2023-03-03T08:32:43.574 INFO:root:teuthology version: 0.0.1.dev49+gc7c3cc8 2023-03-03T08:32:43.576 DEBUG:teuthology.run:Teuthology command: teuthology --name vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi --archive /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192252 --description fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} --owner scheduled_vshankar@teuthology --verbose -- /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192252/orig.config.yaml 2023-03-03T08:32:43.626 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-03-03T08:32:43.707 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192252 branch: wip-vshankar-testing-20230303.023823 description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} email: vshankar@redhat.com first_in_suite: false job_id: '7192252' kernel: client: sha1: distro flavor: default kdb: true sha1: 75534f74ac3dd06f86697ff862e8be2945b3c8e6 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-vshankar-testing-20230303.023823 ceph: cephfs: fs: - name: a - name: b session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: cephfs_name: a client.1: cephfs_name: b install: ceph: flavor: default sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b kclient: client.0: cephfs_name: a client.1: cephfs_name: b syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20230303.023823 sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b sleep_before_teardown: 0 suite: fs suite_branch: wip-vshankar-testing-20230303.023823 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi143.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6 timestamp: 2023-03-03_04:39:14 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.31660 2023-03-03T08:32:43.707 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa; will attempt to use it 2023-03-03T08:32:43.708 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa/tasks 2023-03-03T08:32:43.709 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-03-03T08:32:43.710 INFO:teuthology.task.internal:Checking packages... 2023-03-03T08:32:43.742 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '472aaef4ec3b3a2c29f73ca67854785a910f1f9b' 2023-03-03T08:32:43.743 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-03-03T08:32:43.743 INFO:teuthology.packaging:ref: None 2023-03-03T08:32:43.744 INFO:teuthology.packaging:tag: None 2023-03-03T08:32:43.744 INFO:teuthology.packaging:branch: wip-vshankar-testing-20230303.023823 2023-03-03T08:32:43.744 INFO:teuthology.packaging:sha1: 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T08:32:43.744 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-20230303.023823 2023-03-03T08:32:43.902 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-2739.g472aaef4 2023-03-03T08:32:43.902 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-03-03T08:32:43.909 INFO:teuthology.task.internal:no buildpackages task found 2023-03-03T08:32:43.909 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-03-03T08:32:43.915 INFO:teuthology.task.internal:Saving configuration 2023-03-03T08:32:43.934 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-03-03T08:32:43.941 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-03T08:32:43.971 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192252', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-03 08:27:30.258116', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-03T08:32:43.996 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192252', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-03 08:27:30.259916', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-03T08:32:43.996 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-03-03T08:32:44.008 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-03-03T08:32:44.008 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2023-03-03T08:32:44.009 INFO:teuthology.run_tasks:Running task console_log... 2023-03-03T08:32:44.091 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa663466550>, signals=[15]) 2023-03-03T08:32:44.094 INFO:teuthology.run_tasks:Running task internal.connect... 2023-03-03T08:32:44.103 INFO:teuthology.task.internal:Opening connections... 2023-03-03T08:32:44.103 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2023-03-03T08:32:44.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T08:32:44.175 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2023-03-03T08:32:44.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T08:32:44.244 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-03-03T08:32:44.251 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2023-03-03T08:32:44.272 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2023-03-03T08:32:44.273 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2023-03-03T08:32:44.333 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Red Hat Enterprise Linux" 2023-03-03T08:32:44.334 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="8.6 (Ootpa)" 2023-03-03T08:32:44.334 INFO:teuthology.orchestra.run.smithi007.stdout:ID="rhel" 2023-03-03T08:32:44.334 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="fedora" 2023-03-03T08:32:44.334 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="8.6" 2023-03-03T08:32:44.334 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el8" 2023-03-03T08:32:44.334 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-03T08:32:44.335 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2023-03-03T08:32:44.335 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-03T08:32:44.335 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.redhat.com/" 2023-03-03T08:32:44.335 INFO:teuthology.orchestra.run.smithi007.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-03T08:32:44.335 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-03T08:32:44.335 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:32:44.335 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-03T08:32:44.336 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-03T08:32:44.336 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-03T08:32:44.336 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-03T08:32:44.337 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2023-03-03T08:32:44.363 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2023-03-03T08:32:44.383 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2023-03-03T08:32:44.384 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2023-03-03T08:32:44.442 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="Red Hat Enterprise Linux" 2023-03-03T08:32:44.442 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="8.6 (Ootpa)" 2023-03-03T08:32:44.443 INFO:teuthology.orchestra.run.smithi143.stdout:ID="rhel" 2023-03-03T08:32:44.443 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="fedora" 2023-03-03T08:32:44.443 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="8.6" 2023-03-03T08:32:44.443 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el8" 2023-03-03T08:32:44.443 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-03T08:32:44.443 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2023-03-03T08:32:44.443 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-03T08:32:44.444 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://www.redhat.com/" 2023-03-03T08:32:44.444 INFO:teuthology.orchestra.run.smithi143.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-03T08:32:44.444 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-03T08:32:44.444 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:32:44.444 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-03T08:32:44.444 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-03T08:32:44.444 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-03T08:32:44.444 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-03T08:32:44.445 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2023-03-03T08:32:44.471 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-03-03T08:32:44.479 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-03-03T08:32:44.486 INFO:teuthology.task.internal:Checking for old test directory... 2023-03-03T08:32:44.486 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2023-03-03T08:32:44.488 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2023-03-03T08:32:44.506 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-03-03T08:32:44.518 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-03-03T08:32:44.519 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2023-03-03T08:32:44.548 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2023-03-03T08:32:44.577 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-03-03T08:32:44.641 INFO:teuthology.run_tasks:Running task kernel... 2023-03-03T08:32:44.658 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'} 2023-03-03T08:32:44.658 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}} 2023-03-03T08:32:44.659 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-03-03T08:32:44.659 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-03T08:32:44.660 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-03T08:32:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:32:44.678 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2023-03-03T08:32:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:32:44.679 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-03-03T08:32:44.736 INFO:teuthology.orchestra.run.smithi143.stdout:4.18.0-372.9.1.el8.x86_64 2023-03-03T08:32:44.736 INFO:teuthology.task.kernel:Running kernel on smithi143: 4.18.0-372.9.1.el8.x86_64 2023-03-03T08:32:44.736 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum install -y kernel 2023-03-03T08:32:44.739 INFO:teuthology.orchestra.run.smithi007.stdout:4.18.0-372.9.1.el8.x86_64 2023-03-03T08:32:44.740 INFO:teuthology.task.kernel:Running kernel on smithi007: 4.18.0-372.9.1.el8.x86_64 2023-03-03T08:32:44.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2023-03-03T08:32:45.928 INFO:teuthology.orchestra.run.smithi143.stdout:Updating Subscription Management repositories. 2023-03-03T08:32:45.928 INFO:teuthology.orchestra.run.smithi143.stdout:Unable to read consumer identity 2023-03-03T08:32:46.005 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-03-03T08:32:46.006 INFO:teuthology.orchestra.run.smithi007.stdout:Unable to read consumer identity 2023-03-03T08:32:49.099 INFO:teuthology.orchestra.run.smithi143.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2023-03-03T08:32:49.600 INFO:teuthology.orchestra.run.smithi007.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-03-03T08:32:50.107 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 14 MB/s | 14 MB 00:00 2023-03-03T08:32:50.625 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 14 MB/s | 14 MB 00:01 2023-03-03T08:32:53.124 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 466 kB/s | 24 kB 00:00 2023-03-03T08:32:53.779 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-03-03T08:32:53.830 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2023-03-03T08:32:53.830 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2023-03-03T08:32:53.831 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2023-03-03T08:32:53.883 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 481 kB/s | 24 kB 00:00 2023-03-03T08:32:53.967 DEBUG:teuthology.orchestra.run.smithi143:> echo no | sudo yum reinstall kernel || true 2023-03-03T08:32:54.337 INFO:teuthology.orchestra.run.smithi143.stdout:Updating Subscription Management repositories. 2023-03-03T08:32:54.338 INFO:teuthology.orchestra.run.smithi143.stdout:Unable to read consumer identity 2023-03-03T08:32:54.561 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-03-03T08:32:54.591 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:01 ago on Fri 03 Mar 2023 08:32:53 AM UTC. 2023-03-03T08:32:54.611 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-03-03T08:32:54.612 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-03-03T08:32:54.612 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-03-03T08:32:54.634 INFO:teuthology.orchestra.run.smithi143.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-03T08:32:54.669 INFO:teuthology.orchestra.run.smithi143.stderr:Error: No packages marked for reinstall. 2023-03-03T08:32:54.715 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum reinstall -y kernel || true 2023-03-03T08:32:54.727 DEBUG:teuthology.orchestra.run.smithi007:> echo no | sudo yum reinstall kernel || true 2023-03-03T08:32:55.045 INFO:teuthology.orchestra.run.smithi143.stdout:Updating Subscription Management repositories. 2023-03-03T08:32:55.045 INFO:teuthology.orchestra.run.smithi143.stdout:Unable to read consumer identity 2023-03-03T08:32:55.075 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-03-03T08:32:55.075 INFO:teuthology.orchestra.run.smithi007.stdout:Unable to read consumer identity 2023-03-03T08:32:55.300 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:02 ago on Fri 03 Mar 2023 08:32:53 AM UTC. 2023-03-03T08:32:55.340 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:02 ago on Fri 03 Mar 2023 08:32:53 AM UTC. 2023-03-03T08:32:55.342 INFO:teuthology.orchestra.run.smithi143.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-03T08:32:55.370 INFO:teuthology.orchestra.run.smithi143.stderr:Error: No packages marked for reinstall. 2023-03-03T08:32:55.392 INFO:teuthology.orchestra.run.smithi007.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-03T08:32:55.421 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel --last | head -n 1 2023-03-03T08:32:55.429 INFO:teuthology.orchestra.run.smithi007.stderr:Error: No packages marked for reinstall. 2023-03-03T08:32:55.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum reinstall -y kernel || true 2023-03-03T08:32:55.500 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2023-03-03T08:32:55.502 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-03-03T08:32:55.502 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-03-03T08:32:55.502 DEBUG:teuthology.parallel:result is None 2023-03-03T08:32:55.824 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-03-03T08:32:55.825 INFO:teuthology.orchestra.run.smithi007.stdout:Unable to read consumer identity 2023-03-03T08:32:56.085 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:03 ago on Fri 03 Mar 2023 08:32:53 AM UTC. 2023-03-03T08:32:56.128 INFO:teuthology.orchestra.run.smithi007.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-03T08:32:56.154 INFO:teuthology.orchestra.run.smithi007.stderr:Error: No packages marked for reinstall. 2023-03-03T08:32:56.206 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel --last | head -n 1 2023-03-03T08:32:56.288 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2023-03-03T08:32:56.290 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-03-03T08:32:56.290 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-03-03T08:32:56.290 DEBUG:teuthology.parallel:result is None 2023-03-03T08:32:56.290 INFO:teuthology.run_tasks:Running task internal.base... 2023-03-03T08:32:56.301 INFO:teuthology.task.internal:Creating test directory... 2023-03-03T08:32:56.301 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-03T08:32:56.303 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-03T08:32:56.324 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-03-03T08:32:56.338 INFO:teuthology.run_tasks:Running task internal.archive... 2023-03-03T08:32:56.347 INFO:teuthology.task.internal:Creating archive directory... 2023-03-03T08:32:56.347 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-03T08:32:56.360 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-03T08:32:56.400 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-03-03T08:32:56.413 INFO:teuthology.task.internal:Enabling coredump saving... 2023-03-03T08:32:56.413 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-03T08:32:56.418 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-03T08:32:56.476 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T08:32:56.477 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T08:32:56.487 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T08:32:56.489 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-03T08:32:56.491 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-03-03T08:32:56.500 INFO:teuthology.task.internal:Configuring sudo... 2023-03-03T08:32:56.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-03T08:32:56.502 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-03T08:32:56.561 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-03-03T08:32:56.569 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-03-03T08:32:56.569 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-03T08:32:56.571 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-03T08:32:56.619 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T08:32:56.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T08:32:56.747 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T08:32:56.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T08:32:56.833 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:32:56.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-03T08:32:56.900 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T08:32:56.957 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-03T08:32:57.009 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T08:32:57.059 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-03T08:32:57.098 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:32:57.098 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-03T08:32:57.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2023-03-03T08:32:57.171 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2023-03-03T08:32:57.226 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-03T08:32:57.268 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-03T08:32:57.664 INFO:teuthology.run_tasks:Running task internal.timer... 2023-03-03T08:32:57.672 INFO:teuthology.task.internal:Starting timer... 2023-03-03T08:32:57.672 INFO:teuthology.run_tasks:Running task pcp... 2023-03-03T08:32:57.688 INFO:teuthology.run_tasks:Running task selinux... 2023-03-03T08:32:57.696 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-03-03T08:32:57.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2023-03-03T08:32:57.756 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2023-03-03T08:32:57.758 DEBUG:teuthology.orchestra.run.smithi143:> sudo service auditd rotate 2023-03-03T08:32:57.834 INFO:teuthology.orchestra.run.smithi143.stdout:Rotating logs: 2023-03-03T08:32:57.836 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-03-03T08:32:57.836 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2023-03-03T08:32:57.857 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2023-03-03T08:32:57.857 DEBUG:teuthology.orchestra.run.smithi143:> /usr/sbin/getenforce 2023-03-03T08:32:57.877 INFO:teuthology.orchestra.run.smithi143.stdout:Permissive 2023-03-03T08:32:57.879 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi143.front.sepia.ceph.com': 'permissive'} 2023-03-03T08:32:57.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-03T08:32:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:32:57.925 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-03T08:32:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:32:57.952 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-03-03T08:32:57.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2023-03-03T08:32:57.992 DEBUG:teuthology.orchestra.run.smithi143:> sudo /usr/sbin/setenforce permissive 2023-03-03T08:32:58.018 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-03-03T08:32:58.029 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-03-03T08:32:58.038 INFO:teuthology.repo_utils:Fetching main from origin 2023-03-03T08:32:58.238 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-03-03T08:32:58.255 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-03-03T08:32:58.257 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi143.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-03-03T08:40:17.569 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi143.front.sepia.ceph.com')] 2023-03-03T08:40:17.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-03-03T08:40:17.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T08:40:17.642 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-03-03T08:40:17.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2023-03-03T08:40:17.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-03T08:40:17.736 DEBUG:teuthology.orchestra.run.smithi143:> true 2023-03-03T08:40:17.802 INFO:teuthology.run_tasks:Running task clock... 2023-03-03T08:40:17.814 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-03-03T08:40:17.814 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-03T08:40:17.815 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-03T08:40:17.817 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-03T08:40:17.818 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-03T08:40:17.848 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-03T08:40:17.864 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-03T08:40:17.873 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-03T08:40:17.887 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-03T08:40:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2023-03-03T08:40:17.948 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2023-03-03T08:40:17.959 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: ntpd: command not found 2023-03-03T08:40:17.964 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-03T08:40:17.971 INFO:teuthology.orchestra.run.smithi143.stdout:506 Cannot talk to daemon 2023-03-03T08:40:17.978 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-03T08:40:17.985 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-03T08:40:17.999 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-03T08:40:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:bash: ntpq: command not found 2023-03-03T08:40:18.056 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-03T08:40:18.056 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2023-03-03T08:40:18.056 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T08:40:18.056 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T08:40:18.057 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T08:40:18.057 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T08:40:18.078 INFO:teuthology.orchestra.run.smithi143.stderr:bash: ntpq: command not found 2023-03-03T08:40:18.081 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-03T08:40:18.081 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2023-03-03T08:40:18.081 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T08:40:18.081 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T08:40:18.081 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T08:40:18.082 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-03T08:40:18.082 INFO:teuthology.run_tasks:Running task install... 2023-03-03T08:40:18.091 DEBUG:teuthology.task.install:project ceph 2023-03-03T08:40:18.092 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}} 2023-03-03T08:40:18.092 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'} 2023-03-03T08:40:18.092 INFO:teuthology.task.install:Using flavor: default 2023-03-03T08:40:18.098 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-03-03T08:40:18.098 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-03-03T08:40:18.099 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'tag': None, 'wait_for_package': False} 2023-03-03T08:40:18.100 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T08:40:18.102 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'tag': None, 'wait_for_package': False} 2023-03-03T08:40:18.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T08:40:18.242 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/ 2023-03-03T08:40:18.242 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2739.g472aaef4 2023-03-03T08:40:18.246 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/ 2023-03-03T08:40:18.246 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2739.g472aaef4 2023-03-03T08:40:18.345 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-03T08:40:18.345 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:40:18.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-03T08:40:18.349 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20230303.023823/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-03T08:40:18.349 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:40:18.349 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-03T08:40:18.375 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-03T08:40:18.376 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-03T08:40:18.379 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-03T08:40:18.379 DEBUG:teuthology.orchestra.run.smithi143:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/472aaef4ec3b3a2c29f73ca67854785a910f1f9b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-03T08:40:18.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-03T08:40:18.447 DEBUG:teuthology.orchestra.run.smithi143:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-03T08:40:18.524 DEBUG:teuthology.orchestra.run.smithi143:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-03T08:40:18.525 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-03T08:40:18.590 INFO:teuthology.orchestra.run.smithi143.stdout:check_obsoletes = 1 2023-03-03T08:40:18.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum clean all 2023-03-03T08:40:18.592 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2023-03-03T08:40:18.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2023-03-03T08:40:18.961 INFO:teuthology.orchestra.run.smithi143.stdout:Updating Subscription Management repositories. 2023-03-03T08:40:18.984 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-03-03T08:40:19.194 INFO:teuthology.orchestra.run.smithi143.stdout:44 files removed 2023-03-03T08:40:19.213 INFO:teuthology.orchestra.run.smithi007.stdout:44 files removed 2023-03-03T08:40:19.240 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-03T08:40:19.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-03T08:40:19.569 INFO:teuthology.orchestra.run.smithi143.stdout:Updating Subscription Management repositories. 2023-03-03T08:40:19.612 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-03-03T08:40:20.161 INFO:teuthology.orchestra.run.smithi143.stdout:ceph packages for x86_64 320 kB/s | 80 kB 00:00 2023-03-03T08:40:20.256 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 282 kB/s | 80 kB 00:00 2023-03-03T08:40:20.437 INFO:teuthology.orchestra.run.smithi143.stdout:ceph noarch packages 39 kB/s | 9.4 kB 00:00 2023-03-03T08:40:20.524 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 40 kB/s | 9.4 kB 00:00 2023-03-03T08:40:20.688 INFO:teuthology.orchestra.run.smithi143.stdout:ceph source packages 6.4 kB/s | 1.5 kB 00:00 2023-03-03T08:40:20.808 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 5.7 kB/s | 1.5 kB 00:00 2023-03-03T08:40:23.092 INFO:teuthology.orchestra.run.smithi007.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-03-03T08:40:23.097 INFO:teuthology.orchestra.run.smithi143.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-03-03T08:40:24.056 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 15 MB/s | 14 MB 00:00 2023-03-03T08:40:24.134 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 13 MB/s | 14 MB 00:01 2023-03-03T08:40:27.156 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 344 kB/s | 24 kB 00:00 2023-03-03T08:40:27.451 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 346 kB/s | 24 kB 00:00 2023-03-03T08:40:28.083 INFO:teuthology.orchestra.run.smithi143.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 53 MB 00:00 2023-03-03T08:40:28.378 INFO:teuthology.orchestra.run.smithi007.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 55 MB/s | 47 MB 00:00 2023-03-03T08:40:36.209 INFO:teuthology.orchestra.run.smithi143.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 MB/s | 47 MB 00:00 2023-03-03T08:40:36.887 INFO:teuthology.orchestra.run.smithi007.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 57 MB/s | 53 MB 00:00 2023-03-03T08:40:47.025 INFO:teuthology.orchestra.run.smithi143.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T08:40:47.026 INFO:teuthology.orchestra.run.smithi143.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T08:40:47.041 INFO:teuthology.orchestra.run.smithi143.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-03T08:40:47.042 INFO:teuthology.orchestra.run.smithi143.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-03T08:40:47.043 INFO:teuthology.orchestra.run.smithi143.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-03T08:40:47.044 INFO:teuthology.orchestra.run.smithi143.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-03T08:40:47.047 INFO:teuthology.orchestra.run.smithi143.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-03T08:40:47.050 INFO:teuthology.orchestra.run.smithi143.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-03T08:40:47.051 INFO:teuthology.orchestra.run.smithi143.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-03T08:40:47.187 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2023-03-03T08:40:47.199 INFO:teuthology.orchestra.run.smithi143.stdout:=========================================================================================================================== 2023-03-03T08:40:47.200 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2023-03-03T08:40:47.200 INFO:teuthology.orchestra.run.smithi143.stdout:=========================================================================================================================== 2023-03-03T08:40:47.200 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2023-03-03T08:40:47.200 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-03T08:40:47.200 INFO:teuthology.orchestra.run.smithi143.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-03T08:40:47.200 INFO:teuthology.orchestra.run.smithi143.stdout: ceph x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.4 k 2023-03-03T08:40:47.200 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 5.4 M 2023-03-03T08:40:47.201 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 845 k 2023-03-03T08:40:47.201 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 166 k 2023-03-03T08:40:47.201 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 1.5 M 2023-03-03T08:40:47.201 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 123 k 2023-03-03T08:40:47.201 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 1.7 M 2023-03-03T08:40:47.201 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 7.4 M 2023-03-03T08:40:47.201 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 47 k 2023-03-03T08:40:47.202 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 13 M 2023-03-03T08:40:47.202 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 47 M 2023-03-03T08:40:47.202 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 263 k 2023-03-03T08:40:47.202 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 83 k 2023-03-03T08:40:47.202 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 257 k 2023-03-03T08:40:47.202 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-top noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 18 k 2023-03-03T08:40:47.202 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-03T08:40:47.202 INFO:teuthology.orchestra.run.smithi143.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-03T08:40:47.203 INFO:teuthology.orchestra.run.smithi143.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-03T08:40:47.203 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 30 k 2023-03-03T08:40:47.203 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 767 k 2023-03-03T08:40:47.203 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 139 k 2023-03-03T08:40:47.203 INFO:teuthology.orchestra.run.smithi143.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-03T08:40:47.203 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-03T08:40:47.203 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-03T08:40:47.203 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-03T08:40:47.204 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-03T08:40:47.204 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-03T08:40:47.204 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 190 k 2023-03-03T08:40:47.204 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 371 k 2023-03-03T08:40:47.204 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 364 k 2023-03-03T08:40:47.204 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 107 k 2023-03-03T08:40:47.204 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 94 k 2023-03-03T08:40:47.205 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.4 M 2023-03-03T08:40:47.205 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 180 k 2023-03-03T08:40:47.205 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-03T08:40:47.205 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrading: 2023-03-03T08:40:47.205 INFO:teuthology.orchestra.run.smithi143.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-03T08:40:47.205 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.7 M 2023-03-03T08:40:47.205 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.6 M 2023-03-03T08:40:47.205 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2023-03-03T08:40:47.206 INFO:teuthology.orchestra.run.smithi143.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-03T08:40:47.206 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 22 M 2023-03-03T08:40:47.206 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 23 k 2023-03-03T08:40:47.206 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 2.3 M 2023-03-03T08:40:47.206 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 236 k 2023-03-03T08:40:47.206 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 4.6 M 2023-03-03T08:40:47.206 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 18 M 2023-03-03T08:40:47.207 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 15 k 2023-03-03T08:40:47.207 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 24 k 2023-03-03T08:40:47.207 INFO:teuthology.orchestra.run.smithi143.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-03T08:40:47.207 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-03T08:40:47.207 INFO:teuthology.orchestra.run.smithi143.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-03T08:40:47.207 INFO:teuthology.orchestra.run.smithi143.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-03T08:40:47.207 INFO:teuthology.orchestra.run.smithi143.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-03T08:40:47.207 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 137 k 2023-03-03T08:40:47.208 INFO:teuthology.orchestra.run.smithi143.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-03T08:40:47.208 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-03T08:40:47.208 INFO:teuthology.orchestra.run.smithi143.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-03T08:40:47.208 INFO:teuthology.orchestra.run.smithi143.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-03T08:40:47.208 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-03T08:40:47.208 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-03T08:40:47.208 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 474 k 2023-03-03T08:40:47.208 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-03T08:40:47.209 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.5 M 2023-03-03T08:40:47.209 INFO:teuthology.orchestra.run.smithi143.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-03T08:40:47.209 INFO:teuthology.orchestra.run.smithi143.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-03T08:40:47.209 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-03T08:40:47.209 INFO:teuthology.orchestra.run.smithi143.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-03T08:40:47.209 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-03T08:40:47.209 INFO:teuthology.orchestra.run.smithi143.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-03T08:40:47.210 INFO:teuthology.orchestra.run.smithi143.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-03T08:40:47.210 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-03T08:40:47.210 INFO:teuthology.orchestra.run.smithi143.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-03T08:40:47.210 INFO:teuthology.orchestra.run.smithi143.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-03T08:40:47.210 INFO:teuthology.orchestra.run.smithi143.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-03T08:40:47.210 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-03T08:40:47.210 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-03T08:40:47.210 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-03T08:40:47.211 INFO:teuthology.orchestra.run.smithi143.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-03T08:40:47.211 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-03T08:40:47.211 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 45 k 2023-03-03T08:40:47.211 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 117 k 2023-03-03T08:40:47.211 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-03T08:40:47.211 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-03T08:40:47.211 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-03T08:40:47.211 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-03T08:40:47.212 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-03T08:40:47.212 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-03T08:40:47.212 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-03T08:40:47.212 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-03T08:40:47.212 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-03T08:40:47.212 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-03T08:40:47.212 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-03T08:40:47.212 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-03T08:40:47.213 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-03T08:40:47.213 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-03T08:40:47.213 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-03T08:40:47.213 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-03T08:40:47.213 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-03T08:40:47.213 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-03T08:40:47.213 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-03T08:40:47.214 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-03T08:40:47.214 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-03T08:40:47.214 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-03T08:40:47.214 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-03T08:40:47.214 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-03T08:40:47.214 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-03T08:40:47.214 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-03T08:40:47.214 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-03T08:40:47.215 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-03T08:40:47.215 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-03T08:40:47.215 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-03T08:40:47.215 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-03T08:40:47.215 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-03T08:40:47.215 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-03T08:40:47.215 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-03T08:40:47.215 INFO:teuthology.orchestra.run.smithi143.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-03T08:40:47.216 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-03T08:40:47.216 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-03T08:40:47.216 INFO:teuthology.orchestra.run.smithi143.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-03T08:40:47.216 INFO:teuthology.orchestra.run.smithi143.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-03T08:40:47.216 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-03T08:40:47.216 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-03T08:40:47.216 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-03T08:40:47.216 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-03T08:40:47.217 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-03T08:40:47.217 INFO:teuthology.orchestra.run.smithi143.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-03T08:40:47.217 INFO:teuthology.orchestra.run.smithi143.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-03T08:40:47.217 INFO:teuthology.orchestra.run.smithi143.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-03T08:40:47.217 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-03T08:40:47.217 INFO:teuthology.orchestra.run.smithi143.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-03T08:40:47.217 INFO:teuthology.orchestra.run.smithi143.stdout:Enabling module streams: 2023-03-03T08:40:47.218 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql 10 2023-03-03T08:40:47.218 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:40:47.218 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2023-03-03T08:40:47.218 INFO:teuthology.orchestra.run.smithi143.stdout:=========================================================================================================================== 2023-03-03T08:40:47.218 INFO:teuthology.orchestra.run.smithi143.stdout:Install 128 Packages 2023-03-03T08:40:47.218 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrade 3 Packages 2023-03-03T08:40:47.218 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:40:47.219 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 203 M 2023-03-03T08:40:47.219 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2023-03-03T08:40:47.351 INFO:teuthology.orchestra.run.smithi143.stdout:(1/131): ceph-18.0.0-2739.g472aaef4.el8.x86_64. 47 kB/s | 6.4 kB 00:00 2023-03-03T08:40:47.543 INFO:teuthology.orchestra.run.smithi143.stdout:(2/131): ceph-fuse-18.0.0-2739.g472aaef4.el8.x8 4.3 MB/s | 845 kB 00:00 2023-03-03T08:40:47.602 INFO:teuthology.orchestra.run.smithi143.stdout:(3/131): ceph-immutable-object-cache-18.0.0-273 2.8 MB/s | 166 kB 00:00 2023-03-03T08:40:47.960 INFO:teuthology.orchestra.run.smithi143.stdout:(4/131): ceph-mds-18.0.0-2739.g472aaef4.el8.x86 6.3 MB/s | 2.3 MB 00:00 2023-03-03T08:40:48.035 INFO:teuthology.orchestra.run.smithi143.stdout:(5/131): ceph-base-18.0.0-2739.g472aaef4.el8.x8 6.5 MB/s | 5.4 MB 00:00 2023-03-03T08:40:48.219 INFO:teuthology.orchestra.run.smithi143.stdout:(6/131): ceph-mgr-18.0.0-2739.g472aaef4.el8.x86 6.0 MB/s | 1.5 MB 00:00 2023-03-03T08:40:48.561 INFO:teuthology.orchestra.run.smithi143.stdout:(7/131): ceph-mon-18.0.0-2739.g472aaef4.el8.x86 8.8 MB/s | 4.6 MB 00:00 2023-03-03T08:40:48.768 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T08:40:48.769 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-03T08:40:48.784 INFO:teuthology.orchestra.run.smithi007.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-03T08:40:48.785 INFO:teuthology.orchestra.run.smithi007.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-03T08:40:48.787 INFO:teuthology.orchestra.run.smithi007.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-03T08:40:48.788 INFO:teuthology.orchestra.run.smithi007.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-03T08:40:48.790 INFO:teuthology.orchestra.run.smithi007.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-03T08:40:48.794 INFO:teuthology.orchestra.run.smithi007.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-03T08:40:48.795 INFO:teuthology.orchestra.run.smithi007.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-03T08:40:48.936 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-03-03T08:40:48.949 INFO:teuthology.orchestra.run.smithi007.stdout:=========================================================================================================================== 2023-03-03T08:40:48.950 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-03-03T08:40:48.950 INFO:teuthology.orchestra.run.smithi007.stdout:=========================================================================================================================== 2023-03-03T08:40:48.950 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-03-03T08:40:48.950 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-03T08:40:48.950 INFO:teuthology.orchestra.run.smithi007.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-03T08:40:48.950 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.4 k 2023-03-03T08:40:48.950 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 5.4 M 2023-03-03T08:40:48.951 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 845 k 2023-03-03T08:40:48.951 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 166 k 2023-03-03T08:40:48.951 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 1.5 M 2023-03-03T08:40:48.951 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 123 k 2023-03-03T08:40:48.951 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 1.7 M 2023-03-03T08:40:48.951 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 7.4 M 2023-03-03T08:40:48.951 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 47 k 2023-03-03T08:40:48.952 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 13 M 2023-03-03T08:40:48.952 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 47 M 2023-03-03T08:40:48.952 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 263 k 2023-03-03T08:40:48.952 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 83 k 2023-03-03T08:40:48.952 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 257 k 2023-03-03T08:40:48.952 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-top noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 18 k 2023-03-03T08:40:48.952 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-03T08:40:48.953 INFO:teuthology.orchestra.run.smithi007.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-03T08:40:48.953 INFO:teuthology.orchestra.run.smithi007.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-03T08:40:48.953 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 30 k 2023-03-03T08:40:48.953 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 767 k 2023-03-03T08:40:48.953 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 139 k 2023-03-03T08:40:48.953 INFO:teuthology.orchestra.run.smithi007.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-03T08:40:48.953 INFO:teuthology.orchestra.run.smithi007.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-03T08:40:48.953 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-03T08:40:48.954 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-03T08:40:48.954 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-03T08:40:48.954 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-03T08:40:48.954 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 190 k 2023-03-03T08:40:48.954 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 371 k 2023-03-03T08:40:48.954 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 364 k 2023-03-03T08:40:48.954 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 107 k 2023-03-03T08:40:48.954 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 94 k 2023-03-03T08:40:48.955 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.4 M 2023-03-03T08:40:48.955 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 180 k 2023-03-03T08:40:48.955 INFO:teuthology.orchestra.run.smithi007.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-03T08:40:48.955 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2023-03-03T08:40:48.955 INFO:teuthology.orchestra.run.smithi007.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-03T08:40:48.955 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.7 M 2023-03-03T08:40:48.955 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 3.6 M 2023-03-03T08:40:48.956 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-03-03T08:40:48.956 INFO:teuthology.orchestra.run.smithi007.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-03T08:40:48.956 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 22 M 2023-03-03T08:40:48.956 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 23 k 2023-03-03T08:40:48.956 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 2.3 M 2023-03-03T08:40:48.956 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 236 k 2023-03-03T08:40:48.956 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 4.6 M 2023-03-03T08:40:48.956 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 18 M 2023-03-03T08:40:48.957 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2739.g472aaef4.el8 ceph-noarch 15 k 2023-03-03T08:40:48.957 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 24 k 2023-03-03T08:40:48.957 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-03T08:40:48.957 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-03T08:40:48.957 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-03T08:40:48.957 INFO:teuthology.orchestra.run.smithi007.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-03T08:40:48.957 INFO:teuthology.orchestra.run.smithi007.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-03T08:40:48.958 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 137 k 2023-03-03T08:40:48.958 INFO:teuthology.orchestra.run.smithi007.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-03T08:40:48.958 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-03T08:40:48.958 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-03T08:40:48.958 INFO:teuthology.orchestra.run.smithi007.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-03T08:40:48.958 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-03T08:40:48.958 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-03T08:40:48.958 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 474 k 2023-03-03T08:40:48.959 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-03T08:40:48.959 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 6.5 M 2023-03-03T08:40:48.959 INFO:teuthology.orchestra.run.smithi007.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-03T08:40:48.959 INFO:teuthology.orchestra.run.smithi007.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-03T08:40:48.959 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-03T08:40:48.959 INFO:teuthology.orchestra.run.smithi007.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-03T08:40:48.959 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-03T08:40:48.959 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-03T08:40:48.960 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-03T08:40:48.960 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-03T08:40:48.960 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-03T08:40:48.960 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-03T08:40:48.960 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-03T08:40:48.960 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-03T08:40:48.960 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-03T08:40:48.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-03T08:40:48.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-03T08:40:48.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-03T08:40:48.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 45 k 2023-03-03T08:40:48.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:18.0.0-2739.g472aaef4.el8 ceph 117 k 2023-03-03T08:40:48.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-03T08:40:48.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-03T08:40:48.961 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-03T08:40:48.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-03T08:40:48.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-03T08:40:48.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-03T08:40:48.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-03T08:40:48.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-03T08:40:48.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-03T08:40:48.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-03T08:40:48.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-03T08:40:48.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-03T08:40:48.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-03T08:40:48.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-03T08:40:48.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-03T08:40:48.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-03T08:40:48.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-03T08:40:48.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-03T08:40:48.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-03T08:40:48.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-03T08:40:48.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-03T08:40:48.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-03T08:40:48.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-03T08:40:48.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-03T08:40:48.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-03T08:40:48.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-03T08:40:48.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-03T08:40:48.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-03T08:40:48.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-03T08:40:48.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-03T08:40:48.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-03T08:40:48.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-03T08:40:48.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-03T08:40:48.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-03T08:40:48.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-03T08:40:48.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-03T08:40:48.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-03T08:40:48.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-03T08:40:48.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-03T08:40:48.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-03T08:40:48.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-03T08:40:48.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-03T08:40:48.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-03T08:40:48.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-03T08:40:48.967 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-03T08:40:48.967 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-03T08:40:48.967 INFO:teuthology.orchestra.run.smithi007.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-03T08:40:48.968 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-03T08:40:48.968 INFO:teuthology.orchestra.run.smithi007.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-03T08:40:48.968 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling module streams: 2023-03-03T08:40:48.968 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql 10 2023-03-03T08:40:48.968 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:40:48.968 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-03-03T08:40:48.968 INFO:teuthology.orchestra.run.smithi007.stdout:=========================================================================================================================== 2023-03-03T08:40:48.969 INFO:teuthology.orchestra.run.smithi007.stdout:Install 128 Packages 2023-03-03T08:40:48.969 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 3 Packages 2023-03-03T08:40:48.969 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:40:48.969 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 203 M 2023-03-03T08:40:48.970 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-03-03T08:40:49.262 INFO:teuthology.orchestra.run.smithi007.stdout:(1/131): ceph-18.0.0-2739.g472aaef4.el8.x86_64. 22 kB/s | 6.4 kB 00:00 2023-03-03T08:40:49.453 INFO:teuthology.orchestra.run.smithi143.stdout:(8/131): ceph-common-18.0.0-2739.g472aaef4.el8. 9.6 MB/s | 22 MB 00:02 2023-03-03T08:40:49.537 INFO:teuthology.orchestra.run.smithi143.stdout:(9/131): ceph-selinux-18.0.0-2739.g472aaef4.el8 292 kB/s | 24 kB 00:00 2023-03-03T08:40:50.045 INFO:teuthology.orchestra.run.smithi007.stdout:(2/131): ceph-fuse-18.0.0-2739.g472aaef4.el8.x8 1.1 MB/s | 845 kB 00:00 2023-03-03T08:40:50.220 INFO:teuthology.orchestra.run.smithi007.stdout:(3/131): ceph-immutable-object-cache-18.0.0-273 948 kB/s | 166 kB 00:00 2023-03-03T08:40:50.354 INFO:teuthology.orchestra.run.smithi143.stdout:(10/131): ceph-radosgw-18.0.0-2739.g472aaef4.el 7.3 MB/s | 13 MB 00:01 2023-03-03T08:40:50.454 INFO:teuthology.orchestra.run.smithi143.stdout:(11/131): cephfs-mirror-18.0.0-2739.g472aaef4.e 2.5 MB/s | 257 kB 00:00 2023-03-03T08:40:50.679 INFO:teuthology.orchestra.run.smithi143.stdout:(12/131): ceph-osd-18.0.0-2739.g472aaef4.el8.x8 7.2 MB/s | 18 MB 00:02 2023-03-03T08:40:50.696 INFO:teuthology.orchestra.run.smithi143.stdout:(13/131): libcephfs-devel-18.0.0-2739.g472aaef4 126 kB/s | 30 kB 00:00 2023-03-03T08:40:50.788 INFO:teuthology.orchestra.run.smithi143.stdout:(14/131): libcephsqlite-18.0.0-2739.g472aaef4.e 1.5 MB/s | 137 kB 00:00 2023-03-03T08:40:50.838 INFO:teuthology.orchestra.run.smithi143.stdout:(15/131): libcephfs2-18.0.0-2739.g472aaef4.el8. 4.7 MB/s | 767 kB 00:00 2023-03-03T08:40:50.864 INFO:teuthology.orchestra.run.smithi143.stdout:(16/131): librados-devel-18.0.0-2739.g472aaef4. 1.8 MB/s | 139 kB 00:00 2023-03-03T08:40:50.888 INFO:teuthology.orchestra.run.smithi007.stdout:(4/131): ceph-base-18.0.0-2739.g472aaef4.el8.x8 2.8 MB/s | 5.4 MB 00:01 2023-03-03T08:40:50.923 INFO:teuthology.orchestra.run.smithi143.stdout:(17/131): libradosstriper1-18.0.0-2739.g472aaef 5.5 MB/s | 474 kB 00:00 2023-03-03T08:40:51.014 INFO:teuthology.orchestra.run.smithi143.stdout:(18/131): python3-ceph-argparse-18.0.0-2739.g47 486 kB/s | 45 kB 00:00 2023-03-03T08:40:51.098 INFO:teuthology.orchestra.run.smithi143.stdout:(19/131): python3-ceph-common-18.0.0-2739.g472a 1.4 MB/s | 117 kB 00:00 2023-03-03T08:40:51.146 INFO:teuthology.orchestra.run.smithi007.stdout:(5/131): ceph-mds-18.0.0-2739.g472aaef4.el8.x86 2.5 MB/s | 2.3 MB 00:00 2023-03-03T08:40:51.211 INFO:teuthology.orchestra.run.smithi143.stdout:(20/131): python3-cephfs-18.0.0-2739.g472aaef4. 1.6 MB/s | 190 kB 00:00 2023-03-03T08:40:51.438 INFO:teuthology.orchestra.run.smithi007.stdout:(6/131): ceph-mgr-18.0.0-2739.g472aaef4.el8.x86 2.8 MB/s | 1.5 MB 00:00 2023-03-03T08:40:51.899 INFO:teuthology.orchestra.run.smithi143.stdout:(21/131): python3-rados-18.0.0-2739.g472aaef4.e 540 kB/s | 371 kB 00:00 2023-03-03T08:40:51.991 INFO:teuthology.orchestra.run.smithi143.stdout:(22/131): librgw2-18.0.0-2739.g472aaef4.el8.x86 5.8 MB/s | 6.5 MB 00:01 2023-03-03T08:40:52.016 INFO:teuthology.orchestra.run.smithi143.stdout:(23/131): python3-rbd-18.0.0-2739.g472aaef4.el8 3.0 MB/s | 364 kB 00:00 2023-03-03T08:40:52.083 INFO:teuthology.orchestra.run.smithi143.stdout:(24/131): python3-rgw-18.0.0-2739.g472aaef4.el8 1.1 MB/s | 107 kB 00:00 2023-03-03T08:40:52.108 INFO:teuthology.orchestra.run.smithi143.stdout:(25/131): rbd-fuse-18.0.0-2739.g472aaef4.el8.x8 1.0 MB/s | 94 kB 00:00 2023-03-03T08:40:52.217 INFO:teuthology.orchestra.run.smithi143.stdout:(26/131): rbd-nbd-18.0.0-2739.g472aaef4.el8.x86 1.6 MB/s | 180 kB 00:00 2023-03-03T08:40:52.326 INFO:teuthology.orchestra.run.smithi143.stdout:(27/131): ceph-grafana-dashboards-18.0.0-2739.g 212 kB/s | 23 kB 00:00 2023-03-03T08:40:52.426 INFO:teuthology.orchestra.run.smithi143.stdout:(28/131): ceph-mgr-cephadm-18.0.0-2739.g472aaef 1.2 MB/s | 123 kB 00:00 2023-03-03T08:40:52.726 INFO:teuthology.orchestra.run.smithi143.stdout:(29/131): rbd-mirror-18.0.0-2739.g472aaef4.el8. 5.4 MB/s | 3.4 MB 00:00 2023-03-03T08:40:52.813 INFO:teuthology.orchestra.run.smithi007.stdout:(7/131): ceph-mon-18.0.0-2739.g472aaef4.el8.x86 2.8 MB/s | 4.6 MB 00:01 2023-03-03T08:40:52.860 INFO:teuthology.orchestra.run.smithi143.stdout:(30/131): ceph-mgr-dashboard-18.0.0-2739.g472aa 4.0 MB/s | 1.7 MB 00:00 2023-03-03T08:40:52.952 INFO:teuthology.orchestra.run.smithi143.stdout:(31/131): ceph-mgr-modules-core-18.0.0-2739.g47 2.5 MB/s | 236 kB 00:00 2023-03-03T08:40:53.035 INFO:teuthology.orchestra.run.smithi143.stdout:(32/131): ceph-mgr-rook-18.0.0-2739.g472aaef4.e 570 kB/s | 47 kB 00:00 2023-03-03T08:40:53.119 INFO:teuthology.orchestra.run.smithi143.stdout:(33/131): ceph-prometheus-alerts-18.0.0-2739.g4 174 kB/s | 15 kB 00:00 2023-03-03T08:40:53.211 INFO:teuthology.orchestra.run.smithi143.stdout:(34/131): ceph-volume-18.0.0-2739.g472aaef4.el8 2.8 MB/s | 263 kB 00:00 2023-03-03T08:40:53.295 INFO:teuthology.orchestra.run.smithi143.stdout:(35/131): cephadm-18.0.0-2739.g472aaef4.el8.noa 994 kB/s | 83 kB 00:00 2023-03-03T08:40:53.780 INFO:teuthology.orchestra.run.smithi143.stdout:(36/131): ceph-test-18.0.0-2739.g472aaef4.el8.x 11 MB/s | 47 MB 00:04 2023-03-03T08:40:53.804 INFO:teuthology.orchestra.run.smithi143.stdout:(37/131): cephfs-top-18.0.0-2739.g472aaef4.el8. 35 kB/s | 18 kB 00:00 2023-03-03T08:40:53.905 INFO:teuthology.orchestra.run.smithi143.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 6.3 MB/s | 7.4 MB 00:01 2023-03-03T08:40:54.004 INFO:teuthology.orchestra.run.smithi143.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 294 kB/s | 59 kB 00:00 2023-03-03T08:40:54.038 INFO:teuthology.orchestra.run.smithi143.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 563 kB/s | 75 kB 00:00 2023-03-03T08:40:54.072 INFO:teuthology.orchestra.run.smithi143.stdout:(41/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2023-03-03T08:40:54.089 INFO:teuthology.orchestra.run.smithi143.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 865 kB/s | 44 kB 00:00 2023-03-03T08:40:54.139 INFO:teuthology.orchestra.run.smithi143.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 660 kB/s | 33 kB 00:00 2023-03-03T08:40:54.164 INFO:teuthology.orchestra.run.smithi143.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.0 MB/s | 185 kB 00:00 2023-03-03T08:40:54.190 INFO:teuthology.orchestra.run.smithi143.stdout:(45/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.5 MB/s | 476 kB 00:00 2023-03-03T08:40:54.215 INFO:teuthology.orchestra.run.smithi143.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 216 kB/s | 16 kB 00:00 2023-03-03T08:40:54.249 INFO:teuthology.orchestra.run.smithi143.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.0 MB/s | 173 kB 00:00 2023-03-03T08:40:54.274 INFO:teuthology.orchestra.run.smithi143.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2023-03-03T08:40:54.307 INFO:teuthology.orchestra.run.smithi143.stdout:(49/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.2 MB/s | 384 kB 00:00 2023-03-03T08:40:54.333 INFO:teuthology.orchestra.run.smithi143.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 132 kB/s | 11 kB 00:00 2023-03-03T08:40:54.358 INFO:teuthology.orchestra.run.smithi143.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 210 kB/s | 18 kB 00:00 2023-03-03T08:40:54.384 INFO:teuthology.orchestra.run.smithi143.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 972 kB/s | 49 kB 00:00 2023-03-03T08:40:54.409 INFO:teuthology.orchestra.run.smithi143.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-03-03T08:40:54.434 INFO:teuthology.orchestra.run.smithi143.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-03-03T08:40:54.493 INFO:teuthology.orchestra.run.smithi143.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 267 kB/s | 16 kB 00:00 2023-03-03T08:40:54.518 INFO:teuthology.orchestra.run.smithi143.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2023-03-03T08:40:54.543 INFO:teuthology.orchestra.run.smithi143.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 670 kB/s | 34 kB 00:00 2023-03-03T08:40:54.594 INFO:teuthology.orchestra.run.smithi143.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2023-03-03T08:40:54.619 INFO:teuthology.orchestra.run.smithi143.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-03-03T08:40:54.653 INFO:teuthology.orchestra.run.smithi143.stdout:(60/131): python3-kubernetes-11.0.0-6.el8.noarc 3.4 MB/s | 1.2 MB 00:00 2023-03-03T08:40:54.678 INFO:teuthology.orchestra.run.smithi143.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 223 kB/s | 19 kB 00:00 2023-03-03T08:40:54.703 INFO:teuthology.orchestra.run.smithi143.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 285 kB/s | 24 kB 00:00 2023-03-03T08:40:54.728 INFO:teuthology.orchestra.run.smithi143.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 388 kB/s | 29 kB 00:00 2023-03-03T08:40:54.730 INFO:teuthology.orchestra.run.smithi007.stdout:(8/131): ceph-osd-18.0.0-2739.g472aaef4.el8.x86 5.4 MB/s | 18 MB 00:03 2023-03-03T08:40:54.754 INFO:teuthology.orchestra.run.smithi143.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 550 kB/s | 27 kB 00:00 2023-03-03T08:40:54.804 INFO:teuthology.orchestra.run.smithi143.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-03-03T08:40:54.829 INFO:teuthology.orchestra.run.smithi143.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2023-03-03T08:40:54.854 INFO:teuthology.orchestra.run.smithi143.stdout:(67/131): python3-waitress-1.2.1-2.el8.1.noarch 1.6 MB/s | 241 kB 00:00 2023-03-03T08:40:54.880 INFO:teuthology.orchestra.run.smithi143.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2023-03-03T08:40:54.906 INFO:teuthology.orchestra.run.smithi143.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 303 kB/s | 23 kB 00:00 2023-03-03T08:40:54.931 INFO:teuthology.orchestra.run.smithi143.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.3 MB/s | 70 kB 00:00 2023-03-03T08:40:54.964 INFO:teuthology.orchestra.run.smithi007.stdout:(9/131): ceph-common-18.0.0-2739.g472aaef4.el8. 3.6 MB/s | 22 MB 00:05 2023-03-03T08:40:54.981 INFO:teuthology.orchestra.run.smithi143.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 624 kB/s | 47 kB 00:00 2023-03-03T08:40:54.989 INFO:teuthology.orchestra.run.smithi007.stdout:(10/131): ceph-selinux-18.0.0-2739.g472aaef4.el 94 kB/s | 24 kB 00:00 2023-03-03T08:40:55.006 INFO:teuthology.orchestra.run.smithi143.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 2.8 MB/s | 200 kB 00:00 2023-03-03T08:40:55.032 INFO:teuthology.orchestra.run.smithi143.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 848 kB/s | 43 kB 00:00 2023-03-03T08:40:55.056 INFO:teuthology.orchestra.run.smithi007.stdout:(11/131): cephfs-mirror-18.0.0-2739.g472aaef4.e 3.8 MB/s | 257 kB 00:00 2023-03-03T08:40:55.057 INFO:teuthology.orchestra.run.smithi143.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-03-03T08:40:55.082 INFO:teuthology.orchestra.run.smithi143.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 955 kB/s | 48 kB 00:00 2023-03-03T08:40:55.089 INFO:teuthology.orchestra.run.smithi007.stdout:(12/131): libcephfs-devel-18.0.0-2739.g472aaef4 922 kB/s | 30 kB 00:00 2023-03-03T08:40:55.107 INFO:teuthology.orchestra.run.smithi143.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-03-03T08:40:55.133 INFO:teuthology.orchestra.run.smithi143.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 850 kB/s | 43 kB 00:00 2023-03-03T08:40:55.158 INFO:teuthology.orchestra.run.smithi143.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2023-03-03T08:40:55.183 INFO:teuthology.orchestra.run.smithi143.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2023-03-03T08:40:55.217 INFO:teuthology.orchestra.run.smithi143.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-03-03T08:40:55.234 INFO:teuthology.orchestra.run.smithi143.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 372 kB/s | 19 kB 00:00 2023-03-03T08:40:55.242 INFO:teuthology.orchestra.run.smithi007.stdout:(13/131): ceph-radosgw-18.0.0-2739.g472aaef4.el 5.4 MB/s | 13 MB 00:02 2023-03-03T08:40:55.260 INFO:teuthology.orchestra.run.smithi143.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2023-03-03T08:40:55.290 INFO:teuthology.orchestra.run.smithi007.stdout:(14/131): libcephfs2-18.0.0-2739.g472aaef4.el8. 3.7 MB/s | 767 kB 00:00 2023-03-03T08:40:55.293 INFO:teuthology.orchestra.run.smithi143.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 9.3 MB/s | 559 kB 00:00 2023-03-03T08:40:55.310 INFO:teuthology.orchestra.run.smithi143.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 357 kB/s | 18 kB 00:00 2023-03-03T08:40:55.341 INFO:teuthology.orchestra.run.smithi007.stdout:(15/131): libcephsqlite-18.0.0-2739.g472aaef4.e 1.3 MB/s | 137 kB 00:00 2023-03-03T08:40:55.352 INFO:teuthology.orchestra.run.smithi143.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2023-03-03T08:40:55.374 INFO:teuthology.orchestra.run.smithi007.stdout:(16/131): librados-devel-18.0.0-2739.g472aaef4. 1.6 MB/s | 139 kB 00:00 2023-03-03T08:40:55.394 INFO:teuthology.orchestra.run.smithi143.stdout:(86/131): thrift-0.13.0-2.el8.x86_64.rpm 3.2 MB/s | 1.7 MB 00:00 2023-03-03T08:40:55.419 INFO:teuthology.orchestra.run.smithi143.stdout:(87/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.1 MB/s | 238 kB 00:00 2023-03-03T08:40:55.441 INFO:teuthology.orchestra.run.smithi007.stdout:(17/131): libradosstriper1-18.0.0-2739.g472aaef 4.7 MB/s | 474 kB 00:00 2023-03-03T08:40:55.461 INFO:teuthology.orchestra.run.smithi143.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.1 MB/s | 72 kB 00:00 2023-03-03T08:40:55.487 INFO:teuthology.orchestra.run.smithi143.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.3 MB/s | 83 kB 00:00 2023-03-03T08:40:55.491 INFO:teuthology.orchestra.run.smithi007.stdout:(18/131): python3-ceph-argparse-18.0.0-2739.g47 892 kB/s | 45 kB 00:00 2023-03-03T08:40:55.512 INFO:teuthology.orchestra.run.smithi143.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.2 MB/s | 60 kB 00:00 2023-03-03T08:40:55.533 INFO:teuthology.orchestra.run.smithi007.stdout:(19/131): python3-ceph-common-18.0.0-2739.g472a 2.8 MB/s | 117 kB 00:00 2023-03-03T08:40:55.537 INFO:teuthology.orchestra.run.smithi143.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.9 MB/s | 98 kB 00:00 2023-03-03T08:40:55.583 INFO:teuthology.orchestra.run.smithi007.stdout:(20/131): python3-cephfs-18.0.0-2739.g472aaef4. 3.7 MB/s | 190 kB 00:00 2023-03-03T08:40:55.588 INFO:teuthology.orchestra.run.smithi143.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 767 kB/s | 58 kB 00:00 2023-03-03T08:40:55.613 INFO:teuthology.orchestra.run.smithi143.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 511 kB/s | 39 kB 00:00 2023-03-03T08:40:55.659 INFO:teuthology.orchestra.run.smithi007.stdout:(21/131): python3-rados-18.0.0-2739.g472aaef4.e 4.8 MB/s | 371 kB 00:00 2023-03-03T08:40:55.664 INFO:teuthology.orchestra.run.smithi143.stdout:(94/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 7.5 MB/s | 2.3 MB 00:00 2023-03-03T08:40:55.697 INFO:teuthology.orchestra.run.smithi143.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 5.8 MB/s | 644 kB 00:00 2023-03-03T08:40:55.723 INFO:teuthology.orchestra.run.smithi143.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.5 MB/s | 171 kB 00:00 2023-03-03T08:40:55.748 INFO:teuthology.orchestra.run.smithi143.stdout:(97/131): NetworkManager-initscripts-updown-1.3 1.6 MB/s | 138 kB 00:00 2023-03-03T08:40:55.768 INFO:teuthology.orchestra.run.smithi007.stdout:(22/131): python3-rbd-18.0.0-2739.g472aaef4.el8 3.3 MB/s | 364 kB 00:00 2023-03-03T08:40:55.773 INFO:teuthology.orchestra.run.smithi143.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2023-03-03T08:40:55.798 INFO:teuthology.orchestra.run.smithi143.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-03-03T08:40:55.810 INFO:teuthology.orchestra.run.smithi007.stdout:(23/131): python3-rgw-18.0.0-2739.g472aaef4.el8 2.5 MB/s | 107 kB 00:00 2023-03-03T08:40:55.823 INFO:teuthology.orchestra.run.smithi143.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2023-03-03T08:40:55.857 INFO:teuthology.orchestra.run.smithi143.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-03-03T08:40:55.860 INFO:teuthology.orchestra.run.smithi007.stdout:(24/131): rbd-fuse-18.0.0-2739.g472aaef4.el8.x8 1.8 MB/s | 94 kB 00:00 2023-03-03T08:40:55.874 INFO:teuthology.orchestra.run.smithi143.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 772 kB/s | 39 kB 00:00 2023-03-03T08:40:55.916 INFO:teuthology.orchestra.run.smithi143.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 11 MB/s | 688 kB 00:00 2023-03-03T08:40:55.950 INFO:teuthology.orchestra.run.smithi143.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.0 MB/s | 538 kB 00:00 2023-03-03T08:40:55.975 INFO:teuthology.orchestra.run.smithi143.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-03-03T08:40:56.001 INFO:teuthology.orchestra.run.smithi143.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-03-03T08:40:56.034 INFO:teuthology.orchestra.run.smithi143.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-03-03T08:40:56.068 INFO:teuthology.orchestra.run.smithi143.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 799 kB/s | 27 kB 00:00 2023-03-03T08:40:56.110 INFO:teuthology.orchestra.run.smithi143.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-03-03T08:40:56.143 INFO:teuthology.orchestra.run.smithi143.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-03-03T08:40:56.194 INFO:teuthology.orchestra.run.smithi007.stdout:(25/131): librgw2-18.0.0-2739.g472aaef4.el8.x86 7.9 MB/s | 6.5 MB 00:00 2023-03-03T08:40:56.235 INFO:teuthology.orchestra.run.smithi143.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-03-03T08:40:56.252 INFO:teuthology.orchestra.run.smithi007.stdout:(26/131): rbd-nbd-18.0.0-2739.g472aaef4.el8.x86 3.0 MB/s | 180 kB 00:00 2023-03-03T08:40:56.277 INFO:teuthology.orchestra.run.smithi143.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 11 MB/s | 457 kB 00:00 2023-03-03T08:40:56.302 INFO:teuthology.orchestra.run.smithi007.stdout:(27/131): ceph-grafana-dashboards-18.0.0-2739.g 462 kB/s | 23 kB 00:00 2023-03-03T08:40:56.319 INFO:teuthology.orchestra.run.smithi143.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-03-03T08:40:56.378 INFO:teuthology.orchestra.run.smithi007.stdout:(28/131): rbd-mirror-18.0.0-2739.g472aaef4.el8. 6.7 MB/s | 3.4 MB 00:00 2023-03-03T08:40:56.379 INFO:teuthology.orchestra.run.smithi143.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 16 MB/s | 3.7 MB 00:00 2023-03-03T08:40:56.403 INFO:teuthology.orchestra.run.smithi007.stdout:(29/131): ceph-mgr-cephadm-18.0.0-2739.g472aaef 1.2 MB/s | 123 kB 00:00 2023-03-03T08:40:56.404 INFO:teuthology.orchestra.run.smithi143.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 765 kB/s | 64 kB 00:00 2023-03-03T08:40:56.428 INFO:teuthology.orchestra.run.smithi143.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-03-03T08:40:56.462 INFO:teuthology.orchestra.run.smithi143.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 341 kB/s | 20 kB 00:00 2023-03-03T08:40:56.487 INFO:teuthology.orchestra.run.smithi143.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.7 MB/s | 103 kB 00:00 2023-03-03T08:40:56.529 INFO:teuthology.orchestra.run.smithi143.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-03-03T08:40:56.571 INFO:teuthology.orchestra.run.smithi143.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.6 MB/s | 198 kB 00:00 2023-03-03T08:40:56.620 INFO:teuthology.orchestra.run.smithi007.stdout:(30/131): ceph-mgr-dashboard-18.0.0-2739.g472aa 7.2 MB/s | 1.7 MB 00:00 2023-03-03T08:40:56.704 INFO:teuthology.orchestra.run.smithi007.stdout:(31/131): ceph-mgr-modules-core-18.0.0-2739.g47 2.8 MB/s | 236 kB 00:00 2023-03-03T08:40:56.754 INFO:teuthology.orchestra.run.smithi007.stdout:(32/131): ceph-mgr-rook-18.0.0-2739.g472aaef4.e 949 kB/s | 47 kB 00:00 2023-03-03T08:40:56.755 INFO:teuthology.orchestra.run.smithi143.stdout:(121/131): python3-scipy-1.0.0-21.module+el8.5. 13 MB/s | 14 MB 00:01 2023-03-03T08:40:56.797 INFO:teuthology.orchestra.run.smithi143.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-03-03T08:40:56.804 INFO:teuthology.orchestra.run.smithi007.stdout:(33/131): ceph-prometheus-alerts-18.0.0-2739.g4 291 kB/s | 15 kB 00:00 2023-03-03T08:40:56.847 INFO:teuthology.orchestra.run.smithi143.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-03-03T08:40:56.880 INFO:teuthology.orchestra.run.smithi007.stdout:(34/131): ceph-volume-18.0.0-2739.g472aaef4.el8 3.4 MB/s | 263 kB 00:00 2023-03-03T08:40:56.914 INFO:teuthology.orchestra.run.smithi143.stdout:(124/131): openblas-threads-0.3.15-3.el8.x86_64 11 MB/s | 4.9 MB 00:00 2023-03-03T08:40:56.938 INFO:teuthology.orchestra.run.smithi007.stdout:(35/131): cephadm-18.0.0-2739.g472aaef4.el8.noa 1.4 MB/s | 83 kB 00:00 2023-03-03T08:40:56.965 INFO:teuthology.orchestra.run.smithi143.stdout:(125/131): postgresql-contrib-10.21-2.module+el 16 MB/s | 808 kB 00:00 2023-03-03T08:40:56.988 INFO:teuthology.orchestra.run.smithi007.stdout:(36/131): cephfs-top-18.0.0-2739.g472aaef4.el8. 361 kB/s | 18 kB 00:00 2023-03-03T08:40:57.032 INFO:teuthology.orchestra.run.smithi143.stdout:(126/131): openblas-0.3.15-3.el8.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2023-03-03T08:40:57.082 INFO:teuthology.orchestra.run.smithi143.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 13 MB/s | 1.5 MB 00:00 2023-03-03T08:40:57.166 INFO:teuthology.orchestra.run.smithi143.stdout:(128/131): postgresql-server-10.21-2.module+el8 16 MB/s | 5.1 MB 00:00 2023-03-03T08:40:57.180 INFO:teuthology.orchestra.run.smithi007.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-03-03T08:40:57.216 INFO:teuthology.orchestra.run.smithi143.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-03-03T08:40:57.273 INFO:teuthology.orchestra.run.smithi007.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 8.5 MB/s | 7.4 MB 00:00 2023-03-03T08:40:57.314 INFO:teuthology.orchestra.run.smithi007.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 441 kB/s | 59 kB 00:00 2023-03-03T08:40:57.423 INFO:teuthology.orchestra.run.smithi007.stdout:(40/131): libunwind-1.3.1-3.el8.x86_64.rpm 501 kB/s | 75 kB 00:00 2023-03-03T08:40:57.448 INFO:teuthology.orchestra.run.smithi007.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 3.5 MB/s | 476 kB 00:00 2023-03-03T08:40:57.465 INFO:teuthology.orchestra.run.smithi007.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2023-03-03T08:40:57.466 INFO:teuthology.orchestra.run.smithi143.stdout:(130/131): librbd1-18.0.0-2739.g472aaef4.el8.x8 9.3 MB/s | 3.6 MB 00:00 2023-03-03T08:40:57.507 INFO:teuthology.orchestra.run.smithi007.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 789 kB/s | 33 kB 00:00 2023-03-03T08:40:57.532 INFO:teuthology.orchestra.run.smithi007.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.2 MB/s | 185 kB 00:00 2023-03-03T08:40:57.550 INFO:teuthology.orchestra.run.smithi007.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 389 kB/s | 16 kB 00:00 2023-03-03T08:40:57.591 INFO:teuthology.orchestra.run.smithi143.stdout:(131/131): librados2-18.0.0-2739.g472aaef4.el8. 6.5 MB/s | 3.7 MB 00:00 2023-03-03T08:40:57.593 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2023-03-03T08:40:57.594 INFO:teuthology.orchestra.run.smithi143.stdout:Total 20 MB/s | 203 MB 00:10 2023-03-03T08:40:57.608 INFO:teuthology.orchestra.run.smithi007.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.2 MB/s | 173 kB 00:00 2023-03-03T08:40:57.658 INFO:teuthology.orchestra.run.smithi007.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2023-03-03T08:40:57.709 INFO:teuthology.orchestra.run.smithi007.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.4 MB/s | 384 kB 00:00 2023-03-03T08:40:57.734 INFO:teuthology.orchestra.run.smithi007.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2023-03-03T08:40:57.759 INFO:teuthology.orchestra.run.smithi007.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 349 kB/s | 18 kB 00:00 2023-03-03T08:40:57.810 INFO:teuthology.orchestra.run.smithi007.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 974 kB/s | 49 kB 00:00 2023-03-03T08:40:57.868 INFO:teuthology.orchestra.run.smithi007.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-03-03T08:40:57.918 INFO:teuthology.orchestra.run.smithi007.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-03-03T08:40:58.002 INFO:teuthology.orchestra.run.smithi007.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2023-03-03T08:40:58.052 INFO:teuthology.orchestra.run.smithi007.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 313 kB/s | 16 kB 00:00 2023-03-03T08:40:58.111 INFO:teuthology.orchestra.run.smithi007.stdout:(56/131): python3-kubernetes-11.0.0-6.el8.noarc 3.1 MB/s | 1.2 MB 00:00 2023-03-03T08:40:58.136 INFO:teuthology.orchestra.run.smithi007.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 402 kB/s | 34 kB 00:00 2023-03-03T08:40:58.187 INFO:teuthology.orchestra.run.smithi007.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2023-03-03T08:40:58.204 INFO:teuthology.orchestra.run.smithi007.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 2.1 MB/s | 196 kB 00:00 2023-03-03T08:40:58.237 INFO:teuthology.orchestra.run.smithi007.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 374 kB/s | 19 kB 00:00 2023-03-03T08:40:58.263 INFO:teuthology.orchestra.run.smithi007.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 408 kB/s | 24 kB 00:00 2023-03-03T08:40:58.426 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2023-03-03T08:40:58.475 INFO:teuthology.orchestra.run.smithi007.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 123 kB/s | 29 kB 00:00 2023-03-03T08:40:58.530 INFO:teuthology.orchestra.run.smithi007.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 103 kB/s | 27 kB 00:00 2023-03-03T08:40:58.546 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2023-03-03T08:40:58.546 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2023-03-03T08:40:58.614 INFO:teuthology.orchestra.run.smithi007.stdout:(64/131): python3-waitress-1.2.1-2.el8.1.noarch 1.7 MB/s | 241 kB 00:00 2023-03-03T08:40:58.639 INFO:teuthology.orchestra.run.smithi007.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2023-03-03T08:40:58.664 INFO:teuthology.orchestra.run.smithi007.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-03-03T08:40:58.689 INFO:teuthology.orchestra.run.smithi007.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-03-03T08:40:58.715 INFO:teuthology.orchestra.run.smithi007.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 457 kB/s | 23 kB 00:00 2023-03-03T08:40:58.740 INFO:teuthology.orchestra.run.smithi007.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-03-03T08:40:59.065 INFO:teuthology.orchestra.run.smithi007.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 42 MB/s | 14 MB 00:00 2023-03-03T08:40:59.107 INFO:teuthology.orchestra.run.smithi007.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 4.2 MB/s | 1.7 MB 00:00 2023-03-03T08:40:59.132 INFO:teuthology.orchestra.run.smithi007.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.6 MB/s | 110 kB 00:00 2023-03-03T08:40:59.183 INFO:teuthology.orchestra.run.smithi007.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 720 kB/s | 54 kB 00:00 2023-03-03T08:40:59.200 INFO:teuthology.orchestra.run.smithi007.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.3 MB/s | 259 kB 00:00 2023-03-03T08:40:59.225 INFO:teuthology.orchestra.run.smithi007.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.5 MB/s | 323 kB 00:00 2023-03-03T08:40:59.242 INFO:teuthology.orchestra.run.smithi007.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 919 kB/s | 39 kB 00:00 2023-03-03T08:40:59.309 INFO:teuthology.orchestra.run.smithi007.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 8.0 MB/s | 688 kB 00:00 2023-03-03T08:40:59.343 INFO:teuthology.orchestra.run.smithi007.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 5.2 MB/s | 538 kB 00:00 2023-03-03T08:40:59.368 INFO:teuthology.orchestra.run.smithi007.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-03-03T08:40:59.393 INFO:teuthology.orchestra.run.smithi007.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-03-03T08:40:59.427 INFO:teuthology.orchestra.run.smithi007.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-03-03T08:40:59.452 INFO:teuthology.orchestra.run.smithi007.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 1.0 MB/s | 27 kB 00:00 2023-03-03T08:40:59.494 INFO:teuthology.orchestra.run.smithi007.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-03-03T08:40:59.528 INFO:teuthology.orchestra.run.smithi007.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 1.3 MB/s | 44 kB 00:00 2023-03-03T08:40:59.604 INFO:teuthology.orchestra.run.smithi007.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 23 MB/s | 4.8 MB 00:00 2023-03-03T08:40:59.641 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2023-03-03T08:40:59.645 INFO:teuthology.orchestra.run.smithi007.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2023-03-03T08:40:59.646 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2023-03-03T08:40:59.704 INFO:teuthology.orchestra.run.smithi007.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.8 MB/s | 225 kB 00:00 2023-03-03T08:40:59.771 INFO:teuthology.orchestra.run.smithi007.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 15 MB/s | 3.7 MB 00:00 2023-03-03T08:40:59.796 INFO:teuthology.orchestra.run.smithi007.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 695 kB/s | 64 kB 00:00 2023-03-03T08:40:59.822 INFO:teuthology.orchestra.run.smithi007.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-03-03T08:40:59.847 INFO:teuthology.orchestra.run.smithi007.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 398 kB/s | 20 kB 00:00 2023-03-03T08:40:59.872 INFO:teuthology.orchestra.run.smithi007.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-03-03T08:40:59.906 INFO:teuthology.orchestra.run.smithi007.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-03-03T08:40:59.948 INFO:teuthology.orchestra.run.smithi007.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 4.7 MB/s | 198 kB 00:00 2023-03-03T08:41:00.077 INFO:teuthology.orchestra.run.smithi007.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 24 MB/s | 4.9 MB 00:00 2023-03-03T08:41:00.124 INFO:teuthology.orchestra.run.smithi007.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.2 MB/s | 202 kB 00:00 2023-03-03T08:41:00.199 INFO:teuthology.orchestra.run.smithi007.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2023-03-03T08:41:00.232 INFO:teuthology.orchestra.run.smithi007.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.2 MB/s | 354 kB 00:00 2023-03-03T08:41:00.283 INFO:teuthology.orchestra.run.smithi007.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 16 MB/s | 808 kB 00:00 2023-03-03T08:41:00.358 INFO:teuthology.orchestra.run.smithi007.stdout:(100/131): postgresql-10.21-2.module+el8.6.0+15 20 MB/s | 1.5 MB 00:00 2023-03-03T08:41:00.425 INFO:teuthology.orchestra.run.smithi007.stdout:(101/131): postgresql-server-10.21-2.module+el8 22 MB/s | 5.1 MB 00:00 2023-03-03T08:41:00.442 INFO:teuthology.orchestra.run.smithi007.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 559 kB/s | 47 kB 00:00 2023-03-03T08:41:00.509 INFO:teuthology.orchestra.run.smithi007.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.3 MB/s | 200 kB 00:00 2023-03-03T08:41:00.526 INFO:teuthology.orchestra.run.smithi007.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 507 kB/s | 43 kB 00:00 2023-03-03T08:41:00.552 INFO:teuthology.orchestra.run.smithi007.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2023-03-03T08:41:00.569 INFO:teuthology.orchestra.run.smithi007.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 1.1 MB/s | 48 kB 00:00 2023-03-03T08:41:00.594 INFO:teuthology.orchestra.run.smithi007.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-03-03T08:41:00.611 INFO:teuthology.orchestra.run.smithi007.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 1.0 MB/s | 43 kB 00:00 2023-03-03T08:41:00.636 INFO:teuthology.orchestra.run.smithi007.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.5 MB/s | 109 kB 00:00 2023-03-03T08:41:00.662 INFO:teuthology.orchestra.run.smithi007.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2023-03-03T08:41:00.695 INFO:teuthology.orchestra.run.smithi007.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-03-03T08:41:00.721 INFO:teuthology.orchestra.run.smithi007.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 319 kB/s | 19 kB 00:00 2023-03-03T08:41:01.163 INFO:teuthology.orchestra.run.smithi007.stdout:(113/131): ceph-test-18.0.0-2739.g472aaef4.el8. 7.5 MB/s | 47 MB 00:06 2023-03-03T08:41:01.188 INFO:teuthology.orchestra.run.smithi007.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 315 kB/s | 155 kB 00:00 2023-03-03T08:41:01.230 INFO:teuthology.orchestra.run.smithi007.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 1.1 MB/s | 559 kB 00:00 2023-03-03T08:41:01.285 INFO:teuthology.orchestra.run.smithi007.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 146 kB/s | 18 kB 00:00 2023-03-03T08:41:01.330 INFO:teuthology.orchestra.run.smithi007.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.3 MB/s | 238 kB 00:00 2023-03-03T08:41:01.364 INFO:teuthology.orchestra.run.smithi007.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.5 MB/s | 605 kB 00:00 2023-03-03T08:41:01.389 INFO:teuthology.orchestra.run.smithi007.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.2 MB/s | 72 kB 00:00 2023-03-03T08:41:01.415 INFO:teuthology.orchestra.run.smithi007.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-03-03T08:41:01.440 INFO:teuthology.orchestra.run.smithi007.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-03-03T08:41:01.465 INFO:teuthology.orchestra.run.smithi007.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-03-03T08:41:01.498 INFO:teuthology.orchestra.run.smithi007.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 987 kB/s | 58 kB 00:00 2023-03-03T08:41:01.524 INFO:teuthology.orchestra.run.smithi007.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 659 kB/s | 39 kB 00:00 2023-03-03T08:41:01.574 INFO:teuthology.orchestra.run.smithi007.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 8.4 MB/s | 644 kB 00:00 2023-03-03T08:41:01.599 INFO:teuthology.orchestra.run.smithi007.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.2 MB/s | 171 kB 00:00 2023-03-03T08:41:01.625 INFO:teuthology.orchestra.run.smithi007.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-03-03T08:41:01.700 INFO:teuthology.orchestra.run.smithi007.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 5.6 MB/s | 2.3 MB 00:00 2023-03-03T08:41:01.750 INFO:teuthology.orchestra.run.smithi007.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-03-03T08:41:02.078 INFO:teuthology.orchestra.run.smithi007.stdout:(130/131): librbd1-18.0.0-2739.g472aaef4.el8.x8 7.9 MB/s | 3.6 MB 00:00 2023-03-03T08:41:02.159 INFO:teuthology.orchestra.run.smithi007.stdout:(131/131): librados2-18.0.0-2739.g472aaef4.el8. 6.5 MB/s | 3.7 MB 00:00 2023-03-03T08:41:02.162 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-03-03T08:41:02.163 INFO:teuthology.orchestra.run.smithi007.stdout:Total 15 MB/s | 203 MB 00:13 2023-03-03T08:41:03.035 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-03-03T08:41:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-03-03T08:41:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-03-03T08:41:04.328 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-03-03T08:41:04.332 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-03-03T08:41:04.432 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2023-03-03T08:41:04.873 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-03T08:41:05.065 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-03T08:41:05.224 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-03-03T08:41:05.350 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-03-03T08:41:05.532 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-03-03T08:41:05.934 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-03-03T08:41:06.140 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 5/134 2023-03-03T08:41:06.529 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 5/134 2023-03-03T08:41:06.732 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T08:41:06.842 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T08:41:07.065 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T08:41:07.164 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T08:41:07.312 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 8/134 2023-03-03T08:41:07.490 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-03-03T08:41:07.702 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-03T08:41:07.901 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-03T08:41:08.324 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-03T08:41:08.561 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-03T08:41:08.723 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 14/134 2023-03-03T08:41:08.830 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 14/134 2023-03-03T08:41:09.006 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 15/134 2023-03-03T08:41:09.148 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 15/134 2023-03-03T08:41:09.360 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-03-03T08:41:09.529 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-03-03T08:41:09.587 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-03-03T08:41:09.681 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-03-03T08:41:09.788 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-03T08:41:09.831 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-03-03T08:41:09.955 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-03-03T08:41:09.970 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-03T08:41:10.191 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-03-03T08:41:10.198 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-03-03T08:41:10.357 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-03T08:41:10.498 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-03T08:41:10.698 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-03T08:41:10.808 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-03-03T08:41:10.998 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-03-03T08:41:11.132 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-03T08:41:11.406 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T08:41:11.584 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-03-03T08:41:11.756 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-03-03T08:41:11.808 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T08:41:12.039 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T08:41:12.173 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T08:41:12.323 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 8/134 2023-03-03T08:41:12.443 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 8/134 2023-03-03T08:41:12.457 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-03-03T08:41:12.633 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 9/134 2023-03-03T08:41:12.732 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-03-03T08:41:12.899 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-03T08:41:12.920 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-03-03T08:41:13.123 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-03T08:41:13.127 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-03T08:41:13.287 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 26/134 2023-03-03T08:41:13.400 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 27/134 2023-03-03T08:41:13.590 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-03T08:41:13.695 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 28/134 2023-03-03T08:41:13.844 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-03T08:41:13.897 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-03-03T08:41:14.047 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 14/134 2023-03-03T08:41:14.154 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 14/134 2023-03-03T08:41:14.314 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 15/134 2023-03-03T08:41:14.417 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 15/134 2023-03-03T08:41:14.539 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-03T08:41:14.682 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-03T08:41:15.081 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-03T08:41:15.680 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-03-03T08:41:15.736 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-03T08:41:15.897 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-03T08:41:16.562 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-03T08:41:16.772 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-03-03T08:41:16.861 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-03-03T08:41:17.190 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-03-03T08:41:17.431 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-03-03T08:41:17.507 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-03-03T08:41:17.663 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-03-03T08:41:17.788 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-03-03T08:41:17.866 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-03T08:41:18.047 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-03-03T08:41:18.049 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-03T08:41:18.249 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 36/134 2023-03-03T08:41:18.268 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-03-03T08:41:18.411 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 37/134 2023-03-03T08:41:18.488 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-03-03T08:41:18.554 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-03-03T08:41:18.709 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-03-03T08:41:18.714 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-03-03T08:41:18.888 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-03-03T08:41:18.959 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-03-03T08:41:18.970 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-03T08:41:19.162 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 26/134 2023-03-03T08:41:19.167 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-03-03T08:41:19.284 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 27/134 2023-03-03T08:41:19.391 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-03-03T08:41:19.530 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-03-03T08:41:19.612 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-03-03T08:41:19.833 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-03-03T08:41:20.016 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-03-03T08:41:20.193 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-03-03T08:41:20.231 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 28/134 2023-03-03T08:41:20.414 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-03-03T08:41:20.590 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-03-03T08:41:20.724 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-03-03T08:41:20.960 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-03-03T08:41:21.004 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-03-03T08:41:21.150 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-03-03T08:41:21.305 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-03-03T08:41:21.505 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-03-03T08:41:21.560 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-03-03T08:41:21.771 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-03T08:41:21.806 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-03-03T08:41:22.007 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-03-03T08:41:22.250 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-03-03T08:41:22.467 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-03-03T08:41:22.796 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-03-03T08:41:23.012 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-03-03T08:41:23.180 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-03-03T08:41:23.390 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-03-03T08:41:23.560 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-03-03T08:41:23.690 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-03T08:41:23.757 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-03-03T08:41:24.001 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-03-03T08:41:24.060 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-03-03T08:41:24.195 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-03T08:41:24.307 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-03-03T08:41:24.408 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-03T08:41:24.534 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-03-03T08:41:24.581 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 36/134 2023-03-03T08:41:24.759 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 37/134 2023-03-03T08:41:24.772 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-03-03T08:41:24.931 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-03-03T08:41:24.936 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-03T08:41:25.115 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-03T08:41:25.149 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-03-03T08:41:25.332 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-03-03T08:41:25.377 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-03T08:41:25.468 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-03-03T08:41:25.593 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-03T08:41:25.682 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-03-03T08:41:25.741 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-03T08:41:25.880 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-03T08:41:25.906 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-03-03T08:41:26.103 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-03T08:41:26.285 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-03-03T08:41:26.288 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-03T08:41:26.454 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-03T08:41:26.592 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-03-03T08:41:26.598 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-03T08:41:26.733 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-03T08:41:26.832 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-03-03T08:41:26.961 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-03T08:41:27.066 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-03-03T08:41:27.181 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-03T08:41:27.311 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-03-03T08:41:27.358 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-03T08:41:27.601 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-03T08:41:27.640 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-03-03T08:41:27.827 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-03T08:41:27.895 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-03-03T08:41:27.975 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-03T08:41:28.072 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-03-03T08:41:28.149 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-03T08:41:28.312 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-03-03T08:41:28.478 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-03-03T08:41:28.692 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-03-03T08:41:29.014 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-03-03T08:41:29.269 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-03T08:41:29.312 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-03-03T08:41:29.532 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-03T08:41:29.562 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-03-03T08:41:29.744 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-03-03T08:41:29.778 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-03T08:41:29.936 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-03-03T08:41:30.054 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-03T08:41:30.097 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-03-03T08:41:30.303 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-03-03T08:41:30.309 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-03T08:41:30.375 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-03-03T08:41:30.517 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-03T08:41:30.597 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-03-03T08:41:30.816 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-03-03T08:41:30.961 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-03T08:41:31.079 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-03T08:41:31.263 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-03-03T08:41:31.292 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-03T08:41:31.414 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-03-03T08:41:31.453 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 92/134 2023-03-03T08:41:31.524 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 92/134 2023-03-03T08:41:31.580 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-03T08:41:31.711 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 93/134 2023-03-03T08:41:31.752 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-03T08:41:32.053 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-03T08:41:32.277 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-03T08:41:32.416 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-03T08:41:32.574 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-03T08:41:32.784 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-03T08:41:33.109 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-03T08:41:33.292 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-03T08:41:33.473 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-03T08:41:33.601 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-03T08:41:33.797 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-03T08:41:34.016 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-03T08:41:34.226 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-03T08:41:34.475 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-03T08:41:34.703 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-03T08:41:34.862 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-03T08:41:35.101 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-03T08:41:35.210 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T08:41:35.427 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T08:41:36.029 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T08:41:36.240 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-03T08:41:36.492 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-03T08:41:36.762 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-03T08:41:36.776 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 95/134 2023-03-03T08:41:36.863 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 95/134 2023-03-03T08:41:36.968 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 96/134 2023-03-03T08:41:37.115 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-03T08:41:37.342 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-03T08:41:37.537 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-03T08:41:38.383 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-03T08:41:38.585 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 92/134 2023-03-03T08:41:38.657 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 92/134 2023-03-03T08:41:38.819 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 93/134 2023-03-03T08:41:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T08:41:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T08:41:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 94/134 2023-03-03T08:41:43.904 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 95/134 2023-03-03T08:41:43.992 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 95/134 2023-03-03T08:41:44.142 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 96/134 2023-03-03T08:41:55.659 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 96/134 2023-03-03T08:41:55.660 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /sys 2023-03-03T08:41:55.660 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /proc 2023-03-03T08:41:55.660 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /dev 2023-03-03T08:41:55.660 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /run 2023-03-03T08:41:55.660 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /mnt 2023-03-03T08:41:55.660 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /var/tmp 2023-03-03T08:41:55.660 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /home 2023-03-03T08:41:55.661 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /tmp 2023-03-03T08:41:55.661 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /dev 2023-03-03T08:41:55.661 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:41:55.990 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 97/134 2023-03-03T08:41:57.587 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 97/134 2023-03-03T08:41:57.772 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 98/134 2023-03-03T08:41:57.991 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 98/134 2023-03-03T08:41:58.165 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 99/134 2023-03-03T08:41:58.620 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 99/134 2023-03-03T08:41:58.823 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 100/134 2023-03-03T08:41:58.841 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 100/134 2023-03-03T08:42:00.343 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 101/134 2023-03-03T08:42:00.566 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 101/134 2023-03-03T08:42:00.679 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 102/134 2023-03-03T08:42:00.870 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 103/134 2023-03-03T08:42:01.031 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 104/134 2023-03-03T08:42:01.249 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 104/134 2023-03-03T08:42:01.513 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 105/134 2023-03-03T08:42:02.481 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 105/134 2023-03-03T08:42:02.656 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 106/134 2023-03-03T08:42:02.745 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 106/134 2023-03-03T08:42:03.124 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 107/134 2023-03-03T08:42:03.304 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 108/134 2023-03-03T08:42:03.340 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 108/134 2023-03-03T08:42:03.506 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 109/134 2023-03-03T08:42:03.518 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 109/134 2023-03-03T08:42:03.550 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 96/134 2023-03-03T08:42:03.551 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2023-03-03T08:42:03.551 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2023-03-03T08:42:03.551 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /dev 2023-03-03T08:42:03.551 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /run 2023-03-03T08:42:03.551 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2023-03-03T08:42:03.551 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2023-03-03T08:42:03.552 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2023-03-03T08:42:03.552 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2023-03-03T08:42:03.552 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /dev 2023-03-03T08:42:03.552 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:03.906 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 97/134 2023-03-03T08:42:04.778 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 110/134 2023-03-03T08:42:05.011 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 111/134 2023-03-03T08:42:05.045 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 111/134 2023-03-03T08:42:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 112/134 2023-03-03T08:42:05.241 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2739.g472aa 112/134 2023-03-03T08:42:05.381 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 113/134 2023-03-03T08:42:05.586 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 97/134 2023-03-03T08:42:05.730 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 113/134 2023-03-03T08:42:05.775 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 98/134 2023-03-03T08:42:05.906 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 114/134 2023-03-03T08:42:06.005 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 98/134 2023-03-03T08:42:06.297 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 99/134 2023-03-03T08:42:06.777 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 99/134 2023-03-03T08:42:06.980 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 100/134 2023-03-03T08:42:06.998 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 100/134 2023-03-03T08:42:08.688 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 101/134 2023-03-03T08:42:08.911 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 101/134 2023-03-03T08:42:09.045 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 102/134 2023-03-03T08:42:09.231 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 103/134 2023-03-03T08:42:09.421 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 104/134 2023-03-03T08:42:09.644 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 104/134 2023-03-03T08:42:09.891 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 105/134 2023-03-03T08:42:10.355 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 114/134 2023-03-03T08:42:10.915 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 105/134 2023-03-03T08:42:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 106/134 2023-03-03T08:42:11.175 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 106/134 2023-03-03T08:42:11.263 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 115/134 2023-03-03T08:42:11.553 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 107/134 2023-03-03T08:42:11.578 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-03-03T08:42:11.741 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 108/134 2023-03-03T08:42:11.779 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 108/134 2023-03-03T08:42:11.808 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 117/134 2023-03-03T08:42:11.938 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 109/134 2023-03-03T08:42:11.950 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 109/134 2023-03-03T08:42:12.052 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-03-03T08:42:12.187 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-03-03T08:42:12.380 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-03-03T08:42:12.617 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-03-03T08:42:12.775 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 121/134 2023-03-03T08:42:12.964 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-03T08:42:13.299 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 110/134 2023-03-03T08:42:13.552 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 111/134 2023-03-03T08:42:13.587 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 111/134 2023-03-03T08:42:13.763 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 112/134 2023-03-03T08:42:13.807 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2739.g472aa 112/134 2023-03-03T08:42:13.988 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 113/134 2023-03-03T08:42:14.356 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 113/134 2023-03-03T08:42:14.504 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 114/134 2023-03-03T08:42:15.027 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T08:42:15.271 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T08:42:15.290 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T08:42:15.446 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 124/134 2023-03-03T08:42:15.728 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 125/134 2023-03-03T08:42:16.014 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 126/134 2023-03-03T08:42:16.358 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-03T08:42:16.562 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-03-03T08:42:16.645 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-03-03T08:42:16.829 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-03-03T08:42:16.841 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-03-03T08:42:16.970 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-03-03T08:42:16.978 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-03-03T08:42:17.114 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-03-03T08:42:17.114 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T08:42:17.323 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T08:42:17.324 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-03T08:42:17.568 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-03T08:42:17.744 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T08:42:18.011 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T08:42:18.012 INFO:teuthology.orchestra.run.smithi143.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-03T08:42:18.012 INFO:teuthology.orchestra.run.smithi143.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-03T08:42:18.012 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:18.227 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 1/134 2023-03-03T08:42:18.227 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2/134 2023-03-03T08:42:18.227 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 3/134 2023-03-03T08:42:18.228 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 4/134 2023-03-03T08:42:18.228 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 5/134 2023-03-03T08:42:18.228 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T08:42:18.228 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T08:42:18.228 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 8/134 2023-03-03T08:42:18.228 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 9/134 2023-03-03T08:42:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 10/134 2023-03-03T08:42:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 11/134 2023-03-03T08:42:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 12/134 2023-03-03T08:42:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 13/134 2023-03-03T08:42:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 14/134 2023-03-03T08:42:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 15/134 2023-03-03T08:42:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 16/134 2023-03-03T08:42:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 17/134 2023-03-03T08:42:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/134 2023-03-03T08:42:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 19/134 2023-03-03T08:42:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 20/134 2023-03-03T08:42:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 21/134 2023-03-03T08:42:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 22/134 2023-03-03T08:42:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 23/134 2023-03-03T08:42:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 24/134 2023-03-03T08:42:18.238 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 25/134 2023-03-03T08:42:18.238 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 26/134 2023-03-03T08:42:18.239 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 27/134 2023-03-03T08:42:18.239 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 28/134 2023-03-03T08:42:18.239 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 29/134 2023-03-03T08:42:18.239 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 30/134 2023-03-03T08:42:18.239 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 31/134 2023-03-03T08:42:18.240 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 32/134 2023-03-03T08:42:18.240 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 33/134 2023-03-03T08:42:18.240 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 34/134 2023-03-03T08:42:18.240 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 35/134 2023-03-03T08:42:18.240 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 36/134 2023-03-03T08:42:18.240 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 37/134 2023-03-03T08:42:18.240 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 38/134 2023-03-03T08:42:18.241 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-03T08:42:18.241 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-03T08:42:18.241 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-03T08:42:18.241 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-03T08:42:18.241 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-03T08:42:18.241 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-03T08:42:18.241 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-03T08:42:18.242 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-03T08:42:18.242 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-03T08:42:18.242 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-03T08:42:18.242 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-03T08:42:18.242 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-03T08:42:18.242 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-03T08:42:18.243 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-03T08:42:18.243 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-03T08:42:18.243 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-03T08:42:18.243 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-03T08:42:18.243 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-03T08:42:18.243 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-03T08:42:18.243 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-03T08:42:18.243 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-03T08:42:18.244 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-03T08:42:18.244 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-03T08:42:18.244 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-03T08:42:18.244 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-03T08:42:18.244 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-03T08:42:18.244 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-03T08:42:18.244 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-03T08:42:18.244 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-03T08:42:18.245 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-03T08:42:18.245 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-03T08:42:18.245 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-03T08:42:18.245 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-03T08:42:18.245 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-03-03T08:42:18.245 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-03-03T08:42:18.246 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-03-03T08:42:18.246 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-03-03T08:42:18.246 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-03-03T08:42:18.246 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-03-03T08:42:18.246 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-03-03T08:42:18.247 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-03-03T08:42:18.247 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-03-03T08:42:18.247 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-03-03T08:42:18.247 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-03-03T08:42:18.247 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-03-03T08:42:18.247 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-03-03T08:42:18.248 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-03-03T08:42:18.248 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-03-03T08:42:18.248 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-03-03T08:42:18.248 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-03-03T08:42:18.248 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-03-03T08:42:18.248 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-03-03T08:42:18.248 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-03-03T08:42:18.249 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-03-03T08:42:18.249 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-03-03T08:42:18.249 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-03-03T08:42:18.249 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-03-03T08:42:18.249 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-03-03T08:42:18.250 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-03-03T08:42:18.250 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-03-03T08:42:18.250 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-03-03T08:42:18.250 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-03-03T08:42:18.250 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-03-03T08:42:18.250 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-03-03T08:42:18.250 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-03-03T08:42:18.251 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-03-03T08:42:18.251 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-03-03T08:42:18.251 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-03-03T08:42:18.251 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-03-03T08:42:18.251 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-03-03T08:42:18.251 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-03-03T08:42:18.251 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-03-03T08:42:18.252 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-03-03T08:42:18.252 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-03-03T08:42:18.252 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-03-03T08:42:18.252 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-03-03T08:42:18.252 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-03-03T08:42:18.252 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-03-03T08:42:18.253 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-03-03T08:42:18.253 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-03-03T08:42:18.253 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-03-03T08:42:18.253 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-03-03T08:42:18.253 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-03-03T08:42:18.253 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-03-03T08:42:18.254 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-03-03T08:42:18.254 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-03-03T08:42:18.254 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-03-03T08:42:18.254 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-03-03T08:42:18.254 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-03-03T08:42:18.254 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-03-03T08:42:18.255 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 129/134 2023-03-03T08:42:18.255 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-03T08:42:18.255 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 131/134 2023-03-03T08:42:18.255 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T08:42:18.255 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-03T08:42:19.213 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 114/134 2023-03-03T08:42:19.465 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 115/134 2023-03-03T08:42:19.620 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-03-03T08:42:19.827 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-03-03T08:42:20.779 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-03-03T08:42:21.045 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T08:42:21.045 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:46 ago on Fri 03 Mar 2023 08:40:35 AM UTC. 2023-03-03T08:42:21.130 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-03-03T08:42:21.290 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 119/134 2023-03-03T08:42:21.541 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-03T08:42:21.691 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 121/134 2023-03-03T08:42:21.871 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-03T08:42:23.557 INFO:teuthology.orchestra.run.smithi143.stdout:Installed products updated. 2023-03-03T08:42:23.784 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:23.785 INFO:teuthology.orchestra.run.smithi143.stdout:Upgraded: 2023-03-03T08:42:23.785 INFO:teuthology.orchestra.run.smithi143.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-03T08:42:23.785 INFO:teuthology.orchestra.run.smithi143.stdout: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.785 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.785 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2023-03-03T08:42:23.785 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-03T08:42:23.786 INFO:teuthology.orchestra.run.smithi143.stdout: automake-1.16.1-7.el8.noarch 2023-03-03T08:42:23.786 INFO:teuthology.orchestra.run.smithi143.stdout: bison-3.0.4-10.el8.x86_64 2023-03-03T08:42:23.786 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.786 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.786 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.786 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.786 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:23.786 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.787 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.787 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.787 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:23.787 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:23.787 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:23.787 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:23.787 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:23.788 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.788 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.788 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:23.788 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.788 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.788 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.788 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:23.788 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:23.789 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.789 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:23.789 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-03T08:42:23.789 INFO:teuthology.orchestra.run.smithi143.stdout: flex-2.6.1-9.el8.x86_64 2023-03-03T08:42:23.789 INFO:teuthology.orchestra.run.smithi143.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-03T08:42:23.790 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-03T08:42:23.790 INFO:teuthology.orchestra.run.smithi143.stdout: jq-1.6-3.el8.x86_64 2023-03-03T08:42:23.790 INFO:teuthology.orchestra.run.smithi143.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-03T08:42:23.790 INFO:teuthology.orchestra.run.smithi143.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-03T08:42:23.790 INFO:teuthology.orchestra.run.smithi143.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-03T08:42:23.791 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.791 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.791 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.791 INFO:teuthology.orchestra.run.smithi143.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-03T08:42:23.791 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-03T08:42:23.791 INFO:teuthology.orchestra.run.smithi143.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-03T08:42:23.791 INFO:teuthology.orchestra.run.smithi143.stdout: libpq-13.5-1.el8.x86_64 2023-03-03T08:42:23.792 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-03T08:42:23.792 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-03T08:42:23.792 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.792 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.792 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-03T08:42:23.792 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.792 INFO:teuthology.orchestra.run.smithi143.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-03T08:42:23.792 INFO:teuthology.orchestra.run.smithi143.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-03T08:42:23.793 INFO:teuthology.orchestra.run.smithi143.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-03T08:42:23.793 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-03T08:42:23.793 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-03T08:42:23.793 INFO:teuthology.orchestra.run.smithi143.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-03T08:42:23.793 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-03T08:42:23.793 INFO:teuthology.orchestra.run.smithi143.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-03T08:42:23.793 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-03T08:42:23.794 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-03T08:42:23.794 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-03T08:42:23.794 INFO:teuthology.orchestra.run.smithi143.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-03T08:42:23.794 INFO:teuthology.orchestra.run.smithi143.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-03T08:42:23.794 INFO:teuthology.orchestra.run.smithi143.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-03T08:42:23.794 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T08:42:23.795 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T08:42:23.795 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T08:42:23.795 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-03T08:42:23.795 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-03T08:42:23.795 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-03T08:42:23.795 INFO:teuthology.orchestra.run.smithi143.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-03T08:42:23.795 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-03T08:42:23.795 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.796 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.796 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.796 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-03T08:42:23.796 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-03T08:42:23.796 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-03T08:42:23.796 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-03T08:42:23.796 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-03T08:42:23.796 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-03T08:42:23.797 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-03T08:42:23.797 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-03T08:42:23.797 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-03T08:42:23.797 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-03T08:42:23.797 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-03T08:42:23.798 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-03T08:42:23.798 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-03T08:42:23.798 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-03T08:42:23.798 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-03T08:42:23.798 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-03T08:42:23.798 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-03T08:42:23.798 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-03T08:42:23.799 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-03T08:42:23.799 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-03T08:42:23.799 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-03T08:42:23.799 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-03T08:42:23.799 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend-2.6-1.el8.noarch 2023-03-03T08:42:23.799 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-03T08:42:23.799 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-03T08:42:23.800 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-03T08:42:23.800 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-03T08:42:23.800 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-03T08:42:23.800 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-03T08:42:23.800 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.800 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.800 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-03T08:42:23.801 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-03T08:42:23.801 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.801 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-03T08:42:23.801 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-03T08:42:23.801 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-03T08:42:23.801 INFO:teuthology.orchestra.run.smithi143.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-03T08:42:23.801 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-03T08:42:23.802 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-03T08:42:23.802 INFO:teuthology.orchestra.run.smithi143.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-03T08:42:23.802 INFO:teuthology.orchestra.run.smithi143.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-03T08:42:23.802 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-03T08:42:23.802 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-03T08:42:23.802 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-03T08:42:23.802 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-03T08:42:23.802 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-03T08:42:23.803 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.803 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.803 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:23.803 INFO:teuthology.orchestra.run.smithi143.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-03T08:42:23.803 INFO:teuthology.orchestra.run.smithi143.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-03T08:42:23.803 INFO:teuthology.orchestra.run.smithi143.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-03T08:42:23.803 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-03T08:42:23.803 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-03T08:42:23.804 INFO:teuthology.orchestra.run.smithi143.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-03T08:42:23.804 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:23.804 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2023-03-03T08:42:23.976 DEBUG:teuthology.parallel:result is None 2023-03-03T08:42:24.127 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T08:42:24.394 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T08:42:24.413 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-03T08:42:24.612 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 124/134 2023-03-03T08:42:24.844 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 125/134 2023-03-03T08:42:25.149 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 126/134 2023-03-03T08:42:25.437 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-03T08:42:25.569 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-03T08:42:25.578 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-03T08:42:25.746 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-03-03T08:42:25.927 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-03-03T08:42:26.015 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-03-03T08:42:26.219 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-03-03T08:42:26.232 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-03-03T08:42:26.233 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T08:42:26.422 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T08:42:26.423 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-03T08:42:26.691 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-03T08:42:26.877 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T08:42:27.223 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T08:42:27.223 INFO:teuthology.orchestra.run.smithi007.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-03T08:42:27.223 INFO:teuthology.orchestra.run.smithi007.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-03T08:42:27.223 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:27.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 1/134 2023-03-03T08:42:27.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2/134 2023-03-03T08:42:27.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 3/134 2023-03-03T08:42:27.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 4/134 2023-03-03T08:42:27.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2739.g472aa 5/134 2023-03-03T08:42:27.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 6/134 2023-03-03T08:42:27.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 7/134 2023-03-03T08:42:27.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 8/134 2023-03-03T08:42:27.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 9/134 2023-03-03T08:42:27.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 10/134 2023-03-03T08:42:27.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 11/134 2023-03-03T08:42:27.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 12/134 2023-03-03T08:42:27.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 13/134 2023-03-03T08:42:27.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_ 14/134 2023-03-03T08:42:27.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 15/134 2023-03-03T08:42:27.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 16/134 2023-03-03T08:42:27.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_6 17/134 2023-03-03T08:42:27.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86 18/134 2023-03-03T08:42:27.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 19/134 2023-03-03T08:42:27.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el 20/134 2023-03-03T08:42:27.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:18.0.0-2739.g472aaef4.el8. 21/134 2023-03-03T08:42:27.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_6 22/134 2023-03-03T08:42:27.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 23/134 2023-03-03T08:42:27.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 24/134 2023-03-03T08:42:27.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 25/134 2023-03-03T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 26/134 2023-03-03T08:42:27.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 27/134 2023-03-03T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 28/134 2023-03-03T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4. 29/134 2023-03-03T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noa 30/134 2023-03-03T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.n 31/134 2023-03-03T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2739.g472 32/134 2023-03-03T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el 33/134 2023-03-03T08:42:27.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 34/134 2023-03-03T08:42:27.471 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.e 35/134 2023-03-03T08:42:27.471 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 36/134 2023-03-03T08:42:27.471 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 37/134 2023-03-03T08:42:27.471 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 38/134 2023-03-03T08:42:27.471 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-03T08:42:27.471 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-03T08:42:27.471 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-03T08:42:27.471 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-03T08:42:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-03T08:42:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-03T08:42:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-03T08:42:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-03T08:42:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-03T08:42:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-03T08:42:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-03T08:42:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-03T08:42:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-03T08:42:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-03T08:42:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-03T08:42:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-03T08:42:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-03T08:42:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-03T08:42:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-03T08:42:27.473 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-03T08:42:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-03T08:42:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-03T08:42:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-03T08:42:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-03T08:42:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-03T08:42:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-03T08:42:27.474 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-03T08:42:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-03T08:42:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-03T08:42:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-03T08:42:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-03T08:42:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-03T08:42:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-03T08:42:27.475 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-03-03T08:42:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-03-03T08:42:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-03-03T08:42:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-03-03T08:42:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-03-03T08:42:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-03-03T08:42:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-03-03T08:42:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-03-03T08:42:27.476 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-03-03T08:42:27.477 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-03-03T08:42:27.477 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-03-03T08:42:27.477 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-03-03T08:42:27.477 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-03-03T08:42:27.477 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-03-03T08:42:27.477 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-03-03T08:42:27.478 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-03-03T08:42:27.478 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-03-03T08:42:27.478 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-03-03T08:42:27.478 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-03-03T08:42:27.478 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-03-03T08:42:27.478 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-03-03T08:42:27.478 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-03-03T08:42:27.478 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-03-03T08:42:27.479 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-03-03T08:42:27.479 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-03-03T08:42:27.479 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-03-03T08:42:27.479 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-03-03T08:42:27.479 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-03-03T08:42:27.479 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-03-03T08:42:27.479 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-03-03T08:42:27.480 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-03-03T08:42:27.480 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-03-03T08:42:27.480 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-03-03T08:42:27.480 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-03-03T08:42:27.480 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-03-03T08:42:27.480 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-03-03T08:42:27.480 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-03-03T08:42:27.480 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-03-03T08:42:27.481 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-03-03T08:42:27.481 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-03-03T08:42:27.481 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-03-03T08:42:27.481 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-03-03T08:42:27.481 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-03-03T08:42:27.481 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-03-03T08:42:27.481 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-03-03T08:42:27.481 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-03-03T08:42:27.482 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-03-03T08:42:27.482 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-03-03T08:42:27.482 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-03T08:42:27.482 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-03-03T08:42:27.482 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-03T08:42:27.482 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-03-03T08:42:27.482 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-03-03T08:42:27.482 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-03-03T08:42:27.483 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-03-03T08:42:27.483 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-03-03T08:42:27.483 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-03T08:42:27.483 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 129/134 2023-03-03T08:42:27.483 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-03T08:42:27.483 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 131/134 2023-03-03T08:42:27.483 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-03T08:42:27.483 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-03T08:42:30.230 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-03T08:42:30.230 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:54 ago on Fri 03 Mar 2023 08:40:36 AM UTC. 2023-03-03T08:42:32.764 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-03-03T08:42:33.003 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:33.003 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2023-03-03T08:42:33.003 INFO:teuthology.orchestra.run.smithi007.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-03T08:42:33.004 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.004 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.004 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-03-03T08:42:33.004 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-03T08:42:33.004 INFO:teuthology.orchestra.run.smithi007.stdout: automake-1.16.1-7.el8.noarch 2023-03-03T08:42:33.004 INFO:teuthology.orchestra.run.smithi007.stdout: bison-3.0.4-10.el8.x86_64 2023-03-03T08:42:33.004 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.005 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.005 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.005 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.005 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:33.005 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.005 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.005 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.006 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:33.006 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:33.006 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:33.006 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:33.006 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:33.006 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.006 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.006 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:33.007 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.007 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.007 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.007 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:33.007 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:33.007 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.007 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-top-2:18.0.0-2739.g472aaef4.el8.noarch 2023-03-03T08:42:33.008 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-03T08:42:33.008 INFO:teuthology.orchestra.run.smithi007.stdout: flex-2.6.1-9.el8.x86_64 2023-03-03T08:42:33.008 INFO:teuthology.orchestra.run.smithi007.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-03T08:42:33.008 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-03T08:42:33.008 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-3.el8.x86_64 2023-03-03T08:42:33.008 INFO:teuthology.orchestra.run.smithi007.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-03T08:42:33.008 INFO:teuthology.orchestra.run.smithi007.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-03T08:42:33.008 INFO:teuthology.orchestra.run.smithi007.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-03T08:42:33.009 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.009 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.009 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.009 INFO:teuthology.orchestra.run.smithi007.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-03T08:42:33.009 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-03T08:42:33.009 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-03T08:42:33.009 INFO:teuthology.orchestra.run.smithi007.stdout: libpq-13.5-1.el8.x86_64 2023-03-03T08:42:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-03T08:42:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-03T08:42:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-03T08:42:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-03T08:42:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-03T08:42:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-03T08:42:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-03T08:42:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-03T08:42:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-03T08:42:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-03T08:42:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-03T08:42:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-03T08:42:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-03T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-03T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-03T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-03T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-03T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T08:42:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-03T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-03T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-03T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-03T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-03T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-03T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.013 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-03T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-03T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-03T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-03T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-03T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-03T08:42:33.014 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-03T08:42:33.015 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-03T08:42:33.015 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-03T08:42:33.015 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-03T08:42:33.015 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-03T08:42:33.015 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-03T08:42:33.015 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-03T08:42:33.015 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-03T08:42:33.015 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-03T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-03T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-03T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-03T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-03T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-03T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-03T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-03T08:42:33.016 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-2.6-1.el8.noarch 2023-03-03T08:42:33.017 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-03T08:42:33.017 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-03T08:42:33.017 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-03T08:42:33.017 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-03T08:42:33.017 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-03T08:42:33.017 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-03T08:42:33.017 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.018 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.018 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-03T08:42:33.018 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-03T08:42:33.018 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.018 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-03T08:42:33.018 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-03T08:42:33.018 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-03T08:42:33.018 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-03T08:42:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-03T08:42:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-03T08:42:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-03T08:42:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-03T08:42:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-03T08:42:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-03T08:42:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-03T08:42:33.019 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-03T08:42:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-03T08:42:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:18.0.0-2739.g472aaef4.el8.x86_64 2023-03-03T08:42:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-03T08:42:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-03T08:42:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-03T08:42:33.020 INFO:teuthology.orchestra.run.smithi007.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-03T08:42:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-03T08:42:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-03T08:42:33.021 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:33.022 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-03-03T08:42:33.460 DEBUG:teuthology.parallel:result is None 2023-03-03T08:42:33.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T08:42:33.605 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-03T08:42:33.629 INFO:teuthology.orchestra.run.smithi007.stdout:18.0.0-2739.g472aaef4.el8 2023-03-03T08:42:33.630 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2739.g472aaef4.el8 2023-03-03T08:42:33.631 INFO:teuthology.task.install:The correct ceph version 18.0.0-2739.g472aaef4 is installed. 2023-03-03T08:42:33.631 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T08:42:33.769 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-03T08:42:33.798 INFO:teuthology.orchestra.run.smithi143.stdout:18.0.0-2739.g472aaef4.el8 2023-03-03T08:42:33.799 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2739.g472aaef4.el8 2023-03-03T08:42:33.799 INFO:teuthology.task.install:The correct ceph version 18.0.0-2739.g472aaef4 is installed. 2023-03-03T08:42:33.799 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-03-03T08:42:33.800 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:42:33.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-03T08:42:33.842 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:42:33.843 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-03T08:42:33.887 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-03-03T08:42:33.888 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:42:33.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2023-03-03T08:42:33.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-03T08:42:33.983 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:42:33.983 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2023-03-03T08:42:34.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-03T08:42:34.084 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-03-03T08:42:34.084 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:42:34.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-03T08:42:34.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-03T08:42:34.178 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:42:34.178 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-03T08:42:34.207 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-03T08:42:34.276 INFO:teuthology.run_tasks:Running task ceph... 2023-03-03T08:42:34.353 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-03-03T08:42:34.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2023-03-03T08:42:34.356 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /var/log/ceph 2023-03-03T08:42:34.383 INFO:tasks.ceph:Disabling ceph logrotate... 2023-03-03T08:42:34.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-03T08:42:34.423 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-03T08:42:34.450 INFO:tasks.ceph:Creating extra log directories... 2023-03-03T08:42:34.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-03T08:42:34.490 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-03T08:42:34.526 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-03-03T08:42:34.527 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-03-03T08:42:34.527 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-03-03T08:42:34.527 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:42:34.527 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-03T08:42:34.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:34.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:34.740 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:42:34.741 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-03T08:42:34.757 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:34.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-03T08:42:34.944 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-03-03T08:42:34.944 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-03-03T08:42:34.944 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi/7192252', 'branch': 'wip-vshankar-testing-20230303.023823', 'description': 'fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7192252', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-03-03_04:39:14-fs-wip-vshankar-testing-20230303.023823-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20230303.023823'}, 'ceph': {'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, 'kclient': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}, 'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20230303.023823', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20230303.023823', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_472aaef4ec3b3a2c29f73ca67854785a910f1f9b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=', 'smithi143.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '75534f74ac3dd06f86697ff862e8be2945b3c8e6'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'}}, {'ceph': None}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6', 'timestamp': '2023-03-03_04:39:14', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.31660'} 2023-03-03T08:42:34.945 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-03T08:42:34.948 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-03T08:42:34.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2023-03-03T08:42:35.005 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/run/ceph 2023-03-03T08:42:35.057 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:42:35.057 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2023-03-03T08:42:35.087 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-03T08:42:35.087 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2023-03-03T08:42:35.144 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-03T08:42:35.144 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T08:42:35.144 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 360771 Links: 1 2023-03-03T08:42:35.145 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T08:42:35.145 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T08:42:35.145 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-03-03 08:42:27.138103232 +0000 2023-03-03T08:42:35.145 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-03-03 08:39:47.911863146 +0000 2023-03-03T08:42:35.145 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-03-03 08:39:47.911863146 +0000 2023-03-03T08:42:35.145 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-03-03T08:42:35.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-03T08:42:35.209 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-03-03T08:42:35.209 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-03-03T08:42:35.209 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000107115 s, 4.8 MB/s 2023-03-03T08:42:35.210 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-03T08:42:35.268 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2023-03-03T08:42:35.325 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-03T08:42:35.325 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T08:42:35.325 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 360812 Links: 1 2023-03-03T08:42:35.325 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T08:42:35.326 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T08:42:35.326 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-03-03 08:42:27.138103232 +0000 2023-03-03T08:42:35.326 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-03-03 08:39:48.450855207 +0000 2023-03-03T08:42:35.326 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-03-03 08:39:48.450855207 +0000 2023-03-03T08:42:35.326 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-03-03T08:42:35.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-03T08:42:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-03-03T08:42:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-03-03T08:42:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000107624 s, 4.8 MB/s 2023-03-03T08:42:35.392 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-03T08:42:35.451 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2023-03-03T08:42:35.508 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-03T08:42:35.508 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T08:42:35.508 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 363692 Links: 1 2023-03-03T08:42:35.509 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T08:42:35.509 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T08:42:35.509 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-03-03 08:42:27.138103232 +0000 2023-03-03T08:42:35.509 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-03-03 08:39:48.981847388 +0000 2023-03-03T08:42:35.509 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-03-03 08:39:48.981847388 +0000 2023-03-03T08:42:35.509 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-03-03T08:42:35.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-03T08:42:35.573 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-03-03T08:42:35.574 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-03-03T08:42:35.574 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000110594 s, 4.6 MB/s 2023-03-03T08:42:35.575 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-03T08:42:35.634 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2023-03-03T08:42:35.691 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-03T08:42:35.692 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T08:42:35.692 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 362848 Links: 1 2023-03-03T08:42:35.692 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T08:42:35.692 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T08:42:35.692 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-03-03 08:42:27.138103232 +0000 2023-03-03T08:42:35.692 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-03-03 08:39:49.517839493 +0000 2023-03-03T08:42:35.693 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-03-03 08:39:49.517839493 +0000 2023-03-03T08:42:35.693 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-03-03T08:42:35.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-03T08:42:35.758 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-03-03T08:42:35.758 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-03-03T08:42:35.758 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.9628e-05 s, 5.1 MB/s 2023-03-03T08:42:35.759 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-03T08:42:35.817 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T08:42:35.817 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:42:35.817 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2023-03-03T08:42:35.836 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-03T08:42:35.836 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2023-03-03T08:42:35.895 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-03T08:42:35.895 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T08:42:35.896 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 360171 Links: 1 2023-03-03T08:42:35.896 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T08:42:35.896 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T08:42:35.896 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2023-03-03 08:42:17.929896259 +0000 2023-03-03T08:42:35.896 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2023-03-03 08:39:33.863625000 +0000 2023-03-03T08:42:35.896 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2023-03-03 08:39:33.863625000 +0000 2023-03-03T08:42:35.896 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2023-03-03T08:42:35.897 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-03T08:42:35.963 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2023-03-03T08:42:35.964 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2023-03-03T08:42:35.964 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000289431 s, 1.8 MB/s 2023-03-03T08:42:35.965 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-03T08:42:36.024 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2023-03-03T08:42:36.083 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-03T08:42:36.083 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T08:42:36.083 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 360257 Links: 1 2023-03-03T08:42:36.083 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T08:42:36.083 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T08:42:36.083 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2023-03-03 08:42:17.929896259 +0000 2023-03-03T08:42:36.084 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2023-03-03 08:39:34.394610807 +0000 2023-03-03T08:42:36.084 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2023-03-03 08:39:34.394610807 +0000 2023-03-03T08:42:36.084 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2023-03-03T08:42:36.084 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-03T08:42:36.149 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2023-03-03T08:42:36.149 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2023-03-03T08:42:36.149 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000259665 s, 2.0 MB/s 2023-03-03T08:42:36.150 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-03T08:42:36.209 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2023-03-03T08:42:36.265 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-03T08:42:36.266 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T08:42:36.266 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 356838 Links: 1 2023-03-03T08:42:36.266 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T08:42:36.266 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T08:42:36.266 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2023-03-03 08:42:17.929896259 +0000 2023-03-03T08:42:36.266 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2023-03-03 08:39:34.955595812 +0000 2023-03-03T08:42:36.267 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2023-03-03 08:39:34.955595812 +0000 2023-03-03T08:42:36.267 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2023-03-03T08:42:36.267 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-03T08:42:36.331 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2023-03-03T08:42:36.331 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2023-03-03T08:42:36.332 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000233993 s, 2.2 MB/s 2023-03-03T08:42:36.333 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-03T08:42:36.390 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2023-03-03T08:42:36.447 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-03T08:42:36.448 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-03T08:42:36.448 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 365726 Links: 1 2023-03-03T08:42:36.448 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T08:42:36.448 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2023-03-03T08:42:36.448 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2023-03-03 08:42:17.929896259 +0000 2023-03-03T08:42:36.449 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2023-03-03 08:39:35.463582234 +0000 2023-03-03T08:42:36.449 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2023-03-03 08:39:35.463582234 +0000 2023-03-03T08:42:36.449 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2023-03-03T08:42:36.449 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-03T08:42:36.512 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2023-03-03T08:42:36.513 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2023-03-03T08:42:36.513 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000173897 s, 2.9 MB/s 2023-03-03T08:42:36.514 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-03T08:42:36.572 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T08:42:36.572 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-03T08:42:36.572 INFO:tasks.ceph:Generating config... 2023-03-03T08:42:36.574 INFO:tasks.ceph:[client] client mount timeout = 600 2023-03-03T08:42:36.574 INFO:tasks.ceph:[client] debug client = 20 2023-03-03T08:42:36.574 INFO:tasks.ceph:[client] debug ms = 1 2023-03-03T08:42:36.575 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-03-03T08:42:36.575 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-03-03T08:42:36.575 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-03-03T08:42:36.575 INFO:tasks.ceph:[mds] debug mds = 20 2023-03-03T08:42:36.575 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-03-03T08:42:36.576 INFO:tasks.ceph:[mds] debug ms = 1 2023-03-03T08:42:36.576 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-03-03T08:42:36.576 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-03-03T08:42:36.576 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-03-03T08:42:36.576 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-03-03T08:42:36.577 INFO:tasks.ceph:[mds] mds debug frag = True 2023-03-03T08:42:36.577 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-03-03T08:42:36.577 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-03-03T08:42:36.577 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-03-03T08:42:36.577 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-03-03T08:42:36.578 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-03-03T08:42:36.578 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-03-03T08:42:36.578 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-03-03T08:42:36.578 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-03-03T08:42:36.578 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-03-03T08:42:36.579 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-03-03T08:42:36.579 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-03-03T08:42:36.579 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-03-03T08:42:36.579 INFO:tasks.ceph:[mgr] debug ms = 1 2023-03-03T08:42:36.579 INFO:tasks.ceph:[mon] debug mon = 20 2023-03-03T08:42:36.580 INFO:tasks.ceph:[mon] debug ms = 1 2023-03-03T08:42:36.580 INFO:tasks.ceph:[mon] debug paxos = 20 2023-03-03T08:42:36.580 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-03-03T08:42:36.580 INFO:tasks.ceph:[osd] bdev async discard = True 2023-03-03T08:42:36.580 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-03-03T08:42:36.581 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-03-03T08:42:36.581 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-03-03T08:42:36.581 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-03-03T08:42:36.581 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-03-03T08:42:36.581 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-03-03T08:42:36.582 INFO:tasks.ceph:[osd] debug ms = 1 2023-03-03T08:42:36.582 INFO:tasks.ceph:[osd] debug osd = 20 2023-03-03T08:42:36.582 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-03-03T08:42:36.582 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-03-03T08:42:36.582 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-03-03T08:42:36.583 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-03-03T08:42:36.583 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-03-03T08:42:36.583 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-03-03T08:42:36.583 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-03-03T08:42:36.583 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-03-03T08:42:36.584 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-03-03T08:42:36.584 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-03-03T08:42:36.584 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-03-03T08:42:36.584 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-03-03T08:42:36.584 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-03-03T08:42:36.584 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-03-03T08:42:36.585 INFO:tasks.ceph:Setting up mon.a... 2023-03-03T08:42:36.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-03-03T08:42:36.648 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2023-03-03T08:42:36.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-03-03T08:42:36.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T08:42:36.728 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.7'), ('mon.b', '172.21.15.143'), ('mon.c', '[v2:172.21.15.143:3301,v1:172.21.15.143:6790]')] 2023-03-03T08:42:36.729 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-03-03T08:42:36.730 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:42:36.730 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-03-03T08:42:36.786 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --add b 172.21.15.143 --addv c '[v2:172.21.15.143:3301,v1:172.21.15.143:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:42:36.877 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:42:36.877 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid 4366045e-92da-4bb9-8565-4b6c37871975 2023-03-03T08:42:36.878 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = pacific 2023-03-03T08:42:36.878 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2023-03-03T08:42:36.878 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 4366045e-92da-4bb9-8565-4b6c37871975 2023-03-03T08:42:36.878 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2023-03-03T08:42:36.877123+0000 2023-03-03T08:42:36.878 INFO:teuthology.orchestra.run.smithi007.stdout:created 2023-03-03T08:42:36.877123+0000 2023-03-03T08:42:36.878 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 16 (pacific) 2023-03-03T08:42:36.879 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2023-03-03T08:42:36.879 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-03-03T08:42:36.879 INFO:teuthology.orchestra.run.smithi007.stdout:1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.b 2023-03-03T08:42:36.879 INFO:teuthology.orchestra.run.smithi007.stdout:2: [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] mon.c 2023-03-03T08:42:36.879 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-03-03T08:42:36.880 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-03-03T08:42:36.897 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4366045e-92da-4bb9-8565-4b6c37871975... 2023-03-03T08:42:36.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-03T08:42:36.939 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-03T08:42:36.983 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2023-03-03T08:42:36.984 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2023-03-03T08:42:36.984 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-03T08:42:36.984 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2023-03-03T08:42:36.984 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.984 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2023-03-03T08:42:36.984 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.984 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2023-03-03T08:42:36.984 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.985 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2023-03-03T08:42:36.985 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2023-03-03T08:42:36.985 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.985 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2023-03-03T08:42:36.985 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2023-03-03T08:42:36.985 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.985 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-03T08:42:36.986 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2023-03-03T08:42:36.986 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.986 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2023-03-03T08:42:36.986 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-03T08:42:36.986 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.986 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2023-03-03T08:42:36.986 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.986 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2023-03-03T08:42:36.987 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2023-03-03T08:42:36.987 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2023-03-03T08:42:36.987 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2023-03-03T08:42:36.987 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2023-03-03T08:42:36.987 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2023-03-03T08:42:36.987 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2023-03-03T08:42:36.987 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-03T08:42:36.987 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-03T08:42:36.988 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2023-03-03T08:42:36.988 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.988 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-03T08:42:36.988 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.988 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2023-03-03T08:42:36.988 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.988 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2023-03-03T08:42:36.989 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.989 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2023-03-03T08:42:36.989 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2023-03-03T08:42:36.989 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2023-03-03T08:42:36.989 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = "172.21.15.7,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2023-03-03T08:42:36.989 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on skipped message = False 2023-03-03T08:42:36.989 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 4366045e-92da-4bb9-8565-4b6c37871975 2023-03-03T08:42:36.989 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.990 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2023-03-03T08:42:36.990 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2023-03-03T08:42:36.990 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.990 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2023-03-03T08:42:36.990 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2023-03-03T08:42:36.990 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.990 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2023-03-03T08:42:36.990 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2023-03-03T08:42:36.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.991 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2023-03-03T08:42:36.991 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2023-03-03T08:42:36.991 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2023-03-03T08:42:36.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.991 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2023-03-03T08:42:36.991 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2023-03-03T08:42:36.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.992 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2023-03-03T08:42:36.992 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.992 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2023-03-03T08:42:36.992 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.992 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2023-03-03T08:42:36.992 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.992 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2023-03-03T08:42:36.992 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2023-03-03T08:42:36.993 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.993 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2023-03-03T08:42:36.993 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2023-03-03T08:42:36.993 INFO:teuthology.orchestra.run.smithi007.stdout: bdev async discard = True 2023-03-03T08:42:36.993 INFO:teuthology.orchestra.run.smithi007.stdout: bdev enable discard = True 2023-03-03T08:42:36.993 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore allocator = bitmap 2023-03-03T08:42:36.993 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2023-03-03T08:42:36.994 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2023-03-03T08:42:36.994 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 1/20 2023-03-03T08:42:36.994 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 1/20 2023-03-03T08:42:36.994 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-03-03T08:42:36.994 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2023-03-03T08:42:36.994 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 4/10 2023-03-03T08:42:36.994 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2023-03-03T08:42:36.994 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2023-03-03T08:42:36.995 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2023-03-03T08:42:36.995 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2023-03-03T08:42:36.995 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay max = 1 2023-03-03T08:42:36.995 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay probability = 0.005 2023-03-03T08:42:36.995 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay type = client mds 2023-03-03T08:42:36.995 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 2500 2023-03-03T08:42:36.995 INFO:teuthology.orchestra.run.smithi007.stdout: osd blocked scrub grace period = 3600 2023-03-03T08:42:36.995 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2023-03-03T08:42:36.996 INFO:teuthology.orchestra.run.smithi007.stdout: osd max markdown count = 1000 2023-03-03T08:42:36.996 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2023-03-03T08:42:36.996 INFO:teuthology.orchestra.run.smithi007.stdout: osd op complaint time = 180 2023-03-03T08:42:36.996 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.996 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2023-03-03T08:42:36.996 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-03-03T08:42:36.996 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2023-03-03T08:42:36.996 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2023-03-03T08:42:36.997 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2023-03-03T08:42:36.997 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2023-03-03T08:42:36.997 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2023-03-03T08:42:36.997 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2023-03-03T08:42:36.997 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.997 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2023-03-03T08:42:36.997 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-03-03T08:42:36.998 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2023-03-03T08:42:36.998 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2023-03-03T08:42:36.998 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2023-03-03T08:42:36.998 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2023-03-03T08:42:36.998 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2023-03-03T08:42:36.998 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2023-03-03T08:42:36.998 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2023-03-03T08:42:36.998 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2023-03-03T08:42:36.999 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2023-03-03T08:42:36.999 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.999 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-03T08:42:36.999 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2023-03-03T08:42:36.999 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2023-03-03T08:42:36.999 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:36.999 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2023-03-03T08:42:36.999 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-03T08:42:37.000 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-03T08:42:37.000 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:37.000 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2023-03-03T08:42:37.000 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2023-03-03T08:42:37.000 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:37.000 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2023-03-03T08:42:37.000 INFO:teuthology.orchestra.run.smithi007.stdout: mon op complaint time = 120 2023-03-03T08:42:37.000 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:42:37.001 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2023-03-03T08:42:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2023-03-03T08:42:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2023-03-03T08:42:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2023-03-03T08:42:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-03T08:42:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-03T08:42:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: client mount timeout = 600 2023-03-03T08:42:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: debug client = 20 2023-03-03T08:42:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-03-03T08:42:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: rados mon op timeout = 900 2023-03-03T08:42:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: rados osd op timeout = 900 2023-03-03T08:42:37.002 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2023-03-03T08:42:37.002 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.b] 2023-03-03T08:42:37.002 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.c] 2023-03-03T08:42:37.002 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.a] 2023-03-03T08:42:37.002 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.c] 2023-03-03T08:42:37.003 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.b] 2023-03-03T08:42:37.003 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.d] 2023-03-03T08:42:37.003 INFO:teuthology.orchestra.run.smithi007.stdout:[mds] 2023-03-03T08:42:37.003 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds = 20 2023-03-03T08:42:37.003 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds balancer = 20 2023-03-03T08:42:37.003 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-03-03T08:42:37.003 INFO:teuthology.orchestra.run.smithi007.stdout: mds bal fragment size max = 10000 2023-03-03T08:42:37.003 INFO:teuthology.orchestra.run.smithi007.stdout: mds bal merge size = 5 2023-03-03T08:42:37.004 INFO:teuthology.orchestra.run.smithi007.stdout: mds bal split bits = 3 2023-03-03T08:42:37.004 INFO:teuthology.orchestra.run.smithi007.stdout: mds bal split size = 100 2023-03-03T08:42:37.004 INFO:teuthology.orchestra.run.smithi007.stdout: mds debug frag = True 2023-03-03T08:42:37.004 INFO:teuthology.orchestra.run.smithi007.stdout: mds debug scatterstat = True 2023-03-03T08:42:37.004 INFO:teuthology.orchestra.run.smithi007.stdout: mds op complaint time = 180 2023-03-03T08:42:37.004 INFO:teuthology.orchestra.run.smithi007.stdout: mds verify scatter = True 2023-03-03T08:42:37.004 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2023-03-03T08:42:37.004 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay max = 1 2023-03-03T08:42:37.005 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay probability = 0.005 2023-03-03T08:42:37.005 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay type = client mds osd 2023-03-03T08:42:37.005 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 2500 2023-03-03T08:42:37.005 INFO:teuthology.orchestra.run.smithi007.stdout: osd op complaint time = 180 2023-03-03T08:42:37.005 INFO:teuthology.orchestra.run.smithi007.stdout: rados mon op timeout = 900 2023-03-03T08:42:37.005 INFO:teuthology.orchestra.run.smithi007.stdout: rados osd op timeout = 900 2023-03-03T08:42:37.006 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2023-03-03T08:42:37.007 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2023-03-03T08:42:37.007 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-03T08:42:37.007 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2023-03-03T08:42:37.007 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.007 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2023-03-03T08:42:37.007 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.007 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2023-03-03T08:42:37.008 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.008 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2023-03-03T08:42:37.008 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2023-03-03T08:42:37.008 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.008 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2023-03-03T08:42:37.008 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2023-03-03T08:42:37.008 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.009 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-03T08:42:37.009 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2023-03-03T08:42:37.009 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.009 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2023-03-03T08:42:37.009 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-03T08:42:37.009 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.009 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2023-03-03T08:42:37.009 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.010 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2023-03-03T08:42:37.010 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2023-03-03T08:42:37.010 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2023-03-03T08:42:37.010 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2023-03-03T08:42:37.010 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2023-03-03T08:42:37.010 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2023-03-03T08:42:37.010 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2023-03-03T08:42:37.010 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-03T08:42:37.011 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-03T08:42:37.011 INFO:teuthology.orchestra.run.smithi143.stdout: mon_allow_pool_size_one = true 2023-03-03T08:42:37.011 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.011 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-03T08:42:37.011 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.011 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2023-03-03T08:42:37.011 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.012 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2023-03-03T08:42:37.012 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.012 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2023-03-03T08:42:37.012 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2023-03-03T08:42:37.012 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2023-03-03T08:42:37.012 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = "172.21.15.7,172.21.15.143,[v2:172.21.15.143:3301,v1:172.21.15.143:6790]" 2023-03-03T08:42:37.012 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on skipped message = False 2023-03-03T08:42:37.012 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 4366045e-92da-4bb9-8565-4b6c37871975 2023-03-03T08:42:37.013 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.013 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2023-03-03T08:42:37.013 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2023-03-03T08:42:37.013 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.013 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2023-03-03T08:42:37.013 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2023-03-03T08:42:37.013 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.013 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2023-03-03T08:42:37.014 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2023-03-03T08:42:37.014 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.014 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2023-03-03T08:42:37.014 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2023-03-03T08:42:37.014 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2023-03-03T08:42:37.014 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.014 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2023-03-03T08:42:37.014 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2023-03-03T08:42:37.015 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.015 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2023-03-03T08:42:37.015 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.015 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2023-03-03T08:42:37.015 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.015 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2023-03-03T08:42:37.015 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.016 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2023-03-03T08:42:37.016 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2023-03-03T08:42:37.016 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.016 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2023-03-03T08:42:37.016 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2023-03-03T08:42:37.016 INFO:teuthology.orchestra.run.smithi143.stdout: bdev async discard = True 2023-03-03T08:42:37.016 INFO:teuthology.orchestra.run.smithi143.stdout: bdev enable discard = True 2023-03-03T08:42:37.016 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore allocator = bitmap 2023-03-03T08:42:37.017 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2023-03-03T08:42:37.017 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2023-03-03T08:42:37.017 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 1/20 2023-03-03T08:42:37.017 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 1/20 2023-03-03T08:42:37.017 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-03-03T08:42:37.017 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2023-03-03T08:42:37.017 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 4/10 2023-03-03T08:42:37.017 INFO:teuthology.orchestra.run.smithi143.stdout: mon client directed command retry = 5 2023-03-03T08:42:37.018 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2023-03-03T08:42:37.018 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2023-03-03T08:42:37.018 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2023-03-03T08:42:37.018 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject delay max = 1 2023-03-03T08:42:37.018 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject delay probability = 0.005 2023-03-03T08:42:37.018 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject delay type = client mds 2023-03-03T08:42:37.018 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject socket failures = 2500 2023-03-03T08:42:37.018 INFO:teuthology.orchestra.run.smithi143.stdout: osd blocked scrub grace period = 3600 2023-03-03T08:42:37.019 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2023-03-03T08:42:37.019 INFO:teuthology.orchestra.run.smithi143.stdout: osd max markdown count = 1000 2023-03-03T08:42:37.019 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2023-03-03T08:42:37.019 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2023-03-03T08:42:37.019 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.019 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2023-03-03T08:42:37.019 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-03-03T08:42:37.020 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2023-03-03T08:42:37.020 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2023-03-03T08:42:37.020 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2023-03-03T08:42:37.020 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2023-03-03T08:42:37.020 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2023-03-03T08:42:37.020 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2023-03-03T08:42:37.020 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.020 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2023-03-03T08:42:37.021 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-03-03T08:42:37.021 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2023-03-03T08:42:37.021 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2023-03-03T08:42:37.021 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2023-03-03T08:42:37.021 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2023-03-03T08:42:37.021 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2023-03-03T08:42:37.021 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2023-03-03T08:42:37.022 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2023-03-03T08:42:37.022 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2023-03-03T08:42:37.022 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2023-03-03T08:42:37.022 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.022 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-03T08:42:37.022 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2023-03-03T08:42:37.022 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2023-03-03T08:42:37.023 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.023 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2023-03-03T08:42:37.023 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-03T08:42:37.023 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-03T08:42:37.023 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.023 INFO:teuthology.orchestra.run.smithi143.stdout: # 1m isn't quite enough 2023-03-03T08:42:37.023 INFO:teuthology.orchestra.run.smithi143.stdout: mon_down_mkfs_grace = 2m 2023-03-03T08:42:37.023 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.024 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_filestore_osds = false 2023-03-03T08:42:37.024 INFO:teuthology.orchestra.run.smithi143.stdout: mon op complaint time = 120 2023-03-03T08:42:37.024 INFO:teuthology.orchestra.run.smithi143.stdout: 2023-03-03T08:42:37.024 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2023-03-03T08:42:37.024 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2023-03-03T08:42:37.024 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2023-03-03T08:42:37.024 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2023-03-03T08:42:37.024 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-03T08:42:37.025 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-03T08:42:37.025 INFO:teuthology.orchestra.run.smithi143.stdout: client mount timeout = 600 2023-03-03T08:42:37.025 INFO:teuthology.orchestra.run.smithi143.stdout: debug client = 20 2023-03-03T08:42:37.025 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-03-03T08:42:37.025 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 900 2023-03-03T08:42:37.025 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 900 2023-03-03T08:42:37.025 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2023-03-03T08:42:37.026 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.b] 2023-03-03T08:42:37.026 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.c] 2023-03-03T08:42:37.026 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.a] 2023-03-03T08:42:37.026 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.c] 2023-03-03T08:42:37.026 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.b] 2023-03-03T08:42:37.026 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.d] 2023-03-03T08:42:37.026 INFO:teuthology.orchestra.run.smithi143.stdout:[mds] 2023-03-03T08:42:37.026 INFO:teuthology.orchestra.run.smithi143.stdout: debug mds = 20 2023-03-03T08:42:37.027 INFO:teuthology.orchestra.run.smithi143.stdout: debug mds balancer = 20 2023-03-03T08:42:37.027 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2023-03-03T08:42:37.027 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal fragment size max = 10000 2023-03-03T08:42:37.027 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal merge size = 5 2023-03-03T08:42:37.027 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal split bits = 3 2023-03-03T08:42:37.027 INFO:teuthology.orchestra.run.smithi143.stdout: mds bal split size = 100 2023-03-03T08:42:37.027 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug frag = True 2023-03-03T08:42:37.027 INFO:teuthology.orchestra.run.smithi143.stdout: mds debug scatterstat = True 2023-03-03T08:42:37.028 INFO:teuthology.orchestra.run.smithi143.stdout: mds op complaint time = 180 2023-03-03T08:42:37.028 INFO:teuthology.orchestra.run.smithi143.stdout: mds verify scatter = True 2023-03-03T08:42:37.028 INFO:teuthology.orchestra.run.smithi143.stdout: mon client directed command retry = 5 2023-03-03T08:42:37.028 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject delay max = 1 2023-03-03T08:42:37.028 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject delay probability = 0.005 2023-03-03T08:42:37.028 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject delay type = client mds osd 2023-03-03T08:42:37.028 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject socket failures = 2500 2023-03-03T08:42:37.029 INFO:teuthology.orchestra.run.smithi143.stdout: osd op complaint time = 180 2023-03-03T08:42:37.029 INFO:teuthology.orchestra.run.smithi143.stdout: rados mon op timeout = 900 2023-03-03T08:42:37.029 INFO:teuthology.orchestra.run.smithi143.stdout: rados osd op timeout = 900 2023-03-03T08:42:37.029 INFO:tasks.ceph:Creating admin key on mon.a... 2023-03-03T08:42:37.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-03-03T08:42:37.105 INFO:tasks.ceph:Copying monmap to all nodes... 2023-03-03T08:42:37.105 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:42:37.105 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-03-03T08:42:37.122 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:42:37.122 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-03-03T08:42:37.178 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2023-03-03T08:42:37.179 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:42:37.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-03T08:42:37.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T08:42:37.256 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:42:37.256 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:42:37.313 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi143.front.sepia.ceph.com 2023-03-03T08:42:37.313 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:42:37.314 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-03T08:42:37.314 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-03T08:42:37.357 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:42:37.358 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:42:37.459 INFO:tasks.ceph:Setting up mon nodes... 2023-03-03T08:42:37.459 INFO:tasks.ceph:Setting up mgr nodes... 2023-03-03T08:42:37.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-03-03T08:42:37.515 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-03-03T08:42:37.518 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-03-03T08:42:37.806 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-03-03T08:42:37.809 INFO:tasks.ceph:Setting up mds nodes... 2023-03-03T08:42:37.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-03-03T08:42:37.865 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-03-03T08:42:37.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-03-03T08:42:37.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-03-03T08:42:37.990 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-03-03T08:42:37.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-03-03T08:42:38.020 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-03-03T08:42:38.082 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-03-03T08:42:38.086 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-03-03T08:42:38.115 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-03-03T08:42:38.211 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-03-03T08:42:38.213 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-03-03T08:42:38.239 INFO:tasks.ceph_client:Setting up client nodes... 2023-03-03T08:42:38.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-03-03T08:42:38.284 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-03-03T08:42:38.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-03-03T08:42:38.340 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-03-03T08:42:38.353 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-03-03T08:42:38.353 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-03T08:42:38.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-03-03T08:42:38.381 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T08:42:38.381 INFO:tasks.ceph:role: osd.0 2023-03-03T08:42:38.381 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2023-03-03T08:42:38.382 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-03T08:42:38.448 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T08:42:38.448 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T08:42:38.448 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T08:42:38.449 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T08:42:38.449 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T08:42:38.449 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-03-03T08:42:38.450 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T08:42:38.450 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T08:42:38.450 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T08:42:38.450 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T08:42:38.470 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-03-03T08:42:38.472 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-03-03T08:42:38.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-03-03T08:42:38.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-03-03T08:42:38.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-03-03T08:42:38.697 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T08:42:38.697 INFO:tasks.ceph:role: osd.1 2023-03-03T08:42:38.698 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2023-03-03T08:42:38.698 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-03T08:42:38.764 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T08:42:38.764 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T08:42:38.764 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T08:42:38.764 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T08:42:38.764 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T08:42:38.765 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-03-03T08:42:38.765 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T08:42:38.765 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T08:42:38.765 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T08:42:38.765 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T08:42:38.786 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-03-03T08:42:38.787 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-03-03T08:42:38.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-03-03T08:42:38.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-03-03T08:42:38.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-03-03T08:42:39.006 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T08:42:39.006 INFO:tasks.ceph:role: osd.2 2023-03-03T08:42:39.006 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2023-03-03T08:42:39.007 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-03T08:42:39.073 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T08:42:39.073 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T08:42:39.073 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T08:42:39.073 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T08:42:39.073 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T08:42:39.074 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-03-03T08:42:39.074 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T08:42:39.074 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T08:42:39.074 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T08:42:39.074 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T08:42:39.096 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-03-03T08:42:39.097 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-03-03T08:42:39.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-03-03T08:42:39.178 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-03-03T08:42:39.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-03-03T08:42:39.316 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-03T08:42:39.316 INFO:tasks.ceph:role: osd.3 2023-03-03T08:42:39.316 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com 2023-03-03T08:42:39.317 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-03T08:42:39.382 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T08:42:39.383 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T08:42:39.383 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T08:42:39.383 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T08:42:39.383 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T08:42:39.383 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-03-03T08:42:39.383 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T08:42:39.383 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T08:42:39.384 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T08:42:39.384 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T08:42:39.405 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-03-03T08:42:39.406 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-03-03T08:42:39.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-03-03T08:42:39.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-03-03T08:42:39.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:42:39.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:39.785+0000 7f529fa19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-03-03T08:42:39.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:39.785+0000 7f529fa19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-03-03T08:42:39.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:39.785+0000 7f529fa19640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T08:42:39.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:39.785+0000 7f529fa19640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T08:42:39.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:39.785+0000 7f529fa19640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-03T08:42:39.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:39.786+0000 7f529fa19640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-03-03T08:42:42.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-03T08:42:42.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:42:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:42.779+0000 7fd17bbf8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-03-03T08:42:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:42.779+0000 7fd17bbf8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-03-03T08:42:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:42.779+0000 7fd17bbf8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T08:42:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:42.779+0000 7fd17bbf8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T08:42:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:42.779+0000 7fd17bbf8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-03T08:42:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:42.779+0000 7fd17bbf8640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-03-03T08:42:45.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-03T08:42:45.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:42:45.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:45.764+0000 7f5ab60be640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-03-03T08:42:45.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:45.764+0000 7f5ab60be640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-03-03T08:42:45.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:45.764+0000 7f5ab60be640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T08:42:45.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:45.764+0000 7f5ab60be640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T08:42:45.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:45.764+0000 7f5ab60be640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-03T08:42:45.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:45.765+0000 7f5ab60be640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-03-03T08:42:48.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-03-03T08:42:48.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:42:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:48.762+0000 7ff98719a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-03-03T08:42:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:48.762+0000 7ff98719a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-03-03T08:42:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:48.762+0000 7ff98719a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T08:42:48.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:48.762+0000 7ff98719a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T08:42:48.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:48.762+0000 7ff98719a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-03T08:42:48.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:42:48.764+0000 7ff98719a640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-03-03T08:42:51.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-03T08:42:51.663 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-03-03T08:42:51.696 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T08:42:51.696 INFO:tasks.ceph:role: osd.4 2023-03-03T08:42:51.697 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com 2023-03-03T08:42:51.697 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-03T08:42:51.774 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T08:42:51.775 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T08:42:51.775 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T08:42:51.775 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T08:42:51.775 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T08:42:51.775 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2023-03-03T08:42:51.775 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T08:42:51.776 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T08:42:51.776 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T08:42:51.776 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T08:42:52.204 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2023-03-03T08:42:52.206 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2023-03-03T08:42:52.206 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-03-03T08:42:52.365 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-03-03T08:42:52.401 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-03-03T08:42:52.470 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T08:42:52.470 INFO:tasks.ceph:role: osd.5 2023-03-03T08:42:52.471 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com 2023-03-03T08:42:52.471 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-03T08:42:52.545 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T08:42:52.546 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T08:42:52.546 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T08:42:52.546 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T08:42:52.546 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T08:42:52.547 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2023-03-03T08:42:52.547 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T08:42:52.547 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T08:42:52.547 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T08:42:52.547 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T08:42:52.970 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2023-03-03T08:42:52.972 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2023-03-03T08:42:52.973 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-03-03T08:42:53.083 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-03-03T08:42:53.114 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-03-03T08:42:53.182 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T08:42:53.182 INFO:tasks.ceph:role: osd.6 2023-03-03T08:42:53.182 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com 2023-03-03T08:42:53.183 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-03T08:42:53.256 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T08:42:53.257 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T08:42:53.257 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T08:42:53.257 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T08:42:53.257 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T08:42:53.257 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2023-03-03T08:42:53.257 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T08:42:53.258 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T08:42:53.258 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T08:42:53.258 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T08:42:53.700 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2023-03-03T08:42:53.702 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2023-03-03T08:42:53.703 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-03-03T08:42:53.875 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-03-03T08:42:53.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-03-03T08:42:53.976 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-03T08:42:53.977 INFO:tasks.ceph:role: osd.7 2023-03-03T08:42:53.977 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi143.front.sepia.ceph.com 2023-03-03T08:42:53.977 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-03T08:42:54.050 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-03T08:42:54.051 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-03T08:42:54.051 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-03T08:42:54.051 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-03T08:42:54.051 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-03T08:42:54.051 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2023-03-03T08:42:54.052 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-03T08:42:54.052 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-03T08:42:54.052 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-03T08:42:54.052 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-03T08:42:54.477 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2023-03-03T08:42:54.479 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2023-03-03T08:42:54.479 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-03-03T08:42:54.635 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-03-03T08:42:54.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:42:54.771 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:54.769+0000 7f664d7ad640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-03-03T08:42:54.771 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:54.769+0000 7f664d7ad640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-03-03T08:42:54.771 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:54.769+0000 7f664d7ad640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T08:42:54.771 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:54.769+0000 7f664d7ad640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T08:42:54.772 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:54.769+0000 7f664d7ad640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-03T08:42:54.772 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:54.769+0000 7f664d7ad640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-03-03T08:42:57.642 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-03-03T08:42:57.674 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:42:57.766 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:57.764+0000 7f3bb7516640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-03-03T08:42:57.766 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:57.764+0000 7f3bb7516640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-03-03T08:42:57.766 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:57.764+0000 7f3bb7516640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T08:42:57.767 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:57.764+0000 7f3bb7516640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T08:42:57.767 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:57.764+0000 7f3bb7516640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-03T08:42:57.767 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:42:57.765+0000 7f3bb7516640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-03-03T08:43:00.632 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-03-03T08:43:00.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:43:00.754 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:00.752+0000 7fccceed5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-03-03T08:43:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:00.753+0000 7fccceed5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-03-03T08:43:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:00.753+0000 7fccceed5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T08:43:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:00.753+0000 7fccceed5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T08:43:00.755 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:00.753+0000 7fccceed5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-03T08:43:00.756 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:00.753+0000 7fccceed5640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-03-03T08:43:03.622 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-03-03T08:43:03.675 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:43:03.766 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:03.764+0000 7f95d05a6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-03-03T08:43:03.767 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:03.765+0000 7f95d05a6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-03-03T08:43:03.767 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:03.765+0000 7f95d05a6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T08:43:03.767 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:03.765+0000 7f95d05a6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T08:43:03.767 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:03.765+0000 7f95d05a6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-03T08:43:03.767 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:03.765+0000 7f95d05a6640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-03-03T08:43:04.947 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:04.950 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:06.629 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-03-03T08:43:06.661 INFO:tasks.ceph:Reading keys from all nodes... 2023-03-03T08:43:06.662 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:06.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-03-03T08:43:06.695 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:06.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-03-03T08:43:06.767 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:06.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-03-03T08:43:06.837 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:06.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-03-03T08:43:06.906 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:06.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-03-03T08:43:06.975 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:06.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-03-03T08:43:07.044 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:07.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-03-03T08:43:07.112 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:07.113 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-03-03T08:43:07.144 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:07.145 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-03-03T08:43:07.214 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:07.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-03-03T08:43:07.283 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:07.284 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-03-03T08:43:07.351 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:07.352 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-03-03T08:43:07.419 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:07.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-03-03T08:43:07.486 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:07.486 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-03-03T08:43:07.553 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:07.553 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-03-03T08:43:07.574 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:07.574 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-03-03T08:43:07.609 INFO:tasks.ceph:Adding keys to all mons... 2023-03-03T08:43:07.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-03T08:43:07.616 DEBUG:teuthology.orchestra.run.smithi143:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-03T08:43:07.652 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.y] 2023-03-03T08:43:07.652 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD9sgFknB+yHhAAkBMkIG3LrkbWvc16EGtO3A== 2023-03-03T08:43:07.653 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.a] 2023-03-03T08:43:07.653 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD9sgFkamuUMxAAEoVGspyvNAgO6kfQN/LKug== 2023-03-03T08:43:07.653 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.c] 2023-03-03T08:43:07.654 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD9sgFkD5X9OhAAXrpXSJnpOJ5JAMoieE5JdA== 2023-03-03T08:43:07.654 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2023-03-03T08:43:07.654 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD/sgFk3JnuLhAATQ3R4OiAQaiL/6/0xblTPA== 2023-03-03T08:43:07.654 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2023-03-03T08:43:07.654 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQACswFkSF+MLhAAb/lcT/AiKzRnouQPzsU7sg== 2023-03-03T08:43:07.654 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2023-03-03T08:43:07.654 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAFswFk8sCuLRAA416Snww1W8YCiwLchi35vA== 2023-03-03T08:43:07.654 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.3] 2023-03-03T08:43:07.655 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAIswFklhGSLRAANR1HtkjvvY3EIMSh9kAsiA== 2023-03-03T08:43:07.655 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2023-03-03T08:43:07.655 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD9sgFkKqMHMBAA0gBrttLyEbUBzzQ/fxjnQA== 2023-03-03T08:43:07.655 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.b] 2023-03-03T08:43:07.655 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD+sgFk5wzsBBAAVWCZ+1dpLFnhtIz2YbkUDg== 2023-03-03T08:43:07.655 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.d] 2023-03-03T08:43:07.656 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD+sgFk9wWODBAA2JbJjzGEM+TgdMk5ES/w5w== 2023-03-03T08:43:07.656 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.4] 2023-03-03T08:43:07.656 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAOswFkmPjsLRAAsFAOeewm4OR8ZQPj9HJS/Q== 2023-03-03T08:43:07.656 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.5] 2023-03-03T08:43:07.656 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQARswFkYlWjLRAARiG/oJHoCqrFgHNisdjITQ== 2023-03-03T08:43:07.656 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.6] 2023-03-03T08:43:07.656 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAUswFk5kX1LBAAo0Kzce0NAUtbA2TeF4NgXg== 2023-03-03T08:43:07.656 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.7] 2023-03-03T08:43:07.657 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAXswFkXpmrLRAAVsufNs5+zRAvwkphkiePlg== 2023-03-03T08:43:07.657 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2023-03-03T08:43:07.657 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD+sgFkh+ntEBAARmJJjoXaBqbtEuT9VoTk1w== 2023-03-03T08:43:07.657 INFO:teuthology.orchestra.run.smithi007.stdout:[client.1] 2023-03-03T08:43:07.657 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQD+sgFkQe5MFBAAUavDfCnbgP1h+C8YUKkT0Q== 2023-03-03T08:43:07.673 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.y] 2023-03-03T08:43:07.673 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQD9sgFknB+yHhAAkBMkIG3LrkbWvc16EGtO3A== 2023-03-03T08:43:07.673 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.a] 2023-03-03T08:43:07.673 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQD9sgFkamuUMxAAEoVGspyvNAgO6kfQN/LKug== 2023-03-03T08:43:07.674 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.c] 2023-03-03T08:43:07.674 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQD9sgFkD5X9OhAAXrpXSJnpOJ5JAMoieE5JdA== 2023-03-03T08:43:07.674 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.0] 2023-03-03T08:43:07.674 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQD/sgFk3JnuLhAATQ3R4OiAQaiL/6/0xblTPA== 2023-03-03T08:43:07.674 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.1] 2023-03-03T08:43:07.675 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQACswFkSF+MLhAAb/lcT/AiKzRnouQPzsU7sg== 2023-03-03T08:43:07.675 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.2] 2023-03-03T08:43:07.675 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQAFswFk8sCuLRAA416Snww1W8YCiwLchi35vA== 2023-03-03T08:43:07.675 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.3] 2023-03-03T08:43:07.675 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQAIswFklhGSLRAANR1HtkjvvY3EIMSh9kAsiA== 2023-03-03T08:43:07.675 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.x] 2023-03-03T08:43:07.675 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQD9sgFkKqMHMBAA0gBrttLyEbUBzzQ/fxjnQA== 2023-03-03T08:43:07.675 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.b] 2023-03-03T08:43:07.676 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQD+sgFk5wzsBBAAVWCZ+1dpLFnhtIz2YbkUDg== 2023-03-03T08:43:07.677 INFO:teuthology.orchestra.run.smithi143.stdout:[mds.d] 2023-03-03T08:43:07.677 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQD+sgFk9wWODBAA2JbJjzGEM+TgdMk5ES/w5w== 2023-03-03T08:43:07.677 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.4] 2023-03-03T08:43:07.677 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQAOswFkmPjsLRAAsFAOeewm4OR8ZQPj9HJS/Q== 2023-03-03T08:43:07.677 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.5] 2023-03-03T08:43:07.677 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQARswFkYlWjLRAARiG/oJHoCqrFgHNisdjITQ== 2023-03-03T08:43:07.677 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.6] 2023-03-03T08:43:07.678 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQAUswFk5kX1LBAAo0Kzce0NAUtbA2TeF4NgXg== 2023-03-03T08:43:07.678 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.7] 2023-03-03T08:43:07.678 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQAXswFkXpmrLRAAVsufNs5+zRAvwkphkiePlg== 2023-03-03T08:43:07.678 INFO:teuthology.orchestra.run.smithi143.stdout:[client.0] 2023-03-03T08:43:07.678 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQD+sgFkh+ntEBAARmJJjoXaBqbtEuT9VoTk1w== 2023-03-03T08:43:07.678 INFO:teuthology.orchestra.run.smithi143.stdout:[client.1] 2023-03-03T08:43:07.678 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQD+sgFkQe5MFBAAUavDfCnbgP1h+C8YUKkT0Q== 2023-03-03T08:43:07.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T08:43:07.702 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T08:43:07.772 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T08:43:07.774 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T08:43:07.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T08:43:07.835 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T08:43:07.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:07.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:07.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:07.948 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.056 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T08:43:08.111 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-03T08:43:08.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T08:43:08.166 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T08:43:08.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T08:43:08.225 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-03T08:43:08.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.283 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.393 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-03T08:43:08.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T08:43:08.504 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T08:43:08.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T08:43:08.562 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-03T08:43:08.617 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-03-03T08:43:08.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-03-03T08:43:08.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T08:43:08.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-03-03T08:43:08.771 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-03-03T08:43:08.797 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T08:43:08.895 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-03-03T08:43:08.921 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-03-03T08:43:08.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-03T08:43:09.086 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-03-03T08:43:09.114 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:43:09.116 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-03T08:43:09.170 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-03-03T08:43:09.170 INFO:tasks.ceph.mon.a:Restarting daemon 2023-03-03T08:43:09.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-03-03T08:43:09.174 INFO:tasks.ceph.mon.a:Started 2023-03-03T08:43:09.174 INFO:tasks.ceph.mon.b:Restarting daemon 2023-03-03T08:43:09.174 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-03-03T08:43:09.212 INFO:tasks.ceph.mon.b:Started 2023-03-03T08:43:09.212 INFO:tasks.ceph.mon.c:Restarting daemon 2023-03-03T08:43:09.213 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-03-03T08:43:09.214 INFO:tasks.ceph.mon.c:Started 2023-03-03T08:43:09.215 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-03-03T08:43:09.216 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-03-03T08:43:09.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-03-03T08:43:09.218 INFO:tasks.ceph.mgr.y:Started 2023-03-03T08:43:09.219 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-03-03T08:43:09.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-03-03T08:43:09.221 INFO:tasks.ceph.mgr.x:Started 2023-03-03T08:43:09.221 INFO:tasks.ceph:Setting crush tunables to default 2023-03-03T08:43:09.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2023-03-03T08:43:09.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.348+0000 7fceac8be700 1 Processor -- start 2023-03-03T08:43:09.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.348+0000 7fceac8be700 1 -- start start 2023-03-03T08:43:09.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fceac8be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 0x7fcea81550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:09.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fceac8be700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcea80c3e30 0x7fcea80c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:09.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fceac8be700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea80c7a60 con 0x7fcea806d870 2023-03-03T08:43:09.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fceac8be700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea80c2af0 con 0x7fcea80c3e30 2023-03-03T08:43:09.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fceac8be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80c3620 con 0x7fcea80c1da0 2023-03-03T08:43:09.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea6d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcea80c3e30 0x7fcea80c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:09.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea6d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcea80c3e30 0x7fcea80c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50538/0 (socket says 172.21.15.7:50538) 2023-03-03T08:43:09.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea6d9d700 1 -- 172.21.15.7:0/3837550177 learned_addr learned my addr 172.21.15.7:0/3837550177 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:09.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea5d9b700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 0x7fcea81550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:09.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea6d9d700 1 -- 172.21.15.7:0/3837550177 >> v2:172.21.15.7:3300/0 conn(0x7fcea80c3e30 msgr2=0x7fcea80c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-03-03T08:43:09.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea6d9d700 1 -- 172.21.15.7:0/3837550177 >> v2:172.21.15.7:3300/0 conn(0x7fcea80c3e30 msgr2=0x7fcea80c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-03T08:43:09.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea6d9d700 1 --2- 172.21.15.7:0/3837550177 >> v2:172.21.15.7:3300/0 conn(0x7fcea80c3e30 0x7fcea80c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-03T08:43:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea6d9d700 1 --2- 172.21.15.7:0/3837550177 >> v2:172.21.15.7:3300/0 conn(0x7fcea80c3e30 0x7fcea80c4290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-03T08:43:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea5d9b700 1 -- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 msgr2=0x7fcea81550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-03-03T08:43:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea5d9b700 1 -- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 msgr2=0x7fcea81550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-03T08:43:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea5d9b700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 0x7fcea81550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-03T08:43:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.349+0000 7fcea5d9b700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 0x7fcea81550c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-03T08:43:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.528+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050683033 0 0) 0x7fcea80c7a60 con 0x7fcea806d870 2023-03-03T08:43:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.528+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce900035b0 con 0x7fcea806d870 2023-03-03T08:43:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.530+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4106635886 0 0) 0x7fce900035b0 con 0x7fcea806d870 2023-03-03T08:43:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.530+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 msgr2=0x7fcea81550c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T08:43:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.530+0000 7fcea559a700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 0x7fcea81550c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 >> v2:172.21.15.7:3300/0 conn(0x7fcea80c3e30 msgr2=0x7fcea80c4290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T08:43:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 --2- 172.21.15.7:0/3837550177 >> v2:172.21.15.7:3300/0 conn(0x7fcea80c3e30 0x7fcea80c4290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea81545e0 con 0x7fcea806d870 2023-03-03T08:43:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fcea00019f0 con 0x7fcea806d870 2023-03-03T08:43:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 >> v1:172.21.15.143:6789/0 conn(0x7fcea806d870 legacy=0x7fcea80c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 0x7fce90003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce90004850 0x7fce90004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 0x7fce90005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80c3620 con 0x7fce90003a70 2023-03-03T08:43:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9000bac0 con 0x7fce90004850 2023-03-03T08:43:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9000bc80 con 0x7fcea80c1da0 2023-03-03T08:43:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea659c700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 0x7fce90003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea6d9d700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 0x7fce90005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea5d9b700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce90004850 0x7fce90004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea659c700 1 -- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 msgr2=0x7fce90005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea659c700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea80c1da0 0x7fce90005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea659c700 1 -- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce90004850 msgr2=0x7fce90004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea659c700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce90004850 0x7fce90004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea659c700 1 -- 172.21.15.7:0/3837550177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce900035b0 con 0x7fce90003a70 2023-03-03T08:43:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea659c700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 0x7fce90003e80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcea000a890 tx=0x7fce90005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc7458ee8351a3e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea000a350 con 0x7fce90003a70 2023-03-03T08:43:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea0012640 con 0x7fce90003a70 2023-03-03T08:43:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.531+0000 7fcea559a700 1 -- 172.21.15.7:0/3837550177 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea00129f0 con 0x7fce90003a70 2023-03-03T08:43:09.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.532+0000 7fceac8be700 1 -- 172.21.15.7:0/3837550177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 msgr2=0x7fce90003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:09.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.532+0000 7fceac8be700 1 --2- 172.21.15.7:0/3837550177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 0x7fce90003e80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcea000a890 tx=0x7fce90005880 comp rx=0 tx=0).stop 2023-03-03T08:43:09.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.532+0000 7fcea6d9d700 1 -- 172.21.15.7:0/3837550177 reap_dead start 2023-03-03T08:43:09.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.532+0000 7fceac8be700 1 -- 172.21.15.7:0/3837550177 shutdown_connections 2023-03-03T08:43:09.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.532+0000 7fceac8be700 1 -- 172.21.15.7:0/3837550177 >> 172.21.15.7:0/3837550177 conn(0x7fcea806c110 msgr2=0x7fcea806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:09.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.532+0000 7fceac8be700 1 -- 172.21.15.7:0/3837550177 shutdown_connections 2023-03-03T08:43:09.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.532+0000 7fceac8be700 1 -- 172.21.15.7:0/3837550177 wait complete. 2023-03-03T08:43:09.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.534+0000 7fceac8be700 1 Processor -- start 2023-03-03T08:43:09.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.534+0000 7fceac8be700 1 -- start start 2023-03-03T08:43:09.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fceac8be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 0x7fcea813c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:09.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fceac8be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea81580b0 0x7fcea814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:09.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fceac8be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea814fcc0 0x7fcea81e9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:09.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fceac8be700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea80acdf0 con 0x7fce90003a70 2023-03-03T08:43:09.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fceac8be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea81eac70 con 0x7fcea814fcc0 2023-03-03T08:43:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fceac8be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea813a9b0 con 0x7fcea81580b0 2023-03-03T08:43:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fcea659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 0x7fcea813c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fcea659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 0x7fcea813c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50556/0 (socket says 172.21.15.7:50556) 2023-03-03T08:43:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fcea659c700 1 -- 172.21.15.7:0/1427809970 learned_addr learned my addr 172.21.15.7:0/1427809970 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fcea5d9b700 1 --2- 172.21.15.7:0/1427809970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea81580b0 0x7fcea814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fcea6d9d700 1 --2- 172.21.15.7:0/1427809970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea814fcc0 0x7fcea81e9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fcea659c700 1 -- 172.21.15.7:0/1427809970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea81580b0 msgr2=0x7fcea814f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fcea659c700 1 --2- 172.21.15.7:0/1427809970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea81580b0 0x7fcea814f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fcea659c700 1 -- 172.21.15.7:0/1427809970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea814fcc0 msgr2=0x7fcea81e9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fcea659c700 1 --2- 172.21.15.7:0/1427809970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea814fcc0 0x7fcea81e9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.535+0000 7fcea659c700 1 -- 172.21.15.7:0/1427809970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea8154ba0 con 0x7fce90003a70 2023-03-03T08:43:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.536+0000 7fcea659c700 1 --2- 172.21.15.7:0/1427809970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 0x7fcea813c280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcea0015040 tx=0x7fcea0001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.536+0000 7fce96ffd700 1 -- 172.21.15.7:0/1427809970 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea000a350 con 0x7fce90003a70 2023-03-03T08:43:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.536+0000 7fce96ffd700 1 -- 172.21.15.7:0/1427809970 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea0003c10 con 0x7fce90003a70 2023-03-03T08:43:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.536+0000 7fce96ffd700 1 -- 172.21.15.7:0/1427809970 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea00127b0 con 0x7fce90003a70 2023-03-03T08:43:09.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.536+0000 7fceac8be700 1 -- 172.21.15.7:0/1427809970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea81548c0 con 0x7fce90003a70 2023-03-03T08:43:09.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.536+0000 7fceac8be700 1 -- 172.21.15.7:0/1427809970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea806dc40 con 0x7fce90003a70 2023-03-03T08:43:09.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.536+0000 7fce96ffd700 1 -- 172.21.15.7:0/1427809970 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fcea0010050 con 0x7fce90003a70 2023-03-03T08:43:09.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.536+0000 7fce96ffd700 1 -- 172.21.15.7:0/1427809970 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcea0005280 con 0x7fce90003a70 2023-03-03T08:43:09.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.537+0000 7fceac8be700 1 -- 172.21.15.7:0/1427809970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea8150140 con 0x7fce90003a70 2023-03-03T08:43:09.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.539+0000 7fce96ffd700 1 -- 172.21.15.7:0/1427809970 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fcea0028070 con 0x7fce90003a70 2023-03-03T08:43:09.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.617+0000 7fceac8be700 1 -- 172.21.15.7:0/1427809970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcea80c2b30 con 0x7fce90003a70 2023-03-03T08:43:09.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.619+0000 7fce96ffd700 1 -- 172.21.15.7:0/1427809970 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcea0016020 con 0x7fce90003a70 2023-03-03T08:43:09.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.620+0000 7fceac8be700 1 -- 172.21.15.7:0/1427809970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 msgr2=0x7fcea813c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:09.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.620+0000 7fceac8be700 1 --2- 172.21.15.7:0/1427809970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 0x7fcea813c280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcea0015040 tx=0x7fcea0001360 comp rx=0 tx=0).stop 2023-03-03T08:43:09.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.620+0000 7fceac8be700 1 -- 172.21.15.7:0/1427809970 shutdown_connections 2023-03-03T08:43:09.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.620+0000 7fceac8be700 1 --2- 172.21.15.7:0/1427809970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcea814fcc0 0x7fcea81e9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:09.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.620+0000 7fceac8be700 1 --2- 172.21.15.7:0/1427809970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce90003a70 0x7fcea813c280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:09.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.620+0000 7fceac8be700 1 --2- 172.21.15.7:0/1427809970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcea81580b0 0x7fcea814f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:09.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.620+0000 7fceac8be700 1 -- 172.21.15.7:0/1427809970 >> 172.21.15.7:0/1427809970 conn(0x7fcea806c110 msgr2=0x7fcea806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:09.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.620+0000 7fceac8be700 1 -- 172.21.15.7:0/1427809970 shutdown_connections 2023-03-03T08:43:09.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:09.620+0000 7fceac8be700 1 -- 172.21.15.7:0/1427809970 wait complete. 2023-03-03T08:43:09.625 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2023-03-03T08:43:09.633 INFO:tasks.ceph:check_enable_crimson: False 2023-03-03T08:43:09.634 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-03-03T08:43:09.634 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:09.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-03-03T08:43:09.645 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:09.643+0000 7fc3bfdb8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-03T08:43:09.662 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:09.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-03-03T08:43:09.709 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:09.707+0000 7fc3bfdb8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-03T08:43:09.728 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:09.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-03-03T08:43:09.784 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:09.782+0000 7fc3bfdb8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-03T08:43:09.795 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:09.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-03-03T08:43:09.862 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:09.862 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-03-03T08:43:09.863 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:09.860+0000 7fc3bfdb8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-03T08:43:09.890 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:09.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-03-03T08:43:09.958 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:09.959 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-03-03T08:43:10.013 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:10.012+0000 7fc3bfdb8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-03T08:43:10.018 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:10.017+0000 7f534e58c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-03T08:43:10.025 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:43:10.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-03-03T08:43:10.086 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:10.085+0000 7f534e58c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-03T08:43:10.092 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 0b11573f-bcee-4126-b626-d34ffe8b1720 0 2023-03-03T08:43:10.164 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:10.163+0000 7f534e58c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-03T08:43:10.247 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:10.246+0000 7f534e58c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-03T08:43:10.251 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.249+0000 7fe4644e0700 1 Processor -- start 2023-03-03T08:43:10.251 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.249+0000 7fe4644e0700 1 -- start start 2023-03-03T08:43:10.253 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe4644e0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe45c0c00e0 0x7fe45c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.254 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe4644e0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe45c154ff0 0x7fe45c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.254 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe4644e0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45c0c7310 con 0x7fe45c0caa30 2023-03-03T08:43:10.254 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe4644e0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe45c155430 con 0x7fe45c0c00e0 2023-03-03T08:43:10.254 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe4644e0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe45c0bfd10 con 0x7fe45c154ff0 2023-03-03T08:43:10.254 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe462a7d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe45c154ff0 0x7fe45c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.254 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe462a7d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe45c154ff0 0x7fe45c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54326/0 (socket says 172.21.15.143:54326) 2023-03-03T08:43:10.255 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe462a7d700 1 -- 172.21.15.143:0/560969400 learned_addr learned my addr 172.21.15.143:0/560969400 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:10.255 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe461a7b700 1 --2- 172.21.15.143:0/560969400 >> v2:172.21.15.7:3300/0 conn(0x7fe45c0c00e0 0x7fe45c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.255 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe462a7d700 1 -- 172.21.15.143:0/560969400 >> v2:172.21.15.7:3300/0 conn(0x7fe45c0c00e0 msgr2=0x7fe45c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:10.255 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe462a7d700 1 --2- 172.21.15.143:0/560969400 >> v2:172.21.15.7:3300/0 conn(0x7fe45c0c00e0 0x7fe45c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.255 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe462a7d700 1 -- 172.21.15.143:0/560969400 >> v1:172.21.15.7:6789/0 conn(0x7fe45c0caa30 legacy=0x7fe45c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.255 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe462a7d700 1 -- 172.21.15.143:0/560969400 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c0bfa30 con 0x7fe45c154ff0 2023-03-03T08:43:10.255 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.250+0000 7fe462a7d700 1 --2- 172.21.15.143:0/560969400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c154ff0 0x7fe45c0cae00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe45c071530 tx=0x7fe458009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e2188c6deb1cc45 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:10.255 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe46127a700 1 -- 172.21.15.143:0/560969400 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458011830 con 0x7fe45c154ff0 2023-03-03T08:43:10.256 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe46127a700 1 -- 172.21.15.143:0/560969400 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe458011990 con 0x7fe45c154ff0 2023-03-03T08:43:10.256 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe46127a700 1 -- 172.21.15.143:0/560969400 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe458011d20 con 0x7fe45c154ff0 2023-03-03T08:43:10.256 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe4644e0700 1 -- 172.21.15.143:0/560969400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c154ff0 msgr2=0x7fe45c0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.256 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe4644e0700 1 --2- 172.21.15.143:0/560969400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c154ff0 0x7fe45c0cae00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe45c071530 tx=0x7fe458009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.256 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe4644e0700 1 -- 172.21.15.143:0/560969400 shutdown_connections 2023-03-03T08:43:10.256 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe4644e0700 1 --2- 172.21.15.143:0/560969400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c154ff0 0x7fe45c0cae00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.256 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe4644e0700 1 --2- 172.21.15.143:0/560969400 >> v2:172.21.15.7:3300/0 conn(0x7fe45c0c00e0 0x7fe45c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.256 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe4644e0700 1 -- 172.21.15.143:0/560969400 >> 172.21.15.143:0/560969400 conn(0x7fe45c066e40 msgr2=0x7fe45c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:10.257 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe4644e0700 1 -- 172.21.15.143:0/560969400 shutdown_connections 2023-03-03T08:43:10.257 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.251+0000 7fe4644e0700 1 -- 172.21.15.143:0/560969400 wait complete. 2023-03-03T08:43:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.252+0000 7fe4644e0700 1 Processor -- start 2023-03-03T08:43:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.252+0000 7fe4644e0700 1 -- start start 2023-03-03T08:43:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe4644e0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe45c0c00e0 0x7fe45c150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe4644e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c0caa30 0x7fe45c157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe4644e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe45c154ff0 0x7fe45c158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.259 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe4644e0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c1ed380 con 0x7fe45c0c00e0 2023-03-03T08:43:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe4644e0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c1ed660 con 0x7fe45c0caa30 2023-03-03T08:43:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe4644e0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c1ed940 con 0x7fe45c154ff0 2023-03-03T08:43:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe461a7b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c0caa30 0x7fe45c157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe462a7d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe45c154ff0 0x7fe45c158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe461a7b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c0caa30 0x7fe45c157ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54328/0 (socket says 172.21.15.143:54328) 2023-03-03T08:43:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe461a7b700 1 -- 172.21.15.143:0/4247711672 learned_addr learned my addr 172.21.15.143:0/4247711672 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe46227c700 1 --2- 172.21.15.143:0/4247711672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe45c0c00e0 0x7fe45c150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.260 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe461a7b700 1 -- 172.21.15.143:0/4247711672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe45c154ff0 msgr2=0x7fe45c158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.261 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe461a7b700 1 --2- 172.21.15.143:0/4247711672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe45c154ff0 0x7fe45c158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.261 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe461a7b700 1 -- 172.21.15.143:0/4247711672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe45c0c00e0 msgr2=0x7fe45c150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.261 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe461a7b700 1 --2- 172.21.15.143:0/4247711672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe45c0c00e0 0x7fe45c150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.261 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe461a7b700 1 -- 172.21.15.143:0/4247711672 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe458009550 con 0x7fe45c0caa30 2023-03-03T08:43:10.261 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.253+0000 7fe462a7d700 1 --2- 172.21.15.143:0/4247711672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe45c154ff0 0x7fe45c158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:10.261 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.254+0000 7fe461a7b700 1 --2- 172.21.15.143:0/4247711672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c0caa30 0x7fe45c157ac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00e990 tx=0x7fe44c00eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:10.261 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.254+0000 7fe452ffd700 1 -- 172.21.15.143:0/4247711672 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0096f0 con 0x7fe45c0caa30 2023-03-03T08:43:10.262 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.254+0000 7fe452ffd700 1 -- 172.21.15.143:0/4247711672 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44c00c690 con 0x7fe45c0caa30 2023-03-03T08:43:10.262 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.254+0000 7fe4644e0700 1 -- 172.21.15.143:0/4247711672 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe45c1547f0 con 0x7fe45c0caa30 2023-03-03T08:43:10.262 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.254+0000 7fe452ffd700 1 -- 172.21.15.143:0/4247711672 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c00ca00 con 0x7fe45c0caa30 2023-03-03T08:43:10.262 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.254+0000 7fe4644e0700 1 -- 172.21.15.143:0/4247711672 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45c154510 con 0x7fe45c0caa30 2023-03-03T08:43:10.262 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.254+0000 7fe452ffd700 1 -- 172.21.15.143:0/4247711672 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fe44c023070 con 0x7fe45c0caa30 2023-03-03T08:43:10.262 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.254+0000 7fe452ffd700 1 -- 172.21.15.143:0/4247711672 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe44c01c450 con 0x7fe45c0caa30 2023-03-03T08:43:10.262 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.255+0000 7fe4644e0700 1 -- 172.21.15.143:0/4247711672 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe45c154ad0 con 0x7fe45c0caa30 2023-03-03T08:43:10.262 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.257+0000 7fe452ffd700 1 -- 172.21.15.143:0/4247711672 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fe44c03bb00 con 0x7fe45c0caa30 2023-03-03T08:43:10.351 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.349+0000 7fe4644e0700 1 -- 172.21.15.143:0/4247711672 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0b11573f-bcee-4126-b626-d34ffe8b1720", "id": 0} v 0) v1 -- 0x7fe45c1531f0 con 0x7fe45c0caa30 2023-03-03T08:43:10.354 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.352+0000 7fe452ffd700 1 -- 172.21.15.143:0/4247711672 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0b11573f-bcee-4126-b626-d34ffe8b1720", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe44c02a030 con 0x7fe45c0caa30 2023-03-03T08:43:10.354 INFO:teuthology.orchestra.run.smithi143.stdout:0 2023-03-03T08:43:10.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.353+0000 7fe4644e0700 1 -- 172.21.15.143:0/4247711672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c0caa30 msgr2=0x7fe45c157ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.353+0000 7fe4644e0700 1 --2- 172.21.15.143:0/4247711672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c0caa30 0x7fe45c157ac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00e990 tx=0x7fe44c00eca0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.353+0000 7fe4644e0700 1 -- 172.21.15.143:0/4247711672 shutdown_connections 2023-03-03T08:43:10.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.353+0000 7fe4644e0700 1 --2- 172.21.15.143:0/4247711672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe45c154ff0 0x7fe45c158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.353+0000 7fe4644e0700 1 --2- 172.21.15.143:0/4247711672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe45c0c00e0 0x7fe45c150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.356 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.353+0000 7fe4644e0700 1 --2- 172.21.15.143:0/4247711672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe45c0caa30 0x7fe45c157ac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.357 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.353+0000 7fe4644e0700 1 -- 172.21.15.143:0/4247711672 >> 172.21.15.143:0/4247711672 conn(0x7fe45c066e40 msgr2=0x7fe45c0c62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:10.357 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.353+0000 7fe4644e0700 1 -- 172.21.15.143:0/4247711672 shutdown_connections 2023-03-03T08:43:10.357 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.353+0000 7fe4644e0700 1 -- 172.21.15.143:0/4247711672 wait complete. 2023-03-03T08:43:10.366 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 7384f568-a97f-412b-a9c2-4c4a88b12b80 1 2023-03-03T08:43:10.397 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:10.396+0000 7f534e58c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-03T08:43:10.487 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.485+0000 7f7038df3700 1 Processor -- start 2023-03-03T08:43:10.487 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.485+0000 7f7038df3700 1 -- start start 2023-03-03T08:43:10.488 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f7038df3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 0x7f7034139d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.488 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f7038df3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70340c7b90 con 0x7f70340c8020 2023-03-03T08:43:10.488 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f7038df3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7034154640 con 0x7f703406d4b0 2023-03-03T08:43:10.489 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f7038df3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70340c7330 con 0x7f70340c1da0 2023-03-03T08:43:10.489 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f703259c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f703406d4b0 0x7f703406d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:49950/0 (socket says 172.21.15.143:49950) 2023-03-03T08:43:10.489 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f7032d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 0x7f7034139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.489 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f703259c700 1 -- 172.21.15.143:0/1789423517 learned_addr learned my addr 172.21.15.143:0/1789423517 (peer_addr_for_me v1:172.21.15.143:0/0) 2023-03-03T08:43:10.489 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f703159a700 1 -- 172.21.15.143:0/1789423517 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182126477 0 0) 0x7f7034154640 con 0x7f703406d4b0 2023-03-03T08:43:10.489 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f703159a700 1 -- 172.21.15.143:0/1789423517 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f701c0035b0 con 0x7f703406d4b0 2023-03-03T08:43:10.489 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f7032d9d700 1 -- 172.21.15.143:0/1789423517 >> v1:172.21.15.143:6789/0 conn(0x7f703406d4b0 legacy=0x7f703406d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.490 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f7032d9d700 1 -- 172.21.15.143:0/1789423517 >> v1:172.21.15.7:6789/0 conn(0x7f70340c8020 legacy=0x7f70340c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.490 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f7032d9d700 1 -- 172.21.15.143:0/1789423517 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70340c7540 con 0x7f70340c1da0 2023-03-03T08:43:10.490 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f7032d9d700 1 --2- 172.21.15.143:0/1789423517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 0x7f7034139d80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f702c009dc0 tx=0x7f702c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7721a5b02784f8f7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:10.490 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.486+0000 7f703159a700 1 -- 172.21.15.143:0/1789423517 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c00e070 con 0x7f70340c1da0 2023-03-03T08:43:10.490 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.487+0000 7f703159a700 1 -- 172.21.15.143:0/1789423517 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c00bb20 con 0x7f70340c1da0 2023-03-03T08:43:10.490 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.487+0000 7f703159a700 1 -- 172.21.15.143:0/1789423517 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c012430 con 0x7f70340c1da0 2023-03-03T08:43:10.492 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.487+0000 7f7038df3700 1 -- 172.21.15.143:0/1789423517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 msgr2=0x7f7034139d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.493 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.487+0000 7f7038df3700 1 --2- 172.21.15.143:0/1789423517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 0x7f7034139d80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f702c009dc0 tx=0x7f702c00b330 comp rx=0 tx=0).stop 2023-03-03T08:43:10.493 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.487+0000 7f7038df3700 1 -- 172.21.15.143:0/1789423517 shutdown_connections 2023-03-03T08:43:10.493 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.487+0000 7f7038df3700 1 --2- 172.21.15.143:0/1789423517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 0x7f7034139d80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.493 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.487+0000 7f7038df3700 1 -- 172.21.15.143:0/1789423517 >> 172.21.15.143:0/1789423517 conn(0x7f703406c110 msgr2=0x7f703406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:10.493 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.487+0000 7f7038df3700 1 -- 172.21.15.143:0/1789423517 shutdown_connections 2023-03-03T08:43:10.493 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.487+0000 7f7038df3700 1 -- 172.21.15.143:0/1789423517 wait complete. 2023-03-03T08:43:10.494 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.488+0000 7f7038df3700 1 Processor -- start 2023-03-03T08:43:10.494 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7038df3700 1 -- start start 2023-03-03T08:43:10.494 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7038df3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703406d4b0 0x7f70340ab420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.494 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7038df3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 0x7f70341ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.494 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7038df3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70340c8020 0x7f70341ed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.494 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7038df3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70340aac20 con 0x7f703406d4b0 2023-03-03T08:43:10.494 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7038df3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70341edd20 con 0x7f70340c8020 2023-03-03T08:43:10.494 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7038df3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7034134a40 con 0x7f70340c1da0 2023-03-03T08:43:10.495 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7032d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70340c8020 0x7f70341ed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.495 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7032d9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70340c8020 0x7f70341ed7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54344/0 (socket says 172.21.15.143:54344) 2023-03-03T08:43:10.495 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7032d9d700 1 -- 172.21.15.143:0/3376966546 learned_addr learned my addr 172.21.15.143:0/3376966546 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:10.495 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f7031d9b700 1 --2- 172.21.15.143:0/3376966546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 0x7f70341ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.495 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.489+0000 7f703259c700 1 --2- 172.21.15.143:0/3376966546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703406d4b0 0x7f70340ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.495 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7031d9b700 1 -- 172.21.15.143:0/3376966546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70340c8020 msgr2=0x7f70341ed7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.495 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7031d9b700 1 --2- 172.21.15.143:0/3376966546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70340c8020 0x7f70341ed7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.496 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7031d9b700 1 -- 172.21.15.143:0/3376966546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703406d4b0 msgr2=0x7f70340ab420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.496 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7031d9b700 1 --2- 172.21.15.143:0/3376966546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703406d4b0 0x7f70340ab420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.496 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7031d9b700 1 -- 172.21.15.143:0/3376966546 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c009a70 con 0x7f70340c1da0 2023-03-03T08:43:10.496 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7032d9d700 1 --2- 172.21.15.143:0/3376966546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70340c8020 0x7f70341ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:10.496 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7031d9b700 1 --2- 172.21.15.143:0/3376966546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 0x7f70341ed2a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f702800c4a0 tx=0x7f702800c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:10.496 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7022ffd700 1 -- 172.21.15.143:0/3376966546 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280052d0 con 0x7f70340c1da0 2023-03-03T08:43:10.497 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7022ffd700 1 -- 172.21.15.143:0/3376966546 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7028005430 con 0x7f70340c1da0 2023-03-03T08:43:10.497 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7038df3700 1 -- 172.21.15.143:0/3376966546 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703406e210 con 0x7f70340c1da0 2023-03-03T08:43:10.497 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7022ffd700 1 -- 172.21.15.143:0/3376966546 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7028006a80 con 0x7f70340c1da0 2023-03-03T08:43:10.497 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7038df3700 1 -- 172.21.15.143:0/3376966546 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70340c7820 con 0x7f70340c1da0 2023-03-03T08:43:10.497 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7022ffd700 1 -- 172.21.15.143:0/3376966546 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f7028010050 con 0x7f70340c1da0 2023-03-03T08:43:10.497 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.490+0000 7f7022ffd700 1 -- 172.21.15.143:0/3376966546 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f70280126d0 con 0x7f70340c1da0 2023-03-03T08:43:10.497 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.491+0000 7f7038df3700 1 -- 172.21.15.143:0/3376966546 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70340c0ee0 con 0x7f70340c1da0 2023-03-03T08:43:10.498 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.495+0000 7f7022ffd700 1 -- 172.21.15.143:0/3376966546 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f7028020060 con 0x7f70340c1da0 2023-03-03T08:43:10.579 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.577+0000 7f7038df3700 1 -- 172.21.15.143:0/3376966546 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7384f568-a97f-412b-a9c2-4c4a88b12b80", "id": 1} v 0) v1 -- 0x7f7034074bb0 con 0x7f70340c1da0 2023-03-03T08:43:10.581 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.580+0000 7f7022ffd700 1 -- 172.21.15.143:0/3376966546 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7384f568-a97f-412b-a9c2-4c4a88b12b80", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7028003b50 con 0x7f70340c1da0 2023-03-03T08:43:10.582 INFO:teuthology.orchestra.run.smithi143.stdout:1 2023-03-03T08:43:10.583 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.581+0000 7f7038df3700 1 -- 172.21.15.143:0/3376966546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 msgr2=0x7f70341ed2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.583 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.581+0000 7f7038df3700 1 --2- 172.21.15.143:0/3376966546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 0x7f70341ed2a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f702800c4a0 tx=0x7f702800c7b0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.583 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.581+0000 7f7038df3700 1 -- 172.21.15.143:0/3376966546 shutdown_connections 2023-03-03T08:43:10.583 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.581+0000 7f7038df3700 1 --2- 172.21.15.143:0/3376966546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70340c8020 0x7f70341ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.583 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.581+0000 7f7038df3700 1 --2- 172.21.15.143:0/3376966546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703406d4b0 0x7f70340ab420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.584 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.581+0000 7f7038df3700 1 --2- 172.21.15.143:0/3376966546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70340c1da0 0x7f70341ed2a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.584 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.581+0000 7f7038df3700 1 -- 172.21.15.143:0/3376966546 >> 172.21.15.143:0/3376966546 conn(0x7f703406c110 msgr2=0x7f70341549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:10.584 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.581+0000 7f7038df3700 1 -- 172.21.15.143:0/3376966546 shutdown_connections 2023-03-03T08:43:10.584 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.581+0000 7f7038df3700 1 -- 172.21.15.143:0/3376966546 wait complete. 2023-03-03T08:43:10.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 7f01219e-308c-417d-950f-d87aaa252e35 2 2023-03-03T08:43:10.716 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.714+0000 7fef23be5700 1 Processor -- start 2023-03-03T08:43:10.716 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.714+0000 7fef23be5700 1 -- start start 2023-03-03T08:43:10.717 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef23be5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef1c0c8020 0x7fef1c0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.718 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef23be5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef1c0c1da0 0x7fef1c0c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.718 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef23be5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c0c7b90 con 0x7fef1c06d4b0 2023-03-03T08:43:10.718 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef23be5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef1c0c7330 con 0x7fef1c0c1da0 2023-03-03T08:43:10.718 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef23be5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef1c154420 con 0x7fef1c0c8020 2023-03-03T08:43:10.718 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef21180700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef1c0c8020 0x7fef1c0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.718 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef21180700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef1c0c8020 0x7fef1c0c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54348/0 (socket says 172.21.15.143:54348) 2023-03-03T08:43:10.719 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef21180700 1 -- 172.21.15.143:0/3105397517 learned_addr learned my addr 172.21.15.143:0/3105397517 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:10.719 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef22182700 1 --2- 172.21.15.143:0/3105397517 >> v2:172.21.15.7:3300/0 conn(0x7fef1c0c1da0 0x7fef1c0c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.719 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef21180700 1 -- 172.21.15.143:0/3105397517 >> v2:172.21.15.7:3300/0 conn(0x7fef1c0c1da0 msgr2=0x7fef1c0c0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:10.719 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef21180700 1 --2- 172.21.15.143:0/3105397517 >> v2:172.21.15.7:3300/0 conn(0x7fef1c0c1da0 0x7fef1c0c0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.719 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef21180700 1 -- 172.21.15.143:0/3105397517 >> v1:172.21.15.7:6789/0 conn(0x7fef1c06d4b0 legacy=0x7fef1c06d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:10.719 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.715+0000 7fef21180700 1 -- 172.21.15.143:0/3105397517 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c154640 con 0x7fef1c0c8020 2023-03-03T08:43:10.719 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef21180700 1 --2- 172.21.15.143:0/3105397517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c0c8020 0x7fef1c0c07d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fef1c06fcb0 tx=0x7fef0c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c81e527bdcfc67f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:10.719 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef2097f700 1 -- 172.21.15.143:0/3105397517 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c004230 con 0x7fef1c0c8020 2023-03-03T08:43:10.720 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef2097f700 1 -- 172.21.15.143:0/3105397517 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c004390 con 0x7fef1c0c8020 2023-03-03T08:43:10.720 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef2097f700 1 -- 172.21.15.143:0/3105397517 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c0036a0 con 0x7fef1c0c8020 2023-03-03T08:43:10.722 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef23be5700 1 -- 172.21.15.143:0/3105397517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c0c8020 msgr2=0x7fef1c0c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.722 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef23be5700 1 --2- 172.21.15.143:0/3105397517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c0c8020 0x7fef1c0c07d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fef1c06fcb0 tx=0x7fef0c009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.722 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef23be5700 1 -- 172.21.15.143:0/3105397517 shutdown_connections 2023-03-03T08:43:10.723 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef23be5700 1 --2- 172.21.15.143:0/3105397517 >> v2:172.21.15.7:3300/0 conn(0x7fef1c0c1da0 0x7fef1c0c0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.723 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef23be5700 1 --2- 172.21.15.143:0/3105397517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c0c8020 0x7fef1c0c07d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.723 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef23be5700 1 -- 172.21.15.143:0/3105397517 >> 172.21.15.143:0/3105397517 conn(0x7fef1c06c110 msgr2=0x7fef1c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:10.723 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef23be5700 1 -- 172.21.15.143:0/3105397517 shutdown_connections 2023-03-03T08:43:10.723 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.716+0000 7fef23be5700 1 -- 172.21.15.143:0/3105397517 wait complete. 2023-03-03T08:43:10.723 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.718+0000 7fef23be5700 1 Processor -- start 2023-03-03T08:43:10.723 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.718+0000 7fef23be5700 1 -- start start 2023-03-03T08:43:10.723 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef23be5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c06d4b0 0x7fef1c0714f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.724 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef23be5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1c0c1da0 0x7fef1c0ab880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.724 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef23be5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef1c0c8020 0x7fef1c0abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.724 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef23be5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c0ab340 con 0x7fef1c0c1da0 2023-03-03T08:43:10.724 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef23be5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c0aa450 con 0x7fef1c06d4b0 2023-03-03T08:43:10.724 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef23be5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c0aa730 con 0x7fef1c0c8020 2023-03-03T08:43:10.724 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef21981700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c06d4b0 0x7fef1c0714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.724 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef21981700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c06d4b0 0x7fef1c0714f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54352/0 (socket says 172.21.15.143:54352) 2023-03-03T08:43:10.724 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef21981700 1 -- 172.21.15.143:0/235418345 learned_addr learned my addr 172.21.15.143:0/235418345 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:10.725 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef22182700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef1c0c8020 0x7fef1c0abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.725 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef21180700 1 --2- 172.21.15.143:0/235418345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1c0c1da0 0x7fef1c0ab880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.725 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef21981700 1 -- 172.21.15.143:0/235418345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef1c0c8020 msgr2=0x7fef1c0abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.725 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef21981700 1 --2- 172.21.15.143:0/235418345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef1c0c8020 0x7fef1c0abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.725 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef21981700 1 -- 172.21.15.143:0/235418345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1c0c1da0 msgr2=0x7fef1c0ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.725 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef21981700 1 --2- 172.21.15.143:0/235418345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1c0c1da0 0x7fef1c0ab880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.725 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef21981700 1 -- 172.21.15.143:0/235418345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c009550 con 0x7fef1c06d4b0 2023-03-03T08:43:10.726 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.719+0000 7fef21981700 1 --2- 172.21.15.143:0/235418345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c06d4b0 0x7fef1c0714f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fef1800b4a0 tx=0x7fef1800b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:10.726 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.720+0000 7fef127fc700 1 -- 172.21.15.143:0/235418345 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18011830 con 0x7fef1c06d4b0 2023-03-03T08:43:10.726 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.720+0000 7fef127fc700 1 -- 172.21.15.143:0/235418345 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef18011990 con 0x7fef1c06d4b0 2023-03-03T08:43:10.726 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.720+0000 7fef23be5700 1 -- 172.21.15.143:0/235418345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1c0c7820 con 0x7fef1c06d4b0 2023-03-03T08:43:10.726 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.720+0000 7fef127fc700 1 -- 172.21.15.143:0/235418345 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef18011d40 con 0x7fef1c06d4b0 2023-03-03T08:43:10.726 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.720+0000 7fef23be5700 1 -- 172.21.15.143:0/235418345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1c0c7540 con 0x7fef1c06d4b0 2023-03-03T08:43:10.726 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.720+0000 7fef127fc700 1 -- 172.21.15.143:0/235418345 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fef1800f050 con 0x7fef1c06d4b0 2023-03-03T08:43:10.726 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.720+0000 7fef127fc700 1 -- 172.21.15.143:0/235418345 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fef1801a820 con 0x7fef1c06d4b0 2023-03-03T08:43:10.727 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.721+0000 7fef23be5700 1 -- 172.21.15.143:0/235418345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1c06e210 con 0x7fef1c06d4b0 2023-03-03T08:43:10.727 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.723+0000 7fef127fc700 1 -- 172.21.15.143:0/235418345 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fef18038b00 con 0x7fef1c06d4b0 2023-03-03T08:43:10.740 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:10.738+0000 7fc3bfdb8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-03T08:43:10.808 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.806+0000 7fef23be5700 1 -- 172.21.15.143:0/235418345 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7f01219e-308c-417d-950f-d87aaa252e35", "id": 2} v 0) v1 -- 0x7fef1c0ab620 con 0x7fef1c06d4b0 2023-03-03T08:43:10.811 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.810+0000 7fef127fc700 1 -- 172.21.15.143:0/235418345 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f01219e-308c-417d-950f-d87aaa252e35", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fef18027070 con 0x7fef1c06d4b0 2023-03-03T08:43:10.812 INFO:teuthology.orchestra.run.smithi143.stdout:2 2023-03-03T08:43:10.812 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.811+0000 7fef23be5700 1 -- 172.21.15.143:0/235418345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c06d4b0 msgr2=0x7fef1c0714f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.813 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.811+0000 7fef23be5700 1 --2- 172.21.15.143:0/235418345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c06d4b0 0x7fef1c0714f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fef1800b4a0 tx=0x7fef1800b7b0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.813 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.811+0000 7fef23be5700 1 -- 172.21.15.143:0/235418345 shutdown_connections 2023-03-03T08:43:10.813 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.811+0000 7fef23be5700 1 --2- 172.21.15.143:0/235418345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef1c0c8020 0x7fef1c0abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.813 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.811+0000 7fef23be5700 1 --2- 172.21.15.143:0/235418345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef1c06d4b0 0x7fef1c0714f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.813 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.811+0000 7fef23be5700 1 --2- 172.21.15.143:0/235418345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef1c0c1da0 0x7fef1c0ab880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.813 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.811+0000 7fef23be5700 1 -- 172.21.15.143:0/235418345 >> 172.21.15.143:0/235418345 conn(0x7fef1c06c110 msgr2=0x7fef1c0c54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:10.814 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.811+0000 7fef23be5700 1 -- 172.21.15.143:0/235418345 shutdown_connections 2023-03-03T08:43:10.814 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.811+0000 7fef23be5700 1 -- 172.21.15.143:0/235418345 wait complete. 2023-03-03T08:43:10.817 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:10.816+0000 7fc3bfdb8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-03T08:43:10.824 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 6776045a-e68c-435b-98c8-8462608e1bc4 3 2023-03-03T08:43:10.897 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:10.896+0000 7fc3bfdb8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-03T08:43:10.946 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.944+0000 7f6d50a2d700 1 Processor -- start 2023-03-03T08:43:10.946 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.944+0000 7f6d50a2d700 1 -- start start 2023-03-03T08:43:10.948 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d50a2d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6d4c1562f0 0x7f6d4c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.948 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d50a2d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c155e60 con 0x7f6d4c14f860 2023-03-03T08:43:10.948 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d50a2d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c06cd90 con 0x7f6d4c0c20d0 2023-03-03T08:43:10.948 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d50a2d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4c06d350 con 0x7f6d4c1562f0 2023-03-03T08:43:10.948 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d4ad9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6d4c1562f0 0x7f6d4c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.949 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d49d9b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f6d4c0c20d0 0x7f6d4c0c27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:49974/0 (socket says 172.21.15.143:49974) 2023-03-03T08:43:10.949 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d4ad9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6d4c1562f0 0x7f6d4c0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54362/0 (socket says 172.21.15.143:54362) 2023-03-03T08:43:10.949 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d49d9b700 1 -- 172.21.15.143:0/999660094 learned_addr learned my addr 172.21.15.143:0/999660094 (peer_addr_for_me v1:172.21.15.143:0/0) 2023-03-03T08:43:10.949 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d4ad9d700 1 -- 172.21.15.143:0/999660094 >> v1:172.21.15.143:6789/0 conn(0x7f6d4c0c20d0 legacy=0x7f6d4c0c27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.949 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d4ad9d700 1 -- 172.21.15.143:0/999660094 >> v1:172.21.15.7:6789/0 conn(0x7f6d4c14f860 legacy=0x7f6d4c152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.949 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d4ad9d700 1 -- 172.21.15.143:0/999660094 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c06d070 con 0x7f6d4c1562f0 2023-03-03T08:43:10.949 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d4ad9d700 1 --2- 172.21.15.143:0/999660094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 0x7f6d4c0736f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009d90 tx=0x7f6d4400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a6d63afbda354a4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:10.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.945+0000 7f6d4959a700 1 -- 172.21.15.143:0/999660094 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4400e070 con 0x7f6d4c1562f0 2023-03-03T08:43:10.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.946+0000 7f6d4959a700 1 -- 172.21.15.143:0/999660094 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4400bbd0 con 0x7f6d4c1562f0 2023-03-03T08:43:10.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.946+0000 7f6d4959a700 1 -- 172.21.15.143:0/999660094 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d44012430 con 0x7f6d4c1562f0 2023-03-03T08:43:10.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.946+0000 7f6d50a2d700 1 -- 172.21.15.143:0/999660094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 msgr2=0x7f6d4c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.946+0000 7f6d50a2d700 1 --2- 172.21.15.143:0/999660094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 0x7f6d4c0736f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009d90 tx=0x7f6d4400b330 comp rx=0 tx=0).stop 2023-03-03T08:43:10.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.946+0000 7f6d50a2d700 1 -- 172.21.15.143:0/999660094 shutdown_connections 2023-03-03T08:43:10.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.946+0000 7f6d50a2d700 1 --2- 172.21.15.143:0/999660094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 0x7f6d4c0736f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.950 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.946+0000 7f6d50a2d700 1 -- 172.21.15.143:0/999660094 >> 172.21.15.143:0/999660094 conn(0x7f6d4c0a94d0 msgr2=0x7f6d4c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:10.951 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.946+0000 7f6d50a2d700 1 -- 172.21.15.143:0/999660094 shutdown_connections 2023-03-03T08:43:10.951 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.946+0000 7f6d50a2d700 1 -- 172.21.15.143:0/999660094 wait complete. 2023-03-03T08:43:10.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.948+0000 7f6d50a2d700 1 Processor -- start 2023-03-03T08:43:10.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.948+0000 7f6d50a2d700 1 -- start start 2023-03-03T08:43:10.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.948+0000 7f6d50a2d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d4c0c20d0 0x7f6d4c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.948+0000 7f6d50a2d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d4c14f860 0x7f6d4c0bd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.948+0000 7f6d50a2d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 0x7f6d4c14d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:10.953 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.948+0000 7f6d50a2d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c0c1280 con 0x7f6d4c0c20d0 2023-03-03T08:43:10.954 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.948+0000 7f6d50a2d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c1ea4d0 con 0x7f6d4c1562f0 2023-03-03T08:43:10.954 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.948+0000 7f6d50a2d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c1eb2b0 con 0x7f6d4c14f860 2023-03-03T08:43:10.954 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d4ad9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 0x7f6d4c14d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.954 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d49d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d4c14f860 0x7f6d4c0bd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.954 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d4ad9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 0x7f6d4c14d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54368/0 (socket says 172.21.15.143:54368) 2023-03-03T08:43:10.954 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d4ad9d700 1 -- 172.21.15.143:0/4034596224 learned_addr learned my addr 172.21.15.143:0/4034596224 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:10.954 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d4a59c700 1 --2- 172.21.15.143:0/4034596224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d4c0c20d0 0x7f6d4c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:10.955 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d49d9b700 1 -- 172.21.15.143:0/4034596224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 msgr2=0x7f6d4c14d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.955 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d49d9b700 1 --2- 172.21.15.143:0/4034596224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 0x7f6d4c14d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.955 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d49d9b700 1 -- 172.21.15.143:0/4034596224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d4c0c20d0 msgr2=0x7f6d4c0c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:10.955 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d49d9b700 1 --2- 172.21.15.143:0/4034596224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d4c0c20d0 0x7f6d4c0c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:10.955 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d49d9b700 1 -- 172.21.15.143:0/4034596224 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d44009a70 con 0x7f6d4c14f860 2023-03-03T08:43:10.955 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d4ad9d700 1 --2- 172.21.15.143:0/4034596224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 0x7f6d4c14d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:10.955 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d49d9b700 1 --2- 172.21.15.143:0/4034596224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d4c14f860 0x7f6d4c0bd490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000c4d0 tx=0x7f6d4000c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:10.956 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d3affd700 1 -- 172.21.15.143:0/4034596224 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4000cea0 con 0x7f6d4c14f860 2023-03-03T08:43:10.956 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d3affd700 1 -- 172.21.15.143:0/4034596224 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d40005d70 con 0x7f6d4c14f860 2023-03-03T08:43:10.956 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d50a2d700 1 -- 172.21.15.143:0/4034596224 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c156730 con 0x7f6d4c14f860 2023-03-03T08:43:10.956 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d3affd700 1 -- 172.21.15.143:0/4034596224 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d40007310 con 0x7f6d4c14f860 2023-03-03T08:43:10.956 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.949+0000 7f6d50a2d700 1 -- 172.21.15.143:0/4034596224 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4c155af0 con 0x7f6d4c14f860 2023-03-03T08:43:10.956 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.950+0000 7f6d3affd700 1 -- 172.21.15.143:0/4034596224 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f6d40019030 con 0x7f6d4c14f860 2023-03-03T08:43:10.956 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.950+0000 7f6d3affd700 1 -- 172.21.15.143:0/4034596224 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f6d4001b840 con 0x7f6d4c14f860 2023-03-03T08:43:10.956 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.950+0000 7f6d50a2d700 1 -- 172.21.15.143:0/4034596224 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4c14fc30 con 0x7f6d4c14f860 2023-03-03T08:43:10.957 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:10.952+0000 7f6d3affd700 1 -- 172.21.15.143:0/4034596224 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f6d40017070 con 0x7f6d4c14f860 2023-03-03T08:43:11.037 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.036+0000 7f6d50a2d700 1 -- 172.21.15.143:0/4034596224 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6776045a-e68c-435b-98c8-8462608e1bc4", "id": 3} v 0) v1 -- 0x7f6d4c04e6e0 con 0x7f6d4c14f860 2023-03-03T08:43:11.040 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:11.038+0000 7fc3bfdb8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-03T08:43:11.040 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.039+0000 7f6d3affd700 1 -- 172.21.15.143:0/4034596224 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6776045a-e68c-435b-98c8-8462608e1bc4", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6d4001b3b0 con 0x7f6d4c14f860 2023-03-03T08:43:11.041 INFO:teuthology.orchestra.run.smithi143.stdout:3 2023-03-03T08:43:11.041 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.040+0000 7f6d50a2d700 1 -- 172.21.15.143:0/4034596224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d4c14f860 msgr2=0x7f6d4c0bd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.042 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.040+0000 7f6d50a2d700 1 --2- 172.21.15.143:0/4034596224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d4c14f860 0x7f6d4c0bd490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000c4d0 tx=0x7f6d4000c7e0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.042 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.040+0000 7f6d50a2d700 1 -- 172.21.15.143:0/4034596224 shutdown_connections 2023-03-03T08:43:11.042 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.040+0000 7f6d50a2d700 1 --2- 172.21.15.143:0/4034596224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d4c1562f0 0x7f6d4c14d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.042 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.040+0000 7f6d50a2d700 1 --2- 172.21.15.143:0/4034596224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d4c0c20d0 0x7f6d4c0c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.043 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.040+0000 7f6d50a2d700 1 --2- 172.21.15.143:0/4034596224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d4c14f860 0x7f6d4c0bd490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.043 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.040+0000 7f6d50a2d700 1 -- 172.21.15.143:0/4034596224 >> 172.21.15.143:0/4034596224 conn(0x7f6d4c0a94d0 msgr2=0x7f6d4c0c3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:11.043 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.040+0000 7f6d50a2d700 1 -- 172.21.15.143:0/4034596224 shutdown_connections 2023-03-03T08:43:11.043 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.040+0000 7f6d50a2d700 1 -- 172.21.15.143:0/4034596224 wait complete. 2023-03-03T08:43:11.053 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new f53e6654-cc96-4ed5-a05c-25a543abefba 4 2023-03-03T08:43:11.102 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:11.101+0000 7f534e58c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-03T08:43:11.173 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.171+0000 7fb493c98700 1 Processor -- start 2023-03-03T08:43:11.173 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.171+0000 7fb493c98700 1 -- start start 2023-03-03T08:43:11.175 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.172+0000 7fb493c98700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb48c155390 0x7fb48c0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.175 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.172+0000 7fb493c98700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c06e020 0x7fb48c071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.175 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.172+0000 7fb493c98700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb48c154910 con 0x7fb48c072520 2023-03-03T08:43:11.175 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.172+0000 7fb493c98700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb48c070b90 con 0x7fb48c155390 2023-03-03T08:43:11.175 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.172+0000 7fb493c98700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c151320 con 0x7fb48c06e020 2023-03-03T08:43:11.175 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.172+0000 7fb491a34700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb48c155390 0x7fb48c0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.172+0000 7fb491233700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c06e020 0x7fb48c071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.172+0000 7fb491a34700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb48c155390 0x7fb48c0c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54370/0 (socket says 172.21.15.143:54370) 2023-03-03T08:43:11.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.172+0000 7fb491a34700 1 -- 172.21.15.143:0/2173349477 learned_addr learned my addr 172.21.15.143:0/2173349477 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:11.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb491233700 1 -- 172.21.15.143:0/2173349477 >> v2:172.21.15.143:3300/0 conn(0x7fb48c155390 msgr2=0x7fb48c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:11.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb491233700 1 --2- 172.21.15.143:0/2173349477 >> v2:172.21.15.143:3300/0 conn(0x7fb48c155390 0x7fb48c0c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb491233700 1 -- 172.21.15.143:0/2173349477 >> v1:172.21.15.7:6789/0 conn(0x7fb48c072520 legacy=0x7fb48c151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.176 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb491233700 1 -- 172.21.15.143:0/2173349477 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c0702a0 con 0x7fb48c06e020 2023-03-03T08:43:11.177 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb491a34700 1 --2- 172.21.15.143:0/2173349477 >> v2:172.21.15.143:3300/0 conn(0x7fb48c155390 0x7fb48c0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:11.177 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb491233700 1 --2- 172.21.15.143:0/2173349477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c06e020 0x7fb48c071fe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb488009840 tx=0x7fb488009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3144e3acfe6c0cd8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:11.177 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb490a32700 1 -- 172.21.15.143:0/2173349477 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488011830 con 0x7fb48c06e020 2023-03-03T08:43:11.177 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb490a32700 1 -- 172.21.15.143:0/2173349477 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb488011990 con 0x7fb48c06e020 2023-03-03T08:43:11.177 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb490a32700 1 -- 172.21.15.143:0/2173349477 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488011d20 con 0x7fb48c06e020 2023-03-03T08:43:11.180 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb493c98700 1 -- 172.21.15.143:0/2173349477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c06e020 msgr2=0x7fb48c071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.180 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.173+0000 7fb493c98700 1 --2- 172.21.15.143:0/2173349477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c06e020 0x7fb48c071fe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb488009840 tx=0x7fb488009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:11.180 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.174+0000 7fb493c98700 1 -- 172.21.15.143:0/2173349477 shutdown_connections 2023-03-03T08:43:11.180 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.174+0000 7fb493c98700 1 --2- 172.21.15.143:0/2173349477 >> v2:172.21.15.143:3300/0 conn(0x7fb48c155390 0x7fb48c0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.180 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.174+0000 7fb493c98700 1 --2- 172.21.15.143:0/2173349477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c06e020 0x7fb48c071fe0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.174+0000 7fb493c98700 1 -- 172.21.15.143:0/2173349477 >> 172.21.15.143:0/2173349477 conn(0x7fb48c06c110 msgr2=0x7fb48c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:11.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.174+0000 7fb493c98700 1 -- 172.21.15.143:0/2173349477 shutdown_connections 2023-03-03T08:43:11.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.174+0000 7fb493c98700 1 -- 172.21.15.143:0/2173349477 wait complete. 2023-03-03T08:43:11.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.175+0000 7fb493c98700 1 Processor -- start 2023-03-03T08:43:11.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.175+0000 7fb493c98700 1 -- start start 2023-03-03T08:43:11.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.175+0000 7fb493c98700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb48c06e020 0x7fb48c0aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.181 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.175+0000 7fb493c98700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb48c072520 0x7fb48c149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.182 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb493c98700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c155390 0x7fb48c147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.182 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb493c98700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c14c8a0 con 0x7fb48c072520 2023-03-03T08:43:11.182 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb493c98700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c14d5d0 con 0x7fb48c06e020 2023-03-03T08:43:11.182 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb493c98700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c14bbd0 con 0x7fb48c155390 2023-03-03T08:43:11.182 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb491a34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb48c06e020 0x7fb48c0aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.182 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb492235700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c155390 0x7fb48c147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.182 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb491a34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb48c06e020 0x7fb48c0aa450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54376/0 (socket says 172.21.15.143:54376) 2023-03-03T08:43:11.182 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb491a34700 1 -- 172.21.15.143:0/595626128 learned_addr learned my addr 172.21.15.143:0/595626128 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:11.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb491233700 1 --2- 172.21.15.143:0/595626128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb48c072520 0x7fb48c149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb492235700 1 -- 172.21.15.143:0/595626128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb48c06e020 msgr2=0x7fb48c0aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb492235700 1 --2- 172.21.15.143:0/595626128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb48c06e020 0x7fb48c0aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb492235700 1 -- 172.21.15.143:0/595626128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb48c072520 msgr2=0x7fb48c149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb492235700 1 --2- 172.21.15.143:0/595626128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb48c072520 0x7fb48c149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb492235700 1 -- 172.21.15.143:0/595626128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c009550 con 0x7fb48c155390 2023-03-03T08:43:11.183 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb491a34700 1 --2- 172.21.15.143:0/595626128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb48c06e020 0x7fb48c0aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:11.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb492235700 1 --2- 172.21.15.143:0/595626128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c155390 0x7fb48c147050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb48400cb90 tx=0x7fb48400cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:11.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb4827fc700 1 -- 172.21.15.143:0/595626128 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb484003c00 con 0x7fb48c155390 2023-03-03T08:43:11.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb4827fc700 1 -- 172.21.15.143:0/595626128 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb484003d60 con 0x7fb48c155390 2023-03-03T08:43:11.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb493c98700 1 -- 172.21.15.143:0/595626128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4880094a0 con 0x7fb48c155390 2023-03-03T08:43:11.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb4827fc700 1 -- 172.21.15.143:0/595626128 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48400aa80 con 0x7fb48c155390 2023-03-03T08:43:11.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb493c98700 1 -- 172.21.15.143:0/595626128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48c070820 con 0x7fb48c155390 2023-03-03T08:43:11.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb4827fc700 1 -- 172.21.15.143:0/595626128 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fb484010050 con 0x7fb48c155390 2023-03-03T08:43:11.184 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.176+0000 7fb4827fc700 1 -- 172.21.15.143:0/595626128 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fb4840058a0 con 0x7fb48c155390 2023-03-03T08:43:11.185 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.177+0000 7fb493c98700 1 -- 172.21.15.143:0/595626128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb48c070540 con 0x7fb48c155390 2023-03-03T08:43:11.185 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:11.179+0000 7f534e58c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-03T08:43:11.185 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.179+0000 7fb4827fc700 1 -- 172.21.15.143:0/595626128 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fb484012420 con 0x7fb48c155390 2023-03-03T08:43:11.263 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.262+0000 7fb493c98700 1 -- 172.21.15.143:0/595626128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f53e6654-cc96-4ed5-a05c-25a543abefba", "id": 4} v 0) v1 -- 0x7fb48c155760 con 0x7fb48c155390 2023-03-03T08:43:11.266 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.265+0000 7fb4827fc700 1 -- 172.21.15.143:0/595626128 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f53e6654-cc96-4ed5-a05c-25a543abefba", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb4840125c0 con 0x7fb48c155390 2023-03-03T08:43:11.266 INFO:teuthology.orchestra.run.smithi143.stdout:4 2023-03-03T08:43:11.267 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.266+0000 7fb493c98700 1 -- 172.21.15.143:0/595626128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c155390 msgr2=0x7fb48c147050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.268 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.266+0000 7fb493c98700 1 --2- 172.21.15.143:0/595626128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c155390 0x7fb48c147050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb48400cb90 tx=0x7fb48400cea0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.268 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.266+0000 7fb493c98700 1 -- 172.21.15.143:0/595626128 shutdown_connections 2023-03-03T08:43:11.268 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.266+0000 7fb493c98700 1 --2- 172.21.15.143:0/595626128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb48c155390 0x7fb48c147050 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.268 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.266+0000 7fb493c98700 1 --2- 172.21.15.143:0/595626128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb48c06e020 0x7fb48c0aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.268 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.266+0000 7fb493c98700 1 --2- 172.21.15.143:0/595626128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb48c072520 0x7fb48c149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.269 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.266+0000 7fb493c98700 1 -- 172.21.15.143:0/595626128 >> 172.21.15.143:0/595626128 conn(0x7fb48c06c110 msgr2=0x7fb48c0c1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:11.269 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.266+0000 7fb493c98700 1 -- 172.21.15.143:0/595626128 shutdown_connections 2023-03-03T08:43:11.269 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.266+0000 7fb493c98700 1 -- 172.21.15.143:0/595626128 wait complete. 2023-03-03T08:43:11.274 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:11.273+0000 7f534e58c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-03T08:43:11.280 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 138d395a-16cf-46f5-beed-2bfde299f1b2 5 2023-03-03T08:43:11.402 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.400+0000 7ff2c337a700 1 Processor -- start 2023-03-03T08:43:11.402 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.400+0000 7ff2c337a700 1 -- start start 2023-03-03T08:43:11.404 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c337a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff2bc14f9d0 0x7ff2bc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.404 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c337a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14fda0 0x7ff2bc071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.404 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c337a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2bc0c2f30 con 0x7ff2bc0abe20 2023-03-03T08:43:11.404 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c337a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2bc0c31d0 con 0x7ff2bc14f9d0 2023-03-03T08:43:11.404 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c337a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc1524a0 con 0x7ff2bc14fda0 2023-03-03T08:43:11.405 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c1917700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ff2bc0abe20 0x7ff2bc1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:49998/0 (socket says 172.21.15.143:49998) 2023-03-03T08:43:11.405 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c0915700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14fda0 0x7ff2bc071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.405 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c1917700 1 -- 172.21.15.143:0/2184779476 learned_addr learned my addr 172.21.15.143:0/2184779476 (peer_addr_for_me v1:172.21.15.143:0/0) 2023-03-03T08:43:11.405 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c1116700 1 --2- 172.21.15.143:0/2184779476 >> v2:172.21.15.7:3300/0 conn(0x7ff2bc14f9d0 0x7ff2bc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.405 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2b3fff700 1 -- 172.21.15.143:0/2184779476 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254312049 0 0) 0x7ff2bc0c2f30 con 0x7ff2bc0abe20 2023-03-03T08:43:11.405 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2b3fff700 1 -- 172.21.15.143:0/2184779476 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a80035b0 con 0x7ff2bc0abe20 2023-03-03T08:43:11.405 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c0915700 1 -- 172.21.15.143:0/2184779476 >> v2:172.21.15.7:3300/0 conn(0x7ff2bc14f9d0 msgr2=0x7ff2bc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:11.406 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c0915700 1 --2- 172.21.15.143:0/2184779476 >> v2:172.21.15.7:3300/0 conn(0x7ff2bc14f9d0 0x7ff2bc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.406 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c0915700 1 -- 172.21.15.143:0/2184779476 >> v1:172.21.15.143:6789/0 conn(0x7ff2bc0abe20 legacy=0x7ff2bc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.406 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c0915700 1 -- 172.21.15.143:0/2184779476 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc0c34b0 con 0x7ff2bc14fda0 2023-03-03T08:43:11.406 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.401+0000 7ff2c0915700 1 --2- 172.21.15.143:0/2184779476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14fda0 0x7ff2bc071890 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac009840 tx=0x7ff2ac009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=265cb93e0998f092 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:11.406 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2b3fff700 1 -- 172.21.15.143:0/2184779476 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac011830 con 0x7ff2bc14fda0 2023-03-03T08:43:11.406 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2b3fff700 1 -- 172.21.15.143:0/2184779476 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ac011990 con 0x7ff2bc14fda0 2023-03-03T08:43:11.406 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2b3fff700 1 -- 172.21.15.143:0/2184779476 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac011d00 con 0x7ff2bc14fda0 2023-03-03T08:43:11.407 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2c337a700 1 -- 172.21.15.143:0/2184779476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14fda0 msgr2=0x7ff2bc071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.407 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2c337a700 1 --2- 172.21.15.143:0/2184779476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14fda0 0x7ff2bc071890 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac009840 tx=0x7ff2ac009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:11.407 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2c337a700 1 -- 172.21.15.143:0/2184779476 shutdown_connections 2023-03-03T08:43:11.407 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2c337a700 1 --2- 172.21.15.143:0/2184779476 >> v2:172.21.15.7:3300/0 conn(0x7ff2bc14f9d0 0x7ff2bc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.407 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2c337a700 1 --2- 172.21.15.143:0/2184779476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14fda0 0x7ff2bc071890 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.407 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2c337a700 1 -- 172.21.15.143:0/2184779476 >> 172.21.15.143:0/2184779476 conn(0x7ff2bc0c8900 msgr2=0x7ff2bc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:11.407 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2c337a700 1 -- 172.21.15.143:0/2184779476 shutdown_connections 2023-03-03T08:43:11.407 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.402+0000 7ff2c337a700 1 -- 172.21.15.143:0/2184779476 wait complete. 2023-03-03T08:43:11.409 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.404+0000 7ff2c337a700 1 Processor -- start 2023-03-03T08:43:11.410 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.404+0000 7ff2c337a700 1 -- start start 2023-03-03T08:43:11.410 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c337a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc0abe20 0x7ff2bc14e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.410 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c337a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14f9d0 0x7ff2bc1de290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.410 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c337a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2bc14fda0 0x7ff2bc14c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.410 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c337a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc1dbb40 con 0x7ff2bc14fda0 2023-03-03T08:43:11.410 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c337a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc1ddc00 con 0x7ff2bc0abe20 2023-03-03T08:43:11.410 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c337a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2bc1db860 con 0x7ff2bc14f9d0 2023-03-03T08:43:11.410 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c0915700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14f9d0 0x7ff2bc1de290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.411 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c1116700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc0abe20 0x7ff2bc14e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.411 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c0915700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14f9d0 0x7ff2bc1de290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:39348/0 (socket says 172.21.15.143:39348) 2023-03-03T08:43:11.411 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c0915700 1 -- 172.21.15.143:0/1541446998 learned_addr learned my addr 172.21.15.143:0/1541446998 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:11.411 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.405+0000 7ff2c1116700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc0abe20 0x7ff2bc14e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54386/0 (socket says 172.21.15.143:54386) 2023-03-03T08:43:11.411 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2c1917700 1 --2- 172.21.15.143:0/1541446998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2bc14fda0 0x7ff2bc14c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.411 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2c1116700 1 -- 172.21.15.143:0/1541446998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14f9d0 msgr2=0x7ff2bc1de290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.412 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2c1116700 1 --2- 172.21.15.143:0/1541446998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14f9d0 0x7ff2bc1de290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.412 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2c1116700 1 -- 172.21.15.143:0/1541446998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2bc14fda0 msgr2=0x7ff2bc14c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.412 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2c1116700 1 --2- 172.21.15.143:0/1541446998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2bc14fda0 0x7ff2bc14c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.412 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2c1116700 1 -- 172.21.15.143:0/1541446998 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac0094a0 con 0x7ff2bc0abe20 2023-03-03T08:43:11.412 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2c0915700 1 --2- 172.21.15.143:0/1541446998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14f9d0 0x7ff2bc1de290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:11.412 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2c1116700 1 --2- 172.21.15.143:0/1541446998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc0abe20 0x7ff2bc14e4a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff2b800e900 tx=0x7ff2b800ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:11.412 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2b1ffb700 1 -- 172.21.15.143:0/1541446998 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8013680 con 0x7ff2bc0abe20 2023-03-03T08:43:11.413 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2b1ffb700 1 -- 172.21.15.143:0/1541446998 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b80137e0 con 0x7ff2bc0abe20 2023-03-03T08:43:11.413 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2b1ffb700 1 -- 172.21.15.143:0/1541446998 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8013b50 con 0x7ff2bc0abe20 2023-03-03T08:43:11.413 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2c337a700 1 -- 172.21.15.143:0/1541446998 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2bc0ac350 con 0x7ff2bc0abe20 2023-03-03T08:43:11.413 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2b1ffb700 1 -- 172.21.15.143:0/1541446998 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7ff2b8019030 con 0x7ff2bc0abe20 2023-03-03T08:43:11.413 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2c337a700 1 -- 172.21.15.143:0/1541446998 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2bc0ab630 con 0x7ff2bc0abe20 2023-03-03T08:43:11.413 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.406+0000 7ff2b1ffb700 1 -- 172.21.15.143:0/1541446998 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7ff2b80062a0 con 0x7ff2bc0abe20 2023-03-03T08:43:11.413 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.407+0000 7ff2c337a700 1 -- 172.21.15.143:0/1541446998 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2bc0c3e10 con 0x7ff2bc0abe20 2023-03-03T08:43:11.414 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.409+0000 7ff2b1ffb700 1 -- 172.21.15.143:0/1541446998 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7ff2b803bb00 con 0x7ff2bc0abe20 2023-03-03T08:43:11.416 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:11.415+0000 7f534e58c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-03T08:43:11.453 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:11.451+0000 7fc3bfdb8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-03T08:43:11.496 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.494+0000 7ff2c337a700 1 -- 172.21.15.143:0/1541446998 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "138d395a-16cf-46f5-beed-2bfde299f1b2", "id": 5} v 0) v1 -- 0x7ff2bc074bb0 con 0x7ff2bc0abe20 2023-03-03T08:43:11.498 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.497+0000 7ff2b1ffb700 1 -- 172.21.15.143:0/1541446998 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "138d395a-16cf-46f5-beed-2bfde299f1b2", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff2b802a070 con 0x7ff2bc0abe20 2023-03-03T08:43:11.499 INFO:teuthology.orchestra.run.smithi143.stdout:5 2023-03-03T08:43:11.499 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.498+0000 7ff2c337a700 1 -- 172.21.15.143:0/1541446998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc0abe20 msgr2=0x7ff2bc14e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.500 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.498+0000 7ff2c337a700 1 --2- 172.21.15.143:0/1541446998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc0abe20 0x7ff2bc14e4a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff2b800e900 tx=0x7ff2b800ec10 comp rx=0 tx=0).stop 2023-03-03T08:43:11.500 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.498+0000 7ff2c337a700 1 -- 172.21.15.143:0/1541446998 shutdown_connections 2023-03-03T08:43:11.500 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.498+0000 7ff2c337a700 1 --2- 172.21.15.143:0/1541446998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2bc14fda0 0x7ff2bc14c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.500 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.498+0000 7ff2c337a700 1 --2- 172.21.15.143:0/1541446998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2bc0abe20 0x7ff2bc14e4a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.500 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.498+0000 7ff2c337a700 1 --2- 172.21.15.143:0/1541446998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2bc14f9d0 0x7ff2bc1de290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.500 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.498+0000 7ff2c337a700 1 -- 172.21.15.143:0/1541446998 >> 172.21.15.143:0/1541446998 conn(0x7ff2bc0c8900 msgr2=0x7ff2bc150d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:11.501 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.498+0000 7ff2c337a700 1 -- 172.21.15.143:0/1541446998 shutdown_connections 2023-03-03T08:43:11.501 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.498+0000 7ff2c337a700 1 -- 172.21.15.143:0/1541446998 wait complete. 2023-03-03T08:43:11.513 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new 4c657cd7-4110-46d8-b01a-a5ddf7dfcf15 6 2023-03-03T08:43:11.635 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.633+0000 7fd836d15700 1 Processor -- start 2023-03-03T08:43:11.635 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.633+0000 7fd836d15700 1 -- start start 2023-03-03T08:43:11.638 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd836d15700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd83006bd70 0x7fd8300c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.638 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd836d15700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8301523c0 con 0x7fd83014ec40 2023-03-03T08:43:11.638 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd836d15700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd830151e40 con 0x7fd8300ba1b0 2023-03-03T08:43:11.638 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd836d15700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8300c3290 con 0x7fd83006bd70 2023-03-03T08:43:11.638 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd82ffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd8300ba1b0 0x7fd83006c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:50008/0 (socket says 172.21.15.143:50008) 2023-03-03T08:43:11.639 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd82ffff700 1 -- 172.21.15.143:0/3293464888 learned_addr learned my addr 172.21.15.143:0/3293464888 (peer_addr_for_me v1:172.21.15.143:0/0) 2023-03-03T08:43:11.639 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd8352b2700 1 --2- 172.21.15.143:0/3293464888 >> v2:172.21.15.7:3300/0 conn(0x7fd83006bd70 0x7fd8300c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.639 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933988027 0 0) 0x7fd830151e40 con 0x7fd8300ba1b0 2023-03-03T08:43:11.639 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8200035b0 con 0x7fd8300ba1b0 2023-03-03T08:43:11.639 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98354901 0 0) 0x7fd8301523c0 con 0x7fd83014ec40 2023-03-03T08:43:11.639 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.634+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd830151e40 con 0x7fd83014ec40 2023-03-03T08:43:11.639 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344546797 0 0) 0x7fd8200035b0 con 0x7fd8300ba1b0 2023-03-03T08:43:11.639 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 >> v2:172.21.15.7:3300/0 conn(0x7fd83006bd70 msgr2=0x7fd8300c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:11.640 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 --2- 172.21.15.143:0/3293464888 >> v2:172.21.15.7:3300/0 conn(0x7fd83006bd70 0x7fd8300c4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.640 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 >> v1:172.21.15.7:6789/0 conn(0x7fd83014ec40 legacy=0x7fd8300c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.640 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8301520e0 con 0x7fd8300ba1b0 2023-03-03T08:43:11.640 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd81c001670 con 0x7fd8300ba1b0 2023-03-03T08:43:11.640 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 >> v1:172.21.15.143:6789/0 conn(0x7fd8300ba1b0 legacy=0x7fd83006c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.641 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 0x7fd820003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.641 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd820004850 0x7fd820004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.641 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd820005310 0x7fd82000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.641 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300c3290 con 0x7fd820004850 2023-03-03T08:43:11.641 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82000e420 con 0x7fd820005310 2023-03-03T08:43:11.641 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82000e5e0 con 0x7fd820003c00 2023-03-03T08:43:11.641 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd834ab1700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 0x7fd820003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.642 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82ffff700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd820004850 0x7fd820004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.642 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd8352b2700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd820005310 0x7fd82000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.642 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd834ab1700 1 -- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd820005310 msgr2=0x7fd82000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:11.642 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd834ab1700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd820005310 0x7fd82000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.642 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd834ab1700 1 -- 172.21.15.143:0/3293464888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd820004850 msgr2=0x7fd820004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:11.642 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd834ab1700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd820004850 0x7fd820004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.642 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd834ab1700 1 -- 172.21.15.143:0/3293464888 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8200035b0 con 0x7fd820003c00 2023-03-03T08:43:11.643 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd8352b2700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd820005310 0x7fd82000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:11.643 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd8352b2700 1 -- 172.21.15.143:0/3293464888 reap_dead start 2023-03-03T08:43:11.643 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd834ab1700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 0x7fd820003ff0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd824001940 tx=0x7fd820005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57d597abb05c3c6a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:11.643 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824001430 con 0x7fd820003c00 2023-03-03T08:43:11.643 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd824001d70 con 0x7fd820003c00 2023-03-03T08:43:11.643 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.635+0000 7fd82f7fe700 1 -- 172.21.15.143:0/3293464888 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824005510 con 0x7fd820003c00 2023-03-03T08:43:11.643 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.636+0000 7fd836d15700 1 -- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 msgr2=0x7fd820003ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.643 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.636+0000 7fd836d15700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 0x7fd820003ff0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd824001940 tx=0x7fd820005880 comp rx=0 tx=0).stop 2023-03-03T08:43:11.644 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.636+0000 7fd836d15700 1 -- 172.21.15.143:0/3293464888 shutdown_connections 2023-03-03T08:43:11.644 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.636+0000 7fd836d15700 1 --2- 172.21.15.143:0/3293464888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 0x7fd820003ff0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.644 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.636+0000 7fd836d15700 1 -- 172.21.15.143:0/3293464888 >> 172.21.15.143:0/3293464888 conn(0x7fd83006af40 msgr2=0x7fd8300c6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:11.644 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.636+0000 7fd836d15700 1 -- 172.21.15.143:0/3293464888 shutdown_connections 2023-03-03T08:43:11.644 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.636+0000 7fd836d15700 1 -- 172.21.15.143:0/3293464888 wait complete. 2023-03-03T08:43:11.647 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.637+0000 7fd836d15700 1 Processor -- start 2023-03-03T08:43:11.647 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.637+0000 7fd836d15700 1 -- start start 2023-03-03T08:43:11.647 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd836d15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 0x7fd8301537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.647 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd836d15700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8300a8240 0x7fd8300ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.647 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd836d15700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8300a7170 0x7fd8301e8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.647 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd836d15700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300c2e30 con 0x7fd8300a7170 2023-03-03T08:43:11.647 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd836d15700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300a9540 con 0x7fd8300a8240 2023-03-03T08:43:11.648 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd836d15700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300a86b0 con 0x7fd820003c00 2023-03-03T08:43:11.648 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd82ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8300a8240 0x7fd8300ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.648 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd82ffff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8300a8240 0x7fd8300ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54404/0 (socket says 172.21.15.143:54404) 2023-03-03T08:43:11.648 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd82ffff700 1 -- 172.21.15.143:0/3320077943 learned_addr learned my addr 172.21.15.143:0/3320077943 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:11.648 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd834ab1700 1 --2- 172.21.15.143:0/3320077943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 0x7fd8301537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.648 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd8352b2700 1 --2- 172.21.15.143:0/3320077943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8300a7170 0x7fd8301e8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.649 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd82ffff700 1 -- 172.21.15.143:0/3320077943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 msgr2=0x7fd8301537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.649 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd82ffff700 1 --2- 172.21.15.143:0/3320077943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 0x7fd8301537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.649 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd82ffff700 1 -- 172.21.15.143:0/3320077943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8300a7170 msgr2=0x7fd8301e8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.649 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd82ffff700 1 --2- 172.21.15.143:0/3320077943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8300a7170 0x7fd8301e8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.649 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.638+0000 7fd82ffff700 1 -- 172.21.15.143:0/3320077943 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8300c1ca0 con 0x7fd8300a8240 2023-03-03T08:43:11.649 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.639+0000 7fd82ffff700 1 --2- 172.21.15.143:0/3320077943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8300a8240 0x7fd8300ac050 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd81c007340 tx=0x7fd81c007650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:11.649 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.639+0000 7fd82d7fa700 1 -- 172.21.15.143:0/3320077943 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00f6c0 con 0x7fd8300a8240 2023-03-03T08:43:11.650 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.639+0000 7fd82d7fa700 1 -- 172.21.15.143:0/3320077943 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81c00f860 con 0x7fd8300a8240 2023-03-03T08:43:11.650 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.639+0000 7fd82d7fa700 1 -- 172.21.15.143:0/3320077943 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00fc10 con 0x7fd8300a8240 2023-03-03T08:43:11.650 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.639+0000 7fd836d15700 1 -- 172.21.15.143:0/3320077943 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8300c19c0 con 0x7fd8300a8240 2023-03-03T08:43:11.650 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.639+0000 7fd836d15700 1 -- 172.21.15.143:0/3320077943 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8300c1f80 con 0x7fd8300a8240 2023-03-03T08:43:11.650 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.639+0000 7fd82d7fa700 1 -- 172.21.15.143:0/3320077943 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fd81c01d750 con 0x7fd8300a8240 2023-03-03T08:43:11.650 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.639+0000 7fd82d7fa700 1 -- 172.21.15.143:0/3320077943 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fd81c0139e0 con 0x7fd8300a8240 2023-03-03T08:43:11.650 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.640+0000 7fd836d15700 1 -- 172.21.15.143:0/3320077943 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8300a6e90 con 0x7fd8300a8240 2023-03-03T08:43:11.651 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.642+0000 7fd82d7fa700 1 -- 172.21.15.143:0/3320077943 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fd8300a6e90 con 0x7fd8300a8240 2023-03-03T08:43:11.727 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.725+0000 7fd836d15700 1 -- 172.21.15.143:0/3320077943 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4c657cd7-4110-46d8-b01a-a5ddf7dfcf15", "id": 6} v 0) v1 -- 0x7fd830074bb0 con 0x7fd8300a8240 2023-03-03T08:43:11.733 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.731+0000 7fd82d7fa700 1 -- 172.21.15.143:0/3320077943 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4c657cd7-4110-46d8-b01a-a5ddf7dfcf15", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd81c013c70 con 0x7fd8300a8240 2023-03-03T08:43:11.733 INFO:teuthology.orchestra.run.smithi143.stdout:6 2023-03-03T08:43:11.734 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.732+0000 7fd836d15700 1 -- 172.21.15.143:0/3320077943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8300a8240 msgr2=0x7fd8300ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.734 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.732+0000 7fd836d15700 1 --2- 172.21.15.143:0/3320077943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8300a8240 0x7fd8300ac050 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd81c007340 tx=0x7fd81c007650 comp rx=0 tx=0).stop 2023-03-03T08:43:11.734 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.732+0000 7fd836d15700 1 -- 172.21.15.143:0/3320077943 shutdown_connections 2023-03-03T08:43:11.734 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.732+0000 7fd836d15700 1 --2- 172.21.15.143:0/3320077943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8300a7170 0x7fd8301e8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.735 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.732+0000 7fd836d15700 1 --2- 172.21.15.143:0/3320077943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd820003c00 0x7fd8301537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.735 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.732+0000 7fd836d15700 1 --2- 172.21.15.143:0/3320077943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8300a8240 0x7fd8300ac050 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.735 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.732+0000 7fd836d15700 1 -- 172.21.15.143:0/3320077943 >> 172.21.15.143:0/3320077943 conn(0x7fd83006af40 msgr2=0x7fd83014cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:11.735 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.732+0000 7fd836d15700 1 -- 172.21.15.143:0/3320077943 shutdown_connections 2023-03-03T08:43:11.735 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.732+0000 7fd836d15700 1 -- 172.21.15.143:0/3320077943 wait complete. 2023-03-03T08:43:11.746 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd new d309d51a-84e6-4634-9a68-c6dd8a62f81b 7 2023-03-03T08:43:11.823 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:11.822+0000 7f534e58c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-03T08:43:11.886 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.884+0000 7f59efd6a700 1 Processor -- start 2023-03-03T08:43:11.889 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.885+0000 7f59efd6a700 1 -- start start 2023-03-03T08:43:11.889 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.886+0000 7f59efd6a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e814dd20 0x7f59e80aa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.889 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.886+0000 7f59efd6a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59e80a93d0 0x7f59e8150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.889 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.886+0000 7f59efd6a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59e80c0c60 0x7f59e8151210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.889 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.886+0000 7f59efd6a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e814e0f0 con 0x7f59e80c0c60 2023-03-03T08:43:11.890 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.886+0000 7f59efd6a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e806c100 con 0x7f59e80a93d0 2023-03-03T08:43:11.890 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.886+0000 7f59efd6a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80ac3b0 con 0x7f59e814dd20 2023-03-03T08:43:11.890 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.886+0000 7f59ed305700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59e80a93d0 0x7f59e8150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.890 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.886+0000 7f59edb06700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e814dd20 0x7f59e80aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.890 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.886+0000 7f59ed305700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59e80a93d0 0x7f59e8150cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:54410/0 (socket says 172.21.15.143:54410) 2023-03-03T08:43:11.890 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.886+0000 7f59ed305700 1 -- 172.21.15.143:0/2531433748 learned_addr learned my addr 172.21.15.143:0/2531433748 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:11.890 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ed305700 1 -- 172.21.15.143:0/2531433748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e814dd20 msgr2=0x7f59e80aa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:11.891 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ed305700 1 --2- 172.21.15.143:0/2531433748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e814dd20 0x7f59e80aa110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.891 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ee307700 1 --2- 172.21.15.143:0/2531433748 >> v2:172.21.15.7:3300/0 conn(0x7f59e80c0c60 0x7f59e8151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.891 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ed305700 1 -- 172.21.15.143:0/2531433748 >> v2:172.21.15.7:3300/0 conn(0x7f59e80c0c60 msgr2=0x7f59e8151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:11.891 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ed305700 1 --2- 172.21.15.143:0/2531433748 >> v2:172.21.15.7:3300/0 conn(0x7f59e80c0c60 0x7f59e8151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.891 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ed305700 1 -- 172.21.15.143:0/2531433748 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e8150840 con 0x7f59e80a93d0 2023-03-03T08:43:11.891 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59edb06700 1 --2- 172.21.15.143:0/2531433748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e814dd20 0x7f59e80aa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:11.891 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ee307700 1 --2- 172.21.15.143:0/2531433748 >> v2:172.21.15.7:3300/0 conn(0x7f59e80c0c60 0x7f59e8151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:11.892 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ed305700 1 --2- 172.21.15.143:0/2531433748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e80a93d0 0x7f59e8150cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f59e806edd0 tx=0x7f59d8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9109cffff2488051 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:11.892 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ecb04700 1 -- 172.21.15.143:0/2531433748 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d800b850 con 0x7f59e80a93d0 2023-03-03T08:43:11.892 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ecb04700 1 -- 172.21.15.143:0/2531433748 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d800b9b0 con 0x7f59e80a93d0 2023-03-03T08:43:11.892 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.887+0000 7f59ecb04700 1 -- 172.21.15.143:0/2531433748 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d800bd40 con 0x7f59e80a93d0 2023-03-03T08:43:11.894 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.888+0000 7f59efd6a700 1 -- 172.21.15.143:0/2531433748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e80a93d0 msgr2=0x7f59e8150cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.894 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.888+0000 7f59efd6a700 1 --2- 172.21.15.143:0/2531433748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e80a93d0 0x7f59e8150cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f59e806edd0 tx=0x7f59d8009e30 comp rx=0 tx=0).stop 2023-03-03T08:43:11.894 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.888+0000 7f59efd6a700 1 -- 172.21.15.143:0/2531433748 shutdown_connections 2023-03-03T08:43:11.894 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.888+0000 7f59efd6a700 1 --2- 172.21.15.143:0/2531433748 >> v2:172.21.15.7:3300/0 conn(0x7f59e80c0c60 0x7f59e8151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.894 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.888+0000 7f59efd6a700 1 --2- 172.21.15.143:0/2531433748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e814dd20 0x7f59e80aa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.894 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.888+0000 7f59efd6a700 1 --2- 172.21.15.143:0/2531433748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e80a93d0 0x7f59e8150cd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.895 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.888+0000 7f59efd6a700 1 -- 172.21.15.143:0/2531433748 >> 172.21.15.143:0/2531433748 conn(0x7f59e80c62e0 msgr2=0x7f59e80c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:11.895 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.888+0000 7f59efd6a700 1 -- 172.21.15.143:0/2531433748 shutdown_connections 2023-03-03T08:43:11.895 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.888+0000 7f59efd6a700 1 -- 172.21.15.143:0/2531433748 wait complete. 2023-03-03T08:43:11.895 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.890+0000 7f59efd6a700 1 Processor -- start 2023-03-03T08:43:11.895 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.890+0000 7f59efd6a700 1 -- start start 2023-03-03T08:43:11.895 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59efd6a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e80a93d0 0x7f59e814f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.895 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59efd6a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e80c0c60 0x7f59e81470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.895 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59efd6a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e814dd20 0x7f59e81475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:11.896 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59efd6a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e8153a20 con 0x7f59e80c0c60 2023-03-03T08:43:11.896 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59efd6a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e8152070 con 0x7f59e814dd20 2023-03-03T08:43:11.896 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59efd6a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e81ed9a0 con 0x7f59e80a93d0 2023-03-03T08:43:11.896 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59edb06700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e80a93d0 0x7f59e814f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.896 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59edb06700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e80a93d0 0x7f59e814f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.143:39374/0 (socket says 172.21.15.143:39374) 2023-03-03T08:43:11.896 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59edb06700 1 -- 172.21.15.143:0/2480670090 learned_addr learned my addr 172.21.15.143:0/2480670090 (peer_addr_for_me v2:172.21.15.143:0/0) 2023-03-03T08:43:11.896 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59ee307700 1 --2- 172.21.15.143:0/2480670090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e814dd20 0x7f59e81475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.896 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59ed305700 1 --2- 172.21.15.143:0/2480670090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e80c0c60 0x7f59e81470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:11.897 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59edb06700 1 -- 172.21.15.143:0/2480670090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e814dd20 msgr2=0x7f59e81475f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.897 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59edb06700 1 --2- 172.21.15.143:0/2480670090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e814dd20 0x7f59e81475f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.897 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59edb06700 1 -- 172.21.15.143:0/2480670090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e80c0c60 msgr2=0x7f59e81470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:11.897 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59edb06700 1 --2- 172.21.15.143:0/2480670090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e80c0c60 0x7f59e81470b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:11.897 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.891+0000 7f59edb06700 1 -- 172.21.15.143:0/2480670090 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e40094a0 con 0x7f59e80a93d0 2023-03-03T08:43:11.897 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.892+0000 7f59edb06700 1 --2- 172.21.15.143:0/2480670090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e80a93d0 0x7f59e814f610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f59e400ec10 tx=0x7f59e400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:11.898 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.892+0000 7f59de7fc700 1 -- 172.21.15.143:0/2480670090 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e4009760 con 0x7f59e80a93d0 2023-03-03T08:43:11.898 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.892+0000 7f59de7fc700 1 -- 172.21.15.143:0/2480670090 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e400c850 con 0x7f59e80a93d0 2023-03-03T08:43:11.898 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.892+0000 7f59de7fc700 1 -- 172.21.15.143:0/2480670090 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e400cbe0 con 0x7f59e80a93d0 2023-03-03T08:43:11.898 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.892+0000 7f59efd6a700 1 -- 172.21.15.143:0/2480670090 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d8009550 con 0x7f59e80a93d0 2023-03-03T08:43:11.899 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.892+0000 7f59efd6a700 1 -- 172.21.15.143:0/2480670090 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e80a98a0 con 0x7f59e80a93d0 2023-03-03T08:43:11.899 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.892+0000 7f59de7fc700 1 -- 172.21.15.143:0/2480670090 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f59e4019030 con 0x7f59e80a93d0 2023-03-03T08:43:11.899 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.892+0000 7f59de7fc700 1 -- 172.21.15.143:0/2480670090 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f59e4013470 con 0x7f59e80a93d0 2023-03-03T08:43:11.899 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.893+0000 7f59efd6a700 1 -- 172.21.15.143:0/2480670090 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e80a9b40 con 0x7f59e80a93d0 2023-03-03T08:43:11.899 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:11.897+0000 7f59de7fc700 1 -- 172.21.15.143:0/2480670090 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f59e4021060 con 0x7f59e80a93d0 2023-03-03T08:43:12.002 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.000+0000 7f59efd6a700 1 -- 172.21.15.143:0/2480670090 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d309d51a-84e6-4634-9a68-c6dd8a62f81b", "id": 7} v 0) v1 -- 0x7f59e80ac0d0 con 0x7f59e80a93d0 2023-03-03T08:43:12.010 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.008+0000 7f59de7fc700 1 -- 172.21.15.143:0/2480670090 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d309d51a-84e6-4634-9a68-c6dd8a62f81b", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f59e4017070 con 0x7f59e80a93d0 2023-03-03T08:43:12.010 INFO:teuthology.orchestra.run.smithi143.stdout:7 2023-03-03T08:43:12.011 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.009+0000 7f59efd6a700 1 -- 172.21.15.143:0/2480670090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e80a93d0 msgr2=0x7f59e814f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:12.011 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.009+0000 7f59efd6a700 1 --2- 172.21.15.143:0/2480670090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e80a93d0 0x7f59e814f610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f59e400ec10 tx=0x7f59e400ef20 comp rx=0 tx=0).stop 2023-03-03T08:43:12.011 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.009+0000 7f59efd6a700 1 -- 172.21.15.143:0/2480670090 shutdown_connections 2023-03-03T08:43:12.012 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.009+0000 7f59efd6a700 1 --2- 172.21.15.143:0/2480670090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e814dd20 0x7f59e81475f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.012 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.009+0000 7f59efd6a700 1 --2- 172.21.15.143:0/2480670090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e80a93d0 0x7f59e814f610 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.012 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.009+0000 7f59efd6a700 1 --2- 172.21.15.143:0/2480670090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e80c0c60 0x7f59e81470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.012 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.009+0000 7f59efd6a700 1 -- 172.21.15.143:0/2480670090 >> 172.21.15.143:0/2480670090 conn(0x7f59e80c62e0 msgr2=0x7f59e81d9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:12.012 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.009+0000 7f59efd6a700 1 -- 172.21.15.143:0/2480670090 shutdown_connections 2023-03-03T08:43:12.012 INFO:teuthology.orchestra.run.smithi143.stderr:2023-03-03T08:43:12.009+0000 7f59efd6a700 1 -- 172.21.15.143:0/2480670090 wait complete. 2023-03-03T08:43:12.023 INFO:tasks.ceph.osd.0:Restarting daemon 2023-03-03T08:43:12.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-03-03T08:43:12.025 INFO:tasks.ceph.osd.0:Started 2023-03-03T08:43:12.025 INFO:tasks.ceph.osd.1:Restarting daemon 2023-03-03T08:43:12.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-03-03T08:43:12.027 INFO:tasks.ceph.osd.1:Started 2023-03-03T08:43:12.028 INFO:tasks.ceph.osd.2:Restarting daemon 2023-03-03T08:43:12.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-03-03T08:43:12.030 INFO:tasks.ceph.osd.2:Started 2023-03-03T08:43:12.030 INFO:tasks.ceph.osd.3:Restarting daemon 2023-03-03T08:43:12.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-03-03T08:43:12.032 INFO:tasks.ceph.osd.3:Started 2023-03-03T08:43:12.032 INFO:tasks.ceph.osd.4:Restarting daemon 2023-03-03T08:43:12.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-03-03T08:43:12.035 INFO:tasks.ceph.osd.4:Started 2023-03-03T08:43:12.035 INFO:tasks.ceph.osd.5:Restarting daemon 2023-03-03T08:43:12.035 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-03-03T08:43:12.037 INFO:tasks.ceph.osd.5:Started 2023-03-03T08:43:12.038 INFO:tasks.ceph.osd.6:Restarting daemon 2023-03-03T08:43:12.038 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-03-03T08:43:12.039 INFO:tasks.ceph.osd.6:Started 2023-03-03T08:43:12.040 INFO:tasks.ceph.osd.7:Restarting daemon 2023-03-03T08:43:12.040 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-03-03T08:43:12.042 INFO:tasks.ceph.osd.7:Started 2023-03-03T08:43:12.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:12.059 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:12.057+0000 7fc3bfdb8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-03T08:43:12.142 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:12.140+0000 7fc3bfdb8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-03T08:43:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.206+0000 7feb3159a700 1 Processor -- start 2023-03-03T08:43:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.206+0000 7feb3159a700 1 -- start start 2023-03-03T08:43:12.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.206+0000 7feb3159a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb2c14b020 0x7feb2c0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:12.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.206+0000 7feb3159a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb2c0aad20 0x7feb2c157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:12.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.206+0000 7feb3159a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2c1588f0 con 0x7feb2c071d20 2023-03-03T08:43:12.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.206+0000 7feb3159a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c0728b0 con 0x7feb2c0aad20 2023-03-03T08:43:12.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb3159a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c155070 con 0x7feb2c14b020 2023-03-03T08:43:12.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb2a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb2c0aad20 0x7feb2c157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:12.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb2b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7feb2c071d20 0x7feb2c1581e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44580/0 (socket says 172.21.15.7:44580) 2023-03-03T08:43:12.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb2a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb2c0aad20 0x7feb2c157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50598/0 (socket says 172.21.15.7:50598) 2023-03-03T08:43:12.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb2b7fe700 1 -- 172.21.15.7:0/4151166589 learned_addr learned my addr 172.21.15.7:0/4151166589 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:12.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb2affd700 1 --2- 172.21.15.7:0/4151166589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb2c14b020 0x7feb2c0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:12.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb2a7fc700 1 -- 172.21.15.7:0/4151166589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb2c14b020 msgr2=0x7feb2c0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:12.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb2a7fc700 1 --2- 172.21.15.7:0/4151166589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb2c14b020 0x7feb2c0717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb2a7fc700 1 -- 172.21.15.7:0/4151166589 >> v1:172.21.15.7:6789/0 conn(0x7feb2c071d20 legacy=0x7feb2c1581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:12.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb29ffb700 1 -- 172.21.15.7:0/4151166589 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160015228 0 0) 0x7feb2c1588f0 con 0x7feb2c071d20 2023-03-03T08:43:12.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb2a7fc700 1 -- 172.21.15.7:0/4151166589 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c06f5b0 con 0x7feb2c0aad20 2023-03-03T08:43:12.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb2a7fc700 1 --2- 172.21.15.7:0/4151166589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c0aad20 0x7feb2c157ca0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feb20009840 tx=0x7feb20009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9f5b29bbba5c5f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:12.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb29ffb700 1 -- 172.21.15.7:0/4151166589 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20011830 con 0x7feb2c0aad20 2023-03-03T08:43:12.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb29ffb700 1 -- 172.21.15.7:0/4151166589 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb20011990 con 0x7feb2c0aad20 2023-03-03T08:43:12.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.207+0000 7feb29ffb700 1 -- 172.21.15.7:0/4151166589 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20011d20 con 0x7feb2c0aad20 2023-03-03T08:43:12.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.208+0000 7feb3159a700 1 -- 172.21.15.7:0/4151166589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c0aad20 msgr2=0x7feb2c157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:12.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.208+0000 7feb3159a700 1 --2- 172.21.15.7:0/4151166589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c0aad20 0x7feb2c157ca0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feb20009840 tx=0x7feb20009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:12.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.208+0000 7feb3159a700 1 -- 172.21.15.7:0/4151166589 shutdown_connections 2023-03-03T08:43:12.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.208+0000 7feb3159a700 1 --2- 172.21.15.7:0/4151166589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb2c14b020 0x7feb2c0717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.208+0000 7feb3159a700 1 --2- 172.21.15.7:0/4151166589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c0aad20 0x7feb2c157ca0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.208+0000 7feb3159a700 1 -- 172.21.15.7:0/4151166589 >> 172.21.15.7:0/4151166589 conn(0x7feb2c0c8900 msgr2=0x7feb2c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:12.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.208+0000 7feb3159a700 1 -- 172.21.15.7:0/4151166589 shutdown_connections 2023-03-03T08:43:12.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.208+0000 7feb3159a700 1 -- 172.21.15.7:0/4151166589 wait complete. 2023-03-03T08:43:12.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.209+0000 7feb3159a700 1 Processor -- start 2023-03-03T08:43:12.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.209+0000 7feb3159a700 1 -- start start 2023-03-03T08:43:12.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb3159a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c071d20 0x7feb2c0c7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:12.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb3159a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb2c0aad20 0x7feb2c1dde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:12.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb3159a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c14b020 0x7feb2c1de340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:12.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb3159a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c152500 con 0x7feb2c071d20 2023-03-03T08:43:12.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb3159a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c154030 con 0x7feb2c14b020 2023-03-03T08:43:12.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb3159a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c1534b0 con 0x7feb2c0aad20 2023-03-03T08:43:12.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb2affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c071d20 0x7feb2c0c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:12.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb2affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c071d20 0x7feb2c0c7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50602/0 (socket says 172.21.15.7:50602) 2023-03-03T08:43:12.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb2affd700 1 -- 172.21.15.7:0/3553964183 learned_addr learned my addr 172.21.15.7:0/3553964183 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:12.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb2a7fc700 1 --2- 172.21.15.7:0/3553964183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb2c0aad20 0x7feb2c1dde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:12.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb2b7fe700 1 --2- 172.21.15.7:0/3553964183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c14b020 0x7feb2c1de340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:12.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb2affd700 1 -- 172.21.15.7:0/3553964183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb2c0aad20 msgr2=0x7feb2c1dde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:12.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb2affd700 1 --2- 172.21.15.7:0/3553964183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb2c0aad20 0x7feb2c1dde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb2affd700 1 -- 172.21.15.7:0/3553964183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c14b020 msgr2=0x7feb2c1de340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:12.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb2affd700 1 --2- 172.21.15.7:0/3553964183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c14b020 0x7feb2c1de340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.210+0000 7feb2affd700 1 -- 172.21.15.7:0/3553964183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200094a0 con 0x7feb2c071d20 2023-03-03T08:43:12.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.211+0000 7feb2affd700 1 --2- 172.21.15.7:0/3553964183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c071d20 0x7feb2c0c7630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00e850 tx=0x7feb1c00eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:12.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.211+0000 7feb137fe700 1 -- 172.21.15.7:0/3553964183 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0095b0 con 0x7feb2c071d20 2023-03-03T08:43:12.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.211+0000 7feb137fe700 1 -- 172.21.15.7:0/3553964183 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c013680 con 0x7feb2c071d20 2023-03-03T08:43:12.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.211+0000 7feb3159a700 1 -- 172.21.15.7:0/3553964183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c06fb70 con 0x7feb2c071d20 2023-03-03T08:43:12.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.211+0000 7feb137fe700 1 -- 172.21.15.7:0/3553964183 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c013a10 con 0x7feb2c071d20 2023-03-03T08:43:12.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.211+0000 7feb3159a700 1 -- 172.21.15.7:0/3553964183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c06f890 con 0x7feb2c071d20 2023-03-03T08:43:12.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.212+0000 7feb137fe700 1 -- 172.21.15.7:0/3553964183 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7feb1c019030 con 0x7feb2c071d20 2023-03-03T08:43:12.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.212+0000 7feb3159a700 1 -- 172.21.15.7:0/3553964183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c155350 con 0x7feb2c071d20 2023-03-03T08:43:12.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.212+0000 7feb137fe700 1 -- 172.21.15.7:0/3553964183 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7feb1c0123d0 con 0x7feb2c071d20 2023-03-03T08:43:12.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.214+0000 7feb137fe700 1 -- 172.21.15.7:0/3553964183 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7feb1c021060 con 0x7feb2c071d20 2023-03-03T08:43:12.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.316+0000 7feb3159a700 1 -- 172.21.15.7:0/3553964183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb2c074bb0 con 0x7feb2c071d20 2023-03-03T08:43:12.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.317+0000 7feb137fe700 1 -- 172.21.15.7:0/3553964183 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7feb1c017070 con 0x7feb2c071d20 2023-03-03T08:43:12.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:12.319 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:12.002629+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:12.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.319+0000 7feb3159a700 1 -- 172.21.15.7:0/3553964183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c071d20 msgr2=0x7feb2c0c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:12.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.319+0000 7feb3159a700 1 --2- 172.21.15.7:0/3553964183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c071d20 0x7feb2c0c7630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00e850 tx=0x7feb1c00eb60 comp rx=0 tx=0).stop 2023-03-03T08:43:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.319+0000 7feb3159a700 1 -- 172.21.15.7:0/3553964183 shutdown_connections 2023-03-03T08:43:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.319+0000 7feb3159a700 1 --2- 172.21.15.7:0/3553964183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c14b020 0x7feb2c1de340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.319+0000 7feb3159a700 1 --2- 172.21.15.7:0/3553964183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb2c071d20 0x7feb2c0c7630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.319+0000 7feb3159a700 1 --2- 172.21.15.7:0/3553964183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb2c0aad20 0x7feb2c1dde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.319+0000 7feb3159a700 1 -- 172.21.15.7:0/3553964183 >> 172.21.15.7:0/3553964183 conn(0x7feb2c0c8900 msgr2=0x7feb2c0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.319+0000 7feb3159a700 1 -- 172.21.15.7:0/3553964183 shutdown_connections 2023-03-03T08:43:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:12.319+0000 7feb3159a700 1 -- 172.21.15.7:0/3553964183 wait complete. 2023-03-03T08:43:12.341 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-03-03T08:43:12.342 INFO:tasks.ceph:Waiting for OSDs to come up 2023-03-03T08:43:12.451 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:12.449+0000 7f534e58c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-03T08:43:12.523 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:12.522+0000 7f534e58c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-03T08:43:12.646 INFO:tasks.ceph.osd.6.smithi143.stderr:2023-03-03T08:43:12.644+0000 7f1e7c290640 -1 Falling back to public interface 2023-03-03T08:43:12.649 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-03-03T08:43:12.648+0000 7f94f33ad640 -1 Falling back to public interface 2023-03-03T08:43:12.652 INFO:tasks.ceph.osd.7.smithi143.stderr:2023-03-03T08:43:12.650+0000 7f93f13b1640 -1 Falling back to public interface 2023-03-03T08:43:12.657 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-03-03T08:43:12.656+0000 7fc38070b640 -1 Falling back to public interface 2023-03-03T08:43:12.664 INFO:tasks.ceph.osd.5.smithi143.stderr:2023-03-03T08:43:12.662+0000 7fd290651640 -1 Falling back to public interface 2023-03-03T08:43:12.664 INFO:tasks.ceph.osd.4.smithi143.stderr:2023-03-03T08:43:12.662+0000 7f4ddea94640 -1 Falling back to public interface 2023-03-03T08:43:12.664 INFO:tasks.ceph.osd.5.smithi143.stderr:2023-03-03T08:43:12.663+0000 7fd287a0f700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2023-03-03T08:43:12.665 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-03-03T08:43:12.663+0000 7fbf97ae5640 -1 Falling back to public interface 2023-03-03T08:43:12.671 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-03-03T08:43:12.670+0000 7f6af0166640 -1 Falling back to public interface 2023-03-03T08:43:12.824 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:12.822+0000 7fc3bfdb8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-03T08:43:12.901 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:12.899+0000 7fc3bfdb8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-03T08:43:13.041 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:13.039+0000 7fc3bfdb8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-03T08:43:13.149 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:13.234 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:13.233+0000 7f534e58c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-03T08:43:13.264 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.263+0000 7f2a676a6700 1 Processor -- start 2023-03-03T08:43:13.265 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.263+0000 7f2a676a6700 1 -- start start 2023-03-03T08:43:13.266 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a676a6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2a601553b0 0x7f2a600c1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:13.266 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a676a6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a60158c20 con 0x7f2a601580f0 2023-03-03T08:43:13.266 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a676a6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a601547a0 con 0x7f2a6006dd60 2023-03-03T08:43:13.266 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a676a6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a6006d310 con 0x7f2a601553b0 2023-03-03T08:43:13.267 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a65c43700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2a601580f0 0x7f2a60158510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44588/0 (socket says 172.21.15.7:44588) 2023-03-03T08:43:13.267 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a65c43700 1 -- 172.21.15.7:0/3006294612 learned_addr learned my addr 172.21.15.7:0/3006294612 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:13.267 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a65442700 1 --2- 172.21.15.7:0/3006294612 >> v2:172.21.15.143:3300/0 conn(0x7f2a601553b0 0x7f2a600c1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:13.267 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a5ffff700 1 -- 172.21.15.7:0/3006294612 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629462269 0 0) 0x7f2a60158c20 con 0x7f2a601580f0 2023-03-03T08:43:13.267 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a5ffff700 1 -- 172.21.15.7:0/3006294612 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a480035b0 con 0x7f2a601580f0 2023-03-03T08:43:13.267 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a65442700 1 -- 172.21.15.7:0/3006294612 >> v1:172.21.15.143:6789/0 conn(0x7f2a6006dd60 legacy=0x7f2a600c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:13.267 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.264+0000 7f2a5ffff700 1 -- 172.21.15.7:0/3006294612 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1017481772 0 0) 0x7f2a480035b0 con 0x7f2a601580f0 2023-03-03T08:43:13.268 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a65442700 1 -- 172.21.15.7:0/3006294612 >> v1:172.21.15.7:6789/0 conn(0x7f2a601580f0 legacy=0x7f2a60158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:13.268 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a65442700 1 -- 172.21.15.7:0/3006294612 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a600c1360 con 0x7f2a601553b0 2023-03-03T08:43:13.269 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a65442700 1 --2- 172.21.15.7:0/3006294612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a601553b0 0x7f2a600c1640 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009d90 tx=0x7f2a5000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=340278c4cc847098 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:13.269 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a5ffff700 1 -- 172.21.15.7:0/3006294612 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5000e070 con 0x7f2a601553b0 2023-03-03T08:43:13.269 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a5ffff700 1 -- 172.21.15.7:0/3006294612 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a50003a20 con 0x7f2a601553b0 2023-03-03T08:43:13.269 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a5ffff700 1 -- 172.21.15.7:0/3006294612 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a58001670 con 0x7f2a601553b0 2023-03-03T08:43:13.275 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a676a6700 1 -- 172.21.15.7:0/3006294612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a601553b0 msgr2=0x7f2a600c1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:13.275 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a676a6700 1 --2- 172.21.15.7:0/3006294612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a601553b0 0x7f2a600c1640 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009d90 tx=0x7f2a5000b330 comp rx=0 tx=0).stop 2023-03-03T08:43:13.275 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a676a6700 1 -- 172.21.15.7:0/3006294612 shutdown_connections 2023-03-03T08:43:13.276 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a676a6700 1 --2- 172.21.15.7:0/3006294612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a601553b0 0x7f2a600c1640 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:13.276 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.265+0000 7f2a676a6700 1 -- 172.21.15.7:0/3006294612 >> 172.21.15.7:0/3006294612 conn(0x7f2a600a9ec0 msgr2=0x7f2a600c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:13.276 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.266+0000 7f2a676a6700 1 -- 172.21.15.7:0/3006294612 shutdown_connections 2023-03-03T08:43:13.276 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.266+0000 7f2a676a6700 1 -- 172.21.15.7:0/3006294612 wait complete. 2023-03-03T08:43:13.277 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.267+0000 7f2a676a6700 1 Processor -- start 2023-03-03T08:43:13.277 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.267+0000 7f2a676a6700 1 -- start start 2023-03-03T08:43:13.277 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.267+0000 7f2a676a6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6006dd60 0x7f2a60156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:13.277 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.267+0000 7f2a676a6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a601553b0 0x7f2a6014e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:13.278 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.267+0000 7f2a676a6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a601580f0 0x7f2a6014eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:13.278 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.267+0000 7f2a676a6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60071420 con 0x7f2a601553b0 2023-03-03T08:43:13.278 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.267+0000 7f2a676a6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a600ac3a0 con 0x7f2a6006dd60 2023-03-03T08:43:13.278 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.267+0000 7f2a676a6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6009b170 con 0x7f2a601580f0 2023-03-03T08:43:13.279 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a64c41700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a601553b0 0x7f2a6014e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:13.279 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a64c41700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a601553b0 0x7f2a6014e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50614/0 (socket says 172.21.15.7:50614) 2023-03-03T08:43:13.279 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a64c41700 1 -- 172.21.15.7:0/3011291254 learned_addr learned my addr 172.21.15.7:0/3011291254 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:13.279 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a65442700 1 --2- 172.21.15.7:0/3011291254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6006dd60 0x7f2a60156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:13.280 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a65c43700 1 --2- 172.21.15.7:0/3011291254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a601580f0 0x7f2a6014eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:13.280 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a64c41700 1 -- 172.21.15.7:0/3011291254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a601580f0 msgr2=0x7f2a6014eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:13.280 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a64c41700 1 --2- 172.21.15.7:0/3011291254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a601580f0 0x7f2a6014eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:13.280 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a64c41700 1 -- 172.21.15.7:0/3011291254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6006dd60 msgr2=0x7f2a60156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:13.281 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a64c41700 1 --2- 172.21.15.7:0/3011291254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6006dd60 0x7f2a60156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:13.281 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a64c41700 1 -- 172.21.15.7:0/3011291254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a50009a70 con 0x7f2a601553b0 2023-03-03T08:43:13.281 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a65442700 1 --2- 172.21.15.7:0/3011291254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6006dd60 0x7f2a60156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:13.281 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a64c41700 1 --2- 172.21.15.7:0/3011291254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a601553b0 0x7f2a6014e580 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a5400c550 tx=0x7f2a5400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:13.281 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a5dffb700 1 -- 172.21.15.7:0/3011291254 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a540054e0 con 0x7f2a601553b0 2023-03-03T08:43:13.281 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a5dffb700 1 -- 172.21.15.7:0/3011291254 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a54003680 con 0x7f2a601553b0 2023-03-03T08:43:13.281 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a676a6700 1 -- 172.21.15.7:0/3011291254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a600c8340 con 0x7f2a601553b0 2023-03-03T08:43:13.282 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a5dffb700 1 -- 172.21.15.7:0/3011291254 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54003a30 con 0x7f2a601553b0 2023-03-03T08:43:13.282 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.268+0000 7f2a5dffb700 1 -- 172.21.15.7:0/3011291254 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f2a54019030 con 0x7f2a601553b0 2023-03-03T08:43:13.282 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.269+0000 7f2a676a6700 1 -- 172.21.15.7:0/3011291254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a60074310 con 0x7f2a601553b0 2023-03-03T08:43:13.282 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.269+0000 7f2a5dffb700 1 -- 172.21.15.7:0/3011291254 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2a54012400 con 0x7f2a601553b0 2023-03-03T08:43:13.282 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.269+0000 7f2a676a6700 1 -- 172.21.15.7:0/3011291254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a600c0c70 con 0x7f2a601553b0 2023-03-03T08:43:13.282 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.272+0000 7f2a5dffb700 1 -- 172.21.15.7:0/3011291254 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f2a54017070 con 0x7f2a601553b0 2023-03-03T08:43:13.319 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:13.318+0000 7f534e58c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-03T08:43:13.342 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:13.340+0000 7fc3bfdb8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-03T08:43:13.343 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.342+0000 7f2a676a6700 1 -- 172.21.15.7:0/3011291254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a60074bb0 con 0x7f2a601553b0 2023-03-03T08:43:13.344 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.343+0000 7f2a5dffb700 1 -- 172.21.15.7:0/3011291254 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2a54003be0 con 0x7f2a601553b0 2023-03-03T08:43:13.344 INFO:teuthology.misc.health.smithi007.stdout: 2023-03-03T08:43:13.344 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:12.002629+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:13.345 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.344+0000 7f2a676a6700 1 -- 172.21.15.7:0/3011291254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a601553b0 msgr2=0x7f2a6014e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:13.345 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.344+0000 7f2a676a6700 1 --2- 172.21.15.7:0/3011291254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a601553b0 0x7f2a6014e580 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a5400c550 tx=0x7f2a5400c860 comp rx=0 tx=0).stop 2023-03-03T08:43:13.345 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.344+0000 7f2a676a6700 1 -- 172.21.15.7:0/3011291254 shutdown_connections 2023-03-03T08:43:13.346 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.344+0000 7f2a676a6700 1 --2- 172.21.15.7:0/3011291254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a601580f0 0x7f2a6014eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:13.346 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.344+0000 7f2a676a6700 1 --2- 172.21.15.7:0/3011291254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6006dd60 0x7f2a60156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:13.346 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.344+0000 7f2a676a6700 1 --2- 172.21.15.7:0/3011291254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a601553b0 0x7f2a6014e580 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:13.346 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.344+0000 7f2a676a6700 1 -- 172.21.15.7:0/3011291254 >> 172.21.15.7:0/3011291254 conn(0x7f2a600a9ec0 msgr2=0x7f2a600c3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:13.346 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.344+0000 7f2a676a6700 1 -- 172.21.15.7:0/3011291254 shutdown_connections 2023-03-03T08:43:13.346 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:13.344+0000 7f2a676a6700 1 -- 172.21.15.7:0/3011291254 wait complete. 2023-03-03T08:43:13.355 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-03-03T08:43:13.465 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:13.464+0000 7f534e58c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-03T08:43:13.475 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:13.473+0000 7fc3bfdb8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-03T08:43:13.692 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:13.690+0000 7fc3bfdb8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-03T08:43:13.776 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:13.774+0000 7f534e58c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-03T08:43:13.848 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:13.847+0000 7fc3bfdb8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-03T08:43:13.915 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:13.913+0000 7f534e58c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-03T08:43:13.961 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:13.959+0000 7fc3bfdb8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-03T08:43:14.139 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:14.138+0000 7f534e58c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-03T08:43:14.284 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:14.282+0000 7fc3bfdb8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-03T08:43:14.301 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:14.300+0000 7f534e58c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-03T08:43:14.385 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:14.383+0000 7fc3bfdb8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-03T08:43:14.421 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:14.420+0000 7f534e58c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-03T08:43:14.568 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:14.566+0000 7fc3bfdb8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-03T08:43:14.760 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:14.759+0000 7f534e58c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-03T08:43:14.867 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:14.866+0000 7f534e58c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-03T08:43:15.058 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:15.056+0000 7f534e58c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-03T08:43:15.081 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:15.079+0000 7fc3bfdb8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-03T08:43:15.709 INFO:tasks.ceph.mgr.y.smithi007.stderr:2023-03-03T08:43:15.708+0000 7f534e58c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-03T08:43:16.107 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:16.105+0000 7fc377021700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-03-03T08:43:16.108 INFO:tasks.ceph.mgr.x.smithi143.stderr:2023-03-03T08:43:16.106+0000 7fc377021700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-03-03T08:43:16.170 INFO:tasks.ceph.osd.7.smithi143.stderr:2023-03-03T08:43:16.168+0000 7f93f13b1640 -1 osd.7 0 log_to_monitors true 2023-03-03T08:43:16.196 INFO:tasks.ceph.osd.6.smithi143.stderr:2023-03-03T08:43:16.194+0000 7f1e7c290640 -1 osd.6 0 log_to_monitors true 2023-03-03T08:43:16.197 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-03-03T08:43:16.195+0000 7f6af0166640 -1 osd.2 0 log_to_monitors true 2023-03-03T08:43:16.208 INFO:tasks.ceph.osd.4.smithi143.stderr:2023-03-03T08:43:16.206+0000 7f4ddea94640 -1 osd.4 0 log_to_monitors true 2023-03-03T08:43:16.208 INFO:tasks.ceph.osd.5.smithi143.stderr:2023-03-03T08:43:16.207+0000 7fd290651640 -1 osd.5 0 log_to_monitors true 2023-03-03T08:43:16.222 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-03-03T08:43:16.221+0000 7fbf97ae5640 -1 osd.1 0 log_to_monitors true 2023-03-03T08:43:16.223 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-03-03T08:43:16.222+0000 7f94f33ad640 -1 osd.0 0 log_to_monitors true 2023-03-03T08:43:16.224 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-03-03T08:43:16.223+0000 7fc38070b640 -1 osd.3 0 log_to_monitors true 2023-03-03T08:43:19.488 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-03-03T08:43:19.486+0000 7fbf8b69c700 -1 osd.1 0 waiting for initial osdmap 2023-03-03T08:43:19.490 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-03-03T08:43:19.489+0000 7f6ae3d1d700 -1 osd.2 0 waiting for initial osdmap 2023-03-03T08:43:19.491 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-03-03T08:43:19.490+0000 7f94e6f64700 -1 osd.0 0 waiting for initial osdmap 2023-03-03T08:43:19.493 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-03-03T08:43:19.492+0000 7fc3742c2700 -1 osd.3 0 waiting for initial osdmap 2023-03-03T08:43:19.540 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-03-03T08:43:19.539+0000 7f94df33f700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T08:43:19.541 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-03-03T08:43:19.539+0000 7f6adc0f8700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T08:43:19.557 INFO:tasks.ceph.osd.3.smithi007.stderr:2023-03-03T08:43:19.556+0000 7fc36c69d700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T08:43:19.561 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-03-03T08:43:19.560+0000 7fbf83a77700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T08:43:20.129 INFO:tasks.ceph.osd.5.smithi143.stderr:2023-03-03T08:43:20.128+0000 7fd284208700 -1 osd.5 0 waiting for initial osdmap 2023-03-03T08:43:20.139 INFO:tasks.ceph.osd.4.smithi143.stderr:2023-03-03T08:43:20.138+0000 7f4dd264b700 -1 osd.4 0 waiting for initial osdmap 2023-03-03T08:43:20.142 INFO:tasks.ceph.osd.5.smithi143.stderr:2023-03-03T08:43:20.141+0000 7fd27c5e3700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T08:43:20.145 INFO:tasks.ceph.osd.6.smithi143.stderr:2023-03-03T08:43:20.144+0000 7f1e6fe47700 -1 osd.6 0 waiting for initial osdmap 2023-03-03T08:43:20.148 INFO:tasks.ceph.osd.7.smithi143.stderr:2023-03-03T08:43:20.147+0000 7f93e4f68700 -1 osd.7 0 waiting for initial osdmap 2023-03-03T08:43:20.155 INFO:tasks.ceph.osd.4.smithi143.stderr:2023-03-03T08:43:20.154+0000 7f4dcaa26700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T08:43:20.159 INFO:tasks.ceph.osd.6.smithi143.stderr:2023-03-03T08:43:20.159+0000 7f1e68222700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T08:43:20.162 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:20.163 INFO:tasks.ceph.osd.7.smithi143.stderr:2023-03-03T08:43:20.163+0000 7f93dd343700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-03T08:43:20.283 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.281+0000 7f44ee826700 1 Processor -- start 2023-03-03T08:43:20.283 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.281+0000 7f44ee826700 1 -- start start 2023-03-03T08:43:20.286 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.282+0000 7f44ee826700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e806d150 0x7f44e80c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:20.286 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.282+0000 7f44ee826700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e80c6a60 con 0x7f44e806cd80 2023-03-03T08:43:20.286 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.282+0000 7f44ee826700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e80c6780 con 0x7f44e80ba360 2023-03-03T08:43:20.286 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.282+0000 7f44ee826700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80c2db0 con 0x7f44e806d150 2023-03-03T08:43:20.286 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.282+0000 7f44e7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f44e806cd80 0x7f44e80c1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44652/0 (socket says 172.21.15.7:44652) 2023-03-03T08:43:20.286 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.282+0000 7f44e7fff700 1 -- 172.21.15.7:0/3107453735 learned_addr learned my addr 172.21.15.7:0/3107453735 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:20.286 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.282+0000 7f44e77fe700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e806d150 0x7f44e80c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:20.287 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179855464 0 0) 0x7f44e80c6a60 con 0x7f44e806cd80 2023-03-03T08:43:20.287 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d40035b0 con 0x7f44e806cd80 2023-03-03T08:43:20.287 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719869655 0 0) 0x7f44e80c6780 con 0x7f44e80ba360 2023-03-03T08:43:20.287 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e80c6a60 con 0x7f44e80ba360 2023-03-03T08:43:20.287 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3873027319 0 0) 0x7f44d40035b0 con 0x7f44e806cd80 2023-03-03T08:43:20.287 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e806d150 msgr2=0x7f44e80c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:20.287 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e806d150 0x7f44e80c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:20.288 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 >> v1:172.21.15.143:6789/0 conn(0x7f44e80ba360 legacy=0x7f44e8155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:20.288 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e806fce0 con 0x7f44e806cd80 2023-03-03T08:43:20.288 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f44d8001670 con 0x7f44e806cd80 2023-03-03T08:43:20.288 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 >> v1:172.21.15.7:6789/0 conn(0x7f44e806cd80 legacy=0x7f44e80c1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:20.288 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44d4003bc0 0x7f44d4003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:20.289 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e806d150 0x7f44d4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:20.289 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d4004d90 0x7f44d4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:20.289 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d400bba0 con 0x7f44d4003bc0 2023-03-03T08:43:20.289 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d400bce0 con 0x7f44d4004d90 2023-03-03T08:43:20.289 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d400be50 con 0x7f44e806d150 2023-03-03T08:43:20.289 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e7fff700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44d4003bc0 0x7f44d4003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:20.289 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e77fe700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e806d150 0x7f44d4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:20.290 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44ecdc3700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d4004d90 0x7f44d4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:20.290 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e7fff700 1 -- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e806d150 msgr2=0x7f44d4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:20.290 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e7fff700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44e806d150 0x7f44d4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:20.290 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e7fff700 1 -- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d4004d90 msgr2=0x7f44d4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:20.290 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e7fff700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44d4004d90 0x7f44d4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:20.290 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e7fff700 1 -- 172.21.15.7:0/3107453735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d40035b0 con 0x7f44d4003bc0 2023-03-03T08:43:20.290 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.283+0000 7f44e7fff700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44d4003bc0 0x7f44d4003fb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f44d8001eb0 tx=0x7f44d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c13b04e50ead94c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:20.291 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.284+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d800cc70 con 0x7f44d4003bc0 2023-03-03T08:43:20.291 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.284+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d800cdf0 con 0x7f44d4003bc0 2023-03-03T08:43:20.291 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.284+0000 7f44e6ffd700 1 -- 172.21.15.7:0/3107453735 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8005ba0 con 0x7f44d4003bc0 2023-03-03T08:43:20.292 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.284+0000 7f44ee826700 1 -- 172.21.15.7:0/3107453735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44d4003bc0 msgr2=0x7f44d4003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:20.292 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.284+0000 7f44ee826700 1 --2- 172.21.15.7:0/3107453735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44d4003bc0 0x7f44d4003fb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f44d8001eb0 tx=0x7f44d4005880 comp rx=0 tx=0).stop 2023-03-03T08:43:20.292 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.284+0000 7f44ecdc3700 1 -- 172.21.15.7:0/3107453735 reap_dead start 2023-03-03T08:43:20.292 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.284+0000 7f44ee826700 1 -- 172.21.15.7:0/3107453735 shutdown_connections 2023-03-03T08:43:20.292 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.284+0000 7f44ee826700 1 -- 172.21.15.7:0/3107453735 >> 172.21.15.7:0/3107453735 conn(0x7f44e80a8860 msgr2=0x7f44e80a8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:20.292 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.284+0000 7f44ee826700 1 -- 172.21.15.7:0/3107453735 shutdown_connections 2023-03-03T08:43:20.292 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.284+0000 7f44ee826700 1 -- 172.21.15.7:0/3107453735 wait complete. 2023-03-03T08:43:20.294 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.285+0000 7f44ee826700 1 Processor -- start 2023-03-03T08:43:20.294 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44ee826700 1 -- start start 2023-03-03T08:43:20.295 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44ee826700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44d4003bc0 0x7f44e80a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:20.295 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44ee826700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e81ed420 0x7f44e80bedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:20.295 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44ee826700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e80bf2f0 0x7f44e80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:20.295 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44ee826700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e81eab30 con 0x7f44e81ed420 2023-03-03T08:43:20.295 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44ee826700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80bd240 con 0x7f44e80bf2f0 2023-03-03T08:43:20.296 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44ee826700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e81ea200 con 0x7f44d4003bc0 2023-03-03T08:43:20.296 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44e77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e81ed420 0x7f44e80bedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:20.296 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44e77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e81ed420 0x7f44e80bedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50686/0 (socket says 172.21.15.7:50686) 2023-03-03T08:43:20.296 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44e77fe700 1 -- 172.21.15.7:0/3427366347 learned_addr learned my addr 172.21.15.7:0/3427366347 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:20.296 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44ecdc3700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e80bf2f0 0x7f44e80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:20.296 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.286+0000 7f44e7fff700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44d4003bc0 0x7f44e80a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:20.296 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44e77fe700 1 -- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44d4003bc0 msgr2=0x7f44e80a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:20.296 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44e77fe700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44d4003bc0 0x7f44e80a63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:20.297 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44e77fe700 1 -- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e80bf2f0 msgr2=0x7f44e80a4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:20.297 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44e77fe700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e80bf2f0 0x7f44e80a4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:20.297 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44e77fe700 1 -- 172.21.15.7:0/3427366347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e8072650 con 0x7f44e81ed420 2023-03-03T08:43:20.297 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44e77fe700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e81ed420 0x7f44e80bedb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f44d0011aa0 tx=0x7f44d0011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:20.297 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44e4ff9700 1 -- 172.21.15.7:0/3427366347 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0011ee0 con 0x7f44e81ed420 2023-03-03T08:43:20.297 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44e4ff9700 1 -- 172.21.15.7:0/3427366347 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d000f940 con 0x7f44e81ed420 2023-03-03T08:43:20.297 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44ee826700 1 -- 172.21.15.7:0/3427366347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e80c10c0 con 0x7f44e81ed420 2023-03-03T08:43:20.298 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44e4ff9700 1 -- 172.21.15.7:0/3427366347 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d000fcf0 con 0x7f44e81ed420 2023-03-03T08:43:20.298 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.287+0000 7f44ee826700 1 -- 172.21.15.7:0/3427366347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e806d8e0 con 0x7f44e81ed420 2023-03-03T08:43:20.298 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.288+0000 7f44ee826700 1 -- 172.21.15.7:0/3427366347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e80a93e0 con 0x7f44e81ed420 2023-03-03T08:43:20.298 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.288+0000 7f44e4ff9700 1 -- 172.21.15.7:0/3427366347 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 86375+0+0 (secure 0 0 0) 0x7f44d0014440 con 0x7f44e81ed420 2023-03-03T08:43:20.298 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.289+0000 7f44e4ff9700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44c80682a0 0x7f44c806a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:20.299 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.289+0000 7f44e4ff9700 1 -- 172.21.15.7:0/3427366347 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f44d0085520 con 0x7f44e81ed420 2023-03-03T08:43:20.299 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.289+0000 7f44e7fff700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44c80682a0 0x7f44c806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:20.299 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.289+0000 7f44e7fff700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44c80682a0 0x7f44c806a750 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f44d8000c00 tx=0x7f44d8007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:20.299 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.293+0000 7f44e4ff9700 1 -- 172.21.15.7:0/3427366347 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f44d0052d20 con 0x7f44e81ed420 2023-03-03T08:43:20.489 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.488+0000 7f44ee826700 1 -- 172.21.15.7:0/3427366347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44e8074bb0 con 0x7f44e81ed420 2023-03-03T08:43:20.489 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.488+0000 7f44e4ff9700 1 -- 172.21.15.7:0/3427366347 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f44d0056340 con 0x7f44e81ed420 2023-03-03T08:43:20.490 INFO:teuthology.misc.health.smithi007.stdout: 2023-03-03T08:43:20.490 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":12,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:18.101447+0000","last_up_change":"0.000000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:20.491 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 -- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44c80682a0 msgr2=0x7f44c806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:20.492 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44c80682a0 0x7f44c806a750 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f44d8000c00 tx=0x7f44d8007000 comp rx=0 tx=0).stop 2023-03-03T08:43:20.492 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 -- 172.21.15.7:0/3427366347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e81ed420 msgr2=0x7f44e80bedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:20.492 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e81ed420 0x7f44e80bedb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f44d0011aa0 tx=0x7f44d0011db0 comp rx=0 tx=0).stop 2023-03-03T08:43:20.492 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 -- 172.21.15.7:0/3427366347 shutdown_connections 2023-03-03T08:43:20.492 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44e80bf2f0 0x7f44e80a4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:20.493 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44d4003bc0 0x7f44e80a63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:20.493 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44c80682a0 0x7f44c806a750 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:20.493 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 --2- 172.21.15.7:0/3427366347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e81ed420 0x7f44e80bedb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:20.493 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 -- 172.21.15.7:0/3427366347 >> 172.21.15.7:0/3427366347 conn(0x7f44e80a8860 msgr2=0x7f44e81ddba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:20.493 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 -- 172.21.15.7:0/3427366347 shutdown_connections 2023-03-03T08:43:20.493 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:20.490+0000 7f44ee826700 1 -- 172.21.15.7:0/3427366347 wait complete. 2023-03-03T08:43:20.503 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-03-03T08:43:27.309 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:27.436 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.434+0000 7fba88774700 1 Processor -- start 2023-03-03T08:43:27.438 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.435+0000 7fba88774700 1 -- start start 2023-03-03T08:43:27.440 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.435+0000 7fba88774700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba80071410 0x7fba800be6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.440 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.435+0000 7fba88774700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fba8014fe10 0x7fba80152440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.440 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.435+0000 7fba88774700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8006e250 con 0x7fba800717e0 2023-03-03T08:43:27.440 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.435+0000 7fba88774700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba80152980 con 0x7fba8014fe10 2023-03-03T08:43:27.440 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.435+0000 7fba88774700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba80153430 con 0x7fba80071410 2023-03-03T08:43:27.441 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.436+0000 7fba86d11700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fba8014fe10 0x7fba80152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.441 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.436+0000 7fba86510700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba80071410 0x7fba800be6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.441 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.436+0000 7fba86d11700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fba8014fe10 0x7fba80152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:46644/0 (socket says 172.21.15.7:46644) 2023-03-03T08:43:27.441 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.436+0000 7fba86d11700 1 -- 172.21.15.7:0/3784250512 learned_addr learned my addr 172.21.15.7:0/3784250512 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:27.441 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.436+0000 7fba86510700 1 -- 172.21.15.7:0/3784250512 >> v2:172.21.15.143:3300/0 conn(0x7fba8014fe10 msgr2=0x7fba80152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:27.441 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.436+0000 7fba86510700 1 --2- 172.21.15.7:0/3784250512 >> v2:172.21.15.143:3300/0 conn(0x7fba8014fe10 0x7fba80152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.441 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.436+0000 7fba86510700 1 -- 172.21.15.7:0/3784250512 >> v1:172.21.15.143:6789/0 conn(0x7fba800717e0 legacy=0x7fba8014f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:27.442 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.436+0000 7fba86510700 1 -- 172.21.15.7:0/3784250512 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba80071c30 con 0x7fba80071410 2023-03-03T08:43:27.442 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.436+0000 7fba86d11700 1 --2- 172.21.15.7:0/3784250512 >> v2:172.21.15.143:3300/0 conn(0x7fba8014fe10 0x7fba80152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:27.442 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.436+0000 7fba86510700 1 --2- 172.21.15.7:0/3784250512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba80071410 0x7fba800be6f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fba74009840 tx=0x7fba74009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3624e8bb09c3b1e4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:27.442 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba8550e700 1 -- 172.21.15.7:0/3784250512 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74011830 con 0x7fba80071410 2023-03-03T08:43:27.442 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba8550e700 1 -- 172.21.15.7:0/3784250512 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba74011990 con 0x7fba80071410 2023-03-03T08:43:27.443 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba8550e700 1 -- 172.21.15.7:0/3784250512 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74011d20 con 0x7fba80071410 2023-03-03T08:43:27.443 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba88774700 1 -- 172.21.15.7:0/3784250512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba80071410 msgr2=0x7fba800be6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:27.443 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba88774700 1 --2- 172.21.15.7:0/3784250512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba80071410 0x7fba800be6f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fba74009840 tx=0x7fba74009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:27.443 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba88774700 1 -- 172.21.15.7:0/3784250512 shutdown_connections 2023-03-03T08:43:27.443 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba88774700 1 --2- 172.21.15.7:0/3784250512 >> v2:172.21.15.143:3300/0 conn(0x7fba8014fe10 0x7fba80152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.443 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba88774700 1 --2- 172.21.15.7:0/3784250512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba80071410 0x7fba800be6f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.444 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba88774700 1 -- 172.21.15.7:0/3784250512 >> 172.21.15.7:0/3784250512 conn(0x7fba800c5120 msgr2=0x7fba800c5520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:27.444 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba88774700 1 -- 172.21.15.7:0/3784250512 shutdown_connections 2023-03-03T08:43:27.444 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.437+0000 7fba88774700 1 -- 172.21.15.7:0/3784250512 wait complete. 2023-03-03T08:43:27.444 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.438+0000 7fba88774700 1 Processor -- start 2023-03-03T08:43:27.445 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.439+0000 7fba88774700 1 -- start start 2023-03-03T08:43:27.447 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.439+0000 7fba88774700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba80071410 0x7fba800a7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.447 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.439+0000 7fba88774700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba800717e0 0x7fba8006b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.447 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.439+0000 7fba88774700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8014fe10 0x7fba800c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.447 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.439+0000 7fba88774700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba801ea2b0 con 0x7fba8014fe10 2023-03-03T08:43:27.447 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.439+0000 7fba88774700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba800c21d0 con 0x7fba80071410 2023-03-03T08:43:27.447 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.439+0000 7fba88774700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba801ea6d0 con 0x7fba800717e0 2023-03-03T08:43:27.448 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.439+0000 7fba86d11700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8014fe10 0x7fba800c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.448 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.439+0000 7fba86d11700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8014fe10 0x7fba800c1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50876/0 (socket says 172.21.15.7:50876) 2023-03-03T08:43:27.448 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.439+0000 7fba86d11700 1 -- 172.21.15.7:0/637360913 learned_addr learned my addr 172.21.15.7:0/637360913 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:27.448 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba86510700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba80071410 0x7fba800a7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.448 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba85d0f700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba800717e0 0x7fba8006b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.448 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba86d11700 1 -- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba800717e0 msgr2=0x7fba8006b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:27.448 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba86d11700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba800717e0 0x7fba8006b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.449 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba86d11700 1 -- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba80071410 msgr2=0x7fba800a7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:27.449 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba86d11700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba80071410 0x7fba800a7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.449 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba86d11700 1 -- 172.21.15.7:0/637360913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba78009b30 con 0x7fba8014fe10 2023-03-03T08:43:27.449 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba86d11700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8014fe10 0x7fba800c1120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fba7800efc0 tx=0x7fba7800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:27.449 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba72ffd700 1 -- 172.21.15.7:0/637360913 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78018070 con 0x7fba8014fe10 2023-03-03T08:43:27.449 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba72ffd700 1 -- 172.21.15.7:0/637360913 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7800f040 con 0x7fba8014fe10 2023-03-03T08:43:27.450 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba88774700 1 -- 172.21.15.7:0/637360913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba740094a0 con 0x7fba8014fe10 2023-03-03T08:43:27.450 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba72ffd700 1 -- 172.21.15.7:0/637360913 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78003c20 con 0x7fba8014fe10 2023-03-03T08:43:27.450 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.440+0000 7fba88774700 1 -- 172.21.15.7:0/637360913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba80152e70 con 0x7fba8014fe10 2023-03-03T08:43:27.450 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.441+0000 7fba88774700 1 -- 172.21.15.7:0/637360913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba80152b90 con 0x7fba8014fe10 2023-03-03T08:43:27.450 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.442+0000 7fba72ffd700 1 -- 172.21.15.7:0/637360913 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fba7800c930 con 0x7fba8014fe10 2023-03-03T08:43:27.450 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.442+0000 7fba72ffd700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba640682f0 0x7fba6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.450 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.442+0000 7fba72ffd700 1 -- 172.21.15.7:0/637360913 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fba78085b30 con 0x7fba8014fe10 2023-03-03T08:43:27.451 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.442+0000 7fba86510700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba640682f0 0x7fba6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.451 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.443+0000 7fba86510700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba640682f0 0x7fba6406a7a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba74000c00 tx=0x7fba7401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:27.451 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.447+0000 7fba72ffd700 1 -- 172.21.15.7:0/637360913 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fba78052840 con 0x7fba8014fe10 2023-03-03T08:43:27.643 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.642+0000 7fba88774700 1 -- 172.21.15.7:0/637360913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba80153150 con 0x7fba8014fe10 2023-03-03T08:43:27.644 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.643+0000 7fba72ffd700 1 -- 172.21.15.7:0/637360913 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12334 (secure 0 0 0) 0x7fba78055e60 con 0x7fba8014fe10 2023-03-03T08:43:27.644 INFO:teuthology.misc.health.smithi007.stdout: 2023-03-03T08:43:27.644 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":16,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:23.546605+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:27.646 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 -- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba640682f0 msgr2=0x7fba6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:27.646 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba640682f0 0x7fba6406a7a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fba74000c00 tx=0x7fba7401c040 comp rx=0 tx=0).stop 2023-03-03T08:43:27.646 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 -- 172.21.15.7:0/637360913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8014fe10 msgr2=0x7fba800c1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:27.647 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8014fe10 0x7fba800c1120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fba7800efc0 tx=0x7fba7800c3a0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.647 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 -- 172.21.15.7:0/637360913 shutdown_connections 2023-03-03T08:43:27.647 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba640682f0 0x7fba6406a7a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.647 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8014fe10 0x7fba800c1120 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.647 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba80071410 0x7fba800a7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.647 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 --2- 172.21.15.7:0/637360913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba800717e0 0x7fba8006b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.647 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 -- 172.21.15.7:0/637360913 >> 172.21.15.7:0/637360913 conn(0x7fba800c5120 msgr2=0x7fba80150c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:27.648 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 -- 172.21.15.7:0/637360913 shutdown_connections 2023-03-03T08:43:27.648 INFO:teuthology.misc.health.smithi007.stderr:2023-03-03T08:43:27.645+0000 7fba88774700 1 -- 172.21.15.7:0/637360913 wait complete. 2023-03-03T08:43:27.657 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-03-03T08:43:27.658 INFO:tasks.ceph:Creating RBD pool 2023-03-03T08:43:27.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool create rbd 8 2023-03-03T08:43:27.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.780+0000 7ff9dc9c9700 1 Processor -- start 2023-03-03T08:43:27.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.780+0000 7ff9dc9c9700 1 -- start start 2023-03-03T08:43:27.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.781+0000 7ff9dc9c9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 0x7ff9d8151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.781+0000 7ff9dc9c9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff9d8072250 0x7ff9d814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.781+0000 7ff9dc9c9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9d80c1ce0 0x7ff9d814e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.781+0000 7ff9dc9c9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d81510d0 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.781+0000 7ff9dc9c9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d80c3740 con 0x7ff9d8072250 2023-03-03T08:43:27.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.781+0000 7ff9dc9c9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d8070670 con 0x7ff9d8071e80 2023-03-03T08:43:27.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.781+0000 7ff9d6d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9d80c1ce0 0x7ff9d814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.781+0000 7ff9d6d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9d80c1ce0 0x7ff9d814e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50884/0 (socket says 172.21.15.7:50884) 2023-03-03T08:43:27.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.781+0000 7ff9d6d9d700 1 -- 172.21.15.7:0/3402648112 learned_addr learned my addr 172.21.15.7:0/3402648112 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:27.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d5d9b700 1 --2- 172.21.15.7:0/3402648112 >> v2:172.21.15.143:3300/0 conn(0x7ff9d8072250 0x7ff9d814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d659c700 1 --2- 172.21.15.7:0/3402648112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 0x7ff9d8151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d6d9d700 1 -- 172.21.15.7:0/3402648112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 msgr2=0x7ff9d8151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:27.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d6d9d700 1 --2- 172.21.15.7:0/3402648112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 0x7ff9d8151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d6d9d700 1 -- 172.21.15.7:0/3402648112 >> v2:172.21.15.143:3300/0 conn(0x7ff9d8072250 msgr2=0x7ff9d814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:27.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d6d9d700 1 --2- 172.21.15.7:0/3402648112 >> v2:172.21.15.143:3300/0 conn(0x7ff9d8072250 0x7ff9d814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d6d9d700 1 -- 172.21.15.7:0/3402648112 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d80c39e0 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d659c700 1 --2- 172.21.15.7:0/3402648112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 0x7ff9d8151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:27.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d6d9d700 1 --2- 172.21.15.7:0/3402648112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 0x7ff9d814e730 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80c4e40 tx=0x7ff9d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72c4d3cde57f7d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:27.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d559a700 1 -- 172.21.15.7:0/3402648112 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d000e070 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d559a700 1 -- 172.21.15.7:0/3402648112 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d000b910 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.782+0000 7ff9d559a700 1 -- 172.21.15.7:0/3402648112 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d000bcc0 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.783+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/3402648112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 msgr2=0x7ff9d814e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:27.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.783+0000 7ff9dc9c9700 1 --2- 172.21.15.7:0/3402648112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 0x7ff9d814e730 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80c4e40 tx=0x7ff9d000b330 comp rx=0 tx=0).stop 2023-03-03T08:43:27.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.783+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/3402648112 shutdown_connections 2023-03-03T08:43:27.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.783+0000 7ff9dc9c9700 1 --2- 172.21.15.7:0/3402648112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 0x7ff9d814e730 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.783+0000 7ff9dc9c9700 1 --2- 172.21.15.7:0/3402648112 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 0x7ff9d8151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.783+0000 7ff9dc9c9700 1 --2- 172.21.15.7:0/3402648112 >> v2:172.21.15.143:3300/0 conn(0x7ff9d8072250 0x7ff9d814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.783+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/3402648112 >> 172.21.15.7:0/3402648112 conn(0x7ff9d80a94d0 msgr2=0x7ff9d80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:27.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.783+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/3402648112 shutdown_connections 2023-03-03T08:43:27.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.783+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/3402648112 wait complete. 2023-03-03T08:43:27.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.784+0000 7ff9dc9c9700 1 Processor -- start 2023-03-03T08:43:27.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.784+0000 7ff9dc9c9700 1 -- start start 2023-03-03T08:43:27.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9dc9c9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 0x7ff9d814ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9dc9c9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9d8072250 0x7ff9d81dcc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9dc9c9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 0x7ff9d814c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9dc9c9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d81de1f0 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9dc9c9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d814bc80 con 0x7ff9d8072250 2023-03-03T08:43:27.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9dc9c9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d80c6cc0 con 0x7ff9d8071e80 2023-03-03T08:43:27.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 0x7ff9d814c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 0x7ff9d814c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50890/0 (socket says 172.21.15.7:50890) 2023-03-03T08:43:27.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d6d9d700 1 -- 172.21.15.7:0/1492166681 learned_addr learned my addr 172.21.15.7:0/1492166681 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:27.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d659c700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 0x7ff9d814ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d5d9b700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9d8072250 0x7ff9d81dcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d6d9d700 1 -- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 msgr2=0x7ff9d814ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:27.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d6d9d700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 0x7ff9d814ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d6d9d700 1 -- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9d8072250 msgr2=0x7ff9d81dcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:27.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d6d9d700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9d8072250 0x7ff9d81dcc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:27.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d6d9d700 1 -- 172.21.15.7:0/1492166681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d0009b30 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.785+0000 7ff9d6d9d700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 0x7ff9d814c110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0012000 tx=0x7ff9d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:27.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.786+0000 7ff9c6ffd700 1 -- 172.21.15.7:0/1492166681 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0014a10 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.786+0000 7ff9c6ffd700 1 -- 172.21.15.7:0/1492166681 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d0014b90 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.786+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/1492166681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d806c9a0 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.786+0000 7ff9c6ffd700 1 -- 172.21.15.7:0/1492166681 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d00099b0 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.786+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/1492166681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d8151300 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.787+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/1492166681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d806cb30 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.787+0000 7ff9c6ffd700 1 -- 172.21.15.7:0/1492166681 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff9d0004040 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.788+0000 7ff9c6ffd700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9b80682f0 0x7ff9b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:27.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.788+0000 7ff9c6ffd700 1 -- 172.21.15.7:0/1492166681 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff9d0089900 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.788+0000 7ff9d659c700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9b80682f0 0x7ff9b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:27.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.788+0000 7ff9d659c700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9b80682f0 0x7ff9b806a7a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8155780 tx=0x7ff9cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:27.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.792+0000 7ff9c6ffd700 1 -- 172.21.15.7:0/1492166681 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff9d0056610 con 0x7ff9d80c1ce0 2023-03-03T08:43:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:27.987+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/1492166681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7ff9d8074bb0 con 0x7ff9d80c1ce0 2023-03-03T08:43:28.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.065+0000 7ff9c6ffd700 1 -- 172.21.15.7:0/1492166681 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7ff9d0020020 con 0x7ff9d80c1ce0 2023-03-03T08:43:28.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9b80682f0 msgr2=0x7ff9b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:28.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9b80682f0 0x7ff9b806a7a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8155780 tx=0x7ff9cc008040 comp rx=0 tx=0).stop 2023-03-03T08:43:28.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/1492166681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 msgr2=0x7ff9d814c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:28.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 0x7ff9d814c110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0012000 tx=0x7ff9d0004ad0 comp rx=0 tx=0).stop 2023-03-03T08:43:28.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/1492166681 shutdown_connections 2023-03-03T08:43:28.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9b80682f0 0x7ff9b806a7a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:28.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9d80c1ce0 0x7ff9d814c110 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:28.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9d8071e80 0x7ff9d814ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:28.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 --2- 172.21.15.7:0/1492166681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9d8072250 0x7ff9d81dcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:28.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/1492166681 >> 172.21.15.7:0/1492166681 conn(0x7ff9d80a94d0 msgr2=0x7ff9d806a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:28.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/1492166681 shutdown_connections 2023-03-03T08:43:28.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.067+0000 7ff9dc9c9700 1 -- 172.21.15.7:0/1492166681 wait complete. 2023-03-03T08:43:28.071 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'rbd' created 2023-03-03T08:43:28.081 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-03-03T08:43:28.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.205+0000 7f3d225a0700 1 Processor -- start 2023-03-03T08:43:28.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.205+0000 7f3d225a0700 1 -- start start 2023-03-03T08:43:28.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d225a0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3d1c14de50 0x7f3d1c06a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:28.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d225a0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d1c0c65c0 0x7f3d1c0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:28.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d225a0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c0a65b0 con 0x7f3d1c06a7f0 2023-03-03T08:43:28.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d225a0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1c154e00 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d225a0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1c0c3370 con 0x7f3d1c14de50 2023-03-03T08:43:28.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d1b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d1c0c65c0 0x7f3d1c0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:28.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d20b3d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3d1c06a7f0 0x7f3d1c0c2c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44878/0 (socket says 172.21.15.7:44878) 2023-03-03T08:43:28.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d20b3d700 1 -- 172.21.15.7:0/1664064563 learned_addr learned my addr 172.21.15.7:0/1664064563 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:28.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d1bfff700 1 --2- 172.21.15.7:0/1664064563 >> v2:172.21.15.143:3300/0 conn(0x7f3d1c14de50 0x7f3d1c06a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:28.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d1b7fe700 1 -- 172.21.15.7:0/1664064563 >> v2:172.21.15.143:3300/0 conn(0x7f3d1c14de50 msgr2=0x7f3d1c06a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:28.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d1b7fe700 1 --2- 172.21.15.7:0/1664064563 >> v2:172.21.15.143:3300/0 conn(0x7f3d1c14de50 0x7f3d1c06a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:28.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d1b7fe700 1 -- 172.21.15.7:0/1664064563 >> v1:172.21.15.7:6789/0 conn(0x7f3d1c06a7f0 legacy=0x7f3d1c0c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:28.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d1b7fe700 1 -- 172.21.15.7:0/1664064563 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c1541f0 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d1bfff700 1 --2- 172.21.15.7:0/1664064563 >> v2:172.21.15.143:3300/0 conn(0x7f3d1c14de50 0x7f3d1c06a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:28.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d1b7fe700 1 --2- 172.21.15.7:0/1664064563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 0x7f3d1c0c2720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0c7760 tx=0x7f3d0c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e445654f7eeab3c4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:28.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.206+0000 7f3d1affd700 1 -- 172.21.15.7:0/1664064563 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c011830 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.207+0000 7f3d1affd700 1 -- 172.21.15.7:0/1664064563 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0c011990 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.207+0000 7f3d1affd700 1 -- 172.21.15.7:0/1664064563 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c011d20 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.207+0000 7f3d225a0700 1 -- 172.21.15.7:0/1664064563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 msgr2=0x7f3d1c0c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:28.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.207+0000 7f3d225a0700 1 --2- 172.21.15.7:0/1664064563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 0x7f3d1c0c2720 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0c7760 tx=0x7f3d0c009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:28.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.207+0000 7f3d225a0700 1 -- 172.21.15.7:0/1664064563 shutdown_connections 2023-03-03T08:43:28.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.207+0000 7f3d225a0700 1 --2- 172.21.15.7:0/1664064563 >> v2:172.21.15.143:3300/0 conn(0x7f3d1c14de50 0x7f3d1c06a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:28.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.207+0000 7f3d225a0700 1 --2- 172.21.15.7:0/1664064563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 0x7f3d1c0c2720 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:28.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.207+0000 7f3d225a0700 1 -- 172.21.15.7:0/1664064563 >> 172.21.15.7:0/1664064563 conn(0x7f3d1c0a9430 msgr2=0x7f3d1c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:28.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.207+0000 7f3d225a0700 1 -- 172.21.15.7:0/1664064563 shutdown_connections 2023-03-03T08:43:28.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.207+0000 7f3d225a0700 1 -- 172.21.15.7:0/1664064563 wait complete. 2023-03-03T08:43:28.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.208+0000 7f3d225a0700 1 Processor -- start 2023-03-03T08:43:28.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.208+0000 7f3d225a0700 1 -- start start 2023-03-03T08:43:28.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d225a0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d1c06a7f0 0x7f3d1c0c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d225a0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 0x7f3d1c1522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d225a0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d1c14de50 0x7f3d1c1527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d225a0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c1e8680 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d225a0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c1e8bb0 con 0x7f3d1c14de50 2023-03-03T08:43:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d225a0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c151e20 con 0x7f3d1c06a7f0 2023-03-03T08:43:28.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d1b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 0x7f3d1c1522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d1b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 0x7f3d1c1522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50902/0 (socket says 172.21.15.7:50902) 2023-03-03T08:43:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d1b7fe700 1 -- 172.21.15.7:0/26062304 learned_addr learned my addr 172.21.15.7:0/26062304 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d20b3d700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d1c14de50 0x7f3d1c1527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.209+0000 7f3d1bfff700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d1c06a7f0 0x7f3d1c0c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d1b7fe700 1 -- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d1c06a7f0 msgr2=0x7f3d1c0c3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d1b7fe700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d1c06a7f0 0x7f3d1c0c3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:28.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d1b7fe700 1 -- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d1c14de50 msgr2=0x7f3d1c1527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d1b7fe700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d1c14de50 0x7f3d1c1527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d1b7fe700 1 -- 172.21.15.7:0/26062304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c0094a0 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d1b7fe700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 0x7f3d1c1522b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0090b0 tx=0x7f3d0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d18ff9700 1 -- 172.21.15.7:0/26062304 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c0036a0 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d18ff9700 1 -- 172.21.15.7:0/26062304 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0c01f070 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d225a0700 1 -- 172.21.15.7:0/26062304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1c1547b0 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d18ff9700 1 -- 172.21.15.7:0/26062304 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c003a40 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.210+0000 7f3d225a0700 1 -- 172.21.15.7:0/26062304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c1544d0 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.211+0000 7f3d225a0700 1 -- 172.21.15.7:0/26062304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1c154a90 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.211+0000 7f3d18ff9700 1 -- 172.21.15.7:0/26062304 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d0c00f050 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.212+0000 7f3d18ff9700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cfc0682f0 0x7f3cfc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:28.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.212+0000 7f3d18ff9700 1 -- 172.21.15.7:0/26062304 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f3d0c021070 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.212+0000 7f3d1bfff700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cfc0682f0 0x7f3cfc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:28.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.213+0000 7f3d1bfff700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cfc0682f0 0x7f3cfc06a7a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3d04006fd0 tx=0x7f3d04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:28.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.217+0000 7f3d18ff9700 1 -- 172.21.15.7:0/26062304 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d0c053d80 con 0x7f3d1c0c65c0 2023-03-03T08:43:28.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:28.413+0000 7f3d225a0700 1 -- 172.21.15.7:0/26062304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3d1c071df0 con 0x7f3d1c0c65c0 2023-03-03T08:43:29.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.068+0000 7f3d18ff9700 1 -- 172.21.15.7:0/26062304 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3d0c0150c0 con 0x7f3d1c0c65c0 2023-03-03T08:43:29.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.071+0000 7f3d225a0700 1 -- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cfc0682f0 msgr2=0x7f3cfc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.071+0000 7f3d225a0700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cfc0682f0 0x7f3cfc06a7a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3d04006fd0 tx=0x7f3d04008040 comp rx=0 tx=0).stop 2023-03-03T08:43:29.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.071+0000 7f3d225a0700 1 -- 172.21.15.7:0/26062304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 msgr2=0x7f3d1c1522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.071+0000 7f3d225a0700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 0x7f3d1c1522b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0090b0 tx=0x7f3d0c004750 comp rx=0 tx=0).stop 2023-03-03T08:43:29.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.071+0000 7f3d225a0700 1 -- 172.21.15.7:0/26062304 shutdown_connections 2023-03-03T08:43:29.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.071+0000 7f3d225a0700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d1c14de50 0x7f3d1c1527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.071+0000 7f3d225a0700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d1c06a7f0 0x7f3d1c0c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.071+0000 7f3d225a0700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cfc0682f0 0x7f3cfc06a7a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.071+0000 7f3d225a0700 1 --2- 172.21.15.7:0/26062304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d1c0c65c0 0x7f3d1c1522b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.071+0000 7f3d225a0700 1 -- 172.21.15.7:0/26062304 >> 172.21.15.7:0/26062304 conn(0x7f3d1c0a9430 msgr2=0x7f3d1c06d890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:29.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.072+0000 7f3d225a0700 1 -- 172.21.15.7:0/26062304 shutdown_connections 2023-03-03T08:43:29.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.072+0000 7f3d225a0700 1 -- 172.21.15.7:0/26062304 wait complete. 2023-03-03T08:43:29.075 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rbd' on pool 'rbd' 2023-03-03T08:43:29.088 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-03-03T08:43:29.089 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-03T08:43:29.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-03T08:43:29.091 INFO:tasks.ceph.mds.a:Started 2023-03-03T08:43:29.092 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T08:43:29.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T08:43:29.094 INFO:tasks.ceph.mds.c:Started 2023-03-03T08:43:29.094 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-03T08:43:29.094 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-03T08:43:29.096 INFO:tasks.ceph.mds.b:Started 2023-03-03T08:43:29.097 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T08:43:29.097 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T08:43:29.099 INFO:tasks.ceph.mds.d:Started 2023-03-03T08:43:29.099 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-03-03T08:43:29.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:29.183 INFO:tasks.ceph.mds.c.smithi007.stdout:starting mds.c at 2023-03-03T08:43:29.184 INFO:tasks.ceph.mds.a.smithi007.stdout:starting mds.a at 2023-03-03T08:43:29.185 INFO:tasks.ceph.mds.b.smithi143.stdout:starting mds.b at 2023-03-03T08:43:29.186 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-03-03T08:43:29.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.240+0000 7f8e614b2700 1 Processor -- start 2023-03-03T08:43:29.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.240+0000 7f8e614b2700 1 -- start start 2023-03-03T08:43:29.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.241+0000 7f8e614b2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e5c0c65c0 0x7f8e5c0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.241+0000 7f8e614b2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e5c0a9af0 0x7f8e5c0c2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.241+0000 7f8e614b2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c0a65b0 con 0x7f8e5c14de50 2023-03-03T08:43:29.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.241+0000 7f8e614b2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5c154e00 con 0x7f8e5c0a9af0 2023-03-03T08:43:29.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.241+0000 7f8e614b2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5c1541f0 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.241+0000 7f8e5b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e5c0a9af0 0x7f8e5c0c2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.241+0000 7f8e5b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e5c0a9af0 0x7f8e5c0c2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50938/0 (socket says 172.21.15.7:50938) 2023-03-03T08:43:29.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.241+0000 7f8e5b7fe700 1 -- 172.21.15.7:0/847839878 learned_addr learned my addr 172.21.15.7:0/847839878 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:29.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.241+0000 7f8e5a7fc700 1 --2- 172.21.15.7:0/847839878 >> v2:172.21.15.143:3300/0 conn(0x7f8e5c0c65c0 0x7f8e5c0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e5b7fe700 1 -- 172.21.15.7:0/847839878 >> v2:172.21.15.143:3300/0 conn(0x7f8e5c0c65c0 msgr2=0x7f8e5c0c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:29.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e5b7fe700 1 --2- 172.21.15.7:0/847839878 >> v2:172.21.15.143:3300/0 conn(0x7f8e5c0c65c0 0x7f8e5c0c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e5b7fe700 1 -- 172.21.15.7:0/847839878 >> v1:172.21.15.143:6789/0 conn(0x7f8e5c14de50 legacy=0x7f8e5c06a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e5b7fe700 1 -- 172.21.15.7:0/847839878 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c1544d0 con 0x7f8e5c0a9af0 2023-03-03T08:43:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e5a7fc700 1 --2- 172.21.15.7:0/847839878 >> v2:172.21.15.143:3300/0 conn(0x7f8e5c0c65c0 0x7f8e5c0c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e5b7fe700 1 --2- 172.21.15.7:0/847839878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0a9af0 0x7f8e5c0c2c60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c06f820 tx=0x7f8e5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629858f875924d10 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e59ffb700 1 -- 172.21.15.7:0/847839878 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5400e070 con 0x7f8e5c0a9af0 2023-03-03T08:43:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e59ffb700 1 -- 172.21.15.7:0/847839878 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5400b9c0 con 0x7f8e5c0a9af0 2023-03-03T08:43:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e59ffb700 1 -- 172.21.15.7:0/847839878 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5400bd70 con 0x7f8e5c0a9af0 2023-03-03T08:43:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e614b2700 1 -- 172.21.15.7:0/847839878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0a9af0 msgr2=0x7f8e5c0c2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.242+0000 7f8e614b2700 1 --2- 172.21.15.7:0/847839878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0a9af0 0x7f8e5c0c2c60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c06f820 tx=0x7f8e5400b330 comp rx=0 tx=0).stop 2023-03-03T08:43:29.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.243+0000 7f8e614b2700 1 -- 172.21.15.7:0/847839878 shutdown_connections 2023-03-03T08:43:29.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.243+0000 7f8e614b2700 1 --2- 172.21.15.7:0/847839878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0a9af0 0x7f8e5c0c2c60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.243+0000 7f8e614b2700 1 --2- 172.21.15.7:0/847839878 >> v2:172.21.15.143:3300/0 conn(0x7f8e5c0c65c0 0x7f8e5c0c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.243+0000 7f8e614b2700 1 -- 172.21.15.7:0/847839878 >> 172.21.15.7:0/847839878 conn(0x7f8e5c0a9430 msgr2=0x7f8e5c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:29.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.243+0000 7f8e614b2700 1 -- 172.21.15.7:0/847839878 shutdown_connections 2023-03-03T08:43:29.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.243+0000 7f8e614b2700 1 -- 172.21.15.7:0/847839878 wait complete. 2023-03-03T08:43:29.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.244+0000 7f8e614b2700 1 Processor -- start 2023-03-03T08:43:29.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.244+0000 7f8e614b2700 1 -- start start 2023-03-03T08:43:29.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e614b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e5c0a9af0 0x7f8e5c1dc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e614b2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0c65c0 0x7f8e5c0c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e614b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e5c14de50 0x7f8e5c0c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e614b2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c1e7a20 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e614b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c1e89b0 con 0x7f8e5c14de50 2023-03-03T08:43:29.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e614b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c1e8520 con 0x7f8e5c0a9af0 2023-03-03T08:43:29.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0c65c0 0x7f8e5c0c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0c65c0 0x7f8e5c0c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50942/0 (socket says 172.21.15.7:50942) 2023-03-03T08:43:29.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5a7fc700 1 -- 172.21.15.7:0/2927274640 learned_addr learned my addr 172.21.15.7:0/2927274640 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:29.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5affd700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e5c0a9af0 0x7f8e5c1dc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5b7fe700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e5c14de50 0x7f8e5c0c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5a7fc700 1 -- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e5c0a9af0 msgr2=0x7f8e5c1dc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5a7fc700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e5c0a9af0 0x7f8e5c1dc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5a7fc700 1 -- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e5c14de50 msgr2=0x7f8e5c0c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5a7fc700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e5c14de50 0x7f8e5c0c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5a7fc700 1 -- 172.21.15.7:0/2927274640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e54009b30 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5b7fe700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e5c14de50 0x7f8e5c0c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:29.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e5a7fc700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0c65c0 0x7f8e5c0c3ac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00e7d0 tx=0x7f8e4c00eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:29.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e437fe700 1 -- 172.21.15.7:0/2927274640 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c013610 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e437fe700 1 -- 172.21.15.7:0/2927274640 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c013770 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e614b2700 1 -- 172.21.15.7:0/2927274640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5c154a90 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e437fe700 1 -- 172.21.15.7:0/2927274640 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c013b00 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.245+0000 7f8e614b2700 1 -- 172.21.15.7:0/2927274640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5c1547b0 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.246+0000 7f8e614b2700 1 -- 172.21.15.7:0/2927274640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5c153e80 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.247+0000 7f8e437fe700 1 -- 172.21.15.7:0/2927274640 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e4c023070 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.247+0000 7f8e437fe700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e480682f0 0x7f8e4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.247+0000 7f8e437fe700 1 -- 172.21.15.7:0/2927274640 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8e4c085450 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.247+0000 7f8e5affd700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e480682f0 0x7f8e4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.248+0000 7f8e5affd700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e480682f0 0x7f8e4806a7a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0a5e10 tx=0x7f8e50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.251+0000 7f8e437fe700 1 -- 172.21.15.7:0/2927274640 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e4c052000 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.448+0000 7f8e614b2700 1 -- 172.21.15.7:0/2927274640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e5c074bb0 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.449+0000 7f8e437fe700 1 -- 172.21.15.7:0/2927274640 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+14159 (secure 0 0 0) 0x7f8e4c055620 con 0x7f8e5c0c65c0 2023-03-03T08:43:29.451 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:29.451 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":18,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:29.064428+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 -- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e480682f0 msgr2=0x7f8e4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e480682f0 0x7f8e4806a7a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0a5e10 tx=0x7f8e50009040 comp rx=0 tx=0).stop 2023-03-03T08:43:29.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 -- 172.21.15.7:0/2927274640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0c65c0 msgr2=0x7f8e5c0c3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0c65c0 0x7f8e5c0c3ac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00e7d0 tx=0x7f8e4c00eae0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 -- 172.21.15.7:0/2927274640 shutdown_connections 2023-03-03T08:43:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e5c14de50 0x7f8e5c0c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e5c0a9af0 0x7f8e5c1dc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e480682f0 0x7f8e4806a7a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 --2- 172.21.15.7:0/2927274640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e5c0c65c0 0x7f8e5c0c3ac0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 -- 172.21.15.7:0/2927274640 >> 172.21.15.7:0/2927274640 conn(0x7f8e5c0a9430 msgr2=0x7f8e5c06d110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:29.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.451+0000 7f8e614b2700 1 -- 172.21.15.7:0/2927274640 shutdown_connections 2023-03-03T08:43:29.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.452+0000 7f8e614b2700 1 -- 172.21.15.7:0/2927274640 wait complete. 2023-03-03T08:43:29.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T08:43:21.160680+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T08:43:28.018457+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:43:29.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:43:29.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.591+0000 7f68d0e5e700 1 Processor -- start 2023-03-03T08:43:29.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.591+0000 7f68d0e5e700 1 -- start start 2023-03-03T08:43:29.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.591+0000 7f68d0e5e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f68cc13a0f0 0x7f68cc0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68d0e5e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68cc1586c0 0x7f68cc06d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68d0e5e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc06de90 con 0x7f68cc0c1da0 2023-03-03T08:43:29.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68d0e5e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68cc154840 con 0x7f68cc1586c0 2023-03-03T08:43:29.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68d0e5e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68cc154560 con 0x7f68cc13a0f0 2023-03-03T08:43:29.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c9d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68cc1586c0 0x7f68cc06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c9d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68cc1586c0 0x7f68cc06d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50950/0 (socket says 172.21.15.7:50950) 2023-03-03T08:43:29.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c9d9b700 1 -- 172.21.15.7:0/2742692185 learned_addr learned my addr 172.21.15.7:0/2742692185 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:29.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68ca59c700 1 --2- 172.21.15.7:0/2742692185 >> v2:172.21.15.143:3300/0 conn(0x7f68cc13a0f0 0x7f68cc0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c9d9b700 1 -- 172.21.15.7:0/2742692185 >> v2:172.21.15.143:3300/0 conn(0x7f68cc13a0f0 msgr2=0x7f68cc0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:29.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c9d9b700 1 --2- 172.21.15.7:0/2742692185 >> v2:172.21.15.143:3300/0 conn(0x7f68cc13a0f0 0x7f68cc0c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c9d9b700 1 -- 172.21.15.7:0/2742692185 >> v1:172.21.15.143:6789/0 conn(0x7f68cc0c1da0 legacy=0x7f68cc06d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c9d9b700 1 -- 172.21.15.7:0/2742692185 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc157be0 con 0x7f68cc1586c0 2023-03-03T08:43:29.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c9d9b700 1 --2- 172.21.15.7:0/2742692185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc1586c0 0x7f68cc06d240 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f68cc06cf70 tx=0x7f68b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52b3f149fcb1429 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:29.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c959a700 1 -- 172.21.15.7:0/2742692185 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b400b870 con 0x7f68cc1586c0 2023-03-03T08:43:29.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c959a700 1 -- 172.21.15.7:0/2742692185 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b400b9d0 con 0x7f68cc1586c0 2023-03-03T08:43:29.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.592+0000 7f68c959a700 1 -- 172.21.15.7:0/2742692185 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b400bd60 con 0x7f68cc1586c0 2023-03-03T08:43:29.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.593+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2742692185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc1586c0 msgr2=0x7f68cc06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.593+0000 7f68d0e5e700 1 --2- 172.21.15.7:0/2742692185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc1586c0 0x7f68cc06d240 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f68cc06cf70 tx=0x7f68b4009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.593+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2742692185 shutdown_connections 2023-03-03T08:43:29.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.593+0000 7f68d0e5e700 1 --2- 172.21.15.7:0/2742692185 >> v2:172.21.15.143:3300/0 conn(0x7f68cc13a0f0 0x7f68cc0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.593+0000 7f68d0e5e700 1 --2- 172.21.15.7:0/2742692185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc1586c0 0x7f68cc06d240 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.593+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2742692185 >> 172.21.15.7:0/2742692185 conn(0x7f68cc06c110 msgr2=0x7f68cc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.593+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2742692185 shutdown_connections 2023-03-03T08:43:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.593+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2742692185 wait complete. 2023-03-03T08:43:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.594+0000 7f68d0e5e700 1 Processor -- start 2023-03-03T08:43:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.594+0000 7f68d0e5e700 1 -- start start 2023-03-03T08:43:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68d0e5e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68cc0c1da0 0x7f68cc1e9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68d0e5e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc13a0f0 0x7f68cc1341b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68d0e5e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68cc1586c0 0x7f68cc155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68d0e5e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc1eadc0 con 0x7f68cc13a0f0 2023-03-03T08:43:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68d0e5e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc1514c0 con 0x7f68cc1586c0 2023-03-03T08:43:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68d0e5e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc155970 con 0x7f68cc0c1da0 2023-03-03T08:43:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68c9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc13a0f0 0x7f68cc1341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68c9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc13a0f0 0x7f68cc1341b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50956/0 (socket says 172.21.15.7:50956) 2023-03-03T08:43:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68c9d9b700 1 -- 172.21.15.7:0/2279469742 learned_addr learned my addr 172.21.15.7:0/2279469742 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68ca59c700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68cc0c1da0 0x7f68cc1e9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68cad9d700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68cc1586c0 0x7f68cc155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68c9d9b700 1 -- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68cc0c1da0 msgr2=0x7f68cc1e9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68c9d9b700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68cc0c1da0 0x7f68cc1e9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68c9d9b700 1 -- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68cc1586c0 msgr2=0x7f68cc155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68c9d9b700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68cc1586c0 0x7f68cc155e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68c9d9b700 1 -- 172.21.15.7:0/2279469742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b4009550 con 0x7f68cc13a0f0 2023-03-03T08:43:29.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.595+0000 7f68c9d9b700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc13a0f0 0x7f68cc1341b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68cc06cff0 tx=0x7f68b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:29.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.596+0000 7f68c2ffd700 1 -- 172.21.15.7:0/2279469742 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4004480 con 0x7f68cc13a0f0 2023-03-03T08:43:29.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.596+0000 7f68c2ffd700 1 -- 172.21.15.7:0/2279469742 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b40045e0 con 0x7f68cc13a0f0 2023-03-03T08:43:29.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.596+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2279469742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68cc1581a0 con 0x7f68cc13a0f0 2023-03-03T08:43:29.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.596+0000 7f68c2ffd700 1 -- 172.21.15.7:0/2279469742 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b40038e0 con 0x7f68cc13a0f0 2023-03-03T08:43:29.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.596+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2279469742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68cc157ec0 con 0x7f68cc13a0f0 2023-03-03T08:43:29.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.597+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2279469742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68cc06e1a0 con 0x7f68cc13a0f0 2023-03-03T08:43:29.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.597+0000 7f68c2ffd700 1 -- 172.21.15.7:0/2279469742 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f68b400f050 con 0x7f68cc13a0f0 2023-03-03T08:43:29.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.597+0000 7f68c2ffd700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68ac0682f0 0x7f68ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.598+0000 7f68c2ffd700 1 -- 172.21.15.7:0/2279469742 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f68b4087020 con 0x7f68cc13a0f0 2023-03-03T08:43:29.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.598+0000 7f68ca59c700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68ac0682f0 0x7f68ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.598+0000 7f68ca59c700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68ac0682f0 0x7f68ac06a7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0099d0 tx=0x7f68bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:29.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.602+0000 7f68c2ffd700 1 -- 172.21.15.7:0/2279469742 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68b4053bd0 con 0x7f68cc13a0f0 2023-03-03T08:43:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.793+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2279469742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f68cc155ae0 con 0x7f68cc13a0f0 2023-03-03T08:43:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.793+0000 7f68c2ffd700 1 -- 172.21.15.7:0/2279469742 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f68b40150a0 con 0x7f68cc13a0f0 2023-03-03T08:43:29.795 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-03-03T08:43:29.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68ac0682f0 msgr2=0x7f68ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68ac0682f0 0x7f68ac06a7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0099d0 tx=0x7f68bc008040 comp rx=0 tx=0).stop 2023-03-03T08:43:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2279469742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc13a0f0 msgr2=0x7f68cc1341b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc13a0f0 0x7f68cc1341b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f68cc06cff0 tx=0x7f68b4004750 comp rx=0 tx=0).stop 2023-03-03T08:43:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2279469742 shutdown_connections 2023-03-03T08:43:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68cc1586c0 0x7f68cc155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68cc0c1da0 0x7f68cc1e9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68ac0682f0 0x7f68ac06a7a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 --2- 172.21.15.7:0/2279469742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68cc13a0f0 0x7f68cc1341b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2279469742 >> 172.21.15.7:0/2279469742 conn(0x7f68cc06c110 msgr2=0x7f68cc0c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2279469742 shutdown_connections 2023-03-03T08:43:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.795+0000 7f68d0e5e700 1 -- 172.21.15.7:0/2279469742 wait complete. 2023-03-03T08:43:29.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:43:29.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.935+0000 7f2cfbd25700 1 Processor -- start 2023-03-03T08:43:29.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.935+0000 7f2cfbd25700 1 -- start start 2023-03-03T08:43:29.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cfbd25700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2cf406d4b0 0x7f2cf406d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cfbd25700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf40c7b90 con 0x7f2cf40c1da0 2023-03-03T08:43:29.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cfbd25700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf4154640 con 0x7f2cf40c8020 2023-03-03T08:43:29.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cfbd25700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf40c7330 con 0x7f2cf406d4b0 2023-03-03T08:43:29.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cf9ac1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2cf406d4b0 0x7f2cf406d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cf9ac1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2cf406d4b0 0x7f2cf406d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50962/0 (socket says 172.21.15.7:50962) 2023-03-03T08:43:29.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cf9ac1700 1 -- 172.21.15.7:0/493049844 learned_addr learned my addr 172.21.15.7:0/493049844 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:29.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cfa2c2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2cf40c1da0 0x7f2cf4139d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44946/0 (socket says 172.21.15.7:44946) 2023-03-03T08:43:29.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cf9ac1700 1 -- 172.21.15.7:0/493049844 >> v1:172.21.15.143:6789/0 conn(0x7f2cf40c8020 legacy=0x7f2cf40c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:29.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cf9ac1700 1 -- 172.21.15.7:0/493049844 >> v1:172.21.15.7:6789/0 conn(0x7f2cf40c1da0 legacy=0x7f2cf4139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cf9ac1700 1 -- 172.21.15.7:0/493049844 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf40c7540 con 0x7f2cf406d4b0 2023-03-03T08:43:29.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cf8abf700 1 -- 172.21.15.7:0/493049844 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477043624 0 0) 0x7f2cf40c7b90 con 0x7f2cf40c1da0 2023-03-03T08:43:29.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.936+0000 7f2cf9ac1700 1 --2- 172.21.15.7:0/493049844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf406d4b0 0x7f2cf406d880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cf00098f0 tx=0x7f2cf0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b27f39dcb535bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:29.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.937+0000 7f2cf8abf700 1 -- 172.21.15.7:0/493049844 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0004230 con 0x7f2cf406d4b0 2023-03-03T08:43:29.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.937+0000 7f2cf8abf700 1 -- 172.21.15.7:0/493049844 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf0004390 con 0x7f2cf406d4b0 2023-03-03T08:43:29.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.937+0000 7f2cf8abf700 1 -- 172.21.15.7:0/493049844 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf00036a0 con 0x7f2cf406d4b0 2023-03-03T08:43:29.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.937+0000 7f2cfbd25700 1 -- 172.21.15.7:0/493049844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf406d4b0 msgr2=0x7f2cf406d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.937+0000 7f2cfbd25700 1 --2- 172.21.15.7:0/493049844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf406d4b0 0x7f2cf406d880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cf00098f0 tx=0x7f2cf0009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:29.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.937+0000 7f2cfbd25700 1 -- 172.21.15.7:0/493049844 shutdown_connections 2023-03-03T08:43:29.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.937+0000 7f2cfbd25700 1 --2- 172.21.15.7:0/493049844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf406d4b0 0x7f2cf406d880 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.937+0000 7f2cfbd25700 1 -- 172.21.15.7:0/493049844 >> 172.21.15.7:0/493049844 conn(0x7f2cf406c110 msgr2=0x7f2cf406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:29.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.937+0000 7f2cfbd25700 1 -- 172.21.15.7:0/493049844 shutdown_connections 2023-03-03T08:43:29.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.937+0000 7f2cfbd25700 1 -- 172.21.15.7:0/493049844 wait complete. 2023-03-03T08:43:29.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.939+0000 7f2cfbd25700 1 Processor -- start 2023-03-03T08:43:29.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.939+0000 7f2cfbd25700 1 -- start start 2023-03-03T08:43:29.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfbd25700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf406d4b0 0x7f2cf41e9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfbd25700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c1da0 0x7f2cf41e0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfbd25700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf40c8020 0x7f2cf413c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfbd25700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf4133690 con 0x7f2cf40c8020 2023-03-03T08:43:29.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfbd25700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf41ea4e0 con 0x7f2cf40c1da0 2023-03-03T08:43:29.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfbd25700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf40723f0 con 0x7f2cf406d4b0 2023-03-03T08:43:29.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfa2c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf40c8020 0x7f2cf413c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfa2c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf40c8020 0x7f2cf413c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50972/0 (socket says 172.21.15.7:50972) 2023-03-03T08:43:29.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfa2c2700 1 -- 172.21.15.7:0/930682852 learned_addr learned my addr 172.21.15.7:0/930682852 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:29.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cf9ac1700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf406d4b0 0x7f2cf41e9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cf92c0700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c1da0 0x7f2cf41e0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfa2c2700 1 -- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf406d4b0 msgr2=0x7f2cf41e9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfa2c2700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf406d4b0 0x7f2cf41e9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfa2c2700 1 -- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c1da0 msgr2=0x7f2cf41e0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:29.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfa2c2700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c1da0 0x7f2cf41e0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:29.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfa2c2700 1 -- 172.21.15.7:0/930682852 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf0009550 con 0x7f2cf40c8020 2023-03-03T08:43:29.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cf92c0700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c1da0 0x7f2cf41e0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:29.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cf9ac1700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf406d4b0 0x7f2cf41e9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:29.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfa2c2700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf40c8020 0x7f2cf413c600 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00cc70 tx=0x7f2cec00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:29.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cea7fc700 1 -- 172.21.15.7:0/930682852 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00ad80 con 0x7f2cf40c8020 2023-03-03T08:43:29.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cea7fc700 1 -- 172.21.15.7:0/930682852 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec00ce70 con 0x7f2cf40c8020 2023-03-03T08:43:29.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfbd25700 1 -- 172.21.15.7:0/930682852 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf406ddc0 con 0x7f2cf40c8020 2023-03-03T08:43:29.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cea7fc700 1 -- 172.21.15.7:0/930682852 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec0072f0 con 0x7f2cf40c8020 2023-03-03T08:43:29.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.940+0000 7f2cfbd25700 1 -- 172.21.15.7:0/930682852 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf40c7820 con 0x7f2cf40c8020 2023-03-03T08:43:29.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.941+0000 7f2cfbd25700 1 -- 172.21.15.7:0/930682852 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf406e210 con 0x7f2cf40c8020 2023-03-03T08:43:29.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.942+0000 7f2cea7fc700 1 -- 172.21.15.7:0/930682852 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2cec01a030 con 0x7f2cf40c8020 2023-03-03T08:43:29.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.942+0000 7f2cea7fc700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 0x7f2cd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:29.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.942+0000 7f2cea7fc700 1 -- 172.21.15.7:0/930682852 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f2cec085690 con 0x7f2cf40c8020 2023-03-03T08:43:29.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.942+0000 7f2cf9ac1700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 0x7f2cd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:29.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.943+0000 7f2cf9ac1700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 0x7f2cd806a7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009c40 tx=0x7f2cf0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:29.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:29.947+0000 7f2cea7fc700 1 -- 172.21.15.7:0/930682852 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2cec052240 con 0x7f2cf40c8020 2023-03-03T08:43:30.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.140+0000 7f2cfbd25700 1 -- 172.21.15.7:0/930682852 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2cf4074bb0 con 0x7f2cf40c8020 2023-03-03T08:43:30.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.140+0000 7f2cea7fc700 1 -- 172.21.15.7:0/930682852 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2cec0180a0 con 0x7f2cf40c8020 2023-03-03T08:43:30.142 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 8 2023-03-03T08:43:30.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 -- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 msgr2=0x7f2cd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 0x7f2cd806a7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009c40 tx=0x7f2cf0008030 comp rx=0 tx=0).stop 2023-03-03T08:43:30.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 -- 172.21.15.7:0/930682852 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf40c8020 msgr2=0x7f2cf413c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf40c8020 0x7f2cf413c600 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00cc70 tx=0x7f2cec00a340 comp rx=0 tx=0).stop 2023-03-03T08:43:30.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 -- 172.21.15.7:0/930682852 shutdown_connections 2023-03-03T08:43:30.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 0x7f2cd806a7a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf40c8020 0x7f2cf413c600 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf406d4b0 0x7f2cf41e9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 --2- 172.21.15.7:0/930682852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c1da0 0x7f2cf41e0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 -- 172.21.15.7:0/930682852 >> 172.21.15.7:0/930682852 conn(0x7f2cf406c110 msgr2=0x7f2cf41549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:30.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 -- 172.21.15.7:0/930682852 shutdown_connections 2023-03-03T08:43:30.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.142+0000 7f2cfbd25700 1 -- 172.21.15.7:0/930682852 wait complete. 2023-03-03T08:43:30.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:43:30.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.279+0000 7fd5f7df5700 1 Processor -- start 2023-03-03T08:43:30.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.280+0000 7fd5f7df5700 1 -- start start 2023-03-03T08:43:30.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.280+0000 7fd5f7df5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd5f00beab0 0x7fd5f0155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.280+0000 7fd5f7df5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f01523b0 con 0x7fd5f01562c0 2023-03-03T08:43:30.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.280+0000 7fd5f7df5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f0072010 con 0x7fd5f0152e30 2023-03-03T08:43:30.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.280+0000 7fd5f7df5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f00aaaf0 con 0x7fd5f00beab0 2023-03-03T08:43:30.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.280+0000 7fd5f5390700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd5f00beab0 0x7fd5f0155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.281+0000 7fd5f5390700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd5f00beab0 0x7fd5f0155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50980/0 (socket says 172.21.15.7:50980) 2023-03-03T08:43:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.281+0000 7fd5f5390700 1 -- 172.21.15.7:0/3577972343 learned_addr learned my addr 172.21.15.7:0/3577972343 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.281+0000 7fd5f5390700 1 -- 172.21.15.7:0/3577972343 >> v1:172.21.15.143:6789/0 conn(0x7fd5f0152e30 legacy=0x7fd5f00c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.281+0000 7fd5f5390700 1 -- 172.21.15.7:0/3577972343 >> v1:172.21.15.7:6789/0 conn(0x7fd5f01562c0 legacy=0x7fd5f015cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.281+0000 7fd5f5390700 1 -- 172.21.15.7:0/3577972343 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f00aadd0 con 0x7fd5f00beab0 2023-03-03T08:43:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.281+0000 7fd5f4b8f700 1 -- 172.21.15.7:0/3577972343 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342806686 0 0) 0x7fd5f01523b0 con 0x7fd5f01562c0 2023-03-03T08:43:30.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.281+0000 7fd5f5390700 1 --2- 172.21.15.7:0/3577972343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 0x7fd5f0155d80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40098f0 tx=0x7fd5e4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a236eb83fe6753bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:30.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.281+0000 7fd5f4b8f700 1 -- 172.21.15.7:0/3577972343 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400b8a0 con 0x7fd5f00beab0 2023-03-03T08:43:30.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.281+0000 7fd5f4b8f700 1 -- 172.21.15.7:0/3577972343 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e400ba00 con 0x7fd5f00beab0 2023-03-03T08:43:30.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.281+0000 7fd5f4b8f700 1 -- 172.21.15.7:0/3577972343 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400bd90 con 0x7fd5f00beab0 2023-03-03T08:43:30.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.282+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3577972343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 msgr2=0x7fd5f0155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.282+0000 7fd5f7df5700 1 --2- 172.21.15.7:0/3577972343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 0x7fd5f0155d80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40098f0 tx=0x7fd5e4009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:30.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.282+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3577972343 shutdown_connections 2023-03-03T08:43:30.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.282+0000 7fd5f7df5700 1 --2- 172.21.15.7:0/3577972343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 0x7fd5f0155d80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.282+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3577972343 >> 172.21.15.7:0/3577972343 conn(0x7fd5f00736f0 msgr2=0x7fd5f0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:30.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.282+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3577972343 shutdown_connections 2023-03-03T08:43:30.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.282+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3577972343 wait complete. 2023-03-03T08:43:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.283+0000 7fd5f7df5700 1 Processor -- start 2023-03-03T08:43:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.283+0000 7fd5f7df5700 1 -- start start 2023-03-03T08:43:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.284+0000 7fd5f7df5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 0x7fd5f00ac900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.284+0000 7fd5f7df5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f0152e30 0x7fd5f00c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.284+0000 7fd5f7df5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f01562c0 0x7fd5f00ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.284+0000 7fd5f7df5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f01554d0 con 0x7fd5f00beab0 2023-03-03T08:43:30.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.284+0000 7fd5f7df5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f0155a00 con 0x7fd5f0152e30 2023-03-03T08:43:30.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.284+0000 7fd5f7df5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f00ace40 con 0x7fd5f01562c0 2023-03-03T08:43:30.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.284+0000 7fd5f5b91700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 0x7fd5f00ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.284+0000 7fd5f5b91700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 0x7fd5f00ac900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50984/0 (socket says 172.21.15.7:50984) 2023-03-03T08:43:30.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.284+0000 7fd5f5b91700 1 -- 172.21.15.7:0/3277537423 learned_addr learned my addr 172.21.15.7:0/3277537423 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:30.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f5390700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f0152e30 0x7fd5f00c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f6392700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f01562c0 0x7fd5f00ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f5b91700 1 -- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f01562c0 msgr2=0x7fd5f00ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f5b91700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f01562c0 0x7fd5f00ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f5b91700 1 -- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f0152e30 msgr2=0x7fd5f00c45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f5b91700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f0152e30 0x7fd5f00c45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f5b91700 1 -- 172.21.15.7:0/3277537423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e4009550 con 0x7fd5f00beab0 2023-03-03T08:43:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f5390700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f0152e30 0x7fd5f00c45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f5b91700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 0x7fd5f00ac900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00c510 tx=0x7fd5ec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5e27fc700 1 -- 172.21.15.7:0/3277537423 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0052d0 con 0x7fd5f00beab0 2023-03-03T08:43:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5e27fc700 1 -- 172.21.15.7:0/3277537423 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ec005430 con 0x7fd5f00beab0 2023-03-03T08:43:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3277537423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f0153200 con 0x7fd5f00beab0 2023-03-03T08:43:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5e27fc700 1 -- 172.21.15.7:0/3277537423 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0038c0 con 0x7fd5f00beab0 2023-03-03T08:43:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.285+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3277537423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f0071c70 con 0x7fd5f00beab0 2023-03-03T08:43:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.286+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3277537423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f0153390 con 0x7fd5f00beab0 2023-03-03T08:43:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.286+0000 7fd5e27fc700 1 -- 172.21.15.7:0/3277537423 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd5ec019030 con 0x7fd5f00beab0 2023-03-03T08:43:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.287+0000 7fd5e27fc700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d40682f0 0x7fd5d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.287+0000 7fd5e27fc700 1 -- 172.21.15.7:0/3277537423 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd5ec084cf0 con 0x7fd5f00beab0 2023-03-03T08:43:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.287+0000 7fd5f5390700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d40682f0 0x7fd5d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.287+0000 7fd5f5390700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d40682f0 0x7fd5d406a7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009130 tx=0x7fd5e401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.291+0000 7fd5e27fc700 1 -- 172.21.15.7:0/3277537423 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd5ec0518a0 con 0x7fd5f00beab0 2023-03-03T08:43:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.535+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3277537423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5f00ac620 con 0x7fd5f00beab0 2023-03-03T08:43:30.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.535+0000 7fd5e27fc700 1 -- 172.21.15.7:0/3277537423 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3524 (secure 0 0 0) 0x7fd5ec017de0 con 0x7fd5f00beab0 2023-03-03T08:43:30.537 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:30.537 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-03-03T08:43:30.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d40682f0 msgr2=0x7fd5d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d40682f0 0x7fd5d406a7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009130 tx=0x7fd5e401c040 comp rx=0 tx=0).stop 2023-03-03T08:43:30.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3277537423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 msgr2=0x7fd5f00ac900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 0x7fd5f00ac900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00c510 tx=0x7fd5ec00c8d0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3277537423 shutdown_connections 2023-03-03T08:43:30.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f01562c0 0x7fd5f00ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d40682f0 0x7fd5d406a7a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f00beab0 0x7fd5f00ac900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 --2- 172.21.15.7:0/3277537423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f0152e30 0x7fd5f00c45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3277537423 >> 172.21.15.7:0/3277537423 conn(0x7fd5f00736f0 msgr2=0x7fd5f00aa170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3277537423 shutdown_connections 2023-03-03T08:43:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.537+0000 7fd5f7df5700 1 -- 172.21.15.7:0/3277537423 wait complete. 2023-03-03T08:43:30.541 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 3 2023-03-03T08:43:30.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:30.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.674+0000 7f8ae5687700 1 Processor -- start 2023-03-03T08:43:30.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.674+0000 7f8ae5687700 1 -- start start 2023-03-03T08:43:30.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ae5687700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ae00caa30 0x7f8ae00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ae5687700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ae00c00e0 0x7f8ae0154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ae5687700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ae00cae00 0x7f8ae00c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ae5687700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae00cb280 con 0x7f8ae00c00e0 2023-03-03T08:43:30.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ae5687700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae00bfd10 con 0x7f8ae00caa30 2023-03-03T08:43:30.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ae5687700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00bfa30 con 0x7f8ae00cae00 2023-03-03T08:43:30.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ade7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ae00c00e0 0x7f8ae0154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ade7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ae00c00e0 0x7f8ae0154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50992/0 (socket says 172.21.15.7:50992) 2023-03-03T08:43:30.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ade7fc700 1 -- 172.21.15.7:0/47716142 learned_addr learned my addr 172.21.15.7:0/47716142 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:30.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8adf7fe700 1 --2- 172.21.15.7:0/47716142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ae00cae00 0x7f8ae00c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8adeffd700 1 --2- 172.21.15.7:0/47716142 >> v2:172.21.15.143:3300/0 conn(0x7f8ae00caa30 0x7f8ae00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ade7fc700 1 -- 172.21.15.7:0/47716142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ae00cae00 msgr2=0x7f8ae00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:30.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ade7fc700 1 --2- 172.21.15.7:0/47716142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ae00cae00 0x7f8ae00c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ade7fc700 1 -- 172.21.15.7:0/47716142 >> v2:172.21.15.143:3300/0 conn(0x7f8ae00caa30 msgr2=0x7f8ae00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:30.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ade7fc700 1 --2- 172.21.15.7:0/47716142 >> v2:172.21.15.143:3300/0 conn(0x7f8ae00caa30 0x7f8ae00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ade7fc700 1 -- 172.21.15.7:0/47716142 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0154510 con 0x7f8ae00c00e0 2023-03-03T08:43:30.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.675+0000 7f8ade7fc700 1 --2- 172.21.15.7:0/47716142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00c00e0 0x7f8ae0154ff0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00c81b0 tx=0x7f8ad0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78fb2f1181edcbc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:30.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8addffb700 1 -- 172.21.15.7:0/47716142 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0005e30 con 0x7f8ae00c00e0 2023-03-03T08:43:30.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8addffb700 1 -- 172.21.15.7:0/47716142 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad0004030 con 0x7f8ae00c00e0 2023-03-03T08:43:30.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8addffb700 1 -- 172.21.15.7:0/47716142 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad00043e0 con 0x7f8ae00c00e0 2023-03-03T08:43:30.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8ae5687700 1 -- 172.21.15.7:0/47716142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00c00e0 msgr2=0x7f8ae0154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8ae5687700 1 --2- 172.21.15.7:0/47716142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00c00e0 0x7f8ae0154ff0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00c81b0 tx=0x7f8ad0009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8ae5687700 1 -- 172.21.15.7:0/47716142 shutdown_connections 2023-03-03T08:43:30.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8ae5687700 1 --2- 172.21.15.7:0/47716142 >> v2:172.21.15.143:3300/0 conn(0x7f8ae00caa30 0x7f8ae00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8ae5687700 1 --2- 172.21.15.7:0/47716142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ae00cae00 0x7f8ae00c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8ae5687700 1 --2- 172.21.15.7:0/47716142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00c00e0 0x7f8ae0154ff0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8ae5687700 1 -- 172.21.15.7:0/47716142 >> 172.21.15.7:0/47716142 conn(0x7f8ae0066e40 msgr2=0x7f8ae0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:30.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8ae5687700 1 -- 172.21.15.7:0/47716142 shutdown_connections 2023-03-03T08:43:30.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.676+0000 7f8ae5687700 1 -- 172.21.15.7:0/47716142 wait complete. 2023-03-03T08:43:30.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ae5687700 1 Processor -- start 2023-03-03T08:43:30.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ae5687700 1 -- start start 2023-03-03T08:43:30.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ae5687700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ae00c00e0 0x7f8ae00c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ae5687700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00caa30 0x7f8ae00c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ae5687700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae00cae00 0x7f8ae01eafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ae5687700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00c3d00 con 0x7f8ae00caa30 2023-03-03T08:43:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ae5687700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00c3330 con 0x7f8ae00cae00 2023-03-03T08:43:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ae5687700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae01e9c60 con 0x7f8ae00c00e0 2023-03-03T08:43:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ade7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00caa30 0x7f8ae00c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ade7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00caa30 0x7f8ae00c89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51000/0 (socket says 172.21.15.7:51000) 2023-03-03T08:43:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.678+0000 7f8ade7fc700 1 -- 172.21.15.7:0/2874628730 learned_addr learned my addr 172.21.15.7:0/2874628730 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8adeffd700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ae00c00e0 0x7f8ae00c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8adf7fe700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae00cae00 0x7f8ae01eafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ade7fc700 1 -- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ae00c00e0 msgr2=0x7f8ae00c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ade7fc700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ae00c00e0 0x7f8ae00c37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ade7fc700 1 -- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae00cae00 msgr2=0x7f8ae01eafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ade7fc700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae00cae00 0x7f8ae01eafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ade7fc700 1 -- 172.21.15.7:0/2874628730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad0009500 con 0x7f8ae00caa30 2023-03-03T08:43:30.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ade7fc700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00caa30 0x7f8ae00c89e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0000c00 tx=0x7f8ad0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:30.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ad77fe700 1 -- 172.21.15.7:0/2874628730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0003c30 con 0x7f8ae00caa30 2023-03-03T08:43:30.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ad77fe700 1 -- 172.21.15.7:0/2874628730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad0003d90 con 0x7f8ae00caa30 2023-03-03T08:43:30.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ae5687700 1 -- 172.21.15.7:0/2874628730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae0154ad0 con 0x7f8ae00caa30 2023-03-03T08:43:30.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ad77fe700 1 -- 172.21.15.7:0/2874628730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0011660 con 0x7f8ae00caa30 2023-03-03T08:43:30.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.679+0000 7f8ae5687700 1 -- 172.21.15.7:0/2874628730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae01547f0 con 0x7f8ae00caa30 2023-03-03T08:43:30.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.680+0000 7f8ae5687700 1 -- 172.21.15.7:0/2874628730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae00bf650 con 0x7f8ae00caa30 2023-03-03T08:43:30.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.681+0000 7f8ad77fe700 1 -- 172.21.15.7:0/2874628730 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8ad002c070 con 0x7f8ae00caa30 2023-03-03T08:43:30.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.681+0000 7f8ad77fe700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ac00682f0 0x7f8ac006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:30.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.681+0000 7f8ad77fe700 1 -- 172.21.15.7:0/2874628730 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8ad0086910 con 0x7f8ae00caa30 2023-03-03T08:43:30.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.681+0000 7f8adeffd700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ac00682f0 0x7f8ac006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:30.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.682+0000 7f8adeffd700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ac00682f0 0x7f8ac006a7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8ae014d0f0 tx=0x7f8ac8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:30.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.686+0000 7f8ad77fe700 1 -- 172.21.15.7:0/2874628730 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8ad00534c0 con 0x7f8ae00caa30 2023-03-03T08:43:30.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.881+0000 7f8ae5687700 1 -- 172.21.15.7:0/2874628730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ae0074bb0 con 0x7f8ae00caa30 2023-03-03T08:43:30.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.882+0000 7f8ad77fe700 1 -- 172.21.15.7:0/2874628730 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14146 (secure 0 0 0) 0x7f8ad0056ae0 con 0x7f8ae00caa30 2023-03-03T08:43:30.884 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:30.884 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":19,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:30.071919+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:30.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.884+0000 7f8ae5687700 1 -- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ac00682f0 msgr2=0x7f8ac006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.884+0000 7f8ae5687700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ac00682f0 0x7f8ac006a7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8ae014d0f0 tx=0x7f8ac8008040 comp rx=0 tx=0).stop 2023-03-03T08:43:30.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.884+0000 7f8ae5687700 1 -- 172.21.15.7:0/2874628730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00caa30 msgr2=0x7f8ae00c89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:30.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.884+0000 7f8ae5687700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00caa30 0x7f8ae00c89e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0000c00 tx=0x7f8ad0004750 comp rx=0 tx=0).stop 2023-03-03T08:43:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.884+0000 7f8ae5687700 1 -- 172.21.15.7:0/2874628730 shutdown_connections 2023-03-03T08:43:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.884+0000 7f8ae5687700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ae00cae00 0x7f8ae01eafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.885+0000 7f8ae5687700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ae00c00e0 0x7f8ae00c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.885+0000 7f8ae5687700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ac00682f0 0x7f8ac006a7a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.885+0000 7f8ae5687700 1 --2- 172.21.15.7:0/2874628730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ae00caa30 0x7f8ae00c89e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.885+0000 7f8ae5687700 1 -- 172.21.15.7:0/2874628730 >> 172.21.15.7:0/2874628730 conn(0x7f8ae0066e40 msgr2=0x7f8ae0066040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.885+0000 7f8ae5687700 1 -- 172.21.15.7:0/2874628730 shutdown_connections 2023-03-03T08:43:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:30.885+0000 7f8ae5687700 1 -- 172.21.15.7:0/2874628730 wait complete. 2023-03-03T08:43:30.898 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T08:43:21.160680+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T08:43:28.018457+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:43:30.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:43:31.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.025+0000 7f4579997700 1 Processor -- start 2023-03-03T08:43:31.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.025+0000 7f4579997700 1 -- start start 2023-03-03T08:43:31.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f4579997700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f45740c5c60 0x7f4574154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f4579997700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45740c7310 con 0x7f45740c00e0 2023-03-03T08:43:31.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f4579997700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45740bfd10 con 0x7f45740caa30 2023-03-03T08:43:31.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f4579997700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45740c60a0 con 0x7f45740c5c60 2023-03-03T08:43:31.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f45727fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f45740c00e0 0x7f457415cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44988/0 (socket says 172.21.15.7:44988) 2023-03-03T08:43:31.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f45727fc700 1 -- 172.21.15.7:0/3566712313 learned_addr learned my addr 172.21.15.7:0/3566712313 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:31.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f45737fe700 1 --2- 172.21.15.7:0/3566712313 >> v2:172.21.15.7:3300/0 conn(0x7f45740c5c60 0x7f4574154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f4571ffb700 1 -- 172.21.15.7:0/3566712313 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218190963 0 0) 0x7f45740c7310 con 0x7f45740c00e0 2023-03-03T08:43:31.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f4571ffb700 1 -- 172.21.15.7:0/3566712313 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455c0035b0 con 0x7f45740c00e0 2023-03-03T08:43:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f45737fe700 1 -- 172.21.15.7:0/3566712313 >> v1:172.21.15.143:6789/0 conn(0x7f45740caa30 legacy=0x7f45740c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f45737fe700 1 -- 172.21.15.7:0/3566712313 >> v1:172.21.15.7:6789/0 conn(0x7f45740c00e0 legacy=0x7f457415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f45737fe700 1 -- 172.21.15.7:0/3566712313 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45740bfa30 con 0x7f45740c5c60 2023-03-03T08:43:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f45737fe700 1 --2- 172.21.15.7:0/3566712313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740c5c60 0x7f4574154ff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f456c009d90 tx=0x7f456c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc100d15d3597c94 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f4571ffb700 1 -- 172.21.15.7:0/3566712313 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c00e070 con 0x7f45740c5c60 2023-03-03T08:43:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f4571ffb700 1 -- 172.21.15.7:0/3566712313 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456c00bb20 con 0x7f45740c5c60 2023-03-03T08:43:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.026+0000 7f4571ffb700 1 -- 172.21.15.7:0/3566712313 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c00be90 con 0x7f45740c5c60 2023-03-03T08:43:31.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.027+0000 7f4579997700 1 -- 172.21.15.7:0/3566712313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740c5c60 msgr2=0x7f4574154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.027+0000 7f4579997700 1 --2- 172.21.15.7:0/3566712313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740c5c60 0x7f4574154ff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f456c009d90 tx=0x7f456c00b330 comp rx=0 tx=0).stop 2023-03-03T08:43:31.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.027+0000 7f4579997700 1 -- 172.21.15.7:0/3566712313 shutdown_connections 2023-03-03T08:43:31.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.027+0000 7f4579997700 1 --2- 172.21.15.7:0/3566712313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740c5c60 0x7f4574154ff0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.027+0000 7f4579997700 1 -- 172.21.15.7:0/3566712313 >> 172.21.15.7:0/3566712313 conn(0x7f4574066e40 msgr2=0x7f4574071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:31.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.027+0000 7f4579997700 1 -- 172.21.15.7:0/3566712313 shutdown_connections 2023-03-03T08:43:31.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.027+0000 7f4579997700 1 -- 172.21.15.7:0/3566712313 wait complete. 2023-03-03T08:43:31.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.028+0000 7f4579997700 1 Processor -- start 2023-03-03T08:43:31.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.028+0000 7f4579997700 1 -- start start 2023-03-03T08:43:31.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f4579997700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45740c00e0 0x7f45741564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f4579997700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45740c5c60 0x7f45740c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f4579997700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740caa30 0x7f4574157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f4579997700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45741ed930 con 0x7f45740caa30 2023-03-03T08:43:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f4579997700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45741ede60 con 0x7f45740c00e0 2023-03-03T08:43:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f4579997700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45741de130 con 0x7f45740c5c60 2023-03-03T08:43:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f45737fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740caa30 0x7f4574157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f45737fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740caa30 0x7f4574157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51016/0 (socket says 172.21.15.7:51016) 2023-03-03T08:43:31.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f45737fe700 1 -- 172.21.15.7:0/755209977 learned_addr learned my addr 172.21.15.7:0/755209977 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:31.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f45727fc700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45740c5c60 0x7f45740c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f4572ffd700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45740c00e0 0x7f45741564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f45737fe700 1 -- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45740c5c60 msgr2=0x7f45740c3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f45737fe700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45740c5c60 0x7f45740c3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f45737fe700 1 -- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45740c00e0 msgr2=0x7f45741564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f45737fe700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45740c00e0 0x7f45741564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.029+0000 7f45737fe700 1 -- 172.21.15.7:0/755209977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c009a70 con 0x7f45740caa30 2023-03-03T08:43:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.030+0000 7f45737fe700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740caa30 0x7f4574157520 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f456c00b370 tx=0x7f456c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.030+0000 7f455bfff700 1 -- 172.21.15.7:0/755209977 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c004940 con 0x7f45740caa30 2023-03-03T08:43:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.030+0000 7f455bfff700 1 -- 172.21.15.7:0/755209977 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456c012430 con 0x7f45740caa30 2023-03-03T08:43:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.030+0000 7f4579997700 1 -- 172.21.15.7:0/755209977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45741547f0 con 0x7f45740caa30 2023-03-03T08:43:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.030+0000 7f455bfff700 1 -- 172.21.15.7:0/755209977 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c0127e0 con 0x7f45740caa30 2023-03-03T08:43:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.030+0000 7f4579997700 1 -- 172.21.15.7:0/755209977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4574154510 con 0x7f45740caa30 2023-03-03T08:43:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.031+0000 7f4579997700 1 -- 172.21.15.7:0/755209977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4574154ad0 con 0x7f45740caa30 2023-03-03T08:43:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.031+0000 7f455bfff700 1 -- 172.21.15.7:0/755209977 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f456c012940 con 0x7f45740caa30 2023-03-03T08:43:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.031+0000 7f455bfff700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 0x7f456006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.032+0000 7f455bfff700 1 -- 172.21.15.7:0/755209977 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f456c087ad0 con 0x7f45740caa30 2023-03-03T08:43:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.032+0000 7f4572ffd700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 0x7f456006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.032+0000 7f4572ffd700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 0x7f456006a7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f45740c4550 tx=0x7f4564009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:31.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.036+0000 7f455bfff700 1 -- 172.21.15.7:0/755209977 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f456c0546d0 con 0x7f45740caa30 2023-03-03T08:43:31.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.231+0000 7f4579997700 1 -- 172.21.15.7:0/755209977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4574074bb0 con 0x7f45740caa30 2023-03-03T08:43:31.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.231+0000 7f455bfff700 1 -- 172.21.15.7:0/755209977 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f456c02b0a0 con 0x7f45740caa30 2023-03-03T08:43:31.233 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-03-03T08:43:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 -- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 msgr2=0x7f456006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 0x7f456006a7a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f45740c4550 tx=0x7f4564009040 comp rx=0 tx=0).stop 2023-03-03T08:43:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 -- 172.21.15.7:0/755209977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740caa30 msgr2=0x7f4574157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740caa30 0x7f4574157520 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f456c00b370 tx=0x7f456c004ad0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 -- 172.21.15.7:0/755209977 shutdown_connections 2023-03-03T08:43:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 0x7f456006a7a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45740caa30 0x7f4574157520 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45740c00e0 0x7f45741564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 --2- 172.21.15.7:0/755209977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45740c5c60 0x7f45740c3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 -- 172.21.15.7:0/755209977 >> 172.21.15.7:0/755209977 conn(0x7f4574066e40 msgr2=0x7f457406b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:31.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 -- 172.21.15.7:0/755209977 shutdown_connections 2023-03-03T08:43:31.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.233+0000 7f4579997700 1 -- 172.21.15.7:0/755209977 wait complete. 2023-03-03T08:43:31.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:43:31.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.372+0000 7ff401934700 1 Processor -- start 2023-03-03T08:43:31.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.372+0000 7ff401934700 1 -- start start 2023-03-03T08:43:31.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.373+0000 7ff401934700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0c00e0 0x7ff3fc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.373+0000 7ff401934700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3fc154ff0 0x7ff3fc0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.373+0000 7ff401934700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0c7310 con 0x7ff3fc0caa30 2023-03-03T08:43:31.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.373+0000 7ff401934700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3fc0bfd10 con 0x7ff3fc154ff0 2023-03-03T08:43:31.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.373+0000 7ff401934700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc0bfa30 con 0x7ff3fc0c00e0 2023-03-03T08:43:31.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.373+0000 7ff3fb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3fc154ff0 0x7ff3fc0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.373+0000 7ff3fb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3fc154ff0 0x7ff3fc0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51024/0 (socket says 172.21.15.7:51024) 2023-03-03T08:43:31.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.373+0000 7ff3fb7fe700 1 -- 172.21.15.7:0/780976481 learned_addr learned my addr 172.21.15.7:0/780976481 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:31.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3fa7fc700 1 --2- 172.21.15.7:0/780976481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0c00e0 0x7ff3fc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3fb7fe700 1 -- 172.21.15.7:0/780976481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0c00e0 msgr2=0x7ff3fc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:31.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3fb7fe700 1 --2- 172.21.15.7:0/780976481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0c00e0 0x7ff3fc0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3fb7fe700 1 -- 172.21.15.7:0/780976481 >> v1:172.21.15.7:6789/0 conn(0x7ff3fc0caa30 legacy=0x7ff3fc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3fb7fe700 1 -- 172.21.15.7:0/780976481 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc154510 con 0x7ff3fc154ff0 2023-03-03T08:43:31.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3f9ffb700 1 -- 172.21.15.7:0/780976481 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130796056 0 0) 0x7ff3fc0c7310 con 0x7ff3fc0caa30 2023-03-03T08:43:31.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3fa7fc700 1 --2- 172.21.15.7:0/780976481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0c00e0 0x7ff3fc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:31.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3fb7fe700 1 --2- 172.21.15.7:0/780976481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 0x7ff3fc0cae00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009e50 tx=0x7ff3f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a43912f0ad4575c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:31.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3f9ffb700 1 -- 172.21.15.7:0/780976481 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f400e070 con 0x7ff3fc154ff0 2023-03-03T08:43:31.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3f9ffb700 1 -- 172.21.15.7:0/780976481 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f400b9c0 con 0x7ff3fc154ff0 2023-03-03T08:43:31.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff3f9ffb700 1 -- 172.21.15.7:0/780976481 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f400bd70 con 0x7ff3fc154ff0 2023-03-03T08:43:31.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff401934700 1 -- 172.21.15.7:0/780976481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 msgr2=0x7ff3fc0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff401934700 1 --2- 172.21.15.7:0/780976481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 0x7ff3fc0cae00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009e50 tx=0x7ff3f400b330 comp rx=0 tx=0).stop 2023-03-03T08:43:31.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff401934700 1 -- 172.21.15.7:0/780976481 shutdown_connections 2023-03-03T08:43:31.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff401934700 1 --2- 172.21.15.7:0/780976481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 0x7ff3fc0cae00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff401934700 1 --2- 172.21.15.7:0/780976481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0c00e0 0x7ff3fc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.374+0000 7ff401934700 1 -- 172.21.15.7:0/780976481 >> 172.21.15.7:0/780976481 conn(0x7ff3fc066e40 msgr2=0x7ff3fc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.375+0000 7ff401934700 1 -- 172.21.15.7:0/780976481 shutdown_connections 2023-03-03T08:43:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.375+0000 7ff401934700 1 -- 172.21.15.7:0/780976481 wait complete. 2023-03-03T08:43:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.376+0000 7ff401934700 1 Processor -- start 2023-03-03T08:43:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.376+0000 7ff401934700 1 -- start start 2023-03-03T08:43:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.376+0000 7ff401934700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3fc0c00e0 0x7ff3fc1533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.376+0000 7ff401934700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0caa30 0x7ff3fc0a9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.376+0000 7ff401934700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 0x7ff3fc1e9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff401934700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc1edca0 con 0x7ff3fc154ff0 2023-03-03T08:43:31.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff401934700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc0c6f70 con 0x7ff3fc0c00e0 2023-03-03T08:43:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff401934700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc1ed090 con 0x7ff3fc0caa30 2023-03-03T08:43:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3fb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 0x7ff3fc1e9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3fb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 0x7ff3fc1e9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51032/0 (socket says 172.21.15.7:51032) 2023-03-03T08:43:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3fb7fe700 1 -- 172.21.15.7:0/89201329 learned_addr learned my addr 172.21.15.7:0/89201329 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3fa7fc700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0caa30 0x7ff3fc0a9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3faffd700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3fc0c00e0 0x7ff3fc1533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3fb7fe700 1 -- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0caa30 msgr2=0x7ff3fc0a9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3fb7fe700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0caa30 0x7ff3fc0a9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3fb7fe700 1 -- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3fc0c00e0 msgr2=0x7ff3fc1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3fb7fe700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3fc0c00e0 0x7ff3fc1533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3fb7fe700 1 -- 172.21.15.7:0/89201329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f4009b30 con 0x7ff3fc154ff0 2023-03-03T08:43:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3fb7fe700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 0x7ff3fc1e9b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc06b270 tx=0x7ff3f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3f3fff700 1 -- 172.21.15.7:0/89201329 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f40099b0 con 0x7ff3fc154ff0 2023-03-03T08:43:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3f3fff700 1 -- 172.21.15.7:0/89201329 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f40092e0 con 0x7ff3fc154ff0 2023-03-03T08:43:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff401934700 1 -- 172.21.15.7:0/89201329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3fc154ad0 con 0x7ff3fc154ff0 2023-03-03T08:43:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff3f3fff700 1 -- 172.21.15.7:0/89201329 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f401c640 con 0x7ff3fc154ff0 2023-03-03T08:43:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.377+0000 7ff401934700 1 -- 172.21.15.7:0/89201329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3fc1547f0 con 0x7ff3fc154ff0 2023-03-03T08:43:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.378+0000 7ff401934700 1 -- 172.21.15.7:0/89201329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3fc0bf650 con 0x7ff3fc154ff0 2023-03-03T08:43:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.379+0000 7ff3f3fff700 1 -- 172.21.15.7:0/89201329 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff3f402c070 con 0x7ff3fc154ff0 2023-03-03T08:43:31.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.379+0000 7ff3f3fff700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3dc0682f0 0x7ff3dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.379+0000 7ff3f3fff700 1 -- 172.21.15.7:0/89201329 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff3f4087610 con 0x7ff3fc154ff0 2023-03-03T08:43:31.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.379+0000 7ff3faffd700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3dc0682f0 0x7ff3dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.380+0000 7ff3faffd700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3dc0682f0 0x7ff3dc06a7a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec000f60 tx=0x7ff3ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:31.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.383+0000 7ff3f3fff700 1 -- 172.21.15.7:0/89201329 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff3f40541c0 con 0x7ff3fc154ff0 2023-03-03T08:43:31.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.575+0000 7ff401934700 1 -- 172.21.15.7:0/89201329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff3fc074bb0 con 0x7ff3fc154ff0 2023-03-03T08:43:31.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.575+0000 7ff3f3fff700 1 -- 172.21.15.7:0/89201329 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff3f40577e0 con 0x7ff3fc154ff0 2023-03-03T08:43:31.577 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 8 2023-03-03T08:43:31.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 -- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3dc0682f0 msgr2=0x7ff3dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3dc0682f0 0x7ff3dc06a7a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec000f60 tx=0x7ff3ec009040 comp rx=0 tx=0).stop 2023-03-03T08:43:31.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 -- 172.21.15.7:0/89201329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 msgr2=0x7ff3fc1e9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 0x7ff3fc1e9b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc06b270 tx=0x7ff3f4004ad0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 -- 172.21.15.7:0/89201329 shutdown_connections 2023-03-03T08:43:31.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3dc0682f0 0x7ff3dc06a7a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3fc154ff0 0x7ff3fc1e9b20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3fc0c00e0 0x7ff3fc1533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 --2- 172.21.15.7:0/89201329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3fc0caa30 0x7ff3fc0a9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 -- 172.21.15.7:0/89201329 >> 172.21.15.7:0/89201329 conn(0x7ff3fc066e40 msgr2=0x7ff3fc06b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:31.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 -- 172.21.15.7:0/89201329 shutdown_connections 2023-03-03T08:43:31.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.577+0000 7ff401934700 1 -- 172.21.15.7:0/89201329 wait complete. 2023-03-03T08:43:31.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T08:43:31.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.713+0000 7f4c37ddd700 1 Processor -- start 2023-03-03T08:43:31.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.713+0000 7f4c37ddd700 1 -- start start 2023-03-03T08:43:31.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c37ddd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c30071990 0x7f4c30071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c37ddd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c30070360 con 0x7f4c3014b460 2023-03-03T08:43:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c37ddd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3006a5e0 con 0x7f4c30151c00 2023-03-03T08:43:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c37ddd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c301521a0 con 0x7f4c30071990 2023-03-03T08:43:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c35b79700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c30071990 0x7f4c30071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c35378700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4c3014b460 0x7f4c3006aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45018/0 (socket says 172.21.15.7:45018) 2023-03-03T08:43:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c35b79700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c30071990 0x7f4c30071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51036/0 (socket says 172.21.15.7:51036) 2023-03-03T08:43:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c35378700 1 -- 172.21.15.7:0/3348087744 learned_addr learned my addr 172.21.15.7:0/3348087744 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c35b79700 1 -- 172.21.15.7:0/3348087744 >> v1:172.21.15.143:6789/0 conn(0x7f4c30151c00 legacy=0x7f4c3006b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c34b77700 1 -- 172.21.15.7:0/3348087744 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523031299 0 0) 0x7f4c30070360 con 0x7f4c3014b460 2023-03-03T08:43:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c35b79700 1 -- 172.21.15.7:0/3348087744 >> v1:172.21.15.7:6789/0 conn(0x7f4c3014b460 legacy=0x7f4c3006aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.714+0000 7f4c35b79700 1 -- 172.21.15.7:0/3348087744 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c300c38b0 con 0x7f4c30071990 2023-03-03T08:43:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c35b79700 1 --2- 172.21.15.7:0/3348087744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30071990 0x7f4c30071d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009840 tx=0x7f4c20009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ecb91d0d29c0c3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c34b77700 1 -- 172.21.15.7:0/3348087744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20011830 con 0x7f4c30071990 2023-03-03T08:43:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c34b77700 1 -- 172.21.15.7:0/3348087744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c20011990 con 0x7f4c30071990 2023-03-03T08:43:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c34b77700 1 -- 172.21.15.7:0/3348087744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20011d20 con 0x7f4c30071990 2023-03-03T08:43:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c37ddd700 1 -- 172.21.15.7:0/3348087744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30071990 msgr2=0x7f4c30071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c37ddd700 1 --2- 172.21.15.7:0/3348087744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30071990 0x7f4c30071d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009840 tx=0x7f4c20009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c37ddd700 1 -- 172.21.15.7:0/3348087744 shutdown_connections 2023-03-03T08:43:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c37ddd700 1 --2- 172.21.15.7:0/3348087744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30071990 0x7f4c30071d60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c37ddd700 1 -- 172.21.15.7:0/3348087744 >> 172.21.15.7:0/3348087744 conn(0x7f4c300a94d0 msgr2=0x7f4c300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:31.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c37ddd700 1 -- 172.21.15.7:0/3348087744 shutdown_connections 2023-03-03T08:43:31.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.715+0000 7f4c37ddd700 1 -- 172.21.15.7:0/3348087744 wait complete. 2023-03-03T08:43:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.716+0000 7f4c37ddd700 1 Processor -- start 2023-03-03T08:43:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.716+0000 7f4c37ddd700 1 -- start start 2023-03-03T08:43:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c37ddd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c30071990 0x7f4c301ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c37ddd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c30151c00 0x7f4c301ea200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c37ddd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30146a10 0x7f4c30148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c37ddd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c301ed770 con 0x7f4c30146a10 2023-03-03T08:43:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c37ddd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3014f750 con 0x7f4c30071990 2023-03-03T08:43:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c37ddd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3014fa30 con 0x7f4c30151c00 2023-03-03T08:43:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c3637a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30146a10 0x7f4c30148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c3637a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30146a10 0x7f4c30148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51046/0 (socket says 172.21.15.7:51046) 2023-03-03T08:43:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c3637a700 1 -- 172.21.15.7:0/4120779679 learned_addr learned my addr 172.21.15.7:0/4120779679 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c35378700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c30151c00 0x7f4c301ea200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.717+0000 7f4c35b79700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c30071990 0x7f4c301ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c3637a700 1 -- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c30151c00 msgr2=0x7f4c301ea200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c3637a700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c30151c00 0x7f4c301ea200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c3637a700 1 -- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c30071990 msgr2=0x7f4c301ee1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c3637a700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c30071990 0x7f4c301ee1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c3637a700 1 -- 172.21.15.7:0/4120779679 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c200094a0 con 0x7f4c30146a10 2023-03-03T08:43:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c35378700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c30151c00 0x7f4c301ea200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c35b79700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c30071990 0x7f4c301ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c3637a700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30146a10 0x7f4c30148340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4c2800cb50 tx=0x7f4c2800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c267fc700 1 -- 172.21.15.7:0/4120779679 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c28003c00 con 0x7f4c30146a10 2023-03-03T08:43:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c267fc700 1 -- 172.21.15.7:0/4120779679 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c28003d60 con 0x7f4c30146a10 2023-03-03T08:43:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c267fc700 1 -- 172.21.15.7:0/4120779679 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2800ab70 con 0x7f4c30146a10 2023-03-03T08:43:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c37ddd700 1 -- 172.21.15.7:0/4120779679 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c300c3b50 con 0x7f4c30146a10 2023-03-03T08:43:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.718+0000 7f4c37ddd700 1 -- 172.21.15.7:0/4120779679 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3006a2b0 con 0x7f4c30146a10 2023-03-03T08:43:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.719+0000 7f4c37ddd700 1 -- 172.21.15.7:0/4120779679 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c30074bb0 con 0x7f4c30146a10 2023-03-03T08:43:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.719+0000 7f4c267fc700 1 -- 172.21.15.7:0/4120779679 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4c28010050 con 0x7f4c30146a10 2023-03-03T08:43:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.719+0000 7f4c267fc700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c140682f0 0x7f4c1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.720+0000 7f4c267fc700 1 -- 172.21.15.7:0/4120779679 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4c28085820 con 0x7f4c30146a10 2023-03-03T08:43:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.720+0000 7f4c35b79700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c140682f0 0x7f4c1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:31.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.720+0000 7f4c35b79700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c140682f0 0x7f4c1406a7a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4c20000c00 tx=0x7f4c2001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:31.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.724+0000 7f4c267fc700 1 -- 172.21.15.7:0/4120779679 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c280523d0 con 0x7f4c30146a10 2023-03-03T08:43:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.921+0000 7f4c37ddd700 1 -- 172.21.15.7:0/4120779679 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4c30072220 con 0x7f4c30146a10 2023-03-03T08:43:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.921+0000 7f4c267fc700 1 -- 172.21.15.7:0/4120779679 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7f4c280559f0 con 0x7f4c30146a10 2023-03-03T08:43:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:31.923 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2023-03-03T08:43:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-03-03T08:43:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 1, 2023-03-03T08:43:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": ".mgr" 2023-03-03T08:43:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-03-03T08:43:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-03-03T08:43:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 2, 2023-03-03T08:43:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": "rbd" 2023-03-03T08:43:31.924 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-03-03T08:43:31.925 INFO:teuthology.orchestra.run.smithi007.stdout:] 2023-03-03T08:43:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.923+0000 7f4c37ddd700 1 -- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c140682f0 msgr2=0x7f4c1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.923+0000 7f4c37ddd700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c140682f0 0x7f4c1406a7a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4c20000c00 tx=0x7f4c2001c040 comp rx=0 tx=0).stop 2023-03-03T08:43:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.924+0000 7f4c37ddd700 1 -- 172.21.15.7:0/4120779679 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30146a10 msgr2=0x7f4c30148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.924+0000 7f4c37ddd700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30146a10 0x7f4c30148340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4c2800cb50 tx=0x7f4c2800cf10 comp rx=0 tx=0).stop 2023-03-03T08:43:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.924+0000 7f4c37ddd700 1 -- 172.21.15.7:0/4120779679 shutdown_connections 2023-03-03T08:43:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.924+0000 7f4c37ddd700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c140682f0 0x7f4c1406a7a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.924+0000 7f4c37ddd700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c30146a10 0x7f4c30148340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.924+0000 7f4c37ddd700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c30071990 0x7f4c301ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.924+0000 7f4c37ddd700 1 --2- 172.21.15.7:0/4120779679 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c30151c00 0x7f4c301ea200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:31.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.924+0000 7f4c37ddd700 1 -- 172.21.15.7:0/4120779679 >> 172.21.15.7:0/4120779679 conn(0x7f4c300a94d0 msgr2=0x7f4c300c6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:31.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.924+0000 7f4c37ddd700 1 -- 172.21.15.7:0/4120779679 shutdown_connections 2023-03-03T08:43:31.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:31.924+0000 7f4c37ddd700 1 -- 172.21.15.7:0/4120779679 wait complete. 2023-03-03T08:43:31.937 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'a' 2023-03-03T08:43:31.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_metadata --pg_num_min 64 2023-03-03T08:43:32.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.065+0000 7f80bb381700 1 Processor -- start 2023-03-03T08:43:32.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.065+0000 7f80bb381700 1 -- start start 2023-03-03T08:43:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.066+0000 7f80bb381700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f80b40c5cb0 0x7f80b406a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:32.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.066+0000 7f80bb381700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b41558f0 con 0x7f80b40a6370 2023-03-03T08:43:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.066+0000 7f80bb381700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b40a6180 con 0x7f80b406a2b0 2023-03-03T08:43:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.066+0000 7f80bb381700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b406bab0 con 0x7f80b40c5cb0 2023-03-03T08:43:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.066+0000 7f80b891c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f80b40a6370 0x7f80b415abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45030/0 (socket says 172.21.15.7:45030) 2023-03-03T08:43:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.066+0000 7f80b891c700 1 -- 172.21.15.7:0/2210691361 learned_addr learned my addr 172.21.15.7:0/2210691361 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.066+0000 7f80b991e700 1 --2- 172.21.15.7:0/2210691361 >> v2:172.21.15.143:3300/0 conn(0x7f80b40c5cb0 0x7f80b406a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.066+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214779099 0 0) 0x7f80b41558f0 con 0x7f80b40a6370 2023-03-03T08:43:32.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.066+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a00035b0 con 0x7f80b40a6370 2023-03-03T08:43:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 410758728 0 0) 0x7f80a00035b0 con 0x7f80b40a6370 2023-03-03T08:43:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 >> v2:172.21.15.143:3300/0 conn(0x7f80b40c5cb0 msgr2=0x7f80b406a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 --2- 172.21.15.7:0/2210691361 >> v2:172.21.15.143:3300/0 conn(0x7f80b40c5cb0 0x7f80b406a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 >> v1:172.21.15.143:6789/0 conn(0x7f80b406a2b0 legacy=0x7f80b4155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b40a5ea0 con 0x7f80b40a6370 2023-03-03T08:43:32.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f80b0001670 con 0x7f80b40a6370 2023-03-03T08:43:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 >> v1:172.21.15.7:6789/0 conn(0x7f80b40a6370 legacy=0x7f80b415abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a0003a70 0x7f80a0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a0004850 0x7f80a0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a0005200 0x7f80a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a000c0e0 con 0x7f80a0003a70 2023-03-03T08:43:32.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a000c250 con 0x7f80a0005200 2023-03-03T08:43:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a000c410 con 0x7f80a0004850 2023-03-03T08:43:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80b911d700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a0003a70 0x7f80a0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80b891c700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a0004850 0x7f80a0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80b991e700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a0005200 0x7f80a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80b911d700 1 -- 172.21.15.7:0/2210691361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a0004850 msgr2=0x7f80a0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80b911d700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a0004850 0x7f80a0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80b911d700 1 -- 172.21.15.7:0/2210691361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a0005200 msgr2=0x7f80a000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:32.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80b911d700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a0005200 0x7f80a000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80b911d700 1 -- 172.21.15.7:0/2210691361 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a00035b0 con 0x7f80a0003a70 2023-03-03T08:43:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80b991e700 1 -- 172.21.15.7:0/2210691361 reap_dead start 2023-03-03T08:43:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80b911d700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a0003a70 0x7f80a0003e40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f80a4007fd0 tx=0x7f80a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460f306d062d14f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a40052d0 con 0x7f80a0003a70 2023-03-03T08:43:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a4005430 con 0x7f80a0003a70 2023-03-03T08:43:32.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.067+0000 7f80abfff700 1 -- 172.21.15.7:0/2210691361 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a40038c0 con 0x7f80a0003a70 2023-03-03T08:43:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.068+0000 7f80bb381700 1 -- 172.21.15.7:0/2210691361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a0003a70 msgr2=0x7f80a0003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.068+0000 7f80bb381700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a0003a70 0x7f80a0003e40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f80a4007fd0 tx=0x7f80a0005880 comp rx=0 tx=0).stop 2023-03-03T08:43:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.068+0000 7f80bb381700 1 -- 172.21.15.7:0/2210691361 shutdown_connections 2023-03-03T08:43:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.068+0000 7f80bb381700 1 --2- 172.21.15.7:0/2210691361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a0003a70 0x7f80a0003e40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.068+0000 7f80bb381700 1 -- 172.21.15.7:0/2210691361 >> 172.21.15.7:0/2210691361 conn(0x7f80b40a9430 msgr2=0x7f80b40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.068+0000 7f80bb381700 1 -- 172.21.15.7:0/2210691361 shutdown_connections 2023-03-03T08:43:32.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.068+0000 7f80bb381700 1 -- 172.21.15.7:0/2210691361 wait complete. 2023-03-03T08:43:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.069+0000 7f80bb381700 1 Processor -- start 2023-03-03T08:43:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.069+0000 7f80bb381700 1 -- start start 2023-03-03T08:43:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80bb381700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a0003a70 0x7f80b40715e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80bb381700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b40c0910 0x7f80b40c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80bb381700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b41ed250 0x7f80b4154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80bb381700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b414cb30 con 0x7f80b41ed250 2023-03-03T08:43:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80bb381700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40aaec0 con 0x7f80a0003a70 2023-03-03T08:43:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80bb381700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40ab690 con 0x7f80b40c0910 2023-03-03T08:43:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80b991e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b41ed250 0x7f80b4154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80b991e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b41ed250 0x7f80b4154d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51064/0 (socket says 172.21.15.7:51064) 2023-03-03T08:43:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80b991e700 1 -- 172.21.15.7:0/3272135876 learned_addr learned my addr 172.21.15.7:0/3272135876 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80b891c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b40c0910 0x7f80b40c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80b911d700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a0003a70 0x7f80b40715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80b991e700 1 -- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b40c0910 msgr2=0x7f80b40c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80b991e700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b40c0910 0x7f80b40c00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80b991e700 1 -- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a0003a70 msgr2=0x7f80b40715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80b991e700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a0003a70 0x7f80b40715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.070+0000 7f80b991e700 1 -- 172.21.15.7:0/3272135876 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b4153160 con 0x7f80b41ed250 2023-03-03T08:43:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.071+0000 7f80b991e700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b41ed250 0x7f80b4154d00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f80b406c080 tx=0x7f80ac011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.071+0000 7f80a9ffb700 1 -- 172.21.15.7:0/3272135876 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac00e040 con 0x7f80b41ed250 2023-03-03T08:43:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.071+0000 7f80a9ffb700 1 -- 172.21.15.7:0/3272135876 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac00f040 con 0x7f80b41ed250 2023-03-03T08:43:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.071+0000 7f80bb381700 1 -- 172.21.15.7:0/3272135876 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b4152e80 con 0x7f80b41ed250 2023-03-03T08:43:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.071+0000 7f80a9ffb700 1 -- 172.21.15.7:0/3272135876 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac00fc70 con 0x7f80b41ed250 2023-03-03T08:43:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.071+0000 7f80bb381700 1 -- 172.21.15.7:0/3272135876 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b40c1500 con 0x7f80b41ed250 2023-03-03T08:43:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.072+0000 7f80bb381700 1 -- 172.21.15.7:0/3272135876 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b40a85d0 con 0x7f80b41ed250 2023-03-03T08:43:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.072+0000 7f80a9ffb700 1 -- 172.21.15.7:0/3272135876 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f80ac022460 con 0x7f80b41ed250 2023-03-03T08:43:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.073+0000 7f80a9ffb700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8098068320 0x7f809806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.073+0000 7f80a9ffb700 1 -- 172.21.15.7:0/3272135876 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f80ac085f60 con 0x7f80b41ed250 2023-03-03T08:43:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.073+0000 7f80b911d700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8098068320 0x7f809806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.073+0000 7f80b911d700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8098068320 0x7f809806a7d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80a4000c00 tx=0x7f80a4005000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.077+0000 7f80a9ffb700 1 -- 172.21.15.7:0/3272135876 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80ac052ab0 con 0x7f80b41ed250 2023-03-03T08:43:32.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:32.274+0000 7f80bb381700 1 -- 172.21.15.7:0/3272135876 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f80b4066d20 con 0x7f80b41ed250 2023-03-03T08:43:33.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.088+0000 7f80a9ffb700 1 -- 172.21.15.7:0/3272135876 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64}]=0 pool 'a_metadata' created v20) v1 ==== 128+0+0 (secure 0 0 0) 0x7f80ac0560d0 con 0x7f80b41ed250 2023-03-03T08:43:33.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.092+0000 7f80bb381700 1 -- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8098068320 msgr2=0x7f809806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:33.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.092+0000 7f80bb381700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8098068320 0x7f809806a7d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80a4000c00 tx=0x7f80a4005000 comp rx=0 tx=0).stop 2023-03-03T08:43:33.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.092+0000 7f80bb381700 1 -- 172.21.15.7:0/3272135876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b41ed250 msgr2=0x7f80b4154d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:33.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.092+0000 7f80bb381700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b41ed250 0x7f80b4154d00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f80b406c080 tx=0x7f80ac011ed0 comp rx=0 tx=0).stop 2023-03-03T08:43:33.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.092+0000 7f80bb381700 1 -- 172.21.15.7:0/3272135876 shutdown_connections 2023-03-03T08:43:33.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.092+0000 7f80bb381700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8098068320 0x7f809806a7d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:33.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.092+0000 7f80bb381700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b41ed250 0x7f80b4154d00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:33.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.092+0000 7f80bb381700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a0003a70 0x7f80b40715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:33.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.092+0000 7f80bb381700 1 --2- 172.21.15.7:0/3272135876 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b40c0910 0x7f80b40c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:33.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.092+0000 7f80bb381700 1 -- 172.21.15.7:0/3272135876 >> 172.21.15.7:0/3272135876 conn(0x7f80b40a9430 msgr2=0x7f80b40c2440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:33.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.093+0000 7f80bb381700 1 -- 172.21.15.7:0/3272135876 shutdown_connections 2023-03-03T08:43:33.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.094+0000 7f80bb381700 1 -- 172.21.15.7:0/3272135876 wait complete. 2023-03-03T08:43:33.096 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'a_metadata' created 2023-03-03T08:43:33.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T08:43:33.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.249+0000 7f9e038f5700 1 Processor -- start 2023-03-03T08:43:33.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.249+0000 7f9e038f5700 1 -- start start 2023-03-03T08:43:33.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e038f5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9dfc13a0f0 0x7f9dfc0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:33.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e038f5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc154560 con 0x7f9dfc1586c0 2023-03-03T08:43:33.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e038f5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc157be0 con 0x7f9dfc0c1da0 2023-03-03T08:43:33.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e038f5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dfc154840 con 0x7f9dfc13a0f0 2023-03-03T08:43:33.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e01691700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9dfc13a0f0 0x7f9dfc0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:33.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e00e90700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9dfc1586c0 0x7f9dfc06d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45050/0 (socket says 172.21.15.7:45050) 2023-03-03T08:43:33.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e01691700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9dfc13a0f0 0x7f9dfc0c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51068/0 (socket says 172.21.15.7:51068) 2023-03-03T08:43:33.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e00e90700 1 -- 172.21.15.7:0/1257375880 learned_addr learned my addr 172.21.15.7:0/1257375880 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:33.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e01691700 1 -- 172.21.15.7:0/1257375880 >> v1:172.21.15.143:6789/0 conn(0x7f9dfc0c1da0 legacy=0x7f9dfc15edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:33.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e01691700 1 -- 172.21.15.7:0/1257375880 >> v1:172.21.15.7:6789/0 conn(0x7f9dfc1586c0 legacy=0x7f9dfc06d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:33.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e01691700 1 -- 172.21.15.7:0/1257375880 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc157ec0 con 0x7f9dfc13a0f0 2023-03-03T08:43:33.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9df3fff700 1 -- 172.21.15.7:0/1257375880 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242887497 0 0) 0x7f9dfc154560 con 0x7f9dfc1586c0 2023-03-03T08:43:33.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9e01691700 1 --2- 172.21.15.7:0/1257375880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc13a0f0 0x7f9dfc0c07d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9dec009840 tx=0x7f9dec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518a09d1b4ee654b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:33.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9df3fff700 1 -- 172.21.15.7:0/1257375880 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec004230 con 0x7f9dfc13a0f0 2023-03-03T08:43:33.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9df3fff700 1 -- 172.21.15.7:0/1257375880 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec004390 con 0x7f9dfc13a0f0 2023-03-03T08:43:33.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.250+0000 7f9df3fff700 1 -- 172.21.15.7:0/1257375880 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec0036a0 con 0x7f9dfc13a0f0 2023-03-03T08:43:33.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.251+0000 7f9e038f5700 1 -- 172.21.15.7:0/1257375880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc13a0f0 msgr2=0x7f9dfc0c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:33.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.251+0000 7f9e038f5700 1 --2- 172.21.15.7:0/1257375880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc13a0f0 0x7f9dfc0c07d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9dec009840 tx=0x7f9dec009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:33.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.251+0000 7f9e038f5700 1 -- 172.21.15.7:0/1257375880 shutdown_connections 2023-03-03T08:43:33.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.251+0000 7f9e038f5700 1 --2- 172.21.15.7:0/1257375880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc13a0f0 0x7f9dfc0c07d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:33.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.251+0000 7f9e038f5700 1 -- 172.21.15.7:0/1257375880 >> 172.21.15.7:0/1257375880 conn(0x7f9dfc06c110 msgr2=0x7f9dfc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:33.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.251+0000 7f9e038f5700 1 -- 172.21.15.7:0/1257375880 shutdown_connections 2023-03-03T08:43:33.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.251+0000 7f9e038f5700 1 -- 172.21.15.7:0/1257375880 wait complete. 2023-03-03T08:43:33.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.252+0000 7f9e038f5700 1 Processor -- start 2023-03-03T08:43:33.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.252+0000 7f9e038f5700 1 -- start start 2023-03-03T08:43:33.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e038f5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dfc0c1da0 0x7f9dfc0ac410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:33.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e038f5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dfc13a0f0 0x7f9dfc151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:33.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e038f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc1586c0 0x7f9dfc151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:33.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e038f5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc0aab20 con 0x7f9dfc1586c0 2023-03-03T08:43:33.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e038f5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc151f90 con 0x7f9dfc0c1da0 2023-03-03T08:43:33.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e038f5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc0aad70 con 0x7f9dfc13a0f0 2023-03-03T08:43:33.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01e92700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc1586c0 0x7f9dfc151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:33.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01e92700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc1586c0 0x7f9dfc151a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51078/0 (socket says 172.21.15.7:51078) 2023-03-03T08:43:33.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01e92700 1 -- 172.21.15.7:0/1955843758 learned_addr learned my addr 172.21.15.7:0/1955843758 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:33.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01691700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dfc0c1da0 0x7f9dfc0ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:33.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e00e90700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dfc13a0f0 0x7f9dfc151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:33.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01e92700 1 -- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dfc13a0f0 msgr2=0x7f9dfc151510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:33.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01e92700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dfc13a0f0 0x7f9dfc151510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:33.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01e92700 1 -- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dfc0c1da0 msgr2=0x7f9dfc0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:33.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01e92700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dfc0c1da0 0x7f9dfc0ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:33.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01e92700 1 -- 172.21.15.7:0/1955843758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec0094a0 con 0x7f9dfc1586c0 2023-03-03T08:43:33.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e00e90700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dfc13a0f0 0x7f9dfc151510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:33.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01691700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dfc0c1da0 0x7f9dfc0ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:33.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.253+0000 7f9e01e92700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc1586c0 0x7f9dfc151a50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9df400c7d0 tx=0x7f9df400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:33.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.254+0000 7f9df1ffb700 1 -- 172.21.15.7:0/1955843758 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4003e10 con 0x7f9dfc1586c0 2023-03-03T08:43:33.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.254+0000 7f9df1ffb700 1 -- 172.21.15.7:0/1955843758 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df4005070 con 0x7f9dfc1586c0 2023-03-03T08:43:33.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.254+0000 7f9e038f5700 1 -- 172.21.15.7:0/1955843758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc06e1a0 con 0x7f9dfc1586c0 2023-03-03T08:43:33.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.254+0000 7f9df1ffb700 1 -- 172.21.15.7:0/1955843758 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4005420 con 0x7f9dfc1586c0 2023-03-03T08:43:33.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.254+0000 7f9e038f5700 1 -- 172.21.15.7:0/1955843758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dfc1581a0 con 0x7f9dfc1586c0 2023-03-03T08:43:33.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.255+0000 7f9e038f5700 1 -- 172.21.15.7:0/1955843758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dfc06e330 con 0x7f9dfc1586c0 2023-03-03T08:43:33.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.255+0000 7f9df1ffb700 1 -- 172.21.15.7:0/1955843758 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9df4019030 con 0x7f9dfc1586c0 2023-03-03T08:43:33.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.256+0000 7f9df1ffb700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9de00682f0 0x7f9de006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:33.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.256+0000 7f9df1ffb700 1 -- 172.21.15.7:0/1955843758 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f9df4085060 con 0x7f9dfc1586c0 2023-03-03T08:43:33.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.256+0000 7f9e01691700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9de00682f0 0x7f9de006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:33.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.257+0000 7f9e01691700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9de00682f0 0x7f9de006a7a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9dec000c00 tx=0x7f9dec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:33.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.260+0000 7f9df1ffb700 1 -- 172.21.15.7:0/1955843758 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9df4051aa0 con 0x7f9dfc1586c0 2023-03-03T08:43:33.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:33.453+0000 7f9e038f5700 1 -- 172.21.15.7:0/1955843758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9dfc074bb0 con 0x7f9dfc1586c0 2023-03-03T08:43:34.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.095+0000 7f9df1ffb700 1 -- 172.21.15.7:0/1955843758 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'a_data' created v21) v1 ==== 160+0+0 (secure 0 0 0) 0x7f9df4017020 con 0x7f9dfc1586c0 2023-03-03T08:43:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.103+0000 7f9e038f5700 1 -- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9de00682f0 msgr2=0x7f9de006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.103+0000 7f9e038f5700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9de00682f0 0x7f9de006a7a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9dec000c00 tx=0x7f9dec008030 comp rx=0 tx=0).stop 2023-03-03T08:43:34.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.103+0000 7f9e038f5700 1 -- 172.21.15.7:0/1955843758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc1586c0 msgr2=0x7f9dfc151a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.103+0000 7f9e038f5700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc1586c0 0x7f9dfc151a50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9df400c7d0 tx=0x7f9df400cb90 comp rx=0 tx=0).stop 2023-03-03T08:43:34.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.104+0000 7f9e038f5700 1 -- 172.21.15.7:0/1955843758 shutdown_connections 2023-03-03T08:43:34.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.104+0000 7f9e038f5700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9de00682f0 0x7f9de006a7a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.104+0000 7f9e038f5700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dfc1586c0 0x7f9dfc151a50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.104+0000 7f9e038f5700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dfc0c1da0 0x7f9dfc0ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.104+0000 7f9e038f5700 1 --2- 172.21.15.7:0/1955843758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dfc13a0f0 0x7f9dfc151510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.104+0000 7f9e038f5700 1 -- 172.21.15.7:0/1955843758 >> 172.21.15.7:0/1955843758 conn(0x7f9dfc06c110 msgr2=0x7f9dfc0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.107+0000 7f9e038f5700 1 -- 172.21.15.7:0/1955843758 shutdown_connections 2023-03-03T08:43:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.108+0000 7f9e038f5700 1 -- 172.21.15.7:0/1955843758 wait complete. 2023-03-03T08:43:34.115 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'a_data' created 2023-03-03T08:43:34.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new a a_metadata a_data 2023-03-03T08:43:34.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.263+0000 7f1f0bf9f700 1 Processor -- start 2023-03-03T08:43:34.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.263+0000 7f1f0bf9f700 1 -- start start 2023-03-03T08:43:34.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0bf9f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f040aad90 0x7f1f0406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0bf9f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f0406bfe0 0x7f1f0406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0bf9f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0406eda0 con 0x7f1f0406b920 2023-03-03T08:43:34.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0bf9f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f0406f080 con 0x7f1f0406bfe0 2023-03-03T08:43:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0bf9f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f041584b0 con 0x7f1f040aad90 2023-03-03T08:43:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0a53c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f0406bfe0 0x7f1f0406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0953a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1f0406b920 0x7f1f0406a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45066/0 (socket says 172.21.15.7:45066) 2023-03-03T08:43:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0a53c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f0406bfe0 0x7f1f0406a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51088/0 (socket says 172.21.15.7:51088) 2023-03-03T08:43:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0953a700 1 -- 172.21.15.7:0/866600362 learned_addr learned my addr 172.21.15.7:0/866600362 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f09d3b700 1 --2- 172.21.15.7:0/866600362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f040aad90 0x7f1f0406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0a53c700 1 -- 172.21.15.7:0/866600362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f040aad90 msgr2=0x7f1f0406e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0a53c700 1 --2- 172.21.15.7:0/866600362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f040aad90 0x7f1f0406e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0a53c700 1 -- 172.21.15.7:0/866600362 >> v1:172.21.15.7:6789/0 conn(0x7f1f0406b920 legacy=0x7f1f0406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0a53c700 1 -- 172.21.15.7:0/866600362 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f041586a0 con 0x7f1f0406bfe0 2023-03-03T08:43:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f08d39700 1 -- 172.21.15.7:0/866600362 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691064583 0 0) 0x7f1f0406eda0 con 0x7f1f0406b920 2023-03-03T08:43:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f09d3b700 1 --2- 172.21.15.7:0/866600362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f040aad90 0x7f1f0406e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f0a53c700 1 --2- 172.21.15.7:0/866600362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406bfe0 0x7f1f0406a9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009d90 tx=0x7f1efc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959ea6e4ef45d236 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f08d39700 1 -- 172.21.15.7:0/866600362 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc00e070 con 0x7f1f0406bfe0 2023-03-03T08:43:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f08d39700 1 -- 172.21.15.7:0/866600362 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1efc00bbd0 con 0x7f1f0406bfe0 2023-03-03T08:43:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.264+0000 7f1f08d39700 1 -- 172.21.15.7:0/866600362 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc012430 con 0x7f1f0406bfe0 2023-03-03T08:43:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.265+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/866600362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406bfe0 msgr2=0x7f1f0406a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.265+0000 7f1f0bf9f700 1 --2- 172.21.15.7:0/866600362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406bfe0 0x7f1f0406a9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009d90 tx=0x7f1efc00b330 comp rx=0 tx=0).stop 2023-03-03T08:43:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.265+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/866600362 shutdown_connections 2023-03-03T08:43:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.265+0000 7f1f0bf9f700 1 --2- 172.21.15.7:0/866600362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406bfe0 0x7f1f0406a9c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.265+0000 7f1f0bf9f700 1 --2- 172.21.15.7:0/866600362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f040aad90 0x7f1f0406e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.265+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/866600362 >> 172.21.15.7:0/866600362 conn(0x7f1f040736f0 msgr2=0x7f1f04073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.265+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/866600362 shutdown_connections 2023-03-03T08:43:34.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.265+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/866600362 wait complete. 2023-03-03T08:43:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.267+0000 7f1f0bf9f700 1 Processor -- start 2023-03-03T08:43:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.267+0000 7f1f0bf9f700 1 -- start start 2023-03-03T08:43:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.267+0000 7f1f0bf9f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406b920 0x7f1f040c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f0bf9f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f0406bfe0 0x7f1f040ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f0bf9f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f040aad90 0x7f1f0414b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f0bf9f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0414c360 con 0x7f1f0406b920 2023-03-03T08:43:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f0bf9f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0414c5b0 con 0x7f1f040aad90 2023-03-03T08:43:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f0bf9f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040c7d90 con 0x7f1f0406bfe0 2023-03-03T08:43:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f09d3b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406b920 0x7f1f040c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f09d3b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406b920 0x7f1f040c4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51090/0 (socket says 172.21.15.7:51090) 2023-03-03T08:43:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f09d3b700 1 -- 172.21.15.7:0/354051170 learned_addr learned my addr 172.21.15.7:0/354051170 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f0953a700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f0406bfe0 0x7f1f040ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f0a53c700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f040aad90 0x7f1f0414b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f09d3b700 1 -- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f0406bfe0 msgr2=0x7f1f040ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f09d3b700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f0406bfe0 0x7f1f040ab810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f09d3b700 1 -- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f040aad90 msgr2=0x7f1f0414b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f09d3b700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f040aad90 0x7f1f0414b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f09d3b700 1 -- 172.21.15.7:0/354051170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc009a70 con 0x7f1f0406b920 2023-03-03T08:43:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f0953a700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f0406bfe0 0x7f1f040ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f09d3b700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406b920 0x7f1f040c4ec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000e810 tx=0x7f1f0000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1ef67fc700 1 -- 172.21.15.7:0/354051170 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0000c020 con 0x7f1f0406b920 2023-03-03T08:43:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1ef67fc700 1 -- 172.21.15.7:0/354051170 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f00004250 con 0x7f1f0406b920 2023-03-03T08:43:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/354051170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f04071cd0 con 0x7f1f0406b920 2023-03-03T08:43:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1ef67fc700 1 -- 172.21.15.7:0/354051170 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f000036a0 con 0x7f1f0406b920 2023-03-03T08:43:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.268+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/354051170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f04158980 con 0x7f1f0406b920 2023-03-03T08:43:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.269+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/354051170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f04071270 con 0x7f1f0406b920 2023-03-03T08:43:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.270+0000 7f1ef67fc700 1 -- 172.21.15.7:0/354051170 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f00003800 con 0x7f1f0406b920 2023-03-03T08:43:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.270+0000 7f1ef67fc700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 0x7f1ee806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.270+0000 7f1ef67fc700 1 -- 172.21.15.7:0/354051170 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5952+0+0 (secure 0 0 0) 0x7f1f00085a20 con 0x7f1f0406b920 2023-03-03T08:43:34.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.271+0000 7f1f0953a700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 0x7f1ee806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.271+0000 7f1f0953a700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 0x7f1ee806a7a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8007fd0 tx=0x7f1ef8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:34.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.275+0000 7f1ef67fc700 1 -- 172.21.15.7:0/354051170 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f00052270 con 0x7f1f0406b920 2023-03-03T08:43:34.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.515+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/354051170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"} v 0) v1 -- 0x7f1f040c86c0 con 0x7f1f0406b920 2023-03-03T08:43:34.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.520+0000 7f1ef67fc700 1 -- 172.21.15.7:0/354051170 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 157+0+0 (secure 0 0 0) 0x7f1f0000f040 con 0x7f1f0406b920 2023-03-03T08:43:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.524+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 msgr2=0x7f1ee806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.524+0000 7f1f0bf9f700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 0x7f1ee806a7a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8007fd0 tx=0x7f1ef8009040 comp rx=0 tx=0).stop 2023-03-03T08:43:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.524+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/354051170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406b920 msgr2=0x7f1f040c4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.524+0000 7f1f0bf9f700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406b920 0x7f1f040c4ec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000e810 tx=0x7f1f0000ebd0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.526+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/354051170 shutdown_connections 2023-03-03T08:43:34.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.526+0000 7f1f0bf9f700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f040aad90 0x7f1f0414b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.526+0000 7f1f0bf9f700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 0x7f1ee806a7a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.526+0000 7f1f0bf9f700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0406b920 0x7f1f040c4ec0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.526+0000 7f1f0bf9f700 1 --2- 172.21.15.7:0/354051170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f0406bfe0 0x7f1f040ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.526+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/354051170 >> 172.21.15.7:0/354051170 conn(0x7f1f040736f0 msgr2=0x7f1f04073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.527+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/354051170 shutdown_connections 2023-03-03T08:43:34.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.527+0000 7f1f0bf9f700 1 -- 172.21.15.7:0/354051170 wait complete. 2023-03-03T08:43:34.544 INFO:teuthology.orchestra.run.smithi007.stderr:new fs with metadata pool 3 and data pool 4 2023-03-03T08:43:34.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:34.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.682+0000 7fb002a87700 1 Processor -- start 2023-03-03T08:43:34.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.682+0000 7fb002a87700 1 -- start start 2023-03-03T08:43:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7fb002a87700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faffc0c3ac0 0x7faffc1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7fb002a87700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc154ee0 con 0x7faffc0723f0 2023-03-03T08:43:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7fb002a87700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc0c3f30 con 0x7faffc151960 2023-03-03T08:43:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7fb002a87700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc1545f0 con 0x7faffc0c3ac0 2023-03-03T08:43:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7faffbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faffc0c3ac0 0x7faffc1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7fb001024700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faffc0723f0 0x7faffc157f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45096/0 (socket says 172.21.15.7:45096) 2023-03-03T08:43:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7fb001024700 1 -- 172.21.15.7:0/3719804038 learned_addr learned my addr 172.21.15.7:0/3719804038 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7faffbfff700 1 -- 172.21.15.7:0/3719804038 >> v1:172.21.15.143:6789/0 conn(0x7faffc151960 legacy=0x7faffc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7faffbfff700 1 -- 172.21.15.7:0/3719804038 >> v1:172.21.15.7:6789/0 conn(0x7faffc0723f0 legacy=0x7faffc157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7faffbfff700 1 -- 172.21.15.7:0/3719804038 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc155370 con 0x7faffc0c3ac0 2023-03-03T08:43:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7faffb7fe700 1 -- 172.21.15.7:0/3719804038 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172826853 0 0) 0x7faffc154ee0 con 0x7faffc0723f0 2023-03-03T08:43:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7faffbfff700 1 --2- 172.21.15.7:0/3719804038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 0x7faffc1579f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009840 tx=0x7fafe8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e33929d1f00cad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7faffb7fe700 1 -- 172.21.15.7:0/3719804038 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8004230 con 0x7faffc0c3ac0 2023-03-03T08:43:34.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7faffb7fe700 1 -- 172.21.15.7:0/3719804038 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe8004390 con 0x7faffc0c3ac0 2023-03-03T08:43:34.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.683+0000 7faffb7fe700 1 -- 172.21.15.7:0/3719804038 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe80036a0 con 0x7faffc0c3ac0 2023-03-03T08:43:34.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.684+0000 7fb002a87700 1 -- 172.21.15.7:0/3719804038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 msgr2=0x7faffc1579f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.684+0000 7fb002a87700 1 --2- 172.21.15.7:0/3719804038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 0x7faffc1579f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009840 tx=0x7fafe8009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:34.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.684+0000 7fb002a87700 1 -- 172.21.15.7:0/3719804038 shutdown_connections 2023-03-03T08:43:34.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.684+0000 7fb002a87700 1 --2- 172.21.15.7:0/3719804038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 0x7faffc1579f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.684+0000 7fb002a87700 1 -- 172.21.15.7:0/3719804038 >> 172.21.15.7:0/3719804038 conn(0x7faffc06c060 msgr2=0x7faffc06c460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:34.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.684+0000 7fb002a87700 1 -- 172.21.15.7:0/3719804038 shutdown_connections 2023-03-03T08:43:34.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.684+0000 7fb002a87700 1 -- 172.21.15.7:0/3719804038 wait complete. 2023-03-03T08:43:34.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.685+0000 7fb002a87700 1 Processor -- start 2023-03-03T08:43:34.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.685+0000 7fb002a87700 1 -- start start 2023-03-03T08:43:34.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7fb002a87700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faffc0723f0 0x7faffc0aa770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7fb002a87700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 0x7faffc147e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7fb002a87700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faffc151960 0x7faffc1ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7fb002a87700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc157040 con 0x7faffc0c3ac0 2023-03-03T08:43:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7fb002a87700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc1e9f00 con 0x7faffc0723f0 2023-03-03T08:43:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7fb002a87700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc1483c0 con 0x7faffc151960 2023-03-03T08:43:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7faffbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 0x7faffc147e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7faffbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 0x7faffc147e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51118/0 (socket says 172.21.15.7:51118) 2023-03-03T08:43:34.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7faffbfff700 1 -- 172.21.15.7:0/3453351073 learned_addr learned my addr 172.21.15.7:0/3453351073 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:34.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7fb000823700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faffc0723f0 0x7faffc0aa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7fb001024700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faffc151960 0x7faffc1ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7faffbfff700 1 -- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faffc151960 msgr2=0x7faffc1ea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7faffbfff700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faffc151960 0x7faffc1ea6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7faffbfff700 1 -- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faffc0723f0 msgr2=0x7faffc0aa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7faffbfff700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faffc0723f0 0x7faffc0aa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7faffbfff700 1 -- 172.21.15.7:0/3453351073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe80094a0 con 0x7faffc0c3ac0 2023-03-03T08:43:34.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.686+0000 7fb001024700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faffc151960 0x7faffc1ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:34.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.687+0000 7faffbfff700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 0x7faffc147e80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fafe80090b0 tx=0x7fafe8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:34.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.687+0000 7faff97fa700 1 -- 172.21.15.7:0/3453351073 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8011630 con 0x7faffc0c3ac0 2023-03-03T08:43:34.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.687+0000 7faff97fa700 1 -- 172.21.15.7:0/3453351073 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe801e070 con 0x7faffc0c3ac0 2023-03-03T08:43:34.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.687+0000 7fb002a87700 1 -- 172.21.15.7:0/3453351073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc154b70 con 0x7faffc0c3ac0 2023-03-03T08:43:34.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.687+0000 7faff97fa700 1 -- 172.21.15.7:0/3453351073 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe80119b0 con 0x7faffc0c3ac0 2023-03-03T08:43:34.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.687+0000 7fb002a87700 1 -- 172.21.15.7:0/3453351073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc154890 con 0x7faffc0c3ac0 2023-03-03T08:43:34.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.688+0000 7fb002a87700 1 -- 172.21.15.7:0/3453351073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc073e00 con 0x7faffc0c3ac0 2023-03-03T08:43:34.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.688+0000 7faff97fa700 1 -- 172.21.15.7:0/3453351073 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fafe8017020 con 0x7faffc0c3ac0 2023-03-03T08:43:34.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.689+0000 7faff97fa700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe00682f0 0x7fafe006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:34.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.689+0000 7faff97fa700 1 -- 172.21.15.7:0/3453351073 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6042+0+0 (secure 0 0 0) 0x7fafe8086f90 con 0x7faffc0c3ac0 2023-03-03T08:43:34.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.689+0000 7fb000823700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe00682f0 0x7fafe006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:34.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.690+0000 7fb000823700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe00682f0 0x7fafe006a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faff0009960 tx=0x7faff00090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:34.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.693+0000 7faff97fa700 1 -- 172.21.15.7:0/3453351073 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fafe8053800 con 0x7faffc0c3ac0 2023-03-03T08:43:34.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.887+0000 7fb002a87700 1 -- 172.21.15.7:0/3453351073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faffc074bb0 con 0x7faffc0c3ac0 2023-03-03T08:43:34.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.889+0000 7faff97fa700 1 -- 172.21.15.7:0/3453351073 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17854 (secure 0 0 0) 0x7fafe8020090 con 0x7faffc0c3ac0 2023-03-03T08:43:34.890 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:34.890 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":22,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:34.516631+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:34.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 -- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe00682f0 msgr2=0x7fafe006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe00682f0 0x7fafe006a7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faff0009960 tx=0x7faff00090e0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 -- 172.21.15.7:0/3453351073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 msgr2=0x7faffc147e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:34.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 0x7faffc147e80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fafe80090b0 tx=0x7fafe8004750 comp rx=0 tx=0).stop 2023-03-03T08:43:34.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 -- 172.21.15.7:0/3453351073 shutdown_connections 2023-03-03T08:43:34.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faffc151960 0x7faffc1ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faffc0723f0 0x7faffc0aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe00682f0 0x7fafe006a7a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 --2- 172.21.15.7:0/3453351073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faffc0c3ac0 0x7faffc147e80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:34.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 -- 172.21.15.7:0/3453351073 >> 172.21.15.7:0/3453351073 conn(0x7faffc06c060 msgr2=0x7faffc06c430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:34.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 -- 172.21.15.7:0/3453351073 shutdown_connections 2023-03-03T08:43:34.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:34.891+0000 7fb002a87700 1 -- 172.21.15.7:0/3453351073 wait complete. 2023-03-03T08:43:34.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:35.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:35.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:43:35.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.029+0000 7f0c6633e700 1 Processor -- start 2023-03-03T08:43:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.029+0000 7f0c6633e700 1 -- start start 2023-03-03T08:43:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.030+0000 7f0c6633e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c600c1da0 0x7f0c601550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.030+0000 7f0c6633e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600c7a60 con 0x7f0c6006d870 2023-03-03T08:43:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.030+0000 7f0c6633e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600c3620 con 0x7f0c600c3e30 2023-03-03T08:43:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.030+0000 7f0c6633e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c600c2af0 con 0x7f0c600c1da0 2023-03-03T08:43:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.030+0000 7f0c5f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c600c1da0 0x7f0c601550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.030+0000 7f0c5ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0c6006d870 0x7f0c600c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45106/0 (socket says 172.21.15.7:45106) 2023-03-03T08:43:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.030+0000 7f0c5ffff700 1 -- 172.21.15.7:0/2517347047 learned_addr learned my addr 172.21.15.7:0/2517347047 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.031+0000 7f0c5f7fe700 1 -- 172.21.15.7:0/2517347047 >> v1:172.21.15.143:6789/0 conn(0x7f0c600c3e30 legacy=0x7f0c6015ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.031+0000 7f0c5f7fe700 1 -- 172.21.15.7:0/2517347047 >> v1:172.21.15.7:6789/0 conn(0x7f0c6006d870 legacy=0x7f0c600c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.031+0000 7f0c5effd700 1 -- 172.21.15.7:0/2517347047 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220063304 0 0) 0x7f0c600c7a60 con 0x7f0c6006d870 2023-03-03T08:43:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.031+0000 7f0c5f7fe700 1 -- 172.21.15.7:0/2517347047 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c601545e0 con 0x7f0c600c1da0 2023-03-03T08:43:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.031+0000 7f0c5f7fe700 1 --2- 172.21.15.7:0/2517347047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c1da0 0x7f0c601550c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009840 tx=0x7f0c54009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e71eab3b8408e6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.031+0000 7f0c5effd700 1 -- 172.21.15.7:0/2517347047 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54011830 con 0x7f0c600c1da0 2023-03-03T08:43:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.031+0000 7f0c5effd700 1 -- 172.21.15.7:0/2517347047 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c54011990 con 0x7f0c600c1da0 2023-03-03T08:43:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.031+0000 7f0c5effd700 1 -- 172.21.15.7:0/2517347047 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54011d20 con 0x7f0c600c1da0 2023-03-03T08:43:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.032+0000 7f0c6633e700 1 -- 172.21.15.7:0/2517347047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c1da0 msgr2=0x7f0c601550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.032+0000 7f0c6633e700 1 --2- 172.21.15.7:0/2517347047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c1da0 0x7f0c601550c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009840 tx=0x7f0c54009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.032+0000 7f0c6633e700 1 -- 172.21.15.7:0/2517347047 shutdown_connections 2023-03-03T08:43:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.032+0000 7f0c6633e700 1 --2- 172.21.15.7:0/2517347047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c1da0 0x7f0c601550c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.032+0000 7f0c6633e700 1 -- 172.21.15.7:0/2517347047 >> 172.21.15.7:0/2517347047 conn(0x7f0c6006c110 msgr2=0x7f0c6006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.032+0000 7f0c6633e700 1 -- 172.21.15.7:0/2517347047 shutdown_connections 2023-03-03T08:43:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.032+0000 7f0c6633e700 1 -- 172.21.15.7:0/2517347047 wait complete. 2023-03-03T08:43:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.033+0000 7f0c6633e700 1 Processor -- start 2023-03-03T08:43:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.033+0000 7f0c6633e700 1 -- start start 2023-03-03T08:43:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c6633e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c6006d870 0x7f0c600c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c6633e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c600c1da0 0x7f0c6013a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c6633e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c3e30 0x7f0c6013a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c6633e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6014ee00 con 0x7f0c600c3e30 2023-03-03T08:43:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c6633e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c60156cb0 con 0x7f0c6006d870 2023-03-03T08:43:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c6633e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6014fc00 con 0x7f0c600c1da0 2023-03-03T08:43:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c648db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c3e30 0x7f0c6013a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c648db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c3e30 0x7f0c6013a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51136/0 (socket says 172.21.15.7:51136) 2023-03-03T08:43:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c648db700 1 -- 172.21.15.7:0/2081120457 learned_addr learned my addr 172.21.15.7:0/2081120457 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c5f7fe700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c600c1da0 0x7f0c6013a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c5ffff700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c6006d870 0x7f0c600c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c648db700 1 -- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c600c1da0 msgr2=0x7f0c6013a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c648db700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c600c1da0 0x7f0c6013a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c648db700 1 -- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c6006d870 msgr2=0x7f0c600c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c648db700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c6006d870 0x7f0c600c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c648db700 1 -- 172.21.15.7:0/2081120457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c540094a0 con 0x7f0c600c3e30 2023-03-03T08:43:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.034+0000 7f0c648db700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c3e30 0x7f0c6013a5c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800cc80 tx=0x7f0c5800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.035+0000 7f0c5cff9700 1 -- 172.21.15.7:0/2081120457 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58003c00 con 0x7f0c600c3e30 2023-03-03T08:43:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.035+0000 7f0c5cff9700 1 -- 172.21.15.7:0/2081120457 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c58003d60 con 0x7f0c600c3e30 2023-03-03T08:43:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.035+0000 7f0c6633e700 1 -- 172.21.15.7:0/2081120457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c60154ba0 con 0x7f0c600c3e30 2023-03-03T08:43:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.035+0000 7f0c5cff9700 1 -- 172.21.15.7:0/2081120457 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5800aaa0 con 0x7f0c600c3e30 2023-03-03T08:43:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.035+0000 7f0c6633e700 1 -- 172.21.15.7:0/2081120457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c601548c0 con 0x7f0c600c3e30 2023-03-03T08:43:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.036+0000 7f0c6633e700 1 -- 172.21.15.7:0/2081120457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6006dc40 con 0x7f0c600c3e30 2023-03-03T08:43:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.037+0000 7f0c5cff9700 1 -- 172.21.15.7:0/2081120457 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0c58010050 con 0x7f0c600c3e30 2023-03-03T08:43:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.037+0000 7f0c5cff9700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c4c068320 0x7f0c4c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.037+0000 7f0c5cff9700 1 -- 172.21.15.7:0/2081120457 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f0c58086580 con 0x7f0c600c3e30 2023-03-03T08:43:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.037+0000 7f0c5ffff700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c4c068320 0x7f0c4c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.038+0000 7f0c5ffff700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c4c068320 0x7f0c4c06a7d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0c50000f60 tx=0x7f0c50004a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.041+0000 7f0c5cff9700 1 -- 172.21.15.7:0/2081120457 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0c58052e90 con 0x7f0c600c3e30 2023-03-03T08:43:35.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.237+0000 7f0c6633e700 1 -- 172.21.15.7:0/2081120457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c60074210 con 0x7f0c600c3e30 2023-03-03T08:43:35.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.238+0000 7f0c5cff9700 1 -- 172.21.15.7:0/2081120457 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17854 (secure 0 0 0) 0x7f0c580564b0 con 0x7f0c600c3e30 2023-03-03T08:43:35.240 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:35.240 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":22,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:34.516631+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:35.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.240+0000 7f0c6633e700 1 -- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c4c068320 msgr2=0x7f0c4c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:35.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.240+0000 7f0c6633e700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c4c068320 0x7f0c4c06a7d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0c50000f60 tx=0x7f0c50004a50 comp rx=0 tx=0).stop 2023-03-03T08:43:35.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.240+0000 7f0c6633e700 1 -- 172.21.15.7:0/2081120457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c3e30 msgr2=0x7f0c6013a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:35.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.240+0000 7f0c6633e700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c3e30 0x7f0c6013a5c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800cc80 tx=0x7f0c5800cf90 comp rx=0 tx=0).stop 2023-03-03T08:43:35.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.240+0000 7f0c6633e700 1 -- 172.21.15.7:0/2081120457 shutdown_connections 2023-03-03T08:43:35.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.240+0000 7f0c6633e700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c4c068320 0x7f0c4c06a7d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.240+0000 7f0c6633e700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c600c3e30 0x7f0c6013a5c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.240+0000 7f0c6633e700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c6006d870 0x7f0c600c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.240+0000 7f0c6633e700 1 --2- 172.21.15.7:0/2081120457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c600c1da0 0x7f0c6013a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.240+0000 7f0c6633e700 1 -- 172.21.15.7:0/2081120457 >> 172.21.15.7:0/2081120457 conn(0x7f0c6006c110 msgr2=0x7f0c6006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:35.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.241+0000 7f0c6633e700 1 -- 172.21.15.7:0/2081120457 shutdown_connections 2023-03-03T08:43:35.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.241+0000 7f0c6633e700 1 -- 172.21.15.7:0/2081120457 wait complete. 2023-03-03T08:43:35.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a standby_count_wanted 0 2023-03-03T08:43:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.381+0000 7efcda105700 1 Processor -- start 2023-03-03T08:43:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.381+0000 7efcda105700 1 -- start start 2023-03-03T08:43:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.381+0000 7efcda105700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efcd4071990 0x7efcd4071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.381+0000 7efcda105700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd414b460 0x7efcd40722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.381+0000 7efcda105700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd4070360 con 0x7efcd4151c00 2023-03-03T08:43:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.381+0000 7efcda105700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd41521a0 con 0x7efcd4071990 2023-03-03T08:43:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.381+0000 7efcda105700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd406a5e0 con 0x7efcd414b460 2023-03-03T08:43:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.381+0000 7efcd37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efcd4071990 0x7efcd4071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efcd4071990 0x7efcd4071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51140/0 (socket says 172.21.15.7:51140) 2023-03-03T08:43:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd37fe700 1 -- 172.21.15.7:0/1326363976 learned_addr learned my addr 172.21.15.7:0/1326363976 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd2ffd700 1 --2- 172.21.15.7:0/1326363976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd414b460 0x7efcd40722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd37fe700 1 -- 172.21.15.7:0/1326363976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd414b460 msgr2=0x7efcd40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd37fe700 1 --2- 172.21.15.7:0/1326363976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd414b460 0x7efcd40722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd37fe700 1 -- 172.21.15.7:0/1326363976 >> v1:172.21.15.143:6789/0 conn(0x7efcd4151c00 legacy=0x7efcd406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd37fe700 1 -- 172.21.15.7:0/1326363976 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40c38b0 con 0x7efcd4071990 2023-03-03T08:43:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd37fe700 1 --2- 172.21.15.7:0/1326363976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 0x7efcd4071d60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009e50 tx=0x7efcc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5d5a90c256cde0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd27fc700 1 -- 172.21.15.7:0/1326363976 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc400e070 con 0x7efcd4071990 2023-03-03T08:43:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd27fc700 1 -- 172.21.15.7:0/1326363976 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc4003a20 con 0x7efcd4071990 2023-03-03T08:43:35.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.382+0000 7efcd27fc700 1 -- 172.21.15.7:0/1326363976 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc4003dd0 con 0x7efcd4071990 2023-03-03T08:43:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.383+0000 7efcda105700 1 -- 172.21.15.7:0/1326363976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 msgr2=0x7efcd4071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.383+0000 7efcda105700 1 --2- 172.21.15.7:0/1326363976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 0x7efcd4071d60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efcc4009e50 tx=0x7efcc400b330 comp rx=0 tx=0).stop 2023-03-03T08:43:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.383+0000 7efcda105700 1 -- 172.21.15.7:0/1326363976 shutdown_connections 2023-03-03T08:43:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.383+0000 7efcda105700 1 --2- 172.21.15.7:0/1326363976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 0x7efcd4071d60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.383+0000 7efcda105700 1 --2- 172.21.15.7:0/1326363976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd414b460 0x7efcd40722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.383+0000 7efcda105700 1 -- 172.21.15.7:0/1326363976 >> 172.21.15.7:0/1326363976 conn(0x7efcd40a94d0 msgr2=0x7efcd40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:35.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.383+0000 7efcda105700 1 -- 172.21.15.7:0/1326363976 shutdown_connections 2023-03-03T08:43:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.383+0000 7efcda105700 1 -- 172.21.15.7:0/1326363976 wait complete. 2023-03-03T08:43:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.384+0000 7efcda105700 1 Processor -- start 2023-03-03T08:43:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.384+0000 7efcda105700 1 -- start start 2023-03-03T08:43:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcda105700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 0x7efcd41539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcda105700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd4151c00 0x7efcd41ed800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcda105700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcd4154db0 0x7efcd41eb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcda105700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd4153f10 con 0x7efcd4071990 2023-03-03T08:43:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcda105700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd4153540 con 0x7efcd4154db0 2023-03-03T08:43:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcda105700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd4152f80 con 0x7efcd4151c00 2023-03-03T08:43:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcd37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 0x7efcd41539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcd37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 0x7efcd41539d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51142/0 (socket says 172.21.15.7:51142) 2023-03-03T08:43:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcd37fe700 1 -- 172.21.15.7:0/3554812807 learned_addr learned my addr 172.21.15.7:0/3554812807 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcd37fe700 1 -- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd4151c00 msgr2=0x7efcd41ed800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcd37fe700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd4151c00 0x7efcd41ed800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcd37fe700 1 -- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcd4154db0 msgr2=0x7efcd41eb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcd37fe700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcd4154db0 0x7efcd41eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcd37fe700 1 -- 172.21.15.7:0/3554812807 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc4009b30 con 0x7efcd4071990 2023-03-03T08:43:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcd37fe700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 0x7efcd41539d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efcd40c2040 tx=0x7efcc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efccbfff700 1 -- 172.21.15.7:0/3554812807 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc40044c0 con 0x7efcd4071990 2023-03-03T08:43:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efccbfff700 1 -- 172.21.15.7:0/3554812807 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc4004640 con 0x7efcd4071990 2023-03-03T08:43:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcda105700 1 -- 172.21.15.7:0/3554812807 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd40c3b50 con 0x7efcd4071990 2023-03-03T08:43:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efccbfff700 1 -- 172.21.15.7:0/3554812807 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc401c440 con 0x7efcd4071990 2023-03-03T08:43:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.385+0000 7efcda105700 1 -- 172.21.15.7:0/3554812807 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd406a2b0 con 0x7efcd4071990 2023-03-03T08:43:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.386+0000 7efcda105700 1 -- 172.21.15.7:0/3554812807 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd406b180 con 0x7efcd4071990 2023-03-03T08:43:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.387+0000 7efccbfff700 1 -- 172.21.15.7:0/3554812807 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efcc401c5a0 con 0x7efcd4071990 2023-03-03T08:43:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.387+0000 7efccbfff700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb40682f0 0x7efcb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.387+0000 7efccbfff700 1 -- 172.21.15.7:0/3554812807 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6042+0+0 (secure 0 0 0) 0x7efcc40871e0 con 0x7efcd4071990 2023-03-03T08:43:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.387+0000 7efcd2ffd700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb40682f0 0x7efcb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.388+0000 7efcd2ffd700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb40682f0 0x7efcb406a7a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efcd40be2f0 tx=0x7efcbc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.391+0000 7efccbfff700 1 -- 172.21.15.7:0/3554812807 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efcc4053a50 con 0x7efcd4071990 2023-03-03T08:43:35.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:35.632+0000 7efcda105700 1 -- 172.21.15.7:0/3554812807 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7efcd4074bb0 con 0x7efcd4071990 2023-03-03T08:43:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.525+0000 7efccbfff700 1 -- 172.21.15.7:0/3554812807 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 113+0+0 (secure 0 0 0) 0x7efcc4057070 con 0x7efcd4071990 2023-03-03T08:43:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 -- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb40682f0 msgr2=0x7efcb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb40682f0 0x7efcb406a7a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efcd40be2f0 tx=0x7efcbc008040 comp rx=0 tx=0).stop 2023-03-03T08:43:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 -- 172.21.15.7:0/3554812807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 msgr2=0x7efcd41539d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 0x7efcd41539d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efcd40c2040 tx=0x7efcc4004ad0 comp rx=0 tx=0).stop 2023-03-03T08:43:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 -- 172.21.15.7:0/3554812807 shutdown_connections 2023-03-03T08:43:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcd4154db0 0x7efcd41eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb40682f0 0x7efcb406a7a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd4071990 0x7efcd41539d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 --2- 172.21.15.7:0/3554812807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd4151c00 0x7efcd41ed800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 -- 172.21.15.7:0/3554812807 >> 172.21.15.7:0/3554812807 conn(0x7efcd40a94d0 msgr2=0x7efcd4151240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 -- 172.21.15.7:0/3554812807 shutdown_connections 2023-03-03T08:43:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.527+0000 7efcda105700 1 -- 172.21.15.7:0/3554812807 wait complete. 2023-03-03T08:43:36.543 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2023-03-03T08:43:36.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false 2023-03-03T08:43:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.676+0000 7f39cf626700 1 Processor -- start 2023-03-03T08:43:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.676+0000 7f39cf626700 1 -- start start 2023-03-03T08:43:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39cf626700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f39c8151cd0 0x7f39c8154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39cf626700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c80c2e30 0x7f39c8158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39cf626700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c8157ff0 con 0x7f39c806c6c0 2023-03-03T08:43:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39cf626700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c806d640 con 0x7f39c8151cd0 2023-03-03T08:43:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39cf626700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c806cda0 con 0x7f39c80c2e30 2023-03-03T08:43:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39ccbc1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c80c2e30 0x7f39c8158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39cd3c2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f39c8151cd0 0x7f39c8154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39ccbc1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c80c2e30 0x7f39c8158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59224/0 (socket says 172.21.15.7:59224) 2023-03-03T08:43:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39ccbc1700 1 -- 172.21.15.7:0/613787966 learned_addr learned my addr 172.21.15.7:0/613787966 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39cdbc3700 1 --1- 172.21.15.7:0/613787966 >> v1:172.21.15.143:6789/0 conn(0x7f39c806c6c0 0x7f39c80736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:48654/0 (socket says 172.21.15.7:48654) 2023-03-03T08:43:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39bffff700 1 -- 172.21.15.7:0/613787966 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310715842 0 0) 0x7f39c8157ff0 con 0x7f39c806c6c0 2023-03-03T08:43:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39bffff700 1 -- 172.21.15.7:0/613787966 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b40035b0 con 0x7f39c806c6c0 2023-03-03T08:43:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39ccbc1700 1 -- 172.21.15.7:0/613787966 >> v2:172.21.15.143:3300/0 conn(0x7f39c8151cd0 msgr2=0x7f39c8154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39ccbc1700 1 --2- 172.21.15.7:0/613787966 >> v2:172.21.15.143:3300/0 conn(0x7f39c8151cd0 0x7f39c8154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.677+0000 7f39ccbc1700 1 -- 172.21.15.7:0/613787966 >> v1:172.21.15.143:6789/0 conn(0x7f39c806c6c0 legacy=0x7f39c80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39ccbc1700 1 -- 172.21.15.7:0/613787966 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c806d080 con 0x7f39c80c2e30 2023-03-03T08:43:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39cd3c2700 1 --2- 172.21.15.7:0/613787966 >> v2:172.21.15.143:3300/0 conn(0x7f39c8151cd0 0x7f39c8154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39ccbc1700 1 --2- 172.21.15.7:0/613787966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c80c2e30 0x7f39c8158480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f39c40098f0 tx=0x7f39c4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a67077e629e01266 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39bffff700 1 -- 172.21.15.7:0/613787966 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4011830 con 0x7f39c80c2e30 2023-03-03T08:43:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39bffff700 1 -- 172.21.15.7:0/613787966 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c4011990 con 0x7f39c80c2e30 2023-03-03T08:43:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39bffff700 1 -- 172.21.15.7:0/613787966 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4011d00 con 0x7f39c80c2e30 2023-03-03T08:43:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39cf626700 1 -- 172.21.15.7:0/613787966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c80c2e30 msgr2=0x7f39c8158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39cf626700 1 --2- 172.21.15.7:0/613787966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c80c2e30 0x7f39c8158480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f39c40098f0 tx=0x7f39c4009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39cf626700 1 -- 172.21.15.7:0/613787966 shutdown_connections 2023-03-03T08:43:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39cf626700 1 --2- 172.21.15.7:0/613787966 >> v2:172.21.15.143:3300/0 conn(0x7f39c8151cd0 0x7f39c8154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39cf626700 1 --2- 172.21.15.7:0/613787966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c80c2e30 0x7f39c8158480 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:36.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.678+0000 7f39cf626700 1 -- 172.21.15.7:0/613787966 >> 172.21.15.7:0/613787966 conn(0x7f39c80a94d0 msgr2=0x7f39c80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.679+0000 7f39cf626700 1 -- 172.21.15.7:0/613787966 shutdown_connections 2023-03-03T08:43:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.679+0000 7f39cf626700 1 -- 172.21.15.7:0/613787966 wait complete. 2023-03-03T08:43:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cf626700 1 Processor -- start 2023-03-03T08:43:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cf626700 1 -- start start 2023-03-03T08:43:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cf626700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39c806c6c0 0x7f39c814da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cf626700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c80c2e30 0x7f39c814dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:36.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cf626700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c8151cd0 0x7f39c814b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cf626700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c814fed0 con 0x7f39c806c6c0 2023-03-03T08:43:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cf626700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c814aba0 con 0x7f39c80c2e30 2023-03-03T08:43:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cf626700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c80c6d10 con 0x7f39c8151cd0 2023-03-03T08:43:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cd3c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39c806c6c0 0x7f39c814da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cd3c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39c806c6c0 0x7f39c814da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51156/0 (socket says 172.21.15.7:51156) 2023-03-03T08:43:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.680+0000 7f39cd3c2700 1 -- 172.21.15.7:0/114372359 learned_addr learned my addr 172.21.15.7:0/114372359 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:36.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39ccbc1700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c80c2e30 0x7f39c814dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39cdbc3700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c8151cd0 0x7f39c814b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39cd3c2700 1 -- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c8151cd0 msgr2=0x7f39c814b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39cd3c2700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c8151cd0 0x7f39c814b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39cd3c2700 1 -- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c80c2e30 msgr2=0x7f39c814dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39cd3c2700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c80c2e30 0x7f39c814dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39cd3c2700 1 -- 172.21.15.7:0/114372359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80094a0 con 0x7f39c806c6c0 2023-03-03T08:43:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39cdbc3700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c8151cd0 0x7f39c814b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:36.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39cd3c2700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39c806c6c0 0x7f39c814da60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f39b800ebe0 tx=0x7f39b800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:36.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39bdffb700 1 -- 172.21.15.7:0/114372359 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b800c850 con 0x7f39c806c6c0 2023-03-03T08:43:36.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39bdffb700 1 -- 172.21.15.7:0/114372359 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b800c9b0 con 0x7f39c806c6c0 2023-03-03T08:43:36.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39bdffb700 1 -- 172.21.15.7:0/114372359 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b800cd40 con 0x7f39c806c6c0 2023-03-03T08:43:36.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39cf626700 1 -- 172.21.15.7:0/114372359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c4009550 con 0x7f39c806c6c0 2023-03-03T08:43:36.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.681+0000 7f39cf626700 1 -- 172.21.15.7:0/114372359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c8157c80 con 0x7f39c806c6c0 2023-03-03T08:43:36.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.682+0000 7f39cf626700 1 -- 172.21.15.7:0/114372359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c806d360 con 0x7f39c806c6c0 2023-03-03T08:43:36.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.683+0000 7f39bdffb700 1 -- 172.21.15.7:0/114372359 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f39b8023070 con 0x7f39c806c6c0 2023-03-03T08:43:36.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.683+0000 7f39bdffb700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39ac0682f0 0x7f39ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:36.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.683+0000 7f39bdffb700 1 -- 172.21.15.7:0/114372359 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f39b8085fe0 con 0x7f39c806c6c0 2023-03-03T08:43:36.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.683+0000 7f39ccbc1700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39ac0682f0 0x7f39ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:36.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.684+0000 7f39ccbc1700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39ac0682f0 0x7f39ac06a7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f39c4014040 tx=0x7f39c4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:36.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.687+0000 7f39bdffb700 1 -- 172.21.15.7:0/114372359 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f39b80527c0 con 0x7f39c806c6c0 2023-03-03T08:43:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:36.926+0000 7f39cf626700 1 -- 172.21.15.7:0/114372359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f39c8154a50 con 0x7f39c806c6c0 2023-03-03T08:43:37.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.529+0000 7f39bdffb700 1 -- 172.21.15.7:0/114372359 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 117+0+0 (secure 0 0 0) 0x7f39b800cea0 con 0x7f39c806c6c0 2023-03-03T08:43:37.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 -- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39ac0682f0 msgr2=0x7f39ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:37.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39ac0682f0 0x7f39ac06a7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f39c4014040 tx=0x7f39c4008030 comp rx=0 tx=0).stop 2023-03-03T08:43:37.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 -- 172.21.15.7:0/114372359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39c806c6c0 msgr2=0x7f39c814da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:37.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39c806c6c0 0x7f39c814da60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f39b800ebe0 tx=0x7f39b800eef0 comp rx=0 tx=0).stop 2023-03-03T08:43:37.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 -- 172.21.15.7:0/114372359 shutdown_connections 2023-03-03T08:43:37.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39c8151cd0 0x7f39c814b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:37.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39ac0682f0 0x7f39ac06a7a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:37.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39c806c6c0 0x7f39c814da60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:37.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 --2- 172.21.15.7:0/114372359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39c80c2e30 0x7f39c814dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:37.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 -- 172.21.15.7:0/114372359 >> 172.21.15.7:0/114372359 conn(0x7f39c80a94d0 msgr2=0x7f39c80c2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:37.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 -- 172.21.15.7:0/114372359 shutdown_connections 2023-03-03T08:43:37.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.532+0000 7f39cf626700 1 -- 172.21.15.7:0/114372359 wait complete. 2023-03-03T08:43:37.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a session_timeout 300 2023-03-03T08:43:37.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.682+0000 7f1352a8b700 1 Processor -- start 2023-03-03T08:43:37.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.682+0000 7f1352a8b700 1 -- start start 2023-03-03T08:43:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.683+0000 7f1352a8b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c0aac00 0x7f134c0c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.683+0000 7f1352a8b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f134c155950 0x7f134c155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.683+0000 7f1352a8b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c0713c0 con 0x7f134c071c30 2023-03-03T08:43:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.683+0000 7f1352a8b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134c156310 con 0x7f134c155950 2023-03-03T08:43:37.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.683+0000 7f1352a8b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c152c70 con 0x7f134c0aac00 2023-03-03T08:43:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.683+0000 7f1351028700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f134c155950 0x7f134c155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.683+0000 7f1351028700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f134c155950 0x7f134c155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51168/0 (socket says 172.21.15.7:51168) 2023-03-03T08:43:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.683+0000 7f1351028700 1 -- 172.21.15.7:0/2407631524 learned_addr learned my addr 172.21.15.7:0/2407631524 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.683+0000 7f1350827700 1 --2- 172.21.15.7:0/2407631524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c0aac00 0x7f134c0c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1351028700 1 -- 172.21.15.7:0/2407631524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c0aac00 msgr2=0x7f134c0c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:37.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1351028700 1 --2- 172.21.15.7:0/2407631524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c0aac00 0x7f134c0c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1351028700 1 -- 172.21.15.7:0/2407631524 >> v1:172.21.15.143:6789/0 conn(0x7f134c071c30 legacy=0x7f134c0beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1351028700 1 -- 172.21.15.7:0/2407631524 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c0c4930 con 0x7f134c155950 2023-03-03T08:43:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1350827700 1 --2- 172.21.15.7:0/2407631524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c0aac00 0x7f134c0c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1351028700 1 --2- 172.21.15.7:0/2407631524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c155950 0x7f134c155dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1344009e50 tx=0x7f134400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf391be3ffca8d3e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f134b7fe700 1 -- 172.21.15.7:0/2407631524 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134400e070 con 0x7f134c155950 2023-03-03T08:43:37.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f134b7fe700 1 -- 172.21.15.7:0/2407631524 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134400b9c0 con 0x7f134c155950 2023-03-03T08:43:37.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f134b7fe700 1 -- 172.21.15.7:0/2407631524 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134400bd70 con 0x7f134c155950 2023-03-03T08:43:37.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1352a8b700 1 -- 172.21.15.7:0/2407631524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c155950 msgr2=0x7f134c155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:37.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1352a8b700 1 --2- 172.21.15.7:0/2407631524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c155950 0x7f134c155dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1344009e50 tx=0x7f134400b330 comp rx=0 tx=0).stop 2023-03-03T08:43:37.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1352a8b700 1 -- 172.21.15.7:0/2407631524 shutdown_connections 2023-03-03T08:43:37.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1352a8b700 1 --2- 172.21.15.7:0/2407631524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c155950 0x7f134c155dd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:37.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1352a8b700 1 --2- 172.21.15.7:0/2407631524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c0aac00 0x7f134c0c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:37.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.684+0000 7f1352a8b700 1 -- 172.21.15.7:0/2407631524 >> 172.21.15.7:0/2407631524 conn(0x7f134c0736f0 msgr2=0x7f134c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:37.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.685+0000 7f1352a8b700 1 -- 172.21.15.7:0/2407631524 shutdown_connections 2023-03-03T08:43:37.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.685+0000 7f1352a8b700 1 -- 172.21.15.7:0/2407631524 wait complete. 2023-03-03T08:43:37.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.686+0000 7f1352a8b700 1 Processor -- start 2023-03-03T08:43:37.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.686+0000 7f1352a8b700 1 -- start start 2023-03-03T08:43:37.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.686+0000 7f1352a8b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f134c071c30 0x7f134c14ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:37.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.686+0000 7f1352a8b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c0aac00 0x7f134c1472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:37.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.686+0000 7f1352a8b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c155950 0x7f134c1487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:37.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.686+0000 7f1352a8b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c14e4f0 con 0x7f134c0aac00 2023-03-03T08:43:37.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.686+0000 7f1352a8b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c14cf60 con 0x7f134c071c30 2023-03-03T08:43:37.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.686+0000 7f1352a8b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c0adfa0 con 0x7f134c155950 2023-03-03T08:43:37.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f134bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c0aac00 0x7f134c1472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:37.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f134bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c0aac00 0x7f134c1472c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51172/0 (socket says 172.21.15.7:51172) 2023-03-03T08:43:37.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f134bfff700 1 -- 172.21.15.7:0/1914147731 learned_addr learned my addr 172.21.15.7:0/1914147731 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:37.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f1351028700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c155950 0x7f134c1487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:37.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f1350827700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f134c071c30 0x7f134c14ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:37.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f134bfff700 1 -- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c155950 msgr2=0x7f134c1487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:37.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f134bfff700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c155950 0x7f134c1487f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:37.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f134bfff700 1 -- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f134c071c30 msgr2=0x7f134c14ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:37.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f134bfff700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f134c071c30 0x7f134c14ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:37.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f134bfff700 1 -- 172.21.15.7:0/1914147731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1344009b30 con 0x7f134c0aac00 2023-03-03T08:43:37.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f1350827700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f134c071c30 0x7f134c14ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:37.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f134bfff700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c0aac00 0x7f134c1472c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f133800c510 tx=0x7f133800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:37.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f13497fa700 1 -- 172.21.15.7:0/1914147731 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13380052d0 con 0x7f134c0aac00 2023-03-03T08:43:37.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f13497fa700 1 -- 172.21.15.7:0/1914147731 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1338005430 con 0x7f134c0aac00 2023-03-03T08:43:37.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f1352a8b700 1 -- 172.21.15.7:0/1914147731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c0c4b80 con 0x7f134c0aac00 2023-03-03T08:43:37.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f13497fa700 1 -- 172.21.15.7:0/1914147731 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13380038c0 con 0x7f134c0aac00 2023-03-03T08:43:37.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.687+0000 7f1352a8b700 1 -- 172.21.15.7:0/1914147731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c152360 con 0x7f134c0aac00 2023-03-03T08:43:37.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.688+0000 7f1352a8b700 1 -- 172.21.15.7:0/1914147731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134c152620 con 0x7f134c0aac00 2023-03-03T08:43:37.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.689+0000 7f13497fa700 1 -- 172.21.15.7:0/1914147731 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1338019030 con 0x7f134c0aac00 2023-03-03T08:43:37.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.689+0000 7f13497fa700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13300682f0 0x7f133006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:37.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.689+0000 7f13497fa700 1 -- 172.21.15.7:0/1914147731 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f1338084fe0 con 0x7f134c0aac00 2023-03-03T08:43:37.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.689+0000 7f1350827700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13300682f0 0x7f133006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:37.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.690+0000 7f1350827700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13300682f0 0x7f133006a7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1340009a60 tx=0x7f1340008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:37.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.693+0000 7f13497fa700 1 -- 172.21.15.7:0/1914147731 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1338051850 con 0x7f134c0aac00 2023-03-03T08:43:37.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:37.934+0000 7f1352a8b700 1 -- 172.21.15.7:0/1914147731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f134c148510 con 0x7f134c0aac00 2023-03-03T08:43:38.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.530+0000 7f13497fa700 1 -- 172.21.15.7:0/1914147731 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"}]=0 v9) v1 ==== 110+0+0 (secure 0 0 0) 0x7f134c148510 con 0x7f134c0aac00 2023-03-03T08:43:38.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 -- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13300682f0 msgr2=0x7f133006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:38.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13300682f0 0x7f133006a7a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1340009a60 tx=0x7f1340008040 comp rx=0 tx=0).stop 2023-03-03T08:43:38.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 -- 172.21.15.7:0/1914147731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c0aac00 msgr2=0x7f134c1472c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:38.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c0aac00 0x7f134c1472c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f133800c510 tx=0x7f133800c8d0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 -- 172.21.15.7:0/1914147731 shutdown_connections 2023-03-03T08:43:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f134c155950 0x7f134c1487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f134c071c30 0x7f134c14ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13300682f0 0x7f133006a7a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 --2- 172.21.15.7:0/1914147731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f134c0aac00 0x7f134c1472c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 -- 172.21.15.7:0/1914147731 >> 172.21.15.7:0/1914147731 conn(0x7f134c0736f0 msgr2=0x7f134c0be950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 -- 172.21.15.7:0/1914147731 shutdown_connections 2023-03-03T08:43:38.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.533+0000 7f1352a8b700 1 -- 172.21.15.7:0/1914147731 wait complete. 2023-03-03T08:43:38.549 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'a' 2023-03-03T08:43:38.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:43:38.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.680+0000 7fd52c14e700 1 Processor -- start 2023-03-03T08:43:38.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.680+0000 7fd52c14e700 1 -- start start 2023-03-03T08:43:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.680+0000 7fd52c14e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd5240c2e30 0x7fd524158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.680+0000 7fd52c14e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd524157ff0 con 0x7fd52406c6c0 2023-03-03T08:43:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd52c14e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd52406d640 con 0x7fd524151cd0 2023-03-03T08:43:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd52c14e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52406cda0 con 0x7fd5240c2e30 2023-03-03T08:43:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd52a6eb700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd52406c6c0 0x7fd5240736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45162/0 (socket says 172.21.15.7:45162) 2023-03-03T08:43:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd52a6eb700 1 -- 172.21.15.7:0/2987572884 learned_addr learned my addr 172.21.15.7:0/2987572884 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd5296e9700 1 --2- 172.21.15.7:0/2987572884 >> v2:172.21.15.143:3300/0 conn(0x7fd5240c2e30 0x7fd524158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338573956 0 0) 0x7fd524157ff0 con 0x7fd52406c6c0 2023-03-03T08:43:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5100035b0 con 0x7fd52406c6c0 2023-03-03T08:43:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2782808760 0 0) 0x7fd5100035b0 con 0x7fd52406c6c0 2023-03-03T08:43:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 >> v2:172.21.15.143:3300/0 conn(0x7fd5240c2e30 msgr2=0x7fd524158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 --2- 172.21.15.7:0/2987572884 >> v2:172.21.15.143:3300/0 conn(0x7fd5240c2e30 0x7fd524158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 >> v1:172.21.15.143:6789/0 conn(0x7fd524151cd0 legacy=0x7fd524154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52406d080 con 0x7fd52406c6c0 2023-03-03T08:43:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd51c0019f0 con 0x7fd52406c6c0 2023-03-03T08:43:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 >> v1:172.21.15.7:6789/0 conn(0x7fd52406c6c0 legacy=0x7fd5240736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd510003a70 0x7fd510003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd510004850 0x7fd510004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd510005250 0x7fd51000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52406cda0 con 0x7fd510004850 2023-03-03T08:43:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51000c190 con 0x7fd510005250 2023-03-03T08:43:38.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51000c350 con 0x7fd510003a70 2023-03-03T08:43:38.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd5296e9700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd510004850 0x7fd510004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:38.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd529eea700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd510003a70 0x7fd510003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:38.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.681+0000 7fd52a6eb700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd510005250 0x7fd51000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:38.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd5296e9700 1 -- 172.21.15.7:0/2987572884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd510003a70 msgr2=0x7fd510003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:38.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd5296e9700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd510003a70 0x7fd510003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd5296e9700 1 -- 172.21.15.7:0/2987572884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd510005250 msgr2=0x7fd51000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:38.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd5296e9700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd510005250 0x7fd51000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd5296e9700 1 -- 172.21.15.7:0/2987572884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5100035b0 con 0x7fd510004850 2023-03-03T08:43:38.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd52a6eb700 1 -- 172.21.15.7:0/2987572884 reap_dead start 2023-03-03T08:43:38.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd529eea700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd510003a70 0x7fd510003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:38.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd5296e9700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd510004850 0x7fd510004cc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd518009b30 tx=0x7fd510005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9df45525b0ce95 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:38.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51800e6c0 con 0x7fd510004850 2023-03-03T08:43:38.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51800e820 con 0x7fd510004850 2023-03-03T08:43:38.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd528ee8700 1 -- 172.21.15.7:0/2987572884 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51800eb90 con 0x7fd510004850 2023-03-03T08:43:38.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd52c14e700 1 -- 172.21.15.7:0/2987572884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd510004850 msgr2=0x7fd510004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:38.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd52c14e700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd510004850 0x7fd510004cc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd518009b30 tx=0x7fd510005880 comp rx=0 tx=0).stop 2023-03-03T08:43:38.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd52c14e700 1 -- 172.21.15.7:0/2987572884 shutdown_connections 2023-03-03T08:43:38.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd52c14e700 1 --2- 172.21.15.7:0/2987572884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd510004850 0x7fd510004cc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.682+0000 7fd52c14e700 1 -- 172.21.15.7:0/2987572884 >> 172.21.15.7:0/2987572884 conn(0x7fd5240a94d0 msgr2=0x7fd5240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:38.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.683+0000 7fd52c14e700 1 -- 172.21.15.7:0/2987572884 shutdown_connections 2023-03-03T08:43:38.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.683+0000 7fd52c14e700 1 -- 172.21.15.7:0/2987572884 wait complete. 2023-03-03T08:43:38.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.684+0000 7fd52c14e700 1 Processor -- start 2023-03-03T08:43:38.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.684+0000 7fd52c14e700 1 -- start start 2023-03-03T08:43:38.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.684+0000 7fd52c14e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd510004850 0x7fd52414dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:38.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.684+0000 7fd52c14e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd524155900 0x7fd5240707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:38.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.684+0000 7fd52c14e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524070d20 0x7fd5241ed6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:38.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.684+0000 7fd52c14e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52414ac80 con 0x7fd524155900 2023-03-03T08:43:38.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.684+0000 7fd52c14e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5241575b0 con 0x7fd524070d20 2023-03-03T08:43:38.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.684+0000 7fd52c14e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd524157280 con 0x7fd510004850 2023-03-03T08:43:38.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5296e9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd524155900 0x7fd5240707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:38.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5296e9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd524155900 0x7fd5240707e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51192/0 (socket says 172.21.15.7:51192) 2023-03-03T08:43:38.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5296e9700 1 -- 172.21.15.7:0/248565834 learned_addr learned my addr 172.21.15.7:0/248565834 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:38.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd529eea700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd510004850 0x7fd52414dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:38.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd52a6eb700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524070d20 0x7fd5241ed6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:38.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5296e9700 1 -- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd510004850 msgr2=0x7fd52414dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:38.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5296e9700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd510004850 0x7fd52414dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5296e9700 1 -- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524070d20 msgr2=0x7fd5241ed6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:38.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5296e9700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524070d20 0x7fd5241ed6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5296e9700 1 -- 172.21.15.7:0/248565834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd524157c80 con 0x7fd524155900 2023-03-03T08:43:38.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd52a6eb700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524070d20 0x7fd5241ed6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:38.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5296e9700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd524155900 0x7fd5240707e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd51800e4b0 tx=0x7fd5180047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:38.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5167fc700 1 -- 172.21.15.7:0/248565834 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51800e6c0 con 0x7fd524155900 2023-03-03T08:43:38.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5167fc700 1 -- 172.21.15.7:0/248565834 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd518003ab0 con 0x7fd524155900 2023-03-03T08:43:38.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd52c14e700 1 -- 172.21.15.7:0/248565834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52406d360 con 0x7fd524155900 2023-03-03T08:43:38.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd5167fc700 1 -- 172.21.15.7:0/248565834 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518003e60 con 0x7fd524155900 2023-03-03T08:43:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.685+0000 7fd52c14e700 1 -- 172.21.15.7:0/248565834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd524154510 con 0x7fd524155900 2023-03-03T08:43:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.686+0000 7fd52c14e700 1 -- 172.21.15.7:0/248565834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd524074bb0 con 0x7fd524155900 2023-03-03T08:43:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.687+0000 7fd5167fc700 1 -- 172.21.15.7:0/248565834 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd518019030 con 0x7fd524155900 2023-03-03T08:43:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.687+0000 7fd5167fc700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5080682f0 0x7fd50806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.687+0000 7fd5167fc700 1 -- 172.21.15.7:0/248565834 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7fd518087350 con 0x7fd524155900 2023-03-03T08:43:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.688+0000 7fd529eea700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5080682f0 0x7fd50806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.688+0000 7fd529eea700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5080682f0 0x7fd50806a7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd52404f440 tx=0x7fd520009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:38.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.692+0000 7fd5167fc700 1 -- 172.21.15.7:0/248565834 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd518053bc0 con 0x7fd524155900 2023-03-03T08:43:38.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.927+0000 7fd52c14e700 1 -- 172.21.15.7:0/248565834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd52414ac80 con 0x7fd524155900 2023-03-03T08:43:38.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.927+0000 7fd5167fc700 1 -- 172.21.15.7:0/248565834 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4638 (secure 0 0 0) 0x7fd5180571e0 con 0x7fd524155900 2023-03-03T08:43:38.929 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:38.929 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-03T08:43:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 -- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5080682f0 msgr2=0x7fd50806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5080682f0 0x7fd50806a7a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd52404f440 tx=0x7fd520009040 comp rx=0 tx=0).stop 2023-03-03T08:43:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 -- 172.21.15.7:0/248565834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd524155900 msgr2=0x7fd5240707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd524155900 0x7fd5240707e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd51800e4b0 tx=0x7fd5180047c0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 -- 172.21.15.7:0/248565834 shutdown_connections 2023-03-03T08:43:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd524070d20 0x7fd5241ed6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd510004850 0x7fd52414dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5080682f0 0x7fd50806a7a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 --2- 172.21.15.7:0/248565834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd524155900 0x7fd5240707e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 -- 172.21.15.7:0/248565834 >> 172.21.15.7:0/248565834 conn(0x7fd5240a94d0 msgr2=0x7fd52406a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 -- 172.21.15.7:0/248565834 shutdown_connections 2023-03-03T08:43:38.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:38.930+0000 7fd52c14e700 1 -- 172.21.15.7:0/248565834 wait complete. 2023-03-03T08:43:38.933 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 9 2023-03-03T08:43:38.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:39.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.066+0000 7f63a00b4700 1 Processor -- start 2023-03-03T08:43:39.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.066+0000 7f63a00b4700 1 -- start start 2023-03-03T08:43:39.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.067+0000 7f63a00b4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f63980c1da0 0x7f6398156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.067+0000 7f63a00b4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f63980740f0 0x7f639815cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.067+0000 7f63a00b4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6398155840 con 0x7f639814fb00 2023-03-03T08:43:39.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.067+0000 7f63a00b4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6398072940 con 0x7f63980740f0 2023-03-03T08:43:39.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.067+0000 7f63a00b4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63981567d0 con 0x7f63980c1da0 2023-03-03T08:43:39.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.067+0000 7f639e651700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f63980740f0 0x7f639815cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.067+0000 7f639e651700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f63980740f0 0x7f639815cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51202/0 (socket says 172.21.15.7:51202) 2023-03-03T08:43:39.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.067+0000 7f639e651700 1 -- 172.21.15.7:0/2963254793 learned_addr learned my addr 172.21.15.7:0/2963254793 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:39.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.067+0000 7f639d64f700 1 --2- 172.21.15.7:0/2963254793 >> v2:172.21.15.143:3300/0 conn(0x7f63980c1da0 0x7f6398156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f639e651700 1 -- 172.21.15.7:0/2963254793 >> v2:172.21.15.143:3300/0 conn(0x7f63980c1da0 msgr2=0x7f6398156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:39.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f639e651700 1 --2- 172.21.15.7:0/2963254793 >> v2:172.21.15.143:3300/0 conn(0x7f63980c1da0 0x7f6398156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f639e651700 1 -- 172.21.15.7:0/2963254793 >> v1:172.21.15.143:6789/0 conn(0x7f639814fb00 legacy=0x7f63980c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f639e651700 1 -- 172.21.15.7:0/2963254793 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63980c2330 con 0x7f63980740f0 2023-03-03T08:43:39.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f639e651700 1 --2- 172.21.15.7:0/2963254793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63980740f0 0x7f639815cbd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f639806b300 tx=0x7f639000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f3a743db07e8f1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:39.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f639ce4e700 1 -- 172.21.15.7:0/2963254793 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639000e070 con 0x7f63980740f0 2023-03-03T08:43:39.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f639ce4e700 1 -- 172.21.15.7:0/2963254793 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639000b910 con 0x7f63980740f0 2023-03-03T08:43:39.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f639ce4e700 1 -- 172.21.15.7:0/2963254793 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639000bcc0 con 0x7f63980740f0 2023-03-03T08:43:39.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f63a00b4700 1 -- 172.21.15.7:0/2963254793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63980740f0 msgr2=0x7f639815cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f63a00b4700 1 --2- 172.21.15.7:0/2963254793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63980740f0 0x7f639815cbd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f639806b300 tx=0x7f639000b330 comp rx=0 tx=0).stop 2023-03-03T08:43:39.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f63a00b4700 1 -- 172.21.15.7:0/2963254793 shutdown_connections 2023-03-03T08:43:39.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f63a00b4700 1 --2- 172.21.15.7:0/2963254793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63980740f0 0x7f639815cbd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f63a00b4700 1 --2- 172.21.15.7:0/2963254793 >> v2:172.21.15.143:3300/0 conn(0x7f63980c1da0 0x7f6398156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.068+0000 7f63a00b4700 1 -- 172.21.15.7:0/2963254793 >> 172.21.15.7:0/2963254793 conn(0x7f639806c110 msgr2=0x7f639806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:39.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.069+0000 7f63a00b4700 1 -- 172.21.15.7:0/2963254793 shutdown_connections 2023-03-03T08:43:39.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.069+0000 7f63a00b4700 1 -- 172.21.15.7:0/2963254793 wait complete. 2023-03-03T08:43:39.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f63a00b4700 1 Processor -- start 2023-03-03T08:43:39.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f63a00b4700 1 -- start start 2023-03-03T08:43:39.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f63a00b4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63980740f0 0x7f63980c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f63a00b4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63980c1da0 0x7f63981ed410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f63a00b4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f639814fb00 0x7f63981ed950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f63a00b4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6398072240 con 0x7f639814fb00 2023-03-03T08:43:39.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f63a00b4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63981ecf70 con 0x7f63980740f0 2023-03-03T08:43:39.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f63a00b4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63980c7a90 con 0x7f63980c1da0 2023-03-03T08:43:39.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f639e651700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f639814fb00 0x7f63981ed950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f639e651700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f639814fb00 0x7f63981ed950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51208/0 (socket says 172.21.15.7:51208) 2023-03-03T08:43:39.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.070+0000 7f639e651700 1 -- 172.21.15.7:0/2741610041 learned_addr learned my addr 172.21.15.7:0/2741610041 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:39.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f639e651700 1 -- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63980c1da0 msgr2=0x7f63981ed410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f639d64f700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63980c1da0 0x7f63981ed410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f639de50700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63980740f0 0x7f63980c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f639e651700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63980c1da0 0x7f63981ed410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f639e651700 1 -- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63980740f0 msgr2=0x7f63980c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f639e651700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63980740f0 0x7f63980c1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f639e651700 1 -- 172.21.15.7:0/2741610041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6390009b30 con 0x7f639814fb00 2023-03-03T08:43:39.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f639d64f700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63980c1da0 0x7f63981ed410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:39.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f639de50700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63980740f0 0x7f63980c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:39.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f639e651700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f639814fb00 0x7f63981ed950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f639806cff0 tx=0x7f6390004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:39.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f638e7fc700 1 -- 172.21.15.7:0/2741610041 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63900099b0 con 0x7f639814fb00 2023-03-03T08:43:39.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f638e7fc700 1 -- 172.21.15.7:0/2741610041 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63900092e0 con 0x7f639814fb00 2023-03-03T08:43:39.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f63a00b4700 1 -- 172.21.15.7:0/2741610041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6398153090 con 0x7f639814fb00 2023-03-03T08:43:39.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f638e7fc700 1 -- 172.21.15.7:0/2741610041 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639001b630 con 0x7f639814fb00 2023-03-03T08:43:39.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.071+0000 7f63a00b4700 1 -- 172.21.15.7:0/2741610041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6398152db0 con 0x7f639814fb00 2023-03-03T08:43:39.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.072+0000 7f63a00b4700 1 -- 172.21.15.7:0/2741610041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6398155a90 con 0x7f639814fb00 2023-03-03T08:43:39.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.073+0000 7f638e7fc700 1 -- 172.21.15.7:0/2741610041 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f639002b070 con 0x7f639814fb00 2023-03-03T08:43:39.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.073+0000 7f638e7fc700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f637c0682f0 0x7f637c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.073+0000 7f638e7fc700 1 -- 172.21.15.7:0/2741610041 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f63900874c0 con 0x7f639814fb00 2023-03-03T08:43:39.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.073+0000 7f639de50700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f637c0682f0 0x7f637c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.074+0000 7f639de50700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f637c0682f0 0x7f637c06a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6388000f60 tx=0x7f6388009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:39.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.077+0000 7f638e7fc700 1 -- 172.21.15.7:0/2741610041 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6390053d30 con 0x7f639814fb00 2023-03-03T08:43:39.197 INFO:tasks.ceph.mds.a.smithi007.stderr:2023-03-03T08:43:39.195+0000 7f5cf88f6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:43:39.198 INFO:tasks.ceph.mds.a.smithi007.stderr:2023-03-03T08:43:39.197+0000 7f5cfb0fb700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:43:39.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.271+0000 7f63a00b4700 1 -- 172.21.15.7:0/2741610041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f639806e410 con 0x7f639814fb00 2023-03-03T08:43:39.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.272+0000 7f638e7fc700 1 -- 172.21.15.7:0/2741610041 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+17841 (secure 0 0 0) 0x7f6390057350 con 0x7f639814fb00 2023-03-03T08:43:39.274 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:39.274 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:35.518672+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:39.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 -- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f637c0682f0 msgr2=0x7f637c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f637c0682f0 0x7f637c06a7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6388000f60 tx=0x7f6388009040 comp rx=0 tx=0).stop 2023-03-03T08:43:39.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 -- 172.21.15.7:0/2741610041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f639814fb00 msgr2=0x7f63981ed950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f639814fb00 0x7f63981ed950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f639806cff0 tx=0x7f6390004ad0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 -- 172.21.15.7:0/2741610041 shutdown_connections 2023-03-03T08:43:39.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f637c0682f0 0x7f637c06a7a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f639814fb00 0x7f63981ed950 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63980740f0 0x7f63980c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 --2- 172.21.15.7:0/2741610041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f63980c1da0 0x7f63981ed410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 -- 172.21.15.7:0/2741610041 >> 172.21.15.7:0/2741610041 conn(0x7f639806c110 msgr2=0x7f6398155080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:39.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 -- 172.21.15.7:0/2741610041 shutdown_connections 2023-03-03T08:43:39.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.274+0000 7f63a00b4700 1 -- 172.21.15.7:0/2741610041 wait complete. 2023-03-03T08:43:39.289 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T08:43:39.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:43:39.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.412+0000 7f418233e700 1 Processor -- start 2023-03-03T08:43:39.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.412+0000 7f418233e700 1 -- start start 2023-03-03T08:43:39.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.413+0000 7f418233e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f417c14fc50 0x7f417c0c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.413+0000 7f418233e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 0x7f417c0c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.413+0000 7f418233e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f417c0c3e30 0x7f417c0c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.413+0000 7f418233e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f417c0c30c0 con 0x7f417c0c3e30 2023-03-03T08:43:39.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.413+0000 7f418233e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f417c0ad2b0 con 0x7f417c14fc50 2023-03-03T08:43:39.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.413+0000 7f418233e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417c156420 con 0x7f417c06d240 2023-03-03T08:43:39.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.413+0000 7f41808db700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f417c0c3e30 0x7f417c0c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.413+0000 7f41808db700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f417c0c3e30 0x7f417c0c42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51218/0 (socket says 172.21.15.7:51218) 2023-03-03T08:43:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.413+0000 7f41808db700 1 -- 172.21.15.7:0/2260235602 learned_addr learned my addr 172.21.15.7:0/2260235602 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.413+0000 7f417b7fe700 1 --2- 172.21.15.7:0/2260235602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 0x7f417c0c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f417bfff700 1 --2- 172.21.15.7:0/2260235602 >> v2:172.21.15.143:3300/0 conn(0x7f417c14fc50 0x7f417c0c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f41808db700 1 -- 172.21.15.7:0/2260235602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 msgr2=0x7f417c0c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f41808db700 1 --2- 172.21.15.7:0/2260235602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 0x7f417c0c27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f41808db700 1 -- 172.21.15.7:0/2260235602 >> v2:172.21.15.143:3300/0 conn(0x7f417c14fc50 msgr2=0x7f417c0c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f41808db700 1 --2- 172.21.15.7:0/2260235602 >> v2:172.21.15.143:3300/0 conn(0x7f417c14fc50 0x7f417c0c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f41808db700 1 -- 172.21.15.7:0/2260235602 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c0ad520 con 0x7f417c0c3e30 2023-03-03T08:43:39.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f417b7fe700 1 --2- 172.21.15.7:0/2260235602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 0x7f417c0c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:39.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f417bfff700 1 --2- 172.21.15.7:0/2260235602 >> v2:172.21.15.143:3300/0 conn(0x7f417c14fc50 0x7f417c0c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:39.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f41808db700 1 --2- 172.21.15.7:0/2260235602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c0c3e30 0x7f417c0c42b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f417c070000 tx=0x7f417400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa9900099b69de4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:39.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f417affd700 1 -- 172.21.15.7:0/2260235602 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417400e070 con 0x7f417c0c3e30 2023-03-03T08:43:39.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f417affd700 1 -- 172.21.15.7:0/2260235602 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417400b9c0 con 0x7f417c0c3e30 2023-03-03T08:43:39.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f417affd700 1 -- 172.21.15.7:0/2260235602 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417400bd70 con 0x7f417c0c3e30 2023-03-03T08:43:39.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f418233e700 1 -- 172.21.15.7:0/2260235602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c0c3e30 msgr2=0x7f417c0c42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f418233e700 1 --2- 172.21.15.7:0/2260235602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c0c3e30 0x7f417c0c42b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f417c070000 tx=0x7f417400b330 comp rx=0 tx=0).stop 2023-03-03T08:43:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f418233e700 1 -- 172.21.15.7:0/2260235602 shutdown_connections 2023-03-03T08:43:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f418233e700 1 --2- 172.21.15.7:0/2260235602 >> v2:172.21.15.143:3300/0 conn(0x7f417c14fc50 0x7f417c0c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f418233e700 1 --2- 172.21.15.7:0/2260235602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c0c3e30 0x7f417c0c42b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.414+0000 7f418233e700 1 --2- 172.21.15.7:0/2260235602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 0x7f417c0c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.415+0000 7f418233e700 1 -- 172.21.15.7:0/2260235602 >> 172.21.15.7:0/2260235602 conn(0x7f417c06c110 msgr2=0x7f417c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:39.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.415+0000 7f418233e700 1 -- 172.21.15.7:0/2260235602 shutdown_connections 2023-03-03T08:43:39.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.415+0000 7f418233e700 1 -- 172.21.15.7:0/2260235602 wait complete. 2023-03-03T08:43:39.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.416+0000 7f418233e700 1 Processor -- start 2023-03-03T08:43:39.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.416+0000 7f418233e700 1 -- start start 2023-03-03T08:43:39.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f418233e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 0x7f417c1e8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f418233e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f417c0c3e30 0x7f417c1df0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f418233e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c14fc50 0x7f417c1dbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f418233e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c145d60 con 0x7f417c14fc50 2023-03-03T08:43:39.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f418233e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c146040 con 0x7f417c0c3e30 2023-03-03T08:43:39.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f418233e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417c146320 con 0x7f417c06d240 2023-03-03T08:43:39.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f41808db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c14fc50 0x7f417c1dbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f41808db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c14fc50 0x7f417c1dbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51224/0 (socket says 172.21.15.7:51224) 2023-03-03T08:43:39.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f41808db700 1 -- 172.21.15.7:0/3762956047 learned_addr learned my addr 172.21.15.7:0/3762956047 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:39.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f417bfff700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 0x7f417c1e8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.417+0000 7f417b7fe700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f417c0c3e30 0x7f417c1df0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f41808db700 1 -- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 msgr2=0x7f417c1e8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f41808db700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 0x7f417c1e8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f41808db700 1 -- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f417c0c3e30 msgr2=0x7f417c1df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f41808db700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f417c0c3e30 0x7f417c1df0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f41808db700 1 -- 172.21.15.7:0/3762956047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4174009b30 con 0x7f417c14fc50 2023-03-03T08:43:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f41808db700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c14fc50 0x7f417c1dbca0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f417c06cff0 tx=0x7f4174004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f4178ff9700 1 -- 172.21.15.7:0/3762956047 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41740099b0 con 0x7f417c14fc50 2023-03-03T08:43:39.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f4178ff9700 1 -- 172.21.15.7:0/3762956047 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4174004250 con 0x7f417c14fc50 2023-03-03T08:43:39.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f418233e700 1 -- 172.21.15.7:0/3762956047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417c0c2cf0 con 0x7f417c14fc50 2023-03-03T08:43:39.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f4178ff9700 1 -- 172.21.15.7:0/3762956047 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174004600 con 0x7f417c14fc50 2023-03-03T08:43:39.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.418+0000 7f418233e700 1 -- 172.21.15.7:0/3762956047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417c0c2490 con 0x7f417c14fc50 2023-03-03T08:43:39.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.419+0000 7f418233e700 1 -- 172.21.15.7:0/3762956047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417c074220 con 0x7f417c14fc50 2023-03-03T08:43:39.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.419+0000 7f4178ff9700 1 -- 172.21.15.7:0/3762956047 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4174004760 con 0x7f417c14fc50 2023-03-03T08:43:39.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.420+0000 7f4178ff9700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 0x7f416806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.420+0000 7f4178ff9700 1 -- 172.21.15.7:0/3762956047 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f4174086e30 con 0x7f417c14fc50 2023-03-03T08:43:39.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.420+0000 7f417bfff700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 0x7f416806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.421+0000 7f417bfff700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 0x7f416806a7a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f416c00c940 tx=0x7f416c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:39.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.424+0000 7f4178ff9700 1 -- 172.21.15.7:0/3762956047 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4174053750 con 0x7f417c14fc50 2023-03-03T08:43:39.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.612+0000 7f418233e700 1 -- 172.21.15.7:0/3762956047 --> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f417c145ea0 con 0x7f41680682f0 2023-03-03T08:43:39.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.621+0000 7f4178ff9700 1 -- 172.21.15.7:0/3762956047 <== mgr.4102 v2:172.21.15.143:6832/96367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286764 (secure 0 0 0) 0x7f417c145ea0 con 0x7f41680682f0 2023-03-03T08:43:39.625 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.624+0000 7f418233e700 1 -- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 msgr2=0x7f416806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.624+0000 7f418233e700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 0x7f416806a7a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f416c00c940 tx=0x7f416c008040 comp rx=0 tx=0).stop 2023-03-03T08:43:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.624+0000 7f418233e700 1 -- 172.21.15.7:0/3762956047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c14fc50 msgr2=0x7f417c1dbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.624+0000 7f418233e700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c14fc50 0x7f417c1dbca0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f417c06cff0 tx=0x7f4174004ad0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.625+0000 7f418233e700 1 -- 172.21.15.7:0/3762956047 shutdown_connections 2023-03-03T08:43:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.625+0000 7f418233e700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 0x7f416806a7a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.625+0000 7f418233e700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f417c14fc50 0x7f417c1dbca0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.625+0000 7f418233e700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f417c06d240 0x7f417c1e8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.625+0000 7f418233e700 1 --2- 172.21.15.7:0/3762956047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f417c0c3e30 0x7f417c1df0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.625+0000 7f418233e700 1 -- 172.21.15.7:0/3762956047 >> 172.21.15.7:0/3762956047 conn(0x7f417c06c110 msgr2=0x7f417c0aac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:39.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.625+0000 7f418233e700 1 -- 172.21.15.7:0/3762956047 shutdown_connections 2023-03-03T08:43:39.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.625+0000 7f418233e700 1 -- 172.21.15.7:0/3762956047 wait complete. 2023-03-03T08:43:39.631 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-03-03T08:43:39.638 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-03-03T08:43:39.106315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66112,"kb_used_data":3008,"kb_used_omap":0,"kb_used_meta":63040,"kb_avail":754908608,"statfs":{"total":773094113280,"available":773026414592,"internally_reserved":0,"allocated":3080192,"data_stored":1850057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64552960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001620"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.550421+0000","last_change":"2023-03-03T08:43:34.546272+0000","last_active":"2023-03-03T08:43:34.550421+0000","last_peered":"2023-03-03T08:43:34.550421+0000","last_clean":"2023-03-03T08:43:34.550421+0000","last_became_active":"2023-03-03T08:43:34.546006+0000","last_became_peered":"2023-03-03T08:43:34.546006+0000","last_unstale":"2023-03-03T08:43:34.550421+0000","last_undegraded":"2023-03-03T08:43:34.550421+0000","last_fullsized":"2023-03-03T08:43:34.550421+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:15.121761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530001+0000","last_change":"2023-03-03T08:43:34.541669+0000","last_active":"2023-03-03T08:43:35.530001+0000","last_peered":"2023-03-03T08:43:35.530001+0000","last_clean":"2023-03-03T08:43:35.530001+0000","last_became_active":"2023-03-03T08:43:34.541104+0000","last_became_peered":"2023-03-03T08:43:34.541104+0000","last_unstale":"2023-03-03T08:43:35.530001+0000","last_undegraded":"2023-03-03T08:43:35.530001+0000","last_fullsized":"2023-03-03T08:43:35.530001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.551432+0000","last_change":"2023-03-03T08:43:34.544721+0000","last_active":"2023-03-03T08:43:34.551432+0000","last_peered":"2023-03-03T08:43:34.551432+0000","last_clean":"2023-03-03T08:43:34.551432+0000","last_became_active":"2023-03-03T08:43:34.544461+0000","last_became_peered":"2023-03-03T08:43:34.544461+0000","last_unstale":"2023-03-03T08:43:34.551432+0000","last_undegraded":"2023-03-03T08:43:34.551432+0000","last_fullsized":"2023-03-03T08:43:34.551432+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:52:11.890779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.549458+0000","last_change":"2023-03-03T08:43:34.538467+0000","last_active":"2023-03-03T08:43:34.549458+0000","last_peered":"2023-03-03T08:43:34.549458+0000","last_clean":"2023-03-03T08:43:34.549458+0000","last_became_active":"2023-03-03T08:43:34.538207+0000","last_became_peered":"2023-03-03T08:43:34.538207+0000","last_unstale":"2023-03-03T08:43:34.549458+0000","last_undegraded":"2023-03-03T08:43:34.549458+0000","last_fullsized":"2023-03-03T08:43:34.549458+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:14.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114429+0000","last_change":"2023-03-03T08:43:34.543620+0000","last_active":"2023-03-03T08:43:36.114429+0000","last_peered":"2023-03-03T08:43:36.114429+0000","last_clean":"2023-03-03T08:43:36.114429+0000","last_became_active":"2023-03-03T08:43:34.543370+0000","last_became_peered":"2023-03-03T08:43:34.543370+0000","last_unstale":"2023-03-03T08:43:36.114429+0000","last_undegraded":"2023-03-03T08:43:36.114429+0000","last_fullsized":"2023-03-03T08:43:36.114429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:40:32.808203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529270+0000","last_change":"2023-03-03T08:43:34.537664+0000","last_active":"2023-03-03T08:43:35.529270+0000","last_peered":"2023-03-03T08:43:35.529270+0000","last_clean":"2023-03-03T08:43:35.529270+0000","last_became_active":"2023-03-03T08:43:34.537214+0000","last_became_peered":"2023-03-03T08:43:34.537214+0000","last_unstale":"2023-03-03T08:43:35.529270+0000","last_undegraded":"2023-03-03T08:43:35.529270+0000","last_fullsized":"2023-03-03T08:43:35.529270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:54:09.222135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529177+0000","last_change":"2023-03-03T08:43:34.533673+0000","last_active":"2023-03-03T08:43:35.529177+0000","last_peered":"2023-03-03T08:43:35.529177+0000","last_clean":"2023-03-03T08:43:35.529177+0000","last_became_active":"2023-03-03T08:43:34.532283+0000","last_became_peered":"2023-03-03T08:43:34.532283+0000","last_unstale":"2023-03-03T08:43:35.529177+0000","last_undegraded":"2023-03-03T08:43:35.529177+0000","last_fullsized":"2023-03-03T08:43:35.529177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:25.591188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130569+0000","last_change":"2023-03-03T08:43:34.533496+0000","last_active":"2023-03-03T08:43:36.130569+0000","last_peered":"2023-03-03T08:43:36.130569+0000","last_clean":"2023-03-03T08:43:36.130569+0000","last_became_active":"2023-03-03T08:43:34.532714+0000","last_became_peered":"2023-03-03T08:43:34.532714+0000","last_unstale":"2023-03-03T08:43:36.130569+0000","last_undegraded":"2023-03-03T08:43:36.130569+0000","last_fullsized":"2023-03-03T08:43:36.130569+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:47.800807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114381+0000","last_change":"2023-03-03T08:43:34.549838+0000","last_active":"2023-03-03T08:43:36.114381+0000","last_peered":"2023-03-03T08:43:36.114381+0000","last_clean":"2023-03-03T08:43:36.114381+0000","last_became_active":"2023-03-03T08:43:34.549537+0000","last_became_peered":"2023-03-03T08:43:34.549537+0000","last_unstale":"2023-03-03T08:43:36.114381+0000","last_undegraded":"2023-03-03T08:43:36.114381+0000","last_fullsized":"2023-03-03T08:43:36.114381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:36.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131020+0000","last_change":"2023-03-03T08:43:34.548229+0000","last_active":"2023-03-03T08:43:36.131020+0000","last_peered":"2023-03-03T08:43:36.131020+0000","last_clean":"2023-03-03T08:43:36.131020+0000","last_became_active":"2023-03-03T08:43:34.548013+0000","last_became_peered":"2023-03-03T08:43:34.548013+0000","last_unstale":"2023-03-03T08:43:36.131020+0000","last_undegraded":"2023-03-03T08:43:36.131020+0000","last_fullsized":"2023-03-03T08:43:36.131020+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:27.793716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131431+0000","last_change":"2023-03-03T08:43:34.550889+0000","last_active":"2023-03-03T08:43:36.131431+0000","last_peered":"2023-03-03T08:43:36.131431+0000","last_clean":"2023-03-03T08:43:36.131431+0000","last_became_active":"2023-03-03T08:43:34.550605+0000","last_became_peered":"2023-03-03T08:43:34.550605+0000","last_unstale":"2023-03-03T08:43:36.131431+0000","last_undegraded":"2023-03-03T08:43:36.131431+0000","last_fullsized":"2023-03-03T08:43:36.131431+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:01.118555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.555833+0000","last_change":"2023-03-03T08:43:34.553434+0000","last_active":"2023-03-03T08:43:34.555833+0000","last_peered":"2023-03-03T08:43:34.555833+0000","last_clean":"2023-03-03T08:43:34.555833+0000","last_became_active":"2023-03-03T08:43:34.553125+0000","last_became_peered":"2023-03-03T08:43:34.553125+0000","last_unstale":"2023-03-03T08:43:34.555833+0000","last_undegraded":"2023-03-03T08:43:34.555833+0000","last_fullsized":"2023-03-03T08:43:34.555833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:41:17.553506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131312+0000","last_change":"2023-03-03T08:43:34.544013+0000","last_active":"2023-03-03T08:43:36.131312+0000","last_peered":"2023-03-03T08:43:36.131312+0000","last_clean":"2023-03-03T08:43:36.131312+0000","last_became_active":"2023-03-03T08:43:34.543745+0000","last_became_peered":"2023-03-03T08:43:34.543745+0000","last_unstale":"2023-03-03T08:43:36.131312+0000","last_undegraded":"2023-03-03T08:43:36.131312+0000","last_fullsized":"2023-03-03T08:43:36.131312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:05.647977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131242+0000","last_change":"2023-03-03T08:43:34.549026+0000","last_active":"2023-03-03T08:43:36.131242+0000","last_peered":"2023-03-03T08:43:36.131242+0000","last_clean":"2023-03-03T08:43:36.131242+0000","last_became_active":"2023-03-03T08:43:34.548778+0000","last_became_peered":"2023-03-03T08:43:34.548778+0000","last_unstale":"2023-03-03T08:43:36.131242+0000","last_undegraded":"2023-03-03T08:43:36.131242+0000","last_fullsized":"2023-03-03T08:43:36.131242+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:41.835304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.551257+0000","last_change":"2023-03-03T08:43:34.542729+0000","last_active":"2023-03-03T08:43:34.551257+0000","last_peered":"2023-03-03T08:43:34.551257+0000","last_clean":"2023-03-03T08:43:34.551257+0000","last_became_active":"2023-03-03T08:43:34.542246+0000","last_became_peered":"2023-03-03T08:43:34.542246+0000","last_unstale":"2023-03-03T08:43:34.551257+0000","last_undegraded":"2023-03-03T08:43:34.551257+0000","last_fullsized":"2023-03-03T08:43:34.551257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:14:59.995526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.528932+0000","last_change":"2023-03-03T08:43:34.527373+0000","last_active":"2023-03-03T08:43:35.528932+0000","last_peered":"2023-03-03T08:43:35.528932+0000","last_clean":"2023-03-03T08:43:35.528932+0000","last_became_active":"2023-03-03T08:43:34.527171+0000","last_became_peered":"2023-03-03T08:43:34.527171+0000","last_unstale":"2023-03-03T08:43:35.528932+0000","last_undegraded":"2023-03-03T08:43:35.528932+0000","last_fullsized":"2023-03-03T08:43:35.528932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:20:51.138587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130928+0000","last_change":"2023-03-03T08:43:34.532154+0000","last_active":"2023-03-03T08:43:36.130928+0000","last_peered":"2023-03-03T08:43:36.130928+0000","last_clean":"2023-03-03T08:43:36.130928+0000","last_became_active":"2023-03-03T08:43:34.531844+0000","last_became_peered":"2023-03-03T08:43:34.531844+0000","last_unstale":"2023-03-03T08:43:36.130928+0000","last_undegraded":"2023-03-03T08:43:36.130928+0000","last_fullsized":"2023-03-03T08:43:36.130928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:00.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.549428+0000","last_change":"2023-03-03T08:43:34.537110+0000","last_active":"2023-03-03T08:43:34.549428+0000","last_peered":"2023-03-03T08:43:34.549428+0000","last_clean":"2023-03-03T08:43:34.549428+0000","last_became_active":"2023-03-03T08:43:34.536848+0000","last_became_peered":"2023-03-03T08:43:34.536848+0000","last_unstale":"2023-03-03T08:43:34.549428+0000","last_undegraded":"2023-03-03T08:43:34.549428+0000","last_fullsized":"2023-03-03T08:43:34.549428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.555844+0000","last_change":"2023-03-03T08:43:34.553589+0000","last_active":"2023-03-03T08:43:34.555844+0000","last_peered":"2023-03-03T08:43:34.555844+0000","last_clean":"2023-03-03T08:43:34.555844+0000","last_became_active":"2023-03-03T08:43:34.553242+0000","last_became_peered":"2023-03-03T08:43:34.553242+0000","last_unstale":"2023-03-03T08:43:34.555844+0000","last_undegraded":"2023-03-03T08:43:34.555844+0000","last_fullsized":"2023-03-03T08:43:34.555844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:09.000051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130506+0000","last_change":"2023-03-03T08:43:34.547003+0000","last_active":"2023-03-03T08:43:36.130506+0000","last_peered":"2023-03-03T08:43:36.130506+0000","last_clean":"2023-03-03T08:43:36.130506+0000","last_became_active":"2023-03-03T08:43:34.546744+0000","last_became_peered":"2023-03-03T08:43:34.546744+0000","last_unstale":"2023-03-03T08:43:36.130506+0000","last_undegraded":"2023-03-03T08:43:36.130506+0000","last_fullsized":"2023-03-03T08:43:36.130506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:53:38.996605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131059+0000","last_change":"2023-03-03T08:43:34.549837+0000","last_active":"2023-03-03T08:43:36.131059+0000","last_peered":"2023-03-03T08:43:36.131059+0000","last_clean":"2023-03-03T08:43:36.131059+0000","last_became_active":"2023-03-03T08:43:34.549542+0000","last_became_peered":"2023-03-03T08:43:34.549542+0000","last_unstale":"2023-03-03T08:43:36.131059+0000","last_undegraded":"2023-03-03T08:43:36.131059+0000","last_fullsized":"2023-03-03T08:43:36.131059+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:31.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130796+0000","last_change":"2023-03-03T08:43:34.542661+0000","last_active":"2023-03-03T08:43:36.130796+0000","last_peered":"2023-03-03T08:43:36.130796+0000","last_clean":"2023-03-03T08:43:36.130796+0000","last_became_active":"2023-03-03T08:43:34.542353+0000","last_became_peered":"2023-03-03T08:43:34.542353+0000","last_unstale":"2023-03-03T08:43:36.130796+0000","last_undegraded":"2023-03-03T08:43:36.130796+0000","last_fullsized":"2023-03-03T08:43:36.130796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:38.730662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114146+0000","last_change":"2023-03-03T08:43:34.549771+0000","last_active":"2023-03-03T08:43:36.114146+0000","last_peered":"2023-03-03T08:43:36.114146+0000","last_clean":"2023-03-03T08:43:36.114146+0000","last_became_active":"2023-03-03T08:43:34.549479+0000","last_became_peered":"2023-03-03T08:43:34.549479+0000","last_unstale":"2023-03-03T08:43:36.114146+0000","last_undegraded":"2023-03-03T08:43:36.114146+0000","last_fullsized":"2023-03-03T08:43:36.114146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:41:03.902019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130554+0000","last_change":"2023-03-03T08:43:34.548469+0000","last_active":"2023-03-03T08:43:36.130554+0000","last_peered":"2023-03-03T08:43:36.130554+0000","last_clean":"2023-03-03T08:43:36.130554+0000","last_became_active":"2023-03-03T08:43:34.547647+0000","last_became_peered":"2023-03-03T08:43:34.547647+0000","last_unstale":"2023-03-03T08:43:36.130554+0000","last_undegraded":"2023-03-03T08:43:36.130554+0000","last_fullsized":"2023-03-03T08:43:36.130554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:40.054781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114484+0000","last_change":"2023-03-03T08:43:34.542858+0000","last_active":"2023-03-03T08:43:36.114484+0000","last_peered":"2023-03-03T08:43:36.114484+0000","last_clean":"2023-03-03T08:43:36.114484+0000","last_became_active":"2023-03-03T08:43:34.542510+0000","last_became_peered":"2023-03-03T08:43:34.542510+0000","last_unstale":"2023-03-03T08:43:36.114484+0000","last_undegraded":"2023-03-03T08:43:36.114484+0000","last_fullsized":"2023-03-03T08:43:36.114484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:21.471234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131271+0000","last_change":"2023-03-03T08:43:34.540762+0000","last_active":"2023-03-03T08:43:36.131271+0000","last_peered":"2023-03-03T08:43:36.131271+0000","last_clean":"2023-03-03T08:43:36.131271+0000","last_became_active":"2023-03-03T08:43:34.540467+0000","last_became_peered":"2023-03-03T08:43:34.540467+0000","last_unstale":"2023-03-03T08:43:36.131271+0000","last_undegraded":"2023-03-03T08:43:36.131271+0000","last_fullsized":"2023-03-03T08:43:36.131271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530126+0000","last_change":"2023-03-03T08:43:34.533987+0000","last_active":"2023-03-03T08:43:35.530126+0000","last_peered":"2023-03-03T08:43:35.530126+0000","last_clean":"2023-03-03T08:43:35.530126+0000","last_became_active":"2023-03-03T08:43:34.533729+0000","last_became_peered":"2023-03-03T08:43:34.533729+0000","last_unstale":"2023-03-03T08:43:35.530126+0000","last_undegraded":"2023-03-03T08:43:35.530126+0000","last_fullsized":"2023-03-03T08:43:35.530126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:18:12.407151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131293+0000","last_change":"2023-03-03T08:43:34.108056+0000","last_active":"2023-03-03T08:43:36.131293+0000","last_peered":"2023-03-03T08:43:36.131293+0000","last_clean":"2023-03-03T08:43:36.131293+0000","last_became_active":"2023-03-03T08:43:34.107708+0000","last_became_peered":"2023-03-03T08:43:34.107708+0000","last_unstale":"2023-03-03T08:43:36.131293+0000","last_undegraded":"2023-03-03T08:43:36.131293+0000","last_fullsized":"2023-03-03T08:43:36.131293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:10.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130603+0000","last_change":"2023-03-03T08:43:34.537301+0000","last_active":"2023-03-03T08:43:36.130603+0000","last_peered":"2023-03-03T08:43:36.130603+0000","last_clean":"2023-03-03T08:43:36.130603+0000","last_became_active":"2023-03-03T08:43:34.537007+0000","last_became_peered":"2023-03-03T08:43:34.537007+0000","last_unstale":"2023-03-03T08:43:36.130603+0000","last_undegraded":"2023-03-03T08:43:36.130603+0000","last_fullsized":"2023-03-03T08:43:36.130603+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:59:49.124140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114026+0000","last_change":"2023-03-03T08:43:34.108187+0000","last_active":"2023-03-03T08:43:36.114026+0000","last_peered":"2023-03-03T08:43:36.114026+0000","last_clean":"2023-03-03T08:43:36.114026+0000","last_became_active":"2023-03-03T08:43:34.107859+0000","last_became_peered":"2023-03-03T08:43:34.107859+0000","last_unstale":"2023-03-03T08:43:36.114026+0000","last_undegraded":"2023-03-03T08:43:36.114026+0000","last_fullsized":"2023-03-03T08:43:36.114026+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:46:40.595756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114408+0000","last_change":"2023-03-03T08:43:34.549123+0000","last_active":"2023-03-03T08:43:36.114408+0000","last_peered":"2023-03-03T08:43:36.114408+0000","last_clean":"2023-03-03T08:43:36.114408+0000","last_became_active":"2023-03-03T08:43:34.548839+0000","last_became_peered":"2023-03-03T08:43:34.548839+0000","last_unstale":"2023-03-03T08:43:36.114408+0000","last_undegraded":"2023-03-03T08:43:36.114408+0000","last_fullsized":"2023-03-03T08:43:36.114408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:38.134009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.539564+0000","last_change":"2023-03-03T08:43:34.106505+0000","last_active":"2023-03-03T08:43:34.539564+0000","last_peered":"2023-03-03T08:43:34.539564+0000","last_clean":"2023-03-03T08:43:34.539564+0000","last_became_active":"2023-03-03T08:43:34.106079+0000","last_became_peered":"2023-03-03T08:43:34.106079+0000","last_unstale":"2023-03-03T08:43:34.539564+0000","last_undegraded":"2023-03-03T08:43:34.539564+0000","last_fullsized":"2023-03-03T08:43:34.539564+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:48.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530817+0000","last_change":"2023-03-03T08:43:34.547450+0000","last_active":"2023-03-03T08:43:35.530817+0000","last_peered":"2023-03-03T08:43:35.530817+0000","last_clean":"2023-03-03T08:43:35.530817+0000","last_became_active":"2023-03-03T08:43:34.547168+0000","last_became_peered":"2023-03-03T08:43:34.547168+0000","last_unstale":"2023-03-03T08:43:35.530817+0000","last_undegraded":"2023-03-03T08:43:35.530817+0000","last_fullsized":"2023-03-03T08:43:35.530817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:17.723229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.539993+0000","last_change":"2023-03-03T08:43:34.110802+0000","last_active":"2023-03-03T08:43:34.539993+0000","last_peered":"2023-03-03T08:43:34.539993+0000","last_clean":"2023-03-03T08:43:34.539993+0000","last_became_active":"2023-03-03T08:43:34.110519+0000","last_became_peered":"2023-03-03T08:43:34.110519+0000","last_unstale":"2023-03-03T08:43:34.539993+0000","last_undegraded":"2023-03-03T08:43:34.539993+0000","last_fullsized":"2023-03-03T08:43:34.539993+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:02:18.355043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131083+0000","last_change":"2023-03-03T08:43:34.528573+0000","last_active":"2023-03-03T08:43:36.131083+0000","last_peered":"2023-03-03T08:43:36.131083+0000","last_clean":"2023-03-03T08:43:36.131083+0000","last_became_active":"2023-03-03T08:43:34.528278+0000","last_became_peered":"2023-03-03T08:43:34.528278+0000","last_unstale":"2023-03-03T08:43:36.131083+0000","last_undegraded":"2023-03-03T08:43:36.131083+0000","last_fullsized":"2023-03-03T08:43:36.131083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:51.468323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530140+0000","last_change":"2023-03-03T08:43:34.113168+0000","last_active":"2023-03-03T08:43:35.530140+0000","last_peered":"2023-03-03T08:43:35.530140+0000","last_clean":"2023-03-03T08:43:35.530140+0000","last_became_active":"2023-03-03T08:43:34.112731+0000","last_became_peered":"2023-03-03T08:43:34.112731+0000","last_unstale":"2023-03-03T08:43:35.530140+0000","last_undegraded":"2023-03-03T08:43:35.530140+0000","last_fullsized":"2023-03-03T08:43:35.530140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:57:59.463191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.113969+0000","last_change":"2023-03-03T08:43:34.538703+0000","last_active":"2023-03-03T08:43:36.113969+0000","last_peered":"2023-03-03T08:43:36.113969+0000","last_clean":"2023-03-03T08:43:36.113969+0000","last_became_active":"2023-03-03T08:43:34.538418+0000","last_became_peered":"2023-03-03T08:43:34.538418+0000","last_unstale":"2023-03-03T08:43:36.113969+0000","last_undegraded":"2023-03-03T08:43:36.113969+0000","last_fullsized":"2023-03-03T08:43:36.113969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:02.795203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.551383+0000","last_change":"2023-03-03T08:43:34.110810+0000","last_active":"2023-03-03T08:43:34.551383+0000","last_peered":"2023-03-03T08:43:34.551383+0000","last_clean":"2023-03-03T08:43:34.551383+0000","last_became_active":"2023-03-03T08:43:34.110483+0000","last_became_peered":"2023-03-03T08:43:34.110483+0000","last_unstale":"2023-03-03T08:43:34.551383+0000","last_undegraded":"2023-03-03T08:43:34.551383+0000","last_fullsized":"2023-03-03T08:43:34.551383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:06.970615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114208+0000","last_change":"2023-03-03T08:43:34.544694+0000","last_active":"2023-03-03T08:43:36.114208+0000","last_peered":"2023-03-03T08:43:36.114208+0000","last_clean":"2023-03-03T08:43:36.114208+0000","last_became_active":"2023-03-03T08:43:34.544399+0000","last_became_peered":"2023-03-03T08:43:34.544399+0000","last_unstale":"2023-03-03T08:43:36.114208+0000","last_undegraded":"2023-03-03T08:43:36.114208+0000","last_fullsized":"2023-03-03T08:43:36.114208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530076+0000","last_change":"2023-03-03T08:43:34.117609+0000","last_active":"2023-03-03T08:43:35.530076+0000","last_peered":"2023-03-03T08:43:35.530076+0000","last_clean":"2023-03-03T08:43:35.530076+0000","last_became_active":"2023-03-03T08:43:34.117271+0000","last_became_peered":"2023-03-03T08:43:34.117271+0000","last_unstale":"2023-03-03T08:43:35.530076+0000","last_undegraded":"2023-03-03T08:43:35.530076+0000","last_fullsized":"2023-03-03T08:43:35.530076+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:20.259259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529652+0000","last_change":"2023-03-03T08:43:34.537857+0000","last_active":"2023-03-03T08:43:35.529652+0000","last_peered":"2023-03-03T08:43:35.529652+0000","last_clean":"2023-03-03T08:43:35.529652+0000","last_became_active":"2023-03-03T08:43:34.537590+0000","last_became_peered":"2023-03-03T08:43:34.537590+0000","last_unstale":"2023-03-03T08:43:35.529652+0000","last_undegraded":"2023-03-03T08:43:35.529652+0000","last_fullsized":"2023-03-03T08:43:35.529652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:52.612521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530006+0000","last_change":"2023-03-03T08:43:34.112941+0000","last_active":"2023-03-03T08:43:35.530006+0000","last_peered":"2023-03-03T08:43:35.530006+0000","last_clean":"2023-03-03T08:43:35.530006+0000","last_became_active":"2023-03-03T08:43:34.112646+0000","last_became_peered":"2023-03-03T08:43:34.112646+0000","last_unstale":"2023-03-03T08:43:35.530006+0000","last_undegraded":"2023-03-03T08:43:35.530006+0000","last_fullsized":"2023-03-03T08:43:35.530006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:32.767710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131309+0000","last_change":"2023-03-03T08:43:34.539790+0000","last_active":"2023-03-03T08:43:36.131309+0000","last_peered":"2023-03-03T08:43:36.131309+0000","last_clean":"2023-03-03T08:43:36.131309+0000","last_became_active":"2023-03-03T08:43:34.539443+0000","last_became_peered":"2023-03-03T08:43:34.539443+0000","last_unstale":"2023-03-03T08:43:36.131309+0000","last_undegraded":"2023-03-03T08:43:36.131309+0000","last_fullsized":"2023-03-03T08:43:36.131309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:25.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131521+0000","last_change":"2023-03-03T08:43:34.112905+0000","last_active":"2023-03-03T08:43:36.131521+0000","last_peered":"2023-03-03T08:43:36.131521+0000","last_clean":"2023-03-03T08:43:36.131521+0000","last_became_active":"2023-03-03T08:43:34.112601+0000","last_became_peered":"2023-03-03T08:43:34.112601+0000","last_unstale":"2023-03-03T08:43:36.131521+0000","last_undegraded":"2023-03-03T08:43:36.131521+0000","last_fullsized":"2023-03-03T08:43:36.131521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:01.165783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530721+0000","last_change":"2023-03-03T08:43:34.532656+0000","last_active":"2023-03-03T08:43:35.530721+0000","last_peered":"2023-03-03T08:43:35.530721+0000","last_clean":"2023-03-03T08:43:35.530721+0000","last_became_active":"2023-03-03T08:43:34.532297+0000","last_became_peered":"2023-03-03T08:43:34.532297+0000","last_unstale":"2023-03-03T08:43:35.530721+0000","last_undegraded":"2023-03-03T08:43:35.530721+0000","last_fullsized":"2023-03-03T08:43:35.530721+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:36.347740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529262+0000","last_change":"2023-03-03T08:43:34.115792+0000","last_active":"2023-03-03T08:43:35.529262+0000","last_peered":"2023-03-03T08:43:35.529262+0000","last_clean":"2023-03-03T08:43:35.529262+0000","last_became_active":"2023-03-03T08:43:34.115431+0000","last_became_peered":"2023-03-03T08:43:34.115431+0000","last_unstale":"2023-03-03T08:43:35.529262+0000","last_undegraded":"2023-03-03T08:43:35.529262+0000","last_fullsized":"2023-03-03T08:43:35.529262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:15.484022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.550414+0000","last_change":"2023-03-03T08:43:34.546067+0000","last_active":"2023-03-03T08:43:34.550414+0000","last_peered":"2023-03-03T08:43:34.550414+0000","last_clean":"2023-03-03T08:43:34.550414+0000","last_became_active":"2023-03-03T08:43:34.543786+0000","last_became_peered":"2023-03-03T08:43:34.543786+0000","last_unstale":"2023-03-03T08:43:34.550414+0000","last_undegraded":"2023-03-03T08:43:34.550414+0000","last_fullsized":"2023-03-03T08:43:34.550414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:32.922812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130446+0000","last_change":"2023-03-03T08:43:34.108123+0000","last_active":"2023-03-03T08:43:36.130446+0000","last_peered":"2023-03-03T08:43:36.130446+0000","last_clean":"2023-03-03T08:43:36.130446+0000","last_became_active":"2023-03-03T08:43:34.107757+0000","last_became_peered":"2023-03-03T08:43:34.107757+0000","last_unstale":"2023-03-03T08:43:36.130446+0000","last_undegraded":"2023-03-03T08:43:36.130446+0000","last_fullsized":"2023-03-03T08:43:36.130446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:09.488930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.555860+0000","last_change":"2023-03-03T08:43:34.553597+0000","last_active":"2023-03-03T08:43:34.555860+0000","last_peered":"2023-03-03T08:43:34.555860+0000","last_clean":"2023-03-03T08:43:34.555860+0000","last_became_active":"2023-03-03T08:43:34.553322+0000","last_became_peered":"2023-03-03T08:43:34.553322+0000","last_unstale":"2023-03-03T08:43:34.555860+0000","last_undegraded":"2023-03-03T08:43:34.555860+0000","last_fullsized":"2023-03-03T08:43:34.555860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:24:17.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.552634+0000","last_change":"2023-03-03T08:43:34.110464+0000","last_active":"2023-03-03T08:43:34.552634+0000","last_peered":"2023-03-03T08:43:34.552634+0000","last_clean":"2023-03-03T08:43:34.552634+0000","last_became_active":"2023-03-03T08:43:34.110165+0000","last_became_peered":"2023-03-03T08:43:34.110165+0000","last_unstale":"2023-03-03T08:43:34.552634+0000","last_undegraded":"2023-03-03T08:43:34.552634+0000","last_fullsized":"2023-03-03T08:43:34.552634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:54.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529076+0000","last_change":"2023-03-03T08:43:34.540196+0000","last_active":"2023-03-03T08:43:35.529076+0000","last_peered":"2023-03-03T08:43:35.529076+0000","last_clean":"2023-03-03T08:43:35.529076+0000","last_became_active":"2023-03-03T08:43:34.539931+0000","last_became_peered":"2023-03-03T08:43:34.539931+0000","last_unstale":"2023-03-03T08:43:35.529076+0000","last_undegraded":"2023-03-03T08:43:35.529076+0000","last_fullsized":"2023-03-03T08:43:35.529076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:20.012997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530091+0000","last_change":"2023-03-03T08:43:34.109890+0000","last_active":"2023-03-03T08:43:35.530091+0000","last_peered":"2023-03-03T08:43:35.530091+0000","last_clean":"2023-03-03T08:43:35.530091+0000","last_became_active":"2023-03-03T08:43:34.109583+0000","last_became_peered":"2023-03-03T08:43:34.109583+0000","last_unstale":"2023-03-03T08:43:35.530091+0000","last_undegraded":"2023-03-03T08:43:35.530091+0000","last_fullsized":"2023-03-03T08:43:35.530091+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:27:34.142294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130657+0000","last_change":"2023-03-03T08:43:34.548010+0000","last_active":"2023-03-03T08:43:36.130657+0000","last_peered":"2023-03-03T08:43:36.130657+0000","last_clean":"2023-03-03T08:43:36.130657+0000","last_became_active":"2023-03-03T08:43:34.547662+0000","last_became_peered":"2023-03-03T08:43:34.547662+0000","last_unstale":"2023-03-03T08:43:36.130657+0000","last_undegraded":"2023-03-03T08:43:36.130657+0000","last_fullsized":"2023-03-03T08:43:36.130657+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:52.200064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530215+0000","last_change":"2023-03-03T08:43:34.117734+0000","last_active":"2023-03-03T08:43:35.530215+0000","last_peered":"2023-03-03T08:43:35.530215+0000","last_clean":"2023-03-03T08:43:35.530215+0000","last_became_active":"2023-03-03T08:43:34.117454+0000","last_became_peered":"2023-03-03T08:43:34.117454+0000","last_unstale":"2023-03-03T08:43:35.530215+0000","last_undegraded":"2023-03-03T08:43:35.530215+0000","last_fullsized":"2023-03-03T08:43:35.530215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:42:27.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529314+0000","last_change":"2023-03-03T08:43:34.538817+0000","last_active":"2023-03-03T08:43:35.529314+0000","last_peered":"2023-03-03T08:43:35.529314+0000","last_clean":"2023-03-03T08:43:35.529314+0000","last_became_active":"2023-03-03T08:43:34.538506+0000","last_became_peered":"2023-03-03T08:43:34.538506+0000","last_unstale":"2023-03-03T08:43:35.529314+0000","last_undegraded":"2023-03-03T08:43:35.529314+0000","last_fullsized":"2023-03-03T08:43:35.529314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.531427+0000","last_change":"2023-03-03T08:43:34.109372+0000","last_active":"2023-03-03T08:43:34.531427+0000","last_peered":"2023-03-03T08:43:34.531427+0000","last_clean":"2023-03-03T08:43:34.531427+0000","last_became_active":"2023-03-03T08:43:34.109023+0000","last_became_peered":"2023-03-03T08:43:34.109023+0000","last_unstale":"2023-03-03T08:43:34.531427+0000","last_undegraded":"2023-03-03T08:43:34.531427+0000","last_fullsized":"2023-03-03T08:43:34.531427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:51.374188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.553945+0000","last_change":"2023-03-03T08:43:34.548304+0000","last_active":"2023-03-03T08:43:34.553945+0000","last_peered":"2023-03-03T08:43:34.553945+0000","last_clean":"2023-03-03T08:43:34.553945+0000","last_became_active":"2023-03-03T08:43:34.547960+0000","last_became_peered":"2023-03-03T08:43:34.547960+0000","last_unstale":"2023-03-03T08:43:34.553945+0000","last_undegraded":"2023-03-03T08:43:34.553945+0000","last_fullsized":"2023-03-03T08:43:34.553945+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:04.907753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130849+0000","last_change":"2023-03-03T08:43:34.109698+0000","last_active":"2023-03-03T08:43:36.130849+0000","last_peered":"2023-03-03T08:43:36.130849+0000","last_clean":"2023-03-03T08:43:36.130849+0000","last_became_active":"2023-03-03T08:43:34.109423+0000","last_became_peered":"2023-03-03T08:43:34.109423+0000","last_unstale":"2023-03-03T08:43:36.130849+0000","last_undegraded":"2023-03-03T08:43:36.130849+0000","last_fullsized":"2023-03-03T08:43:36.130849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:49.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.551105+0000","last_change":"2023-03-03T08:43:34.542308+0000","last_active":"2023-03-03T08:43:34.551105+0000","last_peered":"2023-03-03T08:43:34.551105+0000","last_clean":"2023-03-03T08:43:34.551105+0000","last_became_active":"2023-03-03T08:43:34.541991+0000","last_became_peered":"2023-03-03T08:43:34.541991+0000","last_unstale":"2023-03-03T08:43:34.551105+0000","last_undegraded":"2023-03-03T08:43:34.551105+0000","last_fullsized":"2023-03-03T08:43:34.551105+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:06:21.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529147+0000","last_change":"2023-03-03T08:43:34.106777+0000","last_active":"2023-03-03T08:43:35.529147+0000","last_peered":"2023-03-03T08:43:35.529147+0000","last_clean":"2023-03-03T08:43:35.529147+0000","last_became_active":"2023-03-03T08:43:34.106132+0000","last_became_peered":"2023-03-03T08:43:34.106132+0000","last_unstale":"2023-03-03T08:43:35.529147+0000","last_undegraded":"2023-03-03T08:43:35.529147+0000","last_fullsized":"2023-03-03T08:43:35.529147+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:55.965987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530549+0000","last_change":"2023-03-03T08:43:34.550325+0000","last_active":"2023-03-03T08:43:35.530549+0000","last_peered":"2023-03-03T08:43:35.530549+0000","last_clean":"2023-03-03T08:43:35.530549+0000","last_became_active":"2023-03-03T08:43:34.550041+0000","last_became_peered":"2023-03-03T08:43:34.550041+0000","last_unstale":"2023-03-03T08:43:35.530549+0000","last_undegraded":"2023-03-03T08:43:35.530549+0000","last_fullsized":"2023-03-03T08:43:35.530549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:44:34.082003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529221+0000","last_change":"2023-03-03T08:43:34.111335+0000","last_active":"2023-03-03T08:43:35.529221+0000","last_peered":"2023-03-03T08:43:35.529221+0000","last_clean":"2023-03-03T08:43:35.529221+0000","last_became_active":"2023-03-03T08:43:34.110837+0000","last_became_peered":"2023-03-03T08:43:34.110837+0000","last_unstale":"2023-03-03T08:43:35.529221+0000","last_undegraded":"2023-03-03T08:43:35.529221+0000","last_fullsized":"2023-03-03T08:43:35.529221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:03.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530054+0000","last_change":"2023-03-03T08:43:34.541858+0000","last_active":"2023-03-03T08:43:35.530054+0000","last_peered":"2023-03-03T08:43:35.530054+0000","last_clean":"2023-03-03T08:43:35.530054+0000","last_became_active":"2023-03-03T08:43:34.541430+0000","last_became_peered":"2023-03-03T08:43:34.541430+0000","last_unstale":"2023-03-03T08:43:35.530054+0000","last_undegraded":"2023-03-03T08:43:35.530054+0000","last_fullsized":"2023-03-03T08:43:35.530054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:29.397784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114026+0000","last_change":"2023-03-03T08:43:34.109764+0000","last_active":"2023-03-03T08:43:36.114026+0000","last_peered":"2023-03-03T08:43:36.114026+0000","last_clean":"2023-03-03T08:43:36.114026+0000","last_became_active":"2023-03-03T08:43:34.109462+0000","last_became_peered":"2023-03-03T08:43:34.109462+0000","last_unstale":"2023-03-03T08:43:36.114026+0000","last_undegraded":"2023-03-03T08:43:36.114026+0000","last_fullsized":"2023-03-03T08:43:36.114026+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:56.399192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131050+0000","last_change":"2023-03-03T08:43:34.540006+0000","last_active":"2023-03-03T08:43:36.131050+0000","last_peered":"2023-03-03T08:43:36.131050+0000","last_clean":"2023-03-03T08:43:36.131050+0000","last_became_active":"2023-03-03T08:43:34.539579+0000","last_became_peered":"2023-03-03T08:43:34.539579+0000","last_unstale":"2023-03-03T08:43:36.131050+0000","last_undegraded":"2023-03-03T08:43:36.131050+0000","last_fullsized":"2023-03-03T08:43:36.131050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.106922+0000","last_change":"2023-03-03T08:43:29.107536+0000","last_active":"2023-03-03T08:43:34.106922+0000","last_peered":"2023-03-03T08:43:34.106922+0000","last_clean":"2023-03-03T08:43:34.106922+0000","last_became_active":"2023-03-03T08:43:29.107093+0000","last_became_peered":"2023-03-03T08:43:29.107093+0000","last_unstale":"2023-03-03T08:43:34.106922+0000","last_undegraded":"2023-03-03T08:43:34.106922+0000","last_fullsized":"2023-03-03T08:43:34.106922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:25:47.568904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529848+0000","last_change":"2023-03-03T08:43:34.104692+0000","last_active":"2023-03-03T08:43:35.529848+0000","last_peered":"2023-03-03T08:43:35.529848+0000","last_clean":"2023-03-03T08:43:35.529848+0000","last_became_active":"2023-03-03T08:43:34.104263+0000","last_became_peered":"2023-03-03T08:43:34.104263+0000","last_unstale":"2023-03-03T08:43:35.529848+0000","last_undegraded":"2023-03-03T08:43:35.529848+0000","last_fullsized":"2023-03-03T08:43:35.529848+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:24.537703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130617+0000","last_change":"2023-03-03T08:43:34.526797+0000","last_active":"2023-03-03T08:43:36.130617+0000","last_peered":"2023-03-03T08:43:36.130617+0000","last_clean":"2023-03-03T08:43:36.130617+0000","last_became_active":"2023-03-03T08:43:34.526490+0000","last_became_peered":"2023-03-03T08:43:34.526490+0000","last_unstale":"2023-03-03T08:43:36.130617+0000","last_undegraded":"2023-03-03T08:43:36.130617+0000","last_fullsized":"2023-03-03T08:43:36.130617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:11.808336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.521801+0000","last_change":"2023-03-03T08:43:29.107584+0000","last_active":"2023-03-03T08:43:34.521801+0000","last_peered":"2023-03-03T08:43:34.521801+0000","last_clean":"2023-03-03T08:43:34.521801+0000","last_became_active":"2023-03-03T08:43:29.107208+0000","last_became_peered":"2023-03-03T08:43:29.107208+0000","last_unstale":"2023-03-03T08:43:34.521801+0000","last_undegraded":"2023-03-03T08:43:34.521801+0000","last_fullsized":"2023-03-03T08:43:34.521801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:28:24.087122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.531478+0000","last_change":"2023-03-03T08:43:34.113748+0000","last_active":"2023-03-03T08:43:34.531478+0000","last_peered":"2023-03-03T08:43:34.531478+0000","last_clean":"2023-03-03T08:43:34.531478+0000","last_became_active":"2023-03-03T08:43:34.113190+0000","last_became_peered":"2023-03-03T08:43:34.113190+0000","last_unstale":"2023-03-03T08:43:34.531478+0000","last_undegraded":"2023-03-03T08:43:34.531478+0000","last_fullsized":"2023-03-03T08:43:34.531478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:58:55.343571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131714+0000","last_change":"2023-03-03T08:43:34.528593+0000","last_active":"2023-03-03T08:43:36.131714+0000","last_peered":"2023-03-03T08:43:36.131714+0000","last_clean":"2023-03-03T08:43:36.131714+0000","last_became_active":"2023-03-03T08:43:34.528260+0000","last_became_peered":"2023-03-03T08:43:34.528260+0000","last_unstale":"2023-03-03T08:43:36.131714+0000","last_undegraded":"2023-03-03T08:43:36.131714+0000","last_fullsized":"2023-03-03T08:43:36.131714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.524683+0000","last_change":"2023-03-03T08:43:29.076367+0000","last_active":"2023-03-03T08:43:34.524683+0000","last_peered":"2023-03-03T08:43:34.524683+0000","last_clean":"2023-03-03T08:43:34.524683+0000","last_became_active":"2023-03-03T08:43:29.075982+0000","last_became_peered":"2023-03-03T08:43:29.075982+0000","last_unstale":"2023-03-03T08:43:34.524683+0000","last_undegraded":"2023-03-03T08:43:34.524683+0000","last_fullsized":"2023-03-03T08:43:34.524683+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:06.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.539766+0000","last_change":"2023-03-03T08:43:34.106337+0000","last_active":"2023-03-03T08:43:34.539766+0000","last_peered":"2023-03-03T08:43:34.539766+0000","last_clean":"2023-03-03T08:43:34.539766+0000","last_became_active":"2023-03-03T08:43:34.105753+0000","last_became_peered":"2023-03-03T08:43:34.105753+0000","last_unstale":"2023-03-03T08:43:34.539766+0000","last_undegraded":"2023-03-03T08:43:34.539766+0000","last_fullsized":"2023-03-03T08:43:34.539766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:46.761924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.132065+0000","last_change":"2023-03-03T08:43:34.536419+0000","last_active":"2023-03-03T08:43:36.132065+0000","last_peered":"2023-03-03T08:43:36.132065+0000","last_clean":"2023-03-03T08:43:36.132065+0000","last_became_active":"2023-03-03T08:43:34.536150+0000","last_became_peered":"2023-03-03T08:43:34.536150+0000","last_unstale":"2023-03-03T08:43:36.132065+0000","last_undegraded":"2023-03-03T08:43:36.132065+0000","last_fullsized":"2023-03-03T08:43:36.132065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:22:54.362024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.521122+0000","last_change":"2023-03-03T08:43:29.160285+0000","last_active":"2023-03-03T08:43:34.521122+0000","last_peered":"2023-03-03T08:43:34.521122+0000","last_clean":"2023-03-03T08:43:34.521122+0000","last_became_active":"2023-03-03T08:43:29.159943+0000","last_became_peered":"2023-03-03T08:43:29.159943+0000","last_unstale":"2023-03-03T08:43:34.521122+0000","last_undegraded":"2023-03-03T08:43:34.521122+0000","last_fullsized":"2023-03-03T08:43:34.521122+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:48:58.997356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.538374+0000","last_change":"2023-03-03T08:43:23.015460+0000","last_active":"2023-03-03T08:43:34.538374+0000","last_peered":"2023-03-03T08:43:34.538374+0000","last_clean":"2023-03-03T08:43:34.538374+0000","last_became_active":"2023-03-03T08:43:23.014485+0000","last_became_peered":"2023-03-03T08:43:23.014485+0000","last_unstale":"2023-03-03T08:43:34.538374+0000","last_undegraded":"2023-03-03T08:43:34.538374+0000","last_fullsized":"2023-03-03T08:43:34.538374+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_clean_scrub_stamp":"2023-03-03T08:43:21.542883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:07.022017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114090+0000","last_change":"2023-03-03T08:43:34.110484+0000","last_active":"2023-03-03T08:43:36.114090+0000","last_peered":"2023-03-03T08:43:36.114090+0000","last_clean":"2023-03-03T08:43:36.114090+0000","last_became_active":"2023-03-03T08:43:34.110220+0000","last_became_peered":"2023-03-03T08:43:34.110220+0000","last_unstale":"2023-03-03T08:43:36.114090+0000","last_undegraded":"2023-03-03T08:43:36.114090+0000","last_fullsized":"2023-03-03T08:43:36.114090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:04:59.232537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530634+0000","last_change":"2023-03-03T08:43:34.550263+0000","last_active":"2023-03-03T08:43:35.530634+0000","last_peered":"2023-03-03T08:43:35.530634+0000","last_clean":"2023-03-03T08:43:35.530634+0000","last_became_active":"2023-03-03T08:43:34.549560+0000","last_became_peered":"2023-03-03T08:43:34.549560+0000","last_unstale":"2023-03-03T08:43:35.530634+0000","last_undegraded":"2023-03-03T08:43:35.530634+0000","last_fullsized":"2023-03-03T08:43:35.530634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:55.874223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.522038+0000","last_change":"2023-03-03T08:43:29.075836+0000","last_active":"2023-03-03T08:43:34.522038+0000","last_peered":"2023-03-03T08:43:34.522038+0000","last_clean":"2023-03-03T08:43:34.522038+0000","last_became_active":"2023-03-03T08:43:29.075564+0000","last_became_peered":"2023-03-03T08:43:29.075564+0000","last_unstale":"2023-03-03T08:43:34.522038+0000","last_undegraded":"2023-03-03T08:43:34.522038+0000","last_fullsized":"2023-03-03T08:43:34.522038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:39.576714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130591+0000","last_change":"2023-03-03T08:43:34.104492+0000","last_active":"2023-03-03T08:43:36.130591+0000","last_peered":"2023-03-03T08:43:36.130591+0000","last_clean":"2023-03-03T08:43:36.130591+0000","last_became_active":"2023-03-03T08:43:34.104151+0000","last_became_peered":"2023-03-03T08:43:34.104151+0000","last_unstale":"2023-03-03T08:43:36.130591+0000","last_undegraded":"2023-03-03T08:43:36.130591+0000","last_fullsized":"2023-03-03T08:43:36.130591+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:57.214738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.550389+0000","last_change":"2023-03-03T08:43:34.544053+0000","last_active":"2023-03-03T08:43:34.550389+0000","last_peered":"2023-03-03T08:43:34.550389+0000","last_clean":"2023-03-03T08:43:34.550389+0000","last_became_active":"2023-03-03T08:43:34.543523+0000","last_became_peered":"2023-03-03T08:43:34.543523+0000","last_unstale":"2023-03-03T08:43:34.550389+0000","last_undegraded":"2023-03-03T08:43:34.550389+0000","last_fullsized":"2023-03-03T08:43:34.550389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:34:03.662535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.523389+0000","last_change":"2023-03-03T08:43:29.072304+0000","last_active":"2023-03-03T08:43:34.523389+0000","last_peered":"2023-03-03T08:43:34.523389+0000","last_clean":"2023-03-03T08:43:34.523389+0000","last_became_active":"2023-03-03T08:43:29.071962+0000","last_became_peered":"2023-03-03T08:43:29.071962+0000","last_unstale":"2023-03-03T08:43:34.523389+0000","last_undegraded":"2023-03-03T08:43:34.523389+0000","last_fullsized":"2023-03-03T08:43:34.523389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:40:12.229226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131749+0000","last_change":"2023-03-03T08:43:34.107783+0000","last_active":"2023-03-03T08:43:36.131749+0000","last_peered":"2023-03-03T08:43:36.131749+0000","last_clean":"2023-03-03T08:43:36.131749+0000","last_became_active":"2023-03-03T08:43:34.107503+0000","last_became_peered":"2023-03-03T08:43:34.107503+0000","last_unstale":"2023-03-03T08:43:36.131749+0000","last_undegraded":"2023-03-03T08:43:36.131749+0000","last_fullsized":"2023-03-03T08:43:36.131749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:35:42.800814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131418+0000","last_change":"2023-03-03T08:43:34.535435+0000","last_active":"2023-03-03T08:43:36.131418+0000","last_peered":"2023-03-03T08:43:36.131418+0000","last_clean":"2023-03-03T08:43:36.131418+0000","last_became_active":"2023-03-03T08:43:34.535123+0000","last_became_peered":"2023-03-03T08:43:34.535123+0000","last_unstale":"2023-03-03T08:43:36.131418+0000","last_undegraded":"2023-03-03T08:43:36.131418+0000","last_fullsized":"2023-03-03T08:43:36.131418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:52:10.622422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114468+0000","last_change":"2023-03-03T08:43:34.110105+0000","last_active":"2023-03-03T08:43:36.114468+0000","last_peered":"2023-03-03T08:43:36.114468+0000","last_clean":"2023-03-03T08:43:36.114468+0000","last_became_active":"2023-03-03T08:43:34.109736+0000","last_became_peered":"2023-03-03T08:43:34.109736+0000","last_unstale":"2023-03-03T08:43:36.114468+0000","last_undegraded":"2023-03-03T08:43:36.114468+0000","last_fullsized":"2023-03-03T08:43:36.114468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:30.557847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.536239+0000","last_change":"2023-03-03T08:43:29.354519+0000","last_active":"2023-03-03T08:43:34.536239+0000","last_peered":"2023-03-03T08:43:34.536239+0000","last_clean":"2023-03-03T08:43:34.536239+0000","last_became_active":"2023-03-03T08:43:29.354213+0000","last_became_peered":"2023-03-03T08:43:29.354213+0000","last_unstale":"2023-03-03T08:43:34.536239+0000","last_undegraded":"2023-03-03T08:43:34.536239+0000","last_fullsized":"2023-03-03T08:43:34.536239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:00.406507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114040+0000","last_change":"2023-03-03T08:43:34.539018+0000","last_active":"2023-03-03T08:43:36.114040+0000","last_peered":"2023-03-03T08:43:36.114040+0000","last_clean":"2023-03-03T08:43:36.114040+0000","last_became_active":"2023-03-03T08:43:34.538375+0000","last_became_peered":"2023-03-03T08:43:34.538375+0000","last_unstale":"2023-03-03T08:43:36.114040+0000","last_undegraded":"2023-03-03T08:43:36.114040+0000","last_fullsized":"2023-03-03T08:43:36.114040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:29:02.601410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.521322+0000","last_change":"2023-03-03T08:43:29.354627+0000","last_active":"2023-03-03T08:43:34.521322+0000","last_peered":"2023-03-03T08:43:34.521322+0000","last_clean":"2023-03-03T08:43:34.521322+0000","last_became_active":"2023-03-03T08:43:29.354271+0000","last_became_peered":"2023-03-03T08:43:29.354271+0000","last_unstale":"2023-03-03T08:43:34.521322+0000","last_undegraded":"2023-03-03T08:43:34.521322+0000","last_fullsized":"2023-03-03T08:43:34.521322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:05.946053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114400+0000","last_change":"2023-03-03T08:43:34.109127+0000","last_active":"2023-03-03T08:43:36.114400+0000","last_peered":"2023-03-03T08:43:36.114400+0000","last_clean":"2023-03-03T08:43:36.114400+0000","last_became_active":"2023-03-03T08:43:34.108869+0000","last_became_peered":"2023-03-03T08:43:34.108869+0000","last_unstale":"2023-03-03T08:43:36.114400+0000","last_undegraded":"2023-03-03T08:43:36.114400+0000","last_fullsized":"2023-03-03T08:43:36.114400+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:19:00.275620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529000+0000","last_change":"2023-03-03T08:43:34.537850+0000","last_active":"2023-03-03T08:43:35.529000+0000","last_peered":"2023-03-03T08:43:35.529000+0000","last_clean":"2023-03-03T08:43:35.529000+0000","last_became_active":"2023-03-03T08:43:34.537550+0000","last_became_peered":"2023-03-03T08:43:34.537550+0000","last_unstale":"2023-03-03T08:43:35.529000+0000","last_undegraded":"2023-03-03T08:43:35.529000+0000","last_fullsized":"2023-03-03T08:43:35.529000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:37:36.893757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114319+0000","last_change":"2023-03-03T08:43:34.112667+0000","last_active":"2023-03-03T08:43:36.114319+0000","last_peered":"2023-03-03T08:43:36.114319+0000","last_clean":"2023-03-03T08:43:36.114319+0000","last_became_active":"2023-03-03T08:43:34.112386+0000","last_became_peered":"2023-03-03T08:43:34.112386+0000","last_unstale":"2023-03-03T08:43:36.114319+0000","last_undegraded":"2023-03-03T08:43:36.114319+0000","last_fullsized":"2023-03-03T08:43:36.114319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:40.504392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114284+0000","last_change":"2023-03-03T08:43:34.541172+0000","last_active":"2023-03-03T08:43:36.114284+0000","last_peered":"2023-03-03T08:43:36.114284+0000","last_clean":"2023-03-03T08:43:36.114284+0000","last_became_active":"2023-03-03T08:43:34.540739+0000","last_became_peered":"2023-03-03T08:43:34.540739+0000","last_unstale":"2023-03-03T08:43:36.114284+0000","last_undegraded":"2023-03-03T08:43:36.114284+0000","last_fullsized":"2023-03-03T08:43:36.114284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:40.058265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130597+0000","last_change":"2023-03-03T08:43:34.110187+0000","last_active":"2023-03-03T08:43:36.130597+0000","last_peered":"2023-03-03T08:43:36.130597+0000","last_clean":"2023-03-03T08:43:36.130597+0000","last_became_active":"2023-03-03T08:43:34.109845+0000","last_became_peered":"2023-03-03T08:43:34.109845+0000","last_unstale":"2023-03-03T08:43:36.130597+0000","last_undegraded":"2023-03-03T08:43:36.130597+0000","last_fullsized":"2023-03-03T08:43:36.130597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:33.507500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529558+0000","last_change":"2023-03-03T08:43:34.547322+0000","last_active":"2023-03-03T08:43:35.529558+0000","last_peered":"2023-03-03T08:43:35.529558+0000","last_clean":"2023-03-03T08:43:35.529558+0000","last_became_active":"2023-03-03T08:43:34.546692+0000","last_became_peered":"2023-03-03T08:43:34.546692+0000","last_unstale":"2023-03-03T08:43:35.529558+0000","last_undegraded":"2023-03-03T08:43:35.529558+0000","last_fullsized":"2023-03-03T08:43:35.529558+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:11:26.366920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530453+0000","last_change":"2023-03-03T08:43:34.117851+0000","last_active":"2023-03-03T08:43:35.530453+0000","last_peered":"2023-03-03T08:43:35.530453+0000","last_clean":"2023-03-03T08:43:35.530453+0000","last_became_active":"2023-03-03T08:43:34.117548+0000","last_became_peered":"2023-03-03T08:43:34.117548+0000","last_unstale":"2023-03-03T08:43:35.530453+0000","last_undegraded":"2023-03-03T08:43:35.530453+0000","last_fullsized":"2023-03-03T08:43:35.530453+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:20.039817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529223+0000","last_change":"2023-03-03T08:43:34.535451+0000","last_active":"2023-03-03T08:43:35.529223+0000","last_peered":"2023-03-03T08:43:35.529223+0000","last_clean":"2023-03-03T08:43:35.529223+0000","last_became_active":"2023-03-03T08:43:34.535129+0000","last_became_peered":"2023-03-03T08:43:34.535129+0000","last_unstale":"2023-03-03T08:43:35.529223+0000","last_undegraded":"2023-03-03T08:43:35.529223+0000","last_fullsized":"2023-03-03T08:43:35.529223+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:30.645128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.552703+0000","last_change":"2023-03-03T08:43:34.109782+0000","last_active":"2023-03-03T08:43:34.552703+0000","last_peered":"2023-03-03T08:43:34.552703+0000","last_clean":"2023-03-03T08:43:34.552703+0000","last_became_active":"2023-03-03T08:43:34.109513+0000","last_became_peered":"2023-03-03T08:43:34.109513+0000","last_unstale":"2023-03-03T08:43:34.552703+0000","last_undegraded":"2023-03-03T08:43:34.552703+0000","last_fullsized":"2023-03-03T08:43:34.552703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:16.185161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130648+0000","last_change":"2023-03-03T08:43:34.532482+0000","last_active":"2023-03-03T08:43:36.130648+0000","last_peered":"2023-03-03T08:43:36.130648+0000","last_clean":"2023-03-03T08:43:36.130648+0000","last_became_active":"2023-03-03T08:43:34.531751+0000","last_became_peered":"2023-03-03T08:43:34.531751+0000","last_unstale":"2023-03-03T08:43:36.130648+0000","last_undegraded":"2023-03-03T08:43:36.130648+0000","last_fullsized":"2023-03-03T08:43:36.130648+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:47:45.214945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.553022+0000","last_change":"2023-03-03T08:43:34.108426+0000","last_active":"2023-03-03T08:43:34.553022+0000","last_peered":"2023-03-03T08:43:34.553022+0000","last_clean":"2023-03-03T08:43:34.553022+0000","last_became_active":"2023-03-03T08:43:34.108009+0000","last_became_peered":"2023-03-03T08:43:34.108009+0000","last_unstale":"2023-03-03T08:43:34.553022+0000","last_undegraded":"2023-03-03T08:43:34.553022+0000","last_fullsized":"2023-03-03T08:43:34.553022+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:53.632744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.528838+0000","last_change":"2023-03-03T08:43:34.540812+0000","last_active":"2023-03-03T08:43:35.528838+0000","last_peered":"2023-03-03T08:43:35.528838+0000","last_clean":"2023-03-03T08:43:35.528838+0000","last_became_active":"2023-03-03T08:43:34.540549+0000","last_became_peered":"2023-03-03T08:43:34.540549+0000","last_unstale":"2023-03-03T08:43:35.528838+0000","last_undegraded":"2023-03-03T08:43:35.528838+0000","last_fullsized":"2023-03-03T08:43:35.528838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130471+0000","last_change":"2023-03-03T08:43:34.547197+0000","last_active":"2023-03-03T08:43:36.130471+0000","last_peered":"2023-03-03T08:43:36.130471+0000","last_clean":"2023-03-03T08:43:36.130471+0000","last_became_active":"2023-03-03T08:43:34.546927+0000","last_became_peered":"2023-03-03T08:43:34.546927+0000","last_unstale":"2023-03-03T08:43:36.130471+0000","last_undegraded":"2023-03-03T08:43:36.130471+0000","last_fullsized":"2023-03-03T08:43:36.130471+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:21:34.845968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130700+0000","last_change":"2023-03-03T08:43:34.538786+0000","last_active":"2023-03-03T08:43:36.130700+0000","last_peered":"2023-03-03T08:43:36.130700+0000","last_clean":"2023-03-03T08:43:36.130700+0000","last_became_active":"2023-03-03T08:43:34.538494+0000","last_became_peered":"2023-03-03T08:43:34.538494+0000","last_unstale":"2023-03-03T08:43:36.130700+0000","last_undegraded":"2023-03-03T08:43:36.130700+0000","last_fullsized":"2023-03-03T08:43:36.130700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:53.838286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.551228+0000","last_change":"2023-03-03T08:43:34.544408+0000","last_active":"2023-03-03T08:43:34.551228+0000","last_peered":"2023-03-03T08:43:34.551228+0000","last_clean":"2023-03-03T08:43:34.551228+0000","last_became_active":"2023-03-03T08:43:34.543977+0000","last_became_peered":"2023-03-03T08:43:34.543977+0000","last_unstale":"2023-03-03T08:43:34.551228+0000","last_undegraded":"2023-03-03T08:43:34.551228+0000","last_fullsized":"2023-03-03T08:43:34.551228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.074082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.555804+0000","last_change":"2023-03-03T08:43:34.552931+0000","last_active":"2023-03-03T08:43:34.555804+0000","last_peered":"2023-03-03T08:43:34.555804+0000","last_clean":"2023-03-03T08:43:34.555804+0000","last_became_active":"2023-03-03T08:43:34.552623+0000","last_became_peered":"2023-03-03T08:43:34.552623+0000","last_unstale":"2023-03-03T08:43:34.555804+0000","last_undegraded":"2023-03-03T08:43:34.555804+0000","last_fullsized":"2023-03-03T08:43:34.555804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:07:43.954897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.550464+0000","last_change":"2023-03-03T08:43:34.546828+0000","last_active":"2023-03-03T08:43:34.550464+0000","last_peered":"2023-03-03T08:43:34.550464+0000","last_clean":"2023-03-03T08:43:34.550464+0000","last_became_active":"2023-03-03T08:43:34.546559+0000","last_became_peered":"2023-03-03T08:43:34.546559+0000","last_unstale":"2023-03-03T08:43:34.550464+0000","last_undegraded":"2023-03-03T08:43:34.550464+0000","last_fullsized":"2023-03-03T08:43:34.550464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:21:14.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7972,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363868,"statfs":{"total":96636764160,"available":96628600832,"internally_reserved":0,"allocated":225280,"data_stored":81872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7980,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363860,"statfs":{"total":96636764160,"available":96628592640,"internally_reserved":0,"allocated":233472,"data_stored":82981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9092,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362748,"statfs":{"total":96636764160,"available":96627453952,"internally_reserved":0,"allocated":847872,"data_stored":680545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":82018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":83420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8056,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363784,"statfs":{"total":96636764160,"available":96628514816,"internally_reserved":0,"allocated":245760,"data_stored":83524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7976,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363864,"statfs":{"total":96636764160,"available":96628596736,"internally_reserved":0,"allocated":229376,"data_stored":82891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9068,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362772,"statfs":{"total":96636764160,"available":96627478528,"internally_reserved":0,"allocated":823296,"data_stored":672806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:43:39.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:43:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.801+0000 7f1b96d2a700 1 Processor -- start 2023-03-03T08:43:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.801+0000 7f1b96d2a700 1 -- start start 2023-03-03T08:43:39.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.802+0000 7f1b96d2a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1b90071990 0x7f1b90071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.802+0000 7f1b96d2a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b9014b460 0x7f1b900722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.802+0000 7f1b96d2a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b90070360 con 0x7f1b90151c00 2023-03-03T08:43:39.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.802+0000 7f1b96d2a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b901521a0 con 0x7f1b90071990 2023-03-03T08:43:39.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.802+0000 7f1b96d2a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9006a5e0 con 0x7f1b9014b460 2023-03-03T08:43:39.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.802+0000 7f1b8ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b9014b460 0x7f1b900722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.802+0000 7f1b94ac6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1b90071990 0x7f1b90071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.802+0000 7f1b952c7700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1b90151c00 0x7f1b9006aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:48732/0 (socket says 172.21.15.7:48732) 2023-03-03T08:43:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.802+0000 7f1b952c7700 1 -- 172.21.15.7:0/1718281590 learned_addr learned my addr 172.21.15.7:0/1718281590 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b94ac6700 1 -- 172.21.15.7:0/1718281590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b9014b460 msgr2=0x7f1b900722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b94ac6700 1 --2- 172.21.15.7:0/1718281590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b9014b460 0x7f1b900722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b94ac6700 1 -- 172.21.15.7:0/1718281590 >> v1:172.21.15.143:6789/0 conn(0x7f1b90151c00 legacy=0x7f1b9006aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b8f7fe700 1 -- 172.21.15.7:0/1718281590 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281364801 0 0) 0x7f1b90070360 con 0x7f1b90151c00 2023-03-03T08:43:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b94ac6700 1 -- 172.21.15.7:0/1718281590 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b900c38b0 con 0x7f1b90071990 2023-03-03T08:43:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b8ffff700 1 --2- 172.21.15.7:0/1718281590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b9014b460 0x7f1b900722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b94ac6700 1 --2- 172.21.15.7:0/1718281590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b90071990 0x7f1b90071d60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009840 tx=0x7f1b84009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3d4f90c17047ad5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b8f7fe700 1 -- 172.21.15.7:0/1718281590 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84011830 con 0x7f1b90071990 2023-03-03T08:43:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b8f7fe700 1 -- 172.21.15.7:0/1718281590 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b84011990 con 0x7f1b90071990 2023-03-03T08:43:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b8f7fe700 1 -- 172.21.15.7:0/1718281590 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b84011d20 con 0x7f1b90071990 2023-03-03T08:43:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b96d2a700 1 -- 172.21.15.7:0/1718281590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b90071990 msgr2=0x7f1b90071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b96d2a700 1 --2- 172.21.15.7:0/1718281590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b90071990 0x7f1b90071d60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009840 tx=0x7f1b84009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b96d2a700 1 -- 172.21.15.7:0/1718281590 shutdown_connections 2023-03-03T08:43:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.803+0000 7f1b96d2a700 1 --2- 172.21.15.7:0/1718281590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b90071990 0x7f1b90071d60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.804+0000 7f1b96d2a700 1 --2- 172.21.15.7:0/1718281590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b9014b460 0x7f1b900722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.804+0000 7f1b96d2a700 1 -- 172.21.15.7:0/1718281590 >> 172.21.15.7:0/1718281590 conn(0x7f1b900a94d0 msgr2=0x7f1b900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.804+0000 7f1b96d2a700 1 -- 172.21.15.7:0/1718281590 shutdown_connections 2023-03-03T08:43:39.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.804+0000 7f1b96d2a700 1 -- 172.21.15.7:0/1718281590 wait complete. 2023-03-03T08:43:39.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.805+0000 7f1b96d2a700 1 Processor -- start 2023-03-03T08:43:39.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.805+0000 7f1b96d2a700 1 -- start start 2023-03-03T08:43:39.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.805+0000 7f1b96d2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b90071990 0x7f1b90143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.805+0000 7f1b96d2a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b90151c00 0x7f1b90150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.805+0000 7f1b96d2a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b90150420 0x7f1b90150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b96d2a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b901e8f10 con 0x7f1b90150420 2023-03-03T08:43:39.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b96d2a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b901e82c0 con 0x7f1b90071990 2023-03-03T08:43:39.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b96d2a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9014f090 con 0x7f1b90151c00 2023-03-03T08:43:39.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b952c7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b90150420 0x7f1b90150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b952c7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b90150420 0x7f1b90150d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51238/0 (socket says 172.21.15.7:51238) 2023-03-03T08:43:39.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b952c7700 1 -- 172.21.15.7:0/262176486 learned_addr learned my addr 172.21.15.7:0/262176486 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:39.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b94ac6700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b90071990 0x7f1b90143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b8ffff700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b90151c00 0x7f1b90150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b952c7700 1 -- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b90151c00 msgr2=0x7f1b90150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b952c7700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b90151c00 0x7f1b90150810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b952c7700 1 -- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b90071990 msgr2=0x7f1b90143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b952c7700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b90071990 0x7f1b90143fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b952c7700 1 -- 172.21.15.7:0/262176486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c0094a0 con 0x7f1b90150420 2023-03-03T08:43:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b952c7700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b90150420 0x7f1b90150d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800cb30 tx=0x7f1b8800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b8d7fa700 1 -- 172.21.15.7:0/262176486 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b8800ab50 con 0x7f1b90150420 2023-03-03T08:43:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b8d7fa700 1 -- 172.21.15.7:0/262176486 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8800acb0 con 0x7f1b90150420 2023-03-03T08:43:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b8d7fa700 1 -- 172.21.15.7:0/262176486 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88007310 con 0x7f1b90150420 2023-03-03T08:43:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b96d2a700 1 -- 172.21.15.7:0/262176486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b840094a0 con 0x7f1b90150420 2023-03-03T08:43:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.806+0000 7f1b96d2a700 1 -- 172.21.15.7:0/262176486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b900c3b50 con 0x7f1b90150420 2023-03-03T08:43:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.807+0000 7f1b96d2a700 1 -- 172.21.15.7:0/262176486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9006a2b0 con 0x7f1b90150420 2023-03-03T08:43:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.808+0000 7f1b8d7fa700 1 -- 172.21.15.7:0/262176486 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b88010070 con 0x7f1b90150420 2023-03-03T08:43:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.808+0000 7f1b8d7fa700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b740682f0 0x7f1b7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.809+0000 7f1b8d7fa700 1 -- 172.21.15.7:0/262176486 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f1b88085580 con 0x7f1b90150420 2023-03-03T08:43:39.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.809+0000 7f1b94ac6700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b740682f0 0x7f1b7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:39.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.809+0000 7f1b94ac6700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b740682f0 0x7f1b7406a7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1b84000c00 tx=0x7f1b84008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:39.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:39.813+0000 7f1b8d7fa700 1 -- 172.21.15.7:0/262176486 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b88051d90 con 0x7f1b90150420 2023-03-03T08:43:40.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.002+0000 7f1b96d2a700 1 -- 172.21.15.7:0/262176486 --> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b9006b180 con 0x7f1b740682f0 2023-03-03T08:43:40.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.010+0000 7f1b8d7fa700 1 -- 172.21.15.7:0/262176486 <== mgr.4102 v2:172.21.15.143:6832/96367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286764 (secure 0 0 0) 0x7f1b9006b180 con 0x7f1b740682f0 2023-03-03T08:43:40.015 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:40.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 -- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b740682f0 msgr2=0x7f1b7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b740682f0 0x7f1b7406a7a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1b84000c00 tx=0x7f1b84008030 comp rx=0 tx=0).stop 2023-03-03T08:43:40.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 -- 172.21.15.7:0/262176486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b90150420 msgr2=0x7f1b90150d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b90150420 0x7f1b90150d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800cb30 tx=0x7f1b8800cef0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 -- 172.21.15.7:0/262176486 shutdown_connections 2023-03-03T08:43:40.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b740682f0 0x7f1b7406a7a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b90150420 0x7f1b90150d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b90071990 0x7f1b90143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 --2- 172.21.15.7:0/262176486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b90151c00 0x7f1b90150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 -- 172.21.15.7:0/262176486 >> 172.21.15.7:0/262176486 conn(0x7f1b900a94d0 msgr2=0x7f1b900c68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:40.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 -- 172.21.15.7:0/262176486 shutdown_connections 2023-03-03T08:43:40.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.013+0000 7f1b96d2a700 1 -- 172.21.15.7:0/262176486 wait complete. 2023-03-03T08:43:40.020 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-03-03T08:43:40.027 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-03-03T08:43:39.106315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":66112,"kb_used_data":3008,"kb_used_omap":0,"kb_used_meta":63040,"kb_avail":754908608,"statfs":{"total":773094113280,"available":773026414592,"internally_reserved":0,"allocated":3080192,"data_stored":1850057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64552960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001620"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.550421+0000","last_change":"2023-03-03T08:43:34.546272+0000","last_active":"2023-03-03T08:43:34.550421+0000","last_peered":"2023-03-03T08:43:34.550421+0000","last_clean":"2023-03-03T08:43:34.550421+0000","last_became_active":"2023-03-03T08:43:34.546006+0000","last_became_peered":"2023-03-03T08:43:34.546006+0000","last_unstale":"2023-03-03T08:43:34.550421+0000","last_undegraded":"2023-03-03T08:43:34.550421+0000","last_fullsized":"2023-03-03T08:43:34.550421+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:15.121761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530001+0000","last_change":"2023-03-03T08:43:34.541669+0000","last_active":"2023-03-03T08:43:35.530001+0000","last_peered":"2023-03-03T08:43:35.530001+0000","last_clean":"2023-03-03T08:43:35.530001+0000","last_became_active":"2023-03-03T08:43:34.541104+0000","last_became_peered":"2023-03-03T08:43:34.541104+0000","last_unstale":"2023-03-03T08:43:35.530001+0000","last_undegraded":"2023-03-03T08:43:35.530001+0000","last_fullsized":"2023-03-03T08:43:35.530001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.551432+0000","last_change":"2023-03-03T08:43:34.544721+0000","last_active":"2023-03-03T08:43:34.551432+0000","last_peered":"2023-03-03T08:43:34.551432+0000","last_clean":"2023-03-03T08:43:34.551432+0000","last_became_active":"2023-03-03T08:43:34.544461+0000","last_became_peered":"2023-03-03T08:43:34.544461+0000","last_unstale":"2023-03-03T08:43:34.551432+0000","last_undegraded":"2023-03-03T08:43:34.551432+0000","last_fullsized":"2023-03-03T08:43:34.551432+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:52:11.890779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.549458+0000","last_change":"2023-03-03T08:43:34.538467+0000","last_active":"2023-03-03T08:43:34.549458+0000","last_peered":"2023-03-03T08:43:34.549458+0000","last_clean":"2023-03-03T08:43:34.549458+0000","last_became_active":"2023-03-03T08:43:34.538207+0000","last_became_peered":"2023-03-03T08:43:34.538207+0000","last_unstale":"2023-03-03T08:43:34.549458+0000","last_undegraded":"2023-03-03T08:43:34.549458+0000","last_fullsized":"2023-03-03T08:43:34.549458+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:14.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114429+0000","last_change":"2023-03-03T08:43:34.543620+0000","last_active":"2023-03-03T08:43:36.114429+0000","last_peered":"2023-03-03T08:43:36.114429+0000","last_clean":"2023-03-03T08:43:36.114429+0000","last_became_active":"2023-03-03T08:43:34.543370+0000","last_became_peered":"2023-03-03T08:43:34.543370+0000","last_unstale":"2023-03-03T08:43:36.114429+0000","last_undegraded":"2023-03-03T08:43:36.114429+0000","last_fullsized":"2023-03-03T08:43:36.114429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:40:32.808203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529270+0000","last_change":"2023-03-03T08:43:34.537664+0000","last_active":"2023-03-03T08:43:35.529270+0000","last_peered":"2023-03-03T08:43:35.529270+0000","last_clean":"2023-03-03T08:43:35.529270+0000","last_became_active":"2023-03-03T08:43:34.537214+0000","last_became_peered":"2023-03-03T08:43:34.537214+0000","last_unstale":"2023-03-03T08:43:35.529270+0000","last_undegraded":"2023-03-03T08:43:35.529270+0000","last_fullsized":"2023-03-03T08:43:35.529270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:54:09.222135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529177+0000","last_change":"2023-03-03T08:43:34.533673+0000","last_active":"2023-03-03T08:43:35.529177+0000","last_peered":"2023-03-03T08:43:35.529177+0000","last_clean":"2023-03-03T08:43:35.529177+0000","last_became_active":"2023-03-03T08:43:34.532283+0000","last_became_peered":"2023-03-03T08:43:34.532283+0000","last_unstale":"2023-03-03T08:43:35.529177+0000","last_undegraded":"2023-03-03T08:43:35.529177+0000","last_fullsized":"2023-03-03T08:43:35.529177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:25.591188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130569+0000","last_change":"2023-03-03T08:43:34.533496+0000","last_active":"2023-03-03T08:43:36.130569+0000","last_peered":"2023-03-03T08:43:36.130569+0000","last_clean":"2023-03-03T08:43:36.130569+0000","last_became_active":"2023-03-03T08:43:34.532714+0000","last_became_peered":"2023-03-03T08:43:34.532714+0000","last_unstale":"2023-03-03T08:43:36.130569+0000","last_undegraded":"2023-03-03T08:43:36.130569+0000","last_fullsized":"2023-03-03T08:43:36.130569+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:47.800807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114381+0000","last_change":"2023-03-03T08:43:34.549838+0000","last_active":"2023-03-03T08:43:36.114381+0000","last_peered":"2023-03-03T08:43:36.114381+0000","last_clean":"2023-03-03T08:43:36.114381+0000","last_became_active":"2023-03-03T08:43:34.549537+0000","last_became_peered":"2023-03-03T08:43:34.549537+0000","last_unstale":"2023-03-03T08:43:36.114381+0000","last_undegraded":"2023-03-03T08:43:36.114381+0000","last_fullsized":"2023-03-03T08:43:36.114381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:36.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131020+0000","last_change":"2023-03-03T08:43:34.548229+0000","last_active":"2023-03-03T08:43:36.131020+0000","last_peered":"2023-03-03T08:43:36.131020+0000","last_clean":"2023-03-03T08:43:36.131020+0000","last_became_active":"2023-03-03T08:43:34.548013+0000","last_became_peered":"2023-03-03T08:43:34.548013+0000","last_unstale":"2023-03-03T08:43:36.131020+0000","last_undegraded":"2023-03-03T08:43:36.131020+0000","last_fullsized":"2023-03-03T08:43:36.131020+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:27.793716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131431+0000","last_change":"2023-03-03T08:43:34.550889+0000","last_active":"2023-03-03T08:43:36.131431+0000","last_peered":"2023-03-03T08:43:36.131431+0000","last_clean":"2023-03-03T08:43:36.131431+0000","last_became_active":"2023-03-03T08:43:34.550605+0000","last_became_peered":"2023-03-03T08:43:34.550605+0000","last_unstale":"2023-03-03T08:43:36.131431+0000","last_undegraded":"2023-03-03T08:43:36.131431+0000","last_fullsized":"2023-03-03T08:43:36.131431+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:01.118555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.555833+0000","last_change":"2023-03-03T08:43:34.553434+0000","last_active":"2023-03-03T08:43:34.555833+0000","last_peered":"2023-03-03T08:43:34.555833+0000","last_clean":"2023-03-03T08:43:34.555833+0000","last_became_active":"2023-03-03T08:43:34.553125+0000","last_became_peered":"2023-03-03T08:43:34.553125+0000","last_unstale":"2023-03-03T08:43:34.555833+0000","last_undegraded":"2023-03-03T08:43:34.555833+0000","last_fullsized":"2023-03-03T08:43:34.555833+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:41:17.553506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131312+0000","last_change":"2023-03-03T08:43:34.544013+0000","last_active":"2023-03-03T08:43:36.131312+0000","last_peered":"2023-03-03T08:43:36.131312+0000","last_clean":"2023-03-03T08:43:36.131312+0000","last_became_active":"2023-03-03T08:43:34.543745+0000","last_became_peered":"2023-03-03T08:43:34.543745+0000","last_unstale":"2023-03-03T08:43:36.131312+0000","last_undegraded":"2023-03-03T08:43:36.131312+0000","last_fullsized":"2023-03-03T08:43:36.131312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:05.647977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131242+0000","last_change":"2023-03-03T08:43:34.549026+0000","last_active":"2023-03-03T08:43:36.131242+0000","last_peered":"2023-03-03T08:43:36.131242+0000","last_clean":"2023-03-03T08:43:36.131242+0000","last_became_active":"2023-03-03T08:43:34.548778+0000","last_became_peered":"2023-03-03T08:43:34.548778+0000","last_unstale":"2023-03-03T08:43:36.131242+0000","last_undegraded":"2023-03-03T08:43:36.131242+0000","last_fullsized":"2023-03-03T08:43:36.131242+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:41.835304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.551257+0000","last_change":"2023-03-03T08:43:34.542729+0000","last_active":"2023-03-03T08:43:34.551257+0000","last_peered":"2023-03-03T08:43:34.551257+0000","last_clean":"2023-03-03T08:43:34.551257+0000","last_became_active":"2023-03-03T08:43:34.542246+0000","last_became_peered":"2023-03-03T08:43:34.542246+0000","last_unstale":"2023-03-03T08:43:34.551257+0000","last_undegraded":"2023-03-03T08:43:34.551257+0000","last_fullsized":"2023-03-03T08:43:34.551257+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:14:59.995526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.528932+0000","last_change":"2023-03-03T08:43:34.527373+0000","last_active":"2023-03-03T08:43:35.528932+0000","last_peered":"2023-03-03T08:43:35.528932+0000","last_clean":"2023-03-03T08:43:35.528932+0000","last_became_active":"2023-03-03T08:43:34.527171+0000","last_became_peered":"2023-03-03T08:43:34.527171+0000","last_unstale":"2023-03-03T08:43:35.528932+0000","last_undegraded":"2023-03-03T08:43:35.528932+0000","last_fullsized":"2023-03-03T08:43:35.528932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:20:51.138587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130928+0000","last_change":"2023-03-03T08:43:34.532154+0000","last_active":"2023-03-03T08:43:36.130928+0000","last_peered":"2023-03-03T08:43:36.130928+0000","last_clean":"2023-03-03T08:43:36.130928+0000","last_became_active":"2023-03-03T08:43:34.531844+0000","last_became_peered":"2023-03-03T08:43:34.531844+0000","last_unstale":"2023-03-03T08:43:36.130928+0000","last_undegraded":"2023-03-03T08:43:36.130928+0000","last_fullsized":"2023-03-03T08:43:36.130928+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:00.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.549428+0000","last_change":"2023-03-03T08:43:34.537110+0000","last_active":"2023-03-03T08:43:34.549428+0000","last_peered":"2023-03-03T08:43:34.549428+0000","last_clean":"2023-03-03T08:43:34.549428+0000","last_became_active":"2023-03-03T08:43:34.536848+0000","last_became_peered":"2023-03-03T08:43:34.536848+0000","last_unstale":"2023-03-03T08:43:34.549428+0000","last_undegraded":"2023-03-03T08:43:34.549428+0000","last_fullsized":"2023-03-03T08:43:34.549428+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.555844+0000","last_change":"2023-03-03T08:43:34.553589+0000","last_active":"2023-03-03T08:43:34.555844+0000","last_peered":"2023-03-03T08:43:34.555844+0000","last_clean":"2023-03-03T08:43:34.555844+0000","last_became_active":"2023-03-03T08:43:34.553242+0000","last_became_peered":"2023-03-03T08:43:34.553242+0000","last_unstale":"2023-03-03T08:43:34.555844+0000","last_undegraded":"2023-03-03T08:43:34.555844+0000","last_fullsized":"2023-03-03T08:43:34.555844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:09.000051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130506+0000","last_change":"2023-03-03T08:43:34.547003+0000","last_active":"2023-03-03T08:43:36.130506+0000","last_peered":"2023-03-03T08:43:36.130506+0000","last_clean":"2023-03-03T08:43:36.130506+0000","last_became_active":"2023-03-03T08:43:34.546744+0000","last_became_peered":"2023-03-03T08:43:34.546744+0000","last_unstale":"2023-03-03T08:43:36.130506+0000","last_undegraded":"2023-03-03T08:43:36.130506+0000","last_fullsized":"2023-03-03T08:43:36.130506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:53:38.996605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131059+0000","last_change":"2023-03-03T08:43:34.549837+0000","last_active":"2023-03-03T08:43:36.131059+0000","last_peered":"2023-03-03T08:43:36.131059+0000","last_clean":"2023-03-03T08:43:36.131059+0000","last_became_active":"2023-03-03T08:43:34.549542+0000","last_became_peered":"2023-03-03T08:43:34.549542+0000","last_unstale":"2023-03-03T08:43:36.131059+0000","last_undegraded":"2023-03-03T08:43:36.131059+0000","last_fullsized":"2023-03-03T08:43:36.131059+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:31.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130796+0000","last_change":"2023-03-03T08:43:34.542661+0000","last_active":"2023-03-03T08:43:36.130796+0000","last_peered":"2023-03-03T08:43:36.130796+0000","last_clean":"2023-03-03T08:43:36.130796+0000","last_became_active":"2023-03-03T08:43:34.542353+0000","last_became_peered":"2023-03-03T08:43:34.542353+0000","last_unstale":"2023-03-03T08:43:36.130796+0000","last_undegraded":"2023-03-03T08:43:36.130796+0000","last_fullsized":"2023-03-03T08:43:36.130796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:38.730662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114146+0000","last_change":"2023-03-03T08:43:34.549771+0000","last_active":"2023-03-03T08:43:36.114146+0000","last_peered":"2023-03-03T08:43:36.114146+0000","last_clean":"2023-03-03T08:43:36.114146+0000","last_became_active":"2023-03-03T08:43:34.549479+0000","last_became_peered":"2023-03-03T08:43:34.549479+0000","last_unstale":"2023-03-03T08:43:36.114146+0000","last_undegraded":"2023-03-03T08:43:36.114146+0000","last_fullsized":"2023-03-03T08:43:36.114146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:41:03.902019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130554+0000","last_change":"2023-03-03T08:43:34.548469+0000","last_active":"2023-03-03T08:43:36.130554+0000","last_peered":"2023-03-03T08:43:36.130554+0000","last_clean":"2023-03-03T08:43:36.130554+0000","last_became_active":"2023-03-03T08:43:34.547647+0000","last_became_peered":"2023-03-03T08:43:34.547647+0000","last_unstale":"2023-03-03T08:43:36.130554+0000","last_undegraded":"2023-03-03T08:43:36.130554+0000","last_fullsized":"2023-03-03T08:43:36.130554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:40.054781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114484+0000","last_change":"2023-03-03T08:43:34.542858+0000","last_active":"2023-03-03T08:43:36.114484+0000","last_peered":"2023-03-03T08:43:36.114484+0000","last_clean":"2023-03-03T08:43:36.114484+0000","last_became_active":"2023-03-03T08:43:34.542510+0000","last_became_peered":"2023-03-03T08:43:34.542510+0000","last_unstale":"2023-03-03T08:43:36.114484+0000","last_undegraded":"2023-03-03T08:43:36.114484+0000","last_fullsized":"2023-03-03T08:43:36.114484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:21.471234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131271+0000","last_change":"2023-03-03T08:43:34.540762+0000","last_active":"2023-03-03T08:43:36.131271+0000","last_peered":"2023-03-03T08:43:36.131271+0000","last_clean":"2023-03-03T08:43:36.131271+0000","last_became_active":"2023-03-03T08:43:34.540467+0000","last_became_peered":"2023-03-03T08:43:34.540467+0000","last_unstale":"2023-03-03T08:43:36.131271+0000","last_undegraded":"2023-03-03T08:43:36.131271+0000","last_fullsized":"2023-03-03T08:43:36.131271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530126+0000","last_change":"2023-03-03T08:43:34.533987+0000","last_active":"2023-03-03T08:43:35.530126+0000","last_peered":"2023-03-03T08:43:35.530126+0000","last_clean":"2023-03-03T08:43:35.530126+0000","last_became_active":"2023-03-03T08:43:34.533729+0000","last_became_peered":"2023-03-03T08:43:34.533729+0000","last_unstale":"2023-03-03T08:43:35.530126+0000","last_undegraded":"2023-03-03T08:43:35.530126+0000","last_fullsized":"2023-03-03T08:43:35.530126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:18:12.407151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131293+0000","last_change":"2023-03-03T08:43:34.108056+0000","last_active":"2023-03-03T08:43:36.131293+0000","last_peered":"2023-03-03T08:43:36.131293+0000","last_clean":"2023-03-03T08:43:36.131293+0000","last_became_active":"2023-03-03T08:43:34.107708+0000","last_became_peered":"2023-03-03T08:43:34.107708+0000","last_unstale":"2023-03-03T08:43:36.131293+0000","last_undegraded":"2023-03-03T08:43:36.131293+0000","last_fullsized":"2023-03-03T08:43:36.131293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:10.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130603+0000","last_change":"2023-03-03T08:43:34.537301+0000","last_active":"2023-03-03T08:43:36.130603+0000","last_peered":"2023-03-03T08:43:36.130603+0000","last_clean":"2023-03-03T08:43:36.130603+0000","last_became_active":"2023-03-03T08:43:34.537007+0000","last_became_peered":"2023-03-03T08:43:34.537007+0000","last_unstale":"2023-03-03T08:43:36.130603+0000","last_undegraded":"2023-03-03T08:43:36.130603+0000","last_fullsized":"2023-03-03T08:43:36.130603+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:59:49.124140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114026+0000","last_change":"2023-03-03T08:43:34.108187+0000","last_active":"2023-03-03T08:43:36.114026+0000","last_peered":"2023-03-03T08:43:36.114026+0000","last_clean":"2023-03-03T08:43:36.114026+0000","last_became_active":"2023-03-03T08:43:34.107859+0000","last_became_peered":"2023-03-03T08:43:34.107859+0000","last_unstale":"2023-03-03T08:43:36.114026+0000","last_undegraded":"2023-03-03T08:43:36.114026+0000","last_fullsized":"2023-03-03T08:43:36.114026+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:46:40.595756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114408+0000","last_change":"2023-03-03T08:43:34.549123+0000","last_active":"2023-03-03T08:43:36.114408+0000","last_peered":"2023-03-03T08:43:36.114408+0000","last_clean":"2023-03-03T08:43:36.114408+0000","last_became_active":"2023-03-03T08:43:34.548839+0000","last_became_peered":"2023-03-03T08:43:34.548839+0000","last_unstale":"2023-03-03T08:43:36.114408+0000","last_undegraded":"2023-03-03T08:43:36.114408+0000","last_fullsized":"2023-03-03T08:43:36.114408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:38.134009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.539564+0000","last_change":"2023-03-03T08:43:34.106505+0000","last_active":"2023-03-03T08:43:34.539564+0000","last_peered":"2023-03-03T08:43:34.539564+0000","last_clean":"2023-03-03T08:43:34.539564+0000","last_became_active":"2023-03-03T08:43:34.106079+0000","last_became_peered":"2023-03-03T08:43:34.106079+0000","last_unstale":"2023-03-03T08:43:34.539564+0000","last_undegraded":"2023-03-03T08:43:34.539564+0000","last_fullsized":"2023-03-03T08:43:34.539564+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:48.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530817+0000","last_change":"2023-03-03T08:43:34.547450+0000","last_active":"2023-03-03T08:43:35.530817+0000","last_peered":"2023-03-03T08:43:35.530817+0000","last_clean":"2023-03-03T08:43:35.530817+0000","last_became_active":"2023-03-03T08:43:34.547168+0000","last_became_peered":"2023-03-03T08:43:34.547168+0000","last_unstale":"2023-03-03T08:43:35.530817+0000","last_undegraded":"2023-03-03T08:43:35.530817+0000","last_fullsized":"2023-03-03T08:43:35.530817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:17.723229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.539993+0000","last_change":"2023-03-03T08:43:34.110802+0000","last_active":"2023-03-03T08:43:34.539993+0000","last_peered":"2023-03-03T08:43:34.539993+0000","last_clean":"2023-03-03T08:43:34.539993+0000","last_became_active":"2023-03-03T08:43:34.110519+0000","last_became_peered":"2023-03-03T08:43:34.110519+0000","last_unstale":"2023-03-03T08:43:34.539993+0000","last_undegraded":"2023-03-03T08:43:34.539993+0000","last_fullsized":"2023-03-03T08:43:34.539993+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:02:18.355043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131083+0000","last_change":"2023-03-03T08:43:34.528573+0000","last_active":"2023-03-03T08:43:36.131083+0000","last_peered":"2023-03-03T08:43:36.131083+0000","last_clean":"2023-03-03T08:43:36.131083+0000","last_became_active":"2023-03-03T08:43:34.528278+0000","last_became_peered":"2023-03-03T08:43:34.528278+0000","last_unstale":"2023-03-03T08:43:36.131083+0000","last_undegraded":"2023-03-03T08:43:36.131083+0000","last_fullsized":"2023-03-03T08:43:36.131083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:51.468323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530140+0000","last_change":"2023-03-03T08:43:34.113168+0000","last_active":"2023-03-03T08:43:35.530140+0000","last_peered":"2023-03-03T08:43:35.530140+0000","last_clean":"2023-03-03T08:43:35.530140+0000","last_became_active":"2023-03-03T08:43:34.112731+0000","last_became_peered":"2023-03-03T08:43:34.112731+0000","last_unstale":"2023-03-03T08:43:35.530140+0000","last_undegraded":"2023-03-03T08:43:35.530140+0000","last_fullsized":"2023-03-03T08:43:35.530140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:57:59.463191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.113969+0000","last_change":"2023-03-03T08:43:34.538703+0000","last_active":"2023-03-03T08:43:36.113969+0000","last_peered":"2023-03-03T08:43:36.113969+0000","last_clean":"2023-03-03T08:43:36.113969+0000","last_became_active":"2023-03-03T08:43:34.538418+0000","last_became_peered":"2023-03-03T08:43:34.538418+0000","last_unstale":"2023-03-03T08:43:36.113969+0000","last_undegraded":"2023-03-03T08:43:36.113969+0000","last_fullsized":"2023-03-03T08:43:36.113969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:02.795203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.551383+0000","last_change":"2023-03-03T08:43:34.110810+0000","last_active":"2023-03-03T08:43:34.551383+0000","last_peered":"2023-03-03T08:43:34.551383+0000","last_clean":"2023-03-03T08:43:34.551383+0000","last_became_active":"2023-03-03T08:43:34.110483+0000","last_became_peered":"2023-03-03T08:43:34.110483+0000","last_unstale":"2023-03-03T08:43:34.551383+0000","last_undegraded":"2023-03-03T08:43:34.551383+0000","last_fullsized":"2023-03-03T08:43:34.551383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:06.970615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114208+0000","last_change":"2023-03-03T08:43:34.544694+0000","last_active":"2023-03-03T08:43:36.114208+0000","last_peered":"2023-03-03T08:43:36.114208+0000","last_clean":"2023-03-03T08:43:36.114208+0000","last_became_active":"2023-03-03T08:43:34.544399+0000","last_became_peered":"2023-03-03T08:43:34.544399+0000","last_unstale":"2023-03-03T08:43:36.114208+0000","last_undegraded":"2023-03-03T08:43:36.114208+0000","last_fullsized":"2023-03-03T08:43:36.114208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530076+0000","last_change":"2023-03-03T08:43:34.117609+0000","last_active":"2023-03-03T08:43:35.530076+0000","last_peered":"2023-03-03T08:43:35.530076+0000","last_clean":"2023-03-03T08:43:35.530076+0000","last_became_active":"2023-03-03T08:43:34.117271+0000","last_became_peered":"2023-03-03T08:43:34.117271+0000","last_unstale":"2023-03-03T08:43:35.530076+0000","last_undegraded":"2023-03-03T08:43:35.530076+0000","last_fullsized":"2023-03-03T08:43:35.530076+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:20.259259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529652+0000","last_change":"2023-03-03T08:43:34.537857+0000","last_active":"2023-03-03T08:43:35.529652+0000","last_peered":"2023-03-03T08:43:35.529652+0000","last_clean":"2023-03-03T08:43:35.529652+0000","last_became_active":"2023-03-03T08:43:34.537590+0000","last_became_peered":"2023-03-03T08:43:34.537590+0000","last_unstale":"2023-03-03T08:43:35.529652+0000","last_undegraded":"2023-03-03T08:43:35.529652+0000","last_fullsized":"2023-03-03T08:43:35.529652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:52.612521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530006+0000","last_change":"2023-03-03T08:43:34.112941+0000","last_active":"2023-03-03T08:43:35.530006+0000","last_peered":"2023-03-03T08:43:35.530006+0000","last_clean":"2023-03-03T08:43:35.530006+0000","last_became_active":"2023-03-03T08:43:34.112646+0000","last_became_peered":"2023-03-03T08:43:34.112646+0000","last_unstale":"2023-03-03T08:43:35.530006+0000","last_undegraded":"2023-03-03T08:43:35.530006+0000","last_fullsized":"2023-03-03T08:43:35.530006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:32.767710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131309+0000","last_change":"2023-03-03T08:43:34.539790+0000","last_active":"2023-03-03T08:43:36.131309+0000","last_peered":"2023-03-03T08:43:36.131309+0000","last_clean":"2023-03-03T08:43:36.131309+0000","last_became_active":"2023-03-03T08:43:34.539443+0000","last_became_peered":"2023-03-03T08:43:34.539443+0000","last_unstale":"2023-03-03T08:43:36.131309+0000","last_undegraded":"2023-03-03T08:43:36.131309+0000","last_fullsized":"2023-03-03T08:43:36.131309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:25.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131521+0000","last_change":"2023-03-03T08:43:34.112905+0000","last_active":"2023-03-03T08:43:36.131521+0000","last_peered":"2023-03-03T08:43:36.131521+0000","last_clean":"2023-03-03T08:43:36.131521+0000","last_became_active":"2023-03-03T08:43:34.112601+0000","last_became_peered":"2023-03-03T08:43:34.112601+0000","last_unstale":"2023-03-03T08:43:36.131521+0000","last_undegraded":"2023-03-03T08:43:36.131521+0000","last_fullsized":"2023-03-03T08:43:36.131521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:01.165783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530721+0000","last_change":"2023-03-03T08:43:34.532656+0000","last_active":"2023-03-03T08:43:35.530721+0000","last_peered":"2023-03-03T08:43:35.530721+0000","last_clean":"2023-03-03T08:43:35.530721+0000","last_became_active":"2023-03-03T08:43:34.532297+0000","last_became_peered":"2023-03-03T08:43:34.532297+0000","last_unstale":"2023-03-03T08:43:35.530721+0000","last_undegraded":"2023-03-03T08:43:35.530721+0000","last_fullsized":"2023-03-03T08:43:35.530721+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:36.347740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529262+0000","last_change":"2023-03-03T08:43:34.115792+0000","last_active":"2023-03-03T08:43:35.529262+0000","last_peered":"2023-03-03T08:43:35.529262+0000","last_clean":"2023-03-03T08:43:35.529262+0000","last_became_active":"2023-03-03T08:43:34.115431+0000","last_became_peered":"2023-03-03T08:43:34.115431+0000","last_unstale":"2023-03-03T08:43:35.529262+0000","last_undegraded":"2023-03-03T08:43:35.529262+0000","last_fullsized":"2023-03-03T08:43:35.529262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:15.484022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.550414+0000","last_change":"2023-03-03T08:43:34.546067+0000","last_active":"2023-03-03T08:43:34.550414+0000","last_peered":"2023-03-03T08:43:34.550414+0000","last_clean":"2023-03-03T08:43:34.550414+0000","last_became_active":"2023-03-03T08:43:34.543786+0000","last_became_peered":"2023-03-03T08:43:34.543786+0000","last_unstale":"2023-03-03T08:43:34.550414+0000","last_undegraded":"2023-03-03T08:43:34.550414+0000","last_fullsized":"2023-03-03T08:43:34.550414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:32.922812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130446+0000","last_change":"2023-03-03T08:43:34.108123+0000","last_active":"2023-03-03T08:43:36.130446+0000","last_peered":"2023-03-03T08:43:36.130446+0000","last_clean":"2023-03-03T08:43:36.130446+0000","last_became_active":"2023-03-03T08:43:34.107757+0000","last_became_peered":"2023-03-03T08:43:34.107757+0000","last_unstale":"2023-03-03T08:43:36.130446+0000","last_undegraded":"2023-03-03T08:43:36.130446+0000","last_fullsized":"2023-03-03T08:43:36.130446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:09.488930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.555860+0000","last_change":"2023-03-03T08:43:34.553597+0000","last_active":"2023-03-03T08:43:34.555860+0000","last_peered":"2023-03-03T08:43:34.555860+0000","last_clean":"2023-03-03T08:43:34.555860+0000","last_became_active":"2023-03-03T08:43:34.553322+0000","last_became_peered":"2023-03-03T08:43:34.553322+0000","last_unstale":"2023-03-03T08:43:34.555860+0000","last_undegraded":"2023-03-03T08:43:34.555860+0000","last_fullsized":"2023-03-03T08:43:34.555860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:24:17.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.552634+0000","last_change":"2023-03-03T08:43:34.110464+0000","last_active":"2023-03-03T08:43:34.552634+0000","last_peered":"2023-03-03T08:43:34.552634+0000","last_clean":"2023-03-03T08:43:34.552634+0000","last_became_active":"2023-03-03T08:43:34.110165+0000","last_became_peered":"2023-03-03T08:43:34.110165+0000","last_unstale":"2023-03-03T08:43:34.552634+0000","last_undegraded":"2023-03-03T08:43:34.552634+0000","last_fullsized":"2023-03-03T08:43:34.552634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:54.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529076+0000","last_change":"2023-03-03T08:43:34.540196+0000","last_active":"2023-03-03T08:43:35.529076+0000","last_peered":"2023-03-03T08:43:35.529076+0000","last_clean":"2023-03-03T08:43:35.529076+0000","last_became_active":"2023-03-03T08:43:34.539931+0000","last_became_peered":"2023-03-03T08:43:34.539931+0000","last_unstale":"2023-03-03T08:43:35.529076+0000","last_undegraded":"2023-03-03T08:43:35.529076+0000","last_fullsized":"2023-03-03T08:43:35.529076+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:20.012997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530091+0000","last_change":"2023-03-03T08:43:34.109890+0000","last_active":"2023-03-03T08:43:35.530091+0000","last_peered":"2023-03-03T08:43:35.530091+0000","last_clean":"2023-03-03T08:43:35.530091+0000","last_became_active":"2023-03-03T08:43:34.109583+0000","last_became_peered":"2023-03-03T08:43:34.109583+0000","last_unstale":"2023-03-03T08:43:35.530091+0000","last_undegraded":"2023-03-03T08:43:35.530091+0000","last_fullsized":"2023-03-03T08:43:35.530091+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:27:34.142294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130657+0000","last_change":"2023-03-03T08:43:34.548010+0000","last_active":"2023-03-03T08:43:36.130657+0000","last_peered":"2023-03-03T08:43:36.130657+0000","last_clean":"2023-03-03T08:43:36.130657+0000","last_became_active":"2023-03-03T08:43:34.547662+0000","last_became_peered":"2023-03-03T08:43:34.547662+0000","last_unstale":"2023-03-03T08:43:36.130657+0000","last_undegraded":"2023-03-03T08:43:36.130657+0000","last_fullsized":"2023-03-03T08:43:36.130657+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:52.200064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530215+0000","last_change":"2023-03-03T08:43:34.117734+0000","last_active":"2023-03-03T08:43:35.530215+0000","last_peered":"2023-03-03T08:43:35.530215+0000","last_clean":"2023-03-03T08:43:35.530215+0000","last_became_active":"2023-03-03T08:43:34.117454+0000","last_became_peered":"2023-03-03T08:43:34.117454+0000","last_unstale":"2023-03-03T08:43:35.530215+0000","last_undegraded":"2023-03-03T08:43:35.530215+0000","last_fullsized":"2023-03-03T08:43:35.530215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:42:27.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529314+0000","last_change":"2023-03-03T08:43:34.538817+0000","last_active":"2023-03-03T08:43:35.529314+0000","last_peered":"2023-03-03T08:43:35.529314+0000","last_clean":"2023-03-03T08:43:35.529314+0000","last_became_active":"2023-03-03T08:43:34.538506+0000","last_became_peered":"2023-03-03T08:43:34.538506+0000","last_unstale":"2023-03-03T08:43:35.529314+0000","last_undegraded":"2023-03-03T08:43:35.529314+0000","last_fullsized":"2023-03-03T08:43:35.529314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.531427+0000","last_change":"2023-03-03T08:43:34.109372+0000","last_active":"2023-03-03T08:43:34.531427+0000","last_peered":"2023-03-03T08:43:34.531427+0000","last_clean":"2023-03-03T08:43:34.531427+0000","last_became_active":"2023-03-03T08:43:34.109023+0000","last_became_peered":"2023-03-03T08:43:34.109023+0000","last_unstale":"2023-03-03T08:43:34.531427+0000","last_undegraded":"2023-03-03T08:43:34.531427+0000","last_fullsized":"2023-03-03T08:43:34.531427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:51.374188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.553945+0000","last_change":"2023-03-03T08:43:34.548304+0000","last_active":"2023-03-03T08:43:34.553945+0000","last_peered":"2023-03-03T08:43:34.553945+0000","last_clean":"2023-03-03T08:43:34.553945+0000","last_became_active":"2023-03-03T08:43:34.547960+0000","last_became_peered":"2023-03-03T08:43:34.547960+0000","last_unstale":"2023-03-03T08:43:34.553945+0000","last_undegraded":"2023-03-03T08:43:34.553945+0000","last_fullsized":"2023-03-03T08:43:34.553945+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:04.907753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130849+0000","last_change":"2023-03-03T08:43:34.109698+0000","last_active":"2023-03-03T08:43:36.130849+0000","last_peered":"2023-03-03T08:43:36.130849+0000","last_clean":"2023-03-03T08:43:36.130849+0000","last_became_active":"2023-03-03T08:43:34.109423+0000","last_became_peered":"2023-03-03T08:43:34.109423+0000","last_unstale":"2023-03-03T08:43:36.130849+0000","last_undegraded":"2023-03-03T08:43:36.130849+0000","last_fullsized":"2023-03-03T08:43:36.130849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:49.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.551105+0000","last_change":"2023-03-03T08:43:34.542308+0000","last_active":"2023-03-03T08:43:34.551105+0000","last_peered":"2023-03-03T08:43:34.551105+0000","last_clean":"2023-03-03T08:43:34.551105+0000","last_became_active":"2023-03-03T08:43:34.541991+0000","last_became_peered":"2023-03-03T08:43:34.541991+0000","last_unstale":"2023-03-03T08:43:34.551105+0000","last_undegraded":"2023-03-03T08:43:34.551105+0000","last_fullsized":"2023-03-03T08:43:34.551105+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:06:21.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529147+0000","last_change":"2023-03-03T08:43:34.106777+0000","last_active":"2023-03-03T08:43:35.529147+0000","last_peered":"2023-03-03T08:43:35.529147+0000","last_clean":"2023-03-03T08:43:35.529147+0000","last_became_active":"2023-03-03T08:43:34.106132+0000","last_became_peered":"2023-03-03T08:43:34.106132+0000","last_unstale":"2023-03-03T08:43:35.529147+0000","last_undegraded":"2023-03-03T08:43:35.529147+0000","last_fullsized":"2023-03-03T08:43:35.529147+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:55.965987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530549+0000","last_change":"2023-03-03T08:43:34.550325+0000","last_active":"2023-03-03T08:43:35.530549+0000","last_peered":"2023-03-03T08:43:35.530549+0000","last_clean":"2023-03-03T08:43:35.530549+0000","last_became_active":"2023-03-03T08:43:34.550041+0000","last_became_peered":"2023-03-03T08:43:34.550041+0000","last_unstale":"2023-03-03T08:43:35.530549+0000","last_undegraded":"2023-03-03T08:43:35.530549+0000","last_fullsized":"2023-03-03T08:43:35.530549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:44:34.082003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529221+0000","last_change":"2023-03-03T08:43:34.111335+0000","last_active":"2023-03-03T08:43:35.529221+0000","last_peered":"2023-03-03T08:43:35.529221+0000","last_clean":"2023-03-03T08:43:35.529221+0000","last_became_active":"2023-03-03T08:43:34.110837+0000","last_became_peered":"2023-03-03T08:43:34.110837+0000","last_unstale":"2023-03-03T08:43:35.529221+0000","last_undegraded":"2023-03-03T08:43:35.529221+0000","last_fullsized":"2023-03-03T08:43:35.529221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:03.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530054+0000","last_change":"2023-03-03T08:43:34.541858+0000","last_active":"2023-03-03T08:43:35.530054+0000","last_peered":"2023-03-03T08:43:35.530054+0000","last_clean":"2023-03-03T08:43:35.530054+0000","last_became_active":"2023-03-03T08:43:34.541430+0000","last_became_peered":"2023-03-03T08:43:34.541430+0000","last_unstale":"2023-03-03T08:43:35.530054+0000","last_undegraded":"2023-03-03T08:43:35.530054+0000","last_fullsized":"2023-03-03T08:43:35.530054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:29.397784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114026+0000","last_change":"2023-03-03T08:43:34.109764+0000","last_active":"2023-03-03T08:43:36.114026+0000","last_peered":"2023-03-03T08:43:36.114026+0000","last_clean":"2023-03-03T08:43:36.114026+0000","last_became_active":"2023-03-03T08:43:34.109462+0000","last_became_peered":"2023-03-03T08:43:34.109462+0000","last_unstale":"2023-03-03T08:43:36.114026+0000","last_undegraded":"2023-03-03T08:43:36.114026+0000","last_fullsized":"2023-03-03T08:43:36.114026+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:56.399192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131050+0000","last_change":"2023-03-03T08:43:34.540006+0000","last_active":"2023-03-03T08:43:36.131050+0000","last_peered":"2023-03-03T08:43:36.131050+0000","last_clean":"2023-03-03T08:43:36.131050+0000","last_became_active":"2023-03-03T08:43:34.539579+0000","last_became_peered":"2023-03-03T08:43:34.539579+0000","last_unstale":"2023-03-03T08:43:36.131050+0000","last_undegraded":"2023-03-03T08:43:36.131050+0000","last_fullsized":"2023-03-03T08:43:36.131050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.106922+0000","last_change":"2023-03-03T08:43:29.107536+0000","last_active":"2023-03-03T08:43:34.106922+0000","last_peered":"2023-03-03T08:43:34.106922+0000","last_clean":"2023-03-03T08:43:34.106922+0000","last_became_active":"2023-03-03T08:43:29.107093+0000","last_became_peered":"2023-03-03T08:43:29.107093+0000","last_unstale":"2023-03-03T08:43:34.106922+0000","last_undegraded":"2023-03-03T08:43:34.106922+0000","last_fullsized":"2023-03-03T08:43:34.106922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:25:47.568904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529848+0000","last_change":"2023-03-03T08:43:34.104692+0000","last_active":"2023-03-03T08:43:35.529848+0000","last_peered":"2023-03-03T08:43:35.529848+0000","last_clean":"2023-03-03T08:43:35.529848+0000","last_became_active":"2023-03-03T08:43:34.104263+0000","last_became_peered":"2023-03-03T08:43:34.104263+0000","last_unstale":"2023-03-03T08:43:35.529848+0000","last_undegraded":"2023-03-03T08:43:35.529848+0000","last_fullsized":"2023-03-03T08:43:35.529848+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:24.537703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130617+0000","last_change":"2023-03-03T08:43:34.526797+0000","last_active":"2023-03-03T08:43:36.130617+0000","last_peered":"2023-03-03T08:43:36.130617+0000","last_clean":"2023-03-03T08:43:36.130617+0000","last_became_active":"2023-03-03T08:43:34.526490+0000","last_became_peered":"2023-03-03T08:43:34.526490+0000","last_unstale":"2023-03-03T08:43:36.130617+0000","last_undegraded":"2023-03-03T08:43:36.130617+0000","last_fullsized":"2023-03-03T08:43:36.130617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:11.808336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.521801+0000","last_change":"2023-03-03T08:43:29.107584+0000","last_active":"2023-03-03T08:43:34.521801+0000","last_peered":"2023-03-03T08:43:34.521801+0000","last_clean":"2023-03-03T08:43:34.521801+0000","last_became_active":"2023-03-03T08:43:29.107208+0000","last_became_peered":"2023-03-03T08:43:29.107208+0000","last_unstale":"2023-03-03T08:43:34.521801+0000","last_undegraded":"2023-03-03T08:43:34.521801+0000","last_fullsized":"2023-03-03T08:43:34.521801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:28:24.087122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.531478+0000","last_change":"2023-03-03T08:43:34.113748+0000","last_active":"2023-03-03T08:43:34.531478+0000","last_peered":"2023-03-03T08:43:34.531478+0000","last_clean":"2023-03-03T08:43:34.531478+0000","last_became_active":"2023-03-03T08:43:34.113190+0000","last_became_peered":"2023-03-03T08:43:34.113190+0000","last_unstale":"2023-03-03T08:43:34.531478+0000","last_undegraded":"2023-03-03T08:43:34.531478+0000","last_fullsized":"2023-03-03T08:43:34.531478+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:58:55.343571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131714+0000","last_change":"2023-03-03T08:43:34.528593+0000","last_active":"2023-03-03T08:43:36.131714+0000","last_peered":"2023-03-03T08:43:36.131714+0000","last_clean":"2023-03-03T08:43:36.131714+0000","last_became_active":"2023-03-03T08:43:34.528260+0000","last_became_peered":"2023-03-03T08:43:34.528260+0000","last_unstale":"2023-03-03T08:43:36.131714+0000","last_undegraded":"2023-03-03T08:43:36.131714+0000","last_fullsized":"2023-03-03T08:43:36.131714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.524683+0000","last_change":"2023-03-03T08:43:29.076367+0000","last_active":"2023-03-03T08:43:34.524683+0000","last_peered":"2023-03-03T08:43:34.524683+0000","last_clean":"2023-03-03T08:43:34.524683+0000","last_became_active":"2023-03-03T08:43:29.075982+0000","last_became_peered":"2023-03-03T08:43:29.075982+0000","last_unstale":"2023-03-03T08:43:34.524683+0000","last_undegraded":"2023-03-03T08:43:34.524683+0000","last_fullsized":"2023-03-03T08:43:34.524683+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:06.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.539766+0000","last_change":"2023-03-03T08:43:34.106337+0000","last_active":"2023-03-03T08:43:34.539766+0000","last_peered":"2023-03-03T08:43:34.539766+0000","last_clean":"2023-03-03T08:43:34.539766+0000","last_became_active":"2023-03-03T08:43:34.105753+0000","last_became_peered":"2023-03-03T08:43:34.105753+0000","last_unstale":"2023-03-03T08:43:34.539766+0000","last_undegraded":"2023-03-03T08:43:34.539766+0000","last_fullsized":"2023-03-03T08:43:34.539766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:46.761924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.132065+0000","last_change":"2023-03-03T08:43:34.536419+0000","last_active":"2023-03-03T08:43:36.132065+0000","last_peered":"2023-03-03T08:43:36.132065+0000","last_clean":"2023-03-03T08:43:36.132065+0000","last_became_active":"2023-03-03T08:43:34.536150+0000","last_became_peered":"2023-03-03T08:43:34.536150+0000","last_unstale":"2023-03-03T08:43:36.132065+0000","last_undegraded":"2023-03-03T08:43:36.132065+0000","last_fullsized":"2023-03-03T08:43:36.132065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:22:54.362024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.521122+0000","last_change":"2023-03-03T08:43:29.160285+0000","last_active":"2023-03-03T08:43:34.521122+0000","last_peered":"2023-03-03T08:43:34.521122+0000","last_clean":"2023-03-03T08:43:34.521122+0000","last_became_active":"2023-03-03T08:43:29.159943+0000","last_became_peered":"2023-03-03T08:43:29.159943+0000","last_unstale":"2023-03-03T08:43:34.521122+0000","last_undegraded":"2023-03-03T08:43:34.521122+0000","last_fullsized":"2023-03-03T08:43:34.521122+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:48:58.997356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.538374+0000","last_change":"2023-03-03T08:43:23.015460+0000","last_active":"2023-03-03T08:43:34.538374+0000","last_peered":"2023-03-03T08:43:34.538374+0000","last_clean":"2023-03-03T08:43:34.538374+0000","last_became_active":"2023-03-03T08:43:23.014485+0000","last_became_peered":"2023-03-03T08:43:23.014485+0000","last_unstale":"2023-03-03T08:43:34.538374+0000","last_undegraded":"2023-03-03T08:43:34.538374+0000","last_fullsized":"2023-03-03T08:43:34.538374+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_clean_scrub_stamp":"2023-03-03T08:43:21.542883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:07.022017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114090+0000","last_change":"2023-03-03T08:43:34.110484+0000","last_active":"2023-03-03T08:43:36.114090+0000","last_peered":"2023-03-03T08:43:36.114090+0000","last_clean":"2023-03-03T08:43:36.114090+0000","last_became_active":"2023-03-03T08:43:34.110220+0000","last_became_peered":"2023-03-03T08:43:34.110220+0000","last_unstale":"2023-03-03T08:43:36.114090+0000","last_undegraded":"2023-03-03T08:43:36.114090+0000","last_fullsized":"2023-03-03T08:43:36.114090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:04:59.232537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530634+0000","last_change":"2023-03-03T08:43:34.550263+0000","last_active":"2023-03-03T08:43:35.530634+0000","last_peered":"2023-03-03T08:43:35.530634+0000","last_clean":"2023-03-03T08:43:35.530634+0000","last_became_active":"2023-03-03T08:43:34.549560+0000","last_became_peered":"2023-03-03T08:43:34.549560+0000","last_unstale":"2023-03-03T08:43:35.530634+0000","last_undegraded":"2023-03-03T08:43:35.530634+0000","last_fullsized":"2023-03-03T08:43:35.530634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:55.874223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.522038+0000","last_change":"2023-03-03T08:43:29.075836+0000","last_active":"2023-03-03T08:43:34.522038+0000","last_peered":"2023-03-03T08:43:34.522038+0000","last_clean":"2023-03-03T08:43:34.522038+0000","last_became_active":"2023-03-03T08:43:29.075564+0000","last_became_peered":"2023-03-03T08:43:29.075564+0000","last_unstale":"2023-03-03T08:43:34.522038+0000","last_undegraded":"2023-03-03T08:43:34.522038+0000","last_fullsized":"2023-03-03T08:43:34.522038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:39.576714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130591+0000","last_change":"2023-03-03T08:43:34.104492+0000","last_active":"2023-03-03T08:43:36.130591+0000","last_peered":"2023-03-03T08:43:36.130591+0000","last_clean":"2023-03-03T08:43:36.130591+0000","last_became_active":"2023-03-03T08:43:34.104151+0000","last_became_peered":"2023-03-03T08:43:34.104151+0000","last_unstale":"2023-03-03T08:43:36.130591+0000","last_undegraded":"2023-03-03T08:43:36.130591+0000","last_fullsized":"2023-03-03T08:43:36.130591+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:57.214738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.550389+0000","last_change":"2023-03-03T08:43:34.544053+0000","last_active":"2023-03-03T08:43:34.550389+0000","last_peered":"2023-03-03T08:43:34.550389+0000","last_clean":"2023-03-03T08:43:34.550389+0000","last_became_active":"2023-03-03T08:43:34.543523+0000","last_became_peered":"2023-03-03T08:43:34.543523+0000","last_unstale":"2023-03-03T08:43:34.550389+0000","last_undegraded":"2023-03-03T08:43:34.550389+0000","last_fullsized":"2023-03-03T08:43:34.550389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:34:03.662535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.523389+0000","last_change":"2023-03-03T08:43:29.072304+0000","last_active":"2023-03-03T08:43:34.523389+0000","last_peered":"2023-03-03T08:43:34.523389+0000","last_clean":"2023-03-03T08:43:34.523389+0000","last_became_active":"2023-03-03T08:43:29.071962+0000","last_became_peered":"2023-03-03T08:43:29.071962+0000","last_unstale":"2023-03-03T08:43:34.523389+0000","last_undegraded":"2023-03-03T08:43:34.523389+0000","last_fullsized":"2023-03-03T08:43:34.523389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:40:12.229226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131749+0000","last_change":"2023-03-03T08:43:34.107783+0000","last_active":"2023-03-03T08:43:36.131749+0000","last_peered":"2023-03-03T08:43:36.131749+0000","last_clean":"2023-03-03T08:43:36.131749+0000","last_became_active":"2023-03-03T08:43:34.107503+0000","last_became_peered":"2023-03-03T08:43:34.107503+0000","last_unstale":"2023-03-03T08:43:36.131749+0000","last_undegraded":"2023-03-03T08:43:36.131749+0000","last_fullsized":"2023-03-03T08:43:36.131749+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:35:42.800814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.131418+0000","last_change":"2023-03-03T08:43:34.535435+0000","last_active":"2023-03-03T08:43:36.131418+0000","last_peered":"2023-03-03T08:43:36.131418+0000","last_clean":"2023-03-03T08:43:36.131418+0000","last_became_active":"2023-03-03T08:43:34.535123+0000","last_became_peered":"2023-03-03T08:43:34.535123+0000","last_unstale":"2023-03-03T08:43:36.131418+0000","last_undegraded":"2023-03-03T08:43:36.131418+0000","last_fullsized":"2023-03-03T08:43:36.131418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:52:10.622422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114468+0000","last_change":"2023-03-03T08:43:34.110105+0000","last_active":"2023-03-03T08:43:36.114468+0000","last_peered":"2023-03-03T08:43:36.114468+0000","last_clean":"2023-03-03T08:43:36.114468+0000","last_became_active":"2023-03-03T08:43:34.109736+0000","last_became_peered":"2023-03-03T08:43:34.109736+0000","last_unstale":"2023-03-03T08:43:36.114468+0000","last_undegraded":"2023-03-03T08:43:36.114468+0000","last_fullsized":"2023-03-03T08:43:36.114468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:30.557847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.536239+0000","last_change":"2023-03-03T08:43:29.354519+0000","last_active":"2023-03-03T08:43:34.536239+0000","last_peered":"2023-03-03T08:43:34.536239+0000","last_clean":"2023-03-03T08:43:34.536239+0000","last_became_active":"2023-03-03T08:43:29.354213+0000","last_became_peered":"2023-03-03T08:43:29.354213+0000","last_unstale":"2023-03-03T08:43:34.536239+0000","last_undegraded":"2023-03-03T08:43:34.536239+0000","last_fullsized":"2023-03-03T08:43:34.536239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:00.406507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114040+0000","last_change":"2023-03-03T08:43:34.539018+0000","last_active":"2023-03-03T08:43:36.114040+0000","last_peered":"2023-03-03T08:43:36.114040+0000","last_clean":"2023-03-03T08:43:36.114040+0000","last_became_active":"2023-03-03T08:43:34.538375+0000","last_became_peered":"2023-03-03T08:43:34.538375+0000","last_unstale":"2023-03-03T08:43:36.114040+0000","last_undegraded":"2023-03-03T08:43:36.114040+0000","last_fullsized":"2023-03-03T08:43:36.114040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:29:02.601410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.521322+0000","last_change":"2023-03-03T08:43:29.354627+0000","last_active":"2023-03-03T08:43:34.521322+0000","last_peered":"2023-03-03T08:43:34.521322+0000","last_clean":"2023-03-03T08:43:34.521322+0000","last_became_active":"2023-03-03T08:43:29.354271+0000","last_became_peered":"2023-03-03T08:43:29.354271+0000","last_unstale":"2023-03-03T08:43:34.521322+0000","last_undegraded":"2023-03-03T08:43:34.521322+0000","last_fullsized":"2023-03-03T08:43:34.521322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:05.946053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114400+0000","last_change":"2023-03-03T08:43:34.109127+0000","last_active":"2023-03-03T08:43:36.114400+0000","last_peered":"2023-03-03T08:43:36.114400+0000","last_clean":"2023-03-03T08:43:36.114400+0000","last_became_active":"2023-03-03T08:43:34.108869+0000","last_became_peered":"2023-03-03T08:43:34.108869+0000","last_unstale":"2023-03-03T08:43:36.114400+0000","last_undegraded":"2023-03-03T08:43:36.114400+0000","last_fullsized":"2023-03-03T08:43:36.114400+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:19:00.275620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529000+0000","last_change":"2023-03-03T08:43:34.537850+0000","last_active":"2023-03-03T08:43:35.529000+0000","last_peered":"2023-03-03T08:43:35.529000+0000","last_clean":"2023-03-03T08:43:35.529000+0000","last_became_active":"2023-03-03T08:43:34.537550+0000","last_became_peered":"2023-03-03T08:43:34.537550+0000","last_unstale":"2023-03-03T08:43:35.529000+0000","last_undegraded":"2023-03-03T08:43:35.529000+0000","last_fullsized":"2023-03-03T08:43:35.529000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:37:36.893757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114319+0000","last_change":"2023-03-03T08:43:34.112667+0000","last_active":"2023-03-03T08:43:36.114319+0000","last_peered":"2023-03-03T08:43:36.114319+0000","last_clean":"2023-03-03T08:43:36.114319+0000","last_became_active":"2023-03-03T08:43:34.112386+0000","last_became_peered":"2023-03-03T08:43:34.112386+0000","last_unstale":"2023-03-03T08:43:36.114319+0000","last_undegraded":"2023-03-03T08:43:36.114319+0000","last_fullsized":"2023-03-03T08:43:36.114319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:40.504392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.114284+0000","last_change":"2023-03-03T08:43:34.541172+0000","last_active":"2023-03-03T08:43:36.114284+0000","last_peered":"2023-03-03T08:43:36.114284+0000","last_clean":"2023-03-03T08:43:36.114284+0000","last_became_active":"2023-03-03T08:43:34.540739+0000","last_became_peered":"2023-03-03T08:43:34.540739+0000","last_unstale":"2023-03-03T08:43:36.114284+0000","last_undegraded":"2023-03-03T08:43:36.114284+0000","last_fullsized":"2023-03-03T08:43:36.114284+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:40.058265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130597+0000","last_change":"2023-03-03T08:43:34.110187+0000","last_active":"2023-03-03T08:43:36.130597+0000","last_peered":"2023-03-03T08:43:36.130597+0000","last_clean":"2023-03-03T08:43:36.130597+0000","last_became_active":"2023-03-03T08:43:34.109845+0000","last_became_peered":"2023-03-03T08:43:34.109845+0000","last_unstale":"2023-03-03T08:43:36.130597+0000","last_undegraded":"2023-03-03T08:43:36.130597+0000","last_fullsized":"2023-03-03T08:43:36.130597+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:33.507500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529558+0000","last_change":"2023-03-03T08:43:34.547322+0000","last_active":"2023-03-03T08:43:35.529558+0000","last_peered":"2023-03-03T08:43:35.529558+0000","last_clean":"2023-03-03T08:43:35.529558+0000","last_became_active":"2023-03-03T08:43:34.546692+0000","last_became_peered":"2023-03-03T08:43:34.546692+0000","last_unstale":"2023-03-03T08:43:35.529558+0000","last_undegraded":"2023-03-03T08:43:35.529558+0000","last_fullsized":"2023-03-03T08:43:35.529558+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:11:26.366920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.530453+0000","last_change":"2023-03-03T08:43:34.117851+0000","last_active":"2023-03-03T08:43:35.530453+0000","last_peered":"2023-03-03T08:43:35.530453+0000","last_clean":"2023-03-03T08:43:35.530453+0000","last_became_active":"2023-03-03T08:43:34.117548+0000","last_became_peered":"2023-03-03T08:43:34.117548+0000","last_unstale":"2023-03-03T08:43:35.530453+0000","last_undegraded":"2023-03-03T08:43:35.530453+0000","last_fullsized":"2023-03-03T08:43:35.530453+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:20.039817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.529223+0000","last_change":"2023-03-03T08:43:34.535451+0000","last_active":"2023-03-03T08:43:35.529223+0000","last_peered":"2023-03-03T08:43:35.529223+0000","last_clean":"2023-03-03T08:43:35.529223+0000","last_became_active":"2023-03-03T08:43:34.535129+0000","last_became_peered":"2023-03-03T08:43:34.535129+0000","last_unstale":"2023-03-03T08:43:35.529223+0000","last_undegraded":"2023-03-03T08:43:35.529223+0000","last_fullsized":"2023-03-03T08:43:35.529223+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:30.645128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.552703+0000","last_change":"2023-03-03T08:43:34.109782+0000","last_active":"2023-03-03T08:43:34.552703+0000","last_peered":"2023-03-03T08:43:34.552703+0000","last_clean":"2023-03-03T08:43:34.552703+0000","last_became_active":"2023-03-03T08:43:34.109513+0000","last_became_peered":"2023-03-03T08:43:34.109513+0000","last_unstale":"2023-03-03T08:43:34.552703+0000","last_undegraded":"2023-03-03T08:43:34.552703+0000","last_fullsized":"2023-03-03T08:43:34.552703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:16.185161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130648+0000","last_change":"2023-03-03T08:43:34.532482+0000","last_active":"2023-03-03T08:43:36.130648+0000","last_peered":"2023-03-03T08:43:36.130648+0000","last_clean":"2023-03-03T08:43:36.130648+0000","last_became_active":"2023-03-03T08:43:34.531751+0000","last_became_peered":"2023-03-03T08:43:34.531751+0000","last_unstale":"2023-03-03T08:43:36.130648+0000","last_undegraded":"2023-03-03T08:43:36.130648+0000","last_fullsized":"2023-03-03T08:43:36.130648+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:47:45.214945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.553022+0000","last_change":"2023-03-03T08:43:34.108426+0000","last_active":"2023-03-03T08:43:34.553022+0000","last_peered":"2023-03-03T08:43:34.553022+0000","last_clean":"2023-03-03T08:43:34.553022+0000","last_became_active":"2023-03-03T08:43:34.108009+0000","last_became_peered":"2023-03-03T08:43:34.108009+0000","last_unstale":"2023-03-03T08:43:34.553022+0000","last_undegraded":"2023-03-03T08:43:34.553022+0000","last_fullsized":"2023-03-03T08:43:34.553022+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:53.632744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:35.528838+0000","last_change":"2023-03-03T08:43:34.540812+0000","last_active":"2023-03-03T08:43:35.528838+0000","last_peered":"2023-03-03T08:43:35.528838+0000","last_clean":"2023-03-03T08:43:35.528838+0000","last_became_active":"2023-03-03T08:43:34.540549+0000","last_became_peered":"2023-03-03T08:43:34.540549+0000","last_unstale":"2023-03-03T08:43:35.528838+0000","last_undegraded":"2023-03-03T08:43:35.528838+0000","last_fullsized":"2023-03-03T08:43:35.528838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130471+0000","last_change":"2023-03-03T08:43:34.547197+0000","last_active":"2023-03-03T08:43:36.130471+0000","last_peered":"2023-03-03T08:43:36.130471+0000","last_clean":"2023-03-03T08:43:36.130471+0000","last_became_active":"2023-03-03T08:43:34.546927+0000","last_became_peered":"2023-03-03T08:43:34.546927+0000","last_unstale":"2023-03-03T08:43:36.130471+0000","last_undegraded":"2023-03-03T08:43:36.130471+0000","last_fullsized":"2023-03-03T08:43:36.130471+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:21:34.845968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-03-03T08:43:36.130700+0000","last_change":"2023-03-03T08:43:34.538786+0000","last_active":"2023-03-03T08:43:36.130700+0000","last_peered":"2023-03-03T08:43:36.130700+0000","last_clean":"2023-03-03T08:43:36.130700+0000","last_became_active":"2023-03-03T08:43:34.538494+0000","last_became_peered":"2023-03-03T08:43:34.538494+0000","last_unstale":"2023-03-03T08:43:36.130700+0000","last_undegraded":"2023-03-03T08:43:36.130700+0000","last_fullsized":"2023-03-03T08:43:36.130700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:53.838286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.551228+0000","last_change":"2023-03-03T08:43:34.544408+0000","last_active":"2023-03-03T08:43:34.551228+0000","last_peered":"2023-03-03T08:43:34.551228+0000","last_clean":"2023-03-03T08:43:34.551228+0000","last_became_active":"2023-03-03T08:43:34.543977+0000","last_became_peered":"2023-03-03T08:43:34.543977+0000","last_unstale":"2023-03-03T08:43:34.551228+0000","last_undegraded":"2023-03-03T08:43:34.551228+0000","last_fullsized":"2023-03-03T08:43:34.551228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.074082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.555804+0000","last_change":"2023-03-03T08:43:34.552931+0000","last_active":"2023-03-03T08:43:34.555804+0000","last_peered":"2023-03-03T08:43:34.555804+0000","last_clean":"2023-03-03T08:43:34.555804+0000","last_became_active":"2023-03-03T08:43:34.552623+0000","last_became_peered":"2023-03-03T08:43:34.552623+0000","last_unstale":"2023-03-03T08:43:34.555804+0000","last_undegraded":"2023-03-03T08:43:34.555804+0000","last_fullsized":"2023-03-03T08:43:34.555804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:07:43.954897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-03-03T08:43:34.550464+0000","last_change":"2023-03-03T08:43:34.546828+0000","last_active":"2023-03-03T08:43:34.550464+0000","last_peered":"2023-03-03T08:43:34.550464+0000","last_clean":"2023-03-03T08:43:34.550464+0000","last_became_active":"2023-03-03T08:43:34.546559+0000","last_became_peered":"2023-03-03T08:43:34.546559+0000","last_unstale":"2023-03-03T08:43:34.550464+0000","last_undegraded":"2023-03-03T08:43:34.550464+0000","last_fullsized":"2023-03-03T08:43:34.550464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:21:14.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7972,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363868,"statfs":{"total":96636764160,"available":96628600832,"internally_reserved":0,"allocated":225280,"data_stored":81872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7980,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363860,"statfs":{"total":96636764160,"available":96628592640,"internally_reserved":0,"allocated":233472,"data_stored":82981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9092,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362748,"statfs":{"total":96636764160,"available":96627453952,"internally_reserved":0,"allocated":847872,"data_stored":680545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":82018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7984,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363856,"statfs":{"total":96636764160,"available":96628588544,"internally_reserved":0,"allocated":237568,"data_stored":83420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8056,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":94363784,"statfs":{"total":96636764160,"available":96628514816,"internally_reserved":0,"allocated":245760,"data_stored":83524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7976,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363864,"statfs":{"total":96636764160,"available":96628596736,"internally_reserved":0,"allocated":229376,"data_stored":82891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9068,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94362772,"statfs":{"total":96636764160,"available":96627478528,"internally_reserved":0,"allocated":823296,"data_stored":672806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:43:40.036 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T08:43:40.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:40.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.190+0000 7f1852729700 1 Processor -- start 2023-03-03T08:43:40.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.190+0000 7f1852729700 1 -- start start 2023-03-03T08:43:40.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1852729700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c0beab0 0x7f184c155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1852729700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f184c1562c0 0x7f184c15cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1852729700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c1523b0 con 0x7f184c152e30 2023-03-03T08:43:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1852729700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184c072010 con 0x7f184c1562c0 2023-03-03T08:43:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1852729700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0aaaf0 con 0x7f184c0beab0 2023-03-03T08:43:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f184bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f184c152e30 0x7f184c0c43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45222/0 (socket says 172.21.15.7:45222) 2023-03-03T08:43:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f184bfff700 1 -- 172.21.15.7:0/3317280423 learned_addr learned my addr 172.21.15.7:0/3317280423 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1850cc6700 1 --2- 172.21.15.7:0/3317280423 >> v2:172.21.15.7:3300/0 conn(0x7f184c1562c0 0x7f184c15cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f184affd700 1 -- 172.21.15.7:0/3317280423 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401544176 0 0) 0x7f184c1523b0 con 0x7f184c152e30 2023-03-03T08:43:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f184b7fe700 1 --2- 172.21.15.7:0/3317280423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c0beab0 0x7f184c155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f184affd700 1 -- 172.21.15.7:0/3317280423 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18380035b0 con 0x7f184c152e30 2023-03-03T08:43:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1850cc6700 1 -- 172.21.15.7:0/3317280423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c0beab0 msgr2=0x7f184c155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1850cc6700 1 --2- 172.21.15.7:0/3317280423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c0beab0 0x7f184c155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1850cc6700 1 -- 172.21.15.7:0/3317280423 >> v1:172.21.15.7:6789/0 conn(0x7f184c152e30 legacy=0x7f184c0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1850cc6700 1 -- 172.21.15.7:0/3317280423 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c0aadd0 con 0x7f184c1562c0 2023-03-03T08:43:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f1850cc6700 1 --2- 172.21.15.7:0/3317280423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c1562c0 0x7f184c15cd30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1844009d90 tx=0x7f184400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0003a977e440ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.191+0000 7f184affd700 1 -- 172.21.15.7:0/3317280423 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184400e070 con 0x7f184c1562c0 2023-03-03T08:43:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.192+0000 7f184affd700 1 -- 172.21.15.7:0/3317280423 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184400bb20 con 0x7f184c1562c0 2023-03-03T08:43:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.192+0000 7f184affd700 1 -- 172.21.15.7:0/3317280423 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844012430 con 0x7f184c1562c0 2023-03-03T08:43:40.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.192+0000 7f1852729700 1 -- 172.21.15.7:0/3317280423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c1562c0 msgr2=0x7f184c15cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.192+0000 7f1852729700 1 --2- 172.21.15.7:0/3317280423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c1562c0 0x7f184c15cd30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1844009d90 tx=0x7f184400b330 comp rx=0 tx=0).stop 2023-03-03T08:43:40.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.192+0000 7f1852729700 1 -- 172.21.15.7:0/3317280423 shutdown_connections 2023-03-03T08:43:40.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.192+0000 7f1852729700 1 --2- 172.21.15.7:0/3317280423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c1562c0 0x7f184c15cd30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.192+0000 7f1852729700 1 --2- 172.21.15.7:0/3317280423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c0beab0 0x7f184c155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.192+0000 7f1852729700 1 -- 172.21.15.7:0/3317280423 >> 172.21.15.7:0/3317280423 conn(0x7f184c0736f0 msgr2=0x7f184c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:40.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.192+0000 7f1852729700 1 -- 172.21.15.7:0/3317280423 shutdown_connections 2023-03-03T08:43:40.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.192+0000 7f1852729700 1 -- 172.21.15.7:0/3317280423 wait complete. 2023-03-03T08:43:40.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.193+0000 7f1852729700 1 Processor -- start 2023-03-03T08:43:40.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.193+0000 7f1852729700 1 -- start start 2023-03-03T08:43:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f1852729700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c0beab0 0x7f184c0adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f1852729700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c152e30 0x7f184c155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f1852729700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c1562c0 0x7f184c0aa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f1852729700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c153630 con 0x7f184c152e30 2023-03-03T08:43:40.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f1852729700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c154c20 con 0x7f184c1562c0 2023-03-03T08:43:40.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f1852729700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c154130 con 0x7f184c0beab0 2023-03-03T08:43:40.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f184b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c152e30 0x7f184c155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f184b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c152e30 0x7f184c155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51250/0 (socket says 172.21.15.7:51250) 2023-03-03T08:43:40.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f184b7fe700 1 -- 172.21.15.7:0/3473871909 learned_addr learned my addr 172.21.15.7:0/3473871909 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:40.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f184bfff700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c0beab0 0x7f184c0adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f1850cc6700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c1562c0 0x7f184c0aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f184b7fe700 1 -- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c0beab0 msgr2=0x7f184c0adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f184b7fe700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c0beab0 0x7f184c0adf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f184b7fe700 1 -- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c1562c0 msgr2=0x7f184c0aa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.194+0000 7f184b7fe700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c1562c0 0x7f184c0aa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.195+0000 7f184b7fe700 1 -- 172.21.15.7:0/3473871909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1844009a70 con 0x7f184c152e30 2023-03-03T08:43:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.195+0000 7f1850cc6700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c1562c0 0x7f184c0aa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:40.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.195+0000 7f184b7fe700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c152e30 0x7f184c155380 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f183400d800 tx=0x7f183400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.195+0000 7f1848ff9700 1 -- 172.21.15.7:0/3473871909 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18340125f0 con 0x7f184c152e30 2023-03-03T08:43:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.195+0000 7f1852729700 1 -- 172.21.15.7:0/3473871909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c153200 con 0x7f184c152e30 2023-03-03T08:43:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.195+0000 7f1848ff9700 1 -- 172.21.15.7:0/3473871909 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1834012750 con 0x7f184c152e30 2023-03-03T08:43:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.195+0000 7f1848ff9700 1 -- 172.21.15.7:0/3473871909 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1834012b00 con 0x7f184c152e30 2023-03-03T08:43:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.195+0000 7f1852729700 1 -- 172.21.15.7:0/3473871909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c071c70 con 0x7f184c152e30 2023-03-03T08:43:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.196+0000 7f1852729700 1 -- 172.21.15.7:0/3473871909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184c153390 con 0x7f184c152e30 2023-03-03T08:43:40.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.196+0000 7f1848ff9700 1 -- 172.21.15.7:0/3473871909 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1834012c60 con 0x7f184c152e30 2023-03-03T08:43:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.197+0000 7f1848ff9700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f182c0682f0 0x7f182c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.197+0000 7f1848ff9700 1 -- 172.21.15.7:0/3473871909 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f1834086140 con 0x7f184c152e30 2023-03-03T08:43:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.197+0000 7f184bfff700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f182c0682f0 0x7f182c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.198+0000 7f184bfff700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f182c0682f0 0x7f182c06a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f183c007fd0 tx=0x7f183c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:40.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.201+0000 7f1848ff9700 1 -- 172.21.15.7:0/3473871909 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1834052930 con 0x7f184c152e30 2023-03-03T08:43:40.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.396+0000 7f1852729700 1 -- 172.21.15.7:0/3473871909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f184c074bb0 con 0x7f184c152e30 2023-03-03T08:43:40.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.397+0000 7f1848ff9700 1 -- 172.21.15.7:0/3473871909 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+17841 (secure 0 0 0) 0x7f1834055f50 con 0x7f184c152e30 2023-03-03T08:43:40.399 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:40.399 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:35.518672+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 -- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f182c0682f0 msgr2=0x7f182c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f182c0682f0 0x7f182c06a7a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f183c007fd0 tx=0x7f183c009040 comp rx=0 tx=0).stop 2023-03-03T08:43:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 -- 172.21.15.7:0/3473871909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c152e30 msgr2=0x7f184c155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c152e30 0x7f184c155380 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f183400d800 tx=0x7f183400db10 comp rx=0 tx=0).stop 2023-03-03T08:43:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 -- 172.21.15.7:0/3473871909 shutdown_connections 2023-03-03T08:43:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f184c1562c0 0x7f184c0aa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f184c0beab0 0x7f184c0adf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f182c0682f0 0x7f182c06a7a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 --2- 172.21.15.7:0/3473871909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f184c152e30 0x7f184c155380 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 -- 172.21.15.7:0/3473871909 >> 172.21.15.7:0/3473871909 conn(0x7f184c0736f0 msgr2=0x7f184c0be330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 -- 172.21.15.7:0/3473871909 shutdown_connections 2023-03-03T08:43:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.399+0000 7f1852729700 1 -- 172.21.15.7:0/3473871909 wait complete. 2023-03-03T08:43:40.413 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T08:43:21.160680+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T08:43:28.018457+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-03-03T08:43:32.305037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-03-03T08:43:33.484501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:43:40.414 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:43:40.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.539+0000 7f6547a88700 1 Processor -- start 2023-03-03T08:43:40.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.539+0000 7f6547a88700 1 -- start start 2023-03-03T08:43:40.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.540+0000 7f6547a88700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 0x7f6540155a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.540+0000 7f6547a88700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65400c4fc0 con 0x7f6540155f40 2023-03-03T08:43:40.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.540+0000 7f6547a88700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65400c4a90 con 0x7f654014fde0 2023-03-03T08:43:40.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.540+0000 7f6547a88700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65400c5730 con 0x7f65400a5d10 2023-03-03T08:43:40.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.540+0000 7f6546025700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6540155f40 0x7f654015cd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45240/0 (socket says 172.21.15.7:45240) 2023-03-03T08:43:40.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.540+0000 7f6546025700 1 -- 172.21.15.7:0/1010584763 learned_addr learned my addr 172.21.15.7:0/1010584763 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:40.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.540+0000 7f6545023700 1 --2- 172.21.15.7:0/1010584763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 0x7f6540155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.540+0000 7f6544822700 1 -- 172.21.15.7:0/1010584763 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240524356 0 0) 0x7f65400c4fc0 con 0x7f6540155f40 2023-03-03T08:43:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.541+0000 7f6544822700 1 -- 172.21.15.7:0/1010584763 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f652c0035b0 con 0x7f6540155f40 2023-03-03T08:43:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.541+0000 7f6544822700 1 -- 172.21.15.7:0/1010584763 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058959896 0 0) 0x7f652c0035b0 con 0x7f6540155f40 2023-03-03T08:43:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.541+0000 7f6545023700 1 -- 172.21.15.7:0/1010584763 >> v1:172.21.15.143:6789/0 conn(0x7f654014fde0 legacy=0x7f65400abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.541+0000 7f6545023700 1 -- 172.21.15.7:0/1010584763 >> v1:172.21.15.7:6789/0 conn(0x7f6540155f40 legacy=0x7f654015cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.541+0000 7f6545023700 1 -- 172.21.15.7:0/1010584763 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65401501b0 con 0x7f65400a5d10 2023-03-03T08:43:40.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.541+0000 7f6545023700 1 --2- 172.21.15.7:0/1010584763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 0x7f6540155a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6530009870 tx=0x7f6530009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e894e3b96bbbf9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.541+0000 7f6544822700 1 -- 172.21.15.7:0/1010584763 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65380019f0 con 0x7f65400a5d10 2023-03-03T08:43:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.541+0000 7f6544822700 1 -- 172.21.15.7:0/1010584763 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6530004030 con 0x7f65400a5d10 2023-03-03T08:43:40.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.541+0000 7f6544822700 1 -- 172.21.15.7:0/1010584763 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65300043c0 con 0x7f65400a5d10 2023-03-03T08:43:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.542+0000 7f6547a88700 1 -- 172.21.15.7:0/1010584763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 msgr2=0x7f6540155a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.542+0000 7f6547a88700 1 --2- 172.21.15.7:0/1010584763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 0x7f6540155a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6530009870 tx=0x7f6530009b80 comp rx=0 tx=0).stop 2023-03-03T08:43:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.542+0000 7f6547a88700 1 -- 172.21.15.7:0/1010584763 shutdown_connections 2023-03-03T08:43:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.542+0000 7f6547a88700 1 --2- 172.21.15.7:0/1010584763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 0x7f6540155a00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.542+0000 7f6547a88700 1 -- 172.21.15.7:0/1010584763 >> 172.21.15.7:0/1010584763 conn(0x7f654006a2b0 msgr2=0x7f654006a680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:40.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.542+0000 7f6547a88700 1 -- 172.21.15.7:0/1010584763 shutdown_connections 2023-03-03T08:43:40.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.542+0000 7f6547a88700 1 -- 172.21.15.7:0/1010584763 wait complete. 2023-03-03T08:43:40.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.543+0000 7f6547a88700 1 Processor -- start 2023-03-03T08:43:40.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.543+0000 7f6547a88700 1 -- start start 2023-03-03T08:43:40.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6547a88700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 0x7f654014ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6547a88700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f654014fde0 0x7f6540147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6547a88700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6540155f40 0x7f6540147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6547a88700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540147f90 con 0x7f654014fde0 2023-03-03T08:43:40.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6547a88700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540154d90 con 0x7f6540155f40 2023-03-03T08:43:40.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6547a88700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65400bd320 con 0x7f65400a5d10 2023-03-03T08:43:40.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6545023700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f654014fde0 0x7f6540147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6545023700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f654014fde0 0x7f6540147510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51264/0 (socket says 172.21.15.7:51264) 2023-03-03T08:43:40.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6545023700 1 -- 172.21.15.7:0/1495509700 learned_addr learned my addr 172.21.15.7:0/1495509700 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:40.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6545824700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 0x7f654014ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6546025700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6540155f40 0x7f6540147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6545023700 1 -- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 msgr2=0x7f654014ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6545023700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 0x7f654014ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6545023700 1 -- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6540155f40 msgr2=0x7f6540147a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6545023700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6540155f40 0x7f6540147a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6545023700 1 -- 172.21.15.7:0/1495509700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65300094a0 con 0x7f654014fde0 2023-03-03T08:43:40.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.544+0000 7f6545023700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f654014fde0 0x7f6540147510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f65300090b0 tx=0x7f6530004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.545+0000 7f65367fc700 1 -- 172.21.15.7:0/1495509700 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530003ea0 con 0x7f654014fde0 2023-03-03T08:43:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.545+0000 7f65367fc700 1 -- 172.21.15.7:0/1495509700 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653001f070 con 0x7f654014fde0 2023-03-03T08:43:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.545+0000 7f65367fc700 1 -- 172.21.15.7:0/1495509700 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530011850 con 0x7f654014fde0 2023-03-03T08:43:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.545+0000 7f6547a88700 1 -- 172.21.15.7:0/1495509700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65400c5450 con 0x7f654014fde0 2023-03-03T08:43:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.545+0000 7f6547a88700 1 -- 172.21.15.7:0/1495509700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65400c5170 con 0x7f654014fde0 2023-03-03T08:43:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.546+0000 7f6547a88700 1 -- 172.21.15.7:0/1495509700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65400c4ce0 con 0x7f654014fde0 2023-03-03T08:43:40.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.546+0000 7f65367fc700 1 -- 172.21.15.7:0/1495509700 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6530017020 con 0x7f654014fde0 2023-03-03T08:43:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.546+0000 7f65367fc700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65240682f0 0x7f652406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.546+0000 7f65367fc700 1 -- 172.21.15.7:0/1495509700 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f6530021070 con 0x7f654014fde0 2023-03-03T08:43:40.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.547+0000 7f6545824700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65240682f0 0x7f652406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.547+0000 7f6545824700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65240682f0 0x7f652406a7a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f653c000f60 tx=0x7f653c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:40.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.551+0000 7f65367fc700 1 -- 172.21.15.7:0/1495509700 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6530053d90 con 0x7f654014fde0 2023-03-03T08:43:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.744+0000 7f6547a88700 1 -- 172.21.15.7:0/1495509700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6540074bb0 con 0x7f654014fde0 2023-03-03T08:43:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.744+0000 7f65367fc700 1 -- 172.21.15.7:0/1495509700 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f65300573b0 con 0x7f654014fde0 2023-03-03T08:43:40.746 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-03-03T08:43:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 -- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65240682f0 msgr2=0x7f652406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65240682f0 0x7f652406a7a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f653c000f60 tx=0x7f653c009040 comp rx=0 tx=0).stop 2023-03-03T08:43:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 -- 172.21.15.7:0/1495509700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f654014fde0 msgr2=0x7f6540147510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f654014fde0 0x7f6540147510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f65300090b0 tx=0x7f6530004750 comp rx=0 tx=0).stop 2023-03-03T08:43:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 -- 172.21.15.7:0/1495509700 shutdown_connections 2023-03-03T08:43:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6540155f40 0x7f6540147a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65400a5d10 0x7f654014ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65240682f0 0x7f652406a7a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 --2- 172.21.15.7:0/1495509700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f654014fde0 0x7f6540147510 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 -- 172.21.15.7:0/1495509700 >> 172.21.15.7:0/1495509700 conn(0x7f654006a2b0 msgr2=0x7f65400c1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:40.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 -- 172.21.15.7:0/1495509700 shutdown_connections 2023-03-03T08:43:40.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.746+0000 7f6547a88700 1 -- 172.21.15.7:0/1495509700 wait complete. 2023-03-03T08:43:40.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:43:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.885+0000 7f93779f3700 1 Processor -- start 2023-03-03T08:43:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.885+0000 7f93779f3700 1 -- start start 2023-03-03T08:43:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f93779f3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9370155390 0x7f93700c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f93779f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937006e020 0x7f9370071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f93779f3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9370154910 con 0x7f9370072520 2023-03-03T08:43:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f93779f3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9370070b90 con 0x7f9370155390 2023-03-03T08:43:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f93779f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9370151320 con 0x7f937006e020 2023-03-03T08:43:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9375f90700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9370072520 0x7f9370151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45254/0 (socket says 172.21.15.7:45254) 2023-03-03T08:43:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9375f90700 1 -- 172.21.15.7:0/3820108295 learned_addr learned my addr 172.21.15.7:0/3820108295 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9374f8e700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937006e020 0x7f9370071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f937578f700 1 --2- 172.21.15.7:0/3820108295 >> v2:172.21.15.143:3300/0 conn(0x7f9370155390 0x7f93700c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534007683 0 0) 0x7f9370154910 con 0x7f9370072520 2023-03-03T08:43:40.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f935c0035b0 con 0x7f9370072520 2023-03-03T08:43:40.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3813650316 0 0) 0x7f935c0035b0 con 0x7f9370072520 2023-03-03T08:43:40.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937006e020 msgr2=0x7f9370071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:40.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937006e020 0x7f9370071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 >> v2:172.21.15.143:3300/0 conn(0x7f9370155390 msgr2=0x7f93700c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:40.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 --2- 172.21.15.7:0/3820108295 >> v2:172.21.15.143:3300/0 conn(0x7f9370155390 0x7f93700c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93700702a0 con 0x7f9370072520 2023-03-03T08:43:40.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f93680019f0 con 0x7f9370072520 2023-03-03T08:43:40.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 >> v1:172.21.15.7:6789/0 conn(0x7f9370072520 legacy=0x7f9370151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 0x7f935c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c004850 0x7f935c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937006e020 0x7f935c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9370070b90 con 0x7f935c003a70 2023-03-03T08:43:40.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9370151320 con 0x7f935c004850 2023-03-03T08:43:40.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.886+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c00ba40 con 0x7f937006e020 2023-03-03T08:43:40.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f937578f700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 0x7f935c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f9374f8e700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c004850 0x7f935c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f9375f90700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937006e020 0x7f935c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f937578f700 1 -- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937006e020 msgr2=0x7f935c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:40.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f937578f700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f937006e020 0x7f935c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f937578f700 1 -- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c004850 msgr2=0x7f935c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:40.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f937578f700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c004850 0x7f935c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f937578f700 1 -- 172.21.15.7:0/3820108295 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c0035b0 con 0x7f935c003a70 2023-03-03T08:43:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f9374f8e700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f935c004850 0x7f935c004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f937578f700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 0x7f935c003e80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f936c009b60 tx=0x7f935c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60af5caeaf1fc15c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c00e6c0 con 0x7f935c003a70 2023-03-03T08:43:40.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c00e820 con 0x7f935c003a70 2023-03-03T08:43:40.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.887+0000 7f9367fff700 1 -- 172.21.15.7:0/3820108295 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c00eb90 con 0x7f935c003a70 2023-03-03T08:43:40.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.888+0000 7f93779f3700 1 -- 172.21.15.7:0/3820108295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 msgr2=0x7f935c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.888+0000 7f93779f3700 1 --2- 172.21.15.7:0/3820108295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 0x7f935c003e80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f936c009b60 tx=0x7f935c005880 comp rx=0 tx=0).stop 2023-03-03T08:43:40.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.888+0000 7f9375f90700 1 -- 172.21.15.7:0/3820108295 reap_dead start 2023-03-03T08:43:40.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.888+0000 7f93779f3700 1 -- 172.21.15.7:0/3820108295 shutdown_connections 2023-03-03T08:43:40.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.888+0000 7f93779f3700 1 -- 172.21.15.7:0/3820108295 >> 172.21.15.7:0/3820108295 conn(0x7f937006c110 msgr2=0x7f937006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:40.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.888+0000 7f93779f3700 1 -- 172.21.15.7:0/3820108295 shutdown_connections 2023-03-03T08:43:40.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.888+0000 7f93779f3700 1 -- 172.21.15.7:0/3820108295 wait complete. 2023-03-03T08:43:40.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.889+0000 7f93779f3700 1 Processor -- start 2023-03-03T08:43:40.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.889+0000 7f93779f3700 1 -- start start 2023-03-03T08:43:40.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f93779f3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 0x7f93700c5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f93779f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370148090 0x7f937014a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f93779f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93701ea6e0 0x7f9370146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f93779f3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93701ea2c0 con 0x7f935c003a70 2023-03-03T08:43:40.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f93779f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93700c8110 con 0x7f9370148090 2023-03-03T08:43:40.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f93779f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93700c17d0 con 0x7f93701ea6e0 2023-03-03T08:43:40.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f937578f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 0x7f93700c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f937578f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 0x7f93700c5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51282/0 (socket says 172.21.15.7:51282) 2023-03-03T08:43:40.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f937578f700 1 -- 172.21.15.7:0/87988317 learned_addr learned my addr 172.21.15.7:0/87988317 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:40.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f9375f90700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93701ea6e0 0x7f9370146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f9374f8e700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370148090 0x7f937014a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f937578f700 1 -- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93701ea6e0 msgr2=0x7f9370146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f937578f700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93701ea6e0 0x7f9370146f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f937578f700 1 -- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370148090 msgr2=0x7f937014a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:40.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f937578f700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370148090 0x7f937014a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:40.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f937578f700 1 -- 172.21.15.7:0/87988317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9370070820 con 0x7f935c003a70 2023-03-03T08:43:40.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f937578f700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 0x7f93700c5590 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f936c0090b0 tx=0x7f936c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:40.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f9365ffb700 1 -- 172.21.15.7:0/87988317 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c00e6c0 con 0x7f935c003a70 2023-03-03T08:43:40.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f9365ffb700 1 -- 172.21.15.7:0/87988317 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c003ad0 con 0x7f935c003a70 2023-03-03T08:43:40.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f93779f3700 1 -- 172.21.15.7:0/87988317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9370070540 con 0x7f935c003a70 2023-03-03T08:43:40.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.890+0000 7f9365ffb700 1 -- 172.21.15.7:0/87988317 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c003e80 con 0x7f935c003a70 2023-03-03T08:43:40.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.891+0000 7f93779f3700 1 -- 172.21.15.7:0/87988317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9370155760 con 0x7f935c003a70 2023-03-03T08:43:40.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.891+0000 7f93779f3700 1 -- 172.21.15.7:0/87988317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93700715f0 con 0x7f935c003a70 2023-03-03T08:43:40.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.892+0000 7f9365ffb700 1 -- 172.21.15.7:0/87988317 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f936c025070 con 0x7f935c003a70 2023-03-03T08:43:40.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.892+0000 7f9365ffb700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93540682f0 0x7f935406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:40.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.892+0000 7f9365ffb700 1 -- 172.21.15.7:0/87988317 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f936c087520 con 0x7f935c003a70 2023-03-03T08:43:40.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.892+0000 7f9374f8e700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93540682f0 0x7f935406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:40.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.893+0000 7f9374f8e700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93540682f0 0x7f935406a7a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9370155060 tx=0x7f9360008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:40.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:40.896+0000 7f9365ffb700 1 -- 172.21.15.7:0/87988317 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f936c053d90 con 0x7f935c003a70 2023-03-03T08:43:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.089+0000 7f93779f3700 1 -- 172.21.15.7:0/87988317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f93701484e0 con 0x7f935c003a70 2023-03-03T08:43:41.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.089+0000 7f9365ffb700 1 -- 172.21.15.7:0/87988317 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f936c0573b0 con 0x7f935c003a70 2023-03-03T08:43:41.091 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 8 2023-03-03T08:43:41.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 -- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93540682f0 msgr2=0x7f935406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93540682f0 0x7f935406a7a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9370155060 tx=0x7f9360008040 comp rx=0 tx=0).stop 2023-03-03T08:43:41.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 -- 172.21.15.7:0/87988317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 msgr2=0x7f93700c5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 0x7f93700c5590 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f936c0090b0 tx=0x7f936c0047c0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 -- 172.21.15.7:0/87988317 shutdown_connections 2023-03-03T08:43:41.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93701ea6e0 0x7f9370146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93540682f0 0x7f935406a7a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f935c003a70 0x7f93700c5590 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 --2- 172.21.15.7:0/87988317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9370148090 0x7f937014a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 -- 172.21.15.7:0/87988317 >> 172.21.15.7:0/87988317 conn(0x7f937006c110 msgr2=0x7f9370151670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:41.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 -- 172.21.15.7:0/87988317 shutdown_connections 2023-03-03T08:43:41.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.091+0000 7f93779f3700 1 -- 172.21.15.7:0/87988317 wait complete. 2023-03-03T08:43:41.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-03-03T08:43:41.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.232+0000 7f0bcc094700 1 Processor -- start 2023-03-03T08:43:41.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.232+0000 7f0bcc094700 1 -- start start 2023-03-03T08:43:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bcc094700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0bc4155390 0x7f0bc40c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bcc094700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 0x7f0bc4071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bcc094700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc4154910 con 0x7f0bc4072520 2023-03-03T08:43:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bcc094700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc4070b90 con 0x7f0bc4155390 2023-03-03T08:43:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bcc094700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc4151320 con 0x7f0bc406e020 2023-03-03T08:43:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc9e30700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0bc4155390 0x7f0bc40c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bca631700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0bc4072520 0x7f0bc4151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45274/0 (socket says 172.21.15.7:45274) 2023-03-03T08:43:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc9e30700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0bc4155390 0x7f0bc40c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51290/0 (socket says 172.21.15.7:51290) 2023-03-03T08:43:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bca631700 1 -- 172.21.15.7:0/51485197 learned_addr learned my addr 172.21.15.7:0/51485197 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc962f700 1 --2- 172.21.15.7:0/51485197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 0x7f0bc4071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc9e30700 1 -- 172.21.15.7:0/51485197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 msgr2=0x7f0bc4071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc9e30700 1 --2- 172.21.15.7:0/51485197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 0x7f0bc4071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc9e30700 1 -- 172.21.15.7:0/51485197 >> v1:172.21.15.7:6789/0 conn(0x7f0bc4072520 legacy=0x7f0bc4151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc9e30700 1 -- 172.21.15.7:0/51485197 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc40702a0 con 0x7f0bc4155390 2023-03-03T08:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc962f700 1 --2- 172.21.15.7:0/51485197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 0x7f0bc4071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc9e30700 1 --2- 172.21.15.7:0/51485197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 0x7f0bc40c3e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009840 tx=0x7f0bc0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30efc4a156ee30ae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc8e2e700 1 -- 172.21.15.7:0/51485197 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0011830 con 0x7f0bc4155390 2023-03-03T08:43:41.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.233+0000 7f0bc8e2e700 1 -- 172.21.15.7:0/51485197 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc0011990 con 0x7f0bc4155390 2023-03-03T08:43:41.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.234+0000 7f0bc8e2e700 1 -- 172.21.15.7:0/51485197 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0011d20 con 0x7f0bc4155390 2023-03-03T08:43:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.234+0000 7f0bcc094700 1 -- 172.21.15.7:0/51485197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 msgr2=0x7f0bc40c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.234+0000 7f0bcc094700 1 --2- 172.21.15.7:0/51485197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 0x7f0bc40c3e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009840 tx=0x7f0bc0009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.234+0000 7f0bcc094700 1 -- 172.21.15.7:0/51485197 shutdown_connections 2023-03-03T08:43:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.234+0000 7f0bcc094700 1 --2- 172.21.15.7:0/51485197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 0x7f0bc40c3e30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.234+0000 7f0bcc094700 1 --2- 172.21.15.7:0/51485197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 0x7f0bc4071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.234+0000 7f0bcc094700 1 -- 172.21.15.7:0/51485197 >> 172.21.15.7:0/51485197 conn(0x7f0bc406c110 msgr2=0x7f0bc406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.234+0000 7f0bcc094700 1 -- 172.21.15.7:0/51485197 shutdown_connections 2023-03-03T08:43:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.234+0000 7f0bcc094700 1 -- 172.21.15.7:0/51485197 wait complete. 2023-03-03T08:43:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.235+0000 7f0bcc094700 1 Processor -- start 2023-03-03T08:43:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.235+0000 7f0bcc094700 1 -- start start 2023-03-03T08:43:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bcc094700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 0x7f0bc41eb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bcc094700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc4072520 0x7f0bc4153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bcc094700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 0x7f0bc41417f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bcc094700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40ac410 con 0x7f0bc4155390 2023-03-03T08:43:41.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bcc094700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc41510a0 con 0x7f0bc4072520 2023-03-03T08:43:41.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bcc094700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc4150ae0 con 0x7f0bc406e020 2023-03-03T08:43:41.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bca631700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 0x7f0bc41417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bca631700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 0x7f0bc41417f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51300/0 (socket says 172.21.15.7:51300) 2023-03-03T08:43:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bca631700 1 -- 172.21.15.7:0/2339539883 learned_addr learned my addr 172.21.15.7:0/2339539883 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bc962f700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc4072520 0x7f0bc4153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bc9e30700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 0x7f0bc41eb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bca631700 1 -- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 msgr2=0x7f0bc41eb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bca631700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 0x7f0bc41eb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bca631700 1 -- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc4072520 msgr2=0x7f0bc4153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bca631700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc4072520 0x7f0bc4153630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bca631700 1 -- 172.21.15.7:0/2339539883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc00094a0 con 0x7f0bc4155390 2023-03-03T08:43:41.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.236+0000 7f0bca631700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 0x7f0bc41417f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00cbc0 tx=0x7f0bbc00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:41.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.237+0000 7f0bba7fc700 1 -- 172.21.15.7:0/2339539883 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc00ab50 con 0x7f0bc4155390 2023-03-03T08:43:41.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.237+0000 7f0bba7fc700 1 -- 172.21.15.7:0/2339539883 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc00acb0 con 0x7f0bc4155390 2023-03-03T08:43:41.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.237+0000 7f0bcc094700 1 -- 172.21.15.7:0/2339539883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc4070820 con 0x7f0bc4155390 2023-03-03T08:43:41.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.237+0000 7f0bba7fc700 1 -- 172.21.15.7:0/2339539883 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc012600 con 0x7f0bc4155390 2023-03-03T08:43:41.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.237+0000 7f0bcc094700 1 -- 172.21.15.7:0/2339539883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc4070540 con 0x7f0bc4155390 2023-03-03T08:43:41.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.238+0000 7f0bcc094700 1 -- 172.21.15.7:0/2339539883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc4155760 con 0x7f0bc4155390 2023-03-03T08:43:41.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.238+0000 7f0bba7fc700 1 -- 172.21.15.7:0/2339539883 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0bbc019030 con 0x7f0bc4155390 2023-03-03T08:43:41.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.238+0000 7f0bba7fc700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ba8068320 0x7f0ba806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.238+0000 7f0bba7fc700 1 -- 172.21.15.7:0/2339539883 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f0bbc085750 con 0x7f0bc4155390 2023-03-03T08:43:41.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.239+0000 7f0bc9e30700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ba8068320 0x7f0ba806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.239+0000 7f0bc9e30700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ba8068320 0x7f0ba806a7d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00090b0 tx=0x7f0bc0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:41.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.243+0000 7f0bba7fc700 1 -- 172.21.15.7:0/2339539883 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0bbc04b5e0 con 0x7f0bc4155390 2023-03-03T08:43:41.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.434+0000 7f0bcc094700 1 -- 172.21.15.7:0/2339539883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0bc4074bb0 con 0x7f0bc4155390 2023-03-03T08:43:41.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.434+0000 7f0bba7fc700 1 -- 172.21.15.7:0/2339539883 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v23) v1 ==== 99+0+11 (secure 0 0 0) 0x7f0bbc052070 con 0x7f0bc4155390 2023-03-03T08:43:41.436 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:43:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 -- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ba8068320 msgr2=0x7f0ba806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ba8068320 0x7f0ba806a7d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00090b0 tx=0x7f0bc0008030 comp rx=0 tx=0).stop 2023-03-03T08:43:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 -- 172.21.15.7:0/2339539883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 msgr2=0x7f0bc41417f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 0x7f0bc41417f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00cbc0 tx=0x7f0bbc00cf80 comp rx=0 tx=0).stop 2023-03-03T08:43:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 -- 172.21.15.7:0/2339539883 shutdown_connections 2023-03-03T08:43:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ba8068320 0x7f0ba806a7d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bc4155390 0x7f0bc41417f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bc406e020 0x7f0bc41eb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 --2- 172.21.15.7:0/2339539883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc4072520 0x7f0bc4153630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 -- 172.21.15.7:0/2339539883 >> 172.21.15.7:0/2339539883 conn(0x7f0bc406c110 msgr2=0x7f0bc406ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 -- 172.21.15.7:0/2339539883 shutdown_connections 2023-03-03T08:43:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.436+0000 7f0bcc094700 1 -- 172.21.15.7:0/2339539883 wait complete. 2023-03-03T08:43:41.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-03-03T08:43:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.578+0000 7f8112d1d700 1 Processor -- start 2023-03-03T08:43:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.578+0000 7f8112d1d700 1 -- start start 2023-03-03T08:43:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f8112d1d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f810c0c65c0 0x7f810c14de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f8112d1d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f810c0a9af0 0x7f810c066840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f8112d1d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810c0a65b0 con 0x7f810c06a2b0 2023-03-03T08:43:41.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f8112d1d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810c154e00 con 0x7f810c0c65c0 2023-03-03T08:43:41.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f8112d1d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810c14e390 con 0x7f810c0a9af0 2023-03-03T08:43:41.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f81112ba700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f810c06a2b0 0x7f810c0c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45288/0 (socket says 172.21.15.7:45288) 2023-03-03T08:43:41.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f81112ba700 1 -- 172.21.15.7:0/2153480135 learned_addr learned my addr 172.21.15.7:0/2153480135 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:41.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810bfff700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f810c0a9af0 0x7f810c066840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f8110ab9700 1 --2- 172.21.15.7:0/2153480135 >> v2:172.21.15.143:3300/0 conn(0x7f810c0c65c0 0x7f810c14de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918643188 0 0) 0x7f810c0a65b0 con 0x7f810c06a2b0 2023-03-03T08:43:41.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc0035b0 con 0x7f810c06a2b0 2023-03-03T08:43:41.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869878127 0 0) 0x7f80fc0035b0 con 0x7f810c06a2b0 2023-03-03T08:43:41.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f810c0a9af0 msgr2=0x7f810c066840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:41.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f810c0a9af0 0x7f810c066840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 >> v2:172.21.15.143:3300/0 conn(0x7f810c0c65c0 msgr2=0x7f810c14de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 --2- 172.21.15.7:0/2153480135 >> v2:172.21.15.143:3300/0 conn(0x7f810c0c65c0 0x7f810c14de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c1541f0 con 0x7f810c06a2b0 2023-03-03T08:43:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f81040019f0 con 0x7f810c06a2b0 2023-03-03T08:43:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 >> v1:172.21.15.7:6789/0 conn(0x7f810c06a2b0 legacy=0x7f810c0c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc003a70 0x7f80fc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80fc004850 0x7f80fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80fc005200 0x7f80fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c154e00 con 0x7f80fc004850 2023-03-03T08:43:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c14e390 con 0x7f80fc003a70 2023-03-03T08:43:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc00bf50 con 0x7f80fc005200 2023-03-03T08:43:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f8110ab9700 1 --2- 172.21.15.7:0/2153480135 >> v2:172.21.15.143:3300/0 conn(0x7f810c0c65c0 0x7f810c14de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.579+0000 7f810bfff700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f810c0a9af0 0x7f810c066840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f810bfff700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80fc004850 0x7f80fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f81112ba700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80fc005200 0x7f80fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f8110ab9700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc003a70 0x7f80fc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f810bfff700 1 -- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80fc005200 msgr2=0x7f80fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f810bfff700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80fc005200 0x7f80fc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f810bfff700 1 -- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc003a70 msgr2=0x7f80fc003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f810bfff700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc003a70 0x7f80fc003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f810bfff700 1 -- 172.21.15.7:0/2153480135 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc0035b0 con 0x7f80fc004850 2023-03-03T08:43:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f810bfff700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80fc004850 0x7f80fc004cc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009830 tx=0x7f80fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d517c86ec4d38e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8004030 con 0x7f80fc004850 2023-03-03T08:43:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f8004190 con 0x7f80fc004850 2023-03-03T08:43:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.580+0000 7f810b7fe700 1 -- 172.21.15.7:0/2153480135 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8004540 con 0x7f80fc004850 2023-03-03T08:43:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.581+0000 7f8112d1d700 1 -- 172.21.15.7:0/2153480135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80fc004850 msgr2=0x7f80fc004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.581+0000 7f8112d1d700 1 --2- 172.21.15.7:0/2153480135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80fc004850 0x7f80fc004cc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009830 tx=0x7f80fc005880 comp rx=0 tx=0).stop 2023-03-03T08:43:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.581+0000 7f81112ba700 1 -- 172.21.15.7:0/2153480135 reap_dead start 2023-03-03T08:43:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.581+0000 7f8112d1d700 1 -- 172.21.15.7:0/2153480135 shutdown_connections 2023-03-03T08:43:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.581+0000 7f8112d1d700 1 -- 172.21.15.7:0/2153480135 >> 172.21.15.7:0/2153480135 conn(0x7f810c0a9430 msgr2=0x7f810c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.581+0000 7f8112d1d700 1 -- 172.21.15.7:0/2153480135 shutdown_connections 2023-03-03T08:43:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.581+0000 7f8112d1d700 1 -- 172.21.15.7:0/2153480135 wait complete. 2023-03-03T08:43:41.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.582+0000 7f8112d1d700 1 Processor -- start 2023-03-03T08:43:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.582+0000 7f8112d1d700 1 -- start start 2023-03-03T08:43:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.583+0000 7f8112d1d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc004850 0x7f810c14fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.583+0000 7f8112d1d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f810c14e7d0 0x7f810c1e9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.583+0000 7f8112d1d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f810c06d010 0x7f810c1e5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.583+0000 7f8112d1d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c0c17a0 con 0x7f810c14e7d0 2023-03-03T08:43:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.583+0000 7f8112d1d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c072020 con 0x7f80fc004850 2023-03-03T08:43:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.583+0000 7f8112d1d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810c1e59b0 con 0x7f810c06d010 2023-03-03T08:43:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f810bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f810c14e7d0 0x7f810c1e9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f810bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f810c14e7d0 0x7f810c1e9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51318/0 (socket says 172.21.15.7:51318) 2023-03-03T08:43:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f810bfff700 1 -- 172.21.15.7:0/1042593902 learned_addr learned my addr 172.21.15.7:0/1042593902 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f8110ab9700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc004850 0x7f810c14fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f81112ba700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f810c06d010 0x7f810c1e5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f810bfff700 1 -- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f810c06d010 msgr2=0x7f810c1e5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f810bfff700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f810c06d010 0x7f810c1e5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f810bfff700 1 -- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc004850 msgr2=0x7f810c14fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f810bfff700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc004850 0x7f810c14fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f810bfff700 1 -- 172.21.15.7:0/1042593902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c1547b0 con 0x7f810c14e7d0 2023-03-03T08:43:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f8110ab9700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc004850 0x7f810c14fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f810bfff700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f810c14e7d0 0x7f810c1e9490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f80f8003b20 tx=0x7f80f80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f81097fa700 1 -- 172.21.15.7:0/1042593902 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f800e6c0 con 0x7f810c14e7d0 2023-03-03T08:43:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f81097fa700 1 -- 172.21.15.7:0/1042593902 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f800e820 con 0x7f810c14e7d0 2023-03-03T08:43:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f81097fa700 1 -- 172.21.15.7:0/1042593902 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f800eb90 con 0x7f810c14e7d0 2023-03-03T08:43:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f8112d1d700 1 -- 172.21.15.7:0/1042593902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810c1544d0 con 0x7f810c14e7d0 2023-03-03T08:43:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.584+0000 7f8112d1d700 1 -- 172.21.15.7:0/1042593902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810c154a90 con 0x7f810c14e7d0 2023-03-03T08:43:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.585+0000 7f8112d1d700 1 -- 172.21.15.7:0/1042593902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810c074bb0 con 0x7f810c14e7d0 2023-03-03T08:43:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.586+0000 7f81097fa700 1 -- 172.21.15.7:0/1042593902 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f80f8034070 con 0x7f810c14e7d0 2023-03-03T08:43:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.586+0000 7f81097fa700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80f00682f0 0x7f80f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.586+0000 7f81097fa700 1 -- 172.21.15.7:0/1042593902 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f80f8089700 con 0x7f810c14e7d0 2023-03-03T08:43:41.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.586+0000 7f8110ab9700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80f00682f0 0x7f80f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.587+0000 7f8110ab9700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80f00682f0 0x7f80f006a7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8100009df0 tx=0x7f8100008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:41.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.591+0000 7f81097fa700 1 -- 172.21.15.7:0/1042593902 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80f8055fb0 con 0x7f810c14e7d0 2023-03-03T08:43:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.782+0000 7f8112d1d700 1 -- 172.21.15.7:0/1042593902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f810c1e99d0 con 0x7f810c14e7d0 2023-03-03T08:43:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.782+0000 7f81097fa700 1 -- 172.21.15.7:0/1042593902 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v23) v1 ==== 95+0+11 (secure 0 0 0) 0x7f80f8028020 con 0x7f810c14e7d0 2023-03-03T08:43:41.784 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:43:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.784+0000 7f8112d1d700 1 -- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80f00682f0 msgr2=0x7f80f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.784+0000 7f8112d1d700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80f00682f0 0x7f80f006a7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8100009df0 tx=0x7f8100008040 comp rx=0 tx=0).stop 2023-03-03T08:43:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.784+0000 7f8112d1d700 1 -- 172.21.15.7:0/1042593902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f810c14e7d0 msgr2=0x7f810c1e9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.784+0000 7f8112d1d700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f810c14e7d0 0x7f810c1e9490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f80f8003b20 tx=0x7f80f80047c0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.785+0000 7f8112d1d700 1 -- 172.21.15.7:0/1042593902 shutdown_connections 2023-03-03T08:43:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.785+0000 7f8112d1d700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f810c06d010 0x7f810c1e5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.785+0000 7f8112d1d700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80fc004850 0x7f810c14fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.785+0000 7f8112d1d700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80f00682f0 0x7f80f006a7a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.785+0000 7f8112d1d700 1 --2- 172.21.15.7:0/1042593902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f810c14e7d0 0x7f810c1e9490 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.785+0000 7f8112d1d700 1 -- 172.21.15.7:0/1042593902 >> 172.21.15.7:0/1042593902 conn(0x7f810c0a9430 msgr2=0x7f810c1533b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.785+0000 7f8112d1d700 1 -- 172.21.15.7:0/1042593902 shutdown_connections 2023-03-03T08:43:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.785+0000 7f8112d1d700 1 -- 172.21.15.7:0/1042593902 wait complete. 2023-03-03T08:43:41.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-03T08:43:41.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.923+0000 7f8c872a5700 1 Processor -- start 2023-03-03T08:43:41.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.923+0000 7f8c872a5700 1 -- start start 2023-03-03T08:43:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.923+0000 7f8c872a5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8c800c1da0 0x7f8c801550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c872a5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c800c7a60 con 0x7f8c800c3e30 2023-03-03T08:43:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c872a5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c800c3620 con 0x7f8c8006d870 2023-03-03T08:43:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c872a5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c800c2af0 con 0x7f8c800c1da0 2023-03-03T08:43:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c84840700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8c800c1da0 0x7f8c801550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c84840700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8c800c1da0 0x7f8c801550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51326/0 (socket says 172.21.15.7:51326) 2023-03-03T08:43:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c85842700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8c800c3e30 0x7f8c8015ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45308/0 (socket says 172.21.15.7:45308) 2023-03-03T08:43:41.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c84840700 1 -- 172.21.15.7:0/3126738270 learned_addr learned my addr 172.21.15.7:0/3126738270 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:41.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c84840700 1 -- 172.21.15.7:0/3126738270 >> v1:172.21.15.143:6789/0 conn(0x7f8c8006d870 legacy=0x7f8c800c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c84840700 1 -- 172.21.15.7:0/3126738270 >> v1:172.21.15.7:6789/0 conn(0x7f8c800c3e30 legacy=0x7f8c8015ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c73fff700 1 -- 172.21.15.7:0/3126738270 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531686197 0 0) 0x7f8c800c7a60 con 0x7f8c800c3e30 2023-03-03T08:43:41.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c84840700 1 -- 172.21.15.7:0/3126738270 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c801545e0 con 0x7f8c800c1da0 2023-03-03T08:43:41.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c84840700 1 --2- 172.21.15.7:0/3126738270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c1da0 0x7f8c801550c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8c740098f0 tx=0x7f8c74009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770877cebdc3512e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:41.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c73fff700 1 -- 172.21.15.7:0/3126738270 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7400b8a0 con 0x7f8c800c1da0 2023-03-03T08:43:41.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c73fff700 1 -- 172.21.15.7:0/3126738270 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7400ba00 con 0x7f8c800c1da0 2023-03-03T08:43:41.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.924+0000 7f8c73fff700 1 -- 172.21.15.7:0/3126738270 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7400bd90 con 0x7f8c800c1da0 2023-03-03T08:43:41.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.925+0000 7f8c872a5700 1 -- 172.21.15.7:0/3126738270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c1da0 msgr2=0x7f8c801550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.925+0000 7f8c872a5700 1 --2- 172.21.15.7:0/3126738270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c1da0 0x7f8c801550c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8c740098f0 tx=0x7f8c74009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:41.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.925+0000 7f8c872a5700 1 -- 172.21.15.7:0/3126738270 shutdown_connections 2023-03-03T08:43:41.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.925+0000 7f8c872a5700 1 --2- 172.21.15.7:0/3126738270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c1da0 0x7f8c801550c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.925+0000 7f8c872a5700 1 -- 172.21.15.7:0/3126738270 >> 172.21.15.7:0/3126738270 conn(0x7f8c8006c110 msgr2=0x7f8c8006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:41.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.925+0000 7f8c872a5700 1 -- 172.21.15.7:0/3126738270 shutdown_connections 2023-03-03T08:43:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.925+0000 7f8c872a5700 1 -- 172.21.15.7:0/3126738270 wait complete. 2023-03-03T08:43:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.926+0000 7f8c872a5700 1 Processor -- start 2023-03-03T08:43:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.926+0000 7f8c872a5700 1 -- start start 2023-03-03T08:43:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c872a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c8006d870 0x7f8c80071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c872a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c800c1da0 0x7f8c8014ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c872a5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c3e30 0x7f8c8014f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c872a5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c800abfe0 con 0x7f8c800c3e30 2023-03-03T08:43:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c872a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8014ab70 con 0x7f8c8006d870 2023-03-03T08:43:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c872a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c800ac400 con 0x7f8c800c1da0 2023-03-03T08:43:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c85842700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c3e30 0x7f8c8014f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c85842700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c3e30 0x7f8c8014f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51334/0 (socket says 172.21.15.7:51334) 2023-03-03T08:43:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c85842700 1 -- 172.21.15.7:0/4081227203 learned_addr learned my addr 172.21.15.7:0/4081227203 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c85041700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c8006d870 0x7f8c80071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c84840700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c800c1da0 0x7f8c8014ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c85842700 1 -- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c800c1da0 msgr2=0x7f8c8014ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c85842700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c800c1da0 0x7f8c8014ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c85842700 1 -- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c8006d870 msgr2=0x7f8c80071660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c85842700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c8006d870 0x7f8c80071660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.927+0000 7f8c85842700 1 -- 172.21.15.7:0/4081227203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74009550 con 0x7f8c800c3e30 2023-03-03T08:43:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.928+0000 7f8c85842700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c3e30 0x7f8c8014f490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8c7800cca0 tx=0x7f8c7800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.928+0000 7f8c71ffb700 1 -- 172.21.15.7:0/4081227203 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7800ab70 con 0x7f8c800c3e30 2023-03-03T08:43:41.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.928+0000 7f8c71ffb700 1 -- 172.21.15.7:0/4081227203 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7800acd0 con 0x7f8c800c3e30 2023-03-03T08:43:41.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.928+0000 7f8c872a5700 1 -- 172.21.15.7:0/4081227203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c80154ba0 con 0x7f8c800c3e30 2023-03-03T08:43:41.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.928+0000 7f8c71ffb700 1 -- 172.21.15.7:0/4081227203 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78007310 con 0x7f8c800c3e30 2023-03-03T08:43:41.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.928+0000 7f8c872a5700 1 -- 172.21.15.7:0/4081227203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c801548c0 con 0x7f8c800c3e30 2023-03-03T08:43:41.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.929+0000 7f8c872a5700 1 -- 172.21.15.7:0/4081227203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8006dc40 con 0x7f8c800c3e30 2023-03-03T08:43:41.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.929+0000 7f8c71ffb700 1 -- 172.21.15.7:0/4081227203 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c78010070 con 0x7f8c800c3e30 2023-03-03T08:43:41.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.930+0000 7f8c71ffb700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c640682f0 0x7f8c6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:41.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.930+0000 7f8c71ffb700 1 -- 172.21.15.7:0/4081227203 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f8c780856e0 con 0x7f8c800c3e30 2023-03-03T08:43:41.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.930+0000 7f8c85041700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c640682f0 0x7f8c6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:41.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.930+0000 7f8c85041700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c640682f0 0x7f8c6406a7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8c800c4870 tx=0x7f8c7c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:41.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:41.934+0000 7f8c71ffb700 1 -- 172.21.15.7:0/4081227203 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c78051ef0 con 0x7f8c800c3e30 2023-03-03T08:43:42.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.131+0000 7f8c872a5700 1 -- 172.21.15.7:0/4081227203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8c800ac120 con 0x7f8c800c3e30 2023-03-03T08:43:42.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.131+0000 7f8c71ffb700 1 -- 172.21.15.7:0/4081227203 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v23) v1 ==== 84+0+258 (secure 0 0 0) 0x7f8c800ac120 con 0x7f8c800c3e30 2023-03-03T08:43:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:42.133 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2023-03-03T08:43:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-03-03T08:43:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 1, 2023-03-03T08:43:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": ".mgr" 2023-03-03T08:43:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-03-03T08:43:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-03-03T08:43:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 2, 2023-03-03T08:43:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": "rbd" 2023-03-03T08:43:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-03-03T08:43:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-03-03T08:43:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 3, 2023-03-03T08:43:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": "a_metadata" 2023-03-03T08:43:42.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-03-03T08:43:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-03-03T08:43:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 4, 2023-03-03T08:43:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": "a_data" 2023-03-03T08:43:42.135 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-03-03T08:43:42.135 INFO:teuthology.orchestra.run.smithi007.stdout:] 2023-03-03T08:43:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 -- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c640682f0 msgr2=0x7f8c6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c640682f0 0x7f8c6406a7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8c800c4870 tx=0x7f8c7c009040 comp rx=0 tx=0).stop 2023-03-03T08:43:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 -- 172.21.15.7:0/4081227203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c3e30 msgr2=0x7f8c8014f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c3e30 0x7f8c8014f490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8c7800cca0 tx=0x7f8c7800a060 comp rx=0 tx=0).stop 2023-03-03T08:43:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 -- 172.21.15.7:0/4081227203 shutdown_connections 2023-03-03T08:43:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c640682f0 0x7f8c6406a7a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c3e30 0x7f8c8014f490 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c8006d870 0x7f8c80071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 --2- 172.21.15.7:0/4081227203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c800c1da0 0x7f8c8014ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 -- 172.21.15.7:0/4081227203 >> 172.21.15.7:0/4081227203 conn(0x7f8c8006c110 msgr2=0x7f8c8006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 -- 172.21.15.7:0/4081227203 shutdown_connections 2023-03-03T08:43:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.133+0000 7f8c872a5700 1 -- 172.21.15.7:0/4081227203 wait complete. 2023-03-03T08:43:42.146 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'b' 2023-03-03T08:43:42.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_metadata --pg_num_min 64 2023-03-03T08:43:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.273+0000 7fe441aa4700 1 Processor -- start 2023-03-03T08:43:42.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.273+0000 7fe441aa4700 1 -- start start 2023-03-03T08:43:42.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.274+0000 7fe441aa4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 0x7fe43c0c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.274+0000 7fe441aa4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c0713c0 con 0x7fe43c155950 2023-03-03T08:43:42.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.274+0000 7fe441aa4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c152c70 con 0x7fe43c0aac00 2023-03-03T08:43:42.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.274+0000 7fe441aa4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c0c4930 con 0x7fe43c071c30 2023-03-03T08:43:42.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.274+0000 7fe43bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe43c155950 0x7fe43c15cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45324/0 (socket says 172.21.15.7:45324) 2023-03-03T08:43:42.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.274+0000 7fe43bfff700 1 -- 172.21.15.7:0/3019748521 learned_addr learned my addr 172.21.15.7:0/3019748521 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:42.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.274+0000 7fe43affd700 1 --2- 172.21.15.7:0/3019748521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 0x7fe43c0c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43a7fc700 1 -- 172.21.15.7:0/3019748521 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386267451 0 0) 0x7fe43c0713c0 con 0x7fe43c155950 2023-03-03T08:43:42.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43a7fc700 1 -- 172.21.15.7:0/3019748521 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4280035b0 con 0x7fe43c155950 2023-03-03T08:43:42.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43a7fc700 1 -- 172.21.15.7:0/3019748521 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217027450 0 0) 0x7fe43c152c70 con 0x7fe43c0aac00 2023-03-03T08:43:42.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43a7fc700 1 -- 172.21.15.7:0/3019748521 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe43c0713c0 con 0x7fe43c0aac00 2023-03-03T08:43:42.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43affd700 1 -- 172.21.15.7:0/3019748521 >> v1:172.21.15.143:6789/0 conn(0x7fe43c0aac00 legacy=0x7fe43c0beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43affd700 1 -- 172.21.15.7:0/3019748521 >> v1:172.21.15.7:6789/0 conn(0x7fe43c155950 legacy=0x7fe43c15cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43affd700 1 -- 172.21.15.7:0/3019748521 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43c152360 con 0x7fe43c071c30 2023-03-03T08:43:42.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43affd700 1 --2- 172.21.15.7:0/3019748521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 0x7fe43c0c43f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe424009870 tx=0x7fe424009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=894c299f2952bd56 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:42.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43a7fc700 1 -- 172.21.15.7:0/3019748521 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe424011830 con 0x7fe43c071c30 2023-03-03T08:43:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43a7fc700 1 -- 172.21.15.7:0/3019748521 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe424011990 con 0x7fe43c071c30 2023-03-03T08:43:42.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.275+0000 7fe43a7fc700 1 -- 172.21.15.7:0/3019748521 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe424011d00 con 0x7fe43c071c30 2023-03-03T08:43:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.276+0000 7fe441aa4700 1 -- 172.21.15.7:0/3019748521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 msgr2=0x7fe43c0c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.276+0000 7fe441aa4700 1 --2- 172.21.15.7:0/3019748521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 0x7fe43c0c43f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe424009870 tx=0x7fe424009b80 comp rx=0 tx=0).stop 2023-03-03T08:43:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.276+0000 7fe441aa4700 1 -- 172.21.15.7:0/3019748521 shutdown_connections 2023-03-03T08:43:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.276+0000 7fe441aa4700 1 --2- 172.21.15.7:0/3019748521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 0x7fe43c0c43f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.276+0000 7fe441aa4700 1 -- 172.21.15.7:0/3019748521 >> 172.21.15.7:0/3019748521 conn(0x7fe43c0736f0 msgr2=0x7fe43c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:42.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.276+0000 7fe441aa4700 1 -- 172.21.15.7:0/3019748521 shutdown_connections 2023-03-03T08:43:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.276+0000 7fe441aa4700 1 -- 172.21.15.7:0/3019748521 wait complete. 2023-03-03T08:43:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.277+0000 7fe441aa4700 1 Processor -- start 2023-03-03T08:43:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.277+0000 7fe441aa4700 1 -- start start 2023-03-03T08:43:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe441aa4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 0x7fe43c14eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe441aa4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe43c0aac00 0x7fe43c14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe441aa4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe43c155950 0x7fe43c1dcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe441aa4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c141f80 con 0x7fe43c0aac00 2023-03-03T08:43:42.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe441aa4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c1dc960 con 0x7fe43c155950 2023-03-03T08:43:42.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe441aa4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43c14e8d0 con 0x7fe43c071c30 2023-03-03T08:43:42.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe43c0aac00 0x7fe43c14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe43c0aac00 0x7fe43c14f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51348/0 (socket says 172.21.15.7:51348) 2023-03-03T08:43:42.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43affd700 1 -- 172.21.15.7:0/3440541192 learned_addr learned my addr 172.21.15.7:0/3440541192 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:42.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43b7fe700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 0x7fe43c14eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43bfff700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe43c155950 0x7fe43c1dcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43affd700 1 -- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 msgr2=0x7fe43c14eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43affd700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 0x7fe43c14eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43affd700 1 -- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe43c155950 msgr2=0x7fe43c1dcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43affd700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe43c155950 0x7fe43c1dcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43affd700 1 -- 172.21.15.7:0/3440541192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4240094a0 con 0x7fe43c0aac00 2023-03-03T08:43:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43bfff700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe43c155950 0x7fe43c1dcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe43affd700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe43c0aac00 0x7fe43c14f530 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe43c074310 tx=0x7fe424004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe433fff700 1 -- 172.21.15.7:0/3440541192 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4240045d0 con 0x7fe43c0aac00 2023-03-03T08:43:42.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe433fff700 1 -- 172.21.15.7:0/3440541192 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42401f070 con 0x7fe43c0aac00 2023-03-03T08:43:42.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe433fff700 1 -- 172.21.15.7:0/3440541192 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4240038c0 con 0x7fe43c0aac00 2023-03-03T08:43:42.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.278+0000 7fe441aa4700 1 -- 172.21.15.7:0/3440541192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43c152620 con 0x7fe43c0aac00 2023-03-03T08:43:42.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.279+0000 7fe441aa4700 1 -- 172.21.15.7:0/3440541192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43c0c4b80 con 0x7fe43c0aac00 2023-03-03T08:43:42.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.279+0000 7fe441aa4700 1 -- 172.21.15.7:0/3440541192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43c152900 con 0x7fe43c0aac00 2023-03-03T08:43:42.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.280+0000 7fe433fff700 1 -- 172.21.15.7:0/3440541192 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe42400f050 con 0x7fe43c0aac00 2023-03-03T08:43:42.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.280+0000 7fe433fff700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe41c0682f0 0x7fe41c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.280+0000 7fe433fff700 1 -- 172.21.15.7:0/3440541192 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7fe424021070 con 0x7fe43c0aac00 2023-03-03T08:43:42.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.280+0000 7fe43b7fe700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe41c0682f0 0x7fe41c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.281+0000 7fe43b7fe700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe41c0682f0 0x7fe41c06a7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0011e0 tx=0x7fe42c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:42.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.284+0000 7fe433fff700 1 -- 172.21.15.7:0/3440541192 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe424053cd0 con 0x7fe43c0aac00 2023-03-03T08:43:42.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.478+0000 7fe441aa4700 1 -- 172.21.15.7:0/3440541192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe43c1dc730 con 0x7fe43c0aac00 2023-03-03T08:43:42.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.550+0000 7fe433fff700 1 -- 172.21.15.7:0/3440541192 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64}]=0 pool 'b_metadata' created v24) v1 ==== 128+0+0 (secure 0 0 0) 0x7fe43c1dc730 con 0x7fe43c0aac00 2023-03-03T08:43:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.557+0000 7fe441aa4700 1 -- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe41c0682f0 msgr2=0x7fe41c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.557+0000 7fe441aa4700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe41c0682f0 0x7fe41c06a7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe42c0011e0 tx=0x7fe42c009040 comp rx=0 tx=0).stop 2023-03-03T08:43:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.557+0000 7fe441aa4700 1 -- 172.21.15.7:0/3440541192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe43c0aac00 msgr2=0x7fe43c14f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.557+0000 7fe441aa4700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe43c0aac00 0x7fe43c14f530 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe43c074310 tx=0x7fe424004750 comp rx=0 tx=0).stop 2023-03-03T08:43:42.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.558+0000 7fe441aa4700 1 -- 172.21.15.7:0/3440541192 shutdown_connections 2023-03-03T08:43:42.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.558+0000 7fe441aa4700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe43c155950 0x7fe43c1dcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.558+0000 7fe441aa4700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe43c071c30 0x7fe43c14eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.558+0000 7fe441aa4700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe41c0682f0 0x7fe41c06a7a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.558+0000 7fe441aa4700 1 --2- 172.21.15.7:0/3440541192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe43c0aac00 0x7fe43c14f530 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.558+0000 7fe441aa4700 1 -- 172.21.15.7:0/3440541192 >> 172.21.15.7:0/3440541192 conn(0x7fe43c0736f0 msgr2=0x7fe43c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:42.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.558+0000 7fe441aa4700 1 -- 172.21.15.7:0/3440541192 shutdown_connections 2023-03-03T08:43:42.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.558+0000 7fe441aa4700 1 -- 172.21.15.7:0/3440541192 wait complete. 2023-03-03T08:43:42.561 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'b_metadata' created 2023-03-03T08:43:42.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-03-03T08:43:42.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.702+0000 7f95404e0700 1 Processor -- start 2023-03-03T08:43:42.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.702+0000 7f95404e0700 1 -- start start 2023-03-03T08:43:42.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.702+0000 7f95404e0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f95380a6140 0x7f95380c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.702+0000 7f95404e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95380c4fc0 0x7f953806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f95404e0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95380aa890 con 0x7f95380c5410 2023-03-03T08:43:42.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f95404e0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95380c17f0 con 0x7f95380a6140 2023-03-03T08:43:42.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f95404e0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95380c1ad0 con 0x7f95380c4fc0 2023-03-03T08:43:42.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953e27c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f95380a6140 0x7f95380c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953e27c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f95380a6140 0x7f95380c4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51354/0 (socket says 172.21.15.7:51354) 2023-03-03T08:43:42.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953e27c700 1 -- 172.21.15.7:0/863721121 learned_addr learned my addr 172.21.15.7:0/863721121 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:42.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953da7b700 1 --2- 172.21.15.7:0/863721121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95380c4fc0 0x7f953806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953e27c700 1 -- 172.21.15.7:0/863721121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95380c4fc0 msgr2=0x7f953806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:42.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953e27c700 1 --2- 172.21.15.7:0/863721121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95380c4fc0 0x7f953806d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953e27c700 1 -- 172.21.15.7:0/863721121 >> v1:172.21.15.143:6789/0 conn(0x7f95380c5410 legacy=0x7f953806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:42.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953e27c700 1 -- 172.21.15.7:0/863721121 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95381552c0 con 0x7f95380a6140 2023-03-03T08:43:42.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953e27c700 1 --2- 172.21.15.7:0/863721121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380a6140 0x7f95380c4a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f95340098f0 tx=0x7f9534009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fe3bd700200d26 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:42.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953d27a700 1 -- 172.21.15.7:0/863721121 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9534004230 con 0x7f95380a6140 2023-03-03T08:43:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953d27a700 1 -- 172.21.15.7:0/863721121 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9534004390 con 0x7f95380a6140 2023-03-03T08:43:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.703+0000 7f953d27a700 1 -- 172.21.15.7:0/863721121 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95340036a0 con 0x7f95380a6140 2023-03-03T08:43:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.704+0000 7f95404e0700 1 -- 172.21.15.7:0/863721121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380a6140 msgr2=0x7f95380c4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.704+0000 7f95404e0700 1 --2- 172.21.15.7:0/863721121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380a6140 0x7f95380c4a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f95340098f0 tx=0x7f9534009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:42.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.704+0000 7f95404e0700 1 -- 172.21.15.7:0/863721121 shutdown_connections 2023-03-03T08:43:42.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.704+0000 7f95404e0700 1 --2- 172.21.15.7:0/863721121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380a6140 0x7f95380c4a80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.704+0000 7f95404e0700 1 --2- 172.21.15.7:0/863721121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95380c4fc0 0x7f953806d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.704+0000 7f95404e0700 1 -- 172.21.15.7:0/863721121 >> 172.21.15.7:0/863721121 conn(0x7f95380a9430 msgr2=0x7f95380c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:42.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.704+0000 7f95404e0700 1 -- 172.21.15.7:0/863721121 shutdown_connections 2023-03-03T08:43:42.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.704+0000 7f95404e0700 1 -- 172.21.15.7:0/863721121 wait complete. 2023-03-03T08:43:42.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.705+0000 7f95404e0700 1 Processor -- start 2023-03-03T08:43:42.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.705+0000 7f95404e0700 1 -- start start 2023-03-03T08:43:42.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f95404e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95380a6140 0x7f953814f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f95404e0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380c4fc0 0x7f95381509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f95404e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95380c5410 0x7f95381ea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f95404e0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95381e9c60 con 0x7f95380c4fc0 2023-03-03T08:43:42.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f95404e0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95381e9f40 con 0x7f95380a6140 2023-03-03T08:43:42.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f95404e0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9538071ba0 con 0x7f95380c5410 2023-03-03T08:43:42.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953da7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380c4fc0 0x7f95381509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953da7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380c4fc0 0x7f95381509f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51362/0 (socket says 172.21.15.7:51362) 2023-03-03T08:43:42.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953da7b700 1 -- 172.21.15.7:0/1354252643 learned_addr learned my addr 172.21.15.7:0/1354252643 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:42.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953e27c700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95380a6140 0x7f953814f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953ea7d700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95380c5410 0x7f95381ea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953da7b700 1 -- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95380c5410 msgr2=0x7f95381ea3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953da7b700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95380c5410 0x7f95381ea3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953da7b700 1 -- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95380a6140 msgr2=0x7f953814f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953da7b700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95380a6140 0x7f953814f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953da7b700 1 -- 172.21.15.7:0/1354252643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9534009550 con 0x7f95380c4fc0 2023-03-03T08:43:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.706+0000 7f953da7b700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380c4fc0 0x7f95381509f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9528010730 tx=0x7f9528010a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.707+0000 7f952effd700 1 -- 172.21.15.7:0/1354252643 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95280118c0 con 0x7f95380c4fc0 2023-03-03T08:43:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.707+0000 7f952effd700 1 -- 172.21.15.7:0/1354252643 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9528011a20 con 0x7f95380c4fc0 2023-03-03T08:43:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.707+0000 7f952effd700 1 -- 172.21.15.7:0/1354252643 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528011dd0 con 0x7f95380c4fc0 2023-03-03T08:43:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.707+0000 7f95404e0700 1 -- 172.21.15.7:0/1354252643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95380aa520 con 0x7f95380c4fc0 2023-03-03T08:43:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.707+0000 7f95404e0700 1 -- 172.21.15.7:0/1354252643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95380a6510 con 0x7f95380c4fc0 2023-03-03T08:43:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.708+0000 7f95404e0700 1 -- 172.21.15.7:0/1354252643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f953806e2f0 con 0x7f95380c4fc0 2023-03-03T08:43:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.708+0000 7f952effd700 1 -- 172.21.15.7:0/1354252643 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9528019430 con 0x7f95380c4fc0 2023-03-03T08:43:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.708+0000 7f952effd700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f951c0682f0 0x7f951c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.709+0000 7f952effd700 1 -- 172.21.15.7:0/1354252643 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6407+0+0 (secure 0 0 0) 0x7f9528087ec0 con 0x7f95380c4fc0 2023-03-03T08:43:42.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.709+0000 7f953e27c700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f951c0682f0 0x7f951c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:42.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.709+0000 7f953e27c700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f951c0682f0 0x7f951c06a7a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9534009130 tx=0x7f9534008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:42.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.713+0000 7f952effd700 1 -- 172.21.15.7:0/1354252643 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f95280515e0 con 0x7f95380c4fc0 2023-03-03T08:43:42.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:42.906+0000 7f95404e0700 1 -- 172.21.15.7:0/1354252643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f95381ea080 con 0x7f95380c4fc0 2023-03-03T08:43:43.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.556+0000 7f952effd700 1 -- 172.21.15.7:0/1354252643 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'b_data' created v25) v1 ==== 160+0+0 (secure 0 0 0) 0x7f95381ea080 con 0x7f95380c4fc0 2023-03-03T08:43:43.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.563+0000 7f95404e0700 1 -- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f951c0682f0 msgr2=0x7f951c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:43.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.563+0000 7f95404e0700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f951c0682f0 0x7f951c06a7a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9534009130 tx=0x7f9534008030 comp rx=0 tx=0).stop 2023-03-03T08:43:43.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.563+0000 7f95404e0700 1 -- 172.21.15.7:0/1354252643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380c4fc0 msgr2=0x7f95381509f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:43.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.563+0000 7f95404e0700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380c4fc0 0x7f95381509f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9528010730 tx=0x7f9528010a40 comp rx=0 tx=0).stop 2023-03-03T08:43:43.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.564+0000 7f95404e0700 1 -- 172.21.15.7:0/1354252643 shutdown_connections 2023-03-03T08:43:43.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.564+0000 7f95404e0700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95380c5410 0x7f95381ea3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.564+0000 7f95404e0700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95380a6140 0x7f953814f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.564+0000 7f95404e0700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f951c0682f0 0x7f951c06a7a0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9534009130 tx=0x7f9534008030 comp rx=0 tx=0).stop 2023-03-03T08:43:43.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.564+0000 7f95404e0700 1 --2- 172.21.15.7:0/1354252643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95380c4fc0 0x7f95381509f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.564+0000 7f95404e0700 1 -- 172.21.15.7:0/1354252643 >> 172.21.15.7:0/1354252643 conn(0x7f95380a9430 msgr2=0x7f95380c6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:43.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.565+0000 7f95404e0700 1 -- 172.21.15.7:0/1354252643 shutdown_connections 2023-03-03T08:43:43.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.566+0000 7f95404e0700 1 -- 172.21.15.7:0/1354252643 wait complete. 2023-03-03T08:43:43.580 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'b_data' created 2023-03-03T08:43:43.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new b b_metadata b_data 2023-03-03T08:43:43.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.720+0000 7f4cad976700 1 Processor -- start 2023-03-03T08:43:43.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.720+0000 7f4cad976700 1 -- start start 2023-03-03T08:43:43.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.721+0000 7f4cad976700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ca80c1da0 0x7f4ca81550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:43.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.721+0000 7f4cad976700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ca80c3e30 0x7f4ca80c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:43.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.721+0000 7f4cad976700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca80c7a60 con 0x7f4ca806d870 2023-03-03T08:43:43.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.721+0000 7f4cad976700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80c2af0 con 0x7f4ca80c1da0 2023-03-03T08:43:43.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.721+0000 7f4cad976700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80c3620 con 0x7f4ca80c3e30 2023-03-03T08:43:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.721+0000 7f4ca67fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ca80c1da0 0x7f4ca81550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.721+0000 7f4ca67fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ca80c1da0 0x7f4ca81550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51374/0 (socket says 172.21.15.7:51374) 2023-03-03T08:43:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.721+0000 7f4ca67fc700 1 -- 172.21.15.7:0/2747260485 learned_addr learned my addr 172.21.15.7:0/2747260485 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.721+0000 7f4ca77fe700 1 --2- 172.21.15.7:0/2747260485 >> v2:172.21.15.143:3300/0 conn(0x7f4ca80c3e30 0x7f4ca80c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4ca67fc700 1 -- 172.21.15.7:0/2747260485 >> v2:172.21.15.143:3300/0 conn(0x7f4ca80c3e30 msgr2=0x7f4ca80c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4ca67fc700 1 --2- 172.21.15.7:0/2747260485 >> v2:172.21.15.143:3300/0 conn(0x7f4ca80c3e30 0x7f4ca80c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4ca67fc700 1 -- 172.21.15.7:0/2747260485 >> v1:172.21.15.143:6789/0 conn(0x7f4ca806d870 legacy=0x7f4ca80c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4ca67fc700 1 -- 172.21.15.7:0/2747260485 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca81545e0 con 0x7f4ca80c1da0 2023-03-03T08:43:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4ca77fe700 1 --2- 172.21.15.7:0/2747260485 >> v2:172.21.15.143:3300/0 conn(0x7f4ca80c3e30 0x7f4ca80c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4ca67fc700 1 --2- 172.21.15.7:0/2747260485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c1da0 0x7f4ca81550c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8073830 tx=0x7f4c98009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4821df20aef9717b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4ca5ffb700 1 -- 172.21.15.7:0/2747260485 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c98004230 con 0x7f4ca80c1da0 2023-03-03T08:43:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4ca5ffb700 1 -- 172.21.15.7:0/2747260485 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c98004390 con 0x7f4ca80c1da0 2023-03-03T08:43:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4ca5ffb700 1 -- 172.21.15.7:0/2747260485 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c980036a0 con 0x7f4ca80c1da0 2023-03-03T08:43:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4cad976700 1 -- 172.21.15.7:0/2747260485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c1da0 msgr2=0x7f4ca81550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4cad976700 1 --2- 172.21.15.7:0/2747260485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c1da0 0x7f4ca81550c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8073830 tx=0x7f4c98009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4cad976700 1 -- 172.21.15.7:0/2747260485 shutdown_connections 2023-03-03T08:43:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4cad976700 1 --2- 172.21.15.7:0/2747260485 >> v2:172.21.15.143:3300/0 conn(0x7f4ca80c3e30 0x7f4ca80c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4cad976700 1 --2- 172.21.15.7:0/2747260485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c1da0 0x7f4ca81550c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4cad976700 1 -- 172.21.15.7:0/2747260485 >> 172.21.15.7:0/2747260485 conn(0x7f4ca806c110 msgr2=0x7f4ca806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4cad976700 1 -- 172.21.15.7:0/2747260485 shutdown_connections 2023-03-03T08:43:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.722+0000 7f4cad976700 1 -- 172.21.15.7:0/2747260485 wait complete. 2023-03-03T08:43:43.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.724+0000 7f4cad976700 1 Processor -- start 2023-03-03T08:43:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.724+0000 7f4cad976700 1 -- start start 2023-03-03T08:43:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.724+0000 7f4cad976700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca806d870 0x7f4ca81ed670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.724+0000 7f4cad976700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ca80c1da0 0x7f4ca80c7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.724+0000 7f4cad976700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c3e30 0x7f4ca8145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.724+0000 7f4cad976700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80ad1a0 con 0x7f4ca80c3e30 2023-03-03T08:43:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.724+0000 7f4cad976700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca814e700 con 0x7f4ca806d870 2023-03-03T08:43:43.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.724+0000 7f4cad976700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca814def0 con 0x7f4ca80c1da0 2023-03-03T08:43:43.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c3e30 0x7f4ca8145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:43.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c3e30 0x7f4ca8145c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51382/0 (socket says 172.21.15.7:51382) 2023-03-03T08:43:43.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca77fe700 1 -- 172.21.15.7:0/3506563769 learned_addr learned my addr 172.21.15.7:0/3506563769 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:43.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca67fc700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ca80c1da0 0x7f4ca80c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:43.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca6ffd700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca806d870 0x7f4ca81ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:43.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca77fe700 1 -- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ca80c1da0 msgr2=0x7f4ca80c7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:43.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca77fe700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ca80c1da0 0x7f4ca80c7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca77fe700 1 -- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca806d870 msgr2=0x7f4ca81ed670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:43.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca77fe700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca806d870 0x7f4ca81ed670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca77fe700 1 -- 172.21.15.7:0/3506563769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c98009550 con 0x7f4ca80c3e30 2023-03-03T08:43:43.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4ca77fe700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c3e30 0x7f4ca8145c10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4ca000cd50 tx=0x7f4ca000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:43.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4c8ffff700 1 -- 172.21.15.7:0/3506563769 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca000cea0 con 0x7f4ca80c3e30 2023-03-03T08:43:43.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4c8ffff700 1 -- 172.21.15.7:0/3506563769 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca000f480 con 0x7f4ca80c3e30 2023-03-03T08:43:43.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4cad976700 1 -- 172.21.15.7:0/3506563769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca8154ba0 con 0x7f4ca80c3e30 2023-03-03T08:43:43.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4c8ffff700 1 -- 172.21.15.7:0/3506563769 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca000f830 con 0x7f4ca80c3e30 2023-03-03T08:43:43.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.725+0000 7f4cad976700 1 -- 172.21.15.7:0/3506563769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca81548c0 con 0x7f4ca80c3e30 2023-03-03T08:43:43.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.726+0000 7f4cad976700 1 -- 172.21.15.7:0/3506563769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca806dc40 con 0x7f4ca80c3e30 2023-03-03T08:43:43.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.727+0000 7f4c8ffff700 1 -- 172.21.15.7:0/3506563769 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ca0009030 con 0x7f4ca80c3e30 2023-03-03T08:43:43.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.727+0000 7f4c8ffff700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 0x7f4c9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:43.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.727+0000 7f4c8ffff700 1 -- 172.21.15.7:0/3506563769 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6784+0+0 (secure 0 0 0) 0x7f4ca00867d0 con 0x7f4ca80c3e30 2023-03-03T08:43:43.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.727+0000 7f4ca6ffd700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 0x7f4c9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:43.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.728+0000 7f4ca6ffd700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 0x7f4c9406a7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80c3d10 tx=0x7f4c9c007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:43.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.732+0000 7f4c8ffff700 1 -- 172.21.15.7:0/3506563769 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ca0052d00 con 0x7f4ca80c3e30 2023-03-03T08:43:43.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.969+0000 7f4cad976700 1 -- 172.21.15.7:0/3506563769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"} v 0) v1 -- 0x7f4ca8074bb0 con 0x7f4ca80c3e30 2023-03-03T08:43:43.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.974+0000 7f4c8ffff700 1 -- 172.21.15.7:0/3506563769 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"}]=0 new fs with metadata pool 5 and data pool 6 v10) v1 ==== 157+0+0 (secure 0 0 0) 0x7f4ca0056320 con 0x7f4ca80c3e30 2023-03-03T08:43:43.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 -- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 msgr2=0x7f4c9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:43.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 0x7f4c9406a7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80c3d10 tx=0x7f4c9c007ec0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 -- 172.21.15.7:0/3506563769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c3e30 msgr2=0x7f4ca8145c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:43.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c3e30 0x7f4ca8145c10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4ca000cd50 tx=0x7f4ca000a560 comp rx=0 tx=0).stop 2023-03-03T08:43:43.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 -- 172.21.15.7:0/3506563769 shutdown_connections 2023-03-03T08:43:43.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 0x7f4c9406a7a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ca80c3e30 0x7f4ca8145c10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ca806d870 0x7f4ca81ed670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 --2- 172.21.15.7:0/3506563769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ca80c1da0 0x7f4ca80c7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:43.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 -- 172.21.15.7:0/3506563769 >> 172.21.15.7:0/3506563769 conn(0x7f4ca806c110 msgr2=0x7f4ca8152cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:43.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 -- 172.21.15.7:0/3506563769 shutdown_connections 2023-03-03T08:43:43.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:43.977+0000 7f4cad976700 1 -- 172.21.15.7:0/3506563769 wait complete. 2023-03-03T08:43:43.981 INFO:teuthology.orchestra.run.smithi007.stderr:new fs with metadata pool 5 and data pool 6 2023-03-03T08:43:44.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:44.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.129+0000 7f92cf9be700 1 Processor -- start 2023-03-03T08:43:44.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.130+0000 7f92cf9be700 1 -- start start 2023-03-03T08:43:44.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.130+0000 7f92cf9be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c806a2b0 0x7f92c806a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.130+0000 7f92cf9be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f92c8157a40 0x7f92c8157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.130+0000 7f92cf9be700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c80c1c10 con 0x7f92c80ab0c0 2023-03-03T08:43:44.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.130+0000 7f92cf9be700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c8154530 con 0x7f92c8157a40 2023-03-03T08:43:44.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.130+0000 7f92cf9be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8154810 con 0x7f92c806a2b0 2023-03-03T08:43:44.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.130+0000 7f92cd75a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f92c80ab0c0 0x7f92c80c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45384/0 (socket says 172.21.15.7:45384) 2023-03-03T08:43:44.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.130+0000 7f92cd75a700 1 -- 172.21.15.7:0/1144162329 learned_addr learned my addr 172.21.15.7:0/1144162329 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:44.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92ccf59700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c806a2b0 0x7f92c806a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036990136 0 0) 0x7f92bc001110 con 0x7f92c80ab0c0 2023-03-03T08:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92cdf5b700 1 --2- 172.21.15.7:0/1144162329 >> v2:172.21.15.143:3300/0 conn(0x7f92c8157a40 0x7f92c8157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b40035b0 con 0x7f92c80ab0c0 2023-03-03T08:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1686731541 0 0) 0x7f92b40035b0 con 0x7f92c80ab0c0 2023-03-03T08:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c806a2b0 msgr2=0x7f92c806a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c806a2b0 0x7f92c806a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 >> v2:172.21.15.143:3300/0 conn(0x7f92c8157a40 msgr2=0x7f92c8157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 --2- 172.21.15.7:0/1144162329 >> v2:172.21.15.143:3300/0 conn(0x7f92c8157a40 0x7f92c8157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c8154af0 con 0x7f92c80ab0c0 2023-03-03T08:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f92bc001800 con 0x7f92c80ab0c0 2023-03-03T08:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 >> v1:172.21.15.7:6789/0 conn(0x7f92c80ab0c0 legacy=0x7f92c80c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 0x7f92b4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92b4004850 0x7f92b4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c806a2b0 0x7f92b4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8154530 con 0x7f92b4003a70 2023-03-03T08:43:44.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8154810 con 0x7f92b4004850 2023-03-03T08:43:44.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b400ba10 con 0x7f92c806a2b0 2023-03-03T08:43:44.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92cd75a700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 0x7f92b4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92ccf59700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92b4004850 0x7f92b4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92cdf5b700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c806a2b0 0x7f92b4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92cd75a700 1 -- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c806a2b0 msgr2=0x7f92b4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:44.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92cd75a700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c806a2b0 0x7f92b4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92cd75a700 1 -- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92b4004850 msgr2=0x7f92b4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:44.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92cd75a700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92b4004850 0x7f92b4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92cd75a700 1 -- 172.21.15.7:0/1144162329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b40035b0 con 0x7f92b4003a70 2023-03-03T08:43:44.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.131+0000 7f92ccf59700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92b4004850 0x7f92b4004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:44.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92cd75a700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 0x7f92b4003e80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00c320 tx=0x7f92b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a4fe2ff1c575a8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:44.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc00ce20 con 0x7f92b4003a70 2023-03-03T08:43:44.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc005960 con 0x7f92b4003a70 2023-03-03T08:43:44.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92bbfff700 1 -- 172.21.15.7:0/1144162329 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc005d10 con 0x7f92b4003a70 2023-03-03T08:43:44.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92cf9be700 1 -- 172.21.15.7:0/1144162329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 msgr2=0x7f92b4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92cf9be700 1 --2- 172.21.15.7:0/1144162329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 0x7f92b4003e80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f92bc00c320 tx=0x7f92b4005880 comp rx=0 tx=0).stop 2023-03-03T08:43:44.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92cdf5b700 1 -- 172.21.15.7:0/1144162329 reap_dead start 2023-03-03T08:43:44.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92cf9be700 1 -- 172.21.15.7:0/1144162329 shutdown_connections 2023-03-03T08:43:44.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92cf9be700 1 -- 172.21.15.7:0/1144162329 >> 172.21.15.7:0/1144162329 conn(0x7f92c80a9430 msgr2=0x7f92c80c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:44.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92cf9be700 1 -- 172.21.15.7:0/1144162329 shutdown_connections 2023-03-03T08:43:44.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.132+0000 7f92cf9be700 1 -- 172.21.15.7:0/1144162329 wait complete. 2023-03-03T08:43:44.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cf9be700 1 Processor -- start 2023-03-03T08:43:44.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cf9be700 1 -- start start 2023-03-03T08:43:44.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cf9be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 0x7f92c8150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cf9be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c8155580 0x7f92c8156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cf9be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c8156fd0 0x7f92c8146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cf9be700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8155ca0 con 0x7f92b4003a70 2023-03-03T08:43:44.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cf9be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8156700 con 0x7f92c8156fd0 2023-03-03T08:43:44.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cf9be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c80715a0 con 0x7f92c8155580 2023-03-03T08:43:44.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cd75a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 0x7f92c8150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cd75a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 0x7f92c8150b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51416/0 (socket says 172.21.15.7:51416) 2023-03-03T08:43:44.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.134+0000 7f92cd75a700 1 -- 172.21.15.7:0/3728577011 learned_addr learned my addr 172.21.15.7:0/3728577011 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:44.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92ccf59700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c8155580 0x7f92c8156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92cdf5b700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c8156fd0 0x7f92c8146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92cd75a700 1 -- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c8155580 msgr2=0x7f92c8156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92cd75a700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c8155580 0x7f92c8156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92cd75a700 1 -- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c8156fd0 msgr2=0x7f92c8146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92cd75a700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c8156fd0 0x7f92c8146d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92cd75a700 1 -- 172.21.15.7:0/3728577011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c81550b0 con 0x7f92b4003a70 2023-03-03T08:43:44.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92cdf5b700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c8156fd0 0x7f92c8146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:44.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92cd75a700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 0x7f92c8150b40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f92bc000fb0 tx=0x7f92bc000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:44.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92b9ffb700 1 -- 172.21.15.7:0/3728577011 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc004870 con 0x7f92b4003a70 2023-03-03T08:43:44.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92b9ffb700 1 -- 172.21.15.7:0/3728577011 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc0049d0 con 0x7f92b4003a70 2023-03-03T08:43:44.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92cf9be700 1 -- 172.21.15.7:0/3728577011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c8154dd0 con 0x7f92b4003a70 2023-03-03T08:43:44.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92b9ffb700 1 -- 172.21.15.7:0/3728577011 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc004d60 con 0x7f92b4003a70 2023-03-03T08:43:44.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.135+0000 7f92cf9be700 1 -- 172.21.15.7:0/3728577011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c80ab490 con 0x7f92b4003a70 2023-03-03T08:43:44.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.136+0000 7f92cf9be700 1 -- 172.21.15.7:0/3728577011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c8155f40 con 0x7f92b4003a70 2023-03-03T08:43:44.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.137+0000 7f92b9ffb700 1 -- 172.21.15.7:0/3728577011 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f92bc027030 con 0x7f92b4003a70 2023-03-03T08:43:44.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.137+0000 7f92b9ffb700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ac0682f0 0x7f92ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.137+0000 7f92b9ffb700 1 -- 172.21.15.7:0/3728577011 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f92bc086800 con 0x7f92b4003a70 2023-03-03T08:43:44.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.137+0000 7f92ccf59700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ac0682f0 0x7f92ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.138+0000 7f92ccf59700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ac0682f0 0x7f92ac06a7a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f92c81485a0 tx=0x7f92c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:44.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.141+0000 7f92b9ffb700 1 -- 172.21.15.7:0/3728577011 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f92bc052cd0 con 0x7f92b4003a70 2023-03-03T08:43:44.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.335+0000 7f92cf9be700 1 -- 172.21.15.7:0/3728577011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92c80c1c10 con 0x7f92b4003a70 2023-03-03T08:43:44.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.336+0000 7f92b9ffb700 1 -- 172.21.15.7:0/3728577011 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+21546 (secure 0 0 0) 0x7f92c80c1c10 con 0x7f92b4003a70 2023-03-03T08:43:44.339 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:44.339 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":26,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:43.971173+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:44.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 -- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ac0682f0 msgr2=0x7f92ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ac0682f0 0x7f92ac06a7a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f92c81485a0 tx=0x7f92c4008040 comp rx=0 tx=0).stop 2023-03-03T08:43:44.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 -- 172.21.15.7:0/3728577011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 msgr2=0x7f92c8150b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 0x7f92c8150b40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f92bc000fb0 tx=0x7f92bc000fe0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 -- 172.21.15.7:0/3728577011 shutdown_connections 2023-03-03T08:43:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c8156fd0 0x7f92c8146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ac0682f0 0x7f92ac06a7a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92b4003a70 0x7f92c8150b40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 --2- 172.21.15.7:0/3728577011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c8155580 0x7f92c8156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 -- 172.21.15.7:0/3728577011 >> 172.21.15.7:0/3728577011 conn(0x7f92c80a9430 msgr2=0x7f92c806df90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:44.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 -- 172.21.15.7:0/3728577011 shutdown_connections 2023-03-03T08:43:44.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.339+0000 7f92cf9be700 1 -- 172.21.15.7:0/3728577011 wait complete. 2023-03-03T08:43:44.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:44.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.476+0000 7f97d751c700 1 Processor -- start 2023-03-03T08:43:44.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.476+0000 7f97d751c700 1 -- start start 2023-03-03T08:43:44.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d751c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97d00ab0c0 0x7f97d00c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d751c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 0x7f97d006a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d751c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f97d0157a40 0x7f97d0157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d751c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d0154530 con 0x7f97d00ab0c0 2023-03-03T08:43:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d751c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d0154810 con 0x7f97d0157a40 2023-03-03T08:43:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d751c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d0154af0 con 0x7f97d006a2b0 2023-03-03T08:43:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d52b8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97d00ab0c0 0x7f97d00c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d52b8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97d00ab0c0 0x7f97d00c1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51424/0 (socket says 172.21.15.7:51424) 2023-03-03T08:43:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d52b8700 1 -- 172.21.15.7:0/998342894 learned_addr learned my addr 172.21.15.7:0/998342894 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:44.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d4ab7700 1 --2- 172.21.15.7:0/998342894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 0x7f97d006a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d5ab9700 1 --2- 172.21.15.7:0/998342894 >> v2:172.21.15.143:3300/0 conn(0x7f97d0157a40 0x7f97d0157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d52b8700 1 -- 172.21.15.7:0/998342894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 msgr2=0x7f97d006a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:44.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d52b8700 1 --2- 172.21.15.7:0/998342894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 0x7f97d006a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d52b8700 1 -- 172.21.15.7:0/998342894 >> v2:172.21.15.143:3300/0 conn(0x7f97d0157a40 msgr2=0x7f97d0157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d52b8700 1 --2- 172.21.15.7:0/998342894 >> v2:172.21.15.143:3300/0 conn(0x7f97d0157a40 0x7f97d0157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d52b8700 1 -- 172.21.15.7:0/998342894 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d0154dd0 con 0x7f97d00ab0c0 2023-03-03T08:43:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d5ab9700 1 --2- 172.21.15.7:0/998342894 >> v2:172.21.15.143:3300/0 conn(0x7f97d0157a40 0x7f97d0157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97d52b8700 1 --2- 172.21.15.7:0/998342894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d00ab0c0 0x7f97d00c1500 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f97d00c2540 tx=0x7f97cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102f9b630a46e5df server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.477+0000 7f97c7fff700 1 -- 172.21.15.7:0/998342894 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc00b870 con 0x7f97d00ab0c0 2023-03-03T08:43:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97c7fff700 1 -- 172.21.15.7:0/998342894 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97cc00b9d0 con 0x7f97d00ab0c0 2023-03-03T08:43:44.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97c7fff700 1 -- 172.21.15.7:0/998342894 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97cc00bd60 con 0x7f97d00ab0c0 2023-03-03T08:43:44.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97d751c700 1 -- 172.21.15.7:0/998342894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d00ab0c0 msgr2=0x7f97d00c1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97d751c700 1 --2- 172.21.15.7:0/998342894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d00ab0c0 0x7f97d00c1500 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f97d00c2540 tx=0x7f97cc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97d751c700 1 -- 172.21.15.7:0/998342894 shutdown_connections 2023-03-03T08:43:44.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97d751c700 1 --2- 172.21.15.7:0/998342894 >> v2:172.21.15.143:3300/0 conn(0x7f97d0157a40 0x7f97d0157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97d751c700 1 --2- 172.21.15.7:0/998342894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d00ab0c0 0x7f97d00c1500 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97d751c700 1 --2- 172.21.15.7:0/998342894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 0x7f97d006a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97d751c700 1 -- 172.21.15.7:0/998342894 >> 172.21.15.7:0/998342894 conn(0x7f97d00a9430 msgr2=0x7f97d00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:44.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97d751c700 1 -- 172.21.15.7:0/998342894 shutdown_connections 2023-03-03T08:43:44.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.478+0000 7f97d751c700 1 -- 172.21.15.7:0/998342894 wait complete. 2023-03-03T08:43:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.479+0000 7f97d751c700 1 Processor -- start 2023-03-03T08:43:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.479+0000 7f97d751c700 1 -- start start 2023-03-03T08:43:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d751c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 0x7f97d01e0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d751c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97d00ab0c0 0x7f97d01501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d751c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d0157a40 0x7f97d0150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d751c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d014ea60 con 0x7f97d0157a40 2023-03-03T08:43:44.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d751c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d014fd50 con 0x7f97d00ab0c0 2023-03-03T08:43:44.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d751c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d014e780 con 0x7f97d006a2b0 2023-03-03T08:43:44.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d5ab9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d0157a40 0x7f97d0150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d52b8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 0x7f97d01e0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d4ab7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97d00ab0c0 0x7f97d01501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d5ab9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d0157a40 0x7f97d0150720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51434/0 (socket says 172.21.15.7:51434) 2023-03-03T08:43:44.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d52b8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 0x7f97d01e0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59502/0 (socket says 172.21.15.7:59502) 2023-03-03T08:43:44.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.480+0000 7f97d5ab9700 1 -- 172.21.15.7:0/641620274 learned_addr learned my addr 172.21.15.7:0/641620274 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:44.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97d5ab9700 1 -- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 msgr2=0x7f97d01e0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97d5ab9700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 0x7f97d01e0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97d5ab9700 1 -- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97d00ab0c0 msgr2=0x7f97d01501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97d5ab9700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97d00ab0c0 0x7f97d01501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97d5ab9700 1 -- 172.21.15.7:0/641620274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc009550 con 0x7f97d0157a40 2023-03-03T08:43:44.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97d5ab9700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d0157a40 0x7f97d0150720 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009fa0 tx=0x7f97c800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:44.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97c5ffb700 1 -- 172.21.15.7:0/641620274 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c800ddb0 con 0x7f97d0157a40 2023-03-03T08:43:44.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97c5ffb700 1 -- 172.21.15.7:0/641620274 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c800f040 con 0x7f97d0157a40 2023-03-03T08:43:44.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97d751c700 1 -- 172.21.15.7:0/641620274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d00ab490 con 0x7f97d0157a40 2023-03-03T08:43:44.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97c5ffb700 1 -- 172.21.15.7:0/641620274 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8003c60 con 0x7f97d0157a40 2023-03-03T08:43:44.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.481+0000 7f97d751c700 1 -- 172.21.15.7:0/641620274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d01550b0 con 0x7f97d0157a40 2023-03-03T08:43:44.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.482+0000 7f97d751c700 1 -- 172.21.15.7:0/641620274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d00c31c0 con 0x7f97d0157a40 2023-03-03T08:43:44.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.483+0000 7f97c5ffb700 1 -- 172.21.15.7:0/641620274 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f97c801c030 con 0x7f97d0157a40 2023-03-03T08:43:44.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.483+0000 7f97c5ffb700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97b4068320 0x7f97b406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.483+0000 7f97c5ffb700 1 -- 172.21.15.7:0/641620274 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f97c8085ea0 con 0x7f97d0157a40 2023-03-03T08:43:44.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.483+0000 7f97d52b8700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97b4068320 0x7f97b406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.484+0000 7f97d52b8700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97b4068320 0x7f97b406a7d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f97cc000c00 tx=0x7f97cc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:44.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.487+0000 7f97c5ffb700 1 -- 172.21.15.7:0/641620274 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f97c80523d0 con 0x7f97d0157a40 2023-03-03T08:43:44.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.684+0000 7f97d751c700 1 -- 172.21.15.7:0/641620274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97d014f560 con 0x7f97d0157a40 2023-03-03T08:43:44.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.686+0000 7f97c5ffb700 1 -- 172.21.15.7:0/641620274 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+21546 (secure 0 0 0) 0x7f97d014f560 con 0x7f97d0157a40 2023-03-03T08:43:44.688 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:44.689 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":26,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:43.971173+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:44.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 -- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97b4068320 msgr2=0x7f97b406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97b4068320 0x7f97b406a7d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f97cc000c00 tx=0x7f97cc008030 comp rx=0 tx=0).stop 2023-03-03T08:43:44.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 -- 172.21.15.7:0/641620274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d0157a40 msgr2=0x7f97d0150720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d0157a40 0x7f97d0150720 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009fa0 tx=0x7f97c800d3b0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 -- 172.21.15.7:0/641620274 shutdown_connections 2023-03-03T08:43:44.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97b4068320 0x7f97b406a7d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97d0157a40 0x7f97d0150720 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97d006a2b0 0x7f97d01e0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 --2- 172.21.15.7:0/641620274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97d00ab0c0 0x7f97d01501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 -- 172.21.15.7:0/641620274 >> 172.21.15.7:0/641620274 conn(0x7f97d00a9430 msgr2=0x7f97d00c1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:44.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 -- 172.21.15.7:0/641620274 shutdown_connections 2023-03-03T08:43:44.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.688+0000 7f97d751c700 1 -- 172.21.15.7:0/641620274 wait complete. 2023-03-03T08:43:44.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b standby_count_wanted 0 2023-03-03T08:43:44.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.828+0000 7f30b3285700 1 Processor -- start 2023-03-03T08:43:44.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.828+0000 7f30b3285700 1 -- start start 2023-03-03T08:43:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b3285700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac14de50 0x7f30ac06a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b3285700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f30ac0c65c0 0x7f30ac0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b3285700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac0a65b0 con 0x7f30ac06a7f0 2023-03-03T08:43:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b3285700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ac154e00 con 0x7f30ac0c65c0 2023-03-03T08:43:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b3285700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac0c3370 con 0x7f30ac14de50 2023-03-03T08:43:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b0820700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f30ac0c65c0 0x7f30ac0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b0820700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f30ac0c65c0 0x7f30ac0c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51440/0 (socket says 172.21.15.7:51440) 2023-03-03T08:43:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b0820700 1 -- 172.21.15.7:0/20604485 learned_addr learned my addr 172.21.15.7:0/20604485 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b1021700 1 --2- 172.21.15.7:0/20604485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac14de50 0x7f30ac06a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b0820700 1 -- 172.21.15.7:0/20604485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac14de50 msgr2=0x7f30ac06a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b0820700 1 --2- 172.21.15.7:0/20604485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac14de50 0x7f30ac06a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b0820700 1 -- 172.21.15.7:0/20604485 >> v1:172.21.15.143:6789/0 conn(0x7f30ac06a7f0 legacy=0x7f30ac0c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b0820700 1 -- 172.21.15.7:0/20604485 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac1541f0 con 0x7f30ac0c65c0 2023-03-03T08:43:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30b0820700 1 --2- 172.21.15.7:0/20604485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 0x7f30ac0c2720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f309c00bb90 tx=0x7f309c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79961370f190f02 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30a3fff700 1 -- 172.21.15.7:0/20604485 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c00dc00 con 0x7f30ac0c65c0 2023-03-03T08:43:44.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30a3fff700 1 -- 172.21.15.7:0/20604485 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c00dd60 con 0x7f30ac0c65c0 2023-03-03T08:43:44.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.829+0000 7f30a3fff700 1 -- 172.21.15.7:0/20604485 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c004270 con 0x7f30ac0c65c0 2023-03-03T08:43:44.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.830+0000 7f30b3285700 1 -- 172.21.15.7:0/20604485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 msgr2=0x7f30ac0c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.830+0000 7f30b3285700 1 --2- 172.21.15.7:0/20604485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 0x7f30ac0c2720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f309c00bb90 tx=0x7f309c00bea0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.830+0000 7f30b3285700 1 -- 172.21.15.7:0/20604485 shutdown_connections 2023-03-03T08:43:44.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.830+0000 7f30b3285700 1 --2- 172.21.15.7:0/20604485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac14de50 0x7f30ac06a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.830+0000 7f30b3285700 1 --2- 172.21.15.7:0/20604485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 0x7f30ac0c2720 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.830+0000 7f30b3285700 1 -- 172.21.15.7:0/20604485 >> 172.21.15.7:0/20604485 conn(0x7f30ac0a9430 msgr2=0x7f30ac0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:44.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.830+0000 7f30b3285700 1 -- 172.21.15.7:0/20604485 shutdown_connections 2023-03-03T08:43:44.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.830+0000 7f30b3285700 1 -- 172.21.15.7:0/20604485 wait complete. 2023-03-03T08:43:44.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.831+0000 7f30b3285700 1 Processor -- start 2023-03-03T08:43:44.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.831+0000 7f30b3285700 1 -- start start 2023-03-03T08:43:44.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b3285700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30ac06a7f0 0x7f30ac151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b3285700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 0x7f30ac0c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b3285700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac14de50 0x7f30ac1e61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b3285700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac149370 con 0x7f30ac0c65c0 2023-03-03T08:43:44.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b3285700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac149630 con 0x7f30ac06a7f0 2023-03-03T08:43:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b3285700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac14bbb0 con 0x7f30ac14de50 2023-03-03T08:43:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b0820700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 0x7f30ac0c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b0820700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 0x7f30ac0c3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51446/0 (socket says 172.21.15.7:51446) 2023-03-03T08:43:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b0820700 1 -- 172.21.15.7:0/3659781454 learned_addr learned my addr 172.21.15.7:0/3659781454 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b1822700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac14de50 0x7f30ac1e61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b1021700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30ac06a7f0 0x7f30ac151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b0820700 1 -- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac14de50 msgr2=0x7f30ac1e61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b0820700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac14de50 0x7f30ac1e61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b0820700 1 -- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30ac06a7f0 msgr2=0x7f30ac151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b0820700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30ac06a7f0 0x7f30ac151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.832+0000 7f30b0820700 1 -- 172.21.15.7:0/3659781454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c00b7f0 con 0x7f30ac0c65c0 2023-03-03T08:43:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.833+0000 7f30b0820700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 0x7f30ac0c3aa0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f309c019040 tx=0x7f309c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.833+0000 7f30a1ffb700 1 -- 172.21.15.7:0/3659781454 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c003cb0 con 0x7f30ac0c65c0 2023-03-03T08:43:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.833+0000 7f30a1ffb700 1 -- 172.21.15.7:0/3659781454 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c003e10 con 0x7f30ac0c65c0 2023-03-03T08:43:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.833+0000 7f30b3285700 1 -- 172.21.15.7:0/3659781454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac1547b0 con 0x7f30ac0c65c0 2023-03-03T08:43:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.833+0000 7f30a1ffb700 1 -- 172.21.15.7:0/3659781454 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c020680 con 0x7f30ac0c65c0 2023-03-03T08:43:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.833+0000 7f30b3285700 1 -- 172.21.15.7:0/3659781454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac1544d0 con 0x7f30ac0c65c0 2023-03-03T08:43:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.834+0000 7f30b3285700 1 -- 172.21.15.7:0/3659781454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ac154a90 con 0x7f30ac0c65c0 2023-03-03T08:43:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.834+0000 7f30a1ffb700 1 -- 172.21.15.7:0/3659781454 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f309c027020 con 0x7f30ac0c65c0 2023-03-03T08:43:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.835+0000 7f30a1ffb700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30900682f0 0x7f309006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.835+0000 7f30a1ffb700 1 -- 172.21.15.7:0/3659781454 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f309c089de0 con 0x7f30ac0c65c0 2023-03-03T08:43:44.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.835+0000 7f30b1021700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30900682f0 0x7f309006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:44.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.835+0000 7f30b1021700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30900682f0 0x7f309006a7a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f30a8006b20 tx=0x7f30a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:44.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:44.839+0000 7f30a1ffb700 1 -- 172.21.15.7:0/3659781454 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f309c059de0 con 0x7f30ac0c65c0 2023-03-03T08:43:45.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.078+0000 7f30b3285700 1 -- 172.21.15.7:0/3659781454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f30ac154a90 con 0x7f30ac0c65c0 2023-03-03T08:43:45.200 INFO:tasks.ceph.mds.c.smithi007.stderr:2023-03-03T08:43:45.197+0000 7f9933a91700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:43:45.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.983+0000 7f30a1ffb700 1 -- 172.21.15.7:0/3659781454 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"}]=0 v12) v1 ==== 113+0+0 (secure 0 0 0) 0x7f309c0599d0 con 0x7f30ac0c65c0 2023-03-03T08:43:45.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.985+0000 7f30b3285700 1 -- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30900682f0 msgr2=0x7f309006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:45.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.985+0000 7f30b3285700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30900682f0 0x7f309006a7a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f30a8006b20 tx=0x7f30a8008040 comp rx=0 tx=0).stop 2023-03-03T08:43:45.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.985+0000 7f30b3285700 1 -- 172.21.15.7:0/3659781454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 msgr2=0x7f30ac0c3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:45.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.985+0000 7f30b3285700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 0x7f30ac0c3aa0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f309c019040 tx=0x7f309c004750 comp rx=0 tx=0).stop 2023-03-03T08:43:45.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.985+0000 7f30b3285700 1 -- 172.21.15.7:0/3659781454 shutdown_connections 2023-03-03T08:43:45.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.985+0000 7f30b3285700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30ac14de50 0x7f30ac1e61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:45.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.985+0000 7f30b3285700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30ac06a7f0 0x7f30ac151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:45.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.985+0000 7f30b3285700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30900682f0 0x7f309006a7a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:45.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.985+0000 7f30b3285700 1 --2- 172.21.15.7:0/3659781454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30ac0c65c0 0x7f30ac0c3aa0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:45.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.985+0000 7f30b3285700 1 -- 172.21.15.7:0/3659781454 >> 172.21.15.7:0/3659781454 conn(0x7f30ac0a9430 msgr2=0x7f30ac14c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:45.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.986+0000 7f30b3285700 1 -- 172.21.15.7:0/3659781454 shutdown_connections 2023-03-03T08:43:45.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:45.986+0000 7f30b3285700 1 -- 172.21.15.7:0/3659781454 wait complete. 2023-03-03T08:43:46.002 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2023-03-03T08:43:46.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b allow_standby_replay false 2023-03-03T08:43:46.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.136+0000 7f788dbc8700 1 Processor -- start 2023-03-03T08:43:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.136+0000 7f788dbc8700 1 -- start start 2023-03-03T08:43:46.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f788dbc8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f78880c4920 0x7f788806eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:46.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f788dbc8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78880c3020 con 0x7f78880c44a0 2023-03-03T08:43:46.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f788dbc8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78881552d0 con 0x7f78880aae30 2023-03-03T08:43:46.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f788dbc8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788806f080 con 0x7f78880c4920 2023-03-03T08:43:46.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f7887fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f78880c4920 0x7f788806eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:46.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f7886ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f78880c44a0 0x7f788806e430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45434/0 (socket says 172.21.15.7:45434) 2023-03-03T08:43:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f7886ffd700 1 -- 172.21.15.7:0/1196121244 learned_addr learned my addr 172.21.15.7:0/1196121244 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f7887fff700 1 -- 172.21.15.7:0/1196121244 >> v1:172.21.15.143:6789/0 conn(0x7f78880aae30 legacy=0x7f78880beab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f7887fff700 1 -- 172.21.15.7:0/1196121244 >> v1:172.21.15.7:6789/0 conn(0x7f78880c44a0 legacy=0x7f788806e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f7887fff700 1 -- 172.21.15.7:0/1196121244 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78881546e0 con 0x7f78880c4920 2023-03-03T08:43:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f78867fc700 1 -- 172.21.15.7:0/1196121244 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414230586 0 0) 0x7f78880c3020 con 0x7f78880c44a0 2023-03-03T08:43:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.137+0000 7f7887fff700 1 --2- 172.21.15.7:0/1196121244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 0x7f788806eb40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7880009d90 tx=0x7f788000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbaaa580b207fa0b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.138+0000 7f78867fc700 1 -- 172.21.15.7:0/1196121244 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788000e070 con 0x7f78880c4920 2023-03-03T08:43:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.138+0000 7f78867fc700 1 -- 172.21.15.7:0/1196121244 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788000bbd0 con 0x7f78880c4920 2023-03-03T08:43:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.138+0000 7f78867fc700 1 -- 172.21.15.7:0/1196121244 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880012430 con 0x7f78880c4920 2023-03-03T08:43:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.138+0000 7f788dbc8700 1 -- 172.21.15.7:0/1196121244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 msgr2=0x7f788806eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.138+0000 7f788dbc8700 1 --2- 172.21.15.7:0/1196121244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 0x7f788806eb40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7880009d90 tx=0x7f788000b330 comp rx=0 tx=0).stop 2023-03-03T08:43:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.138+0000 7f788dbc8700 1 -- 172.21.15.7:0/1196121244 shutdown_connections 2023-03-03T08:43:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.138+0000 7f788dbc8700 1 --2- 172.21.15.7:0/1196121244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 0x7f788806eb40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.138+0000 7f788dbc8700 1 -- 172.21.15.7:0/1196121244 >> 172.21.15.7:0/1196121244 conn(0x7f78880c8510 msgr2=0x7f78880c8910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.138+0000 7f788dbc8700 1 -- 172.21.15.7:0/1196121244 shutdown_connections 2023-03-03T08:43:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.138+0000 7f788dbc8700 1 -- 172.21.15.7:0/1196121244 wait complete. 2023-03-03T08:43:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.139+0000 7f788dbc8700 1 Processor -- start 2023-03-03T08:43:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f788dbc8700 1 -- start start 2023-03-03T08:43:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f788dbc8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78880aae30 0x7f78880c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f788dbc8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78880c44a0 0x7f7888158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f788dbc8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 0x7f78881ea320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:46.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f788dbc8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78881ed830 con 0x7f78880c4920 2023-03-03T08:43:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f788dbc8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78881edb10 con 0x7f78880aae30 2023-03-03T08:43:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f788dbc8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78881eddf0 con 0x7f78880c44a0 2023-03-03T08:43:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f7887fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 0x7f78881ea320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f7887fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 0x7f78881ea320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51462/0 (socket says 172.21.15.7:51462) 2023-03-03T08:43:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f7887fff700 1 -- 172.21.15.7:0/2960240595 learned_addr learned my addr 172.21.15.7:0/2960240595 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f78877fe700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78880aae30 0x7f78880c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.140+0000 7f7886ffd700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78880c44a0 0x7f7888158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f7887fff700 1 -- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78880c44a0 msgr2=0x7f7888158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f7887fff700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78880c44a0 0x7f7888158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f7887fff700 1 -- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78880aae30 msgr2=0x7f78880c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f7887fff700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78880aae30 0x7f78880c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f7887fff700 1 -- 172.21.15.7:0/2960240595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7880009a70 con 0x7f78880c4920 2023-03-03T08:43:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f7886ffd700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78880c44a0 0x7f7888158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f7887fff700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 0x7f78881ea320 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7880014040 tx=0x7f7880004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f786ffff700 1 -- 172.21.15.7:0/2960240595 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788001b810 con 0x7f78880c4920 2023-03-03T08:43:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f786ffff700 1 -- 172.21.15.7:0/2960240595 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7880020070 con 0x7f78880c4920 2023-03-03T08:43:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f786ffff700 1 -- 172.21.15.7:0/2960240595 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788001bbb0 con 0x7f78880c4920 2023-03-03T08:43:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f788dbc8700 1 -- 172.21.15.7:0/2960240595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78880bf1c0 con 0x7f78880c4920 2023-03-03T08:43:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.141+0000 7f788dbc8700 1 -- 172.21.15.7:0/2960240595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78880c4d20 con 0x7f78880c4920 2023-03-03T08:43:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.142+0000 7f788dbc8700 1 -- 172.21.15.7:0/2960240595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78881549a0 con 0x7f78880c4920 2023-03-03T08:43:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.142+0000 7f786ffff700 1 -- 172.21.15.7:0/2960240595 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f788002e070 con 0x7f78880c4920 2023-03-03T08:43:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.143+0000 7f786ffff700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78740682f0 0x7f787406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.143+0000 7f786ffff700 1 -- 172.21.15.7:0/2960240595 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f7880022070 con 0x7f78880c4920 2023-03-03T08:43:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.143+0000 7f78877fe700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78740682f0 0x7f787406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.143+0000 7f78877fe700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78740682f0 0x7f787406a7a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7878007fd0 tx=0x7f7878009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:46.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.147+0000 7f786ffff700 1 -- 172.21.15.7:0/2960240595 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f78800545d0 con 0x7f78880c4920 2023-03-03T08:43:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.389+0000 7f788dbc8700 1 -- 172.21.15.7:0/2960240595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7888074bb0 con 0x7f78880c4920 2023-03-03T08:43:46.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.984+0000 7f786ffff700 1 -- 172.21.15.7:0/2960240595 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"}]=0 v13) v1 ==== 117+0+0 (secure 0 0 0) 0x7f788002c0b0 con 0x7f78880c4920 2023-03-03T08:43:46.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 -- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78740682f0 msgr2=0x7f787406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:46.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78740682f0 0x7f787406a7a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7878007fd0 tx=0x7f7878009040 comp rx=0 tx=0).stop 2023-03-03T08:43:46.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 -- 172.21.15.7:0/2960240595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 msgr2=0x7f78881ea320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:46.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 0x7f78881ea320 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7880014040 tx=0x7f7880004ad0 comp rx=0 tx=0).stop 2023-03-03T08:43:46.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 -- 172.21.15.7:0/2960240595 shutdown_connections 2023-03-03T08:43:46.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78740682f0 0x7f787406a7a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:46.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78880c4920 0x7f78881ea320 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:46.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78880aae30 0x7f78880c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:46.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 --2- 172.21.15.7:0/2960240595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78880c44a0 0x7f7888158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:46.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 -- 172.21.15.7:0/2960240595 >> 172.21.15.7:0/2960240595 conn(0x7f78880c8510 msgr2=0x7f7888153b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:46.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 -- 172.21.15.7:0/2960240595 shutdown_connections 2023-03-03T08:43:46.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:46.987+0000 7f788dbc8700 1 -- 172.21.15.7:0/2960240595 wait complete. 2023-03-03T08:43:47.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b session_timeout 300 2023-03-03T08:43:47.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.137+0000 7effefd34700 1 Processor -- start 2023-03-03T08:43:47.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.138+0000 7effefd34700 1 -- start start 2023-03-03T08:43:47.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.138+0000 7effefd34700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe814b460 0x7effe806aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:47.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.138+0000 7effefd34700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7effe8072470 0x7effe806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:47.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.138+0000 7effefd34700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe8070360 con 0x7effe8071990 2023-03-03T08:43:47.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.138+0000 7effefd34700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe81521a0 con 0x7effe8072470 2023-03-03T08:43:47.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.138+0000 7effefd34700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe806a5e0 con 0x7effe814b460 2023-03-03T08:43:47.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effed2cf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe814b460 0x7effe806aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:47.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effee2d1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7effe8072470 0x7effe806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:47.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effed2cf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe814b460 0x7effe806aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59538/0 (socket says 172.21.15.7:59538) 2023-03-03T08:43:47.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effed2cf700 1 -- 172.21.15.7:0/2156381843 learned_addr learned my addr 172.21.15.7:0/2156381843 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:47.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effee2d1700 1 -- 172.21.15.7:0/2156381843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe814b460 msgr2=0x7effe806aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:47.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effee2d1700 1 --2- 172.21.15.7:0/2156381843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe814b460 0x7effe806aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:47.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effee2d1700 1 -- 172.21.15.7:0/2156381843 >> v1:172.21.15.143:6789/0 conn(0x7effe8071990 legacy=0x7effe8071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:47.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effee2d1700 1 -- 172.21.15.7:0/2156381843 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe80c38b0 con 0x7effe8072470 2023-03-03T08:43:47.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effed2cf700 1 --2- 172.21.15.7:0/2156381843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe814b460 0x7effe806aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:47.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effee2d1700 1 --2- 172.21.15.7:0/2156381843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe8072470 0x7effe806afb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effe0009d90 tx=0x7effe000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92388ab9b35767a6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:47.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effecace700 1 -- 172.21.15.7:0/2156381843 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe000e070 con 0x7effe8072470 2023-03-03T08:43:47.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effecace700 1 -- 172.21.15.7:0/2156381843 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe000bb20 con 0x7effe8072470 2023-03-03T08:43:47.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.139+0000 7effecace700 1 -- 172.21.15.7:0/2156381843 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe0012430 con 0x7effe8072470 2023-03-03T08:43:47.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.140+0000 7effefd34700 1 -- 172.21.15.7:0/2156381843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe8072470 msgr2=0x7effe806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:47.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.140+0000 7effefd34700 1 --2- 172.21.15.7:0/2156381843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe8072470 0x7effe806afb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effe0009d90 tx=0x7effe000b330 comp rx=0 tx=0).stop 2023-03-03T08:43:47.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.140+0000 7effefd34700 1 -- 172.21.15.7:0/2156381843 shutdown_connections 2023-03-03T08:43:47.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.140+0000 7effefd34700 1 --2- 172.21.15.7:0/2156381843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe8072470 0x7effe806afb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:47.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.140+0000 7effefd34700 1 --2- 172.21.15.7:0/2156381843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe814b460 0x7effe806aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:47.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.140+0000 7effefd34700 1 -- 172.21.15.7:0/2156381843 >> 172.21.15.7:0/2156381843 conn(0x7effe80a94d0 msgr2=0x7effe80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:47.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.140+0000 7effefd34700 1 -- 172.21.15.7:0/2156381843 shutdown_connections 2023-03-03T08:43:47.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.140+0000 7effefd34700 1 -- 172.21.15.7:0/2156381843 wait complete. 2023-03-03T08:43:47.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.141+0000 7effefd34700 1 Processor -- start 2023-03-03T08:43:47.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.141+0000 7effefd34700 1 -- start start 2023-03-03T08:43:47.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effefd34700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8071990 0x7effe80c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:47.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effefd34700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8072470 0x7effe81495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:47.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effefd34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe8151b20 0x7effe8149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:47.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effefd34700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe80aac40 con 0x7effe8072470 2023-03-03T08:43:47.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effefd34700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe80aa3e0 con 0x7effe8151b20 2023-03-03T08:43:47.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effefd34700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe814a030 con 0x7effe8071990 2023-03-03T08:43:47.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effed2cf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8072470 0x7effe81495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:47.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effed2cf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8072470 0x7effe81495b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51474/0 (socket says 172.21.15.7:51474) 2023-03-03T08:43:47.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effed2cf700 1 -- 172.21.15.7:0/2123257008 learned_addr learned my addr 172.21.15.7:0/2123257008 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:47.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effee2d1700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe8151b20 0x7effe8149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:47.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.142+0000 7effedad0700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8071990 0x7effe80c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:47.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effed2cf700 1 -- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8071990 msgr2=0x7effe80c04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:47.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effed2cf700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8071990 0x7effe80c04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:47.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effed2cf700 1 -- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe8151b20 msgr2=0x7effe8149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:47.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effed2cf700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe8151b20 0x7effe8149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:47.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effed2cf700 1 -- 172.21.15.7:0/2123257008 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe4009550 con 0x7effe8072470 2023-03-03T08:43:47.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effed2cf700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8072470 0x7effe81495b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7effe400ebe0 tx=0x7effe400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:47.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effde7fc700 1 -- 172.21.15.7:0/2123257008 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe400c850 con 0x7effe8072470 2023-03-03T08:43:47.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effde7fc700 1 -- 172.21.15.7:0/2123257008 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe400c9b0 con 0x7effe8072470 2023-03-03T08:43:47.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effefd34700 1 -- 172.21.15.7:0/2123257008 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe0009a70 con 0x7effe8072470 2023-03-03T08:43:47.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effde7fc700 1 -- 172.21.15.7:0/2123257008 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe400cd40 con 0x7effe8072470 2023-03-03T08:43:47.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.143+0000 7effefd34700 1 -- 172.21.15.7:0/2123257008 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe80c3b50 con 0x7effe8072470 2023-03-03T08:43:47.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.144+0000 7effefd34700 1 -- 172.21.15.7:0/2123257008 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe806a2b0 con 0x7effe8072470 2023-03-03T08:43:47.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.144+0000 7effde7fc700 1 -- 172.21.15.7:0/2123257008 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7effe4019030 con 0x7effe8072470 2023-03-03T08:43:47.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.145+0000 7effde7fc700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 0x7effcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:47.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.145+0000 7effde7fc700 1 -- 172.21.15.7:0/2123257008 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7effe4085f80 con 0x7effe8072470 2023-03-03T08:43:47.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.145+0000 7effedad0700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 0x7effcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:47.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.145+0000 7effedad0700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 0x7effcc06a7a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7effd8007fd0 tx=0x7effd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:47.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.149+0000 7effde7fc700 1 -- 172.21.15.7:0/2123257008 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7effe40524b0 con 0x7effe8072470 2023-03-03T08:43:47.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.387+0000 7effefd34700 1 -- 172.21.15.7:0/2123257008 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7effe806b4f0 con 0x7effe8072470 2023-03-03T08:43:47.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.996+0000 7effde7fc700 1 -- 172.21.15.7:0/2123257008 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"}]=0 v14) v1 ==== 110+0+0 (secure 0 0 0) 0x7effe40210b0 con 0x7effe8072470 2023-03-03T08:43:48.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 -- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 msgr2=0x7effcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 0x7effcc06a7a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7effd8007fd0 tx=0x7effd8009040 comp rx=0 tx=0).stop 2023-03-03T08:43:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 -- 172.21.15.7:0/2123257008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8072470 msgr2=0x7effe81495b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8072470 0x7effe81495b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7effe400ebe0 tx=0x7effe400eef0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 -- 172.21.15.7:0/2123257008 shutdown_connections 2023-03-03T08:43:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe8151b20 0x7effe8149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8071990 0x7effe80c04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 0x7effcc06a7a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 --2- 172.21.15.7:0/2123257008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8072470 0x7effe81495b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 -- 172.21.15.7:0/2123257008 >> 172.21.15.7:0/2123257008 conn(0x7effe80a94d0 msgr2=0x7effe80c5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 -- 172.21.15.7:0/2123257008 shutdown_connections 2023-03-03T08:43:48.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:47.999+0000 7effefd34700 1 -- 172.21.15.7:0/2123257008 wait complete. 2023-03-03T08:43:48.016 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'b' 2023-03-03T08:43:48.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:43:48.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.145+0000 7f2568f91700 1 Processor -- start 2023-03-03T08:43:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.145+0000 7f2568f91700 1 -- start start 2023-03-03T08:43:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.146+0000 7f2568f91700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f25640aae30 0x7f25640beab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.146+0000 7f2568f91700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f256406e430 0x7f256406e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.146+0000 7f2568f91700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25640c3020 con 0x7f25640beff0 2023-03-03T08:43:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.146+0000 7f2568f91700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25641552d0 con 0x7f25640aae30 2023-03-03T08:43:48.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.146+0000 7f2568f91700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25641546e0 con 0x7f256406e430 2023-03-03T08:43:48.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.146+0000 7f256259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f25640aae30 0x7f25640beab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.146+0000 7f2561d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f25640beff0 0x7f25640c44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45462/0 (socket says 172.21.15.7:45462) 2023-03-03T08:43:48.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.146+0000 7f2561d9b700 1 -- 172.21.15.7:0/2187968522 learned_addr learned my addr 172.21.15.7:0/2187968522 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:48.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f2562d9d700 1 --2- 172.21.15.7:0/2187968522 >> v2:172.21.15.143:3300/0 conn(0x7f256406e430 0x7f256406e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f256259c700 1 -- 172.21.15.7:0/2187968522 >> v2:172.21.15.143:3300/0 conn(0x7f256406e430 msgr2=0x7f256406e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:48.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f256259c700 1 --2- 172.21.15.7:0/2187968522 >> v2:172.21.15.143:3300/0 conn(0x7f256406e430 0x7f256406e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f256259c700 1 -- 172.21.15.7:0/2187968522 >> v1:172.21.15.7:6789/0 conn(0x7f25640beff0 legacy=0x7f25640c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f256259c700 1 -- 172.21.15.7:0/2187968522 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25641549a0 con 0x7f25640aae30 2023-03-03T08:43:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f256159a700 1 -- 172.21.15.7:0/2187968522 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828496062 0 0) 0x7f25640c3020 con 0x7f25640beff0 2023-03-03T08:43:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f256259c700 1 --2- 172.21.15.7:0/2187968522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 0x7f25640beab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f25640711f0 tx=0x7f2558009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f37ee12a5220e1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f256159a700 1 -- 172.21.15.7:0/2187968522 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558011830 con 0x7f25640aae30 2023-03-03T08:43:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f256159a700 1 -- 172.21.15.7:0/2187968522 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2558011990 con 0x7f25640aae30 2023-03-03T08:43:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f256159a700 1 -- 172.21.15.7:0/2187968522 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558011d20 con 0x7f25640aae30 2023-03-03T08:43:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f2568f91700 1 -- 172.21.15.7:0/2187968522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 msgr2=0x7f25640beab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.147+0000 7f2568f91700 1 --2- 172.21.15.7:0/2187968522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 0x7f25640beab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f25640711f0 tx=0x7f2558009b20 comp rx=0 tx=0).stop 2023-03-03T08:43:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.148+0000 7f2568f91700 1 -- 172.21.15.7:0/2187968522 shutdown_connections 2023-03-03T08:43:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.148+0000 7f2568f91700 1 --2- 172.21.15.7:0/2187968522 >> v2:172.21.15.143:3300/0 conn(0x7f256406e430 0x7f256406e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.148+0000 7f2568f91700 1 --2- 172.21.15.7:0/2187968522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 0x7f25640beab0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.148+0000 7f2568f91700 1 -- 172.21.15.7:0/2187968522 >> 172.21.15.7:0/2187968522 conn(0x7f25640c8510 msgr2=0x7f25640c8910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.148+0000 7f2568f91700 1 -- 172.21.15.7:0/2187968522 shutdown_connections 2023-03-03T08:43:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.148+0000 7f2568f91700 1 -- 172.21.15.7:0/2187968522 wait complete. 2023-03-03T08:43:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.149+0000 7f2568f91700 1 Processor -- start 2023-03-03T08:43:48.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.149+0000 7f2568f91700 1 -- start start 2023-03-03T08:43:48.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.150+0000 7f2568f91700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256406e430 0x7f256406ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.150+0000 7f2568f91700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 0x7f25641469d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.150+0000 7f2568f91700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25640beff0 0x7f2564146f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.150+0000 7f2568f91700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25640abcd0 con 0x7f25640aae30 2023-03-03T08:43:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.150+0000 7f2568f91700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256414c120 con 0x7f256406e430 2023-03-03T08:43:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.150+0000 7f2568f91700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25641479a0 con 0x7f25640beff0 2023-03-03T08:43:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.150+0000 7f2561d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 0x7f25641469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.150+0000 7f2561d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 0x7f25641469d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51486/0 (socket says 172.21.15.7:51486) 2023-03-03T08:43:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.150+0000 7f2561d9b700 1 -- 172.21.15.7:0/1614015823 learned_addr learned my addr 172.21.15.7:0/1614015823 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2562d9d700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25640beff0 0x7f2564146f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f256259c700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256406e430 0x7f256406ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2561d9b700 1 -- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25640beff0 msgr2=0x7f2564146f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2561d9b700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25640beff0 0x7f2564146f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2561d9b700 1 -- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256406e430 msgr2=0x7f256406ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2561d9b700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256406e430 0x7f256406ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2561d9b700 1 -- 172.21.15.7:0/1614015823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25580094a0 con 0x7f25640aae30 2023-03-03T08:43:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2562d9d700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25640beff0 0x7f2564146f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f256259c700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256406e430 0x7f256406ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2561d9b700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 0x7f25641469d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f255400c550 tx=0x7f255400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2552ffd700 1 -- 172.21.15.7:0/1614015823 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554004c80 con 0x7f25640aae30 2023-03-03T08:43:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2552ffd700 1 -- 172.21.15.7:0/1614015823 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2554004e00 con 0x7f25640aae30 2023-03-03T08:43:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2568f91700 1 -- 172.21.15.7:0/1614015823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2564154f60 con 0x7f25640aae30 2023-03-03T08:43:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2552ffd700 1 -- 172.21.15.7:0/1614015823 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554007310 con 0x7f25640aae30 2023-03-03T08:43:48.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.151+0000 7f2568f91700 1 -- 172.21.15.7:0/1614015823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2564154c80 con 0x7f25640aae30 2023-03-03T08:43:48.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.152+0000 7f2568f91700 1 -- 172.21.15.7:0/1614015823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2564074bb0 con 0x7f25640aae30 2023-03-03T08:43:48.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.152+0000 7f2552ffd700 1 -- 172.21.15.7:0/1614015823 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f255401a020 con 0x7f25640aae30 2023-03-03T08:43:48.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.153+0000 7f2552ffd700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25440682f0 0x7f254406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.153+0000 7f2552ffd700 1 -- 172.21.15.7:0/1614015823 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f2554085b70 con 0x7f25640aae30 2023-03-03T08:43:48.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.153+0000 7f256259c700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25440682f0 0x7f254406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.153+0000 7f256259c700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25440682f0 0x7f254406a7a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f25580090b0 tx=0x7f2558008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.157+0000 7f2552ffd700 1 -- 172.21.15.7:0/1614015823 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2554052150 con 0x7f25640aae30 2023-03-03T08:43:48.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.394+0000 7f2568f91700 1 -- 172.21.15.7:0/1614015823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f256406fce0 con 0x7f25640aae30 2023-03-03T08:43:48.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.395+0000 7f2552ffd700 1 -- 172.21.15.7:0/1614015823 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f2554055770 con 0x7f25640aae30 2023-03-03T08:43:48.397 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:48.397 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:43:48.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.396+0000 7f2568f91700 1 -- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25440682f0 msgr2=0x7f254406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.396+0000 7f2568f91700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25440682f0 0x7f254406a7a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f25580090b0 tx=0x7f2558008030 comp rx=0 tx=0).stop 2023-03-03T08:43:48.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.396+0000 7f2568f91700 1 -- 172.21.15.7:0/1614015823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 msgr2=0x7f25641469d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.396+0000 7f2568f91700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 0x7f25641469d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f255400c550 tx=0x7f255400c910 comp rx=0 tx=0).stop 2023-03-03T08:43:48.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.396+0000 7f2568f91700 1 -- 172.21.15.7:0/1614015823 shutdown_connections 2023-03-03T08:43:48.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.396+0000 7f2568f91700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25640beff0 0x7f2564146f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.396+0000 7f2568f91700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f256406e430 0x7f256406ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.396+0000 7f2568f91700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25440682f0 0x7f254406a7a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.396+0000 7f2568f91700 1 --2- 172.21.15.7:0/1614015823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25640aae30 0x7f25641469d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.396+0000 7f2568f91700 1 -- 172.21.15.7:0/1614015823 >> 172.21.15.7:0/1614015823 conn(0x7f25640c8510 msgr2=0x7f2564152490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.397+0000 7f2568f91700 1 -- 172.21.15.7:0/1614015823 shutdown_connections 2023-03-03T08:43:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.397+0000 7f2568f91700 1 -- 172.21.15.7:0/1614015823 wait complete. 2023-03-03T08:43:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:43:48.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.533+0000 7fdaa59fd700 1 Processor -- start 2023-03-03T08:43:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.533+0000 7fdaa59fd700 1 -- start start 2023-03-03T08:43:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fdaa59fd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa00ab0c0 0x7fdaa00c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fdaa59fd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdaa006a2b0 0x7fdaa006a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fdaa59fd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdaa0157a40 0x7fdaa0157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fdaa59fd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa0154530 con 0x7fdaa006a2b0 2023-03-03T08:43:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fdaa59fd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa0154810 con 0x7fdaa0157a40 2023-03-03T08:43:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fdaa59fd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa0154af0 con 0x7fdaa00ab0c0 2023-03-03T08:43:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fda9e7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdaa006a2b0 0x7fdaa006a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fda9e7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdaa006a2b0 0x7fdaa006a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51496/0 (socket says 172.21.15.7:51496) 2023-03-03T08:43:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fda9e7fc700 1 -- 172.21.15.7:0/189824253 learned_addr learned my addr 172.21.15.7:0/189824253 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fda9effd700 1 --2- 172.21.15.7:0/189824253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa00ab0c0 0x7fdaa00c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fda9f7fe700 1 --2- 172.21.15.7:0/189824253 >> v2:172.21.15.143:3300/0 conn(0x7fdaa0157a40 0x7fdaa0157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fda9e7fc700 1 -- 172.21.15.7:0/189824253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa00ab0c0 msgr2=0x7fdaa00c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.534+0000 7fda9e7fc700 1 --2- 172.21.15.7:0/189824253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa00ab0c0 0x7fdaa00c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fda9e7fc700 1 -- 172.21.15.7:0/189824253 >> v2:172.21.15.143:3300/0 conn(0x7fdaa0157a40 msgr2=0x7fdaa0157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fda9e7fc700 1 --2- 172.21.15.7:0/189824253 >> v2:172.21.15.143:3300/0 conn(0x7fdaa0157a40 0x7fdaa0157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fda9e7fc700 1 -- 172.21.15.7:0/189824253 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa0154dd0 con 0x7fdaa006a2b0 2023-03-03T08:43:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fda9e7fc700 1 --2- 172.21.15.7:0/189824253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 0x7fdaa006a720 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdaa00c2540 tx=0x7fda88009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d6df262c4df342 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fda9dffb700 1 -- 172.21.15.7:0/189824253 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8800b870 con 0x7fdaa006a2b0 2023-03-03T08:43:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fda9dffb700 1 -- 172.21.15.7:0/189824253 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8800b9d0 con 0x7fdaa006a2b0 2023-03-03T08:43:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fda9dffb700 1 -- 172.21.15.7:0/189824253 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8800bd60 con 0x7fdaa006a2b0 2023-03-03T08:43:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fdaa59fd700 1 -- 172.21.15.7:0/189824253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 msgr2=0x7fdaa006a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fdaa59fd700 1 --2- 172.21.15.7:0/189824253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 0x7fdaa006a720 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdaa00c2540 tx=0x7fda88009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fdaa59fd700 1 -- 172.21.15.7:0/189824253 shutdown_connections 2023-03-03T08:43:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fdaa59fd700 1 --2- 172.21.15.7:0/189824253 >> v2:172.21.15.143:3300/0 conn(0x7fdaa0157a40 0x7fdaa0157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fdaa59fd700 1 --2- 172.21.15.7:0/189824253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa00ab0c0 0x7fdaa00c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fdaa59fd700 1 --2- 172.21.15.7:0/189824253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 0x7fdaa006a720 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fdaa59fd700 1 -- 172.21.15.7:0/189824253 >> 172.21.15.7:0/189824253 conn(0x7fdaa00a9430 msgr2=0x7fdaa00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.535+0000 7fdaa59fd700 1 -- 172.21.15.7:0/189824253 shutdown_connections 2023-03-03T08:43:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.536+0000 7fdaa59fd700 1 -- 172.21.15.7:0/189824253 wait complete. 2023-03-03T08:43:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.537+0000 7fdaa59fd700 1 Processor -- start 2023-03-03T08:43:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.537+0000 7fdaa59fd700 1 -- start start 2023-03-03T08:43:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.537+0000 7fdaa59fd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 0x7fdaa01ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.537+0000 7fdaa59fd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa00ab0c0 0x7fdaa00a7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.537+0000 7fdaa59fd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa0157a40 0x7fdaa0070540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.537+0000 7fdaa59fd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa01edcc0 con 0x7fdaa006a2b0 2023-03-03T08:43:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.537+0000 7fdaa59fd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa01edfa0 con 0x7fdaa00ab0c0 2023-03-03T08:43:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.537+0000 7fdaa59fd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa01e9ee0 con 0x7fdaa0157a40 2023-03-03T08:43:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 0x7fdaa01ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 0x7fdaa01ea6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51500/0 (socket says 172.21.15.7:51500) 2023-03-03T08:43:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9effd700 1 -- 172.21.15.7:0/151332953 learned_addr learned my addr 172.21.15.7:0/151332953 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9f7fe700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa0157a40 0x7fdaa0070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9e7fc700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa00ab0c0 0x7fdaa00a7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9effd700 1 -- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa0157a40 msgr2=0x7fdaa0070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9effd700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa0157a40 0x7fdaa0070540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9effd700 1 -- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa00ab0c0 msgr2=0x7fdaa00a7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9effd700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa00ab0c0 0x7fdaa00a7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9effd700 1 -- 172.21.15.7:0/151332953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda88009550 con 0x7fdaa006a2b0 2023-03-03T08:43:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9f7fe700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa0157a40 0x7fdaa0070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9e7fc700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa00ab0c0 0x7fdaa00a7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda9effd700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 0x7fdaa01ea6e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fda9000e810 tx=0x7fda9000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda97fff700 1 -- 172.21.15.7:0/151332953 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90013610 con 0x7fdaa006a2b0 2023-03-03T08:43:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda97fff700 1 -- 172.21.15.7:0/151332953 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda90013770 con 0x7fdaa006a2b0 2023-03-03T08:43:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fda97fff700 1 -- 172.21.15.7:0/151332953 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90013b20 con 0x7fdaa006a2b0 2023-03-03T08:43:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fdaa59fd700 1 -- 172.21.15.7:0/151332953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa00ab490 con 0x7fdaa006a2b0 2023-03-03T08:43:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.538+0000 7fdaa59fd700 1 -- 172.21.15.7:0/151332953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa01550b0 con 0x7fdaa006a2b0 2023-03-03T08:43:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.539+0000 7fdaa59fd700 1 -- 172.21.15.7:0/151332953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa00c31c0 con 0x7fdaa006a2b0 2023-03-03T08:43:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.540+0000 7fda97fff700 1 -- 172.21.15.7:0/151332953 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda90023070 con 0x7fdaa006a2b0 2023-03-03T08:43:48.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.540+0000 7fda97fff700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda800682f0 0x7fda8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.540+0000 7fda97fff700 1 -- 172.21.15.7:0/151332953 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fda90085ce0 con 0x7fdaa006a2b0 2023-03-03T08:43:48.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.540+0000 7fda9e7fc700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda800682f0 0x7fda8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.541+0000 7fda9e7fc700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda800682f0 0x7fda8006a7a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fda88014040 tx=0x7fda88008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:48.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.545+0000 7fda97fff700 1 -- 172.21.15.7:0/151332953 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda90052190 con 0x7fdaa006a2b0 2023-03-03T08:43:48.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.738+0000 7fdaa59fd700 1 -- 172.21.15.7:0/151332953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdaa00c1ed0 con 0x7fdaa006a2b0 2023-03-03T08:43:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.740+0000 7fda97fff700 1 -- 172.21.15.7:0/151332953 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21533 (secure 0 0 0) 0x7fda900557b0 con 0x7fdaa006a2b0 2023-03-03T08:43:48.742 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:48.743 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:44.973251+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 -- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda800682f0 msgr2=0x7fda8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda800682f0 0x7fda8006a7a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fda88014040 tx=0x7fda88008030 comp rx=0 tx=0).stop 2023-03-03T08:43:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 -- 172.21.15.7:0/151332953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 msgr2=0x7fdaa01ea6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 0x7fdaa01ea6e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fda9000e810 tx=0x7fda9000ebd0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 -- 172.21.15.7:0/151332953 shutdown_connections 2023-03-03T08:43:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaa0157a40 0x7fdaa0070540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda800682f0 0x7fda8006a7a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaa006a2b0 0x7fdaa01ea6e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 --2- 172.21.15.7:0/151332953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaa00ab0c0 0x7fdaa00a7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 -- 172.21.15.7:0/151332953 >> 172.21.15.7:0/151332953 conn(0x7fdaa00a9430 msgr2=0x7fdaa0155dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:48.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 -- 172.21.15.7:0/151332953 shutdown_connections 2023-03-03T08:43:48.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.742+0000 7fdaa59fd700 1 -- 172.21.15.7:0/151332953 wait complete. 2023-03-03T08:43:48.757 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-03T08:43:48.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:43:48.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.878+0000 7fd5e50a2700 1 Processor -- start 2023-03-03T08:43:48.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.879+0000 7fd5e50a2700 1 -- start start 2023-03-03T08:43:48.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.879+0000 7fd5e50a2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd5e006d4b0 0x7fd5e006d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.879+0000 7fd5e50a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e00c1da0 0x7fd5e0139d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.879+0000 7fd5e50a2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5e00c7b90 con 0x7fd5e00c8020 2023-03-03T08:43:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.879+0000 7fd5e50a2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e00c7330 con 0x7fd5e006d4b0 2023-03-03T08:43:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.879+0000 7fd5e50a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e0154420 con 0x7fd5e00c1da0 2023-03-03T08:43:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5df59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e00c1da0 0x7fd5e0139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5ded9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd5e006d4b0 0x7fd5e006d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5de59c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd5e00c8020 0x7fd5e00c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49008/0 (socket says 172.21.15.7:49008) 2023-03-03T08:43:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5de59c700 1 -- 172.21.15.7:0/520863429 learned_addr learned my addr 172.21.15.7:0/520863429 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5ded9d700 1 -- 172.21.15.7:0/520863429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e00c1da0 msgr2=0x7fd5e0139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5ded9d700 1 --2- 172.21.15.7:0/520863429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e00c1da0 0x7fd5e0139d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5ded9d700 1 -- 172.21.15.7:0/520863429 >> v1:172.21.15.143:6789/0 conn(0x7fd5e00c8020 legacy=0x7fd5e00c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5ded9d700 1 -- 172.21.15.7:0/520863429 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e0154640 con 0x7fd5e006d4b0 2023-03-03T08:43:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5df59e700 1 --2- 172.21.15.7:0/520863429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e00c1da0 0x7fd5e0139d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5ded9d700 1 --2- 172.21.15.7:0/520863429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e006d4b0 0x7fd5e006d880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009840 tx=0x7fd5d0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fd096466f31e62a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:48.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5ddd9b700 1 -- 172.21.15.7:0/520863429 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0011830 con 0x7fd5e006d4b0 2023-03-03T08:43:48.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5ddd9b700 1 -- 172.21.15.7:0/520863429 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d0011990 con 0x7fd5e006d4b0 2023-03-03T08:43:48.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.880+0000 7fd5ddd9b700 1 -- 172.21.15.7:0/520863429 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0011d20 con 0x7fd5e006d4b0 2023-03-03T08:43:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.881+0000 7fd5e50a2700 1 -- 172.21.15.7:0/520863429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e006d4b0 msgr2=0x7fd5e006d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.881+0000 7fd5e50a2700 1 --2- 172.21.15.7:0/520863429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e006d4b0 0x7fd5e006d880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009840 tx=0x7fd5d0009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.881+0000 7fd5e50a2700 1 -- 172.21.15.7:0/520863429 shutdown_connections 2023-03-03T08:43:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.881+0000 7fd5e50a2700 1 --2- 172.21.15.7:0/520863429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e00c1da0 0x7fd5e0139d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.881+0000 7fd5e50a2700 1 --2- 172.21.15.7:0/520863429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e006d4b0 0x7fd5e006d880 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.881+0000 7fd5e50a2700 1 -- 172.21.15.7:0/520863429 >> 172.21.15.7:0/520863429 conn(0x7fd5e006c110 msgr2=0x7fd5e006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:48.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.881+0000 7fd5e50a2700 1 -- 172.21.15.7:0/520863429 shutdown_connections 2023-03-03T08:43:48.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.881+0000 7fd5e50a2700 1 -- 172.21.15.7:0/520863429 wait complete. 2023-03-03T08:43:48.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.882+0000 7fd5e50a2700 1 Processor -- start 2023-03-03T08:43:48.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.882+0000 7fd5e50a2700 1 -- start start 2023-03-03T08:43:48.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5e50a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e006d4b0 0x7fd5e01ece50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5e50a2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e00c1da0 0x7fd5e0072200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5e50a2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e00c8020 0x7fd5e0071530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5e50a2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e00abca0 con 0x7fd5e00c1da0 2023-03-03T08:43:48.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5e50a2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e00abe80 con 0x7fd5e00c8020 2023-03-03T08:43:48.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5e50a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e00ace50 con 0x7fd5e006d4b0 2023-03-03T08:43:48.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5de59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e00c1da0 0x7fd5e0072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5de59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e00c1da0 0x7fd5e0072200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51516/0 (socket says 172.21.15.7:51516) 2023-03-03T08:43:48.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5de59c700 1 -- 172.21.15.7:0/1192976307 learned_addr learned my addr 172.21.15.7:0/1192976307 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:48.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5ded9d700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e006d4b0 0x7fd5e01ece50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.883+0000 7fd5df59e700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e00c8020 0x7fd5e0071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5de59c700 1 -- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e006d4b0 msgr2=0x7fd5e01ece50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5de59c700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e006d4b0 0x7fd5e01ece50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5de59c700 1 -- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e00c8020 msgr2=0x7fd5e0071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5de59c700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e00c8020 0x7fd5e0071530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5de59c700 1 -- 172.21.15.7:0/1192976307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d8009a70 con 0x7fd5e00c1da0 2023-03-03T08:43:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5ded9d700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e006d4b0 0x7fd5e01ece50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5de59c700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e00c1da0 0x7fd5e0072200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800c480 tx=0x7fd5c800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5d77fe700 1 -- 172.21.15.7:0/1192976307 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8005d70 con 0x7fd5e00c1da0 2023-03-03T08:43:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5d77fe700 1 -- 172.21.15.7:0/1192976307 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c800ce80 con 0x7fd5e00c1da0 2023-03-03T08:43:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5e50a2700 1 -- 172.21.15.7:0/1192976307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d00094a0 con 0x7fd5e00c1da0 2023-03-03T08:43:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5d77fe700 1 -- 172.21.15.7:0/1192976307 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c80072d0 con 0x7fd5e00c1da0 2023-03-03T08:43:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.884+0000 7fd5e50a2700 1 -- 172.21.15.7:0/1192976307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e00c7820 con 0x7fd5e00c1da0 2023-03-03T08:43:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.885+0000 7fd5e50a2700 1 -- 172.21.15.7:0/1192976307 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e00c7540 con 0x7fd5e00c1da0 2023-03-03T08:43:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.885+0000 7fd5d77fe700 1 -- 172.21.15.7:0/1192976307 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd5c8019030 con 0x7fd5e00c1da0 2023-03-03T08:43:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.886+0000 7fd5d77fe700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5c00682f0 0x7fd5c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.886+0000 7fd5d77fe700 1 -- 172.21.15.7:0/1192976307 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fd5c8085050 con 0x7fd5e00c1da0 2023-03-03T08:43:48.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.886+0000 7fd5ded9d700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5c00682f0 0x7fd5c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:48.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.886+0000 7fd5ded9d700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5c00682f0 0x7fd5c006a7a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0000c00 tx=0x7fd5d001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:48.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:48.890+0000 7fd5d77fe700 1 -- 172.21.15.7:0/1192976307 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd5c804b5e0 con 0x7fd5e00c1da0 2023-03-03T08:43:49.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.078+0000 7fd5e50a2700 1 -- 172.21.15.7:0/1192976307 --> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5e006ddc0 con 0x7fd5c00682f0 2023-03-03T08:43:49.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.093+0000 7fd5d77fe700 1 -- 172.21.15.7:0/1192976307 <== mgr.4102 v2:172.21.15.143:6832/96367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525186 (secure 0 0 0) 0x7fd5e006ddc0 con 0x7fd5c00682f0 2023-03-03T08:43:49.100 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:49.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 -- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5c00682f0 msgr2=0x7fd5c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5c00682f0 0x7fd5c006a7a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0000c00 tx=0x7fd5d001c040 comp rx=0 tx=0).stop 2023-03-03T08:43:49.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 -- 172.21.15.7:0/1192976307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e00c1da0 msgr2=0x7fd5e0072200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e00c1da0 0x7fd5e0072200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800c480 tx=0x7fd5c800c790 comp rx=0 tx=0).stop 2023-03-03T08:43:49.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 -- 172.21.15.7:0/1192976307 shutdown_connections 2023-03-03T08:43:49.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e00c8020 0x7fd5e0071530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e006d4b0 0x7fd5e01ece50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5c00682f0 0x7fd5c006a7a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 --2- 172.21.15.7:0/1192976307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e00c1da0 0x7fd5e0072200 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 -- 172.21.15.7:0/1192976307 >> 172.21.15.7:0/1192976307 conn(0x7fd5e006c110 msgr2=0x7fd5e00c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:49.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 -- 172.21.15.7:0/1192976307 shutdown_connections 2023-03-03T08:43:49.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.097+0000 7fd5e50a2700 1 -- 172.21.15.7:0/1192976307 wait complete. 2023-03-03T08:43:49.105 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-03-03T08:43:49.111 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-03-03T08:43:47.108292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595028,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":248,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":378,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68052,"kb_used_data":3540,"kb_used_omap":0,"kb_used_meta":64448,"kb_avail":754906668,"statfs":{"total":773094113280,"available":773024428032,"internally_reserved":0,"allocated":3624960,"data_stored":2139729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65994752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.557964"},"pg_stats":[{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980166+0000","last_change":"2023-03-03T08:43:43.997091+0000","last_active":"2023-03-03T08:43:44.980166+0000","last_peered":"2023-03-03T08:43:44.980166+0000","last_clean":"2023-03-03T08:43:44.980166+0000","last_became_active":"2023-03-03T08:43:43.996805+0000","last_became_peered":"2023-03-03T08:43:43.996805+0000","last_unstale":"2023-03-03T08:43:44.980166+0000","last_undegraded":"2023-03-03T08:43:44.980166+0000","last_fullsized":"2023-03-03T08:43:44.980166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982505+0000","last_change":"2023-03-03T08:43:34.546828+0000","last_active":"2023-03-03T08:43:43.982505+0000","last_peered":"2023-03-03T08:43:43.982505+0000","last_clean":"2023-03-03T08:43:43.982505+0000","last_became_active":"2023-03-03T08:43:34.546559+0000","last_became_peered":"2023-03-03T08:43:34.546559+0000","last_unstale":"2023-03-03T08:43:43.982505+0000","last_undegraded":"2023-03-03T08:43:43.982505+0000","last_fullsized":"2023-03-03T08:43:43.982505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:21:14.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106061+0000","last_change":"2023-03-03T08:43:43.995073+0000","last_active":"2023-03-03T08:43:45.106061+0000","last_peered":"2023-03-03T08:43:45.106061+0000","last_clean":"2023-03-03T08:43:45.106061+0000","last_became_active":"2023-03-03T08:43:43.994797+0000","last_became_peered":"2023-03-03T08:43:43.994797+0000","last_unstale":"2023-03-03T08:43:45.106061+0000","last_undegraded":"2023-03-03T08:43:45.106061+0000","last_fullsized":"2023-03-03T08:43:45.106061+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:21:52.642936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982397+0000","last_change":"2023-03-03T08:43:34.552931+0000","last_active":"2023-03-03T08:43:43.982397+0000","last_peered":"2023-03-03T08:43:43.982397+0000","last_clean":"2023-03-03T08:43:43.982397+0000","last_became_active":"2023-03-03T08:43:34.552623+0000","last_became_peered":"2023-03-03T08:43:34.552623+0000","last_unstale":"2023-03-03T08:43:43.982397+0000","last_undegraded":"2023-03-03T08:43:43.982397+0000","last_fullsized":"2023-03-03T08:43:43.982397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:07:43.954897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979088+0000","last_change":"2023-03-03T08:43:43.999245+0000","last_active":"2023-03-03T08:43:44.979088+0000","last_peered":"2023-03-03T08:43:44.979088+0000","last_clean":"2023-03-03T08:43:44.979088+0000","last_became_active":"2023-03-03T08:43:43.998592+0000","last_became_peered":"2023-03-03T08:43:43.998592+0000","last_unstale":"2023-03-03T08:43:44.979088+0000","last_undegraded":"2023-03-03T08:43:44.979088+0000","last_fullsized":"2023-03-03T08:43:44.979088+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:23.788724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991054+0000","last_change":"2023-03-03T08:43:34.544408+0000","last_active":"2023-03-03T08:43:43.991054+0000","last_peered":"2023-03-03T08:43:43.991054+0000","last_clean":"2023-03-03T08:43:43.991054+0000","last_became_active":"2023-03-03T08:43:34.543977+0000","last_became_peered":"2023-03-03T08:43:34.543977+0000","last_unstale":"2023-03-03T08:43:43.991054+0000","last_undegraded":"2023-03-03T08:43:43.991054+0000","last_fullsized":"2023-03-03T08:43:43.991054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.074082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105254+0000","last_change":"2023-03-03T08:43:43.994545+0000","last_active":"2023-03-03T08:43:45.105254+0000","last_peered":"2023-03-03T08:43:45.105254+0000","last_clean":"2023-03-03T08:43:45.105254+0000","last_became_active":"2023-03-03T08:43:43.994271+0000","last_became_peered":"2023-03-03T08:43:43.994271+0000","last_unstale":"2023-03-03T08:43:45.105254+0000","last_undegraded":"2023-03-03T08:43:45.105254+0000","last_fullsized":"2023-03-03T08:43:45.105254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:15.925960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989064+0000","last_change":"2023-03-03T08:43:34.538786+0000","last_active":"2023-03-03T08:43:43.989064+0000","last_peered":"2023-03-03T08:43:43.989064+0000","last_clean":"2023-03-03T08:43:43.989064+0000","last_became_active":"2023-03-03T08:43:34.538494+0000","last_became_peered":"2023-03-03T08:43:34.538494+0000","last_unstale":"2023-03-03T08:43:43.989064+0000","last_undegraded":"2023-03-03T08:43:43.989064+0000","last_fullsized":"2023-03-03T08:43:43.989064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:53.838286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105391+0000","last_change":"2023-03-03T08:43:43.993678+0000","last_active":"2023-03-03T08:43:45.105391+0000","last_peered":"2023-03-03T08:43:45.105391+0000","last_clean":"2023-03-03T08:43:45.105391+0000","last_became_active":"2023-03-03T08:43:43.993402+0000","last_became_peered":"2023-03-03T08:43:43.993402+0000","last_unstale":"2023-03-03T08:43:45.105391+0000","last_undegraded":"2023-03-03T08:43:45.105391+0000","last_fullsized":"2023-03-03T08:43:45.105391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:31:00.747216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983479+0000","last_change":"2023-03-03T08:43:34.547197+0000","last_active":"2023-03-03T08:43:43.983479+0000","last_peered":"2023-03-03T08:43:43.983479+0000","last_clean":"2023-03-03T08:43:43.983479+0000","last_became_active":"2023-03-03T08:43:34.546927+0000","last_became_peered":"2023-03-03T08:43:34.546927+0000","last_unstale":"2023-03-03T08:43:43.983479+0000","last_undegraded":"2023-03-03T08:43:43.983479+0000","last_fullsized":"2023-03-03T08:43:43.983479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:21:34.845968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984318+0000","last_change":"2023-03-03T08:43:34.540812+0000","last_active":"2023-03-03T08:43:43.984318+0000","last_peered":"2023-03-03T08:43:43.984318+0000","last_clean":"2023-03-03T08:43:43.984318+0000","last_became_active":"2023-03-03T08:43:34.540549+0000","last_became_peered":"2023-03-03T08:43:34.540549+0000","last_unstale":"2023-03-03T08:43:43.984318+0000","last_undegraded":"2023-03-03T08:43:43.984318+0000","last_fullsized":"2023-03-03T08:43:43.984318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982875+0000","last_change":"2023-03-03T08:43:34.108426+0000","last_active":"2023-03-03T08:43:43.982875+0000","last_peered":"2023-03-03T08:43:43.982875+0000","last_clean":"2023-03-03T08:43:43.982875+0000","last_became_active":"2023-03-03T08:43:34.108009+0000","last_became_peered":"2023-03-03T08:43:34.108009+0000","last_unstale":"2023-03-03T08:43:43.982875+0000","last_undegraded":"2023-03-03T08:43:43.982875+0000","last_fullsized":"2023-03-03T08:43:43.982875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:53.632744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979479+0000","last_change":"2023-03-03T08:43:43.575708+0000","last_active":"2023-03-03T08:43:44.979479+0000","last_peered":"2023-03-03T08:43:44.979479+0000","last_clean":"2023-03-03T08:43:44.979479+0000","last_became_active":"2023-03-03T08:43:43.575417+0000","last_became_peered":"2023-03-03T08:43:43.575417+0000","last_unstale":"2023-03-03T08:43:44.979479+0000","last_undegraded":"2023-03-03T08:43:44.979479+0000","last_fullsized":"2023-03-03T08:43:44.979479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:25.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106018+0000","last_change":"2023-03-03T08:43:43.999274+0000","last_active":"2023-03-03T08:43:45.106018+0000","last_peered":"2023-03-03T08:43:45.106018+0000","last_clean":"2023-03-03T08:43:45.106018+0000","last_became_active":"2023-03-03T08:43:43.998823+0000","last_became_peered":"2023-03-03T08:43:43.998823+0000","last_unstale":"2023-03-03T08:43:45.106018+0000","last_undegraded":"2023-03-03T08:43:45.106018+0000","last_fullsized":"2023-03-03T08:43:45.106018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:48.799261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992918+0000","last_change":"2023-03-03T08:43:34.532482+0000","last_active":"2023-03-03T08:43:43.992918+0000","last_peered":"2023-03-03T08:43:43.992918+0000","last_clean":"2023-03-03T08:43:43.992918+0000","last_became_active":"2023-03-03T08:43:34.531751+0000","last_became_peered":"2023-03-03T08:43:34.531751+0000","last_unstale":"2023-03-03T08:43:43.992918+0000","last_undegraded":"2023-03-03T08:43:43.992918+0000","last_fullsized":"2023-03-03T08:43:43.992918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:47:45.214945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982691+0000","last_change":"2023-03-03T08:43:34.109782+0000","last_active":"2023-03-03T08:43:43.982691+0000","last_peered":"2023-03-03T08:43:43.982691+0000","last_clean":"2023-03-03T08:43:43.982691+0000","last_became_active":"2023-03-03T08:43:34.109513+0000","last_became_peered":"2023-03-03T08:43:34.109513+0000","last_unstale":"2023-03-03T08:43:43.982691+0000","last_undegraded":"2023-03-03T08:43:43.982691+0000","last_fullsized":"2023-03-03T08:43:43.982691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:16.185161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105634+0000","last_change":"2023-03-03T08:43:43.576885+0000","last_active":"2023-03-03T08:43:45.105634+0000","last_peered":"2023-03-03T08:43:45.105634+0000","last_clean":"2023-03-03T08:43:45.105634+0000","last_became_active":"2023-03-03T08:43:43.576577+0000","last_became_peered":"2023-03-03T08:43:43.576577+0000","last_unstale":"2023-03-03T08:43:45.105634+0000","last_undegraded":"2023-03-03T08:43:45.105634+0000","last_fullsized":"2023-03-03T08:43:45.105634+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:26.501956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978988+0000","last_change":"2023-03-03T08:43:43.992020+0000","last_active":"2023-03-03T08:43:44.978988+0000","last_peered":"2023-03-03T08:43:44.978988+0000","last_clean":"2023-03-03T08:43:44.978988+0000","last_became_active":"2023-03-03T08:43:43.991674+0000","last_became_peered":"2023-03-03T08:43:43.991674+0000","last_unstale":"2023-03-03T08:43:44.978988+0000","last_undegraded":"2023-03-03T08:43:44.978988+0000","last_fullsized":"2023-03-03T08:43:44.978988+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:54:30.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985919+0000","last_change":"2023-03-03T08:43:34.535451+0000","last_active":"2023-03-03T08:43:43.985919+0000","last_peered":"2023-03-03T08:43:43.985919+0000","last_clean":"2023-03-03T08:43:43.985919+0000","last_became_active":"2023-03-03T08:43:34.535129+0000","last_became_peered":"2023-03-03T08:43:34.535129+0000","last_unstale":"2023-03-03T08:43:43.985919+0000","last_undegraded":"2023-03-03T08:43:43.985919+0000","last_fullsized":"2023-03-03T08:43:43.985919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:30.645128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981082+0000","last_change":"2023-03-03T08:43:34.117851+0000","last_active":"2023-03-03T08:43:43.981082+0000","last_peered":"2023-03-03T08:43:43.981082+0000","last_clean":"2023-03-03T08:43:43.981082+0000","last_became_active":"2023-03-03T08:43:34.117548+0000","last_became_peered":"2023-03-03T08:43:34.117548+0000","last_unstale":"2023-03-03T08:43:43.981082+0000","last_undegraded":"2023-03-03T08:43:43.981082+0000","last_fullsized":"2023-03-03T08:43:43.981082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:20.039817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106904+0000","last_change":"2023-03-03T08:43:43.569082+0000","last_active":"2023-03-03T08:43:45.106904+0000","last_peered":"2023-03-03T08:43:45.106904+0000","last_clean":"2023-03-03T08:43:45.106904+0000","last_became_active":"2023-03-03T08:43:43.568838+0000","last_became_peered":"2023-03-03T08:43:43.568838+0000","last_unstale":"2023-03-03T08:43:45.106904+0000","last_undegraded":"2023-03-03T08:43:45.106904+0000","last_fullsized":"2023-03-03T08:43:45.106904+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:47.610848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979556+0000","last_change":"2023-03-03T08:43:43.995210+0000","last_active":"2023-03-03T08:43:44.979556+0000","last_peered":"2023-03-03T08:43:44.979556+0000","last_clean":"2023-03-03T08:43:44.979556+0000","last_became_active":"2023-03-03T08:43:43.994899+0000","last_became_peered":"2023-03-03T08:43:43.994899+0000","last_unstale":"2023-03-03T08:43:44.979556+0000","last_undegraded":"2023-03-03T08:43:44.979556+0000","last_fullsized":"2023-03-03T08:43:44.979556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:37:11.974933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985908+0000","last_change":"2023-03-03T08:43:34.547322+0000","last_active":"2023-03-03T08:43:43.985908+0000","last_peered":"2023-03-03T08:43:43.985908+0000","last_clean":"2023-03-03T08:43:43.985908+0000","last_became_active":"2023-03-03T08:43:34.546692+0000","last_became_peered":"2023-03-03T08:43:34.546692+0000","last_unstale":"2023-03-03T08:43:43.985908+0000","last_undegraded":"2023-03-03T08:43:43.985908+0000","last_fullsized":"2023-03-03T08:43:43.985908+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:11:26.366920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982836+0000","last_change":"2023-03-03T08:43:34.110187+0000","last_active":"2023-03-03T08:43:43.982836+0000","last_peered":"2023-03-03T08:43:43.982836+0000","last_clean":"2023-03-03T08:43:43.982836+0000","last_became_active":"2023-03-03T08:43:34.109845+0000","last_became_peered":"2023-03-03T08:43:34.109845+0000","last_unstale":"2023-03-03T08:43:43.982836+0000","last_undegraded":"2023-03-03T08:43:43.982836+0000","last_fullsized":"2023-03-03T08:43:43.982836+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:33.507500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105532+0000","last_change":"2023-03-03T08:43:43.576805+0000","last_active":"2023-03-03T08:43:45.105532+0000","last_peered":"2023-03-03T08:43:45.105532+0000","last_clean":"2023-03-03T08:43:45.105532+0000","last_became_active":"2023-03-03T08:43:43.576241+0000","last_became_peered":"2023-03-03T08:43:43.576241+0000","last_unstale":"2023-03-03T08:43:45.105532+0000","last_undegraded":"2023-03-03T08:43:45.105532+0000","last_fullsized":"2023-03-03T08:43:45.105532+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:34:35.821307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105632+0000","last_change":"2023-03-03T08:43:43.991142+0000","last_active":"2023-03-03T08:43:45.105632+0000","last_peered":"2023-03-03T08:43:45.105632+0000","last_clean":"2023-03-03T08:43:45.105632+0000","last_became_active":"2023-03-03T08:43:43.990868+0000","last_became_peered":"2023-03-03T08:43:43.990868+0000","last_unstale":"2023-03-03T08:43:45.105632+0000","last_undegraded":"2023-03-03T08:43:45.105632+0000","last_fullsized":"2023-03-03T08:43:45.105632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:58.831806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106429+0000","last_change":"2023-03-03T08:43:43.995618+0000","last_active":"2023-03-03T08:43:45.106429+0000","last_peered":"2023-03-03T08:43:45.106429+0000","last_clean":"2023-03-03T08:43:45.106429+0000","last_became_active":"2023-03-03T08:43:43.995350+0000","last_became_peered":"2023-03-03T08:43:43.995350+0000","last_unstale":"2023-03-03T08:43:45.106429+0000","last_undegraded":"2023-03-03T08:43:45.106429+0000","last_fullsized":"2023-03-03T08:43:45.106429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:46.445223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988843+0000","last_change":"2023-03-03T08:43:34.541172+0000","last_active":"2023-03-03T08:43:43.988843+0000","last_peered":"2023-03-03T08:43:43.988843+0000","last_clean":"2023-03-03T08:43:43.988843+0000","last_became_active":"2023-03-03T08:43:34.540739+0000","last_became_peered":"2023-03-03T08:43:34.540739+0000","last_unstale":"2023-03-03T08:43:43.988843+0000","last_undegraded":"2023-03-03T08:43:43.988843+0000","last_fullsized":"2023-03-03T08:43:43.988843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:40.058265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987475+0000","last_change":"2023-03-03T08:43:34.112667+0000","last_active":"2023-03-03T08:43:43.987475+0000","last_peered":"2023-03-03T08:43:43.987475+0000","last_clean":"2023-03-03T08:43:43.987475+0000","last_became_active":"2023-03-03T08:43:34.112386+0000","last_became_peered":"2023-03-03T08:43:34.112386+0000","last_unstale":"2023-03-03T08:43:43.987475+0000","last_undegraded":"2023-03-03T08:43:43.987475+0000","last_fullsized":"2023-03-03T08:43:43.987475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:40.504392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980167+0000","last_change":"2023-03-03T08:43:43.573962+0000","last_active":"2023-03-03T08:43:44.980167+0000","last_peered":"2023-03-03T08:43:44.980167+0000","last_clean":"2023-03-03T08:43:44.980167+0000","last_became_active":"2023-03-03T08:43:43.573200+0000","last_became_peered":"2023-03-03T08:43:43.573200+0000","last_unstale":"2023-03-03T08:43:44.980167+0000","last_undegraded":"2023-03-03T08:43:44.980167+0000","last_fullsized":"2023-03-03T08:43:44.980167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:29.511681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984698+0000","last_change":"2023-03-03T08:43:34.537850+0000","last_active":"2023-03-03T08:43:43.984698+0000","last_peered":"2023-03-03T08:43:43.984698+0000","last_clean":"2023-03-03T08:43:43.984698+0000","last_became_active":"2023-03-03T08:43:34.537550+0000","last_became_peered":"2023-03-03T08:43:34.537550+0000","last_unstale":"2023-03-03T08:43:43.984698+0000","last_undegraded":"2023-03-03T08:43:43.984698+0000","last_fullsized":"2023-03-03T08:43:43.984698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:37:36.893757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990299+0000","last_change":"2023-03-03T08:43:34.109127+0000","last_active":"2023-03-03T08:43:43.990299+0000","last_peered":"2023-03-03T08:43:43.990299+0000","last_clean":"2023-03-03T08:43:43.990299+0000","last_became_active":"2023-03-03T08:43:34.108869+0000","last_became_peered":"2023-03-03T08:43:34.108869+0000","last_unstale":"2023-03-03T08:43:43.990299+0000","last_undegraded":"2023-03-03T08:43:43.990299+0000","last_fullsized":"2023-03-03T08:43:43.990299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:19:00.275620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981298+0000","last_change":"2023-03-03T08:43:29.354627+0000","last_active":"2023-03-03T08:43:43.981298+0000","last_peered":"2023-03-03T08:43:43.981298+0000","last_clean":"2023-03-03T08:43:43.981298+0000","last_became_active":"2023-03-03T08:43:29.354271+0000","last_became_peered":"2023-03-03T08:43:29.354271+0000","last_unstale":"2023-03-03T08:43:43.981298+0000","last_undegraded":"2023-03-03T08:43:43.981298+0000","last_fullsized":"2023-03-03T08:43:43.981298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:05.946053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979432+0000","last_change":"2023-03-03T08:43:43.575567+0000","last_active":"2023-03-03T08:43:44.979432+0000","last_peered":"2023-03-03T08:43:44.979432+0000","last_clean":"2023-03-03T08:43:44.979432+0000","last_became_active":"2023-03-03T08:43:43.575276+0000","last_became_peered":"2023-03-03T08:43:43.575276+0000","last_unstale":"2023-03-03T08:43:44.979432+0000","last_undegraded":"2023-03-03T08:43:44.979432+0000","last_fullsized":"2023-03-03T08:43:44.979432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:12.876790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107250+0000","last_change":"2023-03-03T08:43:43.998276+0000","last_active":"2023-03-03T08:43:45.107250+0000","last_peered":"2023-03-03T08:43:45.107250+0000","last_clean":"2023-03-03T08:43:45.107250+0000","last_became_active":"2023-03-03T08:43:43.996018+0000","last_became_peered":"2023-03-03T08:43:43.996018+0000","last_unstale":"2023-03-03T08:43:45.107250+0000","last_undegraded":"2023-03-03T08:43:45.107250+0000","last_fullsized":"2023-03-03T08:43:45.107250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:08.588278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988844+0000","last_change":"2023-03-03T08:43:34.539018+0000","last_active":"2023-03-03T08:43:43.988844+0000","last_peered":"2023-03-03T08:43:43.988844+0000","last_clean":"2023-03-03T08:43:43.988844+0000","last_became_active":"2023-03-03T08:43:34.538375+0000","last_became_peered":"2023-03-03T08:43:34.538375+0000","last_unstale":"2023-03-03T08:43:43.988844+0000","last_undegraded":"2023-03-03T08:43:43.988844+0000","last_fullsized":"2023-03-03T08:43:43.988844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:29:02.601410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990482+0000","last_change":"2023-03-03T08:43:29.354519+0000","last_active":"2023-03-03T08:43:43.990482+0000","last_peered":"2023-03-03T08:43:43.990482+0000","last_clean":"2023-03-03T08:43:43.990482+0000","last_became_active":"2023-03-03T08:43:29.354213+0000","last_became_peered":"2023-03-03T08:43:29.354213+0000","last_unstale":"2023-03-03T08:43:43.990482+0000","last_undegraded":"2023-03-03T08:43:43.990482+0000","last_fullsized":"2023-03-03T08:43:43.990482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:00.406507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990028+0000","last_change":"2023-03-03T08:43:34.110105+0000","last_active":"2023-03-03T08:43:43.990028+0000","last_peered":"2023-03-03T08:43:43.990028+0000","last_clean":"2023-03-03T08:43:43.990028+0000","last_became_active":"2023-03-03T08:43:34.109736+0000","last_became_peered":"2023-03-03T08:43:34.109736+0000","last_unstale":"2023-03-03T08:43:43.990028+0000","last_undegraded":"2023-03-03T08:43:43.990028+0000","last_fullsized":"2023-03-03T08:43:43.990028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:30.557847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105361+0000","last_change":"2023-03-03T08:43:43.569433+0000","last_active":"2023-03-03T08:43:45.105361+0000","last_peered":"2023-03-03T08:43:45.105361+0000","last_clean":"2023-03-03T08:43:45.105361+0000","last_became_active":"2023-03-03T08:43:43.568711+0000","last_became_peered":"2023-03-03T08:43:43.568711+0000","last_unstale":"2023-03-03T08:43:45.105361+0000","last_undegraded":"2023-03-03T08:43:45.105361+0000","last_fullsized":"2023-03-03T08:43:45.105361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:53.170557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105548+0000","last_change":"2023-03-03T08:43:43.999241+0000","last_active":"2023-03-03T08:43:45.105548+0000","last_peered":"2023-03-03T08:43:45.105548+0000","last_clean":"2023-03-03T08:43:45.105548+0000","last_became_active":"2023-03-03T08:43:43.998992+0000","last_became_peered":"2023-03-03T08:43:43.998992+0000","last_unstale":"2023-03-03T08:43:45.105548+0000","last_undegraded":"2023-03-03T08:43:45.105548+0000","last_fullsized":"2023-03-03T08:43:45.105548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:22.623700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991092+0000","last_change":"2023-03-03T08:43:34.535435+0000","last_active":"2023-03-03T08:43:43.991092+0000","last_peered":"2023-03-03T08:43:43.991092+0000","last_clean":"2023-03-03T08:43:43.991092+0000","last_became_active":"2023-03-03T08:43:34.535123+0000","last_became_peered":"2023-03-03T08:43:34.535123+0000","last_unstale":"2023-03-03T08:43:43.991092+0000","last_undegraded":"2023-03-03T08:43:43.991092+0000","last_fullsized":"2023-03-03T08:43:43.991092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:52:10.622422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990432+0000","last_change":"2023-03-03T08:43:34.107783+0000","last_active":"2023-03-03T08:43:43.990432+0000","last_peered":"2023-03-03T08:43:43.990432+0000","last_clean":"2023-03-03T08:43:43.990432+0000","last_became_active":"2023-03-03T08:43:34.107503+0000","last_became_peered":"2023-03-03T08:43:34.107503+0000","last_unstale":"2023-03-03T08:43:43.990432+0000","last_undegraded":"2023-03-03T08:43:43.990432+0000","last_fullsized":"2023-03-03T08:43:43.990432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:35:42.800814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981038+0000","last_change":"2023-03-03T08:43:29.072304+0000","last_active":"2023-03-03T08:43:43.981038+0000","last_peered":"2023-03-03T08:43:43.981038+0000","last_clean":"2023-03-03T08:43:43.981038+0000","last_became_active":"2023-03-03T08:43:29.071962+0000","last_became_peered":"2023-03-03T08:43:29.071962+0000","last_unstale":"2023-03-03T08:43:43.981038+0000","last_undegraded":"2023-03-03T08:43:43.981038+0000","last_fullsized":"2023-03-03T08:43:43.981038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:40:12.229226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106349+0000","last_change":"2023-03-03T08:43:43.575964+0000","last_active":"2023-03-03T08:43:45.106349+0000","last_peered":"2023-03-03T08:43:45.106349+0000","last_clean":"2023-03-03T08:43:45.106349+0000","last_became_active":"2023-03-03T08:43:43.575697+0000","last_became_peered":"2023-03-03T08:43:43.575697+0000","last_unstale":"2023-03-03T08:43:45.106349+0000","last_undegraded":"2023-03-03T08:43:45.106349+0000","last_fullsized":"2023-03-03T08:43:45.106349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:01.896050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978782+0000","last_change":"2023-03-03T08:43:44.006414+0000","last_active":"2023-03-03T08:43:44.978782+0000","last_peered":"2023-03-03T08:43:44.978782+0000","last_clean":"2023-03-03T08:43:44.978782+0000","last_became_active":"2023-03-03T08:43:44.006132+0000","last_became_peered":"2023-03-03T08:43:44.006132+0000","last_unstale":"2023-03-03T08:43:44.978782+0000","last_undegraded":"2023-03-03T08:43:44.978782+0000","last_fullsized":"2023-03-03T08:43:44.978782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:55.006419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982772+0000","last_change":"2023-03-03T08:43:34.544053+0000","last_active":"2023-03-03T08:43:43.982772+0000","last_peered":"2023-03-03T08:43:43.982772+0000","last_clean":"2023-03-03T08:43:43.982772+0000","last_became_active":"2023-03-03T08:43:34.543523+0000","last_became_peered":"2023-03-03T08:43:34.543523+0000","last_unstale":"2023-03-03T08:43:43.982772+0000","last_undegraded":"2023-03-03T08:43:43.982772+0000","last_fullsized":"2023-03-03T08:43:43.982772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:34:03.662535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989069+0000","last_change":"2023-03-03T08:43:34.104492+0000","last_active":"2023-03-03T08:43:43.989069+0000","last_peered":"2023-03-03T08:43:43.989069+0000","last_clean":"2023-03-03T08:43:43.989069+0000","last_became_active":"2023-03-03T08:43:34.104151+0000","last_became_peered":"2023-03-03T08:43:34.104151+0000","last_unstale":"2023-03-03T08:43:43.989069+0000","last_undegraded":"2023-03-03T08:43:43.989069+0000","last_fullsized":"2023-03-03T08:43:43.989069+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:57.214738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979395+0000","last_change":"2023-03-03T08:43:29.075836+0000","last_active":"2023-03-03T08:43:43.979395+0000","last_peered":"2023-03-03T08:43:43.979395+0000","last_clean":"2023-03-03T08:43:43.979395+0000","last_became_active":"2023-03-03T08:43:29.075564+0000","last_became_peered":"2023-03-03T08:43:29.075564+0000","last_unstale":"2023-03-03T08:43:43.979395+0000","last_undegraded":"2023-03-03T08:43:43.979395+0000","last_fullsized":"2023-03-03T08:43:43.979395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:39.576714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107224+0000","last_change":"2023-03-03T08:43:43.575599+0000","last_active":"2023-03-03T08:43:45.107224+0000","last_peered":"2023-03-03T08:43:45.107224+0000","last_clean":"2023-03-03T08:43:45.107224+0000","last_became_active":"2023-03-03T08:43:43.575318+0000","last_became_peered":"2023-03-03T08:43:43.575318+0000","last_unstale":"2023-03-03T08:43:45.107224+0000","last_undegraded":"2023-03-03T08:43:45.107224+0000","last_fullsized":"2023-03-03T08:43:45.107224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:28.806052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105269+0000","last_change":"2023-03-03T08:43:44.000500+0000","last_active":"2023-03-03T08:43:45.105269+0000","last_peered":"2023-03-03T08:43:45.105269+0000","last_clean":"2023-03-03T08:43:45.105269+0000","last_became_active":"2023-03-03T08:43:43.999560+0000","last_became_peered":"2023-03-03T08:43:43.999560+0000","last_unstale":"2023-03-03T08:43:45.105269+0000","last_undegraded":"2023-03-03T08:43:45.105269+0000","last_fullsized":"2023-03-03T08:43:45.105269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:29:54.240309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981172+0000","last_change":"2023-03-03T08:43:34.550263+0000","last_active":"2023-03-03T08:43:43.981172+0000","last_peered":"2023-03-03T08:43:43.981172+0000","last_clean":"2023-03-03T08:43:43.981172+0000","last_became_active":"2023-03-03T08:43:34.549560+0000","last_became_peered":"2023-03-03T08:43:34.549560+0000","last_unstale":"2023-03-03T08:43:43.981172+0000","last_undegraded":"2023-03-03T08:43:43.981172+0000","last_fullsized":"2023-03-03T08:43:43.981172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:55.874223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987062+0000","last_change":"2023-03-03T08:43:34.110484+0000","last_active":"2023-03-03T08:43:43.987062+0000","last_peered":"2023-03-03T08:43:43.987062+0000","last_clean":"2023-03-03T08:43:43.987062+0000","last_became_active":"2023-03-03T08:43:34.110220+0000","last_became_peered":"2023-03-03T08:43:34.110220+0000","last_unstale":"2023-03-03T08:43:43.987062+0000","last_undegraded":"2023-03-03T08:43:43.987062+0000","last_fullsized":"2023-03-03T08:43:43.987062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:04:59.232537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982439+0000","last_change":"2023-03-03T08:43:23.015460+0000","last_active":"2023-03-03T08:43:43.982439+0000","last_peered":"2023-03-03T08:43:43.982439+0000","last_clean":"2023-03-03T08:43:43.982439+0000","last_became_active":"2023-03-03T08:43:23.014485+0000","last_became_peered":"2023-03-03T08:43:23.014485+0000","last_unstale":"2023-03-03T08:43:43.982439+0000","last_undegraded":"2023-03-03T08:43:43.982439+0000","last_fullsized":"2023-03-03T08:43:43.982439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_clean_scrub_stamp":"2023-03-03T08:43:21.542883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:07.022017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980803+0000","last_change":"2023-03-03T08:43:29.160285+0000","last_active":"2023-03-03T08:43:43.980803+0000","last_peered":"2023-03-03T08:43:43.980803+0000","last_clean":"2023-03-03T08:43:43.980803+0000","last_became_active":"2023-03-03T08:43:29.159943+0000","last_became_peered":"2023-03-03T08:43:29.159943+0000","last_unstale":"2023-03-03T08:43:43.980803+0000","last_undegraded":"2023-03-03T08:43:43.980803+0000","last_fullsized":"2023-03-03T08:43:43.980803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:48:58.997356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107580+0000","last_change":"2023-03-03T08:43:43.575212+0000","last_active":"2023-03-03T08:43:45.107580+0000","last_peered":"2023-03-03T08:43:45.107580+0000","last_clean":"2023-03-03T08:43:45.107580+0000","last_became_active":"2023-03-03T08:43:43.574897+0000","last_became_peered":"2023-03-03T08:43:43.574897+0000","last_unstale":"2023-03-03T08:43:45.107580+0000","last_undegraded":"2023-03-03T08:43:45.107580+0000","last_fullsized":"2023-03-03T08:43:45.107580+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:33:07.674313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107370+0000","last_change":"2023-03-03T08:43:43.992952+0000","last_active":"2023-03-03T08:43:45.107370+0000","last_peered":"2023-03-03T08:43:45.107370+0000","last_clean":"2023-03-03T08:43:45.107370+0000","last_became_active":"2023-03-03T08:43:43.992535+0000","last_became_peered":"2023-03-03T08:43:43.992535+0000","last_unstale":"2023-03-03T08:43:45.107370+0000","last_undegraded":"2023-03-03T08:43:45.107370+0000","last_fullsized":"2023-03-03T08:43:45.107370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:10:41.617430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991335+0000","last_change":"2023-03-03T08:43:34.536419+0000","last_active":"2023-03-03T08:43:43.991335+0000","last_peered":"2023-03-03T08:43:43.991335+0000","last_clean":"2023-03-03T08:43:43.991335+0000","last_became_active":"2023-03-03T08:43:34.536150+0000","last_became_peered":"2023-03-03T08:43:34.536150+0000","last_unstale":"2023-03-03T08:43:43.991335+0000","last_undegraded":"2023-03-03T08:43:43.991335+0000","last_fullsized":"2023-03-03T08:43:43.991335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:22:54.362024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981827+0000","last_change":"2023-03-03T08:43:34.106337+0000","last_active":"2023-03-03T08:43:43.981827+0000","last_peered":"2023-03-03T08:43:43.981827+0000","last_clean":"2023-03-03T08:43:43.981827+0000","last_became_active":"2023-03-03T08:43:34.105753+0000","last_became_peered":"2023-03-03T08:43:34.105753+0000","last_unstale":"2023-03-03T08:43:43.981827+0000","last_undegraded":"2023-03-03T08:43:43.981827+0000","last_fullsized":"2023-03-03T08:43:43.981827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:46.761924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991590+0000","last_change":"2023-03-03T08:43:29.076367+0000","last_active":"2023-03-03T08:43:43.991590+0000","last_peered":"2023-03-03T08:43:43.991590+0000","last_clean":"2023-03-03T08:43:43.991590+0000","last_became_active":"2023-03-03T08:43:29.075982+0000","last_became_peered":"2023-03-03T08:43:29.075982+0000","last_unstale":"2023-03-03T08:43:43.991590+0000","last_undegraded":"2023-03-03T08:43:43.991590+0000","last_fullsized":"2023-03-03T08:43:43.991590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:06.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105779+0000","last_change":"2023-03-03T08:43:43.577670+0000","last_active":"2023-03-03T08:43:45.105779+0000","last_peered":"2023-03-03T08:43:45.105779+0000","last_clean":"2023-03-03T08:43:45.105779+0000","last_became_active":"2023-03-03T08:43:43.577371+0000","last_became_peered":"2023-03-03T08:43:43.577371+0000","last_unstale":"2023-03-03T08:43:45.105779+0000","last_undegraded":"2023-03-03T08:43:45.105779+0000","last_fullsized":"2023-03-03T08:43:45.105779+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:18.377654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108041+0000","last_change":"2023-03-03T08:43:43.993077+0000","last_active":"2023-03-03T08:43:45.108041+0000","last_peered":"2023-03-03T08:43:45.108041+0000","last_clean":"2023-03-03T08:43:45.108041+0000","last_became_active":"2023-03-03T08:43:43.992021+0000","last_became_peered":"2023-03-03T08:43:43.992021+0000","last_unstale":"2023-03-03T08:43:45.108041+0000","last_undegraded":"2023-03-03T08:43:45.108041+0000","last_fullsized":"2023-03-03T08:43:45.108041+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:58.237553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990193+0000","last_change":"2023-03-03T08:43:34.528593+0000","last_active":"2023-03-03T08:43:43.990193+0000","last_peered":"2023-03-03T08:43:43.990193+0000","last_clean":"2023-03-03T08:43:43.990193+0000","last_became_active":"2023-03-03T08:43:34.528260+0000","last_became_peered":"2023-03-03T08:43:34.528260+0000","last_unstale":"2023-03-03T08:43:43.990193+0000","last_undegraded":"2023-03-03T08:43:43.990193+0000","last_fullsized":"2023-03-03T08:43:43.990193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988924+0000","last_change":"2023-03-03T08:43:34.113748+0000","last_active":"2023-03-03T08:43:43.988924+0000","last_peered":"2023-03-03T08:43:43.988924+0000","last_clean":"2023-03-03T08:43:43.988924+0000","last_became_active":"2023-03-03T08:43:34.113190+0000","last_became_peered":"2023-03-03T08:43:34.113190+0000","last_unstale":"2023-03-03T08:43:43.988924+0000","last_undegraded":"2023-03-03T08:43:43.988924+0000","last_fullsized":"2023-03-03T08:43:43.988924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:58:55.343571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980485+0000","last_change":"2023-03-03T08:43:29.107584+0000","last_active":"2023-03-03T08:43:43.980485+0000","last_peered":"2023-03-03T08:43:43.980485+0000","last_clean":"2023-03-03T08:43:43.980485+0000","last_became_active":"2023-03-03T08:43:29.107208+0000","last_became_peered":"2023-03-03T08:43:29.107208+0000","last_unstale":"2023-03-03T08:43:43.980485+0000","last_undegraded":"2023-03-03T08:43:43.980485+0000","last_fullsized":"2023-03-03T08:43:43.980485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:28:24.087122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106004+0000","last_change":"2023-03-03T08:43:43.575742+0000","last_active":"2023-03-03T08:43:45.106004+0000","last_peered":"2023-03-03T08:43:45.106004+0000","last_clean":"2023-03-03T08:43:45.106004+0000","last_became_active":"2023-03-03T08:43:43.575456+0000","last_became_peered":"2023-03-03T08:43:43.575456+0000","last_unstale":"2023-03-03T08:43:45.106004+0000","last_undegraded":"2023-03-03T08:43:45.106004+0000","last_fullsized":"2023-03-03T08:43:45.106004+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:46:17.635306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106635+0000","last_change":"2023-03-03T08:43:43.985961+0000","last_active":"2023-03-03T08:43:45.106635+0000","last_peered":"2023-03-03T08:43:45.106635+0000","last_clean":"2023-03-03T08:43:45.106635+0000","last_became_active":"2023-03-03T08:43:43.985633+0000","last_became_peered":"2023-03-03T08:43:43.985633+0000","last_unstale":"2023-03-03T08:43:45.106635+0000","last_undegraded":"2023-03-03T08:43:45.106635+0000","last_fullsized":"2023-03-03T08:43:45.106635+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:52.517910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989789+0000","last_change":"2023-03-03T08:43:34.526797+0000","last_active":"2023-03-03T08:43:43.989789+0000","last_peered":"2023-03-03T08:43:43.989789+0000","last_clean":"2023-03-03T08:43:43.989789+0000","last_became_active":"2023-03-03T08:43:34.526490+0000","last_became_peered":"2023-03-03T08:43:34.526490+0000","last_unstale":"2023-03-03T08:43:43.989789+0000","last_undegraded":"2023-03-03T08:43:43.989789+0000","last_fullsized":"2023-03-03T08:43:43.989789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:11.808336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986004+0000","last_change":"2023-03-03T08:43:34.104692+0000","last_active":"2023-03-03T08:43:43.986004+0000","last_peered":"2023-03-03T08:43:43.986004+0000","last_clean":"2023-03-03T08:43:43.986004+0000","last_became_active":"2023-03-03T08:43:34.104263+0000","last_became_peered":"2023-03-03T08:43:34.104263+0000","last_unstale":"2023-03-03T08:43:43.986004+0000","last_undegraded":"2023-03-03T08:43:43.986004+0000","last_fullsized":"2023-03-03T08:43:43.986004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:24.537703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983049+0000","last_change":"2023-03-03T08:43:29.107536+0000","last_active":"2023-03-03T08:43:43.983049+0000","last_peered":"2023-03-03T08:43:43.983049+0000","last_clean":"2023-03-03T08:43:43.983049+0000","last_became_active":"2023-03-03T08:43:29.107093+0000","last_became_peered":"2023-03-03T08:43:29.107093+0000","last_unstale":"2023-03-03T08:43:43.983049+0000","last_undegraded":"2023-03-03T08:43:43.983049+0000","last_fullsized":"2023-03-03T08:43:43.983049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:25:47.568904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980025+0000","last_change":"2023-03-03T08:43:43.577221+0000","last_active":"2023-03-03T08:43:44.980025+0000","last_peered":"2023-03-03T08:43:44.980025+0000","last_clean":"2023-03-03T08:43:44.980025+0000","last_became_active":"2023-03-03T08:43:43.576942+0000","last_became_peered":"2023-03-03T08:43:43.576942+0000","last_unstale":"2023-03-03T08:43:44.980025+0000","last_undegraded":"2023-03-03T08:43:44.980025+0000","last_fullsized":"2023-03-03T08:43:44.980025+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:13:09.716078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105102+0000","last_change":"2023-03-03T08:43:44.000318+0000","last_active":"2023-03-03T08:43:45.105102+0000","last_peered":"2023-03-03T08:43:45.105102+0000","last_clean":"2023-03-03T08:43:45.105102+0000","last_became_active":"2023-03-03T08:43:44.000053+0000","last_became_peered":"2023-03-03T08:43:44.000053+0000","last_unstale":"2023-03-03T08:43:45.105102+0000","last_undegraded":"2023-03-03T08:43:45.105102+0000","last_fullsized":"2023-03-03T08:43:45.105102+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:31.059774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994113+0000","last_change":"2023-03-03T08:43:34.540006+0000","last_active":"2023-03-03T08:43:43.994113+0000","last_peered":"2023-03-03T08:43:43.994113+0000","last_clean":"2023-03-03T08:43:43.994113+0000","last_became_active":"2023-03-03T08:43:34.539579+0000","last_became_peered":"2023-03-03T08:43:34.539579+0000","last_unstale":"2023-03-03T08:43:43.994113+0000","last_undegraded":"2023-03-03T08:43:43.994113+0000","last_fullsized":"2023-03-03T08:43:43.994113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988985+0000","last_change":"2023-03-03T08:43:34.109764+0000","last_active":"2023-03-03T08:43:43.988985+0000","last_peered":"2023-03-03T08:43:43.988985+0000","last_clean":"2023-03-03T08:43:43.988985+0000","last_became_active":"2023-03-03T08:43:34.109462+0000","last_became_peered":"2023-03-03T08:43:34.109462+0000","last_unstale":"2023-03-03T08:43:43.988985+0000","last_undegraded":"2023-03-03T08:43:43.988985+0000","last_fullsized":"2023-03-03T08:43:43.988985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:56.399192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106570+0000","last_change":"2023-03-03T08:43:43.576402+0000","last_active":"2023-03-03T08:43:45.106570+0000","last_peered":"2023-03-03T08:43:45.106570+0000","last_clean":"2023-03-03T08:43:45.106570+0000","last_became_active":"2023-03-03T08:43:43.576113+0000","last_became_peered":"2023-03-03T08:43:43.576113+0000","last_unstale":"2023-03-03T08:43:45.106570+0000","last_undegraded":"2023-03-03T08:43:45.106570+0000","last_fullsized":"2023-03-03T08:43:45.106570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:33:26.483688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106437+0000","last_change":"2023-03-03T08:43:44.002074+0000","last_active":"2023-03-03T08:43:45.106437+0000","last_peered":"2023-03-03T08:43:45.106437+0000","last_clean":"2023-03-03T08:43:45.106437+0000","last_became_active":"2023-03-03T08:43:44.001803+0000","last_became_peered":"2023-03-03T08:43:44.001803+0000","last_unstale":"2023-03-03T08:43:45.106437+0000","last_undegraded":"2023-03-03T08:43:45.106437+0000","last_fullsized":"2023-03-03T08:43:45.106437+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:15:16.115135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986358+0000","last_change":"2023-03-03T08:43:34.541858+0000","last_active":"2023-03-03T08:43:43.986358+0000","last_peered":"2023-03-03T08:43:43.986358+0000","last_clean":"2023-03-03T08:43:43.986358+0000","last_became_active":"2023-03-03T08:43:34.541430+0000","last_became_peered":"2023-03-03T08:43:34.541430+0000","last_unstale":"2023-03-03T08:43:43.986358+0000","last_undegraded":"2023-03-03T08:43:43.986358+0000","last_fullsized":"2023-03-03T08:43:43.986358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:29.397784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979830+0000","last_change":"2023-03-03T08:43:34.111335+0000","last_active":"2023-03-03T08:43:43.979830+0000","last_peered":"2023-03-03T08:43:43.979830+0000","last_clean":"2023-03-03T08:43:43.979830+0000","last_became_active":"2023-03-03T08:43:34.110837+0000","last_became_peered":"2023-03-03T08:43:34.110837+0000","last_unstale":"2023-03-03T08:43:43.979830+0000","last_undegraded":"2023-03-03T08:43:43.979830+0000","last_fullsized":"2023-03-03T08:43:43.979830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:03.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106574+0000","last_change":"2023-03-03T08:43:43.577539+0000","last_active":"2023-03-03T08:43:45.106574+0000","last_peered":"2023-03-03T08:43:45.106574+0000","last_clean":"2023-03-03T08:43:45.106574+0000","last_became_active":"2023-03-03T08:43:43.577209+0000","last_became_peered":"2023-03-03T08:43:43.577209+0000","last_unstale":"2023-03-03T08:43:45.106574+0000","last_undegraded":"2023-03-03T08:43:45.106574+0000","last_fullsized":"2023-03-03T08:43:45.106574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:31:01.430495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979296+0000","last_change":"2023-03-03T08:43:43.993565+0000","last_active":"2023-03-03T08:43:44.979296+0000","last_peered":"2023-03-03T08:43:44.979296+0000","last_clean":"2023-03-03T08:43:44.979296+0000","last_became_active":"2023-03-03T08:43:43.992653+0000","last_became_peered":"2023-03-03T08:43:43.992653+0000","last_unstale":"2023-03-03T08:43:44.979296+0000","last_undegraded":"2023-03-03T08:43:44.979296+0000","last_fullsized":"2023-03-03T08:43:44.979296+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:24.295462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981121+0000","last_change":"2023-03-03T08:43:34.550325+0000","last_active":"2023-03-03T08:43:43.981121+0000","last_peered":"2023-03-03T08:43:43.981121+0000","last_clean":"2023-03-03T08:43:43.981121+0000","last_became_active":"2023-03-03T08:43:34.550041+0000","last_became_peered":"2023-03-03T08:43:34.550041+0000","last_unstale":"2023-03-03T08:43:43.981121+0000","last_undegraded":"2023-03-03T08:43:43.981121+0000","last_fullsized":"2023-03-03T08:43:43.981121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:44:34.082003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980457+0000","last_change":"2023-03-03T08:43:34.106777+0000","last_active":"2023-03-03T08:43:43.980457+0000","last_peered":"2023-03-03T08:43:43.980457+0000","last_clean":"2023-03-03T08:43:43.980457+0000","last_became_active":"2023-03-03T08:43:34.106132+0000","last_became_peered":"2023-03-03T08:43:34.106132+0000","last_unstale":"2023-03-03T08:43:43.980457+0000","last_undegraded":"2023-03-03T08:43:43.980457+0000","last_fullsized":"2023-03-03T08:43:43.980457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:55.965987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106224+0000","last_change":"2023-03-03T08:43:43.580312+0000","last_active":"2023-03-03T08:43:45.106224+0000","last_peered":"2023-03-03T08:43:45.106224+0000","last_clean":"2023-03-03T08:43:45.106224+0000","last_became_active":"2023-03-03T08:43:43.580096+0000","last_became_peered":"2023-03-03T08:43:43.580096+0000","last_unstale":"2023-03-03T08:43:45.106224+0000","last_undegraded":"2023-03-03T08:43:45.106224+0000","last_fullsized":"2023-03-03T08:43:45.106224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:06:03.500778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106765+0000","last_change":"2023-03-03T08:43:43.997453+0000","last_active":"2023-03-03T08:43:45.106765+0000","last_peered":"2023-03-03T08:43:45.106765+0000","last_clean":"2023-03-03T08:43:45.106765+0000","last_became_active":"2023-03-03T08:43:43.997090+0000","last_became_peered":"2023-03-03T08:43:43.997090+0000","last_unstale":"2023-03-03T08:43:45.106765+0000","last_undegraded":"2023-03-03T08:43:45.106765+0000","last_fullsized":"2023-03-03T08:43:45.106765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:28:58.322823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985493+0000","last_change":"2023-03-03T08:43:34.542308+0000","last_active":"2023-03-03T08:43:43.985493+0000","last_peered":"2023-03-03T08:43:43.985493+0000","last_clean":"2023-03-03T08:43:43.985493+0000","last_became_active":"2023-03-03T08:43:34.541991+0000","last_became_peered":"2023-03-03T08:43:34.541991+0000","last_unstale":"2023-03-03T08:43:43.985493+0000","last_undegraded":"2023-03-03T08:43:43.985493+0000","last_fullsized":"2023-03-03T08:43:43.985493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:06:21.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992335+0000","last_change":"2023-03-03T08:43:34.109698+0000","last_active":"2023-03-03T08:43:43.992335+0000","last_peered":"2023-03-03T08:43:43.992335+0000","last_clean":"2023-03-03T08:43:43.992335+0000","last_became_active":"2023-03-03T08:43:34.109423+0000","last_became_peered":"2023-03-03T08:43:34.109423+0000","last_unstale":"2023-03-03T08:43:43.992335+0000","last_undegraded":"2023-03-03T08:43:43.992335+0000","last_fullsized":"2023-03-03T08:43:43.992335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:49.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105911+0000","last_change":"2023-03-03T08:43:43.573925+0000","last_active":"2023-03-03T08:43:45.105911+0000","last_peered":"2023-03-03T08:43:45.105911+0000","last_clean":"2023-03-03T08:43:45.105911+0000","last_became_active":"2023-03-03T08:43:43.573474+0000","last_became_peered":"2023-03-03T08:43:43.573474+0000","last_unstale":"2023-03-03T08:43:45.105911+0000","last_undegraded":"2023-03-03T08:43:45.105911+0000","last_fullsized":"2023-03-03T08:43:45.105911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:47.858726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107653+0000","last_change":"2023-03-03T08:43:43.995602+0000","last_active":"2023-03-03T08:43:45.107653+0000","last_peered":"2023-03-03T08:43:45.107653+0000","last_clean":"2023-03-03T08:43:45.107653+0000","last_became_active":"2023-03-03T08:43:43.995404+0000","last_became_peered":"2023-03-03T08:43:43.995404+0000","last_unstale":"2023-03-03T08:43:45.107653+0000","last_undegraded":"2023-03-03T08:43:45.107653+0000","last_fullsized":"2023-03-03T08:43:45.107653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:37.212119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982423+0000","last_change":"2023-03-03T08:43:34.548304+0000","last_active":"2023-03-03T08:43:43.982423+0000","last_peered":"2023-03-03T08:43:43.982423+0000","last_clean":"2023-03-03T08:43:43.982423+0000","last_became_active":"2023-03-03T08:43:34.547960+0000","last_became_peered":"2023-03-03T08:43:34.547960+0000","last_unstale":"2023-03-03T08:43:43.982423+0000","last_undegraded":"2023-03-03T08:43:43.982423+0000","last_fullsized":"2023-03-03T08:43:43.982423+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:04.907753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985574+0000","last_change":"2023-03-03T08:43:34.109372+0000","last_active":"2023-03-03T08:43:43.985574+0000","last_peered":"2023-03-03T08:43:43.985574+0000","last_clean":"2023-03-03T08:43:43.985574+0000","last_became_active":"2023-03-03T08:43:34.109023+0000","last_became_peered":"2023-03-03T08:43:34.109023+0000","last_unstale":"2023-03-03T08:43:43.985574+0000","last_undegraded":"2023-03-03T08:43:43.985574+0000","last_fullsized":"2023-03-03T08:43:43.985574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:51.374188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978866+0000","last_change":"2023-03-03T08:43:43.572240+0000","last_active":"2023-03-03T08:43:44.978866+0000","last_peered":"2023-03-03T08:43:44.978866+0000","last_clean":"2023-03-03T08:43:44.978866+0000","last_became_active":"2023-03-03T08:43:43.571753+0000","last_became_peered":"2023-03-03T08:43:43.571753+0000","last_unstale":"2023-03-03T08:43:44.978866+0000","last_undegraded":"2023-03-03T08:43:44.978866+0000","last_fullsized":"2023-03-03T08:43:44.978866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:02:30.521667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979027+0000","last_change":"2023-03-03T08:43:43.995018+0000","last_active":"2023-03-03T08:43:44.979027+0000","last_peered":"2023-03-03T08:43:44.979027+0000","last_clean":"2023-03-03T08:43:44.979027+0000","last_became_active":"2023-03-03T08:43:43.994690+0000","last_became_peered":"2023-03-03T08:43:43.994690+0000","last_unstale":"2023-03-03T08:43:44.979027+0000","last_undegraded":"2023-03-03T08:43:44.979027+0000","last_fullsized":"2023-03-03T08:43:44.979027+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:06:13.979929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986022+0000","last_change":"2023-03-03T08:43:34.538817+0000","last_active":"2023-03-03T08:43:43.986022+0000","last_peered":"2023-03-03T08:43:43.986022+0000","last_clean":"2023-03-03T08:43:43.986022+0000","last_became_active":"2023-03-03T08:43:34.538506+0000","last_became_peered":"2023-03-03T08:43:34.538506+0000","last_unstale":"2023-03-03T08:43:43.986022+0000","last_undegraded":"2023-03-03T08:43:43.986022+0000","last_fullsized":"2023-03-03T08:43:43.986022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980930+0000","last_change":"2023-03-03T08:43:34.117734+0000","last_active":"2023-03-03T08:43:43.980930+0000","last_peered":"2023-03-03T08:43:43.980930+0000","last_clean":"2023-03-03T08:43:43.980930+0000","last_became_active":"2023-03-03T08:43:34.117454+0000","last_became_peered":"2023-03-03T08:43:34.117454+0000","last_unstale":"2023-03-03T08:43:43.980930+0000","last_undegraded":"2023-03-03T08:43:43.980930+0000","last_fullsized":"2023-03-03T08:43:43.980930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:42:27.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106434+0000","last_change":"2023-03-03T08:43:43.574955+0000","last_active":"2023-03-03T08:43:45.106434+0000","last_peered":"2023-03-03T08:43:45.106434+0000","last_clean":"2023-03-03T08:43:45.106434+0000","last_became_active":"2023-03-03T08:43:43.574677+0000","last_became_peered":"2023-03-03T08:43:43.574677+0000","last_unstale":"2023-03-03T08:43:45.106434+0000","last_undegraded":"2023-03-03T08:43:45.106434+0000","last_fullsized":"2023-03-03T08:43:45.106434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:12:19.538291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.104954+0000","last_change":"2023-03-03T08:43:43.994888+0000","last_active":"2023-03-03T08:43:45.104954+0000","last_peered":"2023-03-03T08:43:45.104954+0000","last_clean":"2023-03-03T08:43:45.104954+0000","last_became_active":"2023-03-03T08:43:43.994630+0000","last_became_peered":"2023-03-03T08:43:43.994630+0000","last_unstale":"2023-03-03T08:43:45.104954+0000","last_undegraded":"2023-03-03T08:43:45.104954+0000","last_fullsized":"2023-03-03T08:43:45.104954+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:15:18.412569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982579+0000","last_change":"2023-03-03T08:43:34.548010+0000","last_active":"2023-03-03T08:43:43.982579+0000","last_peered":"2023-03-03T08:43:43.982579+0000","last_clean":"2023-03-03T08:43:43.982579+0000","last_became_active":"2023-03-03T08:43:34.547662+0000","last_became_peered":"2023-03-03T08:43:34.547662+0000","last_unstale":"2023-03-03T08:43:43.982579+0000","last_undegraded":"2023-03-03T08:43:43.982579+0000","last_fullsized":"2023-03-03T08:43:43.982579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:52.200064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986286+0000","last_change":"2023-03-03T08:43:34.109890+0000","last_active":"2023-03-03T08:43:43.986286+0000","last_peered":"2023-03-03T08:43:43.986286+0000","last_clean":"2023-03-03T08:43:43.986286+0000","last_became_active":"2023-03-03T08:43:34.109583+0000","last_became_peered":"2023-03-03T08:43:34.109583+0000","last_unstale":"2023-03-03T08:43:43.986286+0000","last_undegraded":"2023-03-03T08:43:43.986286+0000","last_fullsized":"2023-03-03T08:43:43.986286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:27:34.142294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978233+0000","last_change":"2023-03-03T08:43:43.575945+0000","last_active":"2023-03-03T08:43:44.978233+0000","last_peered":"2023-03-03T08:43:44.978233+0000","last_clean":"2023-03-03T08:43:44.978233+0000","last_became_active":"2023-03-03T08:43:43.575675+0000","last_became_peered":"2023-03-03T08:43:43.575675+0000","last_unstale":"2023-03-03T08:43:44.978233+0000","last_undegraded":"2023-03-03T08:43:44.978233+0000","last_fullsized":"2023-03-03T08:43:44.978233+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:48.730805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980103+0000","last_change":"2023-03-03T08:43:43.998435+0000","last_active":"2023-03-03T08:43:44.980103+0000","last_peered":"2023-03-03T08:43:44.980103+0000","last_clean":"2023-03-03T08:43:44.980103+0000","last_became_active":"2023-03-03T08:43:43.996508+0000","last_became_peered":"2023-03-03T08:43:43.996508+0000","last_unstale":"2023-03-03T08:43:44.980103+0000","last_undegraded":"2023-03-03T08:43:44.980103+0000","last_fullsized":"2023-03-03T08:43:44.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:31:36.310851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979477+0000","last_change":"2023-03-03T08:43:34.540196+0000","last_active":"2023-03-03T08:43:43.979477+0000","last_peered":"2023-03-03T08:43:43.979477+0000","last_clean":"2023-03-03T08:43:43.979477+0000","last_became_active":"2023-03-03T08:43:34.539931+0000","last_became_peered":"2023-03-03T08:43:34.539931+0000","last_unstale":"2023-03-03T08:43:43.979477+0000","last_undegraded":"2023-03-03T08:43:43.979477+0000","last_fullsized":"2023-03-03T08:43:43.979477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:20.012997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990881+0000","last_change":"2023-03-03T08:43:34.110464+0000","last_active":"2023-03-03T08:43:43.990881+0000","last_peered":"2023-03-03T08:43:43.990881+0000","last_clean":"2023-03-03T08:43:43.990881+0000","last_became_active":"2023-03-03T08:43:34.110165+0000","last_became_peered":"2023-03-03T08:43:34.110165+0000","last_unstale":"2023-03-03T08:43:43.990881+0000","last_undegraded":"2023-03-03T08:43:43.990881+0000","last_fullsized":"2023-03-03T08:43:43.990881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:54.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106139+0000","last_change":"2023-03-03T08:43:43.578298+0000","last_active":"2023-03-03T08:43:45.106139+0000","last_peered":"2023-03-03T08:43:45.106139+0000","last_clean":"2023-03-03T08:43:45.106139+0000","last_became_active":"2023-03-03T08:43:43.578024+0000","last_became_peered":"2023-03-03T08:43:43.578024+0000","last_unstale":"2023-03-03T08:43:45.106139+0000","last_undegraded":"2023-03-03T08:43:45.106139+0000","last_fullsized":"2023-03-03T08:43:45.106139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:00:44.633716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106526+0000","last_change":"2023-03-03T08:43:43.992906+0000","last_active":"2023-03-03T08:43:45.106526+0000","last_peered":"2023-03-03T08:43:45.106526+0000","last_clean":"2023-03-03T08:43:45.106526+0000","last_became_active":"2023-03-03T08:43:43.991823+0000","last_became_peered":"2023-03-03T08:43:43.991823+0000","last_unstale":"2023-03-03T08:43:45.106526+0000","last_undegraded":"2023-03-03T08:43:45.106526+0000","last_fullsized":"2023-03-03T08:43:45.106526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:34.431216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.553597+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.553322+0000","last_became_peered":"2023-03-03T08:43:34.553322+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:24:17.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983377+0000","last_change":"2023-03-03T08:43:34.108123+0000","last_active":"2023-03-03T08:43:43.983377+0000","last_peered":"2023-03-03T08:43:43.983377+0000","last_clean":"2023-03-03T08:43:43.983377+0000","last_became_active":"2023-03-03T08:43:34.107757+0000","last_became_peered":"2023-03-03T08:43:34.107757+0000","last_unstale":"2023-03-03T08:43:43.983377+0000","last_undegraded":"2023-03-03T08:43:43.983377+0000","last_fullsized":"2023-03-03T08:43:43.983377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:09.488930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106648+0000","last_change":"2023-03-03T08:43:43.573369+0000","last_active":"2023-03-03T08:43:45.106648+0000","last_peered":"2023-03-03T08:43:45.106648+0000","last_clean":"2023-03-03T08:43:45.106648+0000","last_became_active":"2023-03-03T08:43:43.573069+0000","last_became_peered":"2023-03-03T08:43:43.573069+0000","last_unstale":"2023-03-03T08:43:45.106648+0000","last_undegraded":"2023-03-03T08:43:45.106648+0000","last_fullsized":"2023-03-03T08:43:45.106648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:09.574926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107410+0000","last_change":"2023-03-03T08:43:43.995725+0000","last_active":"2023-03-03T08:43:45.107410+0000","last_peered":"2023-03-03T08:43:45.107410+0000","last_clean":"2023-03-03T08:43:45.107410+0000","last_became_active":"2023-03-03T08:43:43.993875+0000","last_became_peered":"2023-03-03T08:43:43.993875+0000","last_unstale":"2023-03-03T08:43:45.107410+0000","last_undegraded":"2023-03-03T08:43:45.107410+0000","last_fullsized":"2023-03-03T08:43:45.107410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:38.439539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982673+0000","last_change":"2023-03-03T08:43:34.546067+0000","last_active":"2023-03-03T08:43:43.982673+0000","last_peered":"2023-03-03T08:43:43.982673+0000","last_clean":"2023-03-03T08:43:43.982673+0000","last_became_active":"2023-03-03T08:43:34.543786+0000","last_became_peered":"2023-03-03T08:43:34.543786+0000","last_unstale":"2023-03-03T08:43:43.982673+0000","last_undegraded":"2023-03-03T08:43:43.982673+0000","last_fullsized":"2023-03-03T08:43:43.982673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:32.922812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.978650+0000","last_change":"2023-03-03T08:43:34.115792+0000","last_active":"2023-03-03T08:43:43.978650+0000","last_peered":"2023-03-03T08:43:43.978650+0000","last_clean":"2023-03-03T08:43:43.978650+0000","last_became_active":"2023-03-03T08:43:34.115431+0000","last_became_peered":"2023-03-03T08:43:34.115431+0000","last_unstale":"2023-03-03T08:43:43.978650+0000","last_undegraded":"2023-03-03T08:43:43.978650+0000","last_fullsized":"2023-03-03T08:43:43.978650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:15.484022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980510+0000","last_change":"2023-03-03T08:43:43.579691+0000","last_active":"2023-03-03T08:43:44.980510+0000","last_peered":"2023-03-03T08:43:44.980510+0000","last_clean":"2023-03-03T08:43:44.980510+0000","last_became_active":"2023-03-03T08:43:43.579425+0000","last_became_peered":"2023-03-03T08:43:43.579425+0000","last_unstale":"2023-03-03T08:43:44.980510+0000","last_undegraded":"2023-03-03T08:43:44.980510+0000","last_fullsized":"2023-03-03T08:43:44.980510+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:08.896591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979907+0000","last_change":"2023-03-03T08:43:44.006872+0000","last_active":"2023-03-03T08:43:44.979907+0000","last_peered":"2023-03-03T08:43:44.979907+0000","last_clean":"2023-03-03T08:43:44.979907+0000","last_became_active":"2023-03-03T08:43:44.006283+0000","last_became_peered":"2023-03-03T08:43:44.006283+0000","last_unstale":"2023-03-03T08:43:44.979907+0000","last_undegraded":"2023-03-03T08:43:44.979907+0000","last_fullsized":"2023-03-03T08:43:44.979907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:45.439399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981214+0000","last_change":"2023-03-03T08:43:34.532656+0000","last_active":"2023-03-03T08:43:43.981214+0000","last_peered":"2023-03-03T08:43:43.981214+0000","last_clean":"2023-03-03T08:43:43.981214+0000","last_became_active":"2023-03-03T08:43:34.532297+0000","last_became_peered":"2023-03-03T08:43:34.532297+0000","last_unstale":"2023-03-03T08:43:43.981214+0000","last_undegraded":"2023-03-03T08:43:43.981214+0000","last_fullsized":"2023-03-03T08:43:43.981214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:36.347740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989578+0000","last_change":"2023-03-03T08:43:34.112905+0000","last_active":"2023-03-03T08:43:43.989578+0000","last_peered":"2023-03-03T08:43:43.989578+0000","last_clean":"2023-03-03T08:43:43.989578+0000","last_became_active":"2023-03-03T08:43:34.112601+0000","last_became_peered":"2023-03-03T08:43:34.112601+0000","last_unstale":"2023-03-03T08:43:43.989578+0000","last_undegraded":"2023-03-03T08:43:43.989578+0000","last_fullsized":"2023-03-03T08:43:43.989578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:01.165783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"26'6","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980517+0000","last_change":"2023-03-03T08:43:43.577882+0000","last_active":"2023-03-03T08:43:44.980517+0000","last_peered":"2023-03-03T08:43:44.980517+0000","last_clean":"2023-03-03T08:43:44.980517+0000","last_became_active":"2023-03-03T08:43:43.575625+0000","last_became_peered":"2023-03-03T08:43:43.575625+0000","last_unstale":"2023-03-03T08:43:44.980517+0000","last_undegraded":"2023-03-03T08:43:44.980517+0000","last_fullsized":"2023-03-03T08:43:44.980517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:22:25.919413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106494+0000","last_change":"2023-03-03T08:43:43.994846+0000","last_active":"2023-03-03T08:43:45.106494+0000","last_peered":"2023-03-03T08:43:45.106494+0000","last_clean":"2023-03-03T08:43:45.106494+0000","last_became_active":"2023-03-03T08:43:43.994652+0000","last_became_peered":"2023-03-03T08:43:43.994652+0000","last_unstale":"2023-03-03T08:43:45.106494+0000","last_undegraded":"2023-03-03T08:43:45.106494+0000","last_fullsized":"2023-03-03T08:43:45.106494+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:54:29.238192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990352+0000","last_change":"2023-03-03T08:43:34.539790+0000","last_active":"2023-03-03T08:43:43.990352+0000","last_peered":"2023-03-03T08:43:43.990352+0000","last_clean":"2023-03-03T08:43:43.990352+0000","last_became_active":"2023-03-03T08:43:34.539443+0000","last_became_peered":"2023-03-03T08:43:34.539443+0000","last_unstale":"2023-03-03T08:43:43.990352+0000","last_undegraded":"2023-03-03T08:43:43.990352+0000","last_fullsized":"2023-03-03T08:43:43.990352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:25.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980897+0000","last_change":"2023-03-03T08:43:34.112941+0000","last_active":"2023-03-03T08:43:43.980897+0000","last_peered":"2023-03-03T08:43:43.980897+0000","last_clean":"2023-03-03T08:43:43.980897+0000","last_became_active":"2023-03-03T08:43:34.112646+0000","last_became_peered":"2023-03-03T08:43:34.112646+0000","last_unstale":"2023-03-03T08:43:43.980897+0000","last_undegraded":"2023-03-03T08:43:43.980897+0000","last_fullsized":"2023-03-03T08:43:43.980897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:32.767710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107854+0000","last_change":"2023-03-03T08:43:43.576421+0000","last_active":"2023-03-03T08:43:45.107854+0000","last_peered":"2023-03-03T08:43:45.107854+0000","last_clean":"2023-03-03T08:43:45.107854+0000","last_became_active":"2023-03-03T08:43:43.576086+0000","last_became_peered":"2023-03-03T08:43:43.576086+0000","last_unstale":"2023-03-03T08:43:45.107854+0000","last_undegraded":"2023-03-03T08:43:45.107854+0000","last_fullsized":"2023-03-03T08:43:45.107854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:57:17.814222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980193+0000","last_change":"2023-03-03T08:43:43.995591+0000","last_active":"2023-03-03T08:43:44.980193+0000","last_peered":"2023-03-03T08:43:44.980193+0000","last_clean":"2023-03-03T08:43:44.980193+0000","last_became_active":"2023-03-03T08:43:43.995338+0000","last_became_peered":"2023-03-03T08:43:43.995338+0000","last_unstale":"2023-03-03T08:43:44.980193+0000","last_undegraded":"2023-03-03T08:43:44.980193+0000","last_fullsized":"2023-03-03T08:43:44.980193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:17:56.617398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985713+0000","last_change":"2023-03-03T08:43:34.537857+0000","last_active":"2023-03-03T08:43:43.985713+0000","last_peered":"2023-03-03T08:43:43.985713+0000","last_clean":"2023-03-03T08:43:43.985713+0000","last_became_active":"2023-03-03T08:43:34.537590+0000","last_became_peered":"2023-03-03T08:43:34.537590+0000","last_unstale":"2023-03-03T08:43:43.985713+0000","last_undegraded":"2023-03-03T08:43:43.985713+0000","last_fullsized":"2023-03-03T08:43:43.985713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:52.612521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980688+0000","last_change":"2023-03-03T08:43:34.117609+0000","last_active":"2023-03-03T08:43:43.980688+0000","last_peered":"2023-03-03T08:43:43.980688+0000","last_clean":"2023-03-03T08:43:43.980688+0000","last_became_active":"2023-03-03T08:43:34.117271+0000","last_became_peered":"2023-03-03T08:43:34.117271+0000","last_unstale":"2023-03-03T08:43:43.980688+0000","last_undegraded":"2023-03-03T08:43:43.980688+0000","last_fullsized":"2023-03-03T08:43:43.980688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:20.259259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978772+0000","last_change":"2023-03-03T08:43:43.580489+0000","last_active":"2023-03-03T08:43:44.978772+0000","last_peered":"2023-03-03T08:43:44.978772+0000","last_clean":"2023-03-03T08:43:44.978772+0000","last_became_active":"2023-03-03T08:43:43.580207+0000","last_became_peered":"2023-03-03T08:43:43.580207+0000","last_unstale":"2023-03-03T08:43:44.978772+0000","last_undegraded":"2023-03-03T08:43:44.978772+0000","last_fullsized":"2023-03-03T08:43:44.978772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:58:05.519074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978633+0000","last_change":"2023-03-03T08:43:43.993832+0000","last_active":"2023-03-03T08:43:44.978633+0000","last_peered":"2023-03-03T08:43:44.978633+0000","last_clean":"2023-03-03T08:43:44.978633+0000","last_became_active":"2023-03-03T08:43:43.993524+0000","last_became_peered":"2023-03-03T08:43:43.993524+0000","last_unstale":"2023-03-03T08:43:44.978633+0000","last_undegraded":"2023-03-03T08:43:44.978633+0000","last_fullsized":"2023-03-03T08:43:44.978633+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:23:55.761785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987159+0000","last_change":"2023-03-03T08:43:34.544694+0000","last_active":"2023-03-03T08:43:43.987159+0000","last_peered":"2023-03-03T08:43:43.987159+0000","last_clean":"2023-03-03T08:43:43.987159+0000","last_became_active":"2023-03-03T08:43:34.544399+0000","last_became_peered":"2023-03-03T08:43:34.544399+0000","last_unstale":"2023-03-03T08:43:43.987159+0000","last_undegraded":"2023-03-03T08:43:43.987159+0000","last_fullsized":"2023-03-03T08:43:43.987159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986196+0000","last_change":"2023-03-03T08:43:34.110810+0000","last_active":"2023-03-03T08:43:43.986196+0000","last_peered":"2023-03-03T08:43:43.986196+0000","last_clean":"2023-03-03T08:43:43.986196+0000","last_became_active":"2023-03-03T08:43:34.110483+0000","last_became_peered":"2023-03-03T08:43:34.110483+0000","last_unstale":"2023-03-03T08:43:43.986196+0000","last_undegraded":"2023-03-03T08:43:43.986196+0000","last_fullsized":"2023-03-03T08:43:43.986196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:06.970615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979989+0000","last_change":"2023-03-03T08:43:43.576669+0000","last_active":"2023-03-03T08:43:44.979989+0000","last_peered":"2023-03-03T08:43:44.979989+0000","last_clean":"2023-03-03T08:43:44.979989+0000","last_became_active":"2023-03-03T08:43:43.573885+0000","last_became_peered":"2023-03-03T08:43:43.573885+0000","last_unstale":"2023-03-03T08:43:44.979989+0000","last_undegraded":"2023-03-03T08:43:44.979989+0000","last_fullsized":"2023-03-03T08:43:44.979989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:56:12.054118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979442+0000","last_change":"2023-03-03T08:43:43.996790+0000","last_active":"2023-03-03T08:43:44.979442+0000","last_peered":"2023-03-03T08:43:44.979442+0000","last_clean":"2023-03-03T08:43:44.979442+0000","last_became_active":"2023-03-03T08:43:43.996366+0000","last_became_peered":"2023-03-03T08:43:43.996366+0000","last_unstale":"2023-03-03T08:43:44.979442+0000","last_undegraded":"2023-03-03T08:43:44.979442+0000","last_fullsized":"2023-03-03T08:43:44.979442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:01.662221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988553+0000","last_change":"2023-03-03T08:43:34.538703+0000","last_active":"2023-03-03T08:43:43.988553+0000","last_peered":"2023-03-03T08:43:43.988553+0000","last_clean":"2023-03-03T08:43:43.988553+0000","last_became_active":"2023-03-03T08:43:34.538418+0000","last_became_peered":"2023-03-03T08:43:34.538418+0000","last_unstale":"2023-03-03T08:43:43.988553+0000","last_undegraded":"2023-03-03T08:43:43.988553+0000","last_fullsized":"2023-03-03T08:43:43.988553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:02.795203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986185+0000","last_change":"2023-03-03T08:43:34.113168+0000","last_active":"2023-03-03T08:43:43.986185+0000","last_peered":"2023-03-03T08:43:43.986185+0000","last_clean":"2023-03-03T08:43:43.986185+0000","last_became_active":"2023-03-03T08:43:34.112731+0000","last_became_peered":"2023-03-03T08:43:34.112731+0000","last_unstale":"2023-03-03T08:43:43.986185+0000","last_undegraded":"2023-03-03T08:43:43.986185+0000","last_fullsized":"2023-03-03T08:43:43.986185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:57:59.463191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106795+0000","last_change":"2023-03-03T08:43:43.577965+0000","last_active":"2023-03-03T08:43:45.106795+0000","last_peered":"2023-03-03T08:43:45.106795+0000","last_clean":"2023-03-03T08:43:45.106795+0000","last_became_active":"2023-03-03T08:43:43.577587+0000","last_became_peered":"2023-03-03T08:43:43.577587+0000","last_unstale":"2023-03-03T08:43:45.106795+0000","last_undegraded":"2023-03-03T08:43:45.106795+0000","last_fullsized":"2023-03-03T08:43:45.106795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:24.196773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105366+0000","last_change":"2023-03-03T08:43:43.998210+0000","last_active":"2023-03-03T08:43:45.105366+0000","last_peered":"2023-03-03T08:43:45.105366+0000","last_clean":"2023-03-03T08:43:45.105366+0000","last_became_active":"2023-03-03T08:43:43.997845+0000","last_became_peered":"2023-03-03T08:43:43.997845+0000","last_unstale":"2023-03-03T08:43:45.105366+0000","last_undegraded":"2023-03-03T08:43:45.105366+0000","last_fullsized":"2023-03-03T08:43:45.105366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:57:38.226245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991762+0000","last_change":"2023-03-03T08:43:34.528573+0000","last_active":"2023-03-03T08:43:43.991762+0000","last_peered":"2023-03-03T08:43:43.991762+0000","last_clean":"2023-03-03T08:43:43.991762+0000","last_became_active":"2023-03-03T08:43:34.528278+0000","last_became_peered":"2023-03-03T08:43:34.528278+0000","last_unstale":"2023-03-03T08:43:43.991762+0000","last_undegraded":"2023-03-03T08:43:43.991762+0000","last_fullsized":"2023-03-03T08:43:43.991762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:51.468323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.110802+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.110519+0000","last_became_peered":"2023-03-03T08:43:34.110519+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:02:18.355043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105602+0000","last_change":"2023-03-03T08:43:43.573655+0000","last_active":"2023-03-03T08:43:45.105602+0000","last_peered":"2023-03-03T08:43:45.105602+0000","last_clean":"2023-03-03T08:43:45.105602+0000","last_became_active":"2023-03-03T08:43:43.573162+0000","last_became_peered":"2023-03-03T08:43:43.573162+0000","last_unstale":"2023-03-03T08:43:45.105602+0000","last_undegraded":"2023-03-03T08:43:45.105602+0000","last_fullsized":"2023-03-03T08:43:45.105602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:59:04.568700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106617+0000","last_change":"2023-03-03T08:43:43.993431+0000","last_active":"2023-03-03T08:43:45.106617+0000","last_peered":"2023-03-03T08:43:45.106617+0000","last_clean":"2023-03-03T08:43:45.106617+0000","last_became_active":"2023-03-03T08:43:43.993166+0000","last_became_peered":"2023-03-03T08:43:43.993166+0000","last_unstale":"2023-03-03T08:43:45.106617+0000","last_undegraded":"2023-03-03T08:43:45.106617+0000","last_fullsized":"2023-03-03T08:43:45.106617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:04:52.880667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979475+0000","last_change":"2023-03-03T08:43:43.992992+0000","last_active":"2023-03-03T08:43:44.979475+0000","last_peered":"2023-03-03T08:43:44.979475+0000","last_clean":"2023-03-03T08:43:44.979475+0000","last_became_active":"2023-03-03T08:43:43.992664+0000","last_became_peered":"2023-03-03T08:43:43.992664+0000","last_unstale":"2023-03-03T08:43:44.979475+0000","last_undegraded":"2023-03-03T08:43:44.979475+0000","last_fullsized":"2023-03-03T08:43:44.979475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:07.276715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981207+0000","last_change":"2023-03-03T08:43:34.547450+0000","last_active":"2023-03-03T08:43:43.981207+0000","last_peered":"2023-03-03T08:43:43.981207+0000","last_clean":"2023-03-03T08:43:43.981207+0000","last_became_active":"2023-03-03T08:43:34.547168+0000","last_became_peered":"2023-03-03T08:43:34.547168+0000","last_unstale":"2023-03-03T08:43:43.981207+0000","last_undegraded":"2023-03-03T08:43:43.981207+0000","last_fullsized":"2023-03-03T08:43:43.981207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:17.723229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982025+0000","last_change":"2023-03-03T08:43:34.106505+0000","last_active":"2023-03-03T08:43:43.982025+0000","last_peered":"2023-03-03T08:43:43.982025+0000","last_clean":"2023-03-03T08:43:43.982025+0000","last_became_active":"2023-03-03T08:43:34.106079+0000","last_became_peered":"2023-03-03T08:43:34.106079+0000","last_unstale":"2023-03-03T08:43:43.982025+0000","last_undegraded":"2023-03-03T08:43:43.982025+0000","last_fullsized":"2023-03-03T08:43:43.982025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:48.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979698+0000","last_change":"2023-03-03T08:43:43.575448+0000","last_active":"2023-03-03T08:43:44.979698+0000","last_peered":"2023-03-03T08:43:44.979698+0000","last_clean":"2023-03-03T08:43:44.979698+0000","last_became_active":"2023-03-03T08:43:43.575142+0000","last_became_peered":"2023-03-03T08:43:43.575142+0000","last_unstale":"2023-03-03T08:43:44.979698+0000","last_undegraded":"2023-03-03T08:43:44.979698+0000","last_fullsized":"2023-03-03T08:43:44.979698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:42:09.308616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990396+0000","last_change":"2023-03-03T08:43:34.549123+0000","last_active":"2023-03-03T08:43:43.990396+0000","last_peered":"2023-03-03T08:43:43.990396+0000","last_clean":"2023-03-03T08:43:43.990396+0000","last_became_active":"2023-03-03T08:43:34.548839+0000","last_became_peered":"2023-03-03T08:43:34.548839+0000","last_unstale":"2023-03-03T08:43:43.990396+0000","last_undegraded":"2023-03-03T08:43:43.990396+0000","last_fullsized":"2023-03-03T08:43:43.990396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:38.134009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984959+0000","last_change":"2023-03-03T08:43:34.108187+0000","last_active":"2023-03-03T08:43:43.984959+0000","last_peered":"2023-03-03T08:43:43.984959+0000","last_clean":"2023-03-03T08:43:43.984959+0000","last_became_active":"2023-03-03T08:43:34.107859+0000","last_became_peered":"2023-03-03T08:43:34.107859+0000","last_unstale":"2023-03-03T08:43:43.984959+0000","last_undegraded":"2023-03-03T08:43:43.984959+0000","last_fullsized":"2023-03-03T08:43:43.984959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:46:40.595756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107953+0000","last_change":"2023-03-03T08:43:43.572738+0000","last_active":"2023-03-03T08:43:45.107953+0000","last_peered":"2023-03-03T08:43:45.107953+0000","last_clean":"2023-03-03T08:43:45.107953+0000","last_became_active":"2023-03-03T08:43:43.572435+0000","last_became_peered":"2023-03-03T08:43:43.572435+0000","last_unstale":"2023-03-03T08:43:45.107953+0000","last_undegraded":"2023-03-03T08:43:45.107953+0000","last_fullsized":"2023-03-03T08:43:45.107953+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:14.264653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105420+0000","last_change":"2023-03-03T08:43:43.995021+0000","last_active":"2023-03-03T08:43:45.105420+0000","last_peered":"2023-03-03T08:43:45.105420+0000","last_clean":"2023-03-03T08:43:45.105420+0000","last_became_active":"2023-03-03T08:43:43.994766+0000","last_became_peered":"2023-03-03T08:43:43.994766+0000","last_unstale":"2023-03-03T08:43:45.105420+0000","last_undegraded":"2023-03-03T08:43:45.105420+0000","last_fullsized":"2023-03-03T08:43:45.105420+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:14:03.827057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979582+0000","last_change":"2023-03-03T08:43:43.993391+0000","last_active":"2023-03-03T08:43:44.979582+0000","last_peered":"2023-03-03T08:43:44.979582+0000","last_clean":"2023-03-03T08:43:44.979582+0000","last_became_active":"2023-03-03T08:43:43.993087+0000","last_became_peered":"2023-03-03T08:43:43.993087+0000","last_unstale":"2023-03-03T08:43:44.979582+0000","last_undegraded":"2023-03-03T08:43:44.979582+0000","last_fullsized":"2023-03-03T08:43:44.979582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:13:01.193811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992180+0000","last_change":"2023-03-03T08:43:34.537301+0000","last_active":"2023-03-03T08:43:43.992180+0000","last_peered":"2023-03-03T08:43:43.992180+0000","last_clean":"2023-03-03T08:43:43.992180+0000","last_became_active":"2023-03-03T08:43:34.537007+0000","last_became_peered":"2023-03-03T08:43:34.537007+0000","last_unstale":"2023-03-03T08:43:43.992180+0000","last_undegraded":"2023-03-03T08:43:43.992180+0000","last_fullsized":"2023-03-03T08:43:43.992180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:59:49.124140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982700+0000","last_change":"2023-03-03T08:43:34.108056+0000","last_active":"2023-03-03T08:43:43.982700+0000","last_peered":"2023-03-03T08:43:43.982700+0000","last_clean":"2023-03-03T08:43:43.982700+0000","last_became_active":"2023-03-03T08:43:34.107708+0000","last_became_peered":"2023-03-03T08:43:34.107708+0000","last_unstale":"2023-03-03T08:43:43.982700+0000","last_undegraded":"2023-03-03T08:43:43.982700+0000","last_fullsized":"2023-03-03T08:43:43.982700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:10.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107314+0000","last_change":"2023-03-03T08:43:43.575788+0000","last_active":"2023-03-03T08:43:45.107314+0000","last_peered":"2023-03-03T08:43:45.107314+0000","last_clean":"2023-03-03T08:43:45.107314+0000","last_became_active":"2023-03-03T08:43:43.575507+0000","last_became_peered":"2023-03-03T08:43:43.575507+0000","last_unstale":"2023-03-03T08:43:45.107314+0000","last_undegraded":"2023-03-03T08:43:45.107314+0000","last_fullsized":"2023-03-03T08:43:45.107314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:15.843816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107877+0000","last_change":"2023-03-03T08:43:43.994916+0000","last_active":"2023-03-03T08:43:45.107877+0000","last_peered":"2023-03-03T08:43:45.107877+0000","last_clean":"2023-03-03T08:43:45.107877+0000","last_became_active":"2023-03-03T08:43:43.994639+0000","last_became_peered":"2023-03-03T08:43:43.994639+0000","last_unstale":"2023-03-03T08:43:45.107877+0000","last_undegraded":"2023-03-03T08:43:45.107877+0000","last_fullsized":"2023-03-03T08:43:45.107877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:52.526052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980834+0000","last_change":"2023-03-03T08:43:34.533987+0000","last_active":"2023-03-03T08:43:43.980834+0000","last_peered":"2023-03-03T08:43:43.980834+0000","last_clean":"2023-03-03T08:43:43.980834+0000","last_became_active":"2023-03-03T08:43:34.533729+0000","last_became_peered":"2023-03-03T08:43:34.533729+0000","last_unstale":"2023-03-03T08:43:43.980834+0000","last_undegraded":"2023-03-03T08:43:43.980834+0000","last_fullsized":"2023-03-03T08:43:43.980834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:18:12.407151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106398+0000","last_change":"2023-03-03T08:43:43.996808+0000","last_active":"2023-03-03T08:43:45.106398+0000","last_peered":"2023-03-03T08:43:45.106398+0000","last_clean":"2023-03-03T08:43:45.106398+0000","last_became_active":"2023-03-03T08:43:43.996559+0000","last_became_peered":"2023-03-03T08:43:43.996559+0000","last_unstale":"2023-03-03T08:43:45.106398+0000","last_undegraded":"2023-03-03T08:43:45.106398+0000","last_fullsized":"2023-03-03T08:43:45.106398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:01.223908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.993213+0000","last_change":"2023-03-03T08:43:34.540762+0000","last_active":"2023-03-03T08:43:43.993213+0000","last_peered":"2023-03-03T08:43:43.993213+0000","last_clean":"2023-03-03T08:43:43.993213+0000","last_became_active":"2023-03-03T08:43:34.540467+0000","last_became_peered":"2023-03-03T08:43:34.540467+0000","last_unstale":"2023-03-03T08:43:43.993213+0000","last_undegraded":"2023-03-03T08:43:43.993213+0000","last_fullsized":"2023-03-03T08:43:43.993213+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107993+0000","last_change":"2023-03-03T08:43:43.995286+0000","last_active":"2023-03-03T08:43:45.107993+0000","last_peered":"2023-03-03T08:43:45.107993+0000","last_clean":"2023-03-03T08:43:45.107993+0000","last_became_active":"2023-03-03T08:43:43.994861+0000","last_became_peered":"2023-03-03T08:43:43.994861+0000","last_unstale":"2023-03-03T08:43:45.107993+0000","last_undegraded":"2023-03-03T08:43:45.107993+0000","last_fullsized":"2023-03-03T08:43:45.107993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990722+0000","last_change":"2023-03-03T08:43:34.542858+0000","last_active":"2023-03-03T08:43:43.990722+0000","last_peered":"2023-03-03T08:43:43.990722+0000","last_clean":"2023-03-03T08:43:43.990722+0000","last_became_active":"2023-03-03T08:43:34.542510+0000","last_became_peered":"2023-03-03T08:43:34.542510+0000","last_unstale":"2023-03-03T08:43:43.990722+0000","last_undegraded":"2023-03-03T08:43:43.990722+0000","last_fullsized":"2023-03-03T08:43:43.990722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:21.471234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106711+0000","last_change":"2023-03-03T08:43:43.992084+0000","last_active":"2023-03-03T08:43:45.106711+0000","last_peered":"2023-03-03T08:43:45.106711+0000","last_clean":"2023-03-03T08:43:45.106711+0000","last_became_active":"2023-03-03T08:43:43.991465+0000","last_became_peered":"2023-03-03T08:43:43.991465+0000","last_unstale":"2023-03-03T08:43:45.106711+0000","last_undegraded":"2023-03-03T08:43:45.106711+0000","last_fullsized":"2023-03-03T08:43:45.106711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:19.530923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983571+0000","last_change":"2023-03-03T08:43:34.548469+0000","last_active":"2023-03-03T08:43:43.983571+0000","last_peered":"2023-03-03T08:43:43.983571+0000","last_clean":"2023-03-03T08:43:43.983571+0000","last_became_active":"2023-03-03T08:43:34.547647+0000","last_became_peered":"2023-03-03T08:43:34.547647+0000","last_unstale":"2023-03-03T08:43:43.983571+0000","last_undegraded":"2023-03-03T08:43:43.983571+0000","last_fullsized":"2023-03-03T08:43:43.983571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:40.054781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107161+0000","last_change":"2023-03-03T08:43:43.998061+0000","last_active":"2023-03-03T08:43:45.107161+0000","last_peered":"2023-03-03T08:43:45.107161+0000","last_clean":"2023-03-03T08:43:45.107161+0000","last_became_active":"2023-03-03T08:43:43.997770+0000","last_became_peered":"2023-03-03T08:43:43.997770+0000","last_unstale":"2023-03-03T08:43:45.107161+0000","last_undegraded":"2023-03-03T08:43:45.107161+0000","last_fullsized":"2023-03-03T08:43:45.107161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:17.053610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988635+0000","last_change":"2023-03-03T08:43:34.549771+0000","last_active":"2023-03-03T08:43:43.988635+0000","last_peered":"2023-03-03T08:43:43.988635+0000","last_clean":"2023-03-03T08:43:43.988635+0000","last_became_active":"2023-03-03T08:43:34.549479+0000","last_became_peered":"2023-03-03T08:43:34.549479+0000","last_unstale":"2023-03-03T08:43:43.988635+0000","last_undegraded":"2023-03-03T08:43:43.988635+0000","last_fullsized":"2023-03-03T08:43:43.988635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:41:03.902019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979024+0000","last_change":"2023-03-03T08:43:43.999391+0000","last_active":"2023-03-03T08:43:44.979024+0000","last_peered":"2023-03-03T08:43:44.979024+0000","last_clean":"2023-03-03T08:43:44.979024+0000","last_became_active":"2023-03-03T08:43:43.998904+0000","last_became_peered":"2023-03-03T08:43:43.998904+0000","last_unstale":"2023-03-03T08:43:44.979024+0000","last_undegraded":"2023-03-03T08:43:44.979024+0000","last_fullsized":"2023-03-03T08:43:44.979024+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:17.429776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992365+0000","last_change":"2023-03-03T08:43:34.542661+0000","last_active":"2023-03-03T08:43:43.992365+0000","last_peered":"2023-03-03T08:43:43.992365+0000","last_clean":"2023-03-03T08:43:43.992365+0000","last_became_active":"2023-03-03T08:43:34.542353+0000","last_became_peered":"2023-03-03T08:43:34.542353+0000","last_unstale":"2023-03-03T08:43:43.992365+0000","last_undegraded":"2023-03-03T08:43:43.992365+0000","last_fullsized":"2023-03-03T08:43:43.992365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:38.730662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105103+0000","last_change":"2023-03-03T08:43:43.996729+0000","last_active":"2023-03-03T08:43:45.105103+0000","last_peered":"2023-03-03T08:43:45.105103+0000","last_clean":"2023-03-03T08:43:45.105103+0000","last_became_active":"2023-03-03T08:43:43.986535+0000","last_became_peered":"2023-03-03T08:43:43.986535+0000","last_unstale":"2023-03-03T08:43:45.105103+0000","last_undegraded":"2023-03-03T08:43:45.105103+0000","last_fullsized":"2023-03-03T08:43:45.105103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:22.505305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982406+0000","last_change":"2023-03-03T08:43:34.549837+0000","last_active":"2023-03-03T08:43:43.982406+0000","last_peered":"2023-03-03T08:43:43.982406+0000","last_clean":"2023-03-03T08:43:43.982406+0000","last_became_active":"2023-03-03T08:43:34.549542+0000","last_became_peered":"2023-03-03T08:43:34.549542+0000","last_unstale":"2023-03-03T08:43:43.982406+0000","last_undegraded":"2023-03-03T08:43:43.982406+0000","last_fullsized":"2023-03-03T08:43:43.982406+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:31.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106334+0000","last_change":"2023-03-03T08:43:43.993213+0000","last_active":"2023-03-03T08:43:45.106334+0000","last_peered":"2023-03-03T08:43:45.106334+0000","last_clean":"2023-03-03T08:43:45.106334+0000","last_became_active":"2023-03-03T08:43:43.992920+0000","last_became_peered":"2023-03-03T08:43:43.992920+0000","last_unstale":"2023-03-03T08:43:45.106334+0000","last_undegraded":"2023-03-03T08:43:45.106334+0000","last_fullsized":"2023-03-03T08:43:45.106334+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:42.055599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.547003+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.546744+0000","last_became_peered":"2023-03-03T08:43:34.546744+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:53:38.996605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106934+0000","last_change":"2023-03-03T08:43:43.994195+0000","last_active":"2023-03-03T08:43:45.106934+0000","last_peered":"2023-03-03T08:43:45.106934+0000","last_clean":"2023-03-03T08:43:45.106934+0000","last_became_active":"2023-03-03T08:43:43.993463+0000","last_became_peered":"2023-03-03T08:43:43.993463+0000","last_unstale":"2023-03-03T08:43:45.106934+0000","last_undegraded":"2023-03-03T08:43:45.106934+0000","last_fullsized":"2023-03-03T08:43:45.106934+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:52.365793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982677+0000","last_change":"2023-03-03T08:43:34.553589+0000","last_active":"2023-03-03T08:43:43.982677+0000","last_peered":"2023-03-03T08:43:43.982677+0000","last_clean":"2023-03-03T08:43:43.982677+0000","last_became_active":"2023-03-03T08:43:34.553242+0000","last_became_peered":"2023-03-03T08:43:34.553242+0000","last_unstale":"2023-03-03T08:43:43.982677+0000","last_undegraded":"2023-03-03T08:43:43.982677+0000","last_fullsized":"2023-03-03T08:43:43.982677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:09.000051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107426+0000","last_change":"2023-03-03T08:43:44.004404+0000","last_active":"2023-03-03T08:43:45.107426+0000","last_peered":"2023-03-03T08:43:45.107426+0000","last_clean":"2023-03-03T08:43:45.107426+0000","last_became_active":"2023-03-03T08:43:44.004120+0000","last_became_peered":"2023-03-03T08:43:44.004120+0000","last_unstale":"2023-03-03T08:43:45.107426+0000","last_undegraded":"2023-03-03T08:43:45.107426+0000","last_fullsized":"2023-03-03T08:43:45.107426+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:36.006791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990661+0000","last_change":"2023-03-03T08:43:34.537110+0000","last_active":"2023-03-03T08:43:43.990661+0000","last_peered":"2023-03-03T08:43:43.990661+0000","last_clean":"2023-03-03T08:43:43.990661+0000","last_became_active":"2023-03-03T08:43:34.536848+0000","last_became_peered":"2023-03-03T08:43:34.536848+0000","last_unstale":"2023-03-03T08:43:43.990661+0000","last_undegraded":"2023-03-03T08:43:43.990661+0000","last_fullsized":"2023-03-03T08:43:43.990661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106366+0000","last_change":"2023-03-03T08:43:43.999266+0000","last_active":"2023-03-03T08:43:45.106366+0000","last_peered":"2023-03-03T08:43:45.106366+0000","last_clean":"2023-03-03T08:43:45.106366+0000","last_became_active":"2023-03-03T08:43:43.998929+0000","last_became_peered":"2023-03-03T08:43:43.998929+0000","last_unstale":"2023-03-03T08:43:45.106366+0000","last_undegraded":"2023-03-03T08:43:45.106366+0000","last_fullsized":"2023-03-03T08:43:45.106366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:02.236263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994220+0000","last_change":"2023-03-03T08:43:34.532154+0000","last_active":"2023-03-03T08:43:43.994220+0000","last_peered":"2023-03-03T08:43:43.994220+0000","last_clean":"2023-03-03T08:43:43.994220+0000","last_became_active":"2023-03-03T08:43:34.531844+0000","last_became_peered":"2023-03-03T08:43:34.531844+0000","last_unstale":"2023-03-03T08:43:43.994220+0000","last_undegraded":"2023-03-03T08:43:43.994220+0000","last_fullsized":"2023-03-03T08:43:43.994220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:00.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106991+0000","last_change":"2023-03-03T08:43:44.004469+0000","last_active":"2023-03-03T08:43:45.106991+0000","last_peered":"2023-03-03T08:43:45.106991+0000","last_clean":"2023-03-03T08:43:45.106991+0000","last_became_active":"2023-03-03T08:43:44.004187+0000","last_became_peered":"2023-03-03T08:43:44.004187+0000","last_unstale":"2023-03-03T08:43:45.106991+0000","last_undegraded":"2023-03-03T08:43:45.106991+0000","last_fullsized":"2023-03-03T08:43:45.106991+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:35.298857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979308+0000","last_change":"2023-03-03T08:43:34.527373+0000","last_active":"2023-03-03T08:43:43.979308+0000","last_peered":"2023-03-03T08:43:43.979308+0000","last_clean":"2023-03-03T08:43:43.979308+0000","last_became_active":"2023-03-03T08:43:34.527171+0000","last_became_peered":"2023-03-03T08:43:34.527171+0000","last_unstale":"2023-03-03T08:43:43.979308+0000","last_undegraded":"2023-03-03T08:43:43.979308+0000","last_fullsized":"2023-03-03T08:43:43.979308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:20:51.138587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108220+0000","last_change":"2023-03-03T08:43:43.996329+0000","last_active":"2023-03-03T08:43:45.108220+0000","last_peered":"2023-03-03T08:43:45.108220+0000","last_clean":"2023-03-03T08:43:45.108220+0000","last_became_active":"2023-03-03T08:43:43.995055+0000","last_became_peered":"2023-03-03T08:43:43.995055+0000","last_unstale":"2023-03-03T08:43:45.108220+0000","last_undegraded":"2023-03-03T08:43:45.108220+0000","last_fullsized":"2023-03-03T08:43:45.108220+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:57.593740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984695+0000","last_change":"2023-03-03T08:43:34.542729+0000","last_active":"2023-03-03T08:43:43.984695+0000","last_peered":"2023-03-03T08:43:43.984695+0000","last_clean":"2023-03-03T08:43:43.984695+0000","last_became_active":"2023-03-03T08:43:34.542246+0000","last_became_peered":"2023-03-03T08:43:34.542246+0000","last_unstale":"2023-03-03T08:43:43.984695+0000","last_undegraded":"2023-03-03T08:43:43.984695+0000","last_fullsized":"2023-03-03T08:43:43.984695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:14:59.995526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105365+0000","last_change":"2023-03-03T08:43:44.002358+0000","last_active":"2023-03-03T08:43:45.105365+0000","last_peered":"2023-03-03T08:43:45.105365+0000","last_clean":"2023-03-03T08:43:45.105365+0000","last_became_active":"2023-03-03T08:43:44.002020+0000","last_became_peered":"2023-03-03T08:43:44.002020+0000","last_unstale":"2023-03-03T08:43:45.105365+0000","last_undegraded":"2023-03-03T08:43:45.105365+0000","last_fullsized":"2023-03-03T08:43:45.105365+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:51.524774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982664+0000","last_change":"2023-03-03T08:43:34.549026+0000","last_active":"2023-03-03T08:43:43.982664+0000","last_peered":"2023-03-03T08:43:43.982664+0000","last_clean":"2023-03-03T08:43:43.982664+0000","last_became_active":"2023-03-03T08:43:34.548778+0000","last_became_peered":"2023-03-03T08:43:34.548778+0000","last_unstale":"2023-03-03T08:43:43.982664+0000","last_undegraded":"2023-03-03T08:43:43.982664+0000","last_fullsized":"2023-03-03T08:43:43.982664+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:41.835304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105281+0000","last_change":"2023-03-03T08:43:43.997684+0000","last_active":"2023-03-03T08:43:45.105281+0000","last_peered":"2023-03-03T08:43:45.105281+0000","last_clean":"2023-03-03T08:43:45.105281+0000","last_became_active":"2023-03-03T08:43:43.997503+0000","last_became_peered":"2023-03-03T08:43:43.997503+0000","last_unstale":"2023-03-03T08:43:45.105281+0000","last_undegraded":"2023-03-03T08:43:45.105281+0000","last_fullsized":"2023-03-03T08:43:45.105281+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:29.512524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991178+0000","last_change":"2023-03-03T08:43:34.544013+0000","last_active":"2023-03-03T08:43:43.991178+0000","last_peered":"2023-03-03T08:43:43.991178+0000","last_clean":"2023-03-03T08:43:43.991178+0000","last_became_active":"2023-03-03T08:43:34.543745+0000","last_became_peered":"2023-03-03T08:43:34.543745+0000","last_unstale":"2023-03-03T08:43:43.991178+0000","last_undegraded":"2023-03-03T08:43:43.991178+0000","last_fullsized":"2023-03-03T08:43:43.991178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:05.647977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107560+0000","last_change":"2023-03-03T08:43:43.999884+0000","last_active":"2023-03-03T08:43:45.107560+0000","last_peered":"2023-03-03T08:43:45.107560+0000","last_clean":"2023-03-03T08:43:45.107560+0000","last_became_active":"2023-03-03T08:43:43.999593+0000","last_became_peered":"2023-03-03T08:43:43.999593+0000","last_unstale":"2023-03-03T08:43:45.107560+0000","last_undegraded":"2023-03-03T08:43:45.107560+0000","last_fullsized":"2023-03-03T08:43:45.107560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:45:09.132974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982600+0000","last_change":"2023-03-03T08:43:34.553434+0000","last_active":"2023-03-03T08:43:43.982600+0000","last_peered":"2023-03-03T08:43:43.982600+0000","last_clean":"2023-03-03T08:43:43.982600+0000","last_became_active":"2023-03-03T08:43:34.553125+0000","last_became_peered":"2023-03-03T08:43:34.553125+0000","last_unstale":"2023-03-03T08:43:43.982600+0000","last_undegraded":"2023-03-03T08:43:43.982600+0000","last_fullsized":"2023-03-03T08:43:43.982600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:41:17.553506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105717+0000","last_change":"2023-03-03T08:43:44.006710+0000","last_active":"2023-03-03T08:43:45.105717+0000","last_peered":"2023-03-03T08:43:45.105717+0000","last_clean":"2023-03-03T08:43:45.105717+0000","last_became_active":"2023-03-03T08:43:44.006447+0000","last_became_peered":"2023-03-03T08:43:44.006447+0000","last_unstale":"2023-03-03T08:43:45.105717+0000","last_undegraded":"2023-03-03T08:43:45.105717+0000","last_fullsized":"2023-03-03T08:43:45.105717+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:21.767326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982538+0000","last_change":"2023-03-03T08:43:34.550889+0000","last_active":"2023-03-03T08:43:43.982538+0000","last_peered":"2023-03-03T08:43:43.982538+0000","last_clean":"2023-03-03T08:43:43.982538+0000","last_became_active":"2023-03-03T08:43:34.550605+0000","last_became_peered":"2023-03-03T08:43:34.550605+0000","last_unstale":"2023-03-03T08:43:43.982538+0000","last_undegraded":"2023-03-03T08:43:43.982538+0000","last_fullsized":"2023-03-03T08:43:43.982538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:01.118555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105567+0000","last_change":"2023-03-03T08:43:43.998231+0000","last_active":"2023-03-03T08:43:45.105567+0000","last_peered":"2023-03-03T08:43:45.105567+0000","last_clean":"2023-03-03T08:43:45.105567+0000","last_became_active":"2023-03-03T08:43:43.997941+0000","last_became_peered":"2023-03-03T08:43:43.997941+0000","last_unstale":"2023-03-03T08:43:45.105567+0000","last_undegraded":"2023-03-03T08:43:45.105567+0000","last_fullsized":"2023-03-03T08:43:45.105567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:13:32.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983100+0000","last_change":"2023-03-03T08:43:34.548229+0000","last_active":"2023-03-03T08:43:43.983100+0000","last_peered":"2023-03-03T08:43:43.983100+0000","last_clean":"2023-03-03T08:43:43.983100+0000","last_became_active":"2023-03-03T08:43:34.548013+0000","last_became_peered":"2023-03-03T08:43:34.548013+0000","last_unstale":"2023-03-03T08:43:43.983100+0000","last_undegraded":"2023-03-03T08:43:43.983100+0000","last_fullsized":"2023-03-03T08:43:43.983100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:27.793716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108186+0000","last_change":"2023-03-03T08:43:43.992834+0000","last_active":"2023-03-03T08:43:45.108186+0000","last_peered":"2023-03-03T08:43:45.108186+0000","last_clean":"2023-03-03T08:43:45.108186+0000","last_became_active":"2023-03-03T08:43:43.991577+0000","last_became_peered":"2023-03-03T08:43:43.991577+0000","last_unstale":"2023-03-03T08:43:45.108186+0000","last_undegraded":"2023-03-03T08:43:45.108186+0000","last_fullsized":"2023-03-03T08:43:45.108186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:18:37.119460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990785+0000","last_change":"2023-03-03T08:43:34.549838+0000","last_active":"2023-03-03T08:43:43.990785+0000","last_peered":"2023-03-03T08:43:43.990785+0000","last_clean":"2023-03-03T08:43:43.990785+0000","last_became_active":"2023-03-03T08:43:34.549537+0000","last_became_peered":"2023-03-03T08:43:34.549537+0000","last_unstale":"2023-03-03T08:43:43.990785+0000","last_undegraded":"2023-03-03T08:43:43.990785+0000","last_fullsized":"2023-03-03T08:43:43.990785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:36.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978412+0000","last_change":"2023-03-03T08:43:43.991488+0000","last_active":"2023-03-03T08:43:44.978412+0000","last_peered":"2023-03-03T08:43:44.978412+0000","last_clean":"2023-03-03T08:43:44.978412+0000","last_became_active":"2023-03-03T08:43:43.991139+0000","last_became_peered":"2023-03-03T08:43:43.991139+0000","last_unstale":"2023-03-03T08:43:44.978412+0000","last_undegraded":"2023-03-03T08:43:44.978412+0000","last_fullsized":"2023-03-03T08:43:44.978412+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:21.119183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983652+0000","last_change":"2023-03-03T08:43:34.533496+0000","last_active":"2023-03-03T08:43:43.983652+0000","last_peered":"2023-03-03T08:43:43.983652+0000","last_clean":"2023-03-03T08:43:43.983652+0000","last_became_active":"2023-03-03T08:43:34.532714+0000","last_became_peered":"2023-03-03T08:43:34.532714+0000","last_unstale":"2023-03-03T08:43:43.983652+0000","last_undegraded":"2023-03-03T08:43:43.983652+0000","last_fullsized":"2023-03-03T08:43:43.983652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:47.800807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105250+0000","last_change":"2023-03-03T08:43:43.996215+0000","last_active":"2023-03-03T08:43:45.105250+0000","last_peered":"2023-03-03T08:43:45.105250+0000","last_clean":"2023-03-03T08:43:45.105250+0000","last_became_active":"2023-03-03T08:43:43.995791+0000","last_became_peered":"2023-03-03T08:43:43.995791+0000","last_unstale":"2023-03-03T08:43:45.105250+0000","last_undegraded":"2023-03-03T08:43:45.105250+0000","last_fullsized":"2023-03-03T08:43:45.105250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:50:41.130427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979929+0000","last_change":"2023-03-03T08:43:34.533673+0000","last_active":"2023-03-03T08:43:43.979929+0000","last_peered":"2023-03-03T08:43:43.979929+0000","last_clean":"2023-03-03T08:43:43.979929+0000","last_became_active":"2023-03-03T08:43:34.532283+0000","last_became_peered":"2023-03-03T08:43:34.532283+0000","last_unstale":"2023-03-03T08:43:43.979929+0000","last_undegraded":"2023-03-03T08:43:43.979929+0000","last_fullsized":"2023-03-03T08:43:43.979929+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:25.591188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979205+0000","last_change":"2023-03-03T08:43:43.999435+0000","last_active":"2023-03-03T08:43:44.979205+0000","last_peered":"2023-03-03T08:43:44.979205+0000","last_clean":"2023-03-03T08:43:44.979205+0000","last_became_active":"2023-03-03T08:43:43.997037+0000","last_became_peered":"2023-03-03T08:43:43.997037+0000","last_unstale":"2023-03-03T08:43:44.979205+0000","last_undegraded":"2023-03-03T08:43:44.979205+0000","last_fullsized":"2023-03-03T08:43:44.979205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:54.176343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980585+0000","last_change":"2023-03-03T08:43:34.537664+0000","last_active":"2023-03-03T08:43:43.980585+0000","last_peered":"2023-03-03T08:43:43.980585+0000","last_clean":"2023-03-03T08:43:43.980585+0000","last_became_active":"2023-03-03T08:43:34.537214+0000","last_became_peered":"2023-03-03T08:43:34.537214+0000","last_unstale":"2023-03-03T08:43:43.980585+0000","last_undegraded":"2023-03-03T08:43:43.980585+0000","last_fullsized":"2023-03-03T08:43:43.980585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:54:09.222135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106227+0000","last_change":"2023-03-03T08:43:44.000140+0000","last_active":"2023-03-03T08:43:45.106227+0000","last_peered":"2023-03-03T08:43:45.106227+0000","last_clean":"2023-03-03T08:43:45.106227+0000","last_became_active":"2023-03-03T08:43:43.999778+0000","last_became_peered":"2023-03-03T08:43:43.999778+0000","last_unstale":"2023-03-03T08:43:45.106227+0000","last_undegraded":"2023-03-03T08:43:45.106227+0000","last_fullsized":"2023-03-03T08:43:45.106227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:29:10.745759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987366+0000","last_change":"2023-03-03T08:43:34.543620+0000","last_active":"2023-03-03T08:43:43.987366+0000","last_peered":"2023-03-03T08:43:43.987366+0000","last_clean":"2023-03-03T08:43:43.987366+0000","last_became_active":"2023-03-03T08:43:34.543370+0000","last_became_peered":"2023-03-03T08:43:34.543370+0000","last_unstale":"2023-03-03T08:43:43.987366+0000","last_undegraded":"2023-03-03T08:43:43.987366+0000","last_fullsized":"2023-03-03T08:43:43.987366+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:40:32.808203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979449+0000","last_change":"2023-03-03T08:43:43.995084+0000","last_active":"2023-03-03T08:43:44.979449+0000","last_peered":"2023-03-03T08:43:44.979449+0000","last_clean":"2023-03-03T08:43:44.979449+0000","last_became_active":"2023-03-03T08:43:43.994824+0000","last_became_peered":"2023-03-03T08:43:43.994824+0000","last_unstale":"2023-03-03T08:43:44.979449+0000","last_undegraded":"2023-03-03T08:43:44.979449+0000","last_fullsized":"2023-03-03T08:43:44.979449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:04:17.481969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985302+0000","last_change":"2023-03-03T08:43:34.538467+0000","last_active":"2023-03-03T08:43:43.985302+0000","last_peered":"2023-03-03T08:43:43.985302+0000","last_clean":"2023-03-03T08:43:43.985302+0000","last_became_active":"2023-03-03T08:43:34.538207+0000","last_became_peered":"2023-03-03T08:43:34.538207+0000","last_unstale":"2023-03-03T08:43:43.985302+0000","last_undegraded":"2023-03-03T08:43:43.985302+0000","last_fullsized":"2023-03-03T08:43:43.985302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:14.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980577+0000","last_change":"2023-03-03T08:43:43.996076+0000","last_active":"2023-03-03T08:43:44.980577+0000","last_peered":"2023-03-03T08:43:44.980577+0000","last_clean":"2023-03-03T08:43:44.980577+0000","last_became_active":"2023-03-03T08:43:43.995799+0000","last_became_peered":"2023-03-03T08:43:43.995799+0000","last_unstale":"2023-03-03T08:43:44.980577+0000","last_undegraded":"2023-03-03T08:43:44.980577+0000","last_fullsized":"2023-03-03T08:43:44.980577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:35:35.889895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986289+0000","last_change":"2023-03-03T08:43:34.544721+0000","last_active":"2023-03-03T08:43:43.986289+0000","last_peered":"2023-03-03T08:43:43.986289+0000","last_clean":"2023-03-03T08:43:43.986289+0000","last_became_active":"2023-03-03T08:43:34.544461+0000","last_became_peered":"2023-03-03T08:43:34.544461+0000","last_unstale":"2023-03-03T08:43:43.986289+0000","last_undegraded":"2023-03-03T08:43:43.986289+0000","last_fullsized":"2023-03-03T08:43:43.986289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:52:11.890779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980078+0000","last_change":"2023-03-03T08:43:44.006352+0000","last_active":"2023-03-03T08:43:44.980078+0000","last_peered":"2023-03-03T08:43:44.980078+0000","last_clean":"2023-03-03T08:43:44.980078+0000","last_became_active":"2023-03-03T08:43:44.005802+0000","last_became_peered":"2023-03-03T08:43:44.005802+0000","last_unstale":"2023-03-03T08:43:44.980078+0000","last_undegraded":"2023-03-03T08:43:44.980078+0000","last_fullsized":"2023-03-03T08:43:44.980078+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:52.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986425+0000","last_change":"2023-03-03T08:43:34.541669+0000","last_active":"2023-03-03T08:43:43.986425+0000","last_peered":"2023-03-03T08:43:43.986425+0000","last_clean":"2023-03-03T08:43:43.986425+0000","last_became_active":"2023-03-03T08:43:34.541104+0000","last_became_peered":"2023-03-03T08:43:34.541104+0000","last_unstale":"2023-03-03T08:43:43.986425+0000","last_undegraded":"2023-03-03T08:43:43.986425+0000","last_fullsized":"2023-03-03T08:43:43.986425+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980059+0000","last_change":"2023-03-03T08:43:43.998937+0000","last_active":"2023-03-03T08:43:44.980059+0000","last_peered":"2023-03-03T08:43:44.980059+0000","last_clean":"2023-03-03T08:43:44.980059+0000","last_became_active":"2023-03-03T08:43:43.998164+0000","last_became_peered":"2023-03-03T08:43:43.998164+0000","last_unstale":"2023-03-03T08:43:44.980059+0000","last_undegraded":"2023-03-03T08:43:44.980059+0000","last_fullsized":"2023-03-03T08:43:44.980059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:53.741704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.546272+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.546006+0000","last_became_peered":"2023-03-03T08:43:34.546006+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:15.121761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":118397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":119470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9336,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362504,"statfs":{"total":96636764160,"available":96627204096,"internally_reserved":0,"allocated":901120,"data_stored":710394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8244,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363596,"statfs":{"total":96636764160,"available":96628322304,"internally_reserved":0,"allocated":307200,"data_stored":118565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8188,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363652,"statfs":{"total":96636764160,"available":96628379648,"internally_reserved":0,"allocated":315392,"data_stored":121989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":120069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":121018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9260,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362580,"statfs":{"total":96636764160,"available":96627281920,"internally_reserved":0,"allocated":888832,"data_stored":709827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:43:49.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:43:49.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.274+0000 7fcacb864700 1 Processor -- start 2023-03-03T08:43:49.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.275+0000 7fcacb864700 1 -- start start 2023-03-03T08:43:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.275+0000 7fcacb864700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcac4151cd0 0x7fcac4154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.275+0000 7fcacb864700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcac40c2e30 0x7fcac4158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.275+0000 7fcacb864700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac4157ff0 con 0x7fcac406c6c0 2023-03-03T08:43:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.275+0000 7fcacb864700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac406d640 con 0x7fcac40c2e30 2023-03-03T08:43:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.275+0000 7fcacb864700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac406cda0 con 0x7fcac4151cd0 2023-03-03T08:43:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcac8dff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcac40c2e30 0x7fcac4158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcac8dff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcac40c2e30 0x7fcac4158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51524/0 (socket says 172.21.15.7:51524) 2023-03-03T08:43:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcac8dff700 1 -- 172.21.15.7:0/1240155327 learned_addr learned my addr 172.21.15.7:0/1240155327 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcac9600700 1 --2- 172.21.15.7:0/1240155327 >> v2:172.21.15.143:3300/0 conn(0x7fcac4151cd0 0x7fcac4154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcac8dff700 1 -- 172.21.15.7:0/1240155327 >> v2:172.21.15.143:3300/0 conn(0x7fcac4151cd0 msgr2=0x7fcac4154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcac8dff700 1 --2- 172.21.15.7:0/1240155327 >> v2:172.21.15.143:3300/0 conn(0x7fcac4151cd0 0x7fcac4154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcac8dff700 1 -- 172.21.15.7:0/1240155327 >> v1:172.21.15.143:6789/0 conn(0x7fcac406c6c0 legacy=0x7fcac40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcac8dff700 1 -- 172.21.15.7:0/1240155327 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac406d080 con 0x7fcac40c2e30 2023-03-03T08:43:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcac8dff700 1 --2- 172.21.15.7:0/1240155327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 0x7fcac4158480 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcac406f340 tx=0x7fcab4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1105d694e87e992f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcabbfff700 1 -- 172.21.15.7:0/1240155327 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab400b870 con 0x7fcac40c2e30 2023-03-03T08:43:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcabbfff700 1 -- 172.21.15.7:0/1240155327 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab400b9d0 con 0x7fcac40c2e30 2023-03-03T08:43:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.276+0000 7fcabbfff700 1 -- 172.21.15.7:0/1240155327 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab400bd60 con 0x7fcac40c2e30 2023-03-03T08:43:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.277+0000 7fcacb864700 1 -- 172.21.15.7:0/1240155327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 msgr2=0x7fcac4158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.277+0000 7fcacb864700 1 --2- 172.21.15.7:0/1240155327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 0x7fcac4158480 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcac406f340 tx=0x7fcab4009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.277+0000 7fcacb864700 1 -- 172.21.15.7:0/1240155327 shutdown_connections 2023-03-03T08:43:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.277+0000 7fcacb864700 1 --2- 172.21.15.7:0/1240155327 >> v2:172.21.15.143:3300/0 conn(0x7fcac4151cd0 0x7fcac4154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.277+0000 7fcacb864700 1 --2- 172.21.15.7:0/1240155327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 0x7fcac4158480 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.277+0000 7fcacb864700 1 -- 172.21.15.7:0/1240155327 >> 172.21.15.7:0/1240155327 conn(0x7fcac40a94d0 msgr2=0x7fcac40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.277+0000 7fcacb864700 1 -- 172.21.15.7:0/1240155327 shutdown_connections 2023-03-03T08:43:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.277+0000 7fcacb864700 1 -- 172.21.15.7:0/1240155327 wait complete. 2023-03-03T08:43:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.278+0000 7fcacb864700 1 Processor -- start 2023-03-03T08:43:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.278+0000 7fcacb864700 1 -- start start 2023-03-03T08:43:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcacb864700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac406c6c0 0x7fcac40c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcacb864700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 0x7fcac414a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcacb864700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac4151cd0 0x7fcac414da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcacb864700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac4155e70 con 0x7fcac40c2e30 2023-03-03T08:43:49.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcacb864700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac41515a0 con 0x7fcac4151cd0 2023-03-03T08:43:49.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcacb864700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac414aa80 con 0x7fcac406c6c0 2023-03-03T08:43:49.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcac8dff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 0x7fcac414a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:49.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcac8dff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 0x7fcac414a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51530/0 (socket says 172.21.15.7:51530) 2023-03-03T08:43:49.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcac8dff700 1 -- 172.21.15.7:0/1972487667 learned_addr learned my addr 172.21.15.7:0/1972487667 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:49.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcac9600700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac406c6c0 0x7fcac40c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:49.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.279+0000 7fcac9e01700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac4151cd0 0x7fcac414da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:49.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcac8dff700 1 -- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac406c6c0 msgr2=0x7fcac40c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcac8dff700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac406c6c0 0x7fcac40c6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcac8dff700 1 -- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac4151cd0 msgr2=0x7fcac414da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcac8dff700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac4151cd0 0x7fcac414da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcac8dff700 1 -- 172.21.15.7:0/1972487667 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab4009550 con 0x7fcac40c2e30 2023-03-03T08:43:49.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcac9e01700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac4151cd0 0x7fcac414da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:49.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcac8dff700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 0x7fcac414a540 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcab4014040 tx=0x7fcab4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:49.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcab9ffb700 1 -- 172.21.15.7:0/1972487667 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab4004520 con 0x7fcac40c2e30 2023-03-03T08:43:49.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcab9ffb700 1 -- 172.21.15.7:0/1972487667 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab401e070 con 0x7fcac40c2e30 2023-03-03T08:43:49.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcacb864700 1 -- 172.21.15.7:0/1972487667 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac4157c80 con 0x7fcac40c2e30 2023-03-03T08:43:49.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcab9ffb700 1 -- 172.21.15.7:0/1972487667 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab40038a0 con 0x7fcac40c2e30 2023-03-03T08:43:49.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.280+0000 7fcacb864700 1 -- 172.21.15.7:0/1972487667 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac406d360 con 0x7fcac40c2e30 2023-03-03T08:43:49.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.281+0000 7fcacb864700 1 -- 172.21.15.7:0/1972487667 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac4154a50 con 0x7fcac40c2e30 2023-03-03T08:43:49.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.281+0000 7fcab9ffb700 1 -- 172.21.15.7:0/1972487667 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcab400f050 con 0x7fcac40c2e30 2023-03-03T08:43:49.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.282+0000 7fcab9ffb700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaa80682f0 0x7fcaa806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.282+0000 7fcab9ffb700 1 -- 172.21.15.7:0/1972487667 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fcab4020070 con 0x7fcac40c2e30 2023-03-03T08:43:49.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.282+0000 7fcac9600700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaa80682f0 0x7fcaa806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:49.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.283+0000 7fcac9600700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaa80682f0 0x7fcaa806a7a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcac0006b20 tx=0x7fcac0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:49.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.286+0000 7fcab9ffb700 1 -- 172.21.15.7:0/1972487667 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcab4053ab0 con 0x7fcac40c2e30 2023-03-03T08:43:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.475+0000 7fcacb864700 1 -- 172.21.15.7:0/1972487667 --> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcac4156b30 con 0x7fcaa80682f0 2023-03-03T08:43:49.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.490+0000 7fcab9ffb700 1 -- 172.21.15.7:0/1972487667 <== mgr.4102 v2:172.21.15.143:6832/96367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525186 (secure 0 0 0) 0x7fcac4156b30 con 0x7fcaa80682f0 2023-03-03T08:43:49.496 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:49.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 -- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaa80682f0 msgr2=0x7fcaa806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaa80682f0 0x7fcaa806a7a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcac0006b20 tx=0x7fcac0008040 comp rx=0 tx=0).stop 2023-03-03T08:43:49.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 -- 172.21.15.7:0/1972487667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 msgr2=0x7fcac414a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 0x7fcac414a540 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcab4014040 tx=0x7fcab4004750 comp rx=0 tx=0).stop 2023-03-03T08:43:49.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 -- 172.21.15.7:0/1972487667 shutdown_connections 2023-03-03T08:43:49.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac4151cd0 0x7fcac414da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac406c6c0 0x7fcac40c6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaa80682f0 0x7fcaa806a7a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 --2- 172.21.15.7:0/1972487667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac40c2e30 0x7fcac414a540 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 -- 172.21.15.7:0/1972487667 >> 172.21.15.7:0/1972487667 conn(0x7fcac40a94d0 msgr2=0x7fcac406a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:49.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 -- 172.21.15.7:0/1972487667 shutdown_connections 2023-03-03T08:43:49.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.494+0000 7fcacb864700 1 -- 172.21.15.7:0/1972487667 wait complete. 2023-03-03T08:43:49.501 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-03-03T08:43:49.508 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-03-03T08:43:49.109058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595028,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":248,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":378,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68052,"kb_used_data":3540,"kb_used_omap":0,"kb_used_meta":64448,"kb_avail":754906668,"statfs":{"total":773094113280,"available":773024428032,"internally_reserved":0,"allocated":3624960,"data_stored":2139729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65994752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001787"},"pg_stats":[{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980166+0000","last_change":"2023-03-03T08:43:43.997091+0000","last_active":"2023-03-03T08:43:44.980166+0000","last_peered":"2023-03-03T08:43:44.980166+0000","last_clean":"2023-03-03T08:43:44.980166+0000","last_became_active":"2023-03-03T08:43:43.996805+0000","last_became_peered":"2023-03-03T08:43:43.996805+0000","last_unstale":"2023-03-03T08:43:44.980166+0000","last_undegraded":"2023-03-03T08:43:44.980166+0000","last_fullsized":"2023-03-03T08:43:44.980166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982505+0000","last_change":"2023-03-03T08:43:34.546828+0000","last_active":"2023-03-03T08:43:43.982505+0000","last_peered":"2023-03-03T08:43:43.982505+0000","last_clean":"2023-03-03T08:43:43.982505+0000","last_became_active":"2023-03-03T08:43:34.546559+0000","last_became_peered":"2023-03-03T08:43:34.546559+0000","last_unstale":"2023-03-03T08:43:43.982505+0000","last_undegraded":"2023-03-03T08:43:43.982505+0000","last_fullsized":"2023-03-03T08:43:43.982505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:21:14.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106061+0000","last_change":"2023-03-03T08:43:43.995073+0000","last_active":"2023-03-03T08:43:45.106061+0000","last_peered":"2023-03-03T08:43:45.106061+0000","last_clean":"2023-03-03T08:43:45.106061+0000","last_became_active":"2023-03-03T08:43:43.994797+0000","last_became_peered":"2023-03-03T08:43:43.994797+0000","last_unstale":"2023-03-03T08:43:45.106061+0000","last_undegraded":"2023-03-03T08:43:45.106061+0000","last_fullsized":"2023-03-03T08:43:45.106061+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:21:52.642936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982397+0000","last_change":"2023-03-03T08:43:34.552931+0000","last_active":"2023-03-03T08:43:43.982397+0000","last_peered":"2023-03-03T08:43:43.982397+0000","last_clean":"2023-03-03T08:43:43.982397+0000","last_became_active":"2023-03-03T08:43:34.552623+0000","last_became_peered":"2023-03-03T08:43:34.552623+0000","last_unstale":"2023-03-03T08:43:43.982397+0000","last_undegraded":"2023-03-03T08:43:43.982397+0000","last_fullsized":"2023-03-03T08:43:43.982397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:07:43.954897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979088+0000","last_change":"2023-03-03T08:43:43.999245+0000","last_active":"2023-03-03T08:43:44.979088+0000","last_peered":"2023-03-03T08:43:44.979088+0000","last_clean":"2023-03-03T08:43:44.979088+0000","last_became_active":"2023-03-03T08:43:43.998592+0000","last_became_peered":"2023-03-03T08:43:43.998592+0000","last_unstale":"2023-03-03T08:43:44.979088+0000","last_undegraded":"2023-03-03T08:43:44.979088+0000","last_fullsized":"2023-03-03T08:43:44.979088+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:23.788724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991054+0000","last_change":"2023-03-03T08:43:34.544408+0000","last_active":"2023-03-03T08:43:43.991054+0000","last_peered":"2023-03-03T08:43:43.991054+0000","last_clean":"2023-03-03T08:43:43.991054+0000","last_became_active":"2023-03-03T08:43:34.543977+0000","last_became_peered":"2023-03-03T08:43:34.543977+0000","last_unstale":"2023-03-03T08:43:43.991054+0000","last_undegraded":"2023-03-03T08:43:43.991054+0000","last_fullsized":"2023-03-03T08:43:43.991054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.074082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105254+0000","last_change":"2023-03-03T08:43:43.994545+0000","last_active":"2023-03-03T08:43:45.105254+0000","last_peered":"2023-03-03T08:43:45.105254+0000","last_clean":"2023-03-03T08:43:45.105254+0000","last_became_active":"2023-03-03T08:43:43.994271+0000","last_became_peered":"2023-03-03T08:43:43.994271+0000","last_unstale":"2023-03-03T08:43:45.105254+0000","last_undegraded":"2023-03-03T08:43:45.105254+0000","last_fullsized":"2023-03-03T08:43:45.105254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:15.925960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989064+0000","last_change":"2023-03-03T08:43:34.538786+0000","last_active":"2023-03-03T08:43:43.989064+0000","last_peered":"2023-03-03T08:43:43.989064+0000","last_clean":"2023-03-03T08:43:43.989064+0000","last_became_active":"2023-03-03T08:43:34.538494+0000","last_became_peered":"2023-03-03T08:43:34.538494+0000","last_unstale":"2023-03-03T08:43:43.989064+0000","last_undegraded":"2023-03-03T08:43:43.989064+0000","last_fullsized":"2023-03-03T08:43:43.989064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:53.838286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105391+0000","last_change":"2023-03-03T08:43:43.993678+0000","last_active":"2023-03-03T08:43:45.105391+0000","last_peered":"2023-03-03T08:43:45.105391+0000","last_clean":"2023-03-03T08:43:45.105391+0000","last_became_active":"2023-03-03T08:43:43.993402+0000","last_became_peered":"2023-03-03T08:43:43.993402+0000","last_unstale":"2023-03-03T08:43:45.105391+0000","last_undegraded":"2023-03-03T08:43:45.105391+0000","last_fullsized":"2023-03-03T08:43:45.105391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:31:00.747216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983479+0000","last_change":"2023-03-03T08:43:34.547197+0000","last_active":"2023-03-03T08:43:43.983479+0000","last_peered":"2023-03-03T08:43:43.983479+0000","last_clean":"2023-03-03T08:43:43.983479+0000","last_became_active":"2023-03-03T08:43:34.546927+0000","last_became_peered":"2023-03-03T08:43:34.546927+0000","last_unstale":"2023-03-03T08:43:43.983479+0000","last_undegraded":"2023-03-03T08:43:43.983479+0000","last_fullsized":"2023-03-03T08:43:43.983479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:21:34.845968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984318+0000","last_change":"2023-03-03T08:43:34.540812+0000","last_active":"2023-03-03T08:43:43.984318+0000","last_peered":"2023-03-03T08:43:43.984318+0000","last_clean":"2023-03-03T08:43:43.984318+0000","last_became_active":"2023-03-03T08:43:34.540549+0000","last_became_peered":"2023-03-03T08:43:34.540549+0000","last_unstale":"2023-03-03T08:43:43.984318+0000","last_undegraded":"2023-03-03T08:43:43.984318+0000","last_fullsized":"2023-03-03T08:43:43.984318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982875+0000","last_change":"2023-03-03T08:43:34.108426+0000","last_active":"2023-03-03T08:43:43.982875+0000","last_peered":"2023-03-03T08:43:43.982875+0000","last_clean":"2023-03-03T08:43:43.982875+0000","last_became_active":"2023-03-03T08:43:34.108009+0000","last_became_peered":"2023-03-03T08:43:34.108009+0000","last_unstale":"2023-03-03T08:43:43.982875+0000","last_undegraded":"2023-03-03T08:43:43.982875+0000","last_fullsized":"2023-03-03T08:43:43.982875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:53.632744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979479+0000","last_change":"2023-03-03T08:43:43.575708+0000","last_active":"2023-03-03T08:43:44.979479+0000","last_peered":"2023-03-03T08:43:44.979479+0000","last_clean":"2023-03-03T08:43:44.979479+0000","last_became_active":"2023-03-03T08:43:43.575417+0000","last_became_peered":"2023-03-03T08:43:43.575417+0000","last_unstale":"2023-03-03T08:43:44.979479+0000","last_undegraded":"2023-03-03T08:43:44.979479+0000","last_fullsized":"2023-03-03T08:43:44.979479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:25.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106018+0000","last_change":"2023-03-03T08:43:43.999274+0000","last_active":"2023-03-03T08:43:45.106018+0000","last_peered":"2023-03-03T08:43:45.106018+0000","last_clean":"2023-03-03T08:43:45.106018+0000","last_became_active":"2023-03-03T08:43:43.998823+0000","last_became_peered":"2023-03-03T08:43:43.998823+0000","last_unstale":"2023-03-03T08:43:45.106018+0000","last_undegraded":"2023-03-03T08:43:45.106018+0000","last_fullsized":"2023-03-03T08:43:45.106018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:48.799261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992918+0000","last_change":"2023-03-03T08:43:34.532482+0000","last_active":"2023-03-03T08:43:43.992918+0000","last_peered":"2023-03-03T08:43:43.992918+0000","last_clean":"2023-03-03T08:43:43.992918+0000","last_became_active":"2023-03-03T08:43:34.531751+0000","last_became_peered":"2023-03-03T08:43:34.531751+0000","last_unstale":"2023-03-03T08:43:43.992918+0000","last_undegraded":"2023-03-03T08:43:43.992918+0000","last_fullsized":"2023-03-03T08:43:43.992918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:47:45.214945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982691+0000","last_change":"2023-03-03T08:43:34.109782+0000","last_active":"2023-03-03T08:43:43.982691+0000","last_peered":"2023-03-03T08:43:43.982691+0000","last_clean":"2023-03-03T08:43:43.982691+0000","last_became_active":"2023-03-03T08:43:34.109513+0000","last_became_peered":"2023-03-03T08:43:34.109513+0000","last_unstale":"2023-03-03T08:43:43.982691+0000","last_undegraded":"2023-03-03T08:43:43.982691+0000","last_fullsized":"2023-03-03T08:43:43.982691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:16.185161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105634+0000","last_change":"2023-03-03T08:43:43.576885+0000","last_active":"2023-03-03T08:43:45.105634+0000","last_peered":"2023-03-03T08:43:45.105634+0000","last_clean":"2023-03-03T08:43:45.105634+0000","last_became_active":"2023-03-03T08:43:43.576577+0000","last_became_peered":"2023-03-03T08:43:43.576577+0000","last_unstale":"2023-03-03T08:43:45.105634+0000","last_undegraded":"2023-03-03T08:43:45.105634+0000","last_fullsized":"2023-03-03T08:43:45.105634+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:26.501956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978988+0000","last_change":"2023-03-03T08:43:43.992020+0000","last_active":"2023-03-03T08:43:44.978988+0000","last_peered":"2023-03-03T08:43:44.978988+0000","last_clean":"2023-03-03T08:43:44.978988+0000","last_became_active":"2023-03-03T08:43:43.991674+0000","last_became_peered":"2023-03-03T08:43:43.991674+0000","last_unstale":"2023-03-03T08:43:44.978988+0000","last_undegraded":"2023-03-03T08:43:44.978988+0000","last_fullsized":"2023-03-03T08:43:44.978988+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:54:30.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985919+0000","last_change":"2023-03-03T08:43:34.535451+0000","last_active":"2023-03-03T08:43:43.985919+0000","last_peered":"2023-03-03T08:43:43.985919+0000","last_clean":"2023-03-03T08:43:43.985919+0000","last_became_active":"2023-03-03T08:43:34.535129+0000","last_became_peered":"2023-03-03T08:43:34.535129+0000","last_unstale":"2023-03-03T08:43:43.985919+0000","last_undegraded":"2023-03-03T08:43:43.985919+0000","last_fullsized":"2023-03-03T08:43:43.985919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:30.645128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981082+0000","last_change":"2023-03-03T08:43:34.117851+0000","last_active":"2023-03-03T08:43:43.981082+0000","last_peered":"2023-03-03T08:43:43.981082+0000","last_clean":"2023-03-03T08:43:43.981082+0000","last_became_active":"2023-03-03T08:43:34.117548+0000","last_became_peered":"2023-03-03T08:43:34.117548+0000","last_unstale":"2023-03-03T08:43:43.981082+0000","last_undegraded":"2023-03-03T08:43:43.981082+0000","last_fullsized":"2023-03-03T08:43:43.981082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:20.039817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106904+0000","last_change":"2023-03-03T08:43:43.569082+0000","last_active":"2023-03-03T08:43:45.106904+0000","last_peered":"2023-03-03T08:43:45.106904+0000","last_clean":"2023-03-03T08:43:45.106904+0000","last_became_active":"2023-03-03T08:43:43.568838+0000","last_became_peered":"2023-03-03T08:43:43.568838+0000","last_unstale":"2023-03-03T08:43:45.106904+0000","last_undegraded":"2023-03-03T08:43:45.106904+0000","last_fullsized":"2023-03-03T08:43:45.106904+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:47.610848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979556+0000","last_change":"2023-03-03T08:43:43.995210+0000","last_active":"2023-03-03T08:43:44.979556+0000","last_peered":"2023-03-03T08:43:44.979556+0000","last_clean":"2023-03-03T08:43:44.979556+0000","last_became_active":"2023-03-03T08:43:43.994899+0000","last_became_peered":"2023-03-03T08:43:43.994899+0000","last_unstale":"2023-03-03T08:43:44.979556+0000","last_undegraded":"2023-03-03T08:43:44.979556+0000","last_fullsized":"2023-03-03T08:43:44.979556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:37:11.974933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985908+0000","last_change":"2023-03-03T08:43:34.547322+0000","last_active":"2023-03-03T08:43:43.985908+0000","last_peered":"2023-03-03T08:43:43.985908+0000","last_clean":"2023-03-03T08:43:43.985908+0000","last_became_active":"2023-03-03T08:43:34.546692+0000","last_became_peered":"2023-03-03T08:43:34.546692+0000","last_unstale":"2023-03-03T08:43:43.985908+0000","last_undegraded":"2023-03-03T08:43:43.985908+0000","last_fullsized":"2023-03-03T08:43:43.985908+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:11:26.366920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982836+0000","last_change":"2023-03-03T08:43:34.110187+0000","last_active":"2023-03-03T08:43:43.982836+0000","last_peered":"2023-03-03T08:43:43.982836+0000","last_clean":"2023-03-03T08:43:43.982836+0000","last_became_active":"2023-03-03T08:43:34.109845+0000","last_became_peered":"2023-03-03T08:43:34.109845+0000","last_unstale":"2023-03-03T08:43:43.982836+0000","last_undegraded":"2023-03-03T08:43:43.982836+0000","last_fullsized":"2023-03-03T08:43:43.982836+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:33.507500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105532+0000","last_change":"2023-03-03T08:43:43.576805+0000","last_active":"2023-03-03T08:43:45.105532+0000","last_peered":"2023-03-03T08:43:45.105532+0000","last_clean":"2023-03-03T08:43:45.105532+0000","last_became_active":"2023-03-03T08:43:43.576241+0000","last_became_peered":"2023-03-03T08:43:43.576241+0000","last_unstale":"2023-03-03T08:43:45.105532+0000","last_undegraded":"2023-03-03T08:43:45.105532+0000","last_fullsized":"2023-03-03T08:43:45.105532+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:34:35.821307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105632+0000","last_change":"2023-03-03T08:43:43.991142+0000","last_active":"2023-03-03T08:43:45.105632+0000","last_peered":"2023-03-03T08:43:45.105632+0000","last_clean":"2023-03-03T08:43:45.105632+0000","last_became_active":"2023-03-03T08:43:43.990868+0000","last_became_peered":"2023-03-03T08:43:43.990868+0000","last_unstale":"2023-03-03T08:43:45.105632+0000","last_undegraded":"2023-03-03T08:43:45.105632+0000","last_fullsized":"2023-03-03T08:43:45.105632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:58.831806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106429+0000","last_change":"2023-03-03T08:43:43.995618+0000","last_active":"2023-03-03T08:43:45.106429+0000","last_peered":"2023-03-03T08:43:45.106429+0000","last_clean":"2023-03-03T08:43:45.106429+0000","last_became_active":"2023-03-03T08:43:43.995350+0000","last_became_peered":"2023-03-03T08:43:43.995350+0000","last_unstale":"2023-03-03T08:43:45.106429+0000","last_undegraded":"2023-03-03T08:43:45.106429+0000","last_fullsized":"2023-03-03T08:43:45.106429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:46.445223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988843+0000","last_change":"2023-03-03T08:43:34.541172+0000","last_active":"2023-03-03T08:43:43.988843+0000","last_peered":"2023-03-03T08:43:43.988843+0000","last_clean":"2023-03-03T08:43:43.988843+0000","last_became_active":"2023-03-03T08:43:34.540739+0000","last_became_peered":"2023-03-03T08:43:34.540739+0000","last_unstale":"2023-03-03T08:43:43.988843+0000","last_undegraded":"2023-03-03T08:43:43.988843+0000","last_fullsized":"2023-03-03T08:43:43.988843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:40.058265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987475+0000","last_change":"2023-03-03T08:43:34.112667+0000","last_active":"2023-03-03T08:43:43.987475+0000","last_peered":"2023-03-03T08:43:43.987475+0000","last_clean":"2023-03-03T08:43:43.987475+0000","last_became_active":"2023-03-03T08:43:34.112386+0000","last_became_peered":"2023-03-03T08:43:34.112386+0000","last_unstale":"2023-03-03T08:43:43.987475+0000","last_undegraded":"2023-03-03T08:43:43.987475+0000","last_fullsized":"2023-03-03T08:43:43.987475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:40.504392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980167+0000","last_change":"2023-03-03T08:43:43.573962+0000","last_active":"2023-03-03T08:43:44.980167+0000","last_peered":"2023-03-03T08:43:44.980167+0000","last_clean":"2023-03-03T08:43:44.980167+0000","last_became_active":"2023-03-03T08:43:43.573200+0000","last_became_peered":"2023-03-03T08:43:43.573200+0000","last_unstale":"2023-03-03T08:43:44.980167+0000","last_undegraded":"2023-03-03T08:43:44.980167+0000","last_fullsized":"2023-03-03T08:43:44.980167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:29.511681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984698+0000","last_change":"2023-03-03T08:43:34.537850+0000","last_active":"2023-03-03T08:43:43.984698+0000","last_peered":"2023-03-03T08:43:43.984698+0000","last_clean":"2023-03-03T08:43:43.984698+0000","last_became_active":"2023-03-03T08:43:34.537550+0000","last_became_peered":"2023-03-03T08:43:34.537550+0000","last_unstale":"2023-03-03T08:43:43.984698+0000","last_undegraded":"2023-03-03T08:43:43.984698+0000","last_fullsized":"2023-03-03T08:43:43.984698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:37:36.893757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990299+0000","last_change":"2023-03-03T08:43:34.109127+0000","last_active":"2023-03-03T08:43:43.990299+0000","last_peered":"2023-03-03T08:43:43.990299+0000","last_clean":"2023-03-03T08:43:43.990299+0000","last_became_active":"2023-03-03T08:43:34.108869+0000","last_became_peered":"2023-03-03T08:43:34.108869+0000","last_unstale":"2023-03-03T08:43:43.990299+0000","last_undegraded":"2023-03-03T08:43:43.990299+0000","last_fullsized":"2023-03-03T08:43:43.990299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:19:00.275620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981298+0000","last_change":"2023-03-03T08:43:29.354627+0000","last_active":"2023-03-03T08:43:43.981298+0000","last_peered":"2023-03-03T08:43:43.981298+0000","last_clean":"2023-03-03T08:43:43.981298+0000","last_became_active":"2023-03-03T08:43:29.354271+0000","last_became_peered":"2023-03-03T08:43:29.354271+0000","last_unstale":"2023-03-03T08:43:43.981298+0000","last_undegraded":"2023-03-03T08:43:43.981298+0000","last_fullsized":"2023-03-03T08:43:43.981298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:05.946053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979432+0000","last_change":"2023-03-03T08:43:43.575567+0000","last_active":"2023-03-03T08:43:44.979432+0000","last_peered":"2023-03-03T08:43:44.979432+0000","last_clean":"2023-03-03T08:43:44.979432+0000","last_became_active":"2023-03-03T08:43:43.575276+0000","last_became_peered":"2023-03-03T08:43:43.575276+0000","last_unstale":"2023-03-03T08:43:44.979432+0000","last_undegraded":"2023-03-03T08:43:44.979432+0000","last_fullsized":"2023-03-03T08:43:44.979432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:12.876790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107250+0000","last_change":"2023-03-03T08:43:43.998276+0000","last_active":"2023-03-03T08:43:45.107250+0000","last_peered":"2023-03-03T08:43:45.107250+0000","last_clean":"2023-03-03T08:43:45.107250+0000","last_became_active":"2023-03-03T08:43:43.996018+0000","last_became_peered":"2023-03-03T08:43:43.996018+0000","last_unstale":"2023-03-03T08:43:45.107250+0000","last_undegraded":"2023-03-03T08:43:45.107250+0000","last_fullsized":"2023-03-03T08:43:45.107250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:08.588278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988844+0000","last_change":"2023-03-03T08:43:34.539018+0000","last_active":"2023-03-03T08:43:43.988844+0000","last_peered":"2023-03-03T08:43:43.988844+0000","last_clean":"2023-03-03T08:43:43.988844+0000","last_became_active":"2023-03-03T08:43:34.538375+0000","last_became_peered":"2023-03-03T08:43:34.538375+0000","last_unstale":"2023-03-03T08:43:43.988844+0000","last_undegraded":"2023-03-03T08:43:43.988844+0000","last_fullsized":"2023-03-03T08:43:43.988844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:29:02.601410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990482+0000","last_change":"2023-03-03T08:43:29.354519+0000","last_active":"2023-03-03T08:43:43.990482+0000","last_peered":"2023-03-03T08:43:43.990482+0000","last_clean":"2023-03-03T08:43:43.990482+0000","last_became_active":"2023-03-03T08:43:29.354213+0000","last_became_peered":"2023-03-03T08:43:29.354213+0000","last_unstale":"2023-03-03T08:43:43.990482+0000","last_undegraded":"2023-03-03T08:43:43.990482+0000","last_fullsized":"2023-03-03T08:43:43.990482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:00.406507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990028+0000","last_change":"2023-03-03T08:43:34.110105+0000","last_active":"2023-03-03T08:43:43.990028+0000","last_peered":"2023-03-03T08:43:43.990028+0000","last_clean":"2023-03-03T08:43:43.990028+0000","last_became_active":"2023-03-03T08:43:34.109736+0000","last_became_peered":"2023-03-03T08:43:34.109736+0000","last_unstale":"2023-03-03T08:43:43.990028+0000","last_undegraded":"2023-03-03T08:43:43.990028+0000","last_fullsized":"2023-03-03T08:43:43.990028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:30.557847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105361+0000","last_change":"2023-03-03T08:43:43.569433+0000","last_active":"2023-03-03T08:43:45.105361+0000","last_peered":"2023-03-03T08:43:45.105361+0000","last_clean":"2023-03-03T08:43:45.105361+0000","last_became_active":"2023-03-03T08:43:43.568711+0000","last_became_peered":"2023-03-03T08:43:43.568711+0000","last_unstale":"2023-03-03T08:43:45.105361+0000","last_undegraded":"2023-03-03T08:43:45.105361+0000","last_fullsized":"2023-03-03T08:43:45.105361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:53.170557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105548+0000","last_change":"2023-03-03T08:43:43.999241+0000","last_active":"2023-03-03T08:43:45.105548+0000","last_peered":"2023-03-03T08:43:45.105548+0000","last_clean":"2023-03-03T08:43:45.105548+0000","last_became_active":"2023-03-03T08:43:43.998992+0000","last_became_peered":"2023-03-03T08:43:43.998992+0000","last_unstale":"2023-03-03T08:43:45.105548+0000","last_undegraded":"2023-03-03T08:43:45.105548+0000","last_fullsized":"2023-03-03T08:43:45.105548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:22.623700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991092+0000","last_change":"2023-03-03T08:43:34.535435+0000","last_active":"2023-03-03T08:43:43.991092+0000","last_peered":"2023-03-03T08:43:43.991092+0000","last_clean":"2023-03-03T08:43:43.991092+0000","last_became_active":"2023-03-03T08:43:34.535123+0000","last_became_peered":"2023-03-03T08:43:34.535123+0000","last_unstale":"2023-03-03T08:43:43.991092+0000","last_undegraded":"2023-03-03T08:43:43.991092+0000","last_fullsized":"2023-03-03T08:43:43.991092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:52:10.622422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990432+0000","last_change":"2023-03-03T08:43:34.107783+0000","last_active":"2023-03-03T08:43:43.990432+0000","last_peered":"2023-03-03T08:43:43.990432+0000","last_clean":"2023-03-03T08:43:43.990432+0000","last_became_active":"2023-03-03T08:43:34.107503+0000","last_became_peered":"2023-03-03T08:43:34.107503+0000","last_unstale":"2023-03-03T08:43:43.990432+0000","last_undegraded":"2023-03-03T08:43:43.990432+0000","last_fullsized":"2023-03-03T08:43:43.990432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:35:42.800814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981038+0000","last_change":"2023-03-03T08:43:29.072304+0000","last_active":"2023-03-03T08:43:43.981038+0000","last_peered":"2023-03-03T08:43:43.981038+0000","last_clean":"2023-03-03T08:43:43.981038+0000","last_became_active":"2023-03-03T08:43:29.071962+0000","last_became_peered":"2023-03-03T08:43:29.071962+0000","last_unstale":"2023-03-03T08:43:43.981038+0000","last_undegraded":"2023-03-03T08:43:43.981038+0000","last_fullsized":"2023-03-03T08:43:43.981038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:40:12.229226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106349+0000","last_change":"2023-03-03T08:43:43.575964+0000","last_active":"2023-03-03T08:43:45.106349+0000","last_peered":"2023-03-03T08:43:45.106349+0000","last_clean":"2023-03-03T08:43:45.106349+0000","last_became_active":"2023-03-03T08:43:43.575697+0000","last_became_peered":"2023-03-03T08:43:43.575697+0000","last_unstale":"2023-03-03T08:43:45.106349+0000","last_undegraded":"2023-03-03T08:43:45.106349+0000","last_fullsized":"2023-03-03T08:43:45.106349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:01.896050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978782+0000","last_change":"2023-03-03T08:43:44.006414+0000","last_active":"2023-03-03T08:43:44.978782+0000","last_peered":"2023-03-03T08:43:44.978782+0000","last_clean":"2023-03-03T08:43:44.978782+0000","last_became_active":"2023-03-03T08:43:44.006132+0000","last_became_peered":"2023-03-03T08:43:44.006132+0000","last_unstale":"2023-03-03T08:43:44.978782+0000","last_undegraded":"2023-03-03T08:43:44.978782+0000","last_fullsized":"2023-03-03T08:43:44.978782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:55.006419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982772+0000","last_change":"2023-03-03T08:43:34.544053+0000","last_active":"2023-03-03T08:43:43.982772+0000","last_peered":"2023-03-03T08:43:43.982772+0000","last_clean":"2023-03-03T08:43:43.982772+0000","last_became_active":"2023-03-03T08:43:34.543523+0000","last_became_peered":"2023-03-03T08:43:34.543523+0000","last_unstale":"2023-03-03T08:43:43.982772+0000","last_undegraded":"2023-03-03T08:43:43.982772+0000","last_fullsized":"2023-03-03T08:43:43.982772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:34:03.662535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989069+0000","last_change":"2023-03-03T08:43:34.104492+0000","last_active":"2023-03-03T08:43:43.989069+0000","last_peered":"2023-03-03T08:43:43.989069+0000","last_clean":"2023-03-03T08:43:43.989069+0000","last_became_active":"2023-03-03T08:43:34.104151+0000","last_became_peered":"2023-03-03T08:43:34.104151+0000","last_unstale":"2023-03-03T08:43:43.989069+0000","last_undegraded":"2023-03-03T08:43:43.989069+0000","last_fullsized":"2023-03-03T08:43:43.989069+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:57.214738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979395+0000","last_change":"2023-03-03T08:43:29.075836+0000","last_active":"2023-03-03T08:43:43.979395+0000","last_peered":"2023-03-03T08:43:43.979395+0000","last_clean":"2023-03-03T08:43:43.979395+0000","last_became_active":"2023-03-03T08:43:29.075564+0000","last_became_peered":"2023-03-03T08:43:29.075564+0000","last_unstale":"2023-03-03T08:43:43.979395+0000","last_undegraded":"2023-03-03T08:43:43.979395+0000","last_fullsized":"2023-03-03T08:43:43.979395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:39.576714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107224+0000","last_change":"2023-03-03T08:43:43.575599+0000","last_active":"2023-03-03T08:43:45.107224+0000","last_peered":"2023-03-03T08:43:45.107224+0000","last_clean":"2023-03-03T08:43:45.107224+0000","last_became_active":"2023-03-03T08:43:43.575318+0000","last_became_peered":"2023-03-03T08:43:43.575318+0000","last_unstale":"2023-03-03T08:43:45.107224+0000","last_undegraded":"2023-03-03T08:43:45.107224+0000","last_fullsized":"2023-03-03T08:43:45.107224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:28.806052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105269+0000","last_change":"2023-03-03T08:43:44.000500+0000","last_active":"2023-03-03T08:43:45.105269+0000","last_peered":"2023-03-03T08:43:45.105269+0000","last_clean":"2023-03-03T08:43:45.105269+0000","last_became_active":"2023-03-03T08:43:43.999560+0000","last_became_peered":"2023-03-03T08:43:43.999560+0000","last_unstale":"2023-03-03T08:43:45.105269+0000","last_undegraded":"2023-03-03T08:43:45.105269+0000","last_fullsized":"2023-03-03T08:43:45.105269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:29:54.240309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981172+0000","last_change":"2023-03-03T08:43:34.550263+0000","last_active":"2023-03-03T08:43:43.981172+0000","last_peered":"2023-03-03T08:43:43.981172+0000","last_clean":"2023-03-03T08:43:43.981172+0000","last_became_active":"2023-03-03T08:43:34.549560+0000","last_became_peered":"2023-03-03T08:43:34.549560+0000","last_unstale":"2023-03-03T08:43:43.981172+0000","last_undegraded":"2023-03-03T08:43:43.981172+0000","last_fullsized":"2023-03-03T08:43:43.981172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:55.874223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987062+0000","last_change":"2023-03-03T08:43:34.110484+0000","last_active":"2023-03-03T08:43:43.987062+0000","last_peered":"2023-03-03T08:43:43.987062+0000","last_clean":"2023-03-03T08:43:43.987062+0000","last_became_active":"2023-03-03T08:43:34.110220+0000","last_became_peered":"2023-03-03T08:43:34.110220+0000","last_unstale":"2023-03-03T08:43:43.987062+0000","last_undegraded":"2023-03-03T08:43:43.987062+0000","last_fullsized":"2023-03-03T08:43:43.987062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:04:59.232537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982439+0000","last_change":"2023-03-03T08:43:23.015460+0000","last_active":"2023-03-03T08:43:43.982439+0000","last_peered":"2023-03-03T08:43:43.982439+0000","last_clean":"2023-03-03T08:43:43.982439+0000","last_became_active":"2023-03-03T08:43:23.014485+0000","last_became_peered":"2023-03-03T08:43:23.014485+0000","last_unstale":"2023-03-03T08:43:43.982439+0000","last_undegraded":"2023-03-03T08:43:43.982439+0000","last_fullsized":"2023-03-03T08:43:43.982439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_clean_scrub_stamp":"2023-03-03T08:43:21.542883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:07.022017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980803+0000","last_change":"2023-03-03T08:43:29.160285+0000","last_active":"2023-03-03T08:43:43.980803+0000","last_peered":"2023-03-03T08:43:43.980803+0000","last_clean":"2023-03-03T08:43:43.980803+0000","last_became_active":"2023-03-03T08:43:29.159943+0000","last_became_peered":"2023-03-03T08:43:29.159943+0000","last_unstale":"2023-03-03T08:43:43.980803+0000","last_undegraded":"2023-03-03T08:43:43.980803+0000","last_fullsized":"2023-03-03T08:43:43.980803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:48:58.997356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107580+0000","last_change":"2023-03-03T08:43:43.575212+0000","last_active":"2023-03-03T08:43:45.107580+0000","last_peered":"2023-03-03T08:43:45.107580+0000","last_clean":"2023-03-03T08:43:45.107580+0000","last_became_active":"2023-03-03T08:43:43.574897+0000","last_became_peered":"2023-03-03T08:43:43.574897+0000","last_unstale":"2023-03-03T08:43:45.107580+0000","last_undegraded":"2023-03-03T08:43:45.107580+0000","last_fullsized":"2023-03-03T08:43:45.107580+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:33:07.674313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107370+0000","last_change":"2023-03-03T08:43:43.992952+0000","last_active":"2023-03-03T08:43:45.107370+0000","last_peered":"2023-03-03T08:43:45.107370+0000","last_clean":"2023-03-03T08:43:45.107370+0000","last_became_active":"2023-03-03T08:43:43.992535+0000","last_became_peered":"2023-03-03T08:43:43.992535+0000","last_unstale":"2023-03-03T08:43:45.107370+0000","last_undegraded":"2023-03-03T08:43:45.107370+0000","last_fullsized":"2023-03-03T08:43:45.107370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:10:41.617430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991335+0000","last_change":"2023-03-03T08:43:34.536419+0000","last_active":"2023-03-03T08:43:43.991335+0000","last_peered":"2023-03-03T08:43:43.991335+0000","last_clean":"2023-03-03T08:43:43.991335+0000","last_became_active":"2023-03-03T08:43:34.536150+0000","last_became_peered":"2023-03-03T08:43:34.536150+0000","last_unstale":"2023-03-03T08:43:43.991335+0000","last_undegraded":"2023-03-03T08:43:43.991335+0000","last_fullsized":"2023-03-03T08:43:43.991335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:22:54.362024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981827+0000","last_change":"2023-03-03T08:43:34.106337+0000","last_active":"2023-03-03T08:43:43.981827+0000","last_peered":"2023-03-03T08:43:43.981827+0000","last_clean":"2023-03-03T08:43:43.981827+0000","last_became_active":"2023-03-03T08:43:34.105753+0000","last_became_peered":"2023-03-03T08:43:34.105753+0000","last_unstale":"2023-03-03T08:43:43.981827+0000","last_undegraded":"2023-03-03T08:43:43.981827+0000","last_fullsized":"2023-03-03T08:43:43.981827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:46.761924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991590+0000","last_change":"2023-03-03T08:43:29.076367+0000","last_active":"2023-03-03T08:43:43.991590+0000","last_peered":"2023-03-03T08:43:43.991590+0000","last_clean":"2023-03-03T08:43:43.991590+0000","last_became_active":"2023-03-03T08:43:29.075982+0000","last_became_peered":"2023-03-03T08:43:29.075982+0000","last_unstale":"2023-03-03T08:43:43.991590+0000","last_undegraded":"2023-03-03T08:43:43.991590+0000","last_fullsized":"2023-03-03T08:43:43.991590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:06.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105779+0000","last_change":"2023-03-03T08:43:43.577670+0000","last_active":"2023-03-03T08:43:45.105779+0000","last_peered":"2023-03-03T08:43:45.105779+0000","last_clean":"2023-03-03T08:43:45.105779+0000","last_became_active":"2023-03-03T08:43:43.577371+0000","last_became_peered":"2023-03-03T08:43:43.577371+0000","last_unstale":"2023-03-03T08:43:45.105779+0000","last_undegraded":"2023-03-03T08:43:45.105779+0000","last_fullsized":"2023-03-03T08:43:45.105779+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:18.377654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108041+0000","last_change":"2023-03-03T08:43:43.993077+0000","last_active":"2023-03-03T08:43:45.108041+0000","last_peered":"2023-03-03T08:43:45.108041+0000","last_clean":"2023-03-03T08:43:45.108041+0000","last_became_active":"2023-03-03T08:43:43.992021+0000","last_became_peered":"2023-03-03T08:43:43.992021+0000","last_unstale":"2023-03-03T08:43:45.108041+0000","last_undegraded":"2023-03-03T08:43:45.108041+0000","last_fullsized":"2023-03-03T08:43:45.108041+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:58.237553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990193+0000","last_change":"2023-03-03T08:43:34.528593+0000","last_active":"2023-03-03T08:43:43.990193+0000","last_peered":"2023-03-03T08:43:43.990193+0000","last_clean":"2023-03-03T08:43:43.990193+0000","last_became_active":"2023-03-03T08:43:34.528260+0000","last_became_peered":"2023-03-03T08:43:34.528260+0000","last_unstale":"2023-03-03T08:43:43.990193+0000","last_undegraded":"2023-03-03T08:43:43.990193+0000","last_fullsized":"2023-03-03T08:43:43.990193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988924+0000","last_change":"2023-03-03T08:43:34.113748+0000","last_active":"2023-03-03T08:43:43.988924+0000","last_peered":"2023-03-03T08:43:43.988924+0000","last_clean":"2023-03-03T08:43:43.988924+0000","last_became_active":"2023-03-03T08:43:34.113190+0000","last_became_peered":"2023-03-03T08:43:34.113190+0000","last_unstale":"2023-03-03T08:43:43.988924+0000","last_undegraded":"2023-03-03T08:43:43.988924+0000","last_fullsized":"2023-03-03T08:43:43.988924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:58:55.343571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980485+0000","last_change":"2023-03-03T08:43:29.107584+0000","last_active":"2023-03-03T08:43:43.980485+0000","last_peered":"2023-03-03T08:43:43.980485+0000","last_clean":"2023-03-03T08:43:43.980485+0000","last_became_active":"2023-03-03T08:43:29.107208+0000","last_became_peered":"2023-03-03T08:43:29.107208+0000","last_unstale":"2023-03-03T08:43:43.980485+0000","last_undegraded":"2023-03-03T08:43:43.980485+0000","last_fullsized":"2023-03-03T08:43:43.980485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:28:24.087122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106004+0000","last_change":"2023-03-03T08:43:43.575742+0000","last_active":"2023-03-03T08:43:45.106004+0000","last_peered":"2023-03-03T08:43:45.106004+0000","last_clean":"2023-03-03T08:43:45.106004+0000","last_became_active":"2023-03-03T08:43:43.575456+0000","last_became_peered":"2023-03-03T08:43:43.575456+0000","last_unstale":"2023-03-03T08:43:45.106004+0000","last_undegraded":"2023-03-03T08:43:45.106004+0000","last_fullsized":"2023-03-03T08:43:45.106004+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:46:17.635306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106635+0000","last_change":"2023-03-03T08:43:43.985961+0000","last_active":"2023-03-03T08:43:45.106635+0000","last_peered":"2023-03-03T08:43:45.106635+0000","last_clean":"2023-03-03T08:43:45.106635+0000","last_became_active":"2023-03-03T08:43:43.985633+0000","last_became_peered":"2023-03-03T08:43:43.985633+0000","last_unstale":"2023-03-03T08:43:45.106635+0000","last_undegraded":"2023-03-03T08:43:45.106635+0000","last_fullsized":"2023-03-03T08:43:45.106635+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:52.517910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989789+0000","last_change":"2023-03-03T08:43:34.526797+0000","last_active":"2023-03-03T08:43:43.989789+0000","last_peered":"2023-03-03T08:43:43.989789+0000","last_clean":"2023-03-03T08:43:43.989789+0000","last_became_active":"2023-03-03T08:43:34.526490+0000","last_became_peered":"2023-03-03T08:43:34.526490+0000","last_unstale":"2023-03-03T08:43:43.989789+0000","last_undegraded":"2023-03-03T08:43:43.989789+0000","last_fullsized":"2023-03-03T08:43:43.989789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:11.808336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986004+0000","last_change":"2023-03-03T08:43:34.104692+0000","last_active":"2023-03-03T08:43:43.986004+0000","last_peered":"2023-03-03T08:43:43.986004+0000","last_clean":"2023-03-03T08:43:43.986004+0000","last_became_active":"2023-03-03T08:43:34.104263+0000","last_became_peered":"2023-03-03T08:43:34.104263+0000","last_unstale":"2023-03-03T08:43:43.986004+0000","last_undegraded":"2023-03-03T08:43:43.986004+0000","last_fullsized":"2023-03-03T08:43:43.986004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:24.537703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983049+0000","last_change":"2023-03-03T08:43:29.107536+0000","last_active":"2023-03-03T08:43:43.983049+0000","last_peered":"2023-03-03T08:43:43.983049+0000","last_clean":"2023-03-03T08:43:43.983049+0000","last_became_active":"2023-03-03T08:43:29.107093+0000","last_became_peered":"2023-03-03T08:43:29.107093+0000","last_unstale":"2023-03-03T08:43:43.983049+0000","last_undegraded":"2023-03-03T08:43:43.983049+0000","last_fullsized":"2023-03-03T08:43:43.983049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:25:47.568904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980025+0000","last_change":"2023-03-03T08:43:43.577221+0000","last_active":"2023-03-03T08:43:44.980025+0000","last_peered":"2023-03-03T08:43:44.980025+0000","last_clean":"2023-03-03T08:43:44.980025+0000","last_became_active":"2023-03-03T08:43:43.576942+0000","last_became_peered":"2023-03-03T08:43:43.576942+0000","last_unstale":"2023-03-03T08:43:44.980025+0000","last_undegraded":"2023-03-03T08:43:44.980025+0000","last_fullsized":"2023-03-03T08:43:44.980025+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:13:09.716078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105102+0000","last_change":"2023-03-03T08:43:44.000318+0000","last_active":"2023-03-03T08:43:45.105102+0000","last_peered":"2023-03-03T08:43:45.105102+0000","last_clean":"2023-03-03T08:43:45.105102+0000","last_became_active":"2023-03-03T08:43:44.000053+0000","last_became_peered":"2023-03-03T08:43:44.000053+0000","last_unstale":"2023-03-03T08:43:45.105102+0000","last_undegraded":"2023-03-03T08:43:45.105102+0000","last_fullsized":"2023-03-03T08:43:45.105102+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:31.059774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994113+0000","last_change":"2023-03-03T08:43:34.540006+0000","last_active":"2023-03-03T08:43:43.994113+0000","last_peered":"2023-03-03T08:43:43.994113+0000","last_clean":"2023-03-03T08:43:43.994113+0000","last_became_active":"2023-03-03T08:43:34.539579+0000","last_became_peered":"2023-03-03T08:43:34.539579+0000","last_unstale":"2023-03-03T08:43:43.994113+0000","last_undegraded":"2023-03-03T08:43:43.994113+0000","last_fullsized":"2023-03-03T08:43:43.994113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988985+0000","last_change":"2023-03-03T08:43:34.109764+0000","last_active":"2023-03-03T08:43:43.988985+0000","last_peered":"2023-03-03T08:43:43.988985+0000","last_clean":"2023-03-03T08:43:43.988985+0000","last_became_active":"2023-03-03T08:43:34.109462+0000","last_became_peered":"2023-03-03T08:43:34.109462+0000","last_unstale":"2023-03-03T08:43:43.988985+0000","last_undegraded":"2023-03-03T08:43:43.988985+0000","last_fullsized":"2023-03-03T08:43:43.988985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:56.399192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106570+0000","last_change":"2023-03-03T08:43:43.576402+0000","last_active":"2023-03-03T08:43:45.106570+0000","last_peered":"2023-03-03T08:43:45.106570+0000","last_clean":"2023-03-03T08:43:45.106570+0000","last_became_active":"2023-03-03T08:43:43.576113+0000","last_became_peered":"2023-03-03T08:43:43.576113+0000","last_unstale":"2023-03-03T08:43:45.106570+0000","last_undegraded":"2023-03-03T08:43:45.106570+0000","last_fullsized":"2023-03-03T08:43:45.106570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:33:26.483688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106437+0000","last_change":"2023-03-03T08:43:44.002074+0000","last_active":"2023-03-03T08:43:45.106437+0000","last_peered":"2023-03-03T08:43:45.106437+0000","last_clean":"2023-03-03T08:43:45.106437+0000","last_became_active":"2023-03-03T08:43:44.001803+0000","last_became_peered":"2023-03-03T08:43:44.001803+0000","last_unstale":"2023-03-03T08:43:45.106437+0000","last_undegraded":"2023-03-03T08:43:45.106437+0000","last_fullsized":"2023-03-03T08:43:45.106437+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:15:16.115135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986358+0000","last_change":"2023-03-03T08:43:34.541858+0000","last_active":"2023-03-03T08:43:43.986358+0000","last_peered":"2023-03-03T08:43:43.986358+0000","last_clean":"2023-03-03T08:43:43.986358+0000","last_became_active":"2023-03-03T08:43:34.541430+0000","last_became_peered":"2023-03-03T08:43:34.541430+0000","last_unstale":"2023-03-03T08:43:43.986358+0000","last_undegraded":"2023-03-03T08:43:43.986358+0000","last_fullsized":"2023-03-03T08:43:43.986358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:29.397784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979830+0000","last_change":"2023-03-03T08:43:34.111335+0000","last_active":"2023-03-03T08:43:43.979830+0000","last_peered":"2023-03-03T08:43:43.979830+0000","last_clean":"2023-03-03T08:43:43.979830+0000","last_became_active":"2023-03-03T08:43:34.110837+0000","last_became_peered":"2023-03-03T08:43:34.110837+0000","last_unstale":"2023-03-03T08:43:43.979830+0000","last_undegraded":"2023-03-03T08:43:43.979830+0000","last_fullsized":"2023-03-03T08:43:43.979830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:03.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106574+0000","last_change":"2023-03-03T08:43:43.577539+0000","last_active":"2023-03-03T08:43:45.106574+0000","last_peered":"2023-03-03T08:43:45.106574+0000","last_clean":"2023-03-03T08:43:45.106574+0000","last_became_active":"2023-03-03T08:43:43.577209+0000","last_became_peered":"2023-03-03T08:43:43.577209+0000","last_unstale":"2023-03-03T08:43:45.106574+0000","last_undegraded":"2023-03-03T08:43:45.106574+0000","last_fullsized":"2023-03-03T08:43:45.106574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:31:01.430495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979296+0000","last_change":"2023-03-03T08:43:43.993565+0000","last_active":"2023-03-03T08:43:44.979296+0000","last_peered":"2023-03-03T08:43:44.979296+0000","last_clean":"2023-03-03T08:43:44.979296+0000","last_became_active":"2023-03-03T08:43:43.992653+0000","last_became_peered":"2023-03-03T08:43:43.992653+0000","last_unstale":"2023-03-03T08:43:44.979296+0000","last_undegraded":"2023-03-03T08:43:44.979296+0000","last_fullsized":"2023-03-03T08:43:44.979296+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:24.295462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981121+0000","last_change":"2023-03-03T08:43:34.550325+0000","last_active":"2023-03-03T08:43:43.981121+0000","last_peered":"2023-03-03T08:43:43.981121+0000","last_clean":"2023-03-03T08:43:43.981121+0000","last_became_active":"2023-03-03T08:43:34.550041+0000","last_became_peered":"2023-03-03T08:43:34.550041+0000","last_unstale":"2023-03-03T08:43:43.981121+0000","last_undegraded":"2023-03-03T08:43:43.981121+0000","last_fullsized":"2023-03-03T08:43:43.981121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:44:34.082003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980457+0000","last_change":"2023-03-03T08:43:34.106777+0000","last_active":"2023-03-03T08:43:43.980457+0000","last_peered":"2023-03-03T08:43:43.980457+0000","last_clean":"2023-03-03T08:43:43.980457+0000","last_became_active":"2023-03-03T08:43:34.106132+0000","last_became_peered":"2023-03-03T08:43:34.106132+0000","last_unstale":"2023-03-03T08:43:43.980457+0000","last_undegraded":"2023-03-03T08:43:43.980457+0000","last_fullsized":"2023-03-03T08:43:43.980457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:55.965987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106224+0000","last_change":"2023-03-03T08:43:43.580312+0000","last_active":"2023-03-03T08:43:45.106224+0000","last_peered":"2023-03-03T08:43:45.106224+0000","last_clean":"2023-03-03T08:43:45.106224+0000","last_became_active":"2023-03-03T08:43:43.580096+0000","last_became_peered":"2023-03-03T08:43:43.580096+0000","last_unstale":"2023-03-03T08:43:45.106224+0000","last_undegraded":"2023-03-03T08:43:45.106224+0000","last_fullsized":"2023-03-03T08:43:45.106224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:06:03.500778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106765+0000","last_change":"2023-03-03T08:43:43.997453+0000","last_active":"2023-03-03T08:43:45.106765+0000","last_peered":"2023-03-03T08:43:45.106765+0000","last_clean":"2023-03-03T08:43:45.106765+0000","last_became_active":"2023-03-03T08:43:43.997090+0000","last_became_peered":"2023-03-03T08:43:43.997090+0000","last_unstale":"2023-03-03T08:43:45.106765+0000","last_undegraded":"2023-03-03T08:43:45.106765+0000","last_fullsized":"2023-03-03T08:43:45.106765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:28:58.322823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985493+0000","last_change":"2023-03-03T08:43:34.542308+0000","last_active":"2023-03-03T08:43:43.985493+0000","last_peered":"2023-03-03T08:43:43.985493+0000","last_clean":"2023-03-03T08:43:43.985493+0000","last_became_active":"2023-03-03T08:43:34.541991+0000","last_became_peered":"2023-03-03T08:43:34.541991+0000","last_unstale":"2023-03-03T08:43:43.985493+0000","last_undegraded":"2023-03-03T08:43:43.985493+0000","last_fullsized":"2023-03-03T08:43:43.985493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:06:21.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992335+0000","last_change":"2023-03-03T08:43:34.109698+0000","last_active":"2023-03-03T08:43:43.992335+0000","last_peered":"2023-03-03T08:43:43.992335+0000","last_clean":"2023-03-03T08:43:43.992335+0000","last_became_active":"2023-03-03T08:43:34.109423+0000","last_became_peered":"2023-03-03T08:43:34.109423+0000","last_unstale":"2023-03-03T08:43:43.992335+0000","last_undegraded":"2023-03-03T08:43:43.992335+0000","last_fullsized":"2023-03-03T08:43:43.992335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:49.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105911+0000","last_change":"2023-03-03T08:43:43.573925+0000","last_active":"2023-03-03T08:43:45.105911+0000","last_peered":"2023-03-03T08:43:45.105911+0000","last_clean":"2023-03-03T08:43:45.105911+0000","last_became_active":"2023-03-03T08:43:43.573474+0000","last_became_peered":"2023-03-03T08:43:43.573474+0000","last_unstale":"2023-03-03T08:43:45.105911+0000","last_undegraded":"2023-03-03T08:43:45.105911+0000","last_fullsized":"2023-03-03T08:43:45.105911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:47.858726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107653+0000","last_change":"2023-03-03T08:43:43.995602+0000","last_active":"2023-03-03T08:43:45.107653+0000","last_peered":"2023-03-03T08:43:45.107653+0000","last_clean":"2023-03-03T08:43:45.107653+0000","last_became_active":"2023-03-03T08:43:43.995404+0000","last_became_peered":"2023-03-03T08:43:43.995404+0000","last_unstale":"2023-03-03T08:43:45.107653+0000","last_undegraded":"2023-03-03T08:43:45.107653+0000","last_fullsized":"2023-03-03T08:43:45.107653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:37.212119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982423+0000","last_change":"2023-03-03T08:43:34.548304+0000","last_active":"2023-03-03T08:43:43.982423+0000","last_peered":"2023-03-03T08:43:43.982423+0000","last_clean":"2023-03-03T08:43:43.982423+0000","last_became_active":"2023-03-03T08:43:34.547960+0000","last_became_peered":"2023-03-03T08:43:34.547960+0000","last_unstale":"2023-03-03T08:43:43.982423+0000","last_undegraded":"2023-03-03T08:43:43.982423+0000","last_fullsized":"2023-03-03T08:43:43.982423+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:04.907753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985574+0000","last_change":"2023-03-03T08:43:34.109372+0000","last_active":"2023-03-03T08:43:43.985574+0000","last_peered":"2023-03-03T08:43:43.985574+0000","last_clean":"2023-03-03T08:43:43.985574+0000","last_became_active":"2023-03-03T08:43:34.109023+0000","last_became_peered":"2023-03-03T08:43:34.109023+0000","last_unstale":"2023-03-03T08:43:43.985574+0000","last_undegraded":"2023-03-03T08:43:43.985574+0000","last_fullsized":"2023-03-03T08:43:43.985574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:51.374188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978866+0000","last_change":"2023-03-03T08:43:43.572240+0000","last_active":"2023-03-03T08:43:44.978866+0000","last_peered":"2023-03-03T08:43:44.978866+0000","last_clean":"2023-03-03T08:43:44.978866+0000","last_became_active":"2023-03-03T08:43:43.571753+0000","last_became_peered":"2023-03-03T08:43:43.571753+0000","last_unstale":"2023-03-03T08:43:44.978866+0000","last_undegraded":"2023-03-03T08:43:44.978866+0000","last_fullsized":"2023-03-03T08:43:44.978866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:02:30.521667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979027+0000","last_change":"2023-03-03T08:43:43.995018+0000","last_active":"2023-03-03T08:43:44.979027+0000","last_peered":"2023-03-03T08:43:44.979027+0000","last_clean":"2023-03-03T08:43:44.979027+0000","last_became_active":"2023-03-03T08:43:43.994690+0000","last_became_peered":"2023-03-03T08:43:43.994690+0000","last_unstale":"2023-03-03T08:43:44.979027+0000","last_undegraded":"2023-03-03T08:43:44.979027+0000","last_fullsized":"2023-03-03T08:43:44.979027+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:06:13.979929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986022+0000","last_change":"2023-03-03T08:43:34.538817+0000","last_active":"2023-03-03T08:43:43.986022+0000","last_peered":"2023-03-03T08:43:43.986022+0000","last_clean":"2023-03-03T08:43:43.986022+0000","last_became_active":"2023-03-03T08:43:34.538506+0000","last_became_peered":"2023-03-03T08:43:34.538506+0000","last_unstale":"2023-03-03T08:43:43.986022+0000","last_undegraded":"2023-03-03T08:43:43.986022+0000","last_fullsized":"2023-03-03T08:43:43.986022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980930+0000","last_change":"2023-03-03T08:43:34.117734+0000","last_active":"2023-03-03T08:43:43.980930+0000","last_peered":"2023-03-03T08:43:43.980930+0000","last_clean":"2023-03-03T08:43:43.980930+0000","last_became_active":"2023-03-03T08:43:34.117454+0000","last_became_peered":"2023-03-03T08:43:34.117454+0000","last_unstale":"2023-03-03T08:43:43.980930+0000","last_undegraded":"2023-03-03T08:43:43.980930+0000","last_fullsized":"2023-03-03T08:43:43.980930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:42:27.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106434+0000","last_change":"2023-03-03T08:43:43.574955+0000","last_active":"2023-03-03T08:43:45.106434+0000","last_peered":"2023-03-03T08:43:45.106434+0000","last_clean":"2023-03-03T08:43:45.106434+0000","last_became_active":"2023-03-03T08:43:43.574677+0000","last_became_peered":"2023-03-03T08:43:43.574677+0000","last_unstale":"2023-03-03T08:43:45.106434+0000","last_undegraded":"2023-03-03T08:43:45.106434+0000","last_fullsized":"2023-03-03T08:43:45.106434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:12:19.538291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.104954+0000","last_change":"2023-03-03T08:43:43.994888+0000","last_active":"2023-03-03T08:43:45.104954+0000","last_peered":"2023-03-03T08:43:45.104954+0000","last_clean":"2023-03-03T08:43:45.104954+0000","last_became_active":"2023-03-03T08:43:43.994630+0000","last_became_peered":"2023-03-03T08:43:43.994630+0000","last_unstale":"2023-03-03T08:43:45.104954+0000","last_undegraded":"2023-03-03T08:43:45.104954+0000","last_fullsized":"2023-03-03T08:43:45.104954+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:15:18.412569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982579+0000","last_change":"2023-03-03T08:43:34.548010+0000","last_active":"2023-03-03T08:43:43.982579+0000","last_peered":"2023-03-03T08:43:43.982579+0000","last_clean":"2023-03-03T08:43:43.982579+0000","last_became_active":"2023-03-03T08:43:34.547662+0000","last_became_peered":"2023-03-03T08:43:34.547662+0000","last_unstale":"2023-03-03T08:43:43.982579+0000","last_undegraded":"2023-03-03T08:43:43.982579+0000","last_fullsized":"2023-03-03T08:43:43.982579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:52.200064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986286+0000","last_change":"2023-03-03T08:43:34.109890+0000","last_active":"2023-03-03T08:43:43.986286+0000","last_peered":"2023-03-03T08:43:43.986286+0000","last_clean":"2023-03-03T08:43:43.986286+0000","last_became_active":"2023-03-03T08:43:34.109583+0000","last_became_peered":"2023-03-03T08:43:34.109583+0000","last_unstale":"2023-03-03T08:43:43.986286+0000","last_undegraded":"2023-03-03T08:43:43.986286+0000","last_fullsized":"2023-03-03T08:43:43.986286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:27:34.142294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978233+0000","last_change":"2023-03-03T08:43:43.575945+0000","last_active":"2023-03-03T08:43:44.978233+0000","last_peered":"2023-03-03T08:43:44.978233+0000","last_clean":"2023-03-03T08:43:44.978233+0000","last_became_active":"2023-03-03T08:43:43.575675+0000","last_became_peered":"2023-03-03T08:43:43.575675+0000","last_unstale":"2023-03-03T08:43:44.978233+0000","last_undegraded":"2023-03-03T08:43:44.978233+0000","last_fullsized":"2023-03-03T08:43:44.978233+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:48.730805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980103+0000","last_change":"2023-03-03T08:43:43.998435+0000","last_active":"2023-03-03T08:43:44.980103+0000","last_peered":"2023-03-03T08:43:44.980103+0000","last_clean":"2023-03-03T08:43:44.980103+0000","last_became_active":"2023-03-03T08:43:43.996508+0000","last_became_peered":"2023-03-03T08:43:43.996508+0000","last_unstale":"2023-03-03T08:43:44.980103+0000","last_undegraded":"2023-03-03T08:43:44.980103+0000","last_fullsized":"2023-03-03T08:43:44.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:31:36.310851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979477+0000","last_change":"2023-03-03T08:43:34.540196+0000","last_active":"2023-03-03T08:43:43.979477+0000","last_peered":"2023-03-03T08:43:43.979477+0000","last_clean":"2023-03-03T08:43:43.979477+0000","last_became_active":"2023-03-03T08:43:34.539931+0000","last_became_peered":"2023-03-03T08:43:34.539931+0000","last_unstale":"2023-03-03T08:43:43.979477+0000","last_undegraded":"2023-03-03T08:43:43.979477+0000","last_fullsized":"2023-03-03T08:43:43.979477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:20.012997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990881+0000","last_change":"2023-03-03T08:43:34.110464+0000","last_active":"2023-03-03T08:43:43.990881+0000","last_peered":"2023-03-03T08:43:43.990881+0000","last_clean":"2023-03-03T08:43:43.990881+0000","last_became_active":"2023-03-03T08:43:34.110165+0000","last_became_peered":"2023-03-03T08:43:34.110165+0000","last_unstale":"2023-03-03T08:43:43.990881+0000","last_undegraded":"2023-03-03T08:43:43.990881+0000","last_fullsized":"2023-03-03T08:43:43.990881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:54.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106139+0000","last_change":"2023-03-03T08:43:43.578298+0000","last_active":"2023-03-03T08:43:45.106139+0000","last_peered":"2023-03-03T08:43:45.106139+0000","last_clean":"2023-03-03T08:43:45.106139+0000","last_became_active":"2023-03-03T08:43:43.578024+0000","last_became_peered":"2023-03-03T08:43:43.578024+0000","last_unstale":"2023-03-03T08:43:45.106139+0000","last_undegraded":"2023-03-03T08:43:45.106139+0000","last_fullsized":"2023-03-03T08:43:45.106139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:00:44.633716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106526+0000","last_change":"2023-03-03T08:43:43.992906+0000","last_active":"2023-03-03T08:43:45.106526+0000","last_peered":"2023-03-03T08:43:45.106526+0000","last_clean":"2023-03-03T08:43:45.106526+0000","last_became_active":"2023-03-03T08:43:43.991823+0000","last_became_peered":"2023-03-03T08:43:43.991823+0000","last_unstale":"2023-03-03T08:43:45.106526+0000","last_undegraded":"2023-03-03T08:43:45.106526+0000","last_fullsized":"2023-03-03T08:43:45.106526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:34.431216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.553597+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.553322+0000","last_became_peered":"2023-03-03T08:43:34.553322+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:24:17.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983377+0000","last_change":"2023-03-03T08:43:34.108123+0000","last_active":"2023-03-03T08:43:43.983377+0000","last_peered":"2023-03-03T08:43:43.983377+0000","last_clean":"2023-03-03T08:43:43.983377+0000","last_became_active":"2023-03-03T08:43:34.107757+0000","last_became_peered":"2023-03-03T08:43:34.107757+0000","last_unstale":"2023-03-03T08:43:43.983377+0000","last_undegraded":"2023-03-03T08:43:43.983377+0000","last_fullsized":"2023-03-03T08:43:43.983377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:09.488930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106648+0000","last_change":"2023-03-03T08:43:43.573369+0000","last_active":"2023-03-03T08:43:45.106648+0000","last_peered":"2023-03-03T08:43:45.106648+0000","last_clean":"2023-03-03T08:43:45.106648+0000","last_became_active":"2023-03-03T08:43:43.573069+0000","last_became_peered":"2023-03-03T08:43:43.573069+0000","last_unstale":"2023-03-03T08:43:45.106648+0000","last_undegraded":"2023-03-03T08:43:45.106648+0000","last_fullsized":"2023-03-03T08:43:45.106648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:09.574926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107410+0000","last_change":"2023-03-03T08:43:43.995725+0000","last_active":"2023-03-03T08:43:45.107410+0000","last_peered":"2023-03-03T08:43:45.107410+0000","last_clean":"2023-03-03T08:43:45.107410+0000","last_became_active":"2023-03-03T08:43:43.993875+0000","last_became_peered":"2023-03-03T08:43:43.993875+0000","last_unstale":"2023-03-03T08:43:45.107410+0000","last_undegraded":"2023-03-03T08:43:45.107410+0000","last_fullsized":"2023-03-03T08:43:45.107410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:38.439539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982673+0000","last_change":"2023-03-03T08:43:34.546067+0000","last_active":"2023-03-03T08:43:43.982673+0000","last_peered":"2023-03-03T08:43:43.982673+0000","last_clean":"2023-03-03T08:43:43.982673+0000","last_became_active":"2023-03-03T08:43:34.543786+0000","last_became_peered":"2023-03-03T08:43:34.543786+0000","last_unstale":"2023-03-03T08:43:43.982673+0000","last_undegraded":"2023-03-03T08:43:43.982673+0000","last_fullsized":"2023-03-03T08:43:43.982673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:32.922812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.978650+0000","last_change":"2023-03-03T08:43:34.115792+0000","last_active":"2023-03-03T08:43:43.978650+0000","last_peered":"2023-03-03T08:43:43.978650+0000","last_clean":"2023-03-03T08:43:43.978650+0000","last_became_active":"2023-03-03T08:43:34.115431+0000","last_became_peered":"2023-03-03T08:43:34.115431+0000","last_unstale":"2023-03-03T08:43:43.978650+0000","last_undegraded":"2023-03-03T08:43:43.978650+0000","last_fullsized":"2023-03-03T08:43:43.978650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:15.484022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980510+0000","last_change":"2023-03-03T08:43:43.579691+0000","last_active":"2023-03-03T08:43:44.980510+0000","last_peered":"2023-03-03T08:43:44.980510+0000","last_clean":"2023-03-03T08:43:44.980510+0000","last_became_active":"2023-03-03T08:43:43.579425+0000","last_became_peered":"2023-03-03T08:43:43.579425+0000","last_unstale":"2023-03-03T08:43:44.980510+0000","last_undegraded":"2023-03-03T08:43:44.980510+0000","last_fullsized":"2023-03-03T08:43:44.980510+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:08.896591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979907+0000","last_change":"2023-03-03T08:43:44.006872+0000","last_active":"2023-03-03T08:43:44.979907+0000","last_peered":"2023-03-03T08:43:44.979907+0000","last_clean":"2023-03-03T08:43:44.979907+0000","last_became_active":"2023-03-03T08:43:44.006283+0000","last_became_peered":"2023-03-03T08:43:44.006283+0000","last_unstale":"2023-03-03T08:43:44.979907+0000","last_undegraded":"2023-03-03T08:43:44.979907+0000","last_fullsized":"2023-03-03T08:43:44.979907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:45.439399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981214+0000","last_change":"2023-03-03T08:43:34.532656+0000","last_active":"2023-03-03T08:43:43.981214+0000","last_peered":"2023-03-03T08:43:43.981214+0000","last_clean":"2023-03-03T08:43:43.981214+0000","last_became_active":"2023-03-03T08:43:34.532297+0000","last_became_peered":"2023-03-03T08:43:34.532297+0000","last_unstale":"2023-03-03T08:43:43.981214+0000","last_undegraded":"2023-03-03T08:43:43.981214+0000","last_fullsized":"2023-03-03T08:43:43.981214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:36.347740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989578+0000","last_change":"2023-03-03T08:43:34.112905+0000","last_active":"2023-03-03T08:43:43.989578+0000","last_peered":"2023-03-03T08:43:43.989578+0000","last_clean":"2023-03-03T08:43:43.989578+0000","last_became_active":"2023-03-03T08:43:34.112601+0000","last_became_peered":"2023-03-03T08:43:34.112601+0000","last_unstale":"2023-03-03T08:43:43.989578+0000","last_undegraded":"2023-03-03T08:43:43.989578+0000","last_fullsized":"2023-03-03T08:43:43.989578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:01.165783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"26'6","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980517+0000","last_change":"2023-03-03T08:43:43.577882+0000","last_active":"2023-03-03T08:43:44.980517+0000","last_peered":"2023-03-03T08:43:44.980517+0000","last_clean":"2023-03-03T08:43:44.980517+0000","last_became_active":"2023-03-03T08:43:43.575625+0000","last_became_peered":"2023-03-03T08:43:43.575625+0000","last_unstale":"2023-03-03T08:43:44.980517+0000","last_undegraded":"2023-03-03T08:43:44.980517+0000","last_fullsized":"2023-03-03T08:43:44.980517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:22:25.919413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106494+0000","last_change":"2023-03-03T08:43:43.994846+0000","last_active":"2023-03-03T08:43:45.106494+0000","last_peered":"2023-03-03T08:43:45.106494+0000","last_clean":"2023-03-03T08:43:45.106494+0000","last_became_active":"2023-03-03T08:43:43.994652+0000","last_became_peered":"2023-03-03T08:43:43.994652+0000","last_unstale":"2023-03-03T08:43:45.106494+0000","last_undegraded":"2023-03-03T08:43:45.106494+0000","last_fullsized":"2023-03-03T08:43:45.106494+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:54:29.238192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990352+0000","last_change":"2023-03-03T08:43:34.539790+0000","last_active":"2023-03-03T08:43:43.990352+0000","last_peered":"2023-03-03T08:43:43.990352+0000","last_clean":"2023-03-03T08:43:43.990352+0000","last_became_active":"2023-03-03T08:43:34.539443+0000","last_became_peered":"2023-03-03T08:43:34.539443+0000","last_unstale":"2023-03-03T08:43:43.990352+0000","last_undegraded":"2023-03-03T08:43:43.990352+0000","last_fullsized":"2023-03-03T08:43:43.990352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:25.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980897+0000","last_change":"2023-03-03T08:43:34.112941+0000","last_active":"2023-03-03T08:43:43.980897+0000","last_peered":"2023-03-03T08:43:43.980897+0000","last_clean":"2023-03-03T08:43:43.980897+0000","last_became_active":"2023-03-03T08:43:34.112646+0000","last_became_peered":"2023-03-03T08:43:34.112646+0000","last_unstale":"2023-03-03T08:43:43.980897+0000","last_undegraded":"2023-03-03T08:43:43.980897+0000","last_fullsized":"2023-03-03T08:43:43.980897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:32.767710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107854+0000","last_change":"2023-03-03T08:43:43.576421+0000","last_active":"2023-03-03T08:43:45.107854+0000","last_peered":"2023-03-03T08:43:45.107854+0000","last_clean":"2023-03-03T08:43:45.107854+0000","last_became_active":"2023-03-03T08:43:43.576086+0000","last_became_peered":"2023-03-03T08:43:43.576086+0000","last_unstale":"2023-03-03T08:43:45.107854+0000","last_undegraded":"2023-03-03T08:43:45.107854+0000","last_fullsized":"2023-03-03T08:43:45.107854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:57:17.814222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980193+0000","last_change":"2023-03-03T08:43:43.995591+0000","last_active":"2023-03-03T08:43:44.980193+0000","last_peered":"2023-03-03T08:43:44.980193+0000","last_clean":"2023-03-03T08:43:44.980193+0000","last_became_active":"2023-03-03T08:43:43.995338+0000","last_became_peered":"2023-03-03T08:43:43.995338+0000","last_unstale":"2023-03-03T08:43:44.980193+0000","last_undegraded":"2023-03-03T08:43:44.980193+0000","last_fullsized":"2023-03-03T08:43:44.980193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:17:56.617398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985713+0000","last_change":"2023-03-03T08:43:34.537857+0000","last_active":"2023-03-03T08:43:43.985713+0000","last_peered":"2023-03-03T08:43:43.985713+0000","last_clean":"2023-03-03T08:43:43.985713+0000","last_became_active":"2023-03-03T08:43:34.537590+0000","last_became_peered":"2023-03-03T08:43:34.537590+0000","last_unstale":"2023-03-03T08:43:43.985713+0000","last_undegraded":"2023-03-03T08:43:43.985713+0000","last_fullsized":"2023-03-03T08:43:43.985713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:52.612521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980688+0000","last_change":"2023-03-03T08:43:34.117609+0000","last_active":"2023-03-03T08:43:43.980688+0000","last_peered":"2023-03-03T08:43:43.980688+0000","last_clean":"2023-03-03T08:43:43.980688+0000","last_became_active":"2023-03-03T08:43:34.117271+0000","last_became_peered":"2023-03-03T08:43:34.117271+0000","last_unstale":"2023-03-03T08:43:43.980688+0000","last_undegraded":"2023-03-03T08:43:43.980688+0000","last_fullsized":"2023-03-03T08:43:43.980688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:20.259259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978772+0000","last_change":"2023-03-03T08:43:43.580489+0000","last_active":"2023-03-03T08:43:44.978772+0000","last_peered":"2023-03-03T08:43:44.978772+0000","last_clean":"2023-03-03T08:43:44.978772+0000","last_became_active":"2023-03-03T08:43:43.580207+0000","last_became_peered":"2023-03-03T08:43:43.580207+0000","last_unstale":"2023-03-03T08:43:44.978772+0000","last_undegraded":"2023-03-03T08:43:44.978772+0000","last_fullsized":"2023-03-03T08:43:44.978772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:58:05.519074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978633+0000","last_change":"2023-03-03T08:43:43.993832+0000","last_active":"2023-03-03T08:43:44.978633+0000","last_peered":"2023-03-03T08:43:44.978633+0000","last_clean":"2023-03-03T08:43:44.978633+0000","last_became_active":"2023-03-03T08:43:43.993524+0000","last_became_peered":"2023-03-03T08:43:43.993524+0000","last_unstale":"2023-03-03T08:43:44.978633+0000","last_undegraded":"2023-03-03T08:43:44.978633+0000","last_fullsized":"2023-03-03T08:43:44.978633+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:23:55.761785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987159+0000","last_change":"2023-03-03T08:43:34.544694+0000","last_active":"2023-03-03T08:43:43.987159+0000","last_peered":"2023-03-03T08:43:43.987159+0000","last_clean":"2023-03-03T08:43:43.987159+0000","last_became_active":"2023-03-03T08:43:34.544399+0000","last_became_peered":"2023-03-03T08:43:34.544399+0000","last_unstale":"2023-03-03T08:43:43.987159+0000","last_undegraded":"2023-03-03T08:43:43.987159+0000","last_fullsized":"2023-03-03T08:43:43.987159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986196+0000","last_change":"2023-03-03T08:43:34.110810+0000","last_active":"2023-03-03T08:43:43.986196+0000","last_peered":"2023-03-03T08:43:43.986196+0000","last_clean":"2023-03-03T08:43:43.986196+0000","last_became_active":"2023-03-03T08:43:34.110483+0000","last_became_peered":"2023-03-03T08:43:34.110483+0000","last_unstale":"2023-03-03T08:43:43.986196+0000","last_undegraded":"2023-03-03T08:43:43.986196+0000","last_fullsized":"2023-03-03T08:43:43.986196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:06.970615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979989+0000","last_change":"2023-03-03T08:43:43.576669+0000","last_active":"2023-03-03T08:43:44.979989+0000","last_peered":"2023-03-03T08:43:44.979989+0000","last_clean":"2023-03-03T08:43:44.979989+0000","last_became_active":"2023-03-03T08:43:43.573885+0000","last_became_peered":"2023-03-03T08:43:43.573885+0000","last_unstale":"2023-03-03T08:43:44.979989+0000","last_undegraded":"2023-03-03T08:43:44.979989+0000","last_fullsized":"2023-03-03T08:43:44.979989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:56:12.054118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979442+0000","last_change":"2023-03-03T08:43:43.996790+0000","last_active":"2023-03-03T08:43:44.979442+0000","last_peered":"2023-03-03T08:43:44.979442+0000","last_clean":"2023-03-03T08:43:44.979442+0000","last_became_active":"2023-03-03T08:43:43.996366+0000","last_became_peered":"2023-03-03T08:43:43.996366+0000","last_unstale":"2023-03-03T08:43:44.979442+0000","last_undegraded":"2023-03-03T08:43:44.979442+0000","last_fullsized":"2023-03-03T08:43:44.979442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:01.662221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988553+0000","last_change":"2023-03-03T08:43:34.538703+0000","last_active":"2023-03-03T08:43:43.988553+0000","last_peered":"2023-03-03T08:43:43.988553+0000","last_clean":"2023-03-03T08:43:43.988553+0000","last_became_active":"2023-03-03T08:43:34.538418+0000","last_became_peered":"2023-03-03T08:43:34.538418+0000","last_unstale":"2023-03-03T08:43:43.988553+0000","last_undegraded":"2023-03-03T08:43:43.988553+0000","last_fullsized":"2023-03-03T08:43:43.988553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:02.795203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986185+0000","last_change":"2023-03-03T08:43:34.113168+0000","last_active":"2023-03-03T08:43:43.986185+0000","last_peered":"2023-03-03T08:43:43.986185+0000","last_clean":"2023-03-03T08:43:43.986185+0000","last_became_active":"2023-03-03T08:43:34.112731+0000","last_became_peered":"2023-03-03T08:43:34.112731+0000","last_unstale":"2023-03-03T08:43:43.986185+0000","last_undegraded":"2023-03-03T08:43:43.986185+0000","last_fullsized":"2023-03-03T08:43:43.986185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:57:59.463191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106795+0000","last_change":"2023-03-03T08:43:43.577965+0000","last_active":"2023-03-03T08:43:45.106795+0000","last_peered":"2023-03-03T08:43:45.106795+0000","last_clean":"2023-03-03T08:43:45.106795+0000","last_became_active":"2023-03-03T08:43:43.577587+0000","last_became_peered":"2023-03-03T08:43:43.577587+0000","last_unstale":"2023-03-03T08:43:45.106795+0000","last_undegraded":"2023-03-03T08:43:45.106795+0000","last_fullsized":"2023-03-03T08:43:45.106795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:24.196773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105366+0000","last_change":"2023-03-03T08:43:43.998210+0000","last_active":"2023-03-03T08:43:45.105366+0000","last_peered":"2023-03-03T08:43:45.105366+0000","last_clean":"2023-03-03T08:43:45.105366+0000","last_became_active":"2023-03-03T08:43:43.997845+0000","last_became_peered":"2023-03-03T08:43:43.997845+0000","last_unstale":"2023-03-03T08:43:45.105366+0000","last_undegraded":"2023-03-03T08:43:45.105366+0000","last_fullsized":"2023-03-03T08:43:45.105366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:57:38.226245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991762+0000","last_change":"2023-03-03T08:43:34.528573+0000","last_active":"2023-03-03T08:43:43.991762+0000","last_peered":"2023-03-03T08:43:43.991762+0000","last_clean":"2023-03-03T08:43:43.991762+0000","last_became_active":"2023-03-03T08:43:34.528278+0000","last_became_peered":"2023-03-03T08:43:34.528278+0000","last_unstale":"2023-03-03T08:43:43.991762+0000","last_undegraded":"2023-03-03T08:43:43.991762+0000","last_fullsized":"2023-03-03T08:43:43.991762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:51.468323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.110802+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.110519+0000","last_became_peered":"2023-03-03T08:43:34.110519+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:02:18.355043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105602+0000","last_change":"2023-03-03T08:43:43.573655+0000","last_active":"2023-03-03T08:43:45.105602+0000","last_peered":"2023-03-03T08:43:45.105602+0000","last_clean":"2023-03-03T08:43:45.105602+0000","last_became_active":"2023-03-03T08:43:43.573162+0000","last_became_peered":"2023-03-03T08:43:43.573162+0000","last_unstale":"2023-03-03T08:43:45.105602+0000","last_undegraded":"2023-03-03T08:43:45.105602+0000","last_fullsized":"2023-03-03T08:43:45.105602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:59:04.568700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106617+0000","last_change":"2023-03-03T08:43:43.993431+0000","last_active":"2023-03-03T08:43:45.106617+0000","last_peered":"2023-03-03T08:43:45.106617+0000","last_clean":"2023-03-03T08:43:45.106617+0000","last_became_active":"2023-03-03T08:43:43.993166+0000","last_became_peered":"2023-03-03T08:43:43.993166+0000","last_unstale":"2023-03-03T08:43:45.106617+0000","last_undegraded":"2023-03-03T08:43:45.106617+0000","last_fullsized":"2023-03-03T08:43:45.106617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:04:52.880667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979475+0000","last_change":"2023-03-03T08:43:43.992992+0000","last_active":"2023-03-03T08:43:44.979475+0000","last_peered":"2023-03-03T08:43:44.979475+0000","last_clean":"2023-03-03T08:43:44.979475+0000","last_became_active":"2023-03-03T08:43:43.992664+0000","last_became_peered":"2023-03-03T08:43:43.992664+0000","last_unstale":"2023-03-03T08:43:44.979475+0000","last_undegraded":"2023-03-03T08:43:44.979475+0000","last_fullsized":"2023-03-03T08:43:44.979475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:07.276715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981207+0000","last_change":"2023-03-03T08:43:34.547450+0000","last_active":"2023-03-03T08:43:43.981207+0000","last_peered":"2023-03-03T08:43:43.981207+0000","last_clean":"2023-03-03T08:43:43.981207+0000","last_became_active":"2023-03-03T08:43:34.547168+0000","last_became_peered":"2023-03-03T08:43:34.547168+0000","last_unstale":"2023-03-03T08:43:43.981207+0000","last_undegraded":"2023-03-03T08:43:43.981207+0000","last_fullsized":"2023-03-03T08:43:43.981207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:17.723229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982025+0000","last_change":"2023-03-03T08:43:34.106505+0000","last_active":"2023-03-03T08:43:43.982025+0000","last_peered":"2023-03-03T08:43:43.982025+0000","last_clean":"2023-03-03T08:43:43.982025+0000","last_became_active":"2023-03-03T08:43:34.106079+0000","last_became_peered":"2023-03-03T08:43:34.106079+0000","last_unstale":"2023-03-03T08:43:43.982025+0000","last_undegraded":"2023-03-03T08:43:43.982025+0000","last_fullsized":"2023-03-03T08:43:43.982025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:48.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979698+0000","last_change":"2023-03-03T08:43:43.575448+0000","last_active":"2023-03-03T08:43:44.979698+0000","last_peered":"2023-03-03T08:43:44.979698+0000","last_clean":"2023-03-03T08:43:44.979698+0000","last_became_active":"2023-03-03T08:43:43.575142+0000","last_became_peered":"2023-03-03T08:43:43.575142+0000","last_unstale":"2023-03-03T08:43:44.979698+0000","last_undegraded":"2023-03-03T08:43:44.979698+0000","last_fullsized":"2023-03-03T08:43:44.979698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:42:09.308616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990396+0000","last_change":"2023-03-03T08:43:34.549123+0000","last_active":"2023-03-03T08:43:43.990396+0000","last_peered":"2023-03-03T08:43:43.990396+0000","last_clean":"2023-03-03T08:43:43.990396+0000","last_became_active":"2023-03-03T08:43:34.548839+0000","last_became_peered":"2023-03-03T08:43:34.548839+0000","last_unstale":"2023-03-03T08:43:43.990396+0000","last_undegraded":"2023-03-03T08:43:43.990396+0000","last_fullsized":"2023-03-03T08:43:43.990396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:38.134009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984959+0000","last_change":"2023-03-03T08:43:34.108187+0000","last_active":"2023-03-03T08:43:43.984959+0000","last_peered":"2023-03-03T08:43:43.984959+0000","last_clean":"2023-03-03T08:43:43.984959+0000","last_became_active":"2023-03-03T08:43:34.107859+0000","last_became_peered":"2023-03-03T08:43:34.107859+0000","last_unstale":"2023-03-03T08:43:43.984959+0000","last_undegraded":"2023-03-03T08:43:43.984959+0000","last_fullsized":"2023-03-03T08:43:43.984959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:46:40.595756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107953+0000","last_change":"2023-03-03T08:43:43.572738+0000","last_active":"2023-03-03T08:43:45.107953+0000","last_peered":"2023-03-03T08:43:45.107953+0000","last_clean":"2023-03-03T08:43:45.107953+0000","last_became_active":"2023-03-03T08:43:43.572435+0000","last_became_peered":"2023-03-03T08:43:43.572435+0000","last_unstale":"2023-03-03T08:43:45.107953+0000","last_undegraded":"2023-03-03T08:43:45.107953+0000","last_fullsized":"2023-03-03T08:43:45.107953+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:14.264653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105420+0000","last_change":"2023-03-03T08:43:43.995021+0000","last_active":"2023-03-03T08:43:45.105420+0000","last_peered":"2023-03-03T08:43:45.105420+0000","last_clean":"2023-03-03T08:43:45.105420+0000","last_became_active":"2023-03-03T08:43:43.994766+0000","last_became_peered":"2023-03-03T08:43:43.994766+0000","last_unstale":"2023-03-03T08:43:45.105420+0000","last_undegraded":"2023-03-03T08:43:45.105420+0000","last_fullsized":"2023-03-03T08:43:45.105420+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:14:03.827057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979582+0000","last_change":"2023-03-03T08:43:43.993391+0000","last_active":"2023-03-03T08:43:44.979582+0000","last_peered":"2023-03-03T08:43:44.979582+0000","last_clean":"2023-03-03T08:43:44.979582+0000","last_became_active":"2023-03-03T08:43:43.993087+0000","last_became_peered":"2023-03-03T08:43:43.993087+0000","last_unstale":"2023-03-03T08:43:44.979582+0000","last_undegraded":"2023-03-03T08:43:44.979582+0000","last_fullsized":"2023-03-03T08:43:44.979582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:13:01.193811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992180+0000","last_change":"2023-03-03T08:43:34.537301+0000","last_active":"2023-03-03T08:43:43.992180+0000","last_peered":"2023-03-03T08:43:43.992180+0000","last_clean":"2023-03-03T08:43:43.992180+0000","last_became_active":"2023-03-03T08:43:34.537007+0000","last_became_peered":"2023-03-03T08:43:34.537007+0000","last_unstale":"2023-03-03T08:43:43.992180+0000","last_undegraded":"2023-03-03T08:43:43.992180+0000","last_fullsized":"2023-03-03T08:43:43.992180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:59:49.124140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982700+0000","last_change":"2023-03-03T08:43:34.108056+0000","last_active":"2023-03-03T08:43:43.982700+0000","last_peered":"2023-03-03T08:43:43.982700+0000","last_clean":"2023-03-03T08:43:43.982700+0000","last_became_active":"2023-03-03T08:43:34.107708+0000","last_became_peered":"2023-03-03T08:43:34.107708+0000","last_unstale":"2023-03-03T08:43:43.982700+0000","last_undegraded":"2023-03-03T08:43:43.982700+0000","last_fullsized":"2023-03-03T08:43:43.982700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:10.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107314+0000","last_change":"2023-03-03T08:43:43.575788+0000","last_active":"2023-03-03T08:43:45.107314+0000","last_peered":"2023-03-03T08:43:45.107314+0000","last_clean":"2023-03-03T08:43:45.107314+0000","last_became_active":"2023-03-03T08:43:43.575507+0000","last_became_peered":"2023-03-03T08:43:43.575507+0000","last_unstale":"2023-03-03T08:43:45.107314+0000","last_undegraded":"2023-03-03T08:43:45.107314+0000","last_fullsized":"2023-03-03T08:43:45.107314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:15.843816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107877+0000","last_change":"2023-03-03T08:43:43.994916+0000","last_active":"2023-03-03T08:43:45.107877+0000","last_peered":"2023-03-03T08:43:45.107877+0000","last_clean":"2023-03-03T08:43:45.107877+0000","last_became_active":"2023-03-03T08:43:43.994639+0000","last_became_peered":"2023-03-03T08:43:43.994639+0000","last_unstale":"2023-03-03T08:43:45.107877+0000","last_undegraded":"2023-03-03T08:43:45.107877+0000","last_fullsized":"2023-03-03T08:43:45.107877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:52.526052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980834+0000","last_change":"2023-03-03T08:43:34.533987+0000","last_active":"2023-03-03T08:43:43.980834+0000","last_peered":"2023-03-03T08:43:43.980834+0000","last_clean":"2023-03-03T08:43:43.980834+0000","last_became_active":"2023-03-03T08:43:34.533729+0000","last_became_peered":"2023-03-03T08:43:34.533729+0000","last_unstale":"2023-03-03T08:43:43.980834+0000","last_undegraded":"2023-03-03T08:43:43.980834+0000","last_fullsized":"2023-03-03T08:43:43.980834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:18:12.407151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106398+0000","last_change":"2023-03-03T08:43:43.996808+0000","last_active":"2023-03-03T08:43:45.106398+0000","last_peered":"2023-03-03T08:43:45.106398+0000","last_clean":"2023-03-03T08:43:45.106398+0000","last_became_active":"2023-03-03T08:43:43.996559+0000","last_became_peered":"2023-03-03T08:43:43.996559+0000","last_unstale":"2023-03-03T08:43:45.106398+0000","last_undegraded":"2023-03-03T08:43:45.106398+0000","last_fullsized":"2023-03-03T08:43:45.106398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:01.223908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.993213+0000","last_change":"2023-03-03T08:43:34.540762+0000","last_active":"2023-03-03T08:43:43.993213+0000","last_peered":"2023-03-03T08:43:43.993213+0000","last_clean":"2023-03-03T08:43:43.993213+0000","last_became_active":"2023-03-03T08:43:34.540467+0000","last_became_peered":"2023-03-03T08:43:34.540467+0000","last_unstale":"2023-03-03T08:43:43.993213+0000","last_undegraded":"2023-03-03T08:43:43.993213+0000","last_fullsized":"2023-03-03T08:43:43.993213+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107993+0000","last_change":"2023-03-03T08:43:43.995286+0000","last_active":"2023-03-03T08:43:45.107993+0000","last_peered":"2023-03-03T08:43:45.107993+0000","last_clean":"2023-03-03T08:43:45.107993+0000","last_became_active":"2023-03-03T08:43:43.994861+0000","last_became_peered":"2023-03-03T08:43:43.994861+0000","last_unstale":"2023-03-03T08:43:45.107993+0000","last_undegraded":"2023-03-03T08:43:45.107993+0000","last_fullsized":"2023-03-03T08:43:45.107993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990722+0000","last_change":"2023-03-03T08:43:34.542858+0000","last_active":"2023-03-03T08:43:43.990722+0000","last_peered":"2023-03-03T08:43:43.990722+0000","last_clean":"2023-03-03T08:43:43.990722+0000","last_became_active":"2023-03-03T08:43:34.542510+0000","last_became_peered":"2023-03-03T08:43:34.542510+0000","last_unstale":"2023-03-03T08:43:43.990722+0000","last_undegraded":"2023-03-03T08:43:43.990722+0000","last_fullsized":"2023-03-03T08:43:43.990722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:21.471234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106711+0000","last_change":"2023-03-03T08:43:43.992084+0000","last_active":"2023-03-03T08:43:45.106711+0000","last_peered":"2023-03-03T08:43:45.106711+0000","last_clean":"2023-03-03T08:43:45.106711+0000","last_became_active":"2023-03-03T08:43:43.991465+0000","last_became_peered":"2023-03-03T08:43:43.991465+0000","last_unstale":"2023-03-03T08:43:45.106711+0000","last_undegraded":"2023-03-03T08:43:45.106711+0000","last_fullsized":"2023-03-03T08:43:45.106711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:19.530923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983571+0000","last_change":"2023-03-03T08:43:34.548469+0000","last_active":"2023-03-03T08:43:43.983571+0000","last_peered":"2023-03-03T08:43:43.983571+0000","last_clean":"2023-03-03T08:43:43.983571+0000","last_became_active":"2023-03-03T08:43:34.547647+0000","last_became_peered":"2023-03-03T08:43:34.547647+0000","last_unstale":"2023-03-03T08:43:43.983571+0000","last_undegraded":"2023-03-03T08:43:43.983571+0000","last_fullsized":"2023-03-03T08:43:43.983571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:40.054781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107161+0000","last_change":"2023-03-03T08:43:43.998061+0000","last_active":"2023-03-03T08:43:45.107161+0000","last_peered":"2023-03-03T08:43:45.107161+0000","last_clean":"2023-03-03T08:43:45.107161+0000","last_became_active":"2023-03-03T08:43:43.997770+0000","last_became_peered":"2023-03-03T08:43:43.997770+0000","last_unstale":"2023-03-03T08:43:45.107161+0000","last_undegraded":"2023-03-03T08:43:45.107161+0000","last_fullsized":"2023-03-03T08:43:45.107161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:17.053610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988635+0000","last_change":"2023-03-03T08:43:34.549771+0000","last_active":"2023-03-03T08:43:43.988635+0000","last_peered":"2023-03-03T08:43:43.988635+0000","last_clean":"2023-03-03T08:43:43.988635+0000","last_became_active":"2023-03-03T08:43:34.549479+0000","last_became_peered":"2023-03-03T08:43:34.549479+0000","last_unstale":"2023-03-03T08:43:43.988635+0000","last_undegraded":"2023-03-03T08:43:43.988635+0000","last_fullsized":"2023-03-03T08:43:43.988635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:41:03.902019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979024+0000","last_change":"2023-03-03T08:43:43.999391+0000","last_active":"2023-03-03T08:43:44.979024+0000","last_peered":"2023-03-03T08:43:44.979024+0000","last_clean":"2023-03-03T08:43:44.979024+0000","last_became_active":"2023-03-03T08:43:43.998904+0000","last_became_peered":"2023-03-03T08:43:43.998904+0000","last_unstale":"2023-03-03T08:43:44.979024+0000","last_undegraded":"2023-03-03T08:43:44.979024+0000","last_fullsized":"2023-03-03T08:43:44.979024+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:17.429776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992365+0000","last_change":"2023-03-03T08:43:34.542661+0000","last_active":"2023-03-03T08:43:43.992365+0000","last_peered":"2023-03-03T08:43:43.992365+0000","last_clean":"2023-03-03T08:43:43.992365+0000","last_became_active":"2023-03-03T08:43:34.542353+0000","last_became_peered":"2023-03-03T08:43:34.542353+0000","last_unstale":"2023-03-03T08:43:43.992365+0000","last_undegraded":"2023-03-03T08:43:43.992365+0000","last_fullsized":"2023-03-03T08:43:43.992365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:38.730662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105103+0000","last_change":"2023-03-03T08:43:43.996729+0000","last_active":"2023-03-03T08:43:45.105103+0000","last_peered":"2023-03-03T08:43:45.105103+0000","last_clean":"2023-03-03T08:43:45.105103+0000","last_became_active":"2023-03-03T08:43:43.986535+0000","last_became_peered":"2023-03-03T08:43:43.986535+0000","last_unstale":"2023-03-03T08:43:45.105103+0000","last_undegraded":"2023-03-03T08:43:45.105103+0000","last_fullsized":"2023-03-03T08:43:45.105103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:22.505305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982406+0000","last_change":"2023-03-03T08:43:34.549837+0000","last_active":"2023-03-03T08:43:43.982406+0000","last_peered":"2023-03-03T08:43:43.982406+0000","last_clean":"2023-03-03T08:43:43.982406+0000","last_became_active":"2023-03-03T08:43:34.549542+0000","last_became_peered":"2023-03-03T08:43:34.549542+0000","last_unstale":"2023-03-03T08:43:43.982406+0000","last_undegraded":"2023-03-03T08:43:43.982406+0000","last_fullsized":"2023-03-03T08:43:43.982406+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:31.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106334+0000","last_change":"2023-03-03T08:43:43.993213+0000","last_active":"2023-03-03T08:43:45.106334+0000","last_peered":"2023-03-03T08:43:45.106334+0000","last_clean":"2023-03-03T08:43:45.106334+0000","last_became_active":"2023-03-03T08:43:43.992920+0000","last_became_peered":"2023-03-03T08:43:43.992920+0000","last_unstale":"2023-03-03T08:43:45.106334+0000","last_undegraded":"2023-03-03T08:43:45.106334+0000","last_fullsized":"2023-03-03T08:43:45.106334+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:42.055599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.547003+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.546744+0000","last_became_peered":"2023-03-03T08:43:34.546744+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:53:38.996605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106934+0000","last_change":"2023-03-03T08:43:43.994195+0000","last_active":"2023-03-03T08:43:45.106934+0000","last_peered":"2023-03-03T08:43:45.106934+0000","last_clean":"2023-03-03T08:43:45.106934+0000","last_became_active":"2023-03-03T08:43:43.993463+0000","last_became_peered":"2023-03-03T08:43:43.993463+0000","last_unstale":"2023-03-03T08:43:45.106934+0000","last_undegraded":"2023-03-03T08:43:45.106934+0000","last_fullsized":"2023-03-03T08:43:45.106934+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:52.365793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982677+0000","last_change":"2023-03-03T08:43:34.553589+0000","last_active":"2023-03-03T08:43:43.982677+0000","last_peered":"2023-03-03T08:43:43.982677+0000","last_clean":"2023-03-03T08:43:43.982677+0000","last_became_active":"2023-03-03T08:43:34.553242+0000","last_became_peered":"2023-03-03T08:43:34.553242+0000","last_unstale":"2023-03-03T08:43:43.982677+0000","last_undegraded":"2023-03-03T08:43:43.982677+0000","last_fullsized":"2023-03-03T08:43:43.982677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:09.000051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107426+0000","last_change":"2023-03-03T08:43:44.004404+0000","last_active":"2023-03-03T08:43:45.107426+0000","last_peered":"2023-03-03T08:43:45.107426+0000","last_clean":"2023-03-03T08:43:45.107426+0000","last_became_active":"2023-03-03T08:43:44.004120+0000","last_became_peered":"2023-03-03T08:43:44.004120+0000","last_unstale":"2023-03-03T08:43:45.107426+0000","last_undegraded":"2023-03-03T08:43:45.107426+0000","last_fullsized":"2023-03-03T08:43:45.107426+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:36.006791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990661+0000","last_change":"2023-03-03T08:43:34.537110+0000","last_active":"2023-03-03T08:43:43.990661+0000","last_peered":"2023-03-03T08:43:43.990661+0000","last_clean":"2023-03-03T08:43:43.990661+0000","last_became_active":"2023-03-03T08:43:34.536848+0000","last_became_peered":"2023-03-03T08:43:34.536848+0000","last_unstale":"2023-03-03T08:43:43.990661+0000","last_undegraded":"2023-03-03T08:43:43.990661+0000","last_fullsized":"2023-03-03T08:43:43.990661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106366+0000","last_change":"2023-03-03T08:43:43.999266+0000","last_active":"2023-03-03T08:43:45.106366+0000","last_peered":"2023-03-03T08:43:45.106366+0000","last_clean":"2023-03-03T08:43:45.106366+0000","last_became_active":"2023-03-03T08:43:43.998929+0000","last_became_peered":"2023-03-03T08:43:43.998929+0000","last_unstale":"2023-03-03T08:43:45.106366+0000","last_undegraded":"2023-03-03T08:43:45.106366+0000","last_fullsized":"2023-03-03T08:43:45.106366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:02.236263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994220+0000","last_change":"2023-03-03T08:43:34.532154+0000","last_active":"2023-03-03T08:43:43.994220+0000","last_peered":"2023-03-03T08:43:43.994220+0000","last_clean":"2023-03-03T08:43:43.994220+0000","last_became_active":"2023-03-03T08:43:34.531844+0000","last_became_peered":"2023-03-03T08:43:34.531844+0000","last_unstale":"2023-03-03T08:43:43.994220+0000","last_undegraded":"2023-03-03T08:43:43.994220+0000","last_fullsized":"2023-03-03T08:43:43.994220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:00.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106991+0000","last_change":"2023-03-03T08:43:44.004469+0000","last_active":"2023-03-03T08:43:45.106991+0000","last_peered":"2023-03-03T08:43:45.106991+0000","last_clean":"2023-03-03T08:43:45.106991+0000","last_became_active":"2023-03-03T08:43:44.004187+0000","last_became_peered":"2023-03-03T08:43:44.004187+0000","last_unstale":"2023-03-03T08:43:45.106991+0000","last_undegraded":"2023-03-03T08:43:45.106991+0000","last_fullsized":"2023-03-03T08:43:45.106991+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:35.298857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979308+0000","last_change":"2023-03-03T08:43:34.527373+0000","last_active":"2023-03-03T08:43:43.979308+0000","last_peered":"2023-03-03T08:43:43.979308+0000","last_clean":"2023-03-03T08:43:43.979308+0000","last_became_active":"2023-03-03T08:43:34.527171+0000","last_became_peered":"2023-03-03T08:43:34.527171+0000","last_unstale":"2023-03-03T08:43:43.979308+0000","last_undegraded":"2023-03-03T08:43:43.979308+0000","last_fullsized":"2023-03-03T08:43:43.979308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:20:51.138587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108220+0000","last_change":"2023-03-03T08:43:43.996329+0000","last_active":"2023-03-03T08:43:45.108220+0000","last_peered":"2023-03-03T08:43:45.108220+0000","last_clean":"2023-03-03T08:43:45.108220+0000","last_became_active":"2023-03-03T08:43:43.995055+0000","last_became_peered":"2023-03-03T08:43:43.995055+0000","last_unstale":"2023-03-03T08:43:45.108220+0000","last_undegraded":"2023-03-03T08:43:45.108220+0000","last_fullsized":"2023-03-03T08:43:45.108220+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:57.593740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984695+0000","last_change":"2023-03-03T08:43:34.542729+0000","last_active":"2023-03-03T08:43:43.984695+0000","last_peered":"2023-03-03T08:43:43.984695+0000","last_clean":"2023-03-03T08:43:43.984695+0000","last_became_active":"2023-03-03T08:43:34.542246+0000","last_became_peered":"2023-03-03T08:43:34.542246+0000","last_unstale":"2023-03-03T08:43:43.984695+0000","last_undegraded":"2023-03-03T08:43:43.984695+0000","last_fullsized":"2023-03-03T08:43:43.984695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:14:59.995526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105365+0000","last_change":"2023-03-03T08:43:44.002358+0000","last_active":"2023-03-03T08:43:45.105365+0000","last_peered":"2023-03-03T08:43:45.105365+0000","last_clean":"2023-03-03T08:43:45.105365+0000","last_became_active":"2023-03-03T08:43:44.002020+0000","last_became_peered":"2023-03-03T08:43:44.002020+0000","last_unstale":"2023-03-03T08:43:45.105365+0000","last_undegraded":"2023-03-03T08:43:45.105365+0000","last_fullsized":"2023-03-03T08:43:45.105365+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:51.524774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982664+0000","last_change":"2023-03-03T08:43:34.549026+0000","last_active":"2023-03-03T08:43:43.982664+0000","last_peered":"2023-03-03T08:43:43.982664+0000","last_clean":"2023-03-03T08:43:43.982664+0000","last_became_active":"2023-03-03T08:43:34.548778+0000","last_became_peered":"2023-03-03T08:43:34.548778+0000","last_unstale":"2023-03-03T08:43:43.982664+0000","last_undegraded":"2023-03-03T08:43:43.982664+0000","last_fullsized":"2023-03-03T08:43:43.982664+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:41.835304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105281+0000","last_change":"2023-03-03T08:43:43.997684+0000","last_active":"2023-03-03T08:43:45.105281+0000","last_peered":"2023-03-03T08:43:45.105281+0000","last_clean":"2023-03-03T08:43:45.105281+0000","last_became_active":"2023-03-03T08:43:43.997503+0000","last_became_peered":"2023-03-03T08:43:43.997503+0000","last_unstale":"2023-03-03T08:43:45.105281+0000","last_undegraded":"2023-03-03T08:43:45.105281+0000","last_fullsized":"2023-03-03T08:43:45.105281+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:29.512524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991178+0000","last_change":"2023-03-03T08:43:34.544013+0000","last_active":"2023-03-03T08:43:43.991178+0000","last_peered":"2023-03-03T08:43:43.991178+0000","last_clean":"2023-03-03T08:43:43.991178+0000","last_became_active":"2023-03-03T08:43:34.543745+0000","last_became_peered":"2023-03-03T08:43:34.543745+0000","last_unstale":"2023-03-03T08:43:43.991178+0000","last_undegraded":"2023-03-03T08:43:43.991178+0000","last_fullsized":"2023-03-03T08:43:43.991178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:05.647977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107560+0000","last_change":"2023-03-03T08:43:43.999884+0000","last_active":"2023-03-03T08:43:45.107560+0000","last_peered":"2023-03-03T08:43:45.107560+0000","last_clean":"2023-03-03T08:43:45.107560+0000","last_became_active":"2023-03-03T08:43:43.999593+0000","last_became_peered":"2023-03-03T08:43:43.999593+0000","last_unstale":"2023-03-03T08:43:45.107560+0000","last_undegraded":"2023-03-03T08:43:45.107560+0000","last_fullsized":"2023-03-03T08:43:45.107560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:45:09.132974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982600+0000","last_change":"2023-03-03T08:43:34.553434+0000","last_active":"2023-03-03T08:43:43.982600+0000","last_peered":"2023-03-03T08:43:43.982600+0000","last_clean":"2023-03-03T08:43:43.982600+0000","last_became_active":"2023-03-03T08:43:34.553125+0000","last_became_peered":"2023-03-03T08:43:34.553125+0000","last_unstale":"2023-03-03T08:43:43.982600+0000","last_undegraded":"2023-03-03T08:43:43.982600+0000","last_fullsized":"2023-03-03T08:43:43.982600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:41:17.553506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105717+0000","last_change":"2023-03-03T08:43:44.006710+0000","last_active":"2023-03-03T08:43:45.105717+0000","last_peered":"2023-03-03T08:43:45.105717+0000","last_clean":"2023-03-03T08:43:45.105717+0000","last_became_active":"2023-03-03T08:43:44.006447+0000","last_became_peered":"2023-03-03T08:43:44.006447+0000","last_unstale":"2023-03-03T08:43:45.105717+0000","last_undegraded":"2023-03-03T08:43:45.105717+0000","last_fullsized":"2023-03-03T08:43:45.105717+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:21.767326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982538+0000","last_change":"2023-03-03T08:43:34.550889+0000","last_active":"2023-03-03T08:43:43.982538+0000","last_peered":"2023-03-03T08:43:43.982538+0000","last_clean":"2023-03-03T08:43:43.982538+0000","last_became_active":"2023-03-03T08:43:34.550605+0000","last_became_peered":"2023-03-03T08:43:34.550605+0000","last_unstale":"2023-03-03T08:43:43.982538+0000","last_undegraded":"2023-03-03T08:43:43.982538+0000","last_fullsized":"2023-03-03T08:43:43.982538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:01.118555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105567+0000","last_change":"2023-03-03T08:43:43.998231+0000","last_active":"2023-03-03T08:43:45.105567+0000","last_peered":"2023-03-03T08:43:45.105567+0000","last_clean":"2023-03-03T08:43:45.105567+0000","last_became_active":"2023-03-03T08:43:43.997941+0000","last_became_peered":"2023-03-03T08:43:43.997941+0000","last_unstale":"2023-03-03T08:43:45.105567+0000","last_undegraded":"2023-03-03T08:43:45.105567+0000","last_fullsized":"2023-03-03T08:43:45.105567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:13:32.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983100+0000","last_change":"2023-03-03T08:43:34.548229+0000","last_active":"2023-03-03T08:43:43.983100+0000","last_peered":"2023-03-03T08:43:43.983100+0000","last_clean":"2023-03-03T08:43:43.983100+0000","last_became_active":"2023-03-03T08:43:34.548013+0000","last_became_peered":"2023-03-03T08:43:34.548013+0000","last_unstale":"2023-03-03T08:43:43.983100+0000","last_undegraded":"2023-03-03T08:43:43.983100+0000","last_fullsized":"2023-03-03T08:43:43.983100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:27.793716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108186+0000","last_change":"2023-03-03T08:43:43.992834+0000","last_active":"2023-03-03T08:43:45.108186+0000","last_peered":"2023-03-03T08:43:45.108186+0000","last_clean":"2023-03-03T08:43:45.108186+0000","last_became_active":"2023-03-03T08:43:43.991577+0000","last_became_peered":"2023-03-03T08:43:43.991577+0000","last_unstale":"2023-03-03T08:43:45.108186+0000","last_undegraded":"2023-03-03T08:43:45.108186+0000","last_fullsized":"2023-03-03T08:43:45.108186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:18:37.119460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990785+0000","last_change":"2023-03-03T08:43:34.549838+0000","last_active":"2023-03-03T08:43:43.990785+0000","last_peered":"2023-03-03T08:43:43.990785+0000","last_clean":"2023-03-03T08:43:43.990785+0000","last_became_active":"2023-03-03T08:43:34.549537+0000","last_became_peered":"2023-03-03T08:43:34.549537+0000","last_unstale":"2023-03-03T08:43:43.990785+0000","last_undegraded":"2023-03-03T08:43:43.990785+0000","last_fullsized":"2023-03-03T08:43:43.990785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:36.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978412+0000","last_change":"2023-03-03T08:43:43.991488+0000","last_active":"2023-03-03T08:43:44.978412+0000","last_peered":"2023-03-03T08:43:44.978412+0000","last_clean":"2023-03-03T08:43:44.978412+0000","last_became_active":"2023-03-03T08:43:43.991139+0000","last_became_peered":"2023-03-03T08:43:43.991139+0000","last_unstale":"2023-03-03T08:43:44.978412+0000","last_undegraded":"2023-03-03T08:43:44.978412+0000","last_fullsized":"2023-03-03T08:43:44.978412+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:21.119183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983652+0000","last_change":"2023-03-03T08:43:34.533496+0000","last_active":"2023-03-03T08:43:43.983652+0000","last_peered":"2023-03-03T08:43:43.983652+0000","last_clean":"2023-03-03T08:43:43.983652+0000","last_became_active":"2023-03-03T08:43:34.532714+0000","last_became_peered":"2023-03-03T08:43:34.532714+0000","last_unstale":"2023-03-03T08:43:43.983652+0000","last_undegraded":"2023-03-03T08:43:43.983652+0000","last_fullsized":"2023-03-03T08:43:43.983652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:47.800807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105250+0000","last_change":"2023-03-03T08:43:43.996215+0000","last_active":"2023-03-03T08:43:45.105250+0000","last_peered":"2023-03-03T08:43:45.105250+0000","last_clean":"2023-03-03T08:43:45.105250+0000","last_became_active":"2023-03-03T08:43:43.995791+0000","last_became_peered":"2023-03-03T08:43:43.995791+0000","last_unstale":"2023-03-03T08:43:45.105250+0000","last_undegraded":"2023-03-03T08:43:45.105250+0000","last_fullsized":"2023-03-03T08:43:45.105250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:50:41.130427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979929+0000","last_change":"2023-03-03T08:43:34.533673+0000","last_active":"2023-03-03T08:43:43.979929+0000","last_peered":"2023-03-03T08:43:43.979929+0000","last_clean":"2023-03-03T08:43:43.979929+0000","last_became_active":"2023-03-03T08:43:34.532283+0000","last_became_peered":"2023-03-03T08:43:34.532283+0000","last_unstale":"2023-03-03T08:43:43.979929+0000","last_undegraded":"2023-03-03T08:43:43.979929+0000","last_fullsized":"2023-03-03T08:43:43.979929+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:25.591188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979205+0000","last_change":"2023-03-03T08:43:43.999435+0000","last_active":"2023-03-03T08:43:44.979205+0000","last_peered":"2023-03-03T08:43:44.979205+0000","last_clean":"2023-03-03T08:43:44.979205+0000","last_became_active":"2023-03-03T08:43:43.997037+0000","last_became_peered":"2023-03-03T08:43:43.997037+0000","last_unstale":"2023-03-03T08:43:44.979205+0000","last_undegraded":"2023-03-03T08:43:44.979205+0000","last_fullsized":"2023-03-03T08:43:44.979205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:54.176343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980585+0000","last_change":"2023-03-03T08:43:34.537664+0000","last_active":"2023-03-03T08:43:43.980585+0000","last_peered":"2023-03-03T08:43:43.980585+0000","last_clean":"2023-03-03T08:43:43.980585+0000","last_became_active":"2023-03-03T08:43:34.537214+0000","last_became_peered":"2023-03-03T08:43:34.537214+0000","last_unstale":"2023-03-03T08:43:43.980585+0000","last_undegraded":"2023-03-03T08:43:43.980585+0000","last_fullsized":"2023-03-03T08:43:43.980585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:54:09.222135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106227+0000","last_change":"2023-03-03T08:43:44.000140+0000","last_active":"2023-03-03T08:43:45.106227+0000","last_peered":"2023-03-03T08:43:45.106227+0000","last_clean":"2023-03-03T08:43:45.106227+0000","last_became_active":"2023-03-03T08:43:43.999778+0000","last_became_peered":"2023-03-03T08:43:43.999778+0000","last_unstale":"2023-03-03T08:43:45.106227+0000","last_undegraded":"2023-03-03T08:43:45.106227+0000","last_fullsized":"2023-03-03T08:43:45.106227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:29:10.745759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987366+0000","last_change":"2023-03-03T08:43:34.543620+0000","last_active":"2023-03-03T08:43:43.987366+0000","last_peered":"2023-03-03T08:43:43.987366+0000","last_clean":"2023-03-03T08:43:43.987366+0000","last_became_active":"2023-03-03T08:43:34.543370+0000","last_became_peered":"2023-03-03T08:43:34.543370+0000","last_unstale":"2023-03-03T08:43:43.987366+0000","last_undegraded":"2023-03-03T08:43:43.987366+0000","last_fullsized":"2023-03-03T08:43:43.987366+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:40:32.808203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979449+0000","last_change":"2023-03-03T08:43:43.995084+0000","last_active":"2023-03-03T08:43:44.979449+0000","last_peered":"2023-03-03T08:43:44.979449+0000","last_clean":"2023-03-03T08:43:44.979449+0000","last_became_active":"2023-03-03T08:43:43.994824+0000","last_became_peered":"2023-03-03T08:43:43.994824+0000","last_unstale":"2023-03-03T08:43:44.979449+0000","last_undegraded":"2023-03-03T08:43:44.979449+0000","last_fullsized":"2023-03-03T08:43:44.979449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:04:17.481969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985302+0000","last_change":"2023-03-03T08:43:34.538467+0000","last_active":"2023-03-03T08:43:43.985302+0000","last_peered":"2023-03-03T08:43:43.985302+0000","last_clean":"2023-03-03T08:43:43.985302+0000","last_became_active":"2023-03-03T08:43:34.538207+0000","last_became_peered":"2023-03-03T08:43:34.538207+0000","last_unstale":"2023-03-03T08:43:43.985302+0000","last_undegraded":"2023-03-03T08:43:43.985302+0000","last_fullsized":"2023-03-03T08:43:43.985302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:14.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980577+0000","last_change":"2023-03-03T08:43:43.996076+0000","last_active":"2023-03-03T08:43:44.980577+0000","last_peered":"2023-03-03T08:43:44.980577+0000","last_clean":"2023-03-03T08:43:44.980577+0000","last_became_active":"2023-03-03T08:43:43.995799+0000","last_became_peered":"2023-03-03T08:43:43.995799+0000","last_unstale":"2023-03-03T08:43:44.980577+0000","last_undegraded":"2023-03-03T08:43:44.980577+0000","last_fullsized":"2023-03-03T08:43:44.980577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:35:35.889895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986289+0000","last_change":"2023-03-03T08:43:34.544721+0000","last_active":"2023-03-03T08:43:43.986289+0000","last_peered":"2023-03-03T08:43:43.986289+0000","last_clean":"2023-03-03T08:43:43.986289+0000","last_became_active":"2023-03-03T08:43:34.544461+0000","last_became_peered":"2023-03-03T08:43:34.544461+0000","last_unstale":"2023-03-03T08:43:43.986289+0000","last_undegraded":"2023-03-03T08:43:43.986289+0000","last_fullsized":"2023-03-03T08:43:43.986289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:52:11.890779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980078+0000","last_change":"2023-03-03T08:43:44.006352+0000","last_active":"2023-03-03T08:43:44.980078+0000","last_peered":"2023-03-03T08:43:44.980078+0000","last_clean":"2023-03-03T08:43:44.980078+0000","last_became_active":"2023-03-03T08:43:44.005802+0000","last_became_peered":"2023-03-03T08:43:44.005802+0000","last_unstale":"2023-03-03T08:43:44.980078+0000","last_undegraded":"2023-03-03T08:43:44.980078+0000","last_fullsized":"2023-03-03T08:43:44.980078+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:52.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986425+0000","last_change":"2023-03-03T08:43:34.541669+0000","last_active":"2023-03-03T08:43:43.986425+0000","last_peered":"2023-03-03T08:43:43.986425+0000","last_clean":"2023-03-03T08:43:43.986425+0000","last_became_active":"2023-03-03T08:43:34.541104+0000","last_became_peered":"2023-03-03T08:43:34.541104+0000","last_unstale":"2023-03-03T08:43:43.986425+0000","last_undegraded":"2023-03-03T08:43:43.986425+0000","last_fullsized":"2023-03-03T08:43:43.986425+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980059+0000","last_change":"2023-03-03T08:43:43.998937+0000","last_active":"2023-03-03T08:43:44.980059+0000","last_peered":"2023-03-03T08:43:44.980059+0000","last_clean":"2023-03-03T08:43:44.980059+0000","last_became_active":"2023-03-03T08:43:43.998164+0000","last_became_peered":"2023-03-03T08:43:43.998164+0000","last_unstale":"2023-03-03T08:43:44.980059+0000","last_undegraded":"2023-03-03T08:43:44.980059+0000","last_fullsized":"2023-03-03T08:43:44.980059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:53.741704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.546272+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.546006+0000","last_became_peered":"2023-03-03T08:43:34.546006+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:15.121761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":118397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":119470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9336,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362504,"statfs":{"total":96636764160,"available":96627204096,"internally_reserved":0,"allocated":901120,"data_stored":710394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8244,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363596,"statfs":{"total":96636764160,"available":96628322304,"internally_reserved":0,"allocated":307200,"data_stored":118565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8188,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363652,"statfs":{"total":96636764160,"available":96628379648,"internally_reserved":0,"allocated":315392,"data_stored":121989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":120069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":121018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9260,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362580,"statfs":{"total":96636764160,"available":96627281920,"internally_reserved":0,"allocated":888832,"data_stored":709827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:43:49.521 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-03T08:43:49.521 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-03-03T08:43:49.521 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-03-03T08:43:49.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-03-03T08:43:49.522 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-03-03T08:43:49.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.670+0000 7ff38ac47700 1 Processor -- start 2023-03-03T08:43:49.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.670+0000 7ff38ac47700 1 -- start start 2023-03-03T08:43:49.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.671+0000 7ff38ac47700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff38414fda0 0x7ff384156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.671+0000 7ff38ac47700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3840c2f30 con 0x7ff3840abe20 2023-03-03T08:43:49.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.671+0000 7ff38ac47700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3840c31d0 con 0x7ff38414f9d0 2023-03-03T08:43:49.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.671+0000 7ff38ac47700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3841524a0 con 0x7ff38414fda0 2023-03-03T08:43:49.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.671+0000 7ff383fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff38414fda0 0x7ff384156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:49.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.671+0000 7ff3891e4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff3840abe20 0x7ff3841529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45520/0 (socket says 172.21.15.7:45520) 2023-03-03T08:43:49.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.671+0000 7ff383fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff38414fda0 0x7ff384156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51538/0 (socket says 172.21.15.7:51538) 2023-03-03T08:43:49.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.671+0000 7ff3891e4700 1 -- 172.21.15.7:0/1987578591 learned_addr learned my addr 172.21.15.7:0/1987578591 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:49.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.672+0000 7ff383fff700 1 -- 172.21.15.7:0/1987578591 >> v1:172.21.15.143:6789/0 conn(0x7ff38414f9d0 legacy=0x7ff384071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.672+0000 7ff383fff700 1 -- 172.21.15.7:0/1987578591 >> v1:172.21.15.7:6789/0 conn(0x7ff3840abe20 legacy=0x7ff3841529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.672+0000 7ff383fff700 1 -- 172.21.15.7:0/1987578591 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3840c34b0 con 0x7ff38414fda0 2023-03-03T08:43:49.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.672+0000 7ff383fff700 1 --2- 172.21.15.7:0/1987578591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414fda0 0x7ff384156430 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff370009840 tx=0x7ff370009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb26614b5d1c45a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:49.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.672+0000 7ff3837fe700 1 -- 172.21.15.7:0/1987578591 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370011830 con 0x7ff38414fda0 2023-03-03T08:43:49.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.672+0000 7ff3837fe700 1 -- 172.21.15.7:0/1987578591 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff370011990 con 0x7ff38414fda0 2023-03-03T08:43:49.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.672+0000 7ff3837fe700 1 -- 172.21.15.7:0/1987578591 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370011d20 con 0x7ff38414fda0 2023-03-03T08:43:49.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.673+0000 7ff38ac47700 1 -- 172.21.15.7:0/1987578591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414fda0 msgr2=0x7ff384156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.673+0000 7ff38ac47700 1 --2- 172.21.15.7:0/1987578591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414fda0 0x7ff384156430 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff370009840 tx=0x7ff370009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:49.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.673+0000 7ff38ac47700 1 -- 172.21.15.7:0/1987578591 shutdown_connections 2023-03-03T08:43:49.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.673+0000 7ff38ac47700 1 --2- 172.21.15.7:0/1987578591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414fda0 0x7ff384156430 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.673+0000 7ff38ac47700 1 -- 172.21.15.7:0/1987578591 >> 172.21.15.7:0/1987578591 conn(0x7ff3840c8900 msgr2=0x7ff3840c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:49.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.673+0000 7ff38ac47700 1 -- 172.21.15.7:0/1987578591 shutdown_connections 2023-03-03T08:43:49.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.673+0000 7ff38ac47700 1 -- 172.21.15.7:0/1987578591 wait complete. 2023-03-03T08:43:49.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.674+0000 7ff38ac47700 1 Processor -- start 2023-03-03T08:43:49.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.674+0000 7ff38ac47700 1 -- start start 2023-03-03T08:43:49.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff38ac47700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3840abe20 0x7ff3841eb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff38ac47700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414f9d0 0x7ff38406e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff38ac47700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff38414fda0 0x7ff38414b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff38ac47700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3841dd440 con 0x7ff38414f9d0 2023-03-03T08:43:49.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff38ac47700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff384153300 con 0x7ff38414fda0 2023-03-03T08:43:49.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff38ac47700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3841ebd20 con 0x7ff3840abe20 2023-03-03T08:43:49.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff383fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414f9d0 0x7ff38406e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:49.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff383fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414f9d0 0x7ff38406e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51544/0 (socket says 172.21.15.7:51544) 2023-03-03T08:43:49.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff383fff700 1 -- 172.21.15.7:0/2280693478 learned_addr learned my addr 172.21.15.7:0/2280693478 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:49.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff383fff700 1 -- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3840abe20 msgr2=0x7ff3841eb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff383fff700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3840abe20 0x7ff3841eb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff383fff700 1 -- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff38414fda0 msgr2=0x7ff38414b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff383fff700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff38414fda0 0x7ff38414b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff383fff700 1 -- 172.21.15.7:0/2280693478 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3700094a0 con 0x7ff38414f9d0 2023-03-03T08:43:49.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff383fff700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414f9d0 0x7ff38406e870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff370013000 tx=0x7ff370004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:49.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff3817fa700 1 -- 172.21.15.7:0/2280693478 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700045f0 con 0x7ff38414f9d0 2023-03-03T08:43:49.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff3817fa700 1 -- 172.21.15.7:0/2280693478 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37001f070 con 0x7ff38414f9d0 2023-03-03T08:43:49.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff38ac47700 1 -- 172.21.15.7:0/2280693478 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3840ac350 con 0x7ff38414f9d0 2023-03-03T08:43:49.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff3817fa700 1 -- 172.21.15.7:0/2280693478 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370003b00 con 0x7ff38414f9d0 2023-03-03T08:43:49.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.675+0000 7ff38ac47700 1 -- 172.21.15.7:0/2280693478 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3840ab630 con 0x7ff38414f9d0 2023-03-03T08:43:49.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.676+0000 7ff38ac47700 1 -- 172.21.15.7:0/2280693478 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff384071fa0 con 0x7ff38414f9d0 2023-03-03T08:43:49.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.677+0000 7ff3817fa700 1 -- 172.21.15.7:0/2280693478 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff37000f050 con 0x7ff38414f9d0 2023-03-03T08:43:49.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.677+0000 7ff3817fa700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3680682f0 0x7ff36806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:49.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.677+0000 7ff3817fa700 1 -- 172.21.15.7:0/2280693478 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7ff370021070 con 0x7ff38414f9d0 2023-03-03T08:43:49.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.677+0000 7ff3889e3700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3680682f0 0x7ff36806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:49.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.678+0000 7ff3889e3700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3680682f0 0x7ff36806a7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff378000f60 tx=0x7ff378007a40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:49.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.681+0000 7ff3817fa700 1 -- 172.21.15.7:0/2280693478 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff370053f10 con 0x7ff38414f9d0 2023-03-03T08:43:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.929+0000 7ff38ac47700 1 -- 172.21.15.7:0/2280693478 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff384074c00 con 0x7ff38414f9d0 2023-03-03T08:43:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.932+0000 7ff3817fa700 1 -- 172.21.15.7:0/2280693478 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+166382 (secure 0 0 0) 0x7ff370057530 con 0x7ff38414f9d0 2023-03-03T08:43:49.936 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 -- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3680682f0 msgr2=0x7ff36806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3680682f0 0x7ff36806a7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff378000f60 tx=0x7ff378007a40 comp rx=0 tx=0).stop 2023-03-03T08:43:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 -- 172.21.15.7:0/2280693478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414f9d0 msgr2=0x7ff38406e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414f9d0 0x7ff38406e870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff370013000 tx=0x7ff370004750 comp rx=0 tx=0).stop 2023-03-03T08:43:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 -- 172.21.15.7:0/2280693478 shutdown_connections 2023-03-03T08:43:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff38414fda0 0x7ff38414b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3840abe20 0x7ff3841eb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3680682f0 0x7ff36806a7a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 --2- 172.21.15.7:0/2280693478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38414f9d0 0x7ff38406e870 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 -- 172.21.15.7:0/2280693478 >> 172.21.15.7:0/2280693478 conn(0x7ff3840c8900 msgr2=0x7ff3840c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 -- 172.21.15.7:0/2280693478 shutdown_connections 2023-03-03T08:43:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:49.935+0000 7ff38ac47700 1 -- 172.21.15.7:0/2280693478 wait complete. 2023-03-03T08:43:49.949 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":96367},{"type":"v1","addr":"172.21.15.143:6833","nonce":96367}]},"active_addr":"172.21.15.143:6833/96367","active_change":"2023-03-03T08:43:15.094769+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":1371391700}]},{"addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":853406035}]},{"addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":2010658584}]},{"addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":2155034195}]}]} 2023-03-03T08:43:49.954 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-03-03T08:43:49.954 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-03-03T08:43:49.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:50.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.077+0000 7f8e7a6e9700 1 Processor -- start 2023-03-03T08:43:50.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.077+0000 7f8e7a6e9700 1 -- start start 2023-03-03T08:43:50.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e7a6e9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e74151cd0 0x7f8e74154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e7a6e9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e74157ff0 con 0x7f8e740c2e30 2023-03-03T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e7a6e9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e7406d640 con 0x7f8e7406c6c0 2023-03-03T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e7a6e9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e7406cda0 con 0x7f8e74151cd0 2023-03-03T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e73fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e74151cd0 0x7f8e74154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e73fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e74151cd0 0x7f8e74154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51550/0 (socket says 172.21.15.7:51550) 2023-03-03T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e737fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8e740c2e30 0x7f8e74158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45530/0 (socket says 172.21.15.7:45530) 2023-03-03T08:43:50.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e73fff700 1 -- 172.21.15.7:0/2622625100 learned_addr learned my addr 172.21.15.7:0/2622625100 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e73fff700 1 -- 172.21.15.7:0/2622625100 >> v1:172.21.15.143:6789/0 conn(0x7f8e7406c6c0 legacy=0x7f8e740736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e73fff700 1 -- 172.21.15.7:0/2622625100 >> v1:172.21.15.7:6789/0 conn(0x7f8e740c2e30 legacy=0x7f8e74158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e73fff700 1 -- 172.21.15.7:0/2622625100 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7406d080 con 0x7f8e74151cd0 2023-03-03T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e72ffd700 1 -- 172.21.15.7:0/2622625100 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332957221 0 0) 0x7f8e74157ff0 con 0x7f8e740c2e30 2023-03-03T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e73fff700 1 --2- 172.21.15.7:0/2622625100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e74151cd0 0x7f8e74154510 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0098f0 tx=0x7f8e5c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b5be6386f8794 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.078+0000 7f8e72ffd700 1 -- 172.21.15.7:0/2622625100 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c004230 con 0x7f8e74151cd0 2023-03-03T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.079+0000 7f8e72ffd700 1 -- 172.21.15.7:0/2622625100 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c004390 con 0x7f8e74151cd0 2023-03-03T08:43:50.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.079+0000 7f8e72ffd700 1 -- 172.21.15.7:0/2622625100 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0036a0 con 0x7f8e74151cd0 2023-03-03T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.079+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2622625100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e74151cd0 msgr2=0x7f8e74154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.079+0000 7f8e7a6e9700 1 --2- 172.21.15.7:0/2622625100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e74151cd0 0x7f8e74154510 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0098f0 tx=0x7f8e5c009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.079+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2622625100 shutdown_connections 2023-03-03T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.079+0000 7f8e7a6e9700 1 --2- 172.21.15.7:0/2622625100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e74151cd0 0x7f8e74154510 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.079+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2622625100 >> 172.21.15.7:0/2622625100 conn(0x7f8e740a94d0 msgr2=0x7f8e740a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.079+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2622625100 shutdown_connections 2023-03-03T08:43:50.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.079+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2622625100 wait complete. 2023-03-03T08:43:50.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.080+0000 7f8e7a6e9700 1 Processor -- start 2023-03-03T08:43:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.080+0000 7f8e7a6e9700 1 -- start start 2023-03-03T08:43:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e7a6e9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7406c6c0 0x7f8e74071c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e7a6e9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e740c2e30 0x7f8e740721b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e7a6e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e74151cd0 0x7f8e74141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e7a6e9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7414d720 con 0x7f8e740c2e30 2023-03-03T08:43:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e7a6e9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7414ccd0 con 0x7f8e7406c6c0 2023-03-03T08:43:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e7a6e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7414d290 con 0x7f8e74151cd0 2023-03-03T08:43:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e737fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e740c2e30 0x7f8e740721b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e737fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e740c2e30 0x7f8e740721b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51558/0 (socket says 172.21.15.7:51558) 2023-03-03T08:43:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e737fe700 1 -- 172.21.15.7:0/2991539177 learned_addr learned my addr 172.21.15.7:0/2991539177 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e73fff700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7406c6c0 0x7f8e74071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e78c86700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e74151cd0 0x7f8e74141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e737fe700 1 -- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e74151cd0 msgr2=0x7f8e74141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e737fe700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e74151cd0 0x7f8e74141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e737fe700 1 -- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7406c6c0 msgr2=0x7f8e74071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e737fe700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7406c6c0 0x7f8e74071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.081+0000 7f8e737fe700 1 -- 172.21.15.7:0/2991539177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c009550 con 0x7f8e740c2e30 2023-03-03T08:43:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.082+0000 7f8e78c86700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e74151cd0 0x7f8e74141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:50.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.082+0000 7f8e73fff700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7406c6c0 0x7f8e74071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:50.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.082+0000 7f8e737fe700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e740c2e30 0x7f8e740721b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400c550 tx=0x7f8e6400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.082+0000 7f8e70ff9700 1 -- 172.21.15.7:0/2991539177 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e64004c80 con 0x7f8e740c2e30 2023-03-03T08:43:50.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.082+0000 7f8e70ff9700 1 -- 172.21.15.7:0/2991539177 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e64004e00 con 0x7f8e740c2e30 2023-03-03T08:43:50.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.082+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2991539177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e74157c80 con 0x7f8e740c2e30 2023-03-03T08:43:50.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.082+0000 7f8e70ff9700 1 -- 172.21.15.7:0/2991539177 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e64007310 con 0x7f8e740c2e30 2023-03-03T08:43:50.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.082+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2991539177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7406d360 con 0x7f8e740c2e30 2023-03-03T08:43:50.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.083+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2991539177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e74154a50 con 0x7f8e740c2e30 2023-03-03T08:43:50.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.083+0000 7f8e70ff9700 1 -- 172.21.15.7:0/2991539177 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e6401a020 con 0x7f8e740c2e30 2023-03-03T08:43:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.084+0000 7f8e70ff9700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e540682f0 0x7f8e5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.084+0000 7f8e70ff9700 1 -- 172.21.15.7:0/2991539177 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f8e64085ba0 con 0x7f8e740c2e30 2023-03-03T08:43:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.084+0000 7f8e73fff700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e540682f0 0x7f8e5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.084+0000 7f8e73fff700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e540682f0 0x7f8e5406a7a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009c40 tx=0x7f8e5c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.088+0000 7f8e70ff9700 1 -- 172.21.15.7:0/2991539177 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e64052120 con 0x7f8e740c2e30 2023-03-03T08:43:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.285+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2991539177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e741494a0 con 0x7f8e740c2e30 2023-03-03T08:43:50.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.287+0000 7f8e70ff9700 1 -- 172.21.15.7:0/2991539177 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21533 (secure 0 0 0) 0x7f8e64055740 con 0x7f8e740c2e30 2023-03-03T08:43:50.289 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:50.290 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:44.973251+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:50.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e540682f0 msgr2=0x7f8e5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e540682f0 0x7f8e5406a7a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009c40 tx=0x7f8e5c008030 comp rx=0 tx=0).stop 2023-03-03T08:43:50.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2991539177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e740c2e30 msgr2=0x7f8e740721b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e740c2e30 0x7f8e740721b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400c550 tx=0x7f8e6400c910 comp rx=0 tx=0).stop 2023-03-03T08:43:50.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2991539177 shutdown_connections 2023-03-03T08:43:50.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e74151cd0 0x7f8e74141150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7406c6c0 0x7f8e74071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e540682f0 0x7f8e5406a7a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 --2- 172.21.15.7:0/2991539177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e740c2e30 0x7f8e740721b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2991539177 >> 172.21.15.7:0/2991539177 conn(0x7f8e740a94d0 msgr2=0x7f8e74153600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2991539177 shutdown_connections 2023-03-03T08:43:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.289+0000 7f8e7a6e9700 1 -- 172.21.15.7:0/2991539177 wait complete. 2023-03-03T08:43:50.304 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-03-03T08:43:50.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.427+0000 7fce745f5700 1 Processor -- start 2023-03-03T08:43:50.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.427+0000 7fce745f5700 1 -- start start 2023-03-03T08:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce745f5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c14b460 0x7fce6c06aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce745f5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fce6c072470 0x7fce6c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce745f5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6c070360 con 0x7fce6c071990 2023-03-03T08:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce745f5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6c1521a0 con 0x7fce6c072470 2023-03-03T08:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce745f5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c06a5e0 con 0x7fce6c14b460 2023-03-03T08:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce72b92700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fce6c072470 0x7fce6c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce71b90700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c14b460 0x7fce6c06aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce72b92700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fce6c072470 0x7fce6c06afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47342/0 (socket says 172.21.15.7:47342) 2023-03-03T08:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce72b92700 1 -- 172.21.15.7:0/2391472463 learned_addr learned my addr 172.21.15.7:0/2391472463 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce71b90700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c14b460 0x7fce6c06aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59638/0 (socket says 172.21.15.7:59638) 2023-03-03T08:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce72391700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fce6c071990 0x7fce6c071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49064/0 (socket says 172.21.15.7:49064) 2023-03-03T08:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce71b90700 1 -- 172.21.15.7:0/2391472463 >> v2:172.21.15.143:3300/0 conn(0x7fce6c072470 msgr2=0x7fce6c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce71b90700 1 --2- 172.21.15.7:0/2391472463 >> v2:172.21.15.143:3300/0 conn(0x7fce6c072470 0x7fce6c06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.428+0000 7fce71b90700 1 -- 172.21.15.7:0/2391472463 >> v1:172.21.15.143:6789/0 conn(0x7fce6c071990 legacy=0x7fce6c071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.429+0000 7fce71b90700 1 -- 172.21.15.7:0/2391472463 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c0c38b0 con 0x7fce6c14b460 2023-03-03T08:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.429+0000 7fce72b92700 1 --2- 172.21.15.7:0/2391472463 >> v2:172.21.15.143:3300/0 conn(0x7fce6c072470 0x7fce6c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.429+0000 7fce71b90700 1 --2- 172.21.15.7:0/2391472463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c14b460 0x7fce6c06aa70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fce68009840 tx=0x7fce68009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13ac48c4d90a2dc7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.429+0000 7fce7138f700 1 -- 172.21.15.7:0/2391472463 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68011830 con 0x7fce6c14b460 2023-03-03T08:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.429+0000 7fce7138f700 1 -- 172.21.15.7:0/2391472463 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce68011990 con 0x7fce6c14b460 2023-03-03T08:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.429+0000 7fce7138f700 1 -- 172.21.15.7:0/2391472463 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68011d20 con 0x7fce6c14b460 2023-03-03T08:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.429+0000 7fce745f5700 1 -- 172.21.15.7:0/2391472463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c14b460 msgr2=0x7fce6c06aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.429+0000 7fce745f5700 1 --2- 172.21.15.7:0/2391472463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c14b460 0x7fce6c06aa70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fce68009840 tx=0x7fce68009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.430+0000 7fce745f5700 1 -- 172.21.15.7:0/2391472463 shutdown_connections 2023-03-03T08:43:50.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.430+0000 7fce745f5700 1 --2- 172.21.15.7:0/2391472463 >> v2:172.21.15.143:3300/0 conn(0x7fce6c072470 0x7fce6c06afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.430+0000 7fce745f5700 1 --2- 172.21.15.7:0/2391472463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c14b460 0x7fce6c06aa70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.430+0000 7fce745f5700 1 -- 172.21.15.7:0/2391472463 >> 172.21.15.7:0/2391472463 conn(0x7fce6c0a94d0 msgr2=0x7fce6c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.430+0000 7fce745f5700 1 -- 172.21.15.7:0/2391472463 shutdown_connections 2023-03-03T08:43:50.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.430+0000 7fce745f5700 1 -- 172.21.15.7:0/2391472463 wait complete. 2023-03-03T08:43:50.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.431+0000 7fce745f5700 1 Processor -- start 2023-03-03T08:43:50.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.431+0000 7fce745f5700 1 -- start start 2023-03-03T08:43:50.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.431+0000 7fce745f5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c071990 0x7fce6c0ab330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.431+0000 7fce745f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce6c072470 0x7fce6c0c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce745f5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce6c152570 0x7fce6c154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce745f5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c154760 con 0x7fce6c072470 2023-03-03T08:43:50.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce745f5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c153680 con 0x7fce6c152570 2023-03-03T08:43:50.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce745f5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c154a20 con 0x7fce6c071990 2023-03-03T08:43:50.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce71b90700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce6c072470 0x7fce6c0c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce71b90700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce6c072470 0x7fce6c0c45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51572/0 (socket says 172.21.15.7:51572) 2023-03-03T08:43:50.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce71b90700 1 -- 172.21.15.7:0/2149727242 learned_addr learned my addr 172.21.15.7:0/2149727242 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce72391700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c071990 0x7fce6c0ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce72b92700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce6c152570 0x7fce6c154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce71b90700 1 -- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c071990 msgr2=0x7fce6c0ab330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce71b90700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c071990 0x7fce6c0ab330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce71b90700 1 -- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce6c152570 msgr2=0x7fce6c154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce71b90700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce6c152570 0x7fce6c154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce71b90700 1 -- 172.21.15.7:0/2149727242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6400bd40 con 0x7fce6c072470 2023-03-03T08:43:50.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce72b92700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce6c152570 0x7fce6c154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:50.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce71b90700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce6c072470 0x7fce6c0c45d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fce68009fd0 tx=0x7fce68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce62ffd700 1 -- 172.21.15.7:0/2149727242 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce680036a0 con 0x7fce6c072470 2023-03-03T08:43:50.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce62ffd700 1 -- 172.21.15.7:0/2149727242 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6801f070 con 0x7fce6c072470 2023-03-03T08:43:50.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce745f5700 1 -- 172.21.15.7:0/2149727242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce680094a0 con 0x7fce6c072470 2023-03-03T08:43:50.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce62ffd700 1 -- 172.21.15.7:0/2149727242 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce68003a60 con 0x7fce6c072470 2023-03-03T08:43:50.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.432+0000 7fce745f5700 1 -- 172.21.15.7:0/2149727242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c0c3b50 con 0x7fce6c072470 2023-03-03T08:43:50.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.433+0000 7fce745f5700 1 -- 172.21.15.7:0/2149727242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c06a2b0 con 0x7fce6c072470 2023-03-03T08:43:50.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.434+0000 7fce62ffd700 1 -- 172.21.15.7:0/2149727242 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fce6800f050 con 0x7fce6c072470 2023-03-03T08:43:50.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.434+0000 7fce62ffd700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce500682f0 0x7fce5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.434+0000 7fce62ffd700 1 -- 172.21.15.7:0/2149727242 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fce68021070 con 0x7fce6c072470 2023-03-03T08:43:50.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.434+0000 7fce72391700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce500682f0 0x7fce5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.435+0000 7fce72391700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce500682f0 0x7fce5006a7a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fce5c000f60 tx=0x7fce5c007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.438+0000 7fce62ffd700 1 -- 172.21.15.7:0/2149727242 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fce68053d50 con 0x7fce6c072470 2023-03-03T08:43:50.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.634+0000 7fce745f5700 1 -- 172.21.15.7:0/2149727242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce6c06b4f0 con 0x7fce6c072470 2023-03-03T08:43:50.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.635+0000 7fce62ffd700 1 -- 172.21.15.7:0/2149727242 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21533 (secure 0 0 0) 0x7fce68057370 con 0x7fce6c072470 2023-03-03T08:43:50.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:50.638 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:44.973251+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.637+0000 7fce745f5700 1 -- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce500682f0 msgr2=0x7fce5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce500682f0 0x7fce5006a7a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fce5c000f60 tx=0x7fce5c007d30 comp rx=0 tx=0).stop 2023-03-03T08:43:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 -- 172.21.15.7:0/2149727242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce6c072470 msgr2=0x7fce6c0c45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce6c072470 0x7fce6c0c45d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fce68009fd0 tx=0x7fce68004750 comp rx=0 tx=0).stop 2023-03-03T08:43:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 -- 172.21.15.7:0/2149727242 shutdown_connections 2023-03-03T08:43:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce6c152570 0x7fce6c154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fce6c071990 0x7fce6c0ab330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce500682f0 0x7fce5006a7a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 --2- 172.21.15.7:0/2149727242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce6c072470 0x7fce6c0c45d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 -- 172.21.15.7:0/2149727242 >> 172.21.15.7:0/2149727242 conn(0x7fce6c0a94d0 msgr2=0x7fce6c148970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 -- 172.21.15.7:0/2149727242 shutdown_connections 2023-03-03T08:43:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.638+0000 7fce745f5700 1 -- 172.21.15.7:0/2149727242 wait complete. 2023-03-03T08:43:50.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-03-03T08:43:50.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-03-03T08:43:50.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-03-03T08:43:50.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-03-03T08:43:50.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-03-03T08:43:50.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-03-03T08:43:50.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-03-03T08:43:50.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-03-03T08:43:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.870+0000 7f6b12e74700 1 Processor -- start 2023-03-03T08:43:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.871+0000 7fe687428700 1 Processor -- start 2023-03-03T08:43:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.871+0000 7fe687428700 1 -- start start 2023-03-03T08:43:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.873+0000 7fe687428700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe6800aba70 0x7fe6800abe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.873+0000 7fe687428700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800a8b30 0x7fe680158c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.873+0000 7fe687428700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe680159160 0x7fe68015be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.873+0000 7fe687428700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6800aabe0 con 0x7fe680159160 2023-03-03T08:43:50.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.873+0000 7fe687428700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6800a97e0 con 0x7fe6800aba70 2023-03-03T08:43:50.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.873+0000 7fe687428700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6800a9500 con 0x7fe6800a8b30 2023-03-03T08:43:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.874+0000 7fe6851c4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe6800aba70 0x7fe6800abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.874+0000 7fe6851c4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe6800aba70 0x7fe6800abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47352/0 (socket says 172.21.15.7:47352) 2023-03-03T08:43:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.874+0000 7fe6851c4700 1 -- 172.21.15.7:0/1034084171 learned_addr learned my addr 172.21.15.7:0/1034084171 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.875+0000 7fe6849c3700 1 --2- 172.21.15.7:0/1034084171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800a8b30 0x7fe680158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.875+0000 7fe6851c4700 1 -- 172.21.15.7:0/1034084171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800a8b30 msgr2=0x7fe680158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.875+0000 7fe6851c4700 1 --2- 172.21.15.7:0/1034084171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800a8b30 0x7fe680158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.875+0000 7fe6851c4700 1 -- 172.21.15.7:0/1034084171 >> v2:172.21.15.7:3300/0 conn(0x7fe680159160 msgr2=0x7fe68015be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.875+0000 7fe6851c4700 1 --2- 172.21.15.7:0/1034084171 >> v2:172.21.15.7:3300/0 conn(0x7fe680159160 0x7fe68015be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.875+0000 7fe6851c4700 1 -- 172.21.15.7:0/1034084171 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6800a9e30 con 0x7fe6800aba70 2023-03-03T08:43:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.875+0000 7fe6849c3700 1 --2- 172.21.15.7:0/1034084171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800a8b30 0x7fe680158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.875+0000 7fe6851c4700 1 --2- 172.21.15.7:0/1034084171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6800aba70 0x7fe6800abe40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe68006ebe0 tx=0x7fe670009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94cf6f6329f83fcd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.876+0000 7fe677fff700 1 -- 172.21.15.7:0/1034084171 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000b870 con 0x7fe6800aba70 2023-03-03T08:43:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.876+0000 7fe677fff700 1 -- 172.21.15.7:0/1034084171 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67000b9d0 con 0x7fe6800aba70 2023-03-03T08:43:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.876+0000 7fe677fff700 1 -- 172.21.15.7:0/1034084171 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67000bd60 con 0x7fe6800aba70 2023-03-03T08:43:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.876+0000 7fe687428700 1 -- 172.21.15.7:0/1034084171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6800aba70 msgr2=0x7fe6800abe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.876+0000 7fe687428700 1 --2- 172.21.15.7:0/1034084171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6800aba70 0x7fe6800abe40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe68006ebe0 tx=0x7fe670009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.877+0000 7fe687428700 1 -- 172.21.15.7:0/1034084171 shutdown_connections 2023-03-03T08:43:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.877+0000 7fe687428700 1 --2- 172.21.15.7:0/1034084171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6800aba70 0x7fe6800abe40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.877+0000 7fe687428700 1 --2- 172.21.15.7:0/1034084171 >> v2:172.21.15.7:3300/0 conn(0x7fe680159160 0x7fe68015be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.877+0000 7fe687428700 1 --2- 172.21.15.7:0/1034084171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800a8b30 0x7fe680158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.877+0000 7fe687428700 1 -- 172.21.15.7:0/1034084171 >> 172.21.15.7:0/1034084171 conn(0x7fe6800c62e0 msgr2=0x7fe6800c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.877+0000 7fe687428700 1 -- 172.21.15.7:0/1034084171 shutdown_connections 2023-03-03T08:43:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.877+0000 7fe687428700 1 -- 172.21.15.7:0/1034084171 wait complete. 2023-03-03T08:43:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.877+0000 7f6b12e74700 1 -- start start 2023-03-03T08:43:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.878+0000 7f808a2dd700 1 Processor -- start 2023-03-03T08:43:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.878+0000 7f808a2dd700 1 -- start start 2023-03-03T08:43:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f6b12e74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b0c0c6df0 0x7f6b0c150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f6b12e74700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0c06c100 con 0x7f6b0c0c0c60 2023-03-03T08:43:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f6b12e74700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b0c0ac3b0 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f6b12e74700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c150840 con 0x7f6b0c0c6df0 2023-03-03T08:43:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f808a2dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8084161cd0 0x7f808408b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f808a2dd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f808408b970 0x7f80840c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f808a2dd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f808406bf50 0x7f808406c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f808a2dd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80840aa420 con 0x7f808406bf50 2023-03-03T08:43:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f808a2dd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f808406bac0 con 0x7f808408b970 2023-03-03T08:43:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f808a2dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80840ac270 con 0x7f8084161cd0 2023-03-03T08:43:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f6b10c10700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f6b0c0a93d0 0x7f6b0c14dab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49084/0 (socket says 172.21.15.7:49084) 2023-03-03T08:43:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f6b10c10700 1 -- 172.21.15.7:0/2427800477 learned_addr learned my addr 172.21.15.7:0/2427800477 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7fe687428700 1 Processor -- start 2023-03-03T08:43:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f6b11411700 1 --2- 172.21.15.7:0/2427800477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b0c0c6df0 0x7f6b0c150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.879+0000 7f808887a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f808406bf50 0x7f808406c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f808887a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f808406bf50 0x7f808406c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51590/0 (socket says 172.21.15.7:51590) 2023-03-03T08:43:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f808887a700 1 -- 172.21.15.7:0/1125906812 learned_addr learned my addr 172.21.15.7:0/1125906812 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f8083fff700 1 --2- 172.21.15.7:0/1125906812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8084161cd0 0x7f808408b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f808887a700 1 -- 172.21.15.7:0/1125906812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8084161cd0 msgr2=0x7f808408b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f808887a700 1 --2- 172.21.15.7:0/1125906812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8084161cd0 0x7f808408b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f808887a700 1 -- 172.21.15.7:0/1125906812 >> v2:172.21.15.143:3300/0 conn(0x7f808408b970 msgr2=0x7f80840c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f808887a700 1 --2- 172.21.15.7:0/1125906812 >> v2:172.21.15.143:3300/0 conn(0x7f808408b970 0x7f80840c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f808887a700 1 -- 172.21.15.7:0/1125906812 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80840a9e60 con 0x7f808406bf50 2023-03-03T08:43:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f6b11411700 1 -- 172.21.15.7:0/2427800477 >> v1:172.21.15.143:6789/0 conn(0x7f6b0c0a93d0 legacy=0x7f6b0c14dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f6b11411700 1 -- 172.21.15.7:0/2427800477 >> v1:172.21.15.7:6789/0 conn(0x7f6b0c0c0c60 legacy=0x7f6b0c0aa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7f6b11411700 1 -- 172.21.15.7:0/2427800477 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c0a9b40 con 0x7f6b0c0c6df0 2023-03-03T08:43:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.880+0000 7fe687428700 1 -- start start 2023-03-03T08:43:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7f6b11411700 1 --2- 172.21.15.7:0/2427800477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b0c0c6df0 0x7f6b0c150cd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009870 tx=0x7f6afc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2bfc170cb6d8235 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7f808887a700 1 --2- 172.21.15.7:0/1125906812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f808406bf50 0x7f808406c3c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f80840746d0 tx=0x7f807400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e224b136a54dd75b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7f6b0b7fe700 1 -- 172.21.15.7:0/2427800477 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc004230 con 0x7f6b0c0c6df0 2023-03-03T08:43:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7f6b0b7fe700 1 -- 172.21.15.7:0/2427800477 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6afc004390 con 0x7f6b0c0c6df0 2023-03-03T08:43:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7f6b0b7fe700 1 -- 172.21.15.7:0/2427800477 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0036a0 con 0x7f6b0c0c6df0 2023-03-03T08:43:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7fe687428700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6800a8b30 0x7fe68006f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7fe687428700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800aba70 0x7fe680154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7fe687428700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680159160 0x7fe680155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7fe687428700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68006f2d0 con 0x7fe680159160 2023-03-03T08:43:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7fe687428700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68015b780 con 0x7fe6800a8b30 2023-03-03T08:43:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7fe687428700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68015b9d0 con 0x7fe6800aba70 2023-03-03T08:43:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7f8082ffd700 1 -- 172.21.15.7:0/1125906812 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807400db60 con 0x7f808406bf50 2023-03-03T08:43:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7f8082ffd700 1 -- 172.21.15.7:0/1125906812 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807400dcc0 con 0x7f808406bf50 2023-03-03T08:43:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.881+0000 7f8082ffd700 1 -- 172.21.15.7:0/1125906812 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074017630 con 0x7f808406bf50 2023-03-03T08:43:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6859c5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680159160 0x7fe680155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6859c5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680159160 0x7fe680155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51598/0 (socket says 172.21.15.7:51598) 2023-03-03T08:43:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6859c5700 1 -- 172.21.15.7:0/1490414197 learned_addr learned my addr 172.21.15.7:0/1490414197 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7f63637d4700 1 Processor -- start 2023-03-03T08:43:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6849c3700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800aba70 0x7fe680154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7f63637d4700 1 -- start start 2023-03-03T08:43:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6849c3700 1 -- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6800a8b30 msgr2=0x7fe68006f730 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T08:43:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6849c3700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe6800a8b30 0x7fe68006f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6849c3700 1 -- 172.21.15.7:0/1490414197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680159160 msgr2=0x7fe680155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6849c3700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680159160 0x7fe680155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6849c3700 1 -- 172.21.15.7:0/1490414197 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670009550 con 0x7fe6800aba70 2023-03-03T08:43:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6859c5700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680159160 0x7fe680155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7f808a2dd700 1 -- 172.21.15.7:0/1125906812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f808406bf50 msgr2=0x7f808406c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7f808a2dd700 1 --2- 172.21.15.7:0/1125906812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f808406bf50 0x7f808406c3c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f80840746d0 tx=0x7f807400be70 comp rx=0 tx=0).stop 2023-03-03T08:43:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.882+0000 7fe6849c3700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800aba70 0x7fe680154fe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe67800cde0 tx=0x7fe67800a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f808a2dd700 1 -- 172.21.15.7:0/1125906812 shutdown_connections 2023-03-03T08:43:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f808a2dd700 1 --2- 172.21.15.7:0/1125906812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f808406bf50 0x7f808406c3c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f808a2dd700 1 --2- 172.21.15.7:0/1125906812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8084161cd0 0x7f808408b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f808a2dd700 1 --2- 172.21.15.7:0/1125906812 >> v2:172.21.15.143:3300/0 conn(0x7f808408b970 0x7f80840c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f808a2dd700 1 -- 172.21.15.7:0/1125906812 >> 172.21.15.7:0/1125906812 conn(0x7f80840c62e0 msgr2=0x7f80840c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f63637d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 0x7f635c06b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f63637d4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f635c06be60 0x7f635c0ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f63637d4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f635c14fd60 con 0x7f635c14f990 2023-03-03T08:43:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f63637d4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f635c153340 con 0x7f635c06be60 2023-03-03T08:43:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f63637d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c0c29c0 con 0x7f635c152ed0 2023-03-03T08:43:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f6361570700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f635c14f990 0x7f635c1562b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45584/0 (socket says 172.21.15.7:45584) 2023-03-03T08:43:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7f6361570700 1 -- 172.21.15.7:0/827963572 learned_addr learned my addr 172.21.15.7:0/827963572 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7fe675ffb700 1 -- 172.21.15.7:0/1490414197 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67800f480 con 0x7fe6800aba70 2023-03-03T08:43:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7fe675ffb700 1 -- 172.21.15.7:0/1490414197 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67800f5e0 con 0x7fe6800aba70 2023-03-03T08:43:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7fe675ffb700 1 -- 172.21.15.7:0/1490414197 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67800f970 con 0x7fe6800aba70 2023-03-03T08:43:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6800c2760 con 0x7fe6800aba70 2023-03-03T08:43:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.883+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6800aa110 con 0x7fe6800aba70 2023-03-03T08:43:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.884+0000 7f808a2dd700 1 -- 172.21.15.7:0/1125906812 shutdown_connections 2023-03-03T08:43:50.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.884+0000 7f808a2dd700 1 -- 172.21.15.7:0/1125906812 wait complete. 2023-03-03T08:43:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.884+0000 7f6360d6f700 1 --2- 172.21.15.7:0/827963572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 0x7f635c06b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.884+0000 7f6353fff700 1 -- 172.21.15.7:0/827963572 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396196594 0 0) 0x7f635c14fd60 con 0x7f635c14f990 2023-03-03T08:43:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.884+0000 7f6353fff700 1 -- 172.21.15.7:0/827963572 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63480035b0 con 0x7f635c14f990 2023-03-03T08:43:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.884+0000 7f6360d6f700 1 -- 172.21.15.7:0/827963572 >> v2:172.21.15.143:3300/0 conn(0x7f635c06be60 msgr2=0x7f635c0ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.884+0000 7f6360d6f700 1 --2- 172.21.15.7:0/827963572 >> v2:172.21.15.143:3300/0 conn(0x7f635c06be60 0x7f635c0ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.884+0000 7f6360d6f700 1 -- 172.21.15.7:0/827963572 >> v1:172.21.15.7:6789/0 conn(0x7f635c14f990 legacy=0x7f635c1562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.884+0000 7f6360d6f700 1 -- 172.21.15.7:0/827963572 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c0c2400 con 0x7f635c152ed0 2023-03-03T08:43:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.885+0000 7f6360d6f700 1 --2- 172.21.15.7:0/827963572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 0x7f635c06b920 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6354009d90 tx=0x7f635400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8325744a93580f1a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.885+0000 7f6353fff700 1 -- 172.21.15.7:0/827963572 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635400e070 con 0x7f635c152ed0 2023-03-03T08:43:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.885+0000 7f6353fff700 1 -- 172.21.15.7:0/827963572 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635400bb20 con 0x7f635c152ed0 2023-03-03T08:43:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.885+0000 7f6353fff700 1 -- 172.21.15.7:0/827963572 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354012430 con 0x7f635c152ed0 2023-03-03T08:43:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7f63637d4700 1 -- 172.21.15.7:0/827963572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 msgr2=0x7f635c06b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7f63637d4700 1 --2- 172.21.15.7:0/827963572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 0x7f635c06b920 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6354009d90 tx=0x7f635400b330 comp rx=0 tx=0).stop 2023-03-03T08:43:50.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.885+0000 7fe675ffb700 1 -- 172.21.15.7:0/1490414197 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe678009030 con 0x7fe6800aba70 2023-03-03T08:43:50.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.885+0000 7fe675ffb700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6640683c0 0x7fe66406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7fe675ffb700 1 -- 172.21.15.7:0/1490414197 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fe678086820 con 0x7fe6800aba70 2023-03-03T08:43:50.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7fe6851c4700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6640683c0 0x7fe66406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7f808a2dd700 1 Processor -- start 2023-03-03T08:43:50.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7eff5d2c1700 1 Processor -- start 2023-03-03T08:43:50.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7f808a2dd700 1 -- start start 2023-03-03T08:43:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7eff5d2c1700 1 -- start start 2023-03-03T08:43:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7fe687428700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.7:6800/96730,v1:172.21.15.7:6801/96730] conn(0x7fe68015bec0 0x7fe68015c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 --> [v2:172.21.15.7:6800/96730,v1:172.21.15.7:6801/96730] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe680074bb0 con 0x7fe68015bec0 2023-03-03T08:43:50.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7fd434c99700 1 Processor -- start 2023-03-03T08:43:50.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.886+0000 7fd434c99700 1 -- start start 2023-03-03T08:43:50.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7fe6859c5700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.7:6800/96730,v1:172.21.15.7:6801/96730] conn(0x7fe68015bec0 0x7fe68015c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7f808a2dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f808406bf50 0x7f808406f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7f808a2dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f808408b970 0x7f808420bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7f808a2dd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8084161cd0 0x7f80842085a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7f808a2dd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808420c2c0 con 0x7f8084161cd0 2023-03-03T08:43:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7f808a2dd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80841ef2f0 con 0x7f808408b970 2023-03-03T08:43:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7f808a2dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8084207e00 con 0x7f808406bf50 2023-03-03T08:43:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7f8083fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f808406bf50 0x7f808406f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7f8083fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f808406bf50 0x7f808406f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59684/0 (socket says 172.21.15.7:59684) 2023-03-03T08:43:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7f8083fff700 1 -- 172.21.15.7:0/2000664621 learned_addr learned my addr 172.21.15.7:0/2000664621 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7f80837fe700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f808408b970 0x7f808420bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7eff5d2c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff580c5140 0x7eff58157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7eff5d2c1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff581580e0 0x7eff5806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7eff5d2c1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5806dad0 con 0x7eff58151de0 2023-03-03T08:43:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7eff5d2c1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff5806e370 con 0x7eff581580e0 2023-03-03T08:43:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.887+0000 7eff5d2c1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5806ddb0 con 0x7eff580c5140 2023-03-03T08:43:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fe6851c4700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6640683c0 0x7fe66406a870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe670009c10 tx=0x7fe670008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f808887a700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8084161cd0 0x7f80842085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd434c99700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 0x7fd4300c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd434c99700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd4300c1a40 0x7fd430157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd434c99700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd43006dad0 con 0x7fd4300c5140 2023-03-03T08:43:50.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7eff577fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff581580e0 0x7eff5806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd434c99700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd43006e370 con 0x7fd4300c1a40 2023-03-03T08:43:50.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd434c99700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43006ddb0 con 0x7fd430151de0 2023-03-03T08:43:50.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fe6859c5700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.7:6800/96730,v1:172.21.15.7:6801/96730] conn(0x7fe68015bec0 0x7fe68015c290 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7eff577fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff581580e0 0x7eff5806a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51618/0 (socket says 172.21.15.7:51618) 2023-03-03T08:43:50.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7eff577fe700 1 -- 172.21.15.7:0/3323973235 learned_addr learned my addr 172.21.15.7:0/3323973235 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f80837fe700 1 -- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f808406bf50 msgr2=0x7f808406f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f80837fe700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f808406bf50 0x7f808406f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f80837fe700 1 -- 172.21.15.7:0/2000664621 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8084161cd0 msgr2=0x7f80842085a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f80837fe700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8084161cd0 0x7f80842085a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f80837fe700 1 -- 172.21.15.7:0/2000664621 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807400b7f0 con 0x7f808408b970 2023-03-03T08:43:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd42e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 0x7fd4300c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f6b12e74700 1 -- 172.21.15.7:0/2427800477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b0c0c6df0 msgr2=0x7f6b0c150cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f6b12e74700 1 --2- 172.21.15.7:0/2427800477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b0c0c6df0 0x7f6b0c150cd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009870 tx=0x7f6afc009b80 comp rx=0 tx=0).stop 2023-03-03T08:43:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f80837fe700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f808408b970 0x7f808420bd80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f807c00bd40 tx=0x7f807c00bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd42e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 0x7fd4300c1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59692/0 (socket says 172.21.15.7:59692) 2023-03-03T08:43:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd42e59c700 1 -- 172.21.15.7:0/900785567 learned_addr learned my addr 172.21.15.7:0/900785567 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7eff577fe700 1 -- 172.21.15.7:0/3323973235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff580c5140 msgr2=0x7eff58157ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T08:43:50.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7eff577fe700 1 --2- 172.21.15.7:0/3323973235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff580c5140 0x7eff58157ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7eff577fe700 1 -- 172.21.15.7:0/3323973235 >> v1:172.21.15.143:6789/0 conn(0x7eff58151de0 legacy=0x7eff580c1500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T08:43:50.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7eff577fe700 1 -- 172.21.15.7:0/3323973235 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5806e090 con 0x7eff581580e0 2023-03-03T08:43:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd42e59c700 1 -- 172.21.15.7:0/900785567 >> v2:172.21.15.7:3300/0 conn(0x7fd4300c1a40 msgr2=0x7fd430157ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T08:43:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f6b12e74700 1 -- 172.21.15.7:0/2427800477 shutdown_connections 2023-03-03T08:43:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f6b12e74700 1 --2- 172.21.15.7:0/2427800477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b0c0c6df0 0x7f6b0c150cd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7f6b12e74700 1 -- 172.21.15.7:0/2427800477 >> 172.21.15.7:0/2427800477 conn(0x7f6b0c0c62e0 msgr2=0x7f6b0c0c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd42e59c700 1 --2- 172.21.15.7:0/900785567 >> v2:172.21.15.7:3300/0 conn(0x7fd4300c1a40 0x7fd430157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd42e59c700 1 -- 172.21.15.7:0/900785567 >> v1:172.21.15.7:6789/0 conn(0x7fd4300c5140 legacy=0x7fd43006a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.889+0000 7eff577fe700 1 --2- 172.21.15.7:0/3323973235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff581580e0 0x7eff5806a2b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7eff50009e50 tx=0x7eff5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763c2a09f6e9066a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.889+0000 7f6b12e74700 1 -- 172.21.15.7:0/2427800477 shutdown_connections 2023-03-03T08:43:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd42d59a700 1 -- 172.21.15.7:0/900785567 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056771837 0 0) 0x7fd43006dad0 con 0x7fd4300c5140 2023-03-03T08:43:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.888+0000 7fd42e59c700 1 -- 172.21.15.7:0/900785567 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43006e090 con 0x7fd430151de0 2023-03-03T08:43:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.889+0000 7f6b12e74700 1 -- 172.21.15.7:0/2427800477 wait complete. 2023-03-03T08:43:50.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.889+0000 7fd42e59c700 1 --2- 172.21.15.7:0/900785567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 0x7fd4300c1500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd4240098f0 tx=0x7fd424009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78545455ef89e4f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.889+0000 7fe675ffb700 1 -- 172.21.15.7:0/1490414197 <== osd.0 v2:172.21.15.7:6800/96730 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe680074bb0 con 0x7fe68015bec0 2023-03-03T08:43:50.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.889+0000 7fd42d59a700 1 -- 172.21.15.7:0/900785567 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42400b8a0 con 0x7fd430151de0 2023-03-03T08:43:50.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.889+0000 7fd42d59a700 1 -- 172.21.15.7:0/900785567 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42400ba00 con 0x7fd430151de0 2023-03-03T08:43:50.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.889+0000 7eff55ffb700 1 -- 172.21.15.7:0/3323973235 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5000e070 con 0x7eff581580e0 2023-03-03T08:43:50.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.889+0000 7eff55ffb700 1 -- 172.21.15.7:0/3323973235 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff50003a20 con 0x7eff581580e0 2023-03-03T08:43:50.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.889+0000 7eff55ffb700 1 -- 172.21.15.7:0/3323973235 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50003dd0 con 0x7eff581580e0 2023-03-03T08:43:50.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7fd42d59a700 1 -- 172.21.15.7:0/900785567 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42400bd90 con 0x7fd430151de0 2023-03-03T08:43:50.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7f63637d4700 1 -- 172.21.15.7:0/827963572 shutdown_connections 2023-03-03T08:43:50.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7f63637d4700 1 --2- 172.21.15.7:0/827963572 >> v2:172.21.15.143:3300/0 conn(0x7f635c06be60 0x7f635c0ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7f63637d4700 1 --2- 172.21.15.7:0/827963572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 0x7f635c06b920 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7f63637d4700 1 -- 172.21.15.7:0/827963572 >> 172.21.15.7:0/827963572 conn(0x7f635c0c62e0 msgr2=0x7f635c0c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7eff5d2c1700 1 -- 172.21.15.7:0/3323973235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff581580e0 msgr2=0x7eff5806a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7eff5d2c1700 1 --2- 172.21.15.7:0/3323973235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff581580e0 0x7eff5806a2b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7eff50009e50 tx=0x7eff5000b330 comp rx=0 tx=0).stop 2023-03-03T08:43:50.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7f63637d4700 1 -- 172.21.15.7:0/827963572 shutdown_connections 2023-03-03T08:43:50.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7fd434c99700 1 -- 172.21.15.7:0/900785567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 msgr2=0x7fd4300c1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7fd434c99700 1 --2- 172.21.15.7:0/900785567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 0x7fd4300c1500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd4240098f0 tx=0x7fd424009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:50.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7f6b12e74700 1 Processor -- start 2023-03-03T08:43:50.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7eff5d2c1700 1 -- 172.21.15.7:0/3323973235 shutdown_connections 2023-03-03T08:43:50.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7eff5d2c1700 1 --2- 172.21.15.7:0/3323973235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff581580e0 0x7eff5806a2b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7eff5d2c1700 1 --2- 172.21.15.7:0/3323973235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff580c5140 0x7eff58157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7eff5d2c1700 1 -- 172.21.15.7:0/3323973235 >> 172.21.15.7:0/3323973235 conn(0x7eff580a9430 msgr2=0x7eff580c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7eff5d2c1700 1 -- 172.21.15.7:0/3323973235 shutdown_connections 2023-03-03T08:43:50.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7eff5d2c1700 1 -- 172.21.15.7:0/3323973235 wait complete. 2023-03-03T08:43:50.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7eff5d2c1700 1 Processor -- start 2023-03-03T08:43:50.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7f6b12e74700 1 -- start start 2023-03-03T08:43:50.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.891+0000 7f6b12e74700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b0c0a93d0 0x7f6b0c153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.891+0000 7f6b12e74700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b0c0c0c60 0x7f6b0c13ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.891+0000 7f6b12e74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b0c0c6df0 0x7f6b0c070fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7eff5d2c1700 1 -- start start 2023-03-03T08:43:50.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.891+0000 7f6b12e74700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c1538f0 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f6b12e74700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c0704c0 con 0x7f6b0c0c0c60 2023-03-03T08:43:50.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f6b12e74700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c0707a0 con 0x7f6b0c0c6df0 2023-03-03T08:43:50.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.890+0000 7f63637d4700 1 -- 172.21.15.7:0/827963572 wait complete. 2023-03-03T08:43:50.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f63637d4700 1 Processor -- start 2023-03-03T08:43:50.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f63637d4700 1 -- start start 2023-03-03T08:43:50.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7fd434c99700 1 -- 172.21.15.7:0/900785567 shutdown_connections 2023-03-03T08:43:50.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7fd434c99700 1 --2- 172.21.15.7:0/900785567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 0x7fd4300c1500 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7fd434c99700 1 --2- 172.21.15.7:0/900785567 >> v2:172.21.15.7:3300/0 conn(0x7fd4300c1a40 0x7fd430157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7fd434c99700 1 -- 172.21.15.7:0/900785567 >> 172.21.15.7:0/900785567 conn(0x7fd4300a9430 msgr2=0x7fd4300c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f8080ff9700 1 -- 172.21.15.7:0/2000664621 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c00cc90 con 0x7f808408b970 2023-03-03T08:43:50.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f6b0bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b0c0c0c60 0x7f6b0c13ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f6b10c10700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b0c0a93d0 0x7f6b0c153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f6b0bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b0c0c0c60 0x7f6b0c13ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47400/0 (socket says 172.21.15.7:47400) 2023-03-03T08:43:50.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f6b10c10700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b0c0a93d0 0x7f6b0c153310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51628/0 (socket says 172.21.15.7:51628) 2023-03-03T08:43:50.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f6b0bfff700 1 -- 172.21.15.7:0/2871576077 learned_addr learned my addr 172.21.15.7:0/2871576077 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f6b11411700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b0c0c6df0 0x7f6b0c070fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff5d2c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff580c5140 0x7eff58148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff5d2c1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff58151de0 0x7eff5814d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff5d2c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff581580e0 0x7eff580aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff5d2c1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5814d1c0 con 0x7eff58151de0 2023-03-03T08:43:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff5d2c1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5814c120 con 0x7eff581580e0 2023-03-03T08:43:50.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff5d2c1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5814c400 con 0x7eff580c5140 2023-03-03T08:43:50.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7fd434c99700 1 -- 172.21.15.7:0/900785567 shutdown_connections 2023-03-03T08:43:50.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f63637d4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c06be60 0x7f635c1560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f63637d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f635c14f990 0x7f635c0aa060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f63637d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 0x7f635c14a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f63637d4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c156630 con 0x7f635c06be60 2023-03-03T08:43:50.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f63637d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c155c60 con 0x7f635c14f990 2023-03-03T08:43:50.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f63637d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f635c155650 con 0x7f635c152ed0 2023-03-03T08:43:50.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff56ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff580c5140 0x7eff58148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff56ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff580c5140 0x7eff58148960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59704/0 (socket says 172.21.15.7:59704) 2023-03-03T08:43:50.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff56ffd700 1 -- 172.21.15.7:0/236559808 learned_addr learned my addr 172.21.15.7:0/236559808 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f6b10c10700 1 -- 172.21.15.7:0/2871576077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b0c0c6df0 msgr2=0x7f6b0c070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f6b10c10700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b0c0c6df0 0x7f6b0c070fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f6b10c10700 1 -- 172.21.15.7:0/2871576077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b0c0c0c60 msgr2=0x7f6b0c13ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f6b10c10700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b0c0c0c60 0x7f6b0c13ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f6b10c10700 1 -- 172.21.15.7:0/2871576077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc0094a0 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff577fe700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff581580e0 0x7eff580aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff567fc700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff58151de0 0x7eff5814d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f8080ff9700 1 -- 172.21.15.7:0/2000664621 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807c00cdf0 con 0x7f808408b970 2023-03-03T08:43:50.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f8080ff9700 1 -- 172.21.15.7:0/2000664621 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c014680 con 0x7f808408b970 2023-03-03T08:43:50.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80840c2710 con 0x7f808408b970 2023-03-03T08:43:50.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.892+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80840aa140 con 0x7f808408b970 2023-03-03T08:43:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f6361570700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c06be60 0x7f635c1560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f6361570700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c06be60 0x7f635c1560f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51640/0 (socket says 172.21.15.7:51640) 2023-03-03T08:43:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f6361570700 1 -- 172.21.15.7:0/17901598 learned_addr learned my addr 172.21.15.7:0/17901598 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f6361d71700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 0x7f635c14a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7f6360d6f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f635c14f990 0x7f635c0aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff56ffd700 1 -- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff581580e0 msgr2=0x7eff580aafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff56ffd700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff581580e0 0x7eff580aafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff56ffd700 1 -- 172.21.15.7:0/236559808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff58151de0 msgr2=0x7eff5814d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff56ffd700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff58151de0 0x7eff5814d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.893+0000 7eff56ffd700 1 -- 172.21.15.7:0/236559808 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff50009b30 con 0x7eff580c5140 2023-03-03T08:43:50.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6b10c10700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b0c0a93d0 0x7f6b0c153310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000c4d0 tx=0x7f6b0000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6361d71700 1 -- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f635c14f990 msgr2=0x7f635c0aa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6361d71700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f635c14f990 0x7f635c0aa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6361d71700 1 -- 172.21.15.7:0/17901598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c06be60 msgr2=0x7f635c1560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6361d71700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f635c06be60 0x7f635c1560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6361d71700 1 -- 172.21.15.7:0/17901598 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6354009a70 con 0x7f635c152ed0 2023-03-03T08:43:50.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6b097fa700 1 -- 172.21.15.7:0/2871576077 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b00005d70 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6361d71700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 0x7f635c14a3a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f635800b410 tx=0x7f635800b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6b097fa700 1 -- 172.21.15.7:0/2871576077 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b000070d0 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6b097fa700 1 -- 172.21.15.7:0/2871576077 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b00007480 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7eff56ffd700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff580c5140 0x7eff58148960 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff4800b450 tx=0x7eff4800b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7eff3f7fe700 1 -- 172.21.15.7:0/236559808 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48011830 con 0x7eff580c5140 2023-03-03T08:43:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7eff3f7fe700 1 -- 172.21.15.7:0/236559808 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff480119b0 con 0x7eff580c5140 2023-03-03T08:43:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7eff3f7fe700 1 -- 172.21.15.7:0/236559808 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48011d60 con 0x7eff580c5140 2023-03-03T08:43:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff580c1c10 con 0x7eff580c5140 2023-03-03T08:43:50.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff580c4c20 con 0x7eff580c5140 2023-03-03T08:43:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0c0aa820 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0c0ac0d0 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.895+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f808406c900 con 0x7f808408b970 2023-03-03T08:43:50.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.895+0000 7f6351ffb700 1 -- 172.21.15.7:0/17901598 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358011620 con 0x7f635c152ed0 2023-03-03T08:43:50.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.895+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c1523f0 con 0x7f635c152ed0 2023-03-03T08:43:50.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.895+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c0c26e0 con 0x7f635c152ed0 2023-03-03T08:43:50.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.895+0000 7f6351ffb700 1 -- 172.21.15.7:0/17901598 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63580117a0 con 0x7f635c152ed0 2023-03-03T08:43:50.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.896+0000 7f6351ffb700 1 -- 172.21.15.7:0/17901598 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6358011b50 con 0x7f635c152ed0 2023-03-03T08:43:50.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7f6351ffb700 1 -- 172.21.15.7:0/17901598 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f635800f050 con 0x7f635c152ed0 2023-03-03T08:43:50.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6351ffb700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f63400683c0 0x7f634006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6351ffb700 1 -- 172.21.15.7:0/17901598 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6358085730 con 0x7f635c152ed0 2023-03-03T08:43:50.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6361570700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f63400683c0 0x7f634006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7f6361570700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f63400683c0 0x7f634006a870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f634c0011e0 tx=0x7f634c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.895+0000 7fdf3ccc3700 1 Processor -- start 2023-03-03T08:43:50.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7fdf3ccc3700 1 -- start start 2023-03-03T08:43:50.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf3ccc3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf38151e80 0x7fdf380c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf3ccc3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf380c6fe0 0x7fdf380bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf3ccc3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf380c08d0 con 0x7fdf38155410 2023-03-03T08:43:50.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf3ccc3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf38152250 con 0x7fdf38151e80 2023-03-03T08:43:50.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf3ccc3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380c0fb0 con 0x7fdf380c6fe0 2023-03-03T08:43:50.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf35d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdf38155410 0x7fdf380bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45646/0 (socket says 172.21.15.7:45646) 2023-03-03T08:43:50.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf35d9b700 1 -- 172.21.15.7:0/4069966155 learned_addr learned my addr 172.21.15.7:0/4069966155 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:50.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf3659c700 1 --2- 172.21.15.7:0/4069966155 >> v2:172.21.15.7:3300/0 conn(0x7fdf38151e80 0x7fdf380c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf36d9d700 1 --2- 172.21.15.7:0/4069966155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf380c6fe0 0x7fdf380bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.895+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6b0c06b340 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7f6b097fa700 1 -- 172.21.15.7:0/2871576077 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6b00019030 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f8080ff9700 1 -- 172.21.15.7:0/2000664621 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f807c01b030 con 0x7f808408b970 2023-03-03T08:43:50.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7f8080ff9700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80700683c0 0x7f807006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7f8080ff9700 1 -- 172.21.15.7:0/2000664621 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f807c006e40 con 0x7f808408b970 2023-03-03T08:43:50.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.895+0000 7fb374a74700 1 Processor -- start 2023-03-03T08:43:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.895+0000 7fb374a74700 1 -- start start 2023-03-03T08:43:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7fb374a74700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb3700a8bd0 0x7fb3701593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7fb374a74700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3700a9de0 con 0x7fb37006c0e0 2023-03-03T08:43:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7fb374a74700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3700c2a50 con 0x7fb3700c2680 2023-03-03T08:43:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7fb374a74700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3700aa5f0 con 0x7fb3700a8bd0 2023-03-03T08:43:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7fb36e59c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb3700c2680 0x7fb3700aba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49144/0 (socket says 172.21.15.7:49144) 2023-03-03T08:43:50.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7fb36e59c700 1 -- 172.21.15.7:0/1016352346 learned_addr learned my addr 172.21.15.7:0/1016352346 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7fb36dd9b700 1 --2- 172.21.15.7:0/1016352346 >> v2:172.21.15.143:3300/0 conn(0x7fb3700a8bd0 0x7fb3701593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fb36dd9b700 1 -- 172.21.15.7:0/1016352346 >> v1:172.21.15.143:6789/0 conn(0x7fb3700c2680 legacy=0x7fb3700aba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fb36dd9b700 1 -- 172.21.15.7:0/1016352346 >> v1:172.21.15.7:6789/0 conn(0x7fb37006c0e0 legacy=0x7fb37006c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fb36dd9b700 1 -- 172.21.15.7:0/1016352346 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3700a9b10 con 0x7fb3700a8bd0 2023-03-03T08:43:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fb36dd9b700 1 --2- 172.21.15.7:0/1016352346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3700a8bd0 0x7fb3701593f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb3640097c0 tx=0x7fb364009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e19cc2dc6f1e61 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.894+0000 7fd434c99700 1 -- 172.21.15.7:0/900785567 wait complete. 2023-03-03T08:43:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7fd434c99700 1 Processor -- start 2023-03-03T08:43:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7fd434c99700 1 -- start start 2023-03-03T08:43:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd434c99700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4300c1a40 0x7fd4300aa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd434c99700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4300c5140 0x7fd4300aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd434c99700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 0x7fd4301442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd434c99700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd430148240 con 0x7fd4300c5140 2023-03-03T08:43:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd434c99700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd430147f10 con 0x7fd4300c1a40 2023-03-03T08:43:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd434c99700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43006c990 con 0x7fd430151de0 2023-03-03T08:43:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd42ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 0x7fd4301442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd42ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 0x7fd4301442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59726/0 (socket says 172.21.15.7:59726) 2023-03-03T08:43:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd42ed9d700 1 -- 172.21.15.7:0/2250646095 learned_addr learned my addr 172.21.15.7:0/2250646095 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd42e59c700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4300c1a40 0x7fd4300aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd42dd9b700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4300c5140 0x7fd4300aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd42e59c700 1 -- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 msgr2=0x7fd4301442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7fd42e59c700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd430151de0 0x7fd4301442b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fd42e59c700 1 -- 172.21.15.7:0/2250646095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4300c5140 msgr2=0x7fd4300aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fd42e59c700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4300c5140 0x7fd4300aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fd42e59c700 1 -- 172.21.15.7:0/2250646095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd424009550 con 0x7fd4300c1a40 2023-03-03T08:43:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fd42e59c700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4300c1a40 0x7fd4300aa040 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd424016000 tx=0x7fd424004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.896+0000 7eff3f7fe700 1 -- 172.21.15.7:0/236559808 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7eff48021070 con 0x7eff580c5140 2023-03-03T08:43:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7eff3f7fe700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff440683c0 0x7eff4406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7eff3f7fe700 1 -- 172.21.15.7:0/236559808 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7eff48085de0 con 0x7eff580c5140 2023-03-03T08:43:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.897+0000 7eff567fc700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff440683c0 0x7eff4406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7eff567fc700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff440683c0 0x7eff4406a870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7eff5814b440 tx=0x7eff4c0090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7f8080ff9700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:6816/97029,v1:172.21.15.143:6818/97029] conn(0x7f80700732f0 0x7f8070075700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf3659c700 1 -- 172.21.15.7:0/4069966155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf380c6fe0 msgr2=0x7fdf380bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf3659c700 1 --2- 172.21.15.7:0/4069966155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf380c6fe0 0x7fdf380bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6b097fa700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 0x7f6af806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6b0bfff700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 0x7f6af806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6b097fa700 1 -- 172.21.15.7:0/2871576077 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6b00085b10 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6b097fa700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.7:6808/96731,v1:172.21.15.7:6809/96731] conn(0x7f6af80731a0 0x7f6af80755b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6b097fa700 1 -- 172.21.15.7:0/2871576077 --> [v2:172.21.15.7:6808/96731,v1:172.21.15.7:6809/96731] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6af8075c60 con 0x7f6af80731a0 2023-03-03T08:43:50.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6b097fa700 1 -- 172.21.15.7:0/2871576077 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b00085e80 con 0x7f6b0c0a93d0 2023-03-03T08:43:50.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7f8080ff9700 1 -- 172.21.15.7:0/2000664621 --> [v2:172.21.15.143:6816/97029,v1:172.21.15.143:6818/97029] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8070075db0 con 0x7f80700732f0 2023-03-03T08:43:50.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7f8080ff9700 1 -- 172.21.15.7:0/2000664621 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f807c0882f0 con 0x7f808408b970 2023-03-03T08:43:50.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf3659c700 1 -- 172.21.15.7:0/4069966155 >> v1:172.21.15.7:6789/0 conn(0x7fdf38155410 legacy=0x7fdf380bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6b11411700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.7:6808/96731,v1:172.21.15.7:6809/96731] conn(0x7f6af80731a0 0x7f6af80755b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.898+0000 7f6b0bfff700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 0x7f6af806a7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c072890 tx=0x7f6b04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7f6b11411700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.7:6808/96731,v1:172.21.15.7:6809/96731] conn(0x7f6af80731a0 0x7f6af80755b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.899+0000 7fdf3659c700 1 -- 172.21.15.7:0/4069966155 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf38154930 con 0x7fdf38151e80 2023-03-03T08:43:50.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7fb36d59a700 1 -- 172.21.15.7:0/1016352346 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364004230 con 0x7fb3700a8bd0 2023-03-03T08:43:50.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7fdf3659c700 1 --2- 172.21.15.7:0/4069966155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf38151e80 0x7fdf380c6aa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c009840 tx=0x7fdf2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3197602f4bfad266 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7f8083fff700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80700683c0 0x7f807006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7f63637d4700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.7:6816/96733,v1:172.21.15.7:6817/96733] conn(0x7f633c0015d0 0x7f633c003a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 --> [v2:172.21.15.7:6816/96733,v1:172.21.15.7:6817/96733] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f633c006bb0 con 0x7f633c0015d0 2023-03-03T08:43:50.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7f6360d6f700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.7:6816/96733,v1:172.21.15.7:6817/96733] conn(0x7f633c0015d0 0x7f633c003a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.901+0000 7f6360d6f700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.7:6816/96733,v1:172.21.15.7:6817/96733] conn(0x7f633c0015d0 0x7f633c003a80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7f6b097fa700 1 -- 172.21.15.7:0/2871576077 <== osd.3 v2:172.21.15.7:6808/96731 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6af8075c60 con 0x7f6af80731a0 2023-03-03T08:43:50.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7fb36d59a700 1 -- 172.21.15.7:0/1016352346 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb364004390 con 0x7fb3700a8bd0 2023-03-03T08:43:50.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7fb36d59a700 1 -- 172.21.15.7:0/1016352346 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3640036a0 con 0x7fb3700a8bd0 2023-03-03T08:43:50.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7fd41effd700 1 -- 172.21.15.7:0/2250646095 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4240036a0 con 0x7fd4300c1a40 2023-03-03T08:43:50.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7fd41effd700 1 -- 172.21.15.7:0/2250646095 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42401e070 con 0x7fd4300c1a40 2023-03-03T08:43:50.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7fd41effd700 1 -- 172.21.15.7:0/2250646095 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd424003a40 con 0x7fd4300c1a40 2023-03-03T08:43:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd430066de0 con 0x7fd4300c1a40 2023-03-03T08:43:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.900+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4300c4c20 con 0x7fd4300c1a40 2023-03-03T08:43:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.901+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd43006a9c0 con 0x7fd4300c1a40 2023-03-03T08:43:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.901+0000 7fb374a74700 1 -- 172.21.15.7:0/1016352346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3700a8bd0 msgr2=0x7fb3701593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.901+0000 7fb374a74700 1 --2- 172.21.15.7:0/1016352346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3700a8bd0 0x7fb3701593f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb3640097c0 tx=0x7fb364009ad0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.901+0000 7fb374a74700 1 -- 172.21.15.7:0/1016352346 shutdown_connections 2023-03-03T08:43:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.901+0000 7fb374a74700 1 --2- 172.21.15.7:0/1016352346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3700a8bd0 0x7fb3701593f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.901+0000 7fb374a74700 1 -- 172.21.15.7:0/1016352346 >> 172.21.15.7:0/1016352346 conn(0x7fb3700c62e0 msgr2=0x7fb3700c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7f808887a700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:6816/97029,v1:172.21.15.143:6818/97029] conn(0x7f80700732f0 0x7f8070075700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7eff5d2c1700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:6808/97019,v1:172.21.15.143:6809/97019] conn(0x7eff58148ea0 0x7eff5814c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 --> [v2:172.21.15.143:6808/97019,v1:172.21.15.143:6809/97019] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff5814d010 con 0x7eff58148ea0 2023-03-03T08:43:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7fdf3559a700 1 -- 172.21.15.7:0/4069966155 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c004230 con 0x7fdf38151e80 2023-03-03T08:43:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7fdf3559a700 1 -- 172.21.15.7:0/4069966155 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2c004390 con 0x7fdf38151e80 2023-03-03T08:43:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7fdf3559a700 1 -- 172.21.15.7:0/4069966155 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c0036a0 con 0x7fdf38151e80 2023-03-03T08:43:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7fb374a74700 1 -- 172.21.15.7:0/1016352346 shutdown_connections 2023-03-03T08:43:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7fb374a74700 1 -- 172.21.15.7:0/1016352346 wait complete. 2023-03-03T08:43:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7f8083fff700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80700683c0 0x7f807006a870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f80840c4550 tx=0x7f8078008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7fd41effd700 1 -- 172.21.15.7:0/2250646095 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd42400f050 con 0x7fd4300c1a40 2023-03-03T08:43:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.902+0000 7eff577fe700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:6808/97019,v1:172.21.15.143:6809/97019] conn(0x7eff58148ea0 0x7eff5814c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fd41effd700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd410068340 0x7fd41006a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fd41effd700 1 -- 172.21.15.7:0/2250646095 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fd424020070 con 0x7fd4300c1a40 2023-03-03T08:43:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7f808887a700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:6816/97029,v1:172.21.15.143:6818/97029] conn(0x7f80700732f0 0x7f8070075700 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fd42dd9b700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd410068340 0x7fd41006a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fd41effd700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:6800/97018,v1:172.21.15.143:6801/97018] conn(0x7fd410073270 0x7fd410075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/4069966155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf38151e80 msgr2=0x7fdf380c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fdf3ccc3700 1 --2- 172.21.15.7:0/4069966155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf38151e80 0x7fdf380c6aa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c009840 tx=0x7fdf2c009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fd41effd700 1 -- 172.21.15.7:0/2250646095 --> [v2:172.21.15.143:6800/97018,v1:172.21.15.143:6801/97018] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd410075d30 con 0x7fd410073270 2023-03-03T08:43:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fd41effd700 1 -- 172.21.15.7:0/2250646095 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd424087e60 con 0x7fd4300c1a40 2023-03-03T08:43:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7eff577fe700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:6808/97019,v1:172.21.15.143:6809/97019] conn(0x7eff58148ea0 0x7eff5814c8c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/4069966155 shutdown_connections 2023-03-03T08:43:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fd42ed9d700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:6800/97018,v1:172.21.15.143:6801/97018] conn(0x7fd410073270 0x7fd410075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fdf3ccc3700 1 --2- 172.21.15.7:0/4069966155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf380c6fe0 0x7fdf380bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fdf3ccc3700 1 --2- 172.21.15.7:0/4069966155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf38151e80 0x7fdf380c6aa0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/4069966155 >> 172.21.15.7:0/4069966155 conn(0x7fdf38071630 msgr2=0x7fdf3806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/4069966155 shutdown_connections 2023-03-03T08:43:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/4069966155 wait complete. 2023-03-03T08:43:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.903+0000 7fd42dd9b700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd410068340 0x7fd41006a7f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd43014f330 tx=0x7fd42800d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.904+0000 7fd42ed9d700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:6800/97018,v1:172.21.15.143:6801/97018] conn(0x7fd410073270 0x7fd410075680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.904+0000 7f8080ff9700 1 -- 172.21.15.7:0/2000664621 <== osd.4 v2:172.21.15.143:6816/97029 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8070075db0 con 0x7f80700732f0 2023-03-03T08:43:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.904+0000 7eff3f7fe700 1 -- 172.21.15.7:0/236559808 <== osd.7 v2:172.21.15.143:6808/97019 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7eff5814d010 con 0x7eff58148ea0 2023-03-03T08:43:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.905+0000 7fdf3ccc3700 1 Processor -- start 2023-03-03T08:43:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.905+0000 7fdf3ccc3700 1 -- start start 2023-03-03T08:43:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fdf3ccc3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf380c6fe0 0x7fdf3806f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fdf3ccc3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf38151e80 0x7fdf381f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fdf3ccc3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf38155410 0x7fdf38153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf3ccc3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3806f270 con 0x7fdf380c6fe0 2023-03-03T08:43:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf3ccc3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3806c4c0 con 0x7fdf38151e80 2023-03-03T08:43:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf3ccc3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3806fbd0 con 0x7fdf38155410 2023-03-03T08:43:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf3659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf380c6fe0 0x7fdf3806f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf3659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf380c6fe0 0x7fdf3806f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51688/0 (socket says 172.21.15.7:51688) 2023-03-03T08:43:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf3659c700 1 -- 172.21.15.7:0/1250733761 learned_addr learned my addr 172.21.15.7:0/1250733761 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf35d9b700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf38151e80 0x7fdf381f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf36d9d700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf38155410 0x7fdf38153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf35d9b700 1 -- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf38155410 msgr2=0x7fdf38153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf35d9b700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf38155410 0x7fdf38153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf35d9b700 1 -- 172.21.15.7:0/1250733761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf380c6fe0 msgr2=0x7fdf3806f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf35d9b700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf380c6fe0 0x7fdf3806f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf35d9b700 1 -- 172.21.15.7:0/1250733761 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c0094a0 con 0x7fdf38151e80 2023-03-03T08:43:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fdf35d9b700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf38151e80 0x7fdf381f1b70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdf380c4d60 tx=0x7fdf2800ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.908+0000 7fdf26ffd700 1 -- 172.21.15.7:0/1250733761 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28004c80 con 0x7fdf38151e80 2023-03-03T08:43:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.904+0000 7fb374a74700 1 Processor -- start 2023-03-03T08:43:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.904+0000 7fb374a74700 1 -- start start 2023-03-03T08:43:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb374a74700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb37006c0e0 0x7fb3700bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb374a74700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3700a8bd0 0x7fb37015bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb374a74700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3700c2680 0x7fb37006f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb374a74700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3700706c0 con 0x7fb3700a8bd0 2023-03-03T08:43:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb374a74700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37015cb60 con 0x7fb37006c0e0 2023-03-03T08:43:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb374a74700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3701eead0 con 0x7fb3700c2680 2023-03-03T08:43:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3700a8bd0 0x7fb37015bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb37006c0e0 0x7fb3700bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3700a8bd0 0x7fb37015bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51686/0 (socket says 172.21.15.7:51686) 2023-03-03T08:43:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36dd9b700 1 -- 172.21.15.7:0/3283348011 learned_addr learned my addr 172.21.15.7:0/3283348011 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36ed9d700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3700c2680 0x7fb37006f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36dd9b700 1 -- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3700c2680 msgr2=0x7fb37006f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36dd9b700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3700c2680 0x7fb37006f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36dd9b700 1 -- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb37006c0e0 msgr2=0x7fb3700bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36dd9b700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb37006c0e0 0x7fb3700bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36dd9b700 1 -- 172.21.15.7:0/3283348011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3640094a0 con 0x7fb3700a8bd0 2023-03-03T08:43:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.906+0000 7fb36dd9b700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3700a8bd0 0x7fb37015bcf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb37006d780 tx=0x7fb364004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fb35effd700 1 -- 172.21.15.7:0/3283348011 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364003e50 con 0x7fb3700a8bd0 2023-03-03T08:43:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3700aa310 con 0x7fb3700a8bd0 2023-03-03T08:43:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.907+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3700aa030 con 0x7fb3700a8bd0 2023-03-03T08:43:51.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.908+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf38154ef0 con 0x7fdf38151e80 2023-03-03T08:43:51.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.908+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf38154c10 con 0x7fdf38151e80 2023-03-03T08:43:51.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.908+0000 7fdf26ffd700 1 -- 172.21.15.7:0/1250733761 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf28004de0 con 0x7fdf38151e80 2023-03-03T08:43:51.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.908+0000 7fdf26ffd700 1 -- 172.21.15.7:0/1250733761 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28007310 con 0x7fdf38151e80 2023-03-03T08:43:51.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.908+0000 7fb35effd700 1 -- 172.21.15.7:0/3283348011 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb364011630 con 0x7fb3700a8bd0 2023-03-03T08:43:51.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.908+0000 7fb35effd700 1 -- 172.21.15.7:0/3283348011 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3640119a0 con 0x7fb3700a8bd0 2023-03-03T08:43:51.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.909+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdf1c000fb0 con 0x7fdf38151e80 2023-03-03T08:43:51.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.909+0000 7fdf26ffd700 1 -- 172.21.15.7:0/1250733761 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdf2801a020 con 0x7fdf38151e80 2023-03-03T08:43:51.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.910+0000 7fdf26ffd700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf18068310 0x7fdf1806a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.910+0000 7fdf26ffd700 1 -- 172.21.15.7:0/1250733761 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdf28085b00 con 0x7fdf38151e80 2023-03-03T08:43:51.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.910+0000 7fdf3659c700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf18068310 0x7fdf1806a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.910+0000 7fdf26ffd700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.7:6824/96732,v1:172.21.15.7:6825/96732] conn(0x7fdf18073240 0x7fdf18075650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.910+0000 7fdf26ffd700 1 -- 172.21.15.7:0/1250733761 --> [v2:172.21.15.7:6824/96732,v1:172.21.15.7:6825/96732] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf18075d00 con 0x7fdf18073240 2023-03-03T08:43:51.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.910+0000 7fdf26ffd700 1 -- 172.21.15.7:0/1250733761 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf2808a2a0 con 0x7fdf38151e80 2023-03-03T08:43:51.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.911+0000 7f6351ffb700 1 -- 172.21.15.7:0/17901598 <== osd.1 v2:172.21.15.7:6816/96733 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f633c006bb0 con 0x7f633c0015d0 2023-03-03T08:43:51.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.911+0000 7fdf36d9d700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.7:6824/96732,v1:172.21.15.7:6825/96732] conn(0x7fdf18073240 0x7fdf18075650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.911+0000 7fdf3659c700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf18068310 0x7fdf1806a7c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c004720 tx=0x7fdf2c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.904+0000 7fd41effd700 1 -- 172.21.15.7:0/2250646095 <== osd.6 v2:172.21.15.143:6800/97018 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd410075d30 con 0x7fd410073270 2023-03-03T08:43:51.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.911+0000 7fb35effd700 1 -- 172.21.15.7:0/3283348011 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb364017020 con 0x7fb3700a8bd0 2023-03-03T08:43:51.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.911+0000 7fb35effd700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3500682f0 0x7fb35006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.911+0000 7fb35effd700 1 -- 172.21.15.7:0/3283348011 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb364087c50 con 0x7fb3700a8bd0 2023-03-03T08:43:51.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.911+0000 7fb36e59c700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3500682f0 0x7fb35006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.912+0000 7fb36e59c700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3500682f0 0x7fb35006a7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb360005f30 tx=0x7fb360007d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.916+0000 7fb374a74700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:6817/97030,v1:172.21.15.143:6819/97030] conn(0x7fb3540015d0 0x7fb354003a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.916+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 --> [v2:172.21.15.143:6817/97030,v1:172.21.15.143:6819/97030] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb354006bb0 con 0x7fb3540015d0 2023-03-03T08:43:51.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.916+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 --> [v2:172.21.15.7:6800/96730,v1:172.21.15.7:6801/96730] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe68006c750 con 0x7fe68015bec0 2023-03-03T08:43:51.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.916+0000 7fdf36d9d700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.7:6824/96732,v1:172.21.15.7:6825/96732] conn(0x7fdf18073240 0x7fdf18075650 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.916+0000 7fb36ed9d700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:6817/97030,v1:172.21.15.143:6819/97030] conn(0x7fb3540015d0 0x7fb354003a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.917+0000 7fb36ed9d700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:6817/97030,v1:172.21.15.143:6819/97030] conn(0x7fb3540015d0 0x7fb354003a80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.917+0000 7fe675ffb700 1 -- 172.21.15.7:0/1490414197 <== osd.0 v2:172.21.15.7:6800/96730 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe68006c750 con 0x7fe68015bec0 2023-03-03T08:43:51.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.917+0000 7fdf26ffd700 1 -- 172.21.15.7:0/1250733761 <== osd.2 v2:172.21.15.7:6824/96732 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fdf18075d00 con 0x7fdf18073240 2023-03-03T08:43:51.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.917+0000 7fb35effd700 1 -- 172.21.15.7:0/3283348011 <== osd.5 v2:172.21.15.143:6817/97030 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb354006bb0 con 0x7fb3540015d0 2023-03-03T08:43:51.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.932+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 --> [v2:172.21.15.7:6808/96731,v1:172.21.15.7:6809/96731] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6b0c0702f0 con 0x7f6af80731a0 2023-03-03T08:43:51.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.933+0000 7f6b097fa700 1 -- 172.21.15.7:0/2871576077 <== osd.3 v2:172.21.15.7:6808/96731 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6b0c0702f0 con 0x7f6af80731a0 2023-03-03T08:43:51.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.934+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 >> [v2:172.21.15.7:6808/96731,v1:172.21.15.7:6809/96731] conn(0x7f6af80731a0 msgr2=0x7f6af80755b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.934+0000 7f6b12e74700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.7:6808/96731,v1:172.21.15.7:6809/96731] conn(0x7f6af80731a0 0x7f6af80755b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.934+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 --> [v2:172.21.15.143:6817/97030,v1:172.21.15.143:6819/97030] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb354005c90 con 0x7fb3540015d0 2023-03-03T08:43:51.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 msgr2=0x7f6af806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7f6b12e74700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 0x7f6af806a7a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c072890 tx=0x7f6b04009040 comp rx=0 tx=0).stop 2023-03-03T08:43:51.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b0c0a93d0 msgr2=0x7f6b0c153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7f6b12e74700 1 --2- 172.21.15.7:0/2871576077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b0c0a93d0 0x7f6b0c153310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000c4d0 tx=0x7f6b0000c7e0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7f6b11411700 1 -- 172.21.15.7:0/2871576077 reap_dead start 2023-03-03T08:43:51.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 shutdown_connections 2023-03-03T08:43:51.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 >> 172.21.15.7:0/2871576077 conn(0x7f6b0c0c62e0 msgr2=0x7f6b0c14c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 shutdown_connections 2023-03-03T08:43:51.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7f6b12e74700 1 -- 172.21.15.7:0/2871576077 wait complete. 2023-03-03T08:43:51.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 >> [v2:172.21.15.7:6800/96730,v1:172.21.15.7:6801/96730] conn(0x7fe68015bec0 msgr2=0x7fe68015c290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7fe687428700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.7:6800/96730,v1:172.21.15.7:6801/96730] conn(0x7fe68015bec0 0x7fe68015c290 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6640683c0 msgr2=0x7fe66406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7fe687428700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6640683c0 0x7fe66406a870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe670009c10 tx=0x7fe670008030 comp rx=0 tx=0).stop 2023-03-03T08:43:51.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800aba70 msgr2=0x7fe680154fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7fe687428700 1 --2- 172.21.15.7:0/1490414197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6800aba70 0x7fe680154fe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe67800cde0 tx=0x7fe67800a560 comp rx=0 tx=0).stop 2023-03-03T08:43:51.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7fe6859c5700 1 -- 172.21.15.7:0/1490414197 reap_dead start 2023-03-03T08:43:51.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 shutdown_connections 2023-03-03T08:43:51.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.935+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 >> 172.21.15.7:0/1490414197 conn(0x7fe6800c62e0 msgr2=0x7fe6800c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.936+0000 7fb35effd700 1 -- 172.21.15.7:0/3283348011 <== osd.5 v2:172.21.15.143:6817/97030 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb354005c90 con 0x7fb3540015d0 2023-03-03T08:43:51.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.936+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 --> [v2:172.21.15.143:6816/97029,v1:172.21.15.143:6818/97029] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8084074bb0 con 0x7f80700732f0 2023-03-03T08:43:51.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.936+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 shutdown_connections 2023-03-03T08:43:51.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.936+0000 7fe687428700 1 -- 172.21.15.7:0/1490414197 wait complete. 2023-03-03T08:43:51.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.936+0000 7f8080ff9700 1 -- 172.21.15.7:0/2000664621 <== osd.4 v2:172.21.15.143:6816/97029 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8084074bb0 con 0x7f80700732f0 2023-03-03T08:43:51.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.938+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:6817/97030,v1:172.21.15.143:6819/97030] conn(0x7fb3540015d0 msgr2=0x7fb354003a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.938+0000 7fb374a74700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:6817/97030,v1:172.21.15.143:6819/97030] conn(0x7fb3540015d0 0x7fb354003a80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.938+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3500682f0 msgr2=0x7fb35006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.938+0000 7fb374a74700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3500682f0 0x7fb35006a7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb360005f30 tx=0x7fb360007d50 comp rx=0 tx=0).stop 2023-03-03T08:43:51.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.938+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3700a8bd0 msgr2=0x7fb37015bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.938+0000 7fb374a74700 1 --2- 172.21.15.7:0/3283348011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3700a8bd0 0x7fb37015bcf0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb37006d780 tx=0x7fb364004750 comp rx=0 tx=0).stop 2023-03-03T08:43:51.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.938+0000 7fb36ed9d700 1 -- 172.21.15.7:0/3283348011 reap_dead start 2023-03-03T08:43:51.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.938+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 shutdown_connections 2023-03-03T08:43:51.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.938+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 >> 172.21.15.7:0/3283348011 conn(0x7fb3700c62e0 msgr2=0x7fb3700ab9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.940+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:6816/97029,v1:172.21.15.143:6818/97029] conn(0x7f80700732f0 msgr2=0x7f8070075700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.940+0000 7f808a2dd700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:6816/97029,v1:172.21.15.143:6818/97029] conn(0x7f80700732f0 0x7f8070075700 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.940+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 --> [v2:172.21.15.7:6816/96733,v1:172.21.15.7:6817/96733] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f633c005c30 con 0x7f633c0015d0 2023-03-03T08:43:51.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.941+0000 7f6351ffb700 1 -- 172.21.15.7:0/17901598 <== osd.1 v2:172.21.15.7:6816/96733 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f633c005c30 con 0x7f633c0015d0 2023-03-03T08:43:51.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.941+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 >> [v2:172.21.15.7:6816/96733,v1:172.21.15.7:6817/96733] conn(0x7f633c0015d0 msgr2=0x7f633c003a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.941+0000 7f63637d4700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.7:6816/96733,v1:172.21.15.7:6817/96733] conn(0x7f633c0015d0 0x7f633c003a80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.941+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f63400683c0 msgr2=0x7f634006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.941+0000 7f63637d4700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f63400683c0 0x7f634006a870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f634c0011e0 tx=0x7f634c009040 comp rx=0 tx=0).stop 2023-03-03T08:43:51.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.941+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 msgr2=0x7f635c14a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.941+0000 7f63637d4700 1 --2- 172.21.15.7:0/17901598 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f635c152ed0 0x7f635c14a3a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f635800b410 tx=0x7f635800b720 comp rx=0 tx=0).stop 2023-03-03T08:43:51.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.942+0000 7f6361d71700 1 -- 172.21.15.7:0/17901598 reap_dead start 2023-03-03T08:43:51.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.942+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 shutdown_connections 2023-03-03T08:43:51.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.942+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 >> 172.21.15.7:0/17901598 conn(0x7f635c0c62e0 msgr2=0x7f635c14c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.942+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 shutdown_connections 2023-03-03T08:43:51.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.942+0000 7f63637d4700 1 -- 172.21.15.7:0/17901598 wait complete. 2023-03-03T08:43:51.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.944+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80700683c0 msgr2=0x7f807006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.944+0000 7f808a2dd700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80700683c0 0x7f807006a870 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f80840c4550 tx=0x7f8078008040 comp rx=0 tx=0).stop 2023-03-03T08:43:51.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.944+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f808408b970 msgr2=0x7f808420bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.944+0000 7f808a2dd700 1 --2- 172.21.15.7:0/2000664621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f808408b970 0x7f808420bd80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f807c00bd40 tx=0x7f807c00bd70 comp rx=0 tx=0).stop 2023-03-03T08:43:51.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.944+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 shutdown_connections 2023-03-03T08:43:51.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.944+0000 7fb374a74700 1 -- 172.21.15.7:0/3283348011 wait complete. 2023-03-03T08:43:51.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.944+0000 7f808887a700 1 -- 172.21.15.7:0/2000664621 reap_dead start 2023-03-03T08:43:51.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.945+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 shutdown_connections 2023-03-03T08:43:51.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.945+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 >> 172.21.15.7:0/2000664621 conn(0x7f80840c62e0 msgr2=0x7f80841ec960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.948+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 shutdown_connections 2023-03-03T08:43:51.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.948+0000 7f808a2dd700 1 -- 172.21.15.7:0/2000664621 wait complete. 2023-03-03T08:43:51.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.950+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 --> [v2:172.21.15.143:6800/97018,v1:172.21.15.143:6801/97018] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd430074bb0 con 0x7fd410073270 2023-03-03T08:43:51.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.951+0000 7fd41effd700 1 -- 172.21.15.7:0/2250646095 <== osd.6 v2:172.21.15.143:6800/97018 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd430074bb0 con 0x7fd410073270 2023-03-03T08:43:51.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.951+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 --> [v2:172.21.15.7:6824/96732,v1:172.21.15.7:6825/96732] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdf1c002d30 con 0x7fdf18073240 2023-03-03T08:43:51.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.951+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:6800/97018,v1:172.21.15.143:6801/97018] conn(0x7fd410073270 msgr2=0x7fd410075680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.951+0000 7fd434c99700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:6800/97018,v1:172.21.15.143:6801/97018] conn(0x7fd410073270 0x7fd410075680 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.951+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd410068340 msgr2=0x7fd41006a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.951+0000 7fd434c99700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd410068340 0x7fd41006a7f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd43014f330 tx=0x7fd42800d040 comp rx=0 tx=0).stop 2023-03-03T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.951+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4300c1a40 msgr2=0x7fd4300aa040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.951+0000 7fd434c99700 1 --2- 172.21.15.7:0/2250646095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4300c1a40 0x7fd4300aa040 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd424016000 tx=0x7fd424004750 comp rx=0 tx=0).stop 2023-03-03T08:43:51.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.951+0000 7fd42ed9d700 1 -- 172.21.15.7:0/2250646095 reap_dead start 2023-03-03T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.952+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 shutdown_connections 2023-03-03T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.952+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 >> 172.21.15.7:0/2250646095 conn(0x7fd4300a9430 msgr2=0x7fd43014b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.952+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 shutdown_connections 2023-03-03T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.952+0000 7fd434c99700 1 -- 172.21.15.7:0/2250646095 wait complete. 2023-03-03T08:43:51.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.953+0000 7fdf26ffd700 1 -- 172.21.15.7:0/1250733761 <== osd.2 v2:172.21.15.7:6824/96732 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdf1c002d30 con 0x7fdf18073240 2023-03-03T08:43:51.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.955+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 --> [v2:172.21.15.143:6808/97019,v1:172.21.15.143:6809/97019] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7eff5814ce00 con 0x7eff58148ea0 2023-03-03T08:43:51.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.956+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 >> [v2:172.21.15.7:6824/96732,v1:172.21.15.7:6825/96732] conn(0x7fdf18073240 msgr2=0x7fdf18075650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.956+0000 7fdf3ccc3700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.7:6824/96732,v1:172.21.15.7:6825/96732] conn(0x7fdf18073240 0x7fdf18075650 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.956+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf18068310 msgr2=0x7fdf1806a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.956+0000 7fdf3ccc3700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf18068310 0x7fdf1806a7c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c004720 tx=0x7fdf2c008030 comp rx=0 tx=0).stop 2023-03-03T08:43:51.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.956+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf38151e80 msgr2=0x7fdf381f1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.956+0000 7fdf3ccc3700 1 --2- 172.21.15.7:0/1250733761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf38151e80 0x7fdf381f1b70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdf380c4d60 tx=0x7fdf2800ca30 comp rx=0 tx=0).stop 2023-03-03T08:43:51.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.956+0000 7fdf36d9d700 1 -- 172.21.15.7:0/1250733761 reap_dead start 2023-03-03T08:43:51.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.956+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 shutdown_connections 2023-03-03T08:43:51.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.956+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 >> 172.21.15.7:0/1250733761 conn(0x7fdf38071630 msgr2=0x7fdf380ca910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.956+0000 7eff3f7fe700 1 -- 172.21.15.7:0/236559808 <== osd.7 v2:172.21.15.143:6808/97019 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7eff5814ce00 con 0x7eff58148ea0 2023-03-03T08:43:51.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:6808/97019,v1:172.21.15.143:6809/97019] conn(0x7eff58148ea0 msgr2=0x7eff5814c8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff5d2c1700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:6808/97019,v1:172.21.15.143:6809/97019] conn(0x7eff58148ea0 0x7eff5814c8c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff440683c0 msgr2=0x7eff4406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff5d2c1700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff440683c0 0x7eff4406a870 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7eff5814b440 tx=0x7eff4c0090e0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff580c5140 msgr2=0x7eff58148960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff5d2c1700 1 --2- 172.21.15.7:0/236559808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff580c5140 0x7eff58148960 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff4800b450 tx=0x7eff4800b760 comp rx=0 tx=0).stop 2023-03-03T08:43:51.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff577fe700 1 -- 172.21.15.7:0/236559808 reap_dead start 2023-03-03T08:43:51.037 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-03-03T08:43:51.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-03-03T08:43:51.038 INFO:teuthology.orchestra.run.smithi007.stdout:55834574859 2023-03-03T08:43:51.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-03-03T08:43:51.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 shutdown_connections 2023-03-03T08:43:51.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 >> 172.21.15.7:0/236559808 conn(0x7eff580a9430 msgr2=0x7eff58151110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 shutdown_connections 2023-03-03T08:43:51.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.957+0000 7eff5d2c1700 1 -- 172.21.15.7:0/236559808 wait complete. 2023-03-03T08:43:51.040 INFO:teuthology.orchestra.run.smithi007.stdout:55834574858 2023-03-03T08:43:51.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-03-03T08:43:51.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.964+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 shutdown_connections 2023-03-03T08:43:51.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:50.964+0000 7fdf3ccc3700 1 -- 172.21.15.7:0/1250733761 wait complete. 2023-03-03T08:43:51.043 INFO:teuthology.orchestra.run.smithi007.stdout:55834574857 2023-03-03T08:43:51.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-03-03T08:43:51.044 INFO:teuthology.orchestra.run.smithi007.stdout:55834574856 2023-03-03T08:43:51.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-03-03T08:43:51.045 INFO:teuthology.orchestra.run.smithi007.stdout:55834574857 2023-03-03T08:43:51.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-03-03T08:43:51.046 INFO:teuthology.orchestra.run.smithi007.stdout:55834574857 2023-03-03T08:43:51.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-03-03T08:43:51.048 INFO:teuthology.orchestra.run.smithi007.stdout:55834574857 2023-03-03T08:43:51.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-03-03T08:43:51.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.292+0000 7f0900720700 1 Processor -- start 2023-03-03T08:43:51.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.292+0000 7f0900720700 1 -- start start 2023-03-03T08:43:51.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.293+0000 7f0900720700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f8151b60 0x7f08f80bf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.293+0000 7f0900720700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f08f80aaee0 0x7f08f815ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.293+0000 7f0900720700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f80c1630 con 0x7f08f80c1ac0 2023-03-03T08:43:51.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.293+0000 7f0900720700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f80a96e0 con 0x7f08f80aaee0 2023-03-03T08:43:51.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.293+0000 7f0900720700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f8154400 con 0x7f08f8151b60 2023-03-03T08:43:51.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.294+0000 7f077ac4e700 1 Processor -- start 2023-03-03T08:43:51.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.294+0000 7f08fecbd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f08f80aaee0 0x7f08f815ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.294+0000 7f08fecbd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f08f80aaee0 0x7f08f815ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47478/0 (socket says 172.21.15.7:47478) 2023-03-03T08:43:51.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.294+0000 7f08fecbd700 1 -- 172.21.15.7:0/259550542 learned_addr learned my addr 172.21.15.7:0/259550542 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.294+0000 7f08fecbd700 1 -- 172.21.15.7:0/259550542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f8151b60 msgr2=0x7f08f80bf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:51.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.294+0000 7f08fecbd700 1 --2- 172.21.15.7:0/259550542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f8151b60 0x7f08f80bf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.294+0000 7f08fecbd700 1 -- 172.21.15.7:0/259550542 >> v1:172.21.15.7:6789/0 conn(0x7f08f80c1ac0 legacy=0x7f08f80aa7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:51.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.294+0000 7f08fecbd700 1 -- 172.21.15.7:0/259550542 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f81546c0 con 0x7f08f80aaee0 2023-03-03T08:43:51.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.294+0000 7f08fecbd700 1 --2- 172.21.15.7:0/259550542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80aaee0 0x7f08f815ed80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f08f40098f0 tx=0x7f08f4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99443d5c39227aff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.296+0000 7f08fd4ba700 1 -- 172.21.15.7:0/259550542 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4004230 con 0x7f08f80aaee0 2023-03-03T08:43:51.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.296+0000 7f08fd4ba700 1 -- 172.21.15.7:0/259550542 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f4004390 con 0x7f08f80aaee0 2023-03-03T08:43:51.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.296+0000 7f08fd4ba700 1 -- 172.21.15.7:0/259550542 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f40036a0 con 0x7f08f80aaee0 2023-03-03T08:43:51.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.294+0000 7f077ac4e700 1 -- start start 2023-03-03T08:43:51.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.295+0000 7f077ac4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0774156060 0x7f077415bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.295+0000 7f077ac4e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07740c0e60 con 0x7f07740ab8f0 2023-03-03T08:43:51.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.295+0000 7f077ac4e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07740c1140 con 0x7f07740c5cb0 2023-03-03T08:43:51.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.295+0000 7f077ac4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07740abe60 con 0x7f0774156060 2023-03-03T08:43:51.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.296+0000 7f0773fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f07740ab8f0 0x7f07741590f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45690/0 (socket says 172.21.15.7:45690) 2023-03-03T08:43:51.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.296+0000 7f0773fff700 1 -- 172.21.15.7:0/3539106078 learned_addr learned my addr 172.21.15.7:0/3539106078 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:51.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.296+0000 7f07789ea700 1 --2- 172.21.15.7:0/3539106078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0774156060 0x7f077415bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.296+0000 7f07789ea700 1 -- 172.21.15.7:0/3539106078 >> v1:172.21.15.143:6789/0 conn(0x7f07740c5cb0 legacy=0x7f077406a2b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T08:43:51.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.296+0000 7f07789ea700 1 -- 172.21.15.7:0/3539106078 >> v1:172.21.15.7:6789/0 conn(0x7f07740ab8f0 legacy=0x7f07741590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.297+0000 7f07789ea700 1 -- 172.21.15.7:0/3539106078 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07741588f0 con 0x7f0774156060 2023-03-03T08:43:51.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.297+0000 7f0900720700 1 -- 172.21.15.7:0/259550542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80aaee0 msgr2=0x7f08f815ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.297+0000 7f0900720700 1 --2- 172.21.15.7:0/259550542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80aaee0 0x7f08f815ed80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f08f40098f0 tx=0x7f08f4009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:51.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.299+0000 7f07789ea700 1 --2- 172.21.15.7:0/3539106078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0774156060 0x7f077415bdb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f075c009870 tx=0x7f075c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9ed73c140e34cef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.299+0000 7f0900720700 1 -- 172.21.15.7:0/259550542 shutdown_connections 2023-03-03T08:43:51.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.299+0000 7f0900720700 1 --2- 172.21.15.7:0/259550542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f80aaee0 0x7f08f815ed80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.299+0000 7f0900720700 1 --2- 172.21.15.7:0/259550542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f8151b60 0x7f08f80bf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.299+0000 7f0900720700 1 -- 172.21.15.7:0/259550542 >> 172.21.15.7:0/259550542 conn(0x7f08f80c62e0 msgr2=0x7f08f80c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.299+0000 7f0900720700 1 -- 172.21.15.7:0/259550542 shutdown_connections 2023-03-03T08:43:51.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.299+0000 7f0900720700 1 -- 172.21.15.7:0/259550542 wait complete. 2023-03-03T08:43:51.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.301+0000 7f0900720700 1 Processor -- start 2023-03-03T08:43:51.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.301+0000 7f0900720700 1 -- start start 2023-03-03T08:43:51.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.302+0000 7f0900720700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80aaee0 0x7f08f8153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.302+0000 7f0900720700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f80c1ac0 0x7f08f80bec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.302+0000 7f0900720700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f8151b60 0x7f08f81e7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.302+0000 7f0900720700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f81ee400 con 0x7f08f80c1ac0 2023-03-03T08:43:51.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.302+0000 7f0900720700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f81ee6e0 con 0x7f08f8151b60 2023-03-03T08:43:51.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.302+0000 7f0900720700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f81ee9c0 con 0x7f08f80aaee0 2023-03-03T08:43:51.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.302+0000 7f08fdcbb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f80c1ac0 0x7f08f80bec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.302+0000 7f08fdcbb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f80c1ac0 0x7f08f80bec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51716/0 (socket says 172.21.15.7:51716) 2023-03-03T08:43:51.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.302+0000 7f08fdcbb700 1 -- 172.21.15.7:0/291582954 learned_addr learned my addr 172.21.15.7:0/291582954 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.302+0000 7f08fecbd700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f8151b60 0x7f08f81e7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08fe4bc700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80aaee0 0x7f08f8153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f07737fe700 1 -- 172.21.15.7:0/3539106078 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c004230 con 0x7f0774156060 2023-03-03T08:43:51.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f07737fe700 1 -- 172.21.15.7:0/3539106078 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c004390 con 0x7f0774156060 2023-03-03T08:43:51.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f07737fe700 1 -- 172.21.15.7:0/3539106078 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c0036a0 con 0x7f0774156060 2023-03-03T08:43:51.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.304+0000 7f7d26c49700 1 Processor -- start 2023-03-03T08:43:51.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.304+0000 7f077ac4e700 1 -- 172.21.15.7:0/3539106078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0774156060 msgr2=0x7f077415bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.304+0000 7f077ac4e700 1 --2- 172.21.15.7:0/3539106078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0774156060 0x7f077415bdb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f075c009870 tx=0x7f075c009b80 comp rx=0 tx=0).stop 2023-03-03T08:43:51.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.304+0000 7f077ac4e700 1 -- 172.21.15.7:0/3539106078 shutdown_connections 2023-03-03T08:43:51.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.304+0000 7f077ac4e700 1 --2- 172.21.15.7:0/3539106078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0774156060 0x7f077415bdb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.304+0000 7f077ac4e700 1 -- 172.21.15.7:0/3539106078 >> 172.21.15.7:0/3539106078 conn(0x7f07740a9430 msgr2=0x7f07740c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08fdcbb700 1 -- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80aaee0 msgr2=0x7f08f8153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08fdcbb700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80aaee0 0x7f08f8153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08fdcbb700 1 -- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f8151b60 msgr2=0x7f08f81e7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08fdcbb700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f8151b60 0x7f08f81e7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08fdcbb700 1 -- 172.21.15.7:0/291582954 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f4009550 con 0x7f08f80c1ac0 2023-03-03T08:43:51.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08fecbd700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f8151b60 0x7f08f81e7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:51.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08fdcbb700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f80c1ac0 0x7f08f80bec40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f08e800c4a0 tx=0x7f08e800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08eeffd700 1 -- 172.21.15.7:0/291582954 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e80054e0 con 0x7f08f80c1ac0 2023-03-03T08:43:51.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08eeffd700 1 -- 172.21.15.7:0/291582954 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e8003680 con 0x7f08f80c1ac0 2023-03-03T08:43:51.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f0900720700 1 -- 172.21.15.7:0/291582954 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f80c0fe0 con 0x7f08f80c1ac0 2023-03-03T08:43:51.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f08eeffd700 1 -- 172.21.15.7:0/291582954 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e8003a30 con 0x7f08f80c1ac0 2023-03-03T08:43:51.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.303+0000 7f0900720700 1 -- 172.21.15.7:0/291582954 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f80c0d00 con 0x7f08f80c1ac0 2023-03-03T08:43:51.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.305+0000 7f91ad1f4700 1 Processor -- start 2023-03-03T08:43:51.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.306+0000 7f077ac4e700 1 -- 172.21.15.7:0/3539106078 shutdown_connections 2023-03-03T08:43:51.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.305+0000 7f08eeffd700 1 -- 172.21.15.7:0/291582954 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f08e8019030 con 0x7f08f80c1ac0 2023-03-03T08:43:51.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.306+0000 7f08eeffd700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08dc0682f0 0x7f08dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.306+0000 7f08eeffd700 1 -- 172.21.15.7:0/291582954 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f08e8084fd0 con 0x7f08f80c1ac0 2023-03-03T08:43:51.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.306+0000 7f08fe4bc700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08dc0682f0 0x7f08dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f077ac4e700 1 -- 172.21.15.7:0/3539106078 wait complete. 2023-03-03T08:43:51.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f0900720700 1 -- 172.21.15.7:0/291582954 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f80c12c0 con 0x7f08f80c1ac0 2023-03-03T08:43:51.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.305+0000 7f7d26c49700 1 -- start start 2023-03-03T08:43:51.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f7d26c49700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d200c2680 0x7f7d201593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f7d26c49700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7d2006c0e0 0x7f7d2006c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f7d26c49700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d200a9de0 con 0x7f7d200a8bd0 2023-03-03T08:43:51.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f7d26c49700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d200aa5f0 con 0x7f7d2006c0e0 2023-03-03T08:43:51.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f7d26c49700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2006b9b0 con 0x7f7d200c2680 2023-03-03T08:43:51.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f7d1ffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f7d200a8bd0 0x7f7d200aba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49224/0 (socket says 172.21.15.7:49224) 2023-03-03T08:43:51.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f7d1ffff700 1 -- 172.21.15.7:0/278018187 learned_addr learned my addr 172.21.15.7:0/278018187 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:51.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f7d249e5700 1 --2- 172.21.15.7:0/278018187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d200c2680 0x7f7d201593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f08fe4bc700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08dc0682f0 0x7f08dc06a7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f08f8153fd0 tx=0x7f08f000d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f7d1f7fe700 1 -- 172.21.15.7:0/278018187 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400907495 0 0) 0x7f7d200a9de0 con 0x7f7d200a8bd0 2023-03-03T08:43:51.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f7d1f7fe700 1 -- 172.21.15.7:0/278018187 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d100035b0 con 0x7f7d200a8bd0 2023-03-03T08:43:51.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.305+0000 7f91ad1f4700 1 -- start start 2023-03-03T08:43:51.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.306+0000 7f91ad1f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a81510f0 0x7f91a80c1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.306+0000 7f91ad1f4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f91a8151ec0 0x7f91a8152310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f91ad1f4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a806d460 con 0x7f91a80c35a0 2023-03-03T08:43:51.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f91ad1f4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a80c1c70 con 0x7f91a8151ec0 2023-03-03T08:43:51.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f91ad1f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a814e4e0 con 0x7f91a81510f0 2023-03-03T08:43:51.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f91a6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a81510f0 0x7f91a80c1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f91a6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a81510f0 0x7f91a80c1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59792/0 (socket says 172.21.15.7:59792) 2023-03-03T08:43:51.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.307+0000 7f91a6d9d700 1 -- 172.21.15.7:0/4175618390 learned_addr learned my addr 172.21.15.7:0/4175618390 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f91a6d9d700 1 -- 172.21.15.7:0/4175618390 >> v2:172.21.15.143:3300/0 conn(0x7f91a8151ec0 msgr2=0x7f91a8152310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T08:43:51.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f91a6d9d700 1 --2- 172.21.15.7:0/4175618390 >> v2:172.21.15.143:3300/0 conn(0x7f91a8151ec0 0x7f91a8152310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f91a6d9d700 1 -- 172.21.15.7:0/4175618390 >> v1:172.21.15.7:6789/0 conn(0x7f91a80c35a0 legacy=0x7f91a80c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:51.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f91a6d9d700 1 -- 172.21.15.7:0/4175618390 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a80c2d60 con 0x7f91a81510f0 2023-03-03T08:43:51.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f7d249e5700 1 -- 172.21.15.7:0/278018187 >> v2:172.21.15.7:3300/0 conn(0x7f7d2006c0e0 msgr2=0x7f7d2006c560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T08:43:51.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f7d249e5700 1 --2- 172.21.15.7:0/278018187 >> v2:172.21.15.7:3300/0 conn(0x7f7d2006c0e0 0x7f7d2006c560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f7d249e5700 1 -- 172.21.15.7:0/278018187 >> v1:172.21.15.143:6789/0 conn(0x7f7d200a8bd0 legacy=0x7f7d200aba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f7d249e5700 1 -- 172.21.15.7:0/278018187 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d200c2a50 con 0x7f7d200c2680 2023-03-03T08:43:51.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.306+0000 7f01c0d40700 1 Processor -- start 2023-03-03T08:43:51.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.306+0000 7f01c0d40700 1 -- start start 2023-03-03T08:43:51.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01c0d40700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc161cd0 0x7f01bc08b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01c0d40700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01bc0c2810 con 0x7f01bc08b970 2023-03-03T08:43:51.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01c0d40700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01bc0aa420 con 0x7f01bc0c2440 2023-03-03T08:43:51.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01c0d40700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc06bac0 con 0x7f01bc161cd0 2023-03-03T08:43:51.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01ba59c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f01bc0c2440 0x7f01bc0ab9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49228/0 (socket says 172.21.15.7:49228) 2023-03-03T08:43:51.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01ba59c700 1 -- 172.21.15.7:0/1115012880 learned_addr learned my addr 172.21.15.7:0/1115012880 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:51.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f91a6d9d700 1 --2- 172.21.15.7:0/4175618390 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a81510f0 0x7f91a80c1510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f919c0098f0 tx=0x7f919c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fdf33c20e02f713 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01b9d9b700 1 --2- 172.21.15.7:0/1115012880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc161cd0 0x7f01bc08b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f077ac4e700 1 Processor -- start 2023-03-03T08:43:51.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f7d249e5700 1 --2- 172.21.15.7:0/278018187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d200c2680 0x7f7d201593f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c009840 tx=0x7f7d0c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f022933c1b5a0bcc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.309+0000 7f7d1f7fe700 1 -- 172.21.15.7:0/278018187 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c011830 con 0x7f7d200c2680 2023-03-03T08:43:51.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.309+0000 7f7d1f7fe700 1 -- 172.21.15.7:0/278018187 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0c011990 con 0x7f7d200c2680 2023-03-03T08:43:51.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.309+0000 7f7d1f7fe700 1 -- 172.21.15.7:0/278018187 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c011d20 con 0x7f7d200c2680 2023-03-03T08:43:51.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f077ac4e700 1 -- start start 2023-03-03T08:43:51.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f077ac4e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07740ab8f0 0x7f0774157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f077ac4e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07740c5cb0 0x7f077406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f077ac4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07740724d0 0x7f077406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f077ac4e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741e9980 con 0x7f07740ab8f0 2023-03-03T08:43:51.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f077ac4e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741e9eb0 con 0x7f07740c5cb0 2023-03-03T08:43:51.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f077ac4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077406c6e0 con 0x7f07740724d0 2023-03-03T08:43:51.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f07789ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07740ab8f0 0x7f0774157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01b9d9b700 1 -- 172.21.15.7:0/1115012880 >> v1:172.21.15.143:6789/0 conn(0x7f01bc0c2440 legacy=0x7f01bc0ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01b9d9b700 1 -- 172.21.15.7:0/1115012880 >> v1:172.21.15.7:6789/0 conn(0x7f01bc08b970 legacy=0x7f01bc06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:51.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01b959a700 1 -- 172.21.15.7:0/1115012880 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633649127 0 0) 0x7f01bc0aa420 con 0x7f01bc0c2440 2023-03-03T08:43:51.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.308+0000 7f01b9d9b700 1 -- 172.21.15.7:0/1115012880 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc162160 con 0x7f01bc161cd0 2023-03-03T08:43:51.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.309+0000 7f01b9d9b700 1 --2- 172.21.15.7:0/1115012880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc161cd0 0x7f01bc08b370 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01ac008960 tx=0x7f01ac00d330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a94635c15bd04942 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.309+0000 7f01b959a700 1 -- 172.21.15.7:0/1115012880 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac00d980 con 0x7f01bc161cd0 2023-03-03T08:43:51.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.309+0000 7f01b959a700 1 -- 172.21.15.7:0/1115012880 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01ac00dae0 con 0x7f01bc161cd0 2023-03-03T08:43:51.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.309+0000 7f01b959a700 1 -- 172.21.15.7:0/1115012880 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac00de90 con 0x7f01bc161cd0 2023-03-03T08:43:51.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f7d26c49700 1 -- 172.21.15.7:0/278018187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d200c2680 msgr2=0x7f7d201593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f7d26c49700 1 --2- 172.21.15.7:0/278018187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d200c2680 0x7f7d201593f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c009840 tx=0x7f7d0c009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:51.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.309+0000 7f91a5d9b700 1 -- 172.21.15.7:0/4175618390 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c004230 con 0x7f91a81510f0 2023-03-03T08:43:51.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.309+0000 7f91a5d9b700 1 -- 172.21.15.7:0/4175618390 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919c004390 con 0x7f91a81510f0 2023-03-03T08:43:51.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.309+0000 7f91a5d9b700 1 -- 172.21.15.7:0/4175618390 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c0036a0 con 0x7f91a81510f0 2023-03-03T08:43:51.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f07789ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07740ab8f0 0x7f0774157f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51734/0 (socket says 172.21.15.7:51734) 2023-03-03T08:43:51.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f07789ea700 1 -- 172.21.15.7:0/2629029958 learned_addr learned my addr 172.21.15.7:0/2629029958 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f0773fff700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07740c5cb0 0x7f077406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f07789ea700 1 -- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07740724d0 msgr2=0x7f077406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f07789ea700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07740724d0 0x7f077406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f07789ea700 1 -- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07740c5cb0 msgr2=0x7f077406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f07789ea700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07740c5cb0 0x7f077406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f07789ea700 1 -- 172.21.15.7:0/2629029958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c0094a0 con 0x7f07740ab8f0 2023-03-03T08:43:51.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f01c0d40700 1 -- 172.21.15.7:0/1115012880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc161cd0 msgr2=0x7f01bc08b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f01c0d40700 1 --2- 172.21.15.7:0/1115012880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc161cd0 0x7f01bc08b370 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01ac008960 tx=0x7f01ac00d330 comp rx=0 tx=0).stop 2023-03-03T08:43:51.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f01c0d40700 1 -- 172.21.15.7:0/1115012880 shutdown_connections 2023-03-03T08:43:51.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f01c0d40700 1 --2- 172.21.15.7:0/1115012880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc161cd0 0x7f01bc08b370 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f01c0d40700 1 -- 172.21.15.7:0/1115012880 >> 172.21.15.7:0/1115012880 conn(0x7f01bc0c62e0 msgr2=0x7f01bc0c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f01c0d40700 1 -- 172.21.15.7:0/1115012880 shutdown_connections 2023-03-03T08:43:51.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.310+0000 7f07789ea700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07740ab8f0 0x7f0774157f00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f077406b120 tx=0x7f075c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.311+0000 7f01c0d40700 1 -- 172.21.15.7:0/1115012880 wait complete. 2023-03-03T08:43:51.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.311+0000 7f91ad1f4700 1 -- 172.21.15.7:0/4175618390 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a81510f0 msgr2=0x7f91a80c1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.311+0000 7f91ad1f4700 1 --2- 172.21.15.7:0/4175618390 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a81510f0 0x7f91a80c1510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f919c0098f0 tx=0x7f919c009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:51.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f91ad1f4700 1 -- 172.21.15.7:0/4175618390 shutdown_connections 2023-03-03T08:43:51.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f91ad1f4700 1 --2- 172.21.15.7:0/4175618390 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a81510f0 0x7f91a80c1510 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f91ad1f4700 1 --2- 172.21.15.7:0/4175618390 >> v2:172.21.15.143:3300/0 conn(0x7f91a8151ec0 0x7f91a8152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f91ad1f4700 1 -- 172.21.15.7:0/4175618390 >> 172.21.15.7:0/4175618390 conn(0x7f91a806a2b0 msgr2=0x7f91a806a680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f7d26c49700 1 -- 172.21.15.7:0/278018187 shutdown_connections 2023-03-03T08:43:51.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f7d26c49700 1 --2- 172.21.15.7:0/278018187 >> v2:172.21.15.7:3300/0 conn(0x7f7d2006c0e0 0x7f7d2006c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f7d26c49700 1 --2- 172.21.15.7:0/278018187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d200c2680 0x7f7d201593f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f7d26c49700 1 -- 172.21.15.7:0/278018187 >> 172.21.15.7:0/278018187 conn(0x7f7d200c62e0 msgr2=0x7f7d200c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f01c0d40700 1 Processor -- start 2023-03-03T08:43:51.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f91ad1f4700 1 -- 172.21.15.7:0/4175618390 shutdown_connections 2023-03-03T08:43:51.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.313+0000 7f91ad1f4700 1 -- 172.21.15.7:0/4175618390 wait complete. 2023-03-03T08:43:51.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.312+0000 7f01c0d40700 1 -- start start 2023-03-03T08:43:51.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.313+0000 7f01c0d40700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc08b970 0x7f01bc15f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.313+0000 7f01c0d40700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc0c2440 0x7f01bc159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.313+0000 7f01c0d40700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01bc161cd0 0x7f01bc08a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.313+0000 7f01c0d40700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc1602d0 con 0x7f01bc0c2440 2023-03-03T08:43:51.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.313+0000 7f01c0d40700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc161620 con 0x7f01bc161cd0 2023-03-03T08:43:51.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.313+0000 7f01c0d40700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc160cf0 con 0x7f01bc08b970 2023-03-03T08:43:51.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.313+0000 7f01b9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc0c2440 0x7f01bc159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.313+0000 7f01b9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc0c2440 0x7f01bc159170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51740/0 (socket says 172.21.15.7:51740) 2023-03-03T08:43:51.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.314+0000 7f01b9d9b700 1 -- 172.21.15.7:0/332376068 learned_addr learned my addr 172.21.15.7:0/332376068 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.314+0000 7f01ba59c700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc08b970 0x7f01bc15f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.314+0000 7f01bad9d700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01bc161cd0 0x7f01bc08a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.314+0000 7f01ba59c700 1 -- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01bc161cd0 msgr2=0x7f01bc08a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.314+0000 7f01ba59c700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01bc161cd0 0x7f01bc08a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.314+0000 7f01ba59c700 1 -- 172.21.15.7:0/332376068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc0c2440 msgr2=0x7f01bc159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.314+0000 7f01ba59c700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc0c2440 0x7f01bc159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.314+0000 7f01ba59c700 1 -- 172.21.15.7:0/332376068 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac008610 con 0x7f01bc08b970 2023-03-03T08:43:51.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.314+0000 7f08eeffd700 1 -- 172.21.15.7:0/291582954 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08e808a060 con 0x7f08f80c1ac0 2023-03-03T08:43:51.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.314+0000 7f01ba59c700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc08b970 0x7f01bc15f5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f01b000c460 tx=0x7f01b000c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f07717fa700 1 -- 172.21.15.7:0/2629029958 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c011630 con 0x7f07740ab8f0 2023-03-03T08:43:51.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f91ad1f4700 1 Processor -- start 2023-03-03T08:43:51.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f07717fa700 1 -- 172.21.15.7:0/2629029958 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c011790 con 0x7f07740ab8f0 2023-03-03T08:43:51.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f077ac4e700 1 -- 172.21.15.7:0/2629029958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077406e280 con 0x7f07740ab8f0 2023-03-03T08:43:51.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f07717fa700 1 -- 172.21.15.7:0/2629029958 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c011b20 con 0x7f07740ab8f0 2023-03-03T08:43:51.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f077ac4e700 1 -- 172.21.15.7:0/2629029958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0774158bd0 con 0x7f07740ab8f0 2023-03-03T08:43:51.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f7d26c49700 1 -- 172.21.15.7:0/278018187 shutdown_connections 2023-03-03T08:43:51.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f7d26c49700 1 -- 172.21.15.7:0/278018187 wait complete. 2023-03-03T08:43:51.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7fcfa9569700 1 Processor -- start 2023-03-03T08:43:51.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f01aaffd700 1 -- 172.21.15.7:0/332376068 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0004c80 con 0x7f01bc08b970 2023-03-03T08:43:51.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f01aaffd700 1 -- 172.21.15.7:0/332376068 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b0004de0 con 0x7f01bc08b970 2023-03-03T08:43:51.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f01aaffd700 1 -- 172.21.15.7:0/332376068 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b00072f0 con 0x7f01bc08b970 2023-03-03T08:43:51.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f01c0d40700 1 -- 172.21.15.7:0/332376068 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01bc0aa140 con 0x7f01bc08b970 2023-03-03T08:43:51.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f01c0d40700 1 -- 172.21.15.7:0/332376068 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01bc0a9e60 con 0x7f01bc08b970 2023-03-03T08:43:51.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f7d26c49700 1 Processor -- start 2023-03-03T08:43:51.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f7d26c49700 1 -- start start 2023-03-03T08:43:51.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.316+0000 7f077ac4e700 1 -- 172.21.15.7:0/2629029958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0774074bb0 con 0x7f07740ab8f0 2023-03-03T08:43:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f01c0d40700 1 -- 172.21.15.7:0/332376068 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01bc0ac0f0 con 0x7f01bc08b970 2023-03-03T08:43:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7fcfa9569700 1 -- start start 2023-03-03T08:43:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7f01aaffd700 1 -- 172.21.15.7:0/332376068 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f01b001a020 con 0x7f01bc08b970 2023-03-03T08:43:51.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7f7d26c49700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2006c0e0 0x7f7d20152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7f7d26c49700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d200a8bd0 0x7f7d2015c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7f7d26c49700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d200c2680 0x7f7d200a3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7f7d26c49700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201532b0 con 0x7f7d200c2680 2023-03-03T08:43:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7f01aaffd700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f019c0683c0 0x7f019c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7f01aaffd700 1 -- 172.21.15.7:0/332376068 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f01b0085b40 con 0x7f01bc08b970 2023-03-03T08:43:51.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7f7d26c49700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201ee200 con 0x7f7d2006c0e0 2023-03-03T08:43:51.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7f7d26c49700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d201ee4e0 con 0x7f7d200a8bd0 2023-03-03T08:43:51.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7fcfa9569700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcfa4071990 0x7fcfa4071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7fcfa9569700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfa414b460 0x7fcfa40722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7fcfa9569700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa4070360 con 0x7fcfa4151c00 2023-03-03T08:43:51.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7fcfa9569700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa41521a0 con 0x7fcfa4071990 2023-03-03T08:43:51.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7fcfa9569700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa406a5e0 con 0x7fcfa414b460 2023-03-03T08:43:51.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa27fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfa414b460 0x7fcfa40722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa2ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcfa4071990 0x7fcfa4071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa37fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fcfa4151c00 0x7fcfa406aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49258/0 (socket says 172.21.15.7:49258) 2023-03-03T08:43:51.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa37fe700 1 -- 172.21.15.7:0/3425284930 learned_addr learned my addr 172.21.15.7:0/3425284930 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:51.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.318+0000 7f07717fa700 1 -- 172.21.15.7:0/2629029958 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f075c017020 con 0x7f07740ab8f0 2023-03-03T08:43:51.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7f07717fa700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0754068340 0x7f075406a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7f0773fff700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0754068340 0x7f075406a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa2ffd700 1 -- 172.21.15.7:0/3425284930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfa414b460 msgr2=0x7fcfa40722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:51.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa2ffd700 1 --2- 172.21.15.7:0/3425284930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfa414b460 0x7fcfa40722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa2ffd700 1 -- 172.21.15.7:0/3425284930 >> v1:172.21.15.143:6789/0 conn(0x7fcfa4151c00 legacy=0x7fcfa406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa2ffd700 1 -- 172.21.15.7:0/3425284930 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa40c38b0 con 0x7fcfa4071990 2023-03-03T08:43:51.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa2ffd700 1 --2- 172.21.15.7:0/3425284930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa4071990 0x7fcfa4071d60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800b8f0 tx=0x7fcf9800bc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bb29e0828845fa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa1ffb700 1 -- 172.21.15.7:0/3425284930 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800c9f0 con 0x7fcfa4071990 2023-03-03T08:43:51.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7f0773fff700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0754068340 0x7f075406a7f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0764001dc0 tx=0x7f0764001d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa1ffb700 1 -- 172.21.15.7:0/3425284930 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9800cb50 con 0x7fcfa4071990 2023-03-03T08:43:51.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.319+0000 7fcfa1ffb700 1 -- 172.21.15.7:0/3425284930 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800f460 con 0x7fcfa4071990 2023-03-03T08:43:51.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.320+0000 7fcfa9569700 1 -- 172.21.15.7:0/3425284930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa4071990 msgr2=0x7fcfa4071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.320+0000 7fcfa9569700 1 --2- 172.21.15.7:0/3425284930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa4071990 0x7fcfa4071d60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800b8f0 tx=0x7fcf9800bc00 comp rx=0 tx=0).stop 2023-03-03T08:43:51.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.320+0000 7fcfa9569700 1 -- 172.21.15.7:0/3425284930 shutdown_connections 2023-03-03T08:43:51.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.320+0000 7fcfa9569700 1 --2- 172.21.15.7:0/3425284930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa4071990 0x7fcfa4071d60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.320+0000 7fcfa9569700 1 --2- 172.21.15.7:0/3425284930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfa414b460 0x7fcfa40722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.320+0000 7fcfa9569700 1 -- 172.21.15.7:0/3425284930 >> 172.21.15.7:0/3425284930 conn(0x7fcfa40a94d0 msgr2=0x7fcfa40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.315+0000 7f91ad1f4700 1 -- start start 2023-03-03T08:43:51.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.316+0000 7f91ad1f4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91a80c35a0 0x7f91a80a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.316+0000 7f91ad1f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a8151ec0 0x7f91a80a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.316+0000 7f91ad1f4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91a80bb390 0x7f91a80bcb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.316+0000 7f91ad1f4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a80a8970 con 0x7f91a80c35a0 2023-03-03T08:43:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.316+0000 7f91ad1f4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a80a8e00 con 0x7f91a80bb390 2023-03-03T08:43:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.316+0000 7f91ad1f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a81ef270 con 0x7f91a8151ec0 2023-03-03T08:43:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.316+0000 7f91a759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91a80bb390 0x7f91a80bcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.316+0000 7f91a759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91a80bb390 0x7f91a80bcb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47520/0 (socket says 172.21.15.7:47520) 2023-03-03T08:43:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.316+0000 7f91a759e700 1 -- 172.21.15.7:0/3495668231 learned_addr learned my addr 172.21.15.7:0/3495668231 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f91a659c700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a8151ec0 0x7f91a80a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f91a759e700 1 -- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a8151ec0 msgr2=0x7f91a80a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f91a759e700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a8151ec0 0x7f91a80a7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f91a759e700 1 -- 172.21.15.7:0/3495668231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91a80c35a0 msgr2=0x7f91a80a74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f91a759e700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91a80c35a0 0x7f91a80a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f91a759e700 1 -- 172.21.15.7:0/3495668231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c009550 con 0x7f91a80bb390 2023-03-03T08:43:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f91a759e700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91a80bb390 0x7f91a80bcb50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f91a000b990 tx=0x7f91a000bd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f918f7fe700 1 -- 172.21.15.7:0/3495668231 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a000c9d0 con 0x7f91a80bb390 2023-03-03T08:43:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f918f7fe700 1 -- 172.21.15.7:0/3495668231 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91a000cb30 con 0x7f91a80bb390 2023-03-03T08:43:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.317+0000 7f918f7fe700 1 -- 172.21.15.7:0/3495668231 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a0014440 con 0x7f91a80bb390 2023-03-03T08:43:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.321+0000 7fcfa9569700 1 -- 172.21.15.7:0/3425284930 shutdown_connections 2023-03-03T08:43:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.321+0000 7fcfa9569700 1 -- 172.21.15.7:0/3425284930 wait complete. 2023-03-03T08:43:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.322+0000 7f07717fa700 1 -- 172.21.15.7:0/2629029958 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f075c05f460 con 0x7f07740ab8f0 2023-03-03T08:43:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.322+0000 7f91ad1f4700 1 -- 172.21.15.7:0/3495668231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a80c3000 con 0x7f91a80bb390 2023-03-03T08:43:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.322+0000 7f91ad1f4700 1 -- 172.21.15.7:0/3495668231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a8151820 con 0x7f91a80bb390 2023-03-03T08:43:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.322+0000 7fcfa9569700 1 Processor -- start 2023-03-03T08:43:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.322+0000 7fcfa9569700 1 -- start start 2023-03-03T08:43:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7f91ad1f4700 1 -- 172.21.15.7:0/3495668231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a80c32e0 con 0x7f91a80bb390 2023-03-03T08:43:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.322+0000 7f2eec8ac700 1 Processor -- start 2023-03-03T08:43:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.322+0000 7f2eec8ac700 1 -- start start 2023-03-03T08:43:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7f2eec8ac700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ee80a9c60 0x7f2ee80c3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7f2eec8ac700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2ee80720a0 0x7f2ee8158d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7f2eec8ac700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee815c710 con 0x7f2ee80c31c0 2023-03-03T08:43:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7f2eec8ac700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee80aa8f0 con 0x7f2ee80a9c60 2023-03-03T08:43:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7f2eec8ac700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee806af40 con 0x7f2ee80720a0 2023-03-03T08:43:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7f2ee77fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2ee80c31c0 0x7f2ee8158670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45748/0 (socket says 172.21.15.7:45748) 2023-03-03T08:43:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7f2ee77fe700 1 -- 172.21.15.7:0/1208611825 learned_addr learned my addr 172.21.15.7:0/1208611825 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f2ee6ffd700 1 --2- 172.21.15.7:0/1208611825 >> v2:172.21.15.7:3300/0 conn(0x7f2ee80a9c60 0x7f2ee80c3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f01aaffd700 1 -- 172.21.15.7:0/332376068 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f01b00520b0 con 0x7f01bc08b970 2023-03-03T08:43:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f7d251e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d200c2680 0x7f7d200a3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f7d251e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d200c2680 0x7f7d200a3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51756/0 (socket says 172.21.15.7:51756) 2023-03-03T08:43:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f7d251e6700 1 -- 172.21.15.7:0/2642654867 learned_addr learned my addr 172.21.15.7:0/2642654867 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7fcfa9569700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa4071990 0x7fcfa414eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7fcfa9569700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfa4151c00 0x7fcfa41e8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7fcfa9569700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa40c1590 0x7fcfa40aa600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7fcfa9569700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa41e96b0 con 0x7fcfa4071990 2023-03-03T08:43:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7fcfa9569700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa40be810 con 0x7fcfa40c1590 2023-03-03T08:43:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.323+0000 7fcfa9569700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa41e9950 con 0x7fcfa4151c00 2023-03-03T08:43:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7fcfa2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa4071990 0x7fcfa414eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7fcfa2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa4071990 0x7fcfa414eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51774/0 (socket says 172.21.15.7:51774) 2023-03-03T08:43:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7fcfa2ffd700 1 -- 172.21.15.7:0/3979726543 learned_addr learned my addr 172.21.15.7:0/3979726543 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7fcfa37fe700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa40c1590 0x7fcfa40aa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7fcfa27fc700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfa4151c00 0x7fcfa41e8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f01b9d9b700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f019c0683c0 0x7f019c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f07717fa700 1 -- 172.21.15.7:0/2629029958 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f075c08c060 con 0x7f07740ab8f0 2023-03-03T08:43:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7fcfa2ffd700 1 -- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfa4151c00 msgr2=0x7fcfa41e8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7fcfa2ffd700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfa4151c00 0x7fcfa41e8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7fcfa2ffd700 1 -- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa40c1590 msgr2=0x7fcfa40aa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7fcfa2ffd700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa40c1590 0x7fcfa40aa600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7fcfa2ffd700 1 -- 172.21.15.7:0/3979726543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9800b550 con 0x7fcfa4071990 2023-03-03T08:43:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.325+0000 7fcfa2ffd700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa4071990 0x7fcfa414eff0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800bc40 tx=0x7fcf98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f2ee6ffd700 1 -- 172.21.15.7:0/1208611825 >> v2:172.21.15.143:3300/0 conn(0x7f2ee80720a0 msgr2=0x7f2ee8158d80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T08:43:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f2ee6ffd700 1 --2- 172.21.15.7:0/1208611825 >> v2:172.21.15.143:3300/0 conn(0x7f2ee80720a0 0x7f2ee8158d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f2ee6ffd700 1 -- 172.21.15.7:0/1208611825 >> v1:172.21.15.7:6789/0 conn(0x7f2ee80c31c0 legacy=0x7f2ee8158670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.324+0000 7f2ee6ffd700 1 -- 172.21.15.7:0/1208611825 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee815c9f0 con 0x7f2ee80a9c60 2023-03-03T08:43:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.325+0000 7f2ee6ffd700 1 --2- 172.21.15.7:0/1208611825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ee80a9c60 0x7f2ee80c3590 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8072920 tx=0x7f2edc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4df3c31046f9c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.325+0000 7f2ee67fc700 1 -- 172.21.15.7:0/1208611825 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc011830 con 0x7f2ee80a9c60 2023-03-03T08:43:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.325+0000 7f2ee67fc700 1 -- 172.21.15.7:0/1208611825 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc011990 con 0x7f2ee80a9c60 2023-03-03T08:43:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.325+0000 7f2ee67fc700 1 -- 172.21.15.7:0/1208611825 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc011d20 con 0x7f2ee80a9c60 2023-03-03T08:43:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.325+0000 7f918f7fe700 1 -- 172.21.15.7:0/3495668231 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f91a001b030 con 0x7f91a80bb390 2023-03-03T08:43:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f2eec8ac700 1 -- 172.21.15.7:0/1208611825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ee80a9c60 msgr2=0x7f2ee80c3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f2eec8ac700 1 --2- 172.21.15.7:0/1208611825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ee80a9c60 0x7f2ee80c3590 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8072920 tx=0x7f2edc009b20 comp rx=0 tx=0).stop 2023-03-03T08:43:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d251e6700 1 -- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d200a8bd0 msgr2=0x7f7d2015c020 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T08:43:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d251e6700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d200a8bd0 0x7f7d2015c020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d251e6700 1 -- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2006c0e0 msgr2=0x7f7d20152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d251e6700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2006c0e0 0x7f7d20152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d251e6700 1 -- 172.21.15.7:0/2642654867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c0094a0 con 0x7f7d200c2680 2023-03-03T08:43:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d251e6700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d200c2680 0x7f7d200a3480 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800b990 tx=0x7f7d1800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d1d7fa700 1 -- 172.21.15.7:0/2642654867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1800c9d0 con 0x7f7d200c2680 2023-03-03T08:43:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d1d7fa700 1 -- 172.21.15.7:0/2642654867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1800cb30 con 0x7f7d200c2680 2023-03-03T08:43:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d1d7fa700 1 -- 172.21.15.7:0/2642654867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d18014440 con 0x7f7d200c2680 2023-03-03T08:43:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d26c49700 1 -- 172.21.15.7:0/2642654867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d200aa030 con 0x7f7d200c2680 2023-03-03T08:43:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f7d26c49700 1 -- 172.21.15.7:0/2642654867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d200a9b10 con 0x7f7d200c2680 2023-03-03T08:43:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f918f7fe700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f919408a300 0x7f919408c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.326+0000 7f918f7fe700 1 -- 172.21.15.7:0/3495668231 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f91a0085c60 con 0x7f91a80bb390 2023-03-03T08:43:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.327+0000 7f91a6d9d700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f919408a300 0x7f919408c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.327+0000 7f91a6d9d700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f919408a300 0x7f919408c7b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91a80c16a0 tx=0x7f919c010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.328+0000 7f2eec8ac700 1 -- 172.21.15.7:0/1208611825 shutdown_connections 2023-03-03T08:43:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.328+0000 7f2eec8ac700 1 --2- 172.21.15.7:0/1208611825 >> v2:172.21.15.143:3300/0 conn(0x7f2ee80720a0 0x7f2ee8158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.328+0000 7f2eec8ac700 1 --2- 172.21.15.7:0/1208611825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ee80a9c60 0x7f2ee80c3590 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.328+0000 7f2eec8ac700 1 -- 172.21.15.7:0/1208611825 >> 172.21.15.7:0/1208611825 conn(0x7f2ee80c6cc0 msgr2=0x7f2ee80c2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.329+0000 7f7d1d7fa700 1 -- 172.21.15.7:0/2642654867 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7d1801b030 con 0x7f7d200c2680 2023-03-03T08:43:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.329+0000 7f7d1d7fa700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d040682f0 0x7f7d0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.329+0000 7f7d1d7fa700 1 -- 172.21.15.7:0/2642654867 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f7d18085cc0 con 0x7f7d200c2680 2023-03-03T08:43:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.329+0000 7f7d249e5700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d040682f0 0x7f7d0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.330+0000 7f2eec8ac700 1 -- 172.21.15.7:0/1208611825 shutdown_connections 2023-03-03T08:43:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.330+0000 7f2eec8ac700 1 -- 172.21.15.7:0/1208611825 wait complete. 2023-03-03T08:43:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.330+0000 7f7d249e5700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d040682f0 0x7f7d0406a7a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7d200a4fe0 tx=0x7f7d0c010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.331+0000 7f918f7fe700 1 -- 172.21.15.7:0/3495668231 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f91a008b020 con 0x7f91a80bb390 2023-03-03T08:43:51.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.331+0000 7f2eec8ac700 1 Processor -- start 2023-03-03T08:43:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.332+0000 7f2eec8ac700 1 -- start start 2023-03-03T08:43:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2eec8ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee80720a0 0x7f2ee814d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2eec8ac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee80a9c60 0x7f2ee815a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2eec8ac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ee80c31c0 0x7f2ee814e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2eec8ac700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee814e090 con 0x7f2ee80c31c0 2023-03-03T08:43:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2eec8ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee814ddb0 con 0x7f2ee80720a0 2023-03-03T08:43:51.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2eec8ac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee806f1d0 con 0x7f2ee80a9c60 2023-03-03T08:43:51.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2ee6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee80a9c60 0x7f2ee815a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2ee77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee80720a0 0x7f2ee814d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2ee6ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee80a9c60 0x7f2ee815a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59858/0 (socket says 172.21.15.7:59858) 2023-03-03T08:43:51.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2ee77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee80720a0 0x7f2ee814d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47558/0 (socket says 172.21.15.7:47558) 2023-03-03T08:43:51.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2ee6ffd700 1 -- 172.21.15.7:0/252160881 learned_addr learned my addr 172.21.15.7:0/252160881 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.333+0000 7f2ee7fff700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ee80c31c0 0x7f2ee814e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2ee6ffd700 1 -- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee80720a0 msgr2=0x7f2ee814d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2ee6ffd700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee80720a0 0x7f2ee814d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2ee6ffd700 1 -- 172.21.15.7:0/252160881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ee80c31c0 msgr2=0x7f2ee814e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2ee6ffd700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ee80c31c0 0x7f2ee814e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2ee6ffd700 1 -- 172.21.15.7:0/252160881 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0094a0 con 0x7f2ee80a9c60 2023-03-03T08:43:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2ee6ffd700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee80a9c60 0x7f2ee815a990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2edc000c00 tx=0x7f2edc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2ecffff700 1 -- 172.21.15.7:0/252160881 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc0036a0 con 0x7f2ee80a9c60 2023-03-03T08:43:51.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2ecffff700 1 -- 172.21.15.7:0/252160881 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc01f070 con 0x7f2ee80a9c60 2023-03-03T08:43:51.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2eec8ac700 1 -- 172.21.15.7:0/252160881 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee81592c0 con 0x7f2ee80a9c60 2023-03-03T08:43:51.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2ecffff700 1 -- 172.21.15.7:0/252160881 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc003a40 con 0x7f2ee80a9c60 2023-03-03T08:43:51.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.334+0000 7f2eec8ac700 1 -- 172.21.15.7:0/252160881 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee806ab60 con 0x7f2ee80a9c60 2023-03-03T08:43:51.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.336+0000 7f4c88a23700 1 Processor -- start 2023-03-03T08:43:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.336+0000 7f4c88a23700 1 -- start start 2023-03-03T08:43:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7fcf8b7fe700 1 -- 172.21.15.7:0/3979726543 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9801c070 con 0x7fcfa4071990 2023-03-03T08:43:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7fcf8b7fe700 1 -- 172.21.15.7:0/3979726543 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9800ce30 con 0x7fcfa4071990 2023-03-03T08:43:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7fcf8b7fe700 1 -- 172.21.15.7:0/3979726543 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980038c0 con 0x7fcfa4071990 2023-03-03T08:43:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f7d26c49700 1 -- 172.21.15.7:0/2642654867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d200aa310 con 0x7f7d200c2680 2023-03-03T08:43:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.336+0000 7f2ecffff700 1 -- 172.21.15.7:0/252160881 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2edc00f050 con 0x7f2ee80a9c60 2023-03-03T08:43:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f2ecffff700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ed40682f0 0x7f2ed406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f2ecffff700 1 -- 172.21.15.7:0/252160881 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f2edc021070 con 0x7f2ee80a9c60 2023-03-03T08:43:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f4c88a23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4c8414f9b0 0x7f4c840ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f4c88a23700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c8406f0e0 0x7f4c840c1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f4c88a23700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8406b8e0 con 0x7f4c840abb20 2023-03-03T08:43:51.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f4c88a23700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c840c2450 con 0x7f4c8406f0e0 2023-03-03T08:43:51.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f4c88a23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c840c2bf0 con 0x7f4c8414f9b0 2023-03-03T08:43:51.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f4c81d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c8406f0e0 0x7f4c840c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f4c81d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c8406f0e0 0x7f4c840c1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51792/0 (socket says 172.21.15.7:51792) 2023-03-03T08:43:51.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f4c81d9b700 1 -- 172.21.15.7:0/2116523666 learned_addr learned my addr 172.21.15.7:0/2116523666 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.337+0000 7f4c8259c700 1 --2- 172.21.15.7:0/2116523666 >> v2:172.21.15.143:3300/0 conn(0x7f4c8414f9b0 0x7f4c840ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.338+0000 7f4c81d9b700 1 -- 172.21.15.7:0/2116523666 >> v2:172.21.15.143:3300/0 conn(0x7f4c8414f9b0 msgr2=0x7f4c840ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.338+0000 7f4c81d9b700 1 --2- 172.21.15.7:0/2116523666 >> v2:172.21.15.143:3300/0 conn(0x7f4c8414f9b0 0x7f4c840ab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.338+0000 7f4c81d9b700 1 -- 172.21.15.7:0/2116523666 >> v1:172.21.15.7:6789/0 conn(0x7f4c840abb20 legacy=0x7f4c840c1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.338+0000 7f4c81d9b700 1 -- 172.21.15.7:0/2116523666 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c840c2040 con 0x7f4c8406f0e0 2023-03-03T08:43:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.338+0000 7f4c81d9b700 1 --2- 172.21.15.7:0/2116523666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c8406f0e0 0x7f4c840c1040 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4c840c0b10 tx=0x7f4c7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98cbe1a7ff6a9ad6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.338+0000 7f2ee77fe700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ed40682f0 0x7f2ed406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.338+0000 7f4c8159a700 1 -- 172.21.15.7:0/2116523666 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c00e070 con 0x7f4c8406f0e0 2023-03-03T08:43:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.338+0000 7f4c8159a700 1 -- 172.21.15.7:0/2116523666 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c003a20 con 0x7f4c8406f0e0 2023-03-03T08:43:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.338+0000 7f4c8159a700 1 -- 172.21.15.7:0/2116523666 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c003dd0 con 0x7f4c8406f0e0 2023-03-03T08:43:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.339+0000 7fcfa9569700 1 -- 172.21.15.7:0/3979726543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa40c3b50 con 0x7fcfa4071990 2023-03-03T08:43:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.339+0000 7f2eec8ac700 1 -- 172.21.15.7:0/252160881 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee80aa580 con 0x7f2ee80a9c60 2023-03-03T08:43:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.339+0000 7fcfa9569700 1 -- 172.21.15.7:0/3979726543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa406a2b0 con 0x7fcfa4071990 2023-03-03T08:43:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.339+0000 7f4c88a23700 1 -- 172.21.15.7:0/2116523666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c8406f0e0 msgr2=0x7f4c840c1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.339+0000 7f4c88a23700 1 --2- 172.21.15.7:0/2116523666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c8406f0e0 0x7f4c840c1040 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4c840c0b10 tx=0x7f4c7c00b330 comp rx=0 tx=0).stop 2023-03-03T08:43:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.339+0000 7f2ee77fe700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ed40682f0 0x7f2ed406a7a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8005910 tx=0x7f2ed8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.340+0000 7f4c88a23700 1 -- 172.21.15.7:0/2116523666 shutdown_connections 2023-03-03T08:43:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.340+0000 7f4c88a23700 1 --2- 172.21.15.7:0/2116523666 >> v2:172.21.15.143:3300/0 conn(0x7f4c8414f9b0 0x7f4c840ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.340+0000 7f4c88a23700 1 --2- 172.21.15.7:0/2116523666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c8406f0e0 0x7f4c840c1040 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.340+0000 7f4c88a23700 1 -- 172.21.15.7:0/2116523666 >> 172.21.15.7:0/2116523666 conn(0x7f4c840c62e0 msgr2=0x7f4c840c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.340+0000 7f4c88a23700 1 -- 172.21.15.7:0/2116523666 shutdown_connections 2023-03-03T08:43:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.340+0000 7f4c88a23700 1 -- 172.21.15.7:0/2116523666 wait complete. 2023-03-03T08:43:51.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.342+0000 7f4c88a23700 1 Processor -- start 2023-03-03T08:43:51.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7fcfa9569700 1 -- 172.21.15.7:0/3979726543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa406b180 con 0x7fcfa4071990 2023-03-03T08:43:51.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.342+0000 7f4c88a23700 1 -- start start 2023-03-03T08:43:51.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c88a23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c8406f0e0 0x7f4c84071650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c88a23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c840abb20 0x7f4c84155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c88a23700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c8414f9b0 0x7f4c84144a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c88a23700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c84155a70 con 0x7f4c8414f9b0 2023-03-03T08:43:51.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c88a23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c841442d0 con 0x7f4c8406f0e0 2023-03-03T08:43:51.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c88a23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c84144570 con 0x7f4c840abb20 2023-03-03T08:43:51.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c82d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c8414f9b0 0x7f4c84144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c8259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c8406f0e0 0x7f4c84071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c8259c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c8406f0e0 0x7f4c84071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47572/0 (socket says 172.21.15.7:47572) 2023-03-03T08:43:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c8259c700 1 -- 172.21.15.7:0/2298091507 learned_addr learned my addr 172.21.15.7:0/2298091507 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.343+0000 7f4c82d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c8414f9b0 0x7f4c84144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51800/0 (socket says 172.21.15.7:51800) 2023-03-03T08:43:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c8259c700 1 -- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c840abb20 msgr2=0x7f4c84155f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-03T08:43:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c8259c700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c840abb20 0x7f4c84155f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c8259c700 1 -- 172.21.15.7:0/2298091507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c8414f9b0 msgr2=0x7f4c84144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c8259c700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c8414f9b0 0x7f4c84144a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c8259c700 1 -- 172.21.15.7:0/2298091507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c009b30 con 0x7f4c8406f0e0 2023-03-03T08:43:51.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f2ecffff700 1 -- 172.21.15.7:0/252160881 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2edc053e80 con 0x7f2ee80a9c60 2023-03-03T08:43:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c8259c700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c8406f0e0 0x7f4c84071650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400d830 tx=0x7f4c7400db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c72ffd700 1 -- 172.21.15.7:0/2298091507 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c740125f0 con 0x7f4c8406f0e0 2023-03-03T08:43:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c72ffd700 1 -- 172.21.15.7:0/2298091507 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c74012750 con 0x7f4c8406f0e0 2023-03-03T08:43:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c88a23700 1 -- 172.21.15.7:0/2298091507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c84152ff0 con 0x7f4c8406f0e0 2023-03-03T08:43:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c72ffd700 1 -- 172.21.15.7:0/2298091507 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74012ae0 con 0x7f4c8406f0e0 2023-03-03T08:43:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.344+0000 7f4c88a23700 1 -- 172.21.15.7:0/2298091507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c840ab3e0 con 0x7f4c8406f0e0 2023-03-03T08:43:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.346+0000 7f4c72ffd700 1 -- 172.21.15.7:0/2298091507 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4c74012c40 con 0x7f4c8406f0e0 2023-03-03T08:43:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.346+0000 7f4c72ffd700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c640682c0 0x7f4c6406a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.346+0000 7f4c72ffd700 1 -- 172.21.15.7:0/2298091507 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f4c740864b0 con 0x7f4c8406f0e0 2023-03-03T08:43:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.346+0000 7f4c81d9b700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c640682c0 0x7f4c6406a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.347+0000 7f4c81d9b700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c640682c0 0x7f4c6406a770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4c840c0b10 tx=0x7f4c7c009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.347+0000 7f7d1d7fa700 1 -- 172.21.15.7:0/2642654867 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7d180521f0 con 0x7f7d200c2680 2023-03-03T08:43:51.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.348+0000 7f01b9d9b700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f019c0683c0 0x7f019c06a870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f01bc161910 tx=0x7f01ac011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.349+0000 7fcf8b7fe700 1 -- 172.21.15.7:0/3979726543 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcf98029030 con 0x7fcfa4071990 2023-03-03T08:43:51.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.349+0000 7fcf8b7fe700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf90068340 0x7fcf9006a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.349+0000 7fcf8b7fe700 1 -- 172.21.15.7:0/3979726543 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fcf98087940 con 0x7fcfa4071990 2023-03-03T08:43:51.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.349+0000 7fcfa27fc700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf90068340 0x7fcf9006a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.350+0000 7fcfa27fc700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf90068340 0x7fcf9006a7f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcfa41e6740 tx=0x7fcf9c00f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.357+0000 7fcf8b7fe700 1 -- 172.21.15.7:0/3979726543 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcf98053fc0 con 0x7fcfa4071990 2023-03-03T08:43:51.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.357+0000 7f4c88a23700 1 -- 172.21.15.7:0/2298091507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c840c2630 con 0x7f4c8406f0e0 2023-03-03T08:43:51.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.362+0000 7f4c72ffd700 1 -- 172.21.15.7:0/2298091507 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c74052960 con 0x7f4c8406f0e0 2023-03-03T08:43:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.729+0000 7f0900720700 1 -- 172.21.15.7:0/291582954 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f08f8074bb0 con 0x7f08f80c1ac0 2023-03-03T08:43:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.730+0000 7f08eeffd700 1 -- 172.21.15.7:0/291582954 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f08e8017020 con 0x7f08f80c1ac0 2023-03-03T08:43:51.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.733+0000 7f077ac4e700 1 -- 172.21.15.7:0/2629029958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0774074bb0 con 0x7f07740ab8f0 2023-03-03T08:43:51.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.733+0000 7f07717fa700 1 -- 172.21.15.7:0/2629029958 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f075c08c020 con 0x7f07740ab8f0 2023-03-03T08:43:51.736 INFO:teuthology.orchestra.run.smithi007.stdout:55834574855 2023-03-03T08:43:51.736 INFO:teuthology.orchestra.run.smithi007.stdout:55834574856 2023-03-03T08:43:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.736+0000 7f91ad1f4700 1 -- 172.21.15.7:0/3495668231 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f91a8074bb0 con 0x7f91a80bb390 2023-03-03T08:43:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.736+0000 7f01c0d40700 1 -- 172.21.15.7:0/332376068 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f01bc074bb0 con 0x7f01bc08b970 2023-03-03T08:43:51.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.736+0000 7f077ac4e700 1 -- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0754068340 msgr2=0x7f075406a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.736+0000 7f077ac4e700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0754068340 0x7f075406a7f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0764001dc0 tx=0x7f0764001d50 comp rx=0 tx=0).stop 2023-03-03T08:43:51.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.736+0000 7f077ac4e700 1 -- 172.21.15.7:0/2629029958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07740ab8f0 msgr2=0x7f0774157f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.736+0000 7f077ac4e700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07740ab8f0 0x7f0774157f00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f077406b120 tx=0x7f075c004750 comp rx=0 tx=0).stop 2023-03-03T08:43:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.737+0000 7f0900720700 1 -- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08dc0682f0 msgr2=0x7f08dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.737+0000 7f0900720700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08dc0682f0 0x7f08dc06a7a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f08f8153fd0 tx=0x7f08f000d040 comp rx=0 tx=0).stop 2023-03-03T08:43:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.737+0000 7f0900720700 1 -- 172.21.15.7:0/291582954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f80c1ac0 msgr2=0x7f08f80bec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.737+0000 7f0900720700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f80c1ac0 0x7f08f80bec40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f08e800c4a0 tx=0x7f08e800c7b0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.738+0000 7f918f7fe700 1 -- 172.21.15.7:0/3495668231 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f91a0019090 con 0x7f91a80bb390 2023-03-03T08:43:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.738+0000 7f077ac4e700 1 -- 172.21.15.7:0/2629029958 shutdown_connections 2023-03-03T08:43:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.738+0000 7f077ac4e700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07740724d0 0x7f077406d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.738+0000 7f077ac4e700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0754068340 0x7f075406a7f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.738+0000 7f077ac4e700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07740ab8f0 0x7f0774157f00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.738+0000 7f077ac4e700 1 --2- 172.21.15.7:0/2629029958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07740c5cb0 0x7f077406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.738+0000 7f077ac4e700 1 -- 172.21.15.7:0/2629029958 >> 172.21.15.7:0/2629029958 conn(0x7f07740a9430 msgr2=0x7f07740c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.738+0000 7f077ac4e700 1 -- 172.21.15.7:0/2629029958 shutdown_connections 2023-03-03T08:43:51.742 INFO:teuthology.orchestra.run.smithi007.stdout:55834574857 2023-03-03T08:43:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.739+0000 7f01aaffd700 1 -- 172.21.15.7:0/332376068 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f01b00556d0 con 0x7f01bc08b970 2023-03-03T08:43:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.740+0000 7f077ac4e700 1 -- 172.21.15.7:0/2629029958 wait complete. 2023-03-03T08:43:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.740+0000 7f0900720700 1 -- 172.21.15.7:0/291582954 shutdown_connections 2023-03-03T08:43:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.740+0000 7f0900720700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08f8151b60 0x7f08f81e7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.740+0000 7f0900720700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08f80aaee0 0x7f08f8153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.740+0000 7f0900720700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08dc0682f0 0x7f08dc06a7a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.740+0000 7f0900720700 1 --2- 172.21.15.7:0/291582954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08f80c1ac0 0x7f08f80bec40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.740+0000 7f0900720700 1 -- 172.21.15.7:0/291582954 >> 172.21.15.7:0/291582954 conn(0x7f08f80c62e0 msgr2=0x7f08f814ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.740+0000 7f0900720700 1 -- 172.21.15.7:0/291582954 shutdown_connections 2023-03-03T08:43:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.740+0000 7f0900720700 1 -- 172.21.15.7:0/291582954 wait complete. 2023-03-03T08:43:51.745 INFO:teuthology.orchestra.run.smithi007.stdout:55834574859 2023-03-03T08:43:51.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.745+0000 7f01c0d40700 1 -- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f019c0683c0 msgr2=0x7f019c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.745+0000 7f01c0d40700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f019c0683c0 0x7f019c06a870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f01bc161910 tx=0x7f01ac011000 comp rx=0 tx=0).stop 2023-03-03T08:43:51.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.745+0000 7f01c0d40700 1 -- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc08b970 msgr2=0x7f01bc15f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.745+0000 7f01c0d40700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc08b970 0x7f01bc15f5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f01b000c460 tx=0x7f01b000c770 comp rx=0 tx=0).stop 2023-03-03T08:43:51.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f01c0d40700 1 -- 172.21.15.7:0/332376068 shutdown_connections 2023-03-03T08:43:51.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f01c0d40700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01bc161cd0 0x7f01bc08a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f01c0d40700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc08b970 0x7f01bc15f5c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f01c0d40700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f019c0683c0 0x7f019c06a870 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f01c0d40700 1 --2- 172.21.15.7:0/332376068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc0c2440 0x7f01bc159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f01c0d40700 1 -- 172.21.15.7:0/332376068 >> 172.21.15.7:0/332376068 conn(0x7f01bc0c62e0 msgr2=0x7f01bc0c4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f2eec8ac700 1 -- 172.21.15.7:0/252160881 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2ee8061f40 con 0x7f2ee80a9c60 2023-03-03T08:43:51.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f01c0d40700 1 -- 172.21.15.7:0/332376068 shutdown_connections 2023-03-03T08:43:51.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f01c0d40700 1 -- 172.21.15.7:0/332376068 wait complete. 2023-03-03T08:43:51.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f2ecffff700 1 -- 172.21.15.7:0/252160881 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2edc0574a0 con 0x7f2ee80a9c60 2023-03-03T08:43:51.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f91ad1f4700 1 -- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f919408a300 msgr2=0x7f919408c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f91ad1f4700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f919408a300 0x7f919408c7b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f91a80c16a0 tx=0x7f919c010000 comp rx=0 tx=0).stop 2023-03-03T08:43:51.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f91ad1f4700 1 -- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91a80bb390 msgr2=0x7f91a80bcb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.746+0000 7f91ad1f4700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91a80bb390 0x7f91a80bcb50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f91a000b990 tx=0x7f91a000bd50 comp rx=0 tx=0).stop 2023-03-03T08:43:51.751 INFO:teuthology.orchestra.run.smithi007.stdout:55834574858 2023-03-03T08:43:51.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.750+0000 7f2eec8ac700 1 -- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ed40682f0 msgr2=0x7f2ed406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.750+0000 7f2eec8ac700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ed40682f0 0x7f2ed406a7a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8005910 tx=0x7f2ed8009040 comp rx=0 tx=0).stop 2023-03-03T08:43:51.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.750+0000 7f2eec8ac700 1 -- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee80a9c60 msgr2=0x7f2ee815a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.750+0000 7f2eec8ac700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee80a9c60 0x7f2ee815a990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2edc000c00 tx=0x7f2edc004750 comp rx=0 tx=0).stop 2023-03-03T08:43:51.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.750+0000 7f2eec8ac700 1 -- 172.21.15.7:0/252160881 shutdown_connections 2023-03-03T08:43:51.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.750+0000 7f2eec8ac700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ed40682f0 0x7f2ed406a7a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.750+0000 7f2eec8ac700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ee80c31c0 0x7f2ee814e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.750+0000 7f2eec8ac700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ee80720a0 0x7f2ee814d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.750+0000 7f2eec8ac700 1 --2- 172.21.15.7:0/252160881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ee80a9c60 0x7f2ee815a990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.750+0000 7f2eec8ac700 1 -- 172.21.15.7:0/252160881 >> 172.21.15.7:0/252160881 conn(0x7f2ee80c6cc0 msgr2=0x7f2ee80abad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.751+0000 7fcfa9569700 1 -- 172.21.15.7:0/3979726543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fcfa4074bb0 con 0x7fcfa4071990 2023-03-03T08:43:51.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.751+0000 7fcf8b7fe700 1 -- 172.21.15.7:0/3979726543 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcf980575e0 con 0x7fcfa4071990 2023-03-03T08:43:51.754 INFO:teuthology.orchestra.run.smithi007.stdout:55834574857 2023-03-03T08:43:51.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.752+0000 7f7d26c49700 1 -- 172.21.15.7:0/2642654867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7d2006f010 con 0x7f7d200c2680 2023-03-03T08:43:51.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.752+0000 7f7d1d7fa700 1 -- 172.21.15.7:0/2642654867 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7d18019020 con 0x7f7d200c2680 2023-03-03T08:43:51.755 INFO:teuthology.orchestra.run.smithi007.stdout:55834574857 2023-03-03T08:43:51.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.754+0000 7f2eec8ac700 1 -- 172.21.15.7:0/252160881 shutdown_connections 2023-03-03T08:43:51.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.755+0000 7f2eec8ac700 1 -- 172.21.15.7:0/252160881 wait complete. 2023-03-03T08:43:51.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f7d26c49700 1 -- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d040682f0 msgr2=0x7f7d0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f7d26c49700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d040682f0 0x7f7d0406a7a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7d200a4fe0 tx=0x7f7d0c010000 comp rx=0 tx=0).stop 2023-03-03T08:43:51.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f7d26c49700 1 -- 172.21.15.7:0/2642654867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d200c2680 msgr2=0x7f7d200a3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f7d26c49700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d200c2680 0x7f7d200a3480 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800b990 tx=0x7f7d1800bd50 comp rx=0 tx=0).stop 2023-03-03T08:43:51.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f7d26c49700 1 -- 172.21.15.7:0/2642654867 shutdown_connections 2023-03-03T08:43:51.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f7d26c49700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d040682f0 0x7f7d0406a7a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f7d26c49700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d200c2680 0x7f7d200a3480 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f7d26c49700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d2006c0e0 0x7f7d20152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f7d26c49700 1 --2- 172.21.15.7:0/2642654867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d200a8bd0 0x7f7d2015c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f7d26c49700 1 -- 172.21.15.7:0/2642654867 >> 172.21.15.7:0/2642654867 conn(0x7f7d200c62e0 msgr2=0x7f7d200c3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7fcfa9569700 1 -- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf90068340 msgr2=0x7fcf9006a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7fcfa9569700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf90068340 0x7fcf9006a7f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcfa41e6740 tx=0x7fcf9c00f040 comp rx=0 tx=0).stop 2023-03-03T08:43:51.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7fcfa9569700 1 -- 172.21.15.7:0/3979726543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa4071990 msgr2=0x7fcfa414eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7fcfa9569700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa4071990 0x7fcfa414eff0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800bc40 tx=0x7fcf98004750 comp rx=0 tx=0).stop 2023-03-03T08:43:51.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7fcfa9569700 1 -- 172.21.15.7:0/3979726543 shutdown_connections 2023-03-03T08:43:51.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7fcfa9569700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfa40c1590 0x7fcfa40aa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7fcfa9569700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf90068340 0x7fcf9006a7f0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcfa41e6740 tx=0x7fcf9c00f040 comp rx=0 tx=0).stop 2023-03-03T08:43:51.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7fcfa9569700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfa4071990 0x7fcfa414eff0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7fcfa9569700 1 --2- 172.21.15.7:0/3979726543 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfa4151c00 0x7fcfa41e8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7fcfa9569700 1 -- 172.21.15.7:0/3979726543 >> 172.21.15.7:0/3979726543 conn(0x7fcfa40a94d0 msgr2=0x7fcfa40c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f91ad1f4700 1 -- 172.21.15.7:0/3495668231 shutdown_connections 2023-03-03T08:43:51.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f91ad1f4700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91a80bb390 0x7f91a80bcb50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f91ad1f4700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f919408a300 0x7f919408c7b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f91ad1f4700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91a80c35a0 0x7f91a80a74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f91ad1f4700 1 --2- 172.21.15.7:0/3495668231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91a8151ec0 0x7f91a80a7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.756+0000 7f91ad1f4700 1 -- 172.21.15.7:0/3495668231 >> 172.21.15.7:0/3495668231 conn(0x7f91a806a2b0 msgr2=0x7f91a81501d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7f91ad1f4700 1 -- 172.21.15.7:0/3495668231 shutdown_connections 2023-03-03T08:43:51.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.757+0000 7f91ad1f4700 1 -- 172.21.15.7:0/3495668231 wait complete. 2023-03-03T08:43:51.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.759+0000 7fcfa9569700 1 -- 172.21.15.7:0/3979726543 shutdown_connections 2023-03-03T08:43:51.764 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2023-03-03T08:43:51.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.760+0000 7fcfa9569700 1 -- 172.21.15.7:0/3979726543 wait complete. 2023-03-03T08:43:51.764 DEBUG:teuthology.parallel:result is None 2023-03-03T08:43:51.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.767+0000 7f7d26c49700 1 -- 172.21.15.7:0/2642654867 shutdown_connections 2023-03-03T08:43:51.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.767+0000 7f7d26c49700 1 -- 172.21.15.7:0/2642654867 wait complete. 2023-03-03T08:43:51.772 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2023-03-03T08:43:51.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.770+0000 7f4c88a23700 1 -- 172.21.15.7:0/2298091507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4c84074bb0 con 0x7f4c8406f0e0 2023-03-03T08:43:51.773 DEBUG:teuthology.parallel:result is None 2023-03-03T08:43:51.774 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.1 2023-03-03T08:43:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.772+0000 7f4c72ffd700 1 -- 172.21.15.7:0/2298091507 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4c74012ed0 con 0x7f4c8406f0e0 2023-03-03T08:43:51.774 DEBUG:teuthology.parallel:result is None 2023-03-03T08:43:51.774 INFO:teuthology.orchestra.run.smithi007.stdout:55834574857 2023-03-03T08:43:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.775+0000 7f4c88a23700 1 -- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c640682c0 msgr2=0x7f4c6406a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.775+0000 7f4c88a23700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c640682c0 0x7f4c6406a770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4c840c0b10 tx=0x7f4c7c009000 comp rx=0 tx=0).stop 2023-03-03T08:43:51.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.775+0000 7f4c88a23700 1 -- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c8406f0e0 msgr2=0x7f4c84071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.775+0000 7f4c88a23700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c8406f0e0 0x7f4c84071650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400d830 tx=0x7f4c7400db40 comp rx=0 tx=0).stop 2023-03-03T08:43:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.775+0000 7f4c88a23700 1 -- 172.21.15.7:0/2298091507 shutdown_connections 2023-03-03T08:43:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.775+0000 7f4c88a23700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c640682c0 0x7f4c6406a770 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4c840c0b10 tx=0x7f4c7c009000 comp rx=0 tx=0).stop 2023-03-03T08:43:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.775+0000 7f4c88a23700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c8414f9b0 0x7f4c84144a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.775+0000 7f4c88a23700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c8406f0e0 0x7f4c84071650 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.775+0000 7f4c88a23700 1 --2- 172.21.15.7:0/2298091507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c840abb20 0x7f4c84155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.775+0000 7f4c88a23700 1 -- 172.21.15.7:0/2298091507 >> 172.21.15.7:0/2298091507 conn(0x7f4c840c62e0 msgr2=0x7f4c84070860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.782 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.3 2023-03-03T08:43:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.777+0000 7f4c88a23700 1 -- 172.21.15.7:0/2298091507 shutdown_connections 2023-03-03T08:43:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.777+0000 7f4c88a23700 1 -- 172.21.15.7:0/2298091507 wait complete. 2023-03-03T08:43:51.783 DEBUG:teuthology.parallel:result is None 2023-03-03T08:43:51.784 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2023-03-03T08:43:51.785 DEBUG:teuthology.parallel:result is None 2023-03-03T08:43:51.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2023-03-03T08:43:51.786 DEBUG:teuthology.parallel:result is None 2023-03-03T08:43:51.789 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2023-03-03T08:43:51.790 DEBUG:teuthology.parallel:result is None 2023-03-03T08:43:51.793 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2023-03-03T08:43:51.793 DEBUG:teuthology.parallel:result is None 2023-03-03T08:43:51.794 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-03-03T08:43:51.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:43:51.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.955+0000 7fb042373700 1 Processor -- start 2023-03-03T08:43:51.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.955+0000 7fb042373700 1 -- start start 2023-03-03T08:43:51.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.956+0000 7fb042373700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c13a0f0 0x7fb03c0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.956+0000 7fb042373700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb03c0c1da0 0x7fb03c06d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.956+0000 7fb042373700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c06de90 con 0x7fb03c1586c0 2023-03-03T08:43:51.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.956+0000 7fb042373700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03c154840 con 0x7fb03c0c1da0 2023-03-03T08:43:51.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.956+0000 7fb042373700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c154560 con 0x7fb03c13a0f0 2023-03-03T08:43:51.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.956+0000 7fb03bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c13a0f0 0x7fb03c0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.956+0000 7fb040910700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb03c0c1da0 0x7fb03c06d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.956+0000 7fb03bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c13a0f0 0x7fb03c0c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59880/0 (socket says 172.21.15.7:59880) 2023-03-03T08:43:51.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.956+0000 7fb03bfff700 1 -- 172.21.15.7:0/3201175636 learned_addr learned my addr 172.21.15.7:0/3201175636 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.957+0000 7fb03bfff700 1 -- 172.21.15.7:0/3201175636 >> v2:172.21.15.143:3300/0 conn(0x7fb03c0c1da0 msgr2=0x7fb03c06d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:51.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.957+0000 7fb03bfff700 1 --2- 172.21.15.7:0/3201175636 >> v2:172.21.15.143:3300/0 conn(0x7fb03c0c1da0 0x7fb03c06d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.957+0000 7fb03bfff700 1 -- 172.21.15.7:0/3201175636 >> v1:172.21.15.143:6789/0 conn(0x7fb03c1586c0 legacy=0x7fb03c06d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.957+0000 7fb03bfff700 1 -- 172.21.15.7:0/3201175636 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c157be0 con 0x7fb03c13a0f0 2023-03-03T08:43:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.957+0000 7fb040910700 1 --2- 172.21.15.7:0/3201175636 >> v2:172.21.15.143:3300/0 conn(0x7fb03c0c1da0 0x7fb03c06d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.957+0000 7fb03bfff700 1 --2- 172.21.15.7:0/3201175636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c13a0f0 0x7fb03c0c07d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0098f0 tx=0x7fb02c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6da0ba2b6fbbcbc5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.957+0000 7fb03affd700 1 -- 172.21.15.7:0/3201175636 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c011830 con 0x7fb03c13a0f0 2023-03-03T08:43:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.957+0000 7fb03affd700 1 -- 172.21.15.7:0/3201175636 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02c011990 con 0x7fb03c13a0f0 2023-03-03T08:43:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.957+0000 7fb03affd700 1 -- 172.21.15.7:0/3201175636 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02c011d20 con 0x7fb03c13a0f0 2023-03-03T08:43:51.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.958+0000 7fb042373700 1 -- 172.21.15.7:0/3201175636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c13a0f0 msgr2=0x7fb03c0c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.958+0000 7fb042373700 1 --2- 172.21.15.7:0/3201175636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c13a0f0 0x7fb03c0c07d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0098f0 tx=0x7fb02c009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:51.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.958+0000 7fb042373700 1 -- 172.21.15.7:0/3201175636 shutdown_connections 2023-03-03T08:43:51.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.958+0000 7fb042373700 1 --2- 172.21.15.7:0/3201175636 >> v2:172.21.15.143:3300/0 conn(0x7fb03c0c1da0 0x7fb03c06d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.958+0000 7fb042373700 1 --2- 172.21.15.7:0/3201175636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c13a0f0 0x7fb03c0c07d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.958+0000 7fb042373700 1 -- 172.21.15.7:0/3201175636 >> 172.21.15.7:0/3201175636 conn(0x7fb03c06c110 msgr2=0x7fb03c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:51.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.958+0000 7fb042373700 1 -- 172.21.15.7:0/3201175636 shutdown_connections 2023-03-03T08:43:51.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.958+0000 7fb042373700 1 -- 172.21.15.7:0/3201175636 wait complete. 2023-03-03T08:43:51.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.959+0000 7fb042373700 1 Processor -- start 2023-03-03T08:43:51.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.959+0000 7fb042373700 1 -- start start 2023-03-03T08:43:51.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb042373700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb03c0c1da0 0x7fb03c136d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb042373700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb03c13a0f0 0x7fb03c1e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb042373700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c1586c0 0x7fb03c135d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb042373700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c1eaa50 con 0x7fb03c13a0f0 2023-03-03T08:43:51.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb042373700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c0c5330 con 0x7fb03c0c1da0 2023-03-03T08:43:51.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb042373700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c136a70 con 0x7fb03c1586c0 2023-03-03T08:43:51.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb03b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb03c13a0f0 0x7fb03c1e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb03b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb03c13a0f0 0x7fb03c1e9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51816/0 (socket says 172.21.15.7:51816) 2023-03-03T08:43:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb03b7fe700 1 -- 172.21.15.7:0/2762989381 learned_addr learned my addr 172.21.15.7:0/2762989381 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb03bfff700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb03c0c1da0 0x7fb03c136d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb040910700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c1586c0 0x7fb03c135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb03b7fe700 1 -- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c1586c0 msgr2=0x7fb03c135d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb03b7fe700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c1586c0 0x7fb03c135d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb03b7fe700 1 -- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb03c0c1da0 msgr2=0x7fb03c136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb03b7fe700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb03c0c1da0 0x7fb03c136d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb03b7fe700 1 -- 172.21.15.7:0/2762989381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb034009a70 con 0x7fb03c13a0f0 2023-03-03T08:43:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb03b7fe700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb03c13a0f0 0x7fb03c1e9dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb03000c4b0 tx=0x7fb03000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.960+0000 7fb038ff9700 1 -- 172.21.15.7:0/2762989381 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0300052d0 con 0x7fb03c13a0f0 2023-03-03T08:43:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.961+0000 7fb038ff9700 1 -- 172.21.15.7:0/2762989381 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb030005430 con 0x7fb03c13a0f0 2023-03-03T08:43:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.961+0000 7fb038ff9700 1 -- 172.21.15.7:0/2762989381 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0300038c0 con 0x7fb03c13a0f0 2023-03-03T08:43:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.961+0000 7fb042373700 1 -- 172.21.15.7:0/2762989381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02c009550 con 0x7fb03c13a0f0 2023-03-03T08:43:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.961+0000 7fb042373700 1 -- 172.21.15.7:0/2762989381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03c1581a0 con 0x7fb03c13a0f0 2023-03-03T08:43:51.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.961+0000 7fb042373700 1 -- 172.21.15.7:0/2762989381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03c157ec0 con 0x7fb03c13a0f0 2023-03-03T08:43:51.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.962+0000 7fb038ff9700 1 -- 172.21.15.7:0/2762989381 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb030019030 con 0x7fb03c13a0f0 2023-03-03T08:43:51.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.963+0000 7fb038ff9700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0280682f0 0x7fb02806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:51.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.963+0000 7fb038ff9700 1 -- 172.21.15.7:0/2762989381 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb0300853f0 con 0x7fb03c13a0f0 2023-03-03T08:43:51.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.963+0000 7fb03bfff700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0280682f0 0x7fb02806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:51.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.963+0000 7fb03bfff700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0280682f0 0x7fb02806a7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb02c014040 tx=0x7fb02c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:51.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:51.966+0000 7fb038ff9700 1 -- 172.21.15.7:0/2762989381 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb0300519c0 con 0x7fb03c13a0f0 2023-03-03T08:43:52.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.160+0000 7fb042373700 1 -- 172.21.15.7:0/2762989381 --> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb03c06e1a0 con 0x7fb0280682f0 2023-03-03T08:43:52.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.174+0000 7fb038ff9700 1 -- 172.21.15.7:0/2762989381 <== mgr.4102 v2:172.21.15.143:6832/96367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+524975 (secure 0 0 0) 0x7fb03c06e1a0 con 0x7fb0280682f0 2023-03-03T08:43:52.182 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:52.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 -- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0280682f0 msgr2=0x7fb02806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0280682f0 0x7fb02806a7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb02c014040 tx=0x7fb02c008030 comp rx=0 tx=0).stop 2023-03-03T08:43:52.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 -- 172.21.15.7:0/2762989381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb03c13a0f0 msgr2=0x7fb03c1e9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb03c13a0f0 0x7fb03c1e9dc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb03000c4b0 tx=0x7fb03000c7c0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 -- 172.21.15.7:0/2762989381 shutdown_connections 2023-03-03T08:43:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb03c1586c0 0x7fb03c135d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb03c0c1da0 0x7fb03c136d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0280682f0 0x7fb02806a7a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 --2- 172.21.15.7:0/2762989381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb03c13a0f0 0x7fb03c1e9dc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 -- 172.21.15.7:0/2762989381 >> 172.21.15.7:0/2762989381 conn(0x7fb03c06c110 msgr2=0x7fb03c072110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.178+0000 7fb042373700 1 -- 172.21.15.7:0/2762989381 shutdown_connections 2023-03-03T08:43:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.179+0000 7fb042373700 1 -- 172.21.15.7:0/2762989381 wait complete. 2023-03-03T08:43:52.187 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-03-03T08:43:52.193 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-03-03T08:43:51.109491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595028,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68052,"kb_used_data":3540,"kb_used_omap":0,"kb_used_meta":64448,"kb_avail":754906668,"statfs":{"total":773094113280,"available":773024428032,"internally_reserved":0,"allocated":3624960,"data_stored":2139729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65994752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.554300"},"pg_stats":[{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980166+0000","last_change":"2023-03-03T08:43:43.997091+0000","last_active":"2023-03-03T08:43:44.980166+0000","last_peered":"2023-03-03T08:43:44.980166+0000","last_clean":"2023-03-03T08:43:44.980166+0000","last_became_active":"2023-03-03T08:43:43.996805+0000","last_became_peered":"2023-03-03T08:43:43.996805+0000","last_unstale":"2023-03-03T08:43:44.980166+0000","last_undegraded":"2023-03-03T08:43:44.980166+0000","last_fullsized":"2023-03-03T08:43:44.980166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982505+0000","last_change":"2023-03-03T08:43:34.546828+0000","last_active":"2023-03-03T08:43:43.982505+0000","last_peered":"2023-03-03T08:43:43.982505+0000","last_clean":"2023-03-03T08:43:43.982505+0000","last_became_active":"2023-03-03T08:43:34.546559+0000","last_became_peered":"2023-03-03T08:43:34.546559+0000","last_unstale":"2023-03-03T08:43:43.982505+0000","last_undegraded":"2023-03-03T08:43:43.982505+0000","last_fullsized":"2023-03-03T08:43:43.982505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:21:14.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106061+0000","last_change":"2023-03-03T08:43:43.995073+0000","last_active":"2023-03-03T08:43:45.106061+0000","last_peered":"2023-03-03T08:43:45.106061+0000","last_clean":"2023-03-03T08:43:45.106061+0000","last_became_active":"2023-03-03T08:43:43.994797+0000","last_became_peered":"2023-03-03T08:43:43.994797+0000","last_unstale":"2023-03-03T08:43:45.106061+0000","last_undegraded":"2023-03-03T08:43:45.106061+0000","last_fullsized":"2023-03-03T08:43:45.106061+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:21:52.642936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982397+0000","last_change":"2023-03-03T08:43:34.552931+0000","last_active":"2023-03-03T08:43:43.982397+0000","last_peered":"2023-03-03T08:43:43.982397+0000","last_clean":"2023-03-03T08:43:43.982397+0000","last_became_active":"2023-03-03T08:43:34.552623+0000","last_became_peered":"2023-03-03T08:43:34.552623+0000","last_unstale":"2023-03-03T08:43:43.982397+0000","last_undegraded":"2023-03-03T08:43:43.982397+0000","last_fullsized":"2023-03-03T08:43:43.982397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:07:43.954897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979088+0000","last_change":"2023-03-03T08:43:43.999245+0000","last_active":"2023-03-03T08:43:44.979088+0000","last_peered":"2023-03-03T08:43:44.979088+0000","last_clean":"2023-03-03T08:43:44.979088+0000","last_became_active":"2023-03-03T08:43:43.998592+0000","last_became_peered":"2023-03-03T08:43:43.998592+0000","last_unstale":"2023-03-03T08:43:44.979088+0000","last_undegraded":"2023-03-03T08:43:44.979088+0000","last_fullsized":"2023-03-03T08:43:44.979088+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:23.788724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991054+0000","last_change":"2023-03-03T08:43:34.544408+0000","last_active":"2023-03-03T08:43:43.991054+0000","last_peered":"2023-03-03T08:43:43.991054+0000","last_clean":"2023-03-03T08:43:43.991054+0000","last_became_active":"2023-03-03T08:43:34.543977+0000","last_became_peered":"2023-03-03T08:43:34.543977+0000","last_unstale":"2023-03-03T08:43:43.991054+0000","last_undegraded":"2023-03-03T08:43:43.991054+0000","last_fullsized":"2023-03-03T08:43:43.991054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.074082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105254+0000","last_change":"2023-03-03T08:43:43.994545+0000","last_active":"2023-03-03T08:43:45.105254+0000","last_peered":"2023-03-03T08:43:45.105254+0000","last_clean":"2023-03-03T08:43:45.105254+0000","last_became_active":"2023-03-03T08:43:43.994271+0000","last_became_peered":"2023-03-03T08:43:43.994271+0000","last_unstale":"2023-03-03T08:43:45.105254+0000","last_undegraded":"2023-03-03T08:43:45.105254+0000","last_fullsized":"2023-03-03T08:43:45.105254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:15.925960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989064+0000","last_change":"2023-03-03T08:43:34.538786+0000","last_active":"2023-03-03T08:43:43.989064+0000","last_peered":"2023-03-03T08:43:43.989064+0000","last_clean":"2023-03-03T08:43:43.989064+0000","last_became_active":"2023-03-03T08:43:34.538494+0000","last_became_peered":"2023-03-03T08:43:34.538494+0000","last_unstale":"2023-03-03T08:43:43.989064+0000","last_undegraded":"2023-03-03T08:43:43.989064+0000","last_fullsized":"2023-03-03T08:43:43.989064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:53.838286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105391+0000","last_change":"2023-03-03T08:43:43.993678+0000","last_active":"2023-03-03T08:43:45.105391+0000","last_peered":"2023-03-03T08:43:45.105391+0000","last_clean":"2023-03-03T08:43:45.105391+0000","last_became_active":"2023-03-03T08:43:43.993402+0000","last_became_peered":"2023-03-03T08:43:43.993402+0000","last_unstale":"2023-03-03T08:43:45.105391+0000","last_undegraded":"2023-03-03T08:43:45.105391+0000","last_fullsized":"2023-03-03T08:43:45.105391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:31:00.747216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983479+0000","last_change":"2023-03-03T08:43:34.547197+0000","last_active":"2023-03-03T08:43:43.983479+0000","last_peered":"2023-03-03T08:43:43.983479+0000","last_clean":"2023-03-03T08:43:43.983479+0000","last_became_active":"2023-03-03T08:43:34.546927+0000","last_became_peered":"2023-03-03T08:43:34.546927+0000","last_unstale":"2023-03-03T08:43:43.983479+0000","last_undegraded":"2023-03-03T08:43:43.983479+0000","last_fullsized":"2023-03-03T08:43:43.983479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:21:34.845968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984318+0000","last_change":"2023-03-03T08:43:34.540812+0000","last_active":"2023-03-03T08:43:43.984318+0000","last_peered":"2023-03-03T08:43:43.984318+0000","last_clean":"2023-03-03T08:43:43.984318+0000","last_became_active":"2023-03-03T08:43:34.540549+0000","last_became_peered":"2023-03-03T08:43:34.540549+0000","last_unstale":"2023-03-03T08:43:43.984318+0000","last_undegraded":"2023-03-03T08:43:43.984318+0000","last_fullsized":"2023-03-03T08:43:43.984318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982875+0000","last_change":"2023-03-03T08:43:34.108426+0000","last_active":"2023-03-03T08:43:43.982875+0000","last_peered":"2023-03-03T08:43:43.982875+0000","last_clean":"2023-03-03T08:43:43.982875+0000","last_became_active":"2023-03-03T08:43:34.108009+0000","last_became_peered":"2023-03-03T08:43:34.108009+0000","last_unstale":"2023-03-03T08:43:43.982875+0000","last_undegraded":"2023-03-03T08:43:43.982875+0000","last_fullsized":"2023-03-03T08:43:43.982875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:53.632744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979479+0000","last_change":"2023-03-03T08:43:43.575708+0000","last_active":"2023-03-03T08:43:44.979479+0000","last_peered":"2023-03-03T08:43:44.979479+0000","last_clean":"2023-03-03T08:43:44.979479+0000","last_became_active":"2023-03-03T08:43:43.575417+0000","last_became_peered":"2023-03-03T08:43:43.575417+0000","last_unstale":"2023-03-03T08:43:44.979479+0000","last_undegraded":"2023-03-03T08:43:44.979479+0000","last_fullsized":"2023-03-03T08:43:44.979479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:25.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106018+0000","last_change":"2023-03-03T08:43:43.999274+0000","last_active":"2023-03-03T08:43:45.106018+0000","last_peered":"2023-03-03T08:43:45.106018+0000","last_clean":"2023-03-03T08:43:45.106018+0000","last_became_active":"2023-03-03T08:43:43.998823+0000","last_became_peered":"2023-03-03T08:43:43.998823+0000","last_unstale":"2023-03-03T08:43:45.106018+0000","last_undegraded":"2023-03-03T08:43:45.106018+0000","last_fullsized":"2023-03-03T08:43:45.106018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:48.799261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992918+0000","last_change":"2023-03-03T08:43:34.532482+0000","last_active":"2023-03-03T08:43:43.992918+0000","last_peered":"2023-03-03T08:43:43.992918+0000","last_clean":"2023-03-03T08:43:43.992918+0000","last_became_active":"2023-03-03T08:43:34.531751+0000","last_became_peered":"2023-03-03T08:43:34.531751+0000","last_unstale":"2023-03-03T08:43:43.992918+0000","last_undegraded":"2023-03-03T08:43:43.992918+0000","last_fullsized":"2023-03-03T08:43:43.992918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:47:45.214945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982691+0000","last_change":"2023-03-03T08:43:34.109782+0000","last_active":"2023-03-03T08:43:43.982691+0000","last_peered":"2023-03-03T08:43:43.982691+0000","last_clean":"2023-03-03T08:43:43.982691+0000","last_became_active":"2023-03-03T08:43:34.109513+0000","last_became_peered":"2023-03-03T08:43:34.109513+0000","last_unstale":"2023-03-03T08:43:43.982691+0000","last_undegraded":"2023-03-03T08:43:43.982691+0000","last_fullsized":"2023-03-03T08:43:43.982691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:16.185161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105634+0000","last_change":"2023-03-03T08:43:43.576885+0000","last_active":"2023-03-03T08:43:45.105634+0000","last_peered":"2023-03-03T08:43:45.105634+0000","last_clean":"2023-03-03T08:43:45.105634+0000","last_became_active":"2023-03-03T08:43:43.576577+0000","last_became_peered":"2023-03-03T08:43:43.576577+0000","last_unstale":"2023-03-03T08:43:45.105634+0000","last_undegraded":"2023-03-03T08:43:45.105634+0000","last_fullsized":"2023-03-03T08:43:45.105634+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:26.501956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978988+0000","last_change":"2023-03-03T08:43:43.992020+0000","last_active":"2023-03-03T08:43:44.978988+0000","last_peered":"2023-03-03T08:43:44.978988+0000","last_clean":"2023-03-03T08:43:44.978988+0000","last_became_active":"2023-03-03T08:43:43.991674+0000","last_became_peered":"2023-03-03T08:43:43.991674+0000","last_unstale":"2023-03-03T08:43:44.978988+0000","last_undegraded":"2023-03-03T08:43:44.978988+0000","last_fullsized":"2023-03-03T08:43:44.978988+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:54:30.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985919+0000","last_change":"2023-03-03T08:43:34.535451+0000","last_active":"2023-03-03T08:43:43.985919+0000","last_peered":"2023-03-03T08:43:43.985919+0000","last_clean":"2023-03-03T08:43:43.985919+0000","last_became_active":"2023-03-03T08:43:34.535129+0000","last_became_peered":"2023-03-03T08:43:34.535129+0000","last_unstale":"2023-03-03T08:43:43.985919+0000","last_undegraded":"2023-03-03T08:43:43.985919+0000","last_fullsized":"2023-03-03T08:43:43.985919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:30.645128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981082+0000","last_change":"2023-03-03T08:43:34.117851+0000","last_active":"2023-03-03T08:43:43.981082+0000","last_peered":"2023-03-03T08:43:43.981082+0000","last_clean":"2023-03-03T08:43:43.981082+0000","last_became_active":"2023-03-03T08:43:34.117548+0000","last_became_peered":"2023-03-03T08:43:34.117548+0000","last_unstale":"2023-03-03T08:43:43.981082+0000","last_undegraded":"2023-03-03T08:43:43.981082+0000","last_fullsized":"2023-03-03T08:43:43.981082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:20.039817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106904+0000","last_change":"2023-03-03T08:43:43.569082+0000","last_active":"2023-03-03T08:43:45.106904+0000","last_peered":"2023-03-03T08:43:45.106904+0000","last_clean":"2023-03-03T08:43:45.106904+0000","last_became_active":"2023-03-03T08:43:43.568838+0000","last_became_peered":"2023-03-03T08:43:43.568838+0000","last_unstale":"2023-03-03T08:43:45.106904+0000","last_undegraded":"2023-03-03T08:43:45.106904+0000","last_fullsized":"2023-03-03T08:43:45.106904+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:47.610848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979556+0000","last_change":"2023-03-03T08:43:43.995210+0000","last_active":"2023-03-03T08:43:44.979556+0000","last_peered":"2023-03-03T08:43:44.979556+0000","last_clean":"2023-03-03T08:43:44.979556+0000","last_became_active":"2023-03-03T08:43:43.994899+0000","last_became_peered":"2023-03-03T08:43:43.994899+0000","last_unstale":"2023-03-03T08:43:44.979556+0000","last_undegraded":"2023-03-03T08:43:44.979556+0000","last_fullsized":"2023-03-03T08:43:44.979556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:37:11.974933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985908+0000","last_change":"2023-03-03T08:43:34.547322+0000","last_active":"2023-03-03T08:43:43.985908+0000","last_peered":"2023-03-03T08:43:43.985908+0000","last_clean":"2023-03-03T08:43:43.985908+0000","last_became_active":"2023-03-03T08:43:34.546692+0000","last_became_peered":"2023-03-03T08:43:34.546692+0000","last_unstale":"2023-03-03T08:43:43.985908+0000","last_undegraded":"2023-03-03T08:43:43.985908+0000","last_fullsized":"2023-03-03T08:43:43.985908+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:11:26.366920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982836+0000","last_change":"2023-03-03T08:43:34.110187+0000","last_active":"2023-03-03T08:43:43.982836+0000","last_peered":"2023-03-03T08:43:43.982836+0000","last_clean":"2023-03-03T08:43:43.982836+0000","last_became_active":"2023-03-03T08:43:34.109845+0000","last_became_peered":"2023-03-03T08:43:34.109845+0000","last_unstale":"2023-03-03T08:43:43.982836+0000","last_undegraded":"2023-03-03T08:43:43.982836+0000","last_fullsized":"2023-03-03T08:43:43.982836+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:33.507500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105532+0000","last_change":"2023-03-03T08:43:43.576805+0000","last_active":"2023-03-03T08:43:45.105532+0000","last_peered":"2023-03-03T08:43:45.105532+0000","last_clean":"2023-03-03T08:43:45.105532+0000","last_became_active":"2023-03-03T08:43:43.576241+0000","last_became_peered":"2023-03-03T08:43:43.576241+0000","last_unstale":"2023-03-03T08:43:45.105532+0000","last_undegraded":"2023-03-03T08:43:45.105532+0000","last_fullsized":"2023-03-03T08:43:45.105532+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:34:35.821307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105632+0000","last_change":"2023-03-03T08:43:43.991142+0000","last_active":"2023-03-03T08:43:45.105632+0000","last_peered":"2023-03-03T08:43:45.105632+0000","last_clean":"2023-03-03T08:43:45.105632+0000","last_became_active":"2023-03-03T08:43:43.990868+0000","last_became_peered":"2023-03-03T08:43:43.990868+0000","last_unstale":"2023-03-03T08:43:45.105632+0000","last_undegraded":"2023-03-03T08:43:45.105632+0000","last_fullsized":"2023-03-03T08:43:45.105632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:58.831806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106429+0000","last_change":"2023-03-03T08:43:43.995618+0000","last_active":"2023-03-03T08:43:45.106429+0000","last_peered":"2023-03-03T08:43:45.106429+0000","last_clean":"2023-03-03T08:43:45.106429+0000","last_became_active":"2023-03-03T08:43:43.995350+0000","last_became_peered":"2023-03-03T08:43:43.995350+0000","last_unstale":"2023-03-03T08:43:45.106429+0000","last_undegraded":"2023-03-03T08:43:45.106429+0000","last_fullsized":"2023-03-03T08:43:45.106429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:46.445223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988843+0000","last_change":"2023-03-03T08:43:34.541172+0000","last_active":"2023-03-03T08:43:43.988843+0000","last_peered":"2023-03-03T08:43:43.988843+0000","last_clean":"2023-03-03T08:43:43.988843+0000","last_became_active":"2023-03-03T08:43:34.540739+0000","last_became_peered":"2023-03-03T08:43:34.540739+0000","last_unstale":"2023-03-03T08:43:43.988843+0000","last_undegraded":"2023-03-03T08:43:43.988843+0000","last_fullsized":"2023-03-03T08:43:43.988843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:40.058265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987475+0000","last_change":"2023-03-03T08:43:34.112667+0000","last_active":"2023-03-03T08:43:43.987475+0000","last_peered":"2023-03-03T08:43:43.987475+0000","last_clean":"2023-03-03T08:43:43.987475+0000","last_became_active":"2023-03-03T08:43:34.112386+0000","last_became_peered":"2023-03-03T08:43:34.112386+0000","last_unstale":"2023-03-03T08:43:43.987475+0000","last_undegraded":"2023-03-03T08:43:43.987475+0000","last_fullsized":"2023-03-03T08:43:43.987475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:40.504392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980167+0000","last_change":"2023-03-03T08:43:43.573962+0000","last_active":"2023-03-03T08:43:44.980167+0000","last_peered":"2023-03-03T08:43:44.980167+0000","last_clean":"2023-03-03T08:43:44.980167+0000","last_became_active":"2023-03-03T08:43:43.573200+0000","last_became_peered":"2023-03-03T08:43:43.573200+0000","last_unstale":"2023-03-03T08:43:44.980167+0000","last_undegraded":"2023-03-03T08:43:44.980167+0000","last_fullsized":"2023-03-03T08:43:44.980167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:29.511681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984698+0000","last_change":"2023-03-03T08:43:34.537850+0000","last_active":"2023-03-03T08:43:43.984698+0000","last_peered":"2023-03-03T08:43:43.984698+0000","last_clean":"2023-03-03T08:43:43.984698+0000","last_became_active":"2023-03-03T08:43:34.537550+0000","last_became_peered":"2023-03-03T08:43:34.537550+0000","last_unstale":"2023-03-03T08:43:43.984698+0000","last_undegraded":"2023-03-03T08:43:43.984698+0000","last_fullsized":"2023-03-03T08:43:43.984698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:37:36.893757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990299+0000","last_change":"2023-03-03T08:43:34.109127+0000","last_active":"2023-03-03T08:43:43.990299+0000","last_peered":"2023-03-03T08:43:43.990299+0000","last_clean":"2023-03-03T08:43:43.990299+0000","last_became_active":"2023-03-03T08:43:34.108869+0000","last_became_peered":"2023-03-03T08:43:34.108869+0000","last_unstale":"2023-03-03T08:43:43.990299+0000","last_undegraded":"2023-03-03T08:43:43.990299+0000","last_fullsized":"2023-03-03T08:43:43.990299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:19:00.275620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981298+0000","last_change":"2023-03-03T08:43:29.354627+0000","last_active":"2023-03-03T08:43:43.981298+0000","last_peered":"2023-03-03T08:43:43.981298+0000","last_clean":"2023-03-03T08:43:43.981298+0000","last_became_active":"2023-03-03T08:43:29.354271+0000","last_became_peered":"2023-03-03T08:43:29.354271+0000","last_unstale":"2023-03-03T08:43:43.981298+0000","last_undegraded":"2023-03-03T08:43:43.981298+0000","last_fullsized":"2023-03-03T08:43:43.981298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:05.946053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979432+0000","last_change":"2023-03-03T08:43:43.575567+0000","last_active":"2023-03-03T08:43:44.979432+0000","last_peered":"2023-03-03T08:43:44.979432+0000","last_clean":"2023-03-03T08:43:44.979432+0000","last_became_active":"2023-03-03T08:43:43.575276+0000","last_became_peered":"2023-03-03T08:43:43.575276+0000","last_unstale":"2023-03-03T08:43:44.979432+0000","last_undegraded":"2023-03-03T08:43:44.979432+0000","last_fullsized":"2023-03-03T08:43:44.979432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:12.876790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107250+0000","last_change":"2023-03-03T08:43:43.998276+0000","last_active":"2023-03-03T08:43:45.107250+0000","last_peered":"2023-03-03T08:43:45.107250+0000","last_clean":"2023-03-03T08:43:45.107250+0000","last_became_active":"2023-03-03T08:43:43.996018+0000","last_became_peered":"2023-03-03T08:43:43.996018+0000","last_unstale":"2023-03-03T08:43:45.107250+0000","last_undegraded":"2023-03-03T08:43:45.107250+0000","last_fullsized":"2023-03-03T08:43:45.107250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:08.588278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988844+0000","last_change":"2023-03-03T08:43:34.539018+0000","last_active":"2023-03-03T08:43:43.988844+0000","last_peered":"2023-03-03T08:43:43.988844+0000","last_clean":"2023-03-03T08:43:43.988844+0000","last_became_active":"2023-03-03T08:43:34.538375+0000","last_became_peered":"2023-03-03T08:43:34.538375+0000","last_unstale":"2023-03-03T08:43:43.988844+0000","last_undegraded":"2023-03-03T08:43:43.988844+0000","last_fullsized":"2023-03-03T08:43:43.988844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:29:02.601410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990482+0000","last_change":"2023-03-03T08:43:29.354519+0000","last_active":"2023-03-03T08:43:43.990482+0000","last_peered":"2023-03-03T08:43:43.990482+0000","last_clean":"2023-03-03T08:43:43.990482+0000","last_became_active":"2023-03-03T08:43:29.354213+0000","last_became_peered":"2023-03-03T08:43:29.354213+0000","last_unstale":"2023-03-03T08:43:43.990482+0000","last_undegraded":"2023-03-03T08:43:43.990482+0000","last_fullsized":"2023-03-03T08:43:43.990482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:00.406507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990028+0000","last_change":"2023-03-03T08:43:34.110105+0000","last_active":"2023-03-03T08:43:43.990028+0000","last_peered":"2023-03-03T08:43:43.990028+0000","last_clean":"2023-03-03T08:43:43.990028+0000","last_became_active":"2023-03-03T08:43:34.109736+0000","last_became_peered":"2023-03-03T08:43:34.109736+0000","last_unstale":"2023-03-03T08:43:43.990028+0000","last_undegraded":"2023-03-03T08:43:43.990028+0000","last_fullsized":"2023-03-03T08:43:43.990028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:30.557847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105361+0000","last_change":"2023-03-03T08:43:43.569433+0000","last_active":"2023-03-03T08:43:45.105361+0000","last_peered":"2023-03-03T08:43:45.105361+0000","last_clean":"2023-03-03T08:43:45.105361+0000","last_became_active":"2023-03-03T08:43:43.568711+0000","last_became_peered":"2023-03-03T08:43:43.568711+0000","last_unstale":"2023-03-03T08:43:45.105361+0000","last_undegraded":"2023-03-03T08:43:45.105361+0000","last_fullsized":"2023-03-03T08:43:45.105361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:53.170557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105548+0000","last_change":"2023-03-03T08:43:43.999241+0000","last_active":"2023-03-03T08:43:45.105548+0000","last_peered":"2023-03-03T08:43:45.105548+0000","last_clean":"2023-03-03T08:43:45.105548+0000","last_became_active":"2023-03-03T08:43:43.998992+0000","last_became_peered":"2023-03-03T08:43:43.998992+0000","last_unstale":"2023-03-03T08:43:45.105548+0000","last_undegraded":"2023-03-03T08:43:45.105548+0000","last_fullsized":"2023-03-03T08:43:45.105548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:22.623700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991092+0000","last_change":"2023-03-03T08:43:34.535435+0000","last_active":"2023-03-03T08:43:43.991092+0000","last_peered":"2023-03-03T08:43:43.991092+0000","last_clean":"2023-03-03T08:43:43.991092+0000","last_became_active":"2023-03-03T08:43:34.535123+0000","last_became_peered":"2023-03-03T08:43:34.535123+0000","last_unstale":"2023-03-03T08:43:43.991092+0000","last_undegraded":"2023-03-03T08:43:43.991092+0000","last_fullsized":"2023-03-03T08:43:43.991092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:52:10.622422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990432+0000","last_change":"2023-03-03T08:43:34.107783+0000","last_active":"2023-03-03T08:43:43.990432+0000","last_peered":"2023-03-03T08:43:43.990432+0000","last_clean":"2023-03-03T08:43:43.990432+0000","last_became_active":"2023-03-03T08:43:34.107503+0000","last_became_peered":"2023-03-03T08:43:34.107503+0000","last_unstale":"2023-03-03T08:43:43.990432+0000","last_undegraded":"2023-03-03T08:43:43.990432+0000","last_fullsized":"2023-03-03T08:43:43.990432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:35:42.800814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981038+0000","last_change":"2023-03-03T08:43:29.072304+0000","last_active":"2023-03-03T08:43:43.981038+0000","last_peered":"2023-03-03T08:43:43.981038+0000","last_clean":"2023-03-03T08:43:43.981038+0000","last_became_active":"2023-03-03T08:43:29.071962+0000","last_became_peered":"2023-03-03T08:43:29.071962+0000","last_unstale":"2023-03-03T08:43:43.981038+0000","last_undegraded":"2023-03-03T08:43:43.981038+0000","last_fullsized":"2023-03-03T08:43:43.981038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:40:12.229226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106349+0000","last_change":"2023-03-03T08:43:43.575964+0000","last_active":"2023-03-03T08:43:45.106349+0000","last_peered":"2023-03-03T08:43:45.106349+0000","last_clean":"2023-03-03T08:43:45.106349+0000","last_became_active":"2023-03-03T08:43:43.575697+0000","last_became_peered":"2023-03-03T08:43:43.575697+0000","last_unstale":"2023-03-03T08:43:45.106349+0000","last_undegraded":"2023-03-03T08:43:45.106349+0000","last_fullsized":"2023-03-03T08:43:45.106349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:01.896050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978782+0000","last_change":"2023-03-03T08:43:44.006414+0000","last_active":"2023-03-03T08:43:44.978782+0000","last_peered":"2023-03-03T08:43:44.978782+0000","last_clean":"2023-03-03T08:43:44.978782+0000","last_became_active":"2023-03-03T08:43:44.006132+0000","last_became_peered":"2023-03-03T08:43:44.006132+0000","last_unstale":"2023-03-03T08:43:44.978782+0000","last_undegraded":"2023-03-03T08:43:44.978782+0000","last_fullsized":"2023-03-03T08:43:44.978782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:55.006419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982772+0000","last_change":"2023-03-03T08:43:34.544053+0000","last_active":"2023-03-03T08:43:43.982772+0000","last_peered":"2023-03-03T08:43:43.982772+0000","last_clean":"2023-03-03T08:43:43.982772+0000","last_became_active":"2023-03-03T08:43:34.543523+0000","last_became_peered":"2023-03-03T08:43:34.543523+0000","last_unstale":"2023-03-03T08:43:43.982772+0000","last_undegraded":"2023-03-03T08:43:43.982772+0000","last_fullsized":"2023-03-03T08:43:43.982772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:34:03.662535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989069+0000","last_change":"2023-03-03T08:43:34.104492+0000","last_active":"2023-03-03T08:43:43.989069+0000","last_peered":"2023-03-03T08:43:43.989069+0000","last_clean":"2023-03-03T08:43:43.989069+0000","last_became_active":"2023-03-03T08:43:34.104151+0000","last_became_peered":"2023-03-03T08:43:34.104151+0000","last_unstale":"2023-03-03T08:43:43.989069+0000","last_undegraded":"2023-03-03T08:43:43.989069+0000","last_fullsized":"2023-03-03T08:43:43.989069+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:57.214738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979395+0000","last_change":"2023-03-03T08:43:29.075836+0000","last_active":"2023-03-03T08:43:43.979395+0000","last_peered":"2023-03-03T08:43:43.979395+0000","last_clean":"2023-03-03T08:43:43.979395+0000","last_became_active":"2023-03-03T08:43:29.075564+0000","last_became_peered":"2023-03-03T08:43:29.075564+0000","last_unstale":"2023-03-03T08:43:43.979395+0000","last_undegraded":"2023-03-03T08:43:43.979395+0000","last_fullsized":"2023-03-03T08:43:43.979395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:39.576714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107224+0000","last_change":"2023-03-03T08:43:43.575599+0000","last_active":"2023-03-03T08:43:45.107224+0000","last_peered":"2023-03-03T08:43:45.107224+0000","last_clean":"2023-03-03T08:43:45.107224+0000","last_became_active":"2023-03-03T08:43:43.575318+0000","last_became_peered":"2023-03-03T08:43:43.575318+0000","last_unstale":"2023-03-03T08:43:45.107224+0000","last_undegraded":"2023-03-03T08:43:45.107224+0000","last_fullsized":"2023-03-03T08:43:45.107224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:28.806052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105269+0000","last_change":"2023-03-03T08:43:44.000500+0000","last_active":"2023-03-03T08:43:45.105269+0000","last_peered":"2023-03-03T08:43:45.105269+0000","last_clean":"2023-03-03T08:43:45.105269+0000","last_became_active":"2023-03-03T08:43:43.999560+0000","last_became_peered":"2023-03-03T08:43:43.999560+0000","last_unstale":"2023-03-03T08:43:45.105269+0000","last_undegraded":"2023-03-03T08:43:45.105269+0000","last_fullsized":"2023-03-03T08:43:45.105269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:29:54.240309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981172+0000","last_change":"2023-03-03T08:43:34.550263+0000","last_active":"2023-03-03T08:43:43.981172+0000","last_peered":"2023-03-03T08:43:43.981172+0000","last_clean":"2023-03-03T08:43:43.981172+0000","last_became_active":"2023-03-03T08:43:34.549560+0000","last_became_peered":"2023-03-03T08:43:34.549560+0000","last_unstale":"2023-03-03T08:43:43.981172+0000","last_undegraded":"2023-03-03T08:43:43.981172+0000","last_fullsized":"2023-03-03T08:43:43.981172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:55.874223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987062+0000","last_change":"2023-03-03T08:43:34.110484+0000","last_active":"2023-03-03T08:43:43.987062+0000","last_peered":"2023-03-03T08:43:43.987062+0000","last_clean":"2023-03-03T08:43:43.987062+0000","last_became_active":"2023-03-03T08:43:34.110220+0000","last_became_peered":"2023-03-03T08:43:34.110220+0000","last_unstale":"2023-03-03T08:43:43.987062+0000","last_undegraded":"2023-03-03T08:43:43.987062+0000","last_fullsized":"2023-03-03T08:43:43.987062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:04:59.232537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982439+0000","last_change":"2023-03-03T08:43:23.015460+0000","last_active":"2023-03-03T08:43:43.982439+0000","last_peered":"2023-03-03T08:43:43.982439+0000","last_clean":"2023-03-03T08:43:43.982439+0000","last_became_active":"2023-03-03T08:43:23.014485+0000","last_became_peered":"2023-03-03T08:43:23.014485+0000","last_unstale":"2023-03-03T08:43:43.982439+0000","last_undegraded":"2023-03-03T08:43:43.982439+0000","last_fullsized":"2023-03-03T08:43:43.982439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_clean_scrub_stamp":"2023-03-03T08:43:21.542883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:07.022017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980803+0000","last_change":"2023-03-03T08:43:29.160285+0000","last_active":"2023-03-03T08:43:43.980803+0000","last_peered":"2023-03-03T08:43:43.980803+0000","last_clean":"2023-03-03T08:43:43.980803+0000","last_became_active":"2023-03-03T08:43:29.159943+0000","last_became_peered":"2023-03-03T08:43:29.159943+0000","last_unstale":"2023-03-03T08:43:43.980803+0000","last_undegraded":"2023-03-03T08:43:43.980803+0000","last_fullsized":"2023-03-03T08:43:43.980803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:48:58.997356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107580+0000","last_change":"2023-03-03T08:43:43.575212+0000","last_active":"2023-03-03T08:43:45.107580+0000","last_peered":"2023-03-03T08:43:45.107580+0000","last_clean":"2023-03-03T08:43:45.107580+0000","last_became_active":"2023-03-03T08:43:43.574897+0000","last_became_peered":"2023-03-03T08:43:43.574897+0000","last_unstale":"2023-03-03T08:43:45.107580+0000","last_undegraded":"2023-03-03T08:43:45.107580+0000","last_fullsized":"2023-03-03T08:43:45.107580+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:33:07.674313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107370+0000","last_change":"2023-03-03T08:43:43.992952+0000","last_active":"2023-03-03T08:43:45.107370+0000","last_peered":"2023-03-03T08:43:45.107370+0000","last_clean":"2023-03-03T08:43:45.107370+0000","last_became_active":"2023-03-03T08:43:43.992535+0000","last_became_peered":"2023-03-03T08:43:43.992535+0000","last_unstale":"2023-03-03T08:43:45.107370+0000","last_undegraded":"2023-03-03T08:43:45.107370+0000","last_fullsized":"2023-03-03T08:43:45.107370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:10:41.617430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991335+0000","last_change":"2023-03-03T08:43:34.536419+0000","last_active":"2023-03-03T08:43:43.991335+0000","last_peered":"2023-03-03T08:43:43.991335+0000","last_clean":"2023-03-03T08:43:43.991335+0000","last_became_active":"2023-03-03T08:43:34.536150+0000","last_became_peered":"2023-03-03T08:43:34.536150+0000","last_unstale":"2023-03-03T08:43:43.991335+0000","last_undegraded":"2023-03-03T08:43:43.991335+0000","last_fullsized":"2023-03-03T08:43:43.991335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:22:54.362024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981827+0000","last_change":"2023-03-03T08:43:34.106337+0000","last_active":"2023-03-03T08:43:43.981827+0000","last_peered":"2023-03-03T08:43:43.981827+0000","last_clean":"2023-03-03T08:43:43.981827+0000","last_became_active":"2023-03-03T08:43:34.105753+0000","last_became_peered":"2023-03-03T08:43:34.105753+0000","last_unstale":"2023-03-03T08:43:43.981827+0000","last_undegraded":"2023-03-03T08:43:43.981827+0000","last_fullsized":"2023-03-03T08:43:43.981827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:46.761924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991590+0000","last_change":"2023-03-03T08:43:29.076367+0000","last_active":"2023-03-03T08:43:43.991590+0000","last_peered":"2023-03-03T08:43:43.991590+0000","last_clean":"2023-03-03T08:43:43.991590+0000","last_became_active":"2023-03-03T08:43:29.075982+0000","last_became_peered":"2023-03-03T08:43:29.075982+0000","last_unstale":"2023-03-03T08:43:43.991590+0000","last_undegraded":"2023-03-03T08:43:43.991590+0000","last_fullsized":"2023-03-03T08:43:43.991590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:06.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:46.143717+0000","last_change":"2023-03-03T08:43:43.577670+0000","last_active":"2023-03-03T08:43:46.143717+0000","last_peered":"2023-03-03T08:43:46.143717+0000","last_clean":"2023-03-03T08:43:46.143717+0000","last_became_active":"2023-03-03T08:43:43.577371+0000","last_became_peered":"2023-03-03T08:43:43.577371+0000","last_unstale":"2023-03-03T08:43:46.143717+0000","last_undegraded":"2023-03-03T08:43:46.143717+0000","last_fullsized":"2023-03-03T08:43:46.143717+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:18.377654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108041+0000","last_change":"2023-03-03T08:43:43.993077+0000","last_active":"2023-03-03T08:43:45.108041+0000","last_peered":"2023-03-03T08:43:45.108041+0000","last_clean":"2023-03-03T08:43:45.108041+0000","last_became_active":"2023-03-03T08:43:43.992021+0000","last_became_peered":"2023-03-03T08:43:43.992021+0000","last_unstale":"2023-03-03T08:43:45.108041+0000","last_undegraded":"2023-03-03T08:43:45.108041+0000","last_fullsized":"2023-03-03T08:43:45.108041+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:58.237553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990193+0000","last_change":"2023-03-03T08:43:34.528593+0000","last_active":"2023-03-03T08:43:43.990193+0000","last_peered":"2023-03-03T08:43:43.990193+0000","last_clean":"2023-03-03T08:43:43.990193+0000","last_became_active":"2023-03-03T08:43:34.528260+0000","last_became_peered":"2023-03-03T08:43:34.528260+0000","last_unstale":"2023-03-03T08:43:43.990193+0000","last_undegraded":"2023-03-03T08:43:43.990193+0000","last_fullsized":"2023-03-03T08:43:43.990193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988924+0000","last_change":"2023-03-03T08:43:34.113748+0000","last_active":"2023-03-03T08:43:43.988924+0000","last_peered":"2023-03-03T08:43:43.988924+0000","last_clean":"2023-03-03T08:43:43.988924+0000","last_became_active":"2023-03-03T08:43:34.113190+0000","last_became_peered":"2023-03-03T08:43:34.113190+0000","last_unstale":"2023-03-03T08:43:43.988924+0000","last_undegraded":"2023-03-03T08:43:43.988924+0000","last_fullsized":"2023-03-03T08:43:43.988924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:58:55.343571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980485+0000","last_change":"2023-03-03T08:43:29.107584+0000","last_active":"2023-03-03T08:43:43.980485+0000","last_peered":"2023-03-03T08:43:43.980485+0000","last_clean":"2023-03-03T08:43:43.980485+0000","last_became_active":"2023-03-03T08:43:29.107208+0000","last_became_peered":"2023-03-03T08:43:29.107208+0000","last_unstale":"2023-03-03T08:43:43.980485+0000","last_undegraded":"2023-03-03T08:43:43.980485+0000","last_fullsized":"2023-03-03T08:43:43.980485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:28:24.087122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106004+0000","last_change":"2023-03-03T08:43:43.575742+0000","last_active":"2023-03-03T08:43:45.106004+0000","last_peered":"2023-03-03T08:43:45.106004+0000","last_clean":"2023-03-03T08:43:45.106004+0000","last_became_active":"2023-03-03T08:43:43.575456+0000","last_became_peered":"2023-03-03T08:43:43.575456+0000","last_unstale":"2023-03-03T08:43:45.106004+0000","last_undegraded":"2023-03-03T08:43:45.106004+0000","last_fullsized":"2023-03-03T08:43:45.106004+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:46:17.635306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106635+0000","last_change":"2023-03-03T08:43:43.985961+0000","last_active":"2023-03-03T08:43:45.106635+0000","last_peered":"2023-03-03T08:43:45.106635+0000","last_clean":"2023-03-03T08:43:45.106635+0000","last_became_active":"2023-03-03T08:43:43.985633+0000","last_became_peered":"2023-03-03T08:43:43.985633+0000","last_unstale":"2023-03-03T08:43:45.106635+0000","last_undegraded":"2023-03-03T08:43:45.106635+0000","last_fullsized":"2023-03-03T08:43:45.106635+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:52.517910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989789+0000","last_change":"2023-03-03T08:43:34.526797+0000","last_active":"2023-03-03T08:43:43.989789+0000","last_peered":"2023-03-03T08:43:43.989789+0000","last_clean":"2023-03-03T08:43:43.989789+0000","last_became_active":"2023-03-03T08:43:34.526490+0000","last_became_peered":"2023-03-03T08:43:34.526490+0000","last_unstale":"2023-03-03T08:43:43.989789+0000","last_undegraded":"2023-03-03T08:43:43.989789+0000","last_fullsized":"2023-03-03T08:43:43.989789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:11.808336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986004+0000","last_change":"2023-03-03T08:43:34.104692+0000","last_active":"2023-03-03T08:43:43.986004+0000","last_peered":"2023-03-03T08:43:43.986004+0000","last_clean":"2023-03-03T08:43:43.986004+0000","last_became_active":"2023-03-03T08:43:34.104263+0000","last_became_peered":"2023-03-03T08:43:34.104263+0000","last_unstale":"2023-03-03T08:43:43.986004+0000","last_undegraded":"2023-03-03T08:43:43.986004+0000","last_fullsized":"2023-03-03T08:43:43.986004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:24.537703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983049+0000","last_change":"2023-03-03T08:43:29.107536+0000","last_active":"2023-03-03T08:43:43.983049+0000","last_peered":"2023-03-03T08:43:43.983049+0000","last_clean":"2023-03-03T08:43:43.983049+0000","last_became_active":"2023-03-03T08:43:29.107093+0000","last_became_peered":"2023-03-03T08:43:29.107093+0000","last_unstale":"2023-03-03T08:43:43.983049+0000","last_undegraded":"2023-03-03T08:43:43.983049+0000","last_fullsized":"2023-03-03T08:43:43.983049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:25:47.568904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980025+0000","last_change":"2023-03-03T08:43:43.577221+0000","last_active":"2023-03-03T08:43:44.980025+0000","last_peered":"2023-03-03T08:43:44.980025+0000","last_clean":"2023-03-03T08:43:44.980025+0000","last_became_active":"2023-03-03T08:43:43.576942+0000","last_became_peered":"2023-03-03T08:43:43.576942+0000","last_unstale":"2023-03-03T08:43:44.980025+0000","last_undegraded":"2023-03-03T08:43:44.980025+0000","last_fullsized":"2023-03-03T08:43:44.980025+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:13:09.716078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105102+0000","last_change":"2023-03-03T08:43:44.000318+0000","last_active":"2023-03-03T08:43:45.105102+0000","last_peered":"2023-03-03T08:43:45.105102+0000","last_clean":"2023-03-03T08:43:45.105102+0000","last_became_active":"2023-03-03T08:43:44.000053+0000","last_became_peered":"2023-03-03T08:43:44.000053+0000","last_unstale":"2023-03-03T08:43:45.105102+0000","last_undegraded":"2023-03-03T08:43:45.105102+0000","last_fullsized":"2023-03-03T08:43:45.105102+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:31.059774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994113+0000","last_change":"2023-03-03T08:43:34.540006+0000","last_active":"2023-03-03T08:43:43.994113+0000","last_peered":"2023-03-03T08:43:43.994113+0000","last_clean":"2023-03-03T08:43:43.994113+0000","last_became_active":"2023-03-03T08:43:34.539579+0000","last_became_peered":"2023-03-03T08:43:34.539579+0000","last_unstale":"2023-03-03T08:43:43.994113+0000","last_undegraded":"2023-03-03T08:43:43.994113+0000","last_fullsized":"2023-03-03T08:43:43.994113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988985+0000","last_change":"2023-03-03T08:43:34.109764+0000","last_active":"2023-03-03T08:43:43.988985+0000","last_peered":"2023-03-03T08:43:43.988985+0000","last_clean":"2023-03-03T08:43:43.988985+0000","last_became_active":"2023-03-03T08:43:34.109462+0000","last_became_peered":"2023-03-03T08:43:34.109462+0000","last_unstale":"2023-03-03T08:43:43.988985+0000","last_undegraded":"2023-03-03T08:43:43.988985+0000","last_fullsized":"2023-03-03T08:43:43.988985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:56.399192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106570+0000","last_change":"2023-03-03T08:43:43.576402+0000","last_active":"2023-03-03T08:43:45.106570+0000","last_peered":"2023-03-03T08:43:45.106570+0000","last_clean":"2023-03-03T08:43:45.106570+0000","last_became_active":"2023-03-03T08:43:43.576113+0000","last_became_peered":"2023-03-03T08:43:43.576113+0000","last_unstale":"2023-03-03T08:43:45.106570+0000","last_undegraded":"2023-03-03T08:43:45.106570+0000","last_fullsized":"2023-03-03T08:43:45.106570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:33:26.483688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106437+0000","last_change":"2023-03-03T08:43:44.002074+0000","last_active":"2023-03-03T08:43:45.106437+0000","last_peered":"2023-03-03T08:43:45.106437+0000","last_clean":"2023-03-03T08:43:45.106437+0000","last_became_active":"2023-03-03T08:43:44.001803+0000","last_became_peered":"2023-03-03T08:43:44.001803+0000","last_unstale":"2023-03-03T08:43:45.106437+0000","last_undegraded":"2023-03-03T08:43:45.106437+0000","last_fullsized":"2023-03-03T08:43:45.106437+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:15:16.115135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986358+0000","last_change":"2023-03-03T08:43:34.541858+0000","last_active":"2023-03-03T08:43:43.986358+0000","last_peered":"2023-03-03T08:43:43.986358+0000","last_clean":"2023-03-03T08:43:43.986358+0000","last_became_active":"2023-03-03T08:43:34.541430+0000","last_became_peered":"2023-03-03T08:43:34.541430+0000","last_unstale":"2023-03-03T08:43:43.986358+0000","last_undegraded":"2023-03-03T08:43:43.986358+0000","last_fullsized":"2023-03-03T08:43:43.986358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:29.397784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979830+0000","last_change":"2023-03-03T08:43:34.111335+0000","last_active":"2023-03-03T08:43:43.979830+0000","last_peered":"2023-03-03T08:43:43.979830+0000","last_clean":"2023-03-03T08:43:43.979830+0000","last_became_active":"2023-03-03T08:43:34.110837+0000","last_became_peered":"2023-03-03T08:43:34.110837+0000","last_unstale":"2023-03-03T08:43:43.979830+0000","last_undegraded":"2023-03-03T08:43:43.979830+0000","last_fullsized":"2023-03-03T08:43:43.979830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:03.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106574+0000","last_change":"2023-03-03T08:43:43.577539+0000","last_active":"2023-03-03T08:43:45.106574+0000","last_peered":"2023-03-03T08:43:45.106574+0000","last_clean":"2023-03-03T08:43:45.106574+0000","last_became_active":"2023-03-03T08:43:43.577209+0000","last_became_peered":"2023-03-03T08:43:43.577209+0000","last_unstale":"2023-03-03T08:43:45.106574+0000","last_undegraded":"2023-03-03T08:43:45.106574+0000","last_fullsized":"2023-03-03T08:43:45.106574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:31:01.430495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979296+0000","last_change":"2023-03-03T08:43:43.993565+0000","last_active":"2023-03-03T08:43:44.979296+0000","last_peered":"2023-03-03T08:43:44.979296+0000","last_clean":"2023-03-03T08:43:44.979296+0000","last_became_active":"2023-03-03T08:43:43.992653+0000","last_became_peered":"2023-03-03T08:43:43.992653+0000","last_unstale":"2023-03-03T08:43:44.979296+0000","last_undegraded":"2023-03-03T08:43:44.979296+0000","last_fullsized":"2023-03-03T08:43:44.979296+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:24.295462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981121+0000","last_change":"2023-03-03T08:43:34.550325+0000","last_active":"2023-03-03T08:43:43.981121+0000","last_peered":"2023-03-03T08:43:43.981121+0000","last_clean":"2023-03-03T08:43:43.981121+0000","last_became_active":"2023-03-03T08:43:34.550041+0000","last_became_peered":"2023-03-03T08:43:34.550041+0000","last_unstale":"2023-03-03T08:43:43.981121+0000","last_undegraded":"2023-03-03T08:43:43.981121+0000","last_fullsized":"2023-03-03T08:43:43.981121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:44:34.082003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980457+0000","last_change":"2023-03-03T08:43:34.106777+0000","last_active":"2023-03-03T08:43:43.980457+0000","last_peered":"2023-03-03T08:43:43.980457+0000","last_clean":"2023-03-03T08:43:43.980457+0000","last_became_active":"2023-03-03T08:43:34.106132+0000","last_became_peered":"2023-03-03T08:43:34.106132+0000","last_unstale":"2023-03-03T08:43:43.980457+0000","last_undegraded":"2023-03-03T08:43:43.980457+0000","last_fullsized":"2023-03-03T08:43:43.980457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:55.965987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106224+0000","last_change":"2023-03-03T08:43:43.580312+0000","last_active":"2023-03-03T08:43:45.106224+0000","last_peered":"2023-03-03T08:43:45.106224+0000","last_clean":"2023-03-03T08:43:45.106224+0000","last_became_active":"2023-03-03T08:43:43.580096+0000","last_became_peered":"2023-03-03T08:43:43.580096+0000","last_unstale":"2023-03-03T08:43:45.106224+0000","last_undegraded":"2023-03-03T08:43:45.106224+0000","last_fullsized":"2023-03-03T08:43:45.106224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:06:03.500778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106765+0000","last_change":"2023-03-03T08:43:43.997453+0000","last_active":"2023-03-03T08:43:45.106765+0000","last_peered":"2023-03-03T08:43:45.106765+0000","last_clean":"2023-03-03T08:43:45.106765+0000","last_became_active":"2023-03-03T08:43:43.997090+0000","last_became_peered":"2023-03-03T08:43:43.997090+0000","last_unstale":"2023-03-03T08:43:45.106765+0000","last_undegraded":"2023-03-03T08:43:45.106765+0000","last_fullsized":"2023-03-03T08:43:45.106765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:28:58.322823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985493+0000","last_change":"2023-03-03T08:43:34.542308+0000","last_active":"2023-03-03T08:43:43.985493+0000","last_peered":"2023-03-03T08:43:43.985493+0000","last_clean":"2023-03-03T08:43:43.985493+0000","last_became_active":"2023-03-03T08:43:34.541991+0000","last_became_peered":"2023-03-03T08:43:34.541991+0000","last_unstale":"2023-03-03T08:43:43.985493+0000","last_undegraded":"2023-03-03T08:43:43.985493+0000","last_fullsized":"2023-03-03T08:43:43.985493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:06:21.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992335+0000","last_change":"2023-03-03T08:43:34.109698+0000","last_active":"2023-03-03T08:43:43.992335+0000","last_peered":"2023-03-03T08:43:43.992335+0000","last_clean":"2023-03-03T08:43:43.992335+0000","last_became_active":"2023-03-03T08:43:34.109423+0000","last_became_peered":"2023-03-03T08:43:34.109423+0000","last_unstale":"2023-03-03T08:43:43.992335+0000","last_undegraded":"2023-03-03T08:43:43.992335+0000","last_fullsized":"2023-03-03T08:43:43.992335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:49.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105911+0000","last_change":"2023-03-03T08:43:43.573925+0000","last_active":"2023-03-03T08:43:45.105911+0000","last_peered":"2023-03-03T08:43:45.105911+0000","last_clean":"2023-03-03T08:43:45.105911+0000","last_became_active":"2023-03-03T08:43:43.573474+0000","last_became_peered":"2023-03-03T08:43:43.573474+0000","last_unstale":"2023-03-03T08:43:45.105911+0000","last_undegraded":"2023-03-03T08:43:45.105911+0000","last_fullsized":"2023-03-03T08:43:45.105911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:47.858726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107653+0000","last_change":"2023-03-03T08:43:43.995602+0000","last_active":"2023-03-03T08:43:45.107653+0000","last_peered":"2023-03-03T08:43:45.107653+0000","last_clean":"2023-03-03T08:43:45.107653+0000","last_became_active":"2023-03-03T08:43:43.995404+0000","last_became_peered":"2023-03-03T08:43:43.995404+0000","last_unstale":"2023-03-03T08:43:45.107653+0000","last_undegraded":"2023-03-03T08:43:45.107653+0000","last_fullsized":"2023-03-03T08:43:45.107653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:37.212119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982423+0000","last_change":"2023-03-03T08:43:34.548304+0000","last_active":"2023-03-03T08:43:43.982423+0000","last_peered":"2023-03-03T08:43:43.982423+0000","last_clean":"2023-03-03T08:43:43.982423+0000","last_became_active":"2023-03-03T08:43:34.547960+0000","last_became_peered":"2023-03-03T08:43:34.547960+0000","last_unstale":"2023-03-03T08:43:43.982423+0000","last_undegraded":"2023-03-03T08:43:43.982423+0000","last_fullsized":"2023-03-03T08:43:43.982423+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:04.907753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985574+0000","last_change":"2023-03-03T08:43:34.109372+0000","last_active":"2023-03-03T08:43:43.985574+0000","last_peered":"2023-03-03T08:43:43.985574+0000","last_clean":"2023-03-03T08:43:43.985574+0000","last_became_active":"2023-03-03T08:43:34.109023+0000","last_became_peered":"2023-03-03T08:43:34.109023+0000","last_unstale":"2023-03-03T08:43:43.985574+0000","last_undegraded":"2023-03-03T08:43:43.985574+0000","last_fullsized":"2023-03-03T08:43:43.985574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:51.374188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978866+0000","last_change":"2023-03-03T08:43:43.572240+0000","last_active":"2023-03-03T08:43:44.978866+0000","last_peered":"2023-03-03T08:43:44.978866+0000","last_clean":"2023-03-03T08:43:44.978866+0000","last_became_active":"2023-03-03T08:43:43.571753+0000","last_became_peered":"2023-03-03T08:43:43.571753+0000","last_unstale":"2023-03-03T08:43:44.978866+0000","last_undegraded":"2023-03-03T08:43:44.978866+0000","last_fullsized":"2023-03-03T08:43:44.978866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:02:30.521667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979027+0000","last_change":"2023-03-03T08:43:43.995018+0000","last_active":"2023-03-03T08:43:44.979027+0000","last_peered":"2023-03-03T08:43:44.979027+0000","last_clean":"2023-03-03T08:43:44.979027+0000","last_became_active":"2023-03-03T08:43:43.994690+0000","last_became_peered":"2023-03-03T08:43:43.994690+0000","last_unstale":"2023-03-03T08:43:44.979027+0000","last_undegraded":"2023-03-03T08:43:44.979027+0000","last_fullsized":"2023-03-03T08:43:44.979027+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:06:13.979929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986022+0000","last_change":"2023-03-03T08:43:34.538817+0000","last_active":"2023-03-03T08:43:43.986022+0000","last_peered":"2023-03-03T08:43:43.986022+0000","last_clean":"2023-03-03T08:43:43.986022+0000","last_became_active":"2023-03-03T08:43:34.538506+0000","last_became_peered":"2023-03-03T08:43:34.538506+0000","last_unstale":"2023-03-03T08:43:43.986022+0000","last_undegraded":"2023-03-03T08:43:43.986022+0000","last_fullsized":"2023-03-03T08:43:43.986022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980930+0000","last_change":"2023-03-03T08:43:34.117734+0000","last_active":"2023-03-03T08:43:43.980930+0000","last_peered":"2023-03-03T08:43:43.980930+0000","last_clean":"2023-03-03T08:43:43.980930+0000","last_became_active":"2023-03-03T08:43:34.117454+0000","last_became_peered":"2023-03-03T08:43:34.117454+0000","last_unstale":"2023-03-03T08:43:43.980930+0000","last_undegraded":"2023-03-03T08:43:43.980930+0000","last_fullsized":"2023-03-03T08:43:43.980930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:42:27.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106434+0000","last_change":"2023-03-03T08:43:43.574955+0000","last_active":"2023-03-03T08:43:45.106434+0000","last_peered":"2023-03-03T08:43:45.106434+0000","last_clean":"2023-03-03T08:43:45.106434+0000","last_became_active":"2023-03-03T08:43:43.574677+0000","last_became_peered":"2023-03-03T08:43:43.574677+0000","last_unstale":"2023-03-03T08:43:45.106434+0000","last_undegraded":"2023-03-03T08:43:45.106434+0000","last_fullsized":"2023-03-03T08:43:45.106434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:12:19.538291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.104954+0000","last_change":"2023-03-03T08:43:43.994888+0000","last_active":"2023-03-03T08:43:45.104954+0000","last_peered":"2023-03-03T08:43:45.104954+0000","last_clean":"2023-03-03T08:43:45.104954+0000","last_became_active":"2023-03-03T08:43:43.994630+0000","last_became_peered":"2023-03-03T08:43:43.994630+0000","last_unstale":"2023-03-03T08:43:45.104954+0000","last_undegraded":"2023-03-03T08:43:45.104954+0000","last_fullsized":"2023-03-03T08:43:45.104954+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:15:18.412569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982579+0000","last_change":"2023-03-03T08:43:34.548010+0000","last_active":"2023-03-03T08:43:43.982579+0000","last_peered":"2023-03-03T08:43:43.982579+0000","last_clean":"2023-03-03T08:43:43.982579+0000","last_became_active":"2023-03-03T08:43:34.547662+0000","last_became_peered":"2023-03-03T08:43:34.547662+0000","last_unstale":"2023-03-03T08:43:43.982579+0000","last_undegraded":"2023-03-03T08:43:43.982579+0000","last_fullsized":"2023-03-03T08:43:43.982579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:52.200064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986286+0000","last_change":"2023-03-03T08:43:34.109890+0000","last_active":"2023-03-03T08:43:43.986286+0000","last_peered":"2023-03-03T08:43:43.986286+0000","last_clean":"2023-03-03T08:43:43.986286+0000","last_became_active":"2023-03-03T08:43:34.109583+0000","last_became_peered":"2023-03-03T08:43:34.109583+0000","last_unstale":"2023-03-03T08:43:43.986286+0000","last_undegraded":"2023-03-03T08:43:43.986286+0000","last_fullsized":"2023-03-03T08:43:43.986286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:27:34.142294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978233+0000","last_change":"2023-03-03T08:43:43.575945+0000","last_active":"2023-03-03T08:43:44.978233+0000","last_peered":"2023-03-03T08:43:44.978233+0000","last_clean":"2023-03-03T08:43:44.978233+0000","last_became_active":"2023-03-03T08:43:43.575675+0000","last_became_peered":"2023-03-03T08:43:43.575675+0000","last_unstale":"2023-03-03T08:43:44.978233+0000","last_undegraded":"2023-03-03T08:43:44.978233+0000","last_fullsized":"2023-03-03T08:43:44.978233+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:48.730805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980103+0000","last_change":"2023-03-03T08:43:43.998435+0000","last_active":"2023-03-03T08:43:44.980103+0000","last_peered":"2023-03-03T08:43:44.980103+0000","last_clean":"2023-03-03T08:43:44.980103+0000","last_became_active":"2023-03-03T08:43:43.996508+0000","last_became_peered":"2023-03-03T08:43:43.996508+0000","last_unstale":"2023-03-03T08:43:44.980103+0000","last_undegraded":"2023-03-03T08:43:44.980103+0000","last_fullsized":"2023-03-03T08:43:44.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:31:36.310851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979477+0000","last_change":"2023-03-03T08:43:34.540196+0000","last_active":"2023-03-03T08:43:43.979477+0000","last_peered":"2023-03-03T08:43:43.979477+0000","last_clean":"2023-03-03T08:43:43.979477+0000","last_became_active":"2023-03-03T08:43:34.539931+0000","last_became_peered":"2023-03-03T08:43:34.539931+0000","last_unstale":"2023-03-03T08:43:43.979477+0000","last_undegraded":"2023-03-03T08:43:43.979477+0000","last_fullsized":"2023-03-03T08:43:43.979477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:20.012997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990881+0000","last_change":"2023-03-03T08:43:34.110464+0000","last_active":"2023-03-03T08:43:43.990881+0000","last_peered":"2023-03-03T08:43:43.990881+0000","last_clean":"2023-03-03T08:43:43.990881+0000","last_became_active":"2023-03-03T08:43:34.110165+0000","last_became_peered":"2023-03-03T08:43:34.110165+0000","last_unstale":"2023-03-03T08:43:43.990881+0000","last_undegraded":"2023-03-03T08:43:43.990881+0000","last_fullsized":"2023-03-03T08:43:43.990881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:54.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106139+0000","last_change":"2023-03-03T08:43:43.578298+0000","last_active":"2023-03-03T08:43:45.106139+0000","last_peered":"2023-03-03T08:43:45.106139+0000","last_clean":"2023-03-03T08:43:45.106139+0000","last_became_active":"2023-03-03T08:43:43.578024+0000","last_became_peered":"2023-03-03T08:43:43.578024+0000","last_unstale":"2023-03-03T08:43:45.106139+0000","last_undegraded":"2023-03-03T08:43:45.106139+0000","last_fullsized":"2023-03-03T08:43:45.106139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:00:44.633716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106526+0000","last_change":"2023-03-03T08:43:43.992906+0000","last_active":"2023-03-03T08:43:45.106526+0000","last_peered":"2023-03-03T08:43:45.106526+0000","last_clean":"2023-03-03T08:43:45.106526+0000","last_became_active":"2023-03-03T08:43:43.991823+0000","last_became_peered":"2023-03-03T08:43:43.991823+0000","last_unstale":"2023-03-03T08:43:45.106526+0000","last_undegraded":"2023-03-03T08:43:45.106526+0000","last_fullsized":"2023-03-03T08:43:45.106526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:34.431216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.553597+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.553322+0000","last_became_peered":"2023-03-03T08:43:34.553322+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:24:17.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983377+0000","last_change":"2023-03-03T08:43:34.108123+0000","last_active":"2023-03-03T08:43:43.983377+0000","last_peered":"2023-03-03T08:43:43.983377+0000","last_clean":"2023-03-03T08:43:43.983377+0000","last_became_active":"2023-03-03T08:43:34.107757+0000","last_became_peered":"2023-03-03T08:43:34.107757+0000","last_unstale":"2023-03-03T08:43:43.983377+0000","last_undegraded":"2023-03-03T08:43:43.983377+0000","last_fullsized":"2023-03-03T08:43:43.983377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:09.488930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106648+0000","last_change":"2023-03-03T08:43:43.573369+0000","last_active":"2023-03-03T08:43:45.106648+0000","last_peered":"2023-03-03T08:43:45.106648+0000","last_clean":"2023-03-03T08:43:45.106648+0000","last_became_active":"2023-03-03T08:43:43.573069+0000","last_became_peered":"2023-03-03T08:43:43.573069+0000","last_unstale":"2023-03-03T08:43:45.106648+0000","last_undegraded":"2023-03-03T08:43:45.106648+0000","last_fullsized":"2023-03-03T08:43:45.106648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:09.574926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107410+0000","last_change":"2023-03-03T08:43:43.995725+0000","last_active":"2023-03-03T08:43:45.107410+0000","last_peered":"2023-03-03T08:43:45.107410+0000","last_clean":"2023-03-03T08:43:45.107410+0000","last_became_active":"2023-03-03T08:43:43.993875+0000","last_became_peered":"2023-03-03T08:43:43.993875+0000","last_unstale":"2023-03-03T08:43:45.107410+0000","last_undegraded":"2023-03-03T08:43:45.107410+0000","last_fullsized":"2023-03-03T08:43:45.107410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:38.439539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982673+0000","last_change":"2023-03-03T08:43:34.546067+0000","last_active":"2023-03-03T08:43:43.982673+0000","last_peered":"2023-03-03T08:43:43.982673+0000","last_clean":"2023-03-03T08:43:43.982673+0000","last_became_active":"2023-03-03T08:43:34.543786+0000","last_became_peered":"2023-03-03T08:43:34.543786+0000","last_unstale":"2023-03-03T08:43:43.982673+0000","last_undegraded":"2023-03-03T08:43:43.982673+0000","last_fullsized":"2023-03-03T08:43:43.982673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:32.922812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.978650+0000","last_change":"2023-03-03T08:43:34.115792+0000","last_active":"2023-03-03T08:43:43.978650+0000","last_peered":"2023-03-03T08:43:43.978650+0000","last_clean":"2023-03-03T08:43:43.978650+0000","last_became_active":"2023-03-03T08:43:34.115431+0000","last_became_peered":"2023-03-03T08:43:34.115431+0000","last_unstale":"2023-03-03T08:43:43.978650+0000","last_undegraded":"2023-03-03T08:43:43.978650+0000","last_fullsized":"2023-03-03T08:43:43.978650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:15.484022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980510+0000","last_change":"2023-03-03T08:43:43.579691+0000","last_active":"2023-03-03T08:43:44.980510+0000","last_peered":"2023-03-03T08:43:44.980510+0000","last_clean":"2023-03-03T08:43:44.980510+0000","last_became_active":"2023-03-03T08:43:43.579425+0000","last_became_peered":"2023-03-03T08:43:43.579425+0000","last_unstale":"2023-03-03T08:43:44.980510+0000","last_undegraded":"2023-03-03T08:43:44.980510+0000","last_fullsized":"2023-03-03T08:43:44.980510+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:08.896591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979907+0000","last_change":"2023-03-03T08:43:44.006872+0000","last_active":"2023-03-03T08:43:44.979907+0000","last_peered":"2023-03-03T08:43:44.979907+0000","last_clean":"2023-03-03T08:43:44.979907+0000","last_became_active":"2023-03-03T08:43:44.006283+0000","last_became_peered":"2023-03-03T08:43:44.006283+0000","last_unstale":"2023-03-03T08:43:44.979907+0000","last_undegraded":"2023-03-03T08:43:44.979907+0000","last_fullsized":"2023-03-03T08:43:44.979907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:45.439399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981214+0000","last_change":"2023-03-03T08:43:34.532656+0000","last_active":"2023-03-03T08:43:43.981214+0000","last_peered":"2023-03-03T08:43:43.981214+0000","last_clean":"2023-03-03T08:43:43.981214+0000","last_became_active":"2023-03-03T08:43:34.532297+0000","last_became_peered":"2023-03-03T08:43:34.532297+0000","last_unstale":"2023-03-03T08:43:43.981214+0000","last_undegraded":"2023-03-03T08:43:43.981214+0000","last_fullsized":"2023-03-03T08:43:43.981214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:36.347740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989578+0000","last_change":"2023-03-03T08:43:34.112905+0000","last_active":"2023-03-03T08:43:43.989578+0000","last_peered":"2023-03-03T08:43:43.989578+0000","last_clean":"2023-03-03T08:43:43.989578+0000","last_became_active":"2023-03-03T08:43:34.112601+0000","last_became_peered":"2023-03-03T08:43:34.112601+0000","last_unstale":"2023-03-03T08:43:43.989578+0000","last_undegraded":"2023-03-03T08:43:43.989578+0000","last_fullsized":"2023-03-03T08:43:43.989578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:01.165783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"26'6","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980517+0000","last_change":"2023-03-03T08:43:43.577882+0000","last_active":"2023-03-03T08:43:44.980517+0000","last_peered":"2023-03-03T08:43:44.980517+0000","last_clean":"2023-03-03T08:43:44.980517+0000","last_became_active":"2023-03-03T08:43:43.575625+0000","last_became_peered":"2023-03-03T08:43:43.575625+0000","last_unstale":"2023-03-03T08:43:44.980517+0000","last_undegraded":"2023-03-03T08:43:44.980517+0000","last_fullsized":"2023-03-03T08:43:44.980517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:22:25.919413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106494+0000","last_change":"2023-03-03T08:43:43.994846+0000","last_active":"2023-03-03T08:43:45.106494+0000","last_peered":"2023-03-03T08:43:45.106494+0000","last_clean":"2023-03-03T08:43:45.106494+0000","last_became_active":"2023-03-03T08:43:43.994652+0000","last_became_peered":"2023-03-03T08:43:43.994652+0000","last_unstale":"2023-03-03T08:43:45.106494+0000","last_undegraded":"2023-03-03T08:43:45.106494+0000","last_fullsized":"2023-03-03T08:43:45.106494+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:54:29.238192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990352+0000","last_change":"2023-03-03T08:43:34.539790+0000","last_active":"2023-03-03T08:43:43.990352+0000","last_peered":"2023-03-03T08:43:43.990352+0000","last_clean":"2023-03-03T08:43:43.990352+0000","last_became_active":"2023-03-03T08:43:34.539443+0000","last_became_peered":"2023-03-03T08:43:34.539443+0000","last_unstale":"2023-03-03T08:43:43.990352+0000","last_undegraded":"2023-03-03T08:43:43.990352+0000","last_fullsized":"2023-03-03T08:43:43.990352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:25.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980897+0000","last_change":"2023-03-03T08:43:34.112941+0000","last_active":"2023-03-03T08:43:43.980897+0000","last_peered":"2023-03-03T08:43:43.980897+0000","last_clean":"2023-03-03T08:43:43.980897+0000","last_became_active":"2023-03-03T08:43:34.112646+0000","last_became_peered":"2023-03-03T08:43:34.112646+0000","last_unstale":"2023-03-03T08:43:43.980897+0000","last_undegraded":"2023-03-03T08:43:43.980897+0000","last_fullsized":"2023-03-03T08:43:43.980897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:32.767710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107854+0000","last_change":"2023-03-03T08:43:43.576421+0000","last_active":"2023-03-03T08:43:45.107854+0000","last_peered":"2023-03-03T08:43:45.107854+0000","last_clean":"2023-03-03T08:43:45.107854+0000","last_became_active":"2023-03-03T08:43:43.576086+0000","last_became_peered":"2023-03-03T08:43:43.576086+0000","last_unstale":"2023-03-03T08:43:45.107854+0000","last_undegraded":"2023-03-03T08:43:45.107854+0000","last_fullsized":"2023-03-03T08:43:45.107854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:57:17.814222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980193+0000","last_change":"2023-03-03T08:43:43.995591+0000","last_active":"2023-03-03T08:43:44.980193+0000","last_peered":"2023-03-03T08:43:44.980193+0000","last_clean":"2023-03-03T08:43:44.980193+0000","last_became_active":"2023-03-03T08:43:43.995338+0000","last_became_peered":"2023-03-03T08:43:43.995338+0000","last_unstale":"2023-03-03T08:43:44.980193+0000","last_undegraded":"2023-03-03T08:43:44.980193+0000","last_fullsized":"2023-03-03T08:43:44.980193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:17:56.617398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985713+0000","last_change":"2023-03-03T08:43:34.537857+0000","last_active":"2023-03-03T08:43:43.985713+0000","last_peered":"2023-03-03T08:43:43.985713+0000","last_clean":"2023-03-03T08:43:43.985713+0000","last_became_active":"2023-03-03T08:43:34.537590+0000","last_became_peered":"2023-03-03T08:43:34.537590+0000","last_unstale":"2023-03-03T08:43:43.985713+0000","last_undegraded":"2023-03-03T08:43:43.985713+0000","last_fullsized":"2023-03-03T08:43:43.985713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:52.612521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980688+0000","last_change":"2023-03-03T08:43:34.117609+0000","last_active":"2023-03-03T08:43:43.980688+0000","last_peered":"2023-03-03T08:43:43.980688+0000","last_clean":"2023-03-03T08:43:43.980688+0000","last_became_active":"2023-03-03T08:43:34.117271+0000","last_became_peered":"2023-03-03T08:43:34.117271+0000","last_unstale":"2023-03-03T08:43:43.980688+0000","last_undegraded":"2023-03-03T08:43:43.980688+0000","last_fullsized":"2023-03-03T08:43:43.980688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:20.259259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978772+0000","last_change":"2023-03-03T08:43:43.580489+0000","last_active":"2023-03-03T08:43:44.978772+0000","last_peered":"2023-03-03T08:43:44.978772+0000","last_clean":"2023-03-03T08:43:44.978772+0000","last_became_active":"2023-03-03T08:43:43.580207+0000","last_became_peered":"2023-03-03T08:43:43.580207+0000","last_unstale":"2023-03-03T08:43:44.978772+0000","last_undegraded":"2023-03-03T08:43:44.978772+0000","last_fullsized":"2023-03-03T08:43:44.978772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:58:05.519074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978633+0000","last_change":"2023-03-03T08:43:43.993832+0000","last_active":"2023-03-03T08:43:44.978633+0000","last_peered":"2023-03-03T08:43:44.978633+0000","last_clean":"2023-03-03T08:43:44.978633+0000","last_became_active":"2023-03-03T08:43:43.993524+0000","last_became_peered":"2023-03-03T08:43:43.993524+0000","last_unstale":"2023-03-03T08:43:44.978633+0000","last_undegraded":"2023-03-03T08:43:44.978633+0000","last_fullsized":"2023-03-03T08:43:44.978633+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:23:55.761785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987159+0000","last_change":"2023-03-03T08:43:34.544694+0000","last_active":"2023-03-03T08:43:43.987159+0000","last_peered":"2023-03-03T08:43:43.987159+0000","last_clean":"2023-03-03T08:43:43.987159+0000","last_became_active":"2023-03-03T08:43:34.544399+0000","last_became_peered":"2023-03-03T08:43:34.544399+0000","last_unstale":"2023-03-03T08:43:43.987159+0000","last_undegraded":"2023-03-03T08:43:43.987159+0000","last_fullsized":"2023-03-03T08:43:43.987159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986196+0000","last_change":"2023-03-03T08:43:34.110810+0000","last_active":"2023-03-03T08:43:43.986196+0000","last_peered":"2023-03-03T08:43:43.986196+0000","last_clean":"2023-03-03T08:43:43.986196+0000","last_became_active":"2023-03-03T08:43:34.110483+0000","last_became_peered":"2023-03-03T08:43:34.110483+0000","last_unstale":"2023-03-03T08:43:43.986196+0000","last_undegraded":"2023-03-03T08:43:43.986196+0000","last_fullsized":"2023-03-03T08:43:43.986196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:06.970615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979989+0000","last_change":"2023-03-03T08:43:43.576669+0000","last_active":"2023-03-03T08:43:44.979989+0000","last_peered":"2023-03-03T08:43:44.979989+0000","last_clean":"2023-03-03T08:43:44.979989+0000","last_became_active":"2023-03-03T08:43:43.573885+0000","last_became_peered":"2023-03-03T08:43:43.573885+0000","last_unstale":"2023-03-03T08:43:44.979989+0000","last_undegraded":"2023-03-03T08:43:44.979989+0000","last_fullsized":"2023-03-03T08:43:44.979989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:56:12.054118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979442+0000","last_change":"2023-03-03T08:43:43.996790+0000","last_active":"2023-03-03T08:43:44.979442+0000","last_peered":"2023-03-03T08:43:44.979442+0000","last_clean":"2023-03-03T08:43:44.979442+0000","last_became_active":"2023-03-03T08:43:43.996366+0000","last_became_peered":"2023-03-03T08:43:43.996366+0000","last_unstale":"2023-03-03T08:43:44.979442+0000","last_undegraded":"2023-03-03T08:43:44.979442+0000","last_fullsized":"2023-03-03T08:43:44.979442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:01.662221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988553+0000","last_change":"2023-03-03T08:43:34.538703+0000","last_active":"2023-03-03T08:43:43.988553+0000","last_peered":"2023-03-03T08:43:43.988553+0000","last_clean":"2023-03-03T08:43:43.988553+0000","last_became_active":"2023-03-03T08:43:34.538418+0000","last_became_peered":"2023-03-03T08:43:34.538418+0000","last_unstale":"2023-03-03T08:43:43.988553+0000","last_undegraded":"2023-03-03T08:43:43.988553+0000","last_fullsized":"2023-03-03T08:43:43.988553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:02.795203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986185+0000","last_change":"2023-03-03T08:43:34.113168+0000","last_active":"2023-03-03T08:43:43.986185+0000","last_peered":"2023-03-03T08:43:43.986185+0000","last_clean":"2023-03-03T08:43:43.986185+0000","last_became_active":"2023-03-03T08:43:34.112731+0000","last_became_peered":"2023-03-03T08:43:34.112731+0000","last_unstale":"2023-03-03T08:43:43.986185+0000","last_undegraded":"2023-03-03T08:43:43.986185+0000","last_fullsized":"2023-03-03T08:43:43.986185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:57:59.463191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106795+0000","last_change":"2023-03-03T08:43:43.577965+0000","last_active":"2023-03-03T08:43:45.106795+0000","last_peered":"2023-03-03T08:43:45.106795+0000","last_clean":"2023-03-03T08:43:45.106795+0000","last_became_active":"2023-03-03T08:43:43.577587+0000","last_became_peered":"2023-03-03T08:43:43.577587+0000","last_unstale":"2023-03-03T08:43:45.106795+0000","last_undegraded":"2023-03-03T08:43:45.106795+0000","last_fullsized":"2023-03-03T08:43:45.106795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:24.196773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105366+0000","last_change":"2023-03-03T08:43:43.998210+0000","last_active":"2023-03-03T08:43:45.105366+0000","last_peered":"2023-03-03T08:43:45.105366+0000","last_clean":"2023-03-03T08:43:45.105366+0000","last_became_active":"2023-03-03T08:43:43.997845+0000","last_became_peered":"2023-03-03T08:43:43.997845+0000","last_unstale":"2023-03-03T08:43:45.105366+0000","last_undegraded":"2023-03-03T08:43:45.105366+0000","last_fullsized":"2023-03-03T08:43:45.105366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:57:38.226245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991762+0000","last_change":"2023-03-03T08:43:34.528573+0000","last_active":"2023-03-03T08:43:43.991762+0000","last_peered":"2023-03-03T08:43:43.991762+0000","last_clean":"2023-03-03T08:43:43.991762+0000","last_became_active":"2023-03-03T08:43:34.528278+0000","last_became_peered":"2023-03-03T08:43:34.528278+0000","last_unstale":"2023-03-03T08:43:43.991762+0000","last_undegraded":"2023-03-03T08:43:43.991762+0000","last_fullsized":"2023-03-03T08:43:43.991762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:51.468323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.110802+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.110519+0000","last_became_peered":"2023-03-03T08:43:34.110519+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:02:18.355043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105602+0000","last_change":"2023-03-03T08:43:43.573655+0000","last_active":"2023-03-03T08:43:45.105602+0000","last_peered":"2023-03-03T08:43:45.105602+0000","last_clean":"2023-03-03T08:43:45.105602+0000","last_became_active":"2023-03-03T08:43:43.573162+0000","last_became_peered":"2023-03-03T08:43:43.573162+0000","last_unstale":"2023-03-03T08:43:45.105602+0000","last_undegraded":"2023-03-03T08:43:45.105602+0000","last_fullsized":"2023-03-03T08:43:45.105602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:59:04.568700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106617+0000","last_change":"2023-03-03T08:43:43.993431+0000","last_active":"2023-03-03T08:43:45.106617+0000","last_peered":"2023-03-03T08:43:45.106617+0000","last_clean":"2023-03-03T08:43:45.106617+0000","last_became_active":"2023-03-03T08:43:43.993166+0000","last_became_peered":"2023-03-03T08:43:43.993166+0000","last_unstale":"2023-03-03T08:43:45.106617+0000","last_undegraded":"2023-03-03T08:43:45.106617+0000","last_fullsized":"2023-03-03T08:43:45.106617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:04:52.880667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979475+0000","last_change":"2023-03-03T08:43:43.992992+0000","last_active":"2023-03-03T08:43:44.979475+0000","last_peered":"2023-03-03T08:43:44.979475+0000","last_clean":"2023-03-03T08:43:44.979475+0000","last_became_active":"2023-03-03T08:43:43.992664+0000","last_became_peered":"2023-03-03T08:43:43.992664+0000","last_unstale":"2023-03-03T08:43:44.979475+0000","last_undegraded":"2023-03-03T08:43:44.979475+0000","last_fullsized":"2023-03-03T08:43:44.979475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:07.276715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981207+0000","last_change":"2023-03-03T08:43:34.547450+0000","last_active":"2023-03-03T08:43:43.981207+0000","last_peered":"2023-03-03T08:43:43.981207+0000","last_clean":"2023-03-03T08:43:43.981207+0000","last_became_active":"2023-03-03T08:43:34.547168+0000","last_became_peered":"2023-03-03T08:43:34.547168+0000","last_unstale":"2023-03-03T08:43:43.981207+0000","last_undegraded":"2023-03-03T08:43:43.981207+0000","last_fullsized":"2023-03-03T08:43:43.981207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:17.723229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982025+0000","last_change":"2023-03-03T08:43:34.106505+0000","last_active":"2023-03-03T08:43:43.982025+0000","last_peered":"2023-03-03T08:43:43.982025+0000","last_clean":"2023-03-03T08:43:43.982025+0000","last_became_active":"2023-03-03T08:43:34.106079+0000","last_became_peered":"2023-03-03T08:43:34.106079+0000","last_unstale":"2023-03-03T08:43:43.982025+0000","last_undegraded":"2023-03-03T08:43:43.982025+0000","last_fullsized":"2023-03-03T08:43:43.982025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:48.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979698+0000","last_change":"2023-03-03T08:43:43.575448+0000","last_active":"2023-03-03T08:43:44.979698+0000","last_peered":"2023-03-03T08:43:44.979698+0000","last_clean":"2023-03-03T08:43:44.979698+0000","last_became_active":"2023-03-03T08:43:43.575142+0000","last_became_peered":"2023-03-03T08:43:43.575142+0000","last_unstale":"2023-03-03T08:43:44.979698+0000","last_undegraded":"2023-03-03T08:43:44.979698+0000","last_fullsized":"2023-03-03T08:43:44.979698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:42:09.308616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990396+0000","last_change":"2023-03-03T08:43:34.549123+0000","last_active":"2023-03-03T08:43:43.990396+0000","last_peered":"2023-03-03T08:43:43.990396+0000","last_clean":"2023-03-03T08:43:43.990396+0000","last_became_active":"2023-03-03T08:43:34.548839+0000","last_became_peered":"2023-03-03T08:43:34.548839+0000","last_unstale":"2023-03-03T08:43:43.990396+0000","last_undegraded":"2023-03-03T08:43:43.990396+0000","last_fullsized":"2023-03-03T08:43:43.990396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:38.134009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984959+0000","last_change":"2023-03-03T08:43:34.108187+0000","last_active":"2023-03-03T08:43:43.984959+0000","last_peered":"2023-03-03T08:43:43.984959+0000","last_clean":"2023-03-03T08:43:43.984959+0000","last_became_active":"2023-03-03T08:43:34.107859+0000","last_became_peered":"2023-03-03T08:43:34.107859+0000","last_unstale":"2023-03-03T08:43:43.984959+0000","last_undegraded":"2023-03-03T08:43:43.984959+0000","last_fullsized":"2023-03-03T08:43:43.984959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:46:40.595756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107953+0000","last_change":"2023-03-03T08:43:43.572738+0000","last_active":"2023-03-03T08:43:45.107953+0000","last_peered":"2023-03-03T08:43:45.107953+0000","last_clean":"2023-03-03T08:43:45.107953+0000","last_became_active":"2023-03-03T08:43:43.572435+0000","last_became_peered":"2023-03-03T08:43:43.572435+0000","last_unstale":"2023-03-03T08:43:45.107953+0000","last_undegraded":"2023-03-03T08:43:45.107953+0000","last_fullsized":"2023-03-03T08:43:45.107953+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:14.264653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105420+0000","last_change":"2023-03-03T08:43:43.995021+0000","last_active":"2023-03-03T08:43:45.105420+0000","last_peered":"2023-03-03T08:43:45.105420+0000","last_clean":"2023-03-03T08:43:45.105420+0000","last_became_active":"2023-03-03T08:43:43.994766+0000","last_became_peered":"2023-03-03T08:43:43.994766+0000","last_unstale":"2023-03-03T08:43:45.105420+0000","last_undegraded":"2023-03-03T08:43:45.105420+0000","last_fullsized":"2023-03-03T08:43:45.105420+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:14:03.827057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979582+0000","last_change":"2023-03-03T08:43:43.993391+0000","last_active":"2023-03-03T08:43:44.979582+0000","last_peered":"2023-03-03T08:43:44.979582+0000","last_clean":"2023-03-03T08:43:44.979582+0000","last_became_active":"2023-03-03T08:43:43.993087+0000","last_became_peered":"2023-03-03T08:43:43.993087+0000","last_unstale":"2023-03-03T08:43:44.979582+0000","last_undegraded":"2023-03-03T08:43:44.979582+0000","last_fullsized":"2023-03-03T08:43:44.979582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:13:01.193811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992180+0000","last_change":"2023-03-03T08:43:34.537301+0000","last_active":"2023-03-03T08:43:43.992180+0000","last_peered":"2023-03-03T08:43:43.992180+0000","last_clean":"2023-03-03T08:43:43.992180+0000","last_became_active":"2023-03-03T08:43:34.537007+0000","last_became_peered":"2023-03-03T08:43:34.537007+0000","last_unstale":"2023-03-03T08:43:43.992180+0000","last_undegraded":"2023-03-03T08:43:43.992180+0000","last_fullsized":"2023-03-03T08:43:43.992180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:59:49.124140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982700+0000","last_change":"2023-03-03T08:43:34.108056+0000","last_active":"2023-03-03T08:43:43.982700+0000","last_peered":"2023-03-03T08:43:43.982700+0000","last_clean":"2023-03-03T08:43:43.982700+0000","last_became_active":"2023-03-03T08:43:34.107708+0000","last_became_peered":"2023-03-03T08:43:34.107708+0000","last_unstale":"2023-03-03T08:43:43.982700+0000","last_undegraded":"2023-03-03T08:43:43.982700+0000","last_fullsized":"2023-03-03T08:43:43.982700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:10.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107314+0000","last_change":"2023-03-03T08:43:43.575788+0000","last_active":"2023-03-03T08:43:45.107314+0000","last_peered":"2023-03-03T08:43:45.107314+0000","last_clean":"2023-03-03T08:43:45.107314+0000","last_became_active":"2023-03-03T08:43:43.575507+0000","last_became_peered":"2023-03-03T08:43:43.575507+0000","last_unstale":"2023-03-03T08:43:45.107314+0000","last_undegraded":"2023-03-03T08:43:45.107314+0000","last_fullsized":"2023-03-03T08:43:45.107314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:15.843816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107877+0000","last_change":"2023-03-03T08:43:43.994916+0000","last_active":"2023-03-03T08:43:45.107877+0000","last_peered":"2023-03-03T08:43:45.107877+0000","last_clean":"2023-03-03T08:43:45.107877+0000","last_became_active":"2023-03-03T08:43:43.994639+0000","last_became_peered":"2023-03-03T08:43:43.994639+0000","last_unstale":"2023-03-03T08:43:45.107877+0000","last_undegraded":"2023-03-03T08:43:45.107877+0000","last_fullsized":"2023-03-03T08:43:45.107877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:52.526052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980834+0000","last_change":"2023-03-03T08:43:34.533987+0000","last_active":"2023-03-03T08:43:43.980834+0000","last_peered":"2023-03-03T08:43:43.980834+0000","last_clean":"2023-03-03T08:43:43.980834+0000","last_became_active":"2023-03-03T08:43:34.533729+0000","last_became_peered":"2023-03-03T08:43:34.533729+0000","last_unstale":"2023-03-03T08:43:43.980834+0000","last_undegraded":"2023-03-03T08:43:43.980834+0000","last_fullsized":"2023-03-03T08:43:43.980834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:18:12.407151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106398+0000","last_change":"2023-03-03T08:43:43.996808+0000","last_active":"2023-03-03T08:43:45.106398+0000","last_peered":"2023-03-03T08:43:45.106398+0000","last_clean":"2023-03-03T08:43:45.106398+0000","last_became_active":"2023-03-03T08:43:43.996559+0000","last_became_peered":"2023-03-03T08:43:43.996559+0000","last_unstale":"2023-03-03T08:43:45.106398+0000","last_undegraded":"2023-03-03T08:43:45.106398+0000","last_fullsized":"2023-03-03T08:43:45.106398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:01.223908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.993213+0000","last_change":"2023-03-03T08:43:34.540762+0000","last_active":"2023-03-03T08:43:43.993213+0000","last_peered":"2023-03-03T08:43:43.993213+0000","last_clean":"2023-03-03T08:43:43.993213+0000","last_became_active":"2023-03-03T08:43:34.540467+0000","last_became_peered":"2023-03-03T08:43:34.540467+0000","last_unstale":"2023-03-03T08:43:43.993213+0000","last_undegraded":"2023-03-03T08:43:43.993213+0000","last_fullsized":"2023-03-03T08:43:43.993213+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107993+0000","last_change":"2023-03-03T08:43:43.995286+0000","last_active":"2023-03-03T08:43:45.107993+0000","last_peered":"2023-03-03T08:43:45.107993+0000","last_clean":"2023-03-03T08:43:45.107993+0000","last_became_active":"2023-03-03T08:43:43.994861+0000","last_became_peered":"2023-03-03T08:43:43.994861+0000","last_unstale":"2023-03-03T08:43:45.107993+0000","last_undegraded":"2023-03-03T08:43:45.107993+0000","last_fullsized":"2023-03-03T08:43:45.107993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990722+0000","last_change":"2023-03-03T08:43:34.542858+0000","last_active":"2023-03-03T08:43:43.990722+0000","last_peered":"2023-03-03T08:43:43.990722+0000","last_clean":"2023-03-03T08:43:43.990722+0000","last_became_active":"2023-03-03T08:43:34.542510+0000","last_became_peered":"2023-03-03T08:43:34.542510+0000","last_unstale":"2023-03-03T08:43:43.990722+0000","last_undegraded":"2023-03-03T08:43:43.990722+0000","last_fullsized":"2023-03-03T08:43:43.990722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:21.471234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106711+0000","last_change":"2023-03-03T08:43:43.992084+0000","last_active":"2023-03-03T08:43:45.106711+0000","last_peered":"2023-03-03T08:43:45.106711+0000","last_clean":"2023-03-03T08:43:45.106711+0000","last_became_active":"2023-03-03T08:43:43.991465+0000","last_became_peered":"2023-03-03T08:43:43.991465+0000","last_unstale":"2023-03-03T08:43:45.106711+0000","last_undegraded":"2023-03-03T08:43:45.106711+0000","last_fullsized":"2023-03-03T08:43:45.106711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:19.530923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983571+0000","last_change":"2023-03-03T08:43:34.548469+0000","last_active":"2023-03-03T08:43:43.983571+0000","last_peered":"2023-03-03T08:43:43.983571+0000","last_clean":"2023-03-03T08:43:43.983571+0000","last_became_active":"2023-03-03T08:43:34.547647+0000","last_became_peered":"2023-03-03T08:43:34.547647+0000","last_unstale":"2023-03-03T08:43:43.983571+0000","last_undegraded":"2023-03-03T08:43:43.983571+0000","last_fullsized":"2023-03-03T08:43:43.983571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:40.054781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107161+0000","last_change":"2023-03-03T08:43:43.998061+0000","last_active":"2023-03-03T08:43:45.107161+0000","last_peered":"2023-03-03T08:43:45.107161+0000","last_clean":"2023-03-03T08:43:45.107161+0000","last_became_active":"2023-03-03T08:43:43.997770+0000","last_became_peered":"2023-03-03T08:43:43.997770+0000","last_unstale":"2023-03-03T08:43:45.107161+0000","last_undegraded":"2023-03-03T08:43:45.107161+0000","last_fullsized":"2023-03-03T08:43:45.107161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:17.053610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988635+0000","last_change":"2023-03-03T08:43:34.549771+0000","last_active":"2023-03-03T08:43:43.988635+0000","last_peered":"2023-03-03T08:43:43.988635+0000","last_clean":"2023-03-03T08:43:43.988635+0000","last_became_active":"2023-03-03T08:43:34.549479+0000","last_became_peered":"2023-03-03T08:43:34.549479+0000","last_unstale":"2023-03-03T08:43:43.988635+0000","last_undegraded":"2023-03-03T08:43:43.988635+0000","last_fullsized":"2023-03-03T08:43:43.988635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:41:03.902019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979024+0000","last_change":"2023-03-03T08:43:43.999391+0000","last_active":"2023-03-03T08:43:44.979024+0000","last_peered":"2023-03-03T08:43:44.979024+0000","last_clean":"2023-03-03T08:43:44.979024+0000","last_became_active":"2023-03-03T08:43:43.998904+0000","last_became_peered":"2023-03-03T08:43:43.998904+0000","last_unstale":"2023-03-03T08:43:44.979024+0000","last_undegraded":"2023-03-03T08:43:44.979024+0000","last_fullsized":"2023-03-03T08:43:44.979024+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:17.429776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992365+0000","last_change":"2023-03-03T08:43:34.542661+0000","last_active":"2023-03-03T08:43:43.992365+0000","last_peered":"2023-03-03T08:43:43.992365+0000","last_clean":"2023-03-03T08:43:43.992365+0000","last_became_active":"2023-03-03T08:43:34.542353+0000","last_became_peered":"2023-03-03T08:43:34.542353+0000","last_unstale":"2023-03-03T08:43:43.992365+0000","last_undegraded":"2023-03-03T08:43:43.992365+0000","last_fullsized":"2023-03-03T08:43:43.992365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:38.730662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105103+0000","last_change":"2023-03-03T08:43:43.996729+0000","last_active":"2023-03-03T08:43:45.105103+0000","last_peered":"2023-03-03T08:43:45.105103+0000","last_clean":"2023-03-03T08:43:45.105103+0000","last_became_active":"2023-03-03T08:43:43.986535+0000","last_became_peered":"2023-03-03T08:43:43.986535+0000","last_unstale":"2023-03-03T08:43:45.105103+0000","last_undegraded":"2023-03-03T08:43:45.105103+0000","last_fullsized":"2023-03-03T08:43:45.105103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:22.505305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982406+0000","last_change":"2023-03-03T08:43:34.549837+0000","last_active":"2023-03-03T08:43:43.982406+0000","last_peered":"2023-03-03T08:43:43.982406+0000","last_clean":"2023-03-03T08:43:43.982406+0000","last_became_active":"2023-03-03T08:43:34.549542+0000","last_became_peered":"2023-03-03T08:43:34.549542+0000","last_unstale":"2023-03-03T08:43:43.982406+0000","last_undegraded":"2023-03-03T08:43:43.982406+0000","last_fullsized":"2023-03-03T08:43:43.982406+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:31.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106334+0000","last_change":"2023-03-03T08:43:43.993213+0000","last_active":"2023-03-03T08:43:45.106334+0000","last_peered":"2023-03-03T08:43:45.106334+0000","last_clean":"2023-03-03T08:43:45.106334+0000","last_became_active":"2023-03-03T08:43:43.992920+0000","last_became_peered":"2023-03-03T08:43:43.992920+0000","last_unstale":"2023-03-03T08:43:45.106334+0000","last_undegraded":"2023-03-03T08:43:45.106334+0000","last_fullsized":"2023-03-03T08:43:45.106334+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:42.055599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.547003+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.546744+0000","last_became_peered":"2023-03-03T08:43:34.546744+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:53:38.996605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106934+0000","last_change":"2023-03-03T08:43:43.994195+0000","last_active":"2023-03-03T08:43:45.106934+0000","last_peered":"2023-03-03T08:43:45.106934+0000","last_clean":"2023-03-03T08:43:45.106934+0000","last_became_active":"2023-03-03T08:43:43.993463+0000","last_became_peered":"2023-03-03T08:43:43.993463+0000","last_unstale":"2023-03-03T08:43:45.106934+0000","last_undegraded":"2023-03-03T08:43:45.106934+0000","last_fullsized":"2023-03-03T08:43:45.106934+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:52.365793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982677+0000","last_change":"2023-03-03T08:43:34.553589+0000","last_active":"2023-03-03T08:43:43.982677+0000","last_peered":"2023-03-03T08:43:43.982677+0000","last_clean":"2023-03-03T08:43:43.982677+0000","last_became_active":"2023-03-03T08:43:34.553242+0000","last_became_peered":"2023-03-03T08:43:34.553242+0000","last_unstale":"2023-03-03T08:43:43.982677+0000","last_undegraded":"2023-03-03T08:43:43.982677+0000","last_fullsized":"2023-03-03T08:43:43.982677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:09.000051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107426+0000","last_change":"2023-03-03T08:43:44.004404+0000","last_active":"2023-03-03T08:43:45.107426+0000","last_peered":"2023-03-03T08:43:45.107426+0000","last_clean":"2023-03-03T08:43:45.107426+0000","last_became_active":"2023-03-03T08:43:44.004120+0000","last_became_peered":"2023-03-03T08:43:44.004120+0000","last_unstale":"2023-03-03T08:43:45.107426+0000","last_undegraded":"2023-03-03T08:43:45.107426+0000","last_fullsized":"2023-03-03T08:43:45.107426+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:36.006791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990661+0000","last_change":"2023-03-03T08:43:34.537110+0000","last_active":"2023-03-03T08:43:43.990661+0000","last_peered":"2023-03-03T08:43:43.990661+0000","last_clean":"2023-03-03T08:43:43.990661+0000","last_became_active":"2023-03-03T08:43:34.536848+0000","last_became_peered":"2023-03-03T08:43:34.536848+0000","last_unstale":"2023-03-03T08:43:43.990661+0000","last_undegraded":"2023-03-03T08:43:43.990661+0000","last_fullsized":"2023-03-03T08:43:43.990661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106366+0000","last_change":"2023-03-03T08:43:43.999266+0000","last_active":"2023-03-03T08:43:45.106366+0000","last_peered":"2023-03-03T08:43:45.106366+0000","last_clean":"2023-03-03T08:43:45.106366+0000","last_became_active":"2023-03-03T08:43:43.998929+0000","last_became_peered":"2023-03-03T08:43:43.998929+0000","last_unstale":"2023-03-03T08:43:45.106366+0000","last_undegraded":"2023-03-03T08:43:45.106366+0000","last_fullsized":"2023-03-03T08:43:45.106366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:02.236263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994220+0000","last_change":"2023-03-03T08:43:34.532154+0000","last_active":"2023-03-03T08:43:43.994220+0000","last_peered":"2023-03-03T08:43:43.994220+0000","last_clean":"2023-03-03T08:43:43.994220+0000","last_became_active":"2023-03-03T08:43:34.531844+0000","last_became_peered":"2023-03-03T08:43:34.531844+0000","last_unstale":"2023-03-03T08:43:43.994220+0000","last_undegraded":"2023-03-03T08:43:43.994220+0000","last_fullsized":"2023-03-03T08:43:43.994220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:00.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106991+0000","last_change":"2023-03-03T08:43:44.004469+0000","last_active":"2023-03-03T08:43:45.106991+0000","last_peered":"2023-03-03T08:43:45.106991+0000","last_clean":"2023-03-03T08:43:45.106991+0000","last_became_active":"2023-03-03T08:43:44.004187+0000","last_became_peered":"2023-03-03T08:43:44.004187+0000","last_unstale":"2023-03-03T08:43:45.106991+0000","last_undegraded":"2023-03-03T08:43:45.106991+0000","last_fullsized":"2023-03-03T08:43:45.106991+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:35.298857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979308+0000","last_change":"2023-03-03T08:43:34.527373+0000","last_active":"2023-03-03T08:43:43.979308+0000","last_peered":"2023-03-03T08:43:43.979308+0000","last_clean":"2023-03-03T08:43:43.979308+0000","last_became_active":"2023-03-03T08:43:34.527171+0000","last_became_peered":"2023-03-03T08:43:34.527171+0000","last_unstale":"2023-03-03T08:43:43.979308+0000","last_undegraded":"2023-03-03T08:43:43.979308+0000","last_fullsized":"2023-03-03T08:43:43.979308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:20:51.138587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108220+0000","last_change":"2023-03-03T08:43:43.996329+0000","last_active":"2023-03-03T08:43:45.108220+0000","last_peered":"2023-03-03T08:43:45.108220+0000","last_clean":"2023-03-03T08:43:45.108220+0000","last_became_active":"2023-03-03T08:43:43.995055+0000","last_became_peered":"2023-03-03T08:43:43.995055+0000","last_unstale":"2023-03-03T08:43:45.108220+0000","last_undegraded":"2023-03-03T08:43:45.108220+0000","last_fullsized":"2023-03-03T08:43:45.108220+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:57.593740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984695+0000","last_change":"2023-03-03T08:43:34.542729+0000","last_active":"2023-03-03T08:43:43.984695+0000","last_peered":"2023-03-03T08:43:43.984695+0000","last_clean":"2023-03-03T08:43:43.984695+0000","last_became_active":"2023-03-03T08:43:34.542246+0000","last_became_peered":"2023-03-03T08:43:34.542246+0000","last_unstale":"2023-03-03T08:43:43.984695+0000","last_undegraded":"2023-03-03T08:43:43.984695+0000","last_fullsized":"2023-03-03T08:43:43.984695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:14:59.995526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105365+0000","last_change":"2023-03-03T08:43:44.002358+0000","last_active":"2023-03-03T08:43:45.105365+0000","last_peered":"2023-03-03T08:43:45.105365+0000","last_clean":"2023-03-03T08:43:45.105365+0000","last_became_active":"2023-03-03T08:43:44.002020+0000","last_became_peered":"2023-03-03T08:43:44.002020+0000","last_unstale":"2023-03-03T08:43:45.105365+0000","last_undegraded":"2023-03-03T08:43:45.105365+0000","last_fullsized":"2023-03-03T08:43:45.105365+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:51.524774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982664+0000","last_change":"2023-03-03T08:43:34.549026+0000","last_active":"2023-03-03T08:43:43.982664+0000","last_peered":"2023-03-03T08:43:43.982664+0000","last_clean":"2023-03-03T08:43:43.982664+0000","last_became_active":"2023-03-03T08:43:34.548778+0000","last_became_peered":"2023-03-03T08:43:34.548778+0000","last_unstale":"2023-03-03T08:43:43.982664+0000","last_undegraded":"2023-03-03T08:43:43.982664+0000","last_fullsized":"2023-03-03T08:43:43.982664+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:41.835304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105281+0000","last_change":"2023-03-03T08:43:43.997684+0000","last_active":"2023-03-03T08:43:45.105281+0000","last_peered":"2023-03-03T08:43:45.105281+0000","last_clean":"2023-03-03T08:43:45.105281+0000","last_became_active":"2023-03-03T08:43:43.997503+0000","last_became_peered":"2023-03-03T08:43:43.997503+0000","last_unstale":"2023-03-03T08:43:45.105281+0000","last_undegraded":"2023-03-03T08:43:45.105281+0000","last_fullsized":"2023-03-03T08:43:45.105281+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:29.512524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991178+0000","last_change":"2023-03-03T08:43:34.544013+0000","last_active":"2023-03-03T08:43:43.991178+0000","last_peered":"2023-03-03T08:43:43.991178+0000","last_clean":"2023-03-03T08:43:43.991178+0000","last_became_active":"2023-03-03T08:43:34.543745+0000","last_became_peered":"2023-03-03T08:43:34.543745+0000","last_unstale":"2023-03-03T08:43:43.991178+0000","last_undegraded":"2023-03-03T08:43:43.991178+0000","last_fullsized":"2023-03-03T08:43:43.991178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:05.647977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107560+0000","last_change":"2023-03-03T08:43:43.999884+0000","last_active":"2023-03-03T08:43:45.107560+0000","last_peered":"2023-03-03T08:43:45.107560+0000","last_clean":"2023-03-03T08:43:45.107560+0000","last_became_active":"2023-03-03T08:43:43.999593+0000","last_became_peered":"2023-03-03T08:43:43.999593+0000","last_unstale":"2023-03-03T08:43:45.107560+0000","last_undegraded":"2023-03-03T08:43:45.107560+0000","last_fullsized":"2023-03-03T08:43:45.107560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:45:09.132974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982600+0000","last_change":"2023-03-03T08:43:34.553434+0000","last_active":"2023-03-03T08:43:43.982600+0000","last_peered":"2023-03-03T08:43:43.982600+0000","last_clean":"2023-03-03T08:43:43.982600+0000","last_became_active":"2023-03-03T08:43:34.553125+0000","last_became_peered":"2023-03-03T08:43:34.553125+0000","last_unstale":"2023-03-03T08:43:43.982600+0000","last_undegraded":"2023-03-03T08:43:43.982600+0000","last_fullsized":"2023-03-03T08:43:43.982600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:41:17.553506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105717+0000","last_change":"2023-03-03T08:43:44.006710+0000","last_active":"2023-03-03T08:43:45.105717+0000","last_peered":"2023-03-03T08:43:45.105717+0000","last_clean":"2023-03-03T08:43:45.105717+0000","last_became_active":"2023-03-03T08:43:44.006447+0000","last_became_peered":"2023-03-03T08:43:44.006447+0000","last_unstale":"2023-03-03T08:43:45.105717+0000","last_undegraded":"2023-03-03T08:43:45.105717+0000","last_fullsized":"2023-03-03T08:43:45.105717+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:21.767326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982538+0000","last_change":"2023-03-03T08:43:34.550889+0000","last_active":"2023-03-03T08:43:43.982538+0000","last_peered":"2023-03-03T08:43:43.982538+0000","last_clean":"2023-03-03T08:43:43.982538+0000","last_became_active":"2023-03-03T08:43:34.550605+0000","last_became_peered":"2023-03-03T08:43:34.550605+0000","last_unstale":"2023-03-03T08:43:43.982538+0000","last_undegraded":"2023-03-03T08:43:43.982538+0000","last_fullsized":"2023-03-03T08:43:43.982538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:01.118555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105567+0000","last_change":"2023-03-03T08:43:43.998231+0000","last_active":"2023-03-03T08:43:45.105567+0000","last_peered":"2023-03-03T08:43:45.105567+0000","last_clean":"2023-03-03T08:43:45.105567+0000","last_became_active":"2023-03-03T08:43:43.997941+0000","last_became_peered":"2023-03-03T08:43:43.997941+0000","last_unstale":"2023-03-03T08:43:45.105567+0000","last_undegraded":"2023-03-03T08:43:45.105567+0000","last_fullsized":"2023-03-03T08:43:45.105567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:13:32.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983100+0000","last_change":"2023-03-03T08:43:34.548229+0000","last_active":"2023-03-03T08:43:43.983100+0000","last_peered":"2023-03-03T08:43:43.983100+0000","last_clean":"2023-03-03T08:43:43.983100+0000","last_became_active":"2023-03-03T08:43:34.548013+0000","last_became_peered":"2023-03-03T08:43:34.548013+0000","last_unstale":"2023-03-03T08:43:43.983100+0000","last_undegraded":"2023-03-03T08:43:43.983100+0000","last_fullsized":"2023-03-03T08:43:43.983100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:27.793716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108186+0000","last_change":"2023-03-03T08:43:43.992834+0000","last_active":"2023-03-03T08:43:45.108186+0000","last_peered":"2023-03-03T08:43:45.108186+0000","last_clean":"2023-03-03T08:43:45.108186+0000","last_became_active":"2023-03-03T08:43:43.991577+0000","last_became_peered":"2023-03-03T08:43:43.991577+0000","last_unstale":"2023-03-03T08:43:45.108186+0000","last_undegraded":"2023-03-03T08:43:45.108186+0000","last_fullsized":"2023-03-03T08:43:45.108186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:18:37.119460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990785+0000","last_change":"2023-03-03T08:43:34.549838+0000","last_active":"2023-03-03T08:43:43.990785+0000","last_peered":"2023-03-03T08:43:43.990785+0000","last_clean":"2023-03-03T08:43:43.990785+0000","last_became_active":"2023-03-03T08:43:34.549537+0000","last_became_peered":"2023-03-03T08:43:34.549537+0000","last_unstale":"2023-03-03T08:43:43.990785+0000","last_undegraded":"2023-03-03T08:43:43.990785+0000","last_fullsized":"2023-03-03T08:43:43.990785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:36.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978412+0000","last_change":"2023-03-03T08:43:43.991488+0000","last_active":"2023-03-03T08:43:44.978412+0000","last_peered":"2023-03-03T08:43:44.978412+0000","last_clean":"2023-03-03T08:43:44.978412+0000","last_became_active":"2023-03-03T08:43:43.991139+0000","last_became_peered":"2023-03-03T08:43:43.991139+0000","last_unstale":"2023-03-03T08:43:44.978412+0000","last_undegraded":"2023-03-03T08:43:44.978412+0000","last_fullsized":"2023-03-03T08:43:44.978412+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:21.119183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983652+0000","last_change":"2023-03-03T08:43:34.533496+0000","last_active":"2023-03-03T08:43:43.983652+0000","last_peered":"2023-03-03T08:43:43.983652+0000","last_clean":"2023-03-03T08:43:43.983652+0000","last_became_active":"2023-03-03T08:43:34.532714+0000","last_became_peered":"2023-03-03T08:43:34.532714+0000","last_unstale":"2023-03-03T08:43:43.983652+0000","last_undegraded":"2023-03-03T08:43:43.983652+0000","last_fullsized":"2023-03-03T08:43:43.983652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:47.800807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105250+0000","last_change":"2023-03-03T08:43:43.996215+0000","last_active":"2023-03-03T08:43:45.105250+0000","last_peered":"2023-03-03T08:43:45.105250+0000","last_clean":"2023-03-03T08:43:45.105250+0000","last_became_active":"2023-03-03T08:43:43.995791+0000","last_became_peered":"2023-03-03T08:43:43.995791+0000","last_unstale":"2023-03-03T08:43:45.105250+0000","last_undegraded":"2023-03-03T08:43:45.105250+0000","last_fullsized":"2023-03-03T08:43:45.105250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:50:41.130427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979929+0000","last_change":"2023-03-03T08:43:34.533673+0000","last_active":"2023-03-03T08:43:43.979929+0000","last_peered":"2023-03-03T08:43:43.979929+0000","last_clean":"2023-03-03T08:43:43.979929+0000","last_became_active":"2023-03-03T08:43:34.532283+0000","last_became_peered":"2023-03-03T08:43:34.532283+0000","last_unstale":"2023-03-03T08:43:43.979929+0000","last_undegraded":"2023-03-03T08:43:43.979929+0000","last_fullsized":"2023-03-03T08:43:43.979929+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:25.591188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979205+0000","last_change":"2023-03-03T08:43:43.999435+0000","last_active":"2023-03-03T08:43:44.979205+0000","last_peered":"2023-03-03T08:43:44.979205+0000","last_clean":"2023-03-03T08:43:44.979205+0000","last_became_active":"2023-03-03T08:43:43.997037+0000","last_became_peered":"2023-03-03T08:43:43.997037+0000","last_unstale":"2023-03-03T08:43:44.979205+0000","last_undegraded":"2023-03-03T08:43:44.979205+0000","last_fullsized":"2023-03-03T08:43:44.979205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:54.176343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980585+0000","last_change":"2023-03-03T08:43:34.537664+0000","last_active":"2023-03-03T08:43:43.980585+0000","last_peered":"2023-03-03T08:43:43.980585+0000","last_clean":"2023-03-03T08:43:43.980585+0000","last_became_active":"2023-03-03T08:43:34.537214+0000","last_became_peered":"2023-03-03T08:43:34.537214+0000","last_unstale":"2023-03-03T08:43:43.980585+0000","last_undegraded":"2023-03-03T08:43:43.980585+0000","last_fullsized":"2023-03-03T08:43:43.980585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:54:09.222135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106227+0000","last_change":"2023-03-03T08:43:44.000140+0000","last_active":"2023-03-03T08:43:45.106227+0000","last_peered":"2023-03-03T08:43:45.106227+0000","last_clean":"2023-03-03T08:43:45.106227+0000","last_became_active":"2023-03-03T08:43:43.999778+0000","last_became_peered":"2023-03-03T08:43:43.999778+0000","last_unstale":"2023-03-03T08:43:45.106227+0000","last_undegraded":"2023-03-03T08:43:45.106227+0000","last_fullsized":"2023-03-03T08:43:45.106227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:29:10.745759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987366+0000","last_change":"2023-03-03T08:43:34.543620+0000","last_active":"2023-03-03T08:43:43.987366+0000","last_peered":"2023-03-03T08:43:43.987366+0000","last_clean":"2023-03-03T08:43:43.987366+0000","last_became_active":"2023-03-03T08:43:34.543370+0000","last_became_peered":"2023-03-03T08:43:34.543370+0000","last_unstale":"2023-03-03T08:43:43.987366+0000","last_undegraded":"2023-03-03T08:43:43.987366+0000","last_fullsized":"2023-03-03T08:43:43.987366+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:40:32.808203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979449+0000","last_change":"2023-03-03T08:43:43.995084+0000","last_active":"2023-03-03T08:43:44.979449+0000","last_peered":"2023-03-03T08:43:44.979449+0000","last_clean":"2023-03-03T08:43:44.979449+0000","last_became_active":"2023-03-03T08:43:43.994824+0000","last_became_peered":"2023-03-03T08:43:43.994824+0000","last_unstale":"2023-03-03T08:43:44.979449+0000","last_undegraded":"2023-03-03T08:43:44.979449+0000","last_fullsized":"2023-03-03T08:43:44.979449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:04:17.481969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985302+0000","last_change":"2023-03-03T08:43:34.538467+0000","last_active":"2023-03-03T08:43:43.985302+0000","last_peered":"2023-03-03T08:43:43.985302+0000","last_clean":"2023-03-03T08:43:43.985302+0000","last_became_active":"2023-03-03T08:43:34.538207+0000","last_became_peered":"2023-03-03T08:43:34.538207+0000","last_unstale":"2023-03-03T08:43:43.985302+0000","last_undegraded":"2023-03-03T08:43:43.985302+0000","last_fullsized":"2023-03-03T08:43:43.985302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:14.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980577+0000","last_change":"2023-03-03T08:43:43.996076+0000","last_active":"2023-03-03T08:43:44.980577+0000","last_peered":"2023-03-03T08:43:44.980577+0000","last_clean":"2023-03-03T08:43:44.980577+0000","last_became_active":"2023-03-03T08:43:43.995799+0000","last_became_peered":"2023-03-03T08:43:43.995799+0000","last_unstale":"2023-03-03T08:43:44.980577+0000","last_undegraded":"2023-03-03T08:43:44.980577+0000","last_fullsized":"2023-03-03T08:43:44.980577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:35:35.889895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986289+0000","last_change":"2023-03-03T08:43:34.544721+0000","last_active":"2023-03-03T08:43:43.986289+0000","last_peered":"2023-03-03T08:43:43.986289+0000","last_clean":"2023-03-03T08:43:43.986289+0000","last_became_active":"2023-03-03T08:43:34.544461+0000","last_became_peered":"2023-03-03T08:43:34.544461+0000","last_unstale":"2023-03-03T08:43:43.986289+0000","last_undegraded":"2023-03-03T08:43:43.986289+0000","last_fullsized":"2023-03-03T08:43:43.986289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:52:11.890779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980078+0000","last_change":"2023-03-03T08:43:44.006352+0000","last_active":"2023-03-03T08:43:44.980078+0000","last_peered":"2023-03-03T08:43:44.980078+0000","last_clean":"2023-03-03T08:43:44.980078+0000","last_became_active":"2023-03-03T08:43:44.005802+0000","last_became_peered":"2023-03-03T08:43:44.005802+0000","last_unstale":"2023-03-03T08:43:44.980078+0000","last_undegraded":"2023-03-03T08:43:44.980078+0000","last_fullsized":"2023-03-03T08:43:44.980078+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:52.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986425+0000","last_change":"2023-03-03T08:43:34.541669+0000","last_active":"2023-03-03T08:43:43.986425+0000","last_peered":"2023-03-03T08:43:43.986425+0000","last_clean":"2023-03-03T08:43:43.986425+0000","last_became_active":"2023-03-03T08:43:34.541104+0000","last_became_peered":"2023-03-03T08:43:34.541104+0000","last_unstale":"2023-03-03T08:43:43.986425+0000","last_undegraded":"2023-03-03T08:43:43.986425+0000","last_fullsized":"2023-03-03T08:43:43.986425+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980059+0000","last_change":"2023-03-03T08:43:43.998937+0000","last_active":"2023-03-03T08:43:44.980059+0000","last_peered":"2023-03-03T08:43:44.980059+0000","last_clean":"2023-03-03T08:43:44.980059+0000","last_became_active":"2023-03-03T08:43:43.998164+0000","last_became_peered":"2023-03-03T08:43:43.998164+0000","last_unstale":"2023-03-03T08:43:44.980059+0000","last_undegraded":"2023-03-03T08:43:44.980059+0000","last_fullsized":"2023-03-03T08:43:44.980059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:53.741704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.546272+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.546006+0000","last_became_peered":"2023-03-03T08:43:34.546006+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:15.121761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":118397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":119470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9336,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362504,"statfs":{"total":96636764160,"available":96627204096,"internally_reserved":0,"allocated":901120,"data_stored":710394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8689999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.806}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8244,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363596,"statfs":{"total":96636764160,"available":96628322304,"internally_reserved":0,"allocated":307200,"data_stored":118565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8188,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363652,"statfs":{"total":96636764160,"available":96628379648,"internally_reserved":0,"allocated":315392,"data_stored":121989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":120069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":121018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9260,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362580,"statfs":{"total":96636764160,"available":96627281920,"internally_reserved":0,"allocated":888832,"data_stored":709827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:43:52.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:43:52.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.356+0000 7f72577aa700 1 Processor -- start 2023-03-03T08:43:52.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.357+0000 7f72577aa700 1 -- start start 2023-03-03T08:43:52.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.357+0000 7f72577aa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f72500c2930 0x7f7250157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.357+0000 7f72577aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 0x7f7250157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f72577aa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f725006d430 con 0x7f7250151d40 2023-03-03T08:43:52.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f72577aa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725006c2a0 con 0x7f72500c2930 2023-03-03T08:43:52.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f72577aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725006cb00 con 0x7f725006c480 2023-03-03T08:43:52.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7254d45700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f72500c2930 0x7f7250157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7254d45700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f72500c2930 0x7f7250157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51824/0 (socket says 172.21.15.7:51824) 2023-03-03T08:43:52.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7254d45700 1 -- 172.21.15.7:0/1359376537 learned_addr learned my addr 172.21.15.7:0/1359376537 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:52.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7255d47700 1 --2- 172.21.15.7:0/1359376537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 0x7f7250157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7254d45700 1 -- 172.21.15.7:0/1359376537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 msgr2=0x7f7250157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:52.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7254d45700 1 --2- 172.21.15.7:0/1359376537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 0x7f7250157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7254d45700 1 -- 172.21.15.7:0/1359376537 >> v1:172.21.15.143:6789/0 conn(0x7f7250151d40 legacy=0x7f72500736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:52.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7254d45700 1 -- 172.21.15.7:0/1359376537 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725006cde0 con 0x7f72500c2930 2023-03-03T08:43:52.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7254d45700 1 --2- 172.21.15.7:0/1359376537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72500c2930 0x7f7250157a90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f724c0098f0 tx=0x7f724c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d473240654cee6c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:52.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7247fff700 1 -- 172.21.15.7:0/1359376537 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c00b8a0 con 0x7f72500c2930 2023-03-03T08:43:52.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7247fff700 1 -- 172.21.15.7:0/1359376537 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724c00ba00 con 0x7f72500c2930 2023-03-03T08:43:52.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.358+0000 7f7247fff700 1 -- 172.21.15.7:0/1359376537 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c00bd90 con 0x7f72500c2930 2023-03-03T08:43:52.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.359+0000 7f72577aa700 1 -- 172.21.15.7:0/1359376537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72500c2930 msgr2=0x7f7250157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.359+0000 7f72577aa700 1 --2- 172.21.15.7:0/1359376537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72500c2930 0x7f7250157a90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f724c0098f0 tx=0x7f724c009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:52.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.359+0000 7f72577aa700 1 -- 172.21.15.7:0/1359376537 shutdown_connections 2023-03-03T08:43:52.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.359+0000 7f72577aa700 1 --2- 172.21.15.7:0/1359376537 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 0x7f7250157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.359+0000 7f72577aa700 1 --2- 172.21.15.7:0/1359376537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72500c2930 0x7f7250157a90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.359+0000 7f72577aa700 1 -- 172.21.15.7:0/1359376537 >> 172.21.15.7:0/1359376537 conn(0x7f72500a94d0 msgr2=0x7f72500a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:52.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.359+0000 7f72577aa700 1 -- 172.21.15.7:0/1359376537 shutdown_connections 2023-03-03T08:43:52.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.359+0000 7f72577aa700 1 -- 172.21.15.7:0/1359376537 wait complete. 2023-03-03T08:43:52.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.361+0000 7f72577aa700 1 Processor -- start 2023-03-03T08:43:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.361+0000 7f72577aa700 1 -- start start 2023-03-03T08:43:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.361+0000 7f72577aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 0x7f72501483a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.361+0000 7f72577aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72500c2930 0x7f72501dff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.361+0000 7f72577aa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7250151d40 0x7f725014dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f72577aa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725014d910 con 0x7f7250151d40 2023-03-03T08:43:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f72577aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72501eb2d0 con 0x7f72500c2930 2023-03-03T08:43:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f72577aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72501489d0 con 0x7f725006c480 2023-03-03T08:43:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255d47700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7250151d40 0x7f725014dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255d47700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7250151d40 0x7f725014dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51830/0 (socket says 172.21.15.7:51830) 2023-03-03T08:43:52.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255d47700 1 -- 172.21.15.7:0/773033619 learned_addr learned my addr 172.21.15.7:0/773033619 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:52.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255546700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 0x7f72501483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7254d45700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72500c2930 0x7f72501dff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255d47700 1 -- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 msgr2=0x7f72501483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255d47700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 0x7f72501483a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255d47700 1 -- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72500c2930 msgr2=0x7f72501dff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255d47700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72500c2930 0x7f72501dff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255d47700 1 -- 172.21.15.7:0/773033619 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c009550 con 0x7f7250151d40 2023-03-03T08:43:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7254d45700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72500c2930 0x7f72501dff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255546700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 0x7f72501483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7255d47700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7250151d40 0x7f725014dda0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7248007e60 tx=0x7f724800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7245ffb700 1 -- 172.21.15.7:0/773033619 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724800ede0 con 0x7f7250151d40 2023-03-03T08:43:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7245ffb700 1 -- 172.21.15.7:0/773033619 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7248014420 con 0x7f7250151d40 2023-03-03T08:43:52.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f72577aa700 1 -- 172.21.15.7:0/773033619 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7250073e00 con 0x7f7250151d40 2023-03-03T08:43:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f7245ffb700 1 -- 172.21.15.7:0/773033619 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72480147d0 con 0x7f7250151d40 2023-03-03T08:43:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.362+0000 7f72577aa700 1 -- 172.21.15.7:0/773033619 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725006d0c0 con 0x7f7250151d40 2023-03-03T08:43:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.363+0000 7f72577aa700 1 -- 172.21.15.7:0/773033619 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725006bea0 con 0x7f7250151d40 2023-03-03T08:43:52.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.364+0000 7f7245ffb700 1 -- 172.21.15.7:0/773033619 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7248012050 con 0x7f7250151d40 2023-03-03T08:43:52.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.364+0000 7f7245ffb700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72340682f0 0x7f723406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.364+0000 7f7245ffb700 1 -- 172.21.15.7:0/773033619 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f72480865f0 con 0x7f7250151d40 2023-03-03T08:43:52.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.364+0000 7f7255546700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72340682f0 0x7f723406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.365+0000 7f7255546700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72340682f0 0x7f723406a7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f72500c4770 tx=0x7f7240009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:52.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.368+0000 7f7245ffb700 1 -- 172.21.15.7:0/773033619 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7248052b20 con 0x7f7250151d40 2023-03-03T08:43:52.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.556+0000 7f72577aa700 1 -- 172.21.15.7:0/773033619 --> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72501ded10 con 0x7f72340682f0 2023-03-03T08:43:52.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.571+0000 7f7245ffb700 1 -- 172.21.15.7:0/773033619 <== mgr.4102 v2:172.21.15.143:6832/96367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+524975 (secure 0 0 0) 0x7f7240001c40 con 0x7f72340682f0 2023-03-03T08:43:52.578 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:52.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 -- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72340682f0 msgr2=0x7f723406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72340682f0 0x7f723406a7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f72500c4770 tx=0x7f7240009040 comp rx=0 tx=0).stop 2023-03-03T08:43:52.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 -- 172.21.15.7:0/773033619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7250151d40 msgr2=0x7f725014dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7250151d40 0x7f725014dda0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7248007e60 tx=0x7f724800e3c0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 -- 172.21.15.7:0/773033619 shutdown_connections 2023-03-03T08:43:52.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72340682f0 0x7f723406a7a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7250151d40 0x7f725014dda0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f725006c480 0x7f72501483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 --2- 172.21.15.7:0/773033619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72500c2930 0x7f72501dff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 -- 172.21.15.7:0/773033619 >> 172.21.15.7:0/773033619 conn(0x7f72500a94d0 msgr2=0x7f72501558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:52.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 -- 172.21.15.7:0/773033619 shutdown_connections 2023-03-03T08:43:52.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.575+0000 7f72577aa700 1 -- 172.21.15.7:0/773033619 wait complete. 2023-03-03T08:43:52.583 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-03-03T08:43:52.590 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-03-03T08:43:51.109491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595028,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68052,"kb_used_data":3540,"kb_used_omap":0,"kb_used_meta":64448,"kb_avail":754906668,"statfs":{"total":773094113280,"available":773024428032,"internally_reserved":0,"allocated":3624960,"data_stored":2139729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65994752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.554300"},"pg_stats":[{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980166+0000","last_change":"2023-03-03T08:43:43.997091+0000","last_active":"2023-03-03T08:43:44.980166+0000","last_peered":"2023-03-03T08:43:44.980166+0000","last_clean":"2023-03-03T08:43:44.980166+0000","last_became_active":"2023-03-03T08:43:43.996805+0000","last_became_peered":"2023-03-03T08:43:43.996805+0000","last_unstale":"2023-03-03T08:43:44.980166+0000","last_undegraded":"2023-03-03T08:43:44.980166+0000","last_fullsized":"2023-03-03T08:43:44.980166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982505+0000","last_change":"2023-03-03T08:43:34.546828+0000","last_active":"2023-03-03T08:43:43.982505+0000","last_peered":"2023-03-03T08:43:43.982505+0000","last_clean":"2023-03-03T08:43:43.982505+0000","last_became_active":"2023-03-03T08:43:34.546559+0000","last_became_peered":"2023-03-03T08:43:34.546559+0000","last_unstale":"2023-03-03T08:43:43.982505+0000","last_undegraded":"2023-03-03T08:43:43.982505+0000","last_fullsized":"2023-03-03T08:43:43.982505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:21:14.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106061+0000","last_change":"2023-03-03T08:43:43.995073+0000","last_active":"2023-03-03T08:43:45.106061+0000","last_peered":"2023-03-03T08:43:45.106061+0000","last_clean":"2023-03-03T08:43:45.106061+0000","last_became_active":"2023-03-03T08:43:43.994797+0000","last_became_peered":"2023-03-03T08:43:43.994797+0000","last_unstale":"2023-03-03T08:43:45.106061+0000","last_undegraded":"2023-03-03T08:43:45.106061+0000","last_fullsized":"2023-03-03T08:43:45.106061+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:21:52.642936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982397+0000","last_change":"2023-03-03T08:43:34.552931+0000","last_active":"2023-03-03T08:43:43.982397+0000","last_peered":"2023-03-03T08:43:43.982397+0000","last_clean":"2023-03-03T08:43:43.982397+0000","last_became_active":"2023-03-03T08:43:34.552623+0000","last_became_peered":"2023-03-03T08:43:34.552623+0000","last_unstale":"2023-03-03T08:43:43.982397+0000","last_undegraded":"2023-03-03T08:43:43.982397+0000","last_fullsized":"2023-03-03T08:43:43.982397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:07:43.954897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979088+0000","last_change":"2023-03-03T08:43:43.999245+0000","last_active":"2023-03-03T08:43:44.979088+0000","last_peered":"2023-03-03T08:43:44.979088+0000","last_clean":"2023-03-03T08:43:44.979088+0000","last_became_active":"2023-03-03T08:43:43.998592+0000","last_became_peered":"2023-03-03T08:43:43.998592+0000","last_unstale":"2023-03-03T08:43:44.979088+0000","last_undegraded":"2023-03-03T08:43:44.979088+0000","last_fullsized":"2023-03-03T08:43:44.979088+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:23.788724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991054+0000","last_change":"2023-03-03T08:43:34.544408+0000","last_active":"2023-03-03T08:43:43.991054+0000","last_peered":"2023-03-03T08:43:43.991054+0000","last_clean":"2023-03-03T08:43:43.991054+0000","last_became_active":"2023-03-03T08:43:34.543977+0000","last_became_peered":"2023-03-03T08:43:34.543977+0000","last_unstale":"2023-03-03T08:43:43.991054+0000","last_undegraded":"2023-03-03T08:43:43.991054+0000","last_fullsized":"2023-03-03T08:43:43.991054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.074082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105254+0000","last_change":"2023-03-03T08:43:43.994545+0000","last_active":"2023-03-03T08:43:45.105254+0000","last_peered":"2023-03-03T08:43:45.105254+0000","last_clean":"2023-03-03T08:43:45.105254+0000","last_became_active":"2023-03-03T08:43:43.994271+0000","last_became_peered":"2023-03-03T08:43:43.994271+0000","last_unstale":"2023-03-03T08:43:45.105254+0000","last_undegraded":"2023-03-03T08:43:45.105254+0000","last_fullsized":"2023-03-03T08:43:45.105254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:15.925960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989064+0000","last_change":"2023-03-03T08:43:34.538786+0000","last_active":"2023-03-03T08:43:43.989064+0000","last_peered":"2023-03-03T08:43:43.989064+0000","last_clean":"2023-03-03T08:43:43.989064+0000","last_became_active":"2023-03-03T08:43:34.538494+0000","last_became_peered":"2023-03-03T08:43:34.538494+0000","last_unstale":"2023-03-03T08:43:43.989064+0000","last_undegraded":"2023-03-03T08:43:43.989064+0000","last_fullsized":"2023-03-03T08:43:43.989064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:53.838286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105391+0000","last_change":"2023-03-03T08:43:43.993678+0000","last_active":"2023-03-03T08:43:45.105391+0000","last_peered":"2023-03-03T08:43:45.105391+0000","last_clean":"2023-03-03T08:43:45.105391+0000","last_became_active":"2023-03-03T08:43:43.993402+0000","last_became_peered":"2023-03-03T08:43:43.993402+0000","last_unstale":"2023-03-03T08:43:45.105391+0000","last_undegraded":"2023-03-03T08:43:45.105391+0000","last_fullsized":"2023-03-03T08:43:45.105391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:31:00.747216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983479+0000","last_change":"2023-03-03T08:43:34.547197+0000","last_active":"2023-03-03T08:43:43.983479+0000","last_peered":"2023-03-03T08:43:43.983479+0000","last_clean":"2023-03-03T08:43:43.983479+0000","last_became_active":"2023-03-03T08:43:34.546927+0000","last_became_peered":"2023-03-03T08:43:34.546927+0000","last_unstale":"2023-03-03T08:43:43.983479+0000","last_undegraded":"2023-03-03T08:43:43.983479+0000","last_fullsized":"2023-03-03T08:43:43.983479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:21:34.845968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984318+0000","last_change":"2023-03-03T08:43:34.540812+0000","last_active":"2023-03-03T08:43:43.984318+0000","last_peered":"2023-03-03T08:43:43.984318+0000","last_clean":"2023-03-03T08:43:43.984318+0000","last_became_active":"2023-03-03T08:43:34.540549+0000","last_became_peered":"2023-03-03T08:43:34.540549+0000","last_unstale":"2023-03-03T08:43:43.984318+0000","last_undegraded":"2023-03-03T08:43:43.984318+0000","last_fullsized":"2023-03-03T08:43:43.984318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982875+0000","last_change":"2023-03-03T08:43:34.108426+0000","last_active":"2023-03-03T08:43:43.982875+0000","last_peered":"2023-03-03T08:43:43.982875+0000","last_clean":"2023-03-03T08:43:43.982875+0000","last_became_active":"2023-03-03T08:43:34.108009+0000","last_became_peered":"2023-03-03T08:43:34.108009+0000","last_unstale":"2023-03-03T08:43:43.982875+0000","last_undegraded":"2023-03-03T08:43:43.982875+0000","last_fullsized":"2023-03-03T08:43:43.982875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:53.632744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979479+0000","last_change":"2023-03-03T08:43:43.575708+0000","last_active":"2023-03-03T08:43:44.979479+0000","last_peered":"2023-03-03T08:43:44.979479+0000","last_clean":"2023-03-03T08:43:44.979479+0000","last_became_active":"2023-03-03T08:43:43.575417+0000","last_became_peered":"2023-03-03T08:43:43.575417+0000","last_unstale":"2023-03-03T08:43:44.979479+0000","last_undegraded":"2023-03-03T08:43:44.979479+0000","last_fullsized":"2023-03-03T08:43:44.979479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:25.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106018+0000","last_change":"2023-03-03T08:43:43.999274+0000","last_active":"2023-03-03T08:43:45.106018+0000","last_peered":"2023-03-03T08:43:45.106018+0000","last_clean":"2023-03-03T08:43:45.106018+0000","last_became_active":"2023-03-03T08:43:43.998823+0000","last_became_peered":"2023-03-03T08:43:43.998823+0000","last_unstale":"2023-03-03T08:43:45.106018+0000","last_undegraded":"2023-03-03T08:43:45.106018+0000","last_fullsized":"2023-03-03T08:43:45.106018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:48.799261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992918+0000","last_change":"2023-03-03T08:43:34.532482+0000","last_active":"2023-03-03T08:43:43.992918+0000","last_peered":"2023-03-03T08:43:43.992918+0000","last_clean":"2023-03-03T08:43:43.992918+0000","last_became_active":"2023-03-03T08:43:34.531751+0000","last_became_peered":"2023-03-03T08:43:34.531751+0000","last_unstale":"2023-03-03T08:43:43.992918+0000","last_undegraded":"2023-03-03T08:43:43.992918+0000","last_fullsized":"2023-03-03T08:43:43.992918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:47:45.214945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982691+0000","last_change":"2023-03-03T08:43:34.109782+0000","last_active":"2023-03-03T08:43:43.982691+0000","last_peered":"2023-03-03T08:43:43.982691+0000","last_clean":"2023-03-03T08:43:43.982691+0000","last_became_active":"2023-03-03T08:43:34.109513+0000","last_became_peered":"2023-03-03T08:43:34.109513+0000","last_unstale":"2023-03-03T08:43:43.982691+0000","last_undegraded":"2023-03-03T08:43:43.982691+0000","last_fullsized":"2023-03-03T08:43:43.982691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:16.185161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105634+0000","last_change":"2023-03-03T08:43:43.576885+0000","last_active":"2023-03-03T08:43:45.105634+0000","last_peered":"2023-03-03T08:43:45.105634+0000","last_clean":"2023-03-03T08:43:45.105634+0000","last_became_active":"2023-03-03T08:43:43.576577+0000","last_became_peered":"2023-03-03T08:43:43.576577+0000","last_unstale":"2023-03-03T08:43:45.105634+0000","last_undegraded":"2023-03-03T08:43:45.105634+0000","last_fullsized":"2023-03-03T08:43:45.105634+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:26.501956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978988+0000","last_change":"2023-03-03T08:43:43.992020+0000","last_active":"2023-03-03T08:43:44.978988+0000","last_peered":"2023-03-03T08:43:44.978988+0000","last_clean":"2023-03-03T08:43:44.978988+0000","last_became_active":"2023-03-03T08:43:43.991674+0000","last_became_peered":"2023-03-03T08:43:43.991674+0000","last_unstale":"2023-03-03T08:43:44.978988+0000","last_undegraded":"2023-03-03T08:43:44.978988+0000","last_fullsized":"2023-03-03T08:43:44.978988+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:54:30.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985919+0000","last_change":"2023-03-03T08:43:34.535451+0000","last_active":"2023-03-03T08:43:43.985919+0000","last_peered":"2023-03-03T08:43:43.985919+0000","last_clean":"2023-03-03T08:43:43.985919+0000","last_became_active":"2023-03-03T08:43:34.535129+0000","last_became_peered":"2023-03-03T08:43:34.535129+0000","last_unstale":"2023-03-03T08:43:43.985919+0000","last_undegraded":"2023-03-03T08:43:43.985919+0000","last_fullsized":"2023-03-03T08:43:43.985919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:30.645128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981082+0000","last_change":"2023-03-03T08:43:34.117851+0000","last_active":"2023-03-03T08:43:43.981082+0000","last_peered":"2023-03-03T08:43:43.981082+0000","last_clean":"2023-03-03T08:43:43.981082+0000","last_became_active":"2023-03-03T08:43:34.117548+0000","last_became_peered":"2023-03-03T08:43:34.117548+0000","last_unstale":"2023-03-03T08:43:43.981082+0000","last_undegraded":"2023-03-03T08:43:43.981082+0000","last_fullsized":"2023-03-03T08:43:43.981082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:20.039817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106904+0000","last_change":"2023-03-03T08:43:43.569082+0000","last_active":"2023-03-03T08:43:45.106904+0000","last_peered":"2023-03-03T08:43:45.106904+0000","last_clean":"2023-03-03T08:43:45.106904+0000","last_became_active":"2023-03-03T08:43:43.568838+0000","last_became_peered":"2023-03-03T08:43:43.568838+0000","last_unstale":"2023-03-03T08:43:45.106904+0000","last_undegraded":"2023-03-03T08:43:45.106904+0000","last_fullsized":"2023-03-03T08:43:45.106904+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:47.610848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979556+0000","last_change":"2023-03-03T08:43:43.995210+0000","last_active":"2023-03-03T08:43:44.979556+0000","last_peered":"2023-03-03T08:43:44.979556+0000","last_clean":"2023-03-03T08:43:44.979556+0000","last_became_active":"2023-03-03T08:43:43.994899+0000","last_became_peered":"2023-03-03T08:43:43.994899+0000","last_unstale":"2023-03-03T08:43:44.979556+0000","last_undegraded":"2023-03-03T08:43:44.979556+0000","last_fullsized":"2023-03-03T08:43:44.979556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:37:11.974933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985908+0000","last_change":"2023-03-03T08:43:34.547322+0000","last_active":"2023-03-03T08:43:43.985908+0000","last_peered":"2023-03-03T08:43:43.985908+0000","last_clean":"2023-03-03T08:43:43.985908+0000","last_became_active":"2023-03-03T08:43:34.546692+0000","last_became_peered":"2023-03-03T08:43:34.546692+0000","last_unstale":"2023-03-03T08:43:43.985908+0000","last_undegraded":"2023-03-03T08:43:43.985908+0000","last_fullsized":"2023-03-03T08:43:43.985908+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:11:26.366920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982836+0000","last_change":"2023-03-03T08:43:34.110187+0000","last_active":"2023-03-03T08:43:43.982836+0000","last_peered":"2023-03-03T08:43:43.982836+0000","last_clean":"2023-03-03T08:43:43.982836+0000","last_became_active":"2023-03-03T08:43:34.109845+0000","last_became_peered":"2023-03-03T08:43:34.109845+0000","last_unstale":"2023-03-03T08:43:43.982836+0000","last_undegraded":"2023-03-03T08:43:43.982836+0000","last_fullsized":"2023-03-03T08:43:43.982836+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:33.507500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105532+0000","last_change":"2023-03-03T08:43:43.576805+0000","last_active":"2023-03-03T08:43:45.105532+0000","last_peered":"2023-03-03T08:43:45.105532+0000","last_clean":"2023-03-03T08:43:45.105532+0000","last_became_active":"2023-03-03T08:43:43.576241+0000","last_became_peered":"2023-03-03T08:43:43.576241+0000","last_unstale":"2023-03-03T08:43:45.105532+0000","last_undegraded":"2023-03-03T08:43:45.105532+0000","last_fullsized":"2023-03-03T08:43:45.105532+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:34:35.821307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105632+0000","last_change":"2023-03-03T08:43:43.991142+0000","last_active":"2023-03-03T08:43:45.105632+0000","last_peered":"2023-03-03T08:43:45.105632+0000","last_clean":"2023-03-03T08:43:45.105632+0000","last_became_active":"2023-03-03T08:43:43.990868+0000","last_became_peered":"2023-03-03T08:43:43.990868+0000","last_unstale":"2023-03-03T08:43:45.105632+0000","last_undegraded":"2023-03-03T08:43:45.105632+0000","last_fullsized":"2023-03-03T08:43:45.105632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:58.831806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106429+0000","last_change":"2023-03-03T08:43:43.995618+0000","last_active":"2023-03-03T08:43:45.106429+0000","last_peered":"2023-03-03T08:43:45.106429+0000","last_clean":"2023-03-03T08:43:45.106429+0000","last_became_active":"2023-03-03T08:43:43.995350+0000","last_became_peered":"2023-03-03T08:43:43.995350+0000","last_unstale":"2023-03-03T08:43:45.106429+0000","last_undegraded":"2023-03-03T08:43:45.106429+0000","last_fullsized":"2023-03-03T08:43:45.106429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:46.445223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988843+0000","last_change":"2023-03-03T08:43:34.541172+0000","last_active":"2023-03-03T08:43:43.988843+0000","last_peered":"2023-03-03T08:43:43.988843+0000","last_clean":"2023-03-03T08:43:43.988843+0000","last_became_active":"2023-03-03T08:43:34.540739+0000","last_became_peered":"2023-03-03T08:43:34.540739+0000","last_unstale":"2023-03-03T08:43:43.988843+0000","last_undegraded":"2023-03-03T08:43:43.988843+0000","last_fullsized":"2023-03-03T08:43:43.988843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:40.058265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987475+0000","last_change":"2023-03-03T08:43:34.112667+0000","last_active":"2023-03-03T08:43:43.987475+0000","last_peered":"2023-03-03T08:43:43.987475+0000","last_clean":"2023-03-03T08:43:43.987475+0000","last_became_active":"2023-03-03T08:43:34.112386+0000","last_became_peered":"2023-03-03T08:43:34.112386+0000","last_unstale":"2023-03-03T08:43:43.987475+0000","last_undegraded":"2023-03-03T08:43:43.987475+0000","last_fullsized":"2023-03-03T08:43:43.987475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:40.504392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980167+0000","last_change":"2023-03-03T08:43:43.573962+0000","last_active":"2023-03-03T08:43:44.980167+0000","last_peered":"2023-03-03T08:43:44.980167+0000","last_clean":"2023-03-03T08:43:44.980167+0000","last_became_active":"2023-03-03T08:43:43.573200+0000","last_became_peered":"2023-03-03T08:43:43.573200+0000","last_unstale":"2023-03-03T08:43:44.980167+0000","last_undegraded":"2023-03-03T08:43:44.980167+0000","last_fullsized":"2023-03-03T08:43:44.980167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:29.511681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984698+0000","last_change":"2023-03-03T08:43:34.537850+0000","last_active":"2023-03-03T08:43:43.984698+0000","last_peered":"2023-03-03T08:43:43.984698+0000","last_clean":"2023-03-03T08:43:43.984698+0000","last_became_active":"2023-03-03T08:43:34.537550+0000","last_became_peered":"2023-03-03T08:43:34.537550+0000","last_unstale":"2023-03-03T08:43:43.984698+0000","last_undegraded":"2023-03-03T08:43:43.984698+0000","last_fullsized":"2023-03-03T08:43:43.984698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:37:36.893757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990299+0000","last_change":"2023-03-03T08:43:34.109127+0000","last_active":"2023-03-03T08:43:43.990299+0000","last_peered":"2023-03-03T08:43:43.990299+0000","last_clean":"2023-03-03T08:43:43.990299+0000","last_became_active":"2023-03-03T08:43:34.108869+0000","last_became_peered":"2023-03-03T08:43:34.108869+0000","last_unstale":"2023-03-03T08:43:43.990299+0000","last_undegraded":"2023-03-03T08:43:43.990299+0000","last_fullsized":"2023-03-03T08:43:43.990299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:19:00.275620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981298+0000","last_change":"2023-03-03T08:43:29.354627+0000","last_active":"2023-03-03T08:43:43.981298+0000","last_peered":"2023-03-03T08:43:43.981298+0000","last_clean":"2023-03-03T08:43:43.981298+0000","last_became_active":"2023-03-03T08:43:29.354271+0000","last_became_peered":"2023-03-03T08:43:29.354271+0000","last_unstale":"2023-03-03T08:43:43.981298+0000","last_undegraded":"2023-03-03T08:43:43.981298+0000","last_fullsized":"2023-03-03T08:43:43.981298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:05.946053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979432+0000","last_change":"2023-03-03T08:43:43.575567+0000","last_active":"2023-03-03T08:43:44.979432+0000","last_peered":"2023-03-03T08:43:44.979432+0000","last_clean":"2023-03-03T08:43:44.979432+0000","last_became_active":"2023-03-03T08:43:43.575276+0000","last_became_peered":"2023-03-03T08:43:43.575276+0000","last_unstale":"2023-03-03T08:43:44.979432+0000","last_undegraded":"2023-03-03T08:43:44.979432+0000","last_fullsized":"2023-03-03T08:43:44.979432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:12.876790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107250+0000","last_change":"2023-03-03T08:43:43.998276+0000","last_active":"2023-03-03T08:43:45.107250+0000","last_peered":"2023-03-03T08:43:45.107250+0000","last_clean":"2023-03-03T08:43:45.107250+0000","last_became_active":"2023-03-03T08:43:43.996018+0000","last_became_peered":"2023-03-03T08:43:43.996018+0000","last_unstale":"2023-03-03T08:43:45.107250+0000","last_undegraded":"2023-03-03T08:43:45.107250+0000","last_fullsized":"2023-03-03T08:43:45.107250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:08.588278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988844+0000","last_change":"2023-03-03T08:43:34.539018+0000","last_active":"2023-03-03T08:43:43.988844+0000","last_peered":"2023-03-03T08:43:43.988844+0000","last_clean":"2023-03-03T08:43:43.988844+0000","last_became_active":"2023-03-03T08:43:34.538375+0000","last_became_peered":"2023-03-03T08:43:34.538375+0000","last_unstale":"2023-03-03T08:43:43.988844+0000","last_undegraded":"2023-03-03T08:43:43.988844+0000","last_fullsized":"2023-03-03T08:43:43.988844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:29:02.601410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990482+0000","last_change":"2023-03-03T08:43:29.354519+0000","last_active":"2023-03-03T08:43:43.990482+0000","last_peered":"2023-03-03T08:43:43.990482+0000","last_clean":"2023-03-03T08:43:43.990482+0000","last_became_active":"2023-03-03T08:43:29.354213+0000","last_became_peered":"2023-03-03T08:43:29.354213+0000","last_unstale":"2023-03-03T08:43:43.990482+0000","last_undegraded":"2023-03-03T08:43:43.990482+0000","last_fullsized":"2023-03-03T08:43:43.990482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:00.406507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990028+0000","last_change":"2023-03-03T08:43:34.110105+0000","last_active":"2023-03-03T08:43:43.990028+0000","last_peered":"2023-03-03T08:43:43.990028+0000","last_clean":"2023-03-03T08:43:43.990028+0000","last_became_active":"2023-03-03T08:43:34.109736+0000","last_became_peered":"2023-03-03T08:43:34.109736+0000","last_unstale":"2023-03-03T08:43:43.990028+0000","last_undegraded":"2023-03-03T08:43:43.990028+0000","last_fullsized":"2023-03-03T08:43:43.990028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:30.557847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105361+0000","last_change":"2023-03-03T08:43:43.569433+0000","last_active":"2023-03-03T08:43:45.105361+0000","last_peered":"2023-03-03T08:43:45.105361+0000","last_clean":"2023-03-03T08:43:45.105361+0000","last_became_active":"2023-03-03T08:43:43.568711+0000","last_became_peered":"2023-03-03T08:43:43.568711+0000","last_unstale":"2023-03-03T08:43:45.105361+0000","last_undegraded":"2023-03-03T08:43:45.105361+0000","last_fullsized":"2023-03-03T08:43:45.105361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:53.170557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105548+0000","last_change":"2023-03-03T08:43:43.999241+0000","last_active":"2023-03-03T08:43:45.105548+0000","last_peered":"2023-03-03T08:43:45.105548+0000","last_clean":"2023-03-03T08:43:45.105548+0000","last_became_active":"2023-03-03T08:43:43.998992+0000","last_became_peered":"2023-03-03T08:43:43.998992+0000","last_unstale":"2023-03-03T08:43:45.105548+0000","last_undegraded":"2023-03-03T08:43:45.105548+0000","last_fullsized":"2023-03-03T08:43:45.105548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:22.623700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991092+0000","last_change":"2023-03-03T08:43:34.535435+0000","last_active":"2023-03-03T08:43:43.991092+0000","last_peered":"2023-03-03T08:43:43.991092+0000","last_clean":"2023-03-03T08:43:43.991092+0000","last_became_active":"2023-03-03T08:43:34.535123+0000","last_became_peered":"2023-03-03T08:43:34.535123+0000","last_unstale":"2023-03-03T08:43:43.991092+0000","last_undegraded":"2023-03-03T08:43:43.991092+0000","last_fullsized":"2023-03-03T08:43:43.991092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:52:10.622422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990432+0000","last_change":"2023-03-03T08:43:34.107783+0000","last_active":"2023-03-03T08:43:43.990432+0000","last_peered":"2023-03-03T08:43:43.990432+0000","last_clean":"2023-03-03T08:43:43.990432+0000","last_became_active":"2023-03-03T08:43:34.107503+0000","last_became_peered":"2023-03-03T08:43:34.107503+0000","last_unstale":"2023-03-03T08:43:43.990432+0000","last_undegraded":"2023-03-03T08:43:43.990432+0000","last_fullsized":"2023-03-03T08:43:43.990432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:35:42.800814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981038+0000","last_change":"2023-03-03T08:43:29.072304+0000","last_active":"2023-03-03T08:43:43.981038+0000","last_peered":"2023-03-03T08:43:43.981038+0000","last_clean":"2023-03-03T08:43:43.981038+0000","last_became_active":"2023-03-03T08:43:29.071962+0000","last_became_peered":"2023-03-03T08:43:29.071962+0000","last_unstale":"2023-03-03T08:43:43.981038+0000","last_undegraded":"2023-03-03T08:43:43.981038+0000","last_fullsized":"2023-03-03T08:43:43.981038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:40:12.229226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106349+0000","last_change":"2023-03-03T08:43:43.575964+0000","last_active":"2023-03-03T08:43:45.106349+0000","last_peered":"2023-03-03T08:43:45.106349+0000","last_clean":"2023-03-03T08:43:45.106349+0000","last_became_active":"2023-03-03T08:43:43.575697+0000","last_became_peered":"2023-03-03T08:43:43.575697+0000","last_unstale":"2023-03-03T08:43:45.106349+0000","last_undegraded":"2023-03-03T08:43:45.106349+0000","last_fullsized":"2023-03-03T08:43:45.106349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:01.896050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978782+0000","last_change":"2023-03-03T08:43:44.006414+0000","last_active":"2023-03-03T08:43:44.978782+0000","last_peered":"2023-03-03T08:43:44.978782+0000","last_clean":"2023-03-03T08:43:44.978782+0000","last_became_active":"2023-03-03T08:43:44.006132+0000","last_became_peered":"2023-03-03T08:43:44.006132+0000","last_unstale":"2023-03-03T08:43:44.978782+0000","last_undegraded":"2023-03-03T08:43:44.978782+0000","last_fullsized":"2023-03-03T08:43:44.978782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:55.006419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982772+0000","last_change":"2023-03-03T08:43:34.544053+0000","last_active":"2023-03-03T08:43:43.982772+0000","last_peered":"2023-03-03T08:43:43.982772+0000","last_clean":"2023-03-03T08:43:43.982772+0000","last_became_active":"2023-03-03T08:43:34.543523+0000","last_became_peered":"2023-03-03T08:43:34.543523+0000","last_unstale":"2023-03-03T08:43:43.982772+0000","last_undegraded":"2023-03-03T08:43:43.982772+0000","last_fullsized":"2023-03-03T08:43:43.982772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:34:03.662535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989069+0000","last_change":"2023-03-03T08:43:34.104492+0000","last_active":"2023-03-03T08:43:43.989069+0000","last_peered":"2023-03-03T08:43:43.989069+0000","last_clean":"2023-03-03T08:43:43.989069+0000","last_became_active":"2023-03-03T08:43:34.104151+0000","last_became_peered":"2023-03-03T08:43:34.104151+0000","last_unstale":"2023-03-03T08:43:43.989069+0000","last_undegraded":"2023-03-03T08:43:43.989069+0000","last_fullsized":"2023-03-03T08:43:43.989069+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:57.214738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979395+0000","last_change":"2023-03-03T08:43:29.075836+0000","last_active":"2023-03-03T08:43:43.979395+0000","last_peered":"2023-03-03T08:43:43.979395+0000","last_clean":"2023-03-03T08:43:43.979395+0000","last_became_active":"2023-03-03T08:43:29.075564+0000","last_became_peered":"2023-03-03T08:43:29.075564+0000","last_unstale":"2023-03-03T08:43:43.979395+0000","last_undegraded":"2023-03-03T08:43:43.979395+0000","last_fullsized":"2023-03-03T08:43:43.979395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:39.576714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107224+0000","last_change":"2023-03-03T08:43:43.575599+0000","last_active":"2023-03-03T08:43:45.107224+0000","last_peered":"2023-03-03T08:43:45.107224+0000","last_clean":"2023-03-03T08:43:45.107224+0000","last_became_active":"2023-03-03T08:43:43.575318+0000","last_became_peered":"2023-03-03T08:43:43.575318+0000","last_unstale":"2023-03-03T08:43:45.107224+0000","last_undegraded":"2023-03-03T08:43:45.107224+0000","last_fullsized":"2023-03-03T08:43:45.107224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:28.806052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105269+0000","last_change":"2023-03-03T08:43:44.000500+0000","last_active":"2023-03-03T08:43:45.105269+0000","last_peered":"2023-03-03T08:43:45.105269+0000","last_clean":"2023-03-03T08:43:45.105269+0000","last_became_active":"2023-03-03T08:43:43.999560+0000","last_became_peered":"2023-03-03T08:43:43.999560+0000","last_unstale":"2023-03-03T08:43:45.105269+0000","last_undegraded":"2023-03-03T08:43:45.105269+0000","last_fullsized":"2023-03-03T08:43:45.105269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:29:54.240309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981172+0000","last_change":"2023-03-03T08:43:34.550263+0000","last_active":"2023-03-03T08:43:43.981172+0000","last_peered":"2023-03-03T08:43:43.981172+0000","last_clean":"2023-03-03T08:43:43.981172+0000","last_became_active":"2023-03-03T08:43:34.549560+0000","last_became_peered":"2023-03-03T08:43:34.549560+0000","last_unstale":"2023-03-03T08:43:43.981172+0000","last_undegraded":"2023-03-03T08:43:43.981172+0000","last_fullsized":"2023-03-03T08:43:43.981172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:55.874223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987062+0000","last_change":"2023-03-03T08:43:34.110484+0000","last_active":"2023-03-03T08:43:43.987062+0000","last_peered":"2023-03-03T08:43:43.987062+0000","last_clean":"2023-03-03T08:43:43.987062+0000","last_became_active":"2023-03-03T08:43:34.110220+0000","last_became_peered":"2023-03-03T08:43:34.110220+0000","last_unstale":"2023-03-03T08:43:43.987062+0000","last_undegraded":"2023-03-03T08:43:43.987062+0000","last_fullsized":"2023-03-03T08:43:43.987062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:04:59.232537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982439+0000","last_change":"2023-03-03T08:43:23.015460+0000","last_active":"2023-03-03T08:43:43.982439+0000","last_peered":"2023-03-03T08:43:43.982439+0000","last_clean":"2023-03-03T08:43:43.982439+0000","last_became_active":"2023-03-03T08:43:23.014485+0000","last_became_peered":"2023-03-03T08:43:23.014485+0000","last_unstale":"2023-03-03T08:43:43.982439+0000","last_undegraded":"2023-03-03T08:43:43.982439+0000","last_fullsized":"2023-03-03T08:43:43.982439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_clean_scrub_stamp":"2023-03-03T08:43:21.542883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:07.022017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980803+0000","last_change":"2023-03-03T08:43:29.160285+0000","last_active":"2023-03-03T08:43:43.980803+0000","last_peered":"2023-03-03T08:43:43.980803+0000","last_clean":"2023-03-03T08:43:43.980803+0000","last_became_active":"2023-03-03T08:43:29.159943+0000","last_became_peered":"2023-03-03T08:43:29.159943+0000","last_unstale":"2023-03-03T08:43:43.980803+0000","last_undegraded":"2023-03-03T08:43:43.980803+0000","last_fullsized":"2023-03-03T08:43:43.980803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:48:58.997356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107580+0000","last_change":"2023-03-03T08:43:43.575212+0000","last_active":"2023-03-03T08:43:45.107580+0000","last_peered":"2023-03-03T08:43:45.107580+0000","last_clean":"2023-03-03T08:43:45.107580+0000","last_became_active":"2023-03-03T08:43:43.574897+0000","last_became_peered":"2023-03-03T08:43:43.574897+0000","last_unstale":"2023-03-03T08:43:45.107580+0000","last_undegraded":"2023-03-03T08:43:45.107580+0000","last_fullsized":"2023-03-03T08:43:45.107580+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:33:07.674313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107370+0000","last_change":"2023-03-03T08:43:43.992952+0000","last_active":"2023-03-03T08:43:45.107370+0000","last_peered":"2023-03-03T08:43:45.107370+0000","last_clean":"2023-03-03T08:43:45.107370+0000","last_became_active":"2023-03-03T08:43:43.992535+0000","last_became_peered":"2023-03-03T08:43:43.992535+0000","last_unstale":"2023-03-03T08:43:45.107370+0000","last_undegraded":"2023-03-03T08:43:45.107370+0000","last_fullsized":"2023-03-03T08:43:45.107370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:10:41.617430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991335+0000","last_change":"2023-03-03T08:43:34.536419+0000","last_active":"2023-03-03T08:43:43.991335+0000","last_peered":"2023-03-03T08:43:43.991335+0000","last_clean":"2023-03-03T08:43:43.991335+0000","last_became_active":"2023-03-03T08:43:34.536150+0000","last_became_peered":"2023-03-03T08:43:34.536150+0000","last_unstale":"2023-03-03T08:43:43.991335+0000","last_undegraded":"2023-03-03T08:43:43.991335+0000","last_fullsized":"2023-03-03T08:43:43.991335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:22:54.362024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981827+0000","last_change":"2023-03-03T08:43:34.106337+0000","last_active":"2023-03-03T08:43:43.981827+0000","last_peered":"2023-03-03T08:43:43.981827+0000","last_clean":"2023-03-03T08:43:43.981827+0000","last_became_active":"2023-03-03T08:43:34.105753+0000","last_became_peered":"2023-03-03T08:43:34.105753+0000","last_unstale":"2023-03-03T08:43:43.981827+0000","last_undegraded":"2023-03-03T08:43:43.981827+0000","last_fullsized":"2023-03-03T08:43:43.981827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:46.761924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991590+0000","last_change":"2023-03-03T08:43:29.076367+0000","last_active":"2023-03-03T08:43:43.991590+0000","last_peered":"2023-03-03T08:43:43.991590+0000","last_clean":"2023-03-03T08:43:43.991590+0000","last_became_active":"2023-03-03T08:43:29.075982+0000","last_became_peered":"2023-03-03T08:43:29.075982+0000","last_unstale":"2023-03-03T08:43:43.991590+0000","last_undegraded":"2023-03-03T08:43:43.991590+0000","last_fullsized":"2023-03-03T08:43:43.991590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:06.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:46.143717+0000","last_change":"2023-03-03T08:43:43.577670+0000","last_active":"2023-03-03T08:43:46.143717+0000","last_peered":"2023-03-03T08:43:46.143717+0000","last_clean":"2023-03-03T08:43:46.143717+0000","last_became_active":"2023-03-03T08:43:43.577371+0000","last_became_peered":"2023-03-03T08:43:43.577371+0000","last_unstale":"2023-03-03T08:43:46.143717+0000","last_undegraded":"2023-03-03T08:43:46.143717+0000","last_fullsized":"2023-03-03T08:43:46.143717+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:18.377654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108041+0000","last_change":"2023-03-03T08:43:43.993077+0000","last_active":"2023-03-03T08:43:45.108041+0000","last_peered":"2023-03-03T08:43:45.108041+0000","last_clean":"2023-03-03T08:43:45.108041+0000","last_became_active":"2023-03-03T08:43:43.992021+0000","last_became_peered":"2023-03-03T08:43:43.992021+0000","last_unstale":"2023-03-03T08:43:45.108041+0000","last_undegraded":"2023-03-03T08:43:45.108041+0000","last_fullsized":"2023-03-03T08:43:45.108041+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:58.237553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990193+0000","last_change":"2023-03-03T08:43:34.528593+0000","last_active":"2023-03-03T08:43:43.990193+0000","last_peered":"2023-03-03T08:43:43.990193+0000","last_clean":"2023-03-03T08:43:43.990193+0000","last_became_active":"2023-03-03T08:43:34.528260+0000","last_became_peered":"2023-03-03T08:43:34.528260+0000","last_unstale":"2023-03-03T08:43:43.990193+0000","last_undegraded":"2023-03-03T08:43:43.990193+0000","last_fullsized":"2023-03-03T08:43:43.990193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988924+0000","last_change":"2023-03-03T08:43:34.113748+0000","last_active":"2023-03-03T08:43:43.988924+0000","last_peered":"2023-03-03T08:43:43.988924+0000","last_clean":"2023-03-03T08:43:43.988924+0000","last_became_active":"2023-03-03T08:43:34.113190+0000","last_became_peered":"2023-03-03T08:43:34.113190+0000","last_unstale":"2023-03-03T08:43:43.988924+0000","last_undegraded":"2023-03-03T08:43:43.988924+0000","last_fullsized":"2023-03-03T08:43:43.988924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:58:55.343571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980485+0000","last_change":"2023-03-03T08:43:29.107584+0000","last_active":"2023-03-03T08:43:43.980485+0000","last_peered":"2023-03-03T08:43:43.980485+0000","last_clean":"2023-03-03T08:43:43.980485+0000","last_became_active":"2023-03-03T08:43:29.107208+0000","last_became_peered":"2023-03-03T08:43:29.107208+0000","last_unstale":"2023-03-03T08:43:43.980485+0000","last_undegraded":"2023-03-03T08:43:43.980485+0000","last_fullsized":"2023-03-03T08:43:43.980485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:28:24.087122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106004+0000","last_change":"2023-03-03T08:43:43.575742+0000","last_active":"2023-03-03T08:43:45.106004+0000","last_peered":"2023-03-03T08:43:45.106004+0000","last_clean":"2023-03-03T08:43:45.106004+0000","last_became_active":"2023-03-03T08:43:43.575456+0000","last_became_peered":"2023-03-03T08:43:43.575456+0000","last_unstale":"2023-03-03T08:43:45.106004+0000","last_undegraded":"2023-03-03T08:43:45.106004+0000","last_fullsized":"2023-03-03T08:43:45.106004+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:46:17.635306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106635+0000","last_change":"2023-03-03T08:43:43.985961+0000","last_active":"2023-03-03T08:43:45.106635+0000","last_peered":"2023-03-03T08:43:45.106635+0000","last_clean":"2023-03-03T08:43:45.106635+0000","last_became_active":"2023-03-03T08:43:43.985633+0000","last_became_peered":"2023-03-03T08:43:43.985633+0000","last_unstale":"2023-03-03T08:43:45.106635+0000","last_undegraded":"2023-03-03T08:43:45.106635+0000","last_fullsized":"2023-03-03T08:43:45.106635+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:52.517910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989789+0000","last_change":"2023-03-03T08:43:34.526797+0000","last_active":"2023-03-03T08:43:43.989789+0000","last_peered":"2023-03-03T08:43:43.989789+0000","last_clean":"2023-03-03T08:43:43.989789+0000","last_became_active":"2023-03-03T08:43:34.526490+0000","last_became_peered":"2023-03-03T08:43:34.526490+0000","last_unstale":"2023-03-03T08:43:43.989789+0000","last_undegraded":"2023-03-03T08:43:43.989789+0000","last_fullsized":"2023-03-03T08:43:43.989789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:11.808336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986004+0000","last_change":"2023-03-03T08:43:34.104692+0000","last_active":"2023-03-03T08:43:43.986004+0000","last_peered":"2023-03-03T08:43:43.986004+0000","last_clean":"2023-03-03T08:43:43.986004+0000","last_became_active":"2023-03-03T08:43:34.104263+0000","last_became_peered":"2023-03-03T08:43:34.104263+0000","last_unstale":"2023-03-03T08:43:43.986004+0000","last_undegraded":"2023-03-03T08:43:43.986004+0000","last_fullsized":"2023-03-03T08:43:43.986004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:24.537703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983049+0000","last_change":"2023-03-03T08:43:29.107536+0000","last_active":"2023-03-03T08:43:43.983049+0000","last_peered":"2023-03-03T08:43:43.983049+0000","last_clean":"2023-03-03T08:43:43.983049+0000","last_became_active":"2023-03-03T08:43:29.107093+0000","last_became_peered":"2023-03-03T08:43:29.107093+0000","last_unstale":"2023-03-03T08:43:43.983049+0000","last_undegraded":"2023-03-03T08:43:43.983049+0000","last_fullsized":"2023-03-03T08:43:43.983049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:25:47.568904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980025+0000","last_change":"2023-03-03T08:43:43.577221+0000","last_active":"2023-03-03T08:43:44.980025+0000","last_peered":"2023-03-03T08:43:44.980025+0000","last_clean":"2023-03-03T08:43:44.980025+0000","last_became_active":"2023-03-03T08:43:43.576942+0000","last_became_peered":"2023-03-03T08:43:43.576942+0000","last_unstale":"2023-03-03T08:43:44.980025+0000","last_undegraded":"2023-03-03T08:43:44.980025+0000","last_fullsized":"2023-03-03T08:43:44.980025+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:13:09.716078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105102+0000","last_change":"2023-03-03T08:43:44.000318+0000","last_active":"2023-03-03T08:43:45.105102+0000","last_peered":"2023-03-03T08:43:45.105102+0000","last_clean":"2023-03-03T08:43:45.105102+0000","last_became_active":"2023-03-03T08:43:44.000053+0000","last_became_peered":"2023-03-03T08:43:44.000053+0000","last_unstale":"2023-03-03T08:43:45.105102+0000","last_undegraded":"2023-03-03T08:43:45.105102+0000","last_fullsized":"2023-03-03T08:43:45.105102+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:31.059774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994113+0000","last_change":"2023-03-03T08:43:34.540006+0000","last_active":"2023-03-03T08:43:43.994113+0000","last_peered":"2023-03-03T08:43:43.994113+0000","last_clean":"2023-03-03T08:43:43.994113+0000","last_became_active":"2023-03-03T08:43:34.539579+0000","last_became_peered":"2023-03-03T08:43:34.539579+0000","last_unstale":"2023-03-03T08:43:43.994113+0000","last_undegraded":"2023-03-03T08:43:43.994113+0000","last_fullsized":"2023-03-03T08:43:43.994113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988985+0000","last_change":"2023-03-03T08:43:34.109764+0000","last_active":"2023-03-03T08:43:43.988985+0000","last_peered":"2023-03-03T08:43:43.988985+0000","last_clean":"2023-03-03T08:43:43.988985+0000","last_became_active":"2023-03-03T08:43:34.109462+0000","last_became_peered":"2023-03-03T08:43:34.109462+0000","last_unstale":"2023-03-03T08:43:43.988985+0000","last_undegraded":"2023-03-03T08:43:43.988985+0000","last_fullsized":"2023-03-03T08:43:43.988985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:56.399192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106570+0000","last_change":"2023-03-03T08:43:43.576402+0000","last_active":"2023-03-03T08:43:45.106570+0000","last_peered":"2023-03-03T08:43:45.106570+0000","last_clean":"2023-03-03T08:43:45.106570+0000","last_became_active":"2023-03-03T08:43:43.576113+0000","last_became_peered":"2023-03-03T08:43:43.576113+0000","last_unstale":"2023-03-03T08:43:45.106570+0000","last_undegraded":"2023-03-03T08:43:45.106570+0000","last_fullsized":"2023-03-03T08:43:45.106570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:33:26.483688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106437+0000","last_change":"2023-03-03T08:43:44.002074+0000","last_active":"2023-03-03T08:43:45.106437+0000","last_peered":"2023-03-03T08:43:45.106437+0000","last_clean":"2023-03-03T08:43:45.106437+0000","last_became_active":"2023-03-03T08:43:44.001803+0000","last_became_peered":"2023-03-03T08:43:44.001803+0000","last_unstale":"2023-03-03T08:43:45.106437+0000","last_undegraded":"2023-03-03T08:43:45.106437+0000","last_fullsized":"2023-03-03T08:43:45.106437+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:15:16.115135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986358+0000","last_change":"2023-03-03T08:43:34.541858+0000","last_active":"2023-03-03T08:43:43.986358+0000","last_peered":"2023-03-03T08:43:43.986358+0000","last_clean":"2023-03-03T08:43:43.986358+0000","last_became_active":"2023-03-03T08:43:34.541430+0000","last_became_peered":"2023-03-03T08:43:34.541430+0000","last_unstale":"2023-03-03T08:43:43.986358+0000","last_undegraded":"2023-03-03T08:43:43.986358+0000","last_fullsized":"2023-03-03T08:43:43.986358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:29.397784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979830+0000","last_change":"2023-03-03T08:43:34.111335+0000","last_active":"2023-03-03T08:43:43.979830+0000","last_peered":"2023-03-03T08:43:43.979830+0000","last_clean":"2023-03-03T08:43:43.979830+0000","last_became_active":"2023-03-03T08:43:34.110837+0000","last_became_peered":"2023-03-03T08:43:34.110837+0000","last_unstale":"2023-03-03T08:43:43.979830+0000","last_undegraded":"2023-03-03T08:43:43.979830+0000","last_fullsized":"2023-03-03T08:43:43.979830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:03.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106574+0000","last_change":"2023-03-03T08:43:43.577539+0000","last_active":"2023-03-03T08:43:45.106574+0000","last_peered":"2023-03-03T08:43:45.106574+0000","last_clean":"2023-03-03T08:43:45.106574+0000","last_became_active":"2023-03-03T08:43:43.577209+0000","last_became_peered":"2023-03-03T08:43:43.577209+0000","last_unstale":"2023-03-03T08:43:45.106574+0000","last_undegraded":"2023-03-03T08:43:45.106574+0000","last_fullsized":"2023-03-03T08:43:45.106574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:31:01.430495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979296+0000","last_change":"2023-03-03T08:43:43.993565+0000","last_active":"2023-03-03T08:43:44.979296+0000","last_peered":"2023-03-03T08:43:44.979296+0000","last_clean":"2023-03-03T08:43:44.979296+0000","last_became_active":"2023-03-03T08:43:43.992653+0000","last_became_peered":"2023-03-03T08:43:43.992653+0000","last_unstale":"2023-03-03T08:43:44.979296+0000","last_undegraded":"2023-03-03T08:43:44.979296+0000","last_fullsized":"2023-03-03T08:43:44.979296+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:24.295462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981121+0000","last_change":"2023-03-03T08:43:34.550325+0000","last_active":"2023-03-03T08:43:43.981121+0000","last_peered":"2023-03-03T08:43:43.981121+0000","last_clean":"2023-03-03T08:43:43.981121+0000","last_became_active":"2023-03-03T08:43:34.550041+0000","last_became_peered":"2023-03-03T08:43:34.550041+0000","last_unstale":"2023-03-03T08:43:43.981121+0000","last_undegraded":"2023-03-03T08:43:43.981121+0000","last_fullsized":"2023-03-03T08:43:43.981121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:44:34.082003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980457+0000","last_change":"2023-03-03T08:43:34.106777+0000","last_active":"2023-03-03T08:43:43.980457+0000","last_peered":"2023-03-03T08:43:43.980457+0000","last_clean":"2023-03-03T08:43:43.980457+0000","last_became_active":"2023-03-03T08:43:34.106132+0000","last_became_peered":"2023-03-03T08:43:34.106132+0000","last_unstale":"2023-03-03T08:43:43.980457+0000","last_undegraded":"2023-03-03T08:43:43.980457+0000","last_fullsized":"2023-03-03T08:43:43.980457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:55.965987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106224+0000","last_change":"2023-03-03T08:43:43.580312+0000","last_active":"2023-03-03T08:43:45.106224+0000","last_peered":"2023-03-03T08:43:45.106224+0000","last_clean":"2023-03-03T08:43:45.106224+0000","last_became_active":"2023-03-03T08:43:43.580096+0000","last_became_peered":"2023-03-03T08:43:43.580096+0000","last_unstale":"2023-03-03T08:43:45.106224+0000","last_undegraded":"2023-03-03T08:43:45.106224+0000","last_fullsized":"2023-03-03T08:43:45.106224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:06:03.500778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106765+0000","last_change":"2023-03-03T08:43:43.997453+0000","last_active":"2023-03-03T08:43:45.106765+0000","last_peered":"2023-03-03T08:43:45.106765+0000","last_clean":"2023-03-03T08:43:45.106765+0000","last_became_active":"2023-03-03T08:43:43.997090+0000","last_became_peered":"2023-03-03T08:43:43.997090+0000","last_unstale":"2023-03-03T08:43:45.106765+0000","last_undegraded":"2023-03-03T08:43:45.106765+0000","last_fullsized":"2023-03-03T08:43:45.106765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:28:58.322823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985493+0000","last_change":"2023-03-03T08:43:34.542308+0000","last_active":"2023-03-03T08:43:43.985493+0000","last_peered":"2023-03-03T08:43:43.985493+0000","last_clean":"2023-03-03T08:43:43.985493+0000","last_became_active":"2023-03-03T08:43:34.541991+0000","last_became_peered":"2023-03-03T08:43:34.541991+0000","last_unstale":"2023-03-03T08:43:43.985493+0000","last_undegraded":"2023-03-03T08:43:43.985493+0000","last_fullsized":"2023-03-03T08:43:43.985493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:06:21.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992335+0000","last_change":"2023-03-03T08:43:34.109698+0000","last_active":"2023-03-03T08:43:43.992335+0000","last_peered":"2023-03-03T08:43:43.992335+0000","last_clean":"2023-03-03T08:43:43.992335+0000","last_became_active":"2023-03-03T08:43:34.109423+0000","last_became_peered":"2023-03-03T08:43:34.109423+0000","last_unstale":"2023-03-03T08:43:43.992335+0000","last_undegraded":"2023-03-03T08:43:43.992335+0000","last_fullsized":"2023-03-03T08:43:43.992335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:49.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105911+0000","last_change":"2023-03-03T08:43:43.573925+0000","last_active":"2023-03-03T08:43:45.105911+0000","last_peered":"2023-03-03T08:43:45.105911+0000","last_clean":"2023-03-03T08:43:45.105911+0000","last_became_active":"2023-03-03T08:43:43.573474+0000","last_became_peered":"2023-03-03T08:43:43.573474+0000","last_unstale":"2023-03-03T08:43:45.105911+0000","last_undegraded":"2023-03-03T08:43:45.105911+0000","last_fullsized":"2023-03-03T08:43:45.105911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:47.858726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107653+0000","last_change":"2023-03-03T08:43:43.995602+0000","last_active":"2023-03-03T08:43:45.107653+0000","last_peered":"2023-03-03T08:43:45.107653+0000","last_clean":"2023-03-03T08:43:45.107653+0000","last_became_active":"2023-03-03T08:43:43.995404+0000","last_became_peered":"2023-03-03T08:43:43.995404+0000","last_unstale":"2023-03-03T08:43:45.107653+0000","last_undegraded":"2023-03-03T08:43:45.107653+0000","last_fullsized":"2023-03-03T08:43:45.107653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:37.212119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982423+0000","last_change":"2023-03-03T08:43:34.548304+0000","last_active":"2023-03-03T08:43:43.982423+0000","last_peered":"2023-03-03T08:43:43.982423+0000","last_clean":"2023-03-03T08:43:43.982423+0000","last_became_active":"2023-03-03T08:43:34.547960+0000","last_became_peered":"2023-03-03T08:43:34.547960+0000","last_unstale":"2023-03-03T08:43:43.982423+0000","last_undegraded":"2023-03-03T08:43:43.982423+0000","last_fullsized":"2023-03-03T08:43:43.982423+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:04.907753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985574+0000","last_change":"2023-03-03T08:43:34.109372+0000","last_active":"2023-03-03T08:43:43.985574+0000","last_peered":"2023-03-03T08:43:43.985574+0000","last_clean":"2023-03-03T08:43:43.985574+0000","last_became_active":"2023-03-03T08:43:34.109023+0000","last_became_peered":"2023-03-03T08:43:34.109023+0000","last_unstale":"2023-03-03T08:43:43.985574+0000","last_undegraded":"2023-03-03T08:43:43.985574+0000","last_fullsized":"2023-03-03T08:43:43.985574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:51.374188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978866+0000","last_change":"2023-03-03T08:43:43.572240+0000","last_active":"2023-03-03T08:43:44.978866+0000","last_peered":"2023-03-03T08:43:44.978866+0000","last_clean":"2023-03-03T08:43:44.978866+0000","last_became_active":"2023-03-03T08:43:43.571753+0000","last_became_peered":"2023-03-03T08:43:43.571753+0000","last_unstale":"2023-03-03T08:43:44.978866+0000","last_undegraded":"2023-03-03T08:43:44.978866+0000","last_fullsized":"2023-03-03T08:43:44.978866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:02:30.521667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979027+0000","last_change":"2023-03-03T08:43:43.995018+0000","last_active":"2023-03-03T08:43:44.979027+0000","last_peered":"2023-03-03T08:43:44.979027+0000","last_clean":"2023-03-03T08:43:44.979027+0000","last_became_active":"2023-03-03T08:43:43.994690+0000","last_became_peered":"2023-03-03T08:43:43.994690+0000","last_unstale":"2023-03-03T08:43:44.979027+0000","last_undegraded":"2023-03-03T08:43:44.979027+0000","last_fullsized":"2023-03-03T08:43:44.979027+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:06:13.979929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986022+0000","last_change":"2023-03-03T08:43:34.538817+0000","last_active":"2023-03-03T08:43:43.986022+0000","last_peered":"2023-03-03T08:43:43.986022+0000","last_clean":"2023-03-03T08:43:43.986022+0000","last_became_active":"2023-03-03T08:43:34.538506+0000","last_became_peered":"2023-03-03T08:43:34.538506+0000","last_unstale":"2023-03-03T08:43:43.986022+0000","last_undegraded":"2023-03-03T08:43:43.986022+0000","last_fullsized":"2023-03-03T08:43:43.986022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980930+0000","last_change":"2023-03-03T08:43:34.117734+0000","last_active":"2023-03-03T08:43:43.980930+0000","last_peered":"2023-03-03T08:43:43.980930+0000","last_clean":"2023-03-03T08:43:43.980930+0000","last_became_active":"2023-03-03T08:43:34.117454+0000","last_became_peered":"2023-03-03T08:43:34.117454+0000","last_unstale":"2023-03-03T08:43:43.980930+0000","last_undegraded":"2023-03-03T08:43:43.980930+0000","last_fullsized":"2023-03-03T08:43:43.980930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:42:27.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106434+0000","last_change":"2023-03-03T08:43:43.574955+0000","last_active":"2023-03-03T08:43:45.106434+0000","last_peered":"2023-03-03T08:43:45.106434+0000","last_clean":"2023-03-03T08:43:45.106434+0000","last_became_active":"2023-03-03T08:43:43.574677+0000","last_became_peered":"2023-03-03T08:43:43.574677+0000","last_unstale":"2023-03-03T08:43:45.106434+0000","last_undegraded":"2023-03-03T08:43:45.106434+0000","last_fullsized":"2023-03-03T08:43:45.106434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:12:19.538291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.104954+0000","last_change":"2023-03-03T08:43:43.994888+0000","last_active":"2023-03-03T08:43:45.104954+0000","last_peered":"2023-03-03T08:43:45.104954+0000","last_clean":"2023-03-03T08:43:45.104954+0000","last_became_active":"2023-03-03T08:43:43.994630+0000","last_became_peered":"2023-03-03T08:43:43.994630+0000","last_unstale":"2023-03-03T08:43:45.104954+0000","last_undegraded":"2023-03-03T08:43:45.104954+0000","last_fullsized":"2023-03-03T08:43:45.104954+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:15:18.412569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982579+0000","last_change":"2023-03-03T08:43:34.548010+0000","last_active":"2023-03-03T08:43:43.982579+0000","last_peered":"2023-03-03T08:43:43.982579+0000","last_clean":"2023-03-03T08:43:43.982579+0000","last_became_active":"2023-03-03T08:43:34.547662+0000","last_became_peered":"2023-03-03T08:43:34.547662+0000","last_unstale":"2023-03-03T08:43:43.982579+0000","last_undegraded":"2023-03-03T08:43:43.982579+0000","last_fullsized":"2023-03-03T08:43:43.982579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:52.200064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986286+0000","last_change":"2023-03-03T08:43:34.109890+0000","last_active":"2023-03-03T08:43:43.986286+0000","last_peered":"2023-03-03T08:43:43.986286+0000","last_clean":"2023-03-03T08:43:43.986286+0000","last_became_active":"2023-03-03T08:43:34.109583+0000","last_became_peered":"2023-03-03T08:43:34.109583+0000","last_unstale":"2023-03-03T08:43:43.986286+0000","last_undegraded":"2023-03-03T08:43:43.986286+0000","last_fullsized":"2023-03-03T08:43:43.986286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:27:34.142294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978233+0000","last_change":"2023-03-03T08:43:43.575945+0000","last_active":"2023-03-03T08:43:44.978233+0000","last_peered":"2023-03-03T08:43:44.978233+0000","last_clean":"2023-03-03T08:43:44.978233+0000","last_became_active":"2023-03-03T08:43:43.575675+0000","last_became_peered":"2023-03-03T08:43:43.575675+0000","last_unstale":"2023-03-03T08:43:44.978233+0000","last_undegraded":"2023-03-03T08:43:44.978233+0000","last_fullsized":"2023-03-03T08:43:44.978233+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:48.730805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980103+0000","last_change":"2023-03-03T08:43:43.998435+0000","last_active":"2023-03-03T08:43:44.980103+0000","last_peered":"2023-03-03T08:43:44.980103+0000","last_clean":"2023-03-03T08:43:44.980103+0000","last_became_active":"2023-03-03T08:43:43.996508+0000","last_became_peered":"2023-03-03T08:43:43.996508+0000","last_unstale":"2023-03-03T08:43:44.980103+0000","last_undegraded":"2023-03-03T08:43:44.980103+0000","last_fullsized":"2023-03-03T08:43:44.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:31:36.310851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979477+0000","last_change":"2023-03-03T08:43:34.540196+0000","last_active":"2023-03-03T08:43:43.979477+0000","last_peered":"2023-03-03T08:43:43.979477+0000","last_clean":"2023-03-03T08:43:43.979477+0000","last_became_active":"2023-03-03T08:43:34.539931+0000","last_became_peered":"2023-03-03T08:43:34.539931+0000","last_unstale":"2023-03-03T08:43:43.979477+0000","last_undegraded":"2023-03-03T08:43:43.979477+0000","last_fullsized":"2023-03-03T08:43:43.979477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:20.012997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990881+0000","last_change":"2023-03-03T08:43:34.110464+0000","last_active":"2023-03-03T08:43:43.990881+0000","last_peered":"2023-03-03T08:43:43.990881+0000","last_clean":"2023-03-03T08:43:43.990881+0000","last_became_active":"2023-03-03T08:43:34.110165+0000","last_became_peered":"2023-03-03T08:43:34.110165+0000","last_unstale":"2023-03-03T08:43:43.990881+0000","last_undegraded":"2023-03-03T08:43:43.990881+0000","last_fullsized":"2023-03-03T08:43:43.990881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:54.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106139+0000","last_change":"2023-03-03T08:43:43.578298+0000","last_active":"2023-03-03T08:43:45.106139+0000","last_peered":"2023-03-03T08:43:45.106139+0000","last_clean":"2023-03-03T08:43:45.106139+0000","last_became_active":"2023-03-03T08:43:43.578024+0000","last_became_peered":"2023-03-03T08:43:43.578024+0000","last_unstale":"2023-03-03T08:43:45.106139+0000","last_undegraded":"2023-03-03T08:43:45.106139+0000","last_fullsized":"2023-03-03T08:43:45.106139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:00:44.633716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106526+0000","last_change":"2023-03-03T08:43:43.992906+0000","last_active":"2023-03-03T08:43:45.106526+0000","last_peered":"2023-03-03T08:43:45.106526+0000","last_clean":"2023-03-03T08:43:45.106526+0000","last_became_active":"2023-03-03T08:43:43.991823+0000","last_became_peered":"2023-03-03T08:43:43.991823+0000","last_unstale":"2023-03-03T08:43:45.106526+0000","last_undegraded":"2023-03-03T08:43:45.106526+0000","last_fullsized":"2023-03-03T08:43:45.106526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:34.431216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.553597+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.553322+0000","last_became_peered":"2023-03-03T08:43:34.553322+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:24:17.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983377+0000","last_change":"2023-03-03T08:43:34.108123+0000","last_active":"2023-03-03T08:43:43.983377+0000","last_peered":"2023-03-03T08:43:43.983377+0000","last_clean":"2023-03-03T08:43:43.983377+0000","last_became_active":"2023-03-03T08:43:34.107757+0000","last_became_peered":"2023-03-03T08:43:34.107757+0000","last_unstale":"2023-03-03T08:43:43.983377+0000","last_undegraded":"2023-03-03T08:43:43.983377+0000","last_fullsized":"2023-03-03T08:43:43.983377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:09.488930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106648+0000","last_change":"2023-03-03T08:43:43.573369+0000","last_active":"2023-03-03T08:43:45.106648+0000","last_peered":"2023-03-03T08:43:45.106648+0000","last_clean":"2023-03-03T08:43:45.106648+0000","last_became_active":"2023-03-03T08:43:43.573069+0000","last_became_peered":"2023-03-03T08:43:43.573069+0000","last_unstale":"2023-03-03T08:43:45.106648+0000","last_undegraded":"2023-03-03T08:43:45.106648+0000","last_fullsized":"2023-03-03T08:43:45.106648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:09.574926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107410+0000","last_change":"2023-03-03T08:43:43.995725+0000","last_active":"2023-03-03T08:43:45.107410+0000","last_peered":"2023-03-03T08:43:45.107410+0000","last_clean":"2023-03-03T08:43:45.107410+0000","last_became_active":"2023-03-03T08:43:43.993875+0000","last_became_peered":"2023-03-03T08:43:43.993875+0000","last_unstale":"2023-03-03T08:43:45.107410+0000","last_undegraded":"2023-03-03T08:43:45.107410+0000","last_fullsized":"2023-03-03T08:43:45.107410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:38.439539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982673+0000","last_change":"2023-03-03T08:43:34.546067+0000","last_active":"2023-03-03T08:43:43.982673+0000","last_peered":"2023-03-03T08:43:43.982673+0000","last_clean":"2023-03-03T08:43:43.982673+0000","last_became_active":"2023-03-03T08:43:34.543786+0000","last_became_peered":"2023-03-03T08:43:34.543786+0000","last_unstale":"2023-03-03T08:43:43.982673+0000","last_undegraded":"2023-03-03T08:43:43.982673+0000","last_fullsized":"2023-03-03T08:43:43.982673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:32.922812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.978650+0000","last_change":"2023-03-03T08:43:34.115792+0000","last_active":"2023-03-03T08:43:43.978650+0000","last_peered":"2023-03-03T08:43:43.978650+0000","last_clean":"2023-03-03T08:43:43.978650+0000","last_became_active":"2023-03-03T08:43:34.115431+0000","last_became_peered":"2023-03-03T08:43:34.115431+0000","last_unstale":"2023-03-03T08:43:43.978650+0000","last_undegraded":"2023-03-03T08:43:43.978650+0000","last_fullsized":"2023-03-03T08:43:43.978650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:15.484022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980510+0000","last_change":"2023-03-03T08:43:43.579691+0000","last_active":"2023-03-03T08:43:44.980510+0000","last_peered":"2023-03-03T08:43:44.980510+0000","last_clean":"2023-03-03T08:43:44.980510+0000","last_became_active":"2023-03-03T08:43:43.579425+0000","last_became_peered":"2023-03-03T08:43:43.579425+0000","last_unstale":"2023-03-03T08:43:44.980510+0000","last_undegraded":"2023-03-03T08:43:44.980510+0000","last_fullsized":"2023-03-03T08:43:44.980510+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:08.896591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979907+0000","last_change":"2023-03-03T08:43:44.006872+0000","last_active":"2023-03-03T08:43:44.979907+0000","last_peered":"2023-03-03T08:43:44.979907+0000","last_clean":"2023-03-03T08:43:44.979907+0000","last_became_active":"2023-03-03T08:43:44.006283+0000","last_became_peered":"2023-03-03T08:43:44.006283+0000","last_unstale":"2023-03-03T08:43:44.979907+0000","last_undegraded":"2023-03-03T08:43:44.979907+0000","last_fullsized":"2023-03-03T08:43:44.979907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:45.439399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981214+0000","last_change":"2023-03-03T08:43:34.532656+0000","last_active":"2023-03-03T08:43:43.981214+0000","last_peered":"2023-03-03T08:43:43.981214+0000","last_clean":"2023-03-03T08:43:43.981214+0000","last_became_active":"2023-03-03T08:43:34.532297+0000","last_became_peered":"2023-03-03T08:43:34.532297+0000","last_unstale":"2023-03-03T08:43:43.981214+0000","last_undegraded":"2023-03-03T08:43:43.981214+0000","last_fullsized":"2023-03-03T08:43:43.981214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:36.347740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989578+0000","last_change":"2023-03-03T08:43:34.112905+0000","last_active":"2023-03-03T08:43:43.989578+0000","last_peered":"2023-03-03T08:43:43.989578+0000","last_clean":"2023-03-03T08:43:43.989578+0000","last_became_active":"2023-03-03T08:43:34.112601+0000","last_became_peered":"2023-03-03T08:43:34.112601+0000","last_unstale":"2023-03-03T08:43:43.989578+0000","last_undegraded":"2023-03-03T08:43:43.989578+0000","last_fullsized":"2023-03-03T08:43:43.989578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:01.165783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"26'6","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980517+0000","last_change":"2023-03-03T08:43:43.577882+0000","last_active":"2023-03-03T08:43:44.980517+0000","last_peered":"2023-03-03T08:43:44.980517+0000","last_clean":"2023-03-03T08:43:44.980517+0000","last_became_active":"2023-03-03T08:43:43.575625+0000","last_became_peered":"2023-03-03T08:43:43.575625+0000","last_unstale":"2023-03-03T08:43:44.980517+0000","last_undegraded":"2023-03-03T08:43:44.980517+0000","last_fullsized":"2023-03-03T08:43:44.980517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:22:25.919413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106494+0000","last_change":"2023-03-03T08:43:43.994846+0000","last_active":"2023-03-03T08:43:45.106494+0000","last_peered":"2023-03-03T08:43:45.106494+0000","last_clean":"2023-03-03T08:43:45.106494+0000","last_became_active":"2023-03-03T08:43:43.994652+0000","last_became_peered":"2023-03-03T08:43:43.994652+0000","last_unstale":"2023-03-03T08:43:45.106494+0000","last_undegraded":"2023-03-03T08:43:45.106494+0000","last_fullsized":"2023-03-03T08:43:45.106494+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:54:29.238192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990352+0000","last_change":"2023-03-03T08:43:34.539790+0000","last_active":"2023-03-03T08:43:43.990352+0000","last_peered":"2023-03-03T08:43:43.990352+0000","last_clean":"2023-03-03T08:43:43.990352+0000","last_became_active":"2023-03-03T08:43:34.539443+0000","last_became_peered":"2023-03-03T08:43:34.539443+0000","last_unstale":"2023-03-03T08:43:43.990352+0000","last_undegraded":"2023-03-03T08:43:43.990352+0000","last_fullsized":"2023-03-03T08:43:43.990352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:25.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980897+0000","last_change":"2023-03-03T08:43:34.112941+0000","last_active":"2023-03-03T08:43:43.980897+0000","last_peered":"2023-03-03T08:43:43.980897+0000","last_clean":"2023-03-03T08:43:43.980897+0000","last_became_active":"2023-03-03T08:43:34.112646+0000","last_became_peered":"2023-03-03T08:43:34.112646+0000","last_unstale":"2023-03-03T08:43:43.980897+0000","last_undegraded":"2023-03-03T08:43:43.980897+0000","last_fullsized":"2023-03-03T08:43:43.980897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:32.767710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107854+0000","last_change":"2023-03-03T08:43:43.576421+0000","last_active":"2023-03-03T08:43:45.107854+0000","last_peered":"2023-03-03T08:43:45.107854+0000","last_clean":"2023-03-03T08:43:45.107854+0000","last_became_active":"2023-03-03T08:43:43.576086+0000","last_became_peered":"2023-03-03T08:43:43.576086+0000","last_unstale":"2023-03-03T08:43:45.107854+0000","last_undegraded":"2023-03-03T08:43:45.107854+0000","last_fullsized":"2023-03-03T08:43:45.107854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:57:17.814222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980193+0000","last_change":"2023-03-03T08:43:43.995591+0000","last_active":"2023-03-03T08:43:44.980193+0000","last_peered":"2023-03-03T08:43:44.980193+0000","last_clean":"2023-03-03T08:43:44.980193+0000","last_became_active":"2023-03-03T08:43:43.995338+0000","last_became_peered":"2023-03-03T08:43:43.995338+0000","last_unstale":"2023-03-03T08:43:44.980193+0000","last_undegraded":"2023-03-03T08:43:44.980193+0000","last_fullsized":"2023-03-03T08:43:44.980193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:17:56.617398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985713+0000","last_change":"2023-03-03T08:43:34.537857+0000","last_active":"2023-03-03T08:43:43.985713+0000","last_peered":"2023-03-03T08:43:43.985713+0000","last_clean":"2023-03-03T08:43:43.985713+0000","last_became_active":"2023-03-03T08:43:34.537590+0000","last_became_peered":"2023-03-03T08:43:34.537590+0000","last_unstale":"2023-03-03T08:43:43.985713+0000","last_undegraded":"2023-03-03T08:43:43.985713+0000","last_fullsized":"2023-03-03T08:43:43.985713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:52.612521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980688+0000","last_change":"2023-03-03T08:43:34.117609+0000","last_active":"2023-03-03T08:43:43.980688+0000","last_peered":"2023-03-03T08:43:43.980688+0000","last_clean":"2023-03-03T08:43:43.980688+0000","last_became_active":"2023-03-03T08:43:34.117271+0000","last_became_peered":"2023-03-03T08:43:34.117271+0000","last_unstale":"2023-03-03T08:43:43.980688+0000","last_undegraded":"2023-03-03T08:43:43.980688+0000","last_fullsized":"2023-03-03T08:43:43.980688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:20.259259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978772+0000","last_change":"2023-03-03T08:43:43.580489+0000","last_active":"2023-03-03T08:43:44.978772+0000","last_peered":"2023-03-03T08:43:44.978772+0000","last_clean":"2023-03-03T08:43:44.978772+0000","last_became_active":"2023-03-03T08:43:43.580207+0000","last_became_peered":"2023-03-03T08:43:43.580207+0000","last_unstale":"2023-03-03T08:43:44.978772+0000","last_undegraded":"2023-03-03T08:43:44.978772+0000","last_fullsized":"2023-03-03T08:43:44.978772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:58:05.519074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978633+0000","last_change":"2023-03-03T08:43:43.993832+0000","last_active":"2023-03-03T08:43:44.978633+0000","last_peered":"2023-03-03T08:43:44.978633+0000","last_clean":"2023-03-03T08:43:44.978633+0000","last_became_active":"2023-03-03T08:43:43.993524+0000","last_became_peered":"2023-03-03T08:43:43.993524+0000","last_unstale":"2023-03-03T08:43:44.978633+0000","last_undegraded":"2023-03-03T08:43:44.978633+0000","last_fullsized":"2023-03-03T08:43:44.978633+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:23:55.761785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987159+0000","last_change":"2023-03-03T08:43:34.544694+0000","last_active":"2023-03-03T08:43:43.987159+0000","last_peered":"2023-03-03T08:43:43.987159+0000","last_clean":"2023-03-03T08:43:43.987159+0000","last_became_active":"2023-03-03T08:43:34.544399+0000","last_became_peered":"2023-03-03T08:43:34.544399+0000","last_unstale":"2023-03-03T08:43:43.987159+0000","last_undegraded":"2023-03-03T08:43:43.987159+0000","last_fullsized":"2023-03-03T08:43:43.987159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986196+0000","last_change":"2023-03-03T08:43:34.110810+0000","last_active":"2023-03-03T08:43:43.986196+0000","last_peered":"2023-03-03T08:43:43.986196+0000","last_clean":"2023-03-03T08:43:43.986196+0000","last_became_active":"2023-03-03T08:43:34.110483+0000","last_became_peered":"2023-03-03T08:43:34.110483+0000","last_unstale":"2023-03-03T08:43:43.986196+0000","last_undegraded":"2023-03-03T08:43:43.986196+0000","last_fullsized":"2023-03-03T08:43:43.986196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:06.970615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979989+0000","last_change":"2023-03-03T08:43:43.576669+0000","last_active":"2023-03-03T08:43:44.979989+0000","last_peered":"2023-03-03T08:43:44.979989+0000","last_clean":"2023-03-03T08:43:44.979989+0000","last_became_active":"2023-03-03T08:43:43.573885+0000","last_became_peered":"2023-03-03T08:43:43.573885+0000","last_unstale":"2023-03-03T08:43:44.979989+0000","last_undegraded":"2023-03-03T08:43:44.979989+0000","last_fullsized":"2023-03-03T08:43:44.979989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:56:12.054118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979442+0000","last_change":"2023-03-03T08:43:43.996790+0000","last_active":"2023-03-03T08:43:44.979442+0000","last_peered":"2023-03-03T08:43:44.979442+0000","last_clean":"2023-03-03T08:43:44.979442+0000","last_became_active":"2023-03-03T08:43:43.996366+0000","last_became_peered":"2023-03-03T08:43:43.996366+0000","last_unstale":"2023-03-03T08:43:44.979442+0000","last_undegraded":"2023-03-03T08:43:44.979442+0000","last_fullsized":"2023-03-03T08:43:44.979442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:01.662221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988553+0000","last_change":"2023-03-03T08:43:34.538703+0000","last_active":"2023-03-03T08:43:43.988553+0000","last_peered":"2023-03-03T08:43:43.988553+0000","last_clean":"2023-03-03T08:43:43.988553+0000","last_became_active":"2023-03-03T08:43:34.538418+0000","last_became_peered":"2023-03-03T08:43:34.538418+0000","last_unstale":"2023-03-03T08:43:43.988553+0000","last_undegraded":"2023-03-03T08:43:43.988553+0000","last_fullsized":"2023-03-03T08:43:43.988553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:02.795203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986185+0000","last_change":"2023-03-03T08:43:34.113168+0000","last_active":"2023-03-03T08:43:43.986185+0000","last_peered":"2023-03-03T08:43:43.986185+0000","last_clean":"2023-03-03T08:43:43.986185+0000","last_became_active":"2023-03-03T08:43:34.112731+0000","last_became_peered":"2023-03-03T08:43:34.112731+0000","last_unstale":"2023-03-03T08:43:43.986185+0000","last_undegraded":"2023-03-03T08:43:43.986185+0000","last_fullsized":"2023-03-03T08:43:43.986185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:57:59.463191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106795+0000","last_change":"2023-03-03T08:43:43.577965+0000","last_active":"2023-03-03T08:43:45.106795+0000","last_peered":"2023-03-03T08:43:45.106795+0000","last_clean":"2023-03-03T08:43:45.106795+0000","last_became_active":"2023-03-03T08:43:43.577587+0000","last_became_peered":"2023-03-03T08:43:43.577587+0000","last_unstale":"2023-03-03T08:43:45.106795+0000","last_undegraded":"2023-03-03T08:43:45.106795+0000","last_fullsized":"2023-03-03T08:43:45.106795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:24.196773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105366+0000","last_change":"2023-03-03T08:43:43.998210+0000","last_active":"2023-03-03T08:43:45.105366+0000","last_peered":"2023-03-03T08:43:45.105366+0000","last_clean":"2023-03-03T08:43:45.105366+0000","last_became_active":"2023-03-03T08:43:43.997845+0000","last_became_peered":"2023-03-03T08:43:43.997845+0000","last_unstale":"2023-03-03T08:43:45.105366+0000","last_undegraded":"2023-03-03T08:43:45.105366+0000","last_fullsized":"2023-03-03T08:43:45.105366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:57:38.226245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991762+0000","last_change":"2023-03-03T08:43:34.528573+0000","last_active":"2023-03-03T08:43:43.991762+0000","last_peered":"2023-03-03T08:43:43.991762+0000","last_clean":"2023-03-03T08:43:43.991762+0000","last_became_active":"2023-03-03T08:43:34.528278+0000","last_became_peered":"2023-03-03T08:43:34.528278+0000","last_unstale":"2023-03-03T08:43:43.991762+0000","last_undegraded":"2023-03-03T08:43:43.991762+0000","last_fullsized":"2023-03-03T08:43:43.991762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:51.468323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.110802+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.110519+0000","last_became_peered":"2023-03-03T08:43:34.110519+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:02:18.355043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105602+0000","last_change":"2023-03-03T08:43:43.573655+0000","last_active":"2023-03-03T08:43:45.105602+0000","last_peered":"2023-03-03T08:43:45.105602+0000","last_clean":"2023-03-03T08:43:45.105602+0000","last_became_active":"2023-03-03T08:43:43.573162+0000","last_became_peered":"2023-03-03T08:43:43.573162+0000","last_unstale":"2023-03-03T08:43:45.105602+0000","last_undegraded":"2023-03-03T08:43:45.105602+0000","last_fullsized":"2023-03-03T08:43:45.105602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:59:04.568700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106617+0000","last_change":"2023-03-03T08:43:43.993431+0000","last_active":"2023-03-03T08:43:45.106617+0000","last_peered":"2023-03-03T08:43:45.106617+0000","last_clean":"2023-03-03T08:43:45.106617+0000","last_became_active":"2023-03-03T08:43:43.993166+0000","last_became_peered":"2023-03-03T08:43:43.993166+0000","last_unstale":"2023-03-03T08:43:45.106617+0000","last_undegraded":"2023-03-03T08:43:45.106617+0000","last_fullsized":"2023-03-03T08:43:45.106617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:04:52.880667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979475+0000","last_change":"2023-03-03T08:43:43.992992+0000","last_active":"2023-03-03T08:43:44.979475+0000","last_peered":"2023-03-03T08:43:44.979475+0000","last_clean":"2023-03-03T08:43:44.979475+0000","last_became_active":"2023-03-03T08:43:43.992664+0000","last_became_peered":"2023-03-03T08:43:43.992664+0000","last_unstale":"2023-03-03T08:43:44.979475+0000","last_undegraded":"2023-03-03T08:43:44.979475+0000","last_fullsized":"2023-03-03T08:43:44.979475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:07.276715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981207+0000","last_change":"2023-03-03T08:43:34.547450+0000","last_active":"2023-03-03T08:43:43.981207+0000","last_peered":"2023-03-03T08:43:43.981207+0000","last_clean":"2023-03-03T08:43:43.981207+0000","last_became_active":"2023-03-03T08:43:34.547168+0000","last_became_peered":"2023-03-03T08:43:34.547168+0000","last_unstale":"2023-03-03T08:43:43.981207+0000","last_undegraded":"2023-03-03T08:43:43.981207+0000","last_fullsized":"2023-03-03T08:43:43.981207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:17.723229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982025+0000","last_change":"2023-03-03T08:43:34.106505+0000","last_active":"2023-03-03T08:43:43.982025+0000","last_peered":"2023-03-03T08:43:43.982025+0000","last_clean":"2023-03-03T08:43:43.982025+0000","last_became_active":"2023-03-03T08:43:34.106079+0000","last_became_peered":"2023-03-03T08:43:34.106079+0000","last_unstale":"2023-03-03T08:43:43.982025+0000","last_undegraded":"2023-03-03T08:43:43.982025+0000","last_fullsized":"2023-03-03T08:43:43.982025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:48.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979698+0000","last_change":"2023-03-03T08:43:43.575448+0000","last_active":"2023-03-03T08:43:44.979698+0000","last_peered":"2023-03-03T08:43:44.979698+0000","last_clean":"2023-03-03T08:43:44.979698+0000","last_became_active":"2023-03-03T08:43:43.575142+0000","last_became_peered":"2023-03-03T08:43:43.575142+0000","last_unstale":"2023-03-03T08:43:44.979698+0000","last_undegraded":"2023-03-03T08:43:44.979698+0000","last_fullsized":"2023-03-03T08:43:44.979698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:42:09.308616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990396+0000","last_change":"2023-03-03T08:43:34.549123+0000","last_active":"2023-03-03T08:43:43.990396+0000","last_peered":"2023-03-03T08:43:43.990396+0000","last_clean":"2023-03-03T08:43:43.990396+0000","last_became_active":"2023-03-03T08:43:34.548839+0000","last_became_peered":"2023-03-03T08:43:34.548839+0000","last_unstale":"2023-03-03T08:43:43.990396+0000","last_undegraded":"2023-03-03T08:43:43.990396+0000","last_fullsized":"2023-03-03T08:43:43.990396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:38.134009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984959+0000","last_change":"2023-03-03T08:43:34.108187+0000","last_active":"2023-03-03T08:43:43.984959+0000","last_peered":"2023-03-03T08:43:43.984959+0000","last_clean":"2023-03-03T08:43:43.984959+0000","last_became_active":"2023-03-03T08:43:34.107859+0000","last_became_peered":"2023-03-03T08:43:34.107859+0000","last_unstale":"2023-03-03T08:43:43.984959+0000","last_undegraded":"2023-03-03T08:43:43.984959+0000","last_fullsized":"2023-03-03T08:43:43.984959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:46:40.595756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107953+0000","last_change":"2023-03-03T08:43:43.572738+0000","last_active":"2023-03-03T08:43:45.107953+0000","last_peered":"2023-03-03T08:43:45.107953+0000","last_clean":"2023-03-03T08:43:45.107953+0000","last_became_active":"2023-03-03T08:43:43.572435+0000","last_became_peered":"2023-03-03T08:43:43.572435+0000","last_unstale":"2023-03-03T08:43:45.107953+0000","last_undegraded":"2023-03-03T08:43:45.107953+0000","last_fullsized":"2023-03-03T08:43:45.107953+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:14.264653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105420+0000","last_change":"2023-03-03T08:43:43.995021+0000","last_active":"2023-03-03T08:43:45.105420+0000","last_peered":"2023-03-03T08:43:45.105420+0000","last_clean":"2023-03-03T08:43:45.105420+0000","last_became_active":"2023-03-03T08:43:43.994766+0000","last_became_peered":"2023-03-03T08:43:43.994766+0000","last_unstale":"2023-03-03T08:43:45.105420+0000","last_undegraded":"2023-03-03T08:43:45.105420+0000","last_fullsized":"2023-03-03T08:43:45.105420+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:14:03.827057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979582+0000","last_change":"2023-03-03T08:43:43.993391+0000","last_active":"2023-03-03T08:43:44.979582+0000","last_peered":"2023-03-03T08:43:44.979582+0000","last_clean":"2023-03-03T08:43:44.979582+0000","last_became_active":"2023-03-03T08:43:43.993087+0000","last_became_peered":"2023-03-03T08:43:43.993087+0000","last_unstale":"2023-03-03T08:43:44.979582+0000","last_undegraded":"2023-03-03T08:43:44.979582+0000","last_fullsized":"2023-03-03T08:43:44.979582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:13:01.193811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992180+0000","last_change":"2023-03-03T08:43:34.537301+0000","last_active":"2023-03-03T08:43:43.992180+0000","last_peered":"2023-03-03T08:43:43.992180+0000","last_clean":"2023-03-03T08:43:43.992180+0000","last_became_active":"2023-03-03T08:43:34.537007+0000","last_became_peered":"2023-03-03T08:43:34.537007+0000","last_unstale":"2023-03-03T08:43:43.992180+0000","last_undegraded":"2023-03-03T08:43:43.992180+0000","last_fullsized":"2023-03-03T08:43:43.992180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:59:49.124140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982700+0000","last_change":"2023-03-03T08:43:34.108056+0000","last_active":"2023-03-03T08:43:43.982700+0000","last_peered":"2023-03-03T08:43:43.982700+0000","last_clean":"2023-03-03T08:43:43.982700+0000","last_became_active":"2023-03-03T08:43:34.107708+0000","last_became_peered":"2023-03-03T08:43:34.107708+0000","last_unstale":"2023-03-03T08:43:43.982700+0000","last_undegraded":"2023-03-03T08:43:43.982700+0000","last_fullsized":"2023-03-03T08:43:43.982700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:10.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107314+0000","last_change":"2023-03-03T08:43:43.575788+0000","last_active":"2023-03-03T08:43:45.107314+0000","last_peered":"2023-03-03T08:43:45.107314+0000","last_clean":"2023-03-03T08:43:45.107314+0000","last_became_active":"2023-03-03T08:43:43.575507+0000","last_became_peered":"2023-03-03T08:43:43.575507+0000","last_unstale":"2023-03-03T08:43:45.107314+0000","last_undegraded":"2023-03-03T08:43:45.107314+0000","last_fullsized":"2023-03-03T08:43:45.107314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:15.843816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107877+0000","last_change":"2023-03-03T08:43:43.994916+0000","last_active":"2023-03-03T08:43:45.107877+0000","last_peered":"2023-03-03T08:43:45.107877+0000","last_clean":"2023-03-03T08:43:45.107877+0000","last_became_active":"2023-03-03T08:43:43.994639+0000","last_became_peered":"2023-03-03T08:43:43.994639+0000","last_unstale":"2023-03-03T08:43:45.107877+0000","last_undegraded":"2023-03-03T08:43:45.107877+0000","last_fullsized":"2023-03-03T08:43:45.107877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:52.526052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980834+0000","last_change":"2023-03-03T08:43:34.533987+0000","last_active":"2023-03-03T08:43:43.980834+0000","last_peered":"2023-03-03T08:43:43.980834+0000","last_clean":"2023-03-03T08:43:43.980834+0000","last_became_active":"2023-03-03T08:43:34.533729+0000","last_became_peered":"2023-03-03T08:43:34.533729+0000","last_unstale":"2023-03-03T08:43:43.980834+0000","last_undegraded":"2023-03-03T08:43:43.980834+0000","last_fullsized":"2023-03-03T08:43:43.980834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:18:12.407151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106398+0000","last_change":"2023-03-03T08:43:43.996808+0000","last_active":"2023-03-03T08:43:45.106398+0000","last_peered":"2023-03-03T08:43:45.106398+0000","last_clean":"2023-03-03T08:43:45.106398+0000","last_became_active":"2023-03-03T08:43:43.996559+0000","last_became_peered":"2023-03-03T08:43:43.996559+0000","last_unstale":"2023-03-03T08:43:45.106398+0000","last_undegraded":"2023-03-03T08:43:45.106398+0000","last_fullsized":"2023-03-03T08:43:45.106398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:01.223908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.993213+0000","last_change":"2023-03-03T08:43:34.540762+0000","last_active":"2023-03-03T08:43:43.993213+0000","last_peered":"2023-03-03T08:43:43.993213+0000","last_clean":"2023-03-03T08:43:43.993213+0000","last_became_active":"2023-03-03T08:43:34.540467+0000","last_became_peered":"2023-03-03T08:43:34.540467+0000","last_unstale":"2023-03-03T08:43:43.993213+0000","last_undegraded":"2023-03-03T08:43:43.993213+0000","last_fullsized":"2023-03-03T08:43:43.993213+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107993+0000","last_change":"2023-03-03T08:43:43.995286+0000","last_active":"2023-03-03T08:43:45.107993+0000","last_peered":"2023-03-03T08:43:45.107993+0000","last_clean":"2023-03-03T08:43:45.107993+0000","last_became_active":"2023-03-03T08:43:43.994861+0000","last_became_peered":"2023-03-03T08:43:43.994861+0000","last_unstale":"2023-03-03T08:43:45.107993+0000","last_undegraded":"2023-03-03T08:43:45.107993+0000","last_fullsized":"2023-03-03T08:43:45.107993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990722+0000","last_change":"2023-03-03T08:43:34.542858+0000","last_active":"2023-03-03T08:43:43.990722+0000","last_peered":"2023-03-03T08:43:43.990722+0000","last_clean":"2023-03-03T08:43:43.990722+0000","last_became_active":"2023-03-03T08:43:34.542510+0000","last_became_peered":"2023-03-03T08:43:34.542510+0000","last_unstale":"2023-03-03T08:43:43.990722+0000","last_undegraded":"2023-03-03T08:43:43.990722+0000","last_fullsized":"2023-03-03T08:43:43.990722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:21.471234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106711+0000","last_change":"2023-03-03T08:43:43.992084+0000","last_active":"2023-03-03T08:43:45.106711+0000","last_peered":"2023-03-03T08:43:45.106711+0000","last_clean":"2023-03-03T08:43:45.106711+0000","last_became_active":"2023-03-03T08:43:43.991465+0000","last_became_peered":"2023-03-03T08:43:43.991465+0000","last_unstale":"2023-03-03T08:43:45.106711+0000","last_undegraded":"2023-03-03T08:43:45.106711+0000","last_fullsized":"2023-03-03T08:43:45.106711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:19.530923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983571+0000","last_change":"2023-03-03T08:43:34.548469+0000","last_active":"2023-03-03T08:43:43.983571+0000","last_peered":"2023-03-03T08:43:43.983571+0000","last_clean":"2023-03-03T08:43:43.983571+0000","last_became_active":"2023-03-03T08:43:34.547647+0000","last_became_peered":"2023-03-03T08:43:34.547647+0000","last_unstale":"2023-03-03T08:43:43.983571+0000","last_undegraded":"2023-03-03T08:43:43.983571+0000","last_fullsized":"2023-03-03T08:43:43.983571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:40.054781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107161+0000","last_change":"2023-03-03T08:43:43.998061+0000","last_active":"2023-03-03T08:43:45.107161+0000","last_peered":"2023-03-03T08:43:45.107161+0000","last_clean":"2023-03-03T08:43:45.107161+0000","last_became_active":"2023-03-03T08:43:43.997770+0000","last_became_peered":"2023-03-03T08:43:43.997770+0000","last_unstale":"2023-03-03T08:43:45.107161+0000","last_undegraded":"2023-03-03T08:43:45.107161+0000","last_fullsized":"2023-03-03T08:43:45.107161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:17.053610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988635+0000","last_change":"2023-03-03T08:43:34.549771+0000","last_active":"2023-03-03T08:43:43.988635+0000","last_peered":"2023-03-03T08:43:43.988635+0000","last_clean":"2023-03-03T08:43:43.988635+0000","last_became_active":"2023-03-03T08:43:34.549479+0000","last_became_peered":"2023-03-03T08:43:34.549479+0000","last_unstale":"2023-03-03T08:43:43.988635+0000","last_undegraded":"2023-03-03T08:43:43.988635+0000","last_fullsized":"2023-03-03T08:43:43.988635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:41:03.902019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979024+0000","last_change":"2023-03-03T08:43:43.999391+0000","last_active":"2023-03-03T08:43:44.979024+0000","last_peered":"2023-03-03T08:43:44.979024+0000","last_clean":"2023-03-03T08:43:44.979024+0000","last_became_active":"2023-03-03T08:43:43.998904+0000","last_became_peered":"2023-03-03T08:43:43.998904+0000","last_unstale":"2023-03-03T08:43:44.979024+0000","last_undegraded":"2023-03-03T08:43:44.979024+0000","last_fullsized":"2023-03-03T08:43:44.979024+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:17.429776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992365+0000","last_change":"2023-03-03T08:43:34.542661+0000","last_active":"2023-03-03T08:43:43.992365+0000","last_peered":"2023-03-03T08:43:43.992365+0000","last_clean":"2023-03-03T08:43:43.992365+0000","last_became_active":"2023-03-03T08:43:34.542353+0000","last_became_peered":"2023-03-03T08:43:34.542353+0000","last_unstale":"2023-03-03T08:43:43.992365+0000","last_undegraded":"2023-03-03T08:43:43.992365+0000","last_fullsized":"2023-03-03T08:43:43.992365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:38.730662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105103+0000","last_change":"2023-03-03T08:43:43.996729+0000","last_active":"2023-03-03T08:43:45.105103+0000","last_peered":"2023-03-03T08:43:45.105103+0000","last_clean":"2023-03-03T08:43:45.105103+0000","last_became_active":"2023-03-03T08:43:43.986535+0000","last_became_peered":"2023-03-03T08:43:43.986535+0000","last_unstale":"2023-03-03T08:43:45.105103+0000","last_undegraded":"2023-03-03T08:43:45.105103+0000","last_fullsized":"2023-03-03T08:43:45.105103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:22.505305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982406+0000","last_change":"2023-03-03T08:43:34.549837+0000","last_active":"2023-03-03T08:43:43.982406+0000","last_peered":"2023-03-03T08:43:43.982406+0000","last_clean":"2023-03-03T08:43:43.982406+0000","last_became_active":"2023-03-03T08:43:34.549542+0000","last_became_peered":"2023-03-03T08:43:34.549542+0000","last_unstale":"2023-03-03T08:43:43.982406+0000","last_undegraded":"2023-03-03T08:43:43.982406+0000","last_fullsized":"2023-03-03T08:43:43.982406+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:31.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106334+0000","last_change":"2023-03-03T08:43:43.993213+0000","last_active":"2023-03-03T08:43:45.106334+0000","last_peered":"2023-03-03T08:43:45.106334+0000","last_clean":"2023-03-03T08:43:45.106334+0000","last_became_active":"2023-03-03T08:43:43.992920+0000","last_became_peered":"2023-03-03T08:43:43.992920+0000","last_unstale":"2023-03-03T08:43:45.106334+0000","last_undegraded":"2023-03-03T08:43:45.106334+0000","last_fullsized":"2023-03-03T08:43:45.106334+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:42.055599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.547003+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.546744+0000","last_became_peered":"2023-03-03T08:43:34.546744+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:53:38.996605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106934+0000","last_change":"2023-03-03T08:43:43.994195+0000","last_active":"2023-03-03T08:43:45.106934+0000","last_peered":"2023-03-03T08:43:45.106934+0000","last_clean":"2023-03-03T08:43:45.106934+0000","last_became_active":"2023-03-03T08:43:43.993463+0000","last_became_peered":"2023-03-03T08:43:43.993463+0000","last_unstale":"2023-03-03T08:43:45.106934+0000","last_undegraded":"2023-03-03T08:43:45.106934+0000","last_fullsized":"2023-03-03T08:43:45.106934+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:52.365793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982677+0000","last_change":"2023-03-03T08:43:34.553589+0000","last_active":"2023-03-03T08:43:43.982677+0000","last_peered":"2023-03-03T08:43:43.982677+0000","last_clean":"2023-03-03T08:43:43.982677+0000","last_became_active":"2023-03-03T08:43:34.553242+0000","last_became_peered":"2023-03-03T08:43:34.553242+0000","last_unstale":"2023-03-03T08:43:43.982677+0000","last_undegraded":"2023-03-03T08:43:43.982677+0000","last_fullsized":"2023-03-03T08:43:43.982677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:09.000051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107426+0000","last_change":"2023-03-03T08:43:44.004404+0000","last_active":"2023-03-03T08:43:45.107426+0000","last_peered":"2023-03-03T08:43:45.107426+0000","last_clean":"2023-03-03T08:43:45.107426+0000","last_became_active":"2023-03-03T08:43:44.004120+0000","last_became_peered":"2023-03-03T08:43:44.004120+0000","last_unstale":"2023-03-03T08:43:45.107426+0000","last_undegraded":"2023-03-03T08:43:45.107426+0000","last_fullsized":"2023-03-03T08:43:45.107426+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:36.006791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990661+0000","last_change":"2023-03-03T08:43:34.537110+0000","last_active":"2023-03-03T08:43:43.990661+0000","last_peered":"2023-03-03T08:43:43.990661+0000","last_clean":"2023-03-03T08:43:43.990661+0000","last_became_active":"2023-03-03T08:43:34.536848+0000","last_became_peered":"2023-03-03T08:43:34.536848+0000","last_unstale":"2023-03-03T08:43:43.990661+0000","last_undegraded":"2023-03-03T08:43:43.990661+0000","last_fullsized":"2023-03-03T08:43:43.990661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106366+0000","last_change":"2023-03-03T08:43:43.999266+0000","last_active":"2023-03-03T08:43:45.106366+0000","last_peered":"2023-03-03T08:43:45.106366+0000","last_clean":"2023-03-03T08:43:45.106366+0000","last_became_active":"2023-03-03T08:43:43.998929+0000","last_became_peered":"2023-03-03T08:43:43.998929+0000","last_unstale":"2023-03-03T08:43:45.106366+0000","last_undegraded":"2023-03-03T08:43:45.106366+0000","last_fullsized":"2023-03-03T08:43:45.106366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:02.236263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994220+0000","last_change":"2023-03-03T08:43:34.532154+0000","last_active":"2023-03-03T08:43:43.994220+0000","last_peered":"2023-03-03T08:43:43.994220+0000","last_clean":"2023-03-03T08:43:43.994220+0000","last_became_active":"2023-03-03T08:43:34.531844+0000","last_became_peered":"2023-03-03T08:43:34.531844+0000","last_unstale":"2023-03-03T08:43:43.994220+0000","last_undegraded":"2023-03-03T08:43:43.994220+0000","last_fullsized":"2023-03-03T08:43:43.994220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:00.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106991+0000","last_change":"2023-03-03T08:43:44.004469+0000","last_active":"2023-03-03T08:43:45.106991+0000","last_peered":"2023-03-03T08:43:45.106991+0000","last_clean":"2023-03-03T08:43:45.106991+0000","last_became_active":"2023-03-03T08:43:44.004187+0000","last_became_peered":"2023-03-03T08:43:44.004187+0000","last_unstale":"2023-03-03T08:43:45.106991+0000","last_undegraded":"2023-03-03T08:43:45.106991+0000","last_fullsized":"2023-03-03T08:43:45.106991+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:35.298857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979308+0000","last_change":"2023-03-03T08:43:34.527373+0000","last_active":"2023-03-03T08:43:43.979308+0000","last_peered":"2023-03-03T08:43:43.979308+0000","last_clean":"2023-03-03T08:43:43.979308+0000","last_became_active":"2023-03-03T08:43:34.527171+0000","last_became_peered":"2023-03-03T08:43:34.527171+0000","last_unstale":"2023-03-03T08:43:43.979308+0000","last_undegraded":"2023-03-03T08:43:43.979308+0000","last_fullsized":"2023-03-03T08:43:43.979308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:20:51.138587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108220+0000","last_change":"2023-03-03T08:43:43.996329+0000","last_active":"2023-03-03T08:43:45.108220+0000","last_peered":"2023-03-03T08:43:45.108220+0000","last_clean":"2023-03-03T08:43:45.108220+0000","last_became_active":"2023-03-03T08:43:43.995055+0000","last_became_peered":"2023-03-03T08:43:43.995055+0000","last_unstale":"2023-03-03T08:43:45.108220+0000","last_undegraded":"2023-03-03T08:43:45.108220+0000","last_fullsized":"2023-03-03T08:43:45.108220+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:57.593740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984695+0000","last_change":"2023-03-03T08:43:34.542729+0000","last_active":"2023-03-03T08:43:43.984695+0000","last_peered":"2023-03-03T08:43:43.984695+0000","last_clean":"2023-03-03T08:43:43.984695+0000","last_became_active":"2023-03-03T08:43:34.542246+0000","last_became_peered":"2023-03-03T08:43:34.542246+0000","last_unstale":"2023-03-03T08:43:43.984695+0000","last_undegraded":"2023-03-03T08:43:43.984695+0000","last_fullsized":"2023-03-03T08:43:43.984695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:14:59.995526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105365+0000","last_change":"2023-03-03T08:43:44.002358+0000","last_active":"2023-03-03T08:43:45.105365+0000","last_peered":"2023-03-03T08:43:45.105365+0000","last_clean":"2023-03-03T08:43:45.105365+0000","last_became_active":"2023-03-03T08:43:44.002020+0000","last_became_peered":"2023-03-03T08:43:44.002020+0000","last_unstale":"2023-03-03T08:43:45.105365+0000","last_undegraded":"2023-03-03T08:43:45.105365+0000","last_fullsized":"2023-03-03T08:43:45.105365+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:51.524774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982664+0000","last_change":"2023-03-03T08:43:34.549026+0000","last_active":"2023-03-03T08:43:43.982664+0000","last_peered":"2023-03-03T08:43:43.982664+0000","last_clean":"2023-03-03T08:43:43.982664+0000","last_became_active":"2023-03-03T08:43:34.548778+0000","last_became_peered":"2023-03-03T08:43:34.548778+0000","last_unstale":"2023-03-03T08:43:43.982664+0000","last_undegraded":"2023-03-03T08:43:43.982664+0000","last_fullsized":"2023-03-03T08:43:43.982664+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:41.835304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105281+0000","last_change":"2023-03-03T08:43:43.997684+0000","last_active":"2023-03-03T08:43:45.105281+0000","last_peered":"2023-03-03T08:43:45.105281+0000","last_clean":"2023-03-03T08:43:45.105281+0000","last_became_active":"2023-03-03T08:43:43.997503+0000","last_became_peered":"2023-03-03T08:43:43.997503+0000","last_unstale":"2023-03-03T08:43:45.105281+0000","last_undegraded":"2023-03-03T08:43:45.105281+0000","last_fullsized":"2023-03-03T08:43:45.105281+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:29.512524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991178+0000","last_change":"2023-03-03T08:43:34.544013+0000","last_active":"2023-03-03T08:43:43.991178+0000","last_peered":"2023-03-03T08:43:43.991178+0000","last_clean":"2023-03-03T08:43:43.991178+0000","last_became_active":"2023-03-03T08:43:34.543745+0000","last_became_peered":"2023-03-03T08:43:34.543745+0000","last_unstale":"2023-03-03T08:43:43.991178+0000","last_undegraded":"2023-03-03T08:43:43.991178+0000","last_fullsized":"2023-03-03T08:43:43.991178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:05.647977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107560+0000","last_change":"2023-03-03T08:43:43.999884+0000","last_active":"2023-03-03T08:43:45.107560+0000","last_peered":"2023-03-03T08:43:45.107560+0000","last_clean":"2023-03-03T08:43:45.107560+0000","last_became_active":"2023-03-03T08:43:43.999593+0000","last_became_peered":"2023-03-03T08:43:43.999593+0000","last_unstale":"2023-03-03T08:43:45.107560+0000","last_undegraded":"2023-03-03T08:43:45.107560+0000","last_fullsized":"2023-03-03T08:43:45.107560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:45:09.132974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982600+0000","last_change":"2023-03-03T08:43:34.553434+0000","last_active":"2023-03-03T08:43:43.982600+0000","last_peered":"2023-03-03T08:43:43.982600+0000","last_clean":"2023-03-03T08:43:43.982600+0000","last_became_active":"2023-03-03T08:43:34.553125+0000","last_became_peered":"2023-03-03T08:43:34.553125+0000","last_unstale":"2023-03-03T08:43:43.982600+0000","last_undegraded":"2023-03-03T08:43:43.982600+0000","last_fullsized":"2023-03-03T08:43:43.982600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:41:17.553506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105717+0000","last_change":"2023-03-03T08:43:44.006710+0000","last_active":"2023-03-03T08:43:45.105717+0000","last_peered":"2023-03-03T08:43:45.105717+0000","last_clean":"2023-03-03T08:43:45.105717+0000","last_became_active":"2023-03-03T08:43:44.006447+0000","last_became_peered":"2023-03-03T08:43:44.006447+0000","last_unstale":"2023-03-03T08:43:45.105717+0000","last_undegraded":"2023-03-03T08:43:45.105717+0000","last_fullsized":"2023-03-03T08:43:45.105717+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:21.767326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982538+0000","last_change":"2023-03-03T08:43:34.550889+0000","last_active":"2023-03-03T08:43:43.982538+0000","last_peered":"2023-03-03T08:43:43.982538+0000","last_clean":"2023-03-03T08:43:43.982538+0000","last_became_active":"2023-03-03T08:43:34.550605+0000","last_became_peered":"2023-03-03T08:43:34.550605+0000","last_unstale":"2023-03-03T08:43:43.982538+0000","last_undegraded":"2023-03-03T08:43:43.982538+0000","last_fullsized":"2023-03-03T08:43:43.982538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:01.118555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105567+0000","last_change":"2023-03-03T08:43:43.998231+0000","last_active":"2023-03-03T08:43:45.105567+0000","last_peered":"2023-03-03T08:43:45.105567+0000","last_clean":"2023-03-03T08:43:45.105567+0000","last_became_active":"2023-03-03T08:43:43.997941+0000","last_became_peered":"2023-03-03T08:43:43.997941+0000","last_unstale":"2023-03-03T08:43:45.105567+0000","last_undegraded":"2023-03-03T08:43:45.105567+0000","last_fullsized":"2023-03-03T08:43:45.105567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:13:32.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983100+0000","last_change":"2023-03-03T08:43:34.548229+0000","last_active":"2023-03-03T08:43:43.983100+0000","last_peered":"2023-03-03T08:43:43.983100+0000","last_clean":"2023-03-03T08:43:43.983100+0000","last_became_active":"2023-03-03T08:43:34.548013+0000","last_became_peered":"2023-03-03T08:43:34.548013+0000","last_unstale":"2023-03-03T08:43:43.983100+0000","last_undegraded":"2023-03-03T08:43:43.983100+0000","last_fullsized":"2023-03-03T08:43:43.983100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:27.793716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108186+0000","last_change":"2023-03-03T08:43:43.992834+0000","last_active":"2023-03-03T08:43:45.108186+0000","last_peered":"2023-03-03T08:43:45.108186+0000","last_clean":"2023-03-03T08:43:45.108186+0000","last_became_active":"2023-03-03T08:43:43.991577+0000","last_became_peered":"2023-03-03T08:43:43.991577+0000","last_unstale":"2023-03-03T08:43:45.108186+0000","last_undegraded":"2023-03-03T08:43:45.108186+0000","last_fullsized":"2023-03-03T08:43:45.108186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:18:37.119460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990785+0000","last_change":"2023-03-03T08:43:34.549838+0000","last_active":"2023-03-03T08:43:43.990785+0000","last_peered":"2023-03-03T08:43:43.990785+0000","last_clean":"2023-03-03T08:43:43.990785+0000","last_became_active":"2023-03-03T08:43:34.549537+0000","last_became_peered":"2023-03-03T08:43:34.549537+0000","last_unstale":"2023-03-03T08:43:43.990785+0000","last_undegraded":"2023-03-03T08:43:43.990785+0000","last_fullsized":"2023-03-03T08:43:43.990785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:36.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978412+0000","last_change":"2023-03-03T08:43:43.991488+0000","last_active":"2023-03-03T08:43:44.978412+0000","last_peered":"2023-03-03T08:43:44.978412+0000","last_clean":"2023-03-03T08:43:44.978412+0000","last_became_active":"2023-03-03T08:43:43.991139+0000","last_became_peered":"2023-03-03T08:43:43.991139+0000","last_unstale":"2023-03-03T08:43:44.978412+0000","last_undegraded":"2023-03-03T08:43:44.978412+0000","last_fullsized":"2023-03-03T08:43:44.978412+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:21.119183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983652+0000","last_change":"2023-03-03T08:43:34.533496+0000","last_active":"2023-03-03T08:43:43.983652+0000","last_peered":"2023-03-03T08:43:43.983652+0000","last_clean":"2023-03-03T08:43:43.983652+0000","last_became_active":"2023-03-03T08:43:34.532714+0000","last_became_peered":"2023-03-03T08:43:34.532714+0000","last_unstale":"2023-03-03T08:43:43.983652+0000","last_undegraded":"2023-03-03T08:43:43.983652+0000","last_fullsized":"2023-03-03T08:43:43.983652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:47.800807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105250+0000","last_change":"2023-03-03T08:43:43.996215+0000","last_active":"2023-03-03T08:43:45.105250+0000","last_peered":"2023-03-03T08:43:45.105250+0000","last_clean":"2023-03-03T08:43:45.105250+0000","last_became_active":"2023-03-03T08:43:43.995791+0000","last_became_peered":"2023-03-03T08:43:43.995791+0000","last_unstale":"2023-03-03T08:43:45.105250+0000","last_undegraded":"2023-03-03T08:43:45.105250+0000","last_fullsized":"2023-03-03T08:43:45.105250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:50:41.130427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979929+0000","last_change":"2023-03-03T08:43:34.533673+0000","last_active":"2023-03-03T08:43:43.979929+0000","last_peered":"2023-03-03T08:43:43.979929+0000","last_clean":"2023-03-03T08:43:43.979929+0000","last_became_active":"2023-03-03T08:43:34.532283+0000","last_became_peered":"2023-03-03T08:43:34.532283+0000","last_unstale":"2023-03-03T08:43:43.979929+0000","last_undegraded":"2023-03-03T08:43:43.979929+0000","last_fullsized":"2023-03-03T08:43:43.979929+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:25.591188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979205+0000","last_change":"2023-03-03T08:43:43.999435+0000","last_active":"2023-03-03T08:43:44.979205+0000","last_peered":"2023-03-03T08:43:44.979205+0000","last_clean":"2023-03-03T08:43:44.979205+0000","last_became_active":"2023-03-03T08:43:43.997037+0000","last_became_peered":"2023-03-03T08:43:43.997037+0000","last_unstale":"2023-03-03T08:43:44.979205+0000","last_undegraded":"2023-03-03T08:43:44.979205+0000","last_fullsized":"2023-03-03T08:43:44.979205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:54.176343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980585+0000","last_change":"2023-03-03T08:43:34.537664+0000","last_active":"2023-03-03T08:43:43.980585+0000","last_peered":"2023-03-03T08:43:43.980585+0000","last_clean":"2023-03-03T08:43:43.980585+0000","last_became_active":"2023-03-03T08:43:34.537214+0000","last_became_peered":"2023-03-03T08:43:34.537214+0000","last_unstale":"2023-03-03T08:43:43.980585+0000","last_undegraded":"2023-03-03T08:43:43.980585+0000","last_fullsized":"2023-03-03T08:43:43.980585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:54:09.222135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106227+0000","last_change":"2023-03-03T08:43:44.000140+0000","last_active":"2023-03-03T08:43:45.106227+0000","last_peered":"2023-03-03T08:43:45.106227+0000","last_clean":"2023-03-03T08:43:45.106227+0000","last_became_active":"2023-03-03T08:43:43.999778+0000","last_became_peered":"2023-03-03T08:43:43.999778+0000","last_unstale":"2023-03-03T08:43:45.106227+0000","last_undegraded":"2023-03-03T08:43:45.106227+0000","last_fullsized":"2023-03-03T08:43:45.106227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:29:10.745759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987366+0000","last_change":"2023-03-03T08:43:34.543620+0000","last_active":"2023-03-03T08:43:43.987366+0000","last_peered":"2023-03-03T08:43:43.987366+0000","last_clean":"2023-03-03T08:43:43.987366+0000","last_became_active":"2023-03-03T08:43:34.543370+0000","last_became_peered":"2023-03-03T08:43:34.543370+0000","last_unstale":"2023-03-03T08:43:43.987366+0000","last_undegraded":"2023-03-03T08:43:43.987366+0000","last_fullsized":"2023-03-03T08:43:43.987366+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:40:32.808203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979449+0000","last_change":"2023-03-03T08:43:43.995084+0000","last_active":"2023-03-03T08:43:44.979449+0000","last_peered":"2023-03-03T08:43:44.979449+0000","last_clean":"2023-03-03T08:43:44.979449+0000","last_became_active":"2023-03-03T08:43:43.994824+0000","last_became_peered":"2023-03-03T08:43:43.994824+0000","last_unstale":"2023-03-03T08:43:44.979449+0000","last_undegraded":"2023-03-03T08:43:44.979449+0000","last_fullsized":"2023-03-03T08:43:44.979449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:04:17.481969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985302+0000","last_change":"2023-03-03T08:43:34.538467+0000","last_active":"2023-03-03T08:43:43.985302+0000","last_peered":"2023-03-03T08:43:43.985302+0000","last_clean":"2023-03-03T08:43:43.985302+0000","last_became_active":"2023-03-03T08:43:34.538207+0000","last_became_peered":"2023-03-03T08:43:34.538207+0000","last_unstale":"2023-03-03T08:43:43.985302+0000","last_undegraded":"2023-03-03T08:43:43.985302+0000","last_fullsized":"2023-03-03T08:43:43.985302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:14.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980577+0000","last_change":"2023-03-03T08:43:43.996076+0000","last_active":"2023-03-03T08:43:44.980577+0000","last_peered":"2023-03-03T08:43:44.980577+0000","last_clean":"2023-03-03T08:43:44.980577+0000","last_became_active":"2023-03-03T08:43:43.995799+0000","last_became_peered":"2023-03-03T08:43:43.995799+0000","last_unstale":"2023-03-03T08:43:44.980577+0000","last_undegraded":"2023-03-03T08:43:44.980577+0000","last_fullsized":"2023-03-03T08:43:44.980577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:35:35.889895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986289+0000","last_change":"2023-03-03T08:43:34.544721+0000","last_active":"2023-03-03T08:43:43.986289+0000","last_peered":"2023-03-03T08:43:43.986289+0000","last_clean":"2023-03-03T08:43:43.986289+0000","last_became_active":"2023-03-03T08:43:34.544461+0000","last_became_peered":"2023-03-03T08:43:34.544461+0000","last_unstale":"2023-03-03T08:43:43.986289+0000","last_undegraded":"2023-03-03T08:43:43.986289+0000","last_fullsized":"2023-03-03T08:43:43.986289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:52:11.890779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980078+0000","last_change":"2023-03-03T08:43:44.006352+0000","last_active":"2023-03-03T08:43:44.980078+0000","last_peered":"2023-03-03T08:43:44.980078+0000","last_clean":"2023-03-03T08:43:44.980078+0000","last_became_active":"2023-03-03T08:43:44.005802+0000","last_became_peered":"2023-03-03T08:43:44.005802+0000","last_unstale":"2023-03-03T08:43:44.980078+0000","last_undegraded":"2023-03-03T08:43:44.980078+0000","last_fullsized":"2023-03-03T08:43:44.980078+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:52.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986425+0000","last_change":"2023-03-03T08:43:34.541669+0000","last_active":"2023-03-03T08:43:43.986425+0000","last_peered":"2023-03-03T08:43:43.986425+0000","last_clean":"2023-03-03T08:43:43.986425+0000","last_became_active":"2023-03-03T08:43:34.541104+0000","last_became_peered":"2023-03-03T08:43:34.541104+0000","last_unstale":"2023-03-03T08:43:43.986425+0000","last_undegraded":"2023-03-03T08:43:43.986425+0000","last_fullsized":"2023-03-03T08:43:43.986425+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980059+0000","last_change":"2023-03-03T08:43:43.998937+0000","last_active":"2023-03-03T08:43:44.980059+0000","last_peered":"2023-03-03T08:43:44.980059+0000","last_clean":"2023-03-03T08:43:44.980059+0000","last_became_active":"2023-03-03T08:43:43.998164+0000","last_became_peered":"2023-03-03T08:43:43.998164+0000","last_unstale":"2023-03-03T08:43:44.980059+0000","last_undegraded":"2023-03-03T08:43:44.980059+0000","last_fullsized":"2023-03-03T08:43:44.980059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:53.741704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.546272+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.546006+0000","last_became_peered":"2023-03-03T08:43:34.546006+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:15.121761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":118397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":119470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9336,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362504,"statfs":{"total":96636764160,"available":96627204096,"internally_reserved":0,"allocated":901120,"data_stored":710394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.8689999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.806}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.149}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8244,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363596,"statfs":{"total":96636764160,"available":96628322304,"internally_reserved":0,"allocated":307200,"data_stored":118565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8188,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363652,"statfs":{"total":96636764160,"available":96628379648,"internally_reserved":0,"allocated":315392,"data_stored":121989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":120069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":121018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9260,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362580,"statfs":{"total":96636764160,"available":96627281920,"internally_reserved":0,"allocated":888832,"data_stored":709827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:43:52.601 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-03-03T08:43:52.602 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-03-03T08:43:52.602 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-03-03T08:43:52.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-03-03T08:43:52.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.751+0000 7f1218823700 1 Processor -- start 2023-03-03T08:43:52.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.751+0000 7f1218823700 1 -- start start 2023-03-03T08:43:52.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1218823700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f121414fb00 0x7f1214156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1218823700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12140740f0 0x7f121415cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1218823700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1214155840 con 0x7f12140c1da0 2023-03-03T08:43:52.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1218823700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1214072940 con 0x7f12140740f0 2023-03-03T08:43:52.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1218823700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12141567d0 con 0x7f121414fb00 2023-03-03T08:43:52.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1212d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12140740f0 0x7f121415cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1211d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f12140c1da0 0x7f12140c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45816/0 (socket says 172.21.15.7:45816) 2023-03-03T08:43:52.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1211d9b700 1 -- 172.21.15.7:0/986911055 learned_addr learned my addr 172.21.15.7:0/986911055 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:52.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f121259c700 1 --2- 172.21.15.7:0/986911055 >> v2:172.21.15.143:3300/0 conn(0x7f121414fb00 0x7f1214156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1212d9d700 1 -- 172.21.15.7:0/986911055 >> v2:172.21.15.143:3300/0 conn(0x7f121414fb00 msgr2=0x7f1214156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:52.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1212d9d700 1 --2- 172.21.15.7:0/986911055 >> v2:172.21.15.143:3300/0 conn(0x7f121414fb00 0x7f1214156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1212d9d700 1 -- 172.21.15.7:0/986911055 >> v1:172.21.15.7:6789/0 conn(0x7f12140c1da0 legacy=0x7f12140c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1212d9d700 1 -- 172.21.15.7:0/986911055 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12140c2330 con 0x7f12140740f0 2023-03-03T08:43:52.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f121259c700 1 --2- 172.21.15.7:0/986911055 >> v2:172.21.15.143:3300/0 conn(0x7f121414fb00 0x7f1214156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:52.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.752+0000 7f1212d9d700 1 --2- 172.21.15.7:0/986911055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 0x7f121415cbd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f12140747d0 tx=0x7f120c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0eb016de5ac7676 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:52.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f121159a700 1 -- 172.21.15.7:0/986911055 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c00e070 con 0x7f12140740f0 2023-03-03T08:43:52.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f121159a700 1 -- 172.21.15.7:0/986911055 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c00bbd0 con 0x7f12140740f0 2023-03-03T08:43:52.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f121159a700 1 -- 172.21.15.7:0/986911055 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c012430 con 0x7f12140740f0 2023-03-03T08:43:52.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f1218823700 1 -- 172.21.15.7:0/986911055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 msgr2=0x7f121415cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f1218823700 1 --2- 172.21.15.7:0/986911055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 0x7f121415cbd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f12140747d0 tx=0x7f120c00b330 comp rx=0 tx=0).stop 2023-03-03T08:43:52.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f1218823700 1 -- 172.21.15.7:0/986911055 shutdown_connections 2023-03-03T08:43:52.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f1218823700 1 --2- 172.21.15.7:0/986911055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 0x7f121415cbd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f1218823700 1 --2- 172.21.15.7:0/986911055 >> v2:172.21.15.143:3300/0 conn(0x7f121414fb00 0x7f1214156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f1218823700 1 -- 172.21.15.7:0/986911055 >> 172.21.15.7:0/986911055 conn(0x7f121406c110 msgr2=0x7f121406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:52.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f1218823700 1 -- 172.21.15.7:0/986911055 shutdown_connections 2023-03-03T08:43:52.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.753+0000 7f1218823700 1 -- 172.21.15.7:0/986911055 wait complete. 2023-03-03T08:43:52.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.754+0000 7f1218823700 1 Processor -- start 2023-03-03T08:43:52.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f1218823700 1 -- start start 2023-03-03T08:43:52.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f1218823700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 0x7f12140718d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f1218823700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12140c1da0 0x7f121414acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f1218823700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f121414fb00 0x7f121414b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f1218823700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121414b750 con 0x7f12140740f0 2023-03-03T08:43:52.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f1218823700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1214150c90 con 0x7f121414fb00 2023-03-03T08:43:52.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f1218823700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12141ed170 con 0x7f12140c1da0 2023-03-03T08:43:52.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f121259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 0x7f12140718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f121259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 0x7f12140718d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51842/0 (socket says 172.21.15.7:51842) 2023-03-03T08:43:52.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f121259c700 1 -- 172.21.15.7:0/2745224826 learned_addr learned my addr 172.21.15.7:0/2745224826 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:52.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.755+0000 7f1211d9b700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12140c1da0 0x7f121414acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f1212d9d700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f121414fb00 0x7f121414b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f121259c700 1 -- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12140c1da0 msgr2=0x7f121414acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f121259c700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12140c1da0 0x7f121414acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f121259c700 1 -- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f121414fb00 msgr2=0x7f121414b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:52.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f121259c700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f121414fb00 0x7f121414b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:52.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f121259c700 1 -- 172.21.15.7:0/2745224826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c009a70 con 0x7f12140740f0 2023-03-03T08:43:52.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f1211d9b700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12140c1da0 0x7f121414acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:52.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f121259c700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 0x7f12140718d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f120400c750 tx=0x7f120400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f1202ffd700 1 -- 172.21.15.7:0/2745224826 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204011800 con 0x7f12140740f0 2023-03-03T08:43:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f1202ffd700 1 -- 172.21.15.7:0/2745224826 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1204011960 con 0x7f12140740f0 2023-03-03T08:43:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f1218823700 1 -- 172.21.15.7:0/2745224826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1214153090 con 0x7f12140740f0 2023-03-03T08:43:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f1202ffd700 1 -- 172.21.15.7:0/2745224826 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204011cd0 con 0x7f12140740f0 2023-03-03T08:43:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.756+0000 7f1218823700 1 -- 172.21.15.7:0/2745224826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1214152db0 con 0x7f12140740f0 2023-03-03T08:43:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.757+0000 7f1218823700 1 -- 172.21.15.7:0/2745224826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1214155a90 con 0x7f12140740f0 2023-03-03T08:43:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.757+0000 7f1202ffd700 1 -- 172.21.15.7:0/2745224826 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1204020070 con 0x7f12140740f0 2023-03-03T08:43:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.758+0000 7f1202ffd700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11f40682f0 0x7f11f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:52.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.758+0000 7f1202ffd700 1 -- 172.21.15.7:0/2745224826 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f1204085e10 con 0x7f12140740f0 2023-03-03T08:43:52.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.758+0000 7f1211d9b700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11f40682f0 0x7f11f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:52.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.758+0000 7f1211d9b700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11f40682f0 0x7f11f406a7a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1208007fd0 tx=0x7f1208009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:52.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:52.762+0000 7f1202ffd700 1 -- 172.21.15.7:0/2745224826 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1204052340 con 0x7f12140740f0 2023-03-03T08:43:53.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.000+0000 7f1218823700 1 -- 172.21.15.7:0/2745224826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f121406e410 con 0x7f12140740f0 2023-03-03T08:43:53.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.000+0000 7f1202ffd700 1 -- 172.21.15.7:0/2745224826 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1204055960 con 0x7f12140740f0 2023-03-03T08:43:53.003 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:53.003 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-03-03T08:43:53.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.002+0000 7f1218823700 1 -- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11f40682f0 msgr2=0x7f11f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.002+0000 7f1218823700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11f40682f0 0x7f11f406a7a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1208007fd0 tx=0x7f1208009040 comp rx=0 tx=0).stop 2023-03-03T08:43:53.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.002+0000 7f1218823700 1 -- 172.21.15.7:0/2745224826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 msgr2=0x7f12140718d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.002+0000 7f1218823700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 0x7f12140718d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f120400c750 tx=0x7f120400cb10 comp rx=0 tx=0).stop 2023-03-03T08:43:53.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.002+0000 7f1218823700 1 -- 172.21.15.7:0/2745224826 shutdown_connections 2023-03-03T08:43:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.002+0000 7f1218823700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f121414fb00 0x7f121414b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.002+0000 7f1218823700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11f40682f0 0x7f11f406a7a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.002+0000 7f1218823700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12140740f0 0x7f12140718d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.002+0000 7f1218823700 1 --2- 172.21.15.7:0/2745224826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12140c1da0 0x7f121414acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.002+0000 7f1218823700 1 -- 172.21.15.7:0/2745224826 >> 172.21.15.7:0/2745224826 conn(0x7f121406c110 msgr2=0x7f12140aac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.003+0000 7f1218823700 1 -- 172.21.15.7:0/2745224826 shutdown_connections 2023-03-03T08:43:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.003+0000 7f1218823700 1 -- 172.21.15.7:0/2745224826 wait complete. 2023-03-03T08:43:53.017 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-03-03T08:43:53.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:43:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.140+0000 7efc5becb700 1 Processor -- start 2023-03-03T08:43:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.140+0000 7efc5becb700 1 -- start start 2023-03-03T08:43:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5becb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc540c1da0 0x7efc541550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5becb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc540c3e30 0x7efc540c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5becb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc540c7a60 con 0x7efc5406d870 2023-03-03T08:43:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5becb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc540c2af0 con 0x7efc540c3e30 2023-03-03T08:43:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5becb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc540c3620 con 0x7efc540c1da0 2023-03-03T08:43:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5a468700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc540c3e30 0x7efc540c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc59c67700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7efc5406d870 0x7efc540c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45828/0 (socket says 172.21.15.7:45828) 2023-03-03T08:43:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5a468700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc540c3e30 0x7efc540c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51852/0 (socket says 172.21.15.7:51852) 2023-03-03T08:43:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc59c67700 1 -- 172.21.15.7:0/2332848427 learned_addr learned my addr 172.21.15.7:0/2332848427 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc59466700 1 --2- 172.21.15.7:0/2332848427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc540c1da0 0x7efc541550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5a468700 1 -- 172.21.15.7:0/2332848427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc540c1da0 msgr2=0x7efc541550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5a468700 1 --2- 172.21.15.7:0/2332848427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc540c1da0 0x7efc541550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5a468700 1 -- 172.21.15.7:0/2332848427 >> v1:172.21.15.7:6789/0 conn(0x7efc5406d870 legacy=0x7efc540c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc58c65700 1 -- 172.21.15.7:0/2332848427 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689205564 0 0) 0x7efc540c7a60 con 0x7efc5406d870 2023-03-03T08:43:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.141+0000 7efc5a468700 1 -- 172.21.15.7:0/2332848427 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc541545e0 con 0x7efc540c3e30 2023-03-03T08:43:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc5a468700 1 --2- 172.21.15.7:0/2332848427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc540c3e30 0x7efc540c4290 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efc4c009d90 tx=0x7efc4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4672ec442c206684 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc58c65700 1 -- 172.21.15.7:0/2332848427 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c00e070 con 0x7efc540c3e30 2023-03-03T08:43:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc58c65700 1 -- 172.21.15.7:0/2332848427 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4c00bbd0 con 0x7efc540c3e30 2023-03-03T08:43:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc58c65700 1 -- 172.21.15.7:0/2332848427 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c012430 con 0x7efc540c3e30 2023-03-03T08:43:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc5becb700 1 -- 172.21.15.7:0/2332848427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc540c3e30 msgr2=0x7efc540c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc5becb700 1 --2- 172.21.15.7:0/2332848427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc540c3e30 0x7efc540c4290 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efc4c009d90 tx=0x7efc4c00b330 comp rx=0 tx=0).stop 2023-03-03T08:43:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc5becb700 1 -- 172.21.15.7:0/2332848427 shutdown_connections 2023-03-03T08:43:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc5becb700 1 --2- 172.21.15.7:0/2332848427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc540c3e30 0x7efc540c4290 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc5becb700 1 --2- 172.21.15.7:0/2332848427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc540c1da0 0x7efc541550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc5becb700 1 -- 172.21.15.7:0/2332848427 >> 172.21.15.7:0/2332848427 conn(0x7efc5406c110 msgr2=0x7efc5406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc5becb700 1 -- 172.21.15.7:0/2332848427 shutdown_connections 2023-03-03T08:43:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.142+0000 7efc5becb700 1 -- 172.21.15.7:0/2332848427 wait complete. 2023-03-03T08:43:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.143+0000 7efc5becb700 1 Processor -- start 2023-03-03T08:43:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc5becb700 1 -- start start 2023-03-03T08:43:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc5becb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc5406d870 0x7efc541558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc5becb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc540c1da0 0x7efc541ea2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc5becb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc540c3e30 0x7efc540c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc5becb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc541ea800 con 0x7efc5406d870 2023-03-03T08:43:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc5becb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc540ab540 con 0x7efc540c1da0 2023-03-03T08:43:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc5becb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc540ab820 con 0x7efc540c3e30 2023-03-03T08:43:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc59c67700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc5406d870 0x7efc541558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc59c67700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc5406d870 0x7efc541558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51854/0 (socket says 172.21.15.7:51854) 2023-03-03T08:43:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc59c67700 1 -- 172.21.15.7:0/2107608353 learned_addr learned my addr 172.21.15.7:0/2107608353 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc5a468700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc540c3e30 0x7efc540c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.144+0000 7efc59466700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc540c1da0 0x7efc541ea2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc59c67700 1 -- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc540c3e30 msgr2=0x7efc540c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc59c67700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc540c3e30 0x7efc540c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc59c67700 1 -- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc540c1da0 msgr2=0x7efc541ea2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc59c67700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc540c1da0 0x7efc541ea2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc59c67700 1 -- 172.21.15.7:0/2107608353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c009a70 con 0x7efc5406d870 2023-03-03T08:43:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc59466700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc540c1da0 0x7efc541ea2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc59c67700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc5406d870 0x7efc541558a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efc440011e0 tx=0x7efc4400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc4a7fc700 1 -- 172.21.15.7:0/2107608353 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc44004c80 con 0x7efc5406d870 2023-03-03T08:43:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc4a7fc700 1 -- 172.21.15.7:0/2107608353 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc44004de0 con 0x7efc5406d870 2023-03-03T08:43:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc5becb700 1 -- 172.21.15.7:0/2107608353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc54154ba0 con 0x7efc5406d870 2023-03-03T08:43:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc4a7fc700 1 -- 172.21.15.7:0/2107608353 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc44007310 con 0x7efc5406d870 2023-03-03T08:43:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.145+0000 7efc5becb700 1 -- 172.21.15.7:0/2107608353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc541548c0 con 0x7efc5406d870 2023-03-03T08:43:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.146+0000 7efc5becb700 1 -- 172.21.15.7:0/2107608353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5406dc40 con 0x7efc5406d870 2023-03-03T08:43:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.146+0000 7efc4a7fc700 1 -- 172.21.15.7:0/2107608353 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efc4401a020 con 0x7efc5406d870 2023-03-03T08:43:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.147+0000 7efc4a7fc700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc380682f0 0x7efc3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.147+0000 7efc4a7fc700 1 -- 172.21.15.7:0/2107608353 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7efc44085b70 con 0x7efc5406d870 2023-03-03T08:43:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.147+0000 7efc59466700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc380682f0 0x7efc3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.147+0000 7efc59466700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc380682f0 0x7efc3806a7a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efc540c71d0 tx=0x7efc50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.151+0000 7efc4a7fc700 1 -- 172.21.15.7:0/2107608353 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efc440520a0 con 0x7efc5406d870 2023-03-03T08:43:53.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.347+0000 7efc5becb700 1 -- 172.21.15.7:0/2107608353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc54069730 con 0x7efc5406d870 2023-03-03T08:43:53.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.348+0000 7efc4a7fc700 1 -- 172.21.15.7:0/2107608353 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21533 (secure 0 0 0) 0x7efc440556c0 con 0x7efc5406d870 2023-03-03T08:43:53.351 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:53.351 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:44.973251+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:43:53.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 -- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc380682f0 msgr2=0x7efc3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc380682f0 0x7efc3806a7a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efc540c71d0 tx=0x7efc50008040 comp rx=0 tx=0).stop 2023-03-03T08:43:53.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 -- 172.21.15.7:0/2107608353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc5406d870 msgr2=0x7efc541558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc5406d870 0x7efc541558a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efc440011e0 tx=0x7efc4400ca30 comp rx=0 tx=0).stop 2023-03-03T08:43:53.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 -- 172.21.15.7:0/2107608353 shutdown_connections 2023-03-03T08:43:53.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc540c3e30 0x7efc540c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc380682f0 0x7efc3806a7a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc5406d870 0x7efc541558a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 --2- 172.21.15.7:0/2107608353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc540c1da0 0x7efc541ea2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 -- 172.21.15.7:0/2107608353 >> 172.21.15.7:0/2107608353 conn(0x7efc5406c110 msgr2=0x7efc54152340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 -- 172.21.15.7:0/2107608353 shutdown_connections 2023-03-03T08:43:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.350+0000 7efc5becb700 1 -- 172.21.15.7:0/2107608353 wait complete. 2023-03-03T08:43:53.365 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T08:43:21.160680+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T08:43:28.018457+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-03-03T08:43:32.305037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-03-03T08:43:33.484501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-03-03T08:43:42.509451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-03-03T08:43:42.936898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:43:53.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:43:53.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.492+0000 7faff60a5700 1 Processor -- start 2023-03-03T08:43:53.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.492+0000 7faff60a5700 1 -- start start 2023-03-03T08:43:53.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.493+0000 7faff60a5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faff014de50 0x7faff006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.493+0000 7faff60a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff006a7f0 0x7faff00c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.493+0000 7faff60a5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff00a65b0 con 0x7faff00c65c0 2023-03-03T08:43:53.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.493+0000 7faff60a5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff0154e00 con 0x7faff014de50 2023-03-03T08:43:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.493+0000 7faff60a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff00c3370 con 0x7faff006a7f0 2023-03-03T08:43:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.493+0000 7fafef7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faff014de50 0x7faff006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.493+0000 7fafeeffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faff00c65c0 0x7faff00c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45844/0 (socket says 172.21.15.7:45844) 2023-03-03T08:43:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.493+0000 7fafef7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faff014de50 0x7faff006a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51862/0 (socket says 172.21.15.7:51862) 2023-03-03T08:43:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.493+0000 7fafeeffd700 1 -- 172.21.15.7:0/2414918366 learned_addr learned my addr 172.21.15.7:0/2414918366 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.493+0000 7fafeffff700 1 --2- 172.21.15.7:0/2414918366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff006a7f0 0x7faff00c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7fafef7fe700 1 -- 172.21.15.7:0/2414918366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff006a7f0 msgr2=0x7faff00c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7fafef7fe700 1 --2- 172.21.15.7:0/2414918366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff006a7f0 0x7faff00c2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7fafef7fe700 1 -- 172.21.15.7:0/2414918366 >> v1:172.21.15.7:6789/0 conn(0x7faff00c65c0 legacy=0x7faff00c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7fafef7fe700 1 -- 172.21.15.7:0/2414918366 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff01541f0 con 0x7faff014de50 2023-03-03T08:43:53.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7fafee7fc700 1 -- 172.21.15.7:0/2414918366 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126674200 0 0) 0x7faff00a65b0 con 0x7faff00c65c0 2023-03-03T08:43:53.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7fafef7fe700 1 --2- 172.21.15.7:0/2414918366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff014de50 0x7faff006a2b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fafd800bb10 tx=0x7fafd800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec936f85d4b3938a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:53.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7fafee7fc700 1 -- 172.21.15.7:0/2414918366 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd800dbe0 con 0x7faff014de50 2023-03-03T08:43:53.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7fafee7fc700 1 -- 172.21.15.7:0/2414918366 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd800dd40 con 0x7faff014de50 2023-03-03T08:43:53.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7fafee7fc700 1 -- 172.21.15.7:0/2414918366 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8017630 con 0x7faff014de50 2023-03-03T08:43:53.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7faff60a5700 1 -- 172.21.15.7:0/2414918366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff014de50 msgr2=0x7faff006a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7faff60a5700 1 --2- 172.21.15.7:0/2414918366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff014de50 0x7faff006a2b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fafd800bb10 tx=0x7fafd800be20 comp rx=0 tx=0).stop 2023-03-03T08:43:53.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7faff60a5700 1 -- 172.21.15.7:0/2414918366 shutdown_connections 2023-03-03T08:43:53.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7faff60a5700 1 --2- 172.21.15.7:0/2414918366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff006a7f0 0x7faff00c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7faff60a5700 1 --2- 172.21.15.7:0/2414918366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff014de50 0x7faff006a2b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7faff60a5700 1 -- 172.21.15.7:0/2414918366 >> 172.21.15.7:0/2414918366 conn(0x7faff00a9430 msgr2=0x7faff00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:53.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7faff60a5700 1 -- 172.21.15.7:0/2414918366 shutdown_connections 2023-03-03T08:43:53.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.494+0000 7faff60a5700 1 -- 172.21.15.7:0/2414918366 wait complete. 2023-03-03T08:43:53.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.496+0000 7faff60a5700 1 Processor -- start 2023-03-03T08:43:53.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.496+0000 7faff60a5700 1 -- start start 2023-03-03T08:43:53.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.496+0000 7faff60a5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff006a7f0 0x7faff01e8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.496+0000 7faff60a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff00c65c0 0x7faff014fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.496+0000 7faff60a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff014de50 0x7faff0150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.496+0000 7faff60a5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff014d1f0 con 0x7faff006a7f0 2023-03-03T08:43:53.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.496+0000 7faff60a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff01e97d0 con 0x7faff014de50 2023-03-03T08:43:53.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.496+0000 7faff60a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff0151710 con 0x7faff00c65c0 2023-03-03T08:43:53.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff006a7f0 0x7faff01e8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff006a7f0 0x7faff01e8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51868/0 (socket says 172.21.15.7:51868) 2023-03-03T08:43:53.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafef7fe700 1 -- 172.21.15.7:0/3708480750 learned_addr learned my addr 172.21.15.7:0/3708480750 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:53.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafeeffd700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff00c65c0 0x7faff014fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafeffff700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff014de50 0x7faff0150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafef7fe700 1 -- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff00c65c0 msgr2=0x7faff014fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafef7fe700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff00c65c0 0x7faff014fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafef7fe700 1 -- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff014de50 msgr2=0x7faff0150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafef7fe700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff014de50 0x7faff0150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafef7fe700 1 -- 172.21.15.7:0/3708480750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd800b740 con 0x7faff006a7f0 2023-03-03T08:43:53.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafeeffd700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff00c65c0 0x7faff014fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:53.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafef7fe700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff006a7f0 0x7faff01e8b00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faff00c79c0 tx=0x7fafd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:53.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafe7fff700 1 -- 172.21.15.7:0/3708480750 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8017630 con 0x7faff006a7f0 2023-03-03T08:43:53.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafe7fff700 1 -- 172.21.15.7:0/3708480750 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd80179f0 con 0x7faff006a7f0 2023-03-03T08:43:53.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7faff60a5700 1 -- 172.21.15.7:0/3708480750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff01547b0 con 0x7faff006a7f0 2023-03-03T08:43:53.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7fafe7fff700 1 -- 172.21.15.7:0/3708480750 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8017da0 con 0x7faff006a7f0 2023-03-03T08:43:53.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.497+0000 7faff60a5700 1 -- 172.21.15.7:0/3708480750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff01544d0 con 0x7faff006a7f0 2023-03-03T08:43:53.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.498+0000 7faff60a5700 1 -- 172.21.15.7:0/3708480750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff0154a90 con 0x7faff006a7f0 2023-03-03T08:43:53.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.499+0000 7fafe7fff700 1 -- 172.21.15.7:0/3708480750 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fafd80036a0 con 0x7faff006a7f0 2023-03-03T08:43:53.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.499+0000 7fafe7fff700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafd00682f0 0x7fafd006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.499+0000 7fafe7fff700 1 -- 172.21.15.7:0/3708480750 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fafd8089fe0 con 0x7faff006a7f0 2023-03-03T08:43:53.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.499+0000 7fafeeffd700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafd00682f0 0x7fafd006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.500+0000 7fafeeffd700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafd00682f0 0x7fafd006a7a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faff00c2c30 tx=0x7fafe0007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:53.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.503+0000 7fafe7fff700 1 -- 172.21.15.7:0/3708480750 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fafd8056510 con 0x7faff006a7f0 2023-03-03T08:43:53.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.695+0000 7faff60a5700 1 -- 172.21.15.7:0/3708480750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faff0074bb0 con 0x7faff006a7f0 2023-03-03T08:43:53.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.695+0000 7fafe7fff700 1 -- 172.21.15.7:0/3708480750 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fafd8026020 con 0x7faff006a7f0 2023-03-03T08:43:53.698 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-03-03T08:43:53.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.697+0000 7faff60a5700 1 -- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafd00682f0 msgr2=0x7fafd006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.697+0000 7faff60a5700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafd00682f0 0x7fafd006a7a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faff00c2c30 tx=0x7fafe0007ca0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.697+0000 7faff60a5700 1 -- 172.21.15.7:0/3708480750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff006a7f0 msgr2=0x7faff01e8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.697+0000 7faff60a5700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff006a7f0 0x7faff01e8b00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faff00c79c0 tx=0x7fafd8004750 comp rx=0 tx=0).stop 2023-03-03T08:43:53.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.697+0000 7faff60a5700 1 -- 172.21.15.7:0/3708480750 shutdown_connections 2023-03-03T08:43:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.697+0000 7faff60a5700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff014de50 0x7faff0150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.697+0000 7faff60a5700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafd00682f0 0x7fafd006a7a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.697+0000 7faff60a5700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff006a7f0 0x7faff01e8b00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.697+0000 7faff60a5700 1 --2- 172.21.15.7:0/3708480750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff00c65c0 0x7faff014fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.697+0000 7faff60a5700 1 -- 172.21.15.7:0/3708480750 >> 172.21.15.7:0/3708480750 conn(0x7faff00a9430 msgr2=0x7faff00a65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.698+0000 7faff60a5700 1 -- 172.21.15.7:0/3708480750 shutdown_connections 2023-03-03T08:43:53.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.698+0000 7faff60a5700 1 -- 172.21.15.7:0/3708480750 wait complete. 2023-03-03T08:43:53.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:43:53.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.836+0000 7f9acdf9e700 1 Processor -- start 2023-03-03T08:43:53.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.836+0000 7f9acdf9e700 1 -- start start 2023-03-03T08:43:53.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9acdf9e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9ac814f860 0x7f9ac81562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9acdf9e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9ac80c27c0 0x7f9ac80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9acdf9e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac8155e60 con 0x7f9ac80c20d0 2023-03-03T08:43:53.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9acdf9e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac806d350 con 0x7f9ac80c27c0 2023-03-03T08:43:53.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9acdf9e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac81558e0 con 0x7f9ac814f860 2023-03-03T08:43:53.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9ac7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9ac80c27c0 0x7f9ac80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9ac7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9ac80c27c0 0x7f9ac80736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51880/0 (socket says 172.21.15.7:51880) 2023-03-03T08:43:53.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9ac7fff700 1 -- 172.21.15.7:0/1558862733 learned_addr learned my addr 172.21.15.7:0/1558862733 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:53.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9ac77fe700 1 --2- 172.21.15.7:0/1558862733 >> v2:172.21.15.143:3300/0 conn(0x7f9ac814f860 0x7f9ac81562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9ac7fff700 1 -- 172.21.15.7:0/1558862733 >> v2:172.21.15.143:3300/0 conn(0x7f9ac814f860 msgr2=0x7f9ac81562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:53.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9ac7fff700 1 --2- 172.21.15.7:0/1558862733 >> v2:172.21.15.143:3300/0 conn(0x7f9ac814f860 0x7f9ac81562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9ac7fff700 1 -- 172.21.15.7:0/1558862733 >> v1:172.21.15.143:6789/0 conn(0x7f9ac80c20d0 legacy=0x7f9ac8152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9ac7fff700 1 -- 172.21.15.7:0/1558862733 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac806cd90 con 0x7f9ac80c27c0 2023-03-03T08:43:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.837+0000 7f9ac77fe700 1 --2- 172.21.15.7:0/1558862733 >> v2:172.21.15.143:3300/0 conn(0x7f9ac814f860 0x7f9ac81562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9ac7fff700 1 --2- 172.21.15.7:0/1558862733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 0x7f9ac80736f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9ac806fc80 tx=0x7f9ab8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8e9a7dc449f20d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9ac67fc700 1 -- 172.21.15.7:0/1558862733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8004230 con 0x7f9ac80c27c0 2023-03-03T08:43:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9ac67fc700 1 -- 172.21.15.7:0/1558862733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab8004390 con 0x7f9ac80c27c0 2023-03-03T08:43:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9ac67fc700 1 -- 172.21.15.7:0/1558862733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80036a0 con 0x7f9ac80c27c0 2023-03-03T08:43:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9acdf9e700 1 -- 172.21.15.7:0/1558862733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 msgr2=0x7f9ac80736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9acdf9e700 1 --2- 172.21.15.7:0/1558862733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 0x7f9ac80736f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9ac806fc80 tx=0x7f9ab8009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9acdf9e700 1 -- 172.21.15.7:0/1558862733 shutdown_connections 2023-03-03T08:43:53.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9acdf9e700 1 --2- 172.21.15.7:0/1558862733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 0x7f9ac80736f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9acdf9e700 1 --2- 172.21.15.7:0/1558862733 >> v2:172.21.15.143:3300/0 conn(0x7f9ac814f860 0x7f9ac81562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9acdf9e700 1 -- 172.21.15.7:0/1558862733 >> 172.21.15.7:0/1558862733 conn(0x7f9ac80a94d0 msgr2=0x7f9ac80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:53.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9acdf9e700 1 -- 172.21.15.7:0/1558862733 shutdown_connections 2023-03-03T08:43:53.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.838+0000 7f9acdf9e700 1 -- 172.21.15.7:0/1558862733 wait complete. 2023-03-03T08:43:53.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.839+0000 7f9acdf9e700 1 Processor -- start 2023-03-03T08:43:53.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.840+0000 7f9acdf9e700 1 -- start start 2023-03-03T08:43:53.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.840+0000 7f9acdf9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ac80c20d0 0x7f9ac81549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.840+0000 7f9acdf9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 0x7f9ac80c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.840+0000 7f9acdf9e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ac814f860 0x7f9ac80c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.840+0000 7f9acdf9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80c6dd0 con 0x7f9ac80c27c0 2023-03-03T08:43:53.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.840+0000 7f9acdf9e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac814aab0 con 0x7f9ac814f860 2023-03-03T08:43:53.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.840+0000 7f9acdf9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac80c7290 con 0x7f9ac80c20d0 2023-03-03T08:43:53.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.840+0000 7f9ac6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 0x7f9ac80c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.840+0000 7f9ac6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 0x7f9ac80c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51884/0 (socket says 172.21.15.7:51884) 2023-03-03T08:43:53.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.840+0000 7f9ac6ffd700 1 -- 172.21.15.7:0/2250177495 learned_addr learned my addr 172.21.15.7:0/2250177495 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:53.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9ac77fe700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ac80c20d0 0x7f9ac81549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9ac7fff700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ac814f860 0x7f9ac80c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9ac6ffd700 1 -- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ac80c20d0 msgr2=0x7f9ac81549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9ac6ffd700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ac80c20d0 0x7f9ac81549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9ac6ffd700 1 -- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ac814f860 msgr2=0x7f9ac80c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:53.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9ac6ffd700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ac814f860 0x7f9ac80c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:53.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9ac6ffd700 1 -- 172.21.15.7:0/2250177495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8009550 con 0x7f9ac80c27c0 2023-03-03T08:43:53.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9ac6ffd700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 0x7f9ac80c51f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c4d0 tx=0x7f9ab000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:53.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9abffff700 1 -- 172.21.15.7:0/2250177495 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0011790 con 0x7f9ac80c27c0 2023-03-03T08:43:53.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9abffff700 1 -- 172.21.15.7:0/2250177495 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab00118f0 con 0x7f9ac80c27c0 2023-03-03T08:43:53.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9acdf9e700 1 -- 172.21.15.7:0/2250177495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac8155af0 con 0x7f9ac80c27c0 2023-03-03T08:43:53.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9abffff700 1 -- 172.21.15.7:0/2250177495 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0011ca0 con 0x7f9ac80c27c0 2023-03-03T08:43:53.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.841+0000 7f9acdf9e700 1 -- 172.21.15.7:0/2250177495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac806d070 con 0x7f9ac80c27c0 2023-03-03T08:43:53.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.842+0000 7f9acdf9e700 1 -- 172.21.15.7:0/2250177495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac814fc30 con 0x7f9ac80c27c0 2023-03-03T08:43:53.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.842+0000 7f9abffff700 1 -- 172.21.15.7:0/2250177495 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ab0010050 con 0x7f9ac80c27c0 2023-03-03T08:43:53.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.843+0000 7f9abffff700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9aa80682f0 0x7f9aa806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:53.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.843+0000 7f9abffff700 1 -- 172.21.15.7:0/2250177495 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f9ab0085ba0 con 0x7f9ac80c27c0 2023-03-03T08:43:53.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.843+0000 7f9ac77fe700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9aa80682f0 0x7f9aa806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:53.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.844+0000 7f9ac77fe700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9aa80682f0 0x7f9aa806a7a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9ac81551d0 tx=0x7f9ac0007770 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:53.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:53.847+0000 7f9abffff700 1 -- 172.21.15.7:0/2250177495 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ab00520d0 con 0x7f9ac80c27c0 2023-03-03T08:43:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.037+0000 7f9acdf9e700 1 -- 172.21.15.7:0/2250177495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9ac80aaaa0 con 0x7f9ac80c27c0 2023-03-03T08:43:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.037+0000 7f9abffff700 1 -- 172.21.15.7:0/2250177495 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9ac80aaaa0 con 0x7f9ac80c27c0 2023-03-03T08:43:54.040 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 8 2023-03-03T08:43:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 -- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9aa80682f0 msgr2=0x7f9aa806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9aa80682f0 0x7f9aa806a7a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9ac81551d0 tx=0x7f9ac0007770 comp rx=0 tx=0).stop 2023-03-03T08:43:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 -- 172.21.15.7:0/2250177495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 msgr2=0x7f9ac80c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 0x7f9ac80c51f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c4d0 tx=0x7f9ab000c7e0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 -- 172.21.15.7:0/2250177495 shutdown_connections 2023-03-03T08:43:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ac814f860 0x7f9ac80c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ac80c20d0 0x7f9ac81549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9aa80682f0 0x7f9aa806a7a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 --2- 172.21.15.7:0/2250177495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ac80c27c0 0x7f9ac80c51f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 -- 172.21.15.7:0/2250177495 >> 172.21.15.7:0/2250177495 conn(0x7f9ac80a94d0 msgr2=0x7f9ac8065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 -- 172.21.15.7:0/2250177495 shutdown_connections 2023-03-03T08:43:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.039+0000 7f9acdf9e700 1 -- 172.21.15.7:0/2250177495 wait complete. 2023-03-03T08:43:54.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-03-03T08:43:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.180+0000 7f85ef5b5700 1 Processor -- start 2023-03-03T08:43:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.180+0000 7f85ef5b5700 1 -- start start 2023-03-03T08:43:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85ef5b5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f85e806a2b0 0x7f85e806a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85ef5b5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85e80a6370 0x7f85e8155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85ef5b5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85e80c5cb0 0x7f85e81562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85ef5b5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e806bab0 con 0x7f85e80c5cb0 2023-03-03T08:43:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85ef5b5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e8156800 con 0x7f85e806a2b0 2023-03-03T08:43:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85ef5b5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e81558f0 con 0x7f85e80a6370 2023-03-03T08:43:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85edb52700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85e80c5cb0 0x7f85e81562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85edb52700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85e80c5cb0 0x7f85e81562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51894/0 (socket says 172.21.15.7:51894) 2023-03-03T08:43:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85edb52700 1 -- 172.21.15.7:0/3420369421 learned_addr learned my addr 172.21.15.7:0/3420369421 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85ecb50700 1 --2- 172.21.15.7:0/3420369421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85e80a6370 0x7f85e8155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85ed351700 1 --2- 172.21.15.7:0/3420369421 >> v2:172.21.15.143:3300/0 conn(0x7f85e806a2b0 0x7f85e806a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85edb52700 1 -- 172.21.15.7:0/3420369421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85e80a6370 msgr2=0x7f85e8155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85edb52700 1 --2- 172.21.15.7:0/3420369421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85e80a6370 0x7f85e8155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85edb52700 1 -- 172.21.15.7:0/3420369421 >> v2:172.21.15.143:3300/0 conn(0x7f85e806a2b0 msgr2=0x7f85e806a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85edb52700 1 --2- 172.21.15.7:0/3420369421 >> v2:172.21.15.143:3300/0 conn(0x7f85e806a2b0 0x7f85e806a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85edb52700 1 -- 172.21.15.7:0/3420369421 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e80a6180 con 0x7f85e80c5cb0 2023-03-03T08:43:54.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.181+0000 7f85edb52700 1 --2- 172.21.15.7:0/3420369421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80c5cb0 0x7f85e81562c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f85e80c7a70 tx=0x7f85e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a26a9427803914 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:54.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85dffff700 1 -- 172.21.15.7:0/3420369421 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e000e070 con 0x7f85e80c5cb0 2023-03-03T08:43:54.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85dffff700 1 -- 172.21.15.7:0/3420369421 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e000b910 con 0x7f85e80c5cb0 2023-03-03T08:43:54.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85dffff700 1 -- 172.21.15.7:0/3420369421 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e000bcc0 con 0x7f85e80c5cb0 2023-03-03T08:43:54.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85ef5b5700 1 -- 172.21.15.7:0/3420369421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80c5cb0 msgr2=0x7f85e81562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85ef5b5700 1 --2- 172.21.15.7:0/3420369421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80c5cb0 0x7f85e81562c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f85e80c7a70 tx=0x7f85e000b330 comp rx=0 tx=0).stop 2023-03-03T08:43:54.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85ef5b5700 1 -- 172.21.15.7:0/3420369421 shutdown_connections 2023-03-03T08:43:54.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85ef5b5700 1 --2- 172.21.15.7:0/3420369421 >> v2:172.21.15.143:3300/0 conn(0x7f85e806a2b0 0x7f85e806a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85ef5b5700 1 --2- 172.21.15.7:0/3420369421 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80c5cb0 0x7f85e81562c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85ef5b5700 1 --2- 172.21.15.7:0/3420369421 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85e80a6370 0x7f85e8155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85ef5b5700 1 -- 172.21.15.7:0/3420369421 >> 172.21.15.7:0/3420369421 conn(0x7f85e80a9430 msgr2=0x7f85e80c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:54.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85ef5b5700 1 -- 172.21.15.7:0/3420369421 shutdown_connections 2023-03-03T08:43:54.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.182+0000 7f85ef5b5700 1 -- 172.21.15.7:0/3420369421 wait complete. 2023-03-03T08:43:54.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.183+0000 7f85ef5b5700 1 Processor -- start 2023-03-03T08:43:54.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.183+0000 7f85ef5b5700 1 -- start start 2023-03-03T08:43:54.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ef5b5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85e806a2b0 0x7f85e81ea120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ef5b5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80a6370 0x7f85e80c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ef5b5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85e80c5cb0 0x7f85e8151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ef5b5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e81ea660 con 0x7f85e80a6370 2023-03-03T08:43:54.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ef5b5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e8150480 con 0x7f85e806a2b0 2023-03-03T08:43:54.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ef5b5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e814ff00 con 0x7f85e80c5cb0 2023-03-03T08:43:54.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ecb50700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80a6370 0x7f85e80c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ecb50700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80a6370 0x7f85e80c00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51898/0 (socket says 172.21.15.7:51898) 2023-03-03T08:43:54.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ecb50700 1 -- 172.21.15.7:0/625044657 learned_addr learned my addr 172.21.15.7:0/625044657 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:54.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ed351700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85e806a2b0 0x7f85e81ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85edb52700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85e80c5cb0 0x7f85e8151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ecb50700 1 -- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85e80c5cb0 msgr2=0x7f85e8151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.184+0000 7f85ecb50700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85e80c5cb0 0x7f85e8151950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.185+0000 7f85ecb50700 1 -- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85e806a2b0 msgr2=0x7f85e81ea120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.185+0000 7f85ecb50700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85e806a2b0 0x7f85e81ea120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.185+0000 7f85ecb50700 1 -- 172.21.15.7:0/625044657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e0009b30 con 0x7f85e80a6370 2023-03-03T08:43:54.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.185+0000 7f85ecb50700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80a6370 0x7f85e80c00e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f85e400e9b0 tx=0x7f85e400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:54.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.185+0000 7f85ddffb700 1 -- 172.21.15.7:0/625044657 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e4009710 con 0x7f85e80a6370 2023-03-03T08:43:54.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.185+0000 7f85ddffb700 1 -- 172.21.15.7:0/625044657 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e400ee50 con 0x7f85e80a6370 2023-03-03T08:43:54.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.185+0000 7f85ef5b5700 1 -- 172.21.15.7:0/625044657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e8152e80 con 0x7f85e80a6370 2023-03-03T08:43:54.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.185+0000 7f85ddffb700 1 -- 172.21.15.7:0/625044657 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e400ca50 con 0x7f85e80a6370 2023-03-03T08:43:54.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.185+0000 7f85ef5b5700 1 -- 172.21.15.7:0/625044657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e80a5ea0 con 0x7f85e80a6370 2023-03-03T08:43:54.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.186+0000 7f85ef5b5700 1 -- 172.21.15.7:0/625044657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e8153160 con 0x7f85e80a6370 2023-03-03T08:43:54.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.187+0000 7f85ddffb700 1 -- 172.21.15.7:0/625044657 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f85e4019030 con 0x7f85e80a6370 2023-03-03T08:43:54.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.187+0000 7f85ddffb700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85cc0682f0 0x7f85cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.187+0000 7f85ddffb700 1 -- 172.21.15.7:0/625044657 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f85e4085e00 con 0x7f85e80a6370 2023-03-03T08:43:54.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.187+0000 7f85ed351700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85cc0682f0 0x7f85cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.188+0000 7f85ed351700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85cc0682f0 0x7f85cc06a7a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009b40 tx=0x7f85d8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:54.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.191+0000 7f85ddffb700 1 -- 172.21.15.7:0/625044657 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f85e4052330 con 0x7f85e80a6370 2023-03-03T08:43:54.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.384+0000 7f85ef5b5700 1 -- 172.21.15.7:0/625044657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f85e8074bb0 con 0x7f85e80a6370 2023-03-03T08:43:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.384+0000 7f85ddffb700 1 -- 172.21.15.7:0/625044657 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f85e4055950 con 0x7f85e80a6370 2023-03-03T08:43:54.387 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:43:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.386+0000 7f85ef5b5700 1 -- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85cc0682f0 msgr2=0x7f85cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.386+0000 7f85ef5b5700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85cc0682f0 0x7f85cc06a7a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009b40 tx=0x7f85d8009210 comp rx=0 tx=0).stop 2023-03-03T08:43:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.386+0000 7f85ef5b5700 1 -- 172.21.15.7:0/625044657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80a6370 msgr2=0x7f85e80c00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.386+0000 7f85ef5b5700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80a6370 0x7f85e80c00e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f85e400e9b0 tx=0x7f85e400ecc0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.386+0000 7f85ef5b5700 1 -- 172.21.15.7:0/625044657 shutdown_connections 2023-03-03T08:43:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.386+0000 7f85ef5b5700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85e80c5cb0 0x7f85e8151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.387+0000 7f85ef5b5700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85e806a2b0 0x7f85e81ea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.387+0000 7f85ef5b5700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85cc0682f0 0x7f85cc06a7a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.387+0000 7f85ef5b5700 1 --2- 172.21.15.7:0/625044657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85e80a6370 0x7f85e80c00e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.387+0000 7f85ef5b5700 1 -- 172.21.15.7:0/625044657 >> 172.21.15.7:0/625044657 conn(0x7f85e80a9430 msgr2=0x7f85e80c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.387+0000 7f85ef5b5700 1 -- 172.21.15.7:0/625044657 shutdown_connections 2023-03-03T08:43:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.387+0000 7f85ef5b5700 1 -- 172.21.15.7:0/625044657 wait complete. 2023-03-03T08:43:54.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-03-03T08:43:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.526+0000 7f082c67f700 1 Processor -- start 2023-03-03T08:43:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.526+0000 7f082c67f700 1 -- start start 2023-03-03T08:43:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.527+0000 7f082c67f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 0x7f0824071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.527+0000 7f082c67f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0824151c00 0x7f08240722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.527+0000 7f082c67f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0824070360 con 0x7f082414b460 2023-03-03T08:43:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.527+0000 7f082c67f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08241521a0 con 0x7f0824151c00 2023-03-03T08:43:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.527+0000 7f082c67f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082406a5e0 con 0x7f0824071990 2023-03-03T08:43:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.527+0000 7f082a41b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 0x7f0824071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.527+0000 7f082ac1c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0824151c00 0x7f08240722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.527+0000 7f082a41b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 0x7f0824071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59976/0 (socket says 172.21.15.7:59976) 2023-03-03T08:43:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f082a41b700 1 -- 172.21.15.7:0/1300039591 learned_addr learned my addr 172.21.15.7:0/1300039591 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f0829c1a700 1 --1- 172.21.15.7:0/1300039591 >> v1:172.21.15.143:6789/0 conn(0x7f082414b460 0x7f082406aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49404/0 (socket says 172.21.15.7:49404) 2023-03-03T08:43:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f082a41b700 1 -- 172.21.15.7:0/1300039591 >> v2:172.21.15.143:3300/0 conn(0x7f0824151c00 msgr2=0x7f08240722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f082a41b700 1 --2- 172.21.15.7:0/1300039591 >> v2:172.21.15.143:3300/0 conn(0x7f0824151c00 0x7f08240722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f082a41b700 1 -- 172.21.15.7:0/1300039591 >> v1:172.21.15.143:6789/0 conn(0x7f082414b460 legacy=0x7f082406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f082a41b700 1 -- 172.21.15.7:0/1300039591 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08240c38b0 con 0x7f0824071990 2023-03-03T08:43:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f082ac1c700 1 --2- 172.21.15.7:0/1300039591 >> v2:172.21.15.143:3300/0 conn(0x7f0824151c00 0x7f08240722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f082a41b700 1 --2- 172.21.15.7:0/1300039591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 0x7f0824071d60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f08140098f0 tx=0x7f0814009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80081e8329f829bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f0829419700 1 -- 172.21.15.7:0/1300039591 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0814011830 con 0x7f0824071990 2023-03-03T08:43:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f0829419700 1 -- 172.21.15.7:0/1300039591 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0814011990 con 0x7f0824071990 2023-03-03T08:43:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.528+0000 7f0829419700 1 -- 172.21.15.7:0/1300039591 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0814011d20 con 0x7f0824071990 2023-03-03T08:43:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.529+0000 7f082c67f700 1 -- 172.21.15.7:0/1300039591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 msgr2=0x7f0824071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.529+0000 7f082c67f700 1 --2- 172.21.15.7:0/1300039591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 0x7f0824071d60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f08140098f0 tx=0x7f0814009c00 comp rx=0 tx=0).stop 2023-03-03T08:43:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.529+0000 7f082c67f700 1 -- 172.21.15.7:0/1300039591 shutdown_connections 2023-03-03T08:43:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.529+0000 7f082c67f700 1 --2- 172.21.15.7:0/1300039591 >> v2:172.21.15.143:3300/0 conn(0x7f0824151c00 0x7f08240722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.529+0000 7f082c67f700 1 --2- 172.21.15.7:0/1300039591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 0x7f0824071d60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.529+0000 7f082c67f700 1 -- 172.21.15.7:0/1300039591 >> 172.21.15.7:0/1300039591 conn(0x7f08240a94d0 msgr2=0x7f08240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.529+0000 7f082c67f700 1 -- 172.21.15.7:0/1300039591 shutdown_connections 2023-03-03T08:43:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.529+0000 7f082c67f700 1 -- 172.21.15.7:0/1300039591 wait complete. 2023-03-03T08:43:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.530+0000 7f082c67f700 1 Processor -- start 2023-03-03T08:43:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.530+0000 7f082c67f700 1 -- start start 2023-03-03T08:43:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.531+0000 7f082c67f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 0x7f0824148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.531+0000 7f082c67f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0824151c00 0x7f0824143cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.531+0000 7f082c67f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0824154bc0 0x7f0824144210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.531+0000 7f082c67f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0824153090 con 0x7f0824151c00 2023-03-03T08:43:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.531+0000 7f082c67f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0824144750 con 0x7f0824154bc0 2023-03-03T08:43:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.531+0000 7f082c67f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08241552c0 con 0x7f0824071990 2023-03-03T08:43:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.531+0000 7f0829c1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0824151c00 0x7f0824143cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f0829c1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0824151c00 0x7f0824143cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51914/0 (socket says 172.21.15.7:51914) 2023-03-03T08:43:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f0829c1a700 1 -- 172.21.15.7:0/1912869203 learned_addr learned my addr 172.21.15.7:0/1912869203 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f082a41b700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 0x7f0824148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f082ac1c700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0824154bc0 0x7f0824144210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f0829c1a700 1 -- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 msgr2=0x7f0824148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f0829c1a700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 0x7f0824148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f0829c1a700 1 -- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0824154bc0 msgr2=0x7f0824144210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f0829c1a700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0824154bc0 0x7f0824144210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f0829c1a700 1 -- 172.21.15.7:0/1912869203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c009a70 con 0x7f0824151c00 2023-03-03T08:43:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f082ac1c700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0824154bc0 0x7f0824144210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f0829c1a700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0824151c00 0x7f0824143cd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f082000c550 tx=0x7f082000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f081affd700 1 -- 172.21.15.7:0/1912869203 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0820005cb0 con 0x7f0824151c00 2023-03-03T08:43:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f081affd700 1 -- 172.21.15.7:0/1912869203 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0820005e10 con 0x7f0824151c00 2023-03-03T08:43:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f082c67f700 1 -- 172.21.15.7:0/1912869203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0814009550 con 0x7f0824151c00 2023-03-03T08:43:54.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f081affd700 1 -- 172.21.15.7:0/1912869203 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08200072d0 con 0x7f0824151c00 2023-03-03T08:43:54.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.532+0000 7f082c67f700 1 -- 172.21.15.7:0/1912869203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08240c3b50 con 0x7f0824151c00 2023-03-03T08:43:54.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.533+0000 7f082c67f700 1 -- 172.21.15.7:0/1912869203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082406a2b0 con 0x7f0824151c00 2023-03-03T08:43:54.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.534+0000 7f081affd700 1 -- 172.21.15.7:0/1912869203 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0820019030 con 0x7f0824151c00 2023-03-03T08:43:54.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.534+0000 7f081affd700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08080682f0 0x7f080806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.534+0000 7f081affd700 1 -- 172.21.15.7:0/1912869203 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f0820085140 con 0x7f0824151c00 2023-03-03T08:43:54.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.534+0000 7f082a41b700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08080682f0 0x7f080806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.535+0000 7f082a41b700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08080682f0 0x7f080806a7a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0814009c40 tx=0x7f0814008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:54.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.538+0000 7f081affd700 1 -- 172.21.15.7:0/1912869203 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f082004b5e0 con 0x7f0824151c00 2023-03-03T08:43:54.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.731+0000 7f082c67f700 1 -- 172.21.15.7:0/1912869203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f082406b180 con 0x7f0824151c00 2023-03-03T08:43:54.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.731+0000 7f081affd700 1 -- 172.21.15.7:0/1912869203 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7f08200170a0 con 0x7f0824151c00 2023-03-03T08:43:54.734 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:43:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 -- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08080682f0 msgr2=0x7f080806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08080682f0 0x7f080806a7a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0814009c40 tx=0x7f0814008030 comp rx=0 tx=0).stop 2023-03-03T08:43:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 -- 172.21.15.7:0/1912869203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0824151c00 msgr2=0x7f0824143cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0824151c00 0x7f0824143cd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f082000c550 tx=0x7f082000c860 comp rx=0 tx=0).stop 2023-03-03T08:43:54.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 -- 172.21.15.7:0/1912869203 shutdown_connections 2023-03-03T08:43:54.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0824154bc0 0x7f0824144210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0824071990 0x7f0824148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08080682f0 0x7f080806a7a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 --2- 172.21.15.7:0/1912869203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0824151c00 0x7f0824143cd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 -- 172.21.15.7:0/1912869203 >> 172.21.15.7:0/1912869203 conn(0x7f08240a94d0 msgr2=0x7f0824145760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:54.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 -- 172.21.15.7:0/1912869203 shutdown_connections 2023-03-03T08:43:54.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.733+0000 7f082c67f700 1 -- 172.21.15.7:0/1912869203 wait complete. 2023-03-03T08:43:54.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-03-03T08:43:54.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.872+0000 7f7c64286700 1 Processor -- start 2023-03-03T08:43:54.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.872+0000 7f7c64286700 1 -- start start 2023-03-03T08:43:54.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.873+0000 7f7c64286700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c5c06d870 0x7f7c5c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.873+0000 7f7c64286700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c5c0c1da0 0x7f7c5c0c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.873+0000 7f7c64286700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5c0c3620 con 0x7f7c5c0c3e30 2023-03-03T08:43:54.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.873+0000 7f7c64286700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5c0c2af0 con 0x7f7c5c06d870 2023-03-03T08:43:54.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.873+0000 7f7c64286700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c1545e0 con 0x7f7c5c0c1da0 2023-03-03T08:43:54.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c62022700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c5c06d870 0x7f7c5c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c62823700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7c5c0c3e30 0x7f7c5c157ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45902/0 (socket says 172.21.15.7:45902) 2023-03-03T08:43:54.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c62022700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c5c06d870 0x7f7c5c1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51918/0 (socket says 172.21.15.7:51918) 2023-03-03T08:43:54.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c62823700 1 -- 172.21.15.7:0/671091372 learned_addr learned my addr 172.21.15.7:0/671091372 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:54.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c61821700 1 --2- 172.21.15.7:0/671091372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c5c0c1da0 0x7f7c5c0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c62022700 1 -- 172.21.15.7:0/671091372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c5c0c1da0 msgr2=0x7f7c5c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:54.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c62022700 1 --2- 172.21.15.7:0/671091372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c5c0c1da0 0x7f7c5c0c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c62022700 1 -- 172.21.15.7:0/671091372 >> v1:172.21.15.7:6789/0 conn(0x7f7c5c0c3e30 legacy=0x7f7c5c157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c62022700 1 -- 172.21.15.7:0/671091372 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c1548c0 con 0x7f7c5c06d870 2023-03-03T08:43:54.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c61020700 1 -- 172.21.15.7:0/671091372 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524028747 0 0) 0x7f7c5c0c3620 con 0x7f7c5c0c3e30 2023-03-03T08:43:54.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c62022700 1 --2- 172.21.15.7:0/671091372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 0x7f7c5c1550c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009840 tx=0x7f7c50009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f43361f2e2087cc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:54.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c61020700 1 -- 172.21.15.7:0/671091372 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c50011830 con 0x7f7c5c06d870 2023-03-03T08:43:54.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c61020700 1 -- 172.21.15.7:0/671091372 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c50011990 con 0x7f7c5c06d870 2023-03-03T08:43:54.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.874+0000 7f7c61020700 1 -- 172.21.15.7:0/671091372 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c50011d20 con 0x7f7c5c06d870 2023-03-03T08:43:54.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.875+0000 7f7c64286700 1 -- 172.21.15.7:0/671091372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 msgr2=0x7f7c5c1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.875+0000 7f7c64286700 1 --2- 172.21.15.7:0/671091372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 0x7f7c5c1550c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009840 tx=0x7f7c50009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:54.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.875+0000 7f7c64286700 1 -- 172.21.15.7:0/671091372 shutdown_connections 2023-03-03T08:43:54.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.875+0000 7f7c64286700 1 --2- 172.21.15.7:0/671091372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 0x7f7c5c1550c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.875+0000 7f7c64286700 1 --2- 172.21.15.7:0/671091372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c5c0c1da0 0x7f7c5c0c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.875+0000 7f7c64286700 1 -- 172.21.15.7:0/671091372 >> 172.21.15.7:0/671091372 conn(0x7f7c5c06c110 msgr2=0x7f7c5c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:54.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.875+0000 7f7c64286700 1 -- 172.21.15.7:0/671091372 shutdown_connections 2023-03-03T08:43:54.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.875+0000 7f7c64286700 1 -- 172.21.15.7:0/671091372 wait complete. 2023-03-03T08:43:54.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.876+0000 7f7c64286700 1 Processor -- start 2023-03-03T08:43:54.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.876+0000 7f7c64286700 1 -- start start 2023-03-03T08:43:54.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c64286700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 0x7f7c5c147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c64286700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c5c0c1da0 0x7f7c5c146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c64286700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c5c0c3e30 0x7f7c5c157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c64286700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c071d70 con 0x7f7c5c06d870 2023-03-03T08:43:54.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c64286700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c1ed550 con 0x7f7c5c0c1da0 2023-03-03T08:43:54.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c64286700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c1ed830 con 0x7f7c5c0c3e30 2023-03-03T08:43:54.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c62022700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 0x7f7c5c147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c62022700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 0x7f7c5c147730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51924/0 (socket says 172.21.15.7:51924) 2023-03-03T08:43:54.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c62022700 1 -- 172.21.15.7:0/1827844155 learned_addr learned my addr 172.21.15.7:0/1827844155 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:54.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c62022700 1 -- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c5c0c3e30 msgr2=0x7f7c5c157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c62022700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c5c0c3e30 0x7f7c5c157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c62022700 1 -- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c5c0c1da0 msgr2=0x7f7c5c146840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c62022700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c5c0c1da0 0x7f7c5c146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c62022700 1 -- 172.21.15.7:0/1827844155 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c500094a0 con 0x7f7c5c06d870 2023-03-03T08:43:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c62022700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 0x7f7c5c147730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009fd0 tx=0x7f7c50004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c4effd700 1 -- 172.21.15.7:0/1827844155 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c500038a0 con 0x7f7c5c06d870 2023-03-03T08:43:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c4effd700 1 -- 172.21.15.7:0/1827844155 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5001f070 con 0x7f7c5c06d870 2023-03-03T08:43:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c64286700 1 -- 172.21.15.7:0/1827844155 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5c06dc40 con 0x7f7c5c06d870 2023-03-03T08:43:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c4effd700 1 -- 172.21.15.7:0/1827844155 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c50003c40 con 0x7f7c5c06d870 2023-03-03T08:43:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.877+0000 7f7c64286700 1 -- 172.21.15.7:0/1827844155 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5c154ba0 con 0x7f7c5c06d870 2023-03-03T08:43:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.878+0000 7f7c64286700 1 -- 172.21.15.7:0/1827844155 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5c0c38e0 con 0x7f7c5c06d870 2023-03-03T08:43:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.879+0000 7f7c4effd700 1 -- 172.21.15.7:0/1827844155 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7c5000f050 con 0x7f7c5c06d870 2023-03-03T08:43:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.879+0000 7f7c4effd700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c400682f0 0x7f7c4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:54.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.879+0000 7f7c4effd700 1 -- 172.21.15.7:0/1827844155 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f7c50021070 con 0x7f7c5c06d870 2023-03-03T08:43:54.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.879+0000 7f7c61821700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c400682f0 0x7f7c4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:54.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.880+0000 7f7c61821700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c400682f0 0x7f7c4006a7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7c58006b20 tx=0x7f7c58006ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:54.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:54.883+0000 7f7c4effd700 1 -- 172.21.15.7:0/1827844155 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7c50053ee0 con 0x7f7c5c06d870 2023-03-03T08:43:55.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.075+0000 7f7c64286700 1 -- 172.21.15.7:0/1827844155 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c5c074bb0 con 0x7f7c5c06d870 2023-03-03T08:43:55.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.076+0000 7f7c4effd700 1 -- 172.21.15.7:0/1827844155 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f7c50057500 con 0x7f7c5c06d870 2023-03-03T08:43:55.078 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:43:55.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 -- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c400682f0 msgr2=0x7f7c4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c400682f0 0x7f7c4006a7a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7c58006b20 tx=0x7f7c58006ab0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 -- 172.21.15.7:0/1827844155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 msgr2=0x7f7c5c147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 0x7f7c5c147730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009fd0 tx=0x7f7c50004750 comp rx=0 tx=0).stop 2023-03-03T08:43:55.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 -- 172.21.15.7:0/1827844155 shutdown_connections 2023-03-03T08:43:55.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c5c0c3e30 0x7f7c5c157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c400682f0 0x7f7c4006a7a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c5c06d870 0x7f7c5c147730 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 --2- 172.21.15.7:0/1827844155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c5c0c1da0 0x7f7c5c146840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 -- 172.21.15.7:0/1827844155 >> 172.21.15.7:0/1827844155 conn(0x7f7c5c06c110 msgr2=0x7f7c5c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:55.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 -- 172.21.15.7:0/1827844155 shutdown_connections 2023-03-03T08:43:55.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.078+0000 7f7c64286700 1 -- 172.21.15.7:0/1827844155 wait complete. 2023-03-03T08:43:55.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-03-03T08:43:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.217+0000 7faddfece700 1 Processor -- start 2023-03-03T08:43:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.218+0000 7faddfece700 1 -- start start 2023-03-03T08:43:55.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.218+0000 7faddfece700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fadd813a0f0 0x7fadd80c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.218+0000 7faddfece700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd8154560 con 0x7fadd80c1da0 2023-03-03T08:43:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.218+0000 7faddfece700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd8157be0 con 0x7fadd81586c0 2023-03-03T08:43:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.218+0000 7faddfece700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd8154840 con 0x7fadd813a0f0 2023-03-03T08:43:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7fadde46b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fadd80c1da0 0x7fadd815edd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45916/0 (socket says 172.21.15.7:45916) 2023-03-03T08:43:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7fadde46b700 1 -- 172.21.15.7:0/1987353937 learned_addr learned my addr 172.21.15.7:0/1987353937 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7fadddc6a700 1 --2- 172.21.15.7:0/1987353937 >> v2:172.21.15.143:3300/0 conn(0x7fadd813a0f0 0x7fadd80c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7faddcc68700 1 -- 172.21.15.7:0/1987353937 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730956419 0 0) 0x7fadd8154560 con 0x7fadd80c1da0 2023-03-03T08:43:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7faddcc68700 1 -- 172.21.15.7:0/1987353937 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc40035b0 con 0x7fadd80c1da0 2023-03-03T08:43:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7faddcc68700 1 -- 172.21.15.7:0/1987353937 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518492537 0 0) 0x7fadd8157be0 con 0x7fadd81586c0 2023-03-03T08:43:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7faddcc68700 1 -- 172.21.15.7:0/1987353937 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd8154560 con 0x7fadd81586c0 2023-03-03T08:43:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7fadddc6a700 1 -- 172.21.15.7:0/1987353937 >> v1:172.21.15.143:6789/0 conn(0x7fadd81586c0 legacy=0x7fadd806d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7fadddc6a700 1 -- 172.21.15.7:0/1987353937 >> v1:172.21.15.7:6789/0 conn(0x7fadd80c1da0 legacy=0x7fadd815edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7fadddc6a700 1 -- 172.21.15.7:0/1987353937 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd8157ec0 con 0x7fadd813a0f0 2023-03-03T08:43:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7fadddc6a700 1 --2- 172.21.15.7:0/1987353937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd813a0f0 0x7fadd80c07d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009840 tx=0x7fadc8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce8e5ede8fb6ac2e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7faddcc68700 1 -- 172.21.15.7:0/1987353937 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8011830 con 0x7fadd813a0f0 2023-03-03T08:43:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7faddcc68700 1 -- 172.21.15.7:0/1987353937 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc8011990 con 0x7fadd813a0f0 2023-03-03T08:43:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.219+0000 7faddcc68700 1 -- 172.21.15.7:0/1987353937 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc8011d20 con 0x7fadd813a0f0 2023-03-03T08:43:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.220+0000 7faddfece700 1 -- 172.21.15.7:0/1987353937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd813a0f0 msgr2=0x7fadd80c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.220+0000 7faddfece700 1 --2- 172.21.15.7:0/1987353937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd813a0f0 0x7fadd80c07d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009840 tx=0x7fadc8009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.220+0000 7faddfece700 1 -- 172.21.15.7:0/1987353937 shutdown_connections 2023-03-03T08:43:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.220+0000 7faddfece700 1 --2- 172.21.15.7:0/1987353937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd813a0f0 0x7fadd80c07d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.220+0000 7faddfece700 1 -- 172.21.15.7:0/1987353937 >> 172.21.15.7:0/1987353937 conn(0x7fadd806c110 msgr2=0x7fadd806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.220+0000 7faddfece700 1 -- 172.21.15.7:0/1987353937 shutdown_connections 2023-03-03T08:43:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.220+0000 7faddfece700 1 -- 172.21.15.7:0/1987353937 wait complete. 2023-03-03T08:43:55.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.221+0000 7faddfece700 1 Processor -- start 2023-03-03T08:43:55.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.221+0000 7faddfece700 1 -- start start 2023-03-03T08:43:55.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7faddfece700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80c1da0 0x7fadd8071260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7faddfece700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd813a0f0 0x7fadd806a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7faddfece700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd81586c0 0x7fadd81e9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7faddfece700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80ac230 con 0x7fadd80c1da0 2023-03-03T08:43:55.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7faddfece700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd81edcd0 con 0x7fadd813a0f0 2023-03-03T08:43:55.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7faddfece700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd8071ee0 con 0x7fadd81586c0 2023-03-03T08:43:55.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7fadddc6a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80c1da0 0x7fadd8071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7fadddc6a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80c1da0 0x7fadd8071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51938/0 (socket says 172.21.15.7:51938) 2023-03-03T08:43:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7fadddc6a700 1 -- 172.21.15.7:0/1273920610 learned_addr learned my addr 172.21.15.7:0/1273920610 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7fadde46b700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd81586c0 0x7fadd81e9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7faddd469700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd813a0f0 0x7fadd806a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7fadddc6a700 1 -- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd81586c0 msgr2=0x7fadd81e9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7fadddc6a700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd81586c0 0x7fadd81e9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7fadddc6a700 1 -- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd813a0f0 msgr2=0x7fadd806a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7fadddc6a700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd813a0f0 0x7fadd806a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7fadddc6a700 1 -- 172.21.15.7:0/1273920610 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc80094a0 con 0x7fadd80c1da0 2023-03-03T08:43:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.222+0000 7fadddc6a700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80c1da0 0x7fadd8071260 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fadc8000c00 tx=0x7fadc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.223+0000 7fadce7fc700 1 -- 172.21.15.7:0/1273920610 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc801a6e0 con 0x7fadd80c1da0 2023-03-03T08:43:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.223+0000 7fadce7fc700 1 -- 172.21.15.7:0/1273920610 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc801a840 con 0x7fadd80c1da0 2023-03-03T08:43:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.223+0000 7faddfece700 1 -- 172.21.15.7:0/1273920610 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd806e1a0 con 0x7fadd80c1da0 2023-03-03T08:43:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.223+0000 7fadce7fc700 1 -- 172.21.15.7:0/1273920610 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc801abd0 con 0x7fadd80c1da0 2023-03-03T08:43:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.223+0000 7faddfece700 1 -- 172.21.15.7:0/1273920610 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd81581a0 con 0x7fadd80c1da0 2023-03-03T08:43:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.224+0000 7faddfece700 1 -- 172.21.15.7:0/1273920610 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd806e330 con 0x7fadd80c1da0 2023-03-03T08:43:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.224+0000 7fadce7fc700 1 -- 172.21.15.7:0/1273920610 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fadc802b030 con 0x7fadd80c1da0 2023-03-03T08:43:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.225+0000 7fadce7fc700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadbc0682f0 0x7fadbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.225+0000 7fadce7fc700 1 -- 172.21.15.7:0/1273920610 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fadc8086f10 con 0x7fadd80c1da0 2023-03-03T08:43:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.225+0000 7faddd469700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadbc0682f0 0x7fadbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.225+0000 7faddd469700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadbc0682f0 0x7fadbc06a7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fadd40011e0 tx=0x7fadd4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:55.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.229+0000 7fadce7fc700 1 -- 172.21.15.7:0/1273920610 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fadc8053440 con 0x7fadd80c1da0 2023-03-03T08:43:55.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.421+0000 7faddfece700 1 -- 172.21.15.7:0/1273920610 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7fadd8070ea0 con 0x7fadd80c1da0 2023-03-03T08:43:55.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.421+0000 7fadce7fc700 1 -- 172.21.15.7:0/1273920610 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7fadc80290a0 con 0x7fadd80c1da0 2023-03-03T08:43:55.425 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:43:55.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 -- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadbc0682f0 msgr2=0x7fadbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadbc0682f0 0x7fadbc06a7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fadd40011e0 tx=0x7fadd4009040 comp rx=0 tx=0).stop 2023-03-03T08:43:55.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 -- 172.21.15.7:0/1273920610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80c1da0 msgr2=0x7fadd8071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80c1da0 0x7fadd8071260 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fadc8000c00 tx=0x7fadc8004750 comp rx=0 tx=0).stop 2023-03-03T08:43:55.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 -- 172.21.15.7:0/1273920610 shutdown_connections 2023-03-03T08:43:55.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadd81586c0 0x7fadd81e9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadbc0682f0 0x7fadbc06a7a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadd80c1da0 0x7fadd8071260 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 --2- 172.21.15.7:0/1273920610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadd813a0f0 0x7fadd806a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 -- 172.21.15.7:0/1273920610 >> 172.21.15.7:0/1273920610 conn(0x7fadd806c110 msgr2=0x7fadd8152ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 -- 172.21.15.7:0/1273920610 shutdown_connections 2023-03-03T08:43:55.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.423+0000 7faddfece700 1 -- 172.21.15.7:0/1273920610 wait complete. 2023-03-03T08:43:55.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:43:55.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.560+0000 7f258653d700 1 Processor -- start 2023-03-03T08:43:55.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.560+0000 7f258653d700 1 -- start start 2023-03-03T08:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f258653d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f258006d870 0x7f25801550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f258653d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f25800c1da0 0x7f25800c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f258653d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25800c3620 con 0x7f25800c3e30 2023-03-03T08:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f258653d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25800c2af0 con 0x7f258006d870 2023-03-03T08:43:55.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f258653d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25801545e0 con 0x7f25800c1da0 2023-03-03T08:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f257ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f258006d870 0x7f25801550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f257ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f258006d870 0x7f25801550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51946/0 (socket says 172.21.15.7:51946) 2023-03-03T08:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f2584ada700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f25800c3e30 0x7f2580157ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45930/0 (socket says 172.21.15.7:45930) 2023-03-03T08:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f257ffff700 1 -- 172.21.15.7:0/1713658559 learned_addr learned my addr 172.21.15.7:0/1713658559 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f257ffff700 1 -- 172.21.15.7:0/1713658559 >> v2:172.21.15.143:3300/0 conn(0x7f25800c1da0 msgr2=0x7f25800c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.561+0000 7f257f7fe700 1 --2- 172.21.15.7:0/1713658559 >> v2:172.21.15.143:3300/0 conn(0x7f25800c1da0 0x7f25800c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f257ffff700 1 --2- 172.21.15.7:0/1713658559 >> v2:172.21.15.143:3300/0 conn(0x7f25800c1da0 0x7f25800c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f257effd700 1 -- 172.21.15.7:0/1713658559 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226511938 0 0) 0x7f25800c3620 con 0x7f25800c3e30 2023-03-03T08:43:55.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f257ffff700 1 -- 172.21.15.7:0/1713658559 >> v1:172.21.15.7:6789/0 conn(0x7f25800c3e30 legacy=0x7f2580157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f257ffff700 1 -- 172.21.15.7:0/1713658559 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25801548c0 con 0x7f258006d870 2023-03-03T08:43:55.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f257f7fe700 1 --2- 172.21.15.7:0/1713658559 >> v2:172.21.15.143:3300/0 conn(0x7f25800c1da0 0x7f25800c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:55.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f257ffff700 1 --2- 172.21.15.7:0/1713658559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 0x7f25801550c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f25800c33a0 tx=0x7f2570009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b906a215e0f36b4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:55.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f257effd700 1 -- 172.21.15.7:0/1713658559 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257000b870 con 0x7f258006d870 2023-03-03T08:43:55.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f257effd700 1 -- 172.21.15.7:0/1713658559 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257000b9d0 con 0x7f258006d870 2023-03-03T08:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f257effd700 1 -- 172.21.15.7:0/1713658559 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257000bd60 con 0x7f258006d870 2023-03-03T08:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f258653d700 1 -- 172.21.15.7:0/1713658559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 msgr2=0x7f25801550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f258653d700 1 --2- 172.21.15.7:0/1713658559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 0x7f25801550c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f25800c33a0 tx=0x7f2570009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f258653d700 1 -- 172.21.15.7:0/1713658559 shutdown_connections 2023-03-03T08:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f258653d700 1 --2- 172.21.15.7:0/1713658559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 0x7f25801550c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f258653d700 1 --2- 172.21.15.7:0/1713658559 >> v2:172.21.15.143:3300/0 conn(0x7f25800c1da0 0x7f25800c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f258653d700 1 -- 172.21.15.7:0/1713658559 >> 172.21.15.7:0/1713658559 conn(0x7f258006c110 msgr2=0x7f258006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:55.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f258653d700 1 -- 172.21.15.7:0/1713658559 shutdown_connections 2023-03-03T08:43:55.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.562+0000 7f258653d700 1 -- 172.21.15.7:0/1713658559 wait complete. 2023-03-03T08:43:55.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.564+0000 7f258653d700 1 Processor -- start 2023-03-03T08:43:55.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.564+0000 7f258653d700 1 -- start start 2023-03-03T08:43:55.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f258653d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 0x7f258014dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f258653d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25800c1da0 0x7f25801ece90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f258653d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25800c3e30 0x7f2580146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f258653d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258014bb30 con 0x7f258006d870 2023-03-03T08:43:55.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f258653d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25801ed980 con 0x7f25800c1da0 2023-03-03T08:43:55.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f258653d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2580155df0 con 0x7f25800c3e30 2023-03-03T08:43:55.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f257ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 0x7f258014dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f257ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 0x7f258014dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51952/0 (socket says 172.21.15.7:51952) 2023-03-03T08:43:55.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f257ffff700 1 -- 172.21.15.7:0/630958212 learned_addr learned my addr 172.21.15.7:0/630958212 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:55.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f257f7fe700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25800c1da0 0x7f25801ece90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f2584ada700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25800c3e30 0x7f2580146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f257ffff700 1 -- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25800c3e30 msgr2=0x7f2580146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f257ffff700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25800c3e30 0x7f2580146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f257ffff700 1 -- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25800c1da0 msgr2=0x7f25801ece90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f257ffff700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25800c1da0 0x7f25801ece90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f257ffff700 1 -- 172.21.15.7:0/630958212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2570009550 con 0x7f258006d870 2023-03-03T08:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f257f7fe700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25800c1da0 0x7f25801ece90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.565+0000 7f2584ada700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25800c3e30 0x7f2580146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:55.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.566+0000 7f257ffff700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 0x7f258014dc80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2570009c10 tx=0x7f2570004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.566+0000 7f257cff9700 1 -- 172.21.15.7:0/630958212 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570004310 con 0x7f258006d870 2023-03-03T08:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.566+0000 7f257cff9700 1 -- 172.21.15.7:0/630958212 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257001f070 con 0x7f258006d870 2023-03-03T08:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.566+0000 7f257cff9700 1 -- 172.21.15.7:0/630958212 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25700036a0 con 0x7f258006d870 2023-03-03T08:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.566+0000 7f258653d700 1 -- 172.21.15.7:0/630958212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258006dc40 con 0x7f258006d870 2023-03-03T08:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.566+0000 7f258653d700 1 -- 172.21.15.7:0/630958212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2580154ba0 con 0x7f258006d870 2023-03-03T08:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.567+0000 7f258653d700 1 -- 172.21.15.7:0/630958212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25800c38e0 con 0x7f258006d870 2023-03-03T08:43:55.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.567+0000 7f257cff9700 1 -- 172.21.15.7:0/630958212 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f257002a020 con 0x7f258006d870 2023-03-03T08:43:55.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.568+0000 7f257cff9700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25600682f0 0x7f256006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:55.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.568+0000 7f257cff9700 1 -- 172.21.15.7:0/630958212 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f2570087710 con 0x7f258006d870 2023-03-03T08:43:55.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.568+0000 7f257f7fe700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25600682f0 0x7f256006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:55.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.568+0000 7f257f7fe700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25600682f0 0x7f256006a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25680089b0 tx=0x7f2568008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:55.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.572+0000 7f257cff9700 1 -- 172.21.15.7:0/630958212 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2570053b50 con 0x7f258006d870 2023-03-03T08:43:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.810+0000 7f258653d700 1 -- 172.21.15.7:0/630958212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2580074bb0 con 0x7f258006d870 2023-03-03T08:43:55.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.810+0000 7f257cff9700 1 -- 172.21.15.7:0/630958212 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f2570057170 con 0x7f258006d870 2023-03-03T08:43:55.813 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:43:55.813 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:43:55.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 -- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25600682f0 msgr2=0x7f256006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25600682f0 0x7f256006a7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25680089b0 tx=0x7f2568008040 comp rx=0 tx=0).stop 2023-03-03T08:43:55.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 -- 172.21.15.7:0/630958212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 msgr2=0x7f258014dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:55.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 0x7f258014dc80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2570009c10 tx=0x7f2570004750 comp rx=0 tx=0).stop 2023-03-03T08:43:55.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 -- 172.21.15.7:0/630958212 shutdown_connections 2023-03-03T08:43:55.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25800c3e30 0x7f2580146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25600682f0 0x7f256006a7a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f258006d870 0x7f258014dc80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 --2- 172.21.15.7:0/630958212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25800c1da0 0x7f25801ece90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:55.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 -- 172.21.15.7:0/630958212 >> 172.21.15.7:0/630958212 conn(0x7f258006c110 msgr2=0x7f2580153e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:55.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.812+0000 7f258653d700 1 -- 172.21.15.7:0/630958212 shutdown_connections 2023-03-03T08:43:55.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:55.813+0000 7f258653d700 1 -- 172.21.15.7:0/630958212 wait complete. 2023-03-03T08:43:55.817 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:43:56.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-03T08:43:34.516962+0000', 'modified': '2023-03-03T08:43:38.528223+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4296': {'gid': 4296, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.7:6835/1083851601', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6833', 'nonce': 1083851601}, {'type': 'v1', 'addr': '172.21.15.7:6835', 'nonce': 1083851601}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'a', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-03T08:43:56.230 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-03T08:43:56.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-03-03T08:43:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.358+0000 7fdceccac700 1 Processor -- start 2023-03-03T08:43:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.359+0000 7fdceccac700 1 -- start start 2023-03-03T08:43:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.359+0000 7fdceccac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8155390 0x7fdce80c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.359+0000 7fdceccac700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdce806e020 0x7fdce8071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.359+0000 7fdceccac700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdce8072520 0x7fdce8151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.359+0000 7fdceccac700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce8070b90 con 0x7fdce806e020 2023-03-03T08:43:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.359+0000 7fdceccac700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce8151320 con 0x7fdce8072520 2023-03-03T08:43:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.359+0000 7fdceccac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce80c4780 con 0x7fdce8155390 2023-03-03T08:43:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce5d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdce806e020 0x7fdce8071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce5d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdce806e020 0x7fdce8071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51960/0 (socket says 172.21.15.7:51960) 2023-03-03T08:43:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce5d9b700 1 -- 172.21.15.7:0/2903056340 learned_addr learned my addr 172.21.15.7:0/2903056340 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce659c700 1 --2- 172.21.15.7:0/2903056340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8155390 0x7fdce80c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce6d9d700 1 --2- 172.21.15.7:0/2903056340 >> v2:172.21.15.143:3300/0 conn(0x7fdce8072520 0x7fdce8151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce5d9b700 1 -- 172.21.15.7:0/2903056340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8155390 msgr2=0x7fdce80c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2903056340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8155390 0x7fdce80c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce5d9b700 1 -- 172.21.15.7:0/2903056340 >> v2:172.21.15.143:3300/0 conn(0x7fdce8072520 msgr2=0x7fdce8151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2903056340 >> v2:172.21.15.143:3300/0 conn(0x7fdce8072520 0x7fdce8151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce5d9b700 1 -- 172.21.15.7:0/2903056340 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce8154910 con 0x7fdce806e020 2023-03-03T08:43:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce6d9d700 1 --2- 172.21.15.7:0/2903056340 >> v2:172.21.15.143:3300/0 conn(0x7fdce8072520 0x7fdce8151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2903056340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce806e020 0x7fdce8071fe0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdce80c3d80 tx=0x7fdcdc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402df94c93dd7e0b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce559a700 1 -- 172.21.15.7:0/2903056340 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc00b870 con 0x7fdce806e020 2023-03-03T08:43:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce559a700 1 -- 172.21.15.7:0/2903056340 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcdc00b9d0 con 0x7fdce806e020 2023-03-03T08:43:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.360+0000 7fdce559a700 1 -- 172.21.15.7:0/2903056340 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc00bd60 con 0x7fdce806e020 2023-03-03T08:43:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.361+0000 7fdceccac700 1 -- 172.21.15.7:0/2903056340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce806e020 msgr2=0x7fdce8071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.361+0000 7fdceccac700 1 --2- 172.21.15.7:0/2903056340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce806e020 0x7fdce8071fe0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdce80c3d80 tx=0x7fdcdc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.361+0000 7fdceccac700 1 -- 172.21.15.7:0/2903056340 shutdown_connections 2023-03-03T08:43:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.361+0000 7fdceccac700 1 --2- 172.21.15.7:0/2903056340 >> v2:172.21.15.143:3300/0 conn(0x7fdce8072520 0x7fdce8151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.361+0000 7fdceccac700 1 --2- 172.21.15.7:0/2903056340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8155390 0x7fdce80c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.361+0000 7fdceccac700 1 --2- 172.21.15.7:0/2903056340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce806e020 0x7fdce8071fe0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.361+0000 7fdceccac700 1 -- 172.21.15.7:0/2903056340 >> 172.21.15.7:0/2903056340 conn(0x7fdce806c110 msgr2=0x7fdce806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.361+0000 7fdceccac700 1 -- 172.21.15.7:0/2903056340 shutdown_connections 2023-03-03T08:43:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.361+0000 7fdceccac700 1 -- 172.21.15.7:0/2903056340 wait complete. 2023-03-03T08:43:56.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.362+0000 7fdceccac700 1 Processor -- start 2023-03-03T08:43:56.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.362+0000 7fdceccac700 1 -- start start 2023-03-03T08:43:56.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdceccac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce806e020 0x7fdce80c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdceccac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdce8072520 0x7fdce80aa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdceccac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8155390 0x7fdce80aaa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdceccac700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce8154000 con 0x7fdce8155390 2023-03-03T08:43:56.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdceccac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce8153100 con 0x7fdce8072520 2023-03-03T08:43:56.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdceccac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce8153c90 con 0x7fdce806e020 2023-03-03T08:43:56.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdce6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8155390 0x7fdce80aaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdce6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8155390 0x7fdce80aaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51970/0 (socket says 172.21.15.7:51970) 2023-03-03T08:43:56.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdce6d9d700 1 -- 172.21.15.7:0/3489281444 learned_addr learned my addr 172.21.15.7:0/3489281444 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:56.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdce659c700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce806e020 0x7fdce80c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.363+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdce8072520 0x7fdce80aa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdce6d9d700 1 -- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce806e020 msgr2=0x7fdce80c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdce6d9d700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce806e020 0x7fdce80c64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdce6d9d700 1 -- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdce8072520 msgr2=0x7fdce80aa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdce6d9d700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdce8072520 0x7fdce80aa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdce6d9d700 1 -- 172.21.15.7:0/3489281444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc009550 con 0x7fdce8155390 2023-03-03T08:43:56.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdce6d9d700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8155390 0x7fdce80aaa20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdce000cf20 tx=0x7fdce000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:56.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdcd6ffd700 1 -- 172.21.15.7:0/3489281444 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce000adc0 con 0x7fdce8155390 2023-03-03T08:43:56.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdcd6ffd700 1 -- 172.21.15.7:0/3489281444 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce0015440 con 0x7fdce8155390 2023-03-03T08:43:56.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdceccac700 1 -- 172.21.15.7:0/3489281444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce8070540 con 0x7fdce8155390 2023-03-03T08:43:56.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdcd6ffd700 1 -- 172.21.15.7:0/3489281444 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce00157f0 con 0x7fdce8155390 2023-03-03T08:43:56.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.364+0000 7fdceccac700 1 -- 172.21.15.7:0/3489281444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce80702a0 con 0x7fdce8155390 2023-03-03T08:43:56.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.365+0000 7fdcd6ffd700 1 -- 172.21.15.7:0/3489281444 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdce001c030 con 0x7fdce8155390 2023-03-03T08:43:56.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.366+0000 7fdcd6ffd700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdcc80682f0 0x7fdcc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.366+0000 7fdcd6ffd700 1 -- 172.21.15.7:0/3489281444 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdce0086430 con 0x7fdce8155390 2023-03-03T08:43:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.366+0000 7fdce659c700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdcc80682f0 0x7fdcc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.367+0000 7fdce659c700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdcc80682f0 0x7fdcc806a7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800a950 tx=0x7fdcd8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.368+0000 7fdcf8e35b80 1 Processor -- start 2023-03-03T08:43:56.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.368+0000 7fdcf8e35b80 1 -- start start 2023-03-03T08:43:56.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e72c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e72bbf0 0x56380e74e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74ee00 0x56380e76f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380e61abc0 con 0x56380e74e4f0 2023-03-03T08:43:56.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380e689350 con 0x56380e74ee00 2023-03-03T08:43:56.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x56380e682880 con 0x56380e72bbf0 2023-03-03T08:43:56.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e72c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e72c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51976/0 (socket says 172.21.15.7:51976) 2023-03-03T08:43:56.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce659c700 1 -- 172.21.15.7:0/3607028451 learned_addr learned my addr 172.21.15.7:0/3607028451 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:56.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/3607028451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e72bbf0 0x56380e74e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce6d9d700 1 --2- 172.21.15.7:0/3607028451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74ee00 0x56380e76f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce659c700 1 -- 172.21.15.7:0/3607028451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e72bbf0 msgr2=0x56380e74e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce659c700 1 --2- 172.21.15.7:0/3607028451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e72bbf0 0x56380e74e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce659c700 1 -- 172.21.15.7:0/3607028451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74ee00 msgr2=0x56380e76f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce659c700 1 --2- 172.21.15.7:0/3607028451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74ee00 0x56380e76f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce659c700 1 -- 172.21.15.7:0/3607028451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56380e5045e0 con 0x56380e74e4f0 2023-03-03T08:43:56.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdce659c700 1 --2- 172.21.15.7:0/3607028451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e72c230 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800a340 tx=0x7fdcd8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30376fa74e29b1ad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:56.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdcc7fff700 1 -- 172.21.15.7:0/3607028451 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8013660 con 0x56380e74e4f0 2023-03-03T08:43:56.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdcc7fff700 1 -- 172.21.15.7:0/3607028451 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd8018070 con 0x56380e74e4f0 2023-03-03T08:43:56.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.369+0000 7fdcc7fff700 1 -- 172.21.15.7:0/3607028451 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd80139c0 con 0x56380e74e4f0 2023-03-03T08:43:56.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.370+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/3607028451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 msgr2=0x56380e72c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.370+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/3607028451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e72c230 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800a340 tx=0x7fdcd8011000 comp rx=0 tx=0).stop 2023-03-03T08:43:56.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.370+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/3607028451 shutdown_connections 2023-03-03T08:43:56.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.370+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/3607028451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74ee00 0x56380e76f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.370+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/3607028451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e72c230 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.370+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/3607028451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e72bbf0 0x56380e74e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.370+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/3607028451 >> 172.21.15.7:0/3607028451 conn(0x56380e725720 msgr2=0x56380e730950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:56.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.370+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/3607028451 shutdown_connections 2023-03-03T08:43:56.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.370+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/3607028451 wait complete. 2023-03-03T08:43:56.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.371+0000 7fdcf8e35b80 1 -- start start 2023-03-03T08:43:56.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.371+0000 7fdcf8e35b80 1 Processor -- start 2023-03-03T08:43:56.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.372+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e72bbf0 0x56380e77fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.372+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e765650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.372+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74ee00 0x56380e767f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.372+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380e5068d0 con 0x56380e74e4f0 2023-03-03T08:43:56.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.372+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380e780140 con 0x56380e74ee00 2023-03-03T08:43:56.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.372+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x56380e766370 con 0x56380e72bbf0 2023-03-03T08:43:56.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.372+0000 7fdce5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e765650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.372+0000 7fdcf8e35b80 10 client.0 _ll_register_callbacks cb 0x56380e620630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T08:43:56.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.372+0000 7fdce5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e765650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51982/0 (socket says 172.21.15.7:51982) 2023-03-03T08:43:56.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.372+0000 7fdce5d9b700 1 -- 172.21.15.7:0/1755015455 learned_addr learned my addr 172.21.15.7:0/1755015455 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdce659c700 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e72bbf0 0x56380e77fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdce6d9d700 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74ee00 0x56380e767f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdce5d9b700 1 -- 172.21.15.7:0/1755015455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e72bbf0 msgr2=0x56380e77fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e72bbf0 0x56380e77fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdce5d9b700 1 -- 172.21.15.7:0/1755015455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74ee00 msgr2=0x56380e767f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74ee00 0x56380e767f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdce5d9b700 1 -- 172.21.15.7:0/1755015455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd800e6e0 con 0x56380e74e4f0 2023-03-03T08:43:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e765650 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc013000 tx=0x7fdcdc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61c1cb126d239dd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcc57fa700 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-03-03T08:43:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcc57fa700 1 -- 172.21.15.7:0/1755015455 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc021070 con 0x56380e74e4f0 2023-03-03T08:43:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcc57fa700 1 -- 172.21.15.7:0/1755015455 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdcdc004250 con 0x56380e74e4f0 2023-03-03T08:43:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/1755015455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56380e650140 con 0x56380e74e4f0 2023-03-03T08:43:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcc57fa700 1 -- 172.21.15.7:0/1755015455 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcdc00b840 con 0x56380e74e4f0 2023-03-03T08:43:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcf8e35b80 10 client.4746 fetch_fsmap learned FSMap version 14 2023-03-03T08:43:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/1755015455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x56380e4b0d90 con 0x56380e74e4f0 2023-03-03T08:43:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcc57fa700 1 -- 172.21.15.7:0/1755015455 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56380e650140 con 0x56380e74e4f0 2023-03-03T08:43:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcc57fa700 1 -- 172.21.15.7:0/1755015455 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 14) v1 ==== 3652+0+0 (secure 0 0 0) 0x7fdcdc004410 con 0x56380e74e4f0 2023-03-03T08:43:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcf8e35b80 10 client.4746 fetch_fsmap finished waiting for FSMap version 14 2023-03-03T08:43:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.373+0000 7fdcf8e35b80 10 client.4746 resolve_mds: resolved name 'a' to daemon mds.a 2023-03-03T08:43:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.374+0000 7fdcf8e35b80 20 client.4746 populate_metadata read hostname 'smithi007' 2023-03-03T08:43:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.374+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e768990 0x56380e745960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.374+0000 7fdcf8e35b80 4 client.4746 mds_command: new command op to 4296 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-03T08:43:56.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.374+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/1755015455 --> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56380e506f10 con 0x56380e768990 2023-03-03T08:43:56.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.374+0000 7fdce659c700 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e768990 0x56380e745960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.374+0000 7fdcc57fa700 10 client.4746 ms_handle_connect on v2:172.21.15.7:6833/1083851601 2023-03-03T08:43:56.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.374+0000 7fdce659c700 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e768990 0x56380e745960 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43e51feedf9392ae server_cookie=d6efa16240dd7aef in_seq=0 out_seq=0 2023-03-03T08:43:56.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcc57fa700 1 -- 172.21.15.7:0/1755015455 <== mds.0 v2:172.21.15.7:6833/1083851601 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x56380e506f10 con 0x56380e768990 2023-03-03T08:43:56.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcc57fa700 10 client.4746 handle_command_reply: tid=0 2023-03-03T08:43:56.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 client.4746 shutdown 2023-03-03T08:43:56.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/1755015455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 msgr2=0x56380e765650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e765650 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc013000 tx=0x7fdcdc0047c0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/1755015455 shutdown_connections 2023-03-03T08:43:56.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74ee00 0x56380e767f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e72bbf0 0x56380e77fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e768990 0x56380e745960 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/1755015455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e74e4f0 0x56380e765650 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/1755015455 >> 172.21.15.7:0/1755015455 conn(0x56380e725720 msgr2=0x56380e729bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:56.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcc57fa700 0 client.4746 ms_handle_reset on v2:172.21.15.7:6833/1083851601 2023-03-03T08:43:56.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/1755015455 shutdown_connections 2023-03-03T08:43:56.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/1755015455 wait complete. 2023-03-03T08:43:56.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.375+0000 7fdcf8e35b80 20 client.4746 trim_cache size 0 max 16384 2023-03-03T08:43:56.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.393+0000 7fdcf8e35b80 1 Processor -- start 2023-03-03T08:43:56.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.393+0000 7fdcf8e35b80 1 -- start start 2023-03-03T08:43:56.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e76e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74e4f0 0x56380e76f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e74ee00 0x56380e76f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380e72d9d0 con 0x56380e72bbf0 2023-03-03T08:43:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380e74e9f0 con 0x56380e74e4f0 2023-03-03T08:43:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x56380e72ba50 con 0x56380e74ee00 2023-03-03T08:43:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdce659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e76e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdce659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e76e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51988/0 (socket says 172.21.15.7:51988) 2023-03-03T08:43:56.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdce659c700 1 -- 172.21.15.7:0/2288620305 learned_addr learned my addr 172.21.15.7:0/2288620305 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:56.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdce659c700 1 -- 172.21.15.7:0/2288620305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e74ee00 msgr2=0x56380e76f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdce659c700 1 --2- 172.21.15.7:0/2288620305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e74ee00 0x56380e76f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdce659c700 1 -- 172.21.15.7:0/2288620305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74e4f0 msgr2=0x56380e76f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdce659c700 1 --2- 172.21.15.7:0/2288620305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74e4f0 0x56380e76f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdce659c700 1 -- 172.21.15.7:0/2288620305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56380e649140 con 0x56380e72bbf0 2023-03-03T08:43:56.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdce659c700 1 --2- 172.21.15.7:0/2288620305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e76e6d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800e4b0 tx=0x7fdcd8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b482a43b0b09aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:56.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdcd4ff9700 1 -- 172.21.15.7:0/2288620305 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd802e040 con 0x56380e72bbf0 2023-03-03T08:43:56.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdcd4ff9700 1 -- 172.21.15.7:0/2288620305 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd8018040 con 0x56380e72bbf0 2023-03-03T08:43:56.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.394+0000 7fdcd4ff9700 1 -- 172.21.15.7:0/2288620305 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8031550 con 0x56380e72bbf0 2023-03-03T08:43:56.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.395+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2288620305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 msgr2=0x56380e76e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.395+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/2288620305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e76e6d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800e4b0 tx=0x7fdcd8013000 comp rx=0 tx=0).stop 2023-03-03T08:43:56.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.395+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2288620305 shutdown_connections 2023-03-03T08:43:56.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.395+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/2288620305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e74ee00 0x56380e76f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.395+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/2288620305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e76e6d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.395+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/2288620305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74e4f0 0x56380e76f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.395+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2288620305 >> 172.21.15.7:0/2288620305 conn(0x56380e725720 msgr2=0x56380e729b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:56.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.395+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2288620305 shutdown_connections 2023-03-03T08:43:56.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.395+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2288620305 wait complete. 2023-03-03T08:43:56.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.395+0000 7fdcf8e35b80 1 -- start start 2023-03-03T08:43:56.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.396+0000 7fdcf8e35b80 1 Processor -- start 2023-03-03T08:43:56.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.396+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e7d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.396+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74e4f0 0x56380e7671d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.396+0000 7fdcf8e35b80 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e74ee00 0x56380e742480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.396+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380e76f6b0 con 0x56380e72bbf0 2023-03-03T08:43:56.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.396+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380e767be0 con 0x56380e74e4f0 2023-03-03T08:43:56.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.396+0000 7fdcf8e35b80 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x56380e775c90 con 0x56380e74ee00 2023-03-03T08:43:56.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.396+0000 7fdcf8e35b80 10 client.0 _ll_register_callbacks cb 0x7fdcdc00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-03T08:43:56.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.396+0000 7fdce659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e7d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.396+0000 7fdce659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e7d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51994/0 (socket says 172.21.15.7:51994) 2023-03-03T08:43:56.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdce659c700 1 -- 172.21.15.7:0/2927333118 learned_addr learned my addr 172.21.15.7:0/2927333118 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:56.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74e4f0 0x56380e7671d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdce659c700 1 -- 172.21.15.7:0/2927333118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e74ee00 msgr2=0x56380e742480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdce6d9d700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e74ee00 0x56380e742480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdce659c700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e74ee00 0x56380e742480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdce659c700 1 -- 172.21.15.7:0/2927333118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74e4f0 msgr2=0x56380e7671d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdce659c700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74e4f0 0x56380e7671d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdce659c700 1 -- 172.21.15.7:0/2927333118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56380e5045e0 con 0x56380e72bbf0 2023-03-03T08:43:56.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74e4f0 0x56380e7671d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:43:56.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdce659c700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e7d1af0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8010010 tx=0x7fdcd80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b067d6c020eb15a3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:56.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcc57fa700 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2023-03-03T08:43:56.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcc57fa700 1 -- 172.21.15.7:0/2927333118 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8031ad0 con 0x56380e72bbf0 2023-03-03T08:43:56.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcc57fa700 1 -- 172.21.15.7:0/2927333118 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdcd8031c30 con 0x56380e72bbf0 2023-03-03T08:43:56.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2927333118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56380e640160 con 0x56380e72bbf0 2023-03-03T08:43:56.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcc57fa700 1 -- 172.21.15.7:0/2927333118 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcd8031df0 con 0x56380e72bbf0 2023-03-03T08:43:56.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcf8e35b80 10 client.4752 fetch_fsmap learned FSMap version 14 2023-03-03T08:43:56.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2927333118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x56380e4d7200 con 0x56380e72bbf0 2023-03-03T08:43:56.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcc57fa700 1 -- 172.21.15.7:0/2927333118 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56380e640160 con 0x56380e72bbf0 2023-03-03T08:43:56.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcc57fa700 1 -- 172.21.15.7:0/2927333118 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 14) v1 ==== 3652+0+0 (secure 0 0 0) 0x7fdcd8031050 con 0x56380e72bbf0 2023-03-03T08:43:56.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcf8e35b80 10 client.4752 fetch_fsmap finished waiting for FSMap version 14 2023-03-03T08:43:56.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.397+0000 7fdcf8e35b80 10 client.4752 resolve_mds: resolved name 'a' to daemon mds.a 2023-03-03T08:43:56.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdcf8e35b80 20 client.4752 populate_metadata read hostname 'smithi007' 2023-03-03T08:43:56.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e7429c0 0x56380e77c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdcf8e35b80 4 client.4752 mds_command: new command op to 4296 tid=0[{"prefix": "status"}] 2023-03-03T08:43:56.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2927333118 --> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56380e62a170 con 0x56380e7429c0 2023-03-03T08:43:56.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e7429c0 0x56380e77c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdcc57fa700 10 client.4752 ms_handle_connect on v2:172.21.15.7:6833/1083851601 2023-03-03T08:43:56.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e7429c0 0x56380e77c630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ce9fb83375874d4 server_cookie=84f6009d516ed84f in_seq=0 out_seq=0 2023-03-03T08:43:56.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdce5d9b700 1 -- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e7429c0 msgr2=0x56380e77c630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2023-03-03T08:43:56.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdce5d9b700 1 -- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e7429c0 msgr2=0x56380e77c630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-03T08:43:56.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e7429c0 0x56380e77c630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-03T08:43:56.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e7429c0 0x56380e77c630 unknown :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2023-03-03T08:43:56.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e7429c0 0x56380e77c630 unknown :-1 s=BANNER_CONNECTING pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdcc57fa700 10 client.4752 ms_handle_connect on v2:172.21.15.7:6833/1083851601 2023-03-03T08:43:56.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdce5d9b700 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e7429c0 0x56380e77c630 crc :-1 s=READY pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ce9fb83375874d4 server_cookie=84f6009d516ed84f in_seq=0 out_seq=1 2023-03-03T08:43:56.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdcc57fa700 1 -- 172.21.15.7:0/2927333118 <== mds.0 v2:172.21.15.7:6833/1083851601 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56380e62a170 con 0x56380e7429c0 2023-03-03T08:43:56.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdcc57fa700 10 client.4752 handle_command_reply: tid=0 2023-03-03T08:43:56.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.398+0000 7fdcf8e35b80 1 client.4752 shutdown 2023-03-03T08:43:56.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 msgr2=0x56380e7d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e7d1af0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8010010 tx=0x7fdcd80047c0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2927333118 shutdown_connections 2023-03-03T08:43:56.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x56380e74ee00 0x56380e742480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:6833/1083851601,v1:172.21.15.7:6835/1083851601] conn(0x56380e7429c0 0x56380e77c630 crc :-1 s=READY pgs=14 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56380e72bbf0 0x56380e7d1af0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 1 --2- 172.21.15.7:0/2927333118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56380e74e4f0 0x56380e7671d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2927333118 >> 172.21.15.7:0/2927333118 conn(0x56380e725720 msgr2=0x56380e76e970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:56.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcc57fa700 0 client.4752 ms_handle_reset on v2:172.21.15.7:6833/1083851601 2023-03-03T08:43:56.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2927333118 shutdown_connections 2023-03-03T08:43:56.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 1 -- 172.21.15.7:0/2927333118 wait complete. 2023-03-03T08:43:56.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdcf8e35b80 20 client.4752 trim_cache size 0 max 16384 2023-03-03T08:43:56.413 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-03-03T08:43:56.413 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "4366045e-92da-4bb9-8565-4b6c37871975", 2023-03-03T08:43:56.413 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2023-03-03T08:43:56.413 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 4296, 2023-03-03T08:43:56.414 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2023-03-03T08:43:56.414 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2023-03-03T08:43:56.414 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "a", 2023-03-03T08:43:56.414 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 21.881564599000001, 2023-03-03T08:43:56.414 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 9, 2023-03-03T08:43:56.414 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 27, 2023-03-03T08:43:56.414 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 23, 2023-03-03T08:43:56.414 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 27.214639697999999 2023-03-03T08:43:56.415 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-03-03T08:43:56.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 -- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdcc80682f0 msgr2=0x7fdcc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdcc80682f0 0x7fdcc806a7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800a950 tx=0x7fdcd8008040 comp rx=0 tx=0).stop 2023-03-03T08:43:56.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 -- 172.21.15.7:0/3489281444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8155390 msgr2=0x7fdce80aaa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8155390 0x7fdce80aaa20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdce000cf20 tx=0x7fdce000a340 comp rx=0 tx=0).stop 2023-03-03T08:43:56.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 -- 172.21.15.7:0/3489281444 shutdown_connections 2023-03-03T08:43:56.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdcc80682f0 0x7fdcc806a7a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8155390 0x7fdce80aaa20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce806e020 0x7fdce80c64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 --2- 172.21.15.7:0/3489281444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdce8072520 0x7fdce80aa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 -- 172.21.15.7:0/3489281444 >> 172.21.15.7:0/3489281444 conn(0x7fdce806c110 msgr2=0x7fdce80c7370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:56.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 -- 172.21.15.7:0/3489281444 shutdown_connections 2023-03-03T08:43:56.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.399+0000 7fdceccac700 1 -- 172.21.15.7:0/3489281444 wait complete. 2023-03-03T08:43:56.418 INFO:teuthology.run_tasks:Running task kclient... 2023-03-03T08:43:56.449 INFO:tasks.kclient:Mounting kernel clients... 2023-03-03T08:43:56.449 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2023-03-03T08:43:56.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:56.450 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2023-03-03T08:43:56.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:56.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link delete ceph-brx 2023-03-03T08:43:56.537 INFO:teuthology.orchestra.run.smithi007.stderr:Cannot find device "ceph-brx" 2023-03-03T08:43:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:43:56.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:56.540 DEBUG:teuthology.orchestra.run.smithi143:> ip netns list 2023-03-03T08:43:56.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:56.561 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link delete ceph-brx 2023-03-03T08:43:56.634 INFO:teuthology.orchestra.run.smithi143.stderr:Cannot find device "ceph-brx" 2023-03-03T08:43:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-03T08:43:56.636 INFO:tasks.kclient:client.0 config is {'syntax': 'v2', 'cephfs_name': 'a'} 2023-03-03T08:43:56.636 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-03-03T08:43:56.637 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T08:43:56.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T08:43:56.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa9c9e700 1 Processor -- start 2023-03-03T08:43:56.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa9c9e700 1 -- start start 2023-03-03T08:43:56.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa9c9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa40c65c0 0x7effa414de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa9c9e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7effa406a2b0 0x7effa406a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa9c9e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effa40a65b0 con 0x7effa40a9af0 2023-03-03T08:43:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa9c9e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effa4154e00 con 0x7effa406a2b0 2023-03-03T08:43:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa9c9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa414e390 con 0x7effa40c65c0 2023-03-03T08:43:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa40c65c0 0x7effa414de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7effa406a2b0 0x7effa406a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa2ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7effa40a9af0 0x7effa4066840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49504/0 (socket says 172.21.15.7:49504) 2023-03-03T08:43:56.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa2ffd700 1 -- 172.21.15.7:0/1075631957 learned_addr learned my addr 172.21.15.7:0/1075631957 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7effa406a2b0 0x7effa406a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47778/0 (socket says 172.21.15.7:47778) 2023-03-03T08:43:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa27fc700 1 -- 172.21.15.7:0/1075631957 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391051590 0 0) 0x7effa40a65b0 con 0x7effa40a9af0 2023-03-03T08:43:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa27fc700 1 -- 172.21.15.7:0/1075631957 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff8c0035b0 con 0x7effa40a9af0 2023-03-03T08:43:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa37fe700 1 -- 172.21.15.7:0/1075631957 >> v2:172.21.15.143:3300/0 conn(0x7effa406a2b0 msgr2=0x7effa406a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:43:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa37fe700 1 --2- 172.21.15.7:0/1075631957 >> v2:172.21.15.143:3300/0 conn(0x7effa406a2b0 0x7effa406a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa37fe700 1 -- 172.21.15.7:0/1075631957 >> v1:172.21.15.143:6789/0 conn(0x7effa40a9af0 legacy=0x7effa4066840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa37fe700 1 -- 172.21.15.7:0/1075631957 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa41541f0 con 0x7effa40c65c0 2023-03-03T08:43:56.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa3fff700 1 --2- 172.21.15.7:0/1075631957 >> v2:172.21.15.143:3300/0 conn(0x7effa406a2b0 0x7effa406a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:43:56.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.761+0000 7effa37fe700 1 --2- 172.21.15.7:0/1075631957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa40c65c0 0x7effa414de50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7eff94009840 tx=0x7eff94009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1e1ae40e76a6571 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:56.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa27fc700 1 -- 172.21.15.7:0/1075631957 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94011830 con 0x7effa40c65c0 2023-03-03T08:43:56.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa27fc700 1 -- 172.21.15.7:0/1075631957 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff94011990 con 0x7effa40c65c0 2023-03-03T08:43:56.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa27fc700 1 -- 172.21.15.7:0/1075631957 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94011d20 con 0x7effa40c65c0 2023-03-03T08:43:56.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa9c9e700 1 -- 172.21.15.7:0/1075631957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa40c65c0 msgr2=0x7effa414de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa9c9e700 1 --2- 172.21.15.7:0/1075631957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa40c65c0 0x7effa414de50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7eff94009840 tx=0x7eff94009b50 comp rx=0 tx=0).stop 2023-03-03T08:43:56.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa9c9e700 1 -- 172.21.15.7:0/1075631957 shutdown_connections 2023-03-03T08:43:56.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa9c9e700 1 --2- 172.21.15.7:0/1075631957 >> v2:172.21.15.143:3300/0 conn(0x7effa406a2b0 0x7effa406a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa9c9e700 1 --2- 172.21.15.7:0/1075631957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa40c65c0 0x7effa414de50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa9c9e700 1 -- 172.21.15.7:0/1075631957 >> 172.21.15.7:0/1075631957 conn(0x7effa40a9430 msgr2=0x7effa40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:56.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa9c9e700 1 -- 172.21.15.7:0/1075631957 shutdown_connections 2023-03-03T08:43:56.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.762+0000 7effa9c9e700 1 -- 172.21.15.7:0/1075631957 wait complete. 2023-03-03T08:43:56.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.764+0000 7effa9c9e700 1 Processor -- start 2023-03-03T08:43:56.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.764+0000 7effa9c9e700 1 -- start start 2023-03-03T08:43:56.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.764+0000 7effa9c9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa406a2b0 0x7effa406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.764+0000 7effa9c9e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effa40a9af0 0x7effa414c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa9c9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effa40c65c0 0x7effa414c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa9c9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa414eb20 con 0x7effa40c65c0 2023-03-03T08:43:56.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa9c9e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa40c2bb0 con 0x7effa40a9af0 2023-03-03T08:43:56.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa9c9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effa40c2580 con 0x7effa406a2b0 2023-03-03T08:43:56.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effa40c65c0 0x7effa414c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effa40c65c0 0x7effa414c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52012/0 (socket says 172.21.15.7:52012) 2023-03-03T08:43:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa3fff700 1 -- 172.21.15.7:0/4077270855 learned_addr learned my addr 172.21.15.7:0/4077270855 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa37fe700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa406a2b0 0x7effa406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa2ffd700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effa40a9af0 0x7effa414c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa3fff700 1 -- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa406a2b0 msgr2=0x7effa406fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa3fff700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa406a2b0 0x7effa406fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa3fff700 1 -- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effa40a9af0 msgr2=0x7effa414c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa3fff700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effa40a9af0 0x7effa414c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa3fff700 1 -- 172.21.15.7:0/4077270855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c009a70 con 0x7effa40c65c0 2023-03-03T08:43:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa2ffd700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effa40a9af0 0x7effa414c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:43:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa3fff700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effa40c65c0 0x7effa414c770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00efc0 tx=0x7eff9c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7eff8bfff700 1 -- 172.21.15.7:0/4077270855 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c009cd0 con 0x7effa40c65c0 2023-03-03T08:43:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7eff8bfff700 1 -- 172.21.15.7:0/4077270855 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c00f040 con 0x7effa40c65c0 2023-03-03T08:43:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7eff8bfff700 1 -- 172.21.15.7:0/4077270855 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c00cd60 con 0x7effa40c65c0 2023-03-03T08:43:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa9c9e700 1 -- 172.21.15.7:0/4077270855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff940094a0 con 0x7effa40c65c0 2023-03-03T08:43:56.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.765+0000 7effa9c9e700 1 -- 172.21.15.7:0/4077270855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa41547b0 con 0x7effa40c65c0 2023-03-03T08:43:56.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.766+0000 7effa9c9e700 1 -- 172.21.15.7:0/4077270855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa41544d0 con 0x7effa40c65c0 2023-03-03T08:43:56.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.767+0000 7eff8bfff700 1 -- 172.21.15.7:0/4077270855 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7eff9c014480 con 0x7effa40c65c0 2023-03-03T08:43:56.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.767+0000 7eff8bfff700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff900682f0 0x7eff9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:56.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.767+0000 7eff8bfff700 1 -- 172.21.15.7:0/4077270855 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7eff9c086810 con 0x7effa40c65c0 2023-03-03T08:43:56.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.767+0000 7effa37fe700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff900682f0 0x7eff9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:56.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.768+0000 7effa37fe700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff900682f0 0x7eff9006a7a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7eff94009fd0 tx=0x7eff94008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:56.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:56.771+0000 7eff8bfff700 1 -- 172.21.15.7:0/4077270855 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7eff9c052ce0 con 0x7effa40c65c0 2023-03-03T08:43:57.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.010+0000 7effa9c9e700 1 -- 172.21.15.7:0/4077270855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7effa4154a90 con 0x7effa40c65c0 2023-03-03T08:43:57.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.010+0000 7eff8bfff700 1 -- 172.21.15.7:0/4077270855 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+116 (secure 0 0 0) 0x7eff9c056300 con 0x7effa40c65c0 2023-03-03T08:43:57.013 INFO:teuthology.orchestra.run.smithi007.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2023-03-03T08:43:57.013 INFO:teuthology.orchestra.run.smithi007.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2023-03-03T08:43:57.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.012+0000 7effa9c9e700 1 -- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff900682f0 msgr2=0x7eff9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:57.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.012+0000 7effa9c9e700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff900682f0 0x7eff9006a7a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7eff94009fd0 tx=0x7eff94008030 comp rx=0 tx=0).stop 2023-03-03T08:43:57.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.012+0000 7effa9c9e700 1 -- 172.21.15.7:0/4077270855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effa40c65c0 msgr2=0x7effa414c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:57.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.012+0000 7effa9c9e700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effa40c65c0 0x7effa414c770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00efc0 tx=0x7eff9c00c3a0 comp rx=0 tx=0).stop 2023-03-03T08:43:57.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.012+0000 7effa9c9e700 1 -- 172.21.15.7:0/4077270855 shutdown_connections 2023-03-03T08:43:57.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.012+0000 7effa9c9e700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff900682f0 0x7eff9006a7a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:57.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.012+0000 7effa9c9e700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effa40c65c0 0x7effa414c770 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:57.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.012+0000 7effa9c9e700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effa406a2b0 0x7effa406fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:57.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.012+0000 7effa9c9e700 1 --2- 172.21.15.7:0/4077270855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effa40a9af0 0x7effa414c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:57.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.013+0000 7effa9c9e700 1 -- 172.21.15.7:0/4077270855 >> 172.21.15.7:0/4077270855 conn(0x7effa40a9430 msgr2=0x7effa406b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:57.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.013+0000 7effa9c9e700 1 -- 172.21.15.7:0/4077270855 shutdown_connections 2023-03-03T08:43:57.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:57.013+0000 7effa9c9e700 1 -- 172.21.15.7:0/4077270855 wait complete. 2023-03-03T08:43:57.027 INFO:tasks.cephfs.mount:Mounting Ceph FS a; just confirmed its presence on cluster 2023-03-03T08:43:57.027 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T08:43:57.028 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi007.front.sepia.ceph.com 2023-03-03T08:43:57.028 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-03T08:43:57.028 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-03-03T08:43:57.028 INFO:tasks.cephfs.mount:self.cephfs_name = a 2023-03-03T08:43:57.028 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-03-03T08:43:57.029 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T08:43:57.029 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:43:57.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:57.030 DEBUG:teuthology.orchestra.run.smithi007:> ip addr 2023-03-03T08:43:57.047 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T08:43:57.047 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T08:43:57.047 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T08:43:57.047 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:43:57.047 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2023-03-03T08:43:57.047 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:43:57.048 INFO:teuthology.orchestra.run.smithi007.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:43:57.048 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2023-03-03T08:43:57.048 INFO:teuthology.orchestra.run.smithi007.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:43:57.048 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2023-03-03T08:43:57.048 INFO:teuthology.orchestra.run.smithi007.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T08:43:57.048 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2023-03-03T08:43:57.048 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-03-03T08:43:57.048 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 42467sec preferred_lft 42467sec 2023-03-03T08:43:57.049 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2023-03-03T08:43:57.049 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:43:57.049 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:43:57.049 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2023-03-03T08:43:57.049 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-03-03T08:43:57.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:57.050 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-03-03T08:43:57.050 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-03-03T08:43:57.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add name ceph-brx type bridge 2023-03-03T08:43:57.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr flush dev ceph-brx 2023-03-03T08:43:57.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set ceph-brx up 2023-03-03T08:43:57.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-03-03T08:43:57.051 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-03-03T08:43:57.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:57.225 DEBUG:teuthology.orchestra.run.smithi007:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T08:43:57.254 INFO:teuthology.orchestra.run.smithi007.stdout:1 2023-03-03T08:43:57.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:57.255 DEBUG:teuthology.orchestra.run.smithi007:> route 2023-03-03T08:43:57.334 INFO:teuthology.orchestra.run.smithi007.stdout:Kernel IP routing table 2023-03-03T08:43:57.334 INFO:teuthology.orchestra.run.smithi007.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T08:43:57.334 INFO:teuthology.orchestra.run.smithi007.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-03-03T08:43:57.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-03-03T08:43:57.335 INFO:teuthology.orchestra.run.smithi007.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T08:43:57.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:57.335 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-03-03T08:43:57.336 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-03-03T08:43:57.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-03-03T08:43:57.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-03-03T08:43:57.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-03-03T08:43:57.337 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-03-03T08:43:57.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:57.614 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2023-03-03T08:43:57.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:57.632 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list-id 2023-03-03T08:43:57.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:57.688 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-03-03T08:43:57.689 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-03-03T08:43:57.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-03-03T08:43:57.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-03-03T08:43:57.689 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-03-03T08:43:57.783 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-03-03T08:43:57.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:57.784 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-03-03T08:43:57.785 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-03-03T08:43:57.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-03-03T08:43:57.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-03-03T08:43:57.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-03-03T08:43:57.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-03-03T08:43:57.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-03-03T08:43:57.786 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-03-03T08:43:58.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:58.017 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec bash -c ' 2023-03-03T08:43:58.017 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2023-03-03T08:43:58.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set brx.0 up 2023-03-03T08:43:58.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set dev brx.0 master ceph-brx 2023-03-03T08:43:58.018 DEBUG:teuthology.orchestra.run.smithi007:> ') 2023-03-03T08:43:58.127 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:43:58.127 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-03-03T08:43:58.154 INFO:teuthology.orchestra.run.smithi007.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-03-03T08:43:58.154 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:43:58.154 DEBUG:teuthology.orchestra.run.smithi007:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-03-03T08:43:58.211 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.a=/ 2023-03-03T08:43:58.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:58.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.a=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T08:43:58.308 INFO:teuthology.orchestra.run.smithi007.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T08:43:58.309 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-03-03T08:43:58.435 INFO:teuthology.orchestra.run.smithi007.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T08:43:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-03-03T08:43:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: resolved to: "172.21.15.7:3300,172.21.15.143:3300,172.21.15.143:3301" 2023-03-03T08:43:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: trying mount with new device syntax: 0@4366045e-92da-4bb9-8565-4b6c37871975.a=/ 2023-03-03T08:43:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.7:3300/172.21.15.143:3300/172.21.15.143:3301" will pass to kernel 2023-03-03T08:43:58.436 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: v2 mount support check returned 2 2023-03-03T08:43:58.437 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: kernel does not support v2 syntax 2023-03-03T08:43:58.437 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: switching to using v1 address with old syntax 2023-03-03T08:43:58.522 INFO:teuthology.orchestra.run.smithi007.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T08:43:58.522 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-03-03T08:43:58.523 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: resolved to: "172.21.15.7:3300,172.21.15.143:3300,172.21.15.143:3301" 2023-03-03T08:43:58.523 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: trying mount with new device syntax: 0@4366045e-92da-4bb9-8565-4b6c37871975.a=/ 2023-03-03T08:43:58.523 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.7:3300/172.21.15.143:3300/172.21.15.143:3301" will pass to kernel 2023-03-03T08:43:58.523 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: v2 mount support check returned 2 2023-03-03T08:43:58.523 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: kernel does not support v2 syntax 2023-03-03T08:43:58.523 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: switching to using v1 address with old syntax 2023-03-03T08:43:58.523 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: resolved to: "172.21.15.7:6789,172.21.15.143:6789,172.21.15.143:6790" 2023-03-03T08:43:58.524 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.7:6789,172.21.15.143:6789,172.21.15.143:6790:/ 2023-03-03T08:43:58.524 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=a,fsid=4366045e-92da-4bb9-8565-4b6c37871975" will pass to kernel 2023-03-03T08:43:58.525 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-03-03T08:43:58.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:58.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-03-03T08:43:58.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:58.555 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T08:43:58.957 INFO:teuthology.orchestra.run.smithi007.stdout:client4633 2023-03-03T08:43:58.958 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:43:58.958 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:43:58.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:58.976 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T08:43:59.061 INFO:teuthology.orchestra.run.smithi007.stdout:client4633 2023-03-03T08:43:59.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:59.062 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T08:43:59.143 INFO:teuthology.orchestra.run.smithi007.stdout:client4633 2023-03-03T08:43:59.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:59.144 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/4366045e-92da-4bb9-8565-4b6c37871975.client4633/status') 2023-03-03T08:43:59.225 INFO:teuthology.orchestra.run.smithi007.stdout:instance: client.4633 (3)172.21.15.7:0/680265011 2023-03-03T08:43:59.226 INFO:teuthology.orchestra.run.smithi007.stdout:blocklisted: false 2023-03-03T08:43:59.227 INFO:teuthology.orchestra.run.smithi007.stderr:0+1 records in 2023-03-03T08:43:59.227 INFO:teuthology.orchestra.run.smithi007.stderr:0+1 records out 2023-03-03T08:43:59.227 INFO:teuthology.orchestra.run.smithi007.stderr:68 bytes copied, 2.3456e-05 s, 2.9 MB/s 2023-03-03T08:43:59.227 INFO:tasks.kclient:client.1 config is {'syntax': 'v2', 'cephfs_name': 'b'} 2023-03-03T08:43:59.228 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-03-03T08:43:59.228 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-03-03T08:43:59.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-03-03T08:43:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.390+0000 7fe599fa4700 1 Processor -- start 2023-03-03T08:43:59.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.390+0000 7fe599fa4700 1 -- start start 2023-03-03T08:43:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.391+0000 7fe599fa4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 0x7fe59406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.391+0000 7fe599fa4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59406eda0 con 0x7fe5940aad90 2023-03-03T08:43:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.391+0000 7fe599fa4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5941586a0 con 0x7fe59406bfe0 2023-03-03T08:43:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.391+0000 7fe599fa4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59406f080 con 0x7fe59406b920 2023-03-03T08:43:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.391+0000 7fe5937fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe5940aad90 0x7fe59406e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46006/0 (socket says 172.21.15.7:46006) 2023-03-03T08:43:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.391+0000 7fe5937fe700 1 -- 172.21.15.7:0/2638629279 learned_addr learned my addr 172.21.15.7:0/2638629279 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:43:59.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.391+0000 7fe592ffd700 1 --2- 172.21.15.7:0/2638629279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 0x7fe59406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:59.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe5927fc700 1 -- 172.21.15.7:0/2638629279 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973608520 0 0) 0x7fe59406eda0 con 0x7fe5940aad90 2023-03-03T08:43:59.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe5927fc700 1 -- 172.21.15.7:0/2638629279 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5800035b0 con 0x7fe5940aad90 2023-03-03T08:43:59.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe5927fc700 1 -- 172.21.15.7:0/2638629279 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690587216 0 0) 0x7fe5941586a0 con 0x7fe59406bfe0 2023-03-03T08:43:59.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe5927fc700 1 -- 172.21.15.7:0/2638629279 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe59406eda0 con 0x7fe59406bfe0 2023-03-03T08:43:59.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe592ffd700 1 -- 172.21.15.7:0/2638629279 >> v1:172.21.15.143:6789/0 conn(0x7fe59406bfe0 legacy=0x7fe59406a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:59.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe592ffd700 1 -- 172.21.15.7:0/2638629279 >> v1:172.21.15.7:6789/0 conn(0x7fe5940aad90 legacy=0x7fe59406e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe592ffd700 1 -- 172.21.15.7:0/2638629279 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe594158980 con 0x7fe59406b920 2023-03-03T08:43:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe592ffd700 1 --2- 172.21.15.7:0/2638629279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 0x7fe59406a2b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe584009870 tx=0x7fe584009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee88e46a00780b74 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe5927fc700 1 -- 172.21.15.7:0/2638629279 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584011830 con 0x7fe59406b920 2023-03-03T08:43:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe5927fc700 1 -- 172.21.15.7:0/2638629279 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe584011990 con 0x7fe59406b920 2023-03-03T08:43:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.392+0000 7fe5927fc700 1 -- 172.21.15.7:0/2638629279 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe584011d20 con 0x7fe59406b920 2023-03-03T08:43:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.393+0000 7fe599fa4700 1 -- 172.21.15.7:0/2638629279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 msgr2=0x7fe59406a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.393+0000 7fe599fa4700 1 --2- 172.21.15.7:0/2638629279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 0x7fe59406a2b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe584009870 tx=0x7fe584009b80 comp rx=0 tx=0).stop 2023-03-03T08:43:59.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.393+0000 7fe599fa4700 1 -- 172.21.15.7:0/2638629279 shutdown_connections 2023-03-03T08:43:59.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.393+0000 7fe599fa4700 1 --2- 172.21.15.7:0/2638629279 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 0x7fe59406a2b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:59.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.393+0000 7fe599fa4700 1 -- 172.21.15.7:0/2638629279 >> 172.21.15.7:0/2638629279 conn(0x7fe5940736f0 msgr2=0x7fe594073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:59.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.393+0000 7fe599fa4700 1 -- 172.21.15.7:0/2638629279 shutdown_connections 2023-03-03T08:43:59.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.393+0000 7fe599fa4700 1 -- 172.21.15.7:0/2638629279 wait complete. 2023-03-03T08:43:59.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.394+0000 7fe599fa4700 1 Processor -- start 2023-03-03T08:43:59.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.394+0000 7fe599fa4700 1 -- start start 2023-03-03T08:43:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe599fa4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 0x7fe5940ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe599fa4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59406bfe0 0x7fe5940c6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe599fa4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5940aad90 0x7fe594154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe599fa4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5940abe70 con 0x7fe5940aad90 2023-03-03T08:43:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe599fa4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5940c9150 con 0x7fe59406bfe0 2023-03-03T08:43:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe599fa4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5940ad710 con 0x7fe59406b920 2023-03-03T08:43:59.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe593fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5940aad90 0x7fe594154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:59.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe5937fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 0x7fe5940ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:59.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe593fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5940aad90 0x7fe594154ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52032/0 (socket says 172.21.15.7:52032) 2023-03-03T08:43:59.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe592ffd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59406bfe0 0x7fe5940c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:59.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe593fff700 1 -- 172.21.15.7:0/4248605905 learned_addr learned my addr 172.21.15.7:0/4248605905 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:43:59.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe593fff700 1 -- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 msgr2=0x7fe5940ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:59.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe593fff700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 0x7fe5940ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:59.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe593fff700 1 -- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59406bfe0 msgr2=0x7fe5940c6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:59.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe593fff700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59406bfe0 0x7fe5940c6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:59.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.395+0000 7fe593fff700 1 -- 172.21.15.7:0/4248605905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5840094a0 con 0x7fe5940aad90 2023-03-03T08:43:59.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.396+0000 7fe593fff700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5940aad90 0x7fe594154ae0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe58c00cb50 tx=0x7fe58c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:59.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.396+0000 7fe58bfff700 1 -- 172.21.15.7:0/4248605905 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c003c00 con 0x7fe5940aad90 2023-03-03T08:43:59.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.396+0000 7fe58bfff700 1 -- 172.21.15.7:0/4248605905 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58c003d60 con 0x7fe5940aad90 2023-03-03T08:43:59.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.396+0000 7fe599fa4700 1 -- 172.21.15.7:0/4248605905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe594071270 con 0x7fe5940aad90 2023-03-03T08:43:59.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.396+0000 7fe58bfff700 1 -- 172.21.15.7:0/4248605905 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c00aa20 con 0x7fe5940aad90 2023-03-03T08:43:59.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.396+0000 7fe599fa4700 1 -- 172.21.15.7:0/4248605905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe594071cd0 con 0x7fe5940aad90 2023-03-03T08:43:59.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.397+0000 7fe599fa4700 1 -- 172.21.15.7:0/4248605905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59406ea00 con 0x7fe5940aad90 2023-03-03T08:43:59.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.397+0000 7fe58bfff700 1 -- 172.21.15.7:0/4248605905 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe58c010050 con 0x7fe5940aad90 2023-03-03T08:43:59.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.398+0000 7fe58bfff700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5740682f0 0x7fe57406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:43:59.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.398+0000 7fe58bfff700 1 -- 172.21.15.7:0/4248605905 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fe58c085ea0 con 0x7fe5940aad90 2023-03-03T08:43:59.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.398+0000 7fe5937fe700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5740682f0 0x7fe57406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:43:59.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.399+0000 7fe5937fe700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5740682f0 0x7fe57406a7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe57c007fd0 tx=0x7fe57c007e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:43:59.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.402+0000 7fe58bfff700 1 -- 172.21.15.7:0/4248605905 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe58c0523d0 con 0x7fe5940aad90 2023-03-03T08:43:59.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.641+0000 7fe599fa4700 1 -- 172.21.15.7:0/4248605905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe594074bb0 con 0x7fe5940aad90 2023-03-03T08:43:59.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.641+0000 7fe58bfff700 1 -- 172.21.15.7:0/4248605905 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+116 (secure 0 0 0) 0x7fe58c0559f0 con 0x7fe5940aad90 2023-03-03T08:43:59.643 INFO:teuthology.orchestra.run.smithi007.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2023-03-03T08:43:59.643 INFO:teuthology.orchestra.run.smithi007.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2023-03-03T08:43:59.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 -- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5740682f0 msgr2=0x7fe57406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:59.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5740682f0 0x7fe57406a7a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe57c007fd0 tx=0x7fe57c007e00 comp rx=0 tx=0).stop 2023-03-03T08:43:59.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 -- 172.21.15.7:0/4248605905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5940aad90 msgr2=0x7fe594154ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:43:59.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5940aad90 0x7fe594154ae0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe58c00cb50 tx=0x7fe58c00ce60 comp rx=0 tx=0).stop 2023-03-03T08:43:59.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 -- 172.21.15.7:0/4248605905 shutdown_connections 2023-03-03T08:43:59.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5740682f0 0x7fe57406a7a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:59.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5940aad90 0x7fe594154ae0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe59406b920 0x7fe5940ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 --2- 172.21.15.7:0/4248605905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe59406bfe0 0x7fe5940c6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:43:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 -- 172.21.15.7:0/4248605905 >> 172.21.15.7:0/4248605905 conn(0x7fe5940736f0 msgr2=0x7fe59406b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:43:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 -- 172.21.15.7:0/4248605905 shutdown_connections 2023-03-03T08:43:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:43:59.643+0000 7fe599fa4700 1 -- 172.21.15.7:0/4248605905 wait complete. 2023-03-03T08:43:59.656 INFO:tasks.cephfs.mount:Mounting Ceph FS b; just confirmed its presence on cluster 2023-03-03T08:43:59.657 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-03T08:43:59.657 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi143.front.sepia.ceph.com 2023-03-03T08:43:59.657 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-03-03T08:43:59.658 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-03-03T08:43:59.658 INFO:tasks.cephfs.mount:self.cephfs_name = b 2023-03-03T08:43:59.658 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-03-03T08:43:59.658 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-03T08:43:59.658 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:43:59.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:59.659 DEBUG:teuthology.orchestra.run.smithi143:> ip addr 2023-03-03T08:43:59.680 INFO:teuthology.orchestra.run.smithi143.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-03T08:43:59.680 INFO:teuthology.orchestra.run.smithi143.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-03T08:43:59.680 INFO:teuthology.orchestra.run.smithi143.stdout: inet 127.0.0.1/8 scope host lo 2023-03-03T08:43:59.680 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:43:59.680 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 ::1/128 scope host 2023-03-03T08:43:59.681 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:43:59.681 INFO:teuthology.orchestra.run.smithi143.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:43:59.681 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:d9:3b:94 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:43:59.681 INFO:teuthology.orchestra.run.smithi143.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:43:59.681 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:d9:3b:95 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:43:59.681 INFO:teuthology.orchestra.run.smithi143.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-03T08:43:59.681 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:88:77:82 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:43:59.682 INFO:teuthology.orchestra.run.smithi143.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-03T08:43:59.682 INFO:teuthology.orchestra.run.smithi143.stdout: link/ether 0c:c4:7a:88:77:83 brd ff:ff:ff:ff:ff:ff 2023-03-03T08:43:59.682 INFO:teuthology.orchestra.run.smithi143.stdout: inet 172.21.15.143/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-03-03T08:43:59.682 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft 42496sec preferred_lft 42496sec 2023-03-03T08:43:59.682 INFO:teuthology.orchestra.run.smithi143.stdout: inet6 fe80::8131:81f7:573:c857/64 scope link noprefixroute 2023-03-03T08:43:59.682 INFO:teuthology.orchestra.run.smithi143.stdout: valid_lft forever preferred_lft forever 2023-03-03T08:43:59.682 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-03-03T08:43:59.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:59.683 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-03-03T08:43:59.683 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-03-03T08:43:59.684 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link add name ceph-brx type bridge 2023-03-03T08:43:59.684 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip addr flush dev ceph-brx 2023-03-03T08:43:59.684 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link set ceph-brx up 2023-03-03T08:43:59.685 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-03-03T08:43:59.685 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-03-03T08:43:59.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:59.862 DEBUG:teuthology.orchestra.run.smithi143:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-03T08:43:59.890 INFO:teuthology.orchestra.run.smithi143.stdout:1 2023-03-03T08:43:59.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:59.891 DEBUG:teuthology.orchestra.run.smithi143:> route 2023-03-03T08:43:59.964 INFO:teuthology.orchestra.run.smithi143.stdout:Kernel IP routing table 2023-03-03T08:43:59.965 INFO:teuthology.orchestra.run.smithi143.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-03T08:43:59.965 INFO:teuthology.orchestra.run.smithi143.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-03-03T08:43:59.965 INFO:teuthology.orchestra.run.smithi143.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-03-03T08:43:59.965 INFO:teuthology.orchestra.run.smithi143.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-03T08:43:59.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:43:59.966 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-03-03T08:43:59.966 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-03-03T08:43:59.967 DEBUG:teuthology.orchestra.run.smithi143:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-03-03T08:43:59.967 DEBUG:teuthology.orchestra.run.smithi143:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-03-03T08:43:59.967 DEBUG:teuthology.orchestra.run.smithi143:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-03-03T08:43:59.967 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-03-03T08:44:00.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:00.251 DEBUG:teuthology.orchestra.run.smithi143:> ip netns list 2023-03-03T08:44:00.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:00.268 DEBUG:teuthology.orchestra.run.smithi143:> ip netns list-id 2023-03-03T08:44:00.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:00.325 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-03-03T08:44:00.325 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-03-03T08:44:00.325 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-03-03T08:44:00.326 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-03-03T08:44:00.326 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-03-03T08:44:00.418 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-03-03T08:44:00.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:00.419 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-03-03T08:44:00.419 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-03-03T08:44:00.419 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-03-03T08:44:00.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-03-03T08:44:00.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-03-03T08:44:00.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-03-03T08:44:00.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-03-03T08:44:00.421 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-03-03T08:44:00.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:00.625 DEBUG:teuthology.orchestra.run.smithi143:> (cd / && exec bash -c ' 2023-03-03T08:44:00.626 DEBUG:teuthology.orchestra.run.smithi143:> set -e 2023-03-03T08:44:00.626 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link set brx.0 up 2023-03-03T08:44:00.626 DEBUG:teuthology.orchestra.run.smithi143:> sudo ip link set dev brx.0 master ceph-brx 2023-03-03T08:44:00.626 DEBUG:teuthology.orchestra.run.smithi143:> ') 2023-03-03T08:44:00.725 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:44:00.726 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-03-03T08:44:00.746 INFO:teuthology.orchestra.run.smithi143.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-03-03T08:44:00.746 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-03T08:44:00.747 DEBUG:teuthology.orchestra.run.smithi143:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-03-03T08:44:00.802 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.b=/ 2023-03-03T08:44:00.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:00.803 DEBUG:teuthology.orchestra.run.smithi143:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.b=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T08:44:00.894 INFO:teuthology.orchestra.run.smithi143.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T08:44:00.894 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-03-03T08:44:00.996 INFO:teuthology.orchestra.run.smithi143.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T08:44:00.997 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-03-03T08:44:00.997 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: resolved to: "172.21.15.7:3300,172.21.15.143:3300,172.21.15.143:3301" 2023-03-03T08:44:00.997 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: trying mount with new device syntax: 1@4366045e-92da-4bb9-8565-4b6c37871975.b=/ 2023-03-03T08:44:00.997 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.7:3300/172.21.15.143:3300/172.21.15.143:3301" will pass to kernel 2023-03-03T08:44:00.998 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: v2 mount support check returned 2 2023-03-03T08:44:00.998 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: kernel does not support v2 syntax 2023-03-03T08:44:00.998 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: switching to using v1 address with old syntax 2023-03-03T08:44:01.027 INFO:teuthology.orchestra.run.smithi143.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-03-03T08:44:01.028 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-03-03T08:44:01.028 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: resolved to: "172.21.15.7:3300,172.21.15.143:3300,172.21.15.143:3301" 2023-03-03T08:44:01.028 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: trying mount with new device syntax: 1@4366045e-92da-4bb9-8565-4b6c37871975.b=/ 2023-03-03T08:44:01.029 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.7:3300/172.21.15.143:3300/172.21.15.143:3301" will pass to kernel 2023-03-03T08:44:01.029 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: v2 mount support check returned 2 2023-03-03T08:44:01.029 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: kernel does not support v2 syntax 2023-03-03T08:44:01.029 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: switching to using v1 address with old syntax 2023-03-03T08:44:01.030 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: resolved to: "172.21.15.7:6789,172.21.15.143:6789,172.21.15.143:6790" 2023-03-03T08:44:01.030 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.7:6789,172.21.15.143:6789,172.21.15.143:6790:/ 2023-03-03T08:44:01.030 INFO:teuthology.orchestra.run.smithi143.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,key=1,mds_namespace=b,fsid=4366045e-92da-4bb9-8565-4b6c37871975" will pass to kernel 2023-03-03T08:44:01.031 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-03-03T08:44:01.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:01.032 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-03-03T08:44:01.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:01.063 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T08:44:01.151 INFO:teuthology.orchestra.run.smithi143.stdout:client4541 2023-03-03T08:44:01.151 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:44:01.152 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-03T08:44:01.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:01.169 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T08:44:01.791 INFO:teuthology.orchestra.run.smithi143.stdout:client4541 2023-03-03T08:44:01.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:01.792 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-03T08:44:01.841 INFO:teuthology.orchestra.run.smithi143.stdout:client4541 2023-03-03T08:44:01.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-03T08:44:01.842 DEBUG:teuthology.orchestra.run.smithi143:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/4366045e-92da-4bb9-8565-4b6c37871975.client4541/status') 2023-03-03T08:44:02.180 INFO:teuthology.orchestra.run.smithi143.stdout:instance: client.4541 (3)192.168.0.1:0/3166293783 2023-03-03T08:44:02.180 INFO:teuthology.orchestra.run.smithi143.stdout:blocklisted: false 2023-03-03T08:44:02.182 INFO:teuthology.orchestra.run.smithi143.stderr:0+1 records in 2023-03-03T08:44:02.182 INFO:teuthology.orchestra.run.smithi143.stderr:0+1 records out 2023-03-03T08:44:02.183 INFO:teuthology.orchestra.run.smithi143.stderr:69 bytes copied, 3.7997e-05 s, 1.8 MB/s 2023-03-03T08:44:02.183 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-03-03T08:44:02.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:44:02.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.345+0000 7f6ba02a4700 1 Processor -- start 2023-03-03T08:44:02.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.345+0000 7f6ba02a4700 1 -- start start 2023-03-03T08:44:02.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6ba02a4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b980aac00 0x7f6b980c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6ba02a4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 0x7f6b98155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6ba02a4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b980713c0 con 0x7f6b98071c30 2023-03-03T08:44:02.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6ba02a4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b98156310 con 0x7f6b980aac00 2023-03-03T08:44:02.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6ba02a4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b98152c70 con 0x7f6b98155950 2023-03-03T08:44:02.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6b9e040700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b980aac00 0x7f6b980c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6b9e040700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b980aac00 0x7f6b980c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52038/0 (socket says 172.21.15.7:52038) 2023-03-03T08:44:02.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6b9e040700 1 -- 172.21.15.7:0/3443338535 learned_addr learned my addr 172.21.15.7:0/3443338535 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:02.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6b9e841700 1 --2- 172.21.15.7:0/3443338535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 0x7f6b98155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6b9e040700 1 -- 172.21.15.7:0/3443338535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 msgr2=0x7f6b98155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:02.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6b9e040700 1 --2- 172.21.15.7:0/3443338535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 0x7f6b98155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6b9e040700 1 -- 172.21.15.7:0/3443338535 >> v1:172.21.15.143:6789/0 conn(0x7f6b98071c30 legacy=0x7f6b980beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6b9e040700 1 -- 172.21.15.7:0/3443338535 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b980c4930 con 0x7f6b980aac00 2023-03-03T08:44:02.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6b9e841700 1 --2- 172.21.15.7:0/3443338535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 0x7f6b98155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:02.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.346+0000 7f6b9e040700 1 --2- 172.21.15.7:0/3443338535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 0x7f6b980c43f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6b940098f0 tx=0x7f6b94009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f5548bbb2f6f23 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:02.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6b9d03e700 1 -- 172.21.15.7:0/3443338535 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400b8a0 con 0x7f6b980aac00 2023-03-03T08:44:02.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6b9d03e700 1 -- 172.21.15.7:0/3443338535 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9400ba00 con 0x7f6b980aac00 2023-03-03T08:44:02.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6b9d03e700 1 -- 172.21.15.7:0/3443338535 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400bd90 con 0x7f6b980aac00 2023-03-03T08:44:02.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6ba02a4700 1 -- 172.21.15.7:0/3443338535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 msgr2=0x7f6b980c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6ba02a4700 1 --2- 172.21.15.7:0/3443338535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 0x7f6b980c43f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6b940098f0 tx=0x7f6b94009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:02.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6ba02a4700 1 -- 172.21.15.7:0/3443338535 shutdown_connections 2023-03-03T08:44:02.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6ba02a4700 1 --2- 172.21.15.7:0/3443338535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 0x7f6b98155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6ba02a4700 1 --2- 172.21.15.7:0/3443338535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 0x7f6b980c43f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6ba02a4700 1 -- 172.21.15.7:0/3443338535 >> 172.21.15.7:0/3443338535 conn(0x7f6b980736f0 msgr2=0x7f6b98073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:02.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6ba02a4700 1 -- 172.21.15.7:0/3443338535 shutdown_connections 2023-03-03T08:44:02.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.347+0000 7f6ba02a4700 1 -- 172.21.15.7:0/3443338535 wait complete. 2023-03-03T08:44:02.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.348+0000 7f6ba02a4700 1 Processor -- start 2023-03-03T08:44:02.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.349+0000 7f6ba02a4700 1 -- start start 2023-03-03T08:44:02.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.349+0000 7f6ba02a4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98071c30 0x7f6b9814ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.349+0000 7f6ba02a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 0x7f6b9806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.349+0000 7f6ba02a4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 0x7f6b9806a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.349+0000 7f6ba02a4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b98151470 con 0x7f6b980aac00 2023-03-03T08:44:02.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.349+0000 7f6ba02a4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b980c8390 con 0x7f6b98071c30 2023-03-03T08:44:02.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.349+0000 7f6ba02a4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b980c8630 con 0x7f6b98155950 2023-03-03T08:44:02.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.349+0000 7f6b9d83f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 0x7f6b9806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.349+0000 7f6b9d83f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 0x7f6b9806a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52046/0 (socket says 172.21.15.7:52046) 2023-03-03T08:44:02.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.349+0000 7f6b9d83f700 1 -- 172.21.15.7:0/1784921557 learned_addr learned my addr 172.21.15.7:0/1784921557 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:02.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b9e040700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98071c30 0x7f6b9814ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b9e841700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 0x7f6b9806a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b9d83f700 1 -- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 msgr2=0x7f6b9806a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b9d83f700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 0x7f6b9806a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b9d83f700 1 -- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98071c30 msgr2=0x7f6b9814ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b9d83f700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98071c30 0x7f6b9814ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b9d83f700 1 -- 172.21.15.7:0/1784921557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b94009550 con 0x7f6b980aac00 2023-03-03T08:44:02.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b9d83f700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 0x7f6b9806a2b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800c4e0 tx=0x7f6b8800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:02.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b8effd700 1 -- 172.21.15.7:0/1784921557 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b880052d0 con 0x7f6b980aac00 2023-03-03T08:44:02.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b8effd700 1 -- 172.21.15.7:0/1784921557 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b88005430 con 0x7f6b980aac00 2023-03-03T08:44:02.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6ba02a4700 1 -- 172.21.15.7:0/1784921557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b980c4b80 con 0x7f6b980aac00 2023-03-03T08:44:02.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6b8effd700 1 -- 172.21.15.7:0/1784921557 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b880038c0 con 0x7f6b980aac00 2023-03-03T08:44:02.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.350+0000 7f6ba02a4700 1 -- 172.21.15.7:0/1784921557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b98152360 con 0x7f6b980aac00 2023-03-03T08:44:02.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.351+0000 7f6ba02a4700 1 -- 172.21.15.7:0/1784921557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b98152620 con 0x7f6b980aac00 2023-03-03T08:44:02.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.352+0000 7f6b8effd700 1 -- 172.21.15.7:0/1784921557 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6b88019030 con 0x7f6b980aac00 2023-03-03T08:44:02.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.352+0000 7f6b8effd700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 0x7f6b7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.352+0000 7f6b8effd700 1 -- 172.21.15.7:0/1784921557 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6b88085320 con 0x7f6b980aac00 2023-03-03T08:44:02.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.352+0000 7f6b9e040700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 0x7f6b7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.353+0000 7f6b9e040700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 0x7f6b7c06a7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009c40 tx=0x7f6b94008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:02.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.356+0000 7f6b8effd700 1 -- 172.21.15.7:0/1784921557 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b88051850 con 0x7f6b980aac00 2023-03-03T08:44:02.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.549+0000 7f6ba02a4700 1 -- 172.21.15.7:0/1784921557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b98074bb0 con 0x7f6b980aac00 2023-03-03T08:44:02.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.551+0000 7f6b8effd700 1 -- 172.21.15.7:0/1784921557 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21533 (secure 0 0 0) 0x7f6b88054e70 con 0x7f6b980aac00 2023-03-03T08:44:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:02.553 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:44.973251+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:44:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 -- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 msgr2=0x7f6b7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 0x7f6b7c06a7a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009c40 tx=0x7f6b94008030 comp rx=0 tx=0).stop 2023-03-03T08:44:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 -- 172.21.15.7:0/1784921557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 msgr2=0x7f6b9806a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 0x7f6b9806a2b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800c4e0 tx=0x7f6b8800c8a0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 -- 172.21.15.7:0/1784921557 shutdown_connections 2023-03-03T08:44:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b98155950 0x7f6b9806a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98071c30 0x7f6b9814ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 0x7f6b7c06a7a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 --2- 172.21.15.7:0/1784921557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b980aac00 0x7f6b9806a2b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 -- 172.21.15.7:0/1784921557 >> 172.21.15.7:0/1784921557 conn(0x7f6b980736f0 msgr2=0x7f6b981546f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 -- 172.21.15.7:0/1784921557 shutdown_connections 2023-03-03T08:44:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.553+0000 7f6ba02a4700 1 -- 172.21.15.7:0/1784921557 wait complete. 2023-03-03T08:44:02.566 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T08:43:21.160680+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T08:43:28.018457+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-03-03T08:43:32.305037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-03-03T08:43:33.484501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-03-03T08:43:42.509451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-03-03T08:43:42.936898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:44:02.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:44:02.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.693+0000 7fb6defd3700 1 Processor -- start 2023-03-03T08:44:02.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.693+0000 7fb6defd3700 1 -- start start 2023-03-03T08:44:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6defd3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb6d813a0f0 0x7fb6d80c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6defd3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb6d81586c0 0x7fb6d806d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6defd3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d80c1da0 0x7fb6d806d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6defd3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d8154840 con 0x7fb6d813a0f0 2023-03-03T08:44:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6defd3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6d8154560 con 0x7fb6d81586c0 2023-03-03T08:44:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6defd3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d8157be0 con 0x7fb6d80c1da0 2023-03-03T08:44:02.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6dcd6f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb6d813a0f0 0x7fb6d80c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6dcd6f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb6d813a0f0 0x7fb6d80c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52050/0 (socket says 172.21.15.7:52050) 2023-03-03T08:44:02.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6dcd6f700 1 -- 172.21.15.7:0/1837248504 learned_addr learned my addr 172.21.15.7:0/1837248504 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:02.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6d7fff700 1 --2- 172.21.15.7:0/1837248504 >> v2:172.21.15.143:3300/0 conn(0x7fb6d81586c0 0x7fb6d806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.694+0000 7fb6dd570700 1 --2- 172.21.15.7:0/1837248504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d80c1da0 0x7fb6d806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6dcd6f700 1 -- 172.21.15.7:0/1837248504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d80c1da0 msgr2=0x7fb6d806d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:02.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6dcd6f700 1 --2- 172.21.15.7:0/1837248504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d80c1da0 0x7fb6d806d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6dcd6f700 1 -- 172.21.15.7:0/1837248504 >> v2:172.21.15.143:3300/0 conn(0x7fb6d81586c0 msgr2=0x7fb6d806d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:02.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6dcd6f700 1 --2- 172.21.15.7:0/1837248504 >> v2:172.21.15.143:3300/0 conn(0x7fb6d81586c0 0x7fb6d806d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6dcd6f700 1 -- 172.21.15.7:0/1837248504 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d8157ec0 con 0x7fb6d813a0f0 2023-03-03T08:44:02.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6dd570700 1 --2- 172.21.15.7:0/1837248504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d80c1da0 0x7fb6d806d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:02.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6dcd6f700 1 --2- 172.21.15.7:0/1837248504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d813a0f0 0x7fb6d80c07d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8072a30 tx=0x7fb6cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45385a9758a6c23f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:02.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6d77fe700 1 -- 172.21.15.7:0/1837248504 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc011830 con 0x7fb6d813a0f0 2023-03-03T08:44:02.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6d77fe700 1 -- 172.21.15.7:0/1837248504 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc011990 con 0x7fb6d813a0f0 2023-03-03T08:44:02.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6d77fe700 1 -- 172.21.15.7:0/1837248504 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc011d20 con 0x7fb6d813a0f0 2023-03-03T08:44:02.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6defd3700 1 -- 172.21.15.7:0/1837248504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d813a0f0 msgr2=0x7fb6d80c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6defd3700 1 --2- 172.21.15.7:0/1837248504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d813a0f0 0x7fb6d80c07d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8072a30 tx=0x7fb6cc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6defd3700 1 -- 172.21.15.7:0/1837248504 shutdown_connections 2023-03-03T08:44:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6defd3700 1 --2- 172.21.15.7:0/1837248504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d80c1da0 0x7fb6d806d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6defd3700 1 --2- 172.21.15.7:0/1837248504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d813a0f0 0x7fb6d80c07d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6defd3700 1 --2- 172.21.15.7:0/1837248504 >> v2:172.21.15.143:3300/0 conn(0x7fb6d81586c0 0x7fb6d806d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6defd3700 1 -- 172.21.15.7:0/1837248504 >> 172.21.15.7:0/1837248504 conn(0x7fb6d806c110 msgr2=0x7fb6d806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6defd3700 1 -- 172.21.15.7:0/1837248504 shutdown_connections 2023-03-03T08:44:02.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.695+0000 7fb6defd3700 1 -- 172.21.15.7:0/1837248504 wait complete. 2023-03-03T08:44:02.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.697+0000 7fb6defd3700 1 Processor -- start 2023-03-03T08:44:02.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.697+0000 7fb6defd3700 1 -- start start 2023-03-03T08:44:02.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.697+0000 7fb6defd3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d80c1da0 0x7fb6d81e0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.697+0000 7fb6defd3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6d813a0f0 0x7fb6d80aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6defd3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d81586c0 0x7fb6d81ecf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6defd3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d80ad0c0 con 0x7fb6d80c1da0 2023-03-03T08:44:02.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6defd3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d8071570 con 0x7fb6d813a0f0 2023-03-03T08:44:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6defd3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d813dad0 con 0x7fb6d81586c0 2023-03-03T08:44:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6dcd6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d80c1da0 0x7fb6d81e0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6dcd6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d80c1da0 0x7fb6d81e0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52056/0 (socket says 172.21.15.7:52056) 2023-03-03T08:44:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6dcd6f700 1 -- 172.21.15.7:0/2804725646 learned_addr learned my addr 172.21.15.7:0/2804725646 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6dcd6f700 1 -- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d81586c0 msgr2=0x7fb6d81ecf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6d7fff700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6d813a0f0 0x7fb6d80aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6dd570700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d81586c0 0x7fb6d81ecf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6dcd6f700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d81586c0 0x7fb6d81ecf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6dcd6f700 1 -- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6d813a0f0 msgr2=0x7fb6d80aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6dcd6f700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6d813a0f0 0x7fb6d80aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6dcd6f700 1 -- 172.21.15.7:0/2804725646 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc009550 con 0x7fb6d80c1da0 2023-03-03T08:44:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6d7fff700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6d813a0f0 0x7fb6d80aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6dcd6f700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d80c1da0 0x7fb6d81e0bb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc000c00 tx=0x7fb6cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6d57fa700 1 -- 172.21.15.7:0/2804725646 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc004380 con 0x7fb6d80c1da0 2023-03-03T08:44:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6d57fa700 1 -- 172.21.15.7:0/2804725646 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc01f070 con 0x7fb6d80c1da0 2023-03-03T08:44:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6defd3700 1 -- 172.21.15.7:0/2804725646 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d806e1a0 con 0x7fb6d80c1da0 2023-03-03T08:44:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6d57fa700 1 -- 172.21.15.7:0/2804725646 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc0036a0 con 0x7fb6d80c1da0 2023-03-03T08:44:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.698+0000 7fb6defd3700 1 -- 172.21.15.7:0/2804725646 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d81581a0 con 0x7fb6d80c1da0 2023-03-03T08:44:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.699+0000 7fb6defd3700 1 -- 172.21.15.7:0/2804725646 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d806e330 con 0x7fb6d80c1da0 2023-03-03T08:44:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.700+0000 7fb6d57fa700 1 -- 172.21.15.7:0/2804725646 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb6cc02a020 con 0x7fb6d80c1da0 2023-03-03T08:44:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.700+0000 7fb6d57fa700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6bc0682f0 0x7fb6bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.700+0000 7fb6d57fa700 1 -- 172.21.15.7:0/2804725646 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb6cc087610 con 0x7fb6d80c1da0 2023-03-03T08:44:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.701+0000 7fb6d7fff700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6bc0682f0 0x7fb6bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:02.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.701+0000 7fb6d7fff700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6bc0682f0 0x7fb6bc06a7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8155240 tx=0x7fb6c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:02.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.705+0000 7fb6d57fa700 1 -- 172.21.15.7:0/2804725646 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb6cc053a50 con 0x7fb6d80c1da0 2023-03-03T08:44:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.897+0000 7fb6defd3700 1 -- 172.21.15.7:0/2804725646 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6d8139930 con 0x7fb6d80c1da0 2023-03-03T08:44:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.897+0000 7fb6d57fa700 1 -- 172.21.15.7:0/2804725646 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6cc057070 con 0x7fb6d80c1da0 2023-03-03T08:44:02.899 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-03-03T08:44:02.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 -- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6bc0682f0 msgr2=0x7fb6bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6bc0682f0 0x7fb6bc06a7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8155240 tx=0x7fb6c4008040 comp rx=0 tx=0).stop 2023-03-03T08:44:02.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 -- 172.21.15.7:0/2804725646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d80c1da0 msgr2=0x7fb6d81e0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:02.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d80c1da0 0x7fb6d81e0bb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc000c00 tx=0x7fb6cc004750 comp rx=0 tx=0).stop 2023-03-03T08:44:02.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 -- 172.21.15.7:0/2804725646 shutdown_connections 2023-03-03T08:44:02.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6d81586c0 0x7fb6d81ecf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6bc0682f0 0x7fb6bc06a7a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6d80c1da0 0x7fb6d81e0bb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 --2- 172.21.15.7:0/2804725646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6d813a0f0 0x7fb6d80aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:02.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 -- 172.21.15.7:0/2804725646 >> 172.21.15.7:0/2804725646 conn(0x7fb6d806c110 msgr2=0x7fb6d80736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:02.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 -- 172.21.15.7:0/2804725646 shutdown_connections 2023-03-03T08:44:02.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:02.899+0000 7fb6defd3700 1 -- 172.21.15.7:0/2804725646 wait complete. 2023-03-03T08:44:02.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:44:03.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.038+0000 7f61f0bea700 1 Processor -- start 2023-03-03T08:44:03.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.038+0000 7f61f0bea700 1 -- start start 2023-03-03T08:44:03.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.039+0000 7f61f0bea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c2e30 0x7f61ec158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.039+0000 7f61f0bea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f61ec06c6c0 0x7f61ec0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.039+0000 7f61f0bea700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ec157ff0 con 0x7f61ec151cd0 2023-03-03T08:44:03.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.039+0000 7f61f0bea700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61ec06d640 con 0x7f61ec06c6c0 2023-03-03T08:44:03.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.039+0000 7f61f0bea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec06cda0 con 0x7f61ec0c2e30 2023-03-03T08:44:03.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.039+0000 7f61ead9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f61ec06c6c0 0x7f61ec0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.039+0000 7f61ea59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f61ec151cd0 0x7f61ec154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46044/0 (socket says 172.21.15.7:46044) 2023-03-03T08:44:03.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.039+0000 7f61ea59c700 1 -- 172.21.15.7:0/2113895185 learned_addr learned my addr 172.21.15.7:0/2113895185 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:03.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.039+0000 7f61e9d9b700 1 --2- 172.21.15.7:0/2113895185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c2e30 0x7f61ec158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61ead9d700 1 -- 172.21.15.7:0/2113895185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c2e30 msgr2=0x7f61ec158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:03.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61ead9d700 1 --2- 172.21.15.7:0/2113895185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c2e30 0x7f61ec158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61ead9d700 1 -- 172.21.15.7:0/2113895185 >> v1:172.21.15.7:6789/0 conn(0x7f61ec151cd0 legacy=0x7f61ec154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61e959a700 1 -- 172.21.15.7:0/2113895185 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890740544 0 0) 0x7f61ec157ff0 con 0x7f61ec151cd0 2023-03-03T08:44:03.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61ead9d700 1 -- 172.21.15.7:0/2113895185 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec06d080 con 0x7f61ec06c6c0 2023-03-03T08:44:03.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61e9d9b700 1 --2- 172.21.15.7:0/2113895185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c2e30 0x7f61ec158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:03.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61ead9d700 1 --2- 172.21.15.7:0/2113895185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec06c6c0 0x7f61ec0736f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f61e4009d90 tx=0x7f61e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c900a2056d3ea9f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:03.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61e959a700 1 -- 172.21.15.7:0/2113895185 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e400e070 con 0x7f61ec06c6c0 2023-03-03T08:44:03.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61e959a700 1 -- 172.21.15.7:0/2113895185 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e400bbd0 con 0x7f61ec06c6c0 2023-03-03T08:44:03.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61e959a700 1 -- 172.21.15.7:0/2113895185 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4012430 con 0x7f61ec06c6c0 2023-03-03T08:44:03.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61f0bea700 1 -- 172.21.15.7:0/2113895185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec06c6c0 msgr2=0x7f61ec0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.040+0000 7f61f0bea700 1 --2- 172.21.15.7:0/2113895185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec06c6c0 0x7f61ec0736f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f61e4009d90 tx=0x7f61e400b330 comp rx=0 tx=0).stop 2023-03-03T08:44:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.041+0000 7f61f0bea700 1 -- 172.21.15.7:0/2113895185 shutdown_connections 2023-03-03T08:44:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.041+0000 7f61f0bea700 1 --2- 172.21.15.7:0/2113895185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec06c6c0 0x7f61ec0736f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.041+0000 7f61f0bea700 1 --2- 172.21.15.7:0/2113895185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c2e30 0x7f61ec158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.041+0000 7f61f0bea700 1 -- 172.21.15.7:0/2113895185 >> 172.21.15.7:0/2113895185 conn(0x7f61ec0a94d0 msgr2=0x7f61ec0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.041+0000 7f61f0bea700 1 -- 172.21.15.7:0/2113895185 shutdown_connections 2023-03-03T08:44:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.041+0000 7f61f0bea700 1 -- 172.21.15.7:0/2113895185 wait complete. 2023-03-03T08:44:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.042+0000 7f61f0bea700 1 Processor -- start 2023-03-03T08:44:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.042+0000 7f61f0bea700 1 -- start start 2023-03-03T08:44:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61f0bea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec06c6c0 0x7f61ec155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61f0bea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ec0c2e30 0x7f61ec1eac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61f0bea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec151cd0 0x7f61ec157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61f0bea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec14cbe0 con 0x7f61ec151cd0 2023-03-03T08:44:03.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61f0bea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec14cec0 con 0x7f61ec0c2e30 2023-03-03T08:44:03.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61f0bea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec14d1a0 con 0x7f61ec06c6c0 2023-03-03T08:44:03.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61ead9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec151cd0 0x7f61ec157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61ead9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec151cd0 0x7f61ec157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52074/0 (socket says 172.21.15.7:52074) 2023-03-03T08:44:03.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61ead9d700 1 -- 172.21.15.7:0/1210459237 learned_addr learned my addr 172.21.15.7:0/1210459237 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:03.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61ea59c700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec06c6c0 0x7f61ec155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61e9d9b700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ec0c2e30 0x7f61ec1eac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61ead9d700 1 -- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec06c6c0 msgr2=0x7f61ec155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61ead9d700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec06c6c0 0x7f61ec155900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61ead9d700 1 -- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ec0c2e30 msgr2=0x7f61ec1eac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61ead9d700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ec0c2e30 0x7f61ec1eac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61ead9d700 1 -- 172.21.15.7:0/1210459237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e4009a70 con 0x7f61ec151cd0 2023-03-03T08:44:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.043+0000 7f61ead9d700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec151cd0 0x7f61ec157180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0744a0 tx=0x7f61e4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.044+0000 7f61daffd700 1 -- 172.21.15.7:0/1210459237 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4012430 con 0x7f61ec151cd0 2023-03-03T08:44:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.044+0000 7f61daffd700 1 -- 172.21.15.7:0/1210459237 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e4020070 con 0x7f61ec151cd0 2023-03-03T08:44:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.044+0000 7f61f0bea700 1 -- 172.21.15.7:0/1210459237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ec157c80 con 0x7f61ec151cd0 2023-03-03T08:44:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.044+0000 7f61daffd700 1 -- 172.21.15.7:0/1210459237 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4004280 con 0x7f61ec151cd0 2023-03-03T08:44:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.044+0000 7f61f0bea700 1 -- 172.21.15.7:0/1210459237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ec06d360 con 0x7f61ec151cd0 2023-03-03T08:44:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.044+0000 7f61f0bea700 1 -- 172.21.15.7:0/1210459237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ec1589c0 con 0x7f61ec151cd0 2023-03-03T08:44:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.045+0000 7f61daffd700 1 -- 172.21.15.7:0/1210459237 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f61e40043e0 con 0x7f61ec151cd0 2023-03-03T08:44:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.045+0000 7f61daffd700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61cc0682f0 0x7f61cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.045+0000 7f61daffd700 1 -- 172.21.15.7:0/1210459237 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f61e4022070 con 0x7f61ec151cd0 2023-03-03T08:44:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.046+0000 7f61ea59c700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61cc0682f0 0x7f61cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.046+0000 7f61ea59c700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61cc0682f0 0x7f61cc06a7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f61e0000f60 tx=0x7f61e0007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.050+0000 7f61daffd700 1 -- 172.21.15.7:0/1210459237 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f61e4054a50 con 0x7f61ec151cd0 2023-03-03T08:44:03.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.239+0000 7f61f0bea700 1 -- 172.21.15.7:0/1210459237 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f61ec14d360 con 0x7f61ec151cd0 2023-03-03T08:44:03.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.240+0000 7f61daffd700 1 -- 172.21.15.7:0/1210459237 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f61e4058070 con 0x7f61ec151cd0 2023-03-03T08:44:03.242 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 8 2023-03-03T08:44:03.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 -- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61cc0682f0 msgr2=0x7f61cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61cc0682f0 0x7f61cc06a7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f61e0000f60 tx=0x7f61e0007c90 comp rx=0 tx=0).stop 2023-03-03T08:44:03.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 -- 172.21.15.7:0/1210459237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec151cd0 msgr2=0x7f61ec157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec151cd0 0x7f61ec157180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0744a0 tx=0x7f61e4004ad0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 -- 172.21.15.7:0/1210459237 shutdown_connections 2023-03-03T08:44:03.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61cc0682f0 0x7f61cc06a7a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec151cd0 0x7f61ec157180 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec06c6c0 0x7f61ec155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 --2- 172.21.15.7:0/1210459237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ec0c2e30 0x7f61ec1eac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 -- 172.21.15.7:0/1210459237 >> 172.21.15.7:0/1210459237 conn(0x7f61ec0a94d0 msgr2=0x7f61ec06a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:03.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 -- 172.21.15.7:0/1210459237 shutdown_connections 2023-03-03T08:44:03.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.242+0000 7f61f0bea700 1 -- 172.21.15.7:0/1210459237 wait complete. 2023-03-03T08:44:03.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-03-03T08:44:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.382+0000 7f50ed432700 1 Processor -- start 2023-03-03T08:44:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.382+0000 7f50ed432700 1 -- start start 2023-03-03T08:44:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.382+0000 7f50ed432700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50e80c23a0 0x7f50e80c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.382+0000 7f50ed432700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50e8151ce0 0x7f50e80c3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.382+0000 7f50ed432700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f50e80c3d60 0x7f50e81548b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.382+0000 7f50ed432700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e806d330 con 0x7f50e80c23a0 2023-03-03T08:44:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.382+0000 7f50ed432700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e806bb90 con 0x7f50e80c3d60 2023-03-03T08:44:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.382+0000 7f50ed432700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e80c1a00 con 0x7f50e8151ce0 2023-03-03T08:44:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50e80c23a0 0x7f50e80c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50e80c23a0 0x7f50e80c14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52078/0 (socket says 172.21.15.7:52078) 2023-03-03T08:44:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e6ffd700 1 -- 172.21.15.7:0/3078931221 learned_addr learned my addr 172.21.15.7:0/3078931221 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e6ffd700 1 -- 172.21.15.7:0/3078931221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50e8151ce0 msgr2=0x7f50e80c3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e6ffd700 1 --2- 172.21.15.7:0/3078931221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50e8151ce0 0x7f50e80c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e6ffd700 1 -- 172.21.15.7:0/3078931221 >> v2:172.21.15.143:3300/0 conn(0x7f50e80c3d60 msgr2=0x7f50e81548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e77fe700 1 --2- 172.21.15.7:0/3078931221 >> v2:172.21.15.143:3300/0 conn(0x7f50e80c3d60 0x7f50e81548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e6ffd700 1 --2- 172.21.15.7:0/3078931221 >> v2:172.21.15.143:3300/0 conn(0x7f50e80c3d60 0x7f50e81548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e6ffd700 1 -- 172.21.15.7:0/3078931221 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e806c040 con 0x7f50e80c23a0 2023-03-03T08:44:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e77fe700 1 --2- 172.21.15.7:0/3078931221 >> v2:172.21.15.143:3300/0 conn(0x7f50e80c3d60 0x7f50e81548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e6ffd700 1 --2- 172.21.15.7:0/3078931221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 0x7f50e80c14c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f50e80c41e0 tx=0x7f50d800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d276931784ccaf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:03.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e5ffb700 1 -- 172.21.15.7:0/3078931221 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d800dd30 con 0x7f50e80c23a0 2023-03-03T08:44:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e5ffb700 1 -- 172.21.15.7:0/3078931221 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d800de90 con 0x7f50e80c23a0 2023-03-03T08:44:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.383+0000 7f50e5ffb700 1 -- 172.21.15.7:0/3078931221 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d8010690 con 0x7f50e80c23a0 2023-03-03T08:44:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.384+0000 7f50ed432700 1 -- 172.21.15.7:0/3078931221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 msgr2=0x7f50e80c14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.384+0000 7f50ed432700 1 --2- 172.21.15.7:0/3078931221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 0x7f50e80c14c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f50e80c41e0 tx=0x7f50d800be70 comp rx=0 tx=0).stop 2023-03-03T08:44:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.384+0000 7f50ed432700 1 -- 172.21.15.7:0/3078931221 shutdown_connections 2023-03-03T08:44:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.384+0000 7f50ed432700 1 --2- 172.21.15.7:0/3078931221 >> v2:172.21.15.143:3300/0 conn(0x7f50e80c3d60 0x7f50e81548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.384+0000 7f50ed432700 1 --2- 172.21.15.7:0/3078931221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 0x7f50e80c14c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.384+0000 7f50ed432700 1 --2- 172.21.15.7:0/3078931221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50e8151ce0 0x7f50e80c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.384+0000 7f50ed432700 1 -- 172.21.15.7:0/3078931221 >> 172.21.15.7:0/3078931221 conn(0x7f50e806a490 msgr2=0x7f50e806a860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:03.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.384+0000 7f50ed432700 1 -- 172.21.15.7:0/3078931221 shutdown_connections 2023-03-03T08:44:03.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.384+0000 7f50ed432700 1 -- 172.21.15.7:0/3078931221 wait complete. 2023-03-03T08:44:03.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.385+0000 7f50ed432700 1 Processor -- start 2023-03-03T08:44:03.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.385+0000 7f50ed432700 1 -- start start 2023-03-03T08:44:03.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50ed432700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 0x7f50e81e91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50ed432700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e80c3d60 0x7f50e81e96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50ed432700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50e8151ce0 0x7f50e8154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50ed432700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e8152c30 con 0x7f50e80c23a0 2023-03-03T08:44:03.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50ed432700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e81e9c20 con 0x7f50e80c3d60 2023-03-03T08:44:03.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50ed432700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e81e9e40 con 0x7f50e8151ce0 2023-03-03T08:44:03.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 0x7f50e81e91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 0x7f50e81e91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52084/0 (socket says 172.21.15.7:52084) 2023-03-03T08:44:03.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e6ffd700 1 -- 172.21.15.7:0/2727242520 learned_addr learned my addr 172.21.15.7:0/2727242520 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:03.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e77fe700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50e8151ce0 0x7f50e8154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e67fc700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e80c3d60 0x7f50e81e96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e6ffd700 1 -- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50e8151ce0 msgr2=0x7f50e8154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e6ffd700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50e8151ce0 0x7f50e8154120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e6ffd700 1 -- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e80c3d60 msgr2=0x7f50e81e96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e6ffd700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e80c3d60 0x7f50e81e96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e6ffd700 1 -- 172.21.15.7:0/2727242520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d800b7f0 con 0x7f50e80c23a0 2023-03-03T08:44:03.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.386+0000 7f50e77fe700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50e8151ce0 0x7f50e8154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:03.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.387+0000 7f50e6ffd700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 0x7f50e81e91a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f50e80c45f0 tx=0x7f50d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:03.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.387+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2727242520 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d801f070 con 0x7f50e80c23a0 2023-03-03T08:44:03.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.387+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2727242520 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d8003b20 con 0x7f50e80c23a0 2023-03-03T08:44:03.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.387+0000 7f50ed432700 1 -- 172.21.15.7:0/2727242520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e806b8b0 con 0x7f50e80c23a0 2023-03-03T08:44:03.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.387+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2727242520 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d8010470 con 0x7f50e80c23a0 2023-03-03T08:44:03.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.387+0000 7f50ed432700 1 -- 172.21.15.7:0/2727242520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e806bd60 con 0x7f50e80c23a0 2023-03-03T08:44:03.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.388+0000 7f50ed432700 1 -- 172.21.15.7:0/2727242520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e81514e0 con 0x7f50e80c23a0 2023-03-03T08:44:03.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.388+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2727242520 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f50d802d430 con 0x7f50e80c23a0 2023-03-03T08:44:03.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.388+0000 7f50cf7fe700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50d40682f0 0x7f50d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.389+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2727242520 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f50d8089c80 con 0x7f50e80c23a0 2023-03-03T08:44:03.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.389+0000 7f50e67fc700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50d40682f0 0x7f50d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.389+0000 7f50e67fc700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50d40682f0 0x7f50d406a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f50e80c0cb0 tx=0x7f50dc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:03.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.393+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2727242520 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f50d8056150 con 0x7f50e80c23a0 2023-03-03T08:44:03.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.585+0000 7f50ed432700 1 -- 172.21.15.7:0/2727242520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f50e8074bb0 con 0x7f50e80c23a0 2023-03-03T08:44:03.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.585+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2727242520 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f50d8059770 con 0x7f50e80c23a0 2023-03-03T08:44:03.587 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:44:03.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 -- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50d40682f0 msgr2=0x7f50d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50d40682f0 0x7f50d406a7a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f50e80c0cb0 tx=0x7f50dc009190 comp rx=0 tx=0).stop 2023-03-03T08:44:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 -- 172.21.15.7:0/2727242520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 msgr2=0x7f50e81e91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 0x7f50e81e91a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f50e80c45f0 tx=0x7f50d8004750 comp rx=0 tx=0).stop 2023-03-03T08:44:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 -- 172.21.15.7:0/2727242520 shutdown_connections 2023-03-03T08:44:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50e8151ce0 0x7f50e8154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50d40682f0 0x7f50d406a7a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50e80c23a0 0x7f50e81e91a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 --2- 172.21.15.7:0/2727242520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50e80c3d60 0x7f50e81e96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 -- 172.21.15.7:0/2727242520 >> 172.21.15.7:0/2727242520 conn(0x7f50e806a490 msgr2=0x7f50e814bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:03.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 -- 172.21.15.7:0/2727242520 shutdown_connections 2023-03-03T08:44:03.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.587+0000 7f50ed432700 1 -- 172.21.15.7:0/2727242520 wait complete. 2023-03-03T08:44:03.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-03-03T08:44:03.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.727+0000 7f4a31776700 1 Processor -- start 2023-03-03T08:44:03.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.727+0000 7f4a31776700 1 -- start start 2023-03-03T08:44:03.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.727+0000 7f4a31776700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4a2c0caa30 0x7f4a2c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a31776700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0c00e0 0x7f4a2c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a31776700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0cb280 con 0x7f4a2c0cae00 2023-03-03T08:44:03.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a31776700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a2c0c7310 con 0x7f4a2c0caa30 2023-03-03T08:44:03.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a31776700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c0bfd10 con 0x7f4a2c0c00e0 2023-03-03T08:44:03.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a2affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4a2c0caa30 0x7f4a2c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a2affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4a2c0caa30 0x7f4a2c0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52092/0 (socket says 172.21.15.7:52092) 2023-03-03T08:44:03.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a2affd700 1 -- 172.21.15.7:0/2381898020 learned_addr learned my addr 172.21.15.7:0/2381898020 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:03.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a2a7fc700 1 --2- 172.21.15.7:0/2381898020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0c00e0 0x7f4a2c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a2affd700 1 -- 172.21.15.7:0/2381898020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0c00e0 msgr2=0x7f4a2c154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:03.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a2affd700 1 --2- 172.21.15.7:0/2381898020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0c00e0 0x7f4a2c154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a2affd700 1 -- 172.21.15.7:0/2381898020 >> v1:172.21.15.143:6789/0 conn(0x7f4a2c0cae00 legacy=0x7f4a2c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a2affd700 1 -- 172.21.15.7:0/2381898020 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c0bfa30 con 0x7f4a2c0caa30 2023-03-03T08:44:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a2affd700 1 --2- 172.21.15.7:0/2381898020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 0x7f4a2c0c5c60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c0098f0 tx=0x7f4a1c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4476d03897209ce server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a29ffb700 1 -- 172.21.15.7:0/2381898020 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c004230 con 0x7f4a2c0caa30 2023-03-03T08:44:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a29ffb700 1 -- 172.21.15.7:0/2381898020 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1c004390 con 0x7f4a2c0caa30 2023-03-03T08:44:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.728+0000 7f4a29ffb700 1 -- 172.21.15.7:0/2381898020 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c0036a0 con 0x7f4a2c0caa30 2023-03-03T08:44:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.729+0000 7f4a31776700 1 -- 172.21.15.7:0/2381898020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 msgr2=0x7f4a2c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.729+0000 7f4a31776700 1 --2- 172.21.15.7:0/2381898020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 0x7f4a2c0c5c60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c0098f0 tx=0x7f4a1c009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.729+0000 7f4a31776700 1 -- 172.21.15.7:0/2381898020 shutdown_connections 2023-03-03T08:44:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.729+0000 7f4a31776700 1 --2- 172.21.15.7:0/2381898020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 0x7f4a2c0c5c60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.729+0000 7f4a31776700 1 --2- 172.21.15.7:0/2381898020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0c00e0 0x7f4a2c154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.729+0000 7f4a31776700 1 -- 172.21.15.7:0/2381898020 >> 172.21.15.7:0/2381898020 conn(0x7f4a2c066e40 msgr2=0x7f4a2c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.729+0000 7f4a31776700 1 -- 172.21.15.7:0/2381898020 shutdown_connections 2023-03-03T08:44:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.729+0000 7f4a31776700 1 -- 172.21.15.7:0/2381898020 wait complete. 2023-03-03T08:44:03.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.730+0000 7f4a31776700 1 Processor -- start 2023-03-03T08:44:03.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.730+0000 7f4a31776700 1 -- start start 2023-03-03T08:44:03.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a31776700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a2c0c00e0 0x7f4a2c0c1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a31776700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 0x7f4a2c149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a31776700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0cae00 0x7f4a2c1492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a31776700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c14da90 con 0x7f4a2c0caa30 2023-03-03T08:44:03.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a31776700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c14b230 con 0x7f4a2c0c00e0 2023-03-03T08:44:03.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a31776700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c14d2b0 con 0x7f4a2c0cae00 2023-03-03T08:44:03.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 0x7f4a2c149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 0x7f4a2c149f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52098/0 (socket says 172.21.15.7:52098) 2023-03-03T08:44:03.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2a7fc700 1 -- 172.21.15.7:0/556922382 learned_addr learned my addr 172.21.15.7:0/556922382 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:03.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2affd700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a2c0c00e0 0x7f4a2c0c1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2b7fe700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0cae00 0x7f4a2c1492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2a7fc700 1 -- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0cae00 msgr2=0x7f4a2c1492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2a7fc700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0cae00 0x7f4a2c1492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2a7fc700 1 -- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a2c0c00e0 msgr2=0x7f4a2c0c1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2a7fc700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a2c0c00e0 0x7f4a2c0c1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2a7fc700 1 -- 172.21.15.7:0/556922382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c009550 con 0x7f4a2c0caa30 2023-03-03T08:44:03.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2b7fe700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0cae00 0x7f4a2c1492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:03.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.731+0000 7f4a2a7fc700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 0x7f4a2c149f90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000c7f0 tx=0x7f4a2000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:03.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.732+0000 7f4a137fe700 1 -- 172.21.15.7:0/556922382 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a20004220 con 0x7f4a2c0caa30 2023-03-03T08:44:03.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.732+0000 7f4a137fe700 1 -- 172.21.15.7:0/556922382 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a20004380 con 0x7f4a2c0caa30 2023-03-03T08:44:03.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.732+0000 7f4a31776700 1 -- 172.21.15.7:0/556922382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c1547f0 con 0x7f4a2c0caa30 2023-03-03T08:44:03.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.732+0000 7f4a137fe700 1 -- 172.21.15.7:0/556922382 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a200036a0 con 0x7f4a2c0caa30 2023-03-03T08:44:03.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.732+0000 7f4a31776700 1 -- 172.21.15.7:0/556922382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c154510 con 0x7f4a2c0caa30 2023-03-03T08:44:03.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.733+0000 7f4a31776700 1 -- 172.21.15.7:0/556922382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c154ad0 con 0x7f4a2c0caa30 2023-03-03T08:44:03.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.733+0000 7f4a137fe700 1 -- 172.21.15.7:0/556922382 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4a20020070 con 0x7f4a2c0caa30 2023-03-03T08:44:03.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.734+0000 7f4a137fe700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a180682f0 0x7f4a1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:03.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.734+0000 7f4a137fe700 1 -- 172.21.15.7:0/556922382 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f4a20085eb0 con 0x7f4a2c0caa30 2023-03-03T08:44:03.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.734+0000 7f4a2affd700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a180682f0 0x7f4a1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:03.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.735+0000 7f4a2affd700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a180682f0 0x7f4a1806a7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009130 tx=0x7f4a1c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:03.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.737+0000 7f4a137fe700 1 -- 172.21.15.7:0/556922382 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4a200523e0 con 0x7f4a2c0caa30 2023-03-03T08:44:03.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.930+0000 7f4a31776700 1 -- 172.21.15.7:0/556922382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f4a2c14dbd0 con 0x7f4a2c0caa30 2023-03-03T08:44:03.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.930+0000 7f4a137fe700 1 -- 172.21.15.7:0/556922382 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7f4a2c14dbd0 con 0x7f4a2c0caa30 2023-03-03T08:44:03.932 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:44:03.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 -- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a180682f0 msgr2=0x7f4a1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a180682f0 0x7f4a1806a7a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009130 tx=0x7f4a1c008030 comp rx=0 tx=0).stop 2023-03-03T08:44:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 -- 172.21.15.7:0/556922382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 msgr2=0x7f4a2c149f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 0x7f4a2c149f90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000c7f0 tx=0x7f4a2000cb00 comp rx=0 tx=0).stop 2023-03-03T08:44:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 -- 172.21.15.7:0/556922382 shutdown_connections 2023-03-03T08:44:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a2c0cae00 0x7f4a2c1492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a2c0c00e0 0x7f4a2c0c1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a180682f0 0x7f4a1806a7a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 --2- 172.21.15.7:0/556922382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a2c0caa30 0x7f4a2c149f90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 -- 172.21.15.7:0/556922382 >> 172.21.15.7:0/556922382 conn(0x7f4a2c066e40 msgr2=0x7f4a2c0c6950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 -- 172.21.15.7:0/556922382 shutdown_connections 2023-03-03T08:44:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:03.933+0000 7f4a31776700 1 -- 172.21.15.7:0/556922382 wait complete. 2023-03-03T08:44:03.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-03-03T08:44:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.072+0000 7f96af067700 1 Processor -- start 2023-03-03T08:44:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.072+0000 7f96af067700 1 -- start start 2023-03-03T08:44:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96af067700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f96a814b460 0x7f96a806aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96af067700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f96a8072470 0x7f96a806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96af067700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a8070360 con 0x7f96a8071990 2023-03-03T08:44:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96af067700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96a81521a0 con 0x7f96a814b460 2023-03-03T08:44:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96af067700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96a806a5e0 con 0x7f96a8072470 2023-03-03T08:44:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96a7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f96a814b460 0x7f96a806aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96a7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f96a814b460 0x7f96a806aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52112/0 (socket says 172.21.15.7:52112) 2023-03-03T08:44:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96a7fff700 1 -- 172.21.15.7:0/3266451079 learned_addr learned my addr 172.21.15.7:0/3266451079 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96ad604700 1 --2- 172.21.15.7:0/3266451079 >> v2:172.21.15.143:3300/0 conn(0x7f96a8072470 0x7f96a806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96a7fff700 1 -- 172.21.15.7:0/3266451079 >> v2:172.21.15.143:3300/0 conn(0x7f96a8072470 msgr2=0x7f96a806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96a7fff700 1 --2- 172.21.15.7:0/3266451079 >> v2:172.21.15.143:3300/0 conn(0x7f96a8072470 0x7f96a806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96a7fff700 1 -- 172.21.15.7:0/3266451079 >> v1:172.21.15.143:6789/0 conn(0x7f96a8071990 legacy=0x7f96a8071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.073+0000 7f96a7fff700 1 -- 172.21.15.7:0/3266451079 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a80c38b0 con 0x7f96a814b460 2023-03-03T08:44:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96a7fff700 1 --2- 172.21.15.7:0/3266451079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a814b460 0x7f96a806aa70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f96a8070020 tx=0x7f9694009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae18a2faee884d53 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96a77fe700 1 -- 172.21.15.7:0/3266451079 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969400b870 con 0x7f96a814b460 2023-03-03T08:44:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96a77fe700 1 -- 172.21.15.7:0/3266451079 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969400b9d0 con 0x7f96a814b460 2023-03-03T08:44:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96a77fe700 1 -- 172.21.15.7:0/3266451079 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969400bd60 con 0x7f96a814b460 2023-03-03T08:44:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96af067700 1 -- 172.21.15.7:0/3266451079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a814b460 msgr2=0x7f96a806aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96af067700 1 --2- 172.21.15.7:0/3266451079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a814b460 0x7f96a806aa70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f96a8070020 tx=0x7f9694009bd0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96af067700 1 -- 172.21.15.7:0/3266451079 shutdown_connections 2023-03-03T08:44:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96af067700 1 --2- 172.21.15.7:0/3266451079 >> v2:172.21.15.143:3300/0 conn(0x7f96a8072470 0x7f96a806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96af067700 1 --2- 172.21.15.7:0/3266451079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a814b460 0x7f96a806aa70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96af067700 1 -- 172.21.15.7:0/3266451079 >> 172.21.15.7:0/3266451079 conn(0x7f96a80a94d0 msgr2=0x7f96a80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:04.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96af067700 1 -- 172.21.15.7:0/3266451079 shutdown_connections 2023-03-03T08:44:04.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.074+0000 7f96af067700 1 -- 172.21.15.7:0/3266451079 wait complete. 2023-03-03T08:44:04.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.076+0000 7f96af067700 1 Processor -- start 2023-03-03T08:44:04.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.076+0000 7f96af067700 1 -- start start 2023-03-03T08:44:04.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.076+0000 7f96af067700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a8071990 0x7f96a8146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.076+0000 7f96af067700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96a8072470 0x7f96a806c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.076+0000 7f96af067700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96a80be6f0 0x7f96a80ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.076+0000 7f96af067700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a8146fd0 con 0x7f96a8071990 2023-03-03T08:44:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.076+0000 7f96af067700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a81ee0f0 con 0x7f96a80be6f0 2023-03-03T08:44:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.076+0000 7f96af067700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a814ea80 con 0x7f96a8072470 2023-03-03T08:44:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ace03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a8071990 0x7f96a8146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ace03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a8071990 0x7f96a8146a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52116/0 (socket says 172.21.15.7:52116) 2023-03-03T08:44:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ace03700 1 -- 172.21.15.7:0/3683563608 learned_addr learned my addr 172.21.15.7:0/3683563608 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:04.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ace03700 1 -- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96a8072470 msgr2=0x7f96a806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ad604700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96a80be6f0 0x7f96a80ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96a7fff700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96a8072470 0x7f96a806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ace03700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96a8072470 0x7f96a806c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ace03700 1 -- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96a80be6f0 msgr2=0x7f96a80ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ace03700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96a80be6f0 0x7f96a80ab840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ace03700 1 -- 172.21.15.7:0/3683563608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9694009550 con 0x7f96a8071990 2023-03-03T08:44:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ad604700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96a80be6f0 0x7f96a80ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96ace03700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a8071990 0x7f96a8146a90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f969c00c4e0 tx=0x7f969c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96a57fa700 1 -- 172.21.15.7:0/3683563608 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c0052d0 con 0x7f96a8071990 2023-03-03T08:44:04.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96a57fa700 1 -- 172.21.15.7:0/3683563608 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c005430 con 0x7f96a8071990 2023-03-03T08:44:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96af067700 1 -- 172.21.15.7:0/3683563608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a80c3b50 con 0x7f96a8071990 2023-03-03T08:44:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96a57fa700 1 -- 172.21.15.7:0/3683563608 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c0038c0 con 0x7f96a8071990 2023-03-03T08:44:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.077+0000 7f96af067700 1 -- 172.21.15.7:0/3683563608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a806a2b0 con 0x7f96a8071990 2023-03-03T08:44:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.078+0000 7f96af067700 1 -- 172.21.15.7:0/3683563608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a806b4f0 con 0x7f96a8071990 2023-03-03T08:44:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.079+0000 7f96a57fa700 1 -- 172.21.15.7:0/3683563608 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f969c019030 con 0x7f96a8071990 2023-03-03T08:44:04.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.079+0000 7f96a57fa700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f968c0682f0 0x7f968c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.079+0000 7f96a57fa700 1 -- 172.21.15.7:0/3683563608 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f969c0852f0 con 0x7f96a8071990 2023-03-03T08:44:04.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.079+0000 7f96a7fff700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f968c0682f0 0x7f968c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.080+0000 7f96a7fff700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f968c0682f0 0x7f968c06a7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f96a80ad550 tx=0x7f9694008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:04.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.083+0000 7f96a57fa700 1 -- 172.21.15.7:0/3683563608 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f969c051820 con 0x7f96a8071990 2023-03-03T08:44:04.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.275+0000 7f96af067700 1 -- 172.21.15.7:0/3683563608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96a80ab560 con 0x7f96a8071990 2023-03-03T08:44:04.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.275+0000 7f96a57fa700 1 -- 172.21.15.7:0/3683563608 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f969c054e40 con 0x7f96a8071990 2023-03-03T08:44:04.277 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:44:04.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 -- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f968c0682f0 msgr2=0x7f968c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f968c0682f0 0x7f968c06a7a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f96a80ad550 tx=0x7f9694008030 comp rx=0 tx=0).stop 2023-03-03T08:44:04.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 -- 172.21.15.7:0/3683563608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a8071990 msgr2=0x7f96a8146a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a8071990 0x7f96a8146a90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f969c00c4e0 tx=0x7f969c00c8a0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 -- 172.21.15.7:0/3683563608 shutdown_connections 2023-03-03T08:44:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96a80be6f0 0x7f96a80ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f968c0682f0 0x7f968c06a7a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96a8071990 0x7f96a8146a90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 --2- 172.21.15.7:0/3683563608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96a8072470 0x7f96a806c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 -- 172.21.15.7:0/3683563608 >> 172.21.15.7:0/3683563608 conn(0x7f96a80a94d0 msgr2=0x7f96a8151120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 -- 172.21.15.7:0/3683563608 shutdown_connections 2023-03-03T08:44:04.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.277+0000 7f96af067700 1 -- 172.21.15.7:0/3683563608 wait complete. 2023-03-03T08:44:04.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-03-03T08:44:04.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.417+0000 7f3850aba700 1 Processor -- start 2023-03-03T08:44:04.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.418+0000 7f3850aba700 1 -- start start 2023-03-03T08:44:04.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.418+0000 7f3850aba700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f384c1550c0 0x7f384c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.418+0000 7f3850aba700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c0c7a60 con 0x7f384c0c1da0 2023-03-03T08:44:04.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f3850aba700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384c0c3620 con 0x7f384c06d870 2023-03-03T08:44:04.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f3850aba700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384c1554c0 con 0x7f384c1550c0 2023-03-03T08:44:04.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f3849d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f384c0c1da0 0x7f384c0c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46104/0 (socket says 172.21.15.7:46104) 2023-03-03T08:44:04.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f3849d9b700 1 -- 172.21.15.7:0/28675891 learned_addr learned my addr 172.21.15.7:0/28675891 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:04.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384ad9d700 1 --2- 172.21.15.7:0/28675891 >> v2:172.21.15.143:3300/0 conn(0x7f384c1550c0 0x7f384c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842309684 0 0) 0x7f384c0c7a60 con 0x7f384c0c1da0 2023-03-03T08:44:04.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38340035b0 con 0x7f384c0c1da0 2023-03-03T08:44:04.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430274156 0 0) 0x7f384c0c3620 con 0x7f384c06d870 2023-03-03T08:44:04.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f384c0c7a60 con 0x7f384c06d870 2023-03-03T08:44:04.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1704688313 0 0) 0x7f38340035b0 con 0x7f384c0c1da0 2023-03-03T08:44:04.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 >> v2:172.21.15.143:3300/0 conn(0x7f384c1550c0 msgr2=0x7f384c15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:04.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 --2- 172.21.15.7:0/28675891 >> v2:172.21.15.143:3300/0 conn(0x7f384c1550c0 0x7f384c15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 >> v1:172.21.15.143:6789/0 conn(0x7f384c06d870 legacy=0x7f384c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c1545e0 con 0x7f384c0c1da0 2023-03-03T08:44:04.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f3840001670 con 0x7f384c0c1da0 2023-03-03T08:44:04.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384ad9d700 1 --2- 172.21.15.7:0/28675891 >> v2:172.21.15.143:3300/0 conn(0x7f384c1550c0 0x7f384c15fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:04.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 >> v1:172.21.15.7:6789/0 conn(0x7f384c0c1da0 legacy=0x7f384c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3834003c00 0x7f3834003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f3834004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3834005200 0x7f383400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384c1554c0 con 0x7f3834004850 2023-03-03T08:44:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383400c0e0 con 0x7f3834003c00 2023-03-03T08:44:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383400c2a0 con 0x7f3834005200 2023-03-03T08:44:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.419+0000 7f3849d9b700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f3834004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f384a59c700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3834003c00 0x7f3834003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f384ad9d700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3834005200 0x7f383400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f3849d9b700 1 -- 172.21.15.7:0/28675891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3834005200 msgr2=0x7f383400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f3849d9b700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3834005200 0x7f383400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f3849d9b700 1 -- 172.21.15.7:0/28675891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3834003c00 msgr2=0x7f3834003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f3849d9b700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3834003c00 0x7f3834003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f384ad9d700 1 -- 172.21.15.7:0/28675891 reap_dead start 2023-03-03T08:44:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f3849d9b700 1 -- 172.21.15.7:0/28675891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38340035b0 con 0x7f3834004850 2023-03-03T08:44:04.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f3849d9b700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f3834004cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3840001eb0 tx=0x7f3834005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c745287c010e77 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384000cce0 con 0x7f3834004850 2023-03-03T08:44:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384000ce60 con 0x7f3834004850 2023-03-03T08:44:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.420+0000 7f384959a700 1 -- 172.21.15.7:0/28675891 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840005ba0 con 0x7f3834004850 2023-03-03T08:44:04.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.421+0000 7f3850aba700 1 -- 172.21.15.7:0/28675891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 msgr2=0x7f3834004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.421+0000 7f3850aba700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f3834004cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3840001eb0 tx=0x7f3834005880 comp rx=0 tx=0).stop 2023-03-03T08:44:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.421+0000 7f3850aba700 1 -- 172.21.15.7:0/28675891 shutdown_connections 2023-03-03T08:44:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.421+0000 7f3850aba700 1 --2- 172.21.15.7:0/28675891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f3834004cc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.421+0000 7f3850aba700 1 -- 172.21.15.7:0/28675891 >> 172.21.15.7:0/28675891 conn(0x7f384c06c110 msgr2=0x7f384c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.421+0000 7f3850aba700 1 -- 172.21.15.7:0/28675891 shutdown_connections 2023-03-03T08:44:04.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.421+0000 7f3850aba700 1 -- 172.21.15.7:0/28675891 wait complete. 2023-03-03T08:44:04.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.422+0000 7f3850aba700 1 Processor -- start 2023-03-03T08:44:04.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.423+0000 7f3850aba700 1 -- start start 2023-03-03T08:44:04.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.423+0000 7f3850aba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f384c0aa630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.423+0000 7f3850aba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f384c070720 0x7f384c0aab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.423+0000 7f3850aba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384c13cc60 0x7f384c13d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.423+0000 7f3850aba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384c0ab430 con 0x7f3834004850 2023-03-03T08:44:04.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.423+0000 7f3850aba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384c0ac8e0 con 0x7f384c13cc60 2023-03-03T08:44:04.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.423+0000 7f3850aba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384c1ea160 con 0x7f384c070720 2023-03-03T08:44:04.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f384a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f384c0aa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f384a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f384c0aa630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52136/0 (socket says 172.21.15.7:52136) 2023-03-03T08:44:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f384a59c700 1 -- 172.21.15.7:0/124203485 learned_addr learned my addr 172.21.15.7:0/124203485 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f384a59c700 1 -- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f384c070720 msgr2=0x7f384c0aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f384a59c700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f384c070720 0x7f384c0aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f384ad9d700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384c13cc60 0x7f384c13d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f384a59c700 1 -- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384c13cc60 msgr2=0x7f384c13d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f384a59c700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384c13cc60 0x7f384c13d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f384a59c700 1 -- 172.21.15.7:0/124203485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c154ba0 con 0x7f3834004850 2023-03-03T08:44:04.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f384a59c700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f384c0aa630 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f383c00f770 tx=0x7f383c00fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:04.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f383affd700 1 -- 172.21.15.7:0/124203485 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c005320 con 0x7f3834004850 2023-03-03T08:44:04.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f383affd700 1 -- 172.21.15.7:0/124203485 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383c0054a0 con 0x7f3834004850 2023-03-03T08:44:04.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f3850aba700 1 -- 172.21.15.7:0/124203485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384c1548c0 con 0x7f3834004850 2023-03-03T08:44:04.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f383affd700 1 -- 172.21.15.7:0/124203485 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383c006aa0 con 0x7f3834004850 2023-03-03T08:44:04.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.424+0000 7f3850aba700 1 -- 172.21.15.7:0/124203485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384c06dc40 con 0x7f3834004850 2023-03-03T08:44:04.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.425+0000 7f3850aba700 1 -- 172.21.15.7:0/124203485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384c1ea5f0 con 0x7f3834004850 2023-03-03T08:44:04.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.426+0000 7f383affd700 1 -- 172.21.15.7:0/124203485 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f383c01c030 con 0x7f3834004850 2023-03-03T08:44:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.426+0000 7f383affd700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f382c0682f0 0x7f382c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.426+0000 7f383affd700 1 -- 172.21.15.7:0/124203485 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f383c085710 con 0x7f3834004850 2023-03-03T08:44:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.426+0000 7f3849d9b700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f382c0682f0 0x7f382c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.427+0000 7f3849d9b700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f382c0682f0 0x7f382c06a7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f384c158ac0 tx=0x7f3840009050 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.431+0000 7f383affd700 1 -- 172.21.15.7:0/124203485 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f383c051c40 con 0x7f3834004850 2023-03-03T08:44:04.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.621+0000 7f3850aba700 1 -- 172.21.15.7:0/124203485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f384c074bb0 con 0x7f3834004850 2023-03-03T08:44:04.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.622+0000 7f383affd700 1 -- 172.21.15.7:0/124203485 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7f383c055260 con 0x7f3834004850 2023-03-03T08:44:04.624 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:44:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 -- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f382c0682f0 msgr2=0x7f382c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f382c0682f0 0x7f382c06a7a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f384c158ac0 tx=0x7f3840009050 comp rx=0 tx=0).stop 2023-03-03T08:44:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 -- 172.21.15.7:0/124203485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 msgr2=0x7f384c0aa630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f384c0aa630 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f383c00f770 tx=0x7f383c00fb30 comp rx=0 tx=0).stop 2023-03-03T08:44:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 -- 172.21.15.7:0/124203485 shutdown_connections 2023-03-03T08:44:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f384c13cc60 0x7f384c13d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f382c0682f0 0x7f382c06a7a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834004850 0x7f384c0aa630 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 --2- 172.21.15.7:0/124203485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f384c070720 0x7f384c0aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 -- 172.21.15.7:0/124203485 >> 172.21.15.7:0/124203485 conn(0x7f384c06c110 msgr2=0x7f384c071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 -- 172.21.15.7:0/124203485 shutdown_connections 2023-03-03T08:44:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.624+0000 7f3850aba700 1 -- 172.21.15.7:0/124203485 wait complete. 2023-03-03T08:44:04.637 INFO:tasks.mds_thrash:mds thrasher using random seed: 1677833044 2023-03-03T08:44:04.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:44:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.760+0000 7fda8ef5c700 1 Processor -- start 2023-03-03T08:44:04.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.760+0000 7fda8ef5c700 1 -- start start 2023-03-03T08:44:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.761+0000 7fda8ef5c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda8814de50 0x7fda8806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.761+0000 7fda8ef5c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda880a65b0 con 0x7fda8806a7f0 2023-03-03T08:44:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.761+0000 7fda8ef5c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda881541f0 con 0x7fda880c65c0 2023-03-03T08:44:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.761+0000 7fda8ef5c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda88154e00 con 0x7fda8814de50 2023-03-03T08:44:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.761+0000 7fda8d4f9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fda8806a7f0 0x7fda880c4dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46124/0 (socket says 172.21.15.7:46124) 2023-03-03T08:44:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.761+0000 7fda8d4f9700 1 -- 172.21.15.7:0/707738861 learned_addr learned my addr 172.21.15.7:0/707738861 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.761+0000 7fda8ccf8700 1 --2- 172.21.15.7:0/707738861 >> v2:172.21.15.143:3300/0 conn(0x7fda8814de50 0x7fda8806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.761+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285684593 0 0) 0x7fda880a65b0 con 0x7fda8806a7f0 2023-03-03T08:44:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.761+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda780035b0 con 0x7fda8806a7f0 2023-03-03T08:44:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 314944761 0 0) 0x7fda780035b0 con 0x7fda8806a7f0 2023-03-03T08:44:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 >> v2:172.21.15.143:3300/0 conn(0x7fda8814de50 msgr2=0x7fda8806a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 --2- 172.21.15.7:0/707738861 >> v2:172.21.15.143:3300/0 conn(0x7fda8814de50 0x7fda8806a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 >> v1:172.21.15.143:6789/0 conn(0x7fda880c65c0 legacy=0x7fda880c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda881544d0 con 0x7fda8806a7f0 2023-03-03T08:44:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fda800019f0 con 0x7fda8806a7f0 2023-03-03T08:44:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 >> v1:172.21.15.7:6789/0 conn(0x7fda8806a7f0 legacy=0x7fda880c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda78003a70 0x7fda78003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda78004850 0x7fda78004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda78005250 0x7fda7800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda88154e00 con 0x7fda78004850 2023-03-03T08:44:04.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7800bf50 con 0x7fda78003a70 2023-03-03T08:44:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7800c110 con 0x7fda78005250 2023-03-03T08:44:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda87fff700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda78004850 0x7fda78004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda8ccf8700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda78003a70 0x7fda78003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda8d4f9700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda78005250 0x7fda7800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda87fff700 1 -- 172.21.15.7:0/707738861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda78005250 msgr2=0x7fda7800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda87fff700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda78005250 0x7fda7800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda87fff700 1 -- 172.21.15.7:0/707738861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda78003a70 msgr2=0x7fda78003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:04.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda87fff700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda78003a70 0x7fda78003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda87fff700 1 -- 172.21.15.7:0/707738861 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda780035b0 con 0x7fda78004850 2023-03-03T08:44:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda8d4f9700 1 -- 172.21.15.7:0/707738861 reap_dead start 2023-03-03T08:44:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda87fff700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda78004850 0x7fda78004cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fda74007fd0 tx=0x7fda78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f518c652891d1d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda740052d0 con 0x7fda78004850 2023-03-03T08:44:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda74005430 con 0x7fda78004850 2023-03-03T08:44:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.762+0000 7fda877fe700 1 -- 172.21.15.7:0/707738861 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda740038c0 con 0x7fda78004850 2023-03-03T08:44:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.763+0000 7fda8ef5c700 1 -- 172.21.15.7:0/707738861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda78004850 msgr2=0x7fda78004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.763+0000 7fda8ef5c700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda78004850 0x7fda78004cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fda74007fd0 tx=0x7fda78005880 comp rx=0 tx=0).stop 2023-03-03T08:44:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.763+0000 7fda8ef5c700 1 -- 172.21.15.7:0/707738861 shutdown_connections 2023-03-03T08:44:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.763+0000 7fda8ef5c700 1 --2- 172.21.15.7:0/707738861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda78004850 0x7fda78004cc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.763+0000 7fda8ef5c700 1 -- 172.21.15.7:0/707738861 >> 172.21.15.7:0/707738861 conn(0x7fda880a9430 msgr2=0x7fda880c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.763+0000 7fda8ef5c700 1 -- 172.21.15.7:0/707738861 shutdown_connections 2023-03-03T08:44:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.763+0000 7fda8ef5c700 1 -- 172.21.15.7:0/707738861 wait complete. 2023-03-03T08:44:04.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.764+0000 7fda8ef5c700 1 Processor -- start 2023-03-03T08:44:04.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.764+0000 7fda8ef5c700 1 -- start start 2023-03-03T08:44:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8ef5c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda78004850 0x7fda881e8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8ef5c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda8814a3b0 0x7fda881491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8ef5c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda88149720 0x7fda88071a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8ef5c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8814ec90 con 0x7fda88149720 2023-03-03T08:44:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8ef5c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8814ef70 con 0x7fda78004850 2023-03-03T08:44:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8ef5c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8814a160 con 0x7fda8814a3b0 2023-03-03T08:44:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8d4f9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda88149720 0x7fda88071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8d4f9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda88149720 0x7fda88071a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52156/0 (socket says 172.21.15.7:52156) 2023-03-03T08:44:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8d4f9700 1 -- 172.21.15.7:0/2878155339 learned_addr learned my addr 172.21.15.7:0/2878155339 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8ccf8700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda78004850 0x7fda881e8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda87fff700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda8814a3b0 0x7fda881491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8d4f9700 1 -- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda8814a3b0 msgr2=0x7fda881491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8d4f9700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda8814a3b0 0x7fda881491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8d4f9700 1 -- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda78004850 msgr2=0x7fda881e8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8d4f9700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda78004850 0x7fda881e8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8d4f9700 1 -- 172.21.15.7:0/2878155339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda88154a90 con 0x7fda88149720 2023-03-03T08:44:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.765+0000 7fda8ccf8700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda78004850 0x7fda881e8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.766+0000 7fda8d4f9700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda88149720 0x7fda88071a20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fda880c7800 tx=0x7fda8000e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.766+0000 7fda857fa700 1 -- 172.21.15.7:0/2878155339 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8000ec40 con 0x7fda88149720 2023-03-03T08:44:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.766+0000 7fda857fa700 1 -- 172.21.15.7:0/2878155339 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8000eda0 con 0x7fda88149720 2023-03-03T08:44:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.766+0000 7fda8ef5c700 1 -- 172.21.15.7:0/2878155339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda881547b0 con 0x7fda88149720 2023-03-03T08:44:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.766+0000 7fda857fa700 1 -- 172.21.15.7:0/2878155339 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda80013660 con 0x7fda88149720 2023-03-03T08:44:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.766+0000 7fda8ef5c700 1 -- 172.21.15.7:0/2878155339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda88153e80 con 0x7fda88149720 2023-03-03T08:44:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.767+0000 7fda8ef5c700 1 -- 172.21.15.7:0/2878155339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda88074bb0 con 0x7fda88149720 2023-03-03T08:44:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.767+0000 7fda857fa700 1 -- 172.21.15.7:0/2878155339 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda800137c0 con 0x7fda88149720 2023-03-03T08:44:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.767+0000 7fda857fa700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda6c0682f0 0x7fda6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.768+0000 7fda857fa700 1 -- 172.21.15.7:0/2878155339 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fda800867b0 con 0x7fda88149720 2023-03-03T08:44:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.768+0000 7fda8ccf8700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda6c0682f0 0x7fda6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:04.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.768+0000 7fda8ccf8700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda6c0682f0 0x7fda6c06a7a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fda8814ea10 tx=0x7fda7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:04.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.772+0000 7fda857fa700 1 -- 172.21.15.7:0/2878155339 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda80052b70 con 0x7fda88149720 2023-03-03T08:44:04.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.965+0000 7fda8ef5c700 1 -- 172.21.15.7:0/2878155339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda8814f0b0 con 0x7fda88149720 2023-03-03T08:44:04.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.967+0000 7fda857fa700 1 -- 172.21.15.7:0/2878155339 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21533 (secure 0 0 0) 0x7fda8814f0b0 con 0x7fda88149720 2023-03-03T08:44:04.969 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:04.969 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:44.973251+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:44:04.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 -- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda6c0682f0 msgr2=0x7fda6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda6c0682f0 0x7fda6c06a7a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fda8814ea10 tx=0x7fda7c008040 comp rx=0 tx=0).stop 2023-03-03T08:44:04.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 -- 172.21.15.7:0/2878155339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda88149720 msgr2=0x7fda88071a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:04.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda88149720 0x7fda88071a20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fda880c7800 tx=0x7fda8000e4f0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 -- 172.21.15.7:0/2878155339 shutdown_connections 2023-03-03T08:44:04.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda6c0682f0 0x7fda6c06a7a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda88149720 0x7fda88071a20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda78004850 0x7fda881e8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 --2- 172.21.15.7:0/2878155339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda8814a3b0 0x7fda881491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:04.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 -- 172.21.15.7:0/2878155339 >> 172.21.15.7:0/2878155339 conn(0x7fda880a9430 msgr2=0x7fda88151c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:04.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.969+0000 7fda8ef5c700 1 -- 172.21.15.7:0/2878155339 shutdown_connections 2023-03-03T08:44:04.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:04.970+0000 7fda8ef5c700 1 -- 172.21.15.7:0/2878155339 wait complete. 2023-03-03T08:44:04.983 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T08:43:21.160680+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T08:43:28.018457+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-03-03T08:43:32.305037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-03-03T08:43:33.484501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-03-03T08:43:42.509451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-03-03T08:43:42.936898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:44:04.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:44:05.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:05.103 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:05.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.112+0000 7fe6a2e63700 1 Processor -- start 2023-03-03T08:44:05.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.112+0000 7fe6a2e63700 1 -- start start 2023-03-03T08:44:05.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe6a2e63700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe69c0c17f0 0x7fe69c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe6a2e63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69c0c5190 0x7fe69c06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe6a2e63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe69c14ebe0 con 0x7fe69c0c6010 2023-03-03T08:44:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe6a2e63700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe69c152100 con 0x7fe69c0c17f0 2023-03-03T08:44:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe6a2e63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c0c2e70 con 0x7fe69c0c5190 2023-03-03T08:44:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe69c0c17f0 0x7fe69c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe69c0c17f0 0x7fe69c06d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52162/0 (socket says 172.21.15.7:52162) 2023-03-03T08:44:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69bfff700 1 -- 172.21.15.7:0/25308669 learned_addr learned my addr 172.21.15.7:0/25308669 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69bfff700 1 -- 172.21.15.7:0/25308669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69c0c5190 msgr2=0x7fe69c06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69bfff700 1 --2- 172.21.15.7:0/25308669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69c0c5190 0x7fe69c06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69bfff700 1 -- 172.21.15.7:0/25308669 >> v1:172.21.15.143:6789/0 conn(0x7fe69c0c6010 legacy=0x7fe69c0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69bfff700 1 -- 172.21.15.7:0/25308669 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c0aa680 con 0x7fe69c0c17f0 2023-03-03T08:44:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69bfff700 1 --2- 172.21.15.7:0/25308669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 0x7fe69c06d6a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe6880098f0 tx=0x7fe688009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7469373b7daf295a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:05.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69b7fe700 1 -- 172.21.15.7:0/25308669 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe688004230 con 0x7fe69c0c17f0 2023-03-03T08:44:05.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69b7fe700 1 -- 172.21.15.7:0/25308669 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe688004390 con 0x7fe69c0c17f0 2023-03-03T08:44:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.113+0000 7fe69b7fe700 1 -- 172.21.15.7:0/25308669 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6880036a0 con 0x7fe69c0c17f0 2023-03-03T08:44:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.114+0000 7fe6a2e63700 1 -- 172.21.15.7:0/25308669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 msgr2=0x7fe69c06d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.114+0000 7fe6a2e63700 1 --2- 172.21.15.7:0/25308669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 0x7fe69c06d6a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe6880098f0 tx=0x7fe688009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.114+0000 7fe6a2e63700 1 -- 172.21.15.7:0/25308669 shutdown_connections 2023-03-03T08:44:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.114+0000 7fe6a2e63700 1 --2- 172.21.15.7:0/25308669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69c0c5190 0x7fe69c06dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.114+0000 7fe6a2e63700 1 --2- 172.21.15.7:0/25308669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 0x7fe69c06d6a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.114+0000 7fe6a2e63700 1 -- 172.21.15.7:0/25308669 >> 172.21.15.7:0/25308669 conn(0x7fe69c0a9430 msgr2=0x7fe69c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.114+0000 7fe6a2e63700 1 -- 172.21.15.7:0/25308669 shutdown_connections 2023-03-03T08:44:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.114+0000 7fe6a2e63700 1 -- 172.21.15.7:0/25308669 wait complete. 2023-03-03T08:44:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.116+0000 7fe6a2e63700 1 Processor -- start 2023-03-03T08:44:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.116+0000 7fe6a2e63700 1 -- start start 2023-03-03T08:44:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.116+0000 7fe6a2e63700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 0x7fe69c1dee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.116+0000 7fe6a2e63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69c0c5190 0x7fe69c1536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a2e63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69c0c6010 0x7fe69c153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a2e63700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c153290 con 0x7fe69c0c17f0 2023-03-03T08:44:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a2e63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c1e8850 con 0x7fe69c0c6010 2023-03-03T08:44:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a2e63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69c1da890 con 0x7fe69c0c5190 2023-03-03T08:44:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a0bff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 0x7fe69c1dee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a0bff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 0x7fe69c1dee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52166/0 (socket says 172.21.15.7:52166) 2023-03-03T08:44:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a0bff700 1 -- 172.21.15.7:0/463954915 learned_addr learned my addr 172.21.15.7:0/463954915 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a1400700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69c0c6010 0x7fe69c153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe69bfff700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69c0c5190 0x7fe69c1536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a0bff700 1 -- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69c0c5190 msgr2=0x7fe69c1536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a0bff700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69c0c5190 0x7fe69c1536e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a0bff700 1 -- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69c0c6010 msgr2=0x7fe69c153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a0bff700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69c0c6010 0x7fe69c153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a0bff700 1 -- 172.21.15.7:0/463954915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe688009550 con 0x7fe69c0c17f0 2023-03-03T08:44:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a0bff700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 0x7fe69c1dee00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe69000c4a0 tx=0x7fe69000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6997fa700 1 -- 172.21.15.7:0/463954915 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6900054e0 con 0x7fe69c0c17f0 2023-03-03T08:44:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6997fa700 1 -- 172.21.15.7:0/463954915 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe690003680 con 0x7fe69c0c17f0 2023-03-03T08:44:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6a2e63700 1 -- 172.21.15.7:0/463954915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe69c0c2b00 con 0x7fe69c0c17f0 2023-03-03T08:44:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.117+0000 7fe6997fa700 1 -- 172.21.15.7:0/463954915 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe690003a30 con 0x7fe69c0c17f0 2023-03-03T08:44:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.118+0000 7fe6a2e63700 1 -- 172.21.15.7:0/463954915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69c0aa960 con 0x7fe69c0c17f0 2023-03-03T08:44:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.119+0000 7fe6a2e63700 1 -- 172.21.15.7:0/463954915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69c0c3300 con 0x7fe69c0c17f0 2023-03-03T08:44:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.119+0000 7fe6997fa700 1 -- 172.21.15.7:0/463954915 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe690019030 con 0x7fe69c0c17f0 2023-03-03T08:44:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.120+0000 7fe6997fa700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6800682f0 0x7fe68006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.120+0000 7fe6997fa700 1 -- 172.21.15.7:0/463954915 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fe690085300 con 0x7fe69c0c17f0 2023-03-03T08:44:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.120+0000 7fe69bfff700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6800682f0 0x7fe68006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.120+0000 7fe69bfff700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6800682f0 0x7fe68006a7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe688009130 tx=0x7fe688008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.124+0000 7fe6997fa700 1 -- 172.21.15.7:0/463954915 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe6900517d0 con 0x7fe69c0c17f0 2023-03-03T08:44:05.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.318+0000 7fe6a2e63700 1 -- 172.21.15.7:0/463954915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe69c153530 con 0x7fe69c0c17f0 2023-03-03T08:44:05.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.318+0000 7fe6997fa700 1 -- 172.21.15.7:0/463954915 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe690017020 con 0x7fe69c0c17f0 2023-03-03T08:44:05.320 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-03-03T08:44:05.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 -- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6800682f0 msgr2=0x7fe68006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6800682f0 0x7fe68006a7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe688009130 tx=0x7fe688008030 comp rx=0 tx=0).stop 2023-03-03T08:44:05.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 -- 172.21.15.7:0/463954915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 msgr2=0x7fe69c1dee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 0x7fe69c1dee00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe69000c4a0 tx=0x7fe69000c7b0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 -- 172.21.15.7:0/463954915 shutdown_connections 2023-03-03T08:44:05.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69c0c6010 0x7fe69c153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6800682f0 0x7fe68006a7a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe69c0c17f0 0x7fe69c1dee00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 --2- 172.21.15.7:0/463954915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe69c0c5190 0x7fe69c1536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 -- 172.21.15.7:0/463954915 >> 172.21.15.7:0/463954915 conn(0x7fe69c0a9430 msgr2=0x7fe69c150250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:05.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 -- 172.21.15.7:0/463954915 shutdown_connections 2023-03-03T08:44:05.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.320+0000 7fe6a2e63700 1 -- 172.21.15.7:0/463954915 wait complete. 2023-03-03T08:44:05.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:44:05.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.461+0000 7f342b9cb700 1 Processor -- start 2023-03-03T08:44:05.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.461+0000 7f342b9cb700 1 -- start start 2023-03-03T08:44:05.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.462+0000 7f342b9cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3424151ab0 0x7f34240736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.462+0000 7f342b9cb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3424073c30 0x7f342406bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.462+0000 7f342b9cb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f342406d780 con 0x7f342406c3e0 2023-03-03T08:44:05.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.462+0000 7f342b9cb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f342406cb70 con 0x7f3424073c30 2023-03-03T08:44:05.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.462+0000 7f342b9cb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342406ce50 con 0x7f3424151ab0 2023-03-03T08:44:05.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.462+0000 7f3429f68700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f342406c3e0 0x7f3424157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46158/0 (socket says 172.21.15.7:46158) 2023-03-03T08:44:05.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.462+0000 7f3429f68700 1 -- 172.21.15.7:0/1155738869 learned_addr learned my addr 172.21.15.7:0/1155738869 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:05.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f3428f66700 1 --2- 172.21.15.7:0/1155738869 >> v2:172.21.15.143:3300/0 conn(0x7f3424073c30 0x7f342406bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f3429767700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3424151ab0 0x7f34240736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404239136 0 0) 0x7f342406d780 con 0x7f342406c3e0 2023-03-03T08:44:05.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34100035b0 con 0x7f342406c3e0 2023-03-03T08:44:05.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1788708061 0 0) 0x7f34100035b0 con 0x7f342406c3e0 2023-03-03T08:44:05.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3424151ab0 msgr2=0x7f34240736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:05.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3424151ab0 0x7f34240736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 >> v2:172.21.15.143:3300/0 conn(0x7f3424073c30 msgr2=0x7f342406bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:05.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 --2- 172.21.15.7:0/1155738869 >> v2:172.21.15.143:3300/0 conn(0x7f3424073c30 0x7f342406bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342406d130 con 0x7f342406c3e0 2023-03-03T08:44:05.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f341c0019f0 con 0x7f342406c3e0 2023-03-03T08:44:05.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f3428f66700 1 --2- 172.21.15.7:0/1155738869 >> v2:172.21.15.143:3300/0 conn(0x7f3424073c30 0x7f342406bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:05.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 >> v1:172.21.15.7:6789/0 conn(0x7f342406c3e0 legacy=0x7f3424157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3424151ab0 0x7f3410003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 0x7f3410004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3410004d90 0x7f3410005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342406cb70 con 0x7f3410003ff0 2023-03-03T08:44:05.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342406ce50 con 0x7f3410004d90 2023-03-03T08:44:05.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341000bac0 con 0x7f3424151ab0 2023-03-03T08:44:05.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f3428f66700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 0x7f3410004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f3429767700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3424151ab0 0x7f3410003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f3429f68700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3410004d90 0x7f3410005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f3428f66700 1 -- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3424151ab0 msgr2=0x7f3410003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f3428f66700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3424151ab0 0x7f3410003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f3428f66700 1 -- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3410004d90 msgr2=0x7f3410005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.463+0000 7f3428f66700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3410004d90 0x7f3410005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f3428f66700 1 -- 172.21.15.7:0/1155738869 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34100035b0 con 0x7f3410003ff0 2023-03-03T08:44:05.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f3428f66700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 0x7f3410004850 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3420009d00 tx=0x7f3410005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584a8f7a2cfa8a13 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:05.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420009e50 con 0x7f3410003ff0 2023-03-03T08:44:05.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342000e9a0 con 0x7f3410003ff0 2023-03-03T08:44:05.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f341bfff700 1 -- 172.21.15.7:0/1155738869 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342000ed50 con 0x7f3410003ff0 2023-03-03T08:44:05.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f342b9cb700 1 -- 172.21.15.7:0/1155738869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 msgr2=0x7f3410004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f342b9cb700 1 --2- 172.21.15.7:0/1155738869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 0x7f3410004850 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3420009d00 tx=0x7f3410005880 comp rx=0 tx=0).stop 2023-03-03T08:44:05.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f3429f68700 1 -- 172.21.15.7:0/1155738869 reap_dead start 2023-03-03T08:44:05.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f342b9cb700 1 -- 172.21.15.7:0/1155738869 shutdown_connections 2023-03-03T08:44:05.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f342b9cb700 1 -- 172.21.15.7:0/1155738869 >> 172.21.15.7:0/1155738869 conn(0x7f34240a94d0 msgr2=0x7f34240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:05.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.464+0000 7f342b9cb700 1 -- 172.21.15.7:0/1155738869 shutdown_connections 2023-03-03T08:44:05.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.465+0000 7f342b9cb700 1 -- 172.21.15.7:0/1155738869 wait complete. 2023-03-03T08:44:05.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.466+0000 7f342b9cb700 1 Processor -- start 2023-03-03T08:44:05.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.466+0000 7f342b9cb700 1 -- start start 2023-03-03T08:44:05.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f342b9cb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 0x7f34240c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f342b9cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34240ac9f0 0x7f342414c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f342b9cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f342414cb00 0x7f34241ebad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f342b9cb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34240bfa80 con 0x7f3410003ff0 2023-03-03T08:44:05.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f342b9cb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34240becc0 con 0x7f34240ac9f0 2023-03-03T08:44:05.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f342b9cb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34240befa0 con 0x7f342414cb00 2023-03-03T08:44:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f3429767700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 0x7f34240c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f3429767700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 0x7f34240c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52186/0 (socket says 172.21.15.7:52186) 2023-03-03T08:44:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f3429767700 1 -- 172.21.15.7:0/733280775 learned_addr learned my addr 172.21.15.7:0/733280775 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f3428f66700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34240ac9f0 0x7f342414c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f3429f68700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f342414cb00 0x7f34241ebad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f3429767700 1 -- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f342414cb00 msgr2=0x7f34241ebad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f3429767700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f342414cb00 0x7f34241ebad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f3429767700 1 -- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34240ac9f0 msgr2=0x7f342414c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.467+0000 7f3429767700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34240ac9f0 0x7f342414c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.468+0000 7f3429767700 1 -- 172.21.15.7:0/733280775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3424151e80 con 0x7f3410003ff0 2023-03-03T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.468+0000 7f3428f66700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34240ac9f0 0x7f342414c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.468+0000 7f3429767700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 0x7f34240c69c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3414011aa0 tx=0x7f3414011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.468+0000 7f3419ffb700 1 -- 172.21.15.7:0/733280775 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341400f980 con 0x7f3410003ff0 2023-03-03T08:44:05.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.468+0000 7f3419ffb700 1 -- 172.21.15.7:0/733280775 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341400fae0 con 0x7f3410003ff0 2023-03-03T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.468+0000 7f342b9cb700 1 -- 172.21.15.7:0/733280775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f342406d410 con 0x7f3410003ff0 2023-03-03T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.468+0000 7f3419ffb700 1 -- 172.21.15.7:0/733280775 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341400fe90 con 0x7f3410003ff0 2023-03-03T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.468+0000 7f342b9cb700 1 -- 172.21.15.7:0/733280775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3424152010 con 0x7f3410003ff0 2023-03-03T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.469+0000 7f342b9cb700 1 -- 172.21.15.7:0/733280775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34241ec7e0 con 0x7f3410003ff0 2023-03-03T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.469+0000 7f3419ffb700 1 -- 172.21.15.7:0/733280775 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3414014440 con 0x7f3410003ff0 2023-03-03T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.470+0000 7f3419ffb700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34080682f0 0x7f340806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.470+0000 7f3419ffb700 1 -- 172.21.15.7:0/733280775 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f3414086830 con 0x7f3410003ff0 2023-03-03T08:44:05.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.470+0000 7f3428f66700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34080682f0 0x7f340806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.471+0000 7f3428f66700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34080682f0 0x7f340806a7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3420000c00 tx=0x7f3420008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:05.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.474+0000 7f3419ffb700 1 -- 172.21.15.7:0/733280775 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f34241ec7e0 con 0x7f3410003ff0 2023-03-03T08:44:05.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.668+0000 7f342b9cb700 1 -- 172.21.15.7:0/733280775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f34241ec7e0 con 0x7f3410003ff0 2023-03-03T08:44:05.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.669+0000 7f3419ffb700 1 -- 172.21.15.7:0/733280775 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f34241ec7e0 con 0x7f3410003ff0 2023-03-03T08:44:05.670 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 8 2023-03-03T08:44:05.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 -- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34080682f0 msgr2=0x7f340806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34080682f0 0x7f340806a7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3420000c00 tx=0x7f3420008030 comp rx=0 tx=0).stop 2023-03-03T08:44:05.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 -- 172.21.15.7:0/733280775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 msgr2=0x7f34240c69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 0x7f34240c69c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3414011aa0 tx=0x7f3414011e60 comp rx=0 tx=0).stop 2023-03-03T08:44:05.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 -- 172.21.15.7:0/733280775 shutdown_connections 2023-03-03T08:44:05.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f342414cb00 0x7f34241ebad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34080682f0 0x7f340806a7a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3410003ff0 0x7f34240c69c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 --2- 172.21.15.7:0/733280775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34240ac9f0 0x7f342414c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 -- 172.21.15.7:0/733280775 >> 172.21.15.7:0/733280775 conn(0x7f34240a94d0 msgr2=0x7f3424156940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:05.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 -- 172.21.15.7:0/733280775 shutdown_connections 2023-03-03T08:44:05.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.671+0000 7f342b9cb700 1 -- 172.21.15.7:0/733280775 wait complete. 2023-03-03T08:44:05.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-03-03T08:44:05.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.810+0000 7f3755be7700 1 Processor -- start 2023-03-03T08:44:05.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.810+0000 7f3755be7700 1 -- start start 2023-03-03T08:44:05.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.811+0000 7f3755be7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f375014de50 0x7f375006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.811+0000 7f3755be7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f37500c65c0 0x7f37500c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.811+0000 7f3755be7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500a65b0 con 0x7f375006a7f0 2023-03-03T08:44:05.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.811+0000 7f3755be7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3750154e00 con 0x7f375014de50 2023-03-03T08:44:05.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.811+0000 7f3755be7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37500c3370 con 0x7f37500c65c0 2023-03-03T08:44:05.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.811+0000 7f374f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f375014de50 0x7f375006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.811+0000 7f374ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f375006a7f0 0x7f37500c2c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46178/0 (socket says 172.21.15.7:46178) 2023-03-03T08:44:05.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.811+0000 7f374ffff700 1 -- 172.21.15.7:0/1789506742 learned_addr learned my addr 172.21.15.7:0/1789506742 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:05.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374effd700 1 --2- 172.21.15.7:0/1789506742 >> v2:172.21.15.143:3300/0 conn(0x7f37500c65c0 0x7f37500c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374f7fe700 1 -- 172.21.15.7:0/1789506742 >> v2:172.21.15.143:3300/0 conn(0x7f37500c65c0 msgr2=0x7f37500c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:05.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374f7fe700 1 --2- 172.21.15.7:0/1789506742 >> v2:172.21.15.143:3300/0 conn(0x7f37500c65c0 0x7f37500c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374f7fe700 1 -- 172.21.15.7:0/1789506742 >> v1:172.21.15.7:6789/0 conn(0x7f375006a7f0 legacy=0x7f37500c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374e7fc700 1 -- 172.21.15.7:0/1789506742 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095400206 0 0) 0x7f37500a65b0 con 0x7f375006a7f0 2023-03-03T08:44:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374f7fe700 1 -- 172.21.15.7:0/1789506742 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37501541f0 con 0x7f375014de50 2023-03-03T08:44:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374effd700 1 --2- 172.21.15.7:0/1789506742 >> v2:172.21.15.143:3300/0 conn(0x7f37500c65c0 0x7f37500c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374f7fe700 1 --2- 172.21.15.7:0/1789506742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 0x7f375006a2b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f375006f820 tx=0x7f3740009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de749d427a31f165 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374e7fc700 1 -- 172.21.15.7:0/1789506742 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740011830 con 0x7f375014de50 2023-03-03T08:44:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374e7fc700 1 -- 172.21.15.7:0/1789506742 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3740011990 con 0x7f375014de50 2023-03-03T08:44:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f374e7fc700 1 -- 172.21.15.7:0/1789506742 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740011d20 con 0x7f375014de50 2023-03-03T08:44:05.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f3755be7700 1 -- 172.21.15.7:0/1789506742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 msgr2=0x7f375006a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.812+0000 7f3755be7700 1 --2- 172.21.15.7:0/1789506742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 0x7f375006a2b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f375006f820 tx=0x7f3740009b50 comp rx=0 tx=0).stop 2023-03-03T08:44:05.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.813+0000 7f3755be7700 1 -- 172.21.15.7:0/1789506742 shutdown_connections 2023-03-03T08:44:05.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.813+0000 7f3755be7700 1 --2- 172.21.15.7:0/1789506742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 0x7f375006a2b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.813+0000 7f3755be7700 1 --2- 172.21.15.7:0/1789506742 >> v2:172.21.15.143:3300/0 conn(0x7f37500c65c0 0x7f37500c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.813+0000 7f3755be7700 1 -- 172.21.15.7:0/1789506742 >> 172.21.15.7:0/1789506742 conn(0x7f37500a9430 msgr2=0x7f37500c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:05.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.813+0000 7f3755be7700 1 -- 172.21.15.7:0/1789506742 shutdown_connections 2023-03-03T08:44:05.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.813+0000 7f3755be7700 1 -- 172.21.15.7:0/1789506742 wait complete. 2023-03-03T08:44:05.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.814+0000 7f3755be7700 1 Processor -- start 2023-03-03T08:44:05.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.814+0000 7f3755be7700 1 -- start start 2023-03-03T08:44:05.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3755be7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f375006a7f0 0x7f375014abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3755be7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37500c65c0 0x7f37500ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3755be7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 0x7f375014fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3755be7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37501d9fa0 con 0x7f375014de50 2023-03-03T08:44:05.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3755be7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37501ead10 con 0x7f375006a7f0 2023-03-03T08:44:05.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3755be7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37501ebb70 con 0x7f37500c65c0 2023-03-03T08:44:05.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 0x7f375014fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 0x7f375014fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52204/0 (socket says 172.21.15.7:52204) 2023-03-03T08:44:05.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374ffff700 1 -- 172.21.15.7:0/1151954879 learned_addr learned my addr 172.21.15.7:0/1151954879 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:05.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374f7fe700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f375006a7f0 0x7f375014abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374effd700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37500c65c0 0x7f37500ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374ffff700 1 -- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37500c65c0 msgr2=0x7f37500ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374ffff700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37500c65c0 0x7f37500ab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374ffff700 1 -- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f375006a7f0 msgr2=0x7f375014abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374ffff700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f375006a7f0 0x7f375014abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374ffff700 1 -- 172.21.15.7:0/1151954879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37400094a0 con 0x7f375014de50 2023-03-03T08:44:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374f7fe700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f375006a7f0 0x7f375014abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f374ffff700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 0x7f375014fea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f374800cb60 tx=0x7f374800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:05.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3747fff700 1 -- 172.21.15.7:0/1151954879 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374800ab50 con 0x7f375014de50 2023-03-03T08:44:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3747fff700 1 -- 172.21.15.7:0/1151954879 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374800acb0 con 0x7f375014de50 2023-03-03T08:44:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3755be7700 1 -- 172.21.15.7:0/1151954879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37501547b0 con 0x7f375014de50 2023-03-03T08:44:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3747fff700 1 -- 172.21.15.7:0/1151954879 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3748007310 con 0x7f375014de50 2023-03-03T08:44:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.815+0000 7f3755be7700 1 -- 172.21.15.7:0/1151954879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37501544d0 con 0x7f375014de50 2023-03-03T08:44:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.816+0000 7f3755be7700 1 -- 172.21.15.7:0/1151954879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3750154a90 con 0x7f375014de50 2023-03-03T08:44:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.817+0000 7f3747fff700 1 -- 172.21.15.7:0/1151954879 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f374801a030 con 0x7f375014de50 2023-03-03T08:44:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.817+0000 7f3747fff700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37300682f0 0x7f373006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.817+0000 7f3747fff700 1 -- 172.21.15.7:0/1151954879 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f3748085cb0 con 0x7f375014de50 2023-03-03T08:44:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.817+0000 7f374f7fe700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37300682f0 0x7f373006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.818+0000 7f374f7fe700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37300682f0 0x7f373006a7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f37400097c0 tx=0x7f374001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:05.821+0000 7f3747fff700 1 -- 172.21.15.7:0/1151954879 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3748052180 con 0x7f375014de50 2023-03-03T08:44:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.017+0000 7f3755be7700 1 -- 172.21.15.7:0/1151954879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3750074bb0 con 0x7f375014de50 2023-03-03T08:44:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.018+0000 7f3747fff700 1 -- 172.21.15.7:0/1151954879 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f37480180a0 con 0x7f375014de50 2023-03-03T08:44:06.020 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:44:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 -- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37300682f0 msgr2=0x7f373006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37300682f0 0x7f373006a7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f37400097c0 tx=0x7f374001c040 comp rx=0 tx=0).stop 2023-03-03T08:44:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 -- 172.21.15.7:0/1151954879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 msgr2=0x7f375014fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 0x7f375014fea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f374800cb60 tx=0x7f374800cf20 comp rx=0 tx=0).stop 2023-03-03T08:44:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 -- 172.21.15.7:0/1151954879 shutdown_connections 2023-03-03T08:44:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37300682f0 0x7f373006a7a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f375014de50 0x7f375014fea0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f375006a7f0 0x7f375014abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 --2- 172.21.15.7:0/1151954879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37500c65c0 0x7f37500ab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 -- 172.21.15.7:0/1151954879 >> 172.21.15.7:0/1151954879 conn(0x7f37500a9430 msgr2=0x7f375014d130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:06.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 -- 172.21.15.7:0/1151954879 shutdown_connections 2023-03-03T08:44:06.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.020+0000 7f3755be7700 1 -- 172.21.15.7:0/1151954879 wait complete. 2023-03-03T08:44:06.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-03-03T08:44:06.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.162+0000 7f0638762700 1 Processor -- start 2023-03-03T08:44:06.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.162+0000 7f0638762700 1 -- start start 2023-03-03T08:44:06.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.163+0000 7f0638762700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06300aad20 0x7f0630157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.163+0000 7f0638762700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f06300aa030 0x7f0630158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.163+0000 7f0638762700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06300a9c20 con 0x7f063014b0d0 2023-03-03T08:44:06.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.163+0000 7f0638762700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0630154f10 con 0x7f06300aad20 2023-03-03T08:44:06.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.163+0000 7f0638762700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06300ac3c0 con 0x7f06300aa030 2023-03-03T08:44:06.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.163+0000 7f0635cfd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06300aad20 0x7f0630157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.163+0000 7f06364fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f063014b0d0 0x7f0630071800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46188/0 (socket says 172.21.15.7:46188) 2023-03-03T08:44:06.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.163+0000 7f06364fe700 1 -- 172.21.15.7:0/1565442635 learned_addr learned my addr 172.21.15.7:0/1565442635 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:06.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.164+0000 7f0636cff700 1 --2- 172.21.15.7:0/1565442635 >> v2:172.21.15.143:3300/0 conn(0x7f06300aa030 0x7f0630158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.164+0000 7f0635cfd700 1 -- 172.21.15.7:0/1565442635 >> v2:172.21.15.143:3300/0 conn(0x7f06300aa030 msgr2=0x7f0630158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:06.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.164+0000 7f0635cfd700 1 --2- 172.21.15.7:0/1565442635 >> v2:172.21.15.143:3300/0 conn(0x7f06300aa030 0x7f0630158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.164+0000 7f0635cfd700 1 -- 172.21.15.7:0/1565442635 >> v1:172.21.15.7:6789/0 conn(0x7f063014b0d0 legacy=0x7f0630071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.164+0000 7f0635cfd700 1 -- 172.21.15.7:0/1565442635 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063006fc70 con 0x7f06300aad20 2023-03-03T08:44:06.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.164+0000 7f06354fc700 1 -- 172.21.15.7:0/1565442635 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385141863 0 0) 0x7f06300a9c20 con 0x7f063014b0d0 2023-03-03T08:44:06.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.164+0000 7f0635cfd700 1 --2- 172.21.15.7:0/1565442635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aad20 0x7f0630157df0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f06300a98d0 tx=0x7f0624009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34339ec081291dfd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:06.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.164+0000 7f06354fc700 1 -- 172.21.15.7:0/1565442635 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624011830 con 0x7f06300aad20 2023-03-03T08:44:06.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.164+0000 7f06354fc700 1 -- 172.21.15.7:0/1565442635 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0624011990 con 0x7f06300aad20 2023-03-03T08:44:06.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.164+0000 7f06354fc700 1 -- 172.21.15.7:0/1565442635 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624011d20 con 0x7f06300aad20 2023-03-03T08:44:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.165+0000 7f0638762700 1 -- 172.21.15.7:0/1565442635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aad20 msgr2=0x7f0630157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.165+0000 7f0638762700 1 --2- 172.21.15.7:0/1565442635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aad20 0x7f0630157df0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f06300a98d0 tx=0x7f0624009b20 comp rx=0 tx=0).stop 2023-03-03T08:44:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.165+0000 7f0638762700 1 -- 172.21.15.7:0/1565442635 shutdown_connections 2023-03-03T08:44:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.165+0000 7f0638762700 1 --2- 172.21.15.7:0/1565442635 >> v2:172.21.15.143:3300/0 conn(0x7f06300aa030 0x7f0630158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.165+0000 7f0638762700 1 --2- 172.21.15.7:0/1565442635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aad20 0x7f0630157df0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.165+0000 7f0638762700 1 -- 172.21.15.7:0/1565442635 >> 172.21.15.7:0/1565442635 conn(0x7f06300c8900 msgr2=0x7f06300c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.165+0000 7f0638762700 1 -- 172.21.15.7:0/1565442635 shutdown_connections 2023-03-03T08:44:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.165+0000 7f0638762700 1 -- 172.21.15.7:0/1565442635 wait complete. 2023-03-03T08:44:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.166+0000 7f0638762700 1 Processor -- start 2023-03-03T08:44:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.166+0000 7f0638762700 1 -- start start 2023-03-03T08:44:06.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0638762700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aa030 0x7f06300c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0638762700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300aad20 0x7f06301eca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0638762700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063014b0d0 0x7f0630145e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0638762700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0630158360 con 0x7f06300aa030 2023-03-03T08:44:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0638762700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06301e0370 con 0x7f06300aad20 2023-03-03T08:44:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0638762700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06301e14c0 con 0x7f063014b0d0 2023-03-03T08:44:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f06364fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aa030 0x7f06300c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f06364fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aa030 0x7f06300c5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52214/0 (socket says 172.21.15.7:52214) 2023-03-03T08:44:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f06364fe700 1 -- 172.21.15.7:0/2187302919 learned_addr learned my addr 172.21.15.7:0/2187302919 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0635cfd700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300aad20 0x7f06301eca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0636cff700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063014b0d0 0x7f0630145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f06364fe700 1 -- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063014b0d0 msgr2=0x7f0630145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f06364fe700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063014b0d0 0x7f0630145e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f06364fe700 1 -- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300aad20 msgr2=0x7f06301eca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f06364fe700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300aad20 0x7f06301eca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f06364fe700 1 -- 172.21.15.7:0/2187302919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06240094a0 con 0x7f06300aa030 2023-03-03T08:44:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0635cfd700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300aad20 0x7f06301eca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0636cff700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063014b0d0 0x7f0630145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f06364fe700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aa030 0x7f06300c5480 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f062c0011e0 tx=0x7f062c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0622ffd700 1 -- 172.21.15.7:0/2187302919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c004c80 con 0x7f06300aa030 2023-03-03T08:44:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0622ffd700 1 -- 172.21.15.7:0/2187302919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f062c004e00 con 0x7f06300aa030 2023-03-03T08:44:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0638762700 1 -- 172.21.15.7:0/2187302919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063006f6b0 con 0x7f06300aa030 2023-03-03T08:44:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0622ffd700 1 -- 172.21.15.7:0/2187302919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c007310 con 0x7f06300aa030 2023-03-03T08:44:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.167+0000 7f0638762700 1 -- 172.21.15.7:0/2187302919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0630155160 con 0x7f06300aa030 2023-03-03T08:44:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.169+0000 7f0638762700 1 -- 172.21.15.7:0/2187302919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063006f990 con 0x7f06300aa030 2023-03-03T08:44:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.169+0000 7f0622ffd700 1 -- 172.21.15.7:0/2187302919 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f062c01a020 con 0x7f06300aa030 2023-03-03T08:44:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.169+0000 7f0622ffd700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06140682f0 0x7f061406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.170+0000 7f0622ffd700 1 -- 172.21.15.7:0/2187302919 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f062c085c40 con 0x7f06300aa030 2023-03-03T08:44:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.170+0000 7f0635cfd700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06140682f0 0x7f061406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.170+0000 7f0635cfd700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06140682f0 0x7f061406a7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f06300a98d0 tx=0x7f062401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:06.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.174+0000 7f0622ffd700 1 -- 172.21.15.7:0/2187302919 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f062c052170 con 0x7f06300aa030 2023-03-03T08:44:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.366+0000 7f0638762700 1 -- 172.21.15.7:0/2187302919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f0630074bb0 con 0x7f06300aa030 2023-03-03T08:44:06.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.366+0000 7f0622ffd700 1 -- 172.21.15.7:0/2187302919 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7f062c055790 con 0x7f06300aa030 2023-03-03T08:44:06.368 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:44:06.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 -- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06140682f0 msgr2=0x7f061406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06140682f0 0x7f061406a7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f06300a98d0 tx=0x7f062401c040 comp rx=0 tx=0).stop 2023-03-03T08:44:06.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 -- 172.21.15.7:0/2187302919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aa030 msgr2=0x7f06300c5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aa030 0x7f06300c5480 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f062c0011e0 tx=0x7f062c00cae0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 -- 172.21.15.7:0/2187302919 shutdown_connections 2023-03-03T08:44:06.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f063014b0d0 0x7f0630145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06140682f0 0x7f061406a7a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06300aa030 0x7f06300c5480 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 --2- 172.21.15.7:0/2187302919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06300aad20 0x7f06301eca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 -- 172.21.15.7:0/2187302919 >> 172.21.15.7:0/2187302919 conn(0x7f06300c8900 msgr2=0x7f06300717b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:06.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 -- 172.21.15.7:0/2187302919 shutdown_connections 2023-03-03T08:44:06.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.368+0000 7f0638762700 1 -- 172.21.15.7:0/2187302919 wait complete. 2023-03-03T08:44:06.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-03-03T08:44:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.507+0000 7fdfa4ee9700 1 Processor -- start 2023-03-03T08:44:06.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.508+0000 7fdfa4ee9700 1 -- start start 2023-03-03T08:44:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.508+0000 7fdfa4ee9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdfa00c1da0 0x7fdfa01550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.508+0000 7fdfa4ee9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00c3e30 0x7fdfa00c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.508+0000 7fdfa4ee9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa00c7a60 con 0x7fdfa006d870 2023-03-03T08:44:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.508+0000 7fdfa4ee9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa00c2af0 con 0x7fdfa00c1da0 2023-03-03T08:44:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.508+0000 7fdfa4ee9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00c3620 con 0x7fdfa00c3e30 2023-03-03T08:44:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00c3e30 0x7fdfa00c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9dd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdfa00c1da0 0x7fdfa01550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9dd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdfa00c1da0 0x7fdfa01550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47998/0 (socket says 172.21.15.7:47998) 2023-03-03T08:44:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9dd9b700 1 -- 172.21.15.7:0/3006383232 learned_addr learned my addr 172.21.15.7:0/3006383232 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00c3e30 0x7fdfa00c4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:60298/0 (socket says 172.21.15.7:60298) 2023-03-03T08:44:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9e59c700 1 --1- 172.21.15.7:0/3006383232 >> v1:172.21.15.143:6789/0 conn(0x7fdfa006d870 0x7fdfa00c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49722/0 (socket says 172.21.15.7:49722) 2023-03-03T08:44:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9dd9b700 1 -- 172.21.15.7:0/3006383232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00c3e30 msgr2=0x7fdfa00c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9dd9b700 1 --2- 172.21.15.7:0/3006383232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00c3e30 0x7fdfa00c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9dd9b700 1 -- 172.21.15.7:0/3006383232 >> v1:172.21.15.143:6789/0 conn(0x7fdfa006d870 legacy=0x7fdfa00c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9dd9b700 1 -- 172.21.15.7:0/3006383232 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa01545e0 con 0x7fdfa00c1da0 2023-03-03T08:44:06.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.509+0000 7fdf9dd9b700 1 --2- 172.21.15.7:0/3006383232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00c1da0 0x7fdfa01550c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdf880098f0 tx=0x7fdf88009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f215ec4b324a5f1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:06.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdf9d59a700 1 -- 172.21.15.7:0/3006383232 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf88011830 con 0x7fdfa00c1da0 2023-03-03T08:44:06.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdf9d59a700 1 -- 172.21.15.7:0/3006383232 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf88011990 con 0x7fdfa00c1da0 2023-03-03T08:44:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdf9d59a700 1 -- 172.21.15.7:0/3006383232 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf88011d20 con 0x7fdfa00c1da0 2023-03-03T08:44:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/3006383232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00c1da0 msgr2=0x7fdfa01550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdfa4ee9700 1 --2- 172.21.15.7:0/3006383232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00c1da0 0x7fdfa01550c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdf880098f0 tx=0x7fdf88009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/3006383232 shutdown_connections 2023-03-03T08:44:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdfa4ee9700 1 --2- 172.21.15.7:0/3006383232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa00c3e30 0x7fdfa00c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdfa4ee9700 1 --2- 172.21.15.7:0/3006383232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00c1da0 0x7fdfa01550c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/3006383232 >> 172.21.15.7:0/3006383232 conn(0x7fdfa006c110 msgr2=0x7fdfa006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/3006383232 shutdown_connections 2023-03-03T08:44:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.510+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/3006383232 wait complete. 2023-03-03T08:44:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.511+0000 7fdfa4ee9700 1 Processor -- start 2023-03-03T08:44:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.512+0000 7fdfa4ee9700 1 -- start start 2023-03-03T08:44:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.512+0000 7fdfa4ee9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa006d870 0x7fdfa0155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.512+0000 7fdfa4ee9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00c1da0 0x7fdfa00c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.512+0000 7fdfa4ee9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdfa00c3e30 0x7fdfa0147430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.512+0000 7fdfa4ee9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa01edd00 con 0x7fdfa00c3e30 2023-03-03T08:44:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.512+0000 7fdfa4ee9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa0146c30 con 0x7fdfa00c1da0 2023-03-03T08:44:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.512+0000 7fdfa4ee9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa0146f10 con 0x7fdfa006d870 2023-03-03T08:44:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.512+0000 7fdf9ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdfa00c3e30 0x7fdfa0147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.512+0000 7fdf9ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdfa00c3e30 0x7fdfa0147430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52232/0 (socket says 172.21.15.7:52232) 2023-03-03T08:44:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.512+0000 7fdf9ed9d700 1 -- 172.21.15.7:0/2364491402 learned_addr learned my addr 172.21.15.7:0/2364491402 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:06.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf9dd9b700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00c1da0 0x7fdfa00c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf9e59c700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa006d870 0x7fdfa0155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf9ed9d700 1 -- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa006d870 msgr2=0x7fdfa0155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf9ed9d700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa006d870 0x7fdfa0155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf9ed9d700 1 -- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00c1da0 msgr2=0x7fdfa00c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf9ed9d700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00c1da0 0x7fdfa00c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf9ed9d700 1 -- 172.21.15.7:0/2364491402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf88009550 con 0x7fdfa00c3e30 2023-03-03T08:44:06.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf9ed9d700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdfa00c3e30 0x7fdfa0147430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800efc0 tx=0x7fdf9800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:06.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf96ffd700 1 -- 172.21.15.7:0/2364491402 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9800cad0 con 0x7fdfa00c3e30 2023-03-03T08:44:06.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf96ffd700 1 -- 172.21.15.7:0/2364491402 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9800f040 con 0x7fdfa00c3e30 2023-03-03T08:44:06.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/2364491402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa0154ba0 con 0x7fdfa00c3e30 2023-03-03T08:44:06.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdf96ffd700 1 -- 172.21.15.7:0/2364491402 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9800ce70 con 0x7fdfa00c3e30 2023-03-03T08:44:06.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.513+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/2364491402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa01548c0 con 0x7fdfa00c3e30 2023-03-03T08:44:06.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.514+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/2364491402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa006dc40 con 0x7fdfa00c3e30 2023-03-03T08:44:06.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.515+0000 7fdf96ffd700 1 -- 172.21.15.7:0/2364491402 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdf98014480 con 0x7fdfa00c3e30 2023-03-03T08:44:06.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.515+0000 7fdf96ffd700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf800682f0 0x7fdf8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.515+0000 7fdf96ffd700 1 -- 172.21.15.7:0/2364491402 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdf980868a0 con 0x7fdfa00c3e30 2023-03-03T08:44:06.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.515+0000 7fdf9e59c700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf800682f0 0x7fdf8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.516+0000 7fdf9e59c700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf800682f0 0x7fdf8006a7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdf90000f60 tx=0x7fdf90007ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:06.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.519+0000 7fdf96ffd700 1 -- 172.21.15.7:0/2364491402 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdf98052d70 con 0x7fdfa00c3e30 2023-03-03T08:44:06.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.714+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/2364491402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdfa0147080 con 0x7fdfa00c3e30 2023-03-03T08:44:06.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.714+0000 7fdf96ffd700 1 -- 172.21.15.7:0/2364491402 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7fdf98056390 con 0x7fdfa00c3e30 2023-03-03T08:44:06.716 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:44:06.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf800682f0 msgr2=0x7fdf8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf800682f0 0x7fdf8006a7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdf90000f60 tx=0x7fdf90007ed0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/2364491402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdfa00c3e30 msgr2=0x7fdfa0147430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdfa00c3e30 0x7fdfa0147430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdf9800efc0 tx=0x7fdf9800c0c0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/2364491402 shutdown_connections 2023-03-03T08:44:06.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf800682f0 0x7fdf8006a7a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdfa00c3e30 0x7fdfa0147430 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdfa006d870 0x7fdfa0155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 --2- 172.21.15.7:0/2364491402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfa00c1da0 0x7fdfa00c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/2364491402 >> 172.21.15.7:0/2364491402 conn(0x7fdfa006c110 msgr2=0x7fdfa00c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:06.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/2364491402 shutdown_connections 2023-03-03T08:44:06.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.716+0000 7fdfa4ee9700 1 -- 172.21.15.7:0/2364491402 wait complete. 2023-03-03T08:44:06.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-03-03T08:44:06.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.856+0000 7feb8f960700 1 Processor -- start 2023-03-03T08:44:06.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.856+0000 7feb8f960700 1 -- start start 2023-03-03T08:44:06.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8f960700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb880c6010 0x7feb880c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8f960700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb880c17f0 0x7feb880c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8f960700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7feb8806d6a0 0x7feb8806db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8f960700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb88152100 con 0x7feb880c6010 2023-03-03T08:44:06.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8f960700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb880c5500 con 0x7feb8806d6a0 2023-03-03T08:44:06.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8f960700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8814ebe0 con 0x7feb880c17f0 2023-03-03T08:44:06.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8d6fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb880c6010 0x7feb880c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8d6fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb880c6010 0x7feb880c4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52236/0 (socket says 172.21.15.7:52236) 2023-03-03T08:44:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8d6fc700 1 -- 172.21.15.7:0/1756705190 learned_addr learned my addr 172.21.15.7:0/1756705190 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8defd700 1 --2- 172.21.15.7:0/1756705190 >> v2:172.21.15.143:3300/0 conn(0x7feb8806d6a0 0x7feb8806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8cefb700 1 --2- 172.21.15.7:0/1756705190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb880c17f0 0x7feb880c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8d6fc700 1 -- 172.21.15.7:0/1756705190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb880c17f0 msgr2=0x7feb880c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8d6fc700 1 --2- 172.21.15.7:0/1756705190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb880c17f0 0x7feb880c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8d6fc700 1 -- 172.21.15.7:0/1756705190 >> v2:172.21.15.143:3300/0 conn(0x7feb8806d6a0 msgr2=0x7feb8806db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8d6fc700 1 --2- 172.21.15.7:0/1756705190 >> v2:172.21.15.143:3300/0 conn(0x7feb8806d6a0 0x7feb8806db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8d6fc700 1 -- 172.21.15.7:0/1756705190 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb880c2e70 con 0x7feb880c6010 2023-03-03T08:44:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.857+0000 7feb8d6fc700 1 --2- 172.21.15.7:0/1756705190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 0x7feb880c4a80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7feb88066df0 tx=0x7feb78009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2033d268d3a6f009 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb7ffff700 1 -- 172.21.15.7:0/1756705190 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7800f8a0 con 0x7feb880c6010 2023-03-03T08:44:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb7ffff700 1 -- 172.21.15.7:0/1756705190 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7800fa00 con 0x7feb880c6010 2023-03-03T08:44:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb7ffff700 1 -- 172.21.15.7:0/1756705190 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7800fd90 con 0x7feb880c6010 2023-03-03T08:44:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb8f960700 1 -- 172.21.15.7:0/1756705190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 msgr2=0x7feb880c4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb8f960700 1 --2- 172.21.15.7:0/1756705190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 0x7feb880c4a80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7feb88066df0 tx=0x7feb78009bd0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb8f960700 1 -- 172.21.15.7:0/1756705190 shutdown_connections 2023-03-03T08:44:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb8f960700 1 --2- 172.21.15.7:0/1756705190 >> v2:172.21.15.143:3300/0 conn(0x7feb8806d6a0 0x7feb8806db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb8f960700 1 --2- 172.21.15.7:0/1756705190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 0x7feb880c4a80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb8f960700 1 --2- 172.21.15.7:0/1756705190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb880c17f0 0x7feb880c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb8f960700 1 -- 172.21.15.7:0/1756705190 >> 172.21.15.7:0/1756705190 conn(0x7feb880a9430 msgr2=0x7feb880c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:06.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb8f960700 1 -- 172.21.15.7:0/1756705190 shutdown_connections 2023-03-03T08:44:06.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.858+0000 7feb8f960700 1 -- 172.21.15.7:0/1756705190 wait complete. 2023-03-03T08:44:06.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.859+0000 7feb8f960700 1 Processor -- start 2023-03-03T08:44:06.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8f960700 1 -- start start 2023-03-03T08:44:06.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8f960700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb8806d6a0 0x7feb881e5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8f960700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb880c17f0 0x7feb88071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8f960700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 0x7feb8814df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8f960700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb880bd030 con 0x7feb880c6010 2023-03-03T08:44:06.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8f960700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb88072840 con 0x7feb880c17f0 2023-03-03T08:44:06.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8f960700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb881e6520 con 0x7feb8806d6a0 2023-03-03T08:44:06.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8defd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 0x7feb8814df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8defd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 0x7feb8814df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52246/0 (socket says 172.21.15.7:52246) 2023-03-03T08:44:06.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8defd700 1 -- 172.21.15.7:0/3822474764 learned_addr learned my addr 172.21.15.7:0/3822474764 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8cefb700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb880c17f0 0x7feb88071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.860+0000 7feb8d6fc700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb8806d6a0 0x7feb881e5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb8defd700 1 -- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb8806d6a0 msgr2=0x7feb881e5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb8defd700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb8806d6a0 0x7feb881e5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb8defd700 1 -- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb880c17f0 msgr2=0x7feb88071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb8defd700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb880c17f0 0x7feb88071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb8defd700 1 -- 172.21.15.7:0/3822474764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb78009550 con 0x7feb880c6010 2023-03-03T08:44:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb8defd700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 0x7feb8814df00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feb80007e40 tx=0x7feb8000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb7dffb700 1 -- 172.21.15.7:0/3822474764 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000ed30 con 0x7feb880c6010 2023-03-03T08:44:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb7dffb700 1 -- 172.21.15.7:0/3822474764 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb80014420 con 0x7feb880c6010 2023-03-03T08:44:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb8f960700 1 -- 172.21.15.7:0/3822474764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb880aa960 con 0x7feb880c6010 2023-03-03T08:44:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb7dffb700 1 -- 172.21.15.7:0/3822474764 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800147d0 con 0x7feb880c6010 2023-03-03T08:44:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.861+0000 7feb8f960700 1 -- 172.21.15.7:0/3822474764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb880aa680 con 0x7feb880c6010 2023-03-03T08:44:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.862+0000 7feb8f960700 1 -- 172.21.15.7:0/3822474764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb880c2b00 con 0x7feb880c6010 2023-03-03T08:44:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.862+0000 7feb7dffb700 1 -- 172.21.15.7:0/3822474764 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feb80012050 con 0x7feb880c6010 2023-03-03T08:44:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.863+0000 7feb7dffb700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb6c0682f0 0x7feb6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.863+0000 7feb7dffb700 1 -- 172.21.15.7:0/3822474764 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7feb800864e0 con 0x7feb880c6010 2023-03-03T08:44:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.863+0000 7feb8d6fc700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb6c0682f0 0x7feb6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.863+0000 7feb8d6fc700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb6c0682f0 0x7feb6c06a7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7feb78000c00 tx=0x7feb7801b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:06.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:06.867+0000 7feb7dffb700 1 -- 172.21.15.7:0/3822474764 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb80052a10 con 0x7feb880c6010 2023-03-03T08:44:07.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.056+0000 7feb8f960700 1 -- 172.21.15.7:0/3822474764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7feb88074c30 con 0x7feb880c6010 2023-03-03T08:44:07.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.056+0000 7feb7dffb700 1 -- 172.21.15.7:0/3822474764 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7feb80056030 con 0x7feb880c6010 2023-03-03T08:44:07.059 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:44:07.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.058+0000 7feb8f960700 1 -- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb6c0682f0 msgr2=0x7feb6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.058+0000 7feb8f960700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb6c0682f0 0x7feb6c06a7a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7feb78000c00 tx=0x7feb7801b040 comp rx=0 tx=0).stop 2023-03-03T08:44:07.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.058+0000 7feb8f960700 1 -- 172.21.15.7:0/3822474764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 msgr2=0x7feb8814df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.058+0000 7feb8f960700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 0x7feb8814df00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feb80007e40 tx=0x7feb8000e3c0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.058+0000 7feb8f960700 1 -- 172.21.15.7:0/3822474764 shutdown_connections 2023-03-03T08:44:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.058+0000 7feb8f960700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb6c0682f0 0x7feb6c06a7a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.058+0000 7feb8f960700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb880c6010 0x7feb8814df00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.058+0000 7feb8f960700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb8806d6a0 0x7feb881e5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.058+0000 7feb8f960700 1 --2- 172.21.15.7:0/3822474764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb880c17f0 0x7feb88071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.058+0000 7feb8f960700 1 -- 172.21.15.7:0/3822474764 >> 172.21.15.7:0/3822474764 conn(0x7feb880a9430 msgr2=0x7feb8806dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.059+0000 7feb8f960700 1 -- 172.21.15.7:0/3822474764 shutdown_connections 2023-03-03T08:44:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.059+0000 7feb8f960700 1 -- 172.21.15.7:0/3822474764 wait complete. 2023-03-03T08:44:07.072 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-03-03T08:44:07.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:07.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.196+0000 7f1fb2d95700 1 Processor -- start 2023-03-03T08:44:07.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.196+0000 7f1fb2d95700 1 -- start start 2023-03-03T08:44:07.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fb2d95700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fac071990 0x7f1fac071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fb2d95700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1fac14b460 0x7f1fac0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fb2d95700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fac070360 con 0x7f1fac151c00 2023-03-03T08:44:07.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fb2d95700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fac1521a0 con 0x7f1fac14b460 2023-03-03T08:44:07.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fb2d95700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac06a5e0 con 0x7f1fac071990 2023-03-03T08:44:07.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fabfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1fac14b460 0x7f1fac0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fb1332700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1fac151c00 0x7f1fac06aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46234/0 (socket says 172.21.15.7:46234) 2023-03-03T08:44:07.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fabfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1fac14b460 0x7f1fac0722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52252/0 (socket says 172.21.15.7:52252) 2023-03-03T08:44:07.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fb1332700 1 -- 172.21.15.7:0/38614386 learned_addr learned my addr 172.21.15.7:0/38614386 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:07.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fb0b31700 1 --2- 172.21.15.7:0/38614386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fac071990 0x7f1fac071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fabfff700 1 -- 172.21.15.7:0/38614386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fac071990 msgr2=0x7f1fac071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:07.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fabfff700 1 --2- 172.21.15.7:0/38614386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fac071990 0x7f1fac071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fabfff700 1 -- 172.21.15.7:0/38614386 >> v1:172.21.15.7:6789/0 conn(0x7f1fac151c00 legacy=0x7f1fac06aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fabfff700 1 -- 172.21.15.7:0/38614386 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac0c38b0 con 0x7f1fac14b460 2023-03-03T08:44:07.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.197+0000 7f1fab7fe700 1 -- 172.21.15.7:0/38614386 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891352897 0 0) 0x7f1fac070360 con 0x7f1fac151c00 2023-03-03T08:44:07.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fabfff700 1 --2- 172.21.15.7:0/38614386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac14b460 0x7f1fac0722a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1f98009840 tx=0x7f1f98009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5ac97f335aea74 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:07.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fab7fe700 1 -- 172.21.15.7:0/38614386 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f98011830 con 0x7f1fac14b460 2023-03-03T08:44:07.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fab7fe700 1 -- 172.21.15.7:0/38614386 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f98011990 con 0x7f1fac14b460 2023-03-03T08:44:07.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fab7fe700 1 -- 172.21.15.7:0/38614386 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f98011d20 con 0x7f1fac14b460 2023-03-03T08:44:07.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fb2d95700 1 -- 172.21.15.7:0/38614386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac14b460 msgr2=0x7f1fac0722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fb2d95700 1 --2- 172.21.15.7:0/38614386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac14b460 0x7f1fac0722a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1f98009840 tx=0x7f1f98009b50 comp rx=0 tx=0).stop 2023-03-03T08:44:07.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fb2d95700 1 -- 172.21.15.7:0/38614386 shutdown_connections 2023-03-03T08:44:07.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fb2d95700 1 --2- 172.21.15.7:0/38614386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fac071990 0x7f1fac071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fb2d95700 1 --2- 172.21.15.7:0/38614386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac14b460 0x7f1fac0722a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fb2d95700 1 -- 172.21.15.7:0/38614386 >> 172.21.15.7:0/38614386 conn(0x7f1fac0a94d0 msgr2=0x7f1fac0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:07.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fb2d95700 1 -- 172.21.15.7:0/38614386 shutdown_connections 2023-03-03T08:44:07.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.198+0000 7f1fb2d95700 1 -- 172.21.15.7:0/38614386 wait complete. 2023-03-03T08:44:07.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.200+0000 7f1fb2d95700 1 Processor -- start 2023-03-03T08:44:07.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.200+0000 7f1fb2d95700 1 -- start start 2023-03-03T08:44:07.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb2d95700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fac071990 0x7f1fac1ddcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb2d95700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac151c00 0x7f1fac14b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb2d95700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fac14bed0 0x7f1fac144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb2d95700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac145430 con 0x7f1fac151c00 2023-03-03T08:44:07.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb2d95700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac14c8d0 con 0x7f1fac14bed0 2023-03-03T08:44:07.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb2d95700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac144f40 con 0x7f1fac071990 2023-03-03T08:44:07.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fabfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac151c00 0x7f1fac14b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fabfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac151c00 0x7f1fac14b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52258/0 (socket says 172.21.15.7:52258) 2023-03-03T08:44:07.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fabfff700 1 -- 172.21.15.7:0/2187942585 learned_addr learned my addr 172.21.15.7:0/2187942585 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:07.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb1332700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fac14bed0 0x7f1fac144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb0b31700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fac071990 0x7f1fac1ddcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fabfff700 1 -- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fac071990 msgr2=0x7f1fac1ddcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fabfff700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fac071990 0x7f1fac1ddcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fabfff700 1 -- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fac14bed0 msgr2=0x7f1fac144440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fabfff700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fac14bed0 0x7f1fac144440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fabfff700 1 -- 172.21.15.7:0/2187942585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f980094a0 con 0x7f1fac151c00 2023-03-03T08:44:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb1332700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fac14bed0 0x7f1fac144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fabfff700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac151c00 0x7f1fac14b990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1f980090b0 tx=0x7f1f98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fa97fa700 1 -- 172.21.15.7:0/2187942585 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f980045b0 con 0x7f1fac151c00 2023-03-03T08:44:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fa97fa700 1 -- 172.21.15.7:0/2187942585 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9801f070 con 0x7f1fac151c00 2023-03-03T08:44:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb2d95700 1 -- 172.21.15.7:0/2187942585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fac0c3b50 con 0x7f1fac151c00 2023-03-03T08:44:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fa97fa700 1 -- 172.21.15.7:0/2187942585 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f980038c0 con 0x7f1fac151c00 2023-03-03T08:44:07.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.201+0000 7f1fb2d95700 1 -- 172.21.15.7:0/2187942585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fac06a2b0 con 0x7f1fac151c00 2023-03-03T08:44:07.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.203+0000 7f1fb2d95700 1 -- 172.21.15.7:0/2187942585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fac06b180 con 0x7f1fac151c00 2023-03-03T08:44:07.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.203+0000 7f1fa97fa700 1 -- 172.21.15.7:0/2187942585 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f9802a020 con 0x7f1fac151c00 2023-03-03T08:44:07.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.203+0000 7f1fa97fa700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f900682f0 0x7f1f9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.203+0000 7f1fa97fa700 1 -- 172.21.15.7:0/2187942585 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f1f980874d0 con 0x7f1fac151c00 2023-03-03T08:44:07.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.203+0000 7f1fb0b31700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f900682f0 0x7f1f9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.204+0000 7f1fb0b31700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f900682f0 0x7f1f9006a7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1fac14e840 tx=0x7f1fa0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:07.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.208+0000 7f1fa97fa700 1 -- 172.21.15.7:0/2187942585 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f98053a00 con 0x7f1fac151c00 2023-03-03T08:44:07.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.445+0000 7f1fb2d95700 1 -- 172.21.15.7:0/2187942585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fac06d0d0 con 0x7f1fac151c00 2023-03-03T08:44:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.446+0000 7f1fa97fa700 1 -- 172.21.15.7:0/2187942585 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f1f98057020 con 0x7f1fac151c00 2023-03-03T08:44:07.447 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:07.448 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 -- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f900682f0 msgr2=0x7f1f9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f900682f0 0x7f1f9006a7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1fac14e840 tx=0x7f1fa0008040 comp rx=0 tx=0).stop 2023-03-03T08:44:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 -- 172.21.15.7:0/2187942585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac151c00 msgr2=0x7f1fac14b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac151c00 0x7f1fac14b990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1f980090b0 tx=0x7f1f98004750 comp rx=0 tx=0).stop 2023-03-03T08:44:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 -- 172.21.15.7:0/2187942585 shutdown_connections 2023-03-03T08:44:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fac14bed0 0x7f1fac144440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fac071990 0x7f1fac1ddcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f900682f0 0x7f1f9006a7a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 --2- 172.21.15.7:0/2187942585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fac151c00 0x7f1fac14b990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 -- 172.21.15.7:0/2187942585 >> 172.21.15.7:0/2187942585 conn(0x7f1fac0a94d0 msgr2=0x7f1fac0708a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 -- 172.21.15.7:0/2187942585 shutdown_connections 2023-03-03T08:44:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.448+0000 7f1fb2d95700 1 -- 172.21.15.7:0/2187942585 wait complete. 2023-03-03T08:44:07.452 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:07.464 INFO:tasks.mds_thrash:Ready to start thrashing 2023-03-03T08:44:07.465 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-03-03T08:44:07.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:44:07.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.589+0000 7f645431b700 1 Processor -- start 2023-03-03T08:44:07.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.590+0000 7f645431b700 1 -- start start 2023-03-03T08:44:07.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.590+0000 7f645431b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c14ae20 0x7f644c154e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.590+0000 7f645431b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c1553b0 con 0x7f644c0aabc0 2023-03-03T08:44:07.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.590+0000 7f645431b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c0aa150 con 0x7f644c0c1ac0 2023-03-03T08:44:07.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.590+0000 7f645431b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c06f5b0 con 0x7f644c14ae20 2023-03-03T08:44:07.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64518b6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f644c0aabc0 0x7f644c071840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46246/0 (socket says 172.21.15.7:46246) 2023-03-03T08:44:07.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64518b6700 1 -- 172.21.15.7:0/282998753 learned_addr learned my addr 172.21.15.7:0/282998753 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:07.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64520b7700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c14ae20 0x7f644c154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910535860 0 0) 0x7f644c1553b0 con 0x7f644c0aabc0 2023-03-03T08:44:07.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64380035b0 con 0x7f644c0aabc0 2023-03-03T08:44:07.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 21729701 0 0) 0x7f64380035b0 con 0x7f644c0aabc0 2023-03-03T08:44:07.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c14ae20 msgr2=0x7f644c154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:07.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c14ae20 0x7f644c154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 >> v1:172.21.15.143:6789/0 conn(0x7f644c0c1ac0 legacy=0x7f644c0c1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c06f890 con 0x7f644c0aabc0 2023-03-03T08:44:07.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f643c001670 con 0x7f644c0aabc0 2023-03-03T08:44:07.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 >> v1:172.21.15.7:6789/0 conn(0x7f644c0aabc0 legacy=0x7f644c071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6438003aa0 0x7f6438003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6438004850 0x7f6438004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c14ae20 0x7f6438005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c06f5b0 con 0x7f6438003aa0 2023-03-03T08:44:07.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643800bac0 con 0x7f6438004850 2023-03-03T08:44:07.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643800bc30 con 0x7f644c14ae20 2023-03-03T08:44:07.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.591+0000 7f64520b7700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6438003aa0 0x7f6438003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64518b6700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6438004850 0x7f6438004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64528b8700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c14ae20 0x7f6438005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64520b7700 1 -- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c14ae20 msgr2=0x7f6438005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:07.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64520b7700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c14ae20 0x7f6438005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64520b7700 1 -- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6438004850 msgr2=0x7f6438004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:07.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64520b7700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6438004850 0x7f6438004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64520b7700 1 -- 172.21.15.7:0/282998753 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64380035b0 con 0x7f6438003aa0 2023-03-03T08:44:07.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64518b6700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6438004850 0x7f6438004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:07.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64520b7700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6438003aa0 0x7f6438003e70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6448009b60 tx=0x7f6438005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90712949a902768 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:07.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644800e6c0 con 0x7f6438003aa0 2023-03-03T08:44:07.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644800e820 con 0x7f6438003aa0 2023-03-03T08:44:07.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64510b5700 1 -- 172.21.15.7:0/282998753 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644800eb90 con 0x7f6438003aa0 2023-03-03T08:44:07.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f645431b700 1 -- 172.21.15.7:0/282998753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6438003aa0 msgr2=0x7f6438003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f645431b700 1 --2- 172.21.15.7:0/282998753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6438003aa0 0x7f6438003e70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6448009b60 tx=0x7f6438005880 comp rx=0 tx=0).stop 2023-03-03T08:44:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.592+0000 7f64528b8700 1 -- 172.21.15.7:0/282998753 reap_dead start 2023-03-03T08:44:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.593+0000 7f645431b700 1 -- 172.21.15.7:0/282998753 shutdown_connections 2023-03-03T08:44:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.593+0000 7f645431b700 1 -- 172.21.15.7:0/282998753 >> 172.21.15.7:0/282998753 conn(0x7f644c0c8900 msgr2=0x7f644c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.593+0000 7f645431b700 1 -- 172.21.15.7:0/282998753 shutdown_connections 2023-03-03T08:44:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.593+0000 7f645431b700 1 -- 172.21.15.7:0/282998753 wait complete. 2023-03-03T08:44:07.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.594+0000 7f645431b700 1 Processor -- start 2023-03-03T08:44:07.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.594+0000 7f645431b700 1 -- start start 2023-03-03T08:44:07.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f645431b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6438003aa0 0x7f644c14d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f645431b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f644c14d5f0 0x7f644c14ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f645431b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c14f2a0 0x7f644c1473e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f645431b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c148380 con 0x7f644c14f2a0 2023-03-03T08:44:07.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f645431b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c14da60 con 0x7f644c14d5f0 2023-03-03T08:44:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f645431b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c14e830 con 0x7f6438003aa0 2023-03-03T08:44:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64528b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c14f2a0 0x7f644c1473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64528b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c14f2a0 0x7f644c1473e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52280/0 (socket says 172.21.15.7:52280) 2023-03-03T08:44:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64528b8700 1 -- 172.21.15.7:0/875584264 learned_addr learned my addr 172.21.15.7:0/875584264 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:07.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64520b7700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6438003aa0 0x7f644c14d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64518b6700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f644c14d5f0 0x7f644c14ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64528b8700 1 -- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6438003aa0 msgr2=0x7f644c14d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64528b8700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6438003aa0 0x7f644c14d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64528b8700 1 -- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f644c14d5f0 msgr2=0x7f644c14ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64528b8700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f644c14d5f0 0x7f644c14ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64528b8700 1 -- 172.21.15.7:0/875584264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c0c2610 con 0x7f644c14f2a0 2023-03-03T08:44:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.595+0000 7f64528b8700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c14f2a0 0x7f644c1473e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f644c0ada10 tx=0x7f644400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:07.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.596+0000 7f6442ffd700 1 -- 172.21.15.7:0/875584264 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444003c10 con 0x7f644c14f2a0 2023-03-03T08:44:07.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.596+0000 7f6442ffd700 1 -- 172.21.15.7:0/875584264 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6444003d90 con 0x7f644c14f2a0 2023-03-03T08:44:07.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.596+0000 7f6442ffd700 1 -- 172.21.15.7:0/875584264 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444005310 con 0x7f644c14f2a0 2023-03-03T08:44:07.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.596+0000 7f645431b700 1 -- 172.21.15.7:0/875584264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644c06fb70 con 0x7f644c14f2a0 2023-03-03T08:44:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.596+0000 7f645431b700 1 -- 172.21.15.7:0/875584264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644c0aa430 con 0x7f644c14f2a0 2023-03-03T08:44:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.597+0000 7f645431b700 1 -- 172.21.15.7:0/875584264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644c14ea80 con 0x7f644c14f2a0 2023-03-03T08:44:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.597+0000 7f6442ffd700 1 -- 172.21.15.7:0/875584264 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6444005470 con 0x7f644c14f2a0 2023-03-03T08:44:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.598+0000 7f6442ffd700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64300682c0 0x7f643006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.598+0000 7f6442ffd700 1 -- 172.21.15.7:0/875584264 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6444086080 con 0x7f644c14f2a0 2023-03-03T08:44:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.598+0000 7f64520b7700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64300682c0 0x7f643006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.598+0000 7f64520b7700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64300682c0 0x7f643006a770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f64480090b0 tx=0x7f644801f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.602+0000 7f6442ffd700 1 -- 172.21.15.7:0/875584264 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f644c14ea80 con 0x7f644c14f2a0 2023-03-03T08:44:07.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.793+0000 7f645431b700 1 -- 172.21.15.7:0/875584264 --> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f644c14ea80 con 0x7f64300682c0 2023-03-03T08:44:07.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.808+0000 7f6442ffd700 1 -- 172.21.15.7:0/875584264 <== mgr.4102 v2:172.21.15.143:6832/96367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525124 (secure 0 0 0) 0x7f644c14ea80 con 0x7f64300682c0 2023-03-03T08:44:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 -- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64300682c0 msgr2=0x7f643006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64300682c0 0x7f643006a770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f64480090b0 tx=0x7f644801f040 comp rx=0 tx=0).stop 2023-03-03T08:44:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 -- 172.21.15.7:0/875584264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c14f2a0 msgr2=0x7f644c1473e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c14f2a0 0x7f644c1473e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f644c0ada10 tx=0x7f644400ee70 comp rx=0 tx=0).stop 2023-03-03T08:44:07.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 -- 172.21.15.7:0/875584264 shutdown_connections 2023-03-03T08:44:07.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64300682c0 0x7f643006a770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c14f2a0 0x7f644c1473e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6438003aa0 0x7f644c14d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 --2- 172.21.15.7:0/875584264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f644c14d5f0 0x7f644c14ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 -- 172.21.15.7:0/875584264 >> 172.21.15.7:0/875584264 conn(0x7f644c0c8900 msgr2=0x7f644c1533f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:07.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 -- 172.21.15.7:0/875584264 shutdown_connections 2023-03-03T08:44:07.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.812+0000 7f645431b700 1 -- 172.21.15.7:0/875584264 wait complete. 2023-03-03T08:44:07.820 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-03-03T08:44:07.825 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2023-03-03T08:44:07.114985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597490,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":255,"num_write_kb":2450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68116,"kb_used_data":3540,"kb_used_omap":0,"kb_used_meta":64512,"kb_avail":754906604,"statfs":{"total":773094113280,"available":773024362496,"internally_reserved":0,"allocated":3624960,"data_stored":2144653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66060288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2462,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003970"},"pg_stats":[{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980166+0000","last_change":"2023-03-03T08:43:43.997091+0000","last_active":"2023-03-03T08:43:44.980166+0000","last_peered":"2023-03-03T08:43:44.980166+0000","last_clean":"2023-03-03T08:43:44.980166+0000","last_became_active":"2023-03-03T08:43:43.996805+0000","last_became_peered":"2023-03-03T08:43:43.996805+0000","last_unstale":"2023-03-03T08:43:44.980166+0000","last_undegraded":"2023-03-03T08:43:44.980166+0000","last_fullsized":"2023-03-03T08:43:44.980166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982505+0000","last_change":"2023-03-03T08:43:34.546828+0000","last_active":"2023-03-03T08:43:43.982505+0000","last_peered":"2023-03-03T08:43:43.982505+0000","last_clean":"2023-03-03T08:43:43.982505+0000","last_became_active":"2023-03-03T08:43:34.546559+0000","last_became_peered":"2023-03-03T08:43:34.546559+0000","last_unstale":"2023-03-03T08:43:43.982505+0000","last_undegraded":"2023-03-03T08:43:43.982505+0000","last_fullsized":"2023-03-03T08:43:43.982505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:21:14.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106061+0000","last_change":"2023-03-03T08:43:43.995073+0000","last_active":"2023-03-03T08:43:45.106061+0000","last_peered":"2023-03-03T08:43:45.106061+0000","last_clean":"2023-03-03T08:43:45.106061+0000","last_became_active":"2023-03-03T08:43:43.994797+0000","last_became_peered":"2023-03-03T08:43:43.994797+0000","last_unstale":"2023-03-03T08:43:45.106061+0000","last_undegraded":"2023-03-03T08:43:45.106061+0000","last_fullsized":"2023-03-03T08:43:45.106061+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:21:52.642936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982397+0000","last_change":"2023-03-03T08:43:34.552931+0000","last_active":"2023-03-03T08:43:43.982397+0000","last_peered":"2023-03-03T08:43:43.982397+0000","last_clean":"2023-03-03T08:43:43.982397+0000","last_became_active":"2023-03-03T08:43:34.552623+0000","last_became_peered":"2023-03-03T08:43:34.552623+0000","last_unstale":"2023-03-03T08:43:43.982397+0000","last_undegraded":"2023-03-03T08:43:43.982397+0000","last_fullsized":"2023-03-03T08:43:43.982397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:07:43.954897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979088+0000","last_change":"2023-03-03T08:43:43.999245+0000","last_active":"2023-03-03T08:43:44.979088+0000","last_peered":"2023-03-03T08:43:44.979088+0000","last_clean":"2023-03-03T08:43:44.979088+0000","last_became_active":"2023-03-03T08:43:43.998592+0000","last_became_peered":"2023-03-03T08:43:43.998592+0000","last_unstale":"2023-03-03T08:43:44.979088+0000","last_undegraded":"2023-03-03T08:43:44.979088+0000","last_fullsized":"2023-03-03T08:43:44.979088+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:23.788724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991054+0000","last_change":"2023-03-03T08:43:34.544408+0000","last_active":"2023-03-03T08:43:43.991054+0000","last_peered":"2023-03-03T08:43:43.991054+0000","last_clean":"2023-03-03T08:43:43.991054+0000","last_became_active":"2023-03-03T08:43:34.543977+0000","last_became_peered":"2023-03-03T08:43:34.543977+0000","last_unstale":"2023-03-03T08:43:43.991054+0000","last_undegraded":"2023-03-03T08:43:43.991054+0000","last_fullsized":"2023-03-03T08:43:43.991054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.074082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105254+0000","last_change":"2023-03-03T08:43:43.994545+0000","last_active":"2023-03-03T08:43:45.105254+0000","last_peered":"2023-03-03T08:43:45.105254+0000","last_clean":"2023-03-03T08:43:45.105254+0000","last_became_active":"2023-03-03T08:43:43.994271+0000","last_became_peered":"2023-03-03T08:43:43.994271+0000","last_unstale":"2023-03-03T08:43:45.105254+0000","last_undegraded":"2023-03-03T08:43:45.105254+0000","last_fullsized":"2023-03-03T08:43:45.105254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:15.925960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989064+0000","last_change":"2023-03-03T08:43:34.538786+0000","last_active":"2023-03-03T08:43:43.989064+0000","last_peered":"2023-03-03T08:43:43.989064+0000","last_clean":"2023-03-03T08:43:43.989064+0000","last_became_active":"2023-03-03T08:43:34.538494+0000","last_became_peered":"2023-03-03T08:43:34.538494+0000","last_unstale":"2023-03-03T08:43:43.989064+0000","last_undegraded":"2023-03-03T08:43:43.989064+0000","last_fullsized":"2023-03-03T08:43:43.989064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:53.838286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105391+0000","last_change":"2023-03-03T08:43:43.993678+0000","last_active":"2023-03-03T08:43:45.105391+0000","last_peered":"2023-03-03T08:43:45.105391+0000","last_clean":"2023-03-03T08:43:45.105391+0000","last_became_active":"2023-03-03T08:43:43.993402+0000","last_became_peered":"2023-03-03T08:43:43.993402+0000","last_unstale":"2023-03-03T08:43:45.105391+0000","last_undegraded":"2023-03-03T08:43:45.105391+0000","last_fullsized":"2023-03-03T08:43:45.105391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:31:00.747216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983479+0000","last_change":"2023-03-03T08:43:34.547197+0000","last_active":"2023-03-03T08:43:43.983479+0000","last_peered":"2023-03-03T08:43:43.983479+0000","last_clean":"2023-03-03T08:43:43.983479+0000","last_became_active":"2023-03-03T08:43:34.546927+0000","last_became_peered":"2023-03-03T08:43:34.546927+0000","last_unstale":"2023-03-03T08:43:43.983479+0000","last_undegraded":"2023-03-03T08:43:43.983479+0000","last_fullsized":"2023-03-03T08:43:43.983479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:21:34.845968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984318+0000","last_change":"2023-03-03T08:43:34.540812+0000","last_active":"2023-03-03T08:43:43.984318+0000","last_peered":"2023-03-03T08:43:43.984318+0000","last_clean":"2023-03-03T08:43:43.984318+0000","last_became_active":"2023-03-03T08:43:34.540549+0000","last_became_peered":"2023-03-03T08:43:34.540549+0000","last_unstale":"2023-03-03T08:43:43.984318+0000","last_undegraded":"2023-03-03T08:43:43.984318+0000","last_fullsized":"2023-03-03T08:43:43.984318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982875+0000","last_change":"2023-03-03T08:43:34.108426+0000","last_active":"2023-03-03T08:43:43.982875+0000","last_peered":"2023-03-03T08:43:43.982875+0000","last_clean":"2023-03-03T08:43:43.982875+0000","last_became_active":"2023-03-03T08:43:34.108009+0000","last_became_peered":"2023-03-03T08:43:34.108009+0000","last_unstale":"2023-03-03T08:43:43.982875+0000","last_undegraded":"2023-03-03T08:43:43.982875+0000","last_fullsized":"2023-03-03T08:43:43.982875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:53.632744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979479+0000","last_change":"2023-03-03T08:43:43.575708+0000","last_active":"2023-03-03T08:43:44.979479+0000","last_peered":"2023-03-03T08:43:44.979479+0000","last_clean":"2023-03-03T08:43:44.979479+0000","last_became_active":"2023-03-03T08:43:43.575417+0000","last_became_peered":"2023-03-03T08:43:43.575417+0000","last_unstale":"2023-03-03T08:43:44.979479+0000","last_undegraded":"2023-03-03T08:43:44.979479+0000","last_fullsized":"2023-03-03T08:43:44.979479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:25.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106018+0000","last_change":"2023-03-03T08:43:43.999274+0000","last_active":"2023-03-03T08:43:45.106018+0000","last_peered":"2023-03-03T08:43:45.106018+0000","last_clean":"2023-03-03T08:43:45.106018+0000","last_became_active":"2023-03-03T08:43:43.998823+0000","last_became_peered":"2023-03-03T08:43:43.998823+0000","last_unstale":"2023-03-03T08:43:45.106018+0000","last_undegraded":"2023-03-03T08:43:45.106018+0000","last_fullsized":"2023-03-03T08:43:45.106018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:48.799261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992918+0000","last_change":"2023-03-03T08:43:34.532482+0000","last_active":"2023-03-03T08:43:43.992918+0000","last_peered":"2023-03-03T08:43:43.992918+0000","last_clean":"2023-03-03T08:43:43.992918+0000","last_became_active":"2023-03-03T08:43:34.531751+0000","last_became_peered":"2023-03-03T08:43:34.531751+0000","last_unstale":"2023-03-03T08:43:43.992918+0000","last_undegraded":"2023-03-03T08:43:43.992918+0000","last_fullsized":"2023-03-03T08:43:43.992918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:47:45.214945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982691+0000","last_change":"2023-03-03T08:43:34.109782+0000","last_active":"2023-03-03T08:43:43.982691+0000","last_peered":"2023-03-03T08:43:43.982691+0000","last_clean":"2023-03-03T08:43:43.982691+0000","last_became_active":"2023-03-03T08:43:34.109513+0000","last_became_peered":"2023-03-03T08:43:34.109513+0000","last_unstale":"2023-03-03T08:43:43.982691+0000","last_undegraded":"2023-03-03T08:43:43.982691+0000","last_fullsized":"2023-03-03T08:43:43.982691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:16.185161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105634+0000","last_change":"2023-03-03T08:43:43.576885+0000","last_active":"2023-03-03T08:43:45.105634+0000","last_peered":"2023-03-03T08:43:45.105634+0000","last_clean":"2023-03-03T08:43:45.105634+0000","last_became_active":"2023-03-03T08:43:43.576577+0000","last_became_peered":"2023-03-03T08:43:43.576577+0000","last_unstale":"2023-03-03T08:43:45.105634+0000","last_undegraded":"2023-03-03T08:43:45.105634+0000","last_fullsized":"2023-03-03T08:43:45.105634+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:26.501956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978988+0000","last_change":"2023-03-03T08:43:43.992020+0000","last_active":"2023-03-03T08:43:44.978988+0000","last_peered":"2023-03-03T08:43:44.978988+0000","last_clean":"2023-03-03T08:43:44.978988+0000","last_became_active":"2023-03-03T08:43:43.991674+0000","last_became_peered":"2023-03-03T08:43:43.991674+0000","last_unstale":"2023-03-03T08:43:44.978988+0000","last_undegraded":"2023-03-03T08:43:44.978988+0000","last_fullsized":"2023-03-03T08:43:44.978988+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:54:30.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985919+0000","last_change":"2023-03-03T08:43:34.535451+0000","last_active":"2023-03-03T08:43:43.985919+0000","last_peered":"2023-03-03T08:43:43.985919+0000","last_clean":"2023-03-03T08:43:43.985919+0000","last_became_active":"2023-03-03T08:43:34.535129+0000","last_became_peered":"2023-03-03T08:43:34.535129+0000","last_unstale":"2023-03-03T08:43:43.985919+0000","last_undegraded":"2023-03-03T08:43:43.985919+0000","last_fullsized":"2023-03-03T08:43:43.985919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:30.645128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981082+0000","last_change":"2023-03-03T08:43:34.117851+0000","last_active":"2023-03-03T08:43:43.981082+0000","last_peered":"2023-03-03T08:43:43.981082+0000","last_clean":"2023-03-03T08:43:43.981082+0000","last_became_active":"2023-03-03T08:43:34.117548+0000","last_became_peered":"2023-03-03T08:43:34.117548+0000","last_unstale":"2023-03-03T08:43:43.981082+0000","last_undegraded":"2023-03-03T08:43:43.981082+0000","last_fullsized":"2023-03-03T08:43:43.981082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:20.039817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106904+0000","last_change":"2023-03-03T08:43:43.569082+0000","last_active":"2023-03-03T08:43:45.106904+0000","last_peered":"2023-03-03T08:43:45.106904+0000","last_clean":"2023-03-03T08:43:45.106904+0000","last_became_active":"2023-03-03T08:43:43.568838+0000","last_became_peered":"2023-03-03T08:43:43.568838+0000","last_unstale":"2023-03-03T08:43:45.106904+0000","last_undegraded":"2023-03-03T08:43:45.106904+0000","last_fullsized":"2023-03-03T08:43:45.106904+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:47.610848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979556+0000","last_change":"2023-03-03T08:43:43.995210+0000","last_active":"2023-03-03T08:43:44.979556+0000","last_peered":"2023-03-03T08:43:44.979556+0000","last_clean":"2023-03-03T08:43:44.979556+0000","last_became_active":"2023-03-03T08:43:43.994899+0000","last_became_peered":"2023-03-03T08:43:43.994899+0000","last_unstale":"2023-03-03T08:43:44.979556+0000","last_undegraded":"2023-03-03T08:43:44.979556+0000","last_fullsized":"2023-03-03T08:43:44.979556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:37:11.974933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985908+0000","last_change":"2023-03-03T08:43:34.547322+0000","last_active":"2023-03-03T08:43:43.985908+0000","last_peered":"2023-03-03T08:43:43.985908+0000","last_clean":"2023-03-03T08:43:43.985908+0000","last_became_active":"2023-03-03T08:43:34.546692+0000","last_became_peered":"2023-03-03T08:43:34.546692+0000","last_unstale":"2023-03-03T08:43:43.985908+0000","last_undegraded":"2023-03-03T08:43:43.985908+0000","last_fullsized":"2023-03-03T08:43:43.985908+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:11:26.366920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982836+0000","last_change":"2023-03-03T08:43:34.110187+0000","last_active":"2023-03-03T08:43:43.982836+0000","last_peered":"2023-03-03T08:43:43.982836+0000","last_clean":"2023-03-03T08:43:43.982836+0000","last_became_active":"2023-03-03T08:43:34.109845+0000","last_became_peered":"2023-03-03T08:43:34.109845+0000","last_unstale":"2023-03-03T08:43:43.982836+0000","last_undegraded":"2023-03-03T08:43:43.982836+0000","last_fullsized":"2023-03-03T08:43:43.982836+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:33.507500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105532+0000","last_change":"2023-03-03T08:43:43.576805+0000","last_active":"2023-03-03T08:43:45.105532+0000","last_peered":"2023-03-03T08:43:45.105532+0000","last_clean":"2023-03-03T08:43:45.105532+0000","last_became_active":"2023-03-03T08:43:43.576241+0000","last_became_peered":"2023-03-03T08:43:43.576241+0000","last_unstale":"2023-03-03T08:43:45.105532+0000","last_undegraded":"2023-03-03T08:43:45.105532+0000","last_fullsized":"2023-03-03T08:43:45.105532+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:34:35.821307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105632+0000","last_change":"2023-03-03T08:43:43.991142+0000","last_active":"2023-03-03T08:43:45.105632+0000","last_peered":"2023-03-03T08:43:45.105632+0000","last_clean":"2023-03-03T08:43:45.105632+0000","last_became_active":"2023-03-03T08:43:43.990868+0000","last_became_peered":"2023-03-03T08:43:43.990868+0000","last_unstale":"2023-03-03T08:43:45.105632+0000","last_undegraded":"2023-03-03T08:43:45.105632+0000","last_fullsized":"2023-03-03T08:43:45.105632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:58.831806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106429+0000","last_change":"2023-03-03T08:43:43.995618+0000","last_active":"2023-03-03T08:43:45.106429+0000","last_peered":"2023-03-03T08:43:45.106429+0000","last_clean":"2023-03-03T08:43:45.106429+0000","last_became_active":"2023-03-03T08:43:43.995350+0000","last_became_peered":"2023-03-03T08:43:43.995350+0000","last_unstale":"2023-03-03T08:43:45.106429+0000","last_undegraded":"2023-03-03T08:43:45.106429+0000","last_fullsized":"2023-03-03T08:43:45.106429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:46.445223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988843+0000","last_change":"2023-03-03T08:43:34.541172+0000","last_active":"2023-03-03T08:43:43.988843+0000","last_peered":"2023-03-03T08:43:43.988843+0000","last_clean":"2023-03-03T08:43:43.988843+0000","last_became_active":"2023-03-03T08:43:34.540739+0000","last_became_peered":"2023-03-03T08:43:34.540739+0000","last_unstale":"2023-03-03T08:43:43.988843+0000","last_undegraded":"2023-03-03T08:43:43.988843+0000","last_fullsized":"2023-03-03T08:43:43.988843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:40.058265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987475+0000","last_change":"2023-03-03T08:43:34.112667+0000","last_active":"2023-03-03T08:43:43.987475+0000","last_peered":"2023-03-03T08:43:43.987475+0000","last_clean":"2023-03-03T08:43:43.987475+0000","last_became_active":"2023-03-03T08:43:34.112386+0000","last_became_peered":"2023-03-03T08:43:34.112386+0000","last_unstale":"2023-03-03T08:43:43.987475+0000","last_undegraded":"2023-03-03T08:43:43.987475+0000","last_fullsized":"2023-03-03T08:43:43.987475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:40.504392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980167+0000","last_change":"2023-03-03T08:43:43.573962+0000","last_active":"2023-03-03T08:43:44.980167+0000","last_peered":"2023-03-03T08:43:44.980167+0000","last_clean":"2023-03-03T08:43:44.980167+0000","last_became_active":"2023-03-03T08:43:43.573200+0000","last_became_peered":"2023-03-03T08:43:43.573200+0000","last_unstale":"2023-03-03T08:43:44.980167+0000","last_undegraded":"2023-03-03T08:43:44.980167+0000","last_fullsized":"2023-03-03T08:43:44.980167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:29.511681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984698+0000","last_change":"2023-03-03T08:43:34.537850+0000","last_active":"2023-03-03T08:43:43.984698+0000","last_peered":"2023-03-03T08:43:43.984698+0000","last_clean":"2023-03-03T08:43:43.984698+0000","last_became_active":"2023-03-03T08:43:34.537550+0000","last_became_peered":"2023-03-03T08:43:34.537550+0000","last_unstale":"2023-03-03T08:43:43.984698+0000","last_undegraded":"2023-03-03T08:43:43.984698+0000","last_fullsized":"2023-03-03T08:43:43.984698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:37:36.893757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990299+0000","last_change":"2023-03-03T08:43:34.109127+0000","last_active":"2023-03-03T08:43:43.990299+0000","last_peered":"2023-03-03T08:43:43.990299+0000","last_clean":"2023-03-03T08:43:43.990299+0000","last_became_active":"2023-03-03T08:43:34.108869+0000","last_became_peered":"2023-03-03T08:43:34.108869+0000","last_unstale":"2023-03-03T08:43:43.990299+0000","last_undegraded":"2023-03-03T08:43:43.990299+0000","last_fullsized":"2023-03-03T08:43:43.990299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:19:00.275620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981298+0000","last_change":"2023-03-03T08:43:29.354627+0000","last_active":"2023-03-03T08:43:43.981298+0000","last_peered":"2023-03-03T08:43:43.981298+0000","last_clean":"2023-03-03T08:43:43.981298+0000","last_became_active":"2023-03-03T08:43:29.354271+0000","last_became_peered":"2023-03-03T08:43:29.354271+0000","last_unstale":"2023-03-03T08:43:43.981298+0000","last_undegraded":"2023-03-03T08:43:43.981298+0000","last_fullsized":"2023-03-03T08:43:43.981298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:05.946053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979432+0000","last_change":"2023-03-03T08:43:43.575567+0000","last_active":"2023-03-03T08:43:44.979432+0000","last_peered":"2023-03-03T08:43:44.979432+0000","last_clean":"2023-03-03T08:43:44.979432+0000","last_became_active":"2023-03-03T08:43:43.575276+0000","last_became_peered":"2023-03-03T08:43:43.575276+0000","last_unstale":"2023-03-03T08:43:44.979432+0000","last_undegraded":"2023-03-03T08:43:44.979432+0000","last_fullsized":"2023-03-03T08:43:44.979432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:12.876790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107250+0000","last_change":"2023-03-03T08:43:43.998276+0000","last_active":"2023-03-03T08:43:45.107250+0000","last_peered":"2023-03-03T08:43:45.107250+0000","last_clean":"2023-03-03T08:43:45.107250+0000","last_became_active":"2023-03-03T08:43:43.996018+0000","last_became_peered":"2023-03-03T08:43:43.996018+0000","last_unstale":"2023-03-03T08:43:45.107250+0000","last_undegraded":"2023-03-03T08:43:45.107250+0000","last_fullsized":"2023-03-03T08:43:45.107250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:08.588278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988844+0000","last_change":"2023-03-03T08:43:34.539018+0000","last_active":"2023-03-03T08:43:43.988844+0000","last_peered":"2023-03-03T08:43:43.988844+0000","last_clean":"2023-03-03T08:43:43.988844+0000","last_became_active":"2023-03-03T08:43:34.538375+0000","last_became_peered":"2023-03-03T08:43:34.538375+0000","last_unstale":"2023-03-03T08:43:43.988844+0000","last_undegraded":"2023-03-03T08:43:43.988844+0000","last_fullsized":"2023-03-03T08:43:43.988844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:29:02.601410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990482+0000","last_change":"2023-03-03T08:43:29.354519+0000","last_active":"2023-03-03T08:43:43.990482+0000","last_peered":"2023-03-03T08:43:43.990482+0000","last_clean":"2023-03-03T08:43:43.990482+0000","last_became_active":"2023-03-03T08:43:29.354213+0000","last_became_peered":"2023-03-03T08:43:29.354213+0000","last_unstale":"2023-03-03T08:43:43.990482+0000","last_undegraded":"2023-03-03T08:43:43.990482+0000","last_fullsized":"2023-03-03T08:43:43.990482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:00.406507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990028+0000","last_change":"2023-03-03T08:43:34.110105+0000","last_active":"2023-03-03T08:43:43.990028+0000","last_peered":"2023-03-03T08:43:43.990028+0000","last_clean":"2023-03-03T08:43:43.990028+0000","last_became_active":"2023-03-03T08:43:34.109736+0000","last_became_peered":"2023-03-03T08:43:34.109736+0000","last_unstale":"2023-03-03T08:43:43.990028+0000","last_undegraded":"2023-03-03T08:43:43.990028+0000","last_fullsized":"2023-03-03T08:43:43.990028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:30.557847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105361+0000","last_change":"2023-03-03T08:43:43.569433+0000","last_active":"2023-03-03T08:43:45.105361+0000","last_peered":"2023-03-03T08:43:45.105361+0000","last_clean":"2023-03-03T08:43:45.105361+0000","last_became_active":"2023-03-03T08:43:43.568711+0000","last_became_peered":"2023-03-03T08:43:43.568711+0000","last_unstale":"2023-03-03T08:43:45.105361+0000","last_undegraded":"2023-03-03T08:43:45.105361+0000","last_fullsized":"2023-03-03T08:43:45.105361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:53.170557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105548+0000","last_change":"2023-03-03T08:43:43.999241+0000","last_active":"2023-03-03T08:43:45.105548+0000","last_peered":"2023-03-03T08:43:45.105548+0000","last_clean":"2023-03-03T08:43:45.105548+0000","last_became_active":"2023-03-03T08:43:43.998992+0000","last_became_peered":"2023-03-03T08:43:43.998992+0000","last_unstale":"2023-03-03T08:43:45.105548+0000","last_undegraded":"2023-03-03T08:43:45.105548+0000","last_fullsized":"2023-03-03T08:43:45.105548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:22.623700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991092+0000","last_change":"2023-03-03T08:43:34.535435+0000","last_active":"2023-03-03T08:43:43.991092+0000","last_peered":"2023-03-03T08:43:43.991092+0000","last_clean":"2023-03-03T08:43:43.991092+0000","last_became_active":"2023-03-03T08:43:34.535123+0000","last_became_peered":"2023-03-03T08:43:34.535123+0000","last_unstale":"2023-03-03T08:43:43.991092+0000","last_undegraded":"2023-03-03T08:43:43.991092+0000","last_fullsized":"2023-03-03T08:43:43.991092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:52:10.622422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990432+0000","last_change":"2023-03-03T08:43:34.107783+0000","last_active":"2023-03-03T08:43:43.990432+0000","last_peered":"2023-03-03T08:43:43.990432+0000","last_clean":"2023-03-03T08:43:43.990432+0000","last_became_active":"2023-03-03T08:43:34.107503+0000","last_became_peered":"2023-03-03T08:43:34.107503+0000","last_unstale":"2023-03-03T08:43:43.990432+0000","last_undegraded":"2023-03-03T08:43:43.990432+0000","last_fullsized":"2023-03-03T08:43:43.990432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:35:42.800814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981038+0000","last_change":"2023-03-03T08:43:29.072304+0000","last_active":"2023-03-03T08:43:43.981038+0000","last_peered":"2023-03-03T08:43:43.981038+0000","last_clean":"2023-03-03T08:43:43.981038+0000","last_became_active":"2023-03-03T08:43:29.071962+0000","last_became_peered":"2023-03-03T08:43:29.071962+0000","last_unstale":"2023-03-03T08:43:43.981038+0000","last_undegraded":"2023-03-03T08:43:43.981038+0000","last_fullsized":"2023-03-03T08:43:43.981038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:40:12.229226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106349+0000","last_change":"2023-03-03T08:43:43.575964+0000","last_active":"2023-03-03T08:43:45.106349+0000","last_peered":"2023-03-03T08:43:45.106349+0000","last_clean":"2023-03-03T08:43:45.106349+0000","last_became_active":"2023-03-03T08:43:43.575697+0000","last_became_peered":"2023-03-03T08:43:43.575697+0000","last_unstale":"2023-03-03T08:43:45.106349+0000","last_undegraded":"2023-03-03T08:43:45.106349+0000","last_fullsized":"2023-03-03T08:43:45.106349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:01.896050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978782+0000","last_change":"2023-03-03T08:43:44.006414+0000","last_active":"2023-03-03T08:43:44.978782+0000","last_peered":"2023-03-03T08:43:44.978782+0000","last_clean":"2023-03-03T08:43:44.978782+0000","last_became_active":"2023-03-03T08:43:44.006132+0000","last_became_peered":"2023-03-03T08:43:44.006132+0000","last_unstale":"2023-03-03T08:43:44.978782+0000","last_undegraded":"2023-03-03T08:43:44.978782+0000","last_fullsized":"2023-03-03T08:43:44.978782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:55.006419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982772+0000","last_change":"2023-03-03T08:43:34.544053+0000","last_active":"2023-03-03T08:43:43.982772+0000","last_peered":"2023-03-03T08:43:43.982772+0000","last_clean":"2023-03-03T08:43:43.982772+0000","last_became_active":"2023-03-03T08:43:34.543523+0000","last_became_peered":"2023-03-03T08:43:34.543523+0000","last_unstale":"2023-03-03T08:43:43.982772+0000","last_undegraded":"2023-03-03T08:43:43.982772+0000","last_fullsized":"2023-03-03T08:43:43.982772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:34:03.662535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989069+0000","last_change":"2023-03-03T08:43:34.104492+0000","last_active":"2023-03-03T08:43:43.989069+0000","last_peered":"2023-03-03T08:43:43.989069+0000","last_clean":"2023-03-03T08:43:43.989069+0000","last_became_active":"2023-03-03T08:43:34.104151+0000","last_became_peered":"2023-03-03T08:43:34.104151+0000","last_unstale":"2023-03-03T08:43:43.989069+0000","last_undegraded":"2023-03-03T08:43:43.989069+0000","last_fullsized":"2023-03-03T08:43:43.989069+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:57.214738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979395+0000","last_change":"2023-03-03T08:43:29.075836+0000","last_active":"2023-03-03T08:43:43.979395+0000","last_peered":"2023-03-03T08:43:43.979395+0000","last_clean":"2023-03-03T08:43:43.979395+0000","last_became_active":"2023-03-03T08:43:29.075564+0000","last_became_peered":"2023-03-03T08:43:29.075564+0000","last_unstale":"2023-03-03T08:43:43.979395+0000","last_undegraded":"2023-03-03T08:43:43.979395+0000","last_fullsized":"2023-03-03T08:43:43.979395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:39.576714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107224+0000","last_change":"2023-03-03T08:43:43.575599+0000","last_active":"2023-03-03T08:43:45.107224+0000","last_peered":"2023-03-03T08:43:45.107224+0000","last_clean":"2023-03-03T08:43:45.107224+0000","last_became_active":"2023-03-03T08:43:43.575318+0000","last_became_peered":"2023-03-03T08:43:43.575318+0000","last_unstale":"2023-03-03T08:43:45.107224+0000","last_undegraded":"2023-03-03T08:43:45.107224+0000","last_fullsized":"2023-03-03T08:43:45.107224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:28.806052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105269+0000","last_change":"2023-03-03T08:43:44.000500+0000","last_active":"2023-03-03T08:43:45.105269+0000","last_peered":"2023-03-03T08:43:45.105269+0000","last_clean":"2023-03-03T08:43:45.105269+0000","last_became_active":"2023-03-03T08:43:43.999560+0000","last_became_peered":"2023-03-03T08:43:43.999560+0000","last_unstale":"2023-03-03T08:43:45.105269+0000","last_undegraded":"2023-03-03T08:43:45.105269+0000","last_fullsized":"2023-03-03T08:43:45.105269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:29:54.240309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981172+0000","last_change":"2023-03-03T08:43:34.550263+0000","last_active":"2023-03-03T08:43:43.981172+0000","last_peered":"2023-03-03T08:43:43.981172+0000","last_clean":"2023-03-03T08:43:43.981172+0000","last_became_active":"2023-03-03T08:43:34.549560+0000","last_became_peered":"2023-03-03T08:43:34.549560+0000","last_unstale":"2023-03-03T08:43:43.981172+0000","last_undegraded":"2023-03-03T08:43:43.981172+0000","last_fullsized":"2023-03-03T08:43:43.981172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:55.874223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987062+0000","last_change":"2023-03-03T08:43:34.110484+0000","last_active":"2023-03-03T08:43:43.987062+0000","last_peered":"2023-03-03T08:43:43.987062+0000","last_clean":"2023-03-03T08:43:43.987062+0000","last_became_active":"2023-03-03T08:43:34.110220+0000","last_became_peered":"2023-03-03T08:43:34.110220+0000","last_unstale":"2023-03-03T08:43:43.987062+0000","last_undegraded":"2023-03-03T08:43:43.987062+0000","last_fullsized":"2023-03-03T08:43:43.987062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:04:59.232537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982439+0000","last_change":"2023-03-03T08:43:23.015460+0000","last_active":"2023-03-03T08:43:43.982439+0000","last_peered":"2023-03-03T08:43:43.982439+0000","last_clean":"2023-03-03T08:43:43.982439+0000","last_became_active":"2023-03-03T08:43:23.014485+0000","last_became_peered":"2023-03-03T08:43:23.014485+0000","last_unstale":"2023-03-03T08:43:43.982439+0000","last_undegraded":"2023-03-03T08:43:43.982439+0000","last_fullsized":"2023-03-03T08:43:43.982439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_clean_scrub_stamp":"2023-03-03T08:43:21.542883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:07.022017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980803+0000","last_change":"2023-03-03T08:43:29.160285+0000","last_active":"2023-03-03T08:43:43.980803+0000","last_peered":"2023-03-03T08:43:43.980803+0000","last_clean":"2023-03-03T08:43:43.980803+0000","last_became_active":"2023-03-03T08:43:29.159943+0000","last_became_peered":"2023-03-03T08:43:29.159943+0000","last_unstale":"2023-03-03T08:43:43.980803+0000","last_undegraded":"2023-03-03T08:43:43.980803+0000","last_fullsized":"2023-03-03T08:43:43.980803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:48:58.997356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107580+0000","last_change":"2023-03-03T08:43:43.575212+0000","last_active":"2023-03-03T08:43:45.107580+0000","last_peered":"2023-03-03T08:43:45.107580+0000","last_clean":"2023-03-03T08:43:45.107580+0000","last_became_active":"2023-03-03T08:43:43.574897+0000","last_became_peered":"2023-03-03T08:43:43.574897+0000","last_unstale":"2023-03-03T08:43:45.107580+0000","last_undegraded":"2023-03-03T08:43:45.107580+0000","last_fullsized":"2023-03-03T08:43:45.107580+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:33:07.674313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107370+0000","last_change":"2023-03-03T08:43:43.992952+0000","last_active":"2023-03-03T08:43:45.107370+0000","last_peered":"2023-03-03T08:43:45.107370+0000","last_clean":"2023-03-03T08:43:45.107370+0000","last_became_active":"2023-03-03T08:43:43.992535+0000","last_became_peered":"2023-03-03T08:43:43.992535+0000","last_unstale":"2023-03-03T08:43:45.107370+0000","last_undegraded":"2023-03-03T08:43:45.107370+0000","last_fullsized":"2023-03-03T08:43:45.107370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:10:41.617430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991335+0000","last_change":"2023-03-03T08:43:34.536419+0000","last_active":"2023-03-03T08:43:43.991335+0000","last_peered":"2023-03-03T08:43:43.991335+0000","last_clean":"2023-03-03T08:43:43.991335+0000","last_became_active":"2023-03-03T08:43:34.536150+0000","last_became_peered":"2023-03-03T08:43:34.536150+0000","last_unstale":"2023-03-03T08:43:43.991335+0000","last_undegraded":"2023-03-03T08:43:43.991335+0000","last_fullsized":"2023-03-03T08:43:43.991335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:22:54.362024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981827+0000","last_change":"2023-03-03T08:43:34.106337+0000","last_active":"2023-03-03T08:43:43.981827+0000","last_peered":"2023-03-03T08:43:43.981827+0000","last_clean":"2023-03-03T08:43:43.981827+0000","last_became_active":"2023-03-03T08:43:34.105753+0000","last_became_peered":"2023-03-03T08:43:34.105753+0000","last_unstale":"2023-03-03T08:43:43.981827+0000","last_undegraded":"2023-03-03T08:43:43.981827+0000","last_fullsized":"2023-03-03T08:43:43.981827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:46.761924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991590+0000","last_change":"2023-03-03T08:43:29.076367+0000","last_active":"2023-03-03T08:43:43.991590+0000","last_peered":"2023-03-03T08:43:43.991590+0000","last_clean":"2023-03-03T08:43:43.991590+0000","last_became_active":"2023-03-03T08:43:29.075982+0000","last_became_peered":"2023-03-03T08:43:29.075982+0000","last_unstale":"2023-03-03T08:43:43.991590+0000","last_undegraded":"2023-03-03T08:43:43.991590+0000","last_fullsized":"2023-03-03T08:43:43.991590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:06.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:46.143717+0000","last_change":"2023-03-03T08:43:43.577670+0000","last_active":"2023-03-03T08:43:46.143717+0000","last_peered":"2023-03-03T08:43:46.143717+0000","last_clean":"2023-03-03T08:43:46.143717+0000","last_became_active":"2023-03-03T08:43:43.577371+0000","last_became_peered":"2023-03-03T08:43:43.577371+0000","last_unstale":"2023-03-03T08:43:46.143717+0000","last_undegraded":"2023-03-03T08:43:46.143717+0000","last_fullsized":"2023-03-03T08:43:46.143717+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:18.377654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108041+0000","last_change":"2023-03-03T08:43:43.993077+0000","last_active":"2023-03-03T08:43:45.108041+0000","last_peered":"2023-03-03T08:43:45.108041+0000","last_clean":"2023-03-03T08:43:45.108041+0000","last_became_active":"2023-03-03T08:43:43.992021+0000","last_became_peered":"2023-03-03T08:43:43.992021+0000","last_unstale":"2023-03-03T08:43:45.108041+0000","last_undegraded":"2023-03-03T08:43:45.108041+0000","last_fullsized":"2023-03-03T08:43:45.108041+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:58.237553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990193+0000","last_change":"2023-03-03T08:43:34.528593+0000","last_active":"2023-03-03T08:43:43.990193+0000","last_peered":"2023-03-03T08:43:43.990193+0000","last_clean":"2023-03-03T08:43:43.990193+0000","last_became_active":"2023-03-03T08:43:34.528260+0000","last_became_peered":"2023-03-03T08:43:34.528260+0000","last_unstale":"2023-03-03T08:43:43.990193+0000","last_undegraded":"2023-03-03T08:43:43.990193+0000","last_fullsized":"2023-03-03T08:43:43.990193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988924+0000","last_change":"2023-03-03T08:43:34.113748+0000","last_active":"2023-03-03T08:43:43.988924+0000","last_peered":"2023-03-03T08:43:43.988924+0000","last_clean":"2023-03-03T08:43:43.988924+0000","last_became_active":"2023-03-03T08:43:34.113190+0000","last_became_peered":"2023-03-03T08:43:34.113190+0000","last_unstale":"2023-03-03T08:43:43.988924+0000","last_undegraded":"2023-03-03T08:43:43.988924+0000","last_fullsized":"2023-03-03T08:43:43.988924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:58:55.343571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980485+0000","last_change":"2023-03-03T08:43:29.107584+0000","last_active":"2023-03-03T08:43:43.980485+0000","last_peered":"2023-03-03T08:43:43.980485+0000","last_clean":"2023-03-03T08:43:43.980485+0000","last_became_active":"2023-03-03T08:43:29.107208+0000","last_became_peered":"2023-03-03T08:43:29.107208+0000","last_unstale":"2023-03-03T08:43:43.980485+0000","last_undegraded":"2023-03-03T08:43:43.980485+0000","last_fullsized":"2023-03-03T08:43:43.980485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:28:24.087122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106004+0000","last_change":"2023-03-03T08:43:43.575742+0000","last_active":"2023-03-03T08:43:45.106004+0000","last_peered":"2023-03-03T08:43:45.106004+0000","last_clean":"2023-03-03T08:43:45.106004+0000","last_became_active":"2023-03-03T08:43:43.575456+0000","last_became_peered":"2023-03-03T08:43:43.575456+0000","last_unstale":"2023-03-03T08:43:45.106004+0000","last_undegraded":"2023-03-03T08:43:45.106004+0000","last_fullsized":"2023-03-03T08:43:45.106004+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:46:17.635306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106635+0000","last_change":"2023-03-03T08:43:43.985961+0000","last_active":"2023-03-03T08:43:45.106635+0000","last_peered":"2023-03-03T08:43:45.106635+0000","last_clean":"2023-03-03T08:43:45.106635+0000","last_became_active":"2023-03-03T08:43:43.985633+0000","last_became_peered":"2023-03-03T08:43:43.985633+0000","last_unstale":"2023-03-03T08:43:45.106635+0000","last_undegraded":"2023-03-03T08:43:45.106635+0000","last_fullsized":"2023-03-03T08:43:45.106635+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:52.517910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989789+0000","last_change":"2023-03-03T08:43:34.526797+0000","last_active":"2023-03-03T08:43:43.989789+0000","last_peered":"2023-03-03T08:43:43.989789+0000","last_clean":"2023-03-03T08:43:43.989789+0000","last_became_active":"2023-03-03T08:43:34.526490+0000","last_became_peered":"2023-03-03T08:43:34.526490+0000","last_unstale":"2023-03-03T08:43:43.989789+0000","last_undegraded":"2023-03-03T08:43:43.989789+0000","last_fullsized":"2023-03-03T08:43:43.989789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:11.808336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986004+0000","last_change":"2023-03-03T08:43:34.104692+0000","last_active":"2023-03-03T08:43:43.986004+0000","last_peered":"2023-03-03T08:43:43.986004+0000","last_clean":"2023-03-03T08:43:43.986004+0000","last_became_active":"2023-03-03T08:43:34.104263+0000","last_became_peered":"2023-03-03T08:43:34.104263+0000","last_unstale":"2023-03-03T08:43:43.986004+0000","last_undegraded":"2023-03-03T08:43:43.986004+0000","last_fullsized":"2023-03-03T08:43:43.986004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:24.537703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983049+0000","last_change":"2023-03-03T08:43:29.107536+0000","last_active":"2023-03-03T08:43:43.983049+0000","last_peered":"2023-03-03T08:43:43.983049+0000","last_clean":"2023-03-03T08:43:43.983049+0000","last_became_active":"2023-03-03T08:43:29.107093+0000","last_became_peered":"2023-03-03T08:43:29.107093+0000","last_unstale":"2023-03-03T08:43:43.983049+0000","last_undegraded":"2023-03-03T08:43:43.983049+0000","last_fullsized":"2023-03-03T08:43:43.983049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:25:47.568904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980025+0000","last_change":"2023-03-03T08:43:43.577221+0000","last_active":"2023-03-03T08:43:44.980025+0000","last_peered":"2023-03-03T08:43:44.980025+0000","last_clean":"2023-03-03T08:43:44.980025+0000","last_became_active":"2023-03-03T08:43:43.576942+0000","last_became_peered":"2023-03-03T08:43:43.576942+0000","last_unstale":"2023-03-03T08:43:44.980025+0000","last_undegraded":"2023-03-03T08:43:44.980025+0000","last_fullsized":"2023-03-03T08:43:44.980025+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:13:09.716078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105102+0000","last_change":"2023-03-03T08:43:44.000318+0000","last_active":"2023-03-03T08:43:45.105102+0000","last_peered":"2023-03-03T08:43:45.105102+0000","last_clean":"2023-03-03T08:43:45.105102+0000","last_became_active":"2023-03-03T08:43:44.000053+0000","last_became_peered":"2023-03-03T08:43:44.000053+0000","last_unstale":"2023-03-03T08:43:45.105102+0000","last_undegraded":"2023-03-03T08:43:45.105102+0000","last_fullsized":"2023-03-03T08:43:45.105102+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:31.059774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994113+0000","last_change":"2023-03-03T08:43:34.540006+0000","last_active":"2023-03-03T08:43:43.994113+0000","last_peered":"2023-03-03T08:43:43.994113+0000","last_clean":"2023-03-03T08:43:43.994113+0000","last_became_active":"2023-03-03T08:43:34.539579+0000","last_became_peered":"2023-03-03T08:43:34.539579+0000","last_unstale":"2023-03-03T08:43:43.994113+0000","last_undegraded":"2023-03-03T08:43:43.994113+0000","last_fullsized":"2023-03-03T08:43:43.994113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988985+0000","last_change":"2023-03-03T08:43:34.109764+0000","last_active":"2023-03-03T08:43:43.988985+0000","last_peered":"2023-03-03T08:43:43.988985+0000","last_clean":"2023-03-03T08:43:43.988985+0000","last_became_active":"2023-03-03T08:43:34.109462+0000","last_became_peered":"2023-03-03T08:43:34.109462+0000","last_unstale":"2023-03-03T08:43:43.988985+0000","last_undegraded":"2023-03-03T08:43:43.988985+0000","last_fullsized":"2023-03-03T08:43:43.988985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:56.399192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106570+0000","last_change":"2023-03-03T08:43:43.576402+0000","last_active":"2023-03-03T08:43:45.106570+0000","last_peered":"2023-03-03T08:43:45.106570+0000","last_clean":"2023-03-03T08:43:45.106570+0000","last_became_active":"2023-03-03T08:43:43.576113+0000","last_became_peered":"2023-03-03T08:43:43.576113+0000","last_unstale":"2023-03-03T08:43:45.106570+0000","last_undegraded":"2023-03-03T08:43:45.106570+0000","last_fullsized":"2023-03-03T08:43:45.106570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:33:26.483688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106437+0000","last_change":"2023-03-03T08:43:44.002074+0000","last_active":"2023-03-03T08:43:45.106437+0000","last_peered":"2023-03-03T08:43:45.106437+0000","last_clean":"2023-03-03T08:43:45.106437+0000","last_became_active":"2023-03-03T08:43:44.001803+0000","last_became_peered":"2023-03-03T08:43:44.001803+0000","last_unstale":"2023-03-03T08:43:45.106437+0000","last_undegraded":"2023-03-03T08:43:45.106437+0000","last_fullsized":"2023-03-03T08:43:45.106437+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:15:16.115135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986358+0000","last_change":"2023-03-03T08:43:34.541858+0000","last_active":"2023-03-03T08:43:43.986358+0000","last_peered":"2023-03-03T08:43:43.986358+0000","last_clean":"2023-03-03T08:43:43.986358+0000","last_became_active":"2023-03-03T08:43:34.541430+0000","last_became_peered":"2023-03-03T08:43:34.541430+0000","last_unstale":"2023-03-03T08:43:43.986358+0000","last_undegraded":"2023-03-03T08:43:43.986358+0000","last_fullsized":"2023-03-03T08:43:43.986358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:29.397784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979830+0000","last_change":"2023-03-03T08:43:34.111335+0000","last_active":"2023-03-03T08:43:43.979830+0000","last_peered":"2023-03-03T08:43:43.979830+0000","last_clean":"2023-03-03T08:43:43.979830+0000","last_became_active":"2023-03-03T08:43:34.110837+0000","last_became_peered":"2023-03-03T08:43:34.110837+0000","last_unstale":"2023-03-03T08:43:43.979830+0000","last_undegraded":"2023-03-03T08:43:43.979830+0000","last_fullsized":"2023-03-03T08:43:43.979830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:03.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106574+0000","last_change":"2023-03-03T08:43:43.577539+0000","last_active":"2023-03-03T08:43:45.106574+0000","last_peered":"2023-03-03T08:43:45.106574+0000","last_clean":"2023-03-03T08:43:45.106574+0000","last_became_active":"2023-03-03T08:43:43.577209+0000","last_became_peered":"2023-03-03T08:43:43.577209+0000","last_unstale":"2023-03-03T08:43:45.106574+0000","last_undegraded":"2023-03-03T08:43:45.106574+0000","last_fullsized":"2023-03-03T08:43:45.106574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:31:01.430495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979296+0000","last_change":"2023-03-03T08:43:43.993565+0000","last_active":"2023-03-03T08:43:44.979296+0000","last_peered":"2023-03-03T08:43:44.979296+0000","last_clean":"2023-03-03T08:43:44.979296+0000","last_became_active":"2023-03-03T08:43:43.992653+0000","last_became_peered":"2023-03-03T08:43:43.992653+0000","last_unstale":"2023-03-03T08:43:44.979296+0000","last_undegraded":"2023-03-03T08:43:44.979296+0000","last_fullsized":"2023-03-03T08:43:44.979296+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:24.295462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981121+0000","last_change":"2023-03-03T08:43:34.550325+0000","last_active":"2023-03-03T08:43:43.981121+0000","last_peered":"2023-03-03T08:43:43.981121+0000","last_clean":"2023-03-03T08:43:43.981121+0000","last_became_active":"2023-03-03T08:43:34.550041+0000","last_became_peered":"2023-03-03T08:43:34.550041+0000","last_unstale":"2023-03-03T08:43:43.981121+0000","last_undegraded":"2023-03-03T08:43:43.981121+0000","last_fullsized":"2023-03-03T08:43:43.981121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:44:34.082003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980457+0000","last_change":"2023-03-03T08:43:34.106777+0000","last_active":"2023-03-03T08:43:43.980457+0000","last_peered":"2023-03-03T08:43:43.980457+0000","last_clean":"2023-03-03T08:43:43.980457+0000","last_became_active":"2023-03-03T08:43:34.106132+0000","last_became_peered":"2023-03-03T08:43:34.106132+0000","last_unstale":"2023-03-03T08:43:43.980457+0000","last_undegraded":"2023-03-03T08:43:43.980457+0000","last_fullsized":"2023-03-03T08:43:43.980457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:55.965987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106224+0000","last_change":"2023-03-03T08:43:43.580312+0000","last_active":"2023-03-03T08:43:45.106224+0000","last_peered":"2023-03-03T08:43:45.106224+0000","last_clean":"2023-03-03T08:43:45.106224+0000","last_became_active":"2023-03-03T08:43:43.580096+0000","last_became_peered":"2023-03-03T08:43:43.580096+0000","last_unstale":"2023-03-03T08:43:45.106224+0000","last_undegraded":"2023-03-03T08:43:45.106224+0000","last_fullsized":"2023-03-03T08:43:45.106224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:06:03.500778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106765+0000","last_change":"2023-03-03T08:43:43.997453+0000","last_active":"2023-03-03T08:43:45.106765+0000","last_peered":"2023-03-03T08:43:45.106765+0000","last_clean":"2023-03-03T08:43:45.106765+0000","last_became_active":"2023-03-03T08:43:43.997090+0000","last_became_peered":"2023-03-03T08:43:43.997090+0000","last_unstale":"2023-03-03T08:43:45.106765+0000","last_undegraded":"2023-03-03T08:43:45.106765+0000","last_fullsized":"2023-03-03T08:43:45.106765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:28:58.322823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985493+0000","last_change":"2023-03-03T08:43:34.542308+0000","last_active":"2023-03-03T08:43:43.985493+0000","last_peered":"2023-03-03T08:43:43.985493+0000","last_clean":"2023-03-03T08:43:43.985493+0000","last_became_active":"2023-03-03T08:43:34.541991+0000","last_became_peered":"2023-03-03T08:43:34.541991+0000","last_unstale":"2023-03-03T08:43:43.985493+0000","last_undegraded":"2023-03-03T08:43:43.985493+0000","last_fullsized":"2023-03-03T08:43:43.985493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:06:21.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992335+0000","last_change":"2023-03-03T08:43:34.109698+0000","last_active":"2023-03-03T08:43:43.992335+0000","last_peered":"2023-03-03T08:43:43.992335+0000","last_clean":"2023-03-03T08:43:43.992335+0000","last_became_active":"2023-03-03T08:43:34.109423+0000","last_became_peered":"2023-03-03T08:43:34.109423+0000","last_unstale":"2023-03-03T08:43:43.992335+0000","last_undegraded":"2023-03-03T08:43:43.992335+0000","last_fullsized":"2023-03-03T08:43:43.992335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:49.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105911+0000","last_change":"2023-03-03T08:43:43.573925+0000","last_active":"2023-03-03T08:43:45.105911+0000","last_peered":"2023-03-03T08:43:45.105911+0000","last_clean":"2023-03-03T08:43:45.105911+0000","last_became_active":"2023-03-03T08:43:43.573474+0000","last_became_peered":"2023-03-03T08:43:43.573474+0000","last_unstale":"2023-03-03T08:43:45.105911+0000","last_undegraded":"2023-03-03T08:43:45.105911+0000","last_fullsized":"2023-03-03T08:43:45.105911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:47.858726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107653+0000","last_change":"2023-03-03T08:43:43.995602+0000","last_active":"2023-03-03T08:43:45.107653+0000","last_peered":"2023-03-03T08:43:45.107653+0000","last_clean":"2023-03-03T08:43:45.107653+0000","last_became_active":"2023-03-03T08:43:43.995404+0000","last_became_peered":"2023-03-03T08:43:43.995404+0000","last_unstale":"2023-03-03T08:43:45.107653+0000","last_undegraded":"2023-03-03T08:43:45.107653+0000","last_fullsized":"2023-03-03T08:43:45.107653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:37.212119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982423+0000","last_change":"2023-03-03T08:43:34.548304+0000","last_active":"2023-03-03T08:43:43.982423+0000","last_peered":"2023-03-03T08:43:43.982423+0000","last_clean":"2023-03-03T08:43:43.982423+0000","last_became_active":"2023-03-03T08:43:34.547960+0000","last_became_peered":"2023-03-03T08:43:34.547960+0000","last_unstale":"2023-03-03T08:43:43.982423+0000","last_undegraded":"2023-03-03T08:43:43.982423+0000","last_fullsized":"2023-03-03T08:43:43.982423+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:04.907753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985574+0000","last_change":"2023-03-03T08:43:34.109372+0000","last_active":"2023-03-03T08:43:43.985574+0000","last_peered":"2023-03-03T08:43:43.985574+0000","last_clean":"2023-03-03T08:43:43.985574+0000","last_became_active":"2023-03-03T08:43:34.109023+0000","last_became_peered":"2023-03-03T08:43:34.109023+0000","last_unstale":"2023-03-03T08:43:43.985574+0000","last_undegraded":"2023-03-03T08:43:43.985574+0000","last_fullsized":"2023-03-03T08:43:43.985574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:51.374188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978866+0000","last_change":"2023-03-03T08:43:43.572240+0000","last_active":"2023-03-03T08:43:44.978866+0000","last_peered":"2023-03-03T08:43:44.978866+0000","last_clean":"2023-03-03T08:43:44.978866+0000","last_became_active":"2023-03-03T08:43:43.571753+0000","last_became_peered":"2023-03-03T08:43:43.571753+0000","last_unstale":"2023-03-03T08:43:44.978866+0000","last_undegraded":"2023-03-03T08:43:44.978866+0000","last_fullsized":"2023-03-03T08:43:44.978866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:02:30.521667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979027+0000","last_change":"2023-03-03T08:43:43.995018+0000","last_active":"2023-03-03T08:43:44.979027+0000","last_peered":"2023-03-03T08:43:44.979027+0000","last_clean":"2023-03-03T08:43:44.979027+0000","last_became_active":"2023-03-03T08:43:43.994690+0000","last_became_peered":"2023-03-03T08:43:43.994690+0000","last_unstale":"2023-03-03T08:43:44.979027+0000","last_undegraded":"2023-03-03T08:43:44.979027+0000","last_fullsized":"2023-03-03T08:43:44.979027+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:06:13.979929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986022+0000","last_change":"2023-03-03T08:43:34.538817+0000","last_active":"2023-03-03T08:43:43.986022+0000","last_peered":"2023-03-03T08:43:43.986022+0000","last_clean":"2023-03-03T08:43:43.986022+0000","last_became_active":"2023-03-03T08:43:34.538506+0000","last_became_peered":"2023-03-03T08:43:34.538506+0000","last_unstale":"2023-03-03T08:43:43.986022+0000","last_undegraded":"2023-03-03T08:43:43.986022+0000","last_fullsized":"2023-03-03T08:43:43.986022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980930+0000","last_change":"2023-03-03T08:43:34.117734+0000","last_active":"2023-03-03T08:43:43.980930+0000","last_peered":"2023-03-03T08:43:43.980930+0000","last_clean":"2023-03-03T08:43:43.980930+0000","last_became_active":"2023-03-03T08:43:34.117454+0000","last_became_peered":"2023-03-03T08:43:34.117454+0000","last_unstale":"2023-03-03T08:43:43.980930+0000","last_undegraded":"2023-03-03T08:43:43.980930+0000","last_fullsized":"2023-03-03T08:43:43.980930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:42:27.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106434+0000","last_change":"2023-03-03T08:43:43.574955+0000","last_active":"2023-03-03T08:43:45.106434+0000","last_peered":"2023-03-03T08:43:45.106434+0000","last_clean":"2023-03-03T08:43:45.106434+0000","last_became_active":"2023-03-03T08:43:43.574677+0000","last_became_peered":"2023-03-03T08:43:43.574677+0000","last_unstale":"2023-03-03T08:43:45.106434+0000","last_undegraded":"2023-03-03T08:43:45.106434+0000","last_fullsized":"2023-03-03T08:43:45.106434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:12:19.538291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.104954+0000","last_change":"2023-03-03T08:43:43.994888+0000","last_active":"2023-03-03T08:43:45.104954+0000","last_peered":"2023-03-03T08:43:45.104954+0000","last_clean":"2023-03-03T08:43:45.104954+0000","last_became_active":"2023-03-03T08:43:43.994630+0000","last_became_peered":"2023-03-03T08:43:43.994630+0000","last_unstale":"2023-03-03T08:43:45.104954+0000","last_undegraded":"2023-03-03T08:43:45.104954+0000","last_fullsized":"2023-03-03T08:43:45.104954+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:15:18.412569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982579+0000","last_change":"2023-03-03T08:43:34.548010+0000","last_active":"2023-03-03T08:43:43.982579+0000","last_peered":"2023-03-03T08:43:43.982579+0000","last_clean":"2023-03-03T08:43:43.982579+0000","last_became_active":"2023-03-03T08:43:34.547662+0000","last_became_peered":"2023-03-03T08:43:34.547662+0000","last_unstale":"2023-03-03T08:43:43.982579+0000","last_undegraded":"2023-03-03T08:43:43.982579+0000","last_fullsized":"2023-03-03T08:43:43.982579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:52.200064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986286+0000","last_change":"2023-03-03T08:43:34.109890+0000","last_active":"2023-03-03T08:43:43.986286+0000","last_peered":"2023-03-03T08:43:43.986286+0000","last_clean":"2023-03-03T08:43:43.986286+0000","last_became_active":"2023-03-03T08:43:34.109583+0000","last_became_peered":"2023-03-03T08:43:34.109583+0000","last_unstale":"2023-03-03T08:43:43.986286+0000","last_undegraded":"2023-03-03T08:43:43.986286+0000","last_fullsized":"2023-03-03T08:43:43.986286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:27:34.142294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978233+0000","last_change":"2023-03-03T08:43:43.575945+0000","last_active":"2023-03-03T08:43:44.978233+0000","last_peered":"2023-03-03T08:43:44.978233+0000","last_clean":"2023-03-03T08:43:44.978233+0000","last_became_active":"2023-03-03T08:43:43.575675+0000","last_became_peered":"2023-03-03T08:43:43.575675+0000","last_unstale":"2023-03-03T08:43:44.978233+0000","last_undegraded":"2023-03-03T08:43:44.978233+0000","last_fullsized":"2023-03-03T08:43:44.978233+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:48.730805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980103+0000","last_change":"2023-03-03T08:43:43.998435+0000","last_active":"2023-03-03T08:43:44.980103+0000","last_peered":"2023-03-03T08:43:44.980103+0000","last_clean":"2023-03-03T08:43:44.980103+0000","last_became_active":"2023-03-03T08:43:43.996508+0000","last_became_peered":"2023-03-03T08:43:43.996508+0000","last_unstale":"2023-03-03T08:43:44.980103+0000","last_undegraded":"2023-03-03T08:43:44.980103+0000","last_fullsized":"2023-03-03T08:43:44.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:31:36.310851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979477+0000","last_change":"2023-03-03T08:43:34.540196+0000","last_active":"2023-03-03T08:43:43.979477+0000","last_peered":"2023-03-03T08:43:43.979477+0000","last_clean":"2023-03-03T08:43:43.979477+0000","last_became_active":"2023-03-03T08:43:34.539931+0000","last_became_peered":"2023-03-03T08:43:34.539931+0000","last_unstale":"2023-03-03T08:43:43.979477+0000","last_undegraded":"2023-03-03T08:43:43.979477+0000","last_fullsized":"2023-03-03T08:43:43.979477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:20.012997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990881+0000","last_change":"2023-03-03T08:43:34.110464+0000","last_active":"2023-03-03T08:43:43.990881+0000","last_peered":"2023-03-03T08:43:43.990881+0000","last_clean":"2023-03-03T08:43:43.990881+0000","last_became_active":"2023-03-03T08:43:34.110165+0000","last_became_peered":"2023-03-03T08:43:34.110165+0000","last_unstale":"2023-03-03T08:43:43.990881+0000","last_undegraded":"2023-03-03T08:43:43.990881+0000","last_fullsized":"2023-03-03T08:43:43.990881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:54.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106139+0000","last_change":"2023-03-03T08:43:43.578298+0000","last_active":"2023-03-03T08:43:45.106139+0000","last_peered":"2023-03-03T08:43:45.106139+0000","last_clean":"2023-03-03T08:43:45.106139+0000","last_became_active":"2023-03-03T08:43:43.578024+0000","last_became_peered":"2023-03-03T08:43:43.578024+0000","last_unstale":"2023-03-03T08:43:45.106139+0000","last_undegraded":"2023-03-03T08:43:45.106139+0000","last_fullsized":"2023-03-03T08:43:45.106139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:00:44.633716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106526+0000","last_change":"2023-03-03T08:43:43.992906+0000","last_active":"2023-03-03T08:43:45.106526+0000","last_peered":"2023-03-03T08:43:45.106526+0000","last_clean":"2023-03-03T08:43:45.106526+0000","last_became_active":"2023-03-03T08:43:43.991823+0000","last_became_peered":"2023-03-03T08:43:43.991823+0000","last_unstale":"2023-03-03T08:43:45.106526+0000","last_undegraded":"2023-03-03T08:43:45.106526+0000","last_fullsized":"2023-03-03T08:43:45.106526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:34.431216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.553597+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.553322+0000","last_became_peered":"2023-03-03T08:43:34.553322+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:24:17.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983377+0000","last_change":"2023-03-03T08:43:34.108123+0000","last_active":"2023-03-03T08:43:43.983377+0000","last_peered":"2023-03-03T08:43:43.983377+0000","last_clean":"2023-03-03T08:43:43.983377+0000","last_became_active":"2023-03-03T08:43:34.107757+0000","last_became_peered":"2023-03-03T08:43:34.107757+0000","last_unstale":"2023-03-03T08:43:43.983377+0000","last_undegraded":"2023-03-03T08:43:43.983377+0000","last_fullsized":"2023-03-03T08:43:43.983377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:09.488930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106648+0000","last_change":"2023-03-03T08:43:43.573369+0000","last_active":"2023-03-03T08:43:45.106648+0000","last_peered":"2023-03-03T08:43:45.106648+0000","last_clean":"2023-03-03T08:43:45.106648+0000","last_became_active":"2023-03-03T08:43:43.573069+0000","last_became_peered":"2023-03-03T08:43:43.573069+0000","last_unstale":"2023-03-03T08:43:45.106648+0000","last_undegraded":"2023-03-03T08:43:45.106648+0000","last_fullsized":"2023-03-03T08:43:45.106648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:09.574926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107410+0000","last_change":"2023-03-03T08:43:43.995725+0000","last_active":"2023-03-03T08:43:45.107410+0000","last_peered":"2023-03-03T08:43:45.107410+0000","last_clean":"2023-03-03T08:43:45.107410+0000","last_became_active":"2023-03-03T08:43:43.993875+0000","last_became_peered":"2023-03-03T08:43:43.993875+0000","last_unstale":"2023-03-03T08:43:45.107410+0000","last_undegraded":"2023-03-03T08:43:45.107410+0000","last_fullsized":"2023-03-03T08:43:45.107410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:38.439539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982673+0000","last_change":"2023-03-03T08:43:34.546067+0000","last_active":"2023-03-03T08:43:43.982673+0000","last_peered":"2023-03-03T08:43:43.982673+0000","last_clean":"2023-03-03T08:43:43.982673+0000","last_became_active":"2023-03-03T08:43:34.543786+0000","last_became_peered":"2023-03-03T08:43:34.543786+0000","last_unstale":"2023-03-03T08:43:43.982673+0000","last_undegraded":"2023-03-03T08:43:43.982673+0000","last_fullsized":"2023-03-03T08:43:43.982673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:32.922812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.978650+0000","last_change":"2023-03-03T08:43:34.115792+0000","last_active":"2023-03-03T08:43:43.978650+0000","last_peered":"2023-03-03T08:43:43.978650+0000","last_clean":"2023-03-03T08:43:43.978650+0000","last_became_active":"2023-03-03T08:43:34.115431+0000","last_became_peered":"2023-03-03T08:43:34.115431+0000","last_unstale":"2023-03-03T08:43:43.978650+0000","last_undegraded":"2023-03-03T08:43:43.978650+0000","last_fullsized":"2023-03-03T08:43:43.978650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:15.484022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980510+0000","last_change":"2023-03-03T08:43:43.579691+0000","last_active":"2023-03-03T08:43:44.980510+0000","last_peered":"2023-03-03T08:43:44.980510+0000","last_clean":"2023-03-03T08:43:44.980510+0000","last_became_active":"2023-03-03T08:43:43.579425+0000","last_became_peered":"2023-03-03T08:43:43.579425+0000","last_unstale":"2023-03-03T08:43:44.980510+0000","last_undegraded":"2023-03-03T08:43:44.980510+0000","last_fullsized":"2023-03-03T08:43:44.980510+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:08.896591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979907+0000","last_change":"2023-03-03T08:43:44.006872+0000","last_active":"2023-03-03T08:43:44.979907+0000","last_peered":"2023-03-03T08:43:44.979907+0000","last_clean":"2023-03-03T08:43:44.979907+0000","last_became_active":"2023-03-03T08:43:44.006283+0000","last_became_peered":"2023-03-03T08:43:44.006283+0000","last_unstale":"2023-03-03T08:43:44.979907+0000","last_undegraded":"2023-03-03T08:43:44.979907+0000","last_fullsized":"2023-03-03T08:43:44.979907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:45.439399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981214+0000","last_change":"2023-03-03T08:43:34.532656+0000","last_active":"2023-03-03T08:43:43.981214+0000","last_peered":"2023-03-03T08:43:43.981214+0000","last_clean":"2023-03-03T08:43:43.981214+0000","last_became_active":"2023-03-03T08:43:34.532297+0000","last_became_peered":"2023-03-03T08:43:34.532297+0000","last_unstale":"2023-03-03T08:43:43.981214+0000","last_undegraded":"2023-03-03T08:43:43.981214+0000","last_fullsized":"2023-03-03T08:43:43.981214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:36.347740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989578+0000","last_change":"2023-03-03T08:43:34.112905+0000","last_active":"2023-03-03T08:43:43.989578+0000","last_peered":"2023-03-03T08:43:43.989578+0000","last_clean":"2023-03-03T08:43:43.989578+0000","last_became_active":"2023-03-03T08:43:34.112601+0000","last_became_peered":"2023-03-03T08:43:34.112601+0000","last_unstale":"2023-03-03T08:43:43.989578+0000","last_undegraded":"2023-03-03T08:43:43.989578+0000","last_fullsized":"2023-03-03T08:43:43.989578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:01.165783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"27'9","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:44:01.020445+0000","last_change":"2023-03-03T08:43:43.577882+0000","last_active":"2023-03-03T08:44:01.020445+0000","last_peered":"2023-03-03T08:44:01.020445+0000","last_clean":"2023-03-03T08:44:01.020445+0000","last_became_active":"2023-03-03T08:43:43.575625+0000","last_became_peered":"2023-03-03T08:43:43.575625+0000","last_unstale":"2023-03-03T08:44:01.020445+0000","last_undegraded":"2023-03-03T08:44:01.020445+0000","last_fullsized":"2023-03-03T08:44:01.020445+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:22:25.919413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2779,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106494+0000","last_change":"2023-03-03T08:43:43.994846+0000","last_active":"2023-03-03T08:43:45.106494+0000","last_peered":"2023-03-03T08:43:45.106494+0000","last_clean":"2023-03-03T08:43:45.106494+0000","last_became_active":"2023-03-03T08:43:43.994652+0000","last_became_peered":"2023-03-03T08:43:43.994652+0000","last_unstale":"2023-03-03T08:43:45.106494+0000","last_undegraded":"2023-03-03T08:43:45.106494+0000","last_fullsized":"2023-03-03T08:43:45.106494+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:54:29.238192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990352+0000","last_change":"2023-03-03T08:43:34.539790+0000","last_active":"2023-03-03T08:43:43.990352+0000","last_peered":"2023-03-03T08:43:43.990352+0000","last_clean":"2023-03-03T08:43:43.990352+0000","last_became_active":"2023-03-03T08:43:34.539443+0000","last_became_peered":"2023-03-03T08:43:34.539443+0000","last_unstale":"2023-03-03T08:43:43.990352+0000","last_undegraded":"2023-03-03T08:43:43.990352+0000","last_fullsized":"2023-03-03T08:43:43.990352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:25.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980897+0000","last_change":"2023-03-03T08:43:34.112941+0000","last_active":"2023-03-03T08:43:43.980897+0000","last_peered":"2023-03-03T08:43:43.980897+0000","last_clean":"2023-03-03T08:43:43.980897+0000","last_became_active":"2023-03-03T08:43:34.112646+0000","last_became_peered":"2023-03-03T08:43:34.112646+0000","last_unstale":"2023-03-03T08:43:43.980897+0000","last_undegraded":"2023-03-03T08:43:43.980897+0000","last_fullsized":"2023-03-03T08:43:43.980897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:32.767710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107854+0000","last_change":"2023-03-03T08:43:43.576421+0000","last_active":"2023-03-03T08:43:45.107854+0000","last_peered":"2023-03-03T08:43:45.107854+0000","last_clean":"2023-03-03T08:43:45.107854+0000","last_became_active":"2023-03-03T08:43:43.576086+0000","last_became_peered":"2023-03-03T08:43:43.576086+0000","last_unstale":"2023-03-03T08:43:45.107854+0000","last_undegraded":"2023-03-03T08:43:45.107854+0000","last_fullsized":"2023-03-03T08:43:45.107854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:57:17.814222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980193+0000","last_change":"2023-03-03T08:43:43.995591+0000","last_active":"2023-03-03T08:43:44.980193+0000","last_peered":"2023-03-03T08:43:44.980193+0000","last_clean":"2023-03-03T08:43:44.980193+0000","last_became_active":"2023-03-03T08:43:43.995338+0000","last_became_peered":"2023-03-03T08:43:43.995338+0000","last_unstale":"2023-03-03T08:43:44.980193+0000","last_undegraded":"2023-03-03T08:43:44.980193+0000","last_fullsized":"2023-03-03T08:43:44.980193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:17:56.617398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985713+0000","last_change":"2023-03-03T08:43:34.537857+0000","last_active":"2023-03-03T08:43:43.985713+0000","last_peered":"2023-03-03T08:43:43.985713+0000","last_clean":"2023-03-03T08:43:43.985713+0000","last_became_active":"2023-03-03T08:43:34.537590+0000","last_became_peered":"2023-03-03T08:43:34.537590+0000","last_unstale":"2023-03-03T08:43:43.985713+0000","last_undegraded":"2023-03-03T08:43:43.985713+0000","last_fullsized":"2023-03-03T08:43:43.985713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:52.612521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"27'9","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:58.500261+0000","last_change":"2023-03-03T08:43:34.117609+0000","last_active":"2023-03-03T08:43:58.500261+0000","last_peered":"2023-03-03T08:43:58.500261+0000","last_clean":"2023-03-03T08:43:58.500261+0000","last_became_active":"2023-03-03T08:43:34.117271+0000","last_became_peered":"2023-03-03T08:43:34.117271+0000","last_unstale":"2023-03-03T08:43:58.500261+0000","last_undegraded":"2023-03-03T08:43:58.500261+0000","last_fullsized":"2023-03-03T08:43:58.500261+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:20.259259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2779,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978772+0000","last_change":"2023-03-03T08:43:43.580489+0000","last_active":"2023-03-03T08:43:44.978772+0000","last_peered":"2023-03-03T08:43:44.978772+0000","last_clean":"2023-03-03T08:43:44.978772+0000","last_became_active":"2023-03-03T08:43:43.580207+0000","last_became_peered":"2023-03-03T08:43:43.580207+0000","last_unstale":"2023-03-03T08:43:44.978772+0000","last_undegraded":"2023-03-03T08:43:44.978772+0000","last_fullsized":"2023-03-03T08:43:44.978772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:58:05.519074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978633+0000","last_change":"2023-03-03T08:43:43.993832+0000","last_active":"2023-03-03T08:43:44.978633+0000","last_peered":"2023-03-03T08:43:44.978633+0000","last_clean":"2023-03-03T08:43:44.978633+0000","last_became_active":"2023-03-03T08:43:43.993524+0000","last_became_peered":"2023-03-03T08:43:43.993524+0000","last_unstale":"2023-03-03T08:43:44.978633+0000","last_undegraded":"2023-03-03T08:43:44.978633+0000","last_fullsized":"2023-03-03T08:43:44.978633+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:23:55.761785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987159+0000","last_change":"2023-03-03T08:43:34.544694+0000","last_active":"2023-03-03T08:43:43.987159+0000","last_peered":"2023-03-03T08:43:43.987159+0000","last_clean":"2023-03-03T08:43:43.987159+0000","last_became_active":"2023-03-03T08:43:34.544399+0000","last_became_peered":"2023-03-03T08:43:34.544399+0000","last_unstale":"2023-03-03T08:43:43.987159+0000","last_undegraded":"2023-03-03T08:43:43.987159+0000","last_fullsized":"2023-03-03T08:43:43.987159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986196+0000","last_change":"2023-03-03T08:43:34.110810+0000","last_active":"2023-03-03T08:43:43.986196+0000","last_peered":"2023-03-03T08:43:43.986196+0000","last_clean":"2023-03-03T08:43:43.986196+0000","last_became_active":"2023-03-03T08:43:34.110483+0000","last_became_peered":"2023-03-03T08:43:34.110483+0000","last_unstale":"2023-03-03T08:43:43.986196+0000","last_undegraded":"2023-03-03T08:43:43.986196+0000","last_fullsized":"2023-03-03T08:43:43.986196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:06.970615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979989+0000","last_change":"2023-03-03T08:43:43.576669+0000","last_active":"2023-03-03T08:43:44.979989+0000","last_peered":"2023-03-03T08:43:44.979989+0000","last_clean":"2023-03-03T08:43:44.979989+0000","last_became_active":"2023-03-03T08:43:43.573885+0000","last_became_peered":"2023-03-03T08:43:43.573885+0000","last_unstale":"2023-03-03T08:43:44.979989+0000","last_undegraded":"2023-03-03T08:43:44.979989+0000","last_fullsized":"2023-03-03T08:43:44.979989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:56:12.054118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979442+0000","last_change":"2023-03-03T08:43:43.996790+0000","last_active":"2023-03-03T08:43:44.979442+0000","last_peered":"2023-03-03T08:43:44.979442+0000","last_clean":"2023-03-03T08:43:44.979442+0000","last_became_active":"2023-03-03T08:43:43.996366+0000","last_became_peered":"2023-03-03T08:43:43.996366+0000","last_unstale":"2023-03-03T08:43:44.979442+0000","last_undegraded":"2023-03-03T08:43:44.979442+0000","last_fullsized":"2023-03-03T08:43:44.979442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:01.662221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988553+0000","last_change":"2023-03-03T08:43:34.538703+0000","last_active":"2023-03-03T08:43:43.988553+0000","last_peered":"2023-03-03T08:43:43.988553+0000","last_clean":"2023-03-03T08:43:43.988553+0000","last_became_active":"2023-03-03T08:43:34.538418+0000","last_became_peered":"2023-03-03T08:43:34.538418+0000","last_unstale":"2023-03-03T08:43:43.988553+0000","last_undegraded":"2023-03-03T08:43:43.988553+0000","last_fullsized":"2023-03-03T08:43:43.988553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:02.795203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986185+0000","last_change":"2023-03-03T08:43:34.113168+0000","last_active":"2023-03-03T08:43:43.986185+0000","last_peered":"2023-03-03T08:43:43.986185+0000","last_clean":"2023-03-03T08:43:43.986185+0000","last_became_active":"2023-03-03T08:43:34.112731+0000","last_became_peered":"2023-03-03T08:43:34.112731+0000","last_unstale":"2023-03-03T08:43:43.986185+0000","last_undegraded":"2023-03-03T08:43:43.986185+0000","last_fullsized":"2023-03-03T08:43:43.986185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:57:59.463191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106795+0000","last_change":"2023-03-03T08:43:43.577965+0000","last_active":"2023-03-03T08:43:45.106795+0000","last_peered":"2023-03-03T08:43:45.106795+0000","last_clean":"2023-03-03T08:43:45.106795+0000","last_became_active":"2023-03-03T08:43:43.577587+0000","last_became_peered":"2023-03-03T08:43:43.577587+0000","last_unstale":"2023-03-03T08:43:45.106795+0000","last_undegraded":"2023-03-03T08:43:45.106795+0000","last_fullsized":"2023-03-03T08:43:45.106795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:24.196773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105366+0000","last_change":"2023-03-03T08:43:43.998210+0000","last_active":"2023-03-03T08:43:45.105366+0000","last_peered":"2023-03-03T08:43:45.105366+0000","last_clean":"2023-03-03T08:43:45.105366+0000","last_became_active":"2023-03-03T08:43:43.997845+0000","last_became_peered":"2023-03-03T08:43:43.997845+0000","last_unstale":"2023-03-03T08:43:45.105366+0000","last_undegraded":"2023-03-03T08:43:45.105366+0000","last_fullsized":"2023-03-03T08:43:45.105366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:57:38.226245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991762+0000","last_change":"2023-03-03T08:43:34.528573+0000","last_active":"2023-03-03T08:43:43.991762+0000","last_peered":"2023-03-03T08:43:43.991762+0000","last_clean":"2023-03-03T08:43:43.991762+0000","last_became_active":"2023-03-03T08:43:34.528278+0000","last_became_peered":"2023-03-03T08:43:34.528278+0000","last_unstale":"2023-03-03T08:43:43.991762+0000","last_undegraded":"2023-03-03T08:43:43.991762+0000","last_fullsized":"2023-03-03T08:43:43.991762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:51.468323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.110802+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.110519+0000","last_became_peered":"2023-03-03T08:43:34.110519+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:02:18.355043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105602+0000","last_change":"2023-03-03T08:43:43.573655+0000","last_active":"2023-03-03T08:43:45.105602+0000","last_peered":"2023-03-03T08:43:45.105602+0000","last_clean":"2023-03-03T08:43:45.105602+0000","last_became_active":"2023-03-03T08:43:43.573162+0000","last_became_peered":"2023-03-03T08:43:43.573162+0000","last_unstale":"2023-03-03T08:43:45.105602+0000","last_undegraded":"2023-03-03T08:43:45.105602+0000","last_fullsized":"2023-03-03T08:43:45.105602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:59:04.568700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106617+0000","last_change":"2023-03-03T08:43:43.993431+0000","last_active":"2023-03-03T08:43:45.106617+0000","last_peered":"2023-03-03T08:43:45.106617+0000","last_clean":"2023-03-03T08:43:45.106617+0000","last_became_active":"2023-03-03T08:43:43.993166+0000","last_became_peered":"2023-03-03T08:43:43.993166+0000","last_unstale":"2023-03-03T08:43:45.106617+0000","last_undegraded":"2023-03-03T08:43:45.106617+0000","last_fullsized":"2023-03-03T08:43:45.106617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:04:52.880667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979475+0000","last_change":"2023-03-03T08:43:43.992992+0000","last_active":"2023-03-03T08:43:44.979475+0000","last_peered":"2023-03-03T08:43:44.979475+0000","last_clean":"2023-03-03T08:43:44.979475+0000","last_became_active":"2023-03-03T08:43:43.992664+0000","last_became_peered":"2023-03-03T08:43:43.992664+0000","last_unstale":"2023-03-03T08:43:44.979475+0000","last_undegraded":"2023-03-03T08:43:44.979475+0000","last_fullsized":"2023-03-03T08:43:44.979475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:07.276715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981207+0000","last_change":"2023-03-03T08:43:34.547450+0000","last_active":"2023-03-03T08:43:43.981207+0000","last_peered":"2023-03-03T08:43:43.981207+0000","last_clean":"2023-03-03T08:43:43.981207+0000","last_became_active":"2023-03-03T08:43:34.547168+0000","last_became_peered":"2023-03-03T08:43:34.547168+0000","last_unstale":"2023-03-03T08:43:43.981207+0000","last_undegraded":"2023-03-03T08:43:43.981207+0000","last_fullsized":"2023-03-03T08:43:43.981207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:17.723229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982025+0000","last_change":"2023-03-03T08:43:34.106505+0000","last_active":"2023-03-03T08:43:43.982025+0000","last_peered":"2023-03-03T08:43:43.982025+0000","last_clean":"2023-03-03T08:43:43.982025+0000","last_became_active":"2023-03-03T08:43:34.106079+0000","last_became_peered":"2023-03-03T08:43:34.106079+0000","last_unstale":"2023-03-03T08:43:43.982025+0000","last_undegraded":"2023-03-03T08:43:43.982025+0000","last_fullsized":"2023-03-03T08:43:43.982025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:48.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979698+0000","last_change":"2023-03-03T08:43:43.575448+0000","last_active":"2023-03-03T08:43:44.979698+0000","last_peered":"2023-03-03T08:43:44.979698+0000","last_clean":"2023-03-03T08:43:44.979698+0000","last_became_active":"2023-03-03T08:43:43.575142+0000","last_became_peered":"2023-03-03T08:43:43.575142+0000","last_unstale":"2023-03-03T08:43:44.979698+0000","last_undegraded":"2023-03-03T08:43:44.979698+0000","last_fullsized":"2023-03-03T08:43:44.979698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:42:09.308616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990396+0000","last_change":"2023-03-03T08:43:34.549123+0000","last_active":"2023-03-03T08:43:43.990396+0000","last_peered":"2023-03-03T08:43:43.990396+0000","last_clean":"2023-03-03T08:43:43.990396+0000","last_became_active":"2023-03-03T08:43:34.548839+0000","last_became_peered":"2023-03-03T08:43:34.548839+0000","last_unstale":"2023-03-03T08:43:43.990396+0000","last_undegraded":"2023-03-03T08:43:43.990396+0000","last_fullsized":"2023-03-03T08:43:43.990396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:38.134009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984959+0000","last_change":"2023-03-03T08:43:34.108187+0000","last_active":"2023-03-03T08:43:43.984959+0000","last_peered":"2023-03-03T08:43:43.984959+0000","last_clean":"2023-03-03T08:43:43.984959+0000","last_became_active":"2023-03-03T08:43:34.107859+0000","last_became_peered":"2023-03-03T08:43:34.107859+0000","last_unstale":"2023-03-03T08:43:43.984959+0000","last_undegraded":"2023-03-03T08:43:43.984959+0000","last_fullsized":"2023-03-03T08:43:43.984959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:46:40.595756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107953+0000","last_change":"2023-03-03T08:43:43.572738+0000","last_active":"2023-03-03T08:43:45.107953+0000","last_peered":"2023-03-03T08:43:45.107953+0000","last_clean":"2023-03-03T08:43:45.107953+0000","last_became_active":"2023-03-03T08:43:43.572435+0000","last_became_peered":"2023-03-03T08:43:43.572435+0000","last_unstale":"2023-03-03T08:43:45.107953+0000","last_undegraded":"2023-03-03T08:43:45.107953+0000","last_fullsized":"2023-03-03T08:43:45.107953+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:14.264653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105420+0000","last_change":"2023-03-03T08:43:43.995021+0000","last_active":"2023-03-03T08:43:45.105420+0000","last_peered":"2023-03-03T08:43:45.105420+0000","last_clean":"2023-03-03T08:43:45.105420+0000","last_became_active":"2023-03-03T08:43:43.994766+0000","last_became_peered":"2023-03-03T08:43:43.994766+0000","last_unstale":"2023-03-03T08:43:45.105420+0000","last_undegraded":"2023-03-03T08:43:45.105420+0000","last_fullsized":"2023-03-03T08:43:45.105420+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:14:03.827057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979582+0000","last_change":"2023-03-03T08:43:43.993391+0000","last_active":"2023-03-03T08:43:44.979582+0000","last_peered":"2023-03-03T08:43:44.979582+0000","last_clean":"2023-03-03T08:43:44.979582+0000","last_became_active":"2023-03-03T08:43:43.993087+0000","last_became_peered":"2023-03-03T08:43:43.993087+0000","last_unstale":"2023-03-03T08:43:44.979582+0000","last_undegraded":"2023-03-03T08:43:44.979582+0000","last_fullsized":"2023-03-03T08:43:44.979582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:13:01.193811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992180+0000","last_change":"2023-03-03T08:43:34.537301+0000","last_active":"2023-03-03T08:43:43.992180+0000","last_peered":"2023-03-03T08:43:43.992180+0000","last_clean":"2023-03-03T08:43:43.992180+0000","last_became_active":"2023-03-03T08:43:34.537007+0000","last_became_peered":"2023-03-03T08:43:34.537007+0000","last_unstale":"2023-03-03T08:43:43.992180+0000","last_undegraded":"2023-03-03T08:43:43.992180+0000","last_fullsized":"2023-03-03T08:43:43.992180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:59:49.124140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982700+0000","last_change":"2023-03-03T08:43:34.108056+0000","last_active":"2023-03-03T08:43:43.982700+0000","last_peered":"2023-03-03T08:43:43.982700+0000","last_clean":"2023-03-03T08:43:43.982700+0000","last_became_active":"2023-03-03T08:43:34.107708+0000","last_became_peered":"2023-03-03T08:43:34.107708+0000","last_unstale":"2023-03-03T08:43:43.982700+0000","last_undegraded":"2023-03-03T08:43:43.982700+0000","last_fullsized":"2023-03-03T08:43:43.982700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:10.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107314+0000","last_change":"2023-03-03T08:43:43.575788+0000","last_active":"2023-03-03T08:43:45.107314+0000","last_peered":"2023-03-03T08:43:45.107314+0000","last_clean":"2023-03-03T08:43:45.107314+0000","last_became_active":"2023-03-03T08:43:43.575507+0000","last_became_peered":"2023-03-03T08:43:43.575507+0000","last_unstale":"2023-03-03T08:43:45.107314+0000","last_undegraded":"2023-03-03T08:43:45.107314+0000","last_fullsized":"2023-03-03T08:43:45.107314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:15.843816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107877+0000","last_change":"2023-03-03T08:43:43.994916+0000","last_active":"2023-03-03T08:43:45.107877+0000","last_peered":"2023-03-03T08:43:45.107877+0000","last_clean":"2023-03-03T08:43:45.107877+0000","last_became_active":"2023-03-03T08:43:43.994639+0000","last_became_peered":"2023-03-03T08:43:43.994639+0000","last_unstale":"2023-03-03T08:43:45.107877+0000","last_undegraded":"2023-03-03T08:43:45.107877+0000","last_fullsized":"2023-03-03T08:43:45.107877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:52.526052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980834+0000","last_change":"2023-03-03T08:43:34.533987+0000","last_active":"2023-03-03T08:43:43.980834+0000","last_peered":"2023-03-03T08:43:43.980834+0000","last_clean":"2023-03-03T08:43:43.980834+0000","last_became_active":"2023-03-03T08:43:34.533729+0000","last_became_peered":"2023-03-03T08:43:34.533729+0000","last_unstale":"2023-03-03T08:43:43.980834+0000","last_undegraded":"2023-03-03T08:43:43.980834+0000","last_fullsized":"2023-03-03T08:43:43.980834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:18:12.407151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106398+0000","last_change":"2023-03-03T08:43:43.996808+0000","last_active":"2023-03-03T08:43:45.106398+0000","last_peered":"2023-03-03T08:43:45.106398+0000","last_clean":"2023-03-03T08:43:45.106398+0000","last_became_active":"2023-03-03T08:43:43.996559+0000","last_became_peered":"2023-03-03T08:43:43.996559+0000","last_unstale":"2023-03-03T08:43:45.106398+0000","last_undegraded":"2023-03-03T08:43:45.106398+0000","last_fullsized":"2023-03-03T08:43:45.106398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:01.223908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.993213+0000","last_change":"2023-03-03T08:43:34.540762+0000","last_active":"2023-03-03T08:43:43.993213+0000","last_peered":"2023-03-03T08:43:43.993213+0000","last_clean":"2023-03-03T08:43:43.993213+0000","last_became_active":"2023-03-03T08:43:34.540467+0000","last_became_peered":"2023-03-03T08:43:34.540467+0000","last_unstale":"2023-03-03T08:43:43.993213+0000","last_undegraded":"2023-03-03T08:43:43.993213+0000","last_fullsized":"2023-03-03T08:43:43.993213+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107993+0000","last_change":"2023-03-03T08:43:43.995286+0000","last_active":"2023-03-03T08:43:45.107993+0000","last_peered":"2023-03-03T08:43:45.107993+0000","last_clean":"2023-03-03T08:43:45.107993+0000","last_became_active":"2023-03-03T08:43:43.994861+0000","last_became_peered":"2023-03-03T08:43:43.994861+0000","last_unstale":"2023-03-03T08:43:45.107993+0000","last_undegraded":"2023-03-03T08:43:45.107993+0000","last_fullsized":"2023-03-03T08:43:45.107993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990722+0000","last_change":"2023-03-03T08:43:34.542858+0000","last_active":"2023-03-03T08:43:43.990722+0000","last_peered":"2023-03-03T08:43:43.990722+0000","last_clean":"2023-03-03T08:43:43.990722+0000","last_became_active":"2023-03-03T08:43:34.542510+0000","last_became_peered":"2023-03-03T08:43:34.542510+0000","last_unstale":"2023-03-03T08:43:43.990722+0000","last_undegraded":"2023-03-03T08:43:43.990722+0000","last_fullsized":"2023-03-03T08:43:43.990722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:21.471234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106711+0000","last_change":"2023-03-03T08:43:43.992084+0000","last_active":"2023-03-03T08:43:45.106711+0000","last_peered":"2023-03-03T08:43:45.106711+0000","last_clean":"2023-03-03T08:43:45.106711+0000","last_became_active":"2023-03-03T08:43:43.991465+0000","last_became_peered":"2023-03-03T08:43:43.991465+0000","last_unstale":"2023-03-03T08:43:45.106711+0000","last_undegraded":"2023-03-03T08:43:45.106711+0000","last_fullsized":"2023-03-03T08:43:45.106711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:19.530923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983571+0000","last_change":"2023-03-03T08:43:34.548469+0000","last_active":"2023-03-03T08:43:43.983571+0000","last_peered":"2023-03-03T08:43:43.983571+0000","last_clean":"2023-03-03T08:43:43.983571+0000","last_became_active":"2023-03-03T08:43:34.547647+0000","last_became_peered":"2023-03-03T08:43:34.547647+0000","last_unstale":"2023-03-03T08:43:43.983571+0000","last_undegraded":"2023-03-03T08:43:43.983571+0000","last_fullsized":"2023-03-03T08:43:43.983571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:40.054781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107161+0000","last_change":"2023-03-03T08:43:43.998061+0000","last_active":"2023-03-03T08:43:45.107161+0000","last_peered":"2023-03-03T08:43:45.107161+0000","last_clean":"2023-03-03T08:43:45.107161+0000","last_became_active":"2023-03-03T08:43:43.997770+0000","last_became_peered":"2023-03-03T08:43:43.997770+0000","last_unstale":"2023-03-03T08:43:45.107161+0000","last_undegraded":"2023-03-03T08:43:45.107161+0000","last_fullsized":"2023-03-03T08:43:45.107161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:17.053610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988635+0000","last_change":"2023-03-03T08:43:34.549771+0000","last_active":"2023-03-03T08:43:43.988635+0000","last_peered":"2023-03-03T08:43:43.988635+0000","last_clean":"2023-03-03T08:43:43.988635+0000","last_became_active":"2023-03-03T08:43:34.549479+0000","last_became_peered":"2023-03-03T08:43:34.549479+0000","last_unstale":"2023-03-03T08:43:43.988635+0000","last_undegraded":"2023-03-03T08:43:43.988635+0000","last_fullsized":"2023-03-03T08:43:43.988635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:41:03.902019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979024+0000","last_change":"2023-03-03T08:43:43.999391+0000","last_active":"2023-03-03T08:43:44.979024+0000","last_peered":"2023-03-03T08:43:44.979024+0000","last_clean":"2023-03-03T08:43:44.979024+0000","last_became_active":"2023-03-03T08:43:43.998904+0000","last_became_peered":"2023-03-03T08:43:43.998904+0000","last_unstale":"2023-03-03T08:43:44.979024+0000","last_undegraded":"2023-03-03T08:43:44.979024+0000","last_fullsized":"2023-03-03T08:43:44.979024+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:17.429776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992365+0000","last_change":"2023-03-03T08:43:34.542661+0000","last_active":"2023-03-03T08:43:43.992365+0000","last_peered":"2023-03-03T08:43:43.992365+0000","last_clean":"2023-03-03T08:43:43.992365+0000","last_became_active":"2023-03-03T08:43:34.542353+0000","last_became_peered":"2023-03-03T08:43:34.542353+0000","last_unstale":"2023-03-03T08:43:43.992365+0000","last_undegraded":"2023-03-03T08:43:43.992365+0000","last_fullsized":"2023-03-03T08:43:43.992365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:38.730662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105103+0000","last_change":"2023-03-03T08:43:43.996729+0000","last_active":"2023-03-03T08:43:45.105103+0000","last_peered":"2023-03-03T08:43:45.105103+0000","last_clean":"2023-03-03T08:43:45.105103+0000","last_became_active":"2023-03-03T08:43:43.986535+0000","last_became_peered":"2023-03-03T08:43:43.986535+0000","last_unstale":"2023-03-03T08:43:45.105103+0000","last_undegraded":"2023-03-03T08:43:45.105103+0000","last_fullsized":"2023-03-03T08:43:45.105103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:22.505305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982406+0000","last_change":"2023-03-03T08:43:34.549837+0000","last_active":"2023-03-03T08:43:43.982406+0000","last_peered":"2023-03-03T08:43:43.982406+0000","last_clean":"2023-03-03T08:43:43.982406+0000","last_became_active":"2023-03-03T08:43:34.549542+0000","last_became_peered":"2023-03-03T08:43:34.549542+0000","last_unstale":"2023-03-03T08:43:43.982406+0000","last_undegraded":"2023-03-03T08:43:43.982406+0000","last_fullsized":"2023-03-03T08:43:43.982406+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:31.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106334+0000","last_change":"2023-03-03T08:43:43.993213+0000","last_active":"2023-03-03T08:43:45.106334+0000","last_peered":"2023-03-03T08:43:45.106334+0000","last_clean":"2023-03-03T08:43:45.106334+0000","last_became_active":"2023-03-03T08:43:43.992920+0000","last_became_peered":"2023-03-03T08:43:43.992920+0000","last_unstale":"2023-03-03T08:43:45.106334+0000","last_undegraded":"2023-03-03T08:43:45.106334+0000","last_fullsized":"2023-03-03T08:43:45.106334+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:42.055599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.547003+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.546744+0000","last_became_peered":"2023-03-03T08:43:34.546744+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:53:38.996605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106934+0000","last_change":"2023-03-03T08:43:43.994195+0000","last_active":"2023-03-03T08:43:45.106934+0000","last_peered":"2023-03-03T08:43:45.106934+0000","last_clean":"2023-03-03T08:43:45.106934+0000","last_became_active":"2023-03-03T08:43:43.993463+0000","last_became_peered":"2023-03-03T08:43:43.993463+0000","last_unstale":"2023-03-03T08:43:45.106934+0000","last_undegraded":"2023-03-03T08:43:45.106934+0000","last_fullsized":"2023-03-03T08:43:45.106934+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:52.365793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982677+0000","last_change":"2023-03-03T08:43:34.553589+0000","last_active":"2023-03-03T08:43:43.982677+0000","last_peered":"2023-03-03T08:43:43.982677+0000","last_clean":"2023-03-03T08:43:43.982677+0000","last_became_active":"2023-03-03T08:43:34.553242+0000","last_became_peered":"2023-03-03T08:43:34.553242+0000","last_unstale":"2023-03-03T08:43:43.982677+0000","last_undegraded":"2023-03-03T08:43:43.982677+0000","last_fullsized":"2023-03-03T08:43:43.982677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:09.000051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107426+0000","last_change":"2023-03-03T08:43:44.004404+0000","last_active":"2023-03-03T08:43:45.107426+0000","last_peered":"2023-03-03T08:43:45.107426+0000","last_clean":"2023-03-03T08:43:45.107426+0000","last_became_active":"2023-03-03T08:43:44.004120+0000","last_became_peered":"2023-03-03T08:43:44.004120+0000","last_unstale":"2023-03-03T08:43:45.107426+0000","last_undegraded":"2023-03-03T08:43:45.107426+0000","last_fullsized":"2023-03-03T08:43:45.107426+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:36.006791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990661+0000","last_change":"2023-03-03T08:43:34.537110+0000","last_active":"2023-03-03T08:43:43.990661+0000","last_peered":"2023-03-03T08:43:43.990661+0000","last_clean":"2023-03-03T08:43:43.990661+0000","last_became_active":"2023-03-03T08:43:34.536848+0000","last_became_peered":"2023-03-03T08:43:34.536848+0000","last_unstale":"2023-03-03T08:43:43.990661+0000","last_undegraded":"2023-03-03T08:43:43.990661+0000","last_fullsized":"2023-03-03T08:43:43.990661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106366+0000","last_change":"2023-03-03T08:43:43.999266+0000","last_active":"2023-03-03T08:43:45.106366+0000","last_peered":"2023-03-03T08:43:45.106366+0000","last_clean":"2023-03-03T08:43:45.106366+0000","last_became_active":"2023-03-03T08:43:43.998929+0000","last_became_peered":"2023-03-03T08:43:43.998929+0000","last_unstale":"2023-03-03T08:43:45.106366+0000","last_undegraded":"2023-03-03T08:43:45.106366+0000","last_fullsized":"2023-03-03T08:43:45.106366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:02.236263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994220+0000","last_change":"2023-03-03T08:43:34.532154+0000","last_active":"2023-03-03T08:43:43.994220+0000","last_peered":"2023-03-03T08:43:43.994220+0000","last_clean":"2023-03-03T08:43:43.994220+0000","last_became_active":"2023-03-03T08:43:34.531844+0000","last_became_peered":"2023-03-03T08:43:34.531844+0000","last_unstale":"2023-03-03T08:43:43.994220+0000","last_undegraded":"2023-03-03T08:43:43.994220+0000","last_fullsized":"2023-03-03T08:43:43.994220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:00.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106991+0000","last_change":"2023-03-03T08:43:44.004469+0000","last_active":"2023-03-03T08:43:45.106991+0000","last_peered":"2023-03-03T08:43:45.106991+0000","last_clean":"2023-03-03T08:43:45.106991+0000","last_became_active":"2023-03-03T08:43:44.004187+0000","last_became_peered":"2023-03-03T08:43:44.004187+0000","last_unstale":"2023-03-03T08:43:45.106991+0000","last_undegraded":"2023-03-03T08:43:45.106991+0000","last_fullsized":"2023-03-03T08:43:45.106991+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:35.298857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979308+0000","last_change":"2023-03-03T08:43:34.527373+0000","last_active":"2023-03-03T08:43:43.979308+0000","last_peered":"2023-03-03T08:43:43.979308+0000","last_clean":"2023-03-03T08:43:43.979308+0000","last_became_active":"2023-03-03T08:43:34.527171+0000","last_became_peered":"2023-03-03T08:43:34.527171+0000","last_unstale":"2023-03-03T08:43:43.979308+0000","last_undegraded":"2023-03-03T08:43:43.979308+0000","last_fullsized":"2023-03-03T08:43:43.979308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:20:51.138587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108220+0000","last_change":"2023-03-03T08:43:43.996329+0000","last_active":"2023-03-03T08:43:45.108220+0000","last_peered":"2023-03-03T08:43:45.108220+0000","last_clean":"2023-03-03T08:43:45.108220+0000","last_became_active":"2023-03-03T08:43:43.995055+0000","last_became_peered":"2023-03-03T08:43:43.995055+0000","last_unstale":"2023-03-03T08:43:45.108220+0000","last_undegraded":"2023-03-03T08:43:45.108220+0000","last_fullsized":"2023-03-03T08:43:45.108220+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:57.593740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984695+0000","last_change":"2023-03-03T08:43:34.542729+0000","last_active":"2023-03-03T08:43:43.984695+0000","last_peered":"2023-03-03T08:43:43.984695+0000","last_clean":"2023-03-03T08:43:43.984695+0000","last_became_active":"2023-03-03T08:43:34.542246+0000","last_became_peered":"2023-03-03T08:43:34.542246+0000","last_unstale":"2023-03-03T08:43:43.984695+0000","last_undegraded":"2023-03-03T08:43:43.984695+0000","last_fullsized":"2023-03-03T08:43:43.984695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:14:59.995526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105365+0000","last_change":"2023-03-03T08:43:44.002358+0000","last_active":"2023-03-03T08:43:45.105365+0000","last_peered":"2023-03-03T08:43:45.105365+0000","last_clean":"2023-03-03T08:43:45.105365+0000","last_became_active":"2023-03-03T08:43:44.002020+0000","last_became_peered":"2023-03-03T08:43:44.002020+0000","last_unstale":"2023-03-03T08:43:45.105365+0000","last_undegraded":"2023-03-03T08:43:45.105365+0000","last_fullsized":"2023-03-03T08:43:45.105365+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:51.524774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982664+0000","last_change":"2023-03-03T08:43:34.549026+0000","last_active":"2023-03-03T08:43:43.982664+0000","last_peered":"2023-03-03T08:43:43.982664+0000","last_clean":"2023-03-03T08:43:43.982664+0000","last_became_active":"2023-03-03T08:43:34.548778+0000","last_became_peered":"2023-03-03T08:43:34.548778+0000","last_unstale":"2023-03-03T08:43:43.982664+0000","last_undegraded":"2023-03-03T08:43:43.982664+0000","last_fullsized":"2023-03-03T08:43:43.982664+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:41.835304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105281+0000","last_change":"2023-03-03T08:43:43.997684+0000","last_active":"2023-03-03T08:43:45.105281+0000","last_peered":"2023-03-03T08:43:45.105281+0000","last_clean":"2023-03-03T08:43:45.105281+0000","last_became_active":"2023-03-03T08:43:43.997503+0000","last_became_peered":"2023-03-03T08:43:43.997503+0000","last_unstale":"2023-03-03T08:43:45.105281+0000","last_undegraded":"2023-03-03T08:43:45.105281+0000","last_fullsized":"2023-03-03T08:43:45.105281+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:29.512524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991178+0000","last_change":"2023-03-03T08:43:34.544013+0000","last_active":"2023-03-03T08:43:43.991178+0000","last_peered":"2023-03-03T08:43:43.991178+0000","last_clean":"2023-03-03T08:43:43.991178+0000","last_became_active":"2023-03-03T08:43:34.543745+0000","last_became_peered":"2023-03-03T08:43:34.543745+0000","last_unstale":"2023-03-03T08:43:43.991178+0000","last_undegraded":"2023-03-03T08:43:43.991178+0000","last_fullsized":"2023-03-03T08:43:43.991178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:05.647977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107560+0000","last_change":"2023-03-03T08:43:43.999884+0000","last_active":"2023-03-03T08:43:45.107560+0000","last_peered":"2023-03-03T08:43:45.107560+0000","last_clean":"2023-03-03T08:43:45.107560+0000","last_became_active":"2023-03-03T08:43:43.999593+0000","last_became_peered":"2023-03-03T08:43:43.999593+0000","last_unstale":"2023-03-03T08:43:45.107560+0000","last_undegraded":"2023-03-03T08:43:45.107560+0000","last_fullsized":"2023-03-03T08:43:45.107560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:45:09.132974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982600+0000","last_change":"2023-03-03T08:43:34.553434+0000","last_active":"2023-03-03T08:43:43.982600+0000","last_peered":"2023-03-03T08:43:43.982600+0000","last_clean":"2023-03-03T08:43:43.982600+0000","last_became_active":"2023-03-03T08:43:34.553125+0000","last_became_peered":"2023-03-03T08:43:34.553125+0000","last_unstale":"2023-03-03T08:43:43.982600+0000","last_undegraded":"2023-03-03T08:43:43.982600+0000","last_fullsized":"2023-03-03T08:43:43.982600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:41:17.553506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105717+0000","last_change":"2023-03-03T08:43:44.006710+0000","last_active":"2023-03-03T08:43:45.105717+0000","last_peered":"2023-03-03T08:43:45.105717+0000","last_clean":"2023-03-03T08:43:45.105717+0000","last_became_active":"2023-03-03T08:43:44.006447+0000","last_became_peered":"2023-03-03T08:43:44.006447+0000","last_unstale":"2023-03-03T08:43:45.105717+0000","last_undegraded":"2023-03-03T08:43:45.105717+0000","last_fullsized":"2023-03-03T08:43:45.105717+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:21.767326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982538+0000","last_change":"2023-03-03T08:43:34.550889+0000","last_active":"2023-03-03T08:43:43.982538+0000","last_peered":"2023-03-03T08:43:43.982538+0000","last_clean":"2023-03-03T08:43:43.982538+0000","last_became_active":"2023-03-03T08:43:34.550605+0000","last_became_peered":"2023-03-03T08:43:34.550605+0000","last_unstale":"2023-03-03T08:43:43.982538+0000","last_undegraded":"2023-03-03T08:43:43.982538+0000","last_fullsized":"2023-03-03T08:43:43.982538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:01.118555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105567+0000","last_change":"2023-03-03T08:43:43.998231+0000","last_active":"2023-03-03T08:43:45.105567+0000","last_peered":"2023-03-03T08:43:45.105567+0000","last_clean":"2023-03-03T08:43:45.105567+0000","last_became_active":"2023-03-03T08:43:43.997941+0000","last_became_peered":"2023-03-03T08:43:43.997941+0000","last_unstale":"2023-03-03T08:43:45.105567+0000","last_undegraded":"2023-03-03T08:43:45.105567+0000","last_fullsized":"2023-03-03T08:43:45.105567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:13:32.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983100+0000","last_change":"2023-03-03T08:43:34.548229+0000","last_active":"2023-03-03T08:43:43.983100+0000","last_peered":"2023-03-03T08:43:43.983100+0000","last_clean":"2023-03-03T08:43:43.983100+0000","last_became_active":"2023-03-03T08:43:34.548013+0000","last_became_peered":"2023-03-03T08:43:34.548013+0000","last_unstale":"2023-03-03T08:43:43.983100+0000","last_undegraded":"2023-03-03T08:43:43.983100+0000","last_fullsized":"2023-03-03T08:43:43.983100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:27.793716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108186+0000","last_change":"2023-03-03T08:43:43.992834+0000","last_active":"2023-03-03T08:43:45.108186+0000","last_peered":"2023-03-03T08:43:45.108186+0000","last_clean":"2023-03-03T08:43:45.108186+0000","last_became_active":"2023-03-03T08:43:43.991577+0000","last_became_peered":"2023-03-03T08:43:43.991577+0000","last_unstale":"2023-03-03T08:43:45.108186+0000","last_undegraded":"2023-03-03T08:43:45.108186+0000","last_fullsized":"2023-03-03T08:43:45.108186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:18:37.119460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990785+0000","last_change":"2023-03-03T08:43:34.549838+0000","last_active":"2023-03-03T08:43:43.990785+0000","last_peered":"2023-03-03T08:43:43.990785+0000","last_clean":"2023-03-03T08:43:43.990785+0000","last_became_active":"2023-03-03T08:43:34.549537+0000","last_became_peered":"2023-03-03T08:43:34.549537+0000","last_unstale":"2023-03-03T08:43:43.990785+0000","last_undegraded":"2023-03-03T08:43:43.990785+0000","last_fullsized":"2023-03-03T08:43:43.990785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:36.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978412+0000","last_change":"2023-03-03T08:43:43.991488+0000","last_active":"2023-03-03T08:43:44.978412+0000","last_peered":"2023-03-03T08:43:44.978412+0000","last_clean":"2023-03-03T08:43:44.978412+0000","last_became_active":"2023-03-03T08:43:43.991139+0000","last_became_peered":"2023-03-03T08:43:43.991139+0000","last_unstale":"2023-03-03T08:43:44.978412+0000","last_undegraded":"2023-03-03T08:43:44.978412+0000","last_fullsized":"2023-03-03T08:43:44.978412+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:21.119183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983652+0000","last_change":"2023-03-03T08:43:34.533496+0000","last_active":"2023-03-03T08:43:43.983652+0000","last_peered":"2023-03-03T08:43:43.983652+0000","last_clean":"2023-03-03T08:43:43.983652+0000","last_became_active":"2023-03-03T08:43:34.532714+0000","last_became_peered":"2023-03-03T08:43:34.532714+0000","last_unstale":"2023-03-03T08:43:43.983652+0000","last_undegraded":"2023-03-03T08:43:43.983652+0000","last_fullsized":"2023-03-03T08:43:43.983652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:47.800807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105250+0000","last_change":"2023-03-03T08:43:43.996215+0000","last_active":"2023-03-03T08:43:45.105250+0000","last_peered":"2023-03-03T08:43:45.105250+0000","last_clean":"2023-03-03T08:43:45.105250+0000","last_became_active":"2023-03-03T08:43:43.995791+0000","last_became_peered":"2023-03-03T08:43:43.995791+0000","last_unstale":"2023-03-03T08:43:45.105250+0000","last_undegraded":"2023-03-03T08:43:45.105250+0000","last_fullsized":"2023-03-03T08:43:45.105250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:50:41.130427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979929+0000","last_change":"2023-03-03T08:43:34.533673+0000","last_active":"2023-03-03T08:43:43.979929+0000","last_peered":"2023-03-03T08:43:43.979929+0000","last_clean":"2023-03-03T08:43:43.979929+0000","last_became_active":"2023-03-03T08:43:34.532283+0000","last_became_peered":"2023-03-03T08:43:34.532283+0000","last_unstale":"2023-03-03T08:43:43.979929+0000","last_undegraded":"2023-03-03T08:43:43.979929+0000","last_fullsized":"2023-03-03T08:43:43.979929+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:25.591188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979205+0000","last_change":"2023-03-03T08:43:43.999435+0000","last_active":"2023-03-03T08:43:44.979205+0000","last_peered":"2023-03-03T08:43:44.979205+0000","last_clean":"2023-03-03T08:43:44.979205+0000","last_became_active":"2023-03-03T08:43:43.997037+0000","last_became_peered":"2023-03-03T08:43:43.997037+0000","last_unstale":"2023-03-03T08:43:44.979205+0000","last_undegraded":"2023-03-03T08:43:44.979205+0000","last_fullsized":"2023-03-03T08:43:44.979205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:54.176343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980585+0000","last_change":"2023-03-03T08:43:34.537664+0000","last_active":"2023-03-03T08:43:43.980585+0000","last_peered":"2023-03-03T08:43:43.980585+0000","last_clean":"2023-03-03T08:43:43.980585+0000","last_became_active":"2023-03-03T08:43:34.537214+0000","last_became_peered":"2023-03-03T08:43:34.537214+0000","last_unstale":"2023-03-03T08:43:43.980585+0000","last_undegraded":"2023-03-03T08:43:43.980585+0000","last_fullsized":"2023-03-03T08:43:43.980585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:54:09.222135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106227+0000","last_change":"2023-03-03T08:43:44.000140+0000","last_active":"2023-03-03T08:43:45.106227+0000","last_peered":"2023-03-03T08:43:45.106227+0000","last_clean":"2023-03-03T08:43:45.106227+0000","last_became_active":"2023-03-03T08:43:43.999778+0000","last_became_peered":"2023-03-03T08:43:43.999778+0000","last_unstale":"2023-03-03T08:43:45.106227+0000","last_undegraded":"2023-03-03T08:43:45.106227+0000","last_fullsized":"2023-03-03T08:43:45.106227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:29:10.745759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987366+0000","last_change":"2023-03-03T08:43:34.543620+0000","last_active":"2023-03-03T08:43:43.987366+0000","last_peered":"2023-03-03T08:43:43.987366+0000","last_clean":"2023-03-03T08:43:43.987366+0000","last_became_active":"2023-03-03T08:43:34.543370+0000","last_became_peered":"2023-03-03T08:43:34.543370+0000","last_unstale":"2023-03-03T08:43:43.987366+0000","last_undegraded":"2023-03-03T08:43:43.987366+0000","last_fullsized":"2023-03-03T08:43:43.987366+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:40:32.808203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979449+0000","last_change":"2023-03-03T08:43:43.995084+0000","last_active":"2023-03-03T08:43:44.979449+0000","last_peered":"2023-03-03T08:43:44.979449+0000","last_clean":"2023-03-03T08:43:44.979449+0000","last_became_active":"2023-03-03T08:43:43.994824+0000","last_became_peered":"2023-03-03T08:43:43.994824+0000","last_unstale":"2023-03-03T08:43:44.979449+0000","last_undegraded":"2023-03-03T08:43:44.979449+0000","last_fullsized":"2023-03-03T08:43:44.979449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:04:17.481969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985302+0000","last_change":"2023-03-03T08:43:34.538467+0000","last_active":"2023-03-03T08:43:43.985302+0000","last_peered":"2023-03-03T08:43:43.985302+0000","last_clean":"2023-03-03T08:43:43.985302+0000","last_became_active":"2023-03-03T08:43:34.538207+0000","last_became_peered":"2023-03-03T08:43:34.538207+0000","last_unstale":"2023-03-03T08:43:43.985302+0000","last_undegraded":"2023-03-03T08:43:43.985302+0000","last_fullsized":"2023-03-03T08:43:43.985302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:14.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980577+0000","last_change":"2023-03-03T08:43:43.996076+0000","last_active":"2023-03-03T08:43:44.980577+0000","last_peered":"2023-03-03T08:43:44.980577+0000","last_clean":"2023-03-03T08:43:44.980577+0000","last_became_active":"2023-03-03T08:43:43.995799+0000","last_became_peered":"2023-03-03T08:43:43.995799+0000","last_unstale":"2023-03-03T08:43:44.980577+0000","last_undegraded":"2023-03-03T08:43:44.980577+0000","last_fullsized":"2023-03-03T08:43:44.980577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:35:35.889895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986289+0000","last_change":"2023-03-03T08:43:34.544721+0000","last_active":"2023-03-03T08:43:43.986289+0000","last_peered":"2023-03-03T08:43:43.986289+0000","last_clean":"2023-03-03T08:43:43.986289+0000","last_became_active":"2023-03-03T08:43:34.544461+0000","last_became_peered":"2023-03-03T08:43:34.544461+0000","last_unstale":"2023-03-03T08:43:43.986289+0000","last_undegraded":"2023-03-03T08:43:43.986289+0000","last_fullsized":"2023-03-03T08:43:43.986289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:52:11.890779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980078+0000","last_change":"2023-03-03T08:43:44.006352+0000","last_active":"2023-03-03T08:43:44.980078+0000","last_peered":"2023-03-03T08:43:44.980078+0000","last_clean":"2023-03-03T08:43:44.980078+0000","last_became_active":"2023-03-03T08:43:44.005802+0000","last_became_peered":"2023-03-03T08:43:44.005802+0000","last_unstale":"2023-03-03T08:43:44.980078+0000","last_undegraded":"2023-03-03T08:43:44.980078+0000","last_fullsized":"2023-03-03T08:43:44.980078+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:52.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986425+0000","last_change":"2023-03-03T08:43:34.541669+0000","last_active":"2023-03-03T08:43:43.986425+0000","last_peered":"2023-03-03T08:43:43.986425+0000","last_clean":"2023-03-03T08:43:43.986425+0000","last_became_active":"2023-03-03T08:43:34.541104+0000","last_became_peered":"2023-03-03T08:43:34.541104+0000","last_unstale":"2023-03-03T08:43:43.986425+0000","last_undegraded":"2023-03-03T08:43:43.986425+0000","last_fullsized":"2023-03-03T08:43:43.986425+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980059+0000","last_change":"2023-03-03T08:43:43.998937+0000","last_active":"2023-03-03T08:43:44.980059+0000","last_peered":"2023-03-03T08:43:44.980059+0000","last_clean":"2023-03-03T08:43:44.980059+0000","last_became_active":"2023-03-03T08:43:43.998164+0000","last_became_peered":"2023-03-03T08:43:43.998164+0000","last_unstale":"2023-03-03T08:43:44.980059+0000","last_undegraded":"2023-03-03T08:43:44.980059+0000","last_fullsized":"2023-03-03T08:43:44.980059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:53.741704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.546272+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.546006+0000","last_became_peered":"2023-03-03T08:43:34.546006+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:15.121761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":3561,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3561,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":118397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":119470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9336,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362504,"statfs":{"total":96636764160,"available":96627204096,"internally_reserved":0,"allocated":901120,"data_stored":710394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8244,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363596,"statfs":{"total":96636764160,"available":96628322304,"internally_reserved":0,"allocated":307200,"data_stored":118565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8252,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363588,"statfs":{"total":96636764160,"available":96628314112,"internally_reserved":0,"allocated":315392,"data_stored":124451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":121300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":122249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9260,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362580,"statfs":{"total":96636764160,"available":96627281920,"internally_reserved":0,"allocated":888832,"data_stored":709827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:44:07.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:44:07.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.988+0000 7f07b1a90700 1 Processor -- start 2023-03-03T08:44:07.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.988+0000 7f07b1a90700 1 -- start start 2023-03-03T08:44:07.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07b1a90700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0719e0 0x7f07ac150930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07b1a90700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07ac151e20 0x7f07ac0aa750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07b1a90700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c14f0 0x7f07ac0c1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07b1a90700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ac152270 con 0x7f07ac151e20 2023-03-03T08:44:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07b1a90700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ac0c2760 con 0x7f07ac0c14f0 2023-03-03T08:44:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07b1a90700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac0a93d0 con 0x7f07ac0719e0 2023-03-03T08:44:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aaffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07ac151e20 0x7f07ac0aa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aaffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07ac151e20 0x7f07ac0aa750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52284/0 (socket says 172.21.15.7:52284) 2023-03-03T08:44:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aaffd700 1 -- 172.21.15.7:0/2408112672 learned_addr learned my addr 172.21.15.7:0/2408112672 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:07.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07abfff700 1 --2- 172.21.15.7:0/2408112672 >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c14f0 0x7f07ac0c1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07ab7fe700 1 --2- 172.21.15.7:0/2408112672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0719e0 0x7f07ac150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aaffd700 1 -- 172.21.15.7:0/2408112672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0719e0 msgr2=0x7f07ac150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:07.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aaffd700 1 --2- 172.21.15.7:0/2408112672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0719e0 0x7f07ac150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aaffd700 1 -- 172.21.15.7:0/2408112672 >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c14f0 msgr2=0x7f07ac0c1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:07.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aaffd700 1 --2- 172.21.15.7:0/2408112672 >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c14f0 0x7f07ac0c1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aaffd700 1 -- 172.21.15.7:0/2408112672 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac0a96b0 con 0x7f07ac151e20 2023-03-03T08:44:07.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aaffd700 1 --2- 172.21.15.7:0/2408112672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac151e20 0x7f07ac0aa750 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0c1f90 tx=0x7f07a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1908414e75f3d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:07.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aa7fc700 1 -- 172.21.15.7:0/2408112672 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a000b870 con 0x7f07ac151e20 2023-03-03T08:44:07.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aa7fc700 1 -- 172.21.15.7:0/2408112672 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a000b9d0 con 0x7f07ac151e20 2023-03-03T08:44:07.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.989+0000 7f07aa7fc700 1 -- 172.21.15.7:0/2408112672 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a000bd60 con 0x7f07ac151e20 2023-03-03T08:44:07.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.990+0000 7f07b1a90700 1 -- 172.21.15.7:0/2408112672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac151e20 msgr2=0x7f07ac0aa750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.990+0000 7f07b1a90700 1 --2- 172.21.15.7:0/2408112672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac151e20 0x7f07ac0aa750 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0c1f90 tx=0x7f07a0009bd0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.990+0000 7f07b1a90700 1 -- 172.21.15.7:0/2408112672 shutdown_connections 2023-03-03T08:44:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.990+0000 7f07b1a90700 1 --2- 172.21.15.7:0/2408112672 >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c14f0 0x7f07ac0c1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.990+0000 7f07b1a90700 1 --2- 172.21.15.7:0/2408112672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0719e0 0x7f07ac150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.990+0000 7f07b1a90700 1 --2- 172.21.15.7:0/2408112672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac151e20 0x7f07ac0aa750 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.990+0000 7f07b1a90700 1 -- 172.21.15.7:0/2408112672 >> 172.21.15.7:0/2408112672 conn(0x7f07ac0c62e0 msgr2=0x7f07ac0c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.990+0000 7f07b1a90700 1 -- 172.21.15.7:0/2408112672 shutdown_connections 2023-03-03T08:44:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.990+0000 7f07b1a90700 1 -- 172.21.15.7:0/2408112672 wait complete. 2023-03-03T08:44:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.991+0000 7f07b1a90700 1 Processor -- start 2023-03-03T08:44:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.992+0000 7f07b1a90700 1 -- start start 2023-03-03T08:44:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.992+0000 7f07b1a90700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0719e0 0x7f07ac1eb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.992+0000 7f07b1a90700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0c14f0 0x7f07ac1ebd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.992+0000 7f07b1a90700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac151e20 0x7f07ac150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.992+0000 7f07b1a90700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac142f90 con 0x7f07ac0c14f0 2023-03-03T08:44:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.992+0000 7f07b1a90700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac141720 con 0x7f07ac0719e0 2023-03-03T08:44:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.992+0000 7f07b1a90700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac14f3c0 con 0x7f07ac151e20 2023-03-03T08:44:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07aaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0c14f0 0x7f07ac1ebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07ab7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0719e0 0x7f07ac1eb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07aaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0c14f0 0x7f07ac1ebd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52292/0 (socket says 172.21.15.7:52292) 2023-03-03T08:44:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07aaffd700 1 -- 172.21.15.7:0/159821195 learned_addr learned my addr 172.21.15.7:0/159821195 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07abfff700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac151e20 0x7f07ac150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07aaffd700 1 -- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac151e20 msgr2=0x7f07ac150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07aaffd700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac151e20 0x7f07ac150ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07aaffd700 1 -- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0719e0 msgr2=0x7f07ac1eb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07aaffd700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0719e0 0x7f07ac1eb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07aaffd700 1 -- 172.21.15.7:0/159821195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a0009550 con 0x7f07ac0c14f0 2023-03-03T08:44:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07ab7fe700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0719e0 0x7f07ac1eb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:08.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07aaffd700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0c14f0 0x7f07ac1ebd70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f07a0000c00 tx=0x7f07a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:08.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f0793fff700 1 -- 172.21.15.7:0/159821195 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a0004480 con 0x7f07ac0c14f0 2023-03-03T08:44:08.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f0793fff700 1 -- 172.21.15.7:0/159821195 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a00045e0 con 0x7f07ac0c14f0 2023-03-03T08:44:08.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f07b1a90700 1 -- 172.21.15.7:0/159821195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ac0a9c70 con 0x7f07ac0c14f0 2023-03-03T08:44:08.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.993+0000 7f0793fff700 1 -- 172.21.15.7:0/159821195 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a00038e0 con 0x7f07ac0c14f0 2023-03-03T08:44:08.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.994+0000 7f07b1a90700 1 -- 172.21.15.7:0/159821195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ac0a9990 con 0x7f07ac0c14f0 2023-03-03T08:44:08.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.995+0000 7f07b1a90700 1 -- 172.21.15.7:0/159821195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ac0a9f50 con 0x7f07ac0c14f0 2023-03-03T08:44:08.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.996+0000 7f0793fff700 1 -- 172.21.15.7:0/159821195 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f07a000f050 con 0x7f07ac0c14f0 2023-03-03T08:44:08.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.996+0000 7f0793fff700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07980682f0 0x7f079806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.996+0000 7f0793fff700 1 -- 172.21.15.7:0/159821195 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f07a0087cf0 con 0x7f07ac0c14f0 2023-03-03T08:44:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.997+0000 7f07ab7fe700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07980682f0 0x7f079806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:07.997+0000 7f07ab7fe700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07980682f0 0x7f079806a7a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f079c004630 tx=0x7f079c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.000+0000 7f0793fff700 1 -- 172.21.15.7:0/159821195 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f07a0054260 con 0x7f07ac0c14f0 2023-03-03T08:44:08.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.189+0000 7f07b1a90700 1 -- 172.21.15.7:0/159821195 --> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07ac06b6b0 con 0x7f07980682f0 2023-03-03T08:44:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.204+0000 7f0793fff700 1 -- 172.21.15.7:0/159821195 <== mgr.4102 v2:172.21.15.143:6832/96367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525124 (secure 0 0 0) 0x7f07ac06b6b0 con 0x7f07980682f0 2023-03-03T08:44:08.211 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 -- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07980682f0 msgr2=0x7f079806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07980682f0 0x7f079806a7a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f079c004630 tx=0x7f079c008040 comp rx=0 tx=0).stop 2023-03-03T08:44:08.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 -- 172.21.15.7:0/159821195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0c14f0 msgr2=0x7f07ac1ebd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0c14f0 0x7f07ac1ebd70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f07a0000c00 tx=0x7f07a0004750 comp rx=0 tx=0).stop 2023-03-03T08:44:08.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 -- 172.21.15.7:0/159821195 shutdown_connections 2023-03-03T08:44:08.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac151e20 0x7f07ac150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0719e0 0x7f07ac1eb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07980682f0 0x7f079806a7a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 --2- 172.21.15.7:0/159821195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0c14f0 0x7f07ac1ebd70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 -- 172.21.15.7:0/159821195 >> 172.21.15.7:0/159821195 conn(0x7f07ac0c62e0 msgr2=0x7f07ac06bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:08.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 -- 172.21.15.7:0/159821195 shutdown_connections 2023-03-03T08:44:08.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.208+0000 7f07b1a90700 1 -- 172.21.15.7:0/159821195 wait complete. 2023-03-03T08:44:08.216 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-03-03T08:44:08.222 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2023-03-03T08:44:07.114985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597490,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":255,"num_write_kb":2450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68116,"kb_used_data":3540,"kb_used_omap":0,"kb_used_meta":64512,"kb_avail":754906604,"statfs":{"total":773094113280,"available":773024362496,"internally_reserved":0,"allocated":3624960,"data_stored":2144653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66060288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2462,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003970"},"pg_stats":[{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980166+0000","last_change":"2023-03-03T08:43:43.997091+0000","last_active":"2023-03-03T08:43:44.980166+0000","last_peered":"2023-03-03T08:43:44.980166+0000","last_clean":"2023-03-03T08:43:44.980166+0000","last_became_active":"2023-03-03T08:43:43.996805+0000","last_became_peered":"2023-03-03T08:43:43.996805+0000","last_unstale":"2023-03-03T08:43:44.980166+0000","last_undegraded":"2023-03-03T08:43:44.980166+0000","last_fullsized":"2023-03-03T08:43:44.980166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982505+0000","last_change":"2023-03-03T08:43:34.546828+0000","last_active":"2023-03-03T08:43:43.982505+0000","last_peered":"2023-03-03T08:43:43.982505+0000","last_clean":"2023-03-03T08:43:43.982505+0000","last_became_active":"2023-03-03T08:43:34.546559+0000","last_became_peered":"2023-03-03T08:43:34.546559+0000","last_unstale":"2023-03-03T08:43:43.982505+0000","last_undegraded":"2023-03-03T08:43:43.982505+0000","last_fullsized":"2023-03-03T08:43:43.982505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:21:14.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106061+0000","last_change":"2023-03-03T08:43:43.995073+0000","last_active":"2023-03-03T08:43:45.106061+0000","last_peered":"2023-03-03T08:43:45.106061+0000","last_clean":"2023-03-03T08:43:45.106061+0000","last_became_active":"2023-03-03T08:43:43.994797+0000","last_became_peered":"2023-03-03T08:43:43.994797+0000","last_unstale":"2023-03-03T08:43:45.106061+0000","last_undegraded":"2023-03-03T08:43:45.106061+0000","last_fullsized":"2023-03-03T08:43:45.106061+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:21:52.642936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982397+0000","last_change":"2023-03-03T08:43:34.552931+0000","last_active":"2023-03-03T08:43:43.982397+0000","last_peered":"2023-03-03T08:43:43.982397+0000","last_clean":"2023-03-03T08:43:43.982397+0000","last_became_active":"2023-03-03T08:43:34.552623+0000","last_became_peered":"2023-03-03T08:43:34.552623+0000","last_unstale":"2023-03-03T08:43:43.982397+0000","last_undegraded":"2023-03-03T08:43:43.982397+0000","last_fullsized":"2023-03-03T08:43:43.982397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:07:43.954897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979088+0000","last_change":"2023-03-03T08:43:43.999245+0000","last_active":"2023-03-03T08:43:44.979088+0000","last_peered":"2023-03-03T08:43:44.979088+0000","last_clean":"2023-03-03T08:43:44.979088+0000","last_became_active":"2023-03-03T08:43:43.998592+0000","last_became_peered":"2023-03-03T08:43:43.998592+0000","last_unstale":"2023-03-03T08:43:44.979088+0000","last_undegraded":"2023-03-03T08:43:44.979088+0000","last_fullsized":"2023-03-03T08:43:44.979088+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:23.788724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991054+0000","last_change":"2023-03-03T08:43:34.544408+0000","last_active":"2023-03-03T08:43:43.991054+0000","last_peered":"2023-03-03T08:43:43.991054+0000","last_clean":"2023-03-03T08:43:43.991054+0000","last_became_active":"2023-03-03T08:43:34.543977+0000","last_became_peered":"2023-03-03T08:43:34.543977+0000","last_unstale":"2023-03-03T08:43:43.991054+0000","last_undegraded":"2023-03-03T08:43:43.991054+0000","last_fullsized":"2023-03-03T08:43:43.991054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.074082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105254+0000","last_change":"2023-03-03T08:43:43.994545+0000","last_active":"2023-03-03T08:43:45.105254+0000","last_peered":"2023-03-03T08:43:45.105254+0000","last_clean":"2023-03-03T08:43:45.105254+0000","last_became_active":"2023-03-03T08:43:43.994271+0000","last_became_peered":"2023-03-03T08:43:43.994271+0000","last_unstale":"2023-03-03T08:43:45.105254+0000","last_undegraded":"2023-03-03T08:43:45.105254+0000","last_fullsized":"2023-03-03T08:43:45.105254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:15.925960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989064+0000","last_change":"2023-03-03T08:43:34.538786+0000","last_active":"2023-03-03T08:43:43.989064+0000","last_peered":"2023-03-03T08:43:43.989064+0000","last_clean":"2023-03-03T08:43:43.989064+0000","last_became_active":"2023-03-03T08:43:34.538494+0000","last_became_peered":"2023-03-03T08:43:34.538494+0000","last_unstale":"2023-03-03T08:43:43.989064+0000","last_undegraded":"2023-03-03T08:43:43.989064+0000","last_fullsized":"2023-03-03T08:43:43.989064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:53.838286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105391+0000","last_change":"2023-03-03T08:43:43.993678+0000","last_active":"2023-03-03T08:43:45.105391+0000","last_peered":"2023-03-03T08:43:45.105391+0000","last_clean":"2023-03-03T08:43:45.105391+0000","last_became_active":"2023-03-03T08:43:43.993402+0000","last_became_peered":"2023-03-03T08:43:43.993402+0000","last_unstale":"2023-03-03T08:43:45.105391+0000","last_undegraded":"2023-03-03T08:43:45.105391+0000","last_fullsized":"2023-03-03T08:43:45.105391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:31:00.747216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983479+0000","last_change":"2023-03-03T08:43:34.547197+0000","last_active":"2023-03-03T08:43:43.983479+0000","last_peered":"2023-03-03T08:43:43.983479+0000","last_clean":"2023-03-03T08:43:43.983479+0000","last_became_active":"2023-03-03T08:43:34.546927+0000","last_became_peered":"2023-03-03T08:43:34.546927+0000","last_unstale":"2023-03-03T08:43:43.983479+0000","last_undegraded":"2023-03-03T08:43:43.983479+0000","last_fullsized":"2023-03-03T08:43:43.983479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:21:34.845968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984318+0000","last_change":"2023-03-03T08:43:34.540812+0000","last_active":"2023-03-03T08:43:43.984318+0000","last_peered":"2023-03-03T08:43:43.984318+0000","last_clean":"2023-03-03T08:43:43.984318+0000","last_became_active":"2023-03-03T08:43:34.540549+0000","last_became_peered":"2023-03-03T08:43:34.540549+0000","last_unstale":"2023-03-03T08:43:43.984318+0000","last_undegraded":"2023-03-03T08:43:43.984318+0000","last_fullsized":"2023-03-03T08:43:43.984318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982875+0000","last_change":"2023-03-03T08:43:34.108426+0000","last_active":"2023-03-03T08:43:43.982875+0000","last_peered":"2023-03-03T08:43:43.982875+0000","last_clean":"2023-03-03T08:43:43.982875+0000","last_became_active":"2023-03-03T08:43:34.108009+0000","last_became_peered":"2023-03-03T08:43:34.108009+0000","last_unstale":"2023-03-03T08:43:43.982875+0000","last_undegraded":"2023-03-03T08:43:43.982875+0000","last_fullsized":"2023-03-03T08:43:43.982875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:53.632744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979479+0000","last_change":"2023-03-03T08:43:43.575708+0000","last_active":"2023-03-03T08:43:44.979479+0000","last_peered":"2023-03-03T08:43:44.979479+0000","last_clean":"2023-03-03T08:43:44.979479+0000","last_became_active":"2023-03-03T08:43:43.575417+0000","last_became_peered":"2023-03-03T08:43:43.575417+0000","last_unstale":"2023-03-03T08:43:44.979479+0000","last_undegraded":"2023-03-03T08:43:44.979479+0000","last_fullsized":"2023-03-03T08:43:44.979479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:25.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106018+0000","last_change":"2023-03-03T08:43:43.999274+0000","last_active":"2023-03-03T08:43:45.106018+0000","last_peered":"2023-03-03T08:43:45.106018+0000","last_clean":"2023-03-03T08:43:45.106018+0000","last_became_active":"2023-03-03T08:43:43.998823+0000","last_became_peered":"2023-03-03T08:43:43.998823+0000","last_unstale":"2023-03-03T08:43:45.106018+0000","last_undegraded":"2023-03-03T08:43:45.106018+0000","last_fullsized":"2023-03-03T08:43:45.106018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:48.799261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992918+0000","last_change":"2023-03-03T08:43:34.532482+0000","last_active":"2023-03-03T08:43:43.992918+0000","last_peered":"2023-03-03T08:43:43.992918+0000","last_clean":"2023-03-03T08:43:43.992918+0000","last_became_active":"2023-03-03T08:43:34.531751+0000","last_became_peered":"2023-03-03T08:43:34.531751+0000","last_unstale":"2023-03-03T08:43:43.992918+0000","last_undegraded":"2023-03-03T08:43:43.992918+0000","last_fullsized":"2023-03-03T08:43:43.992918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:47:45.214945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982691+0000","last_change":"2023-03-03T08:43:34.109782+0000","last_active":"2023-03-03T08:43:43.982691+0000","last_peered":"2023-03-03T08:43:43.982691+0000","last_clean":"2023-03-03T08:43:43.982691+0000","last_became_active":"2023-03-03T08:43:34.109513+0000","last_became_peered":"2023-03-03T08:43:34.109513+0000","last_unstale":"2023-03-03T08:43:43.982691+0000","last_undegraded":"2023-03-03T08:43:43.982691+0000","last_fullsized":"2023-03-03T08:43:43.982691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:16.185161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105634+0000","last_change":"2023-03-03T08:43:43.576885+0000","last_active":"2023-03-03T08:43:45.105634+0000","last_peered":"2023-03-03T08:43:45.105634+0000","last_clean":"2023-03-03T08:43:45.105634+0000","last_became_active":"2023-03-03T08:43:43.576577+0000","last_became_peered":"2023-03-03T08:43:43.576577+0000","last_unstale":"2023-03-03T08:43:45.105634+0000","last_undegraded":"2023-03-03T08:43:45.105634+0000","last_fullsized":"2023-03-03T08:43:45.105634+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:26.501956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978988+0000","last_change":"2023-03-03T08:43:43.992020+0000","last_active":"2023-03-03T08:43:44.978988+0000","last_peered":"2023-03-03T08:43:44.978988+0000","last_clean":"2023-03-03T08:43:44.978988+0000","last_became_active":"2023-03-03T08:43:43.991674+0000","last_became_peered":"2023-03-03T08:43:43.991674+0000","last_unstale":"2023-03-03T08:43:44.978988+0000","last_undegraded":"2023-03-03T08:43:44.978988+0000","last_fullsized":"2023-03-03T08:43:44.978988+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:54:30.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985919+0000","last_change":"2023-03-03T08:43:34.535451+0000","last_active":"2023-03-03T08:43:43.985919+0000","last_peered":"2023-03-03T08:43:43.985919+0000","last_clean":"2023-03-03T08:43:43.985919+0000","last_became_active":"2023-03-03T08:43:34.535129+0000","last_became_peered":"2023-03-03T08:43:34.535129+0000","last_unstale":"2023-03-03T08:43:43.985919+0000","last_undegraded":"2023-03-03T08:43:43.985919+0000","last_fullsized":"2023-03-03T08:43:43.985919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:30.645128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981082+0000","last_change":"2023-03-03T08:43:34.117851+0000","last_active":"2023-03-03T08:43:43.981082+0000","last_peered":"2023-03-03T08:43:43.981082+0000","last_clean":"2023-03-03T08:43:43.981082+0000","last_became_active":"2023-03-03T08:43:34.117548+0000","last_became_peered":"2023-03-03T08:43:34.117548+0000","last_unstale":"2023-03-03T08:43:43.981082+0000","last_undegraded":"2023-03-03T08:43:43.981082+0000","last_fullsized":"2023-03-03T08:43:43.981082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:20.039817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106904+0000","last_change":"2023-03-03T08:43:43.569082+0000","last_active":"2023-03-03T08:43:45.106904+0000","last_peered":"2023-03-03T08:43:45.106904+0000","last_clean":"2023-03-03T08:43:45.106904+0000","last_became_active":"2023-03-03T08:43:43.568838+0000","last_became_peered":"2023-03-03T08:43:43.568838+0000","last_unstale":"2023-03-03T08:43:45.106904+0000","last_undegraded":"2023-03-03T08:43:45.106904+0000","last_fullsized":"2023-03-03T08:43:45.106904+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:47.610848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979556+0000","last_change":"2023-03-03T08:43:43.995210+0000","last_active":"2023-03-03T08:43:44.979556+0000","last_peered":"2023-03-03T08:43:44.979556+0000","last_clean":"2023-03-03T08:43:44.979556+0000","last_became_active":"2023-03-03T08:43:43.994899+0000","last_became_peered":"2023-03-03T08:43:43.994899+0000","last_unstale":"2023-03-03T08:43:44.979556+0000","last_undegraded":"2023-03-03T08:43:44.979556+0000","last_fullsized":"2023-03-03T08:43:44.979556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:37:11.974933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985908+0000","last_change":"2023-03-03T08:43:34.547322+0000","last_active":"2023-03-03T08:43:43.985908+0000","last_peered":"2023-03-03T08:43:43.985908+0000","last_clean":"2023-03-03T08:43:43.985908+0000","last_became_active":"2023-03-03T08:43:34.546692+0000","last_became_peered":"2023-03-03T08:43:34.546692+0000","last_unstale":"2023-03-03T08:43:43.985908+0000","last_undegraded":"2023-03-03T08:43:43.985908+0000","last_fullsized":"2023-03-03T08:43:43.985908+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:11:26.366920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982836+0000","last_change":"2023-03-03T08:43:34.110187+0000","last_active":"2023-03-03T08:43:43.982836+0000","last_peered":"2023-03-03T08:43:43.982836+0000","last_clean":"2023-03-03T08:43:43.982836+0000","last_became_active":"2023-03-03T08:43:34.109845+0000","last_became_peered":"2023-03-03T08:43:34.109845+0000","last_unstale":"2023-03-03T08:43:43.982836+0000","last_undegraded":"2023-03-03T08:43:43.982836+0000","last_fullsized":"2023-03-03T08:43:43.982836+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:33.507500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105532+0000","last_change":"2023-03-03T08:43:43.576805+0000","last_active":"2023-03-03T08:43:45.105532+0000","last_peered":"2023-03-03T08:43:45.105532+0000","last_clean":"2023-03-03T08:43:45.105532+0000","last_became_active":"2023-03-03T08:43:43.576241+0000","last_became_peered":"2023-03-03T08:43:43.576241+0000","last_unstale":"2023-03-03T08:43:45.105532+0000","last_undegraded":"2023-03-03T08:43:45.105532+0000","last_fullsized":"2023-03-03T08:43:45.105532+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:34:35.821307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105632+0000","last_change":"2023-03-03T08:43:43.991142+0000","last_active":"2023-03-03T08:43:45.105632+0000","last_peered":"2023-03-03T08:43:45.105632+0000","last_clean":"2023-03-03T08:43:45.105632+0000","last_became_active":"2023-03-03T08:43:43.990868+0000","last_became_peered":"2023-03-03T08:43:43.990868+0000","last_unstale":"2023-03-03T08:43:45.105632+0000","last_undegraded":"2023-03-03T08:43:45.105632+0000","last_fullsized":"2023-03-03T08:43:45.105632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:58.831806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106429+0000","last_change":"2023-03-03T08:43:43.995618+0000","last_active":"2023-03-03T08:43:45.106429+0000","last_peered":"2023-03-03T08:43:45.106429+0000","last_clean":"2023-03-03T08:43:45.106429+0000","last_became_active":"2023-03-03T08:43:43.995350+0000","last_became_peered":"2023-03-03T08:43:43.995350+0000","last_unstale":"2023-03-03T08:43:45.106429+0000","last_undegraded":"2023-03-03T08:43:45.106429+0000","last_fullsized":"2023-03-03T08:43:45.106429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:46.445223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988843+0000","last_change":"2023-03-03T08:43:34.541172+0000","last_active":"2023-03-03T08:43:43.988843+0000","last_peered":"2023-03-03T08:43:43.988843+0000","last_clean":"2023-03-03T08:43:43.988843+0000","last_became_active":"2023-03-03T08:43:34.540739+0000","last_became_peered":"2023-03-03T08:43:34.540739+0000","last_unstale":"2023-03-03T08:43:43.988843+0000","last_undegraded":"2023-03-03T08:43:43.988843+0000","last_fullsized":"2023-03-03T08:43:43.988843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:40.058265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987475+0000","last_change":"2023-03-03T08:43:34.112667+0000","last_active":"2023-03-03T08:43:43.987475+0000","last_peered":"2023-03-03T08:43:43.987475+0000","last_clean":"2023-03-03T08:43:43.987475+0000","last_became_active":"2023-03-03T08:43:34.112386+0000","last_became_peered":"2023-03-03T08:43:34.112386+0000","last_unstale":"2023-03-03T08:43:43.987475+0000","last_undegraded":"2023-03-03T08:43:43.987475+0000","last_fullsized":"2023-03-03T08:43:43.987475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:40.504392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980167+0000","last_change":"2023-03-03T08:43:43.573962+0000","last_active":"2023-03-03T08:43:44.980167+0000","last_peered":"2023-03-03T08:43:44.980167+0000","last_clean":"2023-03-03T08:43:44.980167+0000","last_became_active":"2023-03-03T08:43:43.573200+0000","last_became_peered":"2023-03-03T08:43:43.573200+0000","last_unstale":"2023-03-03T08:43:44.980167+0000","last_undegraded":"2023-03-03T08:43:44.980167+0000","last_fullsized":"2023-03-03T08:43:44.980167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:29.511681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984698+0000","last_change":"2023-03-03T08:43:34.537850+0000","last_active":"2023-03-03T08:43:43.984698+0000","last_peered":"2023-03-03T08:43:43.984698+0000","last_clean":"2023-03-03T08:43:43.984698+0000","last_became_active":"2023-03-03T08:43:34.537550+0000","last_became_peered":"2023-03-03T08:43:34.537550+0000","last_unstale":"2023-03-03T08:43:43.984698+0000","last_undegraded":"2023-03-03T08:43:43.984698+0000","last_fullsized":"2023-03-03T08:43:43.984698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:37:36.893757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990299+0000","last_change":"2023-03-03T08:43:34.109127+0000","last_active":"2023-03-03T08:43:43.990299+0000","last_peered":"2023-03-03T08:43:43.990299+0000","last_clean":"2023-03-03T08:43:43.990299+0000","last_became_active":"2023-03-03T08:43:34.108869+0000","last_became_peered":"2023-03-03T08:43:34.108869+0000","last_unstale":"2023-03-03T08:43:43.990299+0000","last_undegraded":"2023-03-03T08:43:43.990299+0000","last_fullsized":"2023-03-03T08:43:43.990299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:19:00.275620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981298+0000","last_change":"2023-03-03T08:43:29.354627+0000","last_active":"2023-03-03T08:43:43.981298+0000","last_peered":"2023-03-03T08:43:43.981298+0000","last_clean":"2023-03-03T08:43:43.981298+0000","last_became_active":"2023-03-03T08:43:29.354271+0000","last_became_peered":"2023-03-03T08:43:29.354271+0000","last_unstale":"2023-03-03T08:43:43.981298+0000","last_undegraded":"2023-03-03T08:43:43.981298+0000","last_fullsized":"2023-03-03T08:43:43.981298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:05.946053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979432+0000","last_change":"2023-03-03T08:43:43.575567+0000","last_active":"2023-03-03T08:43:44.979432+0000","last_peered":"2023-03-03T08:43:44.979432+0000","last_clean":"2023-03-03T08:43:44.979432+0000","last_became_active":"2023-03-03T08:43:43.575276+0000","last_became_peered":"2023-03-03T08:43:43.575276+0000","last_unstale":"2023-03-03T08:43:44.979432+0000","last_undegraded":"2023-03-03T08:43:44.979432+0000","last_fullsized":"2023-03-03T08:43:44.979432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:12.876790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107250+0000","last_change":"2023-03-03T08:43:43.998276+0000","last_active":"2023-03-03T08:43:45.107250+0000","last_peered":"2023-03-03T08:43:45.107250+0000","last_clean":"2023-03-03T08:43:45.107250+0000","last_became_active":"2023-03-03T08:43:43.996018+0000","last_became_peered":"2023-03-03T08:43:43.996018+0000","last_unstale":"2023-03-03T08:43:45.107250+0000","last_undegraded":"2023-03-03T08:43:45.107250+0000","last_fullsized":"2023-03-03T08:43:45.107250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:08.588278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988844+0000","last_change":"2023-03-03T08:43:34.539018+0000","last_active":"2023-03-03T08:43:43.988844+0000","last_peered":"2023-03-03T08:43:43.988844+0000","last_clean":"2023-03-03T08:43:43.988844+0000","last_became_active":"2023-03-03T08:43:34.538375+0000","last_became_peered":"2023-03-03T08:43:34.538375+0000","last_unstale":"2023-03-03T08:43:43.988844+0000","last_undegraded":"2023-03-03T08:43:43.988844+0000","last_fullsized":"2023-03-03T08:43:43.988844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:29:02.601410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990482+0000","last_change":"2023-03-03T08:43:29.354519+0000","last_active":"2023-03-03T08:43:43.990482+0000","last_peered":"2023-03-03T08:43:43.990482+0000","last_clean":"2023-03-03T08:43:43.990482+0000","last_became_active":"2023-03-03T08:43:29.354213+0000","last_became_peered":"2023-03-03T08:43:29.354213+0000","last_unstale":"2023-03-03T08:43:43.990482+0000","last_undegraded":"2023-03-03T08:43:43.990482+0000","last_fullsized":"2023-03-03T08:43:43.990482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:00.406507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990028+0000","last_change":"2023-03-03T08:43:34.110105+0000","last_active":"2023-03-03T08:43:43.990028+0000","last_peered":"2023-03-03T08:43:43.990028+0000","last_clean":"2023-03-03T08:43:43.990028+0000","last_became_active":"2023-03-03T08:43:34.109736+0000","last_became_peered":"2023-03-03T08:43:34.109736+0000","last_unstale":"2023-03-03T08:43:43.990028+0000","last_undegraded":"2023-03-03T08:43:43.990028+0000","last_fullsized":"2023-03-03T08:43:43.990028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:30.557847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105361+0000","last_change":"2023-03-03T08:43:43.569433+0000","last_active":"2023-03-03T08:43:45.105361+0000","last_peered":"2023-03-03T08:43:45.105361+0000","last_clean":"2023-03-03T08:43:45.105361+0000","last_became_active":"2023-03-03T08:43:43.568711+0000","last_became_peered":"2023-03-03T08:43:43.568711+0000","last_unstale":"2023-03-03T08:43:45.105361+0000","last_undegraded":"2023-03-03T08:43:45.105361+0000","last_fullsized":"2023-03-03T08:43:45.105361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:53.170557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105548+0000","last_change":"2023-03-03T08:43:43.999241+0000","last_active":"2023-03-03T08:43:45.105548+0000","last_peered":"2023-03-03T08:43:45.105548+0000","last_clean":"2023-03-03T08:43:45.105548+0000","last_became_active":"2023-03-03T08:43:43.998992+0000","last_became_peered":"2023-03-03T08:43:43.998992+0000","last_unstale":"2023-03-03T08:43:45.105548+0000","last_undegraded":"2023-03-03T08:43:45.105548+0000","last_fullsized":"2023-03-03T08:43:45.105548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:22.623700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991092+0000","last_change":"2023-03-03T08:43:34.535435+0000","last_active":"2023-03-03T08:43:43.991092+0000","last_peered":"2023-03-03T08:43:43.991092+0000","last_clean":"2023-03-03T08:43:43.991092+0000","last_became_active":"2023-03-03T08:43:34.535123+0000","last_became_peered":"2023-03-03T08:43:34.535123+0000","last_unstale":"2023-03-03T08:43:43.991092+0000","last_undegraded":"2023-03-03T08:43:43.991092+0000","last_fullsized":"2023-03-03T08:43:43.991092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:52:10.622422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990432+0000","last_change":"2023-03-03T08:43:34.107783+0000","last_active":"2023-03-03T08:43:43.990432+0000","last_peered":"2023-03-03T08:43:43.990432+0000","last_clean":"2023-03-03T08:43:43.990432+0000","last_became_active":"2023-03-03T08:43:34.107503+0000","last_became_peered":"2023-03-03T08:43:34.107503+0000","last_unstale":"2023-03-03T08:43:43.990432+0000","last_undegraded":"2023-03-03T08:43:43.990432+0000","last_fullsized":"2023-03-03T08:43:43.990432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:35:42.800814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981038+0000","last_change":"2023-03-03T08:43:29.072304+0000","last_active":"2023-03-03T08:43:43.981038+0000","last_peered":"2023-03-03T08:43:43.981038+0000","last_clean":"2023-03-03T08:43:43.981038+0000","last_became_active":"2023-03-03T08:43:29.071962+0000","last_became_peered":"2023-03-03T08:43:29.071962+0000","last_unstale":"2023-03-03T08:43:43.981038+0000","last_undegraded":"2023-03-03T08:43:43.981038+0000","last_fullsized":"2023-03-03T08:43:43.981038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:40:12.229226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106349+0000","last_change":"2023-03-03T08:43:43.575964+0000","last_active":"2023-03-03T08:43:45.106349+0000","last_peered":"2023-03-03T08:43:45.106349+0000","last_clean":"2023-03-03T08:43:45.106349+0000","last_became_active":"2023-03-03T08:43:43.575697+0000","last_became_peered":"2023-03-03T08:43:43.575697+0000","last_unstale":"2023-03-03T08:43:45.106349+0000","last_undegraded":"2023-03-03T08:43:45.106349+0000","last_fullsized":"2023-03-03T08:43:45.106349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:01.896050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978782+0000","last_change":"2023-03-03T08:43:44.006414+0000","last_active":"2023-03-03T08:43:44.978782+0000","last_peered":"2023-03-03T08:43:44.978782+0000","last_clean":"2023-03-03T08:43:44.978782+0000","last_became_active":"2023-03-03T08:43:44.006132+0000","last_became_peered":"2023-03-03T08:43:44.006132+0000","last_unstale":"2023-03-03T08:43:44.978782+0000","last_undegraded":"2023-03-03T08:43:44.978782+0000","last_fullsized":"2023-03-03T08:43:44.978782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:55.006419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982772+0000","last_change":"2023-03-03T08:43:34.544053+0000","last_active":"2023-03-03T08:43:43.982772+0000","last_peered":"2023-03-03T08:43:43.982772+0000","last_clean":"2023-03-03T08:43:43.982772+0000","last_became_active":"2023-03-03T08:43:34.543523+0000","last_became_peered":"2023-03-03T08:43:34.543523+0000","last_unstale":"2023-03-03T08:43:43.982772+0000","last_undegraded":"2023-03-03T08:43:43.982772+0000","last_fullsized":"2023-03-03T08:43:43.982772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:34:03.662535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989069+0000","last_change":"2023-03-03T08:43:34.104492+0000","last_active":"2023-03-03T08:43:43.989069+0000","last_peered":"2023-03-03T08:43:43.989069+0000","last_clean":"2023-03-03T08:43:43.989069+0000","last_became_active":"2023-03-03T08:43:34.104151+0000","last_became_peered":"2023-03-03T08:43:34.104151+0000","last_unstale":"2023-03-03T08:43:43.989069+0000","last_undegraded":"2023-03-03T08:43:43.989069+0000","last_fullsized":"2023-03-03T08:43:43.989069+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:57.214738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979395+0000","last_change":"2023-03-03T08:43:29.075836+0000","last_active":"2023-03-03T08:43:43.979395+0000","last_peered":"2023-03-03T08:43:43.979395+0000","last_clean":"2023-03-03T08:43:43.979395+0000","last_became_active":"2023-03-03T08:43:29.075564+0000","last_became_peered":"2023-03-03T08:43:29.075564+0000","last_unstale":"2023-03-03T08:43:43.979395+0000","last_undegraded":"2023-03-03T08:43:43.979395+0000","last_fullsized":"2023-03-03T08:43:43.979395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:39.576714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107224+0000","last_change":"2023-03-03T08:43:43.575599+0000","last_active":"2023-03-03T08:43:45.107224+0000","last_peered":"2023-03-03T08:43:45.107224+0000","last_clean":"2023-03-03T08:43:45.107224+0000","last_became_active":"2023-03-03T08:43:43.575318+0000","last_became_peered":"2023-03-03T08:43:43.575318+0000","last_unstale":"2023-03-03T08:43:45.107224+0000","last_undegraded":"2023-03-03T08:43:45.107224+0000","last_fullsized":"2023-03-03T08:43:45.107224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:28.806052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105269+0000","last_change":"2023-03-03T08:43:44.000500+0000","last_active":"2023-03-03T08:43:45.105269+0000","last_peered":"2023-03-03T08:43:45.105269+0000","last_clean":"2023-03-03T08:43:45.105269+0000","last_became_active":"2023-03-03T08:43:43.999560+0000","last_became_peered":"2023-03-03T08:43:43.999560+0000","last_unstale":"2023-03-03T08:43:45.105269+0000","last_undegraded":"2023-03-03T08:43:45.105269+0000","last_fullsized":"2023-03-03T08:43:45.105269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:29:54.240309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981172+0000","last_change":"2023-03-03T08:43:34.550263+0000","last_active":"2023-03-03T08:43:43.981172+0000","last_peered":"2023-03-03T08:43:43.981172+0000","last_clean":"2023-03-03T08:43:43.981172+0000","last_became_active":"2023-03-03T08:43:34.549560+0000","last_became_peered":"2023-03-03T08:43:34.549560+0000","last_unstale":"2023-03-03T08:43:43.981172+0000","last_undegraded":"2023-03-03T08:43:43.981172+0000","last_fullsized":"2023-03-03T08:43:43.981172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:55.874223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987062+0000","last_change":"2023-03-03T08:43:34.110484+0000","last_active":"2023-03-03T08:43:43.987062+0000","last_peered":"2023-03-03T08:43:43.987062+0000","last_clean":"2023-03-03T08:43:43.987062+0000","last_became_active":"2023-03-03T08:43:34.110220+0000","last_became_peered":"2023-03-03T08:43:34.110220+0000","last_unstale":"2023-03-03T08:43:43.987062+0000","last_undegraded":"2023-03-03T08:43:43.987062+0000","last_fullsized":"2023-03-03T08:43:43.987062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:04:59.232537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982439+0000","last_change":"2023-03-03T08:43:23.015460+0000","last_active":"2023-03-03T08:43:43.982439+0000","last_peered":"2023-03-03T08:43:43.982439+0000","last_clean":"2023-03-03T08:43:43.982439+0000","last_became_active":"2023-03-03T08:43:23.014485+0000","last_became_peered":"2023-03-03T08:43:23.014485+0000","last_unstale":"2023-03-03T08:43:43.982439+0000","last_undegraded":"2023-03-03T08:43:43.982439+0000","last_fullsized":"2023-03-03T08:43:43.982439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_clean_scrub_stamp":"2023-03-03T08:43:21.542883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:07.022017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980803+0000","last_change":"2023-03-03T08:43:29.160285+0000","last_active":"2023-03-03T08:43:43.980803+0000","last_peered":"2023-03-03T08:43:43.980803+0000","last_clean":"2023-03-03T08:43:43.980803+0000","last_became_active":"2023-03-03T08:43:29.159943+0000","last_became_peered":"2023-03-03T08:43:29.159943+0000","last_unstale":"2023-03-03T08:43:43.980803+0000","last_undegraded":"2023-03-03T08:43:43.980803+0000","last_fullsized":"2023-03-03T08:43:43.980803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:48:58.997356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107580+0000","last_change":"2023-03-03T08:43:43.575212+0000","last_active":"2023-03-03T08:43:45.107580+0000","last_peered":"2023-03-03T08:43:45.107580+0000","last_clean":"2023-03-03T08:43:45.107580+0000","last_became_active":"2023-03-03T08:43:43.574897+0000","last_became_peered":"2023-03-03T08:43:43.574897+0000","last_unstale":"2023-03-03T08:43:45.107580+0000","last_undegraded":"2023-03-03T08:43:45.107580+0000","last_fullsized":"2023-03-03T08:43:45.107580+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:33:07.674313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107370+0000","last_change":"2023-03-03T08:43:43.992952+0000","last_active":"2023-03-03T08:43:45.107370+0000","last_peered":"2023-03-03T08:43:45.107370+0000","last_clean":"2023-03-03T08:43:45.107370+0000","last_became_active":"2023-03-03T08:43:43.992535+0000","last_became_peered":"2023-03-03T08:43:43.992535+0000","last_unstale":"2023-03-03T08:43:45.107370+0000","last_undegraded":"2023-03-03T08:43:45.107370+0000","last_fullsized":"2023-03-03T08:43:45.107370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:10:41.617430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991335+0000","last_change":"2023-03-03T08:43:34.536419+0000","last_active":"2023-03-03T08:43:43.991335+0000","last_peered":"2023-03-03T08:43:43.991335+0000","last_clean":"2023-03-03T08:43:43.991335+0000","last_became_active":"2023-03-03T08:43:34.536150+0000","last_became_peered":"2023-03-03T08:43:34.536150+0000","last_unstale":"2023-03-03T08:43:43.991335+0000","last_undegraded":"2023-03-03T08:43:43.991335+0000","last_fullsized":"2023-03-03T08:43:43.991335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:22:54.362024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981827+0000","last_change":"2023-03-03T08:43:34.106337+0000","last_active":"2023-03-03T08:43:43.981827+0000","last_peered":"2023-03-03T08:43:43.981827+0000","last_clean":"2023-03-03T08:43:43.981827+0000","last_became_active":"2023-03-03T08:43:34.105753+0000","last_became_peered":"2023-03-03T08:43:34.105753+0000","last_unstale":"2023-03-03T08:43:43.981827+0000","last_undegraded":"2023-03-03T08:43:43.981827+0000","last_fullsized":"2023-03-03T08:43:43.981827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:46.761924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991590+0000","last_change":"2023-03-03T08:43:29.076367+0000","last_active":"2023-03-03T08:43:43.991590+0000","last_peered":"2023-03-03T08:43:43.991590+0000","last_clean":"2023-03-03T08:43:43.991590+0000","last_became_active":"2023-03-03T08:43:29.075982+0000","last_became_peered":"2023-03-03T08:43:29.075982+0000","last_unstale":"2023-03-03T08:43:43.991590+0000","last_undegraded":"2023-03-03T08:43:43.991590+0000","last_fullsized":"2023-03-03T08:43:43.991590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:06.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:46.143717+0000","last_change":"2023-03-03T08:43:43.577670+0000","last_active":"2023-03-03T08:43:46.143717+0000","last_peered":"2023-03-03T08:43:46.143717+0000","last_clean":"2023-03-03T08:43:46.143717+0000","last_became_active":"2023-03-03T08:43:43.577371+0000","last_became_peered":"2023-03-03T08:43:43.577371+0000","last_unstale":"2023-03-03T08:43:46.143717+0000","last_undegraded":"2023-03-03T08:43:46.143717+0000","last_fullsized":"2023-03-03T08:43:46.143717+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:18.377654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108041+0000","last_change":"2023-03-03T08:43:43.993077+0000","last_active":"2023-03-03T08:43:45.108041+0000","last_peered":"2023-03-03T08:43:45.108041+0000","last_clean":"2023-03-03T08:43:45.108041+0000","last_became_active":"2023-03-03T08:43:43.992021+0000","last_became_peered":"2023-03-03T08:43:43.992021+0000","last_unstale":"2023-03-03T08:43:45.108041+0000","last_undegraded":"2023-03-03T08:43:45.108041+0000","last_fullsized":"2023-03-03T08:43:45.108041+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:58.237553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990193+0000","last_change":"2023-03-03T08:43:34.528593+0000","last_active":"2023-03-03T08:43:43.990193+0000","last_peered":"2023-03-03T08:43:43.990193+0000","last_clean":"2023-03-03T08:43:43.990193+0000","last_became_active":"2023-03-03T08:43:34.528260+0000","last_became_peered":"2023-03-03T08:43:34.528260+0000","last_unstale":"2023-03-03T08:43:43.990193+0000","last_undegraded":"2023-03-03T08:43:43.990193+0000","last_fullsized":"2023-03-03T08:43:43.990193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988924+0000","last_change":"2023-03-03T08:43:34.113748+0000","last_active":"2023-03-03T08:43:43.988924+0000","last_peered":"2023-03-03T08:43:43.988924+0000","last_clean":"2023-03-03T08:43:43.988924+0000","last_became_active":"2023-03-03T08:43:34.113190+0000","last_became_peered":"2023-03-03T08:43:34.113190+0000","last_unstale":"2023-03-03T08:43:43.988924+0000","last_undegraded":"2023-03-03T08:43:43.988924+0000","last_fullsized":"2023-03-03T08:43:43.988924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:58:55.343571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980485+0000","last_change":"2023-03-03T08:43:29.107584+0000","last_active":"2023-03-03T08:43:43.980485+0000","last_peered":"2023-03-03T08:43:43.980485+0000","last_clean":"2023-03-03T08:43:43.980485+0000","last_became_active":"2023-03-03T08:43:29.107208+0000","last_became_peered":"2023-03-03T08:43:29.107208+0000","last_unstale":"2023-03-03T08:43:43.980485+0000","last_undegraded":"2023-03-03T08:43:43.980485+0000","last_fullsized":"2023-03-03T08:43:43.980485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:28:24.087122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106004+0000","last_change":"2023-03-03T08:43:43.575742+0000","last_active":"2023-03-03T08:43:45.106004+0000","last_peered":"2023-03-03T08:43:45.106004+0000","last_clean":"2023-03-03T08:43:45.106004+0000","last_became_active":"2023-03-03T08:43:43.575456+0000","last_became_peered":"2023-03-03T08:43:43.575456+0000","last_unstale":"2023-03-03T08:43:45.106004+0000","last_undegraded":"2023-03-03T08:43:45.106004+0000","last_fullsized":"2023-03-03T08:43:45.106004+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:46:17.635306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106635+0000","last_change":"2023-03-03T08:43:43.985961+0000","last_active":"2023-03-03T08:43:45.106635+0000","last_peered":"2023-03-03T08:43:45.106635+0000","last_clean":"2023-03-03T08:43:45.106635+0000","last_became_active":"2023-03-03T08:43:43.985633+0000","last_became_peered":"2023-03-03T08:43:43.985633+0000","last_unstale":"2023-03-03T08:43:45.106635+0000","last_undegraded":"2023-03-03T08:43:45.106635+0000","last_fullsized":"2023-03-03T08:43:45.106635+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:52.517910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989789+0000","last_change":"2023-03-03T08:43:34.526797+0000","last_active":"2023-03-03T08:43:43.989789+0000","last_peered":"2023-03-03T08:43:43.989789+0000","last_clean":"2023-03-03T08:43:43.989789+0000","last_became_active":"2023-03-03T08:43:34.526490+0000","last_became_peered":"2023-03-03T08:43:34.526490+0000","last_unstale":"2023-03-03T08:43:43.989789+0000","last_undegraded":"2023-03-03T08:43:43.989789+0000","last_fullsized":"2023-03-03T08:43:43.989789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:11.808336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986004+0000","last_change":"2023-03-03T08:43:34.104692+0000","last_active":"2023-03-03T08:43:43.986004+0000","last_peered":"2023-03-03T08:43:43.986004+0000","last_clean":"2023-03-03T08:43:43.986004+0000","last_became_active":"2023-03-03T08:43:34.104263+0000","last_became_peered":"2023-03-03T08:43:34.104263+0000","last_unstale":"2023-03-03T08:43:43.986004+0000","last_undegraded":"2023-03-03T08:43:43.986004+0000","last_fullsized":"2023-03-03T08:43:43.986004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:24.537703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983049+0000","last_change":"2023-03-03T08:43:29.107536+0000","last_active":"2023-03-03T08:43:43.983049+0000","last_peered":"2023-03-03T08:43:43.983049+0000","last_clean":"2023-03-03T08:43:43.983049+0000","last_became_active":"2023-03-03T08:43:29.107093+0000","last_became_peered":"2023-03-03T08:43:29.107093+0000","last_unstale":"2023-03-03T08:43:43.983049+0000","last_undegraded":"2023-03-03T08:43:43.983049+0000","last_fullsized":"2023-03-03T08:43:43.983049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:25:47.568904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980025+0000","last_change":"2023-03-03T08:43:43.577221+0000","last_active":"2023-03-03T08:43:44.980025+0000","last_peered":"2023-03-03T08:43:44.980025+0000","last_clean":"2023-03-03T08:43:44.980025+0000","last_became_active":"2023-03-03T08:43:43.576942+0000","last_became_peered":"2023-03-03T08:43:43.576942+0000","last_unstale":"2023-03-03T08:43:44.980025+0000","last_undegraded":"2023-03-03T08:43:44.980025+0000","last_fullsized":"2023-03-03T08:43:44.980025+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:13:09.716078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105102+0000","last_change":"2023-03-03T08:43:44.000318+0000","last_active":"2023-03-03T08:43:45.105102+0000","last_peered":"2023-03-03T08:43:45.105102+0000","last_clean":"2023-03-03T08:43:45.105102+0000","last_became_active":"2023-03-03T08:43:44.000053+0000","last_became_peered":"2023-03-03T08:43:44.000053+0000","last_unstale":"2023-03-03T08:43:45.105102+0000","last_undegraded":"2023-03-03T08:43:45.105102+0000","last_fullsized":"2023-03-03T08:43:45.105102+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:31.059774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994113+0000","last_change":"2023-03-03T08:43:34.540006+0000","last_active":"2023-03-03T08:43:43.994113+0000","last_peered":"2023-03-03T08:43:43.994113+0000","last_clean":"2023-03-03T08:43:43.994113+0000","last_became_active":"2023-03-03T08:43:34.539579+0000","last_became_peered":"2023-03-03T08:43:34.539579+0000","last_unstale":"2023-03-03T08:43:43.994113+0000","last_undegraded":"2023-03-03T08:43:43.994113+0000","last_fullsized":"2023-03-03T08:43:43.994113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988985+0000","last_change":"2023-03-03T08:43:34.109764+0000","last_active":"2023-03-03T08:43:43.988985+0000","last_peered":"2023-03-03T08:43:43.988985+0000","last_clean":"2023-03-03T08:43:43.988985+0000","last_became_active":"2023-03-03T08:43:34.109462+0000","last_became_peered":"2023-03-03T08:43:34.109462+0000","last_unstale":"2023-03-03T08:43:43.988985+0000","last_undegraded":"2023-03-03T08:43:43.988985+0000","last_fullsized":"2023-03-03T08:43:43.988985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:56.399192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106570+0000","last_change":"2023-03-03T08:43:43.576402+0000","last_active":"2023-03-03T08:43:45.106570+0000","last_peered":"2023-03-03T08:43:45.106570+0000","last_clean":"2023-03-03T08:43:45.106570+0000","last_became_active":"2023-03-03T08:43:43.576113+0000","last_became_peered":"2023-03-03T08:43:43.576113+0000","last_unstale":"2023-03-03T08:43:45.106570+0000","last_undegraded":"2023-03-03T08:43:45.106570+0000","last_fullsized":"2023-03-03T08:43:45.106570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:33:26.483688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106437+0000","last_change":"2023-03-03T08:43:44.002074+0000","last_active":"2023-03-03T08:43:45.106437+0000","last_peered":"2023-03-03T08:43:45.106437+0000","last_clean":"2023-03-03T08:43:45.106437+0000","last_became_active":"2023-03-03T08:43:44.001803+0000","last_became_peered":"2023-03-03T08:43:44.001803+0000","last_unstale":"2023-03-03T08:43:45.106437+0000","last_undegraded":"2023-03-03T08:43:45.106437+0000","last_fullsized":"2023-03-03T08:43:45.106437+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:15:16.115135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986358+0000","last_change":"2023-03-03T08:43:34.541858+0000","last_active":"2023-03-03T08:43:43.986358+0000","last_peered":"2023-03-03T08:43:43.986358+0000","last_clean":"2023-03-03T08:43:43.986358+0000","last_became_active":"2023-03-03T08:43:34.541430+0000","last_became_peered":"2023-03-03T08:43:34.541430+0000","last_unstale":"2023-03-03T08:43:43.986358+0000","last_undegraded":"2023-03-03T08:43:43.986358+0000","last_fullsized":"2023-03-03T08:43:43.986358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:29.397784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979830+0000","last_change":"2023-03-03T08:43:34.111335+0000","last_active":"2023-03-03T08:43:43.979830+0000","last_peered":"2023-03-03T08:43:43.979830+0000","last_clean":"2023-03-03T08:43:43.979830+0000","last_became_active":"2023-03-03T08:43:34.110837+0000","last_became_peered":"2023-03-03T08:43:34.110837+0000","last_unstale":"2023-03-03T08:43:43.979830+0000","last_undegraded":"2023-03-03T08:43:43.979830+0000","last_fullsized":"2023-03-03T08:43:43.979830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:03.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106574+0000","last_change":"2023-03-03T08:43:43.577539+0000","last_active":"2023-03-03T08:43:45.106574+0000","last_peered":"2023-03-03T08:43:45.106574+0000","last_clean":"2023-03-03T08:43:45.106574+0000","last_became_active":"2023-03-03T08:43:43.577209+0000","last_became_peered":"2023-03-03T08:43:43.577209+0000","last_unstale":"2023-03-03T08:43:45.106574+0000","last_undegraded":"2023-03-03T08:43:45.106574+0000","last_fullsized":"2023-03-03T08:43:45.106574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:31:01.430495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979296+0000","last_change":"2023-03-03T08:43:43.993565+0000","last_active":"2023-03-03T08:43:44.979296+0000","last_peered":"2023-03-03T08:43:44.979296+0000","last_clean":"2023-03-03T08:43:44.979296+0000","last_became_active":"2023-03-03T08:43:43.992653+0000","last_became_peered":"2023-03-03T08:43:43.992653+0000","last_unstale":"2023-03-03T08:43:44.979296+0000","last_undegraded":"2023-03-03T08:43:44.979296+0000","last_fullsized":"2023-03-03T08:43:44.979296+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:24.295462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981121+0000","last_change":"2023-03-03T08:43:34.550325+0000","last_active":"2023-03-03T08:43:43.981121+0000","last_peered":"2023-03-03T08:43:43.981121+0000","last_clean":"2023-03-03T08:43:43.981121+0000","last_became_active":"2023-03-03T08:43:34.550041+0000","last_became_peered":"2023-03-03T08:43:34.550041+0000","last_unstale":"2023-03-03T08:43:43.981121+0000","last_undegraded":"2023-03-03T08:43:43.981121+0000","last_fullsized":"2023-03-03T08:43:43.981121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:44:34.082003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980457+0000","last_change":"2023-03-03T08:43:34.106777+0000","last_active":"2023-03-03T08:43:43.980457+0000","last_peered":"2023-03-03T08:43:43.980457+0000","last_clean":"2023-03-03T08:43:43.980457+0000","last_became_active":"2023-03-03T08:43:34.106132+0000","last_became_peered":"2023-03-03T08:43:34.106132+0000","last_unstale":"2023-03-03T08:43:43.980457+0000","last_undegraded":"2023-03-03T08:43:43.980457+0000","last_fullsized":"2023-03-03T08:43:43.980457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:55.965987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106224+0000","last_change":"2023-03-03T08:43:43.580312+0000","last_active":"2023-03-03T08:43:45.106224+0000","last_peered":"2023-03-03T08:43:45.106224+0000","last_clean":"2023-03-03T08:43:45.106224+0000","last_became_active":"2023-03-03T08:43:43.580096+0000","last_became_peered":"2023-03-03T08:43:43.580096+0000","last_unstale":"2023-03-03T08:43:45.106224+0000","last_undegraded":"2023-03-03T08:43:45.106224+0000","last_fullsized":"2023-03-03T08:43:45.106224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:06:03.500778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106765+0000","last_change":"2023-03-03T08:43:43.997453+0000","last_active":"2023-03-03T08:43:45.106765+0000","last_peered":"2023-03-03T08:43:45.106765+0000","last_clean":"2023-03-03T08:43:45.106765+0000","last_became_active":"2023-03-03T08:43:43.997090+0000","last_became_peered":"2023-03-03T08:43:43.997090+0000","last_unstale":"2023-03-03T08:43:45.106765+0000","last_undegraded":"2023-03-03T08:43:45.106765+0000","last_fullsized":"2023-03-03T08:43:45.106765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:28:58.322823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985493+0000","last_change":"2023-03-03T08:43:34.542308+0000","last_active":"2023-03-03T08:43:43.985493+0000","last_peered":"2023-03-03T08:43:43.985493+0000","last_clean":"2023-03-03T08:43:43.985493+0000","last_became_active":"2023-03-03T08:43:34.541991+0000","last_became_peered":"2023-03-03T08:43:34.541991+0000","last_unstale":"2023-03-03T08:43:43.985493+0000","last_undegraded":"2023-03-03T08:43:43.985493+0000","last_fullsized":"2023-03-03T08:43:43.985493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:06:21.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992335+0000","last_change":"2023-03-03T08:43:34.109698+0000","last_active":"2023-03-03T08:43:43.992335+0000","last_peered":"2023-03-03T08:43:43.992335+0000","last_clean":"2023-03-03T08:43:43.992335+0000","last_became_active":"2023-03-03T08:43:34.109423+0000","last_became_peered":"2023-03-03T08:43:34.109423+0000","last_unstale":"2023-03-03T08:43:43.992335+0000","last_undegraded":"2023-03-03T08:43:43.992335+0000","last_fullsized":"2023-03-03T08:43:43.992335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:49.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105911+0000","last_change":"2023-03-03T08:43:43.573925+0000","last_active":"2023-03-03T08:43:45.105911+0000","last_peered":"2023-03-03T08:43:45.105911+0000","last_clean":"2023-03-03T08:43:45.105911+0000","last_became_active":"2023-03-03T08:43:43.573474+0000","last_became_peered":"2023-03-03T08:43:43.573474+0000","last_unstale":"2023-03-03T08:43:45.105911+0000","last_undegraded":"2023-03-03T08:43:45.105911+0000","last_fullsized":"2023-03-03T08:43:45.105911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:47.858726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107653+0000","last_change":"2023-03-03T08:43:43.995602+0000","last_active":"2023-03-03T08:43:45.107653+0000","last_peered":"2023-03-03T08:43:45.107653+0000","last_clean":"2023-03-03T08:43:45.107653+0000","last_became_active":"2023-03-03T08:43:43.995404+0000","last_became_peered":"2023-03-03T08:43:43.995404+0000","last_unstale":"2023-03-03T08:43:45.107653+0000","last_undegraded":"2023-03-03T08:43:45.107653+0000","last_fullsized":"2023-03-03T08:43:45.107653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:37.212119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982423+0000","last_change":"2023-03-03T08:43:34.548304+0000","last_active":"2023-03-03T08:43:43.982423+0000","last_peered":"2023-03-03T08:43:43.982423+0000","last_clean":"2023-03-03T08:43:43.982423+0000","last_became_active":"2023-03-03T08:43:34.547960+0000","last_became_peered":"2023-03-03T08:43:34.547960+0000","last_unstale":"2023-03-03T08:43:43.982423+0000","last_undegraded":"2023-03-03T08:43:43.982423+0000","last_fullsized":"2023-03-03T08:43:43.982423+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:04.907753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985574+0000","last_change":"2023-03-03T08:43:34.109372+0000","last_active":"2023-03-03T08:43:43.985574+0000","last_peered":"2023-03-03T08:43:43.985574+0000","last_clean":"2023-03-03T08:43:43.985574+0000","last_became_active":"2023-03-03T08:43:34.109023+0000","last_became_peered":"2023-03-03T08:43:34.109023+0000","last_unstale":"2023-03-03T08:43:43.985574+0000","last_undegraded":"2023-03-03T08:43:43.985574+0000","last_fullsized":"2023-03-03T08:43:43.985574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:51.374188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978866+0000","last_change":"2023-03-03T08:43:43.572240+0000","last_active":"2023-03-03T08:43:44.978866+0000","last_peered":"2023-03-03T08:43:44.978866+0000","last_clean":"2023-03-03T08:43:44.978866+0000","last_became_active":"2023-03-03T08:43:43.571753+0000","last_became_peered":"2023-03-03T08:43:43.571753+0000","last_unstale":"2023-03-03T08:43:44.978866+0000","last_undegraded":"2023-03-03T08:43:44.978866+0000","last_fullsized":"2023-03-03T08:43:44.978866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:02:30.521667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979027+0000","last_change":"2023-03-03T08:43:43.995018+0000","last_active":"2023-03-03T08:43:44.979027+0000","last_peered":"2023-03-03T08:43:44.979027+0000","last_clean":"2023-03-03T08:43:44.979027+0000","last_became_active":"2023-03-03T08:43:43.994690+0000","last_became_peered":"2023-03-03T08:43:43.994690+0000","last_unstale":"2023-03-03T08:43:44.979027+0000","last_undegraded":"2023-03-03T08:43:44.979027+0000","last_fullsized":"2023-03-03T08:43:44.979027+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:06:13.979929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986022+0000","last_change":"2023-03-03T08:43:34.538817+0000","last_active":"2023-03-03T08:43:43.986022+0000","last_peered":"2023-03-03T08:43:43.986022+0000","last_clean":"2023-03-03T08:43:43.986022+0000","last_became_active":"2023-03-03T08:43:34.538506+0000","last_became_peered":"2023-03-03T08:43:34.538506+0000","last_unstale":"2023-03-03T08:43:43.986022+0000","last_undegraded":"2023-03-03T08:43:43.986022+0000","last_fullsized":"2023-03-03T08:43:43.986022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980930+0000","last_change":"2023-03-03T08:43:34.117734+0000","last_active":"2023-03-03T08:43:43.980930+0000","last_peered":"2023-03-03T08:43:43.980930+0000","last_clean":"2023-03-03T08:43:43.980930+0000","last_became_active":"2023-03-03T08:43:34.117454+0000","last_became_peered":"2023-03-03T08:43:34.117454+0000","last_unstale":"2023-03-03T08:43:43.980930+0000","last_undegraded":"2023-03-03T08:43:43.980930+0000","last_fullsized":"2023-03-03T08:43:43.980930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:42:27.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106434+0000","last_change":"2023-03-03T08:43:43.574955+0000","last_active":"2023-03-03T08:43:45.106434+0000","last_peered":"2023-03-03T08:43:45.106434+0000","last_clean":"2023-03-03T08:43:45.106434+0000","last_became_active":"2023-03-03T08:43:43.574677+0000","last_became_peered":"2023-03-03T08:43:43.574677+0000","last_unstale":"2023-03-03T08:43:45.106434+0000","last_undegraded":"2023-03-03T08:43:45.106434+0000","last_fullsized":"2023-03-03T08:43:45.106434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:12:19.538291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.104954+0000","last_change":"2023-03-03T08:43:43.994888+0000","last_active":"2023-03-03T08:43:45.104954+0000","last_peered":"2023-03-03T08:43:45.104954+0000","last_clean":"2023-03-03T08:43:45.104954+0000","last_became_active":"2023-03-03T08:43:43.994630+0000","last_became_peered":"2023-03-03T08:43:43.994630+0000","last_unstale":"2023-03-03T08:43:45.104954+0000","last_undegraded":"2023-03-03T08:43:45.104954+0000","last_fullsized":"2023-03-03T08:43:45.104954+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:15:18.412569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982579+0000","last_change":"2023-03-03T08:43:34.548010+0000","last_active":"2023-03-03T08:43:43.982579+0000","last_peered":"2023-03-03T08:43:43.982579+0000","last_clean":"2023-03-03T08:43:43.982579+0000","last_became_active":"2023-03-03T08:43:34.547662+0000","last_became_peered":"2023-03-03T08:43:34.547662+0000","last_unstale":"2023-03-03T08:43:43.982579+0000","last_undegraded":"2023-03-03T08:43:43.982579+0000","last_fullsized":"2023-03-03T08:43:43.982579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:52.200064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986286+0000","last_change":"2023-03-03T08:43:34.109890+0000","last_active":"2023-03-03T08:43:43.986286+0000","last_peered":"2023-03-03T08:43:43.986286+0000","last_clean":"2023-03-03T08:43:43.986286+0000","last_became_active":"2023-03-03T08:43:34.109583+0000","last_became_peered":"2023-03-03T08:43:34.109583+0000","last_unstale":"2023-03-03T08:43:43.986286+0000","last_undegraded":"2023-03-03T08:43:43.986286+0000","last_fullsized":"2023-03-03T08:43:43.986286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:27:34.142294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978233+0000","last_change":"2023-03-03T08:43:43.575945+0000","last_active":"2023-03-03T08:43:44.978233+0000","last_peered":"2023-03-03T08:43:44.978233+0000","last_clean":"2023-03-03T08:43:44.978233+0000","last_became_active":"2023-03-03T08:43:43.575675+0000","last_became_peered":"2023-03-03T08:43:43.575675+0000","last_unstale":"2023-03-03T08:43:44.978233+0000","last_undegraded":"2023-03-03T08:43:44.978233+0000","last_fullsized":"2023-03-03T08:43:44.978233+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:48.730805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980103+0000","last_change":"2023-03-03T08:43:43.998435+0000","last_active":"2023-03-03T08:43:44.980103+0000","last_peered":"2023-03-03T08:43:44.980103+0000","last_clean":"2023-03-03T08:43:44.980103+0000","last_became_active":"2023-03-03T08:43:43.996508+0000","last_became_peered":"2023-03-03T08:43:43.996508+0000","last_unstale":"2023-03-03T08:43:44.980103+0000","last_undegraded":"2023-03-03T08:43:44.980103+0000","last_fullsized":"2023-03-03T08:43:44.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:31:36.310851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979477+0000","last_change":"2023-03-03T08:43:34.540196+0000","last_active":"2023-03-03T08:43:43.979477+0000","last_peered":"2023-03-03T08:43:43.979477+0000","last_clean":"2023-03-03T08:43:43.979477+0000","last_became_active":"2023-03-03T08:43:34.539931+0000","last_became_peered":"2023-03-03T08:43:34.539931+0000","last_unstale":"2023-03-03T08:43:43.979477+0000","last_undegraded":"2023-03-03T08:43:43.979477+0000","last_fullsized":"2023-03-03T08:43:43.979477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:20.012997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990881+0000","last_change":"2023-03-03T08:43:34.110464+0000","last_active":"2023-03-03T08:43:43.990881+0000","last_peered":"2023-03-03T08:43:43.990881+0000","last_clean":"2023-03-03T08:43:43.990881+0000","last_became_active":"2023-03-03T08:43:34.110165+0000","last_became_peered":"2023-03-03T08:43:34.110165+0000","last_unstale":"2023-03-03T08:43:43.990881+0000","last_undegraded":"2023-03-03T08:43:43.990881+0000","last_fullsized":"2023-03-03T08:43:43.990881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:54.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106139+0000","last_change":"2023-03-03T08:43:43.578298+0000","last_active":"2023-03-03T08:43:45.106139+0000","last_peered":"2023-03-03T08:43:45.106139+0000","last_clean":"2023-03-03T08:43:45.106139+0000","last_became_active":"2023-03-03T08:43:43.578024+0000","last_became_peered":"2023-03-03T08:43:43.578024+0000","last_unstale":"2023-03-03T08:43:45.106139+0000","last_undegraded":"2023-03-03T08:43:45.106139+0000","last_fullsized":"2023-03-03T08:43:45.106139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:00:44.633716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106526+0000","last_change":"2023-03-03T08:43:43.992906+0000","last_active":"2023-03-03T08:43:45.106526+0000","last_peered":"2023-03-03T08:43:45.106526+0000","last_clean":"2023-03-03T08:43:45.106526+0000","last_became_active":"2023-03-03T08:43:43.991823+0000","last_became_peered":"2023-03-03T08:43:43.991823+0000","last_unstale":"2023-03-03T08:43:45.106526+0000","last_undegraded":"2023-03-03T08:43:45.106526+0000","last_fullsized":"2023-03-03T08:43:45.106526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:34.431216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.553597+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.553322+0000","last_became_peered":"2023-03-03T08:43:34.553322+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:24:17.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983377+0000","last_change":"2023-03-03T08:43:34.108123+0000","last_active":"2023-03-03T08:43:43.983377+0000","last_peered":"2023-03-03T08:43:43.983377+0000","last_clean":"2023-03-03T08:43:43.983377+0000","last_became_active":"2023-03-03T08:43:34.107757+0000","last_became_peered":"2023-03-03T08:43:34.107757+0000","last_unstale":"2023-03-03T08:43:43.983377+0000","last_undegraded":"2023-03-03T08:43:43.983377+0000","last_fullsized":"2023-03-03T08:43:43.983377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:09.488930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106648+0000","last_change":"2023-03-03T08:43:43.573369+0000","last_active":"2023-03-03T08:43:45.106648+0000","last_peered":"2023-03-03T08:43:45.106648+0000","last_clean":"2023-03-03T08:43:45.106648+0000","last_became_active":"2023-03-03T08:43:43.573069+0000","last_became_peered":"2023-03-03T08:43:43.573069+0000","last_unstale":"2023-03-03T08:43:45.106648+0000","last_undegraded":"2023-03-03T08:43:45.106648+0000","last_fullsized":"2023-03-03T08:43:45.106648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:09.574926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107410+0000","last_change":"2023-03-03T08:43:43.995725+0000","last_active":"2023-03-03T08:43:45.107410+0000","last_peered":"2023-03-03T08:43:45.107410+0000","last_clean":"2023-03-03T08:43:45.107410+0000","last_became_active":"2023-03-03T08:43:43.993875+0000","last_became_peered":"2023-03-03T08:43:43.993875+0000","last_unstale":"2023-03-03T08:43:45.107410+0000","last_undegraded":"2023-03-03T08:43:45.107410+0000","last_fullsized":"2023-03-03T08:43:45.107410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:38.439539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982673+0000","last_change":"2023-03-03T08:43:34.546067+0000","last_active":"2023-03-03T08:43:43.982673+0000","last_peered":"2023-03-03T08:43:43.982673+0000","last_clean":"2023-03-03T08:43:43.982673+0000","last_became_active":"2023-03-03T08:43:34.543786+0000","last_became_peered":"2023-03-03T08:43:34.543786+0000","last_unstale":"2023-03-03T08:43:43.982673+0000","last_undegraded":"2023-03-03T08:43:43.982673+0000","last_fullsized":"2023-03-03T08:43:43.982673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:32.922812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.978650+0000","last_change":"2023-03-03T08:43:34.115792+0000","last_active":"2023-03-03T08:43:43.978650+0000","last_peered":"2023-03-03T08:43:43.978650+0000","last_clean":"2023-03-03T08:43:43.978650+0000","last_became_active":"2023-03-03T08:43:34.115431+0000","last_became_peered":"2023-03-03T08:43:34.115431+0000","last_unstale":"2023-03-03T08:43:43.978650+0000","last_undegraded":"2023-03-03T08:43:43.978650+0000","last_fullsized":"2023-03-03T08:43:43.978650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:15.484022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980510+0000","last_change":"2023-03-03T08:43:43.579691+0000","last_active":"2023-03-03T08:43:44.980510+0000","last_peered":"2023-03-03T08:43:44.980510+0000","last_clean":"2023-03-03T08:43:44.980510+0000","last_became_active":"2023-03-03T08:43:43.579425+0000","last_became_peered":"2023-03-03T08:43:43.579425+0000","last_unstale":"2023-03-03T08:43:44.980510+0000","last_undegraded":"2023-03-03T08:43:44.980510+0000","last_fullsized":"2023-03-03T08:43:44.980510+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:08.896591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979907+0000","last_change":"2023-03-03T08:43:44.006872+0000","last_active":"2023-03-03T08:43:44.979907+0000","last_peered":"2023-03-03T08:43:44.979907+0000","last_clean":"2023-03-03T08:43:44.979907+0000","last_became_active":"2023-03-03T08:43:44.006283+0000","last_became_peered":"2023-03-03T08:43:44.006283+0000","last_unstale":"2023-03-03T08:43:44.979907+0000","last_undegraded":"2023-03-03T08:43:44.979907+0000","last_fullsized":"2023-03-03T08:43:44.979907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:45.439399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981214+0000","last_change":"2023-03-03T08:43:34.532656+0000","last_active":"2023-03-03T08:43:43.981214+0000","last_peered":"2023-03-03T08:43:43.981214+0000","last_clean":"2023-03-03T08:43:43.981214+0000","last_became_active":"2023-03-03T08:43:34.532297+0000","last_became_peered":"2023-03-03T08:43:34.532297+0000","last_unstale":"2023-03-03T08:43:43.981214+0000","last_undegraded":"2023-03-03T08:43:43.981214+0000","last_fullsized":"2023-03-03T08:43:43.981214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:36.347740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989578+0000","last_change":"2023-03-03T08:43:34.112905+0000","last_active":"2023-03-03T08:43:43.989578+0000","last_peered":"2023-03-03T08:43:43.989578+0000","last_clean":"2023-03-03T08:43:43.989578+0000","last_became_active":"2023-03-03T08:43:34.112601+0000","last_became_peered":"2023-03-03T08:43:34.112601+0000","last_unstale":"2023-03-03T08:43:43.989578+0000","last_undegraded":"2023-03-03T08:43:43.989578+0000","last_fullsized":"2023-03-03T08:43:43.989578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:01.165783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"27'9","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:44:01.020445+0000","last_change":"2023-03-03T08:43:43.577882+0000","last_active":"2023-03-03T08:44:01.020445+0000","last_peered":"2023-03-03T08:44:01.020445+0000","last_clean":"2023-03-03T08:44:01.020445+0000","last_became_active":"2023-03-03T08:43:43.575625+0000","last_became_peered":"2023-03-03T08:43:43.575625+0000","last_unstale":"2023-03-03T08:44:01.020445+0000","last_undegraded":"2023-03-03T08:44:01.020445+0000","last_fullsized":"2023-03-03T08:44:01.020445+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:22:25.919413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2779,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106494+0000","last_change":"2023-03-03T08:43:43.994846+0000","last_active":"2023-03-03T08:43:45.106494+0000","last_peered":"2023-03-03T08:43:45.106494+0000","last_clean":"2023-03-03T08:43:45.106494+0000","last_became_active":"2023-03-03T08:43:43.994652+0000","last_became_peered":"2023-03-03T08:43:43.994652+0000","last_unstale":"2023-03-03T08:43:45.106494+0000","last_undegraded":"2023-03-03T08:43:45.106494+0000","last_fullsized":"2023-03-03T08:43:45.106494+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:54:29.238192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990352+0000","last_change":"2023-03-03T08:43:34.539790+0000","last_active":"2023-03-03T08:43:43.990352+0000","last_peered":"2023-03-03T08:43:43.990352+0000","last_clean":"2023-03-03T08:43:43.990352+0000","last_became_active":"2023-03-03T08:43:34.539443+0000","last_became_peered":"2023-03-03T08:43:34.539443+0000","last_unstale":"2023-03-03T08:43:43.990352+0000","last_undegraded":"2023-03-03T08:43:43.990352+0000","last_fullsized":"2023-03-03T08:43:43.990352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:25.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980897+0000","last_change":"2023-03-03T08:43:34.112941+0000","last_active":"2023-03-03T08:43:43.980897+0000","last_peered":"2023-03-03T08:43:43.980897+0000","last_clean":"2023-03-03T08:43:43.980897+0000","last_became_active":"2023-03-03T08:43:34.112646+0000","last_became_peered":"2023-03-03T08:43:34.112646+0000","last_unstale":"2023-03-03T08:43:43.980897+0000","last_undegraded":"2023-03-03T08:43:43.980897+0000","last_fullsized":"2023-03-03T08:43:43.980897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:32.767710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107854+0000","last_change":"2023-03-03T08:43:43.576421+0000","last_active":"2023-03-03T08:43:45.107854+0000","last_peered":"2023-03-03T08:43:45.107854+0000","last_clean":"2023-03-03T08:43:45.107854+0000","last_became_active":"2023-03-03T08:43:43.576086+0000","last_became_peered":"2023-03-03T08:43:43.576086+0000","last_unstale":"2023-03-03T08:43:45.107854+0000","last_undegraded":"2023-03-03T08:43:45.107854+0000","last_fullsized":"2023-03-03T08:43:45.107854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:57:17.814222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980193+0000","last_change":"2023-03-03T08:43:43.995591+0000","last_active":"2023-03-03T08:43:44.980193+0000","last_peered":"2023-03-03T08:43:44.980193+0000","last_clean":"2023-03-03T08:43:44.980193+0000","last_became_active":"2023-03-03T08:43:43.995338+0000","last_became_peered":"2023-03-03T08:43:43.995338+0000","last_unstale":"2023-03-03T08:43:44.980193+0000","last_undegraded":"2023-03-03T08:43:44.980193+0000","last_fullsized":"2023-03-03T08:43:44.980193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:17:56.617398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985713+0000","last_change":"2023-03-03T08:43:34.537857+0000","last_active":"2023-03-03T08:43:43.985713+0000","last_peered":"2023-03-03T08:43:43.985713+0000","last_clean":"2023-03-03T08:43:43.985713+0000","last_became_active":"2023-03-03T08:43:34.537590+0000","last_became_peered":"2023-03-03T08:43:34.537590+0000","last_unstale":"2023-03-03T08:43:43.985713+0000","last_undegraded":"2023-03-03T08:43:43.985713+0000","last_fullsized":"2023-03-03T08:43:43.985713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:52.612521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"27'9","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:58.500261+0000","last_change":"2023-03-03T08:43:34.117609+0000","last_active":"2023-03-03T08:43:58.500261+0000","last_peered":"2023-03-03T08:43:58.500261+0000","last_clean":"2023-03-03T08:43:58.500261+0000","last_became_active":"2023-03-03T08:43:34.117271+0000","last_became_peered":"2023-03-03T08:43:34.117271+0000","last_unstale":"2023-03-03T08:43:58.500261+0000","last_undegraded":"2023-03-03T08:43:58.500261+0000","last_fullsized":"2023-03-03T08:43:58.500261+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:20.259259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2779,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978772+0000","last_change":"2023-03-03T08:43:43.580489+0000","last_active":"2023-03-03T08:43:44.978772+0000","last_peered":"2023-03-03T08:43:44.978772+0000","last_clean":"2023-03-03T08:43:44.978772+0000","last_became_active":"2023-03-03T08:43:43.580207+0000","last_became_peered":"2023-03-03T08:43:43.580207+0000","last_unstale":"2023-03-03T08:43:44.978772+0000","last_undegraded":"2023-03-03T08:43:44.978772+0000","last_fullsized":"2023-03-03T08:43:44.978772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:58:05.519074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978633+0000","last_change":"2023-03-03T08:43:43.993832+0000","last_active":"2023-03-03T08:43:44.978633+0000","last_peered":"2023-03-03T08:43:44.978633+0000","last_clean":"2023-03-03T08:43:44.978633+0000","last_became_active":"2023-03-03T08:43:43.993524+0000","last_became_peered":"2023-03-03T08:43:43.993524+0000","last_unstale":"2023-03-03T08:43:44.978633+0000","last_undegraded":"2023-03-03T08:43:44.978633+0000","last_fullsized":"2023-03-03T08:43:44.978633+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:23:55.761785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987159+0000","last_change":"2023-03-03T08:43:34.544694+0000","last_active":"2023-03-03T08:43:43.987159+0000","last_peered":"2023-03-03T08:43:43.987159+0000","last_clean":"2023-03-03T08:43:43.987159+0000","last_became_active":"2023-03-03T08:43:34.544399+0000","last_became_peered":"2023-03-03T08:43:34.544399+0000","last_unstale":"2023-03-03T08:43:43.987159+0000","last_undegraded":"2023-03-03T08:43:43.987159+0000","last_fullsized":"2023-03-03T08:43:43.987159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986196+0000","last_change":"2023-03-03T08:43:34.110810+0000","last_active":"2023-03-03T08:43:43.986196+0000","last_peered":"2023-03-03T08:43:43.986196+0000","last_clean":"2023-03-03T08:43:43.986196+0000","last_became_active":"2023-03-03T08:43:34.110483+0000","last_became_peered":"2023-03-03T08:43:34.110483+0000","last_unstale":"2023-03-03T08:43:43.986196+0000","last_undegraded":"2023-03-03T08:43:43.986196+0000","last_fullsized":"2023-03-03T08:43:43.986196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:06.970615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979989+0000","last_change":"2023-03-03T08:43:43.576669+0000","last_active":"2023-03-03T08:43:44.979989+0000","last_peered":"2023-03-03T08:43:44.979989+0000","last_clean":"2023-03-03T08:43:44.979989+0000","last_became_active":"2023-03-03T08:43:43.573885+0000","last_became_peered":"2023-03-03T08:43:43.573885+0000","last_unstale":"2023-03-03T08:43:44.979989+0000","last_undegraded":"2023-03-03T08:43:44.979989+0000","last_fullsized":"2023-03-03T08:43:44.979989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:56:12.054118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979442+0000","last_change":"2023-03-03T08:43:43.996790+0000","last_active":"2023-03-03T08:43:44.979442+0000","last_peered":"2023-03-03T08:43:44.979442+0000","last_clean":"2023-03-03T08:43:44.979442+0000","last_became_active":"2023-03-03T08:43:43.996366+0000","last_became_peered":"2023-03-03T08:43:43.996366+0000","last_unstale":"2023-03-03T08:43:44.979442+0000","last_undegraded":"2023-03-03T08:43:44.979442+0000","last_fullsized":"2023-03-03T08:43:44.979442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:01.662221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988553+0000","last_change":"2023-03-03T08:43:34.538703+0000","last_active":"2023-03-03T08:43:43.988553+0000","last_peered":"2023-03-03T08:43:43.988553+0000","last_clean":"2023-03-03T08:43:43.988553+0000","last_became_active":"2023-03-03T08:43:34.538418+0000","last_became_peered":"2023-03-03T08:43:34.538418+0000","last_unstale":"2023-03-03T08:43:43.988553+0000","last_undegraded":"2023-03-03T08:43:43.988553+0000","last_fullsized":"2023-03-03T08:43:43.988553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:02.795203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986185+0000","last_change":"2023-03-03T08:43:34.113168+0000","last_active":"2023-03-03T08:43:43.986185+0000","last_peered":"2023-03-03T08:43:43.986185+0000","last_clean":"2023-03-03T08:43:43.986185+0000","last_became_active":"2023-03-03T08:43:34.112731+0000","last_became_peered":"2023-03-03T08:43:34.112731+0000","last_unstale":"2023-03-03T08:43:43.986185+0000","last_undegraded":"2023-03-03T08:43:43.986185+0000","last_fullsized":"2023-03-03T08:43:43.986185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:57:59.463191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106795+0000","last_change":"2023-03-03T08:43:43.577965+0000","last_active":"2023-03-03T08:43:45.106795+0000","last_peered":"2023-03-03T08:43:45.106795+0000","last_clean":"2023-03-03T08:43:45.106795+0000","last_became_active":"2023-03-03T08:43:43.577587+0000","last_became_peered":"2023-03-03T08:43:43.577587+0000","last_unstale":"2023-03-03T08:43:45.106795+0000","last_undegraded":"2023-03-03T08:43:45.106795+0000","last_fullsized":"2023-03-03T08:43:45.106795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:24.196773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105366+0000","last_change":"2023-03-03T08:43:43.998210+0000","last_active":"2023-03-03T08:43:45.105366+0000","last_peered":"2023-03-03T08:43:45.105366+0000","last_clean":"2023-03-03T08:43:45.105366+0000","last_became_active":"2023-03-03T08:43:43.997845+0000","last_became_peered":"2023-03-03T08:43:43.997845+0000","last_unstale":"2023-03-03T08:43:45.105366+0000","last_undegraded":"2023-03-03T08:43:45.105366+0000","last_fullsized":"2023-03-03T08:43:45.105366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:57:38.226245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991762+0000","last_change":"2023-03-03T08:43:34.528573+0000","last_active":"2023-03-03T08:43:43.991762+0000","last_peered":"2023-03-03T08:43:43.991762+0000","last_clean":"2023-03-03T08:43:43.991762+0000","last_became_active":"2023-03-03T08:43:34.528278+0000","last_became_peered":"2023-03-03T08:43:34.528278+0000","last_unstale":"2023-03-03T08:43:43.991762+0000","last_undegraded":"2023-03-03T08:43:43.991762+0000","last_fullsized":"2023-03-03T08:43:43.991762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:51.468323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.110802+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.110519+0000","last_became_peered":"2023-03-03T08:43:34.110519+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:02:18.355043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105602+0000","last_change":"2023-03-03T08:43:43.573655+0000","last_active":"2023-03-03T08:43:45.105602+0000","last_peered":"2023-03-03T08:43:45.105602+0000","last_clean":"2023-03-03T08:43:45.105602+0000","last_became_active":"2023-03-03T08:43:43.573162+0000","last_became_peered":"2023-03-03T08:43:43.573162+0000","last_unstale":"2023-03-03T08:43:45.105602+0000","last_undegraded":"2023-03-03T08:43:45.105602+0000","last_fullsized":"2023-03-03T08:43:45.105602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:59:04.568700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106617+0000","last_change":"2023-03-03T08:43:43.993431+0000","last_active":"2023-03-03T08:43:45.106617+0000","last_peered":"2023-03-03T08:43:45.106617+0000","last_clean":"2023-03-03T08:43:45.106617+0000","last_became_active":"2023-03-03T08:43:43.993166+0000","last_became_peered":"2023-03-03T08:43:43.993166+0000","last_unstale":"2023-03-03T08:43:45.106617+0000","last_undegraded":"2023-03-03T08:43:45.106617+0000","last_fullsized":"2023-03-03T08:43:45.106617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:04:52.880667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979475+0000","last_change":"2023-03-03T08:43:43.992992+0000","last_active":"2023-03-03T08:43:44.979475+0000","last_peered":"2023-03-03T08:43:44.979475+0000","last_clean":"2023-03-03T08:43:44.979475+0000","last_became_active":"2023-03-03T08:43:43.992664+0000","last_became_peered":"2023-03-03T08:43:43.992664+0000","last_unstale":"2023-03-03T08:43:44.979475+0000","last_undegraded":"2023-03-03T08:43:44.979475+0000","last_fullsized":"2023-03-03T08:43:44.979475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:07.276715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981207+0000","last_change":"2023-03-03T08:43:34.547450+0000","last_active":"2023-03-03T08:43:43.981207+0000","last_peered":"2023-03-03T08:43:43.981207+0000","last_clean":"2023-03-03T08:43:43.981207+0000","last_became_active":"2023-03-03T08:43:34.547168+0000","last_became_peered":"2023-03-03T08:43:34.547168+0000","last_unstale":"2023-03-03T08:43:43.981207+0000","last_undegraded":"2023-03-03T08:43:43.981207+0000","last_fullsized":"2023-03-03T08:43:43.981207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:17.723229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982025+0000","last_change":"2023-03-03T08:43:34.106505+0000","last_active":"2023-03-03T08:43:43.982025+0000","last_peered":"2023-03-03T08:43:43.982025+0000","last_clean":"2023-03-03T08:43:43.982025+0000","last_became_active":"2023-03-03T08:43:34.106079+0000","last_became_peered":"2023-03-03T08:43:34.106079+0000","last_unstale":"2023-03-03T08:43:43.982025+0000","last_undegraded":"2023-03-03T08:43:43.982025+0000","last_fullsized":"2023-03-03T08:43:43.982025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:48.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979698+0000","last_change":"2023-03-03T08:43:43.575448+0000","last_active":"2023-03-03T08:43:44.979698+0000","last_peered":"2023-03-03T08:43:44.979698+0000","last_clean":"2023-03-03T08:43:44.979698+0000","last_became_active":"2023-03-03T08:43:43.575142+0000","last_became_peered":"2023-03-03T08:43:43.575142+0000","last_unstale":"2023-03-03T08:43:44.979698+0000","last_undegraded":"2023-03-03T08:43:44.979698+0000","last_fullsized":"2023-03-03T08:43:44.979698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:42:09.308616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990396+0000","last_change":"2023-03-03T08:43:34.549123+0000","last_active":"2023-03-03T08:43:43.990396+0000","last_peered":"2023-03-03T08:43:43.990396+0000","last_clean":"2023-03-03T08:43:43.990396+0000","last_became_active":"2023-03-03T08:43:34.548839+0000","last_became_peered":"2023-03-03T08:43:34.548839+0000","last_unstale":"2023-03-03T08:43:43.990396+0000","last_undegraded":"2023-03-03T08:43:43.990396+0000","last_fullsized":"2023-03-03T08:43:43.990396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:38.134009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984959+0000","last_change":"2023-03-03T08:43:34.108187+0000","last_active":"2023-03-03T08:43:43.984959+0000","last_peered":"2023-03-03T08:43:43.984959+0000","last_clean":"2023-03-03T08:43:43.984959+0000","last_became_active":"2023-03-03T08:43:34.107859+0000","last_became_peered":"2023-03-03T08:43:34.107859+0000","last_unstale":"2023-03-03T08:43:43.984959+0000","last_undegraded":"2023-03-03T08:43:43.984959+0000","last_fullsized":"2023-03-03T08:43:43.984959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:46:40.595756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107953+0000","last_change":"2023-03-03T08:43:43.572738+0000","last_active":"2023-03-03T08:43:45.107953+0000","last_peered":"2023-03-03T08:43:45.107953+0000","last_clean":"2023-03-03T08:43:45.107953+0000","last_became_active":"2023-03-03T08:43:43.572435+0000","last_became_peered":"2023-03-03T08:43:43.572435+0000","last_unstale":"2023-03-03T08:43:45.107953+0000","last_undegraded":"2023-03-03T08:43:45.107953+0000","last_fullsized":"2023-03-03T08:43:45.107953+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:14.264653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105420+0000","last_change":"2023-03-03T08:43:43.995021+0000","last_active":"2023-03-03T08:43:45.105420+0000","last_peered":"2023-03-03T08:43:45.105420+0000","last_clean":"2023-03-03T08:43:45.105420+0000","last_became_active":"2023-03-03T08:43:43.994766+0000","last_became_peered":"2023-03-03T08:43:43.994766+0000","last_unstale":"2023-03-03T08:43:45.105420+0000","last_undegraded":"2023-03-03T08:43:45.105420+0000","last_fullsized":"2023-03-03T08:43:45.105420+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:14:03.827057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979582+0000","last_change":"2023-03-03T08:43:43.993391+0000","last_active":"2023-03-03T08:43:44.979582+0000","last_peered":"2023-03-03T08:43:44.979582+0000","last_clean":"2023-03-03T08:43:44.979582+0000","last_became_active":"2023-03-03T08:43:43.993087+0000","last_became_peered":"2023-03-03T08:43:43.993087+0000","last_unstale":"2023-03-03T08:43:44.979582+0000","last_undegraded":"2023-03-03T08:43:44.979582+0000","last_fullsized":"2023-03-03T08:43:44.979582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:13:01.193811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992180+0000","last_change":"2023-03-03T08:43:34.537301+0000","last_active":"2023-03-03T08:43:43.992180+0000","last_peered":"2023-03-03T08:43:43.992180+0000","last_clean":"2023-03-03T08:43:43.992180+0000","last_became_active":"2023-03-03T08:43:34.537007+0000","last_became_peered":"2023-03-03T08:43:34.537007+0000","last_unstale":"2023-03-03T08:43:43.992180+0000","last_undegraded":"2023-03-03T08:43:43.992180+0000","last_fullsized":"2023-03-03T08:43:43.992180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:59:49.124140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982700+0000","last_change":"2023-03-03T08:43:34.108056+0000","last_active":"2023-03-03T08:43:43.982700+0000","last_peered":"2023-03-03T08:43:43.982700+0000","last_clean":"2023-03-03T08:43:43.982700+0000","last_became_active":"2023-03-03T08:43:34.107708+0000","last_became_peered":"2023-03-03T08:43:34.107708+0000","last_unstale":"2023-03-03T08:43:43.982700+0000","last_undegraded":"2023-03-03T08:43:43.982700+0000","last_fullsized":"2023-03-03T08:43:43.982700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:10.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107314+0000","last_change":"2023-03-03T08:43:43.575788+0000","last_active":"2023-03-03T08:43:45.107314+0000","last_peered":"2023-03-03T08:43:45.107314+0000","last_clean":"2023-03-03T08:43:45.107314+0000","last_became_active":"2023-03-03T08:43:43.575507+0000","last_became_peered":"2023-03-03T08:43:43.575507+0000","last_unstale":"2023-03-03T08:43:45.107314+0000","last_undegraded":"2023-03-03T08:43:45.107314+0000","last_fullsized":"2023-03-03T08:43:45.107314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:15.843816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107877+0000","last_change":"2023-03-03T08:43:43.994916+0000","last_active":"2023-03-03T08:43:45.107877+0000","last_peered":"2023-03-03T08:43:45.107877+0000","last_clean":"2023-03-03T08:43:45.107877+0000","last_became_active":"2023-03-03T08:43:43.994639+0000","last_became_peered":"2023-03-03T08:43:43.994639+0000","last_unstale":"2023-03-03T08:43:45.107877+0000","last_undegraded":"2023-03-03T08:43:45.107877+0000","last_fullsized":"2023-03-03T08:43:45.107877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:52.526052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980834+0000","last_change":"2023-03-03T08:43:34.533987+0000","last_active":"2023-03-03T08:43:43.980834+0000","last_peered":"2023-03-03T08:43:43.980834+0000","last_clean":"2023-03-03T08:43:43.980834+0000","last_became_active":"2023-03-03T08:43:34.533729+0000","last_became_peered":"2023-03-03T08:43:34.533729+0000","last_unstale":"2023-03-03T08:43:43.980834+0000","last_undegraded":"2023-03-03T08:43:43.980834+0000","last_fullsized":"2023-03-03T08:43:43.980834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:18:12.407151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106398+0000","last_change":"2023-03-03T08:43:43.996808+0000","last_active":"2023-03-03T08:43:45.106398+0000","last_peered":"2023-03-03T08:43:45.106398+0000","last_clean":"2023-03-03T08:43:45.106398+0000","last_became_active":"2023-03-03T08:43:43.996559+0000","last_became_peered":"2023-03-03T08:43:43.996559+0000","last_unstale":"2023-03-03T08:43:45.106398+0000","last_undegraded":"2023-03-03T08:43:45.106398+0000","last_fullsized":"2023-03-03T08:43:45.106398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:01.223908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.993213+0000","last_change":"2023-03-03T08:43:34.540762+0000","last_active":"2023-03-03T08:43:43.993213+0000","last_peered":"2023-03-03T08:43:43.993213+0000","last_clean":"2023-03-03T08:43:43.993213+0000","last_became_active":"2023-03-03T08:43:34.540467+0000","last_became_peered":"2023-03-03T08:43:34.540467+0000","last_unstale":"2023-03-03T08:43:43.993213+0000","last_undegraded":"2023-03-03T08:43:43.993213+0000","last_fullsized":"2023-03-03T08:43:43.993213+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107993+0000","last_change":"2023-03-03T08:43:43.995286+0000","last_active":"2023-03-03T08:43:45.107993+0000","last_peered":"2023-03-03T08:43:45.107993+0000","last_clean":"2023-03-03T08:43:45.107993+0000","last_became_active":"2023-03-03T08:43:43.994861+0000","last_became_peered":"2023-03-03T08:43:43.994861+0000","last_unstale":"2023-03-03T08:43:45.107993+0000","last_undegraded":"2023-03-03T08:43:45.107993+0000","last_fullsized":"2023-03-03T08:43:45.107993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990722+0000","last_change":"2023-03-03T08:43:34.542858+0000","last_active":"2023-03-03T08:43:43.990722+0000","last_peered":"2023-03-03T08:43:43.990722+0000","last_clean":"2023-03-03T08:43:43.990722+0000","last_became_active":"2023-03-03T08:43:34.542510+0000","last_became_peered":"2023-03-03T08:43:34.542510+0000","last_unstale":"2023-03-03T08:43:43.990722+0000","last_undegraded":"2023-03-03T08:43:43.990722+0000","last_fullsized":"2023-03-03T08:43:43.990722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:21.471234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106711+0000","last_change":"2023-03-03T08:43:43.992084+0000","last_active":"2023-03-03T08:43:45.106711+0000","last_peered":"2023-03-03T08:43:45.106711+0000","last_clean":"2023-03-03T08:43:45.106711+0000","last_became_active":"2023-03-03T08:43:43.991465+0000","last_became_peered":"2023-03-03T08:43:43.991465+0000","last_unstale":"2023-03-03T08:43:45.106711+0000","last_undegraded":"2023-03-03T08:43:45.106711+0000","last_fullsized":"2023-03-03T08:43:45.106711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:19.530923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983571+0000","last_change":"2023-03-03T08:43:34.548469+0000","last_active":"2023-03-03T08:43:43.983571+0000","last_peered":"2023-03-03T08:43:43.983571+0000","last_clean":"2023-03-03T08:43:43.983571+0000","last_became_active":"2023-03-03T08:43:34.547647+0000","last_became_peered":"2023-03-03T08:43:34.547647+0000","last_unstale":"2023-03-03T08:43:43.983571+0000","last_undegraded":"2023-03-03T08:43:43.983571+0000","last_fullsized":"2023-03-03T08:43:43.983571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:40.054781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107161+0000","last_change":"2023-03-03T08:43:43.998061+0000","last_active":"2023-03-03T08:43:45.107161+0000","last_peered":"2023-03-03T08:43:45.107161+0000","last_clean":"2023-03-03T08:43:45.107161+0000","last_became_active":"2023-03-03T08:43:43.997770+0000","last_became_peered":"2023-03-03T08:43:43.997770+0000","last_unstale":"2023-03-03T08:43:45.107161+0000","last_undegraded":"2023-03-03T08:43:45.107161+0000","last_fullsized":"2023-03-03T08:43:45.107161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:17.053610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988635+0000","last_change":"2023-03-03T08:43:34.549771+0000","last_active":"2023-03-03T08:43:43.988635+0000","last_peered":"2023-03-03T08:43:43.988635+0000","last_clean":"2023-03-03T08:43:43.988635+0000","last_became_active":"2023-03-03T08:43:34.549479+0000","last_became_peered":"2023-03-03T08:43:34.549479+0000","last_unstale":"2023-03-03T08:43:43.988635+0000","last_undegraded":"2023-03-03T08:43:43.988635+0000","last_fullsized":"2023-03-03T08:43:43.988635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:41:03.902019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979024+0000","last_change":"2023-03-03T08:43:43.999391+0000","last_active":"2023-03-03T08:43:44.979024+0000","last_peered":"2023-03-03T08:43:44.979024+0000","last_clean":"2023-03-03T08:43:44.979024+0000","last_became_active":"2023-03-03T08:43:43.998904+0000","last_became_peered":"2023-03-03T08:43:43.998904+0000","last_unstale":"2023-03-03T08:43:44.979024+0000","last_undegraded":"2023-03-03T08:43:44.979024+0000","last_fullsized":"2023-03-03T08:43:44.979024+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:17.429776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992365+0000","last_change":"2023-03-03T08:43:34.542661+0000","last_active":"2023-03-03T08:43:43.992365+0000","last_peered":"2023-03-03T08:43:43.992365+0000","last_clean":"2023-03-03T08:43:43.992365+0000","last_became_active":"2023-03-03T08:43:34.542353+0000","last_became_peered":"2023-03-03T08:43:34.542353+0000","last_unstale":"2023-03-03T08:43:43.992365+0000","last_undegraded":"2023-03-03T08:43:43.992365+0000","last_fullsized":"2023-03-03T08:43:43.992365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:38.730662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105103+0000","last_change":"2023-03-03T08:43:43.996729+0000","last_active":"2023-03-03T08:43:45.105103+0000","last_peered":"2023-03-03T08:43:45.105103+0000","last_clean":"2023-03-03T08:43:45.105103+0000","last_became_active":"2023-03-03T08:43:43.986535+0000","last_became_peered":"2023-03-03T08:43:43.986535+0000","last_unstale":"2023-03-03T08:43:45.105103+0000","last_undegraded":"2023-03-03T08:43:45.105103+0000","last_fullsized":"2023-03-03T08:43:45.105103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:22.505305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982406+0000","last_change":"2023-03-03T08:43:34.549837+0000","last_active":"2023-03-03T08:43:43.982406+0000","last_peered":"2023-03-03T08:43:43.982406+0000","last_clean":"2023-03-03T08:43:43.982406+0000","last_became_active":"2023-03-03T08:43:34.549542+0000","last_became_peered":"2023-03-03T08:43:34.549542+0000","last_unstale":"2023-03-03T08:43:43.982406+0000","last_undegraded":"2023-03-03T08:43:43.982406+0000","last_fullsized":"2023-03-03T08:43:43.982406+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:31.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106334+0000","last_change":"2023-03-03T08:43:43.993213+0000","last_active":"2023-03-03T08:43:45.106334+0000","last_peered":"2023-03-03T08:43:45.106334+0000","last_clean":"2023-03-03T08:43:45.106334+0000","last_became_active":"2023-03-03T08:43:43.992920+0000","last_became_peered":"2023-03-03T08:43:43.992920+0000","last_unstale":"2023-03-03T08:43:45.106334+0000","last_undegraded":"2023-03-03T08:43:45.106334+0000","last_fullsized":"2023-03-03T08:43:45.106334+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:42.055599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.547003+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.546744+0000","last_became_peered":"2023-03-03T08:43:34.546744+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:53:38.996605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106934+0000","last_change":"2023-03-03T08:43:43.994195+0000","last_active":"2023-03-03T08:43:45.106934+0000","last_peered":"2023-03-03T08:43:45.106934+0000","last_clean":"2023-03-03T08:43:45.106934+0000","last_became_active":"2023-03-03T08:43:43.993463+0000","last_became_peered":"2023-03-03T08:43:43.993463+0000","last_unstale":"2023-03-03T08:43:45.106934+0000","last_undegraded":"2023-03-03T08:43:45.106934+0000","last_fullsized":"2023-03-03T08:43:45.106934+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:52.365793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982677+0000","last_change":"2023-03-03T08:43:34.553589+0000","last_active":"2023-03-03T08:43:43.982677+0000","last_peered":"2023-03-03T08:43:43.982677+0000","last_clean":"2023-03-03T08:43:43.982677+0000","last_became_active":"2023-03-03T08:43:34.553242+0000","last_became_peered":"2023-03-03T08:43:34.553242+0000","last_unstale":"2023-03-03T08:43:43.982677+0000","last_undegraded":"2023-03-03T08:43:43.982677+0000","last_fullsized":"2023-03-03T08:43:43.982677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:09.000051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107426+0000","last_change":"2023-03-03T08:43:44.004404+0000","last_active":"2023-03-03T08:43:45.107426+0000","last_peered":"2023-03-03T08:43:45.107426+0000","last_clean":"2023-03-03T08:43:45.107426+0000","last_became_active":"2023-03-03T08:43:44.004120+0000","last_became_peered":"2023-03-03T08:43:44.004120+0000","last_unstale":"2023-03-03T08:43:45.107426+0000","last_undegraded":"2023-03-03T08:43:45.107426+0000","last_fullsized":"2023-03-03T08:43:45.107426+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:36.006791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990661+0000","last_change":"2023-03-03T08:43:34.537110+0000","last_active":"2023-03-03T08:43:43.990661+0000","last_peered":"2023-03-03T08:43:43.990661+0000","last_clean":"2023-03-03T08:43:43.990661+0000","last_became_active":"2023-03-03T08:43:34.536848+0000","last_became_peered":"2023-03-03T08:43:34.536848+0000","last_unstale":"2023-03-03T08:43:43.990661+0000","last_undegraded":"2023-03-03T08:43:43.990661+0000","last_fullsized":"2023-03-03T08:43:43.990661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106366+0000","last_change":"2023-03-03T08:43:43.999266+0000","last_active":"2023-03-03T08:43:45.106366+0000","last_peered":"2023-03-03T08:43:45.106366+0000","last_clean":"2023-03-03T08:43:45.106366+0000","last_became_active":"2023-03-03T08:43:43.998929+0000","last_became_peered":"2023-03-03T08:43:43.998929+0000","last_unstale":"2023-03-03T08:43:45.106366+0000","last_undegraded":"2023-03-03T08:43:45.106366+0000","last_fullsized":"2023-03-03T08:43:45.106366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:02.236263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994220+0000","last_change":"2023-03-03T08:43:34.532154+0000","last_active":"2023-03-03T08:43:43.994220+0000","last_peered":"2023-03-03T08:43:43.994220+0000","last_clean":"2023-03-03T08:43:43.994220+0000","last_became_active":"2023-03-03T08:43:34.531844+0000","last_became_peered":"2023-03-03T08:43:34.531844+0000","last_unstale":"2023-03-03T08:43:43.994220+0000","last_undegraded":"2023-03-03T08:43:43.994220+0000","last_fullsized":"2023-03-03T08:43:43.994220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:00.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106991+0000","last_change":"2023-03-03T08:43:44.004469+0000","last_active":"2023-03-03T08:43:45.106991+0000","last_peered":"2023-03-03T08:43:45.106991+0000","last_clean":"2023-03-03T08:43:45.106991+0000","last_became_active":"2023-03-03T08:43:44.004187+0000","last_became_peered":"2023-03-03T08:43:44.004187+0000","last_unstale":"2023-03-03T08:43:45.106991+0000","last_undegraded":"2023-03-03T08:43:45.106991+0000","last_fullsized":"2023-03-03T08:43:45.106991+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:35.298857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979308+0000","last_change":"2023-03-03T08:43:34.527373+0000","last_active":"2023-03-03T08:43:43.979308+0000","last_peered":"2023-03-03T08:43:43.979308+0000","last_clean":"2023-03-03T08:43:43.979308+0000","last_became_active":"2023-03-03T08:43:34.527171+0000","last_became_peered":"2023-03-03T08:43:34.527171+0000","last_unstale":"2023-03-03T08:43:43.979308+0000","last_undegraded":"2023-03-03T08:43:43.979308+0000","last_fullsized":"2023-03-03T08:43:43.979308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:20:51.138587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108220+0000","last_change":"2023-03-03T08:43:43.996329+0000","last_active":"2023-03-03T08:43:45.108220+0000","last_peered":"2023-03-03T08:43:45.108220+0000","last_clean":"2023-03-03T08:43:45.108220+0000","last_became_active":"2023-03-03T08:43:43.995055+0000","last_became_peered":"2023-03-03T08:43:43.995055+0000","last_unstale":"2023-03-03T08:43:45.108220+0000","last_undegraded":"2023-03-03T08:43:45.108220+0000","last_fullsized":"2023-03-03T08:43:45.108220+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:57.593740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984695+0000","last_change":"2023-03-03T08:43:34.542729+0000","last_active":"2023-03-03T08:43:43.984695+0000","last_peered":"2023-03-03T08:43:43.984695+0000","last_clean":"2023-03-03T08:43:43.984695+0000","last_became_active":"2023-03-03T08:43:34.542246+0000","last_became_peered":"2023-03-03T08:43:34.542246+0000","last_unstale":"2023-03-03T08:43:43.984695+0000","last_undegraded":"2023-03-03T08:43:43.984695+0000","last_fullsized":"2023-03-03T08:43:43.984695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:14:59.995526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105365+0000","last_change":"2023-03-03T08:43:44.002358+0000","last_active":"2023-03-03T08:43:45.105365+0000","last_peered":"2023-03-03T08:43:45.105365+0000","last_clean":"2023-03-03T08:43:45.105365+0000","last_became_active":"2023-03-03T08:43:44.002020+0000","last_became_peered":"2023-03-03T08:43:44.002020+0000","last_unstale":"2023-03-03T08:43:45.105365+0000","last_undegraded":"2023-03-03T08:43:45.105365+0000","last_fullsized":"2023-03-03T08:43:45.105365+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:51.524774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982664+0000","last_change":"2023-03-03T08:43:34.549026+0000","last_active":"2023-03-03T08:43:43.982664+0000","last_peered":"2023-03-03T08:43:43.982664+0000","last_clean":"2023-03-03T08:43:43.982664+0000","last_became_active":"2023-03-03T08:43:34.548778+0000","last_became_peered":"2023-03-03T08:43:34.548778+0000","last_unstale":"2023-03-03T08:43:43.982664+0000","last_undegraded":"2023-03-03T08:43:43.982664+0000","last_fullsized":"2023-03-03T08:43:43.982664+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:41.835304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105281+0000","last_change":"2023-03-03T08:43:43.997684+0000","last_active":"2023-03-03T08:43:45.105281+0000","last_peered":"2023-03-03T08:43:45.105281+0000","last_clean":"2023-03-03T08:43:45.105281+0000","last_became_active":"2023-03-03T08:43:43.997503+0000","last_became_peered":"2023-03-03T08:43:43.997503+0000","last_unstale":"2023-03-03T08:43:45.105281+0000","last_undegraded":"2023-03-03T08:43:45.105281+0000","last_fullsized":"2023-03-03T08:43:45.105281+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:29.512524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991178+0000","last_change":"2023-03-03T08:43:34.544013+0000","last_active":"2023-03-03T08:43:43.991178+0000","last_peered":"2023-03-03T08:43:43.991178+0000","last_clean":"2023-03-03T08:43:43.991178+0000","last_became_active":"2023-03-03T08:43:34.543745+0000","last_became_peered":"2023-03-03T08:43:34.543745+0000","last_unstale":"2023-03-03T08:43:43.991178+0000","last_undegraded":"2023-03-03T08:43:43.991178+0000","last_fullsized":"2023-03-03T08:43:43.991178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:05.647977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107560+0000","last_change":"2023-03-03T08:43:43.999884+0000","last_active":"2023-03-03T08:43:45.107560+0000","last_peered":"2023-03-03T08:43:45.107560+0000","last_clean":"2023-03-03T08:43:45.107560+0000","last_became_active":"2023-03-03T08:43:43.999593+0000","last_became_peered":"2023-03-03T08:43:43.999593+0000","last_unstale":"2023-03-03T08:43:45.107560+0000","last_undegraded":"2023-03-03T08:43:45.107560+0000","last_fullsized":"2023-03-03T08:43:45.107560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:45:09.132974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982600+0000","last_change":"2023-03-03T08:43:34.553434+0000","last_active":"2023-03-03T08:43:43.982600+0000","last_peered":"2023-03-03T08:43:43.982600+0000","last_clean":"2023-03-03T08:43:43.982600+0000","last_became_active":"2023-03-03T08:43:34.553125+0000","last_became_peered":"2023-03-03T08:43:34.553125+0000","last_unstale":"2023-03-03T08:43:43.982600+0000","last_undegraded":"2023-03-03T08:43:43.982600+0000","last_fullsized":"2023-03-03T08:43:43.982600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:41:17.553506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105717+0000","last_change":"2023-03-03T08:43:44.006710+0000","last_active":"2023-03-03T08:43:45.105717+0000","last_peered":"2023-03-03T08:43:45.105717+0000","last_clean":"2023-03-03T08:43:45.105717+0000","last_became_active":"2023-03-03T08:43:44.006447+0000","last_became_peered":"2023-03-03T08:43:44.006447+0000","last_unstale":"2023-03-03T08:43:45.105717+0000","last_undegraded":"2023-03-03T08:43:45.105717+0000","last_fullsized":"2023-03-03T08:43:45.105717+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:21.767326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982538+0000","last_change":"2023-03-03T08:43:34.550889+0000","last_active":"2023-03-03T08:43:43.982538+0000","last_peered":"2023-03-03T08:43:43.982538+0000","last_clean":"2023-03-03T08:43:43.982538+0000","last_became_active":"2023-03-03T08:43:34.550605+0000","last_became_peered":"2023-03-03T08:43:34.550605+0000","last_unstale":"2023-03-03T08:43:43.982538+0000","last_undegraded":"2023-03-03T08:43:43.982538+0000","last_fullsized":"2023-03-03T08:43:43.982538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:01.118555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105567+0000","last_change":"2023-03-03T08:43:43.998231+0000","last_active":"2023-03-03T08:43:45.105567+0000","last_peered":"2023-03-03T08:43:45.105567+0000","last_clean":"2023-03-03T08:43:45.105567+0000","last_became_active":"2023-03-03T08:43:43.997941+0000","last_became_peered":"2023-03-03T08:43:43.997941+0000","last_unstale":"2023-03-03T08:43:45.105567+0000","last_undegraded":"2023-03-03T08:43:45.105567+0000","last_fullsized":"2023-03-03T08:43:45.105567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:13:32.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983100+0000","last_change":"2023-03-03T08:43:34.548229+0000","last_active":"2023-03-03T08:43:43.983100+0000","last_peered":"2023-03-03T08:43:43.983100+0000","last_clean":"2023-03-03T08:43:43.983100+0000","last_became_active":"2023-03-03T08:43:34.548013+0000","last_became_peered":"2023-03-03T08:43:34.548013+0000","last_unstale":"2023-03-03T08:43:43.983100+0000","last_undegraded":"2023-03-03T08:43:43.983100+0000","last_fullsized":"2023-03-03T08:43:43.983100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:27.793716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108186+0000","last_change":"2023-03-03T08:43:43.992834+0000","last_active":"2023-03-03T08:43:45.108186+0000","last_peered":"2023-03-03T08:43:45.108186+0000","last_clean":"2023-03-03T08:43:45.108186+0000","last_became_active":"2023-03-03T08:43:43.991577+0000","last_became_peered":"2023-03-03T08:43:43.991577+0000","last_unstale":"2023-03-03T08:43:45.108186+0000","last_undegraded":"2023-03-03T08:43:45.108186+0000","last_fullsized":"2023-03-03T08:43:45.108186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:18:37.119460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990785+0000","last_change":"2023-03-03T08:43:34.549838+0000","last_active":"2023-03-03T08:43:43.990785+0000","last_peered":"2023-03-03T08:43:43.990785+0000","last_clean":"2023-03-03T08:43:43.990785+0000","last_became_active":"2023-03-03T08:43:34.549537+0000","last_became_peered":"2023-03-03T08:43:34.549537+0000","last_unstale":"2023-03-03T08:43:43.990785+0000","last_undegraded":"2023-03-03T08:43:43.990785+0000","last_fullsized":"2023-03-03T08:43:43.990785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:36.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978412+0000","last_change":"2023-03-03T08:43:43.991488+0000","last_active":"2023-03-03T08:43:44.978412+0000","last_peered":"2023-03-03T08:43:44.978412+0000","last_clean":"2023-03-03T08:43:44.978412+0000","last_became_active":"2023-03-03T08:43:43.991139+0000","last_became_peered":"2023-03-03T08:43:43.991139+0000","last_unstale":"2023-03-03T08:43:44.978412+0000","last_undegraded":"2023-03-03T08:43:44.978412+0000","last_fullsized":"2023-03-03T08:43:44.978412+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:21.119183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983652+0000","last_change":"2023-03-03T08:43:34.533496+0000","last_active":"2023-03-03T08:43:43.983652+0000","last_peered":"2023-03-03T08:43:43.983652+0000","last_clean":"2023-03-03T08:43:43.983652+0000","last_became_active":"2023-03-03T08:43:34.532714+0000","last_became_peered":"2023-03-03T08:43:34.532714+0000","last_unstale":"2023-03-03T08:43:43.983652+0000","last_undegraded":"2023-03-03T08:43:43.983652+0000","last_fullsized":"2023-03-03T08:43:43.983652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:47.800807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105250+0000","last_change":"2023-03-03T08:43:43.996215+0000","last_active":"2023-03-03T08:43:45.105250+0000","last_peered":"2023-03-03T08:43:45.105250+0000","last_clean":"2023-03-03T08:43:45.105250+0000","last_became_active":"2023-03-03T08:43:43.995791+0000","last_became_peered":"2023-03-03T08:43:43.995791+0000","last_unstale":"2023-03-03T08:43:45.105250+0000","last_undegraded":"2023-03-03T08:43:45.105250+0000","last_fullsized":"2023-03-03T08:43:45.105250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:50:41.130427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979929+0000","last_change":"2023-03-03T08:43:34.533673+0000","last_active":"2023-03-03T08:43:43.979929+0000","last_peered":"2023-03-03T08:43:43.979929+0000","last_clean":"2023-03-03T08:43:43.979929+0000","last_became_active":"2023-03-03T08:43:34.532283+0000","last_became_peered":"2023-03-03T08:43:34.532283+0000","last_unstale":"2023-03-03T08:43:43.979929+0000","last_undegraded":"2023-03-03T08:43:43.979929+0000","last_fullsized":"2023-03-03T08:43:43.979929+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:25.591188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979205+0000","last_change":"2023-03-03T08:43:43.999435+0000","last_active":"2023-03-03T08:43:44.979205+0000","last_peered":"2023-03-03T08:43:44.979205+0000","last_clean":"2023-03-03T08:43:44.979205+0000","last_became_active":"2023-03-03T08:43:43.997037+0000","last_became_peered":"2023-03-03T08:43:43.997037+0000","last_unstale":"2023-03-03T08:43:44.979205+0000","last_undegraded":"2023-03-03T08:43:44.979205+0000","last_fullsized":"2023-03-03T08:43:44.979205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:54.176343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980585+0000","last_change":"2023-03-03T08:43:34.537664+0000","last_active":"2023-03-03T08:43:43.980585+0000","last_peered":"2023-03-03T08:43:43.980585+0000","last_clean":"2023-03-03T08:43:43.980585+0000","last_became_active":"2023-03-03T08:43:34.537214+0000","last_became_peered":"2023-03-03T08:43:34.537214+0000","last_unstale":"2023-03-03T08:43:43.980585+0000","last_undegraded":"2023-03-03T08:43:43.980585+0000","last_fullsized":"2023-03-03T08:43:43.980585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:54:09.222135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106227+0000","last_change":"2023-03-03T08:43:44.000140+0000","last_active":"2023-03-03T08:43:45.106227+0000","last_peered":"2023-03-03T08:43:45.106227+0000","last_clean":"2023-03-03T08:43:45.106227+0000","last_became_active":"2023-03-03T08:43:43.999778+0000","last_became_peered":"2023-03-03T08:43:43.999778+0000","last_unstale":"2023-03-03T08:43:45.106227+0000","last_undegraded":"2023-03-03T08:43:45.106227+0000","last_fullsized":"2023-03-03T08:43:45.106227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:29:10.745759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987366+0000","last_change":"2023-03-03T08:43:34.543620+0000","last_active":"2023-03-03T08:43:43.987366+0000","last_peered":"2023-03-03T08:43:43.987366+0000","last_clean":"2023-03-03T08:43:43.987366+0000","last_became_active":"2023-03-03T08:43:34.543370+0000","last_became_peered":"2023-03-03T08:43:34.543370+0000","last_unstale":"2023-03-03T08:43:43.987366+0000","last_undegraded":"2023-03-03T08:43:43.987366+0000","last_fullsized":"2023-03-03T08:43:43.987366+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:40:32.808203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979449+0000","last_change":"2023-03-03T08:43:43.995084+0000","last_active":"2023-03-03T08:43:44.979449+0000","last_peered":"2023-03-03T08:43:44.979449+0000","last_clean":"2023-03-03T08:43:44.979449+0000","last_became_active":"2023-03-03T08:43:43.994824+0000","last_became_peered":"2023-03-03T08:43:43.994824+0000","last_unstale":"2023-03-03T08:43:44.979449+0000","last_undegraded":"2023-03-03T08:43:44.979449+0000","last_fullsized":"2023-03-03T08:43:44.979449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:04:17.481969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985302+0000","last_change":"2023-03-03T08:43:34.538467+0000","last_active":"2023-03-03T08:43:43.985302+0000","last_peered":"2023-03-03T08:43:43.985302+0000","last_clean":"2023-03-03T08:43:43.985302+0000","last_became_active":"2023-03-03T08:43:34.538207+0000","last_became_peered":"2023-03-03T08:43:34.538207+0000","last_unstale":"2023-03-03T08:43:43.985302+0000","last_undegraded":"2023-03-03T08:43:43.985302+0000","last_fullsized":"2023-03-03T08:43:43.985302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:14.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980577+0000","last_change":"2023-03-03T08:43:43.996076+0000","last_active":"2023-03-03T08:43:44.980577+0000","last_peered":"2023-03-03T08:43:44.980577+0000","last_clean":"2023-03-03T08:43:44.980577+0000","last_became_active":"2023-03-03T08:43:43.995799+0000","last_became_peered":"2023-03-03T08:43:43.995799+0000","last_unstale":"2023-03-03T08:43:44.980577+0000","last_undegraded":"2023-03-03T08:43:44.980577+0000","last_fullsized":"2023-03-03T08:43:44.980577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:35:35.889895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986289+0000","last_change":"2023-03-03T08:43:34.544721+0000","last_active":"2023-03-03T08:43:43.986289+0000","last_peered":"2023-03-03T08:43:43.986289+0000","last_clean":"2023-03-03T08:43:43.986289+0000","last_became_active":"2023-03-03T08:43:34.544461+0000","last_became_peered":"2023-03-03T08:43:34.544461+0000","last_unstale":"2023-03-03T08:43:43.986289+0000","last_undegraded":"2023-03-03T08:43:43.986289+0000","last_fullsized":"2023-03-03T08:43:43.986289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:52:11.890779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980078+0000","last_change":"2023-03-03T08:43:44.006352+0000","last_active":"2023-03-03T08:43:44.980078+0000","last_peered":"2023-03-03T08:43:44.980078+0000","last_clean":"2023-03-03T08:43:44.980078+0000","last_became_active":"2023-03-03T08:43:44.005802+0000","last_became_peered":"2023-03-03T08:43:44.005802+0000","last_unstale":"2023-03-03T08:43:44.980078+0000","last_undegraded":"2023-03-03T08:43:44.980078+0000","last_fullsized":"2023-03-03T08:43:44.980078+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:52.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986425+0000","last_change":"2023-03-03T08:43:34.541669+0000","last_active":"2023-03-03T08:43:43.986425+0000","last_peered":"2023-03-03T08:43:43.986425+0000","last_clean":"2023-03-03T08:43:43.986425+0000","last_became_active":"2023-03-03T08:43:34.541104+0000","last_became_peered":"2023-03-03T08:43:34.541104+0000","last_unstale":"2023-03-03T08:43:43.986425+0000","last_undegraded":"2023-03-03T08:43:43.986425+0000","last_fullsized":"2023-03-03T08:43:43.986425+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980059+0000","last_change":"2023-03-03T08:43:43.998937+0000","last_active":"2023-03-03T08:43:44.980059+0000","last_peered":"2023-03-03T08:43:44.980059+0000","last_clean":"2023-03-03T08:43:44.980059+0000","last_became_active":"2023-03-03T08:43:43.998164+0000","last_became_peered":"2023-03-03T08:43:43.998164+0000","last_unstale":"2023-03-03T08:43:44.980059+0000","last_undegraded":"2023-03-03T08:43:44.980059+0000","last_fullsized":"2023-03-03T08:43:44.980059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:53.741704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.546272+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.546006+0000","last_became_peered":"2023-03-03T08:43:34.546006+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:15.121761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":3561,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3561,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":118397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":119470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9336,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362504,"statfs":{"total":96636764160,"available":96627204096,"internally_reserved":0,"allocated":901120,"data_stored":710394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8244,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363596,"statfs":{"total":96636764160,"available":96628322304,"internally_reserved":0,"allocated":307200,"data_stored":118565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8252,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363588,"statfs":{"total":96636764160,"available":96628314112,"internally_reserved":0,"allocated":315392,"data_stored":124451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":121300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":122249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9260,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362580,"statfs":{"total":96636764160,"available":96627281920,"internally_reserved":0,"allocated":888832,"data_stored":709827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:44:08.233 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-03-03T08:44:08.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-03T08:44:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.386+0000 7fbc18061700 1 Processor -- start 2023-03-03T08:44:08.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.386+0000 7fbc18061700 1 -- start start 2023-03-03T08:44:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.387+0000 7fbc18061700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbc1006c6c0 0x7fbc100736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.387+0000 7fbc18061700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc10157ff0 con 0x7fbc100c2e30 2023-03-03T08:44:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.387+0000 7fbc18061700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1006d640 con 0x7fbc10151cd0 2023-03-03T08:44:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.387+0000 7fbc18061700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc1006cda0 con 0x7fbc1006c6c0 2023-03-03T08:44:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.387+0000 7fbc155fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbc100c2e30 0x7fbc10158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46280/0 (socket says 172.21.15.7:46280) 2023-03-03T08:44:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.387+0000 7fbc155fc700 1 -- 172.21.15.7:0/2147801111 learned_addr learned my addr 172.21.15.7:0/2147801111 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.387+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459280894 0 0) 0x7fbc10157ff0 con 0x7fbc100c2e30 2023-03-03T08:44:08.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.387+0000 7fbc165fe700 1 --2- 172.21.15.7:0/2147801111 >> v2:172.21.15.143:3300/0 conn(0x7fbc1006c6c0 0x7fbc100736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.387+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbfc0035b0 con 0x7fbc100c2e30 2023-03-03T08:44:08.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 131159993 0 0) 0x7fbbfc0035b0 con 0x7fbc100c2e30 2023-03-03T08:44:08.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 >> v2:172.21.15.143:3300/0 conn(0x7fbc1006c6c0 msgr2=0x7fbc100736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:08.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 --2- 172.21.15.7:0/2147801111 >> v2:172.21.15.143:3300/0 conn(0x7fbc1006c6c0 0x7fbc100736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 >> v1:172.21.15.143:6789/0 conn(0x7fbc10151cd0 legacy=0x7fbc10154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1006d080 con 0x7fbc100c2e30 2023-03-03T08:44:08.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fbc0c001670 con 0x7fbc100c2e30 2023-03-03T08:44:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 >> v1:172.21.15.7:6789/0 conn(0x7fbc100c2e30 legacy=0x7fbc10158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbfc003a70 0x7fbbfc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbfc004850 0x7fbbfc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbfc005320 0x7fbbfc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1006cda0 con 0x7fbbfc005320 2023-03-03T08:44:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbfc00c230 con 0x7fbbfc003a70 2023-03-03T08:44:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbfc00c3f0 con 0x7fbbfc004850 2023-03-03T08:44:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc165fe700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbfc005320 0x7fbbfc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc155fc700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbfc004850 0x7fbbfc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc15dfd700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbfc003a70 0x7fbbfc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc165fe700 1 -- 172.21.15.7:0/2147801111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbfc004850 msgr2=0x7fbbfc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc165fe700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbfc004850 0x7fbbfc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc165fe700 1 -- 172.21.15.7:0/2147801111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbfc003a70 msgr2=0x7fbbfc003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc165fe700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbfc003a70 0x7fbbfc003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc165fe700 1 -- 172.21.15.7:0/2147801111 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbfc0035b0 con 0x7fbbfc005320 2023-03-03T08:44:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc15dfd700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbfc003a70 0x7fbbfc003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc165fe700 1 -- 172.21.15.7:0/2147801111 reap_dead start 2023-03-03T08:44:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.388+0000 7fbc165fe700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbfc005320 0x7fbbfc00ba10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009fc0 tx=0x7fbbfc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6b772fd30ec83a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.389+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08003a20 con 0x7fbbfc005320 2023-03-03T08:44:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.389+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0800f040 con 0x7fbbfc005320 2023-03-03T08:44:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.389+0000 7fbc14dfb700 1 -- 172.21.15.7:0/2147801111 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08003de0 con 0x7fbbfc005320 2023-03-03T08:44:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.389+0000 7fbc18061700 1 -- 172.21.15.7:0/2147801111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbfc005320 msgr2=0x7fbbfc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.389+0000 7fbc18061700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbfc005320 0x7fbbfc00ba10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009fc0 tx=0x7fbbfc005880 comp rx=0 tx=0).stop 2023-03-03T08:44:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.389+0000 7fbc18061700 1 -- 172.21.15.7:0/2147801111 shutdown_connections 2023-03-03T08:44:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.389+0000 7fbc18061700 1 --2- 172.21.15.7:0/2147801111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbfc005320 0x7fbbfc00ba10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.389+0000 7fbc18061700 1 -- 172.21.15.7:0/2147801111 >> 172.21.15.7:0/2147801111 conn(0x7fbc100a94d0 msgr2=0x7fbc100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.389+0000 7fbc18061700 1 -- 172.21.15.7:0/2147801111 shutdown_connections 2023-03-03T08:44:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.389+0000 7fbc18061700 1 -- 172.21.15.7:0/2147801111 wait complete. 2023-03-03T08:44:08.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.390+0000 7fbc18061700 1 Processor -- start 2023-03-03T08:44:08.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.390+0000 7fbc18061700 1 -- start start 2023-03-03T08:44:08.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc18061700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbfc005320 0x7fbc1006a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc18061700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc10155900 0x7fbc10149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc18061700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc10149bd0 0x7fbc101e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc18061700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc101e9a90 con 0x7fbc10155900 2023-03-03T08:44:08.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc18061700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1014b520 con 0x7fbbfc005320 2023-03-03T08:44:08.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc18061700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc100c5400 con 0x7fbc10149bd0 2023-03-03T08:44:08.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc155fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc10155900 0x7fbc10149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc155fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc10155900 0x7fbc10149690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52312/0 (socket says 172.21.15.7:52312) 2023-03-03T08:44:08.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc155fc700 1 -- 172.21.15.7:0/378701832 learned_addr learned my addr 172.21.15.7:0/378701832 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:08.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc15dfd700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbfc005320 0x7fbc1006a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.391+0000 7fbc165fe700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc10149bd0 0x7fbc101e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc155fc700 1 -- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc10149bd0 msgr2=0x7fbc101e9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc155fc700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc10149bd0 0x7fbc101e9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc155fc700 1 -- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbfc005320 msgr2=0x7fbc1006a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc155fc700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbfc005320 0x7fbc1006a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc155fc700 1 -- 172.21.15.7:0/378701832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc10157c80 con 0x7fbc10155900 2023-03-03T08:44:08.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc155fc700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc10155900 0x7fbc10149690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00e7e0 tx=0x7fbc0c00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:08.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc067fc700 1 -- 172.21.15.7:0/378701832 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c010510 con 0x7fbc10155900 2023-03-03T08:44:08.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc067fc700 1 -- 172.21.15.7:0/378701832 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0c010670 con 0x7fbc10155900 2023-03-03T08:44:08.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc067fc700 1 -- 172.21.15.7:0/378701832 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c010a20 con 0x7fbc10155900 2023-03-03T08:44:08.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc18061700 1 -- 172.21.15.7:0/378701832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1006d360 con 0x7fbc10155900 2023-03-03T08:44:08.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.392+0000 7fbc18061700 1 -- 172.21.15.7:0/378701832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc10158480 con 0x7fbc10155900 2023-03-03T08:44:08.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.393+0000 7fbc18061700 1 -- 172.21.15.7:0/378701832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc10074bb0 con 0x7fbc10155900 2023-03-03T08:44:08.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.393+0000 7fbc067fc700 1 -- 172.21.15.7:0/378701832 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbc0c01b030 con 0x7fbc10155900 2023-03-03T08:44:08.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.394+0000 7fbc067fc700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbbf40682f0 0x7fbbf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.394+0000 7fbc067fc700 1 -- 172.21.15.7:0/378701832 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fbc0c085770 con 0x7fbc10155900 2023-03-03T08:44:08.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.394+0000 7fbc15dfd700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbbf40682f0 0x7fbbf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.394+0000 7fbc15dfd700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbbf40682f0 0x7fbbf406a7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbc00000c00 tx=0x7fbc00009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:08.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.398+0000 7fbc067fc700 1 -- 172.21.15.7:0/378701832 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbc0c051c20 con 0x7fbc10155900 2023-03-03T08:44:08.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.584+0000 7fbc18061700 1 -- 172.21.15.7:0/378701832 --> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc100a9fb0 con 0x7fbbf40682f0 2023-03-03T08:44:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.599+0000 7fbc067fc700 1 -- 172.21.15.7:0/378701832 <== mgr.4102 v2:172.21.15.143:6832/96367 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525124 (secure 0 0 0) 0x7fbc100a9fb0 con 0x7fbbf40682f0 2023-03-03T08:44:08.605 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 -- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbbf40682f0 msgr2=0x7fbbf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbbf40682f0 0x7fbbf406a7a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbc00000c00 tx=0x7fbc00009040 comp rx=0 tx=0).stop 2023-03-03T08:44:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 -- 172.21.15.7:0/378701832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc10155900 msgr2=0x7fbc10149690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc10155900 0x7fbc10149690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00e7e0 tx=0x7fbc0c00eaf0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 -- 172.21.15.7:0/378701832 shutdown_connections 2023-03-03T08:44:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc10149bd0 0x7fbc101e9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbfc005320 0x7fbc1006a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbbf40682f0 0x7fbbf406a7a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 --2- 172.21.15.7:0/378701832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc10155900 0x7fbc10149690 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 -- 172.21.15.7:0/378701832 >> 172.21.15.7:0/378701832 conn(0x7fbc100a94d0 msgr2=0x7fbc10071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 -- 172.21.15.7:0/378701832 shutdown_connections 2023-03-03T08:44:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.603+0000 7fbc18061700 1 -- 172.21.15.7:0/378701832 wait complete. 2023-03-03T08:44:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-03-03T08:44:08.616 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2023-03-03T08:44:07.114985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597490,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":255,"num_write_kb":2450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":68116,"kb_used_data":3540,"kb_used_omap":0,"kb_used_meta":64512,"kb_avail":754906604,"statfs":{"total":773094113280,"available":773024362496,"internally_reserved":0,"allocated":3624960,"data_stored":2144653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66060288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2462,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003970"},"pg_stats":[{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980166+0000","last_change":"2023-03-03T08:43:43.997091+0000","last_active":"2023-03-03T08:43:44.980166+0000","last_peered":"2023-03-03T08:43:44.980166+0000","last_clean":"2023-03-03T08:43:44.980166+0000","last_became_active":"2023-03-03T08:43:43.996805+0000","last_became_peered":"2023-03-03T08:43:43.996805+0000","last_unstale":"2023-03-03T08:43:44.980166+0000","last_undegraded":"2023-03-03T08:43:44.980166+0000","last_fullsized":"2023-03-03T08:43:44.980166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982505+0000","last_change":"2023-03-03T08:43:34.546828+0000","last_active":"2023-03-03T08:43:43.982505+0000","last_peered":"2023-03-03T08:43:43.982505+0000","last_clean":"2023-03-03T08:43:43.982505+0000","last_became_active":"2023-03-03T08:43:34.546559+0000","last_became_peered":"2023-03-03T08:43:34.546559+0000","last_unstale":"2023-03-03T08:43:43.982505+0000","last_undegraded":"2023-03-03T08:43:43.982505+0000","last_fullsized":"2023-03-03T08:43:43.982505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:21:14.951107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106061+0000","last_change":"2023-03-03T08:43:43.995073+0000","last_active":"2023-03-03T08:43:45.106061+0000","last_peered":"2023-03-03T08:43:45.106061+0000","last_clean":"2023-03-03T08:43:45.106061+0000","last_became_active":"2023-03-03T08:43:43.994797+0000","last_became_peered":"2023-03-03T08:43:43.994797+0000","last_unstale":"2023-03-03T08:43:45.106061+0000","last_undegraded":"2023-03-03T08:43:45.106061+0000","last_fullsized":"2023-03-03T08:43:45.106061+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:21:52.642936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982397+0000","last_change":"2023-03-03T08:43:34.552931+0000","last_active":"2023-03-03T08:43:43.982397+0000","last_peered":"2023-03-03T08:43:43.982397+0000","last_clean":"2023-03-03T08:43:43.982397+0000","last_became_active":"2023-03-03T08:43:34.552623+0000","last_became_peered":"2023-03-03T08:43:34.552623+0000","last_unstale":"2023-03-03T08:43:43.982397+0000","last_undegraded":"2023-03-03T08:43:43.982397+0000","last_fullsized":"2023-03-03T08:43:43.982397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:07:43.954897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979088+0000","last_change":"2023-03-03T08:43:43.999245+0000","last_active":"2023-03-03T08:43:44.979088+0000","last_peered":"2023-03-03T08:43:44.979088+0000","last_clean":"2023-03-03T08:43:44.979088+0000","last_became_active":"2023-03-03T08:43:43.998592+0000","last_became_peered":"2023-03-03T08:43:43.998592+0000","last_unstale":"2023-03-03T08:43:44.979088+0000","last_undegraded":"2023-03-03T08:43:44.979088+0000","last_fullsized":"2023-03-03T08:43:44.979088+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:23.788724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991054+0000","last_change":"2023-03-03T08:43:34.544408+0000","last_active":"2023-03-03T08:43:43.991054+0000","last_peered":"2023-03-03T08:43:43.991054+0000","last_clean":"2023-03-03T08:43:43.991054+0000","last_became_active":"2023-03-03T08:43:34.543977+0000","last_became_peered":"2023-03-03T08:43:34.543977+0000","last_unstale":"2023-03-03T08:43:43.991054+0000","last_undegraded":"2023-03-03T08:43:43.991054+0000","last_fullsized":"2023-03-03T08:43:43.991054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:55:34.074082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105254+0000","last_change":"2023-03-03T08:43:43.994545+0000","last_active":"2023-03-03T08:43:45.105254+0000","last_peered":"2023-03-03T08:43:45.105254+0000","last_clean":"2023-03-03T08:43:45.105254+0000","last_became_active":"2023-03-03T08:43:43.994271+0000","last_became_peered":"2023-03-03T08:43:43.994271+0000","last_unstale":"2023-03-03T08:43:45.105254+0000","last_undegraded":"2023-03-03T08:43:45.105254+0000","last_fullsized":"2023-03-03T08:43:45.105254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:53:15.925960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989064+0000","last_change":"2023-03-03T08:43:34.538786+0000","last_active":"2023-03-03T08:43:43.989064+0000","last_peered":"2023-03-03T08:43:43.989064+0000","last_clean":"2023-03-03T08:43:43.989064+0000","last_became_active":"2023-03-03T08:43:34.538494+0000","last_became_peered":"2023-03-03T08:43:34.538494+0000","last_unstale":"2023-03-03T08:43:43.989064+0000","last_undegraded":"2023-03-03T08:43:43.989064+0000","last_fullsized":"2023-03-03T08:43:43.989064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:19:53.838286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105391+0000","last_change":"2023-03-03T08:43:43.993678+0000","last_active":"2023-03-03T08:43:45.105391+0000","last_peered":"2023-03-03T08:43:45.105391+0000","last_clean":"2023-03-03T08:43:45.105391+0000","last_became_active":"2023-03-03T08:43:43.993402+0000","last_became_peered":"2023-03-03T08:43:43.993402+0000","last_unstale":"2023-03-03T08:43:45.105391+0000","last_undegraded":"2023-03-03T08:43:45.105391+0000","last_fullsized":"2023-03-03T08:43:45.105391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:31:00.747216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983479+0000","last_change":"2023-03-03T08:43:34.547197+0000","last_active":"2023-03-03T08:43:43.983479+0000","last_peered":"2023-03-03T08:43:43.983479+0000","last_clean":"2023-03-03T08:43:43.983479+0000","last_became_active":"2023-03-03T08:43:34.546927+0000","last_became_peered":"2023-03-03T08:43:34.546927+0000","last_unstale":"2023-03-03T08:43:43.983479+0000","last_undegraded":"2023-03-03T08:43:43.983479+0000","last_fullsized":"2023-03-03T08:43:43.983479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:21:34.845968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984318+0000","last_change":"2023-03-03T08:43:34.540812+0000","last_active":"2023-03-03T08:43:43.984318+0000","last_peered":"2023-03-03T08:43:43.984318+0000","last_clean":"2023-03-03T08:43:43.984318+0000","last_became_active":"2023-03-03T08:43:34.540549+0000","last_became_peered":"2023-03-03T08:43:34.540549+0000","last_unstale":"2023-03-03T08:43:43.984318+0000","last_undegraded":"2023-03-03T08:43:43.984318+0000","last_fullsized":"2023-03-03T08:43:43.984318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982875+0000","last_change":"2023-03-03T08:43:34.108426+0000","last_active":"2023-03-03T08:43:43.982875+0000","last_peered":"2023-03-03T08:43:43.982875+0000","last_clean":"2023-03-03T08:43:43.982875+0000","last_became_active":"2023-03-03T08:43:34.108009+0000","last_became_peered":"2023-03-03T08:43:34.108009+0000","last_unstale":"2023-03-03T08:43:43.982875+0000","last_undegraded":"2023-03-03T08:43:43.982875+0000","last_fullsized":"2023-03-03T08:43:43.982875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:28:53.632744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979479+0000","last_change":"2023-03-03T08:43:43.575708+0000","last_active":"2023-03-03T08:43:44.979479+0000","last_peered":"2023-03-03T08:43:44.979479+0000","last_clean":"2023-03-03T08:43:44.979479+0000","last_became_active":"2023-03-03T08:43:43.575417+0000","last_became_peered":"2023-03-03T08:43:43.575417+0000","last_unstale":"2023-03-03T08:43:44.979479+0000","last_undegraded":"2023-03-03T08:43:44.979479+0000","last_fullsized":"2023-03-03T08:43:44.979479+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:25.571159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106018+0000","last_change":"2023-03-03T08:43:43.999274+0000","last_active":"2023-03-03T08:43:45.106018+0000","last_peered":"2023-03-03T08:43:45.106018+0000","last_clean":"2023-03-03T08:43:45.106018+0000","last_became_active":"2023-03-03T08:43:43.998823+0000","last_became_peered":"2023-03-03T08:43:43.998823+0000","last_unstale":"2023-03-03T08:43:45.106018+0000","last_undegraded":"2023-03-03T08:43:45.106018+0000","last_fullsized":"2023-03-03T08:43:45.106018+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:09:48.799261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992918+0000","last_change":"2023-03-03T08:43:34.532482+0000","last_active":"2023-03-03T08:43:43.992918+0000","last_peered":"2023-03-03T08:43:43.992918+0000","last_clean":"2023-03-03T08:43:43.992918+0000","last_became_active":"2023-03-03T08:43:34.531751+0000","last_became_peered":"2023-03-03T08:43:34.531751+0000","last_unstale":"2023-03-03T08:43:43.992918+0000","last_undegraded":"2023-03-03T08:43:43.992918+0000","last_fullsized":"2023-03-03T08:43:43.992918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:47:45.214945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982691+0000","last_change":"2023-03-03T08:43:34.109782+0000","last_active":"2023-03-03T08:43:43.982691+0000","last_peered":"2023-03-03T08:43:43.982691+0000","last_clean":"2023-03-03T08:43:43.982691+0000","last_became_active":"2023-03-03T08:43:34.109513+0000","last_became_peered":"2023-03-03T08:43:34.109513+0000","last_unstale":"2023-03-03T08:43:43.982691+0000","last_undegraded":"2023-03-03T08:43:43.982691+0000","last_fullsized":"2023-03-03T08:43:43.982691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:16.185161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105634+0000","last_change":"2023-03-03T08:43:43.576885+0000","last_active":"2023-03-03T08:43:45.105634+0000","last_peered":"2023-03-03T08:43:45.105634+0000","last_clean":"2023-03-03T08:43:45.105634+0000","last_became_active":"2023-03-03T08:43:43.576577+0000","last_became_peered":"2023-03-03T08:43:43.576577+0000","last_unstale":"2023-03-03T08:43:45.105634+0000","last_undegraded":"2023-03-03T08:43:45.105634+0000","last_fullsized":"2023-03-03T08:43:45.105634+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:56:26.501956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978988+0000","last_change":"2023-03-03T08:43:43.992020+0000","last_active":"2023-03-03T08:43:44.978988+0000","last_peered":"2023-03-03T08:43:44.978988+0000","last_clean":"2023-03-03T08:43:44.978988+0000","last_became_active":"2023-03-03T08:43:43.991674+0000","last_became_peered":"2023-03-03T08:43:43.991674+0000","last_unstale":"2023-03-03T08:43:44.978988+0000","last_undegraded":"2023-03-03T08:43:44.978988+0000","last_fullsized":"2023-03-03T08:43:44.978988+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:54:30.883459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985919+0000","last_change":"2023-03-03T08:43:34.535451+0000","last_active":"2023-03-03T08:43:43.985919+0000","last_peered":"2023-03-03T08:43:43.985919+0000","last_clean":"2023-03-03T08:43:43.985919+0000","last_became_active":"2023-03-03T08:43:34.535129+0000","last_became_peered":"2023-03-03T08:43:34.535129+0000","last_unstale":"2023-03-03T08:43:43.985919+0000","last_undegraded":"2023-03-03T08:43:43.985919+0000","last_fullsized":"2023-03-03T08:43:43.985919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:24:30.645128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981082+0000","last_change":"2023-03-03T08:43:34.117851+0000","last_active":"2023-03-03T08:43:43.981082+0000","last_peered":"2023-03-03T08:43:43.981082+0000","last_clean":"2023-03-03T08:43:43.981082+0000","last_became_active":"2023-03-03T08:43:34.117548+0000","last_became_peered":"2023-03-03T08:43:34.117548+0000","last_unstale":"2023-03-03T08:43:43.981082+0000","last_undegraded":"2023-03-03T08:43:43.981082+0000","last_fullsized":"2023-03-03T08:43:43.981082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:31:20.039817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106904+0000","last_change":"2023-03-03T08:43:43.569082+0000","last_active":"2023-03-03T08:43:45.106904+0000","last_peered":"2023-03-03T08:43:45.106904+0000","last_clean":"2023-03-03T08:43:45.106904+0000","last_became_active":"2023-03-03T08:43:43.568838+0000","last_became_peered":"2023-03-03T08:43:43.568838+0000","last_unstale":"2023-03-03T08:43:45.106904+0000","last_undegraded":"2023-03-03T08:43:45.106904+0000","last_fullsized":"2023-03-03T08:43:45.106904+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:47.610848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979556+0000","last_change":"2023-03-03T08:43:43.995210+0000","last_active":"2023-03-03T08:43:44.979556+0000","last_peered":"2023-03-03T08:43:44.979556+0000","last_clean":"2023-03-03T08:43:44.979556+0000","last_became_active":"2023-03-03T08:43:43.994899+0000","last_became_peered":"2023-03-03T08:43:43.994899+0000","last_unstale":"2023-03-03T08:43:44.979556+0000","last_undegraded":"2023-03-03T08:43:44.979556+0000","last_fullsized":"2023-03-03T08:43:44.979556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:37:11.974933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985908+0000","last_change":"2023-03-03T08:43:34.547322+0000","last_active":"2023-03-03T08:43:43.985908+0000","last_peered":"2023-03-03T08:43:43.985908+0000","last_clean":"2023-03-03T08:43:43.985908+0000","last_became_active":"2023-03-03T08:43:34.546692+0000","last_became_peered":"2023-03-03T08:43:34.546692+0000","last_unstale":"2023-03-03T08:43:43.985908+0000","last_undegraded":"2023-03-03T08:43:43.985908+0000","last_fullsized":"2023-03-03T08:43:43.985908+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:11:26.366920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982836+0000","last_change":"2023-03-03T08:43:34.110187+0000","last_active":"2023-03-03T08:43:43.982836+0000","last_peered":"2023-03-03T08:43:43.982836+0000","last_clean":"2023-03-03T08:43:43.982836+0000","last_became_active":"2023-03-03T08:43:34.109845+0000","last_became_peered":"2023-03-03T08:43:34.109845+0000","last_unstale":"2023-03-03T08:43:43.982836+0000","last_undegraded":"2023-03-03T08:43:43.982836+0000","last_fullsized":"2023-03-03T08:43:43.982836+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:33.507500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105532+0000","last_change":"2023-03-03T08:43:43.576805+0000","last_active":"2023-03-03T08:43:45.105532+0000","last_peered":"2023-03-03T08:43:45.105532+0000","last_clean":"2023-03-03T08:43:45.105532+0000","last_became_active":"2023-03-03T08:43:43.576241+0000","last_became_peered":"2023-03-03T08:43:43.576241+0000","last_unstale":"2023-03-03T08:43:45.105532+0000","last_undegraded":"2023-03-03T08:43:45.105532+0000","last_fullsized":"2023-03-03T08:43:45.105532+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:34:35.821307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105632+0000","last_change":"2023-03-03T08:43:43.991142+0000","last_active":"2023-03-03T08:43:45.105632+0000","last_peered":"2023-03-03T08:43:45.105632+0000","last_clean":"2023-03-03T08:43:45.105632+0000","last_became_active":"2023-03-03T08:43:43.990868+0000","last_became_peered":"2023-03-03T08:43:43.990868+0000","last_unstale":"2023-03-03T08:43:45.105632+0000","last_undegraded":"2023-03-03T08:43:45.105632+0000","last_fullsized":"2023-03-03T08:43:45.105632+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:42:58.831806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106429+0000","last_change":"2023-03-03T08:43:43.995618+0000","last_active":"2023-03-03T08:43:45.106429+0000","last_peered":"2023-03-03T08:43:45.106429+0000","last_clean":"2023-03-03T08:43:45.106429+0000","last_became_active":"2023-03-03T08:43:43.995350+0000","last_became_peered":"2023-03-03T08:43:43.995350+0000","last_unstale":"2023-03-03T08:43:45.106429+0000","last_undegraded":"2023-03-03T08:43:45.106429+0000","last_fullsized":"2023-03-03T08:43:45.106429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:22:46.445223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988843+0000","last_change":"2023-03-03T08:43:34.541172+0000","last_active":"2023-03-03T08:43:43.988843+0000","last_peered":"2023-03-03T08:43:43.988843+0000","last_clean":"2023-03-03T08:43:43.988843+0000","last_became_active":"2023-03-03T08:43:34.540739+0000","last_became_peered":"2023-03-03T08:43:34.540739+0000","last_unstale":"2023-03-03T08:43:43.988843+0000","last_undegraded":"2023-03-03T08:43:43.988843+0000","last_fullsized":"2023-03-03T08:43:43.988843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:36:40.058265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987475+0000","last_change":"2023-03-03T08:43:34.112667+0000","last_active":"2023-03-03T08:43:43.987475+0000","last_peered":"2023-03-03T08:43:43.987475+0000","last_clean":"2023-03-03T08:43:43.987475+0000","last_became_active":"2023-03-03T08:43:34.112386+0000","last_became_peered":"2023-03-03T08:43:34.112386+0000","last_unstale":"2023-03-03T08:43:43.987475+0000","last_undegraded":"2023-03-03T08:43:43.987475+0000","last_fullsized":"2023-03-03T08:43:43.987475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:43:40.504392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980167+0000","last_change":"2023-03-03T08:43:43.573962+0000","last_active":"2023-03-03T08:43:44.980167+0000","last_peered":"2023-03-03T08:43:44.980167+0000","last_clean":"2023-03-03T08:43:44.980167+0000","last_became_active":"2023-03-03T08:43:43.573200+0000","last_became_peered":"2023-03-03T08:43:43.573200+0000","last_unstale":"2023-03-03T08:43:44.980167+0000","last_undegraded":"2023-03-03T08:43:44.980167+0000","last_fullsized":"2023-03-03T08:43:44.980167+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:28:29.511681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984698+0000","last_change":"2023-03-03T08:43:34.537850+0000","last_active":"2023-03-03T08:43:43.984698+0000","last_peered":"2023-03-03T08:43:43.984698+0000","last_clean":"2023-03-03T08:43:43.984698+0000","last_became_active":"2023-03-03T08:43:34.537550+0000","last_became_peered":"2023-03-03T08:43:34.537550+0000","last_unstale":"2023-03-03T08:43:43.984698+0000","last_undegraded":"2023-03-03T08:43:43.984698+0000","last_fullsized":"2023-03-03T08:43:43.984698+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:37:36.893757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990299+0000","last_change":"2023-03-03T08:43:34.109127+0000","last_active":"2023-03-03T08:43:43.990299+0000","last_peered":"2023-03-03T08:43:43.990299+0000","last_clean":"2023-03-03T08:43:43.990299+0000","last_became_active":"2023-03-03T08:43:34.108869+0000","last_became_peered":"2023-03-03T08:43:34.108869+0000","last_unstale":"2023-03-03T08:43:43.990299+0000","last_undegraded":"2023-03-03T08:43:43.990299+0000","last_fullsized":"2023-03-03T08:43:43.990299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:19:00.275620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981298+0000","last_change":"2023-03-03T08:43:29.354627+0000","last_active":"2023-03-03T08:43:43.981298+0000","last_peered":"2023-03-03T08:43:43.981298+0000","last_clean":"2023-03-03T08:43:43.981298+0000","last_became_active":"2023-03-03T08:43:29.354271+0000","last_became_peered":"2023-03-03T08:43:29.354271+0000","last_unstale":"2023-03-03T08:43:43.981298+0000","last_undegraded":"2023-03-03T08:43:43.981298+0000","last_fullsized":"2023-03-03T08:43:43.981298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:32:05.946053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979432+0000","last_change":"2023-03-03T08:43:43.575567+0000","last_active":"2023-03-03T08:43:44.979432+0000","last_peered":"2023-03-03T08:43:44.979432+0000","last_clean":"2023-03-03T08:43:44.979432+0000","last_became_active":"2023-03-03T08:43:43.575276+0000","last_became_peered":"2023-03-03T08:43:43.575276+0000","last_unstale":"2023-03-03T08:43:44.979432+0000","last_undegraded":"2023-03-03T08:43:44.979432+0000","last_fullsized":"2023-03-03T08:43:44.979432+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:02:12.876790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107250+0000","last_change":"2023-03-03T08:43:43.998276+0000","last_active":"2023-03-03T08:43:45.107250+0000","last_peered":"2023-03-03T08:43:45.107250+0000","last_clean":"2023-03-03T08:43:45.107250+0000","last_became_active":"2023-03-03T08:43:43.996018+0000","last_became_peered":"2023-03-03T08:43:43.996018+0000","last_unstale":"2023-03-03T08:43:45.107250+0000","last_undegraded":"2023-03-03T08:43:45.107250+0000","last_fullsized":"2023-03-03T08:43:45.107250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:49:08.588278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988844+0000","last_change":"2023-03-03T08:43:34.539018+0000","last_active":"2023-03-03T08:43:43.988844+0000","last_peered":"2023-03-03T08:43:43.988844+0000","last_clean":"2023-03-03T08:43:43.988844+0000","last_became_active":"2023-03-03T08:43:34.538375+0000","last_became_peered":"2023-03-03T08:43:34.538375+0000","last_unstale":"2023-03-03T08:43:43.988844+0000","last_undegraded":"2023-03-03T08:43:43.988844+0000","last_fullsized":"2023-03-03T08:43:43.988844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:29:02.601410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990482+0000","last_change":"2023-03-03T08:43:29.354519+0000","last_active":"2023-03-03T08:43:43.990482+0000","last_peered":"2023-03-03T08:43:43.990482+0000","last_clean":"2023-03-03T08:43:43.990482+0000","last_became_active":"2023-03-03T08:43:29.354213+0000","last_became_peered":"2023-03-03T08:43:29.354213+0000","last_unstale":"2023-03-03T08:43:43.990482+0000","last_undegraded":"2023-03-03T08:43:43.990482+0000","last_fullsized":"2023-03-03T08:43:43.990482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:00.406507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990028+0000","last_change":"2023-03-03T08:43:34.110105+0000","last_active":"2023-03-03T08:43:43.990028+0000","last_peered":"2023-03-03T08:43:43.990028+0000","last_clean":"2023-03-03T08:43:43.990028+0000","last_became_active":"2023-03-03T08:43:34.109736+0000","last_became_peered":"2023-03-03T08:43:34.109736+0000","last_unstale":"2023-03-03T08:43:43.990028+0000","last_undegraded":"2023-03-03T08:43:43.990028+0000","last_fullsized":"2023-03-03T08:43:43.990028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:30.557847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105361+0000","last_change":"2023-03-03T08:43:43.569433+0000","last_active":"2023-03-03T08:43:45.105361+0000","last_peered":"2023-03-03T08:43:45.105361+0000","last_clean":"2023-03-03T08:43:45.105361+0000","last_became_active":"2023-03-03T08:43:43.568711+0000","last_became_peered":"2023-03-03T08:43:43.568711+0000","last_unstale":"2023-03-03T08:43:45.105361+0000","last_undegraded":"2023-03-03T08:43:45.105361+0000","last_fullsized":"2023-03-03T08:43:45.105361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:53.170557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105548+0000","last_change":"2023-03-03T08:43:43.999241+0000","last_active":"2023-03-03T08:43:45.105548+0000","last_peered":"2023-03-03T08:43:45.105548+0000","last_clean":"2023-03-03T08:43:45.105548+0000","last_became_active":"2023-03-03T08:43:43.998992+0000","last_became_peered":"2023-03-03T08:43:43.998992+0000","last_unstale":"2023-03-03T08:43:45.105548+0000","last_undegraded":"2023-03-03T08:43:45.105548+0000","last_fullsized":"2023-03-03T08:43:45.105548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:21:22.623700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991092+0000","last_change":"2023-03-03T08:43:34.535435+0000","last_active":"2023-03-03T08:43:43.991092+0000","last_peered":"2023-03-03T08:43:43.991092+0000","last_clean":"2023-03-03T08:43:43.991092+0000","last_became_active":"2023-03-03T08:43:34.535123+0000","last_became_peered":"2023-03-03T08:43:34.535123+0000","last_unstale":"2023-03-03T08:43:43.991092+0000","last_undegraded":"2023-03-03T08:43:43.991092+0000","last_fullsized":"2023-03-03T08:43:43.991092+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:52:10.622422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990432+0000","last_change":"2023-03-03T08:43:34.107783+0000","last_active":"2023-03-03T08:43:43.990432+0000","last_peered":"2023-03-03T08:43:43.990432+0000","last_clean":"2023-03-03T08:43:43.990432+0000","last_became_active":"2023-03-03T08:43:34.107503+0000","last_became_peered":"2023-03-03T08:43:34.107503+0000","last_unstale":"2023-03-03T08:43:43.990432+0000","last_undegraded":"2023-03-03T08:43:43.990432+0000","last_fullsized":"2023-03-03T08:43:43.990432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:35:42.800814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981038+0000","last_change":"2023-03-03T08:43:29.072304+0000","last_active":"2023-03-03T08:43:43.981038+0000","last_peered":"2023-03-03T08:43:43.981038+0000","last_clean":"2023-03-03T08:43:43.981038+0000","last_became_active":"2023-03-03T08:43:29.071962+0000","last_became_peered":"2023-03-03T08:43:29.071962+0000","last_unstale":"2023-03-03T08:43:43.981038+0000","last_undegraded":"2023-03-03T08:43:43.981038+0000","last_fullsized":"2023-03-03T08:43:43.981038+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:40:12.229226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106349+0000","last_change":"2023-03-03T08:43:43.575964+0000","last_active":"2023-03-03T08:43:45.106349+0000","last_peered":"2023-03-03T08:43:45.106349+0000","last_clean":"2023-03-03T08:43:45.106349+0000","last_became_active":"2023-03-03T08:43:43.575697+0000","last_became_peered":"2023-03-03T08:43:43.575697+0000","last_unstale":"2023-03-03T08:43:45.106349+0000","last_undegraded":"2023-03-03T08:43:45.106349+0000","last_fullsized":"2023-03-03T08:43:45.106349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:53:01.896050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978782+0000","last_change":"2023-03-03T08:43:44.006414+0000","last_active":"2023-03-03T08:43:44.978782+0000","last_peered":"2023-03-03T08:43:44.978782+0000","last_clean":"2023-03-03T08:43:44.978782+0000","last_became_active":"2023-03-03T08:43:44.006132+0000","last_became_peered":"2023-03-03T08:43:44.006132+0000","last_unstale":"2023-03-03T08:43:44.978782+0000","last_undegraded":"2023-03-03T08:43:44.978782+0000","last_fullsized":"2023-03-03T08:43:44.978782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:38:55.006419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982772+0000","last_change":"2023-03-03T08:43:34.544053+0000","last_active":"2023-03-03T08:43:43.982772+0000","last_peered":"2023-03-03T08:43:43.982772+0000","last_clean":"2023-03-03T08:43:43.982772+0000","last_became_active":"2023-03-03T08:43:34.543523+0000","last_became_peered":"2023-03-03T08:43:34.543523+0000","last_unstale":"2023-03-03T08:43:43.982772+0000","last_undegraded":"2023-03-03T08:43:43.982772+0000","last_fullsized":"2023-03-03T08:43:43.982772+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:34:03.662535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989069+0000","last_change":"2023-03-03T08:43:34.104492+0000","last_active":"2023-03-03T08:43:43.989069+0000","last_peered":"2023-03-03T08:43:43.989069+0000","last_clean":"2023-03-03T08:43:43.989069+0000","last_became_active":"2023-03-03T08:43:34.104151+0000","last_became_peered":"2023-03-03T08:43:34.104151+0000","last_unstale":"2023-03-03T08:43:43.989069+0000","last_undegraded":"2023-03-03T08:43:43.989069+0000","last_fullsized":"2023-03-03T08:43:43.989069+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:05:57.214738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979395+0000","last_change":"2023-03-03T08:43:29.075836+0000","last_active":"2023-03-03T08:43:43.979395+0000","last_peered":"2023-03-03T08:43:43.979395+0000","last_clean":"2023-03-03T08:43:43.979395+0000","last_became_active":"2023-03-03T08:43:29.075564+0000","last_became_peered":"2023-03-03T08:43:29.075564+0000","last_unstale":"2023-03-03T08:43:43.979395+0000","last_undegraded":"2023-03-03T08:43:43.979395+0000","last_fullsized":"2023-03-03T08:43:43.979395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:51:39.576714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107224+0000","last_change":"2023-03-03T08:43:43.575599+0000","last_active":"2023-03-03T08:43:45.107224+0000","last_peered":"2023-03-03T08:43:45.107224+0000","last_clean":"2023-03-03T08:43:45.107224+0000","last_became_active":"2023-03-03T08:43:43.575318+0000","last_became_peered":"2023-03-03T08:43:43.575318+0000","last_unstale":"2023-03-03T08:43:45.107224+0000","last_undegraded":"2023-03-03T08:43:45.107224+0000","last_fullsized":"2023-03-03T08:43:45.107224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:03:28.806052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105269+0000","last_change":"2023-03-03T08:43:44.000500+0000","last_active":"2023-03-03T08:43:45.105269+0000","last_peered":"2023-03-03T08:43:45.105269+0000","last_clean":"2023-03-03T08:43:45.105269+0000","last_became_active":"2023-03-03T08:43:43.999560+0000","last_became_peered":"2023-03-03T08:43:43.999560+0000","last_unstale":"2023-03-03T08:43:45.105269+0000","last_undegraded":"2023-03-03T08:43:45.105269+0000","last_fullsized":"2023-03-03T08:43:45.105269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:29:54.240309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981172+0000","last_change":"2023-03-03T08:43:34.550263+0000","last_active":"2023-03-03T08:43:43.981172+0000","last_peered":"2023-03-03T08:43:43.981172+0000","last_clean":"2023-03-03T08:43:43.981172+0000","last_became_active":"2023-03-03T08:43:34.549560+0000","last_became_peered":"2023-03-03T08:43:34.549560+0000","last_unstale":"2023-03-03T08:43:43.981172+0000","last_undegraded":"2023-03-03T08:43:43.981172+0000","last_fullsized":"2023-03-03T08:43:43.981172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:47:55.874223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987062+0000","last_change":"2023-03-03T08:43:34.110484+0000","last_active":"2023-03-03T08:43:43.987062+0000","last_peered":"2023-03-03T08:43:43.987062+0000","last_clean":"2023-03-03T08:43:43.987062+0000","last_became_active":"2023-03-03T08:43:34.110220+0000","last_became_peered":"2023-03-03T08:43:34.110220+0000","last_unstale":"2023-03-03T08:43:43.987062+0000","last_undegraded":"2023-03-03T08:43:43.987062+0000","last_fullsized":"2023-03-03T08:43:43.987062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:04:59.232537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982439+0000","last_change":"2023-03-03T08:43:23.015460+0000","last_active":"2023-03-03T08:43:43.982439+0000","last_peered":"2023-03-03T08:43:43.982439+0000","last_clean":"2023-03-03T08:43:43.982439+0000","last_became_active":"2023-03-03T08:43:23.014485+0000","last_became_peered":"2023-03-03T08:43:23.014485+0000","last_unstale":"2023-03-03T08:43:43.982439+0000","last_undegraded":"2023-03-03T08:43:43.982439+0000","last_fullsized":"2023-03-03T08:43:43.982439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:21.542883+0000","last_clean_scrub_stamp":"2023-03-03T08:43:21.542883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:09:07.022017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980803+0000","last_change":"2023-03-03T08:43:29.160285+0000","last_active":"2023-03-03T08:43:43.980803+0000","last_peered":"2023-03-03T08:43:43.980803+0000","last_clean":"2023-03-03T08:43:43.980803+0000","last_became_active":"2023-03-03T08:43:29.159943+0000","last_became_peered":"2023-03-03T08:43:29.159943+0000","last_unstale":"2023-03-03T08:43:43.980803+0000","last_undegraded":"2023-03-03T08:43:43.980803+0000","last_fullsized":"2023-03-03T08:43:43.980803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:48:58.997356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107580+0000","last_change":"2023-03-03T08:43:43.575212+0000","last_active":"2023-03-03T08:43:45.107580+0000","last_peered":"2023-03-03T08:43:45.107580+0000","last_clean":"2023-03-03T08:43:45.107580+0000","last_became_active":"2023-03-03T08:43:43.574897+0000","last_became_peered":"2023-03-03T08:43:43.574897+0000","last_unstale":"2023-03-03T08:43:45.107580+0000","last_undegraded":"2023-03-03T08:43:45.107580+0000","last_fullsized":"2023-03-03T08:43:45.107580+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:33:07.674313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107370+0000","last_change":"2023-03-03T08:43:43.992952+0000","last_active":"2023-03-03T08:43:45.107370+0000","last_peered":"2023-03-03T08:43:45.107370+0000","last_clean":"2023-03-03T08:43:45.107370+0000","last_became_active":"2023-03-03T08:43:43.992535+0000","last_became_peered":"2023-03-03T08:43:43.992535+0000","last_unstale":"2023-03-03T08:43:45.107370+0000","last_undegraded":"2023-03-03T08:43:45.107370+0000","last_fullsized":"2023-03-03T08:43:45.107370+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:10:41.617430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991335+0000","last_change":"2023-03-03T08:43:34.536419+0000","last_active":"2023-03-03T08:43:43.991335+0000","last_peered":"2023-03-03T08:43:43.991335+0000","last_clean":"2023-03-03T08:43:43.991335+0000","last_became_active":"2023-03-03T08:43:34.536150+0000","last_became_peered":"2023-03-03T08:43:34.536150+0000","last_unstale":"2023-03-03T08:43:43.991335+0000","last_undegraded":"2023-03-03T08:43:43.991335+0000","last_fullsized":"2023-03-03T08:43:43.991335+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:22:54.362024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981827+0000","last_change":"2023-03-03T08:43:34.106337+0000","last_active":"2023-03-03T08:43:43.981827+0000","last_peered":"2023-03-03T08:43:43.981827+0000","last_clean":"2023-03-03T08:43:43.981827+0000","last_became_active":"2023-03-03T08:43:34.105753+0000","last_became_peered":"2023-03-03T08:43:34.105753+0000","last_unstale":"2023-03-03T08:43:43.981827+0000","last_undegraded":"2023-03-03T08:43:43.981827+0000","last_fullsized":"2023-03-03T08:43:43.981827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:08:46.761924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991590+0000","last_change":"2023-03-03T08:43:29.076367+0000","last_active":"2023-03-03T08:43:43.991590+0000","last_peered":"2023-03-03T08:43:43.991590+0000","last_clean":"2023-03-03T08:43:43.991590+0000","last_became_active":"2023-03-03T08:43:29.075982+0000","last_became_peered":"2023-03-03T08:43:29.075982+0000","last_unstale":"2023-03-03T08:43:43.991590+0000","last_undegraded":"2023-03-03T08:43:43.991590+0000","last_fullsized":"2023-03-03T08:43:43.991590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:41:06.873363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:46.143717+0000","last_change":"2023-03-03T08:43:43.577670+0000","last_active":"2023-03-03T08:43:46.143717+0000","last_peered":"2023-03-03T08:43:46.143717+0000","last_clean":"2023-03-03T08:43:46.143717+0000","last_became_active":"2023-03-03T08:43:43.577371+0000","last_became_peered":"2023-03-03T08:43:43.577371+0000","last_unstale":"2023-03-03T08:43:46.143717+0000","last_undegraded":"2023-03-03T08:43:46.143717+0000","last_fullsized":"2023-03-03T08:43:46.143717+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:44:18.377654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108041+0000","last_change":"2023-03-03T08:43:43.993077+0000","last_active":"2023-03-03T08:43:45.108041+0000","last_peered":"2023-03-03T08:43:45.108041+0000","last_clean":"2023-03-03T08:43:45.108041+0000","last_became_active":"2023-03-03T08:43:43.992021+0000","last_became_peered":"2023-03-03T08:43:43.992021+0000","last_unstale":"2023-03-03T08:43:45.108041+0000","last_undegraded":"2023-03-03T08:43:45.108041+0000","last_fullsized":"2023-03-03T08:43:45.108041+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:58.237553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990193+0000","last_change":"2023-03-03T08:43:34.528593+0000","last_active":"2023-03-03T08:43:43.990193+0000","last_peered":"2023-03-03T08:43:43.990193+0000","last_clean":"2023-03-03T08:43:43.990193+0000","last_became_active":"2023-03-03T08:43:34.528260+0000","last_became_peered":"2023-03-03T08:43:34.528260+0000","last_unstale":"2023-03-03T08:43:43.990193+0000","last_undegraded":"2023-03-03T08:43:43.990193+0000","last_fullsized":"2023-03-03T08:43:43.990193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988924+0000","last_change":"2023-03-03T08:43:34.113748+0000","last_active":"2023-03-03T08:43:43.988924+0000","last_peered":"2023-03-03T08:43:43.988924+0000","last_clean":"2023-03-03T08:43:43.988924+0000","last_became_active":"2023-03-03T08:43:34.113190+0000","last_became_peered":"2023-03-03T08:43:34.113190+0000","last_unstale":"2023-03-03T08:43:43.988924+0000","last_undegraded":"2023-03-03T08:43:43.988924+0000","last_fullsized":"2023-03-03T08:43:43.988924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:58:55.343571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980485+0000","last_change":"2023-03-03T08:43:29.107584+0000","last_active":"2023-03-03T08:43:43.980485+0000","last_peered":"2023-03-03T08:43:43.980485+0000","last_clean":"2023-03-03T08:43:43.980485+0000","last_became_active":"2023-03-03T08:43:29.107208+0000","last_became_peered":"2023-03-03T08:43:29.107208+0000","last_unstale":"2023-03-03T08:43:43.980485+0000","last_undegraded":"2023-03-03T08:43:43.980485+0000","last_fullsized":"2023-03-03T08:43:43.980485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:28:24.087122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106004+0000","last_change":"2023-03-03T08:43:43.575742+0000","last_active":"2023-03-03T08:43:45.106004+0000","last_peered":"2023-03-03T08:43:45.106004+0000","last_clean":"2023-03-03T08:43:45.106004+0000","last_became_active":"2023-03-03T08:43:43.575456+0000","last_became_peered":"2023-03-03T08:43:43.575456+0000","last_unstale":"2023-03-03T08:43:45.106004+0000","last_undegraded":"2023-03-03T08:43:45.106004+0000","last_fullsized":"2023-03-03T08:43:45.106004+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:46:17.635306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106635+0000","last_change":"2023-03-03T08:43:43.985961+0000","last_active":"2023-03-03T08:43:45.106635+0000","last_peered":"2023-03-03T08:43:45.106635+0000","last_clean":"2023-03-03T08:43:45.106635+0000","last_became_active":"2023-03-03T08:43:43.985633+0000","last_became_peered":"2023-03-03T08:43:43.985633+0000","last_unstale":"2023-03-03T08:43:45.106635+0000","last_undegraded":"2023-03-03T08:43:45.106635+0000","last_fullsized":"2023-03-03T08:43:45.106635+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:52.517910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989789+0000","last_change":"2023-03-03T08:43:34.526797+0000","last_active":"2023-03-03T08:43:43.989789+0000","last_peered":"2023-03-03T08:43:43.989789+0000","last_clean":"2023-03-03T08:43:43.989789+0000","last_became_active":"2023-03-03T08:43:34.526490+0000","last_became_peered":"2023-03-03T08:43:34.526490+0000","last_unstale":"2023-03-03T08:43:43.989789+0000","last_undegraded":"2023-03-03T08:43:43.989789+0000","last_fullsized":"2023-03-03T08:43:43.989789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:46:11.808336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986004+0000","last_change":"2023-03-03T08:43:34.104692+0000","last_active":"2023-03-03T08:43:43.986004+0000","last_peered":"2023-03-03T08:43:43.986004+0000","last_clean":"2023-03-03T08:43:43.986004+0000","last_became_active":"2023-03-03T08:43:34.104263+0000","last_became_peered":"2023-03-03T08:43:34.104263+0000","last_unstale":"2023-03-03T08:43:43.986004+0000","last_undegraded":"2023-03-03T08:43:43.986004+0000","last_fullsized":"2023-03-03T08:43:43.986004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:18:24.537703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983049+0000","last_change":"2023-03-03T08:43:29.107536+0000","last_active":"2023-03-03T08:43:43.983049+0000","last_peered":"2023-03-03T08:43:43.983049+0000","last_clean":"2023-03-03T08:43:43.983049+0000","last_became_active":"2023-03-03T08:43:29.107093+0000","last_became_peered":"2023-03-03T08:43:29.107093+0000","last_unstale":"2023-03-03T08:43:43.983049+0000","last_undegraded":"2023-03-03T08:43:43.983049+0000","last_fullsized":"2023-03-03T08:43:43.983049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:28.060706+0000","last_clean_scrub_stamp":"2023-03-03T08:43:28.060706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:25:47.568904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980025+0000","last_change":"2023-03-03T08:43:43.577221+0000","last_active":"2023-03-03T08:43:44.980025+0000","last_peered":"2023-03-03T08:43:44.980025+0000","last_clean":"2023-03-03T08:43:44.980025+0000","last_became_active":"2023-03-03T08:43:43.576942+0000","last_became_peered":"2023-03-03T08:43:43.576942+0000","last_unstale":"2023-03-03T08:43:44.980025+0000","last_undegraded":"2023-03-03T08:43:44.980025+0000","last_fullsized":"2023-03-03T08:43:44.980025+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:13:09.716078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105102+0000","last_change":"2023-03-03T08:43:44.000318+0000","last_active":"2023-03-03T08:43:45.105102+0000","last_peered":"2023-03-03T08:43:45.105102+0000","last_clean":"2023-03-03T08:43:45.105102+0000","last_became_active":"2023-03-03T08:43:44.000053+0000","last_became_peered":"2023-03-03T08:43:44.000053+0000","last_unstale":"2023-03-03T08:43:45.105102+0000","last_undegraded":"2023-03-03T08:43:45.105102+0000","last_fullsized":"2023-03-03T08:43:45.105102+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:39:31.059774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994113+0000","last_change":"2023-03-03T08:43:34.540006+0000","last_active":"2023-03-03T08:43:43.994113+0000","last_peered":"2023-03-03T08:43:43.994113+0000","last_clean":"2023-03-03T08:43:43.994113+0000","last_became_active":"2023-03-03T08:43:34.539579+0000","last_became_peered":"2023-03-03T08:43:34.539579+0000","last_unstale":"2023-03-03T08:43:43.994113+0000","last_undegraded":"2023-03-03T08:43:43.994113+0000","last_fullsized":"2023-03-03T08:43:43.994113+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988985+0000","last_change":"2023-03-03T08:43:34.109764+0000","last_active":"2023-03-03T08:43:43.988985+0000","last_peered":"2023-03-03T08:43:43.988985+0000","last_clean":"2023-03-03T08:43:43.988985+0000","last_became_active":"2023-03-03T08:43:34.109462+0000","last_became_peered":"2023-03-03T08:43:34.109462+0000","last_unstale":"2023-03-03T08:43:43.988985+0000","last_undegraded":"2023-03-03T08:43:43.988985+0000","last_fullsized":"2023-03-03T08:43:43.988985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:14:56.399192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106570+0000","last_change":"2023-03-03T08:43:43.576402+0000","last_active":"2023-03-03T08:43:45.106570+0000","last_peered":"2023-03-03T08:43:45.106570+0000","last_clean":"2023-03-03T08:43:45.106570+0000","last_became_active":"2023-03-03T08:43:43.576113+0000","last_became_peered":"2023-03-03T08:43:43.576113+0000","last_unstale":"2023-03-03T08:43:45.106570+0000","last_undegraded":"2023-03-03T08:43:45.106570+0000","last_fullsized":"2023-03-03T08:43:45.106570+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:33:26.483688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106437+0000","last_change":"2023-03-03T08:43:44.002074+0000","last_active":"2023-03-03T08:43:45.106437+0000","last_peered":"2023-03-03T08:43:45.106437+0000","last_clean":"2023-03-03T08:43:45.106437+0000","last_became_active":"2023-03-03T08:43:44.001803+0000","last_became_peered":"2023-03-03T08:43:44.001803+0000","last_unstale":"2023-03-03T08:43:45.106437+0000","last_undegraded":"2023-03-03T08:43:45.106437+0000","last_fullsized":"2023-03-03T08:43:45.106437+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:15:16.115135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986358+0000","last_change":"2023-03-03T08:43:34.541858+0000","last_active":"2023-03-03T08:43:43.986358+0000","last_peered":"2023-03-03T08:43:43.986358+0000","last_clean":"2023-03-03T08:43:43.986358+0000","last_became_active":"2023-03-03T08:43:34.541430+0000","last_became_peered":"2023-03-03T08:43:34.541430+0000","last_unstale":"2023-03-03T08:43:43.986358+0000","last_undegraded":"2023-03-03T08:43:43.986358+0000","last_fullsized":"2023-03-03T08:43:43.986358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:06:29.397784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979830+0000","last_change":"2023-03-03T08:43:34.111335+0000","last_active":"2023-03-03T08:43:43.979830+0000","last_peered":"2023-03-03T08:43:43.979830+0000","last_clean":"2023-03-03T08:43:43.979830+0000","last_became_active":"2023-03-03T08:43:34.110837+0000","last_became_peered":"2023-03-03T08:43:34.110837+0000","last_unstale":"2023-03-03T08:43:43.979830+0000","last_undegraded":"2023-03-03T08:43:43.979830+0000","last_fullsized":"2023-03-03T08:43:43.979830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:03.843512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106574+0000","last_change":"2023-03-03T08:43:43.577539+0000","last_active":"2023-03-03T08:43:45.106574+0000","last_peered":"2023-03-03T08:43:45.106574+0000","last_clean":"2023-03-03T08:43:45.106574+0000","last_became_active":"2023-03-03T08:43:43.577209+0000","last_became_peered":"2023-03-03T08:43:43.577209+0000","last_unstale":"2023-03-03T08:43:45.106574+0000","last_undegraded":"2023-03-03T08:43:45.106574+0000","last_fullsized":"2023-03-03T08:43:45.106574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:31:01.430495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979296+0000","last_change":"2023-03-03T08:43:43.993565+0000","last_active":"2023-03-03T08:43:44.979296+0000","last_peered":"2023-03-03T08:43:44.979296+0000","last_clean":"2023-03-03T08:43:44.979296+0000","last_became_active":"2023-03-03T08:43:43.992653+0000","last_became_peered":"2023-03-03T08:43:43.992653+0000","last_unstale":"2023-03-03T08:43:44.979296+0000","last_undegraded":"2023-03-03T08:43:44.979296+0000","last_fullsized":"2023-03-03T08:43:44.979296+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:24.295462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981121+0000","last_change":"2023-03-03T08:43:34.550325+0000","last_active":"2023-03-03T08:43:43.981121+0000","last_peered":"2023-03-03T08:43:43.981121+0000","last_clean":"2023-03-03T08:43:43.981121+0000","last_became_active":"2023-03-03T08:43:34.550041+0000","last_became_peered":"2023-03-03T08:43:34.550041+0000","last_unstale":"2023-03-03T08:43:43.981121+0000","last_undegraded":"2023-03-03T08:43:43.981121+0000","last_fullsized":"2023-03-03T08:43:43.981121+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:44:34.082003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980457+0000","last_change":"2023-03-03T08:43:34.106777+0000","last_active":"2023-03-03T08:43:43.980457+0000","last_peered":"2023-03-03T08:43:43.980457+0000","last_clean":"2023-03-03T08:43:43.980457+0000","last_became_active":"2023-03-03T08:43:34.106132+0000","last_became_peered":"2023-03-03T08:43:34.106132+0000","last_unstale":"2023-03-03T08:43:43.980457+0000","last_undegraded":"2023-03-03T08:43:43.980457+0000","last_fullsized":"2023-03-03T08:43:43.980457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:19:55.965987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106224+0000","last_change":"2023-03-03T08:43:43.580312+0000","last_active":"2023-03-03T08:43:45.106224+0000","last_peered":"2023-03-03T08:43:45.106224+0000","last_clean":"2023-03-03T08:43:45.106224+0000","last_became_active":"2023-03-03T08:43:43.580096+0000","last_became_peered":"2023-03-03T08:43:43.580096+0000","last_unstale":"2023-03-03T08:43:45.106224+0000","last_undegraded":"2023-03-03T08:43:45.106224+0000","last_fullsized":"2023-03-03T08:43:45.106224+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:06:03.500778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106765+0000","last_change":"2023-03-03T08:43:43.997453+0000","last_active":"2023-03-03T08:43:45.106765+0000","last_peered":"2023-03-03T08:43:45.106765+0000","last_clean":"2023-03-03T08:43:45.106765+0000","last_became_active":"2023-03-03T08:43:43.997090+0000","last_became_peered":"2023-03-03T08:43:43.997090+0000","last_unstale":"2023-03-03T08:43:45.106765+0000","last_undegraded":"2023-03-03T08:43:45.106765+0000","last_fullsized":"2023-03-03T08:43:45.106765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:28:58.322823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985493+0000","last_change":"2023-03-03T08:43:34.542308+0000","last_active":"2023-03-03T08:43:43.985493+0000","last_peered":"2023-03-03T08:43:43.985493+0000","last_clean":"2023-03-03T08:43:43.985493+0000","last_became_active":"2023-03-03T08:43:34.541991+0000","last_became_peered":"2023-03-03T08:43:34.541991+0000","last_unstale":"2023-03-03T08:43:43.985493+0000","last_undegraded":"2023-03-03T08:43:43.985493+0000","last_fullsized":"2023-03-03T08:43:43.985493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:06:21.851231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992335+0000","last_change":"2023-03-03T08:43:34.109698+0000","last_active":"2023-03-03T08:43:43.992335+0000","last_peered":"2023-03-03T08:43:43.992335+0000","last_clean":"2023-03-03T08:43:43.992335+0000","last_became_active":"2023-03-03T08:43:34.109423+0000","last_became_peered":"2023-03-03T08:43:34.109423+0000","last_unstale":"2023-03-03T08:43:43.992335+0000","last_undegraded":"2023-03-03T08:43:43.992335+0000","last_fullsized":"2023-03-03T08:43:43.992335+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:07:49.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105911+0000","last_change":"2023-03-03T08:43:43.573925+0000","last_active":"2023-03-03T08:43:45.105911+0000","last_peered":"2023-03-03T08:43:45.105911+0000","last_clean":"2023-03-03T08:43:45.105911+0000","last_became_active":"2023-03-03T08:43:43.573474+0000","last_became_peered":"2023-03-03T08:43:43.573474+0000","last_unstale":"2023-03-03T08:43:45.105911+0000","last_undegraded":"2023-03-03T08:43:45.105911+0000","last_fullsized":"2023-03-03T08:43:45.105911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:47.858726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107653+0000","last_change":"2023-03-03T08:43:43.995602+0000","last_active":"2023-03-03T08:43:45.107653+0000","last_peered":"2023-03-03T08:43:45.107653+0000","last_clean":"2023-03-03T08:43:45.107653+0000","last_became_active":"2023-03-03T08:43:43.995404+0000","last_became_peered":"2023-03-03T08:43:43.995404+0000","last_unstale":"2023-03-03T08:43:45.107653+0000","last_undegraded":"2023-03-03T08:43:45.107653+0000","last_fullsized":"2023-03-03T08:43:45.107653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:37.212119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982423+0000","last_change":"2023-03-03T08:43:34.548304+0000","last_active":"2023-03-03T08:43:43.982423+0000","last_peered":"2023-03-03T08:43:43.982423+0000","last_clean":"2023-03-03T08:43:43.982423+0000","last_became_active":"2023-03-03T08:43:34.547960+0000","last_became_peered":"2023-03-03T08:43:34.547960+0000","last_unstale":"2023-03-03T08:43:43.982423+0000","last_undegraded":"2023-03-03T08:43:43.982423+0000","last_fullsized":"2023-03-03T08:43:43.982423+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:45:04.907753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985574+0000","last_change":"2023-03-03T08:43:34.109372+0000","last_active":"2023-03-03T08:43:43.985574+0000","last_peered":"2023-03-03T08:43:43.985574+0000","last_clean":"2023-03-03T08:43:43.985574+0000","last_became_active":"2023-03-03T08:43:34.109023+0000","last_became_peered":"2023-03-03T08:43:34.109023+0000","last_unstale":"2023-03-03T08:43:43.985574+0000","last_undegraded":"2023-03-03T08:43:43.985574+0000","last_fullsized":"2023-03-03T08:43:43.985574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:27:51.374188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978866+0000","last_change":"2023-03-03T08:43:43.572240+0000","last_active":"2023-03-03T08:43:44.978866+0000","last_peered":"2023-03-03T08:43:44.978866+0000","last_clean":"2023-03-03T08:43:44.978866+0000","last_became_active":"2023-03-03T08:43:43.571753+0000","last_became_peered":"2023-03-03T08:43:43.571753+0000","last_unstale":"2023-03-03T08:43:44.978866+0000","last_undegraded":"2023-03-03T08:43:44.978866+0000","last_fullsized":"2023-03-03T08:43:44.978866+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:02:30.521667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979027+0000","last_change":"2023-03-03T08:43:43.995018+0000","last_active":"2023-03-03T08:43:44.979027+0000","last_peered":"2023-03-03T08:43:44.979027+0000","last_clean":"2023-03-03T08:43:44.979027+0000","last_became_active":"2023-03-03T08:43:43.994690+0000","last_became_peered":"2023-03-03T08:43:43.994690+0000","last_unstale":"2023-03-03T08:43:44.979027+0000","last_undegraded":"2023-03-03T08:43:44.979027+0000","last_fullsized":"2023-03-03T08:43:44.979027+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:06:13.979929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986022+0000","last_change":"2023-03-03T08:43:34.538817+0000","last_active":"2023-03-03T08:43:43.986022+0000","last_peered":"2023-03-03T08:43:43.986022+0000","last_clean":"2023-03-03T08:43:43.986022+0000","last_became_active":"2023-03-03T08:43:34.538506+0000","last_became_peered":"2023-03-03T08:43:34.538506+0000","last_unstale":"2023-03-03T08:43:43.986022+0000","last_undegraded":"2023-03-03T08:43:43.986022+0000","last_fullsized":"2023-03-03T08:43:43.986022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980930+0000","last_change":"2023-03-03T08:43:34.117734+0000","last_active":"2023-03-03T08:43:43.980930+0000","last_peered":"2023-03-03T08:43:43.980930+0000","last_clean":"2023-03-03T08:43:43.980930+0000","last_became_active":"2023-03-03T08:43:34.117454+0000","last_became_peered":"2023-03-03T08:43:34.117454+0000","last_unstale":"2023-03-03T08:43:43.980930+0000","last_undegraded":"2023-03-03T08:43:43.980930+0000","last_fullsized":"2023-03-03T08:43:43.980930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:42:27.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106434+0000","last_change":"2023-03-03T08:43:43.574955+0000","last_active":"2023-03-03T08:43:45.106434+0000","last_peered":"2023-03-03T08:43:45.106434+0000","last_clean":"2023-03-03T08:43:45.106434+0000","last_became_active":"2023-03-03T08:43:43.574677+0000","last_became_peered":"2023-03-03T08:43:43.574677+0000","last_unstale":"2023-03-03T08:43:45.106434+0000","last_undegraded":"2023-03-03T08:43:45.106434+0000","last_fullsized":"2023-03-03T08:43:45.106434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:12:19.538291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.104954+0000","last_change":"2023-03-03T08:43:43.994888+0000","last_active":"2023-03-03T08:43:45.104954+0000","last_peered":"2023-03-03T08:43:45.104954+0000","last_clean":"2023-03-03T08:43:45.104954+0000","last_became_active":"2023-03-03T08:43:43.994630+0000","last_became_peered":"2023-03-03T08:43:43.994630+0000","last_unstale":"2023-03-03T08:43:45.104954+0000","last_undegraded":"2023-03-03T08:43:45.104954+0000","last_fullsized":"2023-03-03T08:43:45.104954+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:15:18.412569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982579+0000","last_change":"2023-03-03T08:43:34.548010+0000","last_active":"2023-03-03T08:43:43.982579+0000","last_peered":"2023-03-03T08:43:43.982579+0000","last_clean":"2023-03-03T08:43:43.982579+0000","last_became_active":"2023-03-03T08:43:34.547662+0000","last_became_peered":"2023-03-03T08:43:34.547662+0000","last_unstale":"2023-03-03T08:43:43.982579+0000","last_undegraded":"2023-03-03T08:43:43.982579+0000","last_fullsized":"2023-03-03T08:43:43.982579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:52.200064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986286+0000","last_change":"2023-03-03T08:43:34.109890+0000","last_active":"2023-03-03T08:43:43.986286+0000","last_peered":"2023-03-03T08:43:43.986286+0000","last_clean":"2023-03-03T08:43:43.986286+0000","last_became_active":"2023-03-03T08:43:34.109583+0000","last_became_peered":"2023-03-03T08:43:34.109583+0000","last_unstale":"2023-03-03T08:43:43.986286+0000","last_undegraded":"2023-03-03T08:43:43.986286+0000","last_fullsized":"2023-03-03T08:43:43.986286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:27:34.142294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978233+0000","last_change":"2023-03-03T08:43:43.575945+0000","last_active":"2023-03-03T08:43:44.978233+0000","last_peered":"2023-03-03T08:43:44.978233+0000","last_clean":"2023-03-03T08:43:44.978233+0000","last_became_active":"2023-03-03T08:43:43.575675+0000","last_became_peered":"2023-03-03T08:43:43.575675+0000","last_unstale":"2023-03-03T08:43:44.978233+0000","last_undegraded":"2023-03-03T08:43:44.978233+0000","last_fullsized":"2023-03-03T08:43:44.978233+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:47:48.730805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980103+0000","last_change":"2023-03-03T08:43:43.998435+0000","last_active":"2023-03-03T08:43:44.980103+0000","last_peered":"2023-03-03T08:43:44.980103+0000","last_clean":"2023-03-03T08:43:44.980103+0000","last_became_active":"2023-03-03T08:43:43.996508+0000","last_became_peered":"2023-03-03T08:43:43.996508+0000","last_unstale":"2023-03-03T08:43:44.980103+0000","last_undegraded":"2023-03-03T08:43:44.980103+0000","last_fullsized":"2023-03-03T08:43:44.980103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:31:36.310851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979477+0000","last_change":"2023-03-03T08:43:34.540196+0000","last_active":"2023-03-03T08:43:43.979477+0000","last_peered":"2023-03-03T08:43:43.979477+0000","last_clean":"2023-03-03T08:43:43.979477+0000","last_became_active":"2023-03-03T08:43:34.539931+0000","last_became_peered":"2023-03-03T08:43:34.539931+0000","last_unstale":"2023-03-03T08:43:43.979477+0000","last_undegraded":"2023-03-03T08:43:43.979477+0000","last_fullsized":"2023-03-03T08:43:43.979477+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:20.012997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990881+0000","last_change":"2023-03-03T08:43:34.110464+0000","last_active":"2023-03-03T08:43:43.990881+0000","last_peered":"2023-03-03T08:43:43.990881+0000","last_clean":"2023-03-03T08:43:43.990881+0000","last_became_active":"2023-03-03T08:43:34.110165+0000","last_became_peered":"2023-03-03T08:43:34.110165+0000","last_unstale":"2023-03-03T08:43:43.990881+0000","last_undegraded":"2023-03-03T08:43:43.990881+0000","last_fullsized":"2023-03-03T08:43:43.990881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:42:54.363804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106139+0000","last_change":"2023-03-03T08:43:43.578298+0000","last_active":"2023-03-03T08:43:45.106139+0000","last_peered":"2023-03-03T08:43:45.106139+0000","last_clean":"2023-03-03T08:43:45.106139+0000","last_became_active":"2023-03-03T08:43:43.578024+0000","last_became_peered":"2023-03-03T08:43:43.578024+0000","last_unstale":"2023-03-03T08:43:45.106139+0000","last_undegraded":"2023-03-03T08:43:45.106139+0000","last_fullsized":"2023-03-03T08:43:45.106139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:00:44.633716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106526+0000","last_change":"2023-03-03T08:43:43.992906+0000","last_active":"2023-03-03T08:43:45.106526+0000","last_peered":"2023-03-03T08:43:45.106526+0000","last_clean":"2023-03-03T08:43:45.106526+0000","last_became_active":"2023-03-03T08:43:43.991823+0000","last_became_peered":"2023-03-03T08:43:43.991823+0000","last_unstale":"2023-03-03T08:43:45.106526+0000","last_undegraded":"2023-03-03T08:43:45.106526+0000","last_fullsized":"2023-03-03T08:43:45.106526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:34.431216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.553597+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.553322+0000","last_became_peered":"2023-03-03T08:43:34.553322+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:24:17.829622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983377+0000","last_change":"2023-03-03T08:43:34.108123+0000","last_active":"2023-03-03T08:43:43.983377+0000","last_peered":"2023-03-03T08:43:43.983377+0000","last_clean":"2023-03-03T08:43:43.983377+0000","last_became_active":"2023-03-03T08:43:34.107757+0000","last_became_peered":"2023-03-03T08:43:34.107757+0000","last_unstale":"2023-03-03T08:43:43.983377+0000","last_undegraded":"2023-03-03T08:43:43.983377+0000","last_fullsized":"2023-03-03T08:43:43.983377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:14:09.488930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106648+0000","last_change":"2023-03-03T08:43:43.573369+0000","last_active":"2023-03-03T08:43:45.106648+0000","last_peered":"2023-03-03T08:43:45.106648+0000","last_clean":"2023-03-03T08:43:45.106648+0000","last_became_active":"2023-03-03T08:43:43.573069+0000","last_became_peered":"2023-03-03T08:43:43.573069+0000","last_unstale":"2023-03-03T08:43:45.106648+0000","last_undegraded":"2023-03-03T08:43:45.106648+0000","last_fullsized":"2023-03-03T08:43:45.106648+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:09.574926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107410+0000","last_change":"2023-03-03T08:43:43.995725+0000","last_active":"2023-03-03T08:43:45.107410+0000","last_peered":"2023-03-03T08:43:45.107410+0000","last_clean":"2023-03-03T08:43:45.107410+0000","last_became_active":"2023-03-03T08:43:43.993875+0000","last_became_peered":"2023-03-03T08:43:43.993875+0000","last_unstale":"2023-03-03T08:43:45.107410+0000","last_undegraded":"2023-03-03T08:43:45.107410+0000","last_fullsized":"2023-03-03T08:43:45.107410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:03:38.439539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982673+0000","last_change":"2023-03-03T08:43:34.546067+0000","last_active":"2023-03-03T08:43:43.982673+0000","last_peered":"2023-03-03T08:43:43.982673+0000","last_clean":"2023-03-03T08:43:43.982673+0000","last_became_active":"2023-03-03T08:43:34.543786+0000","last_became_peered":"2023-03-03T08:43:34.543786+0000","last_unstale":"2023-03-03T08:43:43.982673+0000","last_undegraded":"2023-03-03T08:43:43.982673+0000","last_fullsized":"2023-03-03T08:43:43.982673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:29:32.922812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.978650+0000","last_change":"2023-03-03T08:43:34.115792+0000","last_active":"2023-03-03T08:43:43.978650+0000","last_peered":"2023-03-03T08:43:43.978650+0000","last_clean":"2023-03-03T08:43:43.978650+0000","last_became_active":"2023-03-03T08:43:34.115431+0000","last_became_peered":"2023-03-03T08:43:34.115431+0000","last_unstale":"2023-03-03T08:43:43.978650+0000","last_undegraded":"2023-03-03T08:43:43.978650+0000","last_fullsized":"2023-03-03T08:43:43.978650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:50:15.484022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980510+0000","last_change":"2023-03-03T08:43:43.579691+0000","last_active":"2023-03-03T08:43:44.980510+0000","last_peered":"2023-03-03T08:43:44.980510+0000","last_clean":"2023-03-03T08:43:44.980510+0000","last_became_active":"2023-03-03T08:43:43.579425+0000","last_became_peered":"2023-03-03T08:43:43.579425+0000","last_unstale":"2023-03-03T08:43:44.980510+0000","last_undegraded":"2023-03-03T08:43:44.980510+0000","last_fullsized":"2023-03-03T08:43:44.980510+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:14:08.896591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979907+0000","last_change":"2023-03-03T08:43:44.006872+0000","last_active":"2023-03-03T08:43:44.979907+0000","last_peered":"2023-03-03T08:43:44.979907+0000","last_clean":"2023-03-03T08:43:44.979907+0000","last_became_active":"2023-03-03T08:43:44.006283+0000","last_became_peered":"2023-03-03T08:43:44.006283+0000","last_unstale":"2023-03-03T08:43:44.979907+0000","last_undegraded":"2023-03-03T08:43:44.979907+0000","last_fullsized":"2023-03-03T08:43:44.979907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:01:45.439399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981214+0000","last_change":"2023-03-03T08:43:34.532656+0000","last_active":"2023-03-03T08:43:43.981214+0000","last_peered":"2023-03-03T08:43:43.981214+0000","last_clean":"2023-03-03T08:43:43.981214+0000","last_became_active":"2023-03-03T08:43:34.532297+0000","last_became_peered":"2023-03-03T08:43:34.532297+0000","last_unstale":"2023-03-03T08:43:43.981214+0000","last_undegraded":"2023-03-03T08:43:43.981214+0000","last_fullsized":"2023-03-03T08:43:43.981214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:36.347740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.989578+0000","last_change":"2023-03-03T08:43:34.112905+0000","last_active":"2023-03-03T08:43:43.989578+0000","last_peered":"2023-03-03T08:43:43.989578+0000","last_clean":"2023-03-03T08:43:43.989578+0000","last_became_active":"2023-03-03T08:43:34.112601+0000","last_became_peered":"2023-03-03T08:43:34.112601+0000","last_unstale":"2023-03-03T08:43:43.989578+0000","last_undegraded":"2023-03-03T08:43:43.989578+0000","last_fullsized":"2023-03-03T08:43:43.989578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:01.165783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"27'9","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:44:01.020445+0000","last_change":"2023-03-03T08:43:43.577882+0000","last_active":"2023-03-03T08:44:01.020445+0000","last_peered":"2023-03-03T08:44:01.020445+0000","last_clean":"2023-03-03T08:44:01.020445+0000","last_became_active":"2023-03-03T08:43:43.575625+0000","last_became_peered":"2023-03-03T08:43:43.575625+0000","last_unstale":"2023-03-03T08:44:01.020445+0000","last_undegraded":"2023-03-03T08:44:01.020445+0000","last_fullsized":"2023-03-03T08:44:01.020445+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:22:25.919413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2779,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106494+0000","last_change":"2023-03-03T08:43:43.994846+0000","last_active":"2023-03-03T08:43:45.106494+0000","last_peered":"2023-03-03T08:43:45.106494+0000","last_clean":"2023-03-03T08:43:45.106494+0000","last_became_active":"2023-03-03T08:43:43.994652+0000","last_became_peered":"2023-03-03T08:43:43.994652+0000","last_unstale":"2023-03-03T08:43:45.106494+0000","last_undegraded":"2023-03-03T08:43:45.106494+0000","last_fullsized":"2023-03-03T08:43:45.106494+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:54:29.238192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990352+0000","last_change":"2023-03-03T08:43:34.539790+0000","last_active":"2023-03-03T08:43:43.990352+0000","last_peered":"2023-03-03T08:43:43.990352+0000","last_clean":"2023-03-03T08:43:43.990352+0000","last_became_active":"2023-03-03T08:43:34.539443+0000","last_became_peered":"2023-03-03T08:43:34.539443+0000","last_unstale":"2023-03-03T08:43:43.990352+0000","last_undegraded":"2023-03-03T08:43:43.990352+0000","last_fullsized":"2023-03-03T08:43:43.990352+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:36:25.786904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980897+0000","last_change":"2023-03-03T08:43:34.112941+0000","last_active":"2023-03-03T08:43:43.980897+0000","last_peered":"2023-03-03T08:43:43.980897+0000","last_clean":"2023-03-03T08:43:43.980897+0000","last_became_active":"2023-03-03T08:43:34.112646+0000","last_became_peered":"2023-03-03T08:43:34.112646+0000","last_unstale":"2023-03-03T08:43:43.980897+0000","last_undegraded":"2023-03-03T08:43:43.980897+0000","last_fullsized":"2023-03-03T08:43:43.980897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:20:32.767710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107854+0000","last_change":"2023-03-03T08:43:43.576421+0000","last_active":"2023-03-03T08:43:45.107854+0000","last_peered":"2023-03-03T08:43:45.107854+0000","last_clean":"2023-03-03T08:43:45.107854+0000","last_became_active":"2023-03-03T08:43:43.576086+0000","last_became_peered":"2023-03-03T08:43:43.576086+0000","last_unstale":"2023-03-03T08:43:45.107854+0000","last_undegraded":"2023-03-03T08:43:45.107854+0000","last_fullsized":"2023-03-03T08:43:45.107854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:57:17.814222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980193+0000","last_change":"2023-03-03T08:43:43.995591+0000","last_active":"2023-03-03T08:43:44.980193+0000","last_peered":"2023-03-03T08:43:44.980193+0000","last_clean":"2023-03-03T08:43:44.980193+0000","last_became_active":"2023-03-03T08:43:43.995338+0000","last_became_peered":"2023-03-03T08:43:43.995338+0000","last_unstale":"2023-03-03T08:43:44.980193+0000","last_undegraded":"2023-03-03T08:43:44.980193+0000","last_fullsized":"2023-03-03T08:43:44.980193+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:17:56.617398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985713+0000","last_change":"2023-03-03T08:43:34.537857+0000","last_active":"2023-03-03T08:43:43.985713+0000","last_peered":"2023-03-03T08:43:43.985713+0000","last_clean":"2023-03-03T08:43:43.985713+0000","last_became_active":"2023-03-03T08:43:34.537590+0000","last_became_peered":"2023-03-03T08:43:34.537590+0000","last_unstale":"2023-03-03T08:43:43.985713+0000","last_undegraded":"2023-03-03T08:43:43.985713+0000","last_fullsized":"2023-03-03T08:43:43.985713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:52.612521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"27'9","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:58.500261+0000","last_change":"2023-03-03T08:43:34.117609+0000","last_active":"2023-03-03T08:43:58.500261+0000","last_peered":"2023-03-03T08:43:58.500261+0000","last_clean":"2023-03-03T08:43:58.500261+0000","last_became_active":"2023-03-03T08:43:34.117271+0000","last_became_peered":"2023-03-03T08:43:34.117271+0000","last_unstale":"2023-03-03T08:43:58.500261+0000","last_undegraded":"2023-03-03T08:43:58.500261+0000","last_fullsized":"2023-03-03T08:43:58.500261+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:53:20.259259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2779,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978772+0000","last_change":"2023-03-03T08:43:43.580489+0000","last_active":"2023-03-03T08:43:44.978772+0000","last_peered":"2023-03-03T08:43:44.978772+0000","last_clean":"2023-03-03T08:43:44.978772+0000","last_became_active":"2023-03-03T08:43:43.580207+0000","last_became_peered":"2023-03-03T08:43:43.580207+0000","last_unstale":"2023-03-03T08:43:44.978772+0000","last_undegraded":"2023-03-03T08:43:44.978772+0000","last_fullsized":"2023-03-03T08:43:44.978772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:58:05.519074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978633+0000","last_change":"2023-03-03T08:43:43.993832+0000","last_active":"2023-03-03T08:43:44.978633+0000","last_peered":"2023-03-03T08:43:44.978633+0000","last_clean":"2023-03-03T08:43:44.978633+0000","last_became_active":"2023-03-03T08:43:43.993524+0000","last_became_peered":"2023-03-03T08:43:43.993524+0000","last_unstale":"2023-03-03T08:43:44.978633+0000","last_undegraded":"2023-03-03T08:43:44.978633+0000","last_fullsized":"2023-03-03T08:43:44.978633+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:23:55.761785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987159+0000","last_change":"2023-03-03T08:43:34.544694+0000","last_active":"2023-03-03T08:43:43.987159+0000","last_peered":"2023-03-03T08:43:43.987159+0000","last_clean":"2023-03-03T08:43:43.987159+0000","last_became_active":"2023-03-03T08:43:34.544399+0000","last_became_peered":"2023-03-03T08:43:34.544399+0000","last_unstale":"2023-03-03T08:43:43.987159+0000","last_undegraded":"2023-03-03T08:43:43.987159+0000","last_fullsized":"2023-03-03T08:43:43.987159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:25:22.115139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986196+0000","last_change":"2023-03-03T08:43:34.110810+0000","last_active":"2023-03-03T08:43:43.986196+0000","last_peered":"2023-03-03T08:43:43.986196+0000","last_clean":"2023-03-03T08:43:43.986196+0000","last_became_active":"2023-03-03T08:43:34.110483+0000","last_became_peered":"2023-03-03T08:43:34.110483+0000","last_unstale":"2023-03-03T08:43:43.986196+0000","last_undegraded":"2023-03-03T08:43:43.986196+0000","last_fullsized":"2023-03-03T08:43:43.986196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:11:06.970615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979989+0000","last_change":"2023-03-03T08:43:43.576669+0000","last_active":"2023-03-03T08:43:44.979989+0000","last_peered":"2023-03-03T08:43:44.979989+0000","last_clean":"2023-03-03T08:43:44.979989+0000","last_became_active":"2023-03-03T08:43:43.573885+0000","last_became_peered":"2023-03-03T08:43:43.573885+0000","last_unstale":"2023-03-03T08:43:44.979989+0000","last_undegraded":"2023-03-03T08:43:44.979989+0000","last_fullsized":"2023-03-03T08:43:44.979989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:56:12.054118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979442+0000","last_change":"2023-03-03T08:43:43.996790+0000","last_active":"2023-03-03T08:43:44.979442+0000","last_peered":"2023-03-03T08:43:44.979442+0000","last_clean":"2023-03-03T08:43:44.979442+0000","last_became_active":"2023-03-03T08:43:43.996366+0000","last_became_peered":"2023-03-03T08:43:43.996366+0000","last_unstale":"2023-03-03T08:43:44.979442+0000","last_undegraded":"2023-03-03T08:43:44.979442+0000","last_fullsized":"2023-03-03T08:43:44.979442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:01.662221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988553+0000","last_change":"2023-03-03T08:43:34.538703+0000","last_active":"2023-03-03T08:43:43.988553+0000","last_peered":"2023-03-03T08:43:43.988553+0000","last_clean":"2023-03-03T08:43:43.988553+0000","last_became_active":"2023-03-03T08:43:34.538418+0000","last_became_peered":"2023-03-03T08:43:34.538418+0000","last_unstale":"2023-03-03T08:43:43.988553+0000","last_undegraded":"2023-03-03T08:43:43.988553+0000","last_fullsized":"2023-03-03T08:43:43.988553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:55:02.795203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986185+0000","last_change":"2023-03-03T08:43:34.113168+0000","last_active":"2023-03-03T08:43:43.986185+0000","last_peered":"2023-03-03T08:43:43.986185+0000","last_clean":"2023-03-03T08:43:43.986185+0000","last_became_active":"2023-03-03T08:43:34.112731+0000","last_became_peered":"2023-03-03T08:43:34.112731+0000","last_unstale":"2023-03-03T08:43:43.986185+0000","last_undegraded":"2023-03-03T08:43:43.986185+0000","last_fullsized":"2023-03-03T08:43:43.986185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:57:59.463191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106795+0000","last_change":"2023-03-03T08:43:43.577965+0000","last_active":"2023-03-03T08:43:45.106795+0000","last_peered":"2023-03-03T08:43:45.106795+0000","last_clean":"2023-03-03T08:43:45.106795+0000","last_became_active":"2023-03-03T08:43:43.577587+0000","last_became_peered":"2023-03-03T08:43:43.577587+0000","last_unstale":"2023-03-03T08:43:45.106795+0000","last_undegraded":"2023-03-03T08:43:45.106795+0000","last_fullsized":"2023-03-03T08:43:45.106795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:56:24.196773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105366+0000","last_change":"2023-03-03T08:43:43.998210+0000","last_active":"2023-03-03T08:43:45.105366+0000","last_peered":"2023-03-03T08:43:45.105366+0000","last_clean":"2023-03-03T08:43:45.105366+0000","last_became_active":"2023-03-03T08:43:43.997845+0000","last_became_peered":"2023-03-03T08:43:43.997845+0000","last_unstale":"2023-03-03T08:43:45.105366+0000","last_undegraded":"2023-03-03T08:43:45.105366+0000","last_fullsized":"2023-03-03T08:43:45.105366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:57:38.226245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991762+0000","last_change":"2023-03-03T08:43:34.528573+0000","last_active":"2023-03-03T08:43:43.991762+0000","last_peered":"2023-03-03T08:43:43.991762+0000","last_clean":"2023-03-03T08:43:43.991762+0000","last_became_active":"2023-03-03T08:43:34.528278+0000","last_became_peered":"2023-03-03T08:43:34.528278+0000","last_unstale":"2023-03-03T08:43:43.991762+0000","last_undegraded":"2023-03-03T08:43:43.991762+0000","last_fullsized":"2023-03-03T08:43:43.991762+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:37:51.468323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.110802+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.110519+0000","last_became_peered":"2023-03-03T08:43:34.110519+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:02:18.355043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105602+0000","last_change":"2023-03-03T08:43:43.573655+0000","last_active":"2023-03-03T08:43:45.105602+0000","last_peered":"2023-03-03T08:43:45.105602+0000","last_clean":"2023-03-03T08:43:45.105602+0000","last_became_active":"2023-03-03T08:43:43.573162+0000","last_became_peered":"2023-03-03T08:43:43.573162+0000","last_unstale":"2023-03-03T08:43:45.105602+0000","last_undegraded":"2023-03-03T08:43:45.105602+0000","last_fullsized":"2023-03-03T08:43:45.105602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:59:04.568700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106617+0000","last_change":"2023-03-03T08:43:43.993431+0000","last_active":"2023-03-03T08:43:45.106617+0000","last_peered":"2023-03-03T08:43:45.106617+0000","last_clean":"2023-03-03T08:43:45.106617+0000","last_became_active":"2023-03-03T08:43:43.993166+0000","last_became_peered":"2023-03-03T08:43:43.993166+0000","last_unstale":"2023-03-03T08:43:45.106617+0000","last_undegraded":"2023-03-03T08:43:45.106617+0000","last_fullsized":"2023-03-03T08:43:45.106617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:04:52.880667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979475+0000","last_change":"2023-03-03T08:43:43.992992+0000","last_active":"2023-03-03T08:43:44.979475+0000","last_peered":"2023-03-03T08:43:44.979475+0000","last_clean":"2023-03-03T08:43:44.979475+0000","last_became_active":"2023-03-03T08:43:43.992664+0000","last_became_peered":"2023-03-03T08:43:43.992664+0000","last_unstale":"2023-03-03T08:43:44.979475+0000","last_undegraded":"2023-03-03T08:43:44.979475+0000","last_fullsized":"2023-03-03T08:43:44.979475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:49:07.276715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.981207+0000","last_change":"2023-03-03T08:43:34.547450+0000","last_active":"2023-03-03T08:43:43.981207+0000","last_peered":"2023-03-03T08:43:43.981207+0000","last_clean":"2023-03-03T08:43:43.981207+0000","last_became_active":"2023-03-03T08:43:34.547168+0000","last_became_peered":"2023-03-03T08:43:34.547168+0000","last_unstale":"2023-03-03T08:43:43.981207+0000","last_undegraded":"2023-03-03T08:43:43.981207+0000","last_fullsized":"2023-03-03T08:43:43.981207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:33:17.723229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982025+0000","last_change":"2023-03-03T08:43:34.106505+0000","last_active":"2023-03-03T08:43:43.982025+0000","last_peered":"2023-03-03T08:43:43.982025+0000","last_clean":"2023-03-03T08:43:43.982025+0000","last_became_active":"2023-03-03T08:43:34.106079+0000","last_became_peered":"2023-03-03T08:43:34.106079+0000","last_unstale":"2023-03-03T08:43:43.982025+0000","last_undegraded":"2023-03-03T08:43:43.982025+0000","last_fullsized":"2023-03-03T08:43:43.982025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:48:48.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979698+0000","last_change":"2023-03-03T08:43:43.575448+0000","last_active":"2023-03-03T08:43:44.979698+0000","last_peered":"2023-03-03T08:43:44.979698+0000","last_clean":"2023-03-03T08:43:44.979698+0000","last_became_active":"2023-03-03T08:43:43.575142+0000","last_became_peered":"2023-03-03T08:43:43.575142+0000","last_unstale":"2023-03-03T08:43:44.979698+0000","last_undegraded":"2023-03-03T08:43:44.979698+0000","last_fullsized":"2023-03-03T08:43:44.979698+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:42:09.308616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990396+0000","last_change":"2023-03-03T08:43:34.549123+0000","last_active":"2023-03-03T08:43:43.990396+0000","last_peered":"2023-03-03T08:43:43.990396+0000","last_clean":"2023-03-03T08:43:43.990396+0000","last_became_active":"2023-03-03T08:43:34.548839+0000","last_became_peered":"2023-03-03T08:43:34.548839+0000","last_unstale":"2023-03-03T08:43:43.990396+0000","last_undegraded":"2023-03-03T08:43:43.990396+0000","last_fullsized":"2023-03-03T08:43:43.990396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:32:38.134009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984959+0000","last_change":"2023-03-03T08:43:34.108187+0000","last_active":"2023-03-03T08:43:43.984959+0000","last_peered":"2023-03-03T08:43:43.984959+0000","last_clean":"2023-03-03T08:43:43.984959+0000","last_became_active":"2023-03-03T08:43:34.107859+0000","last_became_peered":"2023-03-03T08:43:34.107859+0000","last_unstale":"2023-03-03T08:43:43.984959+0000","last_undegraded":"2023-03-03T08:43:43.984959+0000","last_fullsized":"2023-03-03T08:43:43.984959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:46:40.595756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107953+0000","last_change":"2023-03-03T08:43:43.572738+0000","last_active":"2023-03-03T08:43:45.107953+0000","last_peered":"2023-03-03T08:43:45.107953+0000","last_clean":"2023-03-03T08:43:45.107953+0000","last_became_active":"2023-03-03T08:43:43.572435+0000","last_became_peered":"2023-03-03T08:43:43.572435+0000","last_unstale":"2023-03-03T08:43:45.107953+0000","last_undegraded":"2023-03-03T08:43:45.107953+0000","last_fullsized":"2023-03-03T08:43:45.107953+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:14.264653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105420+0000","last_change":"2023-03-03T08:43:43.995021+0000","last_active":"2023-03-03T08:43:45.105420+0000","last_peered":"2023-03-03T08:43:45.105420+0000","last_clean":"2023-03-03T08:43:45.105420+0000","last_became_active":"2023-03-03T08:43:43.994766+0000","last_became_peered":"2023-03-03T08:43:43.994766+0000","last_unstale":"2023-03-03T08:43:45.105420+0000","last_undegraded":"2023-03-03T08:43:45.105420+0000","last_fullsized":"2023-03-03T08:43:45.105420+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:14:03.827057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979582+0000","last_change":"2023-03-03T08:43:43.993391+0000","last_active":"2023-03-03T08:43:44.979582+0000","last_peered":"2023-03-03T08:43:44.979582+0000","last_clean":"2023-03-03T08:43:44.979582+0000","last_became_active":"2023-03-03T08:43:43.993087+0000","last_became_peered":"2023-03-03T08:43:43.993087+0000","last_unstale":"2023-03-03T08:43:44.979582+0000","last_undegraded":"2023-03-03T08:43:44.979582+0000","last_fullsized":"2023-03-03T08:43:44.979582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:13:01.193811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992180+0000","last_change":"2023-03-03T08:43:34.537301+0000","last_active":"2023-03-03T08:43:43.992180+0000","last_peered":"2023-03-03T08:43:43.992180+0000","last_clean":"2023-03-03T08:43:43.992180+0000","last_became_active":"2023-03-03T08:43:34.537007+0000","last_became_peered":"2023-03-03T08:43:34.537007+0000","last_unstale":"2023-03-03T08:43:43.992180+0000","last_undegraded":"2023-03-03T08:43:43.992180+0000","last_fullsized":"2023-03-03T08:43:43.992180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:59:49.124140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982700+0000","last_change":"2023-03-03T08:43:34.108056+0000","last_active":"2023-03-03T08:43:43.982700+0000","last_peered":"2023-03-03T08:43:43.982700+0000","last_clean":"2023-03-03T08:43:43.982700+0000","last_became_active":"2023-03-03T08:43:34.107708+0000","last_became_peered":"2023-03-03T08:43:34.107708+0000","last_unstale":"2023-03-03T08:43:43.982700+0000","last_undegraded":"2023-03-03T08:43:43.982700+0000","last_fullsized":"2023-03-03T08:43:43.982700+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:33.082757+0000","last_clean_scrub_stamp":"2023-03-03T08:43:33.082757+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:10.652147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107314+0000","last_change":"2023-03-03T08:43:43.575788+0000","last_active":"2023-03-03T08:43:45.107314+0000","last_peered":"2023-03-03T08:43:45.107314+0000","last_clean":"2023-03-03T08:43:45.107314+0000","last_became_active":"2023-03-03T08:43:43.575507+0000","last_became_peered":"2023-03-03T08:43:43.575507+0000","last_unstale":"2023-03-03T08:43:45.107314+0000","last_undegraded":"2023-03-03T08:43:45.107314+0000","last_fullsized":"2023-03-03T08:43:45.107314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:42.544007+0000","last_clean_scrub_stamp":"2023-03-03T08:43:42.544007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:15.843816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107877+0000","last_change":"2023-03-03T08:43:43.994916+0000","last_active":"2023-03-03T08:43:45.107877+0000","last_peered":"2023-03-03T08:43:45.107877+0000","last_clean":"2023-03-03T08:43:45.107877+0000","last_became_active":"2023-03-03T08:43:43.994639+0000","last_became_peered":"2023-03-03T08:43:43.994639+0000","last_unstale":"2023-03-03T08:43:45.107877+0000","last_undegraded":"2023-03-03T08:43:45.107877+0000","last_fullsized":"2023-03-03T08:43:45.107877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:06:52.526052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980834+0000","last_change":"2023-03-03T08:43:34.533987+0000","last_active":"2023-03-03T08:43:43.980834+0000","last_peered":"2023-03-03T08:43:43.980834+0000","last_clean":"2023-03-03T08:43:43.980834+0000","last_became_active":"2023-03-03T08:43:34.533729+0000","last_became_peered":"2023-03-03T08:43:34.533729+0000","last_unstale":"2023-03-03T08:43:43.980834+0000","last_undegraded":"2023-03-03T08:43:43.980834+0000","last_fullsized":"2023-03-03T08:43:43.980834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:18:12.407151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106398+0000","last_change":"2023-03-03T08:43:43.996808+0000","last_active":"2023-03-03T08:43:45.106398+0000","last_peered":"2023-03-03T08:43:45.106398+0000","last_clean":"2023-03-03T08:43:45.106398+0000","last_became_active":"2023-03-03T08:43:43.996559+0000","last_became_peered":"2023-03-03T08:43:43.996559+0000","last_unstale":"2023-03-03T08:43:45.106398+0000","last_undegraded":"2023-03-03T08:43:45.106398+0000","last_fullsized":"2023-03-03T08:43:45.106398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:57:01.223908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.993213+0000","last_change":"2023-03-03T08:43:34.540762+0000","last_active":"2023-03-03T08:43:43.993213+0000","last_peered":"2023-03-03T08:43:43.993213+0000","last_clean":"2023-03-03T08:43:43.993213+0000","last_became_active":"2023-03-03T08:43:34.540467+0000","last_became_peered":"2023-03-03T08:43:34.540467+0000","last_unstale":"2023-03-03T08:43:43.993213+0000","last_undegraded":"2023-03-03T08:43:43.993213+0000","last_fullsized":"2023-03-03T08:43:43.993213+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:47:11.894605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107993+0000","last_change":"2023-03-03T08:43:43.995286+0000","last_active":"2023-03-03T08:43:45.107993+0000","last_peered":"2023-03-03T08:43:45.107993+0000","last_clean":"2023-03-03T08:43:45.107993+0000","last_became_active":"2023-03-03T08:43:43.994861+0000","last_became_peered":"2023-03-03T08:43:43.994861+0000","last_unstale":"2023-03-03T08:43:45.107993+0000","last_undegraded":"2023-03-03T08:43:45.107993+0000","last_fullsized":"2023-03-03T08:43:45.107993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:14:31.687301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990722+0000","last_change":"2023-03-03T08:43:34.542858+0000","last_active":"2023-03-03T08:43:43.990722+0000","last_peered":"2023-03-03T08:43:43.990722+0000","last_clean":"2023-03-03T08:43:43.990722+0000","last_became_active":"2023-03-03T08:43:34.542510+0000","last_became_peered":"2023-03-03T08:43:34.542510+0000","last_unstale":"2023-03-03T08:43:43.990722+0000","last_undegraded":"2023-03-03T08:43:43.990722+0000","last_fullsized":"2023-03-03T08:43:43.990722+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:51:21.471234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106711+0000","last_change":"2023-03-03T08:43:43.992084+0000","last_active":"2023-03-03T08:43:45.106711+0000","last_peered":"2023-03-03T08:43:45.106711+0000","last_clean":"2023-03-03T08:43:45.106711+0000","last_became_active":"2023-03-03T08:43:43.991465+0000","last_became_peered":"2023-03-03T08:43:43.991465+0000","last_unstale":"2023-03-03T08:43:45.106711+0000","last_undegraded":"2023-03-03T08:43:45.106711+0000","last_fullsized":"2023-03-03T08:43:45.106711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:33:19.530923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983571+0000","last_change":"2023-03-03T08:43:34.548469+0000","last_active":"2023-03-03T08:43:43.983571+0000","last_peered":"2023-03-03T08:43:43.983571+0000","last_clean":"2023-03-03T08:43:43.983571+0000","last_became_active":"2023-03-03T08:43:34.547647+0000","last_became_peered":"2023-03-03T08:43:34.547647+0000","last_unstale":"2023-03-03T08:43:43.983571+0000","last_undegraded":"2023-03-03T08:43:43.983571+0000","last_fullsized":"2023-03-03T08:43:43.983571+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:29:40.054781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107161+0000","last_change":"2023-03-03T08:43:43.998061+0000","last_active":"2023-03-03T08:43:45.107161+0000","last_peered":"2023-03-03T08:43:45.107161+0000","last_clean":"2023-03-03T08:43:45.107161+0000","last_became_active":"2023-03-03T08:43:43.997770+0000","last_became_peered":"2023-03-03T08:43:43.997770+0000","last_unstale":"2023-03-03T08:43:45.107161+0000","last_undegraded":"2023-03-03T08:43:45.107161+0000","last_fullsized":"2023-03-03T08:43:45.107161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:46:17.053610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.988635+0000","last_change":"2023-03-03T08:43:34.549771+0000","last_active":"2023-03-03T08:43:43.988635+0000","last_peered":"2023-03-03T08:43:43.988635+0000","last_clean":"2023-03-03T08:43:43.988635+0000","last_became_active":"2023-03-03T08:43:34.549479+0000","last_became_peered":"2023-03-03T08:43:34.549479+0000","last_unstale":"2023-03-03T08:43:43.988635+0000","last_undegraded":"2023-03-03T08:43:43.988635+0000","last_fullsized":"2023-03-03T08:43:43.988635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:41:03.902019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979024+0000","last_change":"2023-03-03T08:43:43.999391+0000","last_active":"2023-03-03T08:43:44.979024+0000","last_peered":"2023-03-03T08:43:44.979024+0000","last_clean":"2023-03-03T08:43:44.979024+0000","last_became_active":"2023-03-03T08:43:43.998904+0000","last_became_peered":"2023-03-03T08:43:43.998904+0000","last_unstale":"2023-03-03T08:43:44.979024+0000","last_undegraded":"2023-03-03T08:43:44.979024+0000","last_fullsized":"2023-03-03T08:43:44.979024+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:43:17.429776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.992365+0000","last_change":"2023-03-03T08:43:34.542661+0000","last_active":"2023-03-03T08:43:43.992365+0000","last_peered":"2023-03-03T08:43:43.992365+0000","last_clean":"2023-03-03T08:43:43.992365+0000","last_became_active":"2023-03-03T08:43:34.542353+0000","last_became_peered":"2023-03-03T08:43:34.542353+0000","last_unstale":"2023-03-03T08:43:43.992365+0000","last_undegraded":"2023-03-03T08:43:43.992365+0000","last_fullsized":"2023-03-03T08:43:43.992365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:35:38.730662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105103+0000","last_change":"2023-03-03T08:43:43.996729+0000","last_active":"2023-03-03T08:43:45.105103+0000","last_peered":"2023-03-03T08:43:45.105103+0000","last_clean":"2023-03-03T08:43:45.105103+0000","last_became_active":"2023-03-03T08:43:43.986535+0000","last_became_peered":"2023-03-03T08:43:43.986535+0000","last_unstale":"2023-03-03T08:43:45.105103+0000","last_undegraded":"2023-03-03T08:43:45.105103+0000","last_fullsized":"2023-03-03T08:43:45.105103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:59:22.505305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982406+0000","last_change":"2023-03-03T08:43:34.549837+0000","last_active":"2023-03-03T08:43:43.982406+0000","last_peered":"2023-03-03T08:43:43.982406+0000","last_clean":"2023-03-03T08:43:43.982406+0000","last_became_active":"2023-03-03T08:43:34.549542+0000","last_became_peered":"2023-03-03T08:43:34.549542+0000","last_unstale":"2023-03-03T08:43:43.982406+0000","last_undegraded":"2023-03-03T08:43:43.982406+0000","last_fullsized":"2023-03-03T08:43:43.982406+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:36:31.726735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106334+0000","last_change":"2023-03-03T08:43:43.993213+0000","last_active":"2023-03-03T08:43:45.106334+0000","last_peered":"2023-03-03T08:43:45.106334+0000","last_clean":"2023-03-03T08:43:45.106334+0000","last_became_active":"2023-03-03T08:43:43.992920+0000","last_became_peered":"2023-03-03T08:43:43.992920+0000","last_unstale":"2023-03-03T08:43:45.106334+0000","last_undegraded":"2023-03-03T08:43:45.106334+0000","last_fullsized":"2023-03-03T08:43:45.106334+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:42.055599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982924+0000","last_change":"2023-03-03T08:43:34.547003+0000","last_active":"2023-03-03T08:43:43.982924+0000","last_peered":"2023-03-03T08:43:43.982924+0000","last_clean":"2023-03-03T08:43:43.982924+0000","last_became_active":"2023-03-03T08:43:34.546744+0000","last_became_peered":"2023-03-03T08:43:34.546744+0000","last_unstale":"2023-03-03T08:43:43.982924+0000","last_undegraded":"2023-03-03T08:43:43.982924+0000","last_fullsized":"2023-03-03T08:43:43.982924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:53:38.996605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106934+0000","last_change":"2023-03-03T08:43:43.994195+0000","last_active":"2023-03-03T08:43:45.106934+0000","last_peered":"2023-03-03T08:43:45.106934+0000","last_clean":"2023-03-03T08:43:45.106934+0000","last_became_active":"2023-03-03T08:43:43.993463+0000","last_became_peered":"2023-03-03T08:43:43.993463+0000","last_unstale":"2023-03-03T08:43:45.106934+0000","last_undegraded":"2023-03-03T08:43:45.106934+0000","last_fullsized":"2023-03-03T08:43:45.106934+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:52.365793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982677+0000","last_change":"2023-03-03T08:43:34.553589+0000","last_active":"2023-03-03T08:43:43.982677+0000","last_peered":"2023-03-03T08:43:43.982677+0000","last_clean":"2023-03-03T08:43:43.982677+0000","last_became_active":"2023-03-03T08:43:34.553242+0000","last_became_peered":"2023-03-03T08:43:34.553242+0000","last_unstale":"2023-03-03T08:43:43.982677+0000","last_undegraded":"2023-03-03T08:43:43.982677+0000","last_fullsized":"2023-03-03T08:43:43.982677+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:09.000051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107426+0000","last_change":"2023-03-03T08:43:44.004404+0000","last_active":"2023-03-03T08:43:45.107426+0000","last_peered":"2023-03-03T08:43:45.107426+0000","last_clean":"2023-03-03T08:43:45.107426+0000","last_became_active":"2023-03-03T08:43:44.004120+0000","last_became_peered":"2023-03-03T08:43:44.004120+0000","last_unstale":"2023-03-03T08:43:45.107426+0000","last_undegraded":"2023-03-03T08:43:45.107426+0000","last_fullsized":"2023-03-03T08:43:45.107426+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:34:36.006791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990661+0000","last_change":"2023-03-03T08:43:34.537110+0000","last_active":"2023-03-03T08:43:43.990661+0000","last_peered":"2023-03-03T08:43:43.990661+0000","last_clean":"2023-03-03T08:43:43.990661+0000","last_became_active":"2023-03-03T08:43:34.536848+0000","last_became_peered":"2023-03-03T08:43:34.536848+0000","last_unstale":"2023-03-03T08:43:43.990661+0000","last_undegraded":"2023-03-03T08:43:43.990661+0000","last_fullsized":"2023-03-03T08:43:43.990661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:13:12.425595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106366+0000","last_change":"2023-03-03T08:43:43.999266+0000","last_active":"2023-03-03T08:43:45.106366+0000","last_peered":"2023-03-03T08:43:45.106366+0000","last_clean":"2023-03-03T08:43:45.106366+0000","last_became_active":"2023-03-03T08:43:43.998929+0000","last_became_peered":"2023-03-03T08:43:43.998929+0000","last_unstale":"2023-03-03T08:43:45.106366+0000","last_undegraded":"2023-03-03T08:43:45.106366+0000","last_fullsized":"2023-03-03T08:43:45.106366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:02.236263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.994220+0000","last_change":"2023-03-03T08:43:34.532154+0000","last_active":"2023-03-03T08:43:43.994220+0000","last_peered":"2023-03-03T08:43:43.994220+0000","last_clean":"2023-03-03T08:43:43.994220+0000","last_became_active":"2023-03-03T08:43:34.531844+0000","last_became_peered":"2023-03-03T08:43:34.531844+0000","last_unstale":"2023-03-03T08:43:43.994220+0000","last_undegraded":"2023-03-03T08:43:43.994220+0000","last_fullsized":"2023-03-03T08:43:43.994220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:09:00.067605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106991+0000","last_change":"2023-03-03T08:43:44.004469+0000","last_active":"2023-03-03T08:43:45.106991+0000","last_peered":"2023-03-03T08:43:45.106991+0000","last_clean":"2023-03-03T08:43:45.106991+0000","last_became_active":"2023-03-03T08:43:44.004187+0000","last_became_peered":"2023-03-03T08:43:44.004187+0000","last_unstale":"2023-03-03T08:43:45.106991+0000","last_undegraded":"2023-03-03T08:43:45.106991+0000","last_fullsized":"2023-03-03T08:43:45.106991+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:16:35.298857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979308+0000","last_change":"2023-03-03T08:43:34.527373+0000","last_active":"2023-03-03T08:43:43.979308+0000","last_peered":"2023-03-03T08:43:43.979308+0000","last_clean":"2023-03-03T08:43:43.979308+0000","last_became_active":"2023-03-03T08:43:34.527171+0000","last_became_peered":"2023-03-03T08:43:34.527171+0000","last_unstale":"2023-03-03T08:43:43.979308+0000","last_undegraded":"2023-03-03T08:43:43.979308+0000","last_fullsized":"2023-03-03T08:43:43.979308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:20:51.138587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108220+0000","last_change":"2023-03-03T08:43:43.996329+0000","last_active":"2023-03-03T08:43:45.108220+0000","last_peered":"2023-03-03T08:43:45.108220+0000","last_clean":"2023-03-03T08:43:45.108220+0000","last_became_active":"2023-03-03T08:43:43.995055+0000","last_became_peered":"2023-03-03T08:43:43.995055+0000","last_unstale":"2023-03-03T08:43:45.108220+0000","last_undegraded":"2023-03-03T08:43:45.108220+0000","last_fullsized":"2023-03-03T08:43:45.108220+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:33:57.593740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.984695+0000","last_change":"2023-03-03T08:43:34.542729+0000","last_active":"2023-03-03T08:43:43.984695+0000","last_peered":"2023-03-03T08:43:43.984695+0000","last_clean":"2023-03-03T08:43:43.984695+0000","last_became_active":"2023-03-03T08:43:34.542246+0000","last_became_peered":"2023-03-03T08:43:34.542246+0000","last_unstale":"2023-03-03T08:43:43.984695+0000","last_undegraded":"2023-03-03T08:43:43.984695+0000","last_fullsized":"2023-03-03T08:43:43.984695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T13:14:59.995526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105365+0000","last_change":"2023-03-03T08:43:44.002358+0000","last_active":"2023-03-03T08:43:45.105365+0000","last_peered":"2023-03-03T08:43:45.105365+0000","last_clean":"2023-03-03T08:43:45.105365+0000","last_became_active":"2023-03-03T08:43:44.002020+0000","last_became_peered":"2023-03-03T08:43:44.002020+0000","last_unstale":"2023-03-03T08:43:45.105365+0000","last_undegraded":"2023-03-03T08:43:45.105365+0000","last_fullsized":"2023-03-03T08:43:45.105365+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:30:51.524774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982664+0000","last_change":"2023-03-03T08:43:34.549026+0000","last_active":"2023-03-03T08:43:43.982664+0000","last_peered":"2023-03-03T08:43:43.982664+0000","last_clean":"2023-03-03T08:43:43.982664+0000","last_became_active":"2023-03-03T08:43:34.548778+0000","last_became_peered":"2023-03-03T08:43:34.548778+0000","last_unstale":"2023-03-03T08:43:43.982664+0000","last_undegraded":"2023-03-03T08:43:43.982664+0000","last_fullsized":"2023-03-03T08:43:43.982664+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:41.835304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105281+0000","last_change":"2023-03-03T08:43:43.997684+0000","last_active":"2023-03-03T08:43:45.105281+0000","last_peered":"2023-03-03T08:43:45.105281+0000","last_clean":"2023-03-03T08:43:45.105281+0000","last_became_active":"2023-03-03T08:43:43.997503+0000","last_became_peered":"2023-03-03T08:43:43.997503+0000","last_unstale":"2023-03-03T08:43:45.105281+0000","last_undegraded":"2023-03-03T08:43:45.105281+0000","last_fullsized":"2023-03-03T08:43:45.105281+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:15:29.512524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.991178+0000","last_change":"2023-03-03T08:43:34.544013+0000","last_active":"2023-03-03T08:43:43.991178+0000","last_peered":"2023-03-03T08:43:43.991178+0000","last_clean":"2023-03-03T08:43:43.991178+0000","last_became_active":"2023-03-03T08:43:34.543745+0000","last_became_peered":"2023-03-03T08:43:34.543745+0000","last_unstale":"2023-03-03T08:43:43.991178+0000","last_undegraded":"2023-03-03T08:43:43.991178+0000","last_fullsized":"2023-03-03T08:43:43.991178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:47:05.647977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.107560+0000","last_change":"2023-03-03T08:43:43.999884+0000","last_active":"2023-03-03T08:43:45.107560+0000","last_peered":"2023-03-03T08:43:45.107560+0000","last_clean":"2023-03-03T08:43:45.107560+0000","last_became_active":"2023-03-03T08:43:43.999593+0000","last_became_peered":"2023-03-03T08:43:43.999593+0000","last_unstale":"2023-03-03T08:43:45.107560+0000","last_undegraded":"2023-03-03T08:43:45.107560+0000","last_fullsized":"2023-03-03T08:43:45.107560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:45:09.132974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982600+0000","last_change":"2023-03-03T08:43:34.553434+0000","last_active":"2023-03-03T08:43:43.982600+0000","last_peered":"2023-03-03T08:43:43.982600+0000","last_clean":"2023-03-03T08:43:43.982600+0000","last_became_active":"2023-03-03T08:43:34.553125+0000","last_became_peered":"2023-03-03T08:43:34.553125+0000","last_unstale":"2023-03-03T08:43:43.982600+0000","last_undegraded":"2023-03-03T08:43:43.982600+0000","last_fullsized":"2023-03-03T08:43:43.982600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:41:17.553506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105717+0000","last_change":"2023-03-03T08:43:44.006710+0000","last_active":"2023-03-03T08:43:45.105717+0000","last_peered":"2023-03-03T08:43:45.105717+0000","last_clean":"2023-03-03T08:43:45.105717+0000","last_became_active":"2023-03-03T08:43:44.006447+0000","last_became_peered":"2023-03-03T08:43:44.006447+0000","last_unstale":"2023-03-03T08:43:45.105717+0000","last_undegraded":"2023-03-03T08:43:45.105717+0000","last_fullsized":"2023-03-03T08:43:45.105717+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:32:21.767326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.982538+0000","last_change":"2023-03-03T08:43:34.550889+0000","last_active":"2023-03-03T08:43:43.982538+0000","last_peered":"2023-03-03T08:43:43.982538+0000","last_clean":"2023-03-03T08:43:43.982538+0000","last_became_active":"2023-03-03T08:43:34.550605+0000","last_became_peered":"2023-03-03T08:43:34.550605+0000","last_unstale":"2023-03-03T08:43:43.982538+0000","last_undegraded":"2023-03-03T08:43:43.982538+0000","last_fullsized":"2023-03-03T08:43:43.982538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T12:52:01.118555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105567+0000","last_change":"2023-03-03T08:43:43.998231+0000","last_active":"2023-03-03T08:43:45.105567+0000","last_peered":"2023-03-03T08:43:45.105567+0000","last_clean":"2023-03-03T08:43:45.105567+0000","last_became_active":"2023-03-03T08:43:43.997941+0000","last_became_peered":"2023-03-03T08:43:43.997941+0000","last_unstale":"2023-03-03T08:43:45.105567+0000","last_undegraded":"2023-03-03T08:43:45.105567+0000","last_fullsized":"2023-03-03T08:43:45.105567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:13:32.562793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983100+0000","last_change":"2023-03-03T08:43:34.548229+0000","last_active":"2023-03-03T08:43:43.983100+0000","last_peered":"2023-03-03T08:43:43.983100+0000","last_clean":"2023-03-03T08:43:43.983100+0000","last_became_active":"2023-03-03T08:43:34.548013+0000","last_became_peered":"2023-03-03T08:43:34.548013+0000","last_unstale":"2023-03-03T08:43:43.983100+0000","last_undegraded":"2023-03-03T08:43:43.983100+0000","last_fullsized":"2023-03-03T08:43:43.983100+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:24:27.793716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.108186+0000","last_change":"2023-03-03T08:43:43.992834+0000","last_active":"2023-03-03T08:43:45.108186+0000","last_peered":"2023-03-03T08:43:45.108186+0000","last_clean":"2023-03-03T08:43:45.108186+0000","last_became_active":"2023-03-03T08:43:43.991577+0000","last_became_peered":"2023-03-03T08:43:43.991577+0000","last_unstale":"2023-03-03T08:43:45.108186+0000","last_undegraded":"2023-03-03T08:43:45.108186+0000","last_fullsized":"2023-03-03T08:43:45.108186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:18:37.119460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.990785+0000","last_change":"2023-03-03T08:43:34.549838+0000","last_active":"2023-03-03T08:43:43.990785+0000","last_peered":"2023-03-03T08:43:43.990785+0000","last_clean":"2023-03-03T08:43:43.990785+0000","last_became_active":"2023-03-03T08:43:34.549537+0000","last_became_peered":"2023-03-03T08:43:34.549537+0000","last_unstale":"2023-03-03T08:43:43.990785+0000","last_undegraded":"2023-03-03T08:43:43.990785+0000","last_fullsized":"2023-03-03T08:43:43.990785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:16:36.955716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.978412+0000","last_change":"2023-03-03T08:43:43.991488+0000","last_active":"2023-03-03T08:43:44.978412+0000","last_peered":"2023-03-03T08:43:44.978412+0000","last_clean":"2023-03-03T08:43:44.978412+0000","last_became_active":"2023-03-03T08:43:43.991139+0000","last_became_peered":"2023-03-03T08:43:43.991139+0000","last_unstale":"2023-03-03T08:43:44.978412+0000","last_undegraded":"2023-03-03T08:43:44.978412+0000","last_fullsized":"2023-03-03T08:43:44.978412+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:43:21.119183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983652+0000","last_change":"2023-03-03T08:43:34.533496+0000","last_active":"2023-03-03T08:43:43.983652+0000","last_peered":"2023-03-03T08:43:43.983652+0000","last_clean":"2023-03-03T08:43:43.983652+0000","last_became_active":"2023-03-03T08:43:34.532714+0000","last_became_peered":"2023-03-03T08:43:34.532714+0000","last_unstale":"2023-03-03T08:43:43.983652+0000","last_undegraded":"2023-03-03T08:43:43.983652+0000","last_fullsized":"2023-03-03T08:43:43.983652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:44:47.800807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.105250+0000","last_change":"2023-03-03T08:43:43.996215+0000","last_active":"2023-03-03T08:43:45.105250+0000","last_peered":"2023-03-03T08:43:45.105250+0000","last_clean":"2023-03-03T08:43:45.105250+0000","last_became_active":"2023-03-03T08:43:43.995791+0000","last_became_peered":"2023-03-03T08:43:43.995791+0000","last_unstale":"2023-03-03T08:43:45.105250+0000","last_undegraded":"2023-03-03T08:43:45.105250+0000","last_fullsized":"2023-03-03T08:43:45.105250+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:50:41.130427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.979929+0000","last_change":"2023-03-03T08:43:34.533673+0000","last_active":"2023-03-03T08:43:43.979929+0000","last_peered":"2023-03-03T08:43:43.979929+0000","last_clean":"2023-03-03T08:43:43.979929+0000","last_became_active":"2023-03-03T08:43:34.532283+0000","last_became_peered":"2023-03-03T08:43:34.532283+0000","last_unstale":"2023-03-03T08:43:43.979929+0000","last_undegraded":"2023-03-03T08:43:43.979929+0000","last_fullsized":"2023-03-03T08:43:43.979929+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:47:25.591188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979205+0000","last_change":"2023-03-03T08:43:43.999435+0000","last_active":"2023-03-03T08:43:44.979205+0000","last_peered":"2023-03-03T08:43:44.979205+0000","last_clean":"2023-03-03T08:43:44.979205+0000","last_became_active":"2023-03-03T08:43:43.997037+0000","last_became_peered":"2023-03-03T08:43:43.997037+0000","last_unstale":"2023-03-03T08:43:44.979205+0000","last_undegraded":"2023-03-03T08:43:44.979205+0000","last_fullsized":"2023-03-03T08:43:44.979205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T11:35:54.176343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.980585+0000","last_change":"2023-03-03T08:43:34.537664+0000","last_active":"2023-03-03T08:43:43.980585+0000","last_peered":"2023-03-03T08:43:43.980585+0000","last_clean":"2023-03-03T08:43:43.980585+0000","last_became_active":"2023-03-03T08:43:34.537214+0000","last_became_peered":"2023-03-03T08:43:34.537214+0000","last_unstale":"2023-03-03T08:43:43.980585+0000","last_undegraded":"2023-03-03T08:43:43.980585+0000","last_fullsized":"2023-03-03T08:43:43.980585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:54:09.222135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:45.106227+0000","last_change":"2023-03-03T08:43:44.000140+0000","last_active":"2023-03-03T08:43:45.106227+0000","last_peered":"2023-03-03T08:43:45.106227+0000","last_clean":"2023-03-03T08:43:45.106227+0000","last_became_active":"2023-03-03T08:43:43.999778+0000","last_became_peered":"2023-03-03T08:43:43.999778+0000","last_unstale":"2023-03-03T08:43:45.106227+0000","last_undegraded":"2023-03-03T08:43:45.106227+0000","last_fullsized":"2023-03-03T08:43:45.106227+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T18:29:10.745759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.987366+0000","last_change":"2023-03-03T08:43:34.543620+0000","last_active":"2023-03-03T08:43:43.987366+0000","last_peered":"2023-03-03T08:43:43.987366+0000","last_clean":"2023-03-03T08:43:43.987366+0000","last_became_active":"2023-03-03T08:43:34.543370+0000","last_became_peered":"2023-03-03T08:43:34.543370+0000","last_unstale":"2023-03-03T08:43:43.987366+0000","last_undegraded":"2023-03-03T08:43:43.987366+0000","last_fullsized":"2023-03-03T08:43:43.987366+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T14:40:32.808203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.979449+0000","last_change":"2023-03-03T08:43:43.995084+0000","last_active":"2023-03-03T08:43:44.979449+0000","last_peered":"2023-03-03T08:43:44.979449+0000","last_clean":"2023-03-03T08:43:44.979449+0000","last_became_active":"2023-03-03T08:43:43.994824+0000","last_became_peered":"2023-03-03T08:43:43.994824+0000","last_unstale":"2023-03-03T08:43:44.979449+0000","last_undegraded":"2023-03-03T08:43:44.979449+0000","last_fullsized":"2023-03-03T08:43:44.979449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T19:04:17.481969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.985302+0000","last_change":"2023-03-03T08:43:34.538467+0000","last_active":"2023-03-03T08:43:43.985302+0000","last_peered":"2023-03-03T08:43:43.985302+0000","last_clean":"2023-03-03T08:43:43.985302+0000","last_became_active":"2023-03-03T08:43:34.538207+0000","last_became_peered":"2023-03-03T08:43:34.538207+0000","last_unstale":"2023-03-03T08:43:43.985302+0000","last_undegraded":"2023-03-03T08:43:43.985302+0000","last_fullsized":"2023-03-03T08:43:43.985302+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T15:07:14.518566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980577+0000","last_change":"2023-03-03T08:43:43.996076+0000","last_active":"2023-03-03T08:43:44.980577+0000","last_peered":"2023-03-03T08:43:44.980577+0000","last_clean":"2023-03-03T08:43:44.980577+0000","last_became_active":"2023-03-03T08:43:43.995799+0000","last_became_peered":"2023-03-03T08:43:43.995799+0000","last_unstale":"2023-03-03T08:43:44.980577+0000","last_undegraded":"2023-03-03T08:43:44.980577+0000","last_fullsized":"2023-03-03T08:43:44.980577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T20:35:35.889895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986289+0000","last_change":"2023-03-03T08:43:34.544721+0000","last_active":"2023-03-03T08:43:43.986289+0000","last_peered":"2023-03-03T08:43:43.986289+0000","last_clean":"2023-03-03T08:43:43.986289+0000","last_became_active":"2023-03-03T08:43:34.544461+0000","last_became_peered":"2023-03-03T08:43:34.544461+0000","last_unstale":"2023-03-03T08:43:43.986289+0000","last_undegraded":"2023-03-03T08:43:43.986289+0000","last_fullsized":"2023-03-03T08:43:43.986289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T10:52:11.890779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980078+0000","last_change":"2023-03-03T08:43:44.006352+0000","last_active":"2023-03-03T08:43:44.980078+0000","last_peered":"2023-03-03T08:43:44.980078+0000","last_clean":"2023-03-03T08:43:44.980078+0000","last_became_active":"2023-03-03T08:43:44.005802+0000","last_became_peered":"2023-03-03T08:43:44.005802+0000","last_unstale":"2023-03-03T08:43:44.980078+0000","last_undegraded":"2023-03-03T08:43:44.980078+0000","last_fullsized":"2023-03-03T08:43:44.980078+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T16:07:52.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.986425+0000","last_change":"2023-03-03T08:43:34.541669+0000","last_active":"2023-03-03T08:43:43.986425+0000","last_peered":"2023-03-03T08:43:43.986425+0000","last_clean":"2023-03-03T08:43:43.986425+0000","last_became_active":"2023-03-03T08:43:34.541104+0000","last_became_peered":"2023-03-03T08:43:34.541104+0000","last_unstale":"2023-03-03T08:43:43.986425+0000","last_undegraded":"2023-03-03T08:43:43.986425+0000","last_fullsized":"2023-03-03T08:43:43.986425+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T17:18:13.168823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-03-03T08:43:44.980059+0000","last_change":"2023-03-03T08:43:43.998937+0000","last_active":"2023-03-03T08:43:44.980059+0000","last_peered":"2023-03-03T08:43:44.980059+0000","last_clean":"2023-03-03T08:43:44.980059+0000","last_became_active":"2023-03-03T08:43:43.998164+0000","last_became_peered":"2023-03-03T08:43:43.998164+0000","last_unstale":"2023-03-03T08:43:44.980059+0000","last_undegraded":"2023-03-03T08:43:44.980059+0000","last_fullsized":"2023-03-03T08:43:44.980059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:43.549793+0000","last_clean_scrub_stamp":"2023-03-03T08:43:43.549793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T09:19:53.741704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-03-03T08:43:43.983031+0000","last_change":"2023-03-03T08:43:34.546272+0000","last_active":"2023-03-03T08:43:43.983031+0000","last_peered":"2023-03-03T08:43:43.983031+0000","last_clean":"2023-03-03T08:43:43.983031+0000","last_became_active":"2023-03-03T08:43:34.546006+0000","last_became_peered":"2023-03-03T08:43:34.546006+0000","last_unstale":"2023-03-03T08:43:43.983031+0000","last_undegraded":"2023-03-03T08:43:43.983031+0000","last_fullsized":"2023-03-03T08:43:43.983031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-03T08:43:34.087636+0000","last_clean_scrub_stamp":"2023-03-03T08:43:34.087636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-04T08:53:15.121761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":3561,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3561,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8164,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":94363676,"statfs":{"total":96636764160,"available":96628404224,"internally_reserved":0,"allocated":290816,"data_stored":118397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8236,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363604,"statfs":{"total":96636764160,"available":96628330496,"internally_reserved":0,"allocated":299008,"data_stored":119470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9336,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":8448,"kb_avail":94362504,"statfs":{"total":96636764160,"available":96627204096,"internally_reserved":0,"allocated":901120,"data_stored":710394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8650752},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8244,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363596,"statfs":{"total":96636764160,"available":96628322304,"internally_reserved":0,"allocated":307200,"data_stored":118565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8252,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":94363588,"statfs":{"total":96636764160,"available":96628314112,"internally_reserved":0,"allocated":315392,"data_stored":124451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":121300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8312,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":8000,"kb_avail":94363528,"statfs":{"total":96636764160,"available":96628252672,"internally_reserved":0,"allocated":311296,"data_stored":122249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8192000},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9260,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94362580,"statfs":{"total":96636764160,"available":96627281920,"internally_reserved":0,"allocated":888832,"data_stored":709827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-03T08:44:08.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:44:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.779+0000 7f9c53518700 1 Processor -- start 2023-03-03T08:44:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.779+0000 7f9c53518700 1 -- start start 2023-03-03T08:44:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c53518700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c4c0c6010 0x7f9c4c0c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c53518700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c4c0c4fc0 0x7f9c4c06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c53518700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4c14ebe0 con 0x7f9c4c0c17f0 2023-03-03T08:44:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c53518700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c4c152100 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c53518700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c4c0c2e70 con 0x7f9c4c0c6010 2023-03-03T08:44:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c51ab5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c4c0c4fc0 0x7f9c4c06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c51ab5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c4c0c4fc0 0x7f9c4c06ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52322/0 (socket says 172.21.15.7:52322) 2023-03-03T08:44:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c51ab5700 1 -- 172.21.15.7:0/3806294293 learned_addr learned my addr 172.21.15.7:0/3806294293 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c50ab3700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9c4c0c17f0 0x7f9c4c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46298/0 (socket says 172.21.15.7:46298) 2023-03-03T08:44:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c512b4700 1 --2- 172.21.15.7:0/3806294293 >> v2:172.21.15.143:3300/0 conn(0x7f9c4c0c6010 0x7f9c4c0c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c51ab5700 1 -- 172.21.15.7:0/3806294293 >> v2:172.21.15.143:3300/0 conn(0x7f9c4c0c6010 msgr2=0x7f9c4c0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c51ab5700 1 --2- 172.21.15.7:0/3806294293 >> v2:172.21.15.143:3300/0 conn(0x7f9c4c0c6010 0x7f9c4c0c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c51ab5700 1 -- 172.21.15.7:0/3806294293 >> v1:172.21.15.7:6789/0 conn(0x7f9c4c0c17f0 legacy=0x7f9c4c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c51ab5700 1 -- 172.21.15.7:0/3806294293 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c0aa680 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c43fff700 1 -- 172.21.15.7:0/3806294293 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569472883 0 0) 0x7f9c4c14ebe0 con 0x7f9c4c0c17f0 2023-03-03T08:44:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c51ab5700 1 --2- 172.21.15.7:0/3806294293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 0x7f9c4c06ddb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c0c6530 tx=0x7f9c4400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca3688b05bd10dc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c43fff700 1 -- 172.21.15.7:0/3806294293 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44010070 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c43fff700 1 -- 172.21.15.7:0/3806294293 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4400dd90 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.780+0000 7f9c43fff700 1 -- 172.21.15.7:0/3806294293 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4400b310 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.781+0000 7f9c53518700 1 -- 172.21.15.7:0/3806294293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 msgr2=0x7f9c4c06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.781+0000 7f9c53518700 1 --2- 172.21.15.7:0/3806294293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 0x7f9c4c06ddb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c0c6530 tx=0x7f9c4400d430 comp rx=0 tx=0).stop 2023-03-03T08:44:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.781+0000 7f9c53518700 1 -- 172.21.15.7:0/3806294293 shutdown_connections 2023-03-03T08:44:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.781+0000 7f9c53518700 1 --2- 172.21.15.7:0/3806294293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 0x7f9c4c06ddb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.781+0000 7f9c53518700 1 --2- 172.21.15.7:0/3806294293 >> v2:172.21.15.143:3300/0 conn(0x7f9c4c0c6010 0x7f9c4c0c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.781+0000 7f9c53518700 1 -- 172.21.15.7:0/3806294293 >> 172.21.15.7:0/3806294293 conn(0x7f9c4c0a9430 msgr2=0x7f9c4c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.781+0000 7f9c53518700 1 -- 172.21.15.7:0/3806294293 shutdown_connections 2023-03-03T08:44:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.781+0000 7f9c53518700 1 -- 172.21.15.7:0/3806294293 wait complete. 2023-03-03T08:44:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.782+0000 7f9c53518700 1 Processor -- start 2023-03-03T08:44:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.782+0000 7f9c53518700 1 -- start start 2023-03-03T08:44:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c53518700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c4c0c17f0 0x7f9c4c0a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c53518700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 0x7f9c4c06cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c53518700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c4c0c6010 0x7f9c4c14b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c53518700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c1dd550 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c53518700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c06fe80 con 0x7f9c4c0c17f0 2023-03-03T08:44:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c53518700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c1dc890 con 0x7f9c4c0c6010 2023-03-03T08:44:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c50ab3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 0x7f9c4c06cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c50ab3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 0x7f9c4c06cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52326/0 (socket says 172.21.15.7:52326) 2023-03-03T08:44:08.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c50ab3700 1 -- 172.21.15.7:0/1259582114 learned_addr learned my addr 172.21.15.7:0/1259582114 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c512b4700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c4c0c17f0 0x7f9c4c0a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c51ab5700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c4c0c6010 0x7f9c4c14b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c50ab3700 1 -- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c4c0c6010 msgr2=0x7f9c4c14b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c50ab3700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c4c0c6010 0x7f9c4c14b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c50ab3700 1 -- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c4c0c17f0 msgr2=0x7f9c4c0a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c50ab3700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c4c0c17f0 0x7f9c4c0a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c50ab3700 1 -- 172.21.15.7:0/1259582114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4400bd40 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.783+0000 7f9c512b4700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c4c0c17f0 0x7f9c4c0a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.784+0000 7f9c50ab3700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 0x7f9c4c06cf80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0011e0 tx=0x7f9c3c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.784+0000 7f9c41ffb700 1 -- 172.21.15.7:0/1259582114 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c004c80 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.784+0000 7f9c41ffb700 1 -- 172.21.15.7:0/1259582114 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3c004e00 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.784+0000 7f9c53518700 1 -- 172.21.15.7:0/1259582114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4c0c2b00 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.784+0000 7f9c41ffb700 1 -- 172.21.15.7:0/1259582114 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c007310 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.784+0000 7f9c53518700 1 -- 172.21.15.7:0/1259582114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4c0aa960 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.785+0000 7f9c53518700 1 -- 172.21.15.7:0/1259582114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4c0c3300 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.785+0000 7f9c41ffb700 1 -- 172.21.15.7:0/1259582114 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9c3c01a020 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.786+0000 7f9c41ffb700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c300682f0 0x7f9c3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:08.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.786+0000 7f9c41ffb700 1 -- 172.21.15.7:0/1259582114 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f9c3c085bd0 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.786+0000 7f9c512b4700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c300682f0 0x7f9c3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:08.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.786+0000 7f9c512b4700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c300682f0 0x7f9c3006a7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9c48009820 tx=0x7f9c48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:08.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.790+0000 7f9c41ffb700 1 -- 172.21.15.7:0/1259582114 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9c3c052100 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.982+0000 7f9c53518700 1 -- 172.21.15.7:0/1259582114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c4c06fff0 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.984+0000 7f9c41ffb700 1 -- 172.21.15.7:0/1259582114 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21533 (secure 0 0 0) 0x7f9c3c055720 con 0x7f9c4c0c4fc0 2023-03-03T08:44:08.985 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:08.986 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:44.973251+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:44:08.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 -- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c300682f0 msgr2=0x7f9c3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c300682f0 0x7f9c3006a7a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9c48009820 tx=0x7f9c48008040 comp rx=0 tx=0).stop 2023-03-03T08:44:08.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 -- 172.21.15.7:0/1259582114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 msgr2=0x7f9c4c06cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:08.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 0x7f9c4c06cf80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0011e0 tx=0x7f9c3c00cae0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 -- 172.21.15.7:0/1259582114 shutdown_connections 2023-03-03T08:44:08.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c4c0c6010 0x7f9c4c14b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c4c0c17f0 0x7f9c4c0a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c300682f0 0x7f9c3006a7a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 --2- 172.21.15.7:0/1259582114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c4c0c4fc0 0x7f9c4c06cf80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:08.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 -- 172.21.15.7:0/1259582114 >> 172.21.15.7:0/1259582114 conn(0x7f9c4c0a9430 msgr2=0x7f9c4c14cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:08.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 -- 172.21.15.7:0/1259582114 shutdown_connections 2023-03-03T08:44:08.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:08.986+0000 7f9c53518700 1 -- 172.21.15.7:0/1259582114 wait complete. 2023-03-03T08:44:09.005 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T08:43:21.160680+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T08:43:28.018457+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-03-03T08:43:32.305037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-03-03T08:43:33.484501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-03-03T08:43:42.509451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-03-03T08:43:42.936898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:44:09.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:44:09.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.132+0000 7f42b8e30700 1 Processor -- start 2023-03-03T08:44:09.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.132+0000 7f42b8e30700 1 -- start start 2023-03-03T08:44:09.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.133+0000 7f42b8e30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b414e900 0x7f42b415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.133+0000 7f42b8e30700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b41510d0 con 0x7f42b4071e80 2023-03-03T08:44:09.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.133+0000 7f42b8e30700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b40c3740 con 0x7f42b4072250 2023-03-03T08:44:09.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.133+0000 7f42b8e30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b4070670 con 0x7f42b414e900 2023-03-03T08:44:09.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.133+0000 7f42b259c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f42b4071e80 0x7f42b4151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46312/0 (socket says 172.21.15.7:46312) 2023-03-03T08:44:09.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.133+0000 7f42b259c700 1 -- 172.21.15.7:0/947306093 learned_addr learned my addr 172.21.15.7:0/947306093 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:09.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.133+0000 7f42b2d9d700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b414e900 0x7f42b415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.133+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379280272 0 0) 0x7f42b41510d0 con 0x7f42b4071e80 2023-03-03T08:44:09.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.133+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f429c0035b0 con 0x7f42b4071e80 2023-03-03T08:44:09.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666032980 0 0) 0x7f42b40c3740 con 0x7f42b4072250 2023-03-03T08:44:09.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b41510d0 con 0x7f42b4072250 2023-03-03T08:44:09.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 8600891 0 0) 0x7f429c0035b0 con 0x7f42b4071e80 2023-03-03T08:44:09.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b414e900 msgr2=0x7f42b415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:09.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b414e900 0x7f42b415ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 >> v1:172.21.15.143:6789/0 conn(0x7f42b4072250 legacy=0x7f42b414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b40c39e0 con 0x7f42b4071e80 2023-03-03T08:44:09.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f42a4001670 con 0x7f42b4071e80 2023-03-03T08:44:09.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 >> v1:172.21.15.7:6789/0 conn(0x7f42b4071e80 legacy=0x7f42b4151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b414e900 0x7f429c003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f429c004190 0x7f429c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429c004d90 0x7f429c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b4070670 con 0x7f429c004190 2023-03-03T08:44:09.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c00dde0 con 0x7f429c004d90 2023-03-03T08:44:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429c00dfa0 con 0x7f42b414e900 2023-03-03T08:44:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b1d9b700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f429c004190 0x7f429c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b2d9d700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429c004d90 0x7f429c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b259c700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b414e900 0x7f429c003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b1d9b700 1 -- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b414e900 msgr2=0x7f429c003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b1d9b700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b414e900 0x7f429c003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b1d9b700 1 -- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429c004d90 msgr2=0x7f429c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b1d9b700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429c004d90 0x7f429c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b1d9b700 1 -- 172.21.15.7:0/947306093 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c0035b0 con 0x7f429c004190 2023-03-03T08:44:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.134+0000 7f42b1d9b700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f429c004190 0x7f429c004850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f42b40c1780 tx=0x7f429c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9d08c796e4a8db server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.135+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8001d80 con 0x7f429c004190 2023-03-03T08:44:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.135+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a80052d0 con 0x7f429c004190 2023-03-03T08:44:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.135+0000 7f42b159a700 1 -- 172.21.15.7:0/947306093 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8001470 con 0x7f429c004190 2023-03-03T08:44:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.135+0000 7f42b8e30700 1 -- 172.21.15.7:0/947306093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f429c004190 msgr2=0x7f429c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.135+0000 7f42b8e30700 1 --2- 172.21.15.7:0/947306093 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f429c004190 0x7f429c004850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f42b40c1780 tx=0x7f429c005880 comp rx=0 tx=0).stop 2023-03-03T08:44:09.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.135+0000 7f42b2d9d700 1 -- 172.21.15.7:0/947306093 reap_dead start 2023-03-03T08:44:09.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.135+0000 7f42b8e30700 1 -- 172.21.15.7:0/947306093 shutdown_connections 2023-03-03T08:44:09.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.135+0000 7f42b8e30700 1 -- 172.21.15.7:0/947306093 >> 172.21.15.7:0/947306093 conn(0x7f42b40a94d0 msgr2=0x7f42b40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:09.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.135+0000 7f42b8e30700 1 -- 172.21.15.7:0/947306093 shutdown_connections 2023-03-03T08:44:09.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.135+0000 7f42b8e30700 1 -- 172.21.15.7:0/947306093 wait complete. 2023-03-03T08:44:09.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b8e30700 1 Processor -- start 2023-03-03T08:44:09.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b8e30700 1 -- start start 2023-03-03T08:44:09.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b8e30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429c004190 0x7f42b41e67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b8e30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42b41dad30 0x7f42b4149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b8e30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b40aa1d0 0x7f42b40abd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b8e30700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b40aaf20 con 0x7f42b41dad30 2023-03-03T08:44:09.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b8e30700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b41e6330 con 0x7f429c004190 2023-03-03T08:44:09.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b8e30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b41e9d20 con 0x7f42b40aa1d0 2023-03-03T08:44:09.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42b41dad30 0x7f42b4149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42b41dad30 0x7f42b4149bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52346/0 (socket says 172.21.15.7:52346) 2023-03-03T08:44:09.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.137+0000 7f42b1d9b700 1 -- 172.21.15.7:0/182431273 learned_addr learned my addr 172.21.15.7:0/182431273 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:09.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42b1d9b700 1 -- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b40aa1d0 msgr2=0x7f42b40abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42b1d9b700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b40aa1d0 0x7f42b40abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42b259c700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429c004190 0x7f42b41e67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42b1d9b700 1 -- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429c004190 msgr2=0x7f42b41e67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42b1d9b700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429c004190 0x7f42b41e67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42b1d9b700 1 -- 172.21.15.7:0/182431273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b4151f30 con 0x7f42b41dad30 2023-03-03T08:44:09.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42b1d9b700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42b41dad30 0x7f42b4149bf0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f42a800c3c0 tx=0x7f42a8001310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:09.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42a2ffd700 1 -- 172.21.15.7:0/182431273 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8001d80 con 0x7f42b41dad30 2023-03-03T08:44:09.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42a2ffd700 1 -- 172.21.15.7:0/182431273 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a8001410 con 0x7f42b41dad30 2023-03-03T08:44:09.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42b8e30700 1 -- 172.21.15.7:0/182431273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b4151300 con 0x7f42b41dad30 2023-03-03T08:44:09.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42a2ffd700 1 -- 172.21.15.7:0/182431273 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8001110 con 0x7f42b41dad30 2023-03-03T08:44:09.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.138+0000 7f42b8e30700 1 -- 172.21.15.7:0/182431273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b406c9a0 con 0x7f42b41dad30 2023-03-03T08:44:09.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.139+0000 7f42b8e30700 1 -- 172.21.15.7:0/182431273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b4074bb0 con 0x7f42b41dad30 2023-03-03T08:44:09.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.139+0000 7f42a2ffd700 1 -- 172.21.15.7:0/182431273 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f42a802e070 con 0x7f42b41dad30 2023-03-03T08:44:09.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.140+0000 7f42a2ffd700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42940682f0 0x7f429406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.140+0000 7f42a2ffd700 1 -- 172.21.15.7:0/182431273 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f42a8086f00 con 0x7f42b41dad30 2023-03-03T08:44:09.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.140+0000 7f42b259c700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42940682f0 0x7f429406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.141+0000 7f42b259c700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42940682f0 0x7f429406a7a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f42a40011e0 tx=0x7f42a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:09.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.144+0000 7f42a2ffd700 1 -- 172.21.15.7:0/182431273 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f42a8053430 con 0x7f42b41dad30 2023-03-03T08:44:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.335+0000 7f42b8e30700 1 -- 172.21.15.7:0/182431273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f42b41e6470 con 0x7f42b41dad30 2023-03-03T08:44:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.336+0000 7f42a2ffd700 1 -- 172.21.15.7:0/182431273 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f42b41e6470 con 0x7f42b41dad30 2023-03-03T08:44:09.339 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-03-03T08:44:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 -- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42940682f0 msgr2=0x7f429406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42940682f0 0x7f429406a7a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f42a40011e0 tx=0x7f42a4009040 comp rx=0 tx=0).stop 2023-03-03T08:44:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 -- 172.21.15.7:0/182431273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42b41dad30 msgr2=0x7f42b4149bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42b41dad30 0x7f42b4149bf0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f42a800c3c0 tx=0x7f42a8001310 comp rx=0 tx=0).stop 2023-03-03T08:44:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 -- 172.21.15.7:0/182431273 shutdown_connections 2023-03-03T08:44:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42b40aa1d0 0x7f42b40abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f429c004190 0x7f42b41e67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42940682f0 0x7f429406a7a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 --2- 172.21.15.7:0/182431273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42b41dad30 0x7f42b4149bf0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 -- 172.21.15.7:0/182431273 >> 172.21.15.7:0/182431273 conn(0x7f42b40a94d0 msgr2=0x7f42b406a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:09.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 -- 172.21.15.7:0/182431273 shutdown_connections 2023-03-03T08:44:09.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.338+0000 7f42b8e30700 1 -- 172.21.15.7:0/182431273 wait complete. 2023-03-03T08:44:09.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:44:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.479+0000 7fd911de1700 1 Processor -- start 2023-03-03T08:44:09.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.479+0000 7fd911de1700 1 -- start start 2023-03-03T08:44:09.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.479+0000 7fd911de1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd90c155390 0x7fd90c0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.479+0000 7fd911de1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c06e020 0x7fd90c071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.479+0000 7fd911de1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c154910 con 0x7fd90c072520 2023-03-03T08:44:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.479+0000 7fd911de1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90c070b90 con 0x7fd90c155390 2023-03-03T08:44:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.479+0000 7fd911de1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c151320 con 0x7fd90c06e020 2023-03-03T08:44:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd90c155390 0x7fd90c0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd90c155390 0x7fd90c0c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52352/0 (socket says 172.21.15.7:52352) 2023-03-03T08:44:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd90c072520 0x7fd90c151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46332/0 (socket says 172.21.15.7:46332) 2023-03-03T08:44:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90b7fe700 1 -- 172.21.15.7:0/446394494 learned_addr learned my addr 172.21.15.7:0/446394494 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90affd700 1 --2- 172.21.15.7:0/446394494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c06e020 0x7fd90c071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90b7fe700 1 -- 172.21.15.7:0/446394494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c06e020 msgr2=0x7fd90c071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90b7fe700 1 --2- 172.21.15.7:0/446394494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c06e020 0x7fd90c071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90b7fe700 1 -- 172.21.15.7:0/446394494 >> v1:172.21.15.7:6789/0 conn(0x7fd90c072520 legacy=0x7fd90c151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90a7fc700 1 -- 172.21.15.7:0/446394494 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510084339 0 0) 0x7fd90c154910 con 0x7fd90c072520 2023-03-03T08:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90b7fe700 1 -- 172.21.15.7:0/446394494 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c0702a0 con 0x7fd90c155390 2023-03-03T08:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90b7fe700 1 --2- 172.21.15.7:0/446394494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c155390 0x7fd90c0c3e30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd9000098f0 tx=0x7fd900009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf1b8529b35427a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90a7fc700 1 -- 172.21.15.7:0/446394494 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000b8a0 con 0x7fd90c155390 2023-03-03T08:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90a7fc700 1 -- 172.21.15.7:0/446394494 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90000ba00 con 0x7fd90c155390 2023-03-03T08:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.480+0000 7fd90a7fc700 1 -- 172.21.15.7:0/446394494 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000bd90 con 0x7fd90c155390 2023-03-03T08:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.481+0000 7fd911de1700 1 -- 172.21.15.7:0/446394494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c155390 msgr2=0x7fd90c0c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.481+0000 7fd911de1700 1 --2- 172.21.15.7:0/446394494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c155390 0x7fd90c0c3e30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd9000098f0 tx=0x7fd900009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.481+0000 7fd911de1700 1 -- 172.21.15.7:0/446394494 shutdown_connections 2023-03-03T08:44:09.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.481+0000 7fd911de1700 1 --2- 172.21.15.7:0/446394494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c155390 0x7fd90c0c3e30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.481+0000 7fd911de1700 1 --2- 172.21.15.7:0/446394494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c06e020 0x7fd90c071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.481+0000 7fd911de1700 1 -- 172.21.15.7:0/446394494 >> 172.21.15.7:0/446394494 conn(0x7fd90c06c110 msgr2=0x7fd90c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:09.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.481+0000 7fd911de1700 1 -- 172.21.15.7:0/446394494 shutdown_connections 2023-03-03T08:44:09.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.481+0000 7fd911de1700 1 -- 172.21.15.7:0/446394494 wait complete. 2023-03-03T08:44:09.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.482+0000 7fd911de1700 1 Processor -- start 2023-03-03T08:44:09.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.482+0000 7fd911de1700 1 -- start start 2023-03-03T08:44:09.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd911de1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c06e020 0x7fd90c1ece10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd911de1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c072520 0x7fd90c0c33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd911de1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c155390 0x7fd90c14e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd911de1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c154710 con 0x7fd90c06e020 2023-03-03T08:44:09.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd911de1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0c07d0 con 0x7fd90c072520 2023-03-03T08:44:09.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd911de1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0c1860 con 0x7fd90c155390 2023-03-03T08:44:09.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c06e020 0x7fd90c1ece10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c06e020 0x7fd90c1ece10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52358/0 (socket says 172.21.15.7:52358) 2023-03-03T08:44:09.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90b7fe700 1 -- 172.21.15.7:0/1903710577 learned_addr learned my addr 172.21.15.7:0/1903710577 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:09.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90affd700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c072520 0x7fd90c0c33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90bfff700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c155390 0x7fd90c14e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90b7fe700 1 -- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c155390 msgr2=0x7fd90c14e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90b7fe700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c155390 0x7fd90c14e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90b7fe700 1 -- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c072520 msgr2=0x7fd90c0c33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90b7fe700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c072520 0x7fd90c0c33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90b7fe700 1 -- 172.21.15.7:0/1903710577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900009550 con 0x7fd90c06e020 2023-03-03T08:44:09.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.483+0000 7fd90bfff700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c155390 0x7fd90c14e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:09.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.484+0000 7fd90b7fe700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c06e020 0x7fd90c1ece10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd900009c40 tx=0x7fd900004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:09.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.484+0000 7fd8f3fff700 1 -- 172.21.15.7:0/1903710577 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900004380 con 0x7fd90c06e020 2023-03-03T08:44:09.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.484+0000 7fd8f3fff700 1 -- 172.21.15.7:0/1903710577 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90001f070 con 0x7fd90c06e020 2023-03-03T08:44:09.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.484+0000 7fd911de1700 1 -- 172.21.15.7:0/1903710577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c070820 con 0x7fd90c06e020 2023-03-03T08:44:09.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.484+0000 7fd8f3fff700 1 -- 172.21.15.7:0/1903710577 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9000036a0 con 0x7fd90c06e020 2023-03-03T08:44:09.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.484+0000 7fd911de1700 1 -- 172.21.15.7:0/1903710577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c070540 con 0x7fd90c06e020 2023-03-03T08:44:09.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.485+0000 7fd911de1700 1 -- 172.21.15.7:0/1903710577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c155760 con 0x7fd90c06e020 2023-03-03T08:44:09.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.485+0000 7fd8f3fff700 1 -- 172.21.15.7:0/1903710577 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd90002a020 con 0x7fd90c06e020 2023-03-03T08:44:09.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.486+0000 7fd8f3fff700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8f80682f0 0x7fd8f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.486+0000 7fd8f3fff700 1 -- 172.21.15.7:0/1903710577 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fd900087540 con 0x7fd90c06e020 2023-03-03T08:44:09.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.486+0000 7fd90affd700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8f80682f0 0x7fd8f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.486+0000 7fd90affd700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8f80682f0 0x7fd8f806a7a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00b9b0 tx=0x7fd8fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:09.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.490+0000 7fd8f3fff700 1 -- 172.21.15.7:0/1903710577 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd900053a70 con 0x7fd90c06e020 2023-03-03T08:44:09.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.682+0000 7fd911de1700 1 -- 172.21.15.7:0/1903710577 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd90c154850 con 0x7fd90c06e020 2023-03-03T08:44:09.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.682+0000 7fd8f3fff700 1 -- 172.21.15.7:0/1903710577 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd900057090 con 0x7fd90c06e020 2023-03-03T08:44:09.684 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 8 2023-03-03T08:44:09.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 -- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8f80682f0 msgr2=0x7fd8f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8f80682f0 0x7fd8f806a7a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00b9b0 tx=0x7fd8fc008040 comp rx=0 tx=0).stop 2023-03-03T08:44:09.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 -- 172.21.15.7:0/1903710577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c06e020 msgr2=0x7fd90c1ece10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c06e020 0x7fd90c1ece10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd900009c40 tx=0x7fd900004750 comp rx=0 tx=0).stop 2023-03-03T08:44:09.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 -- 172.21.15.7:0/1903710577 shutdown_connections 2023-03-03T08:44:09.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c155390 0x7fd90c14e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8f80682f0 0x7fd8f806a7a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c06e020 0x7fd90c1ece10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 --2- 172.21.15.7:0/1903710577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c072520 0x7fd90c0c33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 -- 172.21.15.7:0/1903710577 >> 172.21.15.7:0/1903710577 conn(0x7fd90c06c110 msgr2=0x7fd90c14ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:09.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 -- 172.21.15.7:0/1903710577 shutdown_connections 2023-03-03T08:44:09.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.684+0000 7fd911de1700 1 -- 172.21.15.7:0/1903710577 wait complete. 2023-03-03T08:44:09.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-03-03T08:44:09.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.823+0000 7f67218a5700 1 Processor -- start 2023-03-03T08:44:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.823+0000 7f67218a5700 1 -- start start 2023-03-03T08:44:09.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f67218a5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f671c0caa30 0x7f671c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f67218a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c0c00e0 0x7f671c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f67218a5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c0cb280 con 0x7f671c0cae00 2023-03-03T08:44:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f67218a5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671c0c7310 con 0x7f671c0caa30 2023-03-03T08:44:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f67218a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c0bfd10 con 0x7f671c0c00e0 2023-03-03T08:44:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f671affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f671c0caa30 0x7f671c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f671affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f671c0caa30 0x7f671c0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52366/0 (socket says 172.21.15.7:52366) 2023-03-03T08:44:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f671b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f671c0cae00 0x7f671c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46348/0 (socket says 172.21.15.7:46348) 2023-03-03T08:44:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f671affd700 1 -- 172.21.15.7:0/107896571 learned_addr learned my addr 172.21.15.7:0/107896571 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f671a7fc700 1 --2- 172.21.15.7:0/107896571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c0c00e0 0x7f671c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f671affd700 1 -- 172.21.15.7:0/107896571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c0c00e0 msgr2=0x7f671c154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f671affd700 1 --2- 172.21.15.7:0/107896571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c0c00e0 0x7f671c154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f671affd700 1 -- 172.21.15.7:0/107896571 >> v1:172.21.15.7:6789/0 conn(0x7f671c0cae00 legacy=0x7f671c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f6719ffb700 1 -- 172.21.15.7:0/107896571 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416971834 0 0) 0x7f671c0cb280 con 0x7f671c0cae00 2023-03-03T08:44:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.824+0000 7f671affd700 1 -- 172.21.15.7:0/107896571 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c0bfa30 con 0x7f671c0caa30 2023-03-03T08:44:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f671affd700 1 --2- 172.21.15.7:0/107896571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 0x7f671c0c5c60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f67040098f0 tx=0x7f6704009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd01b88b2412af0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f6719ffb700 1 -- 172.21.15.7:0/107896571 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400b8a0 con 0x7f671c0caa30 2023-03-03T08:44:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f6719ffb700 1 -- 172.21.15.7:0/107896571 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670400ba00 con 0x7f671c0caa30 2023-03-03T08:44:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f6719ffb700 1 -- 172.21.15.7:0/107896571 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400bd90 con 0x7f671c0caa30 2023-03-03T08:44:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f67218a5700 1 -- 172.21.15.7:0/107896571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 msgr2=0x7f671c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f67218a5700 1 --2- 172.21.15.7:0/107896571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 0x7f671c0c5c60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f67040098f0 tx=0x7f6704009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f67218a5700 1 -- 172.21.15.7:0/107896571 shutdown_connections 2023-03-03T08:44:09.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f67218a5700 1 --2- 172.21.15.7:0/107896571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 0x7f671c0c5c60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f67218a5700 1 --2- 172.21.15.7:0/107896571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c0c00e0 0x7f671c154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f67218a5700 1 -- 172.21.15.7:0/107896571 >> 172.21.15.7:0/107896571 conn(0x7f671c066e40 msgr2=0x7f671c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:09.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f67218a5700 1 -- 172.21.15.7:0/107896571 shutdown_connections 2023-03-03T08:44:09.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.825+0000 7f67218a5700 1 -- 172.21.15.7:0/107896571 wait complete. 2023-03-03T08:44:09.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.827+0000 7f67218a5700 1 Processor -- start 2023-03-03T08:44:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.827+0000 7f67218a5700 1 -- start start 2023-03-03T08:44:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.827+0000 7f67218a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f671c0c00e0 0x7f671c0c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.827+0000 7f67218a5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 0x7f671c14d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.827+0000 7f67218a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c0cae00 0x7f671c14d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.827+0000 7f67218a5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c158820 con 0x7f671c0caa30 2023-03-03T08:44:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.827+0000 7f67218a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c156d10 con 0x7f671c0c00e0 2023-03-03T08:44:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.827+0000 7f67218a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c1e9ed0 con 0x7f671c0cae00 2023-03-03T08:44:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 0x7f671c14d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 0x7f671c14d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52372/0 (socket says 172.21.15.7:52372) 2023-03-03T08:44:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671a7fc700 1 -- 172.21.15.7:0/2158128783 learned_addr learned my addr 172.21.15.7:0/2158128783 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671b7fe700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c0cae00 0x7f671c14d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671affd700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f671c0c00e0 0x7f671c0c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671a7fc700 1 -- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c0cae00 msgr2=0x7f671c14d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671a7fc700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c0cae00 0x7f671c14d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671a7fc700 1 -- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f671c0c00e0 msgr2=0x7f671c0c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671a7fc700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f671c0c00e0 0x7f671c0c23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671a7fc700 1 -- 172.21.15.7:0/2158128783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6704009550 con 0x7f671c0caa30 2023-03-03T08:44:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f671a7fc700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 0x7f671c14d310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f670c00e760 tx=0x7f670c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f6713fff700 1 -- 172.21.15.7:0/2158128783 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c005c20 con 0x7f671c0caa30 2023-03-03T08:44:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f6713fff700 1 -- 172.21.15.7:0/2158128783 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670c005da0 con 0x7f671c0caa30 2023-03-03T08:44:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f67218a5700 1 -- 172.21.15.7:0/2158128783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c1547f0 con 0x7f671c0caa30 2023-03-03T08:44:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f6713fff700 1 -- 172.21.15.7:0/2158128783 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c006260 con 0x7f671c0caa30 2023-03-03T08:44:09.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.828+0000 7f67218a5700 1 -- 172.21.15.7:0/2158128783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c154510 con 0x7f671c0caa30 2023-03-03T08:44:09.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.829+0000 7f67218a5700 1 -- 172.21.15.7:0/2158128783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c154ad0 con 0x7f671c0caa30 2023-03-03T08:44:09.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.830+0000 7f6713fff700 1 -- 172.21.15.7:0/2158128783 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f670c022070 con 0x7f671c0caa30 2023-03-03T08:44:09.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.830+0000 7f6713fff700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66fc0682f0 0x7f66fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:09.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.830+0000 7f6713fff700 1 -- 172.21.15.7:0/2158128783 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f670c085990 con 0x7f671c0caa30 2023-03-03T08:44:09.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.830+0000 7f671affd700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66fc0682f0 0x7f66fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:09.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.831+0000 7f671affd700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66fc0682f0 0x7f66fc06a7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6704000c00 tx=0x7f6704008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:09.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:09.834+0000 7f6713fff700 1 -- 172.21.15.7:0/2158128783 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f670c051ec0 con 0x7f671c0caa30 2023-03-03T08:44:10.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.024+0000 7f67218a5700 1 -- 172.21.15.7:0/2158128783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f671c1ea3c0 con 0x7f671c0caa30 2023-03-03T08:44:10.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.025+0000 7f6713fff700 1 -- 172.21.15.7:0/2158128783 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f670c0554e0 con 0x7f671c0caa30 2023-03-03T08:44:10.026 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:44:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 -- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66fc0682f0 msgr2=0x7f66fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66fc0682f0 0x7f66fc06a7a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6704000c00 tx=0x7f6704008030 comp rx=0 tx=0).stop 2023-03-03T08:44:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 -- 172.21.15.7:0/2158128783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 msgr2=0x7f671c14d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 0x7f671c14d310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f670c00e760 tx=0x7f670c00eb20 comp rx=0 tx=0).stop 2023-03-03T08:44:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 -- 172.21.15.7:0/2158128783 shutdown_connections 2023-03-03T08:44:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f671c0cae00 0x7f671c14d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f671c0c00e0 0x7f671c0c23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66fc0682f0 0x7f66fc06a7a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 --2- 172.21.15.7:0/2158128783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f671c0caa30 0x7f671c14d310 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 -- 172.21.15.7:0/2158128783 >> 172.21.15.7:0/2158128783 conn(0x7f671c066e40 msgr2=0x7f671c153cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 -- 172.21.15.7:0/2158128783 shutdown_connections 2023-03-03T08:44:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.027+0000 7f67218a5700 1 -- 172.21.15.7:0/2158128783 wait complete. 2023-03-03T08:44:10.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-03-03T08:44:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.167+0000 7f957b950700 1 Processor -- start 2023-03-03T08:44:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.167+0000 7f957b950700 1 -- start start 2023-03-03T08:44:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f957b950700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95740c1da0 0x7f95741550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f957b950700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95740c7a60 con 0x7f957406d870 2023-03-03T08:44:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f957b950700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95740c3620 con 0x7f95740c3e30 2023-03-03T08:44:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f957b950700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95740c2af0 con 0x7f95740c1da0 2023-03-03T08:44:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f95796ec700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f957406d870 0x7f95740c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46358/0 (socket says 172.21.15.7:46358) 2023-03-03T08:44:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f95796ec700 1 -- 172.21.15.7:0/2719579354 learned_addr learned my addr 172.21.15.7:0/2719579354 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901628779 0 0) 0x7f95740c7a60 con 0x7f957406d870 2023-03-03T08:44:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f9578eeb700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95740c1da0 0x7f95741550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95600035b0 con 0x7f957406d870 2023-03-03T08:44:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251576412 0 0) 0x7f95600035b0 con 0x7f957406d870 2023-03-03T08:44:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95740c1da0 msgr2=0x7f95741550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95740c1da0 0x7f95741550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 >> v1:172.21.15.143:6789/0 conn(0x7f95740c3e30 legacy=0x7f957415ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95741545e0 con 0x7f957406d870 2023-03-03T08:44:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f9570001670 con 0x7f957406d870 2023-03-03T08:44:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 >> v1:172.21.15.7:6789/0 conn(0x7f957406d870 legacy=0x7f95740c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 0x7f9560003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95740c1da0 0x7f9560004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9560004d90 0x7f9560005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956000baf0 con 0x7f9560003aa0 2023-03-03T08:44:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.168+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956000bc30 con 0x7f9560004d90 2023-03-03T08:44:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956000bdf0 con 0x7f95740c1da0 2023-03-03T08:44:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f95796ec700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 0x7f9560003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f9578eeb700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95740c1da0 0x7f9560004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f9579eed700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9560004d90 0x7f9560005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f95796ec700 1 -- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95740c1da0 msgr2=0x7f9560004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f95796ec700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95740c1da0 0x7f9560004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f95796ec700 1 -- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9560004d90 msgr2=0x7f9560005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f95796ec700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9560004d90 0x7f9560005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f95796ec700 1 -- 172.21.15.7:0/2719579354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600035b0 con 0x7f9560003aa0 2023-03-03T08:44:10.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f9578eeb700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95740c1da0 0x7f9560004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f9579eed700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9560004d90 0x7f9560005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f95796ec700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 0x7f9560003eb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f957000c240 tx=0x7f9560005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77aeaa3ef9d8f65 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9570007c10 con 0x7f9560003aa0 2023-03-03T08:44:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957000cde0 con 0x7f9560003aa0 2023-03-03T08:44:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.169+0000 7f956bfff700 1 -- 172.21.15.7:0/2719579354 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9570005ba0 con 0x7f9560003aa0 2023-03-03T08:44:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.170+0000 7f957b950700 1 -- 172.21.15.7:0/2719579354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 msgr2=0x7f9560003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.170+0000 7f957b950700 1 --2- 172.21.15.7:0/2719579354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 0x7f9560003eb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f957000c240 tx=0x7f9560005880 comp rx=0 tx=0).stop 2023-03-03T08:44:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.170+0000 7f9579eed700 1 -- 172.21.15.7:0/2719579354 reap_dead start 2023-03-03T08:44:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.170+0000 7f957b950700 1 -- 172.21.15.7:0/2719579354 shutdown_connections 2023-03-03T08:44:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.170+0000 7f957b950700 1 -- 172.21.15.7:0/2719579354 >> 172.21.15.7:0/2719579354 conn(0x7f957406c110 msgr2=0x7f957406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.170+0000 7f957b950700 1 -- 172.21.15.7:0/2719579354 shutdown_connections 2023-03-03T08:44:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.170+0000 7f957b950700 1 -- 172.21.15.7:0/2719579354 wait complete. 2023-03-03T08:44:10.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.171+0000 7f957b950700 1 Processor -- start 2023-03-03T08:44:10.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.171+0000 7f957b950700 1 -- start start 2023-03-03T08:44:10.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f957b950700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 0x7f9574156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f957b950700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f957413a430 0x7f9574139770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f957b950700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9574157250 0x7f9574139cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f957b950700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95741ea860 con 0x7f9560003aa0 2023-03-03T08:44:10.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f957b950700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957413a1f0 con 0x7f957413a430 2023-03-03T08:44:10.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f957b950700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95741e9ce0 con 0x7f9574157250 2023-03-03T08:44:10.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f95796ec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 0x7f9574156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f95796ec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 0x7f9574156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52390/0 (socket says 172.21.15.7:52390) 2023-03-03T08:44:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f95796ec700 1 -- 172.21.15.7:0/1604845373 learned_addr learned my addr 172.21.15.7:0/1604845373 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f9578eeb700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f957413a430 0x7f9574139770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f9579eed700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9574157250 0x7f9574139cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f95796ec700 1 -- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9574157250 msgr2=0x7f9574139cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f95796ec700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9574157250 0x7f9574139cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f95796ec700 1 -- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f957413a430 msgr2=0x7f9574139770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f95796ec700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f957413a430 0x7f9574139770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f95796ec700 1 -- 172.21.15.7:0/1604845373 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9574154ba0 con 0x7f9560003aa0 2023-03-03T08:44:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f9578eeb700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f957413a430 0x7f9574139770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f9579eed700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9574157250 0x7f9574139cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f95796ec700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 0x7f9574156d10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f957000c830 tx=0x7f9570001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f9569ffb700 1 -- 172.21.15.7:0/1604845373 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9570001640 con 0x7f9560003aa0 2023-03-03T08:44:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f9569ffb700 1 -- 172.21.15.7:0/1604845373 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95700070d0 con 0x7f9560003aa0 2023-03-03T08:44:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f9569ffb700 1 -- 172.21.15.7:0/1604845373 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9570007460 con 0x7f9560003aa0 2023-03-03T08:44:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.172+0000 7f957b950700 1 -- 172.21.15.7:0/1604845373 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95741548c0 con 0x7f9560003aa0 2023-03-03T08:44:10.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.173+0000 7f957b950700 1 -- 172.21.15.7:0/1604845373 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957406dc40 con 0x7f9560003aa0 2023-03-03T08:44:10.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.174+0000 7f957b950700 1 -- 172.21.15.7:0/1604845373 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9574069730 con 0x7f9560003aa0 2023-03-03T08:44:10.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.174+0000 7f9569ffb700 1 -- 172.21.15.7:0/1604845373 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9570010070 con 0x7f9560003aa0 2023-03-03T08:44:10.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.174+0000 7f9569ffb700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95580682f0 0x7f955806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.175+0000 7f9569ffb700 1 -- 172.21.15.7:0/1604845373 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f9570087490 con 0x7f9560003aa0 2023-03-03T08:44:10.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.175+0000 7f9578eeb700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95580682f0 0x7f955806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.175+0000 7f9578eeb700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95580682f0 0x7f955806a7a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f95740c2df0 tx=0x7f9564008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:10.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.179+0000 7f9569ffb700 1 -- 172.21.15.7:0/1604845373 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f95700539c0 con 0x7f9560003aa0 2023-03-03T08:44:10.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.372+0000 7f957b950700 1 -- 172.21.15.7:0/1604845373 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f95741558a0 con 0x7f9560003aa0 2023-03-03T08:44:10.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.372+0000 7f9569ffb700 1 -- 172.21.15.7:0/1604845373 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7f957002d300 con 0x7f9560003aa0 2023-03-03T08:44:10.374 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:44:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 -- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95580682f0 msgr2=0x7f955806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95580682f0 0x7f955806a7a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f95740c2df0 tx=0x7f9564008040 comp rx=0 tx=0).stop 2023-03-03T08:44:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 -- 172.21.15.7:0/1604845373 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 msgr2=0x7f9574156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 0x7f9574156d10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f957000c830 tx=0x7f9570001a00 comp rx=0 tx=0).stop 2023-03-03T08:44:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 -- 172.21.15.7:0/1604845373 shutdown_connections 2023-03-03T08:44:10.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9574157250 0x7f9574139cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95580682f0 0x7f955806a7a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9560003aa0 0x7f9574156d10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 --2- 172.21.15.7:0/1604845373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f957413a430 0x7f9574139770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 -- 172.21.15.7:0/1604845373 >> 172.21.15.7:0/1604845373 conn(0x7f957406c110 msgr2=0x7f9574152de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:10.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.374+0000 7f957b950700 1 -- 172.21.15.7:0/1604845373 shutdown_connections 2023-03-03T08:44:10.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.375+0000 7f957b950700 1 -- 172.21.15.7:0/1604845373 wait complete. 2023-03-03T08:44:10.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-03-03T08:44:10.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.515+0000 7f6f2af0f700 1 Processor -- start 2023-03-03T08:44:10.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.515+0000 7f6f2af0f700 1 -- start start 2023-03-03T08:44:10.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f2af0f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6f2406d870 0x7f6f241550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f2af0f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f240c7a60 con 0x7f6f240c1da0 2023-03-03T08:44:10.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f2af0f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f240c3620 con 0x7f6f240c3e30 2023-03-03T08:44:10.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f2af0f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f240c2af0 con 0x7f6f2406d870 2023-03-03T08:44:10.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f23fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6f240c1da0 0x7f6f240c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46378/0 (socket says 172.21.15.7:46378) 2023-03-03T08:44:10.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f23fff700 1 -- 172.21.15.7:0/4108041746 learned_addr learned my addr 172.21.15.7:0/4108041746 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:10.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624534104 0 0) 0x7f6f240c7a60 con 0x7f6f240c1da0 2023-03-03T08:44:10.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f28cab700 1 --2- 172.21.15.7:0/4108041746 >> v2:172.21.15.143:3300/0 conn(0x7f6f2406d870 0x7f6f241550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f140035b0 con 0x7f6f240c1da0 2023-03-03T08:44:10.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787868045 0 0) 0x7f6f140035b0 con 0x7f6f240c1da0 2023-03-03T08:44:10.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 >> v2:172.21.15.143:3300/0 conn(0x7f6f2406d870 msgr2=0x7f6f241550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:10.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 --2- 172.21.15.7:0/4108041746 >> v2:172.21.15.143:3300/0 conn(0x7f6f2406d870 0x7f6f241550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 >> v1:172.21.15.143:6789/0 conn(0x7f6f240c3e30 legacy=0x7f6f2415ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f241545e0 con 0x7f6f240c1da0 2023-03-03T08:44:10.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f6f10001670 con 0x7f6f240c1da0 2023-03-03T08:44:10.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 >> v1:172.21.15.7:6789/0 conn(0x7f6f240c1da0 legacy=0x7f6f240c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f14003a70 0x7f6f14003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f14004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f14005200 0x7f6f1400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1400c0e0 con 0x7f6f14004850 2023-03-03T08:44:10.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1400c220 con 0x7f6f14005200 2023-03-03T08:44:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.516+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1400c3e0 con 0x7f6f14003a70 2023-03-03T08:44:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f23fff700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f14004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f28cab700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f14003a70 0x7f6f14003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f294ac700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f14005200 0x7f6f1400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f23fff700 1 -- 172.21.15.7:0/4108041746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f14003a70 msgr2=0x7f6f14003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f23fff700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f14003a70 0x7f6f14003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f23fff700 1 -- 172.21.15.7:0/4108041746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f14005200 msgr2=0x7f6f1400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f23fff700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f14005200 0x7f6f1400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f294ac700 1 -- 172.21.15.7:0/4108041746 reap_dead start 2023-03-03T08:44:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f23fff700 1 -- 172.21.15.7:0/4108041746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f140035b0 con 0x7f6f14004850 2023-03-03T08:44:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f23fff700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f14004cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000c240 tx=0x7f6f14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2beac3c5540b57f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f10007c10 con 0x7f6f14004850 2023-03-03T08:44:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1000cde0 con 0x7f6f14004850 2023-03-03T08:44:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.517+0000 7f6f237fe700 1 -- 172.21.15.7:0/4108041746 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f10005ba0 con 0x7f6f14004850 2023-03-03T08:44:10.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.518+0000 7f6f2af0f700 1 -- 172.21.15.7:0/4108041746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 msgr2=0x7f6f14004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.518+0000 7f6f2af0f700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f14004cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000c240 tx=0x7f6f14005880 comp rx=0 tx=0).stop 2023-03-03T08:44:10.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.518+0000 7f6f2af0f700 1 -- 172.21.15.7:0/4108041746 shutdown_connections 2023-03-03T08:44:10.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.518+0000 7f6f2af0f700 1 --2- 172.21.15.7:0/4108041746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f14004cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.518+0000 7f6f2af0f700 1 -- 172.21.15.7:0/4108041746 >> 172.21.15.7:0/4108041746 conn(0x7f6f2406c110 msgr2=0x7f6f2406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:10.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.518+0000 7f6f2af0f700 1 -- 172.21.15.7:0/4108041746 shutdown_connections 2023-03-03T08:44:10.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.518+0000 7f6f2af0f700 1 -- 172.21.15.7:0/4108041746 wait complete. 2023-03-03T08:44:10.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.519+0000 7f6f2af0f700 1 Processor -- start 2023-03-03T08:44:10.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.519+0000 7f6f2af0f700 1 -- start start 2023-03-03T08:44:10.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.520+0000 7f6f2af0f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f240c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.520+0000 7f6f2af0f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f24153d70 0x7f6f241331a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.520+0000 7f6f2af0f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f24158570 0x7f6f241336e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.520+0000 7f6f2af0f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f241e9e20 con 0x7f6f14004850 2023-03-03T08:44:10.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.520+0000 7f6f2af0f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f241ea080 con 0x7f6f24153d70 2023-03-03T08:44:10.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.520+0000 7f6f2af0f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f241ea360 con 0x7f6f24158570 2023-03-03T08:44:10.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f28cab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f240c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f28cab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f240c6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52410/0 (socket says 172.21.15.7:52410) 2023-03-03T08:44:10.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f28cab700 1 -- 172.21.15.7:0/117157183 learned_addr learned my addr 172.21.15.7:0/117157183 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:10.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f28cab700 1 -- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f24158570 msgr2=0x7f6f241336e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f28cab700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f24158570 0x7f6f241336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f28cab700 1 -- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f24153d70 msgr2=0x7f6f241331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f23fff700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f24153d70 0x7f6f241331a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f28cab700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f24153d70 0x7f6f241331a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f28cab700 1 -- 172.21.15.7:0/117157183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f24154ba0 con 0x7f6f14004850 2023-03-03T08:44:10.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f28cab700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f240c6b00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6f18011a20 tx=0x7f6f18011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:10.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f217fa700 1 -- 172.21.15.7:0/117157183 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1800fa00 con 0x7f6f14004850 2023-03-03T08:44:10.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f217fa700 1 -- 172.21.15.7:0/117157183 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1800fb60 con 0x7f6f14004850 2023-03-03T08:44:10.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f2af0f700 1 -- 172.21.15.7:0/117157183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f241548c0 con 0x7f6f14004850 2023-03-03T08:44:10.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f217fa700 1 -- 172.21.15.7:0/117157183 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f18014440 con 0x7f6f14004850 2023-03-03T08:44:10.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.521+0000 7f6f2af0f700 1 -- 172.21.15.7:0/117157183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2406dc40 con 0x7f6f14004850 2023-03-03T08:44:10.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.522+0000 7f6f2af0f700 1 -- 172.21.15.7:0/117157183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f240c8190 con 0x7f6f14004850 2023-03-03T08:44:10.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.523+0000 7f6f217fa700 1 -- 172.21.15.7:0/117157183 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6f180145a0 con 0x7f6f14004850 2023-03-03T08:44:10.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.523+0000 7f6f217fa700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f080682f0 0x7f6f0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.523+0000 7f6f217fa700 1 -- 172.21.15.7:0/117157183 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6f180869b0 con 0x7f6f14004850 2023-03-03T08:44:10.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.523+0000 7f6f23fff700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f080682f0 0x7f6f0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.524+0000 7f6f23fff700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f080682f0 0x7f6f0806a7a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6f10015040 tx=0x7f6f10007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:10.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.527+0000 7f6f217fa700 1 -- 172.21.15.7:0/117157183 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6f18052d90 con 0x7f6f14004850 2023-03-03T08:44:10.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.720+0000 7f6f2af0f700 1 -- 172.21.15.7:0/117157183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f24074bb0 con 0x7f6f14004850 2023-03-03T08:44:10.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.720+0000 7f6f217fa700 1 -- 172.21.15.7:0/117157183 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f6f180563b0 con 0x7f6f14004850 2023-03-03T08:44:10.722 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:44:10.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 -- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f080682f0 msgr2=0x7f6f0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f080682f0 0x7f6f0806a7a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6f10015040 tx=0x7f6f10007000 comp rx=0 tx=0).stop 2023-03-03T08:44:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 -- 172.21.15.7:0/117157183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 msgr2=0x7f6f240c6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f240c6b00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6f18011a20 tx=0x7f6f18011de0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 -- 172.21.15.7:0/117157183 shutdown_connections 2023-03-03T08:44:10.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f24158570 0x7f6f241336e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f080682f0 0x7f6f0806a7a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f14004850 0x7f6f240c6b00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 --2- 172.21.15.7:0/117157183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f24153d70 0x7f6f241331a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 -- 172.21.15.7:0/117157183 >> 172.21.15.7:0/117157183 conn(0x7f6f2406c110 msgr2=0x7f6f24153660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 -- 172.21.15.7:0/117157183 shutdown_connections 2023-03-03T08:44:10.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.722+0000 7f6f2af0f700 1 -- 172.21.15.7:0/117157183 wait complete. 2023-03-03T08:44:10.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-03-03T08:44:10.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.864+0000 7f06d84b2700 1 Processor -- start 2023-03-03T08:44:10.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.864+0000 7f06d84b2700 1 -- start start 2023-03-03T08:44:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d84b2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06d00719e0 0x7f06d0150930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d84b2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f06d0151e20 0x7f06d00aa750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d84b2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d0152270 con 0x7f06d0153f20 2023-03-03T08:44:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d84b2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d00c2760 con 0x7f06d00719e0 2023-03-03T08:44:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d84b2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d00a93d0 con 0x7f06d0151e20 2023-03-03T08:44:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d624e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06d00719e0 0x7f06d0150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d6a4f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f06d0153f20 0x7f06d0154320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46402/0 (socket says 172.21.15.7:46402) 2023-03-03T08:44:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d6a4f700 1 -- 172.21.15.7:0/882180601 learned_addr learned my addr 172.21.15.7:0/882180601 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d5a4d700 1 --2- 172.21.15.7:0/882180601 >> v2:172.21.15.143:3300/0 conn(0x7f06d0151e20 0x7f06d00aa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d624e700 1 -- 172.21.15.7:0/882180601 >> v2:172.21.15.143:3300/0 conn(0x7f06d0151e20 msgr2=0x7f06d00aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d624e700 1 --2- 172.21.15.7:0/882180601 >> v2:172.21.15.143:3300/0 conn(0x7f06d0151e20 0x7f06d00aa750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d624e700 1 -- 172.21.15.7:0/882180601 >> v1:172.21.15.7:6789/0 conn(0x7f06d0153f20 legacy=0x7f06d0154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d624e700 1 -- 172.21.15.7:0/882180601 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d00a96b0 con 0x7f06d00719e0 2023-03-03T08:44:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d524c700 1 -- 172.21.15.7:0/882180601 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712544318 0 0) 0x7f06d0152270 con 0x7f06d0153f20 2023-03-03T08:44:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.865+0000 7f06d5a4d700 1 --2- 172.21.15.7:0/882180601 >> v2:172.21.15.143:3300/0 conn(0x7f06d0151e20 0x7f06d00aa750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d624e700 1 --2- 172.21.15.7:0/882180601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d00719e0 0x7f06d0150930 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f06d00c73e0 tx=0x7f06c0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59807bba9eaec57 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d524c700 1 -- 172.21.15.7:0/882180601 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0011830 con 0x7f06d00719e0 2023-03-03T08:44:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d524c700 1 -- 172.21.15.7:0/882180601 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c0011990 con 0x7f06d00719e0 2023-03-03T08:44:10.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d524c700 1 -- 172.21.15.7:0/882180601 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c0011d20 con 0x7f06d00719e0 2023-03-03T08:44:10.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d84b2700 1 -- 172.21.15.7:0/882180601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d00719e0 msgr2=0x7f06d0150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d84b2700 1 --2- 172.21.15.7:0/882180601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d00719e0 0x7f06d0150930 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f06d00c73e0 tx=0x7f06c0009b20 comp rx=0 tx=0).stop 2023-03-03T08:44:10.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d84b2700 1 -- 172.21.15.7:0/882180601 shutdown_connections 2023-03-03T08:44:10.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d84b2700 1 --2- 172.21.15.7:0/882180601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d00719e0 0x7f06d0150930 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d84b2700 1 --2- 172.21.15.7:0/882180601 >> v2:172.21.15.143:3300/0 conn(0x7f06d0151e20 0x7f06d00aa750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d84b2700 1 -- 172.21.15.7:0/882180601 >> 172.21.15.7:0/882180601 conn(0x7f06d00c62e0 msgr2=0x7f06d00c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:10.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d84b2700 1 -- 172.21.15.7:0/882180601 shutdown_connections 2023-03-03T08:44:10.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.866+0000 7f06d84b2700 1 -- 172.21.15.7:0/882180601 wait complete. 2023-03-03T08:44:10.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.868+0000 7f06d84b2700 1 Processor -- start 2023-03-03T08:44:10.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.868+0000 7f06d84b2700 1 -- start start 2023-03-03T08:44:10.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d84b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06d00719e0 0x7f06d00c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d84b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06d0151e20 0x7f06d014b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d84b2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d0153f20 0x7f06d014bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d84b2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d00bbaa0 con 0x7f06d0153f20 2023-03-03T08:44:10.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d84b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d00bcf50 con 0x7f06d00719e0 2023-03-03T08:44:10.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d84b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d014c1f0 con 0x7f06d0151e20 2023-03-03T08:44:10.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d6a4f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d0153f20 0x7f06d014bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d6a4f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d0153f20 0x7f06d014bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52428/0 (socket says 172.21.15.7:52428) 2023-03-03T08:44:10.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d6a4f700 1 -- 172.21.15.7:0/3215229122 learned_addr learned my addr 172.21.15.7:0/3215229122 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:10.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d624e700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06d00719e0 0x7f06d00c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d5a4d700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06d0151e20 0x7f06d014b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d6a4f700 1 -- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06d0151e20 msgr2=0x7f06d014b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d6a4f700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06d0151e20 0x7f06d014b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d6a4f700 1 -- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06d00719e0 msgr2=0x7f06d00c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:10.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d6a4f700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06d00719e0 0x7f06d00c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:10.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d6a4f700 1 -- 172.21.15.7:0/3215229122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c00094a0 con 0x7f06d0153f20 2023-03-03T08:44:10.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.869+0000 7f06d5a4d700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06d0151e20 0x7f06d014b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:10.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.870+0000 7f06d6a4f700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d0153f20 0x7f06d014bcb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f06c8002230 tx=0x7f06c800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:10.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.870+0000 7f06c6ffd700 1 -- 172.21.15.7:0/3215229122 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c800ab10 con 0x7f06d0153f20 2023-03-03T08:44:10.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.870+0000 7f06c6ffd700 1 -- 172.21.15.7:0/3215229122 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c800ac90 con 0x7f06d0153f20 2023-03-03T08:44:10.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.870+0000 7f06d84b2700 1 -- 172.21.15.7:0/3215229122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d00a9c70 con 0x7f06d0153f20 2023-03-03T08:44:10.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.870+0000 7f06c6ffd700 1 -- 172.21.15.7:0/3215229122 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c80070d0 con 0x7f06d0153f20 2023-03-03T08:44:10.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.870+0000 7f06d84b2700 1 -- 172.21.15.7:0/3215229122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d00a9990 con 0x7f06d0153f20 2023-03-03T08:44:10.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.871+0000 7f06d84b2700 1 -- 172.21.15.7:0/3215229122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d00a9f50 con 0x7f06d0153f20 2023-03-03T08:44:10.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.871+0000 7f06c6ffd700 1 -- 172.21.15.7:0/3215229122 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f06c8010070 con 0x7f06d0153f20 2023-03-03T08:44:10.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.871+0000 7f06c6ffd700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06b40682f0 0x7f06b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:10.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.872+0000 7f06c6ffd700 1 -- 172.21.15.7:0/3215229122 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f06c8085950 con 0x7f06d0153f20 2023-03-03T08:44:10.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.872+0000 7f06d624e700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06b40682f0 0x7f06b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:10.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.872+0000 7f06d624e700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06b40682f0 0x7f06b406a7a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f06c00090b0 tx=0x7f06c001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:10.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:10.876+0000 7f06c6ffd700 1 -- 172.21.15.7:0/3215229122 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f06c8051e80 con 0x7f06d0153f20 2023-03-03T08:44:11.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.069+0000 7f06d84b2700 1 -- 172.21.15.7:0/3215229122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f06d0074bb0 con 0x7f06d0153f20 2023-03-03T08:44:11.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.069+0000 7f06c6ffd700 1 -- 172.21.15.7:0/3215229122 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7f06c8018020 con 0x7f06d0153f20 2023-03-03T08:44:11.071 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:44:11.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 -- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06b40682f0 msgr2=0x7f06b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06b40682f0 0x7f06b406a7a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f06c00090b0 tx=0x7f06c001c040 comp rx=0 tx=0).stop 2023-03-03T08:44:11.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 -- 172.21.15.7:0/3215229122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d0153f20 msgr2=0x7f06d014bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d0153f20 0x7f06d014bcb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f06c8002230 tx=0x7f06c800cf50 comp rx=0 tx=0).stop 2023-03-03T08:44:11.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 -- 172.21.15.7:0/3215229122 shutdown_connections 2023-03-03T08:44:11.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06b40682f0 0x7f06b406a7a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06d0153f20 0x7f06d014bcb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06d00719e0 0x7f06d00c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 --2- 172.21.15.7:0/3215229122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06d0151e20 0x7f06d014b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 -- 172.21.15.7:0/3215229122 >> 172.21.15.7:0/3215229122 conn(0x7f06d00c62e0 msgr2=0x7f06d0152e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:11.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 -- 172.21.15.7:0/3215229122 shutdown_connections 2023-03-03T08:44:11.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.071+0000 7f06d84b2700 1 -- 172.21.15.7:0/3215229122 wait complete. 2023-03-03T08:44:11.084 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.208+0000 7f94212c2700 1 Processor -- start 2023-03-03T08:44:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.209+0000 7f94212c2700 1 -- start start 2023-03-03T08:44:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.209+0000 7f94212c2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f941c071e80 0x7f941c151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.209+0000 7f94212c2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f941c1510d0 con 0x7f941c14e900 2023-03-03T08:44:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.209+0000 7f94212c2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f941c0c3740 con 0x7f941c072250 2023-03-03T08:44:11.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.209+0000 7f94212c2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f941c070670 con 0x7f941c071e80 2023-03-03T08:44:11.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.209+0000 7f941affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f941c071e80 0x7f941c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.209+0000 7f941b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f941c14e900 0x7f941c15ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46416/0 (socket says 172.21.15.7:46416) 2023-03-03T08:44:11.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.209+0000 7f941b7fe700 1 -- 172.21.15.7:0/2377629246 learned_addr learned my addr 172.21.15.7:0/2377629246 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:11.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.210+0000 7f941affd700 1 -- 172.21.15.7:0/2377629246 >> v1:172.21.15.143:6789/0 conn(0x7f941c072250 legacy=0x7f941c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.210+0000 7f941affd700 1 -- 172.21.15.7:0/2377629246 >> v1:172.21.15.7:6789/0 conn(0x7f941c14e900 legacy=0x7f941c15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.210+0000 7f941affd700 1 -- 172.21.15.7:0/2377629246 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941c0c39e0 con 0x7f941c071e80 2023-03-03T08:44:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.210+0000 7f9419ffb700 1 -- 172.21.15.7:0/2377629246 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617947755 0 0) 0x7f941c1510d0 con 0x7f941c14e900 2023-03-03T08:44:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.210+0000 7f941affd700 1 --2- 172.21.15.7:0/2377629246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 0x7f941c151820 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f940c009840 tx=0x7f940c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91df4b196e2ff01 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.210+0000 7f9419ffb700 1 -- 172.21.15.7:0/2377629246 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c011830 con 0x7f941c071e80 2023-03-03T08:44:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.210+0000 7f9419ffb700 1 -- 172.21.15.7:0/2377629246 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940c011990 con 0x7f941c071e80 2023-03-03T08:44:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.210+0000 7f9419ffb700 1 -- 172.21.15.7:0/2377629246 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c011d20 con 0x7f941c071e80 2023-03-03T08:44:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.211+0000 7f94212c2700 1 -- 172.21.15.7:0/2377629246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 msgr2=0x7f941c151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.211+0000 7f94212c2700 1 --2- 172.21.15.7:0/2377629246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 0x7f941c151820 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f940c009840 tx=0x7f940c009b50 comp rx=0 tx=0).stop 2023-03-03T08:44:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.211+0000 7f94212c2700 1 -- 172.21.15.7:0/2377629246 shutdown_connections 2023-03-03T08:44:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.211+0000 7f94212c2700 1 --2- 172.21.15.7:0/2377629246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 0x7f941c151820 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.211+0000 7f94212c2700 1 -- 172.21.15.7:0/2377629246 >> 172.21.15.7:0/2377629246 conn(0x7f941c0a94d0 msgr2=0x7f941c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.211+0000 7f94212c2700 1 -- 172.21.15.7:0/2377629246 shutdown_connections 2023-03-03T08:44:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.211+0000 7f94212c2700 1 -- 172.21.15.7:0/2377629246 wait complete. 2023-03-03T08:44:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.212+0000 7f94212c2700 1 Processor -- start 2023-03-03T08:44:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.212+0000 7f94212c2700 1 -- start start 2023-03-03T08:44:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94212c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 0x7f941c1e8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94212c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f941c072250 0x7f941c0abba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94212c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f941c14e900 0x7f941c147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94212c2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c1e8ea0 con 0x7f941c071e80 2023-03-03T08:44:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94212c2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941c0c6b00 con 0x7f941c14e900 2023-03-03T08:44:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94212c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941c0c6de0 con 0x7f941c072250 2023-03-03T08:44:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f941affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 0x7f941c1e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f941affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 0x7f941c1e8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52438/0 (socket says 172.21.15.7:52438) 2023-03-03T08:44:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f941affd700 1 -- 172.21.15.7:0/2614234830 learned_addr learned my addr 172.21.15.7:0/2614234830 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f941affd700 1 -- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f941c072250 msgr2=0x7f941c0abba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f941affd700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f941c072250 0x7f941c0abba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f941affd700 1 -- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f941c14e900 msgr2=0x7f941c147870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f941affd700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f941c14e900 0x7f941c147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f941affd700 1 -- 172.21.15.7:0/2614234830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c0094a0 con 0x7f941c071e80 2023-03-03T08:44:11.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f941affd700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 0x7f941c1e8960 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f940c0090b0 tx=0x7f940c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:11.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94137fe700 1 -- 172.21.15.7:0/2614234830 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c01a940 con 0x7f941c071e80 2023-03-03T08:44:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94137fe700 1 -- 172.21.15.7:0/2614234830 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940c01aaa0 con 0x7f941c071e80 2023-03-03T08:44:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94212c2700 1 -- 172.21.15.7:0/2614234830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f941c06c9a0 con 0x7f941c071e80 2023-03-03T08:44:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94137fe700 1 -- 172.21.15.7:0/2614234830 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c01ae10 con 0x7f941c071e80 2023-03-03T08:44:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.213+0000 7f94212c2700 1 -- 172.21.15.7:0/2614234830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f941c151300 con 0x7f941c071e80 2023-03-03T08:44:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.214+0000 7f94212c2700 1 -- 172.21.15.7:0/2614234830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941c06cb30 con 0x7f941c071e80 2023-03-03T08:44:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.215+0000 7f94137fe700 1 -- 172.21.15.7:0/2614234830 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f940c00f050 con 0x7f941c071e80 2023-03-03T08:44:11.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.215+0000 7f94137fe700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93fc0682f0 0x7f93fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.215+0000 7f94137fe700 1 -- 172.21.15.7:0/2614234830 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f940c086fb0 con 0x7f941c071e80 2023-03-03T08:44:11.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.215+0000 7f941a7fc700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93fc0682f0 0x7f93fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.216+0000 7f941a7fc700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93fc0682f0 0x7f93fc06a7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9404000f60 tx=0x7f9404007a40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:11.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.219+0000 7f94137fe700 1 -- 172.21.15.7:0/2614234830 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f940c053510 con 0x7f941c071e80 2023-03-03T08:44:11.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.458+0000 7f94212c2700 1 -- 172.21.15.7:0/2614234830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f941c0c5160 con 0x7f941c071e80 2023-03-03T08:44:11.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.459+0000 7f94137fe700 1 -- 172.21.15.7:0/2614234830 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f941c0c5160 con 0x7f941c071e80 2023-03-03T08:44:11.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:11.461 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:11.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.460+0000 7f94212c2700 1 -- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93fc0682f0 msgr2=0x7f93fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.460+0000 7f94212c2700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93fc0682f0 0x7f93fc06a7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9404000f60 tx=0x7f9404007a40 comp rx=0 tx=0).stop 2023-03-03T08:44:11.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.460+0000 7f94212c2700 1 -- 172.21.15.7:0/2614234830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 msgr2=0x7f941c1e8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.460+0000 7f94212c2700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 0x7f941c1e8960 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f940c0090b0 tx=0x7f940c004750 comp rx=0 tx=0).stop 2023-03-03T08:44:11.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.461+0000 7f94212c2700 1 -- 172.21.15.7:0/2614234830 shutdown_connections 2023-03-03T08:44:11.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.461+0000 7f94212c2700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f941c14e900 0x7f941c147870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.461+0000 7f94212c2700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93fc0682f0 0x7f93fc06a7a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.461+0000 7f94212c2700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f941c071e80 0x7f941c1e8960 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.461+0000 7f94212c2700 1 --2- 172.21.15.7:0/2614234830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f941c072250 0x7f941c0abba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.461+0000 7f94212c2700 1 -- 172.21.15.7:0/2614234830 >> 172.21.15.7:0/2614234830 conn(0x7f941c0a94d0 msgr2=0x7f941c06a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:11.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.461+0000 7f94212c2700 1 -- 172.21.15.7:0/2614234830 shutdown_connections 2023-03-03T08:44:11.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.461+0000 7f94212c2700 1 -- 172.21.15.7:0/2614234830 wait complete. 2023-03-03T08:44:11.464 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:11.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:44:11.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.597+0000 7fd58005f700 1 Processor -- start 2023-03-03T08:44:11.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.597+0000 7fd58005f700 1 -- start start 2023-03-03T08:44:11.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd58005f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd578151e80 0x7fd5780c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd58005f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd5780c6fe0 0x7fd5780bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd58005f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5780c08d0 con 0x7fd578155410 2023-03-03T08:44:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd58005f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd578152250 con 0x7fd5780c6fe0 2023-03-03T08:44:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd58005f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5780c0fb0 con 0x7fd578151e80 2023-03-03T08:44:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd57d5fa700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd578155410 0x7fd5780bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46426/0 (socket says 172.21.15.7:46426) 2023-03-03T08:44:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd57d5fa700 1 -- 172.21.15.7:0/2932352670 learned_addr learned my addr 172.21.15.7:0/2932352670 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:11.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd57e5fc700 1 --2- 172.21.15.7:0/2932352670 >> v2:172.21.15.143:3300/0 conn(0x7fd5780c6fe0 0x7fd5780bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd57ddfb700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd578151e80 0x7fd5780c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828104062 0 0) 0x7fd5780c08d0 con 0x7fd578155410 2023-03-03T08:44:11.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.598+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5640035b0 con 0x7fd578155410 2023-03-03T08:44:11.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3786791377 0 0) 0x7fd5640035b0 con 0x7fd578155410 2023-03-03T08:44:11.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd578151e80 msgr2=0x7fd5780c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:11.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd578151e80 0x7fd5780c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 >> v2:172.21.15.143:3300/0 conn(0x7fd5780c6fe0 msgr2=0x7fd5780bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:11.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 --2- 172.21.15.7:0/2932352670 >> v2:172.21.15.143:3300/0 conn(0x7fd5780c6fe0 0x7fd5780bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd578154930 con 0x7fd578155410 2023-03-03T08:44:11.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd574001670 con 0x7fd578155410 2023-03-03T08:44:11.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57e5fc700 1 --2- 172.21.15.7:0/2932352670 >> v2:172.21.15.143:3300/0 conn(0x7fd5780c6fe0 0x7fd5780bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:11.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 >> v1:172.21.15.7:6789/0 conn(0x7fd578155410 legacy=0x7fd5780bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd564003aa0 0x7fd564003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd564004850 0x7fd564004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd564005200 0x7fd56400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56400bf50 con 0x7fd564003aa0 2023-03-03T08:44:11.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56400c090 con 0x7fd564004850 2023-03-03T08:44:11.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56400c1d0 con 0x7fd564005200 2023-03-03T08:44:11.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57ddfb700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd578151e80 0x7fd5780c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:11.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57ddfb700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd564003aa0 0x7fd564003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57d5fa700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd564004850 0x7fd564004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57e5fc700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd564005200 0x7fd56400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57ddfb700 1 -- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd564005200 msgr2=0x7fd56400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:11.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57ddfb700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd564005200 0x7fd56400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57ddfb700 1 -- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd564004850 msgr2=0x7fd564004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:11.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57ddfb700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd564004850 0x7fd564004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.599+0000 7fd57ddfb700 1 -- 172.21.15.7:0/2932352670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5640035b0 con 0x7fd564003aa0 2023-03-03T08:44:11.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd57ddfb700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd564003aa0 0x7fd564003eb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd57806b320 tx=0x7fd564005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9d16b59e0b86d7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568004030 con 0x7fd564003aa0 2023-03-03T08:44:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd568004190 con 0x7fd564003aa0 2023-03-03T08:44:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd57cdf9700 1 -- 172.21.15.7:0/2932352670 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd568004540 con 0x7fd564003aa0 2023-03-03T08:44:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd58005f700 1 -- 172.21.15.7:0/2932352670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd564003aa0 msgr2=0x7fd564003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd58005f700 1 --2- 172.21.15.7:0/2932352670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd564003aa0 0x7fd564003eb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd57806b320 tx=0x7fd564005880 comp rx=0 tx=0).stop 2023-03-03T08:44:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd57e5fc700 1 -- 172.21.15.7:0/2932352670 reap_dead start 2023-03-03T08:44:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd58005f700 1 -- 172.21.15.7:0/2932352670 shutdown_connections 2023-03-03T08:44:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd58005f700 1 -- 172.21.15.7:0/2932352670 >> 172.21.15.7:0/2932352670 conn(0x7fd578071630 msgr2=0x7fd57806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd58005f700 1 -- 172.21.15.7:0/2932352670 shutdown_connections 2023-03-03T08:44:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.600+0000 7fd58005f700 1 -- 172.21.15.7:0/2932352670 wait complete. 2023-03-03T08:44:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.601+0000 7fd58005f700 1 Processor -- start 2023-03-03T08:44:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd58005f700 1 -- start start 2023-03-03T08:44:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd58005f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd564003aa0 0x7fd5781edab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd58005f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5781e1740 0x7fd57814d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd58005f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57814fc90 0x7fd578150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd58005f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5781edff0 con 0x7fd5781e1740 2023-03-03T08:44:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd58005f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5781ed060 con 0x7fd564003aa0 2023-03-03T08:44:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd58005f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5781ed620 con 0x7fd57814fc90 2023-03-03T08:44:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd57d5fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5781e1740 0x7fd57814d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd57d5fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5781e1740 0x7fd57814d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52458/0 (socket says 172.21.15.7:52458) 2023-03-03T08:44:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd57d5fa700 1 -- 172.21.15.7:0/3684177191 learned_addr learned my addr 172.21.15.7:0/3684177191 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd57ddfb700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd564003aa0 0x7fd5781edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.602+0000 7fd57e5fc700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57814fc90 0x7fd578150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd57d5fa700 1 -- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57814fc90 msgr2=0x7fd578150100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd57d5fa700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57814fc90 0x7fd578150100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd57d5fa700 1 -- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd564003aa0 msgr2=0x7fd5781edab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd57d5fa700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd564003aa0 0x7fd5781edab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd57d5fa700 1 -- 172.21.15.7:0/3684177191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd578154ef0 con 0x7fd5781e1740 2023-03-03T08:44:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd57ddfb700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd564003aa0 0x7fd5781edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd57d5fa700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5781e1740 0x7fd57814d0d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd5740018e0 tx=0x7fd574007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd56e7fc700 1 -- 172.21.15.7:0/3684177191 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57400f6c0 con 0x7fd5781e1740 2023-03-03T08:44:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd56e7fc700 1 -- 172.21.15.7:0/3684177191 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57400f840 con 0x7fd5781e1740 2023-03-03T08:44:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd58005f700 1 -- 172.21.15.7:0/3684177191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd578154c10 con 0x7fd5781e1740 2023-03-03T08:44:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd56e7fc700 1 -- 172.21.15.7:0/3684177191 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57400fbf0 con 0x7fd5781e1740 2023-03-03T08:44:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.603+0000 7fd58005f700 1 -- 172.21.15.7:0/3684177191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5780c5220 con 0x7fd5781e1740 2023-03-03T08:44:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.604+0000 7fd58005f700 1 -- 172.21.15.7:0/3684177191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd578074bb0 con 0x7fd5781e1740 2023-03-03T08:44:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.604+0000 7fd56e7fc700 1 -- 172.21.15.7:0/3684177191 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd57400fd50 con 0x7fd5781e1740 2023-03-03T08:44:11.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.605+0000 7fd56e7fc700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd55c068320 0x7fd55c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.605+0000 7fd56e7fc700 1 -- 172.21.15.7:0/3684177191 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fd574086040 con 0x7fd5781e1740 2023-03-03T08:44:11.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.605+0000 7fd57ddfb700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd55c068320 0x7fd55c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.605+0000 7fd57ddfb700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd55c068320 0x7fd55c06a7d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd57806b320 tx=0x7fd568008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:11.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.609+0000 7fd56e7fc700 1 -- 172.21.15.7:0/3684177191 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd574052570 con 0x7fd5781e1740 2023-03-03T08:44:11.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.805+0000 7fd58005f700 1 -- 172.21.15.7:0/3684177191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5780c6860 con 0x7fd5781e1740 2023-03-03T08:44:11.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.807+0000 7fd56e7fc700 1 -- 172.21.15.7:0/3684177191 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21533 (secure 0 0 0) 0x7fd574055b90 con 0x7fd5781e1740 2023-03-03T08:44:11.808 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:11.809 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:44.973251+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:44:11.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 -- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd55c068320 msgr2=0x7fd55c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd55c068320 0x7fd55c06a7d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd57806b320 tx=0x7fd568008070 comp rx=0 tx=0).stop 2023-03-03T08:44:11.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 -- 172.21.15.7:0/3684177191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5781e1740 msgr2=0x7fd57814d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5781e1740 0x7fd57814d0d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd5740018e0 tx=0x7fd574007620 comp rx=0 tx=0).stop 2023-03-03T08:44:11.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 -- 172.21.15.7:0/3684177191 shutdown_connections 2023-03-03T08:44:11.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57814fc90 0x7fd578150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd564003aa0 0x7fd5781edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd55c068320 0x7fd55c06a7d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 --2- 172.21.15.7:0/3684177191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5781e1740 0x7fd57814d0d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 -- 172.21.15.7:0/3684177191 >> 172.21.15.7:0/3684177191 conn(0x7fd578071630 msgr2=0x7fd5780ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:11.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 -- 172.21.15.7:0/3684177191 shutdown_connections 2023-03-03T08:44:11.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.809+0000 7fd58005f700 1 -- 172.21.15.7:0/3684177191 wait complete. 2023-03-03T08:44:11.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:44:11.823 INFO:tasks.mds_thrash.fs.[a]:starting mds_do_thrash for fs a 2023-03-03T08:44:11.824 INFO:tasks.mds_thrash.fs.[a]:waiting for 93 secs before thrashing 2023-03-03T08:44:11.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.946+0000 7fa7fc054700 1 Processor -- start 2023-03-03T08:44:11.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.946+0000 7fa7fc054700 1 -- start start 2023-03-03T08:44:11.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.947+0000 7fa7fc054700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa7f414de50 0x7fa7f406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.947+0000 7fa7fc054700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7f406a7f0 0x7fa7f40c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.947+0000 7fa7fc054700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f40a65b0 con 0x7fa7f40c65c0 2023-03-03T08:44:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.947+0000 7fa7fc054700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f4154e00 con 0x7fa7f406a7f0 2023-03-03T08:44:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.947+0000 7fa7fc054700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f40c3370 con 0x7fa7f414de50 2023-03-03T08:44:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.947+0000 7fa7fa5f1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7f406a7f0 0x7fa7f40c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.947+0000 7fa7f95ef700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa7f40c65c0 0x7fa7f40c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46444/0 (socket says 172.21.15.7:46444) 2023-03-03T08:44:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.947+0000 7fa7f95ef700 1 -- 172.21.15.7:0/511036131 learned_addr learned my addr 172.21.15.7:0/511036131 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.948+0000 7fa7f9df0700 1 --2- 172.21.15.7:0/511036131 >> v2:172.21.15.143:3300/0 conn(0x7fa7f414de50 0x7fa7f406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.948+0000 7fa7fa5f1700 1 -- 172.21.15.7:0/511036131 >> v2:172.21.15.143:3300/0 conn(0x7fa7f414de50 msgr2=0x7fa7f406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.948+0000 7fa7fa5f1700 1 --2- 172.21.15.7:0/511036131 >> v2:172.21.15.143:3300/0 conn(0x7fa7f414de50 0x7fa7f406a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.948+0000 7fa7fa5f1700 1 -- 172.21.15.7:0/511036131 >> v1:172.21.15.7:6789/0 conn(0x7fa7f40c65c0 legacy=0x7fa7f40c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.948+0000 7fa7f8dee700 1 -- 172.21.15.7:0/511036131 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955477701 0 0) 0x7fa7f40a65b0 con 0x7fa7f40c65c0 2023-03-03T08:44:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.948+0000 7fa7fa5f1700 1 -- 172.21.15.7:0/511036131 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f41541f0 con 0x7fa7f406a7f0 2023-03-03T08:44:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.948+0000 7fa7fa5f1700 1 --2- 172.21.15.7:0/511036131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f406a7f0 0x7fa7f40c2e30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40c7760 tx=0x7fa7ec00d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2463b4ae0a47ecec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.948+0000 7fa7f8dee700 1 -- 172.21.15.7:0/511036131 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec010070 con 0x7fa7f406a7f0 2023-03-03T08:44:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.948+0000 7fa7f8dee700 1 -- 172.21.15.7:0/511036131 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ec00b2e0 con 0x7fa7f406a7f0 2023-03-03T08:44:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.948+0000 7fa7f8dee700 1 -- 172.21.15.7:0/511036131 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec00dcb0 con 0x7fa7f406a7f0 2023-03-03T08:44:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.949+0000 7fa7fc054700 1 -- 172.21.15.7:0/511036131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f406a7f0 msgr2=0x7fa7f40c2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.949+0000 7fa7fc054700 1 --2- 172.21.15.7:0/511036131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f406a7f0 0x7fa7f40c2e30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40c7760 tx=0x7fa7ec00d330 comp rx=0 tx=0).stop 2023-03-03T08:44:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.949+0000 7fa7fc054700 1 -- 172.21.15.7:0/511036131 shutdown_connections 2023-03-03T08:44:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.949+0000 7fa7fc054700 1 --2- 172.21.15.7:0/511036131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f406a7f0 0x7fa7f40c2e30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.949+0000 7fa7fc054700 1 --2- 172.21.15.7:0/511036131 >> v2:172.21.15.143:3300/0 conn(0x7fa7f414de50 0x7fa7f406a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.949+0000 7fa7fc054700 1 -- 172.21.15.7:0/511036131 >> 172.21.15.7:0/511036131 conn(0x7fa7f40a9430 msgr2=0x7fa7f40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.949+0000 7fa7fc054700 1 -- 172.21.15.7:0/511036131 shutdown_connections 2023-03-03T08:44:11.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.949+0000 7fa7fc054700 1 -- 172.21.15.7:0/511036131 wait complete. 2023-03-03T08:44:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.950+0000 7fa7fc054700 1 Processor -- start 2023-03-03T08:44:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.950+0000 7fa7fc054700 1 -- start start 2023-03-03T08:44:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fc054700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7f406a7f0 0x7fa7f40a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fc054700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7f40c65c0 0x7fa7f413ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fc054700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f414de50 0x7fa7f413f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fc054700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f414ee10 con 0x7fa7f414de50 2023-03-03T08:44:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fc054700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f41e5ac0 con 0x7fa7f406a7f0 2023-03-03T08:44:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fc054700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f41e5d80 con 0x7fa7f40c65c0 2023-03-03T08:44:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fa5f1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f414de50 0x7fa7f413f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fa5f1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f414de50 0x7fa7f413f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52474/0 (socket says 172.21.15.7:52474) 2023-03-03T08:44:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fa5f1700 1 -- 172.21.15.7:0/1428374997 learned_addr learned my addr 172.21.15.7:0/1428374997 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7f95ef700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7f40c65c0 0x7fa7f413ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7f9df0700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7f406a7f0 0x7fa7f40a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fa5f1700 1 -- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7f40c65c0 msgr2=0x7fa7f413ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fa5f1700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7f40c65c0 0x7fa7f413ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fa5f1700 1 -- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7f406a7f0 msgr2=0x7fa7f40a7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fa5f1700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7f406a7f0 0x7fa7f40a7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fa5f1700 1 -- 172.21.15.7:0/1428374997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec00bc80 con 0x7fa7f414de50 2023-03-03T08:44:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7f95ef700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7f40c65c0 0x7fa7f413ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7f9df0700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7f406a7f0 0x7fa7f40a7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fa5f1700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f414de50 0x7fa7f413f150 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00bfd0 tx=0x7fa7ec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7ea7fc700 1 -- 172.21.15.7:0/1428374997 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec01d630 con 0x7fa7f414de50 2023-03-03T08:44:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7ea7fc700 1 -- 172.21.15.7:0/1428374997 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ec01d7b0 con 0x7fa7f414de50 2023-03-03T08:44:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7ea7fc700 1 -- 172.21.15.7:0/1428374997 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec01db60 con 0x7fa7f414de50 2023-03-03T08:44:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fc054700 1 -- 172.21.15.7:0/1428374997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f41547b0 con 0x7fa7f414de50 2023-03-03T08:44:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.951+0000 7fa7fc054700 1 -- 172.21.15.7:0/1428374997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f41544d0 con 0x7fa7f414de50 2023-03-03T08:44:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.953+0000 7fa7fc054700 1 -- 172.21.15.7:0/1428374997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f4154a90 con 0x7fa7f414de50 2023-03-03T08:44:11.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.953+0000 7fa7ea7fc700 1 -- 172.21.15.7:0/1428374997 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa7ec025030 con 0x7fa7f414de50 2023-03-03T08:44:11.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.953+0000 7fa7ea7fc700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7d80682f0 0x7fa7d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:11.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.953+0000 7fa7ea7fc700 1 -- 172.21.15.7:0/1428374997 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fa7ec089590 con 0x7fa7f414de50 2023-03-03T08:44:11.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.953+0000 7fa7f9df0700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7d80682f0 0x7fa7d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:11.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.954+0000 7fa7f9df0700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7d80682f0 0x7fa7d806a7a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009a00 tx=0x7fa7e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:11.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:11.957+0000 7fa7ea7fc700 1 -- 172.21.15.7:0/1428374997 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7ec055ac0 con 0x7fa7f414de50 2023-03-03T08:44:12.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.153+0000 7fa7fc054700 1 -- 172.21.15.7:0/1428374997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7f4074bb0 con 0x7fa7f414de50 2023-03-03T08:44:12.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.154+0000 7fa7ea7fc700 1 -- 172.21.15.7:0/1428374997 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21533 (secure 0 0 0) 0x7fa7ec00de60 con 0x7fa7f414de50 2023-03-03T08:44:12.157 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:12.158 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:44.973251+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:44:12.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 -- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7d80682f0 msgr2=0x7fa7d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7d80682f0 0x7fa7d806a7a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009a00 tx=0x7fa7e4008040 comp rx=0 tx=0).stop 2023-03-03T08:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 -- 172.21.15.7:0/1428374997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f414de50 msgr2=0x7fa7f413f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f414de50 0x7fa7f413f150 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00bfd0 tx=0x7fa7ec004ad0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 -- 172.21.15.7:0/1428374997 shutdown_connections 2023-03-03T08:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7d80682f0 0x7fa7d806a7a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f414de50 0x7fa7f413f150 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7f406a7f0 0x7fa7f40a7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 --2- 172.21.15.7:0/1428374997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7f40c65c0 0x7fa7f413ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 -- 172.21.15.7:0/1428374997 >> 172.21.15.7:0/1428374997 conn(0x7fa7f40a9430 msgr2=0x7fa7f40c3680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:12.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 -- 172.21.15.7:0/1428374997 shutdown_connections 2023-03-03T08:44:12.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.156+0000 7fa7fc054700 1 -- 172.21.15.7:0/1428374997 wait complete. 2023-03-03T08:44:12.170 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-03T08:43:21.160680+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-03T08:43:28.018457+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-03-03T08:43:32.305037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-03-03T08:43:33.484501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-03-03T08:43:42.509451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-03-03T08:43:42.936898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-03T08:44:12.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-03T08:44:12.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.297+0000 7feaeb5e2700 1 Processor -- start 2023-03-03T08:44:12.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.297+0000 7feaeb5e2700 1 -- start start 2023-03-03T08:44:12.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feaeb5e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 0x7feae40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feaeb5e2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feae4073c30 0x7feae406bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feaeb5e2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7feae406c3e0 0x7feae4157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feaeb5e2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae406d780 con 0x7feae4073c30 2023-03-03T08:44:12.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feaeb5e2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae406cb70 con 0x7feae406c3e0 2023-03-03T08:44:12.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feaeb5e2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae406ce50 con 0x7feae4151ab0 2023-03-03T08:44:12.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae8b7d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feae4073c30 0x7feae406bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae8b7d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feae4073c30 0x7feae406bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52480/0 (socket says 172.21.15.7:52480) 2023-03-03T08:44:12.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae8b7d700 1 -- 172.21.15.7:0/2357297963 learned_addr learned my addr 172.21.15.7:0/2357297963 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:12.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae937e700 1 --2- 172.21.15.7:0/2357297963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 0x7feae40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae8b7d700 1 -- 172.21.15.7:0/2357297963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 msgr2=0x7feae40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:12.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae8b7d700 1 --2- 172.21.15.7:0/2357297963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 0x7feae40736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae9b7f700 1 --2- 172.21.15.7:0/2357297963 >> v2:172.21.15.143:3300/0 conn(0x7feae406c3e0 0x7feae4157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae8b7d700 1 -- 172.21.15.7:0/2357297963 >> v2:172.21.15.143:3300/0 conn(0x7feae406c3e0 msgr2=0x7feae4157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:12.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae8b7d700 1 --2- 172.21.15.7:0/2357297963 >> v2:172.21.15.143:3300/0 conn(0x7feae406c3e0 0x7feae4157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae8b7d700 1 -- 172.21.15.7:0/2357297963 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae406d130 con 0x7feae4073c30 2023-03-03T08:44:12.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae9b7f700 1 --2- 172.21.15.7:0/2357297963 >> v2:172.21.15.143:3300/0 conn(0x7feae406c3e0 0x7feae4157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:12.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feae8b7d700 1 --2- 172.21.15.7:0/2357297963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae4073c30 0x7feae406bea0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7feae40c3db0 tx=0x7fead4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35658912e47f1290 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:12.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.298+0000 7feadbfff700 1 -- 172.21.15.7:0/2357297963 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead400b870 con 0x7feae4073c30 2023-03-03T08:44:12.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feadbfff700 1 -- 172.21.15.7:0/2357297963 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead400b9d0 con 0x7feae4073c30 2023-03-03T08:44:12.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feadbfff700 1 -- 172.21.15.7:0/2357297963 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead400bd60 con 0x7feae4073c30 2023-03-03T08:44:12.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feaeb5e2700 1 -- 172.21.15.7:0/2357297963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae4073c30 msgr2=0x7feae406bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feaeb5e2700 1 --2- 172.21.15.7:0/2357297963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae4073c30 0x7feae406bea0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7feae40c3db0 tx=0x7fead4009bd0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feaeb5e2700 1 -- 172.21.15.7:0/2357297963 shutdown_connections 2023-03-03T08:44:12.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feaeb5e2700 1 --2- 172.21.15.7:0/2357297963 >> v2:172.21.15.143:3300/0 conn(0x7feae406c3e0 0x7feae4157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feaeb5e2700 1 --2- 172.21.15.7:0/2357297963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 0x7feae40736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feaeb5e2700 1 --2- 172.21.15.7:0/2357297963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae4073c30 0x7feae406bea0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feaeb5e2700 1 -- 172.21.15.7:0/2357297963 >> 172.21.15.7:0/2357297963 conn(0x7feae40a94d0 msgr2=0x7feae40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:12.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feaeb5e2700 1 -- 172.21.15.7:0/2357297963 shutdown_connections 2023-03-03T08:44:12.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.299+0000 7feaeb5e2700 1 -- 172.21.15.7:0/2357297963 wait complete. 2023-03-03T08:44:12.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.300+0000 7feaeb5e2700 1 Processor -- start 2023-03-03T08:44:12.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.300+0000 7feaeb5e2700 1 -- start start 2023-03-03T08:44:12.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feaeb5e2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae406c3e0 0x7feae406a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feaeb5e2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4073c30 0x7feae40bf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feaeb5e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 0x7feae40bf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feaeb5e2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae41ee190 con 0x7feae406c3e0 2023-03-03T08:44:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feaeb5e2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae41edc10 con 0x7feae4073c30 2023-03-03T08:44:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feaeb5e2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae41edeb0 con 0x7feae4151ab0 2023-03-03T08:44:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae937e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae406c3e0 0x7feae406a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae937e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae406c3e0 0x7feae406a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52484/0 (socket says 172.21.15.7:52484) 2023-03-03T08:44:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae937e700 1 -- 172.21.15.7:0/3287050088 learned_addr learned my addr 172.21.15.7:0/3287050088 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae8b7d700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4073c30 0x7feae40bf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae9b7f700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 0x7feae40bf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae937e700 1 -- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 msgr2=0x7feae40bf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae937e700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 0x7feae40bf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae937e700 1 -- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4073c30 msgr2=0x7feae40bf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae937e700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4073c30 0x7feae40bf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae937e700 1 -- 172.21.15.7:0/3287050088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead4009550 con 0x7feae406c3e0 2023-03-03T08:44:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae8b7d700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4073c30 0x7feae40bf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.301+0000 7feae9b7f700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 0x7feae40bf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.302+0000 7feae937e700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae406c3e0 0x7feae406a8e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7feae000c7f0 tx=0x7feae000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.302+0000 7fead9ffb700 1 -- 172.21.15.7:0/3287050088 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae0011800 con 0x7feae406c3e0 2023-03-03T08:44:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.302+0000 7fead9ffb700 1 -- 172.21.15.7:0/3287050088 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae0011960 con 0x7feae406c3e0 2023-03-03T08:44:12.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.302+0000 7feaeb5e2700 1 -- 172.21.15.7:0/3287050088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae4151e80 con 0x7feae406c3e0 2023-03-03T08:44:12.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.302+0000 7fead9ffb700 1 -- 172.21.15.7:0/3287050088 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae0011cf0 con 0x7feae406c3e0 2023-03-03T08:44:12.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.302+0000 7feaeb5e2700 1 -- 172.21.15.7:0/3287050088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae406d410 con 0x7feae406c3e0 2023-03-03T08:44:12.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.303+0000 7feaeb5e2700 1 -- 172.21.15.7:0/3287050088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae4152010 con 0x7feae406c3e0 2023-03-03T08:44:12.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.303+0000 7fead9ffb700 1 -- 172.21.15.7:0/3287050088 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feae0021070 con 0x7feae406c3e0 2023-03-03T08:44:12.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.304+0000 7fead9ffb700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac80682f0 0x7feac806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.304+0000 7fead9ffb700 1 -- 172.21.15.7:0/3287050088 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7feae0085cf0 con 0x7feae406c3e0 2023-03-03T08:44:12.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.304+0000 7feae8b7d700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac80682f0 0x7feac806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.304+0000 7feae8b7d700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac80682f0 0x7feac806a7a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feae40ad260 tx=0x7fead4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:12.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.308+0000 7fead9ffb700 1 -- 172.21.15.7:0/3287050088 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feae0052220 con 0x7feae406c3e0 2023-03-03T08:44:12.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.501+0000 7feaeb5e2700 1 -- 172.21.15.7:0/3287050088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feae40716f0 con 0x7feae406c3e0 2023-03-03T08:44:12.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.501+0000 7fead9ffb700 1 -- 172.21.15.7:0/3287050088 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7feae001f0a0 con 0x7feae406c3e0 2023-03-03T08:44:12.503 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-03-03T08:44:12.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 -- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac80682f0 msgr2=0x7feac806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac80682f0 0x7feac806a7a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feae40ad260 tx=0x7fead4008030 comp rx=0 tx=0).stop 2023-03-03T08:44:12.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 -- 172.21.15.7:0/3287050088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae406c3e0 msgr2=0x7feae406a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae406c3e0 0x7feae406a8e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7feae000c7f0 tx=0x7feae000cbb0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 -- 172.21.15.7:0/3287050088 shutdown_connections 2023-03-03T08:44:12.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae4151ab0 0x7feae40bf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac80682f0 0x7feac806a7a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae406c3e0 0x7feae406a8e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 --2- 172.21.15.7:0/3287050088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae4073c30 0x7feae40bf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 -- 172.21.15.7:0/3287050088 >> 172.21.15.7:0/3287050088 conn(0x7feae40a94d0 msgr2=0x7feae40c4670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:12.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 -- 172.21.15.7:0/3287050088 shutdown_connections 2023-03-03T08:44:12.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.503+0000 7feaeb5e2700 1 -- 172.21.15.7:0/3287050088 wait complete. 2023-03-03T08:44:12.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-03T08:44:12.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.643+0000 7f5bea6ea700 1 Processor -- start 2023-03-03T08:44:12.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.643+0000 7f5bea6ea700 1 -- start start 2023-03-03T08:44:12.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5bea6ea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5be413a0f0 0x7f5be40c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5bea6ea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5be41586c0 0x7f5be406d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5bea6ea700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be406de90 con 0x7f5be40c1da0 2023-03-03T08:44:12.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5bea6ea700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be4154840 con 0x7f5be413a0f0 2023-03-03T08:44:12.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5bea6ea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4154560 con 0x7f5be41586c0 2023-03-03T08:44:12.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5be3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5be413a0f0 0x7f5be40c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5be3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5be413a0f0 0x7f5be40c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52492/0 (socket says 172.21.15.7:52492) 2023-03-03T08:44:12.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5be3fff700 1 -- 172.21.15.7:0/2197858441 learned_addr learned my addr 172.21.15.7:0/2197858441 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:12.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5be37fe700 1 --2- 172.21.15.7:0/2197858441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5be41586c0 0x7f5be406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5be3fff700 1 -- 172.21.15.7:0/2197858441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5be41586c0 msgr2=0x7f5be406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:12.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5be3fff700 1 --2- 172.21.15.7:0/2197858441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5be41586c0 0x7f5be406d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5be3fff700 1 -- 172.21.15.7:0/2197858441 >> v1:172.21.15.143:6789/0 conn(0x7f5be40c1da0 legacy=0x7f5be406d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5be3fff700 1 -- 172.21.15.7:0/2197858441 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be4157be0 con 0x7f5be413a0f0 2023-03-03T08:44:12.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.644+0000 7f5be3fff700 1 --2- 172.21.15.7:0/2197858441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 0x7f5be40c07d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80098f0 tx=0x7f5bd8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf1996d095d38c6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:12.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5be2ffd700 1 -- 172.21.15.7:0/2197858441 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd800b8a0 con 0x7f5be413a0f0 2023-03-03T08:44:12.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5be2ffd700 1 -- 172.21.15.7:0/2197858441 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd800ba00 con 0x7f5be413a0f0 2023-03-03T08:44:12.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5be2ffd700 1 -- 172.21.15.7:0/2197858441 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd800bd90 con 0x7f5be413a0f0 2023-03-03T08:44:12.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5bea6ea700 1 -- 172.21.15.7:0/2197858441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 msgr2=0x7f5be40c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5bea6ea700 1 --2- 172.21.15.7:0/2197858441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 0x7f5be40c07d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80098f0 tx=0x7f5bd8009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:12.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5bea6ea700 1 -- 172.21.15.7:0/2197858441 shutdown_connections 2023-03-03T08:44:12.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5bea6ea700 1 --2- 172.21.15.7:0/2197858441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 0x7f5be40c07d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5bea6ea700 1 --2- 172.21.15.7:0/2197858441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5be41586c0 0x7f5be406d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5bea6ea700 1 -- 172.21.15.7:0/2197858441 >> 172.21.15.7:0/2197858441 conn(0x7f5be406c110 msgr2=0x7f5be406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5bea6ea700 1 -- 172.21.15.7:0/2197858441 shutdown_connections 2023-03-03T08:44:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.645+0000 7f5bea6ea700 1 -- 172.21.15.7:0/2197858441 wait complete. 2023-03-03T08:44:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.646+0000 7f5bea6ea700 1 Processor -- start 2023-03-03T08:44:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.647+0000 7f5bea6ea700 1 -- start start 2023-03-03T08:44:12.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.647+0000 7f5bea6ea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be40c1da0 0x7f5be41ed1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.647+0000 7f5bea6ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 0x7f5be4150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.647+0000 7f5bea6ea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5be41586c0 0x7f5be41509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.647+0000 7f5bea6ea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4151520 con 0x7f5be413a0f0 2023-03-03T08:44:12.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.647+0000 7f5bea6ea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be41edab0 con 0x7f5be40c1da0 2023-03-03T08:44:12.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.647+0000 7f5bea6ea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be40aa4f0 con 0x7f5be41586c0 2023-03-03T08:44:12.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.647+0000 7f5be37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 0x7f5be4150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.647+0000 7f5be37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 0x7f5be4150470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52498/0 (socket says 172.21.15.7:52498) 2023-03-03T08:44:12.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.647+0000 7f5be37fe700 1 -- 172.21.15.7:0/1873473154 learned_addr learned my addr 172.21.15.7:0/1873473154 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:12.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be3fff700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be40c1da0 0x7f5be41ed1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be8c87700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5be41586c0 0x7f5be41509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be37fe700 1 -- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5be41586c0 msgr2=0x7f5be41509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be37fe700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5be41586c0 0x7f5be41509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be37fe700 1 -- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be40c1da0 msgr2=0x7f5be41ed1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be37fe700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be40c1da0 0x7f5be41ed1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be37fe700 1 -- 172.21.15.7:0/1873473154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd8009550 con 0x7f5be413a0f0 2023-03-03T08:44:12.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be37fe700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 0x7f5be4150470 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400e810 tx=0x7f5bd400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:12.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be0ff9700 1 -- 172.21.15.7:0/1873473154 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4013610 con 0x7f5be413a0f0 2023-03-03T08:44:12.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be0ff9700 1 -- 172.21.15.7:0/1873473154 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd4013770 con 0x7f5be413a0f0 2023-03-03T08:44:12.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5be0ff9700 1 -- 172.21.15.7:0/1873473154 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4013b00 con 0x7f5be413a0f0 2023-03-03T08:44:12.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5bea6ea700 1 -- 172.21.15.7:0/1873473154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be41581a0 con 0x7f5be413a0f0 2023-03-03T08:44:12.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.648+0000 7f5bea6ea700 1 -- 172.21.15.7:0/1873473154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be4157ec0 con 0x7f5be413a0f0 2023-03-03T08:44:12.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.649+0000 7f5bea6ea700 1 -- 172.21.15.7:0/1873473154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be406e1a0 con 0x7f5be413a0f0 2023-03-03T08:44:12.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.650+0000 7f5be0ff9700 1 -- 172.21.15.7:0/1873473154 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5bd4023070 con 0x7f5be413a0f0 2023-03-03T08:44:12.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.650+0000 7f5be0ff9700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd0068320 0x7f5bd006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.650+0000 7f5be0ff9700 1 -- 172.21.15.7:0/1873473154 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f5bd4085ca0 con 0x7f5be413a0f0 2023-03-03T08:44:12.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.650+0000 7f5be3fff700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd0068320 0x7f5bd006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.651+0000 7f5be3fff700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd0068320 0x7f5bd006a7d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009c40 tx=0x7f5bd8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:12.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.654+0000 7f5be0ff9700 1 -- 172.21.15.7:0/1873473154 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5bd408a070 con 0x7f5be413a0f0 2023-03-03T08:44:12.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.847+0000 7f5bea6ea700 1 -- 172.21.15.7:0/1873473154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5be4074bb0 con 0x7f5be413a0f0 2023-03-03T08:44:12.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.847+0000 7f5be0ff9700 1 -- 172.21.15.7:0/1873473154 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5bd4052150 con 0x7f5be413a0f0 2023-03-03T08:44:12.849 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 8 2023-03-03T08:44:12.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 -- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd0068320 msgr2=0x7f5bd006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd0068320 0x7f5bd006a7d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009c40 tx=0x7f5bd8008030 comp rx=0 tx=0).stop 2023-03-03T08:44:12.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 -- 172.21.15.7:0/1873473154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 msgr2=0x7f5be4150470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 0x7f5be4150470 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400e810 tx=0x7f5bd400ebd0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 -- 172.21.15.7:0/1873473154 shutdown_connections 2023-03-03T08:44:12.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5be41586c0 0x7f5be41509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5be40c1da0 0x7f5be41ed1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd0068320 0x7f5bd006a7d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 --2- 172.21.15.7:0/1873473154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5be413a0f0 0x7f5be4150470 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 -- 172.21.15.7:0/1873473154 >> 172.21.15.7:0/1873473154 conn(0x7f5be406c110 msgr2=0x7f5be4137950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:12.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 -- 172.21.15.7:0/1873473154 shutdown_connections 2023-03-03T08:44:12.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.849+0000 7f5bea6ea700 1 -- 172.21.15.7:0/1873473154 wait complete. 2023-03-03T08:44:12.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-03-03T08:44:12.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.989+0000 7f46ad1f6700 1 Processor -- start 2023-03-03T08:44:12.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.989+0000 7f46ad1f6700 1 -- start start 2023-03-03T08:44:12.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46ad1f6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a8151cd0 0x7f46a8154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46ad1f6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f46a80c2e30 0x7f46a8158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46ad1f6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a8157ff0 con 0x7f46a806c6c0 2023-03-03T08:44:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46ad1f6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a806d640 con 0x7f46a80c2e30 2023-03-03T08:44:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46ad1f6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a806cda0 con 0x7f46a8151cd0 2023-03-03T08:44:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46a659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f46a80c2e30 0x7f46a8158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46a659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f46a80c2e30 0x7f46a8158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52504/0 (socket says 172.21.15.7:52504) 2023-03-03T08:44:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46a659c700 1 -- 172.21.15.7:0/1309489784 learned_addr learned my addr 172.21.15.7:0/1309489784 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46a6d9d700 1 --2- 172.21.15.7:0/1309489784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a8151cd0 0x7f46a8154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46a659c700 1 -- 172.21.15.7:0/1309489784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a8151cd0 msgr2=0x7f46a8154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46a659c700 1 --2- 172.21.15.7:0/1309489784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a8151cd0 0x7f46a8154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46a659c700 1 -- 172.21.15.7:0/1309489784 >> v1:172.21.15.143:6789/0 conn(0x7f46a806c6c0 legacy=0x7f46a80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.990+0000 7f46a659c700 1 -- 172.21.15.7:0/1309489784 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a806d080 con 0x7f46a80c2e30 2023-03-03T08:44:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46a659c700 1 --2- 172.21.15.7:0/1309489784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 0x7f46a8158480 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f46980098f0 tx=0x7f4698009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da63fda84bb9450 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46a5d9b700 1 -- 172.21.15.7:0/1309489784 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469800b8a0 con 0x7f46a80c2e30 2023-03-03T08:44:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46a5d9b700 1 -- 172.21.15.7:0/1309489784 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469800ba00 con 0x7f46a80c2e30 2023-03-03T08:44:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46a5d9b700 1 -- 172.21.15.7:0/1309489784 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469800bd90 con 0x7f46a80c2e30 2023-03-03T08:44:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46ad1f6700 1 -- 172.21.15.7:0/1309489784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 msgr2=0x7f46a8158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46ad1f6700 1 --2- 172.21.15.7:0/1309489784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 0x7f46a8158480 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f46980098f0 tx=0x7f4698009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46ad1f6700 1 -- 172.21.15.7:0/1309489784 shutdown_connections 2023-03-03T08:44:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46ad1f6700 1 --2- 172.21.15.7:0/1309489784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a8151cd0 0x7f46a8154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46ad1f6700 1 --2- 172.21.15.7:0/1309489784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 0x7f46a8158480 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46ad1f6700 1 -- 172.21.15.7:0/1309489784 >> 172.21.15.7:0/1309489784 conn(0x7f46a80a94d0 msgr2=0x7f46a80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46ad1f6700 1 -- 172.21.15.7:0/1309489784 shutdown_connections 2023-03-03T08:44:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.991+0000 7f46ad1f6700 1 -- 172.21.15.7:0/1309489784 wait complete. 2023-03-03T08:44:12.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.993+0000 7f46ad1f6700 1 Processor -- start 2023-03-03T08:44:12.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.993+0000 7f46ad1f6700 1 -- start start 2023-03-03T08:44:12.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46ad1f6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a806c6c0 0x7f46a814b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46ad1f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 0x7f46a81e0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46ad1f6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46a8151cd0 0x7f46a81eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46ad1f6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a81ea6b0 con 0x7f46a80c2e30 2023-03-03T08:44:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46ad1f6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a81ea310 con 0x7f46a8151cd0 2023-03-03T08:44:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46ad1f6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a80c6bd0 con 0x7f46a806c6c0 2023-03-03T08:44:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 0x7f46a81e0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 0x7f46a81e0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52512/0 (socket says 172.21.15.7:52512) 2023-03-03T08:44:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a659c700 1 -- 172.21.15.7:0/91917966 learned_addr learned my addr 172.21.15.7:0/91917966 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:12.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a6d9d700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a806c6c0 0x7f46a814b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a759e700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46a8151cd0 0x7f46a81eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a659c700 1 -- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a806c6c0 msgr2=0x7f46a814b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a659c700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a806c6c0 0x7f46a814b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a659c700 1 -- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46a8151cd0 msgr2=0x7f46a81eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a659c700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46a8151cd0 0x7f46a81eaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a659c700 1 -- 172.21.15.7:0/91917966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4698009550 con 0x7f46a80c2e30 2023-03-03T08:44:13.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a6d9d700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a806c6c0 0x7f46a814b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:13.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a759e700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46a8151cd0 0x7f46a81eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:13.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46a659c700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 0x7f46a81e0f10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4698009c40 tx=0x7f4698004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:13.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f468f7fe700 1 -- 172.21.15.7:0/91917966 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46980036a0 con 0x7f46a80c2e30 2023-03-03T08:44:13.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f468f7fe700 1 -- 172.21.15.7:0/91917966 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469801e070 con 0x7f46a80c2e30 2023-03-03T08:44:13.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f468f7fe700 1 -- 172.21.15.7:0/91917966 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4698003a60 con 0x7f46a80c2e30 2023-03-03T08:44:13.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.994+0000 7f46ad1f6700 1 -- 172.21.15.7:0/91917966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a8157c80 con 0x7f46a80c2e30 2023-03-03T08:44:13.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.995+0000 7f46ad1f6700 1 -- 172.21.15.7:0/91917966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a806d360 con 0x7f46a80c2e30 2023-03-03T08:44:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.996+0000 7f46ad1f6700 1 -- 172.21.15.7:0/91917966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a8154a50 con 0x7f46a80c2e30 2023-03-03T08:44:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.996+0000 7f468f7fe700 1 -- 172.21.15.7:0/91917966 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f469800f050 con 0x7f46a80c2e30 2023-03-03T08:44:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.996+0000 7f468f7fe700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f46940682f0 0x7f469406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.996+0000 7f468f7fe700 1 -- 172.21.15.7:0/91917966 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f4698020070 con 0x7f46a80c2e30 2023-03-03T08:44:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.997+0000 7f46a6d9d700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f46940682f0 0x7f469406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:12.997+0000 7f46a6d9d700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f46940682f0 0x7f469406a7a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f469c009a00 tx=0x7f469c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.001+0000 7f468f7fe700 1 -- 172.21.15.7:0/91917966 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4698053980 con 0x7f46a80c2e30 2023-03-03T08:44:13.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.192+0000 7f46ad1f6700 1 -- 172.21.15.7:0/91917966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f46a81ea7f0 con 0x7f46a80c2e30 2023-03-03T08:44:13.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.192+0000 7f468f7fe700 1 -- 172.21.15.7:0/91917966 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f46a81ea7f0 con 0x7f46a80c2e30 2023-03-03T08:44:13.194 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:44:13.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 -- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f46940682f0 msgr2=0x7f469406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f46940682f0 0x7f469406a7a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f469c009a00 tx=0x7f469c008040 comp rx=0 tx=0).stop 2023-03-03T08:44:13.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 -- 172.21.15.7:0/91917966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 msgr2=0x7f46a81e0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 0x7f46a81e0f10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4698009c40 tx=0x7f4698004750 comp rx=0 tx=0).stop 2023-03-03T08:44:13.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 -- 172.21.15.7:0/91917966 shutdown_connections 2023-03-03T08:44:13.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46a8151cd0 0x7f46a81eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f46a806c6c0 0x7f46a814b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f46940682f0 0x7f469406a7a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 --2- 172.21.15.7:0/91917966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46a80c2e30 0x7f46a81e0f10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 -- 172.21.15.7:0/91917966 >> 172.21.15.7:0/91917966 conn(0x7f46a80a94d0 msgr2=0x7f46a806a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:13.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 -- 172.21.15.7:0/91917966 shutdown_connections 2023-03-03T08:44:13.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.194+0000 7f46ad1f6700 1 -- 172.21.15.7:0/91917966 wait complete. 2023-03-03T08:44:13.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-03-03T08:44:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.335+0000 7fa15f989700 1 Processor -- start 2023-03-03T08:44:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.335+0000 7fa15f989700 1 -- start start 2023-03-03T08:44:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.335+0000 7fa15f989700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa15814e560 0x7fa158071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.335+0000 7fa15f989700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1580c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15f989700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa158072920 con 0x7fa158154590 2023-03-03T08:44:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15f989700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15806bef0 con 0x7fa15814e560 2023-03-03T08:44:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15f989700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15806b910 con 0x7fa1580c43d0 2023-03-03T08:44:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15cf24700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1580c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15d725700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa15814e560 0x7fa158071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15cf24700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1580c4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:60596/0 (socket says 172.21.15.7:60596) 2023-03-03T08:44:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15cf24700 1 -- 172.21.15.7:0/3293759060 learned_addr learned my addr 172.21.15.7:0/3293759060 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15df26700 1 --1- 172.21.15.7:0/3293759060 >> v1:172.21.15.143:6789/0 conn(0x7fa158154590 0x7fa1581549f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50026/0 (socket says 172.21.15.7:50026) 2023-03-03T08:44:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15d725700 1 -- 172.21.15.7:0/3293759060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 msgr2=0x7fa1580c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15d725700 1 --2- 172.21.15.7:0/3293759060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1580c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15d725700 1 -- 172.21.15.7:0/3293759060 >> v1:172.21.15.143:6789/0 conn(0x7fa158154590 legacy=0x7fa1581549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15d725700 1 -- 172.21.15.7:0/3293759060 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1580aacc0 con 0x7fa15814e560 2023-03-03T08:44:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa14ffff700 1 -- 172.21.15.7:0/3293759060 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89349242 0 0) 0x7fa150001490 con 0x7fa158154590 2023-03-03T08:44:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15cf24700 1 --2- 172.21.15.7:0/3293759060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1580c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.336+0000 7fa15d725700 1 --2- 172.21.15.7:0/3293759060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa15814e560 0x7fa158071c70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa154009840 tx=0x7fa154009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9bb64ddb4cbdff1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa14ffff700 1 -- 172.21.15.7:0/3293759060 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154011830 con 0x7fa15814e560 2023-03-03T08:44:13.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa14ffff700 1 -- 172.21.15.7:0/3293759060 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa154011990 con 0x7fa15814e560 2023-03-03T08:44:13.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa14ffff700 1 -- 172.21.15.7:0/3293759060 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154011d20 con 0x7fa15814e560 2023-03-03T08:44:13.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa15f989700 1 -- 172.21.15.7:0/3293759060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa15814e560 msgr2=0x7fa158071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa15f989700 1 --2- 172.21.15.7:0/3293759060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa15814e560 0x7fa158071c70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa154009840 tx=0x7fa154009b50 comp rx=0 tx=0).stop 2023-03-03T08:44:13.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa15f989700 1 -- 172.21.15.7:0/3293759060 shutdown_connections 2023-03-03T08:44:13.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa15f989700 1 --2- 172.21.15.7:0/3293759060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa15814e560 0x7fa158071c70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa15f989700 1 --2- 172.21.15.7:0/3293759060 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1580c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa15f989700 1 -- 172.21.15.7:0/3293759060 >> 172.21.15.7:0/3293759060 conn(0x7fa1580736f0 msgr2=0x7fa158073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:13.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa15f989700 1 -- 172.21.15.7:0/3293759060 shutdown_connections 2023-03-03T08:44:13.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.337+0000 7fa15f989700 1 -- 172.21.15.7:0/3293759060 wait complete. 2023-03-03T08:44:13.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.338+0000 7fa15f989700 1 Processor -- start 2023-03-03T08:44:13.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.339+0000 7fa15f989700 1 -- start start 2023-03-03T08:44:13.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.339+0000 7fa15f989700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1581ed0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.339+0000 7fa15f989700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15814e560 0x7fa1581ed610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.339+0000 7fa15f989700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa158154590 0x7fa15814bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.339+0000 7fa15f989700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580adce0 con 0x7fa15814e560 2023-03-03T08:44:13.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.339+0000 7fa15f989700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580bfa80 con 0x7fa158154590 2023-03-03T08:44:13.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.339+0000 7fa15f989700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1580ae4f0 con 0x7fa1580c43d0 2023-03-03T08:44:13.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.339+0000 7fa15cf24700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15814e560 0x7fa1581ed610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.339+0000 7fa15cf24700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15814e560 0x7fa1581ed610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52530/0 (socket says 172.21.15.7:52530) 2023-03-03T08:44:13.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.339+0000 7fa15cf24700 1 -- 172.21.15.7:0/2642431063 learned_addr learned my addr 172.21.15.7:0/2642431063 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:13.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15d725700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1581ed0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15df26700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa158154590 0x7fa15814bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15cf24700 1 -- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 msgr2=0x7fa1581ed0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15cf24700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1581ed0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15cf24700 1 -- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa158154590 msgr2=0x7fa15814bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15cf24700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa158154590 0x7fa15814bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15cf24700 1 -- 172.21.15.7:0/2642431063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa148009550 con 0x7fa15814e560 2023-03-03T08:44:13.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15df26700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa158154590 0x7fa15814bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:13.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15d725700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1581ed0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:13.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15cf24700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15814e560 0x7fa1581ed610 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa14800ebe0 tx=0x7fa148009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:13.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa14dffb700 1 -- 172.21.15.7:0/2642431063 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14800c8b0 con 0x7fa15814e560 2023-03-03T08:44:13.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa14dffb700 1 -- 172.21.15.7:0/2642431063 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14800ca10 con 0x7fa15814e560 2023-03-03T08:44:13.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15f989700 1 -- 172.21.15.7:0/2642431063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1540094a0 con 0x7fa15814e560 2023-03-03T08:44:13.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa14dffb700 1 -- 172.21.15.7:0/2642431063 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14800cd80 con 0x7fa15814e560 2023-03-03T08:44:13.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.340+0000 7fa15f989700 1 -- 172.21.15.7:0/2642431063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa158155100 con 0x7fa15814e560 2023-03-03T08:44:13.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.341+0000 7fa15f989700 1 -- 172.21.15.7:0/2642431063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15806bb80 con 0x7fa15814e560 2023-03-03T08:44:13.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.342+0000 7fa14dffb700 1 -- 172.21.15.7:0/2642431063 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa148019030 con 0x7fa15814e560 2023-03-03T08:44:13.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.342+0000 7fa14dffb700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa13c0682f0 0x7fa13c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.342+0000 7fa14dffb700 1 -- 172.21.15.7:0/2642431063 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fa1480860b0 con 0x7fa15814e560 2023-03-03T08:44:13.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.343+0000 7fa15d725700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa13c0682f0 0x7fa13c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.343+0000 7fa15d725700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa13c0682f0 0x7fa13c06a7a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa1540090b0 tx=0x7fa154008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:13.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.346+0000 7fa14dffb700 1 -- 172.21.15.7:0/2642431063 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa148052560 con 0x7fa15814e560 2023-03-03T08:44:13.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.539+0000 7fa15f989700 1 -- 172.21.15.7:0/2642431063 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7fa1581ece90 con 0x7fa15814e560 2023-03-03T08:44:13.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.539+0000 7fa14dffb700 1 -- 172.21.15.7:0/2642431063 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7fa1480170a0 con 0x7fa15814e560 2023-03-03T08:44:13.541 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:44:13.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.541+0000 7fa15f989700 1 -- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa13c0682f0 msgr2=0x7fa13c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.541+0000 7fa15f989700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa13c0682f0 0x7fa13c06a7a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa1540090b0 tx=0x7fa154008030 comp rx=0 tx=0).stop 2023-03-03T08:44:13.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.541+0000 7fa15f989700 1 -- 172.21.15.7:0/2642431063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15814e560 msgr2=0x7fa1581ed610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.541+0000 7fa15f989700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15814e560 0x7fa1581ed610 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa14800ebe0 tx=0x7fa148009e70 comp rx=0 tx=0).stop 2023-03-03T08:44:13.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.541+0000 7fa15f989700 1 -- 172.21.15.7:0/2642431063 shutdown_connections 2023-03-03T08:44:13.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.541+0000 7fa15f989700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa158154590 0x7fa15814bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.541+0000 7fa15f989700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1580c43d0 0x7fa1581ed0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.541+0000 7fa15f989700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa13c0682f0 0x7fa13c06a7a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.541+0000 7fa15f989700 1 --2- 172.21.15.7:0/2642431063 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa15814e560 0x7fa1581ed610 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.541+0000 7fa15f989700 1 -- 172.21.15.7:0/2642431063 >> 172.21.15.7:0/2642431063 conn(0x7fa1580736f0 msgr2=0x7fa158073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:13.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.542+0000 7fa15f989700 1 -- 172.21.15.7:0/2642431063 shutdown_connections 2023-03-03T08:44:13.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.542+0000 7fa15f989700 1 -- 172.21.15.7:0/2642431063 wait complete. 2023-03-03T08:44:13.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-03-03T08:44:13.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.681+0000 7f01efdc8700 1 Processor -- start 2023-03-03T08:44:13.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.681+0000 7f01efdc8700 1 -- start start 2023-03-03T08:44:13.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01efdc8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f01e80c2930 0x7f01e8157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01efdc8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e806d430 con 0x7f01e8151d40 2023-03-03T08:44:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01efdc8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e81586e0 con 0x7f01e806c480 2023-03-03T08:44:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01efdc8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e806c2a0 con 0x7f01e80c2930 2023-03-03T08:44:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01edb64700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f01e8151d40 0x7f01e80736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46516/0 (socket says 172.21.15.7:46516) 2023-03-03T08:44:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01edb64700 1 -- 172.21.15.7:0/1049797815 learned_addr learned my addr 172.21.15.7:0/1049797815 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164047598 0 0) 0x7f01e806d430 con 0x7f01e8151d40 2023-03-03T08:44:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ed363700 1 --2- 172.21.15.7:0/1049797815 >> v2:172.21.15.143:3300/0 conn(0x7f01e80c2930 0x7f01e8157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01d40035b0 con 0x7f01e8151d40 2023-03-03T08:44:13.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2388587125 0 0) 0x7f01d40035b0 con 0x7f01e8151d40 2023-03-03T08:44:13.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 >> v2:172.21.15.143:3300/0 conn(0x7f01e80c2930 msgr2=0x7f01e8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:13.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 --2- 172.21.15.7:0/1049797815 >> v2:172.21.15.143:3300/0 conn(0x7f01e80c2930 0x7f01e8157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 >> v1:172.21.15.143:6789/0 conn(0x7f01e806c480 legacy=0x7f01e8157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e806cb00 con 0x7f01e8151d40 2023-03-03T08:44:13.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f01e4001670 con 0x7f01e8151d40 2023-03-03T08:44:13.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 >> v1:172.21.15.7:6789/0 conn(0x7f01e8151d40 legacy=0x7f01e80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01d4003a70 0x7f01d4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01d4004850 0x7f01d4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01d4005320 0x7f01d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e806c2a0 con 0x7f01d4004850 2023-03-03T08:44:13.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d400c250 con 0x7f01d4005320 2023-03-03T08:44:13.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.682+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d400c410 con 0x7f01d4003a70 2023-03-03T08:44:13.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ed363700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01d4004850 0x7f01d4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01edb64700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01d4003a70 0x7f01d4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ee365700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01d4005320 0x7f01d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ed363700 1 -- 172.21.15.7:0/1049797815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01d4003a70 msgr2=0x7f01d4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:13.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ed363700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01d4003a70 0x7f01d4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ed363700 1 -- 172.21.15.7:0/1049797815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01d4005320 msgr2=0x7f01d400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:13.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ed363700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01d4005320 0x7f01d400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ee365700 1 -- 172.21.15.7:0/1049797815 reap_dead start 2023-03-03T08:44:13.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ed363700 1 -- 172.21.15.7:0/1049797815 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d40035b0 con 0x7f01d4004850 2023-03-03T08:44:13.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ed363700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01d4004850 0x7f01d4004cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009b60 tx=0x7f01d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2177874c56c9a598 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:13.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d800e6c0 con 0x7f01d4004850 2023-03-03T08:44:13.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d800e820 con 0x7f01d4004850 2023-03-03T08:44:13.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.683+0000 7f01ecb62700 1 -- 172.21.15.7:0/1049797815 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d800eb90 con 0x7f01d4004850 2023-03-03T08:44:13.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.684+0000 7f01efdc8700 1 -- 172.21.15.7:0/1049797815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01d4004850 msgr2=0x7f01d4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.684+0000 7f01efdc8700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01d4004850 0x7f01d4004cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009b60 tx=0x7f01d4005880 comp rx=0 tx=0).stop 2023-03-03T08:44:13.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.684+0000 7f01efdc8700 1 -- 172.21.15.7:0/1049797815 shutdown_connections 2023-03-03T08:44:13.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.684+0000 7f01efdc8700 1 --2- 172.21.15.7:0/1049797815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01d4004850 0x7f01d4004cc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.684+0000 7f01efdc8700 1 -- 172.21.15.7:0/1049797815 >> 172.21.15.7:0/1049797815 conn(0x7f01e80a94d0 msgr2=0x7f01e80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:13.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.684+0000 7f01efdc8700 1 -- 172.21.15.7:0/1049797815 shutdown_connections 2023-03-03T08:44:13.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.684+0000 7f01efdc8700 1 -- 172.21.15.7:0/1049797815 wait complete. 2023-03-03T08:44:13.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.685+0000 7f01efdc8700 1 Processor -- start 2023-03-03T08:44:13.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.685+0000 7f01efdc8700 1 -- start start 2023-03-03T08:44:13.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.686+0000 7f01efdc8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01d4004850 0x7f01e80acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.686+0000 7f01efdc8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01e8155290 0x7f01e81523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.686+0000 7f01efdc8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01e8152920 0x7f01e81ed0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.686+0000 7f01efdc8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e80ab930 con 0x7f01e8155290 2023-03-03T08:44:13.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.686+0000 7f01efdc8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e8153b80 con 0x7f01d4004850 2023-03-03T08:44:13.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.686+0000 7f01efdc8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e8153e60 con 0x7f01e8152920 2023-03-03T08:44:13.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.686+0000 7f01ed363700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01e8155290 0x7f01e81523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.686+0000 7f01ed363700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01e8155290 0x7f01e81523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52550/0 (socket says 172.21.15.7:52550) 2023-03-03T08:44:13.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.686+0000 7f01ed363700 1 -- 172.21.15.7:0/1426449651 learned_addr learned my addr 172.21.15.7:0/1426449651 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:13.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01edb64700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01d4004850 0x7f01e80acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01ee365700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01e8152920 0x7f01e81ed0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01ed363700 1 -- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01e8152920 msgr2=0x7f01e81ed0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01ed363700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01e8152920 0x7f01e81ed0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01ed363700 1 -- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01d4004850 msgr2=0x7f01e80acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01ed363700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01d4004850 0x7f01e80acd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01ed363700 1 -- 172.21.15.7:0/1426449651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e806d0c0 con 0x7f01e8155290 2023-03-03T08:44:13.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01ee365700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01e8152920 0x7f01e81ed0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:13.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01ed363700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01e8155290 0x7f01e81523e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f01d400c220 tx=0x7f01d80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:13.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01de7fc700 1 -- 172.21.15.7:0/1426449651 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d800e6c0 con 0x7f01e8155290 2023-03-03T08:44:13.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01de7fc700 1 -- 172.21.15.7:0/1426449651 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d8003ad0 con 0x7f01e8155290 2023-03-03T08:44:13.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01efdc8700 1 -- 172.21.15.7:0/1426449651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e806cde0 con 0x7f01e8155290 2023-03-03T08:44:13.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01de7fc700 1 -- 172.21.15.7:0/1426449651 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8003e80 con 0x7f01e8155290 2023-03-03T08:44:13.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.687+0000 7f01efdc8700 1 -- 172.21.15.7:0/1426449651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e806bea0 con 0x7f01e8155290 2023-03-03T08:44:13.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.688+0000 7f01efdc8700 1 -- 172.21.15.7:0/1426449651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e8074bb0 con 0x7f01e8155290 2023-03-03T08:44:13.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.688+0000 7f01de7fc700 1 -- 172.21.15.7:0/1426449651 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f01d8025070 con 0x7f01e8155290 2023-03-03T08:44:13.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.689+0000 7f01de7fc700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01cc0682f0 0x7f01cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:13.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.689+0000 7f01de7fc700 1 -- 172.21.15.7:0/1426449651 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f01d80878d0 con 0x7f01e8155290 2023-03-03T08:44:13.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.689+0000 7f01edb64700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01cc0682f0 0x7f01cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:13.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.690+0000 7f01edb64700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01cc0682f0 0x7f01cc06a7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f01e4000c00 tx=0x7f01e4000fc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:13.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.693+0000 7f01de7fc700 1 -- 172.21.15.7:0/1426449651 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f01d8053e00 con 0x7f01e8155290 2023-03-03T08:44:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.886+0000 7f01efdc8700 1 -- 172.21.15.7:0/1426449651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f01e80a9fb0 con 0x7f01e8155290 2023-03-03T08:44:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.887+0000 7f01de7fc700 1 -- 172.21.15.7:0/1426449651 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f01d8017020 con 0x7f01e8155290 2023-03-03T08:44:13.889 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2023-03-03T08:44:13.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 -- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01cc0682f0 msgr2=0x7f01cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01cc0682f0 0x7f01cc06a7a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f01e4000c00 tx=0x7f01e4000fc0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 -- 172.21.15.7:0/1426449651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01e8155290 msgr2=0x7f01e81523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:13.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01e8155290 0x7f01e81523e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f01d400c220 tx=0x7f01d80047c0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 -- 172.21.15.7:0/1426449651 shutdown_connections 2023-03-03T08:44:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01e8152920 0x7f01e81ed0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01d4004850 0x7f01e80acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01cc0682f0 0x7f01cc06a7a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 --2- 172.21.15.7:0/1426449651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01e8155290 0x7f01e81523e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 -- 172.21.15.7:0/1426449651 >> 172.21.15.7:0/1426449651 conn(0x7f01e80a94d0 msgr2=0x7f01e81500b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 -- 172.21.15.7:0/1426449651 shutdown_connections 2023-03-03T08:44:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:13.889+0000 7f01efdc8700 1 -- 172.21.15.7:0/1426449651 wait complete. 2023-03-03T08:44:13.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-03-03T08:44:14.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.029+0000 7f3e66997700 1 Processor -- start 2023-03-03T08:44:14.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.029+0000 7f3e66997700 1 -- start start 2023-03-03T08:44:14.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e66997700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3e60071e80 0x7f3e60151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e66997700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e60072250 0x7f3e6014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e66997700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e601510d0 con 0x7f3e600c1ce0 2023-03-03T08:44:14.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e66997700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e600c3740 con 0x7f3e60071e80 2023-03-03T08:44:14.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e66997700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e60070670 con 0x7f3e60072250 2023-03-03T08:44:14.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e5ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3e60071e80 0x7f3e60151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e64f34700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3e600c1ce0 0x7f3e601546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46540/0 (socket says 172.21.15.7:46540) 2023-03-03T08:44:14.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e64f34700 1 -- 172.21.15.7:0/334668468 learned_addr learned my addr 172.21.15.7:0/334668468 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:14.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e5f7fe700 1 --2- 172.21.15.7:0/334668468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e60072250 0x7f3e6014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e5ffff700 1 -- 172.21.15.7:0/334668468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e60072250 msgr2=0x7f3e6014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:14.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e5ffff700 1 --2- 172.21.15.7:0/334668468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e60072250 0x7f3e6014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e5ffff700 1 -- 172.21.15.7:0/334668468 >> v1:172.21.15.7:6789/0 conn(0x7f3e600c1ce0 legacy=0x7f3e601546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e5ffff700 1 -- 172.21.15.7:0/334668468 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e600c39e0 con 0x7f3e60071e80 2023-03-03T08:44:14.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e5effd700 1 -- 172.21.15.7:0/334668468 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303547764 0 0) 0x7f3e601510d0 con 0x7f3e600c1ce0 2023-03-03T08:44:14.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e5ffff700 1 --2- 172.21.15.7:0/334668468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e60071e80 0x7f3e60151820 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009840 tx=0x7f3e48009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ae2f87faafc309 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:14.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.030+0000 7f3e5effd700 1 -- 172.21.15.7:0/334668468 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48011830 con 0x7f3e60071e80 2023-03-03T08:44:14.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.031+0000 7f3e5effd700 1 -- 172.21.15.7:0/334668468 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48011990 con 0x7f3e60071e80 2023-03-03T08:44:14.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.031+0000 7f3e5effd700 1 -- 172.21.15.7:0/334668468 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48011d20 con 0x7f3e60071e80 2023-03-03T08:44:14.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.031+0000 7f3e66997700 1 -- 172.21.15.7:0/334668468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e60071e80 msgr2=0x7f3e60151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.031+0000 7f3e66997700 1 --2- 172.21.15.7:0/334668468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e60071e80 0x7f3e60151820 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009840 tx=0x7f3e48009b50 comp rx=0 tx=0).stop 2023-03-03T08:44:14.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.031+0000 7f3e66997700 1 -- 172.21.15.7:0/334668468 shutdown_connections 2023-03-03T08:44:14.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.031+0000 7f3e66997700 1 --2- 172.21.15.7:0/334668468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e60071e80 0x7f3e60151820 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.031+0000 7f3e66997700 1 --2- 172.21.15.7:0/334668468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e60072250 0x7f3e6014e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.031+0000 7f3e66997700 1 -- 172.21.15.7:0/334668468 >> 172.21.15.7:0/334668468 conn(0x7f3e600a94d0 msgr2=0x7f3e600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:14.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.031+0000 7f3e66997700 1 -- 172.21.15.7:0/334668468 shutdown_connections 2023-03-03T08:44:14.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.031+0000 7f3e66997700 1 -- 172.21.15.7:0/334668468 wait complete. 2023-03-03T08:44:14.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.032+0000 7f3e66997700 1 Processor -- start 2023-03-03T08:44:14.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.032+0000 7f3e66997700 1 -- start start 2023-03-03T08:44:14.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e66997700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e60071e80 0x7f3e601442a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e66997700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e60072250 0x7f3e6014a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e66997700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e600c1ce0 0x7f3e600ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e66997700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e601ebfe0 con 0x7f3e600c1ce0 2023-03-03T08:44:14.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e66997700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e601e7af0 con 0x7f3e60072250 2023-03-03T08:44:14.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e66997700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e601e8ac0 con 0x7f3e60071e80 2023-03-03T08:44:14.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e64f34700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e600c1ce0 0x7f3e600ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e64f34700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e600c1ce0 0x7f3e600ab840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52566/0 (socket says 172.21.15.7:52566) 2023-03-03T08:44:14.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e64f34700 1 -- 172.21.15.7:0/1747260477 learned_addr learned my addr 172.21.15.7:0/1747260477 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:14.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e5f7fe700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e60072250 0x7f3e6014a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.033+0000 7f3e5ffff700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e60071e80 0x7f3e601442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e64f34700 1 -- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e60071e80 msgr2=0x7f3e601442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e64f34700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e60071e80 0x7f3e601442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e64f34700 1 -- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e60072250 msgr2=0x7f3e6014a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e64f34700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e60072250 0x7f3e6014a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e64f34700 1 -- 172.21.15.7:0/1747260477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e480094a0 con 0x7f3e600c1ce0 2023-03-03T08:44:14.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e64f34700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e600c1ce0 0x7f3e600ab840 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800cc10 tx=0x7f3e5800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:14.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e5cff9700 1 -- 172.21.15.7:0/1747260477 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5800aaa0 con 0x7f3e600c1ce0 2023-03-03T08:44:14.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e5cff9700 1 -- 172.21.15.7:0/1747260477 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5800ac20 con 0x7f3e600c1ce0 2023-03-03T08:44:14.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e66997700 1 -- 172.21.15.7:0/1747260477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6006c9a0 con 0x7f3e600c1ce0 2023-03-03T08:44:14.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e5cff9700 1 -- 172.21.15.7:0/1747260477 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e580070d0 con 0x7f3e600c1ce0 2023-03-03T08:44:14.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.034+0000 7f3e66997700 1 -- 172.21.15.7:0/1747260477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e60151300 con 0x7f3e600c1ce0 2023-03-03T08:44:14.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.035+0000 7f3e66997700 1 -- 172.21.15.7:0/1747260477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6006cb30 con 0x7f3e600c1ce0 2023-03-03T08:44:14.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.035+0000 7f3e5cff9700 1 -- 172.21.15.7:0/1747260477 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3e5801a030 con 0x7f3e600c1ce0 2023-03-03T08:44:14.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.036+0000 7f3e5cff9700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3e400682f0 0x7f3e4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.036+0000 7f3e5cff9700 1 -- 172.21.15.7:0/1747260477 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f3e58085b70 con 0x7f3e600c1ce0 2023-03-03T08:44:14.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.036+0000 7f3e5ffff700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3e400682f0 0x7f3e4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.037+0000 7f3e5ffff700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3e400682f0 0x7f3e4006a7a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009fd0 tx=0x7f3e48008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:14.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.040+0000 7f3e5cff9700 1 -- 172.21.15.7:0/1747260477 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3e58052040 con 0x7f3e600c1ce0 2023-03-03T08:44:14.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.233+0000 7f3e66997700 1 -- 172.21.15.7:0/1747260477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f3e60074bb0 con 0x7f3e600c1ce0 2023-03-03T08:44:14.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.233+0000 7f3e5cff9700 1 -- 172.21.15.7:0/1747260477 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7f3e580180a0 con 0x7f3e600c1ce0 2023-03-03T08:44:14.235 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2023-03-03T08:44:14.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 -- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3e400682f0 msgr2=0x7f3e4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3e400682f0 0x7f3e4006a7a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009fd0 tx=0x7f3e48008030 comp rx=0 tx=0).stop 2023-03-03T08:44:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 -- 172.21.15.7:0/1747260477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e600c1ce0 msgr2=0x7f3e600ab840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e600c1ce0 0x7f3e600ab840 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800cc10 tx=0x7f3e5800cf20 comp rx=0 tx=0).stop 2023-03-03T08:44:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 -- 172.21.15.7:0/1747260477 shutdown_connections 2023-03-03T08:44:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3e400682f0 0x7f3e4006a7a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e600c1ce0 0x7f3e600ab840 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e60071e80 0x7f3e601442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 --2- 172.21.15.7:0/1747260477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e60072250 0x7f3e6014a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 -- 172.21.15.7:0/1747260477 >> 172.21.15.7:0/1747260477 conn(0x7f3e600a94d0 msgr2=0x7f3e6006d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 -- 172.21.15.7:0/1747260477 shutdown_connections 2023-03-03T08:44:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.235+0000 7f3e66997700 1 -- 172.21.15.7:0/1747260477 wait complete. 2023-03-03T08:44:14.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:14.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.372+0000 7fdf200d3700 1 Processor -- start 2023-03-03T08:44:14.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.372+0000 7fdf200d3700 1 -- start start 2023-03-03T08:44:14.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.373+0000 7fdf200d3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf18155390 0x7fdf180c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.373+0000 7fdf200d3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdf18151770 0x7fdf18151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.373+0000 7fdf200d3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf18154910 con 0x7fdf1806e020 2023-03-03T08:44:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.373+0000 7fdf200d3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf18070b90 con 0x7fdf18155390 2023-03-03T08:44:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.373+0000 7fdf200d3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf18151320 con 0x7fdf18151770 2023-03-03T08:44:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.373+0000 7fdf1de6f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf18155390 0x7fdf180c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.373+0000 7fdf1de6f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf18155390 0x7fdf180c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52570/0 (socket says 172.21.15.7:52570) 2023-03-03T08:44:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.373+0000 7fdf1de6f700 1 -- 172.21.15.7:0/1055747801 learned_addr learned my addr 172.21.15.7:0/1055747801 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.373+0000 7fdf1e670700 1 --2- 172.21.15.7:0/1055747801 >> v2:172.21.15.143:3300/0 conn(0x7fdf18151770 0x7fdf18151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf1de6f700 1 -- 172.21.15.7:0/1055747801 >> v2:172.21.15.143:3300/0 conn(0x7fdf18151770 msgr2=0x7fdf18151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf1de6f700 1 --2- 172.21.15.7:0/1055747801 >> v2:172.21.15.143:3300/0 conn(0x7fdf18151770 0x7fdf18151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf1de6f700 1 -- 172.21.15.7:0/1055747801 >> v1:172.21.15.143:6789/0 conn(0x7fdf1806e020 legacy=0x7fdf18071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf1de6f700 1 -- 172.21.15.7:0/1055747801 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf180702a0 con 0x7fdf18155390 2023-03-03T08:44:14.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf1de6f700 1 --2- 172.21.15.7:0/1055747801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18155390 0x7fdf180c3e30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdf1806ffe0 tx=0x7fdf0800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd2bca5b5743cb3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:14.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf1ce6d700 1 -- 172.21.15.7:0/1055747801 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0800dc70 con 0x7fdf18155390 2023-03-03T08:44:14.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf1ce6d700 1 -- 172.21.15.7:0/1055747801 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0800ddd0 con 0x7fdf18155390 2023-03-03T08:44:14.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf1ce6d700 1 -- 172.21.15.7:0/1055747801 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08010690 con 0x7fdf18155390 2023-03-03T08:44:14.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf200d3700 1 -- 172.21.15.7:0/1055747801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18155390 msgr2=0x7fdf180c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf200d3700 1 --2- 172.21.15.7:0/1055747801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18155390 0x7fdf180c3e30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdf1806ffe0 tx=0x7fdf0800be70 comp rx=0 tx=0).stop 2023-03-03T08:44:14.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf200d3700 1 -- 172.21.15.7:0/1055747801 shutdown_connections 2023-03-03T08:44:14.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf200d3700 1 --2- 172.21.15.7:0/1055747801 >> v2:172.21.15.143:3300/0 conn(0x7fdf18151770 0x7fdf18151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf200d3700 1 --2- 172.21.15.7:0/1055747801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18155390 0x7fdf180c3e30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.374+0000 7fdf200d3700 1 -- 172.21.15.7:0/1055747801 >> 172.21.15.7:0/1055747801 conn(0x7fdf1806c110 msgr2=0x7fdf1806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:14.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.375+0000 7fdf200d3700 1 -- 172.21.15.7:0/1055747801 shutdown_connections 2023-03-03T08:44:14.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.375+0000 7fdf200d3700 1 -- 172.21.15.7:0/1055747801 wait complete. 2023-03-03T08:44:14.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.376+0000 7fdf200d3700 1 Processor -- start 2023-03-03T08:44:14.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.376+0000 7fdf200d3700 1 -- start start 2023-03-03T08:44:14.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf200d3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf1806e020 0x7fdf180c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf200d3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18151770 0x7fdf180aad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf200d3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf18155390 0x7fdf1814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf200d3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf181e9240 con 0x7fdf18151770 2023-03-03T08:44:14.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf200d3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf181e8930 con 0x7fdf1806e020 2023-03-03T08:44:14.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf200d3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf180ab630 con 0x7fdf18155390 2023-03-03T08:44:14.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf1d66e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18151770 0x7fdf180aad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf1d66e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18151770 0x7fdf180aad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52578/0 (socket says 172.21.15.7:52578) 2023-03-03T08:44:14.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf1d66e700 1 -- 172.21.15.7:0/1140002269 learned_addr learned my addr 172.21.15.7:0/1140002269 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:14.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf1de6f700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf1806e020 0x7fdf180c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf1e670700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf18155390 0x7fdf1814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf1d66e700 1 -- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf18155390 msgr2=0x7fdf1814b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf1d66e700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf18155390 0x7fdf1814b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf1d66e700 1 -- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf1806e020 msgr2=0x7fdf180c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf1d66e700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf1806e020 0x7fdf180c3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.377+0000 7fdf1d66e700 1 -- 172.21.15.7:0/1140002269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0800b7f0 con 0x7fdf18151770 2023-03-03T08:44:14.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.378+0000 7fdf1e670700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf18155390 0x7fdf1814b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:14.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.378+0000 7fdf1d66e700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18151770 0x7fdf180aad30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400c4e0 tx=0x7fdf1400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:14.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.378+0000 7fdf0e7fc700 1 -- 172.21.15.7:0/1140002269 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf140052d0 con 0x7fdf18151770 2023-03-03T08:44:14.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.378+0000 7fdf0e7fc700 1 -- 172.21.15.7:0/1140002269 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf14005430 con 0x7fdf18151770 2023-03-03T08:44:14.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.378+0000 7fdf0e7fc700 1 -- 172.21.15.7:0/1140002269 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf140038c0 con 0x7fdf18151770 2023-03-03T08:44:14.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.378+0000 7fdf200d3700 1 -- 172.21.15.7:0/1140002269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf18070820 con 0x7fdf18151770 2023-03-03T08:44:14.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.378+0000 7fdf200d3700 1 -- 172.21.15.7:0/1140002269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf18070540 con 0x7fdf18151770 2023-03-03T08:44:14.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.379+0000 7fdf200d3700 1 -- 172.21.15.7:0/1140002269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf18155760 con 0x7fdf18151770 2023-03-03T08:44:14.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.379+0000 7fdf0e7fc700 1 -- 172.21.15.7:0/1140002269 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdf14019030 con 0x7fdf18151770 2023-03-03T08:44:14.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.380+0000 7fdf0e7fc700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdefc068320 0x7fdefc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.380+0000 7fdf0e7fc700 1 -- 172.21.15.7:0/1140002269 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdf14085270 con 0x7fdf18151770 2023-03-03T08:44:14.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.380+0000 7fdf1de6f700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdefc068320 0x7fdefc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.380+0000 7fdf1de6f700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdefc068320 0x7fdefc06a7d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800beb0 tx=0x7fdf08013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:14.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.384+0000 7fdf0e7fc700 1 -- 172.21.15.7:0/1140002269 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdf140517a0 con 0x7fdf18151770 2023-03-03T08:44:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.625+0000 7fdf200d3700 1 -- 172.21.15.7:0/1140002269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf180712f0 con 0x7fdf18151770 2023-03-03T08:44:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.625+0000 7fdf0e7fc700 1 -- 172.21.15.7:0/1140002269 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7fdf14054dc0 con 0x7fdf18151770 2023-03-03T08:44:14.627 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:14.627 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:14.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 -- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdefc068320 msgr2=0x7fdefc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdefc068320 0x7fdefc06a7d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800beb0 tx=0x7fdf08013000 comp rx=0 tx=0).stop 2023-03-03T08:44:14.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 -- 172.21.15.7:0/1140002269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18151770 msgr2=0x7fdf180aad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18151770 0x7fdf180aad30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400c4e0 tx=0x7fdf1400c7f0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 -- 172.21.15.7:0/1140002269 shutdown_connections 2023-03-03T08:44:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf18155390 0x7fdf1814b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf1806e020 0x7fdf180c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdefc068320 0x7fdefc06a7d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 --2- 172.21.15.7:0/1140002269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf18151770 0x7fdf180aad30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 -- 172.21.15.7:0/1140002269 >> 172.21.15.7:0/1140002269 conn(0x7fdf1806c110 msgr2=0x7fdf1806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 -- 172.21.15.7:0/1140002269 shutdown_connections 2023-03-03T08:44:14.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.628+0000 7fdf200d3700 1 -- 172.21.15.7:0/1140002269 wait complete. 2023-03-03T08:44:14.631 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:14.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-03T08:44:14.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.764+0000 7f38bd6f6700 1 Processor -- start 2023-03-03T08:44:14.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.764+0000 7f38bd6f6700 1 -- start start 2023-03-03T08:44:14.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38bd6f6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f38b814fda0 0x7f38b8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38bd6f6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b80abe20 0x7f38b81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38bd6f6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b80c2f30 con 0x7f38b814f9d0 2023-03-03T08:44:14.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38bd6f6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b80c31d0 con 0x7f38b814fda0 2023-03-03T08:44:14.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38bd6f6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b81524a0 con 0x7f38b80abe20 2023-03-03T08:44:14.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38b77fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b80abe20 0x7f38b81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38b67fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f38b814fda0 0x7f38b8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38b77fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b80abe20 0x7f38b81529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:60662/0 (socket says 172.21.15.7:60662) 2023-03-03T08:44:14.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38b6ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f38b814f9d0 0x7f38b8071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50086/0 (socket says 172.21.15.7:50086) 2023-03-03T08:44:14.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38b67fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f38b814fda0 0x7f38b8156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:48362/0 (socket says 172.21.15.7:48362) 2023-03-03T08:44:14.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38b77fe700 1 -- 172.21.15.7:0/3421277935 learned_addr learned my addr 172.21.15.7:0/3421277935 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:14.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38b67fc700 1 -- 172.21.15.7:0/3421277935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b80abe20 msgr2=0x7f38b81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:14.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38b67fc700 1 --2- 172.21.15.7:0/3421277935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b80abe20 0x7f38b81529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38b67fc700 1 -- 172.21.15.7:0/3421277935 >> v1:172.21.15.143:6789/0 conn(0x7f38b814f9d0 legacy=0x7f38b8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.765+0000 7f38b67fc700 1 -- 172.21.15.7:0/3421277935 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b80c34b0 con 0x7f38b814fda0 2023-03-03T08:44:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38b67fc700 1 --2- 172.21.15.7:0/3421277935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b814fda0 0x7f38b8156430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009840 tx=0x7f38ac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1ec77926f8e5223 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38b5ffb700 1 -- 172.21.15.7:0/3421277935 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac011830 con 0x7f38b814fda0 2023-03-03T08:44:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38b5ffb700 1 -- 172.21.15.7:0/3421277935 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac011990 con 0x7f38b814fda0 2023-03-03T08:44:14.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38b5ffb700 1 -- 172.21.15.7:0/3421277935 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac011d20 con 0x7f38b814fda0 2023-03-03T08:44:14.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38bd6f6700 1 -- 172.21.15.7:0/3421277935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b814fda0 msgr2=0x7f38b8156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38bd6f6700 1 --2- 172.21.15.7:0/3421277935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b814fda0 0x7f38b8156430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009840 tx=0x7f38ac009b50 comp rx=0 tx=0).stop 2023-03-03T08:44:14.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38bd6f6700 1 -- 172.21.15.7:0/3421277935 shutdown_connections 2023-03-03T08:44:14.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38bd6f6700 1 --2- 172.21.15.7:0/3421277935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b80abe20 0x7f38b81529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38bd6f6700 1 --2- 172.21.15.7:0/3421277935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b814fda0 0x7f38b8156430 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38bd6f6700 1 -- 172.21.15.7:0/3421277935 >> 172.21.15.7:0/3421277935 conn(0x7f38b80c8900 msgr2=0x7f38b80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:14.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38bd6f6700 1 -- 172.21.15.7:0/3421277935 shutdown_connections 2023-03-03T08:44:14.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.766+0000 7f38bd6f6700 1 -- 172.21.15.7:0/3421277935 wait complete. 2023-03-03T08:44:14.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38bd6f6700 1 Processor -- start 2023-03-03T08:44:14.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38bd6f6700 1 -- start start 2023-03-03T08:44:14.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38bd6f6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b80abe20 0x7f38b80c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38bd6f6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b814f9d0 0x7f38b8149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38bd6f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b814fda0 0x7f38b80c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38bd6f6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b81ed7e0 con 0x7f38b814fda0 2023-03-03T08:44:14.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38bd6f6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b81edd10 con 0x7f38b80abe20 2023-03-03T08:44:14.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38bd6f6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b81e9d10 con 0x7f38b814f9d0 2023-03-03T08:44:14.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38b77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b814fda0 0x7f38b80c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38b77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b814fda0 0x7f38b80c6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52596/0 (socket says 172.21.15.7:52596) 2023-03-03T08:44:14.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.768+0000 7f38b77fe700 1 -- 172.21.15.7:0/2325240334 learned_addr learned my addr 172.21.15.7:0/2325240334 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:14.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f38b67fc700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b814f9d0 0x7f38b8149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f38b6ffd700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b80abe20 0x7f38b80c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f38b77fe700 1 -- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b814f9d0 msgr2=0x7f38b8149c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f38b77fe700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b814f9d0 0x7f38b8149c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f38b77fe700 1 -- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b80abe20 msgr2=0x7f38b80c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f38b77fe700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b80abe20 0x7f38b80c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f38b77fe700 1 -- 172.21.15.7:0/2325240334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac0094a0 con 0x7f38b814fda0 2023-03-03T08:44:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f38b77fe700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b814fda0 0x7f38b80c6c40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f38b000efc0 tx=0x7f38b0009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f389f7fe700 1 -- 172.21.15.7:0/2325240334 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b0009b20 con 0x7f38b814fda0 2023-03-03T08:44:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f389f7fe700 1 -- 172.21.15.7:0/2325240334 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b000f040 con 0x7f38b814fda0 2023-03-03T08:44:14.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f38bd6f6700 1 -- 172.21.15.7:0/2325240334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b80ac350 con 0x7f38b814fda0 2023-03-03T08:44:14.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f389f7fe700 1 -- 172.21.15.7:0/2325240334 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b0003e60 con 0x7f38b814fda0 2023-03-03T08:44:14.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.769+0000 7f38bd6f6700 1 -- 172.21.15.7:0/2325240334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b80ab630 con 0x7f38b814fda0 2023-03-03T08:44:14.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.770+0000 7f38bd6f6700 1 -- 172.21.15.7:0/2325240334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b8071fa0 con 0x7f38b814fda0 2023-03-03T08:44:14.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.770+0000 7f389f7fe700 1 -- 172.21.15.7:0/2325240334 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f38b000c620 con 0x7f38b814fda0 2023-03-03T08:44:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.771+0000 7f389f7fe700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38a40682f0 0x7f38a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.771+0000 7f389f7fe700 1 -- 172.21.15.7:0/2325240334 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f38b00862c0 con 0x7f38b814fda0 2023-03-03T08:44:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.771+0000 7f38b6ffd700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38a40682f0 0x7f38a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.771+0000 7f38b6ffd700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38a40682f0 0x7f38a406a7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f38a8000f60 tx=0x7f38a8007ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.775+0000 7f389f7fe700 1 -- 172.21.15.7:0/2325240334 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f38b00527f0 con 0x7f38b814fda0 2023-03-03T08:44:14.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.971+0000 7f38bd6f6700 1 -- 172.21.15.7:0/2325240334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38b80c9100 con 0x7f38b814fda0 2023-03-03T08:44:14.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.972+0000 7f389f7fe700 1 -- 172.21.15.7:0/2325240334 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21533 (secure 0 0 0) 0x7f38b0055e10 con 0x7f38b814fda0 2023-03-03T08:44:14.974 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:14.974 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"fsid":"4366045e-92da-4bb9-8565-4b6c37871975","created":"2023-03-03T08:43:09.505439+0000","modified":"2023-03-03T08:43:44.973251+0000","last_up_change":"2023-03-03T08:43:20.540985+0000","last_in_change":"2023-03-03T08:43:12.002629+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-03T08:43:21.160680+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-03T08:43:28.018457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-03-03T08:43:32.305037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-03-03T08:43:33.484501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-03-03T08:43:42.509451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-03-03T08:43:42.936898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0b11573f-bcee-4126-b626-d34ffe8b1720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":96730},{"type":"v1","addr":"172.21.15.7:6801","nonce":96730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":96730},{"type":"v1","addr":"172.21.15.7:6803","nonce":96730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":96730},{"type":"v1","addr":"172.21.15.7:6807","nonce":96730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":96730},{"type":"v1","addr":"172.21.15.7:6805","nonce":96730}]},"public_addr":"172.21.15.7:6801/96730","cluster_addr":"172.21.15.7:6803/96730","heartbeat_back_addr":"172.21.15.7:6807/96730","heartbeat_front_addr":"172.21.15.7:6805/96730","state":["exists","up"]},{"osd":1,"uuid":"7384f568-a97f-412b-a9c2-4c4a88b12b80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":96733},{"type":"v1","addr":"172.21.15.7:6817","nonce":96733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":96733},{"type":"v1","addr":"172.21.15.7:6819","nonce":96733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":96733},{"type":"v1","addr":"172.21.15.7:6823","nonce":96733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":96733},{"type":"v1","addr":"172.21.15.7:6821","nonce":96733}]},"public_addr":"172.21.15.7:6817/96733","cluster_addr":"172.21.15.7:6819/96733","heartbeat_back_addr":"172.21.15.7:6823/96733","heartbeat_front_addr":"172.21.15.7:6821/96733","state":["exists","up"]},{"osd":2,"uuid":"7f01219e-308c-417d-950f-d87aaa252e35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":96732},{"type":"v1","addr":"172.21.15.7:6825","nonce":96732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":96732},{"type":"v1","addr":"172.21.15.7:6827","nonce":96732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":96732},{"type":"v1","addr":"172.21.15.7:6831","nonce":96732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":96732},{"type":"v1","addr":"172.21.15.7:6829","nonce":96732}]},"public_addr":"172.21.15.7:6825/96732","cluster_addr":"172.21.15.7:6827/96732","heartbeat_back_addr":"172.21.15.7:6831/96732","heartbeat_front_addr":"172.21.15.7:6829/96732","state":["exists","up"]},{"osd":3,"uuid":"6776045a-e68c-435b-98c8-8462608e1bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":96731},{"type":"v1","addr":"172.21.15.7:6809","nonce":96731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":96731},{"type":"v1","addr":"172.21.15.7:6811","nonce":96731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":96731},{"type":"v1","addr":"172.21.15.7:6815","nonce":96731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":96731},{"type":"v1","addr":"172.21.15.7:6813","nonce":96731}]},"public_addr":"172.21.15.7:6809/96731","cluster_addr":"172.21.15.7:6811/96731","heartbeat_back_addr":"172.21.15.7:6815/96731","heartbeat_front_addr":"172.21.15.7:6813/96731","state":["exists","up"]},{"osd":4,"uuid":"f53e6654-cc96-4ed5-a05c-25a543abefba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":97029},{"type":"v1","addr":"172.21.15.143:6818","nonce":97029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":97029},{"type":"v1","addr":"172.21.15.143:6822","nonce":97029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":97029},{"type":"v1","addr":"172.21.15.143:6830","nonce":97029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":97029},{"type":"v1","addr":"172.21.15.143:6826","nonce":97029}]},"public_addr":"172.21.15.143:6818/97029","cluster_addr":"172.21.15.143:6822/97029","heartbeat_back_addr":"172.21.15.143:6830/97029","heartbeat_front_addr":"172.21.15.143:6826/97029","state":["exists","up"]},{"osd":5,"uuid":"138d395a-16cf-46f5-beed-2bfde299f1b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6817","nonce":97030},{"type":"v1","addr":"172.21.15.143:6819","nonce":97030}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6821","nonce":97030},{"type":"v1","addr":"172.21.15.143:6823","nonce":97030}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6829","nonce":97030},{"type":"v1","addr":"172.21.15.143:6831","nonce":97030}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6825","nonce":97030},{"type":"v1","addr":"172.21.15.143:6827","nonce":97030}]},"public_addr":"172.21.15.143:6819/97030","cluster_addr":"172.21.15.143:6823/97030","heartbeat_back_addr":"172.21.15.143:6831/97030","heartbeat_front_addr":"172.21.15.143:6827/97030","state":["exists","up"]},{"osd":6,"uuid":"4c657cd7-4110-46d8-b01a-a5ddf7dfcf15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":97018},{"type":"v1","addr":"172.21.15.143:6801","nonce":97018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":97018},{"type":"v1","addr":"172.21.15.143:6803","nonce":97018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":97018},{"type":"v1","addr":"172.21.15.143:6807","nonce":97018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":97018},{"type":"v1","addr":"172.21.15.143:6805","nonce":97018}]},"public_addr":"172.21.15.143:6801/97018","cluster_addr":"172.21.15.143:6803/97018","heartbeat_back_addr":"172.21.15.143:6807/97018","heartbeat_front_addr":"172.21.15.143:6805/97018","state":["exists","up"]},{"osd":7,"uuid":"d309d51a-84e6-4634-9a68-c6dd8a62f81b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":97019},{"type":"v1","addr":"172.21.15.143:6809","nonce":97019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":97019},{"type":"v1","addr":"172.21.15.143:6811","nonce":97019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":97019},{"type":"v1","addr":"172.21.15.143:6815","nonce":97019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":97019},{"type":"v1","addr":"172.21.15.143:6813","nonce":97019}]},"public_addr":"172.21.15.143:6809/97019","cluster_addr":"172.21.15.143:6811/97019","heartbeat_back_addr":"172.21.15.143:6815/97019","heartbeat_front_addr":"172.21.15.143:6813/97019","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.192790+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.175202+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.197683+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.231809+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.249285+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.173734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.238917+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-03T08:43:17.218478+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-03T08:44:14.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 -- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38a40682f0 msgr2=0x7f38a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38a40682f0 0x7f38a406a7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f38a8000f60 tx=0x7f38a8007ed0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 -- 172.21.15.7:0/2325240334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b814fda0 msgr2=0x7f38b80c6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b814fda0 0x7f38b80c6c40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f38b000efc0 tx=0x7f38b0009f60 comp rx=0 tx=0).stop 2023-03-03T08:44:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 -- 172.21.15.7:0/2325240334 shutdown_connections 2023-03-03T08:44:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38a40682f0 0x7f38a406a7a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b814fda0 0x7f38b80c6c40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b80abe20 0x7f38b80c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 --2- 172.21.15.7:0/2325240334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38b814f9d0 0x7f38b8149c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 -- 172.21.15.7:0/2325240334 >> 172.21.15.7:0/2325240334 conn(0x7f38b80c8900 msgr2=0x7f38b8151d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 -- 172.21.15.7:0/2325240334 shutdown_connections 2023-03-03T08:44:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:14.974+0000 7f38bd6f6700 1 -- 172.21.15.7:0/2325240334 wait complete. 2023-03-03T08:44:14.989 DEBUG:tasks.mds_thrash:Yielding 2023-03-03T08:44:14.990 INFO:teuthology.run_tasks:Running task workunit... 2023-03-03T08:44:15.155 INFO:tasks.workunit:Pulling workunits from ref 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T08:44:15.182 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-03-03T08:44:15.183 INFO:tasks.workunit:timeout=6h 2023-03-03T08:44:15.183 INFO:tasks.workunit:cleanup=True 2023-03-03T08:44:15.183 DEBUG:teuthology.orchestra.run.smithi007:> stat -- /home/ubuntu/cephtest/mnt.0 2023-03-03T08:44:15.184 INFO:tasks.mds_thrash.fs.[b]:starting mds_do_thrash for fs b 2023-03-03T08:44:15.184 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before thrashing 2023-03-03T08:44:15.203 INFO:teuthology.orchestra.run.smithi007.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-03-03T08:44:15.204 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-03-03T08:44:15.204 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 2fh/47d Inode: 1 Links: 2 2023-03-03T08:44:15.204 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T08:44:15.204 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:cephfs_t:s0 2023-03-03T08:44:15.204 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-03-03T08:44:15.204 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-03-03 08:43:34.519264324 +0000 2023-03-03T08:44:15.205 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-03-03 08:43:58.551771977 +0000 2023-03-03T08:44:15.205 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-03-03 08:43:34.519264324 +0000 2023-03-03T08:44:15.205 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-03-03T08:44:15.205 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-03-03T08:44:15.276 DEBUG:teuthology.orchestra.run.smithi143:> stat -- /home/ubuntu/cephtest/mnt.1 2023-03-03T08:44:15.298 INFO:teuthology.orchestra.run.smithi143.stdout: File: /home/ubuntu/cephtest/mnt.1 2023-03-03T08:44:15.299 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-03-03T08:44:15.299 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 2fh/47d Inode: 1 Links: 2 2023-03-03T08:44:15.299 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-03T08:44:15.299 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:cephfs_t:s0 2023-03-03T08:44:15.299 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-03-03T08:44:15.299 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2023-03-03 08:43:43.974812013 +0000 2023-03-03T08:44:15.300 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2023-03-03 08:44:01.059202826 +0000 2023-03-03T08:44:15.300 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2023-03-03 08:43:43.974812013 +0000 2023-03-03T08:44:15.300 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2023-03-03T08:44:15.300 DEBUG:teuthology.orchestra.run.smithi143:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2023-03-03T08:44:15.381 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T08:44:15.381 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 472aaef4ec3b3a2c29f73ca67854785a910f1f9b 2023-03-03T08:44:15.399 INFO:tasks.workunit.client.0.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-03-03T08:44:15.439 INFO:tasks.workunit.client.1.smithi143.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2023-03-03T08:44:22.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:22.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.323+0000 7f6cc8543700 1 Processor -- start 2023-03-03T08:44:22.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.323+0000 7f6cc8543700 1 -- start start 2023-03-03T08:44:22.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.324+0000 7f6cc8543700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6cc014fbe0 0x7f6cc00c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.324+0000 7f6cc8543700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6cc00c3e30 0x7f6cc00c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.324+0000 7f6cc8543700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc00aa530 con 0x7f6cc006d240 2023-03-03T08:44:22.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.324+0000 7f6cc8543700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc00ad2d0 con 0x7f6cc014fbe0 2023-03-03T08:44:22.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.324+0000 7f6cc8543700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc0153300 con 0x7f6cc00c3e30 2023-03-03T08:44:22.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.324+0000 7f6cc62df700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6cc014fbe0 0x7f6cc00c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc62df700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6cc014fbe0 0x7f6cc00c1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52604/0 (socket says 172.21.15.7:52604) 2023-03-03T08:44:22.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc62df700 1 -- 172.21.15.7:0/4093659370 learned_addr learned my addr 172.21.15.7:0/4093659370 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:22.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc6ae0700 1 --2- 172.21.15.7:0/4093659370 >> v2:172.21.15.143:3300/0 conn(0x7f6cc00c3e30 0x7f6cc00c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc62df700 1 -- 172.21.15.7:0/4093659370 >> v2:172.21.15.143:3300/0 conn(0x7f6cc00c3e30 msgr2=0x7f6cc00c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:22.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc62df700 1 --2- 172.21.15.7:0/4093659370 >> v2:172.21.15.143:3300/0 conn(0x7f6cc00c3e30 0x7f6cc00c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc62df700 1 -- 172.21.15.7:0/4093659370 >> v1:172.21.15.7:6789/0 conn(0x7f6cc006d240 legacy=0x7f6cc00c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc62df700 1 -- 172.21.15.7:0/4093659370 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc00c2490 con 0x7f6cc014fbe0 2023-03-03T08:44:22.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc52dd700 1 -- 172.21.15.7:0/4093659370 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981787265 0 0) 0x7f6cc00aa530 con 0x7f6cc006d240 2023-03-03T08:44:22.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc62df700 1 --2- 172.21.15.7:0/4093659370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 0x7f6cc00c1f50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0156a10 tx=0x7f6cb0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1467cd4d068836 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:22.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc52dd700 1 -- 172.21.15.7:0/4093659370 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0011830 con 0x7f6cc014fbe0 2023-03-03T08:44:22.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc52dd700 1 -- 172.21.15.7:0/4093659370 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb0011990 con 0x7f6cc014fbe0 2023-03-03T08:44:22.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.325+0000 7f6cc52dd700 1 -- 172.21.15.7:0/4093659370 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0011d20 con 0x7f6cc014fbe0 2023-03-03T08:44:22.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.326+0000 7f6cc8543700 1 -- 172.21.15.7:0/4093659370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 msgr2=0x7f6cc00c1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.326+0000 7f6cc8543700 1 --2- 172.21.15.7:0/4093659370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 0x7f6cc00c1f50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0156a10 tx=0x7f6cb0009bd0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.326+0000 7f6cc8543700 1 -- 172.21.15.7:0/4093659370 shutdown_connections 2023-03-03T08:44:22.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.326+0000 7f6cc8543700 1 --2- 172.21.15.7:0/4093659370 >> v2:172.21.15.143:3300/0 conn(0x7f6cc00c3e30 0x7f6cc00c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.326+0000 7f6cc8543700 1 --2- 172.21.15.7:0/4093659370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 0x7f6cc00c1f50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.326+0000 7f6cc8543700 1 -- 172.21.15.7:0/4093659370 >> 172.21.15.7:0/4093659370 conn(0x7f6cc006c110 msgr2=0x7f6cc006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:22.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.326+0000 7f6cc8543700 1 -- 172.21.15.7:0/4093659370 shutdown_connections 2023-03-03T08:44:22.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.326+0000 7f6cc8543700 1 -- 172.21.15.7:0/4093659370 wait complete. 2023-03-03T08:44:22.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.327+0000 7f6cc8543700 1 Processor -- start 2023-03-03T08:44:22.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.327+0000 7f6cc8543700 1 -- start start 2023-03-03T08:44:22.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc8543700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cc006d240 0x7f6cc014b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc8543700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc00c3e30 0x7f6cc01e8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc8543700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 0x7f6cc00c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc8543700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc01e7f60 con 0x7f6cc014fbe0 2023-03-03T08:44:22.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc8543700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc01ec460 con 0x7f6cc00c3e30 2023-03-03T08:44:22.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc8543700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc00c8190 con 0x7f6cc006d240 2023-03-03T08:44:22.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc6ae0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 0x7f6cc00c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc6ae0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 0x7f6cc00c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52614/0 (socket says 172.21.15.7:52614) 2023-03-03T08:44:22.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc6ae0700 1 -- 172.21.15.7:0/291803482 learned_addr learned my addr 172.21.15.7:0/291803482 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:22.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc5ade700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc00c3e30 0x7f6cc01e8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc62df700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cc006d240 0x7f6cc014b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc6ae0700 1 -- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cc006d240 msgr2=0x7f6cc014b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc6ae0700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cc006d240 0x7f6cc014b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc6ae0700 1 -- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc00c3e30 msgr2=0x7f6cc01e8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc6ae0700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc00c3e30 0x7f6cc01e8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc6ae0700 1 -- 172.21.15.7:0/291803482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb0009550 con 0x7f6cc014fbe0 2023-03-03T08:44:22.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc5ade700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc00c3e30 0x7f6cc01e8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:22.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.328+0000 7f6cc6ae0700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 0x7f6cc00c6250 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800efc0 tx=0x7f6cb800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:22.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.329+0000 7f6cb6ffd700 1 -- 172.21.15.7:0/291803482 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb800cce0 con 0x7f6cc014fbe0 2023-03-03T08:44:22.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.329+0000 7f6cb6ffd700 1 -- 172.21.15.7:0/291803482 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb800f040 con 0x7f6cc014fbe0 2023-03-03T08:44:22.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.329+0000 7f6cc8543700 1 -- 172.21.15.7:0/291803482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc0074570 con 0x7f6cc014fbe0 2023-03-03T08:44:22.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.329+0000 7f6cb6ffd700 1 -- 172.21.15.7:0/291803482 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8014480 con 0x7f6cc014fbe0 2023-03-03T08:44:22.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.329+0000 7f6cc8543700 1 -- 172.21.15.7:0/291803482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc0074290 con 0x7f6cc014fbe0 2023-03-03T08:44:22.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.330+0000 7f6cc8543700 1 -- 172.21.15.7:0/291803482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc0152f20 con 0x7f6cc014fbe0 2023-03-03T08:44:22.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.330+0000 7f6cb6ffd700 1 -- 172.21.15.7:0/291803482 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6cb80145e0 con 0x7f6cc014fbe0 2023-03-03T08:44:22.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.330+0000 7f6cb6ffd700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ca40682f0 0x7f6ca406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.331+0000 7f6cb6ffd700 1 -- 172.21.15.7:0/291803482 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6932+0+0 (secure 0 0 0) 0x7f6cb8086d30 con 0x7f6cc014fbe0 2023-03-03T08:44:22.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.331+0000 7f6cc62df700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ca40682f0 0x7f6ca406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.331+0000 7f6cc62df700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ca40682f0 0x7f6ca406a7a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0014040 tx=0x7f6cb0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:22.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.335+0000 7f6cb6ffd700 1 -- 172.21.15.7:0/291803482 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6cb8053220 con 0x7f6cc014fbe0 2023-03-03T08:44:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.579+0000 7f6cc8543700 1 -- 172.21.15.7:0/291803482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cc0153aa0 con 0x7f6cc014fbe0 2023-03-03T08:44:22.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.580+0000 7f6cb6ffd700 1 -- 172.21.15.7:0/291803482 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f6cb8018090 con 0x7f6cc014fbe0 2023-03-03T08:44:22.581 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:22.581 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.581+0000 7f6cc8543700 1 -- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ca40682f0 msgr2=0x7f6ca406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.581+0000 7f6cc8543700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ca40682f0 0x7f6ca406a7a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0014040 tx=0x7f6cb0008030 comp rx=0 tx=0).stop 2023-03-03T08:44:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.582+0000 7f6cc8543700 1 -- 172.21.15.7:0/291803482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 msgr2=0x7f6cc00c6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.582+0000 7f6cc8543700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 0x7f6cc00c6250 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800efc0 tx=0x7f6cb800c3a0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.582+0000 7f6cc8543700 1 -- 172.21.15.7:0/291803482 shutdown_connections 2023-03-03T08:44:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.582+0000 7f6cc8543700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ca40682f0 0x7f6ca406a7a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.582+0000 7f6cc8543700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc014fbe0 0x7f6cc00c6250 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.582+0000 7f6cc8543700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6cc006d240 0x7f6cc014b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.582+0000 7f6cc8543700 1 --2- 172.21.15.7:0/291803482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc00c3e30 0x7f6cc01e8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.582+0000 7f6cc8543700 1 -- 172.21.15.7:0/291803482 >> 172.21.15.7:0/291803482 conn(0x7f6cc006c110 msgr2=0x7f6cc0152280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.582+0000 7f6cc8543700 1 -- 172.21.15.7:0/291803482 shutdown_connections 2023-03-03T08:44:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.582+0000 7f6cc8543700 1 -- 172.21.15.7:0/291803482 wait complete. 2023-03-03T08:44:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:22.595 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2023-03-03T08:44:22.595 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-03-03T08:44:22.595 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:44:22.604 INFO:tasks.ceph.mds.c:Stopped 2023-03-03T08:44:22.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:22.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.768+0000 7f0cf5341700 1 Processor -- start 2023-03-03T08:44:22.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.768+0000 7f0cf5341700 1 -- start start 2023-03-03T08:44:22.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.769+0000 7f0cf5341700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0cf006d870 0x7f0cf01550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.769+0000 7f0cf5341700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf00c7a60 con 0x7f0cf00c3e30 2023-03-03T08:44:22.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.769+0000 7f0cf5341700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf00c3620 con 0x7f0cf00c1da0 2023-03-03T08:44:22.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.769+0000 7f0cf5341700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf00c2af0 con 0x7f0cf006d870 2023-03-03T08:44:22.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.769+0000 7f0cef7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0cf00c3e30 0x7f0cf015ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46602/0 (socket says 172.21.15.7:46602) 2023-03-03T08:44:22.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.769+0000 7f0cef7fe700 1 -- 172.21.15.7:0/744346234 learned_addr learned my addr 172.21.15.7:0/744346234 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:22.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.769+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/744346234 >> v2:172.21.15.143:3300/0 conn(0x7f0cf006d870 0x7f0cf01550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.769+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347431305 0 0) 0x7f0cf00c7a60 con 0x7f0cf00c3e30 2023-03-03T08:44:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.769+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd80035b0 con 0x7f0cf00c3e30 2023-03-03T08:44:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4078995580 0 0) 0x7f0cd80035b0 con 0x7f0cf00c3e30 2023-03-03T08:44:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 >> v2:172.21.15.143:3300/0 conn(0x7f0cf006d870 msgr2=0x7f0cf01550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 --2- 172.21.15.7:0/744346234 >> v2:172.21.15.143:3300/0 conn(0x7f0cf006d870 0x7f0cf01550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 >> v1:172.21.15.143:6789/0 conn(0x7f0cf00c1da0 legacy=0x7f0cf00c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf01545e0 con 0x7f0cf00c3e30 2023-03-03T08:44:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0ce80019f0 con 0x7f0cf00c3e30 2023-03-03T08:44:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 >> v1:172.21.15.7:6789/0 conn(0x7f0cf00c3e30 legacy=0x7f0cf015ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cd8003a70 0x7f0cd8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cd8004850 0x7f0cd8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cd8005200 0x7f0cd800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd800c0e0 con 0x7f0cd8003a70 2023-03-03T08:44:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd800c220 con 0x7f0cd8005200 2023-03-03T08:44:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd800c3e0 con 0x7f0cd8004850 2023-03-03T08:44:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cd8003a70 0x7f0cd8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cef7fe700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cd8005200 0x7f0cd800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cee7fc700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cd8004850 0x7f0cd8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0ceeffd700 1 -- 172.21.15.7:0/744346234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cd8004850 msgr2=0x7f0cd8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cd8004850 0x7f0cd8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0ceeffd700 1 -- 172.21.15.7:0/744346234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cd8005200 msgr2=0x7f0cd800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cd8005200 0x7f0cd800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cef7fe700 1 -- 172.21.15.7:0/744346234 reap_dead start 2023-03-03T08:44:22.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0ceeffd700 1 -- 172.21.15.7:0/744346234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd80035b0 con 0x7f0cd8003a70 2023-03-03T08:44:22.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cd8003a70 0x7f0cd8003e80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009b60 tx=0x7f0cd8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e514837137348ec7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:22.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.770+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce400e6c0 con 0x7f0cd8003a70 2023-03-03T08:44:22.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.771+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce400e820 con 0x7f0cd8003a70 2023-03-03T08:44:22.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.771+0000 7f0cedffb700 1 -- 172.21.15.7:0/744346234 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce400eb90 con 0x7f0cd8003a70 2023-03-03T08:44:22.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.771+0000 7f0cf5341700 1 -- 172.21.15.7:0/744346234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cd8003a70 msgr2=0x7f0cd8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.771+0000 7f0cf5341700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cd8003a70 0x7f0cd8003e80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009b60 tx=0x7f0cd8005880 comp rx=0 tx=0).stop 2023-03-03T08:44:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.771+0000 7f0cf5341700 1 -- 172.21.15.7:0/744346234 shutdown_connections 2023-03-03T08:44:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.771+0000 7f0cf5341700 1 --2- 172.21.15.7:0/744346234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cd8003a70 0x7f0cd8003e80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.771+0000 7f0cf5341700 1 -- 172.21.15.7:0/744346234 >> 172.21.15.7:0/744346234 conn(0x7f0cf006c110 msgr2=0x7f0cf006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.771+0000 7f0cf5341700 1 -- 172.21.15.7:0/744346234 shutdown_connections 2023-03-03T08:44:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.771+0000 7f0cf5341700 1 -- 172.21.15.7:0/744346234 wait complete. 2023-03-03T08:44:22.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cf5341700 1 Processor -- start 2023-03-03T08:44:22.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cf5341700 1 -- start start 2023-03-03T08:44:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cf5341700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cd8003a70 0x7f0cf0157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cf5341700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0157110 0x7f0cf00aaa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cf5341700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf013a3a0 0x7f0cf00aaf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cf5341700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf00ac9f0 con 0x7f0cf013a3a0 2023-03-03T08:44:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cf5341700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf0139e50 con 0x7f0cf0157110 2023-03-03T08:44:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cf5341700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf013a110 con 0x7f0cd8003a70 2023-03-03T08:44:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf013a3a0 0x7f0cf00aaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf013a3a0 0x7f0cf00aaf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52634/0 (socket says 172.21.15.7:52634) 2023-03-03T08:44:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.773+0000 7f0cef7fe700 1 -- 172.21.15.7:0/3076480065 learned_addr learned my addr 172.21.15.7:0/3076480065 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cd8003a70 0x7f0cf0157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cee7fc700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0157110 0x7f0cf00aaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cef7fe700 1 -- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cd8003a70 msgr2=0x7f0cf0157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cef7fe700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cd8003a70 0x7f0cf0157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cef7fe700 1 -- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0157110 msgr2=0x7f0cf00aaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cef7fe700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0157110 0x7f0cf00aaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cef7fe700 1 -- 172.21.15.7:0/3076480065 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0154ba0 con 0x7f0cf013a3a0 2023-03-03T08:44:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cd8003a70 0x7f0cf0157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cee7fc700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0157110 0x7f0cf00aaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cef7fe700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf013a3a0 0x7f0cf00aaf60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8000f80 tx=0x7f0ce800a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cd77fe700 1 -- 172.21.15.7:0/3076480065 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800ada0 con 0x7f0cf013a3a0 2023-03-03T08:44:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cd77fe700 1 -- 172.21.15.7:0/3076480065 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce80133f0 con 0x7f0cf013a3a0 2023-03-03T08:44:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cf5341700 1 -- 172.21.15.7:0/3076480065 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf01548c0 con 0x7f0cf013a3a0 2023-03-03T08:44:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cd77fe700 1 -- 172.21.15.7:0/3076480065 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce80137a0 con 0x7f0cf013a3a0 2023-03-03T08:44:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.774+0000 7f0cf5341700 1 -- 172.21.15.7:0/3076480065 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf006dc40 con 0x7f0cf013a3a0 2023-03-03T08:44:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.775+0000 7f0cf5341700 1 -- 172.21.15.7:0/3076480065 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf0156d60 con 0x7f0cf013a3a0 2023-03-03T08:44:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.776+0000 7f0cd77fe700 1 -- 172.21.15.7:0/3076480065 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0ce8013900 con 0x7f0cf013a3a0 2023-03-03T08:44:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.776+0000 7f0cd77fe700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cdc0682f0 0x7f0cdc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.776+0000 7f0cd77fe700 1 -- 172.21.15.7:0/3076480065 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6932+0+0 (secure 0 0 0) 0x7f0ce8086a60 con 0x7f0cf013a3a0 2023-03-03T08:44:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.776+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cdc0682f0 0x7f0cdc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:22.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.777+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cdc0682f0 0x7f0cdc06a7a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40090b0 tx=0x7f0ce4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:22.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:22.780+0000 7f0cd77fe700 1 -- 172.21.15.7:0/3076480065 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0ce8052f50 con 0x7f0cf013a3a0 2023-03-03T08:44:23.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.020+0000 7f0cf5341700 1 -- 172.21.15.7:0/3076480065 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cf0156ef0 con 0x7f0cf013a3a0 2023-03-03T08:44:23.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.020+0000 7f0cd77fe700 1 -- 172.21.15.7:0/3076480065 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f0ce8056570 con 0x7f0cf013a3a0 2023-03-03T08:44:23.022 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:23.022 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.022+0000 7f0cf5341700 1 -- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cdc0682f0 msgr2=0x7f0cdc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.022+0000 7f0cf5341700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cdc0682f0 0x7f0cdc06a7a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40090b0 tx=0x7f0ce4008030 comp rx=0 tx=0).stop 2023-03-03T08:44:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.022+0000 7f0cf5341700 1 -- 172.21.15.7:0/3076480065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf013a3a0 msgr2=0x7f0cf00aaf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.022+0000 7f0cf5341700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf013a3a0 0x7f0cf00aaf60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8000f80 tx=0x7f0ce800a5f0 comp rx=0 tx=0).stop 2023-03-03T08:44:23.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.022+0000 7f0cf5341700 1 -- 172.21.15.7:0/3076480065 shutdown_connections 2023-03-03T08:44:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.022+0000 7f0cf5341700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cdc0682f0 0x7f0cdc06a7a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.022+0000 7f0cf5341700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf013a3a0 0x7f0cf00aaf60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.022+0000 7f0cf5341700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cd8003a70 0x7f0cf0157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.022+0000 7f0cf5341700 1 --2- 172.21.15.7:0/3076480065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0157110 0x7f0cf00aaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.022+0000 7f0cf5341700 1 -- 172.21.15.7:0/3076480065 >> 172.21.15.7:0/3076480065 conn(0x7f0cf006c110 msgr2=0x7f0cf006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.023+0000 7f0cf5341700 1 -- 172.21.15.7:0/3076480065 shutdown_connections 2023-03-03T08:44:23.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:23.023+0000 7f0cf5341700 1 -- 172.21.15.7:0/3076480065 wait complete. 2023-03-03T08:44:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:23.036 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-03-03T08:44:25.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:25.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.175+0000 7ff725ebe700 1 Processor -- start 2023-03-03T08:44:25.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.175+0000 7ff725ebe700 1 -- start start 2023-03-03T08:44:25.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.176+0000 7ff725ebe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff72006a2b0 0x7ff72006a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:25.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.176+0000 7ff725ebe700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7201558f0 con 0x7ff7200a6370 2023-03-03T08:44:25.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.176+0000 7ff725ebe700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7200a6180 con 0x7ff7200c5cb0 2023-03-03T08:44:25.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.176+0000 7ff725ebe700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff72006bab0 con 0x7ff72006a2b0 2023-03-03T08:44:25.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.176+0000 7ff71f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff72006a2b0 0x7ff72006a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:25.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.176+0000 7ff71f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff72006a2b0 0x7ff72006a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52638/0 (socket says 172.21.15.7:52638) 2023-03-03T08:44:25.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.176+0000 7ff71f7fe700 1 -- 172.21.15.7:0/2967190407 learned_addr learned my addr 172.21.15.7:0/2967190407 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:25.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.176+0000 7ff71effd700 1 --1- 172.21.15.7:0/2967190407 >> v1:172.21.15.7:6789/0 conn(0x7ff7200a6370 0x7ff720155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46618/0 (socket says 172.21.15.7:46618) 2023-03-03T08:44:25.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.177+0000 7ff71f7fe700 1 -- 172.21.15.7:0/2967190407 >> v1:172.21.15.143:6789/0 conn(0x7ff7200c5cb0 legacy=0x7ff72015cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:25.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.177+0000 7ff71f7fe700 1 -- 172.21.15.7:0/2967190407 >> v1:172.21.15.7:6789/0 conn(0x7ff7200a6370 legacy=0x7ff720155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:25.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.177+0000 7ff71e7fc700 1 -- 172.21.15.7:0/2967190407 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253683405 0 0) 0x7ff7201558f0 con 0x7ff7200a6370 2023-03-03T08:44:25.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.177+0000 7ff71f7fe700 1 -- 172.21.15.7:0/2967190407 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7200a5ea0 con 0x7ff72006a2b0 2023-03-03T08:44:25.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.177+0000 7ff71f7fe700 1 --2- 172.21.15.7:0/2967190407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72006a2b0 0x7ff72006a680 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff7100098f0 tx=0x7ff710009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d296aa37f8cda1a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:25.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.177+0000 7ff71e7fc700 1 -- 172.21.15.7:0/2967190407 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71000b8a0 con 0x7ff72006a2b0 2023-03-03T08:44:25.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.177+0000 7ff71e7fc700 1 -- 172.21.15.7:0/2967190407 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71000ba00 con 0x7ff72006a2b0 2023-03-03T08:44:25.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.177+0000 7ff71e7fc700 1 -- 172.21.15.7:0/2967190407 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71000bd90 con 0x7ff72006a2b0 2023-03-03T08:44:25.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.178+0000 7ff725ebe700 1 -- 172.21.15.7:0/2967190407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72006a2b0 msgr2=0x7ff72006a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:25.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.178+0000 7ff725ebe700 1 --2- 172.21.15.7:0/2967190407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72006a2b0 0x7ff72006a680 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff7100098f0 tx=0x7ff710009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:25.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.178+0000 7ff725ebe700 1 -- 172.21.15.7:0/2967190407 shutdown_connections 2023-03-03T08:44:25.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.178+0000 7ff725ebe700 1 --2- 172.21.15.7:0/2967190407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff72006a2b0 0x7ff72006a680 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:25.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.178+0000 7ff725ebe700 1 -- 172.21.15.7:0/2967190407 >> 172.21.15.7:0/2967190407 conn(0x7ff7200a9430 msgr2=0x7ff7200c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:25.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.178+0000 7ff725ebe700 1 -- 172.21.15.7:0/2967190407 shutdown_connections 2023-03-03T08:44:25.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.178+0000 7ff725ebe700 1 -- 172.21.15.7:0/2967190407 wait complete. 2023-03-03T08:44:25.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.179+0000 7ff725ebe700 1 Processor -- start 2023-03-03T08:44:25.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.179+0000 7ff725ebe700 1 -- start start 2023-03-03T08:44:25.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff725ebe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff72006a2b0 0x7ff720070240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:25.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff725ebe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7200a6370 0x7ff720148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:25.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff725ebe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7200c5cb0 0x7ff720149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:25.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff725ebe700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff720071240 con 0x7ff7200a6370 2023-03-03T08:44:25.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff725ebe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7201ec1a0 con 0x7ff72006a2b0 2023-03-03T08:44:25.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff725ebe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7201eb3e0 con 0x7ff7200c5cb0 2023-03-03T08:44:25.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7200a6370 0x7ff720148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:25.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7200a6370 0x7ff720148e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52646/0 (socket says 172.21.15.7:52646) 2023-03-03T08:44:25.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71effd700 1 -- 172.21.15.7:0/2493587022 learned_addr learned my addr 172.21.15.7:0/2493587022 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:25.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71f7fe700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff72006a2b0 0x7ff720070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:25.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71ffff700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7200c5cb0 0x7ff720149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:25.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71effd700 1 -- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7200c5cb0 msgr2=0x7ff720149380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:25.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71effd700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7200c5cb0 0x7ff720149380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:25.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71effd700 1 -- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff72006a2b0 msgr2=0x7ff720070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:25.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71effd700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff72006a2b0 0x7ff720070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:25.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71effd700 1 -- 172.21.15.7:0/2493587022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff710009550 con 0x7ff7200a6370 2023-03-03T08:44:25.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.180+0000 7ff71effd700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7200a6370 0x7ff720148e40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff70800c550 tx=0x7ff70800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:25.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.181+0000 7ff717fff700 1 -- 172.21.15.7:0/2493587022 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff708004c80 con 0x7ff7200a6370 2023-03-03T08:44:25.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.181+0000 7ff717fff700 1 -- 172.21.15.7:0/2493587022 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff708004e00 con 0x7ff7200a6370 2023-03-03T08:44:25.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.181+0000 7ff725ebe700 1 -- 172.21.15.7:0/2493587022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff720153160 con 0x7ff7200a6370 2023-03-03T08:44:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.181+0000 7ff717fff700 1 -- 172.21.15.7:0/2493587022 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff708007310 con 0x7ff7200a6370 2023-03-03T08:44:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.181+0000 7ff725ebe700 1 -- 172.21.15.7:0/2493587022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff720152e80 con 0x7ff7200a6370 2023-03-03T08:44:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.182+0000 7ff725ebe700 1 -- 172.21.15.7:0/2493587022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7200c1500 con 0x7ff7200a6370 2023-03-03T08:44:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.182+0000 7ff717fff700 1 -- 172.21.15.7:0/2493587022 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff70801a020 con 0x7ff7200a6370 2023-03-03T08:44:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.182+0000 7ff717fff700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff700068320 0x7ff70006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.183+0000 7ff717fff700 1 -- 172.21.15.7:0/2493587022 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6932+0+0 (secure 0 0 0) 0x7ff708085c10 con 0x7ff7200a6370 2023-03-03T08:44:25.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.183+0000 7ff71f7fe700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff700068320 0x7ff70006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:25.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.183+0000 7ff71f7fe700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff700068320 0x7ff70006a7d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff710009130 tx=0x7ff710008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:25.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.187+0000 7ff717fff700 1 -- 172.21.15.7:0/2493587022 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff708052100 con 0x7ff7200a6370 2023-03-03T08:44:25.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.428+0000 7ff725ebe700 1 -- 172.21.15.7:0/2493587022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7201ebff0 con 0x7ff7200a6370 2023-03-03T08:44:25.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.428+0000 7ff717fff700 1 -- 172.21.15.7:0/2493587022 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7ff708055720 con 0x7ff7200a6370 2023-03-03T08:44:25.430 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:25.430 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:25.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.430+0000 7ff725ebe700 1 -- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff700068320 msgr2=0x7ff70006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:25.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.430+0000 7ff725ebe700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff700068320 0x7ff70006a7d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff710009130 tx=0x7ff710008030 comp rx=0 tx=0).stop 2023-03-03T08:44:25.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.430+0000 7ff725ebe700 1 -- 172.21.15.7:0/2493587022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7200a6370 msgr2=0x7ff720148e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:25.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.430+0000 7ff725ebe700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7200a6370 0x7ff720148e40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff70800c550 tx=0x7ff70800c910 comp rx=0 tx=0).stop 2023-03-03T08:44:25.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.431+0000 7ff725ebe700 1 -- 172.21.15.7:0/2493587022 shutdown_connections 2023-03-03T08:44:25.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.431+0000 7ff725ebe700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7200c5cb0 0x7ff720149380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:25.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.431+0000 7ff725ebe700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff72006a2b0 0x7ff720070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:25.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.431+0000 7ff725ebe700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff700068320 0x7ff70006a7d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:25.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.431+0000 7ff725ebe700 1 --2- 172.21.15.7:0/2493587022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7200a6370 0x7ff720148e40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:25.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.431+0000 7ff725ebe700 1 -- 172.21.15.7:0/2493587022 >> 172.21.15.7:0/2493587022 conn(0x7ff7200a9430 msgr2=0x7ff7200a74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:25.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.431+0000 7ff725ebe700 1 -- 172.21.15.7:0/2493587022 shutdown_connections 2023-03-03T08:44:25.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:25.431+0000 7ff725ebe700 1 -- 172.21.15.7:0/2493587022 wait complete. 2023-03-03T08:44:25.434 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:25.444 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-03-03T08:44:27.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:27.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.583+0000 7f29247e1700 1 Processor -- start 2023-03-03T08:44:27.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.583+0000 7f29247e1700 1 -- start start 2023-03-03T08:44:27.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f29247e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c13a0f0 0x7f291c0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:27.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f29247e1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f291c1586c0 0x7f291c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:27.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f29247e1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291c06de90 con 0x7f291c0c1da0 2023-03-03T08:44:27.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f29247e1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291c154840 con 0x7f291c1586c0 2023-03-03T08:44:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f29247e1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c154560 con 0x7f291c13a0f0 2023-03-03T08:44:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f2922d7e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f291c0c1da0 0x7f291c06d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46634/0 (socket says 172.21.15.7:46634) 2023-03-03T08:44:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f2922d7e700 1 -- 172.21.15.7:0/1030385774 learned_addr learned my addr 172.21.15.7:0/1030385774 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f2921d7c700 1 --2- 172.21.15.7:0/1030385774 >> v2:172.21.15.143:3300/0 conn(0x7f291c1586c0 0x7f291c06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f292257d700 1 --2- 172.21.15.7:0/1030385774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c13a0f0 0x7f291c0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f292157b700 1 -- 172.21.15.7:0/1030385774 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963931927 0 0) 0x7f2914001490 con 0x7f291c0c1da0 2023-03-03T08:44:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f292157b700 1 -- 172.21.15.7:0/1030385774 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29080035b0 con 0x7f291c0c1da0 2023-03-03T08:44:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f292257d700 1 -- 172.21.15.7:0/1030385774 >> v2:172.21.15.143:3300/0 conn(0x7f291c1586c0 msgr2=0x7f291c06d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f292257d700 1 --2- 172.21.15.7:0/1030385774 >> v2:172.21.15.143:3300/0 conn(0x7f291c1586c0 0x7f291c06d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f292257d700 1 -- 172.21.15.7:0/1030385774 >> v1:172.21.15.7:6789/0 conn(0x7f291c0c1da0 legacy=0x7f291c06d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f292157b700 1 -- 172.21.15.7:0/1030385774 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1454397308 0 0) 0x7f29080035b0 con 0x7f291c0c1da0 2023-03-03T08:44:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.584+0000 7f292257d700 1 -- 172.21.15.7:0/1030385774 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c157be0 con 0x7f291c13a0f0 2023-03-03T08:44:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f292257d700 1 --2- 172.21.15.7:0/1030385774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c13a0f0 0x7f291c0c07d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2918009840 tx=0x7f2918009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f181b46c2ed53514 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f292157b700 1 -- 172.21.15.7:0/1030385774 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2914001b80 con 0x7f291c13a0f0 2023-03-03T08:44:27.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f292157b700 1 -- 172.21.15.7:0/1030385774 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2918004030 con 0x7f291c13a0f0 2023-03-03T08:44:27.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f292157b700 1 -- 172.21.15.7:0/1030385774 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29180043a0 con 0x7f291c13a0f0 2023-03-03T08:44:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f29247e1700 1 -- 172.21.15.7:0/1030385774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c13a0f0 msgr2=0x7f291c0c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f29247e1700 1 --2- 172.21.15.7:0/1030385774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c13a0f0 0x7f291c0c07d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2918009840 tx=0x7f2918009b50 comp rx=0 tx=0).stop 2023-03-03T08:44:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f29247e1700 1 -- 172.21.15.7:0/1030385774 shutdown_connections 2023-03-03T08:44:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f29247e1700 1 --2- 172.21.15.7:0/1030385774 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c13a0f0 0x7f291c0c07d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f29247e1700 1 --2- 172.21.15.7:0/1030385774 >> v2:172.21.15.143:3300/0 conn(0x7f291c1586c0 0x7f291c06d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f29247e1700 1 -- 172.21.15.7:0/1030385774 >> 172.21.15.7:0/1030385774 conn(0x7f291c06c110 msgr2=0x7f291c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f29247e1700 1 -- 172.21.15.7:0/1030385774 shutdown_connections 2023-03-03T08:44:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.585+0000 7f29247e1700 1 -- 172.21.15.7:0/1030385774 wait complete. 2023-03-03T08:44:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.587+0000 7f29247e1700 1 Processor -- start 2023-03-03T08:44:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.587+0000 7f29247e1700 1 -- start start 2023-03-03T08:44:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.587+0000 7f29247e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c0c1da0 0x7f291c156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.587+0000 7f29247e1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f291c13a0f0 0x7f291c1ddb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.587+0000 7f29247e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f291c1586c0 0x7f291c1de080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:27.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.587+0000 7f29247e1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1ed880 con 0x7f291c1586c0 2023-03-03T08:44:27.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.587+0000 7f29247e1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1ecfe0 con 0x7f291c13a0f0 2023-03-03T08:44:27.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.587+0000 7f29247e1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c072440 con 0x7f291c0c1da0 2023-03-03T08:44:27.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.587+0000 7f2922d7e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f291c1586c0 0x7f291c1de080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:27.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2922d7e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f291c1586c0 0x7f291c1de080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52662/0 (socket says 172.21.15.7:52662) 2023-03-03T08:44:27.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2922d7e700 1 -- 172.21.15.7:0/2035816202 learned_addr learned my addr 172.21.15.7:0/2035816202 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:27.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2921d7c700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f291c13a0f0 0x7f291c1ddb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:27.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f292257d700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c0c1da0 0x7f291c156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:27.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2922d7e700 1 -- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c0c1da0 msgr2=0x7f291c156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:27.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2922d7e700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c0c1da0 0x7f291c156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:27.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2922d7e700 1 -- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f291c13a0f0 msgr2=0x7f291c1ddb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:27.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2922d7e700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f291c13a0f0 0x7f291c1ddb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:27.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2922d7e700 1 -- 172.21.15.7:0/2035816202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29180094a0 con 0x7f291c1586c0 2023-03-03T08:44:27.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2922d7e700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f291c1586c0 0x7f291c1de080 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f291400cc10 tx=0x7f291400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:27.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2912ffd700 1 -- 172.21.15.7:0/2035816202 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291400ab90 con 0x7f291c1586c0 2023-03-03T08:44:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2912ffd700 1 -- 172.21.15.7:0/2035816202 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291400ad10 con 0x7f291c1586c0 2023-03-03T08:44:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f29247e1700 1 -- 172.21.15.7:0/2035816202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c1581a0 con 0x7f291c1586c0 2023-03-03T08:44:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f2912ffd700 1 -- 172.21.15.7:0/2035816202 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2914007310 con 0x7f291c1586c0 2023-03-03T08:44:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.588+0000 7f29247e1700 1 -- 172.21.15.7:0/2035816202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c157ec0 con 0x7f291c1586c0 2023-03-03T08:44:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.589+0000 7f29247e1700 1 -- 172.21.15.7:0/2035816202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c06e1a0 con 0x7f291c1586c0 2023-03-03T08:44:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.590+0000 7f2912ffd700 1 -- 172.21.15.7:0/2035816202 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f291401a030 con 0x7f291c1586c0 2023-03-03T08:44:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.590+0000 7f2912ffd700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29000682f0 0x7f290006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.590+0000 7f2912ffd700 1 -- 172.21.15.7:0/2035816202 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6932+0+0 (secure 0 0 0) 0x7f2914085bd0 con 0x7f291c1586c0 2023-03-03T08:44:27.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.590+0000 7f292257d700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29000682f0 0x7f290006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:27.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.591+0000 7f292257d700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29000682f0 0x7f290006a7a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f291c134e20 tx=0x7f2918013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:27.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.594+0000 7f2912ffd700 1 -- 172.21.15.7:0/2035816202 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f29140520c0 con 0x7f291c1586c0 2023-03-03T08:44:27.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.831+0000 7f29247e1700 1 -- 172.21.15.7:0/2035816202 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f291c06de90 con 0x7f291c1586c0 2023-03-03T08:44:27.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.832+0000 7f2912ffd700 1 -- 172.21.15.7:0/2035816202 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f291c06de90 con 0x7f291c1586c0 2023-03-03T08:44:27.834 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:27.835 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:27.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 -- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29000682f0 msgr2=0x7f290006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:27.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29000682f0 0x7f290006a7a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f291c134e20 tx=0x7f2918013040 comp rx=0 tx=0).stop 2023-03-03T08:44:27.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 -- 172.21.15.7:0/2035816202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f291c1586c0 msgr2=0x7f291c1de080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:27.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f291c1586c0 0x7f291c1de080 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f291400cc10 tx=0x7f291400a060 comp rx=0 tx=0).stop 2023-03-03T08:44:27.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 -- 172.21.15.7:0/2035816202 shutdown_connections 2023-03-03T08:44:27.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29000682f0 0x7f290006a7a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:27.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f291c1586c0 0x7f291c1de080 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:27.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f291c0c1da0 0x7f291c156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:27.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 --2- 172.21.15.7:0/2035816202 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f291c13a0f0 0x7f291c1ddb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:27.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 -- 172.21.15.7:0/2035816202 >> 172.21.15.7:0/2035816202 conn(0x7f291c06c110 msgr2=0x7f291c138b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:27.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 -- 172.21.15.7:0/2035816202 shutdown_connections 2023-03-03T08:44:27.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:27.834+0000 7f29247e1700 1 -- 172.21.15.7:0/2035816202 wait complete. 2023-03-03T08:44:27.839 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:27.848 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-03-03T08:44:29.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:29.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.987+0000 7f1860fd6700 1 Processor -- start 2023-03-03T08:44:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.987+0000 7f1860fd6700 1 -- start start 2023-03-03T08:44:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f1860fd6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f185c06d870 0x7f185c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f1860fd6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c0c7a60 con 0x7f185c0c3e30 2023-03-03T08:44:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f1860fd6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c0c3620 con 0x7f185c0c1da0 2023-03-03T08:44:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f1860fd6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185c0c2af0 con 0x7f185c06d870 2023-03-03T08:44:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f185a59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f185c06d870 0x7f185c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f185a59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f185c06d870 0x7f185c1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52666/0 (socket says 172.21.15.7:52666) 2023-03-03T08:44:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f185ad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f185c0c3e30 0x7f185c15ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46650/0 (socket says 172.21.15.7:46650) 2023-03-03T08:44:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f185a59c700 1 -- 172.21.15.7:0/3139198385 learned_addr learned my addr 172.21.15.7:0/3139198385 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f185a59c700 1 -- 172.21.15.7:0/3139198385 >> v1:172.21.15.143:6789/0 conn(0x7f185c0c1da0 legacy=0x7f185c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f185a59c700 1 -- 172.21.15.7:0/3139198385 >> v1:172.21.15.7:6789/0 conn(0x7f185c0c3e30 legacy=0x7f185c15ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.988+0000 7f185a59c700 1 -- 172.21.15.7:0/3139198385 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c1545e0 con 0x7f185c06d870 2023-03-03T08:44:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.989+0000 7f185a59c700 1 --2- 172.21.15.7:0/3139198385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c06d870 0x7f185c1550c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f18500098f0 tx=0x7f1850009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc3917341f99be7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.989+0000 7f185959a700 1 -- 172.21.15.7:0/3139198385 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1850004230 con 0x7f185c06d870 2023-03-03T08:44:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.989+0000 7f185959a700 1 -- 172.21.15.7:0/3139198385 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1850004390 con 0x7f185c06d870 2023-03-03T08:44:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.989+0000 7f185959a700 1 -- 172.21.15.7:0/3139198385 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18500036a0 con 0x7f185c06d870 2023-03-03T08:44:29.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.989+0000 7f1860fd6700 1 -- 172.21.15.7:0/3139198385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c06d870 msgr2=0x7f185c1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:29.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.989+0000 7f1860fd6700 1 --2- 172.21.15.7:0/3139198385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c06d870 0x7f185c1550c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f18500098f0 tx=0x7f1850009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:29.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.989+0000 7f1860fd6700 1 -- 172.21.15.7:0/3139198385 shutdown_connections 2023-03-03T08:44:29.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.989+0000 7f1860fd6700 1 --2- 172.21.15.7:0/3139198385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c06d870 0x7f185c1550c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:29.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.989+0000 7f1860fd6700 1 -- 172.21.15.7:0/3139198385 >> 172.21.15.7:0/3139198385 conn(0x7f185c06c110 msgr2=0x7f185c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:29.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.990+0000 7f1860fd6700 1 -- 172.21.15.7:0/3139198385 shutdown_connections 2023-03-03T08:44:29.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.990+0000 7f1860fd6700 1 -- 172.21.15.7:0/3139198385 wait complete. 2023-03-03T08:44:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.991+0000 7f1860fd6700 1 Processor -- start 2023-03-03T08:44:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.991+0000 7f1860fd6700 1 -- start start 2023-03-03T08:44:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1860fd6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f185c06d870 0x7f185c150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1860fd6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c0c1da0 0x7f185c14eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1860fd6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f185c0c3e30 0x7f185c0c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1860fd6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0ad1a0 con 0x7f185c0c1da0 2023-03-03T08:44:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1860fd6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c13cd40 con 0x7f185c06d870 2023-03-03T08:44:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1860fd6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0ace00 con 0x7f185c0c3e30 2023-03-03T08:44:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1859d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c0c1da0 0x7f185c14eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1859d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c0c1da0 0x7f185c14eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52674/0 (socket says 172.21.15.7:52674) 2023-03-03T08:44:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1859d9b700 1 -- 172.21.15.7:0/3052458806 learned_addr learned my addr 172.21.15.7:0/3052458806 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f185ad9d700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f185c0c3e30 0x7f185c0c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f185a59c700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f185c06d870 0x7f185c150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1859d9b700 1 -- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f185c0c3e30 msgr2=0x7f185c0c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1859d9b700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f185c0c3e30 0x7f185c0c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1859d9b700 1 -- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f185c06d870 msgr2=0x7f185c150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1859d9b700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f185c06d870 0x7f185c150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1859d9b700 1 -- 172.21.15.7:0/3052458806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1850009550 con 0x7f185c0c1da0 2023-03-03T08:44:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f185a59c700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f185c06d870 0x7f185c150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:44:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.992+0000 7f1859d9b700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c0c1da0 0x7f185c14eb90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f184c00c4d0 tx=0x7f184c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.993+0000 7f184affd700 1 -- 172.21.15.7:0/3052458806 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c011790 con 0x7f185c0c1da0 2023-03-03T08:44:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.993+0000 7f184affd700 1 -- 172.21.15.7:0/3052458806 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c0118f0 con 0x7f185c0c1da0 2023-03-03T08:44:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.993+0000 7f184affd700 1 -- 172.21.15.7:0/3052458806 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c011ca0 con 0x7f185c0c1da0 2023-03-03T08:44:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.993+0000 7f1860fd6700 1 -- 172.21.15.7:0/3052458806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185c154ba0 con 0x7f185c0c1da0 2023-03-03T08:44:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.993+0000 7f1860fd6700 1 -- 172.21.15.7:0/3052458806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c1548c0 con 0x7f185c0c1da0 2023-03-03T08:44:30.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.994+0000 7f1860fd6700 1 -- 172.21.15.7:0/3052458806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185c06dc40 con 0x7f185c0c1da0 2023-03-03T08:44:30.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.995+0000 7f184affd700 1 -- 172.21.15.7:0/3052458806 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f184c010050 con 0x7f185c0c1da0 2023-03-03T08:44:30.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.995+0000 7f184affd700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f183c0682f0 0x7f183c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:30.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.995+0000 7f184affd700 1 -- 172.21.15.7:0/3052458806 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6932+0+0 (secure 0 0 0) 0x7f184c0868c0 con 0x7f185c0c1da0 2023-03-03T08:44:30.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.996+0000 7f185a59c700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f183c0682f0 0x7f183c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:30.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.996+0000 7f185a59c700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f183c0682f0 0x7f183c06a7a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f185c0aca50 tx=0x7f1850008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:30.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:29.999+0000 7f184affd700 1 -- 172.21.15.7:0/3052458806 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f184c052d40 con 0x7f185c0c1da0 2023-03-03T08:44:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.236+0000 7f1860fd6700 1 -- 172.21.15.7:0/3052458806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f185c074bb0 con 0x7f185c0c1da0 2023-03-03T08:44:30.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.237+0000 7f184affd700 1 -- 172.21.15.7:0/3052458806 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f184c056360 con 0x7f185c0c1da0 2023-03-03T08:44:30.238 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:30.238 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:30.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 -- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f183c0682f0 msgr2=0x7f183c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:30.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f183c0682f0 0x7f183c06a7a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f185c0aca50 tx=0x7f1850008030 comp rx=0 tx=0).stop 2023-03-03T08:44:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 -- 172.21.15.7:0/3052458806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c0c1da0 msgr2=0x7f185c14eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c0c1da0 0x7f185c14eb90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f184c00c4d0 tx=0x7f184c00c890 comp rx=0 tx=0).stop 2023-03-03T08:44:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 -- 172.21.15.7:0/3052458806 shutdown_connections 2023-03-03T08:44:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f185c0c3e30 0x7f185c0c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f185c06d870 0x7f185c150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f183c0682f0 0x7f183c06a7a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 --2- 172.21.15.7:0/3052458806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f185c0c1da0 0x7f185c14eb90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 -- 172.21.15.7:0/3052458806 >> 172.21.15.7:0/3052458806 conn(0x7f185c06c110 msgr2=0x7f185c071300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 -- 172.21.15.7:0/3052458806 shutdown_connections 2023-03-03T08:44:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:30.239+0000 7f1860fd6700 1 -- 172.21.15.7:0/3052458806 wait complete. 2023-03-03T08:44:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:30.252 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-03-03T08:44:32.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:32.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.379+0000 7f5a14416700 1 Processor -- start 2023-03-03T08:44:32.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.379+0000 7f5a14416700 1 -- start start 2023-03-03T08:44:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a14416700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5a0c151ab0 0x7f5a0c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:32.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a14416700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a0c073c30 0x7f5a0c06bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a14416700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c06c3e0 0x7f5a0c157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a14416700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c06d780 con 0x7f5a0c073c30 2023-03-03T08:44:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a14416700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c06cb70 con 0x7f5a0c151ab0 2023-03-03T08:44:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a14416700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c06ce50 con 0x7f5a0c06c3e0 2023-03-03T08:44:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a119b1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a0c073c30 0x7f5a0c06bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a119b1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a0c073c30 0x7f5a0c06bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52684/0 (socket says 172.21.15.7:52684) 2023-03-03T08:44:32.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a119b1700 1 -- 172.21.15.7:0/140960683 learned_addr learned my addr 172.21.15.7:0/140960683 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a121b2700 1 --2- 172.21.15.7:0/140960683 >> v2:172.21.15.143:3300/0 conn(0x7f5a0c151ab0 0x7f5a0c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a129b3700 1 --2- 172.21.15.7:0/140960683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c06c3e0 0x7f5a0c157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a119b1700 1 -- 172.21.15.7:0/140960683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c06c3e0 msgr2=0x7f5a0c157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a119b1700 1 --2- 172.21.15.7:0/140960683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c06c3e0 0x7f5a0c157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a119b1700 1 -- 172.21.15.7:0/140960683 >> v2:172.21.15.143:3300/0 conn(0x7f5a0c151ab0 msgr2=0x7f5a0c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a119b1700 1 --2- 172.21.15.7:0/140960683 >> v2:172.21.15.143:3300/0 conn(0x7f5a0c151ab0 0x7f5a0c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a119b1700 1 -- 172.21.15.7:0/140960683 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c06d130 con 0x7f5a0c073c30 2023-03-03T08:44:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a129b3700 1 --2- 172.21.15.7:0/140960683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c06c3e0 0x7f5a0c157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:44:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.380+0000 7f5a119b1700 1 --2- 172.21.15.7:0/140960683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c073c30 0x7f5a0c06bea0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c06bb20 tx=0x7f59fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d685fe0494bcf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a111b0700 1 -- 172.21.15.7:0/140960683 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc011830 con 0x7f5a0c073c30 2023-03-03T08:44:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a111b0700 1 -- 172.21.15.7:0/140960683 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc011990 con 0x7f5a0c073c30 2023-03-03T08:44:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a111b0700 1 -- 172.21.15.7:0/140960683 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc011d20 con 0x7f5a0c073c30 2023-03-03T08:44:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a14416700 1 -- 172.21.15.7:0/140960683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c073c30 msgr2=0x7f5a0c06bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a14416700 1 --2- 172.21.15.7:0/140960683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c073c30 0x7f5a0c06bea0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c06bb20 tx=0x7f59fc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a14416700 1 -- 172.21.15.7:0/140960683 shutdown_connections 2023-03-03T08:44:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a14416700 1 --2- 172.21.15.7:0/140960683 >> v2:172.21.15.143:3300/0 conn(0x7f5a0c151ab0 0x7f5a0c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a14416700 1 --2- 172.21.15.7:0/140960683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c06c3e0 0x7f5a0c157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a14416700 1 --2- 172.21.15.7:0/140960683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c073c30 0x7f5a0c06bea0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a14416700 1 -- 172.21.15.7:0/140960683 >> 172.21.15.7:0/140960683 conn(0x7f5a0c0a94d0 msgr2=0x7f5a0c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a14416700 1 -- 172.21.15.7:0/140960683 shutdown_connections 2023-03-03T08:44:32.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.381+0000 7f5a14416700 1 -- 172.21.15.7:0/140960683 wait complete. 2023-03-03T08:44:32.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.383+0000 7f5a14416700 1 Processor -- start 2023-03-03T08:44:32.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.383+0000 7f5a14416700 1 -- start start 2023-03-03T08:44:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a14416700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0c06c3e0 0x7f5a0c0c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a14416700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c073c30 0x7f5a0c0bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a14416700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c151ab0 0x7f5a0c0c0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a14416700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0c0a70 con 0x7f5a0c151ab0 2023-03-03T08:44:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a14416700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0be5d0 con 0x7f5a0c06c3e0 2023-03-03T08:44:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a14416700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0be8b0 con 0x7f5a0c073c30 2023-03-03T08:44:32.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a129b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c151ab0 0x7f5a0c0c0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a129b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c151ab0 0x7f5a0c0c0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52692/0 (socket says 172.21.15.7:52692) 2023-03-03T08:44:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a119b1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c073c30 0x7f5a0c0bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a129b3700 1 -- 172.21.15.7:0/3734460589 learned_addr learned my addr 172.21.15.7:0/3734460589 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a121b2700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0c06c3e0 0x7f5a0c0c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a129b3700 1 -- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c073c30 msgr2=0x7f5a0c0bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a129b3700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c073c30 0x7f5a0c0bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a129b3700 1 -- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0c06c3e0 msgr2=0x7f5a0c0c4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a129b3700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0c06c3e0 0x7f5a0c0c4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a129b3700 1 -- 172.21.15.7:0/3734460589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc009550 con 0x7f5a0c151ab0 2023-03-03T08:44:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.384+0000 7f5a129b3700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c151ab0 0x7f5a0c0c0530 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009fa0 tx=0x7f5a0400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.385+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3734460589 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0400ddb0 con 0x7f5a0c151ab0 2023-03-03T08:44:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.385+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3734460589 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0400f040 con 0x7f5a0c151ab0 2023-03-03T08:44:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.385+0000 7f5a14416700 1 -- 172.21.15.7:0/3734460589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0c151e80 con 0x7f5a0c151ab0 2023-03-03T08:44:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.385+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3734460589 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04003c60 con 0x7f5a0c151ab0 2023-03-03T08:44:32.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.385+0000 7f5a14416700 1 -- 172.21.15.7:0/3734460589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c06d410 con 0x7f5a0c151ab0 2023-03-03T08:44:32.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.386+0000 7f5a14416700 1 -- 172.21.15.7:0/3734460589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0c152010 con 0x7f5a0c151ab0 2023-03-03T08:44:32.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.387+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3734460589 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5a0401c030 con 0x7f5a0c151ab0 2023-03-03T08:44:32.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.387+0000 7f5a02ffd700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f00682f0 0x7f59f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:32.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.387+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3734460589 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6932+0+0 (secure 0 0 0) 0x7f5a04085ec0 con 0x7f5a0c151ab0 2023-03-03T08:44:32.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.387+0000 7f5a121b2700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f00682f0 0x7f59f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:32.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.388+0000 7f5a121b2700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f00682f0 0x7f59f006a7a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5a08009ac0 tx=0x7f5a08009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:32.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.392+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3734460589 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a040523b0 con 0x7f5a0c151ab0 2023-03-03T08:44:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.643+0000 7f5a14416700 1 -- 172.21.15.7:0/3734460589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a0c074bb0 con 0x7f5a0c151ab0 2023-03-03T08:44:32.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.643+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3734460589 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f5a040559d0 con 0x7f5a0c151ab0 2023-03-03T08:44:32.645 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:32.645 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:32.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 -- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f00682f0 msgr2=0x7f59f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:32.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f00682f0 0x7f59f006a7a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5a08009ac0 tx=0x7f5a08009210 comp rx=0 tx=0).stop 2023-03-03T08:44:32.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 -- 172.21.15.7:0/3734460589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c151ab0 msgr2=0x7f5a0c0c0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:32.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c151ab0 0x7f5a0c0c0530 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009fa0 tx=0x7f5a0400d3b0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 -- 172.21.15.7:0/3734460589 shutdown_connections 2023-03-03T08:44:32.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f00682f0 0x7f59f006a7a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a0c151ab0 0x7f5a0c0c0530 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0c06c3e0 0x7f5a0c0c4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 --2- 172.21.15.7:0/3734460589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0c073c30 0x7f5a0c0bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:32.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 -- 172.21.15.7:0/3734460589 >> 172.21.15.7:0/3734460589 conn(0x7f5a0c0a94d0 msgr2=0x7f5a0c156f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 -- 172.21.15.7:0/3734460589 shutdown_connections 2023-03-03T08:44:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:32.646+0000 7f5a14416700 1 -- 172.21.15.7:0/3734460589 wait complete. 2023-03-03T08:44:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:32.660 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-03-03T08:44:34.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.788+0000 7fb69a22e700 1 Processor -- start 2023-03-03T08:44:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.788+0000 7fb69a22e700 1 -- start start 2023-03-03T08:44:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.789+0000 7fb69a22e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb694071990 0x7fb694071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.789+0000 7fb69a22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb69414b460 0x7fb6940722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.789+0000 7fb69a22e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb694070360 con 0x7fb694151c00 2023-03-03T08:44:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.789+0000 7fb69a22e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6941521a0 con 0x7fb694071990 2023-03-03T08:44:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.789+0000 7fb69a22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69406a5e0 con 0x7fb69414b460 2023-03-03T08:44:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6937fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb694071990 0x7fb694071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb692ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb69414b460 0x7fb6940722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6937fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb694071990 0x7fb694071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:48470/0 (socket says 172.21.15.7:48470) 2023-03-03T08:44:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb693fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb694151c00 0x7fb69406aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50200/0 (socket says 172.21.15.7:50200) 2023-03-03T08:44:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6937fe700 1 -- 172.21.15.7:0/162605804 learned_addr learned my addr 172.21.15.7:0/162605804 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6937fe700 1 -- 172.21.15.7:0/162605804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb69414b460 msgr2=0x7fb6940722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6937fe700 1 --2- 172.21.15.7:0/162605804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb69414b460 0x7fb6940722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6937fe700 1 -- 172.21.15.7:0/162605804 >> v1:172.21.15.143:6789/0 conn(0x7fb694151c00 legacy=0x7fb69406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6937fe700 1 -- 172.21.15.7:0/162605804 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6940c38b0 con 0x7fb694071990 2023-03-03T08:44:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6927fc700 1 -- 172.21.15.7:0/162605804 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439483302 0 0) 0x7fb694070360 con 0x7fb694151c00 2023-03-03T08:44:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6937fe700 1 --2- 172.21.15.7:0/162605804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb694071990 0x7fb694071d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb6840098f0 tx=0x7fb684009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c860fee18e96e6b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6927fc700 1 -- 172.21.15.7:0/162605804 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684011830 con 0x7fb694071990 2023-03-03T08:44:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6927fc700 1 -- 172.21.15.7:0/162605804 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb684011990 con 0x7fb694071990 2023-03-03T08:44:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.790+0000 7fb6927fc700 1 -- 172.21.15.7:0/162605804 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684011d20 con 0x7fb694071990 2023-03-03T08:44:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.791+0000 7fb69a22e700 1 -- 172.21.15.7:0/162605804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb694071990 msgr2=0x7fb694071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.791+0000 7fb69a22e700 1 --2- 172.21.15.7:0/162605804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb694071990 0x7fb694071d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb6840098f0 tx=0x7fb684009c00 comp rx=0 tx=0).stop 2023-03-03T08:44:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.791+0000 7fb69a22e700 1 -- 172.21.15.7:0/162605804 shutdown_connections 2023-03-03T08:44:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.791+0000 7fb69a22e700 1 --2- 172.21.15.7:0/162605804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb694071990 0x7fb694071d60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.791+0000 7fb69a22e700 1 --2- 172.21.15.7:0/162605804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb69414b460 0x7fb6940722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.791+0000 7fb69a22e700 1 -- 172.21.15.7:0/162605804 >> 172.21.15.7:0/162605804 conn(0x7fb6940a94d0 msgr2=0x7fb6940a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.791+0000 7fb69a22e700 1 -- 172.21.15.7:0/162605804 shutdown_connections 2023-03-03T08:44:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.791+0000 7fb69a22e700 1 -- 172.21.15.7:0/162605804 wait complete. 2023-03-03T08:44:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.792+0000 7fb69a22e700 1 Processor -- start 2023-03-03T08:44:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb69a22e700 1 -- start start 2023-03-03T08:44:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb69a22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb694071990 0x7fb6940c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb69a22e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb694151c00 0x7fb69413f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb69a22e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb694141ee0 0x7fb69413fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb69a22e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6941ebfc0 con 0x7fb694151c00 2023-03-03T08:44:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb69a22e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6941ec2a0 con 0x7fb694141ee0 2023-03-03T08:44:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb69a22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6941537a0 con 0x7fb694071990 2023-03-03T08:44:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb692ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb694151c00 0x7fb69413f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb6937fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb694071990 0x7fb6940c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb692ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb694151c00 0x7fb69413f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52704/0 (socket says 172.21.15.7:52704) 2023-03-03T08:44:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb692ffd700 1 -- 172.21.15.7:0/2457098860 learned_addr learned my addr 172.21.15.7:0/2457098860 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.793+0000 7fb693fff700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb694141ee0 0x7fb69413fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb692ffd700 1 -- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb694071990 msgr2=0x7fb6940c3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb692ffd700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb694071990 0x7fb6940c3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb692ffd700 1 -- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb694141ee0 msgr2=0x7fb69413fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb692ffd700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb694141ee0 0x7fb69413fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb692ffd700 1 -- 172.21.15.7:0/2457098860 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb684009550 con 0x7fb694151c00 2023-03-03T08:44:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb692ffd700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb694151c00 0x7fb69413f680 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb68800e9b0 tx=0x7fb68800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb67bfff700 1 -- 172.21.15.7:0/2457098860 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb688009710 con 0x7fb694151c00 2023-03-03T08:44:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb67bfff700 1 -- 172.21.15.7:0/2457098860 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68800ee50 con 0x7fb694151c00 2023-03-03T08:44:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb67bfff700 1 -- 172.21.15.7:0/2457098860 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68800ca50 con 0x7fb694151c00 2023-03-03T08:44:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb69a22e700 1 -- 172.21.15.7:0/2457098860 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6940c3b50 con 0x7fb694151c00 2023-03-03T08:44:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.794+0000 7fb69a22e700 1 -- 172.21.15.7:0/2457098860 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69406a2b0 con 0x7fb694151c00 2023-03-03T08:44:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.795+0000 7fb69a22e700 1 -- 172.21.15.7:0/2457098860 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69406b180 con 0x7fb694151c00 2023-03-03T08:44:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.796+0000 7fb67bfff700 1 -- 172.21.15.7:0/2457098860 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb688019030 con 0x7fb694151c00 2023-03-03T08:44:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.796+0000 7fb67bfff700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6800682f0 0x7fb68006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.796+0000 7fb67bfff700 1 -- 172.21.15.7:0/2457098860 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6932+0+0 (secure 0 0 0) 0x7fb688085ea0 con 0x7fb694151c00 2023-03-03T08:44:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.796+0000 7fb6937fe700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6800682f0 0x7fb68006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.797+0000 7fb6937fe700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6800682f0 0x7fb68006a7a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb684000c00 tx=0x7fb684008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:34.804+0000 7fb67bfff700 1 -- 172.21.15.7:0/2457098860 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb688052390 con 0x7fb694151c00 2023-03-03T08:44:35.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.043+0000 7fb69a22e700 1 -- 172.21.15.7:0/2457098860 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb694074bb0 con 0x7fb694151c00 2023-03-03T08:44:35.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.043+0000 7fb67bfff700 1 -- 172.21.15.7:0/2457098860 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7fb6880559b0 con 0x7fb694151c00 2023-03-03T08:44:35.045 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:35.045 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 -- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6800682f0 msgr2=0x7fb68006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6800682f0 0x7fb68006a7a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb684000c00 tx=0x7fb684008030 comp rx=0 tx=0).stop 2023-03-03T08:44:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 -- 172.21.15.7:0/2457098860 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb694151c00 msgr2=0x7fb69413f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb694151c00 0x7fb69413f680 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb68800e9b0 tx=0x7fb68800ecc0 comp rx=0 tx=0).stop 2023-03-03T08:44:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 -- 172.21.15.7:0/2457098860 shutdown_connections 2023-03-03T08:44:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb694141ee0 0x7fb69413fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:35.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb694071990 0x7fb6940c3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:35.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6800682f0 0x7fb68006a7a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:35.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 --2- 172.21.15.7:0/2457098860 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb694151c00 0x7fb69413f680 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:35.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 -- 172.21.15.7:0/2457098860 >> 172.21.15.7:0/2457098860 conn(0x7fb6940a94d0 msgr2=0x7fb69406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:35.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.045+0000 7fb69a22e700 1 -- 172.21.15.7:0/2457098860 shutdown_connections 2023-03-03T08:44:35.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:35.046+0000 7fb69a22e700 1 -- 172.21.15.7:0/2457098860 wait complete. 2023-03-03T08:44:35.049 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:35.061 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-03-03T08:44:35.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:35.188 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:44:37.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:37.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.187+0000 7f2eb1ebd700 1 Processor -- start 2023-03-03T08:44:37.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.187+0000 7f2eb1ebd700 1 -- start start 2023-03-03T08:44:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.188+0000 7f2eb1ebd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2eac151de0 0x7f2eac0c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.188+0000 7f2eb1ebd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2eac0c5140 0x7f2eac06a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.188+0000 7f2eb1ebd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eac06dad0 con 0x7f2eac0c1a40 2023-03-03T08:44:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.188+0000 7f2eb1ebd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eac06e370 con 0x7f2eac0c5140 2023-03-03T08:44:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.188+0000 7f2eb1ebd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eac06ddb0 con 0x7f2eac151de0 2023-03-03T08:44:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.188+0000 7f2eabfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2eac0c5140 0x7f2eac06a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.188+0000 7f2eab7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2eac151de0 0x7f2eac0c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.188+0000 7f2eabfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2eac0c5140 0x7f2eac06a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52714/0 (socket says 172.21.15.7:52714) 2023-03-03T08:44:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.188+0000 7f2eabfff700 1 -- 172.21.15.7:0/567307726 learned_addr learned my addr 172.21.15.7:0/567307726 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:37.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.188+0000 7f2ea2ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f2eac0c1a40 0x7f2eac157ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50210/0 (socket says 172.21.15.7:50210) 2023-03-03T08:44:37.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.189+0000 7f2eabfff700 1 -- 172.21.15.7:0/567307726 >> v2:172.21.15.143:3300/0 conn(0x7f2eac151de0 msgr2=0x7f2eac0c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:37.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.189+0000 7f2eabfff700 1 --2- 172.21.15.7:0/567307726 >> v2:172.21.15.143:3300/0 conn(0x7f2eac151de0 0x7f2eac0c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.189+0000 7f2eabfff700 1 -- 172.21.15.7:0/567307726 >> v1:172.21.15.143:6789/0 conn(0x7f2eac0c1a40 legacy=0x7f2eac157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.189+0000 7f2eabfff700 1 -- 172.21.15.7:0/567307726 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eac06e090 con 0x7f2eac0c5140 2023-03-03T08:44:37.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.189+0000 7f2eab7fe700 1 --2- 172.21.15.7:0/567307726 >> v2:172.21.15.143:3300/0 conn(0x7f2eac151de0 0x7f2eac0c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:37.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.189+0000 7f2eabfff700 1 --2- 172.21.15.7:0/567307726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c5140 0x7f2eac06a2b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0c76a0 tx=0x7f2ea400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4038ca07a02705 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:37.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.189+0000 7f2eaaffd700 1 -- 172.21.15.7:0/567307726 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea400e070 con 0x7f2eac0c5140 2023-03-03T08:44:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.189+0000 7f2eaaffd700 1 -- 172.21.15.7:0/567307726 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea400b910 con 0x7f2eac0c5140 2023-03-03T08:44:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.189+0000 7f2eaaffd700 1 -- 172.21.15.7:0/567307726 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea400bcc0 con 0x7f2eac0c5140 2023-03-03T08:44:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.190+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/567307726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c5140 msgr2=0x7f2eac06a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.190+0000 7f2eb1ebd700 1 --2- 172.21.15.7:0/567307726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c5140 0x7f2eac06a2b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2eac0c76a0 tx=0x7f2ea400b330 comp rx=0 tx=0).stop 2023-03-03T08:44:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.190+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/567307726 shutdown_connections 2023-03-03T08:44:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.190+0000 7f2eb1ebd700 1 --2- 172.21.15.7:0/567307726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c5140 0x7f2eac06a2b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.190+0000 7f2eb1ebd700 1 --2- 172.21.15.7:0/567307726 >> v2:172.21.15.143:3300/0 conn(0x7f2eac151de0 0x7f2eac0c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.190+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/567307726 >> 172.21.15.7:0/567307726 conn(0x7f2eac0a9430 msgr2=0x7f2eac0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.190+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/567307726 shutdown_connections 2023-03-03T08:44:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.190+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/567307726 wait complete. 2023-03-03T08:44:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.191+0000 7f2eb1ebd700 1 Processor -- start 2023-03-03T08:44:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.191+0000 7f2eb1ebd700 1 -- start start 2023-03-03T08:44:37.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eb1ebd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c1a40 0x7f2eac1ed4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eb1ebd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac0c5140 0x7f2eac1eda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eb1ebd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eac151de0 0x7f2eac14ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eb1ebd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac1ea980 con 0x7f2eac0c1a40 2023-03-03T08:44:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eb1ebd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac0728f0 con 0x7f2eac151de0 2023-03-03T08:44:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eb1ebd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac1edf40 con 0x7f2eac0c5140 2023-03-03T08:44:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eab7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c1a40 0x7f2eac1ed4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eab7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c1a40 0x7f2eac1ed4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52716/0 (socket says 172.21.15.7:52716) 2023-03-03T08:44:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eab7fe700 1 -- 172.21.15.7:0/3241598907 learned_addr learned my addr 172.21.15.7:0/3241598907 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eabfff700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eac151de0 0x7f2eac14ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2ea2ffd700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac0c5140 0x7f2eac1eda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eab7fe700 1 -- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac0c5140 msgr2=0x7f2eac1eda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eab7fe700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac0c5140 0x7f2eac1eda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eab7fe700 1 -- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eac151de0 msgr2=0x7f2eac14ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eab7fe700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eac151de0 0x7f2eac14ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eab7fe700 1 -- 172.21.15.7:0/3241598907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9c0094a0 con 0x7f2eac0c1a40 2023-03-03T08:44:37.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2eab7fe700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c1a40 0x7f2eac1ed4c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00ec10 tx=0x7f2e9c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:37.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.192+0000 7f2ea8ff9700 1 -- 172.21.15.7:0/3241598907 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c00c850 con 0x7f2eac0c1a40 2023-03-03T08:44:37.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.193+0000 7f2ea8ff9700 1 -- 172.21.15.7:0/3241598907 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9c00c9b0 con 0x7f2eac0c1a40 2023-03-03T08:44:37.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.193+0000 7f2ea8ff9700 1 -- 172.21.15.7:0/3241598907 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c00cd20 con 0x7f2eac0c1a40 2023-03-03T08:44:37.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.193+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/3241598907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea4009b30 con 0x7f2eac0c1a40 2023-03-03T08:44:37.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.193+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/3241598907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eac066de0 con 0x7f2eac0c1a40 2023-03-03T08:44:37.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.193+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/3241598907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eac0c4c20 con 0x7f2eac0c1a40 2023-03-03T08:44:37.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.194+0000 7f2ea8ff9700 1 -- 172.21.15.7:0/3241598907 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2e9c019030 con 0x7f2eac0c1a40 2023-03-03T08:44:37.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.195+0000 7f2ea8ff9700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e8c0682f0 0x7f2e8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:37.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.195+0000 7f2ea8ff9700 1 -- 172.21.15.7:0/3241598907 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6932+0+0 (secure 0 0 0) 0x7f2e9c085ed0 con 0x7f2eac0c1a40 2023-03-03T08:44:37.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.195+0000 7f2ea2ffd700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e8c0682f0 0x7f2e8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:37.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.195+0000 7f2ea2ffd700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e8c0682f0 0x7f2e8c06a7a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2eac151150 tx=0x7f2e94007ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:37.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.199+0000 7f2ea8ff9700 1 -- 172.21.15.7:0/3241598907 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2e9c0523c0 con 0x7f2eac0c1a40 2023-03-03T08:44:37.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.440+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/3241598907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2eac1582b0 con 0x7f2eac0c1a40 2023-03-03T08:44:37.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.440+0000 7f2ea8ff9700 1 -- 172.21.15.7:0/3241598907 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f2e9c017090 con 0x7f2eac0c1a40 2023-03-03T08:44:37.442 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:37.442 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:43:47.991252+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4291},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4291":{"gid":4291,"name":"c","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.7:6834/1392328570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1392328570},{"type":"v1","addr":"172.21.15.7:6834","nonce":1392328570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:37.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.442+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e8c0682f0 msgr2=0x7f2e8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.442+0000 7f2eb1ebd700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e8c0682f0 0x7f2e8c06a7a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2eac151150 tx=0x7f2e94007ed0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.443+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/3241598907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c1a40 msgr2=0x7f2eac1ed4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:37.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.443+0000 7f2eb1ebd700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c1a40 0x7f2eac1ed4c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00ec10 tx=0x7f2e9c00ef20 comp rx=0 tx=0).stop 2023-03-03T08:44:37.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.443+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/3241598907 shutdown_connections 2023-03-03T08:44:37.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.443+0000 7f2eb1ebd700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eac151de0 0x7f2eac14ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.443+0000 7f2eb1ebd700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e8c0682f0 0x7f2e8c06a7a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.443+0000 7f2eb1ebd700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac0c1a40 0x7f2eac1ed4c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.443+0000 7f2eb1ebd700 1 --2- 172.21.15.7:0/3241598907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac0c5140 0x7f2eac1eda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:37.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.443+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/3241598907 >> 172.21.15.7:0/3241598907 conn(0x7f2eac0a9430 msgr2=0x7f2eac14ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:37.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.443+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/3241598907 shutdown_connections 2023-03-03T08:44:37.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:37.443+0000 7f2eb1ebd700 1 -- 172.21.15.7:0/3241598907 wait complete. 2023-03-03T08:44:37.446 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 14 2023-03-03T08:44:37.457 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-03-03T08:44:39.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:39.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.588+0000 7f4c5b846700 1 Processor -- start 2023-03-03T08:44:39.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.588+0000 7f4c5b846700 1 -- start start 2023-03-03T08:44:39.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.589+0000 7f4c5b846700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4c540c1da0 0x7f4c54156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.589+0000 7f4c5b846700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c54155840 con 0x7f4c5414fb00 2023-03-03T08:44:39.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.589+0000 7f4c5b846700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c541567d0 con 0x7f4c540740f0 2023-03-03T08:44:39.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.589+0000 7f4c5b846700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c54072940 con 0x7f4c540c1da0 2023-03-03T08:44:39.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.589+0000 7f4c595e2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4c5414fb00 0x7f4c540c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46708/0 (socket says 172.21.15.7:46708) 2023-03-03T08:44:39.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.589+0000 7f4c595e2700 1 -- 172.21.15.7:0/1801220292 learned_addr learned my addr 172.21.15.7:0/1801220292 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:44:39.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.589+0000 7f4c58de1700 1 --2- 172.21.15.7:0/1801220292 >> v2:172.21.15.143:3300/0 conn(0x7f4c540c1da0 0x7f4c54156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.590+0000 7f4c4bfff700 1 -- 172.21.15.7:0/1801220292 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605963731 0 0) 0x7f4c54155840 con 0x7f4c5414fb00 2023-03-03T08:44:39.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.590+0000 7f4c4bfff700 1 -- 172.21.15.7:0/1801220292 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c400035b0 con 0x7f4c5414fb00 2023-03-03T08:44:39.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.590+0000 7f4c58de1700 1 -- 172.21.15.7:0/1801220292 >> v1:172.21.15.143:6789/0 conn(0x7f4c540740f0 legacy=0x7f4c5415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.590+0000 7f4c58de1700 1 -- 172.21.15.7:0/1801220292 >> v1:172.21.15.7:6789/0 conn(0x7f4c5414fb00 legacy=0x7f4c540c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.590+0000 7f4c58de1700 1 -- 172.21.15.7:0/1801220292 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540c2330 con 0x7f4c540c1da0 2023-03-03T08:44:39.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.590+0000 7f4c58de1700 1 --2- 172.21.15.7:0/1801220292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c540c1da0 0x7f4c54156290 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009840 tx=0x7f4c44009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59b2f24bffb85093 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:39.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.590+0000 7f4c4bfff700 1 -- 172.21.15.7:0/1801220292 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44011830 con 0x7f4c540c1da0 2023-03-03T08:44:39.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.590+0000 7f4c4bfff700 1 -- 172.21.15.7:0/1801220292 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44011990 con 0x7f4c540c1da0 2023-03-03T08:44:39.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.590+0000 7f4c4bfff700 1 -- 172.21.15.7:0/1801220292 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44011d20 con 0x7f4c540c1da0 2023-03-03T08:44:39.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.591+0000 7f4c5b846700 1 -- 172.21.15.7:0/1801220292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c540c1da0 msgr2=0x7f4c54156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.591+0000 7f4c5b846700 1 --2- 172.21.15.7:0/1801220292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c540c1da0 0x7f4c54156290 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009840 tx=0x7f4c44009b50 comp rx=0 tx=0).stop 2023-03-03T08:44:39.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.591+0000 7f4c5b846700 1 -- 172.21.15.7:0/1801220292 shutdown_connections 2023-03-03T08:44:39.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.591+0000 7f4c5b846700 1 --2- 172.21.15.7:0/1801220292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c540c1da0 0x7f4c54156290 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.591+0000 7f4c5b846700 1 -- 172.21.15.7:0/1801220292 >> 172.21.15.7:0/1801220292 conn(0x7f4c5406c110 msgr2=0x7f4c5406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:39.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.591+0000 7f4c5b846700 1 -- 172.21.15.7:0/1801220292 shutdown_connections 2023-03-03T08:44:39.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.591+0000 7f4c5b846700 1 -- 172.21.15.7:0/1801220292 wait complete. 2023-03-03T08:44:39.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.592+0000 7f4c5b846700 1 Processor -- start 2023-03-03T08:44:39.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.592+0000 7f4c5b846700 1 -- start start 2023-03-03T08:44:39.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c5b846700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c540740f0 0x7f4c54153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c5b846700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c540c1da0 0x7f4c5414cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c5b846700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c5414fb00 0x7f4c5414d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c5b846700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540c14e0 con 0x7f4c540c1da0 2023-03-03T08:44:39.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c5b846700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541df260 con 0x7f4c540740f0 2023-03-03T08:44:39.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c5b846700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541ea4d0 con 0x7f4c5414fb00 2023-03-03T08:44:39.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c58de1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c540c1da0 0x7f4c5414cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c58de1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c540c1da0 0x7f4c5414cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52736/0 (socket says 172.21.15.7:52736) 2023-03-03T08:44:39.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c58de1700 1 -- 172.21.15.7:0/1127082335 learned_addr learned my addr 172.21.15.7:0/1127082335 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:39.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c595e2700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c540740f0 0x7f4c54153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c59de3700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c5414fb00 0x7f4c5414d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c58de1700 1 -- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c5414fb00 msgr2=0x7f4c5414d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.593+0000 7f4c58de1700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c5414fb00 0x7f4c5414d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.594+0000 7f4c58de1700 1 -- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c540740f0 msgr2=0x7f4c54153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.594+0000 7f4c58de1700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c540740f0 0x7f4c54153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.594+0000 7f4c58de1700 1 -- 172.21.15.7:0/1127082335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c440094a0 con 0x7f4c540c1da0 2023-03-03T08:44:39.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.594+0000 7f4c58de1700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c540c1da0 0x7f4c5414cd80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009b90 tx=0x7f4c44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:39.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.594+0000 7f4c49ffb700 1 -- 172.21.15.7:0/1127082335 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c440036a0 con 0x7f4c540c1da0 2023-03-03T08:44:39.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.594+0000 7f4c49ffb700 1 -- 172.21.15.7:0/1127082335 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4401e070 con 0x7f4c540c1da0 2023-03-03T08:44:39.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.594+0000 7f4c49ffb700 1 -- 172.21.15.7:0/1127082335 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44003a60 con 0x7f4c540c1da0 2023-03-03T08:44:39.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.594+0000 7f4c5b846700 1 -- 172.21.15.7:0/1127082335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c54153090 con 0x7f4c540c1da0 2023-03-03T08:44:39.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.594+0000 7f4c5b846700 1 -- 172.21.15.7:0/1127082335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c54152db0 con 0x7f4c540c1da0 2023-03-03T08:44:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.596+0000 7f4c49ffb700 1 -- 172.21.15.7:0/1127082335 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4c4400f050 con 0x7f4c540c1da0 2023-03-03T08:44:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.597+0000 7f4c49ffb700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c380682f0 0x7f4c3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.597+0000 7f4c49ffb700 1 -- 172.21.15.7:0/1127082335 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7018+0+0 (secure 0 0 0) 0x7f4c44086e70 con 0x7f4c540c1da0 2023-03-03T08:44:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.597+0000 7f4c5b846700 1 -- 172.21.15.7:0/1127082335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c54155a90 con 0x7f4c540c1da0 2023-03-03T08:44:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.597+0000 7f4c595e2700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c380682f0 0x7f4c3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.597+0000 7f4c595e2700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c380682f0 0x7f4c3806a7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4c50007fd0 tx=0x7f4c50007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.602+0000 7f4c49ffb700 1 -- 172.21.15.7:0/1127082335 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c44020080 con 0x7f4c540c1da0 2023-03-03T08:44:39.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.840+0000 7f4c5b846700 1 -- 172.21.15.7:0/1127082335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c5406e410 con 0x7f4c540c1da0 2023-03-03T08:44:39.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.840+0000 7f4c49ffb700 1 -- 172.21.15.7:0/1127082335 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5005 (secure 0 0 0) 0x7f4c44015030 con 0x7f4c540c1da0 2023-03-03T08:44:39.842 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:39.842 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:39.287875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:39.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.842+0000 7f4c5b846700 1 -- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c380682f0 msgr2=0x7f4c3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.842+0000 7f4c5b846700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c380682f0 0x7f4c3806a7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4c50007fd0 tx=0x7f4c50007dc0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.842+0000 7f4c5b846700 1 -- 172.21.15.7:0/1127082335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c540c1da0 msgr2=0x7f4c5414cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.842+0000 7f4c5b846700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c540c1da0 0x7f4c5414cd80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009b90 tx=0x7f4c44004750 comp rx=0 tx=0).stop 2023-03-03T08:44:39.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.842+0000 7f4c5b846700 1 -- 172.21.15.7:0/1127082335 shutdown_connections 2023-03-03T08:44:39.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.842+0000 7f4c5b846700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c5414fb00 0x7f4c5414d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.842+0000 7f4c5b846700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c540740f0 0x7f4c54153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.842+0000 7f4c5b846700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c380682f0 0x7f4c3806a7a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.842+0000 7f4c5b846700 1 --2- 172.21.15.7:0/1127082335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c540c1da0 0x7f4c5414cd80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.842+0000 7f4c5b846700 1 -- 172.21.15.7:0/1127082335 >> 172.21.15.7:0/1127082335 conn(0x7f4c5406c110 msgr2=0x7f4c540ac3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:39.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.843+0000 7f4c5b846700 1 -- 172.21.15.7:0/1127082335 shutdown_connections 2023-03-03T08:44:39.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.843+0000 7f4c5b846700 1 -- 172.21.15.7:0/1127082335 wait complete. 2023-03-03T08:44:39.846 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 15 2023-03-03T08:44:39.858 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2023-03-03T08:44:39.858 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-03-03T08:44:39.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:39.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.983+0000 7f186d19e700 1 Processor -- start 2023-03-03T08:44:39.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.983+0000 7f186d19e700 1 -- start start 2023-03-03T08:44:39.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.984+0000 7f186d19e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f186806d870 0x7f18681550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.984+0000 7f186d19e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18680c3e30 0x7f18680c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.984+0000 7f186d19e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18680c7a60 con 0x7f18680c1da0 2023-03-03T08:44:39.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.984+0000 7f186d19e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18680c2af0 con 0x7f18680c3e30 2023-03-03T08:44:39.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.984+0000 7f186d19e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18680c3620 con 0x7f186806d870 2023-03-03T08:44:39.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.984+0000 7f186759e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18680c3e30 0x7f18680c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.984+0000 7f186759e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18680c3e30 0x7f18680c4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52746/0 (socket says 172.21.15.7:52746) 2023-03-03T08:44:39.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.985+0000 7f186759e700 1 -- 172.21.15.7:0/1500268844 learned_addr learned my addr 172.21.15.7:0/1500268844 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:39.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.985+0000 7f1866d9d700 1 --2- 172.21.15.7:0/1500268844 >> v2:172.21.15.143:3300/0 conn(0x7f186806d870 0x7f18681550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.985+0000 7f186759e700 1 -- 172.21.15.7:0/1500268844 >> v2:172.21.15.143:3300/0 conn(0x7f186806d870 msgr2=0x7f18681550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:39.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.985+0000 7f186759e700 1 --2- 172.21.15.7:0/1500268844 >> v2:172.21.15.143:3300/0 conn(0x7f186806d870 0x7f18681550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.985+0000 7f186759e700 1 -- 172.21.15.7:0/1500268844 >> v1:172.21.15.143:6789/0 conn(0x7f18680c1da0 legacy=0x7f18680c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.985+0000 7f186759e700 1 -- 172.21.15.7:0/1500268844 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18681545e0 con 0x7f18680c3e30 2023-03-03T08:44:39.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.985+0000 7f186759e700 1 --2- 172.21.15.7:0/1500268844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c3e30 0x7f18680c4230 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1868073880 tx=0x7f186000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9275d7aaa66608 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:39.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.985+0000 7f1865d9b700 1 -- 172.21.15.7:0/1500268844 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186000e070 con 0x7f18680c3e30 2023-03-03T08:44:39.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.985+0000 7f1865d9b700 1 -- 172.21.15.7:0/1500268844 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186000b910 con 0x7f18680c3e30 2023-03-03T08:44:39.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.985+0000 7f1865d9b700 1 -- 172.21.15.7:0/1500268844 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186000bcc0 con 0x7f18680c3e30 2023-03-03T08:44:39.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.986+0000 7f186d19e700 1 -- 172.21.15.7:0/1500268844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c3e30 msgr2=0x7f18680c4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.986+0000 7f186d19e700 1 --2- 172.21.15.7:0/1500268844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c3e30 0x7f18680c4230 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1868073880 tx=0x7f186000b330 comp rx=0 tx=0).stop 2023-03-03T08:44:39.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.986+0000 7f186d19e700 1 -- 172.21.15.7:0/1500268844 shutdown_connections 2023-03-03T08:44:39.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.986+0000 7f186d19e700 1 --2- 172.21.15.7:0/1500268844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c3e30 0x7f18680c4230 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.986+0000 7f186d19e700 1 --2- 172.21.15.7:0/1500268844 >> v2:172.21.15.143:3300/0 conn(0x7f186806d870 0x7f18681550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.986+0000 7f186d19e700 1 -- 172.21.15.7:0/1500268844 >> 172.21.15.7:0/1500268844 conn(0x7f186806c110 msgr2=0x7f186806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:39.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.986+0000 7f186d19e700 1 -- 172.21.15.7:0/1500268844 shutdown_connections 2023-03-03T08:44:39.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.986+0000 7f186d19e700 1 -- 172.21.15.7:0/1500268844 wait complete. 2023-03-03T08:44:39.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.988+0000 7f186d19e700 1 Processor -- start 2023-03-03T08:44:39.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.988+0000 7f186d19e700 1 -- start start 2023-03-03T08:44:39.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.989+0000 7f186d19e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186806d870 0x7f18681ea340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.989+0000 7f186d19e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c1da0 0x7f1868157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.989+0000 7f186d19e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18680c3e30 0x7f186814eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.989+0000 7f186d19e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18681569d0 con 0x7f18680c1da0 2023-03-03T08:44:39.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.989+0000 7f186d19e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1868155dc0 con 0x7f186806d870 2023-03-03T08:44:39.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.989+0000 7f186d19e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18681560a0 con 0x7f18680c3e30 2023-03-03T08:44:39.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c1da0 0x7f1868157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c1da0 0x7f1868157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52750/0 (socket says 172.21.15.7:52750) 2023-03-03T08:44:39.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186659c700 1 -- 172.21.15.7:0/3589122789 learned_addr learned my addr 172.21.15.7:0/3589122789 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:39.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f1866d9d700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186806d870 0x7f18681ea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186759e700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18680c3e30 0x7f186814eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186659c700 1 -- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18680c3e30 msgr2=0x7f186814eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186659c700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18680c3e30 0x7f186814eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186659c700 1 -- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186806d870 msgr2=0x7f18681ea340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:39.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186659c700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186806d870 0x7f18681ea340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:39.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186659c700 1 -- 172.21.15.7:0/3589122789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1860009b30 con 0x7f18680c1da0 2023-03-03T08:44:39.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f1866d9d700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186806d870 0x7f18681ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:39.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186659c700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c1da0 0x7f1868157f60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f185c00c4e0 tx=0x7f185c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:39.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f184f7fe700 1 -- 172.21.15.7:0/3589122789 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0052d0 con 0x7f18680c1da0 2023-03-03T08:44:39.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f184f7fe700 1 -- 172.21.15.7:0/3589122789 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c005430 con 0x7f18680c1da0 2023-03-03T08:44:39.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186d19e700 1 -- 172.21.15.7:0/3589122789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1868154ba0 con 0x7f18680c1da0 2023-03-03T08:44:39.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f184f7fe700 1 -- 172.21.15.7:0/3589122789 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0038c0 con 0x7f18680c1da0 2023-03-03T08:44:39.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.990+0000 7f186d19e700 1 -- 172.21.15.7:0/3589122789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18681548c0 con 0x7f18680c1da0 2023-03-03T08:44:39.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.992+0000 7f186d19e700 1 -- 172.21.15.7:0/3589122789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186806dc40 con 0x7f18680c1da0 2023-03-03T08:44:39.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.992+0000 7f184f7fe700 1 -- 172.21.15.7:0/3589122789 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f185c019030 con 0x7f18680c1da0 2023-03-03T08:44:39.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.993+0000 7f184f7fe700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1854068320 0x7f185406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:39.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.993+0000 7f184f7fe700 1 -- 172.21.15.7:0/3589122789 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7018+0+0 (secure 0 0 0) 0x7f185c0852b0 con 0x7f18680c1da0 2023-03-03T08:44:39.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.993+0000 7f1866d9d700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1854068320 0x7f185406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:39.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.994+0000 7f1866d9d700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1854068320 0x7f185406a7d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1858009fd0 tx=0x7f1858009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:39.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:39.998+0000 7f184f7fe700 1 -- 172.21.15.7:0/3589122789 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f185c0516f0 con 0x7f18680c1da0 2023-03-03T08:44:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.253+0000 7f186d19e700 1 -- 172.21.15.7:0/3589122789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1868154400 con 0x7f18680c1da0 2023-03-03T08:44:40.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.254+0000 7f184f7fe700 1 -- 172.21.15.7:0/3589122789 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5005 (secure 0 0 0) 0x7f185c054d10 con 0x7f18680c1da0 2023-03-03T08:44:40.256 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:40.256 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:39.287875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:40.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 -- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1854068320 msgr2=0x7f185406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:40.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1854068320 0x7f185406a7d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1858009fd0 tx=0x7f1858009210 comp rx=0 tx=0).stop 2023-03-03T08:44:40.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 -- 172.21.15.7:0/3589122789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c1da0 msgr2=0x7f1868157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:40.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c1da0 0x7f1868157f60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f185c00c4e0 tx=0x7f185c00c7f0 comp rx=0 tx=0).stop 2023-03-03T08:44:40.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 -- 172.21.15.7:0/3589122789 shutdown_connections 2023-03-03T08:44:40.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18680c3e30 0x7f186814eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:40.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f186806d870 0x7f18681ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:40.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1854068320 0x7f185406a7d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:40.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 --2- 172.21.15.7:0/3589122789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18680c1da0 0x7f1868157f60 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:40.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 -- 172.21.15.7:0/3589122789 >> 172.21.15.7:0/3589122789 conn(0x7f186806c110 msgr2=0x7f186814abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:40.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 -- 172.21.15.7:0/3589122789 shutdown_connections 2023-03-03T08:44:40.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:40.256+0000 7f186d19e700 1 -- 172.21.15.7:0/3589122789 wait complete. 2023-03-03T08:44:40.259 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 15 2023-03-03T08:44:40.270 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-03-03T08:43:34.516962+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4296": { "addr": "172.21.15.7:6835/1083851601", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6833", "nonce": 1083851601, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 1083851601, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4296, "incarnation": 4, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-03-03T08:43:38.528223+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4296 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-03-03T08:43:43.971492+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4247": { "addr": "172.21.15.143:6837/4239346710", "addrs": { "addrvec": [ { "addr": "172.21.15.143:6836", "nonce": 4239346710, "type": "v2" }, { "addr": "172.21.15.143:6837", "nonce": 4239346710, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4247, "incarnation": 15, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-03-03T08:44:39.287875+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4247 } } } ], "standbys": [ { "addr": "172.21.15.143:6835/2988899186", "addrs": { "addrvec": [ { "addr": "172.21.15.143:6834", "nonce": 2988899186, "type": "v2" }, { "addr": "172.21.15.143:6835", "nonce": 2988899186, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-03-03T08:44:42.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:42.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.425+0000 7f7b5c274700 1 Processor -- start 2023-03-03T08:44:42.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.425+0000 7f7b5c274700 1 -- start start 2023-03-03T08:44:42.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.426+0000 7f7b5c274700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7b5414de50 0x7f7b5406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:42.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.426+0000 7f7b5c274700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b540c65c0 0x7f7b540c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:42.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.426+0000 7f7b5c274700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b540a65b0 con 0x7f7b5406a7f0 2023-03-03T08:44:42.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.426+0000 7f7b5c274700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b54154e00 con 0x7f7b540c65c0 2023-03-03T08:44:42.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.426+0000 7f7b5c274700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b540c3370 con 0x7f7b5414de50 2023-03-03T08:44:42.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.426+0000 7f7b5980f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b540c65c0 0x7f7b540c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:42.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.426+0000 7f7b5980f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b540c65c0 0x7f7b540c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52758/0 (socket says 172.21.15.7:52758) 2023-03-03T08:44:42.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.426+0000 7f7b5a010700 1 --2- 172.21.15.7:0/1730505711 >> v2:172.21.15.143:3300/0 conn(0x7f7b5414de50 0x7f7b5406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:42.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.426+0000 7f7b5980f700 1 -- 172.21.15.7:0/1730505711 learned_addr learned my addr 172.21.15.7:0/1730505711 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:42.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.427+0000 7f7b5a010700 1 -- 172.21.15.7:0/1730505711 >> v2:172.21.15.7:3300/0 conn(0x7f7b540c65c0 msgr2=0x7f7b540c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:42.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.427+0000 7f7b5a010700 1 --2- 172.21.15.7:0/1730505711 >> v2:172.21.15.7:3300/0 conn(0x7f7b540c65c0 0x7f7b540c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:42.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.427+0000 7f7b5a010700 1 -- 172.21.15.7:0/1730505711 >> v1:172.21.15.143:6789/0 conn(0x7f7b5406a7f0 legacy=0x7f7b540c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:42.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.427+0000 7f7b5a010700 1 -- 172.21.15.7:0/1730505711 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b541541f0 con 0x7f7b5414de50 2023-03-03T08:44:42.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.427+0000 7f7b5980f700 1 --2- 172.21.15.7:0/1730505711 >> v2:172.21.15.7:3300/0 conn(0x7f7b540c65c0 0x7f7b540c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:42.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.427+0000 7f7b5a010700 1 --2- 172.21.15.7:0/1730505711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5414de50 0x7f7b5406a2b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7b540c7760 tx=0x7f7b50009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a6c90a9d16c0e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:42.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.427+0000 7f7b5900e700 1 -- 172.21.15.7:0/1730505711 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50004230 con 0x7f7b5414de50 2023-03-03T08:44:42.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.427+0000 7f7b5900e700 1 -- 172.21.15.7:0/1730505711 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b50004390 con 0x7f7b5414de50 2023-03-03T08:44:42.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.427+0000 7f7b5900e700 1 -- 172.21.15.7:0/1730505711 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b500036a0 con 0x7f7b5414de50 2023-03-03T08:44:42.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.428+0000 7f7b5c274700 1 -- 172.21.15.7:0/1730505711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5414de50 msgr2=0x7f7b5406a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:42.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.428+0000 7f7b5c274700 1 --2- 172.21.15.7:0/1730505711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5414de50 0x7f7b5406a2b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7b540c7760 tx=0x7f7b50009b20 comp rx=0 tx=0).stop 2023-03-03T08:44:42.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.428+0000 7f7b5c274700 1 -- 172.21.15.7:0/1730505711 shutdown_connections 2023-03-03T08:44:42.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.428+0000 7f7b5c274700 1 --2- 172.21.15.7:0/1730505711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5414de50 0x7f7b5406a2b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:42.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.428+0000 7f7b5c274700 1 --2- 172.21.15.7:0/1730505711 >> v2:172.21.15.7:3300/0 conn(0x7f7b540c65c0 0x7f7b540c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:42.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.428+0000 7f7b5c274700 1 -- 172.21.15.7:0/1730505711 >> 172.21.15.7:0/1730505711 conn(0x7f7b540a9430 msgr2=0x7f7b540c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:42.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.428+0000 7f7b5c274700 1 -- 172.21.15.7:0/1730505711 shutdown_connections 2023-03-03T08:44:42.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.428+0000 7f7b5c274700 1 -- 172.21.15.7:0/1730505711 wait complete. 2023-03-03T08:44:42.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.430+0000 7f7b5c274700 1 Processor -- start 2023-03-03T08:44:42.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.430+0000 7f7b5c274700 1 -- start start 2023-03-03T08:44:42.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5c274700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5406a7f0 0x7f7b5406b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:42.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5c274700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b540c65c0 0x7f7b540c0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:42.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5c274700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5414de50 0x7f7b541ed2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:42.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5c274700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b540c0bd0 con 0x7f7b540c65c0 2023-03-03T08:44:42.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5c274700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54070f70 con 0x7f7b5406a7f0 2023-03-03T08:44:42.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5c274700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54071250 con 0x7f7b5414de50 2023-03-03T08:44:42.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5a010700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5406a7f0 0x7f7b5406b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:42.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5a010700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5406a7f0 0x7f7b5406b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:48536/0 (socket says 172.21.15.7:48536) 2023-03-03T08:44:42.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5a010700 1 -- 172.21.15.7:0/1737935778 learned_addr learned my addr 172.21.15.7:0/1737935778 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:42.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5980f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b540c65c0 0x7f7b540c0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:42.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.431+0000 7f7b5a811700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5414de50 0x7f7b541ed2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:42.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b5a010700 1 -- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5414de50 msgr2=0x7f7b541ed2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:42.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b5a010700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5414de50 0x7f7b541ed2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:42.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b5a010700 1 -- 172.21.15.7:0/1737935778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b540c65c0 msgr2=0x7f7b540c0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:42.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b5a010700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b540c65c0 0x7f7b540c0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:42.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b5a010700 1 -- 172.21.15.7:0/1737935778 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4400b7f0 con 0x7f7b5406a7f0 2023-03-03T08:44:42.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b5980f700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b540c65c0 0x7f7b540c0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:42.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b5a010700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5406a7f0 0x7f7b5406b680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009e70 tx=0x7f7b50004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:42.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b4affd700 1 -- 172.21.15.7:0/1737935778 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b500105b0 con 0x7f7b5406a7f0 2023-03-03T08:44:42.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b4affd700 1 -- 172.21.15.7:0/1737935778 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b50010710 con 0x7f7b5406a7f0 2023-03-03T08:44:42.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b4affd700 1 -- 172.21.15.7:0/1737935778 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50010ac0 con 0x7f7b5406a7f0 2023-03-03T08:44:42.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b5c274700 1 -- 172.21.15.7:0/1737935778 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b500094a0 con 0x7f7b5406a7f0 2023-03-03T08:44:42.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.432+0000 7f7b5c274700 1 -- 172.21.15.7:0/1737935778 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b541547b0 con 0x7f7b5406a7f0 2023-03-03T08:44:42.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.433+0000 7f7b5c274700 1 -- 172.21.15.7:0/1737935778 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b541544d0 con 0x7f7b5406a7f0 2023-03-03T08:44:42.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.434+0000 7f7b4affd700 1 -- 172.21.15.7:0/1737935778 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7b5002c070 con 0x7f7b5406a7f0 2023-03-03T08:44:42.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.434+0000 7f7b4affd700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b380682f0 0x7f7b3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:42.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.434+0000 7f7b4affd700 1 -- 172.21.15.7:0/1737935778 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7018+0+0 (secure 0 0 0) 0x7f7b50087130 con 0x7f7b5406a7f0 2023-03-03T08:44:42.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.434+0000 7f7b5980f700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b380682f0 0x7f7b3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:42.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.435+0000 7f7b5980f700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b380682f0 0x7f7b3806a7a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400bfd0 tx=0x7f7b4400b4d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:42.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.439+0000 7f7b4affd700 1 -- 172.21.15.7:0/1737935778 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7b50053690 con 0x7f7b5406a7f0 2023-03-03T08:44:42.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.789+0000 7f7b5c274700 1 -- 172.21.15.7:0/1737935778 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b54154a90 con 0x7f7b5406a7f0 2023-03-03T08:44:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.790+0000 7f7b4affd700 1 -- 172.21.15.7:0/1737935778 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5006 (secure 0 0 0) 0x7f7b5002a090 con 0x7f7b5406a7f0 2023-03-03T08:44:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:42.792 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:42.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 -- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b380682f0 msgr2=0x7f7b3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:42.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b380682f0 0x7f7b3806a7a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400bfd0 tx=0x7f7b4400b4d0 comp rx=0 tx=0).stop 2023-03-03T08:44:42.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 -- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5406a7f0 msgr2=0x7f7b5406b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:42.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5406a7f0 0x7f7b5406b680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009e70 tx=0x7f7b50004750 comp rx=0 tx=0).stop 2023-03-03T08:44:42.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 -- 172.21.15.7:0/1737935778 shutdown_connections 2023-03-03T08:44:42.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b5414de50 0x7f7b541ed2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:42.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b5406a7f0 0x7f7b5406b680 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:42.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b380682f0 0x7f7b3806a7a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:42.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 --2- 172.21.15.7:0/1737935778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b540c65c0 0x7f7b540c0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:42.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 -- 172.21.15.7:0/1737935778 >> 172.21.15.7:0/1737935778 conn(0x7f7b540a9430 msgr2=0x7f7b5414d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:42.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 -- 172.21.15.7:0/1737935778 shutdown_connections 2023-03-03T08:44:42.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:42.793+0000 7f7b5c274700 1 -- 172.21.15.7:0/1737935778 wait complete. 2023-03-03T08:44:42.796 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 18 2023-03-03T08:44:42.806 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=4291 2023-03-03T08:44:42.807 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.c 2023-03-03T08:44:44.808 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2023-03-03T08:44:44.809 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-03T08:44:44.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-03T08:44:44.811 INFO:tasks.ceph.mds.c:Started 2023-03-03T08:44:44.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:44.905 INFO:tasks.ceph.mds.c.smithi007.stdout:starting mds.c at 2023-03-03T08:44:44.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.987+0000 7fc7a9670700 1 Processor -- start 2023-03-03T08:44:44.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.987+0000 7fc7a9670700 1 -- start start 2023-03-03T08:44:44.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.988+0000 7fc7a9670700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc7a414de50 0x7fc7a406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:44.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.988+0000 7fc7a9670700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7a406a7f0 0x7fc7a40c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:44.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.988+0000 7fc7a9670700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a40a65b0 con 0x7fc7a40c65c0 2023-03-03T08:44:44.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.988+0000 7fc7a9670700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a4154e00 con 0x7fc7a406a7f0 2023-03-03T08:44:44.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.988+0000 7fc7a9670700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a40c3370 con 0x7fc7a414de50 2023-03-03T08:44:44.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.988+0000 7fc7a37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7a406a7f0 0x7fc7a40c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:44.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.988+0000 7fc7a37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7a406a7f0 0x7fc7a40c2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52784/0 (socket says 172.21.15.7:52784) 2023-03-03T08:44:44.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.988+0000 7fc7a37fe700 1 -- 172.21.15.7:0/606555409 learned_addr learned my addr 172.21.15.7:0/606555409 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:44.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.988+0000 7fc7a2ffd700 1 --2- 172.21.15.7:0/606555409 >> v2:172.21.15.143:3300/0 conn(0x7fc7a414de50 0x7fc7a406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:44.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.989+0000 7fc7a37fe700 1 -- 172.21.15.7:0/606555409 >> v2:172.21.15.143:3300/0 conn(0x7fc7a414de50 msgr2=0x7fc7a406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:44.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.989+0000 7fc7a37fe700 1 --2- 172.21.15.7:0/606555409 >> v2:172.21.15.143:3300/0 conn(0x7fc7a414de50 0x7fc7a406a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:44.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.989+0000 7fc7a37fe700 1 -- 172.21.15.7:0/606555409 >> v1:172.21.15.143:6789/0 conn(0x7fc7a40c65c0 legacy=0x7fc7a40c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:44.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.989+0000 7fc7a37fe700 1 -- 172.21.15.7:0/606555409 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a41541f0 con 0x7fc7a406a7f0 2023-03-03T08:44:44.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.989+0000 7fc7a37fe700 1 --2- 172.21.15.7:0/606555409 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 0x7fc7a40c2e30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40c7760 tx=0x7fc79c00d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=411935aab7c10c28 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:44.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.989+0000 7fc7a1ffb700 1 -- 172.21.15.7:0/606555409 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c010070 con 0x7fc7a406a7f0 2023-03-03T08:44:44.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.989+0000 7fc7a1ffb700 1 -- 172.21.15.7:0/606555409 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c00dac0 con 0x7fc7a406a7f0 2023-03-03T08:44:44.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.989+0000 7fc7a1ffb700 1 -- 172.21.15.7:0/606555409 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c00de70 con 0x7fc7a406a7f0 2023-03-03T08:44:44.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.990+0000 7fc7a9670700 1 -- 172.21.15.7:0/606555409 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 msgr2=0x7fc7a40c2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:44.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.990+0000 7fc7a9670700 1 --2- 172.21.15.7:0/606555409 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 0x7fc7a40c2e30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40c7760 tx=0x7fc79c00d460 comp rx=0 tx=0).stop 2023-03-03T08:44:44.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.990+0000 7fc7a9670700 1 -- 172.21.15.7:0/606555409 shutdown_connections 2023-03-03T08:44:44.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.990+0000 7fc7a9670700 1 --2- 172.21.15.7:0/606555409 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 0x7fc7a40c2e30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:44.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.990+0000 7fc7a9670700 1 --2- 172.21.15.7:0/606555409 >> v2:172.21.15.143:3300/0 conn(0x7fc7a414de50 0x7fc7a406a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:44.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.990+0000 7fc7a9670700 1 -- 172.21.15.7:0/606555409 >> 172.21.15.7:0/606555409 conn(0x7fc7a40a9430 msgr2=0x7fc7a40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:44.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.990+0000 7fc7a9670700 1 -- 172.21.15.7:0/606555409 shutdown_connections 2023-03-03T08:44:44.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.990+0000 7fc7a9670700 1 -- 172.21.15.7:0/606555409 wait complete. 2023-03-03T08:44:44.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.991+0000 7fc7a9670700 1 Processor -- start 2023-03-03T08:44:44.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.991+0000 7fc7a9670700 1 -- start start 2023-03-03T08:44:44.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a9670700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 0x7fc7a4153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:44.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a9670700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7a40c65c0 0x7fc7a40a7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:44.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a9670700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7a414de50 0x7fc7a4071240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:44.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a9670700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a41e5f80 con 0x7fc7a406a7f0 2023-03-03T08:44:44.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a9670700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a41e6260 con 0x7fc7a414de50 2023-03-03T08:44:44.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a9670700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a41e6540 con 0x7fc7a40c65c0 2023-03-03T08:44:44.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 0x7fc7a4153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:44.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 0x7fc7a4153360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52786/0 (socket says 172.21.15.7:52786) 2023-03-03T08:44:44.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a2ffd700 1 -- 172.21.15.7:0/1579204743 learned_addr learned my addr 172.21.15.7:0/1579204743 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:44.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a27fc700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7a40c65c0 0x7fc7a40a7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:44.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a37fe700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7a414de50 0x7fc7a4071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:44.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a2ffd700 1 -- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7a40c65c0 msgr2=0x7fc7a40a7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:44.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a2ffd700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7a40c65c0 0x7fc7a40a7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:44.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a2ffd700 1 -- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7a414de50 msgr2=0x7fc7a4071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:44.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a2ffd700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7a414de50 0x7fc7a4071240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:44.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a2ffd700 1 -- 172.21.15.7:0/1579204743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c00bd40 con 0x7fc7a406a7f0 2023-03-03T08:44:44.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a37fe700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7a414de50 0x7fc7a4071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:44:44.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.993+0000 7fc7a2ffd700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 0x7fc7a4153360 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc79400e9b0 tx=0x7fc79400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:44.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.994+0000 7fc78b7fe700 1 -- 172.21.15.7:0/1579204743 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794009710 con 0x7fc7a406a7f0 2023-03-03T08:44:44.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.994+0000 7fc78b7fe700 1 -- 172.21.15.7:0/1579204743 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79400ee50 con 0x7fc7a406a7f0 2023-03-03T08:44:44.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.994+0000 7fc78b7fe700 1 -- 172.21.15.7:0/1579204743 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79400ca50 con 0x7fc7a406a7f0 2023-03-03T08:44:44.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.994+0000 7fc7a9670700 1 -- 172.21.15.7:0/1579204743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a41547b0 con 0x7fc7a406a7f0 2023-03-03T08:44:44.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.994+0000 7fc7a9670700 1 -- 172.21.15.7:0/1579204743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a41544d0 con 0x7fc7a406a7f0 2023-03-03T08:44:45.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.995+0000 7fc7a9670700 1 -- 172.21.15.7:0/1579204743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a4154a90 con 0x7fc7a406a7f0 2023-03-03T08:44:45.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.996+0000 7fc78b7fe700 1 -- 172.21.15.7:0/1579204743 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc794019030 con 0x7fc7a406a7f0 2023-03-03T08:44:45.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.996+0000 7fc78b7fe700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7900682f0 0x7fc79006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:45.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.996+0000 7fc78b7fe700 1 -- 172.21.15.7:0/1579204743 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7018+0+0 (secure 0 0 0) 0x7fc794085ed0 con 0x7fc7a406a7f0 2023-03-03T08:44:45.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.996+0000 7fc7a27fc700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7900682f0 0x7fc79006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:45.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:44.997+0000 7fc7a27fc700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7900682f0 0x7fc79006a7a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc798007fd0 tx=0x7fc798009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:45.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.000+0000 7fc78b7fe700 1 -- 172.21.15.7:0/1579204743 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc794052370 con 0x7fc7a406a7f0 2023-03-03T08:44:45.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.304+0000 7fc7a9670700 1 -- 172.21.15.7:0/1579204743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7a4074bb0 con 0x7fc7a406a7f0 2023-03-03T08:44:45.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.305+0000 7fc78b7fe700 1 -- 172.21.15.7:0/1579204743 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5006 (secure 0 0 0) 0x7fc794055990 con 0x7fc7a406a7f0 2023-03-03T08:44:45.306 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:45.307 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:45.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 -- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7900682f0 msgr2=0x7fc79006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:45.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7900682f0 0x7fc79006a7a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc798007fd0 tx=0x7fc798009040 comp rx=0 tx=0).stop 2023-03-03T08:44:45.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 -- 172.21.15.7:0/1579204743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 msgr2=0x7fc7a4153360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:45.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 0x7fc7a4153360 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc79400e9b0 tx=0x7fc79400ecc0 comp rx=0 tx=0).stop 2023-03-03T08:44:45.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 -- 172.21.15.7:0/1579204743 shutdown_connections 2023-03-03T08:44:45.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7a414de50 0x7fc7a4071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:45.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7900682f0 0x7fc79006a7a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:45.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7a406a7f0 0x7fc7a4153360 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:45.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 --2- 172.21.15.7:0/1579204743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7a40c65c0 0x7fc7a40a7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:45.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 -- 172.21.15.7:0/1579204743 >> 172.21.15.7:0/1579204743 conn(0x7fc7a40a9430 msgr2=0x7fc7a40c3680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:45.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 -- 172.21.15.7:0/1579204743 shutdown_connections 2023-03-03T08:44:45.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:45.308+0000 7fc7a9670700 1 -- 172.21.15.7:0/1579204743 wait complete. 2023-03-03T08:44:45.313 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 18 2023-03-03T08:44:45.325 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-03-03T08:44:47.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:44:47.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.516+0000 7f5af0cf6700 1 Processor -- start 2023-03-03T08:44:47.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.516+0000 7f5af0cf6700 1 -- start start 2023-03-03T08:44:47.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5af0cf6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec13a0f0 0x7f5aec0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:47.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5af0cf6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5aec1586c0 0x7f5aec06d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5af0cf6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5aec0c1da0 0x7f5aec06d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5af0cf6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec154840 con 0x7f5aec0c1da0 2023-03-03T08:44:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5af0cf6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec154560 con 0x7f5aec1586c0 2023-03-03T08:44:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5af0cf6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec157be0 con 0x7f5aec13a0f0 2023-03-03T08:44:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5aead9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5aec0c1da0 0x7f5aec06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5aead9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5aec0c1da0 0x7f5aec06d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52800/0 (socket says 172.21.15.7:52800) 2023-03-03T08:44:47.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5aead9d700 1 -- 172.21.15.7:0/3184206455 learned_addr learned my addr 172.21.15.7:0/3184206455 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:47.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5ae9d9b700 1 --2- 172.21.15.7:0/3184206455 >> v2:172.21.15.143:3300/0 conn(0x7f5aec1586c0 0x7f5aec06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:47.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5aea59c700 1 --2- 172.21.15.7:0/3184206455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec13a0f0 0x7f5aec0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:47.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5aead9d700 1 -- 172.21.15.7:0/3184206455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec13a0f0 msgr2=0x7f5aec0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:47.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5aead9d700 1 --2- 172.21.15.7:0/3184206455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec13a0f0 0x7f5aec0c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:47.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5aead9d700 1 -- 172.21.15.7:0/3184206455 >> v2:172.21.15.143:3300/0 conn(0x7f5aec1586c0 msgr2=0x7f5aec06d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:44:47.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5aead9d700 1 --2- 172.21.15.7:0/3184206455 >> v2:172.21.15.143:3300/0 conn(0x7f5aec1586c0 0x7f5aec06d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:47.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.518+0000 7f5aead9d700 1 -- 172.21.15.7:0/3184206455 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec157ec0 con 0x7f5aec0c1da0 2023-03-03T08:44:47.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.519+0000 7f5aead9d700 1 --2- 172.21.15.7:0/3184206455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 0x7f5aec06d780 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0aa1d0 tx=0x7f5ae400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf10976f5877065 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:47.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.519+0000 7f5ae959a700 1 -- 172.21.15.7:0/3184206455 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400e070 con 0x7f5aec0c1da0 2023-03-03T08:44:47.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.519+0000 7f5ae959a700 1 -- 172.21.15.7:0/3184206455 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae400b910 con 0x7f5aec0c1da0 2023-03-03T08:44:47.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.519+0000 7f5ae959a700 1 -- 172.21.15.7:0/3184206455 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400bcc0 con 0x7f5aec0c1da0 2023-03-03T08:44:47.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.520+0000 7f5af0cf6700 1 -- 172.21.15.7:0/3184206455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 msgr2=0x7f5aec06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:47.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.520+0000 7f5af0cf6700 1 --2- 172.21.15.7:0/3184206455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 0x7f5aec06d780 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0aa1d0 tx=0x7f5ae400b330 comp rx=0 tx=0).stop 2023-03-03T08:44:47.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.520+0000 7f5af0cf6700 1 -- 172.21.15.7:0/3184206455 shutdown_connections 2023-03-03T08:44:47.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.520+0000 7f5af0cf6700 1 --2- 172.21.15.7:0/3184206455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 0x7f5aec06d780 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:47.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.520+0000 7f5af0cf6700 1 --2- 172.21.15.7:0/3184206455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec13a0f0 0x7f5aec0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:47.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.520+0000 7f5af0cf6700 1 --2- 172.21.15.7:0/3184206455 >> v2:172.21.15.143:3300/0 conn(0x7f5aec1586c0 0x7f5aec06d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:47.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.520+0000 7f5af0cf6700 1 -- 172.21.15.7:0/3184206455 >> 172.21.15.7:0/3184206455 conn(0x7f5aec06c110 msgr2=0x7f5aec06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:47.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.520+0000 7f5af0cf6700 1 -- 172.21.15.7:0/3184206455 shutdown_connections 2023-03-03T08:44:47.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.520+0000 7f5af0cf6700 1 -- 172.21.15.7:0/3184206455 wait complete. 2023-03-03T08:44:47.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.522+0000 7f5af0cf6700 1 Processor -- start 2023-03-03T08:44:47.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.522+0000 7f5af0cf6700 1 -- start start 2023-03-03T08:44:47.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5af0cf6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 0x7f5aec1514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:47.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5af0cf6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec13a0f0 0x7f5aec0acb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:47.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5af0cf6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec1586c0 0x7f5aec071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:47.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5af0cf6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec151a00 con 0x7f5aec0c1da0 2023-03-03T08:44:47.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5af0cf6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec1555d0 con 0x7f5aec13a0f0 2023-03-03T08:44:47.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5af0cf6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec151ca0 con 0x7f5aec1586c0 2023-03-03T08:44:47.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5aea59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 0x7f5aec1514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:47.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5aea59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 0x7f5aec1514c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52802/0 (socket says 172.21.15.7:52802) 2023-03-03T08:44:47.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5aea59c700 1 -- 172.21.15.7:0/1690724752 learned_addr learned my addr 172.21.15.7:0/1690724752 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:44:47.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5ae9d9b700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec13a0f0 0x7f5aec0acb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:47.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.523+0000 7f5aead9d700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec1586c0 0x7f5aec071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:47.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5aea59c700 1 -- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec1586c0 msgr2=0x7f5aec071150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:47.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5aea59c700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec1586c0 0x7f5aec071150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:47.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5aea59c700 1 -- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec13a0f0 msgr2=0x7f5aec0acb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:47.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5aea59c700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec13a0f0 0x7f5aec0acb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:47.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5aea59c700 1 -- 172.21.15.7:0/1690724752 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4009b30 con 0x7f5aec0c1da0 2023-03-03T08:44:47.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5aea59c700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 0x7f5aec1514c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00e9e0 tx=0x7f5adc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5adaffd700 1 -- 172.21.15.7:0/1690724752 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc009710 con 0x7f5aec0c1da0 2023-03-03T08:44:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5af0cf6700 1 -- 172.21.15.7:0/1690724752 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec06e1a0 con 0x7f5aec0c1da0 2023-03-03T08:44:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5af0cf6700 1 -- 172.21.15.7:0/1690724752 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec1581a0 con 0x7f5aec0c1da0 2023-03-03T08:44:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5adaffd700 1 -- 172.21.15.7:0/1690724752 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc00ee80 con 0x7f5aec0c1da0 2023-03-03T08:44:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.524+0000 7f5adaffd700 1 -- 172.21.15.7:0/1690724752 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc00ca50 con 0x7f5aec0c1da0 2023-03-03T08:44:47.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.526+0000 7f5af0cf6700 1 -- 172.21.15.7:0/1690724752 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad00052e0 con 0x7f5aec0c1da0 2023-03-03T08:44:47.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.526+0000 7f5adaffd700 1 -- 172.21.15.7:0/1690724752 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5adc023070 con 0x7f5aec0c1da0 2023-03-03T08:44:47.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.527+0000 7f5adaffd700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5acc0682f0 0x7f5acc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:44:47.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.527+0000 7f5adaffd700 1 -- 172.21.15.7:0/1690724752 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7018+0+0 (secure 0 0 0) 0x7f5adc086150 con 0x7f5aec0c1da0 2023-03-03T08:44:47.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.527+0000 7f5ae9d9b700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5acc0682f0 0x7f5acc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:44:47.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.528+0000 7f5ae9d9b700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5acc0682f0 0x7f5acc06a7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5aec071cc0 tx=0x7f5ae0009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:44:47.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:47.535+0000 7f5adaffd700 1 -- 172.21.15.7:0/1690724752 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5adc0525f0 con 0x7f5aec0c1da0 2023-03-03T08:44:48.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.016+0000 7f5af0cf6700 1 -- 172.21.15.7:0/1690724752 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ad0005b10 con 0x7f5aec0c1da0 2023-03-03T08:44:48.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.017+0000 7f5adaffd700 1 -- 172.21.15.7:0/1690724752 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5760 (secure 0 0 0) 0x7f5adc055c10 con 0x7f5aec0c1da0 2023-03-03T08:44:48.018 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:44:48.019 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:44:48.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.019+0000 7f5af0cf6700 1 -- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5acc0682f0 msgr2=0x7f5acc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:48.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.019+0000 7f5af0cf6700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5acc0682f0 0x7f5acc06a7a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5aec071cc0 tx=0x7f5ae0009210 comp rx=0 tx=0).stop 2023-03-03T08:44:48.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.019+0000 7f5af0cf6700 1 -- 172.21.15.7:0/1690724752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 msgr2=0x7f5aec1514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:44:48.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.019+0000 7f5af0cf6700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 0x7f5aec1514c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00e9e0 tx=0x7f5adc00ecf0 comp rx=0 tx=0).stop 2023-03-03T08:44:48.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.020+0000 7f5af0cf6700 1 -- 172.21.15.7:0/1690724752 shutdown_connections 2023-03-03T08:44:48.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.020+0000 7f5af0cf6700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5aec1586c0 0x7f5aec071150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:48.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.020+0000 7f5af0cf6700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5acc0682f0 0x7f5acc06a7a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:48.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.020+0000 7f5af0cf6700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5aec0c1da0 0x7f5aec1514c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:48.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.020+0000 7f5af0cf6700 1 --2- 172.21.15.7:0/1690724752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5aec13a0f0 0x7f5aec0acb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:44:48.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.020+0000 7f5af0cf6700 1 -- 172.21.15.7:0/1690724752 >> 172.21.15.7:0/1690724752 conn(0x7f5aec06c110 msgr2=0x7f5aec137150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:44:48.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.020+0000 7f5af0cf6700 1 -- 172.21.15.7:0/1690724752 shutdown_connections 2023-03-03T08:44:48.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:44:48.020+0000 7f5af0cf6700 1 -- 172.21.15.7:0/1690724752 wait complete. 2023-03-03T08:44:48.023 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 19 2023-03-03T08:44:48.037 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2023-03-03T08:44:48.038 INFO:tasks.mds_thrash.fs.[b]:waiting for 17 secs before thrashing 2023-03-03T08:45:05.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:05.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.177+0000 7f7fd1b00700 1 Processor -- start 2023-03-03T08:45:05.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.177+0000 7f7fd1b00700 1 -- start start 2023-03-03T08:45:05.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fd1b00700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fcc071e80 0x7f7fcc151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fd1b00700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7fcc072250 0x7f7fcc14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fd1b00700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fcc1510d0 con 0x7f7fcc0c1ce0 2023-03-03T08:45:05.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fd1b00700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fcc0c3740 con 0x7f7fcc072250 2023-03-03T08:45:05.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fd1b00700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc070670 con 0x7f7fcc071e80 2023-03-03T08:45:05.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fcaffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7fcc072250 0x7f7fcc14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fcb7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fcc071e80 0x7f7fcc151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fcbfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f7fcc0c1ce0 0x7f7fcc1546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50330/0 (socket says 172.21.15.7:50330) 2023-03-03T08:45:05.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fcbfff700 1 -- 172.21.15.7:0/2913308614 learned_addr learned my addr 172.21.15.7:0/2913308614 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:45:05.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fcaffd700 1 -- 172.21.15.7:0/2913308614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fcc071e80 msgr2=0x7f7fcc151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:05.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fcaffd700 1 --2- 172.21.15.7:0/2913308614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fcc071e80 0x7f7fcc151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fcaffd700 1 -- 172.21.15.7:0/2913308614 >> v1:172.21.15.143:6789/0 conn(0x7f7fcc0c1ce0 legacy=0x7f7fcc1546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.178+0000 7f7fcaffd700 1 -- 172.21.15.7:0/2913308614 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc0c39e0 con 0x7f7fcc072250 2023-03-03T08:45:05.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fcaffd700 1 --2- 172.21.15.7:0/2913308614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fcc072250 0x7f7fcc14e1f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009840 tx=0x7f7fb4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcafcea6c194a422 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:05.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fca7fc700 1 -- 172.21.15.7:0/2913308614 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400f8a0 con 0x7f7fcc072250 2023-03-03T08:45:05.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fca7fc700 1 -- 172.21.15.7:0/2913308614 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb400fa00 con 0x7f7fcc072250 2023-03-03T08:45:05.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fca7fc700 1 -- 172.21.15.7:0/2913308614 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb400fd90 con 0x7f7fcc072250 2023-03-03T08:45:05.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913308614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fcc072250 msgr2=0x7f7fcc14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fd1b00700 1 --2- 172.21.15.7:0/2913308614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fcc072250 0x7f7fcc14e1f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009840 tx=0x7f7fb4009b50 comp rx=0 tx=0).stop 2023-03-03T08:45:05.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913308614 shutdown_connections 2023-03-03T08:45:05.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fd1b00700 1 --2- 172.21.15.7:0/2913308614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fcc071e80 0x7f7fcc151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fd1b00700 1 --2- 172.21.15.7:0/2913308614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fcc072250 0x7f7fcc14e1f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913308614 >> 172.21.15.7:0/2913308614 conn(0x7f7fcc0a94d0 msgr2=0x7f7fcc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:05.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913308614 shutdown_connections 2023-03-03T08:45:05.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.179+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913308614 wait complete. 2023-03-03T08:45:05.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.181+0000 7f7fd1b00700 1 Processor -- start 2023-03-03T08:45:05.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.181+0000 7f7fd1b00700 1 -- start start 2023-03-03T08:45:05.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fd1b00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fcc071e80 0x7f7fcc145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fd1b00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fcc072250 0x7f7fcc144b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fd1b00700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fcc0c1ce0 0x7f7fcc14aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fd1b00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc142790 con 0x7f7fcc072250 2023-03-03T08:45:05.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fd1b00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc14da10 con 0x7f7fcc071e80 2023-03-03T08:45:05.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fd1b00700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc14d6e0 con 0x7f7fcc0c1ce0 2023-03-03T08:45:05.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fcc072250 0x7f7fcc144b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fcc072250 0x7f7fcc144b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52834/0 (socket says 172.21.15.7:52834) 2023-03-03T08:45:05.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcaffd700 1 -- 172.21.15.7:0/2913192956 learned_addr learned my addr 172.21.15.7:0/2913192956 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:05.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcb7fe700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fcc071e80 0x7f7fcc145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcbfff700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fcc0c1ce0 0x7f7fcc14aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcaffd700 1 -- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fcc0c1ce0 msgr2=0x7f7fcc14aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcaffd700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fcc0c1ce0 0x7f7fcc14aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcaffd700 1 -- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fcc071e80 msgr2=0x7f7fcc145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcaffd700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fcc071e80 0x7f7fcc145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcaffd700 1 -- 172.21.15.7:0/2913192956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb40094a0 con 0x7f7fcc072250 2023-03-03T08:45:05.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcb7fe700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fcc071e80 0x7f7fcc145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:45:05.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.182+0000 7f7fcaffd700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fcc072250 0x7f7fcc144b40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40090b0 tx=0x7f7fb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:05.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.183+0000 7f7fc3fff700 1 -- 172.21.15.7:0/2913192956 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb40038a0 con 0x7f7fcc072250 2023-03-03T08:45:05.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.183+0000 7f7fc3fff700 1 -- 172.21.15.7:0/2913192956 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb4003a20 con 0x7f7fcc072250 2023-03-03T08:45:05.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.183+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913192956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc06c9a0 con 0x7f7fcc072250 2023-03-03T08:45:05.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.183+0000 7f7fc3fff700 1 -- 172.21.15.7:0/2913192956 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4003dd0 con 0x7f7fcc072250 2023-03-03T08:45:05.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.183+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913192956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc151300 con 0x7f7fcc072250 2023-03-03T08:45:05.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.184+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913192956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc06cb30 con 0x7f7fcc072250 2023-03-03T08:45:05.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.184+0000 7f7fc3fff700 1 -- 172.21.15.7:0/2913192956 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7fb4029020 con 0x7f7fcc072250 2023-03-03T08:45:05.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.184+0000 7f7fc3fff700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7fac0682f0 0x7f7fac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.185+0000 7f7fc3fff700 1 -- 172.21.15.7:0/2913192956 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7018+0+0 (secure 0 0 0) 0x7f7fb40872a0 con 0x7f7fcc072250 2023-03-03T08:45:05.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.185+0000 7f7fcb7fe700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7fac0682f0 0x7f7fac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.186+0000 7f7fcb7fe700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7fac0682f0 0x7f7fac06a7a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc1475e0 tx=0x7f7fbc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:05.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.189+0000 7f7fc3fff700 1 -- 172.21.15.7:0/2913192956 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7fb4053740 con 0x7f7fcc072250 2023-03-03T08:45:05.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:05.267 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:05.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.430+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913192956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fcc14dcf0 con 0x7f7fcc072250 2023-03-03T08:45:05.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.431+0000 7f7fc3fff700 1 -- 172.21.15.7:0/2913192956 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5760 (secure 0 0 0) 0x7f7fb4056d60 con 0x7f7fcc072250 2023-03-03T08:45:05.433 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:05.433 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:05.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7fac0682f0 msgr2=0x7f7fac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7fac0682f0 0x7f7fac06a7a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc1475e0 tx=0x7f7fbc009210 comp rx=0 tx=0).stop 2023-03-03T08:45:05.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913192956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fcc072250 msgr2=0x7f7fcc144b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fcc072250 0x7f7fcc144b40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40090b0 tx=0x7f7fb4004750 comp rx=0 tx=0).stop 2023-03-03T08:45:05.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913192956 shutdown_connections 2023-03-03T08:45:05.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7fcc0c1ce0 0x7f7fcc14aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7fcc071e80 0x7f7fcc145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7fac0682f0 0x7f7fac06a7a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 --2- 172.21.15.7:0/2913192956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7fcc072250 0x7f7fcc144b40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913192956 >> 172.21.15.7:0/2913192956 conn(0x7f7fcc0a94d0 msgr2=0x7f7fcc151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:05.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913192956 shutdown_connections 2023-03-03T08:45:05.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.433+0000 7f7fd1b00700 1 -- 172.21.15.7:0/2913192956 wait complete. 2023-03-03T08:45:05.437 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 19 2023-03-03T08:45:05.447 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2023-03-03T08:45:05.448 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-03-03T08:45:05.448 INFO:teuthology.orchestra.run:waiting for 300 2023-03-03T08:45:05.457 INFO:tasks.ceph.mds.d:Stopped 2023-03-03T08:45:05.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:05.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.582+0000 7fe97e207700 1 Processor -- start 2023-03-03T08:45:05.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.582+0000 7fe97e207700 1 -- start start 2023-03-03T08:45:05.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe97e207700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe9781586c0 0x7fe97806d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe97e207700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe97806de90 con 0x7fe9780c1da0 2023-03-03T08:45:05.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe97e207700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe978154560 con 0x7fe97813a0f0 2023-03-03T08:45:05.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe97e207700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe978154840 con 0x7fe9781586c0 2023-03-03T08:45:05.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe977fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe9780c1da0 0x7fe97806d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46824/0 (socket says 172.21.15.7:46824) 2023-03-03T08:45:05.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe977fff700 1 -- 172.21.15.7:0/1064273910 learned_addr learned my addr 172.21.15.7:0/1064273910 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:45:05.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe976ffd700 1 --2- 172.21.15.7:0/1064273910 >> v2:172.21.15.143:3300/0 conn(0x7fe9781586c0 0x7fe97806d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe9767fc700 1 -- 172.21.15.7:0/1064273910 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530541100 0 0) 0x7fe970001490 con 0x7fe9780c1da0 2023-03-03T08:45:05.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe9767fc700 1 -- 172.21.15.7:0/1064273910 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9600035b0 con 0x7fe9780c1da0 2023-03-03T08:45:05.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe976ffd700 1 -- 172.21.15.7:0/1064273910 >> v1:172.21.15.143:6789/0 conn(0x7fe97813a0f0 legacy=0x7fe9780c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe976ffd700 1 -- 172.21.15.7:0/1064273910 >> v1:172.21.15.7:6789/0 conn(0x7fe9780c1da0 legacy=0x7fe97806d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.583+0000 7fe976ffd700 1 -- 172.21.15.7:0/1064273910 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe978157be0 con 0x7fe9781586c0 2023-03-03T08:45:05.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe976ffd700 1 --2- 172.21.15.7:0/1064273910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9781586c0 0x7fe97806d240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe96c009840 tx=0x7fe96c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5271543852dc9cff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:05.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe9767fc700 1 -- 172.21.15.7:0/1064273910 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96c011830 con 0x7fe9781586c0 2023-03-03T08:45:05.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe9767fc700 1 -- 172.21.15.7:0/1064273910 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96c011990 con 0x7fe9781586c0 2023-03-03T08:45:05.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe9767fc700 1 -- 172.21.15.7:0/1064273910 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96c011d20 con 0x7fe9781586c0 2023-03-03T08:45:05.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe97e207700 1 -- 172.21.15.7:0/1064273910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9781586c0 msgr2=0x7fe97806d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe97e207700 1 --2- 172.21.15.7:0/1064273910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9781586c0 0x7fe97806d240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe96c009840 tx=0x7fe96c009b50 comp rx=0 tx=0).stop 2023-03-03T08:45:05.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe97e207700 1 -- 172.21.15.7:0/1064273910 shutdown_connections 2023-03-03T08:45:05.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe97e207700 1 --2- 172.21.15.7:0/1064273910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9781586c0 0x7fe97806d240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe97e207700 1 -- 172.21.15.7:0/1064273910 >> 172.21.15.7:0/1064273910 conn(0x7fe97806c110 msgr2=0x7fe97806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:05.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe97e207700 1 -- 172.21.15.7:0/1064273910 shutdown_connections 2023-03-03T08:45:05.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.584+0000 7fe97e207700 1 -- 172.21.15.7:0/1064273910 wait complete. 2023-03-03T08:45:05.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.585+0000 7fe97e207700 1 Processor -- start 2023-03-03T08:45:05.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.586+0000 7fe97e207700 1 -- start start 2023-03-03T08:45:05.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.586+0000 7fe97e207700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9780c1da0 0x7fe97806a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.586+0000 7fe97e207700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe97813a0f0 0x7fe9781515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.586+0000 7fe97e207700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9781586c0 0x7fe978151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.586+0000 7fe97e207700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9781edb20 con 0x7fe9781586c0 2023-03-03T08:45:05.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.586+0000 7fe97e207700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97813dbb0 con 0x7fe9780c1da0 2023-03-03T08:45:05.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.586+0000 7fe97e207700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe978152020 con 0x7fe97813a0f0 2023-03-03T08:45:05.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe977fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9781586c0 0x7fe978151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe9777fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9780c1da0 0x7fe97806a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe977fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9781586c0 0x7fe978151ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52850/0 (socket says 172.21.15.7:52850) 2023-03-03T08:45:05.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe977fff700 1 -- 172.21.15.7:0/195084477 learned_addr learned my addr 172.21.15.7:0/195084477 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:05.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe976ffd700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe97813a0f0 0x7fe9781515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe977fff700 1 -- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe97813a0f0 msgr2=0x7fe9781515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe977fff700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe97813a0f0 0x7fe9781515a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe977fff700 1 -- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9780c1da0 msgr2=0x7fe97806a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe977fff700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9780c1da0 0x7fe97806a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe977fff700 1 -- 172.21.15.7:0/195084477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96c0094a0 con 0x7fe9781586c0 2023-03-03T08:45:05.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe977fff700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9781586c0 0x7fe978151ae0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe97000cb70 tx=0x7fe97000ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:05.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe95ffff700 1 -- 172.21.15.7:0/195084477 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97000aa60 con 0x7fe9781586c0 2023-03-03T08:45:05.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe95ffff700 1 -- 172.21.15.7:0/195084477 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97000abc0 con 0x7fe9781586c0 2023-03-03T08:45:05.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe97e207700 1 -- 172.21.15.7:0/195084477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9781581a0 con 0x7fe9781586c0 2023-03-03T08:45:05.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe95ffff700 1 -- 172.21.15.7:0/195084477 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe970012400 con 0x7fe9781586c0 2023-03-03T08:45:05.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.587+0000 7fe97e207700 1 -- 172.21.15.7:0/195084477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe978157ec0 con 0x7fe9781586c0 2023-03-03T08:45:05.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.588+0000 7fe97e207700 1 -- 172.21.15.7:0/195084477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe97806e1a0 con 0x7fe9781586c0 2023-03-03T08:45:05.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.589+0000 7fe95ffff700 1 -- 172.21.15.7:0/195084477 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe970019030 con 0x7fe9781586c0 2023-03-03T08:45:05.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.589+0000 7fe95ffff700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9640682f0 0x7fe96406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:05.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.589+0000 7fe95ffff700 1 -- 172.21.15.7:0/195084477 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7018+0+0 (secure 0 0 0) 0x7fe9700859e0 con 0x7fe9781586c0 2023-03-03T08:45:05.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.589+0000 7fe9777fe700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9640682f0 0x7fe96406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:05.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.590+0000 7fe9777fe700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9640682f0 0x7fe96406a7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe968007fd0 tx=0x7fe968009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.594+0000 7fe95ffff700 1 -- 172.21.15.7:0/195084477 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe970051e80 con 0x7fe9781586c0 2023-03-03T08:45:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.832+0000 7fe97e207700 1 -- 172.21.15.7:0/195084477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9781edc90 con 0x7fe9781586c0 2023-03-03T08:45:05.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.832+0000 7fe95ffff700 1 -- 172.21.15.7:0/195084477 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5760 (secure 0 0 0) 0x7fe9781edc90 con 0x7fe9781586c0 2023-03-03T08:45:05.835 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:05.835 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.834+0000 7fe97e207700 1 -- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9640682f0 msgr2=0x7fe96406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.834+0000 7fe97e207700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9640682f0 0x7fe96406a7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe968007fd0 tx=0x7fe968009040 comp rx=0 tx=0).stop 2023-03-03T08:45:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.834+0000 7fe97e207700 1 -- 172.21.15.7:0/195084477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9781586c0 msgr2=0x7fe978151ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.834+0000 7fe97e207700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9781586c0 0x7fe978151ae0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe97000cb70 tx=0x7fe97000ce80 comp rx=0 tx=0).stop 2023-03-03T08:45:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.834+0000 7fe97e207700 1 -- 172.21.15.7:0/195084477 shutdown_connections 2023-03-03T08:45:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.834+0000 7fe97e207700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9640682f0 0x7fe96406a7a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.834+0000 7fe97e207700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9781586c0 0x7fe978151ae0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.834+0000 7fe97e207700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9780c1da0 0x7fe97806a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.834+0000 7fe97e207700 1 --2- 172.21.15.7:0/195084477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe97813a0f0 0x7fe9781515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:05.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.834+0000 7fe97e207700 1 -- 172.21.15.7:0/195084477 >> 172.21.15.7:0/195084477 conn(0x7fe97806c110 msgr2=0x7fe97806d220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:05.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.835+0000 7fe97e207700 1 -- 172.21.15.7:0/195084477 shutdown_connections 2023-03-03T08:45:05.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:05.835+0000 7fe97e207700 1 -- 172.21.15.7:0/195084477 wait complete. 2023-03-03T08:45:05.839 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 19 2023-03-03T08:45:05.849 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-03-03T08:45:06.614 INFO:tasks.workunit.client.1.smithi143.stderr:Note: switching to '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'. 2023-03-03T08:45:06.614 INFO:tasks.workunit.client.1.smithi143.stderr: 2023-03-03T08:45:06.614 INFO:tasks.workunit.client.1.smithi143.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-03-03T08:45:06.614 INFO:tasks.workunit.client.1.smithi143.stderr:changes and commit them, and you can discard any commits you make in this 2023-03-03T08:45:06.614 INFO:tasks.workunit.client.1.smithi143.stderr:state without impacting any branches by switching back to a branch. 2023-03-03T08:45:06.614 INFO:tasks.workunit.client.1.smithi143.stderr: 2023-03-03T08:45:06.615 INFO:tasks.workunit.client.1.smithi143.stderr:If you want to create a new branch to retain commits you create, you may 2023-03-03T08:45:06.615 INFO:tasks.workunit.client.1.smithi143.stderr:do so (now or later) by using -c with the switch command. Example: 2023-03-03T08:45:06.615 INFO:tasks.workunit.client.1.smithi143.stderr: 2023-03-03T08:45:06.615 INFO:tasks.workunit.client.1.smithi143.stderr: git switch -c 2023-03-03T08:45:06.615 INFO:tasks.workunit.client.1.smithi143.stderr: 2023-03-03T08:45:06.615 INFO:tasks.workunit.client.1.smithi143.stderr:Or undo this operation with: 2023-03-03T08:45:06.615 INFO:tasks.workunit.client.1.smithi143.stderr: 2023-03-03T08:45:06.616 INFO:tasks.workunit.client.1.smithi143.stderr: git switch - 2023-03-03T08:45:06.616 INFO:tasks.workunit.client.1.smithi143.stderr: 2023-03-03T08:45:06.616 INFO:tasks.workunit.client.1.smithi143.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-03-03T08:45:06.616 INFO:tasks.workunit.client.1.smithi143.stderr: 2023-03-03T08:45:06.616 INFO:tasks.workunit.client.1.smithi143.stderr:HEAD is now at 472aaef4ec3 Merge PR #45224 into wip-vshankar-testing-20230303.023823 2023-03-03T08:45:06.617 DEBUG:teuthology.orchestra.run.smithi143:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2023-03-03T08:45:06.642 INFO:tasks.workunit.client.1.smithi143.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-03-03T08:45:06.643 INFO:tasks.workunit.client.1.smithi143.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-03-03T08:45:06.644 INFO:tasks.workunit.client.1.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-03-03T08:45:06.730 INFO:tasks.workunit.client.0.smithi007.stderr:Note: switching to '472aaef4ec3b3a2c29f73ca67854785a910f1f9b'. 2023-03-03T08:45:06.730 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-03-03T08:45:06.731 INFO:tasks.workunit.client.0.smithi007.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-03-03T08:45:06.731 INFO:tasks.workunit.client.0.smithi007.stderr:changes and commit them, and you can discard any commits you make in this 2023-03-03T08:45:06.731 INFO:tasks.workunit.client.0.smithi007.stderr:state without impacting any branches by switching back to a branch. 2023-03-03T08:45:06.731 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-03-03T08:45:06.731 INFO:tasks.workunit.client.0.smithi007.stderr:If you want to create a new branch to retain commits you create, you may 2023-03-03T08:45:06.731 INFO:tasks.workunit.client.0.smithi007.stderr:do so (now or later) by using -c with the switch command. Example: 2023-03-03T08:45:06.731 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-03-03T08:45:06.731 INFO:tasks.workunit.client.0.smithi007.stderr: git switch -c 2023-03-03T08:45:06.732 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-03-03T08:45:06.732 INFO:tasks.workunit.client.0.smithi007.stderr:Or undo this operation with: 2023-03-03T08:45:06.732 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-03-03T08:45:06.732 INFO:tasks.workunit.client.0.smithi007.stderr: git switch - 2023-03-03T08:45:06.732 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-03-03T08:45:06.732 INFO:tasks.workunit.client.0.smithi007.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-03-03T08:45:06.732 INFO:tasks.workunit.client.0.smithi007.stderr: 2023-03-03T08:45:06.732 INFO:tasks.workunit.client.0.smithi007.stderr:HEAD is now at 472aaef4ec3 Merge PR #45224 into wip-vshankar-testing-20230303.023823 2023-03-03T08:45:06.733 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-03-03T08:45:06.765 INFO:tasks.workunit.client.0.smithi007.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-03-03T08:45:06.767 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-03-03T08:45:06.768 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-03-03T08:45:07.205 INFO:tasks.workunit.client.1.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-03-03T08:45:07.272 INFO:tasks.workunit.client.1.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-03-03T08:45:07.309 INFO:tasks.workunit.client.1.smithi143.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-03-03T08:45:07.310 INFO:tasks.workunit.client.1.smithi143.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-03-03T08:45:07.311 INFO:tasks.workunit.client.1.smithi143.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-03-03T08:45:07.313 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-03-03T08:45:07.347 INFO:tasks.workunit.client.1.smithi143.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-03-03T08:45:07.350 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2023-03-03T08:45:07.350 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2023-03-03T08:45:07.366 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-03-03T08:45:07.405 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-03-03T08:45:07.407 INFO:tasks.workunit:Running workunits matching fs/snaps on client.1... 2023-03-03T08:45:07.408 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-03-03T08:45:07.408 DEBUG:teuthology.orchestra.run.smithi143:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-03-03T08:45:07.409 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-03-03T08:45:07.409 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-03-03T08:45:07.445 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-03-03T08:45:07.449 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-03-03T08:45:07.449 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-03-03T08:45:07.505 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-03-03T08:45:07.506 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-03-03T08:45:07.507 DEBUG:teuthology.orchestra.run.smithi007:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=472aaef4ec3b3a2c29f73ca67854785a910f1f9b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-03-03T08:45:07.568 INFO:tasks.workunit.client.0.smithi007.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-03-03T08:45:07.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:07.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.976+0000 7fdc0f9e3700 1 Processor -- start 2023-03-03T08:45:07.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.976+0000 7fdc0f9e3700 1 -- start start 2023-03-03T08:45:07.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.977+0000 7fdc0f9e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc0814e900 0x7fdc0815ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.977+0000 7fdc0f9e3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc081510d0 con 0x7fdc08072250 2023-03-03T08:45:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.977+0000 7fdc0f9e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc080c3740 con 0x7fdc08071e80 2023-03-03T08:45:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.977+0000 7fdc0f9e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc08070670 con 0x7fdc0814e900 2023-03-03T08:45:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.977+0000 7fdc0cf7e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdc08072250 0x7fdc0814e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46838/0 (socket says 172.21.15.7:46838) 2023-03-03T08:45:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.977+0000 7fdc0cf7e700 1 -- 172.21.15.7:0/1057384226 learned_addr learned my addr 172.21.15.7:0/1057384226 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:45:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.977+0000 7fdc0df80700 1 --2- 172.21.15.7:0/1057384226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc0814e900 0x7fdc0815ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.978+0000 7fdbfffff700 1 -- 172.21.15.7:0/1057384226 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189882558 0 0) 0x7fdc081510d0 con 0x7fdc08072250 2023-03-03T08:45:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.978+0000 7fdbfffff700 1 -- 172.21.15.7:0/1057384226 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf40035b0 con 0x7fdc08072250 2023-03-03T08:45:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.978+0000 7fdbfffff700 1 -- 172.21.15.7:0/1057384226 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3168135589 0 0) 0x7fdbf40035b0 con 0x7fdc08072250 2023-03-03T08:45:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.978+0000 7fdc0df80700 1 -- 172.21.15.7:0/1057384226 >> v1:172.21.15.143:6789/0 conn(0x7fdc08071e80 legacy=0x7fdc08151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.978+0000 7fdc0df80700 1 -- 172.21.15.7:0/1057384226 >> v1:172.21.15.7:6789/0 conn(0x7fdc08072250 legacy=0x7fdc0814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.978+0000 7fdc0df80700 1 -- 172.21.15.7:0/1057384226 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc080c39e0 con 0x7fdc0814e900 2023-03-03T08:45:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.978+0000 7fdc0df80700 1 --2- 172.21.15.7:0/1057384226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc0814e900 0x7fdc0815ca60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdc00009dc0 tx=0x7fdc0000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a440550b661febd9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:07.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.978+0000 7fdbfffff700 1 -- 172.21.15.7:0/1057384226 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0000e070 con 0x7fdc0814e900 2023-03-03T08:45:07.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.978+0000 7fdbfffff700 1 -- 172.21.15.7:0/1057384226 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc00003a20 con 0x7fdc0814e900 2023-03-03T08:45:07.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.978+0000 7fdbfffff700 1 -- 172.21.15.7:0/1057384226 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04001670 con 0x7fdc0814e900 2023-03-03T08:45:07.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.979+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/1057384226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc0814e900 msgr2=0x7fdc0815ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:07.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.979+0000 7fdc0f9e3700 1 --2- 172.21.15.7:0/1057384226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc0814e900 0x7fdc0815ca60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdc00009dc0 tx=0x7fdc0000b330 comp rx=0 tx=0).stop 2023-03-03T08:45:07.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.979+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/1057384226 shutdown_connections 2023-03-03T08:45:07.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.979+0000 7fdc0f9e3700 1 --2- 172.21.15.7:0/1057384226 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc0814e900 0x7fdc0815ca60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:07.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.979+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/1057384226 >> 172.21.15.7:0/1057384226 conn(0x7fdc080a94d0 msgr2=0x7fdc080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:07.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.979+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/1057384226 shutdown_connections 2023-03-03T08:45:07.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.979+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/1057384226 wait complete. 2023-03-03T08:45:07.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.980+0000 7fdc0f9e3700 1 Processor -- start 2023-03-03T08:45:07.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.980+0000 7fdc0f9e3700 1 -- start start 2023-03-03T08:45:07.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0f9e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc08071e80 0x7fdc081dae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:07.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0f9e3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc08072250 0x7fdc081dcf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:07.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0f9e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc0814e900 0x7fdc0814f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:07.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0f9e3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc081dbb30 con 0x7fdc08072250 2023-03-03T08:45:07.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0f9e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc081db3d0 con 0x7fdc0814e900 2023-03-03T08:45:07.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0f9e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc081ee030 con 0x7fdc08071e80 2023-03-03T08:45:07.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0cf7e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc08072250 0x7fdc081dcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:07.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0cf7e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc08072250 0x7fdc081dcf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52866/0 (socket says 172.21.15.7:52866) 2023-03-03T08:45:07.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0cf7e700 1 -- 172.21.15.7:0/2704680905 learned_addr learned my addr 172.21.15.7:0/2704680905 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:07.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0d77f700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc08071e80 0x7fdc081dae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:07.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0df80700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc0814e900 0x7fdc0814f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:07.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0cf7e700 1 -- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc08071e80 msgr2=0x7fdc081dae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:07.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0cf7e700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc08071e80 0x7fdc081dae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:07.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0cf7e700 1 -- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc0814e900 msgr2=0x7fdc0814f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:07.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0cf7e700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc0814e900 0x7fdc0814f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:07.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0cf7e700 1 -- 172.21.15.7:0/2704680905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc00009a70 con 0x7fdc08072250 2023-03-03T08:45:07.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0df80700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc0814e900 0x7fdc0814f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:45:07.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdc0cf7e700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc08072250 0x7fdc081dcf10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400c7b0 tx=0x7fdc0400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:07.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.981+0000 7fdbfdffb700 1 -- 172.21.15.7:0/2704680905 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04004c80 con 0x7fdc08072250 2023-03-03T08:45:07.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.982+0000 7fdbfdffb700 1 -- 172.21.15.7:0/2704680905 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc04004de0 con 0x7fdc08072250 2023-03-03T08:45:07.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.982+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/2704680905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc08151f30 con 0x7fdc08072250 2023-03-03T08:45:07.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.982+0000 7fdbfdffb700 1 -- 172.21.15.7:0/2704680905 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04007310 con 0x7fdc08072250 2023-03-03T08:45:07.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.982+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/2704680905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc08151300 con 0x7fdc08072250 2023-03-03T08:45:07.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.983+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/2704680905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0806c9a0 con 0x7fdc08072250 2023-03-03T08:45:07.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.983+0000 7fdbfdffb700 1 -- 172.21.15.7:0/2704680905 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc0401a020 con 0x7fdc08072250 2023-03-03T08:45:07.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.983+0000 7fdbfdffb700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbec0682f0 0x7fdbec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:07.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.984+0000 7fdbfdffb700 1 -- 172.21.15.7:0/2704680905 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7018+0+0 (secure 0 0 0) 0x7fdc04085dd0 con 0x7fdc08072250 2023-03-03T08:45:07.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.984+0000 7fdc0d77f700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbec0682f0 0x7fdbec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:07.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.984+0000 7fdc0d77f700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbec0682f0 0x7fdbec06a7a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8000f60 tx=0x7fdbf8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:07.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:07.988+0000 7fdbfdffb700 1 -- 172.21.15.7:0/2704680905 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc04052320 con 0x7fdc08072250 2023-03-03T08:45:08.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.231+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/2704680905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc08074bb0 con 0x7fdc08072250 2023-03-03T08:45:08.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.232+0000 7fdbfdffb700 1 -- 172.21.15.7:0/2704680905 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5760 (secure 0 0 0) 0x7fdc04055940 con 0x7fdc08072250 2023-03-03T08:45:08.234 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:08.234 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:08.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbec0682f0 msgr2=0x7fdbec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:08.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbec0682f0 0x7fdbec06a7a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8000f60 tx=0x7fdbf8009040 comp rx=0 tx=0).stop 2023-03-03T08:45:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/2704680905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc08072250 msgr2=0x7fdc081dcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc08072250 0x7fdc081dcf10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400c7b0 tx=0x7fdc0400cb70 comp rx=0 tx=0).stop 2023-03-03T08:45:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/2704680905 shutdown_connections 2023-03-03T08:45:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc0814e900 0x7fdc0814f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc08071e80 0x7fdc081dae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbec0682f0 0x7fdbec06a7a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 --2- 172.21.15.7:0/2704680905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc08072250 0x7fdc081dcf10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/2704680905 >> 172.21.15.7:0/2704680905 conn(0x7fdc080a94d0 msgr2=0x7fdc080c2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/2704680905 shutdown_connections 2023-03-03T08:45:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:08.234+0000 7fdc0f9e3700 1 -- 172.21.15.7:0/2704680905 wait complete. 2023-03-03T08:45:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 19 2023-03-03T08:45:08.247 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-03-03T08:45:09.908 INFO:tasks.workunit.client.0.smithi007.stderr:+ mkdir foo 2023-03-03T08:45:09.914 INFO:tasks.workunit.client.0.smithi007.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-03-03T08:45:09.958 INFO:tasks.workunit.client.0.smithi007.stderr:+ mkdir foo/.snap/barsnap 2023-03-03T08:45:10.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:10.274 INFO:tasks.workunit.client.0.smithi007.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-03-03T08:45:10.285 INFO:tasks.workunit.client.0.smithi007.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-03-03T08:45:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.417+0000 7fc5bac62700 1 Processor -- start 2023-03-03T08:45:10.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.417+0000 7fc5bac62700 1 -- start start 2023-03-03T08:45:10.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.418+0000 7fc5bac62700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc5b40c65c0 0x7fc5b40c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:10.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.418+0000 7fc5bac62700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b40a65b0 con 0x7fc5b40a9af0 2023-03-03T08:45:10.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.418+0000 7fc5bac62700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b41541f0 con 0x7fc5b414de50 2023-03-03T08:45:10.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.418+0000 7fc5bac62700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b4154e00 con 0x7fc5b40c65c0 2023-03-03T08:45:10.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.418+0000 7fc5b91ff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc5b40a9af0 0x7fc5b40c2c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46878/0 (socket says 172.21.15.7:46878) 2023-03-03T08:45:10.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.418+0000 7fc5b91ff700 1 -- 172.21.15.7:0/1311227144 learned_addr learned my addr 172.21.15.7:0/1311227144 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:45:10.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.418+0000 7fc5b3fff700 1 --2- 172.21.15.7:0/1311227144 >> v2:172.21.15.143:3300/0 conn(0x7fc5b40c65c0 0x7fc5b40c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:10.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.418+0000 7fc5b37fe700 1 -- 172.21.15.7:0/1311227144 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445709547 0 0) 0x7fc5b41541f0 con 0x7fc5b414de50 2023-03-03T08:45:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.419+0000 7fc5b37fe700 1 -- 172.21.15.7:0/1311227144 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a40035b0 con 0x7fc5b414de50 2023-03-03T08:45:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.419+0000 7fc5b37fe700 1 -- 172.21.15.7:0/1311227144 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894031168 0 0) 0x7fc5b40a65b0 con 0x7fc5b40a9af0 2023-03-03T08:45:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.419+0000 7fc5b37fe700 1 -- 172.21.15.7:0/1311227144 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b41541f0 con 0x7fc5b40a9af0 2023-03-03T08:45:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.419+0000 7fc5b3fff700 1 -- 172.21.15.7:0/1311227144 >> v1:172.21.15.143:6789/0 conn(0x7fc5b414de50 legacy=0x7fc5b406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.419+0000 7fc5b3fff700 1 -- 172.21.15.7:0/1311227144 >> v1:172.21.15.7:6789/0 conn(0x7fc5b40a9af0 legacy=0x7fc5b40c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.419+0000 7fc5b3fff700 1 -- 172.21.15.7:0/1311227144 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b41544d0 con 0x7fc5b40c65c0 2023-03-03T08:45:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.419+0000 7fc5b3fff700 1 --2- 172.21.15.7:0/1311227144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b40c65c0 0x7fc5b40c2720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000a9e0 tx=0x7fc5a000acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de015db0df5a1c60 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.419+0000 7fc5b37fe700 1 -- 172.21.15.7:0/1311227144 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a000cb70 con 0x7fc5b40c65c0 2023-03-03T08:45:10.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.419+0000 7fc5b37fe700 1 -- 172.21.15.7:0/1311227144 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a000ccd0 con 0x7fc5b40c65c0 2023-03-03T08:45:10.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.419+0000 7fc5b37fe700 1 -- 172.21.15.7:0/1311227144 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0018620 con 0x7fc5b40c65c0 2023-03-03T08:45:10.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.420+0000 7fc5bac62700 1 -- 172.21.15.7:0/1311227144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b40c65c0 msgr2=0x7fc5b40c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:10.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.420+0000 7fc5bac62700 1 --2- 172.21.15.7:0/1311227144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b40c65c0 0x7fc5b40c2720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000a9e0 tx=0x7fc5a000acf0 comp rx=0 tx=0).stop 2023-03-03T08:45:10.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.420+0000 7fc5bac62700 1 -- 172.21.15.7:0/1311227144 shutdown_connections 2023-03-03T08:45:10.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.420+0000 7fc5bac62700 1 --2- 172.21.15.7:0/1311227144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b40c65c0 0x7fc5b40c2720 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:10.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.420+0000 7fc5bac62700 1 -- 172.21.15.7:0/1311227144 >> 172.21.15.7:0/1311227144 conn(0x7fc5b40a9430 msgr2=0x7fc5b40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:10.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.420+0000 7fc5bac62700 1 -- 172.21.15.7:0/1311227144 shutdown_connections 2023-03-03T08:45:10.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.420+0000 7fc5bac62700 1 -- 172.21.15.7:0/1311227144 wait complete. 2023-03-03T08:45:10.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.422+0000 7fc5bac62700 1 Processor -- start 2023-03-03T08:45:10.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.422+0000 7fc5bac62700 1 -- start start 2023-03-03T08:45:10.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5bac62700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b40a9af0 0x7fc5b4070a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:10.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5bac62700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40c65c0 0x7fc5b41ea220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:10.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5bac62700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b414de50 0x7fc5b4149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:10.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5bac62700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40c3ec0 con 0x7fc5b40a9af0 2023-03-03T08:45:10.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5bac62700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b41e9e00 con 0x7fc5b414de50 2023-03-03T08:45:10.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5bac62700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40c0be0 con 0x7fc5b40c65c0 2023-03-03T08:45:10.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b89fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b40a9af0 0x7fc5b4070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:10.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b89fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b40a9af0 0x7fc5b4070a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52900/0 (socket says 172.21.15.7:52900) 2023-03-03T08:45:10.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b89fe700 1 -- 172.21.15.7:0/576146221 learned_addr learned my addr 172.21.15.7:0/576146221 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:10.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b91ff700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b414de50 0x7fc5b4149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:10.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b3fff700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40c65c0 0x7fc5b41ea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:10.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b89fe700 1 -- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40c65c0 msgr2=0x7fc5b41ea220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:10.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b89fe700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40c65c0 0x7fc5b41ea220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:10.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b89fe700 1 -- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b414de50 msgr2=0x7fc5b4149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:10.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b89fe700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b414de50 0x7fc5b4149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:10.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b89fe700 1 -- 172.21.15.7:0/576146221 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a000a640 con 0x7fc5b40a9af0 2023-03-03T08:45:10.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b89fe700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b40a9af0 0x7fc5b4070a70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800c700 tx=0x7fc5a800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:10.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.423+0000 7fc5b17fa700 1 -- 172.21.15.7:0/576146221 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a80052d0 con 0x7fc5b40a9af0 2023-03-03T08:45:10.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.424+0000 7fc5b17fa700 1 -- 172.21.15.7:0/576146221 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a8005430 con 0x7fc5b40a9af0 2023-03-03T08:45:10.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.424+0000 7fc5b17fa700 1 -- 172.21.15.7:0/576146221 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a80038a0 con 0x7fc5b40a9af0 2023-03-03T08:45:10.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.424+0000 7fc5bac62700 1 -- 172.21.15.7:0/576146221 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b4154a90 con 0x7fc5b40a9af0 2023-03-03T08:45:10.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.424+0000 7fc5bac62700 1 -- 172.21.15.7:0/576146221 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b41547b0 con 0x7fc5b40a9af0 2023-03-03T08:45:10.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.424+0000 7fc5bac62700 1 -- 172.21.15.7:0/576146221 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b4153e80 con 0x7fc5b40a9af0 2023-03-03T08:45:10.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.425+0000 7fc5b17fa700 1 -- 172.21.15.7:0/576146221 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc5a8019030 con 0x7fc5b40a9af0 2023-03-03T08:45:10.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.426+0000 7fc5b17fa700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5980682f0 0x7fc59806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:10.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.426+0000 7fc5b17fa700 1 -- 172.21.15.7:0/576146221 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7018+0+0 (secure 0 0 0) 0x7fc5a80853c0 con 0x7fc5b40a9af0 2023-03-03T08:45:10.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.426+0000 7fc5b3fff700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5980682f0 0x7fc59806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:10.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.427+0000 7fc5b3fff700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5980682f0 0x7fc59806a7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc5b40c6a00 tx=0x7fc5a001a040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:10.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.433+0000 7fc5b17fa700 1 -- 172.21.15.7:0/576146221 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5a8051860 con 0x7fc5b40a9af0 2023-03-03T08:45:10.492 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'passed: files are identical' 2023-03-03T08:45:10.493 INFO:tasks.workunit.client.0.smithi007.stdout:passed: files are identical 2023-03-03T08:45:10.494 INFO:tasks.workunit.client.0.smithi007.stderr:+ rmdir foo/.snap/barsnap 2023-03-03T08:45:10.506 INFO:tasks.workunit.client.0.smithi007.stdout:OK 2023-03-03T08:45:10.507 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo OK 2023-03-03T08:45:10.508 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-03-03T08:45:10.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-03-03T08:45:10.577 INFO:tasks.ceph.mds.a.smithi007.stderr:2023-03-03T08:45:10.576+0000 7f5cfb0fb700 -1 mds.0.cache.den(0x601 10000000002) newly corrupt dentry to be committed: [dentry #0x100/stray1/10000000002 [4,head] auth NULL (dn xlock x=1 by 0x55f68ee64400) (dversion lock w=1 last_client=4633) pv=23 v=22 ap=2 ino=(nil) state=1342177281|new | request=1 lock=2 authpin=1 0x55f68edd1180] 2023-03-03T08:45:10.578 INFO:tasks.ceph.mds.a.smithi007.stderr:2023-03-03T08:45:10.576+0000 7f5cfb0fb700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x601is fatal because it is a system directory for this rank 2023-03-03T08:45:10.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.694+0000 7fc5bac62700 1 -- 172.21.15.7:0/576146221 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5b4151180 con 0x7fc5b40a9af0 2023-03-03T08:45:10.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.695+0000 7fc5b17fa700 1 -- 172.21.15.7:0/576146221 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5760 (secure 0 0 0) 0x7fc5a8054e80 con 0x7fc5b40a9af0 2023-03-03T08:45:10.696 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:10.697 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:43:38.528223+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.7:6835/1083851601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":1083851601},{"type":"v1","addr":"172.21.15.7:6835","nonce":1083851601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 -- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5980682f0 msgr2=0x7fc59806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5980682f0 0x7fc59806a7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc5b40c6a00 tx=0x7fc5a001a040 comp rx=0 tx=0).stop 2023-03-03T08:45:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 -- 172.21.15.7:0/576146221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b40a9af0 msgr2=0x7fc5b4070a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b40a9af0 0x7fc5b4070a70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800c700 tx=0x7fc5a800ca10 comp rx=0 tx=0).stop 2023-03-03T08:45:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 -- 172.21.15.7:0/576146221 shutdown_connections 2023-03-03T08:45:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b414de50 0x7fc5b4149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:10.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5980682f0 0x7fc59806a7a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b40a9af0 0x7fc5b4070a70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 --2- 172.21.15.7:0/576146221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b40c65c0 0x7fc5b41ea220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 -- 172.21.15.7:0/576146221 >> 172.21.15.7:0/576146221 conn(0x7fc5b40a9430 msgr2=0x7fc5b40a65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 -- 172.21.15.7:0/576146221 shutdown_connections 2023-03-03T08:45:10.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:10.697+0000 7fc5bac62700 1 -- 172.21.15.7:0/576146221 wait complete. 2023-03-03T08:45:10.701 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 19 2023-03-03T08:45:10.711 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-03-03T08:45:11.326 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T08:45:11.325+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-03-03T08:45:11.326 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T08:45:11.325+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2023-03-03T08:45:11.346 INFO:tasks.ceph.mds.a.smithi007.stderr: -80> 2023-03-03T08:45:10.576+0000 7f5cfb0fb700 -1 mds.0.cache.den(0x601 10000000002) newly corrupt dentry to be committed: [dentry #0x100/stray1/10000000002 [4,head] auth NULL (dn xlock x=1 by 0x55f68ee64400) (dversion lock w=1 last_client=4633) pv=23 v=22 ap=2 ino=(nil) state=1342177281|new | request=1 lock=2 authpin=1 0x55f68edd1180] 2023-03-03T08:45:11.347 INFO:tasks.ceph.mds.a.smithi007.stderr: -78> 2023-03-03T08:45:10.576+0000 7f5cfb0fb700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x601is fatal because it is a system directory for this rank 2023-03-03T08:45:11.373 INFO:tasks.ceph.mds.a.smithi007.stdout:starting mds.a at 2023-03-03T08:45:12.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:12.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.850+0000 7f5b693e1700 1 Processor -- start 2023-03-03T08:45:12.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.850+0000 7f5b693e1700 1 -- start start 2023-03-03T08:45:12.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.850+0000 7f5b693e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 0x7f5b640c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:12.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.850+0000 7f5b693e1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b640c7310 con 0x7f5b640caa30 2023-03-03T08:45:12.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.850+0000 7f5b693e1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b640bfd10 con 0x7f5b64154ff0 2023-03-03T08:45:12.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.850+0000 7f5b693e1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b640bfa30 con 0x7f5b640c00e0 2023-03-03T08:45:12.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b62ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5b640caa30 0x7f5b640c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46908/0 (socket says 172.21.15.7:46908) 2023-03-03T08:45:12.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b62ffd700 1 -- 172.21.15.7:0/3327814853 learned_addr learned my addr 172.21.15.7:0/3327814853 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:45:12.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b627fc700 1 --2- 172.21.15.7:0/3327814853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 0x7f5b640c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:12.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b61ffb700 1 -- 172.21.15.7:0/3327814853 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587867644 0 0) 0x7f5b640c7310 con 0x7f5b640caa30 2023-03-03T08:45:12.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b61ffb700 1 -- 172.21.15.7:0/3327814853 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b500035b0 con 0x7f5b640caa30 2023-03-03T08:45:12.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b61ffb700 1 -- 172.21.15.7:0/3327814853 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358237812 0 0) 0x7f5b640bfd10 con 0x7f5b64154ff0 2023-03-03T08:45:12.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b61ffb700 1 -- 172.21.15.7:0/3327814853 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b640c7310 con 0x7f5b64154ff0 2023-03-03T08:45:12.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b627fc700 1 -- 172.21.15.7:0/3327814853 >> v1:172.21.15.143:6789/0 conn(0x7f5b64154ff0 legacy=0x7f5b6415ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:12.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b627fc700 1 -- 172.21.15.7:0/3327814853 >> v1:172.21.15.7:6789/0 conn(0x7f5b640caa30 legacy=0x7f5b640c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:12.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b627fc700 1 -- 172.21.15.7:0/3327814853 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b64154510 con 0x7f5b640c00e0 2023-03-03T08:45:12.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b627fc700 1 --2- 172.21.15.7:0/3327814853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 0x7f5b640c5c60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009850 tx=0x7f5b4c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4444230819a084ef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:12.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.851+0000 7f5b61ffb700 1 -- 172.21.15.7:0/3327814853 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c005e30 con 0x7f5b640c00e0 2023-03-03T08:45:12.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.852+0000 7f5b61ffb700 1 -- 172.21.15.7:0/3327814853 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4c006030 con 0x7f5b640c00e0 2023-03-03T08:45:12.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.852+0000 7f5b61ffb700 1 -- 172.21.15.7:0/3327814853 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4c0063e0 con 0x7f5b640c00e0 2023-03-03T08:45:12.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.852+0000 7f5b693e1700 1 -- 172.21.15.7:0/3327814853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 msgr2=0x7f5b640c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:12.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.852+0000 7f5b693e1700 1 --2- 172.21.15.7:0/3327814853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 0x7f5b640c5c60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c009850 tx=0x7f5b4c009b60 comp rx=0 tx=0).stop 2023-03-03T08:45:12.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.852+0000 7f5b693e1700 1 -- 172.21.15.7:0/3327814853 shutdown_connections 2023-03-03T08:45:12.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.852+0000 7f5b693e1700 1 --2- 172.21.15.7:0/3327814853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 0x7f5b640c5c60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:12.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.852+0000 7f5b693e1700 1 -- 172.21.15.7:0/3327814853 >> 172.21.15.7:0/3327814853 conn(0x7f5b64066e40 msgr2=0x7f5b64071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:12.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.852+0000 7f5b693e1700 1 -- 172.21.15.7:0/3327814853 shutdown_connections 2023-03-03T08:45:12.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.852+0000 7f5b693e1700 1 -- 172.21.15.7:0/3327814853 wait complete. 2023-03-03T08:45:12.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.853+0000 7f5b693e1700 1 Processor -- start 2023-03-03T08:45:12.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.853+0000 7f5b693e1700 1 -- start start 2023-03-03T08:45:12.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b693e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 0x7f5b6414e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:12.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b693e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b640caa30 0x7f5b6414be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:12.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b693e1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b64154ff0 0x7f5b6414c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:12.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b693e1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b64157b70 con 0x7f5b640caa30 2023-03-03T08:45:12.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b693e1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b641eda70 con 0x7f5b64154ff0 2023-03-03T08:45:12.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b693e1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b641e9a40 con 0x7f5b640c00e0 2023-03-03T08:45:12.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b627fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b640caa30 0x7f5b6414be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:12.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b62ffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 0x7f5b6414e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:12.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b627fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b640caa30 0x7f5b6414be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52936/0 (socket says 172.21.15.7:52936) 2023-03-03T08:45:12.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b627fc700 1 -- 172.21.15.7:0/4165401384 learned_addr learned my addr 172.21.15.7:0/4165401384 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:12.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.854+0000 7f5b637fe700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b64154ff0 0x7f5b6414c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:12.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b62ffd700 1 -- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b64154ff0 msgr2=0x7f5b6414c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b62ffd700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b64154ff0 0x7f5b6414c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b62ffd700 1 -- 172.21.15.7:0/4165401384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b640caa30 msgr2=0x7f5b6414be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b62ffd700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b640caa30 0x7f5b6414be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b62ffd700 1 -- 172.21.15.7:0/4165401384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4c009430 con 0x7f5b640c00e0 2023-03-03T08:45:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b627fc700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b640caa30 0x7f5b6414be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:45:12.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b62ffd700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 0x7f5b6414e700 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400c4d0 tx=0x7f5b5400c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:12.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b5b7fe700 1 -- 172.21.15.7:0/4165401384 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54005d70 con 0x7f5b640c00e0 2023-03-03T08:45:12.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b5b7fe700 1 -- 172.21.15.7:0/4165401384 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b540070d0 con 0x7f5b640c00e0 2023-03-03T08:45:12.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b693e1700 1 -- 172.21.15.7:0/4165401384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b64154ad0 con 0x7f5b640c00e0 2023-03-03T08:45:12.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b5b7fe700 1 -- 172.21.15.7:0/4165401384 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54007440 con 0x7f5b640c00e0 2023-03-03T08:45:12.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.855+0000 7f5b693e1700 1 -- 172.21.15.7:0/4165401384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b641547f0 con 0x7f5b640c00e0 2023-03-03T08:45:12.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.856+0000 7f5b693e1700 1 -- 172.21.15.7:0/4165401384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b640bf650 con 0x7f5b640c00e0 2023-03-03T08:45:12.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.857+0000 7f5b5b7fe700 1 -- 172.21.15.7:0/4165401384 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5b54019030 con 0x7f5b640c00e0 2023-03-03T08:45:12.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.857+0000 7f5b5b7fe700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b440682f0 0x7f5b4406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:12.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.857+0000 7f5b5b7fe700 1 -- 172.21.15.7:0/4165401384 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7104+0+0 (secure 0 0 0) 0x7f5b54085270 con 0x7f5b640c00e0 2023-03-03T08:45:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.857+0000 7f5b627fc700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b440682f0 0x7f5b4406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.858+0000 7f5b627fc700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b440682f0 0x7f5b4406a7a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5b640af1a0 tx=0x7f5b4c008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:12.862+0000 7f5b5b7fe700 1 -- 172.21.15.7:0/4165401384 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5b5404b5e0 con 0x7f5b640c00e0 2023-03-03T08:45:13.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.097+0000 7f5b693e1700 1 -- 172.21.15.7:0/4165401384 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b641ea370 con 0x7f5b640c00e0 2023-03-03T08:45:13.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.098+0000 7f5b5b7fe700 1 -- 172.21.15.7:0/4165401384 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5754 (secure 0 0 0) 0x7f5b540517b0 con 0x7f5b640c00e0 2023-03-03T08:45:13.100 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:13.101 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5121,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2005962882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":2005962882},{"type":"v1","addr":"172.21.15.7:6835","nonce":2005962882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:13.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 -- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b440682f0 msgr2=0x7f5b4406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:13.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b440682f0 0x7f5b4406a7a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5b640af1a0 tx=0x7f5b4c008070 comp rx=0 tx=0).stop 2023-03-03T08:45:13.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 -- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 msgr2=0x7f5b6414e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:13.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 0x7f5b6414e700 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400c4d0 tx=0x7f5b5400c7e0 comp rx=0 tx=0).stop 2023-03-03T08:45:13.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 -- 172.21.15.7:0/4165401384 shutdown_connections 2023-03-03T08:45:13.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b64154ff0 0x7f5b6414c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:13.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b640c00e0 0x7f5b6414e700 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b440682f0 0x7f5b4406a7a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 --2- 172.21.15.7:0/4165401384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b640caa30 0x7f5b6414be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:13.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 -- 172.21.15.7:0/4165401384 >> 172.21.15.7:0/4165401384 conn(0x7f5b64066e40 msgr2=0x7f5b640c9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:13.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 -- 172.21.15.7:0/4165401384 shutdown_connections 2023-03-03T08:45:13.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:13.100+0000 7f5b693e1700 1 -- 172.21.15.7:0/4165401384 wait complete. 2023-03-03T08:45:13.106 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2023-03-03T08:45:13.114 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-03-03T08:45:15.115 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:15.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.253+0000 7fa70ff44700 1 Processor -- start 2023-03-03T08:45:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.253+0000 7fa70ff44700 1 -- start start 2023-03-03T08:45:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70ff44700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa708152a20 0x7fa7080ba400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70ff44700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7080ba940 0x7fa70815c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70ff44700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70806c420 con 0x7fa7080c1d70 2023-03-03T08:45:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70ff44700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa70806c9e0 con 0x7fa708152a20 2023-03-03T08:45:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70ff44700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa708152df0 con 0x7fa7080ba940 2023-03-03T08:45:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70dce0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa708152a20 0x7fa7080ba400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70dce0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa708152a20 0x7fa7080ba400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52942/0 (socket says 172.21.15.7:52942) 2023-03-03T08:45:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70dce0700 1 -- 172.21.15.7:0/924481430 learned_addr learned my addr 172.21.15.7:0/924481430 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70e4e1700 1 --2- 172.21.15.7:0/924481430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7080ba940 0x7fa70815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70dce0700 1 -- 172.21.15.7:0/924481430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7080ba940 msgr2=0x7fa70815c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70dce0700 1 --2- 172.21.15.7:0/924481430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7080ba940 0x7fa70815c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70dce0700 1 -- 172.21.15.7:0/924481430 >> v1:172.21.15.143:6789/0 conn(0x7fa7080c1d70 legacy=0x7fa70815c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.254+0000 7fa70dce0700 1 -- 172.21.15.7:0/924481430 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa708159450 con 0x7fa708152a20 2023-03-03T08:45:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70dce0700 1 --2- 172.21.15.7:0/924481430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa708152a20 0x7fa7080ba400 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80098f0 tx=0x7fa6f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d27ac8b3ac3a262 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ccde700 1 -- 172.21.15.7:0/924481430 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800b8a0 con 0x7fa708152a20 2023-03-03T08:45:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ccde700 1 -- 172.21.15.7:0/924481430 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f800ba00 con 0x7fa708152a20 2023-03-03T08:45:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ccde700 1 -- 172.21.15.7:0/924481430 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f800bd90 con 0x7fa708152a20 2023-03-03T08:45:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ff44700 1 -- 172.21.15.7:0/924481430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa708152a20 msgr2=0x7fa7080ba400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ff44700 1 --2- 172.21.15.7:0/924481430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa708152a20 0x7fa7080ba400 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80098f0 tx=0x7fa6f8009c00 comp rx=0 tx=0).stop 2023-03-03T08:45:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ff44700 1 -- 172.21.15.7:0/924481430 shutdown_connections 2023-03-03T08:45:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ff44700 1 --2- 172.21.15.7:0/924481430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7080ba940 0x7fa70815c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ff44700 1 --2- 172.21.15.7:0/924481430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa708152a20 0x7fa7080ba400 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ff44700 1 -- 172.21.15.7:0/924481430 >> 172.21.15.7:0/924481430 conn(0x7fa7080a94d0 msgr2=0x7fa7080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ff44700 1 -- 172.21.15.7:0/924481430 shutdown_connections 2023-03-03T08:45:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.255+0000 7fa70ff44700 1 -- 172.21.15.7:0/924481430 wait complete. 2023-03-03T08:45:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70ff44700 1 Processor -- start 2023-03-03T08:45:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70ff44700 1 -- start start 2023-03-03T08:45:15.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70ff44700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7080ba940 0x7fa7081e58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:15.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70ff44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7080c1d70 0x7fa7081f1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:15.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70ff44700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa708152a20 0x7fa7081f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:15.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70ff44700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70814b0d0 con 0x7fa7080c1d70 2023-03-03T08:45:15.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70ff44700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa708157fa0 con 0x7fa708152a20 2023-03-03T08:45:15.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70ff44700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70814b320 con 0x7fa7080ba940 2023-03-03T08:45:15.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70d4df700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7080c1d70 0x7fa7081f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:15.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70d4df700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7080c1d70 0x7fa7081f1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52950/0 (socket says 172.21.15.7:52950) 2023-03-03T08:45:15.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.257+0000 7fa70d4df700 1 -- 172.21.15.7:0/2798919116 learned_addr learned my addr 172.21.15.7:0/2798919116 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:15.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa70e4e1700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa708152a20 0x7fa7081f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:15.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa70dce0700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7080ba940 0x7fa7081e58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:15.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa70d4df700 1 -- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7080ba940 msgr2=0x7fa7081e58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:15.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa70d4df700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7080ba940 0x7fa7081e58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:15.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa70d4df700 1 -- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa708152a20 msgr2=0x7fa7081f1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:15.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa70d4df700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa708152a20 0x7fa7081f1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:15.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa70d4df700 1 -- 172.21.15.7:0/2798919116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f8009550 con 0x7fa7080c1d70 2023-03-03T08:45:15.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa70d4df700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7080c1d70 0x7fa7081f1740 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa70400c4e0 tx=0x7fa70400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:15.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa6fe7fc700 1 -- 172.21.15.7:0/2798919116 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7040052d0 con 0x7fa7080c1d70 2023-03-03T08:45:15.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa6fe7fc700 1 -- 172.21.15.7:0/2798919116 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa704005430 con 0x7fa7080c1d70 2023-03-03T08:45:15.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa70ff44700 1 -- 172.21.15.7:0/2798919116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70806c670 con 0x7fa7080c1d70 2023-03-03T08:45:15.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa6fe7fc700 1 -- 172.21.15.7:0/2798919116 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7040038c0 con 0x7fa7080c1d70 2023-03-03T08:45:15.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.258+0000 7fa70ff44700 1 -- 172.21.15.7:0/2798919116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa708159730 con 0x7fa7080c1d70 2023-03-03T08:45:15.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.259+0000 7fa70ff44700 1 -- 172.21.15.7:0/2798919116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70815cdb0 con 0x7fa7080c1d70 2023-03-03T08:45:15.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.260+0000 7fa6fe7fc700 1 -- 172.21.15.7:0/2798919116 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa704019030 con 0x7fa7080c1d70 2023-03-03T08:45:15.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.260+0000 7fa6fe7fc700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6ec0682f0 0x7fa6ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:15.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.260+0000 7fa6fe7fc700 1 -- 172.21.15.7:0/2798919116 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7104+0+0 (secure 0 0 0) 0x7fa704085300 con 0x7fa7080c1d70 2023-03-03T08:45:15.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.260+0000 7fa70dce0700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6ec0682f0 0x7fa6ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:15.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.261+0000 7fa70dce0700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6ec0682f0 0x7fa6ec06a7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009c40 tx=0x7fa6f8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:15.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.264+0000 7fa6fe7fc700 1 -- 172.21.15.7:0/2798919116 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa704051750 con 0x7fa7080c1d70 2023-03-03T08:45:15.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.501+0000 7fa70ff44700 1 -- 172.21.15.7:0/2798919116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa708074bb0 con 0x7fa7080c1d70 2023-03-03T08:45:15.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.501+0000 7fa6fe7fc700 1 -- 172.21.15.7:0/2798919116 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5754 (secure 0 0 0) 0x7fa704054d70 con 0x7fa7080c1d70 2023-03-03T08:45:15.503 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:15.503 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5121,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2005962882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":2005962882},{"type":"v1","addr":"172.21.15.7:6835","nonce":2005962882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 -- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6ec0682f0 msgr2=0x7fa6ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6ec0682f0 0x7fa6ec06a7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009c40 tx=0x7fa6f8008030 comp rx=0 tx=0).stop 2023-03-03T08:45:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 -- 172.21.15.7:0/2798919116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7080c1d70 msgr2=0x7fa7081f1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7080c1d70 0x7fa7081f1740 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa70400c4e0 tx=0x7fa70400c7f0 comp rx=0 tx=0).stop 2023-03-03T08:45:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 -- 172.21.15.7:0/2798919116 shutdown_connections 2023-03-03T08:45:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa708152a20 0x7fa7081f1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:15.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7080ba940 0x7fa7081e58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:15.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6ec0682f0 0x7fa6ec06a7a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:15.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 --2- 172.21.15.7:0/2798919116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7080c1d70 0x7fa7081f1740 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:15.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 -- 172.21.15.7:0/2798919116 >> 172.21.15.7:0/2798919116 conn(0x7fa7080a94d0 msgr2=0x7fa7081513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:15.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 -- 172.21.15.7:0/2798919116 shutdown_connections 2023-03-03T08:45:15.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:15.503+0000 7fa70ff44700 1 -- 172.21.15.7:0/2798919116 wait complete. 2023-03-03T08:45:15.508 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2023-03-03T08:45:15.517 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-03-03T08:45:17.518 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:17.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.656+0000 7f0f6ce48700 1 Processor -- start 2023-03-03T08:45:17.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.656+0000 7f0f6ce48700 1 -- start start 2023-03-03T08:45:17.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6ce48700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f68155390 0x7f0f680c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6ce48700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0f6806e020 0x7f0f68071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6ce48700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f68072520 0x7f0f68151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6ce48700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f68070b90 con 0x7f0f68155390 2023-03-03T08:45:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6ce48700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f68151320 con 0x7f0f6806e020 2023-03-03T08:45:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6ce48700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f680c4780 con 0x7f0f68072520 2023-03-03T08:45:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f68155390 0x7f0f680c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f68155390 0x7f0f680c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52956/0 (socket says 172.21.15.7:52956) 2023-03-03T08:45:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6659c700 1 -- 172.21.15.7:0/3656417701 learned_addr learned my addr 172.21.15.7:0/3656417701 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f65d9b700 1 --2- 172.21.15.7:0/3656417701 >> v2:172.21.15.143:3300/0 conn(0x7f0f6806e020 0x7f0f68071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f66d9d700 1 --2- 172.21.15.7:0/3656417701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f68072520 0x7f0f68151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6659c700 1 -- 172.21.15.7:0/3656417701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f68072520 msgr2=0x7f0f68151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6659c700 1 --2- 172.21.15.7:0/3656417701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f68072520 0x7f0f68151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6659c700 1 -- 172.21.15.7:0/3656417701 >> v2:172.21.15.143:3300/0 conn(0x7f0f6806e020 msgr2=0x7f0f68071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6659c700 1 --2- 172.21.15.7:0/3656417701 >> v2:172.21.15.143:3300/0 conn(0x7f0f6806e020 0x7f0f68071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6659c700 1 -- 172.21.15.7:0/3656417701 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f68154910 con 0x7f0f68155390 2023-03-03T08:45:17.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f66d9d700 1 --2- 172.21.15.7:0/3656417701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f68072520 0x7f0f68151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:45:17.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f65d9b700 1 --2- 172.21.15.7:0/3656417701 >> v2:172.21.15.143:3300/0 conn(0x7f0f6806e020 0x7f0f68071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:45:17.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.657+0000 7f0f6659c700 1 --2- 172.21.15.7:0/3656417701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68155390 0x7f0f680c3e30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0f680ac9e0 tx=0x7f0f5800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db0ab63dcc813ca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:17.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6559a700 1 -- 172.21.15.7:0/3656417701 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5800dd10 con 0x7f0f68155390 2023-03-03T08:45:17.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6559a700 1 -- 172.21.15.7:0/3656417701 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5800de70 con 0x7f0f68155390 2023-03-03T08:45:17.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6559a700 1 -- 172.21.15.7:0/3656417701 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f58010670 con 0x7f0f68155390 2023-03-03T08:45:17.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6ce48700 1 -- 172.21.15.7:0/3656417701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68155390 msgr2=0x7f0f680c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6ce48700 1 --2- 172.21.15.7:0/3656417701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68155390 0x7f0f680c3e30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0f680ac9e0 tx=0x7f0f5800bea0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6ce48700 1 -- 172.21.15.7:0/3656417701 shutdown_connections 2023-03-03T08:45:17.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6ce48700 1 --2- 172.21.15.7:0/3656417701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f68072520 0x7f0f68151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6ce48700 1 --2- 172.21.15.7:0/3656417701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68155390 0x7f0f680c3e30 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6ce48700 1 --2- 172.21.15.7:0/3656417701 >> v2:172.21.15.143:3300/0 conn(0x7f0f6806e020 0x7f0f68071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6ce48700 1 -- 172.21.15.7:0/3656417701 >> 172.21.15.7:0/3656417701 conn(0x7f0f6806c110 msgr2=0x7f0f6806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:17.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6ce48700 1 -- 172.21.15.7:0/3656417701 shutdown_connections 2023-03-03T08:45:17.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.658+0000 7f0f6ce48700 1 -- 172.21.15.7:0/3656417701 wait complete. 2023-03-03T08:45:17.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.660+0000 7f0f6ce48700 1 Processor -- start 2023-03-03T08:45:17.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.660+0000 7f0f6ce48700 1 -- start start 2023-03-03T08:45:17.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.660+0000 7f0f6ce48700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6806e020 0x7f0f68152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.660+0000 7f0f6ce48700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68072520 0x7f0f68154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.660+0000 7f0f6ce48700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f68155390 0x7f0f680c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.660+0000 7f0f6ce48700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f681ed6b0 con 0x7f0f68072520 2023-03-03T08:45:17.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.660+0000 7f0f6ce48700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f681e96b0 con 0x7f0f68155390 2023-03-03T08:45:17.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.660+0000 7f0f6ce48700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f681edbe0 con 0x7f0f6806e020 2023-03-03T08:45:17.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f65d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68072520 0x7f0f68154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f65d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68072520 0x7f0f68154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52964/0 (socket says 172.21.15.7:52964) 2023-03-03T08:45:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f65d9b700 1 -- 172.21.15.7:0/1668405185 learned_addr learned my addr 172.21.15.7:0/1668405185 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f66d9d700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f68155390 0x7f0f680c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f6659c700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6806e020 0x7f0f68152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f65d9b700 1 -- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6806e020 msgr2=0x7f0f68152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f65d9b700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6806e020 0x7f0f68152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f65d9b700 1 -- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f68155390 msgr2=0x7f0f680c0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f65d9b700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f68155390 0x7f0f680c0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f65d9b700 1 -- 172.21.15.7:0/1668405185 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5800b7f0 con 0x7f0f68072520 2023-03-03T08:45:17.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f6659c700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6806e020 0x7f0f68152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:45:17.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f65d9b700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68072520 0x7f0f68154260 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00d800 tx=0x7f0f5c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:17.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f56ffd700 1 -- 172.21.15.7:0/1668405185 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c004030 con 0x7f0f68072520 2023-03-03T08:45:17.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f56ffd700 1 -- 172.21.15.7:0/1668405185 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f5c004190 con 0x7f0f68072520 2023-03-03T08:45:17.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f6ce48700 1 -- 172.21.15.7:0/1668405185 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f68070540 con 0x7f0f68072520 2023-03-03T08:45:17.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f56ffd700 1 -- 172.21.15.7:0/1668405185 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c004520 con 0x7f0f68072520 2023-03-03T08:45:17.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.661+0000 7f0f6ce48700 1 -- 172.21.15.7:0/1668405185 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f680702a0 con 0x7f0f68072520 2023-03-03T08:45:17.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.662+0000 7f0f6ce48700 1 -- 172.21.15.7:0/1668405185 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f68070820 con 0x7f0f68072520 2023-03-03T08:45:17.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.663+0000 7f0f56ffd700 1 -- 172.21.15.7:0/1668405185 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0f5c01a020 con 0x7f0f68072520 2023-03-03T08:45:17.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.663+0000 7f0f56ffd700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f480682f0 0x7f0f4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:17.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.663+0000 7f0f56ffd700 1 -- 172.21.15.7:0/1668405185 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7104+0+0 (secure 0 0 0) 0x7f0f5c086490 con 0x7f0f68072520 2023-03-03T08:45:17.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.663+0000 7f0f6659c700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f480682f0 0x7f0f4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:17.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.664+0000 7f0f6659c700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f480682f0 0x7f0f4806a7a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0f5800bee0 tx=0x7f0f5801b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:17.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.667+0000 7f0f56ffd700 1 -- 172.21.15.7:0/1668405185 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0f5c052880 con 0x7f0f68072520 2023-03-03T08:45:17.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.903+0000 7f0f6ce48700 1 -- 172.21.15.7:0/1668405185 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f68074bb0 con 0x7f0f68072520 2023-03-03T08:45:17.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.903+0000 7f0f56ffd700 1 -- 172.21.15.7:0/1668405185 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5754 (secure 0 0 0) 0x7f0f5c055ea0 con 0x7f0f68072520 2023-03-03T08:45:17.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:17.905 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5121,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2005962882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":2005962882},{"type":"v1","addr":"172.21.15.7:6835","nonce":2005962882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:17.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 -- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f480682f0 msgr2=0x7f0f4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f480682f0 0x7f0f4806a7a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0f5800bee0 tx=0x7f0f5801b040 comp rx=0 tx=0).stop 2023-03-03T08:45:17.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 -- 172.21.15.7:0/1668405185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68072520 msgr2=0x7f0f68154260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:17.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68072520 0x7f0f68154260 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00d800 tx=0x7f0f5c00dbc0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 -- 172.21.15.7:0/1668405185 shutdown_connections 2023-03-03T08:45:17.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f68155390 0x7f0f680c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6806e020 0x7f0f68152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f480682f0 0x7f0f4806a7a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 --2- 172.21.15.7:0/1668405185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f68072520 0x7f0f68154260 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:17.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 -- 172.21.15.7:0/1668405185 >> 172.21.15.7:0/1668405185 conn(0x7f0f6806c110 msgr2=0x7f0f680aca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:17.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 -- 172.21.15.7:0/1668405185 shutdown_connections 2023-03-03T08:45:17.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:17.906+0000 7f0f6ce48700 1 -- 172.21.15.7:0/1668405185 wait complete. 2023-03-03T08:45:17.909 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2023-03-03T08:45:17.919 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-03-03T08:45:19.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:20.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.060+0000 7f9ebd20c700 1 Processor -- start 2023-03-03T08:45:20.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.060+0000 7f9ebd20c700 1 -- start start 2023-03-03T08:45:20.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.061+0000 7f9ebd20c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 0x7f9eb80c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:20.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.061+0000 7f9ebd20c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9eb80c4fc0 0x7f9eb806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:20.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.061+0000 7f9ebd20c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb814ebe0 con 0x7f9eb80c17f0 2023-03-03T08:45:20.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.061+0000 7f9ebd20c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb8152100 con 0x7f9eb80c4fc0 2023-03-03T08:45:20.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.061+0000 7f9ebd20c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb80c2e70 con 0x7f9eb80c6010 2023-03-03T08:45:20.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 0x7f9eb80c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:20.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9eb80c4fc0 0x7f9eb806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:20.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb759e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9eb80c4fc0 0x7f9eb806ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:48750/0 (socket says 172.21.15.7:48750) 2023-03-03T08:45:20.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 0x7f9eb80c4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:32812/0 (socket says 172.21.15.7:32812) 2023-03-03T08:45:20.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb759e700 1 -- 172.21.15.7:0/1936985450 learned_addr learned my addr 172.21.15.7:0/1936985450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:20.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb659c700 1 --1- 172.21.15.7:0/1936985450 >> v1:172.21.15.143:6789/0 conn(0x7f9eb80c17f0 0x7f9eb806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50474/0 (socket says 172.21.15.7:50474) 2023-03-03T08:45:20.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb759e700 1 -- 172.21.15.7:0/1936985450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 msgr2=0x7f9eb80c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:20.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb759e700 1 --2- 172.21.15.7:0/1936985450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 0x7f9eb80c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:20.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb759e700 1 -- 172.21.15.7:0/1936985450 >> v1:172.21.15.143:6789/0 conn(0x7f9eb80c17f0 legacy=0x7f9eb806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:20.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb759e700 1 -- 172.21.15.7:0/1936985450 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb80aa680 con 0x7f9eb80c4fc0 2023-03-03T08:45:20.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb6d9d700 1 --2- 172.21.15.7:0/1936985450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 0x7f9eb80c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:45:20.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb759e700 1 --2- 172.21.15.7:0/1936985450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eb80c4fc0 0x7f9eb806ddb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009e50 tx=0x7f9eb000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5418cea900f0254 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:20.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb5d9b700 1 -- 172.21.15.7:0/1936985450 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb000e070 con 0x7f9eb80c4fc0 2023-03-03T08:45:20.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb5d9b700 1 -- 172.21.15.7:0/1936985450 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb000b910 con 0x7f9eb80c4fc0 2023-03-03T08:45:20.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.062+0000 7f9eb5d9b700 1 -- 172.21.15.7:0/1936985450 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb000bcc0 con 0x7f9eb80c4fc0 2023-03-03T08:45:20.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.063+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1936985450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eb80c4fc0 msgr2=0x7f9eb806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:20.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.063+0000 7f9ebd20c700 1 --2- 172.21.15.7:0/1936985450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eb80c4fc0 0x7f9eb806ddb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009e50 tx=0x7f9eb000b330 comp rx=0 tx=0).stop 2023-03-03T08:45:20.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.063+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1936985450 shutdown_connections 2023-03-03T08:45:20.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.063+0000 7f9ebd20c700 1 --2- 172.21.15.7:0/1936985450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eb80c4fc0 0x7f9eb806ddb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:20.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.063+0000 7f9ebd20c700 1 --2- 172.21.15.7:0/1936985450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 0x7f9eb80c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:20.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.063+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1936985450 >> 172.21.15.7:0/1936985450 conn(0x7f9eb80a9430 msgr2=0x7f9eb80c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:20.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.063+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1936985450 shutdown_connections 2023-03-03T08:45:20.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.063+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1936985450 wait complete. 2023-03-03T08:45:20.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.064+0000 7f9ebd20c700 1 Processor -- start 2023-03-03T08:45:20.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.064+0000 7f9ebd20c700 1 -- start start 2023-03-03T08:45:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9ebd20c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9eb80c17f0 0x7f9eb81ed7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9ebd20c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eb80c4fc0 0x7f9eb80c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9ebd20c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 0x7f9eb81e9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9ebd20c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb81548c0 con 0x7f9eb80c17f0 2023-03-03T08:45:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9ebd20c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb81edd30 con 0x7f9eb80c4fc0 2023-03-03T08:45:20.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9ebd20c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb8072190 con 0x7f9eb80c6010 2023-03-03T08:45:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9eb6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9eb80c17f0 0x7f9eb81ed7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9eb6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9eb80c17f0 0x7f9eb81ed7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52978/0 (socket says 172.21.15.7:52978) 2023-03-03T08:45:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9eb6d9d700 1 -- 172.21.15.7:0/1123414803 learned_addr learned my addr 172.21.15.7:0/1123414803 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9eb659c700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eb80c4fc0 0x7f9eb80c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:20.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.065+0000 7f9eb759e700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 0x7f9eb81e9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9eb6d9d700 1 -- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 msgr2=0x7f9eb81e9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9eb6d9d700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 0x7f9eb81e9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9eb6d9d700 1 -- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eb80c4fc0 msgr2=0x7f9eb80c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9eb6d9d700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eb80c4fc0 0x7f9eb80c00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9eb6d9d700 1 -- 172.21.15.7:0/1123414803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea800b740 con 0x7f9eb80c17f0 2023-03-03T08:45:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9eb6d9d700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9eb80c17f0 0x7f9eb81ed7f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800ed40 tx=0x7f9ea800d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:20.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9eaf7fe700 1 -- 172.21.15.7:0/1123414803 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea800dd90 con 0x7f9eb80c17f0 2023-03-03T08:45:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9eaf7fe700 1 -- 172.21.15.7:0/1123414803 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea8010440 con 0x7f9eb80c17f0 2023-03-03T08:45:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1123414803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb0009b30 con 0x7f9eb80c17f0 2023-03-03T08:45:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9eaf7fe700 1 -- 172.21.15.7:0/1123414803 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea80107b0 con 0x7f9eb80c17f0 2023-03-03T08:45:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.066+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1123414803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb80c2b00 con 0x7f9eb80c17f0 2023-03-03T08:45:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.067+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1123414803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb80aa960 con 0x7f9eb80c17f0 2023-03-03T08:45:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.067+0000 7f9eaf7fe700 1 -- 172.21.15.7:0/1123414803 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ea8010910 con 0x7f9eb80c17f0 2023-03-03T08:45:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.068+0000 7f9eaf7fe700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e980682f0 0x7f9e9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:20.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.068+0000 7f9eaf7fe700 1 -- 172.21.15.7:0/1123414803 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7104+0+0 (secure 0 0 0) 0x7f9ea80869c0 con 0x7f9eb80c17f0 2023-03-03T08:45:20.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.068+0000 7f9eb659c700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e980682f0 0x7f9e9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:20.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.069+0000 7f9eb659c700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e980682f0 0x7f9e9806a7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80c0ac0 tx=0x7f9ea0007ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:20.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.072+0000 7f9eaf7fe700 1 -- 172.21.15.7:0/1123414803 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ea8052e10 con 0x7f9eb80c17f0 2023-03-03T08:45:20.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.309+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1123414803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9eb80c5420 con 0x7f9eb80c17f0 2023-03-03T08:45:20.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.309+0000 7f9eaf7fe700 1 -- 172.21.15.7:0/1123414803 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5754 (secure 0 0 0) 0x7f9ea8057e60 con 0x7f9eb80c17f0 2023-03-03T08:45:20.311 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:20.311 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5121,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2005962882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":2005962882},{"type":"v1","addr":"172.21.15.7:6835","nonce":2005962882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:20.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e980682f0 msgr2=0x7f9e9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:20.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e980682f0 0x7f9e9806a7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80c0ac0 tx=0x7f9ea0007ed0 comp rx=0 tx=0).stop 2023-03-03T08:45:20.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1123414803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9eb80c17f0 msgr2=0x7f9eb81ed7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:20.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9eb80c17f0 0x7f9eb81ed7f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800ed40 tx=0x7f9ea800d630 comp rx=0 tx=0).stop 2023-03-03T08:45:20.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1123414803 shutdown_connections 2023-03-03T08:45:20.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9eb80c6010 0x7f9eb81e9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:20.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e980682f0 0x7f9e9806a7a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:20.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9eb80c17f0 0x7f9eb81ed7f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:20.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 --2- 172.21.15.7:0/1123414803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9eb80c4fc0 0x7f9eb80c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:20.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1123414803 >> 172.21.15.7:0/1123414803 conn(0x7f9eb80a9430 msgr2=0x7f9eb814f490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:20.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1123414803 shutdown_connections 2023-03-03T08:45:20.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:20.311+0000 7f9ebd20c700 1 -- 172.21.15.7:0/1123414803 wait complete. 2023-03-03T08:45:20.315 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2023-03-03T08:45:20.325 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-03-03T08:45:22.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:22.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.485+0000 7f2d727f8700 1 Processor -- start 2023-03-03T08:45:22.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.486+0000 7f2d727f8700 1 -- start start 2023-03-03T08:45:22.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.486+0000 7f2d727f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d6c14b0d0 0x7f2d6c071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:22.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.486+0000 7f2d727f8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2d6c0aad20 0x7f2d6c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:22.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.486+0000 7f2d727f8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6c0a9c20 con 0x7f2d6c071d40 2023-03-03T08:45:22.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.486+0000 7f2d727f8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d6c154f10 con 0x7f2d6c0aad20 2023-03-03T08:45:22.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.486+0000 7f2d727f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c0ac3c0 con 0x7f2d6c14b0d0 2023-03-03T08:45:22.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.486+0000 7f2d6b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2d6c0aad20 0x7f2d6c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.487+0000 7f2d6b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2d6c0aad20 0x7f2d6c157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52988/0 (socket says 172.21.15.7:52988) 2023-03-03T08:45:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.487+0000 7f2d6b7fe700 1 -- 172.21.15.7:0/2598573333 learned_addr learned my addr 172.21.15.7:0/2598573333 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.487+0000 7f2d6b7fe700 1 -- 172.21.15.7:0/2598573333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d6c14b0d0 msgr2=0x7f2d6c071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.487+0000 7f2d6b7fe700 1 --2- 172.21.15.7:0/2598573333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d6c14b0d0 0x7f2d6c071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.487+0000 7f2d6b7fe700 1 -- 172.21.15.7:0/2598573333 >> v1:172.21.15.143:6789/0 conn(0x7f2d6c071d40 legacy=0x7f2d6c158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.487+0000 7f2d6b7fe700 1 -- 172.21.15.7:0/2598573333 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d6c06fc70 con 0x7f2d6c0aad20 2023-03-03T08:45:22.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.487+0000 7f2d6b7fe700 1 --2- 172.21.15.7:0/2598573333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 0x7f2d6c157df0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2d600098f0 tx=0x7f2d60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da7567a939ff2412 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.487+0000 7f2d6affd700 1 -- 172.21.15.7:0/2598573333 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60004230 con 0x7f2d6c0aad20 2023-03-03T08:45:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.487+0000 7f2d6affd700 1 -- 172.21.15.7:0/2598573333 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d60004390 con 0x7f2d6c0aad20 2023-03-03T08:45:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.487+0000 7f2d6affd700 1 -- 172.21.15.7:0/2598573333 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d600036a0 con 0x7f2d6c0aad20 2023-03-03T08:45:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.488+0000 7f2d727f8700 1 -- 172.21.15.7:0/2598573333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 msgr2=0x7f2d6c157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:22.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.488+0000 7f2d727f8700 1 --2- 172.21.15.7:0/2598573333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 0x7f2d6c157df0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2d600098f0 tx=0x7f2d60009c00 comp rx=0 tx=0).stop 2023-03-03T08:45:22.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.488+0000 7f2d727f8700 1 -- 172.21.15.7:0/2598573333 shutdown_connections 2023-03-03T08:45:22.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.488+0000 7f2d727f8700 1 --2- 172.21.15.7:0/2598573333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d6c14b0d0 0x7f2d6c071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:22.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.488+0000 7f2d727f8700 1 --2- 172.21.15.7:0/2598573333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 0x7f2d6c157df0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:22.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.488+0000 7f2d727f8700 1 -- 172.21.15.7:0/2598573333 >> 172.21.15.7:0/2598573333 conn(0x7f2d6c0c8900 msgr2=0x7f2d6c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:22.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.488+0000 7f2d727f8700 1 -- 172.21.15.7:0/2598573333 shutdown_connections 2023-03-03T08:45:22.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.488+0000 7f2d727f8700 1 -- 172.21.15.7:0/2598573333 wait complete. 2023-03-03T08:45:22.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.489+0000 7f2d727f8700 1 Processor -- start 2023-03-03T08:45:22.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.489+0000 7f2d727f8700 1 -- start start 2023-03-03T08:45:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.490+0000 7f2d727f8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d6c071d40 0x7f2d6c1ed1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.490+0000 7f2d727f8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 0x7f2d6c1440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.490+0000 7f2d727f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d6c14b0d0 0x7f2d6c144610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.490+0000 7f2d727f8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c147e40 con 0x7f2d6c0aad20 2023-03-03T08:45:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.490+0000 7f2d727f8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c1edea0 con 0x7f2d6c071d40 2023-03-03T08:45:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.490+0000 7f2d727f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c152300 con 0x7f2d6c14b0d0 2023-03-03T08:45:22.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.490+0000 7f2d6b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 0x7f2d6c1440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:22.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.490+0000 7f2d6b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 0x7f2d6c1440d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52994/0 (socket says 172.21.15.7:52994) 2023-03-03T08:45:22.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.490+0000 7f2d6b7fe700 1 -- 172.21.15.7:0/4188477586 learned_addr learned my addr 172.21.15.7:0/4188477586 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:22.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d6bfff700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d6c071d40 0x7f2d6c1ed1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:22.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d6b7fe700 1 -- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d6c14b0d0 msgr2=0x7f2d6c144610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:22.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d6b7fe700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d6c14b0d0 0x7f2d6c144610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:22.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d6b7fe700 1 -- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d6c071d40 msgr2=0x7f2d6c1ed1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:22.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d6b7fe700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d6c071d40 0x7f2d6c1ed1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d6b7fe700 1 -- 172.21.15.7:0/4188477586 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d60009550 con 0x7f2d6c0aad20 2023-03-03T08:45:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d6b7fe700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 0x7f2d6c1440d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000b0e0 tx=0x7f2d60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d68ff9700 1 -- 172.21.15.7:0/4188477586 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d60003e30 con 0x7f2d6c0aad20 2023-03-03T08:45:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d68ff9700 1 -- 172.21.15.7:0/4188477586 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6001f070 con 0x7f2d6c0aad20 2023-03-03T08:45:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d727f8700 1 -- 172.21.15.7:0/4188477586 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6c06f6b0 con 0x7f2d6c0aad20 2023-03-03T08:45:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d68ff9700 1 -- 172.21.15.7:0/4188477586 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6000b8a0 con 0x7f2d6c0aad20 2023-03-03T08:45:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.491+0000 7f2d727f8700 1 -- 172.21.15.7:0/4188477586 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6c155160 con 0x7f2d6c0aad20 2023-03-03T08:45:22.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.492+0000 7f2d727f8700 1 -- 172.21.15.7:0/4188477586 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6c06f990 con 0x7f2d6c0aad20 2023-03-03T08:45:22.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.493+0000 7f2d68ff9700 1 -- 172.21.15.7:0/4188477586 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2d6002a020 con 0x7f2d6c0aad20 2023-03-03T08:45:22.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.493+0000 7f2d68ff9700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d580682f0 0x7f2d5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:22.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.493+0000 7f2d68ff9700 1 -- 172.21.15.7:0/4188477586 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7104+0+0 (secure 0 0 0) 0x7f2d60087870 con 0x7f2d6c0aad20 2023-03-03T08:45:22.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.493+0000 7f2d6bfff700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d580682f0 0x7f2d5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:22.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.494+0000 7f2d6bfff700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d580682f0 0x7f2d5806a7a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c006fd0 tx=0x7f2d5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:22.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.497+0000 7f2d68ff9700 1 -- 172.21.15.7:0/4188477586 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2d60053cc0 con 0x7f2d6c0aad20 2023-03-03T08:45:22.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.734+0000 7f2d727f8700 1 -- 172.21.15.7:0/4188477586 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d6c06f990 con 0x7f2d6c0aad20 2023-03-03T08:45:22.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.735+0000 7f2d68ff9700 1 -- 172.21.15.7:0/4188477586 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5754 (secure 0 0 0) 0x7f2d600572e0 con 0x7f2d6c0aad20 2023-03-03T08:45:22.736 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:22.737 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5121,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/2005962882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":2005962882},{"type":"v1","addr":"172.21.15.7:6835","nonce":2005962882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:44:42.296317+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"d","rank":0,"incarnation":15,"state":"up:active","state_seq":21,"addr":"172.21.15.143:6837/4239346710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":4239346710},{"type":"v1","addr":"172.21.15.143:6837","nonce":4239346710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:22.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 -- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d580682f0 msgr2=0x7f2d5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:22.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d580682f0 0x7f2d5806a7a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c006fd0 tx=0x7f2d5c008040 comp rx=0 tx=0).stop 2023-03-03T08:45:22.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 -- 172.21.15.7:0/4188477586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 msgr2=0x7f2d6c1440d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:22.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 0x7f2d6c1440d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000b0e0 tx=0x7f2d60004750 comp rx=0 tx=0).stop 2023-03-03T08:45:22.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 -- 172.21.15.7:0/4188477586 shutdown_connections 2023-03-03T08:45:22.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d6c14b0d0 0x7f2d6c144610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:22.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d6c071d40 0x7f2d6c1ed1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:22.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d580682f0 0x7f2d5806a7a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:22.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 --2- 172.21.15.7:0/4188477586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d6c0aad20 0x7f2d6c1440d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:22.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 -- 172.21.15.7:0/4188477586 >> 172.21.15.7:0/4188477586 conn(0x7f2d6c0c8900 msgr2=0x7f2d6c0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:22.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 -- 172.21.15.7:0/4188477586 shutdown_connections 2023-03-03T08:45:22.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:22.737+0000 7f2d727f8700 1 -- 172.21.15.7:0/4188477586 wait complete. 2023-03-03T08:45:22.740 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 21 2023-03-03T08:45:22.751 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-03-03T08:45:24.752 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:24.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.890+0000 7fd58299d700 1 Processor -- start 2023-03-03T08:45:24.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.890+0000 7fd58299d700 1 -- start start 2023-03-03T08:45:24.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.891+0000 7fd58299d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd57c14e820 0x7fd57c0c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.891+0000 7fd58299d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd57c0ac620 0x7fd57c0c6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.891+0000 7fd58299d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c0715b0 con 0x7fd57c0c6a50 2023-03-03T08:45:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.891+0000 7fd58299d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c070a90 con 0x7fd57c14e820 2023-03-03T08:45:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.891+0000 7fd58299d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c0c3aa0 con 0x7fd57c0ac620 2023-03-03T08:45:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.891+0000 7fd58199b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd57c14e820 0x7fd57c0c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.891+0000 7fd58219c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd57c0c6a50 0x7fd57c151170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46998/0 (socket says 172.21.15.7:46998) 2023-03-03T08:45:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.891+0000 7fd58219c700 1 -- 172.21.15.7:0/2653617916 learned_addr learned my addr 172.21.15.7:0/2653617916 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:45:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.891+0000 7fd58199b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd57c14e820 0x7fd57c0c32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53014/0 (socket says 172.21.15.7:53014) 2023-03-03T08:45:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.892+0000 7fd58119a700 1 --2- 172.21.15.7:0/2653617916 >> v2:172.21.15.143:3300/0 conn(0x7fd57c0ac620 0x7fd57c0c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.892+0000 7fd58199b700 1 -- 172.21.15.7:0/2653617916 >> v2:172.21.15.143:3300/0 conn(0x7fd57c0ac620 msgr2=0x7fd57c0c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.892+0000 7fd58199b700 1 --2- 172.21.15.7:0/2653617916 >> v2:172.21.15.143:3300/0 conn(0x7fd57c0ac620 0x7fd57c0c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.892+0000 7fd58199b700 1 -- 172.21.15.7:0/2653617916 >> v1:172.21.15.7:6789/0 conn(0x7fd57c0c6a50 legacy=0x7fd57c151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.892+0000 7fd58199b700 1 -- 172.21.15.7:0/2653617916 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0720d0 con 0x7fd57c14e820 2023-03-03T08:45:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.892+0000 7fd580999700 1 -- 172.21.15.7:0/2653617916 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151341798 0 0) 0x7fd57c0715b0 con 0x7fd57c0c6a50 2023-03-03T08:45:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.892+0000 7fd58199b700 1 --2- 172.21.15.7:0/2653617916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c14e820 0x7fd57c0c32c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd57c070000 tx=0x7fd578009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf758fd7209d124 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:24.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.892+0000 7fd580999700 1 -- 172.21.15.7:0/2653617916 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578011830 con 0x7fd57c14e820 2023-03-03T08:45:24.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.892+0000 7fd580999700 1 -- 172.21.15.7:0/2653617916 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd578011990 con 0x7fd57c14e820 2023-03-03T08:45:24.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.892+0000 7fd580999700 1 -- 172.21.15.7:0/2653617916 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578011d20 con 0x7fd57c14e820 2023-03-03T08:45:24.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.893+0000 7fd58299d700 1 -- 172.21.15.7:0/2653617916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c14e820 msgr2=0x7fd57c0c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:24.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.893+0000 7fd58299d700 1 --2- 172.21.15.7:0/2653617916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c14e820 0x7fd57c0c32c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd57c070000 tx=0x7fd578009b50 comp rx=0 tx=0).stop 2023-03-03T08:45:24.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.893+0000 7fd58299d700 1 -- 172.21.15.7:0/2653617916 shutdown_connections 2023-03-03T08:45:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.893+0000 7fd58299d700 1 --2- 172.21.15.7:0/2653617916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c14e820 0x7fd57c0c32c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.893+0000 7fd58299d700 1 --2- 172.21.15.7:0/2653617916 >> v2:172.21.15.143:3300/0 conn(0x7fd57c0ac620 0x7fd57c0c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.893+0000 7fd58299d700 1 -- 172.21.15.7:0/2653617916 >> 172.21.15.7:0/2653617916 conn(0x7fd57c0c48b0 msgr2=0x7fd57c0c7180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.893+0000 7fd58299d700 1 -- 172.21.15.7:0/2653617916 shutdown_connections 2023-03-03T08:45:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.893+0000 7fd58299d700 1 -- 172.21.15.7:0/2653617916 wait complete. 2023-03-03T08:45:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.894+0000 7fd58299d700 1 Processor -- start 2023-03-03T08:45:24.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.894+0000 7fd58299d700 1 -- start start 2023-03-03T08:45:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58299d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c0ac620 0x7fd57c0c07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58299d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c6a50 0x7fd57c14d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58299d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd57c14e820 0x7fd57c14d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58299d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c1ea6a0 con 0x7fd57c0c6a50 2023-03-03T08:45:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58299d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c148b60 con 0x7fd57c14e820 2023-03-03T08:45:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58299d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c148e40 con 0x7fd57c0ac620 2023-03-03T08:45:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58119a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c6a50 0x7fd57c14d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:24.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58119a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c6a50 0x7fd57c14d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53022/0 (socket says 172.21.15.7:53022) 2023-03-03T08:45:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58119a700 1 -- 172.21.15.7:0/1128260118 learned_addr learned my addr 172.21.15.7:0/1128260118 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58219c700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd57c14e820 0x7fd57c14d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58199b700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c0ac620 0x7fd57c0c07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58119a700 1 -- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c0ac620 msgr2=0x7fd57c0c07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58119a700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c0ac620 0x7fd57c0c07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58119a700 1 -- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd57c14e820 msgr2=0x7fd57c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58119a700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd57c14e820 0x7fd57c14d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:24.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58119a700 1 -- 172.21.15.7:0/1128260118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5780094a0 con 0x7fd57c0c6a50 2023-03-03T08:45:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58199b700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c0ac620 0x7fd57c0c07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:45:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58119a700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c6a50 0x7fd57c14d4a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd57000e810 tx=0x7fd57000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd56e7fc700 1 -- 172.21.15.7:0/1128260118 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570013610 con 0x7fd57c0c6a50 2023-03-03T08:45:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd56e7fc700 1 -- 172.21.15.7:0/1128260118 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd570013770 con 0x7fd57c0c6a50 2023-03-03T08:45:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd58299d700 1 -- 172.21.15.7:0/1128260118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c0717e0 con 0x7fd57c0c6a50 2023-03-03T08:45:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.895+0000 7fd56e7fc700 1 -- 172.21.15.7:0/1128260118 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570013b20 con 0x7fd57c0c6a50 2023-03-03T08:45:24.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.896+0000 7fd58299d700 1 -- 172.21.15.7:0/1128260118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c14ebf0 con 0x7fd57c0c6a50 2023-03-03T08:45:24.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.897+0000 7fd58299d700 1 -- 172.21.15.7:0/1128260118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c0c3800 con 0x7fd57c0c6a50 2023-03-03T08:45:24.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.897+0000 7fd56e7fc700 1 -- 172.21.15.7:0/1128260118 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd570023070 con 0x7fd57c0c6a50 2023-03-03T08:45:24.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.897+0000 7fd56e7fc700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 0x7fd56006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:24.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.897+0000 7fd56e7fc700 1 -- 172.21.15.7:0/1128260118 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7190+0+0 (secure 0 0 0) 0x7fd570085de0 con 0x7fd57c0c6a50 2023-03-03T08:45:24.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.897+0000 7fd58199b700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 0x7fd56006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:24.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.898+0000 7fd58199b700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 0x7fd56006a7a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd57c1e7580 tx=0x7fd578008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:24.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:24.902+0000 7fd56e7fc700 1 -- 172.21.15.7:0/1128260118 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd570052150 con 0x7fd57c0c6a50 2023-03-03T08:45:25.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.139+0000 7fd58299d700 1 -- 172.21.15.7:0/1128260118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd57c1ecf90 con 0x7fd57c0c6a50 2023-03-03T08:45:25.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.140+0000 7fd56e7fc700 1 -- 172.21.15.7:0/1128260118 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4993 (secure 0 0 0) 0x7fd570055770 con 0x7fd57c0c6a50 2023-03-03T08:45:25.142 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:25.142 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:24.297330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5121},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5121":{"gid":5121,"name":"a","rank":0,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.7:6835/2005962882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":2005962882},{"type":"v1","addr":"172.21.15.7:6835","nonce":2005962882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 -- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 msgr2=0x7fd56006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 0x7fd56006a7a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd57c1e7580 tx=0x7fd578008030 comp rx=0 tx=0).stop 2023-03-03T08:45:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 -- 172.21.15.7:0/1128260118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c6a50 msgr2=0x7fd57c14d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c6a50 0x7fd57c14d4a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd57000e810 tx=0x7fd57000ebd0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 -- 172.21.15.7:0/1128260118 shutdown_connections 2023-03-03T08:45:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd57c14e820 0x7fd57c14d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c0ac620 0x7fd57c0c07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 0x7fd56006a7a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 --2- 172.21.15.7:0/1128260118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c6a50 0x7fd57c14d4a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 -- 172.21.15.7:0/1128260118 >> 172.21.15.7:0/1128260118 conn(0x7fd57c0c48b0 msgr2=0x7fd57c0c4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 -- 172.21.15.7:0/1128260118 shutdown_connections 2023-03-03T08:45:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.142+0000 7fd58299d700 1 -- 172.21.15.7:0/1128260118 wait complete. 2023-03-03T08:45:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 22 2023-03-03T08:45:25.155 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2023-03-03T08:45:25.156 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-03-03T08:45:25.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:25.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.279+0000 7f4ec7bed700 1 Processor -- start 2023-03-03T08:45:25.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.279+0000 7f4ec7bed700 1 -- start start 2023-03-03T08:45:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec7bed700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ec014b460 0x7f4ec006aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec7bed700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec0070360 con 0x7f4ec0072470 2023-03-03T08:45:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec7bed700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec006a5e0 con 0x7f4ec0071990 2023-03-03T08:45:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec7bed700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec01521a0 con 0x7f4ec014b460 2023-03-03T08:45:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec5188700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ec014b460 0x7f4ec006aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec5188700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ec014b460 0x7f4ec006aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53030/0 (socket says 172.21.15.7:53030) 2023-03-03T08:45:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec618a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4ec0072470 0x7f4ec006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47012/0 (socket says 172.21.15.7:47012) 2023-03-03T08:45:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec5188700 1 -- 172.21.15.7:0/2008066361 learned_addr learned my addr 172.21.15.7:0/2008066361 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec5188700 1 -- 172.21.15.7:0/2008066361 >> v1:172.21.15.143:6789/0 conn(0x7f4ec0071990 legacy=0x7f4ec0071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec5188700 1 -- 172.21.15.7:0/2008066361 >> v1:172.21.15.7:6789/0 conn(0x7f4ec0072470 legacy=0x7f4ec006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.280+0000 7f4ec5188700 1 -- 172.21.15.7:0/2008066361 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec00c38b0 con 0x7f4ec014b460 2023-03-03T08:45:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec5188700 1 --2- 172.21.15.7:0/2008066361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec014b460 0x7f4ec006aa70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00bb90 tx=0x7f4ebc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed198df41d0aa9df server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:25.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec4987700 1 -- 172.21.15.7:0/2008066361 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc00dd50 con 0x7f4ec014b460 2023-03-03T08:45:25.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec4987700 1 -- 172.21.15.7:0/2008066361 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc010470 con 0x7f4ec014b460 2023-03-03T08:45:25.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec4987700 1 -- 172.21.15.7:0/2008066361 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc010800 con 0x7f4ec014b460 2023-03-03T08:45:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2008066361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec014b460 msgr2=0x7f4ec006aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec7bed700 1 --2- 172.21.15.7:0/2008066361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec014b460 0x7f4ec006aa70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00bb90 tx=0x7f4ebc00bea0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2008066361 shutdown_connections 2023-03-03T08:45:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec7bed700 1 --2- 172.21.15.7:0/2008066361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec014b460 0x7f4ec006aa70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2008066361 >> 172.21.15.7:0/2008066361 conn(0x7f4ec00a94d0 msgr2=0x7f4ec00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2008066361 shutdown_connections 2023-03-03T08:45:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.281+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2008066361 wait complete. 2023-03-03T08:45:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.283+0000 7f4ec7bed700 1 Processor -- start 2023-03-03T08:45:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.283+0000 7f4ec7bed700 1 -- start start 2023-03-03T08:45:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec7bed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ec0071990 0x7f4ec013ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec7bed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec0072470 0x7f4ec0150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec7bed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ec01513d0 0x7f4ec00bcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec7bed700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec00bb4e0 con 0x7f4ec0072470 2023-03-03T08:45:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec7bed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec0154fb0 con 0x7f4ec01513d0 2023-03-03T08:45:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec7bed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec0155290 con 0x7f4ec0071990 2023-03-03T08:45:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5188700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec0072470 0x7f4ec0150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5188700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec0072470 0x7f4ec0150040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53036/0 (socket says 172.21.15.7:53036) 2023-03-03T08:45:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5188700 1 -- 172.21.15.7:0/2488463193 learned_addr learned my addr 172.21.15.7:0/2488463193 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec618a700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ec01513d0 0x7f4ec00bcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5989700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ec0071990 0x7f4ec013ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5188700 1 -- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ec0071990 msgr2=0x7f4ec013ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5188700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ec0071990 0x7f4ec013ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5188700 1 -- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ec01513d0 msgr2=0x7f4ec00bcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5188700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ec01513d0 0x7f4ec00bcfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5188700 1 -- 172.21.15.7:0/2488463193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc00b7f0 con 0x7f4ec0072470 2023-03-03T08:45:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec618a700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ec01513d0 0x7f4ec00bcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:45:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5989700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ec0071990 0x7f4ec013ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:45:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.284+0000 7f4ec5188700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec0072470 0x7f4ec0150040 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00bee0 tx=0x7f4ebc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.285+0000 7f4eb67fc700 1 -- 172.21.15.7:0/2488463193 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc01f070 con 0x7f4ec0072470 2023-03-03T08:45:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.285+0000 7f4eb67fc700 1 -- 172.21.15.7:0/2488463193 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc0039b0 con 0x7f4ec0072470 2023-03-03T08:45:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.285+0000 7f4eb67fc700 1 -- 172.21.15.7:0/2488463193 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc003d60 con 0x7f4ec0072470 2023-03-03T08:45:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.285+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2488463193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec00c3b50 con 0x7f4ec0072470 2023-03-03T08:45:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.285+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2488463193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec006a2b0 con 0x7f4ec0072470 2023-03-03T08:45:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.286+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2488463193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec006b6c0 con 0x7f4ec0072470 2023-03-03T08:45:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.286+0000 7f4eb67fc700 1 -- 172.21.15.7:0/2488463193 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ebc019400 con 0x7f4ec0072470 2023-03-03T08:45:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.286+0000 7f4eb67fc700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ea40682f0 0x7f4ea406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.287+0000 7f4eb67fc700 1 -- 172.21.15.7:0/2488463193 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7190+0+0 (secure 0 0 0) 0x7f4ebc089fc0 con 0x7f4ec0072470 2023-03-03T08:45:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.287+0000 7f4ec5989700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ea40682f0 0x7f4ea406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:25.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.287+0000 7f4ec5989700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ea40682f0 0x7f4ea406a7a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0000f60 tx=0x7f4eb0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:25.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.291+0000 7f4eb67fc700 1 -- 172.21.15.7:0/2488463193 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ebc056350 con 0x7f4ec0072470 2023-03-03T08:45:25.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.532+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2488463193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ec0074bb0 con 0x7f4ec0072470 2023-03-03T08:45:25.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.533+0000 7f4eb67fc700 1 -- 172.21.15.7:0/2488463193 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4996 (secure 0 0 0) 0x7f4ebc059970 con 0x7f4ec0072470 2023-03-03T08:45:25.535 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:25.535 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:25.302713+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5121},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5121":{"gid":5121,"name":"a","rank":0,"incarnation":22,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.7:6835/2005962882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":2005962882},{"type":"v1","addr":"172.21.15.7:6835","nonce":2005962882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:25.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ea40682f0 msgr2=0x7f4ea406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:25.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ea40682f0 0x7f4ea406a7a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0000f60 tx=0x7f4eb0009040 comp rx=0 tx=0).stop 2023-03-03T08:45:25.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2488463193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec0072470 msgr2=0x7f4ec0150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:25.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec0072470 0x7f4ec0150040 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00bee0 tx=0x7f4ebc004750 comp rx=0 tx=0).stop 2023-03-03T08:45:25.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2488463193 shutdown_connections 2023-03-03T08:45:25.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ec01513d0 0x7f4ec00bcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ec0071990 0x7f4ec013ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ea40682f0 0x7f4ea406a7a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 --2- 172.21.15.7:0/2488463193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ec0072470 0x7f4ec0150040 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2488463193 >> 172.21.15.7:0/2488463193 conn(0x7f4ec00a94d0 msgr2=0x7f4ec0149fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2488463193 shutdown_connections 2023-03-03T08:45:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:25.535+0000 7f4ec7bed700 1 -- 172.21.15.7:0/2488463193 wait complete. 2023-03-03T08:45:25.542 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 23 2023-03-03T08:45:25.549 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-03-03T08:43:34.516962+0000", "damaged": [ 0 ], "data_pools": [ 4 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": {}, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-03-03T08:45:11.325826+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": {} } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-03-03T08:43:43.971492+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5121": { "addr": "172.21.15.7:6835/2005962882", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6833", "nonce": 2005962882, "type": "v2" }, { "addr": "172.21.15.7:6835", "nonce": 2005962882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5121, "incarnation": 22, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-03-03T08:45:25.302713+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5121 } } } ], "standbys": [ { "addr": "172.21.15.143:6835/2988899186", "addrs": { "addrvec": [ { "addr": "172.21.15.143:6834", "nonce": 2988899186, "type": "v2" }, { "addr": "172.21.15.143:6835", "nonce": 2988899186, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4244, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.7:6834/4107062379", "addrs": { "addrvec": [ { "addr": "172.21.15.7:6832", "nonce": 4107062379, "type": "v2" }, { "addr": "172.21.15.7:6834", "nonce": 4107062379, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 19, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4897, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-03-03T08:45:27.336 INFO:tasks.workunit.client.1.smithi143.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-03-03T08:45:27.336 INFO:tasks.ceph.mds.a.smithi007.stderr:2023-03-03T08:45:27.334+0000 7f9e61698700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:45:27.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.689+0000 7f9f02ff3700 1 Processor -- start 2023-03-03T08:45:27.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.690+0000 7f9f02ff3700 1 -- start start 2023-03-03T08:45:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.690+0000 7f9f02ff3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9efc072250 0x7f9efc14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:27.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.690+0000 7f9f02ff3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc1510d0 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.690+0000 7f9f02ff3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc0c3740 con 0x7f9efc071e80 2023-03-03T08:45:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.690+0000 7f9f02ff3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9efc070670 con 0x7f9efc072250 2023-03-03T08:45:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.690+0000 7f9efbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9efc072250 0x7f9efc14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.690+0000 7f9f01590700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9efc0c1ce0 0x7f9efc15ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47030/0 (socket says 172.21.15.7:47030) 2023-03-03T08:45:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.690+0000 7f9f01590700 1 -- 172.21.15.7:0/3168224320 learned_addr learned my addr 172.21.15.7:0/3168224320 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:45:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.691+0000 7f9efbfff700 1 -- 172.21.15.7:0/3168224320 >> v1:172.21.15.143:6789/0 conn(0x7f9efc071e80 legacy=0x7f9efc151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:27.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.691+0000 7f9efbfff700 1 -- 172.21.15.7:0/3168224320 >> v1:172.21.15.7:6789/0 conn(0x7f9efc0c1ce0 legacy=0x7f9efc15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.691+0000 7f9efbfff700 1 -- 172.21.15.7:0/3168224320 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc0c39e0 con 0x7f9efc072250 2023-03-03T08:45:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.691+0000 7f9efbfff700 1 --2- 172.21.15.7:0/3168224320 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc072250 0x7f9efc14e1f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009870 tx=0x7f9ee8009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f40aa251fc2c10d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.691+0000 7f9efb7fe700 1 -- 172.21.15.7:0/3168224320 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8011830 con 0x7f9efc072250 2023-03-03T08:45:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.691+0000 7f9efb7fe700 1 -- 172.21.15.7:0/3168224320 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee8011990 con 0x7f9efc072250 2023-03-03T08:45:27.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.691+0000 7f9efb7fe700 1 -- 172.21.15.7:0/3168224320 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8011d20 con 0x7f9efc072250 2023-03-03T08:45:27.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.692+0000 7f9f02ff3700 1 -- 172.21.15.7:0/3168224320 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc072250 msgr2=0x7f9efc14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:27.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.692+0000 7f9f02ff3700 1 --2- 172.21.15.7:0/3168224320 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc072250 0x7f9efc14e1f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009870 tx=0x7f9ee8009b80 comp rx=0 tx=0).stop 2023-03-03T08:45:27.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.692+0000 7f9f02ff3700 1 -- 172.21.15.7:0/3168224320 shutdown_connections 2023-03-03T08:45:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.692+0000 7f9f02ff3700 1 --2- 172.21.15.7:0/3168224320 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc072250 0x7f9efc14e1f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.692+0000 7f9f02ff3700 1 -- 172.21.15.7:0/3168224320 >> 172.21.15.7:0/3168224320 conn(0x7f9efc0a94d0 msgr2=0x7f9efc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.692+0000 7f9f02ff3700 1 -- 172.21.15.7:0/3168224320 shutdown_connections 2023-03-03T08:45:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.692+0000 7f9f02ff3700 1 -- 172.21.15.7:0/3168224320 wait complete. 2023-03-03T08:45:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.693+0000 7f9f02ff3700 1 Processor -- start 2023-03-03T08:45:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.693+0000 7f9f02ff3700 1 -- start start 2023-03-03T08:45:27.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f02ff3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9efc071e80 0x7f9efc0c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f02ff3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9efc072250 0x7f9efc1da730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f02ff3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc0c1ce0 0x7f9efc1dac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f02ff3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc0c64a0 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f02ff3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc14c690 con 0x7f9efc072250 2023-03-03T08:45:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f02ff3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc14c8e0 con 0x7f9efc071e80 2023-03-03T08:45:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f01590700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc0c1ce0 0x7f9efc1dac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:27.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f01590700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc0c1ce0 0x7f9efc1dac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53056/0 (socket says 172.21.15.7:53056) 2023-03-03T08:45:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f01590700 1 -- 172.21.15.7:0/1692392168 learned_addr learned my addr 172.21.15.7:0/1692392168 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9efbfff700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9efc072250 0x7f9efc1da730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f01590700 1 -- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9efc071e80 msgr2=0x7f9efc0c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f00d8f700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9efc071e80 0x7f9efc0c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f01590700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9efc071e80 0x7f9efc0c6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f01590700 1 -- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9efc072250 msgr2=0x7f9efc1da730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f01590700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9efc072250 0x7f9efc1da730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:27.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f01590700 1 -- 172.21.15.7:0/1692392168 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee80094a0 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.694+0000 7f9f01590700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc0c1ce0 0x7f9efc1dac70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9ef400caa0 tx=0x7f9ef400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.695+0000 7f9ef97fa700 1 -- 172.21.15.7:0/1692392168 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef400aa80 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.695+0000 7f9ef97fa700 1 -- 172.21.15.7:0/1692392168 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef400abe0 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.695+0000 7f9f02ff3700 1 -- 172.21.15.7:0/1692392168 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc151f30 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.695+0000 7f9ef97fa700 1 -- 172.21.15.7:0/1692392168 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4012400 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.695+0000 7f9f02ff3700 1 -- 172.21.15.7:0/1692392168 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc151300 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.696+0000 7f9f02ff3700 1 -- 172.21.15.7:0/1692392168 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc06c9a0 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.696+0000 7f9ef97fa700 1 -- 172.21.15.7:0/1692392168 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ef4019030 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.697+0000 7f9ef97fa700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ee00682f0 0x7f9ee006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:27.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.697+0000 7f9ef97fa700 1 -- 172.21.15.7:0/1692392168 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7190+0+0 (secure 0 0 0) 0x7f9ef4085b40 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.697+0000 7f9f00d8f700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ee00682f0 0x7f9ee006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:27.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.698+0000 7f9f00d8f700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ee00682f0 0x7f9ee006a7a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0007fd0 tx=0x7f9ef0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:27.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.701+0000 7f9ef97fa700 1 -- 172.21.15.7:0/1692392168 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ef404b5e0 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.939+0000 7f9f02ff3700 1 -- 172.21.15.7:0/1692392168 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9efc074bb0 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.939+0000 7f9ef97fa700 1 -- 172.21.15.7:0/1692392168 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4993 (secure 0 0 0) 0x7f9ef4051fe0 con 0x7f9efc0c1ce0 2023-03-03T08:45:27.941 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:27.941 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:27.305326+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5121},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5121":{"gid":5121,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.7:6835/2005962882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":2005962882},{"type":"v1","addr":"172.21.15.7:6835","nonce":2005962882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:27.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 -- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ee00682f0 msgr2=0x7f9ee006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:27.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ee00682f0 0x7f9ee006a7a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0007fd0 tx=0x7f9ef0009040 comp rx=0 tx=0).stop 2023-03-03T08:45:27.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 -- 172.21.15.7:0/1692392168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc0c1ce0 msgr2=0x7f9efc1dac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:27.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc0c1ce0 0x7f9efc1dac70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9ef400caa0 tx=0x7f9ef400ce60 comp rx=0 tx=0).stop 2023-03-03T08:45:27.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 -- 172.21.15.7:0/1692392168 shutdown_connections 2023-03-03T08:45:27.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ee00682f0 0x7f9ee006a7a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:27.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9efc0c1ce0 0x7f9efc1dac70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:27.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9efc071e80 0x7f9efc0c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:27.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 --2- 172.21.15.7:0/1692392168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9efc072250 0x7f9efc1da730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:27.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 -- 172.21.15.7:0/1692392168 >> 172.21.15.7:0/1692392168 conn(0x7f9efc0a94d0 msgr2=0x7f9efc06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:27.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 -- 172.21.15.7:0/1692392168 shutdown_connections 2023-03-03T08:45:27.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:27.942+0000 7f9f02ff3700 1 -- 172.21.15.7:0/1692392168 wait complete. 2023-03-03T08:45:27.945 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 25 2023-03-03T08:45:27.956 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=4247 2023-03-03T08:45:27.956 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.d 2023-03-03T08:45:29.702 INFO:tasks.workunit.client.1.smithi143.stderr:+ mkdir foo 2023-03-03T08:45:29.709 INFO:tasks.workunit.client.1.smithi143.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-03-03T08:45:30.601 INFO:tasks.workunit.client.1.smithi143.stderr:+ mkdir foo/.snap/barsnap 2023-03-03T08:45:30.642 INFO:tasks.workunit.client.1.smithi143.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-03-03T08:45:31.310 INFO:tasks.workunit.client.1.smithi143.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-03-03T08:45:31.743 INFO:tasks.workunit.client.1.smithi143.stderr:+ echo 'passed: files are identical' 2023-03-03T08:45:31.744 INFO:tasks.workunit.client.1.smithi143.stdout:passed: files are identical 2023-03-03T08:45:31.744 INFO:tasks.workunit.client.1.smithi143.stderr:+ rmdir foo/.snap/barsnap 2023-03-03T08:45:31.759 INFO:tasks.workunit.client.1.smithi143.stdout:OK 2023-03-03T08:45:31.760 INFO:tasks.workunit.client.1.smithi143.stderr:+ echo OK 2023-03-03T08:45:31.761 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-03-03T08:45:31.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2023-03-03T08:45:31.788 INFO:tasks.ceph.mds.a.smithi007.stderr:2023-03-03T08:45:31.787+0000 7f9e63e9d700 -1 mds.0.cache.den(0x600 10000000002) newly corrupt dentry to be committed: [dentry #0x100/stray0/10000000002 [4,head] auth NULL (dn xlock x=1 by 0x55792723e800) (dversion lock w=1 last_client=4541) pv=27 v=26 ap=2 ino=(nil) state=1342177281|new | request=1 lock=2 authpin=1 0x557927169b80] 2023-03-03T08:45:31.789 INFO:tasks.ceph.mds.a.smithi007.stderr:2023-03-03T08:45:31.787+0000 7f9e63e9d700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x600is fatal because it is a system directory for this rank 2023-03-03T08:45:32.316 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T08:45:32.314+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : Health check update: 2 filesystems are offline (MDS_ALL_DOWN) 2023-03-03T08:45:32.316 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T08:45:32.314+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : Health check update: 2 mds daemons damaged (MDS_DAMAGE) 2023-03-03T08:45:32.327 INFO:tasks.ceph.mds.a.smithi007.stderr: -5754> 2023-03-03T08:45:27.334+0000 7f9e61698700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-03T08:45:32.334 INFO:tasks.ceph.mds.a.smithi007.stderr: -36> 2023-03-03T08:45:31.787+0000 7f9e63e9d700 -1 mds.0.cache.den(0x600 10000000002) newly corrupt dentry to be committed: [dentry #0x100/stray0/10000000002 [4,head] auth NULL (dn xlock x=1 by 0x55792723e800) (dversion lock w=1 last_client=4541) pv=27 v=26 ap=2 ino=(nil) state=1342177281|new | request=1 lock=2 authpin=1 0x557927169b80] 2023-03-03T08:45:32.334 INFO:tasks.ceph.mds.a.smithi007.stderr: -34> 2023-03-03T08:45:31.787+0000 7f9e63e9d700 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x600is fatal because it is a system directory for this rank 2023-03-03T08:45:32.360 INFO:tasks.ceph.mds.a.smithi007.stdout:starting mds.a at 2023-03-03T08:45:33.956 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2023-03-03T08:45:33.957 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-03T08:45:33.957 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-03T08:45:33.960 INFO:tasks.ceph.mds.d:Started 2023-03-03T08:45:33.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:34.047 INFO:tasks.ceph.mds.d.smithi143.stdout:starting mds.d at 2023-03-03T08:45:34.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.098+0000 7f0abb2d7700 1 Processor -- start 2023-03-03T08:45:34.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.098+0000 7f0abb2d7700 1 -- start start 2023-03-03T08:45:34.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.099+0000 7f0abb2d7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0ab414fda0 0x7f0ab4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:34.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.099+0000 7f0abb2d7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0ab40abe20 0x7f0ab41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:34.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.099+0000 7f0abb2d7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab40c2f30 con 0x7f0ab414f9d0 2023-03-03T08:45:34.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.099+0000 7f0abb2d7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab40c31d0 con 0x7f0ab40abe20 2023-03-03T08:45:34.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.099+0000 7f0abb2d7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab41524a0 con 0x7f0ab414fda0 2023-03-03T08:45:34.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.099+0000 7f0ab9874700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0ab40abe20 0x7f0ab41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:34.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.099+0000 7f0ab9073700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f0ab414f9d0 0x7f0ab4071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50578/0 (socket says 172.21.15.7:50578) 2023-03-03T08:45:34.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.099+0000 7f0ab9874700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0ab40abe20 0x7f0ab41529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53082/0 (socket says 172.21.15.7:53082) 2023-03-03T08:45:34.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.099+0000 7f0ab9073700 1 -- 172.21.15.7:0/4107433939 learned_addr learned my addr 172.21.15.7:0/4107433939 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:45:34.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.099+0000 7f0ab8872700 1 --2- 172.21.15.7:0/4107433939 >> v2:172.21.15.143:3300/0 conn(0x7f0ab414fda0 0x7f0ab4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:34.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.100+0000 7f0aa7fff700 1 -- 172.21.15.7:0/4107433939 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636602372 0 0) 0x7f0ab40c2f30 con 0x7f0ab414f9d0 2023-03-03T08:45:34.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.100+0000 7f0aa7fff700 1 -- 172.21.15.7:0/4107433939 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aa00035b0 con 0x7f0ab414f9d0 2023-03-03T08:45:34.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.100+0000 7f0ab9874700 1 -- 172.21.15.7:0/4107433939 >> v2:172.21.15.143:3300/0 conn(0x7f0ab414fda0 msgr2=0x7f0ab4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:34.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.100+0000 7f0ab9874700 1 --2- 172.21.15.7:0/4107433939 >> v2:172.21.15.143:3300/0 conn(0x7f0ab414fda0 0x7f0ab4156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:34.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.100+0000 7f0ab9874700 1 -- 172.21.15.7:0/4107433939 >> v1:172.21.15.143:6789/0 conn(0x7f0ab414f9d0 legacy=0x7f0ab4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:34.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.100+0000 7f0ab9874700 1 -- 172.21.15.7:0/4107433939 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab40c34b0 con 0x7f0ab40abe20 2023-03-03T08:45:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.100+0000 7f0ab9874700 1 --2- 172.21.15.7:0/4107433939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab40abe20 0x7f0ab41529c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0ab40709b0 tx=0x7f0aac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c2b42d0c7ccca2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.100+0000 7f0aa7fff700 1 -- 172.21.15.7:0/4107433939 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac00e070 con 0x7f0ab40abe20 2023-03-03T08:45:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.100+0000 7f0aa7fff700 1 -- 172.21.15.7:0/4107433939 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aac00bb20 con 0x7f0ab40abe20 2023-03-03T08:45:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.100+0000 7f0aa7fff700 1 -- 172.21.15.7:0/4107433939 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac012430 con 0x7f0ab40abe20 2023-03-03T08:45:34.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.101+0000 7f0abb2d7700 1 -- 172.21.15.7:0/4107433939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab40abe20 msgr2=0x7f0ab41529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:34.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.101+0000 7f0abb2d7700 1 --2- 172.21.15.7:0/4107433939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab40abe20 0x7f0ab41529c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0ab40709b0 tx=0x7f0aac00b330 comp rx=0 tx=0).stop 2023-03-03T08:45:34.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.101+0000 7f0abb2d7700 1 -- 172.21.15.7:0/4107433939 shutdown_connections 2023-03-03T08:45:34.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.101+0000 7f0abb2d7700 1 --2- 172.21.15.7:0/4107433939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab40abe20 0x7f0ab41529c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:34.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.101+0000 7f0abb2d7700 1 --2- 172.21.15.7:0/4107433939 >> v2:172.21.15.143:3300/0 conn(0x7f0ab414fda0 0x7f0ab4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:34.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.101+0000 7f0abb2d7700 1 -- 172.21.15.7:0/4107433939 >> 172.21.15.7:0/4107433939 conn(0x7f0ab40c8900 msgr2=0x7f0ab40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:34.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.101+0000 7f0abb2d7700 1 -- 172.21.15.7:0/4107433939 shutdown_connections 2023-03-03T08:45:34.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.101+0000 7f0abb2d7700 1 -- 172.21.15.7:0/4107433939 wait complete. 2023-03-03T08:45:34.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.102+0000 7f0abb2d7700 1 Processor -- start 2023-03-03T08:45:34.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.102+0000 7f0abb2d7700 1 -- start start 2023-03-03T08:45:34.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0abb2d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab40abe20 0x7f0ab41e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0abb2d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab414f9d0 0x7f0ab40c1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0abb2d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ab414fda0 0x7f0ab40bfd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0abb2d7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab40bd600 con 0x7f0ab414f9d0 2023-03-03T08:45:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0abb2d7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab41e8080 con 0x7f0ab40abe20 2023-03-03T08:45:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0abb2d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab406d510 con 0x7f0ab414fda0 2023-03-03T08:45:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0ab8872700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab414f9d0 0x7f0ab40c1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0ab8872700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab414f9d0 0x7f0ab40c1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53086/0 (socket says 172.21.15.7:53086) 2023-03-03T08:45:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0ab8872700 1 -- 172.21.15.7:0/3352760811 learned_addr learned my addr 172.21.15.7:0/3352760811 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0ab9073700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab40abe20 0x7f0ab41e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0ab8872700 1 -- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ab414fda0 msgr2=0x7f0ab40bfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0ab8872700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ab414fda0 0x7f0ab40bfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0ab8872700 1 -- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab40abe20 msgr2=0x7f0ab41e7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0ab8872700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab40abe20 0x7f0ab41e7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0ab8872700 1 -- 172.21.15.7:0/3352760811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac009a70 con 0x7f0ab414f9d0 2023-03-03T08:45:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0ab8872700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab414f9d0 0x7f0ab40c1ef0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000e9b0 tx=0x7f0ab000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0aa5ffb700 1 -- 172.21.15.7:0/3352760811 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0009710 con 0x7f0ab414f9d0 2023-03-03T08:45:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0aa5ffb700 1 -- 172.21.15.7:0/3352760811 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab000c850 con 0x7f0ab414f9d0 2023-03-03T08:45:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0abb2d7700 1 -- 172.21.15.7:0/3352760811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab40ac350 con 0x7f0ab414f9d0 2023-03-03T08:45:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.103+0000 7f0aa5ffb700 1 -- 172.21.15.7:0/3352760811 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab000cc00 con 0x7f0ab414f9d0 2023-03-03T08:45:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.104+0000 7f0abb2d7700 1 -- 172.21.15.7:0/3352760811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab40ab630 con 0x7f0ab414f9d0 2023-03-03T08:45:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.104+0000 7f0abb2d7700 1 -- 172.21.15.7:0/3352760811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab4071fa0 con 0x7f0ab414f9d0 2023-03-03T08:45:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.105+0000 7f0aa5ffb700 1 -- 172.21.15.7:0/3352760811 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0ab0019030 con 0x7f0ab414f9d0 2023-03-03T08:45:34.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.105+0000 7f0aa5ffb700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a980682f0 0x7f0a9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:34.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.105+0000 7f0aa5ffb700 1 -- 172.21.15.7:0/3352760811 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0ab00860e0 con 0x7f0ab414f9d0 2023-03-03T08:45:34.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.106+0000 7f0ab9073700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a980682f0 0x7f0a9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:34.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.106+0000 7f0ab9073700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a980682f0 0x7f0a9806a7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80011e0 tx=0x7f0aa8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:34.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.110+0000 7f0aa5ffb700 1 -- 172.21.15.7:0/3352760811 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0ab0052480 con 0x7f0ab414f9d0 2023-03-03T08:45:34.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.353+0000 7f0abb2d7700 1 -- 172.21.15.7:0/3352760811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ab4074bb0 con 0x7f0ab414f9d0 2023-03-03T08:45:34.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.353+0000 7f0aa5ffb700 1 -- 172.21.15.7:0/3352760811 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f0ab0017090 con 0x7f0ab414f9d0 2023-03-03T08:45:34.355 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:34.355 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 -- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a980682f0 msgr2=0x7f0a9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a980682f0 0x7f0a9806a7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80011e0 tx=0x7f0aa8009040 comp rx=0 tx=0).stop 2023-03-03T08:45:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 -- 172.21.15.7:0/3352760811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab414f9d0 msgr2=0x7f0ab40c1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:34.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab414f9d0 0x7f0ab40c1ef0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000e9b0 tx=0x7f0ab000ed70 comp rx=0 tx=0).stop 2023-03-03T08:45:34.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 -- 172.21.15.7:0/3352760811 shutdown_connections 2023-03-03T08:45:34.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ab414fda0 0x7f0ab40bfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:34.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ab40abe20 0x7f0ab41e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:34.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a980682f0 0x7f0a9806a7a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:34.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 --2- 172.21.15.7:0/3352760811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ab414f9d0 0x7f0ab40c1ef0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:34.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 -- 172.21.15.7:0/3352760811 >> 172.21.15.7:0/3352760811 conn(0x7f0ab40c8900 msgr2=0x7f0ab4155550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:34.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 -- 172.21.15.7:0/3352760811 shutdown_connections 2023-03-03T08:45:34.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:34.356+0000 7f0abb2d7700 1 -- 172.21.15.7:0/3352760811 wait complete. 2023-03-03T08:45:34.359 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2023-03-03T08:45:34.369 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-03-03T08:45:35.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:35.325 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:45:36.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:36.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.509+0000 7fbea29fd700 1 Processor -- start 2023-03-03T08:45:36.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.510+0000 7fbea29fd700 1 -- start start 2023-03-03T08:45:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.510+0000 7fbea29fd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbe9c14de50 0x7fbe9c06a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:36.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.510+0000 7fbea29fd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe9c06a7f0 0x7fbe9c0c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.510+0000 7fbea29fd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c0a65b0 con 0x7fbe9c0c65c0 2023-03-03T08:45:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.510+0000 7fbea29fd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c154e00 con 0x7fbe9c14de50 2023-03-03T08:45:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.510+0000 7fbea29fd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c3370 con 0x7fbe9c06a7f0 2023-03-03T08:45:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.510+0000 7fbe9b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbe9c0c65c0 0x7fbe9c0c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47074/0 (socket says 172.21.15.7:47074) 2023-03-03T08:45:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9b7fe700 1 -- 172.21.15.7:0/824942461 learned_addr learned my addr 172.21.15.7:0/824942461 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:45:36.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9bfff700 1 --2- 172.21.15.7:0/824942461 >> v2:172.21.15.143:3300/0 conn(0x7fbe9c14de50 0x7fbe9c06a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbea0f9a700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe9c06a7f0 0x7fbe9c0c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844108225 0 0) 0x7fbe9c0a65b0 con 0x7fbe9c0c65c0 2023-03-03T08:45:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe840035b0 con 0x7fbe9c0c65c0 2023-03-03T08:45:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1247914031 0 0) 0x7fbe840035b0 con 0x7fbe9c0c65c0 2023-03-03T08:45:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe9c06a7f0 msgr2=0x7fbe9c0c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe9c06a7f0 0x7fbe9c0c2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:36.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 >> v2:172.21.15.143:3300/0 conn(0x7fbe9c14de50 msgr2=0x7fbe9c06a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:36.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 --2- 172.21.15.7:0/824942461 >> v2:172.21.15.143:3300/0 conn(0x7fbe9c14de50 0x7fbe9c06a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:36.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c1541f0 con 0x7fbe9c0c65c0 2023-03-03T08:45:36.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fbe90001670 con 0x7fbe9c0c65c0 2023-03-03T08:45:36.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 >> v1:172.21.15.7:6789/0 conn(0x7fbe9c0c65c0 legacy=0x7fbe9c0c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:36.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe84003aa0 0x7fbe84003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:36.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe84004850 0x7fbe84004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe84005200 0x7fbe8400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c154e00 con 0x7fbe84004850 2023-03-03T08:45:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8400bf50 con 0x7fbe84005200 2023-03-03T08:45:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8400c090 con 0x7fbe84003aa0 2023-03-03T08:45:36.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbea0f9a700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe9c06a7f0 0x7fbe9c0c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:45:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbe9b7fe700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe84004850 0x7fbe84004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.511+0000 7fbea0f9a700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe84005200 0x7fbe8400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbe9b7fe700 1 -- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe84003aa0 msgr2=0x7fbe84003eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-03T08:45:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbe9b7fe700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe84003aa0 0x7fbe84003eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbe9b7fe700 1 -- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe84005200 msgr2=0x7fbe8400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbe9b7fe700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe84005200 0x7fbe8400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:36.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbe9b7fe700 1 -- 172.21.15.7:0/824942461 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe840035b0 con 0x7fbe84004850 2023-03-03T08:45:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbea0f9a700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe84005200 0x7fbe8400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:45:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbe9b7fe700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe84004850 0x7fbe84004cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbe9000c2f0 tx=0x7fbe84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795bc8d40c4e4e02 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9000cd90 con 0x7fbe84004850 2023-03-03T08:45:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe90005960 con 0x7fbe84004850 2023-03-03T08:45:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbe9affd700 1 -- 172.21.15.7:0/824942461 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe90005d10 con 0x7fbe84004850 2023-03-03T08:45:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.512+0000 7fbea29fd700 1 -- 172.21.15.7:0/824942461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe84004850 msgr2=0x7fbe84004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:36.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.513+0000 7fbea29fd700 1 --2- 172.21.15.7:0/824942461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe84004850 0x7fbe84004cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbe9000c2f0 tx=0x7fbe84005880 comp rx=0 tx=0).stop 2023-03-03T08:45:36.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.513+0000 7fbea0f9a700 1 -- 172.21.15.7:0/824942461 reap_dead start 2023-03-03T08:45:36.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.513+0000 7fbea29fd700 1 -- 172.21.15.7:0/824942461 shutdown_connections 2023-03-03T08:45:36.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.513+0000 7fbea29fd700 1 -- 172.21.15.7:0/824942461 >> 172.21.15.7:0/824942461 conn(0x7fbe9c0a9430 msgr2=0x7fbe9c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:36.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.513+0000 7fbe9bfff700 1 --2- 172.21.15.7:0/824942461 >> v2:172.21.15.143:3300/0 conn(0x7fbe9c14de50 0x7fbe9c06a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:45:36.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.513+0000 7fbea29fd700 1 -- 172.21.15.7:0/824942461 shutdown_connections 2023-03-03T08:45:36.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.513+0000 7fbea29fd700 1 -- 172.21.15.7:0/824942461 wait complete. 2023-03-03T08:45:36.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.514+0000 7fbea29fd700 1 Processor -- start 2023-03-03T08:45:36.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.514+0000 7fbea29fd700 1 -- start start 2023-03-03T08:45:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea29fd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe84004850 0x7fbe9c144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea29fd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9c145240 0x7fbe9c0c3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea29fd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe9c0c3960 0x7fbe9c06fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:36.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea29fd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0725d0 con 0x7fbe9c0c3960 2023-03-03T08:45:36.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea29fd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c13ef90 con 0x7fbe9c145240 2023-03-03T08:45:36.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea29fd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c06d870 con 0x7fbe84004850 2023-03-03T08:45:36.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea0f9a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe9c0c3960 0x7fbe9c06fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:36.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea0f9a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe9c0c3960 0x7fbe9c06fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53106/0 (socket says 172.21.15.7:53106) 2023-03-03T08:45:36.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea0f9a700 1 -- 172.21.15.7:0/602719884 learned_addr learned my addr 172.21.15.7:0/602719884 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:36.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbe9b7fe700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9c145240 0x7fbe9c0c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:36.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbe9bfff700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe84004850 0x7fbe9c144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea0f9a700 1 -- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe84004850 msgr2=0x7fbe9c144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea0f9a700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe84004850 0x7fbe9c144d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea0f9a700 1 -- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9c145240 msgr2=0x7fbe9c0c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:36.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea0f9a700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9c145240 0x7fbe9c0c3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea0f9a700 1 -- 172.21.15.7:0/602719884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c1547b0 con 0x7fbe9c0c3960 2023-03-03T08:45:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbe9b7fe700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9c145240 0x7fbe9c0c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:45:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.515+0000 7fbea0f9a700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe9c0c3960 0x7fbe9c06fb00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0c5f20 tx=0x7fbe94011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.516+0000 7fbe98ff9700 1 -- 172.21.15.7:0/602719884 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94003c20 con 0x7fbe9c0c3960 2023-03-03T08:45:36.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.516+0000 7fbe98ff9700 1 -- 172.21.15.7:0/602719884 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe94003d80 con 0x7fbe9c0c3960 2023-03-03T08:45:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.516+0000 7fbea29fd700 1 -- 172.21.15.7:0/602719884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c1544d0 con 0x7fbe9c0c3960 2023-03-03T08:45:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.516+0000 7fbe98ff9700 1 -- 172.21.15.7:0/602719884 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94004250 con 0x7fbe9c0c3960 2023-03-03T08:45:36.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.516+0000 7fbea29fd700 1 -- 172.21.15.7:0/602719884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c154a90 con 0x7fbe9c0c3960 2023-03-03T08:45:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.517+0000 7fbea29fd700 1 -- 172.21.15.7:0/602719884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c070810 con 0x7fbe9c0c3960 2023-03-03T08:45:36.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.517+0000 7fbe98ff9700 1 -- 172.21.15.7:0/602719884 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbe940043b0 con 0x7fbe9c0c3960 2023-03-03T08:45:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.517+0000 7fbe98ff9700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe880682f0 0x7fbe8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.517+0000 7fbe98ff9700 1 -- 172.21.15.7:0/602719884 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbe94089000 con 0x7fbe9c0c3960 2023-03-03T08:45:36.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.518+0000 7fbe9bfff700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe880682f0 0x7fbe8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.518+0000 7fbe9bfff700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe880682f0 0x7fbe8806a7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0c7620 tx=0x7fbe8c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:36.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.522+0000 7fbe98ff9700 1 -- 172.21.15.7:0/602719884 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbe940553a0 con 0x7fbe9c0c3960 2023-03-03T08:45:36.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.765+0000 7fbea29fd700 1 -- 172.21.15.7:0/602719884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe9c074bb0 con 0x7fbe9c0c3960 2023-03-03T08:45:36.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.766+0000 7fbe98ff9700 1 -- 172.21.15.7:0/602719884 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbe940589c0 con 0x7fbe9c0c3960 2023-03-03T08:45:36.768 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:36.768 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:36.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 -- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe880682f0 msgr2=0x7fbe8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:36.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe880682f0 0x7fbe8806a7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0c7620 tx=0x7fbe8c009210 comp rx=0 tx=0).stop 2023-03-03T08:45:36.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 -- 172.21.15.7:0/602719884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe9c0c3960 msgr2=0x7fbe9c06fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:36.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe9c0c3960 0x7fbe9c06fb00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0c5f20 tx=0x7fbe94011870 comp rx=0 tx=0).stop 2023-03-03T08:45:36.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 -- 172.21.15.7:0/602719884 shutdown_connections 2023-03-03T08:45:36.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe880682f0 0x7fbe8806a7a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:36.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe9c0c3960 0x7fbe9c06fb00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:36.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe84004850 0x7fbe9c144d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:36.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 --2- 172.21.15.7:0/602719884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe9c145240 0x7fbe9c0c3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:36.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 -- 172.21.15.7:0/602719884 >> 172.21.15.7:0/602719884 conn(0x7fbe9c0a9430 msgr2=0x7fbe9c0a72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:36.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 -- 172.21.15.7:0/602719884 shutdown_connections 2023-03-03T08:45:36.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:36.768+0000 7fbea29fd700 1 -- 172.21.15.7:0/602719884 wait complete. 2023-03-03T08:45:36.772 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:45:36.782 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2023-03-03T08:45:36.782 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2023-03-03T08:45:44.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:45:44.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.963+0000 7f7e14ff6700 1 Processor -- start 2023-03-03T08:45:44.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.963+0000 7f7e14ff6700 1 -- start start 2023-03-03T08:45:44.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e14ff6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7e100ac220 0x7f7e100736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:44.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e14ff6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e10073c30 0x7f7e10152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:44.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e14ff6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e100ac900 con 0x7f7e100ace30 2023-03-03T08:45:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e14ff6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e10071e30 con 0x7f7e100ac220 2023-03-03T08:45:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e14ff6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100acb50 con 0x7f7e10073c30 2023-03-03T08:45:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e0dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e10073c30 0x7f7e10152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e0e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7e100ac220 0x7f7e100736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e0ed9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f7e100ace30 0x7f7e1006a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50616/0 (socket says 172.21.15.7:50616) 2023-03-03T08:45:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e0ed9d700 1 -- 172.21.15.7:0/1308873602 learned_addr learned my addr 172.21.15.7:0/1308873602 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:45:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e0d59a700 1 -- 172.21.15.7:0/1308873602 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236194349 0 0) 0x7f7e100ac900 con 0x7f7e100ace30 2023-03-03T08:45:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.964+0000 7f7e0d59a700 1 -- 172.21.15.7:0/1308873602 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df80035b0 con 0x7f7e100ace30 2023-03-03T08:45:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e0e59c700 1 -- 172.21.15.7:0/1308873602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e10073c30 msgr2=0x7f7e10152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:45:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e0e59c700 1 --2- 172.21.15.7:0/1308873602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e10073c30 0x7f7e10152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e0e59c700 1 -- 172.21.15.7:0/1308873602 >> v1:172.21.15.143:6789/0 conn(0x7f7e100ace30 legacy=0x7f7e1006a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e0e59c700 1 -- 172.21.15.7:0/1308873602 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e101559c0 con 0x7f7e100ac220 2023-03-03T08:45:44.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e0dd9b700 1 --2- 172.21.15.7:0/1308873602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e10073c30 0x7f7e10152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:45:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e0e59c700 1 --2- 172.21.15.7:0/1308873602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e100ac220 0x7f7e100736f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009840 tx=0x7f7e04009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9820c596beccdf8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e0d59a700 1 -- 172.21.15.7:0/1308873602 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04011830 con 0x7f7e100ac220 2023-03-03T08:45:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e0d59a700 1 -- 172.21.15.7:0/1308873602 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e04011990 con 0x7f7e100ac220 2023-03-03T08:45:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e0d59a700 1 -- 172.21.15.7:0/1308873602 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04011d20 con 0x7f7e100ac220 2023-03-03T08:45:44.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e14ff6700 1 -- 172.21.15.7:0/1308873602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e100ac220 msgr2=0x7f7e100736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:44.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.965+0000 7f7e14ff6700 1 --2- 172.21.15.7:0/1308873602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e100ac220 0x7f7e100736f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009840 tx=0x7f7e04009b50 comp rx=0 tx=0).stop 2023-03-03T08:45:44.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.966+0000 7f7e14ff6700 1 -- 172.21.15.7:0/1308873602 shutdown_connections 2023-03-03T08:45:44.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.966+0000 7f7e14ff6700 1 --2- 172.21.15.7:0/1308873602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e100ac220 0x7f7e100736f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:44.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.966+0000 7f7e14ff6700 1 --2- 172.21.15.7:0/1308873602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e10073c30 0x7f7e10152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:44.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.966+0000 7f7e14ff6700 1 -- 172.21.15.7:0/1308873602 >> 172.21.15.7:0/1308873602 conn(0x7f7e100a94d0 msgr2=0x7f7e100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:44.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.966+0000 7f7e14ff6700 1 -- 172.21.15.7:0/1308873602 shutdown_connections 2023-03-03T08:45:44.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.966+0000 7f7e14ff6700 1 -- 172.21.15.7:0/1308873602 wait complete. 2023-03-03T08:45:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.967+0000 7f7e14ff6700 1 Processor -- start 2023-03-03T08:45:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.967+0000 7f7e14ff6700 1 -- start start 2023-03-03T08:45:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e14ff6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e10073c30 0x7f7e10154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e14ff6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e100ac220 0x7f7e100c0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e14ff6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e100ace30 0x7f7e100c10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:44.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e14ff6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100c1600 con 0x7f7e100ace30 2023-03-03T08:45:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e14ff6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1006e130 con 0x7f7e10073c30 2023-03-03T08:45:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e14ff6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1006d400 con 0x7f7e100ac220 2023-03-03T08:45:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e100ace30 0x7f7e100c10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e100ace30 0x7f7e100c10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53122/0 (socket says 172.21.15.7:53122) 2023-03-03T08:45:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0ed9d700 1 -- 172.21.15.7:0/2664110329 learned_addr learned my addr 172.21.15.7:0/2664110329 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:45:44.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0e59c700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e10073c30 0x7f7e10154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0dd9b700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e100ac220 0x7f7e100c0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0ed9d700 1 -- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e100ac220 msgr2=0x7f7e100c0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0ed9d700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e100ac220 0x7f7e100c0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0ed9d700 1 -- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e10073c30 msgr2=0x7f7e10154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0ed9d700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e10073c30 0x7f7e10154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0ed9d700 1 -- 172.21.15.7:0/2664110329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e000094a0 con 0x7f7e100ace30 2023-03-03T08:45:44.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0e59c700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e10073c30 0x7f7e10154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:45:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.968+0000 7f7e0ed9d700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e100ace30 0x7f7e100c10c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800cb70 tx=0x7f7e0800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.969+0000 7f7dfeffd700 1 -- 172.21.15.7:0/2664110329 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0800aab0 con 0x7f7e100ace30 2023-03-03T08:45:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.969+0000 7f7dfeffd700 1 -- 172.21.15.7:0/2664110329 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0800ac10 con 0x7f7e100ace30 2023-03-03T08:45:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.969+0000 7f7e14ff6700 1 -- 172.21.15.7:0/2664110329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e040094a0 con 0x7f7e100ace30 2023-03-03T08:45:44.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.969+0000 7f7dfeffd700 1 -- 172.21.15.7:0/2664110329 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08012400 con 0x7f7e100ace30 2023-03-03T08:45:44.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.969+0000 7f7e14ff6700 1 -- 172.21.15.7:0/2664110329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e10155f80 con 0x7f7e100ace30 2023-03-03T08:45:44.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.970+0000 7f7e14ff6700 1 -- 172.21.15.7:0/2664110329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e10155ca0 con 0x7f7e100ace30 2023-03-03T08:45:44.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.970+0000 7f7dfeffd700 1 -- 172.21.15.7:0/2664110329 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7e08019030 con 0x7f7e100ace30 2023-03-03T08:45:44.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.970+0000 7f7dfeffd700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7df00682f0 0x7f7df006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:45:44.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.971+0000 7f7dfeffd700 1 -- 172.21.15.7:0/2664110329 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7e08085d70 con 0x7f7e100ace30 2023-03-03T08:45:44.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.971+0000 7f7e0e59c700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7df00682f0 0x7f7df006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:45:44.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.971+0000 7f7e0e59c700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7df00682f0 0x7f7df006a7a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e04000c00 tx=0x7f7e0401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:45:44.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:44.975+0000 7f7dfeffd700 1 -- 172.21.15.7:0/2664110329 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7e08052110 con 0x7f7e100ace30 2023-03-03T08:45:45.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.216+0000 7f7e14ff6700 1 -- 172.21.15.7:0/2664110329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e10156260 con 0x7f7e100ace30 2023-03-03T08:45:45.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.217+0000 7f7dfeffd700 1 -- 172.21.15.7:0/2664110329 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7e08055730 con 0x7f7e100ace30 2023-03-03T08:45:45.218 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:45:45.219 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:45:45.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 -- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7df00682f0 msgr2=0x7f7df006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:45.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7df00682f0 0x7f7df006a7a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7e04000c00 tx=0x7f7e0401c040 comp rx=0 tx=0).stop 2023-03-03T08:45:45.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 -- 172.21.15.7:0/2664110329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e100ace30 msgr2=0x7f7e100c10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:45:45.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e100ace30 0x7f7e100c10c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800cb70 tx=0x7f7e0800cf30 comp rx=0 tx=0).stop 2023-03-03T08:45:45.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 -- 172.21.15.7:0/2664110329 shutdown_connections 2023-03-03T08:45:45.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7df00682f0 0x7f7df006a7a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:45.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e100ace30 0x7f7e100c10c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:45.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e10073c30 0x7f7e10154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:45.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 --2- 172.21.15.7:0/2664110329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e100ac220 0x7f7e100c0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:45:45.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 -- 172.21.15.7:0/2664110329 >> 172.21.15.7:0/2664110329 conn(0x7f7e100a94d0 msgr2=0x7f7e10070720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:45:45.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 -- 172.21.15.7:0/2664110329 shutdown_connections 2023-03-03T08:45:45.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:45:45.219+0000 7f7e14ff6700 1 -- 172.21.15.7:0/2664110329 wait complete. 2023-03-03T08:45:45.223 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:45:45.233 INFO:tasks.mds_thrash.fs.[a]:waiting for 80 secs before thrashing 2023-03-03T08:46:05.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:05.470 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:10.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:46:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.919+0000 7f36792a7700 1 Processor -- start 2023-03-03T08:46:10.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.919+0000 7f36792a7700 1 -- start start 2023-03-03T08:46:10.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f36792a7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f36740c5c60 0x7f3674154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f36792a7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36740c7310 con 0x7f36740caa30 2023-03-03T08:46:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f36792a7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36740bfd10 con 0x7f36740c00e0 2023-03-03T08:46:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f36792a7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36740c60a0 con 0x7f36740c5c60 2023-03-03T08:46:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f36737fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f36740c5c60 0x7f3674154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f3672ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f36740caa30 0x7f36740c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47112/0 (socket says 172.21.15.7:47112) 2023-03-03T08:46:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f36737fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f36740c5c60 0x7f3674154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53136/0 (socket says 172.21.15.7:53136) 2023-03-03T08:46:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f3672ffd700 1 -- 172.21.15.7:0/145615152 learned_addr learned my addr 172.21.15.7:0/145615152 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:46:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f36737fe700 1 -- 172.21.15.7:0/145615152 >> v1:172.21.15.143:6789/0 conn(0x7f36740c00e0 legacy=0x7f367415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:46:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f36737fe700 1 -- 172.21.15.7:0/145615152 >> v1:172.21.15.7:6789/0 conn(0x7f36740caa30 legacy=0x7f36740c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f3671ffb700 1 -- 172.21.15.7:0/145615152 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483994582 0 0) 0x7f36740c7310 con 0x7f36740caa30 2023-03-03T08:46:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.920+0000 7f36737fe700 1 -- 172.21.15.7:0/145615152 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36740bfa30 con 0x7f36740c5c60 2023-03-03T08:46:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f36737fe700 1 --2- 172.21.15.7:0/145615152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 0x7f3674154ff0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f366c009d90 tx=0x7f366c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1bbf96670f9032a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:46:10.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f3671ffb700 1 -- 172.21.15.7:0/145615152 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c00e070 con 0x7f36740c5c60 2023-03-03T08:46:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f3671ffb700 1 -- 172.21.15.7:0/145615152 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c00bb90 con 0x7f36740c5c60 2023-03-03T08:46:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f3671ffb700 1 -- 172.21.15.7:0/145615152 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c003a20 con 0x7f36740c5c60 2023-03-03T08:46:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f36792a7700 1 -- 172.21.15.7:0/145615152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 msgr2=0x7f3674154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f36792a7700 1 --2- 172.21.15.7:0/145615152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 0x7f3674154ff0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f366c009d90 tx=0x7f366c00b330 comp rx=0 tx=0).stop 2023-03-03T08:46:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f36792a7700 1 -- 172.21.15.7:0/145615152 shutdown_connections 2023-03-03T08:46:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f36792a7700 1 --2- 172.21.15.7:0/145615152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 0x7f3674154ff0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:10.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f36792a7700 1 -- 172.21.15.7:0/145615152 >> 172.21.15.7:0/145615152 conn(0x7f3674066e40 msgr2=0x7f3674071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:46:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f36792a7700 1 -- 172.21.15.7:0/145615152 shutdown_connections 2023-03-03T08:46:10.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.921+0000 7f36792a7700 1 -- 172.21.15.7:0/145615152 wait complete. 2023-03-03T08:46:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36792a7700 1 Processor -- start 2023-03-03T08:46:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36792a7700 1 -- start start 2023-03-03T08:46:10.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36792a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36740c00e0 0x7f36741e9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36792a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 0x7f36741e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36792a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36740caa30 0x7f36740c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36792a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3674158390 con 0x7f36740c5c60 2023-03-03T08:46:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36792a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36740c3860 con 0x7f36740c00e0 2023-03-03T08:46:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36792a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367414d0f0 con 0x7f36740caa30 2023-03-03T08:46:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36727fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 0x7f36741e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:10.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36727fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 0x7f36741e9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53140/0 (socket says 172.21.15.7:53140) 2023-03-03T08:46:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.923+0000 7f36727fc700 1 -- 172.21.15.7:0/665305047 learned_addr learned my addr 172.21.15.7:0/665305047 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:46:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f3672ffd700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36740c00e0 0x7f36741e9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f36737fe700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36740caa30 0x7f36740c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f36727fc700 1 -- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36740caa30 msgr2=0x7f36740c19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f36727fc700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36740caa30 0x7f36740c19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f36727fc700 1 -- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36740c00e0 msgr2=0x7f36741e9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f36727fc700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36740c00e0 0x7f36741e9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f36727fc700 1 -- 172.21.15.7:0/665305047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c009a70 con 0x7f36740c5c60 2023-03-03T08:46:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f36727fc700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 0x7f36741e9ec0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f366400b450 tx=0x7f366400b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:46:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f365b7fe700 1 -- 172.21.15.7:0/665305047 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366400be60 con 0x7f36740c5c60 2023-03-03T08:46:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f365b7fe700 1 -- 172.21.15.7:0/665305047 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3664004020 con 0x7f36740c5c60 2023-03-03T08:46:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f365b7fe700 1 -- 172.21.15.7:0/665305047 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640043d0 con 0x7f36740c5c60 2023-03-03T08:46:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f36792a7700 1 -- 172.21.15.7:0/665305047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36741547f0 con 0x7f36740c5c60 2023-03-03T08:46:10.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.924+0000 7f36792a7700 1 -- 172.21.15.7:0/665305047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3674154510 con 0x7f36740c5c60 2023-03-03T08:46:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.925+0000 7f36792a7700 1 -- 172.21.15.7:0/665305047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3674154ad0 con 0x7f36740c5c60 2023-03-03T08:46:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.926+0000 7f365b7fe700 1 -- 172.21.15.7:0/665305047 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f366400f050 con 0x7f36740c5c60 2023-03-03T08:46:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.926+0000 7f365b7fe700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36600682f0 0x7f366006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:46:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.926+0000 7f365b7fe700 1 -- 172.21.15.7:0/665305047 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3664085720 con 0x7f36740c5c60 2023-03-03T08:46:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.926+0000 7f3672ffd700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36600682f0 0x7f366006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:46:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.927+0000 7f3672ffd700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36600682f0 0x7f366006a7a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3668007fd0 tx=0x7f3668007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:46:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:10.930+0000 7f365b7fe700 1 -- 172.21.15.7:0/665305047 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3664051ac0 con 0x7f36740c5c60 2023-03-03T08:46:11.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.167+0000 7f36792a7700 1 -- 172.21.15.7:0/665305047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3674074bb0 con 0x7f36740c5c60 2023-03-03T08:46:11.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.168+0000 7f365b7fe700 1 -- 172.21.15.7:0/665305047 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3664015090 con 0x7f36740c5c60 2023-03-03T08:46:11.171 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:46:11.171 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:46:11.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 -- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36600682f0 msgr2=0x7f366006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:11.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36600682f0 0x7f366006a7a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3668007fd0 tx=0x7f3668007de0 comp rx=0 tx=0).stop 2023-03-03T08:46:11.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 -- 172.21.15.7:0/665305047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 msgr2=0x7f36741e9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:46:11.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 0x7f36741e9ec0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f366400b450 tx=0x7f366400b760 comp rx=0 tx=0).stop 2023-03-03T08:46:11.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 -- 172.21.15.7:0/665305047 shutdown_connections 2023-03-03T08:46:11.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36740caa30 0x7f36740c19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:11.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36740c00e0 0x7f36741e9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:11.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36600682f0 0x7f366006a7a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:11.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 --2- 172.21.15.7:0/665305047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36740c5c60 0x7f36741e9ec0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:46:11.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 -- 172.21.15.7:0/665305047 >> 172.21.15.7:0/665305047 conn(0x7f3674066e40 msgr2=0x7f3674153da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:46:11.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 -- 172.21.15.7:0/665305047 shutdown_connections 2023-03-03T08:46:11.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:46:11.170+0000 7f36792a7700 1 -- 172.21.15.7:0/665305047 wait complete. 2023-03-03T08:46:11.175 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:46:11.185 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2023-03-03T08:46:35.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:46:35.696 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:05.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:47:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.375+0000 7f66caec8700 1 Processor -- start 2023-03-03T08:47:05.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.375+0000 7f66caec8700 1 -- start start 2023-03-03T08:47:05.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.376+0000 7f66caec8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 0x7f66c4152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:05.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.376+0000 7f66caec8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c414fd60 con 0x7f66c41562b0 2023-03-03T08:47:05.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.376+0000 7f66caec8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c406c030 con 0x7f66c40ab5e0 2023-03-03T08:47:05.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.376+0000 7f66caec8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c40c29c0 con 0x7f66c414f990 2023-03-03T08:47:05.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.376+0000 7f66c3fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f66c41562b0 0x7f66c406b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47166/0 (socket says 172.21.15.7:47166) 2023-03-03T08:47:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.376+0000 7f66c3fff700 1 -- 172.21.15.7:0/3420062049 learned_addr learned my addr 172.21.15.7:0/3420062049 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:47:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.376+0000 7f66c8c64700 1 --2- 172.21.15.7:0/3420062049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 0x7f66c4152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.376+0000 7f66c37fe700 1 -- 172.21.15.7:0/3420062049 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025016222 0 0) 0x7f66c414fd60 con 0x7f66c41562b0 2023-03-03T08:47:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.376+0000 7f66c37fe700 1 -- 172.21.15.7:0/3420062049 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b40035b0 con 0x7f66c41562b0 2023-03-03T08:47:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.377+0000 7f66c37fe700 1 -- 172.21.15.7:0/3420062049 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515067574 0 0) 0x7f66c406c030 con 0x7f66c40ab5e0 2023-03-03T08:47:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.377+0000 7f66c37fe700 1 -- 172.21.15.7:0/3420062049 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c414fd60 con 0x7f66c40ab5e0 2023-03-03T08:47:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.377+0000 7f66c37fe700 1 -- 172.21.15.7:0/3420062049 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4283647117 0 0) 0x7f66b40035b0 con 0x7f66c41562b0 2023-03-03T08:47:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.377+0000 7f66c8c64700 1 -- 172.21.15.7:0/3420062049 >> v1:172.21.15.143:6789/0 conn(0x7f66c40ab5e0 legacy=0x7f66c415cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.377+0000 7f66c8c64700 1 -- 172.21.15.7:0/3420062049 >> v1:172.21.15.7:6789/0 conn(0x7f66c41562b0 legacy=0x7f66c406b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.377+0000 7f66c8c64700 1 -- 172.21.15.7:0/3420062049 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c40c2400 con 0x7f66c414f990 2023-03-03T08:47:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.377+0000 7f66c8c64700 1 --2- 172.21.15.7:0/3420062049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 0x7f66c4152ed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009870 tx=0x7f66b8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57e55cf4c6df9e94 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:47:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.377+0000 7f66c37fe700 1 -- 172.21.15.7:0/3420062049 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0001670 con 0x7f66c414f990 2023-03-03T08:47:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.377+0000 7f66c37fe700 1 -- 172.21.15.7:0/3420062049 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b8004030 con 0x7f66c414f990 2023-03-03T08:47:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.377+0000 7f66c37fe700 1 -- 172.21.15.7:0/3420062049 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b80043c0 con 0x7f66c414f990 2023-03-03T08:47:05.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.378+0000 7f66caec8700 1 -- 172.21.15.7:0/3420062049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 msgr2=0x7f66c4152ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:05.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.378+0000 7f66caec8700 1 --2- 172.21.15.7:0/3420062049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 0x7f66c4152ed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009870 tx=0x7f66b8009b80 comp rx=0 tx=0).stop 2023-03-03T08:47:05.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.378+0000 7f66caec8700 1 -- 172.21.15.7:0/3420062049 shutdown_connections 2023-03-03T08:47:05.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.378+0000 7f66caec8700 1 --2- 172.21.15.7:0/3420062049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 0x7f66c4152ed0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:05.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.378+0000 7f66caec8700 1 -- 172.21.15.7:0/3420062049 >> 172.21.15.7:0/3420062049 conn(0x7f66c40c62e0 msgr2=0x7f66c40c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:47:05.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.378+0000 7f66caec8700 1 -- 172.21.15.7:0/3420062049 shutdown_connections 2023-03-03T08:47:05.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.378+0000 7f66caec8700 1 -- 172.21.15.7:0/3420062049 wait complete. 2023-03-03T08:47:05.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.379+0000 7f66caec8700 1 Processor -- start 2023-03-03T08:47:05.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.379+0000 7f66caec8700 1 -- start start 2023-03-03T08:47:05.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66caec8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c40ab5e0 0x7f66c4154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:05.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66caec8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 0x7f66c40aa580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:05.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66caec8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c41562b0 0x7f66c41e9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:05.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66caec8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c41edff0 con 0x7f66c40ab5e0 2023-03-03T08:47:05.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66caec8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c41ed450 con 0x7f66c41562b0 2023-03-03T08:47:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66caec8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c41ed6c0 con 0x7f66c414f990 2023-03-03T08:47:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c8c64700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c40ab5e0 0x7f66c4154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c8c64700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c40ab5e0 0x7f66c4154d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53190/0 (socket says 172.21.15.7:53190) 2023-03-03T08:47:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c8c64700 1 -- 172.21.15.7:0/2041054085 learned_addr learned my addr 172.21.15.7:0/2041054085 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:47:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c3fff700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 0x7f66c40aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c9465700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c41562b0 0x7f66c41e9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:05.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c8c64700 1 -- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 msgr2=0x7f66c40aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c8c64700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 0x7f66c40aa580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c8c64700 1 -- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c41562b0 msgr2=0x7f66c41e9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c8c64700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c41562b0 0x7f66c41e9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c8c64700 1 -- 172.21.15.7:0/2041054085 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b80094a0 con 0x7f66c40ab5e0 2023-03-03T08:47:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c9465700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c41562b0 0x7f66c41e9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:47:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c3fff700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 0x7f66c40aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:47:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.380+0000 7f66c8c64700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c40ab5e0 0x7f66c4154d00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f66b80097c0 tx=0x7f66b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:47:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.381+0000 7f66c17fa700 1 -- 172.21.15.7:0/2041054085 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b8003d70 con 0x7f66c40ab5e0 2023-03-03T08:47:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.381+0000 7f66c17fa700 1 -- 172.21.15.7:0/2041054085 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b801f070 con 0x7f66c40ab5e0 2023-03-03T08:47:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.381+0000 7f66caec8700 1 -- 172.21.15.7:0/2041054085 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c41523f0 con 0x7f66c40ab5e0 2023-03-03T08:47:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.381+0000 7f66c17fa700 1 -- 172.21.15.7:0/2041054085 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b8011870 con 0x7f66c40ab5e0 2023-03-03T08:47:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.381+0000 7f66caec8700 1 -- 172.21.15.7:0/2041054085 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c40c26e0 con 0x7f66c40ab5e0 2023-03-03T08:47:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.382+0000 7f66caec8700 1 -- 172.21.15.7:0/2041054085 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c41526d0 con 0x7f66c40ab5e0 2023-03-03T08:47:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.382+0000 7f66c17fa700 1 -- 172.21.15.7:0/2041054085 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f66b802a020 con 0x7f66c40ab5e0 2023-03-03T08:47:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.382+0000 7f66c17fa700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66a80682f0 0x7f66a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.382+0000 7f66c17fa700 1 -- 172.21.15.7:0/2041054085 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f66b8087580 con 0x7f66c40ab5e0 2023-03-03T08:47:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.383+0000 7f66c3fff700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66a80682f0 0x7f66a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.383+0000 7f66c3fff700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66a80682f0 0x7f66a806a7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f66b00017a0 tx=0x7f66b0001550 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:47:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.387+0000 7f66c17fa700 1 -- 172.21.15.7:0/2041054085 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f66b8053920 con 0x7f66c40ab5e0 2023-03-03T08:47:05.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.627+0000 7f66caec8700 1 -- 172.21.15.7:0/2041054085 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66c4074bb0 con 0x7f66c40ab5e0 2023-03-03T08:47:05.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.627+0000 7f66c17fa700 1 -- 172.21.15.7:0/2041054085 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f66b8056f40 con 0x7f66c40ab5e0 2023-03-03T08:47:05.629 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:47:05.630 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:47:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 -- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66a80682f0 msgr2=0x7f66a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66a80682f0 0x7f66a806a7a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f66b00017a0 tx=0x7f66b0001550 comp rx=0 tx=0).stop 2023-03-03T08:47:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 -- 172.21.15.7:0/2041054085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c40ab5e0 msgr2=0x7f66c4154d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:05.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c40ab5e0 0x7f66c4154d00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f66b80097c0 tx=0x7f66b8004750 comp rx=0 tx=0).stop 2023-03-03T08:47:05.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 -- 172.21.15.7:0/2041054085 shutdown_connections 2023-03-03T08:47:05.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c41562b0 0x7f66c41e9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:05.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66a80682f0 0x7f66a806a7a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:05.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c40ab5e0 0x7f66c4154d00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:05.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 --2- 172.21.15.7:0/2041054085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c414f990 0x7f66c40aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:05.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 -- 172.21.15.7:0/2041054085 >> 172.21.15.7:0/2041054085 conn(0x7f66c40c62e0 msgr2=0x7f66c40aba60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:47:05.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 -- 172.21.15.7:0/2041054085 shutdown_connections 2023-03-03T08:47:05.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:05.630+0000 7f66caec8700 1 -- 172.21.15.7:0/2041054085 wait complete. 2023-03-03T08:47:05.633 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:47:05.643 INFO:tasks.mds_thrash.fs.[a]:waiting for 96 secs before thrashing 2023-03-03T08:47:05.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:05.764 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:35.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:35.832 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:47:48.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:47:48.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.325+0000 7f3d41277700 1 Processor -- start 2023-03-03T08:47:48.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.325+0000 7f3d41277700 1 -- start start 2023-03-03T08:47:48.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.326+0000 7f3d41277700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d3c0c5c60 0x7f3d3c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:48.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.326+0000 7f3d41277700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c0c7310 con 0x7f3d3c0caa30 2023-03-03T08:47:48.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.326+0000 7f3d41277700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c0bfd10 con 0x7f3d3c0c00e0 2023-03-03T08:47:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.326+0000 7f3d41277700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3c0c60a0 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.326+0000 7f3d3b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d3c0c5c60 0x7f3d3c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.326+0000 7f3d3b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d3c0c5c60 0x7f3d3c154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53246/0 (socket says 172.21.15.7:53246) 2023-03-03T08:47:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.326+0000 7f3d3b7fe700 1 -- 172.21.15.7:0/1350873995 learned_addr learned my addr 172.21.15.7:0/1350873995 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:47:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.326+0000 7f3d3affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3d3c0caa30 0x7f3d3c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47224/0 (socket says 172.21.15.7:47224) 2023-03-03T08:47:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.326+0000 7f3d3b7fe700 1 -- 172.21.15.7:0/1350873995 >> v1:172.21.15.143:6789/0 conn(0x7f3d3c0c00e0 legacy=0x7f3d3c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:47:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.326+0000 7f3d3b7fe700 1 -- 172.21.15.7:0/1350873995 >> v1:172.21.15.7:6789/0 conn(0x7f3d3c0caa30 legacy=0x7f3d3c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d3b7fe700 1 -- 172.21.15.7:0/1350873995 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c0bfa30 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d39ffb700 1 -- 172.21.15.7:0/1350873995 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670951640 0 0) 0x7f3d3c0c7310 con 0x7f3d3c0caa30 2023-03-03T08:47:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d3b7fe700 1 --2- 172.21.15.7:0/1350873995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 0x7f3d3c154ff0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3d34009e50 tx=0x7f3d3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4421632247625a4f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:47:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d39ffb700 1 -- 172.21.15.7:0/1350873995 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3400e070 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d39ffb700 1 -- 172.21.15.7:0/1350873995 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d34003a20 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d39ffb700 1 -- 172.21.15.7:0/1350873995 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34003dd0 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d41277700 1 -- 172.21.15.7:0/1350873995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 msgr2=0x7f3d3c154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:48.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d41277700 1 --2- 172.21.15.7:0/1350873995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 0x7f3d3c154ff0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3d34009e50 tx=0x7f3d3400b330 comp rx=0 tx=0).stop 2023-03-03T08:47:48.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d41277700 1 -- 172.21.15.7:0/1350873995 shutdown_connections 2023-03-03T08:47:48.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d41277700 1 --2- 172.21.15.7:0/1350873995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 0x7f3d3c154ff0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:48.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d41277700 1 -- 172.21.15.7:0/1350873995 >> 172.21.15.7:0/1350873995 conn(0x7f3d3c066e40 msgr2=0x7f3d3c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:47:48.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.327+0000 7f3d41277700 1 -- 172.21.15.7:0/1350873995 shutdown_connections 2023-03-03T08:47:48.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.328+0000 7f3d41277700 1 -- 172.21.15.7:0/1350873995 wait complete. 2023-03-03T08:47:48.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.329+0000 7f3d41277700 1 Processor -- start 2023-03-03T08:47:48.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.329+0000 7f3d41277700 1 -- start start 2023-03-03T08:47:48.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.330+0000 7f3d41277700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 0x7f3d3c1ed140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:48.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.330+0000 7f3d41277700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 0x7f3d3c157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:48.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.330+0000 7f3d41277700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c0caa30 0x7f3d3c1583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:48.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.330+0000 7f3d41277700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1edc50 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.330+0000 7f3d41277700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1567f0 con 0x7f3d3c0c00e0 2023-03-03T08:47:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.330+0000 7f3d41277700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0c2a30 con 0x7f3d3c0caa30 2023-03-03T08:47:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.330+0000 7f3d3a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 0x7f3d3c157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.330+0000 7f3d3a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 0x7f3d3c157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53250/0 (socket says 172.21.15.7:53250) 2023-03-03T08:47:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.330+0000 7f3d3a7fc700 1 -- 172.21.15.7:0/3561370131 learned_addr learned my addr 172.21.15.7:0/3561370131 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:47:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d3affd700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 0x7f3d3c1ed140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d3b7fe700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c0caa30 0x7f3d3c1583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d3a7fc700 1 -- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c0caa30 msgr2=0x7f3d3c1583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d3a7fc700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c0caa30 0x7f3d3c1583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d3a7fc700 1 -- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 msgr2=0x7f3d3c1ed140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d3a7fc700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 0x7f3d3c1ed140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d3a7fc700 1 -- 172.21.15.7:0/3561370131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34009b30 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d3b7fe700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c0caa30 0x7f3d3c1583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:47:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d3affd700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 0x7f3d3c1ed140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:47:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d3a7fc700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 0x7f3d3c157e60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00b450 tx=0x7f3d2c00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:47:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d237fe700 1 -- 172.21.15.7:0/3561370131 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c004020 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d237fe700 1 -- 172.21.15.7:0/3561370131 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c004180 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d41277700 1 -- 172.21.15.7:0/3561370131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c1547f0 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d237fe700 1 -- 172.21.15.7:0/3561370131 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c004530 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.331+0000 7f3d41277700 1 -- 172.21.15.7:0/3561370131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3c154510 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.332+0000 7f3d41277700 1 -- 172.21.15.7:0/3561370131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3c154ad0 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.333+0000 7f3d237fe700 1 -- 172.21.15.7:0/3561370131 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d2c00f050 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.333+0000 7f3d237fe700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 0x7f3d2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:47:48.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.333+0000 7f3d237fe700 1 -- 172.21.15.7:0/3561370131 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3d2c085840 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.333+0000 7f3d3affd700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 0x7f3d2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:47:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.334+0000 7f3d3affd700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 0x7f3d2806a7a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3d30000f60 tx=0x7f3d30007cc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:47:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.337+0000 7f3d237fe700 1 -- 172.21.15.7:0/3561370131 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d2c051be0 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.576+0000 7f3d41277700 1 -- 172.21.15.7:0/3561370131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d3c156640 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.577+0000 7f3d237fe700 1 -- 172.21.15.7:0/3561370131 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3d2c015090 con 0x7f3d3c0c5c60 2023-03-03T08:47:48.579 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:47:48.579 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:47:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 -- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 msgr2=0x7f3d2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 0x7f3d2806a7a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3d30000f60 tx=0x7f3d30007cc0 comp rx=0 tx=0).stop 2023-03-03T08:47:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 -- 172.21.15.7:0/3561370131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 msgr2=0x7f3d3c157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:47:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 0x7f3d3c157e60 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00b450 tx=0x7f3d2c00b810 comp rx=0 tx=0).stop 2023-03-03T08:47:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 -- 172.21.15.7:0/3561370131 shutdown_connections 2023-03-03T08:47:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c0caa30 0x7f3d3c1583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 0x7f3d3c1ed140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 0x7f3d2806a7a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 --2- 172.21.15.7:0/3561370131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0c5c60 0x7f3d3c157e60 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:47:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 -- 172.21.15.7:0/3561370131 >> 172.21.15.7:0/3561370131 conn(0x7f3d3c066e40 msgr2=0x7f3d3c0c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:47:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 -- 172.21.15.7:0/3561370131 shutdown_connections 2023-03-03T08:47:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:47:48.579+0000 7f3d41277700 1 -- 172.21.15.7:0/3561370131 wait complete. 2023-03-03T08:47:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:47:48.593 INFO:tasks.mds_thrash.fs.[b]:waiting for 118 secs before thrashing 2023-03-03T08:48:05.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:05.963 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:36.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:36.034 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:48:41.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:48:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.782+0000 7fcc2702c700 1 Processor -- start 2023-03-03T08:48:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.782+0000 7fcc2702c700 1 -- start start 2023-03-03T08:48:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc2702c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcc200c5140 0x7fcc20157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc2702c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc201580e0 0x7fcc2006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc2702c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2006dad0 con 0x7fcc20151de0 2023-03-03T08:48:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc2702c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2006e370 con 0x7fcc200c5140 2023-03-03T08:48:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc2702c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2006ddb0 con 0x7fcc201580e0 2023-03-03T08:48:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc255c9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc201580e0 0x7fcc2006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc1ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcc200c5140 0x7fcc20157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc1ffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcc200c5140 0x7fcc20157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:49062/0 (socket says 172.21.15.7:49062) 2023-03-03T08:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc24dc8700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fcc20151de0 0x7fcc200c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50788/0 (socket says 172.21.15.7:50788) 2023-03-03T08:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc1ffff700 1 -- 172.21.15.7:0/2702562429 learned_addr learned my addr 172.21.15.7:0/2702562429 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.783+0000 7fcc255c9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc201580e0 0x7fcc2006a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:33130/0 (socket says 172.21.15.7:33130) 2023-03-03T08:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.784+0000 7fcc255c9700 1 -- 172.21.15.7:0/2702562429 >> v2:172.21.15.143:3300/0 conn(0x7fcc200c5140 msgr2=0x7fcc20157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.784+0000 7fcc255c9700 1 --2- 172.21.15.7:0/2702562429 >> v2:172.21.15.143:3300/0 conn(0x7fcc200c5140 0x7fcc20157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.784+0000 7fcc255c9700 1 -- 172.21.15.7:0/2702562429 >> v1:172.21.15.143:6789/0 conn(0x7fcc20151de0 legacy=0x7fcc200c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.784+0000 7fcc255c9700 1 -- 172.21.15.7:0/2702562429 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2006e090 con 0x7fcc201580e0 2023-03-03T08:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.784+0000 7fcc1ffff700 1 --2- 172.21.15.7:0/2702562429 >> v2:172.21.15.143:3300/0 conn(0x7fcc200c5140 0x7fcc20157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.784+0000 7fcc255c9700 1 --2- 172.21.15.7:0/2702562429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc201580e0 0x7fcc2006a2b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009d90 tx=0x7fcc1800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2284e13779a92740 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.784+0000 7fcc1f7fe700 1 -- 172.21.15.7:0/2702562429 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1800e070 con 0x7fcc201580e0 2023-03-03T08:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.784+0000 7fcc1f7fe700 1 -- 172.21.15.7:0/2702562429 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1800bb20 con 0x7fcc201580e0 2023-03-03T08:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.784+0000 7fcc1f7fe700 1 -- 172.21.15.7:0/2702562429 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18012430 con 0x7fcc201580e0 2023-03-03T08:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.785+0000 7fcc2702c700 1 -- 172.21.15.7:0/2702562429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc201580e0 msgr2=0x7fcc2006a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.785+0000 7fcc2702c700 1 --2- 172.21.15.7:0/2702562429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc201580e0 0x7fcc2006a2b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009d90 tx=0x7fcc1800b330 comp rx=0 tx=0).stop 2023-03-03T08:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.785+0000 7fcc2702c700 1 -- 172.21.15.7:0/2702562429 shutdown_connections 2023-03-03T08:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.785+0000 7fcc2702c700 1 --2- 172.21.15.7:0/2702562429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc201580e0 0x7fcc2006a2b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.785+0000 7fcc2702c700 1 --2- 172.21.15.7:0/2702562429 >> v2:172.21.15.143:3300/0 conn(0x7fcc200c5140 0x7fcc20157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.785+0000 7fcc2702c700 1 -- 172.21.15.7:0/2702562429 >> 172.21.15.7:0/2702562429 conn(0x7fcc200a9430 msgr2=0x7fcc200c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.785+0000 7fcc2702c700 1 -- 172.21.15.7:0/2702562429 shutdown_connections 2023-03-03T08:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.785+0000 7fcc2702c700 1 -- 172.21.15.7:0/2702562429 wait complete. 2023-03-03T08:48:41.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.786+0000 7fcc2702c700 1 Processor -- start 2023-03-03T08:48:41.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.786+0000 7fcc2702c700 1 -- start start 2023-03-03T08:48:41.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc2702c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc200c5140 0x7fcc200a7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:41.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc2702c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc20151de0 0x7fcc20146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:41.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc2702c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc201580e0 0x7fcc201472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:41.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc2702c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2014e470 con 0x7fcc20151de0 2023-03-03T08:48:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc2702c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2006cf10 con 0x7fcc201580e0 2023-03-03T08:48:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc2702c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2006d6d0 con 0x7fcc200c5140 2023-03-03T08:48:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc1ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc20151de0 0x7fcc20146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc1ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc20151de0 0x7fcc20146d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53294/0 (socket says 172.21.15.7:53294) 2023-03-03T08:48:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc1ffff700 1 -- 172.21.15.7:0/3508880944 learned_addr learned my addr 172.21.15.7:0/3508880944 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:48:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc255c9700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc201580e0 0x7fcc201472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc24dc8700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc200c5140 0x7fcc200a7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc1ffff700 1 -- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc200c5140 msgr2=0x7fcc200a7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc1ffff700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc200c5140 0x7fcc200a7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc1ffff700 1 -- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc201580e0 msgr2=0x7fcc201472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc1ffff700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc201580e0 0x7fcc201472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc1ffff700 1 -- 172.21.15.7:0/3508880944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c009550 con 0x7fcc20151de0 2023-03-03T08:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc24dc8700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc200c5140 0x7fcc200a7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.787+0000 7fcc1ffff700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc20151de0 0x7fcc20146d90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00ebe0 tx=0x7fcc0c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.788+0000 7fcc1d7fa700 1 -- 172.21.15.7:0/3508880944 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00c850 con 0x7fcc20151de0 2023-03-03T08:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.788+0000 7fcc1d7fa700 1 -- 172.21.15.7:0/3508880944 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0c00c9b0 con 0x7fcc20151de0 2023-03-03T08:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.788+0000 7fcc2702c700 1 -- 172.21.15.7:0/3508880944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc18009a70 con 0x7fcc20151de0 2023-03-03T08:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.788+0000 7fcc1d7fa700 1 -- 172.21.15.7:0/3508880944 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00cd60 con 0x7fcc20151de0 2023-03-03T08:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.788+0000 7fcc2702c700 1 -- 172.21.15.7:0/3508880944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc200c1c10 con 0x7fcc20151de0 2023-03-03T08:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.789+0000 7fcc2702c700 1 -- 172.21.15.7:0/3508880944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc200c4c20 con 0x7fcc20151de0 2023-03-03T08:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.789+0000 7fcc1d7fa700 1 -- 172.21.15.7:0/3508880944 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcc0c019030 con 0x7fcc20151de0 2023-03-03T08:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.790+0000 7fcc1d7fa700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc040682f0 0x7fcc0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.790+0000 7fcc1d7fa700 1 -- 172.21.15.7:0/3508880944 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcc0c086250 con 0x7fcc20151de0 2023-03-03T08:48:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.790+0000 7fcc24dc8700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc040682f0 0x7fcc0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:48:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.790+0000 7fcc24dc8700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc040682f0 0x7fcc0406a7a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcc14000f60 tx=0x7fcc14007dd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:48:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:41.794+0000 7fcc1d7fa700 1 -- 172.21.15.7:0/3508880944 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcc0c0524a0 con 0x7fcc20151de0 2023-03-03T08:48:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.030+0000 7fcc2702c700 1 -- 172.21.15.7:0/3508880944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc2006ef40 con 0x7fcc20151de0 2023-03-03T08:48:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.031+0000 7fcc1d7fa700 1 -- 172.21.15.7:0/3508880944 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcc0c055ac0 con 0x7fcc20151de0 2023-03-03T08:48:42.033 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:48:42.033 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:48:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 -- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc040682f0 msgr2=0x7fcc0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc040682f0 0x7fcc0406a7a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcc14000f60 tx=0x7fcc14007dd0 comp rx=0 tx=0).stop 2023-03-03T08:48:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 -- 172.21.15.7:0/3508880944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc20151de0 msgr2=0x7fcc20146d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:48:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc20151de0 0x7fcc20146d90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00ebe0 tx=0x7fcc0c00eef0 comp rx=0 tx=0).stop 2023-03-03T08:48:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 -- 172.21.15.7:0/3508880944 shutdown_connections 2023-03-03T08:48:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc201580e0 0x7fcc201472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc200c5140 0x7fcc200a7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc040682f0 0x7fcc0406a7a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 --2- 172.21.15.7:0/3508880944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc20151de0 0x7fcc20146d90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:48:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 -- 172.21.15.7:0/3508880944 >> 172.21.15.7:0/3508880944 conn(0x7fcc200a9430 msgr2=0x7fcc200bfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:48:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 -- 172.21.15.7:0/3508880944 shutdown_connections 2023-03-03T08:48:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:48:42.033+0000 7fcc2702c700 1 -- 172.21.15.7:0/3508880944 wait complete. 2023-03-03T08:48:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:48:42.046 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2023-03-03T08:49:06.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:06.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:36.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:36.212 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:49:46.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:49:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.731+0000 7ff8821f4700 1 Processor -- start 2023-03-03T08:49:46.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.731+0000 7ff8821f4700 1 -- start start 2023-03-03T08:49:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.732+0000 7ff8821f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c151d40 0x7ff87c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.732+0000 7ff8821f4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff87c0c2930 0x7ff87c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.732+0000 7ff8821f4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c06d430 con 0x7ff87c073c30 2023-03-03T08:49:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.732+0000 7ff8821f4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87c06c2a0 con 0x7ff87c0c2930 2023-03-03T08:49:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.732+0000 7ff8821f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c1586e0 con 0x7ff87c151d40 2023-03-03T08:49:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c151d40 0x7ff87c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87affd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff87c0c2930 0x7ff87c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c151d40 0x7ff87c0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:33160/0 (socket says 172.21.15.7:33160) 2023-03-03T08:49:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87b7fe700 1 -- 172.21.15.7:0/3156730480 learned_addr learned my addr 172.21.15.7:0/3156730480 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:49:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87bfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ff87c073c30 0x7ff87c157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50824/0 (socket says 172.21.15.7:50824) 2023-03-03T08:49:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87b7fe700 1 -- 172.21.15.7:0/3156730480 >> v2:172.21.15.143:3300/0 conn(0x7ff87c0c2930 msgr2=0x7ff87c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:49:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87b7fe700 1 --2- 172.21.15.7:0/3156730480 >> v2:172.21.15.143:3300/0 conn(0x7ff87c0c2930 0x7ff87c157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87b7fe700 1 -- 172.21.15.7:0/3156730480 >> v1:172.21.15.143:6789/0 conn(0x7ff87c073c30 legacy=0x7ff87c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87b7fe700 1 -- 172.21.15.7:0/3156730480 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c06cb00 con 0x7ff87c151d40 2023-03-03T08:49:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87b7fe700 1 --2- 172.21.15.7:0/3156730480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c151d40 0x7ff87c0736f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff86c0098f0 tx=0x7ff86c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc84480eaf1f4fe6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:49:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87a7fc700 1 -- 172.21.15.7:0/3156730480 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c011830 con 0x7ff87c151d40 2023-03-03T08:49:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87a7fc700 1 -- 172.21.15.7:0/3156730480 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c011990 con 0x7ff87c151d40 2023-03-03T08:49:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.733+0000 7ff87a7fc700 1 -- 172.21.15.7:0/3156730480 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c011d20 con 0x7ff87c151d40 2023-03-03T08:49:46.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.734+0000 7ff8821f4700 1 -- 172.21.15.7:0/3156730480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c151d40 msgr2=0x7ff87c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.734+0000 7ff8821f4700 1 --2- 172.21.15.7:0/3156730480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c151d40 0x7ff87c0736f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff86c0098f0 tx=0x7ff86c009c00 comp rx=0 tx=0).stop 2023-03-03T08:49:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.734+0000 7ff8821f4700 1 -- 172.21.15.7:0/3156730480 shutdown_connections 2023-03-03T08:49:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.734+0000 7ff8821f4700 1 --2- 172.21.15.7:0/3156730480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c151d40 0x7ff87c0736f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.734+0000 7ff8821f4700 1 --2- 172.21.15.7:0/3156730480 >> v2:172.21.15.143:3300/0 conn(0x7ff87c0c2930 0x7ff87c157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.734+0000 7ff8821f4700 1 -- 172.21.15.7:0/3156730480 >> 172.21.15.7:0/3156730480 conn(0x7ff87c0a94d0 msgr2=0x7ff87c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:49:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.734+0000 7ff8821f4700 1 -- 172.21.15.7:0/3156730480 shutdown_connections 2023-03-03T08:49:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.734+0000 7ff8821f4700 1 -- 172.21.15.7:0/3156730480 wait complete. 2023-03-03T08:49:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.735+0000 7ff8821f4700 1 Processor -- start 2023-03-03T08:49:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.735+0000 7ff8821f4700 1 -- start start 2023-03-03T08:49:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff8821f4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff87c073c30 0x7ff87c152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff8821f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c0c2930 0x7ff87c14e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff8821f4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff87c151d40 0x7ff87c0c0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff8821f4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0c5be0 con 0x7ff87c073c30 2023-03-03T08:49:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff8821f4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c153110 con 0x7ff87c151d40 2023-03-03T08:49:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff8821f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c14e860 con 0x7ff87c0c2930 2023-03-03T08:49:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff87b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff87c073c30 0x7ff87c152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff87b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff87c073c30 0x7ff87c152bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53326/0 (socket says 172.21.15.7:53326) 2023-03-03T08:49:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff87b7fe700 1 -- 172.21.15.7:0/3008325254 learned_addr learned my addr 172.21.15.7:0/3008325254 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:49:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff87affd700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c0c2930 0x7ff87c14e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff87bfff700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff87c151d40 0x7ff87c0c0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff87b7fe700 1 -- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c0c2930 msgr2=0x7ff87c14e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff87b7fe700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c0c2930 0x7ff87c14e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff87b7fe700 1 -- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff87c151d40 msgr2=0x7ff87c0c0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.736+0000 7ff87b7fe700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff87c151d40 0x7ff87c0c0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.737+0000 7ff87b7fe700 1 -- 172.21.15.7:0/3008325254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c009550 con 0x7ff87c073c30 2023-03-03T08:49:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.737+0000 7ff87b7fe700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff87c073c30 0x7ff87c152bd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff87c158820 tx=0x7ff86c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:49:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.737+0000 7ff863fff700 1 -- 172.21.15.7:0/3008325254 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c004450 con 0x7ff87c073c30 2023-03-03T08:49:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.737+0000 7ff863fff700 1 -- 172.21.15.7:0/3008325254 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c0045d0 con 0x7ff87c073c30 2023-03-03T08:49:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.737+0000 7ff8821f4700 1 -- 172.21.15.7:0/3008325254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c06d0c0 con 0x7ff87c073c30 2023-03-03T08:49:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.737+0000 7ff863fff700 1 -- 172.21.15.7:0/3008325254 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c0038e0 con 0x7ff87c073c30 2023-03-03T08:49:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.737+0000 7ff8821f4700 1 -- 172.21.15.7:0/3008325254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c06cde0 con 0x7ff87c073c30 2023-03-03T08:49:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.738+0000 7ff8821f4700 1 -- 172.21.15.7:0/3008325254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c06bea0 con 0x7ff87c073c30 2023-03-03T08:49:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.739+0000 7ff863fff700 1 -- 172.21.15.7:0/3008325254 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff86c02a030 con 0x7ff87c073c30 2023-03-03T08:49:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.739+0000 7ff863fff700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8680682f0 0x7ff86806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:49:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.740+0000 7ff863fff700 1 -- 172.21.15.7:0/3008325254 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff86c0874e0 con 0x7ff87c073c30 2023-03-03T08:49:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.740+0000 7ff87affd700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8680682f0 0x7ff86806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:49:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.740+0000 7ff87affd700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8680682f0 0x7ff86806a7a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0bdd90 tx=0x7ff870009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:49:46.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.743+0000 7ff863fff700 1 -- 172.21.15.7:0/3008325254 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff86c053810 con 0x7ff87c073c30 2023-03-03T08:49:46.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.979+0000 7ff8821f4700 1 -- 172.21.15.7:0/3008325254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff87c074bb0 con 0x7ff87c073c30 2023-03-03T08:49:46.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.979+0000 7ff863fff700 1 -- 172.21.15.7:0/3008325254 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff86c015090 con 0x7ff87c073c30 2023-03-03T08:49:46.981 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:49:46.981 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:49:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 -- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8680682f0 msgr2=0x7ff86806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8680682f0 0x7ff86806a7a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0bdd90 tx=0x7ff870009210 comp rx=0 tx=0).stop 2023-03-03T08:49:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 -- 172.21.15.7:0/3008325254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff87c073c30 msgr2=0x7ff87c152bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:49:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff87c073c30 0x7ff87c152bd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff87c158820 tx=0x7ff86c004750 comp rx=0 tx=0).stop 2023-03-03T08:49:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 -- 172.21.15.7:0/3008325254 shutdown_connections 2023-03-03T08:49:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff87c151d40 0x7ff87c0c0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8680682f0 0x7ff86806a7a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff87c073c30 0x7ff87c152bd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 --2- 172.21.15.7:0/3008325254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff87c0c2930 0x7ff87c14e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:49:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 -- 172.21.15.7:0/3008325254 >> 172.21.15.7:0/3008325254 conn(0x7ff87c0a94d0 msgr2=0x7ff87c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:49:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 -- 172.21.15.7:0/3008325254 shutdown_connections 2023-03-03T08:49:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:49:46.981+0000 7ff8821f4700 1 -- 172.21.15.7:0/3008325254 wait complete. 2023-03-03T08:49:46.985 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:49:46.995 INFO:tasks.mds_thrash.fs.[b]:waiting for 38 secs before thrashing 2023-03-03T08:50:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T08:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T08:50:01.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:50:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.186+0000 7f2b52226700 1 Processor -- start 2023-03-03T08:50:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.186+0000 7f2b52226700 1 -- start start 2023-03-03T08:50:01.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b52226700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2b4c152e30 0x7f2b4c0c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:01.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b52226700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c0beab0 0x7f2b4c0c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:01.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b52226700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2b4c155d80 0x7f2b4c156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:01.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b52226700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b4c1523b0 con 0x7f2b4c152e30 2023-03-03T08:50:01.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b52226700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b4c072010 con 0x7f2b4c155d80 2023-03-03T08:50:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b52226700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c0aaaf0 con 0x7f2b4c0beab0 2023-03-03T08:50:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2b4c152e30 0x7f2b4c0c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2b4c152e30 0x7f2b4c0c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53342/0 (socket says 172.21.15.7:53342) 2023-03-03T08:50:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4b7fe700 1 -- 172.21.15.7:0/2774237204 learned_addr learned my addr 172.21.15.7:0/2774237204 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:50:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4affd700 1 --2- 172.21.15.7:0/2774237204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c0beab0 0x7f2b4c0c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4bfff700 1 --2- 172.21.15.7:0/2774237204 >> v2:172.21.15.143:3300/0 conn(0x7f2b4c155d80 0x7f2b4c156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:01.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4b7fe700 1 -- 172.21.15.7:0/2774237204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c0beab0 msgr2=0x7f2b4c0c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:50:01.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4b7fe700 1 --2- 172.21.15.7:0/2774237204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c0beab0 0x7f2b4c0c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4b7fe700 1 -- 172.21.15.7:0/2774237204 >> v2:172.21.15.143:3300/0 conn(0x7f2b4c155d80 msgr2=0x7f2b4c156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:50:01.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4b7fe700 1 --2- 172.21.15.7:0/2774237204 >> v2:172.21.15.143:3300/0 conn(0x7f2b4c155d80 0x7f2b4c156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4b7fe700 1 -- 172.21.15.7:0/2774237204 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c0aadd0 con 0x7f2b4c152e30 2023-03-03T08:50:01.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.187+0000 7f2b4affd700 1 --2- 172.21.15.7:0/2774237204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c0beab0 0x7f2b4c0c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:50:01.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b4b7fe700 1 --2- 172.21.15.7:0/2774237204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c152e30 0x7f2b4c0c43d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0c5970 tx=0x7f2b3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f3e0d65c8ed0b3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:01.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b4a7fc700 1 -- 172.21.15.7:0/2774237204 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c00b870 con 0x7f2b4c152e30 2023-03-03T08:50:01.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b4a7fc700 1 -- 172.21.15.7:0/2774237204 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c00b9d0 con 0x7f2b4c152e30 2023-03-03T08:50:01.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b4a7fc700 1 -- 172.21.15.7:0/2774237204 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c00bd60 con 0x7f2b4c152e30 2023-03-03T08:50:01.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b52226700 1 -- 172.21.15.7:0/2774237204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c152e30 msgr2=0x7f2b4c0c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:01.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b52226700 1 --2- 172.21.15.7:0/2774237204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c152e30 0x7f2b4c0c43d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0c5970 tx=0x7f2b3c009bd0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b52226700 1 -- 172.21.15.7:0/2774237204 shutdown_connections 2023-03-03T08:50:01.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b52226700 1 --2- 172.21.15.7:0/2774237204 >> v2:172.21.15.143:3300/0 conn(0x7f2b4c155d80 0x7f2b4c156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b52226700 1 --2- 172.21.15.7:0/2774237204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c152e30 0x7f2b4c0c43d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b52226700 1 --2- 172.21.15.7:0/2774237204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c0beab0 0x7f2b4c0c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b52226700 1 -- 172.21.15.7:0/2774237204 >> 172.21.15.7:0/2774237204 conn(0x7f2b4c0736f0 msgr2=0x7f2b4c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:50:01.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b52226700 1 -- 172.21.15.7:0/2774237204 shutdown_connections 2023-03-03T08:50:01.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.188+0000 7f2b52226700 1 -- 172.21.15.7:0/2774237204 wait complete. 2023-03-03T08:50:01.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.190+0000 7f2b52226700 1 Processor -- start 2023-03-03T08:50:01.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.190+0000 7f2b52226700 1 -- start start 2023-03-03T08:50:01.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.190+0000 7f2b52226700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c0beab0 0x7f2b4c154080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:01.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b52226700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b4c152e30 0x7f2b4c0c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:01.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b52226700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c155d80 0x7f2b4c1e7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:01.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b52226700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c0ad710 con 0x7f2b4c0beab0 2023-03-03T08:50:01.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b52226700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c1e76e0 con 0x7f2b4c152e30 2023-03-03T08:50:01.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b52226700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c1545c0 con 0x7f2b4c155d80 2023-03-03T08:50:01.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c0beab0 0x7f2b4c154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:01.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c0beab0 0x7f2b4c154080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53348/0 (socket says 172.21.15.7:53348) 2023-03-03T08:50:01.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4b7fe700 1 -- 172.21.15.7:0/2281077710 learned_addr learned my addr 172.21.15.7:0/2281077710 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:50:01.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4bfff700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c155d80 0x7f2b4c1e7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:01.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4affd700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b4c152e30 0x7f2b4c0c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:01.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4b7fe700 1 -- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c155d80 msgr2=0x7f2b4c1e7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:01.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4b7fe700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c155d80 0x7f2b4c1e7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4b7fe700 1 -- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b4c152e30 msgr2=0x7f2b4c0c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:01.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4b7fe700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b4c152e30 0x7f2b4c0c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4b7fe700 1 -- 172.21.15.7:0/2281077710 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c009550 con 0x7f2b4c0beab0 2023-03-03T08:50:01.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4bfff700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c155d80 0x7f2b4c1e7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:50:01.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4affd700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b4c152e30 0x7f2b4c0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:50:01.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b4b7fe700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c0beab0 0x7f2b4c154080 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c000c00 tx=0x7f2b3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:01.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b43fff700 1 -- 172.21.15.7:0/2281077710 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c004540 con 0x7f2b4c0beab0 2023-03-03T08:50:01.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b43fff700 1 -- 172.21.15.7:0/2281077710 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c0036a0 con 0x7f2b4c0beab0 2023-03-03T08:50:01.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b52226700 1 -- 172.21.15.7:0/2281077710 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4c156760 con 0x7f2b4c0beab0 2023-03-03T08:50:01.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b43fff700 1 -- 172.21.15.7:0/2281077710 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c003a50 con 0x7f2b4c0beab0 2023-03-03T08:50:01.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.191+0000 7f2b52226700 1 -- 172.21.15.7:0/2281077710 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b4c071c70 con 0x7f2b4c0beab0 2023-03-03T08:50:01.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.192+0000 7f2b52226700 1 -- 172.21.15.7:0/2281077710 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4c1568f0 con 0x7f2b4c0beab0 2023-03-03T08:50:01.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.193+0000 7f2b43fff700 1 -- 172.21.15.7:0/2281077710 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2b3c00f050 con 0x7f2b4c0beab0 2023-03-03T08:50:01.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.193+0000 7f2b43fff700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b2c0682f0 0x7f2b2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:01.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.193+0000 7f2b43fff700 1 -- 172.21.15.7:0/2281077710 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2b3c0879e0 con 0x7f2b4c0beab0 2023-03-03T08:50:01.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.193+0000 7f2b4affd700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b2c0682f0 0x7f2b2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:01.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.194+0000 7f2b4affd700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b2c0682f0 0x7f2b2c06a7a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c14e620 tx=0x7f2b34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:01.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.198+0000 7f2b43fff700 1 -- 172.21.15.7:0/2281077710 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2b3c053d80 con 0x7f2b4c0beab0 2023-03-03T08:50:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.437+0000 7f2b52226700 1 -- 172.21.15.7:0/2281077710 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b4c074bb0 con 0x7f2b4c0beab0 2023-03-03T08:50:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.437+0000 7f2b43fff700 1 -- 172.21.15.7:0/2281077710 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2b3c0573a0 con 0x7f2b4c0beab0 2023-03-03T08:50:01.439 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:50:01.439 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:50:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 -- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b2c0682f0 msgr2=0x7f2b2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b2c0682f0 0x7f2b2c06a7a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c14e620 tx=0x7f2b34008040 comp rx=0 tx=0).stop 2023-03-03T08:50:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 -- 172.21.15.7:0/2281077710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c0beab0 msgr2=0x7f2b4c154080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c0beab0 0x7f2b4c154080 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c000c00 tx=0x7f2b3c004750 comp rx=0 tx=0).stop 2023-03-03T08:50:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 -- 172.21.15.7:0/2281077710 shutdown_connections 2023-03-03T08:50:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b4c155d80 0x7f2b4c1e7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b2c0682f0 0x7f2b2c06a7a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b4c0beab0 0x7f2b4c154080 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 --2- 172.21.15.7:0/2281077710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b4c152e30 0x7f2b4c0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 -- 172.21.15.7:0/2281077710 >> 172.21.15.7:0/2281077710 conn(0x7f2b4c0736f0 msgr2=0x7f2b4c0c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:50:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 -- 172.21.15.7:0/2281077710 shutdown_connections 2023-03-03T08:50:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:01.439+0000 7f2b52226700 1 -- 172.21.15.7:0/2281077710 wait complete. 2023-03-03T08:50:01.443 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:50:01.452 INFO:tasks.mds_thrash.fs.[a]:waiting for 37 secs before thrashing 2023-03-03T08:50:06.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:06.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:24.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:50:25.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.134+0000 7f76d3ce3700 1 Processor -- start 2023-03-03T08:50:25.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.134+0000 7f76d3ce3700 1 -- start start 2023-03-03T08:50:25.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d3ce3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f76cc06d870 0x7f76cc1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:25.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d3ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 0x7f76cc0c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:25.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d3ce3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc0c7a60 con 0x7f76cc0c1da0 2023-03-03T08:50:25.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d3ce3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0c2af0 con 0x7f76cc06d870 2023-03-03T08:50:25.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d3ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0c3620 con 0x7f76cc0c3e30 2023-03-03T08:50:25.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d127e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f76cc0c1da0 0x7f76cc0c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47354/0 (socket says 172.21.15.7:47354) 2023-03-03T08:50:25.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d127e700 1 -- 172.21.15.7:0/2618718935 learned_addr learned my addr 172.21.15.7:0/2618718935 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:50:25.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d1a7f700 1 --2- 172.21.15.7:0/2618718935 >> v2:172.21.15.143:3300/0 conn(0x7f76cc06d870 0x7f76cc1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:25.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d2280700 1 --2- 172.21.15.7:0/2618718935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 0x7f76cc0c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:25.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d0a7d700 1 -- 172.21.15.7:0/2618718935 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364237563 0 0) 0x7f76cc0c7a60 con 0x7f76cc0c1da0 2023-03-03T08:50:25.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d0a7d700 1 -- 172.21.15.7:0/2618718935 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b80035b0 con 0x7f76cc0c1da0 2023-03-03T08:50:25.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d2280700 1 -- 172.21.15.7:0/2618718935 >> v2:172.21.15.143:3300/0 conn(0x7f76cc06d870 msgr2=0x7f76cc1550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:50:25.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d2280700 1 --2- 172.21.15.7:0/2618718935 >> v2:172.21.15.143:3300/0 conn(0x7f76cc06d870 0x7f76cc1550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d2280700 1 -- 172.21.15.7:0/2618718935 >> v1:172.21.15.7:6789/0 conn(0x7f76cc0c1da0 legacy=0x7f76cc0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:25.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d2280700 1 -- 172.21.15.7:0/2618718935 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc1545e0 con 0x7f76cc0c3e30 2023-03-03T08:50:25.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.135+0000 7f76d1a7f700 1 --2- 172.21.15.7:0/2618718935 >> v2:172.21.15.143:3300/0 conn(0x7f76cc06d870 0x7f76cc1550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:50:25.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d2280700 1 --2- 172.21.15.7:0/2618718935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 0x7f76cc0c4270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009d90 tx=0x7f76c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c25888d37f4ea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:25.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d0a7d700 1 -- 172.21.15.7:0/2618718935 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c400e070 con 0x7f76cc0c3e30 2023-03-03T08:50:25.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d0a7d700 1 -- 172.21.15.7:0/2618718935 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c400bb20 con 0x7f76cc0c3e30 2023-03-03T08:50:25.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d0a7d700 1 -- 172.21.15.7:0/2618718935 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4012430 con 0x7f76cc0c3e30 2023-03-03T08:50:25.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2618718935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 msgr2=0x7f76cc0c4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:25.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d3ce3700 1 --2- 172.21.15.7:0/2618718935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 0x7f76cc0c4270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009d90 tx=0x7f76c400b330 comp rx=0 tx=0).stop 2023-03-03T08:50:25.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2618718935 shutdown_connections 2023-03-03T08:50:25.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d3ce3700 1 --2- 172.21.15.7:0/2618718935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 0x7f76cc0c4270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d3ce3700 1 --2- 172.21.15.7:0/2618718935 >> v2:172.21.15.143:3300/0 conn(0x7f76cc06d870 0x7f76cc1550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2618718935 >> 172.21.15.7:0/2618718935 conn(0x7f76cc06c110 msgr2=0x7f76cc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:50:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2618718935 shutdown_connections 2023-03-03T08:50:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.136+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2618718935 wait complete. 2023-03-03T08:50:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d3ce3700 1 Processor -- start 2023-03-03T08:50:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d3ce3700 1 -- start start 2023-03-03T08:50:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d3ce3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc06d870 0x7f76cc1505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:25.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d3ce3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc0c1da0 0x7f76cc1ecc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d3ce3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 0x7f76cc152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d3ce3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc1e9720 con 0x7f76cc06d870 2023-03-03T08:50:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d3ce3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc1ed520 con 0x7f76cc0c1da0 2023-03-03T08:50:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d3ce3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc150bb0 con 0x7f76cc0c3e30 2023-03-03T08:50:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d1a7f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc06d870 0x7f76cc1505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:25.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d1a7f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc06d870 0x7f76cc1505d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53378/0 (socket says 172.21.15.7:53378) 2023-03-03T08:50:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.138+0000 7f76d1a7f700 1 -- 172.21.15.7:0/2062843213 learned_addr learned my addr 172.21.15.7:0/2062843213 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:50:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76d127e700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc0c1da0 0x7f76cc1ecc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76d2280700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 0x7f76cc152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76d1a7f700 1 -- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 msgr2=0x7f76cc152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76d1a7f700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 0x7f76cc152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76d1a7f700 1 -- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc0c1da0 msgr2=0x7f76cc1ecc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:25.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76d1a7f700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc0c1da0 0x7f76cc1ecc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76d1a7f700 1 -- 172.21.15.7:0/2062843213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c80094a0 con 0x7f76cc06d870 2023-03-03T08:50:25.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76d1a7f700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc06d870 0x7f76cc1505d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f76c800ebe0 tx=0x7f76c800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:25.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76c27fc700 1 -- 172.21.15.7:0/2062843213 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c800c850 con 0x7f76cc06d870 2023-03-03T08:50:25.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76c27fc700 1 -- 172.21.15.7:0/2062843213 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c800c9b0 con 0x7f76cc06d870 2023-03-03T08:50:25.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76c27fc700 1 -- 172.21.15.7:0/2062843213 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c800cd40 con 0x7f76cc06d870 2023-03-03T08:50:25.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2062843213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c4009a70 con 0x7f76cc06d870 2023-03-03T08:50:25.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.139+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2062843213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc154ba0 con 0x7f76cc06d870 2023-03-03T08:50:25.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.140+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2062843213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc1548c0 con 0x7f76cc06d870 2023-03-03T08:50:25.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.141+0000 7f76c27fc700 1 -- 172.21.15.7:0/2062843213 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f76c8019030 con 0x7f76cc06d870 2023-03-03T08:50:25.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.141+0000 7f76c27fc700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 0x7f76b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:25.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.141+0000 7f76c27fc700 1 -- 172.21.15.7:0/2062843213 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f76c80860b0 con 0x7f76cc06d870 2023-03-03T08:50:25.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.141+0000 7f76d127e700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 0x7f76b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:25.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.142+0000 7f76d127e700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 0x7f76b006a7a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f76cc1e7560 tx=0x7f76bc007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:25.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.146+0000 7f76c27fc700 1 -- 172.21.15.7:0/2062843213 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f76c8052450 con 0x7f76cc06d870 2023-03-03T08:50:25.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.385+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2062843213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76cc06dc40 con 0x7f76cc06d870 2023-03-03T08:50:25.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.385+0000 7f76c27fc700 1 -- 172.21.15.7:0/2062843213 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f76c8017090 con 0x7f76cc06d870 2023-03-03T08:50:25.387 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:50:25.387 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:50:25.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.387+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 msgr2=0x7f76b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:25.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.387+0000 7f76d3ce3700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 0x7f76b006a7a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f76cc1e7560 tx=0x7f76bc007dc0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.387+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2062843213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc06d870 msgr2=0x7f76cc1505d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:25.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.387+0000 7f76d3ce3700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc06d870 0x7f76cc1505d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f76c800ebe0 tx=0x7f76c800eef0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.387+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2062843213 shutdown_connections 2023-03-03T08:50:25.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.387+0000 7f76d3ce3700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc0c3e30 0x7f76cc152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.387+0000 7f76d3ce3700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 0x7f76b006a7a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.387+0000 7f76d3ce3700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc06d870 0x7f76cc1505d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.387+0000 7f76d3ce3700 1 --2- 172.21.15.7:0/2062843213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc0c1da0 0x7f76cc1ecc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:25.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.387+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2062843213 >> 172.21.15.7:0/2062843213 conn(0x7f76cc06c110 msgr2=0x7f76cc153c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:50:25.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.388+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2062843213 shutdown_connections 2023-03-03T08:50:25.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:25.388+0000 7f76d3ce3700 1 -- 172.21.15.7:0/2062843213 wait complete. 2023-03-03T08:50:25.391 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:50:25.401 INFO:tasks.mds_thrash.fs.[b]:waiting for 70 secs before thrashing 2023-03-03T08:50:36.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:36.679 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:50:38.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:50:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.591+0000 7fb30868f700 1 Processor -- start 2023-03-03T08:50:38.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.591+0000 7fb30868f700 1 -- start start 2023-03-03T08:50:38.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.592+0000 7fb30868f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb300155390 0x7fb3000c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.592+0000 7fb30868f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb30006e020 0x7fb300071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.592+0000 7fb30868f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb300154910 con 0x7fb300072520 2023-03-03T08:50:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.592+0000 7fb30868f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb300070b90 con 0x7fb300155390 2023-03-03T08:50:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.592+0000 7fb30868f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb300151320 con 0x7fb30006e020 2023-03-03T08:50:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.592+0000 7fb30642b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb300155390 0x7fb3000c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:38.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.592+0000 7fb30642b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb300155390 0x7fb3000c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53390/0 (socket says 172.21.15.7:53390) 2023-03-03T08:50:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.592+0000 7fb30642b700 1 -- 172.21.15.7:0/2477320196 learned_addr learned my addr 172.21.15.7:0/2477320196 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:50:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.592+0000 7fb306c2c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb300072520 0x7fb300151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47374/0 (socket says 172.21.15.7:47374) 2023-03-03T08:50:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.593+0000 7fb305c2a700 1 --2- 172.21.15.7:0/2477320196 >> v2:172.21.15.143:3300/0 conn(0x7fb30006e020 0x7fb300071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.593+0000 7fb30642b700 1 -- 172.21.15.7:0/2477320196 >> v2:172.21.15.143:3300/0 conn(0x7fb30006e020 msgr2=0x7fb300071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:50:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.593+0000 7fb30642b700 1 --2- 172.21.15.7:0/2477320196 >> v2:172.21.15.143:3300/0 conn(0x7fb30006e020 0x7fb300071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.593+0000 7fb30642b700 1 -- 172.21.15.7:0/2477320196 >> v1:172.21.15.7:6789/0 conn(0x7fb300072520 legacy=0x7fb300151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.593+0000 7fb30642b700 1 -- 172.21.15.7:0/2477320196 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3000702a0 con 0x7fb300155390 2023-03-03T08:50:38.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.593+0000 7fb305429700 1 -- 172.21.15.7:0/2477320196 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205085603 0 0) 0x7fb300154910 con 0x7fb300072520 2023-03-03T08:50:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.593+0000 7fb30642b700 1 --2- 172.21.15.7:0/2477320196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 0x7fb3000c3e30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb3000ac9e0 tx=0x7fb2fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843aa67cbc2828aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.593+0000 7fb305429700 1 -- 172.21.15.7:0/2477320196 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc00b870 con 0x7fb300155390 2023-03-03T08:50:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.593+0000 7fb305429700 1 -- 172.21.15.7:0/2477320196 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2fc00b9d0 con 0x7fb300155390 2023-03-03T08:50:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.593+0000 7fb305429700 1 -- 172.21.15.7:0/2477320196 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc00bd60 con 0x7fb300155390 2023-03-03T08:50:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.594+0000 7fb30868f700 1 -- 172.21.15.7:0/2477320196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 msgr2=0x7fb3000c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.594+0000 7fb30868f700 1 --2- 172.21.15.7:0/2477320196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 0x7fb3000c3e30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb3000ac9e0 tx=0x7fb2fc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:50:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.594+0000 7fb30868f700 1 -- 172.21.15.7:0/2477320196 shutdown_connections 2023-03-03T08:50:38.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.594+0000 7fb30868f700 1 --2- 172.21.15.7:0/2477320196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 0x7fb3000c3e30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:38.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.594+0000 7fb30868f700 1 --2- 172.21.15.7:0/2477320196 >> v2:172.21.15.143:3300/0 conn(0x7fb30006e020 0x7fb300071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:38.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.594+0000 7fb30868f700 1 -- 172.21.15.7:0/2477320196 >> 172.21.15.7:0/2477320196 conn(0x7fb30006c110 msgr2=0x7fb30006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:50:38.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.594+0000 7fb30868f700 1 -- 172.21.15.7:0/2477320196 shutdown_connections 2023-03-03T08:50:38.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.594+0000 7fb30868f700 1 -- 172.21.15.7:0/2477320196 wait complete. 2023-03-03T08:50:38.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.595+0000 7fb30868f700 1 Processor -- start 2023-03-03T08:50:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.595+0000 7fb30868f700 1 -- start start 2023-03-03T08:50:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb30868f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb30006e020 0x7fb30014dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb30868f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb300072520 0x7fb300150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb30868f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 0x7fb30014c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb30868f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3001e9610 con 0x7fb300155390 2023-03-03T08:50:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb30868f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30014e870 con 0x7fb300072520 2023-03-03T08:50:38.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb30868f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30014e2d0 con 0x7fb30006e020 2023-03-03T08:50:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb306c2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 0x7fb30014c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb306c2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 0x7fb30014c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53400/0 (socket says 172.21.15.7:53400) 2023-03-03T08:50:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb306c2c700 1 -- 172.21.15.7:0/789484182 learned_addr learned my addr 172.21.15.7:0/789484182 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:50:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb30642b700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb30006e020 0x7fb30014dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb305c2a700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb300072520 0x7fb300150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb306c2c700 1 -- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb30006e020 msgr2=0x7fb30014dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:38.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb306c2c700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb30006e020 0x7fb30014dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb306c2c700 1 -- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb300072520 msgr2=0x7fb300150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb306c2c700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb300072520 0x7fb300150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb306c2c700 1 -- 172.21.15.7:0/789484182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc009550 con 0x7fb300155390 2023-03-03T08:50:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb305c2a700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb300072520 0x7fb300150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:50:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb306c2c700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 0x7fb30014c080 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8002230 tx=0x7fb2f800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.596+0000 7fb2f2ffd700 1 -- 172.21.15.7:0/789484182 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f800ac20 con 0x7fb300155390 2023-03-03T08:50:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.597+0000 7fb2f2ffd700 1 -- 172.21.15.7:0/789484182 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f800ada0 con 0x7fb300155390 2023-03-03T08:50:38.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.597+0000 7fb2f2ffd700 1 -- 172.21.15.7:0/789484182 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8007310 con 0x7fb300155390 2023-03-03T08:50:38.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.597+0000 7fb30868f700 1 -- 172.21.15.7:0/789484182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb300070820 con 0x7fb300155390 2023-03-03T08:50:38.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.597+0000 7fb30868f700 1 -- 172.21.15.7:0/789484182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb300070540 con 0x7fb300155390 2023-03-03T08:50:38.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.598+0000 7fb30868f700 1 -- 172.21.15.7:0/789484182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb300155760 con 0x7fb300155390 2023-03-03T08:50:38.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.598+0000 7fb2f2ffd700 1 -- 172.21.15.7:0/789484182 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb2f8010070 con 0x7fb300155390 2023-03-03T08:50:38.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.598+0000 7fb2f2ffd700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2e40682f0 0x7fb2e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:50:38.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.598+0000 7fb2f2ffd700 1 -- 172.21.15.7:0/789484182 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb2f8085b40 con 0x7fb300155390 2023-03-03T08:50:38.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.599+0000 7fb30642b700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2e40682f0 0x7fb2e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:50:38.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.599+0000 7fb30642b700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2e40682f0 0x7fb2e406a7a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc014040 tx=0x7fb2fc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:50:38.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.603+0000 7fb2f2ffd700 1 -- 172.21.15.7:0/789484182 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb2f8051ee0 con 0x7fb300155390 2023-03-03T08:50:38.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.840+0000 7fb30868f700 1 -- 172.21.15.7:0/789484182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb300074bb0 con 0x7fb300155390 2023-03-03T08:50:38.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.840+0000 7fb2f2ffd700 1 -- 172.21.15.7:0/789484182 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb2f8055500 con 0x7fb300155390 2023-03-03T08:50:38.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:50:38.843 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:50:38.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.842+0000 7fb30868f700 1 -- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2e40682f0 msgr2=0x7fb2e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:38.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.842+0000 7fb30868f700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2e40682f0 0x7fb2e406a7a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc014040 tx=0x7fb2fc008030 comp rx=0 tx=0).stop 2023-03-03T08:50:38.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.842+0000 7fb30868f700 1 -- 172.21.15.7:0/789484182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 msgr2=0x7fb30014c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:50:38.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.842+0000 7fb30868f700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 0x7fb30014c080 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8002230 tx=0x7fb2f800a060 comp rx=0 tx=0).stop 2023-03-03T08:50:38.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.843+0000 7fb30868f700 1 -- 172.21.15.7:0/789484182 shutdown_connections 2023-03-03T08:50:38.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.843+0000 7fb30868f700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2e40682f0 0x7fb2e406a7a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:38.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.843+0000 7fb30868f700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb300155390 0x7fb30014c080 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:38.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.843+0000 7fb30868f700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb30006e020 0x7fb30014dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:38.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.843+0000 7fb30868f700 1 --2- 172.21.15.7:0/789484182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb300072520 0x7fb300150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:50:38.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.843+0000 7fb30868f700 1 -- 172.21.15.7:0/789484182 >> 172.21.15.7:0/789484182 conn(0x7fb30006c110 msgr2=0x7fb30014ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:50:38.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.843+0000 7fb30868f700 1 -- 172.21.15.7:0/789484182 shutdown_connections 2023-03-03T08:50:38.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:50:38.843+0000 7fb30868f700 1 -- 172.21.15.7:0/789484182 wait complete. 2023-03-03T08:50:38.847 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:50:38.857 INFO:tasks.mds_thrash.fs.[a]:waiting for 118 secs before thrashing 2023-03-03T08:51:06.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:06.751 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:35.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:51:35.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.540+0000 7fb34cd30700 1 Processor -- start 2023-03-03T08:51:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.540+0000 7fb34cd30700 1 -- start start 2023-03-03T08:51:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.541+0000 7fb34cd30700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb3480aad20 0x7fb348157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.541+0000 7fb34cd30700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3480a9c20 con 0x7fb3480aa030 2023-03-03T08:51:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.541+0000 7fb34cd30700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb348154f10 con 0x7fb34814b0d0 2023-03-03T08:51:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.541+0000 7fb34cd30700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3480ac3c0 con 0x7fb3480aad20 2023-03-03T08:51:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.541+0000 7fb346d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb3480aa030 0x7fb348158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47442/0 (socket says 172.21.15.7:47442) 2023-03-03T08:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.541+0000 7fb346d9d700 1 -- 172.21.15.7:0/2498176523 learned_addr learned my addr 172.21.15.7:0/2498176523 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.541+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063179917 0 0) 0x7fb3480a9c20 con 0x7fb3480aa030 2023-03-03T08:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3340035b0 con 0x7fb3480aa030 2023-03-03T08:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb345d9b700 1 --2- 172.21.15.7:0/2498176523 >> v2:172.21.15.143:3300/0 conn(0x7fb3480aad20 0x7fb348157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 937430069 0 0) 0x7fb3340035b0 con 0x7fb3480aa030 2023-03-03T08:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 >> v2:172.21.15.143:3300/0 conn(0x7fb3480aad20 msgr2=0x7fb348157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 --2- 172.21.15.7:0/2498176523 >> v2:172.21.15.143:3300/0 conn(0x7fb3480aad20 0x7fb348157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 >> v1:172.21.15.143:6789/0 conn(0x7fb34814b0d0 legacy=0x7fb348071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34806fc70 con 0x7fb3480aa030 2023-03-03T08:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb3400019f0 con 0x7fb3480aa030 2023-03-03T08:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 >> v1:172.21.15.7:6789/0 conn(0x7fb3480aa030 legacy=0x7fb348158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334003a70 0x7fb334003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb334004850 0x7fb334004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334005250 0x7fb33400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480ac3c0 con 0x7fb334004850 2023-03-03T08:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33400c190 con 0x7fb334003a70 2023-03-03T08:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33400c350 con 0x7fb334005250 2023-03-03T08:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb345d9b700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb334004850 0x7fb334004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb34659c700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334003a70 0x7fb334003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb346d9d700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334005250 0x7fb33400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb345d9b700 1 -- 172.21.15.7:0/2498176523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334005250 msgr2=0x7fb33400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb345d9b700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334005250 0x7fb33400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb345d9b700 1 -- 172.21.15.7:0/2498176523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334003a70 msgr2=0x7fb334003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb345d9b700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334003a70 0x7fb334003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb345d9b700 1 -- 172.21.15.7:0/2498176523 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3340035b0 con 0x7fb334004850 2023-03-03T08:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb346d9d700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb334005250 0x7fb33400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.542+0000 7fb346d9d700 1 -- 172.21.15.7:0/2498176523 reap_dead start 2023-03-03T08:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb345d9b700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb334004850 0x7fb334004cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb3480ab1a0 tx=0x7fb334005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b1c15d4357db42 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33800e6c0 con 0x7fb334004850 2023-03-03T08:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33800e820 con 0x7fb334004850 2023-03-03T08:51:35.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb34559a700 1 -- 172.21.15.7:0/2498176523 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33800eb90 con 0x7fb334004850 2023-03-03T08:51:35.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb34cd30700 1 -- 172.21.15.7:0/2498176523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb334004850 msgr2=0x7fb334004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb34cd30700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb334004850 0x7fb334004cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb3480ab1a0 tx=0x7fb334005880 comp rx=0 tx=0).stop 2023-03-03T08:51:35.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb34cd30700 1 -- 172.21.15.7:0/2498176523 shutdown_connections 2023-03-03T08:51:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb34cd30700 1 --2- 172.21.15.7:0/2498176523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb334004850 0x7fb334004cc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb34cd30700 1 -- 172.21.15.7:0/2498176523 >> 172.21.15.7:0/2498176523 conn(0x7fb3480c8900 msgr2=0x7fb3480c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:51:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb34cd30700 1 -- 172.21.15.7:0/2498176523 shutdown_connections 2023-03-03T08:51:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.543+0000 7fb34cd30700 1 -- 172.21.15.7:0/2498176523 wait complete. 2023-03-03T08:51:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.545+0000 7fb34cd30700 1 Processor -- start 2023-03-03T08:51:35.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.545+0000 7fb34cd30700 1 -- start start 2023-03-03T08:51:35.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.545+0000 7fb34cd30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334004850 0x7fb348156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.545+0000 7fb34cd30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3480c7ad0 0x7fb3481ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.545+0000 7fb34cd30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3480c7fe0 0x7fb3481ec1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.545+0000 7fb34cd30700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3481eb7d0 con 0x7fb3480c7ad0 2023-03-03T08:51:35.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.545+0000 7fb34cd30700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480c73e0 con 0x7fb334004850 2023-03-03T08:51:35.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.545+0000 7fb34cd30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480c7100 con 0x7fb3480c7fe0 2023-03-03T08:51:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.545+0000 7fb345d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3480c7ad0 0x7fb3481ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb345d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3480c7ad0 0x7fb3481ebc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53472/0 (socket says 172.21.15.7:53472) 2023-03-03T08:51:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb345d9b700 1 -- 172.21.15.7:0/2635827891 learned_addr learned my addr 172.21.15.7:0/2635827891 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:51:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb34659c700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334004850 0x7fb348156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb346d9d700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3480c7fe0 0x7fb3481ec1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb345d9b700 1 -- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3480c7fe0 msgr2=0x7fb3481ec1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb345d9b700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3480c7fe0 0x7fb3481ec1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb345d9b700 1 -- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334004850 msgr2=0x7fb348156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb345d9b700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334004850 0x7fb348156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb345d9b700 1 -- 172.21.15.7:0/2635827891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34806f6b0 con 0x7fb3480c7ad0 2023-03-03T08:51:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb346d9d700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3480c7fe0 0x7fb3481ec1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:51:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb345d9b700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3480c7ad0 0x7fb3481ebc60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb338000c00 tx=0x7fb3380047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:51:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb33effd700 1 -- 172.21.15.7:0/2635827891 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33800e6c0 con 0x7fb3480c7ad0 2023-03-03T08:51:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb33effd700 1 -- 172.21.15.7:0/2635827891 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3380038c0 con 0x7fb3480c7ad0 2023-03-03T08:51:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb33effd700 1 -- 172.21.15.7:0/2635827891 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338006e60 con 0x7fb3480c7ad0 2023-03-03T08:51:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb34cd30700 1 -- 172.21.15.7:0/2635827891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb348155160 con 0x7fb3480c7ad0 2023-03-03T08:51:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.546+0000 7fb34cd30700 1 -- 172.21.15.7:0/2635827891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34806f990 con 0x7fb3480c7ad0 2023-03-03T08:51:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.547+0000 7fb34cd30700 1 -- 172.21.15.7:0/2635827891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3481eb980 con 0x7fb3480c7ad0 2023-03-03T08:51:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.548+0000 7fb33effd700 1 -- 172.21.15.7:0/2635827891 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb338025070 con 0x7fb3480c7ad0 2023-03-03T08:51:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.548+0000 7fb33effd700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb328068320 0x7fb32806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.548+0000 7fb33effd700 1 -- 172.21.15.7:0/2635827891 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb3380879c0 con 0x7fb3480c7ad0 2023-03-03T08:51:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.548+0000 7fb34659c700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb328068320 0x7fb32806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.549+0000 7fb34659c700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb328068320 0x7fb32806a7d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb330000c00 tx=0x7fb330009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:51:35.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.552+0000 7fb33effd700 1 -- 172.21.15.7:0/2635827891 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb3481eb980 con 0x7fb3480c7ad0 2023-03-03T08:51:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.791+0000 7fb34cd30700 1 -- 172.21.15.7:0/2635827891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb348074bb0 con 0x7fb3480c7ad0 2023-03-03T08:51:35.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.792+0000 7fb33effd700 1 -- 172.21.15.7:0/2635827891 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb338053d60 con 0x7fb3480c7ad0 2023-03-03T08:51:35.793 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:51:35.794 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:51:35.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 -- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb328068320 msgr2=0x7fb32806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb328068320 0x7fb32806a7d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb330000c00 tx=0x7fb330009040 comp rx=0 tx=0).stop 2023-03-03T08:51:35.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 -- 172.21.15.7:0/2635827891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3480c7ad0 msgr2=0x7fb3481ebc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3480c7ad0 0x7fb3481ebc60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb338000c00 tx=0x7fb3380047c0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 -- 172.21.15.7:0/2635827891 shutdown_connections 2023-03-03T08:51:35.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3480c7fe0 0x7fb3481ec1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb334004850 0x7fb348156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb328068320 0x7fb32806a7d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 --2- 172.21.15.7:0/2635827891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3480c7ad0 0x7fb3481ebc60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 -- 172.21.15.7:0/2635827891 >> 172.21.15.7:0/2635827891 conn(0x7fb3480c8900 msgr2=0x7fb348157cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:51:35.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 -- 172.21.15.7:0/2635827891 shutdown_connections 2023-03-03T08:51:35.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.794+0000 7fb34cd30700 1 -- 172.21.15.7:0/2635827891 wait complete. 2023-03-03T08:51:35.797 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:51:35.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:51:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.930+0000 7ff8a70d3700 1 Processor -- start 2023-03-03T08:51:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.930+0000 7ff8a70d3700 1 -- start start 2023-03-03T08:51:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a70d3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff8a00ab0c0 0x7ff8a00c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a70d3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8a006a2b0 0x7ff8a006a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a70d3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a0154530 con 0x7ff8a0157a40 2023-03-03T08:51:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a70d3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a0154810 con 0x7ff8a00ab0c0 2023-03-03T08:51:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a70d3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a0154af0 con 0x7ff8a006a2b0 2023-03-03T08:51:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a4e6f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff8a00ab0c0 0x7ff8a00c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a5670700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff8a0157a40 0x7ff8a0157ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47462/0 (socket says 172.21.15.7:47462) 2023-03-03T08:51:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a5670700 1 -- 172.21.15.7:0/3301817307 learned_addr learned my addr 172.21.15.7:0/3301817307 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:51:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff89ffff700 1 --2- 172.21.15.7:0/3301817307 >> v2:172.21.15.143:3300/0 conn(0x7ff8a006a2b0 0x7ff8a006a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff89f7fe700 1 -- 172.21.15.7:0/3301817307 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897017493 0 0) 0x7ff8a0154530 con 0x7ff8a0157a40 2023-03-03T08:51:35.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff89f7fe700 1 -- 172.21.15.7:0/3301817307 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8900035b0 con 0x7ff8a0157a40 2023-03-03T08:51:35.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a4e6f700 1 -- 172.21.15.7:0/3301817307 >> v2:172.21.15.143:3300/0 conn(0x7ff8a006a2b0 msgr2=0x7ff8a006a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:51:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a4e6f700 1 --2- 172.21.15.7:0/3301817307 >> v2:172.21.15.143:3300/0 conn(0x7ff8a006a2b0 0x7ff8a006a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a4e6f700 1 -- 172.21.15.7:0/3301817307 >> v1:172.21.15.7:6789/0 conn(0x7ff8a0157a40 legacy=0x7ff8a0157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.931+0000 7ff8a4e6f700 1 -- 172.21.15.7:0/3301817307 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a0154dd0 con 0x7ff8a00ab0c0 2023-03-03T08:51:35.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff8a4e6f700 1 --2- 172.21.15.7:0/3301817307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a00ab0c0 0x7ff8a00c1500 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff8a006f4d0 tx=0x7ff894009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f6ec17b43068218 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:51:35.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff89f7fe700 1 -- 172.21.15.7:0/3301817307 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894011830 con 0x7ff8a00ab0c0 2023-03-03T08:51:35.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff89f7fe700 1 -- 172.21.15.7:0/3301817307 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff894011990 con 0x7ff8a00ab0c0 2023-03-03T08:51:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff89f7fe700 1 -- 172.21.15.7:0/3301817307 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894011d20 con 0x7ff8a00ab0c0 2023-03-03T08:51:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff8a70d3700 1 -- 172.21.15.7:0/3301817307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a00ab0c0 msgr2=0x7ff8a00c1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff8a70d3700 1 --2- 172.21.15.7:0/3301817307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a00ab0c0 0x7ff8a00c1500 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff8a006f4d0 tx=0x7ff894009b20 comp rx=0 tx=0).stop 2023-03-03T08:51:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff8a70d3700 1 -- 172.21.15.7:0/3301817307 shutdown_connections 2023-03-03T08:51:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff8a70d3700 1 --2- 172.21.15.7:0/3301817307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a00ab0c0 0x7ff8a00c1500 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff8a70d3700 1 --2- 172.21.15.7:0/3301817307 >> v2:172.21.15.143:3300/0 conn(0x7ff8a006a2b0 0x7ff8a006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff8a70d3700 1 -- 172.21.15.7:0/3301817307 >> 172.21.15.7:0/3301817307 conn(0x7ff8a00a9430 msgr2=0x7ff8a00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:51:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff8a70d3700 1 -- 172.21.15.7:0/3301817307 shutdown_connections 2023-03-03T08:51:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.932+0000 7ff8a70d3700 1 -- 172.21.15.7:0/3301817307 wait complete. 2023-03-03T08:51:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.933+0000 7ff8a70d3700 1 Processor -- start 2023-03-03T08:51:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.933+0000 7ff8a70d3700 1 -- start start 2023-03-03T08:51:35.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.934+0000 7ff8a70d3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8a006a2b0 0x7ff8a00c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.934+0000 7ff8a70d3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8a00ab0c0 0x7ff8a00c38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.934+0000 7ff8a70d3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a0157a40 0x7ff8a014acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.934+0000 7ff8a70d3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a0155af0 con 0x7ff8a0157a40 2023-03-03T08:51:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.934+0000 7ff8a70d3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00c34b0 con 0x7ff8a00ab0c0 2023-03-03T08:51:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.934+0000 7ff8a70d3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00c31a0 con 0x7ff8a006a2b0 2023-03-03T08:51:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.934+0000 7ff8a5670700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a0157a40 0x7ff8a014acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.934+0000 7ff8a5670700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a0157a40 0x7ff8a014acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53488/0 (socket says 172.21.15.7:53488) 2023-03-03T08:51:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.934+0000 7ff8a5670700 1 -- 172.21.15.7:0/1660463123 learned_addr learned my addr 172.21.15.7:0/1660463123 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:51:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff89ffff700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8a00ab0c0 0x7ff8a00c38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff8a4e6f700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8a006a2b0 0x7ff8a00c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff8a5670700 1 -- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8a006a2b0 msgr2=0x7ff8a00c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff8a5670700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8a006a2b0 0x7ff8a00c00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff8a5670700 1 -- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8a00ab0c0 msgr2=0x7ff8a00c38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff8a5670700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8a00ab0c0 0x7ff8a00c38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff8a5670700 1 -- 172.21.15.7:0/1660463123 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8940094a0 con 0x7ff8a0157a40 2023-03-03T08:51:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff89ffff700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8a00ab0c0 0x7ff8a00c38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:51:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff8a5670700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a0157a40 0x7ff8a014acb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff89800cb70 tx=0x7ff89800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:51:35.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff89d7fa700 1 -- 172.21.15.7:0/1660463123 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89800ab10 con 0x7ff8a0157a40 2023-03-03T08:51:35.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff89d7fa700 1 -- 172.21.15.7:0/1660463123 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89800ac70 con 0x7ff8a0157a40 2023-03-03T08:51:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff8a70d3700 1 -- 172.21.15.7:0/1660463123 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a01585f0 con 0x7ff8a0157a40 2023-03-03T08:51:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff89d7fa700 1 -- 172.21.15.7:0/1660463123 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898012400 con 0x7ff8a0157a40 2023-03-03T08:51:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.935+0000 7ff8a70d3700 1 -- 172.21.15.7:0/1660463123 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a01550b0 con 0x7ff8a0157a40 2023-03-03T08:51:35.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.936+0000 7ff8a70d3700 1 -- 172.21.15.7:0/1660463123 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a00ab490 con 0x7ff8a0157a40 2023-03-03T08:51:35.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.936+0000 7ff89d7fa700 1 -- 172.21.15.7:0/1660463123 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff898019030 con 0x7ff8a0157a40 2023-03-03T08:51:35.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.937+0000 7ff89d7fa700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8840682f0 0x7ff88406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:51:35.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.937+0000 7ff89d7fa700 1 -- 172.21.15.7:0/1660463123 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff898085e10 con 0x7ff8a0157a40 2023-03-03T08:51:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.937+0000 7ff8a4e6f700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8840682f0 0x7ff88406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:51:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.938+0000 7ff8a4e6f700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8840682f0 0x7ff88406a7a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00c5200 tx=0x7ff894008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:51:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:35.941+0000 7ff89d7fa700 1 -- 172.21.15.7:0/1660463123 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff8980521b0 con 0x7ff8a0157a40 2023-03-03T08:51:36.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.180+0000 7ff8a70d3700 1 -- 172.21.15.7:0/1660463123 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8a00c35f0 con 0x7ff8a0157a40 2023-03-03T08:51:36.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.181+0000 7ff89d7fa700 1 -- 172.21.15.7:0/1660463123 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff8a00c35f0 con 0x7ff8a0157a40 2023-03-03T08:51:36.183 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:51:36.183 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:51:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 -- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8840682f0 msgr2=0x7ff88406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8840682f0 0x7ff88406a7a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00c5200 tx=0x7ff894008030 comp rx=0 tx=0).stop 2023-03-03T08:51:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 -- 172.21.15.7:0/1660463123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a0157a40 msgr2=0x7ff8a014acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:51:36.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a0157a40 0x7ff8a014acb0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff89800cb70 tx=0x7ff89800cf30 comp rx=0 tx=0).stop 2023-03-03T08:51:36.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 -- 172.21.15.7:0/1660463123 shutdown_connections 2023-03-03T08:51:36.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8840682f0 0x7ff88406a7a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:36.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8a0157a40 0x7ff8a014acb0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:36.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8a006a2b0 0x7ff8a00c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:36.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 --2- 172.21.15.7:0/1660463123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8a00ab0c0 0x7ff8a00c38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:51:36.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 -- 172.21.15.7:0/1660463123 >> 172.21.15.7:0/1660463123 conn(0x7ff8a00a9430 msgr2=0x7ff8a006b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:51:36.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 -- 172.21.15.7:0/1660463123 shutdown_connections 2023-03-03T08:51:36.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:51:36.183+0000 7ff8a70d3700 1 -- 172.21.15.7:0/1660463123 wait complete. 2023-03-03T08:51:36.186 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:51:36.196 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2023-03-03T08:51:36.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:51:36.813 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:06.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:06.873 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:10.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:52:10.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.335+0000 7f318dcb4700 1 Processor -- start 2023-03-03T08:52:10.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.335+0000 7f318dcb4700 1 -- start start 2023-03-03T08:52:10.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.336+0000 7f318dcb4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f318814fda0 0x7f3188156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.336+0000 7f318dcb4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31880abe20 0x7f31881529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.336+0000 7f318dcb4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31880c2f30 con 0x7f318814f9d0 2023-03-03T08:52:10.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.336+0000 7f318dcb4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31880c31d0 con 0x7f318814fda0 2023-03-03T08:52:10.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.336+0000 7f318dcb4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31881524a0 con 0x7f31880abe20 2023-03-03T08:52:10.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3187fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31880abe20 0x7f31881529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3186ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f318814fda0 0x7f3188156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3187fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31880abe20 0x7f31881529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:33366/0 (socket says 172.21.15.7:33366) 2023-03-03T08:52:10.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f31877fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f318814f9d0 0x7f3188071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:51022/0 (socket says 172.21.15.7:51022) 2023-03-03T08:52:10.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3186ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f318814fda0 0x7f3188156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:49298/0 (socket says 172.21.15.7:49298) 2023-03-03T08:52:10.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3187fff700 1 -- 172.21.15.7:0/2981973431 learned_addr learned my addr 172.21.15.7:0/2981973431 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:52:10.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3187fff700 1 -- 172.21.15.7:0/2981973431 >> v2:172.21.15.143:3300/0 conn(0x7f318814fda0 msgr2=0x7f3188156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:52:10.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3187fff700 1 --2- 172.21.15.7:0/2981973431 >> v2:172.21.15.143:3300/0 conn(0x7f318814fda0 0x7f3188156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3187fff700 1 -- 172.21.15.7:0/2981973431 >> v1:172.21.15.143:6789/0 conn(0x7f318814f9d0 legacy=0x7f3188071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:10.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3187fff700 1 -- 172.21.15.7:0/2981973431 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880c34b0 con 0x7f31880abe20 2023-03-03T08:52:10.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3186ffd700 1 --2- 172.21.15.7:0/2981973431 >> v2:172.21.15.143:3300/0 conn(0x7f318814fda0 0x7f3188156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:52:10.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f3187fff700 1 --2- 172.21.15.7:0/2981973431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31880abe20 0x7f31881529c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3180009e50 tx=0x7f318000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7157a03a9a53fcea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:10.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f31867fc700 1 -- 172.21.15.7:0/2981973431 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318000e070 con 0x7f31880abe20 2023-03-03T08:52:10.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f31867fc700 1 -- 172.21.15.7:0/2981973431 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318000b910 con 0x7f31880abe20 2023-03-03T08:52:10.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.337+0000 7f31867fc700 1 -- 172.21.15.7:0/2981973431 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318000bcc0 con 0x7f31880abe20 2023-03-03T08:52:10.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.338+0000 7f318dcb4700 1 -- 172.21.15.7:0/2981973431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31880abe20 msgr2=0x7f31881529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:10.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.338+0000 7f318dcb4700 1 --2- 172.21.15.7:0/2981973431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31880abe20 0x7f31881529c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3180009e50 tx=0x7f318000b330 comp rx=0 tx=0).stop 2023-03-03T08:52:10.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.338+0000 7f318dcb4700 1 -- 172.21.15.7:0/2981973431 shutdown_connections 2023-03-03T08:52:10.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.338+0000 7f318dcb4700 1 --2- 172.21.15.7:0/2981973431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31880abe20 0x7f31881529c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.338+0000 7f318dcb4700 1 --2- 172.21.15.7:0/2981973431 >> v2:172.21.15.143:3300/0 conn(0x7f318814fda0 0x7f3188156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.338+0000 7f318dcb4700 1 -- 172.21.15.7:0/2981973431 >> 172.21.15.7:0/2981973431 conn(0x7f31880c8900 msgr2=0x7f31880c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:52:10.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.338+0000 7f318dcb4700 1 -- 172.21.15.7:0/2981973431 shutdown_connections 2023-03-03T08:52:10.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.338+0000 7f318dcb4700 1 -- 172.21.15.7:0/2981973431 wait complete. 2023-03-03T08:52:10.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.339+0000 7f318dcb4700 1 Processor -- start 2023-03-03T08:52:10.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.339+0000 7f318dcb4700 1 -- start start 2023-03-03T08:52:10.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f318dcb4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31880abe20 0x7f31881e9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f318dcb4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f318814f9d0 0x7f31881dc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f318dcb4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f318814fda0 0x7f3188152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f318dcb4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31881dce80 con 0x7f31880abe20 2023-03-03T08:52:10.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f318dcb4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31881db8a0 con 0x7f318814f9d0 2023-03-03T08:52:10.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f318dcb4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31881dbb80 con 0x7f318814fda0 2023-03-03T08:52:10.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f31877fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31880abe20 0x7f31881e9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f31877fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31880abe20 0x7f31881e9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53528/0 (socket says 172.21.15.7:53528) 2023-03-03T08:52:10.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f31877fe700 1 -- 172.21.15.7:0/4034798052 learned_addr learned my addr 172.21.15.7:0/4034798052 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:52:10.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f3186ffd700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f318814f9d0 0x7f31881dc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f3187fff700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f318814fda0 0x7f3188152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f31877fe700 1 -- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f318814fda0 msgr2=0x7f3188152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:10.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f31877fe700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f318814fda0 0x7f3188152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f31877fe700 1 -- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f318814f9d0 msgr2=0x7f31881dc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:10.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.340+0000 7f31877fe700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f318814f9d0 0x7f31881dc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.341+0000 7f31877fe700 1 -- 172.21.15.7:0/4034798052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31700094a0 con 0x7f31880abe20 2023-03-03T08:52:10.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.341+0000 7f31877fe700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31880abe20 0x7f31881e9740 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f317800c520 tx=0x7f317800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:10.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.341+0000 7f317ffff700 1 -- 172.21.15.7:0/4034798052 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178005cb0 con 0x7f31880abe20 2023-03-03T08:52:10.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.341+0000 7f318dcb4700 1 -- 172.21.15.7:0/4034798052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3180009b30 con 0x7f31880abe20 2023-03-03T08:52:10.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.341+0000 7f317ffff700 1 -- 172.21.15.7:0/4034798052 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3178005e10 con 0x7f31880abe20 2023-03-03T08:52:10.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.341+0000 7f317ffff700 1 -- 172.21.15.7:0/4034798052 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31780072d0 con 0x7f31880abe20 2023-03-03T08:52:10.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.341+0000 7f318dcb4700 1 -- 172.21.15.7:0/4034798052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31880ac350 con 0x7f31880abe20 2023-03-03T08:52:10.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.342+0000 7f318dcb4700 1 -- 172.21.15.7:0/4034798052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31880ab630 con 0x7f31880abe20 2023-03-03T08:52:10.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.342+0000 7f317ffff700 1 -- 172.21.15.7:0/4034798052 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3178019030 con 0x7f31880abe20 2023-03-03T08:52:10.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.343+0000 7f317ffff700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31680682f0 0x7f316806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:10.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.343+0000 7f317ffff700 1 -- 172.21.15.7:0/4034798052 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f31780852e0 con 0x7f31880abe20 2023-03-03T08:52:10.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.343+0000 7f3186ffd700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31680682f0 0x7f316806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:10.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.343+0000 7f3186ffd700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31680682f0 0x7f316806a7a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3188155d90 tx=0x7f3170009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:10.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.347+0000 7f317ffff700 1 -- 172.21.15.7:0/4034798052 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f317804b5e0 con 0x7f31880abe20 2023-03-03T08:52:10.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.585+0000 7f318dcb4700 1 -- 172.21.15.7:0/4034798052 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3188071fa0 con 0x7f31880abe20 2023-03-03T08:52:10.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.586+0000 7f317ffff700 1 -- 172.21.15.7:0/4034798052 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3178017090 con 0x7f31880abe20 2023-03-03T08:52:10.587 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:52:10.588 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:52:10.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 -- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31680682f0 msgr2=0x7f316806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:10.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31680682f0 0x7f316806a7a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3188155d90 tx=0x7f3170009210 comp rx=0 tx=0).stop 2023-03-03T08:52:10.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 -- 172.21.15.7:0/4034798052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31880abe20 msgr2=0x7f31881e9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:10.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31880abe20 0x7f31881e9740 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f317800c520 tx=0x7f317800c830 comp rx=0 tx=0).stop 2023-03-03T08:52:10.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 -- 172.21.15.7:0/4034798052 shutdown_connections 2023-03-03T08:52:10.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f318814fda0 0x7f3188152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31680682f0 0x7f316806a7a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31880abe20 0x7f31881e9740 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 --2- 172.21.15.7:0/4034798052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f318814f9d0 0x7f31881dc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:10.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 -- 172.21.15.7:0/4034798052 >> 172.21.15.7:0/4034798052 conn(0x7f31880c8900 msgr2=0x7f3188151b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:52:10.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 -- 172.21.15.7:0/4034798052 shutdown_connections 2023-03-03T08:52:10.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:10.588+0000 7f318dcb4700 1 -- 172.21.15.7:0/4034798052 wait complete. 2023-03-03T08:52:10.591 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:52:10.604 INFO:tasks.mds_thrash.fs.[b]:waiting for 46 secs before thrashing 2023-03-03T08:52:36.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:52:36.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:36.940 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:52:37.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.007+0000 7f43a0c7f700 1 Processor -- start 2023-03-03T08:52:37.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.007+0000 7f43a0c7f700 1 -- start start 2023-03-03T08:52:37.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.008+0000 7f43a0c7f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c1547a0 0x7f439c0c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:37.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.008+0000 7f43a0c7f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f439c0a60f0 0x7f439c154b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:37.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.008+0000 7f43a0c7f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439c0c2c20 con 0x7f439c0aa670 2023-03-03T08:52:37.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.008+0000 7f43a0c7f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f439c0a65f0 con 0x7f439c0a60f0 2023-03-03T08:52:37.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.008+0000 7f43a0c7f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0c1e10 con 0x7f439c1547a0 2023-03-03T08:52:37.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.008+0000 7f4399d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f439c0a60f0 0x7f439c154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:37.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.008+0000 7f439ad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f439c0aa670 0x7f439c0c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47524/0 (socket says 172.21.15.7:47524) 2023-03-03T08:52:37.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.008+0000 7f4399d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f439c0a60f0 0x7f439c154b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53540/0 (socket says 172.21.15.7:53540) 2023-03-03T08:52:37.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.008+0000 7f439ad9d700 1 -- 172.21.15.7:0/1396269664 learned_addr learned my addr 172.21.15.7:0/1396269664 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:52:37.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.009+0000 7f439a59c700 1 --2- 172.21.15.7:0/1396269664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c1547a0 0x7f439c0c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:37.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.009+0000 7f4399d9b700 1 -- 172.21.15.7:0/1396269664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c1547a0 msgr2=0x7f439c0c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:52:37.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.009+0000 7f4399d9b700 1 --2- 172.21.15.7:0/1396269664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c1547a0 0x7f439c0c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:37.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.009+0000 7f4399d9b700 1 -- 172.21.15.7:0/1396269664 >> v1:172.21.15.7:6789/0 conn(0x7f439c0aa670 legacy=0x7f439c0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:37.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.009+0000 7f439959a700 1 -- 172.21.15.7:0/1396269664 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530202628 0 0) 0x7f439c0c2c20 con 0x7f439c0aa670 2023-03-03T08:52:37.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.009+0000 7f4399d9b700 1 -- 172.21.15.7:0/1396269664 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c1550b0 con 0x7f439c0a60f0 2023-03-03T08:52:37.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.009+0000 7f4399d9b700 1 --2- 172.21.15.7:0/1396269664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 0x7f439c154b70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f438c009840 tx=0x7f438c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efca625860d77f2f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:37.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.009+0000 7f439959a700 1 -- 172.21.15.7:0/1396269664 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c011830 con 0x7f439c0a60f0 2023-03-03T08:52:37.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.009+0000 7f439959a700 1 -- 172.21.15.7:0/1396269664 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438c011990 con 0x7f439c0a60f0 2023-03-03T08:52:37.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.009+0000 7f439959a700 1 -- 172.21.15.7:0/1396269664 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c011d20 con 0x7f439c0a60f0 2023-03-03T08:52:37.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.010+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1396269664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 msgr2=0x7f439c154b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:37.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.010+0000 7f43a0c7f700 1 --2- 172.21.15.7:0/1396269664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 0x7f439c154b70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f438c009840 tx=0x7f438c009b50 comp rx=0 tx=0).stop 2023-03-03T08:52:37.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.010+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1396269664 shutdown_connections 2023-03-03T08:52:37.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.010+0000 7f43a0c7f700 1 --2- 172.21.15.7:0/1396269664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c1547a0 0x7f439c0c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:37.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.010+0000 7f43a0c7f700 1 --2- 172.21.15.7:0/1396269664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 0x7f439c154b70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:37.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.010+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1396269664 >> 172.21.15.7:0/1396269664 conn(0x7f439c0a9430 msgr2=0x7f439c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:52:37.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.010+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1396269664 shutdown_connections 2023-03-03T08:52:37.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.010+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1396269664 wait complete. 2023-03-03T08:52:37.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.011+0000 7f43a0c7f700 1 Processor -- start 2023-03-03T08:52:37.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.011+0000 7f43a0c7f700 1 -- start start 2023-03-03T08:52:37.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f43a0c7f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 0x7f439c153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:37.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f43a0c7f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c0aa670 0x7f439c14dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:37.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f43a0c7f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f439c1547a0 0x7f439c149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:37.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f43a0c7f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0726e0 con 0x7f439c0a60f0 2023-03-03T08:52:37.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f43a0c7f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c154320 con 0x7f439c1547a0 2023-03-03T08:52:37.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f43a0c7f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439c1e8bb0 con 0x7f439c0aa670 2023-03-03T08:52:37.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f439a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 0x7f439c153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:37.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f439a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 0x7f439c153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53546/0 (socket says 172.21.15.7:53546) 2023-03-03T08:52:37.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f439a59c700 1 -- 172.21.15.7:0/1245946790 learned_addr learned my addr 172.21.15.7:0/1245946790 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:52:37.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f4399d9b700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c0aa670 0x7f439c14dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:37.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f439a59c700 1 -- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c0aa670 msgr2=0x7f439c14dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:37.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f439a59c700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c0aa670 0x7f439c14dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:37.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f439ad9d700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f439c1547a0 0x7f439c149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:37.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f439a59c700 1 -- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f439c1547a0 msgr2=0x7f439c149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:37.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f439a59c700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f439c1547a0 0x7f439c149670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:37.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f439a59c700 1 -- 172.21.15.7:0/1245946790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c0094a0 con 0x7f439c0a60f0 2023-03-03T08:52:37.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f4399d9b700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c0aa670 0x7f439c14dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:52:37.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.012+0000 7f439ad9d700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f439c1547a0 0x7f439c149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:52:37.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.013+0000 7f439a59c700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 0x7f439c153de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f439000d800 tx=0x7f439000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:37.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.013+0000 7f438affd700 1 -- 172.21.15.7:0/1245946790 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43900125f0 con 0x7f439c0a60f0 2023-03-03T08:52:37.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.013+0000 7f438affd700 1 -- 172.21.15.7:0/1245946790 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4390012750 con 0x7f439c0a60f0 2023-03-03T08:52:37.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.013+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1245946790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439c0c2ec0 con 0x7f439c0a60f0 2023-03-03T08:52:37.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.013+0000 7f438affd700 1 -- 172.21.15.7:0/1245946790 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4390012b00 con 0x7f439c0a60f0 2023-03-03T08:52:37.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.013+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1245946790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439c155350 con 0x7f439c0a60f0 2023-03-03T08:52:37.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.014+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1245946790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439c155630 con 0x7f439c0a60f0 2023-03-03T08:52:37.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.014+0000 7f438affd700 1 -- 172.21.15.7:0/1245946790 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4390012c60 con 0x7f439c0a60f0 2023-03-03T08:52:37.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.015+0000 7f438affd700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f437c0682f0 0x7f437c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:37.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.015+0000 7f438affd700 1 -- 172.21.15.7:0/1245946790 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f43900866d0 con 0x7f439c0a60f0 2023-03-03T08:52:37.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.015+0000 7f4399d9b700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f437c0682f0 0x7f437c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:37.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.015+0000 7f4399d9b700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f437c0682f0 0x7f437c06a7a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f439c0a6520 tx=0x7f438c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:37.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.019+0000 7f438affd700 1 -- 172.21.15.7:0/1245946790 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4390052a70 con 0x7f439c0a60f0 2023-03-03T08:52:37.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.262+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1245946790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f439c074bb0 con 0x7f439c0a60f0 2023-03-03T08:52:37.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.263+0000 7f438affd700 1 -- 172.21.15.7:0/1245946790 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4390056090 con 0x7f439c0a60f0 2023-03-03T08:52:37.264 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:52:37.264 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:52:37.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.264+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f437c0682f0 msgr2=0x7f437c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:37.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.264+0000 7f43a0c7f700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f437c0682f0 0x7f437c06a7a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f439c0a6520 tx=0x7f438c008030 comp rx=0 tx=0).stop 2023-03-03T08:52:37.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.264+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1245946790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 msgr2=0x7f439c153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:37.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.264+0000 7f43a0c7f700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 0x7f439c153de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f439000d800 tx=0x7f439000dbc0 comp rx=0 tx=0).stop 2023-03-03T08:52:37.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.264+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1245946790 shutdown_connections 2023-03-03T08:52:37.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.264+0000 7f43a0c7f700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f439c1547a0 0x7f439c149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:37.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.265+0000 7f43a0c7f700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f437c0682f0 0x7f437c06a7a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:37.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.265+0000 7f43a0c7f700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f439c0a60f0 0x7f439c153de0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:37.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.265+0000 7f43a0c7f700 1 --2- 172.21.15.7:0/1245946790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f439c0aa670 0x7f439c14dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:37.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.265+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1245946790 >> 172.21.15.7:0/1245946790 conn(0x7f439c0a9430 msgr2=0x7f439c0c1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:52:37.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.265+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1245946790 shutdown_connections 2023-03-03T08:52:37.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:37.265+0000 7f43a0c7f700 1 -- 172.21.15.7:0/1245946790 wait complete. 2023-03-03T08:52:37.268 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:52:37.278 INFO:tasks.mds_thrash.fs.[a]:waiting for 116 secs before thrashing 2023-03-03T08:52:56.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:52:56.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.741+0000 7f0b27b04700 1 Processor -- start 2023-03-03T08:52:56.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.742+0000 7f0b27b04700 1 -- start start 2023-03-03T08:52:56.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.742+0000 7f0b27b04700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0b2006bea0 0x7f0b2006c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:56.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.742+0000 7f0b27b04700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2006d780 con 0x7f0b20157dc0 2023-03-03T08:52:56.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.742+0000 7f0b27b04700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2006cb70 con 0x7f0b20151ab0 2023-03-03T08:52:56.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.742+0000 7f0b27b04700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2006c830 con 0x7f0b2006bea0 2023-03-03T08:52:56.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b260a1700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0b20157dc0 0x7f0b20158240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47538/0 (socket says 172.21.15.7:47538) 2023-03-03T08:52:56.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b260a1700 1 -- 172.21.15.7:0/2820949420 learned_addr learned my addr 172.21.15.7:0/2820949420 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:52:56.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2509f700 1 --2- 172.21.15.7:0/2820949420 >> v2:172.21.15.143:3300/0 conn(0x7f0b2006bea0 0x7f0b2006c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:56.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891781475 0 0) 0x7f0b2006d780 con 0x7f0b20157dc0 2023-03-03T08:52:56.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b0c0035b0 con 0x7f0b20157dc0 2023-03-03T08:52:56.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1316543511 0 0) 0x7f0b0c0035b0 con 0x7f0b20157dc0 2023-03-03T08:52:56.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 >> v2:172.21.15.143:3300/0 conn(0x7f0b2006bea0 msgr2=0x7f0b2006c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:52:56.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 --2- 172.21.15.7:0/2820949420 >> v2:172.21.15.143:3300/0 conn(0x7f0b2006bea0 0x7f0b2006c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:56.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 >> v1:172.21.15.143:6789/0 conn(0x7f0b20151ab0 legacy=0x7f0b200736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:56.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2006ce50 con 0x7f0b20157dc0 2023-03-03T08:52:56.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0b180019f0 con 0x7f0b20157dc0 2023-03-03T08:52:56.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 >> v1:172.21.15.7:6789/0 conn(0x7f0b20157dc0 legacy=0x7f0b20158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:56.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c003a70 0x7f0b0c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:56.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c004850 0x7f0b0c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:56.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b0c005250 0x7f0b0c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:56.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c00c160 con 0x7f0b0c005250 2023-03-03T08:52:56.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c00c2d0 con 0x7f0b0c004850 2023-03-03T08:52:56.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c00c490 con 0x7f0b0c003a70 2023-03-03T08:52:56.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b260a1700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b0c005250 0x7f0b0c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:56.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b2509f700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c004850 0x7f0b0c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:56.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.743+0000 7f0b258a0700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c003a70 0x7f0b0c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:56.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b260a1700 1 -- 172.21.15.7:0/2820949420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c003a70 msgr2=0x7f0b0c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:52:56.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b260a1700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b0c003a70 0x7f0b0c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:56.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b260a1700 1 -- 172.21.15.7:0/2820949420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c004850 msgr2=0x7f0b0c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:52:56.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b260a1700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c004850 0x7f0b0c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:56.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b260a1700 1 -- 172.21.15.7:0/2820949420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0035b0 con 0x7f0b0c005250 2023-03-03T08:52:56.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b260a1700 1 -- 172.21.15.7:0/2820949420 reap_dead start 2023-03-03T08:52:56.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b260a1700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b0c005250 0x7f0b0c00ba10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800a800 tx=0x7f0b0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f97c7e18d2d0aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:56.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1800a2f0 con 0x7f0b0c005250 2023-03-03T08:52:56.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b18012600 con 0x7f0b0c005250 2023-03-03T08:52:56.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b2489e700 1 -- 172.21.15.7:0/2820949420 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b180129b0 con 0x7f0b0c005250 2023-03-03T08:52:56.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b27b04700 1 -- 172.21.15.7:0/2820949420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b0c005250 msgr2=0x7f0b0c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:56.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b27b04700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b0c005250 0x7f0b0c00ba10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0b1800a800 tx=0x7f0b0c005880 comp rx=0 tx=0).stop 2023-03-03T08:52:56.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b27b04700 1 -- 172.21.15.7:0/2820949420 shutdown_connections 2023-03-03T08:52:56.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b27b04700 1 --2- 172.21.15.7:0/2820949420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b0c005250 0x7f0b0c00ba10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:56.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.744+0000 7f0b27b04700 1 -- 172.21.15.7:0/2820949420 >> 172.21.15.7:0/2820949420 conn(0x7f0b200a94d0 msgr2=0x7f0b200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:52:56.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.745+0000 7f0b27b04700 1 -- 172.21.15.7:0/2820949420 shutdown_connections 2023-03-03T08:52:56.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.745+0000 7f0b27b04700 1 -- 172.21.15.7:0/2820949420 wait complete. 2023-03-03T08:52:56.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.746+0000 7f0b27b04700 1 Processor -- start 2023-03-03T08:52:56.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.746+0000 7f0b27b04700 1 -- start start 2023-03-03T08:52:56.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.746+0000 7f0b27b04700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c005250 0x7f0b200accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:56.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.746+0000 7f0b27b04700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b20154f90 0x7f0b200aaec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:56.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.746+0000 7f0b27b04700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b200ab400 0x7f0b201ea580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:56.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.746+0000 7f0b27b04700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b200c7010 con 0x7f0b20154f90 2023-03-03T08:52:56.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.746+0000 7f0b27b04700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b201e9e30 con 0x7f0b0c005250 2023-03-03T08:52:56.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.746+0000 7f0b27b04700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b200c61b0 con 0x7f0b200ab400 2023-03-03T08:52:56.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b2509f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b20154f90 0x7f0b200aaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:56.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b2509f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b20154f90 0x7f0b200aaec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53568/0 (socket says 172.21.15.7:53568) 2023-03-03T08:52:56.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b2509f700 1 -- 172.21.15.7:0/2422739820 learned_addr learned my addr 172.21.15.7:0/2422739820 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:52:56.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b258a0700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c005250 0x7f0b200accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:56.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b260a1700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b200ab400 0x7f0b201ea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:56.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b2509f700 1 -- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b200ab400 msgr2=0x7f0b201ea580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:56.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b2509f700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b200ab400 0x7f0b201ea580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:56.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b2509f700 1 -- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c005250 msgr2=0x7f0b200accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:56.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b2509f700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c005250 0x7f0b200accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:56.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b2509f700 1 -- 172.21.15.7:0/2422739820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2006d410 con 0x7f0b20154f90 2023-03-03T08:52:56.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b260a1700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b200ab400 0x7f0b201ea580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:52:56.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b2509f700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b20154f90 0x7f0b200aaec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0b100119a0 tx=0x7f0b10011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:56.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b167fc700 1 -- 172.21.15.7:0/2422739820 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000f940 con 0x7f0b20154f90 2023-03-03T08:52:56.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b167fc700 1 -- 172.21.15.7:0/2422739820 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1000faa0 con 0x7f0b20154f90 2023-03-03T08:52:56.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b27b04700 1 -- 172.21.15.7:0/2422739820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2006d130 con 0x7f0b20154f90 2023-03-03T08:52:56.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b167fc700 1 -- 172.21.15.7:0/2422739820 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000fe30 con 0x7f0b20154f90 2023-03-03T08:52:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.747+0000 7f0b27b04700 1 -- 172.21.15.7:0/2422739820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b20151e80 con 0x7f0b20154f90 2023-03-03T08:52:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.748+0000 7f0b27b04700 1 -- 172.21.15.7:0/2422739820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b201ed5b0 con 0x7f0b20154f90 2023-03-03T08:52:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.749+0000 7f0b167fc700 1 -- 172.21.15.7:0/2422739820 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0b10014440 con 0x7f0b20154f90 2023-03-03T08:52:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.749+0000 7f0b167fc700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b040682f0 0x7f0b0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:52:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.749+0000 7f0b167fc700 1 -- 172.21.15.7:0/2422739820 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0b10086a20 con 0x7f0b20154f90 2023-03-03T08:52:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.749+0000 7f0b258a0700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b040682f0 0x7f0b0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:52:56.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.750+0000 7f0b258a0700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b040682f0 0x7f0b0406a7a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c007fd0 tx=0x7f0b1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:52:56.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.754+0000 7f0b167fc700 1 -- 172.21.15.7:0/2422739820 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0b10052cd0 con 0x7f0b20154f90 2023-03-03T08:52:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.993+0000 7f0b27b04700 1 -- 172.21.15.7:0/2422739820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b201ed3a0 con 0x7f0b20154f90 2023-03-03T08:52:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.994+0000 7f0b167fc700 1 -- 172.21.15.7:0/2422739820 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0b100562f0 con 0x7f0b20154f90 2023-03-03T08:52:56.997 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:52:56.997 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:52:56.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 -- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b040682f0 msgr2=0x7f0b0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b040682f0 0x7f0b0406a7a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c007fd0 tx=0x7f0b1c009040 comp rx=0 tx=0).stop 2023-03-03T08:52:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 -- 172.21.15.7:0/2422739820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b20154f90 msgr2=0x7f0b200aaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:52:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b20154f90 0x7f0b200aaec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0b100119a0 tx=0x7f0b10011d60 comp rx=0 tx=0).stop 2023-03-03T08:52:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 -- 172.21.15.7:0/2422739820 shutdown_connections 2023-03-03T08:52:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b200ab400 0x7f0b201ea580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b0c005250 0x7f0b200accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b040682f0 0x7f0b0406a7a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:57.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 --2- 172.21.15.7:0/2422739820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b20154f90 0x7f0b200aaec0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:52:57.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 -- 172.21.15.7:0/2422739820 >> 172.21.15.7:0/2422739820 conn(0x7f0b200a94d0 msgr2=0x7f0b2014f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:52:57.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 -- 172.21.15.7:0/2422739820 shutdown_connections 2023-03-03T08:52:57.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:52:56.996+0000 7f0b27b04700 1 -- 172.21.15.7:0/2422739820 wait complete. 2023-03-03T08:52:57.000 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:52:57.010 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before thrashing 2023-03-03T08:53:01.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:53:01.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.149+0000 7f702d631700 1 Processor -- start 2023-03-03T08:53:01.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.149+0000 7f702d631700 1 -- start start 2023-03-03T08:53:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.149+0000 7f702d631700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f70280caa30 0x7f70280c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.149+0000 7f702d631700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70280c00e0 0x7f7028154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.149+0000 7f702d631700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70280cae00 0x7f70280c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.149+0000 7f702d631700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70280cb280 con 0x7f70280cae00 2023-03-03T08:53:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.149+0000 7f702d631700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70280bfd10 con 0x7f70280caa30 2023-03-03T08:53:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.149+0000 7f702d631700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70280bfa30 con 0x7f70280c00e0 2023-03-03T08:53:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f70277fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70280cae00 0x7f70280c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f70277fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70280cae00 0x7f70280c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53578/0 (socket says 172.21.15.7:53578) 2023-03-03T08:53:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f70277fe700 1 -- 172.21.15.7:0/1568696358 learned_addr learned my addr 172.21.15.7:0/1568696358 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:53:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f7026ffd700 1 --2- 172.21.15.7:0/1568696358 >> v2:172.21.15.143:3300/0 conn(0x7f70280caa30 0x7f70280c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f70267fc700 1 --2- 172.21.15.7:0/1568696358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70280c00e0 0x7f7028154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:01.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f70277fe700 1 -- 172.21.15.7:0/1568696358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70280c00e0 msgr2=0x7f7028154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:53:01.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f70277fe700 1 --2- 172.21.15.7:0/1568696358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70280c00e0 0x7f7028154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f70277fe700 1 -- 172.21.15.7:0/1568696358 >> v2:172.21.15.143:3300/0 conn(0x7f70280caa30 msgr2=0x7f70280c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:53:01.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f70277fe700 1 --2- 172.21.15.7:0/1568696358 >> v2:172.21.15.143:3300/0 conn(0x7f70280caa30 0x7f70280c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f70277fe700 1 -- 172.21.15.7:0/1568696358 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7028154510 con 0x7f70280cae00 2023-03-03T08:53:01.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f7026ffd700 1 --2- 172.21.15.7:0/1568696358 >> v2:172.21.15.143:3300/0 conn(0x7f70280caa30 0x7f70280c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:53:01.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f70277fe700 1 --2- 172.21.15.7:0/1568696358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280cae00 0x7f70280c6c00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f70280c81b0 tx=0x7f702000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b10a1897578bdb8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:53:01.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f7025ffb700 1 -- 172.21.15.7:0/1568696358 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702000e070 con 0x7f70280cae00 2023-03-03T08:53:01.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f7025ffb700 1 -- 172.21.15.7:0/1568696358 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702000b9c0 con 0x7f70280cae00 2023-03-03T08:53:01.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.150+0000 7f7025ffb700 1 -- 172.21.15.7:0/1568696358 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702000bd70 con 0x7f70280cae00 2023-03-03T08:53:01.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.151+0000 7f702d631700 1 -- 172.21.15.7:0/1568696358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280cae00 msgr2=0x7f70280c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:01.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.151+0000 7f702d631700 1 --2- 172.21.15.7:0/1568696358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280cae00 0x7f70280c6c00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f70280c81b0 tx=0x7f702000b330 comp rx=0 tx=0).stop 2023-03-03T08:53:01.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.151+0000 7f702d631700 1 -- 172.21.15.7:0/1568696358 shutdown_connections 2023-03-03T08:53:01.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.151+0000 7f702d631700 1 --2- 172.21.15.7:0/1568696358 >> v2:172.21.15.143:3300/0 conn(0x7f70280caa30 0x7f70280c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.151+0000 7f702d631700 1 --2- 172.21.15.7:0/1568696358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280cae00 0x7f70280c6c00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.151+0000 7f702d631700 1 --2- 172.21.15.7:0/1568696358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70280c00e0 0x7f7028154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.151+0000 7f702d631700 1 -- 172.21.15.7:0/1568696358 >> 172.21.15.7:0/1568696358 conn(0x7f7028066e40 msgr2=0x7f7028071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:53:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.151+0000 7f702d631700 1 -- 172.21.15.7:0/1568696358 shutdown_connections 2023-03-03T08:53:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.151+0000 7f702d631700 1 -- 172.21.15.7:0/1568696358 wait complete. 2023-03-03T08:53:01.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.152+0000 7f702d631700 1 Processor -- start 2023-03-03T08:53:01.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.152+0000 7f702d631700 1 -- start start 2023-03-03T08:53:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f702d631700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280c00e0 0x7f70281ea040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f702d631700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70280caa30 0x7f7028153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f702d631700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70280cae00 0x7f70281ed150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f702d631700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70280aa6e0 con 0x7f70280c00e0 2023-03-03T08:53:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f702d631700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70280c2810 con 0x7f70280cae00 2023-03-03T08:53:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f702d631700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70281ea580 con 0x7f70280caa30 2023-03-03T08:53:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f7026ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280c00e0 0x7f70281ea040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f7026ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280c00e0 0x7f70281ea040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53580/0 (socket says 172.21.15.7:53580) 2023-03-03T08:53:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f7026ffd700 1 -- 172.21.15.7:0/1704216784 learned_addr learned my addr 172.21.15.7:0/1704216784 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:53:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f70267fc700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70280caa30 0x7f7028153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f70277fe700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70280cae00 0x7f70281ed150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f7026ffd700 1 -- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70280caa30 msgr2=0x7f7028153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f7026ffd700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70280caa30 0x7f7028153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f7026ffd700 1 -- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70280cae00 msgr2=0x7f70281ed150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f7026ffd700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70280cae00 0x7f70281ed150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.153+0000 7f7026ffd700 1 -- 172.21.15.7:0/1704216784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020009b30 con 0x7f70280c00e0 2023-03-03T08:53:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.154+0000 7f7026ffd700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280c00e0 0x7f70281ea040 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f701800e810 tx=0x7f701800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:53:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.154+0000 7f700f7fe700 1 -- 172.21.15.7:0/1704216784 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018013610 con 0x7f70280c00e0 2023-03-03T08:53:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.154+0000 7f700f7fe700 1 -- 172.21.15.7:0/1704216784 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7018013770 con 0x7f70280c00e0 2023-03-03T08:53:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.154+0000 7f702d631700 1 -- 172.21.15.7:0/1704216784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7028154ad0 con 0x7f70280c00e0 2023-03-03T08:53:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.154+0000 7f700f7fe700 1 -- 172.21.15.7:0/1704216784 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018013b20 con 0x7f70280c00e0 2023-03-03T08:53:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.154+0000 7f702d631700 1 -- 172.21.15.7:0/1704216784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70281547f0 con 0x7f70280c00e0 2023-03-03T08:53:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.155+0000 7f702d631700 1 -- 172.21.15.7:0/1704216784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70280bf650 con 0x7f70280c00e0 2023-03-03T08:53:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.155+0000 7f700f7fe700 1 -- 172.21.15.7:0/1704216784 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7018023070 con 0x7f70280c00e0 2023-03-03T08:53:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.156+0000 7f700f7fe700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70140682f0 0x7f701406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.156+0000 7f700f7fe700 1 -- 172.21.15.7:0/1704216784 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7018085c90 con 0x7f70280c00e0 2023-03-03T08:53:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.156+0000 7f70267fc700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70140682f0 0x7f701406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.156+0000 7f70267fc700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70140682f0 0x7f701406a7a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f70280c9a90 tx=0x7f701c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:53:01.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.160+0000 7f700f7fe700 1 -- 172.21.15.7:0/1704216784 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7018052030 con 0x7f70280c00e0 2023-03-03T08:53:01.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.396+0000 7f702d631700 1 -- 172.21.15.7:0/1704216784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70280c6cd0 con 0x7f70280c00e0 2023-03-03T08:53:01.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.396+0000 7f700f7fe700 1 -- 172.21.15.7:0/1704216784 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7018055650 con 0x7f70280c00e0 2023-03-03T08:53:01.399 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:53:01.399 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:53:01.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 -- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70140682f0 msgr2=0x7f701406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:01.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70140682f0 0x7f701406a7a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f70280c9a90 tx=0x7f701c008040 comp rx=0 tx=0).stop 2023-03-03T08:53:01.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 -- 172.21.15.7:0/1704216784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280c00e0 msgr2=0x7f70281ea040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:01.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280c00e0 0x7f70281ea040 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f701800e810 tx=0x7f701800eb20 comp rx=0 tx=0).stop 2023-03-03T08:53:01.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 -- 172.21.15.7:0/1704216784 shutdown_connections 2023-03-03T08:53:01.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70280cae00 0x7f70281ed150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70140682f0 0x7f701406a7a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70280c00e0 0x7f70281ea040 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 --2- 172.21.15.7:0/1704216784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70280caa30 0x7f7028153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:01.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 -- 172.21.15.7:0/1704216784 >> 172.21.15.7:0/1704216784 conn(0x7f7028066e40 msgr2=0x7f702806b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:53:01.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 -- 172.21.15.7:0/1704216784 shutdown_connections 2023-03-03T08:53:01.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:01.398+0000 7f702d631700 1 -- 172.21.15.7:0/1704216784 wait complete. 2023-03-03T08:53:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:53:01.412 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before thrashing 2023-03-03T08:53:02.414 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:53:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.552+0000 7fd973161700 1 Processor -- start 2023-03-03T08:53:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.552+0000 7fd973161700 1 -- start start 2023-03-03T08:53:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd973161700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd96c151d40 0x7fd96c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd973161700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd96c0c2930 0x7fd96c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd973161700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd96c073c30 0x7fd96c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd973161700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96c06c2a0 con 0x7fd96c073c30 2023-03-03T08:53:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd973161700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96c06d430 con 0x7fd96c0c2930 2023-03-03T08:53:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd973161700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c06cb00 con 0x7fd96c151d40 2023-03-03T08:53:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd9716fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd96c073c30 0x7fd96c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd9716fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd96c073c30 0x7fd96c157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53592/0 (socket says 172.21.15.7:53592) 2023-03-03T08:53:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd9716fe700 1 -- 172.21.15.7:0/3895489846 learned_addr learned my addr 172.21.15.7:0/3895489846 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:53:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd970efd700 1 --2- 172.21.15.7:0/3895489846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd96c151d40 0x7fd96c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.553+0000 7fd96bfff700 1 --2- 172.21.15.7:0/3895489846 >> v2:172.21.15.143:3300/0 conn(0x7fd96c0c2930 0x7fd96c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd9716fe700 1 -- 172.21.15.7:0/3895489846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd96c151d40 msgr2=0x7fd96c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:53:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd9716fe700 1 --2- 172.21.15.7:0/3895489846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd96c151d40 0x7fd96c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd9716fe700 1 -- 172.21.15.7:0/3895489846 >> v2:172.21.15.143:3300/0 conn(0x7fd96c0c2930 msgr2=0x7fd96c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:53:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd9716fe700 1 --2- 172.21.15.7:0/3895489846 >> v2:172.21.15.143:3300/0 conn(0x7fd96c0c2930 0x7fd96c157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd9716fe700 1 -- 172.21.15.7:0/3895489846 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c06cde0 con 0x7fd96c073c30 2023-03-03T08:53:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd9716fe700 1 --2- 172.21.15.7:0/3895489846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c073c30 0x7fd96c157fd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd96c06b820 tx=0x7fd96400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ab57db097d2c8d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:53:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd96b7fe700 1 -- 172.21.15.7:0/3895489846 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96400e070 con 0x7fd96c073c30 2023-03-03T08:53:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd96b7fe700 1 -- 172.21.15.7:0/3895489846 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96400b910 con 0x7fd96c073c30 2023-03-03T08:53:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd96b7fe700 1 -- 172.21.15.7:0/3895489846 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96400bcc0 con 0x7fd96c073c30 2023-03-03T08:53:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd973161700 1 -- 172.21.15.7:0/3895489846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c073c30 msgr2=0x7fd96c157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd973161700 1 --2- 172.21.15.7:0/3895489846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c073c30 0x7fd96c157fd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd96c06b820 tx=0x7fd96400b330 comp rx=0 tx=0).stop 2023-03-03T08:53:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd973161700 1 -- 172.21.15.7:0/3895489846 shutdown_connections 2023-03-03T08:53:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd973161700 1 --2- 172.21.15.7:0/3895489846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c073c30 0x7fd96c157fd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd973161700 1 --2- 172.21.15.7:0/3895489846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd96c151d40 0x7fd96c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd973161700 1 --2- 172.21.15.7:0/3895489846 >> v2:172.21.15.143:3300/0 conn(0x7fd96c0c2930 0x7fd96c157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.554+0000 7fd973161700 1 -- 172.21.15.7:0/3895489846 >> 172.21.15.7:0/3895489846 conn(0x7fd96c0a94d0 msgr2=0x7fd96c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:53:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.555+0000 7fd973161700 1 -- 172.21.15.7:0/3895489846 shutdown_connections 2023-03-03T08:53:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.555+0000 7fd973161700 1 -- 172.21.15.7:0/3895489846 wait complete. 2023-03-03T08:53:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.556+0000 7fd973161700 1 Processor -- start 2023-03-03T08:53:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.556+0000 7fd973161700 1 -- start start 2023-03-03T08:53:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd973161700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd96c073c30 0x7fd96c0acb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd973161700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd96c0c2930 0x7fd96c153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd973161700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c151d40 0x7fd96c1ed020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd973161700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c1ed960 con 0x7fd96c151d40 2023-03-03T08:53:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd973161700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c1edc40 con 0x7fd96c0c2930 2023-03-03T08:53:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd973161700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c0aa270 con 0x7fd96c073c30 2023-03-03T08:53:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd9716fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c151d40 0x7fd96c1ed020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd9716fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c151d40 0x7fd96c1ed020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53598/0 (socket says 172.21.15.7:53598) 2023-03-03T08:53:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd9716fe700 1 -- 172.21.15.7:0/2420313943 learned_addr learned my addr 172.21.15.7:0/2420313943 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:53:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd970efd700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd96c073c30 0x7fd96c0acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd96bfff700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd96c0c2930 0x7fd96c153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd9716fe700 1 -- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd96c073c30 msgr2=0x7fd96c0acb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd9716fe700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd96c073c30 0x7fd96c0acb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd9716fe700 1 -- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd96c0c2930 msgr2=0x7fd96c153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd9716fe700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd96c0c2930 0x7fd96c153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.557+0000 7fd9716fe700 1 -- 172.21.15.7:0/2420313943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd964009b30 con 0x7fd96c151d40 2023-03-03T08:53:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.558+0000 7fd9716fe700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c151d40 0x7fd96c1ed020 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd96400b370 tx=0x7fd964004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:53:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.558+0000 7fd9697fa700 1 -- 172.21.15.7:0/2420313943 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96400ce80 con 0x7fd96c151d40 2023-03-03T08:53:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.558+0000 7fd9697fa700 1 -- 172.21.15.7:0/2420313943 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd964004040 con 0x7fd96c151d40 2023-03-03T08:53:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.558+0000 7fd9697fa700 1 -- 172.21.15.7:0/2420313943 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9640043f0 con 0x7fd96c151d40 2023-03-03T08:53:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.558+0000 7fd973161700 1 -- 172.21.15.7:0/2420313943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96c06bea0 con 0x7fd96c151d40 2023-03-03T08:53:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.558+0000 7fd973161700 1 -- 172.21.15.7:0/2420313943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c06d0c0 con 0x7fd96c151d40 2023-03-03T08:53:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.559+0000 7fd973161700 1 -- 172.21.15.7:0/2420313943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96c06c030 con 0x7fd96c151d40 2023-03-03T08:53:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.559+0000 7fd9697fa700 1 -- 172.21.15.7:0/2420313943 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd96402c070 con 0x7fd96c151d40 2023-03-03T08:53:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.560+0000 7fd9697fa700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9500682f0 0x7fd95006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:53:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.560+0000 7fd9697fa700 1 -- 172.21.15.7:0/2420313943 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd964087230 con 0x7fd96c151d40 2023-03-03T08:53:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.560+0000 7fd970efd700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9500682f0 0x7fd95006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:53:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.561+0000 7fd970efd700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9500682f0 0x7fd95006a7a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd960009ac0 tx=0x7fd960009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:53:02.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.564+0000 7fd9697fa700 1 -- 172.21.15.7:0/2420313943 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd9640534e0 con 0x7fd96c151d40 2023-03-03T08:53:02.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.806+0000 7fd973161700 1 -- 172.21.15.7:0/2420313943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd96c074bb0 con 0x7fd96c151d40 2023-03-03T08:53:02.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.806+0000 7fd9697fa700 1 -- 172.21.15.7:0/2420313943 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd964020090 con 0x7fd96c151d40 2023-03-03T08:53:02.809 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:53:02.809 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:53:02.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 -- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9500682f0 msgr2=0x7fd95006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:02.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9500682f0 0x7fd95006a7a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd960009ac0 tx=0x7fd960009210 comp rx=0 tx=0).stop 2023-03-03T08:53:02.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 -- 172.21.15.7:0/2420313943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c151d40 msgr2=0x7fd96c1ed020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:53:02.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c151d40 0x7fd96c1ed020 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd96400b370 tx=0x7fd964004ad0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 -- 172.21.15.7:0/2420313943 shutdown_connections 2023-03-03T08:53:02.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9500682f0 0x7fd95006a7a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd96c151d40 0x7fd96c1ed020 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd96c073c30 0x7fd96c0acb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 --2- 172.21.15.7:0/2420313943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd96c0c2930 0x7fd96c153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:53:02.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 -- 172.21.15.7:0/2420313943 >> 172.21.15.7:0/2420313943 conn(0x7fd96c0a94d0 msgr2=0x7fd96c0c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:53:02.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 -- 172.21.15.7:0/2420313943 shutdown_connections 2023-03-03T08:53:02.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:53:02.809+0000 7fd973161700 1 -- 172.21.15.7:0/2420313943 wait complete. 2023-03-03T08:53:02.813 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:53:02.823 INFO:tasks.mds_thrash.fs.[b]:waiting for 117 secs before thrashing 2023-03-03T08:53:07.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:07.029 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:37.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:53:37.230 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:07.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:07.332 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:33.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:54:33.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.417+0000 7f9310def700 1 Processor -- start 2023-03-03T08:54:33.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.417+0000 7f9310def700 1 -- start start 2023-03-03T08:54:33.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9310def700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f930c072250 0x7f930c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:33.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9310def700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f930c0c1ce0 0x7f930c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:33.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9310def700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c1510d0 con 0x7f930c071e80 2023-03-03T08:54:33.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9310def700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930c0c3740 con 0x7f930c072250 2023-03-03T08:54:33.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9310def700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930c070670 con 0x7f930c0c1ce0 2023-03-03T08:54:33.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9309d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f930c072250 0x7f930c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:33.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9309d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f930c072250 0x7f930c14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53650/0 (socket says 172.21.15.7:53650) 2023-03-03T08:54:33.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9309d9b700 1 -- 172.21.15.7:0/3465286420 learned_addr learned my addr 172.21.15.7:0/3465286420 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:54:33.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f930ad9d700 1 --2- 172.21.15.7:0/3465286420 >> v2:172.21.15.143:3300/0 conn(0x7f930c0c1ce0 0x7f930c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:33.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9309d9b700 1 -- 172.21.15.7:0/3465286420 >> v2:172.21.15.143:3300/0 conn(0x7f930c0c1ce0 msgr2=0x7f930c14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:54:33.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9309d9b700 1 --2- 172.21.15.7:0/3465286420 >> v2:172.21.15.143:3300/0 conn(0x7f930c0c1ce0 0x7f930c14e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:33.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.418+0000 7f9309d9b700 1 -- 172.21.15.7:0/3465286420 >> v1:172.21.15.143:6789/0 conn(0x7f930c071e80 legacy=0x7f930c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:54:33.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.419+0000 7f9309d9b700 1 -- 172.21.15.7:0/3465286420 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0c39e0 con 0x7f930c072250 2023-03-03T08:54:33.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.419+0000 7f9309d9b700 1 --2- 172.21.15.7:0/3465286420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 0x7f930c14e1f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f930c071090 tx=0x7f92fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9298b0b38291504d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:33.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.419+0000 7f930959a700 1 -- 172.21.15.7:0/3465286420 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc004230 con 0x7f930c072250 2023-03-03T08:54:33.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.419+0000 7f930959a700 1 -- 172.21.15.7:0/3465286420 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc004390 con 0x7f930c072250 2023-03-03T08:54:33.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.419+0000 7f930959a700 1 -- 172.21.15.7:0/3465286420 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc0036a0 con 0x7f930c072250 2023-03-03T08:54:33.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.419+0000 7f9310def700 1 -- 172.21.15.7:0/3465286420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 msgr2=0x7f930c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:33.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.419+0000 7f9310def700 1 --2- 172.21.15.7:0/3465286420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 0x7f930c14e1f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f930c071090 tx=0x7f92fc009bd0 comp rx=0 tx=0).stop 2023-03-03T08:54:33.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.419+0000 7f9310def700 1 -- 172.21.15.7:0/3465286420 shutdown_connections 2023-03-03T08:54:33.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.419+0000 7f9310def700 1 --2- 172.21.15.7:0/3465286420 >> v2:172.21.15.143:3300/0 conn(0x7f930c0c1ce0 0x7f930c14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:33.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.419+0000 7f9310def700 1 --2- 172.21.15.7:0/3465286420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 0x7f930c14e1f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:33.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.420+0000 7f9310def700 1 -- 172.21.15.7:0/3465286420 >> 172.21.15.7:0/3465286420 conn(0x7f930c0a94d0 msgr2=0x7f930c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:54:33.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.420+0000 7f9310def700 1 -- 172.21.15.7:0/3465286420 shutdown_connections 2023-03-03T08:54:33.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.420+0000 7f9310def700 1 -- 172.21.15.7:0/3465286420 wait complete. 2023-03-03T08:54:33.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.421+0000 7f9310def700 1 Processor -- start 2023-03-03T08:54:33.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.421+0000 7f9310def700 1 -- start start 2023-03-03T08:54:33.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9310def700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f930c071e80 0x7f930c1496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:33.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9310def700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 0x7f930c148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:33.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9310def700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f930c0c1ce0 0x7f930c143bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:33.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9310def700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c1491c0 con 0x7f930c072250 2023-03-03T08:54:33.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9310def700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c1e9cf0 con 0x7f930c0c1ce0 2023-03-03T08:54:33.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9310def700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c14a4a0 con 0x7f930c071e80 2023-03-03T08:54:33.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9309d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 0x7f930c148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9309d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 0x7f930c148700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53656/0 (socket says 172.21.15.7:53656) 2023-03-03T08:54:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9309d9b700 1 -- 172.21.15.7:0/3945637553 learned_addr learned my addr 172.21.15.7:0/3945637553 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:54:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9309d9b700 1 -- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f930c071e80 msgr2=0x7f930c1496d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9309d9b700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f930c071e80 0x7f930c1496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9309d9b700 1 -- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f930c0c1ce0 msgr2=0x7f930c143bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9309d9b700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f930c0c1ce0 0x7f930c143bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9309d9b700 1 -- 172.21.15.7:0/3945637553 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc009550 con 0x7f930c072250 2023-03-03T08:54:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9309d9b700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 0x7f930c148700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009130 tx=0x7f92fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:33.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f92faffd700 1 -- 172.21.15.7:0/3945637553 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc003dc0 con 0x7f930c072250 2023-03-03T08:54:33.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f92faffd700 1 -- 172.21.15.7:0/3945637553 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc01f070 con 0x7f930c072250 2023-03-03T08:54:33.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f92faffd700 1 -- 172.21.15.7:0/3945637553 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc011870 con 0x7f930c072250 2023-03-03T08:54:33.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9310def700 1 -- 172.21.15.7:0/3945637553 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c151f30 con 0x7f930c072250 2023-03-03T08:54:33.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.422+0000 7f9310def700 1 -- 172.21.15.7:0/3945637553 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c151300 con 0x7f930c072250 2023-03-03T08:54:33.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.423+0000 7f9310def700 1 -- 172.21.15.7:0/3945637553 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c06c9a0 con 0x7f930c072250 2023-03-03T08:54:33.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.424+0000 7f92faffd700 1 -- 172.21.15.7:0/3945637553 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f92fc017020 con 0x7f930c072250 2023-03-03T08:54:33.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.424+0000 7f92faffd700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ec0682f0 0x7f92ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:33.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.425+0000 7f92faffd700 1 -- 172.21.15.7:0/3945637553 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f92fc087940 con 0x7f930c072250 2023-03-03T08:54:33.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.425+0000 7f930a59c700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ec0682f0 0x7f92ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:33.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.425+0000 7f930a59c700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ec0682f0 0x7f92ec06a7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9300001600 tx=0x7f9300007a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.429+0000 7f92faffd700 1 -- 172.21.15.7:0/3945637553 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f92fc053ce0 con 0x7f930c072250 2023-03-03T08:54:33.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.670+0000 7f9310def700 1 -- 172.21.15.7:0/3945637553 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f930c074bb0 con 0x7f930c072250 2023-03-03T08:54:33.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.670+0000 7f92faffd700 1 -- 172.21.15.7:0/3945637553 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f92fc057300 con 0x7f930c072250 2023-03-03T08:54:33.672 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:54:33.672 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:54:33.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.672+0000 7f9310def700 1 -- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ec0682f0 msgr2=0x7f92ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:33.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.672+0000 7f9310def700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ec0682f0 0x7f92ec06a7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9300001600 tx=0x7f9300007a20 comp rx=0 tx=0).stop 2023-03-03T08:54:33.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.672+0000 7f9310def700 1 -- 172.21.15.7:0/3945637553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 msgr2=0x7f930c148700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:33.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.672+0000 7f9310def700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 0x7f930c148700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f92fc009130 tx=0x7f92fc004750 comp rx=0 tx=0).stop 2023-03-03T08:54:33.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.672+0000 7f9310def700 1 -- 172.21.15.7:0/3945637553 shutdown_connections 2023-03-03T08:54:33.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.672+0000 7f9310def700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f930c0c1ce0 0x7f930c143bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:33.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.672+0000 7f9310def700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f930c071e80 0x7f930c1496d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:33.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.672+0000 7f9310def700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92ec0682f0 0x7f92ec06a7a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:33.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.672+0000 7f9310def700 1 --2- 172.21.15.7:0/3945637553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f930c072250 0x7f930c148700 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:33.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.672+0000 7f9310def700 1 -- 172.21.15.7:0/3945637553 >> 172.21.15.7:0/3945637553 conn(0x7f930c0a94d0 msgr2=0x7f930c06a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:54:33.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.673+0000 7f9310def700 1 -- 172.21.15.7:0/3945637553 shutdown_connections 2023-03-03T08:54:33.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:33.673+0000 7f9310def700 1 -- 172.21.15.7:0/3945637553 wait complete. 2023-03-03T08:54:33.676 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:54:33.686 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before thrashing 2023-03-03T08:54:35.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:54:35.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.825+0000 7fa541813700 1 Processor -- start 2023-03-03T08:54:35.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.825+0000 7fa541813700 1 -- start start 2023-03-03T08:54:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.825+0000 7fa541813700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa53c0b00b0 0x7fa53c0c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.825+0000 7fa541813700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c0c7590 0x7fa53c15ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa541813700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53c0c6cf0 con 0x7fa53c151cc0 2023-03-03T08:54:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa541813700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c152090 con 0x7fa53c0b00b0 2023-03-03T08:54:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa541813700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c1548f0 con 0x7fa53c0c7590 2023-03-03T08:54:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa53affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa53c151cc0 0x7fa53c0ca6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47646/0 (socket says 172.21.15.7:47646) 2023-03-03T08:54:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa53affd700 1 -- 172.21.15.7:0/455527964 learned_addr learned my addr 172.21.15.7:0/455527964 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:54:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa53a7fc700 1 --2- 172.21.15.7:0/455527964 >> v2:172.21.15.143:3300/0 conn(0x7fa53c0b00b0 0x7fa53c0c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa53b7fe700 1 --2- 172.21.15.7:0/455527964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c0c7590 0x7fa53c15ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa539ffb700 1 -- 172.21.15.7:0/455527964 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131230231 0 0) 0x7fa53c0c6cf0 con 0x7fa53c151cc0 2023-03-03T08:54:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa539ffb700 1 -- 172.21.15.7:0/455527964 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5240035b0 con 0x7fa53c151cc0 2023-03-03T08:54:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa539ffb700 1 -- 172.21.15.7:0/455527964 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4091035166 0 0) 0x7fa5240035b0 con 0x7fa53c151cc0 2023-03-03T08:54:35.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa53a7fc700 1 -- 172.21.15.7:0/455527964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c0c7590 msgr2=0x7fa53c15ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:54:35.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa53a7fc700 1 --2- 172.21.15.7:0/455527964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c0c7590 0x7fa53c15ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:35.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa53a7fc700 1 -- 172.21.15.7:0/455527964 >> v1:172.21.15.7:6789/0 conn(0x7fa53c151cc0 legacy=0x7fa53c0ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:35.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa53a7fc700 1 -- 172.21.15.7:0/455527964 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c154bd0 con 0x7fa53c0b00b0 2023-03-03T08:54:35.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa53b7fe700 1 --2- 172.21.15.7:0/455527964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c0c7590 0x7fa53c15ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:54:35.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.826+0000 7fa53a7fc700 1 --2- 172.21.15.7:0/455527964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c0b00b0 0x7fa53c0c7050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa530009840 tx=0x7fa530009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88ef4829871a7080 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:35.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa539ffb700 1 -- 172.21.15.7:0/455527964 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c001670 con 0x7fa53c0b00b0 2023-03-03T08:54:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa539ffb700 1 -- 172.21.15.7:0/455527964 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa530004030 con 0x7fa53c0b00b0 2023-03-03T08:54:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa539ffb700 1 -- 172.21.15.7:0/455527964 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5300043c0 con 0x7fa53c0b00b0 2023-03-03T08:54:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa541813700 1 -- 172.21.15.7:0/455527964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c0b00b0 msgr2=0x7fa53c0c7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa541813700 1 --2- 172.21.15.7:0/455527964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c0b00b0 0x7fa53c0c7050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa530009840 tx=0x7fa530009b50 comp rx=0 tx=0).stop 2023-03-03T08:54:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa541813700 1 -- 172.21.15.7:0/455527964 shutdown_connections 2023-03-03T08:54:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa541813700 1 --2- 172.21.15.7:0/455527964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c0c7590 0x7fa53c15ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa541813700 1 --2- 172.21.15.7:0/455527964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c0b00b0 0x7fa53c0c7050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa541813700 1 -- 172.21.15.7:0/455527964 >> 172.21.15.7:0/455527964 conn(0x7fa53c06b980 msgr2=0x7fa53c071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:54:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa541813700 1 -- 172.21.15.7:0/455527964 shutdown_connections 2023-03-03T08:54:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.827+0000 7fa541813700 1 -- 172.21.15.7:0/455527964 wait complete. 2023-03-03T08:54:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.828+0000 7fa541813700 1 Processor -- start 2023-03-03T08:54:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa541813700 1 -- start start 2023-03-03T08:54:35.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa541813700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c0b00b0 0x7fa53c1507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:35.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa541813700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c0c7590 0x7fa53c1ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:35.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa541813700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c151cc0 0x7fa53c1ead00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa541813700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c147510 con 0x7fa53c0c7590 2023-03-03T08:54:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa541813700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c156e40 con 0x7fa53c0b00b0 2023-03-03T08:54:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa541813700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c157120 con 0x7fa53c151cc0 2023-03-03T08:54:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa53a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c0c7590 0x7fa53c1ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa53a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c0c7590 0x7fa53c1ea7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53674/0 (socket says 172.21.15.7:53674) 2023-03-03T08:54:35.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa53a7fc700 1 -- 172.21.15.7:0/3164178780 learned_addr learned my addr 172.21.15.7:0/3164178780 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:54:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.829+0000 7fa53affd700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c0b00b0 0x7fa53c1507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa53b7fe700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c151cc0 0x7fa53c1ead00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa53a7fc700 1 -- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c151cc0 msgr2=0x7fa53c1ead00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa53a7fc700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c151cc0 0x7fa53c1ead00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa53a7fc700 1 -- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c0b00b0 msgr2=0x7fa53c1507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:35.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa53a7fc700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c0b00b0 0x7fa53c1507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa53a7fc700 1 -- 172.21.15.7:0/3164178780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa534009a70 con 0x7fa53c0c7590 2023-03-03T08:54:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa53a7fc700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c0c7590 0x7fa53c1ea7c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa5300090b0 tx=0x7fa530004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa523fff700 1 -- 172.21.15.7:0/3164178780 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530003e20 con 0x7fa53c0c7590 2023-03-03T08:54:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa523fff700 1 -- 172.21.15.7:0/3164178780 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53001f070 con 0x7fa53c0c7590 2023-03-03T08:54:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa541813700 1 -- 172.21.15.7:0/3164178780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5300094a0 con 0x7fa53c0c7590 2023-03-03T08:54:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa523fff700 1 -- 172.21.15.7:0/3164178780 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530011870 con 0x7fa53c0c7590 2023-03-03T08:54:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.830+0000 7fa541813700 1 -- 172.21.15.7:0/3164178780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c155190 con 0x7fa53c0c7590 2023-03-03T08:54:35.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.831+0000 7fa541813700 1 -- 172.21.15.7:0/3164178780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c154eb0 con 0x7fa53c0c7590 2023-03-03T08:54:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.832+0000 7fa523fff700 1 -- 172.21.15.7:0/3164178780 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa530017020 con 0x7fa53c0c7590 2023-03-03T08:54:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.832+0000 7fa523fff700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 0x7fa52806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.832+0000 7fa523fff700 1 -- 172.21.15.7:0/3164178780 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa530021070 con 0x7fa53c0c7590 2023-03-03T08:54:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.832+0000 7fa53affd700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 0x7fa52806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.833+0000 7fa53affd700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 0x7fa52806a7a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa52c001eb0 tx=0x7fa52c001c40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:35.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:35.837+0000 7fa523fff700 1 -- 172.21.15.7:0/3164178780 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa530053dc0 con 0x7fa53c0c7590 2023-03-03T08:54:36.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.070+0000 7fa541813700 1 -- 172.21.15.7:0/3164178780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa53c1572e0 con 0x7fa53c0c7590 2023-03-03T08:54:36.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.071+0000 7fa523fff700 1 -- 172.21.15.7:0/3164178780 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa5300573e0 con 0x7fa53c0c7590 2023-03-03T08:54:36.072 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:54:36.073 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:54:36.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 -- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 msgr2=0x7fa52806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:36.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 0x7fa52806a7a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa52c001eb0 tx=0x7fa52c001c40 comp rx=0 tx=0).stop 2023-03-03T08:54:36.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 -- 172.21.15.7:0/3164178780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c0c7590 msgr2=0x7fa53c1ea7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:36.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c0c7590 0x7fa53c1ea7c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa5300090b0 tx=0x7fa530004750 comp rx=0 tx=0).stop 2023-03-03T08:54:36.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 -- 172.21.15.7:0/3164178780 shutdown_connections 2023-03-03T08:54:36.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c151cc0 0x7fa53c1ead00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:36.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c0b00b0 0x7fa53c1507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:36.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 0x7fa52806a7a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 --2- 172.21.15.7:0/3164178780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c0c7590 0x7fa53c1ea7c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 -- 172.21.15.7:0/3164178780 >> 172.21.15.7:0/3164178780 conn(0x7fa53c06b980 msgr2=0x7fa53c0c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:54:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 -- 172.21.15.7:0/3164178780 shutdown_connections 2023-03-03T08:54:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:36.073+0000 7fa541813700 1 -- 172.21.15.7:0/3164178780 wait complete. 2023-03-03T08:54:36.076 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:54:36.086 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2023-03-03T08:54:37.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:37.495 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:54:59.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:54:59.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.962+0000 7fac0f744700 1 Processor -- start 2023-03-03T08:54:59.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.962+0000 7fac0f744700 1 -- start start 2023-03-03T08:54:59.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0f744700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac0806c480 0x7fac081581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0f744700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0806cb70 con 0x7fac080c2950 2023-03-03T08:54:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0f744700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0806ce50 con 0x7fac08151b20 2023-03-03T08:54:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0f744700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0806c2f0 con 0x7fac0806c480 2023-03-03T08:54:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0dce1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac0806c480 0x7fac081581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0ccdf700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fac080c2950 0x7fac08157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47680/0 (socket says 172.21.15.7:47680) 2023-03-03T08:54:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0ccdf700 1 -- 172.21.15.7:0/3081695672 learned_addr learned my addr 172.21.15.7:0/3081695672 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:54:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0dce1700 1 -- 172.21.15.7:0/3081695672 >> v1:172.21.15.143:6789/0 conn(0x7fac08151b20 legacy=0x7fac080736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:54:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0dce1700 1 -- 172.21.15.7:0/3081695672 >> v1:172.21.15.7:6789/0 conn(0x7fac080c2950 legacy=0x7fac08157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0dce1700 1 -- 172.21.15.7:0/3081695672 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0806d130 con 0x7fac0806c480 2023-03-03T08:54:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fabfffff700 1 -- 172.21.15.7:0/3081695672 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142950437 0 0) 0x7fac0806cb70 con 0x7fac080c2950 2023-03-03T08:54:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.963+0000 7fac0dce1700 1 --2- 172.21.15.7:0/3081695672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 0x7fac081581f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fac00009d90 tx=0x7fac0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94cba0233608c05 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.964+0000 7fabfffff700 1 -- 172.21.15.7:0/3081695672 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0000e070 con 0x7fac0806c480 2023-03-03T08:54:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.964+0000 7fabfffff700 1 -- 172.21.15.7:0/3081695672 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0000bb90 con 0x7fac0806c480 2023-03-03T08:54:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.964+0000 7fabfffff700 1 -- 172.21.15.7:0/3081695672 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac00003a20 con 0x7fac0806c480 2023-03-03T08:54:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.964+0000 7fac0f744700 1 -- 172.21.15.7:0/3081695672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 msgr2=0x7fac081581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.964+0000 7fac0f744700 1 --2- 172.21.15.7:0/3081695672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 0x7fac081581f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fac00009d90 tx=0x7fac0000b330 comp rx=0 tx=0).stop 2023-03-03T08:54:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.964+0000 7fac0f744700 1 -- 172.21.15.7:0/3081695672 shutdown_connections 2023-03-03T08:54:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.964+0000 7fac0f744700 1 --2- 172.21.15.7:0/3081695672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 0x7fac081581f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.964+0000 7fac0f744700 1 -- 172.21.15.7:0/3081695672 >> 172.21.15.7:0/3081695672 conn(0x7fac080a94d0 msgr2=0x7fac080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:54:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.964+0000 7fac0f744700 1 -- 172.21.15.7:0/3081695672 shutdown_connections 2023-03-03T08:54:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.964+0000 7fac0f744700 1 -- 172.21.15.7:0/3081695672 wait complete. 2023-03-03T08:54:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.965+0000 7fac0f744700 1 Processor -- start 2023-03-03T08:54:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.965+0000 7fac0f744700 1 -- start start 2023-03-03T08:54:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0f744700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 0x7fac0814a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0f744700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac080c2950 0x7fac0814d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0f744700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac08151b20 0x7fac081ea4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0f744700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac08154170 con 0x7fac0806c480 2023-03-03T08:54:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0f744700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0814c730 con 0x7fac08151b20 2023-03-03T08:54:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0f744700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0814ca10 con 0x7fac080c2950 2023-03-03T08:54:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0d4e0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 0x7fac0814a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0d4e0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 0x7fac0814a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53704/0 (socket says 172.21.15.7:53704) 2023-03-03T08:54:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0d4e0700 1 -- 172.21.15.7:0/2370299814 learned_addr learned my addr 172.21.15.7:0/2370299814 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:54:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0dce1700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac08151b20 0x7fac081ea4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.966+0000 7fac0ccdf700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac080c2950 0x7fac0814d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fac0d4e0700 1 -- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac080c2950 msgr2=0x7fac0814d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fac0d4e0700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac080c2950 0x7fac0814d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fac0d4e0700 1 -- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac08151b20 msgr2=0x7fac081ea4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:54:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fac0d4e0700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac08151b20 0x7fac081ea4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:54:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fac0d4e0700 1 -- 172.21.15.7:0/2370299814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac00009a70 con 0x7fac0806c480 2023-03-03T08:54:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fac0dce1700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac08151b20 0x7fac081ea4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:54:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fac0d4e0700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 0x7fac0814a8c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fac0400b4a0 tx=0x7fac0400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fabfdffb700 1 -- 172.21.15.7:0/2370299814 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac04011830 con 0x7fac0806c480 2023-03-03T08:54:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fabfdffb700 1 -- 172.21.15.7:0/2370299814 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac04011990 con 0x7fac0806c480 2023-03-03T08:54:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fac0f744700 1 -- 172.21.15.7:0/2370299814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac08158730 con 0x7fac0806c480 2023-03-03T08:54:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fabfdffb700 1 -- 172.21.15.7:0/2370299814 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac04011d40 con 0x7fac0806c480 2023-03-03T08:54:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.967+0000 7fac0f744700 1 -- 172.21.15.7:0/2370299814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0806d410 con 0x7fac0806c480 2023-03-03T08:54:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.968+0000 7fac0f744700 1 -- 172.21.15.7:0/2370299814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac08151ef0 con 0x7fac0806c480 2023-03-03T08:54:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.968+0000 7fabfdffb700 1 -- 172.21.15.7:0/2370299814 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fac0400f050 con 0x7fac0806c480 2023-03-03T08:54:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.969+0000 7fabfdffb700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabec0682f0 0x7fabec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:54:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.969+0000 7fabfdffb700 1 -- 172.21.15.7:0/2370299814 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fac04085df0 con 0x7fac0806c480 2023-03-03T08:54:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.969+0000 7fac0ccdf700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabec0682f0 0x7fabec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:54:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.970+0000 7fac0ccdf700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabec0682f0 0x7fabec06a7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fac0814de30 tx=0x7fabf8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:54:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:54:59.973+0000 7fabfdffb700 1 -- 172.21.15.7:0/2370299814 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fac04052130 con 0x7fac0806c480 2023-03-03T08:55:00.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.214+0000 7fac0f744700 1 -- 172.21.15.7:0/2370299814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac081ea320 con 0x7fac0806c480 2023-03-03T08:55:00.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.215+0000 7fabfdffb700 1 -- 172.21.15.7:0/2370299814 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fac04055750 con 0x7fac0806c480 2023-03-03T08:55:00.216 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:55:00.216 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:55:00.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 -- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabec0682f0 msgr2=0x7fabec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:00.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabec0682f0 0x7fabec06a7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fac0814de30 tx=0x7fabf8009040 comp rx=0 tx=0).stop 2023-03-03T08:55:00.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 -- 172.21.15.7:0/2370299814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 msgr2=0x7fac0814a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:00.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 0x7fac0814a8c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fac0400b4a0 tx=0x7fac0400b860 comp rx=0 tx=0).stop 2023-03-03T08:55:00.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 -- 172.21.15.7:0/2370299814 shutdown_connections 2023-03-03T08:55:00.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac08151b20 0x7fac081ea4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:00.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabec0682f0 0x7fabec06a7a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:00.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac0806c480 0x7fac0814a8c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:00.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 --2- 172.21.15.7:0/2370299814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac080c2950 0x7fac0814d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:00.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 -- 172.21.15.7:0/2370299814 >> 172.21.15.7:0/2370299814 conn(0x7fac080a94d0 msgr2=0x7fac08154e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:55:00.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 -- 172.21.15.7:0/2370299814 shutdown_connections 2023-03-03T08:55:00.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:00.217+0000 7fac0f744700 1 -- 172.21.15.7:0/2370299814 wait complete. 2023-03-03T08:55:00.220 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:55:00.230 INFO:tasks.mds_thrash.fs.[b]:waiting for 58 secs before thrashing 2023-03-03T08:55:07.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:07.558 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:37.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:37.626 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:55:44.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:55:44.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.224+0000 7f7494bae700 1 Processor -- start 2023-03-03T08:55:44.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.225+0000 7f7494bae700 1 -- start start 2023-03-03T08:55:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.225+0000 7f7494bae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74901525d0 0x7f74900c1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.225+0000 7f7494bae700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749006d460 con 0x7f74901510f0 2023-03-03T08:55:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.225+0000 7f7494bae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749014e4e0 con 0x7f74900c35a0 2023-03-03T08:55:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.225+0000 7f7494bae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74900c1c70 con 0x7f74901525d0 2023-03-03T08:55:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748e59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f74901510f0 0x7f7490151ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47724/0 (socket says 172.21.15.7:47724) 2023-03-03T08:55:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748e59c700 1 -- 172.21.15.7:0/198683927 learned_addr learned my addr 172.21.15.7:0/198683927 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:55:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748dd9b700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74901525d0 0x7f74900c1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880768558 0 0) 0x7f749006d460 con 0x7f74901510f0 2023-03-03T08:55:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f747c0046c0 con 0x7f74901510f0 2023-03-03T08:55:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 358507493 0 0) 0x7f747c0046c0 con 0x7f74901510f0 2023-03-03T08:55:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74901525d0 msgr2=0x7f74900c1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:55:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74901525d0 0x7f74900c1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 >> v1:172.21.15.143:6789/0 conn(0x7f74900c35a0 legacy=0x7f74900c3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900c2d60 con 0x7f74901510f0 2023-03-03T08:55:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7480001670 con 0x7f74901510f0 2023-03-03T08:55:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 >> v1:172.21.15.7:6789/0 conn(0x7f74901510f0 legacy=0x7f7490151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:44.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f747c004b40 0x7f747c004f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:44.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74901525d0 0x7f747c005960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:44.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 0x7f747c006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:44.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c00cbd0 con 0x7f747c005ea0 2023-03-03T08:55:44.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c00cd10 con 0x7f747c004b40 2023-03-03T08:55:44.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.226+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c003520 con 0x7f74901525d0 2023-03-03T08:55:44.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748ed9d700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 0x7f747c006310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:44.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748dd9b700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74901525d0 0x7f747c005960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:44.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748e59c700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f747c004b40 0x7f747c004f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:44.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748ed9d700 1 -- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74901525d0 msgr2=0x7f747c005960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:55:44.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748ed9d700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74901525d0 0x7f747c005960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:44.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748ed9d700 1 -- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f747c004b40 msgr2=0x7f747c004f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:55:44.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748ed9d700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f747c004b40 0x7f747c004f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:44.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748ed9d700 1 -- 172.21.15.7:0/198683927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c0046c0 con 0x7f747c005ea0 2023-03-03T08:55:44.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748ed9d700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 0x7f747c006310 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f748800c190 tx=0x7f747c006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b609c0419c52d9a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:44.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488001db0 con 0x7f747c005ea0 2023-03-03T08:55:44.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748800ccd0 con 0x7f747c005ea0 2023-03-03T08:55:44.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.227+0000 7f748d59a700 1 -- 172.21.15.7:0/198683927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488003c40 con 0x7f747c005ea0 2023-03-03T08:55:44.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.228+0000 7f7494bae700 1 -- 172.21.15.7:0/198683927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 msgr2=0x7f747c006310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:44.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.228+0000 7f7494bae700 1 --2- 172.21.15.7:0/198683927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 0x7f747c006310 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f748800c190 tx=0x7f747c006990 comp rx=0 tx=0).stop 2023-03-03T08:55:44.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.228+0000 7f748ed9d700 1 -- 172.21.15.7:0/198683927 reap_dead start 2023-03-03T08:55:44.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.228+0000 7f7494bae700 1 -- 172.21.15.7:0/198683927 shutdown_connections 2023-03-03T08:55:44.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.228+0000 7f7494bae700 1 -- 172.21.15.7:0/198683927 >> 172.21.15.7:0/198683927 conn(0x7f749006a2b0 msgr2=0x7f749006a680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:55:44.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.228+0000 7f7494bae700 1 -- 172.21.15.7:0/198683927 shutdown_connections 2023-03-03T08:55:44.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.228+0000 7f7494bae700 1 -- 172.21.15.7:0/198683927 wait complete. 2023-03-03T08:55:44.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.230+0000 7f7494bae700 1 Processor -- start 2023-03-03T08:55:44.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.230+0000 7f7494bae700 1 -- start start 2023-03-03T08:55:44.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f7494bae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 0x7f74901e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:44.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f7494bae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74900708b0 0x7f74900ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:44.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f7494bae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7490071fb0 0x7f7490072420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:44.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f7494bae700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74901ea2b0 con 0x7f747c005ea0 2023-03-03T08:55:44.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f7494bae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74900aa7b0 con 0x7f74900708b0 2023-03-03T08:55:44.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f7494bae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74900c6db0 con 0x7f7490071fb0 2023-03-03T08:55:44.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 0x7f74901e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:44.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 0x7f74901e9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53756/0 (socket says 172.21.15.7:53756) 2023-03-03T08:55:44.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748e59c700 1 -- 172.21.15.7:0/305323926 learned_addr learned my addr 172.21.15.7:0/305323926 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:55:44.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748dd9b700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74900708b0 0x7f74900ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:44.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748ed9d700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7490071fb0 0x7f7490072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:44.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748e59c700 1 -- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7490071fb0 msgr2=0x7f7490072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:44.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748e59c700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7490071fb0 0x7f7490072420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:44.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748e59c700 1 -- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74900708b0 msgr2=0x7f74900ac1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:44.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748e59c700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74900708b0 0x7f74900ac1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:44.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748e59c700 1 -- 172.21.15.7:0/305323926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74900c3000 con 0x7f747c005ea0 2023-03-03T08:55:44.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f748e59c700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 0x7f74901e9c60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f74800072f0 tx=0x7f74800076b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:44.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f7486ffd700 1 -- 172.21.15.7:0/305323926 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74800014a0 con 0x7f747c005ea0 2023-03-03T08:55:44.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f7486ffd700 1 -- 172.21.15.7:0/305323926 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748000f8d0 con 0x7f747c005ea0 2023-03-03T08:55:44.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f7494bae700 1 -- 172.21.15.7:0/305323926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7490151820 con 0x7f747c005ea0 2023-03-03T08:55:44.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.231+0000 7f7486ffd700 1 -- 172.21.15.7:0/305323926 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748000fc80 con 0x7f747c005ea0 2023-03-03T08:55:44.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.232+0000 7f7494bae700 1 -- 172.21.15.7:0/305323926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74900c32e0 con 0x7f747c005ea0 2023-03-03T08:55:44.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.233+0000 7f7494bae700 1 -- 172.21.15.7:0/305323926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7490153cc0 con 0x7f747c005ea0 2023-03-03T08:55:44.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.233+0000 7f7486ffd700 1 -- 172.21.15.7:0/305323926 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f748001d460 con 0x7f747c005ea0 2023-03-03T08:55:44.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.233+0000 7f7486ffd700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 0x7f747006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:44.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.234+0000 7f7486ffd700 1 -- 172.21.15.7:0/305323926 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f74800863b0 con 0x7f747c005ea0 2023-03-03T08:55:44.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.234+0000 7f748dd9b700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 0x7f747006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:44.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.234+0000 7f748dd9b700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 0x7f747006a7a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f74900c1020 tx=0x7f7478008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:44.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.238+0000 7f7486ffd700 1 -- 172.21.15.7:0/305323926 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f74800526f0 con 0x7f747c005ea0 2023-03-03T08:55:44.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.476+0000 7f7494bae700 1 -- 172.21.15.7:0/305323926 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7490074bb0 con 0x7f747c005ea0 2023-03-03T08:55:44.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.477+0000 7f7486ffd700 1 -- 172.21.15.7:0/305323926 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7480055d10 con 0x7f747c005ea0 2023-03-03T08:55:44.479 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:55:44.479 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:55:44.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 -- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 msgr2=0x7f747006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:44.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 0x7f747006a7a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f74900c1020 tx=0x7f7478008040 comp rx=0 tx=0).stop 2023-03-03T08:55:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 -- 172.21.15.7:0/305323926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 msgr2=0x7f74901e9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 0x7f74901e9c60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f74800072f0 tx=0x7f74800076b0 comp rx=0 tx=0).stop 2023-03-03T08:55:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 -- 172.21.15.7:0/305323926 shutdown_connections 2023-03-03T08:55:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7490071fb0 0x7f7490072420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 0x7f747006a7a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:44.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f747c005ea0 0x7f74901e9c60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:44.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 --2- 172.21.15.7:0/305323926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74900708b0 0x7f74900ac1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:44.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 -- 172.21.15.7:0/305323926 >> 172.21.15.7:0/305323926 conn(0x7f749006a2b0 msgr2=0x7f74900a6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:55:44.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 -- 172.21.15.7:0/305323926 shutdown_connections 2023-03-03T08:55:44.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:44.479+0000 7f7494bae700 1 -- 172.21.15.7:0/305323926 wait complete. 2023-03-03T08:55:44.482 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:55:44.492 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T08:55:58.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:55:58.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.368+0000 7f7cd39a9700 1 Processor -- start 2023-03-03T08:55:58.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.369+0000 7f7cd39a9700 1 -- start start 2023-03-03T08:55:58.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.369+0000 7f7cd39a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14fda0 0x7f7ccc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:58.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.369+0000 7f7cd39a9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7ccc0abe20 0x7f7ccc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:58.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.369+0000 7f7cd39a9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc0c2f30 con 0x7f7ccc14f9d0 2023-03-03T08:55:58.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.369+0000 7f7cd39a9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc0c31d0 con 0x7f7ccc0abe20 2023-03-03T08:55:58.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.369+0000 7f7cd39a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc1524a0 con 0x7f7ccc14fda0 2023-03-03T08:55:58.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.369+0000 7f7cd1745700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7ccc14f9d0 0x7f7ccc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47748/0 (socket says 172.21.15.7:47748) 2023-03-03T08:55:58.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.369+0000 7f7cd1745700 1 -- 172.21.15.7:0/469702609 learned_addr learned my addr 172.21.15.7:0/469702609 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:55:58.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898588680 0 0) 0x7f7ccc0c2f30 con 0x7f7ccc14f9d0 2023-03-03T08:55:58.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb80035b0 con 0x7f7ccc14f9d0 2023-03-03T08:55:58.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cd1f46700 1 --2- 172.21.15.7:0/469702609 >> v2:172.21.15.143:3300/0 conn(0x7f7ccc0abe20 0x7f7ccc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:58.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cd0f44700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14fda0 0x7f7ccc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:58.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1806670202 0 0) 0x7f7cb80035b0 con 0x7f7ccc14f9d0 2023-03-03T08:55:58.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14fda0 msgr2=0x7f7ccc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:55:58.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14fda0 0x7f7ccc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:58.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 >> v2:172.21.15.143:3300/0 conn(0x7f7ccc0abe20 msgr2=0x7f7ccc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:55:58.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 --2- 172.21.15.7:0/469702609 >> v2:172.21.15.143:3300/0 conn(0x7f7ccc0abe20 0x7f7ccc1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:58.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc0c34b0 con 0x7f7ccc14f9d0 2023-03-03T08:55:58.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7cc8001670 con 0x7f7ccc14f9d0 2023-03-03T08:55:58.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 >> v1:172.21.15.7:6789/0 conn(0x7f7ccc14f9d0 legacy=0x7f7ccc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:58.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb8003a70 0x7f7cb8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:58.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14fda0 0x7f7cb8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:58.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cb8004d90 0x7f7cb8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:58.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0c31d0 con 0x7f7cb8004d90 2023-03-03T08:55:58.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc1524a0 con 0x7f7cb8003a70 2023-03-03T08:55:58.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb800baf0 con 0x7f7ccc14fda0 2023-03-03T08:55:58.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cd1f46700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cb8004d90 0x7f7cb8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:58.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cd1745700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb8003a70 0x7f7cb8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:58.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.370+0000 7f7cd0f44700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14fda0 0x7f7cb8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:58.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cd1f46700 1 -- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14fda0 msgr2=0x7f7cb8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:55:58.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cd1f46700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14fda0 0x7f7cb8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:58.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cd1f46700 1 -- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb8003a70 msgr2=0x7f7cb8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:55:58.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cd1f46700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb8003a70 0x7f7cb8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:58.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cd1f46700 1 -- 172.21.15.7:0/469702609 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb80035b0 con 0x7f7cb8004d90 2023-03-03T08:55:58.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cd1f46700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cb8004d90 0x7f7cb8005200 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009fc0 tx=0x7f7cb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425bef5d3a11128c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:58.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc400ec30 con 0x7f7cb8004d90 2023-03-03T08:55:58.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc400f040 con 0x7f7cb8004d90 2023-03-03T08:55:58.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cbffff700 1 -- 172.21.15.7:0/469702609 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4014480 con 0x7f7cb8004d90 2023-03-03T08:55:58.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cd39a9700 1 -- 172.21.15.7:0/469702609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cb8004d90 msgr2=0x7f7cb8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:58.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cd39a9700 1 --2- 172.21.15.7:0/469702609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cb8004d90 0x7f7cb8005200 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009fc0 tx=0x7f7cb8005880 comp rx=0 tx=0).stop 2023-03-03T08:55:58.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.371+0000 7f7cd1f46700 1 -- 172.21.15.7:0/469702609 reap_dead start 2023-03-03T08:55:58.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.372+0000 7f7cd39a9700 1 -- 172.21.15.7:0/469702609 shutdown_connections 2023-03-03T08:55:58.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.372+0000 7f7cd39a9700 1 -- 172.21.15.7:0/469702609 >> 172.21.15.7:0/469702609 conn(0x7f7ccc0c8900 msgr2=0x7f7ccc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:55:58.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.372+0000 7f7cd39a9700 1 -- 172.21.15.7:0/469702609 shutdown_connections 2023-03-03T08:55:58.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.372+0000 7f7cd39a9700 1 -- 172.21.15.7:0/469702609 wait complete. 2023-03-03T08:55:58.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.373+0000 7f7cd39a9700 1 Processor -- start 2023-03-03T08:55:58.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.373+0000 7f7cd39a9700 1 -- start start 2023-03-03T08:55:58.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.373+0000 7f7cd39a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb8004d90 0x7f7ccc1eb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:58.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.373+0000 7f7cd39a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ccc1549c0 0x7f7ccc1ed9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:58.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd39a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14d670 0x7f7ccc14daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:58.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd39a9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc122210 con 0x7f7ccc14d670 2023-03-03T08:55:58.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd39a9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc1ed1d0 con 0x7f7ccc1549c0 2023-03-03T08:55:58.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd39a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc1ed4b0 con 0x7f7cb8004d90 2023-03-03T08:55:58.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd1f46700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14d670 0x7f7ccc14daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:58.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd1f46700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14d670 0x7f7ccc14daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53784/0 (socket says 172.21.15.7:53784) 2023-03-03T08:55:58.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd1f46700 1 -- 172.21.15.7:0/974286263 learned_addr learned my addr 172.21.15.7:0/974286263 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:55:58.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd1745700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb8004d90 0x7f7ccc1eb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:58.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd0f44700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ccc1549c0 0x7f7ccc1ed9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:58.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd1f46700 1 -- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb8004d90 msgr2=0x7f7ccc1eb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:58.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd1f46700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb8004d90 0x7f7ccc1eb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:58.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd1f46700 1 -- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ccc1549c0 msgr2=0x7f7ccc1ed9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:58.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd1f46700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ccc1549c0 0x7f7ccc1ed9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:58.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd1f46700 1 -- 172.21.15.7:0/974286263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc0ac350 con 0x7f7ccc14d670 2023-03-03T08:55:58.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd1f46700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14d670 0x7f7ccc14daf0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009f60 tx=0x7f7cc4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:58.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cbdffb700 1 -- 172.21.15.7:0/974286263 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4007000 con 0x7f7ccc14d670 2023-03-03T08:55:58.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cbdffb700 1 -- 172.21.15.7:0/974286263 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc4014cf0 con 0x7f7ccc14d670 2023-03-03T08:55:58.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cd39a9700 1 -- 172.21.15.7:0/974286263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc0ab630 con 0x7f7ccc14d670 2023-03-03T08:55:58.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.374+0000 7f7cbdffb700 1 -- 172.21.15.7:0/974286263 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4014ea0 con 0x7f7ccc14d670 2023-03-03T08:55:58.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.375+0000 7f7cd39a9700 1 -- 172.21.15.7:0/974286263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc071fa0 con 0x7f7ccc14d670 2023-03-03T08:55:58.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.375+0000 7f7cd39a9700 1 -- 172.21.15.7:0/974286263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc1ed820 con 0x7f7ccc14d670 2023-03-03T08:55:58.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.377+0000 7f7cbdffb700 1 -- 172.21.15.7:0/974286263 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7cc402e070 con 0x7f7ccc14d670 2023-03-03T08:55:58.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.377+0000 7f7cbdffb700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb00682f0 0x7f7cb006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:55:58.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.377+0000 7f7cbdffb700 1 -- 172.21.15.7:0/974286263 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7cc40883d0 con 0x7f7ccc14d670 2023-03-03T08:55:58.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.377+0000 7f7cd1745700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb00682f0 0x7f7cb006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:55:58.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.378+0000 7f7cd1745700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb00682f0 0x7f7cb006a7a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8000fb0 tx=0x7f7cc8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:55:58.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.381+0000 7f7cbdffb700 1 -- 172.21.15.7:0/974286263 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7cc40547b0 con 0x7f7ccc14d670 2023-03-03T08:55:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.621+0000 7f7cd39a9700 1 -- 172.21.15.7:0/974286263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ccc074bb0 con 0x7f7ccc14d670 2023-03-03T08:55:58.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.621+0000 7f7cbdffb700 1 -- 172.21.15.7:0/974286263 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7cc4057dd0 con 0x7f7ccc14d670 2023-03-03T08:55:58.623 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:55:58.623 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:55:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 -- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb00682f0 msgr2=0x7f7cb006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb00682f0 0x7f7cb006a7a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8000fb0 tx=0x7f7cc8009040 comp rx=0 tx=0).stop 2023-03-03T08:55:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 -- 172.21.15.7:0/974286263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14d670 msgr2=0x7f7ccc14daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:55:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14d670 0x7f7ccc14daf0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009f60 tx=0x7f7cc4009750 comp rx=0 tx=0).stop 2023-03-03T08:55:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 -- 172.21.15.7:0/974286263 shutdown_connections 2023-03-03T08:55:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb00682f0 0x7f7cb006a7a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:58.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14d670 0x7f7ccc14daf0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cb8004d90 0x7f7ccc1eb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 --2- 172.21.15.7:0/974286263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ccc1549c0 0x7f7ccc1ed9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:55:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 -- 172.21.15.7:0/974286263 >> 172.21.15.7:0/974286263 conn(0x7f7ccc0c8900 msgr2=0x7f7ccc150c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:55:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 -- 172.21.15.7:0/974286263 shutdown_connections 2023-03-03T08:55:58.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:55:58.623+0000 7f7cd39a9700 1 -- 172.21.15.7:0/974286263 wait complete. 2023-03-03T08:55:58.627 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:55:58.636 INFO:tasks.mds_thrash.fs.[b]:waiting for 80 secs before thrashing 2023-03-03T08:56:07.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:07.689 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:26.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:56:26.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.631+0000 7f1fe8040700 1 Processor -- start 2023-03-03T08:56:26.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.631+0000 7f1fe8040700 1 -- start start 2023-03-03T08:56:26.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.632+0000 7f1fe8040700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1fe00c5140 0x7f1fe0157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:26.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.632+0000 7f1fe8040700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe01580e0 0x7f1fe006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:26.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.632+0000 7f1fe8040700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe006dad0 con 0x7f1fe0151de0 2023-03-03T08:56:26.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.632+0000 7f1fe8040700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe006e370 con 0x7f1fe00c5140 2023-03-03T08:56:26.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.632+0000 7f1fe8040700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe006ddb0 con 0x7f1fe01580e0 2023-03-03T08:56:26.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.632+0000 7f1fe5ddc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1fe0151de0 0x7f1fe00c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47796/0 (socket says 172.21.15.7:47796) 2023-03-03T08:56:26.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.632+0000 7f1fe5ddc700 1 -- 172.21.15.7:0/3346763331 learned_addr learned my addr 172.21.15.7:0/3346763331 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:56:26.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.632+0000 7f1fe55db700 1 --2- 172.21.15.7:0/3346763331 >> v2:172.21.15.143:3300/0 conn(0x7f1fe00c5140 0x7f1fe0157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.632+0000 7f1fe65dd700 1 --2- 172.21.15.7:0/3346763331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe01580e0 0x7f1fe006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.632+0000 7f1fe4dda700 1 -- 172.21.15.7:0/3346763331 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288259833 0 0) 0x7f1fe006dad0 con 0x7f1fe0151de0 2023-03-03T08:56:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe4dda700 1 -- 172.21.15.7:0/3346763331 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc0035b0 con 0x7f1fe0151de0 2023-03-03T08:56:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe4dda700 1 -- 172.21.15.7:0/3346763331 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4069582677 0 0) 0x7f1fcc0035b0 con 0x7f1fe0151de0 2023-03-03T08:56:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe55db700 1 -- 172.21.15.7:0/3346763331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe01580e0 msgr2=0x7f1fe006a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:56:26.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe55db700 1 --2- 172.21.15.7:0/3346763331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe01580e0 0x7f1fe006a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:26.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe55db700 1 -- 172.21.15.7:0/3346763331 >> v1:172.21.15.7:6789/0 conn(0x7f1fe0151de0 legacy=0x7f1fe00c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:26.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe55db700 1 -- 172.21.15.7:0/3346763331 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe006e090 con 0x7f1fe00c5140 2023-03-03T08:56:26.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe65dd700 1 --2- 172.21.15.7:0/3346763331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe01580e0 0x7f1fe006a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T08:56:26.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe55db700 1 --2- 172.21.15.7:0/3346763331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe00c5140 0x7f1fe0157ba0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009840 tx=0x7f1fdc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a3a7b6b8212196 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:56:26.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe4dda700 1 -- 172.21.15.7:0/3346763331 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0001670 con 0x7f1fe00c5140 2023-03-03T08:56:26.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe4dda700 1 -- 172.21.15.7:0/3346763331 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc004030 con 0x7f1fe00c5140 2023-03-03T08:56:26.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.633+0000 7f1fe4dda700 1 -- 172.21.15.7:0/3346763331 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0043c0 con 0x7f1fe00c5140 2023-03-03T08:56:26.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.634+0000 7f1fe8040700 1 -- 172.21.15.7:0/3346763331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe00c5140 msgr2=0x7f1fe0157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:26.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.634+0000 7f1fe8040700 1 --2- 172.21.15.7:0/3346763331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe00c5140 0x7f1fe0157ba0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009840 tx=0x7f1fdc009b50 comp rx=0 tx=0).stop 2023-03-03T08:56:26.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.634+0000 7f1fe8040700 1 -- 172.21.15.7:0/3346763331 shutdown_connections 2023-03-03T08:56:26.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.634+0000 7f1fe8040700 1 --2- 172.21.15.7:0/3346763331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe01580e0 0x7f1fe006a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:26.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.634+0000 7f1fe8040700 1 --2- 172.21.15.7:0/3346763331 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe00c5140 0x7f1fe0157ba0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:26.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.634+0000 7f1fe8040700 1 -- 172.21.15.7:0/3346763331 >> 172.21.15.7:0/3346763331 conn(0x7f1fe00a9430 msgr2=0x7f1fe00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:56:26.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.634+0000 7f1fe8040700 1 -- 172.21.15.7:0/3346763331 shutdown_connections 2023-03-03T08:56:26.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.634+0000 7f1fe8040700 1 -- 172.21.15.7:0/3346763331 wait complete. 2023-03-03T08:56:26.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.635+0000 7f1fe8040700 1 Processor -- start 2023-03-03T08:56:26.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.635+0000 7f1fe8040700 1 -- start start 2023-03-03T08:56:26.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe8040700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fe00c5140 0x7f1fe0155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:26.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe8040700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe0151de0 0x7f1fe00abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:26.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe8040700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe01580e0 0x7f1fe014d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe8040700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe014df80 con 0x7f1fe00c5140 2023-03-03T08:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe8040700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00ac0c0 con 0x7f1fe01580e0 2023-03-03T08:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe8040700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe0157030 con 0x7f1fe0151de0 2023-03-03T08:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe5ddc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fe00c5140 0x7f1fe0155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe5ddc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fe00c5140 0x7f1fe0155930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53822/0 (socket says 172.21.15.7:53822) 2023-03-03T08:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe5ddc700 1 -- 172.21.15.7:0/2117279415 learned_addr learned my addr 172.21.15.7:0/2117279415 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe55db700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe0151de0 0x7f1fe00abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:26.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe65dd700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe01580e0 0x7f1fe014d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe5ddc700 1 -- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe0151de0 msgr2=0x7f1fe00abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe5ddc700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe0151de0 0x7f1fe00abb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe5ddc700 1 -- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe01580e0 msgr2=0x7f1fe014d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe5ddc700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe01580e0 0x7f1fe014d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.636+0000 7f1fe5ddc700 1 -- 172.21.15.7:0/2117279415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd8009a70 con 0x7f1fe00c5140 2023-03-03T08:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.637+0000 7f1fe5ddc700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fe00c5140 0x7f1fe0155930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c780 tx=0x7f1fd000ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:56:26.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.637+0000 7f1fd67fc700 1 -- 172.21.15.7:0/2117279415 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0004c80 con 0x7f1fe00c5140 2023-03-03T08:56:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.637+0000 7f1fd67fc700 1 -- 172.21.15.7:0/2117279415 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd0004de0 con 0x7f1fe00c5140 2023-03-03T08:56:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.637+0000 7f1fd67fc700 1 -- 172.21.15.7:0/2117279415 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00072f0 con 0x7f1fe00c5140 2023-03-03T08:56:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.637+0000 7f1fe8040700 1 -- 172.21.15.7:0/2117279415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fdc0094a0 con 0x7f1fe00c5140 2023-03-03T08:56:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.637+0000 7f1fe8040700 1 -- 172.21.15.7:0/2117279415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe00c1c10 con 0x7f1fe00c5140 2023-03-03T08:56:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.638+0000 7f1fe8040700 1 -- 172.21.15.7:0/2117279415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe00c4c20 con 0x7f1fe00c5140 2023-03-03T08:56:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.638+0000 7f1fd67fc700 1 -- 172.21.15.7:0/2117279415 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1fd0010070 con 0x7f1fe00c5140 2023-03-03T08:56:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.639+0000 7f1fd67fc700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fc4068320 0x7f1fc406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:56:26.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.639+0000 7f1fd67fc700 1 -- 172.21.15.7:0/2117279415 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1fd0084fe0 con 0x7f1fe00c5140 2023-03-03T08:56:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.639+0000 7f1fe55db700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fc4068320 0x7f1fc406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:56:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.639+0000 7f1fe55db700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fc4068320 0x7f1fc406a7d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009840 tx=0x7f1fdc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:56:26.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.643+0000 7f1fd67fc700 1 -- 172.21.15.7:0/2117279415 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1fd0051380 con 0x7f1fe00c5140 2023-03-03T08:56:26.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.879+0000 7f1fe8040700 1 -- 172.21.15.7:0/2117279415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fe0074bb0 con 0x7f1fe00c5140 2023-03-03T08:56:26.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.880+0000 7f1fd67fc700 1 -- 172.21.15.7:0/2117279415 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1fd0018090 con 0x7f1fe00c5140 2023-03-03T08:56:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:56:26.882 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:56:26.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 -- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fc4068320 msgr2=0x7f1fc406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:26.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fc4068320 0x7f1fc406a7d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009840 tx=0x7f1fdc008030 comp rx=0 tx=0).stop 2023-03-03T08:56:26.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 -- 172.21.15.7:0/2117279415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fe00c5140 msgr2=0x7f1fe0155930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:56:26.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fe00c5140 0x7f1fe0155930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c780 tx=0x7f1fd000ca90 comp rx=0 tx=0).stop 2023-03-03T08:56:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 -- 172.21.15.7:0/2117279415 shutdown_connections 2023-03-03T08:56:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fe01580e0 0x7f1fe014d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fc4068320 0x7f1fc406a7d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fe00c5140 0x7f1fe0155930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 --2- 172.21.15.7:0/2117279415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fe0151de0 0x7f1fe00abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:56:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 -- 172.21.15.7:0/2117279415 >> 172.21.15.7:0/2117279415 conn(0x7f1fe00a9430 msgr2=0x7f1fe00c1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:56:26.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 -- 172.21.15.7:0/2117279415 shutdown_connections 2023-03-03T08:56:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:56:26.882+0000 7f1fe8040700 1 -- 172.21.15.7:0/2117279415 wait complete. 2023-03-03T08:56:26.885 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:56:26.895 INFO:tasks.mds_thrash.fs.[a]:waiting for 33 secs before thrashing 2023-03-03T08:56:38.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:38.068 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:56:59.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:57:00.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.033+0000 7f66265ad700 1 Processor -- start 2023-03-03T08:57:00.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.033+0000 7f66265ad700 1 -- start start 2023-03-03T08:57:00.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.034+0000 7f66265ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f662006d870 0x7f66201550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:00.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.034+0000 7f66265ad700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66200c3e30 0x7f66200c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:00.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.034+0000 7f66265ad700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66200c7a60 con 0x7f66200c1da0 2023-03-03T08:57:00.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.034+0000 7f66265ad700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66200c2af0 con 0x7f66200c3e30 2023-03-03T08:57:00.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.034+0000 7f66265ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66200c3620 con 0x7f662006d870 2023-03-03T08:57:00.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.034+0000 7f6624b4a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66200c3e30 0x7f66200c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:00.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.034+0000 7f6624b4a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66200c3e30 0x7f66200c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53850/0 (socket says 172.21.15.7:53850) 2023-03-03T08:57:00.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.034+0000 7f6624b4a700 1 -- 172.21.15.7:0/1761819016 learned_addr learned my addr 172.21.15.7:0/1761819016 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:57:00.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f661ffff700 1 --2- 172.21.15.7:0/1761819016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f662006d870 0x7f66201550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:00.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f6624b4a700 1 -- 172.21.15.7:0/1761819016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f662006d870 msgr2=0x7f66201550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:57:00.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f6624b4a700 1 --2- 172.21.15.7:0/1761819016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f662006d870 0x7f66201550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:00.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f6624b4a700 1 -- 172.21.15.7:0/1761819016 >> v1:172.21.15.143:6789/0 conn(0x7f66200c1da0 legacy=0x7f66200c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:57:00.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f6624b4a700 1 -- 172.21.15.7:0/1761819016 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66201545e0 con 0x7f66200c3e30 2023-03-03T08:57:00.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f6624b4a700 1 --2- 172.21.15.7:0/1761819016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c3e30 0x7f66200c4290 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6618009e50 tx=0x7f661800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6d18ce573d30ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:00.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f661effd700 1 -- 172.21.15.7:0/1761819016 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661800e070 con 0x7f66200c3e30 2023-03-03T08:57:00.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f661effd700 1 -- 172.21.15.7:0/1761819016 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6618003a20 con 0x7f66200c3e30 2023-03-03T08:57:00.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f661effd700 1 -- 172.21.15.7:0/1761819016 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6618003dd0 con 0x7f66200c3e30 2023-03-03T08:57:00.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f66265ad700 1 -- 172.21.15.7:0/1761819016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c3e30 msgr2=0x7f66200c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:00.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.035+0000 7f66265ad700 1 --2- 172.21.15.7:0/1761819016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c3e30 0x7f66200c4290 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6618009e50 tx=0x7f661800b330 comp rx=0 tx=0).stop 2023-03-03T08:57:00.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.036+0000 7f66265ad700 1 -- 172.21.15.7:0/1761819016 shutdown_connections 2023-03-03T08:57:00.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.036+0000 7f66265ad700 1 --2- 172.21.15.7:0/1761819016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c3e30 0x7f66200c4290 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:00.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.036+0000 7f66265ad700 1 --2- 172.21.15.7:0/1761819016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f662006d870 0x7f66201550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:00.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.036+0000 7f66265ad700 1 -- 172.21.15.7:0/1761819016 >> 172.21.15.7:0/1761819016 conn(0x7f662006c110 msgr2=0x7f662006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:57:00.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.036+0000 7f66265ad700 1 -- 172.21.15.7:0/1761819016 shutdown_connections 2023-03-03T08:57:00.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.036+0000 7f66265ad700 1 -- 172.21.15.7:0/1761819016 wait complete. 2023-03-03T08:57:00.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.037+0000 7f66265ad700 1 Processor -- start 2023-03-03T08:57:00.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.037+0000 7f66265ad700 1 -- start start 2023-03-03T08:57:00.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f66265ad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f662006d870 0x7f66200ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:00.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f66265ad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c1da0 0x7f66200aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:00.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f66265ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66200c3e30 0x7f6620149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f66265ad700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66200c6a80 con 0x7f66200c1da0 2023-03-03T08:57:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f66265ad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66200c5cd0 con 0x7f662006d870 2023-03-03T08:57:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f66265ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66200c64c0 con 0x7f66200c3e30 2023-03-03T08:57:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c1da0 0x7f66200aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c1da0 0x7f66200aa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53854/0 (socket says 172.21.15.7:53854) 2023-03-03T08:57:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661f7fe700 1 -- 172.21.15.7:0/3555249146 learned_addr learned my addr 172.21.15.7:0/3555249146 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:57:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661f7fe700 1 -- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66200c3e30 msgr2=0x7f6620149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f6624b4a700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66200c3e30 0x7f6620149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661ffff700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f662006d870 0x7f66200ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661f7fe700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66200c3e30 0x7f6620149750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661f7fe700 1 -- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f662006d870 msgr2=0x7f66200ac2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661f7fe700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f662006d870 0x7f66200ac2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661f7fe700 1 -- 172.21.15.7:0/3555249146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6618009b30 con 0x7f66200c1da0 2023-03-03T08:57:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661ffff700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f662006d870 0x7f66200ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:57:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661f7fe700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c1da0 0x7f66200aa6c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f661000c500 tx=0x7f661000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661cff9700 1 -- 172.21.15.7:0/3555249146 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610011790 con 0x7f66200c1da0 2023-03-03T08:57:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661cff9700 1 -- 172.21.15.7:0/3555249146 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66100118f0 con 0x7f66200c1da0 2023-03-03T08:57:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f66265ad700 1 -- 172.21.15.7:0/3555249146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6620154ba0 con 0x7f66200c1da0 2023-03-03T08:57:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f661cff9700 1 -- 172.21.15.7:0/3555249146 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610011ca0 con 0x7f66200c1da0 2023-03-03T08:57:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.038+0000 7f66265ad700 1 -- 172.21.15.7:0/3555249146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66201548c0 con 0x7f66200c1da0 2023-03-03T08:57:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.039+0000 7f66265ad700 1 -- 172.21.15.7:0/3555249146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662006dc40 con 0x7f66200c1da0 2023-03-03T08:57:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.040+0000 7f661cff9700 1 -- 172.21.15.7:0/3555249146 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6610010050 con 0x7f66200c1da0 2023-03-03T08:57:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.040+0000 7f661cff9700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66000682f0 0x7f660006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.041+0000 7f661cff9700 1 -- 172.21.15.7:0/3555249146 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6610085e30 con 0x7f66200c1da0 2023-03-03T08:57:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.041+0000 7f661ffff700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66000682f0 0x7f660006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.041+0000 7f661ffff700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66000682f0 0x7f660006a7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6608009a00 tx=0x7f6608008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:00.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.045+0000 7f661cff9700 1 -- 172.21.15.7:0/3555249146 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f66100521d0 con 0x7f66200c1da0 2023-03-03T08:57:00.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.286+0000 7f66265ad700 1 -- 172.21.15.7:0/3555249146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6620074bb0 con 0x7f66200c1da0 2023-03-03T08:57:00.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.287+0000 7f661cff9700 1 -- 172.21.15.7:0/3555249146 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f66100557f0 con 0x7f66200c1da0 2023-03-03T08:57:00.288 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:57:00.289 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:57:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 -- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66000682f0 msgr2=0x7f660006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66000682f0 0x7f660006a7a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6608009a00 tx=0x7f6608008040 comp rx=0 tx=0).stop 2023-03-03T08:57:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 -- 172.21.15.7:0/3555249146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c1da0 msgr2=0x7f66200aa6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c1da0 0x7f66200aa6c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f661000c500 tx=0x7f661000c8c0 comp rx=0 tx=0).stop 2023-03-03T08:57:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 -- 172.21.15.7:0/3555249146 shutdown_connections 2023-03-03T08:57:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66200c3e30 0x7f6620149750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f662006d870 0x7f66200ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66000682f0 0x7f660006a7a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 --2- 172.21.15.7:0/3555249146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66200c1da0 0x7f66200aa6c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 -- 172.21.15.7:0/3555249146 >> 172.21.15.7:0/3555249146 conn(0x7f662006c110 msgr2=0x7f662006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:57:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 -- 172.21.15.7:0/3555249146 shutdown_connections 2023-03-03T08:57:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:00.289+0000 7f66265ad700 1 -- 172.21.15.7:0/3555249146 wait complete. 2023-03-03T08:57:00.292 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:57:00.302 INFO:tasks.mds_thrash.fs.[a]:waiting for 32 secs before thrashing 2023-03-03T08:57:08.141 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:08.144 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:18.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:57:18.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.775+0000 7fc7c5a8d700 1 Processor -- start 2023-03-03T08:57:18.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.775+0000 7fc7c5a8d700 1 -- start start 2023-03-03T08:57:18.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.776+0000 7fc7c5a8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7c006e020 0x7fc7c0071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:18.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7c5a8d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7c0072520 0x7fc7c0151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:18.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7c5a8d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c0154910 con 0x7fc7c0155390 2023-03-03T08:57:18.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7c5a8d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7c0070b90 con 0x7fc7c0072520 2023-03-03T08:57:18.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7c5a8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c0151320 con 0x7fc7c006e020 2023-03-03T08:57:18.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7bffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7c0072520 0x7fc7c0151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:18.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7bf7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc7c0155390 0x7fc7c00c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47852/0 (socket says 172.21.15.7:47852) 2023-03-03T08:57:18.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7bf7fe700 1 -- 172.21.15.7:0/805183846 learned_addr learned my addr 172.21.15.7:0/805183846 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:57:18.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7beffd700 1 --2- 172.21.15.7:0/805183846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7c006e020 0x7fc7c0071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:18.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7bffff700 1 -- 172.21.15.7:0/805183846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7c006e020 msgr2=0x7fc7c0071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:57:18.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7bffff700 1 --2- 172.21.15.7:0/805183846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7c006e020 0x7fc7c0071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:18.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7bffff700 1 -- 172.21.15.7:0/805183846 >> v1:172.21.15.7:6789/0 conn(0x7fc7c0155390 legacy=0x7fc7c00c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:18.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7bffff700 1 -- 172.21.15.7:0/805183846 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c00c4540 con 0x7fc7c0072520 2023-03-03T08:57:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7be7fc700 1 -- 172.21.15.7:0/805183846 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016762815 0 0) 0x7fc7c0154910 con 0x7fc7c0155390 2023-03-03T08:57:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7bffff700 1 --2- 172.21.15.7:0/805183846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 0x7fc7c0151770 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009d90 tx=0x7fc7b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e42f7d62455919 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7be7fc700 1 -- 172.21.15.7:0/805183846 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b800e070 con 0x7fc7c0072520 2023-03-03T08:57:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7be7fc700 1 -- 172.21.15.7:0/805183846 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b800bbd0 con 0x7fc7c0072520 2023-03-03T08:57:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.777+0000 7fc7be7fc700 1 -- 172.21.15.7:0/805183846 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8012430 con 0x7fc7c0072520 2023-03-03T08:57:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.778+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/805183846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 msgr2=0x7fc7c0151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:18.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.778+0000 7fc7c5a8d700 1 --2- 172.21.15.7:0/805183846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 0x7fc7c0151770 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009d90 tx=0x7fc7b800b330 comp rx=0 tx=0).stop 2023-03-03T08:57:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.778+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/805183846 shutdown_connections 2023-03-03T08:57:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.778+0000 7fc7c5a8d700 1 --2- 172.21.15.7:0/805183846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 0x7fc7c0151770 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.778+0000 7fc7c5a8d700 1 --2- 172.21.15.7:0/805183846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7c006e020 0x7fc7c0071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.778+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/805183846 >> 172.21.15.7:0/805183846 conn(0x7fc7c006c110 msgr2=0x7fc7c006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:57:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.778+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/805183846 shutdown_connections 2023-03-03T08:57:18.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.778+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/805183846 wait complete. 2023-03-03T08:57:18.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.779+0000 7fc7c5a8d700 1 Processor -- start 2023-03-03T08:57:18.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7c5a8d700 1 -- start start 2023-03-03T08:57:18.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7c5a8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7c006e020 0x7fc7c014c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:18.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7c5a8d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 0x7fc7c01ed990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:18.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7c5a8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7c0155390 0x7fc7c00c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:18.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7c5a8d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c01e99d0 con 0x7fc7c0072520 2023-03-03T08:57:18.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7c5a8d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c01e9c90 con 0x7fc7c0155390 2023-03-03T08:57:18.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7c5a8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c00c22c0 con 0x7fc7c006e020 2023-03-03T08:57:18.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7beffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 0x7fc7c01ed990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:18.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7beffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 0x7fc7c01ed990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53880/0 (socket says 172.21.15.7:53880) 2023-03-03T08:57:18.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7beffd700 1 -- 172.21.15.7:0/3971629452 learned_addr learned my addr 172.21.15.7:0/3971629452 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:57:18.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7bf7fe700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7c006e020 0x7fc7c014c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:18.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.780+0000 7fc7bffff700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7c0155390 0x7fc7c00c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:18.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7beffd700 1 -- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7c006e020 msgr2=0x7fc7c014c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:18.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7beffd700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7c006e020 0x7fc7c014c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:18.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7beffd700 1 -- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7c0155390 msgr2=0x7fc7c00c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7beffd700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7c0155390 0x7fc7c00c5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7beffd700 1 -- 172.21.15.7:0/3971629452 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b8009a70 con 0x7fc7c0072520 2023-03-03T08:57:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7beffd700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 0x7fc7c01ed990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800d800 tx=0x7fc7a800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7b7fff700 1 -- 172.21.15.7:0/3971629452 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80125f0 con 0x7fc7c0072520 2023-03-03T08:57:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7b7fff700 1 -- 172.21.15.7:0/3971629452 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a8012750 con 0x7fc7c0072520 2023-03-03T08:57:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/3971629452 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c0070540 con 0x7fc7c0072520 2023-03-03T08:57:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7b7fff700 1 -- 172.21.15.7:0/3971629452 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8012b00 con 0x7fc7c0072520 2023-03-03T08:57:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.781+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/3971629452 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c00702a0 con 0x7fc7c0072520 2023-03-03T08:57:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.782+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/3971629452 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c0070820 con 0x7fc7c0072520 2023-03-03T08:57:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.783+0000 7fc7b7fff700 1 -- 172.21.15.7:0/3971629452 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc7a8012c60 con 0x7fc7c0072520 2023-03-03T08:57:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.783+0000 7fc7b7fff700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7a00682f0 0x7fc7a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:18.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.783+0000 7fc7b7fff700 1 -- 172.21.15.7:0/3971629452 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc7a8086320 con 0x7fc7c0072520 2023-03-03T08:57:18.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.783+0000 7fc7bf7fe700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7a00682f0 0x7fc7a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:18.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.784+0000 7fc7bf7fe700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7a00682f0 0x7fc7a006a7a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0007fd0 tx=0x7fc7b0007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:18.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:18.787+0000 7fc7b7fff700 1 -- 172.21.15.7:0/3971629452 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc7a80526c0 con 0x7fc7c0072520 2023-03-03T08:57:19.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.024+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/3971629452 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7c01e9e00 con 0x7fc7c0072520 2023-03-03T08:57:19.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.025+0000 7fc7b7fff700 1 -- 172.21.15.7:0/3971629452 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc7a8055ce0 con 0x7fc7c0072520 2023-03-03T08:57:19.028 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:57:19.028 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:57:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7a00682f0 msgr2=0x7fc7a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7a00682f0 0x7fc7a006a7a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0007fd0 tx=0x7fc7b0007de0 comp rx=0 tx=0).stop 2023-03-03T08:57:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/3971629452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 msgr2=0x7fc7c01ed990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 0x7fc7c01ed990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800d800 tx=0x7fc7a800db10 comp rx=0 tx=0).stop 2023-03-03T08:57:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/3971629452 shutdown_connections 2023-03-03T08:57:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7c0155390 0x7fc7c00c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7c006e020 0x7fc7c014c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7a00682f0 0x7fc7a006a7a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 --2- 172.21.15.7:0/3971629452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7c0072520 0x7fc7c01ed990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/3971629452 >> 172.21.15.7:0/3971629452 conn(0x7fc7c006c110 msgr2=0x7fc7c00c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:57:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/3971629452 shutdown_connections 2023-03-03T08:57:19.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:19.027+0000 7fc7c5a8d700 1 -- 172.21.15.7:0/3971629452 wait complete. 2023-03-03T08:57:19.032 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:57:19.041 INFO:tasks.mds_thrash.fs.[b]:waiting for 31 secs before thrashing 2023-03-03T08:57:32.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:57:32.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.441+0000 7f1e4b236700 1 Processor -- start 2023-03-03T08:57:32.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.441+0000 7f1e4b236700 1 -- start start 2023-03-03T08:57:32.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e4b236700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1e4414e560 0x7f1e44071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:32.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e4b236700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e440c43d0 0x7f1e440c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:32.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e4b236700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e44072920 con 0x7f1e44154590 2023-03-03T08:57:32.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e4b236700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4406bef0 con 0x7f1e4414e560 2023-03-03T08:57:32.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e4b236700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4406b910 con 0x7f1e440c43d0 2023-03-03T08:57:32.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e48fd2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1e4414e560 0x7f1e44071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:32.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e48fd2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1e4414e560 0x7f1e44071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53892/0 (socket says 172.21.15.7:53892) 2023-03-03T08:57:32.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e48fd2700 1 -- 172.21.15.7:0/4023039672 learned_addr learned my addr 172.21.15.7:0/4023039672 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:57:32.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e43fff700 1 --2- 172.21.15.7:0/4023039672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e440c43d0 0x7f1e440c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:32.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e48fd2700 1 -- 172.21.15.7:0/4023039672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e440c43d0 msgr2=0x7f1e440c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:57:32.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e48fd2700 1 --2- 172.21.15.7:0/4023039672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e440c43d0 0x7f1e440c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:32.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e48fd2700 1 -- 172.21.15.7:0/4023039672 >> v1:172.21.15.143:6789/0 conn(0x7f1e44154590 legacy=0x7f1e441549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:32.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e48fd2700 1 -- 172.21.15.7:0/4023039672 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e440aacc0 con 0x7f1e4414e560 2023-03-03T08:57:32.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e48fd2700 1 --2- 172.21.15.7:0/4023039672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 0x7f1e44071c70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1e380098f0 tx=0x7f1e38009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c04d42591f52361 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:32.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.442+0000 7f1e437fe700 1 -- 172.21.15.7:0/4023039672 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800b8a0 con 0x7f1e4414e560 2023-03-03T08:57:32.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.443+0000 7f1e437fe700 1 -- 172.21.15.7:0/4023039672 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3800ba00 con 0x7f1e4414e560 2023-03-03T08:57:32.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.443+0000 7f1e437fe700 1 -- 172.21.15.7:0/4023039672 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800bd90 con 0x7f1e4414e560 2023-03-03T08:57:32.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.443+0000 7f1e4b236700 1 -- 172.21.15.7:0/4023039672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 msgr2=0x7f1e44071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:32.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.443+0000 7f1e4b236700 1 --2- 172.21.15.7:0/4023039672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 0x7f1e44071c70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1e380098f0 tx=0x7f1e38009c00 comp rx=0 tx=0).stop 2023-03-03T08:57:32.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.443+0000 7f1e4b236700 1 -- 172.21.15.7:0/4023039672 shutdown_connections 2023-03-03T08:57:32.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.443+0000 7f1e4b236700 1 --2- 172.21.15.7:0/4023039672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 0x7f1e44071c70 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:32.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.443+0000 7f1e4b236700 1 --2- 172.21.15.7:0/4023039672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e440c43d0 0x7f1e440c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:32.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.443+0000 7f1e4b236700 1 -- 172.21.15.7:0/4023039672 >> 172.21.15.7:0/4023039672 conn(0x7f1e440736f0 msgr2=0x7f1e44073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:57:32.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.443+0000 7f1e4b236700 1 -- 172.21.15.7:0/4023039672 shutdown_connections 2023-03-03T08:57:32.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.443+0000 7f1e4b236700 1 -- 172.21.15.7:0/4023039672 wait complete. 2023-03-03T08:57:32.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.445+0000 7f1e4b236700 1 Processor -- start 2023-03-03T08:57:32.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.445+0000 7f1e4b236700 1 -- start start 2023-03-03T08:57:32.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e4b236700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e440c43d0 0x7f1e440add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:32.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e4b236700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 0x7f1e440abde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:32.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e4b236700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e44154590 0x7f1e4414d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:32.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e4b236700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4414cd30 con 0x7f1e4414e560 2023-03-03T08:57:32.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e4b236700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4414c120 con 0x7f1e440c43d0 2023-03-03T08:57:32.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e4b236700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4414c400 con 0x7f1e44154590 2023-03-03T08:57:32.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e43fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 0x7f1e440abde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:32.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e43fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 0x7f1e440abde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53900/0 (socket says 172.21.15.7:53900) 2023-03-03T08:57:32.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e43fff700 1 -- 172.21.15.7:0/2275239235 learned_addr learned my addr 172.21.15.7:0/2275239235 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:57:32.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e48fd2700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e440c43d0 0x7f1e440add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:32.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e497d3700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e44154590 0x7f1e4414d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:32.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e43fff700 1 -- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e44154590 msgr2=0x7f1e4414d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:32.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e43fff700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e44154590 0x7f1e4414d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:32.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e43fff700 1 -- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e440c43d0 msgr2=0x7f1e440add70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:32.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e43fff700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e440c43d0 0x7f1e440add70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:32.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e43fff700 1 -- 172.21.15.7:0/2275239235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e38009550 con 0x7f1e4414e560 2023-03-03T08:57:32.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.446+0000 7f1e43fff700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 0x7f1e440abde0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000d800 tx=0x7f1e3000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:32.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.447+0000 7f1e417fa700 1 -- 172.21.15.7:0/2275239235 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e300125f0 con 0x7f1e4414e560 2023-03-03T08:57:32.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.447+0000 7f1e417fa700 1 -- 172.21.15.7:0/2275239235 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e30012750 con 0x7f1e4414e560 2023-03-03T08:57:32.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.447+0000 7f1e4b236700 1 -- 172.21.15.7:0/2275239235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e44155100 con 0x7f1e4414e560 2023-03-03T08:57:32.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.447+0000 7f1e417fa700 1 -- 172.21.15.7:0/2275239235 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30012b00 con 0x7f1e4414e560 2023-03-03T08:57:32.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.447+0000 7f1e4b236700 1 -- 172.21.15.7:0/2275239235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4406bb80 con 0x7f1e4414e560 2023-03-03T08:57:32.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.448+0000 7f1e4b236700 1 -- 172.21.15.7:0/2275239235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e44155870 con 0x7f1e4414e560 2023-03-03T08:57:32.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.448+0000 7f1e417fa700 1 -- 172.21.15.7:0/2275239235 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1e30012c60 con 0x7f1e4414e560 2023-03-03T08:57:32.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.449+0000 7f1e417fa700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e280682f0 0x7f1e2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:32.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.449+0000 7f1e417fa700 1 -- 172.21.15.7:0/2275239235 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1e30086570 con 0x7f1e4414e560 2023-03-03T08:57:32.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.449+0000 7f1e48fd2700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e280682f0 0x7f1e2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:32.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.449+0000 7f1e48fd2700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e280682f0 0x7f1e2806a7a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009c40 tx=0x7f1e38008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:32.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.453+0000 7f1e417fa700 1 -- 172.21.15.7:0/2275239235 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1e30052910 con 0x7f1e4414e560 2023-03-03T08:57:32.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.691+0000 7f1e4b236700 1 -- 172.21.15.7:0/2275239235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e44074bb0 con 0x7f1e4414e560 2023-03-03T08:57:32.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.692+0000 7f1e417fa700 1 -- 172.21.15.7:0/2275239235 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1e30055f30 con 0x7f1e4414e560 2023-03-03T08:57:32.694 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:57:32.694 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:57:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 -- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e280682f0 msgr2=0x7f1e2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e280682f0 0x7f1e2806a7a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1e38009c40 tx=0x7f1e38008030 comp rx=0 tx=0).stop 2023-03-03T08:57:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 -- 172.21.15.7:0/2275239235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 msgr2=0x7f1e440abde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 0x7f1e440abde0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000d800 tx=0x7f1e3000db10 comp rx=0 tx=0).stop 2023-03-03T08:57:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 -- 172.21.15.7:0/2275239235 shutdown_connections 2023-03-03T08:57:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e44154590 0x7f1e4414d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e440c43d0 0x7f1e440add70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e280682f0 0x7f1e2806a7a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 --2- 172.21.15.7:0/2275239235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e4414e560 0x7f1e440abde0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 -- 172.21.15.7:0/2275239235 >> 172.21.15.7:0/2275239235 conn(0x7f1e440736f0 msgr2=0x7f1e44073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:57:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 -- 172.21.15.7:0/2275239235 shutdown_connections 2023-03-03T08:57:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:32.694+0000 7f1e4b236700 1 -- 172.21.15.7:0/2275239235 wait complete. 2023-03-03T08:57:32.698 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:57:32.708 INFO:tasks.mds_thrash.fs.[a]:waiting for 60 secs before thrashing 2023-03-03T08:57:38.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:38.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:57:50.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:57:50.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.180+0000 7fcfcfeed700 1 Processor -- start 2023-03-03T08:57:50.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.180+0000 7fcfcfeed700 1 -- start start 2023-03-03T08:57:50.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfcfeed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc814e560 0x7fcfc8071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:50.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfcfeed700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcfc80c4840 0x7fcfc8154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:50.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfcfeed700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc8072920 con 0x7fcfc80c43d0 2023-03-03T08:57:50.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfcfeed700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc806bef0 con 0x7fcfc80c4840 2023-03-03T08:57:50.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfcfeed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc81551e0 con 0x7fcfc814e560 2023-03-03T08:57:50.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfce48a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcfc80c4840 0x7fcfc8154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:50.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfcd488700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcfc80c43d0 0x7fcfc8154590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47896/0 (socket says 172.21.15.7:47896) 2023-03-03T08:57:50.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfcd488700 1 -- 172.21.15.7:0/1858751310 learned_addr learned my addr 172.21.15.7:0/1858751310 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:57:50.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfce48a700 1 -- 172.21.15.7:0/1858751310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc814e560 msgr2=0x7fcfc8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:57:50.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfcdc89700 1 --2- 172.21.15.7:0/1858751310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc814e560 0x7fcfc8071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:50.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfccc87700 1 -- 172.21.15.7:0/1858751310 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184217303 0 0) 0x7fcfc4001110 con 0x7fcfc80c43d0 2023-03-03T08:57:50.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfce48a700 1 --2- 172.21.15.7:0/1858751310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc814e560 0x7fcfc8071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:50.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfce48a700 1 -- 172.21.15.7:0/1858751310 >> v1:172.21.15.7:6789/0 conn(0x7fcfc80c43d0 legacy=0x7fcfc8154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:50.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.181+0000 7fcfce48a700 1 -- 172.21.15.7:0/1858751310 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc806b910 con 0x7fcfc80c4840 2023-03-03T08:57:50.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfce48a700 1 --2- 172.21.15.7:0/1858751310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc80c4840 0x7fcfc8154ca0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009d90 tx=0x7fcfc000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f5f5665dac79a81 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:50.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfccc87700 1 -- 172.21.15.7:0/1858751310 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc000e070 con 0x7fcfc80c4840 2023-03-03T08:57:50.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfccc87700 1 -- 172.21.15.7:0/1858751310 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc000bbd0 con 0x7fcfc80c4840 2023-03-03T08:57:50.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfccc87700 1 -- 172.21.15.7:0/1858751310 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0012430 con 0x7fcfc80c4840 2023-03-03T08:57:50.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfcfeed700 1 -- 172.21.15.7:0/1858751310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc80c4840 msgr2=0x7fcfc8154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:50.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfcfeed700 1 --2- 172.21.15.7:0/1858751310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc80c4840 0x7fcfc8154ca0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009d90 tx=0x7fcfc000b330 comp rx=0 tx=0).stop 2023-03-03T08:57:50.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfcfeed700 1 -- 172.21.15.7:0/1858751310 shutdown_connections 2023-03-03T08:57:50.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfcfeed700 1 --2- 172.21.15.7:0/1858751310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc80c4840 0x7fcfc8154ca0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:50.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfcfeed700 1 --2- 172.21.15.7:0/1858751310 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc814e560 0x7fcfc8071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:50.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfcfeed700 1 -- 172.21.15.7:0/1858751310 >> 172.21.15.7:0/1858751310 conn(0x7fcfc80736f0 msgr2=0x7fcfc8073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:57:50.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfcfeed700 1 -- 172.21.15.7:0/1858751310 shutdown_connections 2023-03-03T08:57:50.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.182+0000 7fcfcfeed700 1 -- 172.21.15.7:0/1858751310 wait complete. 2023-03-03T08:57:50.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.184+0000 7fcfcfeed700 1 Processor -- start 2023-03-03T08:57:50.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.184+0000 7fcfcfeed700 1 -- start start 2023-03-03T08:57:50.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.184+0000 7fcfcfeed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc80c43d0 0x7fcfc814a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:50.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.184+0000 7fcfcfeed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc80c4840 0x7fcfc814e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:50.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.184+0000 7fcfcfeed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc814e560 0x7fcfc814ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:50.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.184+0000 7fcfcfeed700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc81eda20 con 0x7fcfc814e560 2023-03-03T08:57:50.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.184+0000 7fcfcfeed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc8151410 con 0x7fcfc80c4840 2023-03-03T08:57:50.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.184+0000 7fcfcfeed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc81518a0 con 0x7fcfc80c43d0 2023-03-03T08:57:50.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfce48a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc814e560 0x7fcfc814ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:50.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfcdc89700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc80c43d0 0x7fcfc814a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:50.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfce48a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc814e560 0x7fcfc814ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53926/0 (socket says 172.21.15.7:53926) 2023-03-03T08:57:50.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfce48a700 1 -- 172.21.15.7:0/4212889273 learned_addr learned my addr 172.21.15.7:0/4212889273 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:57:50.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfcd488700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc80c4840 0x7fcfc814e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:50.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfce48a700 1 -- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc80c43d0 msgr2=0x7fcfc814a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:50.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfce48a700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc80c43d0 0x7fcfc814a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:50.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfce48a700 1 -- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc80c4840 msgr2=0x7fcfc814e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:50.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfce48a700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc80c4840 0x7fcfc814e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:50.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfce48a700 1 -- 172.21.15.7:0/4212889273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc0009a70 con 0x7fcfc814e560 2023-03-03T08:57:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfcd488700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc80c4840 0x7fcfc814e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:57:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfce48a700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc814e560 0x7fcfc814ee70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000b730 tx=0x7fcfc0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfbe7fc700 1 -- 172.21.15.7:0/4212889273 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0012430 con 0x7fcfc814e560 2023-03-03T08:57:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfbe7fc700 1 -- 172.21.15.7:0/4212889273 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc0020070 con 0x7fcfc814e560 2023-03-03T08:57:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfcfeed700 1 -- 172.21.15.7:0/4212889273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc80aacc0 con 0x7fcfc814e560 2023-03-03T08:57:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfbe7fc700 1 -- 172.21.15.7:0/4212889273 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0003ea0 con 0x7fcfc814e560 2023-03-03T08:57:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.185+0000 7fcfcfeed700 1 -- 172.21.15.7:0/4212889273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc80c4ca0 con 0x7fcfc814e560 2023-03-03T08:57:50.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.186+0000 7fcfcfeed700 1 -- 172.21.15.7:0/4212889273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc806bb80 con 0x7fcfc814e560 2023-03-03T08:57:50.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.187+0000 7fcfbe7fc700 1 -- 172.21.15.7:0/4212889273 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcfc002e070 con 0x7fcfc814e560 2023-03-03T08:57:50.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.187+0000 7fcfbe7fc700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 0x7fcfac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:57:50.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.187+0000 7fcfbe7fc700 1 -- 172.21.15.7:0/4212889273 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcfc0022070 con 0x7fcfc814e560 2023-03-03T08:57:50.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.187+0000 7fcfcdc89700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 0x7fcfac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:57:50.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.188+0000 7fcfcdc89700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 0x7fcfac06a7a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8006fd0 tx=0x7fcfb8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:57:50.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.192+0000 7fcfbe7fc700 1 -- 172.21.15.7:0/4212889273 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcfc0054320 con 0x7fcfc814e560 2023-03-03T08:57:50.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.430+0000 7fcfcfeed700 1 -- 172.21.15.7:0/4212889273 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfc81515c0 con 0x7fcfc814e560 2023-03-03T08:57:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.431+0000 7fcfbe7fc700 1 -- 172.21.15.7:0/4212889273 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcfc81515c0 con 0x7fcfc814e560 2023-03-03T08:57:50.432 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:57:50.433 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:57:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 -- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 msgr2=0x7fcfac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 0x7fcfac06a7a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8006fd0 tx=0x7fcfb8008040 comp rx=0 tx=0).stop 2023-03-03T08:57:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 -- 172.21.15.7:0/4212889273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc814e560 msgr2=0x7fcfc814ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:57:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc814e560 0x7fcfc814ee70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000b730 tx=0x7fcfc0004ad0 comp rx=0 tx=0).stop 2023-03-03T08:57:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 -- 172.21.15.7:0/4212889273 shutdown_connections 2023-03-03T08:57:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 0x7fcfac06a7a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc814e560 0x7fcfc814ee70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc80c43d0 0x7fcfc814a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 --2- 172.21.15.7:0/4212889273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc80c4840 0x7fcfc814e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:57:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 -- 172.21.15.7:0/4212889273 >> 172.21.15.7:0/4212889273 conn(0x7fcfc80736f0 msgr2=0x7fcfc806aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:57:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 -- 172.21.15.7:0/4212889273 shutdown_connections 2023-03-03T08:57:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:57:50.433+0000 7fcfcfeed700 1 -- 172.21.15.7:0/4212889273 wait complete. 2023-03-03T08:57:50.436 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:57:50.446 INFO:tasks.mds_thrash.fs.[b]:waiting for 12 secs before thrashing 2023-03-03T08:58:02.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:58:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.585+0000 7f11de4b0700 1 Processor -- start 2023-03-03T08:58:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.585+0000 7f11de4b0700 1 -- start start 2023-03-03T08:58:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11de4b0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11d8151de0 0x7f11d80c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11de4b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d80c1a40 0x7f11d8157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11de4b0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d806dad0 con 0x7f11d80c5140 2023-03-03T08:58:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11de4b0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d806e370 con 0x7f11d8151de0 2023-03-03T08:58:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11de4b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d806ddb0 con 0x7f11d80c1a40 2023-03-03T08:58:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11d7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11d8151de0 0x7f11d80c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11dca4d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f11d80c5140 0x7f11d806a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47920/0 (socket says 172.21.15.7:47920) 2023-03-03T08:58:02.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11d7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11d8151de0 0x7f11d80c1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53938/0 (socket says 172.21.15.7:53938) 2023-03-03T08:58:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11dca4d700 1 -- 172.21.15.7:0/1285673636 learned_addr learned my addr 172.21.15.7:0/1285673636 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:58:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11d77fe700 1 --2- 172.21.15.7:0/1285673636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d80c1a40 0x7f11d8157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11d7fff700 1 -- 172.21.15.7:0/1285673636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d80c1a40 msgr2=0x7f11d8157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:58:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11d7fff700 1 --2- 172.21.15.7:0/1285673636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d80c1a40 0x7f11d8157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11d7fff700 1 -- 172.21.15.7:0/1285673636 >> v1:172.21.15.7:6789/0 conn(0x7f11d80c5140 legacy=0x7f11d806a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11d7fff700 1 -- 172.21.15.7:0/1285673636 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d806e090 con 0x7f11d8151de0 2023-03-03T08:58:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.586+0000 7f11d6ffd700 1 -- 172.21.15.7:0/1285673636 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766670126 0 0) 0x7f11d806dad0 con 0x7f11d80c5140 2023-03-03T08:58:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11d77fe700 1 --2- 172.21.15.7:0/1285673636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d80c1a40 0x7f11d8157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T08:58:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11d7fff700 1 --2- 172.21.15.7:0/1285673636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8151de0 0x7f11d80c1500 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009840 tx=0x7f11c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68b11394e06e417 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11d6ffd700 1 -- 172.21.15.7:0/1285673636 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8011830 con 0x7f11d8151de0 2023-03-03T08:58:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11d6ffd700 1 -- 172.21.15.7:0/1285673636 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c8011990 con 0x7f11d8151de0 2023-03-03T08:58:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11d6ffd700 1 -- 172.21.15.7:0/1285673636 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8011d20 con 0x7f11d8151de0 2023-03-03T08:58:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11de4b0700 1 -- 172.21.15.7:0/1285673636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8151de0 msgr2=0x7f11d80c1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11de4b0700 1 --2- 172.21.15.7:0/1285673636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8151de0 0x7f11d80c1500 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009840 tx=0x7f11c8009b50 comp rx=0 tx=0).stop 2023-03-03T08:58:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11de4b0700 1 -- 172.21.15.7:0/1285673636 shutdown_connections 2023-03-03T08:58:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11de4b0700 1 --2- 172.21.15.7:0/1285673636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d8151de0 0x7f11d80c1500 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11de4b0700 1 --2- 172.21.15.7:0/1285673636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d80c1a40 0x7f11d8157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11de4b0700 1 -- 172.21.15.7:0/1285673636 >> 172.21.15.7:0/1285673636 conn(0x7f11d80a9430 msgr2=0x7f11d80c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:58:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11de4b0700 1 -- 172.21.15.7:0/1285673636 shutdown_connections 2023-03-03T08:58:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.587+0000 7f11de4b0700 1 -- 172.21.15.7:0/1285673636 wait complete. 2023-03-03T08:58:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.589+0000 7f11de4b0700 1 Processor -- start 2023-03-03T08:58:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.589+0000 7f11de4b0700 1 -- start start 2023-03-03T08:58:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11de4b0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11d80c1a40 0x7f11d80aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11de4b0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d80c5140 0x7f11d81483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11de4b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d8151de0 0x7f11d814da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11de4b0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d814d610 con 0x7f11d80c5140 2023-03-03T08:58:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11de4b0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d814f130 con 0x7f11d80c1a40 2023-03-03T08:58:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11de4b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d814cce0 con 0x7f11d8151de0 2023-03-03T08:58:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11d77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d80c5140 0x7f11d81483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11d77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d80c5140 0x7f11d81483f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53946/0 (socket says 172.21.15.7:53946) 2023-03-03T08:58:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11d77fe700 1 -- 172.21.15.7:0/4161192940 learned_addr learned my addr 172.21.15.7:0/4161192940 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:58:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11d7fff700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11d80c1a40 0x7f11d80aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.590+0000 7f11dca4d700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d8151de0 0x7f11d814da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11d77fe700 1 -- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d8151de0 msgr2=0x7f11d814da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11d77fe700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d8151de0 0x7f11d814da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11d77fe700 1 -- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11d80c1a40 msgr2=0x7f11d80aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11d77fe700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11d80c1a40 0x7f11d80aaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11d77fe700 1 -- 172.21.15.7:0/4161192940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80094a0 con 0x7f11d80c5140 2023-03-03T08:58:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11d77fe700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d80c5140 0x7f11d81483f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00c820 tx=0x7f11cc00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11d4ff9700 1 -- 172.21.15.7:0/4161192940 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc011800 con 0x7f11d80c5140 2023-03-03T08:58:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11d4ff9700 1 -- 172.21.15.7:0/4161192940 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11cc011960 con 0x7f11d80c5140 2023-03-03T08:58:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11de4b0700 1 -- 172.21.15.7:0/4161192940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d8066de0 con 0x7f11d80c5140 2023-03-03T08:58:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11d4ff9700 1 -- 172.21.15.7:0/4161192940 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc011cf0 con 0x7f11d80c5140 2023-03-03T08:58:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.591+0000 7f11de4b0700 1 -- 172.21.15.7:0/4161192940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d80c4c20 con 0x7f11d80c5140 2023-03-03T08:58:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.592+0000 7f11de4b0700 1 -- 172.21.15.7:0/4161192940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d806a9c0 con 0x7f11d80c5140 2023-03-03T08:58:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.592+0000 7f11d4ff9700 1 -- 172.21.15.7:0/4161192940 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f11cc017030 con 0x7f11d80c5140 2023-03-03T08:58:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.593+0000 7f11d4ff9700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c40682f0 0x7f11c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.593+0000 7f11d4ff9700 1 -- 172.21.15.7:0/4161192940 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f11cc086220 con 0x7f11d80c5140 2023-03-03T08:58:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.593+0000 7f11d7fff700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c40682f0 0x7f11c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.593+0000 7f11d7fff700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c40682f0 0x7f11c406a7a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f11c80090b0 tx=0x7f11c8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:02.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.597+0000 7f11d4ff9700 1 -- 172.21.15.7:0/4161192940 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11cc052540 con 0x7f11d80c5140 2023-03-03T08:58:02.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.837+0000 7f11de4b0700 1 -- 172.21.15.7:0/4161192940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11d814d880 con 0x7f11d80c5140 2023-03-03T08:58:02.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.837+0000 7f11d4ff9700 1 -- 172.21.15.7:0/4161192940 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f11cc055b60 con 0x7f11d80c5140 2023-03-03T08:58:02.839 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:58:02.839 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:58:02.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 -- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c40682f0 msgr2=0x7f11c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:02.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c40682f0 0x7f11c406a7a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f11c80090b0 tx=0x7f11c8008030 comp rx=0 tx=0).stop 2023-03-03T08:58:02.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 -- 172.21.15.7:0/4161192940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d80c5140 msgr2=0x7f11d81483f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:02.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d80c5140 0x7f11d81483f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f11cc00c820 tx=0x7f11cc00cb30 comp rx=0 tx=0).stop 2023-03-03T08:58:02.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 -- 172.21.15.7:0/4161192940 shutdown_connections 2023-03-03T08:58:02.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11d8151de0 0x7f11d814da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:02.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11d80c1a40 0x7f11d80aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:02.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c40682f0 0x7f11c406a7a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:02.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 --2- 172.21.15.7:0/4161192940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11d80c5140 0x7f11d81483f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:02.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 -- 172.21.15.7:0/4161192940 >> 172.21.15.7:0/4161192940 conn(0x7f11d80a9430 msgr2=0x7f11d8149150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:58:02.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 -- 172.21.15.7:0/4161192940 shutdown_connections 2023-03-03T08:58:02.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:02.840+0000 7f11de4b0700 1 -- 172.21.15.7:0/4161192940 wait complete. 2023-03-03T08:58:02.843 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:58:02.853 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before thrashing 2023-03-03T08:58:07.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:58:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.990+0000 7f3116bbf700 1 Processor -- start 2023-03-03T08:58:07.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.990+0000 7f3116bbf700 1 -- start start 2023-03-03T08:58:07.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.991+0000 7f3116bbf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f311014f9d0 0x7f3110156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:07.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.991+0000 7f3116bbf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31100c2f30 con 0x7f311014fda0 2023-03-03T08:58:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.991+0000 7f3116bbf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31100c31d0 con 0x7f31100abe20 2023-03-03T08:58:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.991+0000 7f3116bbf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31101524a0 con 0x7f311014f9d0 2023-03-03T08:58:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.991+0000 7f310ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f311014fda0 0x7f3110071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47942/0 (socket says 172.21.15.7:47942) 2023-03-03T08:58:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.991+0000 7f310ffff700 1 -- 172.21.15.7:0/861449349 learned_addr learned my addr 172.21.15.7:0/861449349 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:58:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f311495b700 1 --2- 172.21.15.7:0/861449349 >> v2:172.21.15.143:3300/0 conn(0x7f311014f9d0 0x7f3110156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803495773 0 0) 0x7f31100c2f30 con 0x7f311014fda0 2023-03-03T08:58:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31000035b0 con 0x7f311014fda0 2023-03-03T08:58:07.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2698026434 0 0) 0x7f31000035b0 con 0x7f311014fda0 2023-03-03T08:58:07.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 >> v2:172.21.15.143:3300/0 conn(0x7f311014f9d0 msgr2=0x7f3110156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:58:07.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 --2- 172.21.15.7:0/861449349 >> v2:172.21.15.143:3300/0 conn(0x7f311014f9d0 0x7f3110156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:07.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 >> v1:172.21.15.143:6789/0 conn(0x7f31100abe20 legacy=0x7f31101529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:07.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100c34b0 con 0x7f311014fda0 2023-03-03T08:58:07.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f30fc001670 con 0x7f311014fda0 2023-03-03T08:58:07.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 >> v1:172.21.15.7:6789/0 conn(0x7f311014fda0 legacy=0x7f3110071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:07.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3100003a70 0x7f3100003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3100004850 0x7f3100004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3100005320 0x7f310000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31101524a0 con 0x7f3100003a70 2023-03-03T08:58:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310000c230 con 0x7f3100005320 2023-03-03T08:58:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310000c3f0 con 0x7f3100004850 2023-03-03T08:58:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f311495b700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3100003a70 0x7f3100003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:07.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f311515c700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3100005320 0x7f310000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.992+0000 7f310ffff700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3100004850 0x7f3100004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.993+0000 7f311495b700 1 -- 172.21.15.7:0/861449349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3100004850 msgr2=0x7f3100004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:58:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.993+0000 7f311495b700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3100004850 0x7f3100004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.993+0000 7f311495b700 1 -- 172.21.15.7:0/861449349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3100005320 msgr2=0x7f310000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:58:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.993+0000 7f311495b700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3100005320 0x7f310000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.993+0000 7f311495b700 1 -- 172.21.15.7:0/861449349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31000035b0 con 0x7f3100003a70 2023-03-03T08:58:07.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.993+0000 7f311515c700 1 -- 172.21.15.7:0/861449349 reap_dead start 2023-03-03T08:58:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.993+0000 7f311495b700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3100003a70 0x7f3100003e80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3104009b60 tx=0x7f3100005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e1f3d19b1860b0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.993+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400e6c0 con 0x7f3100003a70 2023-03-03T08:58:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.993+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310400e820 con 0x7f3100003a70 2023-03-03T08:58:07.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.993+0000 7f310f7fe700 1 -- 172.21.15.7:0/861449349 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400eb90 con 0x7f3100003a70 2023-03-03T08:58:08.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.994+0000 7f3116bbf700 1 -- 172.21.15.7:0/861449349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3100003a70 msgr2=0x7f3100003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:08.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.994+0000 7f3116bbf700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3100003a70 0x7f3100003e80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3104009b60 tx=0x7f3100005880 comp rx=0 tx=0).stop 2023-03-03T08:58:08.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.994+0000 7f3116bbf700 1 -- 172.21.15.7:0/861449349 shutdown_connections 2023-03-03T08:58:08.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.994+0000 7f3116bbf700 1 --2- 172.21.15.7:0/861449349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3100003a70 0x7f3100003e80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:08.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.994+0000 7f3116bbf700 1 -- 172.21.15.7:0/861449349 >> 172.21.15.7:0/861449349 conn(0x7f31100c8900 msgr2=0x7f31100c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:58:08.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.994+0000 7f3116bbf700 1 -- 172.21.15.7:0/861449349 shutdown_connections 2023-03-03T08:58:08.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.994+0000 7f3116bbf700 1 -- 172.21.15.7:0/861449349 wait complete. 2023-03-03T08:58:08.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.995+0000 7f3116bbf700 1 Processor -- start 2023-03-03T08:58:08.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.995+0000 7f3116bbf700 1 -- start start 2023-03-03T08:58:08.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f3116bbf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3100003a70 0x7f31101ded00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:08.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f3116bbf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3110155210 0x7f3110146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:08.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f3116bbf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3110147140 0x7f31101ecbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:08.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f3116bbf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311014a3a0 con 0x7f3110155210 2023-03-03T08:58:08.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f3116bbf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31101475c0 con 0x7f3100003a70 2023-03-03T08:58:08.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f3116bbf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31101e99d0 con 0x7f3110147140 2023-03-03T08:58:08.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3110155210 0x7f3110146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3110155210 0x7f3110146c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53976/0 (socket says 172.21.15.7:53976) 2023-03-03T08:58:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310ffff700 1 -- 172.21.15.7:0/2994957975 learned_addr learned my addr 172.21.15.7:0/2994957975 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:58:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310ffff700 1 -- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3110147140 msgr2=0x7f31101ecbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f311495b700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3100003a70 0x7f31101ded00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f311515c700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3110147140 0x7f31101ecbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310ffff700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3110147140 0x7f31101ecbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310ffff700 1 -- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3100003a70 msgr2=0x7f31101ded00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:08.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310ffff700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3100003a70 0x7f31101ded00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:08.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310ffff700 1 -- 172.21.15.7:0/2994957975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100ac350 con 0x7f3110155210 2023-03-03T08:58:08.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310ffff700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3110155210 0x7f3110146c00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f30fc007290 tx=0x7f30fc0075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:08.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310d7fa700 1 -- 172.21.15.7:0/2994957975 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc011580 con 0x7f3110155210 2023-03-03T08:58:08.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310d7fa700 1 -- 172.21.15.7:0/2994957975 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30fc0116e0 con 0x7f3110155210 2023-03-03T08:58:08.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f3116bbf700 1 -- 172.21.15.7:0/2994957975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31100ab630 con 0x7f3110155210 2023-03-03T08:58:08.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.996+0000 7f310d7fa700 1 -- 172.21.15.7:0/2994957975 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc011a90 con 0x7f3110155210 2023-03-03T08:58:08.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.997+0000 7f3116bbf700 1 -- 172.21.15.7:0/2994957975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31100c3e10 con 0x7f3110155210 2023-03-03T08:58:08.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.997+0000 7f3116bbf700 1 -- 172.21.15.7:0/2994957975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3110074bb0 con 0x7f3110155210 2023-03-03T08:58:08.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.998+0000 7f310d7fa700 1 -- 172.21.15.7:0/2994957975 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f30fc011bf0 con 0x7f3110155210 2023-03-03T08:58:08.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.998+0000 7f310d7fa700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30f40682f0 0x7f30f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:08.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.998+0000 7f310d7fa700 1 -- 172.21.15.7:0/2994957975 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f30fc086240 con 0x7f3110155210 2023-03-03T08:58:08.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.999+0000 7f311495b700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30f40682f0 0x7f30f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:08.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:07.999+0000 7f311495b700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30f40682f0 0x7f30f406a7a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f311014af30 tx=0x7f3104008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:08.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.003+0000 7f310d7fa700 1 -- 172.21.15.7:0/2994957975 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30fc052500 con 0x7f3110155210 2023-03-03T08:58:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.243+0000 7f3116bbf700 1 -- 172.21.15.7:0/2994957975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31100c93d0 con 0x7f3110155210 2023-03-03T08:58:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.243+0000 7f310d7fa700 1 -- 172.21.15.7:0/2994957975 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f30fc055b20 con 0x7f3110155210 2023-03-03T08:58:08.245 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:58:08.245 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:58:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.245+0000 7f3116bbf700 1 -- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30f40682f0 msgr2=0x7f30f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.245+0000 7f3116bbf700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30f40682f0 0x7f30f406a7a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f311014af30 tx=0x7f3104008030 comp rx=0 tx=0).stop 2023-03-03T08:58:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.245+0000 7f3116bbf700 1 -- 172.21.15.7:0/2994957975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3110155210 msgr2=0x7f3110146c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.245+0000 7f3116bbf700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3110155210 0x7f3110146c00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f30fc007290 tx=0x7f30fc0075a0 comp rx=0 tx=0).stop 2023-03-03T08:58:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.246+0000 7f3116bbf700 1 -- 172.21.15.7:0/2994957975 shutdown_connections 2023-03-03T08:58:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.246+0000 7f3116bbf700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3110147140 0x7f31101ecbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.246+0000 7f3116bbf700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3100003a70 0x7f31101ded00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.246+0000 7f3116bbf700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30f40682f0 0x7f30f406a7a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.246+0000 7f3116bbf700 1 --2- 172.21.15.7:0/2994957975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3110155210 0x7f3110146c00 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.246+0000 7f3116bbf700 1 -- 172.21.15.7:0/2994957975 >> 172.21.15.7:0/2994957975 conn(0x7f31100c8900 msgr2=0x7f31100c6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:58:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.246+0000 7f3116bbf700 1 -- 172.21.15.7:0/2994957975 shutdown_connections 2023-03-03T08:58:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:08.246+0000 7f3116bbf700 1 -- 172.21.15.7:0/2994957975 wait complete. 2023-03-03T08:58:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:58:08.259 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T08:58:08.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:08.287 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:26.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:58:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.396+0000 7fac523e0700 1 Processor -- start 2023-03-03T08:58:26.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.396+0000 7fac523e0700 1 -- start start 2023-03-03T08:58:26.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.397+0000 7fac523e0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fac4c151b20 0x7fac4c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:26.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.397+0000 7fac523e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 0x7fac4c157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:26.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.397+0000 7fac523e0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac4c073c30 0x7fac4c158020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:26.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.397+0000 7fac523e0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4c06c2f0 con 0x7fac4c073c30 2023-03-03T08:58:26.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.397+0000 7fac523e0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4c06cb70 con 0x7fac4c151b20 2023-03-03T08:58:26.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.397+0000 7fac523e0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c06ce50 con 0x7fac4c0c2950 2023-03-03T08:58:26.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.397+0000 7fac5097d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac4c073c30 0x7fac4c158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:26.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.397+0000 7fac5097d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac4c073c30 0x7fac4c158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53994/0 (socket says 172.21.15.7:53994) 2023-03-03T08:58:26.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.397+0000 7fac5097d700 1 -- 172.21.15.7:0/2142247795 learned_addr learned my addr 172.21.15.7:0/2142247795 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:58:26.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac4bfff700 1 --2- 172.21.15.7:0/2142247795 >> v2:172.21.15.143:3300/0 conn(0x7fac4c151b20 0x7fac4c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:26.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac4b7fe700 1 --2- 172.21.15.7:0/2142247795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 0x7fac4c157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:26.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac5097d700 1 -- 172.21.15.7:0/2142247795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 msgr2=0x7fac4c157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:58:26.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac5097d700 1 --2- 172.21.15.7:0/2142247795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 0x7fac4c157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac5097d700 1 -- 172.21.15.7:0/2142247795 >> v2:172.21.15.143:3300/0 conn(0x7fac4c151b20 msgr2=0x7fac4c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:58:26.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac5097d700 1 --2- 172.21.15.7:0/2142247795 >> v2:172.21.15.143:3300/0 conn(0x7fac4c151b20 0x7fac4c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac5097d700 1 -- 172.21.15.7:0/2142247795 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c06d130 con 0x7fac4c073c30 2023-03-03T08:58:26.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac4b7fe700 1 --2- 172.21.15.7:0/2142247795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 0x7fac4c157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:58:26.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac5097d700 1 --2- 172.21.15.7:0/2142247795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c073c30 0x7fac4c158020 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fac4c06b9c0 tx=0x7fac4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d0d417cce6f27c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:26.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac4affd700 1 -- 172.21.15.7:0/2142247795 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4400e070 con 0x7fac4c073c30 2023-03-03T08:58:26.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac4affd700 1 -- 172.21.15.7:0/2142247795 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4400b910 con 0x7fac4c073c30 2023-03-03T08:58:26.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.398+0000 7fac4affd700 1 -- 172.21.15.7:0/2142247795 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4400bcc0 con 0x7fac4c073c30 2023-03-03T08:58:26.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.399+0000 7fac523e0700 1 -- 172.21.15.7:0/2142247795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c073c30 msgr2=0x7fac4c158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:26.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.399+0000 7fac523e0700 1 --2- 172.21.15.7:0/2142247795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c073c30 0x7fac4c158020 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fac4c06b9c0 tx=0x7fac4400b330 comp rx=0 tx=0).stop 2023-03-03T08:58:26.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.399+0000 7fac523e0700 1 -- 172.21.15.7:0/2142247795 shutdown_connections 2023-03-03T08:58:26.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.399+0000 7fac523e0700 1 --2- 172.21.15.7:0/2142247795 >> v2:172.21.15.143:3300/0 conn(0x7fac4c151b20 0x7fac4c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.399+0000 7fac523e0700 1 --2- 172.21.15.7:0/2142247795 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c073c30 0x7fac4c158020 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.399+0000 7fac523e0700 1 --2- 172.21.15.7:0/2142247795 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 0x7fac4c157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.399+0000 7fac523e0700 1 -- 172.21.15.7:0/2142247795 >> 172.21.15.7:0/2142247795 conn(0x7fac4c0a94d0 msgr2=0x7fac4c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:58:26.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.399+0000 7fac523e0700 1 -- 172.21.15.7:0/2142247795 shutdown_connections 2023-03-03T08:58:26.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.399+0000 7fac523e0700 1 -- 172.21.15.7:0/2142247795 wait complete. 2023-03-03T08:58:26.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.400+0000 7fac523e0700 1 Processor -- start 2023-03-03T08:58:26.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.400+0000 7fac523e0700 1 -- start start 2023-03-03T08:58:26.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac523e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac4c073c30 0x7fac4c0c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:26.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac523e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 0x7fac4c14d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:26.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac523e0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c151b20 0x7fac4c14d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:26.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac523e0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c1562b0 con 0x7fac4c151b20 2023-03-03T08:58:26.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac523e0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c14d1b0 con 0x7fac4c073c30 2023-03-03T08:58:26.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac523e0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c14c860 con 0x7fac4c0c2950 2023-03-03T08:58:26.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac5097d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c151b20 0x7fac4c14d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:26.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac5097d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c151b20 0x7fac4c14d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54000/0 (socket says 172.21.15.7:54000) 2023-03-03T08:58:26.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac5097d700 1 -- 172.21.15.7:0/157490524 learned_addr learned my addr 172.21.15.7:0/157490524 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:58:26.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac4bfff700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac4c073c30 0x7fac4c0c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:26.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac4b7fe700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 0x7fac4c14d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:26.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac5097d700 1 -- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 msgr2=0x7fac4c14d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:26.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac5097d700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 0x7fac4c14d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac5097d700 1 -- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac4c073c30 msgr2=0x7fac4c0c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:26.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac5097d700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac4c073c30 0x7fac4c0c3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.401+0000 7fac5097d700 1 -- 172.21.15.7:0/157490524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac44009b30 con 0x7fac4c151b20 2023-03-03T08:58:26.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.402+0000 7fac5097d700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c151b20 0x7fac4c14d9f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fac4c06bb30 tx=0x7fac44004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:26.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.402+0000 7fac48ff9700 1 -- 172.21.15.7:0/157490524 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac440099b0 con 0x7fac4c151b20 2023-03-03T08:58:26.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.402+0000 7fac48ff9700 1 -- 172.21.15.7:0/157490524 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac44004250 con 0x7fac4c151b20 2023-03-03T08:58:26.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.402+0000 7fac523e0700 1 -- 172.21.15.7:0/157490524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4c151ef0 con 0x7fac4c151b20 2023-03-03T08:58:26.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.402+0000 7fac48ff9700 1 -- 172.21.15.7:0/157490524 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac44004600 con 0x7fac4c151b20 2023-03-03T08:58:26.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.402+0000 7fac523e0700 1 -- 172.21.15.7:0/157490524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4c06d410 con 0x7fac4c151b20 2023-03-03T08:58:26.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.403+0000 7fac523e0700 1 -- 172.21.15.7:0/157490524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4c06bea0 con 0x7fac4c151b20 2023-03-03T08:58:26.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.403+0000 7fac48ff9700 1 -- 172.21.15.7:0/157490524 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fac4402b070 con 0x7fac4c151b20 2023-03-03T08:58:26.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.404+0000 7fac48ff9700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac380682c0 0x7fac3806a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:26.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.404+0000 7fac48ff9700 1 -- 172.21.15.7:0/157490524 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fac44087440 con 0x7fac4c151b20 2023-03-03T08:58:26.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.404+0000 7fac4bfff700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac380682c0 0x7fac3806a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:26.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.404+0000 7fac4bfff700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac380682c0 0x7fac3806a770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fac4c14e530 tx=0x7fac3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:26.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.408+0000 7fac48ff9700 1 -- 172.21.15.7:0/157490524 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fac440537e0 con 0x7fac4c151b20 2023-03-03T08:58:26.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.648+0000 7fac523e0700 1 -- 172.21.15.7:0/157490524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac4c1563f0 con 0x7fac4c151b20 2023-03-03T08:58:26.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.648+0000 7fac48ff9700 1 -- 172.21.15.7:0/157490524 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fac4c1563f0 con 0x7fac4c151b20 2023-03-03T08:58:26.651 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:58:26.651 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:58:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.650+0000 7fac523e0700 1 -- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac380682c0 msgr2=0x7fac3806a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.650+0000 7fac523e0700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac380682c0 0x7fac3806a770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fac4c14e530 tx=0x7fac3c008040 comp rx=0 tx=0).stop 2023-03-03T08:58:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.650+0000 7fac523e0700 1 -- 172.21.15.7:0/157490524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c151b20 msgr2=0x7fac4c14d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.650+0000 7fac523e0700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c151b20 0x7fac4c14d9f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fac4c06bb30 tx=0x7fac44004ad0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.650+0000 7fac523e0700 1 -- 172.21.15.7:0/157490524 shutdown_connections 2023-03-03T08:58:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.650+0000 7fac523e0700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac380682c0 0x7fac3806a770 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.650+0000 7fac523e0700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4c151b20 0x7fac4c14d9f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.650+0000 7fac523e0700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac4c073c30 0x7fac4c0c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.650+0000 7fac523e0700 1 --2- 172.21.15.7:0/157490524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac4c0c2950 0x7fac4c14d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.650+0000 7fac523e0700 1 -- 172.21.15.7:0/157490524 >> 172.21.15.7:0/157490524 conn(0x7fac4c0a94d0 msgr2=0x7fac4c0740e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:58:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.651+0000 7fac523e0700 1 -- 172.21.15.7:0/157490524 shutdown_connections 2023-03-03T08:58:26.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:26.651+0000 7fac523e0700 1 -- 172.21.15.7:0/157490524 wait complete. 2023-03-03T08:58:26.655 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:58:26.665 INFO:tasks.mds_thrash.fs.[b]:waiting for 101 secs before thrashing 2023-03-03T08:58:32.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:58:32.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.845+0000 7f281ab94700 1 Processor -- start 2023-03-03T08:58:32.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.845+0000 7f281ab94700 1 -- start start 2023-03-03T08:58:32.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.846+0000 7f281ab94700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f281414de50 0x7f281406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:32.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.846+0000 7f281ab94700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28140c65c0 0x7f28140c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.846+0000 7f281ab94700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28140a65b0 con 0x7f281406a7f0 2023-03-03T08:58:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.846+0000 7f281ab94700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2814154e00 con 0x7f28140c65c0 2023-03-03T08:58:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.846+0000 7f281ab94700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28140c3370 con 0x7f281414de50 2023-03-03T08:58:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.846+0000 7f2813fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28140c65c0 0x7f28140c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.846+0000 7f2813fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28140c65c0 0x7f28140c2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54012/0 (socket says 172.21.15.7:54012) 2023-03-03T08:58:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.846+0000 7f2813fff700 1 -- 172.21.15.7:0/305126775 learned_addr learned my addr 172.21.15.7:0/305126775 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:58:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.847+0000 7f2818930700 1 --2- 172.21.15.7:0/305126775 >> v2:172.21.15.143:3300/0 conn(0x7f281414de50 0x7f281406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.847+0000 7f2813fff700 1 -- 172.21.15.7:0/305126775 >> v2:172.21.15.143:3300/0 conn(0x7f281414de50 msgr2=0x7f281406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:58:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.847+0000 7f2813fff700 1 --2- 172.21.15.7:0/305126775 >> v2:172.21.15.143:3300/0 conn(0x7f281414de50 0x7f281406a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.847+0000 7f2813fff700 1 -- 172.21.15.7:0/305126775 >> v1:172.21.15.143:6789/0 conn(0x7f281406a7f0 legacy=0x7f28140c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.847+0000 7f2813fff700 1 -- 172.21.15.7:0/305126775 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28141541f0 con 0x7f28140c65c0 2023-03-03T08:58:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.847+0000 7f2813fff700 1 --2- 172.21.15.7:0/305126775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28140c65c0 0x7f28140c2720 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f28140c7760 tx=0x7f280000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89efd88f1b281f5e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.847+0000 7f28137fe700 1 -- 172.21.15.7:0/305126775 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280000dcf0 con 0x7f28140c65c0 2023-03-03T08:58:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.847+0000 7f28137fe700 1 -- 172.21.15.7:0/305126775 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280000de50 con 0x7f28140c65c0 2023-03-03T08:58:32.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.847+0000 7f28137fe700 1 -- 172.21.15.7:0/305126775 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2800010690 con 0x7f28140c65c0 2023-03-03T08:58:32.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.848+0000 7f281ab94700 1 -- 172.21.15.7:0/305126775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28140c65c0 msgr2=0x7f28140c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:32.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.848+0000 7f281ab94700 1 --2- 172.21.15.7:0/305126775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28140c65c0 0x7f28140c2720 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f28140c7760 tx=0x7f280000bea0 comp rx=0 tx=0).stop 2023-03-03T08:58:32.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.848+0000 7f281ab94700 1 -- 172.21.15.7:0/305126775 shutdown_connections 2023-03-03T08:58:32.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.848+0000 7f281ab94700 1 --2- 172.21.15.7:0/305126775 >> v2:172.21.15.143:3300/0 conn(0x7f281414de50 0x7f281406a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:32.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.848+0000 7f281ab94700 1 --2- 172.21.15.7:0/305126775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28140c65c0 0x7f28140c2720 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:32.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.848+0000 7f281ab94700 1 -- 172.21.15.7:0/305126775 >> 172.21.15.7:0/305126775 conn(0x7f28140a9430 msgr2=0x7f28140c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:58:32.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.848+0000 7f281ab94700 1 -- 172.21.15.7:0/305126775 shutdown_connections 2023-03-03T08:58:32.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.848+0000 7f281ab94700 1 -- 172.21.15.7:0/305126775 wait complete. 2023-03-03T08:58:32.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.849+0000 7f281ab94700 1 Processor -- start 2023-03-03T08:58:32.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.849+0000 7f281ab94700 1 -- start start 2023-03-03T08:58:32.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f281ab94700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f281406a7f0 0x7f281406d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:32.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f281ab94700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28140c65c0 0x7f28141e9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:32.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f281ab94700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f281414de50 0x7f281414a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:32.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f281ab94700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28141eaab0 con 0x7f281414de50 2023-03-03T08:58:32.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f281ab94700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28141e9ff0 con 0x7f281406a7f0 2023-03-03T08:58:32.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f281ab94700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2814153580 con 0x7f28140c65c0 2023-03-03T08:58:32.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f2819131700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f281414de50 0x7f281414a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:32.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f2819131700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f281414de50 0x7f281414a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54018/0 (socket says 172.21.15.7:54018) 2023-03-03T08:58:32.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f2819131700 1 -- 172.21.15.7:0/2912373951 learned_addr learned my addr 172.21.15.7:0/2912373951 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:58:32.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f2818930700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f281406a7f0 0x7f281406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:32.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f2813fff700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28140c65c0 0x7f28141e9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:32.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f2819131700 1 -- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28140c65c0 msgr2=0x7f28141e9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:32.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f2819131700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28140c65c0 0x7f28141e9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f2819131700 1 -- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f281406a7f0 msgr2=0x7f281406d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f2819131700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f281406a7f0 0x7f281406d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.850+0000 7f2819131700 1 -- 172.21.15.7:0/2912373951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280000b7f0 con 0x7f281414de50 2023-03-03T08:58:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.851+0000 7f2819131700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f281414de50 0x7f281414a530 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f280c00cb50 tx=0x7f280c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.851+0000 7f28117fa700 1 -- 172.21.15.7:0/2912373951 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c003c00 con 0x7f281414de50 2023-03-03T08:58:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.851+0000 7f28117fa700 1 -- 172.21.15.7:0/2912373951 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c003d60 con 0x7f281414de50 2023-03-03T08:58:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.851+0000 7f281ab94700 1 -- 172.21.15.7:0/2912373951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28141547b0 con 0x7f281414de50 2023-03-03T08:58:32.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.851+0000 7f28117fa700 1 -- 172.21.15.7:0/2912373951 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c00aa20 con 0x7f281414de50 2023-03-03T08:58:32.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.851+0000 7f281ab94700 1 -- 172.21.15.7:0/2912373951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28141544d0 con 0x7f281414de50 2023-03-03T08:58:32.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.852+0000 7f281ab94700 1 -- 172.21.15.7:0/2912373951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2814154a90 con 0x7f281414de50 2023-03-03T08:58:32.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.852+0000 7f28117fa700 1 -- 172.21.15.7:0/2912373951 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f280c010050 con 0x7f281414de50 2023-03-03T08:58:32.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.853+0000 7f28117fa700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27f80682f0 0x7f27f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:58:32.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.853+0000 7f28117fa700 1 -- 172.21.15.7:0/2912373951 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f280c085fe0 con 0x7f281414de50 2023-03-03T08:58:32.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.853+0000 7f2818930700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27f80682f0 0x7f27f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:58:32.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.853+0000 7f2818930700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27f80682f0 0x7f27f806a7a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2814072a40 tx=0x7f2808009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:58:32.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:32.857+0000 7f28117fa700 1 -- 172.21.15.7:0/2912373951 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f280c052380 con 0x7f281414de50 2023-03-03T08:58:33.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.093+0000 7f281ab94700 1 -- 172.21.15.7:0/2912373951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2814074bb0 con 0x7f281414de50 2023-03-03T08:58:33.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.093+0000 7f28117fa700 1 -- 172.21.15.7:0/2912373951 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f280c0559a0 con 0x7f281414de50 2023-03-03T08:58:33.096 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:58:33.096 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:58:33.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 -- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27f80682f0 msgr2=0x7f27f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27f80682f0 0x7f27f806a7a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2814072a40 tx=0x7f2808009210 comp rx=0 tx=0).stop 2023-03-03T08:58:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 -- 172.21.15.7:0/2912373951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f281414de50 msgr2=0x7f281414a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:58:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f281414de50 0x7f281414a530 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f280c00cb50 tx=0x7f280c00ce60 comp rx=0 tx=0).stop 2023-03-03T08:58:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 -- 172.21.15.7:0/2912373951 shutdown_connections 2023-03-03T08:58:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27f80682f0 0x7f27f806a7a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f281414de50 0x7f281414a530 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:33.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f281406a7f0 0x7f281406d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 --2- 172.21.15.7:0/2912373951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28140c65c0 0x7f28141e9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:58:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 -- 172.21.15.7:0/2912373951 >> 172.21.15.7:0/2912373951 conn(0x7f28140a9430 msgr2=0x7f28140a7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:58:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 -- 172.21.15.7:0/2912373951 shutdown_connections 2023-03-03T08:58:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:58:33.095+0000 7f281ab94700 1 -- 172.21.15.7:0/2912373951 wait complete. 2023-03-03T08:58:33.099 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:58:33.109 INFO:tasks.mds_thrash.fs.[a]:waiting for 55 secs before thrashing 2023-03-03T08:58:38.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:58:38.688 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:08.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:08.771 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:28.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T08:59:28.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.246+0000 7f458b6fa700 1 Processor -- start 2023-03-03T08:59:28.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.247+0000 7f458b6fa700 1 -- start start 2023-03-03T08:59:28.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.247+0000 7f458b6fa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f458414f9d0 0x7f4584156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.247+0000 7f458b6fa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f45840abe20 0x7f45841529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.247+0000 7f458b6fa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45840c2f30 con 0x7f458414fda0 2023-03-03T08:59:28.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.247+0000 7f458b6fa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45840c31d0 con 0x7f45840abe20 2023-03-03T08:59:28.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.247+0000 7f458b6fa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45841524a0 con 0x7f458414f9d0 2023-03-03T08:59:28.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.247+0000 7f4589c97700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f45840abe20 0x7f45841529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4588c95700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f458414fda0 0x7f4584071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48012/0 (socket says 172.21.15.7:48012) 2023-03-03T08:59:28.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4588c95700 1 -- 172.21.15.7:0/2389104115 learned_addr learned my addr 172.21.15.7:0/2389104115 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T08:59:28.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4589496700 1 --2- 172.21.15.7:0/2389104115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f458414f9d0 0x7f4584156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4589c97700 1 -- 172.21.15.7:0/2389104115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f458414f9d0 msgr2=0x7f4584156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T08:59:28.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4589c97700 1 --2- 172.21.15.7:0/2389104115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f458414f9d0 0x7f4584156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4589c97700 1 -- 172.21.15.7:0/2389104115 >> v1:172.21.15.7:6789/0 conn(0x7f458414fda0 legacy=0x7f4584071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4589c97700 1 -- 172.21.15.7:0/2389104115 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45840c34b0 con 0x7f45840abe20 2023-03-03T08:59:28.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4577fff700 1 -- 172.21.15.7:0/2389104115 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438171806 0 0) 0x7f45840c2f30 con 0x7f458414fda0 2023-03-03T08:59:28.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4589c97700 1 --2- 172.21.15.7:0/2389104115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 0x7f45841529c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f457c009d90 tx=0x7f457c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12df3be6302e577e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:28.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4577fff700 1 -- 172.21.15.7:0/2389104115 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c00e070 con 0x7f45840abe20 2023-03-03T08:59:28.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4577fff700 1 -- 172.21.15.7:0/2389104115 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c00bbd0 con 0x7f45840abe20 2023-03-03T08:59:28.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.248+0000 7f4577fff700 1 -- 172.21.15.7:0/2389104115 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c012430 con 0x7f45840abe20 2023-03-03T08:59:28.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.249+0000 7f458b6fa700 1 -- 172.21.15.7:0/2389104115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 msgr2=0x7f45841529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.249+0000 7f458b6fa700 1 --2- 172.21.15.7:0/2389104115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 0x7f45841529c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f457c009d90 tx=0x7f457c00b330 comp rx=0 tx=0).stop 2023-03-03T08:59:28.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.249+0000 7f458b6fa700 1 -- 172.21.15.7:0/2389104115 shutdown_connections 2023-03-03T08:59:28.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.249+0000 7f458b6fa700 1 --2- 172.21.15.7:0/2389104115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 0x7f45841529c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.249+0000 7f458b6fa700 1 --2- 172.21.15.7:0/2389104115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f458414f9d0 0x7f4584156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.249+0000 7f458b6fa700 1 -- 172.21.15.7:0/2389104115 >> 172.21.15.7:0/2389104115 conn(0x7f45840c8900 msgr2=0x7f45840c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:59:28.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.249+0000 7f458b6fa700 1 -- 172.21.15.7:0/2389104115 shutdown_connections 2023-03-03T08:59:28.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.249+0000 7f458b6fa700 1 -- 172.21.15.7:0/2389104115 wait complete. 2023-03-03T08:59:28.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.250+0000 7f458b6fa700 1 Processor -- start 2023-03-03T08:59:28.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.250+0000 7f458b6fa700 1 -- start start 2023-03-03T08:59:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f458b6fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 0x7f45841e8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f458b6fa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f458414f9d0 0x7f458406f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f458b6fa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f458414fda0 0x7f4584154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f458b6fa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45840c2a40 con 0x7f45840abe20 2023-03-03T08:59:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f458b6fa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458406ec50 con 0x7f458414fda0 2023-03-03T08:59:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f458b6fa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45840bffa0 con 0x7f458414f9d0 2023-03-03T08:59:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589496700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 0x7f45841e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589496700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 0x7f45841e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54038/0 (socket says 172.21.15.7:54038) 2023-03-03T08:59:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589496700 1 -- 172.21.15.7:0/559196527 learned_addr learned my addr 172.21.15.7:0/559196527 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T08:59:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4588c95700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f458414f9d0 0x7f458406f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589c97700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f458414fda0 0x7f4584154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589496700 1 -- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f458414f9d0 msgr2=0x7f458406f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589496700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f458414f9d0 0x7f458406f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589496700 1 -- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f458414fda0 msgr2=0x7f4584154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589496700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f458414fda0 0x7f4584154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589496700 1 -- 172.21.15.7:0/559196527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c009a70 con 0x7f45840abe20 2023-03-03T08:59:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589c97700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f458414fda0 0x7f4584154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T08:59:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4589496700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 0x7f45841e8170 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f458000e810 tx=0x7f458000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.251+0000 7f4575ffb700 1 -- 172.21.15.7:0/559196527 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580013610 con 0x7f45840abe20 2023-03-03T08:59:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.252+0000 7f4575ffb700 1 -- 172.21.15.7:0/559196527 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4580013770 con 0x7f45840abe20 2023-03-03T08:59:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.252+0000 7f458b6fa700 1 -- 172.21.15.7:0/559196527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45840ac350 con 0x7f45840abe20 2023-03-03T08:59:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.252+0000 7f4575ffb700 1 -- 172.21.15.7:0/559196527 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580013b20 con 0x7f45840abe20 2023-03-03T08:59:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.252+0000 7f458b6fa700 1 -- 172.21.15.7:0/559196527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45840ab630 con 0x7f45840abe20 2023-03-03T08:59:28.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.253+0000 7f458b6fa700 1 -- 172.21.15.7:0/559196527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4584071fa0 con 0x7f45840abe20 2023-03-03T08:59:28.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.253+0000 7f4575ffb700 1 -- 172.21.15.7:0/559196527 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4580023070 con 0x7f45840abe20 2023-03-03T08:59:28.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.253+0000 7f4575ffb700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45680682f0 0x7f456806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T08:59:28.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.254+0000 7f4575ffb700 1 -- 172.21.15.7:0/559196527 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4580085e00 con 0x7f45840abe20 2023-03-03T08:59:28.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.254+0000 7f4588c95700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45680682f0 0x7f456806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T08:59:28.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.254+0000 7f4588c95700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45680682f0 0x7f456806a7a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f45840c0380 tx=0x7f4578007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T08:59:28.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.258+0000 7f4575ffb700 1 -- 172.21.15.7:0/559196527 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4580052120 con 0x7f45840abe20 2023-03-03T08:59:28.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.495+0000 7f458b6fa700 1 -- 172.21.15.7:0/559196527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f458406ef30 con 0x7f45840abe20 2023-03-03T08:59:28.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.496+0000 7f4575ffb700 1 -- 172.21.15.7:0/559196527 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4580059680 con 0x7f45840abe20 2023-03-03T08:59:28.498 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T08:59:28.498 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T08:59:28.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 -- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45680682f0 msgr2=0x7f456806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45680682f0 0x7f456806a7a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f45840c0380 tx=0x7f4578007de0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 -- 172.21.15.7:0/559196527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 msgr2=0x7f45841e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T08:59:28.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 0x7f45841e8170 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f458000e810 tx=0x7f458000eb20 comp rx=0 tx=0).stop 2023-03-03T08:59:28.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 -- 172.21.15.7:0/559196527 shutdown_connections 2023-03-03T08:59:28.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f458414fda0 0x7f4584154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45680682f0 0x7f456806a7a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45840abe20 0x7f45841e8170 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 --2- 172.21.15.7:0/559196527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f458414f9d0 0x7f458406f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T08:59:28.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 -- 172.21.15.7:0/559196527 >> 172.21.15.7:0/559196527 conn(0x7f45840c8900 msgr2=0x7f45840c8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T08:59:28.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 -- 172.21.15.7:0/559196527 shutdown_connections 2023-03-03T08:59:28.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T08:59:28.498+0000 7f458b6fa700 1 -- 172.21.15.7:0/559196527 wait complete. 2023-03-03T08:59:28.502 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T08:59:28.512 INFO:tasks.mds_thrash.fs.[a]:waiting for 38 secs before thrashing 2023-03-03T08:59:39.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T08:59:39.029 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T08:59:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T09:00:06.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:00:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.653+0000 7f557384c700 1 Processor -- start 2023-03-03T09:00:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.653+0000 7f557384c700 1 -- start start 2023-03-03T09:00:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.653+0000 7f557384c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c14b340 0x7f556c152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f557384c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f556c1546f0 0x7f556c154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f557384c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c06c920 con 0x7f556c0c2790 2023-03-03T09:00:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f557384c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c0abd90 con 0x7f556c1546f0 2023-03-03T09:00:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f557384c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0c1300 con 0x7f556c14b340 2023-03-03T09:00:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f5570de7700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f556c0c2790 0x7f556c152b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48048/0 (socket says 172.21.15.7:48048) 2023-03-03T09:00:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f5570de7700 1 -- 172.21.15.7:0/3145378141 learned_addr learned my addr 172.21.15.7:0/3145378141 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:00:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f55715e8700 1 --2- 172.21.15.7:0/3145378141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c14b340 0x7f556c152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f5571de9700 1 --2- 172.21.15.7:0/3145378141 >> v2:172.21.15.143:3300/0 conn(0x7f556c1546f0 0x7f556c154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f5563fff700 1 -- 172.21.15.7:0/3145378141 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656855383 0 0) 0x7f556c06c920 con 0x7f556c0c2790 2023-03-03T09:00:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f5563fff700 1 -- 172.21.15.7:0/3145378141 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55580035b0 con 0x7f556c0c2790 2023-03-03T09:00:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f5563fff700 1 -- 172.21.15.7:0/3145378141 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1062095121 0 0) 0x7f55580035b0 con 0x7f556c0c2790 2023-03-03T09:00:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f5571de9700 1 -- 172.21.15.7:0/3145378141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c14b340 msgr2=0x7f556c152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:00:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f5571de9700 1 --2- 172.21.15.7:0/3145378141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c14b340 0x7f556c152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f5571de9700 1 -- 172.21.15.7:0/3145378141 >> v1:172.21.15.7:6789/0 conn(0x7f556c0c2790 legacy=0x7f556c152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f5571de9700 1 -- 172.21.15.7:0/3145378141 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c153280 con 0x7f556c1546f0 2023-03-03T09:00:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.654+0000 7f55715e8700 1 --2- 172.21.15.7:0/3145378141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c14b340 0x7f556c152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:00:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.655+0000 7f5571de9700 1 --2- 172.21.15.7:0/3145378141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f556c1546f0 0x7f556c154b40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5564009d90 tx=0x7f556400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc2e88ed3588f1f7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:00:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.655+0000 7f5563fff700 1 -- 172.21.15.7:0/3145378141 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556400e070 con 0x7f556c1546f0 2023-03-03T09:00:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.655+0000 7f5563fff700 1 -- 172.21.15.7:0/3145378141 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5564003a20 con 0x7f556c1546f0 2023-03-03T09:00:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.655+0000 7f5563fff700 1 -- 172.21.15.7:0/3145378141 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c001670 con 0x7f556c1546f0 2023-03-03T09:00:06.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.655+0000 7f557384c700 1 -- 172.21.15.7:0/3145378141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f556c1546f0 msgr2=0x7f556c154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.655+0000 7f557384c700 1 --2- 172.21.15.7:0/3145378141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f556c1546f0 0x7f556c154b40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5564009d90 tx=0x7f556400b330 comp rx=0 tx=0).stop 2023-03-03T09:00:06.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.655+0000 7f557384c700 1 -- 172.21.15.7:0/3145378141 shutdown_connections 2023-03-03T09:00:06.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.655+0000 7f557384c700 1 --2- 172.21.15.7:0/3145378141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f556c1546f0 0x7f556c154b40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.655+0000 7f557384c700 1 --2- 172.21.15.7:0/3145378141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c14b340 0x7f556c152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.655+0000 7f557384c700 1 -- 172.21.15.7:0/3145378141 >> 172.21.15.7:0/3145378141 conn(0x7f556c0c62e0 msgr2=0x7f556c0c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:00:06.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.656+0000 7f557384c700 1 -- 172.21.15.7:0/3145378141 shutdown_connections 2023-03-03T09:00:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.656+0000 7f557384c700 1 -- 172.21.15.7:0/3145378141 wait complete. 2023-03-03T09:00:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.657+0000 7f557384c700 1 Processor -- start 2023-03-03T09:00:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.657+0000 7f557384c700 1 -- start start 2023-03-03T09:00:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f557384c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556c0c2790 0x7f556c1ea2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f557384c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f556c14b340 0x7f556c0bf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f557384c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c1546f0 0x7f556c0a9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f557384c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0c5d20 con 0x7f556c0c2790 2023-03-03T09:00:06.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f557384c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c144cd0 con 0x7f556c14b340 2023-03-03T09:00:06.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f557384c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c1ea820 con 0x7f556c1546f0 2023-03-03T09:00:06.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f55715e8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556c0c2790 0x7f556c1ea2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f55715e8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556c0c2790 0x7f556c1ea2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54072/0 (socket says 172.21.15.7:54072) 2023-03-03T09:00:06.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f55715e8700 1 -- 172.21.15.7:0/690286757 learned_addr learned my addr 172.21.15.7:0/690286757 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:00:06.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f5570de7700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f556c14b340 0x7f556c0bf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f5571de9700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c1546f0 0x7f556c0a9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f55715e8700 1 -- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c1546f0 msgr2=0x7f556c0a9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f55715e8700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c1546f0 0x7f556c0a9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f55715e8700 1 -- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f556c14b340 msgr2=0x7f556c0bf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f55715e8700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f556c14b340 0x7f556c0bf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f55715e8700 1 -- 172.21.15.7:0/690286757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55680094a0 con 0x7f556c0c2790 2023-03-03T09:00:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f5570de7700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f556c14b340 0x7f556c0bf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:00:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f5571de9700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c1546f0 0x7f556c0a9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:00:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.658+0000 7f55715e8700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556c0c2790 0x7f556c1ea2e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f556800ebe0 tx=0x7f5568009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:00:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.659+0000 7f5561ffb700 1 -- 172.21.15.7:0/690286757 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556800c8b0 con 0x7f556c0c2790 2023-03-03T09:00:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.659+0000 7f5561ffb700 1 -- 172.21.15.7:0/690286757 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556800ca10 con 0x7f556c0c2790 2023-03-03T09:00:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.659+0000 7f557384c700 1 -- 172.21.15.7:0/690286757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5564009a70 con 0x7f556c0c2790 2023-03-03T09:00:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.659+0000 7f5561ffb700 1 -- 172.21.15.7:0/690286757 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556800cda0 con 0x7f556c0c2790 2023-03-03T09:00:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.659+0000 7f557384c700 1 -- 172.21.15.7:0/690286757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c06bff0 con 0x7f556c0c2790 2023-03-03T09:00:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.660+0000 7f557384c700 1 -- 172.21.15.7:0/690286757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c06bd50 con 0x7f556c0c2790 2023-03-03T09:00:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.660+0000 7f5561ffb700 1 -- 172.21.15.7:0/690286757 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5568019030 con 0x7f556c0c2790 2023-03-03T09:00:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.661+0000 7f5561ffb700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55500682f0 0x7f555006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.661+0000 7f5561ffb700 1 -- 172.21.15.7:0/690286757 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5568086130 con 0x7f556c0c2790 2023-03-03T09:00:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.661+0000 7f5570de7700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55500682f0 0x7f555006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:06.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.661+0000 7f5570de7700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55500682f0 0x7f555006a7a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f555c001eb0 tx=0x7f555c001c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:00:06.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.665+0000 7f5561ffb700 1 -- 172.21.15.7:0/690286757 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f55680524d0 con 0x7f556c0c2790 2023-03-03T09:00:06.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.902+0000 7f557384c700 1 -- 172.21.15.7:0/690286757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f556c06c2d0 con 0x7f556c0c2790 2023-03-03T09:00:06.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.902+0000 7f5561ffb700 1 -- 172.21.15.7:0/690286757 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5568055af0 con 0x7f556c0c2790 2023-03-03T09:00:06.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:00:06.904 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:00:06.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 -- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55500682f0 msgr2=0x7f555006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55500682f0 0x7f555006a7a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f555c001eb0 tx=0x7f555c001c60 comp rx=0 tx=0).stop 2023-03-03T09:00:06.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 -- 172.21.15.7:0/690286757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556c0c2790 msgr2=0x7f556c1ea2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:06.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556c0c2790 0x7f556c1ea2e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f556800ebe0 tx=0x7f5568009e20 comp rx=0 tx=0).stop 2023-03-03T09:00:06.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 -- 172.21.15.7:0/690286757 shutdown_connections 2023-03-03T09:00:06.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f556c1546f0 0x7f556c0a9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55500682f0 0x7f555006a7a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556c0c2790 0x7f556c1ea2e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 --2- 172.21.15.7:0/690286757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f556c14b340 0x7f556c0bf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 -- 172.21.15.7:0/690286757 >> 172.21.15.7:0/690286757 conn(0x7f556c0c62e0 msgr2=0x7f556c148290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:00:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 -- 172.21.15.7:0/690286757 shutdown_connections 2023-03-03T09:00:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:06.904+0000 7f557384c700 1 -- 172.21.15.7:0/690286757 wait complete. 2023-03-03T09:00:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:00:06.917 INFO:tasks.mds_thrash.fs.[a]:waiting for 69 secs before thrashing 2023-03-03T09:00:07.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:00:07.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.804+0000 7f4cdfd30700 1 Processor -- start 2023-03-03T09:00:07.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.804+0000 7f4cdfd30700 1 -- start start 2023-03-03T09:00:07.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.805+0000 7f4cdfd30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd814e560 0x7f4cd8071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:07.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.805+0000 7f4cdfd30700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4cd80c4840 0x7f4cd8154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:07.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.805+0000 7f4cdfd30700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd8072920 con 0x7f4cd80c43d0 2023-03-03T09:00:07.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.805+0000 7f4cdfd30700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd806bef0 con 0x7f4cd80c4840 2023-03-03T09:00:07.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.805+0000 7f4cdfd30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd81551e0 con 0x7f4cd814e560 2023-03-03T09:00:07.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.805+0000 7f4cddacc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd814e560 0x7f4cd8071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:07.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.805+0000 7f4cde2cd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4cd80c4840 0x7f4cd8154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:07.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.805+0000 7f4cddacc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd814e560 0x7f4cd8071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:33924/0 (socket says 172.21.15.7:33924) 2023-03-03T09:00:07.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.805+0000 7f4cddacc700 1 -- 172.21.15.7:0/568243657 learned_addr learned my addr 172.21.15.7:0/568243657 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:00:07.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cddacc700 1 -- 172.21.15.7:0/568243657 >> v2:172.21.15.143:3300/0 conn(0x7f4cd80c4840 msgr2=0x7f4cd8154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:00:07.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cddacc700 1 --2- 172.21.15.7:0/568243657 >> v2:172.21.15.143:3300/0 conn(0x7f4cd80c4840 0x7f4cd8154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:07.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cddacc700 1 -- 172.21.15.7:0/568243657 >> v1:172.21.15.143:6789/0 conn(0x7f4cd80c43d0 legacy=0x7f4cd8154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:07.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cddacc700 1 -- 172.21.15.7:0/568243657 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd806b910 con 0x7f4cd814e560 2023-03-03T09:00:07.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cdcaca700 1 -- 172.21.15.7:0/568243657 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704607055 0 0) 0x7f4cd4001110 con 0x7f4cd80c43d0 2023-03-03T09:00:07.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cde2cd700 1 --2- 172.21.15.7:0/568243657 >> v2:172.21.15.143:3300/0 conn(0x7f4cd80c4840 0x7f4cd8154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:00:07.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cddacc700 1 --2- 172.21.15.7:0/568243657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd814e560 0x7f4cd8071c70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80098f0 tx=0x7f4cc8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14ca3591fbdd892e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:00:07.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cdcaca700 1 -- 172.21.15.7:0/568243657 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8011830 con 0x7f4cd814e560 2023-03-03T09:00:07.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cdcaca700 1 -- 172.21.15.7:0/568243657 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc8011990 con 0x7f4cd814e560 2023-03-03T09:00:07.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cdcaca700 1 -- 172.21.15.7:0/568243657 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8011d20 con 0x7f4cd814e560 2023-03-03T09:00:07.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cdfd30700 1 -- 172.21.15.7:0/568243657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd814e560 msgr2=0x7f4cd8071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:07.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.806+0000 7f4cdfd30700 1 --2- 172.21.15.7:0/568243657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd814e560 0x7f4cd8071c70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4cc80098f0 tx=0x7f4cc8009c00 comp rx=0 tx=0).stop 2023-03-03T09:00:07.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.807+0000 7f4cdfd30700 1 -- 172.21.15.7:0/568243657 shutdown_connections 2023-03-03T09:00:07.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.807+0000 7f4cdfd30700 1 --2- 172.21.15.7:0/568243657 >> v2:172.21.15.143:3300/0 conn(0x7f4cd80c4840 0x7f4cd8154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:07.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.807+0000 7f4cdfd30700 1 --2- 172.21.15.7:0/568243657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd814e560 0x7f4cd8071c70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:07.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.807+0000 7f4cdfd30700 1 -- 172.21.15.7:0/568243657 >> 172.21.15.7:0/568243657 conn(0x7f4cd80736f0 msgr2=0x7f4cd8073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:00:07.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.807+0000 7f4cdfd30700 1 -- 172.21.15.7:0/568243657 shutdown_connections 2023-03-03T09:00:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.807+0000 7f4cdfd30700 1 -- 172.21.15.7:0/568243657 wait complete. 2023-03-03T09:00:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.808+0000 7f4cdfd30700 1 Processor -- start 2023-03-03T09:00:07.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.808+0000 7f4cdfd30700 1 -- start start 2023-03-03T09:00:07.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cdfd30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cd80c43d0 0x7f4cd80adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:07.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cdfd30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd80c4840 0x7f4cd80ae080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:07.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cdfd30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cd814e560 0x7f4cd80c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:07.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cdfd30700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80c66c0 con 0x7f4cd814e560 2023-03-03T09:00:07.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cdfd30700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80ad420 con 0x7f4cd80c43d0 2023-03-03T09:00:07.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cdfd30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd80c6b50 con 0x7f4cd80c4840 2023-03-03T09:00:07.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cde2cd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cd814e560 0x7f4cd80c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:07.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cde2cd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cd814e560 0x7f4cd80c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54094/0 (socket says 172.21.15.7:54094) 2023-03-03T09:00:07.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cde2cd700 1 -- 172.21.15.7:0/2622314639 learned_addr learned my addr 172.21.15.7:0/2622314639 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:00:07.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cddacc700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cd80c43d0 0x7f4cd80adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:07.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cdd2cb700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd80c4840 0x7f4cd80ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:07.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cde2cd700 1 -- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd80c4840 msgr2=0x7f4cd80ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:07.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cde2cd700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd80c4840 0x7f4cd80ae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:07.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cde2cd700 1 -- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cd80c43d0 msgr2=0x7f4cd80adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:07.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cde2cd700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cd80c43d0 0x7f4cd80adb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:07.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cde2cd700 1 -- 172.21.15.7:0/2622314639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd0009a70 con 0x7f4cd814e560 2023-03-03T09:00:07.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cdd2cb700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd80c4840 0x7f4cd80ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:00:07.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.809+0000 7f4cde2cd700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cd814e560 0x7f4cd80c7960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000efc0 tx=0x7f4cd000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:00:07.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.810+0000 7f4cce7fc700 1 -- 172.21.15.7:0/2622314639 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd000cb40 con 0x7f4cd814e560 2023-03-03T09:00:07.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.810+0000 7f4cce7fc700 1 -- 172.21.15.7:0/2622314639 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd000f040 con 0x7f4cd814e560 2023-03-03T09:00:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.810+0000 7f4cdfd30700 1 -- 172.21.15.7:0/2622314639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc8009550 con 0x7f4cd814e560 2023-03-03T09:00:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.810+0000 7f4cce7fc700 1 -- 172.21.15.7:0/2622314639 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0003a20 con 0x7f4cd814e560 2023-03-03T09:00:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.810+0000 7f4cdfd30700 1 -- 172.21.15.7:0/2622314639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd80aacc0 con 0x7f4cd814e560 2023-03-03T09:00:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.811+0000 7f4cdfd30700 1 -- 172.21.15.7:0/2622314639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd80c4ca0 con 0x7f4cd814e560 2023-03-03T09:00:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.811+0000 7f4cce7fc700 1 -- 172.21.15.7:0/2622314639 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4cd0014480 con 0x7f4cd814e560 2023-03-03T09:00:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.811+0000 7f4cce7fc700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4cbc0682f0 0x7f4cbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:00:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.811+0000 7f4cce7fc700 1 -- 172.21.15.7:0/2622314639 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4cd0086c20 con 0x7f4cd814e560 2023-03-03T09:00:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.812+0000 7f4cddacc700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4cbc0682f0 0x7f4cbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:00:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.812+0000 7f4cddacc700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4cbc0682f0 0x7f4cbc06a7a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8014040 tx=0x7f4cc8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:00:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:07.816+0000 7f4cce7fc700 1 -- 172.21.15.7:0/2622314639 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4cd0052f60 con 0x7f4cd814e560 2023-03-03T09:00:08.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.056+0000 7f4cdfd30700 1 -- 172.21.15.7:0/2622314639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cd806bb80 con 0x7f4cd814e560 2023-03-03T09:00:08.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.056+0000 7f4cce7fc700 1 -- 172.21.15.7:0/2622314639 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4cd0056580 con 0x7f4cd814e560 2023-03-03T09:00:08.058 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:00:08.058 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:00:08.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 -- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4cbc0682f0 msgr2=0x7f4cbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:08.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4cbc0682f0 0x7f4cbc06a7a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8014040 tx=0x7f4cc8008030 comp rx=0 tx=0).stop 2023-03-03T09:00:08.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 -- 172.21.15.7:0/2622314639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cd814e560 msgr2=0x7f4cd80c7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:00:08.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cd814e560 0x7f4cd80c7960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000efc0 tx=0x7f4cd000c3a0 comp rx=0 tx=0).stop 2023-03-03T09:00:08.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 -- 172.21.15.7:0/2622314639 shutdown_connections 2023-03-03T09:00:08.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4cbc0682f0 0x7f4cbc06a7a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:08.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cd814e560 0x7f4cd80c7960 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:08.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cd80c43d0 0x7f4cd80adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:08.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 --2- 172.21.15.7:0/2622314639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cd80c4840 0x7f4cd80ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:00:08.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 -- 172.21.15.7:0/2622314639 >> 172.21.15.7:0/2622314639 conn(0x7f4cd80736f0 msgr2=0x7f4cd814bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:00:08.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 -- 172.21.15.7:0/2622314639 shutdown_connections 2023-03-03T09:00:08.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:00:08.058+0000 7f4cdfd30700 1 -- 172.21.15.7:0/2622314639 wait complete. 2023-03-03T09:00:08.061 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:00:08.071 INFO:tasks.mds_thrash.fs.[b]:waiting for 114 secs before thrashing 2023-03-03T09:00:09.114 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:09.117 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:39.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:00:39.228 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:09.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:09.328 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:15.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:01:16.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.056+0000 7fb23a6bc700 1 Processor -- start 2023-03-03T09:01:16.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.056+0000 7fb23a6bc700 1 -- start start 2023-03-03T09:01:16.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb23a6bc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2341586c0 0x7fb23406d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:16.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb23a6bc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2340c1da0 0x7fb23406d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb23a6bc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb234154560 con 0x7fb23413a0f0 2023-03-03T09:01:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb23a6bc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb234154840 con 0x7fb2340c1da0 2023-03-03T09:01:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb23a6bc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb234157be0 con 0x7fb2341586c0 2023-03-03T09:01:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb238c59700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2340c1da0 0x7fb23406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb238c59700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2340c1da0 0x7fb23406d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54134/0 (socket says 172.21.15.7:54134) 2023-03-03T09:01:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb238c59700 1 -- 172.21.15.7:0/613863072 learned_addr learned my addr 172.21.15.7:0/613863072 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:01:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb2337fe700 1 --2- 172.21.15.7:0/613863072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2341586c0 0x7fb23406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:16.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb238c59700 1 -- 172.21.15.7:0/613863072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2341586c0 msgr2=0x7fb23406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:01:16.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb238c59700 1 --2- 172.21.15.7:0/613863072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2341586c0 0x7fb23406d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:16.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb238c59700 1 -- 172.21.15.7:0/613863072 >> v1:172.21.15.143:6789/0 conn(0x7fb23413a0f0 legacy=0x7fb2340c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:16.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb238c59700 1 -- 172.21.15.7:0/613863072 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb234157ec0 con 0x7fb2340c1da0 2023-03-03T09:01:16.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.057+0000 7fb238c59700 1 --2- 172.21.15.7:0/613863072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2340c1da0 0x7fb23406d780 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009e50 tx=0x7fb22c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f6da34b505f498 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:01:16.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb232ffd700 1 -- 172.21.15.7:0/613863072 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c00e070 con 0x7fb2340c1da0 2023-03-03T09:01:16.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb232ffd700 1 -- 172.21.15.7:0/613863072 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c00b910 con 0x7fb2340c1da0 2023-03-03T09:01:16.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb232ffd700 1 -- 172.21.15.7:0/613863072 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c00bcc0 con 0x7fb2340c1da0 2023-03-03T09:01:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb23a6bc700 1 -- 172.21.15.7:0/613863072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2340c1da0 msgr2=0x7fb23406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb23a6bc700 1 --2- 172.21.15.7:0/613863072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2340c1da0 0x7fb23406d780 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009e50 tx=0x7fb22c00b330 comp rx=0 tx=0).stop 2023-03-03T09:01:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb23a6bc700 1 -- 172.21.15.7:0/613863072 shutdown_connections 2023-03-03T09:01:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb23a6bc700 1 --2- 172.21.15.7:0/613863072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2340c1da0 0x7fb23406d780 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb23a6bc700 1 --2- 172.21.15.7:0/613863072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2341586c0 0x7fb23406d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb23a6bc700 1 -- 172.21.15.7:0/613863072 >> 172.21.15.7:0/613863072 conn(0x7fb23406c110 msgr2=0x7fb23406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:01:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb23a6bc700 1 -- 172.21.15.7:0/613863072 shutdown_connections 2023-03-03T09:01:16.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.058+0000 7fb23a6bc700 1 -- 172.21.15.7:0/613863072 wait complete. 2023-03-03T09:01:16.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.059+0000 7fb23a6bc700 1 Processor -- start 2023-03-03T09:01:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.060+0000 7fb23a6bc700 1 -- start start 2023-03-03T09:01:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.060+0000 7fb23a6bc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2340c1da0 0x7fb2340c0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.060+0000 7fb23a6bc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb23413a0f0 0x7fb2341ed040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.060+0000 7fb23a6bc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2341586c0 0x7fb2341ed580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.060+0000 7fb23a6bc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340aa450 con 0x7fb2341586c0 2023-03-03T09:01:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.060+0000 7fb23a6bc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb234151370 con 0x7fb2340c1da0 2023-03-03T09:01:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.060+0000 7fb23a6bc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb234156370 con 0x7fb23413a0f0 2023-03-03T09:01:16.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.060+0000 7fb238c59700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2341586c0 0x7fb2341ed580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:16.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.060+0000 7fb238c59700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2341586c0 0x7fb2341ed580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54140/0 (socket says 172.21.15.7:54140) 2023-03-03T09:01:16.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.060+0000 7fb238c59700 1 -- 172.21.15.7:0/1698621431 learned_addr learned my addr 172.21.15.7:0/1698621431 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:01:16.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb233fff700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2340c1da0 0x7fb2340c0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:16.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb2337fe700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb23413a0f0 0x7fb2341ed040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:16.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb238c59700 1 -- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb23413a0f0 msgr2=0x7fb2341ed040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:16.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb238c59700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb23413a0f0 0x7fb2341ed040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:16.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb238c59700 1 -- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2340c1da0 msgr2=0x7fb2340c0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:16.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb238c59700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2340c1da0 0x7fb2340c0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:16.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb238c59700 1 -- 172.21.15.7:0/1698621431 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c009b30 con 0x7fb2341586c0 2023-03-03T09:01:16.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb238c59700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2341586c0 0x7fb2341ed580 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb22c012000 tx=0x7fb22c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:01:16.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb230ff9700 1 -- 172.21.15.7:0/1698621431 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c004800 con 0x7fb2341586c0 2023-03-03T09:01:16.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb230ff9700 1 -- 172.21.15.7:0/1698621431 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c004960 con 0x7fb2341586c0 2023-03-03T09:01:16.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb23a6bc700 1 -- 172.21.15.7:0/1698621431 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23406e1a0 con 0x7fb2341586c0 2023-03-03T09:01:16.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb230ff9700 1 -- 172.21.15.7:0/1698621431 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c01c680 con 0x7fb2341586c0 2023-03-03T09:01:16.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.061+0000 7fb23a6bc700 1 -- 172.21.15.7:0/1698621431 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2341581a0 con 0x7fb2341586c0 2023-03-03T09:01:16.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.062+0000 7fb23a6bc700 1 -- 172.21.15.7:0/1698621431 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23406e330 con 0x7fb2341586c0 2023-03-03T09:01:16.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.063+0000 7fb230ff9700 1 -- 172.21.15.7:0/1698621431 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb22c02c070 con 0x7fb2341586c0 2023-03-03T09:01:16.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.063+0000 7fb230ff9700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2200682f0 0x7fb22006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:01:16.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.063+0000 7fb230ff9700 1 -- 172.21.15.7:0/1698621431 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb22c087bf0 con 0x7fb2341586c0 2023-03-03T09:01:16.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.063+0000 7fb233fff700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2200682f0 0x7fb22006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:01:16.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.064+0000 7fb233fff700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2200682f0 0x7fb22006a7a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb224007fd0 tx=0x7fb224009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:01:16.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.067+0000 7fb230ff9700 1 -- 172.21.15.7:0/1698621431 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb22c053f10 con 0x7fb2341586c0 2023-03-03T09:01:16.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.304+0000 7fb23a6bc700 1 -- 172.21.15.7:0/1698621431 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb234074bb0 con 0x7fb2341586c0 2023-03-03T09:01:16.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.305+0000 7fb230ff9700 1 -- 172.21.15.7:0/1698621431 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb22c057530 con 0x7fb2341586c0 2023-03-03T09:01:16.306 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:01:16.307 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:01:16.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 -- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2200682f0 msgr2=0x7fb22006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:16.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2200682f0 0x7fb22006a7a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb224007fd0 tx=0x7fb224009040 comp rx=0 tx=0).stop 2023-03-03T09:01:16.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 -- 172.21.15.7:0/1698621431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2341586c0 msgr2=0x7fb2341ed580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:01:16.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2341586c0 0x7fb2341ed580 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb22c012000 tx=0x7fb22c004ad0 comp rx=0 tx=0).stop 2023-03-03T09:01:16.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 -- 172.21.15.7:0/1698621431 shutdown_connections 2023-03-03T09:01:16.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2200682f0 0x7fb22006a7a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:16.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2341586c0 0x7fb2341ed580 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:16.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2340c1da0 0x7fb2340c0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:16.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 --2- 172.21.15.7:0/1698621431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb23413a0f0 0x7fb2341ed040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:01:16.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 -- 172.21.15.7:0/1698621431 >> 172.21.15.7:0/1698621431 conn(0x7fb23406c110 msgr2=0x7fb234137150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:01:16.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 -- 172.21.15.7:0/1698621431 shutdown_connections 2023-03-03T09:01:16.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:01:16.307+0000 7fb23a6bc700 1 -- 172.21.15.7:0/1698621431 wait complete. 2023-03-03T09:01:16.310 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:01:16.320 INFO:tasks.mds_thrash.fs.[a]:waiting for 45 secs before thrashing 2023-03-03T09:01:39.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:01:39.397 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:01.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:02:01.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.461+0000 7ff6e1675700 1 Processor -- start 2023-03-03T09:02:01.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.461+0000 7ff6e1675700 1 -- start start 2023-03-03T09:02:01.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.462+0000 7ff6e1675700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6dc151d40 0x7ff6dc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:01.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.462+0000 7ff6e1675700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 0x7ff6dc157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:01.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.462+0000 7ff6e1675700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6dc06d430 con 0x7ff6dc073c30 2023-03-03T09:02:01.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.462+0000 7ff6e1675700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6dc06c2a0 con 0x7ff6dc151d40 2023-03-03T09:02:01.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.462+0000 7ff6e1675700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc1586e0 con 0x7ff6dc0c2930 2023-03-03T09:02:01.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.462+0000 7ff6daffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6dc151d40 0x7ff6dc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:01.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.462+0000 7ff6daffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6dc151d40 0x7ff6dc0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54164/0 (socket says 172.21.15.7:54164) 2023-03-03T09:02:01.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.462+0000 7ff6daffd700 1 -- 172.21.15.7:0/1073354092 learned_addr learned my addr 172.21.15.7:0/1073354092 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:02:01.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.462+0000 7ff6da7fc700 1 --2- 172.21.15.7:0/1073354092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 0x7ff6dc157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:01.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6daffd700 1 -- 172.21.15.7:0/1073354092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 msgr2=0x7ff6dc157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:02:01.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6daffd700 1 --2- 172.21.15.7:0/1073354092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 0x7ff6dc157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:01.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6daffd700 1 -- 172.21.15.7:0/1073354092 >> v1:172.21.15.143:6789/0 conn(0x7ff6dc073c30 legacy=0x7ff6dc157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:01.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6daffd700 1 -- 172.21.15.7:0/1073354092 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc06cb00 con 0x7ff6dc151d40 2023-03-03T09:02:01.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6da7fc700 1 --2- 172.21.15.7:0/1073354092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 0x7ff6dc157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:02:01.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6daffd700 1 --2- 172.21.15.7:0/1073354092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc151d40 0x7ff6dc0736f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00098f0 tx=0x7ff6d0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0924151c1c31a6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:01.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6d9ffb700 1 -- 172.21.15.7:0/1073354092 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d000b8a0 con 0x7ff6dc151d40 2023-03-03T09:02:01.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6d9ffb700 1 -- 172.21.15.7:0/1073354092 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d000ba00 con 0x7ff6dc151d40 2023-03-03T09:02:01.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6d9ffb700 1 -- 172.21.15.7:0/1073354092 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d000bd90 con 0x7ff6dc151d40 2023-03-03T09:02:01.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6e1675700 1 -- 172.21.15.7:0/1073354092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc151d40 msgr2=0x7ff6dc0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:01.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6e1675700 1 --2- 172.21.15.7:0/1073354092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc151d40 0x7ff6dc0736f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00098f0 tx=0x7ff6d0009c00 comp rx=0 tx=0).stop 2023-03-03T09:02:01.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6e1675700 1 -- 172.21.15.7:0/1073354092 shutdown_connections 2023-03-03T09:02:01.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6e1675700 1 --2- 172.21.15.7:0/1073354092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc151d40 0x7ff6dc0736f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:01.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6e1675700 1 --2- 172.21.15.7:0/1073354092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 0x7ff6dc157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:01.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.463+0000 7ff6e1675700 1 -- 172.21.15.7:0/1073354092 >> 172.21.15.7:0/1073354092 conn(0x7ff6dc0a94d0 msgr2=0x7ff6dc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:02:01.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.464+0000 7ff6e1675700 1 -- 172.21.15.7:0/1073354092 shutdown_connections 2023-03-03T09:02:01.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.464+0000 7ff6e1675700 1 -- 172.21.15.7:0/1073354092 wait complete. 2023-03-03T09:02:01.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.465+0000 7ff6e1675700 1 Processor -- start 2023-03-03T09:02:01.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.465+0000 7ff6e1675700 1 -- start start 2023-03-03T09:02:01.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6e1675700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc073c30 0x7ff6dc14ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:01.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6e1675700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 0x7ff6dc1ea310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:01.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6e1675700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6dc151d40 0x7ff6dc14d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:01.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6e1675700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc1e9ef0 con 0x7ff6dc073c30 2023-03-03T09:02:01.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6e1675700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc1523e0 con 0x7ff6dc151d40 2023-03-03T09:02:01.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6e1675700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc1ed270 con 0x7ff6dc0c2930 2023-03-03T09:02:01.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6daffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc073c30 0x7ff6dc14ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:01.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6daffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc073c30 0x7ff6dc14ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54170/0 (socket says 172.21.15.7:54170) 2023-03-03T09:02:01.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6daffd700 1 -- 172.21.15.7:0/533008386 learned_addr learned my addr 172.21.15.7:0/533008386 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:02:01.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6da7fc700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 0x7ff6dc1ea310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:01.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6db7fe700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6dc151d40 0x7ff6dc14d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:01.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6daffd700 1 -- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 msgr2=0x7ff6dc1ea310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:01.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6daffd700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 0x7ff6dc1ea310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:01.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6daffd700 1 -- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6dc151d40 msgr2=0x7ff6dc14d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:01.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6daffd700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6dc151d40 0x7ff6dc14d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:01.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6daffd700 1 -- 172.21.15.7:0/533008386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d0009550 con 0x7ff6dc073c30 2023-03-03T09:02:01.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6db7fe700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6dc151d40 0x7ff6dc14d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:02:01.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.466+0000 7ff6daffd700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc073c30 0x7ff6dc14ddb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009c40 tx=0x7ff6d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:01.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.467+0000 7ff6c37fe700 1 -- 172.21.15.7:0/533008386 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0004590 con 0x7ff6dc073c30 2023-03-03T09:02:01.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.467+0000 7ff6c37fe700 1 -- 172.21.15.7:0/533008386 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d00036a0 con 0x7ff6dc073c30 2023-03-03T09:02:01.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.467+0000 7ff6e1675700 1 -- 172.21.15.7:0/533008386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc06d0c0 con 0x7ff6dc073c30 2023-03-03T09:02:01.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.467+0000 7ff6c37fe700 1 -- 172.21.15.7:0/533008386 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d0003a50 con 0x7ff6dc073c30 2023-03-03T09:02:01.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.467+0000 7ff6e1675700 1 -- 172.21.15.7:0/533008386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc06cde0 con 0x7ff6dc073c30 2023-03-03T09:02:01.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.468+0000 7ff6e1675700 1 -- 172.21.15.7:0/533008386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc06bea0 con 0x7ff6dc073c30 2023-03-03T09:02:01.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.468+0000 7ff6c37fe700 1 -- 172.21.15.7:0/533008386 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff6d000f050 con 0x7ff6dc073c30 2023-03-03T09:02:01.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.468+0000 7ff6c37fe700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6c80682f0 0x7ff6c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:01.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.469+0000 7ff6c37fe700 1 -- 172.21.15.7:0/533008386 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff6d0087740 con 0x7ff6dc073c30 2023-03-03T09:02:01.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.469+0000 7ff6da7fc700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6c80682f0 0x7ff6c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:01.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.469+0000 7ff6da7fc700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6c80682f0 0x7ff6c806a7a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc151720 tx=0x7ff6cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:01.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.473+0000 7ff6c37fe700 1 -- 172.21.15.7:0/533008386 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff6d0053ae0 con 0x7ff6dc073c30 2023-03-03T09:02:01.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.709+0000 7ff6e1675700 1 -- 172.21.15.7:0/533008386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6dc1e9970 con 0x7ff6dc073c30 2023-03-03T09:02:01.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.710+0000 7ff6c37fe700 1 -- 172.21.15.7:0/533008386 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff6dc1e9970 con 0x7ff6dc073c30 2023-03-03T09:02:01.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:02:01.711 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:02:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.711+0000 7ff6e1675700 1 -- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6c80682f0 msgr2=0x7ff6c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.711+0000 7ff6e1675700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6c80682f0 0x7ff6c806a7a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc151720 tx=0x7ff6cc008040 comp rx=0 tx=0).stop 2023-03-03T09:02:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.711+0000 7ff6e1675700 1 -- 172.21.15.7:0/533008386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc073c30 msgr2=0x7ff6dc14ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.711+0000 7ff6e1675700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc073c30 0x7ff6dc14ddb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009c40 tx=0x7ff6d0004750 comp rx=0 tx=0).stop 2023-03-03T09:02:01.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.711+0000 7ff6e1675700 1 -- 172.21.15.7:0/533008386 shutdown_connections 2023-03-03T09:02:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.711+0000 7ff6e1675700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6dc151d40 0x7ff6dc14d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.711+0000 7ff6e1675700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6c80682f0 0x7ff6c806a7a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.711+0000 7ff6e1675700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6dc073c30 0x7ff6dc14ddb0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.711+0000 7ff6e1675700 1 --2- 172.21.15.7:0/533008386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6dc0c2930 0x7ff6dc1ea310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.711+0000 7ff6e1675700 1 -- 172.21.15.7:0/533008386 >> 172.21.15.7:0/533008386 conn(0x7ff6dc0a94d0 msgr2=0x7ff6dc06af00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:02:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.712+0000 7ff6e1675700 1 -- 172.21.15.7:0/533008386 shutdown_connections 2023-03-03T09:02:01.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:01.712+0000 7ff6e1675700 1 -- 172.21.15.7:0/533008386 wait complete. 2023-03-03T09:02:01.715 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:02:01.725 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before thrashing 2023-03-03T09:02:02.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:02:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.198+0000 7f5c3f770700 1 Processor -- start 2023-03-03T09:02:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.198+0000 7f5c3f770700 1 -- start start 2023-03-03T09:02:02.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.199+0000 7f5c3f770700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5c3813a0f0 0x7f5c380c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:02.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.199+0000 7f5c3f770700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c38154560 con 0x7f5c380c1da0 2023-03-03T09:02:02.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.199+0000 7f5c3f770700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c38157be0 con 0x7f5c381586c0 2023-03-03T09:02:02.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.199+0000 7f5c3f770700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c38154840 con 0x7f5c3813a0f0 2023-03-03T09:02:02.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.199+0000 7f5c3dd0d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5c380c1da0 0x7f5c3815edd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48162/0 (socket says 172.21.15.7:48162) 2023-03-03T09:02:02.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.199+0000 7f5c3dd0d700 1 -- 172.21.15.7:0/1590518385 learned_addr learned my addr 172.21.15.7:0/1590518385 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:02:02.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.199+0000 7f5c3d50c700 1 --2- 172.21.15.7:0/1590518385 >> v2:172.21.15.143:3300/0 conn(0x7f5c3813a0f0 0x7f5c380c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:02.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.199+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898761018 0 0) 0x7f5c38154560 con 0x7f5c380c1da0 2023-03-03T09:02:02.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c240035b0 con 0x7f5c380c1da0 2023-03-03T09:02:02.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540981900 0 0) 0x7f5c240035b0 con 0x7f5c380c1da0 2023-03-03T09:02:02.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 >> v2:172.21.15.143:3300/0 conn(0x7f5c3813a0f0 msgr2=0x7f5c380c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:02:02.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 --2- 172.21.15.7:0/1590518385 >> v2:172.21.15.143:3300/0 conn(0x7f5c3813a0f0 0x7f5c380c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 >> v1:172.21.15.143:6789/0 conn(0x7f5c381586c0 legacy=0x7f5c3806d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:02.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c38157ec0 con 0x7f5c380c1da0 2023-03-03T09:02:02.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f5c300019f0 con 0x7f5c380c1da0 2023-03-03T09:02:02.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 >> v1:172.21.15.7:6789/0 conn(0x7f5c380c1da0 legacy=0x7f5c3815edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:02.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c24003a70 0x7f5c24003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:02.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c24004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:02.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c24005320 0x7f5c2400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:02.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c38154840 con 0x7f5c24004850 2023-03-03T09:02:02.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2400c230 con 0x7f5c24003a70 2023-03-03T09:02:02.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2400c3f0 con 0x7f5c24005320 2023-03-03T09:02:02.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3cd0b700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c24004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:02.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3d50c700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c24003a70 0x7f5c24003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:02.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3dd0d700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c24005320 0x7f5c2400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:02.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3cd0b700 1 -- 172.21.15.7:0/1590518385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c24005320 msgr2=0x7f5c2400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:02:02.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3cd0b700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c24005320 0x7f5c2400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3cd0b700 1 -- 172.21.15.7:0/1590518385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c24003a70 msgr2=0x7f5c24003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:02:02.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3cd0b700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c24003a70 0x7f5c24003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3cd0b700 1 -- 172.21.15.7:0/1590518385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c240035b0 con 0x7f5c24004850 2023-03-03T09:02:02.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3dd0d700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c24005320 0x7f5c2400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:02:02.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3dd0d700 1 -- 172.21.15.7:0/1590518385 reap_dead start 2023-03-03T09:02:02.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.200+0000 7f5c3cd0b700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c24004cc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5c280017c0 tx=0x7f5c24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7623b0ac69bb18ca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:02.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.201+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c280052d0 con 0x7f5c24004850 2023-03-03T09:02:02.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.201+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c28005430 con 0x7f5c24004850 2023-03-03T09:02:02.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.201+0000 7f5c2ffff700 1 -- 172.21.15.7:0/1590518385 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c280038c0 con 0x7f5c24004850 2023-03-03T09:02:02.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.201+0000 7f5c3f770700 1 -- 172.21.15.7:0/1590518385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 msgr2=0x7f5c24004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:02.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.201+0000 7f5c3f770700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c24004cc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5c280017c0 tx=0x7f5c24005880 comp rx=0 tx=0).stop 2023-03-03T09:02:02.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.201+0000 7f5c3f770700 1 -- 172.21.15.7:0/1590518385 shutdown_connections 2023-03-03T09:02:02.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.201+0000 7f5c3f770700 1 --2- 172.21.15.7:0/1590518385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c24004cc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.201+0000 7f5c3f770700 1 -- 172.21.15.7:0/1590518385 >> 172.21.15.7:0/1590518385 conn(0x7f5c3806c110 msgr2=0x7f5c3806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:02:02.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.201+0000 7f5c3f770700 1 -- 172.21.15.7:0/1590518385 shutdown_connections 2023-03-03T09:02:02.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.201+0000 7f5c3f770700 1 -- 172.21.15.7:0/1590518385 wait complete. 2023-03-03T09:02:02.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.202+0000 7f5c3f770700 1 Processor -- start 2023-03-03T09:02:02.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.202+0000 7f5c3f770700 1 -- start start 2023-03-03T09:02:02.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3f770700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c38133ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:02.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3f770700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c380c6100 0x7f5c381ed000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:02.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3f770700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c381513e0 0x7f5c38151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:02.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3f770700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380aa500 con 0x7f5c24004850 2023-03-03T09:02:02.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3f770700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380aac70 con 0x7f5c380c6100 2023-03-03T09:02:02.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3f770700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380c5200 con 0x7f5c381513e0 2023-03-03T09:02:02.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3d50c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c38133ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:02.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3d50c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c38133ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54190/0 (socket says 172.21.15.7:54190) 2023-03-03T09:02:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3d50c700 1 -- 172.21.15.7:0/3195955907 learned_addr learned my addr 172.21.15.7:0/3195955907 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:02:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3cd0b700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c380c6100 0x7f5c381ed000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.203+0000 7f5c3dd0d700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c381513e0 0x7f5c38151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c3d50c700 1 -- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c381513e0 msgr2=0x7f5c38151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c3d50c700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c381513e0 0x7f5c38151860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c3d50c700 1 -- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c380c6100 msgr2=0x7f5c381ed000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c3d50c700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c380c6100 0x7f5c381ed000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c3d50c700 1 -- 172.21.15.7:0/3195955907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3806e1a0 con 0x7f5c24004850 2023-03-03T09:02:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c3dd0d700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c381513e0 0x7f5c38151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:02:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c3d50c700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c38133ed0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5c34011aa0 tx=0x7f5c34011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c2dffb700 1 -- 172.21.15.7:0/3195955907 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3400f940 con 0x7f5c24004850 2023-03-03T09:02:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c2dffb700 1 -- 172.21.15.7:0/3195955907 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3400faa0 con 0x7f5c24004850 2023-03-03T09:02:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c3f770700 1 -- 172.21.15.7:0/3195955907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c381581a0 con 0x7f5c24004850 2023-03-03T09:02:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c2dffb700 1 -- 172.21.15.7:0/3195955907 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3400fe50 con 0x7f5c24004850 2023-03-03T09:02:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.204+0000 7f5c3f770700 1 -- 172.21.15.7:0/3195955907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3806e330 con 0x7f5c24004850 2023-03-03T09:02:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.205+0000 7f5c3f770700 1 -- 172.21.15.7:0/3195955907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c380aaac0 con 0x7f5c24004850 2023-03-03T09:02:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.206+0000 7f5c2dffb700 1 -- 172.21.15.7:0/3195955907 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5c34014440 con 0x7f5c24004850 2023-03-03T09:02:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.206+0000 7f5c2dffb700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c1c0682f0 0x7f5c1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.206+0000 7f5c2dffb700 1 -- 172.21.15.7:0/3195955907 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5c34086ae0 con 0x7f5c24004850 2023-03-03T09:02:02.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.206+0000 7f5c3cd0b700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c1c0682f0 0x7f5c1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:02.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.207+0000 7f5c3cd0b700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c1c0682f0 0x7f5c1c06a7a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c28015040 tx=0x7f5c28009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:02.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.211+0000 7f5c2dffb700 1 -- 172.21.15.7:0/3195955907 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5c34052e20 con 0x7f5c24004850 2023-03-03T09:02:02.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.452+0000 7f5c3f770700 1 -- 172.21.15.7:0/3195955907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c38134410 con 0x7f5c24004850 2023-03-03T09:02:02.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.452+0000 7f5c2dffb700 1 -- 172.21.15.7:0/3195955907 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5c34056440 con 0x7f5c24004850 2023-03-03T09:02:02.454 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:02:02.454 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:02:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 -- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c1c0682f0 msgr2=0x7f5c1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c1c0682f0 0x7f5c1c06a7a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c28015040 tx=0x7f5c28009030 comp rx=0 tx=0).stop 2023-03-03T09:02:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 -- 172.21.15.7:0/3195955907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 msgr2=0x7f5c38133ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c38133ed0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5c34011aa0 tx=0x7f5c34011db0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 -- 172.21.15.7:0/3195955907 shutdown_connections 2023-03-03T09:02:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c381513e0 0x7f5c38151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c1c0682f0 0x7f5c1c06a7a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c24004850 0x7f5c38133ed0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 --2- 172.21.15.7:0/3195955907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c380c6100 0x7f5c381ed000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:02.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 -- 172.21.15.7:0/3195955907 >> 172.21.15.7:0/3195955907 conn(0x7f5c3806c110 msgr2=0x7f5c38139120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:02:02.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.454+0000 7f5c3f770700 1 -- 172.21.15.7:0/3195955907 shutdown_connections 2023-03-03T09:02:02.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:02.455+0000 7f5c3f770700 1 -- 172.21.15.7:0/3195955907 wait complete. 2023-03-03T09:02:02.458 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:02:02.468 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2023-03-03T09:02:04.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:02:04.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.861+0000 7fc1ca518700 1 Processor -- start 2023-03-03T09:02:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.861+0000 7fc1ca518700 1 -- start start 2023-03-03T09:02:04.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.862+0000 7fc1ca518700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c414b0d0 0x7fc1c4071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:04.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.862+0000 7fc1ca518700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1c40aad20 0x7fc1c4157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:04.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.862+0000 7fc1ca518700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c40a9c20 con 0x7fc1c4071d40 2023-03-03T09:02:04.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.862+0000 7fc1ca518700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c4154f10 con 0x7fc1c40aad20 2023-03-03T09:02:04.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.862+0000 7fc1ca518700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40ac3c0 con 0x7fc1c414b0d0 2023-03-03T09:02:04.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.862+0000 7fc1c37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1c40aad20 0x7fc1c4157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:04.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.862+0000 7fc1c37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1c40aad20 0x7fc1c4157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54200/0 (socket says 172.21.15.7:54200) 2023-03-03T09:02:04.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.862+0000 7fc1c37fe700 1 -- 172.21.15.7:0/719582111 learned_addr learned my addr 172.21.15.7:0/719582111 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:02:04.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.862+0000 7fc1c3fff700 1 --2- 172.21.15.7:0/719582111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c414b0d0 0x7fc1c4071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:04.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1c37fe700 1 -- 172.21.15.7:0/719582111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c414b0d0 msgr2=0x7fc1c4071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:02:04.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1c37fe700 1 --2- 172.21.15.7:0/719582111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c414b0d0 0x7fc1c4071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:04.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1c37fe700 1 -- 172.21.15.7:0/719582111 >> v1:172.21.15.143:6789/0 conn(0x7fc1c4071d40 legacy=0x7fc1c4158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:04.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1c37fe700 1 -- 172.21.15.7:0/719582111 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c406fc70 con 0x7fc1c40aad20 2023-03-03T09:02:04.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1c37fe700 1 --2- 172.21.15.7:0/719582111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c40aad20 0x7fc1c4157df0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0098f0 tx=0x7fc1ac009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a7edce9f3cb21c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:04.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1c2ffd700 1 -- 172.21.15.7:0/719582111 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac011830 con 0x7fc1c40aad20 2023-03-03T09:02:04.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1c2ffd700 1 -- 172.21.15.7:0/719582111 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac011990 con 0x7fc1c40aad20 2023-03-03T09:02:04.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1c2ffd700 1 -- 172.21.15.7:0/719582111 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac011d20 con 0x7fc1c40aad20 2023-03-03T09:02:04.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1ca518700 1 -- 172.21.15.7:0/719582111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c40aad20 msgr2=0x7fc1c4157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:04.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1ca518700 1 --2- 172.21.15.7:0/719582111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c40aad20 0x7fc1c4157df0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0098f0 tx=0x7fc1ac009c00 comp rx=0 tx=0).stop 2023-03-03T09:02:04.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1ca518700 1 -- 172.21.15.7:0/719582111 shutdown_connections 2023-03-03T09:02:04.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1ca518700 1 --2- 172.21.15.7:0/719582111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c414b0d0 0x7fc1c4071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:04.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1ca518700 1 --2- 172.21.15.7:0/719582111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c40aad20 0x7fc1c4157df0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:04.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.863+0000 7fc1ca518700 1 -- 172.21.15.7:0/719582111 >> 172.21.15.7:0/719582111 conn(0x7fc1c40c8900 msgr2=0x7fc1c40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:02:04.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.864+0000 7fc1ca518700 1 -- 172.21.15.7:0/719582111 shutdown_connections 2023-03-03T09:02:04.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.864+0000 7fc1ca518700 1 -- 172.21.15.7:0/719582111 wait complete. 2023-03-03T09:02:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.865+0000 7fc1ca518700 1 Processor -- start 2023-03-03T09:02:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.865+0000 7fc1ca518700 1 -- start start 2023-03-03T09:02:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1ca518700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c4071d40 0x7fc1c4142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1ca518700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c40aad20 0x7fc1c41431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1ca518700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c414b0d0 0x7fc1c4152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1ca518700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c4153e80 con 0x7fc1c414b0d0 2023-03-03T09:02:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1ca518700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c41444d0 con 0x7fc1c40aad20 2023-03-03T09:02:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1ca518700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c41edd70 con 0x7fc1c4071d40 2023-03-03T09:02:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c8ab5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c414b0d0 0x7fc1c4152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c8ab5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c414b0d0 0x7fc1c4152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54208/0 (socket says 172.21.15.7:54208) 2023-03-03T09:02:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c8ab5700 1 -- 172.21.15.7:0/2158367043 learned_addr learned my addr 172.21.15.7:0/2158367043 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:02:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c37fe700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c40aad20 0x7fc1c41431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c3fff700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c4071d40 0x7fc1c4142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c8ab5700 1 -- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c4071d40 msgr2=0x7fc1c4142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c8ab5700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c4071d40 0x7fc1c4142c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c8ab5700 1 -- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c40aad20 msgr2=0x7fc1c41431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c8ab5700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c40aad20 0x7fc1c41431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c8ab5700 1 -- 172.21.15.7:0/2158367043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac009550 con 0x7fc1c414b0d0 2023-03-03T09:02:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c37fe700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c40aad20 0x7fc1c41431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:02:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c3fff700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c4071d40 0x7fc1c4142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:02:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c8ab5700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c414b0d0 0x7fc1c4152220 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00cb50 tx=0x7fc1bc00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c0ff9700 1 -- 172.21.15.7:0/2158367043 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc003c00 con 0x7fc1c414b0d0 2023-03-03T09:02:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c0ff9700 1 -- 172.21.15.7:0/2158367043 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc003d60 con 0x7fc1c414b0d0 2023-03-03T09:02:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1ca518700 1 -- 172.21.15.7:0/2158367043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c406f6b0 con 0x7fc1c414b0d0 2023-03-03T09:02:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1c0ff9700 1 -- 172.21.15.7:0/2158367043 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc00ab70 con 0x7fc1c414b0d0 2023-03-03T09:02:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.866+0000 7fc1ca518700 1 -- 172.21.15.7:0/2158367043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c4155160 con 0x7fc1c414b0d0 2023-03-03T09:02:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.867+0000 7fc1ca518700 1 -- 172.21.15.7:0/2158367043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c406f990 con 0x7fc1c414b0d0 2023-03-03T09:02:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.868+0000 7fc1c0ff9700 1 -- 172.21.15.7:0/2158367043 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc1bc010050 con 0x7fc1c414b0d0 2023-03-03T09:02:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.868+0000 7fc1c0ff9700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a40682f0 0x7fc1a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:02:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.868+0000 7fc1c0ff9700 1 -- 172.21.15.7:0/2158367043 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc1bc0861e0 con 0x7fc1c414b0d0 2023-03-03T09:02:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.869+0000 7fc1c3fff700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a40682f0 0x7fc1a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:02:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.869+0000 7fc1c3fff700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a40682f0 0x7fc1a406a7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009b60 tx=0x7fc1b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:02:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:04.873+0000 7fc1c0ff9700 1 -- 172.21.15.7:0/2158367043 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc1bc052490 con 0x7fc1c414b0d0 2023-03-03T09:02:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.111+0000 7fc1ca518700 1 -- 172.21.15.7:0/2158367043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1c4074bb0 con 0x7fc1c414b0d0 2023-03-03T09:02:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.111+0000 7fc1c0ff9700 1 -- 172.21.15.7:0/2158367043 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc1bc055ab0 con 0x7fc1c414b0d0 2023-03-03T09:02:05.113 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:02:05.113 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:02:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.113+0000 7fc1ca518700 1 -- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a40682f0 msgr2=0x7fc1a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.113+0000 7fc1ca518700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a40682f0 0x7fc1a406a7a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009b60 tx=0x7fc1b4008040 comp rx=0 tx=0).stop 2023-03-03T09:02:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.113+0000 7fc1ca518700 1 -- 172.21.15.7:0/2158367043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c414b0d0 msgr2=0x7fc1c4152220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:02:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.113+0000 7fc1ca518700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c414b0d0 0x7fc1c4152220 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00cb50 tx=0x7fc1bc00cf10 comp rx=0 tx=0).stop 2023-03-03T09:02:05.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.113+0000 7fc1ca518700 1 -- 172.21.15.7:0/2158367043 shutdown_connections 2023-03-03T09:02:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.113+0000 7fc1ca518700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a40682f0 0x7fc1a406a7a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.113+0000 7fc1ca518700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c414b0d0 0x7fc1c4152220 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.113+0000 7fc1ca518700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c4071d40 0x7fc1c4142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.113+0000 7fc1ca518700 1 --2- 172.21.15.7:0/2158367043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c40aad20 0x7fc1c41431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:02:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.113+0000 7fc1ca518700 1 -- 172.21.15.7:0/2158367043 >> 172.21.15.7:0/2158367043 conn(0x7fc1c40c8900 msgr2=0x7fc1c40c17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:02:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.114+0000 7fc1ca518700 1 -- 172.21.15.7:0/2158367043 shutdown_connections 2023-03-03T09:02:05.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:02:05.114+0000 7fc1ca518700 1 -- 172.21.15.7:0/2158367043 wait complete. 2023-03-03T09:02:05.117 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:02:05.127 INFO:tasks.mds_thrash.fs.[a]:waiting for 92 secs before thrashing 2023-03-03T09:02:09.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:09.464 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:39.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:02:39.534 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:09.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:09.634 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:37.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:03:37.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.266+0000 7f5a3b697700 1 Processor -- start 2023-03-03T09:03:37.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.266+0000 7f5a3b697700 1 -- start start 2023-03-03T09:03:37.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.267+0000 7f5a3b697700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5a340aad20 0x7f5a34157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:37.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.267+0000 7f5a3b697700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a340aa030 0x7f5a341581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:37.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.267+0000 7f5a3b697700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a340728b0 con 0x7f5a3414b020 2023-03-03T09:03:37.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.267+0000 7f5a3b697700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a34155070 con 0x7f5a340aa030 2023-03-03T09:03:37.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.267+0000 7f5a3b697700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a3406f5b0 con 0x7f5a340aad20 2023-03-03T09:03:37.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.267+0000 7f5a39c34700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a340aa030 0x7f5a341581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:37.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.267+0000 7f5a39c34700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a340aa030 0x7f5a341581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54270/0 (socket says 172.21.15.7:54270) 2023-03-03T09:03:37.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.267+0000 7f5a39c34700 1 -- 172.21.15.7:0/2885763541 learned_addr learned my addr 172.21.15.7:0/2885763541 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:03:37.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a38c32700 1 --2- 172.21.15.7:0/2885763541 >> v2:172.21.15.143:3300/0 conn(0x7f5a340aad20 0x7f5a34157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:37.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a39c34700 1 -- 172.21.15.7:0/2885763541 >> v2:172.21.15.143:3300/0 conn(0x7f5a340aad20 msgr2=0x7f5a34157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:03:37.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a39c34700 1 --2- 172.21.15.7:0/2885763541 >> v2:172.21.15.143:3300/0 conn(0x7f5a340aad20 0x7f5a34157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a39c34700 1 -- 172.21.15.7:0/2885763541 >> v1:172.21.15.143:6789/0 conn(0x7f5a3414b020 legacy=0x7f5a340717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a39c34700 1 -- 172.21.15.7:0/2885763541 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3406f890 con 0x7f5a340aa030 2023-03-03T09:03:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a39c34700 1 --2- 172.21.15.7:0/2885763541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a340aa030 0x7f5a341581e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5a340a98d0 tx=0x7f5a2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1dbe92a5212c1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:03:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a2bfff700 1 -- 172.21.15.7:0/2885763541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00e070 con 0x7f5a340aa030 2023-03-03T09:03:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a2bfff700 1 -- 172.21.15.7:0/2885763541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c00b9c0 con 0x7f5a340aa030 2023-03-03T09:03:37.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a2bfff700 1 -- 172.21.15.7:0/2885763541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00bd70 con 0x7f5a340aa030 2023-03-03T09:03:37.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a3b697700 1 -- 172.21.15.7:0/2885763541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a340aa030 msgr2=0x7f5a341581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:37.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.268+0000 7f5a3b697700 1 --2- 172.21.15.7:0/2885763541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a340aa030 0x7f5a341581e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5a340a98d0 tx=0x7f5a2c00b330 comp rx=0 tx=0).stop 2023-03-03T09:03:37.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.269+0000 7f5a3b697700 1 -- 172.21.15.7:0/2885763541 shutdown_connections 2023-03-03T09:03:37.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.269+0000 7f5a3b697700 1 --2- 172.21.15.7:0/2885763541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a340aa030 0x7f5a341581e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:37.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.269+0000 7f5a3b697700 1 --2- 172.21.15.7:0/2885763541 >> v2:172.21.15.143:3300/0 conn(0x7f5a340aad20 0x7f5a34157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:37.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.269+0000 7f5a3b697700 1 -- 172.21.15.7:0/2885763541 >> 172.21.15.7:0/2885763541 conn(0x7f5a340c8900 msgr2=0x7f5a340c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:03:37.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.269+0000 7f5a3b697700 1 -- 172.21.15.7:0/2885763541 shutdown_connections 2023-03-03T09:03:37.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.269+0000 7f5a3b697700 1 -- 172.21.15.7:0/2885763541 wait complete. 2023-03-03T09:03:37.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.270+0000 7f5a3b697700 1 Processor -- start 2023-03-03T09:03:37.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.270+0000 7f5a3b697700 1 -- start start 2023-03-03T09:03:37.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.270+0000 7f5a3b697700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a340aa030 0x7f5a341501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:37.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.270+0000 7f5a3b697700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a340aad20 0x7f5a34157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:37.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a3b697700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3414b020 0x7f5a3414d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:37.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a3b697700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a34150f60 con 0x7f5a3414b020 2023-03-03T09:03:37.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a3b697700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a341ed160 con 0x7f5a340aa030 2023-03-03T09:03:37.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a3b697700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a341ed440 con 0x7f5a340aad20 2023-03-03T09:03:37.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a39c34700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3414b020 0x7f5a3414d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:37.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a39c34700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3414b020 0x7f5a3414d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54276/0 (socket says 172.21.15.7:54276) 2023-03-03T09:03:37.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a39c34700 1 -- 172.21.15.7:0/3526926319 learned_addr learned my addr 172.21.15.7:0/3526926319 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:03:37.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a38c32700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a340aad20 0x7f5a34157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:37.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a39433700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a340aa030 0x7f5a341501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:37.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a39c34700 1 -- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a340aad20 msgr2=0x7f5a34157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:37.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a39c34700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a340aad20 0x7f5a34157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:37.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a39c34700 1 -- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a340aa030 msgr2=0x7f5a341501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:37.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a39c34700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a340aa030 0x7f5a341501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:37.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a39c34700 1 -- 172.21.15.7:0/3526926319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c009b30 con 0x7f5a3414b020 2023-03-03T09:03:37.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a39c34700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3414b020 0x7f5a3414d510 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5a340c2b10 tx=0x7f5a2c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:03:37.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a29ffb700 1 -- 172.21.15.7:0/3526926319 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0099b0 con 0x7f5a3414b020 2023-03-03T09:03:37.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a29ffb700 1 -- 172.21.15.7:0/3526926319 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c0092e0 con 0x7f5a3414b020 2023-03-03T09:03:37.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a29ffb700 1 -- 172.21.15.7:0/3526926319 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c01b630 con 0x7f5a3414b020 2023-03-03T09:03:37.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.271+0000 7f5a3b697700 1 -- 172.21.15.7:0/3526926319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a34155350 con 0x7f5a3414b020 2023-03-03T09:03:37.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.272+0000 7f5a3b697700 1 -- 172.21.15.7:0/3526926319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3406fb70 con 0x7f5a3414b020 2023-03-03T09:03:37.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.272+0000 7f5a3b697700 1 -- 172.21.15.7:0/3526926319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a34071ef0 con 0x7f5a3414b020 2023-03-03T09:03:37.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.273+0000 7f5a29ffb700 1 -- 172.21.15.7:0/3526926319 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5a2c02b070 con 0x7f5a3414b020 2023-03-03T09:03:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.273+0000 7f5a29ffb700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a18068320 0x7f5a1806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.273+0000 7f5a29ffb700 1 -- 172.21.15.7:0/3526926319 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5a2c087910 con 0x7f5a3414b020 2023-03-03T09:03:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.274+0000 7f5a39433700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a18068320 0x7f5a1806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.274+0000 7f5a39433700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a18068320 0x7f5a1806a7d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5a24000f60 tx=0x7f5a24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:03:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.278+0000 7f5a29ffb700 1 -- 172.21.15.7:0/3526926319 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a2c053cb0 con 0x7f5a3414b020 2023-03-03T09:03:37.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.515+0000 7f5a3b697700 1 -- 172.21.15.7:0/3526926319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a3406f390 con 0x7f5a3414b020 2023-03-03T09:03:37.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.516+0000 7f5a29ffb700 1 -- 172.21.15.7:0/3526926319 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5a2c0572d0 con 0x7f5a3414b020 2023-03-03T09:03:37.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:03:37.518 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:03:37.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 -- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a18068320 msgr2=0x7f5a1806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:37.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a18068320 0x7f5a1806a7d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5a24000f60 tx=0x7f5a24009040 comp rx=0 tx=0).stop 2023-03-03T09:03:37.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 -- 172.21.15.7:0/3526926319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3414b020 msgr2=0x7f5a3414d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:37.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3414b020 0x7f5a3414d510 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5a340c2b10 tx=0x7f5a2c004ad0 comp rx=0 tx=0).stop 2023-03-03T09:03:37.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 -- 172.21.15.7:0/3526926319 shutdown_connections 2023-03-03T09:03:37.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a18068320 0x7f5a1806a7d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:37.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3414b020 0x7f5a3414d510 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:37.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a340aa030 0x7f5a341501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:37.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 --2- 172.21.15.7:0/3526926319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a340aad20 0x7f5a34157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:37.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 -- 172.21.15.7:0/3526926319 >> 172.21.15.7:0/3526926319 conn(0x7f5a340c8900 msgr2=0x7f5a340ac310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:03:37.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 -- 172.21.15.7:0/3526926319 shutdown_connections 2023-03-03T09:03:37.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:37.518+0000 7f5a3b697700 1 -- 172.21.15.7:0/3526926319 wait complete. 2023-03-03T09:03:37.521 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:03:37.531 INFO:tasks.mds_thrash.fs.[a]:waiting for 83 secs before thrashing 2023-03-03T09:03:39.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:03:39.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.607+0000 7f0b0361d700 1 Processor -- start 2023-03-03T09:03:39.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.607+0000 7f0b0361d700 1 -- start start 2023-03-03T09:03:39.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.608+0000 7f0b0361d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0afc155390 0x7f0afc0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.608+0000 7f0b0361d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0afc154910 con 0x7f0afc06e020 2023-03-03T09:03:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.608+0000 7f0b0361d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0afc1522e0 con 0x7f0afc151770 2023-03-03T09:03:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.608+0000 7f0b0361d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0afc070b90 con 0x7f0afc155390 2023-03-03T09:03:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.608+0000 7f0b00bb8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0afc06e020 0x7f0afc071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48264/0 (socket says 172.21.15.7:48264) 2023-03-03T09:03:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.608+0000 7f0b00bb8700 1 -- 172.21.15.7:0/557855501 learned_addr learned my addr 172.21.15.7:0/557855501 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:03:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.608+0000 7f0b013b9700 1 --2- 172.21.15.7:0/557855501 >> v2:172.21.15.143:3300/0 conn(0x7f0afc155390 0x7f0afc0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:39.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092870286 0 0) 0x7f0afc154910 con 0x7f0afc06e020 2023-03-03T09:03:39.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae80035b0 con 0x7f0afc06e020 2023-03-03T09:03:39.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3391497389 0 0) 0x7f0ae80035b0 con 0x7f0afc06e020 2023-03-03T09:03:39.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 >> v2:172.21.15.143:3300/0 conn(0x7f0afc155390 msgr2=0x7f0afc0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:03:39.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 --2- 172.21.15.7:0/557855501 >> v2:172.21.15.143:3300/0 conn(0x7f0afc155390 0x7f0afc0c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:39.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 >> v1:172.21.15.143:6789/0 conn(0x7f0afc151770 legacy=0x7f0afc151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:39.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc0702a0 con 0x7f0afc06e020 2023-03-03T09:03:39.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0af8001670 con 0x7f0afc06e020 2023-03-03T09:03:39.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0b013b9700 1 --2- 172.21.15.7:0/557855501 >> v2:172.21.15.143:3300/0 conn(0x7f0afc155390 0x7f0afc0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:03:39.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 >> v1:172.21.15.7:6789/0 conn(0x7f0afc06e020 legacy=0x7f0afc071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:39.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ae8003a70 0x7f0ae8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:39.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ae8004850 0x7f0ae8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:39.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ae8005250 0x7f0ae800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:39.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc070b90 con 0x7f0ae8004850 2023-03-03T09:03:39.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae800c190 con 0x7f0ae8003a70 2023-03-03T09:03:39.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae800c350 con 0x7f0ae8005250 2023-03-03T09:03:39.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0b00bb8700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ae8004850 0x7f0ae8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:39.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0b013b9700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ae8003a70 0x7f0ae8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:39.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0b01bba700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ae8005250 0x7f0ae800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:39.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0b00bb8700 1 -- 172.21.15.7:0/557855501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ae8005250 msgr2=0x7f0ae800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:03:39.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0b00bb8700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ae8005250 0x7f0ae800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:39.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0b00bb8700 1 -- 172.21.15.7:0/557855501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ae8003a70 msgr2=0x7f0ae8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:03:39.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0b00bb8700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ae8003a70 0x7f0ae8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:39.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0b00bb8700 1 -- 172.21.15.7:0/557855501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae80035b0 con 0x7f0ae8004850 2023-03-03T09:03:39.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.609+0000 7f0b01bba700 1 -- 172.21.15.7:0/557855501 reap_dead start 2023-03-03T09:03:39.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0b00bb8700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ae8004850 0x7f0ae8004cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0af800c270 tx=0x7f0ae8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55085c8ddbff266 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:03:39.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8007c10 con 0x7f0ae8004850 2023-03-03T09:03:39.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af800cdb0 con 0x7f0ae8004850 2023-03-03T09:03:39.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0af3fff700 1 -- 172.21.15.7:0/557855501 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8005ba0 con 0x7f0ae8004850 2023-03-03T09:03:39.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0b0361d700 1 -- 172.21.15.7:0/557855501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ae8004850 msgr2=0x7f0ae8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:39.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0b0361d700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ae8004850 0x7f0ae8004cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0af800c270 tx=0x7f0ae8005880 comp rx=0 tx=0).stop 2023-03-03T09:03:39.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0b0361d700 1 -- 172.21.15.7:0/557855501 shutdown_connections 2023-03-03T09:03:39.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0b0361d700 1 --2- 172.21.15.7:0/557855501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ae8004850 0x7f0ae8004cc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:39.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0b0361d700 1 -- 172.21.15.7:0/557855501 >> 172.21.15.7:0/557855501 conn(0x7f0afc06c110 msgr2=0x7f0afc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:03:39.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0b0361d700 1 -- 172.21.15.7:0/557855501 shutdown_connections 2023-03-03T09:03:39.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.610+0000 7f0b0361d700 1 -- 172.21.15.7:0/557855501 wait complete. 2023-03-03T09:03:39.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.611+0000 7f0b0361d700 1 Processor -- start 2023-03-03T09:03:39.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b0361d700 1 -- start start 2023-03-03T09:03:39.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b0361d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ae8004850 0x7f0afc1461b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:39.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b0361d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0afc152eb0 0x7f0afc071f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:39.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b0361d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc072470 0x7f0afc1e95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:39.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b0361d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc0c17a0 con 0x7f0afc152eb0 2023-03-03T09:03:39.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b0361d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc1466f0 con 0x7f0ae8004850 2023-03-03T09:03:39.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b0361d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0afc154790 con 0x7f0afc072470 2023-03-03T09:03:39.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b00bb8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0afc152eb0 0x7f0afc071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:39.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b00bb8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0afc152eb0 0x7f0afc071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54298/0 (socket says 172.21.15.7:54298) 2023-03-03T09:03:39.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b00bb8700 1 -- 172.21.15.7:0/4279724832 learned_addr learned my addr 172.21.15.7:0/4279724832 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:03:39.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b013b9700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ae8004850 0x7f0afc1461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:39.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.612+0000 7f0b01bba700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc072470 0x7f0afc1e95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:39.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0b00bb8700 1 -- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc072470 msgr2=0x7f0afc1e95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:39.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0b00bb8700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc072470 0x7f0afc1e95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:39.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0b00bb8700 1 -- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ae8004850 msgr2=0x7f0afc1461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:39.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0b00bb8700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ae8004850 0x7f0afc1461b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:39.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0b00bb8700 1 -- 172.21.15.7:0/4279724832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc070820 con 0x7f0afc152eb0 2023-03-03T09:03:39.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0b01bba700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc072470 0x7f0afc1e95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:03:39.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0b00bb8700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0afc152eb0 0x7f0afc071f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0af800c860 tx=0x7f0afc071920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:03:39.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0af1ffb700 1 -- 172.21.15.7:0/4279724832 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8001630 con 0x7f0afc152eb0 2023-03-03T09:03:39.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0af1ffb700 1 -- 172.21.15.7:0/4279724832 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af80070d0 con 0x7f0afc152eb0 2023-03-03T09:03:39.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0b0361d700 1 -- 172.21.15.7:0/4279724832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0afc070540 con 0x7f0afc152eb0 2023-03-03T09:03:39.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0af1ffb700 1 -- 172.21.15.7:0/4279724832 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8007440 con 0x7f0afc152eb0 2023-03-03T09:03:39.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.613+0000 7f0b0361d700 1 -- 172.21.15.7:0/4279724832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0afc155760 con 0x7f0afc152eb0 2023-03-03T09:03:39.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.614+0000 7f0b0361d700 1 -- 172.21.15.7:0/4279724832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0afc152d00 con 0x7f0afc152eb0 2023-03-03T09:03:39.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.615+0000 7f0af1ffb700 1 -- 172.21.15.7:0/4279724832 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0af8024030 con 0x7f0afc152eb0 2023-03-03T09:03:39.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.615+0000 7f0af1ffb700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ae00682f0 0x7f0ae006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:03:39.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.615+0000 7f0af1ffb700 1 -- 172.21.15.7:0/4279724832 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0af8087450 con 0x7f0afc152eb0 2023-03-03T09:03:39.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.616+0000 7f0b013b9700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ae00682f0 0x7f0ae006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:03:39.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.616+0000 7f0b013b9700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ae00682f0 0x7f0ae006a7a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0afc150070 tx=0x7f0aec009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:03:39.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.620+0000 7f0af1ffb700 1 -- 172.21.15.7:0/4279724832 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0af8053700 con 0x7f0afc152eb0 2023-03-03T09:03:39.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:39.702 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:03:39.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.860+0000 7f0b0361d700 1 -- 172.21.15.7:0/4279724832 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0afc074bb0 con 0x7f0afc152eb0 2023-03-03T09:03:39.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.861+0000 7f0af1ffb700 1 -- 172.21.15.7:0/4279724832 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0af8056d20 con 0x7f0afc152eb0 2023-03-03T09:03:39.862 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:03:39.862 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:03:39.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 -- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ae00682f0 msgr2=0x7f0ae006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:39.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ae00682f0 0x7f0ae006a7a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0afc150070 tx=0x7f0aec009210 comp rx=0 tx=0).stop 2023-03-03T09:03:39.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 -- 172.21.15.7:0/4279724832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0afc152eb0 msgr2=0x7f0afc071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:03:39.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0afc152eb0 0x7f0afc071f30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0af800c860 tx=0x7f0afc071920 comp rx=0 tx=0).stop 2023-03-03T09:03:39.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 -- 172.21.15.7:0/4279724832 shutdown_connections 2023-03-03T09:03:39.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0afc072470 0x7f0afc1e95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:39.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ae8004850 0x7f0afc1461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ae00682f0 0x7f0ae006a7a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 --2- 172.21.15.7:0/4279724832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0afc152eb0 0x7f0afc071f30 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:03:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 -- 172.21.15.7:0/4279724832 >> 172.21.15.7:0/4279724832 conn(0x7f0afc06c110 msgr2=0x7f0afc1dce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:03:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 -- 172.21.15.7:0/4279724832 shutdown_connections 2023-03-03T09:03:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:03:39.863+0000 7f0b0361d700 1 -- 172.21.15.7:0/4279724832 wait complete. 2023-03-03T09:03:39.866 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:03:39.876 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T09:04:09.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:09.766 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:39.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:04:39.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:00.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:05:00.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.669+0000 7f2cc4e8e700 1 Processor -- start 2023-03-03T09:05:00.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.669+0000 7f2cc4e8e700 1 -- start start 2023-03-03T09:05:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.670+0000 7f2cc4e8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cc00c1da0 0x7f2cc0139d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.670+0000 7f2cc4e8e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc00c7b90 con 0x7f2cc00c8020 2023-03-03T09:05:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.670+0000 7f2cc4e8e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc0154640 con 0x7f2cc006d4b0 2023-03-03T09:05:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.670+0000 7f2cc4e8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00c7330 con 0x7f2cc00c1da0 2023-03-03T09:05:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbdd9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2cc00c8020 0x7f2cc00c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48334/0 (socket says 172.21.15.7:48334) 2023-03-03T09:05:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbdd9b700 1 -- 172.21.15.7:0/1998760918 learned_addr learned my addr 172.21.15.7:0/1998760918 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:05:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027323696 0 0) 0x7f2cc00c7b90 con 0x7f2cc00c8020 2023-03-03T09:05:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca80035b0 con 0x7f2cc00c8020 2023-03-03T09:05:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbed9d700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cc00c1da0 0x7f2cc0139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3812982125 0 0) 0x7f2ca80035b0 con 0x7f2cc00c8020 2023-03-03T09:05:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cc00c1da0 msgr2=0x7f2cc0139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:05:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cc00c1da0 0x7f2cc0139d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 >> v1:172.21.15.143:6789/0 conn(0x7f2cc006d4b0 legacy=0x7f2cc006d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc00c7540 con 0x7f2cc00c8020 2023-03-03T09:05:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f2cb0001670 con 0x7f2cc00c8020 2023-03-03T09:05:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbed9d700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cc00c1da0 0x7f2cc0139d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:05:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 >> v1:172.21.15.7:6789/0 conn(0x7f2cc00c8020 legacy=0x7f2cc00c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cc00c1da0 0x7f2ca8003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca8004090 0x7f2ca8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ca8004d90 0x7f2ca8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc00c7330 con 0x7f2ca8004090 2023-03-03T09:05:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca800bc70 con 0x7f2ca8004d90 2023-03-03T09:05:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca800be30 con 0x7f2cc00c1da0 2023-03-03T09:05:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbdd9b700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca8004090 0x7f2ca8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbe59c700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cc00c1da0 0x7f2ca8003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:00.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.671+0000 7f2cbed9d700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ca8004d90 0x7f2ca8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:00.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.672+0000 7f2cbdd9b700 1 -- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cc00c1da0 msgr2=0x7f2ca8003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:05:00.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.672+0000 7f2cbdd9b700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cc00c1da0 0x7f2ca8003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:00.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.672+0000 7f2cbdd9b700 1 -- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ca8004d90 msgr2=0x7f2ca8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:05:00.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.672+0000 7f2cbdd9b700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ca8004d90 0x7f2ca8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:00.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.672+0000 7f2cbdd9b700 1 -- 172.21.15.7:0/1998760918 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca80035b0 con 0x7f2ca8004090 2023-03-03T09:05:00.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.672+0000 7f2cbdd9b700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca8004090 0x7f2ca8004850 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000c210 tx=0x7f2ca8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277020cec733d2f1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:00.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.672+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0007c10 con 0x7f2ca8004090 2023-03-03T09:05:00.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.672+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb000cdd0 con 0x7f2ca8004090 2023-03-03T09:05:00.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.672+0000 7f2cbd59a700 1 -- 172.21.15.7:0/1998760918 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0005ba0 con 0x7f2ca8004090 2023-03-03T09:05:00.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.673+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/1998760918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca8004090 msgr2=0x7f2ca8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:00.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.673+0000 7f2cc4e8e700 1 --2- 172.21.15.7:0/1998760918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca8004090 0x7f2ca8004850 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000c210 tx=0x7f2ca8005880 comp rx=0 tx=0).stop 2023-03-03T09:05:00.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.673+0000 7f2cbed9d700 1 -- 172.21.15.7:0/1998760918 reap_dead start 2023-03-03T09:05:00.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.673+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/1998760918 shutdown_connections 2023-03-03T09:05:00.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.673+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/1998760918 >> 172.21.15.7:0/1998760918 conn(0x7f2cc006c110 msgr2=0x7f2cc006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:05:00.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.673+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/1998760918 shutdown_connections 2023-03-03T09:05:00.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.673+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/1998760918 wait complete. 2023-03-03T09:05:00.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.674+0000 7f2cc4e8e700 1 Processor -- start 2023-03-03T09:05:00.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.674+0000 7f2cc4e8e700 1 -- start start 2023-03-03T09:05:00.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cc4e8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca8004090 0x7f2cc01ed150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:00.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cc4e8e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cc0071b60 0x7f2cc01ed690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:00.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cc4e8e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cc013b3a0 0x7f2cc013b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:00.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cc4e8e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc013cf00 con 0x7f2cc0071b60 2023-03-03T09:05:00.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cc4e8e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc013d260 con 0x7f2cc013b3a0 2023-03-03T09:05:00.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cc4e8e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc013da30 con 0x7f2ca8004090 2023-03-03T09:05:00.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbdd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cc0071b60 0x7f2cc01ed690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:00.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbdd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cc0071b60 0x7f2cc01ed690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54368/0 (socket says 172.21.15.7:54368) 2023-03-03T09:05:00.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbdd9b700 1 -- 172.21.15.7:0/4133948428 learned_addr learned my addr 172.21.15.7:0/4133948428 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:05:00.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbed9d700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cc013b3a0 0x7f2cc013b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:00.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbe59c700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca8004090 0x7f2cc01ed150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:00.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbdd9b700 1 -- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca8004090 msgr2=0x7f2cc01ed150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:00.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbdd9b700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca8004090 0x7f2cc01ed150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:00.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbdd9b700 1 -- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cc013b3a0 msgr2=0x7f2cc013b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:00.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbdd9b700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cc013b3a0 0x7f2cc013b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:00.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbdd9b700 1 -- 172.21.15.7:0/4133948428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc006e210 con 0x7f2cc0071b60 2023-03-03T09:05:00.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.675+0000 7f2cbdd9b700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cc0071b60 0x7f2cc01ed690 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000c800 tx=0x7f2cb0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:00.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.676+0000 7f2caeffd700 1 -- 172.21.15.7:0/4133948428 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0001630 con 0x7f2cc0071b60 2023-03-03T09:05:00.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.676+0000 7f2caeffd700 1 -- 172.21.15.7:0/4133948428 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb00070d0 con 0x7f2cc0071b60 2023-03-03T09:05:00.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.676+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/4133948428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc00c7820 con 0x7f2cc0071b60 2023-03-03T09:05:00.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.676+0000 7f2caeffd700 1 -- 172.21.15.7:0/4133948428 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb00074a0 con 0x7f2cc0071b60 2023-03-03T09:05:00.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.676+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/4133948428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc00c0ee0 con 0x7f2cc0071b60 2023-03-03T09:05:00.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.677+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/4133948428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc0074bb0 con 0x7f2cc0071b60 2023-03-03T09:05:00.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.677+0000 7f2caeffd700 1 -- 172.21.15.7:0/4133948428 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2cb0010070 con 0x7f2cc0071b60 2023-03-03T09:05:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.677+0000 7f2caeffd700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 0x7f2ca006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.678+0000 7f2caeffd700 1 -- 172.21.15.7:0/4133948428 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2cb0087250 con 0x7f2cc0071b60 2023-03-03T09:05:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.678+0000 7f2cbe59c700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 0x7f2ca006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.678+0000 7f2cbe59c700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 0x7f2ca006a7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4000c00 tx=0x7f2cb4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.682+0000 7f2caeffd700 1 -- 172.21.15.7:0/4133948428 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2cb0053570 con 0x7f2cc0071b60 2023-03-03T09:05:00.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.918+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/4133948428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cc00c2ed0 con 0x7f2cc0071b60 2023-03-03T09:05:00.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.919+0000 7f2caeffd700 1 -- 172.21.15.7:0/4133948428 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2cb0056b90 con 0x7f2cc0071b60 2023-03-03T09:05:00.922 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:05:00.922 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:05:00.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 msgr2=0x7f2ca006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 0x7f2ca006a7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4000c00 tx=0x7f2cb4009040 comp rx=0 tx=0).stop 2023-03-03T09:05:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/4133948428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cc0071b60 msgr2=0x7f2cc01ed690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cc0071b60 0x7f2cc01ed690 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000c800 tx=0x7f2cb0001a00 comp rx=0 tx=0).stop 2023-03-03T09:05:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/4133948428 shutdown_connections 2023-03-03T09:05:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cc013b3a0 0x7f2cc013b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca8004090 0x7f2cc01ed150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:00.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 0x7f2ca006a7a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 --2- 172.21.15.7:0/4133948428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cc0071b60 0x7f2cc01ed690 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/4133948428 >> 172.21.15.7:0/4133948428 conn(0x7f2cc006c110 msgr2=0x7f2cc01537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:05:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/4133948428 shutdown_connections 2023-03-03T09:05:00.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:00.921+0000 7f2cc4e8e700 1 -- 172.21.15.7:0/4133948428 wait complete. 2023-03-03T09:05:00.926 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:05:00.935 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before thrashing 2023-03-03T09:05:08.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:05:09.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.074+0000 7f3033374700 1 Processor -- start 2023-03-03T09:05:09.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.074+0000 7f3033374700 1 -- start start 2023-03-03T09:05:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.075+0000 7f3033374700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f302c151d40 0x7f302c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.075+0000 7f3033374700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c06d430 con 0x7f302c0c2930 2023-03-03T09:05:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.075+0000 7f3033374700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c06cb00 con 0x7f302c073c30 2023-03-03T09:05:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.075+0000 7f3033374700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302c06c2a0 con 0x7f302c151d40 2023-03-03T09:05:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.075+0000 7f3031110700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f302c151d40 0x7f302c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.075+0000 7f3031110700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f302c151d40 0x7f302c0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54384/0 (socket says 172.21.15.7:54384) 2023-03-03T09:05:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.075+0000 7f3031110700 1 -- 172.21.15.7:0/3237453785 learned_addr learned my addr 172.21.15.7:0/3237453785 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:05:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.075+0000 7f303090f700 1 --1- 172.21.15.7:0/3237453785 >> v1:172.21.15.7:6789/0 conn(0x7f302c0c2930 0x7f302c157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48366/0 (socket says 172.21.15.7:48366) 2023-03-03T09:05:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.076+0000 7f3031110700 1 -- 172.21.15.7:0/3237453785 >> v1:172.21.15.143:6789/0 conn(0x7f302c073c30 legacy=0x7f302c1581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.076+0000 7f3031110700 1 -- 172.21.15.7:0/3237453785 >> v1:172.21.15.7:6789/0 conn(0x7f302c0c2930 legacy=0x7f302c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.076+0000 7f3023fff700 1 -- 172.21.15.7:0/3237453785 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120154471 0 0) 0x7f302c06d430 con 0x7f302c0c2930 2023-03-03T09:05:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.076+0000 7f3031110700 1 -- 172.21.15.7:0/3237453785 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c06cde0 con 0x7f302c151d40 2023-03-03T09:05:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.076+0000 7f3031110700 1 --2- 172.21.15.7:0/3237453785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c151d40 0x7f302c0736f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f301c009920 tx=0x7f301c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b22f9645dbb3419 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:09.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.076+0000 7f3023fff700 1 -- 172.21.15.7:0/3237453785 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c00ba20 con 0x7f302c151d40 2023-03-03T09:05:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.076+0000 7f3023fff700 1 -- 172.21.15.7:0/3237453785 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c00bb80 con 0x7f302c151d40 2023-03-03T09:05:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.076+0000 7f3023fff700 1 -- 172.21.15.7:0/3237453785 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c009e70 con 0x7f302c151d40 2023-03-03T09:05:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.077+0000 7f3033374700 1 -- 172.21.15.7:0/3237453785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c151d40 msgr2=0x7f302c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.077+0000 7f3033374700 1 --2- 172.21.15.7:0/3237453785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c151d40 0x7f302c0736f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f301c009920 tx=0x7f301c009c30 comp rx=0 tx=0).stop 2023-03-03T09:05:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.077+0000 7f3033374700 1 -- 172.21.15.7:0/3237453785 shutdown_connections 2023-03-03T09:05:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.077+0000 7f3033374700 1 --2- 172.21.15.7:0/3237453785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c151d40 0x7f302c0736f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.077+0000 7f3033374700 1 -- 172.21.15.7:0/3237453785 >> 172.21.15.7:0/3237453785 conn(0x7f302c0a94d0 msgr2=0x7f302c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:05:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.077+0000 7f3033374700 1 -- 172.21.15.7:0/3237453785 shutdown_connections 2023-03-03T09:05:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.077+0000 7f3033374700 1 -- 172.21.15.7:0/3237453785 wait complete. 2023-03-03T09:05:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.078+0000 7f3033374700 1 Processor -- start 2023-03-03T09:05:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.078+0000 7f3033374700 1 -- start start 2023-03-03T09:05:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3033374700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f302c073c30 0x7f302c0aa440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3033374700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c0c2930 0x7f302c14b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3033374700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c151d40 0x7f302c14bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3033374700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0abef0 con 0x7f302c0c2930 2023-03-03T09:05:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3033374700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0ab860 con 0x7f302c151d40 2023-03-03T09:05:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3033374700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0ad0d0 con 0x7f302c073c30 2023-03-03T09:05:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f303090f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c0c2930 0x7f302c14b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f303090f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c0c2930 0x7f302c14b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54392/0 (socket says 172.21.15.7:54392) 2023-03-03T09:05:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f303090f700 1 -- 172.21.15.7:0/2184458120 learned_addr learned my addr 172.21.15.7:0/2184458120 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:05:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3031911700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c151d40 0x7f302c14bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:09.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f303090f700 1 -- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f302c073c30 msgr2=0x7f302c0aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:09.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f303090f700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f302c073c30 0x7f302c0aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:09.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f303090f700 1 -- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c151d40 msgr2=0x7f302c14bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:09.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f303090f700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c151d40 0x7f302c14bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:09.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f303090f700 1 -- 172.21.15.7:0/2184458120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c009550 con 0x7f302c0c2930 2023-03-03T09:05:09.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3031911700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c151d40 0x7f302c14bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:05:09.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f303090f700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c0c2930 0x7f302c14b6b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f302800c520 tx=0x7f302800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:09.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3021ffb700 1 -- 172.21.15.7:0/2184458120 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3028004c80 con 0x7f302c0c2930 2023-03-03T09:05:09.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3021ffb700 1 -- 172.21.15.7:0/2184458120 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3028004e00 con 0x7f302c0c2930 2023-03-03T09:05:09.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3033374700 1 -- 172.21.15.7:0/2184458120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c06bea0 con 0x7f302c0c2930 2023-03-03T09:05:09.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.079+0000 7f3021ffb700 1 -- 172.21.15.7:0/2184458120 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3028007310 con 0x7f302c0c2930 2023-03-03T09:05:09.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.080+0000 7f3033374700 1 -- 172.21.15.7:0/2184458120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c06d0c0 con 0x7f302c0c2930 2023-03-03T09:05:09.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.081+0000 7f3033374700 1 -- 172.21.15.7:0/2184458120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302c06c030 con 0x7f302c0c2930 2023-03-03T09:05:09.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.081+0000 7f3021ffb700 1 -- 172.21.15.7:0/2184458120 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f302801a020 con 0x7f302c0c2930 2023-03-03T09:05:09.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.081+0000 7f3021ffb700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30100682f0 0x7f301006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:09.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.081+0000 7f3021ffb700 1 -- 172.21.15.7:0/2184458120 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3028085d50 con 0x7f302c0c2930 2023-03-03T09:05:09.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.082+0000 7f3031110700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30100682f0 0x7f301006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:09.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.082+0000 7f3031110700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30100682f0 0x7f301006a7a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f301c000c00 tx=0x7f301c010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:09.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.086+0000 7f3021ffb700 1 -- 172.21.15.7:0/2184458120 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30280520f0 con 0x7f302c0c2930 2023-03-03T09:05:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.323+0000 7f3033374700 1 -- 172.21.15.7:0/2184458120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f302c074bb0 con 0x7f302c0c2930 2023-03-03T09:05:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.323+0000 7f3021ffb700 1 -- 172.21.15.7:0/2184458120 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3028055710 con 0x7f302c0c2930 2023-03-03T09:05:09.325 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:05:09.325 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:05:09.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 -- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30100682f0 msgr2=0x7f301006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30100682f0 0x7f301006a7a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f301c000c00 tx=0x7f301c010000 comp rx=0 tx=0).stop 2023-03-03T09:05:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 -- 172.21.15.7:0/2184458120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c0c2930 msgr2=0x7f302c14b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c0c2930 0x7f302c14b6b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f302800c520 tx=0x7f302800c8e0 comp rx=0 tx=0).stop 2023-03-03T09:05:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 -- 172.21.15.7:0/2184458120 shutdown_connections 2023-03-03T09:05:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f302c151d40 0x7f302c14bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f302c073c30 0x7f302c0aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30100682f0 0x7f301006a7a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 --2- 172.21.15.7:0/2184458120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f302c0c2930 0x7f302c14b6b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 -- 172.21.15.7:0/2184458120 >> 172.21.15.7:0/2184458120 conn(0x7f302c0a94d0 msgr2=0x7f302c158280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:05:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 -- 172.21.15.7:0/2184458120 shutdown_connections 2023-03-03T09:05:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:09.325+0000 7f3033374700 1 -- 172.21.15.7:0/2184458120 wait complete. 2023-03-03T09:05:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:05:09.338 INFO:tasks.mds_thrash.fs.[a]:waiting for 112 secs before thrashing 2023-03-03T09:05:10.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:10.186 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:35.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:05:36.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.015+0000 7f5b2c6df700 1 Processor -- start 2023-03-03T09:05:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.015+0000 7f5b2c6df700 1 -- start start 2023-03-03T09:05:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b2c6df700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240caa30 0x7f5b240c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b2c6df700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240c7310 con 0x7f5b240c00e0 2023-03-03T09:05:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b2c6df700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240bfd10 con 0x7f5b24154ff0 2023-03-03T09:05:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b2c6df700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240bfa30 con 0x7f5b240caa30 2023-03-03T09:05:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29c7a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5b240c00e0 0x7f5b240c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48406/0 (socket says 172.21.15.7:48406) 2023-03-03T09:05:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29c7a700 1 -- 172.21.15.7:0/4070880617 learned_addr learned my addr 172.21.15.7:0/4070880617 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:05:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b2a47b700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240caa30 0x7f5b240c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618834861 0 0) 0x7f5b240c7310 con 0x7f5b240c00e0 2023-03-03T09:05:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b100035b0 con 0x7f5b240c00e0 2023-03-03T09:05:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350623938 0 0) 0x7f5b240bfd10 con 0x7f5b24154ff0 2023-03-03T09:05:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b240c7310 con 0x7f5b24154ff0 2023-03-03T09:05:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2808911340 0 0) 0x7f5b100035b0 con 0x7f5b240c00e0 2023-03-03T09:05:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240caa30 msgr2=0x7f5b240c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:05:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29479700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240caa30 0x7f5b240c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:36.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 >> v1:172.21.15.143:6789/0 conn(0x7f5b24154ff0 legacy=0x7f5b2415ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:36.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.016+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24154510 con 0x7f5b240c00e0 2023-03-03T09:05:36.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f5b140017f0 con 0x7f5b240c00e0 2023-03-03T09:05:36.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b2a47b700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240caa30 0x7f5b240c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:05:36.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 >> v1:172.21.15.7:6789/0 conn(0x7f5b240c00e0 legacy=0x7f5b240c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:36.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b10003c30 0x7f5b10004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:36.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b10004850 0x7f5b10004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:36.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240caa30 0x7f5b10005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:36.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240bfa30 con 0x7f5b10004850 2023-03-03T09:05:36.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1000bb20 con 0x7f5b10003c30 2023-03-03T09:05:36.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1000bce0 con 0x7f5b240caa30 2023-03-03T09:05:36.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29c7a700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b10004850 0x7f5b10004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b2a47b700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b10003c30 0x7f5b10004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b2ac7c700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240caa30 0x7f5b10005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29c7a700 1 -- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240caa30 msgr2=0x7f5b10005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:05:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29c7a700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240caa30 0x7f5b10005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29c7a700 1 -- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b10003c30 msgr2=0x7f5b10004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:05:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29c7a700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b10003c30 0x7f5b10004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29c7a700 1 -- 172.21.15.7:0/4070880617 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b200094a0 con 0x7f5b10004850 2023-03-03T09:05:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29c7a700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b10004850 0x7f5b10004cc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400c240 tx=0x7f5b10005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479d4aacb0af2b6b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1400cd80 con 0x7f5b10004850 2023-03-03T09:05:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b14001c70 con 0x7f5b10004850 2023-03-03T09:05:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.017+0000 7f5b29479700 1 -- 172.21.15.7:0/4070880617 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14007310 con 0x7f5b10004850 2023-03-03T09:05:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.018+0000 7f5b2c6df700 1 -- 172.21.15.7:0/4070880617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b10004850 msgr2=0x7f5b10004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.018+0000 7f5b2c6df700 1 --2- 172.21.15.7:0/4070880617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b10004850 0x7f5b10004cc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400c240 tx=0x7f5b10005880 comp rx=0 tx=0).stop 2023-03-03T09:05:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.018+0000 7f5b2ac7c700 1 -- 172.21.15.7:0/4070880617 reap_dead start 2023-03-03T09:05:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.018+0000 7f5b2c6df700 1 -- 172.21.15.7:0/4070880617 shutdown_connections 2023-03-03T09:05:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.018+0000 7f5b2c6df700 1 -- 172.21.15.7:0/4070880617 >> 172.21.15.7:0/4070880617 conn(0x7f5b24066e40 msgr2=0x7f5b24071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:05:36.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.018+0000 7f5b2c6df700 1 -- 172.21.15.7:0/4070880617 shutdown_connections 2023-03-03T09:05:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.018+0000 7f5b2c6df700 1 -- 172.21.15.7:0/4070880617 wait complete. 2023-03-03T09:05:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.019+0000 7f5b2c6df700 1 Processor -- start 2023-03-03T09:05:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.019+0000 7f5b2c6df700 1 -- start start 2023-03-03T09:05:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b2c6df700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b10004850 0x7f5b241ead60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b2c6df700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414c8b0 0x7f5b241e7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b2c6df700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240c19e0 0x7f5b24157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b2c6df700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24145c20 con 0x7f5b2414c8b0 2023-03-03T09:05:36.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b2c6df700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24158900 con 0x7f5b10004850 2023-03-03T09:05:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b2c6df700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b241e75b0 con 0x7f5b240c19e0 2023-03-03T09:05:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b29c7a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414c8b0 0x7f5b241e7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b29c7a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414c8b0 0x7f5b241e7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54438/0 (socket says 172.21.15.7:54438) 2023-03-03T09:05:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b29c7a700 1 -- 172.21.15.7:0/340455178 learned_addr learned my addr 172.21.15.7:0/340455178 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:05:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b2a47b700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b10004850 0x7f5b241ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b2ac7c700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240c19e0 0x7f5b24157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:36.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b29c7a700 1 -- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240c19e0 msgr2=0x7f5b24157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b29c7a700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240c19e0 0x7f5b24157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b29c7a700 1 -- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b10004850 msgr2=0x7f5b241ead60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b29c7a700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b10004850 0x7f5b241ead60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.020+0000 7f5b29c7a700 1 -- 172.21.15.7:0/340455178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24154ad0 con 0x7f5b2414c8b0 2023-03-03T09:05:36.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.021+0000 7f5b29c7a700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414c8b0 0x7f5b241e7070 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400c830 tx=0x7f5b14001b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.021+0000 7f5b1affd700 1 -- 172.21.15.7:0/340455178 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14005c40 con 0x7f5b2414c8b0 2023-03-03T09:05:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.021+0000 7f5b1affd700 1 -- 172.21.15.7:0/340455178 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b14005da0 con 0x7f5b2414c8b0 2023-03-03T09:05:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.021+0000 7f5b2c6df700 1 -- 172.21.15.7:0/340455178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b241547f0 con 0x7f5b2414c8b0 2023-03-03T09:05:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.021+0000 7f5b1affd700 1 -- 172.21.15.7:0/340455178 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14007310 con 0x7f5b2414c8b0 2023-03-03T09:05:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.021+0000 7f5b2c6df700 1 -- 172.21.15.7:0/340455178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b240bf650 con 0x7f5b2414c8b0 2023-03-03T09:05:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.022+0000 7f5b2c6df700 1 -- 172.21.15.7:0/340455178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b24074bb0 con 0x7f5b2414c8b0 2023-03-03T09:05:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.022+0000 7f5b1affd700 1 -- 172.21.15.7:0/340455178 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5b1402e070 con 0x7f5b2414c8b0 2023-03-03T09:05:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.023+0000 7f5b1affd700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b080682f0 0x7f5b0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:05:36.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.023+0000 7f5b1affd700 1 -- 172.21.15.7:0/340455178 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5b14087360 con 0x7f5b2414c8b0 2023-03-03T09:05:36.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.023+0000 7f5b2a47b700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b080682f0 0x7f5b0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:05:36.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.023+0000 7f5b2a47b700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b080682f0 0x7f5b0806a7a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009470 tx=0x7f5b20009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:05:36.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.027+0000 7f5b1affd700 1 -- 172.21.15.7:0/340455178 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5b140536c0 con 0x7f5b2414c8b0 2023-03-03T09:05:36.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.267+0000 7f5b2c6df700 1 -- 172.21.15.7:0/340455178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b24145c00 con 0x7f5b2414c8b0 2023-03-03T09:05:36.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.267+0000 7f5b1affd700 1 -- 172.21.15.7:0/340455178 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5b14056ce0 con 0x7f5b2414c8b0 2023-03-03T09:05:36.269 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:05:36.269 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:05:36.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 -- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b080682f0 msgr2=0x7f5b0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:36.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b080682f0 0x7f5b0806a7a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5b20009470 tx=0x7f5b20009210 comp rx=0 tx=0).stop 2023-03-03T09:05:36.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 -- 172.21.15.7:0/340455178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414c8b0 msgr2=0x7f5b241e7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:05:36.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414c8b0 0x7f5b241e7070 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400c830 tx=0x7f5b14001b70 comp rx=0 tx=0).stop 2023-03-03T09:05:36.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 -- 172.21.15.7:0/340455178 shutdown_connections 2023-03-03T09:05:36.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b240c19e0 0x7f5b24157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:36.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b10004850 0x7f5b241ead60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:36.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b080682f0 0x7f5b0806a7a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:36.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 --2- 172.21.15.7:0/340455178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414c8b0 0x7f5b241e7070 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:05:36.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 -- 172.21.15.7:0/340455178 >> 172.21.15.7:0/340455178 conn(0x7f5b24066e40 msgr2=0x7f5b240c8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:05:36.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 -- 172.21.15.7:0/340455178 shutdown_connections 2023-03-03T09:05:36.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:05:36.269+0000 7f5b2c6df700 1 -- 172.21.15.7:0/340455178 wait complete. 2023-03-03T09:05:36.273 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:05:36.282 INFO:tasks.mds_thrash.fs.[b]:waiting for 90 secs before thrashing 2023-03-03T09:05:40.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:05:40.274 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:10.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:10.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:40.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:06:40.408 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:01.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:07:01.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.477+0000 7f5c43502700 1 Processor -- start 2023-03-03T09:07:01.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.477+0000 7f5c43502700 1 -- start start 2023-03-03T09:07:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.478+0000 7f5c43502700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5c3c06c480 0x7f5c3c1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.478+0000 7f5c43502700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3c06d430 con 0x7f5c3c0c2930 2023-03-03T09:07:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.478+0000 7f5c43502700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3c1586e0 con 0x7f5c3c151d40 2023-03-03T09:07:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.478+0000 7f5c43502700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3c06c2a0 con 0x7f5c3c06c480 2023-03-03T09:07:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.478+0000 7f5c41a9f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5c3c06c480 0x7f5c3c1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.478+0000 7f5c40a9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5c3c0c2930 0x7f5c3c157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48494/0 (socket says 172.21.15.7:48494) 2023-03-03T09:07:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.478+0000 7f5c40a9d700 1 -- 172.21.15.7:0/1669938007 learned_addr learned my addr 172.21.15.7:0/1669938007 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:07:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c41a9f700 1 -- 172.21.15.7:0/1669938007 >> v1:172.21.15.143:6789/0 conn(0x7f5c3c151d40 legacy=0x7f5c3c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:07:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c41a9f700 1 -- 172.21.15.7:0/1669938007 >> v1:172.21.15.7:6789/0 conn(0x7f5c3c0c2930 legacy=0x7f5c3c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c41a9f700 1 -- 172.21.15.7:0/1669938007 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c06cb00 con 0x7f5c3c06c480 2023-03-03T09:07:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c41a9f700 1 --2- 172.21.15.7:0/1669938007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c06c480 0x7f5c3c1581a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009dc0 tx=0x7f5c3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a36c3d05a090ac8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c33fff700 1 -- 172.21.15.7:0/1669938007 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3400e070 con 0x7f5c3c06c480 2023-03-03T09:07:01.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c33fff700 1 -- 172.21.15.7:0/1669938007 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3400bb50 con 0x7f5c3c06c480 2023-03-03T09:07:01.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c33fff700 1 -- 172.21.15.7:0/1669938007 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34003a20 con 0x7f5c3c06c480 2023-03-03T09:07:01.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c43502700 1 -- 172.21.15.7:0/1669938007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c06c480 msgr2=0x7f5c3c1581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:01.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c43502700 1 --2- 172.21.15.7:0/1669938007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c06c480 0x7f5c3c1581a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009dc0 tx=0x7f5c3400b330 comp rx=0 tx=0).stop 2023-03-03T09:07:01.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c43502700 1 -- 172.21.15.7:0/1669938007 shutdown_connections 2023-03-03T09:07:01.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c43502700 1 --2- 172.21.15.7:0/1669938007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c06c480 0x7f5c3c1581a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:01.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.479+0000 7f5c43502700 1 -- 172.21.15.7:0/1669938007 >> 172.21.15.7:0/1669938007 conn(0x7f5c3c0a94d0 msgr2=0x7f5c3c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:07:01.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.480+0000 7f5c43502700 1 -- 172.21.15.7:0/1669938007 shutdown_connections 2023-03-03T09:07:01.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.480+0000 7f5c43502700 1 -- 172.21.15.7:0/1669938007 wait complete. 2023-03-03T09:07:01.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.481+0000 7f5c43502700 1 Processor -- start 2023-03-03T09:07:01.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.481+0000 7f5c43502700 1 -- start start 2023-03-03T09:07:01.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.481+0000 7f5c43502700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c3c06c480 0x7f5c3c1475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:01.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.481+0000 7f5c43502700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c0c2930 0x7f5c3c147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:01.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.481+0000 7f5c43502700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c3c151d40 0x7f5c3c1e9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:01.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.481+0000 7f5c43502700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c1ddda0 con 0x7f5c3c0c2930 2023-03-03T09:07:01.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.481+0000 7f5c43502700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c148060 con 0x7f5c3c06c480 2023-03-03T09:07:01.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.481+0000 7f5c43502700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c1de510 con 0x7f5c3c151d40 2023-03-03T09:07:01.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c40a9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c0c2930 0x7f5c3c147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:01.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c40a9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c0c2930 0x7f5c3c147b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54520/0 (socket says 172.21.15.7:54520) 2023-03-03T09:07:01.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c40a9d700 1 -- 172.21.15.7:0/2724970848 learned_addr learned my addr 172.21.15.7:0/2724970848 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:07:01.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c4129e700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c3c06c480 0x7f5c3c1475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:01.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c41a9f700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c3c151d40 0x7f5c3c1e9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:01.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c40a9d700 1 -- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c3c151d40 msgr2=0x7f5c3c1e9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c40a9d700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c3c151d40 0x7f5c3c1e9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c40a9d700 1 -- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c3c06c480 msgr2=0x7f5c3c1475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c40a9d700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c3c06c480 0x7f5c3c1475e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c40a9d700 1 -- 172.21.15.7:0/2724970848 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34009a70 con 0x7f5c3c0c2930 2023-03-03T09:07:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c41a9f700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c3c151d40 0x7f5c3c1e9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:07:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c40a9d700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c0c2930 0x7f5c3c147b20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800c4b0 tx=0x7f5c3800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c31ffb700 1 -- 172.21.15.7:0/2724970848 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c38005d70 con 0x7f5c3c0c2930 2023-03-03T09:07:01.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c31ffb700 1 -- 172.21.15.7:0/2724970848 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c380070d0 con 0x7f5c3c0c2930 2023-03-03T09:07:01.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c31ffb700 1 -- 172.21.15.7:0/2724970848 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c38007440 con 0x7f5c3c0c2930 2023-03-03T09:07:01.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.482+0000 7f5c43502700 1 -- 172.21.15.7:0/2724970848 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3c06d0c0 con 0x7f5c3c0c2930 2023-03-03T09:07:01.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.483+0000 7f5c43502700 1 -- 172.21.15.7:0/2724970848 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3c06cde0 con 0x7f5c3c0c2930 2023-03-03T09:07:01.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.484+0000 7f5c43502700 1 -- 172.21.15.7:0/2724970848 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c074bb0 con 0x7f5c3c0c2930 2023-03-03T09:07:01.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.484+0000 7f5c31ffb700 1 -- 172.21.15.7:0/2724970848 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5c38019030 con 0x7f5c3c0c2930 2023-03-03T09:07:01.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.484+0000 7f5c31ffb700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 0x7f5c2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:01.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.484+0000 7f5c31ffb700 1 -- 172.21.15.7:0/2724970848 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5c380853b0 con 0x7f5c3c0c2930 2023-03-03T09:07:01.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.485+0000 7f5c4129e700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 0x7f5c2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:01.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.485+0000 7f5c4129e700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 0x7f5c2006a7a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c004f30 tx=0x7f5c2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:01.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.489+0000 7f5c31ffb700 1 -- 172.21.15.7:0/2724970848 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5c3804b5e0 con 0x7f5c3c0c2930 2023-03-03T09:07:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.728+0000 7f5c43502700 1 -- 172.21.15.7:0/2724970848 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c3c073d80 con 0x7f5c3c0c2930 2023-03-03T09:07:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.728+0000 7f5c31ffb700 1 -- 172.21.15.7:0/2724970848 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5c380517a0 con 0x7f5c3c0c2930 2023-03-03T09:07:01.730 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:07:01.730 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:07:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.730+0000 7f5c43502700 1 -- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 msgr2=0x7f5c2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.730+0000 7f5c43502700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 0x7f5c2006a7a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c004f30 tx=0x7f5c2c009040 comp rx=0 tx=0).stop 2023-03-03T09:07:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.730+0000 7f5c43502700 1 -- 172.21.15.7:0/2724970848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c0c2930 msgr2=0x7f5c3c147b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.730+0000 7f5c43502700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c0c2930 0x7f5c3c147b20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800c4b0 tx=0x7f5c3800c870 comp rx=0 tx=0).stop 2023-03-03T09:07:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.731+0000 7f5c43502700 1 -- 172.21.15.7:0/2724970848 shutdown_connections 2023-03-03T09:07:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.731+0000 7f5c43502700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c3c151d40 0x7f5c3c1e9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.731+0000 7f5c43502700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c3c06c480 0x7f5c3c1475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.731+0000 7f5c43502700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 0x7f5c2006a7a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.731+0000 7f5c43502700 1 --2- 172.21.15.7:0/2724970848 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c3c0c2930 0x7f5c3c147b20 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.731+0000 7f5c43502700 1 -- 172.21.15.7:0/2724970848 >> 172.21.15.7:0/2724970848 conn(0x7f5c3c0a94d0 msgr2=0x7f5c3c0c34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:07:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.731+0000 7f5c43502700 1 -- 172.21.15.7:0/2724970848 shutdown_connections 2023-03-03T09:07:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:01.731+0000 7f5c43502700 1 -- 172.21.15.7:0/2724970848 wait complete. 2023-03-03T09:07:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:07:01.744 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before thrashing 2023-03-03T09:07:04.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:07:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.882+0000 7fc6a82a6700 1 Processor -- start 2023-03-03T09:07:04.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.882+0000 7fc6a82a6700 1 -- start start 2023-03-03T09:07:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a82a6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 0x7fc6a014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:04.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a82a6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6a00c1ce0 0x7fc6a014e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:04.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a82a6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a01510d0 con 0x7fc6a0071e80 2023-03-03T09:07:04.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a82a6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a00c3740 con 0x7fc6a00c1ce0 2023-03-03T09:07:04.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a82a6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0070670 con 0x7fc6a0072250 2023-03-03T09:07:04.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a6843700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6a00c1ce0 0x7fc6a014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:04.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a6042700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc6a0071e80 0x7fc6a0151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48510/0 (socket says 172.21.15.7:48510) 2023-03-03T09:07:04.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a6042700 1 -- 172.21.15.7:0/333042140 learned_addr learned my addr 172.21.15.7:0/333042140 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:07:04.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a5841700 1 --2- 172.21.15.7:0/333042140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 0x7fc6a014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:04.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a5040700 1 -- 172.21.15.7:0/333042140 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773325930 0 0) 0x7fc6a01510d0 con 0x7fc6a0071e80 2023-03-03T09:07:04.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a5040700 1 -- 172.21.15.7:0/333042140 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc68c0035b0 con 0x7fc6a0071e80 2023-03-03T09:07:04.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a6843700 1 -- 172.21.15.7:0/333042140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 msgr2=0x7fc6a014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:07:04.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a6843700 1 --2- 172.21.15.7:0/333042140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 0x7fc6a014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:04.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a6843700 1 -- 172.21.15.7:0/333042140 >> v1:172.21.15.7:6789/0 conn(0x7fc6a0071e80 legacy=0x7fc6a0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:04.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.883+0000 7fc6a6843700 1 -- 172.21.15.7:0/333042140 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a00c39e0 con 0x7fc6a00c1ce0 2023-03-03T09:07:04.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a6843700 1 --2- 172.21.15.7:0/333042140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a00c1ce0 0x7fc6a014e730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc698009d90 tx=0x7fc69800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718b2bff89c03311 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a5040700 1 -- 172.21.15.7:0/333042140 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69800e070 con 0x7fc6a00c1ce0 2023-03-03T09:07:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a5040700 1 -- 172.21.15.7:0/333042140 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69800bb20 con 0x7fc6a00c1ce0 2023-03-03T09:07:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a5040700 1 -- 172.21.15.7:0/333042140 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698012430 con 0x7fc6a00c1ce0 2023-03-03T09:07:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a82a6700 1 -- 172.21.15.7:0/333042140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a00c1ce0 msgr2=0x7fc6a014e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a82a6700 1 --2- 172.21.15.7:0/333042140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a00c1ce0 0x7fc6a014e730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc698009d90 tx=0x7fc69800b330 comp rx=0 tx=0).stop 2023-03-03T09:07:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a82a6700 1 -- 172.21.15.7:0/333042140 shutdown_connections 2023-03-03T09:07:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a82a6700 1 --2- 172.21.15.7:0/333042140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a00c1ce0 0x7fc6a014e730 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a82a6700 1 --2- 172.21.15.7:0/333042140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 0x7fc6a014e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a82a6700 1 -- 172.21.15.7:0/333042140 >> 172.21.15.7:0/333042140 conn(0x7fc6a00a94d0 msgr2=0x7fc6a00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:07:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a82a6700 1 -- 172.21.15.7:0/333042140 shutdown_connections 2023-03-03T09:07:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.884+0000 7fc6a82a6700 1 -- 172.21.15.7:0/333042140 wait complete. 2023-03-03T09:07:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.886+0000 7fc6a82a6700 1 Processor -- start 2023-03-03T09:07:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.886+0000 7fc6a82a6700 1 -- start start 2023-03-03T09:07:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a82a6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a0071e80 0x7fc6a00c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a82a6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 0x7fc6a0149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a82a6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6a00c1ce0 0x7fc6a01e7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a82a6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a00c5a10 con 0x7fc6a0071e80 2023-03-03T09:07:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a82a6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a00acb20 con 0x7fc6a00c1ce0 2023-03-03T09:07:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a82a6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0146af0 con 0x7fc6a0072250 2023-03-03T09:07:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a6042700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a0071e80 0x7fc6a00c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a6042700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a0071e80 0x7fc6a00c54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54536/0 (socket says 172.21.15.7:54536) 2023-03-03T09:07:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a6042700 1 -- 172.21.15.7:0/3599943179 learned_addr learned my addr 172.21.15.7:0/3599943179 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:07:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a5841700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 0x7fc6a0149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a6843700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6a00c1ce0 0x7fc6a01e7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a6042700 1 -- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 msgr2=0x7fc6a0149fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a6042700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 0x7fc6a0149fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a6042700 1 -- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6a00c1ce0 msgr2=0x7fc6a01e7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a6042700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6a00c1ce0 0x7fc6a01e7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a6042700 1 -- 172.21.15.7:0/3599943179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc698009a70 con 0x7fc6a0071e80 2023-03-03T09:07:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.887+0000 7fc6a5841700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 0x7fc6a0149fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:07:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.888+0000 7fc6a6042700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a0071e80 0x7fc6a00c54d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc69000c730 tx=0x7fc69000caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.888+0000 7fc696ffd700 1 -- 172.21.15.7:0/3599943179 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690005d70 con 0x7fc6a0071e80 2023-03-03T09:07:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.888+0000 7fc696ffd700 1 -- 172.21.15.7:0/3599943179 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6900070d0 con 0x7fc6a0071e80 2023-03-03T09:07:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.888+0000 7fc6a82a6700 1 -- 172.21.15.7:0/3599943179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a0151f30 con 0x7fc6a0071e80 2023-03-03T09:07:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.888+0000 7fc696ffd700 1 -- 172.21.15.7:0/3599943179 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690007440 con 0x7fc6a0071e80 2023-03-03T09:07:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.888+0000 7fc6a82a6700 1 -- 172.21.15.7:0/3599943179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a0151300 con 0x7fc6a0071e80 2023-03-03T09:07:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.889+0000 7fc6a82a6700 1 -- 172.21.15.7:0/3599943179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a006c9a0 con 0x7fc6a0071e80 2023-03-03T09:07:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.889+0000 7fc696ffd700 1 -- 172.21.15.7:0/3599943179 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc690019030 con 0x7fc6a0071e80 2023-03-03T09:07:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.890+0000 7fc696ffd700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6840682f0 0x7fc68406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.890+0000 7fc696ffd700 1 -- 172.21.15.7:0/3599943179 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc690085410 con 0x7fc6a0071e80 2023-03-03T09:07:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.890+0000 7fc6a5841700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6840682f0 0x7fc68406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.890+0000 7fc6a5841700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6840682f0 0x7fc68406a7a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc6a01554a0 tx=0x7fc69c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:04.894+0000 7fc696ffd700 1 -- 172.21.15.7:0/3599943179 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc69004b5e0 con 0x7fc6a0071e80 2023-03-03T09:07:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.130+0000 7fc6a82a6700 1 -- 172.21.15.7:0/3599943179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6a0074bb0 con 0x7fc6a0071e80 2023-03-03T09:07:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.130+0000 7fc696ffd700 1 -- 172.21.15.7:0/3599943179 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc690051860 con 0x7fc6a0071e80 2023-03-03T09:07:05.132 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:07:05.132 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:07:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 -- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6840682f0 msgr2=0x7fc68406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6840682f0 0x7fc68406a7a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc6a01554a0 tx=0x7fc69c008040 comp rx=0 tx=0).stop 2023-03-03T09:07:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 -- 172.21.15.7:0/3599943179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a0071e80 msgr2=0x7fc6a00c54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a0071e80 0x7fc6a00c54d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc69000c730 tx=0x7fc69000caf0 comp rx=0 tx=0).stop 2023-03-03T09:07:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 -- 172.21.15.7:0/3599943179 shutdown_connections 2023-03-03T09:07:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6a00c1ce0 0x7fc6a01e7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:05.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6840682f0 0x7fc68406a7a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:05.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6a0071e80 0x7fc6a00c54d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:05.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 --2- 172.21.15.7:0/3599943179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6a0072250 0x7fc6a0149fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:05.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 -- 172.21.15.7:0/3599943179 >> 172.21.15.7:0/3599943179 conn(0x7fc6a00a94d0 msgr2=0x7fc6a014dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:07:05.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.132+0000 7fc6a82a6700 1 -- 172.21.15.7:0/3599943179 shutdown_connections 2023-03-03T09:07:05.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:05.133+0000 7fc6a82a6700 1 -- 172.21.15.7:0/3599943179 wait complete. 2023-03-03T09:07:05.136 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:07:05.145 INFO:tasks.mds_thrash.fs.[a]:waiting for 61 secs before thrashing 2023-03-03T09:07:06.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:07:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.420+0000 7f8e49e9b700 1 Processor -- start 2023-03-03T09:07:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.420+0000 7f8e49e9b700 1 -- start start 2023-03-03T09:07:06.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.421+0000 7f8e49e9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e4406a2b0 0x7f8e4406a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.421+0000 7f8e49e9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e440a6370 0x7f8e44155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.421+0000 7f8e49e9b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e441558f0 con 0x7f8e440c5cb0 2023-03-03T09:07:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.421+0000 7f8e49e9b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4406bab0 con 0x7f8e440a6370 2023-03-03T09:07:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.421+0000 7f8e49e9b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e440a6180 con 0x7f8e4406a2b0 2023-03-03T09:07:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.421+0000 7f8e43fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8e440c5cb0 0x7f8e440c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48530/0 (socket says 172.21.15.7:48530) 2023-03-03T09:07:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.421+0000 7f8e43fff700 1 -- 172.21.15.7:0/3295518970 learned_addr learned my addr 172.21.15.7:0/3295518970 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:07:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.421+0000 7f8e437fe700 1 --2- 172.21.15.7:0/3295518970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e4406a2b0 0x7f8e4406a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.421+0000 7f8e42ffd700 1 --2- 172.21.15.7:0/3295518970 >> v2:172.21.15.143:3300/0 conn(0x7f8e440a6370 0x7f8e44155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e427fc700 1 -- 172.21.15.7:0/3295518970 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146839348 0 0) 0x7f8e441558f0 con 0x7f8e440c5cb0 2023-03-03T09:07:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e427fc700 1 -- 172.21.15.7:0/3295518970 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e2c0035b0 con 0x7f8e440c5cb0 2023-03-03T09:07:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e427fc700 1 -- 172.21.15.7:0/3295518970 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393461102 0 0) 0x7f8e2c0035b0 con 0x7f8e440c5cb0 2023-03-03T09:07:06.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e42ffd700 1 -- 172.21.15.7:0/3295518970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e4406a2b0 msgr2=0x7f8e4406a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:07:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e42ffd700 1 --2- 172.21.15.7:0/3295518970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e4406a2b0 0x7f8e4406a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e42ffd700 1 -- 172.21.15.7:0/3295518970 >> v1:172.21.15.7:6789/0 conn(0x7f8e440c5cb0 legacy=0x7f8e440c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e42ffd700 1 -- 172.21.15.7:0/3295518970 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e440a5ea0 con 0x7f8e440a6370 2023-03-03T09:07:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e42ffd700 1 --2- 172.21.15.7:0/3295518970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e440a6370 0x7f8e44155d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009840 tx=0x7f8e38009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32197791038c799a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e427fc700 1 -- 172.21.15.7:0/3295518970 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c0019f0 con 0x7f8e440a6370 2023-03-03T09:07:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e427fc700 1 -- 172.21.15.7:0/3295518970 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e38004030 con 0x7f8e440a6370 2023-03-03T09:07:06.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.422+0000 7f8e427fc700 1 -- 172.21.15.7:0/3295518970 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e380043a0 con 0x7f8e440a6370 2023-03-03T09:07:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.423+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3295518970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e440a6370 msgr2=0x7f8e44155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.423+0000 7f8e49e9b700 1 --2- 172.21.15.7:0/3295518970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e440a6370 0x7f8e44155d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8e38009840 tx=0x7f8e38009b50 comp rx=0 tx=0).stop 2023-03-03T09:07:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.423+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3295518970 shutdown_connections 2023-03-03T09:07:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.423+0000 7f8e49e9b700 1 --2- 172.21.15.7:0/3295518970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e4406a2b0 0x7f8e4406a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.423+0000 7f8e49e9b700 1 --2- 172.21.15.7:0/3295518970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e440a6370 0x7f8e44155d80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.423+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3295518970 >> 172.21.15.7:0/3295518970 conn(0x7f8e440a9430 msgr2=0x7f8e440c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:07:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.423+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3295518970 shutdown_connections 2023-03-03T09:07:06.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.423+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3295518970 wait complete. 2023-03-03T09:07:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.424+0000 7f8e49e9b700 1 Processor -- start 2023-03-03T09:07:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.424+0000 7f8e49e9b700 1 -- start start 2023-03-03T09:07:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e49e9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e4406a2b0 0x7f8e4414db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e49e9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e440a6370 0x7f8e440a9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e49e9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e440c5cb0 0x7f8e440aa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e49e9b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e441538c0 con 0x7f8e4406a2b0 2023-03-03T09:07:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e49e9b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e440c0830 con 0x7f8e440a6370 2023-03-03T09:07:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e49e9b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e44155430 con 0x7f8e440c5cb0 2023-03-03T09:07:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e4406a2b0 0x7f8e4414db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e4406a2b0 0x7f8e4414db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54552/0 (socket says 172.21.15.7:54552) 2023-03-03T09:07:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e437fe700 1 -- 172.21.15.7:0/3897759058 learned_addr learned my addr 172.21.15.7:0/3897759058 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:07:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e42ffd700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e440a6370 0x7f8e440a9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e43fff700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e440c5cb0 0x7f8e440aa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e437fe700 1 -- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e440c5cb0 msgr2=0x7f8e440aa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.425+0000 7f8e437fe700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e440c5cb0 0x7f8e440aa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:06.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.426+0000 7f8e437fe700 1 -- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e440a6370 msgr2=0x7f8e440a9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.426+0000 7f8e437fe700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e440a6370 0x7f8e440a9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.426+0000 7f8e437fe700 1 -- 172.21.15.7:0/3897759058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e380094a0 con 0x7f8e4406a2b0 2023-03-03T09:07:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.426+0000 7f8e437fe700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e4406a2b0 0x7f8e4414db90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400e8a0 tx=0x7f8e3400ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.426+0000 7f8e2bfff700 1 -- 172.21.15.7:0/3897759058 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34009590 con 0x7f8e4406a2b0 2023-03-03T09:07:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.426+0000 7f8e2bfff700 1 -- 172.21.15.7:0/3897759058 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e34013680 con 0x7f8e4406a2b0 2023-03-03T09:07:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.426+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3897759058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e44153160 con 0x7f8e4406a2b0 2023-03-03T09:07:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.426+0000 7f8e2bfff700 1 -- 172.21.15.7:0/3897759058 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e340139f0 con 0x7f8e4406a2b0 2023-03-03T09:07:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.426+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3897759058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e44152e80 con 0x7f8e4406a2b0 2023-03-03T09:07:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.427+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3897759058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e440c1500 con 0x7f8e4406a2b0 2023-03-03T09:07:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.427+0000 7f8e2bfff700 1 -- 172.21.15.7:0/3897759058 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e34023070 con 0x7f8e4406a2b0 2023-03-03T09:07:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.428+0000 7f8e2bfff700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e300682f0 0x7f8e3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:07:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.428+0000 7f8e2bfff700 1 -- 172.21.15.7:0/3897759058 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8e34086280 con 0x7f8e4406a2b0 2023-03-03T09:07:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.428+0000 7f8e42ffd700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e300682f0 0x7f8e3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:07:06.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.429+0000 7f8e42ffd700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e300682f0 0x7f8e3006a7a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8e38000c00 tx=0x7f8e38008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:07:06.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.432+0000 7f8e2bfff700 1 -- 172.21.15.7:0/3897759058 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e340525c0 con 0x7f8e4406a2b0 2023-03-03T09:07:06.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.667+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3897759058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e44074bb0 con 0x7f8e4406a2b0 2023-03-03T09:07:06.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.668+0000 7f8e2bfff700 1 -- 172.21.15.7:0/3897759058 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8e34055be0 con 0x7f8e4406a2b0 2023-03-03T09:07:06.670 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:07:06.670 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:07:06.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e300682f0 msgr2=0x7f8e3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:06.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e300682f0 0x7f8e3006a7a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8e38000c00 tx=0x7f8e38008030 comp rx=0 tx=0).stop 2023-03-03T09:07:06.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3897759058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e4406a2b0 msgr2=0x7f8e4414db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:07:06.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e4406a2b0 0x7f8e4414db90 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400e8a0 tx=0x7f8e3400ebb0 comp rx=0 tx=0).stop 2023-03-03T09:07:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3897759058 shutdown_connections 2023-03-03T09:07:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e440c5cb0 0x7f8e440aa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e300682f0 0x7f8e3006a7a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e4406a2b0 0x7f8e4414db90 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 --2- 172.21.15.7:0/3897759058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e440a6370 0x7f8e440a9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:07:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3897759058 >> 172.21.15.7:0/3897759058 conn(0x7f8e440a9430 msgr2=0x7f8e4406c670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:07:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3897759058 shutdown_connections 2023-03-03T09:07:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:07:06.670+0000 7f8e49e9b700 1 -- 172.21.15.7:0/3897759058 wait complete. 2023-03-03T09:07:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:07:06.684 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T09:07:10.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:10.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:40.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:07:40.846 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:06.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:08:06.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.285+0000 7f8293069700 1 Processor -- start 2023-03-03T09:08:06.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.285+0000 7f8293069700 1 -- start start 2023-03-03T09:08:06.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8293069700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f828c14fda0 0x7f828c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:06.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8293069700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f828c0abe20 0x7f828c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:06.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8293069700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f828c0c2f30 con 0x7f828c14f9d0 2023-03-03T09:08:06.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8293069700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f828c0c31d0 con 0x7f828c0abe20 2023-03-03T09:08:06.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8293069700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f828c1524a0 con 0x7f828c14fda0 2023-03-03T09:08:06.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8291606700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f828c0abe20 0x7f828c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:06.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8291606700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f828c0abe20 0x7f828c1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54612/0 (socket says 172.21.15.7:54612) 2023-03-03T09:08:06.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8291606700 1 -- 172.21.15.7:0/3573154894 learned_addr learned my addr 172.21.15.7:0/3573154894 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:08:06.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f828bfff700 1 --2- 172.21.15.7:0/3573154894 >> v2:172.21.15.143:3300/0 conn(0x7f828c14fda0 0x7f828c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:06.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8291606700 1 -- 172.21.15.7:0/3573154894 >> v2:172.21.15.143:3300/0 conn(0x7f828c14fda0 msgr2=0x7f828c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:08:06.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8291606700 1 --2- 172.21.15.7:0/3573154894 >> v2:172.21.15.143:3300/0 conn(0x7f828c14fda0 0x7f828c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:06.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8291606700 1 -- 172.21.15.7:0/3573154894 >> v1:172.21.15.143:6789/0 conn(0x7f828c14f9d0 legacy=0x7f828c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:08:06.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8291606700 1 -- 172.21.15.7:0/3573154894 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c0c34b0 con 0x7f828c0abe20 2023-03-03T09:08:06.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.286+0000 7f8291606700 1 --2- 172.21.15.7:0/3573154894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 0x7f828c1529c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f828c0709b0 tx=0x7f828400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c83dc7c77f232dc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:08:06.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f828b7fe700 1 -- 172.21.15.7:0/3573154894 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828400e070 con 0x7f828c0abe20 2023-03-03T09:08:06.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f828b7fe700 1 -- 172.21.15.7:0/3573154894 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8284003a20 con 0x7f828c0abe20 2023-03-03T09:08:06.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f828b7fe700 1 -- 172.21.15.7:0/3573154894 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284003dd0 con 0x7f828c0abe20 2023-03-03T09:08:06.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f8293069700 1 -- 172.21.15.7:0/3573154894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 msgr2=0x7f828c1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:06.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f8293069700 1 --2- 172.21.15.7:0/3573154894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 0x7f828c1529c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f828c0709b0 tx=0x7f828400b330 comp rx=0 tx=0).stop 2023-03-03T09:08:06.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f8293069700 1 -- 172.21.15.7:0/3573154894 shutdown_connections 2023-03-03T09:08:06.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f8293069700 1 --2- 172.21.15.7:0/3573154894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 0x7f828c1529c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:06.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f8293069700 1 --2- 172.21.15.7:0/3573154894 >> v2:172.21.15.143:3300/0 conn(0x7f828c14fda0 0x7f828c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:06.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f8293069700 1 -- 172.21.15.7:0/3573154894 >> 172.21.15.7:0/3573154894 conn(0x7f828c0c8900 msgr2=0x7f828c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:08:06.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f8293069700 1 -- 172.21.15.7:0/3573154894 shutdown_connections 2023-03-03T09:08:06.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.287+0000 7f8293069700 1 -- 172.21.15.7:0/3573154894 wait complete. 2023-03-03T09:08:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.288+0000 7f8293069700 1 Processor -- start 2023-03-03T09:08:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.288+0000 7f8293069700 1 -- start start 2023-03-03T09:08:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.289+0000 7f8293069700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 0x7f828c1e8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.289+0000 7f8293069700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f828c14f9d0 0x7f828c152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.289+0000 7f8293069700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f828c14fda0 0x7f828c1db6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.289+0000 7f8293069700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c06f870 con 0x7f828c0abe20 2023-03-03T09:08:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.289+0000 7f8293069700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828c0bda30 con 0x7f828c14fda0 2023-03-03T09:08:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.289+0000 7f8293069700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f828c1e8b80 con 0x7f828c14f9d0 2023-03-03T09:08:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.289+0000 7f8290e05700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 0x7f828c1e8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.289+0000 7f8290e05700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 0x7f828c1e8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54614/0 (socket says 172.21.15.7:54614) 2023-03-03T09:08:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.289+0000 7f8290e05700 1 -- 172.21.15.7:0/3959840970 learned_addr learned my addr 172.21.15.7:0/3959840970 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:08:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f8291606700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f828c14fda0 0x7f828c1db6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f8290e05700 1 -- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f828c14f9d0 msgr2=0x7f828c152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f828bfff700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f828c14f9d0 0x7f828c152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f8290e05700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f828c14f9d0 0x7f828c152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f8290e05700 1 -- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f828c14fda0 msgr2=0x7f828c1db6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f8290e05700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f828c14fda0 0x7f828c1db6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f8290e05700 1 -- 172.21.15.7:0/3959840970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8284009b30 con 0x7f828c0abe20 2023-03-03T09:08:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f8290e05700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 0x7f828c1e8640 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f828000c4d0 tx=0x7f828000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:08:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f82897fa700 1 -- 172.21.15.7:0/3959840970 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8280011790 con 0x7f828c0abe20 2023-03-03T09:08:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f82897fa700 1 -- 172.21.15.7:0/3959840970 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82800118f0 con 0x7f828c0abe20 2023-03-03T09:08:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f82897fa700 1 -- 172.21.15.7:0/3959840970 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8280011ca0 con 0x7f828c0abe20 2023-03-03T09:08:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f8293069700 1 -- 172.21.15.7:0/3959840970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f828c0ac350 con 0x7f828c0abe20 2023-03-03T09:08:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.290+0000 7f8293069700 1 -- 172.21.15.7:0/3959840970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828c0ab630 con 0x7f828c0abe20 2023-03-03T09:08:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.291+0000 7f8293069700 1 -- 172.21.15.7:0/3959840970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828c071fa0 con 0x7f828c0abe20 2023-03-03T09:08:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.292+0000 7f82897fa700 1 -- 172.21.15.7:0/3959840970 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8280010050 con 0x7f828c0abe20 2023-03-03T09:08:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.292+0000 7f82897fa700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82700682f0 0x7f827006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:08:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.292+0000 7f82897fa700 1 -- 172.21.15.7:0/3959840970 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8280085de0 con 0x7f828c0abe20 2023-03-03T09:08:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.292+0000 7f828bfff700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82700682f0 0x7f827006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:08:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.293+0000 7f828bfff700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82700682f0 0x7f827006a7a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f828c1501d0 tx=0x7f8278008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:08:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.296+0000 7f82897fa700 1 -- 172.21.15.7:0/3959840970 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8280052180 con 0x7f828c0abe20 2023-03-03T09:08:06.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.536+0000 7f8293069700 1 -- 172.21.15.7:0/3959840970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f828c1dd440 con 0x7f828c0abe20 2023-03-03T09:08:06.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.537+0000 7f82897fa700 1 -- 172.21.15.7:0/3959840970 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f82800557a0 con 0x7f828c0abe20 2023-03-03T09:08:06.538 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:08:06.538 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:08:06.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 -- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82700682f0 msgr2=0x7f827006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:06.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82700682f0 0x7f827006a7a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f828c1501d0 tx=0x7f8278008040 comp rx=0 tx=0).stop 2023-03-03T09:08:06.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 -- 172.21.15.7:0/3959840970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 msgr2=0x7f828c1e8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:08:06.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 0x7f828c1e8640 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f828000c4d0 tx=0x7f828000c890 comp rx=0 tx=0).stop 2023-03-03T09:08:06.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 -- 172.21.15.7:0/3959840970 shutdown_connections 2023-03-03T09:08:06.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f828c14fda0 0x7f828c1db6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:06.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82700682f0 0x7f827006a7a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:06.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f828c0abe20 0x7f828c1e8640 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:06.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 --2- 172.21.15.7:0/3959840970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f828c14f9d0 0x7f828c152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:08:06.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 -- 172.21.15.7:0/3959840970 >> 172.21.15.7:0/3959840970 conn(0x7f828c0c8900 msgr2=0x7f828c150200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:08:06.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 -- 172.21.15.7:0/3959840970 shutdown_connections 2023-03-03T09:08:06.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:08:06.539+0000 7f8293069700 1 -- 172.21.15.7:0/3959840970 wait complete. 2023-03-03T09:08:06.542 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:08:06.552 INFO:tasks.mds_thrash.fs.[a]:waiting for 97 secs before thrashing 2023-03-03T09:08:11.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:11.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:41.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:08:41.071 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:02.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:09:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.823+0000 7f6448551700 1 Processor -- start 2023-03-03T09:09:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.823+0000 7f6448551700 1 -- start start 2023-03-03T09:09:02.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f6448551700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f644014f860 0x7f64401562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:02.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f6448551700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f64400c20d0 0x7f6440152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:02.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f6448551700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c27c0 0x7f64400736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:02.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f6448551700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644006d350 con 0x7f64400c20d0 2023-03-03T09:09:02.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f6448551700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64401558e0 con 0x7f644014f860 2023-03-03T09:09:02.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f6448551700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6440156830 con 0x7f64400c27c0 2023-03-03T09:09:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f6445aec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f64400c20d0 0x7f6440152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f6445aec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f64400c20d0 0x7f6440152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54666/0 (socket says 172.21.15.7:54666) 2023-03-03T09:09:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f6445aec700 1 -- 172.21.15.7:0/1563936787 learned_addr learned my addr 172.21.15.7:0/1563936787 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:09:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f64462ed700 1 --2- 172.21.15.7:0/1563936787 >> v2:172.21.15.143:3300/0 conn(0x7f644014f860 0x7f64401562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.824+0000 7f6446aee700 1 --2- 172.21.15.7:0/1563936787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c27c0 0x7f64400736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f6445aec700 1 -- 172.21.15.7:0/1563936787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c27c0 msgr2=0x7f64400736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:09:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f6445aec700 1 --2- 172.21.15.7:0/1563936787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c27c0 0x7f64400736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f6445aec700 1 -- 172.21.15.7:0/1563936787 >> v2:172.21.15.143:3300/0 conn(0x7f644014f860 msgr2=0x7f64401562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:09:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f6445aec700 1 --2- 172.21.15.7:0/1563936787 >> v2:172.21.15.143:3300/0 conn(0x7f644014f860 0x7f64401562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f6445aec700 1 -- 172.21.15.7:0/1563936787 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6440155e60 con 0x7f64400c20d0 2023-03-03T09:09:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f64462ed700 1 --2- 172.21.15.7:0/1563936787 >> v2:172.21.15.143:3300/0 conn(0x7f644014f860 0x7f64401562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:09:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f6446aee700 1 --2- 172.21.15.7:0/1563936787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c27c0 0x7f64400736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:09:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f6445aec700 1 --2- 172.21.15.7:0/1563936787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64400c20d0 0x7f6440152d70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f644006fc80 tx=0x7f643c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9d82b2b56ad5ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:09:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f64452eb700 1 -- 172.21.15.7:0/1563936787 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c00b860 con 0x7f64400c20d0 2023-03-03T09:09:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f64452eb700 1 -- 172.21.15.7:0/1563936787 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c00b9c0 con 0x7f64400c20d0 2023-03-03T09:09:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f64452eb700 1 -- 172.21.15.7:0/1563936787 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c00bd50 con 0x7f64400c20d0 2023-03-03T09:09:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f6448551700 1 -- 172.21.15.7:0/1563936787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64400c20d0 msgr2=0x7f6440152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.825+0000 7f6448551700 1 --2- 172.21.15.7:0/1563936787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64400c20d0 0x7f6440152d70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f644006fc80 tx=0x7f643c009c00 comp rx=0 tx=0).stop 2023-03-03T09:09:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.826+0000 7f6448551700 1 -- 172.21.15.7:0/1563936787 shutdown_connections 2023-03-03T09:09:02.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.826+0000 7f6448551700 1 --2- 172.21.15.7:0/1563936787 >> v2:172.21.15.143:3300/0 conn(0x7f644014f860 0x7f64401562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:02.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.826+0000 7f6448551700 1 --2- 172.21.15.7:0/1563936787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c27c0 0x7f64400736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:02.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.826+0000 7f6448551700 1 --2- 172.21.15.7:0/1563936787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64400c20d0 0x7f6440152d70 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:02.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.826+0000 7f6448551700 1 -- 172.21.15.7:0/1563936787 >> 172.21.15.7:0/1563936787 conn(0x7f64400a94d0 msgr2=0x7f64400a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:09:02.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.826+0000 7f6448551700 1 -- 172.21.15.7:0/1563936787 shutdown_connections 2023-03-03T09:09:02.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.826+0000 7f6448551700 1 -- 172.21.15.7:0/1563936787 wait complete. 2023-03-03T09:09:02.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.827+0000 7f6448551700 1 Processor -- start 2023-03-03T09:09:02.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.827+0000 7f6448551700 1 -- start start 2023-03-03T09:09:02.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.827+0000 7f6448551700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c20d0 0x7f64400aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:02.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.827+0000 7f6448551700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64400c27c0 0x7f644014d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:02.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.827+0000 7f6448551700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644014f860 0x7f64400bf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:02.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.827+0000 7f6448551700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64400bde10 con 0x7f644014f860 2023-03-03T09:09:02.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.827+0000 7f6448551700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64400c08a0 con 0x7f64400c27c0 2023-03-03T09:09:02.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.827+0000 7f6448551700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64400bdff0 con 0x7f64400c20d0 2023-03-03T09:09:02.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6446aee700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644014f860 0x7f64400bf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:02.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6446aee700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644014f860 0x7f64400bf240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54674/0 (socket says 172.21.15.7:54674) 2023-03-03T09:09:02.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6446aee700 1 -- 172.21.15.7:0/644606141 learned_addr learned my addr 172.21.15.7:0/644606141 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:09:02.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f64462ed700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c20d0 0x7f64400aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:02.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6445aec700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64400c27c0 0x7f644014d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:02.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6446aee700 1 -- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c20d0 msgr2=0x7f64400aa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:02.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6446aee700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c20d0 0x7f64400aa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:02.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6446aee700 1 -- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64400c27c0 msgr2=0x7f644014d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:02.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6446aee700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64400c27c0 0x7f644014d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:02.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6446aee700 1 -- 172.21.15.7:0/644606141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c009550 con 0x7f644014f860 2023-03-03T09:09:02.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6446aee700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644014f860 0x7f64400bf240 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f643800cf20 tx=0x7f643800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:09:02.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6436ffd700 1 -- 172.21.15.7:0/644606141 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643800aca0 con 0x7f644014f860 2023-03-03T09:09:02.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6436ffd700 1 -- 172.21.15.7:0/644606141 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643800ae00 con 0x7f644014f860 2023-03-03T09:09:02.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6436ffd700 1 -- 172.21.15.7:0/644606141 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6438015680 con 0x7f644014f860 2023-03-03T09:09:02.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6448551700 1 -- 172.21.15.7:0/644606141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644006cd90 con 0x7f644014f860 2023-03-03T09:09:02.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.828+0000 7f6448551700 1 -- 172.21.15.7:0/644606141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64401532b0 con 0x7f644014f860 2023-03-03T09:09:02.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.829+0000 7f6448551700 1 -- 172.21.15.7:0/644606141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644006d070 con 0x7f644014f860 2023-03-03T09:09:02.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.830+0000 7f6436ffd700 1 -- 172.21.15.7:0/644606141 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f643801c030 con 0x7f644014f860 2023-03-03T09:09:02.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.830+0000 7f6436ffd700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64240682f0 0x7f642406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:02.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.830+0000 7f6436ffd700 1 -- 172.21.15.7:0/644606141 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f64380864b0 con 0x7f644014f860 2023-03-03T09:09:02.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.830+0000 7f64462ed700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64240682f0 0x7f642406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:02.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.831+0000 7f64462ed700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64240682f0 0x7f642406a7a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6430006fd0 tx=0x7f6430008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:09:02.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:02.835+0000 7f6436ffd700 1 -- 172.21.15.7:0/644606141 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6438052700 con 0x7f644014f860 2023-03-03T09:09:03.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.074+0000 7f6448551700 1 -- 172.21.15.7:0/644606141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6440074bb0 con 0x7f644014f860 2023-03-03T09:09:03.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.075+0000 7f6436ffd700 1 -- 172.21.15.7:0/644606141 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f643801a090 con 0x7f644014f860 2023-03-03T09:09:03.076 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:09:03.076 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:09:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 -- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64240682f0 msgr2=0x7f642406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64240682f0 0x7f642406a7a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6430006fd0 tx=0x7f6430008040 comp rx=0 tx=0).stop 2023-03-03T09:09:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 -- 172.21.15.7:0/644606141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644014f860 msgr2=0x7f64400bf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644014f860 0x7f64400bf240 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f643800cf20 tx=0x7f643800cf50 comp rx=0 tx=0).stop 2023-03-03T09:09:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 -- 172.21.15.7:0/644606141 shutdown_connections 2023-03-03T09:09:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64240682f0 0x7f642406a7a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644014f860 0x7f64400bf240 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64400c20d0 0x7f64400aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 --2- 172.21.15.7:0/644606141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64400c27c0 0x7f644014d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 -- 172.21.15.7:0/644606141 >> 172.21.15.7:0/644606141 conn(0x7f64400a94d0 msgr2=0x7f6440155100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:09:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 -- 172.21.15.7:0/644606141 shutdown_connections 2023-03-03T09:09:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:03.077+0000 7f6448551700 1 -- 172.21.15.7:0/644606141 wait complete. 2023-03-03T09:09:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:09:03.090 INFO:tasks.mds_thrash.fs.[b]:waiting for 41 secs before thrashing 2023-03-03T09:09:11.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:11.141 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:41.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:41.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:09:43.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:09:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.692+0000 7f8cf7bbb700 1 Processor -- start 2023-03-03T09:09:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.692+0000 7f8cf7bbb700 1 -- start start 2023-03-03T09:09:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf7bbb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8cf014fda0 0x7f8cf0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf7bbb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8cf00abe20 0x7f8cf01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf7bbb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf00c2f30 con 0x7f8cf014f9d0 2023-03-03T09:09:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf7bbb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf00c31d0 con 0x7f8cf014fda0 2023-03-03T09:09:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf7bbb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf01524a0 con 0x7f8cf00abe20 2023-03-03T09:09:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf5156700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8cf014fda0 0x7f8cf0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf5957700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8cf014f9d0 0x7f8cf0071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48694/0 (socket says 172.21.15.7:48694) 2023-03-03T09:09:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf5957700 1 -- 172.21.15.7:0/4032555216 learned_addr learned my addr 172.21.15.7:0/4032555216 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:09:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf6158700 1 --2- 172.21.15.7:0/4032555216 >> v2:172.21.15.143:3300/0 conn(0x7f8cf00abe20 0x7f8cf01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf5156700 1 -- 172.21.15.7:0/4032555216 >> v2:172.21.15.143:3300/0 conn(0x7f8cf00abe20 msgr2=0x7f8cf01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:09:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf5156700 1 --2- 172.21.15.7:0/4032555216 >> v2:172.21.15.143:3300/0 conn(0x7f8cf00abe20 0x7f8cf01529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf5156700 1 -- 172.21.15.7:0/4032555216 >> v1:172.21.15.7:6789/0 conn(0x7f8cf014f9d0 legacy=0x7f8cf0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf5156700 1 -- 172.21.15.7:0/4032555216 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf00c34b0 con 0x7f8cf014fda0 2023-03-03T09:09:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf4955700 1 -- 172.21.15.7:0/4032555216 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483585404 0 0) 0x7f8cf00c2f30 con 0x7f8cf014f9d0 2023-03-03T09:09:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf5156700 1 --2- 172.21.15.7:0/4032555216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 0x7f8cf0156430 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0071230 tx=0x7f8ce0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ca530c6cfc1d7c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:09:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.693+0000 7f8cf4955700 1 -- 172.21.15.7:0/4032555216 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0011830 con 0x7f8cf014fda0 2023-03-03T09:09:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.694+0000 7f8cf4955700 1 -- 172.21.15.7:0/4032555216 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce0011990 con 0x7f8cf014fda0 2023-03-03T09:09:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.694+0000 7f8cf4955700 1 -- 172.21.15.7:0/4032555216 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0011d20 con 0x7f8cf014fda0 2023-03-03T09:09:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.694+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/4032555216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 msgr2=0x7f8cf0156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.694+0000 7f8cf7bbb700 1 --2- 172.21.15.7:0/4032555216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 0x7f8cf0156430 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8cf0071230 tx=0x7f8ce0009b20 comp rx=0 tx=0).stop 2023-03-03T09:09:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.694+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/4032555216 shutdown_connections 2023-03-03T09:09:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.694+0000 7f8cf7bbb700 1 --2- 172.21.15.7:0/4032555216 >> v2:172.21.15.143:3300/0 conn(0x7f8cf00abe20 0x7f8cf01529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.694+0000 7f8cf7bbb700 1 --2- 172.21.15.7:0/4032555216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 0x7f8cf0156430 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.694+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/4032555216 >> 172.21.15.7:0/4032555216 conn(0x7f8cf00c8900 msgr2=0x7f8cf00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:09:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.694+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/4032555216 shutdown_connections 2023-03-03T09:09:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.694+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/4032555216 wait complete. 2023-03-03T09:09:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.695+0000 7f8cf7bbb700 1 Processor -- start 2023-03-03T09:09:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.695+0000 7f8cf7bbb700 1 -- start start 2023-03-03T09:09:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf7bbb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cf00abe20 0x7f8cf01e7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf7bbb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cf014f9d0 0x7f8cf01549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf7bbb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 0x7f8cf014acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:43.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf7bbb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf006e880 con 0x7f8cf014fda0 2023-03-03T09:09:43.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf7bbb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf01e8410 con 0x7f8cf00abe20 2023-03-03T09:09:43.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf7bbb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf00bdb30 con 0x7f8cf014f9d0 2023-03-03T09:09:43.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf6158700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 0x7f8cf014acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:43.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf6158700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 0x7f8cf014acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54724/0 (socket says 172.21.15.7:54724) 2023-03-03T09:09:43.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf6158700 1 -- 172.21.15.7:0/1549001165 learned_addr learned my addr 172.21.15.7:0/1549001165 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:09:43.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf5957700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cf00abe20 0x7f8cf01e7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:43.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf5156700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cf014f9d0 0x7f8cf01549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:43.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.696+0000 7f8cf6158700 1 -- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cf014f9d0 msgr2=0x7f8cf01549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:43.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.697+0000 7f8cf6158700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cf014f9d0 0x7f8cf01549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.697+0000 7f8cf6158700 1 -- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cf00abe20 msgr2=0x7f8cf01e7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:43.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.697+0000 7f8cf6158700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cf00abe20 0x7f8cf01e7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.697+0000 7f8cf6158700 1 -- 172.21.15.7:0/1549001165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce00094a0 con 0x7f8cf014fda0 2023-03-03T09:09:43.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.697+0000 7f8cf6158700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 0x7f8cf014acd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009fd0 tx=0x7f8ce800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:09:43.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.697+0000 7f8ce67fc700 1 -- 172.21.15.7:0/1549001165 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8007ea0 con 0x7f8cf014fda0 2023-03-03T09:09:43.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.697+0000 7f8ce67fc700 1 -- 172.21.15.7:0/1549001165 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce800ed30 con 0x7f8cf014fda0 2023-03-03T09:09:43.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.697+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/1549001165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf00ac350 con 0x7f8cf014fda0 2023-03-03T09:09:43.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.697+0000 7f8ce67fc700 1 -- 172.21.15.7:0/1549001165 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce800eea0 con 0x7f8cf014fda0 2023-03-03T09:09:43.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.697+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/1549001165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf00ab630 con 0x7f8cf014fda0 2023-03-03T09:09:43.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.698+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/1549001165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf0071fa0 con 0x7f8cf014fda0 2023-03-03T09:09:43.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.699+0000 7f8ce67fc700 1 -- 172.21.15.7:0/1549001165 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8ce8012050 con 0x7f8cf014fda0 2023-03-03T09:09:43.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.699+0000 7f8ce67fc700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd40682f0 0x7f8cd406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:43.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.699+0000 7f8ce67fc700 1 -- 172.21.15.7:0/1549001165 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8ce80866c0 con 0x7f8cf014fda0 2023-03-03T09:09:43.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.699+0000 7f8cf5957700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd40682f0 0x7f8cd406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:43.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.700+0000 7f8cf5957700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd40682f0 0x7f8cd406a7a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8cec007fd0 tx=0x7f8cec007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:09:43.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.703+0000 7f8ce67fc700 1 -- 172.21.15.7:0/1549001165 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8ce8052970 con 0x7f8cf014fda0 2023-03-03T09:09:43.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.940+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/1549001165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cf004e6e0 con 0x7f8cf014fda0 2023-03-03T09:09:43.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.940+0000 7f8ce67fc700 1 -- 172.21.15.7:0/1549001165 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8ce8055f90 con 0x7f8cf014fda0 2023-03-03T09:09:43.942 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:09:43.942 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:09:43.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.942+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd40682f0 msgr2=0x7f8cd406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:43.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.942+0000 7f8cf7bbb700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd40682f0 0x7f8cd406a7a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8cec007fd0 tx=0x7f8cec007de0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.942+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/1549001165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 msgr2=0x7f8cf014acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:43.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.942+0000 7f8cf7bbb700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 0x7f8cf014acd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009fd0 tx=0x7f8ce800e3c0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.943+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/1549001165 shutdown_connections 2023-03-03T09:09:43.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.943+0000 7f8cf7bbb700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd40682f0 0x7f8cd406a7a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.943+0000 7f8cf7bbb700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cf014fda0 0x7f8cf014acd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.943+0000 7f8cf7bbb700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cf00abe20 0x7f8cf01e7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.943+0000 7f8cf7bbb700 1 --2- 172.21.15.7:0/1549001165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cf014f9d0 0x7f8cf01549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:43.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.943+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/1549001165 >> 172.21.15.7:0/1549001165 conn(0x7f8cf00c8900 msgr2=0x7f8cf014f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:09:43.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.943+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/1549001165 shutdown_connections 2023-03-03T09:09:43.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:43.943+0000 7f8cf7bbb700 1 -- 172.21.15.7:0/1549001165 wait complete. 2023-03-03T09:09:43.946 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:09:43.956 INFO:tasks.mds_thrash.fs.[a]:waiting for 23 secs before thrashing 2023-03-03T09:09:44.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:09:44.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.215+0000 7f3c64bd8700 1 Processor -- start 2023-03-03T09:09:44.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.215+0000 7f3c64bd8700 1 -- start start 2023-03-03T09:09:44.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.215+0000 7f3c64bd8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014f9d0 0x7f3c60156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.215+0000 7f3c64bd8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3c6014fda0 0x7f3c60071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.215+0000 7f3c64bd8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c600c2f30 con 0x7f3c600abe20 2023-03-03T09:09:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.215+0000 7f3c64bd8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c600c31d0 con 0x7f3c6014fda0 2023-03-03T09:09:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.215+0000 7f3c64bd8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601524a0 con 0x7f3c6014f9d0 2023-03-03T09:09:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5ed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3c600abe20 0x7f3c601529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48712/0 (socket says 172.21.15.7:48712) 2023-03-03T09:09:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5ed9d700 1 -- 172.21.15.7:0/959779050 learned_addr learned my addr 172.21.15.7:0/959779050 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:09:44.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5e59c700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014f9d0 0x7f3c60156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:44.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5dd9b700 1 --2- 172.21.15.7:0/959779050 >> v2:172.21.15.143:3300/0 conn(0x7f3c6014fda0 0x7f3c60071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:44.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244324645 0 0) 0x7f3c600c2f30 con 0x7f3c600abe20 2023-03-03T09:09:44.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c4c0035b0 con 0x7f3c600abe20 2023-03-03T09:09:44.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 563595726 0 0) 0x7f3c4c0035b0 con 0x7f3c600abe20 2023-03-03T09:09:44.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014f9d0 msgr2=0x7f3c60156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:09:44.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014f9d0 0x7f3c60156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:44.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 >> v2:172.21.15.143:3300/0 conn(0x7f3c6014fda0 msgr2=0x7f3c60071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:09:44.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 --2- 172.21.15.7:0/959779050 >> v2:172.21.15.143:3300/0 conn(0x7f3c6014fda0 0x7f3c60071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:44.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c600c34b0 con 0x7f3c600abe20 2023-03-03T09:09:44.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f3c580019f0 con 0x7f3c600abe20 2023-03-03T09:09:44.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 >> v1:172.21.15.7:6789/0 conn(0x7f3c600abe20 legacy=0x7f3c601529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:44.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c4c003a70 0x7f3c4c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:44.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014f9d0 0x7f3c4c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:44.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c004d90 0x7f3c4c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:44.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c600c31d0 con 0x7f3c4c003a70 2023-03-03T09:09:44.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601524a0 con 0x7f3c4c004d90 2023-03-03T09:09:44.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c00baf0 con 0x7f3c6014f9d0 2023-03-03T09:09:44.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.216+0000 7f3c5e59c700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c4c003a70 0x7f3c4c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:44.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5dd9b700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014f9d0 0x7f3c4c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:44.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5ed9d700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c004d90 0x7f3c4c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:44.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5e59c700 1 -- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014f9d0 msgr2=0x7f3c4c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:09:44.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5e59c700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014f9d0 0x7f3c4c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:44.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5e59c700 1 -- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c004d90 msgr2=0x7f3c4c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:09:44.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5e59c700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c004d90 0x7f3c4c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:44.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5e59c700 1 -- 172.21.15.7:0/959779050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c0035b0 con 0x7f3c4c003a70 2023-03-03T09:09:44.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5e59c700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c4c003a70 0x7f3c4c003e80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009b60 tx=0x7f3c4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7511115512cf41e5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:09:44.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5000e6c0 con 0x7f3c4c003a70 2023-03-03T09:09:44.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5000e820 con 0x7f3c4c003a70 2023-03-03T09:09:44.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5d59a700 1 -- 172.21.15.7:0/959779050 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5000eb90 con 0x7f3c4c003a70 2023-03-03T09:09:44.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c64bd8700 1 -- 172.21.15.7:0/959779050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c4c003a70 msgr2=0x7f3c4c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:44.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c64bd8700 1 --2- 172.21.15.7:0/959779050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c4c003a70 0x7f3c4c003e80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009b60 tx=0x7f3c4c005880 comp rx=0 tx=0).stop 2023-03-03T09:09:44.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.217+0000 7f3c5ed9d700 1 -- 172.21.15.7:0/959779050 reap_dead start 2023-03-03T09:09:44.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.218+0000 7f3c64bd8700 1 -- 172.21.15.7:0/959779050 shutdown_connections 2023-03-03T09:09:44.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.218+0000 7f3c64bd8700 1 -- 172.21.15.7:0/959779050 >> 172.21.15.7:0/959779050 conn(0x7f3c600c8900 msgr2=0x7f3c600c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:09:44.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.218+0000 7f3c64bd8700 1 -- 172.21.15.7:0/959779050 shutdown_connections 2023-03-03T09:09:44.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.218+0000 7f3c64bd8700 1 -- 172.21.15.7:0/959779050 wait complete. 2023-03-03T09:09:44.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.219+0000 7f3c64bd8700 1 Processor -- start 2023-03-03T09:09:44.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.219+0000 7f3c64bd8700 1 -- start start 2023-03-03T09:09:44.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c64bd8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c003a70 0x7f3c6006f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:44.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c64bd8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014c240 0x7f3c60148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:44.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c64bd8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c6006c830 0x7f3c601eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:44.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c64bd8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601df2e0 con 0x7f3c6006c830 2023-03-03T09:09:44.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c64bd8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601df090 con 0x7f3c4c003a70 2023-03-03T09:09:44.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c64bd8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6014c6b0 con 0x7f3c6014c240 2023-03-03T09:09:44.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c6006c830 0x7f3c601eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c6006c830 0x7f3c601eaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54744/0 (socket says 172.21.15.7:54744) 2023-03-03T09:09:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5ed9d700 1 -- 172.21.15.7:0/1718163583 learned_addr learned my addr 172.21.15.7:0/1718163583 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:09:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5e59c700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c003a70 0x7f3c6006f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5dd9b700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014c240 0x7f3c60148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5ed9d700 1 -- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014c240 msgr2=0x7f3c60148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5ed9d700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014c240 0x7f3c60148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5ed9d700 1 -- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c003a70 msgr2=0x7f3c6006f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:44.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5ed9d700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c003a70 0x7f3c6006f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5ed9d700 1 -- 172.21.15.7:0/1718163583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c600ac350 con 0x7f3c6006c830 2023-03-03T09:09:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5e59c700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c003a70 0x7f3c6006f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:09:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5dd9b700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014c240 0x7f3c60148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:09:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c5ed9d700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c6006c830 0x7f3c601eaab0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3c600ab150 tx=0x7f3c58007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:09:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c56ffd700 1 -- 172.21.15.7:0/1718163583 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58017070 con 0x7f3c6006c830 2023-03-03T09:09:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c56ffd700 1 -- 172.21.15.7:0/1718163583 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5800eb60 con 0x7f3c6006c830 2023-03-03T09:09:44.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c64bd8700 1 -- 172.21.15.7:0/1718163583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c600ab630 con 0x7f3c6006c830 2023-03-03T09:09:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c56ffd700 1 -- 172.21.15.7:0/1718163583 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58001f00 con 0x7f3c6006c830 2023-03-03T09:09:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.220+0000 7f3c64bd8700 1 -- 172.21.15.7:0/1718163583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c600c3e10 con 0x7f3c6006c830 2023-03-03T09:09:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.221+0000 7f3c64bd8700 1 -- 172.21.15.7:0/1718163583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c60074bb0 con 0x7f3c6006c830 2023-03-03T09:09:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.222+0000 7f3c56ffd700 1 -- 172.21.15.7:0/1718163583 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3c5800ed10 con 0x7f3c6006c830 2023-03-03T09:09:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.222+0000 7f3c56ffd700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c400682f0 0x7f3c4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:09:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.222+0000 7f3c56ffd700 1 -- 172.21.15.7:0/1718163583 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3c58085f10 con 0x7f3c6006c830 2023-03-03T09:09:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.222+0000 7f3c5e59c700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c400682f0 0x7f3c4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:09:44.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.223+0000 7f3c5e59c700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c400682f0 0x7f3c4006a7a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3c600aa470 tx=0x7f3c50008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:09:44.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.226+0000 7f3c56ffd700 1 -- 172.21.15.7:0/1718163583 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c58052250 con 0x7f3c6006c830 2023-03-03T09:09:44.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.464+0000 7f3c64bd8700 1 -- 172.21.15.7:0/1718163583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c6014bf60 con 0x7f3c6006c830 2023-03-03T09:09:44.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.465+0000 7f3c56ffd700 1 -- 172.21.15.7:0/1718163583 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3c6014bf60 con 0x7f3c6006c830 2023-03-03T09:09:44.466 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:09:44.466 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:09:44.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 -- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c400682f0 msgr2=0x7f3c4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:44.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c400682f0 0x7f3c4006a7a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3c600aa470 tx=0x7f3c50008030 comp rx=0 tx=0).stop 2023-03-03T09:09:44.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 -- 172.21.15.7:0/1718163583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c6006c830 msgr2=0x7f3c601eaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:09:44.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c6006c830 0x7f3c601eaab0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3c600ab150 tx=0x7f3c58007c90 comp rx=0 tx=0).stop 2023-03-03T09:09:44.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 -- 172.21.15.7:0/1718163583 shutdown_connections 2023-03-03T09:09:44.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c400682f0 0x7f3c4006a7a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:44.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c6006c830 0x7f3c601eaab0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:44.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c4c003a70 0x7f3c6006f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:44.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 --2- 172.21.15.7:0/1718163583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c6014c240 0x7f3c60148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:09:44.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 -- 172.21.15.7:0/1718163583 >> 172.21.15.7:0/1718163583 conn(0x7f3c600c8900 msgr2=0x7f3c60150c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:09:44.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 -- 172.21.15.7:0/1718163583 shutdown_connections 2023-03-03T09:09:44.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:09:44.467+0000 7f3c64bd8700 1 -- 172.21.15.7:0/1718163583 wait complete. 2023-03-03T09:09:44.470 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:09:44.480 INFO:tasks.mds_thrash.fs.[b]:waiting for 46 secs before thrashing 2023-03-03T09:10:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T09:09:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T09:10:06.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:10:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.095+0000 7f080b76c700 1 Processor -- start 2023-03-03T09:10:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.095+0000 7f080b76c700 1 -- start start 2023-03-03T09:10:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.096+0000 7f080b76c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f080414b340 0x7f0804152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.096+0000 7f080b76c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 0x7f0804152b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.096+0000 7f080b76c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080406c920 con 0x7f0804151140 2023-03-03T09:10:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.096+0000 7f080b76c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08040abd90 con 0x7f080414b340 2023-03-03T09:10:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.096+0000 7f080b76c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08040c1300 con 0x7f08040c2790 2023-03-03T09:10:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f0808d07700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 0x7f0804152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f0809508700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f080414b340 0x7f0804152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f0808d07700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 0x7f0804152b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:34598/0 (socket says 172.21.15.7:34598) 2023-03-03T09:10:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f0809508700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f080414b340 0x7f0804152630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:50532/0 (socket says 172.21.15.7:50532) 2023-03-03T09:10:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f0808d07700 1 -- 172.21.15.7:0/1268087003 learned_addr learned my addr 172.21.15.7:0/1268087003 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:10:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f0809508700 1 -- 172.21.15.7:0/1268087003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 msgr2=0x7f0804152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:10:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f0809508700 1 --2- 172.21.15.7:0/1268087003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 0x7f0804152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f0809508700 1 -- 172.21.15.7:0/1268087003 >> v1:172.21.15.143:6789/0 conn(0x7f0804151140 legacy=0x7f0804151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:07.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f0809508700 1 -- 172.21.15.7:0/1268087003 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080406bd50 con 0x7f080414b340 2023-03-03T09:10:07.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f07fbfff700 1 -- 172.21.15.7:0/1268087003 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154093756 0 0) 0x7f080406c920 con 0x7f0804151140 2023-03-03T09:10:07.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f0809508700 1 --2- 172.21.15.7:0/1268087003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f080414b340 0x7f0804152630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009840 tx=0x7f07f4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=360b4fb7693559 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:07.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f07fbfff700 1 -- 172.21.15.7:0/1268087003 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f4011830 con 0x7f080414b340 2023-03-03T09:10:07.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f07fbfff700 1 -- 172.21.15.7:0/1268087003 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f4011990 con 0x7f080414b340 2023-03-03T09:10:07.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.097+0000 7f07fbfff700 1 -- 172.21.15.7:0/1268087003 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f4011d20 con 0x7f080414b340 2023-03-03T09:10:07.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.098+0000 7f080b76c700 1 -- 172.21.15.7:0/1268087003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f080414b340 msgr2=0x7f0804152630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:07.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.098+0000 7f080b76c700 1 --2- 172.21.15.7:0/1268087003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f080414b340 0x7f0804152630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009840 tx=0x7f07f4009b50 comp rx=0 tx=0).stop 2023-03-03T09:10:07.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.098+0000 7f080b76c700 1 -- 172.21.15.7:0/1268087003 shutdown_connections 2023-03-03T09:10:07.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.098+0000 7f080b76c700 1 --2- 172.21.15.7:0/1268087003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f080414b340 0x7f0804152630 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:07.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.098+0000 7f080b76c700 1 --2- 172.21.15.7:0/1268087003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 0x7f0804152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:07.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.098+0000 7f080b76c700 1 -- 172.21.15.7:0/1268087003 >> 172.21.15.7:0/1268087003 conn(0x7f08040c62e0 msgr2=0x7f08040c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:10:07.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.098+0000 7f080b76c700 1 -- 172.21.15.7:0/1268087003 shutdown_connections 2023-03-03T09:10:07.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.098+0000 7f080b76c700 1 -- 172.21.15.7:0/1268087003 wait complete. 2023-03-03T09:10:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.100+0000 7f080b76c700 1 Processor -- start 2023-03-03T09:10:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.100+0000 7f080b76c700 1 -- start start 2023-03-03T09:10:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f080b76c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 0x7f0804070f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f080b76c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f080414b340 0x7f08041ea450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f080b76c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0804151140 0x7f08041ea990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f080b76c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0804071a40 con 0x7f0804151140 2023-03-03T09:10:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f080b76c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08041eaed0 con 0x7f080414b340 2023-03-03T09:10:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f080b76c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080414aad0 con 0x7f08040c2790 2023-03-03T09:10:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0809d09700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0804151140 0x7f08041ea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0809d09700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0804151140 0x7f08041ea990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54768/0 (socket says 172.21.15.7:54768) 2023-03-03T09:10:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0809d09700 1 -- 172.21.15.7:0/4165081339 learned_addr learned my addr 172.21.15.7:0/4165081339 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:10:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0808d07700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f080414b340 0x7f08041ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0809508700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 0x7f0804070f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0809d09700 1 -- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 msgr2=0x7f0804070f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0809d09700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 0x7f0804070f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0809d09700 1 -- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f080414b340 msgr2=0x7f08041ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0809d09700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f080414b340 0x7f08041ea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0809d09700 1 -- 172.21.15.7:0/4165081339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f40094a0 con 0x7f0804151140 2023-03-03T09:10:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f0809d09700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0804151140 0x7f08041ea990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00e9b0 tx=0x7f07fc00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:07.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.101+0000 7f07f9ffb700 1 -- 172.21.15.7:0/4165081339 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc003e30 con 0x7f0804151140 2023-03-03T09:10:07.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.102+0000 7f07f9ffb700 1 -- 172.21.15.7:0/4165081339 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07fc00ee50 con 0x7f0804151140 2023-03-03T09:10:07.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.102+0000 7f080b76c700 1 -- 172.21.15.7:0/4165081339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080406c2d0 con 0x7f0804151140 2023-03-03T09:10:07.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.102+0000 7f07f9ffb700 1 -- 172.21.15.7:0/4165081339 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc0052b0 con 0x7f0804151140 2023-03-03T09:10:07.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.102+0000 7f080b76c700 1 -- 172.21.15.7:0/4165081339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080406bff0 con 0x7f0804151140 2023-03-03T09:10:07.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.103+0000 7f080b76c700 1 -- 172.21.15.7:0/4165081339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080406c5b0 con 0x7f0804151140 2023-03-03T09:10:07.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.103+0000 7f07f9ffb700 1 -- 172.21.15.7:0/4165081339 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f07fc021070 con 0x7f0804151140 2023-03-03T09:10:07.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.103+0000 7f07f9ffb700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07e80682f0 0x7f07e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:07.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.103+0000 7f07f9ffb700 1 -- 172.21.15.7:0/4165081339 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f07fc087f10 con 0x7f0804151140 2023-03-03T09:10:07.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.104+0000 7f0809508700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07e80682f0 0x7f07e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:07.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.104+0000 7f0809508700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07e80682f0 0x7f07e806a7a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009fd0 tx=0x7f07f401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:07.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.108+0000 7f07f9ffb700 1 -- 172.21.15.7:0/4165081339 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f07fc0542b0 con 0x7f0804151140 2023-03-03T09:10:07.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.344+0000 7f080b76c700 1 -- 172.21.15.7:0/4165081339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0804074bb0 con 0x7f0804151140 2023-03-03T09:10:07.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.345+0000 7f07f9ffb700 1 -- 172.21.15.7:0/4165081339 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f07fc015090 con 0x7f0804151140 2023-03-03T09:10:07.346 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:10:07.346 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:10:07.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 -- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07e80682f0 msgr2=0x7f07e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:07.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07e80682f0 0x7f07e806a7a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009fd0 tx=0x7f07f401c040 comp rx=0 tx=0).stop 2023-03-03T09:10:07.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 -- 172.21.15.7:0/4165081339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0804151140 msgr2=0x7f08041ea990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:07.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0804151140 0x7f08041ea990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00e9b0 tx=0x7f07fc00ecc0 comp rx=0 tx=0).stop 2023-03-03T09:10:07.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 -- 172.21.15.7:0/4165081339 shutdown_connections 2023-03-03T09:10:07.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07e80682f0 0x7f07e806a7a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:07.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0804151140 0x7f08041ea990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:07.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08040c2790 0x7f0804070f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:07.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 --2- 172.21.15.7:0/4165081339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f080414b340 0x7f08041ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:07.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 -- 172.21.15.7:0/4165081339 >> 172.21.15.7:0/4165081339 conn(0x7f08040c62e0 msgr2=0x7f0804152ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:10:07.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 -- 172.21.15.7:0/4165081339 shutdown_connections 2023-03-03T09:10:07.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:07.347+0000 7f080b76c700 1 -- 172.21.15.7:0/4165081339 wait complete. 2023-03-03T09:10:07.350 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:10:07.360 INFO:tasks.mds_thrash.fs.[a]:waiting for 12 secs before thrashing 2023-03-03T09:10:11.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:11.320 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:19.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:10:19.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.499+0000 7f051d0fb700 1 Processor -- start 2023-03-03T09:10:19.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.499+0000 7f051d0fb700 1 -- start start 2023-03-03T09:10:19.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.499+0000 7f051d0fb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f051814e900 0x7f051815ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:19.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.499+0000 7f051d0fb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05181510d0 con 0x7f0518071e80 2023-03-03T09:10:19.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.499+0000 7f051d0fb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05180c3740 con 0x7f0518072250 2023-03-03T09:10:19.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.499+0000 7f051d0fb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0518070670 con 0x7f051814e900 2023-03-03T09:10:19.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.499+0000 7f051759e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f051814e900 0x7f051815ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:19.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.500+0000 7f0516d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0518071e80 0x7f0518151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48754/0 (socket says 172.21.15.7:48754) 2023-03-03T09:10:19.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.500+0000 7f0516d9d700 1 -- 172.21.15.7:0/438400559 learned_addr learned my addr 172.21.15.7:0/438400559 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:10:19.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.500+0000 7f051759e700 1 -- 172.21.15.7:0/438400559 >> v1:172.21.15.143:6789/0 conn(0x7f0518072250 legacy=0x7f051814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:19.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.500+0000 7f051759e700 1 -- 172.21.15.7:0/438400559 >> v1:172.21.15.7:6789/0 conn(0x7f0518071e80 legacy=0x7f0518151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:19.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.500+0000 7f051759e700 1 -- 172.21.15.7:0/438400559 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05180c39e0 con 0x7f051814e900 2023-03-03T09:10:19.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.500+0000 7f0515d9b700 1 -- 172.21.15.7:0/438400559 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528698531 0 0) 0x7f05181510d0 con 0x7f0518071e80 2023-03-03T09:10:19.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.500+0000 7f051759e700 1 --2- 172.21.15.7:0/438400559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 0x7f051815ca60 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0510009d90 tx=0x7f051000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fee8d8ccd657ac2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:19.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.500+0000 7f0515d9b700 1 -- 172.21.15.7:0/438400559 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051000e070 con 0x7f051814e900 2023-03-03T09:10:19.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.500+0000 7f0515d9b700 1 -- 172.21.15.7:0/438400559 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051000bbd0 con 0x7f051814e900 2023-03-03T09:10:19.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.500+0000 7f0515d9b700 1 -- 172.21.15.7:0/438400559 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510012430 con 0x7f051814e900 2023-03-03T09:10:19.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.501+0000 7f051d0fb700 1 -- 172.21.15.7:0/438400559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 msgr2=0x7f051815ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:19.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.501+0000 7f051d0fb700 1 --2- 172.21.15.7:0/438400559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 0x7f051815ca60 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0510009d90 tx=0x7f051000b330 comp rx=0 tx=0).stop 2023-03-03T09:10:19.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.501+0000 7f051d0fb700 1 -- 172.21.15.7:0/438400559 shutdown_connections 2023-03-03T09:10:19.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.501+0000 7f051d0fb700 1 --2- 172.21.15.7:0/438400559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 0x7f051815ca60 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:19.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.501+0000 7f051d0fb700 1 -- 172.21.15.7:0/438400559 >> 172.21.15.7:0/438400559 conn(0x7f05180a94d0 msgr2=0x7f05180a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:10:19.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.501+0000 7f051d0fb700 1 -- 172.21.15.7:0/438400559 shutdown_connections 2023-03-03T09:10:19.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.501+0000 7f051d0fb700 1 -- 172.21.15.7:0/438400559 wait complete. 2023-03-03T09:10:19.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.502+0000 7f051d0fb700 1 Processor -- start 2023-03-03T09:10:19.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.502+0000 7f051d0fb700 1 -- start start 2023-03-03T09:10:19.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f051d0fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0518071e80 0x7f05181e4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:19.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f051d0fb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0518072250 0x7f0518149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:19.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f051d0fb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 0x7f0518145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:19.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f051d0fb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0518149480 con 0x7f051814e900 2023-03-03T09:10:19.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f051d0fb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051814b2d0 con 0x7f0518072250 2023-03-03T09:10:19.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f051d0fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051814b5b0 con 0x7f0518071e80 2023-03-03T09:10:19.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f051759e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 0x7f0518145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:19.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f051759e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 0x7f0518145100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54784/0 (socket says 172.21.15.7:54784) 2023-03-03T09:10:19.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f051759e700 1 -- 172.21.15.7:0/2608351615 learned_addr learned my addr 172.21.15.7:0/2608351615 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:10:19.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f051659c700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0518072250 0x7f0518149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:19.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.503+0000 7f0516d9d700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0518071e80 0x7f05181e4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:19.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f051759e700 1 -- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0518071e80 msgr2=0x7f05181e4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:19.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f051759e700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0518071e80 0x7f05181e4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:19.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f051759e700 1 -- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0518072250 msgr2=0x7f0518149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:19.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f051759e700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0518072250 0x7f0518149910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:19.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f051759e700 1 -- 172.21.15.7:0/2608351615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0510009a70 con 0x7f051814e900 2023-03-03T09:10:19.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f0516d9d700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0518071e80 0x7f05181e4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:10:19.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f051759e700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 0x7f0518145100 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0510014040 tx=0x7f0510004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:19.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f04ff7fe700 1 -- 172.21.15.7:0/2608351615 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510012430 con 0x7f051814e900 2023-03-03T09:10:19.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f051d0fb700 1 -- 172.21.15.7:0/2608351615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0518151f30 con 0x7f051814e900 2023-03-03T09:10:19.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f04ff7fe700 1 -- 172.21.15.7:0/2608351615 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0510020070 con 0x7f051814e900 2023-03-03T09:10:19.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f04ff7fe700 1 -- 172.21.15.7:0/2608351615 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510007000 con 0x7f051814e900 2023-03-03T09:10:19.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.504+0000 7f051d0fb700 1 -- 172.21.15.7:0/2608351615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0518151300 con 0x7f051814e900 2023-03-03T09:10:19.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.505+0000 7f051d0fb700 1 -- 172.21.15.7:0/2608351615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051806c9a0 con 0x7f051814e900 2023-03-03T09:10:19.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.506+0000 7f04ff7fe700 1 -- 172.21.15.7:0/2608351615 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0510004280 con 0x7f051814e900 2023-03-03T09:10:19.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.506+0000 7f04ff7fe700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05040682f0 0x7f050406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:19.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.506+0000 7f04ff7fe700 1 -- 172.21.15.7:0/2608351615 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0510022070 con 0x7f051814e900 2023-03-03T09:10:19.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.506+0000 7f0516d9d700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05040682f0 0x7f050406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:19.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.507+0000 7f0516d9d700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05040682f0 0x7f050406a7a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f050c000f60 tx=0x7f050c007d00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:19.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.510+0000 7f04ff7fe700 1 -- 172.21.15.7:0/2608351615 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f05100549b0 con 0x7f051814e900 2023-03-03T09:10:19.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.751+0000 7f051d0fb700 1 -- 172.21.15.7:0/2608351615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0518145e10 con 0x7f051814e900 2023-03-03T09:10:19.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.751+0000 7f04ff7fe700 1 -- 172.21.15.7:0/2608351615 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0510057fd0 con 0x7f051814e900 2023-03-03T09:10:19.753 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:10:19.753 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:10:19.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 -- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05040682f0 msgr2=0x7f050406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:19.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05040682f0 0x7f050406a7a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f050c000f60 tx=0x7f050c007d00 comp rx=0 tx=0).stop 2023-03-03T09:10:19.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 -- 172.21.15.7:0/2608351615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 msgr2=0x7f0518145100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:19.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 0x7f0518145100 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0510014040 tx=0x7f0510004ad0 comp rx=0 tx=0).stop 2023-03-03T09:10:19.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 -- 172.21.15.7:0/2608351615 shutdown_connections 2023-03-03T09:10:19.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05040682f0 0x7f050406a7a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:19.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f051814e900 0x7f0518145100 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:19.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0518071e80 0x7f05181e4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:19.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 --2- 172.21.15.7:0/2608351615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0518072250 0x7f0518149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:19.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 -- 172.21.15.7:0/2608351615 >> 172.21.15.7:0/2608351615 conn(0x7f05180a94d0 msgr2=0x7f0518065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:10:19.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 -- 172.21.15.7:0/2608351615 shutdown_connections 2023-03-03T09:10:19.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:19.753+0000 7f051d0fb700 1 -- 172.21.15.7:0/2608351615 wait complete. 2023-03-03T09:10:19.756 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:10:19.766 INFO:tasks.mds_thrash.fs.[a]:waiting for 100 secs before thrashing 2023-03-03T09:10:30.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:10:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.619+0000 7fd94d7c9700 1 Processor -- start 2023-03-03T09:10:30.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.619+0000 7fd94d7c9700 1 -- start start 2023-03-03T09:10:30.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd94d7c9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9480caa30 0x7fd9480c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:30.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd94d7c9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd948154ff0 0x7fd9480cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:30.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd94d7c9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9480c7310 con 0x7fd9480c00e0 2023-03-03T09:10:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd94d7c9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9480bfd10 con 0x7fd948154ff0 2023-03-03T09:10:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd94d7c9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9480bfa30 con 0x7fd9480caa30 2023-03-03T09:10:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd9467fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd9480c00e0 0x7fd9480c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48772/0 (socket says 172.21.15.7:48772) 2023-03-03T09:10:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd9467fc700 1 -- 172.21.15.7:0/838687664 learned_addr learned my addr 172.21.15.7:0/838687664 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:10:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147390569 0 0) 0x7fd9480c7310 con 0x7fd9480c00e0 2023-03-03T09:10:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9300035b0 con 0x7fd9480c00e0 2023-03-03T09:10:30.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd9477fe700 1 --2- 172.21.15.7:0/838687664 >> v2:172.21.15.143:3300/0 conn(0x7fd948154ff0 0x7fd9480cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.620+0000 7fd946ffd700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9480caa30 0x7fd9480c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635039573 0 0) 0x7fd9300035b0 con 0x7fd9480c00e0 2023-03-03T09:10:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9480caa30 msgr2=0x7fd9480c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:10:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9480caa30 0x7fd9480c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 >> v2:172.21.15.143:3300/0 conn(0x7fd948154ff0 msgr2=0x7fd9480cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:10:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 --2- 172.21.15.7:0/838687664 >> v2:172.21.15.143:3300/0 conn(0x7fd948154ff0 0x7fd9480cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd948154510 con 0x7fd9480c00e0 2023-03-03T09:10:30.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd9380017f0 con 0x7fd9480c00e0 2023-03-03T09:10:30.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 >> v1:172.21.15.7:6789/0 conn(0x7fd9480c00e0 legacy=0x7fd9480c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:30.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd930003a70 0x7fd930003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:30.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9480caa30 0x7fd930004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:30.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd930004d90 0x7fd930005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:30.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9480bfd10 con 0x7fd930004d90 2023-03-03T09:10:30.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9480bfa30 con 0x7fd930003a70 2023-03-03T09:10:30.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93000ba60 con 0x7fd9480caa30 2023-03-03T09:10:30.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd9477fe700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd930004d90 0x7fd930005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:30.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd946ffd700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd930003a70 0x7fd930003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:30.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd9467fc700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9480caa30 0x7fd930004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:30.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd9477fe700 1 -- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9480caa30 msgr2=0x7fd930004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:10:30.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd9477fe700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9480caa30 0x7fd930004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:30.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd9477fe700 1 -- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd930003a70 msgr2=0x7fd930003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:10:30.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd9477fe700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd930003a70 0x7fd930003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:30.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd9477fe700 1 -- 172.21.15.7:0/838687664 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9300035b0 con 0x7fd930004d90 2023-03-03T09:10:30.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd9477fe700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd930004d90 0x7fd930005200 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd94000e170 tx=0x7fd930005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa8a12e9f399f1c3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:30.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.621+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000ec90 con 0x7fd930004d90 2023-03-03T09:10:30.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.622+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94000f040 con 0x7fd930004d90 2023-03-03T09:10:30.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.622+0000 7fd945ffb700 1 -- 172.21.15.7:0/838687664 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940014480 con 0x7fd930004d90 2023-03-03T09:10:30.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.622+0000 7fd94d7c9700 1 -- 172.21.15.7:0/838687664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd930004d90 msgr2=0x7fd930005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:30.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.622+0000 7fd94d7c9700 1 --2- 172.21.15.7:0/838687664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd930004d90 0x7fd930005200 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd94000e170 tx=0x7fd930005880 comp rx=0 tx=0).stop 2023-03-03T09:10:30.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.622+0000 7fd9477fe700 1 -- 172.21.15.7:0/838687664 reap_dead start 2023-03-03T09:10:30.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.622+0000 7fd94d7c9700 1 -- 172.21.15.7:0/838687664 shutdown_connections 2023-03-03T09:10:30.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.622+0000 7fd94d7c9700 1 -- 172.21.15.7:0/838687664 >> 172.21.15.7:0/838687664 conn(0x7fd948066e40 msgr2=0x7fd948071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:10:30.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.622+0000 7fd94d7c9700 1 -- 172.21.15.7:0/838687664 shutdown_connections 2023-03-03T09:10:30.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.622+0000 7fd94d7c9700 1 -- 172.21.15.7:0/838687664 wait complete. 2023-03-03T09:10:30.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.623+0000 7fd94d7c9700 1 Processor -- start 2023-03-03T09:10:30.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.624+0000 7fd94d7c9700 1 -- start start 2023-03-03T09:10:30.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.624+0000 7fd94d7c9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd948158350 0x7fd9481e0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:30.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.624+0000 7fd94d7c9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9481553c0 0x7fd9480c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:30.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.624+0000 7fd94d7c9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9481561c0 0x7fd9480c3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:30.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.624+0000 7fd94d7c9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9480c3d40 con 0x7fd9481553c0 2023-03-03T09:10:30.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.624+0000 7fd94d7c9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9481edf70 con 0x7fd948158350 2023-03-03T09:10:30.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.624+0000 7fd94d7c9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9481ed1b0 con 0x7fd9481561c0 2023-03-03T09:10:30.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.624+0000 7fd9467fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9481553c0 0x7fd9480c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:30.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.624+0000 7fd9467fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9481553c0 0x7fd9480c32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54804/0 (socket says 172.21.15.7:54804) 2023-03-03T09:10:30.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.624+0000 7fd9467fc700 1 -- 172.21.15.7:0/2301189689 learned_addr learned my addr 172.21.15.7:0/2301189689 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:10:30.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd946ffd700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd948158350 0x7fd9481e0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:30.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd9477fe700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9481561c0 0x7fd9480c3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:30.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd9467fc700 1 -- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9481561c0 msgr2=0x7fd9480c3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:30.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd9467fc700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9481561c0 0x7fd9480c3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:30.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd9467fc700 1 -- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd948158350 msgr2=0x7fd9481e0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:30.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd9467fc700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd948158350 0x7fd9481e0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:30.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd9467fc700 1 -- 172.21.15.7:0/2301189689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd948154ad0 con 0x7fd9481553c0 2023-03-03T09:10:30.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd9467fc700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9481553c0 0x7fd9480c32c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd938001c20 tx=0x7fd938001890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:30.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd92f7fe700 1 -- 172.21.15.7:0/2301189689 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9380073a0 con 0x7fd9481553c0 2023-03-03T09:10:30.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd92f7fe700 1 -- 172.21.15.7:0/2301189689 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd938007520 con 0x7fd9481553c0 2023-03-03T09:10:30.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd92f7fe700 1 -- 172.21.15.7:0/2301189689 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd938005a70 con 0x7fd9481553c0 2023-03-03T09:10:30.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd94d7c9700 1 -- 172.21.15.7:0/2301189689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9481547f0 con 0x7fd9481553c0 2023-03-03T09:10:30.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.625+0000 7fd94d7c9700 1 -- 172.21.15.7:0/2301189689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9480bf650 con 0x7fd9481553c0 2023-03-03T09:10:30.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.626+0000 7fd94d7c9700 1 -- 172.21.15.7:0/2301189689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd948074bb0 con 0x7fd9481553c0 2023-03-03T09:10:30.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.627+0000 7fd92f7fe700 1 -- 172.21.15.7:0/2301189689 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd938005bd0 con 0x7fd9481553c0 2023-03-03T09:10:30.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.627+0000 7fd92f7fe700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd934068320 0x7fd93406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:10:30.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.627+0000 7fd92f7fe700 1 -- 172.21.15.7:0/2301189689 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd938086350 con 0x7fd9481553c0 2023-03-03T09:10:30.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.627+0000 7fd946ffd700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd934068320 0x7fd93406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:10:30.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.628+0000 7fd946ffd700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd934068320 0x7fd93406a7d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009b20 tx=0x7fd93c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:10:30.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.631+0000 7fd92f7fe700 1 -- 172.21.15.7:0/2301189689 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd9380526f0 con 0x7fd9481553c0 2023-03-03T09:10:30.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.871+0000 7fd94d7c9700 1 -- 172.21.15.7:0/2301189689 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9480c5030 con 0x7fd9481553c0 2023-03-03T09:10:30.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.872+0000 7fd92f7fe700 1 -- 172.21.15.7:0/2301189689 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd938005e60 con 0x7fd9481553c0 2023-03-03T09:10:30.875 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:10:30.875 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:10:30.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 -- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd934068320 msgr2=0x7fd93406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:30.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd934068320 0x7fd93406a7d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009b20 tx=0x7fd93c008040 comp rx=0 tx=0).stop 2023-03-03T09:10:30.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 -- 172.21.15.7:0/2301189689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9481553c0 msgr2=0x7fd9480c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:10:30.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9481553c0 0x7fd9480c32c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd938001c20 tx=0x7fd938001890 comp rx=0 tx=0).stop 2023-03-03T09:10:30.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 -- 172.21.15.7:0/2301189689 shutdown_connections 2023-03-03T09:10:30.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9481561c0 0x7fd9480c3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:30.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd948158350 0x7fd9481e0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:30.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd934068320 0x7fd93406a7d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:30.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 --2- 172.21.15.7:0/2301189689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9481553c0 0x7fd9480c32c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:10:30.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 -- 172.21.15.7:0/2301189689 >> 172.21.15.7:0/2301189689 conn(0x7fd948066e40 msgr2=0x7fd9480c8490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:10:30.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 -- 172.21.15.7:0/2301189689 shutdown_connections 2023-03-03T09:10:30.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:10:30.874+0000 7fd94d7c9700 1 -- 172.21.15.7:0/2301189689 wait complete. 2023-03-03T09:10:30.878 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:10:30.888 INFO:tasks.mds_thrash.fs.[b]:waiting for 106 secs before thrashing 2023-03-03T09:10:41.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:10:41.710 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:11.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:11.772 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:41.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:41.836 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:11:59.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:11:59.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.905+0000 7f3cda7ae700 1 Processor -- start 2023-03-03T09:11:59.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.905+0000 7f3cda7ae700 1 -- start start 2023-03-03T09:11:59.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.906+0000 7f3cda7ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3cd40ac220 0x7f3cd40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:59.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.906+0000 7f3cda7ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 0x7f3cd4152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:59.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.906+0000 7f3cda7ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd40ac900 con 0x7f3cd40ace30 2023-03-03T09:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.906+0000 7f3cda7ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd4071e30 con 0x7f3cd40ac220 2023-03-03T09:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.906+0000 7f3cda7ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd40acb50 con 0x7f3cd4073c30 2023-03-03T09:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.906+0000 7f3cd3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3cd40ac220 0x7f3cd40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.906+0000 7f3cd37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 0x7f3cd4152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.906+0000 7f3cd3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3cd40ac220 0x7f3cd40736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:50660/0 (socket says 172.21.15.7:50660) 2023-03-03T09:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.906+0000 7f3cd3fff700 1 -- 172.21.15.7:0/1124247494 learned_addr learned my addr 172.21.15.7:0/1124247494 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd2ffd700 1 -- 172.21.15.7:0/1124247494 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868098395 0 0) 0x7f3cd40ac900 con 0x7f3cd40ace30 2023-03-03T09:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd2ffd700 1 -- 172.21.15.7:0/1124247494 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cc00035b0 con 0x7f3cd40ace30 2023-03-03T09:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd3fff700 1 -- 172.21.15.7:0/1124247494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 msgr2=0x7f3cd4152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd3fff700 1 --2- 172.21.15.7:0/1124247494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 0x7f3cd4152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd3fff700 1 -- 172.21.15.7:0/1124247494 >> v1:172.21.15.143:6789/0 conn(0x7f3cd40ace30 legacy=0x7f3cd406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd3fff700 1 -- 172.21.15.7:0/1124247494 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd41559c0 con 0x7f3cd40ac220 2023-03-03T09:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd37fe700 1 --2- 172.21.15.7:0/1124247494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 0x7f3cd4152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd3fff700 1 --2- 172.21.15.7:0/1124247494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cd40ac220 0x7f3cd40736f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40098f0 tx=0x7f3cc4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f60dfe14491cf95 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd2ffd700 1 -- 172.21.15.7:0/1124247494 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4011830 con 0x7f3cd40ac220 2023-03-03T09:11:59.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd2ffd700 1 -- 172.21.15.7:0/1124247494 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc4011990 con 0x7f3cd40ac220 2023-03-03T09:11:59.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.907+0000 7f3cd2ffd700 1 -- 172.21.15.7:0/1124247494 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4011d20 con 0x7f3cd40ac220 2023-03-03T09:11:59.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.908+0000 7f3cda7ae700 1 -- 172.21.15.7:0/1124247494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cd40ac220 msgr2=0x7f3cd40736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:11:59.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.908+0000 7f3cda7ae700 1 --2- 172.21.15.7:0/1124247494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cd40ac220 0x7f3cd40736f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40098f0 tx=0x7f3cc4009c00 comp rx=0 tx=0).stop 2023-03-03T09:11:59.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.908+0000 7f3cda7ae700 1 -- 172.21.15.7:0/1124247494 shutdown_connections 2023-03-03T09:11:59.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.908+0000 7f3cda7ae700 1 --2- 172.21.15.7:0/1124247494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cd40ac220 0x7f3cd40736f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:59.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.908+0000 7f3cda7ae700 1 --2- 172.21.15.7:0/1124247494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 0x7f3cd4152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:59.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.908+0000 7f3cda7ae700 1 -- 172.21.15.7:0/1124247494 >> 172.21.15.7:0/1124247494 conn(0x7f3cd40a94d0 msgr2=0x7f3cd40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:11:59.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.908+0000 7f3cda7ae700 1 -- 172.21.15.7:0/1124247494 shutdown_connections 2023-03-03T09:11:59.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.908+0000 7f3cda7ae700 1 -- 172.21.15.7:0/1124247494 wait complete. 2023-03-03T09:11:59.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.909+0000 7f3cda7ae700 1 Processor -- start 2023-03-03T09:11:59.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.909+0000 7f3cda7ae700 1 -- start start 2023-03-03T09:11:59.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cda7ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 0x7f3cd4154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:59.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cda7ae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cd40ac220 0x7f3cd40be7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:59.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cda7ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cd40ace30 0x7f3cd414dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:59.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cda7ae700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd41ebd80 con 0x7f3cd40ac220 2023-03-03T09:11:59.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cda7ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd414d290 con 0x7f3cd40ace30 2023-03-03T09:11:59.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cda7ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd41eb800 con 0x7f3cd4073c30 2023-03-03T09:11:59.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cd37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cd40ac220 0x7f3cd40be7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:59.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cd37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cd40ac220 0x7f3cd40be7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54894/0 (socket says 172.21.15.7:54894) 2023-03-03T09:11:59.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cd37fe700 1 -- 172.21.15.7:0/328139956 learned_addr learned my addr 172.21.15.7:0/328139956 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:11:59.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cd3fff700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 0x7f3cd4154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:59.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cd8d4b700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cd40ace30 0x7f3cd414dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:59.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cd37fe700 1 -- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 msgr2=0x7f3cd4154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:11:59.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cd37fe700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 0x7f3cd4154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:59.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cd37fe700 1 -- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cd40ace30 msgr2=0x7f3cd414dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:11:59.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cd37fe700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cd40ace30 0x7f3cd414dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:11:59.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.910+0000 7f3cd37fe700 1 -- 172.21.15.7:0/328139956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc0094a0 con 0x7f3cd40ac220 2023-03-03T09:11:59.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.911+0000 7f3cd37fe700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cd40ac220 0x7f3cd40be7b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00ebe0 tx=0x7f3cbc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:11:59.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.911+0000 7f3cd0ff9700 1 -- 172.21.15.7:0/328139956 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc009760 con 0x7f3cd40ac220 2023-03-03T09:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.911+0000 7f3cd0ff9700 1 -- 172.21.15.7:0/328139956 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cbc00c850 con 0x7f3cd40ac220 2023-03-03T09:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.911+0000 7f3cd0ff9700 1 -- 172.21.15.7:0/328139956 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc00cbe0 con 0x7f3cd40ac220 2023-03-03T09:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.911+0000 7f3cda7ae700 1 -- 172.21.15.7:0/328139956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc4009550 con 0x7f3cd40ac220 2023-03-03T09:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.911+0000 7f3cda7ae700 1 -- 172.21.15.7:0/328139956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd4155f80 con 0x7f3cd40ac220 2023-03-03T09:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.912+0000 7f3cda7ae700 1 -- 172.21.15.7:0/328139956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd4155ca0 con 0x7f3cd40ac220 2023-03-03T09:11:59.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.912+0000 7f3cd0ff9700 1 -- 172.21.15.7:0/328139956 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3cbc023070 con 0x7f3cd40ac220 2023-03-03T09:11:59.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.912+0000 7f3cd0ff9700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cb40682f0 0x7f3cb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:11:59.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.913+0000 7f3cd0ff9700 1 -- 172.21.15.7:0/328139956 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3cbc086380 con 0x7f3cd40ac220 2023-03-03T09:11:59.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.913+0000 7f3cd3fff700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cb40682f0 0x7f3cb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:11:59.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.913+0000 7f3cd3fff700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cb40682f0 0x7f3cb406a7a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4014040 tx=0x7f3cc4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:11:59.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:11:59.917+0000 7f3cd0ff9700 1 -- 172.21.15.7:0/328139956 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3cbc0526a0 con 0x7f3cd40ac220 2023-03-03T09:12:00.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.158+0000 7f3cda7ae700 1 -- 172.21.15.7:0/328139956 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cd4156260 con 0x7f3cd40ac220 2023-03-03T09:12:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.159+0000 7f3cd0ff9700 1 -- 172.21.15.7:0/328139956 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3cbc055cc0 con 0x7f3cd40ac220 2023-03-03T09:12:00.160 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:12:00.161 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:12:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 -- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cb40682f0 msgr2=0x7f3cb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cb40682f0 0x7f3cb406a7a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4014040 tx=0x7f3cc4008030 comp rx=0 tx=0).stop 2023-03-03T09:12:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 -- 172.21.15.7:0/328139956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cd40ac220 msgr2=0x7f3cd40be7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cd40ac220 0x7f3cd40be7b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00ebe0 tx=0x7f3cbc00eef0 comp rx=0 tx=0).stop 2023-03-03T09:12:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 -- 172.21.15.7:0/328139956 shutdown_connections 2023-03-03T09:12:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cd40ace30 0x7f3cd414dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cd4073c30 0x7f3cd4154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:00.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cb40682f0 0x7f3cb406a7a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:00.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 --2- 172.21.15.7:0/328139956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cd40ac220 0x7f3cd40be7b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:00.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 -- 172.21.15.7:0/328139956 >> 172.21.15.7:0/328139956 conn(0x7f3cd40a94d0 msgr2=0x7f3cd4070720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:00.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 -- 172.21.15.7:0/328139956 shutdown_connections 2023-03-03T09:12:00.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:00.161+0000 7f3cda7ae700 1 -- 172.21.15.7:0/328139956 wait complete. 2023-03-03T09:12:00.164 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:12:00.174 INFO:tasks.mds_thrash.fs.[a]:waiting for 54 secs before thrashing 2023-03-03T09:12:11.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:11.913 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:16.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:12:17.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.027+0000 7fa5da7b6700 1 Processor -- start 2023-03-03T09:12:17.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.027+0000 7fa5da7b6700 1 -- start start 2023-03-03T09:12:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5da7b6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa5d4151d40 0x7fa5d40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5da7b6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4073c30 0x7fa5d41581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5da7b6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d406d430 con 0x7fa5d40c2930 2023-03-03T09:12:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5da7b6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d406c2a0 con 0x7fa5d4151d40 2023-03-03T09:12:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5da7b6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d41586e0 con 0x7fa5d4073c30 2023-03-03T09:12:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d8d53700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4073c30 0x7fa5d41581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa5d4151d40 0x7fa5d40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d8d53700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4073c30 0x7fa5d41581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:34746/0 (socket says 172.21.15.7:34746) 2023-03-03T09:12:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa5d4151d40 0x7fa5d40736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:50676/0 (socket says 172.21.15.7:50676) 2023-03-03T09:12:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d8d53700 1 -- 172.21.15.7:0/829713159 learned_addr learned my addr 172.21.15.7:0/829713159 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:12:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d37fe700 1 --1- 172.21.15.7:0/829713159 >> v1:172.21.15.143:6789/0 conn(0x7fa5d40c2930 0x7fa5d4157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:52402/0 (socket says 172.21.15.7:52402) 2023-03-03T09:12:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d8d53700 1 -- 172.21.15.7:0/829713159 >> v2:172.21.15.143:3300/0 conn(0x7fa5d4151d40 msgr2=0x7fa5d40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:12:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d8d53700 1 --2- 172.21.15.7:0/829713159 >> v2:172.21.15.143:3300/0 conn(0x7fa5d4151d40 0x7fa5d40736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d8d53700 1 -- 172.21.15.7:0/829713159 >> v1:172.21.15.143:6789/0 conn(0x7fa5d40c2930 legacy=0x7fa5d4157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d8d53700 1 -- 172.21.15.7:0/829713159 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d406cb00 con 0x7fa5d4073c30 2023-03-03T09:12:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d3fff700 1 --2- 172.21.15.7:0/829713159 >> v2:172.21.15.143:3300/0 conn(0x7fa5d4151d40 0x7fa5d40736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:12:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.028+0000 7fa5d2ffd700 1 -- 172.21.15.7:0/829713159 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940097090 0 0) 0x7fa5d406d430 con 0x7fa5d40c2930 2023-03-03T09:12:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5d8d53700 1 --2- 172.21.15.7:0/829713159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4073c30 0x7fa5d41581a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc009e50 tx=0x7fa5cc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fa45e5301c8c338 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:17.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5d2ffd700 1 -- 172.21.15.7:0/829713159 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc00e070 con 0x7fa5d4073c30 2023-03-03T09:12:17.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5d2ffd700 1 -- 172.21.15.7:0/829713159 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5cc00b910 con 0x7fa5d4073c30 2023-03-03T09:12:17.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5d2ffd700 1 -- 172.21.15.7:0/829713159 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc00bcc0 con 0x7fa5d4073c30 2023-03-03T09:12:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5da7b6700 1 -- 172.21.15.7:0/829713159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4073c30 msgr2=0x7fa5d41581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5da7b6700 1 --2- 172.21.15.7:0/829713159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4073c30 0x7fa5d41581a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc009e50 tx=0x7fa5cc00b330 comp rx=0 tx=0).stop 2023-03-03T09:12:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5da7b6700 1 -- 172.21.15.7:0/829713159 shutdown_connections 2023-03-03T09:12:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5da7b6700 1 --2- 172.21.15.7:0/829713159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4073c30 0x7fa5d41581a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5da7b6700 1 --2- 172.21.15.7:0/829713159 >> v2:172.21.15.143:3300/0 conn(0x7fa5d4151d40 0x7fa5d40736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5da7b6700 1 -- 172.21.15.7:0/829713159 >> 172.21.15.7:0/829713159 conn(0x7fa5d40a94d0 msgr2=0x7fa5d40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5da7b6700 1 -- 172.21.15.7:0/829713159 shutdown_connections 2023-03-03T09:12:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.029+0000 7fa5da7b6700 1 -- 172.21.15.7:0/829713159 wait complete. 2023-03-03T09:12:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.030+0000 7fa5da7b6700 1 Processor -- start 2023-03-03T09:12:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.031+0000 7fa5da7b6700 1 -- start start 2023-03-03T09:12:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.031+0000 7fa5da7b6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5d4073c30 0x7fa5d40aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.031+0000 7fa5da7b6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5d40c2930 0x7fa5d40aca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.031+0000 7fa5da7b6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4151d40 0x7fa5d41ed7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.031+0000 7fa5da7b6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d40c0a40 con 0x7fa5d4073c30 2023-03-03T09:12:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.031+0000 7fa5da7b6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d40c0d20 con 0x7fa5d40c2930 2023-03-03T09:12:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.031+0000 7fa5da7b6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d40c1000 con 0x7fa5d4151d40 2023-03-03T09:12:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5d4073c30 0x7fa5d40aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5d4073c30 0x7fa5d40aa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54908/0 (socket says 172.21.15.7:54908) 2023-03-03T09:12:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d3fff700 1 -- 172.21.15.7:0/2075869796 learned_addr learned my addr 172.21.15.7:0/2075869796 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:12:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d8d53700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4151d40 0x7fa5d41ed7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d37fe700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5d40c2930 0x7fa5d40aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d3fff700 1 -- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4151d40 msgr2=0x7fa5d41ed7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d3fff700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4151d40 0x7fa5d41ed7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d3fff700 1 -- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5d40c2930 msgr2=0x7fa5d40aca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d3fff700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5d40c2930 0x7fa5d40aca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d3fff700 1 -- 172.21.15.7:0/2075869796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c40094a0 con 0x7fa5d4073c30 2023-03-03T09:12:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d8d53700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4151d40 0x7fa5d41ed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:12:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d37fe700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5d40c2930 0x7fa5d40aca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:12:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d3fff700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5d4073c30 0x7fa5d40aa1d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400ebe0 tx=0x7fa5c4009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d0ff9700 1 -- 172.21.15.7:0/2075869796 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c400c910 con 0x7fa5d4073c30 2023-03-03T09:12:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d0ff9700 1 -- 172.21.15.7:0/2075869796 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c400ca70 con 0x7fa5d4073c30 2023-03-03T09:12:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5d0ff9700 1 -- 172.21.15.7:0/2075869796 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c400cde0 con 0x7fa5d4073c30 2023-03-03T09:12:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5da7b6700 1 -- 172.21.15.7:0/2075869796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc009b30 con 0x7fa5d4073c30 2023-03-03T09:12:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.032+0000 7fa5da7b6700 1 -- 172.21.15.7:0/2075869796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d406d0c0 con 0x7fa5d4073c30 2023-03-03T09:12:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.033+0000 7fa5da7b6700 1 -- 172.21.15.7:0/2075869796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d406cde0 con 0x7fa5d4073c30 2023-03-03T09:12:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.034+0000 7fa5d0ff9700 1 -- 172.21.15.7:0/2075869796 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa5c4019030 con 0x7fa5d4073c30 2023-03-03T09:12:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.034+0000 7fa5d0ff9700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5c00682f0 0x7fa5c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.034+0000 7fa5d0ff9700 1 -- 172.21.15.7:0/2075869796 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa5c4086210 con 0x7fa5d4073c30 2023-03-03T09:12:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.035+0000 7fa5d37fe700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5c00682f0 0x7fa5c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.035+0000 7fa5d37fe700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5c00682f0 0x7fa5c006a7a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40c4900 tx=0x7fa5c8007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.039+0000 7fa5d0ff9700 1 -- 172.21.15.7:0/2075869796 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa5c40525b0 con 0x7fa5d4073c30 2023-03-03T09:12:17.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.277+0000 7fa5da7b6700 1 -- 172.21.15.7:0/2075869796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5d41ed620 con 0x7fa5d4073c30 2023-03-03T09:12:17.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.277+0000 7fa5d0ff9700 1 -- 172.21.15.7:0/2075869796 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa5c4055bd0 con 0x7fa5d4073c30 2023-03-03T09:12:17.279 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:12:17.279 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:12:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 -- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5c00682f0 msgr2=0x7fa5c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5c00682f0 0x7fa5c006a7a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40c4900 tx=0x7fa5c8007d30 comp rx=0 tx=0).stop 2023-03-03T09:12:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 -- 172.21.15.7:0/2075869796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5d4073c30 msgr2=0x7fa5d40aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5d4073c30 0x7fa5d40aa1d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400ebe0 tx=0x7fa5c4009ec0 comp rx=0 tx=0).stop 2023-03-03T09:12:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 -- 172.21.15.7:0/2075869796 shutdown_connections 2023-03-03T09:12:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5d4151d40 0x7fa5d41ed7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5c00682f0 0x7fa5c006a7a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5d4073c30 0x7fa5d40aa1d0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 --2- 172.21.15.7:0/2075869796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5d40c2930 0x7fa5d40aca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 -- 172.21.15.7:0/2075869796 >> 172.21.15.7:0/2075869796 conn(0x7fa5d40a94d0 msgr2=0x7fa5d4071330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:17.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 -- 172.21.15.7:0/2075869796 shutdown_connections 2023-03-03T09:12:17.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:17.280+0000 7fa5da7b6700 1 -- 172.21.15.7:0/2075869796 wait complete. 2023-03-03T09:12:17.283 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:12:17.293 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T09:12:35.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:12:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.431+0000 7fa2dda0c700 1 Processor -- start 2023-03-03T09:12:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.431+0000 7fa2dda0c700 1 -- start start 2023-03-03T09:12:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.431+0000 7fa2dda0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80caa30 0x7fa2d80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2dda0c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d80c7310 con 0x7fa2d80c00e0 2023-03-03T09:12:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2dda0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d80bfd10 con 0x7fa2d8154ff0 2023-03-03T09:12:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2dda0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d80bfa30 con 0x7fa2d80caa30 2023-03-03T09:12:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d67fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa2d80c00e0 0x7fa2d80c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48912/0 (socket says 172.21.15.7:48912) 2023-03-03T09:12:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d67fc700 1 -- 172.21.15.7:0/3404012342 learned_addr learned my addr 172.21.15.7:0/3404012342 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:12:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d6ffd700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80caa30 0x7fa2d80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678599862 0 0) 0x7fa2d80c7310 con 0x7fa2d80c00e0 2023-03-03T09:12:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c00035b0 con 0x7fa2d80c00e0 2023-03-03T09:12:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798881789 0 0) 0x7fa2d80bfd10 con 0x7fa2d8154ff0 2023-03-03T09:12:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d80c7310 con 0x7fa2d8154ff0 2023-03-03T09:12:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 740699404 0 0) 0x7fa2c00035b0 con 0x7fa2d80c00e0 2023-03-03T09:12:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80caa30 msgr2=0x7fa2d80c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:12:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80caa30 0x7fa2d80c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 >> v1:172.21.15.143:6789/0 conn(0x7fa2d8154ff0 legacy=0x7fa2d815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d8154510 con 0x7fa2d80c00e0 2023-03-03T09:12:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa2cc0017f0 con 0x7fa2d80c00e0 2023-03-03T09:12:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 >> v1:172.21.15.7:6789/0 conn(0x7fa2d80c00e0 legacy=0x7fa2d80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2c0003c30 0x7fa2c0004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80caa30 0x7fa2c0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2c0004d90 0x7fa2c0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d80bfa30 con 0x7fa2c0003c30 2023-03-03T09:12:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c000bba0 con 0x7fa2c0004d90 2023-03-03T09:12:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.432+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c000bd30 con 0x7fa2d80caa30 2023-03-03T09:12:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d6ffd700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2c0003c30 0x7fa2c0004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d67fc700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80caa30 0x7fa2c0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d77fe700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2c0004d90 0x7fa2c0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d6ffd700 1 -- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80caa30 msgr2=0x7fa2c0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:12:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d6ffd700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80caa30 0x7fa2c0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d6ffd700 1 -- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2c0004d90 msgr2=0x7fa2c0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:12:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d6ffd700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2c0004d90 0x7fa2c0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d6ffd700 1 -- 172.21.15.7:0/3404012342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c00035b0 con 0x7fa2c0003c30 2023-03-03T09:12:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d67fc700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80caa30 0x7fa2c0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:12:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d6ffd700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2c0003c30 0x7fa2c0004020 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009b60 tx=0x7fa2c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b63a59ce8c6c3c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c800e6c0 con 0x7fa2c0003c30 2023-03-03T09:12:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c800e820 con 0x7fa2c0003c30 2023-03-03T09:12:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.433+0000 7fa2d5ffb700 1 -- 172.21.15.7:0/3404012342 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c800eb90 con 0x7fa2c0003c30 2023-03-03T09:12:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.434+0000 7fa2dda0c700 1 -- 172.21.15.7:0/3404012342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2c0003c30 msgr2=0x7fa2c0004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.434+0000 7fa2dda0c700 1 --2- 172.21.15.7:0/3404012342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2c0003c30 0x7fa2c0004020 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009b60 tx=0x7fa2c0005880 comp rx=0 tx=0).stop 2023-03-03T09:12:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.434+0000 7fa2d77fe700 1 -- 172.21.15.7:0/3404012342 reap_dead start 2023-03-03T09:12:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.434+0000 7fa2dda0c700 1 -- 172.21.15.7:0/3404012342 shutdown_connections 2023-03-03T09:12:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.434+0000 7fa2dda0c700 1 -- 172.21.15.7:0/3404012342 >> 172.21.15.7:0/3404012342 conn(0x7fa2d8066e40 msgr2=0x7fa2d8071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.434+0000 7fa2dda0c700 1 -- 172.21.15.7:0/3404012342 shutdown_connections 2023-03-03T09:12:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.434+0000 7fa2dda0c700 1 -- 172.21.15.7:0/3404012342 wait complete. 2023-03-03T09:12:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.435+0000 7fa2dda0c700 1 Processor -- start 2023-03-03T09:12:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.435+0000 7fa2dda0c700 1 -- start start 2023-03-03T09:12:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2dda0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2c0003c30 0x7fa2d814c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2dda0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80c9690 0x7fa2d80c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2dda0c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2d814a320 0x7fa2d80c4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2dda0c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d8156820 con 0x7fa2d814a320 2023-03-03T09:12:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2dda0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d80c2290 con 0x7fa2c0003c30 2023-03-03T09:12:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2dda0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d8156d50 con 0x7fa2d80c9690 2023-03-03T09:12:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2d814a320 0x7fa2d80c4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2d814a320 0x7fa2d80c4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54948/0 (socket says 172.21.15.7:54948) 2023-03-03T09:12:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d77fe700 1 -- 172.21.15.7:0/4057566799 learned_addr learned my addr 172.21.15.7:0/4057566799 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:12:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d77fe700 1 -- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80c9690 msgr2=0x7fa2d80c3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d6ffd700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2c0003c30 0x7fa2d814c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d67fc700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80c9690 0x7fa2d80c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d77fe700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80c9690 0x7fa2d80c3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d77fe700 1 -- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2c0003c30 msgr2=0x7fa2d814c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d77fe700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2c0003c30 0x7fa2d814c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d77fe700 1 -- 172.21.15.7:0/4057566799 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d8154ad0 con 0x7fa2d814a320 2023-03-03T09:12:35.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2d77fe700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2d814a320 0x7fa2d80c4220 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa2d806c210 tx=0x7fa2d000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:35.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.436+0000 7fa2bffff700 1 -- 172.21.15.7:0/4057566799 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0003c10 con 0x7fa2d814a320 2023-03-03T09:12:35.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.437+0000 7fa2bffff700 1 -- 172.21.15.7:0/4057566799 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d0003d90 con 0x7fa2d814a320 2023-03-03T09:12:35.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.437+0000 7fa2dda0c700 1 -- 172.21.15.7:0/4057566799 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d81547f0 con 0x7fa2d814a320 2023-03-03T09:12:35.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.437+0000 7fa2bffff700 1 -- 172.21.15.7:0/4057566799 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0005310 con 0x7fa2d814a320 2023-03-03T09:12:35.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.437+0000 7fa2dda0c700 1 -- 172.21.15.7:0/4057566799 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d80bf650 con 0x7fa2d814a320 2023-03-03T09:12:35.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.438+0000 7fa2dda0c700 1 -- 172.21.15.7:0/4057566799 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d8074bb0 con 0x7fa2d814a320 2023-03-03T09:12:35.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.438+0000 7fa2bffff700 1 -- 172.21.15.7:0/4057566799 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa2d0005470 con 0x7fa2d814a320 2023-03-03T09:12:35.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.439+0000 7fa2bffff700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2c40682f0 0x7fa2c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:35.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.439+0000 7fa2bffff700 1 -- 172.21.15.7:0/4057566799 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa2d0086330 con 0x7fa2d814a320 2023-03-03T09:12:35.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.439+0000 7fa2d6ffd700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2c40682f0 0x7fa2c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:35.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.440+0000 7fa2d6ffd700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2c40682f0 0x7fa2c406a7a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8000c00 tx=0x7fa2c8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:35.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.443+0000 7fa2bffff700 1 -- 172.21.15.7:0/4057566799 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa2d00526d0 con 0x7fa2d814a320 2023-03-03T09:12:35.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.679+0000 7fa2dda0c700 1 -- 172.21.15.7:0/4057566799 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2d80c23d0 con 0x7fa2d814a320 2023-03-03T09:12:35.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.679+0000 7fa2bffff700 1 -- 172.21.15.7:0/4057566799 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa2d0055cf0 con 0x7fa2d814a320 2023-03-03T09:12:35.681 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:12:35.681 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:12:35.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 -- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2c40682f0 msgr2=0x7fa2c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:35.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2c40682f0 0x7fa2c406a7a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8000c00 tx=0x7fa2c8008030 comp rx=0 tx=0).stop 2023-03-03T09:12:35.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 -- 172.21.15.7:0/4057566799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2d814a320 msgr2=0x7fa2d80c4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:35.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2d814a320 0x7fa2d80c4220 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa2d806c210 tx=0x7fa2d000ef20 comp rx=0 tx=0).stop 2023-03-03T09:12:35.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 -- 172.21.15.7:0/4057566799 shutdown_connections 2023-03-03T09:12:35.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2c40682f0 0x7fa2c406a7a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:35.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2d814a320 0x7fa2d80c4220 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:35.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2c0003c30 0x7fa2d814c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:35.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 --2- 172.21.15.7:0/4057566799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2d80c9690 0x7fa2d80c3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:35.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 -- 172.21.15.7:0/4057566799 >> 172.21.15.7:0/4057566799 conn(0x7fa2d8066e40 msgr2=0x7fa2d80c7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:35.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 -- 172.21.15.7:0/4057566799 shutdown_connections 2023-03-03T09:12:35.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:35.681+0000 7fa2dda0c700 1 -- 172.21.15.7:0/4057566799 wait complete. 2023-03-03T09:12:35.684 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:12:35.694 INFO:tasks.mds_thrash.fs.[b]:waiting for 10 secs before thrashing 2023-03-03T09:12:41.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:41.992 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:12:45.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:12:45.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.831+0000 7fdca2723700 1 Processor -- start 2023-03-03T09:12:45.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.831+0000 7fdca2723700 1 -- start start 2023-03-03T09:12:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdca2723700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdc9c0c2e30 0x7fdc9c158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdca2723700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c157ff0 con 0x7fdc9c06c6c0 2023-03-03T09:12:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdca2723700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9c06d640 con 0x7fdc9c151cd0 2023-03-03T09:12:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdca2723700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc9c06cda0 con 0x7fdc9c0c2e30 2023-03-03T09:12:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdca0cc0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdc9c06c6c0 0x7fdc9c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48964/0 (socket says 172.21.15.7:48964) 2023-03-03T09:12:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdc9b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdc9c0c2e30 0x7fdc9c158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdca0cc0700 1 -- 172.21.15.7:0/2096557725 learned_addr learned my addr 172.21.15.7:0/2096557725 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:12:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdc9b7fe700 1 -- 172.21.15.7:0/2096557725 >> v1:172.21.15.143:6789/0 conn(0x7fdc9c151cd0 legacy=0x7fdc9c154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:12:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdc9b7fe700 1 -- 172.21.15.7:0/2096557725 >> v1:172.21.15.7:6789/0 conn(0x7fdc9c06c6c0 legacy=0x7fdc9c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdc9affd700 1 -- 172.21.15.7:0/2096557725 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297226059 0 0) 0x7fdc9c157ff0 con 0x7fdc9c06c6c0 2023-03-03T09:12:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.832+0000 7fdc9b7fe700 1 -- 172.21.15.7:0/2096557725 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c06d080 con 0x7fdc9c0c2e30 2023-03-03T09:12:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdc9b7fe700 1 --2- 172.21.15.7:0/2096557725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c0c2e30 0x7fdc9c158480 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009840 tx=0x7fdc8c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820232b8b49963bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdc9affd700 1 -- 172.21.15.7:0/2096557725 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c004230 con 0x7fdc9c0c2e30 2023-03-03T09:12:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdc9affd700 1 -- 172.21.15.7:0/2096557725 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8c004390 con 0x7fdc9c0c2e30 2023-03-03T09:12:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdc9affd700 1 -- 172.21.15.7:0/2096557725 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c0036a0 con 0x7fdc9c0c2e30 2023-03-03T09:12:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdca2723700 1 -- 172.21.15.7:0/2096557725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c0c2e30 msgr2=0x7fdc9c158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdca2723700 1 --2- 172.21.15.7:0/2096557725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c0c2e30 0x7fdc9c158480 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009840 tx=0x7fdc8c009b50 comp rx=0 tx=0).stop 2023-03-03T09:12:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdca2723700 1 -- 172.21.15.7:0/2096557725 shutdown_connections 2023-03-03T09:12:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdca2723700 1 --2- 172.21.15.7:0/2096557725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c0c2e30 0x7fdc9c158480 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdca2723700 1 -- 172.21.15.7:0/2096557725 >> 172.21.15.7:0/2096557725 conn(0x7fdc9c0a94d0 msgr2=0x7fdc9c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdca2723700 1 -- 172.21.15.7:0/2096557725 shutdown_connections 2023-03-03T09:12:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.833+0000 7fdca2723700 1 -- 172.21.15.7:0/2096557725 wait complete. 2023-03-03T09:12:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.835+0000 7fdca2723700 1 Processor -- start 2023-03-03T09:12:45.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.835+0000 7fdca2723700 1 -- start start 2023-03-03T09:12:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdca2723700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc9c06c6c0 0x7fdc9c1542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdca2723700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc9c0c2e30 0x7fdc9c1eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdca2723700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c151cd0 0x7fdc9c1edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdca2723700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c1ea5d0 con 0x7fdc9c151cd0 2023-03-03T09:12:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdca2723700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c1ea870 con 0x7fdc9c06c6c0 2023-03-03T09:12:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdca2723700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9c0c0e10 con 0x7fdc9c0c2e30 2023-03-03T09:12:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdca0cc0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c151cd0 0x7fdc9c1edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdca0cc0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c151cd0 0x7fdc9c1edb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54990/0 (socket says 172.21.15.7:54990) 2023-03-03T09:12:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdca0cc0700 1 -- 172.21.15.7:0/1692002859 learned_addr learned my addr 172.21.15.7:0/1692002859 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:12:45.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdc9bfff700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc9c06c6c0 0x7fdc9c1542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:45.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.836+0000 7fdc9b7fe700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc9c0c2e30 0x7fdc9c1eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:45.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdca0cc0700 1 -- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc9c0c2e30 msgr2=0x7fdc9c1eacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:45.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdca0cc0700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc9c0c2e30 0x7fdc9c1eacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:45.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdca0cc0700 1 -- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc9c06c6c0 msgr2=0x7fdc9c1542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:45.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdca0cc0700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc9c06c6c0 0x7fdc9c1542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:45.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdca0cc0700 1 -- 172.21.15.7:0/1692002859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c0094a0 con 0x7fdc9c151cd0 2023-03-03T09:12:45.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdca0cc0700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c151cd0 0x7fdc9c1edb40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400cbd0 tx=0x7fdc9400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdc98ff9700 1 -- 172.21.15.7:0/1692002859 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9400ab70 con 0x7fdc9c151cd0 2023-03-03T09:12:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdc98ff9700 1 -- 172.21.15.7:0/1692002859 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9400acf0 con 0x7fdc9c151cd0 2023-03-03T09:12:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdca2723700 1 -- 172.21.15.7:0/1692002859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9c157c80 con 0x7fdc9c151cd0 2023-03-03T09:12:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdc98ff9700 1 -- 172.21.15.7:0/1692002859 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94007310 con 0x7fdc9c151cd0 2023-03-03T09:12:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.837+0000 7fdca2723700 1 -- 172.21.15.7:0/1692002859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9c06d360 con 0x7fdc9c151cd0 2023-03-03T09:12:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.838+0000 7fdca2723700 1 -- 172.21.15.7:0/1692002859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9c1589c0 con 0x7fdc9c151cd0 2023-03-03T09:12:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.838+0000 7fdc98ff9700 1 -- 172.21.15.7:0/1692002859 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc9401a030 con 0x7fdc9c151cd0 2023-03-03T09:12:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.839+0000 7fdc98ff9700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc7c0682f0 0x7fdc7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.839+0000 7fdc98ff9700 1 -- 172.21.15.7:0/1692002859 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdc94086010 con 0x7fdc9c151cd0 2023-03-03T09:12:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.839+0000 7fdc9bfff700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc7c0682f0 0x7fdc7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.840+0000 7fdc9bfff700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc7c0682f0 0x7fdc7c06a7a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009870 tx=0x7fdc84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:45.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:45.843+0000 7fdc98ff9700 1 -- 172.21.15.7:0/1692002859 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc94052230 con 0x7fdc9c151cd0 2023-03-03T09:12:46.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.082+0000 7fdca2723700 1 -- 172.21.15.7:0/1692002859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc9c1ea9e0 con 0x7fdc9c151cd0 2023-03-03T09:12:46.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.082+0000 7fdc98ff9700 1 -- 172.21.15.7:0/1692002859 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdc9c1ea9e0 con 0x7fdc9c151cd0 2023-03-03T09:12:46.085 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:12:46.085 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:12:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.084+0000 7fdca2723700 1 -- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc7c0682f0 msgr2=0x7fdc7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.084+0000 7fdca2723700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc7c0682f0 0x7fdc7c06a7a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009870 tx=0x7fdc84008040 comp rx=0 tx=0).stop 2023-03-03T09:12:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.084+0000 7fdca2723700 1 -- 172.21.15.7:0/1692002859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c151cd0 msgr2=0x7fdc9c1edb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.084+0000 7fdca2723700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c151cd0 0x7fdc9c1edb40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400cbd0 tx=0x7fdc9400cf90 comp rx=0 tx=0).stop 2023-03-03T09:12:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.084+0000 7fdca2723700 1 -- 172.21.15.7:0/1692002859 shutdown_connections 2023-03-03T09:12:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.084+0000 7fdca2723700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc7c0682f0 0x7fdc7c06a7a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:46.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.084+0000 7fdca2723700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc9c151cd0 0x7fdc9c1edb40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:46.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.084+0000 7fdca2723700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc9c06c6c0 0x7fdc9c1542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:46.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.084+0000 7fdca2723700 1 --2- 172.21.15.7:0/1692002859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc9c0c2e30 0x7fdc9c1eacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:46.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.084+0000 7fdca2723700 1 -- 172.21.15.7:0/1692002859 >> 172.21.15.7:0/1692002859 conn(0x7fdc9c0a94d0 msgr2=0x7fdc9c0710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:46.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.085+0000 7fdca2723700 1 -- 172.21.15.7:0/1692002859 shutdown_connections 2023-03-03T09:12:46.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:46.085+0000 7fdca2723700 1 -- 172.21.15.7:0/1692002859 wait complete. 2023-03-03T09:12:46.089 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:12:46.098 INFO:tasks.mds_thrash.fs.[b]:waiting for 113 secs before thrashing 2023-03-03T09:12:54.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:12:54.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.314+0000 7f861c244700 1 Processor -- start 2023-03-03T09:12:54.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.315+0000 7f861c244700 1 -- start start 2023-03-03T09:12:54.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.315+0000 7f861c244700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614151ab0 0x7f86140736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:54.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.315+0000 7f861c244700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f861406c5b0 0x7f8614157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:54.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.315+0000 7f861c244700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861406d780 con 0x7f8614073c30 2023-03-03T09:12:54.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.315+0000 7f861c244700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861406cb70 con 0x7f861406c5b0 2023-03-03T09:12:54.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.315+0000 7f861c244700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861406ce50 con 0x7f8614151ab0 2023-03-03T09:12:54.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f861a7e1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f861406c5b0 0x7f8614157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:54.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f861a7e1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f861406c5b0 0x7f8614157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55006/0 (socket says 172.21.15.7:55006) 2023-03-03T09:12:54.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f861a7e1700 1 -- 172.21.15.7:0/2946517205 learned_addr learned my addr 172.21.15.7:0/2946517205 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:12:54.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f8619fe0700 1 --2- 172.21.15.7:0/2946517205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614151ab0 0x7f86140736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:54.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f861a7e1700 1 -- 172.21.15.7:0/2946517205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614151ab0 msgr2=0x7f86140736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:12:54.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f861a7e1700 1 --2- 172.21.15.7:0/2946517205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614151ab0 0x7f86140736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:54.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f861a7e1700 1 -- 172.21.15.7:0/2946517205 >> v1:172.21.15.143:6789/0 conn(0x7f8614073c30 legacy=0x7f861406bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:12:54.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f861a7e1700 1 -- 172.21.15.7:0/2946517205 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861406d130 con 0x7f861406c5b0 2023-03-03T09:12:54.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f8619fe0700 1 --2- 172.21.15.7:0/2946517205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614151ab0 0x7f86140736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:12:54.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f861a7e1700 1 --2- 172.21.15.7:0/2946517205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406c5b0 0x7f8614157dc0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f860c009e50 tx=0x7f860c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c857ba2bcda37110 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:54.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f8618fde700 1 -- 172.21.15.7:0/2946517205 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c00e070 con 0x7f861406c5b0 2023-03-03T09:12:54.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f8618fde700 1 -- 172.21.15.7:0/2946517205 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860c003a20 con 0x7f861406c5b0 2023-03-03T09:12:54.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.316+0000 7f8618fde700 1 -- 172.21.15.7:0/2946517205 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c003dd0 con 0x7f861406c5b0 2023-03-03T09:12:54.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.317+0000 7f861c244700 1 -- 172.21.15.7:0/2946517205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406c5b0 msgr2=0x7f8614157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:54.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.317+0000 7f861c244700 1 --2- 172.21.15.7:0/2946517205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406c5b0 0x7f8614157dc0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f860c009e50 tx=0x7f860c00b330 comp rx=0 tx=0).stop 2023-03-03T09:12:54.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.317+0000 7f861c244700 1 -- 172.21.15.7:0/2946517205 shutdown_connections 2023-03-03T09:12:54.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.317+0000 7f861c244700 1 --2- 172.21.15.7:0/2946517205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406c5b0 0x7f8614157dc0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:54.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.317+0000 7f861c244700 1 --2- 172.21.15.7:0/2946517205 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614151ab0 0x7f86140736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:54.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.317+0000 7f861c244700 1 -- 172.21.15.7:0/2946517205 >> 172.21.15.7:0/2946517205 conn(0x7f86140a94d0 msgr2=0x7f86140a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:54.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.317+0000 7f861c244700 1 -- 172.21.15.7:0/2946517205 shutdown_connections 2023-03-03T09:12:54.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.317+0000 7f861c244700 1 -- 172.21.15.7:0/2946517205 wait complete. 2023-03-03T09:12:54.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.318+0000 7f861c244700 1 Processor -- start 2023-03-03T09:12:54.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.318+0000 7f861c244700 1 -- start start 2023-03-03T09:12:54.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861c244700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f861406c5b0 0x7f86140718d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:54.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861c244700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614073c30 0x7f86141ea040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:54.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861c244700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8614151ab0 0x7f86141ea580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:54.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861c244700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8614147d30 con 0x7f8614151ab0 2023-03-03T09:12:54.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861c244700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8614154b80 con 0x7f861406c5b0 2023-03-03T09:12:54.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861c244700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8614155390 con 0x7f8614073c30 2023-03-03T09:12:54.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861a7e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8614151ab0 0x7f86141ea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:54.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861a7e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8614151ab0 0x7f86141ea580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55012/0 (socket says 172.21.15.7:55012) 2023-03-03T09:12:54.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861a7e1700 1 -- 172.21.15.7:0/3705644582 learned_addr learned my addr 172.21.15.7:0/3705644582 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:12:54.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f8619fe0700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f861406c5b0 0x7f86140718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:54.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f86197df700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614073c30 0x7f86141ea040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:54.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861a7e1700 1 -- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614073c30 msgr2=0x7f86141ea040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:54.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861a7e1700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614073c30 0x7f86141ea040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:54.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861a7e1700 1 -- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f861406c5b0 msgr2=0x7f86140718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:54.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861a7e1700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f861406c5b0 0x7f86140718d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:54.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f861a7e1700 1 -- 172.21.15.7:0/3705644582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c009b30 con 0x7f8614151ab0 2023-03-03T09:12:54.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.319+0000 7f86197df700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614073c30 0x7f86141ea040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:12:54.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.320+0000 7f861a7e1700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8614151ab0 0x7f86141ea580 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f860c015040 tx=0x7f860c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:54.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.320+0000 7f860a7fc700 1 -- 172.21.15.7:0/3705644582 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c004440 con 0x7f8614151ab0 2023-03-03T09:12:54.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.320+0000 7f860a7fc700 1 -- 172.21.15.7:0/3705644582 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860c0045c0 con 0x7f8614151ab0 2023-03-03T09:12:54.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.320+0000 7f861c244700 1 -- 172.21.15.7:0/3705644582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8614151e80 con 0x7f8614151ab0 2023-03-03T09:12:54.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.320+0000 7f860a7fc700 1 -- 172.21.15.7:0/3705644582 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c004970 con 0x7f8614151ab0 2023-03-03T09:12:54.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.320+0000 7f861c244700 1 -- 172.21.15.7:0/3705644582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861406d410 con 0x7f8614151ab0 2023-03-03T09:12:54.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.321+0000 7f861c244700 1 -- 172.21.15.7:0/3705644582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8614152010 con 0x7f8614151ab0 2023-03-03T09:12:54.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.321+0000 7f860a7fc700 1 -- 172.21.15.7:0/3705644582 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f860c022030 con 0x7f8614151ab0 2023-03-03T09:12:54.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.322+0000 7f860a7fc700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f80682f0 0x7f85f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:12:54.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.322+0000 7f860a7fc700 1 -- 172.21.15.7:0/3705644582 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f860c087470 con 0x7f8614151ab0 2023-03-03T09:12:54.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.322+0000 7f8619fe0700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f80682f0 0x7f85f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:12:54.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.323+0000 7f8619fe0700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f80682f0 0x7f85f806a7a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8614146af0 tx=0x7f8604008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:12:54.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.326+0000 7f860a7fc700 1 -- 172.21.15.7:0/3705644582 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f860c053810 con 0x7f8614151ab0 2023-03-03T09:12:54.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.565+0000 7f861c244700 1 -- 172.21.15.7:0/3705644582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8614074bb0 con 0x7f8614151ab0 2023-03-03T09:12:54.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.565+0000 7f860a7fc700 1 -- 172.21.15.7:0/3705644582 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f860c007310 con 0x7f8614151ab0 2023-03-03T09:12:54.567 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:12:54.567 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:12:54.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 -- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f80682f0 msgr2=0x7f85f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:54.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f80682f0 0x7f85f806a7a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8614146af0 tx=0x7f8604008040 comp rx=0 tx=0).stop 2023-03-03T09:12:54.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 -- 172.21.15.7:0/3705644582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8614151ab0 msgr2=0x7f86141ea580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:12:54.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8614151ab0 0x7f86141ea580 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f860c015040 tx=0x7f860c004ad0 comp rx=0 tx=0).stop 2023-03-03T09:12:54.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 -- 172.21.15.7:0/3705644582 shutdown_connections 2023-03-03T09:12:54.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f80682f0 0x7f85f806a7a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:54.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8614151ab0 0x7f86141ea580 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:54.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f861406c5b0 0x7f86140718d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:54.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 --2- 172.21.15.7:0/3705644582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8614073c30 0x7f86141ea040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:12:54.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 -- 172.21.15.7:0/3705644582 >> 172.21.15.7:0/3705644582 conn(0x7f86140a94d0 msgr2=0x7f86140c3050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:12:54.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 -- 172.21.15.7:0/3705644582 shutdown_connections 2023-03-03T09:12:54.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:12:54.567+0000 7f861c244700 1 -- 172.21.15.7:0/3705644582 wait complete. 2023-03-03T09:12:54.570 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:12:54.580 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T09:13:12.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:12.076 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:42.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:42.178 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:13:59.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:13:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.721+0000 7f7ed0cc2700 1 Processor -- start 2023-03-03T09:13:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.721+0000 7f7ed0cc2700 1 -- start start 2023-03-03T09:13:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.722+0000 7f7ed0cc2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7ecc0ace30 0x7f7ecc152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.722+0000 7f7ed0cc2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ecc0c26e0 0x7f7ecc152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.722+0000 7f7ed0cc2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ecc0ac900 con 0x7f7ecc0ac220 2023-03-03T09:13:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.722+0000 7f7ed0cc2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ecc071e30 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.722+0000 7f7ed0cc2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ecc152ed0 con 0x7f7ecc0ace30 2023-03-03T09:13:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.722+0000 7f7ecad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ecc0c26e0 0x7f7ecc152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.722+0000 7f7eca59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7ecc0ac220 0x7f7ecc0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49042/0 (socket says 172.21.15.7:49042) 2023-03-03T09:13:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.722+0000 7f7eca59c700 1 -- 172.21.15.7:0/3221712127 learned_addr learned my addr 172.21.15.7:0/3221712127 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:13:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ec9d9b700 1 --2- 172.21.15.7:0/3221712127 >> v2:172.21.15.143:3300/0 conn(0x7f7ecc0ace30 0x7f7ecc152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ec959a700 1 -- 172.21.15.7:0/3221712127 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404100405 0 0) 0x7f7ecc0ac900 con 0x7f7ecc0ac220 2023-03-03T09:13:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ec959a700 1 -- 172.21.15.7:0/3221712127 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb80035b0 con 0x7f7ecc0ac220 2023-03-03T09:13:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ecad9d700 1 -- 172.21.15.7:0/3221712127 >> v2:172.21.15.143:3300/0 conn(0x7f7ecc0ace30 msgr2=0x7f7ecc152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:13:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ecad9d700 1 --2- 172.21.15.7:0/3221712127 >> v2:172.21.15.143:3300/0 conn(0x7f7ecc0ace30 0x7f7ecc152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ecad9d700 1 -- 172.21.15.7:0/3221712127 >> v1:172.21.15.7:6789/0 conn(0x7f7ecc0ac220 legacy=0x7f7ecc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ecad9d700 1 -- 172.21.15.7:0/3221712127 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc0acb50 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ec9d9b700 1 --2- 172.21.15.7:0/3221712127 >> v2:172.21.15.143:3300/0 conn(0x7f7ecc0ace30 0x7f7ecc152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:13:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ecad9d700 1 --2- 172.21.15.7:0/3221712127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 0x7f7ecc152990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0c4eb0 tx=0x7f7ec400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff480545a5ac58b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:13:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ec959a700 1 -- 172.21.15.7:0/3221712127 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec400e070 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ec959a700 1 -- 172.21.15.7:0/3221712127 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec400bb20 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.723+0000 7f7ec959a700 1 -- 172.21.15.7:0/3221712127 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4012430 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.724+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/3221712127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 msgr2=0x7f7ecc152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.724+0000 7f7ed0cc2700 1 --2- 172.21.15.7:0/3221712127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 0x7f7ecc152990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0c4eb0 tx=0x7f7ec400b330 comp rx=0 tx=0).stop 2023-03-03T09:13:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.724+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/3221712127 shutdown_connections 2023-03-03T09:13:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.724+0000 7f7ed0cc2700 1 --2- 172.21.15.7:0/3221712127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 0x7f7ecc152990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.724+0000 7f7ed0cc2700 1 --2- 172.21.15.7:0/3221712127 >> v2:172.21.15.143:3300/0 conn(0x7f7ecc0ace30 0x7f7ecc152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.724+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/3221712127 >> 172.21.15.7:0/3221712127 conn(0x7f7ecc0a94d0 msgr2=0x7f7ecc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:13:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.724+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/3221712127 shutdown_connections 2023-03-03T09:13:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.724+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/3221712127 wait complete. 2023-03-03T09:13:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.726+0000 7f7ed0cc2700 1 Processor -- start 2023-03-03T09:13:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.726+0000 7f7ed0cc2700 1 -- start start 2023-03-03T09:13:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ed0cc2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ecc0ac220 0x7f7ecc14db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ed0cc2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0ace30 0x7f7ecc0c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ed0cc2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 0x7f7ecc1e7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ed0cc2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc1e7730 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ed0cc2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc0c0460 con 0x7f7ecc0ace30 2023-03-03T09:13:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ed0cc2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc0bd990 con 0x7f7ecc0ac220 2023-03-03T09:13:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ecad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 0x7f7ecc1e7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ecad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 0x7f7ecc1e7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55072/0 (socket says 172.21.15.7:55072) 2023-03-03T09:13:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ecad9d700 1 -- 172.21.15.7:0/2957743997 learned_addr learned my addr 172.21.15.7:0/2957743997 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:13:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7eca59c700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ecc0ac220 0x7f7ecc14db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ec9d9b700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0ace30 0x7f7ecc0c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ecad9d700 1 -- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ecc0ac220 msgr2=0x7f7ecc14db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ecad9d700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ecc0ac220 0x7f7ecc14db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:59.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ecad9d700 1 -- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0ace30 msgr2=0x7f7ecc0c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:59.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ecad9d700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0ace30 0x7f7ecc0c61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:59.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ecad9d700 1 -- 172.21.15.7:0/2957743997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec4009a70 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7eca59c700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ecc0ac220 0x7f7ecc14db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:13:59.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ec9d9b700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0ace30 0x7f7ecc0c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:13:59.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ecad9d700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 0x7f7ecc1e7b80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc06b980 tx=0x7f7ec4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:13:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.727+0000 7f7ec2ffd700 1 -- 172.21.15.7:0/2957743997 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4012430 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.728+0000 7f7ec2ffd700 1 -- 172.21.15.7:0/2957743997 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec4020070 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.728+0000 7f7ec2ffd700 1 -- 172.21.15.7:0/2957743997 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4004280 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.728+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/2957743997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ecc155ca0 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.728+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/2957743997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc1559c0 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.729+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/2957743997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ecc155f80 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.729+0000 7f7ec2ffd700 1 -- 172.21.15.7:0/2957743997 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7ec402e070 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.729+0000 7f7ec2ffd700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7eac0682f0 0x7f7eac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:13:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.730+0000 7f7ec2ffd700 1 -- 172.21.15.7:0/2957743997 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7ec4022070 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.730+0000 7f7eca59c700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7eac0682f0 0x7f7eac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:13:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.730+0000 7f7eca59c700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7eac0682f0 0x7f7eac06a7a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc005910 tx=0x7f7ebc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:13:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.734+0000 7f7ec2ffd700 1 -- 172.21.15.7:0/2957743997 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7ec4054240 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.972+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/2957743997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ecc1e7870 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.972+0000 7f7ec2ffd700 1 -- 172.21.15.7:0/2957743997 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7ec405be60 con 0x7f7ecc0c26e0 2023-03-03T09:13:59.974 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:13:59.974 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:13:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.974+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7eac0682f0 msgr2=0x7f7eac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.974+0000 7f7ed0cc2700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7eac0682f0 0x7f7eac06a7a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc005910 tx=0x7f7ebc009040 comp rx=0 tx=0).stop 2023-03-03T09:13:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.974+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/2957743997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 msgr2=0x7f7ecc1e7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:13:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.974+0000 7f7ed0cc2700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 0x7f7ecc1e7b80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc06b980 tx=0x7f7ec4004ad0 comp rx=0 tx=0).stop 2023-03-03T09:13:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.974+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/2957743997 shutdown_connections 2023-03-03T09:13:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.974+0000 7f7ed0cc2700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7eac0682f0 0x7f7eac06a7a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.974+0000 7f7ed0cc2700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ecc0c26e0 0x7f7ecc1e7b80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.974+0000 7f7ed0cc2700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ecc0ac220 0x7f7ecc14db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.974+0000 7f7ed0cc2700 1 --2- 172.21.15.7:0/2957743997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ecc0ace30 0x7f7ecc0c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:13:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.974+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/2957743997 >> 172.21.15.7:0/2957743997 conn(0x7f7ecc0a94d0 msgr2=0x7f7ecc0aa920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:13:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.975+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/2957743997 shutdown_connections 2023-03-03T09:13:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:13:59.975+0000 7f7ed0cc2700 1 -- 172.21.15.7:0/2957743997 wait complete. 2023-03-03T09:13:59.977 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:13:59.988 INFO:tasks.mds_thrash.fs.[a]:waiting for 63 secs before thrashing 2023-03-03T09:14:12.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:12.251 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:39.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:14:39.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.238+0000 7f3233ed9700 1 Processor -- start 2023-03-03T09:14:39.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.238+0000 7f3233ed9700 1 -- start start 2023-03-03T09:14:39.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3233ed9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f322c14f9d0 0x7f322c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:14:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3233ed9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f322c0abe20 0x7f322c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:14:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3233ed9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c0c2f30 con 0x7f322c14fda0 2023-03-03T09:14:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3233ed9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c0c31d0 con 0x7f322c14f9d0 2023-03-03T09:14:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3233ed9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c1524a0 con 0x7f322c0abe20 2023-03-03T09:14:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3231c75700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f322c14f9d0 0x7f322c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:14:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3231c75700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f322c14f9d0 0x7f322c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55106/0 (socket says 172.21.15.7:55106) 2023-03-03T09:14:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3231474700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f322c14fda0 0x7f322c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49086/0 (socket says 172.21.15.7:49086) 2023-03-03T09:14:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3231c75700 1 -- 172.21.15.7:0/670068932 learned_addr learned my addr 172.21.15.7:0/670068932 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:14:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3232476700 1 --2- 172.21.15.7:0/670068932 >> v2:172.21.15.143:3300/0 conn(0x7f322c0abe20 0x7f322c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:14:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3231c75700 1 -- 172.21.15.7:0/670068932 >> v2:172.21.15.143:3300/0 conn(0x7f322c0abe20 msgr2=0x7f322c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:14:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3231c75700 1 --2- 172.21.15.7:0/670068932 >> v2:172.21.15.143:3300/0 conn(0x7f322c0abe20 0x7f322c1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3231c75700 1 -- 172.21.15.7:0/670068932 >> v1:172.21.15.7:6789/0 conn(0x7f322c14fda0 legacy=0x7f322c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:14:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3231c75700 1 -- 172.21.15.7:0/670068932 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0c34b0 con 0x7f322c14f9d0 2023-03-03T09:14:39.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.239+0000 7f3231c75700 1 --2- 172.21.15.7:0/670068932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c14f9d0 0x7f322c156430 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f322c071230 tx=0x7f3228009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6291c33d950c95a1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:14:39.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3230c73700 1 -- 172.21.15.7:0/670068932 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322800b870 con 0x7f322c14f9d0 2023-03-03T09:14:39.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3230c73700 1 -- 172.21.15.7:0/670068932 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322800b9d0 con 0x7f322c14f9d0 2023-03-03T09:14:39.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3230c73700 1 -- 172.21.15.7:0/670068932 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322800bd60 con 0x7f322c14f9d0 2023-03-03T09:14:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3233ed9700 1 -- 172.21.15.7:0/670068932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c14f9d0 msgr2=0x7f322c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:14:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3233ed9700 1 --2- 172.21.15.7:0/670068932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c14f9d0 0x7f322c156430 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f322c071230 tx=0x7f3228009bd0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3233ed9700 1 -- 172.21.15.7:0/670068932 shutdown_connections 2023-03-03T09:14:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3233ed9700 1 --2- 172.21.15.7:0/670068932 >> v2:172.21.15.143:3300/0 conn(0x7f322c0abe20 0x7f322c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3233ed9700 1 --2- 172.21.15.7:0/670068932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c14f9d0 0x7f322c156430 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3233ed9700 1 -- 172.21.15.7:0/670068932 >> 172.21.15.7:0/670068932 conn(0x7f322c0c8900 msgr2=0x7f322c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:14:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3233ed9700 1 -- 172.21.15.7:0/670068932 shutdown_connections 2023-03-03T09:14:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.240+0000 7f3233ed9700 1 -- 172.21.15.7:0/670068932 wait complete. 2023-03-03T09:14:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.241+0000 7f3233ed9700 1 Processor -- start 2023-03-03T09:14:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3233ed9700 1 -- start start 2023-03-03T09:14:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3233ed9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c0abe20 0x7f322c06e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:14:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3233ed9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322c14f9d0 0x7f322c1eacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:14:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3233ed9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f322c14fda0 0x7f322c151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:14:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3233ed9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c1dd000 con 0x7f322c0abe20 2023-03-03T09:14:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3233ed9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c1de020 con 0x7f322c14f9d0 2023-03-03T09:14:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3233ed9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c06ced0 con 0x7f322c14fda0 2023-03-03T09:14:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3231c75700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c0abe20 0x7f322c06e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:14:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3231c75700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c0abe20 0x7f322c06e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55112/0 (socket says 172.21.15.7:55112) 2023-03-03T09:14:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3231c75700 1 -- 172.21.15.7:0/1601016001 learned_addr learned my addr 172.21.15.7:0/1601016001 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:14:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.242+0000 7f3231474700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322c14f9d0 0x7f322c1eacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:14:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f3232476700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f322c14fda0 0x7f322c151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:14:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f3231c75700 1 -- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f322c14fda0 msgr2=0x7f322c151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:14:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f3231c75700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f322c14fda0 0x7f322c151910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f3231c75700 1 -- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322c14f9d0 msgr2=0x7f322c1eacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:14:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f3231c75700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322c14f9d0 0x7f322c1eacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f3231c75700 1 -- 172.21.15.7:0/1601016001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3228009550 con 0x7f322c0abe20 2023-03-03T09:14:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f3231474700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322c14f9d0 0x7f322c1eacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:14:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f3231c75700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c0abe20 0x7f322c06e170 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f322c072520 tx=0x7f3228004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:14:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f32227fc700 1 -- 172.21.15.7:0/1601016001 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228004310 con 0x7f322c0abe20 2023-03-03T09:14:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f32227fc700 1 -- 172.21.15.7:0/1601016001 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322801f070 con 0x7f322c0abe20 2023-03-03T09:14:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f3233ed9700 1 -- 172.21.15.7:0/1601016001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c0ac350 con 0x7f322c0abe20 2023-03-03T09:14:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f32227fc700 1 -- 172.21.15.7:0/1601016001 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32280036a0 con 0x7f322c0abe20 2023-03-03T09:14:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.243+0000 7f3233ed9700 1 -- 172.21.15.7:0/1601016001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c0ab630 con 0x7f322c0abe20 2023-03-03T09:14:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.244+0000 7f3233ed9700 1 -- 172.21.15.7:0/1601016001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c071fa0 con 0x7f322c0abe20 2023-03-03T09:14:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.244+0000 7f32227fc700 1 -- 172.21.15.7:0/1601016001 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f322802a020 con 0x7f322c0abe20 2023-03-03T09:14:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.245+0000 7f32227fc700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32100682f0 0x7f321006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:14:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.245+0000 7f32227fc700 1 -- 172.21.15.7:0/1601016001 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3228087700 con 0x7f322c0abe20 2023-03-03T09:14:39.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.245+0000 7f3231474700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32100682f0 0x7f321006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:14:39.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.246+0000 7f3231474700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32100682f0 0x7f321006a7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f322c14cd60 tx=0x7f321c007dd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:14:39.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.249+0000 7f32227fc700 1 -- 172.21.15.7:0/1601016001 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3228053aa0 con 0x7f322c0abe20 2023-03-03T09:14:39.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.487+0000 7f3233ed9700 1 -- 172.21.15.7:0/1601016001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f322c0c8230 con 0x7f322c0abe20 2023-03-03T09:14:39.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.487+0000 7f32227fc700 1 -- 172.21.15.7:0/1601016001 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f32280570c0 con 0x7f322c0abe20 2023-03-03T09:14:39.488 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:14:39.489 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:14:39.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 -- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32100682f0 msgr2=0x7f321006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:14:39.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32100682f0 0x7f321006a7a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f322c14cd60 tx=0x7f321c007dd0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 -- 172.21.15.7:0/1601016001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c0abe20 msgr2=0x7f322c06e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:14:39.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c0abe20 0x7f322c06e170 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f322c072520 tx=0x7f3228004750 comp rx=0 tx=0).stop 2023-03-03T09:14:39.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 -- 172.21.15.7:0/1601016001 shutdown_connections 2023-03-03T09:14:39.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f322c14fda0 0x7f322c151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32100682f0 0x7f321006a7a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f322c0abe20 0x7f322c06e170 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 --2- 172.21.15.7:0/1601016001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322c14f9d0 0x7f322c1eacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:14:39.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 -- 172.21.15.7:0/1601016001 >> 172.21.15.7:0/1601016001 conn(0x7f322c0c8900 msgr2=0x7f322c150ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:14:39.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 -- 172.21.15.7:0/1601016001 shutdown_connections 2023-03-03T09:14:39.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:14:39.489+0000 7f3233ed9700 1 -- 172.21.15.7:0/1601016001 wait complete. 2023-03-03T09:14:39.493 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:14:39.503 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2023-03-03T09:14:42.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:14:42.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:02.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:15:03.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.126+0000 7f80faf4a700 1 Processor -- start 2023-03-03T09:15:03.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.126+0000 7f80faf4a700 1 -- start start 2023-03-03T09:15:03.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80faf4a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80f40aad20 0x7f80f4157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:03.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80faf4a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f80f40aa030 0x7f80f41581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:03.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80faf4a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f40728b0 con 0x7f80f414b020 2023-03-03T09:15:03.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80faf4a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f4155070 con 0x7f80f40aad20 2023-03-03T09:15:03.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80faf4a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f406f5b0 con 0x7f80f40aa030 2023-03-03T09:15:03.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80f3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80f40aad20 0x7f80f4157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:03.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80f8ce6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f80f414b020 0x7f80f40717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49132/0 (socket says 172.21.15.7:49132) 2023-03-03T09:15:03.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80f8ce6700 1 -- 172.21.15.7:0/4132491996 learned_addr learned my addr 172.21.15.7:0/4132491996 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:15:03.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80f94e7700 1 --2- 172.21.15.7:0/4132491996 >> v2:172.21.15.143:3300/0 conn(0x7f80f40aa030 0x7f80f41581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:03.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80f3fff700 1 -- 172.21.15.7:0/4132491996 >> v2:172.21.15.143:3300/0 conn(0x7f80f40aa030 msgr2=0x7f80f41581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:15:03.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80f3fff700 1 --2- 172.21.15.7:0/4132491996 >> v2:172.21.15.143:3300/0 conn(0x7f80f40aa030 0x7f80f41581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:03.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80f3fff700 1 -- 172.21.15.7:0/4132491996 >> v1:172.21.15.7:6789/0 conn(0x7f80f414b020 legacy=0x7f80f40717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:03.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80f37fe700 1 -- 172.21.15.7:0/4132491996 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733676722 0 0) 0x7f80f40728b0 con 0x7f80f414b020 2023-03-03T09:15:03.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80f3fff700 1 -- 172.21.15.7:0/4132491996 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f406f890 con 0x7f80f40aad20 2023-03-03T09:15:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.127+0000 7f80f3fff700 1 --2- 172.21.15.7:0/4132491996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f40aad20 0x7f80f4157ca0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f80f40adb00 tx=0x7f80e0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1f6120a3c9ddd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:15:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80f37fe700 1 -- 172.21.15.7:0/4132491996 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e0011830 con 0x7f80f40aad20 2023-03-03T09:15:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80f37fe700 1 -- 172.21.15.7:0/4132491996 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e0011990 con 0x7f80f40aad20 2023-03-03T09:15:03.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80f37fe700 1 -- 172.21.15.7:0/4132491996 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e0011d20 con 0x7f80f40aad20 2023-03-03T09:15:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80faf4a700 1 -- 172.21.15.7:0/4132491996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f40aad20 msgr2=0x7f80f4157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80faf4a700 1 --2- 172.21.15.7:0/4132491996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f40aad20 0x7f80f4157ca0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f80f40adb00 tx=0x7f80e0009b20 comp rx=0 tx=0).stop 2023-03-03T09:15:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80faf4a700 1 -- 172.21.15.7:0/4132491996 shutdown_connections 2023-03-03T09:15:03.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80faf4a700 1 --2- 172.21.15.7:0/4132491996 >> v2:172.21.15.143:3300/0 conn(0x7f80f40aa030 0x7f80f41581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80faf4a700 1 --2- 172.21.15.7:0/4132491996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f40aad20 0x7f80f4157ca0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80faf4a700 1 -- 172.21.15.7:0/4132491996 >> 172.21.15.7:0/4132491996 conn(0x7f80f40c8900 msgr2=0x7f80f40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:15:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80faf4a700 1 -- 172.21.15.7:0/4132491996 shutdown_connections 2023-03-03T09:15:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.128+0000 7f80faf4a700 1 -- 172.21.15.7:0/4132491996 wait complete. 2023-03-03T09:15:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.130+0000 7f80faf4a700 1 Processor -- start 2023-03-03T09:15:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.130+0000 7f80faf4a700 1 -- start start 2023-03-03T09:15:03.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80faf4a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f40aa030 0x7f80f41ec950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80faf4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f40aad20 0x7f80f41ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80faf4a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f414b020 0x7f80f41ed930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80faf4a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f41ede70 con 0x7f80f414b020 2023-03-03T09:15:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80faf4a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f41ddb70 con 0x7f80f40aa030 2023-03-03T09:15:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80faf4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f41dde50 con 0x7f80f40aad20 2023-03-03T09:15:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80f94e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f414b020 0x7f80f41ed930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80f94e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f414b020 0x7f80f41ed930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55162/0 (socket says 172.21.15.7:55162) 2023-03-03T09:15:03.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80f94e7700 1 -- 172.21.15.7:0/3901416668 learned_addr learned my addr 172.21.15.7:0/3901416668 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:15:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80f3fff700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f40aad20 0x7f80f41ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80f8ce6700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f40aa030 0x7f80f41ec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80f94e7700 1 -- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f40aad20 msgr2=0x7f80f41ed3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80f94e7700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f40aad20 0x7f80f41ed3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80f94e7700 1 -- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f40aa030 msgr2=0x7f80f41ec950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80f94e7700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f40aa030 0x7f80f41ec950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.131+0000 7f80f94e7700 1 -- 172.21.15.7:0/3901416668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e00094a0 con 0x7f80f414b020 2023-03-03T09:15:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.132+0000 7f80f94e7700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f414b020 0x7f80f41ed930 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f80ec007e60 tx=0x7f80ec00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:15:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.132+0000 7f80f17fa700 1 -- 172.21.15.7:0/3901416668 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec00ede0 con 0x7f80f414b020 2023-03-03T09:15:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.132+0000 7f80f17fa700 1 -- 172.21.15.7:0/3901416668 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ec014420 con 0x7f80f414b020 2023-03-03T09:15:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.132+0000 7f80f17fa700 1 -- 172.21.15.7:0/3901416668 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec0147d0 con 0x7f80f414b020 2023-03-03T09:15:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.132+0000 7f80faf4a700 1 -- 172.21.15.7:0/3901416668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f4155350 con 0x7f80f414b020 2023-03-03T09:15:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.132+0000 7f80faf4a700 1 -- 172.21.15.7:0/3901416668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f406fb70 con 0x7f80f414b020 2023-03-03T09:15:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.133+0000 7f80faf4a700 1 -- 172.21.15.7:0/3901416668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f4071ef0 con 0x7f80f414b020 2023-03-03T09:15:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.133+0000 7f80f17fa700 1 -- 172.21.15.7:0/3901416668 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f80ec012050 con 0x7f80f414b020 2023-03-03T09:15:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.134+0000 7f80f17fa700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80d80682f0 0x7f80d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.134+0000 7f80f17fa700 1 -- 172.21.15.7:0/3901416668 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f80ec086750 con 0x7f80f414b020 2023-03-03T09:15:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.134+0000 7f80f8ce6700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80d80682f0 0x7f80d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.134+0000 7f80f8ce6700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80d80682f0 0x7f80d806a7a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f80e8000f60 tx=0x7f80e8007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:15:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.138+0000 7f80f17fa700 1 -- 172.21.15.7:0/3901416668 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80ec052af0 con 0x7f80f414b020 2023-03-03T09:15:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.373+0000 7f80faf4a700 1 -- 172.21.15.7:0/3901416668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80f40a9810 con 0x7f80f414b020 2023-03-03T09:15:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.373+0000 7f80f17fa700 1 -- 172.21.15.7:0/3901416668 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f80ec056110 con 0x7f80f414b020 2023-03-03T09:15:03.375 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:15:03.375 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:15:03.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 -- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80d80682f0 msgr2=0x7f80d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:03.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80d80682f0 0x7f80d806a7a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f80e8000f60 tx=0x7f80e8007c90 comp rx=0 tx=0).stop 2023-03-03T09:15:03.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 -- 172.21.15.7:0/3901416668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f414b020 msgr2=0x7f80f41ed930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:03.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f414b020 0x7f80f41ed930 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f80ec007e60 tx=0x7f80ec00e3c0 comp rx=0 tx=0).stop 2023-03-03T09:15:03.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 -- 172.21.15.7:0/3901416668 shutdown_connections 2023-03-03T09:15:03.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80d80682f0 0x7f80d806a7a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:03.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80f414b020 0x7f80f41ed930 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:03.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80f40aa030 0x7f80f41ec950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:03.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 --2- 172.21.15.7:0/3901416668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80f40aad20 0x7f80f41ed3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:03.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 -- 172.21.15.7:0/3901416668 >> 172.21.15.7:0/3901416668 conn(0x7f80f40c8900 msgr2=0x7f80f41489d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:15:03.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 -- 172.21.15.7:0/3901416668 shutdown_connections 2023-03-03T09:15:03.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:03.375+0000 7f80faf4a700 1 -- 172.21.15.7:0/3901416668 wait complete. 2023-03-03T09:15:03.378 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:15:03.388 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T09:15:12.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:12.419 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:42.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:42.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:15:52.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:15:52.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.527+0000 7ff70f131700 1 Processor -- start 2023-03-03T09:15:52.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.527+0000 7ff70f131700 1 -- start start 2023-03-03T09:15:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70f131700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff7080c1da0 0x7ff7081550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70f131700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff7080c3e30 0x7ff7080c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70f131700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7080c7a60 con 0x7ff70806d870 2023-03-03T09:15:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70f131700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7080c2af0 con 0x7ff7080c3e30 2023-03-03T09:15:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70f131700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7080c3620 con 0x7ff7080c1da0 2023-03-03T09:15:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70d6ce700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff7080c3e30 0x7ff7080c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70d6ce700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff7080c3e30 0x7ff7080c4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55222/0 (socket says 172.21.15.7:55222) 2023-03-03T09:15:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70d6ce700 1 -- 172.21.15.7:0/1507779960 learned_addr learned my addr 172.21.15.7:0/1507779960 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:15:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff707fff700 1 --2- 172.21.15.7:0/1507779960 >> v2:172.21.15.143:3300/0 conn(0x7ff7080c1da0 0x7ff7081550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70d6ce700 1 -- 172.21.15.7:0/1507779960 >> v2:172.21.15.143:3300/0 conn(0x7ff7080c1da0 msgr2=0x7ff7081550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:15:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70d6ce700 1 --2- 172.21.15.7:0/1507779960 >> v2:172.21.15.143:3300/0 conn(0x7ff7080c1da0 0x7ff7081550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70d6ce700 1 -- 172.21.15.7:0/1507779960 >> v1:172.21.15.143:6789/0 conn(0x7ff70806d870 legacy=0x7ff7080c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70d6ce700 1 -- 172.21.15.7:0/1507779960 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7081545e0 con 0x7ff7080c3e30 2023-03-03T09:15:52.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.528+0000 7ff70d6ce700 1 --2- 172.21.15.7:0/1507779960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7080c3e30 0x7ff7080c4230 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff7080c33a0 tx=0x7ff70000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e96cea1a3551245 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:15:52.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff7077fe700 1 -- 172.21.15.7:0/1507779960 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70000e070 con 0x7ff7080c3e30 2023-03-03T09:15:52.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff7077fe700 1 -- 172.21.15.7:0/1507779960 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70000b9c0 con 0x7ff7080c3e30 2023-03-03T09:15:52.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff7077fe700 1 -- 172.21.15.7:0/1507779960 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70000bd70 con 0x7ff7080c3e30 2023-03-03T09:15:52.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff70f131700 1 -- 172.21.15.7:0/1507779960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7080c3e30 msgr2=0x7ff7080c4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:52.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff70f131700 1 --2- 172.21.15.7:0/1507779960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7080c3e30 0x7ff7080c4230 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff7080c33a0 tx=0x7ff70000b330 comp rx=0 tx=0).stop 2023-03-03T09:15:52.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff70f131700 1 -- 172.21.15.7:0/1507779960 shutdown_connections 2023-03-03T09:15:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff70f131700 1 --2- 172.21.15.7:0/1507779960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7080c3e30 0x7ff7080c4230 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff70f131700 1 --2- 172.21.15.7:0/1507779960 >> v2:172.21.15.143:3300/0 conn(0x7ff7080c1da0 0x7ff7081550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff70f131700 1 -- 172.21.15.7:0/1507779960 >> 172.21.15.7:0/1507779960 conn(0x7ff70806c110 msgr2=0x7ff70806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:15:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff70f131700 1 -- 172.21.15.7:0/1507779960 shutdown_connections 2023-03-03T09:15:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.529+0000 7ff70f131700 1 -- 172.21.15.7:0/1507779960 wait complete. 2023-03-03T09:15:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.530+0000 7ff70f131700 1 Processor -- start 2023-03-03T09:15:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.530+0000 7ff70f131700 1 -- start start 2023-03-03T09:15:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.531+0000 7ff70f131700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff70806d870 0x7ff708149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:52.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.531+0000 7ff70f131700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7080c1da0 0x7ff708146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:52.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.531+0000 7ff70f131700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c3e30 0x7ff708147500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:52.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.531+0000 7ff70f131700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7081ea900 con 0x7ff70806d870 2023-03-03T09:15:52.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.531+0000 7ff70f131700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7081ea380 con 0x7ff7080c1da0 2023-03-03T09:15:52.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.531+0000 7ff70f131700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7081ea620 con 0x7ff7080c3e30 2023-03-03T09:15:52.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.531+0000 7ff70cecd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff70806d870 0x7ff708149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:52.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.531+0000 7ff70cecd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff70806d870 0x7ff708149600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55224/0 (socket says 172.21.15.7:55224) 2023-03-03T09:15:52.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.531+0000 7ff70cecd700 1 -- 172.21.15.7:0/2021603265 learned_addr learned my addr 172.21.15.7:0/2021603265 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:15:52.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.531+0000 7ff707fff700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7080c1da0 0x7ff708146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:52.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff70d6ce700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c3e30 0x7ff708147500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:52.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff70cecd700 1 -- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c3e30 msgr2=0x7ff708147500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:52.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff70cecd700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c3e30 0x7ff708147500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:52.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff70cecd700 1 -- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7080c1da0 msgr2=0x7ff708146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:52.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff70cecd700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7080c1da0 0x7ff708146fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:52.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff70cecd700 1 -- 172.21.15.7:0/2021603265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff700009b30 con 0x7ff70806d870 2023-03-03T09:15:52.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff707fff700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7080c1da0 0x7ff708146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:15:52.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff70d6ce700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c3e30 0x7ff708147500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:15:52.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff70cecd700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff70806d870 0x7ff708149600 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c4e0 tx=0x7ff6fc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:15:52.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff7057fa700 1 -- 172.21.15.7:0/2021603265 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc0052d0 con 0x7ff70806d870 2023-03-03T09:15:52.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff7057fa700 1 -- 172.21.15.7:0/2021603265 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc005430 con 0x7ff70806d870 2023-03-03T09:15:52.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff70f131700 1 -- 172.21.15.7:0/2021603265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff708154ba0 con 0x7ff70806d870 2023-03-03T09:15:52.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff7057fa700 1 -- 172.21.15.7:0/2021603265 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc0038c0 con 0x7ff70806d870 2023-03-03T09:15:52.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.532+0000 7ff70f131700 1 -- 172.21.15.7:0/2021603265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7081548c0 con 0x7ff70806d870 2023-03-03T09:15:52.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.533+0000 7ff70f131700 1 -- 172.21.15.7:0/2021603265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70806dc40 con 0x7ff70806d870 2023-03-03T09:15:52.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.534+0000 7ff7057fa700 1 -- 172.21.15.7:0/2021603265 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff6fc019030 con 0x7ff70806d870 2023-03-03T09:15:52.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.534+0000 7ff7057fa700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6ec0682f0 0x7ff6ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:15:52.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.534+0000 7ff7057fa700 1 -- 172.21.15.7:0/2021603265 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff6fc085500 con 0x7ff70806d870 2023-03-03T09:15:52.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.534+0000 7ff707fff700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6ec0682f0 0x7ff6ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:15:52.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.535+0000 7ff707fff700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6ec0682f0 0x7ff6ec06a7a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff708070400 tx=0x7ff6f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:15:52.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.538+0000 7ff7057fa700 1 -- 172.21.15.7:0/2021603265 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff6fc0518a0 con 0x7ff70806d870 2023-03-03T09:15:52.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.779+0000 7ff70f131700 1 -- 172.21.15.7:0/2021603265 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7080ac6d0 con 0x7ff70806d870 2023-03-03T09:15:52.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.780+0000 7ff7057fa700 1 -- 172.21.15.7:0/2021603265 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff7080ac6d0 con 0x7ff70806d870 2023-03-03T09:15:52.781 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:15:52.782 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:15:52.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 -- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6ec0682f0 msgr2=0x7ff6ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:52.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6ec0682f0 0x7ff6ec06a7a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff708070400 tx=0x7ff6f4008040 comp rx=0 tx=0).stop 2023-03-03T09:15:52.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 -- 172.21.15.7:0/2021603265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff70806d870 msgr2=0x7ff708149600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:15:52.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff70806d870 0x7ff708149600 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c4e0 tx=0x7ff6fc00c8a0 comp rx=0 tx=0).stop 2023-03-03T09:15:52.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 -- 172.21.15.7:0/2021603265 shutdown_connections 2023-03-03T09:15:52.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7080c3e30 0x7ff708147500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:52.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6ec0682f0 0x7ff6ec06a7a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:52.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff70806d870 0x7ff708149600 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 --2- 172.21.15.7:0/2021603265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7080c1da0 0x7ff708146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:15:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 -- 172.21.15.7:0/2021603265 >> 172.21.15.7:0/2021603265 conn(0x7ff70806c110 msgr2=0x7ff7080c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:15:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 -- 172.21.15.7:0/2021603265 shutdown_connections 2023-03-03T09:15:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:15:52.782+0000 7ff70f131700 1 -- 172.21.15.7:0/2021603265 wait complete. 2023-03-03T09:15:52.785 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:15:52.795 INFO:tasks.mds_thrash.fs.[a]:waiting for 101 secs before thrashing 2023-03-03T09:16:05.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:16:05.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.640+0000 7f1cea41c700 1 Processor -- start 2023-03-03T09:16:05.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.640+0000 7f1cea41c700 1 -- start start 2023-03-03T09:16:05.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1cea41c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce40ac220 0x7f1ce40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:05.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1cea41c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce40ac900 con 0x7f1ce4073c30 2023-03-03T09:16:05.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1cea41c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce40acb50 con 0x7f1ce4152b60 2023-03-03T09:16:05.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1cea41c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce4071e30 con 0x7f1ce40ac220 2023-03-03T09:16:05.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1ce37fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1ce4073c30 0x7f1ce4152450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49226/0 (socket says 172.21.15.7:49226) 2023-03-03T09:16:05.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1ce37fe700 1 -- 172.21.15.7:0/203129122 learned_addr learned my addr 172.21.15.7:0/203129122 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:16:05.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1ce3fff700 1 --2- 172.21.15.7:0/203129122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce40ac220 0x7f1ce40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:05.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1ce2ffd700 1 -- 172.21.15.7:0/203129122 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035943540 0 0) 0x7f1ce40ac900 con 0x7f1ce4073c30 2023-03-03T09:16:05.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1ce2ffd700 1 -- 172.21.15.7:0/203129122 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd00035b0 con 0x7f1ce4073c30 2023-03-03T09:16:05.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1ce2ffd700 1 -- 172.21.15.7:0/203129122 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097914552 0 0) 0x7f1ce40acb50 con 0x7f1ce4152b60 2023-03-03T09:16:05.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.641+0000 7f1ce2ffd700 1 -- 172.21.15.7:0/203129122 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce40ac900 con 0x7f1ce4152b60 2023-03-03T09:16:05.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.642+0000 7f1ce2ffd700 1 -- 172.21.15.7:0/203129122 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2284201741 0 0) 0x7f1cd00035b0 con 0x7f1ce4073c30 2023-03-03T09:16:05.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.642+0000 7f1ce3fff700 1 -- 172.21.15.7:0/203129122 >> v1:172.21.15.143:6789/0 conn(0x7f1ce4152b60 legacy=0x7f1ce415ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:05.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.642+0000 7f1ce3fff700 1 -- 172.21.15.7:0/203129122 >> v1:172.21.15.7:6789/0 conn(0x7f1ce4073c30 legacy=0x7f1ce4152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:05.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.642+0000 7f1ce3fff700 1 -- 172.21.15.7:0/203129122 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce41559c0 con 0x7f1ce40ac220 2023-03-03T09:16:05.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.642+0000 7f1ce3fff700 1 --2- 172.21.15.7:0/203129122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce40ac220 0x7f1ce40736f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009870 tx=0x7f1cd4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b7a3f06ff972adc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:16:05.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.642+0000 7f1ce2ffd700 1 -- 172.21.15.7:0/203129122 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc001670 con 0x7f1ce40ac220 2023-03-03T09:16:05.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.642+0000 7f1ce2ffd700 1 -- 172.21.15.7:0/203129122 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd4004030 con 0x7f1ce40ac220 2023-03-03T09:16:05.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.642+0000 7f1ce2ffd700 1 -- 172.21.15.7:0/203129122 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd40043c0 con 0x7f1ce40ac220 2023-03-03T09:16:05.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.643+0000 7f1cea41c700 1 -- 172.21.15.7:0/203129122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce40ac220 msgr2=0x7f1ce40736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:05.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.643+0000 7f1cea41c700 1 --2- 172.21.15.7:0/203129122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce40ac220 0x7f1ce40736f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009870 tx=0x7f1cd4009b80 comp rx=0 tx=0).stop 2023-03-03T09:16:05.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.643+0000 7f1cea41c700 1 -- 172.21.15.7:0/203129122 shutdown_connections 2023-03-03T09:16:05.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.643+0000 7f1cea41c700 1 --2- 172.21.15.7:0/203129122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce40ac220 0x7f1ce40736f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:05.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.643+0000 7f1cea41c700 1 -- 172.21.15.7:0/203129122 >> 172.21.15.7:0/203129122 conn(0x7f1ce40a94d0 msgr2=0x7f1ce40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:16:05.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.643+0000 7f1cea41c700 1 -- 172.21.15.7:0/203129122 shutdown_connections 2023-03-03T09:16:05.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.643+0000 7f1cea41c700 1 -- 172.21.15.7:0/203129122 wait complete. 2023-03-03T09:16:05.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.644+0000 7f1cea41c700 1 Processor -- start 2023-03-03T09:16:05.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.644+0000 7f1cea41c700 1 -- start start 2023-03-03T09:16:05.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1cea41c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce4073c30 0x7f1ce40ba490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:05.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1cea41c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce40ac220 0x7f1ce4151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:05.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1cea41c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce4152b60 0x7f1ce414ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:05.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1cea41c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce41ed3d0 con 0x7f1ce40ac220 2023-03-03T09:16:05.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1cea41c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce41ed670 con 0x7f1ce4073c30 2023-03-03T09:16:05.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1cea41c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce41ed950 con 0x7f1ce4152b60 2023-03-03T09:16:05.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce40ac220 0x7f1ce4151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:05.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce40ac220 0x7f1ce4151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55252/0 (socket says 172.21.15.7:55252) 2023-03-03T09:16:05.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce37fe700 1 -- 172.21.15.7:0/515235877 learned_addr learned my addr 172.21.15.7:0/515235877 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:16:05.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce89b9700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce4152b60 0x7f1ce414ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:05.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce3fff700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce4073c30 0x7f1ce40ba490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:05.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce37fe700 1 -- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce4152b60 msgr2=0x7f1ce414ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:05.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce37fe700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce4152b60 0x7f1ce414ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:05.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce37fe700 1 -- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce4073c30 msgr2=0x7f1ce40ba490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:05.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce37fe700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce4073c30 0x7f1ce40ba490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:05.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce37fe700 1 -- 172.21.15.7:0/515235877 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40094a0 con 0x7f1ce40ac220 2023-03-03T09:16:05.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.645+0000 7f1ce3fff700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce4073c30 0x7f1ce40ba490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:16:05.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.646+0000 7f1ce37fe700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce40ac220 0x7f1ce4151220 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00c6b0 tx=0x7f1ccc00ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:16:05.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.646+0000 7f1ce0ff9700 1 -- 172.21.15.7:0/515235877 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc004c80 con 0x7f1ce40ac220 2023-03-03T09:16:05.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.646+0000 7f1ce0ff9700 1 -- 172.21.15.7:0/515235877 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc004de0 con 0x7f1ce40ac220 2023-03-03T09:16:05.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.646+0000 7f1cea41c700 1 -- 172.21.15.7:0/515235877 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce4155f80 con 0x7f1ce40ac220 2023-03-03T09:16:05.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.646+0000 7f1ce0ff9700 1 -- 172.21.15.7:0/515235877 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc007310 con 0x7f1ce40ac220 2023-03-03T09:16:05.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.646+0000 7f1cea41c700 1 -- 172.21.15.7:0/515235877 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce4155ca0 con 0x7f1ce40ac220 2023-03-03T09:16:05.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.647+0000 7f1cea41c700 1 -- 172.21.15.7:0/515235877 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce4156260 con 0x7f1ce40ac220 2023-03-03T09:16:05.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.647+0000 7f1ce0ff9700 1 -- 172.21.15.7:0/515235877 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1ccc01a020 con 0x7f1ce40ac220 2023-03-03T09:16:05.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.648+0000 7f1ce0ff9700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc40682f0 0x7f1cc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:05.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.648+0000 7f1ce0ff9700 1 -- 172.21.15.7:0/515235877 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1ccc085e60 con 0x7f1ce40ac220 2023-03-03T09:16:05.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.648+0000 7f1ce3fff700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc40682f0 0x7f1cc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:05.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.648+0000 7f1ce3fff700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc40682f0 0x7f1cc406a7a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009bc0 tx=0x7f1cd4013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:16:05.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.652+0000 7f1ce0ff9700 1 -- 172.21.15.7:0/515235877 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1ccc0521a0 con 0x7f1ce40ac220 2023-03-03T09:16:05.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.888+0000 7f1cea41c700 1 -- 172.21.15.7:0/515235877 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ce41ee150 con 0x7f1ce40ac220 2023-03-03T09:16:05.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.889+0000 7f1ce0ff9700 1 -- 172.21.15.7:0/515235877 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1ce41ee150 con 0x7f1ce40ac220 2023-03-03T09:16:05.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:16:05.892 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:16:05.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 -- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc40682f0 msgr2=0x7f1cc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:05.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc40682f0 0x7f1cc406a7a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009bc0 tx=0x7f1cd4013040 comp rx=0 tx=0).stop 2023-03-03T09:16:05.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 -- 172.21.15.7:0/515235877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce40ac220 msgr2=0x7f1ce4151220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:05.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce40ac220 0x7f1ce4151220 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00c6b0 tx=0x7f1ccc00ca70 comp rx=0 tx=0).stop 2023-03-03T09:16:05.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 -- 172.21.15.7:0/515235877 shutdown_connections 2023-03-03T09:16:05.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce4152b60 0x7f1ce414ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:05.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce4073c30 0x7f1ce40ba490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:05.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc40682f0 0x7f1cc406a7a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:05.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 --2- 172.21.15.7:0/515235877 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1ce40ac220 0x7f1ce4151220 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:05.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 -- 172.21.15.7:0/515235877 >> 172.21.15.7:0/515235877 conn(0x7f1ce40a94d0 msgr2=0x7f1ce40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:16:05.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 -- 172.21.15.7:0/515235877 shutdown_connections 2023-03-03T09:16:05.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:05.891+0000 7f1cea41c700 1 -- 172.21.15.7:0/515235877 wait complete. 2023-03-03T09:16:05.895 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:16:05.905 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before thrashing 2023-03-03T09:16:08.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:16:09.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.044+0000 7f2907e8a700 1 Processor -- start 2023-03-03T09:16:09.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.044+0000 7f2907e8a700 1 -- start start 2023-03-03T09:16:09.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2907e8a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2900151ab0 0x7f29000736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:09.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2907e8a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2900073c30 0x7f290006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:09.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2907e8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290006c3e0 0x7f2900157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:09.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2907e8a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290006d780 con 0x7f2900151ab0 2023-03-03T09:16:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2907e8a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290006cb70 con 0x7f2900073c30 2023-03-03T09:16:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2907e8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290006ce50 con 0x7f290006c3e0 2023-03-03T09:16:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2905c26700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2900151ab0 0x7f29000736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2905c26700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2900151ab0 0x7f29000736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55258/0 (socket says 172.21.15.7:55258) 2023-03-03T09:16:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2905c26700 1 -- 172.21.15.7:0/1481050127 learned_addr learned my addr 172.21.15.7:0/1481050127 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:16:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2906427700 1 --2- 172.21.15.7:0/1481050127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290006c3e0 0x7f2900157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:09.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.045+0000 7f2905425700 1 --2- 172.21.15.7:0/1481050127 >> v2:172.21.15.143:3300/0 conn(0x7f2900073c30 0x7f290006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:09.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2905c26700 1 -- 172.21.15.7:0/1481050127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290006c3e0 msgr2=0x7f2900157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:16:09.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2905c26700 1 --2- 172.21.15.7:0/1481050127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290006c3e0 0x7f2900157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2905c26700 1 -- 172.21.15.7:0/1481050127 >> v2:172.21.15.143:3300/0 conn(0x7f2900073c30 msgr2=0x7f290006bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:16:09.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2905c26700 1 --2- 172.21.15.7:0/1481050127 >> v2:172.21.15.143:3300/0 conn(0x7f2900073c30 0x7f290006bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2905c26700 1 -- 172.21.15.7:0/1481050127 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290006d130 con 0x7f2900151ab0 2023-03-03T09:16:09.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2905c26700 1 --2- 172.21.15.7:0/1481050127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900151ab0 0x7f29000736f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f29000701f0 tx=0x7f28f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbbe3e96df08ce8a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:16:09.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2904c24700 1 -- 172.21.15.7:0/1481050127 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0011830 con 0x7f2900151ab0 2023-03-03T09:16:09.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2904c24700 1 -- 172.21.15.7:0/1481050127 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f0011990 con 0x7f2900151ab0 2023-03-03T09:16:09.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2904c24700 1 -- 172.21.15.7:0/1481050127 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0011d20 con 0x7f2900151ab0 2023-03-03T09:16:09.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2907e8a700 1 -- 172.21.15.7:0/1481050127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900151ab0 msgr2=0x7f29000736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:09.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.046+0000 7f2907e8a700 1 --2- 172.21.15.7:0/1481050127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900151ab0 0x7f29000736f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f29000701f0 tx=0x7f28f0009bd0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.047+0000 7f2907e8a700 1 -- 172.21.15.7:0/1481050127 shutdown_connections 2023-03-03T09:16:09.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.047+0000 7f2907e8a700 1 --2- 172.21.15.7:0/1481050127 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290006c3e0 0x7f2900157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.047+0000 7f2907e8a700 1 --2- 172.21.15.7:0/1481050127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900151ab0 0x7f29000736f0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.047+0000 7f2907e8a700 1 --2- 172.21.15.7:0/1481050127 >> v2:172.21.15.143:3300/0 conn(0x7f2900073c30 0x7f290006bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.047+0000 7f2907e8a700 1 -- 172.21.15.7:0/1481050127 >> 172.21.15.7:0/1481050127 conn(0x7f29000a94d0 msgr2=0x7f29000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:16:09.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.047+0000 7f2907e8a700 1 -- 172.21.15.7:0/1481050127 shutdown_connections 2023-03-03T09:16:09.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.047+0000 7f2907e8a700 1 -- 172.21.15.7:0/1481050127 wait complete. 2023-03-03T09:16:09.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.048+0000 7f2907e8a700 1 Processor -- start 2023-03-03T09:16:09.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.048+0000 7f2907e8a700 1 -- start start 2023-03-03T09:16:09.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2907e8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f290006c3e0 0x7f29001eabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:09.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2907e8a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900073c30 0x7f290014bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:09.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2907e8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2900151ab0 0x7f29000bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:09.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2907e8a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000beaf0 con 0x7f2900073c30 2023-03-03T09:16:09.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2907e8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000c5ea0 con 0x7f290006c3e0 2023-03-03T09:16:09.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2907e8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29001eb100 con 0x7f2900151ab0 2023-03-03T09:16:09.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905425700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900073c30 0x7f290014bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:09.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905425700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900073c30 0x7f290014bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55266/0 (socket says 172.21.15.7:55266) 2023-03-03T09:16:09.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905425700 1 -- 172.21.15.7:0/932284833 learned_addr learned my addr 172.21.15.7:0/932284833 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:16:09.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905c26700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f290006c3e0 0x7f29001eabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:09.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2906427700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2900151ab0 0x7f29000bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:09.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905425700 1 -- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2900151ab0 msgr2=0x7f29000bf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:09.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905425700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2900151ab0 0x7f29000bf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905425700 1 -- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f290006c3e0 msgr2=0x7f29001eabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:09.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905425700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f290006c3e0 0x7f29001eabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905425700 1 -- 172.21.15.7:0/932284833 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f0009550 con 0x7f2900073c30 2023-03-03T09:16:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905c26700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f290006c3e0 0x7f29001eabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:16:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2905425700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900073c30 0x7f290014bee0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00e9e0 tx=0x7f28fc00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:16:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f28f67fc700 1 -- 172.21.15.7:0/932284833 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc009710 con 0x7f2900073c30 2023-03-03T09:16:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f28f67fc700 1 -- 172.21.15.7:0/932284833 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28fc00c7f0 con 0x7f2900073c30 2023-03-03T09:16:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2907e8a700 1 -- 172.21.15.7:0/932284833 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2900151e80 con 0x7f2900073c30 2023-03-03T09:16:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f28f67fc700 1 -- 172.21.15.7:0/932284833 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc00cb60 con 0x7f2900073c30 2023-03-03T09:16:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.049+0000 7f2907e8a700 1 -- 172.21.15.7:0/932284833 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290006d410 con 0x7f2900073c30 2023-03-03T09:16:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.050+0000 7f2907e8a700 1 -- 172.21.15.7:0/932284833 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2900152010 con 0x7f2900073c30 2023-03-03T09:16:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.051+0000 7f28f67fc700 1 -- 172.21.15.7:0/932284833 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f28fc019030 con 0x7f2900073c30 2023-03-03T09:16:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.051+0000 7f28f67fc700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28e40682f0 0x7f28e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:16:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.052+0000 7f28f67fc700 1 -- 172.21.15.7:0/932284833 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f28fc086120 con 0x7f2900073c30 2023-03-03T09:16:09.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.052+0000 7f2905c26700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28e40682f0 0x7f28e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:16:09.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.052+0000 7f2905c26700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28e40682f0 0x7f28e406a7a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f28f0014040 tx=0x7f28f0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:16:09.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.056+0000 7f28f67fc700 1 -- 172.21.15.7:0/932284833 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f28fc0523d0 con 0x7f2900073c30 2023-03-03T09:16:09.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.292+0000 7f2907e8a700 1 -- 172.21.15.7:0/932284833 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29000c6050 con 0x7f2900073c30 2023-03-03T09:16:09.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.292+0000 7f28f67fc700 1 -- 172.21.15.7:0/932284833 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f29000c6050 con 0x7f2900073c30 2023-03-03T09:16:09.294 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:16:09.294 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:16:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 -- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28e40682f0 msgr2=0x7f28e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28e40682f0 0x7f28e406a7a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f28f0014040 tx=0x7f28f0008030 comp rx=0 tx=0).stop 2023-03-03T09:16:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 -- 172.21.15.7:0/932284833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900073c30 msgr2=0x7f290014bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:16:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900073c30 0x7f290014bee0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00e9e0 tx=0x7f28fc00eda0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 -- 172.21.15.7:0/932284833 shutdown_connections 2023-03-03T09:16:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2900151ab0 0x7f29000bf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f290006c3e0 0x7f29001eabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28e40682f0 0x7f28e406a7a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 --2- 172.21.15.7:0/932284833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2900073c30 0x7f290014bee0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:16:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 -- 172.21.15.7:0/932284833 >> 172.21.15.7:0/932284833 conn(0x7f29000a94d0 msgr2=0x7f29000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:16:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 -- 172.21.15.7:0/932284833 shutdown_connections 2023-03-03T09:16:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:16:09.294+0000 7f2907e8a700 1 -- 172.21.15.7:0/932284833 wait complete. 2023-03-03T09:16:09.298 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:16:09.307 INFO:tasks.mds_thrash.fs.[b]:waiting for 57 secs before thrashing 2023-03-03T09:16:12.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:12.868 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:43.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:16:43.120 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:06.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:17:06.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.444+0000 7f8492124700 1 Processor -- start 2023-03-03T09:17:06.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.445+0000 7f8492124700 1 -- start start 2023-03-03T09:17:06.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.445+0000 7f8492124700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f848c151e80 0x7f848c0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:06.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.445+0000 7f8492124700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f848c0c6fe0 0x7f848c0bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:06.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.445+0000 7f8492124700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848c0c08d0 con 0x7f848c155410 2023-03-03T09:17:06.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.445+0000 7f8492124700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848c152250 con 0x7f848c0c6fe0 2023-03-03T09:17:06.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.445+0000 7f8492124700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848c0c0fb0 con 0x7f848c151e80 2023-03-03T09:17:06.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.445+0000 7f848bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f848c0c6fe0 0x7f848c0bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:06.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f848c155410 0x7f848c0bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49306/0 (socket says 172.21.15.7:49306) 2023-03-03T09:17:06.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848affd700 1 -- 172.21.15.7:0/3033214594 learned_addr learned my addr 172.21.15.7:0/3033214594 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:17:06.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848b7fe700 1 --2- 172.21.15.7:0/3033214594 >> v2:172.21.15.143:3300/0 conn(0x7f848c151e80 0x7f848c0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:06.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848bfff700 1 -- 172.21.15.7:0/3033214594 >> v2:172.21.15.143:3300/0 conn(0x7f848c151e80 msgr2=0x7f848c0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:17:06.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848bfff700 1 --2- 172.21.15.7:0/3033214594 >> v2:172.21.15.143:3300/0 conn(0x7f848c151e80 0x7f848c0c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:06.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848bfff700 1 -- 172.21.15.7:0/3033214594 >> v1:172.21.15.7:6789/0 conn(0x7f848c155410 legacy=0x7f848c0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:06.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848bfff700 1 -- 172.21.15.7:0/3033214594 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848c154930 con 0x7f848c0c6fe0 2023-03-03T09:17:06.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848a7fc700 1 -- 172.21.15.7:0/3033214594 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813105310 0 0) 0x7f848c0c08d0 con 0x7f848c155410 2023-03-03T09:17:06.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848bfff700 1 --2- 172.21.15.7:0/3033214594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 0x7f848c0bfff0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f848c0cb250 tx=0x7f848400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34aab817958df69d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:06.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848a7fc700 1 -- 172.21.15.7:0/3033214594 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848400e070 con 0x7f848c0c6fe0 2023-03-03T09:17:06.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848a7fc700 1 -- 172.21.15.7:0/3033214594 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848400bbd0 con 0x7f848c0c6fe0 2023-03-03T09:17:06.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.446+0000 7f848a7fc700 1 -- 172.21.15.7:0/3033214594 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8484012430 con 0x7f848c0c6fe0 2023-03-03T09:17:06.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.447+0000 7f8492124700 1 -- 172.21.15.7:0/3033214594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 msgr2=0x7f848c0bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:06.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.447+0000 7f8492124700 1 --2- 172.21.15.7:0/3033214594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 0x7f848c0bfff0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f848c0cb250 tx=0x7f848400b330 comp rx=0 tx=0).stop 2023-03-03T09:17:06.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.447+0000 7f8492124700 1 -- 172.21.15.7:0/3033214594 shutdown_connections 2023-03-03T09:17:06.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.447+0000 7f8492124700 1 --2- 172.21.15.7:0/3033214594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 0x7f848c0bfff0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:06.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.447+0000 7f8492124700 1 --2- 172.21.15.7:0/3033214594 >> v2:172.21.15.143:3300/0 conn(0x7f848c151e80 0x7f848c0c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:06.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.447+0000 7f8492124700 1 -- 172.21.15.7:0/3033214594 >> 172.21.15.7:0/3033214594 conn(0x7f848c071630 msgr2=0x7f848c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:17:06.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.447+0000 7f8492124700 1 -- 172.21.15.7:0/3033214594 shutdown_connections 2023-03-03T09:17:06.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.447+0000 7f8492124700 1 -- 172.21.15.7:0/3033214594 wait complete. 2023-03-03T09:17:06.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.448+0000 7f8492124700 1 Processor -- start 2023-03-03T09:17:06.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.448+0000 7f8492124700 1 -- start start 2023-03-03T09:17:06.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f8492124700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 0x7f848c14e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:06.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f8492124700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f848c151e80 0x7f848c0bf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:06.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f8492124700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f848c155410 0x7f848c1eaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:06.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f8492124700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c14f830 con 0x7f848c0c6fe0 2023-03-03T09:17:06.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f8492124700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848c0c8ad0 con 0x7f848c155410 2023-03-03T09:17:06.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f8492124700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848c1ea3c0 con 0x7f848c151e80 2023-03-03T09:17:06.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f848b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 0x7f848c14e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:06.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f848b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 0x7f848c14e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55330/0 (socket says 172.21.15.7:55330) 2023-03-03T09:17:06.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f848b7fe700 1 -- 172.21.15.7:0/2736728167 learned_addr learned my addr 172.21.15.7:0/2736728167 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:17:06.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f848bfff700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f848c155410 0x7f848c1eaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:06.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.449+0000 7f848affd700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f848c151e80 0x7f848c0bf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:06.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f848b7fe700 1 -- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f848c151e80 msgr2=0x7f848c0bf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:06.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f848b7fe700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f848c151e80 0x7f848c0bf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:06.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f848b7fe700 1 -- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f848c155410 msgr2=0x7f848c1eaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:06.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f848b7fe700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f848c155410 0x7f848c1eaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:06.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f848b7fe700 1 -- 172.21.15.7:0/2736728167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8484009a70 con 0x7f848c0c6fe0 2023-03-03T09:17:06.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f848affd700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f848c151e80 0x7f848c0bf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:17:06.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f848b7fe700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 0x7f848c14e610 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f847c00e810 tx=0x7f847c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:06.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f8483fff700 1 -- 172.21.15.7:0/2736728167 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c013610 con 0x7f848c0c6fe0 2023-03-03T09:17:06.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f8483fff700 1 -- 172.21.15.7:0/2736728167 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c013770 con 0x7f848c0c6fe0 2023-03-03T09:17:06.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f8492124700 1 -- 172.21.15.7:0/2736728167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848c154ef0 con 0x7f848c0c6fe0 2023-03-03T09:17:06.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f8483fff700 1 -- 172.21.15.7:0/2736728167 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c013b20 con 0x7f848c0c6fe0 2023-03-03T09:17:06.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.450+0000 7f8492124700 1 -- 172.21.15.7:0/2736728167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848c154c10 con 0x7f848c0c6fe0 2023-03-03T09:17:06.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.451+0000 7f8492124700 1 -- 172.21.15.7:0/2736728167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848c0c5220 con 0x7f848c0c6fe0 2023-03-03T09:17:06.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.451+0000 7f8483fff700 1 -- 172.21.15.7:0/2736728167 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f847c023070 con 0x7f848c0c6fe0 2023-03-03T09:17:06.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.452+0000 7f8483fff700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f846c0682f0 0x7f846c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:06.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.452+0000 7f8483fff700 1 -- 172.21.15.7:0/2736728167 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f847c085db0 con 0x7f848c0c6fe0 2023-03-03T09:17:06.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.452+0000 7f848affd700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f846c0682f0 0x7f846c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:06.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.453+0000 7f848affd700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f846c0682f0 0x7f846c06a7a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f848c14d630 tx=0x7f8474007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:06.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.456+0000 7f8483fff700 1 -- 172.21.15.7:0/2736728167 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f847c0520d0 con 0x7f848c0c6fe0 2023-03-03T09:17:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.692+0000 7f8492124700 1 -- 172.21.15.7:0/2736728167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f848c0c23a0 con 0x7f848c0c6fe0 2023-03-03T09:17:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.693+0000 7f8483fff700 1 -- 172.21.15.7:0/2736728167 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f848c0c23a0 con 0x7f848c0c6fe0 2023-03-03T09:17:06.695 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:17:06.696 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:17:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 -- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f846c0682f0 msgr2=0x7f846c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f846c0682f0 0x7f846c06a7a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f848c14d630 tx=0x7f8474007d30 comp rx=0 tx=0).stop 2023-03-03T09:17:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 -- 172.21.15.7:0/2736728167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 msgr2=0x7f848c14e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:06.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 0x7f848c14e610 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f847c00e810 tx=0x7f847c00eb20 comp rx=0 tx=0).stop 2023-03-03T09:17:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 -- 172.21.15.7:0/2736728167 shutdown_connections 2023-03-03T09:17:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f848c155410 0x7f848c1eaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f846c0682f0 0x7f846c06a7a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f848c0c6fe0 0x7f848c14e610 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 --2- 172.21.15.7:0/2736728167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f848c151e80 0x7f848c0bf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 -- 172.21.15.7:0/2736728167 >> 172.21.15.7:0/2736728167 conn(0x7f848c071630 msgr2=0x7f848c0c4330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:17:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 -- 172.21.15.7:0/2736728167 shutdown_connections 2023-03-03T09:17:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:06.695+0000 7f8492124700 1 -- 172.21.15.7:0/2736728167 wait complete. 2023-03-03T09:17:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:17:06.709 INFO:tasks.mds_thrash.fs.[b]:waiting for 71 secs before thrashing 2023-03-03T09:17:13.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:13.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:33.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:17:33.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.934+0000 7f82fb538700 1 Processor -- start 2023-03-03T09:17:33.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.934+0000 7f82fb538700 1 -- start start 2023-03-03T09:17:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82fb538700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82f40c2930 0x7f82f4157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82fb538700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f406c480 0x7f82f4157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82fb538700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f406d430 con 0x7f82f4151d40 2023-03-03T09:17:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82fb538700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f406c2a0 con 0x7f82f40c2930 2023-03-03T09:17:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82fb538700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f406cb00 con 0x7f82f406c480 2023-03-03T09:17:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82f8ad3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82f40c2930 0x7f82f4157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82f9ad5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f406c480 0x7f82f4157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82f8ad3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82f40c2930 0x7f82f4157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:51124/0 (socket says 172.21.15.7:51124) 2023-03-03T09:17:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82f9ad5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f406c480 0x7f82f4157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:35192/0 (socket says 172.21.15.7:35192) 2023-03-03T09:17:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82f8ad3700 1 -- 172.21.15.7:0/2811561720 learned_addr learned my addr 172.21.15.7:0/2811561720 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:17:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.935+0000 7f82f92d4700 1 --1- 172.21.15.7:0/2811561720 >> v1:172.21.15.143:6789/0 conn(0x7f82f4151d40 0x7f82f40736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:52848/0 (socket says 172.21.15.7:52848) 2023-03-03T09:17:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.936+0000 7f82f9ad5700 1 -- 172.21.15.7:0/2811561720 >> v2:172.21.15.143:3300/0 conn(0x7f82f40c2930 msgr2=0x7f82f4157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:17:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.936+0000 7f82f9ad5700 1 --2- 172.21.15.7:0/2811561720 >> v2:172.21.15.143:3300/0 conn(0x7f82f40c2930 0x7f82f4157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.936+0000 7f82f9ad5700 1 -- 172.21.15.7:0/2811561720 >> v1:172.21.15.143:6789/0 conn(0x7f82f4151d40 legacy=0x7f82f40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.936+0000 7f82f9ad5700 1 -- 172.21.15.7:0/2811561720 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f406cde0 con 0x7f82f406c480 2023-03-03T09:17:33.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.936+0000 7f82ebfff700 1 -- 172.21.15.7:0/2811561720 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230871420 0 0) 0x7f82f406d430 con 0x7f82f4151d40 2023-03-03T09:17:33.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.936+0000 7f82f8ad3700 1 --2- 172.21.15.7:0/2811561720 >> v2:172.21.15.143:3300/0 conn(0x7f82f40c2930 0x7f82f4157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:17:33.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.936+0000 7f82f9ad5700 1 --2- 172.21.15.7:0/2811561720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f406c480 0x7f82f4157fd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009d90 tx=0x7f82ec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=843afa7f673db8e3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:33.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.936+0000 7f82ebfff700 1 -- 172.21.15.7:0/2811561720 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec00e070 con 0x7f82f406c480 2023-03-03T09:17:33.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.936+0000 7f82ebfff700 1 -- 172.21.15.7:0/2811561720 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec00bb20 con 0x7f82f406c480 2023-03-03T09:17:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.936+0000 7f82ebfff700 1 -- 172.21.15.7:0/2811561720 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec012430 con 0x7f82f406c480 2023-03-03T09:17:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.937+0000 7f82fb538700 1 -- 172.21.15.7:0/2811561720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f406c480 msgr2=0x7f82f4157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.937+0000 7f82fb538700 1 --2- 172.21.15.7:0/2811561720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f406c480 0x7f82f4157fd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009d90 tx=0x7f82ec00b330 comp rx=0 tx=0).stop 2023-03-03T09:17:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.937+0000 7f82fb538700 1 -- 172.21.15.7:0/2811561720 shutdown_connections 2023-03-03T09:17:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.937+0000 7f82fb538700 1 --2- 172.21.15.7:0/2811561720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f406c480 0x7f82f4157fd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.937+0000 7f82fb538700 1 --2- 172.21.15.7:0/2811561720 >> v2:172.21.15.143:3300/0 conn(0x7f82f40c2930 0x7f82f4157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.937+0000 7f82fb538700 1 -- 172.21.15.7:0/2811561720 >> 172.21.15.7:0/2811561720 conn(0x7f82f40a94d0 msgr2=0x7f82f40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:17:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.937+0000 7f82fb538700 1 -- 172.21.15.7:0/2811561720 shutdown_connections 2023-03-03T09:17:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.937+0000 7f82fb538700 1 -- 172.21.15.7:0/2811561720 wait complete. 2023-03-03T09:17:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.938+0000 7f82fb538700 1 Processor -- start 2023-03-03T09:17:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.938+0000 7f82fb538700 1 -- start start 2023-03-03T09:17:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82fb538700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f406c480 0x7f82f40c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82fb538700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f40c2930 0x7f82f40aca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82fb538700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f4151d40 0x7f82f41ed590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82fb538700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40bf960 con 0x7f82f4151d40 2023-03-03T09:17:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82fb538700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41ed100 con 0x7f82f406c480 2023-03-03T09:17:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82fb538700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40bebc0 con 0x7f82f40c2930 2023-03-03T09:17:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f9ad5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f4151d40 0x7f82f41ed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f9ad5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f4151d40 0x7f82f41ed590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55358/0 (socket says 172.21.15.7:55358) 2023-03-03T09:17:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f9ad5700 1 -- 172.21.15.7:0/2818942455 learned_addr learned my addr 172.21.15.7:0/2818942455 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:17:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f92d4700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f406c480 0x7f82f40c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f8ad3700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f40c2930 0x7f82f40aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f9ad5700 1 -- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f40c2930 msgr2=0x7f82f40aca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f9ad5700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f40c2930 0x7f82f40aca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f9ad5700 1 -- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f406c480 msgr2=0x7f82f40c3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f9ad5700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f406c480 0x7f82f40c3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f9ad5700 1 -- 172.21.15.7:0/2818942455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f0009550 con 0x7f82f4151d40 2023-03-03T09:17:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82f9ad5700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f4151d40 0x7f82f41ed590 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f82f406cc40 tx=0x7f82ec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.939+0000 7f82e9ffb700 1 -- 172.21.15.7:0/2818942455 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0098f0 con 0x7f82f4151d40 2023-03-03T09:17:33.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.940+0000 7f82e9ffb700 1 -- 172.21.15.7:0/2818942455 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec020070 con 0x7f82f4151d40 2023-03-03T09:17:33.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.940+0000 7f82e9ffb700 1 -- 172.21.15.7:0/2818942455 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec0276a0 con 0x7f82f4151d40 2023-03-03T09:17:33.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.940+0000 7f82fb538700 1 -- 172.21.15.7:0/2818942455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec009a70 con 0x7f82f4151d40 2023-03-03T09:17:33.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.940+0000 7f82fb538700 1 -- 172.21.15.7:0/2818942455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f4073e00 con 0x7f82f4151d40 2023-03-03T09:17:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.941+0000 7f82fb538700 1 -- 172.21.15.7:0/2818942455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f406d0c0 con 0x7f82f4151d40 2023-03-03T09:17:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.941+0000 7f82e9ffb700 1 -- 172.21.15.7:0/2818942455 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f82ec02e070 con 0x7f82f4151d40 2023-03-03T09:17:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.942+0000 7f82e9ffb700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 0x7f82d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:17:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.942+0000 7f82e9ffb700 1 -- 172.21.15.7:0/2818942455 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f82ec022070 con 0x7f82f4151d40 2023-03-03T09:17:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.942+0000 7f82f92d4700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 0x7f82d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:17:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.942+0000 7f82f92d4700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 0x7f82d806a7a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f82e4000f60 tx=0x7f82e4007c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:17:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:33.946+0000 7f82e9ffb700 1 -- 172.21.15.7:0/2818942455 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f82ec08e080 con 0x7f82f4151d40 2023-03-03T09:17:34.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.185+0000 7f82fb538700 1 -- 172.21.15.7:0/2818942455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82f406bea0 con 0x7f82f4151d40 2023-03-03T09:17:34.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.186+0000 7f82e9ffb700 1 -- 172.21.15.7:0/2818942455 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f82ec054130 con 0x7f82f4151d40 2023-03-03T09:17:34.187 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:17:34.187 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:17:34.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 -- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 msgr2=0x7f82d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:34.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 0x7f82d806a7a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f82e4000f60 tx=0x7f82e4007c00 comp rx=0 tx=0).stop 2023-03-03T09:17:34.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 -- 172.21.15.7:0/2818942455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f4151d40 msgr2=0x7f82f41ed590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:17:34.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f4151d40 0x7f82f41ed590 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f82f406cc40 tx=0x7f82ec004ad0 comp rx=0 tx=0).stop 2023-03-03T09:17:34.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 -- 172.21.15.7:0/2818942455 shutdown_connections 2023-03-03T09:17:34.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 0x7f82d806a7a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:34.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f4151d40 0x7f82f41ed590 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:34.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f406c480 0x7f82f40c3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:34.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 --2- 172.21.15.7:0/2818942455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f40c2930 0x7f82f40aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:17:34.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 -- 172.21.15.7:0/2818942455 >> 172.21.15.7:0/2818942455 conn(0x7f82f40a94d0 msgr2=0x7f82f4072010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:17:34.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 -- 172.21.15.7:0/2818942455 shutdown_connections 2023-03-03T09:17:34.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:17:34.188+0000 7f82fb538700 1 -- 172.21.15.7:0/2818942455 wait complete. 2023-03-03T09:17:34.191 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:17:34.201 INFO:tasks.mds_thrash.fs.[a]:waiting for 56 secs before thrashing 2023-03-03T09:17:43.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:17:43.421 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:13.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:13.501 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:17.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:18:17.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.847+0000 7f6846275700 1 Processor -- start 2023-03-03T09:18:17.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.847+0000 7f6846275700 1 -- start start 2023-03-03T09:18:17.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f6846275700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f684014b0d0 0x7f6840071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:17.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f6846275700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6840071d40 0x7f6840158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:17.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f6846275700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68400a9c20 con 0x7f68400aad20 2023-03-03T09:18:17.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f6846275700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6840154f10 con 0x7f6840071d40 2023-03-03T09:18:17.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f6846275700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68400ac3c0 con 0x7f684014b0d0 2023-03-03T09:18:17.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f68400aad20 0x7f6840157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49380/0 (socket says 172.21.15.7:49380) 2023-03-03T09:18:17.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683f7fe700 1 -- 172.21.15.7:0/2637955361 learned_addr learned my addr 172.21.15.7:0/2637955361 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:18:17.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618640418 0 0) 0x7f68400a9c20 con 0x7f68400aad20 2023-03-03T09:18:17.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683ffff700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f684014b0d0 0x7f6840071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:17.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f6844812700 1 --2- 172.21.15.7:0/2637955361 >> v2:172.21.15.143:3300/0 conn(0x7f6840071d40 0x7f6840158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:17.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68280035b0 con 0x7f68400aad20 2023-03-03T09:18:17.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2048872928 0 0) 0x7f68280035b0 con 0x7f68400aad20 2023-03-03T09:18:17.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f684014b0d0 msgr2=0x7f6840071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:18:17.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683effd700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f684014b0d0 0x7f6840071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:17.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 >> v2:172.21.15.143:3300/0 conn(0x7f6840071d40 msgr2=0x7f6840158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:18:17.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683effd700 1 --2- 172.21.15.7:0/2637955361 >> v2:172.21.15.143:3300/0 conn(0x7f6840071d40 0x7f6840158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:17.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684006fc70 con 0x7f68400aad20 2023-03-03T09:18:17.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f6834001670 con 0x7f68400aad20 2023-03-03T09:18:17.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.848+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 >> v1:172.21.15.7:6789/0 conn(0x7f68400aad20 legacy=0x7f6840157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:17.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683effd700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f684014b0d0 0x7f6828003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:17.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683effd700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6828003ff0 0x7f6828004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:17.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683effd700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6828004d90 0x7f6828005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:17.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6840154f10 con 0x7f6828003ff0 2023-03-03T09:18:17.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68400ac3c0 con 0x7f6828004d90 2023-03-03T09:18:17.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682800bac0 con 0x7f684014b0d0 2023-03-03T09:18:17.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683f7fe700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6828003ff0 0x7f6828004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:17.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683ffff700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f684014b0d0 0x7f6828003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:17.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f6844812700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6828004d90 0x7f6828005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:17.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683f7fe700 1 -- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f684014b0d0 msgr2=0x7f6828003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:18:17.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683f7fe700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f684014b0d0 0x7f6828003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:17.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683f7fe700 1 -- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6828004d90 msgr2=0x7f6828005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:18:17.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683f7fe700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6828004d90 0x7f6828005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:17.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683f7fe700 1 -- 172.21.15.7:0/2637955361 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68280035b0 con 0x7f6828003ff0 2023-03-03T09:18:17.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683f7fe700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6828003ff0 0x7f6828004850 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6834001050 tx=0x7f6828005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1a4cf460febbf9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:18:17.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834007c60 con 0x7f6828003ff0 2023-03-03T09:18:17.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683400ce80 con 0x7f6828003ff0 2023-03-03T09:18:17.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.849+0000 7f683effd700 1 -- 172.21.15.7:0/2637955361 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834005ba0 con 0x7f6828003ff0 2023-03-03T09:18:17.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.850+0000 7f6846275700 1 -- 172.21.15.7:0/2637955361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6828003ff0 msgr2=0x7f6828004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:17.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.850+0000 7f6846275700 1 --2- 172.21.15.7:0/2637955361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6828003ff0 0x7f6828004850 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6834001050 tx=0x7f6828005880 comp rx=0 tx=0).stop 2023-03-03T09:18:17.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.850+0000 7f6844812700 1 -- 172.21.15.7:0/2637955361 reap_dead start 2023-03-03T09:18:17.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.850+0000 7f6846275700 1 -- 172.21.15.7:0/2637955361 shutdown_connections 2023-03-03T09:18:17.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.850+0000 7f6846275700 1 -- 172.21.15.7:0/2637955361 >> 172.21.15.7:0/2637955361 conn(0x7f68400c8900 msgr2=0x7f68400c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:18:17.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.850+0000 7f6846275700 1 -- 172.21.15.7:0/2637955361 shutdown_connections 2023-03-03T09:18:17.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.850+0000 7f6846275700 1 -- 172.21.15.7:0/2637955361 wait complete. 2023-03-03T09:18:17.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.851+0000 7f6846275700 1 Processor -- start 2023-03-03T09:18:17.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.851+0000 7f6846275700 1 -- start start 2023-03-03T09:18:17.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.852+0000 7f6846275700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68400c75d0 0x7f6840143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:17.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.852+0000 7f6846275700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68400c79a0 0x7f6840143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:17.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.852+0000 7f6846275700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68401e1540 0x7f68401ee540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:17.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.852+0000 7f6846275700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68401f1840 con 0x7f68401e1540 2023-03-03T09:18:17.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.852+0000 7f6846275700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68401f1a90 con 0x7f68400c79a0 2023-03-03T09:18:17.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.852+0000 7f6846275700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68401f1d70 con 0x7f68400c75d0 2023-03-03T09:18:17.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.852+0000 7f6844812700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68401e1540 0x7f68401ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:17.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.852+0000 7f6844812700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68401e1540 0x7f68401ee540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55414/0 (socket says 172.21.15.7:55414) 2023-03-03T09:18:17.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.852+0000 7f6844812700 1 -- 172.21.15.7:0/1993867639 learned_addr learned my addr 172.21.15.7:0/1993867639 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:18:17.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f683ffff700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68400c75d0 0x7f6840143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f683f7fe700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68400c79a0 0x7f6840143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f6844812700 1 -- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68400c75d0 msgr2=0x7f6840143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f6844812700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68400c75d0 0x7f6840143a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f6844812700 1 -- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68400c79a0 msgr2=0x7f6840143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f6844812700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68400c79a0 0x7f6840143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f6844812700 1 -- 172.21.15.7:0/1993867639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684006f6b0 con 0x7f68401e1540 2023-03-03T09:18:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f6844812700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68401e1540 0x7f68401ee540 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f68400ac500 tx=0x7f6838011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:18:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f683cff9700 1 -- 172.21.15.7:0/1993867639 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683800e040 con 0x7f68401e1540 2023-03-03T09:18:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f683cff9700 1 -- 172.21.15.7:0/1993867639 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683800f040 con 0x7f68401e1540 2023-03-03T09:18:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f683cff9700 1 -- 172.21.15.7:0/1993867639 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683800fc40 con 0x7f68401e1540 2023-03-03T09:18:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f6846275700 1 -- 172.21.15.7:0/1993867639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6840155160 con 0x7f68401e1540 2023-03-03T09:18:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.853+0000 7f6846275700 1 -- 172.21.15.7:0/1993867639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684006f990 con 0x7f68401e1540 2023-03-03T09:18:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.854+0000 7f6846275700 1 -- 172.21.15.7:0/1993867639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68401ef250 con 0x7f68401e1540 2023-03-03T09:18:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.855+0000 7f683cff9700 1 -- 172.21.15.7:0/1993867639 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f683800fda0 con 0x7f68401e1540 2023-03-03T09:18:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.855+0000 7f683cff9700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f682c0682c0 0x7f682c06a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.855+0000 7f683cff9700 1 -- 172.21.15.7:0/1993867639 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6838086670 con 0x7f68401e1540 2023-03-03T09:18:17.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.855+0000 7f683ffff700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f682c0682c0 0x7f682c06a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:17.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.856+0000 7f683ffff700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f682c0682c0 0x7f682c06a770 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6830009b80 tx=0x7f6830009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:18:17.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:17.859+0000 7f683cff9700 1 -- 172.21.15.7:0/1993867639 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68401ef250 con 0x7f68401e1540 2023-03-03T09:18:18.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.098+0000 7f6846275700 1 -- 172.21.15.7:0/1993867639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68401ef250 con 0x7f68401e1540 2023-03-03T09:18:18.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.098+0000 7f683cff9700 1 -- 172.21.15.7:0/1993867639 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f68401ef250 con 0x7f68401e1540 2023-03-03T09:18:18.101 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:18:18.101 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:18:18.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 -- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f682c0682c0 msgr2=0x7f682c06a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:18.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f682c0682c0 0x7f682c06a770 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6830009b80 tx=0x7f6830009210 comp rx=0 tx=0).stop 2023-03-03T09:18:18.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 -- 172.21.15.7:0/1993867639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68401e1540 msgr2=0x7f68401ee540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:18.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68401e1540 0x7f68401ee540 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f68400ac500 tx=0x7f6838011ed0 comp rx=0 tx=0).stop 2023-03-03T09:18:18.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 -- 172.21.15.7:0/1993867639 shutdown_connections 2023-03-03T09:18:18.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f682c0682c0 0x7f682c06a770 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:18.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68401e1540 0x7f68401ee540 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:18.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68400c75d0 0x7f6840143a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:18.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 --2- 172.21.15.7:0/1993867639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68400c79a0 0x7f6840143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:18.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 -- 172.21.15.7:0/1993867639 >> 172.21.15.7:0/1993867639 conn(0x7f68400c8900 msgr2=0x7f6840148570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:18:18.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 -- 172.21.15.7:0/1993867639 shutdown_connections 2023-03-03T09:18:18.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:18.100+0000 7f6846275700 1 -- 172.21.15.7:0/1993867639 wait complete. 2023-03-03T09:18:18.104 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:18:18.114 INFO:tasks.mds_thrash.fs.[b]:waiting for 117 secs before thrashing 2023-03-03T09:18:30.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:18:30.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.340+0000 7f6bb69a4700 1 Processor -- start 2023-03-03T09:18:30.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.340+0000 7f6bb69a4700 1 -- start start 2023-03-03T09:18:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6bb69a4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bb0072250 0x7f6bb014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6bb69a4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bb00c1ce0 0x7f6bb014e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6bb69a4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb01510d0 con 0x7f6bb0071e80 2023-03-03T09:18:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6bb69a4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb00c3740 con 0x7f6bb0072250 2023-03-03T09:18:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6bb69a4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb0070670 con 0x7f6bb00c1ce0 2023-03-03T09:18:30.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6ba7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bb0072250 0x7f6bb014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:30.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6ba7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bb0072250 0x7f6bb014e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55424/0 (socket says 172.21.15.7:55424) 2023-03-03T09:18:30.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6ba7fff700 1 -- 172.21.15.7:0/1127076428 learned_addr learned my addr 172.21.15.7:0/1127076428 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:18:30.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6bb4f41700 1 --2- 172.21.15.7:0/1127076428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bb00c1ce0 0x7f6bb014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:30.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6ba7fff700 1 -- 172.21.15.7:0/1127076428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bb00c1ce0 msgr2=0x7f6bb014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:18:30.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6ba7fff700 1 --2- 172.21.15.7:0/1127076428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bb00c1ce0 0x7f6bb014e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:30.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6ba7fff700 1 -- 172.21.15.7:0/1127076428 >> v1:172.21.15.143:6789/0 conn(0x7f6bb0071e80 legacy=0x7f6bb0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:30.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.341+0000 7f6ba7fff700 1 -- 172.21.15.7:0/1127076428 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00c39e0 con 0x7f6bb0072250 2023-03-03T09:18:30.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6ba7fff700 1 --2- 172.21.15.7:0/1127076428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 0x7f6bb014e1f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6b980098f0 tx=0x7f6b98009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20236898e8c2e9d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:18:30.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6baf7fe700 1 -- 172.21.15.7:0/1127076428 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9800b8a0 con 0x7f6bb0072250 2023-03-03T09:18:30.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6baf7fe700 1 -- 172.21.15.7:0/1127076428 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9800ba00 con 0x7f6bb0072250 2023-03-03T09:18:30.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6baf7fe700 1 -- 172.21.15.7:0/1127076428 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9800bd90 con 0x7f6bb0072250 2023-03-03T09:18:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6bb69a4700 1 -- 172.21.15.7:0/1127076428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 msgr2=0x7f6bb014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6bb69a4700 1 --2- 172.21.15.7:0/1127076428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 0x7f6bb014e1f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6b980098f0 tx=0x7f6b98009c00 comp rx=0 tx=0).stop 2023-03-03T09:18:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6bb69a4700 1 -- 172.21.15.7:0/1127076428 shutdown_connections 2023-03-03T09:18:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6bb69a4700 1 --2- 172.21.15.7:0/1127076428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bb00c1ce0 0x7f6bb014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6bb69a4700 1 --2- 172.21.15.7:0/1127076428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 0x7f6bb014e1f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6bb69a4700 1 -- 172.21.15.7:0/1127076428 >> 172.21.15.7:0/1127076428 conn(0x7f6bb00a94d0 msgr2=0x7f6bb00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:18:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6bb69a4700 1 -- 172.21.15.7:0/1127076428 shutdown_connections 2023-03-03T09:18:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.342+0000 7f6bb69a4700 1 -- 172.21.15.7:0/1127076428 wait complete. 2023-03-03T09:18:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6bb69a4700 1 Processor -- start 2023-03-03T09:18:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6bb69a4700 1 -- start start 2023-03-03T09:18:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6bb69a4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bb0071e80 0x7f6bb01ddc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6bb69a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 0x7f6bb014f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:30.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6bb69a4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bb00c1ce0 0x7f6bb014fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:30.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6bb69a4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb00ba5f0 con 0x7f6bb0072250 2023-03-03T09:18:30.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6bb69a4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb014f070 con 0x7f6bb00c1ce0 2023-03-03T09:18:30.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6bb69a4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb0148770 con 0x7f6bb0071e80 2023-03-03T09:18:30.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6ba7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 0x7f6bb014f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:30.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6ba7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 0x7f6bb014f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55430/0 (socket says 172.21.15.7:55430) 2023-03-03T09:18:30.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.344+0000 7f6ba7fff700 1 -- 172.21.15.7:0/502724775 learned_addr learned my addr 172.21.15.7:0/502724775 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:18:30.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6bb4f41700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bb00c1ce0 0x7f6bb014fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:30.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6baffff700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bb0071e80 0x7f6bb01ddc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:30.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6ba7fff700 1 -- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bb0071e80 msgr2=0x7f6bb01ddc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:30.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6ba7fff700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bb0071e80 0x7f6bb01ddc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:30.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6ba7fff700 1 -- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bb00c1ce0 msgr2=0x7f6bb014fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:30.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6ba7fff700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bb00c1ce0 0x7f6bb014fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:30.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6ba7fff700 1 -- 172.21.15.7:0/502724775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b98009550 con 0x7f6bb0072250 2023-03-03T09:18:30.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6ba7fff700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 0x7f6bb014f500 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6b98000c00 tx=0x7f6b98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:18:30.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6bad7fa700 1 -- 172.21.15.7:0/502724775 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b98004300 con 0x7f6bb0072250 2023-03-03T09:18:30.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6bad7fa700 1 -- 172.21.15.7:0/502724775 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9801f070 con 0x7f6bb0072250 2023-03-03T09:18:30.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6bad7fa700 1 -- 172.21.15.7:0/502724775 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b980036a0 con 0x7f6bb0072250 2023-03-03T09:18:30.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6bb69a4700 1 -- 172.21.15.7:0/502724775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb0151f30 con 0x7f6bb0072250 2023-03-03T09:18:30.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.345+0000 7f6bb69a4700 1 -- 172.21.15.7:0/502724775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb0151300 con 0x7f6bb0072250 2023-03-03T09:18:30.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.346+0000 7f6bb69a4700 1 -- 172.21.15.7:0/502724775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb006c9a0 con 0x7f6bb0072250 2023-03-03T09:18:30.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.347+0000 7f6bad7fa700 1 -- 172.21.15.7:0/502724775 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6b9802a020 con 0x7f6bb0072250 2023-03-03T09:18:30.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.347+0000 7f6bad7fa700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b900682f0 0x7f6b9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:18:30.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.347+0000 7f6bad7fa700 1 -- 172.21.15.7:0/502724775 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6b98087680 con 0x7f6bb0072250 2023-03-03T09:18:30.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.347+0000 7f6baffff700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b900682f0 0x7f6b9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:18:30.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.348+0000 7f6baffff700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b900682f0 0x7f6b9006a7a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0007fd0 tx=0x7f6ba0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:18:30.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.351+0000 7f6bad7fa700 1 -- 172.21.15.7:0/502724775 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b980539a0 con 0x7f6bb0072250 2023-03-03T09:18:30.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.587+0000 7f6bb69a4700 1 -- 172.21.15.7:0/502724775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bb0074bb0 con 0x7f6bb0072250 2023-03-03T09:18:30.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.587+0000 7f6bad7fa700 1 -- 172.21.15.7:0/502724775 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6b98056fc0 con 0x7f6bb0072250 2023-03-03T09:18:30.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:18:30.589 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:18:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 -- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b900682f0 msgr2=0x7f6b9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b900682f0 0x7f6b9006a7a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0007fd0 tx=0x7f6ba0009040 comp rx=0 tx=0).stop 2023-03-03T09:18:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 -- 172.21.15.7:0/502724775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 msgr2=0x7f6bb014f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:18:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 0x7f6bb014f500 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6b98000c00 tx=0x7f6b98004750 comp rx=0 tx=0).stop 2023-03-03T09:18:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 -- 172.21.15.7:0/502724775 shutdown_connections 2023-03-03T09:18:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bb00c1ce0 0x7f6bb014fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bb0071e80 0x7f6bb01ddc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b900682f0 0x7f6b9006a7a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 --2- 172.21.15.7:0/502724775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb0072250 0x7f6bb014f500 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:18:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 -- 172.21.15.7:0/502724775 >> 172.21.15.7:0/502724775 conn(0x7f6bb00a94d0 msgr2=0x7f6bb00c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:18:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 -- 172.21.15.7:0/502724775 shutdown_connections 2023-03-03T09:18:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:18:30.589+0000 7f6bb69a4700 1 -- 172.21.15.7:0/502724775 wait complete. 2023-03-03T09:18:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:18:30.603 INFO:tasks.mds_thrash.fs.[a]:waiting for 31 secs before thrashing 2023-03-03T09:18:43.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:18:43.846 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:01.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:19:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.741+0000 7f10733f2700 1 Processor -- start 2023-03-03T09:19:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.742+0000 7f10733f2700 1 -- start start 2023-03-03T09:19:01.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.742+0000 7f10733f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c0aad20 0x7f106c157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:19:01.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.742+0000 7f10733f2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f106c0aa030 0x7f106c1581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:19:01.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.742+0000 7f10733f2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c0728b0 con 0x7f106c14b020 2023-03-03T09:19:01.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.742+0000 7f10733f2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c155070 con 0x7f106c0aa030 2023-03-03T09:19:01.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.742+0000 7f10733f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c06f5b0 con 0x7f106c0aad20 2023-03-03T09:19:01.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.742+0000 7f107118e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f106c14b020 0x7f106c0717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49430/0 (socket says 172.21.15.7:49430) 2023-03-03T09:19:01.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.742+0000 7f107118e700 1 -- 172.21.15.7:0/3820870094 learned_addr learned my addr 172.21.15.7:0/3820870094 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:19:01.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f107198f700 1 --2- 172.21.15.7:0/3820870094 >> v2:172.21.15.143:3300/0 conn(0x7f106c0aa030 0x7f106c1581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:19:01.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f107098d700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c0aad20 0x7f106c157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:19:01.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431525444 0 0) 0x7f106c0728b0 con 0x7f106c14b020 2023-03-03T09:19:01.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10580035b0 con 0x7f106c14b020 2023-03-03T09:19:01.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789549025 0 0) 0x7f10580035b0 con 0x7f106c14b020 2023-03-03T09:19:01.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c0aad20 msgr2=0x7f106c157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:19:01.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c0aad20 0x7f106c157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 >> v2:172.21.15.143:3300/0 conn(0x7f106c0aa030 msgr2=0x7f106c1581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:19:01.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 --2- 172.21.15.7:0/3820870094 >> v2:172.21.15.143:3300/0 conn(0x7f106c0aa030 0x7f106c1581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c06f890 con 0x7f106c14b020 2023-03-03T09:19:01.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f1060001670 con 0x7f106c14b020 2023-03-03T09:19:01.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 >> v1:172.21.15.7:6789/0 conn(0x7f106c14b020 legacy=0x7f106c0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:19:01.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1058003a70 0x7f1058003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:19:01.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c0aad20 0x7f1058004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:19:01.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 0x7f1058005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:19:01.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c155070 con 0x7f1058004d90 2023-03-03T09:19:01.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c06f5b0 con 0x7f1058003a70 2023-03-03T09:19:01.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105800ba90 con 0x7f106c0aad20 2023-03-03T09:19:01.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f107198f700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 0x7f1058005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:19:01.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f107118e700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1058003a70 0x7f1058003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:19:01.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.743+0000 7f107098d700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c0aad20 0x7f1058004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:19:01.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f107198f700 1 -- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c0aad20 msgr2=0x7f1058004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:19:01.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f107198f700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c0aad20 0x7f1058004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f107198f700 1 -- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1058003a70 msgr2=0x7f1058003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:19:01.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f107198f700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1058003a70 0x7f1058003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f107198f700 1 -- 172.21.15.7:0/3820870094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10580035b0 con 0x7f1058004d90 2023-03-03T09:19:01.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f107118e700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1058003a70 0x7f1058003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:19:01.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f107198f700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 0x7f1058005200 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1064009fc0 tx=0x7f1058005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2529f7b537e3e2bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:19:01.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106400ec30 con 0x7f1058004d90 2023-03-03T09:19:01.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106400f040 con 0x7f1058004d90 2023-03-03T09:19:01.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f105ffff700 1 -- 172.21.15.7:0/3820870094 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064014480 con 0x7f1058004d90 2023-03-03T09:19:01.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f10733f2700 1 -- 172.21.15.7:0/3820870094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 msgr2=0x7f1058005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:19:01.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f10733f2700 1 --2- 172.21.15.7:0/3820870094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 0x7f1058005200 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1064009fc0 tx=0x7f1058005880 comp rx=0 tx=0).stop 2023-03-03T09:19:01.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f107198f700 1 -- 172.21.15.7:0/3820870094 reap_dead start 2023-03-03T09:19:01.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f10733f2700 1 -- 172.21.15.7:0/3820870094 shutdown_connections 2023-03-03T09:19:01.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.744+0000 7f10733f2700 1 -- 172.21.15.7:0/3820870094 >> 172.21.15.7:0/3820870094 conn(0x7f106c0c8900 msgr2=0x7f106c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:19:01.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.745+0000 7f10733f2700 1 -- 172.21.15.7:0/3820870094 shutdown_connections 2023-03-03T09:19:01.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.745+0000 7f10733f2700 1 -- 172.21.15.7:0/3820870094 wait complete. 2023-03-03T09:19:01.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.746+0000 7f10733f2700 1 Processor -- start 2023-03-03T09:19:01.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.746+0000 7f10733f2700 1 -- start start 2023-03-03T09:19:01.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.746+0000 7f10733f2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 0x7f106c14db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:19:01.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.746+0000 7f10733f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c14e0a0 0x7f106c148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:19:01.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.746+0000 7f10733f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c148ea0 0x7f106c1ea2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:19:01.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.746+0000 7f10733f2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1eb380 con 0x7f1058004d90 2023-03-03T09:19:01.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.746+0000 7f10733f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1ede90 con 0x7f106c14e0a0 2023-03-03T09:19:01.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.746+0000 7f10733f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1ee3c0 con 0x7f106c148ea0 2023-03-03T09:19:01.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f107118e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 0x7f106c14db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:19:01.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f107118e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 0x7f106c14db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55462/0 (socket says 172.21.15.7:55462) 2023-03-03T09:19:01.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f107118e700 1 -- 172.21.15.7:0/1917388546 learned_addr learned my addr 172.21.15.7:0/1917388546 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:19:01.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f107098d700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c14e0a0 0x7f106c148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:19:01.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f107118e700 1 -- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c148ea0 msgr2=0x7f106c1ea2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:19:01.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f107118e700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c148ea0 0x7f106c1ea2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f107118e700 1 -- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c14e0a0 msgr2=0x7f106c148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:19:01.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f107118e700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c14e0a0 0x7f106c148960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f107118e700 1 -- 172.21.15.7:0/1917388546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c155350 con 0x7f1058004d90 2023-03-03T09:19:01.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f107118e700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 0x7f106c14db60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1060000c00 tx=0x7f10600076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:19:01.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f105dffb700 1 -- 172.21.15.7:0/1917388546 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106000f910 con 0x7f1058004d90 2023-03-03T09:19:01.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f105dffb700 1 -- 172.21.15.7:0/1917388546 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106000fa70 con 0x7f1058004d90 2023-03-03T09:19:01.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f10733f2700 1 -- 172.21.15.7:0/1917388546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c06fb70 con 0x7f1058004d90 2023-03-03T09:19:01.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f105dffb700 1 -- 172.21.15.7:0/1917388546 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106000fe00 con 0x7f1058004d90 2023-03-03T09:19:01.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.747+0000 7f10733f2700 1 -- 172.21.15.7:0/1917388546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c071ef0 con 0x7f1058004d90 2023-03-03T09:19:01.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.748+0000 7f10733f2700 1 -- 172.21.15.7:0/1917388546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c074bb0 con 0x7f1058004d90 2023-03-03T09:19:01.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.749+0000 7f105dffb700 1 -- 172.21.15.7:0/1917388546 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f106001d460 con 0x7f1058004d90 2023-03-03T09:19:01.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.749+0000 7f105dffb700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10500682f0 0x7f105006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:19:01.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.749+0000 7f105dffb700 1 -- 172.21.15.7:0/1917388546 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1060086450 con 0x7f1058004d90 2023-03-03T09:19:01.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.749+0000 7f107098d700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10500682f0 0x7f105006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:19:01.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.750+0000 7f107098d700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10500682f0 0x7f105006a7a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f106c0ab1a0 tx=0x7f1068008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:19:01.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.753+0000 7f105dffb700 1 -- 172.21.15.7:0/1917388546 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1060052790 con 0x7f1058004d90 2023-03-03T09:19:01.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.989+0000 7f10733f2700 1 -- 172.21.15.7:0/1917388546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f106c1e9f10 con 0x7f1058004d90 2023-03-03T09:19:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.989+0000 7f105dffb700 1 -- 172.21.15.7:0/1917388546 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1060055db0 con 0x7f1058004d90 2023-03-03T09:19:01.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:19:01.991 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:19:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 -- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10500682f0 msgr2=0x7f105006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:19:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10500682f0 0x7f105006a7a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f106c0ab1a0 tx=0x7f1068008040 comp rx=0 tx=0).stop 2023-03-03T09:19:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 -- 172.21.15.7:0/1917388546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 msgr2=0x7f106c14db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:19:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 0x7f106c14db60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1060000c00 tx=0x7f10600076a0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 -- 172.21.15.7:0/1917388546 shutdown_connections 2023-03-03T09:19:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106c148ea0 0x7f106c1ea2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10500682f0 0x7f105006a7a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1058004d90 0x7f106c14db60 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 --2- 172.21.15.7:0/1917388546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f106c14e0a0 0x7f106c148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:19:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 -- 172.21.15.7:0/1917388546 >> 172.21.15.7:0/1917388546 conn(0x7f106c0c8900 msgr2=0x7f106c0c8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:19:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 -- 172.21.15.7:0/1917388546 shutdown_connections 2023-03-03T09:19:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:19:01.991+0000 7f10733f2700 1 -- 172.21.15.7:0/1917388546 wait complete. 2023-03-03T09:19:01.995 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:19:02.004 INFO:tasks.mds_thrash.fs.[a]:waiting for 113 secs before thrashing 2023-03-03T09:19:13.907 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:13.910 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:43.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:19:43.981 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T09:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T09:20:14.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:14.044 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:15.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:20:15.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.252+0000 7f4f63fc8700 1 Processor -- start 2023-03-03T09:20:15.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.253+0000 7f4f63fc8700 1 -- start start 2023-03-03T09:20:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.253+0000 7f4f63fc8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4f5c0caa30 0x7f4f5c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.253+0000 7f4f63fc8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4f5c0c00e0 0x7f4f5c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.253+0000 7f4f63fc8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f5c0cae00 0x7f4f5c0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.253+0000 7f4f63fc8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f5c0cb280 con 0x7f4f5c0caa30 2023-03-03T09:20:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.253+0000 7f4f63fc8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f5c0bfd10 con 0x7f4f5c0c00e0 2023-03-03T09:20:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.253+0000 7f4f63fc8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c0bfa30 con 0x7f4f5c0cae00 2023-03-03T09:20:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.253+0000 7f4f61d64700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4f5c0caa30 0x7f4f5c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f61d64700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4f5c0caa30 0x7f4f5c0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55546/0 (socket says 172.21.15.7:55546) 2023-03-03T09:20:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f61d64700 1 -- 172.21.15.7:0/3354313923 learned_addr learned my addr 172.21.15.7:0/3354313923 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:20:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f62565700 1 --2- 172.21.15.7:0/3354313923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f5c0cae00 0x7f4f5c0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f61563700 1 --2- 172.21.15.7:0/3354313923 >> v2:172.21.15.143:3300/0 conn(0x7f4f5c0c00e0 0x7f4f5c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f61d64700 1 -- 172.21.15.7:0/3354313923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f5c0cae00 msgr2=0x7f4f5c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:20:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f61d64700 1 --2- 172.21.15.7:0/3354313923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f5c0cae00 0x7f4f5c0c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f61d64700 1 -- 172.21.15.7:0/3354313923 >> v2:172.21.15.143:3300/0 conn(0x7f4f5c0c00e0 msgr2=0x7f4f5c154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:20:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f61d64700 1 --2- 172.21.15.7:0/3354313923 >> v2:172.21.15.143:3300/0 conn(0x7f4f5c0c00e0 0x7f4f5c154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f61d64700 1 -- 172.21.15.7:0/3354313923 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c154510 con 0x7f4f5c0caa30 2023-03-03T09:20:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f61d64700 1 --2- 172.21.15.7:0/3354313923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0caa30 0x7f4f5c0c5c60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0ca910 tx=0x7f4f58009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f003ae264331d2f6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f60d62700 1 -- 172.21.15.7:0/3354313923 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5800b870 con 0x7f4f5c0caa30 2023-03-03T09:20:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f60d62700 1 -- 172.21.15.7:0/3354313923 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5800b9d0 con 0x7f4f5c0caa30 2023-03-03T09:20:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.254+0000 7f4f60d62700 1 -- 172.21.15.7:0/3354313923 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5800bd60 con 0x7f4f5c0caa30 2023-03-03T09:20:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.255+0000 7f4f63fc8700 1 -- 172.21.15.7:0/3354313923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0caa30 msgr2=0x7f4f5c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.255+0000 7f4f63fc8700 1 --2- 172.21.15.7:0/3354313923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0caa30 0x7f4f5c0c5c60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0ca910 tx=0x7f4f58009bd0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.255+0000 7f4f63fc8700 1 -- 172.21.15.7:0/3354313923 shutdown_connections 2023-03-03T09:20:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.255+0000 7f4f63fc8700 1 --2- 172.21.15.7:0/3354313923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f5c0cae00 0x7f4f5c0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.255+0000 7f4f63fc8700 1 --2- 172.21.15.7:0/3354313923 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0caa30 0x7f4f5c0c5c60 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.255+0000 7f4f63fc8700 1 --2- 172.21.15.7:0/3354313923 >> v2:172.21.15.143:3300/0 conn(0x7f4f5c0c00e0 0x7f4f5c154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.255+0000 7f4f63fc8700 1 -- 172.21.15.7:0/3354313923 >> 172.21.15.7:0/3354313923 conn(0x7f4f5c066e40 msgr2=0x7f4f5c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:20:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.255+0000 7f4f63fc8700 1 -- 172.21.15.7:0/3354313923 shutdown_connections 2023-03-03T09:20:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.255+0000 7f4f63fc8700 1 -- 172.21.15.7:0/3354313923 wait complete. 2023-03-03T09:20:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.256+0000 7f4f63fc8700 1 Processor -- start 2023-03-03T09:20:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.256+0000 7f4f63fc8700 1 -- start start 2023-03-03T09:20:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f63fc8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f5c0c00e0 0x7f4f5c1ed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f63fc8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f5c0caa30 0x7f4f5c14d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f63fc8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0cae00 0x7f4f5c14d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f63fc8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c1edf30 con 0x7f4f5c0cae00 2023-03-03T09:20:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f63fc8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c0c2a30 con 0x7f4f5c0c00e0 2023-03-03T09:20:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f63fc8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c1edc50 con 0x7f4f5c0caa30 2023-03-03T09:20:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f62565700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0cae00 0x7f4f5c14d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f62565700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0cae00 0x7f4f5c14d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55556/0 (socket says 172.21.15.7:55556) 2023-03-03T09:20:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f62565700 1 -- 172.21.15.7:0/2620020966 learned_addr learned my addr 172.21.15.7:0/2620020966 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:20:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f61d64700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f5c0c00e0 0x7f4f5c1ed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f61563700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f5c0caa30 0x7f4f5c14d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f62565700 1 -- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f5c0caa30 msgr2=0x7f4f5c14d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f62565700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f5c0caa30 0x7f4f5c14d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f62565700 1 -- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f5c0c00e0 msgr2=0x7f4f5c1ed3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f62565700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f5c0c00e0 0x7f4f5c1ed3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f62565700 1 -- 172.21.15.7:0/2620020966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f58009550 con 0x7f4f5c0cae00 2023-03-03T09:20:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.257+0000 7f4f62565700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0cae00 0x7f4f5c14d9e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4f54007e40 tx=0x7f4f5400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.258+0000 7f4f4e7fc700 1 -- 172.21.15.7:0/2620020966 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5400ecd0 con 0x7f4f5c0cae00 2023-03-03T09:20:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.258+0000 7f4f4e7fc700 1 -- 172.21.15.7:0/2620020966 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5400ee50 con 0x7f4f5c0cae00 2023-03-03T09:20:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.258+0000 7f4f63fc8700 1 -- 172.21.15.7:0/2620020966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5c154ad0 con 0x7f4f5c0cae00 2023-03-03T09:20:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.258+0000 7f4f4e7fc700 1 -- 172.21.15.7:0/2620020966 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54014660 con 0x7f4f5c0cae00 2023-03-03T09:20:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.258+0000 7f4f63fc8700 1 -- 172.21.15.7:0/2620020966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5c1547f0 con 0x7f4f5c0cae00 2023-03-03T09:20:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.259+0000 7f4f63fc8700 1 -- 172.21.15.7:0/2620020966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5c0bf650 con 0x7f4f5c0cae00 2023-03-03T09:20:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.259+0000 7f4f4e7fc700 1 -- 172.21.15.7:0/2620020966 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4f54012050 con 0x7f4f5c0cae00 2023-03-03T09:20:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.259+0000 7f4f4e7fc700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f400682f0 0x7f4f4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.260+0000 7f4f4e7fc700 1 -- 172.21.15.7:0/2620020966 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4f54086600 con 0x7f4f5c0cae00 2023-03-03T09:20:15.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.260+0000 7f4f61d64700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f400682f0 0x7f4f4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:15.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.260+0000 7f4f61d64700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f400682f0 0x7f4f4006a7a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4f58000c00 tx=0x7f4f58008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:15.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.264+0000 7f4f4e7fc700 1 -- 172.21.15.7:0/2620020966 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4f540529a0 con 0x7f4f5c0cae00 2023-03-03T09:20:15.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.500+0000 7f4f63fc8700 1 -- 172.21.15.7:0/2620020966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f5c0c2b80 con 0x7f4f5c0cae00 2023-03-03T09:20:15.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.500+0000 7f4f4e7fc700 1 -- 172.21.15.7:0/2620020966 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4f5c0c2b80 con 0x7f4f5c0cae00 2023-03-03T09:20:15.502 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:20:15.502 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:20:15.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 -- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f400682f0 msgr2=0x7f4f4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:15.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f400682f0 0x7f4f4006a7a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4f58000c00 tx=0x7f4f58008030 comp rx=0 tx=0).stop 2023-03-03T09:20:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 -- 172.21.15.7:0/2620020966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0cae00 msgr2=0x7f4f5c14d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0cae00 0x7f4f5c14d9e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4f54007e40 tx=0x7f4f5400e3c0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 -- 172.21.15.7:0/2620020966 shutdown_connections 2023-03-03T09:20:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f400682f0 0x7f4f4006a7a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f5c0cae00 0x7f4f5c14d9e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f5c0c00e0 0x7f4f5c1ed3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 --2- 172.21.15.7:0/2620020966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f5c0caa30 0x7f4f5c14d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 -- 172.21.15.7:0/2620020966 >> 172.21.15.7:0/2620020966 conn(0x7f4f5c066e40 msgr2=0x7f4f5c1530d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:20:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 -- 172.21.15.7:0/2620020966 shutdown_connections 2023-03-03T09:20:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:15.502+0000 7f4f63fc8700 1 -- 172.21.15.7:0/2620020966 wait complete. 2023-03-03T09:20:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:20:15.515 INFO:tasks.mds_thrash.fs.[b]:waiting for 66 secs before thrashing 2023-03-03T09:20:44.139 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:44.143 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:20:55.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:20:55.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.143+0000 7f94d53c3700 1 Processor -- start 2023-03-03T09:20:55.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.143+0000 7f94d53c3700 1 -- start start 2023-03-03T09:20:55.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94d53c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d00c00e0 0x7f94d00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:55.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94d53c3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94d0154ff0 0x7f94d00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:55.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94d53c3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d00c7310 con 0x7f94d00caa30 2023-03-03T09:20:55.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94d53c3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d00bfd10 con 0x7f94d0154ff0 2023-03-03T09:20:55.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94d53c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d00bfa30 con 0x7f94d00c00e0 2023-03-03T09:20:55.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94cf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94d0154ff0 0x7f94d00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:55.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94cf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94d0154ff0 0x7f94d00cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55608/0 (socket says 172.21.15.7:55608) 2023-03-03T09:20:55.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94cf7fe700 1 -- 172.21.15.7:0/4028908486 learned_addr learned my addr 172.21.15.7:0/4028908486 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:20:55.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94ce7fc700 1 --2- 172.21.15.7:0/4028908486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d00c00e0 0x7f94d00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:55.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94cf7fe700 1 -- 172.21.15.7:0/4028908486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d00c00e0 msgr2=0x7f94d00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:20:55.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94cf7fe700 1 --2- 172.21.15.7:0/4028908486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d00c00e0 0x7f94d00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:55.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94cf7fe700 1 -- 172.21.15.7:0/4028908486 >> v1:172.21.15.143:6789/0 conn(0x7f94d00caa30 legacy=0x7f94d00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:55.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94cf7fe700 1 -- 172.21.15.7:0/4028908486 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d0154510 con 0x7f94d0154ff0 2023-03-03T09:20:55.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94cf7fe700 1 --2- 172.21.15.7:0/4028908486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d0154ff0 0x7f94d00cae00 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009e50 tx=0x7f94c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b72bc3f65b5001 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:55.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94cdffb700 1 -- 172.21.15.7:0/4028908486 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c800e070 con 0x7f94d0154ff0 2023-03-03T09:20:55.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.144+0000 7f94cdffb700 1 -- 172.21.15.7:0/4028908486 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c800b9c0 con 0x7f94d0154ff0 2023-03-03T09:20:55.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.145+0000 7f94cdffb700 1 -- 172.21.15.7:0/4028908486 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c800bd70 con 0x7f94d0154ff0 2023-03-03T09:20:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.145+0000 7f94d53c3700 1 -- 172.21.15.7:0/4028908486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d0154ff0 msgr2=0x7f94d00cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.145+0000 7f94d53c3700 1 --2- 172.21.15.7:0/4028908486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d0154ff0 0x7f94d00cae00 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009e50 tx=0x7f94c800b330 comp rx=0 tx=0).stop 2023-03-03T09:20:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.145+0000 7f94d53c3700 1 -- 172.21.15.7:0/4028908486 shutdown_connections 2023-03-03T09:20:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.145+0000 7f94d53c3700 1 --2- 172.21.15.7:0/4028908486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d0154ff0 0x7f94d00cae00 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.145+0000 7f94d53c3700 1 --2- 172.21.15.7:0/4028908486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d00c00e0 0x7f94d00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.145+0000 7f94d53c3700 1 -- 172.21.15.7:0/4028908486 >> 172.21.15.7:0/4028908486 conn(0x7f94d0066e40 msgr2=0x7f94d0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:20:55.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.145+0000 7f94d53c3700 1 -- 172.21.15.7:0/4028908486 shutdown_connections 2023-03-03T09:20:55.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.145+0000 7f94d53c3700 1 -- 172.21.15.7:0/4028908486 wait complete. 2023-03-03T09:20:55.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.146+0000 7f94d53c3700 1 Processor -- start 2023-03-03T09:20:55.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.146+0000 7f94d53c3700 1 -- start start 2023-03-03T09:20:55.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94d53c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d00c00e0 0x7f94d014b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:55.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94d53c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00caa30 0x7f94d0157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:55.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94d53c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94d0154ff0 0x7f94d0148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:55.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94d53c3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d014a4d0 con 0x7f94d00caa30 2023-03-03T09:20:55.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94d53c3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d014a7b0 con 0x7f94d0154ff0 2023-03-03T09:20:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94d53c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d0157230 con 0x7f94d00c00e0 2023-03-03T09:20:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94ce7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00caa30 0x7f94d0157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94ce7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00caa30 0x7f94d0157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55612/0 (socket says 172.21.15.7:55612) 2023-03-03T09:20:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94ce7fc700 1 -- 172.21.15.7:0/2940111838 learned_addr learned my addr 172.21.15.7:0/2940111838 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:20:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94ceffd700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d00c00e0 0x7f94d014b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.147+0000 7f94cf7fe700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94d0154ff0 0x7f94d0148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:55.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94ce7fc700 1 -- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d00c00e0 msgr2=0x7f94d014b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:55.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94ce7fc700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d00c00e0 0x7f94d014b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:55.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94ce7fc700 1 -- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94d0154ff0 msgr2=0x7f94d0148930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:55.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94ce7fc700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94d0154ff0 0x7f94d0148930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:55.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94ce7fc700 1 -- 172.21.15.7:0/2940111838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c8009b30 con 0x7f94d00caa30 2023-03-03T09:20:55.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94ce7fc700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00caa30 0x7f94d0157d10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f94c000d7d0 tx=0x7f94c000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:55.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94b77fe700 1 -- 172.21.15.7:0/2940111838 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c0017070 con 0x7f94d00caa30 2023-03-03T09:20:55.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94b77fe700 1 -- 172.21.15.7:0/2940111838 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c0005a20 con 0x7f94d00caa30 2023-03-03T09:20:55.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94d53c3700 1 -- 172.21.15.7:0/2940111838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d0154ad0 con 0x7f94d00caa30 2023-03-03T09:20:55.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94b77fe700 1 -- 172.21.15.7:0/2940111838 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c0005bd0 con 0x7f94d00caa30 2023-03-03T09:20:55.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.148+0000 7f94d53c3700 1 -- 172.21.15.7:0/2940111838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d01547f0 con 0x7f94d00caa30 2023-03-03T09:20:55.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.149+0000 7f94d53c3700 1 -- 172.21.15.7:0/2940111838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d00bf650 con 0x7f94d00caa30 2023-03-03T09:20:55.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.149+0000 7f94b77fe700 1 -- 172.21.15.7:0/2940111838 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f94c0005d30 con 0x7f94d00caa30 2023-03-03T09:20:55.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.150+0000 7f94b77fe700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94bc0682f0 0x7f94bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:20:55.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.150+0000 7f94b77fe700 1 -- 172.21.15.7:0/2940111838 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f94c0085c90 con 0x7f94d00caa30 2023-03-03T09:20:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.150+0000 7f94ceffd700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94bc0682f0 0x7f94bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:20:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.150+0000 7f94ceffd700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94bc0682f0 0x7f94bc06a7a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f94d00af1a0 tx=0x7f94c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:20:55.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.154+0000 7f94b77fe700 1 -- 172.21.15.7:0/2940111838 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f94c0052030 con 0x7f94d00caa30 2023-03-03T09:20:55.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.391+0000 7f94d53c3700 1 -- 172.21.15.7:0/2940111838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94d0074bb0 con 0x7f94d00caa30 2023-03-03T09:20:55.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.392+0000 7f94b77fe700 1 -- 172.21.15.7:0/2940111838 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f94c0055650 con 0x7f94d00caa30 2023-03-03T09:20:55.394 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:20:55.394 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:20:55.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 -- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94bc0682f0 msgr2=0x7f94bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:55.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94bc0682f0 0x7f94bc06a7a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f94d00af1a0 tx=0x7f94c4009040 comp rx=0 tx=0).stop 2023-03-03T09:20:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 -- 172.21.15.7:0/2940111838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00caa30 msgr2=0x7f94d0157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:20:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00caa30 0x7f94d0157d10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f94c000d7d0 tx=0x7f94c000dae0 comp rx=0 tx=0).stop 2023-03-03T09:20:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 -- 172.21.15.7:0/2940111838 shutdown_connections 2023-03-03T09:20:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94d0154ff0 0x7f94d0148930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94d00c00e0 0x7f94d014b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94bc0682f0 0x7f94bc06a7a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 --2- 172.21.15.7:0/2940111838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94d00caa30 0x7f94d0157d10 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:20:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 -- 172.21.15.7:0/2940111838 >> 172.21.15.7:0/2940111838 conn(0x7f94d0066e40 msgr2=0x7f94d00c68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:20:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 -- 172.21.15.7:0/2940111838 shutdown_connections 2023-03-03T09:20:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:20:55.394+0000 7f94d53c3700 1 -- 172.21.15.7:0/2940111838 wait complete. 2023-03-03T09:20:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:20:55.407 INFO:tasks.mds_thrash.fs.[a]:waiting for 24 secs before thrashing 2023-03-03T09:21:14.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:14.340 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:19.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:21:19.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.545+0000 7f2eb3e72700 1 Processor -- start 2023-03-03T09:21:19.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.545+0000 7f2eb3e72700 1 -- start start 2023-03-03T09:21:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.546+0000 7f2eb3e72700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac151e80 0x7f2eac0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.546+0000 7f2eb3e72700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2eac155410 0x7f2eac0bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:19.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.546+0000 7f2eb3e72700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eac0c08d0 con 0x7f2eac0c6fe0 2023-03-03T09:21:19.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.546+0000 7f2eb3e72700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eac152250 con 0x7f2eac155410 2023-03-03T09:21:19.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.546+0000 7f2eb3e72700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac0c0fb0 con 0x7f2eac151e80 2023-03-03T09:21:19.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.546+0000 7f2eb240f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2eac0c6fe0 0x7f2eac0bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49620/0 (socket says 172.21.15.7:49620) 2023-03-03T09:21:19.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.546+0000 7f2eb240f700 1 -- 172.21.15.7:0/4085791316 learned_addr learned my addr 172.21.15.7:0/4085791316 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:21:19.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.546+0000 7f2eb1c0e700 1 --2- 172.21.15.7:0/4085791316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac151e80 0x7f2eac0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:19.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.546+0000 7f2eb140d700 1 --2- 172.21.15.7:0/4085791316 >> v2:172.21.15.143:3300/0 conn(0x7f2eac155410 0x7f2eac0bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:19.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb0c0c700 1 -- 172.21.15.7:0/4085791316 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328513725 0 0) 0x7f2eac0c08d0 con 0x7f2eac0c6fe0 2023-03-03T09:21:19.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb0c0c700 1 -- 172.21.15.7:0/4085791316 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e980035b0 con 0x7f2eac0c6fe0 2023-03-03T09:21:19.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb1c0e700 1 -- 172.21.15.7:0/4085791316 >> v2:172.21.15.143:3300/0 conn(0x7f2eac155410 msgr2=0x7f2eac0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:21:19.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb1c0e700 1 --2- 172.21.15.7:0/4085791316 >> v2:172.21.15.143:3300/0 conn(0x7f2eac155410 0x7f2eac0bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:19.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb1c0e700 1 -- 172.21.15.7:0/4085791316 >> v1:172.21.15.7:6789/0 conn(0x7f2eac0c6fe0 legacy=0x7f2eac0bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:19.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb1c0e700 1 -- 172.21.15.7:0/4085791316 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eac154930 con 0x7f2eac151e80 2023-03-03T09:21:19.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb140d700 1 --2- 172.21.15.7:0/4085791316 >> v2:172.21.15.143:3300/0 conn(0x7f2eac155410 0x7f2eac0bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:21:19.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb1c0e700 1 --2- 172.21.15.7:0/4085791316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac151e80 0x7f2eac0c6aa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009840 tx=0x7f2e9c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2df9b2a51098c527 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:19.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb0c0c700 1 -- 172.21.15.7:0/4085791316 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c011830 con 0x7f2eac151e80 2023-03-03T09:21:19.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb0c0c700 1 -- 172.21.15.7:0/4085791316 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9c011990 con 0x7f2eac151e80 2023-03-03T09:21:19.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.547+0000 7f2eb0c0c700 1 -- 172.21.15.7:0/4085791316 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c011d20 con 0x7f2eac151e80 2023-03-03T09:21:19.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.548+0000 7f2eb3e72700 1 -- 172.21.15.7:0/4085791316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac151e80 msgr2=0x7f2eac0c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:19.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.548+0000 7f2eb3e72700 1 --2- 172.21.15.7:0/4085791316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac151e80 0x7f2eac0c6aa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009840 tx=0x7f2e9c009b50 comp rx=0 tx=0).stop 2023-03-03T09:21:19.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.548+0000 7f2eb3e72700 1 -- 172.21.15.7:0/4085791316 shutdown_connections 2023-03-03T09:21:19.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.548+0000 7f2eb3e72700 1 --2- 172.21.15.7:0/4085791316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac151e80 0x7f2eac0c6aa0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:19.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.548+0000 7f2eb3e72700 1 --2- 172.21.15.7:0/4085791316 >> v2:172.21.15.143:3300/0 conn(0x7f2eac155410 0x7f2eac0bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:19.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.548+0000 7f2eb3e72700 1 -- 172.21.15.7:0/4085791316 >> 172.21.15.7:0/4085791316 conn(0x7f2eac071630 msgr2=0x7f2eac06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:21:19.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.548+0000 7f2eb3e72700 1 -- 172.21.15.7:0/4085791316 shutdown_connections 2023-03-03T09:21:19.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.548+0000 7f2eb3e72700 1 -- 172.21.15.7:0/4085791316 wait complete. 2023-03-03T09:21:19.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.549+0000 7f2eb3e72700 1 Processor -- start 2023-03-03T09:21:19.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.549+0000 7f2eb3e72700 1 -- start start 2023-03-03T09:21:19.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb3e72700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac0c6fe0 0x7f2eac14dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:19.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb3e72700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac151e80 0x7f2eac0c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:19.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb3e72700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eac155410 0x7f2eac14ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:19.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb3e72700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac1df130 con 0x7f2eac151e80 2023-03-03T09:21:19.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb3e72700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac14e0f0 con 0x7f2eac155410 2023-03-03T09:21:19.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb3e72700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eac1de930 con 0x7f2eac0c6fe0 2023-03-03T09:21:19.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb140d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac151e80 0x7f2eac0c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:19.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb140d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac151e80 0x7f2eac0c9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55644/0 (socket says 172.21.15.7:55644) 2023-03-03T09:21:19.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb140d700 1 -- 172.21.15.7:0/3705327415 learned_addr learned my addr 172.21.15.7:0/3705327415 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:21:19.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb1c0e700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac0c6fe0 0x7f2eac14dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:19.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb240f700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eac155410 0x7f2eac14ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:19.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb140d700 1 -- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac0c6fe0 msgr2=0x7f2eac14dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:19.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb140d700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac0c6fe0 0x7f2eac14dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:19.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb140d700 1 -- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eac155410 msgr2=0x7f2eac14ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:19.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb140d700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eac155410 0x7f2eac14ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:19.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb140d700 1 -- 172.21.15.7:0/3705327415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea80094a0 con 0x7f2eac151e80 2023-03-03T09:21:19.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.550+0000 7f2eb140d700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac151e80 0x7f2eac0c9d20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800ebe0 tx=0x7f2ea800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:19.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.551+0000 7f2ea27fc700 1 -- 172.21.15.7:0/3705327415 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea800c850 con 0x7f2eac151e80 2023-03-03T09:21:19.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.551+0000 7f2ea27fc700 1 -- 172.21.15.7:0/3705327415 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea800c9b0 con 0x7f2eac151e80 2023-03-03T09:21:19.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.551+0000 7f2eb3e72700 1 -- 172.21.15.7:0/3705327415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9c0094a0 con 0x7f2eac151e80 2023-03-03T09:21:19.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.551+0000 7f2ea27fc700 1 -- 172.21.15.7:0/3705327415 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea800cd40 con 0x7f2eac151e80 2023-03-03T09:21:19.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.551+0000 7f2eb3e72700 1 -- 172.21.15.7:0/3705327415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eac154ef0 con 0x7f2eac151e80 2023-03-03T09:21:19.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.552+0000 7f2eb3e72700 1 -- 172.21.15.7:0/3705327415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eac154c10 con 0x7f2eac151e80 2023-03-03T09:21:19.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.552+0000 7f2ea27fc700 1 -- 172.21.15.7:0/3705327415 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ea8019030 con 0x7f2eac151e80 2023-03-03T09:21:19.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.553+0000 7f2ea27fc700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e900682f0 0x7f2e9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:19.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.553+0000 7f2ea27fc700 1 -- 172.21.15.7:0/3705327415 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2ea8086050 con 0x7f2eac151e80 2023-03-03T09:21:19.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.553+0000 7f2eb1c0e700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e900682f0 0x7f2e9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:19.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.554+0000 7f2eb1c0e700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e900682f0 0x7f2e9006a7a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009fd0 tx=0x7f2e9c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:19.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.557+0000 7f2ea27fc700 1 -- 172.21.15.7:0/3705327415 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ea80523f0 con 0x7f2eac151e80 2023-03-03T09:21:19.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.794+0000 7f2eb3e72700 1 -- 172.21.15.7:0/3705327415 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2eac0c5220 con 0x7f2eac151e80 2023-03-03T09:21:19.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.795+0000 7f2ea27fc700 1 -- 172.21.15.7:0/3705327415 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2ea8017090 con 0x7f2eac151e80 2023-03-03T09:21:19.796 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:21:19.796 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:21:19.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 -- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e900682f0 msgr2=0x7f2e9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:19.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e900682f0 0x7f2e9006a7a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009fd0 tx=0x7f2e9c008030 comp rx=0 tx=0).stop 2023-03-03T09:21:19.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 -- 172.21.15.7:0/3705327415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac151e80 msgr2=0x7f2eac0c9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:19.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac151e80 0x7f2eac0c9d20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800ebe0 tx=0x7f2ea800eef0 comp rx=0 tx=0).stop 2023-03-03T09:21:19.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 -- 172.21.15.7:0/3705327415 shutdown_connections 2023-03-03T09:21:19.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eac155410 0x7f2eac14ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:19.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eac0c6fe0 0x7f2eac14dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:19.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e900682f0 0x7f2e9006a7a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:19.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 --2- 172.21.15.7:0/3705327415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eac151e80 0x7f2eac0c9d20 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:19.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 -- 172.21.15.7:0/3705327415 >> 172.21.15.7:0/3705327415 conn(0x7f2eac071630 msgr2=0x7f2eac0c3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:21:19.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 -- 172.21.15.7:0/3705327415 shutdown_connections 2023-03-03T09:21:19.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:19.797+0000 7f2eb3e72700 1 -- 172.21.15.7:0/3705327415 wait complete. 2023-03-03T09:21:19.800 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:21:19.810 INFO:tasks.mds_thrash.fs.[a]:waiting for 74 secs before thrashing 2023-03-03T09:21:21.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:21:21.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.653+0000 7f8aa5b87700 1 Processor -- start 2023-03-03T09:21:21.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.653+0000 7f8aa5b87700 1 -- start start 2023-03-03T09:21:21.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.654+0000 7f8aa5b87700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8aa0151e80 0x7f8aa00c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:21.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.654+0000 7f8aa5b87700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8aa00c6fe0 0x7f8aa00bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:21.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.654+0000 7f8aa5b87700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa00c08d0 con 0x7f8aa0155410 2023-03-03T09:21:21.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.654+0000 7f8aa5b87700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa0152250 con 0x7f8aa0151e80 2023-03-03T09:21:21.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.654+0000 7f8aa5b87700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa00c0fb0 con 0x7f8aa00c6fe0 2023-03-03T09:21:21.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8aa00c6fe0 0x7f8aa00bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:21.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9f7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8aa0151e80 0x7f8aa00c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:21.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9effd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8aa0155410 0x7f8aa00bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:53150/0 (socket says 172.21.15.7:53150) 2023-03-03T09:21:21.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9effd700 1 -- 172.21.15.7:0/4206069742 learned_addr learned my addr 172.21.15.7:0/4206069742 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:21:21.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9f7fe700 1 -- 172.21.15.7:0/4206069742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8aa00c6fe0 msgr2=0x7f8aa00bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:21:21.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9f7fe700 1 --2- 172.21.15.7:0/4206069742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8aa00c6fe0 0x7f8aa00bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:21.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9f7fe700 1 -- 172.21.15.7:0/4206069742 >> v1:172.21.15.143:6789/0 conn(0x7f8aa0155410 legacy=0x7f8aa00bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:21.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9f7fe700 1 -- 172.21.15.7:0/4206069742 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa0154930 con 0x7f8aa0151e80 2023-03-03T09:21:21.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9e7fc700 1 -- 172.21.15.7:0/4206069742 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580912009 0 0) 0x7f8aa00c08d0 con 0x7f8aa0155410 2023-03-03T09:21:21.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9f7fe700 1 --2- 172.21.15.7:0/4206069742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aa0151e80 0x7f8aa00c6aa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009840 tx=0x7f8a90009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=731cfe8d1ca57a14 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:21.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.655+0000 7f8a9e7fc700 1 -- 172.21.15.7:0/4206069742 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90011830 con 0x7f8aa0151e80 2023-03-03T09:21:21.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.656+0000 7f8a9e7fc700 1 -- 172.21.15.7:0/4206069742 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a90011990 con 0x7f8aa0151e80 2023-03-03T09:21:21.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.656+0000 7f8a9e7fc700 1 -- 172.21.15.7:0/4206069742 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90011d20 con 0x7f8aa0151e80 2023-03-03T09:21:21.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.656+0000 7f8aa5b87700 1 -- 172.21.15.7:0/4206069742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aa0151e80 msgr2=0x7f8aa00c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:21.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.656+0000 7f8aa5b87700 1 --2- 172.21.15.7:0/4206069742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aa0151e80 0x7f8aa00c6aa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009840 tx=0x7f8a90009b50 comp rx=0 tx=0).stop 2023-03-03T09:21:21.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.656+0000 7f8aa5b87700 1 -- 172.21.15.7:0/4206069742 shutdown_connections 2023-03-03T09:21:21.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.656+0000 7f8aa5b87700 1 --2- 172.21.15.7:0/4206069742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8aa00c6fe0 0x7f8aa00bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:21.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.656+0000 7f8aa5b87700 1 --2- 172.21.15.7:0/4206069742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aa0151e80 0x7f8aa00c6aa0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:21.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.656+0000 7f8aa5b87700 1 -- 172.21.15.7:0/4206069742 >> 172.21.15.7:0/4206069742 conn(0x7f8aa0071630 msgr2=0x7f8aa006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:21:21.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.656+0000 7f8aa5b87700 1 -- 172.21.15.7:0/4206069742 shutdown_connections 2023-03-03T09:21:21.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.656+0000 7f8aa5b87700 1 -- 172.21.15.7:0/4206069742 wait complete. 2023-03-03T09:21:21.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.657+0000 7f8aa5b87700 1 Processor -- start 2023-03-03T09:21:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.658+0000 7f8aa5b87700 1 -- start start 2023-03-03T09:21:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.658+0000 7f8aa5b87700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8aa00c6fe0 0x7f8aa00c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.658+0000 7f8aa5b87700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aa0151e80 0x7f8aa0146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.658+0000 7f8aa5b87700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8aa0155410 0x7f8aa01466a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.658+0000 7f8aa5b87700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa01eab10 con 0x7f8aa0155410 2023-03-03T09:21:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.658+0000 7f8aa5b87700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa01e9ed0 con 0x7f8aa0151e80 2023-03-03T09:21:21.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.658+0000 7f8aa5b87700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa01ea170 con 0x7f8aa00c6fe0 2023-03-03T09:21:21.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.658+0000 7f8a9ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8aa0155410 0x7f8aa01466a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:21.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.658+0000 7f8a9ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8aa0155410 0x7f8aa01466a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55660/0 (socket says 172.21.15.7:55660) 2023-03-03T09:21:21.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.658+0000 7f8a9ffff700 1 -- 172.21.15.7:0/525296944 learned_addr learned my addr 172.21.15.7:0/525296944 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:21:21.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a9effd700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aa0151e80 0x7f8aa0146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:21.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a9f7fe700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8aa00c6fe0 0x7f8aa00c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:21.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a9ffff700 1 -- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8aa00c6fe0 msgr2=0x7f8aa00c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:21.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a9ffff700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8aa00c6fe0 0x7f8aa00c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:21.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a9ffff700 1 -- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aa0151e80 msgr2=0x7f8aa0146160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:21.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a9ffff700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aa0151e80 0x7f8aa0146160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:21.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a9ffff700 1 -- 172.21.15.7:0/525296944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a900094a0 con 0x7f8aa0155410 2023-03-03T09:21:21.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a9ffff700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8aa0155410 0x7f8aa01466a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800efc0 tx=0x7f8a9800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:21.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a97fff700 1 -- 172.21.15.7:0/525296944 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9800cad0 con 0x7f8aa0155410 2023-03-03T09:21:21.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a97fff700 1 -- 172.21.15.7:0/525296944 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9800f040 con 0x7f8aa0155410 2023-03-03T09:21:21.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8aa5b87700 1 -- 172.21.15.7:0/525296944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa0154ef0 con 0x7f8aa0155410 2023-03-03T09:21:21.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8a97fff700 1 -- 172.21.15.7:0/525296944 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9800ce70 con 0x7f8aa0155410 2023-03-03T09:21:21.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.659+0000 7f8aa5b87700 1 -- 172.21.15.7:0/525296944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa0154c10 con 0x7f8aa0155410 2023-03-03T09:21:21.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.660+0000 7f8aa5b87700 1 -- 172.21.15.7:0/525296944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa00c5220 con 0x7f8aa0155410 2023-03-03T09:21:21.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.661+0000 7f8a97fff700 1 -- 172.21.15.7:0/525296944 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a98014480 con 0x7f8aa0155410 2023-03-03T09:21:21.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.661+0000 7f8a97fff700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a800682c0 0x7f8a8006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:21.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.661+0000 7f8a97fff700 1 -- 172.21.15.7:0/525296944 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8a98086a40 con 0x7f8aa0155410 2023-03-03T09:21:21.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.661+0000 7f8a9f7fe700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a800682c0 0x7f8a8006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:21.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.662+0000 7f8a9f7fe700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a800682c0 0x7f8a8006a770 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009fd0 tx=0x7f8a9001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:21.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.665+0000 7f8a97fff700 1 -- 172.21.15.7:0/525296944 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8a98052d80 con 0x7f8aa0155410 2023-03-03T09:21:21.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.905+0000 7f8aa5b87700 1 -- 172.21.15.7:0/525296944 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8aa01eac50 con 0x7f8aa0155410 2023-03-03T09:21:21.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.905+0000 7f8a97fff700 1 -- 172.21.15.7:0/525296944 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8aa01eac50 con 0x7f8aa0155410 2023-03-03T09:21:21.907 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:21:21.907 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:21:21.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 -- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a800682c0 msgr2=0x7f8a8006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:21.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a800682c0 0x7f8a8006a770 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009fd0 tx=0x7f8a9001c040 comp rx=0 tx=0).stop 2023-03-03T09:21:21.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 -- 172.21.15.7:0/525296944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8aa0155410 msgr2=0x7f8aa01466a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:21.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8aa0155410 0x7f8aa01466a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800efc0 tx=0x7f8a9800c0c0 comp rx=0 tx=0).stop 2023-03-03T09:21:21.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 -- 172.21.15.7:0/525296944 shutdown_connections 2023-03-03T09:21:21.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a800682c0 0x7f8a8006a770 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:21.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8aa0155410 0x7f8aa01466a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:21.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8aa00c6fe0 0x7f8aa00c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:21.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 --2- 172.21.15.7:0/525296944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8aa0151e80 0x7f8aa0146160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:21.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 -- 172.21.15.7:0/525296944 >> 172.21.15.7:0/525296944 conn(0x7f8aa0071630 msgr2=0x7f8aa00c40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:21:21.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 -- 172.21.15.7:0/525296944 shutdown_connections 2023-03-03T09:21:21.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:21.908+0000 7f8aa5b87700 1 -- 172.21.15.7:0/525296944 wait complete. 2023-03-03T09:21:21.911 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:21:21.921 INFO:tasks.mds_thrash.fs.[b]:waiting for 13 secs before thrashing 2023-03-03T09:21:34.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:21:35.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.059+0000 7fa8d96c1700 1 Processor -- start 2023-03-03T09:21:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.059+0000 7fa8d96c1700 1 -- start start 2023-03-03T09:21:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.060+0000 7fa8d96c1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa8d406a2b0 0x7fa8d406a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.060+0000 7fa8d96c1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d40c1c10 con 0x7fa8d4151c60 2023-03-03T09:21:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.060+0000 7fa8d96c1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d4152030 con 0x7fa8d4158030 2023-03-03T09:21:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.060+0000 7fa8d96c1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d40c26b0 con 0x7fa8d406a2b0 2023-03-03T09:21:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.060+0000 7fa8d2ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa8d4151c60 0x7fa8d40c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49650/0 (socket says 172.21.15.7:49650) 2023-03-03T09:21:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.060+0000 7fa8d2ffd700 1 -- 172.21.15.7:0/250805957 learned_addr learned my addr 172.21.15.7:0/250805957 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:21:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.060+0000 7fa8d27fc700 1 --2- 172.21.15.7:0/250805957 >> v2:172.21.15.143:3300/0 conn(0x7fa8d406a2b0 0x7fa8d406a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.060+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263239563 0 0) 0x7fa8c4001110 con 0x7fa8d4151c60 2023-03-03T09:21:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8bc0035b0 con 0x7fa8d4151c60 2023-03-03T09:21:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2979486894 0 0) 0x7fa8bc0035b0 con 0x7fa8d4151c60 2023-03-03T09:21:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 >> v2:172.21.15.143:3300/0 conn(0x7fa8d406a2b0 msgr2=0x7fa8d406a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:21:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 --2- 172.21.15.7:0/250805957 >> v2:172.21.15.143:3300/0 conn(0x7fa8d406a2b0 0x7fa8d406a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 >> v1:172.21.15.143:6789/0 conn(0x7fa8d4158030 legacy=0x7fa8d4158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:35.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d4157b30 con 0x7fa8d4151c60 2023-03-03T09:21:35.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa8c4001800 con 0x7fa8d4151c60 2023-03-03T09:21:35.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 >> v1:172.21.15.7:6789/0 conn(0x7fa8d4151c60 legacy=0x7fa8d40c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:35.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc003a70 0x7fa8bc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:35.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8bc004850 0x7fa8bc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:35.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc005200 0x7fa8bc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:35.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc00c0e0 con 0x7fa8bc004850 2023-03-03T09:21:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc00c250 con 0x7fa8bc003a70 2023-03-03T09:21:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc00c410 con 0x7fa8bc005200 2023-03-03T09:21:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d27fc700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8bc004850 0x7fa8bc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d2ffd700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc003a70 0x7fa8bc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d37fe700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc005200 0x7fa8bc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d27fc700 1 -- 172.21.15.7:0/250805957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc005200 msgr2=0x7fa8bc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:21:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d27fc700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc005200 0x7fa8bc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d27fc700 1 -- 172.21.15.7:0/250805957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc003a70 msgr2=0x7fa8bc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:21:35.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d27fc700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8bc003a70 0x7fa8bc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d27fc700 1 -- 172.21.15.7:0/250805957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc0035b0 con 0x7fa8bc004850 2023-03-03T09:21:35.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d37fe700 1 -- 172.21.15.7:0/250805957 reap_dead start 2023-03-03T09:21:35.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.061+0000 7fa8d27fc700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8bc004850 0x7fa8bc004cc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009b30 tx=0x7fa8bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c513fb4d316265e6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:35.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.062+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800e6c0 con 0x7fa8bc004850 2023-03-03T09:21:35.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.062+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c800e820 con 0x7fa8bc004850 2023-03-03T09:21:35.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.062+0000 7fa8d1ffb700 1 -- 172.21.15.7:0/250805957 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800eb90 con 0x7fa8bc004850 2023-03-03T09:21:35.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.062+0000 7fa8d96c1700 1 -- 172.21.15.7:0/250805957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8bc004850 msgr2=0x7fa8bc004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:35.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.062+0000 7fa8d96c1700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8bc004850 0x7fa8bc004cc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009b30 tx=0x7fa8bc005880 comp rx=0 tx=0).stop 2023-03-03T09:21:35.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.062+0000 7fa8d96c1700 1 -- 172.21.15.7:0/250805957 shutdown_connections 2023-03-03T09:21:35.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.062+0000 7fa8d96c1700 1 --2- 172.21.15.7:0/250805957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8bc004850 0x7fa8bc004cc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.062+0000 7fa8d96c1700 1 -- 172.21.15.7:0/250805957 >> 172.21.15.7:0/250805957 conn(0x7fa8d40a9430 msgr2=0x7fa8d40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:21:35.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.062+0000 7fa8d96c1700 1 -- 172.21.15.7:0/250805957 shutdown_connections 2023-03-03T09:21:35.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.062+0000 7fa8d96c1700 1 -- 172.21.15.7:0/250805957 wait complete. 2023-03-03T09:21:35.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.064+0000 7fa8d96c1700 1 Processor -- start 2023-03-03T09:21:35.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.064+0000 7fa8d96c1700 1 -- start start 2023-03-03T09:21:35.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.064+0000 7fa8d96c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc004850 0x7fa8d41ed610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:35.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.064+0000 7fa8d96c1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8d4070cf0 0x7fa8d414ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:35.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.064+0000 7fa8d96c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d4150480 0x7fa8d4153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.064+0000 7fa8d96c1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40a63c0 con 0x7fa8d4070cf0 2023-03-03T09:21:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.064+0000 7fa8d96c1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d41edb50 con 0x7fa8d4150480 2023-03-03T09:21:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.064+0000 7fa8d96c1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40a60e0 con 0x7fa8bc004850 2023-03-03T09:21:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.064+0000 7fa8d27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8d4070cf0 0x7fa8d414ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8d4070cf0 0x7fa8d414ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55684/0 (socket says 172.21.15.7:55684) 2023-03-03T09:21:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d27fc700 1 -- 172.21.15.7:0/2414951584 learned_addr learned my addr 172.21.15.7:0/2414951584 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:21:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d2ffd700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc004850 0x7fa8d41ed610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d37fe700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d4150480 0x7fa8d4153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d27fc700 1 -- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc004850 msgr2=0x7fa8d41ed610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d27fc700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc004850 0x7fa8d41ed610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d27fc700 1 -- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d4150480 msgr2=0x7fa8d4153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d27fc700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d4150480 0x7fa8d4153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d27fc700 1 -- 172.21.15.7:0/2414951584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d40c4d60 con 0x7fa8d4070cf0 2023-03-03T09:21:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d27fc700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8d4070cf0 0x7fa8d414ff40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c220 tx=0x7fa8c80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8bb7fe700 1 -- 172.21.15.7:0/2414951584 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c800e6c0 con 0x7fa8d4070cf0 2023-03-03T09:21:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8bb7fe700 1 -- 172.21.15.7:0/2414951584 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c80038a0 con 0x7fa8d4070cf0 2023-03-03T09:21:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d96c1700 1 -- 172.21.15.7:0/2414951584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d40c4a80 con 0x7fa8d4070cf0 2023-03-03T09:21:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8bb7fe700 1 -- 172.21.15.7:0/2414951584 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8003c50 con 0x7fa8d4070cf0 2023-03-03T09:21:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.065+0000 7fa8d96c1700 1 -- 172.21.15.7:0/2414951584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d40c5040 con 0x7fa8d4070cf0 2023-03-03T09:21:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.066+0000 7fa8d96c1700 1 -- 172.21.15.7:0/2414951584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d41ecc70 con 0x7fa8d4070cf0 2023-03-03T09:21:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.067+0000 7fa8bb7fe700 1 -- 172.21.15.7:0/2414951584 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa8c8019030 con 0x7fa8d4070cf0 2023-03-03T09:21:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.067+0000 7fa8bb7fe700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8c0068320 0x7fa8c006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:21:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.067+0000 7fa8bb7fe700 1 -- 172.21.15.7:0/2414951584 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa8c80876c0 con 0x7fa8d4070cf0 2023-03-03T09:21:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.067+0000 7fa8d2ffd700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8c0068320 0x7fa8c006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:21:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.068+0000 7fa8d2ffd700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8c0068320 0x7fa8c006a7d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4007fa0 tx=0x7fa8c4007f30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:21:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.072+0000 7fa8bb7fe700 1 -- 172.21.15.7:0/2414951584 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa8d41ecc70 con 0x7fa8d4070cf0 2023-03-03T09:21:35.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.310+0000 7fa8d96c1700 1 -- 172.21.15.7:0/2414951584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8d41ecc70 con 0x7fa8d4070cf0 2023-03-03T09:21:35.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.311+0000 7fa8bb7fe700 1 -- 172.21.15.7:0/2414951584 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa8d41ecc70 con 0x7fa8d4070cf0 2023-03-03T09:21:35.313 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:21:35.314 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:21:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 -- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8c0068320 msgr2=0x7fa8c006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:35.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8c0068320 0x7fa8c006a7d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4007fa0 tx=0x7fa8c4007f30 comp rx=0 tx=0).stop 2023-03-03T09:21:35.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 -- 172.21.15.7:0/2414951584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8d4070cf0 msgr2=0x7fa8d414ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:21:35.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8d4070cf0 0x7fa8d414ff40 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c220 tx=0x7fa8c80047c0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 -- 172.21.15.7:0/2414951584 shutdown_connections 2023-03-03T09:21:35.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d4150480 0x7fa8d4153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa8bc004850 0x7fa8d41ed610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8c0068320 0x7fa8c006a7d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 --2- 172.21.15.7:0/2414951584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8d4070cf0 0x7fa8d414ff40 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:21:35.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 -- 172.21.15.7:0/2414951584 >> 172.21.15.7:0/2414951584 conn(0x7fa8d40a9430 msgr2=0x7fa8d40aa670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:21:35.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 -- 172.21.15.7:0/2414951584 shutdown_connections 2023-03-03T09:21:35.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:21:35.313+0000 7fa8d96c1700 1 -- 172.21.15.7:0/2414951584 wait complete. 2023-03-03T09:21:35.318 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:21:35.327 INFO:tasks.mds_thrash.fs.[b]:waiting for 53 secs before thrashing 2023-03-03T09:21:44.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:21:44.410 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:14.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:14.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:28.329 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:22:28.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.465+0000 7fc95b673700 1 Processor -- start 2023-03-03T09:22:28.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.465+0000 7fc95b673700 1 -- start start 2023-03-03T09:22:28.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.466+0000 7fc95b673700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9540aa030 0x7fc9541583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:28.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.466+0000 7fc95b673700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9541588f0 con 0x7fc9540aad20 2023-03-03T09:22:28.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.466+0000 7fc95b673700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9540728b0 con 0x7fc95414b020 2023-03-03T09:22:28.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.466+0000 7fc95b673700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc954155070 con 0x7fc9540aa030 2023-03-03T09:22:28.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.466+0000 7fc959c10700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9540aa030 0x7fc9541583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:28.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.466+0000 7fc958c0e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc9540aad20 0x7fc954157ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49712/0 (socket says 172.21.15.7:49712) 2023-03-03T09:22:28.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.466+0000 7fc958c0e700 1 -- 172.21.15.7:0/94325095 learned_addr learned my addr 172.21.15.7:0/94325095 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:22:28.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.467+0000 7fc94bfff700 1 -- 172.21.15.7:0/94325095 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544463346 0 0) 0x7fc9541588f0 con 0x7fc9540aad20 2023-03-03T09:22:28.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.467+0000 7fc94bfff700 1 -- 172.21.15.7:0/94325095 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9400035b0 con 0x7fc9540aad20 2023-03-03T09:22:28.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.467+0000 7fc959c10700 1 -- 172.21.15.7:0/94325095 >> v1:172.21.15.143:6789/0 conn(0x7fc95414b020 legacy=0x7fc9540717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:28.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.467+0000 7fc959c10700 1 -- 172.21.15.7:0/94325095 >> v1:172.21.15.7:6789/0 conn(0x7fc9540aad20 legacy=0x7fc954157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:28.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.467+0000 7fc959c10700 1 -- 172.21.15.7:0/94325095 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95406f5b0 con 0x7fc9540aa030 2023-03-03T09:22:28.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.467+0000 7fc959c10700 1 --2- 172.21.15.7:0/94325095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9540aa030 0x7fc9541583b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc94c009d90 tx=0x7fc94c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91996e6b19dd0cd2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:28.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.467+0000 7fc94bfff700 1 -- 172.21.15.7:0/94325095 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c00e070 con 0x7fc9540aa030 2023-03-03T09:22:28.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.467+0000 7fc94bfff700 1 -- 172.21.15.7:0/94325095 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c00bb20 con 0x7fc9540aa030 2023-03-03T09:22:28.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.467+0000 7fc94bfff700 1 -- 172.21.15.7:0/94325095 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c012430 con 0x7fc9540aa030 2023-03-03T09:22:28.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.468+0000 7fc95b673700 1 -- 172.21.15.7:0/94325095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9540aa030 msgr2=0x7fc9541583b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:28.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.468+0000 7fc95b673700 1 --2- 172.21.15.7:0/94325095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9540aa030 0x7fc9541583b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc94c009d90 tx=0x7fc94c00b330 comp rx=0 tx=0).stop 2023-03-03T09:22:28.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.468+0000 7fc95b673700 1 -- 172.21.15.7:0/94325095 shutdown_connections 2023-03-03T09:22:28.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.468+0000 7fc95b673700 1 --2- 172.21.15.7:0/94325095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9540aa030 0x7fc9541583b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:28.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.468+0000 7fc95b673700 1 -- 172.21.15.7:0/94325095 >> 172.21.15.7:0/94325095 conn(0x7fc9540c8900 msgr2=0x7fc9540c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:22:28.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.468+0000 7fc95b673700 1 -- 172.21.15.7:0/94325095 shutdown_connections 2023-03-03T09:22:28.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.468+0000 7fc95b673700 1 -- 172.21.15.7:0/94325095 wait complete. 2023-03-03T09:22:28.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.469+0000 7fc95b673700 1 Processor -- start 2023-03-03T09:22:28.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.469+0000 7fc95b673700 1 -- start start 2023-03-03T09:22:28.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc95b673700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9540aa030 0x7fc9541ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:28.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc95b673700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9540aad20 0x7fc9541535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:28.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc95b673700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95414b020 0x7fc954155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:28.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc95b673700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95414dbf0 con 0x7fc95414b020 2023-03-03T09:22:28.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc95b673700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9541eae00 con 0x7fc9540aad20 2023-03-03T09:22:28.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc95b673700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc954156ec0 con 0x7fc9540aa030 2023-03-03T09:22:28.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc959c10700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95414b020 0x7fc954155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:28.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc959c10700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95414b020 0x7fc954155e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55742/0 (socket says 172.21.15.7:55742) 2023-03-03T09:22:28.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc959c10700 1 -- 172.21.15.7:0/4242746786 learned_addr learned my addr 172.21.15.7:0/4242746786 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:22:28.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc958c0e700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9540aad20 0x7fc9541535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:28.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc95940f700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9540aa030 0x7fc9541ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:28.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc959c10700 1 -- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9540aa030 msgr2=0x7fc9541ea8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:28.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc959c10700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9540aa030 0x7fc9541ea8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:28.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc959c10700 1 -- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9540aad20 msgr2=0x7fc9541535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:28.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc959c10700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9540aad20 0x7fc9541535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:28.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc959c10700 1 -- 172.21.15.7:0/4242746786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c009a70 con 0x7fc95414b020 2023-03-03T09:22:28.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc959c10700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95414b020 0x7fc954155e10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc9540c2b10 tx=0x7fc94c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:28.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc949ffb700 1 -- 172.21.15.7:0/4242746786 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c012430 con 0x7fc95414b020 2023-03-03T09:22:28.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc949ffb700 1 -- 172.21.15.7:0/4242746786 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94c020070 con 0x7fc95414b020 2023-03-03T09:22:28.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc95b673700 1 -- 172.21.15.7:0/4242746786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95406fb70 con 0x7fc95414b020 2023-03-03T09:22:28.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.470+0000 7fc949ffb700 1 -- 172.21.15.7:0/4242746786 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94c004280 con 0x7fc95414b020 2023-03-03T09:22:28.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.471+0000 7fc95b673700 1 -- 172.21.15.7:0/4242746786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95406f890 con 0x7fc95414b020 2023-03-03T09:22:28.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.472+0000 7fc95b673700 1 -- 172.21.15.7:0/4242746786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc954155350 con 0x7fc95414b020 2023-03-03T09:22:28.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.473+0000 7fc949ffb700 1 -- 172.21.15.7:0/4242746786 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc94c0043e0 con 0x7fc95414b020 2023-03-03T09:22:28.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.473+0000 7fc949ffb700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9380682f0 0x7fc93806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:28.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.473+0000 7fc949ffb700 1 -- 172.21.15.7:0/4242746786 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc94c022070 con 0x7fc95414b020 2023-03-03T09:22:28.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.473+0000 7fc95940f700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9380682f0 0x7fc93806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:28.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.474+0000 7fc95940f700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9380682f0 0x7fc93806a7a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc9540c4dd0 tx=0x7fc944009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:28.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.477+0000 7fc949ffb700 1 -- 172.21.15.7:0/4242746786 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc94c054b40 con 0x7fc95414b020 2023-03-03T09:22:28.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.718+0000 7fc95b673700 1 -- 172.21.15.7:0/4242746786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc954074bb0 con 0x7fc95414b020 2023-03-03T09:22:28.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.718+0000 7fc949ffb700 1 -- 172.21.15.7:0/4242746786 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc94c058160 con 0x7fc95414b020 2023-03-03T09:22:28.721 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:22:28.721 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:22:28.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 -- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9380682f0 msgr2=0x7fc93806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:28.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9380682f0 0x7fc93806a7a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc9540c4dd0 tx=0x7fc944009040 comp rx=0 tx=0).stop 2023-03-03T09:22:28.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 -- 172.21.15.7:0/4242746786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95414b020 msgr2=0x7fc954155e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:28.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95414b020 0x7fc954155e10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc9540c2b10 tx=0x7fc94c004ad0 comp rx=0 tx=0).stop 2023-03-03T09:22:28.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 -- 172.21.15.7:0/4242746786 shutdown_connections 2023-03-03T09:22:28.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9380682f0 0x7fc93806a7a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:28.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95414b020 0x7fc954155e10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:28.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9540aa030 0x7fc9541ea8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:28.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 --2- 172.21.15.7:0/4242746786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9540aad20 0x7fc9541535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:28.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 -- 172.21.15.7:0/4242746786 >> 172.21.15.7:0/4242746786 conn(0x7fc9540c8900 msgr2=0x7fc9540ac310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:22:28.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 -- 172.21.15.7:0/4242746786 shutdown_connections 2023-03-03T09:22:28.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:28.720+0000 7fc95b673700 1 -- 172.21.15.7:0/4242746786 wait complete. 2023-03-03T09:22:28.724 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:22:28.734 INFO:tasks.mds_thrash.fs.[b]:waiting for 12 secs before thrashing 2023-03-03T09:22:33.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:22:33.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.948+0000 7fc6ed444700 1 Processor -- start 2023-03-03T09:22:33.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.948+0000 7fc6ed444700 1 -- start start 2023-03-03T09:22:33.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.949+0000 7fc6ed444700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e80aad20 0x7fc6e8157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:33.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.949+0000 7fc6ed444700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc6e80aa030 0x7fc6e81581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:33.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.949+0000 7fc6ed444700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e80728b0 con 0x7fc6e814b020 2023-03-03T09:22:33.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.949+0000 7fc6ed444700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e8155070 con 0x7fc6e80aa030 2023-03-03T09:22:33.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.949+0000 7fc6ed444700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e806f5b0 con 0x7fc6e80aad20 2023-03-03T09:22:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.949+0000 7fc6e6ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc6e814b020 0x7fc6e80717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49734/0 (socket says 172.21.15.7:49734) 2023-03-03T09:22:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.949+0000 7fc6e6ffd700 1 -- 172.21.15.7:0/3708986904 learned_addr learned my addr 172.21.15.7:0/3708986904 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:22:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.949+0000 7fc6e67fc700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e80aad20 0x7fc6e8157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.949+0000 7fc6e77fe700 1 --2- 172.21.15.7:0/3708986904 >> v2:172.21.15.143:3300/0 conn(0x7fc6e80aa030 0x7fc6e81581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.949+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407688735 0 0) 0x7fc6e80728b0 con 0x7fc6e814b020 2023-03-03T09:22:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d00035b0 con 0x7fc6e814b020 2023-03-03T09:22:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938846471 0 0) 0x7fc6d00035b0 con 0x7fc6e814b020 2023-03-03T09:22:33.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e80aad20 msgr2=0x7fc6e8157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:33.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e80aad20 0x7fc6e8157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:33.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 >> v2:172.21.15.143:3300/0 conn(0x7fc6e80aa030 msgr2=0x7fc6e81581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:33.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 --2- 172.21.15.7:0/3708986904 >> v2:172.21.15.143:3300/0 conn(0x7fc6e80aa030 0x7fc6e81581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:33.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e806f890 con 0x7fc6e814b020 2023-03-03T09:22:33.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc6dc001670 con 0x7fc6e814b020 2023-03-03T09:22:33.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 >> v1:172.21.15.7:6789/0 conn(0x7fc6e814b020 legacy=0x7fc6e80717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:33.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e80aad20 0x7fc6d0003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:33.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 0x7fc6d0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:33.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d0004d90 0x7fc6d0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:33.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e8155070 con 0x7fc6d0003ff0 2023-03-03T09:22:33.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e806f5b0 con 0x7fc6d0004d90 2023-03-03T09:22:33.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d000bac0 con 0x7fc6e80aad20 2023-03-03T09:22:33.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e67fc700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 0x7fc6d0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:33.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e6ffd700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e80aad20 0x7fc6d0003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:33.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e77fe700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d0004d90 0x7fc6d0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:33.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e67fc700 1 -- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e80aad20 msgr2=0x7fc6d0003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:33.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e67fc700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e80aad20 0x7fc6d0003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:33.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e67fc700 1 -- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d0004d90 msgr2=0x7fc6d0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:33.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e67fc700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6d0004d90 0x7fc6d0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:33.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e67fc700 1 -- 172.21.15.7:0/3708986904 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d00035b0 con 0x7fc6d0003ff0 2023-03-03T09:22:33.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.950+0000 7fc6e67fc700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 0x7fc6d0004850 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80ab1a0 tx=0x7fc6d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28f48ae4fa1d41 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:33.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.951+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d800e6c0 con 0x7fc6d0003ff0 2023-03-03T09:22:33.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.951+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d800e820 con 0x7fc6d0003ff0 2023-03-03T09:22:33.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.951+0000 7fc6e5ffb700 1 -- 172.21.15.7:0/3708986904 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d800eb90 con 0x7fc6d0003ff0 2023-03-03T09:22:33.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.951+0000 7fc6ed444700 1 -- 172.21.15.7:0/3708986904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 msgr2=0x7fc6d0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:33.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.951+0000 7fc6ed444700 1 --2- 172.21.15.7:0/3708986904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 0x7fc6d0004850 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80ab1a0 tx=0x7fc6d0005880 comp rx=0 tx=0).stop 2023-03-03T09:22:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.951+0000 7fc6e77fe700 1 -- 172.21.15.7:0/3708986904 reap_dead start 2023-03-03T09:22:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.951+0000 7fc6ed444700 1 -- 172.21.15.7:0/3708986904 shutdown_connections 2023-03-03T09:22:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.951+0000 7fc6ed444700 1 -- 172.21.15.7:0/3708986904 >> 172.21.15.7:0/3708986904 conn(0x7fc6e80c8900 msgr2=0x7fc6e80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:22:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.951+0000 7fc6ed444700 1 -- 172.21.15.7:0/3708986904 shutdown_connections 2023-03-03T09:22:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.951+0000 7fc6ed444700 1 -- 172.21.15.7:0/3708986904 wait complete. 2023-03-03T09:22:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.953+0000 7fc6ed444700 1 Processor -- start 2023-03-03T09:22:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.953+0000 7fc6ed444700 1 -- start start 2023-03-03T09:22:33.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6ed444700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 0x7fc6e8156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6ed444700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6e81ed050 0x7fc6e8150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6ed444700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e8150c50 0x7fc6e814eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6ed444700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e814cd90 con 0x7fc6d0003ff0 2023-03-03T09:22:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6ed444700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e814e7a0 con 0x7fc6e81ed050 2023-03-03T09:22:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6ed444700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e8151b90 con 0x7fc6e8150c50 2023-03-03T09:22:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 0x7fc6e8156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:33.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 0x7fc6e8156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55766/0 (socket says 172.21.15.7:55766) 2023-03-03T09:22:33.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e6ffd700 1 -- 172.21.15.7:0/3418058423 learned_addr learned my addr 172.21.15.7:0/3418058423 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:22:33.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e67fc700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6e81ed050 0x7fc6e8150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:33.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e77fe700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e8150c50 0x7fc6e814eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:33.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e6ffd700 1 -- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e8150c50 msgr2=0x7fc6e814eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:33.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e6ffd700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e8150c50 0x7fc6e814eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:33.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e6ffd700 1 -- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6e81ed050 msgr2=0x7fc6e8150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:33.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e6ffd700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6e81ed050 0x7fc6e8150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:33.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e6ffd700 1 -- 172.21.15.7:0/3418058423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e8155350 con 0x7fc6d0003ff0 2023-03-03T09:22:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.954+0000 7fc6e77fe700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e8150c50 0x7fc6e814eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:22:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.955+0000 7fc6e6ffd700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 0x7fc6e8156d40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc007310 tx=0x7fc6dc007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.955+0000 7fc6cf7fe700 1 -- 172.21.15.7:0/3418058423 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc00f5e0 con 0x7fc6d0003ff0 2023-03-03T09:22:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.955+0000 7fc6cf7fe700 1 -- 172.21.15.7:0/3418058423 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc00f740 con 0x7fc6d0003ff0 2023-03-03T09:22:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.955+0000 7fc6ed444700 1 -- 172.21.15.7:0/3418058423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e806fb70 con 0x7fc6d0003ff0 2023-03-03T09:22:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.955+0000 7fc6cf7fe700 1 -- 172.21.15.7:0/3418058423 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc00fad0 con 0x7fc6d0003ff0 2023-03-03T09:22:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.955+0000 7fc6ed444700 1 -- 172.21.15.7:0/3418058423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e8071ef0 con 0x7fc6d0003ff0 2023-03-03T09:22:33.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.956+0000 7fc6ed444700 1 -- 172.21.15.7:0/3418058423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e8074bb0 con 0x7fc6d0003ff0 2023-03-03T09:22:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.956+0000 7fc6cf7fe700 1 -- 172.21.15.7:0/3418058423 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc6dc00fc30 con 0x7fc6d0003ff0 2023-03-03T09:22:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.957+0000 7fc6cf7fe700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6d40682f0 0x7fc6d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.957+0000 7fc6cf7fe700 1 -- 172.21.15.7:0/3418058423 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc6dc086220 con 0x7fc6d0003ff0 2023-03-03T09:22:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.957+0000 7fc6e67fc700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6d40682f0 0x7fc6d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.958+0000 7fc6e67fc700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6d40682f0 0x7fc6d406a7a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8000c00 tx=0x7fc6d801f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:33.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:33.961+0000 7fc6cf7fe700 1 -- 172.21.15.7:0/3418058423 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc6dc0525c0 con 0x7fc6d0003ff0 2023-03-03T09:22:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.199+0000 7fc6ed444700 1 -- 172.21.15.7:0/3418058423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6e80a9e10 con 0x7fc6d0003ff0 2023-03-03T09:22:34.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.200+0000 7fc6cf7fe700 1 -- 172.21.15.7:0/3418058423 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc6dc055be0 con 0x7fc6d0003ff0 2023-03-03T09:22:34.202 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:22:34.202 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:22:34.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 -- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6d40682f0 msgr2=0x7fc6d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:34.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6d40682f0 0x7fc6d406a7a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8000c00 tx=0x7fc6d801f040 comp rx=0 tx=0).stop 2023-03-03T09:22:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 -- 172.21.15.7:0/3418058423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 msgr2=0x7fc6e8156d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 0x7fc6e8156d40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc007310 tx=0x7fc6dc007620 comp rx=0 tx=0).stop 2023-03-03T09:22:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 -- 172.21.15.7:0/3418058423 shutdown_connections 2023-03-03T09:22:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc6e8150c50 0x7fc6e814eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc6d40682f0 0x7fc6d406a7a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6d0003ff0 0x7fc6e8156d40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 --2- 172.21.15.7:0/3418058423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6e81ed050 0x7fc6e8150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 -- 172.21.15.7:0/3418058423 >> 172.21.15.7:0/3418058423 conn(0x7fc6e80c8900 msgr2=0x7fc6e80accd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:22:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 -- 172.21.15.7:0/3418058423 shutdown_connections 2023-03-03T09:22:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:34.202+0000 7fc6ed444700 1 -- 172.21.15.7:0/3418058423 wait complete. 2023-03-03T09:22:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:22:34.216 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2023-03-03T09:22:40.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:22:40.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.872+0000 7f55781c9700 1 Processor -- start 2023-03-03T09:22:40.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.873+0000 7f55781c9700 1 -- start start 2023-03-03T09:22:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.873+0000 7f55781c9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f557014c610 0x7f5570152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.873+0000 7f55781c9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55700c1f80 0x7f557006b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.873+0000 7f55781c9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55700c7060 con 0x7f557006bd20 2023-03-03T09:22:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.873+0000 7f55781c9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f557014c9e0 con 0x7f55700c1f80 2023-03-03T09:22:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.873+0000 7f55781c9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5570156250 con 0x7f557014c610 2023-03-03T09:22:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.873+0000 7f5576766700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f557006bd20 0x7f55700ab5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49762/0 (socket says 172.21.15.7:49762) 2023-03-03T09:22:40.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.873+0000 7f5576766700 1 -- 172.21.15.7:0/144274913 learned_addr learned my addr 172.21.15.7:0/144274913 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:22:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5575764700 1 --2- 172.21.15.7:0/144274913 >> v2:172.21.15.143:3300/0 conn(0x7f55700c1f80 0x7f557006b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5575f65700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f557014c610 0x7f5570152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97560417 0 0) 0x7f55700c7060 con 0x7f557006bd20 2023-03-03T09:22:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f555c0035b0 con 0x7f557006bd20 2023-03-03T09:22:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2478802146 0 0) 0x7f555c0035b0 con 0x7f557006bd20 2023-03-03T09:22:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f557014c610 msgr2=0x7f5570152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f557014c610 0x7f5570152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:40.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 >> v2:172.21.15.143:3300/0 conn(0x7f55700c1f80 msgr2=0x7f557006b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 --2- 172.21.15.7:0/144274913 >> v2:172.21.15.143:3300/0 conn(0x7f55700c1f80 0x7f557006b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5570152a40 con 0x7f557006bd20 2023-03-03T09:22:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f55680019f0 con 0x7f557006bd20 2023-03-03T09:22:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 >> v1:172.21.15.7:6789/0 conn(0x7f557006bd20 legacy=0x7f55700ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f555c003a70 0x7f555c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f557014c610 0x7f555c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:40.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f555c004d90 0x7f555c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557014c9e0 con 0x7f555c004d90 2023-03-03T09:22:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5570156250 con 0x7f555c003a70 2023-03-03T09:22:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555c00baf0 con 0x7f557014c610 2023-03-03T09:22:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5576766700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f555c004d90 0x7f555c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5575764700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f557014c610 0x7f555c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.874+0000 7f5575f65700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f555c003a70 0x7f555c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:40.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f5576766700 1 -- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f557014c610 msgr2=0x7f555c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:40.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f5576766700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f557014c610 0x7f555c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:40.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f5576766700 1 -- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f555c003a70 msgr2=0x7f555c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:40.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f5576766700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f555c003a70 0x7f555c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:40.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f5576766700 1 -- 172.21.15.7:0/144274913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c0035b0 con 0x7f555c004d90 2023-03-03T09:22:40.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f5576766700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f555c004d90 0x7f555c005200 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f557006c120 tx=0x7f555c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08c7954eab55a73 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:40.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568012600 con 0x7f555c004d90 2023-03-03T09:22:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5568012760 con 0x7f555c004d90 2023-03-03T09:22:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f5574f63700 1 -- 172.21.15.7:0/144274913 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568012b10 con 0x7f555c004d90 2023-03-03T09:22:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f55781c9700 1 -- 172.21.15.7:0/144274913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f555c004d90 msgr2=0x7f555c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f55781c9700 1 --2- 172.21.15.7:0/144274913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f555c004d90 0x7f555c005200 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f557006c120 tx=0x7f555c005880 comp rx=0 tx=0).stop 2023-03-03T09:22:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f5576766700 1 -- 172.21.15.7:0/144274913 reap_dead start 2023-03-03T09:22:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f55781c9700 1 -- 172.21.15.7:0/144274913 shutdown_connections 2023-03-03T09:22:40.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.875+0000 7f55781c9700 1 -- 172.21.15.7:0/144274913 >> 172.21.15.7:0/144274913 conn(0x7f55700c62e0 msgr2=0x7f55700c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:22:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.876+0000 7f55781c9700 1 -- 172.21.15.7:0/144274913 shutdown_connections 2023-03-03T09:22:40.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.876+0000 7f55781c9700 1 -- 172.21.15.7:0/144274913 wait complete. 2023-03-03T09:22:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.877+0000 7f55781c9700 1 Processor -- start 2023-03-03T09:22:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.877+0000 7f55781c9700 1 -- start start 2023-03-03T09:22:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.877+0000 7f55781c9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f555c004d90 0x7f55700c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.877+0000 7f55781c9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55701540f0 0x7f55701eb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.877+0000 7f55781c9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55701ebc40 0x7f5570150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.877+0000 7f55781c9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55701e7eb0 con 0x7f55701ebc40 2023-03-03T09:22:40.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.877+0000 7f55781c9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5570151d50 con 0x7f555c004d90 2023-03-03T09:22:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.877+0000 7f55781c9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55701455a0 con 0x7f55701540f0 2023-03-03T09:22:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5576766700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55701ebc40 0x7f5570150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5576766700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55701ebc40 0x7f5570150070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55794/0 (socket says 172.21.15.7:55794) 2023-03-03T09:22:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5576766700 1 -- 172.21.15.7:0/3362899625 learned_addr learned my addr 172.21.15.7:0/3362899625 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:22:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5575764700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55701540f0 0x7f55701eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5575f65700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f555c004d90 0x7f55700c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:40.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5576766700 1 -- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55701540f0 msgr2=0x7f55701eb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:40.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5576766700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55701540f0 0x7f55701eb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:40.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5576766700 1 -- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f555c004d90 msgr2=0x7f55700c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:40.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5576766700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f555c004d90 0x7f55700c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:40.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5576766700 1 -- 172.21.15.7:0/3362899625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55701564a0 con 0x7f55701ebc40 2023-03-03T09:22:40.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f5576766700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55701ebc40 0x7f5570150070 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5568012630 tx=0x7f55701529b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:40.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f55627fc700 1 -- 172.21.15.7:0/3362899625 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556801e600 con 0x7f55701ebc40 2023-03-03T09:22:40.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f55627fc700 1 -- 172.21.15.7:0/3362899625 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5568001360 con 0x7f55701ebc40 2023-03-03T09:22:40.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f55781c9700 1 -- 172.21.15.7:0/3362899625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5570156780 con 0x7f55701ebc40 2023-03-03T09:22:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f55627fc700 1 -- 172.21.15.7:0/3362899625 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556801e9c0 con 0x7f55701ebc40 2023-03-03T09:22:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.878+0000 7f55781c9700 1 -- 172.21.15.7:0/3362899625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55701523f0 con 0x7f55701ebc40 2023-03-03T09:22:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.879+0000 7f55781c9700 1 -- 172.21.15.7:0/3362899625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5570074bb0 con 0x7f55701ebc40 2023-03-03T09:22:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.880+0000 7f55627fc700 1 -- 172.21.15.7:0/3362899625 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f556802d070 con 0x7f55701ebc40 2023-03-03T09:22:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.880+0000 7f55627fc700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55540682f0 0x7f555406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.880+0000 7f55627fc700 1 -- 172.21.15.7:0/3362899625 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5568087b20 con 0x7f55701ebc40 2023-03-03T09:22:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.880+0000 7f5575f65700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55540682f0 0x7f555406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:40.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.881+0000 7f5575f65700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55540682f0 0x7f555406a7a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f556c009be0 tx=0x7f556c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:40.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:40.885+0000 7f55627fc700 1 -- 172.21.15.7:0/3362899625 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5568053f00 con 0x7f55701ebc40 2023-03-03T09:22:41.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.121+0000 7f55781c9700 1 -- 172.21.15.7:0/3362899625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f557014fc70 con 0x7f55701ebc40 2023-03-03T09:22:41.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.122+0000 7f55627fc700 1 -- 172.21.15.7:0/3362899625 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f556802b090 con 0x7f55701ebc40 2023-03-03T09:22:41.124 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:22:41.125 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:22:41.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 -- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55540682f0 msgr2=0x7f555406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:41.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55540682f0 0x7f555406a7a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f556c009be0 tx=0x7f556c009210 comp rx=0 tx=0).stop 2023-03-03T09:22:41.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 -- 172.21.15.7:0/3362899625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55701ebc40 msgr2=0x7f5570150070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:41.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55701ebc40 0x7f5570150070 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5568012630 tx=0x7f55701529b0 comp rx=0 tx=0).stop 2023-03-03T09:22:41.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 -- 172.21.15.7:0/3362899625 shutdown_connections 2023-03-03T09:22:41.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55540682f0 0x7f555406a7a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:41.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55701ebc40 0x7f5570150070 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:41.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f555c004d90 0x7f55700c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:41.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 --2- 172.21.15.7:0/3362899625 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55701540f0 0x7f55701eb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:41.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 -- 172.21.15.7:0/3362899625 >> 172.21.15.7:0/3362899625 conn(0x7f55700c62e0 msgr2=0x7f55700c12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:22:41.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 -- 172.21.15.7:0/3362899625 shutdown_connections 2023-03-03T09:22:41.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:41.124+0000 7f55781c9700 1 -- 172.21.15.7:0/3362899625 wait complete. 2023-03-03T09:22:41.132 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:22:41.138 INFO:tasks.mds_thrash.fs.[b]:waiting for 90 secs before thrashing 2023-03-03T09:22:44.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:44.562 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:22:49.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:22:49.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.354+0000 7f3d38bfb700 1 Processor -- start 2023-03-03T09:22:49.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.354+0000 7f3d38bfb700 1 -- start start 2023-03-03T09:22:49.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.355+0000 7f3d38bfb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340736f0 0x7f3d34073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:49.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.355+0000 7f3d38bfb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d3406bea0 0x7f3d34158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:49.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.355+0000 7f3d38bfb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3406c3e0 con 0x7f3d3406c910 2023-03-03T09:22:49.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.355+0000 7f3d38bfb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d34158960 con 0x7f3d3406bea0 2023-03-03T09:22:49.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.355+0000 7f3d38bfb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340c32e0 con 0x7f3d340736f0 2023-03-03T09:22:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.355+0000 7f3d32d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d3406bea0 0x7f3d34158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.355+0000 7f3d32d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d3406bea0 0x7f3d34158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55810/0 (socket says 172.21.15.7:55810) 2023-03-03T09:22:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.355+0000 7f3d32d9d700 1 -- 172.21.15.7:0/2663581097 learned_addr learned my addr 172.21.15.7:0/2663581097 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:22:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.356+0000 7f3d3259c700 1 --2- 172.21.15.7:0/2663581097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340736f0 0x7f3d34073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.356+0000 7f3d32d9d700 1 -- 172.21.15.7:0/2663581097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340736f0 msgr2=0x7f3d34073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.356+0000 7f3d32d9d700 1 --2- 172.21.15.7:0/2663581097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340736f0 0x7f3d34073ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:49.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.356+0000 7f3d32d9d700 1 -- 172.21.15.7:0/2663581097 >> v1:172.21.15.143:6789/0 conn(0x7f3d3406c910 legacy=0x7f3d34157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:22:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.356+0000 7f3d32d9d700 1 -- 172.21.15.7:0/2663581097 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3406c630 con 0x7f3d3406bea0 2023-03-03T09:22:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.356+0000 7f3d3259c700 1 --2- 172.21.15.7:0/2663581097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340736f0 0x7f3d34073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:22:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.356+0000 7f3d32d9d700 1 --2- 172.21.15.7:0/2663581097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 0x7f3d34158420 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009e50 tx=0x7f3d2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91166b90e988ab48 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.356+0000 7f3d3159a700 1 -- 172.21.15.7:0/2663581097 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00e070 con 0x7f3d3406bea0 2023-03-03T09:22:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.356+0000 7f3d3159a700 1 -- 172.21.15.7:0/2663581097 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c00b9c0 con 0x7f3d3406bea0 2023-03-03T09:22:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.356+0000 7f3d3159a700 1 -- 172.21.15.7:0/2663581097 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00bd70 con 0x7f3d3406bea0 2023-03-03T09:22:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.357+0000 7f3d38bfb700 1 -- 172.21.15.7:0/2663581097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 msgr2=0x7f3d34158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:49.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.357+0000 7f3d38bfb700 1 --2- 172.21.15.7:0/2663581097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 0x7f3d34158420 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009e50 tx=0x7f3d2c00b330 comp rx=0 tx=0).stop 2023-03-03T09:22:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.357+0000 7f3d38bfb700 1 -- 172.21.15.7:0/2663581097 shutdown_connections 2023-03-03T09:22:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.357+0000 7f3d38bfb700 1 --2- 172.21.15.7:0/2663581097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 0x7f3d34158420 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.357+0000 7f3d38bfb700 1 --2- 172.21.15.7:0/2663581097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d340736f0 0x7f3d34073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.357+0000 7f3d38bfb700 1 -- 172.21.15.7:0/2663581097 >> 172.21.15.7:0/2663581097 conn(0x7f3d340a94d0 msgr2=0x7f3d340a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:22:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.357+0000 7f3d38bfb700 1 -- 172.21.15.7:0/2663581097 shutdown_connections 2023-03-03T09:22:49.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.357+0000 7f3d38bfb700 1 -- 172.21.15.7:0/2663581097 wait complete. 2023-03-03T09:22:49.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.358+0000 7f3d38bfb700 1 Processor -- start 2023-03-03T09:22:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.358+0000 7f3d38bfb700 1 -- start start 2023-03-03T09:22:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d38bfb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 0x7f3d341ec170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d38bfb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3406c910 0x7f3d341ec6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d38bfb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d340736f0 0x7f3d341ed6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d38bfb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d341ebd50 con 0x7f3d3406bea0 2023-03-03T09:22:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d38bfb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340c0d40 con 0x7f3d340736f0 2023-03-03T09:22:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d38bfb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340c0f90 con 0x7f3d3406c910 2023-03-03T09:22:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d3259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 0x7f3d341ec170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d3259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 0x7f3d341ec170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55812/0 (socket says 172.21.15.7:55812) 2023-03-03T09:22:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d3259c700 1 -- 172.21.15.7:0/3703269165 learned_addr learned my addr 172.21.15.7:0/3703269165 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:22:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d32d9d700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d340736f0 0x7f3d341ed6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d31d9b700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3406c910 0x7f3d341ec6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d3259c700 1 -- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3406c910 msgr2=0x7f3d341ec6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d3259c700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3406c910 0x7f3d341ec6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d3259c700 1 -- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d340736f0 msgr2=0x7f3d341ed6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d3259c700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d340736f0 0x7f3d341ed6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d3259c700 1 -- 172.21.15.7:0/3703269165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c009b30 con 0x7f3d3406bea0 2023-03-03T09:22:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.359+0000 7f3d3259c700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 0x7f3d341ec170 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400c750 tx=0x7f3d2400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.360+0000 7f3d2affd700 1 -- 172.21.15.7:0/3703269165 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24011800 con 0x7f3d3406bea0 2023-03-03T09:22:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.360+0000 7f3d2affd700 1 -- 172.21.15.7:0/3703269165 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d24011960 con 0x7f3d3406bea0 2023-03-03T09:22:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.360+0000 7f3d38bfb700 1 -- 172.21.15.7:0/3703269165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d340c3b60 con 0x7f3d3406bea0 2023-03-03T09:22:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.360+0000 7f3d2affd700 1 -- 172.21.15.7:0/3703269165 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24011cd0 con 0x7f3d3406bea0 2023-03-03T09:22:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.360+0000 7f3d38bfb700 1 -- 172.21.15.7:0/3703269165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d340c3880 con 0x7f3d3406bea0 2023-03-03T09:22:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.361+0000 7f3d38bfb700 1 -- 172.21.15.7:0/3703269165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d340c3580 con 0x7f3d3406bea0 2023-03-03T09:22:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.361+0000 7f3d2affd700 1 -- 172.21.15.7:0/3703269165 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d24020070 con 0x7f3d3406bea0 2023-03-03T09:22:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.361+0000 7f3d2affd700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d140682f0 0x7f3d1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:22:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.362+0000 7f3d2affd700 1 -- 172.21.15.7:0/3703269165 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3d24085f30 con 0x7f3d3406bea0 2023-03-03T09:22:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.362+0000 7f3d31d9b700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d140682f0 0x7f3d1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:22:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.362+0000 7f3d31d9b700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d140682f0 0x7f3d1406a7a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3d340712c0 tx=0x7f3d1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:22:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.366+0000 7f3d2affd700 1 -- 172.21.15.7:0/3703269165 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d240522d0 con 0x7f3d3406bea0 2023-03-03T09:22:49.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.607+0000 7f3d38bfb700 1 -- 172.21.15.7:0/3703269165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d34074bb0 con 0x7f3d3406bea0 2023-03-03T09:22:49.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.608+0000 7f3d2affd700 1 -- 172.21.15.7:0/3703269165 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3d240558f0 con 0x7f3d3406bea0 2023-03-03T09:22:49.609 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:22:49.610 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:22:49.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.609+0000 7f3d38bfb700 1 -- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d140682f0 msgr2=0x7f3d1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:49.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.609+0000 7f3d38bfb700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d140682f0 0x7f3d1406a7a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3d340712c0 tx=0x7f3d1c009040 comp rx=0 tx=0).stop 2023-03-03T09:22:49.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.609+0000 7f3d38bfb700 1 -- 172.21.15.7:0/3703269165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 msgr2=0x7f3d341ec170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:22:49.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.609+0000 7f3d38bfb700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 0x7f3d341ec170 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400c750 tx=0x7f3d2400ca60 comp rx=0 tx=0).stop 2023-03-03T09:22:49.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.609+0000 7f3d38bfb700 1 -- 172.21.15.7:0/3703269165 shutdown_connections 2023-03-03T09:22:49.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.609+0000 7f3d38bfb700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d340736f0 0x7f3d341ed6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:49.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.609+0000 7f3d38bfb700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d140682f0 0x7f3d1406a7a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:49.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.609+0000 7f3d38bfb700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3406bea0 0x7f3d341ec170 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:49.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.609+0000 7f3d38bfb700 1 --2- 172.21.15.7:0/3703269165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3406c910 0x7f3d341ec6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:22:49.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.610+0000 7f3d38bfb700 1 -- 172.21.15.7:0/3703269165 >> 172.21.15.7:0/3703269165 conn(0x7f3d340a94d0 msgr2=0x7f3d3414a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:22:49.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.610+0000 7f3d38bfb700 1 -- 172.21.15.7:0/3703269165 shutdown_connections 2023-03-03T09:22:49.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:22:49.610+0000 7f3d38bfb700 1 -- 172.21.15.7:0/3703269165 wait complete. 2023-03-03T09:22:49.613 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:22:49.623 INFO:tasks.mds_thrash.fs.[a]:waiting for 55 secs before thrashing 2023-03-03T09:23:14.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:14.643 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:44.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:23:44.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:44.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:23:44.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.761+0000 7fdd78ddd700 1 Processor -- start 2023-03-03T09:23:44.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.761+0000 7fdd78ddd700 1 -- start start 2023-03-03T09:23:44.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd78ddd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdd740c5140 0x7fdd74157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:44.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd78ddd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741580e0 0x7fdd7406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:44.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd78ddd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7406dad0 con 0x7fdd74151de0 2023-03-03T09:23:44.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd78ddd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7406e370 con 0x7fdd740c5140 2023-03-03T09:23:44.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd78ddd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7406ddb0 con 0x7fdd741580e0 2023-03-03T09:23:44.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd7259c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdd74151de0 0x7fdd740c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49820/0 (socket says 172.21.15.7:49820) 2023-03-03T09:23:44.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd7259c700 1 -- 172.21.15.7:0/1453792012 learned_addr learned my addr 172.21.15.7:0/1453792012 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:23:44.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd71d9b700 1 --2- 172.21.15.7:0/1453792012 >> v2:172.21.15.143:3300/0 conn(0x7fdd740c5140 0x7fdd74157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:44.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd72d9d700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741580e0 0x7fdd7406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:44.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197140590 0 0) 0x7fdd7406dad0 con 0x7fdd74151de0 2023-03-03T09:23:44.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd5c0035b0 con 0x7fdd74151de0 2023-03-03T09:23:44.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.762+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4008750690 0 0) 0x7fdd5c0035b0 con 0x7fdd74151de0 2023-03-03T09:23:44.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741580e0 msgr2=0x7fdd7406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:23:44.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741580e0 0x7fdd7406a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:44.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 >> v2:172.21.15.143:3300/0 conn(0x7fdd740c5140 msgr2=0x7fdd74157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:23:44.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 --2- 172.21.15.7:0/1453792012 >> v2:172.21.15.143:3300/0 conn(0x7fdd740c5140 0x7fdd74157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:44.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7406e090 con 0x7fdd74151de0 2023-03-03T09:23:44.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fdd64001670 con 0x7fdd74151de0 2023-03-03T09:23:44.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 >> v1:172.21.15.7:6789/0 conn(0x7fdd74151de0 legacy=0x7fdd740c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:44.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 0x7fdd5c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:44.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd5c004850 0x7fdd5c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:44.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741580e0 0x7fdd5c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:44.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7406e370 con 0x7fdd5c003a70 2023-03-03T09:23:44.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7406ddb0 con 0x7fdd5c004850 2023-03-03T09:23:44.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c00ba10 con 0x7fdd741580e0 2023-03-03T09:23:44.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd71d9b700 1 --2- 172.21.15.7:0/1453792012 >> v2:172.21.15.143:3300/0 conn(0x7fdd740c5140 0x7fdd74157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:23:44.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7259c700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 0x7fdd5c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:44.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd71d9b700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd5c004850 0x7fdd5c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:44.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd72d9d700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741580e0 0x7fdd5c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:44.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7259c700 1 -- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741580e0 msgr2=0x7fdd5c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:23:44.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7259c700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741580e0 0x7fdd5c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:44.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7259c700 1 -- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd5c004850 msgr2=0x7fdd5c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:23:44.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7259c700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd5c004850 0x7fdd5c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:44.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7259c700 1 -- 172.21.15.7:0/1453792012 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0035b0 con 0x7fdd5c003a70 2023-03-03T09:23:44.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd71d9b700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd5c004850 0x7fdd5c004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:23:44.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7259c700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 0x7fdd5c003e80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400c320 tx=0x7fdd5c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a61cb5c6b56e404 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:23:44.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6400ce20 con 0x7fdd5c003a70 2023-03-03T09:23:44.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd64005960 con 0x7fdd5c003a70 2023-03-03T09:23:44.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.763+0000 7fdd7159a700 1 -- 172.21.15.7:0/1453792012 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64005d10 con 0x7fdd5c003a70 2023-03-03T09:23:44.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.764+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1453792012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 msgr2=0x7fdd5c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:44.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.764+0000 7fdd78ddd700 1 --2- 172.21.15.7:0/1453792012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 0x7fdd5c003e80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400c320 tx=0x7fdd5c005880 comp rx=0 tx=0).stop 2023-03-03T09:23:44.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.764+0000 7fdd72d9d700 1 -- 172.21.15.7:0/1453792012 reap_dead start 2023-03-03T09:23:44.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.764+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1453792012 shutdown_connections 2023-03-03T09:23:44.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.764+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1453792012 >> 172.21.15.7:0/1453792012 conn(0x7fdd740a9430 msgr2=0x7fdd740c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:23:44.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.764+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1453792012 shutdown_connections 2023-03-03T09:23:44.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.764+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1453792012 wait complete. 2023-03-03T09:23:44.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.765+0000 7fdd78ddd700 1 Processor -- start 2023-03-03T09:23:44.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.765+0000 7fdd78ddd700 1 -- start start 2023-03-03T09:23:44.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd78ddd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 0x7fdd74148490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:44.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd78ddd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd741489d0 0x7fdd741ed0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:44.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd78ddd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741ed600 0x7fdd7414c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:44.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd78ddd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74148e20 con 0x7fdd5c003a70 2023-03-03T09:23:44.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd78ddd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7414df60 con 0x7fdd741489d0 2023-03-03T09:23:44.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd78ddd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7414c520 con 0x7fdd741ed600 2023-03-03T09:23:44.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd7259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 0x7fdd74148490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:44.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd7259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 0x7fdd74148490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55852/0 (socket says 172.21.15.7:55852) 2023-03-03T09:23:44.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd7259c700 1 -- 172.21.15.7:0/1291928284 learned_addr learned my addr 172.21.15.7:0/1291928284 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:23:44.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd72d9d700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741ed600 0x7fdd7414c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:44.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.766+0000 7fdd71d9b700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd741489d0 0x7fdd741ed0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:44.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd7259c700 1 -- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741ed600 msgr2=0x7fdd7414c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:44.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd7259c700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741ed600 0x7fdd7414c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:44.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd7259c700 1 -- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd741489d0 msgr2=0x7fdd741ed0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:44.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd7259c700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd741489d0 0x7fdd741ed0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:44.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd7259c700 1 -- 172.21.15.7:0/1291928284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd740c1c10 con 0x7fdd5c003a70 2023-03-03T09:23:44.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd7259c700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 0x7fdd74148490 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fdd64001660 tx=0x7fdd64001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:23:44.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd62ffd700 1 -- 172.21.15.7:0/1291928284 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64004870 con 0x7fdd5c003a70 2023-03-03T09:23:44.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd62ffd700 1 -- 172.21.15.7:0/1291928284 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd64004a10 con 0x7fdd5c003a70 2023-03-03T09:23:44.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd62ffd700 1 -- 172.21.15.7:0/1291928284 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64004dc0 con 0x7fdd5c003a70 2023-03-03T09:23:44.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1291928284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd740c4c20 con 0x7fdd5c003a70 2023-03-03T09:23:44.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.767+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1291928284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd74066de0 con 0x7fdd5c003a70 2023-03-03T09:23:44.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.768+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1291928284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7414c800 con 0x7fdd5c003a70 2023-03-03T09:23:44.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.768+0000 7fdd62ffd700 1 -- 172.21.15.7:0/1291928284 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdd64010070 con 0x7fdd5c003a70 2023-03-03T09:23:44.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.769+0000 7fdd62ffd700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd54068320 0x7fdd5406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:44.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.769+0000 7fdd62ffd700 1 -- 172.21.15.7:0/1291928284 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdd64087540 con 0x7fdd5c003a70 2023-03-03T09:23:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.769+0000 7fdd71d9b700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd54068320 0x7fdd5406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.770+0000 7fdd71d9b700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd54068320 0x7fdd5406a7d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdd7406c910 tx=0x7fdd68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:23:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:44.773+0000 7fdd62ffd700 1 -- 172.21.15.7:0/1291928284 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdd640538e0 con 0x7fdd5c003a70 2023-03-03T09:23:45.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.008+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1291928284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd74074bb0 con 0x7fdd5c003a70 2023-03-03T09:23:45.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.008+0000 7fdd62ffd700 1 -- 172.21.15.7:0/1291928284 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdd64007dc0 con 0x7fdd5c003a70 2023-03-03T09:23:45.011 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:23:45.011 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:23:45.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.010+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd54068320 msgr2=0x7fdd5406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:45.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.010+0000 7fdd78ddd700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd54068320 0x7fdd5406a7d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdd7406c910 tx=0x7fdd68008040 comp rx=0 tx=0).stop 2023-03-03T09:23:45.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.011+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1291928284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 msgr2=0x7fdd74148490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:45.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.011+0000 7fdd78ddd700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 0x7fdd74148490 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fdd64001660 tx=0x7fdd64001a00 comp rx=0 tx=0).stop 2023-03-03T09:23:45.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.011+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1291928284 shutdown_connections 2023-03-03T09:23:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.011+0000 7fdd78ddd700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd741ed600 0x7fdd7414c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.011+0000 7fdd78ddd700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd54068320 0x7fdd5406a7d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.011+0000 7fdd78ddd700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd5c003a70 0x7fdd74148490 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.011+0000 7fdd78ddd700 1 --2- 172.21.15.7:0/1291928284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd741489d0 0x7fdd741ed0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.011+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1291928284 >> 172.21.15.7:0/1291928284 conn(0x7fdd740a9430 msgr2=0x7fdd740a5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:23:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.011+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1291928284 shutdown_connections 2023-03-03T09:23:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:45.011+0000 7fdd78ddd700 1 -- 172.21.15.7:0/1291928284 wait complete. 2023-03-03T09:23:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:23:45.025 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before thrashing 2023-03-03T09:23:50.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:23:50.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.164+0000 7fcbef4c6700 1 Processor -- start 2023-03-03T09:23:50.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.164+0000 7fcbef4c6700 1 -- start start 2023-03-03T09:23:50.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.165+0000 7fcbef4c6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcbe80c1da0 0x7fcbe8139d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:50.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.165+0000 7fcbef4c6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe80c7b90 con 0x7fcbe806d4b0 2023-03-03T09:23:50.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.165+0000 7fcbef4c6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe8154640 con 0x7fcbe80c8020 2023-03-03T09:23:50.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.165+0000 7fcbef4c6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe80c7330 con 0x7fcbe80c1da0 2023-03-03T09:23:50.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.165+0000 7fcbed262700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcbe806d4b0 0x7fcbe806d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49852/0 (socket says 172.21.15.7:49852) 2023-03-03T09:23:50.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.165+0000 7fcbed262700 1 -- 172.21.15.7:0/2241166240 learned_addr learned my addr 172.21.15.7:0/2241166240 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:23:50.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.165+0000 7fcbeda63700 1 --2- 172.21.15.7:0/2241166240 >> v2:172.21.15.143:3300/0 conn(0x7fcbe80c1da0 0x7fcbe8139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:50.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.165+0000 7fcbdffff700 1 -- 172.21.15.7:0/2241166240 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173441062 0 0) 0x7fcbe80c7b90 con 0x7fcbe806d4b0 2023-03-03T09:23:50.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.165+0000 7fcbdffff700 1 -- 172.21.15.7:0/2241166240 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd40035b0 con 0x7fcbe806d4b0 2023-03-03T09:23:50.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.166+0000 7fcbdffff700 1 -- 172.21.15.7:0/2241166240 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633388730 0 0) 0x7fcbe8154640 con 0x7fcbe80c8020 2023-03-03T09:23:50.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.166+0000 7fcbdffff700 1 -- 172.21.15.7:0/2241166240 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe80c7b90 con 0x7fcbe80c8020 2023-03-03T09:23:50.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.166+0000 7fcbdffff700 1 -- 172.21.15.7:0/2241166240 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2674488337 0 0) 0x7fcbd40035b0 con 0x7fcbe806d4b0 2023-03-03T09:23:50.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.166+0000 7fcbeda63700 1 -- 172.21.15.7:0/2241166240 >> v1:172.21.15.143:6789/0 conn(0x7fcbe80c8020 legacy=0x7fcbe80c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:50.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.166+0000 7fcbeda63700 1 -- 172.21.15.7:0/2241166240 >> v1:172.21.15.7:6789/0 conn(0x7fcbe806d4b0 legacy=0x7fcbe806d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:50.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.166+0000 7fcbeda63700 1 -- 172.21.15.7:0/2241166240 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80c7540 con 0x7fcbe80c1da0 2023-03-03T09:23:50.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.166+0000 7fcbeda63700 1 --2- 172.21.15.7:0/2241166240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe80c1da0 0x7fcbe8139d80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009d90 tx=0x7fcbe000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ab2bda42eb86947 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:23:50.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.166+0000 7fcbdffff700 1 -- 172.21.15.7:0/2241166240 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe000e070 con 0x7fcbe80c1da0 2023-03-03T09:23:50.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.166+0000 7fcbdffff700 1 -- 172.21.15.7:0/2241166240 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe0003a20 con 0x7fcbe80c1da0 2023-03-03T09:23:50.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.166+0000 7fcbdffff700 1 -- 172.21.15.7:0/2241166240 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8001670 con 0x7fcbe80c1da0 2023-03-03T09:23:50.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.167+0000 7fcbef4c6700 1 -- 172.21.15.7:0/2241166240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe80c1da0 msgr2=0x7fcbe8139d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:50.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.167+0000 7fcbef4c6700 1 --2- 172.21.15.7:0/2241166240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe80c1da0 0x7fcbe8139d80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009d90 tx=0x7fcbe000b330 comp rx=0 tx=0).stop 2023-03-03T09:23:50.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.167+0000 7fcbef4c6700 1 -- 172.21.15.7:0/2241166240 shutdown_connections 2023-03-03T09:23:50.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.167+0000 7fcbef4c6700 1 --2- 172.21.15.7:0/2241166240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe80c1da0 0x7fcbe8139d80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:50.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.167+0000 7fcbef4c6700 1 -- 172.21.15.7:0/2241166240 >> 172.21.15.7:0/2241166240 conn(0x7fcbe806c110 msgr2=0x7fcbe806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:23:50.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.167+0000 7fcbef4c6700 1 -- 172.21.15.7:0/2241166240 shutdown_connections 2023-03-03T09:23:50.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.167+0000 7fcbef4c6700 1 -- 172.21.15.7:0/2241166240 wait complete. 2023-03-03T09:23:50.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.168+0000 7fcbef4c6700 1 Processor -- start 2023-03-03T09:23:50.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.168+0000 7fcbef4c6700 1 -- start start 2023-03-03T09:23:50.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbef4c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe806d4b0 0x7fcbe81dc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:50.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbef4c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe80c1da0 0x7fcbe80714f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:50.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbef4c6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe80c8020 0x7fcbe81dd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:50.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbef4c6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81db3c0 con 0x7fcbe80c8020 2023-03-03T09:23:50.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbef4c6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81dab60 con 0x7fcbe806d4b0 2023-03-03T09:23:50.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbef4c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81dd490 con 0x7fcbe80c1da0 2023-03-03T09:23:50.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbeda63700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe80c8020 0x7fcbe81dd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:50.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbeda63700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe80c8020 0x7fcbe81dd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55882/0 (socket says 172.21.15.7:55882) 2023-03-03T09:23:50.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbeda63700 1 -- 172.21.15.7:0/669538047 learned_addr learned my addr 172.21.15.7:0/669538047 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:23:50.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbeca61700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe80c1da0 0x7fcbe80714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:50.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbed262700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe806d4b0 0x7fcbe81dc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:50.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbeda63700 1 -- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe80c1da0 msgr2=0x7fcbe80714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:50.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbeda63700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe80c1da0 0x7fcbe80714f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:50.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbeda63700 1 -- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe806d4b0 msgr2=0x7fcbe81dc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:50.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbeda63700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe806d4b0 0x7fcbe81dc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:50.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbeda63700 1 -- 172.21.15.7:0/669538047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe0009a70 con 0x7fcbe80c8020 2023-03-03T09:23:50.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.169+0000 7fcbeda63700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe80c8020 0x7fcbe81dd990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0015040 tx=0x7fcbe0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:23:50.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.170+0000 7fcbddffb700 1 -- 172.21.15.7:0/669538047 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe00044c0 con 0x7fcbe80c8020 2023-03-03T09:23:50.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.170+0000 7fcbddffb700 1 -- 172.21.15.7:0/669538047 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe0004620 con 0x7fcbe80c8020 2023-03-03T09:23:50.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.170+0000 7fcbef4c6700 1 -- 172.21.15.7:0/669538047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe806e210 con 0x7fcbe80c8020 2023-03-03T09:23:50.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.170+0000 7fcbddffb700 1 -- 172.21.15.7:0/669538047 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0012430 con 0x7fcbe80c8020 2023-03-03T09:23:50.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.170+0000 7fcbef4c6700 1 -- 172.21.15.7:0/669538047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe80c7820 con 0x7fcbe80c8020 2023-03-03T09:23:50.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.171+0000 7fcbef4c6700 1 -- 172.21.15.7:0/669538047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe80c0ee0 con 0x7fcbe80c8020 2023-03-03T09:23:50.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.171+0000 7fcbddffb700 1 -- 172.21.15.7:0/669538047 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcbe0023030 con 0x7fcbe80c8020 2023-03-03T09:23:50.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.172+0000 7fcbddffb700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:23:50.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.172+0000 7fcbddffb700 1 -- 172.21.15.7:0/669538047 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcbe0087490 con 0x7fcbe80c8020 2023-03-03T09:23:50.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.172+0000 7fcbed262700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:23:50.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.173+0000 7fcbed262700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8000fd0 tx=0x7fcbd8001670 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:23:50.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.176+0000 7fcbddffb700 1 -- 172.21.15.7:0/669538047 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcbe0053830 con 0x7fcbe80c8020 2023-03-03T09:23:50.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.414+0000 7fcbef4c6700 1 -- 172.21.15.7:0/669538047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbe81dd6b0 con 0x7fcbe80c8020 2023-03-03T09:23:50.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.414+0000 7fcbddffb700 1 -- 172.21.15.7:0/669538047 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcbe81dd6b0 con 0x7fcbe80c8020 2023-03-03T09:23:50.416 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:23:50.416 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:23:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 -- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 msgr2=0x7fcbcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8000fd0 tx=0x7fcbd8001670 comp rx=0 tx=0).stop 2023-03-03T09:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 -- 172.21.15.7:0/669538047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe80c8020 msgr2=0x7fcbe81dd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe80c8020 0x7fcbe81dd990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0015040 tx=0x7fcbe0004ad0 comp rx=0 tx=0).stop 2023-03-03T09:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 -- 172.21.15.7:0/669538047 shutdown_connections 2023-03-03T09:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:50.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe80c8020 0x7fcbe81dd990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe806d4b0 0x7fcbe81dc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 --2- 172.21.15.7:0/669538047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe80c1da0 0x7fcbe80714f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 -- 172.21.15.7:0/669538047 >> 172.21.15.7:0/669538047 conn(0x7fcbe806c110 msgr2=0x7fcbe80c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 -- 172.21.15.7:0/669538047 shutdown_connections 2023-03-03T09:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:23:50.416+0000 7fcbef4c6700 1 -- 172.21.15.7:0/669538047 wait complete. 2023-03-03T09:23:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:23:50.429 INFO:tasks.mds_thrash.fs.[a]:waiting for 88 secs before thrashing 2023-03-03T09:24:11.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:24:11.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.275+0000 7f29b8ae6700 1 Processor -- start 2023-03-03T09:24:11.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.275+0000 7f29b8ae6700 1 -- start start 2023-03-03T09:24:11.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.276+0000 7f29b8ae6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29b40caa30 0x7f29b40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:11.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.276+0000 7f29b8ae6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 0x7f29b4154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:11.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.276+0000 7f29b8ae6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b40cb280 con 0x7f29b40cae00 2023-03-03T09:24:11.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.276+0000 7f29b8ae6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b40c7310 con 0x7f29b40caa30 2023-03-03T09:24:11.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.276+0000 7f29b8ae6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b40bfd10 con 0x7f29b40c00e0 2023-03-03T09:24:11.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29b40caa30 0x7f29b40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:11.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b2d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f29b40cae00 0x7f29b40c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49872/0 (socket says 172.21.15.7:49872) 2023-03-03T09:24:11.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b2d9d700 1 -- 172.21.15.7:0/1019096592 learned_addr learned my addr 172.21.15.7:0/1019096592 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:24:11.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b1d9b700 1 --2- 172.21.15.7:0/1019096592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 0x7f29b4154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:11.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b259c700 1 -- 172.21.15.7:0/1019096592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 msgr2=0x7f29b4154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:24:11.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b259c700 1 --2- 172.21.15.7:0/1019096592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 0x7f29b4154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:11.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b159a700 1 -- 172.21.15.7:0/1019096592 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572493597 0 0) 0x7f29b40cb280 con 0x7f29b40cae00 2023-03-03T09:24:11.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b259c700 1 -- 172.21.15.7:0/1019096592 >> v1:172.21.15.7:6789/0 conn(0x7f29b40cae00 legacy=0x7f29b40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:11.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b259c700 1 -- 172.21.15.7:0/1019096592 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b40bfa30 con 0x7f29b40caa30 2023-03-03T09:24:11.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b259c700 1 --2- 172.21.15.7:0/1019096592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40caa30 0x7f29b40c5c60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009840 tx=0x7f29a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbfa6a5ffad5b45a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:24:11.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b159a700 1 -- 172.21.15.7:0/1019096592 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4011830 con 0x7f29b40caa30 2023-03-03T09:24:11.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b159a700 1 -- 172.21.15.7:0/1019096592 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a4011990 con 0x7f29b40caa30 2023-03-03T09:24:11.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.277+0000 7f29b159a700 1 -- 172.21.15.7:0/1019096592 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4011d20 con 0x7f29b40caa30 2023-03-03T09:24:11.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.278+0000 7f29b8ae6700 1 -- 172.21.15.7:0/1019096592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40caa30 msgr2=0x7f29b40c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:11.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.278+0000 7f29b8ae6700 1 --2- 172.21.15.7:0/1019096592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40caa30 0x7f29b40c5c60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009840 tx=0x7f29a4009b50 comp rx=0 tx=0).stop 2023-03-03T09:24:11.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.278+0000 7f29b8ae6700 1 -- 172.21.15.7:0/1019096592 shutdown_connections 2023-03-03T09:24:11.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.278+0000 7f29b8ae6700 1 --2- 172.21.15.7:0/1019096592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40caa30 0x7f29b40c5c60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:11.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.278+0000 7f29b8ae6700 1 --2- 172.21.15.7:0/1019096592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 0x7f29b4154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:11.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.278+0000 7f29b8ae6700 1 -- 172.21.15.7:0/1019096592 >> 172.21.15.7:0/1019096592 conn(0x7f29b4066e40 msgr2=0x7f29b4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:24:11.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.278+0000 7f29b8ae6700 1 -- 172.21.15.7:0/1019096592 shutdown_connections 2023-03-03T09:24:11.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.278+0000 7f29b8ae6700 1 -- 172.21.15.7:0/1019096592 wait complete. 2023-03-03T09:24:11.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.279+0000 7f29b8ae6700 1 Processor -- start 2023-03-03T09:24:11.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.279+0000 7f29b8ae6700 1 -- start start 2023-03-03T09:24:11.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b8ae6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 0x7f29b40c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:11.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b8ae6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b40caa30 0x7f29b4157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:11.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b8ae6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40cae00 0x7f29b41ed120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:11.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b8ae6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b41dda40 con 0x7f29b40cae00 2023-03-03T09:24:11.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b8ae6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b4158860 con 0x7f29b40caa30 2023-03-03T09:24:11.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b8ae6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b40c4760 con 0x7f29b40c00e0 2023-03-03T09:24:11.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40cae00 0x7f29b41ed120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:11.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40cae00 0x7f29b41ed120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55898/0 (socket says 172.21.15.7:55898) 2023-03-03T09:24:11.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b2d9d700 1 -- 172.21.15.7:0/2065968433 learned_addr learned my addr 172.21.15.7:0/2065968433 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:24:11.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b259c700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 0x7f29b40c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:11.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b1d9b700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b40caa30 0x7f29b4157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:11.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.280+0000 7f29b2d9d700 1 -- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 msgr2=0x7f29b40c26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:11.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29b2d9d700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 0x7f29b40c26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:11.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29b2d9d700 1 -- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b40caa30 msgr2=0x7f29b4157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:11.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29b2d9d700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b40caa30 0x7f29b4157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:11.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29b2d9d700 1 -- 172.21.15.7:0/2065968433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a40094a0 con 0x7f29b40cae00 2023-03-03T09:24:11.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29b259c700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 0x7f29b40c26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:24:11.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29b2d9d700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40cae00 0x7f29b41ed120 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00cbd0 tx=0x7f29ac00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:24:11.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29a2ffd700 1 -- 172.21.15.7:0/2065968433 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac00ab70 con 0x7f29b40cae00 2023-03-03T09:24:11.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29a2ffd700 1 -- 172.21.15.7:0/2065968433 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ac00acf0 con 0x7f29b40cae00 2023-03-03T09:24:11.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29b8ae6700 1 -- 172.21.15.7:0/2065968433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b41547f0 con 0x7f29b40cae00 2023-03-03T09:24:11.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29a2ffd700 1 -- 172.21.15.7:0/2065968433 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac007310 con 0x7f29b40cae00 2023-03-03T09:24:11.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.281+0000 7f29b8ae6700 1 -- 172.21.15.7:0/2065968433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b4154510 con 0x7f29b40cae00 2023-03-03T09:24:11.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.282+0000 7f29b8ae6700 1 -- 172.21.15.7:0/2065968433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b4154ad0 con 0x7f29b40cae00 2023-03-03T09:24:11.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.282+0000 7f29a2ffd700 1 -- 172.21.15.7:0/2065968433 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f29ac01a030 con 0x7f29b40cae00 2023-03-03T09:24:11.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.283+0000 7f29a2ffd700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29940682f0 0x7f299406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:24:11.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.283+0000 7f29a2ffd700 1 -- 172.21.15.7:0/2065968433 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f29ac085e70 con 0x7f29b40cae00 2023-03-03T09:24:11.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.283+0000 7f29b259c700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29940682f0 0x7f299406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:24:11.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.283+0000 7f29b259c700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29940682f0 0x7f299406a7a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009fd0 tx=0x7f29a4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:24:11.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.287+0000 7f29a2ffd700 1 -- 172.21.15.7:0/2065968433 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f29ac052210 con 0x7f29b40cae00 2023-03-03T09:24:11.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.526+0000 7f29b8ae6700 1 -- 172.21.15.7:0/2065968433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29b4074bb0 con 0x7f29b40cae00 2023-03-03T09:24:11.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.527+0000 7f29a2ffd700 1 -- 172.21.15.7:0/2065968433 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f29ac018090 con 0x7f29b40cae00 2023-03-03T09:24:11.528 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:24:11.528 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:24:11.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 -- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29940682f0 msgr2=0x7f299406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:11.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29940682f0 0x7f299406a7a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f29a4009fd0 tx=0x7f29a4008030 comp rx=0 tx=0).stop 2023-03-03T09:24:11.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 -- 172.21.15.7:0/2065968433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40cae00 msgr2=0x7f29b41ed120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:24:11.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40cae00 0x7f29b41ed120 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00cbd0 tx=0x7f29ac00cf90 comp rx=0 tx=0).stop 2023-03-03T09:24:11.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 -- 172.21.15.7:0/2065968433 shutdown_connections 2023-03-03T09:24:11.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29940682f0 0x7f299406a7a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:11.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b40cae00 0x7f29b41ed120 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:11.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29b40c00e0 0x7f29b40c26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 --2- 172.21.15.7:0/2065968433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b40caa30 0x7f29b4157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 -- 172.21.15.7:0/2065968433 >> 172.21.15.7:0/2065968433 conn(0x7f29b4066e40 msgr2=0x7f29b4153c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 -- 172.21.15.7:0/2065968433 shutdown_connections 2023-03-03T09:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:24:11.529+0000 7f29b8ae6700 1 -- 172.21.15.7:0/2065968433 wait complete. 2023-03-03T09:24:11.532 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:24:11.542 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-03-03T09:24:14.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:14.784 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:44.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:24:44.847 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:14.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:14.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:18.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:25:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.567+0000 7f18d102a700 1 Processor -- start 2023-03-03T09:25:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.567+0000 7f18d102a700 1 -- start start 2023-03-03T09:25:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18d102a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18cc13a0f0 0x7f18cc0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18d102a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f18cc1586c0 0x7f18cc06d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18d102a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18cc0c1da0 0x7f18cc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18d102a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18cc154840 con 0x7f18cc0c1da0 2023-03-03T09:25:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18d102a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18cc154560 con 0x7f18cc1586c0 2023-03-03T09:25:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18d102a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc157be0 con 0x7f18cc13a0f0 2023-03-03T09:25:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18cb59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18cc0c1da0 0x7f18cc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18cb59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18cc0c1da0 0x7f18cc06d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55944/0 (socket says 172.21.15.7:55944) 2023-03-03T09:25:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18cb59e700 1 -- 172.21.15.7:0/3659978561 learned_addr learned my addr 172.21.15.7:0/3659978561 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:25:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18ca59c700 1 --2- 172.21.15.7:0/3659978561 >> v2:172.21.15.143:3300/0 conn(0x7f18cc1586c0 0x7f18cc06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18cad9d700 1 --2- 172.21.15.7:0/3659978561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18cc13a0f0 0x7f18cc0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18cb59e700 1 -- 172.21.15.7:0/3659978561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18cc13a0f0 msgr2=0x7f18cc0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:25:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18cb59e700 1 --2- 172.21.15.7:0/3659978561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18cc13a0f0 0x7f18cc0c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18cb59e700 1 -- 172.21.15.7:0/3659978561 >> v2:172.21.15.143:3300/0 conn(0x7f18cc1586c0 msgr2=0x7f18cc06d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:25:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18cb59e700 1 --2- 172.21.15.7:0/3659978561 >> v2:172.21.15.143:3300/0 conn(0x7f18cc1586c0 0x7f18cc06d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18cb59e700 1 -- 172.21.15.7:0/3659978561 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18cc157ec0 con 0x7f18cc0c1da0 2023-03-03T09:25:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18ca59c700 1 --2- 172.21.15.7:0/3659978561 >> v2:172.21.15.143:3300/0 conn(0x7f18cc1586c0 0x7f18cc06d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:25:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18cb59e700 1 --2- 172.21.15.7:0/3659978561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc0c1da0 0x7f18cc06d780 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f18cc0aa1d0 tx=0x7f18c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c15011ebddc5ed server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:25:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.568+0000 7f18c9d9b700 1 -- 172.21.15.7:0/3659978561 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c400e070 con 0x7f18cc0c1da0 2023-03-03T09:25:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18c9d9b700 1 -- 172.21.15.7:0/3659978561 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c400b9c0 con 0x7f18cc0c1da0 2023-03-03T09:25:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18c9d9b700 1 -- 172.21.15.7:0/3659978561 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c400bd70 con 0x7f18cc0c1da0 2023-03-03T09:25:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18d102a700 1 -- 172.21.15.7:0/3659978561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc0c1da0 msgr2=0x7f18cc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18d102a700 1 --2- 172.21.15.7:0/3659978561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc0c1da0 0x7f18cc06d780 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f18cc0aa1d0 tx=0x7f18c400b330 comp rx=0 tx=0).stop 2023-03-03T09:25:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18d102a700 1 -- 172.21.15.7:0/3659978561 shutdown_connections 2023-03-03T09:25:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18d102a700 1 --2- 172.21.15.7:0/3659978561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc0c1da0 0x7f18cc06d780 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18d102a700 1 --2- 172.21.15.7:0/3659978561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18cc13a0f0 0x7f18cc0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18d102a700 1 --2- 172.21.15.7:0/3659978561 >> v2:172.21.15.143:3300/0 conn(0x7f18cc1586c0 0x7f18cc06d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18d102a700 1 -- 172.21.15.7:0/3659978561 >> 172.21.15.7:0/3659978561 conn(0x7f18cc06c110 msgr2=0x7f18cc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:25:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18d102a700 1 -- 172.21.15.7:0/3659978561 shutdown_connections 2023-03-03T09:25:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.569+0000 7f18d102a700 1 -- 172.21.15.7:0/3659978561 wait complete. 2023-03-03T09:25:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.571+0000 7f18d102a700 1 Processor -- start 2023-03-03T09:25:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.571+0000 7f18d102a700 1 -- start start 2023-03-03T09:25:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.571+0000 7f18d102a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18cc0c1da0 0x7f18cc071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.571+0000 7f18d102a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18cc13a0f0 0x7f18cc1510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.571+0000 7f18d102a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc1586c0 0x7f18cc151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.571+0000 7f18d102a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc156d20 con 0x7f18cc1586c0 2023-03-03T09:25:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.571+0000 7f18d102a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc1566f0 con 0x7f18cc13a0f0 2023-03-03T09:25:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.571+0000 7f18d102a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc1ea1e0 con 0x7f18cc0c1da0 2023-03-03T09:25:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18cb59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc1586c0 0x7f18cc151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18cb59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc1586c0 0x7f18cc151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55948/0 (socket says 172.21.15.7:55948) 2023-03-03T09:25:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18cb59e700 1 -- 172.21.15.7:0/3316795173 learned_addr learned my addr 172.21.15.7:0/3316795173 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:25:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18cad9d700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18cc0c1da0 0x7f18cc071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18ca59c700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18cc13a0f0 0x7f18cc1510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18cb59e700 1 -- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18cc0c1da0 msgr2=0x7f18cc071150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18cb59e700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18cc0c1da0 0x7f18cc071150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18cb59e700 1 -- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18cc13a0f0 msgr2=0x7f18cc1510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18cb59e700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18cc13a0f0 0x7f18cc1510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18cb59e700 1 -- 172.21.15.7:0/3316795173 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c4009b30 con 0x7f18cc1586c0 2023-03-03T09:25:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18cb59e700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc1586c0 0x7f18cc151600 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f18cc06cff0 tx=0x7f18c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:25:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18b37fe700 1 -- 172.21.15.7:0/3316795173 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4004040 con 0x7f18cc1586c0 2023-03-03T09:25:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18b37fe700 1 -- 172.21.15.7:0/3316795173 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c40041c0 con 0x7f18cc1586c0 2023-03-03T09:25:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18b37fe700 1 -- 172.21.15.7:0/3316795173 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4004570 con 0x7f18cc1586c0 2023-03-03T09:25:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18d102a700 1 -- 172.21.15.7:0/3316795173 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18cc06e1a0 con 0x7f18cc1586c0 2023-03-03T09:25:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.572+0000 7f18d102a700 1 -- 172.21.15.7:0/3316795173 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18cc1581a0 con 0x7f18cc1586c0 2023-03-03T09:25:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.573+0000 7f18d102a700 1 -- 172.21.15.7:0/3316795173 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18cc06e330 con 0x7f18cc1586c0 2023-03-03T09:25:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.574+0000 7f18b37fe700 1 -- 172.21.15.7:0/3316795173 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f18c40046d0 con 0x7f18cc1586c0 2023-03-03T09:25:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.574+0000 7f18b37fe700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18b80682f0 0x7f18b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:25:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.575+0000 7f18b37fe700 1 -- 172.21.15.7:0/3316795173 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f18c4087610 con 0x7f18cc1586c0 2023-03-03T09:25:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.575+0000 7f18cad9d700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18b80682f0 0x7f18b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:25:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.575+0000 7f18cad9d700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18b80682f0 0x7f18b806a7a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f18cc0ac170 tx=0x7f18bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:25:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.579+0000 7f18b37fe700 1 -- 172.21.15.7:0/3316795173 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18c40539b0 con 0x7f18cc1586c0 2023-03-03T09:25:18.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.819+0000 7f18d102a700 1 -- 172.21.15.7:0/3316795173 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18cc156e60 con 0x7f18cc1586c0 2023-03-03T09:25:18.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.820+0000 7f18b37fe700 1 -- 172.21.15.7:0/3316795173 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f18cc156e60 con 0x7f18cc1586c0 2023-03-03T09:25:18.821 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:25:18.821 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:25:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 -- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18b80682f0 msgr2=0x7f18b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18b80682f0 0x7f18b806a7a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f18cc0ac170 tx=0x7f18bc008040 comp rx=0 tx=0).stop 2023-03-03T09:25:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 -- 172.21.15.7:0/3316795173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc1586c0 msgr2=0x7f18cc151600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:25:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc1586c0 0x7f18cc151600 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f18cc06cff0 tx=0x7f18c4004ad0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 -- 172.21.15.7:0/3316795173 shutdown_connections 2023-03-03T09:25:18.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18b80682f0 0x7f18b806a7a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18cc1586c0 0x7f18cc151600 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18cc0c1da0 0x7f18cc071150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 --2- 172.21.15.7:0/3316795173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18cc13a0f0 0x7f18cc1510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:25:18.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 -- 172.21.15.7:0/3316795173 >> 172.21.15.7:0/3316795173 conn(0x7f18cc06c110 msgr2=0x7f18cc137150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:25:18.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 -- 172.21.15.7:0/3316795173 shutdown_connections 2023-03-03T09:25:18.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:25:18.822+0000 7f18d102a700 1 -- 172.21.15.7:0/3316795173 wait complete. 2023-03-03T09:25:18.825 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:25:18.835 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T09:25:44.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:25:45.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:00.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:26:00.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.681+0000 7ff1380dd700 1 Processor -- start 2023-03-03T09:26:00.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.681+0000 7ff1380dd700 1 -- start start 2023-03-03T09:26:00.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff1380dd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff13006d870 0x7ff1301550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff1380dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1300c3e30 0x7ff1300c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff1380dd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1300c7a60 con 0x7ff1300c1da0 2023-03-03T09:26:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff1380dd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1300c2af0 con 0x7ff13006d870 2023-03-03T09:26:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff1380dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1300c3620 con 0x7ff1300c3e30 2023-03-03T09:26:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff135678700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff1300c1da0 0x7ff1300c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49964/0 (socket says 172.21.15.7:49964) 2023-03-03T09:26:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff135678700 1 -- 172.21.15.7:0/2028759917 learned_addr learned my addr 172.21.15.7:0/2028759917 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:26:00.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff135e79700 1 --2- 172.21.15.7:0/2028759917 >> v2:172.21.15.143:3300/0 conn(0x7ff13006d870 0x7ff1301550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:00.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff13667a700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1300c3e30 0x7ff1300c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:00.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277609755 0 0) 0x7ff1300c7a60 con 0x7ff1300c1da0 2023-03-03T09:26:00.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.682+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c0035b0 con 0x7ff1300c1da0 2023-03-03T09:26:00.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3027858462 0 0) 0x7ff11c0035b0 con 0x7ff1300c1da0 2023-03-03T09:26:00.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1300c3e30 msgr2=0x7ff1300c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:00.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1300c3e30 0x7ff1300c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:00.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 >> v2:172.21.15.143:3300/0 conn(0x7ff13006d870 msgr2=0x7ff1301550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:00.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 --2- 172.21.15.7:0/2028759917 >> v2:172.21.15.143:3300/0 conn(0x7ff13006d870 0x7ff1301550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:00.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1301545e0 con 0x7ff1300c1da0 2023-03-03T09:26:00.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff12c001670 con 0x7ff1300c1da0 2023-03-03T09:26:00.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 >> v1:172.21.15.7:6789/0 conn(0x7ff1300c1da0 legacy=0x7ff1300c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:00.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff11c003a70 0x7ff11c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:00.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1300c3e30 0x7ff11c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:00.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff11c004d90 0x7ff11c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:00.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1300c3620 con 0x7ff11c003a70 2023-03-03T09:26:00.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c00ba10 con 0x7ff11c004d90 2023-03-03T09:26:00.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c00bba0 con 0x7ff1300c3e30 2023-03-03T09:26:00.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff135e79700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff11c003a70 0x7ff11c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:00.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff135678700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1300c3e30 0x7ff11c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:00.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff13667a700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff11c004d90 0x7ff11c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:00.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff135e79700 1 -- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1300c3e30 msgr2=0x7ff11c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:00.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff135e79700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1300c3e30 0x7ff11c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:00.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff135e79700 1 -- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff11c004d90 msgr2=0x7ff11c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:00.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff135e79700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff11c004d90 0x7ff11c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:00.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff135e79700 1 -- 172.21.15.7:0/2028759917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c0035b0 con 0x7ff11c003a70 2023-03-03T09:26:00.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff135678700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff1300c3e30 0x7ff11c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:26:00.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff135e79700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff11c003a70 0x7ff11c003e80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff120009b90 tx=0x7ff11c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636f8e3d540a02df server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:00.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.683+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12000e6c0 con 0x7ff11c003a70 2023-03-03T09:26:00.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.684+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12000e820 con 0x7ff11c003a70 2023-03-03T09:26:00.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.684+0000 7ff134e77700 1 -- 172.21.15.7:0/2028759917 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12000eb90 con 0x7ff11c003a70 2023-03-03T09:26:00.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.684+0000 7ff1380dd700 1 -- 172.21.15.7:0/2028759917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff11c003a70 msgr2=0x7ff11c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:00.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.684+0000 7ff1380dd700 1 --2- 172.21.15.7:0/2028759917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff11c003a70 0x7ff11c003e80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff120009b90 tx=0x7ff11c005880 comp rx=0 tx=0).stop 2023-03-03T09:26:00.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.684+0000 7ff13667a700 1 -- 172.21.15.7:0/2028759917 reap_dead start 2023-03-03T09:26:00.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.684+0000 7ff1380dd700 1 -- 172.21.15.7:0/2028759917 shutdown_connections 2023-03-03T09:26:00.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.684+0000 7ff1380dd700 1 -- 172.21.15.7:0/2028759917 >> 172.21.15.7:0/2028759917 conn(0x7ff13006c110 msgr2=0x7ff13006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:26:00.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.684+0000 7ff1380dd700 1 -- 172.21.15.7:0/2028759917 shutdown_connections 2023-03-03T09:26:00.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.684+0000 7ff1380dd700 1 -- 172.21.15.7:0/2028759917 wait complete. 2023-03-03T09:26:00.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.685+0000 7ff1380dd700 1 Processor -- start 2023-03-03T09:26:00.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.685+0000 7ff1380dd700 1 -- start start 2023-03-03T09:26:00.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff1380dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff11c003a70 0x7ff1301ed7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:00.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff1380dd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1300acea0 0x7ff13014ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:00.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff1380dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff13014f340 0x7ff130150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:00.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff1380dd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1301ea380 con 0x7ff1300acea0 2023-03-03T09:26:00.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff1380dd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1301ea8b0 con 0x7ff13014f340 2023-03-03T09:26:00.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff1380dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1301ed310 con 0x7ff11c003a70 2023-03-03T09:26:00.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff135678700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1300acea0 0x7ff13014ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:00.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff135678700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1300acea0 0x7ff13014ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55996/0 (socket says 172.21.15.7:55996) 2023-03-03T09:26:00.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff135678700 1 -- 172.21.15.7:0/2201083454 learned_addr learned my addr 172.21.15.7:0/2201083454 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:26:00.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff135e79700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff11c003a70 0x7ff1301ed7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:00.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.686+0000 7ff13667a700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff13014f340 0x7ff130150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:00.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff135678700 1 -- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff11c003a70 msgr2=0x7ff1301ed7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:00.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff135678700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff11c003a70 0x7ff1301ed7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:00.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff135678700 1 -- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff13014f340 msgr2=0x7ff130150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:00.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff135678700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff13014f340 0x7ff130150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:00.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff135678700 1 -- 172.21.15.7:0/2201083454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff130154ba0 con 0x7ff1300acea0 2023-03-03T09:26:00.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff135678700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1300acea0 0x7ff13014ee00 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00e800 tx=0x7ff12c00eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:00.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff1267fc700 1 -- 172.21.15.7:0/2201083454 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c012590 con 0x7ff1300acea0 2023-03-03T09:26:00.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff1267fc700 1 -- 172.21.15.7:0/2201083454 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12c0126f0 con 0x7ff1300acea0 2023-03-03T09:26:00.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff1380dd700 1 -- 172.21.15.7:0/2201083454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1301548c0 con 0x7ff1300acea0 2023-03-03T09:26:00.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff1267fc700 1 -- 172.21.15.7:0/2201083454 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c012aa0 con 0x7ff1300acea0 2023-03-03T09:26:00.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.687+0000 7ff1380dd700 1 -- 172.21.15.7:0/2201083454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13006dc40 con 0x7ff1300acea0 2023-03-03T09:26:00.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.688+0000 7ff1380dd700 1 -- 172.21.15.7:0/2201083454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13014f7c0 con 0x7ff1300acea0 2023-03-03T09:26:00.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.688+0000 7ff1267fc700 1 -- 172.21.15.7:0/2201083454 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff12c012c00 con 0x7ff1300acea0 2023-03-03T09:26:00.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.689+0000 7ff1267fc700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff1140682f0 0x7ff11406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:00.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.689+0000 7ff1267fc700 1 -- 172.21.15.7:0/2201083454 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff12c085f30 con 0x7ff1300acea0 2023-03-03T09:26:00.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.689+0000 7ff135e79700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff1140682f0 0x7ff11406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:00.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.690+0000 7ff135e79700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff1140682f0 0x7ff11406a7a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff1200090b0 tx=0x7ff12001f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:00.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.693+0000 7ff1267fc700 1 -- 172.21.15.7:0/2201083454 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff12c0522d0 con 0x7ff1300acea0 2023-03-03T09:26:00.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.938+0000 7ff1380dd700 1 -- 172.21.15.7:0/2201083454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1301ed450 con 0x7ff1300acea0 2023-03-03T09:26:00.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.939+0000 7ff1267fc700 1 -- 172.21.15.7:0/2201083454 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff12c0558f0 con 0x7ff1300acea0 2023-03-03T09:26:00.940 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:26:00.941 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:26:00.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 -- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff1140682f0 msgr2=0x7ff11406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:00.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff1140682f0 0x7ff11406a7a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff1200090b0 tx=0x7ff12001f040 comp rx=0 tx=0).stop 2023-03-03T09:26:00.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 -- 172.21.15.7:0/2201083454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1300acea0 msgr2=0x7ff13014ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:00.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1300acea0 0x7ff13014ee00 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00e800 tx=0x7ff12c00eb10 comp rx=0 tx=0).stop 2023-03-03T09:26:00.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 -- 172.21.15.7:0/2201083454 shutdown_connections 2023-03-03T09:26:00.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff13014f340 0x7ff130150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:00.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff11c003a70 0x7ff1301ed7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:00.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff1140682f0 0x7ff11406a7a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:00.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 --2- 172.21.15.7:0/2201083454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1300acea0 0x7ff13014ee00 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:00.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 -- 172.21.15.7:0/2201083454 >> 172.21.15.7:0/2201083454 conn(0x7ff13006c110 msgr2=0x7ff130071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:26:00.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 -- 172.21.15.7:0/2201083454 shutdown_connections 2023-03-03T09:26:00.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:00.941+0000 7ff1380dd700 1 -- 172.21.15.7:0/2201083454 wait complete. 2023-03-03T09:26:00.944 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:26:00.954 INFO:tasks.mds_thrash.fs.[b]:waiting for 37 secs before thrashing 2023-03-03T09:26:15.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:15.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:24.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:26:24.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.974+0000 7fceda71a700 1 Processor -- start 2023-03-03T09:26:24.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.974+0000 7fceda71a700 1 -- start start 2023-03-03T09:26:24.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.975+0000 7fceda71a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fced413a0f0 0x7fced40c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:24.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.975+0000 7fceda71a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced41586c0 0x7fced406d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:24.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.975+0000 7fceda71a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced406de90 con 0x7fced40c1da0 2023-03-03T09:26:24.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.975+0000 7fceda71a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced4154840 con 0x7fced413a0f0 2023-03-03T09:26:24.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.975+0000 7fceda71a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced4154560 con 0x7fced41586c0 2023-03-03T09:26:24.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.975+0000 7fced3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fced413a0f0 0x7fced40c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:24.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.975+0000 7fced3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fced413a0f0 0x7fced40c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:51810/0 (socket says 172.21.15.7:51810) 2023-03-03T09:26:24.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.975+0000 7fced8cb7700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fced40c1da0 0x7fced406d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:53540/0 (socket says 172.21.15.7:53540) 2023-03-03T09:26:24.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.975+0000 7fced3fff700 1 -- 172.21.15.7:0/1727569898 learned_addr learned my addr 172.21.15.7:0/1727569898 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:26:24.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.975+0000 7fced37fe700 1 --2- 172.21.15.7:0/1727569898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced41586c0 0x7fced406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:24.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.976+0000 7fced3fff700 1 -- 172.21.15.7:0/1727569898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced41586c0 msgr2=0x7fced406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:24.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.976+0000 7fced3fff700 1 --2- 172.21.15.7:0/1727569898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced41586c0 0x7fced406d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:24.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.976+0000 7fced3fff700 1 -- 172.21.15.7:0/1727569898 >> v1:172.21.15.143:6789/0 conn(0x7fced40c1da0 legacy=0x7fced406d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:24.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.976+0000 7fced3fff700 1 -- 172.21.15.7:0/1727569898 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced4157be0 con 0x7fced413a0f0 2023-03-03T09:26:24.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.976+0000 7fced2ffd700 1 -- 172.21.15.7:0/1727569898 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413830155 0 0) 0x7fcecc001490 con 0x7fced40c1da0 2023-03-03T09:26:24.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.976+0000 7fced37fe700 1 --2- 172.21.15.7:0/1727569898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced41586c0 0x7fced406d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:26:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.976+0000 7fced3fff700 1 --2- 172.21.15.7:0/1727569898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fced413a0f0 0x7fced40c07d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcec40098f0 tx=0x7fcec4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f05df2e58e776f88 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.976+0000 7fced2ffd700 1 -- 172.21.15.7:0/1727569898 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4011830 con 0x7fced413a0f0 2023-03-03T09:26:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.976+0000 7fced2ffd700 1 -- 172.21.15.7:0/1727569898 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec4011990 con 0x7fced413a0f0 2023-03-03T09:26:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.976+0000 7fced2ffd700 1 -- 172.21.15.7:0/1727569898 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4011d20 con 0x7fced413a0f0 2023-03-03T09:26:24.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.977+0000 7fceda71a700 1 -- 172.21.15.7:0/1727569898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fced413a0f0 msgr2=0x7fced40c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:24.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.977+0000 7fceda71a700 1 --2- 172.21.15.7:0/1727569898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fced413a0f0 0x7fced40c07d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcec40098f0 tx=0x7fcec4009c00 comp rx=0 tx=0).stop 2023-03-03T09:26:24.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.977+0000 7fceda71a700 1 -- 172.21.15.7:0/1727569898 shutdown_connections 2023-03-03T09:26:24.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.977+0000 7fceda71a700 1 --2- 172.21.15.7:0/1727569898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fced413a0f0 0x7fced40c07d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:24.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.977+0000 7fceda71a700 1 --2- 172.21.15.7:0/1727569898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced41586c0 0x7fced406d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:24.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.977+0000 7fceda71a700 1 -- 172.21.15.7:0/1727569898 >> 172.21.15.7:0/1727569898 conn(0x7fced406c110 msgr2=0x7fced406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:26:24.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.977+0000 7fceda71a700 1 -- 172.21.15.7:0/1727569898 shutdown_connections 2023-03-03T09:26:24.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.977+0000 7fceda71a700 1 -- 172.21.15.7:0/1727569898 wait complete. 2023-03-03T09:26:24.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.978+0000 7fceda71a700 1 Processor -- start 2023-03-03T09:26:24.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.978+0000 7fceda71a700 1 -- start start 2023-03-03T09:26:24.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fceda71a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced40c1da0 0x7fced40ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:24.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fceda71a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fced413a0f0 0x7fced4070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:24.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fceda71a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced41586c0 0x7fced41ed750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:24.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fceda71a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced4070ba0 con 0x7fced41586c0 2023-03-03T09:26:24.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fceda71a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced41ed2c0 con 0x7fced413a0f0 2023-03-03T09:26:24.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fceda71a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced4155e60 con 0x7fced40c1da0 2023-03-03T09:26:24.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fced8cb7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced41586c0 0x7fced41ed750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:24.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fced8cb7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced41586c0 0x7fced41ed750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56046/0 (socket says 172.21.15.7:56046) 2023-03-03T09:26:24.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fced8cb7700 1 -- 172.21.15.7:0/3944234835 learned_addr learned my addr 172.21.15.7:0/3944234835 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:26:24.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fced37fe700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fced413a0f0 0x7fced4070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:24.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fced3fff700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced40c1da0 0x7fced40ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:24.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fced8cb7700 1 -- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced40c1da0 msgr2=0x7fced40ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:24.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fced8cb7700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced40c1da0 0x7fced40ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:24.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fced8cb7700 1 -- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fced413a0f0 msgr2=0x7fced4070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:24.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fced8cb7700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fced413a0f0 0x7fced4070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:24.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.979+0000 7fced8cb7700 1 -- 172.21.15.7:0/3944234835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec80094a0 con 0x7fced41586c0 2023-03-03T09:26:24.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.980+0000 7fced8cb7700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced41586c0 0x7fced41ed750 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00cb60 tx=0x7fcecc00ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:24.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.980+0000 7fced0ff9700 1 -- 172.21.15.7:0/3944234835 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc00a9a0 con 0x7fced41586c0 2023-03-03T09:26:24.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.980+0000 7fced0ff9700 1 -- 172.21.15.7:0/3944234835 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc00ab00 con 0x7fced41586c0 2023-03-03T09:26:24.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.980+0000 7fceda71a700 1 -- 172.21.15.7:0/3944234835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec4009550 con 0x7fced41586c0 2023-03-03T09:26:24.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.980+0000 7fced0ff9700 1 -- 172.21.15.7:0/3944234835 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc0070d0 con 0x7fced41586c0 2023-03-03T09:26:24.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.980+0000 7fceda71a700 1 -- 172.21.15.7:0/3944234835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced41581a0 con 0x7fced41586c0 2023-03-03T09:26:24.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.981+0000 7fceda71a700 1 -- 172.21.15.7:0/3944234835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced4157ec0 con 0x7fced41586c0 2023-03-03T09:26:24.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.981+0000 7fced0ff9700 1 -- 172.21.15.7:0/3944234835 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcecc01a030 con 0x7fced41586c0 2023-03-03T09:26:24.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.982+0000 7fced0ff9700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcec00682f0 0x7fcec006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:24.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.982+0000 7fced0ff9700 1 -- 172.21.15.7:0/3944234835 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcecc085af0 con 0x7fced41586c0 2023-03-03T09:26:24.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.982+0000 7fced3fff700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcec00682f0 0x7fcec006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.982+0000 7fced3fff700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcec00682f0 0x7fcec006a7a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcec4000c00 tx=0x7fcec4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:24.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:24.986+0000 7fced0ff9700 1 -- 172.21.15.7:0/3944234835 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcecc051e90 con 0x7fced41586c0 2023-03-03T09:26:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.224+0000 7fceda71a700 1 -- 172.21.15.7:0/3944234835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fced406e1a0 con 0x7fced41586c0 2023-03-03T09:26:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.224+0000 7fced0ff9700 1 -- 172.21.15.7:0/3944234835 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcecc0554b0 con 0x7fced41586c0 2023-03-03T09:26:25.226 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:26:25.226 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:26:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 -- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcec00682f0 msgr2=0x7fcec006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcec00682f0 0x7fcec006a7a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcec4000c00 tx=0x7fcec4008030 comp rx=0 tx=0).stop 2023-03-03T09:26:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 -- 172.21.15.7:0/3944234835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced41586c0 msgr2=0x7fced41ed750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced41586c0 0x7fced41ed750 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00cb60 tx=0x7fcecc00ce70 comp rx=0 tx=0).stop 2023-03-03T09:26:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 -- 172.21.15.7:0/3944234835 shutdown_connections 2023-03-03T09:26:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcec00682f0 0x7fcec006a7a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fced41586c0 0x7fced41ed750 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fced40c1da0 0x7fced40ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 --2- 172.21.15.7:0/3944234835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fced413a0f0 0x7fced4070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 -- 172.21.15.7:0/3944234835 >> 172.21.15.7:0/3944234835 conn(0x7fced406c110 msgr2=0x7fced4138490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:26:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 -- 172.21.15.7:0/3944234835 shutdown_connections 2023-03-03T09:26:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:25.226+0000 7fceda71a700 1 -- 172.21.15.7:0/3944234835 wait complete. 2023-03-03T09:26:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:26:25.239 INFO:tasks.mds_thrash.fs.[a]:waiting for 39 secs before thrashing 2023-03-03T09:26:37.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:26:38.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.093+0000 7f490aa62700 1 Processor -- start 2023-03-03T09:26:38.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.093+0000 7f490aa62700 1 -- start start 2023-03-03T09:26:38.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.094+0000 7f490aa62700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f49040caa30 0x7f49040c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:38.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.094+0000 7f490aa62700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4904154ff0 0x7f49040cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:38.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.094+0000 7f490aa62700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49040c7310 con 0x7f49040c00e0 2023-03-03T09:26:38.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.094+0000 7f490aa62700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49040bfd10 con 0x7f49040caa30 2023-03-03T09:26:38.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.094+0000 7f490aa62700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49040bfa30 con 0x7f4904154ff0 2023-03-03T09:26:38.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.094+0000 7f49037fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f49040c00e0 0x7f49040c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50034/0 (socket says 172.21.15.7:50034) 2023-03-03T09:26:38.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.094+0000 7f49037fe700 1 -- 172.21.15.7:0/3223845775 learned_addr learned my addr 172.21.15.7:0/3223845775 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:26:38.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4908fff700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4904154ff0 0x7f49040cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:38.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260525058 0 0) 0x7f49040c7310 con 0x7f49040c00e0 2023-03-03T09:26:38.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4903fff700 1 --2- 172.21.15.7:0/3223845775 >> v2:172.21.15.143:3300/0 conn(0x7f49040caa30 0x7f49040c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:38.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ec0035b0 con 0x7f49040c00e0 2023-03-03T09:26:38.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 74621331 0 0) 0x7f48ec0035b0 con 0x7f49040c00e0 2023-03-03T09:26:38.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4904154ff0 msgr2=0x7f49040cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:38.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4904154ff0 0x7f49040cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 >> v2:172.21.15.143:3300/0 conn(0x7f49040caa30 msgr2=0x7f49040c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:38.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 --2- 172.21.15.7:0/3223845775 >> v2:172.21.15.143:3300/0 conn(0x7f49040caa30 0x7f49040c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4904154510 con 0x7f49040c00e0 2023-03-03T09:26:38.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f48f80017f0 con 0x7f49040c00e0 2023-03-03T09:26:38.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 >> v1:172.21.15.7:6789/0 conn(0x7f49040c00e0 legacy=0x7f49040c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:38.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4904154ff0 0x7f48ec003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:38.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48ec003ff0 0x7f48ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:38.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec004d90 0x7f48ec005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:38.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49040bfd10 con 0x7f48ec003ff0 2023-03-03T09:26:38.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49040bfa30 con 0x7f48ec004d90 2023-03-03T09:26:38.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec00bac0 con 0x7f4904154ff0 2023-03-03T09:26:38.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f49037fe700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48ec003ff0 0x7f48ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:38.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4903fff700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4904154ff0 0x7f48ec003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:38.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f4908fff700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec004d90 0x7f48ec005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:38.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f49037fe700 1 -- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4904154ff0 msgr2=0x7f48ec003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:38.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f49037fe700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4904154ff0 0x7f48ec003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f49037fe700 1 -- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec004d90 msgr2=0x7f48ec005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:26:38.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f49037fe700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec004d90 0x7f48ec005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.095+0000 7f49037fe700 1 -- 172.21.15.7:0/3223845775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec0035b0 con 0x7f48ec003ff0 2023-03-03T09:26:38.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.096+0000 7f49037fe700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48ec003ff0 0x7f48ec004850 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f48f800c2f0 tx=0x7f48ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c3d3b5c1f1b9ec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:38.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.096+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f800ce30 con 0x7f48ec003ff0 2023-03-03T09:26:38.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.096+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f8007bf0 con 0x7f48ec003ff0 2023-03-03T09:26:38.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.096+0000 7f4902ffd700 1 -- 172.21.15.7:0/3223845775 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8007310 con 0x7f48ec003ff0 2023-03-03T09:26:38.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.096+0000 7f490aa62700 1 -- 172.21.15.7:0/3223845775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48ec003ff0 msgr2=0x7f48ec004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:38.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.096+0000 7f490aa62700 1 --2- 172.21.15.7:0/3223845775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48ec003ff0 0x7f48ec004850 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f48f800c2f0 tx=0x7f48ec005880 comp rx=0 tx=0).stop 2023-03-03T09:26:38.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.096+0000 7f4908fff700 1 -- 172.21.15.7:0/3223845775 reap_dead start 2023-03-03T09:26:38.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.096+0000 7f490aa62700 1 -- 172.21.15.7:0/3223845775 shutdown_connections 2023-03-03T09:26:38.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.096+0000 7f490aa62700 1 -- 172.21.15.7:0/3223845775 >> 172.21.15.7:0/3223845775 conn(0x7f4904066e40 msgr2=0x7f4904071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:26:38.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.096+0000 7f490aa62700 1 -- 172.21.15.7:0/3223845775 shutdown_connections 2023-03-03T09:26:38.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.097+0000 7f490aa62700 1 -- 172.21.15.7:0/3223845775 wait complete. 2023-03-03T09:26:38.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.098+0000 7f490aa62700 1 Processor -- start 2023-03-03T09:26:38.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.098+0000 7f490aa62700 1 -- start start 2023-03-03T09:26:38.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f490aa62700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec003ff0 0x7f49041edbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:38.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f490aa62700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49040c8e70 0x7f49040c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:38.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f490aa62700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49040c3720 0x7f490414cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:38.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f490aa62700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4904152de0 con 0x7f49040c8e70 2023-03-03T09:26:38.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f490aa62700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49041ed720 con 0x7f48ec003ff0 2023-03-03T09:26:38.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f490aa62700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49041ddbe0 con 0x7f49040c3720 2023-03-03T09:26:38.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f49037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49040c8e70 0x7f49040c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:38.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f49037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49040c8e70 0x7f49040c31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56068/0 (socket says 172.21.15.7:56068) 2023-03-03T09:26:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f49037fe700 1 -- 172.21.15.7:0/2607717427 learned_addr learned my addr 172.21.15.7:0/2607717427 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:26:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f4903fff700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec003ff0 0x7f49041edbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f4908fff700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49040c3720 0x7f490414cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f49037fe700 1 -- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49040c3720 msgr2=0x7f490414cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f49037fe700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49040c3720 0x7f490414cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f49037fe700 1 -- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec003ff0 msgr2=0x7f49041edbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f49037fe700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec003ff0 0x7f49041edbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f49037fe700 1 -- 172.21.15.7:0/2607717427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4904154ad0 con 0x7f49040c8e70 2023-03-03T09:26:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f4903fff700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec003ff0 0x7f49041edbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:26:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f49037fe700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49040c8e70 0x7f49040c31e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f48f8000ef0 tx=0x7f48f8001bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f4900ff9700 1 -- 172.21.15.7:0/2607717427 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8005dc0 con 0x7f49040c8e70 2023-03-03T09:26:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f4900ff9700 1 -- 172.21.15.7:0/2607717427 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f80070d0 con 0x7f49040c8e70 2023-03-03T09:26:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f4900ff9700 1 -- 172.21.15.7:0/2607717427 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8007480 con 0x7f49040c8e70 2023-03-03T09:26:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.099+0000 7f490aa62700 1 -- 172.21.15.7:0/2607717427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49041547f0 con 0x7f49040c8e70 2023-03-03T09:26:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.100+0000 7f490aa62700 1 -- 172.21.15.7:0/2607717427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49040bf650 con 0x7f49040c8e70 2023-03-03T09:26:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.100+0000 7f490aa62700 1 -- 172.21.15.7:0/2607717427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4904074bb0 con 0x7f49040c8e70 2023-03-03T09:26:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.101+0000 7f4900ff9700 1 -- 172.21.15.7:0/2607717427 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f48f802e070 con 0x7f49040c8e70 2023-03-03T09:26:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.101+0000 7f4900ff9700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48f00682f0 0x7f48f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:26:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.101+0000 7f4900ff9700 1 -- 172.21.15.7:0/2607717427 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f48f8087530 con 0x7f49040c8e70 2023-03-03T09:26:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.102+0000 7f4903fff700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48f00682f0 0x7f48f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:26:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.102+0000 7f4903fff700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48f00682f0 0x7f48f006a7a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f49041525e0 tx=0x7f48f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:26:38.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.106+0000 7f4900ff9700 1 -- 172.21.15.7:0/2607717427 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f48f80538d0 con 0x7f49040c8e70 2023-03-03T09:26:38.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.344+0000 7f490aa62700 1 -- 172.21.15.7:0/2607717427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4904152de0 con 0x7f49040c8e70 2023-03-03T09:26:38.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.345+0000 7f4900ff9700 1 -- 172.21.15.7:0/2607717427 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f48f8056ef0 con 0x7f49040c8e70 2023-03-03T09:26:38.347 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:26:38.347 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:26:38.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 -- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48f00682f0 msgr2=0x7f48f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:38.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48f00682f0 0x7f48f006a7a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f49041525e0 tx=0x7f48f4008040 comp rx=0 tx=0).stop 2023-03-03T09:26:38.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 -- 172.21.15.7:0/2607717427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49040c8e70 msgr2=0x7f49040c31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:26:38.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49040c8e70 0x7f49040c31e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f48f8000ef0 tx=0x7f48f8001bf0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 -- 172.21.15.7:0/2607717427 shutdown_connections 2023-03-03T09:26:38.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49040c3720 0x7f490414cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48ec003ff0 0x7f49041edbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48f00682f0 0x7f48f006a7a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 --2- 172.21.15.7:0/2607717427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49040c8e70 0x7f49040c31e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:26:38.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 -- 172.21.15.7:0/2607717427 >> 172.21.15.7:0/2607717427 conn(0x7f4904066e40 msgr2=0x7f49040c7740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:26:38.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 -- 172.21.15.7:0/2607717427 shutdown_connections 2023-03-03T09:26:38.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:26:38.347+0000 7f490aa62700 1 -- 172.21.15.7:0/2607717427 wait complete. 2023-03-03T09:26:38.350 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:26:38.360 INFO:tasks.mds_thrash.fs.[b]:waiting for 52 secs before thrashing 2023-03-03T09:26:45.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:26:45.179 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:04.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:27:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.378+0000 7f003d6db700 1 Processor -- start 2023-03-03T09:27:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.378+0000 7f003d6db700 1 -- start start 2023-03-03T09:27:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.379+0000 7f003d6db700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00380c5c60 0x7f0038154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.379+0000 7f003d6db700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00380c7310 con 0x7f00380caa30 2023-03-03T09:27:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.379+0000 7f003d6db700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00380bfd10 con 0x7f00380c00e0 2023-03-03T09:27:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.379+0000 7f003d6db700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00380c60a0 con 0x7f00380c5c60 2023-03-03T09:27:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.379+0000 7f00377fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00380c5c60 0x7f0038154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.379+0000 7f0036ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f00380caa30 0x7f00380c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50072/0 (socket says 172.21.15.7:50072) 2023-03-03T09:27:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.379+0000 7f0036ffd700 1 -- 172.21.15.7:0/3950169929 learned_addr learned my addr 172.21.15.7:0/3950169929 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:27:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.379+0000 7f00377fe700 1 -- 172.21.15.7:0/3950169929 >> v1:172.21.15.143:6789/0 conn(0x7f00380c00e0 legacy=0x7f003815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.379+0000 7f00377fe700 1 -- 172.21.15.7:0/3950169929 >> v1:172.21.15.7:6789/0 conn(0x7f00380caa30 legacy=0x7f00380c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.379+0000 7f00377fe700 1 -- 172.21.15.7:0/3950169929 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00380bfa30 con 0x7f00380c5c60 2023-03-03T09:27:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f00377fe700 1 --2- 172.21.15.7:0/3950169929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 0x7f0038154ff0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0030009d90 tx=0x7f003000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e80d331dd8b924 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:27:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f0035ffb700 1 -- 172.21.15.7:0/3950169929 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003000e070 con 0x7f00380c5c60 2023-03-03T09:27:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f0035ffb700 1 -- 172.21.15.7:0/3950169929 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003000bbd0 con 0x7f00380c5c60 2023-03-03T09:27:04.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f0035ffb700 1 -- 172.21.15.7:0/3950169929 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030012430 con 0x7f00380c5c60 2023-03-03T09:27:04.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f003d6db700 1 -- 172.21.15.7:0/3950169929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 msgr2=0x7f0038154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:04.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f003d6db700 1 --2- 172.21.15.7:0/3950169929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 0x7f0038154ff0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0030009d90 tx=0x7f003000b330 comp rx=0 tx=0).stop 2023-03-03T09:27:04.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f003d6db700 1 -- 172.21.15.7:0/3950169929 shutdown_connections 2023-03-03T09:27:04.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f003d6db700 1 --2- 172.21.15.7:0/3950169929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 0x7f0038154ff0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:04.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f003d6db700 1 -- 172.21.15.7:0/3950169929 >> 172.21.15.7:0/3950169929 conn(0x7f0038066e40 msgr2=0x7f0038071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:27:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f003d6db700 1 -- 172.21.15.7:0/3950169929 shutdown_connections 2023-03-03T09:27:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.380+0000 7f003d6db700 1 -- 172.21.15.7:0/3950169929 wait complete. 2023-03-03T09:27:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f003d6db700 1 Processor -- start 2023-03-03T09:27:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f003d6db700 1 -- start start 2023-03-03T09:27:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f003d6db700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00380c00e0 0x7f00381ed660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f003d6db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 0x7f00381eb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f003d6db700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00380caa30 0x7f00381ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f003d6db700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00380c1b00 con 0x7f00380c5c60 2023-03-03T09:27:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f003d6db700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00381ee150 con 0x7f00380caa30 2023-03-03T09:27:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f003d6db700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00380c1d10 con 0x7f00380c00e0 2023-03-03T09:27:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00367fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 0x7f00381eb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00367fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 0x7f00381eb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56098/0 (socket says 172.21.15.7:56098) 2023-03-03T09:27:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00367fc700 1 -- 172.21.15.7:0/1846648392 learned_addr learned my addr 172.21.15.7:0/1846648392 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:27:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00367fc700 1 -- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00380c00e0 msgr2=0x7f00381ed660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f0036ffd700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00380c00e0 0x7f00381ed660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00377fe700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00380caa30 0x7f00381ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00367fc700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00380c00e0 0x7f00381ed660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00367fc700 1 -- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00380caa30 msgr2=0x7f00381ebba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00367fc700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00380caa30 0x7f00381ebba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00367fc700 1 -- 172.21.15.7:0/1846648392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0030009a70 con 0x7f00380c5c60 2023-03-03T09:27:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00377fe700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00380caa30 0x7f00381ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:27:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f0036ffd700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00380c00e0 0x7f00381ed660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:27:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f00367fc700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 0x7f00381eb660 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f002800c4d0 tx=0x7f002800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:27:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f001f7fe700 1 -- 172.21.15.7:0/1846648392 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00280052d0 con 0x7f00380c5c60 2023-03-03T09:27:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f001f7fe700 1 -- 172.21.15.7:0/1846648392 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0028005430 con 0x7f00380c5c60 2023-03-03T09:27:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f003d6db700 1 -- 172.21.15.7:0/1846648392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00381547f0 con 0x7f00380c5c60 2023-03-03T09:27:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f001f7fe700 1 -- 172.21.15.7:0/1846648392 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00280038f0 con 0x7f00380c5c60 2023-03-03T09:27:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.383+0000 7f003d6db700 1 -- 172.21.15.7:0/1846648392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0038154510 con 0x7f00380c5c60 2023-03-03T09:27:04.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.384+0000 7f003d6db700 1 -- 172.21.15.7:0/1846648392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0038154ad0 con 0x7f00380c5c60 2023-03-03T09:27:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.385+0000 7f001f7fe700 1 -- 172.21.15.7:0/1846648392 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0028019030 con 0x7f00380c5c60 2023-03-03T09:27:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.385+0000 7f001f7fe700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00240682f0 0x7f002406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.386+0000 7f001f7fe700 1 -- 172.21.15.7:0/1846648392 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0028085a00 con 0x7f00380c5c60 2023-03-03T09:27:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.386+0000 7f0036ffd700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00240682f0 0x7f002406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.386+0000 7f0036ffd700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00240682f0 0x7f002406a7a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f002c005910 tx=0x7f002c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:27:04.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.390+0000 7f001f7fe700 1 -- 172.21.15.7:0/1846648392 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0028051d40 con 0x7f00380c5c60 2023-03-03T09:27:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.627+0000 7f003d6db700 1 -- 172.21.15.7:0/1846648392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0038074bb0 con 0x7f00380c5c60 2023-03-03T09:27:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.627+0000 7f001f7fe700 1 -- 172.21.15.7:0/1846648392 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0028055360 con 0x7f00380c5c60 2023-03-03T09:27:04.630 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:27:04.630 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:27:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 -- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00240682f0 msgr2=0x7f002406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00240682f0 0x7f002406a7a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f002c005910 tx=0x7f002c009040 comp rx=0 tx=0).stop 2023-03-03T09:27:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 -- 172.21.15.7:0/1846648392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 msgr2=0x7f00381eb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 0x7f00381eb660 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f002800c4d0 tx=0x7f002800c890 comp rx=0 tx=0).stop 2023-03-03T09:27:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 -- 172.21.15.7:0/1846648392 shutdown_connections 2023-03-03T09:27:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00380caa30 0x7f00381ebba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00380c00e0 0x7f00381ed660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00240682f0 0x7f002406a7a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 --2- 172.21.15.7:0/1846648392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00380c5c60 0x7f00381eb660 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 -- 172.21.15.7:0/1846648392 >> 172.21.15.7:0/1846648392 conn(0x7f0038066e40 msgr2=0x7f00380c8920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:27:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 -- 172.21.15.7:0/1846648392 shutdown_connections 2023-03-03T09:27:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:04.629+0000 7f003d6db700 1 -- 172.21.15.7:0/1846648392 wait complete. 2023-03-03T09:27:04.634 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:27:04.643 INFO:tasks.mds_thrash.fs.[a]:waiting for 33 secs before thrashing 2023-03-03T09:27:15.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:15.247 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:30.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:27:30.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.499+0000 7f411b634700 1 Processor -- start 2023-03-03T09:27:30.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.499+0000 7f411b634700 1 -- start start 2023-03-03T09:27:30.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f411b634700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f411406e0c0 0x7f4114157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:30.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f411b634700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41140728c0 con 0x7f41140c07d0 2023-03-03T09:27:30.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f411b634700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4114155140 con 0x7f4114151e30 2023-03-03T09:27:30.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f411b634700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41140c1aa0 con 0x7f411406e0c0 2023-03-03T09:27:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f4119bd1700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f41140c07d0 0x7f4114158430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50116/0 (socket says 172.21.15.7:50116) 2023-03-03T09:27:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f4119bd1700 1 -- 172.21.15.7:0/517283778 learned_addr learned my addr 172.21.15.7:0/517283778 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:27:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f4118bcf700 1 --2- 172.21.15.7:0/517283778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f411406e0c0 0x7f4114157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f410bfff700 1 -- 172.21.15.7:0/517283778 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619565421 0 0) 0x7f41140728c0 con 0x7f41140c07d0 2023-03-03T09:27:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f410bfff700 1 -- 172.21.15.7:0/517283778 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41000035b0 con 0x7f41140c07d0 2023-03-03T09:27:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f410bfff700 1 -- 172.21.15.7:0/517283778 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590813659 0 0) 0x7f4114155140 con 0x7f4114151e30 2023-03-03T09:27:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.500+0000 7f410bfff700 1 -- 172.21.15.7:0/517283778 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41140728c0 con 0x7f4114151e30 2023-03-03T09:27:30.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.501+0000 7f4118bcf700 1 -- 172.21.15.7:0/517283778 >> v1:172.21.15.143:6789/0 conn(0x7f4114151e30 legacy=0x7f41140736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:30.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.501+0000 7f4118bcf700 1 -- 172.21.15.7:0/517283778 >> v1:172.21.15.7:6789/0 conn(0x7f41140c07d0 legacy=0x7f4114158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:30.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.501+0000 7f4118bcf700 1 -- 172.21.15.7:0/517283778 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4114155670 con 0x7f411406e0c0 2023-03-03T09:27:30.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.501+0000 7f4118bcf700 1 --2- 172.21.15.7:0/517283778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f411406e0c0 0x7f4114157ef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4104009870 tx=0x7f4104009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc97454dac34654f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:27:30.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.501+0000 7f410bfff700 1 -- 172.21.15.7:0/517283778 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104011830 con 0x7f411406e0c0 2023-03-03T09:27:30.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.501+0000 7f410bfff700 1 -- 172.21.15.7:0/517283778 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104011990 con 0x7f411406e0c0 2023-03-03T09:27:30.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.501+0000 7f410bfff700 1 -- 172.21.15.7:0/517283778 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104011d20 con 0x7f411406e0c0 2023-03-03T09:27:30.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.502+0000 7f411b634700 1 -- 172.21.15.7:0/517283778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f411406e0c0 msgr2=0x7f4114157ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:30.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.502+0000 7f411b634700 1 --2- 172.21.15.7:0/517283778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f411406e0c0 0x7f4114157ef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4104009870 tx=0x7f4104009b80 comp rx=0 tx=0).stop 2023-03-03T09:27:30.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.502+0000 7f411b634700 1 -- 172.21.15.7:0/517283778 shutdown_connections 2023-03-03T09:27:30.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.502+0000 7f411b634700 1 --2- 172.21.15.7:0/517283778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f411406e0c0 0x7f4114157ef0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:30.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.502+0000 7f411b634700 1 -- 172.21.15.7:0/517283778 >> 172.21.15.7:0/517283778 conn(0x7f411406c1c0 msgr2=0x7f411406c590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:27:30.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.502+0000 7f411b634700 1 -- 172.21.15.7:0/517283778 shutdown_connections 2023-03-03T09:27:30.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.502+0000 7f411b634700 1 -- 172.21.15.7:0/517283778 wait complete. 2023-03-03T09:27:30.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.503+0000 7f411b634700 1 Processor -- start 2023-03-03T09:27:30.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.503+0000 7f411b634700 1 -- start start 2023-03-03T09:27:30.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f411b634700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f411406e0c0 0x7f41140aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f411b634700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41140c07d0 0x7f4114150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f411b634700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4114151e30 0x7f41141ed8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f411b634700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41141ed4d0 con 0x7f41140c07d0 2023-03-03T09:27:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f411b634700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41141e7980 con 0x7f411406e0c0 2023-03-03T09:27:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f411b634700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4114157350 con 0x7f4114151e30 2023-03-03T09:27:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4118bcf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41140c07d0 0x7f4114150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4118bcf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41140c07d0 0x7f4114150880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56140/0 (socket says 172.21.15.7:56140) 2023-03-03T09:27:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4118bcf700 1 -- 172.21.15.7:0/1973719697 learned_addr learned my addr 172.21.15.7:0/1973719697 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:27:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f41193d0700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f411406e0c0 0x7f41140aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4118bcf700 1 -- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4114151e30 msgr2=0x7f41141ed8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4118bcf700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4114151e30 0x7f41141ed8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4118bcf700 1 -- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f411406e0c0 msgr2=0x7f41140aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4118bcf700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f411406e0c0 0x7f41140aa450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4118bcf700 1 -- 172.21.15.7:0/1973719697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41040094a0 con 0x7f41140c07d0 2023-03-03T09:27:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4118bcf700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41140c07d0 0x7f4114150880 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f41040090b0 tx=0x7f4104004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:27:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4109ffb700 1 -- 172.21.15.7:0/1973719697 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410401aa70 con 0x7f41140c07d0 2023-03-03T09:27:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4109ffb700 1 -- 172.21.15.7:0/1973719697 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410401abd0 con 0x7f41140c07d0 2023-03-03T09:27:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f411b634700 1 -- 172.21.15.7:0/1973719697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4114155390 con 0x7f41140c07d0 2023-03-03T09:27:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.504+0000 7f4109ffb700 1 -- 172.21.15.7:0/1973719697 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040243f0 con 0x7f41140c07d0 2023-03-03T09:27:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.505+0000 7f411b634700 1 -- 172.21.15.7:0/1973719697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41140c1490 con 0x7f41140c07d0 2023-03-03T09:27:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.506+0000 7f411b634700 1 -- 172.21.15.7:0/1973719697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41140c1730 con 0x7f41140c07d0 2023-03-03T09:27:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.506+0000 7f4109ffb700 1 -- 172.21.15.7:0/1973719697 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f410400f050 con 0x7f41140c07d0 2023-03-03T09:27:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.507+0000 7f4109ffb700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40f8068320 0x7f40f806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.507+0000 7f4109ffb700 1 -- 172.21.15.7:0/1973719697 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4104087310 con 0x7f41140c07d0 2023-03-03T09:27:30.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.507+0000 7f41193d0700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40f8068320 0x7f40f806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.507+0000 7f41193d0700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40f8068320 0x7f40f806a7d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4110001230 tx=0x7f4110009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:27:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.511+0000 7f4109ffb700 1 -- 172.21.15.7:0/1973719697 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f41040536b0 con 0x7f41140c07d0 2023-03-03T09:27:30.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.749+0000 7f411b634700 1 -- 172.21.15.7:0/1973719697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4114074bb0 con 0x7f41140c07d0 2023-03-03T09:27:30.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.749+0000 7f4109ffb700 1 -- 172.21.15.7:0/1973719697 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4104056cd0 con 0x7f41140c07d0 2023-03-03T09:27:30.751 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:27:30.751 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:27:30.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.751+0000 7f411b634700 1 -- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40f8068320 msgr2=0x7f40f806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:30.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.751+0000 7f411b634700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40f8068320 0x7f40f806a7d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4110001230 tx=0x7f4110009040 comp rx=0 tx=0).stop 2023-03-03T09:27:30.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.751+0000 7f411b634700 1 -- 172.21.15.7:0/1973719697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41140c07d0 msgr2=0x7f4114150880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:30.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.751+0000 7f411b634700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41140c07d0 0x7f4114150880 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f41040090b0 tx=0x7f4104004750 comp rx=0 tx=0).stop 2023-03-03T09:27:30.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.751+0000 7f411b634700 1 -- 172.21.15.7:0/1973719697 shutdown_connections 2023-03-03T09:27:30.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.751+0000 7f411b634700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4114151e30 0x7f41141ed8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:30.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.751+0000 7f411b634700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f411406e0c0 0x7f41140aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:30.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.751+0000 7f411b634700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40f8068320 0x7f40f806a7d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:30.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.751+0000 7f411b634700 1 --2- 172.21.15.7:0/1973719697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41140c07d0 0x7f4114150880 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:30.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.751+0000 7f411b634700 1 -- 172.21.15.7:0/1973719697 >> 172.21.15.7:0/1973719697 conn(0x7f411406c1c0 msgr2=0x7f411414a320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:27:30.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.752+0000 7f411b634700 1 -- 172.21.15.7:0/1973719697 shutdown_connections 2023-03-03T09:27:30.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:30.752+0000 7f411b634700 1 -- 172.21.15.7:0/1973719697 wait complete. 2023-03-03T09:27:30.755 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:27:30.764 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T09:27:37.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:27:37.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.781+0000 7f8eabaad700 1 Processor -- start 2023-03-03T09:27:37.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.781+0000 7f8eabaad700 1 -- start start 2023-03-03T09:27:37.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.782+0000 7f8eabaad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ea40c2e30 0x7f8ea4158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:37.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.782+0000 7f8eabaad700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea4157ff0 con 0x7f8ea406c6c0 2023-03-03T09:27:37.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.782+0000 7f8eabaad700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea406d640 con 0x7f8ea4151cd0 2023-03-03T09:27:37.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.782+0000 7f8eabaad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea406cda0 con 0x7f8ea40c2e30 2023-03-03T09:27:37.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.782+0000 7f8eaa04a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8ea406c6c0 0x7f8ea40736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50138/0 (socket says 172.21.15.7:50138) 2023-03-03T09:27:37.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.782+0000 7f8eaa04a700 1 -- 172.21.15.7:0/2460692174 learned_addr learned my addr 172.21.15.7:0/2460692174 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:27:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea9048700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ea40c2e30 0x7f8ea4158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792499479 0 0) 0x7f8ea4157ff0 con 0x7f8ea406c6c0 2023-03-03T09:27:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e900035b0 con 0x7f8ea406c6c0 2023-03-03T09:27:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4077290223 0 0) 0x7f8e900035b0 con 0x7f8ea406c6c0 2023-03-03T09:27:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ea40c2e30 msgr2=0x7f8ea4158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:27:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ea40c2e30 0x7f8ea4158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 >> v1:172.21.15.143:6789/0 conn(0x7f8ea4151cd0 legacy=0x7f8ea4154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea406d080 con 0x7f8ea406c6c0 2023-03-03T09:27:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8e9c0019f0 con 0x7f8ea406c6c0 2023-03-03T09:27:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 >> v1:172.21.15.7:6789/0 conn(0x7f8ea406c6c0 legacy=0x7f8ea40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:37.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ea40c2e30 0x7f8e90003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:37.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e90004020 0x7f8e90004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:37.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e90004d90 0x7f8e90005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:37.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea406cda0 con 0x7f8e90004020 2023-03-03T09:27:37.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9000bcf0 con 0x7f8e90004d90 2023-03-03T09:27:37.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9000beb0 con 0x7f8ea40c2e30 2023-03-03T09:27:37.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea9048700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e90004020 0x7f8e90004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:37.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea9849700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ea40c2e30 0x7f8e90003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:37.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8eaa04a700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e90004d90 0x7f8e90005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:37.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.783+0000 7f8ea9048700 1 -- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ea40c2e30 msgr2=0x7f8e90003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:27:37.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8ea9048700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ea40c2e30 0x7f8e90003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:37.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8ea9048700 1 -- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e90004d90 msgr2=0x7f8e90005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:27:37.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8ea9048700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e90004d90 0x7f8e90005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:37.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8ea9048700 1 -- 172.21.15.7:0/2460692174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e900035b0 con 0x7f8e90004020 2023-03-03T09:27:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8eaa04a700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e90004d90 0x7f8e90005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:27:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8ea9048700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e90004020 0x7f8e90004850 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0009b30 tx=0x7f8e90005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ae267fb00b1496 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:27:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea000e6c0 con 0x7f8e90004020 2023-03-03T09:27:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea000e820 con 0x7f8e90004020 2023-03-03T09:27:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8ea8847700 1 -- 172.21.15.7:0/2460692174 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea000eb90 con 0x7f8e90004020 2023-03-03T09:27:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8eabaad700 1 -- 172.21.15.7:0/2460692174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e90004020 msgr2=0x7f8e90004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8eabaad700 1 --2- 172.21.15.7:0/2460692174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e90004020 0x7f8e90004850 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0009b30 tx=0x7f8e90005880 comp rx=0 tx=0).stop 2023-03-03T09:27:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8eaa04a700 1 -- 172.21.15.7:0/2460692174 reap_dead start 2023-03-03T09:27:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8eabaad700 1 -- 172.21.15.7:0/2460692174 shutdown_connections 2023-03-03T09:27:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8eabaad700 1 -- 172.21.15.7:0/2460692174 >> 172.21.15.7:0/2460692174 conn(0x7f8ea40a94d0 msgr2=0x7f8ea40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:27:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.784+0000 7f8eabaad700 1 -- 172.21.15.7:0/2460692174 shutdown_connections 2023-03-03T09:27:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.785+0000 7f8eabaad700 1 -- 172.21.15.7:0/2460692174 wait complete. 2023-03-03T09:27:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.786+0000 7f8eabaad700 1 Processor -- start 2023-03-03T09:27:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.786+0000 7f8eabaad700 1 -- start start 2023-03-03T09:27:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.786+0000 7f8eabaad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e90004020 0x7f8ea4147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.786+0000 7f8eabaad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ea414db70 0x7f8ea40716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.786+0000 7f8eabaad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ea4071c10 0x7f8ea41e9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.786+0000 7f8eabaad700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea414bbe0 con 0x7f8ea4071c10 2023-03-03T09:27:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.786+0000 7f8eabaad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea414b900 con 0x7f8ea414db70 2023-03-03T09:27:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.786+0000 7f8eabaad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea41ed610 con 0x7f8e90004020 2023-03-03T09:27:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.786+0000 7f8eaa04a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ea4071c10 0x7f8ea41e9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8eaa04a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ea4071c10 0x7f8ea41e9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56170/0 (socket says 172.21.15.7:56170) 2023-03-03T09:27:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8eaa04a700 1 -- 172.21.15.7:0/2006510268 learned_addr learned my addr 172.21.15.7:0/2006510268 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:27:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8ea9048700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ea414db70 0x7f8ea40716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8ea9849700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e90004020 0x7f8ea4147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8eaa04a700 1 -- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e90004020 msgr2=0x7f8ea4147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8eaa04a700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e90004020 0x7f8ea4147a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8eaa04a700 1 -- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ea414db70 msgr2=0x7f8ea40716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8eaa04a700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ea414db70 0x7f8ea40716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8eaa04a700 1 -- 172.21.15.7:0/2006510268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea4157c80 con 0x7f8ea4071c10 2023-03-03T09:27:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8ea9048700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ea414db70 0x7f8ea40716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:27:37.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8eaa04a700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ea4071c10 0x7f8ea41e9e80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8ea406f310 tx=0x7f8e9c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:27:37.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8e9a7fc700 1 -- 172.21.15.7:0/2006510268 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c00ebf0 con 0x7f8ea4071c10 2023-03-03T09:27:37.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8e9a7fc700 1 -- 172.21.15.7:0/2006510268 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c00ed50 con 0x7f8ea4071c10 2023-03-03T09:27:37.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8eabaad700 1 -- 172.21.15.7:0/2006510268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea406d360 con 0x7f8ea4071c10 2023-03-03T09:27:37.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8e9a7fc700 1 -- 172.21.15.7:0/2006510268 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c013660 con 0x7f8ea4071c10 2023-03-03T09:27:37.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.787+0000 7f8eabaad700 1 -- 172.21.15.7:0/2006510268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea4073e00 con 0x7f8ea4071c10 2023-03-03T09:27:37.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.788+0000 7f8eabaad700 1 -- 172.21.15.7:0/2006510268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea41edf40 con 0x7f8ea4071c10 2023-03-03T09:27:37.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.789+0000 7f8e9a7fc700 1 -- 172.21.15.7:0/2006510268 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e9c0137c0 con 0x7f8ea4071c10 2023-03-03T09:27:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.789+0000 7f8e9a7fc700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e880682f0 0x7f8e8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:27:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.789+0000 7f8e9a7fc700 1 -- 172.21.15.7:0/2006510268 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8e9c0867e0 con 0x7f8ea4071c10 2023-03-03T09:27:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.789+0000 7f8ea9849700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e880682f0 0x7f8e8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:27:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.790+0000 7f8ea9849700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e880682f0 0x7f8e8806a7a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8e94007fd0 tx=0x7f8e94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:27:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:37.794+0000 7f8e9a7fc700 1 -- 172.21.15.7:0/2006510268 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e9c052b80 con 0x7f8ea4071c10 2023-03-03T09:27:38.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.032+0000 7f8eabaad700 1 -- 172.21.15.7:0/2006510268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ea41ed470 con 0x7f8ea4071c10 2023-03-03T09:27:38.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.033+0000 7f8e9a7fc700 1 -- 172.21.15.7:0/2006510268 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8ea41ed470 con 0x7f8ea4071c10 2023-03-03T09:27:38.034 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:27:38.034 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:27:38.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.034+0000 7f8eabaad700 1 -- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e880682f0 msgr2=0x7f8e8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:38.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.034+0000 7f8eabaad700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e880682f0 0x7f8e8806a7a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8e94007fd0 tx=0x7f8e94009040 comp rx=0 tx=0).stop 2023-03-03T09:27:38.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.034+0000 7f8eabaad700 1 -- 172.21.15.7:0/2006510268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ea4071c10 msgr2=0x7f8ea41e9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:27:38.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.034+0000 7f8eabaad700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ea4071c10 0x7f8ea41e9e80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8ea406f310 tx=0x7f8e9c00e440 comp rx=0 tx=0).stop 2023-03-03T09:27:38.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.035+0000 7f8eabaad700 1 -- 172.21.15.7:0/2006510268 shutdown_connections 2023-03-03T09:27:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.035+0000 7f8eabaad700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e880682f0 0x7f8e8806a7a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.035+0000 7f8eabaad700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ea4071c10 0x7f8ea41e9e80 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.035+0000 7f8eabaad700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e90004020 0x7f8ea4147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.035+0000 7f8eabaad700 1 --2- 172.21.15.7:0/2006510268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ea414db70 0x7f8ea40716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:27:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.035+0000 7f8eabaad700 1 -- 172.21.15.7:0/2006510268 >> 172.21.15.7:0/2006510268 conn(0x7f8ea40a94d0 msgr2=0x7f8ea406b430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:27:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.035+0000 7f8eabaad700 1 -- 172.21.15.7:0/2006510268 shutdown_connections 2023-03-03T09:27:38.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:27:38.035+0000 7f8eabaad700 1 -- 172.21.15.7:0/2006510268 wait complete. 2023-03-03T09:27:38.038 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:27:38.047 INFO:tasks.mds_thrash.fs.[a]:waiting for 39 secs before thrashing 2023-03-03T09:27:45.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:27:45.315 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:15.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:15.396 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:17.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:28:17.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.186+0000 7f7215893700 1 Processor -- start 2023-03-03T09:28:17.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.186+0000 7f7215893700 1 -- start start 2023-03-03T09:28:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.186+0000 7f7215893700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7210151cc0 0x7f72100ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.186+0000 7f7215893700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72100b00b0 0x7f72100c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f7215893700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100c7ca0 con 0x7f72100cac10 2023-03-03T09:28:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f7215893700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7210152090 con 0x7f7210151cc0 2023-03-03T09:28:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f7215893700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72100c6cf0 con 0x7f72100b00b0 2023-03-03T09:28:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f72100cac10 0x7f72100c7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50182/0 (socket says 172.21.15.7:50182) 2023-03-03T09:28:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720f7fe700 1 -- 172.21.15.7:0/1499159646 learned_addr learned my addr 172.21.15.7:0/1499159646 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:28:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720effd700 1 --2- 172.21.15.7:0/1499159646 >> v2:172.21.15.143:3300/0 conn(0x7f7210151cc0 0x7f72100ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720e7fc700 1 --2- 172.21.15.7:0/1499159646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72100b00b0 0x7f72100c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720dffb700 1 -- 172.21.15.7:0/1499159646 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901258033 0 0) 0x7f72100c7ca0 con 0x7f72100cac10 2023-03-03T09:28:17.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720dffb700 1 -- 172.21.15.7:0/1499159646 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f80035b0 con 0x7f72100cac10 2023-03-03T09:28:17.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720effd700 1 -- 172.21.15.7:0/1499159646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72100b00b0 msgr2=0x7f72100c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:28:17.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720effd700 1 --2- 172.21.15.7:0/1499159646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72100b00b0 0x7f72100c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:17.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720effd700 1 -- 172.21.15.7:0/1499159646 >> v1:172.21.15.7:6789/0 conn(0x7f72100cac10 legacy=0x7f72100c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:17.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720effd700 1 -- 172.21.15.7:0/1499159646 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72101548f0 con 0x7f7210151cc0 2023-03-03T09:28:17.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.187+0000 7f720effd700 1 --2- 172.21.15.7:0/1499159646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7210151cc0 0x7f72100ca6d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7200009840 tx=0x7f7200009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75105cef364b2c0e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:17.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f720dffb700 1 -- 172.21.15.7:0/1499159646 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200011830 con 0x7f7210151cc0 2023-03-03T09:28:17.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f720dffb700 1 -- 172.21.15.7:0/1499159646 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7200011990 con 0x7f7210151cc0 2023-03-03T09:28:17.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f720dffb700 1 -- 172.21.15.7:0/1499159646 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200011d20 con 0x7f7210151cc0 2023-03-03T09:28:17.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f7215893700 1 -- 172.21.15.7:0/1499159646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7210151cc0 msgr2=0x7f72100ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:17.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f7215893700 1 --2- 172.21.15.7:0/1499159646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7210151cc0 0x7f72100ca6d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7200009840 tx=0x7f7200009b50 comp rx=0 tx=0).stop 2023-03-03T09:28:17.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f7215893700 1 -- 172.21.15.7:0/1499159646 shutdown_connections 2023-03-03T09:28:17.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f7215893700 1 --2- 172.21.15.7:0/1499159646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7210151cc0 0x7f72100ca6d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:17.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f7215893700 1 --2- 172.21.15.7:0/1499159646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72100b00b0 0x7f72100c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:17.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f7215893700 1 -- 172.21.15.7:0/1499159646 >> 172.21.15.7:0/1499159646 conn(0x7f721006b980 msgr2=0x7f7210071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:28:17.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f7215893700 1 -- 172.21.15.7:0/1499159646 shutdown_connections 2023-03-03T09:28:17.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.188+0000 7f7215893700 1 -- 172.21.15.7:0/1499159646 wait complete. 2023-03-03T09:28:17.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.189+0000 7f7215893700 1 Processor -- start 2023-03-03T09:28:17.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f7215893700 1 -- start start 2023-03-03T09:28:17.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f7215893700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72100b00b0 0x7f72101ea390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:17.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f7215893700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72100cac10 0x7f7210158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:17.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f7215893700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7210151cc0 0x7f7210155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:17.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f7215893700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100caac0 con 0x7f72100cac10 2023-03-03T09:28:17.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f7215893700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7210156b20 con 0x7f72100b00b0 2023-03-03T09:28:17.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f7215893700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72101e98d0 con 0x7f7210151cc0 2023-03-03T09:28:17.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f720e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72100cac10 0x7f7210158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:17.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f720e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72100cac10 0x7f7210158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56204/0 (socket says 172.21.15.7:56204) 2023-03-03T09:28:17.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f720e7fc700 1 -- 172.21.15.7:0/3825889733 learned_addr learned my addr 172.21.15.7:0/3825889733 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:28:17.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f720effd700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72100b00b0 0x7f72101ea390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:17.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.190+0000 7f720f7fe700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7210151cc0 0x7f7210155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:17.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f720e7fc700 1 -- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7210151cc0 msgr2=0x7f7210155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:17.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f720e7fc700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7210151cc0 0x7f7210155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:17.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f720e7fc700 1 -- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72100b00b0 msgr2=0x7f72101ea390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:17.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f720e7fc700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72100b00b0 0x7f72101ea390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:17.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f720e7fc700 1 -- 172.21.15.7:0/3825889733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72000094a0 con 0x7f72100cac10 2023-03-03T09:28:17.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f720e7fc700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72100cac10 0x7f7210158210 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f720400e9b0 tx=0x7f720400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:17.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f71f7fff700 1 -- 172.21.15.7:0/3825889733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7204009710 con 0x7f72100cac10 2023-03-03T09:28:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f71f7fff700 1 -- 172.21.15.7:0/3825889733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720400ee50 con 0x7f72100cac10 2023-03-03T09:28:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f7215893700 1 -- 172.21.15.7:0/3825889733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7210154eb0 con 0x7f72100cac10 2023-03-03T09:28:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f71f7fff700 1 -- 172.21.15.7:0/3825889733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720400ca50 con 0x7f72100cac10 2023-03-03T09:28:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.191+0000 7f7215893700 1 -- 172.21.15.7:0/3825889733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7210154bd0 con 0x7f72100cac10 2023-03-03T09:28:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.192+0000 7f7215893700 1 -- 172.21.15.7:0/3825889733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7210155190 con 0x7f72100cac10 2023-03-03T09:28:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.192+0000 7f71f7fff700 1 -- 172.21.15.7:0/3825889733 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7204019030 con 0x7f72100cac10 2023-03-03T09:28:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.193+0000 7f71f7fff700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71fc0682f0 0x7f71fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.193+0000 7f71f7fff700 1 -- 172.21.15.7:0/3825889733 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7204086070 con 0x7f72100cac10 2023-03-03T09:28:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.193+0000 7f720effd700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71fc0682f0 0x7f71fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.194+0000 7f720effd700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71fc0682f0 0x7f71fc06a7a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7200000c00 tx=0x7f720001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.197+0000 7f71f7fff700 1 -- 172.21.15.7:0/3825889733 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7204052390 con 0x7f72100cac10 2023-03-03T09:28:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.431+0000 7f7215893700 1 -- 172.21.15.7:0/3825889733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72101ea1e0 con 0x7f72100cac10 2023-03-03T09:28:17.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.431+0000 7f71f7fff700 1 -- 172.21.15.7:0/3825889733 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f72040559b0 con 0x7f72100cac10 2023-03-03T09:28:17.433 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:28:17.433 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:28:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 -- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71fc0682f0 msgr2=0x7f71fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71fc0682f0 0x7f71fc06a7a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7200000c00 tx=0x7f720001c040 comp rx=0 tx=0).stop 2023-03-03T09:28:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 -- 172.21.15.7:0/3825889733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72100cac10 msgr2=0x7f7210158210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72100cac10 0x7f7210158210 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f720400e9b0 tx=0x7f720400ecc0 comp rx=0 tx=0).stop 2023-03-03T09:28:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 -- 172.21.15.7:0/3825889733 shutdown_connections 2023-03-03T09:28:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7210151cc0 0x7f7210155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72100b00b0 0x7f72101ea390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71fc0682f0 0x7f71fc06a7a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 --2- 172.21.15.7:0/3825889733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72100cac10 0x7f7210158210 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 -- 172.21.15.7:0/3825889733 >> 172.21.15.7:0/3825889733 conn(0x7f721006b980 msgr2=0x7f721014b410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:28:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 -- 172.21.15.7:0/3825889733 shutdown_connections 2023-03-03T09:28:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:17.433+0000 7f7215893700 1 -- 172.21.15.7:0/3825889733 wait complete. 2023-03-03T09:28:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:28:17.446 INFO:tasks.mds_thrash.fs.[a]:waiting for 41 secs before thrashing 2023-03-03T09:28:34.765 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:28:34.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.903+0000 7ff6da1fe700 1 Processor -- start 2023-03-03T09:28:34.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.903+0000 7ff6da1fe700 1 -- start start 2023-03-03T09:28:34.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.904+0000 7ff6da1fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d4155390 0x7ff6d40c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:34.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.904+0000 7ff6da1fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff6d4151770 0x7ff6d4151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:34.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.904+0000 7ff6da1fe700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d4154910 con 0x7ff6d406e020 2023-03-03T09:28:34.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.904+0000 7ff6da1fe700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d4070b90 con 0x7ff6d4151770 2023-03-03T09:28:34.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.904+0000 7ff6da1fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d4151320 con 0x7ff6d4155390 2023-03-03T09:28:34.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.904+0000 7ff6d37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d4155390 0x7ff6d40c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:34.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.904+0000 7ff6d2ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ff6d406e020 0x7ff6d4071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:53726/0 (socket says 172.21.15.7:53726) 2023-03-03T09:28:34.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.904+0000 7ff6d3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff6d4151770 0x7ff6d4151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:34.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.904+0000 7ff6d2ffd700 1 -- 172.21.15.7:0/1831450121 learned_addr learned my addr 172.21.15.7:0/1831450121 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:28:34.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.905+0000 7ff6d37fe700 1 -- 172.21.15.7:0/1831450121 >> v2:172.21.15.143:3300/0 conn(0x7ff6d4151770 msgr2=0x7ff6d4151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:28:34.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.905+0000 7ff6d37fe700 1 --2- 172.21.15.7:0/1831450121 >> v2:172.21.15.143:3300/0 conn(0x7ff6d4151770 0x7ff6d4151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:34.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.905+0000 7ff6d37fe700 1 -- 172.21.15.7:0/1831450121 >> v1:172.21.15.143:6789/0 conn(0x7ff6d406e020 legacy=0x7ff6d4071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:34.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.905+0000 7ff6d37fe700 1 -- 172.21.15.7:0/1831450121 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d40702a0 con 0x7ff6d4155390 2023-03-03T09:28:34.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.905+0000 7ff6d27fc700 1 -- 172.21.15.7:0/1831450121 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029305024 0 0) 0x7ff6d4154910 con 0x7ff6d406e020 2023-03-03T09:28:34.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.905+0000 7ff6d3fff700 1 --2- 172.21.15.7:0/1831450121 >> v2:172.21.15.143:3300/0 conn(0x7ff6d4151770 0x7ff6d4151bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:28:34.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.905+0000 7ff6d37fe700 1 --2- 172.21.15.7:0/1831450121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d4155390 0x7ff6d40c3e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400b8f0 tx=0x7ff6c400bc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac76967189464f67 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:34.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.905+0000 7ff6d27fc700 1 -- 172.21.15.7:0/1831450121 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c400c9f0 con 0x7ff6d4155390 2023-03-03T09:28:34.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.905+0000 7ff6d27fc700 1 -- 172.21.15.7:0/1831450121 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c400cb50 con 0x7ff6d4155390 2023-03-03T09:28:34.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.905+0000 7ff6d27fc700 1 -- 172.21.15.7:0/1831450121 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c400f460 con 0x7ff6d4155390 2023-03-03T09:28:34.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.906+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1831450121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d4155390 msgr2=0x7ff6d40c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:34.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.906+0000 7ff6da1fe700 1 --2- 172.21.15.7:0/1831450121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d4155390 0x7ff6d40c3e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400b8f0 tx=0x7ff6c400bc00 comp rx=0 tx=0).stop 2023-03-03T09:28:34.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.906+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1831450121 shutdown_connections 2023-03-03T09:28:34.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.906+0000 7ff6da1fe700 1 --2- 172.21.15.7:0/1831450121 >> v2:172.21.15.143:3300/0 conn(0x7ff6d4151770 0x7ff6d4151bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:34.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.906+0000 7ff6da1fe700 1 --2- 172.21.15.7:0/1831450121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d4155390 0x7ff6d40c3e30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:34.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.906+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1831450121 >> 172.21.15.7:0/1831450121 conn(0x7ff6d406c110 msgr2=0x7ff6d406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:28:34.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.906+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1831450121 shutdown_connections 2023-03-03T09:28:34.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.906+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1831450121 wait complete. 2023-03-03T09:28:34.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.907+0000 7ff6da1fe700 1 Processor -- start 2023-03-03T09:28:34.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.907+0000 7ff6da1fe700 1 -- start start 2023-03-03T09:28:34.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6da1fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d406e020 0x7ff6d4153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:34.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6da1fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6d4151770 0x7ff6d40c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:34.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6da1fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6d4155390 0x7ff6d41e57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:34.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6da1fe700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d4152be0 con 0x7ff6d4155390 2023-03-03T09:28:34.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6da1fe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d4073e80 con 0x7ff6d4151770 2023-03-03T09:28:34.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6da1fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d414d580 con 0x7ff6d406e020 2023-03-03T09:28:34.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6d4155390 0x7ff6d41e57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:34.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6d4155390 0x7ff6d41e57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56236/0 (socket says 172.21.15.7:56236) 2023-03-03T09:28:34.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d3fff700 1 -- 172.21.15.7:0/1064124592 learned_addr learned my addr 172.21.15.7:0/1064124592 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:28:34.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d37fe700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d406e020 0x7ff6d4153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:34.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d2ffd700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6d4151770 0x7ff6d40c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:34.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d3fff700 1 -- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d406e020 msgr2=0x7ff6d4153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:34.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d3fff700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d406e020 0x7ff6d4153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:34.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d3fff700 1 -- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6d4151770 msgr2=0x7ff6d40c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:34.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d3fff700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6d4151770 0x7ff6d40c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:34.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d3fff700 1 -- 172.21.15.7:0/1064124592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc00f050 con 0x7ff6d4155390 2023-03-03T09:28:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d37fe700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d406e020 0x7ff6d4153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:28:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.908+0000 7ff6d2ffd700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6d4151770 0x7ff6d40c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:28:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.909+0000 7ff6d3fff700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6d4155390 0x7ff6d41e57b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00f500 tx=0x7ff6cc00f8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.909+0000 7ff6cbfff700 1 -- 172.21.15.7:0/1064124592 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc012050 con 0x7ff6d4155390 2023-03-03T09:28:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.909+0000 7ff6cbfff700 1 -- 172.21.15.7:0/1064124592 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc013040 con 0x7ff6d4155390 2023-03-03T09:28:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.909+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1064124592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c400b550 con 0x7ff6d4155390 2023-03-03T09:28:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.909+0000 7ff6cbfff700 1 -- 172.21.15.7:0/1064124592 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc003e60 con 0x7ff6d4155390 2023-03-03T09:28:34.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.909+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1064124592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d4070820 con 0x7ff6d4155390 2023-03-03T09:28:34.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.910+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1064124592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d4070540 con 0x7ff6d4155390 2023-03-03T09:28:34.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.910+0000 7ff6cbfff700 1 -- 172.21.15.7:0/1064124592 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff6cc0175d0 con 0x7ff6d4155390 2023-03-03T09:28:34.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.911+0000 7ff6cbfff700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6b40682c0 0x7ff6b406a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:34.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.911+0000 7ff6cbfff700 1 -- 172.21.15.7:0/1064124592 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff6cc08aeb0 con 0x7ff6d4155390 2023-03-03T09:28:34.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.911+0000 7ff6d37fe700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6b40682c0 0x7ff6b406a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.912+0000 7ff6d37fe700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6b40682c0 0x7ff6b406a770 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff6c40090b0 tx=0x7ff6c4019040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:34.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:34.915+0000 7ff6cbfff700 1 -- 172.21.15.7:0/1064124592 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff6cc057100 con 0x7ff6d4155390 2023-03-03T09:28:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.155+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1064124592 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6d4155760 con 0x7ff6d4155390 2023-03-03T09:28:35.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.155+0000 7ff6cbfff700 1 -- 172.21.15.7:0/1064124592 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff6cc05a720 con 0x7ff6d4155390 2023-03-03T09:28:35.157 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:28:35.157 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:28:35.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.157+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6b40682c0 msgr2=0x7ff6b406a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:35.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.157+0000 7ff6da1fe700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6b40682c0 0x7ff6b406a770 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff6c40090b0 tx=0x7ff6c4019040 comp rx=0 tx=0).stop 2023-03-03T09:28:35.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.157+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1064124592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6d4155390 msgr2=0x7ff6d41e57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:35.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.157+0000 7ff6da1fe700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6d4155390 0x7ff6d41e57b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00f500 tx=0x7ff6cc00f8c0 comp rx=0 tx=0).stop 2023-03-03T09:28:35.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.158+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1064124592 shutdown_connections 2023-03-03T09:28:35.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.158+0000 7ff6da1fe700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6b40682c0 0x7ff6b406a770 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:35.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.158+0000 7ff6da1fe700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6d4155390 0x7ff6d41e57b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:35.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.158+0000 7ff6da1fe700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6d406e020 0x7ff6d4153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:35.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.158+0000 7ff6da1fe700 1 --2- 172.21.15.7:0/1064124592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6d4151770 0x7ff6d40c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:35.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.158+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1064124592 >> 172.21.15.7:0/1064124592 conn(0x7ff6d406c110 msgr2=0x7ff6d40c3320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:28:35.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.158+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1064124592 shutdown_connections 2023-03-03T09:28:35.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:35.158+0000 7ff6da1fe700 1 -- 172.21.15.7:0/1064124592 wait complete. 2023-03-03T09:28:35.161 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:28:35.171 INFO:tasks.mds_thrash.fs.[b]:waiting for 15 secs before thrashing 2023-03-03T09:28:45.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:45.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:28:50.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:28:50.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.309+0000 7f06ad51c700 1 Processor -- start 2023-03-03T09:28:50.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.309+0000 7f06ad51c700 1 -- start start 2023-03-03T09:28:50.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06ad51c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f06a80caa30 0x7f06a80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:50.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06ad51c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a8154ff0 0x7f06a80cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:50.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06ad51c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a80c7310 con 0x7f06a80c00e0 2023-03-03T09:28:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06ad51c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a80bfd10 con 0x7f06a80caa30 2023-03-03T09:28:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06ad51c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a80bfa30 con 0x7f06a8154ff0 2023-03-03T09:28:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06a67fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f06a80c00e0 0x7f06a80c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50228/0 (socket says 172.21.15.7:50228) 2023-03-03T09:28:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06a67fc700 1 -- 172.21.15.7:0/2680741354 learned_addr learned my addr 172.21.15.7:0/2680741354 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:28:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06a77fe700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a8154ff0 0x7f06a80cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06a6ffd700 1 --2- 172.21.15.7:0/2680741354 >> v2:172.21.15.143:3300/0 conn(0x7f06a80caa30 0x7f06a80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:50.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501480373 0 0) 0x7f06a80c7310 con 0x7f06a80c00e0 2023-03-03T09:28:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.310+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06900035b0 con 0x7f06a80c00e0 2023-03-03T09:28:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1773697609 0 0) 0x7f06900035b0 con 0x7f06a80c00e0 2023-03-03T09:28:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a8154ff0 msgr2=0x7f06a80cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:28:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a8154ff0 0x7f06a80cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 >> v2:172.21.15.143:3300/0 conn(0x7f06a80caa30 msgr2=0x7f06a80c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:28:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 --2- 172.21.15.7:0/2680741354 >> v2:172.21.15.143:3300/0 conn(0x7f06a80caa30 0x7f06a80c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a8154510 con 0x7f06a80c00e0 2023-03-03T09:28:50.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f06980017f0 con 0x7f06a80c00e0 2023-03-03T09:28:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 >> v1:172.21.15.7:6789/0 conn(0x7f06a80c00e0 legacy=0x7f06a80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 0x7f0690003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a8154ff0 0x7f0690004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0690004d90 0x7f0690005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a80bfd10 con 0x7f0690003a70 2023-03-03T09:28:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a80bfa30 con 0x7f0690004d90 2023-03-03T09:28:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069000ba60 con 0x7f06a8154ff0 2023-03-03T09:28:50.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a6ffd700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 0x7f0690003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:50.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a67fc700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a8154ff0 0x7f0690004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:50.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a77fe700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0690004d90 0x7f0690005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:50.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a6ffd700 1 -- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a8154ff0 msgr2=0x7f0690004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:28:50.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a6ffd700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a8154ff0 0x7f0690004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a6ffd700 1 -- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0690004d90 msgr2=0x7f0690005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:28:50.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a6ffd700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0690004d90 0x7f0690005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a6ffd700 1 -- 172.21.15.7:0/2680741354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06900035b0 con 0x7f0690003a70 2023-03-03T09:28:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a6ffd700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 0x7f0690003e80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f069c009b90 tx=0x7f0690005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac4e9a4207d31d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c00e6c0 con 0x7f0690003a70 2023-03-03T09:28:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069c00e820 con 0x7f0690003a70 2023-03-03T09:28:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.311+0000 7f06a5ffb700 1 -- 172.21.15.7:0/2680741354 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c00eb90 con 0x7f0690003a70 2023-03-03T09:28:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.312+0000 7f06ad51c700 1 -- 172.21.15.7:0/2680741354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 msgr2=0x7f0690003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.312+0000 7f06ad51c700 1 --2- 172.21.15.7:0/2680741354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 0x7f0690003e80 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f069c009b90 tx=0x7f0690005880 comp rx=0 tx=0).stop 2023-03-03T09:28:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.312+0000 7f06a77fe700 1 -- 172.21.15.7:0/2680741354 reap_dead start 2023-03-03T09:28:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.312+0000 7f06ad51c700 1 -- 172.21.15.7:0/2680741354 shutdown_connections 2023-03-03T09:28:50.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.312+0000 7f06ad51c700 1 -- 172.21.15.7:0/2680741354 >> 172.21.15.7:0/2680741354 conn(0x7f06a8066e40 msgr2=0x7f06a8071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:28:50.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.312+0000 7f06ad51c700 1 -- 172.21.15.7:0/2680741354 shutdown_connections 2023-03-03T09:28:50.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.312+0000 7f06ad51c700 1 -- 172.21.15.7:0/2680741354 wait complete. 2023-03-03T09:28:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.314+0000 7f06ad51c700 1 Processor -- start 2023-03-03T09:28:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.314+0000 7f06ad51c700 1 -- start start 2023-03-03T09:28:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06ad51c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 0x7f06a80c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06ad51c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a814daf0 0x7f06a814df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06ad51c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06a814cfc0 0x7f06a814d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06ad51c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a80c2cb0 con 0x7f0690003a70 2023-03-03T09:28:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06ad51c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a81ea040 con 0x7f06a814cfc0 2023-03-03T09:28:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06ad51c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a81584e0 con 0x7f06a814daf0 2023-03-03T09:28:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 0x7f06a80c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 0x7f06a80c1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56260/0 (socket says 172.21.15.7:56260) 2023-03-03T09:28:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a6ffd700 1 -- 172.21.15.7:0/475629770 learned_addr learned my addr 172.21.15.7:0/475629770 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:28:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a67fc700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a814daf0 0x7f06a814df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:50.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a77fe700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06a814cfc0 0x7f06a814d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:50.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a6ffd700 1 -- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a814daf0 msgr2=0x7f06a814df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:50.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a6ffd700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a814daf0 0x7f06a814df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a6ffd700 1 -- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06a814cfc0 msgr2=0x7f06a814d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:50.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a6ffd700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06a814cfc0 0x7f06a814d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a6ffd700 1 -- 172.21.15.7:0/475629770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a8154ad0 con 0x7f0690003a70 2023-03-03T09:28:50.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a77fe700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06a814cfc0 0x7f06a814d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:28:50.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a67fc700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a814daf0 0x7f06a814df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:28:50.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.315+0000 7f06a6ffd700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 0x7f06a80c1c50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f069c009a80 tx=0x7f069c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:50.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.316+0000 7f068f7fe700 1 -- 172.21.15.7:0/475629770 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c021070 con 0x7f0690003a70 2023-03-03T09:28:50.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.316+0000 7f068f7fe700 1 -- 172.21.15.7:0/475629770 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069c004550 con 0x7f0690003a70 2023-03-03T09:28:50.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.316+0000 7f06ad51c700 1 -- 172.21.15.7:0/475629770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a81547f0 con 0x7f0690003a70 2023-03-03T09:28:50.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.316+0000 7f068f7fe700 1 -- 172.21.15.7:0/475629770 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c00e6c0 con 0x7f0690003a70 2023-03-03T09:28:50.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.316+0000 7f06ad51c700 1 -- 172.21.15.7:0/475629770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a80bf650 con 0x7f0690003a70 2023-03-03T09:28:50.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.317+0000 7f06ad51c700 1 -- 172.21.15.7:0/475629770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a8074bb0 con 0x7f0690003a70 2023-03-03T09:28:50.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.317+0000 7f068f7fe700 1 -- 172.21.15.7:0/475629770 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f069c02e030 con 0x7f0690003a70 2023-03-03T09:28:50.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.317+0000 7f068f7fe700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06940682f0 0x7f069406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:50.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.318+0000 7f068f7fe700 1 -- 172.21.15.7:0/475629770 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f069c087320 con 0x7f0690003a70 2023-03-03T09:28:50.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.318+0000 7f06a67fc700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06940682f0 0x7f069406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:50.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.318+0000 7f06a67fc700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06940682f0 0x7f069406a7a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0698000c00 tx=0x7f0698009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:50.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.322+0000 7f068f7fe700 1 -- 172.21.15.7:0/475629770 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f069c0536c0 con 0x7f0690003a70 2023-03-03T09:28:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.559+0000 7f06ad51c700 1 -- 172.21.15.7:0/475629770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06a80c5030 con 0x7f0690003a70 2023-03-03T09:28:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.559+0000 7f068f7fe700 1 -- 172.21.15.7:0/475629770 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f069c056ce0 con 0x7f0690003a70 2023-03-03T09:28:50.561 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:28:50.561 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:28:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 -- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06940682f0 msgr2=0x7f069406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06940682f0 0x7f069406a7a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0698000c00 tx=0x7f0698009040 comp rx=0 tx=0).stop 2023-03-03T09:28:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 -- 172.21.15.7:0/475629770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 msgr2=0x7f06a80c1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 0x7f06a80c1c50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f069c009a80 tx=0x7f069c0047c0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 -- 172.21.15.7:0/475629770 shutdown_connections 2023-03-03T09:28:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06a814cfc0 0x7f06a814d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06940682f0 0x7f069406a7a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0690003a70 0x7f06a80c1c50 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 --2- 172.21.15.7:0/475629770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06a814daf0 0x7f06a814df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 -- 172.21.15.7:0/475629770 >> 172.21.15.7:0/475629770 conn(0x7f06a8066e40 msgr2=0x7f06a80c90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:28:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 -- 172.21.15.7:0/475629770 shutdown_connections 2023-03-03T09:28:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:50.561+0000 7f06ad51c700 1 -- 172.21.15.7:0/475629770 wait complete. 2023-03-03T09:28:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:28:50.574 INFO:tasks.mds_thrash.fs.[b]:waiting for 51 secs before thrashing 2023-03-03T09:28:58.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:28:58.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.583+0000 7f69a7a85700 1 Processor -- start 2023-03-03T09:28:58.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.584+0000 7f69a7a85700 1 -- start start 2023-03-03T09:28:58.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.584+0000 7f69a7a85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f69a0154f10 0x7f69a0158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:58.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.584+0000 7f69a7a85700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69a00aa010 0x7f69a0158540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:58.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.584+0000 7f69a7a85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a0151ef0 con 0x7f69a0151b20 2023-03-03T09:28:58.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.584+0000 7f69a7a85700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a0157b20 con 0x7f69a00aa010 2023-03-03T09:28:58.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.584+0000 7f69a7a85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a0155410 con 0x7f69a0154f10 2023-03-03T09:28:58.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a6022700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69a00aa010 0x7f69a0158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:58.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a6022700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69a00aa010 0x7f69a0158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56280/0 (socket says 172.21.15.7:56280) 2023-03-03T09:28:58.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a6022700 1 -- 172.21.15.7:0/2133848708 learned_addr learned my addr 172.21.15.7:0/2133848708 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:28:58.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a5020700 1 --2- 172.21.15.7:0/2133848708 >> v2:172.21.15.143:3300/0 conn(0x7f69a0154f10 0x7f69a0158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:58.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a6022700 1 -- 172.21.15.7:0/2133848708 >> v2:172.21.15.143:3300/0 conn(0x7f69a0154f10 msgr2=0x7f69a0158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:28:58.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a6022700 1 --2- 172.21.15.7:0/2133848708 >> v2:172.21.15.143:3300/0 conn(0x7f69a0154f10 0x7f69a0158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:58.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a6022700 1 -- 172.21.15.7:0/2133848708 >> v1:172.21.15.143:6789/0 conn(0x7f69a0151b20 legacy=0x7f69a0071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:28:58.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a6022700 1 -- 172.21.15.7:0/2133848708 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a006f640 con 0x7f69a00aa010 2023-03-03T09:28:58.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a6022700 1 --2- 172.21.15.7:0/2133848708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 0x7f69a0158540 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f69a0070850 tx=0x7f699800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0f6b2fc04f4cc8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:58.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a481f700 1 -- 172.21.15.7:0/2133848708 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699800e070 con 0x7f69a00aa010 2023-03-03T09:28:58.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a481f700 1 -- 172.21.15.7:0/2133848708 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6998003a20 con 0x7f69a00aa010 2023-03-03T09:28:58.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.585+0000 7f69a481f700 1 -- 172.21.15.7:0/2133848708 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998003dd0 con 0x7f69a00aa010 2023-03-03T09:28:58.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.586+0000 7f69a7a85700 1 -- 172.21.15.7:0/2133848708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 msgr2=0x7f69a0158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:58.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.586+0000 7f69a7a85700 1 --2- 172.21.15.7:0/2133848708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 0x7f69a0158540 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f69a0070850 tx=0x7f699800b330 comp rx=0 tx=0).stop 2023-03-03T09:28:58.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.586+0000 7f69a7a85700 1 -- 172.21.15.7:0/2133848708 shutdown_connections 2023-03-03T09:28:58.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.586+0000 7f69a7a85700 1 --2- 172.21.15.7:0/2133848708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 0x7f69a0158540 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:58.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.586+0000 7f69a7a85700 1 --2- 172.21.15.7:0/2133848708 >> v2:172.21.15.143:3300/0 conn(0x7f69a0154f10 0x7f69a0158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:58.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.586+0000 7f69a7a85700 1 -- 172.21.15.7:0/2133848708 >> 172.21.15.7:0/2133848708 conn(0x7f69a00c8900 msgr2=0x7f69a00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:28:58.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.586+0000 7f69a7a85700 1 -- 172.21.15.7:0/2133848708 shutdown_connections 2023-03-03T09:28:58.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.586+0000 7f69a7a85700 1 -- 172.21.15.7:0/2133848708 wait complete. 2023-03-03T09:28:58.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.587+0000 7f69a7a85700 1 Processor -- start 2023-03-03T09:28:58.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.587+0000 7f69a7a85700 1 -- start start 2023-03-03T09:28:58.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a7a85700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 0x7f69a01ead50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:58.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a7a85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69a0151b20 0x7f69a01535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:58.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a7a85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a0154f10 0x7f69a0149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:58.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a7a85700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a0150fb0 con 0x7f69a00aa010 2023-03-03T09:28:58.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a7a85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a006ef40 con 0x7f69a0154f10 2023-03-03T09:28:58.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a7a85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a006f220 con 0x7f69a0151b20 2023-03-03T09:28:58.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a5821700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 0x7f69a01ead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:58.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a5821700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 0x7f69a01ead50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56282/0 (socket says 172.21.15.7:56282) 2023-03-03T09:28:58.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a5821700 1 -- 172.21.15.7:0/2030061106 learned_addr learned my addr 172.21.15.7:0/2030061106 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:28:58.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a6022700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a0154f10 0x7f69a0149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:58.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a5020700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69a0151b20 0x7f69a01535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:58.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a5821700 1 -- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69a0151b20 msgr2=0x7f69a01535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:58.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a5821700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69a0151b20 0x7f69a01535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:58.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a5821700 1 -- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a0154f10 msgr2=0x7f69a0149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:58.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a5821700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a0154f10 0x7f69a0149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:58.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a5821700 1 -- 172.21.15.7:0/2030061106 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6998009b30 con 0x7f69a00aa010 2023-03-03T09:28:58.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a6022700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a0154f10 0x7f69a0149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:28:58.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69a5821700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 0x7f69a01ead50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f699c00c4d0 tx=0x7f699c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:58.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.588+0000 7f69927fc700 1 -- 172.21.15.7:0/2030061106 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c011790 con 0x7f69a00aa010 2023-03-03T09:28:58.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.589+0000 7f69927fc700 1 -- 172.21.15.7:0/2030061106 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c0118f0 con 0x7f69a00aa010 2023-03-03T09:28:58.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.589+0000 7f69a7a85700 1 -- 172.21.15.7:0/2030061106 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a006fc00 con 0x7f69a00aa010 2023-03-03T09:28:58.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.589+0000 7f69927fc700 1 -- 172.21.15.7:0/2030061106 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699c011ca0 con 0x7f69a00aa010 2023-03-03T09:28:58.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.589+0000 7f69a7a85700 1 -- 172.21.15.7:0/2030061106 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a006f920 con 0x7f69a00aa010 2023-03-03T09:28:58.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.590+0000 7f69a7a85700 1 -- 172.21.15.7:0/2030061106 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a0071f30 con 0x7f69a00aa010 2023-03-03T09:28:58.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.590+0000 7f69927fc700 1 -- 172.21.15.7:0/2030061106 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f699c010050 con 0x7f69a00aa010 2023-03-03T09:28:58.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.590+0000 7f69927fc700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69840682f0 0x7f698406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:28:58.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.590+0000 7f69927fc700 1 -- 172.21.15.7:0/2030061106 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f699c085ed0 con 0x7f69a00aa010 2023-03-03T09:28:58.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.591+0000 7f69a5020700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69840682f0 0x7f698406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:28:58.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.591+0000 7f69a5020700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69840682f0 0x7f698406a7a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f69a00ac9e0 tx=0x7f6994008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:28:58.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.595+0000 7f69927fc700 1 -- 172.21.15.7:0/2030061106 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f699c052270 con 0x7f69a00aa010 2023-03-03T09:28:58.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.832+0000 7f69a7a85700 1 -- 172.21.15.7:0/2030061106 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69a0074bb0 con 0x7f69a00aa010 2023-03-03T09:28:58.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.833+0000 7f69927fc700 1 -- 172.21.15.7:0/2030061106 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f699c055890 con 0x7f69a00aa010 2023-03-03T09:28:58.834 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:28:58.834 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:28:58.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 -- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69840682f0 msgr2=0x7f698406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:58.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69840682f0 0x7f698406a7a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f69a00ac9e0 tx=0x7f6994008040 comp rx=0 tx=0).stop 2023-03-03T09:28:58.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 -- 172.21.15.7:0/2030061106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 msgr2=0x7f69a01ead50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:28:58.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 0x7f69a01ead50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f699c00c4d0 tx=0x7f699c00c890 comp rx=0 tx=0).stop 2023-03-03T09:28:58.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 -- 172.21.15.7:0/2030061106 shutdown_connections 2023-03-03T09:28:58.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69a0154f10 0x7f69a0149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:58.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69840682f0 0x7f698406a7a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:58.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69a00aa010 0x7f69a01ead50 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:58.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 --2- 172.21.15.7:0/2030061106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69a0151b20 0x7f69a01535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:28:58.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 -- 172.21.15.7:0/2030061106 >> 172.21.15.7:0/2030061106 conn(0x7f69a00c8900 msgr2=0x7f69a014b170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:28:58.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 -- 172.21.15.7:0/2030061106 shutdown_connections 2023-03-03T09:28:58.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:28:58.835+0000 7f69a7a85700 1 -- 172.21.15.7:0/2030061106 wait complete. 2023-03-03T09:28:58.838 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:28:58.848 INFO:tasks.mds_thrash.fs.[a]:waiting for 70 secs before thrashing 2023-03-03T09:29:15.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:15.538 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:41.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:29:41.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.713+0000 7f2a7fbaa700 1 Processor -- start 2023-03-03T09:29:41.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.713+0000 7f2a7fbaa700 1 -- start start 2023-03-03T09:29:41.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7fbaa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 0x7f2a780736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:41.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7fbaa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a78073c30 0x7f2a781581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:41.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7fbaa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7806cb70 con 0x7f2a780c2950 2023-03-03T09:29:41.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7fbaa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a7806c2f0 con 0x7f2a78073c30 2023-03-03T09:29:41.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7fbaa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7806ce50 con 0x7f2a78151b20 2023-03-03T09:29:41.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7e147700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a78073c30 0x7f2a781581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:41.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7d145700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2a780c2950 0x7f2a78157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50282/0 (socket says 172.21.15.7:50282) 2023-03-03T09:29:41.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7d145700 1 -- 172.21.15.7:0/3514471983 learned_addr learned my addr 172.21.15.7:0/3514471983 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:29:41.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7d946700 1 --2- 172.21.15.7:0/3514471983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 0x7f2a780736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:41.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7e147700 1 -- 172.21.15.7:0/3514471983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 msgr2=0x7f2a780736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:29:41.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7e147700 1 --2- 172.21.15.7:0/3514471983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 0x7f2a780736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:41.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7e147700 1 -- 172.21.15.7:0/3514471983 >> v1:172.21.15.7:6789/0 conn(0x7f2a780c2950 legacy=0x7f2a78157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:41.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7c944700 1 -- 172.21.15.7:0/3514471983 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333566588 0 0) 0x7f2a7806cb70 con 0x7f2a780c2950 2023-03-03T09:29:41.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7e147700 1 -- 172.21.15.7:0/3514471983 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7806d130 con 0x7f2a78073c30 2023-03-03T09:29:41.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.714+0000 7f2a7e147700 1 --2- 172.21.15.7:0/3514471983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a78073c30 0x7f2a781581f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009d90 tx=0x7f2a7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab6ac3a32f3293f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:29:41.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7c944700 1 -- 172.21.15.7:0/3514471983 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7000e070 con 0x7f2a78073c30 2023-03-03T09:29:41.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7c944700 1 -- 172.21.15.7:0/3514471983 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7000bbd0 con 0x7f2a78073c30 2023-03-03T09:29:41.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7c944700 1 -- 172.21.15.7:0/3514471983 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a70012430 con 0x7f2a78073c30 2023-03-03T09:29:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/3514471983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a78073c30 msgr2=0x7f2a781581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:41.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7fbaa700 1 --2- 172.21.15.7:0/3514471983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a78073c30 0x7f2a781581f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009d90 tx=0x7f2a7000b330 comp rx=0 tx=0).stop 2023-03-03T09:29:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/3514471983 shutdown_connections 2023-03-03T09:29:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7fbaa700 1 --2- 172.21.15.7:0/3514471983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a78073c30 0x7f2a781581f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7fbaa700 1 --2- 172.21.15.7:0/3514471983 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 0x7f2a780736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/3514471983 >> 172.21.15.7:0/3514471983 conn(0x7f2a780a94d0 msgr2=0x7f2a780a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:29:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/3514471983 shutdown_connections 2023-03-03T09:29:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.715+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/3514471983 wait complete. 2023-03-03T09:29:41.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.716+0000 7f2a7fbaa700 1 Processor -- start 2023-03-03T09:29:41.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.716+0000 7f2a7fbaa700 1 -- start start 2023-03-03T09:29:41.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7fbaa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a78073c30 0x7f2a781569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:41.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7fbaa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a780c2950 0x7f2a78157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:41.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7fbaa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 0x7f2a780bed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:41.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7fbaa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a780bf8e0 con 0x7f2a780c2950 2023-03-03T09:29:41.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7fbaa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a780c0620 con 0x7f2a78073c30 2023-03-03T09:29:41.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7fbaa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a78157920 con 0x7f2a78151b20 2023-03-03T09:29:41.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7d145700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a780c2950 0x7f2a78157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:41.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7d145700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a780c2950 0x7f2a78157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56308/0 (socket says 172.21.15.7:56308) 2023-03-03T09:29:41.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7d145700 1 -- 172.21.15.7:0/32834513 learned_addr learned my addr 172.21.15.7:0/32834513 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:29:41.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7e147700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 0x7f2a780bed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:41.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7d946700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a78073c30 0x7f2a781569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:41.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7d145700 1 -- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 msgr2=0x7f2a780bed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:41.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7d145700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 0x7f2a780bed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:41.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.717+0000 7f2a7d145700 1 -- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a78073c30 msgr2=0x7f2a781569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.718+0000 7f2a7d145700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a78073c30 0x7f2a781569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.718+0000 7f2a7d145700 1 -- 172.21.15.7:0/32834513 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a70009a70 con 0x7f2a780c2950 2023-03-03T09:29:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.718+0000 7f2a7d946700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a78073c30 0x7f2a781569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:29:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.718+0000 7f2a7e147700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 0x7f2a780bed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:29:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.718+0000 7f2a7d145700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a780c2950 0x7f2a78157240 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800c580 tx=0x7f2a6800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:29:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.718+0000 7f2a6e7fc700 1 -- 172.21.15.7:0/32834513 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68004c80 con 0x7f2a780c2950 2023-03-03T09:29:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.718+0000 7f2a6e7fc700 1 -- 172.21.15.7:0/32834513 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a68004e00 con 0x7f2a780c2950 2023-03-03T09:29:41.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.718+0000 7f2a6e7fc700 1 -- 172.21.15.7:0/32834513 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68007310 con 0x7f2a780c2950 2023-03-03T09:29:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.718+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/32834513 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a78158730 con 0x7f2a780c2950 2023-03-03T09:29:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.718+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/32834513 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7806d410 con 0x7f2a780c2950 2023-03-03T09:29:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.719+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/32834513 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a78151ef0 con 0x7f2a780c2950 2023-03-03T09:29:41.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.719+0000 7f2a6e7fc700 1 -- 172.21.15.7:0/32834513 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2a6801a020 con 0x7f2a780c2950 2023-03-03T09:29:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.720+0000 7f2a6e7fc700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a5c0682f0 0x7f2a5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:29:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.720+0000 7f2a6e7fc700 1 -- 172.21.15.7:0/32834513 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2a68085cf0 con 0x7f2a780c2950 2023-03-03T09:29:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.720+0000 7f2a7d946700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a5c0682f0 0x7f2a5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:29:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.721+0000 7f2a7d946700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a5c0682f0 0x7f2a5c06a7a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2a780c2030 tx=0x7f2a74008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:29:41.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.724+0000 7f2a6e7fc700 1 -- 172.21.15.7:0/32834513 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2a68052140 con 0x7f2a780c2950 2023-03-03T09:29:41.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.962+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/32834513 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a78074bb0 con 0x7f2a780c2950 2023-03-03T09:29:41.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.963+0000 7f2a6e7fc700 1 -- 172.21.15.7:0/32834513 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2a68055760 con 0x7f2a780c2950 2023-03-03T09:29:41.964 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:29:41.965 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:29:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a5c0682f0 msgr2=0x7f2a5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a5c0682f0 0x7f2a5c06a7a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2a780c2030 tx=0x7f2a74008040 comp rx=0 tx=0).stop 2023-03-03T09:29:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/32834513 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a780c2950 msgr2=0x7f2a78157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:29:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a780c2950 0x7f2a78157240 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800c580 tx=0x7f2a6800c940 comp rx=0 tx=0).stop 2023-03-03T09:29:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/32834513 shutdown_connections 2023-03-03T09:29:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a78151b20 0x7f2a780bed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a78073c30 0x7f2a781569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a5c0682f0 0x7f2a5c06a7a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 --2- 172.21.15.7:0/32834513 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a780c2950 0x7f2a78157240 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:29:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/32834513 >> 172.21.15.7:0/32834513 conn(0x7f2a780a94d0 msgr2=0x7f2a78071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:29:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/32834513 shutdown_connections 2023-03-03T09:29:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:29:41.965+0000 7f2a7fbaa700 1 -- 172.21.15.7:0/32834513 wait complete. 2023-03-03T09:29:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:29:41.978 INFO:tasks.mds_thrash.fs.[b]:waiting for 65 secs before thrashing 2023-03-03T09:29:45.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:29:45.609 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T09:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T09:30:08.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:30:08.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.986+0000 7f07b2a50700 1 Processor -- start 2023-03-03T09:30:08.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.986+0000 7f07b2a50700 1 -- start start 2023-03-03T09:30:08.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.987+0000 7f07b2a50700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 0x7f07ac0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:08.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.987+0000 7f07b2a50700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c00e0 0x7f07ac154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:08.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.987+0000 7f07b2a50700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac0cb280 con 0x7f07ac0cae00 2023-03-03T09:30:08.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.987+0000 7f07b2a50700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ac0c7310 con 0x7f07ac0c00e0 2023-03-03T09:30:08.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.987+0000 7f07b2a50700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac0bfd10 con 0x7f07ac0caa30 2023-03-03T09:30:08.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.987+0000 7f07b0fed700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f07ac0cae00 0x7f07ac0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:53830/0 (socket says 172.21.15.7:53830) 2023-03-03T09:30:08.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.987+0000 7f07abfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 0x7f07ac0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:08.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.987+0000 7f07b0fed700 1 -- 172.21.15.7:0/4061676422 learned_addr learned my addr 172.21.15.7:0/4061676422 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:30:08.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.987+0000 7f07ab7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c00e0 0x7f07ac154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:08.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07aaffd700 1 -- 172.21.15.7:0/4061676422 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774629070 0 0) 0x7f07ac0cb280 con 0x7f07ac0cae00 2023-03-03T09:30:08.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07aaffd700 1 -- 172.21.15.7:0/4061676422 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07980035b0 con 0x7f07ac0cae00 2023-03-03T09:30:08.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07abfff700 1 -- 172.21.15.7:0/4061676422 >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c00e0 msgr2=0x7f07ac154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:30:08.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07abfff700 1 --2- 172.21.15.7:0/4061676422 >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c00e0 0x7f07ac154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:08.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07abfff700 1 -- 172.21.15.7:0/4061676422 >> v1:172.21.15.143:6789/0 conn(0x7f07ac0cae00 legacy=0x7f07ac0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:08.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07abfff700 1 -- 172.21.15.7:0/4061676422 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac0bfa30 con 0x7f07ac0caa30 2023-03-03T09:30:08.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07ab7fe700 1 --2- 172.21.15.7:0/4061676422 >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c00e0 0x7f07ac154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:30:08.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07abfff700 1 --2- 172.21.15.7:0/4061676422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 0x7f07ac0c5c60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f079c009840 tx=0x7f079c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaed0148302cf354 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:30:08.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07aaffd700 1 -- 172.21.15.7:0/4061676422 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c011830 con 0x7f07ac0caa30 2023-03-03T09:30:08.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07aaffd700 1 -- 172.21.15.7:0/4061676422 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079c011990 con 0x7f07ac0caa30 2023-03-03T09:30:08.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.988+0000 7f07aaffd700 1 -- 172.21.15.7:0/4061676422 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c011d20 con 0x7f07ac0caa30 2023-03-03T09:30:08.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.989+0000 7f07b2a50700 1 -- 172.21.15.7:0/4061676422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 msgr2=0x7f07ac0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:08.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.989+0000 7f07b2a50700 1 --2- 172.21.15.7:0/4061676422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 0x7f07ac0c5c60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f079c009840 tx=0x7f079c009b50 comp rx=0 tx=0).stop 2023-03-03T09:30:08.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.989+0000 7f07b2a50700 1 -- 172.21.15.7:0/4061676422 shutdown_connections 2023-03-03T09:30:08.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.989+0000 7f07b2a50700 1 --2- 172.21.15.7:0/4061676422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 0x7f07ac0c5c60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:08.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.989+0000 7f07b2a50700 1 --2- 172.21.15.7:0/4061676422 >> v2:172.21.15.143:3300/0 conn(0x7f07ac0c00e0 0x7f07ac154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:08.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.989+0000 7f07b2a50700 1 -- 172.21.15.7:0/4061676422 >> 172.21.15.7:0/4061676422 conn(0x7f07ac066e40 msgr2=0x7f07ac071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:30:08.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.989+0000 7f07b2a50700 1 -- 172.21.15.7:0/4061676422 shutdown_connections 2023-03-03T09:30:08.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.989+0000 7f07b2a50700 1 -- 172.21.15.7:0/4061676422 wait complete. 2023-03-03T09:30:08.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.990+0000 7f07b2a50700 1 Processor -- start 2023-03-03T09:30:08.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.990+0000 7f07b2a50700 1 -- start start 2023-03-03T09:30:08.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b2a50700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0c00e0 0x7f07ac0c2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:08.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b2a50700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 0x7f07ac143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:08.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b2a50700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0cae00 0x7f07ac158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:08.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b2a50700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac143890 con 0x7f07ac0cae00 2023-03-03T09:30:08.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b2a50700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac0a9dc0 con 0x7f07ac0c00e0 2023-03-03T09:30:08.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b2a50700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac0aa380 con 0x7f07ac0caa30 2023-03-03T09:30:08.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b0fed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0cae00 0x7f07ac158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:08.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b0fed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0cae00 0x7f07ac158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56336/0 (socket says 172.21.15.7:56336) 2023-03-03T09:30:08.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b0fed700 1 -- 172.21.15.7:0/398618501 learned_addr learned my addr 172.21.15.7:0/398618501 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:30:08.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07abfff700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0c00e0 0x7f07ac0c2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:08.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07ab7fe700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 0x7f07ac143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:08.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b0fed700 1 -- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 msgr2=0x7f07ac143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:08.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b0fed700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 0x7f07ac143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:08.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b0fed700 1 -- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0c00e0 msgr2=0x7f07ac0c2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:08.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b0fed700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0c00e0 0x7f07ac0c2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:08.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.991+0000 7f07b0fed700 1 -- 172.21.15.7:0/398618501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0794009430 con 0x7f07ac0cae00 2023-03-03T09:30:08.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.992+0000 7f07b0fed700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0cae00 0x7f07ac158130 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f07a400caa0 tx=0x7f07a400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:30:08.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.992+0000 7f07a8ff9700 1 -- 172.21.15.7:0/398618501 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a400a9e0 con 0x7f07ac0cae00 2023-03-03T09:30:08.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.992+0000 7f07a8ff9700 1 -- 172.21.15.7:0/398618501 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a400ab40 con 0x7f07ac0cae00 2023-03-03T09:30:08.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.992+0000 7f07b2a50700 1 -- 172.21.15.7:0/398618501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079c0094a0 con 0x7f07ac0cae00 2023-03-03T09:30:08.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.992+0000 7f07a8ff9700 1 -- 172.21.15.7:0/398618501 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a4012400 con 0x7f07ac0cae00 2023-03-03T09:30:08.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.992+0000 7f07b2a50700 1 -- 172.21.15.7:0/398618501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ac1547f0 con 0x7f07ac0cae00 2023-03-03T09:30:09.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.993+0000 7f07b2a50700 1 -- 172.21.15.7:0/398618501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ac154510 con 0x7f07ac0cae00 2023-03-03T09:30:09.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.993+0000 7f07a8ff9700 1 -- 172.21.15.7:0/398618501 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f07a4019030 con 0x7f07ac0cae00 2023-03-03T09:30:09.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.994+0000 7f07a8ff9700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f078c0682f0 0x7f078c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:09.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.994+0000 7f07a8ff9700 1 -- 172.21.15.7:0/398618501 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f07a4085c20 con 0x7f07ac0cae00 2023-03-03T09:30:09.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.994+0000 7f07abfff700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f078c0682f0 0x7f078c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:09.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.995+0000 7f07abfff700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f078c0682f0 0x7f078c06a7a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f079c009fd0 tx=0x7f079c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:30:09.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:08.998+0000 7f07a8ff9700 1 -- 172.21.15.7:0/398618501 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f07a4052080 con 0x7f07ac0cae00 2023-03-03T09:30:09.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.235+0000 7f07b2a50700 1 -- 172.21.15.7:0/398618501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07ac154ad0 con 0x7f07ac0cae00 2023-03-03T09:30:09.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.236+0000 7f07a8ff9700 1 -- 172.21.15.7:0/398618501 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f07a40556a0 con 0x7f07ac0cae00 2023-03-03T09:30:09.238 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:30:09.239 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:30:09.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 -- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f078c0682f0 msgr2=0x7f078c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:09.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f078c0682f0 0x7f078c06a7a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f079c009fd0 tx=0x7f079c008030 comp rx=0 tx=0).stop 2023-03-03T09:30:09.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 -- 172.21.15.7:0/398618501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0cae00 msgr2=0x7f07ac158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:09.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0cae00 0x7f07ac158130 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f07a400caa0 tx=0x7f07a400cdb0 comp rx=0 tx=0).stop 2023-03-03T09:30:09.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 -- 172.21.15.7:0/398618501 shutdown_connections 2023-03-03T09:30:09.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f078c0682f0 0x7f078c06a7a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:09.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07ac0cae00 0x7f07ac158130 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:09.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07ac0c00e0 0x7f07ac0c2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:09.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 --2- 172.21.15.7:0/398618501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07ac0caa30 0x7f07ac143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:09.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 -- 172.21.15.7:0/398618501 >> 172.21.15.7:0/398618501 conn(0x7f07ac066e40 msgr2=0x7f07ac1513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:30:09.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 -- 172.21.15.7:0/398618501 shutdown_connections 2023-03-03T09:30:09.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:09.238+0000 7f07b2a50700 1 -- 172.21.15.7:0/398618501 wait complete. 2023-03-03T09:30:09.242 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:30:09.252 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2023-03-03T09:30:15.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:15.689 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:45.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:45.921 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:30:46.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:30:47.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.116+0000 7f53d22bf700 1 Processor -- start 2023-03-03T09:30:47.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.116+0000 7f53d22bf700 1 -- start start 2023-03-03T09:30:47.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.117+0000 7f53d22bf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f53cc06d4b0 0x7f53cc06d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:47.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.117+0000 7f53d22bf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc0c7b90 con 0x7f53cc0c1da0 2023-03-03T09:30:47.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.117+0000 7f53d22bf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53cc154640 con 0x7f53cc0c8020 2023-03-03T09:30:47.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.117+0000 7f53d22bf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53cc0c7330 con 0x7f53cc06d4b0 2023-03-03T09:30:47.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.117+0000 7f53cbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f53cc06d4b0 0x7f53cc06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:47.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.117+0000 7f53d085c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f53cc0c1da0 0x7f53cc139d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50344/0 (socket says 172.21.15.7:50344) 2023-03-03T09:30:47.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.117+0000 7f53d085c700 1 -- 172.21.15.7:0/3157063682 learned_addr learned my addr 172.21.15.7:0/3157063682 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:30:47.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.117+0000 7f53cbfff700 1 -- 172.21.15.7:0/3157063682 >> v1:172.21.15.143:6789/0 conn(0x7f53cc0c8020 legacy=0x7f53cc0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:30:47.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53cbfff700 1 -- 172.21.15.7:0/3157063682 >> v1:172.21.15.7:6789/0 conn(0x7f53cc0c1da0 legacy=0x7f53cc139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:47.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53cbfff700 1 -- 172.21.15.7:0/3157063682 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc0c7540 con 0x7f53cc06d4b0 2023-03-03T09:30:47.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53cbfff700 1 --2- 172.21.15.7:0/3157063682 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc06d4b0 0x7f53cc06d880 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009840 tx=0x7f53bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc0acb366158e73 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:30:47.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53caffd700 1 -- 172.21.15.7:0/3157063682 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc004230 con 0x7f53cc06d4b0 2023-03-03T09:30:47.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53caffd700 1 -- 172.21.15.7:0/3157063682 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53bc004390 con 0x7f53cc06d4b0 2023-03-03T09:30:47.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53caffd700 1 -- 172.21.15.7:0/3157063682 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc0036a0 con 0x7f53cc06d4b0 2023-03-03T09:30:47.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53d22bf700 1 -- 172.21.15.7:0/3157063682 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc06d4b0 msgr2=0x7f53cc06d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:47.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53d22bf700 1 --2- 172.21.15.7:0/3157063682 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc06d4b0 0x7f53cc06d880 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009840 tx=0x7f53bc009b50 comp rx=0 tx=0).stop 2023-03-03T09:30:47.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53d22bf700 1 -- 172.21.15.7:0/3157063682 shutdown_connections 2023-03-03T09:30:47.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53d22bf700 1 --2- 172.21.15.7:0/3157063682 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc06d4b0 0x7f53cc06d880 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:47.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.118+0000 7f53d22bf700 1 -- 172.21.15.7:0/3157063682 >> 172.21.15.7:0/3157063682 conn(0x7f53cc06c110 msgr2=0x7f53cc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:30:47.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.119+0000 7f53d22bf700 1 -- 172.21.15.7:0/3157063682 shutdown_connections 2023-03-03T09:30:47.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.119+0000 7f53d22bf700 1 -- 172.21.15.7:0/3157063682 wait complete. 2023-03-03T09:30:47.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.120+0000 7f53d22bf700 1 Processor -- start 2023-03-03T09:30:47.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.120+0000 7f53d22bf700 1 -- start start 2023-03-03T09:30:47.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.120+0000 7f53d22bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc06d4b0 0x7f53cc0acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:47.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.120+0000 7f53d22bf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc0c1da0 0x7f53cc136f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:47.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.120+0000 7f53d22bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc0c8020 0x7f53cc134720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:47.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.120+0000 7f53d22bf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc071420 con 0x7f53cc0c1da0 2023-03-03T09:30:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53d22bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc1de430 con 0x7f53cc06d4b0 2023-03-03T09:30:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53d22bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc071b20 con 0x7f53cc0c8020 2023-03-03T09:30:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc0c1da0 0x7f53cc136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc0c1da0 0x7f53cc136f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56370/0 (socket says 172.21.15.7:56370) 2023-03-03T09:30:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cb7fe700 1 -- 172.21.15.7:0/2862577450 learned_addr learned my addr 172.21.15.7:0/2862577450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:30:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cbfff700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc06d4b0 0x7f53cc0acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53d085c700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc0c8020 0x7f53cc134720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cb7fe700 1 -- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc0c8020 msgr2=0x7f53cc134720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cb7fe700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc0c8020 0x7f53cc134720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cb7fe700 1 -- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc06d4b0 msgr2=0x7f53cc0acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cb7fe700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc06d4b0 0x7f53cc0acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cb7fe700 1 -- 172.21.15.7:0/2862577450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0094a0 con 0x7f53cc0c1da0 2023-03-03T09:30:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cbfff700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc06d4b0 0x7f53cc0acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:30:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53cb7fe700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc0c1da0 0x7f53cc136f90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f53c000c4d0 tx=0x7f53c000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:30:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53c8ff9700 1 -- 172.21.15.7:0/2862577450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c0011790 con 0x7f53cc0c1da0 2023-03-03T09:30:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53c8ff9700 1 -- 172.21.15.7:0/2862577450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c00118f0 con 0x7f53cc0c1da0 2023-03-03T09:30:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53c8ff9700 1 -- 172.21.15.7:0/2862577450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c0011ca0 con 0x7f53cc0c1da0 2023-03-03T09:30:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53d22bf700 1 -- 172.21.15.7:0/2862577450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53cc06ddc0 con 0x7f53cc0c1da0 2023-03-03T09:30:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.121+0000 7f53d22bf700 1 -- 172.21.15.7:0/2862577450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53cc0c7820 con 0x7f53cc0c1da0 2023-03-03T09:30:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.122+0000 7f53d22bf700 1 -- 172.21.15.7:0/2862577450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53cc06e210 con 0x7f53cc0c1da0 2023-03-03T09:30:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.123+0000 7f53c8ff9700 1 -- 172.21.15.7:0/2862577450 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f53c0010050 con 0x7f53cc0c1da0 2023-03-03T09:30:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.123+0000 7f53c8ff9700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53b80682f0 0x7f53b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:30:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.123+0000 7f53c8ff9700 1 -- 172.21.15.7:0/2862577450 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f53c0085fc0 con 0x7f53cc0c1da0 2023-03-03T09:30:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.124+0000 7f53cbfff700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53b80682f0 0x7f53b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:30:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.124+0000 7f53cbfff700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53b80682f0 0x7f53b806a7a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f53bc0090b0 tx=0x7f53bc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:30:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.128+0000 7f53c8ff9700 1 -- 172.21.15.7:0/2862577450 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f53c0052270 con 0x7f53cc0c1da0 2023-03-03T09:30:47.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.366+0000 7f53d22bf700 1 -- 172.21.15.7:0/2862577450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53cc074bb0 con 0x7f53cc0c1da0 2023-03-03T09:30:47.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.367+0000 7f53c8ff9700 1 -- 172.21.15.7:0/2862577450 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f53c0055890 con 0x7f53cc0c1da0 2023-03-03T09:30:47.368 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:30:47.369 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:30:47.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 -- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53b80682f0 msgr2=0x7f53b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:47.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53b80682f0 0x7f53b806a7a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f53bc0090b0 tx=0x7f53bc008030 comp rx=0 tx=0).stop 2023-03-03T09:30:47.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 -- 172.21.15.7:0/2862577450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc0c1da0 msgr2=0x7f53cc136f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:30:47.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc0c1da0 0x7f53cc136f90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f53c000c4d0 tx=0x7f53c000c890 comp rx=0 tx=0).stop 2023-03-03T09:30:47.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 -- 172.21.15.7:0/2862577450 shutdown_connections 2023-03-03T09:30:47.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53cc0c8020 0x7f53cc134720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:47.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53cc06d4b0 0x7f53cc0acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:47.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53b80682f0 0x7f53b806a7a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:47.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 --2- 172.21.15.7:0/2862577450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53cc0c1da0 0x7f53cc136f90 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:30:47.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 -- 172.21.15.7:0/2862577450 >> 172.21.15.7:0/2862577450 conn(0x7f53cc06c110 msgr2=0x7f53cc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:30:47.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 -- 172.21.15.7:0/2862577450 shutdown_connections 2023-03-03T09:30:47.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:30:47.369+0000 7f53d22bf700 1 -- 172.21.15.7:0/2862577450 wait complete. 2023-03-03T09:30:47.372 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:30:47.382 INFO:tasks.mds_thrash.fs.[b]:waiting for 105 secs before thrashing 2023-03-03T09:31:15.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:15.992 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:17.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:31:17.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.389+0000 7f4b67773700 1 Processor -- start 2023-03-03T09:31:17.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.390+0000 7f4b67773700 1 -- start start 2023-03-03T09:31:17.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.390+0000 7f4b67773700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4b6006d870 0x7f4b601550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:17.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.390+0000 7f4b67773700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b600c7a60 con 0x7f4b600c1da0 2023-03-03T09:31:17.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.390+0000 7f4b67773700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b600c3620 con 0x7f4b600c3e30 2023-03-03T09:31:17.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.390+0000 7f4b67773700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b600c2af0 con 0x7f4b6006d870 2023-03-03T09:31:17.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b64d0e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4b600c1da0 0x7f4b600c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50376/0 (socket says 172.21.15.7:50376) 2023-03-03T09:31:17.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b64d0e700 1 -- 172.21.15.7:0/2286099868 learned_addr learned my addr 172.21.15.7:0/2286099868 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:31:17.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b6550f700 1 --2- 172.21.15.7:0/2286099868 >> v2:172.21.15.143:3300/0 conn(0x7f4b6006d870 0x7f4b601550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:17.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b53fff700 1 -- 172.21.15.7:0/2286099868 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3081186038 0 0) 0x7f4b600c7a60 con 0x7f4b600c1da0 2023-03-03T09:31:17.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b53fff700 1 -- 172.21.15.7:0/2286099868 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b4c0035b0 con 0x7f4b600c1da0 2023-03-03T09:31:17.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b53fff700 1 -- 172.21.15.7:0/2286099868 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258114508 0 0) 0x7f4b600c3620 con 0x7f4b600c3e30 2023-03-03T09:31:17.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b53fff700 1 -- 172.21.15.7:0/2286099868 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b600c7a60 con 0x7f4b600c3e30 2023-03-03T09:31:17.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b6550f700 1 -- 172.21.15.7:0/2286099868 >> v1:172.21.15.143:6789/0 conn(0x7f4b600c3e30 legacy=0x7f4b6015ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:17.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b6550f700 1 -- 172.21.15.7:0/2286099868 >> v1:172.21.15.7:6789/0 conn(0x7f4b600c1da0 legacy=0x7f4b600c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:17.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b53fff700 1 -- 172.21.15.7:0/2286099868 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 587196879 0 0) 0x7f4b4c0035b0 con 0x7f4b600c1da0 2023-03-03T09:31:17.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b6550f700 1 -- 172.21.15.7:0/2286099868 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b601545e0 con 0x7f4b6006d870 2023-03-03T09:31:17.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.391+0000 7f4b6550f700 1 --2- 172.21.15.7:0/2286099868 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b6006d870 0x7f4b601550c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009840 tx=0x7f4b5c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24b9b3952493d10a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:31:17.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.392+0000 7f4b53fff700 1 -- 172.21.15.7:0/2286099868 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b54001670 con 0x7f4b6006d870 2023-03-03T09:31:17.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.392+0000 7f4b53fff700 1 -- 172.21.15.7:0/2286099868 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5c004030 con 0x7f4b6006d870 2023-03-03T09:31:17.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.392+0000 7f4b53fff700 1 -- 172.21.15.7:0/2286099868 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c0043c0 con 0x7f4b6006d870 2023-03-03T09:31:17.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.392+0000 7f4b67773700 1 -- 172.21.15.7:0/2286099868 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b6006d870 msgr2=0x7f4b601550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:17.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.392+0000 7f4b67773700 1 --2- 172.21.15.7:0/2286099868 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b6006d870 0x7f4b601550c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009840 tx=0x7f4b5c009b50 comp rx=0 tx=0).stop 2023-03-03T09:31:17.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.392+0000 7f4b67773700 1 -- 172.21.15.7:0/2286099868 shutdown_connections 2023-03-03T09:31:17.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.392+0000 7f4b67773700 1 --2- 172.21.15.7:0/2286099868 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b6006d870 0x7f4b601550c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:17.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.392+0000 7f4b67773700 1 -- 172.21.15.7:0/2286099868 >> 172.21.15.7:0/2286099868 conn(0x7f4b6006c110 msgr2=0x7f4b6006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:31:17.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.392+0000 7f4b67773700 1 -- 172.21.15.7:0/2286099868 shutdown_connections 2023-03-03T09:31:17.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.392+0000 7f4b67773700 1 -- 172.21.15.7:0/2286099868 wait complete. 2023-03-03T09:31:17.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.393+0000 7f4b67773700 1 Processor -- start 2023-03-03T09:31:17.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.393+0000 7f4b67773700 1 -- start start 2023-03-03T09:31:17.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b67773700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b6006d870 0x7f4b60151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:17.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b67773700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b600c1da0 0x7f4b60151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:17.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b67773700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b600c3e30 0x7f4b601eb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:17.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b67773700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b60150350 con 0x7f4b600c1da0 2023-03-03T09:31:17.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b67773700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6014f6d0 con 0x7f4b600c3e30 2023-03-03T09:31:17.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b67773700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6014f9b0 con 0x7f4b6006d870 2023-03-03T09:31:17.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b64d0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b600c1da0 0x7f4b60151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:17.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b64d0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b600c1da0 0x7f4b60151650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56402/0 (socket says 172.21.15.7:56402) 2023-03-03T09:31:17.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b64d0e700 1 -- 172.21.15.7:0/2992119162 learned_addr learned my addr 172.21.15.7:0/2992119162 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:31:17.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b6550f700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b6006d870 0x7f4b60151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:17.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b65d10700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b600c3e30 0x7f4b601eb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:17.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b64d0e700 1 -- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b6006d870 msgr2=0x7f4b60151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:17.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b64d0e700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b6006d870 0x7f4b60151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:17.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b64d0e700 1 -- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b600c3e30 msgr2=0x7f4b601eb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:17.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b64d0e700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b600c3e30 0x7f4b601eb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:17.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.394+0000 7f4b64d0e700 1 -- 172.21.15.7:0/2992119162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c0094a0 con 0x7f4b600c1da0 2023-03-03T09:31:17.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.395+0000 7f4b64d0e700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b600c1da0 0x7f4b60151650 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400c750 tx=0x7f4b5400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:31:17.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.395+0000 7f4b51ffb700 1 -- 172.21.15.7:0/2992119162 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b54007560 con 0x7f4b600c1da0 2023-03-03T09:31:17.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.395+0000 7f4b51ffb700 1 -- 172.21.15.7:0/2992119162 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b540076c0 con 0x7f4b600c1da0 2023-03-03T09:31:17.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.395+0000 7f4b67773700 1 -- 172.21.15.7:0/2992119162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b60154ba0 con 0x7f4b600c1da0 2023-03-03T09:31:17.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.395+0000 7f4b51ffb700 1 -- 172.21.15.7:0/2992119162 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b54004ab0 con 0x7f4b600c1da0 2023-03-03T09:31:17.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.395+0000 7f4b67773700 1 -- 172.21.15.7:0/2992119162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b601548c0 con 0x7f4b600c1da0 2023-03-03T09:31:17.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.396+0000 7f4b67773700 1 -- 172.21.15.7:0/2992119162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6006dc40 con 0x7f4b600c1da0 2023-03-03T09:31:17.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.396+0000 7f4b51ffb700 1 -- 172.21.15.7:0/2992119162 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4b54010070 con 0x7f4b600c1da0 2023-03-03T09:31:17.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.397+0000 7f4b51ffb700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b44068320 0x7f4b4406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.397+0000 7f4b51ffb700 1 -- 172.21.15.7:0/2992119162 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4b54085300 con 0x7f4b600c1da0 2023-03-03T09:31:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.397+0000 7f4b6550f700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b44068320 0x7f4b4406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.398+0000 7f4b6550f700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b44068320 0x7f4b4406a7d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00a010 tx=0x7f4b5c013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:31:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.401+0000 7f4b51ffb700 1 -- 172.21.15.7:0/2992119162 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b540515c0 con 0x7f4b600c1da0 2023-03-03T09:31:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.639+0000 7f4b67773700 1 -- 172.21.15.7:0/2992119162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b60074bb0 con 0x7f4b600c1da0 2023-03-03T09:31:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.640+0000 7f4b51ffb700 1 -- 172.21.15.7:0/2992119162 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4b54018090 con 0x7f4b600c1da0 2023-03-03T09:31:17.642 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:31:17.643 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:31:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 -- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b44068320 msgr2=0x7f4b4406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b44068320 0x7f4b4406a7d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00a010 tx=0x7f4b5c013040 comp rx=0 tx=0).stop 2023-03-03T09:31:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 -- 172.21.15.7:0/2992119162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b600c1da0 msgr2=0x7f4b60151650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b600c1da0 0x7f4b60151650 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4b5400c750 tx=0x7f4b5400cb10 comp rx=0 tx=0).stop 2023-03-03T09:31:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 -- 172.21.15.7:0/2992119162 shutdown_connections 2023-03-03T09:31:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b600c3e30 0x7f4b601eb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b6006d870 0x7f4b60151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b44068320 0x7f4b4406a7d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 --2- 172.21.15.7:0/2992119162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b600c1da0 0x7f4b60151650 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 -- 172.21.15.7:0/2992119162 >> 172.21.15.7:0/2992119162 conn(0x7f4b6006c110 msgr2=0x7f4b60070310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:31:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 -- 172.21.15.7:0/2992119162 shutdown_connections 2023-03-03T09:31:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:17.642+0000 7f4b67773700 1 -- 172.21.15.7:0/2992119162 wait complete. 2023-03-03T09:31:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:31:17.656 INFO:tasks.mds_thrash.fs.[a]:waiting for 32 secs before thrashing 2023-03-03T09:31:46.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:46.063 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:31:49.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:31:49.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.794+0000 7fe126304700 1 Processor -- start 2023-03-03T09:31:49.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.794+0000 7fe126304700 1 -- start start 2023-03-03T09:31:49.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.795+0000 7fe126304700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12014f9d0 0x7fe120156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:49.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.795+0000 7fe126304700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1200abe20 0x7fe1201529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:49.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.795+0000 7fe126304700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1200c2f30 con 0x7fe12014fda0 2023-03-03T09:31:49.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.795+0000 7fe126304700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1200c31d0 con 0x7fe1200abe20 2023-03-03T09:31:49.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.795+0000 7fe126304700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1201524a0 con 0x7fe12014f9d0 2023-03-03T09:31:49.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.795+0000 7fe1248a1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1200abe20 0x7fe1201529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:49.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.795+0000 7fe11f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe12014fda0 0x7fe120071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50400/0 (socket says 172.21.15.7:50400) 2023-03-03T09:31:49.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.795+0000 7fe11f7fe700 1 -- 172.21.15.7:0/1415886281 learned_addr learned my addr 172.21.15.7:0/1415886281 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:31:49.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.795+0000 7fe11ffff700 1 --2- 172.21.15.7:0/1415886281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12014f9d0 0x7fe120156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:49.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe1248a1700 1 -- 172.21.15.7:0/1415886281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12014f9d0 msgr2=0x7fe120156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:31:49.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe1248a1700 1 --2- 172.21.15.7:0/1415886281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12014f9d0 0x7fe120156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:49.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe1248a1700 1 -- 172.21.15.7:0/1415886281 >> v1:172.21.15.7:6789/0 conn(0x7fe12014fda0 legacy=0x7fe120071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:49.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe1248a1700 1 -- 172.21.15.7:0/1415886281 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200c34b0 con 0x7fe1200abe20 2023-03-03T09:31:49.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe11effd700 1 -- 172.21.15.7:0/1415886281 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348174777 0 0) 0x7fe1200c2f30 con 0x7fe12014fda0 2023-03-03T09:31:49.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe1248a1700 1 --2- 172.21.15.7:0/1415886281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1200abe20 0x7fe1201529c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe118009d90 tx=0x7fe11800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d75e33d575c8826 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:31:49.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe11effd700 1 -- 172.21.15.7:0/1415886281 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11800e070 con 0x7fe1200abe20 2023-03-03T09:31:49.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe11effd700 1 -- 172.21.15.7:0/1415886281 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11800bbd0 con 0x7fe1200abe20 2023-03-03T09:31:49.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe11effd700 1 -- 172.21.15.7:0/1415886281 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118012430 con 0x7fe1200abe20 2023-03-03T09:31:49.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe126304700 1 -- 172.21.15.7:0/1415886281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1200abe20 msgr2=0x7fe1201529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:49.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.796+0000 7fe126304700 1 --2- 172.21.15.7:0/1415886281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1200abe20 0x7fe1201529c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fe118009d90 tx=0x7fe11800b330 comp rx=0 tx=0).stop 2023-03-03T09:31:49.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.797+0000 7fe126304700 1 -- 172.21.15.7:0/1415886281 shutdown_connections 2023-03-03T09:31:49.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.797+0000 7fe126304700 1 --2- 172.21.15.7:0/1415886281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1200abe20 0x7fe1201529c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:49.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.797+0000 7fe126304700 1 --2- 172.21.15.7:0/1415886281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12014f9d0 0x7fe120156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:49.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.797+0000 7fe126304700 1 -- 172.21.15.7:0/1415886281 >> 172.21.15.7:0/1415886281 conn(0x7fe1200c8900 msgr2=0x7fe1200c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:31:49.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.797+0000 7fe126304700 1 -- 172.21.15.7:0/1415886281 shutdown_connections 2023-03-03T09:31:49.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.797+0000 7fe126304700 1 -- 172.21.15.7:0/1415886281 wait complete. 2023-03-03T09:31:49.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.798+0000 7fe126304700 1 Processor -- start 2023-03-03T09:31:49.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.798+0000 7fe126304700 1 -- start start 2023-03-03T09:31:49.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe126304700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1200abe20 0x7fe120153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:49.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe126304700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12014f9d0 0x7fe12014ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe126304700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe1200bd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe126304700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12014f380 con 0x7fe12014fda0 2023-03-03T09:31:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe126304700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200bedf0 con 0x7fe12014f9d0 2023-03-03T09:31:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe126304700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12014eb60 con 0x7fe1200abe20 2023-03-03T09:31:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe1248a1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe1200bd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe1248a1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe1200bd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56426/0 (socket says 172.21.15.7:56426) 2023-03-03T09:31:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe1248a1700 1 -- 172.21.15.7:0/2127037742 learned_addr learned my addr 172.21.15.7:0/2127037742 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:31:49.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe11ffff700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1200abe20 0x7fe120153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:49.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe11f7fe700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12014f9d0 0x7fe12014ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:49.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe1248a1700 1 -- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1200abe20 msgr2=0x7fe120153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:49.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe1248a1700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1200abe20 0x7fe120153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:49.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe1248a1700 1 -- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12014f9d0 msgr2=0x7fe12014ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:49.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe1248a1700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12014f9d0 0x7fe12014ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:49.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe1248a1700 1 -- 172.21.15.7:0/2127037742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118009a70 con 0x7fe12014fda0 2023-03-03T09:31:49.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.799+0000 7fe1248a1700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe1200bd640 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe120072520 tx=0x7fe118004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:31:49.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.800+0000 7fe11cff9700 1 -- 172.21.15.7:0/2127037742 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118012430 con 0x7fe12014fda0 2023-03-03T09:31:49.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.800+0000 7fe11cff9700 1 -- 172.21.15.7:0/2127037742 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe118020070 con 0x7fe12014fda0 2023-03-03T09:31:49.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.800+0000 7fe126304700 1 -- 172.21.15.7:0/2127037742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1200ac350 con 0x7fe12014fda0 2023-03-03T09:31:49.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.800+0000 7fe11cff9700 1 -- 172.21.15.7:0/2127037742 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118012e70 con 0x7fe12014fda0 2023-03-03T09:31:49.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.800+0000 7fe126304700 1 -- 172.21.15.7:0/2127037742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1200ab630 con 0x7fe12014fda0 2023-03-03T09:31:49.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.800+0000 7fe126304700 1 -- 172.21.15.7:0/2127037742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe120071fa0 con 0x7fe12014fda0 2023-03-03T09:31:49.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.801+0000 7fe11cff9700 1 -- 172.21.15.7:0/2127037742 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe118004040 con 0x7fe12014fda0 2023-03-03T09:31:49.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.802+0000 7fe11cff9700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1000682f0 0x7fe10006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:31:49.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.802+0000 7fe11cff9700 1 -- 172.21.15.7:0/2127037742 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe118022070 con 0x7fe12014fda0 2023-03-03T09:31:49.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.802+0000 7fe11ffff700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1000682f0 0x7fe10006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:31:49.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.803+0000 7fe11ffff700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1000682f0 0x7fe10006a7a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe12006d510 tx=0x7fe110008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:31:49.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:49.806+0000 7fe11cff9700 1 -- 172.21.15.7:0/2127037742 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe118054b10 con 0x7fe12014fda0 2023-03-03T09:31:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.043+0000 7fe126304700 1 -- 172.21.15.7:0/2127037742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe120074bb0 con 0x7fe12014fda0 2023-03-03T09:31:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.044+0000 7fe11cff9700 1 -- 172.21.15.7:0/2127037742 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe118058130 con 0x7fe12014fda0 2023-03-03T09:31:50.046 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:31:50.046 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:31:50.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 -- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1000682f0 msgr2=0x7fe10006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:50.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1000682f0 0x7fe10006a7a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe12006d510 tx=0x7fe110008040 comp rx=0 tx=0).stop 2023-03-03T09:31:50.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 -- 172.21.15.7:0/2127037742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 msgr2=0x7fe1200bd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:31:50.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe1200bd640 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe120072520 tx=0x7fe118004ad0 comp rx=0 tx=0).stop 2023-03-03T09:31:50.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 -- 172.21.15.7:0/2127037742 shutdown_connections 2023-03-03T09:31:50.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1000682f0 0x7fe10006a7a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:50.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe1200bd640 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:50.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1200abe20 0x7fe120153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:50.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 --2- 172.21.15.7:0/2127037742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12014f9d0 0x7fe12014ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:31:50.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 -- 172.21.15.7:0/2127037742 >> 172.21.15.7:0/2127037742 conn(0x7fe1200c8900 msgr2=0x7fe120151820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:31:50.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 -- 172.21.15.7:0/2127037742 shutdown_connections 2023-03-03T09:31:50.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:31:50.046+0000 7fe126304700 1 -- 172.21.15.7:0/2127037742 wait complete. 2023-03-03T09:31:50.049 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:31:50.059 INFO:tasks.mds_thrash.fs.[a]:waiting for 107 secs before thrashing 2023-03-03T09:32:16.281 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:16.284 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:32.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:32:32.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.519+0000 7fad48994700 1 Processor -- start 2023-03-03T09:32:32.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.519+0000 7fad48994700 1 -- start start 2023-03-03T09:32:32.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad48994700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fad4406d870 0x7fad441550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:32.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad48994700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad440c1da0 0x7fad440c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:32.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad48994700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad440c3e30 0x7fad440c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:32.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad48994700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad440c2af0 con 0x7fad4406d870 2023-03-03T09:32:32.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad48994700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad440c3620 con 0x7fad440c1da0 2023-03-03T09:32:32.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad48994700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad441545e0 con 0x7fad440c3e30 2023-03-03T09:32:32.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad4259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fad4406d870 0x7fad441550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:32.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad4259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fad4406d870 0x7fad441550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56476/0 (socket says 172.21.15.7:56476) 2023-03-03T09:32:32.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad4259c700 1 -- 172.21.15.7:0/588028698 learned_addr learned my addr 172.21.15.7:0/588028698 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:32:32.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad42d9d700 1 --2- 172.21.15.7:0/588028698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad440c3e30 0x7fad440c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:32.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad4259c700 1 -- 172.21.15.7:0/588028698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad440c3e30 msgr2=0x7fad440c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:32:32.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad4259c700 1 --2- 172.21.15.7:0/588028698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad440c3e30 0x7fad440c4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad41d9b700 1 --2- 172.21.15.7:0/588028698 >> v2:172.21.15.143:3300/0 conn(0x7fad440c1da0 0x7fad440c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:32.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad4259c700 1 -- 172.21.15.7:0/588028698 >> v2:172.21.15.143:3300/0 conn(0x7fad440c1da0 msgr2=0x7fad440c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:32:32.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad4259c700 1 --2- 172.21.15.7:0/588028698 >> v2:172.21.15.143:3300/0 conn(0x7fad440c1da0 0x7fad440c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.520+0000 7fad4259c700 1 -- 172.21.15.7:0/588028698 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad441548c0 con 0x7fad4406d870 2023-03-03T09:32:32.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad4259c700 1 --2- 172.21.15.7:0/588028698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 0x7fad441550c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fad440c33a0 tx=0x7fad34009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b5f9b11b60170f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:32:32.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad4159a700 1 -- 172.21.15.7:0/588028698 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3400b870 con 0x7fad4406d870 2023-03-03T09:32:32.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad4159a700 1 -- 172.21.15.7:0/588028698 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3400b9d0 con 0x7fad4406d870 2023-03-03T09:32:32.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad4159a700 1 -- 172.21.15.7:0/588028698 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3400bd60 con 0x7fad4406d870 2023-03-03T09:32:32.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad48994700 1 -- 172.21.15.7:0/588028698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 msgr2=0x7fad441550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:32.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad48994700 1 --2- 172.21.15.7:0/588028698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 0x7fad441550c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fad440c33a0 tx=0x7fad34009bd0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad48994700 1 -- 172.21.15.7:0/588028698 shutdown_connections 2023-03-03T09:32:32.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad48994700 1 --2- 172.21.15.7:0/588028698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad440c3e30 0x7fad440c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad48994700 1 --2- 172.21.15.7:0/588028698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 0x7fad441550c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad48994700 1 --2- 172.21.15.7:0/588028698 >> v2:172.21.15.143:3300/0 conn(0x7fad440c1da0 0x7fad440c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.521+0000 7fad48994700 1 -- 172.21.15.7:0/588028698 >> 172.21.15.7:0/588028698 conn(0x7fad4406c110 msgr2=0x7fad4406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:32:32.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.522+0000 7fad48994700 1 -- 172.21.15.7:0/588028698 shutdown_connections 2023-03-03T09:32:32.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.522+0000 7fad48994700 1 -- 172.21.15.7:0/588028698 wait complete. 2023-03-03T09:32:32.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.523+0000 7fad48994700 1 Processor -- start 2023-03-03T09:32:32.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.523+0000 7fad48994700 1 -- start start 2023-03-03T09:32:32.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad48994700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 0x7fad441575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:32.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad48994700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad440c1da0 0x7fad441ed860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:32.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad48994700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad440c3e30 0x7fad44146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:32.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad48994700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad441ee2e0 con 0x7fad4406d870 2023-03-03T09:32:32.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad48994700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad44156dc0 con 0x7fad440c3e30 2023-03-03T09:32:32.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad48994700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad441570a0 con 0x7fad440c1da0 2023-03-03T09:32:32.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad4259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 0x7fad441575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:32.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad4259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 0x7fad441575c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56482/0 (socket says 172.21.15.7:56482) 2023-03-03T09:32:32.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad4259c700 1 -- 172.21.15.7:0/598254503 learned_addr learned my addr 172.21.15.7:0/598254503 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:32:32.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad4259c700 1 -- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad440c1da0 msgr2=0x7fad441ed860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:32.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad4259c700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad440c1da0 0x7fad441ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad4259c700 1 -- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad440c3e30 msgr2=0x7fad44146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:32.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad42d9d700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad440c3e30 0x7fad44146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:32.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad4259c700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad440c3e30 0x7fad44146c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad4259c700 1 -- 172.21.15.7:0/598254503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad34009550 con 0x7fad4406d870 2023-03-03T09:32:32.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad4259c700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 0x7fad441575c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fad34014040 tx=0x7fad34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:32:32.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad3affd700 1 -- 172.21.15.7:0/598254503 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34004520 con 0x7fad4406d870 2023-03-03T09:32:32.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.524+0000 7fad3affd700 1 -- 172.21.15.7:0/598254503 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3401e070 con 0x7fad4406d870 2023-03-03T09:32:32.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.525+0000 7fad48994700 1 -- 172.21.15.7:0/598254503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4406dc40 con 0x7fad4406d870 2023-03-03T09:32:32.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.525+0000 7fad3affd700 1 -- 172.21.15.7:0/598254503 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad340038e0 con 0x7fad4406d870 2023-03-03T09:32:32.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.525+0000 7fad48994700 1 -- 172.21.15.7:0/598254503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad44154ba0 con 0x7fad4406d870 2023-03-03T09:32:32.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.526+0000 7fad48994700 1 -- 172.21.15.7:0/598254503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad440c38e0 con 0x7fad4406d870 2023-03-03T09:32:32.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.526+0000 7fad3affd700 1 -- 172.21.15.7:0/598254503 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fad3400f050 con 0x7fad4406d870 2023-03-03T09:32:32.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.526+0000 7fad3affd700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad240682f0 0x7fad2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:32:32.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.527+0000 7fad3affd700 1 -- 172.21.15.7:0/598254503 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fad34020070 con 0x7fad4406d870 2023-03-03T09:32:32.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.527+0000 7fad41d9b700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad240682f0 0x7fad2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:32:32.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.527+0000 7fad41d9b700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad240682f0 0x7fad2406a7a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fad2c006ba0 tx=0x7fad2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:32:32.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.531+0000 7fad3affd700 1 -- 172.21.15.7:0/598254503 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fad34053b50 con 0x7fad4406d870 2023-03-03T09:32:32.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.772+0000 7fad48994700 1 -- 172.21.15.7:0/598254503 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad44153350 con 0x7fad4406d870 2023-03-03T09:32:32.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.773+0000 7fad3affd700 1 -- 172.21.15.7:0/598254503 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fad34057170 con 0x7fad4406d870 2023-03-03T09:32:32.775 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:32:32.775 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:32:32.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 -- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad240682f0 msgr2=0x7fad2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:32.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad240682f0 0x7fad2406a7a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fad2c006ba0 tx=0x7fad2c008040 comp rx=0 tx=0).stop 2023-03-03T09:32:32.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 -- 172.21.15.7:0/598254503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 msgr2=0x7fad441575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:32:32.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 0x7fad441575c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fad34014040 tx=0x7fad34004750 comp rx=0 tx=0).stop 2023-03-03T09:32:32.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 -- 172.21.15.7:0/598254503 shutdown_connections 2023-03-03T09:32:32.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad440c3e30 0x7fad44146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad240682f0 0x7fad2406a7a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad4406d870 0x7fad441575c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 --2- 172.21.15.7:0/598254503 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad440c1da0 0x7fad441ed860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:32:32.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 -- 172.21.15.7:0/598254503 >> 172.21.15.7:0/598254503 conn(0x7fad4406c110 msgr2=0x7fad440c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:32:32.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 -- 172.21.15.7:0/598254503 shutdown_connections 2023-03-03T09:32:32.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:32:32.775+0000 7fad48994700 1 -- 172.21.15.7:0/598254503 wait complete. 2023-03-03T09:32:32.779 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:32:32.789 INFO:tasks.mds_thrash.fs.[b]:waiting for 75 secs before thrashing 2023-03-03T09:32:46.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:32:46.356 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:16.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:16.470 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:37.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:33:37.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.197+0000 7f18fdffa700 1 Processor -- start 2023-03-03T09:33:37.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.197+0000 7f18fdffa700 1 -- start start 2023-03-03T09:33:37.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.198+0000 7f18fdffa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f80aad20 0x7f18f8157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:37.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.198+0000 7f18fdffa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f81588f0 con 0x7f18f814b020 2023-03-03T09:33:37.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.198+0000 7f18fdffa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f80728b0 con 0x7f18f80aa030 2023-03-03T09:33:37.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.198+0000 7f18fdffa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f8155070 con 0x7f18f80aad20 2023-03-03T09:33:37.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.198+0000 7f18f77fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f18f814b020 0x7f18f80717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50500/0 (socket says 172.21.15.7:50500) 2023-03-03T09:33:37.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.198+0000 7f18f77fe700 1 -- 172.21.15.7:0/4046964351 learned_addr learned my addr 172.21.15.7:0/4046964351 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:33:37.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f6ffd700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f80aad20 0x7f18f8157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:37.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872541453 0 0) 0x7f18f81588f0 con 0x7f18f814b020 2023-03-03T09:33:37.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e00035b0 con 0x7f18f814b020 2023-03-03T09:33:37.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938368391 0 0) 0x7f18f80728b0 con 0x7f18f80aa030 2023-03-03T09:33:37.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f81588f0 con 0x7f18f80aa030 2023-03-03T09:33:37.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1576721012 0 0) 0x7f18e00035b0 con 0x7f18f814b020 2023-03-03T09:33:37.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f80aad20 msgr2=0x7f18f8157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:33:37.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f80aad20 0x7f18f8157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:37.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 >> v1:172.21.15.143:6789/0 conn(0x7f18f80aa030 legacy=0x7f18f81581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:37.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f806f5b0 con 0x7f18f814b020 2023-03-03T09:33:37.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f18e8001670 con 0x7f18f814b020 2023-03-03T09:33:37.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 >> v1:172.21.15.7:6789/0 conn(0x7f18f814b020 legacy=0x7f18f80717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:37.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18e0003c30 0x7f18e0004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:37.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f80aad20 0x7f18e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:37.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e0004d90 0x7f18e0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:37.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f8155070 con 0x7f18e0004d90 2023-03-03T09:33:37.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e000bba0 con 0x7f18e0003c30 2023-03-03T09:33:37.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e000bd60 con 0x7f18f80aad20 2023-03-03T09:33:37.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f77fe700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18e0003c30 0x7f18e0004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:37.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f6ffd700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f80aad20 0x7f18e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:37.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.199+0000 7f18f7fff700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e0004d90 0x7f18e0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:37.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f7fff700 1 -- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f80aad20 msgr2=0x7f18e0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:33:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f7fff700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f80aad20 0x7f18e0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f7fff700 1 -- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18e0003c30 msgr2=0x7f18e0004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:33:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f7fff700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18e0003c30 0x7f18e0004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f7fff700 1 -- 172.21.15.7:0/4046964351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e00035b0 con 0x7f18e0004d90 2023-03-03T09:33:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f6ffd700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f80aad20 0x7f18e0004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:33:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f7fff700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e0004d90 0x7f18e0005200 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f18f000c2f0 tx=0x7f18e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c1d8a48fe73442 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f0001f30 con 0x7f18e0004d90 2023-03-03T09:33:37.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f000ce90 con 0x7f18e0004d90 2023-03-03T09:33:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f67fc700 1 -- 172.21.15.7:0/4046964351 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f0003c40 con 0x7f18e0004d90 2023-03-03T09:33:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18fdffa700 1 -- 172.21.15.7:0/4046964351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e0004d90 msgr2=0x7f18e0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18fdffa700 1 --2- 172.21.15.7:0/4046964351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18e0004d90 0x7f18e0005200 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f18f000c2f0 tx=0x7f18e0005880 comp rx=0 tx=0).stop 2023-03-03T09:33:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.200+0000 7f18f7fff700 1 -- 172.21.15.7:0/4046964351 reap_dead start 2023-03-03T09:33:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.201+0000 7f18fdffa700 1 -- 172.21.15.7:0/4046964351 shutdown_connections 2023-03-03T09:33:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.201+0000 7f18fdffa700 1 -- 172.21.15.7:0/4046964351 >> 172.21.15.7:0/4046964351 conn(0x7f18f80c8900 msgr2=0x7f18f80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:33:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.201+0000 7f18fdffa700 1 -- 172.21.15.7:0/4046964351 shutdown_connections 2023-03-03T09:33:37.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.201+0000 7f18fdffa700 1 -- 172.21.15.7:0/4046964351 wait complete. 2023-03-03T09:33:37.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.202+0000 7f18fdffa700 1 Processor -- start 2023-03-03T09:33:37.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.202+0000 7f18fdffa700 1 -- start start 2023-03-03T09:33:37.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18fdffa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f8153580 0x7f18f814ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:37.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18fdffa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18f8153950 0x7f18f814efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:37.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18fdffa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f80c5530 0x7f18f814d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:37.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18fdffa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f814b490 con 0x7f18f8153950 2023-03-03T09:33:37.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18fdffa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f81edbb0 con 0x7f18f80c5530 2023-03-03T09:33:37.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18fdffa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f814b730 con 0x7f18f8153580 2023-03-03T09:33:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18f8153950 0x7f18f814efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18f8153950 0x7f18f814efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56534/0 (socket says 172.21.15.7:56534) 2023-03-03T09:33:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f6ffd700 1 -- 172.21.15.7:0/2128583450 learned_addr learned my addr 172.21.15.7:0/2128583450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:33:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f7fff700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f80c5530 0x7f18f814d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f77fe700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f8153580 0x7f18f814ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f6ffd700 1 -- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f8153580 msgr2=0x7f18f814ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f6ffd700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f8153580 0x7f18f814ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f6ffd700 1 -- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f80c5530 msgr2=0x7f18f814d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f6ffd700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f80c5530 0x7f18f814d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f6ffd700 1 -- 172.21.15.7:0/2128583450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f806fb70 con 0x7f18f8153950 2023-03-03T09:33:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f77fe700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f8153580 0x7f18f814ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:33:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f7fff700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f80c5530 0x7f18f814d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:33:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18f6ffd700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18f8153950 0x7f18f814efa0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f18ec011cd0 tx=0x7f18ec00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18dffff700 1 -- 172.21.15.7:0/2128583450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec00fb30 con 0x7f18f8153950 2023-03-03T09:33:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18dffff700 1 -- 172.21.15.7:0/2128583450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ec00fc90 con 0x7f18f8153950 2023-03-03T09:33:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18dffff700 1 -- 172.21.15.7:0/2128583450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec004020 con 0x7f18f8153950 2023-03-03T09:33:37.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.203+0000 7f18fdffa700 1 -- 172.21.15.7:0/2128583450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f806f890 con 0x7f18f8153950 2023-03-03T09:33:37.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.204+0000 7f18fdffa700 1 -- 172.21.15.7:0/2128583450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f8155350 con 0x7f18f8153950 2023-03-03T09:33:37.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.204+0000 7f18fdffa700 1 -- 172.21.15.7:0/2128583450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f814b8f0 con 0x7f18f8153950 2023-03-03T09:33:37.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.205+0000 7f18dffff700 1 -- 172.21.15.7:0/2128583450 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f18ec024070 con 0x7f18f8153950 2023-03-03T09:33:37.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.205+0000 7f18dffff700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18e40682f0 0x7f18e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:37.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.205+0000 7f18dffff700 1 -- 172.21.15.7:0/2128583450 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f18ec086660 con 0x7f18f8153950 2023-03-03T09:33:37.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.206+0000 7f18f77fe700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18e40682f0 0x7f18e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:37.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.206+0000 7f18f77fe700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18e40682f0 0x7f18e406a7a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f18e80048d0 tx=0x7f18e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:37.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.210+0000 7f18dffff700 1 -- 172.21.15.7:0/2128583450 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18ec052a00 con 0x7f18f8153950 2023-03-03T09:33:37.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.446+0000 7f18fdffa700 1 -- 172.21.15.7:0/2128583450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18f8074bb0 con 0x7f18f8153950 2023-03-03T09:33:37.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.447+0000 7f18dffff700 1 -- 172.21.15.7:0/2128583450 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f18ec056020 con 0x7f18f8153950 2023-03-03T09:33:37.450 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:33:37.450 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:33:37.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 -- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18e40682f0 msgr2=0x7f18e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:37.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18e40682f0 0x7f18e406a7a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f18e80048d0 tx=0x7f18e8009040 comp rx=0 tx=0).stop 2023-03-03T09:33:37.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 -- 172.21.15.7:0/2128583450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18f8153950 msgr2=0x7f18f814efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:37.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18f8153950 0x7f18f814efa0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f18ec011cd0 tx=0x7f18ec00f520 comp rx=0 tx=0).stop 2023-03-03T09:33:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 -- 172.21.15.7:0/2128583450 shutdown_connections 2023-03-03T09:33:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18f80c5530 0x7f18f814d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18f8153580 0x7f18f814ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18e40682f0 0x7f18e406a7a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 --2- 172.21.15.7:0/2128583450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18f8153950 0x7f18f814efa0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 -- 172.21.15.7:0/2128583450 >> 172.21.15.7:0/2128583450 conn(0x7f18f80c8900 msgr2=0x7f18f80ace30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:33:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 -- 172.21.15.7:0/2128583450 shutdown_connections 2023-03-03T09:33:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:37.449+0000 7f18fdffa700 1 -- 172.21.15.7:0/2128583450 wait complete. 2023-03-03T09:33:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:33:37.463 INFO:tasks.mds_thrash.fs.[a]:waiting for 91 secs before thrashing 2023-03-03T09:33:46.529 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:46.532 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:33:47.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:33:47.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.926+0000 7feeda74a700 1 Processor -- start 2023-03-03T09:33:47.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.926+0000 7feeda74a700 1 -- start start 2023-03-03T09:33:47.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feeda74a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feed414b0d0 0x7feed4071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:47.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feeda74a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7feed4071d40 0x7feed4158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:47.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feeda74a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed40a9c20 con 0x7feed40aad20 2023-03-03T09:33:47.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feeda74a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed4154f10 con 0x7feed414b0d0 2023-03-03T09:33:47.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feeda74a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed40ac3c0 con 0x7feed4071d40 2023-03-03T09:33:47.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feed3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feed414b0d0 0x7feed4071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:47.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feed3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feed414b0d0 0x7feed4071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56544/0 (socket says 172.21.15.7:56544) 2023-03-03T09:33:47.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feed37fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7feed40aad20 0x7feed4157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50524/0 (socket says 172.21.15.7:50524) 2023-03-03T09:33:47.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feed3fff700 1 -- 172.21.15.7:0/3112436076 learned_addr learned my addr 172.21.15.7:0/3112436076 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:33:47.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feed8ce7700 1 --2- 172.21.15.7:0/3112436076 >> v2:172.21.15.143:3300/0 conn(0x7feed4071d40 0x7feed4158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:47.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feed3fff700 1 -- 172.21.15.7:0/3112436076 >> v2:172.21.15.143:3300/0 conn(0x7feed4071d40 msgr2=0x7feed4158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:33:47.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feed3fff700 1 --2- 172.21.15.7:0/3112436076 >> v2:172.21.15.143:3300/0 conn(0x7feed4071d40 0x7feed4158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:47.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feed3fff700 1 -- 172.21.15.7:0/3112436076 >> v1:172.21.15.7:6789/0 conn(0x7feed40aad20 legacy=0x7feed4157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:47.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feed2ffd700 1 -- 172.21.15.7:0/3112436076 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255097571 0 0) 0x7feed40a9c20 con 0x7feed40aad20 2023-03-03T09:33:47.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.927+0000 7feed3fff700 1 -- 172.21.15.7:0/3112436076 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed406fc70 con 0x7feed414b0d0 2023-03-03T09:33:47.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feed3fff700 1 --2- 172.21.15.7:0/3112436076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed414b0d0 0x7feed4071800 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7feed40a98d0 tx=0x7feebc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d974eb5db0538cc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:47.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feed2ffd700 1 -- 172.21.15.7:0/3112436076 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc00b870 con 0x7feed414b0d0 2023-03-03T09:33:47.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feed2ffd700 1 -- 172.21.15.7:0/3112436076 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feebc00b9d0 con 0x7feed414b0d0 2023-03-03T09:33:47.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feed2ffd700 1 -- 172.21.15.7:0/3112436076 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc00bd60 con 0x7feed414b0d0 2023-03-03T09:33:47.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feeda74a700 1 -- 172.21.15.7:0/3112436076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed414b0d0 msgr2=0x7feed4071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:47.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feeda74a700 1 --2- 172.21.15.7:0/3112436076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed414b0d0 0x7feed4071800 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7feed40a98d0 tx=0x7feebc009bd0 comp rx=0 tx=0).stop 2023-03-03T09:33:47.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feeda74a700 1 -- 172.21.15.7:0/3112436076 shutdown_connections 2023-03-03T09:33:47.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feeda74a700 1 --2- 172.21.15.7:0/3112436076 >> v2:172.21.15.143:3300/0 conn(0x7feed4071d40 0x7feed4158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:47.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feeda74a700 1 --2- 172.21.15.7:0/3112436076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed414b0d0 0x7feed4071800 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:47.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feeda74a700 1 -- 172.21.15.7:0/3112436076 >> 172.21.15.7:0/3112436076 conn(0x7feed40c8900 msgr2=0x7feed40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:33:47.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feeda74a700 1 -- 172.21.15.7:0/3112436076 shutdown_connections 2023-03-03T09:33:47.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.928+0000 7feeda74a700 1 -- 172.21.15.7:0/3112436076 wait complete. 2023-03-03T09:33:47.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.929+0000 7feeda74a700 1 Processor -- start 2023-03-03T09:33:47.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.930+0000 7feeda74a700 1 -- start start 2023-03-03T09:33:47.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.930+0000 7feeda74a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feed4071d40 0x7feed4145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:47.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.930+0000 7feeda74a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed40aad20 0x7feed4152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:47.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.930+0000 7feeda74a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feed414b0d0 0x7feed4152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:47.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.930+0000 7feeda74a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed40c6d20 con 0x7feed40aad20 2023-03-03T09:33:47.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.930+0000 7feeda74a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed41520d0 con 0x7feed414b0d0 2023-03-03T09:33:47.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.930+0000 7feeda74a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed40c7250 con 0x7feed4071d40 2023-03-03T09:33:47.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.930+0000 7feed37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed40aad20 0x7feed4152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:47.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.930+0000 7feed37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed40aad20 0x7feed4152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56552/0 (socket says 172.21.15.7:56552) 2023-03-03T09:33:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.930+0000 7feed37fe700 1 -- 172.21.15.7:0/1611920464 learned_addr learned my addr 172.21.15.7:0/1611920464 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:33:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feed37fe700 1 -- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feed4071d40 msgr2=0x7feed4145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feed37fe700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feed4071d40 0x7feed4145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feed37fe700 1 -- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feed414b0d0 msgr2=0x7feed4152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feed37fe700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feed414b0d0 0x7feed4152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feed37fe700 1 -- 172.21.15.7:0/1611920464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc009550 con 0x7feed40aad20 2023-03-03T09:33:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feed37fe700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed40aad20 0x7feed4152450 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7feec400c520 tx=0x7feec400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feed0ff9700 1 -- 172.21.15.7:0/1611920464 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec4004c80 con 0x7feed40aad20 2023-03-03T09:33:47.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feed0ff9700 1 -- 172.21.15.7:0/1611920464 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec4004de0 con 0x7feed40aad20 2023-03-03T09:33:47.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feed0ff9700 1 -- 172.21.15.7:0/1611920464 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec40072d0 con 0x7feed40aad20 2023-03-03T09:33:47.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feeda74a700 1 -- 172.21.15.7:0/1611920464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed406f6b0 con 0x7feed40aad20 2023-03-03T09:33:47.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.931+0000 7feeda74a700 1 -- 172.21.15.7:0/1611920464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed4155160 con 0x7feed40aad20 2023-03-03T09:33:47.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.932+0000 7feeda74a700 1 -- 172.21.15.7:0/1611920464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed406f990 con 0x7feed40aad20 2023-03-03T09:33:47.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.933+0000 7feed0ff9700 1 -- 172.21.15.7:0/1611920464 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feec401a020 con 0x7feed40aad20 2023-03-03T09:33:47.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.933+0000 7feed0ff9700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feeb40682f0 0x7feeb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:47.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.933+0000 7feed0ff9700 1 -- 172.21.15.7:0/1611920464 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7feec4085e80 con 0x7feed40aad20 2023-03-03T09:33:47.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.933+0000 7feed3fff700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feeb40682f0 0x7feeb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:47.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.934+0000 7feed3fff700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feeb40682f0 0x7feeb406a7a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7feebc000c00 tx=0x7feebc010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:47.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:47.938+0000 7feed0ff9700 1 -- 172.21.15.7:0/1611920464 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feec40521e0 con 0x7feed40aad20 2023-03-03T09:33:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.176+0000 7feeda74a700 1 -- 172.21.15.7:0/1611920464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feed4074bb0 con 0x7feed40aad20 2023-03-03T09:33:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.176+0000 7feed0ff9700 1 -- 172.21.15.7:0/1611920464 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7feec4055800 con 0x7feed40aad20 2023-03-03T09:33:48.178 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:33:48.178 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:33:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 -- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feeb40682f0 msgr2=0x7feeb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feeb40682f0 0x7feeb406a7a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7feebc000c00 tx=0x7feebc010000 comp rx=0 tx=0).stop 2023-03-03T09:33:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 -- 172.21.15.7:0/1611920464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed40aad20 msgr2=0x7feed4152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed40aad20 0x7feed4152450 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7feec400c520 tx=0x7feec400c8e0 comp rx=0 tx=0).stop 2023-03-03T09:33:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 -- 172.21.15.7:0/1611920464 shutdown_connections 2023-03-03T09:33:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feed414b0d0 0x7feed4152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feed4071d40 0x7feed4145ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feeb40682f0 0x7feeb406a7a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 --2- 172.21.15.7:0/1611920464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feed40aad20 0x7feed4152450 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 -- 172.21.15.7:0/1611920464 >> 172.21.15.7:0/1611920464 conn(0x7feed40c8900 msgr2=0x7feed40ac5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:33:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 -- 172.21.15.7:0/1611920464 shutdown_connections 2023-03-03T09:33:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:48.178+0000 7feeda74a700 1 -- 172.21.15.7:0/1611920464 wait complete. 2023-03-03T09:33:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:33:48.191 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before thrashing 2023-03-03T09:33:52.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:33:52.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.329+0000 7f085f1a1700 1 Processor -- start 2023-03-03T09:33:52.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.329+0000 7f085f1a1700 1 -- start start 2023-03-03T09:33:52.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.330+0000 7f085f1a1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f085814de50 0x7f085806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:52.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.330+0000 7f085f1a1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f08580c65c0 0x7f08580c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:52.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.330+0000 7f085f1a1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08580a65b0 con 0x7f085806a7f0 2023-03-03T09:33:52.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.330+0000 7f085f1a1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0858154e00 con 0x7f08580c65c0 2023-03-03T09:33:52.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.330+0000 7f085f1a1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08580c3370 con 0x7f085814de50 2023-03-03T09:33:52.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.330+0000 7f0857fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f08580c65c0 0x7f08580c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:52.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.330+0000 7f085d73e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f085806a7f0 0x7f08580c2c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50540/0 (socket says 172.21.15.7:50540) 2023-03-03T09:33:52.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.330+0000 7f085d73e700 1 -- 172.21.15.7:0/4243775718 learned_addr learned my addr 172.21.15.7:0/4243775718 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:33:52.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.331+0000 7f085cf3d700 1 --2- 172.21.15.7:0/4243775718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f085814de50 0x7f085806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:52.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.331+0000 7f0857fff700 1 -- 172.21.15.7:0/4243775718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f085814de50 msgr2=0x7f085806a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:33:52.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.331+0000 7f0857fff700 1 --2- 172.21.15.7:0/4243775718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f085814de50 0x7f085806a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:52.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.331+0000 7f0857fff700 1 -- 172.21.15.7:0/4243775718 >> v1:172.21.15.7:6789/0 conn(0x7f085806a7f0 legacy=0x7f08580c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:52.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.331+0000 7f0857fff700 1 -- 172.21.15.7:0/4243775718 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08581541f0 con 0x7f08580c65c0 2023-03-03T09:33:52.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.331+0000 7f08577fe700 1 -- 172.21.15.7:0/4243775718 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003209748 0 0) 0x7f08580a65b0 con 0x7f085806a7f0 2023-03-03T09:33:52.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.331+0000 7f0857fff700 1 --2- 172.21.15.7:0/4243775718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 0x7f08580c2720 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0844009840 tx=0x7f0844009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b47c75dedc4df5c9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:52.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.331+0000 7f08577fe700 1 -- 172.21.15.7:0/4243775718 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844011830 con 0x7f08580c65c0 2023-03-03T09:33:52.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.331+0000 7f08577fe700 1 -- 172.21.15.7:0/4243775718 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0844011990 con 0x7f08580c65c0 2023-03-03T09:33:52.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.331+0000 7f08577fe700 1 -- 172.21.15.7:0/4243775718 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844011d20 con 0x7f08580c65c0 2023-03-03T09:33:52.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.332+0000 7f085f1a1700 1 -- 172.21.15.7:0/4243775718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 msgr2=0x7f08580c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:52.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.332+0000 7f085f1a1700 1 --2- 172.21.15.7:0/4243775718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 0x7f08580c2720 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0844009840 tx=0x7f0844009b50 comp rx=0 tx=0).stop 2023-03-03T09:33:52.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.332+0000 7f085f1a1700 1 -- 172.21.15.7:0/4243775718 shutdown_connections 2023-03-03T09:33:52.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.332+0000 7f085f1a1700 1 --2- 172.21.15.7:0/4243775718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f085814de50 0x7f085806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:52.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.332+0000 7f085f1a1700 1 --2- 172.21.15.7:0/4243775718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 0x7f08580c2720 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:52.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.332+0000 7f085f1a1700 1 -- 172.21.15.7:0/4243775718 >> 172.21.15.7:0/4243775718 conn(0x7f08580a9430 msgr2=0x7f08580c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:33:52.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.332+0000 7f085f1a1700 1 -- 172.21.15.7:0/4243775718 shutdown_connections 2023-03-03T09:33:52.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.332+0000 7f085f1a1700 1 -- 172.21.15.7:0/4243775718 wait complete. 2023-03-03T09:33:52.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.333+0000 7f085f1a1700 1 Processor -- start 2023-03-03T09:33:52.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.333+0000 7f085f1a1700 1 -- start start 2023-03-03T09:33:52.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.334+0000 7f085f1a1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f085806a7f0 0x7f08580c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:52.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.334+0000 7f085f1a1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 0x7f08581ed620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:52.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.334+0000 7f085f1a1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f085814de50 0x7f08581edb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:52.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f085f1a1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0858071020 con 0x7f08580c65c0 2023-03-03T09:33:52.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f085f1a1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0858070040 con 0x7f085806a7f0 2023-03-03T09:33:52.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f085f1a1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08581ed1d0 con 0x7f085814de50 2023-03-03T09:33:52.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f0857fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 0x7f08581ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:52.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f0857fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 0x7f08581ed620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56566/0 (socket says 172.21.15.7:56566) 2023-03-03T09:33:52.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f0857fff700 1 -- 172.21.15.7:0/4031692651 learned_addr learned my addr 172.21.15.7:0/4031692651 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:33:52.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f0857fff700 1 -- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f085814de50 msgr2=0x7f08581edb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:52.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f0857fff700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f085814de50 0x7f08581edb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:52.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f0857fff700 1 -- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f085806a7f0 msgr2=0x7f08580c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:52.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f085cf3d700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f085806a7f0 0x7f08580c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:52.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f0857fff700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f085806a7f0 0x7f08580c00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:52.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f0857fff700 1 -- 172.21.15.7:0/4031692651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08440094a0 con 0x7f08580c65c0 2023-03-03T09:33:52.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f0857fff700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 0x7f08581ed620 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0844009fd0 tx=0x7f0844004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:52.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f08557fa700 1 -- 172.21.15.7:0/4031692651 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08440036a0 con 0x7f08580c65c0 2023-03-03T09:33:52.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f08557fa700 1 -- 172.21.15.7:0/4031692651 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084401f070 con 0x7f08580c65c0 2023-03-03T09:33:52.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f085f1a1700 1 -- 172.21.15.7:0/4031692651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08581547b0 con 0x7f08580c65c0 2023-03-03T09:33:52.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f08557fa700 1 -- 172.21.15.7:0/4031692651 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844003a60 con 0x7f08580c65c0 2023-03-03T09:33:52.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.335+0000 7f085f1a1700 1 -- 172.21.15.7:0/4031692651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08581544d0 con 0x7f08580c65c0 2023-03-03T09:33:52.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.336+0000 7f085f1a1700 1 -- 172.21.15.7:0/4031692651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0858154a90 con 0x7f08580c65c0 2023-03-03T09:33:52.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.337+0000 7f08557fa700 1 -- 172.21.15.7:0/4031692651 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f084400f050 con 0x7f08580c65c0 2023-03-03T09:33:52.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.337+0000 7f08557fa700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f083c0682f0 0x7f083c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:33:52.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.337+0000 7f08557fa700 1 -- 172.21.15.7:0/4031692651 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0844021070 con 0x7f08580c65c0 2023-03-03T09:33:52.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.337+0000 7f085cf3d700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f083c0682f0 0x7f083c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:33:52.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.338+0000 7f085cf3d700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f083c0682f0 0x7f083c06a7a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f084c009ab0 tx=0x7f084c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:33:52.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.341+0000 7f08557fa700 1 -- 172.21.15.7:0/4031692651 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0844053d50 con 0x7f08580c65c0 2023-03-03T09:33:52.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.581+0000 7f085f1a1700 1 -- 172.21.15.7:0/4031692651 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0858074bb0 con 0x7f08580c65c0 2023-03-03T09:33:52.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.581+0000 7f08557fa700 1 -- 172.21.15.7:0/4031692651 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0844057370 con 0x7f08580c65c0 2023-03-03T09:33:52.583 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:33:52.583 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:33:52.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 -- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f083c0682f0 msgr2=0x7f083c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:52.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f083c0682f0 0x7f083c06a7a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f084c009ab0 tx=0x7f084c008040 comp rx=0 tx=0).stop 2023-03-03T09:33:52.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 -- 172.21.15.7:0/4031692651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 msgr2=0x7f08581ed620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:33:52.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 0x7f08581ed620 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0844009fd0 tx=0x7f0844004750 comp rx=0 tx=0).stop 2023-03-03T09:33:52.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 -- 172.21.15.7:0/4031692651 shutdown_connections 2023-03-03T09:33:52.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f085814de50 0x7f08581edb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f085806a7f0 0x7f08580c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f083c0682f0 0x7f083c06a7a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 --2- 172.21.15.7:0/4031692651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08580c65c0 0x7f08581ed620 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:33:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 -- 172.21.15.7:0/4031692651 >> 172.21.15.7:0/4031692651 conn(0x7f08580a9430 msgr2=0x7f08580c3590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:33:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 -- 172.21.15.7:0/4031692651 shutdown_connections 2023-03-03T09:33:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:33:52.583+0000 7f085f1a1700 1 -- 172.21.15.7:0/4031692651 wait complete. 2023-03-03T09:33:52.587 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:33:52.596 INFO:tasks.mds_thrash.fs.[b]:waiting for 36 secs before thrashing 2023-03-03T09:34:16.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:16.596 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:28.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:34:28.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.734+0000 7fd3f7d35700 1 Processor -- start 2023-03-03T09:34:28.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.734+0000 7fd3f7d35700 1 -- start start 2023-03-03T09:34:28.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.735+0000 7fd3f7d35700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f00c00e0 0x7fd3f00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:28.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.735+0000 7fd3f7d35700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f00c7310 con 0x7fd3f00caa30 2023-03-03T09:34:28.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.735+0000 7fd3f7d35700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f00bfd10 con 0x7fd3f0154ff0 2023-03-03T09:34:28.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.735+0000 7fd3f7d35700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f00bfa30 con 0x7fd3f00c00e0 2023-03-03T09:34:28.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.735+0000 7fd3f5ad1700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd3f00caa30 0x7fd3f00c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50572/0 (socket says 172.21.15.7:50572) 2023-03-03T09:34:28.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.735+0000 7fd3f5ad1700 1 -- 172.21.15.7:0/621167855 learned_addr learned my addr 172.21.15.7:0/621167855 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:34:28.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.735+0000 7fd3f52d0700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f00c00e0 0x7fd3f00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:28.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.735+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847708608 0 0) 0x7fd3f00c7310 con 0x7fd3f00caa30 2023-03-03T09:34:28.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.735+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3dc0035b0 con 0x7fd3f00caa30 2023-03-03T09:34:28.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3182194383 0 0) 0x7fd3dc0035b0 con 0x7fd3f00caa30 2023-03-03T09:34:28.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f00c00e0 msgr2=0x7fd3f00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:34:28.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f00c00e0 0x7fd3f00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:28.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 >> v1:172.21.15.143:6789/0 conn(0x7fd3f0154ff0 legacy=0x7fd3f015ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:28.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f0154510 con 0x7fd3f00caa30 2023-03-03T09:34:28.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd3e0001670 con 0x7fd3f00caa30 2023-03-03T09:34:28.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 >> v1:172.21.15.7:6789/0 conn(0x7fd3f00caa30 legacy=0x7fd3f00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:28.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f00c00e0 0x7fd3dc003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:28.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3dc004070 0x7fd3dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:28.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3dc004d90 0x7fd3dc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:28.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f00bfa30 con 0x7fd3dc004d90 2023-03-03T09:34:28.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc00bc50 con 0x7fd3dc004070 2023-03-03T09:34:28.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc00be10 con 0x7fd3f00c00e0 2023-03-03T09:34:28.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f62d2700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3dc004d90 0x7fd3dc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:28.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f5ad1700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f00c00e0 0x7fd3dc003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:28.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f52d0700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3dc004070 0x7fd3dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:28.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f62d2700 1 -- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f00c00e0 msgr2=0x7fd3dc003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:34:28.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f62d2700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f00c00e0 0x7fd3dc003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f62d2700 1 -- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3dc004070 msgr2=0x7fd3dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:34:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f62d2700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3dc004070 0x7fd3dc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f62d2700 1 -- 172.21.15.7:0/621167855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc0035b0 con 0x7fd3dc004d90 2023-03-03T09:34:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.736+0000 7fd3f62d2700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3dc004d90 0x7fd3dc005200 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800c160 tx=0x7fd3dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47ff1e9a5ca36ed server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:34:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.737+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8001db0 con 0x7fd3dc004d90 2023-03-03T09:34:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.737+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e800cca0 con 0x7fd3dc004d90 2023-03-03T09:34:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.737+0000 7fd3f4acf700 1 -- 172.21.15.7:0/621167855 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8003c40 con 0x7fd3dc004d90 2023-03-03T09:34:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.737+0000 7fd3f7d35700 1 -- 172.21.15.7:0/621167855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3dc004d90 msgr2=0x7fd3dc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.737+0000 7fd3f7d35700 1 --2- 172.21.15.7:0/621167855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3dc004d90 0x7fd3dc005200 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800c160 tx=0x7fd3dc005880 comp rx=0 tx=0).stop 2023-03-03T09:34:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.737+0000 7fd3f62d2700 1 -- 172.21.15.7:0/621167855 reap_dead start 2023-03-03T09:34:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.737+0000 7fd3f7d35700 1 -- 172.21.15.7:0/621167855 shutdown_connections 2023-03-03T09:34:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.737+0000 7fd3f7d35700 1 -- 172.21.15.7:0/621167855 >> 172.21.15.7:0/621167855 conn(0x7fd3f0066e40 msgr2=0x7fd3f0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:34:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.737+0000 7fd3f7d35700 1 -- 172.21.15.7:0/621167855 shutdown_connections 2023-03-03T09:34:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.737+0000 7fd3f7d35700 1 -- 172.21.15.7:0/621167855 wait complete. 2023-03-03T09:34:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.738+0000 7fd3f7d35700 1 Processor -- start 2023-03-03T09:34:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f7d35700 1 -- start start 2023-03-03T09:34:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f7d35700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3f00c1e90 0x7fd3f01577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f7d35700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3f00c2260 0x7fd3f0157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f7d35700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f01508d0 0x7fd3f014bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f7d35700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f0156820 con 0x7fd3f00c2260 2023-03-03T09:34:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f7d35700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f01de0c0 con 0x7fd3f00c1e90 2023-03-03T09:34:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f7d35700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f01e1980 con 0x7fd3f01508d0 2023-03-03T09:34:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f52d0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3f00c2260 0x7fd3f0157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f52d0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3f00c2260 0x7fd3f0157d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56608/0 (socket says 172.21.15.7:56608) 2023-03-03T09:34:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f52d0700 1 -- 172.21.15.7:0/1219047076 learned_addr learned my addr 172.21.15.7:0/1219047076 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:34:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f62d2700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f01508d0 0x7fd3f014bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.739+0000 7fd3f5ad1700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3f00c1e90 0x7fd3f01577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3f52d0700 1 -- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f01508d0 msgr2=0x7fd3f014bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3f52d0700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f01508d0 0x7fd3f014bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3f52d0700 1 -- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3f00c1e90 msgr2=0x7fd3f01577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3f52d0700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3f00c1e90 0x7fd3f01577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3f52d0700 1 -- 172.21.15.7:0/1219047076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f0154ad0 con 0x7fd3f00c2260 2023-03-03T09:34:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3f62d2700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f01508d0 0x7fd3f014bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:34:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3f52d0700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3f00c2260 0x7fd3f0157d00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0119f0 tx=0x7fd3ec011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:34:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3e67fc700 1 -- 172.21.15.7:0/1219047076 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec005c30 con 0x7fd3f00c2260 2023-03-03T09:34:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3e67fc700 1 -- 172.21.15.7:0/1219047076 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec005d90 con 0x7fd3f00c2260 2023-03-03T09:34:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3f7d35700 1 -- 172.21.15.7:0/1219047076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f01547f0 con 0x7fd3f00c2260 2023-03-03T09:34:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3e67fc700 1 -- 172.21.15.7:0/1219047076 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec00f950 con 0x7fd3f00c2260 2023-03-03T09:34:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.740+0000 7fd3f7d35700 1 -- 172.21.15.7:0/1219047076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f00bf650 con 0x7fd3f00c2260 2023-03-03T09:34:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.741+0000 7fd3f7d35700 1 -- 172.21.15.7:0/1219047076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f0074bb0 con 0x7fd3f00c2260 2023-03-03T09:34:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.741+0000 7fd3e67fc700 1 -- 172.21.15.7:0/1219047076 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd3ec00fab0 con 0x7fd3f00c2260 2023-03-03T09:34:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.742+0000 7fd3e67fc700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3d40682f0 0x7fd3d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.742+0000 7fd3e67fc700 1 -- 172.21.15.7:0/1219047076 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd3ec086970 con 0x7fd3f00c2260 2023-03-03T09:34:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.742+0000 7fd3f5ad1700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3d40682f0 0x7fd3d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.743+0000 7fd3f5ad1700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3d40682f0 0x7fd3d406a7a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd3f01582e0 tx=0x7fd3e0000f80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:34:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.746+0000 7fd3e67fc700 1 -- 172.21.15.7:0/1219047076 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd3ec052dd0 con 0x7fd3f00c2260 2023-03-03T09:34:28.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.989+0000 7fd3f7d35700 1 -- 172.21.15.7:0/1219047076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3f00c5030 con 0x7fd3f00c2260 2023-03-03T09:34:28.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.990+0000 7fd3e67fc700 1 -- 172.21.15.7:0/1219047076 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd3ec0563f0 con 0x7fd3f00c2260 2023-03-03T09:34:28.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:34:28.991 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:34:28.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 -- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3d40682f0 msgr2=0x7fd3d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:28.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3d40682f0 0x7fd3d406a7a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd3f01582e0 tx=0x7fd3e0000f80 comp rx=0 tx=0).stop 2023-03-03T09:34:28.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 -- 172.21.15.7:0/1219047076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3f00c2260 msgr2=0x7fd3f0157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:28.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3f00c2260 0x7fd3f0157d00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0119f0 tx=0x7fd3ec011d00 comp rx=0 tx=0).stop 2023-03-03T09:34:28.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 -- 172.21.15.7:0/1219047076 shutdown_connections 2023-03-03T09:34:28.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3f01508d0 0x7fd3f014bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:28.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3f00c1e90 0x7fd3f01577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:28.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3d40682f0 0x7fd3d406a7a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:28.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 --2- 172.21.15.7:0/1219047076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3f00c2260 0x7fd3f0157d00 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:28.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 -- 172.21.15.7:0/1219047076 >> 172.21.15.7:0/1219047076 conn(0x7fd3f0066e40 msgr2=0x7fd3f0065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:34:28.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 -- 172.21.15.7:0/1219047076 shutdown_connections 2023-03-03T09:34:28.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:28.992+0000 7fd3f7d35700 1 -- 172.21.15.7:0/1219047076 wait complete. 2023-03-03T09:34:28.995 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:34:29.005 INFO:tasks.mds_thrash.fs.[b]:waiting for 23 secs before thrashing 2023-03-03T09:34:46.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:46.667 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:34:52.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:34:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.144+0000 7f8d44d6f700 1 Processor -- start 2023-03-03T09:34:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.144+0000 7f8d44d6f700 1 -- start start 2023-03-03T09:34:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.145+0000 7f8d44d6f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d40071990 0x7f8d40071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.145+0000 7f8d44d6f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8d4014b460 0x7f8d400722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.145+0000 7f8d44d6f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d40070360 con 0x7f8d40151c00 2023-03-03T09:34:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.145+0000 7f8d44d6f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d401521a0 con 0x7f8d4014b460 2023-03-03T09:34:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.145+0000 7f8d44d6f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4006a5e0 con 0x7f8d40071990 2023-03-03T09:34:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.145+0000 7f8d3dd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8d4014b460 0x7f8d400722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.145+0000 7f8d3e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d40071990 0x7f8d40071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.145+0000 7f8d3ed9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8d40151c00 0x7f8d4006aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:54132/0 (socket says 172.21.15.7:54132) 2023-03-03T09:34:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3ed9d700 1 -- 172.21.15.7:0/839883306 learned_addr learned my addr 172.21.15.7:0/839883306 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:34:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3dd9b700 1 -- 172.21.15.7:0/839883306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d40071990 msgr2=0x7f8d40071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:34:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3dd9b700 1 --2- 172.21.15.7:0/839883306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d40071990 0x7f8d40071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3dd9b700 1 -- 172.21.15.7:0/839883306 >> v1:172.21.15.143:6789/0 conn(0x7f8d40151c00 legacy=0x7f8d4006aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3d59a700 1 -- 172.21.15.7:0/839883306 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866571225 0 0) 0x7f8d40070360 con 0x7f8d40151c00 2023-03-03T09:34:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3dd9b700 1 -- 172.21.15.7:0/839883306 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400c38b0 con 0x7f8d4014b460 2023-03-03T09:34:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3e59c700 1 --2- 172.21.15.7:0/839883306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d40071990 0x7f8d40071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:34:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3dd9b700 1 --2- 172.21.15.7:0/839883306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d4014b460 0x7f8d400722a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009840 tx=0x7f8d28009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b4d70abe50cf027 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:34:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3d59a700 1 -- 172.21.15.7:0/839883306 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2800f8a0 con 0x7f8d4014b460 2023-03-03T09:34:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3d59a700 1 -- 172.21.15.7:0/839883306 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2800fa00 con 0x7f8d4014b460 2023-03-03T09:34:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.146+0000 7f8d3d59a700 1 -- 172.21.15.7:0/839883306 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2800fd90 con 0x7f8d4014b460 2023-03-03T09:34:52.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.147+0000 7f8d44d6f700 1 -- 172.21.15.7:0/839883306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d4014b460 msgr2=0x7f8d400722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:52.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.147+0000 7f8d44d6f700 1 --2- 172.21.15.7:0/839883306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d4014b460 0x7f8d400722a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009840 tx=0x7f8d28009b50 comp rx=0 tx=0).stop 2023-03-03T09:34:52.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.147+0000 7f8d44d6f700 1 -- 172.21.15.7:0/839883306 shutdown_connections 2023-03-03T09:34:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.147+0000 7f8d44d6f700 1 --2- 172.21.15.7:0/839883306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d40071990 0x7f8d40071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.147+0000 7f8d44d6f700 1 --2- 172.21.15.7:0/839883306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d4014b460 0x7f8d400722a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.147+0000 7f8d44d6f700 1 -- 172.21.15.7:0/839883306 >> 172.21.15.7:0/839883306 conn(0x7f8d400a94d0 msgr2=0x7f8d400a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:34:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.147+0000 7f8d44d6f700 1 -- 172.21.15.7:0/839883306 shutdown_connections 2023-03-03T09:34:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.147+0000 7f8d44d6f700 1 -- 172.21.15.7:0/839883306 wait complete. 2023-03-03T09:34:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.148+0000 7f8d44d6f700 1 Processor -- start 2023-03-03T09:34:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.148+0000 7f8d44d6f700 1 -- start start 2023-03-03T09:34:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d44d6f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d40071990 0x7f8d400bb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d44d6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d40151c00 0x7f8d400bf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d44d6f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d401515a0 0x7f8d401ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d44d6f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400bd980 con 0x7f8d40151c00 2023-03-03T09:34:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d44d6f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400bcde0 con 0x7f8d40071990 2023-03-03T09:34:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d44d6f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400bd050 con 0x7f8d401515a0 2023-03-03T09:34:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d3dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d40151c00 0x7f8d400bf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d3dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d40151c00 0x7f8d400bf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56636/0 (socket says 172.21.15.7:56636) 2023-03-03T09:34:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d3e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d40071990 0x7f8d400bb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d3dd9b700 1 -- 172.21.15.7:0/3751916858 learned_addr learned my addr 172.21.15.7:0/3751916858 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:34:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.149+0000 7f8d3ed9d700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d401515a0 0x7f8d401ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d3dd9b700 1 -- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d401515a0 msgr2=0x7f8d401ebbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d3dd9b700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d401515a0 0x7f8d401ebbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d3dd9b700 1 -- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d40071990 msgr2=0x7f8d400bb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d3dd9b700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d40071990 0x7f8d400bb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d3dd9b700 1 -- 172.21.15.7:0/3751916858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d300094a0 con 0x7f8d40151c00 2023-03-03T09:34:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d3dd9b700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d40151c00 0x7f8d400bf920 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009fd0 tx=0x7f8d28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:34:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d36ffd700 1 -- 172.21.15.7:0/3751916858 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d280038a0 con 0x7f8d40151c00 2023-03-03T09:34:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d36ffd700 1 -- 172.21.15.7:0/3751916858 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d28003a20 con 0x7f8d40151c00 2023-03-03T09:34:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d44d6f700 1 -- 172.21.15.7:0/3751916858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d280094a0 con 0x7f8d40151c00 2023-03-03T09:34:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d36ffd700 1 -- 172.21.15.7:0/3751916858 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d28003dd0 con 0x7f8d40151c00 2023-03-03T09:34:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.150+0000 7f8d44d6f700 1 -- 172.21.15.7:0/3751916858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d400c3b50 con 0x7f8d40151c00 2023-03-03T09:34:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.151+0000 7f8d44d6f700 1 -- 172.21.15.7:0/3751916858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4006a2b0 con 0x7f8d40151c00 2023-03-03T09:34:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.151+0000 7f8d36ffd700 1 -- 172.21.15.7:0/3751916858 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8d28029020 con 0x7f8d40151c00 2023-03-03T09:34:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.152+0000 7f8d36ffd700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d200682f0 0x7f8d2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:34:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.152+0000 7f8d36ffd700 1 -- 172.21.15.7:0/3751916858 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8d280873c0 con 0x7f8d40151c00 2023-03-03T09:34:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.152+0000 7f8d3e59c700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d200682f0 0x7f8d2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:34:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.153+0000 7f8d3e59c700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d200682f0 0x7f8d2006a7a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009d70 tx=0x7f8d30009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:34:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.156+0000 7f8d36ffd700 1 -- 172.21.15.7:0/3751916858 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8d28053810 con 0x7f8d40151c00 2023-03-03T09:34:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.395+0000 7f8d44d6f700 1 -- 172.21.15.7:0/3751916858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d40074bb0 con 0x7f8d40151c00 2023-03-03T09:34:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.395+0000 7f8d36ffd700 1 -- 172.21.15.7:0/3751916858 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8d28056e30 con 0x7f8d40151c00 2023-03-03T09:34:52.397 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:34:52.397 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:34:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 -- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d200682f0 msgr2=0x7f8d2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d200682f0 0x7f8d2006a7a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009d70 tx=0x7f8d30009210 comp rx=0 tx=0).stop 2023-03-03T09:34:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 -- 172.21.15.7:0/3751916858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d40151c00 msgr2=0x7f8d400bf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:34:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d40151c00 0x7f8d400bf920 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009fd0 tx=0x7f8d28004750 comp rx=0 tx=0).stop 2023-03-03T09:34:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 -- 172.21.15.7:0/3751916858 shutdown_connections 2023-03-03T09:34:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d401515a0 0x7f8d401ebbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d40071990 0x7f8d400bb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d200682f0 0x7f8d2006a7a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 --2- 172.21.15.7:0/3751916858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d40151c00 0x7f8d400bf920 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:34:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 -- 172.21.15.7:0/3751916858 >> 172.21.15.7:0/3751916858 conn(0x7f8d400a94d0 msgr2=0x7f8d400c5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:34:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.397+0000 7f8d44d6f700 1 -- 172.21.15.7:0/3751916858 shutdown_connections 2023-03-03T09:34:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:34:52.398+0000 7f8d44d6f700 1 -- 172.21.15.7:0/3751916858 wait complete. 2023-03-03T09:34:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:34:52.411 INFO:tasks.mds_thrash.fs.[b]:waiting for 70 secs before thrashing 2023-03-03T09:35:08.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:35:08.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.602+0000 7f446de0d700 1 Processor -- start 2023-03-03T09:35:08.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.602+0000 7f446de0d700 1 -- start start 2023-03-03T09:35:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f446de0d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f446814f860 0x7f44681562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f446de0d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44680c20d0 0x7f4468152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f446de0d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4468155e60 con 0x7f44680c27c0 2023-03-03T09:35:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f446de0d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446806d350 con 0x7f44680c20d0 2023-03-03T09:35:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f446de0d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44681558e0 con 0x7f446814f860 2023-03-03T09:35:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f4466ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44680c20d0 0x7f4468152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f4467fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f44680c27c0 0x7f44680736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50632/0 (socket says 172.21.15.7:50632) 2023-03-03T09:35:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f4467fff700 1 -- 172.21.15.7:0/278470836 learned_addr learned my addr 172.21.15.7:0/278470836 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:35:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f44677fe700 1 --2- 172.21.15.7:0/278470836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f446814f860 0x7f44681562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f4466ffd700 1 -- 172.21.15.7:0/278470836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f446814f860 msgr2=0x7f44681562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:35:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f4466ffd700 1 --2- 172.21.15.7:0/278470836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f446814f860 0x7f44681562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:08.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f4466ffd700 1 -- 172.21.15.7:0/278470836 >> v1:172.21.15.7:6789/0 conn(0x7f44680c27c0 legacy=0x7f44680736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f4466ffd700 1 -- 172.21.15.7:0/278470836 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44681532b0 con 0x7f44680c20d0 2023-03-03T09:35:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f44667fc700 1 -- 172.21.15.7:0/278470836 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287966428 0 0) 0x7f4468155e60 con 0x7f44680c27c0 2023-03-03T09:35:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.603+0000 7f44677fe700 1 --2- 172.21.15.7:0/278470836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f446814f860 0x7f44681562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:35:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f4466ffd700 1 --2- 172.21.15.7:0/278470836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44680c20d0 0x7f4468152d70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4450009840 tx=0x7f4450009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2ab498e8c644a3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:35:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f44667fc700 1 -- 172.21.15.7:0/278470836 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450011830 con 0x7f44680c20d0 2023-03-03T09:35:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f44667fc700 1 -- 172.21.15.7:0/278470836 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4450011990 con 0x7f44680c20d0 2023-03-03T09:35:08.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f44667fc700 1 -- 172.21.15.7:0/278470836 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450011d20 con 0x7f44680c20d0 2023-03-03T09:35:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f446de0d700 1 -- 172.21.15.7:0/278470836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44680c20d0 msgr2=0x7f4468152d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f446de0d700 1 --2- 172.21.15.7:0/278470836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44680c20d0 0x7f4468152d70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4450009840 tx=0x7f4450009b50 comp rx=0 tx=0).stop 2023-03-03T09:35:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f446de0d700 1 -- 172.21.15.7:0/278470836 shutdown_connections 2023-03-03T09:35:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f446de0d700 1 --2- 172.21.15.7:0/278470836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f446814f860 0x7f44681562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f446de0d700 1 --2- 172.21.15.7:0/278470836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44680c20d0 0x7f4468152d70 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f446de0d700 1 -- 172.21.15.7:0/278470836 >> 172.21.15.7:0/278470836 conn(0x7f44680a94d0 msgr2=0x7f44680a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:35:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f446de0d700 1 -- 172.21.15.7:0/278470836 shutdown_connections 2023-03-03T09:35:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.604+0000 7f446de0d700 1 -- 172.21.15.7:0/278470836 wait complete. 2023-03-03T09:35:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.605+0000 7f446de0d700 1 Processor -- start 2023-03-03T09:35:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.606+0000 7f446de0d700 1 -- start start 2023-03-03T09:35:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.606+0000 7f446de0d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44680c20d0 0x7f446806a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.606+0000 7f446de0d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44680c27c0 0x7f44680c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:08.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.606+0000 7f446de0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f446814f860 0x7f446814a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.606+0000 7f446de0d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44680ab150 con 0x7f446814f860 2023-03-03T09:35:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.606+0000 7f446de0d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44681edea0 con 0x7f44680c20d0 2023-03-03T09:35:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.606+0000 7f446de0d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44680ac1c0 con 0x7f44680c27c0 2023-03-03T09:35:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.606+0000 7f4467fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f446814f860 0x7f446814a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.606+0000 7f4467fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f446814f860 0x7f446814a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56658/0 (socket says 172.21.15.7:56658) 2023-03-03T09:35:08.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.606+0000 7f4467fff700 1 -- 172.21.15.7:0/2676816064 learned_addr learned my addr 172.21.15.7:0/2676816064 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:35:08.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f44677fe700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44680c20d0 0x7f446806a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:08.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f4466ffd700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44680c27c0 0x7f44680c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:08.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f4467fff700 1 -- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44680c27c0 msgr2=0x7f44680c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:08.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f4467fff700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44680c27c0 0x7f44680c48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:08.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f4467fff700 1 -- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44680c20d0 msgr2=0x7f446806a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:08.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f4467fff700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44680c20d0 0x7f446806a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:08.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f4467fff700 1 -- 172.21.15.7:0/2676816064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44500094a0 con 0x7f446814f860 2023-03-03T09:35:08.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f4467fff700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f446814f860 0x7f446814a4a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4460002230 tx=0x7f446000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:35:08.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f445ffff700 1 -- 172.21.15.7:0/2676816064 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446000aac0 con 0x7f446814f860 2023-03-03T09:35:08.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f445ffff700 1 -- 172.21.15.7:0/2676816064 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446000ac40 con 0x7f446814f860 2023-03-03T09:35:08.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f446de0d700 1 -- 172.21.15.7:0/2676816064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446806d070 con 0x7f446814f860 2023-03-03T09:35:08.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f445ffff700 1 -- 172.21.15.7:0/2676816064 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44600070d0 con 0x7f446814f860 2023-03-03T09:35:08.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.607+0000 7f446de0d700 1 -- 172.21.15.7:0/2676816064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446806cd90 con 0x7f446814f860 2023-03-03T09:35:08.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.608+0000 7f446de0d700 1 -- 172.21.15.7:0/2676816064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4468155af0 con 0x7f446814f860 2023-03-03T09:35:08.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.609+0000 7f445ffff700 1 -- 172.21.15.7:0/2676816064 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4460010070 con 0x7f446814f860 2023-03-03T09:35:08.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.609+0000 7f445ffff700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44480682f0 0x7f444806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:35:08.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.609+0000 7f445ffff700 1 -- 172.21.15.7:0/2676816064 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4460085b30 con 0x7f446814f860 2023-03-03T09:35:08.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.609+0000 7f44677fe700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44480682f0 0x7f444806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:35:08.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.610+0000 7f44677fe700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44480682f0 0x7f444806a7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f445800c940 tx=0x7f4458008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:35:08.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.613+0000 7f445ffff700 1 -- 172.21.15.7:0/2676816064 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4460051df0 con 0x7f446814f860 2023-03-03T09:35:08.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.851+0000 7f446de0d700 1 -- 172.21.15.7:0/2676816064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44680ac020 con 0x7f446814f860 2023-03-03T09:35:08.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.851+0000 7f445ffff700 1 -- 172.21.15.7:0/2676816064 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f44680ac020 con 0x7f446814f860 2023-03-03T09:35:08.853 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:35:08.853 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:35:08.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 -- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44480682f0 msgr2=0x7f444806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:08.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44480682f0 0x7f444806a7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f445800c940 tx=0x7f4458008040 comp rx=0 tx=0).stop 2023-03-03T09:35:08.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 -- 172.21.15.7:0/2676816064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f446814f860 msgr2=0x7f446814a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:35:08.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f446814f860 0x7f446814a4a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4460002230 tx=0x7f446000cf50 comp rx=0 tx=0).stop 2023-03-03T09:35:08.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 -- 172.21.15.7:0/2676816064 shutdown_connections 2023-03-03T09:35:08.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44480682f0 0x7f444806a7a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:08.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f446814f860 0x7f446814a4a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:08.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44680c20d0 0x7f446806a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 --2- 172.21.15.7:0/2676816064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44680c27c0 0x7f44680c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:35:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 -- 172.21.15.7:0/2676816064 >> 172.21.15.7:0/2676816064 conn(0x7f44680a94d0 msgr2=0x7f4468151200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:35:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 -- 172.21.15.7:0/2676816064 shutdown_connections 2023-03-03T09:35:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:35:08.853+0000 7f446de0d700 1 -- 172.21.15.7:0/2676816064 wait complete. 2023-03-03T09:35:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:35:08.866 INFO:tasks.mds_thrash.fs.[a]:waiting for 114 secs before thrashing 2023-03-03T09:35:16.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:16.734 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:46.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:35:46.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:02.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:36:02.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.550+0000 7f90c1620700 1 Processor -- start 2023-03-03T09:36:02.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.550+0000 7f90c1620700 1 -- start start 2023-03-03T09:36:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.551+0000 7f90c1620700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f90bc14f9d0 0x7f90bc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:36:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.551+0000 7f90c1620700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f90bc14fda0 0x7f90bc071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:36:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.551+0000 7f90c1620700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90bc0c2f30 con 0x7f90bc0abe20 2023-03-03T09:36:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.551+0000 7f90c1620700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc0c31d0 con 0x7f90bc14f9d0 2023-03-03T09:36:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.551+0000 7f90c1620700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc1524a0 con 0x7f90bc14fda0 2023-03-03T09:36:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.551+0000 7f90baffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f90bc14f9d0 0x7f90bc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:36:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.551+0000 7f90baffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f90bc14f9d0 0x7f90bc156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56706/0 (socket says 172.21.15.7:56706) 2023-03-03T09:36:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.551+0000 7f90baffd700 1 -- 172.21.15.7:0/1899599455 learned_addr learned my addr 172.21.15.7:0/1899599455 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:36:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.551+0000 7f90ba7fc700 1 --2- 172.21.15.7:0/1899599455 >> v2:172.21.15.143:3300/0 conn(0x7f90bc14fda0 0x7f90bc071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:36:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90baffd700 1 -- 172.21.15.7:0/1899599455 >> v2:172.21.15.143:3300/0 conn(0x7f90bc14fda0 msgr2=0x7f90bc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:36:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90baffd700 1 --2- 172.21.15.7:0/1899599455 >> v2:172.21.15.143:3300/0 conn(0x7f90bc14fda0 0x7f90bc071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:36:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90baffd700 1 -- 172.21.15.7:0/1899599455 >> v1:172.21.15.143:6789/0 conn(0x7f90bc0abe20 legacy=0x7f90bc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:36:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90baffd700 1 -- 172.21.15.7:0/1899599455 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc0c34b0 con 0x7f90bc14f9d0 2023-03-03T09:36:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90baffd700 1 --2- 172.21.15.7:0/1899599455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 0x7f90bc156430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0709b0 tx=0x7f90b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7db91392102e71 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:36:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90b9ffb700 1 -- 172.21.15.7:0/1899599455 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b0011830 con 0x7f90bc14f9d0 2023-03-03T09:36:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90b9ffb700 1 -- 172.21.15.7:0/1899599455 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b0011990 con 0x7f90bc14f9d0 2023-03-03T09:36:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90b9ffb700 1 -- 172.21.15.7:0/1899599455 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b0011d20 con 0x7f90bc14f9d0 2023-03-03T09:36:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90c1620700 1 -- 172.21.15.7:0/1899599455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 msgr2=0x7f90bc156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:36:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90c1620700 1 --2- 172.21.15.7:0/1899599455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 0x7f90bc156430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0709b0 tx=0x7f90b0009bd0 comp rx=0 tx=0).stop 2023-03-03T09:36:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90c1620700 1 -- 172.21.15.7:0/1899599455 shutdown_connections 2023-03-03T09:36:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90c1620700 1 --2- 172.21.15.7:0/1899599455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 0x7f90bc156430 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:36:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90c1620700 1 --2- 172.21.15.7:0/1899599455 >> v2:172.21.15.143:3300/0 conn(0x7f90bc14fda0 0x7f90bc071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:36:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90c1620700 1 -- 172.21.15.7:0/1899599455 >> 172.21.15.7:0/1899599455 conn(0x7f90bc0c8900 msgr2=0x7f90bc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:36:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.552+0000 7f90c1620700 1 -- 172.21.15.7:0/1899599455 shutdown_connections 2023-03-03T09:36:02.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.553+0000 7f90c1620700 1 -- 172.21.15.7:0/1899599455 wait complete. 2023-03-03T09:36:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.554+0000 7f90c1620700 1 Processor -- start 2023-03-03T09:36:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.554+0000 7f90c1620700 1 -- start start 2023-03-03T09:36:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90c1620700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90bc0abe20 0x7f90bc152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:36:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90c1620700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 0x7f90bc1e72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:36:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90c1620700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc14fda0 0x7f90bc1e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:36:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90c1620700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc06e8f0 con 0x7f90bc14f9d0 2023-03-03T09:36:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90c1620700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc1eaa30 con 0x7f90bc14fda0 2023-03-03T09:36:02.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90c1620700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc1ead10 con 0x7f90bc0abe20 2023-03-03T09:36:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90ba7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 0x7f90bc1e72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:36:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90ba7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 0x7f90bc1e72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56712/0 (socket says 172.21.15.7:56712) 2023-03-03T09:36:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90ba7fc700 1 -- 172.21.15.7:0/1938543572 learned_addr learned my addr 172.21.15.7:0/1938543572 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:36:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90bb7fe700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc14fda0 0x7f90bc1e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:36:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90baffd700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90bc0abe20 0x7f90bc152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:36:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90ba7fc700 1 -- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90bc0abe20 msgr2=0x7f90bc152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:36:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90ba7fc700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90bc0abe20 0x7f90bc152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:36:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90ba7fc700 1 -- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc14fda0 msgr2=0x7f90bc1e7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:36:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90ba7fc700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc14fda0 0x7f90bc1e7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:36:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90ba7fc700 1 -- 172.21.15.7:0/1938543572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b0009550 con 0x7f90bc14f9d0 2023-03-03T09:36:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90ba7fc700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 0x7f90bc1e72c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00e980 tx=0x7f90ac00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:36:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90a37fe700 1 -- 172.21.15.7:0/1938543572 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac009710 con 0x7f90bc14f9d0 2023-03-03T09:36:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.555+0000 7f90a37fe700 1 -- 172.21.15.7:0/1938543572 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac00c690 con 0x7f90bc14f9d0 2023-03-03T09:36:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.556+0000 7f90c1620700 1 -- 172.21.15.7:0/1938543572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90bc0ac350 con 0x7f90bc14f9d0 2023-03-03T09:36:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.556+0000 7f90a37fe700 1 -- 172.21.15.7:0/1938543572 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac00ca40 con 0x7f90bc14f9d0 2023-03-03T09:36:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.556+0000 7f90c1620700 1 -- 172.21.15.7:0/1938543572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90bc0ab630 con 0x7f90bc14f9d0 2023-03-03T09:36:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.556+0000 7f90c1620700 1 -- 172.21.15.7:0/1938543572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc0c3e10 con 0x7f90bc14f9d0 2023-03-03T09:36:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.557+0000 7f90a37fe700 1 -- 172.21.15.7:0/1938543572 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f90ac019030 con 0x7f90bc14f9d0 2023-03-03T09:36:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.557+0000 7f90a37fe700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a80682f0 0x7f90a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:36:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.558+0000 7f90a37fe700 1 -- 172.21.15.7:0/1938543572 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f90ac0860f0 con 0x7f90bc14f9d0 2023-03-03T09:36:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.558+0000 7f90baffd700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a80682f0 0x7f90a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:36:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.558+0000 7f90baffd700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a80682f0 0x7f90a806a7a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f90b0009c10 tx=0x7f90b0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:36:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.562+0000 7f90a37fe700 1 -- 172.21.15.7:0/1938543572 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f90ac0523a0 con 0x7f90bc14f9d0 2023-03-03T09:36:02.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.796+0000 7f90c1620700 1 -- 172.21.15.7:0/1938543572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90bc074bb0 con 0x7f90bc14f9d0 2023-03-03T09:36:02.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.796+0000 7f90a37fe700 1 -- 172.21.15.7:0/1938543572 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f90ac0559c0 con 0x7f90bc14f9d0 2023-03-03T09:36:02.798 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:36:02.798 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:36:02.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 -- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a80682f0 msgr2=0x7f90a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:36:02.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a80682f0 0x7f90a806a7a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f90b0009c10 tx=0x7f90b0008030 comp rx=0 tx=0).stop 2023-03-03T09:36:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 -- 172.21.15.7:0/1938543572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 msgr2=0x7f90bc1e72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:36:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 0x7f90bc1e72c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00e980 tx=0x7f90ac00ec90 comp rx=0 tx=0).stop 2023-03-03T09:36:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 -- 172.21.15.7:0/1938543572 shutdown_connections 2023-03-03T09:36:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc14fda0 0x7f90bc1e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:36:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90bc0abe20 0x7f90bc152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:36:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a80682f0 0x7f90a806a7a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:36:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 --2- 172.21.15.7:0/1938543572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc14f9d0 0x7f90bc1e72c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:36:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 -- 172.21.15.7:0/1938543572 >> 172.21.15.7:0/1938543572 conn(0x7f90bc0c8900 msgr2=0x7f90bc151af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:36:02.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 -- 172.21.15.7:0/1938543572 shutdown_connections 2023-03-03T09:36:02.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:36:02.798+0000 7f90c1620700 1 -- 172.21.15.7:0/1938543572 wait complete. 2023-03-03T09:36:02.801 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:36:02.811 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-03-03T09:36:16.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:16.899 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:46.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:36:46.971 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:02.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:37:03.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.004+0000 7f2ad3fa7700 1 Processor -- start 2023-03-03T09:37:03.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.005+0000 7f2ad3fa7700 1 -- start start 2023-03-03T09:37:03.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.005+0000 7f2ad3fa7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2acc14b0d0 0x7f2acc071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:03.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.005+0000 7f2ad3fa7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc0a9c20 con 0x7f2acc071d40 2023-03-03T09:37:03.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.005+0000 7f2ad3fa7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc154f10 con 0x7f2acc0aad20 2023-03-03T09:37:03.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.005+0000 7f2ad3fa7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2acc0ac3c0 con 0x7f2acc14b0d0 2023-03-03T09:37:03.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.005+0000 7f2ad1d43700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2acc14b0d0 0x7f2acc071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:03.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad1d43700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2acc14b0d0 0x7f2acc071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56760/0 (socket says 172.21.15.7:56760) 2023-03-03T09:37:03.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad2544700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2acc071d40 0x7f2acc15ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50744/0 (socket says 172.21.15.7:50744) 2023-03-03T09:37:03.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad1d43700 1 -- 172.21.15.7:0/137319433 learned_addr learned my addr 172.21.15.7:0/137319433 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:37:03.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad1d43700 1 -- 172.21.15.7:0/137319433 >> v1:172.21.15.143:6789/0 conn(0x7f2acc0aad20 legacy=0x7f2acc157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:03.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad1d43700 1 -- 172.21.15.7:0/137319433 >> v1:172.21.15.7:6789/0 conn(0x7f2acc071d40 legacy=0x7f2acc15ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:03.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad1d43700 1 -- 172.21.15.7:0/137319433 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc06fc70 con 0x7f2acc14b0d0 2023-03-03T09:37:03.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad0d41700 1 -- 172.21.15.7:0/137319433 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499417849 0 0) 0x7f2acc0a9c20 con 0x7f2acc071d40 2023-03-03T09:37:03.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad1d43700 1 --2- 172.21.15.7:0/137319433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14b0d0 0x7f2acc071800 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80098f0 tx=0x7f2ac8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3e01f0f0ed9e7b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:03.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad0d41700 1 -- 172.21.15.7:0/137319433 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac800b8a0 con 0x7f2acc14b0d0 2023-03-03T09:37:03.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad0d41700 1 -- 172.21.15.7:0/137319433 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac800ba00 con 0x7f2acc14b0d0 2023-03-03T09:37:03.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.006+0000 7f2ad0d41700 1 -- 172.21.15.7:0/137319433 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac800bd90 con 0x7f2acc14b0d0 2023-03-03T09:37:03.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.007+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/137319433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14b0d0 msgr2=0x7f2acc071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:03.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.007+0000 7f2ad3fa7700 1 --2- 172.21.15.7:0/137319433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14b0d0 0x7f2acc071800 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2ac80098f0 tx=0x7f2ac8009c00 comp rx=0 tx=0).stop 2023-03-03T09:37:03.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.007+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/137319433 shutdown_connections 2023-03-03T09:37:03.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.007+0000 7f2ad3fa7700 1 --2- 172.21.15.7:0/137319433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14b0d0 0x7f2acc071800 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:03.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.007+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/137319433 >> 172.21.15.7:0/137319433 conn(0x7f2acc0c8900 msgr2=0x7f2acc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:37:03.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.007+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/137319433 shutdown_connections 2023-03-03T09:37:03.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.007+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/137319433 wait complete. 2023-03-03T09:37:03.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.008+0000 7f2ad3fa7700 1 Processor -- start 2023-03-03T09:37:03.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.008+0000 7f2ad3fa7700 1 -- start start 2023-03-03T09:37:03.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad3fa7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc071d40 0x7f2acc1dfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:03.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad3fa7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0aad20 0x7f2acc152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:03.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad3fa7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14b0d0 0x7f2acc152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:03.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad3fa7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc152d80 con 0x7f2acc071d40 2023-03-03T09:37:03.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad3fa7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc1eaea0 con 0x7f2acc14b0d0 2023-03-03T09:37:03.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad3fa7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc1ea0e0 con 0x7f2acc0aad20 2023-03-03T09:37:03.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad1d43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc071d40 0x7f2acc1dfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:03.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad1d43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc071d40 0x7f2acc1dfeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56766/0 (socket says 172.21.15.7:56766) 2023-03-03T09:37:03.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad1d43700 1 -- 172.21.15.7:0/3993355496 learned_addr learned my addr 172.21.15.7:0/3993355496 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:37:03.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad2544700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14b0d0 0x7f2acc152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:03.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad1542700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0aad20 0x7f2acc152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:03.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad1d43700 1 -- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0aad20 msgr2=0x7f2acc152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:03.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad1d43700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0aad20 0x7f2acc152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:03.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad1d43700 1 -- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14b0d0 msgr2=0x7f2acc152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:03.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad1d43700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14b0d0 0x7f2acc152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:03.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad1d43700 1 -- 172.21.15.7:0/3993355496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac8009550 con 0x7f2acc071d40 2023-03-03T09:37:03.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.009+0000 7f2ad1d43700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc071d40 0x7f2acc1dfeb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8000c00 tx=0x7f2ac8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:03.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.010+0000 7f2ac27fc700 1 -- 172.21.15.7:0/3993355496 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac80119f0 con 0x7f2acc071d40 2023-03-03T09:37:03.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.010+0000 7f2ac27fc700 1 -- 172.21.15.7:0/3993355496 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac801f070 con 0x7f2acc071d40 2023-03-03T09:37:03.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.010+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/3993355496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc06f6b0 con 0x7f2acc071d40 2023-03-03T09:37:03.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.010+0000 7f2ac27fc700 1 -- 172.21.15.7:0/3993355496 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8011d70 con 0x7f2acc071d40 2023-03-03T09:37:03.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.010+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/3993355496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc155160 con 0x7f2acc071d40 2023-03-03T09:37:03.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.010+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/3993355496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc06f990 con 0x7f2acc071d40 2023-03-03T09:37:03.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.011+0000 7f2ac27fc700 1 -- 172.21.15.7:0/3993355496 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ac800f050 con 0x7f2acc071d40 2023-03-03T09:37:03.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.011+0000 7f2ac27fc700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab00682f0 0x7f2ab006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:03.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.012+0000 7f2ac27fc700 1 -- 172.21.15.7:0/3993355496 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2ac8021070 con 0x7f2acc071d40 2023-03-03T09:37:03.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.012+0000 7f2ad1542700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab00682f0 0x7f2ab006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:03.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.012+0000 7f2ad1542700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab00682f0 0x7f2ab006a7a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2acc153db0 tx=0x7f2abc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:03.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.016+0000 7f2ac27fc700 1 -- 172.21.15.7:0/3993355496 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ac8053ee0 con 0x7f2acc071d40 2023-03-03T09:37:03.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.252+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/3993355496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2acc1eacf0 con 0x7f2acc071d40 2023-03-03T09:37:03.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.253+0000 7f2ac27fc700 1 -- 172.21.15.7:0/3993355496 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2ac8057500 con 0x7f2acc071d40 2023-03-03T09:37:03.254 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:37:03.255 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:37:03.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab00682f0 msgr2=0x7f2ab006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:03.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab00682f0 0x7f2ab006a7a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2acc153db0 tx=0x7f2abc009040 comp rx=0 tx=0).stop 2023-03-03T09:37:03.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/3993355496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc071d40 msgr2=0x7f2acc1dfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:03.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc071d40 0x7f2acc1dfeb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8000c00 tx=0x7f2ac8004750 comp rx=0 tx=0).stop 2023-03-03T09:37:03.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/3993355496 shutdown_connections 2023-03-03T09:37:03.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14b0d0 0x7f2acc152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:03.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab00682f0 0x7f2ab006a7a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:03.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc071d40 0x7f2acc1dfeb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:03.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 --2- 172.21.15.7:0/3993355496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0aad20 0x7f2acc152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:03.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/3993355496 >> 172.21.15.7:0/3993355496 conn(0x7f2acc0c8900 msgr2=0x7f2acc157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:37:03.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/3993355496 shutdown_connections 2023-03-03T09:37:03.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:03.255+0000 7f2ad3fa7700 1 -- 172.21.15.7:0/3993355496 wait complete. 2023-03-03T09:37:03.258 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:37:03.268 INFO:tasks.mds_thrash.fs.[a]:waiting for 70 secs before thrashing 2023-03-03T09:37:17.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:17.038 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:24.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:37:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.949+0000 7f503ff3d700 1 Processor -- start 2023-03-03T09:37:24.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.950+0000 7f503ff3d700 1 -- start start 2023-03-03T09:37:24.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.950+0000 7f503ff3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50380c6010 0x7f50380c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:24.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503ff3d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50380c4fc0 0x7f503806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:24.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503ff3d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503814ebe0 con 0x7f50380c17f0 2023-03-03T09:37:24.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503ff3d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5038152100 con 0x7f50380c4fc0 2023-03-03T09:37:24.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503ff3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50380c2e70 con 0x7f50380c6010 2023-03-03T09:37:24.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503e4da700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50380c4fc0 0x7f503806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:24.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503d4d8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f50380c17f0 0x7f503806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50782/0 (socket says 172.21.15.7:50782) 2023-03-03T09:37:24.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503d4d8700 1 -- 172.21.15.7:0/1874885304 learned_addr learned my addr 172.21.15.7:0/1874885304 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:37:24.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503e4da700 1 -- 172.21.15.7:0/1874885304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50380c6010 msgr2=0x7f50380c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:37:24.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503e4da700 1 --2- 172.21.15.7:0/1874885304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50380c6010 0x7f50380c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:24.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503e4da700 1 -- 172.21.15.7:0/1874885304 >> v1:172.21.15.7:6789/0 conn(0x7f50380c17f0 legacy=0x7f503806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:24.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503e4da700 1 -- 172.21.15.7:0/1874885304 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50380aa680 con 0x7f50380c4fc0 2023-03-03T09:37:24.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503ccd7700 1 -- 172.21.15.7:0/1874885304 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124631812 0 0) 0x7f503814ebe0 con 0x7f50380c17f0 2023-03-03T09:37:24.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503e4da700 1 --2- 172.21.15.7:0/1874885304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 0x7f503806ddb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5030009d90 tx=0x7f503000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e850de868945faa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:24.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503ccd7700 1 -- 172.21.15.7:0/1874885304 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503000e070 con 0x7f50380c4fc0 2023-03-03T09:37:24.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503ccd7700 1 -- 172.21.15.7:0/1874885304 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503000bbd0 con 0x7f50380c4fc0 2023-03-03T09:37:24.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.951+0000 7f503ccd7700 1 -- 172.21.15.7:0/1874885304 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5030012430 con 0x7f50380c4fc0 2023-03-03T09:37:24.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.952+0000 7f503ff3d700 1 -- 172.21.15.7:0/1874885304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 msgr2=0x7f503806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:24.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.952+0000 7f503ff3d700 1 --2- 172.21.15.7:0/1874885304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 0x7f503806ddb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5030009d90 tx=0x7f503000b330 comp rx=0 tx=0).stop 2023-03-03T09:37:24.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.952+0000 7f503ff3d700 1 -- 172.21.15.7:0/1874885304 shutdown_connections 2023-03-03T09:37:24.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.952+0000 7f503ff3d700 1 --2- 172.21.15.7:0/1874885304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 0x7f503806ddb0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:24.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.952+0000 7f503ff3d700 1 --2- 172.21.15.7:0/1874885304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50380c6010 0x7f50380c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:24.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.952+0000 7f503ff3d700 1 -- 172.21.15.7:0/1874885304 >> 172.21.15.7:0/1874885304 conn(0x7f50380a9430 msgr2=0x7f50380c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:37:24.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.952+0000 7f503ff3d700 1 -- 172.21.15.7:0/1874885304 shutdown_connections 2023-03-03T09:37:24.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.952+0000 7f503ff3d700 1 -- 172.21.15.7:0/1874885304 wait complete. 2023-03-03T09:37:24.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.953+0000 7f503ff3d700 1 Processor -- start 2023-03-03T09:37:24.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.953+0000 7f503ff3d700 1 -- start start 2023-03-03T09:37:24.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503ff3d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50380c17f0 0x7f5038070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:24.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503ff3d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 0x7f5038153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:24.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503ff3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50380c6010 0x7f50381eafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503ff3d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5038071860 con 0x7f50380c4fc0 2023-03-03T09:37:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503ff3d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50380712a0 con 0x7f50380c17f0 2023-03-03T09:37:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503ff3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50380728c0 con 0x7f50380c6010 2023-03-03T09:37:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503d4d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 0x7f5038153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503d4d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 0x7f5038153040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56808/0 (socket says 172.21.15.7:56808) 2023-03-03T09:37:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503d4d8700 1 -- 172.21.15.7:0/4167915410 learned_addr learned my addr 172.21.15.7:0/4167915410 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:37:24.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503e4da700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50380c6010 0x7f50381eafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503dcd9700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50380c17f0 0x7f5038070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503d4d8700 1 -- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50380c6010 msgr2=0x7f50381eafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503d4d8700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50380c6010 0x7f50381eafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503d4d8700 1 -- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50380c17f0 msgr2=0x7f5038070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503d4d8700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50380c17f0 0x7f5038070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503d4d8700 1 -- 172.21.15.7:0/4167915410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5030009a70 con 0x7f50380c4fc0 2023-03-03T09:37:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503dcd9700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50380c17f0 0x7f5038070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:37:24.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.954+0000 7f503d4d8700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 0x7f5038153040 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c750 tx=0x7f502c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:24.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.955+0000 7f502a7fc700 1 -- 172.21.15.7:0/4167915410 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c004c80 con 0x7f50380c4fc0 2023-03-03T09:37:24.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.955+0000 7f502a7fc700 1 -- 172.21.15.7:0/4167915410 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502c004de0 con 0x7f50380c4fc0 2023-03-03T09:37:24.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.955+0000 7f503ff3d700 1 -- 172.21.15.7:0/4167915410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50380c2b00 con 0x7f50380c4fc0 2023-03-03T09:37:24.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.955+0000 7f502a7fc700 1 -- 172.21.15.7:0/4167915410 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c007310 con 0x7f50380c4fc0 2023-03-03T09:37:24.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.955+0000 7f503ff3d700 1 -- 172.21.15.7:0/4167915410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50380aa960 con 0x7f50380c4fc0 2023-03-03T09:37:24.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.956+0000 7f503ff3d700 1 -- 172.21.15.7:0/4167915410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50380c5420 con 0x7f50380c4fc0 2023-03-03T09:37:24.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.956+0000 7f502a7fc700 1 -- 172.21.15.7:0/4167915410 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f502c01a020 con 0x7f50380c4fc0 2023-03-03T09:37:24.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.957+0000 7f502a7fc700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f501c0682f0 0x7f501c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:24.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.957+0000 7f502a7fc700 1 -- 172.21.15.7:0/4167915410 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f502c085c60 con 0x7f50380c4fc0 2023-03-03T09:37:24.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.957+0000 7f503dcd9700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f501c0682f0 0x7f501c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:24.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.958+0000 7f503dcd9700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f501c0682f0 0x7f501c06a7a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5034006fd0 tx=0x7f5034008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:24.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:24.961+0000 7f502a7fc700 1 -- 172.21.15.7:0/4167915410 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f502c052000 con 0x7f50380c4fc0 2023-03-03T09:37:25.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.200+0000 7f503ff3d700 1 -- 172.21.15.7:0/4167915410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5038074bb0 con 0x7f50380c4fc0 2023-03-03T09:37:25.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.200+0000 7f502a7fc700 1 -- 172.21.15.7:0/4167915410 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f502c055620 con 0x7f50380c4fc0 2023-03-03T09:37:25.202 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:37:25.202 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:37:25.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 -- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f501c0682f0 msgr2=0x7f501c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:25.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f501c0682f0 0x7f501c06a7a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5034006fd0 tx=0x7f5034008040 comp rx=0 tx=0).stop 2023-03-03T09:37:25.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 -- 172.21.15.7:0/4167915410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 msgr2=0x7f5038153040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:25.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 0x7f5038153040 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c750 tx=0x7f502c00ca60 comp rx=0 tx=0).stop 2023-03-03T09:37:25.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 -- 172.21.15.7:0/4167915410 shutdown_connections 2023-03-03T09:37:25.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50380c6010 0x7f50381eafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:25.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50380c17f0 0x7f5038070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:25.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f501c0682f0 0x7f501c06a7a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:25.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 --2- 172.21.15.7:0/4167915410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50380c4fc0 0x7f5038153040 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:25.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 -- 172.21.15.7:0/4167915410 >> 172.21.15.7:0/4167915410 conn(0x7f50380a9430 msgr2=0x7f50380c36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:37:25.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 -- 172.21.15.7:0/4167915410 shutdown_connections 2023-03-03T09:37:25.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:25.202+0000 7f503ff3d700 1 -- 172.21.15.7:0/4167915410 wait complete. 2023-03-03T09:37:25.206 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:37:25.215 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before thrashing 2023-03-03T09:37:29.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:37:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.353+0000 7fd9aad24700 1 Processor -- start 2023-03-03T09:37:29.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.353+0000 7fd9aad24700 1 -- start start 2023-03-03T09:37:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9aad24700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a4155390 0x7fd9a40c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9aad24700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a4154910 con 0x7fd9a406e020 2023-03-03T09:37:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9aad24700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9a4152300 con 0x7fd9a4151770 2023-03-03T09:37:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9aad24700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a4070b90 con 0x7fd9a4155390 2023-03-03T09:37:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a3fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd9a406e020 0x7fd9a4071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50794/0 (socket says 172.21.15.7:50794) 2023-03-03T09:37:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a3fff700 1 -- 172.21.15.7:0/346612319 learned_addr learned my addr 172.21.15.7:0/346612319 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:37:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a8ac0700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a4155390 0x7fd9a40c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709577114 0 0) 0x7fd9a4154910 con 0x7fd9a406e020 2023-03-03T09:37:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9940035b0 con 0x7fd9a406e020 2023-03-03T09:37:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1797454522 0 0) 0x7fd9940035b0 con 0x7fd9a406e020 2023-03-03T09:37:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a4155390 msgr2=0x7fd9a40c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:37:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a37fe700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a4155390 0x7fd9a40c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 >> v1:172.21.15.143:6789/0 conn(0x7fd9a4151770 legacy=0x7fd9a4151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a40702a0 con 0x7fd9a406e020 2023-03-03T09:37:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.354+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd990001670 con 0x7fd9a406e020 2023-03-03T09:37:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 >> v1:172.21.15.7:6789/0 conn(0x7fd9a406e020 legacy=0x7fd9a4071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a37fe700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a4155390 0x7fd994003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a37fe700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd994004020 0x7fd994004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a37fe700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd994004d90 0x7fd994005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a4070b90 con 0x7fd994004d90 2023-03-03T09:37:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99400bc00 con 0x7fd994004020 2023-03-03T09:37:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99400dfd0 con 0x7fd9a4155390 2023-03-03T09:37:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a92c1700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd994004d90 0x7fd994005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:29.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a8ac0700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a4155390 0x7fd994003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:29.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a3fff700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd994004020 0x7fd994004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:29.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a92c1700 1 -- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a4155390 msgr2=0x7fd994003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:37:29.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a92c1700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a4155390 0x7fd994003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:29.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a92c1700 1 -- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd994004020 msgr2=0x7fd994004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:37:29.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a92c1700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd994004020 0x7fd994004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:29.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a92c1700 1 -- 172.21.15.7:0/346612319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9940035b0 con 0x7fd994004d90 2023-03-03T09:37:29.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a92c1700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd994004d90 0x7fd994005200 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c160 tx=0x7fd994005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebe1d47cbf1020f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:29.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c001db0 con 0x7fd994004d90 2023-03-03T09:37:29.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c00cca0 con 0x7fd994004d90 2023-03-03T09:37:29.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.355+0000 7fd9a37fe700 1 -- 172.21.15.7:0/346612319 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c003c40 con 0x7fd994004d90 2023-03-03T09:37:29.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.356+0000 7fd9aad24700 1 -- 172.21.15.7:0/346612319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd994004d90 msgr2=0x7fd994005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:29.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.356+0000 7fd9aad24700 1 --2- 172.21.15.7:0/346612319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd994004d90 0x7fd994005200 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c160 tx=0x7fd994005880 comp rx=0 tx=0).stop 2023-03-03T09:37:29.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.356+0000 7fd9a92c1700 1 -- 172.21.15.7:0/346612319 reap_dead start 2023-03-03T09:37:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.356+0000 7fd9aad24700 1 -- 172.21.15.7:0/346612319 shutdown_connections 2023-03-03T09:37:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.356+0000 7fd9aad24700 1 -- 172.21.15.7:0/346612319 >> 172.21.15.7:0/346612319 conn(0x7fd9a406c110 msgr2=0x7fd9a406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:37:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.356+0000 7fd9aad24700 1 -- 172.21.15.7:0/346612319 shutdown_connections 2023-03-03T09:37:29.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.356+0000 7fd9aad24700 1 -- 172.21.15.7:0/346612319 wait complete. 2023-03-03T09:37:29.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.357+0000 7fd9aad24700 1 Processor -- start 2023-03-03T09:37:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.358+0000 7fd9aad24700 1 -- start start 2023-03-03T09:37:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9aad24700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd994004d90 0x7fd9a4150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9aad24700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a41e5560 0x7fd9a414d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9aad24700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a41e8ad0 0x7fd9a414d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9aad24700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a40ab4e0 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9aad24700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a41e5a60 con 0x7fd994004d90 2023-03-03T09:37:29.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9aad24700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a41e5d40 con 0x7fd9a41e5560 2023-03-03T09:37:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a92c1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a41e8ad0 0x7fd9a414d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a92c1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a41e8ad0 0x7fd9a414d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56830/0 (socket says 172.21.15.7:56830) 2023-03-03T09:37:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a92c1700 1 -- 172.21.15.7:0/2998505932 learned_addr learned my addr 172.21.15.7:0/2998505932 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:37:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a8ac0700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd994004d90 0x7fd9a4150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a3fff700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a41e5560 0x7fd9a414d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a92c1700 1 -- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a41e5560 msgr2=0x7fd9a414d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:29.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a92c1700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a41e5560 0x7fd9a414d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:29.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a92c1700 1 -- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd994004d90 msgr2=0x7fd9a4150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:29.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a92c1700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd994004d90 0x7fd9a4150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:29.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a92c1700 1 -- 172.21.15.7:0/2998505932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4070820 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a8ac0700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd994004d90 0x7fd9a4150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:37:29.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a92c1700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a41e8ad0 0x7fd9a414d9c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd99c001220 tx=0x7fd99c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:29.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a17fa700 1 -- 172.21.15.7:0/2998505932 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c003e10 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.359+0000 7fd9a17fa700 1 -- 172.21.15.7:0/2998505932 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c0050d0 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.360+0000 7fd9aad24700 1 -- 172.21.15.7:0/2998505932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a4070540 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.360+0000 7fd9a17fa700 1 -- 172.21.15.7:0/2998505932 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c005480 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.360+0000 7fd9aad24700 1 -- 172.21.15.7:0/2998505932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a4155760 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.360+0000 7fd9aad24700 1 -- 172.21.15.7:0/2998505932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a4074bb0 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.361+0000 7fd9a17fa700 1 -- 172.21.15.7:0/2998505932 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd99c010050 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.361+0000 7fd9a17fa700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9880682f0 0x7fd98806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:37:29.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.361+0000 7fd9a17fa700 1 -- 172.21.15.7:0/2998505932 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd99c087460 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.362+0000 7fd9a8ac0700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9880682f0 0x7fd98806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:37:29.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.362+0000 7fd9a8ac0700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9880682f0 0x7fd98806a7a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd9a41db4f0 tx=0x7fd998008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:37:29.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.366+0000 7fd9a17fa700 1 -- 172.21.15.7:0/2998505932 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd99c053800 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.603+0000 7fd9aad24700 1 -- 172.21.15.7:0/2998505932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9a40ab620 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.603+0000 7fd9a17fa700 1 -- 172.21.15.7:0/2998505932 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd9a40ab620 con 0x7fd9a41e8ad0 2023-03-03T09:37:29.605 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:37:29.605 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:37:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 -- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9880682f0 msgr2=0x7fd98806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9880682f0 0x7fd98806a7a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd9a41db4f0 tx=0x7fd998008040 comp rx=0 tx=0).stop 2023-03-03T09:37:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 -- 172.21.15.7:0/2998505932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a41e8ad0 msgr2=0x7fd9a414d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:37:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a41e8ad0 0x7fd9a414d9c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd99c001220 tx=0x7fd99c0014a0 comp rx=0 tx=0).stop 2023-03-03T09:37:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 -- 172.21.15.7:0/2998505932 shutdown_connections 2023-03-03T09:37:29.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd9880682f0 0x7fd98806a7a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:29.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9a41e8ad0 0x7fd9a414d9c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:29.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd994004d90 0x7fd9a4150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:29.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 --2- 172.21.15.7:0/2998505932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9a41e5560 0x7fd9a414d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:37:29.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 -- 172.21.15.7:0/2998505932 >> 172.21.15.7:0/2998505932 conn(0x7fd9a406c110 msgr2=0x7fd9a4151bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:37:29.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 -- 172.21.15.7:0/2998505932 shutdown_connections 2023-03-03T09:37:29.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:37:29.605+0000 7fd9aad24700 1 -- 172.21.15.7:0/2998505932 wait complete. 2023-03-03T09:37:29.609 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:37:29.619 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T09:37:47.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:37:47.105 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:13.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:38:13.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.406+0000 7faec18bf700 1 Processor -- start 2023-03-03T09:38:13.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.406+0000 7faec18bf700 1 -- start start 2023-03-03T09:38:13.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faec18bf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faebc0c00e0 0x7faebc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:13.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faec18bf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faebc0c7310 con 0x7faebc154ff0 2023-03-03T09:38:13.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faec18bf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faebc0bfd10 con 0x7faebc0caa30 2023-03-03T09:38:13.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faec18bf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faebc155430 con 0x7faebc0c00e0 2023-03-03T09:38:13.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faebb7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faebc154ff0 0x7faebc15ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50856/0 (socket says 172.21.15.7:50856) 2023-03-03T09:38:13.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faeba7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faebc0c00e0 0x7faebc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:13.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faebb7fe700 1 -- 172.21.15.7:0/2902145242 learned_addr learned my addr 172.21.15.7:0/2902145242 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:38:13.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faeba7fc700 1 -- 172.21.15.7:0/2902145242 >> v1:172.21.15.143:6789/0 conn(0x7faebc0caa30 legacy=0x7faebc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:13.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faeba7fc700 1 -- 172.21.15.7:0/2902145242 >> v1:172.21.15.7:6789/0 conn(0x7faebc154ff0 legacy=0x7faebc15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:13.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faeba7fc700 1 -- 172.21.15.7:0/2902145242 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc0bfa30 con 0x7faebc0c00e0 2023-03-03T09:38:13.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faeb9ffb700 1 -- 172.21.15.7:0/2902145242 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949514922 0 0) 0x7faebc0c7310 con 0x7faebc154ff0 2023-03-03T09:38:13.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.407+0000 7faeba7fc700 1 --2- 172.21.15.7:0/2902145242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 0x7faebc0c5c60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faeac009820 tx=0x7faeac009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba7b2f86f12a931 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:13.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.408+0000 7faeb9ffb700 1 -- 172.21.15.7:0/2902145242 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac005e30 con 0x7faebc0c00e0 2023-03-03T09:38:13.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.408+0000 7faeb9ffb700 1 -- 172.21.15.7:0/2902145242 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac006030 con 0x7faebc0c00e0 2023-03-03T09:38:13.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.408+0000 7faeb9ffb700 1 -- 172.21.15.7:0/2902145242 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac0063e0 con 0x7faebc0c00e0 2023-03-03T09:38:13.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.408+0000 7faec18bf700 1 -- 172.21.15.7:0/2902145242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 msgr2=0x7faebc0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:13.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.408+0000 7faec18bf700 1 --2- 172.21.15.7:0/2902145242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 0x7faebc0c5c60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faeac009820 tx=0x7faeac009b30 comp rx=0 tx=0).stop 2023-03-03T09:38:13.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.408+0000 7faec18bf700 1 -- 172.21.15.7:0/2902145242 shutdown_connections 2023-03-03T09:38:13.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.408+0000 7faec18bf700 1 --2- 172.21.15.7:0/2902145242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 0x7faebc0c5c60 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:13.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.408+0000 7faec18bf700 1 -- 172.21.15.7:0/2902145242 >> 172.21.15.7:0/2902145242 conn(0x7faebc066e40 msgr2=0x7faebc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:38:13.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.408+0000 7faec18bf700 1 -- 172.21.15.7:0/2902145242 shutdown_connections 2023-03-03T09:38:13.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.408+0000 7faec18bf700 1 -- 172.21.15.7:0/2902145242 wait complete. 2023-03-03T09:38:13.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.409+0000 7faec18bf700 1 Processor -- start 2023-03-03T09:38:13.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.410+0000 7faec18bf700 1 -- start start 2023-03-03T09:38:13.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.410+0000 7faec18bf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 0x7faebc0c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:13.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.410+0000 7faec18bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faebc0caa30 0x7faebc13ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:13.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.410+0000 7faec18bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faebc154ff0 0x7faebc13f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:13.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.410+0000 7faec18bf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1584f0 con 0x7faebc0c00e0 2023-03-03T09:38:13.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.410+0000 7faec18bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faebc13f860 con 0x7faebc0caa30 2023-03-03T09:38:13.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.410+0000 7faec18bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faebc1578e0 con 0x7faebc154ff0 2023-03-03T09:38:13.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.410+0000 7faebaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 0x7faebc0c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:13.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.410+0000 7faebaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 0x7faebc0c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56878/0 (socket says 172.21.15.7:56878) 2023-03-03T09:38:13.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.410+0000 7faebaffd700 1 -- 172.21.15.7:0/966286766 learned_addr learned my addr 172.21.15.7:0/966286766 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:38:13.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faebaffd700 1 -- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faebc154ff0 msgr2=0x7faebc13f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:13.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faebb7fe700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faebc154ff0 0x7faebc13f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:13.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faeba7fc700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faebc0caa30 0x7faebc13ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:13.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faebaffd700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faebc154ff0 0x7faebc13f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:13.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faebaffd700 1 -- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faebc0caa30 msgr2=0x7faebc13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:13.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faebaffd700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faebc0caa30 0x7faebc13ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:13.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faebaffd700 1 -- 172.21.15.7:0/966286766 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac009430 con 0x7faebc0c00e0 2023-03-03T09:38:13.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faeba7fc700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faebc0caa30 0x7faebc13ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:38:13.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faebaffd700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 0x7faebc0c3e20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7faeb000c700 tx=0x7faeb000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:13.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faea3fff700 1 -- 172.21.15.7:0/966286766 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb00052d0 con 0x7faebc0c00e0 2023-03-03T09:38:13.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faea3fff700 1 -- 172.21.15.7:0/966286766 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb0005430 con 0x7faebc0c00e0 2023-03-03T09:38:13.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faec18bf700 1 -- 172.21.15.7:0/966286766 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faebc1547f0 con 0x7faebc0c00e0 2023-03-03T09:38:13.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faea3fff700 1 -- 172.21.15.7:0/966286766 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb0003880 con 0x7faebc0c00e0 2023-03-03T09:38:13.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.411+0000 7faec18bf700 1 -- 172.21.15.7:0/966286766 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faebc154510 con 0x7faebc0c00e0 2023-03-03T09:38:13.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.412+0000 7faec18bf700 1 -- 172.21.15.7:0/966286766 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faebc154ad0 con 0x7faebc0c00e0 2023-03-03T09:38:13.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.413+0000 7faea3fff700 1 -- 172.21.15.7:0/966286766 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faeb0019030 con 0x7faebc0c00e0 2023-03-03T09:38:13.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.413+0000 7faea3fff700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faea80682f0 0x7faea806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:13.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.413+0000 7faea3fff700 1 -- 172.21.15.7:0/966286766 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faeb0085430 con 0x7faebc0c00e0 2023-03-03T09:38:13.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.413+0000 7faeba7fc700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faea80682f0 0x7faea806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:13.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.414+0000 7faeba7fc700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faea80682f0 0x7faea806a7a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faebc14e030 tx=0x7faeac008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:13.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.418+0000 7faea3fff700 1 -- 172.21.15.7:0/966286766 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faeb004b5e0 con 0x7faebc0c00e0 2023-03-03T09:38:13.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.655+0000 7faec18bf700 1 -- 172.21.15.7:0/966286766 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faebc074bb0 con 0x7faebc0c00e0 2023-03-03T09:38:13.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.656+0000 7faea3fff700 1 -- 172.21.15.7:0/966286766 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faeb00517a0 con 0x7faebc0c00e0 2023-03-03T09:38:13.657 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:38:13.658 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:38:13.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 -- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faea80682f0 msgr2=0x7faea806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:13.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faea80682f0 0x7faea806a7a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faebc14e030 tx=0x7faeac008070 comp rx=0 tx=0).stop 2023-03-03T09:38:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 -- 172.21.15.7:0/966286766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 msgr2=0x7faebc0c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 0x7faebc0c3e20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7faeb000c700 tx=0x7faeb000cac0 comp rx=0 tx=0).stop 2023-03-03T09:38:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 -- 172.21.15.7:0/966286766 shutdown_connections 2023-03-03T09:38:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faebc154ff0 0x7faebc13f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faea80682f0 0x7faea806a7a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faebc0c00e0 0x7faebc0c3e20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 --2- 172.21.15.7:0/966286766 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faebc0caa30 0x7faebc13ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:13.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 -- 172.21.15.7:0/966286766 >> 172.21.15.7:0/966286766 conn(0x7faebc066e40 msgr2=0x7faebc071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:38:13.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 -- 172.21.15.7:0/966286766 shutdown_connections 2023-03-03T09:38:13.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:13.658+0000 7faec18bf700 1 -- 172.21.15.7:0/966286766 wait complete. 2023-03-03T09:38:13.661 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:38:13.671 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-03-03T09:38:17.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:17.368 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:39.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:38:39.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.811+0000 7f9a72ef3700 1 Processor -- start 2023-03-03T09:38:39.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.811+0000 7f9a72ef3700 1 -- start start 2023-03-03T09:38:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a72ef3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a6c0ace30 0x7f9a6c152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a72ef3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a6c0c26e0 0x7f9a6c152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a72ef3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6c0ac900 con 0x7f9a6c0ac220 2023-03-03T09:38:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a72ef3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c071e30 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a72ef3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c152ed0 con 0x7f9a6c0ace30 2023-03-03T09:38:39.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a71490700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a6c0c26e0 0x7f9a6c152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a70c8f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9a6c0ac220 0x7f9a6c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50880/0 (socket says 172.21.15.7:50880) 2023-03-03T09:38:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a70c8f700 1 -- 172.21.15.7:0/4077312740 learned_addr learned my addr 172.21.15.7:0/4077312740 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:38:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a6bfff700 1 --2- 172.21.15.7:0/4077312740 >> v2:172.21.15.143:3300/0 conn(0x7f9a6c0ace30 0x7f9a6c152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a71490700 1 -- 172.21.15.7:0/4077312740 >> v2:172.21.15.143:3300/0 conn(0x7f9a6c0ace30 msgr2=0x7f9a6c152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:38:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a71490700 1 --2- 172.21.15.7:0/4077312740 >> v2:172.21.15.143:3300/0 conn(0x7f9a6c0ace30 0x7f9a6c152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a71490700 1 -- 172.21.15.7:0/4077312740 >> v1:172.21.15.7:6789/0 conn(0x7f9a6c0ac220 legacy=0x7f9a6c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:39.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a71490700 1 -- 172.21.15.7:0/4077312740 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c0acb50 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a6b7fe700 1 -- 172.21.15.7:0/4077312740 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835769313 0 0) 0x7f9a6c0ac900 con 0x7f9a6c0ac220 2023-03-03T09:38:39.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.812+0000 7f9a6bfff700 1 --2- 172.21.15.7:0/4077312740 >> v2:172.21.15.143:3300/0 conn(0x7f9a6c0ace30 0x7f9a6c152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:38:39.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a71490700 1 --2- 172.21.15.7:0/4077312740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 0x7f9a6c152990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0c4eb0 tx=0x7f9a6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55168f0123ef325 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:39.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a6b7fe700 1 -- 172.21.15.7:0/4077312740 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6400e070 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a6b7fe700 1 -- 172.21.15.7:0/4077312740 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6400bbd0 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a6b7fe700 1 -- 172.21.15.7:0/4077312740 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64012430 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a72ef3700 1 -- 172.21.15.7:0/4077312740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 msgr2=0x7f9a6c152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:39.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a72ef3700 1 --2- 172.21.15.7:0/4077312740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 0x7f9a6c152990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0c4eb0 tx=0x7f9a6400b330 comp rx=0 tx=0).stop 2023-03-03T09:38:39.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a72ef3700 1 -- 172.21.15.7:0/4077312740 shutdown_connections 2023-03-03T09:38:39.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a72ef3700 1 --2- 172.21.15.7:0/4077312740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 0x7f9a6c152990 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:39.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a72ef3700 1 --2- 172.21.15.7:0/4077312740 >> v2:172.21.15.143:3300/0 conn(0x7f9a6c0ace30 0x7f9a6c152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:39.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a72ef3700 1 -- 172.21.15.7:0/4077312740 >> 172.21.15.7:0/4077312740 conn(0x7f9a6c0a94d0 msgr2=0x7f9a6c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:38:39.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a72ef3700 1 -- 172.21.15.7:0/4077312740 shutdown_connections 2023-03-03T09:38:39.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.813+0000 7f9a72ef3700 1 -- 172.21.15.7:0/4077312740 wait complete. 2023-03-03T09:38:39.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.815+0000 7f9a72ef3700 1 Processor -- start 2023-03-03T09:38:39.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.815+0000 7f9a72ef3700 1 -- start start 2023-03-03T09:38:39.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a72ef3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c0ac220 0x7f9a6c14bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:39.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a72ef3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ace30 0x7f9a6c0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:39.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a72ef3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 0x7f9a6c144b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:39.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a72ef3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c14a750 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a72ef3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c0c6e60 con 0x7f9a6c0ace30 2023-03-03T09:38:39.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a72ef3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c0c1150 con 0x7f9a6c0ac220 2023-03-03T09:38:39.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a71490700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 0x7f9a6c144b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:39.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a71490700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 0x7f9a6c144b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56908/0 (socket says 172.21.15.7:56908) 2023-03-03T09:38:39.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a71490700 1 -- 172.21.15.7:0/2857866475 learned_addr learned my addr 172.21.15.7:0/2857866475 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:38:39.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a6bfff700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ace30 0x7f9a6c0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:39.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a70c8f700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c0ac220 0x7f9a6c14bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:39.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a71490700 1 -- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c0ac220 msgr2=0x7f9a6c14bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:39.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a71490700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c0ac220 0x7f9a6c14bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:39.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a71490700 1 -- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ace30 msgr2=0x7f9a6c0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:39.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a71490700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ace30 0x7f9a6c0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:39.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.816+0000 7f9a71490700 1 -- 172.21.15.7:0/2857866475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a64009a70 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.817+0000 7f9a71490700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 0x7f9a6c144b90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c06b980 tx=0x7f9a64004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:39.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.817+0000 7f9a697fa700 1 -- 172.21.15.7:0/2857866475 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64012430 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.817+0000 7f9a697fa700 1 -- 172.21.15.7:0/2857866475 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a64020070 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.817+0000 7f9a72ef3700 1 -- 172.21.15.7:0/2857866475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6c155ca0 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.817+0000 7f9a697fa700 1 -- 172.21.15.7:0/2857866475 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64004280 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.817+0000 7f9a72ef3700 1 -- 172.21.15.7:0/2857866475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6c1559c0 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.818+0000 7f9a72ef3700 1 -- 172.21.15.7:0/2857866475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6c155f80 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.818+0000 7f9a697fa700 1 -- 172.21.15.7:0/2857866475 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a640043e0 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.819+0000 7f9a697fa700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a500682f0 0x7f9a5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:39.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.819+0000 7f9a697fa700 1 -- 172.21.15.7:0/2857866475 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9a64022070 con 0x7f9a6c0c26e0 2023-03-03T09:38:39.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.819+0000 7f9a70c8f700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a500682f0 0x7f9a5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:39.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.819+0000 7f9a70c8f700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a500682f0 0x7f9a5006a7a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9a60007fd0 tx=0x7f9a60007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:39.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:39.823+0000 7f9a697fa700 1 -- 172.21.15.7:0/2857866475 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a64054810 con 0x7f9a6c0c26e0 2023-03-03T09:38:40.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.060+0000 7f9a72ef3700 1 -- 172.21.15.7:0/2857866475 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a6c0c1610 con 0x7f9a6c0c26e0 2023-03-03T09:38:40.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.061+0000 7f9a697fa700 1 -- 172.21.15.7:0/2857866475 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9a64057e30 con 0x7f9a6c0c26e0 2023-03-03T09:38:40.062 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:38:40.063 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:38:40.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 -- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a500682f0 msgr2=0x7f9a5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a500682f0 0x7f9a5006a7a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9a60007fd0 tx=0x7f9a60007de0 comp rx=0 tx=0).stop 2023-03-03T09:38:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 -- 172.21.15.7:0/2857866475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 msgr2=0x7f9a6c144b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 0x7f9a6c144b90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c06b980 tx=0x7f9a64004ad0 comp rx=0 tx=0).stop 2023-03-03T09:38:40.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 -- 172.21.15.7:0/2857866475 shutdown_connections 2023-03-03T09:38:40.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a500682f0 0x7f9a5006a7a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:40.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6c0c26e0 0x7f9a6c144b90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:40.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a6c0ac220 0x7f9a6c14bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:40.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 --2- 172.21.15.7:0/2857866475 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a6c0ace30 0x7f9a6c0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:40.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 -- 172.21.15.7:0/2857866475 >> 172.21.15.7:0/2857866475 conn(0x7f9a6c0a94d0 msgr2=0x7f9a6c152670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:38:40.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 -- 172.21.15.7:0/2857866475 shutdown_connections 2023-03-03T09:38:40.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:40.063+0000 7f9a72ef3700 1 -- 172.21.15.7:0/2857866475 wait complete. 2023-03-03T09:38:40.067 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:38:40.076 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before thrashing 2023-03-03T09:38:42.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:38:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.215+0000 7f480f87f700 1 Processor -- start 2023-03-03T09:38:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.215+0000 7f480f87f700 1 -- start start 2023-03-03T09:38:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.216+0000 7f480f87f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 0x7f480806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.216+0000 7f480f87f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480806dad0 con 0x7f4808151de0 2023-03-03T09:38:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.216+0000 7f480f87f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480806ddb0 con 0x7f48080c5140 2023-03-03T09:38:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.216+0000 7f480f87f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480806e370 con 0x7f480806a2b0 2023-03-03T09:38:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.216+0000 7f480d61b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4808151de0 0x7f48080c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50892/0 (socket says 172.21.15.7:50892) 2023-03-03T09:38:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.216+0000 7f480d61b700 1 -- 172.21.15.7:0/2087642540 learned_addr learned my addr 172.21.15.7:0/2087642540 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:38:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.216+0000 7f480de1c700 1 --2- 172.21.15.7:0/2087642540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 0x7f480806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f47fffff700 1 -- 172.21.15.7:0/2087642540 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035084444 0 0) 0x7f480806dad0 con 0x7f4808151de0 2023-03-03T09:38:42.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f47fffff700 1 -- 172.21.15.7:0/2087642540 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f40035b0 con 0x7f4808151de0 2023-03-03T09:38:42.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f47fffff700 1 -- 172.21.15.7:0/2087642540 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099651683 0 0) 0x7f480806ddb0 con 0x7f48080c5140 2023-03-03T09:38:42.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f47fffff700 1 -- 172.21.15.7:0/2087642540 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480806dad0 con 0x7f48080c5140 2023-03-03T09:38:42.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f480de1c700 1 -- 172.21.15.7:0/2087642540 >> v1:172.21.15.143:6789/0 conn(0x7f48080c5140 legacy=0x7f4808157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f480de1c700 1 -- 172.21.15.7:0/2087642540 >> v1:172.21.15.7:6789/0 conn(0x7f4808151de0 legacy=0x7f48080c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f480de1c700 1 -- 172.21.15.7:0/2087642540 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480806e090 con 0x7f480806a2b0 2023-03-03T09:38:42.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f480de1c700 1 --2- 172.21.15.7:0/2087642540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 0x7f480806a750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4800009dc0 tx=0x7f480000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1b98ea1d3a2e452 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f47fffff700 1 -- 172.21.15.7:0/2087642540 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480000e070 con 0x7f480806a2b0 2023-03-03T09:38:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f47fffff700 1 -- 172.21.15.7:0/2087642540 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480000bb20 con 0x7f480806a2b0 2023-03-03T09:38:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.217+0000 7f47fffff700 1 -- 172.21.15.7:0/2087642540 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800012430 con 0x7f480806a2b0 2023-03-03T09:38:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.218+0000 7f480f87f700 1 -- 172.21.15.7:0/2087642540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 msgr2=0x7f480806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.218+0000 7f480f87f700 1 --2- 172.21.15.7:0/2087642540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 0x7f480806a750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4800009dc0 tx=0x7f480000b330 comp rx=0 tx=0).stop 2023-03-03T09:38:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.218+0000 7f480f87f700 1 -- 172.21.15.7:0/2087642540 shutdown_connections 2023-03-03T09:38:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.218+0000 7f480f87f700 1 --2- 172.21.15.7:0/2087642540 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 0x7f480806a750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.218+0000 7f480f87f700 1 -- 172.21.15.7:0/2087642540 >> 172.21.15.7:0/2087642540 conn(0x7f48080a9430 msgr2=0x7f48080c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:38:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.218+0000 7f480f87f700 1 -- 172.21.15.7:0/2087642540 shutdown_connections 2023-03-03T09:38:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.218+0000 7f480f87f700 1 -- 172.21.15.7:0/2087642540 wait complete. 2023-03-03T09:38:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.219+0000 7f480f87f700 1 Processor -- start 2023-03-03T09:38:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.219+0000 7f480f87f700 1 -- start start 2023-03-03T09:38:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480f87f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 0x7f4808150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480f87f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48080c5140 0x7f480814d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480f87f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4808151de0 0x7f480814d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480f87f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4808071ef0 con 0x7f4808151de0 2023-03-03T09:38:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480f87f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48081ed0f0 con 0x7f48080c5140 2023-03-03T09:38:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480f87f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48081ea930 con 0x7f480806a2b0 2023-03-03T09:38:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480de1c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4808151de0 0x7f480814d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480de1c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4808151de0 0x7f480814d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56922/0 (socket says 172.21.15.7:56922) 2023-03-03T09:38:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480de1c700 1 -- 172.21.15.7:0/1713566485 learned_addr learned my addr 172.21.15.7:0/1713566485 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:38:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480d61b700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 0x7f4808150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480ce1a700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48080c5140 0x7f480814d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480de1c700 1 -- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 msgr2=0x7f4808150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480de1c700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 0x7f4808150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480de1c700 1 -- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48080c5140 msgr2=0x7f480814d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480de1c700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48080c5140 0x7f480814d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480de1c700 1 -- 172.21.15.7:0/1713566485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4800009a70 con 0x7f4808151de0 2023-03-03T09:38:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480ce1a700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48080c5140 0x7f480814d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:38:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480de1c700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4808151de0 0x7f480814d9b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f480000b370 tx=0x7f4800004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f47fdffb700 1 -- 172.21.15.7:0/1713566485 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480001b810 con 0x7f4808151de0 2023-03-03T09:38:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f47fdffb700 1 -- 172.21.15.7:0/1713566485 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4800020070 con 0x7f4808151de0 2023-03-03T09:38:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f47fdffb700 1 -- 172.21.15.7:0/1713566485 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480001bbf0 con 0x7f4808151de0 2023-03-03T09:38:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.220+0000 7f480f87f700 1 -- 172.21.15.7:0/1713566485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48080c1c10 con 0x7f4808151de0 2023-03-03T09:38:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.221+0000 7f480f87f700 1 -- 172.21.15.7:0/1713566485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48080c4c20 con 0x7f4808151de0 2023-03-03T09:38:42.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.222+0000 7f480f87f700 1 -- 172.21.15.7:0/1713566485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f480806c800 con 0x7f4808151de0 2023-03-03T09:38:42.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.222+0000 7f47fdffb700 1 -- 172.21.15.7:0/1713566485 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f480001bd50 con 0x7f4808151de0 2023-03-03T09:38:42.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.223+0000 7f47fdffb700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47ec0682f0 0x7f47ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:38:42.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.223+0000 7f47fdffb700 1 -- 172.21.15.7:0/1713566485 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4800022070 con 0x7f4808151de0 2023-03-03T09:38:42.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.223+0000 7f480d61b700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47ec0682f0 0x7f47ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:38:42.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.223+0000 7f480d61b700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47ec0682f0 0x7f47ec06a7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4804000f60 tx=0x7f4804009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:38:42.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.227+0000 7f47fdffb700 1 -- 172.21.15.7:0/1713566485 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4800054a70 con 0x7f4808151de0 2023-03-03T09:38:42.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.466+0000 7f480f87f700 1 -- 172.21.15.7:0/1713566485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4808074bb0 con 0x7f4808151de0 2023-03-03T09:38:42.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.466+0000 7f47fdffb700 1 -- 172.21.15.7:0/1713566485 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f480002c090 con 0x7f4808151de0 2023-03-03T09:38:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:38:42.468 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:38:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 -- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47ec0682f0 msgr2=0x7f47ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47ec0682f0 0x7f47ec06a7a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4804000f60 tx=0x7f4804009040 comp rx=0 tx=0).stop 2023-03-03T09:38:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 -- 172.21.15.7:0/1713566485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4808151de0 msgr2=0x7f480814d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:38:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4808151de0 0x7f480814d9b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f480000b370 tx=0x7f4800004ad0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 -- 172.21.15.7:0/1713566485 shutdown_connections 2023-03-03T09:38:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47ec0682f0 0x7f47ec06a7a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4808151de0 0x7f480814d9b0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f480806a2b0 0x7f4808150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 --2- 172.21.15.7:0/1713566485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48080c5140 0x7f480814d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:38:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 -- 172.21.15.7:0/1713566485 >> 172.21.15.7:0/1713566485 conn(0x7f48080a9430 msgr2=0x7f48080abb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:38:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 -- 172.21.15.7:0/1713566485 shutdown_connections 2023-03-03T09:38:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:38:42.468+0000 7f480f87f700 1 -- 172.21.15.7:0/1713566485 wait complete. 2023-03-03T09:38:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:38:42.482 INFO:tasks.mds_thrash.fs.[a]:waiting for 95 secs before thrashing 2023-03-03T09:38:47.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:38:47.465 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:17.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:17.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:25.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:39:25.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.757+0000 7f28e0c0b700 1 Processor -- start 2023-03-03T09:39:25.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.757+0000 7f28e0c0b700 1 -- start start 2023-03-03T09:39:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.758+0000 7f28e0c0b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28dc071e80 0x7f28dc151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.758+0000 7f28e0c0b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f28dc14e900 0x7f28dc15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.758+0000 7f28e0c0b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28dc1510d0 con 0x7f28dc072250 2023-03-03T09:39:25.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.758+0000 7f28e0c0b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc0c3740 con 0x7f28dc071e80 2023-03-03T09:39:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.758+0000 7f28e0c0b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc070670 con 0x7f28dc14e900 2023-03-03T09:39:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.758+0000 7f28da59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28dc071e80 0x7f28dc151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.758+0000 7f28da59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28dc071e80 0x7f28dc151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56950/0 (socket says 172.21.15.7:56950) 2023-03-03T09:39:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.758+0000 7f28da59c700 1 -- 172.21.15.7:0/3525111468 learned_addr learned my addr 172.21.15.7:0/3525111468 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:39:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.758+0000 7f28dad9d700 1 --2- 172.21.15.7:0/3525111468 >> v2:172.21.15.143:3300/0 conn(0x7f28dc14e900 0x7f28dc15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28da59c700 1 -- 172.21.15.7:0/3525111468 >> v2:172.21.15.143:3300/0 conn(0x7f28dc14e900 msgr2=0x7f28dc15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:39:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28da59c700 1 --2- 172.21.15.7:0/3525111468 >> v2:172.21.15.143:3300/0 conn(0x7f28dc14e900 0x7f28dc15ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:25.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28da59c700 1 -- 172.21.15.7:0/3525111468 >> v1:172.21.15.143:6789/0 conn(0x7f28dc072250 legacy=0x7f28dc14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28da59c700 1 -- 172.21.15.7:0/3525111468 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc0c39e0 con 0x7f28dc071e80 2023-03-03T09:39:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28dad9d700 1 --2- 172.21.15.7:0/3525111468 >> v2:172.21.15.143:3300/0 conn(0x7f28dc14e900 0x7f28dc15ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:39:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28da59c700 1 --2- 172.21.15.7:0/3525111468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 0x7f28dc151820 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f28dc070de0 tx=0x7f28c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d7e3a329e7918b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28d959a700 1 -- 172.21.15.7:0/3525111468 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c400b870 con 0x7f28dc071e80 2023-03-03T09:39:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28d959a700 1 -- 172.21.15.7:0/3525111468 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c400b9d0 con 0x7f28dc071e80 2023-03-03T09:39:25.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28d959a700 1 -- 172.21.15.7:0/3525111468 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c400bd60 con 0x7f28dc071e80 2023-03-03T09:39:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28e0c0b700 1 -- 172.21.15.7:0/3525111468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 msgr2=0x7f28dc151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.759+0000 7f28e0c0b700 1 --2- 172.21.15.7:0/3525111468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 0x7f28dc151820 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f28dc070de0 tx=0x7f28c4009bd0 comp rx=0 tx=0).stop 2023-03-03T09:39:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.760+0000 7f28e0c0b700 1 -- 172.21.15.7:0/3525111468 shutdown_connections 2023-03-03T09:39:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.760+0000 7f28e0c0b700 1 --2- 172.21.15.7:0/3525111468 >> v2:172.21.15.143:3300/0 conn(0x7f28dc14e900 0x7f28dc15ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.760+0000 7f28e0c0b700 1 --2- 172.21.15.7:0/3525111468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 0x7f28dc151820 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.760+0000 7f28e0c0b700 1 -- 172.21.15.7:0/3525111468 >> 172.21.15.7:0/3525111468 conn(0x7f28dc0a94d0 msgr2=0x7f28dc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:39:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.760+0000 7f28e0c0b700 1 -- 172.21.15.7:0/3525111468 shutdown_connections 2023-03-03T09:39:25.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.760+0000 7f28e0c0b700 1 -- 172.21.15.7:0/3525111468 wait complete. 2023-03-03T09:39:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.761+0000 7f28e0c0b700 1 Processor -- start 2023-03-03T09:39:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.761+0000 7f28e0c0b700 1 -- start start 2023-03-03T09:39:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28e0c0b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 0x7f28dc0c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28e0c0b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28dc072250 0x7f28dc1de260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28e0c0b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28dc14e900 0x7f28dc0c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28e0c0b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc149b10 con 0x7f28dc071e80 2023-03-03T09:39:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28e0c0b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc143790 con 0x7f28dc072250 2023-03-03T09:39:25.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28e0c0b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc1417b0 con 0x7f28dc14e900 2023-03-03T09:39:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28da59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 0x7f28dc0c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28da59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 0x7f28dc0c5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56956/0 (socket says 172.21.15.7:56956) 2023-03-03T09:39:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28da59c700 1 -- 172.21.15.7:0/2758937959 learned_addr learned my addr 172.21.15.7:0/2758937959 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:39:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28d9d9b700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28dc072250 0x7f28dc1de260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28da59c700 1 -- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28dc14e900 msgr2=0x7f28dc0c6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28dad9d700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28dc14e900 0x7f28dc0c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:25.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28da59c700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28dc14e900 0x7f28dc0c6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28da59c700 1 -- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28dc072250 msgr2=0x7f28dc1de260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28da59c700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28dc072250 0x7f28dc1de260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28da59c700 1 -- 172.21.15.7:0/2758937959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c4009550 con 0x7f28dc071e80 2023-03-03T09:39:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28da59c700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 0x7f28dc0c5160 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f28dc06b550 tx=0x7f28c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.762+0000 7f28d2ffd700 1 -- 172.21.15.7:0/2758937959 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c4004540 con 0x7f28dc071e80 2023-03-03T09:39:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.763+0000 7f28d2ffd700 1 -- 172.21.15.7:0/2758937959 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c40036a0 con 0x7f28dc071e80 2023-03-03T09:39:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.763+0000 7f28e0c0b700 1 -- 172.21.15.7:0/2758937959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28dc06c9a0 con 0x7f28dc071e80 2023-03-03T09:39:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.763+0000 7f28d2ffd700 1 -- 172.21.15.7:0/2758937959 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c4003a30 con 0x7f28dc071e80 2023-03-03T09:39:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.763+0000 7f28e0c0b700 1 -- 172.21.15.7:0/2758937959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc151300 con 0x7f28dc071e80 2023-03-03T09:39:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.763+0000 7f28e0c0b700 1 -- 172.21.15.7:0/2758937959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28dc06cb30 con 0x7f28dc071e80 2023-03-03T09:39:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.764+0000 7f28d2ffd700 1 -- 172.21.15.7:0/2758937959 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f28c402b030 con 0x7f28dc071e80 2023-03-03T09:39:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.764+0000 7f28d2ffd700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28bc0682f0 0x7f28bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.765+0000 7f28d2ffd700 1 -- 172.21.15.7:0/2758937959 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f28c4089260 con 0x7f28dc071e80 2023-03-03T09:39:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.765+0000 7f28d9d9b700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28bc0682f0 0x7f28bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.765+0000 7f28d9d9b700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28bc0682f0 0x7f28bc06a7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f28dc1ddfc0 tx=0x7f28cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:25.769+0000 7f28d2ffd700 1 -- 172.21.15.7:0/2758937959 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f28c4055510 con 0x7f28dc071e80 2023-03-03T09:39:26.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.005+0000 7f28e0c0b700 1 -- 172.21.15.7:0/2758937959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28dc074bb0 con 0x7f28dc071e80 2023-03-03T09:39:26.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.006+0000 7f28d2ffd700 1 -- 172.21.15.7:0/2758937959 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f28c4015090 con 0x7f28dc071e80 2023-03-03T09:39:26.008 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:39:26.008 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:39:26.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 -- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28bc0682f0 msgr2=0x7f28bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:26.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28bc0682f0 0x7f28bc06a7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f28dc1ddfc0 tx=0x7f28cc009040 comp rx=0 tx=0).stop 2023-03-03T09:39:26.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 -- 172.21.15.7:0/2758937959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 msgr2=0x7f28dc0c5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:26.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 0x7f28dc0c5160 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f28dc06b550 tx=0x7f28c4004750 comp rx=0 tx=0).stop 2023-03-03T09:39:26.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 -- 172.21.15.7:0/2758937959 shutdown_connections 2023-03-03T09:39:26.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28dc14e900 0x7f28dc0c6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:26.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28bc0682f0 0x7f28bc06a7a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:26.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28dc071e80 0x7f28dc0c5160 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:26.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 --2- 172.21.15.7:0/2758937959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28dc072250 0x7f28dc1de260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:26.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 -- 172.21.15.7:0/2758937959 >> 172.21.15.7:0/2758937959 conn(0x7f28dc0a94d0 msgr2=0x7f28dc150110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:39:26.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 -- 172.21.15.7:0/2758937959 shutdown_connections 2023-03-03T09:39:26.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:26.008+0000 7f28e0c0b700 1 -- 172.21.15.7:0/2758937959 wait complete. 2023-03-03T09:39:26.011 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:39:26.021 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before thrashing 2023-03-03T09:39:35.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:39:35.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.159+0000 7f2031bc2700 1 Processor -- start 2023-03-03T09:39:35.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.159+0000 7f2031bc2700 1 -- start start 2023-03-03T09:39:35.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.160+0000 7f2031bc2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 0x7f202c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.160+0000 7f2031bc2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f202c0c00e0 0x7f202c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.160+0000 7f2031bc2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f202c0cae00 0x7f202c0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.160+0000 7f2031bc2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202c0cb280 con 0x7f202c0c00e0 2023-03-03T09:39:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.160+0000 7f2031bc2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202c0bfd10 con 0x7f202c0cae00 2023-03-03T09:39:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.160+0000 7f2031bc2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202c0bfa30 con 0x7f202c0caa30 2023-03-03T09:39:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.160+0000 7f202affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f202c0c00e0 0x7f202c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:35.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.160+0000 7f202affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f202c0c00e0 0x7f202c154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56970/0 (socket says 172.21.15.7:56970) 2023-03-03T09:39:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.160+0000 7f202affd700 1 -- 172.21.15.7:0/104027022 learned_addr learned my addr 172.21.15.7:0/104027022 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:39:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202b7fe700 1 --2- 172.21.15.7:0/104027022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 0x7f202c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202bfff700 1 --2- 172.21.15.7:0/104027022 >> v2:172.21.15.143:3300/0 conn(0x7f202c0cae00 0x7f202c0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202affd700 1 -- 172.21.15.7:0/104027022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 msgr2=0x7f202c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:39:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202affd700 1 --2- 172.21.15.7:0/104027022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 0x7f202c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202affd700 1 -- 172.21.15.7:0/104027022 >> v2:172.21.15.143:3300/0 conn(0x7f202c0cae00 msgr2=0x7f202c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:39:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202affd700 1 --2- 172.21.15.7:0/104027022 >> v2:172.21.15.143:3300/0 conn(0x7f202c0cae00 0x7f202c0c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202affd700 1 -- 172.21.15.7:0/104027022 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c154510 con 0x7f202c0c00e0 2023-03-03T09:39:35.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202bfff700 1 --2- 172.21.15.7:0/104027022 >> v2:172.21.15.143:3300/0 conn(0x7f202c0cae00 0x7f202c0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:39:35.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202affd700 1 --2- 172.21.15.7:0/104027022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0c00e0 0x7f202c154ff0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f202c158a70 tx=0x7f2020009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166aa1cf8ba04d20 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:35.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202a7fc700 1 -- 172.21.15.7:0/104027022 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020005e30 con 0x7f202c0c00e0 2023-03-03T09:39:35.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202a7fc700 1 -- 172.21.15.7:0/104027022 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2020004030 con 0x7f202c0c00e0 2023-03-03T09:39:35.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.161+0000 7f202a7fc700 1 -- 172.21.15.7:0/104027022 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20200043e0 con 0x7f202c0c00e0 2023-03-03T09:39:35.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.162+0000 7f2031bc2700 1 -- 172.21.15.7:0/104027022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0c00e0 msgr2=0x7f202c154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:35.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.162+0000 7f2031bc2700 1 --2- 172.21.15.7:0/104027022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0c00e0 0x7f202c154ff0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f202c158a70 tx=0x7f2020009bd0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.162+0000 7f2031bc2700 1 -- 172.21.15.7:0/104027022 shutdown_connections 2023-03-03T09:39:35.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.162+0000 7f2031bc2700 1 --2- 172.21.15.7:0/104027022 >> v2:172.21.15.143:3300/0 conn(0x7f202c0cae00 0x7f202c0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.162+0000 7f2031bc2700 1 --2- 172.21.15.7:0/104027022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 0x7f202c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.162+0000 7f2031bc2700 1 --2- 172.21.15.7:0/104027022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0c00e0 0x7f202c154ff0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.162+0000 7f2031bc2700 1 -- 172.21.15.7:0/104027022 >> 172.21.15.7:0/104027022 conn(0x7f202c066e40 msgr2=0x7f202c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:39:35.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.162+0000 7f2031bc2700 1 -- 172.21.15.7:0/104027022 shutdown_connections 2023-03-03T09:39:35.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.162+0000 7f2031bc2700 1 -- 172.21.15.7:0/104027022 wait complete. 2023-03-03T09:39:35.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.163+0000 7f2031bc2700 1 Processor -- start 2023-03-03T09:39:35.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.164+0000 7f2031bc2700 1 -- start start 2023-03-03T09:39:35.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.164+0000 7f2031bc2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202c0c00e0 0x7f202c1edbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:35.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.164+0000 7f2031bc2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 0x7f202c156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:35.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.164+0000 7f2031bc2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0cae00 0x7f202c158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:35.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.164+0000 7f2031bc2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c1557d0 con 0x7f202c0cae00 2023-03-03T09:39:35.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f2031bc2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202c1ee0f0 con 0x7f202c0c00e0 2023-03-03T09:39:35.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f2031bc2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202c156c40 con 0x7f202c0caa30 2023-03-03T09:39:35.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0cae00 0x7f202c158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:35.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0cae00 0x7f202c158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56978/0 (socket says 172.21.15.7:56978) 2023-03-03T09:39:35.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202bfff700 1 -- 172.21.15.7:0/3644866443 learned_addr learned my addr 172.21.15.7:0/3644866443 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:39:35.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202b7fe700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202c0c00e0 0x7f202c1edbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:35.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202affd700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 0x7f202c156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:35.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202bfff700 1 -- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 msgr2=0x7f202c156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:35.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202bfff700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 0x7f202c156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202bfff700 1 -- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202c0c00e0 msgr2=0x7f202c1edbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:35.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202bfff700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202c0c00e0 0x7f202c1edbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202bfff700 1 -- 172.21.15.7:0/3644866443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2020009500 con 0x7f202c0cae00 2023-03-03T09:39:35.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202affd700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 0x7f202c156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:39:35.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f202bfff700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0cae00 0x7f202c158100 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2024009fa0 tx=0x7f202400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:35.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f2013fff700 1 -- 172.21.15.7:0/3644866443 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202400de60 con 0x7f202c0cae00 2023-03-03T09:39:35.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f2013fff700 1 -- 172.21.15.7:0/3644866443 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202400f040 con 0x7f202c0cae00 2023-03-03T09:39:35.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f2031bc2700 1 -- 172.21.15.7:0/3644866443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c154ad0 con 0x7f202c0cae00 2023-03-03T09:39:35.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f2013fff700 1 -- 172.21.15.7:0/3644866443 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024003c80 con 0x7f202c0cae00 2023-03-03T09:39:35.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.165+0000 7f2031bc2700 1 -- 172.21.15.7:0/3644866443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202c1547f0 con 0x7f202c0cae00 2023-03-03T09:39:35.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.166+0000 7f2031bc2700 1 -- 172.21.15.7:0/3644866443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f202c0bf650 con 0x7f202c0cae00 2023-03-03T09:39:35.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.167+0000 7f2013fff700 1 -- 172.21.15.7:0/3644866443 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f202401c030 con 0x7f202c0cae00 2023-03-03T09:39:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.167+0000 7f2013fff700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f20180682f0 0x7f201806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:39:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.167+0000 7f2013fff700 1 -- 172.21.15.7:0/3644866443 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2024086130 con 0x7f202c0cae00 2023-03-03T09:39:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.168+0000 7f202b7fe700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f20180682f0 0x7f201806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:39:35.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.168+0000 7f202b7fe700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f20180682f0 0x7f201806a7a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f202c0af020 tx=0x7f201c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:39:35.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.172+0000 7f2013fff700 1 -- 172.21.15.7:0/3644866443 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f20240524d0 con 0x7f202c0cae00 2023-03-03T09:39:35.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.409+0000 7f2031bc2700 1 -- 172.21.15.7:0/3644866443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f202c074bb0 con 0x7f202c0cae00 2023-03-03T09:39:35.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.409+0000 7f2013fff700 1 -- 172.21.15.7:0/3644866443 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2024055af0 con 0x7f202c0cae00 2023-03-03T09:39:35.411 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:39:35.411 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:39:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 -- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f20180682f0 msgr2=0x7f201806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f20180682f0 0x7f201806a7a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f202c0af020 tx=0x7f201c009210 comp rx=0 tx=0).stop 2023-03-03T09:39:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 -- 172.21.15.7:0/3644866443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0cae00 msgr2=0x7f202c158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:39:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0cae00 0x7f202c158100 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2024009fa0 tx=0x7f202400d3b0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 -- 172.21.15.7:0/3644866443 shutdown_connections 2023-03-03T09:39:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f20180682f0 0x7f201806a7a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f202c0cae00 0x7f202c158100 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f202c0c00e0 0x7f202c1edbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 --2- 172.21.15.7:0/3644866443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f202c0caa30 0x7f202c156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:39:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 -- 172.21.15.7:0/3644866443 >> 172.21.15.7:0/3644866443 conn(0x7f202c066e40 msgr2=0x7f202c071710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:39:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 -- 172.21.15.7:0/3644866443 shutdown_connections 2023-03-03T09:39:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:39:35.411+0000 7f2031bc2700 1 -- 172.21.15.7:0/3644866443 wait complete. 2023-03-03T09:39:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:39:35.425 INFO:tasks.mds_thrash.fs.[b]:waiting for 99 secs before thrashing 2023-03-03T09:39:47.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:39:47.624 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:00.002 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T09:40:00.000+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T09:40:17.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:40:17.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.622+0000 7fbe8019c700 1 Processor -- start 2023-03-03T09:40:17.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.622+0000 7fbe8019c700 1 -- start start 2023-03-03T09:40:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe8019c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe78152e30 0x7fbe780c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:40:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe8019c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbe780c4910 0x7fbe78156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:40:17.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe8019c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe781523b0 con 0x7fbe780beab0 2023-03-03T09:40:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe8019c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe78072010 con 0x7fbe780c4910 2023-03-03T09:40:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe8019c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780aaaf0 con 0x7fbe78152e30 2023-03-03T09:40:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe7e739700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbe780c4910 0x7fbe78156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:40:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe7e739700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbe780c4910 0x7fbe78156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57008/0 (socket says 172.21.15.7:57008) 2023-03-03T09:40:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe7e739700 1 -- 172.21.15.7:0/1638558567 learned_addr learned my addr 172.21.15.7:0/1638558567 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:40:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe7df38700 1 --2- 172.21.15.7:0/1638558567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe78152e30 0x7fbe780c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:40:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe7e739700 1 -- 172.21.15.7:0/1638558567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe78152e30 msgr2=0x7fbe780c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:40:17.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe7e739700 1 --2- 172.21.15.7:0/1638558567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe78152e30 0x7fbe780c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:40:17.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.623+0000 7fbe7e739700 1 -- 172.21.15.7:0/1638558567 >> v1:172.21.15.143:6789/0 conn(0x7fbe780beab0 legacy=0x7fbe78155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:40:17.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe7e739700 1 -- 172.21.15.7:0/1638558567 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe780aadd0 con 0x7fbe780c4910 2023-03-03T09:40:17.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe7df38700 1 --2- 172.21.15.7:0/1638558567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe78152e30 0x7fbe780c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:40:17.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe7e739700 1 --2- 172.21.15.7:0/1638558567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 0x7fbe78156490 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009e50 tx=0x7fbe7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f88942e0150ce7b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:40:17.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe7cf36700 1 -- 172.21.15.7:0/1638558567 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7000e070 con 0x7fbe780c4910 2023-03-03T09:40:17.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe7cf36700 1 -- 172.21.15.7:0/1638558567 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe70003a20 con 0x7fbe780c4910 2023-03-03T09:40:17.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe7cf36700 1 -- 172.21.15.7:0/1638558567 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70003dd0 con 0x7fbe780c4910 2023-03-03T09:40:17.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe8019c700 1 -- 172.21.15.7:0/1638558567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 msgr2=0x7fbe78156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:40:17.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe8019c700 1 --2- 172.21.15.7:0/1638558567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 0x7fbe78156490 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009e50 tx=0x7fbe7000b330 comp rx=0 tx=0).stop 2023-03-03T09:40:17.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe8019c700 1 -- 172.21.15.7:0/1638558567 shutdown_connections 2023-03-03T09:40:17.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe8019c700 1 --2- 172.21.15.7:0/1638558567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 0x7fbe78156490 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:40:17.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe8019c700 1 --2- 172.21.15.7:0/1638558567 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe78152e30 0x7fbe780c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:40:17.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.624+0000 7fbe8019c700 1 -- 172.21.15.7:0/1638558567 >> 172.21.15.7:0/1638558567 conn(0x7fbe780736f0 msgr2=0x7fbe78073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:40:17.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.625+0000 7fbe8019c700 1 -- 172.21.15.7:0/1638558567 shutdown_connections 2023-03-03T09:40:17.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.625+0000 7fbe8019c700 1 -- 172.21.15.7:0/1638558567 wait complete. 2023-03-03T09:40:17.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.626+0000 7fbe8019c700 1 Processor -- start 2023-03-03T09:40:17.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.626+0000 7fbe8019c700 1 -- start start 2023-03-03T09:40:17.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.626+0000 7fbe8019c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe780beab0 0x7fbe780a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:40:17.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.626+0000 7fbe8019c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 0x7fbe781ed3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:40:17.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.626+0000 7fbe8019c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe78152e30 0x7fbe781ed910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:40:17.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.626+0000 7fbe8019c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe781551c0 con 0x7fbe780c4910 2023-03-03T09:40:17.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.626+0000 7fbe8019c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe781ecf40 con 0x7fbe78152e30 2023-03-03T09:40:17.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.626+0000 7fbe8019c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7806b4a0 con 0x7fbe780beab0 2023-03-03T09:40:17.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.626+0000 7fbe7d737700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 0x7fbe781ed3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:40:17.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe7d737700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 0x7fbe781ed3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57012/0 (socket says 172.21.15.7:57012) 2023-03-03T09:40:17.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe7d737700 1 -- 172.21.15.7:0/805290109 learned_addr learned my addr 172.21.15.7:0/805290109 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:40:17.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe7e739700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe78152e30 0x7fbe781ed910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:40:17.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe7df38700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe780beab0 0x7fbe780a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:40:17.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe7d737700 1 -- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe780beab0 msgr2=0x7fbe780a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:40:17.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe7d737700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe780beab0 0x7fbe780a9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:40:17.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe7d737700 1 -- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe78152e30 msgr2=0x7fbe781ed910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:40:17.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe7d737700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe78152e30 0x7fbe781ed910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:40:17.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe7d737700 1 -- 172.21.15.7:0/805290109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70009b30 con 0x7fbe780c4910 2023-03-03T09:40:17.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe7d737700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 0x7fbe781ed3d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbe7400b4a0 tx=0x7fbe7400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:40:17.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe6e7fc700 1 -- 172.21.15.7:0/805290109 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74011830 con 0x7fbe780c4910 2023-03-03T09:40:17.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe6e7fc700 1 -- 172.21.15.7:0/805290109 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe74011990 con 0x7fbe780c4910 2023-03-03T09:40:17.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe8019c700 1 -- 172.21.15.7:0/805290109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe78153200 con 0x7fbe780c4910 2023-03-03T09:40:17.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe6e7fc700 1 -- 172.21.15.7:0/805290109 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74011d40 con 0x7fbe780c4910 2023-03-03T09:40:17.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.627+0000 7fbe8019c700 1 -- 172.21.15.7:0/805290109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe78071c70 con 0x7fbe780c4910 2023-03-03T09:40:17.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.628+0000 7fbe8019c700 1 -- 172.21.15.7:0/805290109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe78153390 con 0x7fbe780c4910 2023-03-03T09:40:17.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.629+0000 7fbe6e7fc700 1 -- 172.21.15.7:0/805290109 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbe7400f050 con 0x7fbe780c4910 2023-03-03T09:40:17.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.629+0000 7fbe6e7fc700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe5c0682f0 0x7fbe5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:40:17.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.629+0000 7fbe6e7fc700 1 -- 172.21.15.7:0/805290109 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbe74085cd0 con 0x7fbe780c4910 2023-03-03T09:40:17.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.629+0000 7fbe7df38700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe5c0682f0 0x7fbe5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:40:17.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.630+0000 7fbe7df38700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe5c0682f0 0x7fbe5c06a7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbe68006fd0 tx=0x7fbe68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:40:17.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.634+0000 7fbe6e7fc700 1 -- 172.21.15.7:0/805290109 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbe74052010 con 0x7fbe780c4910 2023-03-03T09:40:17.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:17.695 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:17.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.873+0000 7fbe8019c700 1 -- 172.21.15.7:0/805290109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe781ed0b0 con 0x7fbe780c4910 2023-03-03T09:40:17.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.874+0000 7fbe6e7fc700 1 -- 172.21.15.7:0/805290109 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbe74055630 con 0x7fbe780c4910 2023-03-03T09:40:17.875 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:40:17.875 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:40:17.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.875+0000 7fbe8019c700 1 -- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe5c0682f0 msgr2=0x7fbe5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:40:17.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.875+0000 7fbe8019c700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe5c0682f0 0x7fbe5c06a7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbe68006fd0 tx=0x7fbe68008040 comp rx=0 tx=0).stop 2023-03-03T09:40:17.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.875+0000 7fbe8019c700 1 -- 172.21.15.7:0/805290109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 msgr2=0x7fbe781ed3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:40:17.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.875+0000 7fbe8019c700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 0x7fbe781ed3d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbe7400b4a0 tx=0x7fbe7400b860 comp rx=0 tx=0).stop 2023-03-03T09:40:17.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.875+0000 7fbe8019c700 1 -- 172.21.15.7:0/805290109 shutdown_connections 2023-03-03T09:40:17.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.875+0000 7fbe8019c700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe78152e30 0x7fbe781ed910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:40:17.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.876+0000 7fbe8019c700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe780beab0 0x7fbe780a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:40:17.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.876+0000 7fbe8019c700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbe5c0682f0 0x7fbe5c06a7a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:40:17.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.876+0000 7fbe8019c700 1 --2- 172.21.15.7:0/805290109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe780c4910 0x7fbe781ed3d0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:40:17.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.876+0000 7fbe8019c700 1 -- 172.21.15.7:0/805290109 >> 172.21.15.7:0/805290109 conn(0x7fbe780736f0 msgr2=0x7fbe780bd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:40:17.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.876+0000 7fbe8019c700 1 -- 172.21.15.7:0/805290109 shutdown_connections 2023-03-03T09:40:17.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:40:17.876+0000 7fbe8019c700 1 -- 172.21.15.7:0/805290109 wait complete. 2023-03-03T09:40:17.879 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:40:17.889 INFO:tasks.mds_thrash.fs.[a]:waiting for 59 secs before thrashing 2023-03-03T09:40:47.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:40:47.767 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:14.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:41:14.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.562+0000 7f0944eaf700 1 Processor -- start 2023-03-03T09:41:14.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.562+0000 7f0944eaf700 1 -- start start 2023-03-03T09:41:14.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f0944eaf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0940151e80 0x7f09400c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:14.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f0944eaf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 0x7f09400bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:14.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f0944eaf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09400c08d0 con 0x7f0940155410 2023-03-03T09:41:14.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f0944eaf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0940152250 con 0x7f0940151e80 2023-03-03T09:41:14.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f0944eaf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09400c0fb0 con 0x7f09400c6fe0 2023-03-03T09:41:14.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f093dd9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0940155410 0x7f09400bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51052/0 (socket says 172.21.15.7:51052) 2023-03-03T09:41:14.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f093dd9b700 1 -- 172.21.15.7:0/659441926 learned_addr learned my addr 172.21.15.7:0/659441926 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:41:14.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f093e59c700 1 --2- 172.21.15.7:0/659441926 >> v2:172.21.15.7:3300/0 conn(0x7f0940151e80 0x7f09400c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:14.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f093ed9d700 1 --2- 172.21.15.7:0/659441926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 0x7f09400bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:14.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f093e59c700 1 -- 172.21.15.7:0/659441926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 msgr2=0x7f09400bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:41:14.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f093e59c700 1 --2- 172.21.15.7:0/659441926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 0x7f09400bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:14.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f093e59c700 1 -- 172.21.15.7:0/659441926 >> v1:172.21.15.7:6789/0 conn(0x7f0940155410 legacy=0x7f09400bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:14.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f093d59a700 1 -- 172.21.15.7:0/659441926 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746625272 0 0) 0x7f09400c08d0 con 0x7f0940155410 2023-03-03T09:41:14.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.563+0000 7f093e59c700 1 -- 172.21.15.7:0/659441926 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0940154930 con 0x7f0940151e80 2023-03-03T09:41:14.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f093e59c700 1 --2- 172.21.15.7:0/659441926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 0x7f09400c6aa0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0930009840 tx=0x7f0930009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd30bca0a0d7a1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:14.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f093d59a700 1 -- 172.21.15.7:0/659441926 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930011830 con 0x7f0940151e80 2023-03-03T09:41:14.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f093d59a700 1 -- 172.21.15.7:0/659441926 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0930011990 con 0x7f0940151e80 2023-03-03T09:41:14.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f093d59a700 1 -- 172.21.15.7:0/659441926 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930011d20 con 0x7f0940151e80 2023-03-03T09:41:14.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f0944eaf700 1 -- 172.21.15.7:0/659441926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 msgr2=0x7f09400c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:14.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f0944eaf700 1 --2- 172.21.15.7:0/659441926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 0x7f09400c6aa0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0930009840 tx=0x7f0930009b50 comp rx=0 tx=0).stop 2023-03-03T09:41:14.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f0944eaf700 1 -- 172.21.15.7:0/659441926 shutdown_connections 2023-03-03T09:41:14.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f0944eaf700 1 --2- 172.21.15.7:0/659441926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 0x7f09400bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:14.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f0944eaf700 1 --2- 172.21.15.7:0/659441926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 0x7f09400c6aa0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:14.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f0944eaf700 1 -- 172.21.15.7:0/659441926 >> 172.21.15.7:0/659441926 conn(0x7f0940071630 msgr2=0x7f094006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:41:14.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f0944eaf700 1 -- 172.21.15.7:0/659441926 shutdown_connections 2023-03-03T09:41:14.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.564+0000 7f0944eaf700 1 -- 172.21.15.7:0/659441926 wait complete. 2023-03-03T09:41:14.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.566+0000 7f0944eaf700 1 Processor -- start 2023-03-03T09:41:14.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.566+0000 7f0944eaf700 1 -- start start 2023-03-03T09:41:14.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.566+0000 7f0944eaf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 0x7f09400bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:14.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.566+0000 7f0944eaf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 0x7f094014c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:14.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.566+0000 7f0944eaf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0940155410 0x7f094014cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:14.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.566+0000 7f0944eaf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0940152720 con 0x7f0940151e80 2023-03-03T09:41:14.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.566+0000 7f0944eaf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094014bcf0 con 0x7f0940155410 2023-03-03T09:41:14.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.566+0000 7f0944eaf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f094014c550 con 0x7f09400c6fe0 2023-03-03T09:41:14.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.566+0000 7f093dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 0x7f094014c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:14.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 0x7f094014c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57078/0 (socket says 172.21.15.7:57078) 2023-03-03T09:41:14.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093dd9b700 1 -- 172.21.15.7:0/2595315466 learned_addr learned my addr 172.21.15.7:0/2595315466 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:41:14.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093e59c700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 0x7f09400bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:14.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093ed9d700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0940155410 0x7f094014cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:14.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093dd9b700 1 -- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 msgr2=0x7f09400bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:14.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093dd9b700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 0x7f09400bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:14.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093dd9b700 1 -- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0940155410 msgr2=0x7f094014cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:14.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093dd9b700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0940155410 0x7f094014cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:14.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093dd9b700 1 -- 172.21.15.7:0/2595315466 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09300094a0 con 0x7f0940151e80 2023-03-03T09:41:14.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093ed9d700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0940155410 0x7f094014cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:41:14.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093e59c700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 0x7f09400bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:41:14.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f093dd9b700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 0x7f094014c9e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f093400c520 tx=0x7f093400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:14.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f092effd700 1 -- 172.21.15.7:0/2595315466 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934004c80 con 0x7f0940151e80 2023-03-03T09:41:14.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f092effd700 1 -- 172.21.15.7:0/2595315466 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0934004e00 con 0x7f0940151e80 2023-03-03T09:41:14.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f0944eaf700 1 -- 172.21.15.7:0/2595315466 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0940154ef0 con 0x7f0940151e80 2023-03-03T09:41:14.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f092effd700 1 -- 172.21.15.7:0/2595315466 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934007310 con 0x7f0940151e80 2023-03-03T09:41:14.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.567+0000 7f0944eaf700 1 -- 172.21.15.7:0/2595315466 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0940154c10 con 0x7f0940151e80 2023-03-03T09:41:14.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.568+0000 7f0944eaf700 1 -- 172.21.15.7:0/2595315466 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09400c5220 con 0x7f0940151e80 2023-03-03T09:41:14.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.569+0000 7f092effd700 1 -- 172.21.15.7:0/2595315466 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f093401a020 con 0x7f0940151e80 2023-03-03T09:41:14.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.569+0000 7f092effd700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 0x7f092006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:14.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.569+0000 7f092effd700 1 -- 172.21.15.7:0/2595315466 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0934085d00 con 0x7f0940151e80 2023-03-03T09:41:14.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.569+0000 7f093e59c700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 0x7f092006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:14.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.570+0000 7f093e59c700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 0x7f092006a7a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0930009fd0 tx=0x7f093001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:14.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.573+0000 7f092effd700 1 -- 172.21.15.7:0/2595315466 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0934089030 con 0x7f0940151e80 2023-03-03T09:41:14.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.814+0000 7f0944eaf700 1 -- 172.21.15.7:0/2595315466 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f094014c6c0 con 0x7f0940151e80 2023-03-03T09:41:14.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.814+0000 7f092effd700 1 -- 172.21.15.7:0/2595315466 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f09340520a0 con 0x7f0940151e80 2023-03-03T09:41:14.817 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:41:14.817 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:41:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.816+0000 7f0944eaf700 1 -- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 msgr2=0x7f092006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.816+0000 7f0944eaf700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 0x7f092006a7a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0930009fd0 tx=0x7f093001c040 comp rx=0 tx=0).stop 2023-03-03T09:41:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.816+0000 7f0944eaf700 1 -- 172.21.15.7:0/2595315466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 msgr2=0x7f094014c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.816+0000 7f0944eaf700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 0x7f094014c9e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f093400c520 tx=0x7f093400c8e0 comp rx=0 tx=0).stop 2023-03-03T09:41:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.816+0000 7f0944eaf700 1 -- 172.21.15.7:0/2595315466 shutdown_connections 2023-03-03T09:41:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.816+0000 7f0944eaf700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0940155410 0x7f094014cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.816+0000 7f0944eaf700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c6fe0 0x7f09400bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.816+0000 7f0944eaf700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 0x7f092006a7a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.816+0000 7f0944eaf700 1 --2- 172.21.15.7:0/2595315466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0940151e80 0x7f094014c9e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.816+0000 7f0944eaf700 1 -- 172.21.15.7:0/2595315466 >> 172.21.15.7:0/2595315466 conn(0x7f0940071630 msgr2=0x7f09400c40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:41:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.817+0000 7f0944eaf700 1 -- 172.21.15.7:0/2595315466 shutdown_connections 2023-03-03T09:41:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:14.817+0000 7f0944eaf700 1 -- 172.21.15.7:0/2595315466 wait complete. 2023-03-03T09:41:14.821 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:41:14.831 INFO:tasks.mds_thrash.fs.[b]:waiting for 58 secs before thrashing 2023-03-03T09:41:16.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:41:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.028+0000 7fc3ec6f3700 1 Processor -- start 2023-03-03T09:41:17.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.028+0000 7fc3ec6f3700 1 -- start start 2023-03-03T09:41:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3ec6f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4151d40 0x7fc3e40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3ec6f3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc3e40c2930 0x7fc3e4157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3ec6f3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e406d430 con 0x7fc3e4073c30 2023-03-03T09:41:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3ec6f3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e406c2a0 con 0x7fc3e40c2930 2023-03-03T09:41:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3ec6f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e41586e0 con 0x7fc3e4151d40 2023-03-03T09:41:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3eac90700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc3e4073c30 0x7fc3e4157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51072/0 (socket says 172.21.15.7:51072) 2023-03-03T09:41:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3eac90700 1 -- 172.21.15.7:0/298793657 learned_addr learned my addr 172.21.15.7:0/298793657 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:41:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3e9c8e700 1 --2- 172.21.15.7:0/298793657 >> v2:172.21.15.143:3300/0 conn(0x7fc3e40c2930 0x7fc3e4157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097670080 0 0) 0x7fc3e406d430 con 0x7fc3e4073c30 2023-03-03T09:41:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d00035b0 con 0x7fc3e4073c30 2023-03-03T09:41:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3ea48f700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4151d40 0x7fc3e40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.029+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2569445518 0 0) 0x7fc3d00035b0 con 0x7fc3e4073c30 2023-03-03T09:41:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4151d40 msgr2=0x7fc3e40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:41:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4151d40 0x7fc3e40736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 >> v2:172.21.15.143:3300/0 conn(0x7fc3e40c2930 msgr2=0x7fc3e4157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:41:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 --2- 172.21.15.7:0/298793657 >> v2:172.21.15.143:3300/0 conn(0x7fc3e40c2930 0x7fc3e4157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e406cb00 con 0x7fc3e4073c30 2023-03-03T09:41:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc3dc0019f0 con 0x7fc3e4073c30 2023-03-03T09:41:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 >> v1:172.21.15.7:6789/0 conn(0x7fc3e4073c30 legacy=0x7fc3e4157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4151d40 0x7fc3d0003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3d0003ff0 0x7fc3d0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d0004d90 0x7fc3d0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e406c2a0 con 0x7fc3d0004d90 2023-03-03T09:41:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e41586e0 con 0x7fc3d0003ff0 2023-03-03T09:41:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d000bb70 con 0x7fc3e4151d40 2023-03-03T09:41:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3eac90700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d0004d90 0x7fc3d0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3ea48f700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4151d40 0x7fc3d0003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e9c8e700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3d0003ff0 0x7fc3d0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3eac90700 1 -- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4151d40 msgr2=0x7fc3d0003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:41:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3eac90700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e4151d40 0x7fc3d0003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3eac90700 1 -- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3d0003ff0 msgr2=0x7fc3d0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:41:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3eac90700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3d0003ff0 0x7fc3d0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3eac90700 1 -- 172.21.15.7:0/298793657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d00035b0 con 0x7fc3d0004d90 2023-03-03T09:41:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3eac90700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d0004d90 0x7fc3d0005200 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0013d0 tx=0x7fc3d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b6dab7f6c782beb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00a340 con 0x7fc3d0004d90 2023-03-03T09:41:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc012600 con 0x7fc3d0004d90 2023-03-03T09:41:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.030+0000 7fc3e948d700 1 -- 172.21.15.7:0/298793657 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0129b0 con 0x7fc3d0004d90 2023-03-03T09:41:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.031+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/298793657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d0004d90 msgr2=0x7fc3d0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.031+0000 7fc3ec6f3700 1 --2- 172.21.15.7:0/298793657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d0004d90 0x7fc3d0005200 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0013d0 tx=0x7fc3d0005880 comp rx=0 tx=0).stop 2023-03-03T09:41:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.031+0000 7fc3eac90700 1 -- 172.21.15.7:0/298793657 reap_dead start 2023-03-03T09:41:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.031+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/298793657 shutdown_connections 2023-03-03T09:41:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.031+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/298793657 >> 172.21.15.7:0/298793657 conn(0x7fc3e40a94d0 msgr2=0x7fc3e40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:41:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.031+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/298793657 shutdown_connections 2023-03-03T09:41:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.031+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/298793657 wait complete. 2023-03-03T09:41:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.032+0000 7fc3ec6f3700 1 Processor -- start 2023-03-03T09:41:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.032+0000 7fc3ec6f3700 1 -- start start 2023-03-03T09:41:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3ec6f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3d0004d90 0x7fc3e40bd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3ec6f3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e40becb0 0x7fc3e4155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3ec6f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c5160 0x7fc3e414d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3ec6f3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40bd8c0 con 0x7fc3e40becb0 2023-03-03T09:41:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3ec6f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e414ef30 con 0x7fc3e40c5160 2023-03-03T09:41:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3ec6f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40c6f30 con 0x7fc3d0004d90 2023-03-03T09:41:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3e9c8e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e40becb0 0x7fc3e4155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3e9c8e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e40becb0 0x7fc3e4155b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57102/0 (socket says 172.21.15.7:57102) 2023-03-03T09:41:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3e9c8e700 1 -- 172.21.15.7:0/3731098088 learned_addr learned my addr 172.21.15.7:0/3731098088 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:41:17.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3ea48f700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3d0004d90 0x7fc3e40bd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:17.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3eac90700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c5160 0x7fc3e414d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:17.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3e9c8e700 1 -- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3d0004d90 msgr2=0x7fc3e40bd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:17.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3e9c8e700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3d0004d90 0x7fc3e40bd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:17.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.033+0000 7fc3e9c8e700 1 -- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c5160 msgr2=0x7fc3e414d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:17.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.034+0000 7fc3e9c8e700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c5160 0x7fc3e414d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.034+0000 7fc3e9c8e700 1 -- 172.21.15.7:0/3731098088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e406d0c0 con 0x7fc3e40becb0 2023-03-03T09:41:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.034+0000 7fc3e9c8e700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e40becb0 0x7fc3e4155b50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4011aa0 tx=0x7fc3d4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.034+0000 7fc3daffd700 1 -- 172.21.15.7:0/3731098088 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d400f980 con 0x7fc3e40becb0 2023-03-03T09:41:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.034+0000 7fc3daffd700 1 -- 172.21.15.7:0/3731098088 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d400fb00 con 0x7fc3e40becb0 2023-03-03T09:41:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.034+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/3731098088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e406cde0 con 0x7fc3e40becb0 2023-03-03T09:41:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.034+0000 7fc3daffd700 1 -- 172.21.15.7:0/3731098088 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d400feb0 con 0x7fc3e40becb0 2023-03-03T09:41:17.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.034+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/3731098088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e406bea0 con 0x7fc3e40becb0 2023-03-03T09:41:17.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.035+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/3731098088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e41eabd0 con 0x7fc3e40becb0 2023-03-03T09:41:17.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.035+0000 7fc3daffd700 1 -- 172.21.15.7:0/3731098088 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc3d4014440 con 0x7fc3e40becb0 2023-03-03T09:41:17.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.036+0000 7fc3daffd700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 0x7fc3c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:17.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.036+0000 7fc3daffd700 1 -- 172.21.15.7:0/3731098088 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc3d4086b40 con 0x7fc3e40becb0 2023-03-03T09:41:17.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.036+0000 7fc3ea48f700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 0x7fc3c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:17.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.036+0000 7fc3ea48f700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 0x7fc3c806a7a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009ba0 tx=0x7fc3e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:17.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.040+0000 7fc3daffd700 1 -- 172.21.15.7:0/3731098088 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc3d4052f00 con 0x7fc3e40becb0 2023-03-03T09:41:17.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.277+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/3731098088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3e40724f0 con 0x7fc3e40becb0 2023-03-03T09:41:17.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.277+0000 7fc3daffd700 1 -- 172.21.15.7:0/3731098088 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc3d4056520 con 0x7fc3e40becb0 2023-03-03T09:41:17.279 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:41:17.279 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:41:17.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 msgr2=0x7fc3c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:17.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 0x7fc3c806a7a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009ba0 tx=0x7fc3e0008040 comp rx=0 tx=0).stop 2023-03-03T09:41:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/3731098088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e40becb0 msgr2=0x7fc3e4155b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e40becb0 0x7fc3e4155b50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4011aa0 tx=0x7fc3d4011e60 comp rx=0 tx=0).stop 2023-03-03T09:41:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/3731098088 shutdown_connections 2023-03-03T09:41:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e40c5160 0x7fc3e414d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3d0004d90 0x7fc3e40bd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 0x7fc3c806a7a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 --2- 172.21.15.7:0/3731098088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e40becb0 0x7fc3e4155b50 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/3731098088 >> 172.21.15.7:0/3731098088 conn(0x7fc3e40a94d0 msgr2=0x7fc3e40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:41:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/3731098088 shutdown_connections 2023-03-03T09:41:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:17.279+0000 7fc3ec6f3700 1 -- 172.21.15.7:0/3731098088 wait complete. 2023-03-03T09:41:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:41:17.292 INFO:tasks.mds_thrash.fs.[a]:waiting for 40 secs before thrashing 2023-03-03T09:41:17.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:18.079 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:48.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:48.143 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:41:57.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:41:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.430+0000 7efcdb9e2700 1 Processor -- start 2023-03-03T09:41:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.431+0000 7efcdb9e2700 1 -- start start 2023-03-03T09:41:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.431+0000 7efcdb9e2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efcd414f9d0 0x7efcd4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.431+0000 7efcdb9e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 0x7efcd41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.431+0000 7efcdb9e2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd40c2f30 con 0x7efcd414fda0 2023-03-03T09:41:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.431+0000 7efcdb9e2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd40c31d0 con 0x7efcd414f9d0 2023-03-03T09:41:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.431+0000 7efcdb9e2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd41524a0 con 0x7efcd40abe20 2023-03-03T09:41:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcd8f7d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7efcd414fda0 0x7efcd4071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51138/0 (socket says 172.21.15.7:51138) 2023-03-03T09:41:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcd8f7d700 1 -- 172.21.15.7:0/2543997317 learned_addr learned my addr 172.21.15.7:0/2543997317 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:41:57.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcd977e700 1 --2- 172.21.15.7:0/2543997317 >> v2:172.21.15.143:3300/0 conn(0x7efcd414f9d0 0x7efcd4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcd9f7f700 1 --2- 172.21.15.7:0/2543997317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 0x7efcd41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcc7fff700 1 -- 172.21.15.7:0/2543997317 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910733647 0 0) 0x7efcd40c2f30 con 0x7efcd414fda0 2023-03-03T09:41:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcc7fff700 1 -- 172.21.15.7:0/2543997317 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc00035b0 con 0x7efcd414fda0 2023-03-03T09:41:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcc7fff700 1 -- 172.21.15.7:0/2543997317 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2607040858 0 0) 0x7efcc00035b0 con 0x7efcd414fda0 2023-03-03T09:41:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcd9f7f700 1 -- 172.21.15.7:0/2543997317 >> v2:172.21.15.143:3300/0 conn(0x7efcd414f9d0 msgr2=0x7efcd4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:41:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcd9f7f700 1 --2- 172.21.15.7:0/2543997317 >> v2:172.21.15.143:3300/0 conn(0x7efcd414f9d0 0x7efcd4156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:57.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcd9f7f700 1 -- 172.21.15.7:0/2543997317 >> v1:172.21.15.7:6789/0 conn(0x7efcd414fda0 legacy=0x7efcd4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcd9f7f700 1 -- 172.21.15.7:0/2543997317 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40c34b0 con 0x7efcd40abe20 2023-03-03T09:41:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcd977e700 1 --2- 172.21.15.7:0/2543997317 >> v2:172.21.15.143:3300/0 conn(0x7efcd414f9d0 0x7efcd4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:41:57.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.432+0000 7efcd9f7f700 1 --2- 172.21.15.7:0/2543997317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 0x7efcd41529c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efccc009d90 tx=0x7efccc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb768226d4de61c1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:57.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcc7fff700 1 -- 172.21.15.7:0/2543997317 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc00e070 con 0x7efcd40abe20 2023-03-03T09:41:57.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcc7fff700 1 -- 172.21.15.7:0/2543997317 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc003a20 con 0x7efcd40abe20 2023-03-03T09:41:57.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcc7fff700 1 -- 172.21.15.7:0/2543997317 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc8001670 con 0x7efcd40abe20 2023-03-03T09:41:57.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2543997317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 msgr2=0x7efcd41529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:57.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcdb9e2700 1 --2- 172.21.15.7:0/2543997317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 0x7efcd41529c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efccc009d90 tx=0x7efccc00b330 comp rx=0 tx=0).stop 2023-03-03T09:41:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2543997317 shutdown_connections 2023-03-03T09:41:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcdb9e2700 1 --2- 172.21.15.7:0/2543997317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 0x7efcd41529c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcdb9e2700 1 --2- 172.21.15.7:0/2543997317 >> v2:172.21.15.143:3300/0 conn(0x7efcd414f9d0 0x7efcd4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2543997317 >> 172.21.15.7:0/2543997317 conn(0x7efcd40c8900 msgr2=0x7efcd40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:41:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2543997317 shutdown_connections 2023-03-03T09:41:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.433+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2543997317 wait complete. 2023-03-03T09:41:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.434+0000 7efcdb9e2700 1 Processor -- start 2023-03-03T09:41:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcdb9e2700 1 -- start start 2023-03-03T09:41:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcdb9e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 0x7efcd4143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcdb9e2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcd414f9d0 0x7efcd40c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcdb9e2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd414fda0 0x7efcd406aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcdb9e2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd41e9b10 con 0x7efcd414fda0 2023-03-03T09:41:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcdb9e2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd406aa10 con 0x7efcd414f9d0 2023-03-03T09:41:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcdb9e2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd41e7340 con 0x7efcd40abe20 2023-03-03T09:41:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcd9f7f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd414fda0 0x7efcd406aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcd9f7f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd414fda0 0x7efcd406aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57166/0 (socket says 172.21.15.7:57166) 2023-03-03T09:41:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcd9f7f700 1 -- 172.21.15.7:0/2369359946 learned_addr learned my addr 172.21.15.7:0/2369359946 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:41:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.435+0000 7efcd977e700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 0x7efcd4143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcd8f7d700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcd414f9d0 0x7efcd40c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcd9f7f700 1 -- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 msgr2=0x7efcd4143b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcd9f7f700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 0x7efcd4143b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcd9f7f700 1 -- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcd414f9d0 msgr2=0x7efcd40c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcd9f7f700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcd414f9d0 0x7efcd40c17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcd9f7f700 1 -- 172.21.15.7:0/2369359946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd00094a0 con 0x7efcd414fda0 2023-03-03T09:41:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcd9f7f700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd414fda0 0x7efcd406aea0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7efcd41525e0 tx=0x7efccc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcc5ffb700 1 -- 172.21.15.7:0/2369359946 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc004440 con 0x7efcd414fda0 2023-03-03T09:41:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcc5ffb700 1 -- 172.21.15.7:0/2369359946 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc0045c0 con 0x7efcd414fda0 2023-03-03T09:41:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2369359946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efccc009a70 con 0x7efcd414fda0 2023-03-03T09:41:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcc5ffb700 1 -- 172.21.15.7:0/2369359946 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc004970 con 0x7efcd414fda0 2023-03-03T09:41:57.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.436+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2369359946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd40ac350 con 0x7efcd414fda0 2023-03-03T09:41:57.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.437+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2369359946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd40ab630 con 0x7efcd414fda0 2023-03-03T09:41:57.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.438+0000 7efcc5ffb700 1 -- 172.21.15.7:0/2369359946 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efccc012430 con 0x7efcd414fda0 2023-03-03T09:41:57.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.438+0000 7efcc5ffb700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb80682f0 0x7efcb806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:41:57.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.438+0000 7efcc5ffb700 1 -- 172.21.15.7:0/2369359946 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efccc087d50 con 0x7efcd414fda0 2023-03-03T09:41:57.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.438+0000 7efcd977e700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb80682f0 0x7efcb806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:41:57.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.439+0000 7efcd977e700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb80682f0 0x7efcb806a7a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009d70 tx=0x7efcd0009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:41:57.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.443+0000 7efcc5ffb700 1 -- 172.21.15.7:0/2369359946 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efccc054090 con 0x7efcd414fda0 2023-03-03T09:41:57.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.680+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2369359946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcd4071fa0 con 0x7efcd414fda0 2023-03-03T09:41:57.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.680+0000 7efcc5ffb700 1 -- 172.21.15.7:0/2369359946 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efccc0576b0 con 0x7efcd414fda0 2023-03-03T09:41:57.682 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:41:57.682 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:41:57.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.682+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb80682f0 msgr2=0x7efcb806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:57.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.682+0000 7efcdb9e2700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb80682f0 0x7efcb806a7a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009d70 tx=0x7efcd0009210 comp rx=0 tx=0).stop 2023-03-03T09:41:57.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.682+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2369359946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd414fda0 msgr2=0x7efcd406aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:41:57.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.682+0000 7efcdb9e2700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd414fda0 0x7efcd406aea0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7efcd41525e0 tx=0x7efccc004ad0 comp rx=0 tx=0).stop 2023-03-03T09:41:57.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.682+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2369359946 shutdown_connections 2023-03-03T09:41:57.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.682+0000 7efcdb9e2700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcb80682f0 0x7efcb806a7a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:57.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.682+0000 7efcdb9e2700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcd414fda0 0x7efcd406aea0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:57.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.682+0000 7efcdb9e2700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcd40abe20 0x7efcd4143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:57.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.682+0000 7efcdb9e2700 1 --2- 172.21.15.7:0/2369359946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcd414f9d0 0x7efcd40c17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:41:57.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.682+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2369359946 >> 172.21.15.7:0/2369359946 conn(0x7efcd40c8900 msgr2=0x7efcd40c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:41:57.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.683+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2369359946 shutdown_connections 2023-03-03T09:41:57.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:41:57.683+0000 7efcdb9e2700 1 -- 172.21.15.7:0/2369359946 wait complete. 2023-03-03T09:41:57.686 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:41:57.696 INFO:tasks.mds_thrash.fs.[a]:waiting for 18 secs before thrashing 2023-03-03T09:42:12.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:42:12.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.968+0000 7faa6b021700 1 Processor -- start 2023-03-03T09:42:12.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.968+0000 7faa6b021700 1 -- start start 2023-03-03T09:42:12.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.969+0000 7faa6b021700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faa640c2e30 0x7faa64158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:12.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.969+0000 7faa6b021700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6406c6c0 0x7faa640736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:12.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.969+0000 7faa6b021700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa64157ff0 con 0x7faa64151cd0 2023-03-03T09:42:12.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.969+0000 7faa6b021700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6406d640 con 0x7faa640c2e30 2023-03-03T09:42:12.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.969+0000 7faa6b021700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6406cda0 con 0x7faa6406c6c0 2023-03-03T09:42:12.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.969+0000 7faa68dbd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faa64151cd0 0x7faa64154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51170/0 (socket says 172.21.15.7:51170) 2023-03-03T09:42:12.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.969+0000 7faa68dbd700 1 -- 172.21.15.7:0/2605363186 learned_addr learned my addr 172.21.15.7:0/2605363186 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:42:12.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.969+0000 7faa695be700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6406c6c0 0x7faa640736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:12.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa63fff700 1 --2- 172.21.15.7:0/2605363186 >> v2:172.21.15.143:3300/0 conn(0x7faa640c2e30 0x7faa64158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:12.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804486443 0 0) 0x7faa64157ff0 con 0x7faa64151cd0 2023-03-03T09:42:12.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa540035b0 con 0x7faa64151cd0 2023-03-03T09:42:12.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3956805859 0 0) 0x7faa540035b0 con 0x7faa64151cd0 2023-03-03T09:42:12.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6406c6c0 msgr2=0x7faa640736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:42:12.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6406c6c0 0x7faa640736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:12.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 >> v2:172.21.15.143:3300/0 conn(0x7faa640c2e30 msgr2=0x7faa64158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:42:12.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 --2- 172.21.15.7:0/2605363186 >> v2:172.21.15.143:3300/0 conn(0x7faa640c2e30 0x7faa64158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:12.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6406d080 con 0x7faa64151cd0 2023-03-03T09:42:12.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7faa58001670 con 0x7faa64151cd0 2023-03-03T09:42:12.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 >> v1:172.21.15.7:6789/0 conn(0x7faa64151cd0 legacy=0x7faa64154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:12.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa54003a70 0x7faa54003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:12.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa54004850 0x7faa54004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:12.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6406c6c0 0x7faa54005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:12.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6406d640 con 0x7faa54004850 2023-03-03T09:42:12.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6406cda0 con 0x7faa54003a70 2023-03-03T09:42:12.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5400ba40 con 0x7faa6406c6c0 2023-03-03T09:42:12.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa63fff700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa54004850 0x7faa54004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:12.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa68dbd700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa54003a70 0x7faa54003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:12.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa695be700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6406c6c0 0x7faa54005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:12.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa63fff700 1 -- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6406c6c0 msgr2=0x7faa54005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:42:12.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa63fff700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6406c6c0 0x7faa54005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:12.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa63fff700 1 -- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa54003a70 msgr2=0x7faa54003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:42:12.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa63fff700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa54003a70 0x7faa54003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:12.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.970+0000 7faa63fff700 1 -- 172.21.15.7:0/2605363186 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa540035b0 con 0x7faa54004850 2023-03-03T09:42:12.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa63fff700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa54004850 0x7faa54004cc0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faa50009b60 tx=0x7faa54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45faaf2d021bf863 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:42:12.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5000e6c0 con 0x7faa54004850 2023-03-03T09:42:12.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5000e820 con 0x7faa54004850 2023-03-03T09:42:12.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa637fe700 1 -- 172.21.15.7:0/2605363186 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5000eb90 con 0x7faa54004850 2023-03-03T09:42:12.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa6b021700 1 -- 172.21.15.7:0/2605363186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa54004850 msgr2=0x7faa54004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:12.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa6b021700 1 --2- 172.21.15.7:0/2605363186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa54004850 0x7faa54004cc0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faa50009b60 tx=0x7faa54005880 comp rx=0 tx=0).stop 2023-03-03T09:42:12.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa695be700 1 -- 172.21.15.7:0/2605363186 reap_dead start 2023-03-03T09:42:12.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa6b021700 1 -- 172.21.15.7:0/2605363186 shutdown_connections 2023-03-03T09:42:12.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa6b021700 1 -- 172.21.15.7:0/2605363186 >> 172.21.15.7:0/2605363186 conn(0x7faa640a94d0 msgr2=0x7faa640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:42:12.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa6b021700 1 -- 172.21.15.7:0/2605363186 shutdown_connections 2023-03-03T09:42:12.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.971+0000 7faa6b021700 1 -- 172.21.15.7:0/2605363186 wait complete. 2023-03-03T09:42:12.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.973+0000 7faa6b021700 1 Processor -- start 2023-03-03T09:42:12.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.973+0000 7faa6b021700 1 -- start start 2023-03-03T09:42:12.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.973+0000 7faa6b021700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa54004850 0x7faa641e0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:12.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.973+0000 7faa6b021700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6414c270 0x7faa6414b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:12.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa6b021700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa6414bb10 0x7faa641534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:12.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa6b021700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa640c6a70 con 0x7faa6414bb10 2023-03-03T09:42:12.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa6b021700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa641ede70 con 0x7faa54004850 2023-03-03T09:42:12.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa6b021700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa640c7140 con 0x7faa6414c270 2023-03-03T09:42:12.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa695be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa6414bb10 0x7faa641534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:12.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa695be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa6414bb10 0x7faa641534e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57206/0 (socket says 172.21.15.7:57206) 2023-03-03T09:42:12.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa695be700 1 -- 172.21.15.7:0/174800098 learned_addr learned my addr 172.21.15.7:0/174800098 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:42:12.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa68dbd700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa54004850 0x7faa641e0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:12.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa63fff700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6414c270 0x7faa6414b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:12.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa695be700 1 -- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6414c270 msgr2=0x7faa6414b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:12.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa695be700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6414c270 0x7faa6414b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:12.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa695be700 1 -- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa54004850 msgr2=0x7faa641e0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:12.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa695be700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa54004850 0x7faa641e0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:12.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa695be700 1 -- 172.21.15.7:0/174800098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa64157c80 con 0x7faa6414bb10 2023-03-03T09:42:12.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa695be700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa6414bb10 0x7faa641534e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7faa6406e230 tx=0x7faa5c011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:42:12.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa617fa700 1 -- 172.21.15.7:0/174800098 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c00e040 con 0x7faa6414bb10 2023-03-03T09:42:12.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa617fa700 1 -- 172.21.15.7:0/174800098 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c00f040 con 0x7faa6414bb10 2023-03-03T09:42:12.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa617fa700 1 -- 172.21.15.7:0/174800098 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c00fd50 con 0x7faa6414bb10 2023-03-03T09:42:12.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa6b021700 1 -- 172.21.15.7:0/174800098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6406d360 con 0x7faa6414bb10 2023-03-03T09:42:12.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.974+0000 7faa6b021700 1 -- 172.21.15.7:0/174800098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa641589c0 con 0x7faa6414bb10 2023-03-03T09:42:12.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.975+0000 7faa6b021700 1 -- 172.21.15.7:0/174800098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa641541f0 con 0x7faa6414bb10 2023-03-03T09:42:12.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.976+0000 7faa617fa700 1 -- 172.21.15.7:0/174800098 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faa5c022460 con 0x7faa6414bb10 2023-03-03T09:42:12.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.976+0000 7faa617fa700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa480682f0 0x7faa4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:12.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.976+0000 7faa617fa700 1 -- 172.21.15.7:0/174800098 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faa5c086700 con 0x7faa6414bb10 2023-03-03T09:42:12.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.976+0000 7faa68dbd700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa480682f0 0x7faa4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:12.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.977+0000 7faa68dbd700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa480682f0 0x7faa4806a7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faa58000fb0 tx=0x7faa58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:42:12.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:12.980+0000 7faa617fa700 1 -- 172.21.15.7:0/174800098 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faa5c052a40 con 0x7faa6414bb10 2023-03-03T09:42:13.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.216+0000 7faa6b021700 1 -- 172.21.15.7:0/174800098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa6414ecc0 con 0x7faa6414bb10 2023-03-03T09:42:13.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.217+0000 7faa617fa700 1 -- 172.21.15.7:0/174800098 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faa5c056060 con 0x7faa6414bb10 2023-03-03T09:42:13.219 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:42:13.220 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:42:13.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 -- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa480682f0 msgr2=0x7faa4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:13.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa480682f0 0x7faa4806a7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7faa58000fb0 tx=0x7faa58009040 comp rx=0 tx=0).stop 2023-03-03T09:42:13.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 -- 172.21.15.7:0/174800098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa6414bb10 msgr2=0x7faa641534e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:13.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa6414bb10 0x7faa641534e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7faa6406e230 tx=0x7faa5c011f80 comp rx=0 tx=0).stop 2023-03-03T09:42:13.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 -- 172.21.15.7:0/174800098 shutdown_connections 2023-03-03T09:42:13.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa480682f0 0x7faa4806a7a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:13.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa6414bb10 0x7faa641534e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:13.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa54004850 0x7faa641e0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:13.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 --2- 172.21.15.7:0/174800098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa6414c270 0x7faa6414b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:13.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 -- 172.21.15.7:0/174800098 >> 172.21.15.7:0/174800098 conn(0x7faa640a94d0 msgr2=0x7faa640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:42:13.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 -- 172.21.15.7:0/174800098 shutdown_connections 2023-03-03T09:42:13.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:13.219+0000 7faa6b021700 1 -- 172.21.15.7:0/174800098 wait complete. 2023-03-03T09:42:13.223 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:42:13.233 INFO:tasks.mds_thrash.fs.[b]:waiting for 57 secs before thrashing 2023-03-03T09:42:15.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:42:15.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.833+0000 7efc81768700 1 Processor -- start 2023-03-03T09:42:15.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.833+0000 7efc81768700 1 -- start start 2023-03-03T09:42:15.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.834+0000 7efc81768700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efc7c151b20 0x7efc7c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:15.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.834+0000 7efc81768700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc7c0c2950 0x7efc7c157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:15.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.834+0000 7efc81768700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c06cb70 con 0x7efc7c073c30 2023-03-03T09:42:15.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.834+0000 7efc81768700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c06c2f0 con 0x7efc7c0c2950 2023-03-03T09:42:15.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.834+0000 7efc81768700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c06ce50 con 0x7efc7c151b20 2023-03-03T09:42:15.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.834+0000 7efc7a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc7c0c2950 0x7efc7c157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:15.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.834+0000 7efc7a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc7c0c2950 0x7efc7c157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57216/0 (socket says 172.21.15.7:57216) 2023-03-03T09:42:15.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.834+0000 7efc7a7fc700 1 -- 172.21.15.7:0/685739740 learned_addr learned my addr 172.21.15.7:0/685739740 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:42:15.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.835+0000 7efc7affd700 1 --2- 172.21.15.7:0/685739740 >> v2:172.21.15.143:3300/0 conn(0x7efc7c151b20 0x7efc7c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:15.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.835+0000 7efc7a7fc700 1 -- 172.21.15.7:0/685739740 >> v2:172.21.15.143:3300/0 conn(0x7efc7c151b20 msgr2=0x7efc7c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:42:15.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.835+0000 7efc7a7fc700 1 --2- 172.21.15.7:0/685739740 >> v2:172.21.15.143:3300/0 conn(0x7efc7c151b20 0x7efc7c0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:15.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.835+0000 7efc7a7fc700 1 -- 172.21.15.7:0/685739740 >> v1:172.21.15.143:6789/0 conn(0x7efc7c073c30 legacy=0x7efc7c158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:42:15.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.835+0000 7efc7a7fc700 1 -- 172.21.15.7:0/685739740 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c06d130 con 0x7efc7c0c2950 2023-03-03T09:42:15.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.835+0000 7efc7a7fc700 1 --2- 172.21.15.7:0/685739740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 0x7efc7c157ae0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efc7c06b9c0 tx=0x7efc6c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48ec9612256959e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:42:15.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.835+0000 7efc79ffb700 1 -- 172.21.15.7:0/685739740 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c004230 con 0x7efc7c0c2950 2023-03-03T09:42:15.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.835+0000 7efc79ffb700 1 -- 172.21.15.7:0/685739740 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c004390 con 0x7efc7c0c2950 2023-03-03T09:42:15.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.835+0000 7efc79ffb700 1 -- 172.21.15.7:0/685739740 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0036a0 con 0x7efc7c0c2950 2023-03-03T09:42:15.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.836+0000 7efc81768700 1 -- 172.21.15.7:0/685739740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 msgr2=0x7efc7c157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:15.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.836+0000 7efc81768700 1 --2- 172.21.15.7:0/685739740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 0x7efc7c157ae0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efc7c06b9c0 tx=0x7efc6c009bd0 comp rx=0 tx=0).stop 2023-03-03T09:42:15.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.836+0000 7efc81768700 1 -- 172.21.15.7:0/685739740 shutdown_connections 2023-03-03T09:42:15.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.836+0000 7efc81768700 1 --2- 172.21.15.7:0/685739740 >> v2:172.21.15.143:3300/0 conn(0x7efc7c151b20 0x7efc7c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:15.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.836+0000 7efc81768700 1 --2- 172.21.15.7:0/685739740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 0x7efc7c157ae0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:15.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.836+0000 7efc81768700 1 -- 172.21.15.7:0/685739740 >> 172.21.15.7:0/685739740 conn(0x7efc7c0a94d0 msgr2=0x7efc7c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:42:15.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.836+0000 7efc81768700 1 -- 172.21.15.7:0/685739740 shutdown_connections 2023-03-03T09:42:15.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.836+0000 7efc81768700 1 -- 172.21.15.7:0/685739740 wait complete. 2023-03-03T09:42:15.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.837+0000 7efc81768700 1 Processor -- start 2023-03-03T09:42:15.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.837+0000 7efc81768700 1 -- start start 2023-03-03T09:42:15.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc81768700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc7c073c30 0x7efc7c0c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:15.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc81768700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 0x7efc7c0c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:15.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc81768700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc7c151b20 0x7efc7c0c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:15.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc81768700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c1575c0 con 0x7efc7c0c2950 2023-03-03T09:42:15.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc81768700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c1560e0 con 0x7efc7c151b20 2023-03-03T09:42:15.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc81768700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c1566f0 con 0x7efc7c073c30 2023-03-03T09:42:15.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 0x7efc7c0c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:15.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 0x7efc7c0c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57222/0 (socket says 172.21.15.7:57222) 2023-03-03T09:42:15.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7a7fc700 1 -- 172.21.15.7:0/869296820 learned_addr learned my addr 172.21.15.7:0/869296820 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:42:15.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7affd700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc7c073c30 0x7efc7c0c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:15.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7b7fe700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc7c151b20 0x7efc7c0c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:15.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7a7fc700 1 -- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc7c073c30 msgr2=0x7efc7c0c3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:15.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7a7fc700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc7c073c30 0x7efc7c0c3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:15.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7a7fc700 1 -- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc7c151b20 msgr2=0x7efc7c0c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:15.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7a7fc700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc7c151b20 0x7efc7c0c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:15.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7a7fc700 1 -- 172.21.15.7:0/869296820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c009550 con 0x7efc7c0c2950 2023-03-03T09:42:15.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.838+0000 7efc7a7fc700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 0x7efc7c0c5440 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efc7c06bb30 tx=0x7efc6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:42:15.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.839+0000 7efc637fe700 1 -- 172.21.15.7:0/869296820 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c011630 con 0x7efc7c0c2950 2023-03-03T09:42:15.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.839+0000 7efc637fe700 1 -- 172.21.15.7:0/869296820 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c011790 con 0x7efc7c0c2950 2023-03-03T09:42:15.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.839+0000 7efc81768700 1 -- 172.21.15.7:0/869296820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c158730 con 0x7efc7c0c2950 2023-03-03T09:42:15.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.839+0000 7efc637fe700 1 -- 172.21.15.7:0/869296820 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c011b40 con 0x7efc7c0c2950 2023-03-03T09:42:15.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.839+0000 7efc81768700 1 -- 172.21.15.7:0/869296820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c06d410 con 0x7efc7c0c2950 2023-03-03T09:42:15.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.840+0000 7efc81768700 1 -- 172.21.15.7:0/869296820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c151ef0 con 0x7efc7c0c2950 2023-03-03T09:42:15.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.840+0000 7efc637fe700 1 -- 172.21.15.7:0/869296820 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efc6c017020 con 0x7efc7c0c2950 2023-03-03T09:42:15.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.841+0000 7efc637fe700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc680682f0 0x7efc6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:42:15.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.841+0000 7efc637fe700 1 -- 172.21.15.7:0/869296820 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efc6c0876a0 con 0x7efc7c0c2950 2023-03-03T09:42:15.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.841+0000 7efc7affd700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc680682f0 0x7efc6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:42:15.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.841+0000 7efc7affd700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc680682f0 0x7efc6806a7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efc7c155550 tx=0x7efc70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:42:15.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:15.845+0000 7efc637fe700 1 -- 172.21.15.7:0/869296820 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efc6c0539e0 con 0x7efc7c0c2950 2023-03-03T09:42:16.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.082+0000 7efc81768700 1 -- 172.21.15.7:0/869296820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc7c0c5160 con 0x7efc7c0c2950 2023-03-03T09:42:16.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.083+0000 7efc637fe700 1 -- 172.21.15.7:0/869296820 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efc7c0c5160 con 0x7efc7c0c2950 2023-03-03T09:42:16.085 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:42:16.085 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:42:16.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 -- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc680682f0 msgr2=0x7efc6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:16.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc680682f0 0x7efc6806a7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efc7c155550 tx=0x7efc70008040 comp rx=0 tx=0).stop 2023-03-03T09:42:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 -- 172.21.15.7:0/869296820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 msgr2=0x7efc7c0c5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:42:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 0x7efc7c0c5440 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efc7c06bb30 tx=0x7efc6c004750 comp rx=0 tx=0).stop 2023-03-03T09:42:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 -- 172.21.15.7:0/869296820 shutdown_connections 2023-03-03T09:42:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc7c151b20 0x7efc7c0c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc7c073c30 0x7efc7c0c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc680682f0 0x7efc6806a7a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 --2- 172.21.15.7:0/869296820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc7c0c2950 0x7efc7c0c5440 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:42:16.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 -- 172.21.15.7:0/869296820 >> 172.21.15.7:0/869296820 conn(0x7efc7c0a94d0 msgr2=0x7efc7c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:42:16.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 -- 172.21.15.7:0/869296820 shutdown_connections 2023-03-03T09:42:16.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:42:16.085+0000 7efc81768700 1 -- 172.21.15.7:0/869296820 wait complete. 2023-03-03T09:42:16.089 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:42:16.100 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T09:42:18.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:18.215 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:48.285 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:42:48.288 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:10.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:43:10.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.373+0000 7f6ce5883700 1 Processor -- start 2023-03-03T09:43:10.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.373+0000 7f6ce5883700 1 -- start start 2023-03-03T09:43:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.373+0000 7f6ce5883700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce00c00e0 0x7f6ce00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.373+0000 7f6ce5883700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6ce0154ff0 0x7f6ce00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:10.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.373+0000 7f6ce5883700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce00c7310 con 0x7f6ce00caa30 2023-03-03T09:43:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6ce5883700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce00bfd10 con 0x7f6ce0154ff0 2023-03-03T09:43:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6ce5883700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00bfa30 con 0x7f6ce00c00e0 2023-03-03T09:43:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cdeffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6ce00caa30 0x7f6ce00c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51268/0 (socket says 172.21.15.7:51268) 2023-03-03T09:43:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cdeffd700 1 -- 172.21.15.7:0/1074453853 learned_addr learned my addr 172.21.15.7:0/1074453853 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:43:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588410118 0 0) 0x7f6ce00c7310 con 0x7f6ce00caa30 2023-03-03T09:43:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cdf7fe700 1 --2- 172.21.15.7:0/1074453853 >> v2:172.21.15.143:3300/0 conn(0x7f6ce0154ff0 0x7f6ce00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc80035b0 con 0x7f6ce00caa30 2023-03-03T09:43:10.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cde7fc700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce00c00e0 0x7f6ce00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 231632819 0 0) 0x7f6cc80035b0 con 0x7f6ce00caa30 2023-03-03T09:43:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce00c00e0 msgr2=0x7f6ce00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:43:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce00c00e0 0x7f6ce00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 >> v2:172.21.15.143:3300/0 conn(0x7f6ce0154ff0 msgr2=0x7f6ce00cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:43:10.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 --2- 172.21.15.7:0/1074453853 >> v2:172.21.15.143:3300/0 conn(0x7f6ce0154ff0 0x7f6ce00cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0154510 con 0x7f6ce00caa30 2023-03-03T09:43:10.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f6cd4001670 con 0x7f6ce00caa30 2023-03-03T09:43:10.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 >> v1:172.21.15.7:6789/0 conn(0x7f6ce00caa30 legacy=0x7f6ce00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:10.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc8003a70 0x7f6cc8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:10.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce00c00e0 0x7f6cc8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:10.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc8004d90 0x7f6cc8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:10.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00bfd10 con 0x7f6cc8003a70 2023-03-03T09:43:10.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00bfa30 con 0x7f6cc8004d90 2023-03-03T09:43:10.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc800ba60 con 0x7f6ce00c00e0 2023-03-03T09:43:10.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.374+0000 7f6cdeffd700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc8003a70 0x7f6cc8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:10.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cde7fc700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce00c00e0 0x7f6cc8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:10.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cdf7fe700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc8004d90 0x7f6cc8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cdeffd700 1 -- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce00c00e0 msgr2=0x7f6cc8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:43:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cdeffd700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce00c00e0 0x7f6cc8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cdeffd700 1 -- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc8004d90 msgr2=0x7f6cc8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:43:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cdeffd700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc8004d90 0x7f6cc8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cdeffd700 1 -- 172.21.15.7:0/1074453853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc80035b0 con 0x7f6cc8003a70 2023-03-03T09:43:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cdf7fe700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc8004d90 0x7f6cc8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:43:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cdeffd700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc8003a70 0x7f6cc8003e80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400c3a0 tx=0x7f6cc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d21df2240f98702 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4007c60 con 0x7f6cc8003a70 2023-03-03T09:43:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4005960 con 0x7f6cc8003a70 2023-03-03T09:43:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6cddffb700 1 -- 172.21.15.7:0/1074453853 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd400cea0 con 0x7f6cc8003a70 2023-03-03T09:43:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6ce5883700 1 -- 172.21.15.7:0/1074453853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc8003a70 msgr2=0x7f6cc8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.375+0000 7f6ce5883700 1 --2- 172.21.15.7:0/1074453853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cc8003a70 0x7f6cc8003e80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400c3a0 tx=0x7f6cc8005880 comp rx=0 tx=0).stop 2023-03-03T09:43:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.376+0000 7f6cdf7fe700 1 -- 172.21.15.7:0/1074453853 reap_dead start 2023-03-03T09:43:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.376+0000 7f6ce5883700 1 -- 172.21.15.7:0/1074453853 shutdown_connections 2023-03-03T09:43:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.376+0000 7f6ce5883700 1 -- 172.21.15.7:0/1074453853 >> 172.21.15.7:0/1074453853 conn(0x7f6ce0066e40 msgr2=0x7f6ce0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:43:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.376+0000 7f6ce5883700 1 -- 172.21.15.7:0/1074453853 shutdown_connections 2023-03-03T09:43:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.376+0000 7f6ce5883700 1 -- 172.21.15.7:0/1074453853 wait complete. 2023-03-03T09:43:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.377+0000 7f6ce5883700 1 Processor -- start 2023-03-03T09:43:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.377+0000 7f6ce5883700 1 -- start start 2023-03-03T09:43:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6ce5883700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc8003a70 0x7f6ce014d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6ce5883700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce01edfe0 0x7f6ce00c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6ce5883700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ce00c2850 0x7f6ce01e9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6ce5883700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce00c2e10 con 0x7f6ce00c2850 2023-03-03T09:43:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6ce5883700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01ed500 con 0x7f6cc8003a70 2023-03-03T09:43:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6ce5883700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01ed7e0 con 0x7f6ce01edfe0 2023-03-03T09:43:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cdf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ce00c2850 0x7f6ce01e9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cdf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ce00c2850 0x7f6ce01e9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57304/0 (socket says 172.21.15.7:57304) 2023-03-03T09:43:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cdf7fe700 1 -- 172.21.15.7:0/1053389292 learned_addr learned my addr 172.21.15.7:0/1053389292 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:43:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cdeffd700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc8003a70 0x7f6ce014d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cde7fc700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce01edfe0 0x7f6ce00c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cdf7fe700 1 -- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce01edfe0 msgr2=0x7f6ce00c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cdf7fe700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce01edfe0 0x7f6ce00c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cdf7fe700 1 -- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc8003a70 msgr2=0x7f6ce014d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cdf7fe700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc8003a70 0x7f6ce014d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cdf7fe700 1 -- 172.21.15.7:0/1053389292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0154ad0 con 0x7f6ce00c2850 2023-03-03T09:43:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cdf7fe700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ce00c2850 0x7f6ce01e9e10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6ce01553c0 tx=0x7f6cd80104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cc7fff700 1 -- 172.21.15.7:0/1053389292 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd800e050 con 0x7f6ce00c2850 2023-03-03T09:43:10.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cc7fff700 1 -- 172.21.15.7:0/1053389292 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd800f040 con 0x7f6ce00c2850 2023-03-03T09:43:10.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6ce5883700 1 -- 172.21.15.7:0/1053389292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce01547f0 con 0x7f6ce00c2850 2023-03-03T09:43:10.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6cc7fff700 1 -- 172.21.15.7:0/1053389292 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8003c40 con 0x7f6ce00c2850 2023-03-03T09:43:10.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.378+0000 7f6ce5883700 1 -- 172.21.15.7:0/1053389292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce00bf650 con 0x7f6ce00c2850 2023-03-03T09:43:10.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.379+0000 7f6ce5883700 1 -- 172.21.15.7:0/1053389292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce00afa20 con 0x7f6ce00c2850 2023-03-03T09:43:10.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.380+0000 7f6cc7fff700 1 -- 172.21.15.7:0/1053389292 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6cd8010920 con 0x7f6ce00c2850 2023-03-03T09:43:10.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.380+0000 7f6cc7fff700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ccc0682f0 0x7f6ccc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:10.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.380+0000 7f6cc7fff700 1 -- 172.21.15.7:0/1053389292 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6cd8086650 con 0x7f6ce00c2850 2023-03-03T09:43:10.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.380+0000 7f6cdeffd700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ccc0682f0 0x7f6ccc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:10.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.381+0000 7f6cdeffd700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ccc0682f0 0x7f6ccc06a7a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4001050 tx=0x7f6cd400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:10.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.385+0000 7f6cc7fff700 1 -- 172.21.15.7:0/1053389292 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6cd80529f0 con 0x7f6ce00c2850 2023-03-03T09:43:10.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.625+0000 7f6ce5883700 1 -- 172.21.15.7:0/1053389292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ce0074bb0 con 0x7f6ce00c2850 2023-03-03T09:43:10.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.625+0000 7f6cc7fff700 1 -- 172.21.15.7:0/1053389292 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6cd8056010 con 0x7f6ce00c2850 2023-03-03T09:43:10.627 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:43:10.627 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:43:10.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 -- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ccc0682f0 msgr2=0x7f6ccc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:10.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ccc0682f0 0x7f6ccc06a7a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4001050 tx=0x7f6cd400a000 comp rx=0 tx=0).stop 2023-03-03T09:43:10.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 -- 172.21.15.7:0/1053389292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ce00c2850 msgr2=0x7f6ce01e9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:10.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ce00c2850 0x7f6ce01e9e10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6ce01553c0 tx=0x7f6cd80104a0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 -- 172.21.15.7:0/1053389292 shutdown_connections 2023-03-03T09:43:10.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ccc0682f0 0x7f6ccc06a7a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ce00c2850 0x7f6ce01e9e10 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6cc8003a70 0x7f6ce014d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 --2- 172.21.15.7:0/1053389292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ce01edfe0 0x7f6ce00c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:10.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 -- 172.21.15.7:0/1053389292 >> 172.21.15.7:0/1053389292 conn(0x7f6ce0066e40 msgr2=0x7f6ce0152910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:43:10.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 -- 172.21.15.7:0/1053389292 shutdown_connections 2023-03-03T09:43:10.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:10.627+0000 7f6ce5883700 1 -- 172.21.15.7:0/1053389292 wait complete. 2023-03-03T09:43:10.631 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:43:10.641 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T09:43:18.371 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:18.374 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:20.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:43:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.239+0000 7ff478605700 1 Processor -- start 2023-03-03T09:43:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.240+0000 7ff478605700 1 -- start start 2023-03-03T09:43:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.240+0000 7ff478605700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff470073960 0x7ff4701524e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.240+0000 7ff478605700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff47014ffa0 0x7ff470152a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.240+0000 7ff478605700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4700acb10 con 0x7ff470073d30 2023-03-03T09:43:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.240+0000 7ff478605700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4700c2b70 con 0x7ff470073960 2023-03-03T09:43:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.240+0000 7ff478605700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff470155bd0 con 0x7ff47014ffa0 2023-03-03T09:43:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff4763a1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff470073960 0x7ff4701524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff475ba0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff470073d30 0x7ff47014f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51304/0 (socket says 172.21.15.7:51304) 2023-03-03T09:43:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff475ba0700 1 -- 172.21.15.7:0/4215316036 learned_addr learned my addr 172.21.15.7:0/4215316036 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:43:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff476ba2700 1 --2- 172.21.15.7:0/4215316036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff47014ffa0 0x7ff470152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff4763a1700 1 -- 172.21.15.7:0/4215316036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff47014ffa0 msgr2=0x7ff470152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:43:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff4763a1700 1 --2- 172.21.15.7:0/4215316036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff47014ffa0 0x7ff470152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff4763a1700 1 -- 172.21.15.7:0/4215316036 >> v1:172.21.15.7:6789/0 conn(0x7ff470073d30 legacy=0x7ff47014f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff47539f700 1 -- 172.21.15.7:0/4215316036 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126096081 0 0) 0x7ff4700acb10 con 0x7ff470073d30 2023-03-03T09:43:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff4763a1700 1 -- 172.21.15.7:0/4215316036 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff470155eb0 con 0x7ff470073960 2023-03-03T09:43:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff4763a1700 1 --2- 172.21.15.7:0/4215316036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 0x7ff4701524e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009840 tx=0x7ff46c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9ce58b6ed41f32 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff47539f700 1 -- 172.21.15.7:0/4215316036 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c011830 con 0x7ff470073960 2023-03-03T09:43:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff47539f700 1 -- 172.21.15.7:0/4215316036 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46c011990 con 0x7ff470073960 2023-03-03T09:43:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.241+0000 7ff47539f700 1 -- 172.21.15.7:0/4215316036 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c011d20 con 0x7ff470073960 2023-03-03T09:43:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.242+0000 7ff478605700 1 -- 172.21.15.7:0/4215316036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 msgr2=0x7ff4701524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.242+0000 7ff478605700 1 --2- 172.21.15.7:0/4215316036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 0x7ff4701524e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009840 tx=0x7ff46c009b50 comp rx=0 tx=0).stop 2023-03-03T09:43:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.242+0000 7ff478605700 1 -- 172.21.15.7:0/4215316036 shutdown_connections 2023-03-03T09:43:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.242+0000 7ff478605700 1 --2- 172.21.15.7:0/4215316036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff47014ffa0 0x7ff470152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.242+0000 7ff478605700 1 --2- 172.21.15.7:0/4215316036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 0x7ff4701524e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.242+0000 7ff478605700 1 -- 172.21.15.7:0/4215316036 >> 172.21.15.7:0/4215316036 conn(0x7ff4700a94d0 msgr2=0x7ff4700a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:43:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.242+0000 7ff478605700 1 -- 172.21.15.7:0/4215316036 shutdown_connections 2023-03-03T09:43:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.242+0000 7ff478605700 1 -- 172.21.15.7:0/4215316036 wait complete. 2023-03-03T09:43:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.243+0000 7ff478605700 1 Processor -- start 2023-03-03T09:43:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.243+0000 7ff478605700 1 -- start start 2023-03-03T09:43:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff478605700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 0x7ff470070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff478605700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff470073d30 0x7ff4701457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff478605700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff47014ffa0 0x7ff470145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff478605700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4701e7f40 con 0x7ff470073960 2023-03-03T09:43:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff478605700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff470071560 con 0x7ff470073d30 2023-03-03T09:43:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff478605700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4701536a0 con 0x7ff47014ffa0 2023-03-03T09:43:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff4763a1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 0x7ff470070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff4763a1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 0x7ff470070a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57328/0 (socket says 172.21.15.7:57328) 2023-03-03T09:43:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff4763a1700 1 -- 172.21.15.7:0/2580395844 learned_addr learned my addr 172.21.15.7:0/2580395844 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:43:20.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff4763a1700 1 -- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff47014ffa0 msgr2=0x7ff470145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff4763a1700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff47014ffa0 0x7ff470145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff4763a1700 1 -- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff470073d30 msgr2=0x7ff4701457f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff4763a1700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff470073d30 0x7ff4701457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff4763a1700 1 -- 172.21.15.7:0/2580395844 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c0094a0 con 0x7ff470073960 2023-03-03T09:43:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff4763a1700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 0x7ff470070a80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0090b0 tx=0x7ff46c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff462ffd700 1 -- 172.21.15.7:0/2580395844 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c0045b0 con 0x7ff470073960 2023-03-03T09:43:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.244+0000 7ff462ffd700 1 -- 172.21.15.7:0/2580395844 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46c01f070 con 0x7ff470073960 2023-03-03T09:43:20.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.245+0000 7ff462ffd700 1 -- 172.21.15.7:0/2580395844 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c0038c0 con 0x7ff470073960 2023-03-03T09:43:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.245+0000 7ff478605700 1 -- 172.21.15.7:0/2580395844 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4700c1ce0 con 0x7ff470073960 2023-03-03T09:43:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.245+0000 7ff478605700 1 -- 172.21.15.7:0/2580395844 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff470156190 con 0x7ff470073960 2023-03-03T09:43:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.245+0000 7ff478605700 1 -- 172.21.15.7:0/2580395844 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4700c1e70 con 0x7ff470073960 2023-03-03T09:43:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.246+0000 7ff462ffd700 1 -- 172.21.15.7:0/2580395844 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff46c02a020 con 0x7ff470073960 2023-03-03T09:43:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.246+0000 7ff462ffd700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4540682f0 0x7ff45406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.246+0000 7ff462ffd700 1 -- 172.21.15.7:0/2580395844 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff46c087640 con 0x7ff470073960 2023-03-03T09:43:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.247+0000 7ff475ba0700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4540682f0 0x7ff45406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.247+0000 7ff475ba0700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4540682f0 0x7ff45406a7a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff4640011e0 tx=0x7ff4640079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.251+0000 7ff462ffd700 1 -- 172.21.15.7:0/2580395844 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff46c0539e0 con 0x7ff470073960 2023-03-03T09:43:20.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.486+0000 7ff478605700 1 -- 172.21.15.7:0/2580395844 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff47004e6e0 con 0x7ff470073960 2023-03-03T09:43:20.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.486+0000 7ff462ffd700 1 -- 172.21.15.7:0/2580395844 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff46c057000 con 0x7ff470073960 2023-03-03T09:43:20.489 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:43:20.489 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:43:20.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 -- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4540682f0 msgr2=0x7ff45406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:20.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4540682f0 0x7ff45406a7a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff4640011e0 tx=0x7ff4640079b0 comp rx=0 tx=0).stop 2023-03-03T09:43:20.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 -- 172.21.15.7:0/2580395844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 msgr2=0x7ff470070a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:20.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 0x7ff470070a80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0090b0 tx=0x7ff46c004750 comp rx=0 tx=0).stop 2023-03-03T09:43:20.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 -- 172.21.15.7:0/2580395844 shutdown_connections 2023-03-03T09:43:20.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff47014ffa0 0x7ff470145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:20.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4540682f0 0x7ff45406a7a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:20.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff470073960 0x7ff470070a80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:20.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 --2- 172.21.15.7:0/2580395844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff470073d30 0x7ff4701457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:20.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 -- 172.21.15.7:0/2580395844 >> 172.21.15.7:0/2580395844 conn(0x7ff4700a94d0 msgr2=0x7ff47014f660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:43:20.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 -- 172.21.15.7:0/2580395844 shutdown_connections 2023-03-03T09:43:20.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:20.488+0000 7ff478605700 1 -- 172.21.15.7:0/2580395844 wait complete. 2023-03-03T09:43:20.493 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:43:20.502 INFO:tasks.mds_thrash.fs.[a]:waiting for 38 secs before thrashing 2023-03-03T09:43:48.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:48.672 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:43:58.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:43:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.641+0000 7f4f23db4700 1 Processor -- start 2023-03-03T09:43:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.641+0000 7f4f23db4700 1 -- start start 2023-03-03T09:43:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f23db4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0aadb0 0x7f4f1c157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f23db4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4f1c0aa030 0x7f4f1c158390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f23db4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c1588d0 con 0x7f4f1c14e410 2023-03-03T09:43:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f23db4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c06fce0 con 0x7f4f1c0aa030 2023-03-03T09:43:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f23db4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c06f510 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f22351700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4f1c0aa030 0x7f4f1c158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f21b50700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4f1c14e410 0x7f4f1c0717c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51352/0 (socket says 172.21.15.7:51352) 2023-03-03T09:43:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f21b50700 1 -- 172.21.15.7:0/3942898404 learned_addr learned my addr 172.21.15.7:0/3942898404 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:43:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f2134f700 1 --2- 172.21.15.7:0/3942898404 >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0aadb0 0x7f4f1c157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f22351700 1 -- 172.21.15.7:0/3942898404 >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0aadb0 msgr2=0x7f4f1c157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:43:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f22351700 1 --2- 172.21.15.7:0/3942898404 >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0aadb0 0x7f4f1c157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f22351700 1 -- 172.21.15.7:0/3942898404 >> v1:172.21.15.7:6789/0 conn(0x7f4f1c14e410 legacy=0x7f4f1c0717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f20b4e700 1 -- 172.21.15.7:0/3942898404 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921665202 0 0) 0x7f4f1c1588d0 con 0x7f4f1c14e410 2023-03-03T09:43:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f22351700 1 -- 172.21.15.7:0/3942898404 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c154f00 con 0x7f4f1c0aa030 2023-03-03T09:43:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.642+0000 7f4f2134f700 1 --2- 172.21.15.7:0/3942898404 >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0aadb0 0x7f4f1c157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:43:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f22351700 1 --2- 172.21.15.7:0/3942898404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aa030 0x7f4f1c158390 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0adb30 tx=0x7f4f1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba01e772be0b581 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f20b4e700 1 -- 172.21.15.7:0/3942898404 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1400e070 con 0x7f4f1c0aa030 2023-03-03T09:43:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f20b4e700 1 -- 172.21.15.7:0/3942898404 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1400bbd0 con 0x7f4f1c0aa030 2023-03-03T09:43:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f20b4e700 1 -- 172.21.15.7:0/3942898404 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14012430 con 0x7f4f1c0aa030 2023-03-03T09:43:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f23db4700 1 -- 172.21.15.7:0/3942898404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aa030 msgr2=0x7f4f1c158390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f23db4700 1 --2- 172.21.15.7:0/3942898404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aa030 0x7f4f1c158390 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0adb30 tx=0x7f4f1400b330 comp rx=0 tx=0).stop 2023-03-03T09:43:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f23db4700 1 -- 172.21.15.7:0/3942898404 shutdown_connections 2023-03-03T09:43:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f23db4700 1 --2- 172.21.15.7:0/3942898404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aa030 0x7f4f1c158390 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f23db4700 1 --2- 172.21.15.7:0/3942898404 >> v2:172.21.15.143:3300/0 conn(0x7f4f1c0aadb0 0x7f4f1c157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f23db4700 1 -- 172.21.15.7:0/3942898404 >> 172.21.15.7:0/3942898404 conn(0x7f4f1c0c8900 msgr2=0x7f4f1c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:43:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f23db4700 1 -- 172.21.15.7:0/3942898404 shutdown_connections 2023-03-03T09:43:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.643+0000 7f4f23db4700 1 -- 172.21.15.7:0/3942898404 wait complete. 2023-03-03T09:43:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.645+0000 7f4f23db4700 1 Processor -- start 2023-03-03T09:43:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.645+0000 7f4f23db4700 1 -- start start 2023-03-03T09:43:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f23db4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0aa030 0x7f4f1c1ea1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f23db4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aadb0 0x7f4f1c0c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f23db4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c14e410 0x7f4f1c150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f23db4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0c36b0 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f23db4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c1ea700 con 0x7f4f1c14e410 2023-03-03T09:43:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f23db4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c151830 con 0x7f4f1c0aa030 2023-03-03T09:43:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f2134f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aadb0 0x7f4f1c0c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f2134f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aadb0 0x7f4f1c0c75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57380/0 (socket says 172.21.15.7:57380) 2023-03-03T09:43:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f2134f700 1 -- 172.21.15.7:0/3441605517 learned_addr learned my addr 172.21.15.7:0/3441605517 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:43:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f21b50700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0aa030 0x7f4f1c1ea1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f22351700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c14e410 0x7f4f1c150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f2134f700 1 -- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0aa030 msgr2=0x7f4f1c1ea1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f2134f700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0aa030 0x7f4f1c1ea1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f2134f700 1 -- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c14e410 msgr2=0x7f4f1c150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f2134f700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c14e410 0x7f4f1c150a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f2134f700 1 -- 172.21.15.7:0/3441605517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f14009a70 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f21b50700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0aa030 0x7f4f1c1ea1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:43:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.646+0000 7f4f22351700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c14e410 0x7f4f1c150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:43:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.647+0000 7f4f2134f700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aadb0 0x7f4f1c0c75d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00c780 tx=0x7f4f0c00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.647+0000 7f4f127fc700 1 -- 172.21.15.7:0/3441605517 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c011800 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.647+0000 7f4f127fc700 1 -- 172.21.15.7:0/3441605517 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c011960 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.647+0000 7f4f23db4700 1 -- 172.21.15.7:0/3441605517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c06f720 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.647+0000 7f4f127fc700 1 -- 172.21.15.7:0/3441605517 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c011d10 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.647+0000 7f4f23db4700 1 -- 172.21.15.7:0/3441605517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c1551e0 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.648+0000 7f4f23db4700 1 -- 172.21.15.7:0/3441605517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c06fa00 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.648+0000 7f4f127fc700 1 -- 172.21.15.7:0/3441605517 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4f0c017030 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.649+0000 7f4f127fc700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f00068320 0x7f4f0006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:43:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.649+0000 7f4f127fc700 1 -- 172.21.15.7:0/3441605517 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4f0c085d20 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.649+0000 7f4f21b50700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f00068320 0x7f4f0006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:43:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.650+0000 7f4f21b50700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f00068320 0x7f4f0006a7d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4f18000f60 tx=0x7f4f18009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:43:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.653+0000 7f4f127fc700 1 -- 172.21.15.7:0/3441605517 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4f0c052040 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.901+0000 7f4f23db4700 1 -- 172.21.15.7:0/3441605517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f1c074bb0 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.901+0000 7f4f127fc700 1 -- 172.21.15.7:0/3441605517 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4f0c055660 con 0x7f4f1c0aadb0 2023-03-03T09:43:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:43:58.903 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:43:58.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 -- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f00068320 msgr2=0x7f4f0006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:58.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f00068320 0x7f4f0006a7d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4f18000f60 tx=0x7f4f18009040 comp rx=0 tx=0).stop 2023-03-03T09:43:58.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 -- 172.21.15.7:0/3441605517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aadb0 msgr2=0x7f4f1c0c75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:43:58.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aadb0 0x7f4f1c0c75d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00c780 tx=0x7f4f0c00cb40 comp rx=0 tx=0).stop 2023-03-03T09:43:58.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 -- 172.21.15.7:0/3441605517 shutdown_connections 2023-03-03T09:43:58.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f1c14e410 0x7f4f1c150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:58.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f1c0aa030 0x7f4f1c1ea1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:58.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f00068320 0x7f4f0006a7d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:58.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 --2- 172.21.15.7:0/3441605517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f1c0aadb0 0x7f4f1c0c75d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:43:58.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 -- 172.21.15.7:0/3441605517 >> 172.21.15.7:0/3441605517 conn(0x7f4f1c0c8900 msgr2=0x7f4f1c157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:43:58.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 -- 172.21.15.7:0/3441605517 shutdown_connections 2023-03-03T09:43:58.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:43:58.904+0000 7f4f23db4700 1 -- 172.21.15.7:0/3441605517 wait complete. 2023-03-03T09:43:58.907 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:43:58.917 INFO:tasks.mds_thrash.fs.[a]:waiting for 58 secs before thrashing 2023-03-03T09:44:12.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:44:12.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.779+0000 7f9745cd2700 1 Processor -- start 2023-03-03T09:44:12.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.779+0000 7f9745cd2700 1 -- start start 2023-03-03T09:44:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f9745cd2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f97400aad20 0x7f9740157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f9745cd2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97400aa030 0x7f9740158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f9745cd2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97400a9c20 con 0x7f974014b0d0 2023-03-03T09:44:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f9745cd2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9740154f10 con 0x7f97400aad20 2023-03-03T09:44:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f9745cd2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97400ac3c0 con 0x7f97400aa030 2023-03-03T09:44:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973f7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f974014b0d0 0x7f9740071800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:54898/0 (socket says 172.21.15.7:54898) 2023-03-03T09:44:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973f7fe700 1 -- 172.21.15.7:0/266770172 learned_addr learned my addr 172.21.15.7:0/266770172 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:44:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973ffff700 1 --2- 172.21.15.7:0/266770172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97400aa030 0x7f9740158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973effd700 1 --2- 172.21.15.7:0/266770172 >> v2:172.21.15.143:3300/0 conn(0x7f97400aad20 0x7f9740157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973e7fc700 1 -- 172.21.15.7:0/266770172 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139155282 0 0) 0x7f97400a9c20 con 0x7f974014b0d0 2023-03-03T09:44:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973e7fc700 1 -- 172.21.15.7:0/266770172 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97280035b0 con 0x7f974014b0d0 2023-03-03T09:44:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973ffff700 1 -- 172.21.15.7:0/266770172 >> v2:172.21.15.143:3300/0 conn(0x7f97400aad20 msgr2=0x7f9740157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:44:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973ffff700 1 --2- 172.21.15.7:0/266770172 >> v2:172.21.15.143:3300/0 conn(0x7f97400aad20 0x7f9740157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973ffff700 1 -- 172.21.15.7:0/266770172 >> v1:172.21.15.143:6789/0 conn(0x7f974014b0d0 legacy=0x7f9740071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973ffff700 1 -- 172.21.15.7:0/266770172 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974006fc70 con 0x7f97400aa030 2023-03-03T09:44:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.780+0000 7f973effd700 1 --2- 172.21.15.7:0/266770172 >> v2:172.21.15.143:3300/0 conn(0x7f97400aad20 0x7f9740157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:44:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f973ffff700 1 --2- 172.21.15.7:0/266770172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97400aa030 0x7f9740158330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9738009d90 tx=0x7f973800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a30fca52374220c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:44:12.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f973e7fc700 1 -- 172.21.15.7:0/266770172 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973800e070 con 0x7f97400aa030 2023-03-03T09:44:12.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f973e7fc700 1 -- 172.21.15.7:0/266770172 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973800bb20 con 0x7f97400aa030 2023-03-03T09:44:12.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f973e7fc700 1 -- 172.21.15.7:0/266770172 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738012430 con 0x7f97400aa030 2023-03-03T09:44:12.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f9745cd2700 1 -- 172.21.15.7:0/266770172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97400aa030 msgr2=0x7f9740158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:12.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f9745cd2700 1 --2- 172.21.15.7:0/266770172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97400aa030 0x7f9740158330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9738009d90 tx=0x7f973800b330 comp rx=0 tx=0).stop 2023-03-03T09:44:12.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f9745cd2700 1 -- 172.21.15.7:0/266770172 shutdown_connections 2023-03-03T09:44:12.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f9745cd2700 1 --2- 172.21.15.7:0/266770172 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97400aa030 0x7f9740158330 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:12.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f9745cd2700 1 --2- 172.21.15.7:0/266770172 >> v2:172.21.15.143:3300/0 conn(0x7f97400aad20 0x7f9740157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:12.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f9745cd2700 1 -- 172.21.15.7:0/266770172 >> 172.21.15.7:0/266770172 conn(0x7f97400c8900 msgr2=0x7f97400c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:44:12.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f9745cd2700 1 -- 172.21.15.7:0/266770172 shutdown_connections 2023-03-03T09:44:12.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.781+0000 7f9745cd2700 1 -- 172.21.15.7:0/266770172 wait complete. 2023-03-03T09:44:12.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.783+0000 7f9745cd2700 1 Processor -- start 2023-03-03T09:44:12.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.783+0000 7f9745cd2700 1 -- start start 2023-03-03T09:44:12.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.783+0000 7f9745cd2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97400aa030 0x7f9740153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:12.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.783+0000 7f9745cd2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97400aad20 0x7f97401ed8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:12.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.783+0000 7f9745cd2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f974014b0d0 0x7f97401ea240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:12.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.783+0000 7f9745cd2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97401ea780 con 0x7f97400aad20 2023-03-03T09:44:12.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.783+0000 7f9745cd2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9740153b20 con 0x7f97400aa030 2023-03-03T09:44:12.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.783+0000 7f9745cd2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97401ecdd0 con 0x7f974014b0d0 2023-03-03T09:44:12.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97400aad20 0x7f97401ed8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:12.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97400aad20 0x7f97401ed8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57406/0 (socket says 172.21.15.7:57406) 2023-03-03T09:44:12.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973effd700 1 -- 172.21.15.7:0/451038599 learned_addr learned my addr 172.21.15.7:0/451038599 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:44:12.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973ffff700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f974014b0d0 0x7f97401ea240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:12.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973f7fe700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97400aa030 0x7f9740153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:12.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973effd700 1 -- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f974014b0d0 msgr2=0x7f97401ea240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:12.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973effd700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f974014b0d0 0x7f97401ea240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973effd700 1 -- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97400aa030 msgr2=0x7f9740153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973effd700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97400aa030 0x7f9740153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973effd700 1 -- 172.21.15.7:0/451038599 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97340094a0 con 0x7f97400aad20 2023-03-03T09:44:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f973effd700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97400aad20 0x7f97401ed8b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f973400ebb0 tx=0x7f973400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:44:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f9727fff700 1 -- 172.21.15.7:0/451038599 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973400c910 con 0x7f97400aad20 2023-03-03T09:44:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f9727fff700 1 -- 172.21.15.7:0/451038599 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973400ca70 con 0x7f97400aad20 2023-03-03T09:44:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f9745cd2700 1 -- 172.21.15.7:0/451038599 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9738009a70 con 0x7f97400aad20 2023-03-03T09:44:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f9727fff700 1 -- 172.21.15.7:0/451038599 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973400ce00 con 0x7f97400aad20 2023-03-03T09:44:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.784+0000 7f9745cd2700 1 -- 172.21.15.7:0/451038599 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974006f6b0 con 0x7f97400aad20 2023-03-03T09:44:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.785+0000 7f9745cd2700 1 -- 172.21.15.7:0/451038599 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9740155160 con 0x7f97400aad20 2023-03-03T09:44:12.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.786+0000 7f9727fff700 1 -- 172.21.15.7:0/451038599 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9734019030 con 0x7f97400aad20 2023-03-03T09:44:12.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.786+0000 7f9727fff700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f972c0682f0 0x7f972c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:12.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.786+0000 7f9727fff700 1 -- 172.21.15.7:0/451038599 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9734086200 con 0x7f97400aad20 2023-03-03T09:44:12.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.787+0000 7f973f7fe700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f972c0682f0 0x7f972c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:12.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.787+0000 7f973f7fe700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f972c0682f0 0x7f972c06a7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f97300011e0 tx=0x7f9730009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:44:12.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:12.791+0000 7f9727fff700 1 -- 172.21.15.7:0/451038599 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f97340525a0 con 0x7f97400aad20 2023-03-03T09:44:13.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.029+0000 7f9745cd2700 1 -- 172.21.15.7:0/451038599 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f974006f990 con 0x7f97400aad20 2023-03-03T09:44:13.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.029+0000 7f9727fff700 1 -- 172.21.15.7:0/451038599 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9734055bc0 con 0x7f97400aad20 2023-03-03T09:44:13.031 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:44:13.031 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:44:13.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.031+0000 7f9745cd2700 1 -- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f972c0682f0 msgr2=0x7f972c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:13.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.031+0000 7f9745cd2700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f972c0682f0 0x7f972c06a7a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f97300011e0 tx=0x7f9730009040 comp rx=0 tx=0).stop 2023-03-03T09:44:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.032+0000 7f9745cd2700 1 -- 172.21.15.7:0/451038599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97400aad20 msgr2=0x7f97401ed8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.032+0000 7f9745cd2700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97400aad20 0x7f97401ed8b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f973400ebb0 tx=0x7f973400ef70 comp rx=0 tx=0).stop 2023-03-03T09:44:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.032+0000 7f9745cd2700 1 -- 172.21.15.7:0/451038599 shutdown_connections 2023-03-03T09:44:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.032+0000 7f9745cd2700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f974014b0d0 0x7f97401ea240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.032+0000 7f9745cd2700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97400aa030 0x7f9740153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.032+0000 7f9745cd2700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f972c0682f0 0x7f972c06a7a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.032+0000 7f9745cd2700 1 --2- 172.21.15.7:0/451038599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97400aad20 0x7f97401ed8b0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.032+0000 7f9745cd2700 1 -- 172.21.15.7:0/451038599 >> 172.21.15.7:0/451038599 conn(0x7f97400c8900 msgr2=0x7f97400ace30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:44:13.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.032+0000 7f9745cd2700 1 -- 172.21.15.7:0/451038599 shutdown_connections 2023-03-03T09:44:13.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:13.032+0000 7f9745cd2700 1 -- 172.21.15.7:0/451038599 wait complete. 2023-03-03T09:44:13.035 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:44:13.045 INFO:tasks.mds_thrash.fs.[b]:waiting for 117 secs before thrashing 2023-03-03T09:44:18.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:18.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:48.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:48.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:44:56.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:44:57.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.055+0000 7f4ebf2df700 1 Processor -- start 2023-03-03T09:44:57.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.055+0000 7f4ebf2df700 1 -- start start 2023-03-03T09:44:57.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.056+0000 7f4ebf2df700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb806e020 0x7f4eb8071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:57.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.056+0000 7f4ebf2df700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb8154910 con 0x7f4eb8155390 2023-03-03T09:44:57.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.056+0000 7f4ebf2df700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb80c4540 con 0x7f4eb8072520 2023-03-03T09:44:57.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.056+0000 7f4ebf2df700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8070b90 con 0x7f4eb806e020 2023-03-03T09:44:57.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.056+0000 7f4ebd07b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4eb8155390 0x7f4eb80c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51428/0 (socket says 172.21.15.7:51428) 2023-03-03T09:44:57.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.056+0000 7f4ebd07b700 1 -- 172.21.15.7:0/2039961114 learned_addr learned my addr 172.21.15.7:0/2039961114 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:44:57.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.056+0000 7f4ebc87a700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb806e020 0x7f4eb8071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:57.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.056+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811272997 0 0) 0x7f4eb8154910 con 0x7f4eb8155390 2023-03-03T09:44:57.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.056+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea40035b0 con 0x7f4eb8155390 2023-03-03T09:44:57.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4197065643 0 0) 0x7f4ea40035b0 con 0x7f4eb8155390 2023-03-03T09:44:57.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb806e020 msgr2=0x7f4eb8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:44:57.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb806e020 0x7f4eb8071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:57.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 >> v1:172.21.15.143:6789/0 conn(0x7f4eb8072520 legacy=0x7f4eb8151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:57.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb80702a0 con 0x7f4eb8155390 2023-03-03T09:44:57.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f4ea8001670 con 0x7f4eb8155390 2023-03-03T09:44:57.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 >> v1:172.21.15.7:6789/0 conn(0x7f4eb8155390 legacy=0x7f4eb80c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:57.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ea4003aa0 0x7f4ea4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:57.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ea4004850 0x7f4ea4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:57.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb806e020 0x7f4ea4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:57.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8070b90 con 0x7f4ea4004850 2023-03-03T09:44:57.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea400baf0 con 0x7f4ea4003aa0 2023-03-03T09:44:57.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea400bcb0 con 0x7f4eb806e020 2023-03-03T09:44:57.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4ebc87a700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ea4004850 0x7f4ea4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:57.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4ebd07b700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ea4003aa0 0x7f4ea4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:57.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4ebd87c700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb806e020 0x7f4ea4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:57.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4ebc87a700 1 -- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb806e020 msgr2=0x7f4ea4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:44:57.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4ebc87a700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eb806e020 0x7f4ea4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:57.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4ebc87a700 1 -- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ea4003aa0 msgr2=0x7f4ea4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:44:57.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4ebc87a700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ea4003aa0 0x7f4ea4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:57.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4ebc87a700 1 -- 172.21.15.7:0/2039961114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea40035b0 con 0x7f4ea4004850 2023-03-03T09:44:57.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.057+0000 7f4ebc87a700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ea4004850 0x7f4ea4004cc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40096f0 tx=0x7f4ea4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ab212ced735ad9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:44:57.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.058+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb400e6c0 con 0x7f4ea4004850 2023-03-03T09:44:57.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.058+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb400e820 con 0x7f4ea4004850 2023-03-03T09:44:57.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.058+0000 7f4eaffff700 1 -- 172.21.15.7:0/2039961114 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb400ebb0 con 0x7f4ea4004850 2023-03-03T09:44:57.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.058+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2039961114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ea4004850 msgr2=0x7f4ea4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:57.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.058+0000 7f4ebf2df700 1 --2- 172.21.15.7:0/2039961114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ea4004850 0x7f4ea4004cc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40096f0 tx=0x7f4ea4005880 comp rx=0 tx=0).stop 2023-03-03T09:44:57.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.058+0000 7f4ebd87c700 1 -- 172.21.15.7:0/2039961114 reap_dead start 2023-03-03T09:44:57.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.058+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2039961114 shutdown_connections 2023-03-03T09:44:57.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.058+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2039961114 >> 172.21.15.7:0/2039961114 conn(0x7f4eb806c110 msgr2=0x7f4eb806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:44:57.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.058+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2039961114 shutdown_connections 2023-03-03T09:44:57.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.058+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2039961114 wait complete. 2023-03-03T09:44:57.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.059+0000 7f4ebf2df700 1 Processor -- start 2023-03-03T09:44:57.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.059+0000 7f4ebf2df700 1 -- start start 2023-03-03T09:44:57.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebf2df700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ea4004850 0x7f4eb814a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:57.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebf2df700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb814e110 0x7f4eb8146790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:57.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebf2df700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eb8147460 0x7f4eb8146cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:57.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebf2df700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb814b9e0 con 0x7f4eb8147460 2023-03-03T09:44:57.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebf2df700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb81ee250 con 0x7f4eb814e110 2023-03-03T09:44:57.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebf2df700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8147210 con 0x7f4ea4004850 2023-03-03T09:44:57.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebd87c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eb8147460 0x7f4eb8146cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:57.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebd87c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eb8147460 0x7f4eb8146cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57464/0 (socket says 172.21.15.7:57464) 2023-03-03T09:44:57.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebd87c700 1 -- 172.21.15.7:0/2643196753 learned_addr learned my addr 172.21.15.7:0/2643196753 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:44:57.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebd07b700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ea4004850 0x7f4eb814a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:57.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.060+0000 7f4ebc87a700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb814e110 0x7f4eb8146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:57.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4ebd87c700 1 -- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ea4004850 msgr2=0x7f4eb814a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:57.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4ebd87c700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ea4004850 0x7f4eb814a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:57.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4ebd87c700 1 -- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb814e110 msgr2=0x7f4eb8146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:57.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4ebd87c700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb814e110 0x7f4eb8146790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:57.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4ebd87c700 1 -- 172.21.15.7:0/2643196753 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb8070820 con 0x7f4eb8147460 2023-03-03T09:44:57.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4ebd87c700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eb8147460 0x7f4eb8146cd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8071d60 tx=0x7f4eb000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:44:57.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4eadffb700 1 -- 172.21.15.7:0/2643196753 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb000dc70 con 0x7f4eb8147460 2023-03-03T09:44:57.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4eadffb700 1 -- 172.21.15.7:0/2643196753 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb000ddf0 con 0x7f4eb8147460 2023-03-03T09:44:57.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4eadffb700 1 -- 172.21.15.7:0/2643196753 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0003c40 con 0x7f4eb8147460 2023-03-03T09:44:57.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2643196753 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb8070540 con 0x7f4eb8147460 2023-03-03T09:44:57.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.061+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2643196753 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb8155760 con 0x7f4eb8147460 2023-03-03T09:44:57.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.062+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2643196753 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb8074bb0 con 0x7f4eb8147460 2023-03-03T09:44:57.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.063+0000 7f4eadffb700 1 -- 172.21.15.7:0/2643196753 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4eb00050d0 con 0x7f4eb8147460 2023-03-03T09:44:57.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.063+0000 7f4eadffb700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e9c0682f0 0x7f4e9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:44:57.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.063+0000 7f4eadffb700 1 -- 172.21.15.7:0/2643196753 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4eb00864d0 con 0x7f4eb8147460 2023-03-03T09:44:57.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.063+0000 7f4ebd07b700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e9c0682f0 0x7f4e9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:44:57.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.064+0000 7f4ebd07b700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e9c0682f0 0x7f4e9c06a7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8001200 tx=0x7f4ea8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:44:57.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.067+0000 7f4eadffb700 1 -- 172.21.15.7:0/2643196753 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4eb0052870 con 0x7f4eb8147460 2023-03-03T09:44:57.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.306+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2643196753 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4eb80c4540 con 0x7f4eb8147460 2023-03-03T09:44:57.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.306+0000 7f4eadffb700 1 -- 172.21.15.7:0/2643196753 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4eb80c4540 con 0x7f4eb8147460 2023-03-03T09:44:57.308 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:44:57.308 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:44:57.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.308+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e9c0682f0 msgr2=0x7f4e9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:57.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.308+0000 7f4ebf2df700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e9c0682f0 0x7f4e9c06a7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8001200 tx=0x7f4ea8009040 comp rx=0 tx=0).stop 2023-03-03T09:44:57.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.308+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2643196753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eb8147460 msgr2=0x7f4eb8146cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:44:57.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.308+0000 7f4ebf2df700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eb8147460 0x7f4eb8146cd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8071d60 tx=0x7f4eb000d710 comp rx=0 tx=0).stop 2023-03-03T09:44:57.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.309+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2643196753 shutdown_connections 2023-03-03T09:44:57.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.309+0000 7f4ebf2df700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e9c0682f0 0x7f4e9c06a7a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:57.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.309+0000 7f4ebf2df700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eb8147460 0x7f4eb8146cd0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:57.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.309+0000 7f4ebf2df700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ea4004850 0x7f4eb814a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:57.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.309+0000 7f4ebf2df700 1 --2- 172.21.15.7:0/2643196753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eb814e110 0x7f4eb8146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:44:57.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.309+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2643196753 >> 172.21.15.7:0/2643196753 conn(0x7f4eb806c110 msgr2=0x7f4eb8073ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:44:57.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.309+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2643196753 shutdown_connections 2023-03-03T09:44:57.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:44:57.309+0000 7f4ebf2df700 1 -- 172.21.15.7:0/2643196753 wait complete. 2023-03-03T09:44:57.312 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:44:57.322 INFO:tasks.mds_thrash.fs.[a]:waiting for 77 secs before thrashing 2023-03-03T09:45:10.555 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2023-03-03T09:45:10.557 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2023-03-03T09:45:19.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:19.145 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:31.765 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2023-03-03T09:45:31.766 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_c7c3cc84dcd2bd1fdc61d0abde9057ccda2484b6/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2023-03-03T09:45:49.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:45:49.207 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:10.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:46:10.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.186+0000 7f9a264ea700 1 Processor -- start 2023-03-03T09:46:10.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.186+0000 7f9a264ea700 1 -- start start 2023-03-03T09:46:10.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.186+0000 7f9a264ea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a2014f9d0 0x7f9a20156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:10.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.186+0000 7f9a264ea700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a200c2f30 con 0x7f9a200abe20 2023-03-03T09:46:10.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.186+0000 7f9a264ea700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a200c31d0 con 0x7f9a2014fda0 2023-03-03T09:46:10.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.186+0000 7f9a264ea700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a201524a0 con 0x7f9a2014f9d0 2023-03-03T09:46:10.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a1ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a2014f9d0 0x7f9a20156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:10.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a24a87700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9a200abe20 0x7f9a201529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51520/0 (socket says 172.21.15.7:51520) 2023-03-03T09:46:10.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a1ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a2014f9d0 0x7f9a20156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57538/0 (socket says 172.21.15.7:57538) 2023-03-03T09:46:10.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a24a87700 1 -- 172.21.15.7:0/2025594224 learned_addr learned my addr 172.21.15.7:0/2025594224 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:46:10.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a1ffff700 1 -- 172.21.15.7:0/2025594224 >> v1:172.21.15.143:6789/0 conn(0x7f9a2014fda0 legacy=0x7f9a20071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:46:10.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a1ffff700 1 -- 172.21.15.7:0/2025594224 >> v1:172.21.15.7:6789/0 conn(0x7f9a200abe20 legacy=0x7f9a201529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:10.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a1ffff700 1 -- 172.21.15.7:0/2025594224 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a200c34b0 con 0x7f9a2014f9d0 2023-03-03T09:46:10.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a1ffff700 1 --2- 172.21.15.7:0/2025594224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 0x7f9a20156430 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009840 tx=0x7f9a10009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f54cc46730b81ab server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:10.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a1effd700 1 -- 172.21.15.7:0/2025594224 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10004230 con 0x7f9a2014f9d0 2023-03-03T09:46:10.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a1effd700 1 -- 172.21.15.7:0/2025594224 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10004390 con 0x7f9a2014f9d0 2023-03-03T09:46:10.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.187+0000 7f9a1effd700 1 -- 172.21.15.7:0/2025594224 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a100036a0 con 0x7f9a2014f9d0 2023-03-03T09:46:10.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.188+0000 7f9a264ea700 1 -- 172.21.15.7:0/2025594224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 msgr2=0x7f9a20156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:10.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.188+0000 7f9a264ea700 1 --2- 172.21.15.7:0/2025594224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 0x7f9a20156430 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009840 tx=0x7f9a10009b50 comp rx=0 tx=0).stop 2023-03-03T09:46:10.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.188+0000 7f9a264ea700 1 -- 172.21.15.7:0/2025594224 shutdown_connections 2023-03-03T09:46:10.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.188+0000 7f9a264ea700 1 --2- 172.21.15.7:0/2025594224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 0x7f9a20156430 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:10.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.188+0000 7f9a264ea700 1 -- 172.21.15.7:0/2025594224 >> 172.21.15.7:0/2025594224 conn(0x7f9a200c8900 msgr2=0x7f9a200c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:46:10.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.188+0000 7f9a264ea700 1 -- 172.21.15.7:0/2025594224 shutdown_connections 2023-03-03T09:46:10.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.188+0000 7f9a264ea700 1 -- 172.21.15.7:0/2025594224 wait complete. 2023-03-03T09:46:10.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.189+0000 7f9a264ea700 1 Processor -- start 2023-03-03T09:46:10.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.189+0000 7f9a264ea700 1 -- start start 2023-03-03T09:46:10.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a264ea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a200abe20 0x7f9a201492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:10.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a264ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 0x7f9a201dbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:10.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a264ea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a2014fda0 0x7f9a200c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:10.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a264ea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a20149b90 con 0x7f9a2014f9d0 2023-03-03T09:46:10.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a264ea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a20149e70 con 0x7f9a200abe20 2023-03-03T09:46:10.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a264ea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a201edef0 con 0x7f9a2014fda0 2023-03-03T09:46:10.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a1f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 0x7f9a201dbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:10.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a1f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 0x7f9a201dbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57544/0 (socket says 172.21.15.7:57544) 2023-03-03T09:46:10.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a1f7fe700 1 -- 172.21.15.7:0/1378519709 learned_addr learned my addr 172.21.15.7:0/1378519709 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:46:10.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a1ffff700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a200abe20 0x7f9a201492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:10.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.190+0000 7f9a24a87700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a2014fda0 0x7f9a200c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:10.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a1f7fe700 1 -- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a2014fda0 msgr2=0x7f9a200c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:10.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a1f7fe700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a2014fda0 0x7f9a200c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:10.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a1f7fe700 1 -- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a200abe20 msgr2=0x7f9a201492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:10.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a1f7fe700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a200abe20 0x7f9a201492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:10.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a1f7fe700 1 -- 172.21.15.7:0/1378519709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a100094a0 con 0x7f9a2014f9d0 2023-03-03T09:46:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a24a87700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a2014fda0 0x7f9a200c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:46:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a1f7fe700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 0x7f9a201dbf20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9a1400c4a0 tx=0x7f9a1400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a1cff9700 1 -- 172.21.15.7:0/1378519709 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a14011790 con 0x7f9a2014f9d0 2023-03-03T09:46:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a1cff9700 1 -- 172.21.15.7:0/1378519709 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a140118f0 con 0x7f9a2014f9d0 2023-03-03T09:46:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a264ea700 1 -- 172.21.15.7:0/1378519709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a200ac350 con 0x7f9a2014f9d0 2023-03-03T09:46:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a1cff9700 1 -- 172.21.15.7:0/1378519709 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a14011ca0 con 0x7f9a2014f9d0 2023-03-03T09:46:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.191+0000 7f9a264ea700 1 -- 172.21.15.7:0/1378519709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a200ab630 con 0x7f9a2014f9d0 2023-03-03T09:46:10.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.192+0000 7f9a264ea700 1 -- 172.21.15.7:0/1378519709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a20071fa0 con 0x7f9a2014f9d0 2023-03-03T09:46:10.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.192+0000 7f9a1cff9700 1 -- 172.21.15.7:0/1378519709 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a14010050 con 0x7f9a2014f9d0 2023-03-03T09:46:10.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.193+0000 7f9a1cff9700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a0c0682f0 0x7f9a0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.193+0000 7f9a1cff9700 1 -- 172.21.15.7:0/1378519709 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9a14085e60 con 0x7f9a2014f9d0 2023-03-03T09:46:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.193+0000 7f9a1ffff700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a0c0682f0 0x7f9a0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.194+0000 7f9a1ffff700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a0c0682f0 0x7f9a0c06a7a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9a100090b0 tx=0x7f9a1001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:10.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.198+0000 7f9a1cff9700 1 -- 172.21.15.7:0/1378519709 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a14052170 con 0x7f9a2014f9d0 2023-03-03T09:46:10.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.437+0000 7f9a264ea700 1 -- 172.21.15.7:0/1378519709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a201ee130 con 0x7f9a2014f9d0 2023-03-03T09:46:10.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.437+0000 7f9a1cff9700 1 -- 172.21.15.7:0/1378519709 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9a14055790 con 0x7f9a2014f9d0 2023-03-03T09:46:10.439 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:46:10.439 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:46:10.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.439+0000 7f9a264ea700 1 -- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a0c0682f0 msgr2=0x7f9a0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:10.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.439+0000 7f9a264ea700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a0c0682f0 0x7f9a0c06a7a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9a100090b0 tx=0x7f9a1001c040 comp rx=0 tx=0).stop 2023-03-03T09:46:10.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.439+0000 7f9a264ea700 1 -- 172.21.15.7:0/1378519709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 msgr2=0x7f9a201dbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:10.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.439+0000 7f9a264ea700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 0x7f9a201dbf20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9a1400c4a0 tx=0x7f9a1400c7b0 comp rx=0 tx=0).stop 2023-03-03T09:46:10.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.439+0000 7f9a264ea700 1 -- 172.21.15.7:0/1378519709 shutdown_connections 2023-03-03T09:46:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.439+0000 7f9a264ea700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a2014fda0 0x7f9a200c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.439+0000 7f9a264ea700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a200abe20 0x7f9a201492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.439+0000 7f9a264ea700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a0c0682f0 0x7f9a0c06a7a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.439+0000 7f9a264ea700 1 --2- 172.21.15.7:0/1378519709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a2014f9d0 0x7f9a201dbf20 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.439+0000 7f9a264ea700 1 -- 172.21.15.7:0/1378519709 >> 172.21.15.7:0/1378519709 conn(0x7f9a200c8900 msgr2=0x7f9a200c6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:46:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.440+0000 7f9a264ea700 1 -- 172.21.15.7:0/1378519709 shutdown_connections 2023-03-03T09:46:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:10.440+0000 7f9a264ea700 1 -- 172.21.15.7:0/1378519709 wait complete. 2023-03-03T09:46:10.443 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:46:10.453 INFO:tasks.mds_thrash.fs.[b]:waiting for 40 secs before thrashing 2023-03-03T09:46:14.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:46:14.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.460+0000 7fa7154d8700 1 Processor -- start 2023-03-03T09:46:14.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.461+0000 7fa7154d8700 1 -- start start 2023-03-03T09:46:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa7154d8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa71006b6f0 0x7fa710155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa7154d8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa710157bb0 con 0x7fa7100c7a30 2023-03-03T09:46:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa7154d8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71006a2f0 con 0x7fa7100c7e00 2023-03-03T09:46:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa7154d8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa71006ab00 con 0x7fa71006b6f0 2023-03-03T09:46:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa7100c7a30 0x7fa71014e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51530/0 (socket says 172.21.15.7:51530) 2023-03-03T09:46:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70effd700 1 -- 172.21.15.7:0/357338462 learned_addr learned my addr 172.21.15.7:0/357338462 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:46:14.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492249369 0 0) 0x7fa710157bb0 con 0x7fa7100c7a30 2023-03-03T09:46:14.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70f7fe700 1 --2- 172.21.15.7:0/357338462 >> v2:172.21.15.143:3300/0 conn(0x7fa71006b6f0 0x7fa710155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:14.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f80035b0 con 0x7fa7100c7a30 2023-03-03T09:46:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 954206195 0 0) 0x7fa6f80035b0 con 0x7fa7100c7a30 2023-03-03T09:46:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 >> v2:172.21.15.143:3300/0 conn(0x7fa71006b6f0 msgr2=0x7fa710155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:46:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70dffb700 1 --2- 172.21.15.7:0/357338462 >> v2:172.21.15.143:3300/0 conn(0x7fa71006b6f0 0x7fa710155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 >> v1:172.21.15.143:6789/0 conn(0x7fa7100c7e00 legacy=0x7fa710154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:14.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7100bfbb0 con 0x7fa7100c7a30 2023-03-03T09:46:14.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa7000017f0 con 0x7fa7100c7a30 2023-03-03T09:46:14.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 >> v1:172.21.15.7:6789/0 conn(0x7fa7100c7a30 legacy=0x7fa71014e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:14.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.462+0000 7fa70dffb700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f8003a70 0x7fa6f8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:14.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70dffb700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6f8004850 0x7fa6f8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:14.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70dffb700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f8005250 0x7fa6f800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:14.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71006ab00 con 0x7fa6f8005250 2023-03-03T09:46:14.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f800c190 con 0x7fa6f8003a70 2023-03-03T09:46:14.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f800c350 con 0x7fa6f8004850 2023-03-03T09:46:14.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70f7fe700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f8005250 0x7fa6f800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:14.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70effd700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f8003a70 0x7fa6f8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:14.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70e7fc700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6f8004850 0x7fa6f8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:14.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70f7fe700 1 -- 172.21.15.7:0/357338462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6f8004850 msgr2=0x7fa6f8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:46:14.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70f7fe700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6f8004850 0x7fa6f8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:14.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70f7fe700 1 -- 172.21.15.7:0/357338462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f8003a70 msgr2=0x7fa6f8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:46:14.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70f7fe700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f8003a70 0x7fa6f8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:14.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70f7fe700 1 -- 172.21.15.7:0/357338462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f80035b0 con 0x7fa6f8005250 2023-03-03T09:46:14.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70e7fc700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6f8004850 0x7fa6f8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:46:14.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70f7fe700 1 -- 172.21.15.7:0/357338462 reap_dead start 2023-03-03T09:46:14.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70f7fe700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f8005250 0x7fa6f800ba10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa7100bfed0 tx=0x7fa6f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62340fd90763ffa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:14.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708018070 con 0x7fa6f8005250 2023-03-03T09:46:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70800f040 con 0x7fa6f8005250 2023-03-03T09:46:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.463+0000 7fa70dffb700 1 -- 172.21.15.7:0/357338462 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70800ec10 con 0x7fa6f8005250 2023-03-03T09:46:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.464+0000 7fa7154d8700 1 -- 172.21.15.7:0/357338462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f8005250 msgr2=0x7fa6f800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.464+0000 7fa7154d8700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f8005250 0x7fa6f800ba10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa7100bfed0 tx=0x7fa6f8005880 comp rx=0 tx=0).stop 2023-03-03T09:46:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.464+0000 7fa7154d8700 1 -- 172.21.15.7:0/357338462 shutdown_connections 2023-03-03T09:46:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.464+0000 7fa7154d8700 1 --2- 172.21.15.7:0/357338462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f8005250 0x7fa6f800ba10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.464+0000 7fa7154d8700 1 -- 172.21.15.7:0/357338462 >> 172.21.15.7:0/357338462 conn(0x7fa710073d20 msgr2=0x7fa7100716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:46:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.464+0000 7fa7154d8700 1 -- 172.21.15.7:0/357338462 shutdown_connections 2023-03-03T09:46:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.464+0000 7fa7154d8700 1 -- 172.21.15.7:0/357338462 wait complete. 2023-03-03T09:46:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.465+0000 7fa7154d8700 1 Processor -- start 2023-03-03T09:46:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.465+0000 7fa7154d8700 1 -- start start 2023-03-03T09:46:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa7154d8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7101536f0 0x7fa7101402b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa7154d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa710153ac0 0x7fa7101407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa7154d8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa710141340 0x7fa7101e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa7154d8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7100c22b0 con 0x7fa710153ac0 2023-03-03T09:46:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa7154d8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7101edcc0 con 0x7fa710141340 2023-03-03T09:46:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa7154d8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa710156f30 con 0x7fa7101536f0 2023-03-03T09:46:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa70e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa710153ac0 0x7fa7101407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa70e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa710153ac0 0x7fa7101407f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57564/0 (socket says 172.21.15.7:57564) 2023-03-03T09:46:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa70e7fc700 1 -- 172.21.15.7:0/1636623256 learned_addr learned my addr 172.21.15.7:0/1636623256 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:46:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa70f7fe700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa710141340 0x7fa7101e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa70effd700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7101536f0 0x7fa7101402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa70e7fc700 1 -- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7101536f0 msgr2=0x7fa7101402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:14.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa70e7fc700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7101536f0 0x7fa7101402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:14.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa70e7fc700 1 -- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa710141340 msgr2=0x7fa7101e9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:14.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa70e7fc700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa710141340 0x7fa7101e9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:14.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.466+0000 7fa70e7fc700 1 -- 172.21.15.7:0/1636623256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71006a820 con 0x7fa710153ac0 2023-03-03T09:46:14.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.467+0000 7fa70e7fc700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa710153ac0 0x7fa7101407f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa70400e580 tx=0x7fa70400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:14.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.467+0000 7fa6f77fe700 1 -- 172.21.15.7:0/1636623256 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7040052e0 con 0x7fa710153ac0 2023-03-03T09:46:14.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.467+0000 7fa6f77fe700 1 -- 172.21.15.7:0/1636623256 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa704005440 con 0x7fa710153ac0 2023-03-03T09:46:14.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.467+0000 7fa7154d8700 1 -- 172.21.15.7:0/1636623256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa71006a540 con 0x7fa710153ac0 2023-03-03T09:46:14.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.467+0000 7fa6f77fe700 1 -- 172.21.15.7:0/1636623256 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7040038c0 con 0x7fa710153ac0 2023-03-03T09:46:14.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.467+0000 7fa7154d8700 1 -- 172.21.15.7:0/1636623256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7100bf6d0 con 0x7fa710153ac0 2023-03-03T09:46:14.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.468+0000 7fa7154d8700 1 -- 172.21.15.7:0/1636623256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa710074bb0 con 0x7fa710153ac0 2023-03-03T09:46:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.468+0000 7fa6f77fe700 1 -- 172.21.15.7:0/1636623256 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa70401b030 con 0x7fa710153ac0 2023-03-03T09:46:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.469+0000 7fa6f77fe700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6fc0682f0 0x7fa6fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.469+0000 7fa6f77fe700 1 -- 172.21.15.7:0/1636623256 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa7040856a0 con 0x7fa710153ac0 2023-03-03T09:46:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.469+0000 7fa70effd700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6fc0682f0 0x7fa6fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.470+0000 7fa70effd700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6fc0682f0 0x7fa6fc06a7a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa700000ef0 tx=0x7fa700009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:14.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.473+0000 7fa6f77fe700 1 -- 172.21.15.7:0/1636623256 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa70404d5e0 con 0x7fa710153ac0 2023-03-03T09:46:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.711+0000 7fa7154d8700 1 -- 172.21.15.7:0/1636623256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa71014a740 con 0x7fa710153ac0 2023-03-03T09:46:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.712+0000 7fa6f77fe700 1 -- 172.21.15.7:0/1636623256 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa704051bb0 con 0x7fa710153ac0 2023-03-03T09:46:14.714 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:46:14.714 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:46:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 -- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6fc0682f0 msgr2=0x7fa6fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6fc0682f0 0x7fa6fc06a7a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa700000ef0 tx=0x7fa700009040 comp rx=0 tx=0).stop 2023-03-03T09:46:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 -- 172.21.15.7:0/1636623256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa710153ac0 msgr2=0x7fa7101407f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa710153ac0 0x7fa7101407f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fa70400e580 tx=0x7fa70400e940 comp rx=0 tx=0).stop 2023-03-03T09:46:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 -- 172.21.15.7:0/1636623256 shutdown_connections 2023-03-03T09:46:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa710141340 0x7fa7101e9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7101536f0 0x7fa7101402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6fc0682f0 0x7fa6fc06a7a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 --2- 172.21.15.7:0/1636623256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa710153ac0 0x7fa7101407f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 -- 172.21.15.7:0/1636623256 >> 172.21.15.7:0/1636623256 conn(0x7fa6f8005250 msgr2=0x7fa71006acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:46:14.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 -- 172.21.15.7:0/1636623256 shutdown_connections 2023-03-03T09:46:14.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:14.714+0000 7fa7154d8700 1 -- 172.21.15.7:0/1636623256 wait complete. 2023-03-03T09:46:14.717 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:46:14.727 INFO:tasks.mds_thrash.fs.[a]:waiting for 104 secs before thrashing 2023-03-03T09:46:19.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:19.281 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:49.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:49.686 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:46:50.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:46:50.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.591+0000 7fd00cc93700 1 Processor -- start 2023-03-03T09:46:50.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.591+0000 7fd00cc93700 1 -- start start 2023-03-03T09:46:50.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.592+0000 7fd00cc93700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd008151e80 0x7fd0080c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:50.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.592+0000 7fd00cc93700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0080c08d0 con 0x7fd0080c6fe0 2023-03-03T09:46:50.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.592+0000 7fd00cc93700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd008154930 con 0x7fd008155410 2023-03-03T09:46:50.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.592+0000 7fd00cc93700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd008152250 con 0x7fd008151e80 2023-03-03T09:46:50.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.592+0000 7fd00659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd008151e80 0x7fd0080c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:50.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.592+0000 7fd006d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd0080c6fe0 0x7fd0080bfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51582/0 (socket says 172.21.15.7:51582) 2023-03-03T09:46:50.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.592+0000 7fd006d9d700 1 -- 172.21.15.7:0/498328853 learned_addr learned my addr 172.21.15.7:0/498328853 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:46:50.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.593+0000 7fd00659c700 1 -- 172.21.15.7:0/498328853 >> v1:172.21.15.143:6789/0 conn(0x7fd008155410 legacy=0x7fd0080bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:50.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.593+0000 7fd00659c700 1 -- 172.21.15.7:0/498328853 >> v1:172.21.15.7:6789/0 conn(0x7fd0080c6fe0 legacy=0x7fd0080bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:50.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.593+0000 7fd00659c700 1 -- 172.21.15.7:0/498328853 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd008154c10 con 0x7fd008151e80 2023-03-03T09:46:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.593+0000 7fd00559a700 1 -- 172.21.15.7:0/498328853 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163500798 0 0) 0x7fd0080c08d0 con 0x7fd0080c6fe0 2023-03-03T09:46:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.593+0000 7fd00659c700 1 --2- 172.21.15.7:0/498328853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 0x7fd0080c6aa0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009840 tx=0x7fcff8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f715219f3ce966 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.593+0000 7fd00559a700 1 -- 172.21.15.7:0/498328853 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff8011830 con 0x7fd008151e80 2023-03-03T09:46:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.593+0000 7fd00559a700 1 -- 172.21.15.7:0/498328853 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff8011990 con 0x7fd008151e80 2023-03-03T09:46:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.593+0000 7fd00559a700 1 -- 172.21.15.7:0/498328853 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff8011d20 con 0x7fd008151e80 2023-03-03T09:46:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.594+0000 7fd00cc93700 1 -- 172.21.15.7:0/498328853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 msgr2=0x7fd0080c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.594+0000 7fd00cc93700 1 --2- 172.21.15.7:0/498328853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 0x7fd0080c6aa0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009840 tx=0x7fcff8009b50 comp rx=0 tx=0).stop 2023-03-03T09:46:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.594+0000 7fd00cc93700 1 -- 172.21.15.7:0/498328853 shutdown_connections 2023-03-03T09:46:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.594+0000 7fd00cc93700 1 --2- 172.21.15.7:0/498328853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 0x7fd0080c6aa0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.594+0000 7fd00cc93700 1 -- 172.21.15.7:0/498328853 >> 172.21.15.7:0/498328853 conn(0x7fd008071630 msgr2=0x7fd00806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:46:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.594+0000 7fd00cc93700 1 -- 172.21.15.7:0/498328853 shutdown_connections 2023-03-03T09:46:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.594+0000 7fd00cc93700 1 -- 172.21.15.7:0/498328853 wait complete. 2023-03-03T09:46:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.595+0000 7fd00cc93700 1 Processor -- start 2023-03-03T09:46:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.595+0000 7fd00cc93700 1 -- start start 2023-03-03T09:46:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd00cc93700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0080c6fe0 0x7fd0081ea470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd00cc93700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 0x7fd0080c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd00cc93700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd008155410 0x7fd00814aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd00cc93700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080ca3c0 con 0x7fd008151e80 2023-03-03T09:46:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd00cc93700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0081584c0 con 0x7fd008155410 2023-03-03T09:46:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd00cc93700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0081ee440 con 0x7fd0080c6fe0 2023-03-03T09:46:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd005d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 0x7fd0080c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd005d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 0x7fd0080c88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57606/0 (socket says 172.21.15.7:57606) 2023-03-03T09:46:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd005d9b700 1 -- 172.21.15.7:0/2139998425 learned_addr learned my addr 172.21.15.7:0/2139998425 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:46:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd006d9d700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd008155410 0x7fd00814aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd00659c700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0080c6fe0 0x7fd0081ea470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd005d9b700 1 -- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0080c6fe0 msgr2=0x7fd0081ea470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd005d9b700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0080c6fe0 0x7fd0081ea470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd005d9b700 1 -- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd008155410 msgr2=0x7fd00814aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd005d9b700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd008155410 0x7fd00814aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:50.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd005d9b700 1 -- 172.21.15.7:0/2139998425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff80094a0 con 0x7fd008151e80 2023-03-03T09:46:50.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd006d9d700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd008155410 0x7fd00814aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:46:50.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.596+0000 7fd005d9b700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 0x7fd0080c88c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00c490 tx=0x7fcffc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:50.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.597+0000 7fcff6ffd700 1 -- 172.21.15.7:0/2139998425 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc0052d0 con 0x7fd008151e80 2023-03-03T09:46:50.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.597+0000 7fcff6ffd700 1 -- 172.21.15.7:0/2139998425 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc005430 con 0x7fd008151e80 2023-03-03T09:46:50.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.597+0000 7fd00cc93700 1 -- 172.21.15.7:0/2139998425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0080c5220 con 0x7fd008151e80 2023-03-03T09:46:50.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.597+0000 7fcff6ffd700 1 -- 172.21.15.7:0/2139998425 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc0038c0 con 0x7fd008151e80 2023-03-03T09:46:50.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.597+0000 7fd00cc93700 1 -- 172.21.15.7:0/2139998425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd008154ef0 con 0x7fd008151e80 2023-03-03T09:46:50.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.598+0000 7fd00cc93700 1 -- 172.21.15.7:0/2139998425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0080c85e0 con 0x7fd008151e80 2023-03-03T09:46:50.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.598+0000 7fcff6ffd700 1 -- 172.21.15.7:0/2139998425 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcffc019030 con 0x7fd008151e80 2023-03-03T09:46:50.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.598+0000 7fcff6ffd700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 0x7fcfe806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:46:50.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.598+0000 7fcff6ffd700 1 -- 172.21.15.7:0/2139998425 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcffc085390 con 0x7fd008151e80 2023-03-03T09:46:50.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.599+0000 7fd00659c700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 0x7fcfe806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:46:50.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.599+0000 7fd00659c700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 0x7fcfe806a7a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcff80090b0 tx=0x7fcff8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:46:50.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.603+0000 7fcff6ffd700 1 -- 172.21.15.7:0/2139998425 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd0080c85e0 con 0x7fd008151e80 2023-03-03T09:46:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.840+0000 7fd00cc93700 1 -- 172.21.15.7:0/2139998425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0080c85e0 con 0x7fd008151e80 2023-03-03T09:46:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.840+0000 7fcff6ffd700 1 -- 172.21.15.7:0/2139998425 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd0080c85e0 con 0x7fd008151e80 2023-03-03T09:46:50.842 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:46:50.842 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:46:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 -- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 msgr2=0x7fcfe806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 0x7fcfe806a7a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcff80090b0 tx=0x7fcff8008030 comp rx=0 tx=0).stop 2023-03-03T09:46:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 -- 172.21.15.7:0/2139998425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 msgr2=0x7fd0080c88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:46:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 0x7fd0080c88c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00c490 tx=0x7fcffc00c850 comp rx=0 tx=0).stop 2023-03-03T09:46:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 -- 172.21.15.7:0/2139998425 shutdown_connections 2023-03-03T09:46:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd008155410 0x7fd00814aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0080c6fe0 0x7fd0081ea470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 0x7fcfe806a7a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 --2- 172.21.15.7:0/2139998425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008151e80 0x7fd0080c88c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:46:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 -- 172.21.15.7:0/2139998425 >> 172.21.15.7:0/2139998425 conn(0x7fd008071630 msgr2=0x7fd0081540f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:46:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 -- 172.21.15.7:0/2139998425 shutdown_connections 2023-03-03T09:46:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:46:50.842+0000 7fd00cc93700 1 -- 172.21.15.7:0/2139998425 wait complete. 2023-03-03T09:46:50.846 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:46:50.855 INFO:tasks.mds_thrash.fs.[b]:waiting for 115 secs before thrashing 2023-03-03T09:47:19.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:19.748 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:50.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:50.107 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:47:58.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:47:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.862+0000 7f515b44b700 1 Processor -- start 2023-03-03T09:47:58.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.862+0000 7f515b44b700 1 -- start start 2023-03-03T09:47:58.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f515b44b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f515406d870 0x7f51541550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:58.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f515b44b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 0x7f51540c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:58.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f515b44b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51540c7a60 con 0x7f51540c1da0 2023-03-03T09:47:58.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f515b44b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51540c2af0 con 0x7f515406d870 2023-03-03T09:47:58.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f515b44b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51540c3620 con 0x7f51540c3e30 2023-03-03T09:47:58.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f51591e7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f515406d870 0x7f51541550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:58.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f51599e8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 0x7f51540c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:58.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f51589e6700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f51540c1da0 0x7f51540c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:55160/0 (socket says 172.21.15.7:55160) 2023-03-03T09:47:58.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f51589e6700 1 -- 172.21.15.7:0/272919618 learned_addr learned my addr 172.21.15.7:0/272919618 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:47:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f51591e7700 1 -- 172.21.15.7:0/272919618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 msgr2=0x7f51540c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:47:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f51591e7700 1 --2- 172.21.15.7:0/272919618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 0x7f51540c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f51591e7700 1 -- 172.21.15.7:0/272919618 >> v1:172.21.15.143:6789/0 conn(0x7f51540c1da0 legacy=0x7f51540c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f5147fff700 1 -- 172.21.15.7:0/272919618 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308927907 0 0) 0x7f51540c7a60 con 0x7f51540c1da0 2023-03-03T09:47:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f51591e7700 1 -- 172.21.15.7:0/272919618 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51541545e0 con 0x7f515406d870 2023-03-03T09:47:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.863+0000 7f51599e8700 1 --2- 172.21.15.7:0/272919618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 0x7f51540c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:47:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f51591e7700 1 --2- 172.21.15.7:0/272919618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f515406d870 0x7f51541550c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5150009840 tx=0x7f5150009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dcb503aa6d4c8dd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:47:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f5147fff700 1 -- 172.21.15.7:0/272919618 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150011830 con 0x7f515406d870 2023-03-03T09:47:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f5147fff700 1 -- 172.21.15.7:0/272919618 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5150011990 con 0x7f515406d870 2023-03-03T09:47:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f5147fff700 1 -- 172.21.15.7:0/272919618 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150011d20 con 0x7f515406d870 2023-03-03T09:47:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f515b44b700 1 -- 172.21.15.7:0/272919618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f515406d870 msgr2=0x7f51541550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f515b44b700 1 --2- 172.21.15.7:0/272919618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f515406d870 0x7f51541550c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5150009840 tx=0x7f5150009b50 comp rx=0 tx=0).stop 2023-03-03T09:47:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f515b44b700 1 -- 172.21.15.7:0/272919618 shutdown_connections 2023-03-03T09:47:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f515b44b700 1 --2- 172.21.15.7:0/272919618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 0x7f51540c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:58.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f515b44b700 1 --2- 172.21.15.7:0/272919618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f515406d870 0x7f51541550c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:58.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f515b44b700 1 -- 172.21.15.7:0/272919618 >> 172.21.15.7:0/272919618 conn(0x7f515406c110 msgr2=0x7f515406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:47:58.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f515b44b700 1 -- 172.21.15.7:0/272919618 shutdown_connections 2023-03-03T09:47:58.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.864+0000 7f515b44b700 1 -- 172.21.15.7:0/272919618 wait complete. 2023-03-03T09:47:58.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.866+0000 7f515b44b700 1 Processor -- start 2023-03-03T09:47:58.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.866+0000 7f515b44b700 1 -- start start 2023-03-03T09:47:58.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.866+0000 7f515b44b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f515406d870 0x7f51541ecfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:58.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.866+0000 7f515b44b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51540c1da0 0x7f515414fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:58.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.866+0000 7f515b44b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 0x7f515414ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:58.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.866+0000 7f515b44b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5154150e10 con 0x7f51540c1da0 2023-03-03T09:47:58.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.866+0000 7f515b44b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515414f5a0 con 0x7f515406d870 2023-03-03T09:47:58.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.866+0000 7f515b44b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5154151bf0 con 0x7f51540c3e30 2023-03-03T09:47:58.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51589e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51540c1da0 0x7f515414fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:58.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51589e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51540c1da0 0x7f515414fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57668/0 (socket says 172.21.15.7:57668) 2023-03-03T09:47:58.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51589e6700 1 -- 172.21.15.7:0/3014952077 learned_addr learned my addr 172.21.15.7:0/3014952077 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:47:58.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51599e8700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 0x7f515414ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:58.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51591e7700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f515406d870 0x7f51541ecfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:58.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51589e6700 1 -- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 msgr2=0x7f515414ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:58.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51589e6700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 0x7f515414ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:58.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51589e6700 1 -- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f515406d870 msgr2=0x7f51541ecfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:58.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51589e6700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f515406d870 0x7f51541ecfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:58.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51589e6700 1 -- 172.21.15.7:0/3014952077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c009a70 con 0x7f51540c1da0 2023-03-03T09:47:58.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f51589e6700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51540c1da0 0x7f515414fa30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f514800c750 tx=0x7f514800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:47:58.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f5145ffb700 1 -- 172.21.15.7:0/3014952077 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148004c80 con 0x7f51540c1da0 2023-03-03T09:47:58.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f5145ffb700 1 -- 172.21.15.7:0/3014952077 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5148004e00 con 0x7f51540c1da0 2023-03-03T09:47:58.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f515b44b700 1 -- 172.21.15.7:0/3014952077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51500094a0 con 0x7f51540c1da0 2023-03-03T09:47:58.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f5145ffb700 1 -- 172.21.15.7:0/3014952077 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148007310 con 0x7f51540c1da0 2023-03-03T09:47:58.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.867+0000 7f515b44b700 1 -- 172.21.15.7:0/3014952077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5154154ba0 con 0x7f51540c1da0 2023-03-03T09:47:58.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.868+0000 7f515b44b700 1 -- 172.21.15.7:0/3014952077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51541548c0 con 0x7f51540c1da0 2023-03-03T09:47:58.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.869+0000 7f5145ffb700 1 -- 172.21.15.7:0/3014952077 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f514801a020 con 0x7f51540c1da0 2023-03-03T09:47:58.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.869+0000 7f5145ffb700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5138068320 0x7f513806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:47:58.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.869+0000 7f5145ffb700 1 -- 172.21.15.7:0/3014952077 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5148085c80 con 0x7f51540c1da0 2023-03-03T09:47:58.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.869+0000 7f51591e7700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5138068320 0x7f513806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:47:58.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.870+0000 7f51591e7700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5138068320 0x7f513806a7d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f51500090b0 tx=0x7f5150008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:47:58.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:58.873+0000 7f5145ffb700 1 -- 172.21.15.7:0/3014952077 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5148052070 con 0x7f51540c1da0 2023-03-03T09:47:59.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.108+0000 7f515b44b700 1 -- 172.21.15.7:0/3014952077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f515406dc40 con 0x7f51540c1da0 2023-03-03T09:47:59.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.108+0000 7f5145ffb700 1 -- 172.21.15.7:0/3014952077 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5148055690 con 0x7f51540c1da0 2023-03-03T09:47:59.110 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:47:59.110 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:47:59.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 -- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5138068320 msgr2=0x7f513806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:59.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5138068320 0x7f513806a7d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f51500090b0 tx=0x7f5150008030 comp rx=0 tx=0).stop 2023-03-03T09:47:59.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 -- 172.21.15.7:0/3014952077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51540c1da0 msgr2=0x7f515414fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:47:59.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51540c1da0 0x7f515414fa30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f514800c750 tx=0x7f514800ca60 comp rx=0 tx=0).stop 2023-03-03T09:47:59.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 -- 172.21.15.7:0/3014952077 shutdown_connections 2023-03-03T09:47:59.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51540c3e30 0x7f515414ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:59.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f515406d870 0x7f51541ecfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:59.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5138068320 0x7f513806a7d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:59.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 --2- 172.21.15.7:0/3014952077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51540c1da0 0x7f515414fa30 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:47:59.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 -- 172.21.15.7:0/3014952077 >> 172.21.15.7:0/3014952077 conn(0x7f515406c110 msgr2=0x7f515406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:47:59.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 -- 172.21.15.7:0/3014952077 shutdown_connections 2023-03-03T09:47:59.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:47:59.111+0000 7f515b44b700 1 -- 172.21.15.7:0/3014952077 wait complete. 2023-03-03T09:47:59.114 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:47:59.124 INFO:tasks.mds_thrash.fs.[a]:waiting for 35 secs before thrashing 2023-03-03T09:48:20.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:20.183 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:34.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:48:34.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.263+0000 7f09ccae5700 1 Processor -- start 2023-03-03T09:48:34.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.263+0000 7f09ccae5700 1 -- start start 2023-03-03T09:48:34.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09ccae5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f09c8072250 0x7f09c814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:34.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09ccae5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09c80c1ce0 0x7f09c814e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:34.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09ccae5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c81510d0 con 0x7f09c8071e80 2023-03-03T09:48:34.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09ccae5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c80c3740 con 0x7f09c8072250 2023-03-03T09:48:34.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09ccae5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c8070670 con 0x7f09c80c1ce0 2023-03-03T09:48:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f09c8071e80 0x7f09c8151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51680/0 (socket says 172.21.15.7:51680) 2023-03-03T09:48:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c659c700 1 -- 172.21.15.7:0/1310422123 learned_addr learned my addr 172.21.15.7:0/1310422123 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:48:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c5d9b700 1 --2- 172.21.15.7:0/1310422123 >> v2:172.21.15.143:3300/0 conn(0x7f09c8072250 0x7f09c814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c6d9d700 1 --2- 172.21.15.7:0/1310422123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09c80c1ce0 0x7f09c814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c559a700 1 -- 172.21.15.7:0/1310422123 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835357718 0 0) 0x7f09c81510d0 con 0x7f09c8071e80 2023-03-03T09:48:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c559a700 1 -- 172.21.15.7:0/1310422123 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b00035b0 con 0x7f09c8071e80 2023-03-03T09:48:34.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c559a700 1 -- 172.21.15.7:0/1310422123 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 888930749 0 0) 0x7f09b00035b0 con 0x7f09c8071e80 2023-03-03T09:48:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c5d9b700 1 -- 172.21.15.7:0/1310422123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09c80c1ce0 msgr2=0x7f09c814e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:48:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c5d9b700 1 --2- 172.21.15.7:0/1310422123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09c80c1ce0 0x7f09c814e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c5d9b700 1 -- 172.21.15.7:0/1310422123 >> v1:172.21.15.7:6789/0 conn(0x7f09c8071e80 legacy=0x7f09c8151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.264+0000 7f09c5d9b700 1 -- 172.21.15.7:0/1310422123 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c80c39e0 con 0x7f09c8072250 2023-03-03T09:48:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09c5d9b700 1 --2- 172.21.15.7:0/1310422123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09c8072250 0x7f09c814e1f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009840 tx=0x7f09b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=273bf1b784057116 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09c559a700 1 -- 172.21.15.7:0/1310422123 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc001670 con 0x7f09c8072250 2023-03-03T09:48:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09c559a700 1 -- 172.21.15.7:0/1310422123 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b8004030 con 0x7f09c8072250 2023-03-03T09:48:34.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09c559a700 1 -- 172.21.15.7:0/1310422123 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b80043c0 con 0x7f09c8072250 2023-03-03T09:48:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09ccae5700 1 -- 172.21.15.7:0/1310422123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09c8072250 msgr2=0x7f09c814e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09ccae5700 1 --2- 172.21.15.7:0/1310422123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09c8072250 0x7f09c814e1f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009840 tx=0x7f09b8009b50 comp rx=0 tx=0).stop 2023-03-03T09:48:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09ccae5700 1 -- 172.21.15.7:0/1310422123 shutdown_connections 2023-03-03T09:48:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09ccae5700 1 --2- 172.21.15.7:0/1310422123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09c80c1ce0 0x7f09c814e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09ccae5700 1 --2- 172.21.15.7:0/1310422123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09c8072250 0x7f09c814e1f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:34.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09ccae5700 1 -- 172.21.15.7:0/1310422123 >> 172.21.15.7:0/1310422123 conn(0x7f09c80a94d0 msgr2=0x7f09c80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:48:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.265+0000 7f09ccae5700 1 -- 172.21.15.7:0/1310422123 shutdown_connections 2023-03-03T09:48:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.266+0000 7f09ccae5700 1 -- 172.21.15.7:0/1310422123 wait complete. 2023-03-03T09:48:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.267+0000 7f09ccae5700 1 Processor -- start 2023-03-03T09:48:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.267+0000 7f09ccae5700 1 -- start start 2023-03-03T09:48:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09ccae5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09c8071e80 0x7f09c81ed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09ccae5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c8072250 0x7f09c81df160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:34.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09ccae5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09c80c1ce0 0x7f09c8144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09ccae5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c81ee950 con 0x7f09c8072250 2023-03-03T09:48:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09ccae5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c81456f0 con 0x7f09c8071e80 2023-03-03T09:48:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09ccae5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c80c5160 con 0x7f09c80c1ce0 2023-03-03T09:48:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c8072250 0x7f09c81df160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c8072250 0x7f09c81df160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57708/0 (socket says 172.21.15.7:57708) 2023-03-03T09:48:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c5d9b700 1 -- 172.21.15.7:0/4042509025 learned_addr learned my addr 172.21.15.7:0/4042509025 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:48:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c659c700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09c8071e80 0x7f09c81ed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:34.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c6d9d700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09c80c1ce0 0x7f09c8144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c5d9b700 1 -- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09c80c1ce0 msgr2=0x7f09c8144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c5d9b700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09c80c1ce0 0x7f09c8144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c5d9b700 1 -- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09c8071e80 msgr2=0x7f09c81ed9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c5d9b700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09c8071e80 0x7f09c81ed9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c5d9b700 1 -- 172.21.15.7:0/4042509025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b80094a0 con 0x7f09c8072250 2023-03-03T09:48:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c659c700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09c8071e80 0x7f09c81ed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:48:34.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09c5d9b700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c8072250 0x7f09c81df160 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f09b80090b0 tx=0x7f09b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09b6ffd700 1 -- 172.21.15.7:0/4042509025 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8003e20 con 0x7f09c8072250 2023-03-03T09:48:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09b6ffd700 1 -- 172.21.15.7:0/4042509025 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b801f070 con 0x7f09c8072250 2023-03-03T09:48:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09ccae5700 1 -- 172.21.15.7:0/4042509025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c8151f30 con 0x7f09c8072250 2023-03-03T09:48:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.268+0000 7f09b6ffd700 1 -- 172.21.15.7:0/4042509025 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8011870 con 0x7f09c8072250 2023-03-03T09:48:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.269+0000 7f09ccae5700 1 -- 172.21.15.7:0/4042509025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c8151300 con 0x7f09c8072250 2023-03-03T09:48:34.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.269+0000 7f09ccae5700 1 -- 172.21.15.7:0/4042509025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c806c9a0 con 0x7f09c8072250 2023-03-03T09:48:34.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.270+0000 7f09b6ffd700 1 -- 172.21.15.7:0/4042509025 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f09b8017020 con 0x7f09c8072250 2023-03-03T09:48:34.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.270+0000 7f09b6ffd700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09a80682f0 0x7f09a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:34.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.270+0000 7f09b6ffd700 1 -- 172.21.15.7:0/4042509025 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f09b8021070 con 0x7f09c8072250 2023-03-03T09:48:34.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.271+0000 7f09c659c700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09a80682f0 0x7f09a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:34.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.271+0000 7f09c659c700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09a80682f0 0x7f09a806a7a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f09bc001eb0 tx=0x7f09bc001d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:34.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.275+0000 7f09b6ffd700 1 -- 172.21.15.7:0/4042509025 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f09b8053df0 con 0x7f09c8072250 2023-03-03T09:48:34.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.516+0000 7f09ccae5700 1 -- 172.21.15.7:0/4042509025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09c8074bb0 con 0x7f09c8072250 2023-03-03T09:48:34.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.517+0000 7f09b6ffd700 1 -- 172.21.15.7:0/4042509025 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f09b8057410 con 0x7f09c8072250 2023-03-03T09:48:34.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:48:34.518 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:48:34.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 -- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09a80682f0 msgr2=0x7f09a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09a80682f0 0x7f09a806a7a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f09bc001eb0 tx=0x7f09bc001d80 comp rx=0 tx=0).stop 2023-03-03T09:48:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 -- 172.21.15.7:0/4042509025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c8072250 msgr2=0x7f09c81df160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c8072250 0x7f09c81df160 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f09b80090b0 tx=0x7f09b8004750 comp rx=0 tx=0).stop 2023-03-03T09:48:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 -- 172.21.15.7:0/4042509025 shutdown_connections 2023-03-03T09:48:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09c80c1ce0 0x7f09c8144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09c8071e80 0x7f09c81ed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:34.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09a80682f0 0x7f09a806a7a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:34.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 --2- 172.21.15.7:0/4042509025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09c8072250 0x7f09c81df160 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:34.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 -- 172.21.15.7:0/4042509025 >> 172.21.15.7:0/4042509025 conn(0x7f09c80a94d0 msgr2=0x7f09c814dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:48:34.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 -- 172.21.15.7:0/4042509025 shutdown_connections 2023-03-03T09:48:34.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:34.519+0000 7f09ccae5700 1 -- 172.21.15.7:0/4042509025 wait complete. 2023-03-03T09:48:34.522 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:48:34.532 INFO:tasks.mds_thrash.fs.[a]:waiting for 39 secs before thrashing 2023-03-03T09:48:45.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:48:45.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.994+0000 7fe12f335700 1 Processor -- start 2023-03-03T09:48:45.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.994+0000 7fe12f335700 1 -- start start 2023-03-03T09:48:45.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.995+0000 7fe12f335700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1280aad90 0x7fe12806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:45.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.995+0000 7fe12f335700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806b920 0x7fe12806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:45.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.995+0000 7fe12f335700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe12806eda0 con 0x7fe12806bfe0 2023-03-03T09:48:45.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.995+0000 7fe12f335700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12806f080 con 0x7fe1280aad90 2023-03-03T09:48:45.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.995+0000 7fe12f335700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281584b0 con 0x7fe12806b920 2023-03-03T09:48:45.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.995+0000 7fe12d0d1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1280aad90 0x7fe12806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:45.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.995+0000 7fe12d0d1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1280aad90 0x7fe12806e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57730/0 (socket says 172.21.15.7:57730) 2023-03-03T09:48:45.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.995+0000 7fe12d0d1700 1 -- 172.21.15.7:0/1496559008 learned_addr learned my addr 172.21.15.7:0/1496559008 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:48:45.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.995+0000 7fe12c8d0700 1 --2- 172.21.15.7:0/1496559008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806b920 0x7fe12806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:45.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.996+0000 7fe12d0d1700 1 -- 172.21.15.7:0/1496559008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806b920 msgr2=0x7fe12806a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:48:45.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.996+0000 7fe12d0d1700 1 --2- 172.21.15.7:0/1496559008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806b920 0x7fe12806a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:45.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.996+0000 7fe12d0d1700 1 -- 172.21.15.7:0/1496559008 >> v1:172.21.15.143:6789/0 conn(0x7fe12806bfe0 legacy=0x7fe12806a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:46.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.996+0000 7fe12d0d1700 1 -- 172.21.15.7:0/1496559008 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1281586a0 con 0x7fe1280aad90 2023-03-03T09:48:46.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.996+0000 7fe12d0d1700 1 --2- 172.21.15.7:0/1496559008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 0x7fe12806e2f0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe1180098f0 tx=0x7fe118009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18305708ba831e11 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:46.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.996+0000 7fe11ffff700 1 -- 172.21.15.7:0/1496559008 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118011830 con 0x7fe1280aad90 2023-03-03T09:48:46.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.996+0000 7fe11ffff700 1 -- 172.21.15.7:0/1496559008 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe118011990 con 0x7fe1280aad90 2023-03-03T09:48:46.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.996+0000 7fe11ffff700 1 -- 172.21.15.7:0/1496559008 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118011d20 con 0x7fe1280aad90 2023-03-03T09:48:46.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.997+0000 7fe12f335700 1 -- 172.21.15.7:0/1496559008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 msgr2=0x7fe12806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:46.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.997+0000 7fe12f335700 1 --2- 172.21.15.7:0/1496559008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 0x7fe12806e2f0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe1180098f0 tx=0x7fe118009c00 comp rx=0 tx=0).stop 2023-03-03T09:48:46.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.997+0000 7fe12f335700 1 -- 172.21.15.7:0/1496559008 shutdown_connections 2023-03-03T09:48:46.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.997+0000 7fe12f335700 1 --2- 172.21.15.7:0/1496559008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 0x7fe12806e2f0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:46.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.997+0000 7fe12f335700 1 --2- 172.21.15.7:0/1496559008 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806b920 0x7fe12806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:46.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.997+0000 7fe12f335700 1 -- 172.21.15.7:0/1496559008 >> 172.21.15.7:0/1496559008 conn(0x7fe1280736f0 msgr2=0x7fe128073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:48:46.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.997+0000 7fe12f335700 1 -- 172.21.15.7:0/1496559008 shutdown_connections 2023-03-03T09:48:46.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.997+0000 7fe12f335700 1 -- 172.21.15.7:0/1496559008 wait complete. 2023-03-03T09:48:46.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.998+0000 7fe12f335700 1 Processor -- start 2023-03-03T09:48:46.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.998+0000 7fe12f335700 1 -- start start 2023-03-03T09:48:46.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12f335700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12806b920 0x7fe1280c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:46.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12f335700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806bfe0 0x7fe1281ed330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:46.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12f335700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 0x7fe1281ed870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:46.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12f335700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1280abb00 con 0x7fe1280aad90 2023-03-03T09:48:46.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12f335700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1280acd30 con 0x7fe12806b920 2023-03-03T09:48:46.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12f335700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1280adcf0 con 0x7fe12806bfe0 2023-03-03T09:48:46.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d8d2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 0x7fe1281ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:46.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d8d2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 0x7fe1281ed870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57740/0 (socket says 172.21.15.7:57740) 2023-03-03T09:48:46.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d8d2700 1 -- 172.21.15.7:0/288798024 learned_addr learned my addr 172.21.15.7:0/288798024 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:48:46.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12c8d0700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806bfe0 0x7fe1281ed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:46.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d0d1700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12806b920 0x7fe1280c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:46.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d8d2700 1 -- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806bfe0 msgr2=0x7fe1281ed330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:46.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d8d2700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806bfe0 0x7fe1281ed330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:46.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d8d2700 1 -- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12806b920 msgr2=0x7fe1280c5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:46.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d8d2700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12806b920 0x7fe1280c5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:46.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d8d2700 1 -- 172.21.15.7:0/288798024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118009550 con 0x7fe1280aad90 2023-03-03T09:48:46.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12c8d0700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806bfe0 0x7fe1281ed330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:48:46.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d0d1700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12806b920 0x7fe1280c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:48:46.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:45.999+0000 7fe12d8d2700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 0x7fe1281ed870 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe12000cbf0 tx=0x7fe1200021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:46.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.000+0000 7fe11dffb700 1 -- 172.21.15.7:0/288798024 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe120003c00 con 0x7fe1280aad90 2023-03-03T09:48:46.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.000+0000 7fe11dffb700 1 -- 172.21.15.7:0/288798024 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe120003d60 con 0x7fe1280aad90 2023-03-03T09:48:46.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.000+0000 7fe12f335700 1 -- 172.21.15.7:0/288798024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe128071cd0 con 0x7fe1280aad90 2023-03-03T09:48:46.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.000+0000 7fe11dffb700 1 -- 172.21.15.7:0/288798024 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12000ab80 con 0x7fe1280aad90 2023-03-03T09:48:46.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.000+0000 7fe12f335700 1 -- 172.21.15.7:0/288798024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe128158980 con 0x7fe1280aad90 2023-03-03T09:48:46.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.001+0000 7fe12f335700 1 -- 172.21.15.7:0/288798024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe128071270 con 0x7fe1280aad90 2023-03-03T09:48:46.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.001+0000 7fe11dffb700 1 -- 172.21.15.7:0/288798024 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe120010050 con 0x7fe1280aad90 2023-03-03T09:48:46.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.002+0000 7fe11dffb700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 0x7fe10c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:48:46.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.002+0000 7fe11dffb700 1 -- 172.21.15.7:0/288798024 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe120086010 con 0x7fe1280aad90 2023-03-03T09:48:46.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.002+0000 7fe12d0d1700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 0x7fe10c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:48:46.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.003+0000 7fe12d0d1700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 0x7fe10c06a7a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe118000c00 tx=0x7fe118008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:48:46.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.006+0000 7fe11dffb700 1 -- 172.21.15.7:0/288798024 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe1200523b0 con 0x7fe1280aad90 2023-03-03T09:48:46.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.243+0000 7fe12f335700 1 -- 172.21.15.7:0/288798024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe128074bb0 con 0x7fe1280aad90 2023-03-03T09:48:46.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.244+0000 7fe11dffb700 1 -- 172.21.15.7:0/288798024 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe1200559d0 con 0x7fe1280aad90 2023-03-03T09:48:46.246 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:48:46.246 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:48:46.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 -- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 msgr2=0x7fe10c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:46.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 0x7fe10c06a7a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe118000c00 tx=0x7fe118008030 comp rx=0 tx=0).stop 2023-03-03T09:48:46.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 -- 172.21.15.7:0/288798024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 msgr2=0x7fe1281ed870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:48:46.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 0x7fe1281ed870 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe12000cbf0 tx=0x7fe1200021c0 comp rx=0 tx=0).stop 2023-03-03T09:48:46.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 -- 172.21.15.7:0/288798024 shutdown_connections 2023-03-03T09:48:46.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 0x7fe10c06a7a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:46.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1280aad90 0x7fe1281ed870 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:46.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12806b920 0x7fe1280c5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:46.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 --2- 172.21.15.7:0/288798024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe12806bfe0 0x7fe1281ed330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:48:46.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 -- 172.21.15.7:0/288798024 >> 172.21.15.7:0/288798024 conn(0x7fe1280736f0 msgr2=0x7fe1280c4480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:48:46.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 -- 172.21.15.7:0/288798024 shutdown_connections 2023-03-03T09:48:46.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:48:46.246+0000 7fe12f335700 1 -- 172.21.15.7:0/288798024 wait complete. 2023-03-03T09:48:46.250 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:48:46.260 INFO:tasks.mds_thrash.fs.[b]:waiting for 105 secs before thrashing 2023-03-03T09:48:50.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:48:50.263 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:13.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:49:13.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.670+0000 7fda9f7ec700 1 Processor -- start 2023-03-03T09:49:13.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.671+0000 7fda9f7ec700 1 -- start start 2023-03-03T09:49:13.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.671+0000 7fda9f7ec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda9814b460 0x7fda9806aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.671+0000 7fda9f7ec700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda98070360 con 0x7fda98071990 2023-03-03T09:49:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.671+0000 7fda9f7ec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9806a5e0 con 0x7fda98072470 2023-03-03T09:49:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.671+0000 7fda9f7ec700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda981521a0 con 0x7fda9814b460 2023-03-03T09:49:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.671+0000 7fda9d588700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fda98071990 0x7fda98071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51752/0 (socket says 172.21.15.7:51752) 2023-03-03T09:49:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda9d588700 1 -- 172.21.15.7:0/1771734614 learned_addr learned my addr 172.21.15.7:0/1771734614 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:49:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760278039 0 0) 0x7fda98070360 con 0x7fda98071990 2023-03-03T09:49:13.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda9cd87700 1 --2- 172.21.15.7:0/1771734614 >> v2:172.21.15.143:3300/0 conn(0x7fda9814b460 0x7fda9806aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:13.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda840035b0 con 0x7fda98071990 2023-03-03T09:49:13.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4263331726 0 0) 0x7fda840035b0 con 0x7fda98071990 2023-03-03T09:49:13.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 >> v2:172.21.15.143:3300/0 conn(0x7fda9814b460 msgr2=0x7fda9806aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:49:13.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 --2- 172.21.15.7:0/1771734614 >> v2:172.21.15.143:3300/0 conn(0x7fda9814b460 0x7fda9806aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 >> v1:172.21.15.143:6789/0 conn(0x7fda98072470 legacy=0x7fda9806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:13.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda980c38b0 con 0x7fda98071990 2023-03-03T09:49:13.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fda94001670 con 0x7fda98071990 2023-03-03T09:49:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 >> v1:172.21.15.7:6789/0 conn(0x7fda98071990 legacy=0x7fda98071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84003a70 0x7fda84003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda84004850 0x7fda84004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda84005250 0x7fda8400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda981521a0 con 0x7fda84005250 2023-03-03T09:49:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8400c190 con 0x7fda84004850 2023-03-03T09:49:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8400c350 con 0x7fda84003a70 2023-03-03T09:49:13.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda9dd89700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda84005250 0x7fda8400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:13.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda9cd87700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda84004850 0x7fda84004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:13.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.672+0000 7fda9d588700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84003a70 0x7fda84003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:13.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9dd89700 1 -- 172.21.15.7:0/1771734614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84003a70 msgr2=0x7fda84003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:49:13.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9dd89700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84003a70 0x7fda84003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9dd89700 1 -- 172.21.15.7:0/1771734614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda84004850 msgr2=0x7fda84004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:49:13.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9dd89700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda84004850 0x7fda84004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9dd89700 1 -- 172.21.15.7:0/1771734614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda840035b0 con 0x7fda84005250 2023-03-03T09:49:13.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9d588700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84003a70 0x7fda84003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:49:13.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9dd89700 1 -- 172.21.15.7:0/1771734614 reap_dead start 2023-03-03T09:49:13.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9dd89700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda84005250 0x7fda8400ba10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fda9000c160 tx=0x7fda84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77586e79c01a1fee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:49:13.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90001db0 con 0x7fda84005250 2023-03-03T09:49:13.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9000cca0 con 0x7fda84005250 2023-03-03T09:49:13.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda8bfff700 1 -- 172.21.15.7:0/1771734614 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90003c20 con 0x7fda84005250 2023-03-03T09:49:13.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1771734614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda84005250 msgr2=0x7fda8400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:13.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9f7ec700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda84005250 0x7fda8400ba10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fda9000c160 tx=0x7fda84005880 comp rx=0 tx=0).stop 2023-03-03T09:49:13.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1771734614 shutdown_connections 2023-03-03T09:49:13.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9f7ec700 1 --2- 172.21.15.7:0/1771734614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda84005250 0x7fda8400ba10 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.673+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1771734614 >> 172.21.15.7:0/1771734614 conn(0x7fda980a94d0 msgr2=0x7fda980a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:49:13.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.674+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1771734614 shutdown_connections 2023-03-03T09:49:13.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.674+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1771734614 wait complete. 2023-03-03T09:49:13.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.675+0000 7fda9f7ec700 1 Processor -- start 2023-03-03T09:49:13.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.675+0000 7fda9f7ec700 1 -- start start 2023-03-03T09:49:13.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.675+0000 7fda9f7ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84005250 0x7fda981e7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:13.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.675+0000 7fda9f7ec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda981e7680 0x7fda98150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:13.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.675+0000 7fda9f7ec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda98150960 0x7fda981ea2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:13.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.675+0000 7fda9f7ec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980b9e70 con 0x7fda98150960 2023-03-03T09:49:13.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.675+0000 7fda9f7ec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980ba460 con 0x7fda981e7680 2023-03-03T09:49:13.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.675+0000 7fda9f7ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda980ba740 con 0x7fda84005250 2023-03-03T09:49:13.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9dd89700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda98150960 0x7fda981ea2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:13.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9dd89700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda98150960 0x7fda981ea2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57786/0 (socket says 172.21.15.7:57786) 2023-03-03T09:49:13.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9dd89700 1 -- 172.21.15.7:0/1216174920 learned_addr learned my addr 172.21.15.7:0/1216174920 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:49:13.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9d588700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84005250 0x7fda981e7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:13.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9cd87700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda981e7680 0x7fda98150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:13.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9dd89700 1 -- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84005250 msgr2=0x7fda981e7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:13.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9dd89700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84005250 0x7fda981e7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9dd89700 1 -- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda981e7680 msgr2=0x7fda98150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:13.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9dd89700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda981e7680 0x7fda98150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9dd89700 1 -- 172.21.15.7:0/1216174920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda980c3b50 con 0x7fda98150960 2023-03-03T09:49:13.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9d588700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84005250 0x7fda981e7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:49:13.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9dd89700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda98150960 0x7fda981ea2f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fda9000c160 tx=0x7fda900014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:49:13.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda89ffb700 1 -- 172.21.15.7:0/1216174920 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90005270 con 0x7fda98150960 2023-03-03T09:49:13.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda89ffb700 1 -- 172.21.15.7:0/1216174920 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda900053d0 con 0x7fda98150960 2023-03-03T09:49:13.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda89ffb700 1 -- 172.21.15.7:0/1216174920 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90005780 con 0x7fda98150960 2023-03-03T09:49:13.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1216174920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda9806a2b0 con 0x7fda98150960 2023-03-03T09:49:13.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.676+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1216174920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda98151c00 con 0x7fda98150960 2023-03-03T09:49:13.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.678+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1216174920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda981ea140 con 0x7fda98150960 2023-03-03T09:49:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.678+0000 7fda89ffb700 1 -- 172.21.15.7:0/1216174920 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda90026030 con 0x7fda98150960 2023-03-03T09:49:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.678+0000 7fda89ffb700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda7c0682f0 0x7fda7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.679+0000 7fda89ffb700 1 -- 172.21.15.7:0/1216174920 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fda90086e40 con 0x7fda98150960 2023-03-03T09:49:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.679+0000 7fda9d588700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda7c0682f0 0x7fda7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.679+0000 7fda9d588700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda7c0682f0 0x7fda7c06a7a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fda94001eb0 tx=0x7fda94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:49:13.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.683+0000 7fda89ffb700 1 -- 172.21.15.7:0/1216174920 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda900530f0 con 0x7fda98150960 2023-03-03T09:49:13.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.922+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1216174920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda98074bb0 con 0x7fda98150960 2023-03-03T09:49:13.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.923+0000 7fda89ffb700 1 -- 172.21.15.7:0/1216174920 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fda90016090 con 0x7fda98150960 2023-03-03T09:49:13.924 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:49:13.925 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:49:13.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda7c0682f0 msgr2=0x7fda7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:13.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda7c0682f0 0x7fda7c06a7a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fda94001eb0 tx=0x7fda94009040 comp rx=0 tx=0).stop 2023-03-03T09:49:13.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1216174920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda98150960 msgr2=0x7fda981ea2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:13.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda98150960 0x7fda981ea2f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fda9000c160 tx=0x7fda900014a0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1216174920 shutdown_connections 2023-03-03T09:49:13.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda7c0682f0 0x7fda7c06a7a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda98150960 0x7fda981ea2f0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda84005250 0x7fda981e7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 --2- 172.21.15.7:0/1216174920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda981e7680 0x7fda98150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:13.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1216174920 >> 172.21.15.7:0/1216174920 conn(0x7fda980a94d0 msgr2=0x7fda980c1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:49:13.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1216174920 shutdown_connections 2023-03-03T09:49:13.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:13.925+0000 7fda9f7ec700 1 -- 172.21.15.7:0/1216174920 wait complete. 2023-03-03T09:49:13.928 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:49:13.938 INFO:tasks.mds_thrash.fs.[a]:waiting for 18 secs before thrashing 2023-03-03T09:49:20.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:20.334 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:31.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:49:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.077+0000 7f8c03de5700 1 Processor -- start 2023-03-03T09:49:32.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.078+0000 7f8c03de5700 1 -- start start 2023-03-03T09:49:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.078+0000 7f8c03de5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc151960 0x7f8bfc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.078+0000 7f8c03de5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc154ee0 con 0x7f8bfc0c3ac0 2023-03-03T09:49:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.078+0000 7f8c03de5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0c3f30 con 0x7f8bfc0723f0 2023-03-03T09:49:32.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.078+0000 7f8c03de5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1545f0 con 0x7f8bfc151960 2023-03-03T09:49:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c01380700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8bfc0c3ac0 0x7f8bfc1579f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51788/0 (socket says 172.21.15.7:51788) 2023-03-03T09:49:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c01380700 1 -- 172.21.15.7:0/542868410 learned_addr learned my addr 172.21.15.7:0/542868410 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:49:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c01b81700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc151960 0x7f8bfc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659600021 0 0) 0x7f8bfc154ee0 con 0x7f8bfc0c3ac0 2023-03-03T09:49:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be80035b0 con 0x7f8bfc0c3ac0 2023-03-03T09:49:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812926846 0 0) 0x7f8bfc0c3f30 con 0x7f8bfc0723f0 2023-03-03T09:49:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bfc154ee0 con 0x7f8bfc0723f0 2023-03-03T09:49:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2715961021 0 0) 0x7f8be80035b0 con 0x7f8bfc0c3ac0 2023-03-03T09:49:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc151960 msgr2=0x7f8bfc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:49:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc151960 0x7f8bfc0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 >> v1:172.21.15.143:6789/0 conn(0x7f8bfc0723f0 legacy=0x7f8bfc15edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc155370 con 0x7f8bfc0c3ac0 2023-03-03T09:49:32.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8bf00015f0 con 0x7f8bfc0c3ac0 2023-03-03T09:49:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 >> v1:172.21.15.7:6789/0 conn(0x7f8bfc0c3ac0 legacy=0x7f8bfc1579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8be8003bc0 0x7f8be8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be8004850 0x7f8be8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc151960 0x7f8be8005260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1545f0 con 0x7f8be8003bc0 2023-03-03T09:49:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be800bba0 con 0x7f8be8004850 2023-03-03T09:49:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be800bd60 con 0x7f8bfc151960 2023-03-03T09:49:32.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.079+0000 7f8c01b81700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8be8003bc0 0x7f8be8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c01380700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be8004850 0x7f8be8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c02382700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc151960 0x7f8be8005260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c01b81700 1 -- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc151960 msgr2=0x7f8be8005260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:49:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c01b81700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc151960 0x7f8be8005260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c01b81700 1 -- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be8004850 msgr2=0x7f8be8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:49:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c01b81700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8be8004850 0x7f8be8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c01b81700 1 -- 172.21.15.7:0/542868410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be80035b0 con 0x7f8be8003bc0 2023-03-03T09:49:32.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c01b81700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8be8003bc0 0x7f8be8003fb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009b60 tx=0x7f8be8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa188314d6314e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:49:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf800e6c0 con 0x7f8be8003bc0 2023-03-03T09:49:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf800e820 con 0x7f8be8003bc0 2023-03-03T09:49:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.080+0000 7f8c00b7f700 1 -- 172.21.15.7:0/542868410 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf800eb90 con 0x7f8be8003bc0 2023-03-03T09:49:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.081+0000 7f8c03de5700 1 -- 172.21.15.7:0/542868410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8be8003bc0 msgr2=0x7f8be8003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.081+0000 7f8c03de5700 1 --2- 172.21.15.7:0/542868410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8be8003bc0 0x7f8be8003fb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009b60 tx=0x7f8be8005880 comp rx=0 tx=0).stop 2023-03-03T09:49:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.081+0000 7f8c02382700 1 -- 172.21.15.7:0/542868410 reap_dead start 2023-03-03T09:49:32.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.081+0000 7f8c03de5700 1 -- 172.21.15.7:0/542868410 shutdown_connections 2023-03-03T09:49:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.081+0000 7f8c03de5700 1 -- 172.21.15.7:0/542868410 >> 172.21.15.7:0/542868410 conn(0x7f8bfc06c060 msgr2=0x7f8bfc06c460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:49:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.081+0000 7f8c03de5700 1 -- 172.21.15.7:0/542868410 shutdown_connections 2023-03-03T09:49:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.081+0000 7f8c03de5700 1 -- 172.21.15.7:0/542868410 wait complete. 2023-03-03T09:49:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.082+0000 7f8c03de5700 1 Processor -- start 2023-03-03T09:49:32.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.082+0000 7f8c03de5700 1 -- start start 2023-03-03T09:49:32.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c03de5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be8003bc0 0x7f8bfc14a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:32.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c03de5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc1535d0 0x7f8bfc0aa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:32.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c03de5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0aa920 0x7f8bfc145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:32.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c03de5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc146e40 con 0x7f8bfc0aa920 2023-03-03T09:49:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c03de5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1ede80 con 0x7f8bfc1535d0 2023-03-03T09:49:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c03de5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1ed0c0 con 0x7f8be8003bc0 2023-03-03T09:49:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c02382700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0aa920 0x7f8bfc145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c02382700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0aa920 0x7f8bfc145e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57824/0 (socket says 172.21.15.7:57824) 2023-03-03T09:49:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c02382700 1 -- 172.21.15.7:0/2821629157 learned_addr learned my addr 172.21.15.7:0/2821629157 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:49:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c01b81700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be8003bc0 0x7f8bfc14a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c01380700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc1535d0 0x7f8bfc0aa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:32.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c02382700 1 -- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be8003bc0 msgr2=0x7f8bfc14a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c02382700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be8003bc0 0x7f8bfc14a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c02382700 1 -- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc1535d0 msgr2=0x7f8bfc0aa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c02382700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc1535d0 0x7f8bfc0aa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c02382700 1 -- 172.21.15.7:0/2821629157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc154b70 con 0x7f8bfc0aa920 2023-03-03T09:49:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c01380700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc1535d0 0x7f8bfc0aa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:49:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c02382700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0aa920 0x7f8bfc145e40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0c4f00 tx=0x7f8bf400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:49:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8bee7fc700 1 -- 172.21.15.7:0/2821629157 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4003c10 con 0x7f8bfc0aa920 2023-03-03T09:49:32.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8bee7fc700 1 -- 172.21.15.7:0/2821629157 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf4003d90 con 0x7f8bfc0aa920 2023-03-03T09:49:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8bee7fc700 1 -- 172.21.15.7:0/2821629157 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4005310 con 0x7f8bfc0aa920 2023-03-03T09:49:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.083+0000 7f8c03de5700 1 -- 172.21.15.7:0/2821629157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bfc154890 con 0x7f8bfc0aa920 2023-03-03T09:49:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.084+0000 7f8c03de5700 1 -- 172.21.15.7:0/2821629157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc151d30 con 0x7f8bfc0aa920 2023-03-03T09:49:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.084+0000 7f8c03de5700 1 -- 172.21.15.7:0/2821629157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bfc1edcd0 con 0x7f8bfc0aa920 2023-03-03T09:49:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.085+0000 7f8bee7fc700 1 -- 172.21.15.7:0/2821629157 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8bf4005470 con 0x7f8bfc0aa920 2023-03-03T09:49:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.085+0000 7f8bee7fc700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be00682f0 0x7f8be006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:49:32.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.086+0000 7f8bee7fc700 1 -- 172.21.15.7:0/2821629157 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8bf40863c0 con 0x7f8bfc0aa920 2023-03-03T09:49:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.086+0000 7f8c01b81700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be00682f0 0x7f8be006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:49:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.086+0000 7f8c01b81700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be00682f0 0x7f8be006a7a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80090b0 tx=0x7f8bf801f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:49:32.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.090+0000 7f8bee7fc700 1 -- 172.21.15.7:0/2821629157 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8bf4052670 con 0x7f8bfc0aa920 2023-03-03T09:49:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.327+0000 7f8c03de5700 1 -- 172.21.15.7:0/2821629157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bfc074bb0 con 0x7f8bfc0aa920 2023-03-03T09:49:32.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.328+0000 7f8bee7fc700 1 -- 172.21.15.7:0/2821629157 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8bf4055c90 con 0x7f8bfc0aa920 2023-03-03T09:49:32.329 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:49:32.329 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:49:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 -- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be00682f0 msgr2=0x7f8be006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be00682f0 0x7f8be006a7a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80090b0 tx=0x7f8bf801f040 comp rx=0 tx=0).stop 2023-03-03T09:49:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 -- 172.21.15.7:0/2821629157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0aa920 msgr2=0x7f8bfc145e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:49:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0aa920 0x7f8bfc145e40 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc0c4f00 tx=0x7f8bf400ee70 comp rx=0 tx=0).stop 2023-03-03T09:49:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 -- 172.21.15.7:0/2821629157 shutdown_connections 2023-03-03T09:49:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be00682f0 0x7f8be006a7a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0aa920 0x7f8bfc145e40 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8be8003bc0 0x7f8bfc14a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 --2- 172.21.15.7:0/2821629157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc1535d0 0x7f8bfc0aa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:49:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 -- 172.21.15.7:0/2821629157 >> 172.21.15.7:0/2821629157 conn(0x7f8bfc06c060 msgr2=0x7f8bfc06a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:49:32.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 -- 172.21.15.7:0/2821629157 shutdown_connections 2023-03-03T09:49:32.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:49:32.330+0000 7f8c03de5700 1 -- 172.21.15.7:0/2821629157 wait complete. 2023-03-03T09:49:32.333 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:49:32.343 INFO:tasks.mds_thrash.fs.[a]:waiting for 112 secs before thrashing 2023-03-03T09:49:50.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:49:50.533 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T09:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T09:50:20.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:20.652 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:31.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:50:31.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.399+0000 7f1bd4f63700 1 Processor -- start 2023-03-03T09:50:31.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.399+0000 7f1bd4f63700 1 -- start start 2023-03-03T09:50:31.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.400+0000 7f1bd4f63700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1bd014fda0 0x7f1bd0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:31.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.400+0000 7f1bd4f63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bd00abe20 0x7f1bd01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:31.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.400+0000 7f1bd4f63700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd00c2f30 con 0x7f1bd014f9d0 2023-03-03T09:50:31.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.400+0000 7f1bd4f63700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd00c31d0 con 0x7f1bd014fda0 2023-03-03T09:50:31.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.400+0000 7f1bd4f63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd01524a0 con 0x7f1bd00abe20 2023-03-03T09:50:31.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bce59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1bd014f9d0 0x7f1bd0071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51868/0 (socket says 172.21.15.7:51868) 2023-03-03T09:50:31.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bce59c700 1 -- 172.21.15.7:0/2745155501 learned_addr learned my addr 172.21.15.7:0/2745155501 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:50:31.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcdd9b700 1 --2- 172.21.15.7:0/2745155501 >> v2:172.21.15.143:3300/0 conn(0x7f1bd014fda0 0x7f1bd0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:31.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bced9d700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bd00abe20 0x7f1bd01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:31.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381033476 0 0) 0x7f1bd00c2f30 con 0x7f1bd014f9d0 2023-03-03T09:50:31.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb80035b0 con 0x7f1bd014f9d0 2023-03-03T09:50:31.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1413931919 0 0) 0x7f1bb80035b0 con 0x7f1bd014f9d0 2023-03-03T09:50:31.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bd00abe20 msgr2=0x7f1bd01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:50:31.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bd00abe20 0x7f1bd01529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:31.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 >> v2:172.21.15.143:3300/0 conn(0x7f1bd014fda0 msgr2=0x7f1bd0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:50:31.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 --2- 172.21.15.7:0/2745155501 >> v2:172.21.15.143:3300/0 conn(0x7f1bd014fda0 0x7f1bd0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:31.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd00c34b0 con 0x7f1bd014f9d0 2023-03-03T09:50:31.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f1bc4001670 con 0x7f1bd014f9d0 2023-03-03T09:50:31.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 >> v1:172.21.15.7:6789/0 conn(0x7f1bd014f9d0 legacy=0x7f1bd0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:31.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8003a70 0x7f1bb8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:31.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bb8004850 0x7f1bb8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:31.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bd00abe20 0x7f1bb8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:31.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd00c31d0 con 0x7f1bb8003a70 2023-03-03T09:50:31.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd01524a0 con 0x7f1bb8004850 2023-03-03T09:50:31.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb800bb40 con 0x7f1bd00abe20 2023-03-03T09:50:31.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bce59c700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8003a70 0x7f1bb8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:31.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bcdd9b700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bb8004850 0x7f1bb8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:31.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.401+0000 7f1bced9d700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bd00abe20 0x7f1bb8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:31.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bce59c700 1 -- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bd00abe20 msgr2=0x7f1bb8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:50:31.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bce59c700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bd00abe20 0x7f1bb8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:31.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bce59c700 1 -- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bb8004850 msgr2=0x7f1bb8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:50:31.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bce59c700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bb8004850 0x7f1bb8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:31.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bce59c700 1 -- 172.21.15.7:0/2745155501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb80035b0 con 0x7f1bb8003a70 2023-03-03T09:50:31.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bce59c700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8003a70 0x7f1bb8003e80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400c370 tx=0x7f1bb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b713640fd7d1e59b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:50:31.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4007c60 con 0x7f1bb8003a70 2023-03-03T09:50:31.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc4005960 con 0x7f1bb8003a70 2023-03-03T09:50:31.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bcd59a700 1 -- 172.21.15.7:0/2745155501 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4005d10 con 0x7f1bb8003a70 2023-03-03T09:50:31.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bd4f63700 1 -- 172.21.15.7:0/2745155501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8003a70 msgr2=0x7f1bb8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:31.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bd4f63700 1 --2- 172.21.15.7:0/2745155501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8003a70 0x7f1bb8003e80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400c370 tx=0x7f1bb8005880 comp rx=0 tx=0).stop 2023-03-03T09:50:31.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.402+0000 7f1bced9d700 1 -- 172.21.15.7:0/2745155501 reap_dead start 2023-03-03T09:50:31.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.403+0000 7f1bd4f63700 1 -- 172.21.15.7:0/2745155501 shutdown_connections 2023-03-03T09:50:31.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.403+0000 7f1bd4f63700 1 -- 172.21.15.7:0/2745155501 >> 172.21.15.7:0/2745155501 conn(0x7f1bd00c8900 msgr2=0x7f1bd00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:50:31.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.403+0000 7f1bd4f63700 1 -- 172.21.15.7:0/2745155501 shutdown_connections 2023-03-03T09:50:31.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.403+0000 7f1bd4f63700 1 -- 172.21.15.7:0/2745155501 wait complete. 2023-03-03T09:50:31.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.404+0000 7f1bd4f63700 1 Processor -- start 2023-03-03T09:50:31.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.404+0000 7f1bd4f63700 1 -- start start 2023-03-03T09:50:31.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bd4f63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bb8003a70 0x7f1bd01e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:31.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bd4f63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd01dc0d0 0x7f1bd01548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:31.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bd4f63700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bd0154e20 0x7f1bd014b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:31.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bd4f63700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd01dcf90 con 0x7f1bd0154e20 2023-03-03T09:50:31.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bd4f63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd01eb960 con 0x7f1bd01dc0d0 2023-03-03T09:50:31.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bd4f63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd01ebc40 con 0x7f1bb8003a70 2023-03-03T09:50:31.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bced9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bd0154e20 0x7f1bd014b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:31.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bced9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bd0154e20 0x7f1bd014b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57904/0 (socket says 172.21.15.7:57904) 2023-03-03T09:50:31.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bced9d700 1 -- 172.21.15.7:0/3310897721 learned_addr learned my addr 172.21.15.7:0/3310897721 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:50:31.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bcdd9b700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd01dc0d0 0x7f1bd01548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:31.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bce59c700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bb8003a70 0x7f1bd01e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:31.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bced9d700 1 -- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bb8003a70 msgr2=0x7f1bd01e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:31.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bced9d700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bb8003a70 0x7f1bd01e8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:31.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bced9d700 1 -- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd01dc0d0 msgr2=0x7f1bd01548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:31.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bced9d700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd01dc0d0 0x7f1bd01548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:31.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bced9d700 1 -- 172.21.15.7:0/3310897721 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd00ac350 con 0x7f1bd0154e20 2023-03-03T09:50:31.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bcdd9b700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd01dc0d0 0x7f1bd01548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:50:31.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bced9d700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bd0154e20 0x7f1bd014b190 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1bd00ab150 tx=0x7f1bc8011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:50:31.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bbeffd700 1 -- 172.21.15.7:0/3310897721 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc800e040 con 0x7f1bd0154e20 2023-03-03T09:50:31.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.405+0000 7f1bbeffd700 1 -- 172.21.15.7:0/3310897721 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc800f040 con 0x7f1bd0154e20 2023-03-03T09:50:31.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.406+0000 7f1bbeffd700 1 -- 172.21.15.7:0/3310897721 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc800fdb0 con 0x7f1bd0154e20 2023-03-03T09:50:31.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.406+0000 7f1bd4f63700 1 -- 172.21.15.7:0/3310897721 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd00ab630 con 0x7f1bd0154e20 2023-03-03T09:50:31.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.406+0000 7f1bd4f63700 1 -- 172.21.15.7:0/3310897721 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd0071fa0 con 0x7f1bd0154e20 2023-03-03T09:50:31.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.406+0000 7f1bd4f63700 1 -- 172.21.15.7:0/3310897721 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd00c7c70 con 0x7f1bd0154e20 2023-03-03T09:50:31.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.407+0000 7f1bbeffd700 1 -- 172.21.15.7:0/3310897721 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1bc8015430 con 0x7f1bd0154e20 2023-03-03T09:50:31.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.407+0000 7f1bbeffd700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bb00682f0 0x7f1bb006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:31.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.407+0000 7f1bbeffd700 1 -- 172.21.15.7:0/3310897721 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1bc8086cf0 con 0x7f1bd0154e20 2023-03-03T09:50:31.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.408+0000 7f1bce59c700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bb00682f0 0x7f1bb006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:31.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.408+0000 7f1bce59c700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bb00682f0 0x7f1bb006a7a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0151c50 tx=0x7f1bc400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:50:31.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.412+0000 7f1bbeffd700 1 -- 172.21.15.7:0/3310897721 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1bc8052f40 con 0x7f1bd0154e20 2023-03-03T09:50:31.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.649+0000 7f1bd4f63700 1 -- 172.21.15.7:0/3310897721 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bd0074bb0 con 0x7f1bd0154e20 2023-03-03T09:50:31.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.650+0000 7f1bbeffd700 1 -- 172.21.15.7:0/3310897721 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1bc8056560 con 0x7f1bd0154e20 2023-03-03T09:50:31.651 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:50:31.651 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:50:31.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 -- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bb00682f0 msgr2=0x7f1bb006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:31.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bb00682f0 0x7f1bb006a7a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0151c50 tx=0x7f1bc400a000 comp rx=0 tx=0).stop 2023-03-03T09:50:31.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 -- 172.21.15.7:0/3310897721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bd0154e20 msgr2=0x7f1bd014b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:31.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bd0154e20 0x7f1bd014b190 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1bd00ab150 tx=0x7f1bc8011f80 comp rx=0 tx=0).stop 2023-03-03T09:50:31.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 -- 172.21.15.7:0/3310897721 shutdown_connections 2023-03-03T09:50:31.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bb00682f0 0x7f1bb006a7a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:31.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bd0154e20 0x7f1bd014b190 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:31.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bb8003a70 0x7f1bd01e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:31.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 --2- 172.21.15.7:0/3310897721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd01dc0d0 0x7f1bd01548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:31.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 -- 172.21.15.7:0/3310897721 >> 172.21.15.7:0/3310897721 conn(0x7f1bd00c8900 msgr2=0x7f1bd0151480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:50:31.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 -- 172.21.15.7:0/3310897721 shutdown_connections 2023-03-03T09:50:31.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:31.652+0000 7f1bd4f63700 1 -- 172.21.15.7:0/3310897721 wait complete. 2023-03-03T09:50:31.655 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:50:31.665 INFO:tasks.mds_thrash.fs.[b]:waiting for 23 secs before thrashing 2023-03-03T09:50:50.721 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:50.724 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:50:54.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:50:54.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.806+0000 7f07de9a0700 1 Processor -- start 2023-03-03T09:50:54.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.806+0000 7f07de9a0700 1 -- start start 2023-03-03T09:50:54.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.807+0000 7f07de9a0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8151770 0x7f07d8151c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:54.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.807+0000 7f07de9a0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d8154910 con 0x7f07d8155390 2023-03-03T09:50:54.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.807+0000 7f07de9a0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d80c4540 con 0x7f07d806e020 2023-03-03T09:50:54.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.807+0000 7f07de9a0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8070b90 con 0x7f07d8151770 2023-03-03T09:50:54.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.807+0000 7f07d7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f07d8155390 0x7f07d80c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51900/0 (socket says 172.21.15.7:51900) 2023-03-03T09:50:54.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.807+0000 7f07d7fff700 1 -- 172.21.15.7:0/196572177 learned_addr learned my addr 172.21.15.7:0/196572177 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:50:54.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.807+0000 7f07dcf3d700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8151770 0x7f07d8151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:54.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.807+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809170517 0 0) 0x7f07d8154910 con 0x7f07d8155390 2023-03-03T09:50:54.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.807+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07c40035b0 con 0x7f07d8155390 2023-03-03T09:50:54.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912004165 0 0) 0x7f07c40035b0 con 0x7f07d8155390 2023-03-03T09:50:54.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8151770 msgr2=0x7f07d8151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:50:54.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8151770 0x7f07d8151c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:54.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 >> v1:172.21.15.143:6789/0 conn(0x7f07d806e020 legacy=0x7f07d8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:54.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d80702a0 con 0x7f07d8155390 2023-03-03T09:50:54.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f07c8001670 con 0x7f07d8155390 2023-03-03T09:50:54.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 >> v1:172.21.15.7:6789/0 conn(0x7f07d8155390 legacy=0x7f07d80c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:54.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07c4003aa0 0x7f07c4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:54.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c4004850 0x7f07c4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:54.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8151770 0x7f07c4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:54.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8070b90 con 0x7f07c4003aa0 2023-03-03T09:50:54.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c400baf0 con 0x7f07c4004850 2023-03-03T09:50:54.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c400bcb0 con 0x7f07d8151770 2023-03-03T09:50:54.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d7fff700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07c4003aa0 0x7f07c4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:54.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d77fe700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c4004850 0x7f07c4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:54.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07dcf3d700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8151770 0x7f07c4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:54.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d7fff700 1 -- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8151770 msgr2=0x7f07c4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:50:54.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d7fff700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8151770 0x7f07c4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:54.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d7fff700 1 -- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c4004850 msgr2=0x7f07c4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:50:54.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d7fff700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c4004850 0x7f07c4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:54.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d7fff700 1 -- 172.21.15.7:0/196572177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c40035b0 con 0x7f07c4003aa0 2023-03-03T09:50:54.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d7fff700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07c4003aa0 0x7f07c4003e70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f07c800c2f0 tx=0x7f07c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b638801795fcb55 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:50:54.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.808+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c800ce30 con 0x7f07c4003aa0 2023-03-03T09:50:54.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.809+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c8005960 con 0x7f07c4003aa0 2023-03-03T09:50:54.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.809+0000 7f07d6ffd700 1 -- 172.21.15.7:0/196572177 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c8005cf0 con 0x7f07c4003aa0 2023-03-03T09:50:54.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.809+0000 7f07de9a0700 1 -- 172.21.15.7:0/196572177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07c4003aa0 msgr2=0x7f07c4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:54.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.809+0000 7f07de9a0700 1 --2- 172.21.15.7:0/196572177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07c4003aa0 0x7f07c4003e70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f07c800c2f0 tx=0x7f07c4005880 comp rx=0 tx=0).stop 2023-03-03T09:50:54.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.809+0000 7f07dcf3d700 1 -- 172.21.15.7:0/196572177 reap_dead start 2023-03-03T09:50:54.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.809+0000 7f07de9a0700 1 -- 172.21.15.7:0/196572177 shutdown_connections 2023-03-03T09:50:54.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.809+0000 7f07de9a0700 1 -- 172.21.15.7:0/196572177 >> 172.21.15.7:0/196572177 conn(0x7f07d806c110 msgr2=0x7f07d806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:50:54.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.809+0000 7f07de9a0700 1 -- 172.21.15.7:0/196572177 shutdown_connections 2023-03-03T09:50:54.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.809+0000 7f07de9a0700 1 -- 172.21.15.7:0/196572177 wait complete. 2023-03-03T09:50:54.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.810+0000 7f07de9a0700 1 Processor -- start 2023-03-03T09:50:54.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.811+0000 7f07de9a0700 1 -- start start 2023-03-03T09:50:54.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.811+0000 7f07de9a0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c4003aa0 0x7f07d81edb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:54.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.811+0000 7f07de9a0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07d80abe80 0x7f07d8154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:54.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.811+0000 7f07de9a0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8149030 0x7f07d81dc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:54.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.811+0000 7f07de9a0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d81ed6d0 con 0x7f07d80abe80 2023-03-03T09:50:54.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.811+0000 7f07de9a0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80ac310 con 0x7f07c4003aa0 2023-03-03T09:50:54.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.811+0000 7f07de9a0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80ac5f0 con 0x7f07d8149030 2023-03-03T09:50:54.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.811+0000 7f07d77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07d80abe80 0x7f07d8154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:54.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.811+0000 7f07d77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07d80abe80 0x7f07d8154390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57934/0 (socket says 172.21.15.7:57934) 2023-03-03T09:50:54.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.811+0000 7f07d77fe700 1 -- 172.21.15.7:0/1198703017 learned_addr learned my addr 172.21.15.7:0/1198703017 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:50:54.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07dcf3d700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8149030 0x7f07d81dc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:54.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d7fff700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c4003aa0 0x7f07d81edb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:54.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d77fe700 1 -- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8149030 msgr2=0x7f07d81dc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:54.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d77fe700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8149030 0x7f07d81dc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:54.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d77fe700 1 -- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c4003aa0 msgr2=0x7f07d81edb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:54.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d77fe700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c4003aa0 0x7f07d81edb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:54.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d77fe700 1 -- 172.21.15.7:0/1198703017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d8070820 con 0x7f07d80abe80 2023-03-03T09:50:54.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07dcf3d700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8149030 0x7f07d81dc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:50:54.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d7fff700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c4003aa0 0x7f07d81edb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:50:54.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d77fe700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07d80abe80 0x7f07d8154390 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f07c000f7f0 tx=0x7f07c000fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:50:54.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d4ff9700 1 -- 172.21.15.7:0/1198703017 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c00052e0 con 0x7f07d80abe80 2023-03-03T09:50:54.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d4ff9700 1 -- 172.21.15.7:0/1198703017 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c0005460 con 0x7f07d80abe80 2023-03-03T09:50:54.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07de9a0700 1 -- 172.21.15.7:0/1198703017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d8070540 con 0x7f07d80abe80 2023-03-03T09:50:54.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07d4ff9700 1 -- 172.21.15.7:0/1198703017 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c00038c0 con 0x7f07d80abe80 2023-03-03T09:50:54.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.812+0000 7f07de9a0700 1 -- 172.21.15.7:0/1198703017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d8155760 con 0x7f07d80abe80 2023-03-03T09:50:54.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.813+0000 7f07de9a0700 1 -- 172.21.15.7:0/1198703017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d8074bb0 con 0x7f07d80abe80 2023-03-03T09:50:54.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.813+0000 7f07d4ff9700 1 -- 172.21.15.7:0/1198703017 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f07c0003a20 con 0x7f07d80abe80 2023-03-03T09:50:54.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.814+0000 7f07d4ff9700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07b8068320 0x7f07b806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:50:54.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.814+0000 7f07d4ff9700 1 -- 172.21.15.7:0/1198703017 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f07c0086000 con 0x7f07d80abe80 2023-03-03T09:50:54.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.814+0000 7f07d7fff700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07b8068320 0x7f07b806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:50:54.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.815+0000 7f07d7fff700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07b8068320 0x7f07b806a7d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f07c8015040 tx=0x7f07c8007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:50:54.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:54.818+0000 7f07d4ff9700 1 -- 172.21.15.7:0/1198703017 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f07c0052320 con 0x7f07d80abe80 2023-03-03T09:50:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.057+0000 7f07de9a0700 1 -- 172.21.15.7:0/1198703017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07d80c4540 con 0x7f07d80abe80 2023-03-03T09:50:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.057+0000 7f07d4ff9700 1 -- 172.21.15.7:0/1198703017 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f07d80c4540 con 0x7f07d80abe80 2023-03-03T09:50:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:50:55.059 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:50:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.059+0000 7f07de9a0700 1 -- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07b8068320 msgr2=0x7f07b806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.059+0000 7f07de9a0700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07b8068320 0x7f07b806a7d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f07c8015040 tx=0x7f07c8007000 comp rx=0 tx=0).stop 2023-03-03T09:50:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.059+0000 7f07de9a0700 1 -- 172.21.15.7:0/1198703017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07d80abe80 msgr2=0x7f07d8154390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:50:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.059+0000 7f07de9a0700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07d80abe80 0x7f07d8154390 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f07c000f7f0 tx=0x7f07c000fbb0 comp rx=0 tx=0).stop 2023-03-03T09:50:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.059+0000 7f07de9a0700 1 -- 172.21.15.7:0/1198703017 shutdown_connections 2023-03-03T09:50:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.059+0000 7f07de9a0700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07d8149030 0x7f07d81dc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.059+0000 7f07de9a0700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07c4003aa0 0x7f07d81edb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.059+0000 7f07de9a0700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07b8068320 0x7f07b806a7d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.059+0000 7f07de9a0700 1 --2- 172.21.15.7:0/1198703017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07d80abe80 0x7f07d8154390 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:50:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.059+0000 7f07de9a0700 1 -- 172.21.15.7:0/1198703017 >> 172.21.15.7:0/1198703017 conn(0x7f07d806c110 msgr2=0x7f07d8150740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:50:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.060+0000 7f07de9a0700 1 -- 172.21.15.7:0/1198703017 shutdown_connections 2023-03-03T09:50:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:50:55.060+0000 7f07de9a0700 1 -- 172.21.15.7:0/1198703017 wait complete. 2023-03-03T09:50:55.063 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:50:55.073 INFO:tasks.mds_thrash.fs.[b]:waiting for 31 secs before thrashing 2023-03-03T09:51:20.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:20.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:24.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:51:24.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.480+0000 7f9a7a68f700 1 Processor -- start 2023-03-03T09:51:24.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.480+0000 7f9a7a68f700 1 -- start start 2023-03-03T09:51:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a7a68f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a7414de50 0x7f9a7406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a7a68f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a740c65c0 0x7f9a740c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a7a68f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a740a65b0 con 0x7f9a7406a7f0 2023-03-03T09:51:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a7a68f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a74154e00 con 0x7f9a7414de50 2023-03-03T09:51:24.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a7a68f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740c3370 con 0x7f9a740c65c0 2023-03-03T09:51:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a73fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a7414de50 0x7f9a7406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a78c2c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9a7406a7f0 0x7f9a740c2c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51938/0 (socket says 172.21.15.7:51938) 2023-03-03T09:51:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a78c2c700 1 -- 172.21.15.7:0/2925554715 learned_addr learned my addr 172.21.15.7:0/2925554715 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:51:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a737fe700 1 --2- 172.21.15.7:0/2925554715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a740c65c0 0x7f9a740c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a73fff700 1 -- 172.21.15.7:0/2925554715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a740c65c0 msgr2=0x7f9a740c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:51:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a73fff700 1 --2- 172.21.15.7:0/2925554715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a740c65c0 0x7f9a740c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a73fff700 1 -- 172.21.15.7:0/2925554715 >> v1:172.21.15.7:6789/0 conn(0x7f9a7406a7f0 legacy=0x7f9a740c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a72ffd700 1 -- 172.21.15.7:0/2925554715 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439485615 0 0) 0x7f9a740a65b0 con 0x7f9a7406a7f0 2023-03-03T09:51:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.481+0000 7f9a73fff700 1 -- 172.21.15.7:0/2925554715 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a741541f0 con 0x7f9a7414de50 2023-03-03T09:51:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a73fff700 1 --2- 172.21.15.7:0/2925554715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 0x7f9a7406a2b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009840 tx=0x7f9a64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1a7cbf6d80dbfe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a72ffd700 1 -- 172.21.15.7:0/2925554715 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64011830 con 0x7f9a7414de50 2023-03-03T09:51:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a72ffd700 1 -- 172.21.15.7:0/2925554715 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a64011990 con 0x7f9a7414de50 2023-03-03T09:51:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a72ffd700 1 -- 172.21.15.7:0/2925554715 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64011d20 con 0x7f9a7414de50 2023-03-03T09:51:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a7a68f700 1 -- 172.21.15.7:0/2925554715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 msgr2=0x7f9a7406a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a7a68f700 1 --2- 172.21.15.7:0/2925554715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 0x7f9a7406a2b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009840 tx=0x7f9a64009b50 comp rx=0 tx=0).stop 2023-03-03T09:51:24.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a7a68f700 1 -- 172.21.15.7:0/2925554715 shutdown_connections 2023-03-03T09:51:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a7a68f700 1 --2- 172.21.15.7:0/2925554715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 0x7f9a7406a2b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a7a68f700 1 --2- 172.21.15.7:0/2925554715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a740c65c0 0x7f9a740c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a7a68f700 1 -- 172.21.15.7:0/2925554715 >> 172.21.15.7:0/2925554715 conn(0x7f9a740a9430 msgr2=0x7f9a740c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:51:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a7a68f700 1 -- 172.21.15.7:0/2925554715 shutdown_connections 2023-03-03T09:51:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.482+0000 7f9a7a68f700 1 -- 172.21.15.7:0/2925554715 wait complete. 2023-03-03T09:51:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.484+0000 7f9a7a68f700 1 Processor -- start 2023-03-03T09:51:24.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.484+0000 7f9a7a68f700 1 -- start start 2023-03-03T09:51:24.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a7a68f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a7406a7f0 0x7f9a7414cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:24.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a7a68f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a740c65c0 0x7f9a74153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:24.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a7a68f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 0x7f9a7414bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:24.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a7a68f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a741e92e0 con 0x7f9a7414de50 2023-03-03T09:51:24.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a7a68f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7406fb50 con 0x7f9a740c65c0 2023-03-03T09:51:24.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a7a68f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7406ffe0 con 0x7f9a7406a7f0 2023-03-03T09:51:24.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a78c2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 0x7f9a7414bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a78c2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 0x7f9a7414bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57964/0 (socket says 172.21.15.7:57964) 2023-03-03T09:51:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a78c2c700 1 -- 172.21.15.7:0/3206497335 learned_addr learned my addr 172.21.15.7:0/3206497335 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:51:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a737fe700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a740c65c0 0x7f9a74153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a73fff700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a7406a7f0 0x7f9a7414cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a78c2c700 1 -- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a7406a7f0 msgr2=0x7f9a7414cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a78c2c700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a7406a7f0 0x7f9a7414cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a78c2c700 1 -- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a740c65c0 msgr2=0x7f9a74153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:24.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a78c2c700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a740c65c0 0x7f9a74153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.485+0000 7f9a78c2c700 1 -- 172.21.15.7:0/3206497335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640094a0 con 0x7f9a7414de50 2023-03-03T09:51:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.486+0000 7f9a78c2c700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 0x7f9a7414bc30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00cb60 tx=0x7f9a6c00ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.486+0000 7f9a70ff9700 1 -- 172.21.15.7:0/3206497335 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c00a9e0 con 0x7f9a7414de50 2023-03-03T09:51:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.486+0000 7f9a70ff9700 1 -- 172.21.15.7:0/3206497335 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6c00ab40 con 0x7f9a7414de50 2023-03-03T09:51:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.486+0000 7f9a7a68f700 1 -- 172.21.15.7:0/3206497335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a741547b0 con 0x7f9a7414de50 2023-03-03T09:51:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.486+0000 7f9a70ff9700 1 -- 172.21.15.7:0/3206497335 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0070d0 con 0x7f9a7414de50 2023-03-03T09:51:24.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.486+0000 7f9a7a68f700 1 -- 172.21.15.7:0/3206497335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a741544d0 con 0x7f9a7414de50 2023-03-03T09:51:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.487+0000 7f9a7a68f700 1 -- 172.21.15.7:0/3206497335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a74154a90 con 0x7f9a7414de50 2023-03-03T09:51:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.487+0000 7f9a70ff9700 1 -- 172.21.15.7:0/3206497335 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a6c010070 con 0x7f9a7414de50 2023-03-03T09:51:24.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.488+0000 7f9a70ff9700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a540682f0 0x7f9a5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:24.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.488+0000 7f9a70ff9700 1 -- 172.21.15.7:0/3206497335 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9a6c085b80 con 0x7f9a7414de50 2023-03-03T09:51:24.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.488+0000 7f9a73fff700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a540682f0 0x7f9a5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:24.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.488+0000 7f9a73fff700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a540682f0 0x7f9a5406a7a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9a640090b0 tx=0x7f9a64008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:24.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.492+0000 7f9a70ff9700 1 -- 172.21.15.7:0/3206497335 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a6c051d50 con 0x7f9a7414de50 2023-03-03T09:51:24.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.732+0000 7f9a7a68f700 1 -- 172.21.15.7:0/3206497335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a740729a0 con 0x7f9a7414de50 2023-03-03T09:51:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.733+0000 7f9a70ff9700 1 -- 172.21.15.7:0/3206497335 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9a740729a0 con 0x7f9a7414de50 2023-03-03T09:51:24.734 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:51:24.734 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:51:24.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 -- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a540682f0 msgr2=0x7f9a5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:24.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a540682f0 0x7f9a5406a7a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9a640090b0 tx=0x7f9a64008030 comp rx=0 tx=0).stop 2023-03-03T09:51:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 -- 172.21.15.7:0/3206497335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 msgr2=0x7f9a7414bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 0x7f9a7414bc30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00cb60 tx=0x7f9a6c00ce70 comp rx=0 tx=0).stop 2023-03-03T09:51:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 -- 172.21.15.7:0/3206497335 shutdown_connections 2023-03-03T09:51:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a540682f0 0x7f9a5406a7a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a7414de50 0x7f9a7414bc30 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a7406a7f0 0x7f9a7414cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 --2- 172.21.15.7:0/3206497335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a740c65c0 0x7f9a74153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:24.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 -- 172.21.15.7:0/3206497335 >> 172.21.15.7:0/3206497335 conn(0x7f9a740a9430 msgr2=0x7f9a740c2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:51:24.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 -- 172.21.15.7:0/3206497335 shutdown_connections 2023-03-03T09:51:24.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:24.735+0000 7f9a7a68f700 1 -- 172.21.15.7:0/3206497335 wait complete. 2023-03-03T09:51:24.738 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:51:24.748 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before thrashing 2023-03-03T09:51:26.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:51:26.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.211+0000 7f3c8881e700 1 Processor -- start 2023-03-03T09:51:26.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.211+0000 7f3c8881e700 1 -- start start 2023-03-03T09:51:26.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.212+0000 7f3c8881e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3c8414fda0 0x7f3c84156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:26.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.212+0000 7f3c8881e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3c840abe20 0x7f3c841529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:26.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.212+0000 7f3c8881e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c840c2f30 con 0x7f3c8414f9d0 2023-03-03T09:51:26.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.212+0000 7f3c8881e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c840c31d0 con 0x7f3c8414fda0 2023-03-03T09:51:26.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.212+0000 7f3c8881e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c841524a0 con 0x7f3c840abe20 2023-03-03T09:51:26.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.212+0000 7f3c81d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3c8414fda0 0x7f3c84156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:26.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c81d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3c8414fda0 0x7f3c84156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57972/0 (socket says 172.21.15.7:57972) 2023-03-03T09:51:26.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c81d9b700 1 -- 172.21.15.7:0/3287789248 learned_addr learned my addr 172.21.15.7:0/3287789248 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:51:26.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c82d9d700 1 --2- 172.21.15.7:0/3287789248 >> v2:172.21.15.143:3300/0 conn(0x7f3c840abe20 0x7f3c841529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:26.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c81d9b700 1 -- 172.21.15.7:0/3287789248 >> v2:172.21.15.143:3300/0 conn(0x7f3c840abe20 msgr2=0x7f3c841529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:51:26.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c81d9b700 1 --2- 172.21.15.7:0/3287789248 >> v2:172.21.15.143:3300/0 conn(0x7f3c840abe20 0x7f3c841529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c81d9b700 1 -- 172.21.15.7:0/3287789248 >> v1:172.21.15.143:6789/0 conn(0x7f3c8414f9d0 legacy=0x7f3c84071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:51:26.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c81d9b700 1 -- 172.21.15.7:0/3287789248 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c840c34b0 con 0x7f3c8414fda0 2023-03-03T09:51:26.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c82d9d700 1 --2- 172.21.15.7:0/3287789248 >> v2:172.21.15.143:3300/0 conn(0x7f3c840abe20 0x7f3c841529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:51:26.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c81d9b700 1 --2- 172.21.15.7:0/3287789248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8414fda0 0x7f3c84156430 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3c84071230 tx=0x7f3c6c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52a26054fa6430d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:26.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c8159a700 1 -- 172.21.15.7:0/3287789248 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c004230 con 0x7f3c8414fda0 2023-03-03T09:51:26.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c8159a700 1 -- 172.21.15.7:0/3287789248 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c004390 con 0x7f3c8414fda0 2023-03-03T09:51:26.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.213+0000 7f3c8159a700 1 -- 172.21.15.7:0/3287789248 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c0036a0 con 0x7f3c8414fda0 2023-03-03T09:51:26.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.214+0000 7f3c8881e700 1 -- 172.21.15.7:0/3287789248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8414fda0 msgr2=0x7f3c84156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:26.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.214+0000 7f3c8881e700 1 --2- 172.21.15.7:0/3287789248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8414fda0 0x7f3c84156430 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3c84071230 tx=0x7f3c6c009bd0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.214+0000 7f3c8881e700 1 -- 172.21.15.7:0/3287789248 shutdown_connections 2023-03-03T09:51:26.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.214+0000 7f3c8881e700 1 --2- 172.21.15.7:0/3287789248 >> v2:172.21.15.143:3300/0 conn(0x7f3c840abe20 0x7f3c841529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.214+0000 7f3c8881e700 1 --2- 172.21.15.7:0/3287789248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c8414fda0 0x7f3c84156430 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.214+0000 7f3c8881e700 1 -- 172.21.15.7:0/3287789248 >> 172.21.15.7:0/3287789248 conn(0x7f3c840c8900 msgr2=0x7f3c840c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:51:26.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.214+0000 7f3c8881e700 1 -- 172.21.15.7:0/3287789248 shutdown_connections 2023-03-03T09:51:26.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.214+0000 7f3c8881e700 1 -- 172.21.15.7:0/3287789248 wait complete. 2023-03-03T09:51:26.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.215+0000 7f3c8881e700 1 Processor -- start 2023-03-03T09:51:26.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.215+0000 7f3c8881e700 1 -- start start 2023-03-03T09:51:26.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8881e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c840abe20 0x7f3c84152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:26.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8881e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8414f9d0 0x7f3c841dbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:26.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8881e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c8414fda0 0x7f3c841dc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:26.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8881e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c840c1fb0 con 0x7f3c840abe20 2023-03-03T09:51:26.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8881e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c841dc720 con 0x7f3c8414f9d0 2023-03-03T09:51:26.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8881e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8414f000 con 0x7f3c8414fda0 2023-03-03T09:51:26.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c840abe20 0x7f3c84152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:26.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c840abe20 0x7f3c84152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57976/0 (socket says 172.21.15.7:57976) 2023-03-03T09:51:26.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8259c700 1 -- 172.21.15.7:0/3672897486 learned_addr learned my addr 172.21.15.7:0/3672897486 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:51:26.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c81d9b700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8414f9d0 0x7f3c841dbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:26.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c82d9d700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c8414fda0 0x7f3c841dc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:26.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8259c700 1 -- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c8414fda0 msgr2=0x7f3c841dc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:26.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8259c700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c8414fda0 0x7f3c841dc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8259c700 1 -- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8414f9d0 msgr2=0x7f3c841dbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:26.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8259c700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8414f9d0 0x7f3c841dbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8259c700 1 -- 172.21.15.7:0/3672897486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c009550 con 0x7f3c840abe20 2023-03-03T09:51:26.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c81d9b700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8414f9d0 0x7f3c841dbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:51:26.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.216+0000 7f3c8259c700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c840abe20 0x7f3c84152ad0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400c500 tx=0x7f3c7400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:26.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.217+0000 7f3c7affd700 1 -- 172.21.15.7:0/3672897486 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74011790 con 0x7f3c840abe20 2023-03-03T09:51:26.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.217+0000 7f3c7affd700 1 -- 172.21.15.7:0/3672897486 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c740118f0 con 0x7f3c840abe20 2023-03-03T09:51:26.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.217+0000 7f3c8881e700 1 -- 172.21.15.7:0/3672897486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c840ac350 con 0x7f3c840abe20 2023-03-03T09:51:26.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.217+0000 7f3c7affd700 1 -- 172.21.15.7:0/3672897486 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74011ca0 con 0x7f3c840abe20 2023-03-03T09:51:26.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.217+0000 7f3c8881e700 1 -- 172.21.15.7:0/3672897486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c840ab630 con 0x7f3c840abe20 2023-03-03T09:51:26.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.218+0000 7f3c8881e700 1 -- 172.21.15.7:0/3672897486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c84071fa0 con 0x7f3c840abe20 2023-03-03T09:51:26.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.218+0000 7f3c7affd700 1 -- 172.21.15.7:0/3672897486 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3c74010050 con 0x7f3c840abe20 2023-03-03T09:51:26.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.219+0000 7f3c7affd700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c640682f0 0x7f3c6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:26.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.219+0000 7f3c7affd700 1 -- 172.21.15.7:0/3672897486 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3c74085fe0 con 0x7f3c840abe20 2023-03-03T09:51:26.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.219+0000 7f3c81d9b700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c640682f0 0x7f3c6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:26.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.219+0000 7f3c81d9b700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c640682f0 0x7f3c6406a7a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3c841501d0 tx=0x7f3c6c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:26.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.223+0000 7f3c7affd700 1 -- 172.21.15.7:0/3672897486 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c74052230 con 0x7f3c840abe20 2023-03-03T09:51:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.463+0000 7f3c8881e700 1 -- 172.21.15.7:0/3672897486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c840c60e0 con 0x7f3c840abe20 2023-03-03T09:51:26.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.463+0000 7f3c7affd700 1 -- 172.21.15.7:0/3672897486 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3c74055850 con 0x7f3c840abe20 2023-03-03T09:51:26.465 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:51:26.465 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:51:26.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 -- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c640682f0 msgr2=0x7f3c6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:26.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c640682f0 0x7f3c6406a7a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3c841501d0 tx=0x7f3c6c008030 comp rx=0 tx=0).stop 2023-03-03T09:51:26.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 -- 172.21.15.7:0/3672897486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c840abe20 msgr2=0x7f3c84152ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:26.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c840abe20 0x7f3c84152ad0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400c500 tx=0x7f3c7400c8c0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 -- 172.21.15.7:0/3672897486 shutdown_connections 2023-03-03T09:51:26.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c8414fda0 0x7f3c841dc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c640682f0 0x7f3c6406a7a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c840abe20 0x7f3c84152ad0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 --2- 172.21.15.7:0/3672897486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c8414f9d0 0x7f3c841dbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:26.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 -- 172.21.15.7:0/3672897486 >> 172.21.15.7:0/3672897486 conn(0x7f3c840c8900 msgr2=0x7f3c840c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:51:26.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 -- 172.21.15.7:0/3672897486 shutdown_connections 2023-03-03T09:51:26.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:26.465+0000 7f3c8881e700 1 -- 172.21.15.7:0/3672897486 wait complete. 2023-03-03T09:51:26.468 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:51:26.478 INFO:tasks.mds_thrash.fs.[b]:waiting for 40 secs before thrashing 2023-03-03T09:51:31.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:51:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.886+0000 7f601343c700 1 Processor -- start 2023-03-03T09:51:31.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.886+0000 7f601343c700 1 -- start start 2023-03-03T09:51:31.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.887+0000 7f601343c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f600c06d870 0x7f600c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:31.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.887+0000 7f601343c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0c7a60 con 0x7f600c0c1da0 2023-03-03T09:51:31.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.887+0000 7f601343c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0c3620 con 0x7f600c0c3e30 2023-03-03T09:51:31.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.887+0000 7f601343c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600c0c2af0 con 0x7f600c06d870 2023-03-03T09:51:31.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.887+0000 7f60109d7700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f600c0c1da0 0x7f600c0c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51968/0 (socket says 172.21.15.7:51968) 2023-03-03T09:51:31.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.887+0000 7f60109d7700 1 -- 172.21.15.7:0/1480924864 learned_addr learned my addr 172.21.15.7:0/1480924864 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:51:31.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.887+0000 7f60111d8700 1 --2- 172.21.15.7:0/1480924864 >> v2:172.21.15.143:3300/0 conn(0x7f600c06d870 0x7f600c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:31.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.887+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417275531 0 0) 0x7f600c0c7a60 con 0x7f600c0c1da0 2023-03-03T09:51:31.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.887+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff80035b0 con 0x7f600c0c1da0 2023-03-03T09:51:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4013460010 0 0) 0x7f5ff80035b0 con 0x7f600c0c1da0 2023-03-03T09:51:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 >> v2:172.21.15.143:3300/0 conn(0x7f600c06d870 msgr2=0x7f600c1550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:51:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 --2- 172.21.15.7:0/1480924864 >> v2:172.21.15.143:3300/0 conn(0x7f600c06d870 0x7f600c1550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 >> v1:172.21.15.143:6789/0 conn(0x7f600c0c3e30 legacy=0x7f600c15ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c1545e0 con 0x7f600c0c1da0 2023-03-03T09:51:31.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f5ffc001670 con 0x7f600c0c1da0 2023-03-03T09:51:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 >> v1:172.21.15.7:6789/0 conn(0x7f600c0c1da0 legacy=0x7f600c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ff8003a70 0x7f5ff8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f5ff8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ff8005200 0x7f5ff800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff800c0e0 con 0x7f5ff8004850 2023-03-03T09:51:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff800c220 con 0x7f5ff8005200 2023-03-03T09:51:31.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff800c3e0 con 0x7f5ff8003a70 2023-03-03T09:51:31.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f60109d7700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f5ff8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:31.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f60111d8700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ff8003a70 0x7f5ff8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:31.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f60119d9700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ff8005200 0x7f5ff800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:31.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f60109d7700 1 -- 172.21.15.7:0/1480924864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ff8003a70 msgr2=0x7f5ff8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:51:31.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f60109d7700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ff8003a70 0x7f5ff8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:31.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f60109d7700 1 -- 172.21.15.7:0/1480924864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ff8005200 msgr2=0x7f5ff800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:51:31.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f60109d7700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ff8005200 0x7f5ff800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:31.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f60109d7700 1 -- 172.21.15.7:0/1480924864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff80035b0 con 0x7f5ff8004850 2023-03-03T09:51:31.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f60119d9700 1 -- 172.21.15.7:0/1480924864 reap_dead start 2023-03-03T09:51:31.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f60109d7700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f5ff8004cc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00c240 tx=0x7f5ff8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a750444c7034a35 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:31.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.888+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc007c10 con 0x7f5ff8004850 2023-03-03T09:51:31.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.889+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc00cde0 con 0x7f5ff8004850 2023-03-03T09:51:31.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.889+0000 7f6003fff700 1 -- 172.21.15.7:0/1480924864 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc005ba0 con 0x7f5ff8004850 2023-03-03T09:51:31.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.889+0000 7f601343c700 1 -- 172.21.15.7:0/1480924864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 msgr2=0x7f5ff8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:31.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.889+0000 7f601343c700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f5ff8004cc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00c240 tx=0x7f5ff8005880 comp rx=0 tx=0).stop 2023-03-03T09:51:31.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.889+0000 7f601343c700 1 -- 172.21.15.7:0/1480924864 shutdown_connections 2023-03-03T09:51:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.889+0000 7f601343c700 1 --2- 172.21.15.7:0/1480924864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f5ff8004cc0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.889+0000 7f601343c700 1 -- 172.21.15.7:0/1480924864 >> 172.21.15.7:0/1480924864 conn(0x7f600c06c110 msgr2=0x7f600c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:51:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.889+0000 7f601343c700 1 -- 172.21.15.7:0/1480924864 shutdown_connections 2023-03-03T09:51:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.889+0000 7f601343c700 1 -- 172.21.15.7:0/1480924864 wait complete. 2023-03-03T09:51:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.890+0000 7f601343c700 1 Processor -- start 2023-03-03T09:51:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.891+0000 7f601343c700 1 -- start start 2023-03-03T09:51:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.891+0000 7f601343c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f600c1e95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.891+0000 7f601343c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f600c1eaa60 0x7f600c14db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.891+0000 7f601343c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f600c151120 0x7f600c151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.891+0000 7f601343c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0ab130 con 0x7f5ff8004850 2023-03-03T09:51:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.891+0000 7f601343c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0aa4f0 con 0x7f600c151120 2023-03-03T09:51:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.891+0000 7f601343c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c14e080 con 0x7f600c1eaa60 2023-03-03T09:51:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.891+0000 7f60111d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f600c1e95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.891+0000 7f60111d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f600c1e95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58000/0 (socket says 172.21.15.7:58000) 2023-03-03T09:51:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.891+0000 7f60111d8700 1 -- 172.21.15.7:0/3913414102 learned_addr learned my addr 172.21.15.7:0/3913414102 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:51:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f60109d7700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f600c1eaa60 0x7f600c14db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:31.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f60119d9700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f600c151120 0x7f600c151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f60111d8700 1 -- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f600c1eaa60 msgr2=0x7f600c14db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f60111d8700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f600c1eaa60 0x7f600c14db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f60111d8700 1 -- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f600c151120 msgr2=0x7f600c151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f60111d8700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f600c151120 0x7f600c151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f60111d8700 1 -- 172.21.15.7:0/3913414102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c154ba0 con 0x7f5ff8004850 2023-03-03T09:51:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f60119d9700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f600c151120 0x7f600c151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:51:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f60111d8700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f600c1e95f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f6008011aa0 tx=0x7f6008011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f6001ffb700 1 -- 172.21.15.7:0/3913414102 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600800f9e0 con 0x7f5ff8004850 2023-03-03T09:51:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f6001ffb700 1 -- 172.21.15.7:0/3913414102 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600800fb40 con 0x7f5ff8004850 2023-03-03T09:51:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f601343c700 1 -- 172.21.15.7:0/3913414102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c1548c0 con 0x7f5ff8004850 2023-03-03T09:51:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f6001ffb700 1 -- 172.21.15.7:0/3913414102 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008014440 con 0x7f5ff8004850 2023-03-03T09:51:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.892+0000 7f601343c700 1 -- 172.21.15.7:0/3913414102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c06dc40 con 0x7f5ff8004850 2023-03-03T09:51:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.893+0000 7f601343c700 1 -- 172.21.15.7:0/3913414102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600c0ab3a0 con 0x7f5ff8004850 2023-03-03T09:51:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.895+0000 7f6001ffb700 1 -- 172.21.15.7:0/3913414102 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f600801d400 con 0x7f5ff8004850 2023-03-03T09:51:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.895+0000 7f6001ffb700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ff00682f0 0x7f5ff006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:51:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.895+0000 7f6001ffb700 1 -- 172.21.15.7:0/3913414102 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f60080871a0 con 0x7f5ff8004850 2023-03-03T09:51:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.895+0000 7f60109d7700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ff00682f0 0x7f5ff006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:51:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.896+0000 7f60109d7700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ff00682f0 0x7f5ff006a7a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc001930 tx=0x7f5ffc007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:51:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:31.899+0000 7f6001ffb700 1 -- 172.21.15.7:0/3913414102 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6008053470 con 0x7f5ff8004850 2023-03-03T09:51:32.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.137+0000 7f601343c700 1 -- 172.21.15.7:0/3913414102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f600c0c4230 con 0x7f5ff8004850 2023-03-03T09:51:32.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.137+0000 7f6001ffb700 1 -- 172.21.15.7:0/3913414102 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6008056a90 con 0x7f5ff8004850 2023-03-03T09:51:32.139 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:51:32.139 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:51:32.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 -- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ff00682f0 msgr2=0x7f5ff006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:32.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ff00682f0 0x7f5ff006a7a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc001930 tx=0x7f5ffc007000 comp rx=0 tx=0).stop 2023-03-03T09:51:32.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 -- 172.21.15.7:0/3913414102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 msgr2=0x7f600c1e95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:51:32.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f600c1e95f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f6008011aa0 tx=0x7f6008011e60 comp rx=0 tx=0).stop 2023-03-03T09:51:32.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 -- 172.21.15.7:0/3913414102 shutdown_connections 2023-03-03T09:51:32.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f600c151120 0x7f600c151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:32.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ff00682f0 0x7f5ff006a7a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:32.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ff8004850 0x7f600c1e95f0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:32.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 --2- 172.21.15.7:0/3913414102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f600c1eaa60 0x7f600c14db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:51:32.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 -- 172.21.15.7:0/3913414102 >> 172.21.15.7:0/3913414102 conn(0x7f600c06c110 msgr2=0x7f600c152340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:51:32.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 -- 172.21.15.7:0/3913414102 shutdown_connections 2023-03-03T09:51:32.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:51:32.139+0000 7f601343c700 1 -- 172.21.15.7:0/3913414102 wait complete. 2023-03-03T09:51:32.143 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:51:32.153 INFO:tasks.mds_thrash.fs.[a]:waiting for 109 secs before thrashing 2023-03-03T09:51:50.947 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:51:50.951 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:06.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:52:06.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.616+0000 7fad61d32700 1 Processor -- start 2023-03-03T09:52:06.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.616+0000 7fad61d32700 1 -- start start 2023-03-03T09:52:06.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad61d32700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fad5c151950 0x7fad5c154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:06.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad61d32700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad5c0c5dc0 0x7fad5c0bf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:06.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad61d32700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c0c5940 con 0x7fad5c0b0490 2023-03-03T09:52:06.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad61d32700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5c1584a0 con 0x7fad5c151950 2023-03-03T09:52:06.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad61d32700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c158780 con 0x7fad5c0c5dc0 2023-03-03T09:52:06.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad5b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fad5c151950 0x7fad5c154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:06.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad5b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fad5c151950 0x7fad5c154ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58048/0 (socket says 172.21.15.7:58048) 2023-03-03T09:52:06.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad5b7fe700 1 -- 172.21.15.7:0/221876300 learned_addr learned my addr 172.21.15.7:0/221876300 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:52:06.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad5affd700 1 --2- 172.21.15.7:0/221876300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad5c0c5dc0 0x7fad5c0bf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:06.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad5b7fe700 1 -- 172.21.15.7:0/221876300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad5c0c5dc0 msgr2=0x7fad5c0bf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:52:06.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad5b7fe700 1 --2- 172.21.15.7:0/221876300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad5c0c5dc0 0x7fad5c0bf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:06.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad5b7fe700 1 -- 172.21.15.7:0/221876300 >> v1:172.21.15.143:6789/0 conn(0x7fad5c0b0490 legacy=0x7fad5c0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:52:06.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad5b7fe700 1 -- 172.21.15.7:0/221876300 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c1546c0 con 0x7fad5c151950 2023-03-03T09:52:06.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.617+0000 7fad5b7fe700 1 --2- 172.21.15.7:0/221876300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 0x7fad5c154ec0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0098f0 tx=0x7fad4c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c21a61079b522d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:52:06.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad5a7fc700 1 -- 172.21.15.7:0/221876300 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c00b8a0 con 0x7fad5c151950 2023-03-03T09:52:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad5a7fc700 1 -- 172.21.15.7:0/221876300 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c00ba00 con 0x7fad5c151950 2023-03-03T09:52:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad5a7fc700 1 -- 172.21.15.7:0/221876300 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c00bd90 con 0x7fad5c151950 2023-03-03T09:52:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad61d32700 1 -- 172.21.15.7:0/221876300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 msgr2=0x7fad5c154ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:52:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad61d32700 1 --2- 172.21.15.7:0/221876300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 0x7fad5c154ec0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0098f0 tx=0x7fad4c009c00 comp rx=0 tx=0).stop 2023-03-03T09:52:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad61d32700 1 -- 172.21.15.7:0/221876300 shutdown_connections 2023-03-03T09:52:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad61d32700 1 --2- 172.21.15.7:0/221876300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 0x7fad5c154ec0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad61d32700 1 --2- 172.21.15.7:0/221876300 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad5c0c5dc0 0x7fad5c0bf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad61d32700 1 -- 172.21.15.7:0/221876300 >> 172.21.15.7:0/221876300 conn(0x7fad5c066840 msgr2=0x7fad5c06ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:52:06.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad61d32700 1 -- 172.21.15.7:0/221876300 shutdown_connections 2023-03-03T09:52:06.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.618+0000 7fad61d32700 1 -- 172.21.15.7:0/221876300 wait complete. 2023-03-03T09:52:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.619+0000 7fad61d32700 1 Processor -- start 2023-03-03T09:52:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.620+0000 7fad61d32700 1 -- start start 2023-03-03T09:52:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.620+0000 7fad61d32700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad5c0b0490 0x7fad5c156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.620+0000 7fad61d32700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c0c5dc0 0x7fad5c0c25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.620+0000 7fad61d32700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 0x7fad5c14f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.620+0000 7fad61d32700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c151120 con 0x7fad5c151950 2023-03-03T09:52:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.620+0000 7fad61d32700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c0c4260 con 0x7fad5c0c5dc0 2023-03-03T09:52:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.620+0000 7fad61d32700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c150e40 con 0x7fad5c0b0490 2023-03-03T09:52:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 0x7fad5c14f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 0x7fad5c14f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58058/0 (socket says 172.21.15.7:58058) 2023-03-03T09:52:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5bfff700 1 -- 172.21.15.7:0/748385071 learned_addr learned my addr 172.21.15.7:0/748385071 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:52:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5affd700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c0c5dc0 0x7fad5c0c25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5b7fe700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad5c0b0490 0x7fad5c156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5bfff700 1 -- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad5c0b0490 msgr2=0x7fad5c156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:52:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5bfff700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad5c0b0490 0x7fad5c156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5bfff700 1 -- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c0c5dc0 msgr2=0x7fad5c0c25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:52:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5bfff700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c0c5dc0 0x7fad5c0c25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5bfff700 1 -- 172.21.15.7:0/748385071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c009550 con 0x7fad5c151950 2023-03-03T09:52:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad5bfff700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 0x7fad5c14f840 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fad5400cb50 tx=0x7fad5400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:52:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad43fff700 1 -- 172.21.15.7:0/748385071 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54003c00 con 0x7fad5c151950 2023-03-03T09:52:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad43fff700 1 -- 172.21.15.7:0/748385071 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad54003d60 con 0x7fad5c151950 2023-03-03T09:52:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad43fff700 1 -- 172.21.15.7:0/748385071 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5400aa20 con 0x7fad5c151950 2023-03-03T09:52:06.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad61d32700 1 -- 172.21.15.7:0/748385071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c151d20 con 0x7fad5c151950 2023-03-03T09:52:06.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.621+0000 7fad61d32700 1 -- 172.21.15.7:0/748385071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c1549a0 con 0x7fad5c151950 2023-03-03T09:52:06.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.622+0000 7fad61d32700 1 -- 172.21.15.7:0/748385071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5c154370 con 0x7fad5c151950 2023-03-03T09:52:06.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.623+0000 7fad43fff700 1 -- 172.21.15.7:0/748385071 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fad54010050 con 0x7fad5c151950 2023-03-03T09:52:06.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.623+0000 7fad43fff700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad480682f0 0x7fad4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:06.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.623+0000 7fad43fff700 1 -- 172.21.15.7:0/748385071 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fad54086160 con 0x7fad5c151950 2023-03-03T09:52:06.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.623+0000 7fad5b7fe700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad480682f0 0x7fad4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:06.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.624+0000 7fad5b7fe700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad480682f0 0x7fad4806a7a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fad4c000c00 tx=0x7fad4c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:52:06.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.628+0000 7fad43fff700 1 -- 172.21.15.7:0/748385071 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fad54052380 con 0x7fad5c151950 2023-03-03T09:52:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.867+0000 7fad61d32700 1 -- 172.21.15.7:0/748385071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad5c0c44a0 con 0x7fad5c151950 2023-03-03T09:52:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.868+0000 7fad43fff700 1 -- 172.21.15.7:0/748385071 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fad5c0c44a0 con 0x7fad5c151950 2023-03-03T09:52:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:52:06.871 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:52:06.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 -- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad480682f0 msgr2=0x7fad4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:52:06.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad480682f0 0x7fad4806a7a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fad4c000c00 tx=0x7fad4c008030 comp rx=0 tx=0).stop 2023-03-03T09:52:06.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 -- 172.21.15.7:0/748385071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 msgr2=0x7fad5c14f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:52:06.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 0x7fad5c14f840 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fad5400cb50 tx=0x7fad5400ce60 comp rx=0 tx=0).stop 2023-03-03T09:52:06.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 -- 172.21.15.7:0/748385071 shutdown_connections 2023-03-03T09:52:06.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad480682f0 0x7fad4806a7a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:06.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad5c151950 0x7fad5c14f840 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:06.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad5c0b0490 0x7fad5c156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:06.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 --2- 172.21.15.7:0/748385071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c0c5dc0 0x7fad5c0c25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:06.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 -- 172.21.15.7:0/748385071 >> 172.21.15.7:0/748385071 conn(0x7fad5c066840 msgr2=0x7fad5c0c05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:52:06.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 -- 172.21.15.7:0/748385071 shutdown_connections 2023-03-03T09:52:06.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:06.870+0000 7fad61d32700 1 -- 172.21.15.7:0/748385071 wait complete. 2023-03-03T09:52:06.874 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:52:06.884 INFO:tasks.mds_thrash.fs.[b]:waiting for 33 secs before thrashing 2023-03-03T09:52:21.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:21.347 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:39.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:52:40.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.024+0000 7f29f5675700 1 Processor -- start 2023-03-03T09:52:40.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.024+0000 7f29f5675700 1 -- start start 2023-03-03T09:52:40.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.025+0000 7f29f5675700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f29f0151cc0 0x7f29f00ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:40.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.025+0000 7f29f5675700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29f00b00b0 0x7f29f00c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:40.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.025+0000 7f29f5675700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f00c7ca0 con 0x7f29f00cac10 2023-03-03T09:52:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.025+0000 7f29f5675700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f0152090 con 0x7f29f00b00b0 2023-03-03T09:52:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.025+0000 7f29f5675700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f00c6cf0 con 0x7f29f0151cc0 2023-03-03T09:52:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.025+0000 7f29ee7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29f00b00b0 0x7f29f00c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.025+0000 7f29ee7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29f00b00b0 0x7f29f00c7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58092/0 (socket says 172.21.15.7:58092) 2023-03-03T09:52:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.025+0000 7f29ee7fc700 1 -- 172.21.15.7:0/153221659 learned_addr learned my addr 172.21.15.7:0/153221659 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:52:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29eeffd700 1 --2- 172.21.15.7:0/153221659 >> v2:172.21.15.143:3300/0 conn(0x7f29f0151cc0 0x7f29f00ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29ee7fc700 1 -- 172.21.15.7:0/153221659 >> v2:172.21.15.143:3300/0 conn(0x7f29f0151cc0 msgr2=0x7f29f00ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:52:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29ee7fc700 1 --2- 172.21.15.7:0/153221659 >> v2:172.21.15.143:3300/0 conn(0x7f29f0151cc0 0x7f29f00ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29ee7fc700 1 -- 172.21.15.7:0/153221659 >> v1:172.21.15.143:6789/0 conn(0x7f29f00cac10 legacy=0x7f29f00c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:52:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29ee7fc700 1 -- 172.21.15.7:0/153221659 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f01548f0 con 0x7f29f00b00b0 2023-03-03T09:52:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29ee7fc700 1 --2- 172.21.15.7:0/153221659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00b00b0 0x7f29f00c7050 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f29f00cb350 tx=0x7f29e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645dbbb97a29df51 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:52:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29edffb700 1 -- 172.21.15.7:0/153221659 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0004230 con 0x7f29f00b00b0 2023-03-03T09:52:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29edffb700 1 -- 172.21.15.7:0/153221659 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e0004390 con 0x7f29f00b00b0 2023-03-03T09:52:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29edffb700 1 -- 172.21.15.7:0/153221659 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e00036a0 con 0x7f29f00b00b0 2023-03-03T09:52:40.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29f5675700 1 -- 172.21.15.7:0/153221659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00b00b0 msgr2=0x7f29f00c7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:52:40.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29f5675700 1 --2- 172.21.15.7:0/153221659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00b00b0 0x7f29f00c7050 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f29f00cb350 tx=0x7f29e0009bd0 comp rx=0 tx=0).stop 2023-03-03T09:52:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29f5675700 1 -- 172.21.15.7:0/153221659 shutdown_connections 2023-03-03T09:52:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29f5675700 1 --2- 172.21.15.7:0/153221659 >> v2:172.21.15.143:3300/0 conn(0x7f29f0151cc0 0x7f29f00ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29f5675700 1 --2- 172.21.15.7:0/153221659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00b00b0 0x7f29f00c7050 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.026+0000 7f29f5675700 1 -- 172.21.15.7:0/153221659 >> 172.21.15.7:0/153221659 conn(0x7f29f006b980 msgr2=0x7f29f0071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:52:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.027+0000 7f29f5675700 1 -- 172.21.15.7:0/153221659 shutdown_connections 2023-03-03T09:52:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.027+0000 7f29f5675700 1 -- 172.21.15.7:0/153221659 wait complete. 2023-03-03T09:52:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.028+0000 7f29f5675700 1 Processor -- start 2023-03-03T09:52:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.028+0000 7f29f5675700 1 -- start start 2023-03-03T09:52:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29f5675700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29f00b00b0 0x7f29f00c28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29f5675700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00cac10 0x7f29f0146b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29f5675700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29f0151cc0 0x7f29f00c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29f5675700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f01e9780 con 0x7f29f00cac10 2023-03-03T09:52:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29f5675700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f01ecd10 con 0x7f29f0151cc0 2023-03-03T09:52:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29f5675700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f00c2e30 con 0x7f29f00b00b0 2023-03-03T09:52:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ee7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00cac10 0x7f29f0146b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ee7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00cac10 0x7f29f0146b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58098/0 (socket says 172.21.15.7:58098) 2023-03-03T09:52:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ee7fc700 1 -- 172.21.15.7:0/987626209 learned_addr learned my addr 172.21.15.7:0/987626209 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:52:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29eeffd700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29f00b00b0 0x7f29f00c28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ef7fe700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29f0151cc0 0x7f29f00c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ee7fc700 1 -- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29f00b00b0 msgr2=0x7f29f00c28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:52:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ee7fc700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29f00b00b0 0x7f29f00c28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ee7fc700 1 -- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29f0151cc0 msgr2=0x7f29f00c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:52:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ee7fc700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29f0151cc0 0x7f29f00c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ee7fc700 1 -- 172.21.15.7:0/987626209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e0009550 con 0x7f29f00cac10 2023-03-03T09:52:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29eeffd700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29f00b00b0 0x7f29f00c28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:52:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ef7fe700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29f0151cc0 0x7f29f00c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:52:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29ee7fc700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00cac10 0x7f29f0146b50 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f29e0004000 tx=0x7f29e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:52:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29d77fe700 1 -- 172.21.15.7:0/987626209 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0003d80 con 0x7f29f00cac10 2023-03-03T09:52:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29d77fe700 1 -- 172.21.15.7:0/987626209 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e001f070 con 0x7f29f00cac10 2023-03-03T09:52:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29f5675700 1 -- 172.21.15.7:0/987626209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f0154eb0 con 0x7f29f00cac10 2023-03-03T09:52:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.029+0000 7f29d77fe700 1 -- 172.21.15.7:0/987626209 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e0011870 con 0x7f29f00cac10 2023-03-03T09:52:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.030+0000 7f29f5675700 1 -- 172.21.15.7:0/987626209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f0154bd0 con 0x7f29f00cac10 2023-03-03T09:52:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.031+0000 7f29f5675700 1 -- 172.21.15.7:0/987626209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f0155190 con 0x7f29f00cac10 2023-03-03T09:52:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.031+0000 7f29d77fe700 1 -- 172.21.15.7:0/987626209 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f29e0017020 con 0x7f29f00cac10 2023-03-03T09:52:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.031+0000 7f29d77fe700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29dc0682f0 0x7f29dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:52:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.032+0000 7f29d77fe700 1 -- 172.21.15.7:0/987626209 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f29e0087950 con 0x7f29f00cac10 2023-03-03T09:52:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.032+0000 7f29eeffd700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29dc0682f0 0x7f29dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:52:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.032+0000 7f29eeffd700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29dc0682f0 0x7f29dc06a7a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f29e400a9b0 tx=0x7f29e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:52:40.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.036+0000 7f29d77fe700 1 -- 172.21.15.7:0/987626209 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f29e0053cf0 con 0x7f29f00cac10 2023-03-03T09:52:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.279+0000 7f29f5675700 1 -- 172.21.15.7:0/987626209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29f0074bb0 con 0x7f29f00cac10 2023-03-03T09:52:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.279+0000 7f29d77fe700 1 -- 172.21.15.7:0/987626209 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f29e0057310 con 0x7f29f00cac10 2023-03-03T09:52:40.281 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:52:40.281 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:52:40.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.281+0000 7f29f5675700 1 -- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29dc0682f0 msgr2=0x7f29dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:52:40.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.281+0000 7f29f5675700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29dc0682f0 0x7f29dc06a7a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f29e400a9b0 tx=0x7f29e4008040 comp rx=0 tx=0).stop 2023-03-03T09:52:40.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.282+0000 7f29f5675700 1 -- 172.21.15.7:0/987626209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00cac10 msgr2=0x7f29f0146b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:52:40.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.282+0000 7f29f5675700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00cac10 0x7f29f0146b50 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f29e0004000 tx=0x7f29e0004750 comp rx=0 tx=0).stop 2023-03-03T09:52:40.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.282+0000 7f29f5675700 1 -- 172.21.15.7:0/987626209 shutdown_connections 2023-03-03T09:52:40.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.282+0000 7f29f5675700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29f0151cc0 0x7f29f00c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:40.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.282+0000 7f29f5675700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29f00b00b0 0x7f29f00c28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:40.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.282+0000 7f29f5675700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29dc0682f0 0x7f29dc06a7a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:40.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.282+0000 7f29f5675700 1 --2- 172.21.15.7:0/987626209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29f00cac10 0x7f29f0146b50 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:52:40.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.282+0000 7f29f5675700 1 -- 172.21.15.7:0/987626209 >> 172.21.15.7:0/987626209 conn(0x7f29f006b980 msgr2=0x7f29f00c6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:52:40.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.282+0000 7f29f5675700 1 -- 172.21.15.7:0/987626209 shutdown_connections 2023-03-03T09:52:40.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:52:40.282+0000 7f29f5675700 1 -- 172.21.15.7:0/987626209 wait complete. 2023-03-03T09:52:40.285 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:52:40.295 INFO:tasks.mds_thrash.fs.[b]:waiting for 66 secs before thrashing 2023-03-03T09:52:51.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:52:51.593 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:21.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:53:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.291+0000 7f002afd8700 1 Processor -- start 2023-03-03T09:53:21.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.291+0000 7f002afd8700 1 -- start start 2023-03-03T09:53:21.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.291+0000 7f002afd8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f002406c480 0x7f00241581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.291+0000 7f002afd8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002406d430 con 0x7f0024151d40 2023-03-03T09:53:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f002afd8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00241586e0 con 0x7f00240c2930 2023-03-03T09:53:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f002afd8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002406c2a0 con 0x7f002406c480 2023-03-03T09:53:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f0028d74700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0024151d40 0x7f00240736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52116/0 (socket says 172.21.15.7:52116) 2023-03-03T09:53:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f0028d74700 1 -- 172.21.15.7:0/450661135 learned_addr learned my addr 172.21.15.7:0/450661135 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:53:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f0029575700 1 --2- 172.21.15.7:0/450661135 >> v2:172.21.15.7:3300/0 conn(0x7f002406c480 0x7f00241581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:21.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f00237fe700 1 -- 172.21.15.7:0/450661135 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960650193 0 0) 0x7f002406d430 con 0x7f0024151d40 2023-03-03T09:53:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f00237fe700 1 -- 172.21.15.7:0/450661135 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00140035b0 con 0x7f0024151d40 2023-03-03T09:53:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f0029575700 1 -- 172.21.15.7:0/450661135 >> v1:172.21.15.143:6789/0 conn(0x7f00240c2930 legacy=0x7f0024157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:53:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f0029575700 1 -- 172.21.15.7:0/450661135 >> v1:172.21.15.7:6789/0 conn(0x7f0024151d40 legacy=0x7f00240736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f0029575700 1 -- 172.21.15.7:0/450661135 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002406cb00 con 0x7f002406c480 2023-03-03T09:53:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f0029575700 1 --2- 172.21.15.7:0/450661135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f002406c480 0x7f00241581a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f001c009dc0 tx=0x7f001c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bef8e6be9212fff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:53:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f00237fe700 1 -- 172.21.15.7:0/450661135 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c00e070 con 0x7f002406c480 2023-03-03T09:53:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f00237fe700 1 -- 172.21.15.7:0/450661135 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001c00baa0 con 0x7f002406c480 2023-03-03T09:53:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.292+0000 7f00237fe700 1 -- 172.21.15.7:0/450661135 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c00be10 con 0x7f002406c480 2023-03-03T09:53:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.293+0000 7f002afd8700 1 -- 172.21.15.7:0/450661135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f002406c480 msgr2=0x7f00241581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.293+0000 7f002afd8700 1 --2- 172.21.15.7:0/450661135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f002406c480 0x7f00241581a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f001c009dc0 tx=0x7f001c00b330 comp rx=0 tx=0).stop 2023-03-03T09:53:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.293+0000 7f002afd8700 1 -- 172.21.15.7:0/450661135 shutdown_connections 2023-03-03T09:53:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.293+0000 7f002afd8700 1 --2- 172.21.15.7:0/450661135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f002406c480 0x7f00241581a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.293+0000 7f002afd8700 1 -- 172.21.15.7:0/450661135 >> 172.21.15.7:0/450661135 conn(0x7f00240a94d0 msgr2=0x7f00240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:53:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.293+0000 7f002afd8700 1 -- 172.21.15.7:0/450661135 shutdown_connections 2023-03-03T09:53:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.293+0000 7f002afd8700 1 -- 172.21.15.7:0/450661135 wait complete. 2023-03-03T09:53:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.294+0000 7f002afd8700 1 Processor -- start 2023-03-03T09:53:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.294+0000 7f002afd8700 1 -- start start 2023-03-03T09:53:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f002afd8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002406c480 0x7f00241ed420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f002afd8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00240c2930 0x7f00240ab930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f002afd8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0024151d40 0x7f00240abe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f002afd8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00240ab4c0 con 0x7f0024151d40 2023-03-03T09:53:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f002afd8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00241ee0f0 con 0x7f002406c480 2023-03-03T09:53:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f002afd8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00241ede10 con 0x7f00240c2930 2023-03-03T09:53:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f0029575700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0024151d40 0x7f00240abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f0029575700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0024151d40 0x7f00240abe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58146/0 (socket says 172.21.15.7:58146) 2023-03-03T09:53:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f0029575700 1 -- 172.21.15.7:0/1671546587 learned_addr learned my addr 172.21.15.7:0/1671546587 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:53:21.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f0028d74700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002406c480 0x7f00241ed420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:21.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f0023fff700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00240c2930 0x7f00240ab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:21.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f0029575700 1 -- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00240c2930 msgr2=0x7f00240ab930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:21.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.295+0000 7f0029575700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00240c2930 0x7f00240ab930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:21.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.296+0000 7f0029575700 1 -- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002406c480 msgr2=0x7f00241ed420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:21.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.296+0000 7f0029575700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002406c480 0x7f00241ed420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.296+0000 7f0029575700 1 -- 172.21.15.7:0/1671546587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c009a70 con 0x7f0024151d40 2023-03-03T09:53:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.296+0000 7f0029575700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0024151d40 0x7f00240abe70 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f001c009470 tx=0x7f001c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:53:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.296+0000 7f00217fa700 1 -- 172.21.15.7:0/1671546587 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c003ea0 con 0x7f0024151d40 2023-03-03T09:53:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.296+0000 7f00217fa700 1 -- 172.21.15.7:0/1671546587 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001c012650 con 0x7f0024151d40 2023-03-03T09:53:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.296+0000 7f002afd8700 1 -- 172.21.15.7:0/1671546587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002406d0c0 con 0x7f0024151d40 2023-03-03T09:53:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.296+0000 7f00217fa700 1 -- 172.21.15.7:0/1671546587 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c012a00 con 0x7f0024151d40 2023-03-03T09:53:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.296+0000 7f002afd8700 1 -- 172.21.15.7:0/1671546587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002406cde0 con 0x7f0024151d40 2023-03-03T09:53:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.297+0000 7f002afd8700 1 -- 172.21.15.7:0/1671546587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0024074bb0 con 0x7f0024151d40 2023-03-03T09:53:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.297+0000 7f00217fa700 1 -- 172.21.15.7:0/1671546587 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f001c012b60 con 0x7f0024151d40 2023-03-03T09:53:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.298+0000 7f00217fa700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 0x7f000806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.298+0000 7f00217fa700 1 -- 172.21.15.7:0/1671546587 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f001c088420 con 0x7f0024151d40 2023-03-03T09:53:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.298+0000 7f0028d74700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 0x7f000806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.299+0000 7f0028d74700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 0x7f000806a7a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0018007fd0 tx=0x7f0018009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:53:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.303+0000 7f00217fa700 1 -- 172.21.15.7:0/1671546587 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f001c054670 con 0x7f0024151d40 2023-03-03T09:53:21.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.542+0000 7f002afd8700 1 -- 172.21.15.7:0/1671546587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00240ab600 con 0x7f0024151d40 2023-03-03T09:53:21.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.543+0000 7f00217fa700 1 -- 172.21.15.7:0/1671546587 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f001c057c90 con 0x7f0024151d40 2023-03-03T09:53:21.545 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:53:21.546 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:53:21.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 -- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 msgr2=0x7f000806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:21.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 0x7f000806a7a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0018007fd0 tx=0x7f0018009040 comp rx=0 tx=0).stop 2023-03-03T09:53:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 -- 172.21.15.7:0/1671546587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0024151d40 msgr2=0x7f00240abe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0024151d40 0x7f00240abe70 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f001c009470 tx=0x7f001c004ad0 comp rx=0 tx=0).stop 2023-03-03T09:53:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 -- 172.21.15.7:0/1671546587 shutdown_connections 2023-03-03T09:53:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 0x7f000806a7a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0024151d40 0x7f00240abe70 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002406c480 0x7f00241ed420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 --2- 172.21.15.7:0/1671546587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00240c2930 0x7f00240ab930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 -- 172.21.15.7:0/1671546587 >> 172.21.15.7:0/1671546587 conn(0x7f00240a94d0 msgr2=0x7f00240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:53:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 -- 172.21.15.7:0/1671546587 shutdown_connections 2023-03-03T09:53:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:21.545+0000 7f002afd8700 1 -- 172.21.15.7:0/1671546587 wait complete. 2023-03-03T09:53:21.550 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:53:21.559 INFO:tasks.mds_thrash.fs.[a]:waiting for 116 secs before thrashing 2023-03-03T09:53:21.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:21.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:46.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:53:46.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.433+0000 7fee6fc78700 1 Processor -- start 2023-03-03T09:53:46.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.433+0000 7fee6fc78700 1 -- start start 2023-03-03T09:53:46.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.434+0000 7fee6fc78700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fee68151d40 0x7fee680736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:46.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.434+0000 7fee6fc78700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fee680c2930 0x7fee68157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:46.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.434+0000 7fee6fc78700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6806d430 con 0x7fee68073c30 2023-03-03T09:53:46.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.434+0000 7fee6fc78700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6806c2a0 con 0x7fee680c2930 2023-03-03T09:53:46.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.434+0000 7fee6fc78700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee681586e0 con 0x7fee68151d40 2023-03-03T09:53:46.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.434+0000 7fee6d213700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fee680c2930 0x7fee68157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:46.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.434+0000 7fee6e215700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fee68073c30 0x7fee68157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52158/0 (socket says 172.21.15.7:52158) 2023-03-03T09:53:46.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.434+0000 7fee6e215700 1 -- 172.21.15.7:0/3789033561 learned_addr learned my addr 172.21.15.7:0/3789033561 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:53:46.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6da14700 1 --2- 172.21.15.7:0/3789033561 >> v2:172.21.15.143:3300/0 conn(0x7fee68151d40 0x7fee680736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6ca12700 1 -- 172.21.15.7:0/3789033561 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537712247 0 0) 0x7fee6806d430 con 0x7fee68073c30 2023-03-03T09:53:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6ca12700 1 -- 172.21.15.7:0/3789033561 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee540035b0 con 0x7fee68073c30 2023-03-03T09:53:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6d213700 1 -- 172.21.15.7:0/3789033561 >> v2:172.21.15.143:3300/0 conn(0x7fee68151d40 msgr2=0x7fee680736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:53:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6d213700 1 --2- 172.21.15.7:0/3789033561 >> v2:172.21.15.143:3300/0 conn(0x7fee68151d40 0x7fee680736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6d213700 1 -- 172.21.15.7:0/3789033561 >> v1:172.21.15.7:6789/0 conn(0x7fee68073c30 legacy=0x7fee68157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6d213700 1 -- 172.21.15.7:0/3789033561 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6806cb00 con 0x7fee680c2930 2023-03-03T09:53:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6da14700 1 --2- 172.21.15.7:0/3789033561 >> v2:172.21.15.143:3300/0 conn(0x7fee68151d40 0x7fee680736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:53:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6d213700 1 --2- 172.21.15.7:0/3789033561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee680c2930 0x7fee68157a90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fee58009840 tx=0x7fee58009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19e446d74c6e599 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:53:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6ca12700 1 -- 172.21.15.7:0/3789033561 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58011830 con 0x7fee680c2930 2023-03-03T09:53:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6ca12700 1 -- 172.21.15.7:0/3789033561 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee58011990 con 0x7fee680c2930 2023-03-03T09:53:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.435+0000 7fee6ca12700 1 -- 172.21.15.7:0/3789033561 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58011d20 con 0x7fee680c2930 2023-03-03T09:53:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.436+0000 7fee6fc78700 1 -- 172.21.15.7:0/3789033561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee680c2930 msgr2=0x7fee68157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.436+0000 7fee6fc78700 1 --2- 172.21.15.7:0/3789033561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee680c2930 0x7fee68157a90 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fee58009840 tx=0x7fee58009b50 comp rx=0 tx=0).stop 2023-03-03T09:53:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.436+0000 7fee6fc78700 1 -- 172.21.15.7:0/3789033561 shutdown_connections 2023-03-03T09:53:46.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.436+0000 7fee6fc78700 1 --2- 172.21.15.7:0/3789033561 >> v2:172.21.15.143:3300/0 conn(0x7fee68151d40 0x7fee680736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:46.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.436+0000 7fee6fc78700 1 --2- 172.21.15.7:0/3789033561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee680c2930 0x7fee68157a90 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:46.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.436+0000 7fee6fc78700 1 -- 172.21.15.7:0/3789033561 >> 172.21.15.7:0/3789033561 conn(0x7fee680a94d0 msgr2=0x7fee680a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:53:46.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.436+0000 7fee6fc78700 1 -- 172.21.15.7:0/3789033561 shutdown_connections 2023-03-03T09:53:46.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.436+0000 7fee6fc78700 1 -- 172.21.15.7:0/3789033561 wait complete. 2023-03-03T09:53:46.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.437+0000 7fee6fc78700 1 Processor -- start 2023-03-03T09:53:46.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.437+0000 7fee6fc78700 1 -- start start 2023-03-03T09:53:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6fc78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee68073c30 0x7fee681e0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6fc78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee680c2930 0x7fee6814eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6fc78700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee68151d40 0x7fee680c42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6fc78700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680c0ff0 con 0x7fee68151d40 2023-03-03T09:53:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6fc78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee681df550 con 0x7fee68073c30 2023-03-03T09:53:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6fc78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee681eae00 con 0x7fee680c2930 2023-03-03T09:53:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6e215700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee68151d40 0x7fee680c42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6e215700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee68151d40 0x7fee680c42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58184/0 (socket says 172.21.15.7:58184) 2023-03-03T09:53:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6e215700 1 -- 172.21.15.7:0/715368153 learned_addr learned my addr 172.21.15.7:0/715368153 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:53:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6da14700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee68073c30 0x7fee681e0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6d213700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee680c2930 0x7fee6814eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6e215700 1 -- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee680c2930 msgr2=0x7fee6814eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6e215700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee680c2930 0x7fee6814eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6e215700 1 -- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee68073c30 msgr2=0x7fee681e0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6e215700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee68073c30 0x7fee681e0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6e215700 1 -- 172.21.15.7:0/715368153 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee580094a0 con 0x7fee68151d40 2023-03-03T09:53:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6d213700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee680c2930 0x7fee6814eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:53:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.438+0000 7fee6e215700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee68151d40 0x7fee680c42c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fee6000caa0 tx=0x7fee6000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:53:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.439+0000 7fee5e7fc700 1 -- 172.21.15.7:0/715368153 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6000a9a0 con 0x7fee68151d40 2023-03-03T09:53:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.439+0000 7fee5e7fc700 1 -- 172.21.15.7:0/715368153 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6000ab00 con 0x7fee68151d40 2023-03-03T09:53:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.439+0000 7fee6fc78700 1 -- 172.21.15.7:0/715368153 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6806d0c0 con 0x7fee68151d40 2023-03-03T09:53:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.439+0000 7fee5e7fc700 1 -- 172.21.15.7:0/715368153 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6000ae70 con 0x7fee68151d40 2023-03-03T09:53:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.439+0000 7fee6fc78700 1 -- 172.21.15.7:0/715368153 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6806cde0 con 0x7fee68151d40 2023-03-03T09:53:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.440+0000 7fee6fc78700 1 -- 172.21.15.7:0/715368153 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6806bea0 con 0x7fee68151d40 2023-03-03T09:53:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.440+0000 7fee5e7fc700 1 -- 172.21.15.7:0/715368153 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fee60019030 con 0x7fee68151d40 2023-03-03T09:53:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.440+0000 7fee5e7fc700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee4c068320 0x7fee4c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:53:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.440+0000 7fee5e7fc700 1 -- 172.21.15.7:0/715368153 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fee60085ba0 con 0x7fee68151d40 2023-03-03T09:53:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.441+0000 7fee6da14700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee4c068320 0x7fee4c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:53:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.441+0000 7fee6da14700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee4c068320 0x7fee4c06a7d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fee64006f10 tx=0x7fee64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:53:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.445+0000 7fee5e7fc700 1 -- 172.21.15.7:0/715368153 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fee6004b5e0 con 0x7fee68151d40 2023-03-03T09:53:46.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.684+0000 7fee6fc78700 1 -- 172.21.15.7:0/715368153 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee68074bb0 con 0x7fee68151d40 2023-03-03T09:53:46.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.685+0000 7fee5e7fc700 1 -- 172.21.15.7:0/715368153 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fee60051ff0 con 0x7fee68151d40 2023-03-03T09:53:46.687 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:53:46.687 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:53:46.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 -- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee4c068320 msgr2=0x7fee4c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:46.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee4c068320 0x7fee4c06a7d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fee64006f10 tx=0x7fee64008040 comp rx=0 tx=0).stop 2023-03-03T09:53:46.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 -- 172.21.15.7:0/715368153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee68151d40 msgr2=0x7fee680c42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:53:46.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee68151d40 0x7fee680c42c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fee6000caa0 tx=0x7fee6000cdb0 comp rx=0 tx=0).stop 2023-03-03T09:53:46.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 -- 172.21.15.7:0/715368153 shutdown_connections 2023-03-03T09:53:46.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee4c068320 0x7fee4c06a7d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:46.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee68151d40 0x7fee680c42c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:46.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee68073c30 0x7fee681e0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:46.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 --2- 172.21.15.7:0/715368153 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee680c2930 0x7fee6814eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:53:46.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 -- 172.21.15.7:0/715368153 >> 172.21.15.7:0/715368153 conn(0x7fee680a94d0 msgr2=0x7fee68157290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:53:46.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 -- 172.21.15.7:0/715368153 shutdown_connections 2023-03-03T09:53:46.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:53:46.687+0000 7fee6fc78700 1 -- 172.21.15.7:0/715368153 wait complete. 2023-03-03T09:53:46.690 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:53:46.700 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-03-03T09:53:51.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:53:51.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:21.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:21.794 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:46.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:54:46.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.839+0000 7f12dd19f700 1 Processor -- start 2023-03-03T09:54:46.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.840+0000 7f12dd19f700 1 -- start start 2023-03-03T09:54:46.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.840+0000 7f12dd19f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12d814e820 0x7f12d80c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.840+0000 7f12dd19f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f12d80ac620 0x7f12d80c6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.840+0000 7f12dd19f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d80c6a50 0x7f12d8151170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.840+0000 7f12dd19f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d8070a90 con 0x7f12d814e820 2023-03-03T09:54:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.840+0000 7f12dd19f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d80c3aa0 con 0x7f12d80ac620 2023-03-03T09:54:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.840+0000 7f12dd19f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d80715b0 con 0x7f12d80c6a50 2023-03-03T09:54:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12d814e820 0x7f12d80c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12d814e820 0x7f12d80c32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58238/0 (socket says 172.21.15.7:58238) 2023-03-03T09:54:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d7fff700 1 -- 172.21.15.7:0/1722674827 learned_addr learned my addr 172.21.15.7:0/1722674827 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:54:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12cffff700 1 --2- 172.21.15.7:0/1722674827 >> v2:172.21.15.143:3300/0 conn(0x7f12d80ac620 0x7f12d80c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12dc99e700 1 --2- 172.21.15.7:0/1722674827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d80c6a50 0x7f12d8151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d7fff700 1 -- 172.21.15.7:0/1722674827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d80c6a50 msgr2=0x7f12d8151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:54:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d7fff700 1 --2- 172.21.15.7:0/1722674827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d80c6a50 0x7f12d8151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d7fff700 1 -- 172.21.15.7:0/1722674827 >> v2:172.21.15.143:3300/0 conn(0x7f12d80ac620 msgr2=0x7f12d80c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:54:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d7fff700 1 --2- 172.21.15.7:0/1722674827 >> v2:172.21.15.143:3300/0 conn(0x7f12d80ac620 0x7f12d80c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d7fff700 1 -- 172.21.15.7:0/1722674827 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d80720d0 con 0x7f12d814e820 2023-03-03T09:54:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d7fff700 1 --2- 172.21.15.7:0/1722674827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 0x7f12d80c32c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f12d8070000 tx=0x7f12c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0365aed405abb4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:54:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d77fe700 1 -- 172.21.15.7:0/1722674827 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c800b860 con 0x7f12d814e820 2023-03-03T09:54:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d77fe700 1 -- 172.21.15.7:0/1722674827 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c800b9c0 con 0x7f12d814e820 2023-03-03T09:54:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.841+0000 7f12d77fe700 1 -- 172.21.15.7:0/1722674827 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c800bd50 con 0x7f12d814e820 2023-03-03T09:54:46.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.842+0000 7f12dd19f700 1 -- 172.21.15.7:0/1722674827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 msgr2=0x7f12d80c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:46.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.842+0000 7f12dd19f700 1 --2- 172.21.15.7:0/1722674827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 0x7f12d80c32c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f12d8070000 tx=0x7f12c8009c00 comp rx=0 tx=0).stop 2023-03-03T09:54:46.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.842+0000 7f12dd19f700 1 -- 172.21.15.7:0/1722674827 shutdown_connections 2023-03-03T09:54:46.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.842+0000 7f12dd19f700 1 --2- 172.21.15.7:0/1722674827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d80c6a50 0x7f12d8151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.842+0000 7f12dd19f700 1 --2- 172.21.15.7:0/1722674827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 0x7f12d80c32c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.842+0000 7f12dd19f700 1 --2- 172.21.15.7:0/1722674827 >> v2:172.21.15.143:3300/0 conn(0x7f12d80ac620 0x7f12d80c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.842+0000 7f12dd19f700 1 -- 172.21.15.7:0/1722674827 >> 172.21.15.7:0/1722674827 conn(0x7f12d80c48b0 msgr2=0x7f12d80c7180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:54:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.842+0000 7f12dd19f700 1 -- 172.21.15.7:0/1722674827 shutdown_connections 2023-03-03T09:54:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.842+0000 7f12dd19f700 1 -- 172.21.15.7:0/1722674827 wait complete. 2023-03-03T09:54:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.843+0000 7f12dd19f700 1 Processor -- start 2023-03-03T09:54:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.843+0000 7f12dd19f700 1 -- start start 2023-03-03T09:54:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dd19f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d80ac620 0x7f12d81ed990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dd19f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d80c6a50 0x7f12d80ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dd19f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 0x7f12d814d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dd19f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d80ac1a0 con 0x7f12d814e820 2023-03-03T09:54:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dd19f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d8141690 con 0x7f12d80ac620 2023-03-03T09:54:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dd19f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d80abec0 con 0x7f12d80c6a50 2023-03-03T09:54:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dc99e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 0x7f12d814d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12d7fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d80ac620 0x7f12d81ed990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12cffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d80c6a50 0x7f12d80ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dc99e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 0x7f12d814d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58248/0 (socket says 172.21.15.7:58248) 2023-03-03T09:54:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dc99e700 1 -- 172.21.15.7:0/866232356 learned_addr learned my addr 172.21.15.7:0/866232356 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:54:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dc99e700 1 -- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d80c6a50 msgr2=0x7f12d80ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dc99e700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d80c6a50 0x7f12d80ab8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dc99e700 1 -- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d80ac620 msgr2=0x7f12d81ed990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dc99e700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d80ac620 0x7f12d81ed990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.844+0000 7f12dc99e700 1 -- 172.21.15.7:0/866232356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c8009550 con 0x7f12d814e820 2023-03-03T09:54:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.845+0000 7f12dc99e700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 0x7f12d814d9d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f12d0007e40 tx=0x7f12d000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:54:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.845+0000 7f12d57fa700 1 -- 172.21.15.7:0/866232356 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d000ed30 con 0x7f12d814e820 2023-03-03T09:54:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.845+0000 7f12d57fa700 1 -- 172.21.15.7:0/866232356 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d0014420 con 0x7f12d814e820 2023-03-03T09:54:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.845+0000 7f12dd19f700 1 -- 172.21.15.7:0/866232356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d80717e0 con 0x7f12d814e820 2023-03-03T09:54:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.845+0000 7f12d57fa700 1 -- 172.21.15.7:0/866232356 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d00147d0 con 0x7f12d814e820 2023-03-03T09:54:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.845+0000 7f12dd19f700 1 -- 172.21.15.7:0/866232356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d814ebf0 con 0x7f12d814e820 2023-03-03T09:54:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.846+0000 7f12dd19f700 1 -- 172.21.15.7:0/866232356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d80c3800 con 0x7f12d814e820 2023-03-03T09:54:46.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.846+0000 7f12d57fa700 1 -- 172.21.15.7:0/866232356 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f12d0012050 con 0x7f12d814e820 2023-03-03T09:54:46.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.847+0000 7f12d57fa700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12b80682f0 0x7f12b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:54:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.847+0000 7f12d57fa700 1 -- 172.21.15.7:0/866232356 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f12d0086670 con 0x7f12d814e820 2023-03-03T09:54:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.847+0000 7f12d7fff700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12b80682f0 0x7f12b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:54:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.847+0000 7f12d7fff700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12b80682f0 0x7f12b806a7a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f12c8000c00 tx=0x7f12c8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:54:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:46.851+0000 7f12d57fa700 1 -- 172.21.15.7:0/866232356 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f12d0052a10 con 0x7f12d814e820 2023-03-03T09:54:47.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.089+0000 7f12dd19f700 1 -- 172.21.15.7:0/866232356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12d8061f40 con 0x7f12d814e820 2023-03-03T09:54:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.089+0000 7f12d57fa700 1 -- 172.21.15.7:0/866232356 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f12d0056030 con 0x7f12d814e820 2023-03-03T09:54:47.091 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:54:47.091 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:54:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.091+0000 7f12dd19f700 1 -- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12b80682f0 msgr2=0x7f12b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.091+0000 7f12dd19f700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12b80682f0 0x7f12b806a7a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f12c8000c00 tx=0x7f12c8008030 comp rx=0 tx=0).stop 2023-03-03T09:54:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.091+0000 7f12dd19f700 1 -- 172.21.15.7:0/866232356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 msgr2=0x7f12d814d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:54:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.091+0000 7f12dd19f700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 0x7f12d814d9d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f12d0007e40 tx=0x7f12d000e3c0 comp rx=0 tx=0).stop 2023-03-03T09:54:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.092+0000 7f12dd19f700 1 -- 172.21.15.7:0/866232356 shutdown_connections 2023-03-03T09:54:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.092+0000 7f12dd19f700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12b80682f0 0x7f12b806a7a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.092+0000 7f12dd19f700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d814e820 0x7f12d814d9d0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.092+0000 7f12dd19f700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d80ac620 0x7f12d81ed990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.092+0000 7f12dd19f700 1 --2- 172.21.15.7:0/866232356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d80c6a50 0x7f12d80ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:54:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.092+0000 7f12dd19f700 1 -- 172.21.15.7:0/866232356 >> 172.21.15.7:0/866232356 conn(0x7f12d80c48b0 msgr2=0x7f12d806afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:54:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.092+0000 7f12dd19f700 1 -- 172.21.15.7:0/866232356 shutdown_connections 2023-03-03T09:54:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:54:47.092+0000 7f12dd19f700 1 -- 172.21.15.7:0/866232356 wait complete. 2023-03-03T09:54:47.095 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:54:47.105 INFO:tasks.mds_thrash.fs.[b]:waiting for 54 secs before thrashing 2023-03-03T09:54:52.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:54:52.129 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:17.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:55:17.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.698+0000 7f6451f59700 1 Processor -- start 2023-03-03T09:55:17.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.698+0000 7f6451f59700 1 -- start start 2023-03-03T09:55:17.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.698+0000 7f6451f59700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f644c071e80 0x7f644c151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:17.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.698+0000 7f6451f59700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f644c072250 0x7f644c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:17.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.698+0000 7f6451f59700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c0c1ce0 0x7f644c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.698+0000 7f6451f59700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644c1510d0 con 0x7f644c072250 2023-03-03T09:55:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.698+0000 7f6451f59700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644c0c3740 con 0x7f644c071e80 2023-03-03T09:55:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.698+0000 7f6451f59700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c070670 con 0x7f644c0c1ce0 2023-03-03T09:55:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f644c072250 0x7f644c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f644c072250 0x7f644c14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58272/0 (socket says 172.21.15.7:58272) 2023-03-03T09:55:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644affd700 1 -- 172.21.15.7:0/1197603781 learned_addr learned my addr 172.21.15.7:0/1197603781 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:55:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644b7fe700 1 --2- 172.21.15.7:0/1197603781 >> v2:172.21.15.143:3300/0 conn(0x7f644c071e80 0x7f644c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644bfff700 1 --2- 172.21.15.7:0/1197603781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c0c1ce0 0x7f644c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644affd700 1 -- 172.21.15.7:0/1197603781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c0c1ce0 msgr2=0x7f644c14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:55:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644affd700 1 --2- 172.21.15.7:0/1197603781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c0c1ce0 0x7f644c14e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644affd700 1 -- 172.21.15.7:0/1197603781 >> v2:172.21.15.143:3300/0 conn(0x7f644c071e80 msgr2=0x7f644c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:55:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644affd700 1 --2- 172.21.15.7:0/1197603781 >> v2:172.21.15.143:3300/0 conn(0x7f644c071e80 0x7f644c151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644affd700 1 -- 172.21.15.7:0/1197603781 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c0c39e0 con 0x7f644c072250 2023-03-03T09:55:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644affd700 1 --2- 172.21.15.7:0/1197603781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c072250 0x7f644c14e1f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f644c06bd10 tx=0x7f6434009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ed394970e382dd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:55:17.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644a7fc700 1 -- 172.21.15.7:0/1197603781 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643400b870 con 0x7f644c072250 2023-03-03T09:55:17.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644a7fc700 1 -- 172.21.15.7:0/1197603781 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643400b9d0 con 0x7f644c072250 2023-03-03T09:55:17.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.699+0000 7f644a7fc700 1 -- 172.21.15.7:0/1197603781 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643400bd60 con 0x7f644c072250 2023-03-03T09:55:17.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.700+0000 7f6451f59700 1 -- 172.21.15.7:0/1197603781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c072250 msgr2=0x7f644c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:17.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.700+0000 7f6451f59700 1 --2- 172.21.15.7:0/1197603781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c072250 0x7f644c14e1f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f644c06bd10 tx=0x7f6434009bd0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.700+0000 7f6451f59700 1 -- 172.21.15.7:0/1197603781 shutdown_connections 2023-03-03T09:55:17.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.700+0000 7f6451f59700 1 --2- 172.21.15.7:0/1197603781 >> v2:172.21.15.143:3300/0 conn(0x7f644c071e80 0x7f644c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.700+0000 7f6451f59700 1 --2- 172.21.15.7:0/1197603781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c0c1ce0 0x7f644c14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.700+0000 7f6451f59700 1 --2- 172.21.15.7:0/1197603781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c072250 0x7f644c14e1f0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.700+0000 7f6451f59700 1 -- 172.21.15.7:0/1197603781 >> 172.21.15.7:0/1197603781 conn(0x7f644c0a94d0 msgr2=0x7f644c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:55:17.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.700+0000 7f6451f59700 1 -- 172.21.15.7:0/1197603781 shutdown_connections 2023-03-03T09:55:17.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.700+0000 7f6451f59700 1 -- 172.21.15.7:0/1197603781 wait complete. 2023-03-03T09:55:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.701+0000 7f6451f59700 1 Processor -- start 2023-03-03T09:55:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.701+0000 7f6451f59700 1 -- start start 2023-03-03T09:55:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f6451f59700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c071e80 0x7f644c1e90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f6451f59700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f644c072250 0x7f644c1ebbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f6451f59700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c0c1ce0 0x7f644c141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f6451f59700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c1ec100 con 0x7f644c0c1ce0 2023-03-03T09:55:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f6451f59700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c144de0 con 0x7f644c072250 2023-03-03T09:55:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f6451f59700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c144540 con 0x7f644c071e80 2023-03-03T09:55:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c0c1ce0 0x7f644c141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c0c1ce0 0x7f644c141330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58282/0 (socket says 172.21.15.7:58282) 2023-03-03T09:55:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644bfff700 1 -- 172.21.15.7:0/56840472 learned_addr learned my addr 172.21.15.7:0/56840472 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:55:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644b7fe700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c071e80 0x7f644c1e90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644affd700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f644c072250 0x7f644c1ebbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644bfff700 1 -- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c071e80 msgr2=0x7f644c1e90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644bfff700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c071e80 0x7f644c1e90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644bfff700 1 -- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f644c072250 msgr2=0x7f644c1ebbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644bfff700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f644c072250 0x7f644c1ebbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644bfff700 1 -- 172.21.15.7:0/56840472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6434009550 con 0x7f644c0c1ce0 2023-03-03T09:55:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.702+0000 7f644bfff700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c0c1ce0 0x7f644c141330 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f644400cc80 tx=0x7f644400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:55:17.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.703+0000 7f6443fff700 1 -- 172.21.15.7:0/56840472 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444003a20 con 0x7f644c0c1ce0 2023-03-03T09:55:17.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.703+0000 7f6443fff700 1 -- 172.21.15.7:0/56840472 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6444003ba0 con 0x7f644c0c1ce0 2023-03-03T09:55:17.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.703+0000 7f6451f59700 1 -- 172.21.15.7:0/56840472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644c06c9a0 con 0x7f644c0c1ce0 2023-03-03T09:55:17.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.703+0000 7f6443fff700 1 -- 172.21.15.7:0/56840472 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644400a9f0 con 0x7f644c0c1ce0 2023-03-03T09:55:17.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.703+0000 7f6451f59700 1 -- 172.21.15.7:0/56840472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644c151300 con 0x7f644c0c1ce0 2023-03-03T09:55:17.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.704+0000 7f6451f59700 1 -- 172.21.15.7:0/56840472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644c06cb30 con 0x7f644c0c1ce0 2023-03-03T09:55:17.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.704+0000 7f6443fff700 1 -- 172.21.15.7:0/56840472 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f644401f070 con 0x7f644c0c1ce0 2023-03-03T09:55:17.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.704+0000 7f6443fff700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f642c0682f0 0x7f642c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:17.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.705+0000 7f6443fff700 1 -- 172.21.15.7:0/56840472 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6444086120 con 0x7f644c0c1ce0 2023-03-03T09:55:17.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.705+0000 7f644b7fe700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f642c0682f0 0x7f642c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:17.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.705+0000 7f644b7fe700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f642c0682f0 0x7f642c06a7a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f644c141eb0 tx=0x7f643c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:55:17.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.709+0000 7f6443fff700 1 -- 172.21.15.7:0/56840472 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f64440523e0 con 0x7f644c0c1ce0 2023-03-03T09:55:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.945+0000 7f6451f59700 1 -- 172.21.15.7:0/56840472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f644c074bb0 con 0x7f644c0c1ce0 2023-03-03T09:55:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.946+0000 7f6443fff700 1 -- 172.21.15.7:0/56840472 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6444055a00 con 0x7f644c0c1ce0 2023-03-03T09:55:17.947 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:55:17.947 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:55:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 -- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f642c0682f0 msgr2=0x7f642c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f642c0682f0 0x7f642c06a7a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f644c141eb0 tx=0x7f643c008040 comp rx=0 tx=0).stop 2023-03-03T09:55:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 -- 172.21.15.7:0/56840472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c0c1ce0 msgr2=0x7f644c141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c0c1ce0 0x7f644c141330 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f644400cc80 tx=0x7f644400cf90 comp rx=0 tx=0).stop 2023-03-03T09:55:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 -- 172.21.15.7:0/56840472 shutdown_connections 2023-03-03T09:55:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f642c0682f0 0x7f642c06a7a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f644c0c1ce0 0x7f644c141330 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f644c071e80 0x7f644c1e90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 --2- 172.21.15.7:0/56840472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f644c072250 0x7f644c1ebbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 -- 172.21.15.7:0/56840472 >> 172.21.15.7:0/56840472 conn(0x7f644c0a94d0 msgr2=0x7f644c06d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:55:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 -- 172.21.15.7:0/56840472 shutdown_connections 2023-03-03T09:55:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:17.948+0000 7f6451f59700 1 -- 172.21.15.7:0/56840472 wait complete. 2023-03-03T09:55:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:55:17.961 INFO:tasks.mds_thrash.fs.[a]:waiting for 69 secs before thrashing 2023-03-03T09:55:22.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:22.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:41.106 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:55:41.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.243+0000 7fe22d6f2700 1 Processor -- start 2023-03-03T09:55:41.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.243+0000 7fe22d6f2700 1 -- start start 2023-03-03T09:55:41.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.244+0000 7fe22d6f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe22814b060 0x7fe22806f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:41.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.244+0000 7fe22d6f2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe2280c1850 0x7fe2280c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:41.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.244+0000 7fe22d6f2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe22806fd40 con 0x7fe2280aa170 2023-03-03T09:55:41.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.244+0000 7fe22d6f2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2280c4100 con 0x7fe2280c1850 2023-03-03T09:55:41.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.244+0000 7fe22d6f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe228154790 con 0x7fe22814b060 2023-03-03T09:55:41.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.244+0000 7fe2277fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe2280aa170 0x7fe2280aa5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52288/0 (socket says 172.21.15.7:52288) 2023-03-03T09:55:41.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.244+0000 7fe2277fe700 1 -- 172.21.15.7:0/2980754515 learned_addr learned my addr 172.21.15.7:0/2980754515 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:55:41.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249942304 0 0) 0x7fe22806fd40 con 0x7fe2280aa170 2023-03-03T09:55:41.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2100035b0 con 0x7fe2280aa170 2023-03-03T09:55:41.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe2267fc700 1 --2- 172.21.15.7:0/2980754515 >> v2:172.21.15.143:3300/0 conn(0x7fe2280c1850 0x7fe2280c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:41.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe226ffd700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe22814b060 0x7fe22806f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:41.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1102952371 0 0) 0x7fe2100035b0 con 0x7fe2280aa170 2023-03-03T09:55:41.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe22814b060 msgr2=0x7fe22806f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:55:41.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe22814b060 0x7fe22806f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 >> v2:172.21.15.143:3300/0 conn(0x7fe2280c1850 msgr2=0x7fe2280c1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:55:41.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 --2- 172.21.15.7:0/2980754515 >> v2:172.21.15.143:3300/0 conn(0x7fe2280c1850 0x7fe2280c1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe228154a70 con 0x7fe2280aa170 2023-03-03T09:55:41.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe2200019f0 con 0x7fe2280aa170 2023-03-03T09:55:41.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 >> v1:172.21.15.7:6789/0 conn(0x7fe2280aa170 legacy=0x7fe2280aa5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:41.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe210003a70 0x7fe210003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:41.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe210004850 0x7fe210004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:41.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe22814b060 0x7fe210005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:41.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2280c4100 con 0x7fe210003a70 2023-03-03T09:55:41.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe228154790 con 0x7fe210004850 2023-03-03T09:55:41.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21000ba40 con 0x7fe22814b060 2023-03-03T09:55:41.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe226ffd700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe210003a70 0x7fe210003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:41.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe2267fc700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe210004850 0x7fe210004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:41.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe2277fe700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe22814b060 0x7fe210005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:41.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe226ffd700 1 -- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe22814b060 msgr2=0x7fe210005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:55:41.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe226ffd700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe22814b060 0x7fe210005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe226ffd700 1 -- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe210004850 msgr2=0x7fe210004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:55:41.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.245+0000 7fe226ffd700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe210004850 0x7fe210004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe226ffd700 1 -- 172.21.15.7:0/2980754515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2100035b0 con 0x7fe210003a70 2023-03-03T09:55:41.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe2277fe700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe22814b060 0x7fe210005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:55:41.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe226ffd700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe210003a70 0x7fe210003e80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe21c009b60 tx=0x7fe210005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc17901fed92d760 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:55:41.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c00e6c0 con 0x7fe210003a70 2023-03-03T09:55:41.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c00e820 con 0x7fe210003a70 2023-03-03T09:55:41.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe225ffb700 1 -- 172.21.15.7:0/2980754515 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c00eb90 con 0x7fe210003a70 2023-03-03T09:55:41.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe22d6f2700 1 -- 172.21.15.7:0/2980754515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe210003a70 msgr2=0x7fe210003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:41.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe22d6f2700 1 --2- 172.21.15.7:0/2980754515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe210003a70 0x7fe210003e80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe21c009b60 tx=0x7fe210005880 comp rx=0 tx=0).stop 2023-03-03T09:55:41.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe2277fe700 1 -- 172.21.15.7:0/2980754515 reap_dead start 2023-03-03T09:55:41.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe22d6f2700 1 -- 172.21.15.7:0/2980754515 shutdown_connections 2023-03-03T09:55:41.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe22d6f2700 1 -- 172.21.15.7:0/2980754515 >> 172.21.15.7:0/2980754515 conn(0x7fe2280c4f20 msgr2=0x7fe2280c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:55:41.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe22d6f2700 1 -- 172.21.15.7:0/2980754515 shutdown_connections 2023-03-03T09:55:41.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.246+0000 7fe22d6f2700 1 -- 172.21.15.7:0/2980754515 wait complete. 2023-03-03T09:55:41.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.248+0000 7fe22d6f2700 1 Processor -- start 2023-03-03T09:55:41.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.248+0000 7fe22d6f2700 1 -- start start 2023-03-03T09:55:41.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.248+0000 7fe22d6f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe210003a70 0x7fe2281448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:41.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.248+0000 7fe22d6f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2281ed9b0 0x7fe22806a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:41.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.248+0000 7fe22d6f2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe22806ab20 0x7fe2281e97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:41.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.248+0000 7fe22d6f2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2280c7cf0 con 0x7fe22806ab20 2023-03-03T09:55:41.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.248+0000 7fe22d6f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2280ac270 con 0x7fe210003a70 2023-03-03T09:55:41.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.248+0000 7fe22d6f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2280ac4c0 con 0x7fe2281ed9b0 2023-03-03T09:55:41.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2277fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe22806ab20 0x7fe2281e97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:41.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2277fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe22806ab20 0x7fe2281e97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58320/0 (socket says 172.21.15.7:58320) 2023-03-03T09:55:41.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2277fe700 1 -- 172.21.15.7:0/766211802 learned_addr learned my addr 172.21.15.7:0/766211802 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:55:41.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe226ffd700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe210003a70 0x7fe2281448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:41.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2267fc700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2281ed9b0 0x7fe22806a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:41.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2277fe700 1 -- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2281ed9b0 msgr2=0x7fe22806a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:41.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2277fe700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2281ed9b0 0x7fe22806a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2277fe700 1 -- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe210003a70 msgr2=0x7fe2281448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:41.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2277fe700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe210003a70 0x7fe2281448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2277fe700 1 -- 172.21.15.7:0/766211802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe228155030 con 0x7fe22806ab20 2023-03-03T09:55:41.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2267fc700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2281ed9b0 0x7fe22806a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:55:41.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe226ffd700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe210003a70 0x7fe2281448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:55:41.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe2277fe700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe22806ab20 0x7fe2281e97e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe220000f80 tx=0x7fe22000e4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:55:41.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe20f7fe700 1 -- 172.21.15.7:0/766211802 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22000ed20 con 0x7fe22806ab20 2023-03-03T09:55:41.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe20f7fe700 1 -- 172.21.15.7:0/766211802 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22000eea0 con 0x7fe22806ab20 2023-03-03T09:55:41.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe22d6f2700 1 -- 172.21.15.7:0/766211802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe228154d50 con 0x7fe22806ab20 2023-03-03T09:55:41.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe20f7fe700 1 -- 172.21.15.7:0/766211802 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220013660 con 0x7fe22806ab20 2023-03-03T09:55:41.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.249+0000 7fe22d6f2700 1 -- 172.21.15.7:0/766211802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe228155310 con 0x7fe22806ab20 2023-03-03T09:55:41.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.250+0000 7fe22d6f2700 1 -- 172.21.15.7:0/766211802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe228074bb0 con 0x7fe22806ab20 2023-03-03T09:55:41.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.251+0000 7fe20f7fe700 1 -- 172.21.15.7:0/766211802 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe2200137c0 con 0x7fe22806ab20 2023-03-03T09:55:41.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.251+0000 7fe20f7fe700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2140682f0 0x7fe21406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:55:41.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.251+0000 7fe20f7fe700 1 -- 172.21.15.7:0/766211802 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe220086940 con 0x7fe22806ab20 2023-03-03T09:55:41.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.251+0000 7fe226ffd700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2140682f0 0x7fe21406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:55:41.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.252+0000 7fe226ffd700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2140682f0 0x7fe21406a7a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0090b0 tx=0x7fe21c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:55:41.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.255+0000 7fe20f7fe700 1 -- 172.21.15.7:0/766211802 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe220052ce0 con 0x7fe22806ab20 2023-03-03T09:55:41.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.493+0000 7fe22d6f2700 1 -- 172.21.15.7:0/766211802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe22814a270 con 0x7fe22806ab20 2023-03-03T09:55:41.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.494+0000 7fe20f7fe700 1 -- 172.21.15.7:0/766211802 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe220056300 con 0x7fe22806ab20 2023-03-03T09:55:41.495 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:55:41.496 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:55:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 -- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2140682f0 msgr2=0x7fe21406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2140682f0 0x7fe21406a7a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0090b0 tx=0x7fe21c008030 comp rx=0 tx=0).stop 2023-03-03T09:55:41.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 -- 172.21.15.7:0/766211802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe22806ab20 msgr2=0x7fe2281e97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:55:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe22806ab20 0x7fe2281e97e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe220000f80 tx=0x7fe22000e4f0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 -- 172.21.15.7:0/766211802 shutdown_connections 2023-03-03T09:55:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2140682f0 0x7fe21406a7a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe22806ab20 0x7fe2281e97e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe210003a70 0x7fe2281448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 --2- 172.21.15.7:0/766211802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2281ed9b0 0x7fe22806a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:55:41.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 -- 172.21.15.7:0/766211802 >> 172.21.15.7:0/766211802 conn(0x7fe2280c4f20 msgr2=0x7fe228149390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:55:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 -- 172.21.15.7:0/766211802 shutdown_connections 2023-03-03T09:55:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:55:41.496+0000 7fe22d6f2700 1 -- 172.21.15.7:0/766211802 wait complete. 2023-03-03T09:55:41.499 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:55:41.509 INFO:tasks.mds_thrash.fs.[b]:waiting for 53 secs before thrashing 2023-03-03T09:55:52.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:55:52.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:22.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:22.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:26.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:56:27.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.099+0000 7f5731c8d700 1 Processor -- start 2023-03-03T09:56:27.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.099+0000 7f5731c8d700 1 -- start start 2023-03-03T09:56:27.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f5731c8d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f572c0ac220 0x7f572c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:27.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f5731c8d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f572c073c30 0x7f572c152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:27.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f5731c8d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572c0ac900 con 0x7f572c0ace30 2023-03-03T09:56:27.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f5731c8d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f572c071e30 con 0x7f572c0ac220 2023-03-03T09:56:27.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f5731c8d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f572c0acb50 con 0x7f572c073c30 2023-03-03T09:56:27.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f572c0ac220 0x7f572c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:27.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f572c0ace30 0x7f572c06a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52346/0 (socket says 172.21.15.7:52346) 2023-03-03T09:56:27.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572bfff700 1 -- 172.21.15.7:0/3279956735 learned_addr learned my addr 172.21.15.7:0/3279956735 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:56:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572affd700 1 --2- 172.21.15.7:0/3279956735 >> v2:172.21.15.143:3300/0 conn(0x7f572c073c30 0x7f572c152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572b7fe700 1 -- 172.21.15.7:0/3279956735 >> v2:172.21.15.143:3300/0 conn(0x7f572c073c30 msgr2=0x7f572c152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:56:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572b7fe700 1 --2- 172.21.15.7:0/3279956735 >> v2:172.21.15.143:3300/0 conn(0x7f572c073c30 0x7f572c152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572b7fe700 1 -- 172.21.15.7:0/3279956735 >> v1:172.21.15.7:6789/0 conn(0x7f572c0ace30 legacy=0x7f572c06a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572a7fc700 1 -- 172.21.15.7:0/3279956735 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738456902 0 0) 0x7f572c0ac900 con 0x7f572c0ace30 2023-03-03T09:56:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572b7fe700 1 -- 172.21.15.7:0/3279956735 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c1559c0 con 0x7f572c0ac220 2023-03-03T09:56:27.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572affd700 1 --2- 172.21.15.7:0/3279956735 >> v2:172.21.15.143:3300/0 conn(0x7f572c073c30 0x7f572c152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:56:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572b7fe700 1 --2- 172.21.15.7:0/3279956735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ac220 0x7f572c0736f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f572c0c4eb0 tx=0x7f571c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cd4eb126a1c5a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.100+0000 7f572a7fc700 1 -- 172.21.15.7:0/3279956735 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c011830 con 0x7f572c0ac220 2023-03-03T09:56:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.101+0000 7f572a7fc700 1 -- 172.21.15.7:0/3279956735 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c011990 con 0x7f572c0ac220 2023-03-03T09:56:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.101+0000 7f572a7fc700 1 -- 172.21.15.7:0/3279956735 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571c011d20 con 0x7f572c0ac220 2023-03-03T09:56:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.101+0000 7f5731c8d700 1 -- 172.21.15.7:0/3279956735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ac220 msgr2=0x7f572c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.101+0000 7f5731c8d700 1 --2- 172.21.15.7:0/3279956735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ac220 0x7f572c0736f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f572c0c4eb0 tx=0x7f571c009b20 comp rx=0 tx=0).stop 2023-03-03T09:56:27.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.101+0000 7f5731c8d700 1 -- 172.21.15.7:0/3279956735 shutdown_connections 2023-03-03T09:56:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.101+0000 7f5731c8d700 1 --2- 172.21.15.7:0/3279956735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ac220 0x7f572c0736f0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.101+0000 7f5731c8d700 1 --2- 172.21.15.7:0/3279956735 >> v2:172.21.15.143:3300/0 conn(0x7f572c073c30 0x7f572c152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.101+0000 7f5731c8d700 1 -- 172.21.15.7:0/3279956735 >> 172.21.15.7:0/3279956735 conn(0x7f572c0a94d0 msgr2=0x7f572c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:56:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.101+0000 7f5731c8d700 1 -- 172.21.15.7:0/3279956735 shutdown_connections 2023-03-03T09:56:27.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.101+0000 7f5731c8d700 1 -- 172.21.15.7:0/3279956735 wait complete. 2023-03-03T09:56:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.102+0000 7f5731c8d700 1 Processor -- start 2023-03-03T09:56:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.102+0000 7f5731c8d700 1 -- start start 2023-03-03T09:56:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f5731c8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f572c073c30 0x7f572c0c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f5731c8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f572c0ac220 0x7f572c0c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f5731c8d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ace30 0x7f572c14dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f5731c8d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c0bf400 con 0x7f572c0ace30 2023-03-03T09:56:27.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f5731c8d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c0c6240 con 0x7f572c0ac220 2023-03-03T09:56:27.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f5731c8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572c0c0eb0 con 0x7f572c073c30 2023-03-03T09:56:27.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f572bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ace30 0x7f572c14dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:27.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f572bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ace30 0x7f572c14dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58372/0 (socket says 172.21.15.7:58372) 2023-03-03T09:56:27.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f572bfff700 1 -- 172.21.15.7:0/432426731 learned_addr learned my addr 172.21.15.7:0/432426731 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:56:27.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f572b7fe700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f572c073c30 0x7f572c0c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:27.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.103+0000 7f572affd700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f572c0ac220 0x7f572c0c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:27.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f572bfff700 1 -- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f572c073c30 msgr2=0x7f572c0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:27.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f572bfff700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f572c073c30 0x7f572c0c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:27.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f572bfff700 1 -- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f572c0ac220 msgr2=0x7f572c0c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:27.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f572bfff700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f572c0ac220 0x7f572c0c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:27.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f572bfff700 1 -- 172.21.15.7:0/432426731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c0094a0 con 0x7f572c0ace30 2023-03-03T09:56:27.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f572bfff700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ace30 0x7f572c14dde0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5724002230 tx=0x7f572400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:27.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f5723fff700 1 -- 172.21.15.7:0/432426731 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572400aac0 con 0x7f572c0ace30 2023-03-03T09:56:27.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f5723fff700 1 -- 172.21.15.7:0/432426731 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572400ac40 con 0x7f572c0ace30 2023-03-03T09:56:27.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f5731c8d700 1 -- 172.21.15.7:0/432426731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572c155f80 con 0x7f572c0ace30 2023-03-03T09:56:27.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f5723fff700 1 -- 172.21.15.7:0/432426731 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57240070d0 con 0x7f572c0ace30 2023-03-03T09:56:27.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.104+0000 7f5731c8d700 1 -- 172.21.15.7:0/432426731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572c155ca0 con 0x7f572c0ace30 2023-03-03T09:56:27.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.105+0000 7f5731c8d700 1 -- 172.21.15.7:0/432426731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572c156260 con 0x7f572c0ace30 2023-03-03T09:56:27.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.106+0000 7f5723fff700 1 -- 172.21.15.7:0/432426731 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5724010070 con 0x7f572c0ace30 2023-03-03T09:56:27.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.106+0000 7f5723fff700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f570c0682f0 0x7f570c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:27.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.106+0000 7f5723fff700 1 -- 172.21.15.7:0/432426731 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5724085a30 con 0x7f572c0ace30 2023-03-03T09:56:27.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.106+0000 7f572b7fe700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f570c0682f0 0x7f570c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:27.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.107+0000 7f572b7fe700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f570c0682f0 0x7f570c06a7a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f571c0090b0 tx=0x7f571c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:27.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.110+0000 7f5723fff700 1 -- 172.21.15.7:0/432426731 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5724051d50 con 0x7f572c0ace30 2023-03-03T09:56:27.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.348+0000 7f5731c8d700 1 -- 172.21.15.7:0/432426731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f572c074bb0 con 0x7f572c0ace30 2023-03-03T09:56:27.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.348+0000 7f5723fff700 1 -- 172.21.15.7:0/432426731 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5724055370 con 0x7f572c0ace30 2023-03-03T09:56:27.351 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:56:27.351 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:56:27.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 -- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f570c0682f0 msgr2=0x7f570c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:27.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f570c0682f0 0x7f570c06a7a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f571c0090b0 tx=0x7f571c008030 comp rx=0 tx=0).stop 2023-03-03T09:56:27.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 -- 172.21.15.7:0/432426731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ace30 msgr2=0x7f572c14dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ace30 0x7f572c14dde0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5724002230 tx=0x7f572400cf50 comp rx=0 tx=0).stop 2023-03-03T09:56:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 -- 172.21.15.7:0/432426731 shutdown_connections 2023-03-03T09:56:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f570c0682f0 0x7f570c06a7a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f572c0ace30 0x7f572c14dde0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f572c073c30 0x7f572c0c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:27.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 --2- 172.21.15.7:0/432426731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f572c0ac220 0x7f572c0c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 -- 172.21.15.7:0/432426731 >> 172.21.15.7:0/432426731 conn(0x7f572c0a94d0 msgr2=0x7f572c152df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:56:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 -- 172.21.15.7:0/432426731 shutdown_connections 2023-03-03T09:56:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:27.351+0000 7f5731c8d700 1 -- 172.21.15.7:0/432426731 wait complete. 2023-03-03T09:56:27.355 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:56:27.365 INFO:tasks.mds_thrash.fs.[a]:waiting for 105 secs before thrashing 2023-03-03T09:56:34.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:56:34.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.647+0000 7f7e0b3b1700 1 Processor -- start 2023-03-03T09:56:34.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.647+0000 7f7e0b3b1700 1 -- start start 2023-03-03T09:56:34.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7e0b3b1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7e0406e020 0x7f7e04071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:34.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7e0b3b1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e04154910 con 0x7f7e04155390 2023-03-03T09:56:34.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7e0b3b1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e040c4540 con 0x7f7e04072520 2023-03-03T09:56:34.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7e0b3b1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e04070b90 con 0x7f7e0406e020 2023-03-03T09:56:34.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7e0914d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7e04155390 0x7f7e040c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52356/0 (socket says 172.21.15.7:52356) 2023-03-03T09:56:34.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7e0914d700 1 -- 172.21.15.7:0/452166350 learned_addr learned my addr 172.21.15.7:0/452166350 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:56:34.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7e0894c700 1 --2- 172.21.15.7:0/452166350 >> v2:172.21.15.143:3300/0 conn(0x7f7e0406e020 0x7f7e04071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:34.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934539965 0 0) 0x7f7e04154910 con 0x7f7e04155390 2023-03-03T09:56:34.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df00035b0 con 0x7f7e04155390 2023-03-03T09:56:34.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745351651 0 0) 0x7f7dfc001490 con 0x7f7e04072520 2023-03-03T09:56:34.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e04154910 con 0x7f7e04072520 2023-03-03T09:56:34.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881323955 0 0) 0x7f7df00035b0 con 0x7f7e04155390 2023-03-03T09:56:34.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 >> v2:172.21.15.143:3300/0 conn(0x7f7e0406e020 msgr2=0x7f7e04071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:56:34.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7dfbfff700 1 --2- 172.21.15.7:0/452166350 >> v2:172.21.15.143:3300/0 conn(0x7f7e0406e020 0x7f7e04071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.648+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 >> v1:172.21.15.143:6789/0 conn(0x7f7e04072520 legacy=0x7f7e04151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:34.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e040702a0 con 0x7f7e04155390 2023-03-03T09:56:34.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7e00001670 con 0x7f7e04155390 2023-03-03T09:56:34.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0894c700 1 --2- 172.21.15.7:0/452166350 >> v2:172.21.15.143:3300/0 conn(0x7f7e0406e020 0x7f7e04071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:56:34.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 >> v1:172.21.15.7:6789/0 conn(0x7f7e04155390 legacy=0x7f7e040c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:34.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7df0003c00 0x7f7df0003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:34.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df0004850 0x7f7df0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:34.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df0005250 0x7f7df000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:34.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e04070b90 con 0x7f7df0004850 2023-03-03T09:56:34.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df000c160 con 0x7f7df0005250 2023-03-03T09:56:34.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df000c320 con 0x7f7df0003c00 2023-03-03T09:56:34.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0894c700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df0004850 0x7f7df0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:34.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0994e700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df0005250 0x7f7df000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:34.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0914d700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7df0003c00 0x7f7df0003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:34.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0894c700 1 -- 172.21.15.7:0/452166350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7df0003c00 msgr2=0x7f7df0003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:56:34.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0894c700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7df0003c00 0x7f7df0003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0894c700 1 -- 172.21.15.7:0/452166350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df0005250 msgr2=0x7f7df000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:56:34.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0894c700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df0005250 0x7f7df000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0994e700 1 -- 172.21.15.7:0/452166350 reap_dead start 2023-03-03T09:56:34.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0894c700 1 -- 172.21.15.7:0/452166350 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df00035b0 con 0x7f7df0004850 2023-03-03T09:56:34.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7e0894c700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df0004850 0x7f7df0004cc0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009d30 tx=0x7f7df0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfbbc16c805368a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:34.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df4009e80 con 0x7f7df0004850 2023-03-03T09:56:34.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df400e9a0 con 0x7f7df0004850 2023-03-03T09:56:34.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.649+0000 7f7dfbfff700 1 -- 172.21.15.7:0/452166350 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df400ed10 con 0x7f7df0004850 2023-03-03T09:56:34.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.650+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/452166350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df0004850 msgr2=0x7f7df0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:34.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.650+0000 7f7e0b3b1700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df0004850 0x7f7df0004cc0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009d30 tx=0x7f7df0005880 comp rx=0 tx=0).stop 2023-03-03T09:56:34.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.650+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/452166350 shutdown_connections 2023-03-03T09:56:34.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.650+0000 7f7e0b3b1700 1 --2- 172.21.15.7:0/452166350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df0004850 0x7f7df0004cc0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.650+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/452166350 >> 172.21.15.7:0/452166350 conn(0x7f7e0406c110 msgr2=0x7f7e0406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:56:34.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.650+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/452166350 shutdown_connections 2023-03-03T09:56:34.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.650+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/452166350 wait complete. 2023-03-03T09:56:34.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.652+0000 7f7e0b3b1700 1 Processor -- start 2023-03-03T09:56:34.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.652+0000 7f7e0b3b1700 1 -- start start 2023-03-03T09:56:34.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0b3b1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df0004850 0x7f7e0414fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:34.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0b3b1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c7bc0 0x7f7e040c0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:34.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0b3b1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c0c70 0x7f7e041db750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:34.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0b3b1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e040c1110 con 0x7f7e040c0c70 2023-03-03T09:56:34.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0b3b1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e041eb600 con 0x7f7df0004850 2023-03-03T09:56:34.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0b3b1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e041eb8e0 con 0x7f7e040c7bc0 2023-03-03T09:56:34.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0994e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c0c70 0x7f7e041db750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:34.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0994e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c0c70 0x7f7e041db750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58392/0 (socket says 172.21.15.7:58392) 2023-03-03T09:56:34.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0994e700 1 -- 172.21.15.7:0/142359952 learned_addr learned my addr 172.21.15.7:0/142359952 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:56:34.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0914d700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df0004850 0x7f7e0414fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:34.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0894c700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c7bc0 0x7f7e040c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:34.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0994e700 1 -- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c7bc0 msgr2=0x7f7e040c0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:34.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0994e700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c7bc0 0x7f7e040c0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0994e700 1 -- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df0004850 msgr2=0x7f7e0414fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:34.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0994e700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df0004850 0x7f7e0414fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0994e700 1 -- 172.21.15.7:0/142359952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04070820 con 0x7f7e040c0c70 2023-03-03T09:56:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0894c700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c7bc0 0x7f7e040c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:56:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0994e700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c0c70 0x7f7e041db750 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7e040c2580 tx=0x7f7dfc00d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7df9ffb700 1 -- 172.21.15.7:0/142359952 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc00dbf0 con 0x7f7e040c0c70 2023-03-03T09:56:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7df9ffb700 1 -- 172.21.15.7:0/142359952 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc00dd50 con 0x7f7e040c0c70 2023-03-03T09:56:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.653+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/142359952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e04070540 con 0x7f7e040c0c70 2023-03-03T09:56:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.654+0000 7f7df9ffb700 1 -- 172.21.15.7:0/142359952 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc003c40 con 0x7f7e040c0c70 2023-03-03T09:56:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.654+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/142359952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e04155760 con 0x7f7e040c0c70 2023-03-03T09:56:34.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.655+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/142359952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e04074bb0 con 0x7f7e040c0c70 2023-03-03T09:56:34.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.655+0000 7f7df9ffb700 1 -- 172.21.15.7:0/142359952 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7dfc0050d0 con 0x7f7e040c0c70 2023-03-03T09:56:34.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.655+0000 7f7df9ffb700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 0x7f7de806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:56:34.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.656+0000 7f7df9ffb700 1 -- 172.21.15.7:0/142359952 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7dfc0863b0 con 0x7f7e040c0c70 2023-03-03T09:56:34.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.656+0000 7f7e0914d700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 0x7f7de806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:56:34.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.656+0000 7f7e0914d700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 0x7f7de806a7a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7e00007c10 tx=0x7f7e00001df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:56:34.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.660+0000 7f7df9ffb700 1 -- 172.21.15.7:0/142359952 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7dfc0527a0 con 0x7f7e040c0c70 2023-03-03T09:56:34.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.896+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/142359952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e040c4540 con 0x7f7e040c0c70 2023-03-03T09:56:34.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.897+0000 7f7df9ffb700 1 -- 172.21.15.7:0/142359952 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7e040c4540 con 0x7f7e040c0c70 2023-03-03T09:56:34.899 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:56:34.900 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:56:34.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 msgr2=0x7f7de806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:34.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 0x7f7de806a7a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7e00007c10 tx=0x7f7e00001df0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/142359952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c0c70 msgr2=0x7f7e041db750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:56:34.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c0c70 0x7f7e041db750 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7e040c2580 tx=0x7f7dfc00d660 comp rx=0 tx=0).stop 2023-03-03T09:56:34.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/142359952 shutdown_connections 2023-03-03T09:56:34.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 0x7f7de806a7a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c0c70 0x7f7e041db750 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df0004850 0x7f7e0414fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 --2- 172.21.15.7:0/142359952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c7bc0 0x7f7e040c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:56:34.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/142359952 >> 172.21.15.7:0/142359952 conn(0x7f7e0406c110 msgr2=0x7f7e040c2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:56:34.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/142359952 shutdown_connections 2023-03-03T09:56:34.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:56:34.899+0000 7f7e0b3b1700 1 -- 172.21.15.7:0/142359952 wait complete. 2023-03-03T09:56:34.904 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:56:34.914 INFO:tasks.mds_thrash.fs.[b]:waiting for 66 secs before thrashing 2023-03-03T09:56:52.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:56:52.748 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:23.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:23.003 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:40.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:57:41.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.051+0000 7f9ce2763700 1 Processor -- start 2023-03-03T09:57:41.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.051+0000 7f9ce2763700 1 -- start start 2023-03-03T09:57:41.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.052+0000 7f9ce2763700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cdc14fbc0 0x7f9cdc152e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:41.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.052+0000 7f9ce2763700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cdc0ba3d0 con 0x7f9cdc0a5d20 2023-03-03T09:57:41.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.052+0000 7f9ce2763700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cdc0c1540 con 0x7f9cdc0c25e0 2023-03-03T09:57:41.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.052+0000 7f9ce2763700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc0c1820 con 0x7f9cdc14fbc0 2023-03-03T09:57:41.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.052+0000 7f9cdb7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9cdc0a5d20 0x7f9cdc0a85a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52418/0 (socket says 172.21.15.7:52418) 2023-03-03T09:57:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.052+0000 7f9cdb7fe700 1 -- 172.21.15.7:0/1725744849 learned_addr learned my addr 172.21.15.7:0/1725744849 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:57:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.052+0000 7f9cdbfff700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cdc14fbc0 0x7f9cdc152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209926796 0 0) 0x7f9cdc0ba3d0 con 0x7f9cdc0a5d20 2023-03-03T09:57:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cc40035b0 con 0x7f9cdc0a5d20 2023-03-03T09:57:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661005862 0 0) 0x7f9cdc0c1540 con 0x7f9cdc0c25e0 2023-03-03T09:57:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cdc0ba3d0 con 0x7f9cdc0c25e0 2023-03-03T09:57:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3941465671 0 0) 0x7f9cc40035b0 con 0x7f9cdc0a5d20 2023-03-03T09:57:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cdc14fbc0 msgr2=0x7f9cdc152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:57:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cdc14fbc0 0x7f9cdc152e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 >> v1:172.21.15.143:6789/0 conn(0x7f9cdc0c25e0 legacy=0x7f9cdc15cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc0c1b00 con 0x7f9cdc0a5d20 2023-03-03T09:57:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f9cd0001670 con 0x7f9cdc0a5d20 2023-03-03T09:57:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdbfff700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cdc14fbc0 0x7f9cdc152e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T09:57:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 >> v1:172.21.15.7:6789/0 conn(0x7f9cdc0a5d20 legacy=0x7f9cdc0a85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cdc14fbc0 0x7f9cc4003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:41.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc4004170 0x7f9cc4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:41.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cc4004d90 0x7f9cc4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:41.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc0c1820 con 0x7f9cc4004170 2023-03-03T09:57:41.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc400bba0 con 0x7f9cc4004d90 2023-03-03T09:57:41.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc400bd60 con 0x7f9cdc14fbc0 2023-03-03T09:57:41.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdb7fe700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc4004170 0x7f9cc4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:41.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdbfff700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cdc14fbc0 0x7f9cc4003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:41.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9ce0d00700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cc4004d90 0x7f9cc4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:41.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdb7fe700 1 -- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cdc14fbc0 msgr2=0x7f9cc4003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:57:41.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdb7fe700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cdc14fbc0 0x7f9cc4003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:41.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdb7fe700 1 -- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cc4004d90 msgr2=0x7f9cc4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:57:41.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdb7fe700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cc4004d90 0x7f9cc4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:41.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdb7fe700 1 -- 172.21.15.7:0/1725744849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc0094a0 con 0x7f9cc4004170 2023-03-03T09:57:41.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9ce0d00700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cc4004d90 0x7f9cc4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:57:41.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.053+0000 7f9cdb7fe700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc4004170 0x7f9cc4004850 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000c210 tx=0x7f9cc4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998faa0b678d72e2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:57:41.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.054+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd000cd50 con 0x7f9cc4004170 2023-03-03T09:57:41.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.054+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd0001af0 con 0x7f9cc4004170 2023-03-03T09:57:41.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.054+0000 7f9cdaffd700 1 -- 172.21.15.7:0/1725744849 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0005ba0 con 0x7f9cc4004170 2023-03-03T09:57:41.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.054+0000 7f9ce2763700 1 -- 172.21.15.7:0/1725744849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc4004170 msgr2=0x7f9cc4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:41.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.054+0000 7f9ce2763700 1 --2- 172.21.15.7:0/1725744849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc4004170 0x7f9cc4004850 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000c210 tx=0x7f9cc4005880 comp rx=0 tx=0).stop 2023-03-03T09:57:41.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.054+0000 7f9ce0d00700 1 -- 172.21.15.7:0/1725744849 reap_dead start 2023-03-03T09:57:41.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.054+0000 7f9ce2763700 1 -- 172.21.15.7:0/1725744849 shutdown_connections 2023-03-03T09:57:41.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.054+0000 7f9ce2763700 1 -- 172.21.15.7:0/1725744849 >> 172.21.15.7:0/1725744849 conn(0x7f9cdc06af40 msgr2=0x7f9cdc0c6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:57:41.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.054+0000 7f9ce2763700 1 -- 172.21.15.7:0/1725744849 shutdown_connections 2023-03-03T09:57:41.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.054+0000 7f9ce2763700 1 -- 172.21.15.7:0/1725744849 wait complete. 2023-03-03T09:57:41.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.055+0000 7f9ce2763700 1 Processor -- start 2023-03-03T09:57:41.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.056+0000 7f9ce2763700 1 -- start start 2023-03-03T09:57:41.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.056+0000 7f9ce2763700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc4004170 0x7f9cdc0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:41.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.056+0000 7f9ce2763700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc14d1e0 0x7f9cdc1df3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:41.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.056+0000 7f9ce2763700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cdc0a9a30 0x7f9cdc1e8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:41.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.056+0000 7f9ce2763700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc14d6e0 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.056+0000 7f9ce2763700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc0ab6e0 con 0x7f9cdc0a9a30 2023-03-03T09:57:41.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.056+0000 7f9ce2763700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1e81c0 con 0x7f9cc4004170 2023-03-03T09:57:41.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.056+0000 7f9cdb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc14d1e0 0x7f9cdc1df3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:41.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.056+0000 7f9cdb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc14d1e0 0x7f9cdc1df3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58448/0 (socket says 172.21.15.7:58448) 2023-03-03T09:57:41.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.056+0000 7f9cdb7fe700 1 -- 172.21.15.7:0/175469636 learned_addr learned my addr 172.21.15.7:0/175469636 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:57:41.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9cdbfff700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc4004170 0x7f9cdc0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:41.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9ce0d00700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cdc0a9a30 0x7f9cdc1e8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:41.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9cdb7fe700 1 -- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc4004170 msgr2=0x7f9cdc0ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:41.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9cdb7fe700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc4004170 0x7f9cdc0ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:41.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9cdb7fe700 1 -- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cdc0a9a30 msgr2=0x7f9cdc1e8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:41.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9cdb7fe700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cdc0a9a30 0x7f9cdc1e8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:41.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9cdb7fe700 1 -- 172.21.15.7:0/175469636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc0c20c0 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9cdb7fe700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc14d1e0 0x7f9cdc1df3d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4005830 tx=0x7f9cd0001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:57:41.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9cd8ff9700 1 -- 172.21.15.7:0/175469636 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0004c80 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9cd8ff9700 1 -- 172.21.15.7:0/175469636 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd0004de0 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9ce2763700 1 -- 172.21.15.7:0/175469636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc0c1de0 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9cd8ff9700 1 -- 172.21.15.7:0/175469636 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd00072f0 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.057+0000 7f9ce2763700 1 -- 172.21.15.7:0/175469636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc0a8cb0 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.058+0000 7f9ce2763700 1 -- 172.21.15.7:0/175469636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cdc074bb0 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.059+0000 7f9cd8ff9700 1 -- 172.21.15.7:0/175469636 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9cd002e070 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.059+0000 7f9cd8ff9700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9cc80682f0 0x7f9cc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:57:41.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.059+0000 7f9cd8ff9700 1 -- 172.21.15.7:0/175469636 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9cd00872e0 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.059+0000 7f9cdbfff700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9cc80682f0 0x7f9cc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:57:41.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.060+0000 7f9cdbfff700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9cc80682f0 0x7f9cc806a7a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009c50 tx=0x7f9ccc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:57:41.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.064+0000 7f9cd8ff9700 1 -- 172.21.15.7:0/175469636 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9cd0053680 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.309+0000 7f9ce2763700 1 -- 172.21.15.7:0/175469636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cdc14b090 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.309+0000 7f9cd8ff9700 1 -- 172.21.15.7:0/175469636 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9cd0056ca0 con 0x7f9cdc14d1e0 2023-03-03T09:57:41.312 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:57:41.312 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:57:41.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 -- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9cc80682f0 msgr2=0x7f9cc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:41.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9cc80682f0 0x7f9cc806a7a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009c50 tx=0x7f9ccc008040 comp rx=0 tx=0).stop 2023-03-03T09:57:41.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 -- 172.21.15.7:0/175469636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc14d1e0 msgr2=0x7f9cdc1df3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:57:41.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc14d1e0 0x7f9cdc1df3d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4005830 tx=0x7f9cd0001980 comp rx=0 tx=0).stop 2023-03-03T09:57:41.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 -- 172.21.15.7:0/175469636 shutdown_connections 2023-03-03T09:57:41.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cdc0a9a30 0x7f9cdc1e8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:41.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc4004170 0x7f9cdc0ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:41.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9cc80682f0 0x7f9cc806a7a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:41.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 --2- 172.21.15.7:0/175469636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc14d1e0 0x7f9cdc1df3d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:57:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 -- 172.21.15.7:0/175469636 >> 172.21.15.7:0/175469636 conn(0x7f9cdc06af40 msgr2=0x7f9cdc0a7870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:57:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 -- 172.21.15.7:0/175469636 shutdown_connections 2023-03-03T09:57:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:57:41.312+0000 7f9ce2763700 1 -- 172.21.15.7:0/175469636 wait complete. 2023-03-03T09:57:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:57:41.326 INFO:tasks.mds_thrash.fs.[b]:waiting for 110 secs before thrashing 2023-03-03T09:57:53.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:57:53.073 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:12.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:58:12.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.504+0000 7f84e6ecf700 1 Processor -- start 2023-03-03T09:58:12.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.504+0000 7f84e6ecf700 1 -- start start 2023-03-03T09:58:12.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.504+0000 7f84e6ecf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00c43d0 0x7f84e00c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:12.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.504+0000 7f84e6ecf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f84e0154590 0x7f84e01549f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:12.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84e6ecf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e0072920 con 0x7f84e014e560 2023-03-03T09:58:12.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84e6ecf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e0072380 con 0x7f84e0154590 2023-03-03T09:58:12.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84e6ecf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e006bef0 con 0x7f84e00c43d0 2023-03-03T09:58:12.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84e4c6b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f84e014e560 0x7f84e0071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52458/0 (socket says 172.21.15.7:52458) 2023-03-03T09:58:12.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84e4c6b700 1 -- 172.21.15.7:0/3720143460 learned_addr learned my addr 172.21.15.7:0/3720143460 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:58:12.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486102346 0 0) 0x7f84d4001110 con 0x7f84e014e560 2023-03-03T09:58:12.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d00035b0 con 0x7f84e014e560 2023-03-03T09:58:12.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84e546c700 1 --2- 172.21.15.7:0/3720143460 >> v2:172.21.15.143:3300/0 conn(0x7f84e0154590 0x7f84e01549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:12.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84dffff700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00c43d0 0x7f84e00c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:12.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1109385636 0 0) 0x7f84d00035b0 con 0x7f84e014e560 2023-03-03T09:58:12.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00c43d0 msgr2=0x7f84e00c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:58:12.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00c43d0 0x7f84e00c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 >> v2:172.21.15.143:3300/0 conn(0x7f84e0154590 msgr2=0x7f84e01549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:58:12.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 --2- 172.21.15.7:0/3720143460 >> v2:172.21.15.143:3300/0 conn(0x7f84e0154590 0x7f84e01549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e006b910 con 0x7f84e014e560 2023-03-03T09:58:12.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f84d4001800 con 0x7f84e014e560 2023-03-03T09:58:12.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 >> v1:172.21.15.7:6789/0 conn(0x7f84e014e560 legacy=0x7f84e0071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:12.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d0003a70 0x7f84d0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:12.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0004850 0x7f84d0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:12.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00c43d0 0x7f84d0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:12.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e0072380 con 0x7f84d0003a70 2023-03-03T09:58:12.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e006bef0 con 0x7f84d0004850 2023-03-03T09:58:12.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d000ba10 con 0x7f84e00c43d0 2023-03-03T09:58:12.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.505+0000 7f84e4c6b700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d0003a70 0x7f84d0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:12.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84e546c700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00c43d0 0x7f84d0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:12.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84dffff700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0004850 0x7f84d0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:12.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84e4c6b700 1 -- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00c43d0 msgr2=0x7f84d0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:58:12.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84e4c6b700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00c43d0 0x7f84d0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84e4c6b700 1 -- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0004850 msgr2=0x7f84d0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:58:12.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84e4c6b700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0004850 0x7f84d0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84e4c6b700 1 -- 172.21.15.7:0/3720143460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d00035b0 con 0x7f84d0003a70 2023-03-03T09:58:12.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84e546c700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00c43d0 0x7f84d0005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:58:12.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84dffff700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0004850 0x7f84d0004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:58:12.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84e4c6b700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d0003a70 0x7f84d0003e80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f84d400c240 tx=0x7f84d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704c2c19f097c8f3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:58:12.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4007db0 con 0x7f84d0003a70 2023-03-03T09:58:12.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d400cde0 con 0x7f84d0003a70 2023-03-03T09:58:12.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84df7fe700 1 -- 172.21.15.7:0/3720143460 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4005b80 con 0x7f84d0003a70 2023-03-03T09:58:12.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84e6ecf700 1 -- 172.21.15.7:0/3720143460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d0003a70 msgr2=0x7f84d0003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:12.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.506+0000 7f84e6ecf700 1 --2- 172.21.15.7:0/3720143460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d0003a70 0x7f84d0003e80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f84d400c240 tx=0x7f84d0005880 comp rx=0 tx=0).stop 2023-03-03T09:58:12.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.507+0000 7f84e546c700 1 -- 172.21.15.7:0/3720143460 reap_dead start 2023-03-03T09:58:12.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.507+0000 7f84e6ecf700 1 -- 172.21.15.7:0/3720143460 shutdown_connections 2023-03-03T09:58:12.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.507+0000 7f84e6ecf700 1 -- 172.21.15.7:0/3720143460 >> 172.21.15.7:0/3720143460 conn(0x7f84e00736f0 msgr2=0x7f84e0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:58:12.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.507+0000 7f84e6ecf700 1 -- 172.21.15.7:0/3720143460 shutdown_connections 2023-03-03T09:58:12.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.507+0000 7f84e6ecf700 1 -- 172.21.15.7:0/3720143460 wait complete. 2023-03-03T09:58:12.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.508+0000 7f84e6ecf700 1 Processor -- start 2023-03-03T09:58:12.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.508+0000 7f84e6ecf700 1 -- start start 2023-03-03T09:58:12.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e6ecf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0003a70 0x7f84e01edeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:12.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e6ecf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00ab460 0x7f84e00c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:12.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e6ecf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e00c78c0 0x7f84e0151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:12.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e6ecf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00be7c0 con 0x7f84e00c78c0 2023-03-03T09:58:12.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e6ecf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00befd0 con 0x7f84d0003a70 2023-03-03T09:58:12.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e6ecf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e00bea10 con 0x7f84e00ab460 2023-03-03T09:58:12.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e546c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e00c78c0 0x7f84e0151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:12.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e546c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e00c78c0 0x7f84e0151140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58494/0 (socket says 172.21.15.7:58494) 2023-03-03T09:58:12.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e546c700 1 -- 172.21.15.7:0/497739450 learned_addr learned my addr 172.21.15.7:0/497739450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:58:12.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e4c6b700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0003a70 0x7f84e01edeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:12.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84dffff700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00ab460 0x7f84e00c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:12.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e546c700 1 -- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00ab460 msgr2=0x7f84e00c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:12.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e546c700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00ab460 0x7f84e00c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e546c700 1 -- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0003a70 msgr2=0x7f84e01edeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:12.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e546c700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0003a70 0x7f84e01edeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e546c700 1 -- 172.21.15.7:0/497739450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e006bb80 con 0x7f84e00c78c0 2023-03-03T09:58:12.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e4c6b700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0003a70 0x7f84e01edeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:58:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84e546c700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e00c78c0 0x7f84e0151140 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f84e00c6040 tx=0x7f84d80104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:58:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.509+0000 7f84dd7fa700 1 -- 172.21.15.7:0/497739450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d800e050 con 0x7f84e00c78c0 2023-03-03T09:58:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.510+0000 7f84dd7fa700 1 -- 172.21.15.7:0/497739450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d800f040 con 0x7f84e00c78c0 2023-03-03T09:58:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.510+0000 7f84dd7fa700 1 -- 172.21.15.7:0/497739450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8003a20 con 0x7f84e00c78c0 2023-03-03T09:58:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.510+0000 7f84e6ecf700 1 -- 172.21.15.7:0/497739450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e00aacc0 con 0x7f84e00c78c0 2023-03-03T09:58:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.510+0000 7f84e6ecf700 1 -- 172.21.15.7:0/497739450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e00aa970 con 0x7f84e00c78c0 2023-03-03T09:58:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.511+0000 7f84e6ecf700 1 -- 172.21.15.7:0/497739450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e0074bb0 con 0x7f84e00c78c0 2023-03-03T09:58:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.511+0000 7f84dd7fa700 1 -- 172.21.15.7:0/497739450 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f84d8003b80 con 0x7f84e00c78c0 2023-03-03T09:58:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.511+0000 7f84dd7fa700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84c40682f0 0x7f84c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.512+0000 7f84dd7fa700 1 -- 172.21.15.7:0/497739450 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f84d8086ad0 con 0x7f84e00c78c0 2023-03-03T09:58:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.512+0000 7f84e4c6b700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84c40682f0 0x7f84c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.512+0000 7f84e4c6b700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84c40682f0 0x7f84c406a7a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f84d40012b0 tx=0x7f84d400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:58:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.516+0000 7f84dd7fa700 1 -- 172.21.15.7:0/497739450 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f84d8052e70 con 0x7f84e00c78c0 2023-03-03T09:58:12.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.753+0000 7f84e6ecf700 1 -- 172.21.15.7:0/497739450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84e006ad60 con 0x7f84e00c78c0 2023-03-03T09:58:12.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.753+0000 7f84dd7fa700 1 -- 172.21.15.7:0/497739450 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f84d8056490 con 0x7f84e00c78c0 2023-03-03T09:58:12.755 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:58:12.755 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:58:12.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 -- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84c40682f0 msgr2=0x7f84c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:12.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84c40682f0 0x7f84c406a7a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f84d40012b0 tx=0x7f84d400a000 comp rx=0 tx=0).stop 2023-03-03T09:58:12.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 -- 172.21.15.7:0/497739450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e00c78c0 msgr2=0x7f84e0151140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:12.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e00c78c0 0x7f84e0151140 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f84e00c6040 tx=0x7f84d80104a0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 -- 172.21.15.7:0/497739450 shutdown_connections 2023-03-03T09:58:12.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84c40682f0 0x7f84c406a7a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e00c78c0 0x7f84e0151140 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84d0003a70 0x7f84e01edeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 --2- 172.21.15.7:0/497739450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84e00ab460 0x7f84e00c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:12.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 -- 172.21.15.7:0/497739450 >> 172.21.15.7:0/497739450 conn(0x7f84e00736f0 msgr2=0x7f84e0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:58:12.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 -- 172.21.15.7:0/497739450 shutdown_connections 2023-03-03T09:58:12.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:12.756+0000 7f84e6ecf700 1 -- 172.21.15.7:0/497739450 wait complete. 2023-03-03T09:58:12.759 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:58:12.769 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before thrashing 2023-03-03T09:58:13.770 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:58:13.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.907+0000 7fef8ed5a700 1 Processor -- start 2023-03-03T09:58:13.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.907+0000 7fef8ed5a700 1 -- start start 2023-03-03T09:58:13.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8ed5a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef88151b20 0x7fef88071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:13.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8ed5a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef88154f10 0x7fef88158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:13.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8ed5a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef88151ef0 con 0x7fef88071d60 2023-03-03T09:58:13.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8ed5a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef88157b20 con 0x7fef88151b20 2023-03-03T09:58:13.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8ed5a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef88155410 con 0x7fef88154f10 2023-03-03T09:58:13.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8caf6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef88151b20 0x7fef88071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:13.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8caf6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef88151b20 0x7fef88071820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58498/0 (socket says 172.21.15.7:58498) 2023-03-03T09:58:13.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8caf6700 1 -- 172.21.15.7:0/3469866904 learned_addr learned my addr 172.21.15.7:0/3469866904 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:58:13.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef87fff700 1 --2- 172.21.15.7:0/3469866904 >> v2:172.21.15.143:3300/0 conn(0x7fef88154f10 0x7fef88158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:13.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8caf6700 1 -- 172.21.15.7:0/3469866904 >> v2:172.21.15.143:3300/0 conn(0x7fef88154f10 msgr2=0x7fef88158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:58:13.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8caf6700 1 --2- 172.21.15.7:0/3469866904 >> v2:172.21.15.143:3300/0 conn(0x7fef88154f10 0x7fef88158000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:13.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.908+0000 7fef8caf6700 1 -- 172.21.15.7:0/3469866904 >> v1:172.21.15.143:6789/0 conn(0x7fef88071d60 legacy=0x7fef880c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:58:13.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef8caf6700 1 -- 172.21.15.7:0/3469866904 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8806f640 con 0x7fef88151b20 2023-03-03T09:58:13.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef8caf6700 1 --2- 172.21.15.7:0/3469866904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88151b20 0x7fef88071820 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fef88070850 tx=0x7fef7c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922d2c85ded9d238 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:58:13.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef877fe700 1 -- 172.21.15.7:0/3469866904 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c004230 con 0x7fef88151b20 2023-03-03T09:58:13.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef877fe700 1 -- 172.21.15.7:0/3469866904 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c004390 con 0x7fef88151b20 2023-03-03T09:58:13.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef877fe700 1 -- 172.21.15.7:0/3469866904 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0036a0 con 0x7fef88151b20 2023-03-03T09:58:13.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3469866904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88151b20 msgr2=0x7fef88071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:13.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef8ed5a700 1 --2- 172.21.15.7:0/3469866904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88151b20 0x7fef88071820 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fef88070850 tx=0x7fef7c009bd0 comp rx=0 tx=0).stop 2023-03-03T09:58:13.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3469866904 shutdown_connections 2023-03-03T09:58:13.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef8ed5a700 1 --2- 172.21.15.7:0/3469866904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88151b20 0x7fef88071820 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:13.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef8ed5a700 1 --2- 172.21.15.7:0/3469866904 >> v2:172.21.15.143:3300/0 conn(0x7fef88154f10 0x7fef88158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:13.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.909+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3469866904 >> 172.21.15.7:0/3469866904 conn(0x7fef880c8900 msgr2=0x7fef880c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:58:13.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.910+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3469866904 shutdown_connections 2023-03-03T09:58:13.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.910+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3469866904 wait complete. 2023-03-03T09:58:13.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.911+0000 7fef8ed5a700 1 Processor -- start 2023-03-03T09:58:13.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.911+0000 7fef8ed5a700 1 -- start start 2023-03-03T09:58:13.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.911+0000 7fef8ed5a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88071d60 0x7fef880c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:13.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8ed5a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef88151b20 0x7fef8814f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:13.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8ed5a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef88154f10 0x7fef881ed1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:13.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8ed5a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef88150c20 con 0x7fef88071d60 2023-03-03T09:58:13.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8ed5a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef88150940 con 0x7fef88151b20 2023-03-03T09:58:13.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8ed5a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef88156600 con 0x7fef88154f10 2023-03-03T09:58:13.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8caf6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88071d60 0x7fef880c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:13.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8caf6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88071d60 0x7fef880c6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58502/0 (socket says 172.21.15.7:58502) 2023-03-03T09:58:13.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8caf6700 1 -- 172.21.15.7:0/3514835015 learned_addr learned my addr 172.21.15.7:0/3514835015 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:58:13.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef87fff700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef88151b20 0x7fef8814f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:13.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8d2f7700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef88154f10 0x7fef881ed1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:13.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8caf6700 1 -- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef88154f10 msgr2=0x7fef881ed1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:13.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8caf6700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef88154f10 0x7fef881ed1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:13.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8caf6700 1 -- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef88151b20 msgr2=0x7fef8814f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:13.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8caf6700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef88151b20 0x7fef8814f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:13.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8caf6700 1 -- 172.21.15.7:0/3514835015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c009550 con 0x7fef88071d60 2023-03-03T09:58:13.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef87fff700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef88151b20 0x7fef8814f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:58:13.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8d2f7700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef88154f10 0x7fef881ed1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:58:13.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8caf6700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88071d60 0x7fef880c6d20 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009130 tx=0x7fef7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:58:13.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef857fa700 1 -- 172.21.15.7:0/3514835015 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c011630 con 0x7fef88071d60 2023-03-03T09:58:13.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef857fa700 1 -- 172.21.15.7:0/3514835015 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c011790 con 0x7fef88071d60 2023-03-03T09:58:13.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3514835015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8806fc00 con 0x7fef88071d60 2023-03-03T09:58:13.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef857fa700 1 -- 172.21.15.7:0/3514835015 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c011b00 con 0x7fef88071d60 2023-03-03T09:58:13.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.912+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3514835015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8806f920 con 0x7fef88071d60 2023-03-03T09:58:13.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.913+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3514835015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef880aae90 con 0x7fef88071d60 2023-03-03T09:58:13.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.914+0000 7fef857fa700 1 -- 172.21.15.7:0/3514835015 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fef7c017020 con 0x7fef88071d60 2023-03-03T09:58:13.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.914+0000 7fef857fa700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef6c0682f0 0x7fef6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:58:13.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.914+0000 7fef857fa700 1 -- 172.21.15.7:0/3514835015 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fef7c0876f0 con 0x7fef88071d60 2023-03-03T09:58:13.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.914+0000 7fef87fff700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef6c0682f0 0x7fef6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:58:13.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.915+0000 7fef87fff700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef6c0682f0 0x7fef6c06a7a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fef88144de0 tx=0x7fef74008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:58:13.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:13.919+0000 7fef857fa700 1 -- 172.21.15.7:0/3514835015 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fef7c053a30 con 0x7fef88071d60 2023-03-03T09:58:14.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.157+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3514835015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef88074bb0 con 0x7fef88071d60 2023-03-03T09:58:14.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.158+0000 7fef857fa700 1 -- 172.21.15.7:0/3514835015 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fef7c057050 con 0x7fef88071d60 2023-03-03T09:58:14.159 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:58:14.159 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:58:14.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef6c0682f0 msgr2=0x7fef6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:14.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef6c0682f0 0x7fef6c06a7a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fef88144de0 tx=0x7fef74008040 comp rx=0 tx=0).stop 2023-03-03T09:58:14.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3514835015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88071d60 msgr2=0x7fef880c6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:58:14.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88071d60 0x7fef880c6d20 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009130 tx=0x7fef7c004750 comp rx=0 tx=0).stop 2023-03-03T09:58:14.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3514835015 shutdown_connections 2023-03-03T09:58:14.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef88154f10 0x7fef881ed1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:14.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef6c0682f0 0x7fef6c06a7a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:14.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef88071d60 0x7fef880c6d20 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:14.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 --2- 172.21.15.7:0/3514835015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef88151b20 0x7fef8814f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:58:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3514835015 >> 172.21.15.7:0/3514835015 conn(0x7fef880c8900 msgr2=0x7fef880c3690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:58:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3514835015 shutdown_connections 2023-03-03T09:58:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:58:14.160+0000 7fef8ed5a700 1 -- 172.21.15.7:0/3514835015 wait complete. 2023-03-03T09:58:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:58:14.173 INFO:tasks.mds_thrash.fs.[a]:waiting for 55 secs before thrashing 2023-03-03T09:58:23.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:23.154 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:53.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:58:53.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:09.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:59:09.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.312+0000 7fdf269f4700 1 Processor -- start 2023-03-03T09:59:09.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.312+0000 7fdf269f4700 1 -- start start 2023-03-03T09:59:09.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf269f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf2006e020 0x7fdf20071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:09.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf269f4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf20072520 0x7fdf20151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:09.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf269f4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf20154910 con 0x7fdf20155390 2023-03-03T09:59:09.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf269f4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf20070b90 con 0x7fdf20072520 2023-03-03T09:59:09.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf269f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf20151320 con 0x7fdf2006e020 2023-03-03T09:59:09.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf24f91700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf20072520 0x7fdf20151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:09.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf24f91700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf20072520 0x7fdf20151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58574/0 (socket says 172.21.15.7:58574) 2023-03-03T09:59:09.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf24f91700 1 -- 172.21.15.7:0/1703144584 learned_addr learned my addr 172.21.15.7:0/1703144584 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:59:09.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf1f7fe700 1 --2- 172.21.15.7:0/1703144584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf2006e020 0x7fdf20071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:09.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf24f91700 1 -- 172.21.15.7:0/1703144584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf2006e020 msgr2=0x7fdf20071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:59:09.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf24f91700 1 --2- 172.21.15.7:0/1703144584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf2006e020 0x7fdf20071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:09.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.313+0000 7fdf24f91700 1 -- 172.21.15.7:0/1703144584 >> v1:172.21.15.143:6789/0 conn(0x7fdf20155390 legacy=0x7fdf200c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:09.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf24f91700 1 -- 172.21.15.7:0/1703144584 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf200c4540 con 0x7fdf20072520 2023-03-03T09:59:09.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf24f91700 1 --2- 172.21.15.7:0/1703144584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20072520 0x7fdf20151770 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009e50 tx=0x7fdf1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15a1cadc0bbc559 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:09.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf1effd700 1 -- 172.21.15.7:0/1703144584 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1800e070 con 0x7fdf20072520 2023-03-03T09:59:09.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf1effd700 1 -- 172.21.15.7:0/1703144584 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1800b910 con 0x7fdf20072520 2023-03-03T09:59:09.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf1effd700 1 -- 172.21.15.7:0/1703144584 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1800bcc0 con 0x7fdf20072520 2023-03-03T09:59:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf269f4700 1 -- 172.21.15.7:0/1703144584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20072520 msgr2=0x7fdf20151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf269f4700 1 --2- 172.21.15.7:0/1703144584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20072520 0x7fdf20151770 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fdf18009e50 tx=0x7fdf1800b330 comp rx=0 tx=0).stop 2023-03-03T09:59:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf269f4700 1 -- 172.21.15.7:0/1703144584 shutdown_connections 2023-03-03T09:59:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf269f4700 1 --2- 172.21.15.7:0/1703144584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20072520 0x7fdf20151770 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf269f4700 1 --2- 172.21.15.7:0/1703144584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf2006e020 0x7fdf20071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf269f4700 1 -- 172.21.15.7:0/1703144584 >> 172.21.15.7:0/1703144584 conn(0x7fdf2006c110 msgr2=0x7fdf2006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:59:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf269f4700 1 -- 172.21.15.7:0/1703144584 shutdown_connections 2023-03-03T09:59:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.314+0000 7fdf269f4700 1 -- 172.21.15.7:0/1703144584 wait complete. 2023-03-03T09:59:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.316+0000 7fdf269f4700 1 Processor -- start 2023-03-03T09:59:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.316+0000 7fdf269f4700 1 -- start start 2023-03-03T09:59:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.316+0000 7fdf269f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf2006e020 0x7fdf2014e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.316+0000 7fdf269f4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf20072520 0x7fdf201e5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.316+0000 7fdf269f4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20155390 0x7fdf2014c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf269f4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2014d9f0 con 0x7fdf20155390 2023-03-03T09:59:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf269f4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf201533a0 con 0x7fdf20072520 2023-03-03T09:59:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf269f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf201540b0 con 0x7fdf2006e020 2023-03-03T09:59:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf24f91700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20155390 0x7fdf2014c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf24f91700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20155390 0x7fdf2014c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58580/0 (socket says 172.21.15.7:58580) 2023-03-03T09:59:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf24f91700 1 -- 172.21.15.7:0/1201670266 learned_addr learned my addr 172.21.15.7:0/1201670266 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:59:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf1f7fe700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf20072520 0x7fdf201e5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf1ffff700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf2006e020 0x7fdf2014e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf24f91700 1 -- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf2006e020 msgr2=0x7fdf2014e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf24f91700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf2006e020 0x7fdf2014e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf24f91700 1 -- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf20072520 msgr2=0x7fdf201e5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf24f91700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf20072520 0x7fdf201e5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf24f91700 1 -- 172.21.15.7:0/1201670266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf18009b30 con 0x7fdf20155390 2023-03-03T09:59:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf1f7fe700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf20072520 0x7fdf201e5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T09:59:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf24f91700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20155390 0x7fdf2014c3d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fdf200c4a90 tx=0x7fdf18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf1cff9700 1 -- 172.21.15.7:0/1201670266 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf18004960 con 0x7fdf20155390 2023-03-03T09:59:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf1cff9700 1 -- 172.21.15.7:0/1201670266 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf180099b0 con 0x7fdf20155390 2023-03-03T09:59:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf269f4700 1 -- 172.21.15.7:0/1201670266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf20070540 con 0x7fdf20155390 2023-03-03T09:59:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf1cff9700 1 -- 172.21.15.7:0/1201670266 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1801c680 con 0x7fdf20155390 2023-03-03T09:59:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.317+0000 7fdf269f4700 1 -- 172.21.15.7:0/1201670266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf200702a0 con 0x7fdf20155390 2023-03-03T09:59:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.319+0000 7fdf269f4700 1 -- 172.21.15.7:0/1201670266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf20070820 con 0x7fdf20155390 2023-03-03T09:59:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.319+0000 7fdf1cff9700 1 -- 172.21.15.7:0/1201670266 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdf1802c070 con 0x7fdf20155390 2023-03-03T09:59:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.319+0000 7fdf1cff9700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf000682f0 0x7fdf0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.319+0000 7fdf1cff9700 1 -- 172.21.15.7:0/1201670266 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdf18087ba0 con 0x7fdf20155390 2023-03-03T09:59:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.320+0000 7fdf1ffff700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf000682f0 0x7fdf0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.320+0000 7fdf1ffff700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf000682f0 0x7fdf0006a7a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdf10000f60 tx=0x7fdf10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.324+0000 7fdf1cff9700 1 -- 172.21.15.7:0/1201670266 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdf18053f40 con 0x7fdf20155390 2023-03-03T09:59:09.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.564+0000 7fdf269f4700 1 -- 172.21.15.7:0/1201670266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf200c4540 con 0x7fdf20155390 2023-03-03T09:59:09.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.565+0000 7fdf1cff9700 1 -- 172.21.15.7:0/1201670266 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdf200c4540 con 0x7fdf20155390 2023-03-03T09:59:09.566 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:59:09.567 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:59:09.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 -- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf000682f0 msgr2=0x7fdf0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:09.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf000682f0 0x7fdf0006a7a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdf10000f60 tx=0x7fdf10009040 comp rx=0 tx=0).stop 2023-03-03T09:59:09.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 -- 172.21.15.7:0/1201670266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20155390 msgr2=0x7fdf2014c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:09.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20155390 0x7fdf2014c3d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fdf200c4a90 tx=0x7fdf18004ad0 comp rx=0 tx=0).stop 2023-03-03T09:59:09.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 -- 172.21.15.7:0/1201670266 shutdown_connections 2023-03-03T09:59:09.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdf000682f0 0x7fdf0006a7a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:09.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf20155390 0x7fdf2014c3d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:09.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf2006e020 0x7fdf2014e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:09.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 --2- 172.21.15.7:0/1201670266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf20072520 0x7fdf201e5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:09.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 -- 172.21.15.7:0/1201670266 >> 172.21.15.7:0/1201670266 conn(0x7fdf2006c110 msgr2=0x7fdf200c1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:59:09.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 -- 172.21.15.7:0/1201670266 shutdown_connections 2023-03-03T09:59:09.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:09.567+0000 7fdf269f4700 1 -- 172.21.15.7:0/1201670266 wait complete. 2023-03-03T09:59:09.570 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:59:09.580 INFO:tasks.mds_thrash.fs.[a]:waiting for 67 secs before thrashing 2023-03-03T09:59:23.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:23.381 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:31.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:59:31.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.464+0000 7f40ebc97700 1 Processor -- start 2023-03-03T09:59:31.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.464+0000 7f40ebc97700 1 -- start start 2023-03-03T09:59:31.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40ebc97700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40e4071c30 0x7f40e40c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:31.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40ebc97700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40e4155950 0x7f40e4155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:31.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40ebc97700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e40713c0 con 0x7f40e40aac00 2023-03-03T09:59:31.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40ebc97700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e4156310 con 0x7f40e4071c30 2023-03-03T09:59:31.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40ebc97700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e4152c70 con 0x7f40e4155950 2023-03-03T09:59:31.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40e9232700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40e4071c30 0x7f40e40c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:31.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40e9232700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40e4071c30 0x7f40e40c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58606/0 (socket says 172.21.15.7:58606) 2023-03-03T09:59:31.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40e9a33700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f40e40aac00 0x7f40e40beb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52584/0 (socket says 172.21.15.7:52584) 2023-03-03T09:59:31.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40e9232700 1 -- 172.21.15.7:0/3964392579 learned_addr learned my addr 172.21.15.7:0/3964392579 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:59:31.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40ea234700 1 --2- 172.21.15.7:0/3964392579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40e4155950 0x7f40e4155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:31.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40e9232700 1 -- 172.21.15.7:0/3964392579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40e4155950 msgr2=0x7f40e4155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T09:59:31.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40e9232700 1 --2- 172.21.15.7:0/3964392579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40e4155950 0x7f40e4155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40e9232700 1 -- 172.21.15.7:0/3964392579 >> v1:172.21.15.7:6789/0 conn(0x7f40e40aac00 legacy=0x7f40e40beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:31.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40e9232700 1 -- 172.21.15.7:0/3964392579 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e40c4930 con 0x7f40e4071c30 2023-03-03T09:59:31.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.465+0000 7f40e8a31700 1 -- 172.21.15.7:0/3964392579 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612284510 0 0) 0x7f40e40713c0 con 0x7f40e40aac00 2023-03-03T09:59:31.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40e9232700 1 --2- 172.21.15.7:0/3964392579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4071c30 0x7f40e40c43f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f40d40098f0 tx=0x7f40d4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40a9d575998bc32 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:31.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40e8a31700 1 -- 172.21.15.7:0/3964392579 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d400b8a0 con 0x7f40e4071c30 2023-03-03T09:59:31.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40e8a31700 1 -- 172.21.15.7:0/3964392579 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d400ba00 con 0x7f40e4071c30 2023-03-03T09:59:31.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40e8a31700 1 -- 172.21.15.7:0/3964392579 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d400bd90 con 0x7f40e4071c30 2023-03-03T09:59:31.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40ebc97700 1 -- 172.21.15.7:0/3964392579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4071c30 msgr2=0x7f40e40c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:31.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40ebc97700 1 --2- 172.21.15.7:0/3964392579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4071c30 0x7f40e40c43f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f40d40098f0 tx=0x7f40d4009c00 comp rx=0 tx=0).stop 2023-03-03T09:59:31.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40ebc97700 1 -- 172.21.15.7:0/3964392579 shutdown_connections 2023-03-03T09:59:31.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40ebc97700 1 --2- 172.21.15.7:0/3964392579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40e4155950 0x7f40e4155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40ebc97700 1 --2- 172.21.15.7:0/3964392579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4071c30 0x7f40e40c43f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40ebc97700 1 -- 172.21.15.7:0/3964392579 >> 172.21.15.7:0/3964392579 conn(0x7f40e40736f0 msgr2=0x7f40e4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:59:31.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40ebc97700 1 -- 172.21.15.7:0/3964392579 shutdown_connections 2023-03-03T09:59:31.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.466+0000 7f40ebc97700 1 -- 172.21.15.7:0/3964392579 wait complete. 2023-03-03T09:59:31.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ebc97700 1 Processor -- start 2023-03-03T09:59:31.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ebc97700 1 -- start start 2023-03-03T09:59:31.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ebc97700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40e4071c30 0x7f40e40c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:31.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ebc97700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40e40aac00 0x7f40e40ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:31.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ebc97700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4155950 0x7f40e41eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:31.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ebc97700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40c6e10 con 0x7f40e4155950 2023-03-03T09:59:31.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ebc97700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40ad980 con 0x7f40e4071c30 2023-03-03T09:59:31.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ebc97700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40adc60 con 0x7f40e40aac00 2023-03-03T09:59:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ea234700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4155950 0x7f40e41eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ea234700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4155950 0x7f40e41eaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58614/0 (socket says 172.21.15.7:58614) 2023-03-03T09:59:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.468+0000 7f40ea234700 1 -- 172.21.15.7:0/1350367546 learned_addr learned my addr 172.21.15.7:0/1350367546 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:59:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40e9a33700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40e4071c30 0x7f40e40c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40e9232700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40e40aac00 0x7f40e40ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40ea234700 1 -- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40e40aac00 msgr2=0x7f40e40ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40ea234700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40e40aac00 0x7f40e40ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40ea234700 1 -- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40e4071c30 msgr2=0x7f40e40c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40ea234700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40e4071c30 0x7f40e40c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40ea234700 1 -- 172.21.15.7:0/1350367546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4009550 con 0x7f40e4155950 2023-03-03T09:59:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40ea234700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4155950 0x7f40e41eaf80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f40dc007e60 tx=0x7f40dc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40da7fc700 1 -- 172.21.15.7:0/1350367546 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc00ed30 con 0x7f40e4155950 2023-03-03T09:59:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40da7fc700 1 -- 172.21.15.7:0/1350367546 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40dc014420 con 0x7f40e4155950 2023-03-03T09:59:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40ebc97700 1 -- 172.21.15.7:0/1350367546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e40c4b80 con 0x7f40e4155950 2023-03-03T09:59:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40da7fc700 1 -- 172.21.15.7:0/1350367546 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc0147d0 con 0x7f40e4155950 2023-03-03T09:59:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.469+0000 7f40ebc97700 1 -- 172.21.15.7:0/1350367546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e4152360 con 0x7f40e4155950 2023-03-03T09:59:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.470+0000 7f40ebc97700 1 -- 172.21.15.7:0/1350367546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e4152620 con 0x7f40e4155950 2023-03-03T09:59:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.471+0000 7f40da7fc700 1 -- 172.21.15.7:0/1350367546 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f40dc012050 con 0x7f40e4155950 2023-03-03T09:59:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.471+0000 7f40da7fc700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c80682f0 0x7f40c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.471+0000 7f40da7fc700 1 -- 172.21.15.7:0/1350367546 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f40dc086670 con 0x7f40e4155950 2023-03-03T09:59:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.471+0000 7f40e9a33700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c80682f0 0x7f40c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.472+0000 7f40e9a33700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c80682f0 0x7f40c806a7a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f40e0007fd0 tx=0x7f40e0007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.475+0000 7f40da7fc700 1 -- 172.21.15.7:0/1350367546 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f40dc052a10 con 0x7f40e4155950 2023-03-03T09:59:31.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.713+0000 7f40ebc97700 1 -- 172.21.15.7:0/1350367546 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40e40721c0 con 0x7f40e4155950 2023-03-03T09:59:31.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.714+0000 7f40da7fc700 1 -- 172.21.15.7:0/1350367546 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f40dc056030 con 0x7f40e4155950 2023-03-03T09:59:31.716 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:59:31.716 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:59:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 -- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c80682f0 msgr2=0x7f40c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c80682f0 0x7f40c806a7a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f40e0007fd0 tx=0x7f40e0007de0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 -- 172.21.15.7:0/1350367546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4155950 msgr2=0x7f40e41eaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4155950 0x7f40e41eaf80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f40dc007e60 tx=0x7f40dc00e3c0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 -- 172.21.15.7:0/1350367546 shutdown_connections 2023-03-03T09:59:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c80682f0 0x7f40c806a7a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40e4155950 0x7f40e41eaf80 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40e4071c30 0x7f40e40c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 --2- 172.21.15.7:0/1350367546 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40e40aac00 0x7f40e40ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 -- 172.21.15.7:0/1350367546 >> 172.21.15.7:0/1350367546 conn(0x7f40e40736f0 msgr2=0x7f40e4153290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:59:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 -- 172.21.15.7:0/1350367546 shutdown_connections 2023-03-03T09:59:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:31.716+0000 7f40ebc97700 1 -- 172.21.15.7:0/1350367546 wait complete. 2023-03-03T09:59:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:59:31.729 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T09:59:42.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T09:59:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.868+0000 7fd8687c3700 1 Processor -- start 2023-03-03T09:59:42.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.868+0000 7fd8687c3700 1 -- start start 2023-03-03T09:59:42.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd8687c3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd86006c480 0x7fd8601581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:42.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd8687c3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86006d430 con 0x7fd8600c2930 2023-03-03T09:59:42.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd8687c3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8601586e0 con 0x7fd860151d40 2023-03-03T09:59:42.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd8687c3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86006c2a0 con 0x7fd86006c480 2023-03-03T09:59:42.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd866d60700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd86006c480 0x7fd8601581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:42.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd865d5e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd8600c2930 0x7fd860157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52604/0 (socket says 172.21.15.7:52604) 2023-03-03T09:59:42.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd865d5e700 1 -- 172.21.15.7:0/3941410454 learned_addr learned my addr 172.21.15.7:0/3941410454 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T09:59:42.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd866d60700 1 -- 172.21.15.7:0/3941410454 >> v1:172.21.15.143:6789/0 conn(0x7fd860151d40 legacy=0x7fd8600736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:42.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd866d60700 1 -- 172.21.15.7:0/3941410454 >> v1:172.21.15.7:6789/0 conn(0x7fd8600c2930 legacy=0x7fd860157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:42.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd86555d700 1 -- 172.21.15.7:0/3941410454 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430290025 0 0) 0x7fd86006d430 con 0x7fd8600c2930 2023-03-03T09:59:42.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd866d60700 1 -- 172.21.15.7:0/3941410454 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86006cb00 con 0x7fd86006c480 2023-03-03T09:59:42.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.869+0000 7fd866d60700 1 --2- 172.21.15.7:0/3941410454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 0x7fd8601581a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd858009dc0 tx=0x7fd85800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a00b507e0d9d5d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.870+0000 7fd86555d700 1 -- 172.21.15.7:0/3941410454 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85800e070 con 0x7fd86006c480 2023-03-03T09:59:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.870+0000 7fd86555d700 1 -- 172.21.15.7:0/3941410454 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85800bb90 con 0x7fd86006c480 2023-03-03T09:59:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.870+0000 7fd86555d700 1 -- 172.21.15.7:0/3941410454 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858012430 con 0x7fd86006c480 2023-03-03T09:59:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.870+0000 7fd8687c3700 1 -- 172.21.15.7:0/3941410454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 msgr2=0x7fd8601581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.870+0000 7fd8687c3700 1 --2- 172.21.15.7:0/3941410454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 0x7fd8601581a0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd858009dc0 tx=0x7fd85800b330 comp rx=0 tx=0).stop 2023-03-03T09:59:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.870+0000 7fd8687c3700 1 -- 172.21.15.7:0/3941410454 shutdown_connections 2023-03-03T09:59:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.870+0000 7fd8687c3700 1 --2- 172.21.15.7:0/3941410454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 0x7fd8601581a0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:42.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.870+0000 7fd8687c3700 1 -- 172.21.15.7:0/3941410454 >> 172.21.15.7:0/3941410454 conn(0x7fd8600a94d0 msgr2=0x7fd8600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:59:42.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.870+0000 7fd8687c3700 1 -- 172.21.15.7:0/3941410454 shutdown_connections 2023-03-03T09:59:42.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.870+0000 7fd8687c3700 1 -- 172.21.15.7:0/3941410454 wait complete. 2023-03-03T09:59:42.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.871+0000 7fd8687c3700 1 Processor -- start 2023-03-03T09:59:42.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.871+0000 7fd8687c3700 1 -- start start 2023-03-03T09:59:42.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd8687c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 0x7fd8600be960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:42.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd8687c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c2930 0x7fd8601eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:42.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd8687c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd860151d40 0x7fd8600aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:42.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd8687c3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601ea0b0 con 0x7fd86006c480 2023-03-03T09:59:42.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd8687c3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600beea0 con 0x7fd860151d40 2023-03-03T09:59:42.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd8687c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600be430 con 0x7fd8600c2930 2023-03-03T09:59:42.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd86655f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 0x7fd8600be960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:42.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd86655f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 0x7fd8600be960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58628/0 (socket says 172.21.15.7:58628) 2023-03-03T09:59:42.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd86655f700 1 -- 172.21.15.7:0/2094056197 learned_addr learned my addr 172.21.15.7:0/2094056197 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T09:59:42.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd866d60700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd860151d40 0x7fd8600aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:42.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd86655f700 1 -- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c2930 msgr2=0x7fd8601eaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:42.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.872+0000 7fd865d5e700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c2930 0x7fd8601eaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:42.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd86655f700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c2930 0x7fd8601eaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:42.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd86655f700 1 -- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd860151d40 msgr2=0x7fd8600aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:42.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd86655f700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd860151d40 0x7fd8600aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:42.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd86655f700 1 -- 172.21.15.7:0/2094056197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd858009a70 con 0x7fd86006c480 2023-03-03T09:59:42.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd866d60700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd860151d40 0x7fd8600aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T09:59:42.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd86655f700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 0x7fd8600be960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00c510 tx=0x7fd85c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:42.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd856ffd700 1 -- 172.21.15.7:0/2094056197 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0052d0 con 0x7fd86006c480 2023-03-03T09:59:42.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd856ffd700 1 -- 172.21.15.7:0/2094056197 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c005450 con 0x7fd86006c480 2023-03-03T09:59:42.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd856ffd700 1 -- 172.21.15.7:0/2094056197 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0038c0 con 0x7fd86006c480 2023-03-03T09:59:42.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd8687c3700 1 -- 172.21.15.7:0/2094056197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86006d0c0 con 0x7fd86006c480 2023-03-03T09:59:42.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.873+0000 7fd8687c3700 1 -- 172.21.15.7:0/2094056197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86006cde0 con 0x7fd86006c480 2023-03-03T09:59:42.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.874+0000 7fd8687c3700 1 -- 172.21.15.7:0/2094056197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860073e00 con 0x7fd86006c480 2023-03-03T09:59:42.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.874+0000 7fd856ffd700 1 -- 172.21.15.7:0/2094056197 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd85c019030 con 0x7fd86006c480 2023-03-03T09:59:42.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.875+0000 7fd856ffd700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 0x7fd84406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T09:59:42.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.875+0000 7fd856ffd700 1 -- 172.21.15.7:0/2094056197 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd85c0854a0 con 0x7fd86006c480 2023-03-03T09:59:42.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.875+0000 7fd865d5e700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 0x7fd84406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T09:59:42.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.876+0000 7fd865d5e700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 0x7fd84406a7a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd850007fd0 tx=0x7fd850009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T09:59:42.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:42.879+0000 7fd856ffd700 1 -- 172.21.15.7:0/2094056197 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd85c051840 con 0x7fd86006c480 2023-03-03T09:59:43.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.117+0000 7fd8687c3700 1 -- 172.21.15.7:0/2094056197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd860074bb0 con 0x7fd86006c480 2023-03-03T09:59:43.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.118+0000 7fd856ffd700 1 -- 172.21.15.7:0/2094056197 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd85c054e60 con 0x7fd86006c480 2023-03-03T09:59:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T09:59:43.119 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T09:59:43.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 -- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 msgr2=0x7fd84406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:43.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 0x7fd84406a7a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd850007fd0 tx=0x7fd850009040 comp rx=0 tx=0).stop 2023-03-03T09:59:43.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 -- 172.21.15.7:0/2094056197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 msgr2=0x7fd8600be960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T09:59:43.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 0x7fd8600be960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00c510 tx=0x7fd85c00c8d0 comp rx=0 tx=0).stop 2023-03-03T09:59:43.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 -- 172.21.15.7:0/2094056197 shutdown_connections 2023-03-03T09:59:43.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd860151d40 0x7fd8600aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:43.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 0x7fd84406a7a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:43.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86006c480 0x7fd8600be960 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:43.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 --2- 172.21.15.7:0/2094056197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c2930 0x7fd8601eaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T09:59:43.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 -- 172.21.15.7:0/2094056197 >> 172.21.15.7:0/2094056197 conn(0x7fd8600a94d0 msgr2=0x7fd860071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T09:59:43.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 -- 172.21.15.7:0/2094056197 shutdown_connections 2023-03-03T09:59:43.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T09:59:43.120+0000 7fd8687c3700 1 -- 172.21.15.7:0/2094056197 wait complete. 2023-03-03T09:59:43.123 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T09:59:43.133 INFO:tasks.mds_thrash.fs.[b]:waiting for 32 secs before thrashing 2023-03-03T09:59:53.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T09:59:53.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T09:59:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T10:00:15.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:00:15.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.274+0000 7f56a0d92700 1 Processor -- start 2023-03-03T10:00:15.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.274+0000 7f56a0d92700 1 -- start start 2023-03-03T10:00:15.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.275+0000 7f56a0d92700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f569c0aad20 0x7f569c157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:15.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.275+0000 7f56a0d92700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c1588f0 con 0x7f569c0aa030 2023-03-03T10:00:15.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.275+0000 7f56a0d92700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c0728b0 con 0x7f569c14b020 2023-03-03T10:00:15.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.275+0000 7f56a0d92700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569c155070 con 0x7f569c0aad20 2023-03-03T10:00:15.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.275+0000 7f569ad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f569c0aa030 0x7f569c1581e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52640/0 (socket says 172.21.15.7:52640) 2023-03-03T10:00:15.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.275+0000 7f569ad9d700 1 -- 172.21.15.7:0/2359449403 learned_addr learned my addr 172.21.15.7:0/2359449403 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:00:15.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f5699d9b700 1 --2- 172.21.15.7:0/2359449403 >> v2:172.21.15.143:3300/0 conn(0x7f569c0aad20 0x7f569c157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:15.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f569959a700 1 -- 172.21.15.7:0/2359449403 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389579090 0 0) 0x7f569c1588f0 con 0x7f569c0aa030 2023-03-03T10:00:15.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f569959a700 1 -- 172.21.15.7:0/2359449403 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56840035b0 con 0x7f569c0aa030 2023-03-03T10:00:15.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f569959a700 1 -- 172.21.15.7:0/2359449403 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61173509 0 0) 0x7f569c0728b0 con 0x7f569c14b020 2023-03-03T10:00:15.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f569959a700 1 -- 172.21.15.7:0/2359449403 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f569c1588f0 con 0x7f569c14b020 2023-03-03T10:00:15.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f569959a700 1 -- 172.21.15.7:0/2359449403 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4089879529 0 0) 0x7f56840035b0 con 0x7f569c0aa030 2023-03-03T10:00:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f5699d9b700 1 -- 172.21.15.7:0/2359449403 >> v1:172.21.15.143:6789/0 conn(0x7f569c14b020 legacy=0x7f569c0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f5699d9b700 1 -- 172.21.15.7:0/2359449403 >> v1:172.21.15.7:6789/0 conn(0x7f569c0aa030 legacy=0x7f569c1581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f5699d9b700 1 -- 172.21.15.7:0/2359449403 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c06f5b0 con 0x7f569c0aad20 2023-03-03T10:00:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f5699d9b700 1 --2- 172.21.15.7:0/2359449403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f569c0aad20 0x7f569c157ca0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5690009840 tx=0x7f5690009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b04a623d05ce02f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.276+0000 7f569959a700 1 -- 172.21.15.7:0/2359449403 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56940019f0 con 0x7f569c0aad20 2023-03-03T10:00:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.277+0000 7f569959a700 1 -- 172.21.15.7:0/2359449403 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5690004030 con 0x7f569c0aad20 2023-03-03T10:00:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.277+0000 7f569959a700 1 -- 172.21.15.7:0/2359449403 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56900043c0 con 0x7f569c0aad20 2023-03-03T10:00:15.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.277+0000 7f56a0d92700 1 -- 172.21.15.7:0/2359449403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f569c0aad20 msgr2=0x7f569c157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:15.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.277+0000 7f56a0d92700 1 --2- 172.21.15.7:0/2359449403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f569c0aad20 0x7f569c157ca0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5690009840 tx=0x7f5690009b50 comp rx=0 tx=0).stop 2023-03-03T10:00:15.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.277+0000 7f56a0d92700 1 -- 172.21.15.7:0/2359449403 shutdown_connections 2023-03-03T10:00:15.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.277+0000 7f56a0d92700 1 --2- 172.21.15.7:0/2359449403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f569c0aad20 0x7f569c157ca0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:15.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.277+0000 7f56a0d92700 1 -- 172.21.15.7:0/2359449403 >> 172.21.15.7:0/2359449403 conn(0x7f569c0c8900 msgr2=0x7f569c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:00:15.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.277+0000 7f56a0d92700 1 -- 172.21.15.7:0/2359449403 shutdown_connections 2023-03-03T10:00:15.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.277+0000 7f56a0d92700 1 -- 172.21.15.7:0/2359449403 wait complete. 2023-03-03T10:00:15.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.278+0000 7f56a0d92700 1 Processor -- start 2023-03-03T10:00:15.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f56a0d92700 1 -- start start 2023-03-03T10:00:15.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f56a0d92700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f569c0aa030 0x7f569c153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:15.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f56a0d92700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f569c0aad20 0x7f569c1ea870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:15.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f56a0d92700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f569c14b020 0x7f569c147a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:15.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f56a0d92700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c13ee60 con 0x7f569c14b020 2023-03-03T10:00:15.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f56a0d92700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c1e9e80 con 0x7f569c0aa030 2023-03-03T10:00:15.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f56a0d92700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c145a00 con 0x7f569c0aad20 2023-03-03T10:00:15.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f569ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f569c14b020 0x7f569c147a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f569ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f569c14b020 0x7f569c147a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58668/0 (socket says 172.21.15.7:58668) 2023-03-03T10:00:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f569ad9d700 1 -- 172.21.15.7:0/2414077932 learned_addr learned my addr 172.21.15.7:0/2414077932 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:00:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f569a59c700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f569c0aa030 0x7f569c153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.279+0000 7f5699d9b700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f569c0aad20 0x7f569c1ea870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f569ad9d700 1 -- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f569c0aad20 msgr2=0x7f569c1ea870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f569ad9d700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f569c0aad20 0x7f569c1ea870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:15.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f569ad9d700 1 -- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f569c0aa030 msgr2=0x7f569c153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f569ad9d700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f569c0aa030 0x7f569c153020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f569ad9d700 1 -- 172.21.15.7:0/2414077932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56900094a0 con 0x7f569c14b020 2023-03-03T10:00:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f569ad9d700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f569c14b020 0x7f569c147a10 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f569400cbd0 tx=0x7f569400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f568affd700 1 -- 172.21.15.7:0/2414077932 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569400aa60 con 0x7f569c14b020 2023-03-03T10:00:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f568affd700 1 -- 172.21.15.7:0/2414077932 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569400abe0 con 0x7f569c14b020 2023-03-03T10:00:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f568affd700 1 -- 172.21.15.7:0/2414077932 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56940070d0 con 0x7f569c14b020 2023-03-03T10:00:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f56a0d92700 1 -- 172.21.15.7:0/2414077932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569c06fb70 con 0x7f569c14b020 2023-03-03T10:00:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.280+0000 7f56a0d92700 1 -- 172.21.15.7:0/2414077932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569c06f890 con 0x7f569c14b020 2023-03-03T10:00:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.281+0000 7f56a0d92700 1 -- 172.21.15.7:0/2414077932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569c155350 con 0x7f569c14b020 2023-03-03T10:00:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.281+0000 7f568affd700 1 -- 172.21.15.7:0/2414077932 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5694010070 con 0x7f569c14b020 2023-03-03T10:00:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.282+0000 7f568affd700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f567c0682f0 0x7f567c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.282+0000 7f568affd700 1 -- 172.21.15.7:0/2414077932 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5694085a20 con 0x7f569c14b020 2023-03-03T10:00:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.282+0000 7f569a59c700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f567c0682f0 0x7f567c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.283+0000 7f569a59c700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f567c0682f0 0x7f567c06a7a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f568c0011e0 tx=0x7f568c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.286+0000 7f568affd700 1 -- 172.21.15.7:0/2414077932 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5694051dc0 con 0x7f569c14b020 2023-03-03T10:00:15.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.522+0000 7f56a0d92700 1 -- 172.21.15.7:0/2414077932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f569c074bb0 con 0x7f569c14b020 2023-03-03T10:00:15.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.523+0000 7f568affd700 1 -- 172.21.15.7:0/2414077932 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f56940553e0 con 0x7f569c14b020 2023-03-03T10:00:15.524 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:00:15.525 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:00:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 -- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f567c0682f0 msgr2=0x7f567c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f567c0682f0 0x7f567c06a7a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f568c0011e0 tx=0x7f568c009040 comp rx=0 tx=0).stop 2023-03-03T10:00:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 -- 172.21.15.7:0/2414077932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f569c14b020 msgr2=0x7f569c147a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f569c14b020 0x7f569c147a10 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f569400cbd0 tx=0x7f569400cee0 comp rx=0 tx=0).stop 2023-03-03T10:00:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 -- 172.21.15.7:0/2414077932 shutdown_connections 2023-03-03T10:00:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f567c0682f0 0x7f567c06a7a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f569c14b020 0x7f569c147a10 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f569c0aa030 0x7f569c153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 --2- 172.21.15.7:0/2414077932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f569c0aad20 0x7f569c1ea870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:15.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 -- 172.21.15.7:0/2414077932 >> 172.21.15.7:0/2414077932 conn(0x7f569c0c8900 msgr2=0x7f569c0c3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:00:15.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 -- 172.21.15.7:0/2414077932 shutdown_connections 2023-03-03T10:00:15.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:15.525+0000 7f56a0d92700 1 -- 172.21.15.7:0/2414077932 wait complete. 2023-03-03T10:00:15.528 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:00:15.538 INFO:tasks.mds_thrash.fs.[b]:waiting for 26 secs before thrashing 2023-03-03T10:00:16.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:00:16.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.719+0000 7ffa371b7700 1 Processor -- start 2023-03-03T10:00:16.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.719+0000 7ffa371b7700 1 -- start start 2023-03-03T10:00:16.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa371b7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffa30151ab0 0x7ffa300736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:16.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa371b7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa3006c5b0 0x7ffa30157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:16.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa371b7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa3006d780 con 0x7ffa30073c30 2023-03-03T10:00:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa371b7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa3006cb70 con 0x7ffa30151ab0 2023-03-03T10:00:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa371b7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3006ce50 con 0x7ffa3006c5b0 2023-03-03T10:00:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa34f53700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffa30151ab0 0x7ffa300736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa2ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ffa30073c30 0x7ffa3006bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52654/0 (socket says 172.21.15.7:52654) 2023-03-03T10:00:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa2ffff700 1 -- 172.21.15.7:0/891699447 learned_addr learned my addr 172.21.15.7:0/891699447 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:00:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa35754700 1 --2- 172.21.15.7:0/891699447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa3006c5b0 0x7ffa30157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa34f53700 1 -- 172.21.15.7:0/891699447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa3006c5b0 msgr2=0x7ffa30157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:00:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa34f53700 1 --2- 172.21.15.7:0/891699447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa3006c5b0 0x7ffa30157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa34f53700 1 -- 172.21.15.7:0/891699447 >> v1:172.21.15.7:6789/0 conn(0x7ffa30073c30 legacy=0x7ffa3006bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa34f53700 1 -- 172.21.15.7:0/891699447 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3006d130 con 0x7ffa30151ab0 2023-03-03T10:00:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa2f7fe700 1 -- 172.21.15.7:0/891699447 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232018792 0 0) 0x7ffa3006d780 con 0x7ffa30073c30 2023-03-03T10:00:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa35754700 1 --2- 172.21.15.7:0/891699447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa3006c5b0 0x7ffa30157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:00:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa34f53700 1 --2- 172.21.15.7:0/891699447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa30151ab0 0x7ffa300736f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009840 tx=0x7ffa24009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f65d0b9cd3e37af server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.720+0000 7ffa2f7fe700 1 -- 172.21.15.7:0/891699447 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24011830 con 0x7ffa30151ab0 2023-03-03T10:00:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.721+0000 7ffa2f7fe700 1 -- 172.21.15.7:0/891699447 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24011990 con 0x7ffa30151ab0 2023-03-03T10:00:16.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.721+0000 7ffa2f7fe700 1 -- 172.21.15.7:0/891699447 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24011d20 con 0x7ffa30151ab0 2023-03-03T10:00:16.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.721+0000 7ffa371b7700 1 -- 172.21.15.7:0/891699447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa30151ab0 msgr2=0x7ffa300736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:16.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.721+0000 7ffa371b7700 1 --2- 172.21.15.7:0/891699447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa30151ab0 0x7ffa300736f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009840 tx=0x7ffa24009b50 comp rx=0 tx=0).stop 2023-03-03T10:00:16.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.721+0000 7ffa371b7700 1 -- 172.21.15.7:0/891699447 shutdown_connections 2023-03-03T10:00:16.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.721+0000 7ffa371b7700 1 --2- 172.21.15.7:0/891699447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa3006c5b0 0x7ffa30157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:16.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.721+0000 7ffa371b7700 1 --2- 172.21.15.7:0/891699447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa30151ab0 0x7ffa300736f0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:16.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.721+0000 7ffa371b7700 1 -- 172.21.15.7:0/891699447 >> 172.21.15.7:0/891699447 conn(0x7ffa300a94d0 msgr2=0x7ffa300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:00:16.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.721+0000 7ffa371b7700 1 -- 172.21.15.7:0/891699447 shutdown_connections 2023-03-03T10:00:16.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.721+0000 7ffa371b7700 1 -- 172.21.15.7:0/891699447 wait complete. 2023-03-03T10:00:16.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.722+0000 7ffa371b7700 1 Processor -- start 2023-03-03T10:00:16.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.722+0000 7ffa371b7700 1 -- start start 2023-03-03T10:00:16.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.723+0000 7ffa371b7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa3006c5b0 0x7ffa30146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:16.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.723+0000 7ffa371b7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa30073c30 0x7ffa30146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:16.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.723+0000 7ffa371b7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa30151ab0 0x7ffa300c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:16.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.723+0000 7ffa371b7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa301471b0 con 0x7ffa3006c5b0 2023-03-03T10:00:16.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.723+0000 7ffa371b7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3014c220 con 0x7ffa30073c30 2023-03-03T10:00:16.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.723+0000 7ffa371b7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3014c500 con 0x7ffa30151ab0 2023-03-03T10:00:16.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.723+0000 7ffa34f53700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa3006c5b0 0x7ffa30146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:16.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.723+0000 7ffa34f53700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa3006c5b0 0x7ffa30146b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58680/0 (socket says 172.21.15.7:58680) 2023-03-03T10:00:16.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.723+0000 7ffa34f53700 1 -- 172.21.15.7:0/2967494480 learned_addr learned my addr 172.21.15.7:0/2967494480 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:00:16.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.723+0000 7ffa2ffff700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa30073c30 0x7ffa30146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:16.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa35754700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa30151ab0 0x7ffa300c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:16.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa34f53700 1 -- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa30151ab0 msgr2=0x7ffa300c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:16.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa34f53700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa30151ab0 0x7ffa300c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:16.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa34f53700 1 -- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa30073c30 msgr2=0x7ffa30146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:16.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa34f53700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa30073c30 0x7ffa30146110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:16.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa34f53700 1 -- 172.21.15.7:0/2967494480 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa240094a0 con 0x7ffa3006c5b0 2023-03-03T10:00:16.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa34f53700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa3006c5b0 0x7ffa30146b40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ffa24014040 tx=0x7ffa24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:16.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa2d7fa700 1 -- 172.21.15.7:0/2967494480 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa240036a0 con 0x7ffa3006c5b0 2023-03-03T10:00:16.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa2d7fa700 1 -- 172.21.15.7:0/2967494480 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2401e070 con 0x7ffa3006c5b0 2023-03-03T10:00:16.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa371b7700 1 -- 172.21.15.7:0/2967494480 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa30151e80 con 0x7ffa3006c5b0 2023-03-03T10:00:16.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa2d7fa700 1 -- 172.21.15.7:0/2967494480 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24003a40 con 0x7ffa3006c5b0 2023-03-03T10:00:16.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.724+0000 7ffa371b7700 1 -- 172.21.15.7:0/2967494480 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3006d410 con 0x7ffa3006c5b0 2023-03-03T10:00:16.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.725+0000 7ffa371b7700 1 -- 172.21.15.7:0/2967494480 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa30152010 con 0x7ffa3006c5b0 2023-03-03T10:00:16.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.725+0000 7ffa2d7fa700 1 -- 172.21.15.7:0/2967494480 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffa2400f050 con 0x7ffa3006c5b0 2023-03-03T10:00:16.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.726+0000 7ffa2d7fa700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa140682f0 0x7ffa1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:16.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.726+0000 7ffa2d7fa700 1 -- 172.21.15.7:0/2967494480 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ffa24087320 con 0x7ffa3006c5b0 2023-03-03T10:00:16.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.726+0000 7ffa2ffff700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa140682f0 0x7ffa1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:16.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.726+0000 7ffa2ffff700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa140682f0 0x7ffa1406a7a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ffa30147bf0 tx=0x7ffa1c007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:16.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.730+0000 7ffa2d7fa700 1 -- 172.21.15.7:0/2967494480 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffa240535d0 con 0x7ffa3006c5b0 2023-03-03T10:00:16.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.969+0000 7ffa371b7700 1 -- 172.21.15.7:0/2967494480 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa30074bb0 con 0x7ffa3006c5b0 2023-03-03T10:00:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.969+0000 7ffa2d7fa700 1 -- 172.21.15.7:0/2967494480 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ffa24015090 con 0x7ffa3006c5b0 2023-03-03T10:00:16.971 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:00:16.971 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:00:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.971+0000 7ffa371b7700 1 -- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa140682f0 msgr2=0x7ffa1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.971+0000 7ffa371b7700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa140682f0 0x7ffa1406a7a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ffa30147bf0 tx=0x7ffa1c007de0 comp rx=0 tx=0).stop 2023-03-03T10:00:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.971+0000 7ffa371b7700 1 -- 172.21.15.7:0/2967494480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa3006c5b0 msgr2=0x7ffa30146b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.971+0000 7ffa371b7700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa3006c5b0 0x7ffa30146b40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ffa24014040 tx=0x7ffa24004750 comp rx=0 tx=0).stop 2023-03-03T10:00:16.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.972+0000 7ffa371b7700 1 -- 172.21.15.7:0/2967494480 shutdown_connections 2023-03-03T10:00:16.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.972+0000 7ffa371b7700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa30151ab0 0x7ffa300c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:16.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.972+0000 7ffa371b7700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa140682f0 0x7ffa1406a7a0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:16.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.972+0000 7ffa371b7700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa3006c5b0 0x7ffa30146b40 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:16.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.972+0000 7ffa371b7700 1 --2- 172.21.15.7:0/2967494480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa30073c30 0x7ffa30146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:16.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.972+0000 7ffa371b7700 1 -- 172.21.15.7:0/2967494480 >> 172.21.15.7:0/2967494480 conn(0x7ffa300a94d0 msgr2=0x7ffa30071210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:00:16.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.972+0000 7ffa371b7700 1 -- 172.21.15.7:0/2967494480 shutdown_connections 2023-03-03T10:00:16.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:16.972+0000 7ffa371b7700 1 -- 172.21.15.7:0/2967494480 wait complete. 2023-03-03T10:00:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:00:16.985 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before thrashing 2023-03-03T10:00:20.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:00:21.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.123+0000 7f4b242fe700 1 Processor -- start 2023-03-03T10:00:21.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.123+0000 7f4b242fe700 1 -- start start 2023-03-03T10:00:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.124+0000 7f4b242fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 0x7f4b1c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.124+0000 7f4b242fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4b1c152b60 0x7f4b1c15ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.124+0000 7f4b242fe700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1c0ac900 con 0x7f4b1c073c30 2023-03-03T10:00:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.124+0000 7f4b242fe700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b1c071e30 con 0x7f4b1c152b60 2023-03-03T10:00:21.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.124+0000 7f4b242fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c0acb50 con 0x7f4b1c0ac220 2023-03-03T10:00:21.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.124+0000 7f4b2209a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 0x7f4b1c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:21.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b21899700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f4b1c073c30 0x7f4b1c152450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:56188/0 (socket says 172.21.15.7:56188) 2023-03-03T10:00:21.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.124+0000 7f4b2289b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4b1c152b60 0x7f4b1c15ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:21.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b21899700 1 -- 172.21.15.7:0/3522454660 learned_addr learned my addr 172.21.15.7:0/3522454660 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:00:21.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b21098700 1 -- 172.21.15.7:0/3522454660 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435601517 0 0) 0x7f4b1c0ac900 con 0x7f4b1c073c30 2023-03-03T10:00:21.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b21098700 1 -- 172.21.15.7:0/3522454660 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b080035b0 con 0x7f4b1c073c30 2023-03-03T10:00:21.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b2209a700 1 -- 172.21.15.7:0/3522454660 >> v2:172.21.15.143:3300/0 conn(0x7f4b1c152b60 msgr2=0x7f4b1c15ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:00:21.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b2209a700 1 --2- 172.21.15.7:0/3522454660 >> v2:172.21.15.143:3300/0 conn(0x7f4b1c152b60 0x7f4b1c15ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:21.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b2209a700 1 -- 172.21.15.7:0/3522454660 >> v1:172.21.15.143:6789/0 conn(0x7f4b1c073c30 legacy=0x7f4b1c152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:21.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b2209a700 1 -- 172.21.15.7:0/3522454660 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c1559c0 con 0x7f4b1c0ac220 2023-03-03T10:00:21.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b2289b700 1 --2- 172.21.15.7:0/3522454660 >> v2:172.21.15.143:3300/0 conn(0x7f4b1c152b60 0x7f4b1c15ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:00:21.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b2209a700 1 --2- 172.21.15.7:0/3522454660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 0x7f4b1c0736f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b18009840 tx=0x7f4b18009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=712592afeeee332e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:21.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b21098700 1 -- 172.21.15.7:0/3522454660 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18011830 con 0x7f4b1c0ac220 2023-03-03T10:00:21.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b21098700 1 -- 172.21.15.7:0/3522454660 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b18011990 con 0x7f4b1c0ac220 2023-03-03T10:00:21.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.125+0000 7f4b21098700 1 -- 172.21.15.7:0/3522454660 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18011d00 con 0x7f4b1c0ac220 2023-03-03T10:00:21.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.126+0000 7f4b242fe700 1 -- 172.21.15.7:0/3522454660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 msgr2=0x7f4b1c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:21.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.126+0000 7f4b242fe700 1 --2- 172.21.15.7:0/3522454660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 0x7f4b1c0736f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b18009840 tx=0x7f4b18009b50 comp rx=0 tx=0).stop 2023-03-03T10:00:21.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.126+0000 7f4b242fe700 1 -- 172.21.15.7:0/3522454660 shutdown_connections 2023-03-03T10:00:21.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.126+0000 7f4b242fe700 1 --2- 172.21.15.7:0/3522454660 >> v2:172.21.15.143:3300/0 conn(0x7f4b1c152b60 0x7f4b1c15ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:21.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.126+0000 7f4b242fe700 1 --2- 172.21.15.7:0/3522454660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 0x7f4b1c0736f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:21.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.126+0000 7f4b242fe700 1 -- 172.21.15.7:0/3522454660 >> 172.21.15.7:0/3522454660 conn(0x7f4b1c0a94d0 msgr2=0x7f4b1c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:00:21.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.126+0000 7f4b242fe700 1 -- 172.21.15.7:0/3522454660 shutdown_connections 2023-03-03T10:00:21.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.126+0000 7f4b242fe700 1 -- 172.21.15.7:0/3522454660 wait complete. 2023-03-03T10:00:21.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.127+0000 7f4b242fe700 1 Processor -- start 2023-03-03T10:00:21.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.127+0000 7f4b242fe700 1 -- start start 2023-03-03T10:00:21.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b242fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b1c073c30 0x7f4b1c0bddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:21.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b242fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 0x7f4b1c0c1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:21.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b242fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b1c152b60 0x7f4b1c0bae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:21.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b242fe700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c1ed980 con 0x7f4b1c152b60 2023-03-03T10:00:21.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b242fe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c1edeb0 con 0x7f4b1c073c30 2023-03-03T10:00:21.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b242fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c0bb390 con 0x7f4b1c0ac220 2023-03-03T10:00:21.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b2289b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b1c152b60 0x7f4b1c0bae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:21.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b2289b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b1c152b60 0x7f4b1c0bae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58694/0 (socket says 172.21.15.7:58694) 2023-03-03T10:00:21.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b2289b700 1 -- 172.21.15.7:0/4241361122 learned_addr learned my addr 172.21.15.7:0/4241361122 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:00:21.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b21899700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 0x7f4b1c0c1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:21.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b2209a700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b1c073c30 0x7f4b1c0bddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:21.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b2289b700 1 -- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 msgr2=0x7f4b1c0c1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:21.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b2289b700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 0x7f4b1c0c1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:21.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b2289b700 1 -- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b1c073c30 msgr2=0x7f4b1c0bddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:21.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b2289b700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b1c073c30 0x7f4b1c0bddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:21.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b2289b700 1 -- 172.21.15.7:0/4241361122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b14009a70 con 0x7f4b1c152b60 2023-03-03T10:00:21.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.128+0000 7f4b2209a700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b1c073c30 0x7f4b1c0bddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:00:21.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.129+0000 7f4b2289b700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b1c152b60 0x7f4b1c0bae50 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4b14009f00 tx=0x7f4b1400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:21.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.129+0000 7f4b0effd700 1 -- 172.21.15.7:0/4241361122 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b14009d70 con 0x7f4b1c152b60 2023-03-03T10:00:21.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.129+0000 7f4b0effd700 1 -- 172.21.15.7:0/4241361122 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1400f040 con 0x7f4b1c152b60 2023-03-03T10:00:21.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.129+0000 7f4b242fe700 1 -- 172.21.15.7:0/4241361122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b180094a0 con 0x7f4b1c152b60 2023-03-03T10:00:21.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.129+0000 7f4b0effd700 1 -- 172.21.15.7:0/4241361122 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1400cd70 con 0x7f4b1c152b60 2023-03-03T10:00:21.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.129+0000 7f4b242fe700 1 -- 172.21.15.7:0/4241361122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1c155f80 con 0x7f4b1c152b60 2023-03-03T10:00:21.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.130+0000 7f4b242fe700 1 -- 172.21.15.7:0/4241361122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1c155ca0 con 0x7f4b1c152b60 2023-03-03T10:00:21.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.130+0000 7f4b0effd700 1 -- 172.21.15.7:0/4241361122 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4b14014480 con 0x7f4b1c152b60 2023-03-03T10:00:21.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.131+0000 7f4b0effd700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b000682f0 0x7f4b0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:21.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.131+0000 7f4b0effd700 1 -- 172.21.15.7:0/4241361122 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4b14086920 con 0x7f4b1c152b60 2023-03-03T10:00:21.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.131+0000 7f4b2209a700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b000682f0 0x7f4b0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:21.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.132+0000 7f4b2209a700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b000682f0 0x7f4b0006a7a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4b18000c00 tx=0x7f4b1801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:21.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.135+0000 7f4b0effd700 1 -- 172.21.15.7:0/4241361122 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b14052c40 con 0x7f4b1c152b60 2023-03-03T10:00:21.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.372+0000 7f4b242fe700 1 -- 172.21.15.7:0/4241361122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b1c0c0340 con 0x7f4b1c152b60 2023-03-03T10:00:21.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.373+0000 7f4b0effd700 1 -- 172.21.15.7:0/4241361122 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4b14056260 con 0x7f4b1c152b60 2023-03-03T10:00:21.374 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:00:21.374 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:00:21.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 -- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b000682f0 msgr2=0x7f4b0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:21.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b000682f0 0x7f4b0006a7a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4b18000c00 tx=0x7f4b1801c040 comp rx=0 tx=0).stop 2023-03-03T10:00:21.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 -- 172.21.15.7:0/4241361122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b1c152b60 msgr2=0x7f4b1c0bae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:21.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b1c152b60 0x7f4b1c0bae50 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f4b14009f00 tx=0x7f4b1400c3a0 comp rx=0 tx=0).stop 2023-03-03T10:00:21.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 -- 172.21.15.7:0/4241361122 shutdown_connections 2023-03-03T10:00:21.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b000682f0 0x7f4b0006a7a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:21.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b1c152b60 0x7f4b1c0bae50 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:21.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b1c073c30 0x7f4b1c0bddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:21.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 --2- 172.21.15.7:0/4241361122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b1c0ac220 0x7f4b1c0c1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:21.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 -- 172.21.15.7:0/4241361122 >> 172.21.15.7:0/4241361122 conn(0x7f4b1c0a94d0 msgr2=0x7f4b1c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:00:21.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 -- 172.21.15.7:0/4241361122 shutdown_connections 2023-03-03T10:00:21.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:21.375+0000 7f4b242fe700 1 -- 172.21.15.7:0/4241361122 wait complete. 2023-03-03T10:00:21.378 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:00:21.389 INFO:tasks.mds_thrash.fs.[a]:waiting for 119 secs before thrashing 2023-03-03T10:00:23.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:23.832 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:41.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:00:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.678+0000 7fd13a2a8700 1 Processor -- start 2023-03-03T10:00:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.678+0000 7fd13a2a8700 1 -- start start 2023-03-03T10:00:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.678+0000 7fd13a2a8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd13406c480 0x7fd1341581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.678+0000 7fd13a2a8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13406d430 con 0x7fd134151d40 2023-03-03T10:00:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.678+0000 7fd13a2a8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1341586e0 con 0x7fd1340c2930 2023-03-03T10:00:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.678+0000 7fd13a2a8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd13406c2a0 con 0x7fd13406c480 2023-03-03T10:00:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.679+0000 7fd138845700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd13406c480 0x7fd1341581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.679+0000 7fd133fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd134151d40 0x7fd1340736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52688/0 (socket says 172.21.15.7:52688) 2023-03-03T10:00:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.679+0000 7fd133fff700 1 -- 172.21.15.7:0/2732230002 learned_addr learned my addr 172.21.15.7:0/2732230002 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:00:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.679+0000 7fd138845700 1 -- 172.21.15.7:0/2732230002 >> v1:172.21.15.143:6789/0 conn(0x7fd1340c2930 legacy=0x7fd134157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.679+0000 7fd138845700 1 -- 172.21.15.7:0/2732230002 >> v1:172.21.15.7:6789/0 conn(0x7fd134151d40 legacy=0x7fd1340736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.679+0000 7fd138845700 1 -- 172.21.15.7:0/2732230002 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13406cb00 con 0x7fd13406c480 2023-03-03T10:00:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.679+0000 7fd138845700 1 --2- 172.21.15.7:0/2732230002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 0x7fd1341581a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009dc0 tx=0x7fd12c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ad51ecb506afde server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.679+0000 7fd132ffd700 1 -- 172.21.15.7:0/2732230002 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c00e070 con 0x7fd13406c480 2023-03-03T10:00:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.679+0000 7fd132ffd700 1 -- 172.21.15.7:0/2732230002 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12c00bb90 con 0x7fd13406c480 2023-03-03T10:00:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.679+0000 7fd132ffd700 1 -- 172.21.15.7:0/2732230002 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12c012430 con 0x7fd13406c480 2023-03-03T10:00:41.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.680+0000 7fd13a2a8700 1 -- 172.21.15.7:0/2732230002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 msgr2=0x7fd1341581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:41.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.680+0000 7fd13a2a8700 1 --2- 172.21.15.7:0/2732230002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 0x7fd1341581a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009dc0 tx=0x7fd12c00b330 comp rx=0 tx=0).stop 2023-03-03T10:00:41.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.680+0000 7fd13a2a8700 1 -- 172.21.15.7:0/2732230002 shutdown_connections 2023-03-03T10:00:41.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.680+0000 7fd13a2a8700 1 --2- 172.21.15.7:0/2732230002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 0x7fd1341581a0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.680+0000 7fd13a2a8700 1 -- 172.21.15.7:0/2732230002 >> 172.21.15.7:0/2732230002 conn(0x7fd1340a94d0 msgr2=0x7fd1340a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:00:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.680+0000 7fd13a2a8700 1 -- 172.21.15.7:0/2732230002 shutdown_connections 2023-03-03T10:00:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.680+0000 7fd13a2a8700 1 -- 172.21.15.7:0/2732230002 wait complete. 2023-03-03T10:00:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.681+0000 7fd13a2a8700 1 Processor -- start 2023-03-03T10:00:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.682+0000 7fd13a2a8700 1 -- start start 2023-03-03T10:00:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.682+0000 7fd13a2a8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 0x7fd1340bee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.682+0000 7fd13a2a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1340c2930 0x7fd1340c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd13a2a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd134151d40 0x7fd1340bf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd13a2a8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd134153220 con 0x7fd13406c480 2023-03-03T10:00:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd13a2a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1340be9f0 con 0x7fd134151d40 2023-03-03T10:00:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd13a2a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1340be430 con 0x7fd1340c2930 2023-03-03T10:00:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd133fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 0x7fd1340bee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd133fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 0x7fd1340bee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58714/0 (socket says 172.21.15.7:58714) 2023-03-03T10:00:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd133fff700 1 -- 172.21.15.7:0/1466535974 learned_addr learned my addr 172.21.15.7:0/1466535974 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:00:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd1337fe700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1340c2930 0x7fd1340c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd138845700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd134151d40 0x7fd1340bf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd133fff700 1 -- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1340c2930 msgr2=0x7fd1340c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd133fff700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1340c2930 0x7fd1340c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd133fff700 1 -- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd134151d40 msgr2=0x7fd1340bf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd133fff700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd134151d40 0x7fd1340bf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd133fff700 1 -- 172.21.15.7:0/1466535974 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c009a70 con 0x7fd13406c480 2023-03-03T10:00:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd133fff700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 0x7fd1340bee80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd12400c520 tx=0x7fd12400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd130ff9700 1 -- 172.21.15.7:0/1466535974 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd124005d70 con 0x7fd13406c480 2023-03-03T10:00:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd130ff9700 1 -- 172.21.15.7:0/1466535974 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1240070d0 con 0x7fd13406c480 2023-03-03T10:00:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd13a2a8700 1 -- 172.21.15.7:0/1466535974 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13406d0c0 con 0x7fd13406c480 2023-03-03T10:00:41.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd130ff9700 1 -- 172.21.15.7:0/1466535974 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd124007440 con 0x7fd13406c480 2023-03-03T10:00:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.683+0000 7fd13a2a8700 1 -- 172.21.15.7:0/1466535974 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13406cde0 con 0x7fd13406c480 2023-03-03T10:00:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.684+0000 7fd13a2a8700 1 -- 172.21.15.7:0/1466535974 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd134073e00 con 0x7fd13406c480 2023-03-03T10:00:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.685+0000 7fd130ff9700 1 -- 172.21.15.7:0/1466535974 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd124019030 con 0x7fd13406c480 2023-03-03T10:00:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.685+0000 7fd130ff9700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1200682f0 0x7fd12006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:00:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.686+0000 7fd130ff9700 1 -- 172.21.15.7:0/1466535974 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd1240853e0 con 0x7fd13406c480 2023-03-03T10:00:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.686+0000 7fd1337fe700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1200682f0 0x7fd12006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:00:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.686+0000 7fd1337fe700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1200682f0 0x7fd12006a7a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd134148f80 tx=0x7fd128009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:00:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.690+0000 7fd130ff9700 1 -- 172.21.15.7:0/1466535974 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd12404b5e0 con 0x7fd13406c480 2023-03-03T10:00:41.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.926+0000 7fd13a2a8700 1 -- 172.21.15.7:0/1466535974 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd134074bb0 con 0x7fd13406c480 2023-03-03T10:00:41.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.927+0000 7fd130ff9700 1 -- 172.21.15.7:0/1466535974 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd124051830 con 0x7fd13406c480 2023-03-03T10:00:41.928 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:00:41.929 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:00:41.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 -- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1200682f0 msgr2=0x7fd12006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:41.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1200682f0 0x7fd12006a7a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd134148f80 tx=0x7fd128009040 comp rx=0 tx=0).stop 2023-03-03T10:00:41.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 -- 172.21.15.7:0/1466535974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 msgr2=0x7fd1340bee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:00:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 0x7fd1340bee80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd12400c520 tx=0x7fd12400c8e0 comp rx=0 tx=0).stop 2023-03-03T10:00:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 -- 172.21.15.7:0/1466535974 shutdown_connections 2023-03-03T10:00:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd134151d40 0x7fd1340bf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1200682f0 0x7fd12006a7a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd13406c480 0x7fd1340bee80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 --2- 172.21.15.7:0/1466535974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1340c2930 0x7fd1340c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:00:41.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 -- 172.21.15.7:0/1466535974 >> 172.21.15.7:0/1466535974 conn(0x7fd1340a94d0 msgr2=0x7fd1340c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:00:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 -- 172.21.15.7:0/1466535974 shutdown_connections 2023-03-03T10:00:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:00:41.929+0000 7fd13a2a8700 1 -- 172.21.15.7:0/1466535974 wait complete. 2023-03-03T10:00:41.932 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:00:41.942 INFO:tasks.mds_thrash.fs.[b]:waiting for 111 secs before thrashing 2023-03-03T10:00:53.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:00:53.904 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:23.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:23.968 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:54.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:01:54.041 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:20.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:02:20.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.528+0000 7ff69287b700 1 Processor -- start 2023-03-03T10:02:20.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.528+0000 7ff69287b700 1 -- start start 2023-03-03T10:02:20.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.529+0000 7ff69287b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff68c06a2b0 0x7ff68c06a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:20.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.529+0000 7ff69287b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff68c158030 0x7ff68c158470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:20.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.529+0000 7ff69287b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68c0c1c10 con 0x7ff68c151c60 2023-03-03T10:02:20.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.529+0000 7ff69287b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68c0c26b0 con 0x7ff68c158030 2023-03-03T10:02:20.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.529+0000 7ff69287b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68c1555c0 con 0x7ff68c06a2b0 2023-03-03T10:02:20.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.529+0000 7ff690e18700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff68c158030 0x7ff68c158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:20.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.529+0000 7ff690e18700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff68c158030 0x7ff68c158470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58776/0 (socket says 172.21.15.7:58776) 2023-03-03T10:02:20.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.529+0000 7ff690e18700 1 -- 172.21.15.7:0/1130150729 learned_addr learned my addr 172.21.15.7:0/1130150729 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:02:20.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.530+0000 7ff68b7fe700 1 --2- 172.21.15.7:0/1130150729 >> v2:172.21.15.143:3300/0 conn(0x7ff68c06a2b0 0x7ff68c06a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:20.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.530+0000 7ff690e18700 1 -- 172.21.15.7:0/1130150729 >> v2:172.21.15.143:3300/0 conn(0x7ff68c06a2b0 msgr2=0x7ff68c06a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:02:20.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.530+0000 7ff690e18700 1 --2- 172.21.15.7:0/1130150729 >> v2:172.21.15.143:3300/0 conn(0x7ff68c06a2b0 0x7ff68c06a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:20.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.530+0000 7ff690e18700 1 -- 172.21.15.7:0/1130150729 >> v1:172.21.15.143:6789/0 conn(0x7ff68c151c60 legacy=0x7ff68c0c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:02:20.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.530+0000 7ff690e18700 1 -- 172.21.15.7:0/1130150729 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c152030 con 0x7ff68c158030 2023-03-03T10:02:20.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.530+0000 7ff690e18700 1 --2- 172.21.15.7:0/1130150729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c158030 0x7ff68c158470 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0a6730 tx=0x7ff68400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5a7addf509eb8a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:02:20.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.530+0000 7ff68affd700 1 -- 172.21.15.7:0/1130150729 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68400e070 con 0x7ff68c158030 2023-03-03T10:02:20.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.530+0000 7ff68affd700 1 -- 172.21.15.7:0/1130150729 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff684003a20 con 0x7ff68c158030 2023-03-03T10:02:20.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.530+0000 7ff68affd700 1 -- 172.21.15.7:0/1130150729 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684003dd0 con 0x7ff68c158030 2023-03-03T10:02:20.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.531+0000 7ff69287b700 1 -- 172.21.15.7:0/1130150729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c158030 msgr2=0x7ff68c158470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:20.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.531+0000 7ff69287b700 1 --2- 172.21.15.7:0/1130150729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c158030 0x7ff68c158470 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0a6730 tx=0x7ff68400b330 comp rx=0 tx=0).stop 2023-03-03T10:02:20.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.531+0000 7ff69287b700 1 -- 172.21.15.7:0/1130150729 shutdown_connections 2023-03-03T10:02:20.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.531+0000 7ff69287b700 1 --2- 172.21.15.7:0/1130150729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c158030 0x7ff68c158470 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:20.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.531+0000 7ff69287b700 1 --2- 172.21.15.7:0/1130150729 >> v2:172.21.15.143:3300/0 conn(0x7ff68c06a2b0 0x7ff68c06a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:20.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.531+0000 7ff69287b700 1 -- 172.21.15.7:0/1130150729 >> 172.21.15.7:0/1130150729 conn(0x7ff68c0a9430 msgr2=0x7ff68c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:02:20.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.531+0000 7ff69287b700 1 -- 172.21.15.7:0/1130150729 shutdown_connections 2023-03-03T10:02:20.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.531+0000 7ff69287b700 1 -- 172.21.15.7:0/1130150729 wait complete. 2023-03-03T10:02:20.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.532+0000 7ff69287b700 1 Processor -- start 2023-03-03T10:02:20.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.532+0000 7ff69287b700 1 -- start start 2023-03-03T10:02:20.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff69287b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68c06a2b0 0x7ff68c06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:20.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff69287b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c151c60 0x7ff68c1e9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:20.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff69287b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff68c158030 0x7ff68c156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:20.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff69287b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c1ed180 con 0x7ff68c151c60 2023-03-03T10:02:20.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff69287b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c1ece30 con 0x7ff68c158030 2023-03-03T10:02:20.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff69287b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c1ea140 con 0x7ff68c06a2b0 2023-03-03T10:02:20.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c151c60 0x7ff68c1e9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:20.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c151c60 0x7ff68c1e9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58780/0 (socket says 172.21.15.7:58780) 2023-03-03T10:02:20.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68b7fe700 1 -- 172.21.15.7:0/293179139 learned_addr learned my addr 172.21.15.7:0/293179139 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:02:20.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff690e18700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff68c158030 0x7ff68c156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:20.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68bfff700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68c06a2b0 0x7ff68c06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:20.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68b7fe700 1 -- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68c06a2b0 msgr2=0x7ff68c06ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:20.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68b7fe700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68c06a2b0 0x7ff68c06ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:20.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68b7fe700 1 -- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff68c158030 msgr2=0x7ff68c156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:20.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68b7fe700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff68c158030 0x7ff68c156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:20.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68b7fe700 1 -- 172.21.15.7:0/293179139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff684009b30 con 0x7ff68c151c60 2023-03-03T10:02:20.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68bfff700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68c06a2b0 0x7ff68c06ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:02:20.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.533+0000 7ff68b7fe700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c151c60 0x7ff68c1e9880 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff67400c7f0 tx=0x7ff67400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:02:20.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.534+0000 7ff688ff9700 1 -- 172.21.15.7:0/293179139 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674011800 con 0x7ff68c151c60 2023-03-03T10:02:20.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.534+0000 7ff688ff9700 1 -- 172.21.15.7:0/293179139 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff674011960 con 0x7ff68c151c60 2023-03-03T10:02:20.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.534+0000 7ff69287b700 1 -- 172.21.15.7:0/293179139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68c0c4a80 con 0x7ff68c151c60 2023-03-03T10:02:20.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.534+0000 7ff688ff9700 1 -- 172.21.15.7:0/293179139 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674011cf0 con 0x7ff68c151c60 2023-03-03T10:02:20.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.534+0000 7ff69287b700 1 -- 172.21.15.7:0/293179139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff68c157b30 con 0x7ff68c151c60 2023-03-03T10:02:20.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.535+0000 7ff69287b700 1 -- 172.21.15.7:0/293179139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68c0c4d60 con 0x7ff68c151c60 2023-03-03T10:02:20.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.535+0000 7ff688ff9700 1 -- 172.21.15.7:0/293179139 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff674021070 con 0x7ff68c151c60 2023-03-03T10:02:20.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.536+0000 7ff688ff9700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff66c068320 0x7ff66c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:20.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.536+0000 7ff688ff9700 1 -- 172.21.15.7:0/293179139 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff674085fc0 con 0x7ff68c151c60 2023-03-03T10:02:20.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.536+0000 7ff68bfff700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff66c068320 0x7ff66c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:20.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.536+0000 7ff68bfff700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff66c068320 0x7ff66c06a7d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff67c003f40 tx=0x7ff67c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:02:20.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.540+0000 7ff688ff9700 1 -- 172.21.15.7:0/293179139 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff674052270 con 0x7ff68c151c60 2023-03-03T10:02:20.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.776+0000 7ff69287b700 1 -- 172.21.15.7:0/293179139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff68c074bb0 con 0x7ff68c151c60 2023-03-03T10:02:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.777+0000 7ff688ff9700 1 -- 172.21.15.7:0/293179139 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff67401f020 con 0x7ff68c151c60 2023-03-03T10:02:20.778 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:02:20.779 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:02:20.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 -- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff66c068320 msgr2=0x7ff66c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff66c068320 0x7ff66c06a7d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff67c003f40 tx=0x7ff67c009040 comp rx=0 tx=0).stop 2023-03-03T10:02:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 -- 172.21.15.7:0/293179139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c151c60 msgr2=0x7ff68c1e9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c151c60 0x7ff68c1e9880 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ff67400c7f0 tx=0x7ff67400cbb0 comp rx=0 tx=0).stop 2023-03-03T10:02:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 -- 172.21.15.7:0/293179139 shutdown_connections 2023-03-03T10:02:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff68c158030 0x7ff68c156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68c06a2b0 0x7ff68c06ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff66c068320 0x7ff66c06a7d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 --2- 172.21.15.7:0/293179139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff68c151c60 0x7ff68c1e9880 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 -- 172.21.15.7:0/293179139 >> 172.21.15.7:0/293179139 conn(0x7ff68c0a9430 msgr2=0x7ff68c0c74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:02:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 -- 172.21.15.7:0/293179139 shutdown_connections 2023-03-03T10:02:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:20.779+0000 7ff69287b700 1 -- 172.21.15.7:0/293179139 wait complete. 2023-03-03T10:02:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:02:20.792 INFO:tasks.mds_thrash.fs.[a]:waiting for 111 secs before thrashing 2023-03-03T10:02:24.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:24.111 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:32.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:02:33.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.081+0000 7f943add1700 1 Processor -- start 2023-03-03T10:02:33.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.081+0000 7f943add1700 1 -- start start 2023-03-03T10:02:33.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f943add1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f943406d870 0x7f94341550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:33.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f943add1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f94340c3e30 0x7f94340c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:33.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f943add1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94340c7a60 con 0x7f94340c1da0 2023-03-03T10:02:33.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f943add1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94340c2af0 con 0x7f943406d870 2023-03-03T10:02:33.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f943add1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94340c3620 con 0x7f94340c3e30 2023-03-03T10:02:33.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f9438b6d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f943406d870 0x7f94341550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:33.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f9433fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f94340c1da0 0x7f94340c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52780/0 (socket says 172.21.15.7:52780) 2023-03-03T10:02:33.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f9433fff700 1 -- 172.21.15.7:0/52638508 learned_addr learned my addr 172.21.15.7:0/52638508 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:02:33.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f943936e700 1 --2- 172.21.15.7:0/52638508 >> v2:172.21.15.143:3300/0 conn(0x7f94340c3e30 0x7f94340c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f9438b6d700 1 -- 172.21.15.7:0/52638508 >> v2:172.21.15.143:3300/0 conn(0x7f94340c3e30 msgr2=0x7f94340c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:02:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f9438b6d700 1 --2- 172.21.15.7:0/52638508 >> v2:172.21.15.143:3300/0 conn(0x7f94340c3e30 0x7f94340c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f9438b6d700 1 -- 172.21.15.7:0/52638508 >> v1:172.21.15.7:6789/0 conn(0x7f94340c1da0 legacy=0x7f94340c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f9438b6d700 1 -- 172.21.15.7:0/52638508 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94341545e0 con 0x7f943406d870 2023-03-03T10:02:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f943936e700 1 --2- 172.21.15.7:0/52638508 >> v2:172.21.15.143:3300/0 conn(0x7f94340c3e30 0x7f94340c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:02:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f94337fe700 1 -- 172.21.15.7:0/52638508 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953383361 0 0) 0x7f94340c7a60 con 0x7f94340c1da0 2023-03-03T10:02:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.082+0000 7f9438b6d700 1 --2- 172.21.15.7:0/52638508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f943406d870 0x7f94341550c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9434073830 tx=0x7f9428009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aee8efffc230556 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:02:33.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f94337fe700 1 -- 172.21.15.7:0/52638508 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9428011830 con 0x7f943406d870 2023-03-03T10:02:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f94337fe700 1 -- 172.21.15.7:0/52638508 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9428011990 con 0x7f943406d870 2023-03-03T10:02:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f94337fe700 1 -- 172.21.15.7:0/52638508 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9428011d20 con 0x7f943406d870 2023-03-03T10:02:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f943add1700 1 -- 172.21.15.7:0/52638508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f943406d870 msgr2=0x7f94341550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f943add1700 1 --2- 172.21.15.7:0/52638508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f943406d870 0x7f94341550c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9434073830 tx=0x7f9428009b20 comp rx=0 tx=0).stop 2023-03-03T10:02:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f943add1700 1 -- 172.21.15.7:0/52638508 shutdown_connections 2023-03-03T10:02:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f943add1700 1 --2- 172.21.15.7:0/52638508 >> v2:172.21.15.143:3300/0 conn(0x7f94340c3e30 0x7f94340c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:33.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f943add1700 1 --2- 172.21.15.7:0/52638508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f943406d870 0x7f94341550c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f943add1700 1 -- 172.21.15.7:0/52638508 >> 172.21.15.7:0/52638508 conn(0x7f943406c110 msgr2=0x7f943406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:02:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f943add1700 1 -- 172.21.15.7:0/52638508 shutdown_connections 2023-03-03T10:02:33.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.083+0000 7f943add1700 1 -- 172.21.15.7:0/52638508 wait complete. 2023-03-03T10:02:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.084+0000 7f943add1700 1 Processor -- start 2023-03-03T10:02:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.084+0000 7f943add1700 1 -- start start 2023-03-03T10:02:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.085+0000 7f943add1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f943406d870 0x7f94341ed860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.085+0000 7f943add1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94340c1da0 0x7f94340c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.085+0000 7f943add1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94340c3e30 0x7f9434142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.085+0000 7f943add1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94341510b0 con 0x7f94340c3e30 2023-03-03T10:02:33.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.085+0000 7f943add1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94341508a0 con 0x7f943406d870 2023-03-03T10:02:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.085+0000 7f943add1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9434151930 con 0x7f94340c1da0 2023-03-03T10:02:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.085+0000 7f943936e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94340c3e30 0x7f9434142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.085+0000 7f943936e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94340c3e30 0x7f9434142800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58808/0 (socket says 172.21.15.7:58808) 2023-03-03T10:02:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.085+0000 7f943936e700 1 -- 172.21.15.7:0/89336429 learned_addr learned my addr 172.21.15.7:0/89336429 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:02:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f9438b6d700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f943406d870 0x7f94341ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f9433fff700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94340c1da0 0x7f94340c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:33.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f943936e700 1 -- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94340c1da0 msgr2=0x7f94340c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:33.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f943936e700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94340c1da0 0x7f94340c73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:33.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f943936e700 1 -- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f943406d870 msgr2=0x7f94341ed860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:33.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f943936e700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f943406d870 0x7f94341ed860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:33.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f943936e700 1 -- 172.21.15.7:0/89336429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94280094a0 con 0x7f94340c3e30 2023-03-03T10:02:33.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f9433fff700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94340c1da0 0x7f94340c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:02:33.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f9438b6d700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f943406d870 0x7f94341ed860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:02:33.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f943936e700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94340c3e30 0x7f9434142800 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f942c007df0 tx=0x7f942c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:02:33.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f94317fa700 1 -- 172.21.15.7:0/89336429 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c00ddd0 con 0x7f94340c3e30 2023-03-03T10:02:33.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f94317fa700 1 -- 172.21.15.7:0/89336429 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c00f040 con 0x7f94340c3e30 2023-03-03T10:02:33.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f943add1700 1 -- 172.21.15.7:0/89336429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9434154ba0 con 0x7f94340c3e30 2023-03-03T10:02:33.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f94317fa700 1 -- 172.21.15.7:0/89336429 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c003c60 con 0x7f94340c3e30 2023-03-03T10:02:33.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.086+0000 7f943add1700 1 -- 172.21.15.7:0/89336429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94341548c0 con 0x7f94340c3e30 2023-03-03T10:02:33.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.087+0000 7f943add1700 1 -- 172.21.15.7:0/89336429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943406dc40 con 0x7f94340c3e30 2023-03-03T10:02:33.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.087+0000 7f94317fa700 1 -- 172.21.15.7:0/89336429 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f942c01c030 con 0x7f94340c3e30 2023-03-03T10:02:33.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.088+0000 7f94317fa700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94180682f0 0x7f941806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:33.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.088+0000 7f94317fa700 1 -- 172.21.15.7:0/89336429 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f942c086330 con 0x7f94340c3e30 2023-03-03T10:02:33.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.088+0000 7f9438b6d700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94180682f0 0x7f941806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:33.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.089+0000 7f9438b6d700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94180682f0 0x7f941806a7a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f94280090b0 tx=0x7f9428008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:02:33.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.092+0000 7f94317fa700 1 -- 172.21.15.7:0/89336429 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f942c0525e0 con 0x7f94340c3e30 2023-03-03T10:02:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.328+0000 7f943add1700 1 -- 172.21.15.7:0/89336429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9434074bb0 con 0x7f94340c3e30 2023-03-03T10:02:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.328+0000 7f94317fa700 1 -- 172.21.15.7:0/89336429 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f942c055c00 con 0x7f94340c3e30 2023-03-03T10:02:33.330 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:02:33.330 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:02:33.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 -- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94180682f0 msgr2=0x7f941806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:33.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94180682f0 0x7f941806a7a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f94280090b0 tx=0x7f9428008030 comp rx=0 tx=0).stop 2023-03-03T10:02:33.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 -- 172.21.15.7:0/89336429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94340c3e30 msgr2=0x7f9434142800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94340c3e30 0x7f9434142800 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f942c007df0 tx=0x7f942c00d3b0 comp rx=0 tx=0).stop 2023-03-03T10:02:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 -- 172.21.15.7:0/89336429 shutdown_connections 2023-03-03T10:02:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94180682f0 0x7f941806a7a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94340c3e30 0x7f9434142800 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f943406d870 0x7f94341ed860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 --2- 172.21.15.7:0/89336429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94340c1da0 0x7f94340c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 -- 172.21.15.7:0/89336429 >> 172.21.15.7:0/89336429 conn(0x7f943406c110 msgr2=0x7f94340708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:02:33.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 -- 172.21.15.7:0/89336429 shutdown_connections 2023-03-03T10:02:33.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:33.331+0000 7f943add1700 1 -- 172.21.15.7:0/89336429 wait complete. 2023-03-03T10:02:33.334 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:02:33.344 INFO:tasks.mds_thrash.fs.[b]:waiting for 14 secs before thrashing 2023-03-03T10:02:47.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:02:47.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.483+0000 7efd802d3700 1 Processor -- start 2023-03-03T10:02:47.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.483+0000 7efd802d3700 1 -- start start 2023-03-03T10:02:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd802d3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efd7814fda0 0x7efd78156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd802d3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd780c2f30 con 0x7efd7814f9d0 2023-03-03T10:02:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd802d3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd780c31d0 con 0x7efd780abe20 2023-03-03T10:02:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd802d3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd781524a0 con 0x7efd7814fda0 2023-03-03T10:02:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7e06f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7efd7814f9d0 0x7efd78071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52798/0 (socket says 172.21.15.7:52798) 2023-03-03T10:02:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7e06f700 1 -- 172.21.15.7:0/591839007 learned_addr learned my addr 172.21.15.7:0/591839007 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:02:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016237626 0 0) 0x7efd780c2f30 con 0x7efd7814f9d0 2023-03-03T10:02:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd640035b0 con 0x7efd7814f9d0 2023-03-03T10:02:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7d86e700 1 --2- 172.21.15.7:0/591839007 >> v2:172.21.15.143:3300/0 conn(0x7efd7814fda0 0x7efd78156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1864163458 0 0) 0x7efd640035b0 con 0x7efd7814f9d0 2023-03-03T10:02:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 >> v2:172.21.15.143:3300/0 conn(0x7efd7814fda0 msgr2=0x7efd78156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:02:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7d06d700 1 --2- 172.21.15.7:0/591839007 >> v2:172.21.15.143:3300/0 conn(0x7efd7814fda0 0x7efd78156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:47.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 >> v1:172.21.15.143:6789/0 conn(0x7efd780abe20 legacy=0x7efd781529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:47.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780c34b0 con 0x7efd7814f9d0 2023-03-03T10:02:47.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7efd74001670 con 0x7efd7814f9d0 2023-03-03T10:02:47.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.484+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 >> v1:172.21.15.7:6789/0 conn(0x7efd7814f9d0 legacy=0x7efd78071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:47.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7d06d700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd64003a70 0x7efd64003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:47.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7d06d700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd64004850 0x7efd64004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:47.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7d06d700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd64005320 0x7efd6400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:47.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd781524a0 con 0x7efd64003a70 2023-03-03T10:02:47.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6400c230 con 0x7efd64005320 2023-03-03T10:02:47.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6400c3f0 con 0x7efd64004850 2023-03-03T10:02:47.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7e06f700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd64003a70 0x7efd64003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:47.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7e870700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd64005320 0x7efd6400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:47.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7d86e700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd64004850 0x7efd64004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:47.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7e06f700 1 -- 172.21.15.7:0/591839007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd64004850 msgr2=0x7efd64004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:02:47.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7e06f700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd64004850 0x7efd64004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:47.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7e06f700 1 -- 172.21.15.7:0/591839007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd64005320 msgr2=0x7efd6400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:02:47.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7e06f700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd64005320 0x7efd6400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:47.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7e06f700 1 -- 172.21.15.7:0/591839007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd640035b0 con 0x7efd64003a70 2023-03-03T10:02:47.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7e870700 1 -- 172.21.15.7:0/591839007 reap_dead start 2023-03-03T10:02:47.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7e06f700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd64003a70 0x7efd64003e80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7efd7400c240 tx=0x7efd64005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc261599204a5fa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:02:47.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74007c10 con 0x7efd64003a70 2023-03-03T10:02:47.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7400cde0 con 0x7efd64003a70 2023-03-03T10:02:47.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.485+0000 7efd7d06d700 1 -- 172.21.15.7:0/591839007 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74005b80 con 0x7efd64003a70 2023-03-03T10:02:47.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.486+0000 7efd802d3700 1 -- 172.21.15.7:0/591839007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd64003a70 msgr2=0x7efd64003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:47.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.486+0000 7efd802d3700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd64003a70 0x7efd64003e80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7efd7400c240 tx=0x7efd64005880 comp rx=0 tx=0).stop 2023-03-03T10:02:47.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.486+0000 7efd802d3700 1 -- 172.21.15.7:0/591839007 shutdown_connections 2023-03-03T10:02:47.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.486+0000 7efd802d3700 1 --2- 172.21.15.7:0/591839007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd64003a70 0x7efd64003e80 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:47.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.486+0000 7efd802d3700 1 -- 172.21.15.7:0/591839007 >> 172.21.15.7:0/591839007 conn(0x7efd780c8900 msgr2=0x7efd780c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:02:47.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.486+0000 7efd802d3700 1 -- 172.21.15.7:0/591839007 shutdown_connections 2023-03-03T10:02:47.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.486+0000 7efd802d3700 1 -- 172.21.15.7:0/591839007 wait complete. 2023-03-03T10:02:47.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.487+0000 7efd802d3700 1 Processor -- start 2023-03-03T10:02:47.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.487+0000 7efd802d3700 1 -- start start 2023-03-03T10:02:47.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd802d3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd64003a70 0x7efd7814ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:47.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd802d3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7814f1c0 0x7efd781eae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:47.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd802d3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd781eb3b0 0x7efd781ddcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:47.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd802d3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd780bdac0 con 0x7efd781eb3b0 2023-03-03T10:02:47.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd802d3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd780bd470 con 0x7efd7814f1c0 2023-03-03T10:02:47.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd802d3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd781dee00 con 0x7efd64003a70 2023-03-03T10:02:47.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd7e870700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd781eb3b0 0x7efd781ddcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:47.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd7e870700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd781eb3b0 0x7efd781ddcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58834/0 (socket says 172.21.15.7:58834) 2023-03-03T10:02:47.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd7e870700 1 -- 172.21.15.7:0/2181350119 learned_addr learned my addr 172.21.15.7:0/2181350119 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:02:47.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd7d86e700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7814f1c0 0x7efd781eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:47.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd7e06f700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd64003a70 0x7efd7814ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:47.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd7e870700 1 -- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd64003a70 msgr2=0x7efd7814ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:47.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd7e870700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd64003a70 0x7efd7814ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:47.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd7e870700 1 -- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7814f1c0 msgr2=0x7efd781eae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:47.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd7e870700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7814f1c0 0x7efd781eae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:47.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.488+0000 7efd7e870700 1 -- 172.21.15.7:0/2181350119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780ac350 con 0x7efd781eb3b0 2023-03-03T10:02:47.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.489+0000 7efd7e870700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd781eb3b0 0x7efd781ddcb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7efd64005830 tx=0x7efd7000ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:02:47.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.489+0000 7efd6effd700 1 -- 172.21.15.7:0/2181350119 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd70003cd0 con 0x7efd781eb3b0 2023-03-03T10:02:47.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.489+0000 7efd6effd700 1 -- 172.21.15.7:0/2181350119 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd70003e70 con 0x7efd781eb3b0 2023-03-03T10:02:47.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.489+0000 7efd802d3700 1 -- 172.21.15.7:0/2181350119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd780ab630 con 0x7efd781eb3b0 2023-03-03T10:02:47.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.489+0000 7efd6effd700 1 -- 172.21.15.7:0/2181350119 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd700052f0 con 0x7efd781eb3b0 2023-03-03T10:02:47.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.489+0000 7efd802d3700 1 -- 172.21.15.7:0/2181350119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd780c3e10 con 0x7efd781eb3b0 2023-03-03T10:02:47.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.490+0000 7efd802d3700 1 -- 172.21.15.7:0/2181350119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd780bdc00 con 0x7efd781eb3b0 2023-03-03T10:02:47.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.490+0000 7efd6effd700 1 -- 172.21.15.7:0/2181350119 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efd70005450 con 0x7efd781eb3b0 2023-03-03T10:02:47.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.490+0000 7efd6effd700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd5c0682f0 0x7efd5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:02:47.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.491+0000 7efd6effd700 1 -- 172.21.15.7:0/2181350119 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efd70086380 con 0x7efd781eb3b0 2023-03-03T10:02:47.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.491+0000 7efd7e06f700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd5c0682f0 0x7efd5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:02:47.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.491+0000 7efd7e06f700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd5c0682f0 0x7efd5c06a7a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7efd74000c00 tx=0x7efd74006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:02:47.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.495+0000 7efd6effd700 1 -- 172.21.15.7:0/2181350119 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efd780bdc00 con 0x7efd781eb3b0 2023-03-03T10:02:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.734+0000 7efd802d3700 1 -- 172.21.15.7:0/2181350119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd780bdc00 con 0x7efd781eb3b0 2023-03-03T10:02:47.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.734+0000 7efd6effd700 1 -- 172.21.15.7:0/2181350119 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efd700526c0 con 0x7efd781eb3b0 2023-03-03T10:02:47.736 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:02:47.736 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:02:47.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 -- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd5c0682f0 msgr2=0x7efd5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd5c0682f0 0x7efd5c06a7a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7efd74000c00 tx=0x7efd74006040 comp rx=0 tx=0).stop 2023-03-03T10:02:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 -- 172.21.15.7:0/2181350119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd781eb3b0 msgr2=0x7efd781ddcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:02:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd781eb3b0 0x7efd781ddcb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7efd64005830 tx=0x7efd7000ff00 comp rx=0 tx=0).stop 2023-03-03T10:02:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 -- 172.21.15.7:0/2181350119 shutdown_connections 2023-03-03T10:02:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd5c0682f0 0x7efd5c06a7a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd781eb3b0 0x7efd781ddcb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd64003a70 0x7efd7814ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:47.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 --2- 172.21.15.7:0/2181350119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7814f1c0 0x7efd781eae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:02:47.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 -- 172.21.15.7:0/2181350119 >> 172.21.15.7:0/2181350119 conn(0x7efd780c8900 msgr2=0x7efd781508f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:02:47.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 -- 172.21.15.7:0/2181350119 shutdown_connections 2023-03-03T10:02:47.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:02:47.736+0000 7efd802d3700 1 -- 172.21.15.7:0/2181350119 wait complete. 2023-03-03T10:02:47.739 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:02:47.749 INFO:tasks.mds_thrash.fs.[b]:waiting for 43 secs before thrashing 2023-03-03T10:02:54.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:02:54.174 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:24.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:24.289 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:30.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:03:30.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.888+0000 7fb401bf7700 1 Processor -- start 2023-03-03T10:03:30.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.888+0000 7fb401bf7700 1 -- start start 2023-03-03T10:03:30.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb401bf7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 0x7fb3fc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:30.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb401bf7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb3fc0abe20 0x7fb3fc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:30.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb401bf7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3fc0c2f30 con 0x7fb3fc14fda0 2023-03-03T10:03:30.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb401bf7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3fc0c31d0 con 0x7fb3fc0abe20 2023-03-03T10:03:30.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb401bf7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc1524a0 con 0x7fb3fc14f9d0 2023-03-03T10:03:30.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3fbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb3fc0abe20 0x7fb3fc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3faffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb3fc14fda0 0x7fb3fc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52844/0 (socket says 172.21.15.7:52844) 2023-03-03T10:03:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3faffd700 1 -- 172.21.15.7:0/2786168533 learned_addr learned my addr 172.21.15.7:0/2786168533 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:03:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3fa7fc700 1 -- 172.21.15.7:0/2786168533 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121356110 0 0) 0x7fb3fc0c2f30 con 0x7fb3fc14fda0 2023-03-03T10:03:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3fb7fe700 1 --2- 172.21.15.7:0/2786168533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 0x7fb3fc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3fa7fc700 1 -- 172.21.15.7:0/2786168533 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e80035b0 con 0x7fb3fc14fda0 2023-03-03T10:03:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3fbfff700 1 -- 172.21.15.7:0/2786168533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 msgr2=0x7fb3fc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:03:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3fbfff700 1 --2- 172.21.15.7:0/2786168533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 0x7fb3fc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3fbfff700 1 -- 172.21.15.7:0/2786168533 >> v1:172.21.15.7:6789/0 conn(0x7fb3fc14fda0 legacy=0x7fb3fc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3fbfff700 1 -- 172.21.15.7:0/2786168533 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc0c34b0 con 0x7fb3fc0abe20 2023-03-03T10:03:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3fb7fe700 1 --2- 172.21.15.7:0/2786168533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 0x7fb3fc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:03:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.889+0000 7fb3fbfff700 1 --2- 172.21.15.7:0/2786168533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc0abe20 0x7fb3fc1529c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009d90 tx=0x7fb3f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca72578607da3ee2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:03:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb3fa7fc700 1 -- 172.21.15.7:0/2786168533 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f400e070 con 0x7fb3fc0abe20 2023-03-03T10:03:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb3fa7fc700 1 -- 172.21.15.7:0/2786168533 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f400bb20 con 0x7fb3fc0abe20 2023-03-03T10:03:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb3fa7fc700 1 -- 172.21.15.7:0/2786168533 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4012430 con 0x7fb3fc0abe20 2023-03-03T10:03:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb401bf7700 1 -- 172.21.15.7:0/2786168533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc0abe20 msgr2=0x7fb3fc1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb401bf7700 1 --2- 172.21.15.7:0/2786168533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc0abe20 0x7fb3fc1529c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009d90 tx=0x7fb3f400b330 comp rx=0 tx=0).stop 2023-03-03T10:03:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb401bf7700 1 -- 172.21.15.7:0/2786168533 shutdown_connections 2023-03-03T10:03:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb401bf7700 1 --2- 172.21.15.7:0/2786168533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc0abe20 0x7fb3fc1529c0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb401bf7700 1 --2- 172.21.15.7:0/2786168533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 0x7fb3fc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb401bf7700 1 -- 172.21.15.7:0/2786168533 >> 172.21.15.7:0/2786168533 conn(0x7fb3fc0c8900 msgr2=0x7fb3fc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:03:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb401bf7700 1 -- 172.21.15.7:0/2786168533 shutdown_connections 2023-03-03T10:03:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.890+0000 7fb401bf7700 1 -- 172.21.15.7:0/2786168533 wait complete. 2023-03-03T10:03:30.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.891+0000 7fb401bf7700 1 Processor -- start 2023-03-03T10:03:30.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.892+0000 7fb401bf7700 1 -- start start 2023-03-03T10:03:30.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.892+0000 7fb401bf7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc0abe20 0x7fb3fc155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:30.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.892+0000 7fb401bf7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 0x7fb3fc0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:30.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.892+0000 7fb401bf7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc14fda0 0x7fb3fc14a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:30.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.892+0000 7fb401bf7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc0bdb90 con 0x7fb3fc14fda0 2023-03-03T10:03:30.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.892+0000 7fb401bf7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc0c2350 con 0x7fb3fc0abe20 2023-03-03T10:03:30.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.892+0000 7fb401bf7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc1e7010 con 0x7fb3fc14f9d0 2023-03-03T10:03:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.892+0000 7fb3fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc14fda0 0x7fb3fc14a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.892+0000 7fb3fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc14fda0 0x7fb3fc14a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58874/0 (socket says 172.21.15.7:58874) 2023-03-03T10:03:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.892+0000 7fb3fbfff700 1 -- 172.21.15.7:0/4294675681 learned_addr learned my addr 172.21.15.7:0/4294675681 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:03:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3fb7fe700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc0abe20 0x7fb3fc155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3faffd700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 0x7fb3fc0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3fbfff700 1 -- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 msgr2=0x7fb3fc0c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3fbfff700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 0x7fb3fc0c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3fbfff700 1 -- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc0abe20 msgr2=0x7fb3fc155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3fbfff700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc0abe20 0x7fb3fc155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3fbfff700 1 -- 172.21.15.7:0/4294675681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4009a70 con 0x7fb3fc14fda0 2023-03-03T10:03:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3fb7fe700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc0abe20 0x7fb3fc155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:03:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3fbfff700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc14fda0 0x7fb3fc14a980 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4014040 tx=0x7fb3f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:03:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3f3fff700 1 -- 172.21.15.7:0/4294675681 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4012430 con 0x7fb3fc14fda0 2023-03-03T10:03:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3f3fff700 1 -- 172.21.15.7:0/4294675681 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f4020070 con 0x7fb3fc14fda0 2023-03-03T10:03:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb401bf7700 1 -- 172.21.15.7:0/4294675681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc0ac350 con 0x7fb3fc14fda0 2023-03-03T10:03:30.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb3f3fff700 1 -- 172.21.15.7:0/4294675681 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4004280 con 0x7fb3fc14fda0 2023-03-03T10:03:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.893+0000 7fb401bf7700 1 -- 172.21.15.7:0/4294675681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc0ab630 con 0x7fb3fc14fda0 2023-03-03T10:03:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.894+0000 7fb401bf7700 1 -- 172.21.15.7:0/4294675681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc071fa0 con 0x7fb3fc14fda0 2023-03-03T10:03:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.895+0000 7fb3f3fff700 1 -- 172.21.15.7:0/4294675681 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb3f40043e0 con 0x7fb3fc14fda0 2023-03-03T10:03:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.895+0000 7fb3f3fff700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3dc0682f0 0x7fb3dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:03:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.895+0000 7fb3f3fff700 1 -- 172.21.15.7:0/4294675681 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb3f4022070 con 0x7fb3fc14fda0 2023-03-03T10:03:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.895+0000 7fb3fb7fe700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3dc0682f0 0x7fb3dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:03:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.896+0000 7fb3fb7fe700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3dc0682f0 0x7fb3dc06a7a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec006fd0 tx=0x7fb3ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:03:30.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:30.899+0000 7fb3f3fff700 1 -- 172.21.15.7:0/4294675681 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb3f4054620 con 0x7fb3fc14fda0 2023-03-03T10:03:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.141+0000 7fb401bf7700 1 -- 172.21.15.7:0/4294675681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3fc074bb0 con 0x7fb3fc14fda0 2023-03-03T10:03:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.142+0000 7fb3f3fff700 1 -- 172.21.15.7:0/4294675681 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb3f4057c40 con 0x7fb3fc14fda0 2023-03-03T10:03:31.144 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:03:31.144 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:03:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 -- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3dc0682f0 msgr2=0x7fb3dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3dc0682f0 0x7fb3dc06a7a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec006fd0 tx=0x7fb3ec008040 comp rx=0 tx=0).stop 2023-03-03T10:03:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 -- 172.21.15.7:0/4294675681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc14fda0 msgr2=0x7fb3fc14a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:03:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc14fda0 0x7fb3fc14a980 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4014040 tx=0x7fb3f4004ad0 comp rx=0 tx=0).stop 2023-03-03T10:03:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 -- 172.21.15.7:0/4294675681 shutdown_connections 2023-03-03T10:03:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3dc0682f0 0x7fb3dc06a7a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3fc14fda0 0x7fb3fc14a980 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3fc0abe20 0x7fb3fc155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 --2- 172.21.15.7:0/4294675681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3fc14f9d0 0x7fb3fc0c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:03:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 -- 172.21.15.7:0/4294675681 >> 172.21.15.7:0/4294675681 conn(0x7fb3fc0c8900 msgr2=0x7fb3fc14efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:03:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 -- 172.21.15.7:0/4294675681 shutdown_connections 2023-03-03T10:03:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:03:31.144+0000 7fb401bf7700 1 -- 172.21.15.7:0/4294675681 wait complete. 2023-03-03T10:03:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:03:31.157 INFO:tasks.mds_thrash.fs.[b]:waiting for 50 secs before thrashing 2023-03-03T10:03:54.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:03:54.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:11.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:04:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.924+0000 7f9b1a8fa700 1 Processor -- start 2023-03-03T10:04:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.924+0000 7f9b1a8fa700 1 -- start start 2023-03-03T10:04:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.925+0000 7f9b1a8fa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9b140caa30 0x7f9b140c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.925+0000 7f9b1a8fa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b14154ff0 0x7f9b140cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.925+0000 7f9b1a8fa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b140c7310 con 0x7f9b140c00e0 2023-03-03T10:04:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.925+0000 7f9b1a8fa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b14155430 con 0x7f9b14154ff0 2023-03-03T10:04:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.925+0000 7f9b1a8fa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b140bfd10 con 0x7f9b140caa30 2023-03-03T10:04:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.925+0000 7f9b18e97700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b14154ff0 0x7f9b140cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.925+0000 7f9b18e97700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b14154ff0 0x7f9b140cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58908/0 (socket says 172.21.15.7:58908) 2023-03-03T10:04:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.925+0000 7f9b18e97700 1 -- 172.21.15.7:0/453448601 learned_addr learned my addr 172.21.15.7:0/453448601 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:04:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.926+0000 7f9b13fff700 1 --2- 172.21.15.7:0/453448601 >> v2:172.21.15.143:3300/0 conn(0x7f9b140caa30 0x7f9b140c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.926+0000 7f9b18e97700 1 -- 172.21.15.7:0/453448601 >> v2:172.21.15.143:3300/0 conn(0x7f9b140caa30 msgr2=0x7f9b140c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:04:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.926+0000 7f9b18e97700 1 --2- 172.21.15.7:0/453448601 >> v2:172.21.15.143:3300/0 conn(0x7f9b140caa30 0x7f9b140c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.926+0000 7f9b18e97700 1 -- 172.21.15.7:0/453448601 >> v1:172.21.15.143:6789/0 conn(0x7f9b140c00e0 legacy=0x7f9b140c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:04:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.926+0000 7f9b18e97700 1 -- 172.21.15.7:0/453448601 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b140bfa30 con 0x7f9b14154ff0 2023-03-03T10:04:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.926+0000 7f9b18e97700 1 --2- 172.21.15.7:0/453448601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b14154ff0 0x7f9b140cae00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9b140c81b0 tx=0x7f9b0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f9c5dce073a00d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.926+0000 7f9b12ffd700 1 -- 172.21.15.7:0/453448601 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c00e070 con 0x7f9b14154ff0 2023-03-03T10:04:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.926+0000 7f9b12ffd700 1 -- 172.21.15.7:0/453448601 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0c00b9c0 con 0x7f9b14154ff0 2023-03-03T10:04:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.926+0000 7f9b12ffd700 1 -- 172.21.15.7:0/453448601 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c00bd70 con 0x7f9b14154ff0 2023-03-03T10:04:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.927+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/453448601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b14154ff0 msgr2=0x7f9b140cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:11.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.927+0000 7f9b1a8fa700 1 --2- 172.21.15.7:0/453448601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b14154ff0 0x7f9b140cae00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9b140c81b0 tx=0x7f9b0c00b330 comp rx=0 tx=0).stop 2023-03-03T10:04:11.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.927+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/453448601 shutdown_connections 2023-03-03T10:04:11.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.927+0000 7f9b1a8fa700 1 --2- 172.21.15.7:0/453448601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b14154ff0 0x7f9b140cae00 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:11.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.927+0000 7f9b1a8fa700 1 --2- 172.21.15.7:0/453448601 >> v2:172.21.15.143:3300/0 conn(0x7f9b140caa30 0x7f9b140c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:11.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.927+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/453448601 >> 172.21.15.7:0/453448601 conn(0x7f9b14066e40 msgr2=0x7f9b14071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:04:11.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.927+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/453448601 shutdown_connections 2023-03-03T10:04:11.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.927+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/453448601 wait complete. 2023-03-03T10:04:11.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.928+0000 7f9b1a8fa700 1 Processor -- start 2023-03-03T10:04:11.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.928+0000 7f9b1a8fa700 1 -- start start 2023-03-03T10:04:11.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b1a8fa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b140c00e0 0x7f9b141dff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:11.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b1a8fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b140caa30 0x7f9b141464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:11.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b1a8fa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b14154ff0 0x7f9b14146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:11.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b1a8fa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1414c9d0 con 0x7f9b140caa30 2023-03-03T10:04:11.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b1a8fa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1414d4c0 con 0x7f9b140c00e0 2023-03-03T10:04:11.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b1a8fa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1414c0a0 con 0x7f9b14154ff0 2023-03-03T10:04:11.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b140caa30 0x7f9b141464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:11.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b140caa30 0x7f9b141464c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58912/0 (socket says 172.21.15.7:58912) 2023-03-03T10:04:11.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b137fe700 1 -- 172.21.15.7:0/3605662759 learned_addr learned my addr 172.21.15.7:0/3605662759 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:04:11.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b13fff700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b140c00e0 0x7f9b141dff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:11.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b18e97700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b14154ff0 0x7f9b14146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:11.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b137fe700 1 -- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b14154ff0 msgr2=0x7f9b14146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:11.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b137fe700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b14154ff0 0x7f9b14146a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:11.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b137fe700 1 -- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b140c00e0 msgr2=0x7f9b141dff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:11.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b137fe700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b140c00e0 0x7f9b141dff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:11.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b137fe700 1 -- 172.21.15.7:0/3605662759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c009b30 con 0x7f9b140caa30 2023-03-03T10:04:11.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b13fff700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b140c00e0 0x7f9b141dff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:04:11.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b137fe700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b140caa30 0x7f9b141464c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00c4d0 tx=0x7f9afc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:11.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.929+0000 7f9b10ff9700 1 -- 172.21.15.7:0/3605662759 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc0052d0 con 0x7f9b140caa30 2023-03-03T10:04:11.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.930+0000 7f9b10ff9700 1 -- 172.21.15.7:0/3605662759 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc005430 con 0x7f9b140caa30 2023-03-03T10:04:11.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.930+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/3605662759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b141547f0 con 0x7f9b140caa30 2023-03-03T10:04:11.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.930+0000 7f9b10ff9700 1 -- 172.21.15.7:0/3605662759 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc0038f0 con 0x7f9b140caa30 2023-03-03T10:04:11.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.930+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/3605662759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b14154510 con 0x7f9b140caa30 2023-03-03T10:04:11.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.931+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/3605662759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b14154ad0 con 0x7f9b140caa30 2023-03-03T10:04:11.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.931+0000 7f9b10ff9700 1 -- 172.21.15.7:0/3605662759 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9afc019030 con 0x7f9b140caa30 2023-03-03T10:04:11.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.932+0000 7f9b10ff9700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9af4068320 0x7f9af406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:11.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.932+0000 7f9b10ff9700 1 -- 172.21.15.7:0/3605662759 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9afc085520 con 0x7f9b140caa30 2023-03-03T10:04:11.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.932+0000 7f9b13fff700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9af4068320 0x7f9af406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:11.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.932+0000 7f9b13fff700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9af4068320 0x7f9af406a7d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9b040099f0 tx=0x7f9b04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:11.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:11.936+0000 7f9b10ff9700 1 -- 172.21.15.7:0/3605662759 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9afc051860 con 0x7f9b140caa30 2023-03-03T10:04:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.180+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/3605662759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b1414cb10 con 0x7f9b140caa30 2023-03-03T10:04:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.180+0000 7f9b10ff9700 1 -- 172.21.15.7:0/3605662759 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9b1414cb10 con 0x7f9b140caa30 2023-03-03T10:04:12.183 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:04:12.183 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:04:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9af4068320 msgr2=0x7f9af406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9af4068320 0x7f9af406a7d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9b040099f0 tx=0x7f9b04008040 comp rx=0 tx=0).stop 2023-03-03T10:04:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/3605662759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b140caa30 msgr2=0x7f9b141464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b140caa30 0x7f9b141464c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00c4d0 tx=0x7f9afc00c890 comp rx=0 tx=0).stop 2023-03-03T10:04:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/3605662759 shutdown_connections 2023-03-03T10:04:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b14154ff0 0x7f9b14146a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b140c00e0 0x7f9b141dff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9af4068320 0x7f9af406a7d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 --2- 172.21.15.7:0/3605662759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b140caa30 0x7f9b141464c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/3605662759 >> 172.21.15.7:0/3605662759 conn(0x7f9b14066e40 msgr2=0x7f9b140c8850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:04:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/3605662759 shutdown_connections 2023-03-03T10:04:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:12.182+0000 7f9b1a8fa700 1 -- 172.21.15.7:0/3605662759 wait complete. 2023-03-03T10:04:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:04:12.197 INFO:tasks.mds_thrash.fs.[a]:waiting for 31 secs before thrashing 2023-03-03T10:04:21.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:04:21.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.295+0000 7f014f7a8700 1 Processor -- start 2023-03-03T10:04:21.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.295+0000 7f014f7a8700 1 -- start start 2023-03-03T10:04:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.296+0000 7f014f7a8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0148151e80 0x7f01480c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.296+0000 7f014f7a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0148155410 0x7f01480bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:21.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.296+0000 7f014f7a8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01480c08d0 con 0x7f01480c6fe0 2023-03-03T10:04:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.296+0000 7f014f7a8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0148152250 con 0x7f0148151e80 2023-03-03T10:04:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.296+0000 7f014f7a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01480c0fb0 con 0x7f0148155410 2023-03-03T10:04:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.296+0000 7f014d544700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0148151e80 0x7f01480c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.296+0000 7f014dd45700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f01480c6fe0 0x7f01480bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52902/0 (socket says 172.21.15.7:52902) 2023-03-03T10:04:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.296+0000 7f014dd45700 1 -- 172.21.15.7:0/3504311694 learned_addr learned my addr 172.21.15.7:0/3504311694 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:04:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.296+0000 7f014cd43700 1 --2- 172.21.15.7:0/3504311694 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0148155410 0x7f01480bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.297+0000 7f014d544700 1 -- 172.21.15.7:0/3504311694 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0148155410 msgr2=0x7f01480bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:04:21.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.297+0000 7f014d544700 1 --2- 172.21.15.7:0/3504311694 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0148155410 0x7f01480bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.297+0000 7f014d544700 1 -- 172.21.15.7:0/3504311694 >> v1:172.21.15.7:6789/0 conn(0x7f01480c6fe0 legacy=0x7f01480bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.297+0000 7f014d544700 1 -- 172.21.15.7:0/3504311694 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0148154930 con 0x7f0148151e80 2023-03-03T10:04:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.297+0000 7f013ffff700 1 -- 172.21.15.7:0/3504311694 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457844098 0 0) 0x7f01480c08d0 con 0x7f01480c6fe0 2023-03-03T10:04:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.297+0000 7f014d544700 1 --2- 172.21.15.7:0/3504311694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0148151e80 0x7f01480c6aa0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0138009870 tx=0x7f0138009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d929d839336271 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.297+0000 7f013ffff700 1 -- 172.21.15.7:0/3504311694 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138011830 con 0x7f0148151e80 2023-03-03T10:04:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.297+0000 7f013ffff700 1 -- 172.21.15.7:0/3504311694 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138011990 con 0x7f0148151e80 2023-03-03T10:04:21.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.297+0000 7f013ffff700 1 -- 172.21.15.7:0/3504311694 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138011d20 con 0x7f0148151e80 2023-03-03T10:04:21.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.298+0000 7f014f7a8700 1 -- 172.21.15.7:0/3504311694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0148151e80 msgr2=0x7f01480c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:21.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.298+0000 7f014f7a8700 1 --2- 172.21.15.7:0/3504311694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0148151e80 0x7f01480c6aa0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0138009870 tx=0x7f0138009b80 comp rx=0 tx=0).stop 2023-03-03T10:04:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.298+0000 7f014f7a8700 1 -- 172.21.15.7:0/3504311694 shutdown_connections 2023-03-03T10:04:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.298+0000 7f014f7a8700 1 --2- 172.21.15.7:0/3504311694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0148151e80 0x7f01480c6aa0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.298+0000 7f014f7a8700 1 --2- 172.21.15.7:0/3504311694 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0148155410 0x7f01480bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.298+0000 7f014f7a8700 1 -- 172.21.15.7:0/3504311694 >> 172.21.15.7:0/3504311694 conn(0x7f0148071630 msgr2=0x7f014806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:04:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.298+0000 7f014f7a8700 1 -- 172.21.15.7:0/3504311694 shutdown_connections 2023-03-03T10:04:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.298+0000 7f014f7a8700 1 -- 172.21.15.7:0/3504311694 wait complete. 2023-03-03T10:04:21.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.299+0000 7f014f7a8700 1 Processor -- start 2023-03-03T10:04:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.299+0000 7f014f7a8700 1 -- start start 2023-03-03T10:04:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014f7a8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01480c6fe0 0x7f01481e1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014f7a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0148151e80 0x7f0148148b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014f7a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0148155410 0x7f01481490c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014f7a8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01481ee5c0 con 0x7f01480c6fe0 2023-03-03T10:04:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014f7a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01481ed800 con 0x7f0148151e80 2023-03-03T10:04:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014f7a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01481edae0 con 0x7f0148155410 2023-03-03T10:04:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014d544700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01480c6fe0 0x7f01481e1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014d544700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01480c6fe0 0x7f01481e1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58924/0 (socket says 172.21.15.7:58924) 2023-03-03T10:04:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014d544700 1 -- 172.21.15.7:0/751354541 learned_addr learned my addr 172.21.15.7:0/751354541 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:04:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014d544700 1 -- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0148155410 msgr2=0x7f01481490c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014d544700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0148155410 0x7f01481490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014d544700 1 -- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0148151e80 msgr2=0x7f0148148b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014d544700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0148151e80 0x7f0148148b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014d544700 1 -- 172.21.15.7:0/751354541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380094a0 con 0x7f01480c6fe0 2023-03-03T10:04:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014d544700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01480c6fe0 0x7f01481e1440 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f01380090b0 tx=0x7f0138004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f013dffb700 1 -- 172.21.15.7:0/751354541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013801aa10 con 0x7f01480c6fe0 2023-03-03T10:04:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f013dffb700 1 -- 172.21.15.7:0/751354541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013801ab70 con 0x7f01480c6fe0 2023-03-03T10:04:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f014f7a8700 1 -- 172.21.15.7:0/751354541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0148154ef0 con 0x7f01480c6fe0 2023-03-03T10:04:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.300+0000 7f013dffb700 1 -- 172.21.15.7:0/751354541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01380243f0 con 0x7f01480c6fe0 2023-03-03T10:04:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.301+0000 7f014f7a8700 1 -- 172.21.15.7:0/751354541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0148154c10 con 0x7f01480c6fe0 2023-03-03T10:04:21.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.302+0000 7f014f7a8700 1 -- 172.21.15.7:0/751354541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01480c5220 con 0x7f01480c6fe0 2023-03-03T10:04:21.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.302+0000 7f013dffb700 1 -- 172.21.15.7:0/751354541 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f013800f050 con 0x7f01480c6fe0 2023-03-03T10:04:21.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.302+0000 7f013dffb700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f012c0682f0 0x7f012c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:21.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.302+0000 7f013dffb700 1 -- 172.21.15.7:0/751354541 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0138087670 con 0x7f01480c6fe0 2023-03-03T10:04:21.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.303+0000 7f014cd43700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f012c0682f0 0x7f012c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:21.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.303+0000 7f014cd43700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f012c0682f0 0x7f012c06a7a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0144006b20 tx=0x7f0144006ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:21.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.308+0000 7f013dffb700 1 -- 172.21.15.7:0/751354541 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0138053920 con 0x7f01480c6fe0 2023-03-03T10:04:21.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.544+0000 7f014f7a8700 1 -- 172.21.15.7:0/751354541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0148074bb0 con 0x7f01480c6fe0 2023-03-03T10:04:21.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.545+0000 7f013dffb700 1 -- 172.21.15.7:0/751354541 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0138056f40 con 0x7f01480c6fe0 2023-03-03T10:04:21.548 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:04:21.548 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:04:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 -- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f012c0682f0 msgr2=0x7f012c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f012c0682f0 0x7f012c06a7a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0144006b20 tx=0x7f0144006ab0 comp rx=0 tx=0).stop 2023-03-03T10:04:21.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 -- 172.21.15.7:0/751354541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01480c6fe0 msgr2=0x7f01481e1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:21.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01480c6fe0 0x7f01481e1440 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f01380090b0 tx=0x7f0138004750 comp rx=0 tx=0).stop 2023-03-03T10:04:21.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 -- 172.21.15.7:0/751354541 shutdown_connections 2023-03-03T10:04:21.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0148155410 0x7f01481490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:21.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f012c0682f0 0x7f012c06a7a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:21.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01480c6fe0 0x7f01481e1440 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:21.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 --2- 172.21.15.7:0/751354541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0148151e80 0x7f0148148b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:21.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 -- 172.21.15.7:0/751354541 >> 172.21.15.7:0/751354541 conn(0x7f0148071630 msgr2=0x7f01480c36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:04:21.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 -- 172.21.15.7:0/751354541 shutdown_connections 2023-03-03T10:04:21.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:21.547+0000 7f014f7a8700 1 -- 172.21.15.7:0/751354541 wait complete. 2023-03-03T10:04:21.551 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:04:21.561 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T10:04:24.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:24.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:39.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:04:39.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.701+0000 7fb24b0fc700 1 Processor -- start 2023-03-03T10:04:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.701+0000 7fb24b0fc700 1 -- start start 2023-03-03T10:04:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb24b0fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2440717a0 0x7fb2440bdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb24b0fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb244151c30 0x7fb244071e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb24b0fc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2441517a0 con 0x7fb24406e7f0 2023-03-03T10:04:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb24b0fc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2440bd1a0 con 0x7fb244151c30 2023-03-03T10:04:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb24b0fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb244151170 con 0x7fb2440717a0 2023-03-03T10:04:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb243fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb24406e7f0 0x7fb244154760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52922/0 (socket says 172.21.15.7:52922) 2023-03-03T10:04:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb243fff700 1 -- 172.21.15.7:0/1620116628 learned_addr learned my addr 172.21.15.7:0/1620116628 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:04:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb249699700 1 --2- 172.21.15.7:0/1620116628 >> v2:172.21.15.7:3300/0 conn(0x7fb244151c30 0x7fb244071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb248e98700 1 --2- 172.21.15.7:0/1620116628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2440717a0 0x7fb2440bdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb2437fe700 1 -- 172.21.15.7:0/1620116628 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229621081 0 0) 0x7fb2441517a0 con 0x7fb24406e7f0 2023-03-03T10:04:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb2437fe700 1 -- 172.21.15.7:0/1620116628 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2340035b0 con 0x7fb24406e7f0 2023-03-03T10:04:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb249699700 1 -- 172.21.15.7:0/1620116628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2440717a0 msgr2=0x7fb2440bdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:04:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb249699700 1 --2- 172.21.15.7:0/1620116628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2440717a0 0x7fb2440bdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb249699700 1 -- 172.21.15.7:0/1620116628 >> v1:172.21.15.7:6789/0 conn(0x7fb24406e7f0 legacy=0x7fb244154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:39.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.702+0000 7fb249699700 1 -- 172.21.15.7:0/1620116628 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2440bc2f0 con 0x7fb244151c30 2023-03-03T10:04:39.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb249699700 1 --2- 172.21.15.7:0/1620116628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb244151c30 0x7fb244071e10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009d90 tx=0x7fb23c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14fca8615b5dfcd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:39.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb2437fe700 1 -- 172.21.15.7:0/1620116628 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c00e070 con 0x7fb244151c30 2023-03-03T10:04:39.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb2437fe700 1 -- 172.21.15.7:0/1620116628 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c00bb20 con 0x7fb244151c30 2023-03-03T10:04:39.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb2437fe700 1 -- 172.21.15.7:0/1620116628 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c012430 con 0x7fb244151c30 2023-03-03T10:04:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb24b0fc700 1 -- 172.21.15.7:0/1620116628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb244151c30 msgr2=0x7fb244071e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb24b0fc700 1 --2- 172.21.15.7:0/1620116628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb244151c30 0x7fb244071e10 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009d90 tx=0x7fb23c00b330 comp rx=0 tx=0).stop 2023-03-03T10:04:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb24b0fc700 1 -- 172.21.15.7:0/1620116628 shutdown_connections 2023-03-03T10:04:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb24b0fc700 1 --2- 172.21.15.7:0/1620116628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb244151c30 0x7fb244071e10 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb24b0fc700 1 --2- 172.21.15.7:0/1620116628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2440717a0 0x7fb2440bdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb24b0fc700 1 -- 172.21.15.7:0/1620116628 >> 172.21.15.7:0/1620116628 conn(0x7fb2440a94c0 msgr2=0x7fb2440a98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:04:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb24b0fc700 1 -- 172.21.15.7:0/1620116628 shutdown_connections 2023-03-03T10:04:39.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.703+0000 7fb24b0fc700 1 -- 172.21.15.7:0/1620116628 wait complete. 2023-03-03T10:04:39.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.704+0000 7fb24b0fc700 1 Processor -- start 2023-03-03T10:04:39.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.705+0000 7fb24b0fc700 1 -- start start 2023-03-03T10:04:39.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.705+0000 7fb24b0fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb24406e7f0 0x7fb2441dcec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:39.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.705+0000 7fb24b0fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2440717a0 0x7fb244148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:39.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.705+0000 7fb24b0fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb244151c30 0x7fb244148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:39.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.705+0000 7fb24b0fc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2441e8560 con 0x7fb24406e7f0 2023-03-03T10:04:39.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.705+0000 7fb24b0fc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440c5f40 con 0x7fb244151c30 2023-03-03T10:04:39.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.705+0000 7fb24b0fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24414f970 con 0x7fb2440717a0 2023-03-03T10:04:39.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb248e98700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb24406e7f0 0x7fb2441dcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:39.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb248e98700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb24406e7f0 0x7fb2441dcec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58946/0 (socket says 172.21.15.7:58946) 2023-03-03T10:04:39.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb248e98700 1 -- 172.21.15.7:0/246952640 learned_addr learned my addr 172.21.15.7:0/246952640 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:04:39.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb243fff700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2440717a0 0x7fb244148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:39.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb248e98700 1 -- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2440717a0 msgr2=0x7fb244148a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:39.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb249699700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb244151c30 0x7fb244148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:39.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb248e98700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2440717a0 0x7fb244148a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:39.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb248e98700 1 -- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb244151c30 msgr2=0x7fb244148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:39.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb248e98700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb244151c30 0x7fb244148f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:39.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb248e98700 1 -- 172.21.15.7:0/246952640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c009a70 con 0x7fb24406e7f0 2023-03-03T10:04:39.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb249699700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb244151c30 0x7fb244148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:04:39.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb248e98700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb24406e7f0 0x7fb2441dcec0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb23800e810 tx=0x7fb23800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:39.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb2417fa700 1 -- 172.21.15.7:0/246952640 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238013610 con 0x7fb24406e7f0 2023-03-03T10:04:39.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb2417fa700 1 -- 172.21.15.7:0/246952640 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb238013770 con 0x7fb24406e7f0 2023-03-03T10:04:39.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb24b0fc700 1 -- 172.21.15.7:0/246952640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb244151430 con 0x7fb24406e7f0 2023-03-03T10:04:39.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb2417fa700 1 -- 172.21.15.7:0/246952640 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238013b20 con 0x7fb24406e7f0 2023-03-03T10:04:39.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.706+0000 7fb24b0fc700 1 -- 172.21.15.7:0/246952640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2440bc5d0 con 0x7fb24406e7f0 2023-03-03T10:04:39.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.707+0000 7fb24b0fc700 1 -- 172.21.15.7:0/246952640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2441520d0 con 0x7fb24406e7f0 2023-03-03T10:04:39.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.708+0000 7fb2417fa700 1 -- 172.21.15.7:0/246952640 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb238023070 con 0x7fb24406e7f0 2023-03-03T10:04:39.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.708+0000 7fb2417fa700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2280682f0 0x7fb22806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:39.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.708+0000 7fb2417fa700 1 -- 172.21.15.7:0/246952640 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb238085ea0 con 0x7fb24406e7f0 2023-03-03T10:04:39.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.708+0000 7fb243fff700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2280682f0 0x7fb22806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:39.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.709+0000 7fb243fff700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2280682f0 0x7fb22806a7a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb2440c17c0 tx=0x7fb230009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:39.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.712+0000 7fb2417fa700 1 -- 172.21.15.7:0/246952640 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb2380521c0 con 0x7fb24406e7f0 2023-03-03T10:04:39.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.950+0000 7fb24b0fc700 1 -- 172.21.15.7:0/246952640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb244074bb0 con 0x7fb24406e7f0 2023-03-03T10:04:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.950+0000 7fb2417fa700 1 -- 172.21.15.7:0/246952640 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb2380557e0 con 0x7fb24406e7f0 2023-03-03T10:04:39.952 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:04:39.952 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 -- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2280682f0 msgr2=0x7fb22806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2280682f0 0x7fb22806a7a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fb2440c17c0 tx=0x7fb230009040 comp rx=0 tx=0).stop 2023-03-03T10:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 -- 172.21.15.7:0/246952640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb24406e7f0 msgr2=0x7fb2441dcec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb24406e7f0 0x7fb2441dcec0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fb23800e810 tx=0x7fb23800ebd0 comp rx=0 tx=0).stop 2023-03-03T10:04:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 -- 172.21.15.7:0/246952640 shutdown_connections 2023-03-03T10:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb244151c30 0x7fb244148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2280682f0 0x7fb22806a7a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb24406e7f0 0x7fb2441dcec0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 --2- 172.21.15.7:0/246952640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2440717a0 0x7fb244148a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 -- 172.21.15.7:0/246952640 >> 172.21.15.7:0/246952640 conn(0x7fb2440a94c0 msgr2=0x7fb2440a9890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 -- 172.21.15.7:0/246952640 shutdown_connections 2023-03-03T10:04:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:39.952+0000 7fb24b0fc700 1 -- 172.21.15.7:0/246952640 wait complete. 2023-03-03T10:04:39.956 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:04:39.966 INFO:tasks.mds_thrash.fs.[b]:waiting for 118 secs before thrashing 2023-03-03T10:04:43.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:04:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.336+0000 7f98bf1e0700 1 Processor -- start 2023-03-03T10:04:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.336+0000 7f98bf1e0700 1 -- start start 2023-03-03T10:04:43.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98bf1e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98b80beab0 0x7f98b8155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98bf1e0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f98b81562c0 0x7f98b815cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98bf1e0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b81523b0 con 0x7f98b8152e30 2023-03-03T10:04:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98bf1e0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b8072010 con 0x7f98b81562c0 2023-03-03T10:04:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98bf1e0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80aaaf0 con 0x7f98b80beab0 2023-03-03T10:04:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98bcf7c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f98b8152e30 0x7f98b80c43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52934/0 (socket says 172.21.15.7:52934) 2023-03-03T10:04:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98bcf7c700 1 -- 172.21.15.7:0/336983718 learned_addr learned my addr 172.21.15.7:0/336983718 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:04:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98bd77d700 1 --2- 172.21.15.7:0/336983718 >> v2:172.21.15.143:3300/0 conn(0x7f98b81562c0 0x7f98b815cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98b7fff700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98b80beab0 0x7f98b8155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173924631 0 0) 0x7f98b81523b0 con 0x7f98b8152e30 2023-03-03T10:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a80035b0 con 0x7f98b8152e30 2023-03-03T10:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728802337 0 0) 0x7f98a80035b0 con 0x7f98b8152e30 2023-03-03T10:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98b80beab0 msgr2=0x7f98b8155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98b77fe700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98b80beab0 0x7f98b8155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 >> v2:172.21.15.143:3300/0 conn(0x7f98b81562c0 msgr2=0x7f98b815cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:04:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98b77fe700 1 --2- 172.21.15.7:0/336983718 >> v2:172.21.15.143:3300/0 conn(0x7f98b81562c0 0x7f98b815cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.337+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b80aadd0 con 0x7f98b8152e30 2023-03-03T10:04:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f98ac001670 con 0x7f98b8152e30 2023-03-03T10:04:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 >> v1:172.21.15.7:6789/0 conn(0x7f98b8152e30 legacy=0x7f98b80c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a8003a70 0x7f98a8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98a8004850 0x7f98a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98a8005200 0x7f98a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b8072010 con 0x7f98a8003a70 2023-03-03T10:04:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b80aaaf0 con 0x7f98a8004850 2023-03-03T10:04:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a800bfd0 con 0x7f98a8005200 2023-03-03T10:04:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b7fff700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98b80beab0 0x7f98b8155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:04:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98bd77d700 1 --2- 172.21.15.7:0/336983718 >> v2:172.21.15.143:3300/0 conn(0x7f98b81562c0 0x7f98b815cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:04:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98bcf7c700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a8003a70 0x7f98a8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b7fff700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98a8004850 0x7f98a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98bd77d700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98a8005200 0x7f98a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98bcf7c700 1 -- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98a8005200 msgr2=0x7f98a800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98bcf7c700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98a8005200 0x7f98a800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98bcf7c700 1 -- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98a8004850 msgr2=0x7f98a8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98bcf7c700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98a8004850 0x7f98a8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98bcf7c700 1 -- 172.21.15.7:0/336983718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a80035b0 con 0x7f98a8003a70 2023-03-03T10:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98bcf7c700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a8003a70 0x7f98a8003e80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00c320 tx=0x7f98a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881eb28c8e3fd948 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac00ce60 con 0x7f98a8003a70 2023-03-03T10:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac005960 con 0x7f98a8003a70 2023-03-03T10:04:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.338+0000 7f98b77fe700 1 -- 172.21.15.7:0/336983718 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac005cf0 con 0x7f98a8003a70 2023-03-03T10:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.339+0000 7f98bf1e0700 1 -- 172.21.15.7:0/336983718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a8003a70 msgr2=0x7f98a8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.339+0000 7f98bf1e0700 1 --2- 172.21.15.7:0/336983718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98a8003a70 0x7f98a8003e80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00c320 tx=0x7f98a8005880 comp rx=0 tx=0).stop 2023-03-03T10:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.339+0000 7f98bd77d700 1 -- 172.21.15.7:0/336983718 reap_dead start 2023-03-03T10:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.339+0000 7f98bf1e0700 1 -- 172.21.15.7:0/336983718 shutdown_connections 2023-03-03T10:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.339+0000 7f98bf1e0700 1 -- 172.21.15.7:0/336983718 >> 172.21.15.7:0/336983718 conn(0x7f98b80736f0 msgr2=0x7f98b8073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.339+0000 7f98bf1e0700 1 -- 172.21.15.7:0/336983718 shutdown_connections 2023-03-03T10:04:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.339+0000 7f98bf1e0700 1 -- 172.21.15.7:0/336983718 wait complete. 2023-03-03T10:04:43.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.340+0000 7f98bf1e0700 1 Processor -- start 2023-03-03T10:04:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.340+0000 7f98bf1e0700 1 -- start start 2023-03-03T10:04:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bf1e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98a8003a70 0x7f98b80ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bf1e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98b81eb760 0x7f98b80c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bf1e0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98b81e77e0 0x7f98b81e7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bf1e0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b81559c0 con 0x7f98b81e77e0 2023-03-03T10:04:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bf1e0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b81ebbd0 con 0x7f98a8003a70 2023-03-03T10:04:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bf1e0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b81543f0 con 0x7f98b81eb760 2023-03-03T10:04:43.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bd77d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98b81e77e0 0x7f98b81e7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:43.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bd77d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98b81e77e0 0x7f98b81e7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58970/0 (socket says 172.21.15.7:58970) 2023-03-03T10:04:43.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bd77d700 1 -- 172.21.15.7:0/2224457213 learned_addr learned my addr 172.21.15.7:0/2224457213 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:04:43.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bcf7c700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98a8003a70 0x7f98b80ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:43.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98b7fff700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98b81eb760 0x7f98b80c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:43.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bd77d700 1 -- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98b81eb760 msgr2=0x7f98b80c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:43.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bd77d700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98b81eb760 0x7f98b80c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:43.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bd77d700 1 -- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98a8003a70 msgr2=0x7f98b80ae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:43.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bd77d700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98a8003a70 0x7f98b80ae0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:43.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.341+0000 7f98bd77d700 1 -- 172.21.15.7:0/2224457213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8153200 con 0x7f98b81e77e0 2023-03-03T10:04:43.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.342+0000 7f98bd77d700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98b81e77e0 0x7f98b81e7c80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f98b80c5970 tx=0x7f98b0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:43.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.342+0000 7f98b57fa700 1 -- 172.21.15.7:0/2224457213 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b000e070 con 0x7f98b81e77e0 2023-03-03T10:04:43.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.342+0000 7f98b57fa700 1 -- 172.21.15.7:0/2224457213 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b0003c30 con 0x7f98b81e77e0 2023-03-03T10:04:43.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.342+0000 7f98bf1e0700 1 -- 172.21.15.7:0/2224457213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b8071c70 con 0x7f98b81e77e0 2023-03-03T10:04:43.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.342+0000 7f98b57fa700 1 -- 172.21.15.7:0/2224457213 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b0003dc0 con 0x7f98b81e77e0 2023-03-03T10:04:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.342+0000 7f98bf1e0700 1 -- 172.21.15.7:0/2224457213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b8153390 con 0x7f98b81e77e0 2023-03-03T10:04:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.343+0000 7f98bf1e0700 1 -- 172.21.15.7:0/2224457213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b8155810 con 0x7f98b81e77e0 2023-03-03T10:04:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.344+0000 7f98b57fa700 1 -- 172.21.15.7:0/2224457213 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f98b00119a0 con 0x7f98b81e77e0 2023-03-03T10:04:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.344+0000 7f98b57fa700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f989c0682f0 0x7f989c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:04:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.344+0000 7f98b57fa700 1 -- 172.21.15.7:0/2224457213 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f98b0087160 con 0x7f98b81e77e0 2023-03-03T10:04:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.344+0000 7f98bcf7c700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f989c0682f0 0x7f989c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:04:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.345+0000 7f98bcf7c700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f989c0682f0 0x7f989c06a7a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f98ac008fd0 tx=0x7f98ac00a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:04:43.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.348+0000 7f98b57fa700 1 -- 172.21.15.7:0/2224457213 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f98b00535a0 con 0x7f98b81e77e0 2023-03-03T10:04:43.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.586+0000 7f98bf1e0700 1 -- 172.21.15.7:0/2224457213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98b8074bb0 con 0x7f98b81e77e0 2023-03-03T10:04:43.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.586+0000 7f98b57fa700 1 -- 172.21.15.7:0/2224457213 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f98b0056bc0 con 0x7f98b81e77e0 2023-03-03T10:04:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:04:43.588 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:04:43.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 -- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f989c0682f0 msgr2=0x7f989c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:43.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f989c0682f0 0x7f989c06a7a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f98ac008fd0 tx=0x7f98ac00a000 comp rx=0 tx=0).stop 2023-03-03T10:04:43.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 -- 172.21.15.7:0/2224457213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98b81e77e0 msgr2=0x7f98b81e7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:04:43.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98b81e77e0 0x7f98b81e7c80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f98b80c5970 tx=0x7f98b0009f90 comp rx=0 tx=0).stop 2023-03-03T10:04:43.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 -- 172.21.15.7:0/2224457213 shutdown_connections 2023-03-03T10:04:43.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f989c0682f0 0x7f989c06a7a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:43.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98b81e77e0 0x7f98b81e7c80 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:43.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98a8003a70 0x7f98b80ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:43.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 --2- 172.21.15.7:0/2224457213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98b81eb760 0x7f98b80c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:04:43.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 -- 172.21.15.7:0/2224457213 >> 172.21.15.7:0/2224457213 conn(0x7f98b80736f0 msgr2=0x7f98b80c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:04:43.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 -- 172.21.15.7:0/2224457213 shutdown_connections 2023-03-03T10:04:43.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:04:43.588+0000 7f98bf1e0700 1 -- 172.21.15.7:0/2224457213 wait complete. 2023-03-03T10:04:43.591 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:04:43.601 INFO:tasks.mds_thrash.fs.[a]:waiting for 45 secs before thrashing 2023-03-03T10:04:54.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:04:54.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:24.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:24.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:28.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:05:28.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.740+0000 7f2df516b700 1 Processor -- start 2023-03-03T10:05:28.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.740+0000 7f2df516b700 1 -- start start 2023-03-03T10:05:28.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.740+0000 7f2df516b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2df00c1da0 0x7f2df01550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:28.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2df516b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df00c7a60 con 0x7f2df00c3e30 2023-03-03T10:05:28.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2df516b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df00c3620 con 0x7f2df006d870 2023-03-03T10:05:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2df516b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df00c2af0 con 0x7f2df00c1da0 2023-03-03T10:05:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2def59e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2df00c3e30 0x7f2df015ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52990/0 (socket says 172.21.15.7:52990) 2023-03-03T10:05:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2def59e700 1 -- 172.21.15.7:0/3647165525 learned_addr learned my addr 172.21.15.7:0/3647165525 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:05:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dee59c700 1 --2- 172.21.15.7:0/3647165525 >> v2:172.21.15.143:3300/0 conn(0x7f2df00c1da0 0x7f2df01550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765807674 0 0) 0x7f2df00c7a60 con 0x7f2df00c3e30 2023-03-03T10:05:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd80035b0 con 0x7f2df00c3e30 2023-03-03T10:05:28.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1627833929 0 0) 0x7f2dd80035b0 con 0x7f2df00c3e30 2023-03-03T10:05:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 >> v2:172.21.15.143:3300/0 conn(0x7f2df00c1da0 msgr2=0x7f2df01550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:05:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 --2- 172.21.15.7:0/3647165525 >> v2:172.21.15.143:3300/0 conn(0x7f2df00c1da0 0x7f2df01550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 >> v1:172.21.15.143:6789/0 conn(0x7f2df006d870 legacy=0x7f2df00c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df01545e0 con 0x7f2df00c3e30 2023-03-03T10:05:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f2de80019f0 con 0x7f2df00c3e30 2023-03-03T10:05:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 >> v1:172.21.15.7:6789/0 conn(0x7f2df00c3e30 legacy=0x7f2df015ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:28.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd8003a70 0x7f2dd8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:28.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd8004850 0x7f2dd8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:28.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2dd800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:28.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd800c030 con 0x7f2dd8005200 2023-03-03T10:05:28.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd800c1a0 con 0x7f2dd8004850 2023-03-03T10:05:28.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.741+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd800c360 con 0x7f2dd8003a70 2023-03-03T10:05:28.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2def59e700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2dd800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:28.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2dee59c700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd8004850 0x7f2dd8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2deed9d700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd8003a70 0x7f2dd8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2def59e700 1 -- 172.21.15.7:0/3647165525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd8003a70 msgr2=0x7f2dd8003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:05:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2def59e700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2dd8003a70 0x7f2dd8003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2def59e700 1 -- 172.21.15.7:0/3647165525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd8004850 msgr2=0x7f2dd8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:05:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2def59e700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd8004850 0x7f2dd8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2def59e700 1 -- 172.21.15.7:0/3647165525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd80035b0 con 0x7f2dd8005200 2023-03-03T10:05:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2dee59c700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2dd8004850 0x7f2dd8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:05:28.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2def59e700 1 -- 172.21.15.7:0/3647165525 reap_dead start 2023-03-03T10:05:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2def59e700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2dd800ba10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2de800a830 tx=0x7f2dd8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca491f93644efb1f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:05:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8012600 con 0x7f2dd8005200 2023-03-03T10:05:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de8012780 con 0x7f2dd8005200 2023-03-03T10:05:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.742+0000 7f2dedd9b700 1 -- 172.21.15.7:0/3647165525 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8012b30 con 0x7f2dd8005200 2023-03-03T10:05:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.743+0000 7f2df516b700 1 -- 172.21.15.7:0/3647165525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 msgr2=0x7f2dd800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.743+0000 7f2df516b700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2dd800ba10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f2de800a830 tx=0x7f2dd8005880 comp rx=0 tx=0).stop 2023-03-03T10:05:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.743+0000 7f2df516b700 1 -- 172.21.15.7:0/3647165525 shutdown_connections 2023-03-03T10:05:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.743+0000 7f2df516b700 1 --2- 172.21.15.7:0/3647165525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2dd800ba10 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.743+0000 7f2df516b700 1 -- 172.21.15.7:0/3647165525 >> 172.21.15.7:0/3647165525 conn(0x7f2df006c110 msgr2=0x7f2df006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:05:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.743+0000 7f2df516b700 1 -- 172.21.15.7:0/3647165525 shutdown_connections 2023-03-03T10:05:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.743+0000 7f2df516b700 1 -- 172.21.15.7:0/3647165525 wait complete. 2023-03-03T10:05:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.744+0000 7f2df516b700 1 Processor -- start 2023-03-03T10:05:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.744+0000 7f2df516b700 1 -- start start 2023-03-03T10:05:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2df516b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2df0157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2df516b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2df01584d0 0x7f2df0151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2df516b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2df00ab9a0 0x7f2df0151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2df516b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df00aaf00 con 0x7f2dd8005200 2023-03-03T10:05:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2df516b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df0151d80 con 0x7f2df00ab9a0 2023-03-03T10:05:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2df516b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df00abf50 con 0x7f2df01584d0 2023-03-03T10:05:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2deed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2df0157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2deed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2df0157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59018/0 (socket says 172.21.15.7:59018) 2023-03-03T10:05:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2deed9d700 1 -- 172.21.15.7:0/750387806 learned_addr learned my addr 172.21.15.7:0/750387806 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:05:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2def59e700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2df00ab9a0 0x7f2df0151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2dee59c700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2df01584d0 0x7f2df0151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2deed9d700 1 -- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2df01584d0 msgr2=0x7f2df0151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2deed9d700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2df01584d0 0x7f2df0151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2deed9d700 1 -- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2df00ab9a0 msgr2=0x7f2df0151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2deed9d700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2df00ab9a0 0x7f2df0151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2deed9d700 1 -- 172.21.15.7:0/750387806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df0154ba0 con 0x7f2dd8005200 2023-03-03T10:05:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.745+0000 7f2dee59c700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2df01584d0 0x7f2df0151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:05:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.746+0000 7f2deed9d700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2df0157900 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2de000f780 tx=0x7f2de000fa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:05:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.746+0000 7f2dd77fe700 1 -- 172.21.15.7:0/750387806 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de00052e0 con 0x7f2dd8005200 2023-03-03T10:05:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.746+0000 7f2dd77fe700 1 -- 172.21.15.7:0/750387806 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de0005460 con 0x7f2dd8005200 2023-03-03T10:05:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.746+0000 7f2dd77fe700 1 -- 172.21.15.7:0/750387806 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de00038c0 con 0x7f2dd8005200 2023-03-03T10:05:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.746+0000 7f2df516b700 1 -- 172.21.15.7:0/750387806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df01548c0 con 0x7f2dd8005200 2023-03-03T10:05:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.746+0000 7f2df516b700 1 -- 172.21.15.7:0/750387806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df006dc40 con 0x7f2dd8005200 2023-03-03T10:05:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.747+0000 7f2df516b700 1 -- 172.21.15.7:0/750387806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df0074bb0 con 0x7f2dd8005200 2023-03-03T10:05:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.747+0000 7f2dd77fe700 1 -- 172.21.15.7:0/750387806 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2de0003a20 con 0x7f2dd8005200 2023-03-03T10:05:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.748+0000 7f2dd77fe700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ddc068320 0x7f2ddc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:05:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.748+0000 7f2dd77fe700 1 -- 172.21.15.7:0/750387806 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2de00860c0 con 0x7f2dd8005200 2023-03-03T10:05:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.748+0000 7f2dee59c700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ddc068320 0x7f2ddc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:05:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.748+0000 7f2dee59c700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ddc068320 0x7f2ddc06a7d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2df0153190 tx=0x7f2de4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:05:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.752+0000 7f2dd77fe700 1 -- 172.21.15.7:0/750387806 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2de0052460 con 0x7f2dd8005200 2023-03-03T10:05:28.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.992+0000 7f2df516b700 1 -- 172.21.15.7:0/750387806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2df00744b0 con 0x7f2dd8005200 2023-03-03T10:05:28.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.992+0000 7f2dd77fe700 1 -- 172.21.15.7:0/750387806 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2de0003cb0 con 0x7f2dd8005200 2023-03-03T10:05:28.994 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:05:28.994 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:05:28.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.994+0000 7f2df516b700 1 -- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ddc068320 msgr2=0x7f2ddc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:28.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.994+0000 7f2df516b700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ddc068320 0x7f2ddc06a7d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2df0153190 tx=0x7f2de4008040 comp rx=0 tx=0).stop 2023-03-03T10:05:28.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.994+0000 7f2df516b700 1 -- 172.21.15.7:0/750387806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 msgr2=0x7f2df0157900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:05:28.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.994+0000 7f2df516b700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2df0157900 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2de000f780 tx=0x7f2de000fa90 comp rx=0 tx=0).stop 2023-03-03T10:05:28.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.995+0000 7f2df516b700 1 -- 172.21.15.7:0/750387806 shutdown_connections 2023-03-03T10:05:28.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.995+0000 7f2df516b700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2df00ab9a0 0x7f2df0151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:28.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.995+0000 7f2df516b700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ddc068320 0x7f2ddc06a7d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:28.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.995+0000 7f2df516b700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2dd8005200 0x7f2df0157900 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:28.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.995+0000 7f2df516b700 1 --2- 172.21.15.7:0/750387806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2df01584d0 0x7f2df0151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:05:28.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.995+0000 7f2df516b700 1 -- 172.21.15.7:0/750387806 >> 172.21.15.7:0/750387806 conn(0x7f2df006c110 msgr2=0x7f2df00c1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:05:28.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.995+0000 7f2df516b700 1 -- 172.21.15.7:0/750387806 shutdown_connections 2023-03-03T10:05:28.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:05:28.995+0000 7f2df516b700 1 -- 172.21.15.7:0/750387806 wait complete. 2023-03-03T10:05:28.998 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:05:29.008 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2023-03-03T10:05:54.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:05:54.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:24.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:24.760 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:37.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:06:37.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.147+0000 7fda6b3fc700 1 Processor -- start 2023-03-03T10:06:37.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.147+0000 7fda6b3fc700 1 -- start start 2023-03-03T10:06:37.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda6b3fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda640ab0c0 0x7fda640c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:37.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda6b3fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda6406a720 0x7fda64158150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:37.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda6b3fc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda64158690 con 0x7fda6406a2b0 2023-03-03T10:06:37.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda6b3fc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda64154530 con 0x7fda6406a720 2023-03-03T10:06:37.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda6b3fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda64154810 con 0x7fda640ab0c0 2023-03-03T10:06:37.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda68997700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fda6406a2b0 0x7fda64157a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53064/0 (socket says 172.21.15.7:53064) 2023-03-03T10:06:37.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda68997700 1 -- 172.21.15.7:0/4168136106 learned_addr learned my addr 172.21.15.7:0/4168136106 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:06:37.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137269818 0 0) 0x7fda64158690 con 0x7fda6406a2b0 2023-03-03T10:06:37.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda69198700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda640ab0c0 0x7fda640c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:37.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda69999700 1 --2- 172.21.15.7:0/4168136106 >> v2:172.21.15.143:3300/0 conn(0x7fda6406a720 0x7fda64158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:37.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda500035b0 con 0x7fda6406a2b0 2023-03-03T10:06:37.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1437132527 0 0) 0x7fda500035b0 con 0x7fda6406a2b0 2023-03-03T10:06:37.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda640ab0c0 msgr2=0x7fda640c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:06:37.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda640ab0c0 0x7fda640c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 >> v2:172.21.15.143:3300/0 conn(0x7fda6406a720 msgr2=0x7fda64158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:06:37.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 --2- 172.21.15.7:0/4168136106 >> v2:172.21.15.143:3300/0 conn(0x7fda6406a720 0x7fda64158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda64154af0 con 0x7fda6406a2b0 2023-03-03T10:06:37.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fda54001670 con 0x7fda6406a2b0 2023-03-03T10:06:37.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 >> v1:172.21.15.7:6789/0 conn(0x7fda6406a2b0 legacy=0x7fda64157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:37.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda640ab0c0 0x7fda50003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.148+0000 7fda5bfff700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 0x7fda50004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda5bfff700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda50004d90 0x7fda50005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda64154530 con 0x7fda50003ff0 2023-03-03T10:06:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda64154810 con 0x7fda50004d90 2023-03-03T10:06:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5000ba10 con 0x7fda640ab0c0 2023-03-03T10:06:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda68997700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 0x7fda50004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:37.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda69198700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda640ab0c0 0x7fda50003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:37.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda69999700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda50004d90 0x7fda50005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:37.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda68997700 1 -- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda640ab0c0 msgr2=0x7fda50003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:06:37.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda68997700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda640ab0c0 0x7fda50003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda68997700 1 -- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda50004d90 msgr2=0x7fda50005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:06:37.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda68997700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda50004d90 0x7fda50005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda68997700 1 -- 172.21.15.7:0/4168136106 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500035b0 con 0x7fda50003ff0 2023-03-03T10:06:37.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda68997700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 0x7fda50004850 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fda5400c350 tx=0x7fda50005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b123eefb17b04872 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:37.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5400ce50 con 0x7fda50003ff0 2023-03-03T10:06:37.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda54005960 con 0x7fda50003ff0 2023-03-03T10:06:37.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.149+0000 7fda5bfff700 1 -- 172.21.15.7:0/4168136106 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda54005d10 con 0x7fda50003ff0 2023-03-03T10:06:37.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.150+0000 7fda6b3fc700 1 -- 172.21.15.7:0/4168136106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 msgr2=0x7fda50004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:37.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.150+0000 7fda6b3fc700 1 --2- 172.21.15.7:0/4168136106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 0x7fda50004850 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fda5400c350 tx=0x7fda50005880 comp rx=0 tx=0).stop 2023-03-03T10:06:37.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.150+0000 7fda69999700 1 -- 172.21.15.7:0/4168136106 reap_dead start 2023-03-03T10:06:37.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.150+0000 7fda6b3fc700 1 -- 172.21.15.7:0/4168136106 shutdown_connections 2023-03-03T10:06:37.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.150+0000 7fda6b3fc700 1 -- 172.21.15.7:0/4168136106 >> 172.21.15.7:0/4168136106 conn(0x7fda640a9430 msgr2=0x7fda640c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:06:37.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.150+0000 7fda6b3fc700 1 -- 172.21.15.7:0/4168136106 shutdown_connections 2023-03-03T10:06:37.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.150+0000 7fda6b3fc700 1 -- 172.21.15.7:0/4168136106 wait complete. 2023-03-03T10:06:37.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.151+0000 7fda6b3fc700 1 Processor -- start 2023-03-03T10:06:37.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.151+0000 7fda6b3fc700 1 -- start start 2023-03-03T10:06:37.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda6b3fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 0x7fda641eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:37.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda6b3fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda640711f0 0x7fda641e12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:37.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda6b3fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda641ed8f0 0x7fda6414b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:37.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda6b3fc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda641ed460 con 0x7fda50003ff0 2023-03-03T10:06:37.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda6b3fc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda641dfdf0 con 0x7fda640711f0 2023-03-03T10:06:37.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda6b3fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda640704d0 con 0x7fda641ed8f0 2023-03-03T10:06:37.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda69198700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 0x7fda641eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:37.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda69198700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 0x7fda641eafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59096/0 (socket says 172.21.15.7:59096) 2023-03-03T10:06:37.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda69198700 1 -- 172.21.15.7:0/1054622612 learned_addr learned my addr 172.21.15.7:0/1054622612 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:06:37.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda69999700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda641ed8f0 0x7fda6414b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:37.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda68997700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda640711f0 0x7fda641e12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:37.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda69198700 1 -- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda641ed8f0 msgr2=0x7fda6414b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:37.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda69198700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda641ed8f0 0x7fda6414b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda69198700 1 -- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda640711f0 msgr2=0x7fda641e12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:37.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda69198700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda640711f0 0x7fda641e12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda69198700 1 -- 172.21.15.7:0/1054622612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda641550b0 con 0x7fda50003ff0 2023-03-03T10:06:37.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.152+0000 7fda69198700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 0x7fda641eafc0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fda60011aa0 tx=0x7fda60011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:37.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.153+0000 7fda59ffb700 1 -- 172.21.15.7:0/1054622612 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda60011ee0 con 0x7fda50003ff0 2023-03-03T10:06:37.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.153+0000 7fda59ffb700 1 -- 172.21.15.7:0/1054622612 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6000f940 con 0x7fda50003ff0 2023-03-03T10:06:37.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.153+0000 7fda6b3fc700 1 -- 172.21.15.7:0/1054622612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda64154dd0 con 0x7fda50003ff0 2023-03-03T10:06:37.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.153+0000 7fda59ffb700 1 -- 172.21.15.7:0/1054622612 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6000fcf0 con 0x7fda50003ff0 2023-03-03T10:06:37.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.153+0000 7fda6b3fc700 1 -- 172.21.15.7:0/1054622612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda640ab490 con 0x7fda50003ff0 2023-03-03T10:06:37.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.154+0000 7fda6b3fc700 1 -- 172.21.15.7:0/1054622612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda640702a0 con 0x7fda50003ff0 2023-03-03T10:06:37.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.154+0000 7fda59ffb700 1 -- 172.21.15.7:0/1054622612 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda6001d400 con 0x7fda50003ff0 2023-03-03T10:06:37.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.155+0000 7fda59ffb700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda480682f0 0x7fda4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:37.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.155+0000 7fda59ffb700 1 -- 172.21.15.7:0/1054622612 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fda60086840 con 0x7fda50003ff0 2023-03-03T10:06:37.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.155+0000 7fda68997700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda480682f0 0x7fda4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:37.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.155+0000 7fda68997700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda480682f0 0x7fda4806a7a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fda54008fd0 tx=0x7fda5400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:37.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.159+0000 7fda59ffb700 1 -- 172.21.15.7:0/1054622612 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda60052c40 con 0x7fda50003ff0 2023-03-03T10:06:37.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.401+0000 7fda6b3fc700 1 -- 172.21.15.7:0/1054622612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda641ed740 con 0x7fda50003ff0 2023-03-03T10:06:37.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.401+0000 7fda59ffb700 1 -- 172.21.15.7:0/1054622612 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fda60056260 con 0x7fda50003ff0 2023-03-03T10:06:37.403 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:06:37.403 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:06:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.403+0000 7fda6b3fc700 1 -- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda480682f0 msgr2=0x7fda4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.403+0000 7fda6b3fc700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda480682f0 0x7fda4806a7a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fda54008fd0 tx=0x7fda5400a000 comp rx=0 tx=0).stop 2023-03-03T10:06:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.403+0000 7fda6b3fc700 1 -- 172.21.15.7:0/1054622612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 msgr2=0x7fda641eafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.403+0000 7fda6b3fc700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 0x7fda641eafc0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fda60011aa0 tx=0x7fda60011db0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.403+0000 7fda6b3fc700 1 -- 172.21.15.7:0/1054622612 shutdown_connections 2023-03-03T10:06:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.403+0000 7fda6b3fc700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda641ed8f0 0x7fda6414b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.403+0000 7fda6b3fc700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda480682f0 0x7fda4806a7a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.403+0000 7fda6b3fc700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda50003ff0 0x7fda641eafc0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.403+0000 7fda6b3fc700 1 --2- 172.21.15.7:0/1054622612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda640711f0 0x7fda641e12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.403+0000 7fda6b3fc700 1 -- 172.21.15.7:0/1054622612 >> 172.21.15.7:0/1054622612 conn(0x7fda640a9430 msgr2=0x7fda640c6790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:06:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.404+0000 7fda6b3fc700 1 -- 172.21.15.7:0/1054622612 shutdown_connections 2023-03-03T10:06:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:37.404+0000 7fda6b3fc700 1 -- 172.21.15.7:0/1054622612 wait complete. 2023-03-03T10:06:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:06:37.417 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2023-03-03T10:06:37.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:06:38.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.102+0000 7f1802554700 1 Processor -- start 2023-03-03T10:06:38.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.102+0000 7f1802554700 1 -- start start 2023-03-03T10:06:38.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f1802554700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc06d870 0x7f17fc1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f1802554700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f17fc0c1da0 0x7f17fc0c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f1802554700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17fc0c3620 con 0x7f17fc0c3e30 2023-03-03T10:06:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f1802554700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17fc0c2af0 con 0x7f17fc0c1da0 2023-03-03T10:06:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f1802554700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc1545e0 con 0x7f17fc06d870 2023-03-03T10:06:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f17fb7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f17fc0c1da0 0x7f17fc0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:38.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f17fbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc06d870 0x7f17fc1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f1800af1700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f17fc0c3e30 0x7f17fc157ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:56608/0 (socket says 172.21.15.7:56608) 2023-03-03T10:06:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f1800af1700 1 -- 172.21.15.7:0/4293113178 learned_addr learned my addr 172.21.15.7:0/4293113178 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:06:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f17faffd700 1 -- 172.21.15.7:0/4293113178 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930189510 0 0) 0x7f17fc0c3620 con 0x7f17fc0c3e30 2023-03-03T10:06:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.103+0000 7f17faffd700 1 -- 172.21.15.7:0/4293113178 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e40035b0 con 0x7f17fc0c3e30 2023-03-03T10:06:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.104+0000 7f17fbfff700 1 -- 172.21.15.7:0/4293113178 >> v2:172.21.15.143:3300/0 conn(0x7f17fc0c1da0 msgr2=0x7f17fc0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:06:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.104+0000 7f17fbfff700 1 --2- 172.21.15.7:0/4293113178 >> v2:172.21.15.143:3300/0 conn(0x7f17fc0c1da0 0x7f17fc0c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.104+0000 7f17fbfff700 1 -- 172.21.15.7:0/4293113178 >> v1:172.21.15.143:6789/0 conn(0x7f17fc0c3e30 legacy=0x7f17fc157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:38.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.104+0000 7f17fbfff700 1 -- 172.21.15.7:0/4293113178 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc1548c0 con 0x7f17fc06d870 2023-03-03T10:06:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.104+0000 7f17fb7fe700 1 --2- 172.21.15.7:0/4293113178 >> v2:172.21.15.143:3300/0 conn(0x7f17fc0c1da0 0x7f17fc0c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:06:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.104+0000 7f17fbfff700 1 --2- 172.21.15.7:0/4293113178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc06d870 0x7f17fc1550c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17f0009840 tx=0x7f17f0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79a98339e92861ce server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.104+0000 7f17faffd700 1 -- 172.21.15.7:0/4293113178 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0011830 con 0x7f17fc06d870 2023-03-03T10:06:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.104+0000 7f17faffd700 1 -- 172.21.15.7:0/4293113178 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f0011990 con 0x7f17fc06d870 2023-03-03T10:06:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.104+0000 7f17faffd700 1 -- 172.21.15.7:0/4293113178 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0011d20 con 0x7f17fc06d870 2023-03-03T10:06:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.105+0000 7f1802554700 1 -- 172.21.15.7:0/4293113178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc06d870 msgr2=0x7f17fc1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:38.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.105+0000 7f1802554700 1 --2- 172.21.15.7:0/4293113178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc06d870 0x7f17fc1550c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17f0009840 tx=0x7f17f0009b50 comp rx=0 tx=0).stop 2023-03-03T10:06:38.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.105+0000 7f1802554700 1 -- 172.21.15.7:0/4293113178 shutdown_connections 2023-03-03T10:06:38.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.105+0000 7f1802554700 1 --2- 172.21.15.7:0/4293113178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc06d870 0x7f17fc1550c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:38.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.105+0000 7f1802554700 1 --2- 172.21.15.7:0/4293113178 >> v2:172.21.15.143:3300/0 conn(0x7f17fc0c1da0 0x7f17fc0c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:38.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.105+0000 7f1802554700 1 -- 172.21.15.7:0/4293113178 >> 172.21.15.7:0/4293113178 conn(0x7f17fc06c110 msgr2=0x7f17fc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:06:38.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.105+0000 7f1802554700 1 -- 172.21.15.7:0/4293113178 shutdown_connections 2023-03-03T10:06:38.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.105+0000 7f1802554700 1 -- 172.21.15.7:0/4293113178 wait complete. 2023-03-03T10:06:38.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.106+0000 7f1802554700 1 Processor -- start 2023-03-03T10:06:38.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.107+0000 7f1802554700 1 -- start start 2023-03-03T10:06:38.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.107+0000 7f1802554700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17fc06d870 0x7f17fc0c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:38.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.107+0000 7f1802554700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17fc0c1da0 0x7f17fc1ed420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:38.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.107+0000 7f1802554700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc0c3e30 0x7f17fc1ed960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:38.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.107+0000 7f1802554700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc0ac9f0 con 0x7f17fc06d870 2023-03-03T10:06:38.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.107+0000 7f1802554700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc151b10 con 0x7f17fc0c1da0 2023-03-03T10:06:38.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.107+0000 7f1802554700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc147030 con 0x7f17fc0c3e30 2023-03-03T10:06:38.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.107+0000 7f17fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17fc06d870 0x7f17fc0c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:38.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.107+0000 7f17fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17fc06d870 0x7f17fc0c0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59110/0 (socket says 172.21.15.7:59110) 2023-03-03T10:06:38.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17fbfff700 1 -- 172.21.15.7:0/2131909841 learned_addr learned my addr 172.21.15.7:0/2131909841 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:06:38.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17fb7fe700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17fc0c1da0 0x7f17fc1ed420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:38.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f1800af1700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc0c3e30 0x7f17fc1ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:38.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17fbfff700 1 -- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc0c3e30 msgr2=0x7f17fc1ed960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:38.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17fbfff700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc0c3e30 0x7f17fc1ed960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:38.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17fbfff700 1 -- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17fc0c1da0 msgr2=0x7f17fc1ed420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:38.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17fbfff700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17fc0c1da0 0x7f17fc1ed420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:38.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17fbfff700 1 -- 172.21.15.7:0/2131909841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec0094a0 con 0x7f17fc06d870 2023-03-03T10:06:38.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17fbfff700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17fc06d870 0x7f17fc0c0a40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f17f00090b0 tx=0x7f17f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:38.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17f8ff9700 1 -- 172.21.15.7:0/2131909841 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0004590 con 0x7f17fc06d870 2023-03-03T10:06:38.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17f8ff9700 1 -- 172.21.15.7:0/2131909841 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f001e070 con 0x7f17fc06d870 2023-03-03T10:06:38.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f17f8ff9700 1 -- 172.21.15.7:0/2131909841 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f00038e0 con 0x7f17fc06d870 2023-03-03T10:06:38.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f1802554700 1 -- 172.21.15.7:0/2131909841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f00094a0 con 0x7f17fc06d870 2023-03-03T10:06:38.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.108+0000 7f1802554700 1 -- 172.21.15.7:0/2131909841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17fc06dc40 con 0x7f17fc06d870 2023-03-03T10:06:38.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.109+0000 7f1802554700 1 -- 172.21.15.7:0/2131909841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17fc154ba0 con 0x7f17fc06d870 2023-03-03T10:06:38.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.110+0000 7f17f8ff9700 1 -- 172.21.15.7:0/2131909841 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f17f000f050 con 0x7f17fc06d870 2023-03-03T10:06:38.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.110+0000 7f17f8ff9700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17e80682f0 0x7f17e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:06:38.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.110+0000 7f17f8ff9700 1 -- 172.21.15.7:0/2131909841 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f17f0086ef0 con 0x7f17fc06d870 2023-03-03T10:06:38.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.110+0000 7f17fb7fe700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17e80682f0 0x7f17e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:06:38.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.111+0000 7f17fb7fe700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17e80682f0 0x7f17e806a7a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f17fc1479f0 tx=0x7f17ec009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:06:38.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.114+0000 7f17f8ff9700 1 -- 172.21.15.7:0/2131909841 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f17f0053290 con 0x7f17fc06d870 2023-03-03T10:06:38.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.352+0000 7f1802554700 1 -- 172.21.15.7:0/2131909841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17fc0c38e0 con 0x7f17fc06d870 2023-03-03T10:06:38.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.353+0000 7f17f8ff9700 1 -- 172.21.15.7:0/2131909841 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f17f0015090 con 0x7f17fc06d870 2023-03-03T10:06:38.354 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:06:38.354 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:06:38.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 -- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17e80682f0 msgr2=0x7f17e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:38.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17e80682f0 0x7f17e806a7a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f17fc1479f0 tx=0x7f17ec009210 comp rx=0 tx=0).stop 2023-03-03T10:06:38.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 -- 172.21.15.7:0/2131909841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17fc06d870 msgr2=0x7f17fc0c0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:06:38.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17fc06d870 0x7f17fc0c0a40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f17f00090b0 tx=0x7f17f0004750 comp rx=0 tx=0).stop 2023-03-03T10:06:38.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 -- 172.21.15.7:0/2131909841 shutdown_connections 2023-03-03T10:06:38.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17fc0c3e30 0x7f17fc1ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:38.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17e80682f0 0x7f17e806a7a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:38.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17fc06d870 0x7f17fc0c0a40 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:38.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 --2- 172.21.15.7:0/2131909841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17fc0c1da0 0x7f17fc1ed420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:06:38.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 -- 172.21.15.7:0/2131909841 >> 172.21.15.7:0/2131909841 conn(0x7f17fc06c110 msgr2=0x7f17fc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:06:38.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 -- 172.21.15.7:0/2131909841 shutdown_connections 2023-03-03T10:06:38.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:06:38.355+0000 7f1802554700 1 -- 172.21.15.7:0/2131909841 wait complete. 2023-03-03T10:06:38.358 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:06:38.368 INFO:tasks.mds_thrash.fs.[b]:waiting for 42 secs before thrashing 2023-03-03T10:06:54.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:06:54.831 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:20.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:07:20.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:07:20.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.512+0000 7f89aa6d3700 1 Processor -- start 2023-03-03T10:07:20.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.512+0000 7f89aa6d3700 1 -- start start 2023-03-03T10:07:20.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.512+0000 7f89aa6d3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89a4151e80 0x7f89a40c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.512+0000 7f89aa6d3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a40c6fe0 0x7f89a40bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.512+0000 7f89aa6d3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a40c08d0 con 0x7f89a4155410 2023-03-03T10:07:20.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.512+0000 7f89aa6d3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a4152250 con 0x7f89a4151e80 2023-03-03T10:07:20.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.512+0000 7f89aa6d3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40c0fb0 con 0x7f89a40c6fe0 2023-03-03T10:07:20.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89a4151e80 0x7f89a40c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a37fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f89a4155410 0x7f89a40bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53136/0 (socket says 172.21.15.7:53136) 2023-03-03T10:07:20.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a37fe700 1 -- 172.21.15.7:0/806042072 learned_addr learned my addr 172.21.15.7:0/806042072 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:07:20.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a8c70700 1 --2- 172.21.15.7:0/806042072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a40c6fe0 0x7f89a40bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a3fff700 1 -- 172.21.15.7:0/806042072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a40c6fe0 msgr2=0x7f89a40bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:07:20.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a3fff700 1 --2- 172.21.15.7:0/806042072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a40c6fe0 0x7f89a40bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a3fff700 1 -- 172.21.15.7:0/806042072 >> v1:172.21.15.7:6789/0 conn(0x7f89a4155410 legacy=0x7f89a40bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a3fff700 1 -- 172.21.15.7:0/806042072 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4154930 con 0x7f89a4151e80 2023-03-03T10:07:20.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a2ffd700 1 -- 172.21.15.7:0/806042072 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122494152 0 0) 0x7f89a40c08d0 con 0x7f89a4155410 2023-03-03T10:07:20.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a8c70700 1 --2- 172.21.15.7:0/806042072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a40c6fe0 0x7f89a40bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:07:20.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a3fff700 1 --2- 172.21.15.7:0/806042072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 0x7f89a40c6aa0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8994009840 tx=0x7f8994009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df986f70a31a7f82 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:07:20.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a2ffd700 1 -- 172.21.15.7:0/806042072 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994011830 con 0x7f89a4151e80 2023-03-03T10:07:20.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a2ffd700 1 -- 172.21.15.7:0/806042072 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8994011990 con 0x7f89a4151e80 2023-03-03T10:07:20.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.513+0000 7f89a2ffd700 1 -- 172.21.15.7:0/806042072 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994011d20 con 0x7f89a4151e80 2023-03-03T10:07:20.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.514+0000 7f89aa6d3700 1 -- 172.21.15.7:0/806042072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 msgr2=0x7f89a40c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.514+0000 7f89aa6d3700 1 --2- 172.21.15.7:0/806042072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 0x7f89a40c6aa0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8994009840 tx=0x7f8994009b50 comp rx=0 tx=0).stop 2023-03-03T10:07:20.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.514+0000 7f89aa6d3700 1 -- 172.21.15.7:0/806042072 shutdown_connections 2023-03-03T10:07:20.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.514+0000 7f89aa6d3700 1 --2- 172.21.15.7:0/806042072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a40c6fe0 0x7f89a40bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.514+0000 7f89aa6d3700 1 --2- 172.21.15.7:0/806042072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 0x7f89a40c6aa0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.514+0000 7f89aa6d3700 1 -- 172.21.15.7:0/806042072 >> 172.21.15.7:0/806042072 conn(0x7f89a4071630 msgr2=0x7f89a406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:07:20.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.514+0000 7f89aa6d3700 1 -- 172.21.15.7:0/806042072 shutdown_connections 2023-03-03T10:07:20.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.514+0000 7f89aa6d3700 1 -- 172.21.15.7:0/806042072 wait complete. 2023-03-03T10:07:20.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.515+0000 7f89aa6d3700 1 Processor -- start 2023-03-03T10:07:20.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.515+0000 7f89aa6d3700 1 -- start start 2023-03-03T10:07:20.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89aa6d3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89a40c6fe0 0x7f89a414caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89aa6d3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 0x7f89a4150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89aa6d3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a4155410 0x7f89a4156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89aa6d3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a4158420 con 0x7f89a4151e80 2023-03-03T10:07:20.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89aa6d3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40c30b0 con 0x7f89a40c6fe0 2023-03-03T10:07:20.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89aa6d3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40c8ad0 con 0x7f89a4155410 2023-03-03T10:07:20.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89a37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 0x7f89a4150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89a37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 0x7f89a4150100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59162/0 (socket says 172.21.15.7:59162) 2023-03-03T10:07:20.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89a37fe700 1 -- 172.21.15.7:0/3732939825 learned_addr learned my addr 172.21.15.7:0/3732939825 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:07:20.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89a3fff700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89a40c6fe0 0x7f89a414caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.516+0000 7f89a8c70700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a4155410 0x7f89a4156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89a37fe700 1 -- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a4155410 msgr2=0x7f89a4156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89a37fe700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a4155410 0x7f89a4156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89a37fe700 1 -- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89a40c6fe0 msgr2=0x7f89a414caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89a37fe700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89a40c6fe0 0x7f89a414caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89a37fe700 1 -- 172.21.15.7:0/3732939825 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89940094a0 con 0x7f89a4151e80 2023-03-03T10:07:20.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89a8c70700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a4155410 0x7f89a4156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:07:20.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89a37fe700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 0x7f89a4150100 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f899800c520 tx=0x7f899800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:07:20.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89a0ff9700 1 -- 172.21.15.7:0/3732939825 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998004c80 con 0x7f89a4151e80 2023-03-03T10:07:20.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89a0ff9700 1 -- 172.21.15.7:0/3732939825 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8998004de0 con 0x7f89a4151e80 2023-03-03T10:07:20.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89aa6d3700 1 -- 172.21.15.7:0/3732939825 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a4154ef0 con 0x7f89a4151e80 2023-03-03T10:07:20.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89a0ff9700 1 -- 172.21.15.7:0/3732939825 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8998007310 con 0x7f89a4151e80 2023-03-03T10:07:20.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.517+0000 7f89aa6d3700 1 -- 172.21.15.7:0/3732939825 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a4154c10 con 0x7f89a4151e80 2023-03-03T10:07:20.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.518+0000 7f89aa6d3700 1 -- 172.21.15.7:0/3732939825 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a40c5220 con 0x7f89a4151e80 2023-03-03T10:07:20.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.519+0000 7f89a0ff9700 1 -- 172.21.15.7:0/3732939825 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f899801a020 con 0x7f89a4151e80 2023-03-03T10:07:20.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.519+0000 7f89a0ff9700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89900682f0 0x7f899006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.519+0000 7f89a0ff9700 1 -- 172.21.15.7:0/3732939825 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8998085c70 con 0x7f89a4151e80 2023-03-03T10:07:20.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.519+0000 7f89a3fff700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89900682f0 0x7f899006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.520+0000 7f89a3fff700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89900682f0 0x7f899006a7a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f89940090b0 tx=0x7f899401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:07:20.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.523+0000 7f89a0ff9700 1 -- 172.21.15.7:0/3732939825 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8998052010 con 0x7f89a4151e80 2023-03-03T10:07:20.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.552+0000 7fcca1d13700 1 Processor -- start 2023-03-03T10:07:20.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.552+0000 7fcca1d13700 1 -- start start 2023-03-03T10:07:20.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.553+0000 7fcca1d13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c155410 0x7fcc9c0bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.553+0000 7fcca1d13700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcc9c0c5b00 0x7fcc9c0bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.553+0000 7fcca1d13700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc9c0c08d0 con 0x7fcc9c151e80 2023-03-03T10:07:20.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.553+0000 7fcca1d13700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc9c152250 con 0x7fcc9c0c5b00 2023-03-03T10:07:20.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.553+0000 7fcca1d13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c0c0fb0 con 0x7fcc9c155410 2023-03-03T10:07:20.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.553+0000 7fcc9affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c155410 0x7fcc9c0bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.553+0000 7fcc9bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcc9c0c5b00 0x7fcc9c0bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.553+0000 7fcc9b7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fcc9c151e80 0x7fcc9c0c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:56668/0 (socket says 172.21.15.7:56668) 2023-03-03T10:07:20.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.553+0000 7fcc9b7fe700 1 -- 172.21.15.7:0/2721386886 learned_addr learned my addr 172.21.15.7:0/2721386886 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:07:20.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.554+0000 7fcc9affd700 1 -- 172.21.15.7:0/2721386886 >> v2:172.21.15.143:3300/0 conn(0x7fcc9c0c5b00 msgr2=0x7fcc9c0bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:07:20.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.554+0000 7fcc9affd700 1 --2- 172.21.15.7:0/2721386886 >> v2:172.21.15.143:3300/0 conn(0x7fcc9c0c5b00 0x7fcc9c0bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.554+0000 7fcc9affd700 1 -- 172.21.15.7:0/2721386886 >> v1:172.21.15.143:6789/0 conn(0x7fcc9c151e80 legacy=0x7fcc9c0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.554+0000 7fcc9affd700 1 -- 172.21.15.7:0/2721386886 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c154930 con 0x7fcc9c155410 2023-03-03T10:07:20.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.554+0000 7fcc9a7fc700 1 -- 172.21.15.7:0/2721386886 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117489684 0 0) 0x7fcc9c0c08d0 con 0x7fcc9c151e80 2023-03-03T10:07:20.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.554+0000 7fcc9affd700 1 --2- 172.21.15.7:0/2721386886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c155410 0x7fcc9c0bf8e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009840 tx=0x7fcc90009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd4f59345398760d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:07:20.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.554+0000 7fcc9a7fc700 1 -- 172.21.15.7:0/2721386886 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc90011830 con 0x7fcc9c155410 2023-03-03T10:07:20.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.554+0000 7fcc9a7fc700 1 -- 172.21.15.7:0/2721386886 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc90011990 con 0x7fcc9c155410 2023-03-03T10:07:20.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.554+0000 7fcc9a7fc700 1 -- 172.21.15.7:0/2721386886 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc90011d20 con 0x7fcc9c155410 2023-03-03T10:07:20.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.555+0000 7fcca1d13700 1 -- 172.21.15.7:0/2721386886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c155410 msgr2=0x7fcc9c0bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.555+0000 7fcca1d13700 1 --2- 172.21.15.7:0/2721386886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c155410 0x7fcc9c0bf8e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009840 tx=0x7fcc90009b50 comp rx=0 tx=0).stop 2023-03-03T10:07:20.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.555+0000 7fcca1d13700 1 -- 172.21.15.7:0/2721386886 shutdown_connections 2023-03-03T10:07:20.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.555+0000 7fcca1d13700 1 --2- 172.21.15.7:0/2721386886 >> v2:172.21.15.143:3300/0 conn(0x7fcc9c0c5b00 0x7fcc9c0bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.555+0000 7fcca1d13700 1 --2- 172.21.15.7:0/2721386886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c155410 0x7fcc9c0bf8e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.555+0000 7fcca1d13700 1 -- 172.21.15.7:0/2721386886 >> 172.21.15.7:0/2721386886 conn(0x7fcc9c071630 msgr2=0x7fcc9c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:07:20.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.555+0000 7fcca1d13700 1 -- 172.21.15.7:0/2721386886 shutdown_connections 2023-03-03T10:07:20.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.555+0000 7fcca1d13700 1 -- 172.21.15.7:0/2721386886 wait complete. 2023-03-03T10:07:20.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.556+0000 7fcca1d13700 1 Processor -- start 2023-03-03T10:07:20.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.556+0000 7fcca1d13700 1 -- start start 2023-03-03T10:07:20.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcca1d13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c0c5b00 0x7fcc9c14a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcca1d13700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc9c151e80 0x7fcc9c0c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcca1d13700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc9c155410 0x7fcc9c1ddf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcca1d13700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c1df290 con 0x7fcc9c155410 2023-03-03T10:07:20.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcca1d13700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c0c9660 con 0x7fcc9c151e80 2023-03-03T10:07:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcca1d13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c1e0620 con 0x7fcc9c0c5b00 2023-03-03T10:07:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc9c155410 0x7fcc9c1ddf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc9c155410 0x7fcc9c1ddf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59178/0 (socket says 172.21.15.7:59178) 2023-03-03T10:07:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9bfff700 1 -- 172.21.15.7:0/694681515 learned_addr learned my addr 172.21.15.7:0/694681515 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:07:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9affd700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc9c151e80 0x7fcc9c0c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9b7fe700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c0c5b00 0x7fcc9c14a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9bfff700 1 -- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c0c5b00 msgr2=0x7fcc9c14a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9bfff700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c0c5b00 0x7fcc9c14a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9bfff700 1 -- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc9c151e80 msgr2=0x7fcc9c0c8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9bfff700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc9c151e80 0x7fcc9c0c8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9bfff700 1 -- 172.21.15.7:0/694681515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc900094a0 con 0x7fcc9c155410 2023-03-03T10:07:20.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc9bfff700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc9c155410 0x7fcc9c1ddf90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400efc0 tx=0x7fcc9400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:07:20.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.557+0000 7fcc83fff700 1 -- 172.21.15.7:0/694681515 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9400cad0 con 0x7fcc9c155410 2023-03-03T10:07:20.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.558+0000 7fcc83fff700 1 -- 172.21.15.7:0/694681515 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9400f040 con 0x7fcc9c155410 2023-03-03T10:07:20.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.558+0000 7fcca1d13700 1 -- 172.21.15.7:0/694681515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9c154ef0 con 0x7fcc9c155410 2023-03-03T10:07:20.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.558+0000 7fcc83fff700 1 -- 172.21.15.7:0/694681515 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9400ce70 con 0x7fcc9c155410 2023-03-03T10:07:20.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.558+0000 7fcca1d13700 1 -- 172.21.15.7:0/694681515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9c154c10 con 0x7fcc9c155410 2023-03-03T10:07:20.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.559+0000 7fcca1d13700 1 -- 172.21.15.7:0/694681515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9c0c5220 con 0x7fcc9c155410 2023-03-03T10:07:20.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.559+0000 7fcc83fff700 1 -- 172.21.15.7:0/694681515 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcc94014480 con 0x7fcc9c155410 2023-03-03T10:07:20.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.560+0000 7fcc83fff700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc88068320 0x7fcc8806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:07:20.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.560+0000 7fcc83fff700 1 -- 172.21.15.7:0/694681515 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcc94087180 con 0x7fcc9c155410 2023-03-03T10:07:20.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.560+0000 7fcc9b7fe700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc88068320 0x7fcc8806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:07:20.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.560+0000 7fcc9b7fe700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc88068320 0x7fcc8806a7d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c007fd0 tx=0x7fcc8c007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:07:20.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.563+0000 7fcc83fff700 1 -- 172.21.15.7:0/694681515 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcc940534c0 con 0x7fcc9c155410 2023-03-03T10:07:20.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.763+0000 7f89aa6d3700 1 -- 172.21.15.7:0/3732939825 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89a4074bb0 con 0x7f89a4151e80 2023-03-03T10:07:20.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.763+0000 7f89a0ff9700 1 -- 172.21.15.7:0/3732939825 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8998055630 con 0x7f89a4151e80 2023-03-03T10:07:20.765 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:07:20.765 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:07:20.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 -- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89900682f0 msgr2=0x7f899006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89900682f0 0x7f899006a7a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f89940090b0 tx=0x7f899401c040 comp rx=0 tx=0).stop 2023-03-03T10:07:20.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 -- 172.21.15.7:0/3732939825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 msgr2=0x7f89a4150100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 0x7f89a4150100 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f899800c520 tx=0x7f899800c830 comp rx=0 tx=0).stop 2023-03-03T10:07:20.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 -- 172.21.15.7:0/3732939825 shutdown_connections 2023-03-03T10:07:20.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89a4155410 0x7f89a4156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89a40c6fe0 0x7f89a414caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89900682f0 0x7f899006a7a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 --2- 172.21.15.7:0/3732939825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89a4151e80 0x7f89a4150100 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 -- 172.21.15.7:0/3732939825 >> 172.21.15.7:0/3732939825 conn(0x7f89a4071630 msgr2=0x7f89a40bf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:07:20.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 -- 172.21.15.7:0/3732939825 shutdown_connections 2023-03-03T10:07:20.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.765+0000 7f89aa6d3700 1 -- 172.21.15.7:0/3732939825 wait complete. 2023-03-03T10:07:20.769 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:07:20.779 INFO:tasks.mds_thrash.fs.[b]:waiting for 115 secs before thrashing 2023-03-03T10:07:20.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.803+0000 7fcca1d13700 1 -- 172.21.15.7:0/694681515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc9c0c97d0 con 0x7fcc9c155410 2023-03-03T10:07:20.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.804+0000 7fcc83fff700 1 -- 172.21.15.7:0/694681515 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcc94056ae0 con 0x7fcc9c155410 2023-03-03T10:07:20.805 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:07:20.806 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:07:20.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 -- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc88068320 msgr2=0x7fcc8806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc88068320 0x7fcc8806a7d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c007fd0 tx=0x7fcc8c007de0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 -- 172.21.15.7:0/694681515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc9c155410 msgr2=0x7fcc9c1ddf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:07:20.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc9c155410 0x7fcc9c1ddf90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400efc0 tx=0x7fcc9400c0c0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 -- 172.21.15.7:0/694681515 shutdown_connections 2023-03-03T10:07:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc88068320 0x7fcc8806a7d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc9c155410 0x7fcc9c1ddf90 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc9c0c5b00 0x7fcc9c14a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 --2- 172.21.15.7:0/694681515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc9c151e80 0x7fcc9c0c8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:07:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 -- 172.21.15.7:0/694681515 >> 172.21.15.7:0/694681515 conn(0x7fcc9c071630 msgr2=0x7fcc9c0c3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:07:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 -- 172.21.15.7:0/694681515 shutdown_connections 2023-03-03T10:07:20.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:07:20.806+0000 7fcca1d13700 1 -- 172.21.15.7:0/694681515 wait complete. 2023-03-03T10:07:20.810 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:07:20.820 INFO:tasks.mds_thrash.fs.[a]:waiting for 101 secs before thrashing 2023-03-03T10:07:25.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:25.167 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:55.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:07:55.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:25.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:25.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:55.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:08:55.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:01.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:09:01.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.956+0000 7f3b0c352700 1 Processor -- start 2023-03-03T10:09:01.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.956+0000 7f3b0c352700 1 -- start start 2023-03-03T10:09:01.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.957+0000 7f3b0c352700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04155410 0x7f3b040bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.957+0000 7f3b0c352700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b040c08d0 con 0x7f3b040c5b00 2023-03-03T10:09:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.957+0000 7f3b0c352700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b04154930 con 0x7f3b04151e80 2023-03-03T10:09:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.957+0000 7f3b0c352700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b04152250 con 0x7f3b04155410 2023-03-03T10:09:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.957+0000 7f3b0a8ef700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3b040c5b00 0x7f3b040bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53246/0 (socket says 172.21.15.7:53246) 2023-03-03T10:09:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.957+0000 7f3b0a8ef700 1 -- 172.21.15.7:0/4096735891 learned_addr learned my addr 172.21.15.7:0/4096735891 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:09:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b098ed700 1 --2- 172.21.15.7:0/4096735891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04155410 0x7f3b040bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b090ec700 1 -- 172.21.15.7:0/4096735891 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268143016 0 0) 0x7f3b040c08d0 con 0x7f3b040c5b00 2023-03-03T10:09:01.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b090ec700 1 -- 172.21.15.7:0/4096735891 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af00035b0 con 0x7f3b040c5b00 2023-03-03T10:09:01.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b090ec700 1 -- 172.21.15.7:0/4096735891 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846024027 0 0) 0x7f3b04154930 con 0x7f3b04151e80 2023-03-03T10:09:01.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b090ec700 1 -- 172.21.15.7:0/4096735891 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b040c08d0 con 0x7f3b04151e80 2023-03-03T10:09:01.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b098ed700 1 -- 172.21.15.7:0/4096735891 >> v1:172.21.15.143:6789/0 conn(0x7f3b04151e80 legacy=0x7f3b040c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:01.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b098ed700 1 -- 172.21.15.7:0/4096735891 >> v1:172.21.15.7:6789/0 conn(0x7f3b040c5b00 legacy=0x7f3b040bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:01.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b098ed700 1 -- 172.21.15.7:0/4096735891 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b04154c10 con 0x7f3b04155410 2023-03-03T10:09:01.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b098ed700 1 --2- 172.21.15.7:0/4096735891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04155410 0x7f3b040bf8e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009870 tx=0x7f3af4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eae2605bbcdd9ec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b090ec700 1 -- 172.21.15.7:0/4096735891 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4011830 con 0x7f3b04155410 2023-03-03T10:09:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b090ec700 1 -- 172.21.15.7:0/4096735891 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af4011990 con 0x7f3b04155410 2023-03-03T10:09:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.958+0000 7f3b090ec700 1 -- 172.21.15.7:0/4096735891 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af4011d20 con 0x7f3b04155410 2023-03-03T10:09:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.959+0000 7f3b0c352700 1 -- 172.21.15.7:0/4096735891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04155410 msgr2=0x7f3b040bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.959+0000 7f3b0c352700 1 --2- 172.21.15.7:0/4096735891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04155410 0x7f3b040bf8e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009870 tx=0x7f3af4009b80 comp rx=0 tx=0).stop 2023-03-03T10:09:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.959+0000 7f3b0c352700 1 -- 172.21.15.7:0/4096735891 shutdown_connections 2023-03-03T10:09:01.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.959+0000 7f3b0c352700 1 --2- 172.21.15.7:0/4096735891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04155410 0x7f3b040bf8e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.959+0000 7f3b0c352700 1 -- 172.21.15.7:0/4096735891 >> 172.21.15.7:0/4096735891 conn(0x7f3b04071630 msgr2=0x7f3b0406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:09:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.959+0000 7f3b0c352700 1 -- 172.21.15.7:0/4096735891 shutdown_connections 2023-03-03T10:09:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.959+0000 7f3b0c352700 1 -- 172.21.15.7:0/4096735891 wait complete. 2023-03-03T10:09:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.960+0000 7f3b0c352700 1 Processor -- start 2023-03-03T10:09:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.960+0000 7f3b0c352700 1 -- start start 2023-03-03T10:09:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0c352700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b040c5b00 0x7f3b041e11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:01.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0c352700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04151e80 0x7f3b040c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0c352700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b04155410 0x7f3b041edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0c352700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b041eb4e0 con 0x7f3b040c5b00 2023-03-03T10:09:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0c352700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b041eb000 con 0x7f3b04155410 2023-03-03T10:09:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0c352700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b041587c0 con 0x7f3b04151e80 2023-03-03T10:09:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0a0ee700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b040c5b00 0x7f3b041e11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0a0ee700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b040c5b00 0x7f3b041e11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59268/0 (socket says 172.21.15.7:59268) 2023-03-03T10:09:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0a0ee700 1 -- 172.21.15.7:0/1717445023 learned_addr learned my addr 172.21.15.7:0/1717445023 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:09:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0a8ef700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b04155410 0x7f3b041edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b098ed700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04151e80 0x7f3b040c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0a0ee700 1 -- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04151e80 msgr2=0x7f3b040c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0a0ee700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04151e80 0x7f3b040c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0a0ee700 1 -- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b04155410 msgr2=0x7f3b041edb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0a0ee700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b04155410 0x7f3b041edb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0a0ee700 1 -- 172.21.15.7:0/1717445023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40094a0 con 0x7f3b040c5b00 2023-03-03T10:09:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.961+0000 7f3b0a8ef700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b04155410 0x7f3b041edb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:09:01.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.962+0000 7f3b0a0ee700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b040c5b00 0x7f3b041e11d0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000c500 tx=0x7f3b0000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.962+0000 7f3afaffd700 1 -- 172.21.15.7:0/1717445023 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b000052d0 con 0x7f3b040c5b00 2023-03-03T10:09:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.962+0000 7f3afaffd700 1 -- 172.21.15.7:0/1717445023 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b00005430 con 0x7f3b040c5b00 2023-03-03T10:09:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.962+0000 7f3b0c352700 1 -- 172.21.15.7:0/1717445023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b040c5220 con 0x7f3b040c5b00 2023-03-03T10:09:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.962+0000 7f3afaffd700 1 -- 172.21.15.7:0/1717445023 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b000038c0 con 0x7f3b040c5b00 2023-03-03T10:09:01.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.962+0000 7f3b0c352700 1 -- 172.21.15.7:0/1717445023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b04154ef0 con 0x7f3b040c5b00 2023-03-03T10:09:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.963+0000 7f3b0c352700 1 -- 172.21.15.7:0/1717445023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b040c0530 con 0x7f3b040c5b00 2023-03-03T10:09:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.964+0000 7f3afaffd700 1 -- 172.21.15.7:0/1717445023 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3b00019030 con 0x7f3b040c5b00 2023-03-03T10:09:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.964+0000 7f3afaffd700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ae80682f0 0x7f3ae806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.964+0000 7f3afaffd700 1 -- 172.21.15.7:0/1717445023 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3b00085560 con 0x7f3b040c5b00 2023-03-03T10:09:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.964+0000 7f3b098ed700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ae80682f0 0x7f3ae806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.965+0000 7f3b098ed700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ae80682f0 0x7f3ae806a7a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3af4000c00 tx=0x7f3af4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:01.968+0000 7f3afaffd700 1 -- 172.21.15.7:0/1717445023 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3b00051900 con 0x7f3b040c5b00 2023-03-03T10:09:02.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.205+0000 7f3b0c352700 1 -- 172.21.15.7:0/1717445023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b040c3930 con 0x7f3b040c5b00 2023-03-03T10:09:02.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.206+0000 7f3afaffd700 1 -- 172.21.15.7:0/1717445023 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3b00054f20 con 0x7f3b040c5b00 2023-03-03T10:09:02.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:09:02.208 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:09:02.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 -- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ae80682f0 msgr2=0x7f3ae806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:02.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ae80682f0 0x7f3ae806a7a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3af4000c00 tx=0x7f3af4008030 comp rx=0 tx=0).stop 2023-03-03T10:09:02.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 -- 172.21.15.7:0/1717445023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b040c5b00 msgr2=0x7f3b041e11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:02.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b040c5b00 0x7f3b041e11d0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000c500 tx=0x7f3b0000c8c0 comp rx=0 tx=0).stop 2023-03-03T10:09:02.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 -- 172.21.15.7:0/1717445023 shutdown_connections 2023-03-03T10:09:02.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b04155410 0x7f3b041edb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:02.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ae80682f0 0x7f3ae806a7a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:02.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b040c5b00 0x7f3b041e11d0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:02.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 --2- 172.21.15.7:0/1717445023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b04151e80 0x7f3b040c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:02.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 -- 172.21.15.7:0/1717445023 >> 172.21.15.7:0/1717445023 conn(0x7f3b04071630 msgr2=0x7f3b041537f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:09:02.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 -- 172.21.15.7:0/1717445023 shutdown_connections 2023-03-03T10:09:02.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:02.208+0000 7f3b0c352700 1 -- 172.21.15.7:0/1717445023 wait complete. 2023-03-03T10:09:02.211 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:09:02.221 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before thrashing 2023-03-03T10:09:04.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:09:04.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.359+0000 7f3699d09700 1 Processor -- start 2023-03-03T10:09:04.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.359+0000 7f3699d09700 1 -- start start 2023-03-03T10:09:04.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f3699d09700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f369414b020 0x7f36940717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:04.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f3699d09700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36940728b0 con 0x7f3694071d20 2023-03-03T10:09:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f3699d09700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3694155070 con 0x7f36940aad20 2023-03-03T10:09:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f3699d09700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f369406f5b0 con 0x7f369414b020 2023-03-03T10:09:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f3693fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3694071d20 0x7f36941583b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53264/0 (socket says 172.21.15.7:53264) 2023-03-03T10:09:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f3693fff700 1 -- 172.21.15.7:0/2874922988 learned_addr learned my addr 172.21.15.7:0/2874922988 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:09:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f36937fe700 1 --2- 172.21.15.7:0/2874922988 >> v2:172.21.15.143:3300/0 conn(0x7f369414b020 0x7f36940717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f36927fc700 1 -- 172.21.15.7:0/2874922988 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117152900 0 0) 0x7f36940728b0 con 0x7f3694071d20 2023-03-03T10:09:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f36927fc700 1 -- 172.21.15.7:0/2874922988 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36800035b0 con 0x7f3694071d20 2023-03-03T10:09:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f36927fc700 1 -- 172.21.15.7:0/2874922988 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318325967 0 0) 0x7f3694155070 con 0x7f36940aad20 2023-03-03T10:09:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.360+0000 7f36927fc700 1 -- 172.21.15.7:0/2874922988 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36940728b0 con 0x7f36940aad20 2023-03-03T10:09:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.361+0000 7f36937fe700 1 -- 172.21.15.7:0/2874922988 >> v1:172.21.15.143:6789/0 conn(0x7f36940aad20 legacy=0x7f3694157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.361+0000 7f36937fe700 1 -- 172.21.15.7:0/2874922988 >> v1:172.21.15.7:6789/0 conn(0x7f3694071d20 legacy=0x7f36941583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.361+0000 7f36927fc700 1 -- 172.21.15.7:0/2874922988 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981056736 0 0) 0x7f36800035b0 con 0x7f3694071d20 2023-03-03T10:09:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.361+0000 7f36937fe700 1 -- 172.21.15.7:0/2874922988 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369406f890 con 0x7f369414b020 2023-03-03T10:09:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.361+0000 7f36937fe700 1 --2- 172.21.15.7:0/2874922988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f369414b020 0x7f36940717e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f367c009840 tx=0x7f367c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5b786db5112ca12 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.361+0000 7f36927fc700 1 -- 172.21.15.7:0/2874922988 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c0019f0 con 0x7f369414b020 2023-03-03T10:09:04.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.361+0000 7f36927fc700 1 -- 172.21.15.7:0/2874922988 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367c004030 con 0x7f369414b020 2023-03-03T10:09:04.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.361+0000 7f36927fc700 1 -- 172.21.15.7:0/2874922988 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c0043c0 con 0x7f369414b020 2023-03-03T10:09:04.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.361+0000 7f3699d09700 1 -- 172.21.15.7:0/2874922988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f369414b020 msgr2=0x7f36940717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:04.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.361+0000 7f3699d09700 1 --2- 172.21.15.7:0/2874922988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f369414b020 0x7f36940717e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f367c009840 tx=0x7f367c009b50 comp rx=0 tx=0).stop 2023-03-03T10:09:04.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.362+0000 7f3699d09700 1 -- 172.21.15.7:0/2874922988 shutdown_connections 2023-03-03T10:09:04.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.362+0000 7f3699d09700 1 --2- 172.21.15.7:0/2874922988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f369414b020 0x7f36940717e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:04.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.362+0000 7f3699d09700 1 -- 172.21.15.7:0/2874922988 >> 172.21.15.7:0/2874922988 conn(0x7f36940c8900 msgr2=0x7f36940c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:09:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.362+0000 7f3699d09700 1 -- 172.21.15.7:0/2874922988 shutdown_connections 2023-03-03T10:09:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.362+0000 7f3699d09700 1 -- 172.21.15.7:0/2874922988 wait complete. 2023-03-03T10:09:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.363+0000 7f3699d09700 1 Processor -- start 2023-03-03T10:09:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.363+0000 7f3699d09700 1 -- start start 2023-03-03T10:09:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3699d09700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3694071d20 0x7f36941ed2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3699d09700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940aad20 0x7f3694146940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3699d09700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f369414b020 0x7f3694146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3699d09700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36941e0440 con 0x7f36940aad20 2023-03-03T10:09:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3699d09700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36941e0a60 con 0x7f3694071d20 2023-03-03T10:09:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3699d09700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36941edf90 con 0x7f369414b020 2023-03-03T10:09:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3692ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940aad20 0x7f3694146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3692ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940aad20 0x7f3694146940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59288/0 (socket says 172.21.15.7:59288) 2023-03-03T10:09:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3692ffd700 1 -- 172.21.15.7:0/2865593613 learned_addr learned my addr 172.21.15.7:0/2865593613 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:09:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f36937fe700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3694071d20 0x7f36941ed2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3693fff700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f369414b020 0x7f3694146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3692ffd700 1 -- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f369414b020 msgr2=0x7f3694146e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3692ffd700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f369414b020 0x7f3694146e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3692ffd700 1 -- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3694071d20 msgr2=0x7f36941ed2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3692ffd700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3694071d20 0x7f36941ed2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3692ffd700 1 -- 172.21.15.7:0/2865593613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c0094a0 con 0x7f36940aad20 2023-03-03T10:09:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f36937fe700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3694071d20 0x7f36941ed2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:09:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3692ffd700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940aad20 0x7f3694146940 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f368400c4f0 tx=0x7f368400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f368bfff700 1 -- 172.21.15.7:0/2865593613 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36840052d0 con 0x7f36940aad20 2023-03-03T10:09:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f368bfff700 1 -- 172.21.15.7:0/2865593613 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3684005430 con 0x7f36940aad20 2023-03-03T10:09:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.364+0000 7f3699d09700 1 -- 172.21.15.7:0/2865593613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3694155350 con 0x7f36940aad20 2023-03-03T10:09:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.365+0000 7f368bfff700 1 -- 172.21.15.7:0/2865593613 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3684003880 con 0x7f36940aad20 2023-03-03T10:09:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.365+0000 7f3699d09700 1 -- 172.21.15.7:0/2865593613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369406fb70 con 0x7f36940aad20 2023-03-03T10:09:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.365+0000 7f3699d09700 1 -- 172.21.15.7:0/2865593613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3694074bb0 con 0x7f36940aad20 2023-03-03T10:09:04.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.366+0000 7f368bfff700 1 -- 172.21.15.7:0/2865593613 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3684019030 con 0x7f36940aad20 2023-03-03T10:09:04.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.366+0000 7f368bfff700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36740682f0 0x7f367406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:04.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.366+0000 7f368bfff700 1 -- 172.21.15.7:0/2865593613 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3684085400 con 0x7f36940aad20 2023-03-03T10:09:04.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.367+0000 7f36937fe700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36740682f0 0x7f367406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:04.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.367+0000 7f36937fe700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36740682f0 0x7f367406a7a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f367c0090b0 tx=0x7f367c013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:04.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.371+0000 7f368bfff700 1 -- 172.21.15.7:0/2865593613 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f368404b5e0 con 0x7f36940aad20 2023-03-03T10:09:04.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.612+0000 7f3699d09700 1 -- 172.21.15.7:0/2865593613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36940c8fb0 con 0x7f36940aad20 2023-03-03T10:09:04.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.612+0000 7f368bfff700 1 -- 172.21.15.7:0/2865593613 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3684051850 con 0x7f36940aad20 2023-03-03T10:09:04.614 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:09:04.614 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:09:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 -- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36740682f0 msgr2=0x7f367406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36740682f0 0x7f367406a7a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f367c0090b0 tx=0x7f367c013040 comp rx=0 tx=0).stop 2023-03-03T10:09:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 -- 172.21.15.7:0/2865593613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940aad20 msgr2=0x7f3694146940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940aad20 0x7f3694146940 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f368400c4f0 tx=0x7f368400c8b0 comp rx=0 tx=0).stop 2023-03-03T10:09:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 -- 172.21.15.7:0/2865593613 shutdown_connections 2023-03-03T10:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f369414b020 0x7f3694146e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3694071d20 0x7f36941ed2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36740682f0 0x7f367406a7a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 --2- 172.21.15.7:0/2865593613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36940aad20 0x7f3694146940 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 -- 172.21.15.7:0/2865593613 >> 172.21.15.7:0/2865593613 conn(0x7f36940c8900 msgr2=0x7f3694155e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 -- 172.21.15.7:0/2865593613 shutdown_connections 2023-03-03T10:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:04.614+0000 7f3699d09700 1 -- 172.21.15.7:0/2865593613 wait complete. 2023-03-03T10:09:04.618 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:09:04.627 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before thrashing 2023-03-03T10:09:08.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:09:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.766+0000 7fe2cfaff700 1 Processor -- start 2023-03-03T10:09:08.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.766+0000 7fe2cfaff700 1 -- start start 2023-03-03T10:09:08.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cfaff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c814fc50 0x7fe2c80c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:08.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cfaff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2c806d240 0x7fe2c80c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cfaff700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2c80ad2b0 con 0x7fe2c80c3e30 2023-03-03T10:09:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cfaff700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2c80c30c0 con 0x7fe2c806d240 2023-03-03T10:09:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cfaff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c8156420 con 0x7fe2c814fc50 2023-03-03T10:09:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cd09a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2c806d240 0x7fe2c80c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2ce09c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe2c80c3e30 0x7fe2c80c7390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53282/0 (socket says 172.21.15.7:53282) 2023-03-03T10:09:08.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2ce09c700 1 -- 172.21.15.7:0/1888237366 learned_addr learned my addr 172.21.15.7:0/1888237366 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:09:08.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cd09a700 1 -- 172.21.15.7:0/1888237366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c814fc50 msgr2=0x7fe2c80c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:09:08.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cd89b700 1 --2- 172.21.15.7:0/1888237366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c814fc50 0x7fe2c80c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:08.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cc899700 1 -- 172.21.15.7:0/1888237366 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877371862 0 0) 0x7fe2c80ad2b0 con 0x7fe2c80c3e30 2023-03-03T10:09:08.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cd09a700 1 --2- 172.21.15.7:0/1888237366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c814fc50 0x7fe2c80c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:08.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cd09a700 1 -- 172.21.15.7:0/1888237366 >> v1:172.21.15.7:6789/0 conn(0x7fe2c80c3e30 legacy=0x7fe2c80c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:08.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cd09a700 1 -- 172.21.15.7:0/1888237366 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c80ad520 con 0x7fe2c806d240 2023-03-03T10:09:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cd89b700 1 --2- 172.21.15.7:0/1888237366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c814fc50 0x7fe2c80c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:09:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.767+0000 7fe2cd09a700 1 --2- 172.21.15.7:0/1888237366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c806d240 0x7fe2c80c27b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009840 tx=0x7fe2b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9c0446254a1260 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cc899700 1 -- 172.21.15.7:0/1888237366 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8011830 con 0x7fe2c806d240 2023-03-03T10:09:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cc899700 1 -- 172.21.15.7:0/1888237366 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b8011990 con 0x7fe2c806d240 2023-03-03T10:09:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cc899700 1 -- 172.21.15.7:0/1888237366 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8011d20 con 0x7fe2c806d240 2023-03-03T10:09:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cfaff700 1 -- 172.21.15.7:0/1888237366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c806d240 msgr2=0x7fe2c80c27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:08.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cfaff700 1 --2- 172.21.15.7:0/1888237366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c806d240 0x7fe2c80c27b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8009840 tx=0x7fe2b8009b50 comp rx=0 tx=0).stop 2023-03-03T10:09:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cfaff700 1 -- 172.21.15.7:0/1888237366 shutdown_connections 2023-03-03T10:09:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cfaff700 1 --2- 172.21.15.7:0/1888237366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c814fc50 0x7fe2c80c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cfaff700 1 --2- 172.21.15.7:0/1888237366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c806d240 0x7fe2c80c27b0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cfaff700 1 -- 172.21.15.7:0/1888237366 >> 172.21.15.7:0/1888237366 conn(0x7fe2c806c110 msgr2=0x7fe2c806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:09:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cfaff700 1 -- 172.21.15.7:0/1888237366 shutdown_connections 2023-03-03T10:09:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.768+0000 7fe2cfaff700 1 -- 172.21.15.7:0/1888237366 wait complete. 2023-03-03T10:09:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.769+0000 7fe2cfaff700 1 Processor -- start 2023-03-03T10:09:08.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.769+0000 7fe2cfaff700 1 -- start start 2023-03-03T10:09:08.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2cfaff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c806d240 0x7fe2c81e5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:08.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2cfaff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c80c3e30 0x7fe2c8154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:08.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2cfaff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c814fc50 0x7fe2c80c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:08.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2cfaff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c81469d0 con 0x7fe2c80c3e30 2023-03-03T10:09:08.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2cfaff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c81e8c70 con 0x7fe2c814fc50 2023-03-03T10:09:08.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2cfaff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2c814eca0 con 0x7fe2c806d240 2023-03-03T10:09:08.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2cd09a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c80c3e30 0x7fe2c8154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:08.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2cd09a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c80c3e30 0x7fe2c8154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59306/0 (socket says 172.21.15.7:59306) 2023-03-03T10:09:08.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2cd09a700 1 -- 172.21.15.7:0/3878184183 learned_addr learned my addr 172.21.15.7:0/3878184183 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:09:08.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2cd89b700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c806d240 0x7fe2c81e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:08.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.770+0000 7fe2ce09c700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c814fc50 0x7fe2c80c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:08.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2cd09a700 1 -- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c806d240 msgr2=0x7fe2c81e5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:08.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2cd09a700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c806d240 0x7fe2c81e5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:08.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2cd09a700 1 -- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c814fc50 msgr2=0x7fe2c80c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:08.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2cd09a700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c814fc50 0x7fe2c80c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:08.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2cd09a700 1 -- 172.21.15.7:0/3878184183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b80094a0 con 0x7fe2c80c3e30 2023-03-03T10:09:08.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2cd09a700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c80c3e30 0x7fe2c8154750 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8014040 tx=0x7fe2b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2be7fc700 1 -- 172.21.15.7:0/3878184183 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b80038a0 con 0x7fe2c80c3e30 2023-03-03T10:09:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2be7fc700 1 -- 172.21.15.7:0/3878184183 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b801e070 con 0x7fe2c80c3e30 2023-03-03T10:09:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2cfaff700 1 -- 172.21.15.7:0/3878184183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c80c2cf0 con 0x7fe2c80c3e30 2023-03-03T10:09:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2be7fc700 1 -- 172.21.15.7:0/3878184183 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b8003c60 con 0x7fe2c80c3e30 2023-03-03T10:09:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.771+0000 7fe2cfaff700 1 -- 172.21.15.7:0/3878184183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c80c2490 con 0x7fe2c80c3e30 2023-03-03T10:09:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.772+0000 7fe2abfff700 1 -- 172.21.15.7:0/3878184183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a40052e0 con 0x7fe2c80c3e30 2023-03-03T10:09:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.772+0000 7fe2be7fc700 1 -- 172.21.15.7:0/3878184183 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe2b800f050 con 0x7fe2c80c3e30 2023-03-03T10:09:08.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.773+0000 7fe2be7fc700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2ac0682f0 0x7fe2ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.773+0000 7fe2be7fc700 1 -- 172.21.15.7:0/3878184183 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe2b8020070 con 0x7fe2c80c3e30 2023-03-03T10:09:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.773+0000 7fe2cd89b700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2ac0682f0 0x7fe2ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.774+0000 7fe2cd89b700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2ac0682f0 0x7fe2ac06a7a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4006fd0 tx=0x7fe2c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:08.778+0000 7fe2be7fc700 1 -- 172.21.15.7:0/3878184183 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2b8053ac0 con 0x7fe2c80c3e30 2023-03-03T10:09:09.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.016+0000 7fe2abfff700 1 -- 172.21.15.7:0/3878184183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2a4005b10 con 0x7fe2c80c3e30 2023-03-03T10:09:09.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.017+0000 7fe2be7fc700 1 -- 172.21.15.7:0/3878184183 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe2b80570e0 con 0x7fe2c80c3e30 2023-03-03T10:09:09.018 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:09:09.019 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:09:09.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 -- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2ac0682f0 msgr2=0x7fe2ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:09.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2ac0682f0 0x7fe2ac06a7a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4006fd0 tx=0x7fe2c4008040 comp rx=0 tx=0).stop 2023-03-03T10:09:09.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 -- 172.21.15.7:0/3878184183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c80c3e30 msgr2=0x7fe2c8154750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:09.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c80c3e30 0x7fe2c8154750 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe2b8014040 tx=0x7fe2b8004750 comp rx=0 tx=0).stop 2023-03-03T10:09:09.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 -- 172.21.15.7:0/3878184183 shutdown_connections 2023-03-03T10:09:09.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2c814fc50 0x7fe2c80c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:09.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2c806d240 0x7fe2c81e5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:09.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2ac0682f0 0x7fe2ac06a7a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:09.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 --2- 172.21.15.7:0/3878184183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2c80c3e30 0x7fe2c8154750 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:09.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 -- 172.21.15.7:0/3878184183 >> 172.21.15.7:0/3878184183 conn(0x7fe2c806c110 msgr2=0x7fe2c8155cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:09:09.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 -- 172.21.15.7:0/3878184183 shutdown_connections 2023-03-03T10:09:09.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:09.019+0000 7fe2abfff700 1 -- 172.21.15.7:0/3878184183 wait complete. 2023-03-03T10:09:09.022 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:09:09.032 INFO:tasks.mds_thrash.fs.[a]:waiting for 96 secs before thrashing 2023-03-03T10:09:15.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:09:15.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.916+0000 7fda59fb5700 1 Processor -- start 2023-03-03T10:09:15.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.916+0000 7fda59fb5700 1 -- start start 2023-03-03T10:09:15.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.917+0000 7fda59fb5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda54071e80 0x7fda54151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:15.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.917+0000 7fda59fb5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda5414e900 0x7fda5415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.917+0000 7fda59fb5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda541510d0 con 0x7fda54072250 2023-03-03T10:09:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.917+0000 7fda59fb5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda540c3740 con 0x7fda5414e900 2023-03-03T10:09:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.917+0000 7fda59fb5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda54070670 con 0x7fda54071e80 2023-03-03T10:09:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.917+0000 7fda53fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda5414e900 0x7fda5415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.917+0000 7fda52ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fda54072250 0x7fda5414e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53302/0 (socket says 172.21.15.7:53302) 2023-03-03T10:09:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.917+0000 7fda52ffd700 1 -- 172.21.15.7:0/2415487013 learned_addr learned my addr 172.21.15.7:0/2415487013 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:09:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.918+0000 7fda537fe700 1 --2- 172.21.15.7:0/2415487013 >> v2:172.21.15.143:3300/0 conn(0x7fda54071e80 0x7fda54151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:15.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.918+0000 7fda53fff700 1 -- 172.21.15.7:0/2415487013 >> v2:172.21.15.143:3300/0 conn(0x7fda54071e80 msgr2=0x7fda54151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:09:15.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.918+0000 7fda53fff700 1 --2- 172.21.15.7:0/2415487013 >> v2:172.21.15.143:3300/0 conn(0x7fda54071e80 0x7fda54151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:15.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.918+0000 7fda53fff700 1 -- 172.21.15.7:0/2415487013 >> v1:172.21.15.7:6789/0 conn(0x7fda54072250 legacy=0x7fda5414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:15.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.918+0000 7fda53fff700 1 -- 172.21.15.7:0/2415487013 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda540c39e0 con 0x7fda5414e900 2023-03-03T10:09:15.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.918+0000 7fda53fff700 1 --2- 172.21.15.7:0/2415487013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 0x7fda5415ba50 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fda5406bd10 tx=0x7fda4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a7355594a3e5d7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:15.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.918+0000 7fda527fc700 1 -- 172.21.15.7:0/2415487013 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c00e070 con 0x7fda5414e900 2023-03-03T10:09:15.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.918+0000 7fda527fc700 1 -- 172.21.15.7:0/2415487013 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c00bbd0 con 0x7fda5414e900 2023-03-03T10:09:15.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.918+0000 7fda527fc700 1 -- 172.21.15.7:0/2415487013 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c012430 con 0x7fda5414e900 2023-03-03T10:09:15.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.919+0000 7fda59fb5700 1 -- 172.21.15.7:0/2415487013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 msgr2=0x7fda5415ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:15.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.919+0000 7fda59fb5700 1 --2- 172.21.15.7:0/2415487013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 0x7fda5415ba50 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fda5406bd10 tx=0x7fda4c00b330 comp rx=0 tx=0).stop 2023-03-03T10:09:15.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.919+0000 7fda59fb5700 1 -- 172.21.15.7:0/2415487013 shutdown_connections 2023-03-03T10:09:15.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.919+0000 7fda59fb5700 1 --2- 172.21.15.7:0/2415487013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 0x7fda5415ba50 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:15.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.919+0000 7fda59fb5700 1 --2- 172.21.15.7:0/2415487013 >> v2:172.21.15.143:3300/0 conn(0x7fda54071e80 0x7fda54151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:15.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.919+0000 7fda59fb5700 1 -- 172.21.15.7:0/2415487013 >> 172.21.15.7:0/2415487013 conn(0x7fda540a94d0 msgr2=0x7fda540a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:09:15.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.919+0000 7fda59fb5700 1 -- 172.21.15.7:0/2415487013 shutdown_connections 2023-03-03T10:09:15.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.919+0000 7fda59fb5700 1 -- 172.21.15.7:0/2415487013 wait complete. 2023-03-03T10:09:15.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.920+0000 7fda59fb5700 1 Processor -- start 2023-03-03T10:09:15.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.921+0000 7fda59fb5700 1 -- start start 2023-03-03T10:09:15.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.921+0000 7fda59fb5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda54071e80 0x7fda54143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.921+0000 7fda59fb5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda54072250 0x7fda540c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.921+0000 7fda59fb5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 0x7fda540bfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.921+0000 7fda59fb5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda541ea780 con 0x7fda5414e900 2023-03-03T10:09:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.921+0000 7fda59fb5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda540c7150 con 0x7fda54072250 2023-03-03T10:09:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.921+0000 7fda59fb5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda541e9b70 con 0x7fda54071e80 2023-03-03T10:09:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda53fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 0x7fda540bfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda53fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 0x7fda540bfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59332/0 (socket says 172.21.15.7:59332) 2023-03-03T10:09:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda53fff700 1 -- 172.21.15.7:0/58442264 learned_addr learned my addr 172.21.15.7:0/58442264 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:09:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda52ffd700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda54072250 0x7fda540c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda537fe700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda54071e80 0x7fda54143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda53fff700 1 -- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda54071e80 msgr2=0x7fda54143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda53fff700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda54071e80 0x7fda54143ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda53fff700 1 -- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda54072250 msgr2=0x7fda540c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda53fff700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda54072250 0x7fda540c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda53fff700 1 -- 172.21.15.7:0/58442264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c009a70 con 0x7fda5414e900 2023-03-03T10:09:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda52ffd700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda54072250 0x7fda540c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:09:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda53fff700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 0x7fda540bfaa0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fda4c014040 tx=0x7fda4c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda3bfff700 1 -- 172.21.15.7:0/58442264 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c012430 con 0x7fda5414e900 2023-03-03T10:09:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda3bfff700 1 -- 172.21.15.7:0/58442264 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c020070 con 0x7fda5414e900 2023-03-03T10:09:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda3bfff700 1 -- 172.21.15.7:0/58442264 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c004280 con 0x7fda5414e900 2023-03-03T10:09:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda59fb5700 1 -- 172.21.15.7:0/58442264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5406c9a0 con 0x7fda5414e900 2023-03-03T10:09:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.922+0000 7fda59fb5700 1 -- 172.21.15.7:0/58442264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda54151300 con 0x7fda5414e900 2023-03-03T10:09:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.923+0000 7fda59fb5700 1 -- 172.21.15.7:0/58442264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5406cb30 con 0x7fda5414e900 2023-03-03T10:09:15.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.924+0000 7fda3bfff700 1 -- 172.21.15.7:0/58442264 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda4c02e070 con 0x7fda5414e900 2023-03-03T10:09:15.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.924+0000 7fda3bfff700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda400682f0 0x7fda4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:09:15.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.924+0000 7fda3bfff700 1 -- 172.21.15.7:0/58442264 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fda4c022070 con 0x7fda5414e900 2023-03-03T10:09:15.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.924+0000 7fda537fe700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda400682f0 0x7fda4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:09:15.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.925+0000 7fda537fe700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda400682f0 0x7fda4006a7a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fda5414dd00 tx=0x7fda48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:09:15.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:15.929+0000 7fda3bfff700 1 -- 172.21.15.7:0/58442264 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda4c054330 con 0x7fda5414e900 2023-03-03T10:09:16.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.163+0000 7fda59fb5700 1 -- 172.21.15.7:0/58442264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda54074bb0 con 0x7fda5414e900 2023-03-03T10:09:16.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.163+0000 7fda3bfff700 1 -- 172.21.15.7:0/58442264 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fda4c057950 con 0x7fda5414e900 2023-03-03T10:09:16.166 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:09:16.166 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:09:16.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.165+0000 7fda59fb5700 1 -- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda400682f0 msgr2=0x7fda4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:16.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.165+0000 7fda59fb5700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda400682f0 0x7fda4006a7a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fda5414dd00 tx=0x7fda48008040 comp rx=0 tx=0).stop 2023-03-03T10:09:16.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.165+0000 7fda59fb5700 1 -- 172.21.15.7:0/58442264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 msgr2=0x7fda540bfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:09:16.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.165+0000 7fda59fb5700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 0x7fda540bfaa0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fda4c014040 tx=0x7fda4c004ad0 comp rx=0 tx=0).stop 2023-03-03T10:09:16.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.165+0000 7fda59fb5700 1 -- 172.21.15.7:0/58442264 shutdown_connections 2023-03-03T10:09:16.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.165+0000 7fda59fb5700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda400682f0 0x7fda4006a7a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:16.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.165+0000 7fda59fb5700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda5414e900 0x7fda540bfaa0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:16.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.165+0000 7fda59fb5700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda54071e80 0x7fda54143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:16.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.165+0000 7fda59fb5700 1 --2- 172.21.15.7:0/58442264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda54072250 0x7fda540c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:09:16.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.165+0000 7fda59fb5700 1 -- 172.21.15.7:0/58442264 >> 172.21.15.7:0/58442264 conn(0x7fda540a94d0 msgr2=0x7fda5406a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:09:16.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.166+0000 7fda59fb5700 1 -- 172.21.15.7:0/58442264 shutdown_connections 2023-03-03T10:09:16.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:09:16.166+0000 7fda59fb5700 1 -- 172.21.15.7:0/58442264 wait complete. 2023-03-03T10:09:16.170 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:09:16.180 INFO:tasks.mds_thrash.fs.[b]:waiting for 71 secs before thrashing 2023-03-03T10:09:25.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:25.803 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:56.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:09:56.208 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T10:09:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T10:10:26.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:26.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:27.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:10:27.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.317+0000 7f0cf599e700 1 Processor -- start 2023-03-03T10:10:27.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.317+0000 7f0cf599e700 1 -- start start 2023-03-03T10:10:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.318+0000 7f0cf599e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0cf00caa30 0x7f0cf00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.318+0000 7f0cf599e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0cf0154ff0 0x7f0cf00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.318+0000 7f0cf599e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf00c7310 con 0x7f0cf00c00e0 2023-03-03T10:10:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.318+0000 7f0cf599e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf0155430 con 0x7f0cf0154ff0 2023-03-03T10:10:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.318+0000 7f0cf599e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf00bfd10 con 0x7f0cf00caa30 2023-03-03T10:10:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.318+0000 7f0cef7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0cf0154ff0 0x7f0cf00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.318+0000 7f0cef7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0cf0154ff0 0x7f0cf00cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59382/0 (socket says 172.21.15.7:59382) 2023-03-03T10:10:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.318+0000 7f0cef7fe700 1 -- 172.21.15.7:0/525804007 learned_addr learned my addr 172.21.15.7:0/525804007 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:10:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.319+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/525804007 >> v2:172.21.15.143:3300/0 conn(0x7f0cf00caa30 0x7f0cf00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.319+0000 7f0cef7fe700 1 -- 172.21.15.7:0/525804007 >> v2:172.21.15.143:3300/0 conn(0x7f0cf00caa30 msgr2=0x7f0cf00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:10:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.319+0000 7f0cef7fe700 1 --2- 172.21.15.7:0/525804007 >> v2:172.21.15.143:3300/0 conn(0x7f0cf00caa30 0x7f0cf00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.319+0000 7f0cef7fe700 1 -- 172.21.15.7:0/525804007 >> v1:172.21.15.143:6789/0 conn(0x7f0cf00c00e0 legacy=0x7f0cf00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.319+0000 7f0cef7fe700 1 -- 172.21.15.7:0/525804007 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf00bfa30 con 0x7f0cf0154ff0 2023-03-03T10:10:27.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.319+0000 7f0cef7fe700 1 --2- 172.21.15.7:0/525804007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf0154ff0 0x7f0cf00cae00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00c81b0 tx=0x7f0ce800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5250ad1d13e93c6a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:27.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.319+0000 7f0cedffb700 1 -- 172.21.15.7:0/525804007 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800e070 con 0x7f0cf0154ff0 2023-03-03T10:10:27.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.319+0000 7f0cedffb700 1 -- 172.21.15.7:0/525804007 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce800b9c0 con 0x7f0cf0154ff0 2023-03-03T10:10:27.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.319+0000 7f0cedffb700 1 -- 172.21.15.7:0/525804007 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800bd70 con 0x7f0cf0154ff0 2023-03-03T10:10:27.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.320+0000 7f0cf599e700 1 -- 172.21.15.7:0/525804007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf0154ff0 msgr2=0x7f0cf00cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:27.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.320+0000 7f0cf599e700 1 --2- 172.21.15.7:0/525804007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf0154ff0 0x7f0cf00cae00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00c81b0 tx=0x7f0ce800b330 comp rx=0 tx=0).stop 2023-03-03T10:10:27.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.320+0000 7f0cf599e700 1 -- 172.21.15.7:0/525804007 shutdown_connections 2023-03-03T10:10:27.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.320+0000 7f0cf599e700 1 --2- 172.21.15.7:0/525804007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf0154ff0 0x7f0cf00cae00 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:27.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.320+0000 7f0cf599e700 1 --2- 172.21.15.7:0/525804007 >> v2:172.21.15.143:3300/0 conn(0x7f0cf00caa30 0x7f0cf00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:27.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.320+0000 7f0cf599e700 1 -- 172.21.15.7:0/525804007 >> 172.21.15.7:0/525804007 conn(0x7f0cf0066e40 msgr2=0x7f0cf0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:10:27.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.320+0000 7f0cf599e700 1 -- 172.21.15.7:0/525804007 shutdown_connections 2023-03-03T10:10:27.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.320+0000 7f0cf599e700 1 -- 172.21.15.7:0/525804007 wait complete. 2023-03-03T10:10:27.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.321+0000 7f0cf599e700 1 Processor -- start 2023-03-03T10:10:27.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.321+0000 7f0cf599e700 1 -- start start 2023-03-03T10:10:27.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0cf599e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf00c00e0 0x7f0cf01dff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:27.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0cf599e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf00caa30 0x7f0cf01464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0cf599e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0154ff0 0x7f0cf0146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0cf599e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf014c9d0 con 0x7f0cf00c00e0 2023-03-03T10:10:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0cf599e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf014d4c0 con 0x7f0cf0154ff0 2023-03-03T10:10:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0cf599e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf014c0a0 con 0x7f0cf00caa30 2023-03-03T10:10:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0ceeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf00c00e0 0x7f0cf01dff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0ceeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf00c00e0 0x7f0cf01dff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59384/0 (socket says 172.21.15.7:59384) 2023-03-03T10:10:27.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0ceeffd700 1 -- 172.21.15.7:0/652115203 learned_addr learned my addr 172.21.15.7:0/652115203 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:10:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0cef7fe700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0154ff0 0x7f0cf0146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0cee7fc700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf00caa30 0x7f0cf01464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0ceeffd700 1 -- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf00caa30 msgr2=0x7f0cf01464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf00caa30 0x7f0cf01464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0ceeffd700 1 -- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0154ff0 msgr2=0x7f0cf0146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0154ff0 0x7f0cf0146a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0ceeffd700 1 -- 172.21.15.7:0/652115203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8009b30 con 0x7f0cf00c00e0 2023-03-03T10:10:27.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.322+0000 7f0cef7fe700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0154ff0 0x7f0cf0146a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:10:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.323+0000 7f0ceeffd700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf00c00e0 0x7f0cf01dff40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000e810 tx=0x7f0ce000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.323+0000 7f0ce7fff700 1 -- 172.21.15.7:0/652115203 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0013610 con 0x7f0cf00c00e0 2023-03-03T10:10:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.323+0000 7f0ce7fff700 1 -- 172.21.15.7:0/652115203 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce0013770 con 0x7f0cf00c00e0 2023-03-03T10:10:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.323+0000 7f0cf599e700 1 -- 172.21.15.7:0/652115203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf01547f0 con 0x7f0cf00c00e0 2023-03-03T10:10:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.323+0000 7f0ce7fff700 1 -- 172.21.15.7:0/652115203 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0013b20 con 0x7f0cf00c00e0 2023-03-03T10:10:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.323+0000 7f0cf599e700 1 -- 172.21.15.7:0/652115203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf0154510 con 0x7f0cf00c00e0 2023-03-03T10:10:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.324+0000 7f0cf599e700 1 -- 172.21.15.7:0/652115203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf0154ad0 con 0x7f0cf00c00e0 2023-03-03T10:10:27.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.324+0000 7f0ce7fff700 1 -- 172.21.15.7:0/652115203 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0ce0023070 con 0x7f0cf00c00e0 2023-03-03T10:10:27.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.325+0000 7f0ce7fff700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cd0068320 0x7f0cd006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:27.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.325+0000 7f0ce7fff700 1 -- 172.21.15.7:0/652115203 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0ce0085e30 con 0x7f0cf00c00e0 2023-03-03T10:10:27.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.325+0000 7f0cee7fc700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cd0068320 0x7f0cd006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:27.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.325+0000 7f0cee7fc700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cd0068320 0x7f0cd006a7d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00c1230 tx=0x7f0cd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:27.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.329+0000 7f0ce7fff700 1 -- 172.21.15.7:0/652115203 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0ce0052150 con 0x7f0cf00c00e0 2023-03-03T10:10:27.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.566+0000 7f0cf599e700 1 -- 172.21.15.7:0/652115203 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cf0074bb0 con 0x7f0cf00c00e0 2023-03-03T10:10:27.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.566+0000 7f0ce7fff700 1 -- 172.21.15.7:0/652115203 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0ce0055770 con 0x7f0cf00c00e0 2023-03-03T10:10:27.569 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:10:27.569 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:10:27.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 -- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cd0068320 msgr2=0x7f0cd006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:27.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cd0068320 0x7f0cd006a7d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00c1230 tx=0x7f0cd8009040 comp rx=0 tx=0).stop 2023-03-03T10:10:27.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 -- 172.21.15.7:0/652115203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf00c00e0 msgr2=0x7f0cf01dff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:27.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf00c00e0 0x7f0cf01dff40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000e810 tx=0x7f0ce000ebd0 comp rx=0 tx=0).stop 2023-03-03T10:10:27.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 -- 172.21.15.7:0/652115203 shutdown_connections 2023-03-03T10:10:27.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cf0154ff0 0x7f0cf0146a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:27.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cd0068320 0x7f0cd006a7d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:27.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cf00c00e0 0x7f0cf01dff40 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:27.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 --2- 172.21.15.7:0/652115203 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0cf00caa30 0x7f0cf01464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:27.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 -- 172.21.15.7:0/652115203 >> 172.21.15.7:0/652115203 conn(0x7f0cf0066e40 msgr2=0x7f0cf00c8850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:10:27.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.568+0000 7f0cf599e700 1 -- 172.21.15.7:0/652115203 shutdown_connections 2023-03-03T10:10:27.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:27.569+0000 7f0cf599e700 1 -- 172.21.15.7:0/652115203 wait complete. 2023-03-03T10:10:27.573 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:10:27.583 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-03-03T10:10:45.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:10:45.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.170+0000 7f0b89a58700 1 Processor -- start 2023-03-03T10:10:45.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.170+0000 7f0b89a58700 1 -- start start 2023-03-03T10:10:45.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b89a58700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b840caa30 0x7f0b840c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b89a58700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0b840c00e0 0x7f0b84154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b89a58700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0b840cae00 0x7f0b840c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b89a58700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b840cb280 con 0x7f0b840cae00 2023-03-03T10:10:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b89a58700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b840bfd10 con 0x7f0b840c00e0 2023-03-03T10:10:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b89a58700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840bfa30 con 0x7f0b840caa30 2023-03-03T10:10:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b837fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0b840cae00 0x7f0b840c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b837fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0b840cae00 0x7f0b840c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59406/0 (socket says 172.21.15.7:59406) 2023-03-03T10:10:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b837fe700 1 -- 172.21.15.7:0/2705246466 learned_addr learned my addr 172.21.15.7:0/2705246466 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:10:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b827fc700 1 --2- 172.21.15.7:0/2705246466 >> v2:172.21.15.143:3300/0 conn(0x7f0b840c00e0 0x7f0b84154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b82ffd700 1 --2- 172.21.15.7:0/2705246466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b840caa30 0x7f0b840c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b837fe700 1 -- 172.21.15.7:0/2705246466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b840caa30 msgr2=0x7f0b840c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:10:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b837fe700 1 --2- 172.21.15.7:0/2705246466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b840caa30 0x7f0b840c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b837fe700 1 -- 172.21.15.7:0/2705246466 >> v2:172.21.15.143:3300/0 conn(0x7f0b840c00e0 msgr2=0x7f0b84154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:10:45.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b837fe700 1 --2- 172.21.15.7:0/2705246466 >> v2:172.21.15.143:3300/0 conn(0x7f0b840c00e0 0x7f0b84154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b837fe700 1 -- 172.21.15.7:0/2705246466 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b84154510 con 0x7f0b840cae00 2023-03-03T10:10:45.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.171+0000 7f0b827fc700 1 --2- 172.21.15.7:0/2705246466 >> v2:172.21.15.143:3300/0 conn(0x7f0b840c00e0 0x7f0b84154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:10:45.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b837fe700 1 --2- 172.21.15.7:0/2705246466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 0x7f0b840c6c00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0b840c81e0 tx=0x7f0b7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157e7483ec2d91db server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:45.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b81ffb700 1 -- 172.21.15.7:0/2705246466 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c00e070 con 0x7f0b840cae00 2023-03-03T10:10:45.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b81ffb700 1 -- 172.21.15.7:0/2705246466 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7c00b9c0 con 0x7f0b840cae00 2023-03-03T10:10:45.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b81ffb700 1 -- 172.21.15.7:0/2705246466 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c00bd70 con 0x7f0b840cae00 2023-03-03T10:10:45.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b89a58700 1 -- 172.21.15.7:0/2705246466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 msgr2=0x7f0b840c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:45.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b89a58700 1 --2- 172.21.15.7:0/2705246466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 0x7f0b840c6c00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0b840c81e0 tx=0x7f0b7c00b330 comp rx=0 tx=0).stop 2023-03-03T10:10:45.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b89a58700 1 -- 172.21.15.7:0/2705246466 shutdown_connections 2023-03-03T10:10:45.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b89a58700 1 --2- 172.21.15.7:0/2705246466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 0x7f0b840c6c00 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b89a58700 1 --2- 172.21.15.7:0/2705246466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b840caa30 0x7f0b840c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b89a58700 1 --2- 172.21.15.7:0/2705246466 >> v2:172.21.15.143:3300/0 conn(0x7f0b840c00e0 0x7f0b84154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b89a58700 1 -- 172.21.15.7:0/2705246466 >> 172.21.15.7:0/2705246466 conn(0x7f0b84066e40 msgr2=0x7f0b84071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:10:45.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.172+0000 7f0b89a58700 1 -- 172.21.15.7:0/2705246466 shutdown_connections 2023-03-03T10:10:45.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.173+0000 7f0b89a58700 1 -- 172.21.15.7:0/2705246466 wait complete. 2023-03-03T10:10:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.174+0000 7f0b89a58700 1 Processor -- start 2023-03-03T10:10:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.174+0000 7f0b89a58700 1 -- start start 2023-03-03T10:10:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.174+0000 7f0b89a58700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b840c00e0 0x7f0b841561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.174+0000 7f0b89a58700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b840caa30 0x7f0b8414d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.174+0000 7f0b89a58700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 0x7f0b8414d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.174+0000 7f0b89a58700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840ab3e0 con 0x7f0b840cae00 2023-03-03T10:10:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.174+0000 7f0b89a58700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b841edbd0 con 0x7f0b840caa30 2023-03-03T10:10:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.174+0000 7f0b89a58700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b841570f0 con 0x7f0b840c00e0 2023-03-03T10:10:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b837fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 0x7f0b8414d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b837fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 0x7f0b8414d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59412/0 (socket says 172.21.15.7:59412) 2023-03-03T10:10:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b837fe700 1 -- 172.21.15.7:0/2249906512 learned_addr learned my addr 172.21.15.7:0/2249906512 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:10:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b827fc700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b840caa30 0x7f0b8414d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b837fe700 1 -- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b840c00e0 msgr2=0x7f0b841561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b82ffd700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b840c00e0 0x7f0b841561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b837fe700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b840c00e0 0x7f0b841561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b837fe700 1 -- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b840caa30 msgr2=0x7f0b8414d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b837fe700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b840caa30 0x7f0b8414d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b837fe700 1 -- 172.21.15.7:0/2249906512 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c009b30 con 0x7f0b840cae00 2023-03-03T10:10:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b837fe700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 0x7f0b8414d810 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0b8406b270 tx=0x7f0b7c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b6bfff700 1 -- 172.21.15.7:0/2249906512 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c004250 con 0x7f0b840cae00 2023-03-03T10:10:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b6bfff700 1 -- 172.21.15.7:0/2249906512 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7c0043d0 con 0x7f0b840cae00 2023-03-03T10:10:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b89a58700 1 -- 172.21.15.7:0/2249906512 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b84154ad0 con 0x7f0b840cae00 2023-03-03T10:10:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b6bfff700 1 -- 172.21.15.7:0/2249906512 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c004780 con 0x7f0b840cae00 2023-03-03T10:10:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.175+0000 7f0b89a58700 1 -- 172.21.15.7:0/2249906512 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b841547f0 con 0x7f0b840cae00 2023-03-03T10:10:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.176+0000 7f0b89a58700 1 -- 172.21.15.7:0/2249906512 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b840bf650 con 0x7f0b840cae00 2023-03-03T10:10:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.177+0000 7f0b6bfff700 1 -- 172.21.15.7:0/2249906512 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0b7c01c440 con 0x7f0b840cae00 2023-03-03T10:10:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.177+0000 7f0b6bfff700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b70068320 0x7f0b7006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:10:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.177+0000 7f0b6bfff700 1 -- 172.21.15.7:0/2249906512 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0b7c087800 con 0x7f0b840cae00 2023-03-03T10:10:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.177+0000 7f0b82ffd700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b70068320 0x7f0b7006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:10:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.178+0000 7f0b82ffd700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b70068320 0x7f0b7006a7d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800d8d0 tx=0x7f0b78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:10:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.182+0000 7f0b6bfff700 1 -- 172.21.15.7:0/2249906512 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0b7c053b40 con 0x7f0b840cae00 2023-03-03T10:10:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.418+0000 7f0b89a58700 1 -- 172.21.15.7:0/2249906512 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b84074bb0 con 0x7f0b840cae00 2023-03-03T10:10:45.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.418+0000 7f0b6bfff700 1 -- 172.21.15.7:0/2249906512 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0b7c057160 con 0x7f0b840cae00 2023-03-03T10:10:45.421 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:10:45.421 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:10:45.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 -- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b70068320 msgr2=0x7f0b7006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:45.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b70068320 0x7f0b7006a7d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0b7800d8d0 tx=0x7f0b78008040 comp rx=0 tx=0).stop 2023-03-03T10:10:45.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 -- 172.21.15.7:0/2249906512 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 msgr2=0x7f0b8414d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:10:45.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 0x7f0b8414d810 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0b8406b270 tx=0x7f0b7c004ad0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 -- 172.21.15.7:0/2249906512 shutdown_connections 2023-03-03T10:10:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b70068320 0x7f0b7006a7d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b840cae00 0x7f0b8414d810 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b840c00e0 0x7f0b841561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 --2- 172.21.15.7:0/2249906512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b840caa30 0x7f0b8414d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:10:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 -- 172.21.15.7:0/2249906512 >> 172.21.15.7:0/2249906512 conn(0x7f0b84066e40 msgr2=0x7f0b841512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:10:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 -- 172.21.15.7:0/2249906512 shutdown_connections 2023-03-03T10:10:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:10:45.421+0000 7f0b89a58700 1 -- 172.21.15.7:0/2249906512 wait complete. 2023-03-03T10:10:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:10:45.435 INFO:tasks.mds_thrash.fs.[a]:waiting for 89 secs before thrashing 2023-03-03T10:10:56.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:10:56.359 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:26.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:26.441 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:27.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:11:27.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.719+0000 7fbd144be700 1 Processor -- start 2023-03-03T10:11:27.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.719+0000 7fbd144be700 1 -- start start 2023-03-03T10:11:27.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.720+0000 7fbd144be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd0c14b0d0 0x7fbd0c071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:27.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.720+0000 7fbd144be700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbd0c0aad20 0x7fbd0c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:27.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.720+0000 7fbd144be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbd0c071d40 0x7fbd0c158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:27.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.720+0000 7fbd144be700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd0c0a9c20 con 0x7fbd0c0aad20 2023-03-03T10:11:27.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.720+0000 7fbd144be700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd0c154f10 con 0x7fbd0c071d40 2023-03-03T10:11:27.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.720+0000 7fbd144be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c0ac3c0 con 0x7fbd0c14b0d0 2023-03-03T10:11:27.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11a59700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbd0c0aad20 0x7fbd0c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:27.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11a59700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbd0c0aad20 0x7fbd0c157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59446/0 (socket says 172.21.15.7:59446) 2023-03-03T10:11:27.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11a59700 1 -- 172.21.15.7:0/3046553331 learned_addr learned my addr 172.21.15.7:0/3046553331 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:11:27.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd12a5b700 1 --2- 172.21.15.7:0/3046553331 >> v2:172.21.15.143:3300/0 conn(0x7fbd0c071d40 0x7fbd0c158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:27.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd1225a700 1 --2- 172.21.15.7:0/3046553331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd0c14b0d0 0x7fbd0c071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:27.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11a59700 1 -- 172.21.15.7:0/3046553331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd0c14b0d0 msgr2=0x7fbd0c071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:11:27.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11a59700 1 --2- 172.21.15.7:0/3046553331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd0c14b0d0 0x7fbd0c071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11a59700 1 -- 172.21.15.7:0/3046553331 >> v2:172.21.15.143:3300/0 conn(0x7fbd0c071d40 msgr2=0x7fbd0c158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:11:27.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11a59700 1 --2- 172.21.15.7:0/3046553331 >> v2:172.21.15.143:3300/0 conn(0x7fbd0c071d40 0x7fbd0c158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11a59700 1 -- 172.21.15.7:0/3046553331 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c06fc70 con 0x7fbd0c0aad20 2023-03-03T10:11:27.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11a59700 1 --2- 172.21.15.7:0/3046553331 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 0x7fbd0c157df0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0a98d0 tx=0x7fbd00009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f18fb0765e32f9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:11:27.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11258700 1 -- 172.21.15.7:0/3046553331 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00011830 con 0x7fbd0c0aad20 2023-03-03T10:11:27.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11258700 1 -- 172.21.15.7:0/3046553331 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd00011990 con 0x7fbd0c0aad20 2023-03-03T10:11:27.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.721+0000 7fbd11258700 1 -- 172.21.15.7:0/3046553331 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00011d20 con 0x7fbd0c0aad20 2023-03-03T10:11:27.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.722+0000 7fbd144be700 1 -- 172.21.15.7:0/3046553331 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 msgr2=0x7fbd0c157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:11:27.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.722+0000 7fbd144be700 1 --2- 172.21.15.7:0/3046553331 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 0x7fbd0c157df0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0a98d0 tx=0x7fbd00009bd0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.722+0000 7fbd144be700 1 -- 172.21.15.7:0/3046553331 shutdown_connections 2023-03-03T10:11:27.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.722+0000 7fbd144be700 1 --2- 172.21.15.7:0/3046553331 >> v2:172.21.15.143:3300/0 conn(0x7fbd0c071d40 0x7fbd0c158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.722+0000 7fbd144be700 1 --2- 172.21.15.7:0/3046553331 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd0c14b0d0 0x7fbd0c071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.722+0000 7fbd144be700 1 --2- 172.21.15.7:0/3046553331 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 0x7fbd0c157df0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.722+0000 7fbd144be700 1 -- 172.21.15.7:0/3046553331 >> 172.21.15.7:0/3046553331 conn(0x7fbd0c0c8900 msgr2=0x7fbd0c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:11:27.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.722+0000 7fbd144be700 1 -- 172.21.15.7:0/3046553331 shutdown_connections 2023-03-03T10:11:27.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.722+0000 7fbd144be700 1 -- 172.21.15.7:0/3046553331 wait complete. 2023-03-03T10:11:27.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.723+0000 7fbd144be700 1 Processor -- start 2023-03-03T10:11:27.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.723+0000 7fbd144be700 1 -- start start 2023-03-03T10:11:27.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd144be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd0c071d40 0x7fbd0c152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:27.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd144be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 0x7fbd0c157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:27.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd144be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd0c14b0d0 0x7fbd0c0c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:27.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd144be700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c14a6c0 con 0x7fbd0c0aad20 2023-03-03T10:11:27.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd144be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c156f60 con 0x7fbd0c14b0d0 2023-03-03T10:11:27.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd144be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c152300 con 0x7fbd0c071d40 2023-03-03T10:11:27.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd11a59700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 0x7fbd0c157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:27.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd11a59700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 0x7fbd0c157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59454/0 (socket says 172.21.15.7:59454) 2023-03-03T10:11:27.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd11a59700 1 -- 172.21.15.7:0/2121264470 learned_addr learned my addr 172.21.15.7:0/2121264470 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:11:27.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd1225a700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd0c071d40 0x7fbd0c152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:27.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd12a5b700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd0c14b0d0 0x7fbd0c0c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:27.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd11a59700 1 -- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd0c071d40 msgr2=0x7fbd0c152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:11:27.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd11a59700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd0c071d40 0x7fbd0c152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd11a59700 1 -- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd0c14b0d0 msgr2=0x7fbd0c0c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:11:27.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd11a59700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd0c14b0d0 0x7fbd0c0c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.724+0000 7fbd11a59700 1 -- 172.21.15.7:0/2121264470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd00009550 con 0x7fbd0c0aad20 2023-03-03T10:11:27.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.725+0000 7fbd11a59700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 0x7fbd0c157f60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009c10 tx=0x7fbd00004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:11:27.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.725+0000 7fbcfeffd700 1 -- 172.21.15.7:0/2121264470 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00004450 con 0x7fbd0c0aad20 2023-03-03T10:11:27.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.725+0000 7fbcfeffd700 1 -- 172.21.15.7:0/2121264470 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0001e070 con 0x7fbd0c0aad20 2023-03-03T10:11:27.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.725+0000 7fbd144be700 1 -- 172.21.15.7:0/2121264470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0c06f6b0 con 0x7fbd0c0aad20 2023-03-03T10:11:27.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.725+0000 7fbcfeffd700 1 -- 172.21.15.7:0/2121264470 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd000038e0 con 0x7fbd0c0aad20 2023-03-03T10:11:27.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.725+0000 7fbd144be700 1 -- 172.21.15.7:0/2121264470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c155160 con 0x7fbd0c0aad20 2023-03-03T10:11:27.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.726+0000 7fbd144be700 1 -- 172.21.15.7:0/2121264470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0c153020 con 0x7fbd0c0aad20 2023-03-03T10:11:27.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.726+0000 7fbcfeffd700 1 -- 172.21.15.7:0/2121264470 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd0000f050 con 0x7fbd0c0aad20 2023-03-03T10:11:27.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.727+0000 7fbcfeffd700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbcf00682f0 0x7fbcf006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:27.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.727+0000 7fbcfeffd700 1 -- 172.21.15.7:0/2121264470 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbd00086e20 con 0x7fbd0c0aad20 2023-03-03T10:11:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.727+0000 7fbd1225a700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbcf00682f0 0x7fbcf006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.727+0000 7fbd1225a700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbcf00682f0 0x7fbcf006a7a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009be0 tx=0x7fbd08009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:11:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.731+0000 7fbcfeffd700 1 -- 172.21.15.7:0/2121264470 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbd000531c0 con 0x7fbd0c0aad20 2023-03-03T10:11:27.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.970+0000 7fbd144be700 1 -- 172.21.15.7:0/2121264470 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd0c06f990 con 0x7fbd0c0aad20 2023-03-03T10:11:27.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.970+0000 7fbcfeffd700 1 -- 172.21.15.7:0/2121264470 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbd00020090 con 0x7fbd0c0aad20 2023-03-03T10:11:27.973 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:11:27.973 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:11:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 -- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbcf00682f0 msgr2=0x7fbcf006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:11:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbcf00682f0 0x7fbcf006a7a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbd08009be0 tx=0x7fbd08009210 comp rx=0 tx=0).stop 2023-03-03T10:11:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 -- 172.21.15.7:0/2121264470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 msgr2=0x7fbd0c157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:11:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 0x7fbd0c157f60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009c10 tx=0x7fbd00004750 comp rx=0 tx=0).stop 2023-03-03T10:11:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 -- 172.21.15.7:0/2121264470 shutdown_connections 2023-03-03T10:11:27.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd0c14b0d0 0x7fbd0c0c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd0c071d40 0x7fbd0c152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbcf00682f0 0x7fbcf006a7a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 --2- 172.21.15.7:0/2121264470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd0c0aad20 0x7fbd0c157f60 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:27.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 -- 172.21.15.7:0/2121264470 >> 172.21.15.7:0/2121264470 conn(0x7fbd0c0c8900 msgr2=0x7fbd0c148570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:11:27.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.972+0000 7fbd144be700 1 -- 172.21.15.7:0/2121264470 shutdown_connections 2023-03-03T10:11:27.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:27.973+0000 7fbd144be700 1 -- 172.21.15.7:0/2121264470 wait complete. 2023-03-03T10:11:27.977 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:11:27.986 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T10:11:45.987 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:11:46.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.124+0000 7fc030bc2700 1 Processor -- start 2023-03-03T10:11:46.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.124+0000 7fc030bc2700 1 -- start start 2023-03-03T10:11:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.125+0000 7fc030bc2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc02c155410 0x7fc02c0bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.125+0000 7fc030bc2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc02c0c5b00 0x7fc02c0bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.125+0000 7fc030bc2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0c08d0 con 0x7fc02c151e80 2023-03-03T10:11:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.125+0000 7fc030bc2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02c152250 con 0x7fc02c0c5b00 2023-03-03T10:11:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.125+0000 7fc030bc2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02c0c0fb0 con 0x7fc02c155410 2023-03-03T10:11:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.125+0000 7fc02ad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc02c0c5b00 0x7fc02c0bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.125+0000 7fc02ad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc02c0c5b00 0x7fc02c0bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59464/0 (socket says 172.21.15.7:59464) 2023-03-03T10:11:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.125+0000 7fc02ad9d700 1 -- 172.21.15.7:0/2309255978 learned_addr learned my addr 172.21.15.7:0/2309255978 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:11:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.125+0000 7fc029d9b700 1 --2- 172.21.15.7:0/2309255978 >> v2:172.21.15.143:3300/0 conn(0x7fc02c155410 0x7fc02c0bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.125+0000 7fc02ad9d700 1 -- 172.21.15.7:0/2309255978 >> v2:172.21.15.143:3300/0 conn(0x7fc02c155410 msgr2=0x7fc02c0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:11:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc02ad9d700 1 --2- 172.21.15.7:0/2309255978 >> v2:172.21.15.143:3300/0 conn(0x7fc02c155410 0x7fc02c0bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc02ad9d700 1 -- 172.21.15.7:0/2309255978 >> v1:172.21.15.143:6789/0 conn(0x7fc02c151e80 legacy=0x7fc02c0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:11:46.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc02ad9d700 1 -- 172.21.15.7:0/2309255978 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c154930 con 0x7fc02c0c5b00 2023-03-03T10:11:46.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc029d9b700 1 --2- 172.21.15.7:0/2309255978 >> v2:172.21.15.143:3300/0 conn(0x7fc02c155410 0x7fc02c0bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:11:46.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc02ad9d700 1 --2- 172.21.15.7:0/2309255978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c5b00 0x7fc02c0bfe20 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc02c06c380 tx=0x7fc02400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85d51e76be5dd4f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:11:46.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc02959a700 1 -- 172.21.15.7:0/2309255978 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02400e070 con 0x7fc02c0c5b00 2023-03-03T10:11:46.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc02959a700 1 -- 172.21.15.7:0/2309255978 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc024003a20 con 0x7fc02c0c5b00 2023-03-03T10:11:46.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc02959a700 1 -- 172.21.15.7:0/2309255978 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024003dd0 con 0x7fc02c0c5b00 2023-03-03T10:11:46.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc030bc2700 1 -- 172.21.15.7:0/2309255978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c5b00 msgr2=0x7fc02c0bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:11:46.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc030bc2700 1 --2- 172.21.15.7:0/2309255978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c5b00 0x7fc02c0bfe20 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc02c06c380 tx=0x7fc02400b330 comp rx=0 tx=0).stop 2023-03-03T10:11:46.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc030bc2700 1 -- 172.21.15.7:0/2309255978 shutdown_connections 2023-03-03T10:11:46.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc030bc2700 1 --2- 172.21.15.7:0/2309255978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c5b00 0x7fc02c0bfe20 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:46.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc030bc2700 1 --2- 172.21.15.7:0/2309255978 >> v2:172.21.15.143:3300/0 conn(0x7fc02c155410 0x7fc02c0bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:46.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.126+0000 7fc030bc2700 1 -- 172.21.15.7:0/2309255978 >> 172.21.15.7:0/2309255978 conn(0x7fc02c071630 msgr2=0x7fc02c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:11:46.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.127+0000 7fc030bc2700 1 -- 172.21.15.7:0/2309255978 shutdown_connections 2023-03-03T10:11:46.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.127+0000 7fc030bc2700 1 -- 172.21.15.7:0/2309255978 wait complete. 2023-03-03T10:11:46.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.128+0000 7fc030bc2700 1 Processor -- start 2023-03-03T10:11:46.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.128+0000 7fc030bc2700 1 -- start start 2023-03-03T10:11:46.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc030bc2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0c5b00 0x7fc02c0c3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:46.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc030bc2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c151e80 0x7fc02c1ed740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:46.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc030bc2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c155410 0x7fc02c1edc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:46.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc030bc2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c157fe0 con 0x7fc02c155410 2023-03-03T10:11:46.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc030bc2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c155fb0 con 0x7fc02c0c5b00 2023-03-03T10:11:46.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc030bc2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c1576b0 con 0x7fc02c151e80 2023-03-03T10:11:46.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c155410 0x7fc02c1edc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:46.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c155410 0x7fc02c1edc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59472/0 (socket says 172.21.15.7:59472) 2023-03-03T10:11:46.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02ad9d700 1 -- 172.21.15.7:0/2095596192 learned_addr learned my addr 172.21.15.7:0/2095596192 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:11:46.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02a59c700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0c5b00 0x7fc02c0c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc029d9b700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c151e80 0x7fc02c1ed740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02ad9d700 1 -- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c151e80 msgr2=0x7fc02c1ed740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:11:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02ad9d700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c151e80 0x7fc02c1ed740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02ad9d700 1 -- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0c5b00 msgr2=0x7fc02c0c3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:11:46.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02ad9d700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0c5b00 0x7fc02c0c3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:46.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02ad9d700 1 -- 172.21.15.7:0/2095596192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc024009b30 con 0x7fc02c155410 2023-03-03T10:11:46.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02a59c700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0c5b00 0x7fc02c0c3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:11:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc02ad9d700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c155410 0x7fc02c1edc80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc024015040 tx=0x7fc024004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:11:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc01affd700 1 -- 172.21.15.7:0/2095596192 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024004440 con 0x7fc02c155410 2023-03-03T10:11:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc01affd700 1 -- 172.21.15.7:0/2095596192 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0240045a0 con 0x7fc02c155410 2023-03-03T10:11:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.129+0000 7fc030bc2700 1 -- 172.21.15.7:0/2095596192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02c154ef0 con 0x7fc02c155410 2023-03-03T10:11:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.130+0000 7fc01affd700 1 -- 172.21.15.7:0/2095596192 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024004950 con 0x7fc02c155410 2023-03-03T10:11:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.130+0000 7fc030bc2700 1 -- 172.21.15.7:0/2095596192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02c154c10 con 0x7fc02c155410 2023-03-03T10:11:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.131+0000 7fc030bc2700 1 -- 172.21.15.7:0/2095596192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02c0c5220 con 0x7fc02c155410 2023-03-03T10:11:46.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.131+0000 7fc01affd700 1 -- 172.21.15.7:0/2095596192 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc024022030 con 0x7fc02c155410 2023-03-03T10:11:46.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.131+0000 7fc01affd700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc00c0682f0 0x7fc00c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:11:46.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.131+0000 7fc01affd700 1 -- 172.21.15.7:0/2095596192 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc024087630 con 0x7fc02c155410 2023-03-03T10:11:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.132+0000 7fc02a59c700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc00c0682f0 0x7fc00c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:11:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.132+0000 7fc02a59c700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc00c0682f0 0x7fc00c06a7a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc02c0cad50 tx=0x7fc020009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:11:46.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.136+0000 7fc01affd700 1 -- 172.21.15.7:0/2095596192 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc024053950 con 0x7fc02c155410 2023-03-03T10:11:46.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.373+0000 7fc030bc2700 1 -- 172.21.15.7:0/2095596192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc02c074bb0 con 0x7fc02c155410 2023-03-03T10:11:46.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.374+0000 7fc01affd700 1 -- 172.21.15.7:0/2095596192 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc024007310 con 0x7fc02c155410 2023-03-03T10:11:46.376 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:11:46.376 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:11:46.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 -- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc00c0682f0 msgr2=0x7fc00c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:11:46.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc00c0682f0 0x7fc00c06a7a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc02c0cad50 tx=0x7fc020009040 comp rx=0 tx=0).stop 2023-03-03T10:11:46.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 -- 172.21.15.7:0/2095596192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c155410 msgr2=0x7fc02c1edc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:11:46.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c155410 0x7fc02c1edc80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc024015040 tx=0x7fc024004ad0 comp rx=0 tx=0).stop 2023-03-03T10:11:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 -- 172.21.15.7:0/2095596192 shutdown_connections 2023-03-03T10:11:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc00c0682f0 0x7fc00c06a7a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c155410 0x7fc02c1edc80 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0c5b00 0x7fc02c0c3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 --2- 172.21.15.7:0/2095596192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c151e80 0x7fc02c1ed740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:11:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 -- 172.21.15.7:0/2095596192 >> 172.21.15.7:0/2095596192 conn(0x7fc02c071630 msgr2=0x7fc02c1525b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:11:46.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 -- 172.21.15.7:0/2095596192 shutdown_connections 2023-03-03T10:11:46.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:11:46.376+0000 7fc030bc2700 1 -- 172.21.15.7:0/2095596192 wait complete. 2023-03-03T10:11:46.380 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:11:46.390 INFO:tasks.mds_thrash.fs.[b]:waiting for 104 secs before thrashing 2023-03-03T10:11:56.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:11:56.515 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:14.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:12:14.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.573+0000 7fd47ac71700 1 Processor -- start 2023-03-03T10:12:14.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.573+0000 7fd47ac71700 1 -- start start 2023-03-03T10:12:14.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd47ac71700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd47406e020 0x7fd474071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:12:14.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd47ac71700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd474154910 con 0x7fd474155390 2023-03-03T10:12:14.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd47ac71700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4740c4540 con 0x7fd474072520 2023-03-03T10:12:14.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd47ac71700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd474070b90 con 0x7fd47406e020 2023-03-03T10:12:14.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd478a0d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd474155390 0x7fd4740c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53462/0 (socket says 172.21.15.7:53462) 2023-03-03T10:12:14.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd478a0d700 1 -- 172.21.15.7:0/3329869802 learned_addr learned my addr 172.21.15.7:0/3329869802 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:12:14.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd473fff700 1 --2- 172.21.15.7:0/3329869802 >> v2:172.21.15.143:3300/0 conn(0x7fd47406e020 0x7fd474071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:12:14.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd4737fe700 1 -- 172.21.15.7:0/3329869802 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807735188 0 0) 0x7fd474154910 con 0x7fd474155390 2023-03-03T10:12:14.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd4737fe700 1 -- 172.21.15.7:0/3329869802 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4640035b0 con 0x7fd474155390 2023-03-03T10:12:14.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd473fff700 1 -- 172.21.15.7:0/3329869802 >> v1:172.21.15.143:6789/0 conn(0x7fd474072520 legacy=0x7fd474151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:14.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd473fff700 1 -- 172.21.15.7:0/3329869802 >> v1:172.21.15.7:6789/0 conn(0x7fd474155390 legacy=0x7fd4740c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:14.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.574+0000 7fd473fff700 1 -- 172.21.15.7:0/3329869802 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4740702a0 con 0x7fd47406e020 2023-03-03T10:12:14.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd473fff700 1 --2- 172.21.15.7:0/3329869802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd47406e020 0x7fd474071fe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd460009840 tx=0x7fd460009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fedd2204d49cb5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:12:14.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd4737fe700 1 -- 172.21.15.7:0/3329869802 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460011830 con 0x7fd47406e020 2023-03-03T10:12:14.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd4737fe700 1 -- 172.21.15.7:0/3329869802 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd460011990 con 0x7fd47406e020 2023-03-03T10:12:14.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd4737fe700 1 -- 172.21.15.7:0/3329869802 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460011d20 con 0x7fd47406e020 2023-03-03T10:12:14.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd47ac71700 1 -- 172.21.15.7:0/3329869802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd47406e020 msgr2=0x7fd474071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:14.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd47ac71700 1 --2- 172.21.15.7:0/3329869802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd47406e020 0x7fd474071fe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd460009840 tx=0x7fd460009b50 comp rx=0 tx=0).stop 2023-03-03T10:12:14.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd47ac71700 1 -- 172.21.15.7:0/3329869802 shutdown_connections 2023-03-03T10:12:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd47ac71700 1 --2- 172.21.15.7:0/3329869802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd47406e020 0x7fd474071fe0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd47ac71700 1 -- 172.21.15.7:0/3329869802 >> 172.21.15.7:0/3329869802 conn(0x7fd47406c110 msgr2=0x7fd47406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:12:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd47ac71700 1 -- 172.21.15.7:0/3329869802 shutdown_connections 2023-03-03T10:12:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.575+0000 7fd47ac71700 1 -- 172.21.15.7:0/3329869802 wait complete. 2023-03-03T10:12:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.576+0000 7fd47ac71700 1 Processor -- start 2023-03-03T10:12:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.577+0000 7fd47ac71700 1 -- start start 2023-03-03T10:12:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.577+0000 7fd47ac71700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd47406e020 0x7fd4741dc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:12:14.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.577+0000 7fd47ac71700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd474072520 0x7fd474147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:12:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.577+0000 7fd47ac71700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd474155390 0x7fd474145c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:12:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.577+0000 7fd47ac71700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4741ed440 con 0x7fd474155390 2023-03-03T10:12:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.577+0000 7fd47ac71700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4741dc780 con 0x7fd474072520 2023-03-03T10:12:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.577+0000 7fd47ac71700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4741ed700 con 0x7fd47406e020 2023-03-03T10:12:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47920e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd474155390 0x7fd474145c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:12:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47920e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd474155390 0x7fd474145c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59490/0 (socket says 172.21.15.7:59490) 2023-03-03T10:12:14.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47920e700 1 -- 172.21.15.7:0/2831423363 learned_addr learned my addr 172.21.15.7:0/2831423363 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:12:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd473fff700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd474072520 0x7fd474147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:12:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd478a0d700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd47406e020 0x7fd4741dc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:12:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47920e700 1 -- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd47406e020 msgr2=0x7fd4741dc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47920e700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd47406e020 0x7fd4741dc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47920e700 1 -- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd474072520 msgr2=0x7fd474147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47920e700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd474072520 0x7fd474147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47920e700 1 -- 172.21.15.7:0/2831423363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4600094a0 con 0x7fd474155390 2023-03-03T10:12:14.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47920e700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd474155390 0x7fd474145c80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00cc50 tx=0x7fd46c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:12:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd4717fa700 1 -- 172.21.15.7:0/2831423363 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c003c00 con 0x7fd474155390 2023-03-03T10:12:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd4717fa700 1 -- 172.21.15.7:0/2831423363 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c003d60 con 0x7fd474155390 2023-03-03T10:12:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47ac71700 1 -- 172.21.15.7:0/2831423363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd474070820 con 0x7fd474155390 2023-03-03T10:12:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd4717fa700 1 -- 172.21.15.7:0/2831423363 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c00aab0 con 0x7fd474155390 2023-03-03T10:12:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.578+0000 7fd47ac71700 1 -- 172.21.15.7:0/2831423363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd474070540 con 0x7fd474155390 2023-03-03T10:12:14.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.579+0000 7fd47ac71700 1 -- 172.21.15.7:0/2831423363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd474155760 con 0x7fd474155390 2023-03-03T10:12:14.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.580+0000 7fd4717fa700 1 -- 172.21.15.7:0/2831423363 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd46c010050 con 0x7fd474155390 2023-03-03T10:12:14.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.580+0000 7fd4717fa700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4580682f0 0x7fd45806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:12:14.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.580+0000 7fd4717fa700 1 -- 172.21.15.7:0/2831423363 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd46c085ea0 con 0x7fd474155390 2023-03-03T10:12:14.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.580+0000 7fd478a0d700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4580682f0 0x7fd45806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:12:14.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.581+0000 7fd478a0d700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4580682f0 0x7fd45806a7a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd468007fd0 tx=0x7fd468007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:12:14.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.584+0000 7fd4717fa700 1 -- 172.21.15.7:0/2831423363 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd46c0521c0 con 0x7fd474155390 2023-03-03T10:12:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.826+0000 7fd47ac71700 1 -- 172.21.15.7:0/2831423363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4740c4540 con 0x7fd474155390 2023-03-03T10:12:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.827+0000 7fd4717fa700 1 -- 172.21.15.7:0/2831423363 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd4740c4540 con 0x7fd474155390 2023-03-03T10:12:14.830 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:12:14.831 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:12:14.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 -- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4580682f0 msgr2=0x7fd45806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:14.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4580682f0 0x7fd45806a7a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd468007fd0 tx=0x7fd468007dc0 comp rx=0 tx=0).stop 2023-03-03T10:12:14.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 -- 172.21.15.7:0/2831423363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd474155390 msgr2=0x7fd474145c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:12:14.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd474155390 0x7fd474145c80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00cc50 tx=0x7fd46c00cf60 comp rx=0 tx=0).stop 2023-03-03T10:12:14.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 -- 172.21.15.7:0/2831423363 shutdown_connections 2023-03-03T10:12:14.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4580682f0 0x7fd45806a7a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:14.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd474155390 0x7fd474145c80 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:14.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd47406e020 0x7fd4741dc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:14.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 --2- 172.21.15.7:0/2831423363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd474072520 0x7fd474147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:12:14.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 -- 172.21.15.7:0/2831423363 >> 172.21.15.7:0/2831423363 conn(0x7fd47406c110 msgr2=0x7fd4740c1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:12:14.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 -- 172.21.15.7:0/2831423363 shutdown_connections 2023-03-03T10:12:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:12:14.829+0000 7fd47ac71700 1 -- 172.21.15.7:0/2831423363 wait complete. 2023-03-03T10:12:14.836 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:12:14.843 INFO:tasks.mds_thrash.fs.[a]:waiting for 72 secs before thrashing 2023-03-03T10:12:26.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:26.596 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:56.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:12:56.679 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:26.772 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:26.776 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:26.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:13:26.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.976+0000 7fd492ed2700 1 Processor -- start 2023-03-03T10:13:26.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.976+0000 7fd492ed2700 1 -- start start 2023-03-03T10:13:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd492ed2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd48c1530d0 0x7fd48c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd492ed2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c0c2f30 con 0x7fd48c14f9d0 2023-03-03T10:13:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd492ed2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48c0c31d0 con 0x7fd48c14fda0 2023-03-03T10:13:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd492ed2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48c1524a0 con 0x7fd48c1530d0 2023-03-03T10:13:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd490c6e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd48c14f9d0 0x7fd48c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53526/0 (socket says 172.21.15.7:53526) 2023-03-03T10:13:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd490c6e700 1 -- 172.21.15.7:0/1819640547 learned_addr learned my addr 172.21.15.7:0/1819640547 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:13:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981589019 0 0) 0x7fd48c0c2f30 con 0x7fd48c14f9d0 2023-03-03T10:13:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd49146f700 1 --2- 172.21.15.7:0/1819640547 >> v2:172.21.15.143:3300/0 conn(0x7fd48c1530d0 0x7fd48c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd47c0035b0 con 0x7fd48c14f9d0 2023-03-03T10:13:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2898150512 0 0) 0x7fd47c0035b0 con 0x7fd48c14f9d0 2023-03-03T10:13:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 >> v2:172.21.15.143:3300/0 conn(0x7fd48c1530d0 msgr2=0x7fd48c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:13:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd48b7fe700 1 --2- 172.21.15.7:0/1819640547 >> v2:172.21.15.143:3300/0 conn(0x7fd48c1530d0 0x7fd48c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 >> v1:172.21.15.143:6789/0 conn(0x7fd48c14fda0 legacy=0x7fd48c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c0c34b0 con 0x7fd48c14f9d0 2023-03-03T10:13:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd480001670 con 0x7fd48c14f9d0 2023-03-03T10:13:26.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.977+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 >> v1:172.21.15.7:6789/0 conn(0x7fd48c14f9d0 legacy=0x7fd48c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:26.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48b7fe700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd47c003a70 0x7fd47c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:26.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48b7fe700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd47c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:26.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48b7fe700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd47c005320 0x7fd47c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:26.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c1524a0 con 0x7fd47c004850 2023-03-03T10:13:26.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c00c230 con 0x7fd47c005320 2023-03-03T10:13:26.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c00c3f0 con 0x7fd47c003a70 2023-03-03T10:13:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48bfff700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd47c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd49146f700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd47c005320 0x7fd47c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd490c6e700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd47c003a70 0x7fd47c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48bfff700 1 -- 172.21.15.7:0/1819640547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd47c003a70 msgr2=0x7fd47c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:13:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48bfff700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd47c003a70 0x7fd47c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48bfff700 1 -- 172.21.15.7:0/1819640547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd47c005320 msgr2=0x7fd47c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:13:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48bfff700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd47c005320 0x7fd47c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd49146f700 1 -- 172.21.15.7:0/1819640547 reap_dead start 2023-03-03T10:13:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48bfff700 1 -- 172.21.15.7:0/1819640547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c0035b0 con 0x7fd47c004850 2023-03-03T10:13:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48bfff700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd47c004cc0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd48c1501d0 tx=0x7fd47c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef4d865dd8372f3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:13:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4780052d0 con 0x7fd47c004850 2023-03-03T10:13:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd478005450 con 0x7fd47c004850 2023-03-03T10:13:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.978+0000 7fd48b7fe700 1 -- 172.21.15.7:0/1819640547 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4780038c0 con 0x7fd47c004850 2023-03-03T10:13:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.979+0000 7fd492ed2700 1 -- 172.21.15.7:0/1819640547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 msgr2=0x7fd47c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.979+0000 7fd492ed2700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd47c004cc0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd48c1501d0 tx=0x7fd47c005880 comp rx=0 tx=0).stop 2023-03-03T10:13:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.979+0000 7fd492ed2700 1 -- 172.21.15.7:0/1819640547 shutdown_connections 2023-03-03T10:13:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.979+0000 7fd492ed2700 1 --2- 172.21.15.7:0/1819640547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd47c004cc0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.979+0000 7fd492ed2700 1 -- 172.21.15.7:0/1819640547 >> 172.21.15.7:0/1819640547 conn(0x7fd48c0c8900 msgr2=0x7fd48c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:13:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.979+0000 7fd492ed2700 1 -- 172.21.15.7:0/1819640547 shutdown_connections 2023-03-03T10:13:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.979+0000 7fd492ed2700 1 -- 172.21.15.7:0/1819640547 wait complete. 2023-03-03T10:13:26.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.980+0000 7fd492ed2700 1 Processor -- start 2023-03-03T10:13:26.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.980+0000 7fd492ed2700 1 -- start start 2023-03-03T10:13:26.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd492ed2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd48c148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:26.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd492ed2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd48c149c00 0x7fd48c0c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:26.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd492ed2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd48c0c1d20 0x7fd48c14d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:26.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd492ed2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c1539c0 con 0x7fd47c004850 2023-03-03T10:13:26.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd492ed2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c14cfc0 con 0x7fd48c149c00 2023-03-03T10:13:26.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd492ed2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c14ae50 con 0x7fd48c0c1d20 2023-03-03T10:13:26.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd490c6e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd48c148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:26.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd490c6e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd48c148bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59560/0 (socket says 172.21.15.7:59560) 2023-03-03T10:13:26.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd490c6e700 1 -- 172.21.15.7:0/1134441614 learned_addr learned my addr 172.21.15.7:0/1134441614 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:13:26.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd49146f700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd48c0c1d20 0x7fd48c14d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:26.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd48bfff700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd48c149c00 0x7fd48c0c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:26.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd490c6e700 1 -- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd48c0c1d20 msgr2=0x7fd48c14d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:26.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd490c6e700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd48c0c1d20 0x7fd48c14d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:26.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd490c6e700 1 -- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd48c149c00 msgr2=0x7fd48c0c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:26.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd490c6e700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd48c149c00 0x7fd48c0c17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:26.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.981+0000 7fd490c6e700 1 -- 172.21.15.7:0/1134441614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c0ac350 con 0x7fd47c004850 2023-03-03T10:13:26.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.982+0000 7fd490c6e700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd48c148bd0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd4800072c0 tx=0x7fd480007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:13:26.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.982+0000 7fd4897fa700 1 -- 172.21.15.7:0/1134441614 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4800010b0 con 0x7fd47c004850 2023-03-03T10:13:26.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.982+0000 7fd4897fa700 1 -- 172.21.15.7:0/1134441614 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4800077a0 con 0x7fd47c004850 2023-03-03T10:13:26.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.982+0000 7fd492ed2700 1 -- 172.21.15.7:0/1134441614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48c0ab630 con 0x7fd47c004850 2023-03-03T10:13:26.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.982+0000 7fd4897fa700 1 -- 172.21.15.7:0/1134441614 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48000fa90 con 0x7fd47c004850 2023-03-03T10:13:26.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.982+0000 7fd492ed2700 1 -- 172.21.15.7:0/1134441614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48c0c3e10 con 0x7fd47c004850 2023-03-03T10:13:26.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.983+0000 7fd492ed2700 1 -- 172.21.15.7:0/1134441614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48c1ea720 con 0x7fd47c004850 2023-03-03T10:13:26.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.983+0000 7fd4897fa700 1 -- 172.21.15.7:0/1134441614 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd48000fbf0 con 0x7fd47c004850 2023-03-03T10:13:26.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.983+0000 7fd4897fa700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4700682f0 0x7fd47006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:26.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.984+0000 7fd4897fa700 1 -- 172.21.15.7:0/1134441614 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd4800861e0 con 0x7fd47c004850 2023-03-03T10:13:26.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.984+0000 7fd48bfff700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4700682f0 0x7fd47006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:26.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.984+0000 7fd48bfff700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4700682f0 0x7fd47006a7a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd478005040 tx=0x7fd478009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:13:26.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:26.988+0000 7fd4897fa700 1 -- 172.21.15.7:0/1134441614 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd480052520 con 0x7fd47c004850 2023-03-03T10:13:27.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.223+0000 7fd492ed2700 1 -- 172.21.15.7:0/1134441614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd48c074bb0 con 0x7fd47c004850 2023-03-03T10:13:27.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.224+0000 7fd4897fa700 1 -- 172.21.15.7:0/1134441614 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd480055b40 con 0x7fd47c004850 2023-03-03T10:13:27.226 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:13:27.227 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:13:27.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 -- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4700682f0 msgr2=0x7fd47006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:27.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4700682f0 0x7fd47006a7a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd478005040 tx=0x7fd478009030 comp rx=0 tx=0).stop 2023-03-03T10:13:27.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 -- 172.21.15.7:0/1134441614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 msgr2=0x7fd48c148bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:27.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd48c148bd0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd4800072c0 tx=0x7fd480007680 comp rx=0 tx=0).stop 2023-03-03T10:13:27.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 -- 172.21.15.7:0/1134441614 shutdown_connections 2023-03-03T10:13:27.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd48c0c1d20 0x7fd48c14d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:27.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd4700682f0 0x7fd47006a7a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:27.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd47c004850 0x7fd48c148bd0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:27.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 --2- 172.21.15.7:0/1134441614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd48c149c00 0x7fd48c0c17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 -- 172.21.15.7:0/1134441614 >> 172.21.15.7:0/1134441614 conn(0x7fd48c0c8900 msgr2=0x7fd48c0716c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:13:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 -- 172.21.15.7:0/1134441614 shutdown_connections 2023-03-03T10:13:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:27.226+0000 7fd492ed2700 1 -- 172.21.15.7:0/1134441614 wait complete. 2023-03-03T10:13:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:13:27.240 INFO:tasks.mds_thrash.fs.[a]:waiting for 82 secs before thrashing 2023-03-03T10:13:30.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:13:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.528+0000 7f232a321700 1 Processor -- start 2023-03-03T10:13:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.529+0000 7f232a321700 1 -- start start 2023-03-03T10:13:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f232a321700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23240c5190 0x7f232406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f232a321700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f232406dbe0 0x7f232406e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f232a321700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23240aa890 con 0x7f23240a6140 2023-03-03T10:13:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f232a321700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23240c17f0 con 0x7f232406dbe0 2023-03-03T10:13:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f232a321700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23240c1ad0 con 0x7f23240c5190 2023-03-03T10:13:30.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f23288be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f232406dbe0 0x7f232406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:30.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f2323fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f23240a6140 0x7f23240c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:57066/0 (socket says 172.21.15.7:57066) 2023-03-03T10:13:30.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f23237fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23240c5190 0x7f232406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:30.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f2323fff700 1 -- 172.21.15.7:0/1890594017 learned_addr learned my addr 172.21.15.7:0/1890594017 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:13:30.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f2322ffd700 1 -- 172.21.15.7:0/1890594017 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097048508 0 0) 0x7f23240aa890 con 0x7f23240a6140 2023-03-03T10:13:30.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f2322ffd700 1 -- 172.21.15.7:0/1890594017 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23100035b0 con 0x7f23240a6140 2023-03-03T10:13:30.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f23288be700 1 -- 172.21.15.7:0/1890594017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23240c5190 msgr2=0x7f232406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:13:30.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f23288be700 1 --2- 172.21.15.7:0/1890594017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23240c5190 0x7f232406d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:30.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f23288be700 1 -- 172.21.15.7:0/1890594017 >> v1:172.21.15.143:6789/0 conn(0x7f23240a6140 legacy=0x7f23240c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:30.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f23288be700 1 -- 172.21.15.7:0/1890594017 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23241552c0 con 0x7f232406dbe0 2023-03-03T10:13:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.530+0000 7f23237fe700 1 --2- 172.21.15.7:0/1890594017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23240c5190 0x7f232406d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:13:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f23288be700 1 --2- 172.21.15.7:0/1890594017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f232406dbe0 0x7f232406e040 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f231c009d90 tx=0x7f231c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2dd5bc999f7fd1e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:13:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f2322ffd700 1 -- 172.21.15.7:0/1890594017 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c00e070 con 0x7f232406dbe0 2023-03-03T10:13:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f2322ffd700 1 -- 172.21.15.7:0/1890594017 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c00bb20 con 0x7f232406dbe0 2023-03-03T10:13:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f2322ffd700 1 -- 172.21.15.7:0/1890594017 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c00be90 con 0x7f232406dbe0 2023-03-03T10:13:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f232a321700 1 -- 172.21.15.7:0/1890594017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f232406dbe0 msgr2=0x7f232406e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:30.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f232a321700 1 --2- 172.21.15.7:0/1890594017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f232406dbe0 0x7f232406e040 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f231c009d90 tx=0x7f231c00b330 comp rx=0 tx=0).stop 2023-03-03T10:13:30.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f232a321700 1 -- 172.21.15.7:0/1890594017 shutdown_connections 2023-03-03T10:13:30.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f232a321700 1 --2- 172.21.15.7:0/1890594017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f232406dbe0 0x7f232406e040 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:30.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f232a321700 1 --2- 172.21.15.7:0/1890594017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23240c5190 0x7f232406d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:30.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f232a321700 1 -- 172.21.15.7:0/1890594017 >> 172.21.15.7:0/1890594017 conn(0x7f23240a9430 msgr2=0x7f23240c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:13:30.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f232a321700 1 -- 172.21.15.7:0/1890594017 shutdown_connections 2023-03-03T10:13:30.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.531+0000 7f232a321700 1 -- 172.21.15.7:0/1890594017 wait complete. 2023-03-03T10:13:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.533+0000 7f232a321700 1 Processor -- start 2023-03-03T10:13:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.533+0000 7f232a321700 1 -- start start 2023-03-03T10:13:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f232a321700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232406dbe0 0x7f23241ed720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f232a321700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23240a6140 0x7f2324070730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f232a321700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23240c5190 0x7f23241ea130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f232a321700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232414f550 con 0x7f23240c5190 2023-03-03T10:13:30.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f232a321700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23241e9d10 con 0x7f23240a6140 2023-03-03T10:13:30.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f232a321700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23241edc60 con 0x7f232406dbe0 2023-03-03T10:13:30.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f23288be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23240c5190 0x7f23241ea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:30.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f23288be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23240c5190 0x7f23241ea130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59576/0 (socket says 172.21.15.7:59576) 2023-03-03T10:13:30.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f23288be700 1 -- 172.21.15.7:0/1716341751 learned_addr learned my addr 172.21.15.7:0/1716341751 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:13:30.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f23237fe700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23240a6140 0x7f2324070730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:30.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.534+0000 7f2323fff700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232406dbe0 0x7f23241ed720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:30.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f23288be700 1 -- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232406dbe0 msgr2=0x7f23241ed720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:30.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f23288be700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232406dbe0 0x7f23241ed720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:30.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f23288be700 1 -- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23240a6140 msgr2=0x7f2324070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:30.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f23288be700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23240a6140 0x7f2324070730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:30.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f23288be700 1 -- 172.21.15.7:0/1716341751 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23140094a0 con 0x7f23240c5190 2023-03-03T10:13:30.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f23288be700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23240c5190 0x7f23241ea130 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f23240c6870 tx=0x7f231c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:13:30.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f2320ff9700 1 -- 172.21.15.7:0/1716341751 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c004800 con 0x7f23240c5190 2023-03-03T10:13:30.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f2320ff9700 1 -- 172.21.15.7:0/1716341751 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c020070 con 0x7f23240c5190 2023-03-03T10:13:30.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f232a321700 1 -- 172.21.15.7:0/1716341751 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c009a70 con 0x7f23240c5190 2023-03-03T10:13:30.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f2320ff9700 1 -- 172.21.15.7:0/1716341751 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c0276c0 con 0x7f23240c5190 2023-03-03T10:13:30.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.535+0000 7f232a321700 1 -- 172.21.15.7:0/1716341751 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23240aa520 con 0x7f23240c5190 2023-03-03T10:13:30.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.536+0000 7f232a321700 1 -- 172.21.15.7:0/1716341751 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23240a6510 con 0x7f23240c5190 2023-03-03T10:13:30.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.536+0000 7f2320ff9700 1 -- 172.21.15.7:0/1716341751 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f231c02e070 con 0x7f23240c5190 2023-03-03T10:13:30.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.537+0000 7f2320ff9700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23040682f0 0x7f230406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:30.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.537+0000 7f2320ff9700 1 -- 172.21.15.7:0/1716341751 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f231c022070 con 0x7f23240c5190 2023-03-03T10:13:30.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.537+0000 7f2323fff700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23040682f0 0x7f230406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:30.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.538+0000 7f2323fff700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23040682f0 0x7f230406a7a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f230c007fd0 tx=0x7f230c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:13:30.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.541+0000 7f2320ff9700 1 -- 172.21.15.7:0/1716341751 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f231c054010 con 0x7f23240c5190 2023-03-03T10:13:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.779+0000 7f232a321700 1 -- 172.21.15.7:0/1716341751 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23240c55e0 con 0x7f23240c5190 2023-03-03T10:13:30.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.779+0000 7f2320ff9700 1 -- 172.21.15.7:0/1716341751 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f231c057630 con 0x7f23240c5190 2023-03-03T10:13:30.782 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:13:30.782 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:13:30.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 -- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23040682f0 msgr2=0x7f230406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:30.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23040682f0 0x7f230406a7a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f230c007fd0 tx=0x7f230c009040 comp rx=0 tx=0).stop 2023-03-03T10:13:30.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 -- 172.21.15.7:0/1716341751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23240c5190 msgr2=0x7f23241ea130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23240c5190 0x7f23241ea130 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f23240c6870 tx=0x7f231c004ad0 comp rx=0 tx=0).stop 2023-03-03T10:13:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 -- 172.21.15.7:0/1716341751 shutdown_connections 2023-03-03T10:13:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23040682f0 0x7f230406a7a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23240c5190 0x7f23241ea130 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f232406dbe0 0x7f23241ed720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:30.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 --2- 172.21.15.7:0/1716341751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23240a6140 0x7f2324070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 -- 172.21.15.7:0/1716341751 >> 172.21.15.7:0/1716341751 conn(0x7f23240a9430 msgr2=0x7f2324154340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:13:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 -- 172.21.15.7:0/1716341751 shutdown_connections 2023-03-03T10:13:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:30.781+0000 7f232a321700 1 -- 172.21.15.7:0/1716341751 wait complete. 2023-03-03T10:13:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:13:30.796 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before thrashing 2023-03-03T10:13:32.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:13:32.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.934+0000 7f255852c700 1 Processor -- start 2023-03-03T10:13:32.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.934+0000 7f255852c700 1 -- start start 2023-03-03T10:13:32.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f255852c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f255006e020 0x7f2550071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:32.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f255852c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2550072520 0x7f2550151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f255852c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2550154910 con 0x7f2550155390 2023-03-03T10:13:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f255852c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2550070b90 con 0x7f255006e020 2023-03-03T10:13:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f255852c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2550151320 con 0x7f2550072520 2023-03-03T10:13:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f2556ac9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2550072520 0x7f2550151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f2555ac7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f255006e020 0x7f2550071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f2555ac7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f255006e020 0x7f2550071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:55358/0 (socket says 172.21.15.7:55358) 2023-03-03T10:13:32.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f2556ac9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2550072520 0x7f2550151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:39426/0 (socket says 172.21.15.7:39426) 2023-03-03T10:13:32.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f2555ac7700 1 -- 172.21.15.7:0/4274043145 learned_addr learned my addr 172.21.15.7:0/4274043145 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:13:32.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.935+0000 7f25562c8700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f2550155390 0x7f25500c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:57082/0 (socket says 172.21.15.7:57082) 2023-03-03T10:13:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f2556ac9700 1 -- 172.21.15.7:0/4274043145 >> v2:172.21.15.143:3300/0 conn(0x7f255006e020 msgr2=0x7f2550071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:13:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f2556ac9700 1 --2- 172.21.15.7:0/4274043145 >> v2:172.21.15.143:3300/0 conn(0x7f255006e020 0x7f2550071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f2556ac9700 1 -- 172.21.15.7:0/4274043145 >> v1:172.21.15.143:6789/0 conn(0x7f2550155390 legacy=0x7f25500c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f25552c6700 1 -- 172.21.15.7:0/4274043145 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39058090 0 0) 0x7f2550154910 con 0x7f2550155390 2023-03-03T10:13:32.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f2556ac9700 1 -- 172.21.15.7:0/4274043145 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25500c4540 con 0x7f2550072520 2023-03-03T10:13:32.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f2555ac7700 1 --2- 172.21.15.7:0/4274043145 >> v2:172.21.15.143:3300/0 conn(0x7f255006e020 0x7f2550071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:13:32.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f2556ac9700 1 --2- 172.21.15.7:0/4274043145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2550072520 0x7f2550151770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2548009d90 tx=0x7f254800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab90bccb287530da server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:13:32.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f25552c6700 1 -- 172.21.15.7:0/4274043145 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254800e070 con 0x7f2550072520 2023-03-03T10:13:32.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f25552c6700 1 -- 172.21.15.7:0/4274043145 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254800bb20 con 0x7f2550072520 2023-03-03T10:13:32.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f25552c6700 1 -- 172.21.15.7:0/4274043145 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548012430 con 0x7f2550072520 2023-03-03T10:13:32.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f255852c700 1 -- 172.21.15.7:0/4274043145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2550072520 msgr2=0x7f2550151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:32.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.936+0000 7f255852c700 1 --2- 172.21.15.7:0/4274043145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2550072520 0x7f2550151770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2548009d90 tx=0x7f254800b330 comp rx=0 tx=0).stop 2023-03-03T10:13:32.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.937+0000 7f255852c700 1 -- 172.21.15.7:0/4274043145 shutdown_connections 2023-03-03T10:13:32.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.937+0000 7f255852c700 1 --2- 172.21.15.7:0/4274043145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2550072520 0x7f2550151770 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:32.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.937+0000 7f255852c700 1 --2- 172.21.15.7:0/4274043145 >> v2:172.21.15.143:3300/0 conn(0x7f255006e020 0x7f2550071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:32.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.937+0000 7f255852c700 1 -- 172.21.15.7:0/4274043145 >> 172.21.15.7:0/4274043145 conn(0x7f255006c110 msgr2=0x7f255006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:13:32.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.937+0000 7f255852c700 1 -- 172.21.15.7:0/4274043145 shutdown_connections 2023-03-03T10:13:32.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.937+0000 7f255852c700 1 -- 172.21.15.7:0/4274043145 wait complete. 2023-03-03T10:13:32.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.938+0000 7f255852c700 1 Processor -- start 2023-03-03T10:13:32.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.938+0000 7f255852c700 1 -- start start 2023-03-03T10:13:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.938+0000 7f255852c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f255006e020 0x7f25501e5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.938+0000 7f255852c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2550072520 0x7f25500c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.938+0000 7f255852c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2550155390 0x7f25500aab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.938+0000 7f255852c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25500ac8d0 con 0x7f2550155390 2023-03-03T10:13:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.938+0000 7f255852c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25501e6200 con 0x7f2550072520 2023-03-03T10:13:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.938+0000 7f255852c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25500abd50 con 0x7f255006e020 2023-03-03T10:13:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2556ac9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2550155390 0x7f25500aab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2556ac9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2550155390 0x7f25500aab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59592/0 (socket says 172.21.15.7:59592) 2023-03-03T10:13:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2556ac9700 1 -- 172.21.15.7:0/3405767873 learned_addr learned my addr 172.21.15.7:0/3405767873 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:13:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2555ac7700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2550072520 0x7f25500c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f25562c8700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f255006e020 0x7f25501e5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2556ac9700 1 -- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f255006e020 msgr2=0x7f25501e5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2556ac9700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f255006e020 0x7f25501e5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2556ac9700 1 -- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2550072520 msgr2=0x7f25500c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2556ac9700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2550072520 0x7f25500c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2556ac9700 1 -- 172.21.15.7:0/3405767873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254000b7f0 con 0x7f2550155390 2023-03-03T10:13:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2556ac9700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2550155390 0x7f25500aab10 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2550151460 tx=0x7f2548004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:13:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2546ffd700 1 -- 172.21.15.7:0/3405767873 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548012430 con 0x7f2550155390 2023-03-03T10:13:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2546ffd700 1 -- 172.21.15.7:0/3405767873 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2548020070 con 0x7f2550155390 2023-03-03T10:13:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f255852c700 1 -- 172.21.15.7:0/3405767873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2548009a70 con 0x7f2550155390 2023-03-03T10:13:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f2546ffd700 1 -- 172.21.15.7:0/3405767873 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548004280 con 0x7f2550155390 2023-03-03T10:13:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.939+0000 7f255852c700 1 -- 172.21.15.7:0/3405767873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2550070540 con 0x7f2550155390 2023-03-03T10:13:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.940+0000 7f255852c700 1 -- 172.21.15.7:0/3405767873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25500702a0 con 0x7f2550155390 2023-03-03T10:13:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.941+0000 7f2546ffd700 1 -- 172.21.15.7:0/3405767873 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f25480043e0 con 0x7f2550155390 2023-03-03T10:13:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.941+0000 7f2546ffd700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25340682f0 0x7f253406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:13:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.941+0000 7f2546ffd700 1 -- 172.21.15.7:0/3405767873 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2548022070 con 0x7f2550155390 2023-03-03T10:13:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.941+0000 7f25562c8700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25340682f0 0x7f253406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:13:32.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.942+0000 7f25562c8700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25340682f0 0x7f253406a7a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f254c000f60 tx=0x7f254c007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:13:32.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:32.946+0000 7f2546ffd700 1 -- 172.21.15.7:0/3405767873 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2548054540 con 0x7f2550155390 2023-03-03T10:13:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.186+0000 7f255852c700 1 -- 172.21.15.7:0/3405767873 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2550070820 con 0x7f2550155390 2023-03-03T10:13:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.186+0000 7f2546ffd700 1 -- 172.21.15.7:0/3405767873 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2548057b60 con 0x7f2550155390 2023-03-03T10:13:33.189 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:13:33.189 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:13:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.188+0000 7f255852c700 1 -- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25340682f0 msgr2=0x7f253406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.188+0000 7f255852c700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25340682f0 0x7f253406a7a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f254c000f60 tx=0x7f254c007c90 comp rx=0 tx=0).stop 2023-03-03T10:13:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.188+0000 7f255852c700 1 -- 172.21.15.7:0/3405767873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2550155390 msgr2=0x7f25500aab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:13:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.188+0000 7f255852c700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2550155390 0x7f25500aab10 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2550151460 tx=0x7f2548004ad0 comp rx=0 tx=0).stop 2023-03-03T10:13:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.189+0000 7f255852c700 1 -- 172.21.15.7:0/3405767873 shutdown_connections 2023-03-03T10:13:33.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.189+0000 7f255852c700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25340682f0 0x7f253406a7a0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:33.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.189+0000 7f255852c700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2550155390 0x7f25500aab10 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:33.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.189+0000 7f255852c700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f255006e020 0x7f25501e5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:33.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.189+0000 7f255852c700 1 --2- 172.21.15.7:0/3405767873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2550072520 0x7f25500c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:13:33.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.189+0000 7f255852c700 1 -- 172.21.15.7:0/3405767873 >> 172.21.15.7:0/3405767873 conn(0x7f255006c110 msgr2=0x7f25500c25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:13:33.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.189+0000 7f255852c700 1 -- 172.21.15.7:0/3405767873 shutdown_connections 2023-03-03T10:13:33.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:13:33.189+0000 7f255852c700 1 -- 172.21.15.7:0/3405767873 wait complete. 2023-03-03T10:13:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:13:33.203 INFO:tasks.mds_thrash.fs.[b]:waiting for 55 secs before thrashing 2023-03-03T10:13:57.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:13:57.127 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:27.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:27.192 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:28.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:14:28.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.341+0000 7ffbc51ba700 1 Processor -- start 2023-03-03T10:14:28.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.341+0000 7ffbc51ba700 1 -- start start 2023-03-03T10:14:28.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbc51ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 0x7ffbc00717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:28.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbc51ba700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffbc0071d20 0x7ffbc01583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:28.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbc51ba700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc01588f0 con 0x7ffbc00aad20 2023-03-03T10:14:28.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbc51ba700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc00728b0 con 0x7ffbc0071d20 2023-03-03T10:14:28.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbc51ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0155070 con 0x7ffbc014b020 2023-03-03T10:14:28.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbe59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ffbc00aad20 0x7ffbc0157ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53604/0 (socket says 172.21.15.7:53604) 2023-03-03T10:14:28.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbe59c700 1 -- 172.21.15.7:0/4202897431 learned_addr learned my addr 172.21.15.7:0/4202897431 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:14:28.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbf59e700 1 --2- 172.21.15.7:0/4202897431 >> v2:172.21.15.7:3300/0 conn(0x7ffbc0071d20 0x7ffbc01583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:28.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbed9d700 1 --2- 172.21.15.7:0/4202897431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 0x7ffbc00717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:28.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbdd9b700 1 -- 172.21.15.7:0/4202897431 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854971466 0 0) 0x7ffbc01588f0 con 0x7ffbc00aad20 2023-03-03T10:14:28.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbdd9b700 1 -- 172.21.15.7:0/4202897431 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba80035b0 con 0x7ffbc00aad20 2023-03-03T10:14:28.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbf59e700 1 -- 172.21.15.7:0/4202897431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 msgr2=0x7ffbc00717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:14:28.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbf59e700 1 --2- 172.21.15.7:0/4202897431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 0x7ffbc00717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:28.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbf59e700 1 -- 172.21.15.7:0/4202897431 >> v1:172.21.15.7:6789/0 conn(0x7ffbc00aad20 legacy=0x7ffbc0157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:28.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbf59e700 1 -- 172.21.15.7:0/4202897431 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc006f5b0 con 0x7ffbc0071d20 2023-03-03T10:14:28.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbf59e700 1 --2- 172.21.15.7:0/4202897431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc0071d20 0x7ffbc01583b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009d90 tx=0x7ffbb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e1fa34a65b6c0c6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:14:28.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.342+0000 7ffbbdd9b700 1 -- 172.21.15.7:0/4202897431 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb800e070 con 0x7ffbc0071d20 2023-03-03T10:14:28.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.343+0000 7ffbbdd9b700 1 -- 172.21.15.7:0/4202897431 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb800bb20 con 0x7ffbc0071d20 2023-03-03T10:14:28.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.343+0000 7ffbbdd9b700 1 -- 172.21.15.7:0/4202897431 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8012430 con 0x7ffbc0071d20 2023-03-03T10:14:28.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.343+0000 7ffbc51ba700 1 -- 172.21.15.7:0/4202897431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc0071d20 msgr2=0x7ffbc01583b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:28.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.343+0000 7ffbc51ba700 1 --2- 172.21.15.7:0/4202897431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc0071d20 0x7ffbc01583b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8009d90 tx=0x7ffbb800b330 comp rx=0 tx=0).stop 2023-03-03T10:14:28.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.343+0000 7ffbc51ba700 1 -- 172.21.15.7:0/4202897431 shutdown_connections 2023-03-03T10:14:28.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.343+0000 7ffbc51ba700 1 --2- 172.21.15.7:0/4202897431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc0071d20 0x7ffbc01583b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:28.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.343+0000 7ffbc51ba700 1 --2- 172.21.15.7:0/4202897431 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 0x7ffbc00717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:28.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.343+0000 7ffbc51ba700 1 -- 172.21.15.7:0/4202897431 >> 172.21.15.7:0/4202897431 conn(0x7ffbc00c8900 msgr2=0x7ffbc00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:14:28.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.343+0000 7ffbc51ba700 1 -- 172.21.15.7:0/4202897431 shutdown_connections 2023-03-03T10:14:28.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.343+0000 7ffbc51ba700 1 -- 172.21.15.7:0/4202897431 wait complete. 2023-03-03T10:14:28.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbc51ba700 1 Processor -- start 2023-03-03T10:14:28.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbc51ba700 1 -- start start 2023-03-03T10:14:28.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbc51ba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0071d20 0x7ffbc0157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:28.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbc51ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc00aad20 0x7ffbc01ed860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:28.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbc51ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 0x7ffbc01ea270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:28.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbc51ba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0156820 con 0x7ffbc00aad20 2023-03-03T10:14:28.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbc51ba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc01e9d90 con 0x7ffbc0071d20 2023-03-03T10:14:28.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbc51ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0155f80 con 0x7ffbc014b020 2023-03-03T10:14:28.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbbe59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc00aad20 0x7ffbc01ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:28.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbbe59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc00aad20 0x7ffbc01ed860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59630/0 (socket says 172.21.15.7:59630) 2023-03-03T10:14:28.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.345+0000 7ffbbe59c700 1 -- 172.21.15.7:0/984814218 learned_addr learned my addr 172.21.15.7:0/984814218 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:14:28.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbbf59e700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 0x7ffbc01ea270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:28.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbbed9d700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0071d20 0x7ffbc0157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:28.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbbe59c700 1 -- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 msgr2=0x7ffbc01ea270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:28.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbbe59c700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 0x7ffbc01ea270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:28.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbbe59c700 1 -- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0071d20 msgr2=0x7ffbc0157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:28.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbbe59c700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0071d20 0x7ffbc0157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:28.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbbe59c700 1 -- 172.21.15.7:0/984814218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb8009a70 con 0x7ffbc00aad20 2023-03-03T10:14:28.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbbf59e700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 0x7ffbc01ea270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:14:28.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbbed9d700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0071d20 0x7ffbc0157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:14:28.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbbe59c700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc00aad20 0x7ffbc01ed860 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0005910 tx=0x7ffbb000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:14:28.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffba77fe700 1 -- 172.21.15.7:0/984814218 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0005d70 con 0x7ffbc00aad20 2023-03-03T10:14:28.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffba77fe700 1 -- 172.21.15.7:0/984814218 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb00070d0 con 0x7ffbc00aad20 2023-03-03T10:14:28.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbc51ba700 1 -- 172.21.15.7:0/984814218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc006fb70 con 0x7ffbc00aad20 2023-03-03T10:14:28.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffba77fe700 1 -- 172.21.15.7:0/984814218 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0007440 con 0x7ffbc00aad20 2023-03-03T10:14:28.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.346+0000 7ffbc51ba700 1 -- 172.21.15.7:0/984814218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc006f890 con 0x7ffbc00aad20 2023-03-03T10:14:28.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.347+0000 7ffbc51ba700 1 -- 172.21.15.7:0/984814218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc0155350 con 0x7ffbc00aad20 2023-03-03T10:14:28.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.348+0000 7ffba77fe700 1 -- 172.21.15.7:0/984814218 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffbb0019030 con 0x7ffbc00aad20 2023-03-03T10:14:28.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.348+0000 7ffba77fe700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbac068320 0x7ffbac06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:28.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.348+0000 7ffba77fe700 1 -- 172.21.15.7:0/984814218 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ffbb0085370 con 0x7ffbc00aad20 2023-03-03T10:14:28.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.348+0000 7ffbbed9d700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbac068320 0x7ffbac06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:28.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.349+0000 7ffbbed9d700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbac068320 0x7ffbac06a7d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009b90 tx=0x7ffbb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:14:28.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.353+0000 7ffba77fe700 1 -- 172.21.15.7:0/984814218 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffbb004b5e0 con 0x7ffbc00aad20 2023-03-03T10:14:28.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.589+0000 7ffbc51ba700 1 -- 172.21.15.7:0/984814218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbc01e9f00 con 0x7ffbc00aad20 2023-03-03T10:14:28.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.589+0000 7ffba77fe700 1 -- 172.21.15.7:0/984814218 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ffbc01e9f00 con 0x7ffbc00aad20 2023-03-03T10:14:28.592 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:14:28.592 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:14:28.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 -- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbac068320 msgr2=0x7ffbac06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbac068320 0x7ffbac06a7d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009b90 tx=0x7ffbb4008040 comp rx=0 tx=0).stop 2023-03-03T10:14:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 -- 172.21.15.7:0/984814218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc00aad20 msgr2=0x7ffbc01ed860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc00aad20 0x7ffbc01ed860 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0005910 tx=0x7ffbb000c860 comp rx=0 tx=0).stop 2023-03-03T10:14:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 -- 172.21.15.7:0/984814218 shutdown_connections 2023-03-03T10:14:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbc014b020 0x7ffbc01ea270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:28.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbc0071d20 0x7ffbc0157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbac068320 0x7ffbac06a7d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 --2- 172.21.15.7:0/984814218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbc00aad20 0x7ffbc01ed860 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 -- 172.21.15.7:0/984814218 >> 172.21.15.7:0/984814218 conn(0x7ffbc00c8900 msgr2=0x7ffbc00ac740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:14:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 -- 172.21.15.7:0/984814218 shutdown_connections 2023-03-03T10:14:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:28.591+0000 7ffbc51ba700 1 -- 172.21.15.7:0/984814218 wait complete. 2023-03-03T10:14:28.595 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:14:28.605 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-03-03T10:14:49.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:14:49.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.378+0000 7f98cf1d0700 1 Processor -- start 2023-03-03T10:14:49.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.378+0000 7f98cf1d0700 1 -- start start 2023-03-03T10:14:49.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98cf1d0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f98c814fb00 0x7f98c8156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:49.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98cf1d0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f98c80c1da0 0x7f98c80c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98cf1d0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c8155840 con 0x7f98c80c4370 2023-03-03T10:14:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98cf1d0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c8072940 con 0x7f98c80c1da0 2023-03-03T10:14:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98cf1d0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c81567d0 con 0x7f98c814fb00 2023-03-03T10:14:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98c7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f98c80c1da0 0x7f98c80c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98c7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f98c80c1da0 0x7f98c80c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59648/0 (socket says 172.21.15.7:59648) 2023-03-03T10:14:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98c7fff700 1 -- 172.21.15.7:0/1113321367 learned_addr learned my addr 172.21.15.7:0/1113321367 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:14:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98ccf6c700 1 --2- 172.21.15.7:0/1113321367 >> v2:172.21.15.143:3300/0 conn(0x7f98c814fb00 0x7f98c8156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98c7fff700 1 -- 172.21.15.7:0/1113321367 >> v2:172.21.15.143:3300/0 conn(0x7f98c814fb00 msgr2=0x7f98c8156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:14:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98c7fff700 1 --2- 172.21.15.7:0/1113321367 >> v2:172.21.15.143:3300/0 conn(0x7f98c814fb00 0x7f98c8156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98c7fff700 1 -- 172.21.15.7:0/1113321367 >> v1:172.21.15.143:6789/0 conn(0x7f98c80c4370 legacy=0x7f98c80c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:14:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.379+0000 7f98c7fff700 1 -- 172.21.15.7:0/1113321367 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c80c2330 con 0x7f98c80c1da0 2023-03-03T10:14:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98c7fff700 1 --2- 172.21.15.7:0/1113321367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c1da0 0x7f98c80c3e30 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f98c80747d0 tx=0x7f98b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c764e8d8466e550 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:14:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98c77fe700 1 -- 172.21.15.7:0/1113321367 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b400b870 con 0x7f98c80c1da0 2023-03-03T10:14:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98c77fe700 1 -- 172.21.15.7:0/1113321367 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b400b9d0 con 0x7f98c80c1da0 2023-03-03T10:14:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98c77fe700 1 -- 172.21.15.7:0/1113321367 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b400bd60 con 0x7f98c80c1da0 2023-03-03T10:14:49.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1113321367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c1da0 msgr2=0x7f98c80c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:49.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98cf1d0700 1 --2- 172.21.15.7:0/1113321367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c1da0 0x7f98c80c3e30 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f98c80747d0 tx=0x7f98b4009bd0 comp rx=0 tx=0).stop 2023-03-03T10:14:49.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1113321367 shutdown_connections 2023-03-03T10:14:49.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98cf1d0700 1 --2- 172.21.15.7:0/1113321367 >> v2:172.21.15.143:3300/0 conn(0x7f98c814fb00 0x7f98c8156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:49.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98cf1d0700 1 --2- 172.21.15.7:0/1113321367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c1da0 0x7f98c80c3e30 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:49.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1113321367 >> 172.21.15.7:0/1113321367 conn(0x7f98c806c110 msgr2=0x7f98c806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:14:49.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1113321367 shutdown_connections 2023-03-03T10:14:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.380+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1113321367 wait complete. 2023-03-03T10:14:49.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.381+0000 7f98cf1d0700 1 Processor -- start 2023-03-03T10:14:49.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.382+0000 7f98cf1d0700 1 -- start start 2023-03-03T10:14:49.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.382+0000 7f98cf1d0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c1da0 0x7f98c81502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:49.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.382+0000 7f98cf1d0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c4370 0x7f98c8154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:49.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.382+0000 7f98cf1d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c814fb00 0x7f98c80c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:49.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.382+0000 7f98cf1d0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81ec200 con 0x7f98c80c4370 2023-03-03T10:14:49.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.382+0000 7f98cf1d0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c814b220 con 0x7f98c814fb00 2023-03-03T10:14:49.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.382+0000 7f98cf1d0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81555e0 con 0x7f98c80c1da0 2023-03-03T10:14:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.382+0000 7f98c7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c4370 0x7f98c8154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c4370 0x7f98c8154050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59652/0 (socket says 172.21.15.7:59652) 2023-03-03T10:14:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c7fff700 1 -- 172.21.15.7:0/1947748972 learned_addr learned my addr 172.21.15.7:0/1947748972 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:14:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98cd76d700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c814fb00 0x7f98c80c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98ccf6c700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c1da0 0x7f98c81502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c7fff700 1 -- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c1da0 msgr2=0x7f98c81502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c7fff700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c1da0 0x7f98c81502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c7fff700 1 -- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c814fb00 msgr2=0x7f98c80c5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c7fff700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c814fb00 0x7f98c80c5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c7fff700 1 -- 172.21.15.7:0/1947748972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b4009550 con 0x7f98c80c4370 2023-03-03T10:14:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c7fff700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c4370 0x7f98c8154050 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f98b4000c00 tx=0x7f98b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:14:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c57fa700 1 -- 172.21.15.7:0/1947748972 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4004310 con 0x7f98c80c4370 2023-03-03T10:14:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c57fa700 1 -- 172.21.15.7:0/1947748972 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b401e070 con 0x7f98c80c4370 2023-03-03T10:14:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1947748972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c8153090 con 0x7f98c80c4370 2023-03-03T10:14:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98c57fa700 1 -- 172.21.15.7:0/1947748972 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b40036a0 con 0x7f98c80c4370 2023-03-03T10:14:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.383+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1947748972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c8152db0 con 0x7f98c80c4370 2023-03-03T10:14:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.384+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1947748972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c8155a90 con 0x7f98c80c4370 2023-03-03T10:14:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.385+0000 7f98c57fa700 1 -- 172.21.15.7:0/1947748972 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f98b4029020 con 0x7f98c80c4370 2023-03-03T10:14:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.385+0000 7f98c57fa700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98ac0682f0 0x7f98ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:14:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.385+0000 7f98c57fa700 1 -- 172.21.15.7:0/1947748972 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f98b4087130 con 0x7f98c80c4370 2023-03-03T10:14:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.385+0000 7f98ccf6c700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98ac0682f0 0x7f98ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:14:49.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.386+0000 7f98ccf6c700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98ac0682f0 0x7f98ac06a7a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f98bc006b20 tx=0x7f98bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:14:49.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.390+0000 7f98c57fa700 1 -- 172.21.15.7:0/1947748972 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f98b4053450 con 0x7f98c80c4370 2023-03-03T10:14:49.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.631+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1947748972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98c806e410 con 0x7f98c80c4370 2023-03-03T10:14:49.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.631+0000 7f98c57fa700 1 -- 172.21.15.7:0/1947748972 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f98b4056a70 con 0x7f98c80c4370 2023-03-03T10:14:49.634 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:14:49.634 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:14:49.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98ac0682f0 msgr2=0x7f98ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98ac0682f0 0x7f98ac06a7a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f98bc006b20 tx=0x7f98bc008040 comp rx=0 tx=0).stop 2023-03-03T10:14:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1947748972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c4370 msgr2=0x7f98c8154050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:14:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c4370 0x7f98c8154050 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f98b4000c00 tx=0x7f98b4004750 comp rx=0 tx=0).stop 2023-03-03T10:14:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1947748972 shutdown_connections 2023-03-03T10:14:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c814fb00 0x7f98c80c5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:49.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c1da0 0x7f98c81502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98ac0682f0 0x7f98ac06a7a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 --2- 172.21.15.7:0/1947748972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c4370 0x7f98c8154050 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:14:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1947748972 >> 172.21.15.7:0/1947748972 conn(0x7f98c806c110 msgr2=0x7f98c806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:14:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1947748972 shutdown_connections 2023-03-03T10:14:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:14:49.633+0000 7f98cf1d0700 1 -- 172.21.15.7:0/1947748972 wait complete. 2023-03-03T10:14:49.637 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:14:49.648 INFO:tasks.mds_thrash.fs.[a]:waiting for 78 secs before thrashing 2023-03-03T10:14:57.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:14:57.337 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:27.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:27.409 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:50.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.744+0000 7f7e0b4ef700 1 Processor -- start 2023-03-03T10:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.745+0000 7f7e0b4ef700 1 -- start start 2023-03-03T10:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.745+0000 7f7e0b4ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c5b00 0x7f7e040bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.745+0000 7f7e0b4ef700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e040c08d0 con 0x7f7e04151e80 2023-03-03T10:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.745+0000 7f7e0b4ef700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e04154930 con 0x7f7e04155410 2023-03-03T10:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.745+0000 7f7e0b4ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e04152250 con 0x7f7e040c5b00 2023-03-03T10:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.745+0000 7f7e0928b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7e04151e80 0x7f7e040c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53666/0 (socket says 172.21.15.7:53666) 2023-03-03T10:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.745+0000 7f7e0928b700 1 -- 172.21.15.7:0/3085838779 learned_addr learned my addr 172.21.15.7:0/3085838779 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7e09a8c700 1 --2- 172.21.15.7:0/3085838779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c5b00 0x7f7e040bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7dfbfff700 1 -- 172.21.15.7:0/3085838779 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814436530 0 0) 0x7f7e040c08d0 con 0x7f7e04151e80 2023-03-03T10:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7dfbfff700 1 -- 172.21.15.7:0/3085838779 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df00035b0 con 0x7f7e04151e80 2023-03-03T10:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7dfbfff700 1 -- 172.21.15.7:0/3085838779 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272142565 0 0) 0x7f7e04154930 con 0x7f7e04155410 2023-03-03T10:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7dfbfff700 1 -- 172.21.15.7:0/3085838779 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e040c08d0 con 0x7f7e04155410 2023-03-03T10:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7e09a8c700 1 -- 172.21.15.7:0/3085838779 >> v1:172.21.15.143:6789/0 conn(0x7f7e04155410 legacy=0x7f7e040bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7e09a8c700 1 -- 172.21.15.7:0/3085838779 >> v1:172.21.15.7:6789/0 conn(0x7f7e04151e80 legacy=0x7f7e040c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7dfbfff700 1 -- 172.21.15.7:0/3085838779 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1276329037 0 0) 0x7f7df00035b0 con 0x7f7e04151e80 2023-03-03T10:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7e09a8c700 1 -- 172.21.15.7:0/3085838779 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04154c10 con 0x7f7e040c5b00 2023-03-03T10:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7e09a8c700 1 --2- 172.21.15.7:0/3085838779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c5b00 0x7f7e040bfff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc009dc0 tx=0x7f7dfc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1177bb0dd1548f9e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7dfbfff700 1 -- 172.21.15.7:0/3085838779 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc00e070 con 0x7f7e040c5b00 2023-03-03T10:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7dfbfff700 1 -- 172.21.15.7:0/3085838779 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc003a20 con 0x7f7e040c5b00 2023-03-03T10:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.746+0000 7f7dfbfff700 1 -- 172.21.15.7:0/3085838779 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00001670 con 0x7f7e040c5b00 2023-03-03T10:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.747+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/3085838779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c5b00 msgr2=0x7f7e040bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.747+0000 7f7e0b4ef700 1 --2- 172.21.15.7:0/3085838779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c5b00 0x7f7e040bfff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc009dc0 tx=0x7f7dfc00b330 comp rx=0 tx=0).stop 2023-03-03T10:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.747+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/3085838779 shutdown_connections 2023-03-03T10:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.747+0000 7f7e0b4ef700 1 --2- 172.21.15.7:0/3085838779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e040c5b00 0x7f7e040bfff0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.747+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/3085838779 >> 172.21.15.7:0/3085838779 conn(0x7f7e04071630 msgr2=0x7f7e0406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.747+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/3085838779 shutdown_connections 2023-03-03T10:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.747+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/3085838779 wait complete. 2023-03-03T10:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.748+0000 7f7e0b4ef700 1 Processor -- start 2023-03-03T10:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.748+0000 7f7e0b4ef700 1 -- start start 2023-03-03T10:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.749+0000 7f7e0b4ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c5b00 0x7f7e0414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.749+0000 7f7e0b4ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e04151e80 0x7f7e0414e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.749+0000 7f7e0b4ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e04155410 0x7f7e0414e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.749+0000 7f7e0b4ef700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0414f590 con 0x7f7e040c5b00 2023-03-03T10:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.749+0000 7f7e0b4ef700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0414ed10 con 0x7f7e04155410 2023-03-03T10:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.749+0000 7f7e0b4ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0414efd0 con 0x7f7e04151e80 2023-03-03T10:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.749+0000 7f7e0928b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c5b00 0x7f7e0414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.749+0000 7f7e0928b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c5b00 0x7f7e0414ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59692/0 (socket says 172.21.15.7:59692) 2023-03-03T10:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.749+0000 7f7e0928b700 1 -- 172.21.15.7:0/1248561968 learned_addr learned my addr 172.21.15.7:0/1248561968 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e08a8a700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e04151e80 0x7f7e0414e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e09a8c700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e04155410 0x7f7e0414e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e0928b700 1 -- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e04151e80 msgr2=0x7f7e0414e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:15:50.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e0928b700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e04151e80 0x7f7e0414e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:15:50.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e0928b700 1 -- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e04155410 msgr2=0x7f7e0414e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:15:50.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e0928b700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e04155410 0x7f7e0414e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:15:50.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e0928b700 1 -- 172.21.15.7:0/1248561968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dfc009a70 con 0x7f7e040c5b00 2023-03-03T10:15:50.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e08a8a700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e04151e80 0x7f7e0414e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:15:50.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e0928b700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c5b00 0x7f7e0414ffd0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000c750 tx=0x7f7e0000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:15:50.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7df9ffb700 1 -- 172.21.15.7:0/1248561968 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00007560 con 0x7f7e040c5b00 2023-03-03T10:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7df9ffb700 1 -- 172.21.15.7:0/1248561968 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e000076c0 con 0x7f7e040c5b00 2023-03-03T10:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7df9ffb700 1 -- 172.21.15.7:0/1248561968 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00004ab0 con 0x7f7e040c5b00 2023-03-03T10:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/1248561968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e040c5220 con 0x7f7e040c5b00 2023-03-03T10:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.750+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/1248561968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e04154ef0 con 0x7f7e040c5b00 2023-03-03T10:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.751+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/1248561968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e040c0530 con 0x7f7e040c5b00 2023-03-03T10:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.751+0000 7f7df9ffb700 1 -- 172.21.15.7:0/1248561968 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7e00010070 con 0x7f7e040c5b00 2023-03-03T10:15:50.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.752+0000 7f7df9ffb700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 0x7f7de806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:15:50.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.752+0000 7f7df9ffb700 1 -- 172.21.15.7:0/1248561968 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7e00085260 con 0x7f7e040c5b00 2023-03-03T10:15:50.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.752+0000 7f7e08a8a700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 0x7f7de806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:15:50.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.753+0000 7f7e08a8a700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 0x7f7de806a7a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7df40011e0 tx=0x7f7df4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:15:50.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.757+0000 7f7df9ffb700 1 -- 172.21.15.7:0/1248561968 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7e00051600 con 0x7f7e040c5b00 2023-03-03T10:15:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.996+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/1248561968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e04074bb0 con 0x7f7e040c5b00 2023-03-03T10:15:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.997+0000 7f7df9ffb700 1 -- 172.21.15.7:0/1248561968 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7e00018090 con 0x7f7e040c5b00 2023-03-03T10:15:50.999 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:15:51.000 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:15:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 msgr2=0x7f7de806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:15:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 0x7f7de806a7a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7df40011e0 tx=0x7f7df4009040 comp rx=0 tx=0).stop 2023-03-03T10:15:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/1248561968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c5b00 msgr2=0x7f7e0414ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:15:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c5b00 0x7f7e0414ffd0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000c750 tx=0x7f7e0000cb10 comp rx=0 tx=0).stop 2023-03-03T10:15:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/1248561968 shutdown_connections 2023-03-03T10:15:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e04155410 0x7f7e0414e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:15:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7de80682f0 0x7f7de806a7a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:15:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e040c5b00 0x7f7e0414ffd0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:15:51.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 --2- 172.21.15.7:0/1248561968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e04151e80 0x7f7e0414e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:15:51.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/1248561968 >> 172.21.15.7:0/1248561968 conn(0x7f7e04071630 msgr2=0x7f7e04153750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:15:51.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/1248561968 shutdown_connections 2023-03-03T10:15:51.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:15:50.999+0000 7f7e0b4ef700 1 -- 172.21.15.7:0/1248561968 wait complete. 2023-03-03T10:15:51.003 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:15:51.013 INFO:tasks.mds_thrash.fs.[b]:waiting for 69 secs before thrashing 2023-03-03T10:15:57.477 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:15:57.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:07.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:16:07.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.786+0000 7f0d19caa700 1 Processor -- start 2023-03-03T10:16:07.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.786+0000 7f0d19caa700 1 -- start start 2023-03-03T10:16:07.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d19caa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0d1414b020 0x7f0d140717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:07.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d19caa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0d140aad20 0x7f0d14157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:07.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d19caa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d14071d20 0x7f0d141581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:07.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d19caa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d140728b0 con 0x7f0d140aad20 2023-03-03T10:16:07.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d19caa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d14155070 con 0x7f0d1414b020 2023-03-03T10:16:07.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d19caa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1406f5b0 con 0x7f0d14071d20 2023-03-03T10:16:07.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d0bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0d140aad20 0x7f0d14157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:07.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d0bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0d140aad20 0x7f0d14157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59704/0 (socket says 172.21.15.7:59704) 2023-03-03T10:16:07.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d0bfff700 1 -- 172.21.15.7:0/3600941439 learned_addr learned my addr 172.21.15.7:0/3600941439 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:16:07.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d13fff700 1 --2- 172.21.15.7:0/3600941439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d14071d20 0x7f0d141581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:07.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d137fe700 1 --2- 172.21.15.7:0/3600941439 >> v2:172.21.15.143:3300/0 conn(0x7f0d1414b020 0x7f0d140717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:07.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d0bfff700 1 -- 172.21.15.7:0/3600941439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d14071d20 msgr2=0x7f0d141581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:16:07.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d0bfff700 1 --2- 172.21.15.7:0/3600941439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d14071d20 0x7f0d141581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:07.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d0bfff700 1 -- 172.21.15.7:0/3600941439 >> v2:172.21.15.143:3300/0 conn(0x7f0d1414b020 msgr2=0x7f0d140717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:16:07.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d0bfff700 1 --2- 172.21.15.7:0/3600941439 >> v2:172.21.15.143:3300/0 conn(0x7f0d1414b020 0x7f0d140717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:07.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.787+0000 7f0d0bfff700 1 -- 172.21.15.7:0/3600941439 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1406f890 con 0x7f0d140aad20 2023-03-03T10:16:07.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d0bfff700 1 --2- 172.21.15.7:0/3600941439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d140aad20 0x7f0d14157ca0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0d14070940 tx=0x7f0cfc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97dc9f5bcb548be9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:16:07.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d12ffd700 1 -- 172.21.15.7:0/3600941439 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc011830 con 0x7f0d140aad20 2023-03-03T10:16:07.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d12ffd700 1 -- 172.21.15.7:0/3600941439 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc011990 con 0x7f0d140aad20 2023-03-03T10:16:07.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d12ffd700 1 -- 172.21.15.7:0/3600941439 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc011d20 con 0x7f0d140aad20 2023-03-03T10:16:07.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d19caa700 1 -- 172.21.15.7:0/3600941439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d140aad20 msgr2=0x7f0d14157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d19caa700 1 --2- 172.21.15.7:0/3600941439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d140aad20 0x7f0d14157ca0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0d14070940 tx=0x7f0cfc009bd0 comp rx=0 tx=0).stop 2023-03-03T10:16:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d19caa700 1 -- 172.21.15.7:0/3600941439 shutdown_connections 2023-03-03T10:16:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d19caa700 1 --2- 172.21.15.7:0/3600941439 >> v2:172.21.15.143:3300/0 conn(0x7f0d1414b020 0x7f0d140717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d19caa700 1 --2- 172.21.15.7:0/3600941439 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d14071d20 0x7f0d141581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:07.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d19caa700 1 --2- 172.21.15.7:0/3600941439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d140aad20 0x7f0d14157ca0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:07.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d19caa700 1 -- 172.21.15.7:0/3600941439 >> 172.21.15.7:0/3600941439 conn(0x7f0d140c8900 msgr2=0x7f0d140c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:16:07.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.788+0000 7f0d19caa700 1 -- 172.21.15.7:0/3600941439 shutdown_connections 2023-03-03T10:16:07.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.789+0000 7f0d19caa700 1 -- 172.21.15.7:0/3600941439 wait complete. 2023-03-03T10:16:07.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.790+0000 7f0d19caa700 1 Processor -- start 2023-03-03T10:16:07.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.790+0000 7f0d19caa700 1 -- start start 2023-03-03T10:16:07.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d19caa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d14071d20 0x7f0d14148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:07.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d19caa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d140aad20 0x7f0d141516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:07.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d19caa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1414b020 0x7f0d14147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:07.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d19caa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d141ea490 con 0x7f0d1414b020 2023-03-03T10:16:07.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d19caa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d141e9af0 con 0x7f0d140aad20 2023-03-03T10:16:07.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d19caa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d141e9dd0 con 0x7f0d14071d20 2023-03-03T10:16:07.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d13fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1414b020 0x7f0d14147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:07.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d13fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1414b020 0x7f0d14147c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59714/0 (socket says 172.21.15.7:59714) 2023-03-03T10:16:07.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d13fff700 1 -- 172.21.15.7:0/426533846 learned_addr learned my addr 172.21.15.7:0/426533846 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:16:07.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d0bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d140aad20 0x7f0d141516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:07.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.791+0000 7f0d137fe700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d14071d20 0x7f0d14148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:07.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d13fff700 1 -- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d14071d20 msgr2=0x7f0d14148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:07.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d13fff700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d14071d20 0x7f0d14148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:07.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d13fff700 1 -- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d140aad20 msgr2=0x7f0d141516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:07.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d13fff700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d140aad20 0x7f0d141516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:07.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d13fff700 1 -- 172.21.15.7:0/426533846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc009550 con 0x7f0d1414b020 2023-03-03T10:16:07.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d13fff700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1414b020 0x7f0d14147c80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00eed0 tx=0x7f0d0c00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:16:07.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d10ff9700 1 -- 172.21.15.7:0/426533846 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c010040 con 0x7f0d1414b020 2023-03-03T10:16:07.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d10ff9700 1 -- 172.21.15.7:0/426533846 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c00b9f0 con 0x7f0d1414b020 2023-03-03T10:16:07.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d19caa700 1 -- 172.21.15.7:0/426533846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d14155350 con 0x7f0d1414b020 2023-03-03T10:16:07.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d10ff9700 1 -- 172.21.15.7:0/426533846 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c00bda0 con 0x7f0d1414b020 2023-03-03T10:16:07.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.792+0000 7f0d19caa700 1 -- 172.21.15.7:0/426533846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1406fb70 con 0x7f0d1414b020 2023-03-03T10:16:07.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.793+0000 7f0d19caa700 1 -- 172.21.15.7:0/426533846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d14158720 con 0x7f0d1414b020 2023-03-03T10:16:07.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.793+0000 7f0d10ff9700 1 -- 172.21.15.7:0/426533846 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0d0c014400 con 0x7f0d1414b020 2023-03-03T10:16:07.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.794+0000 7f0d10ff9700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 0x7f0cf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:16:07.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.794+0000 7f0d10ff9700 1 -- 172.21.15.7:0/426533846 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0d0c086560 con 0x7f0d1414b020 2023-03-03T10:16:07.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.794+0000 7f0d137fe700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 0x7f0cf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:16:07.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.795+0000 7f0d137fe700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 0x7f0cf406a7a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009be0 tx=0x7f0d04009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:16:07.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:07.798+0000 7f0d10ff9700 1 -- 172.21.15.7:0/426533846 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0d0c0528a0 con 0x7f0d1414b020 2023-03-03T10:16:08.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.035+0000 7f0d19caa700 1 -- 172.21.15.7:0/426533846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d141ea5d0 con 0x7f0d1414b020 2023-03-03T10:16:08.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.036+0000 7f0d10ff9700 1 -- 172.21.15.7:0/426533846 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0d141ea5d0 con 0x7f0d1414b020 2023-03-03T10:16:08.039 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:16:08.039 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:16:08.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 -- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 msgr2=0x7f0cf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:08.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 0x7f0cf406a7a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009be0 tx=0x7f0d04009210 comp rx=0 tx=0).stop 2023-03-03T10:16:08.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 -- 172.21.15.7:0/426533846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1414b020 msgr2=0x7f0d14147c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:16:08.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1414b020 0x7f0d14147c80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00eed0 tx=0x7f0d0c00ef00 comp rx=0 tx=0).stop 2023-03-03T10:16:08.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 -- 172.21.15.7:0/426533846 shutdown_connections 2023-03-03T10:16:08.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 0x7f0cf406a7a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:08.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1414b020 0x7f0d14147c80 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:08.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d14071d20 0x7f0d14148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:08.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 --2- 172.21.15.7:0/426533846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d140aad20 0x7f0d141516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:16:08.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 -- 172.21.15.7:0/426533846 >> 172.21.15.7:0/426533846 conn(0x7f0d140c8900 msgr2=0x7f0d140c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:16:08.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 -- 172.21.15.7:0/426533846 shutdown_connections 2023-03-03T10:16:08.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:16:08.038+0000 7f0d19caa700 1 -- 172.21.15.7:0/426533846 wait complete. 2023-03-03T10:16:08.042 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:16:08.052 INFO:tasks.mds_thrash.fs.[a]:waiting for 62 secs before thrashing 2023-03-03T10:16:27.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:27.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:57.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:16:57.922 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:00.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:17:00.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.150+0000 7fc3b211d700 1 Processor -- start 2023-03-03T10:17:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.150+0000 7fc3b211d700 1 -- start start 2023-03-03T10:17:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.151+0000 7fc3b211d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc3ac06a2b0 0x7fc3ac06a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.151+0000 7fc3b211d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc3ac156490 0x7fc3ac15cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.151+0000 7fc3b211d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac1558f0 con 0x7fc3ac0c5cb0 2023-03-03T10:17:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.151+0000 7fc3b211d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ac0c3290 con 0x7fc3ac156490 2023-03-03T10:17:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.151+0000 7fc3b211d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ac06bab0 con 0x7fc3ac06a2b0 2023-03-03T10:17:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.151+0000 7fc3abfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc3ac156490 0x7fc3ac15cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.151+0000 7fc3abfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc3ac156490 0x7fc3ac15cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59742/0 (socket says 172.21.15.7:59742) 2023-03-03T10:17:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.151+0000 7fc3abfff700 1 -- 172.21.15.7:0/1219270801 learned_addr learned my addr 172.21.15.7:0/1219270801 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:17:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.151+0000 7fc3abfff700 1 -- 172.21.15.7:0/1219270801 >> v2:172.21.15.143:3300/0 conn(0x7fc3ac06a2b0 msgr2=0x7fc3ac06a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:17:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3abfff700 1 --2- 172.21.15.7:0/1219270801 >> v2:172.21.15.143:3300/0 conn(0x7fc3ac06a2b0 0x7fc3ac06a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3abfff700 1 -- 172.21.15.7:0/1219270801 >> v1:172.21.15.143:6789/0 conn(0x7fc3ac0c5cb0 legacy=0x7fc3ac155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:17:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3abfff700 1 -- 172.21.15.7:0/1219270801 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac0a6180 con 0x7fc3ac156490 2023-03-03T10:17:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3abfff700 1 --2- 172.21.15.7:0/1219270801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 0x7fc3ac15cbd0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0c7c00 tx=0x7fc3a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb20f63155864221 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3aa7fc700 1 -- 172.21.15.7:0/1219270801 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a400e070 con 0x7fc3ac156490 2023-03-03T10:17:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3aa7fc700 1 -- 172.21.15.7:0/1219270801 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a4003a20 con 0x7fc3ac156490 2023-03-03T10:17:00.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3aa7fc700 1 -- 172.21.15.7:0/1219270801 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4003dd0 con 0x7fc3ac156490 2023-03-03T10:17:00.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3b211d700 1 -- 172.21.15.7:0/1219270801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 msgr2=0x7fc3ac15cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:00.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3b211d700 1 --2- 172.21.15.7:0/1219270801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 0x7fc3ac15cbd0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0c7c00 tx=0x7fc3a400b330 comp rx=0 tx=0).stop 2023-03-03T10:17:00.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3b211d700 1 -- 172.21.15.7:0/1219270801 shutdown_connections 2023-03-03T10:17:00.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3b211d700 1 --2- 172.21.15.7:0/1219270801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 0x7fc3ac15cbd0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:00.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3b211d700 1 --2- 172.21.15.7:0/1219270801 >> v2:172.21.15.143:3300/0 conn(0x7fc3ac06a2b0 0x7fc3ac06a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:00.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3b211d700 1 -- 172.21.15.7:0/1219270801 >> 172.21.15.7:0/1219270801 conn(0x7fc3ac0a9430 msgr2=0x7fc3ac0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:17:00.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.152+0000 7fc3b211d700 1 -- 172.21.15.7:0/1219270801 shutdown_connections 2023-03-03T10:17:00.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.153+0000 7fc3b211d700 1 -- 172.21.15.7:0/1219270801 wait complete. 2023-03-03T10:17:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.154+0000 7fc3b211d700 1 Processor -- start 2023-03-03T10:17:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.154+0000 7fc3b211d700 1 -- start start 2023-03-03T10:17:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.154+0000 7fc3b211d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3ac06a2b0 0x7fc3ac0c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3b211d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3ac0c5cb0 0x7fc3ac149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3b211d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 0x7fc3ac1496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3b211d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac150a50 con 0x7fc3ac156490 2023-03-03T10:17:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3b211d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac1e9cc0 con 0x7fc3ac0c5cb0 2023-03-03T10:17:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3b211d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac14d520 con 0x7fc3ac06a2b0 2023-03-03T10:17:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3abfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 0x7fc3ac1496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3abfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 0x7fc3ac1496b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59748/0 (socket says 172.21.15.7:59748) 2023-03-03T10:17:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3abfff700 1 -- 172.21.15.7:0/2359056771 learned_addr learned my addr 172.21.15.7:0/2359056771 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:17:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3abfff700 1 -- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3ac06a2b0 msgr2=0x7fc3ac0c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3ab7fe700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3ac06a2b0 0x7fc3ac0c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3aaffd700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3ac0c5cb0 0x7fc3ac149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3abfff700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3ac06a2b0 0x7fc3ac0c00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3abfff700 1 -- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3ac0c5cb0 msgr2=0x7fc3ac149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3abfff700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3ac0c5cb0 0x7fc3ac149170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3abfff700 1 -- 172.21.15.7:0/2359056771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a4009b30 con 0x7fc3ac156490 2023-03-03T10:17:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3ab7fe700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3ac06a2b0 0x7fc3ac0c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:17:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3abfff700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 0x7fc3ac1496b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4015040 tx=0x7fc3a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3a3fff700 1 -- 172.21.15.7:0/2359056771 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a40092e0 con 0x7fc3ac156490 2023-03-03T10:17:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3a3fff700 1 -- 172.21.15.7:0/2359056771 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a4004240 con 0x7fc3ac156490 2023-03-03T10:17:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3a3fff700 1 -- 172.21.15.7:0/2359056771 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4004630 con 0x7fc3ac156490 2023-03-03T10:17:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3b211d700 1 -- 172.21.15.7:0/2359056771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ac152e80 con 0x7fc3ac156490 2023-03-03T10:17:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.155+0000 7fc3b211d700 1 -- 172.21.15.7:0/2359056771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ac0a5ea0 con 0x7fc3ac156490 2023-03-03T10:17:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.156+0000 7fc3b211d700 1 -- 172.21.15.7:0/2359056771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ac153160 con 0x7fc3ac156490 2023-03-03T10:17:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.157+0000 7fc3a3fff700 1 -- 172.21.15.7:0/2359056771 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc3a4004790 con 0x7fc3ac156490 2023-03-03T10:17:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.157+0000 7fc3a3fff700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc38c0682f0 0x7fc38c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:00.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.157+0000 7fc3a3fff700 1 -- 172.21.15.7:0/2359056771 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc3a4087610 con 0x7fc3ac156490 2023-03-03T10:17:00.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.158+0000 7fc3ab7fe700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc38c0682f0 0x7fc38c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:00.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.158+0000 7fc3ab7fe700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc38c0682f0 0x7fc38c06a7a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0a84c0 tx=0x7fc394008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:00.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.162+0000 7fc3a3fff700 1 -- 172.21.15.7:0/2359056771 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc3a4053930 con 0x7fc3ac156490 2023-03-03T10:17:00.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.399+0000 7fc3b211d700 1 -- 172.21.15.7:0/2359056771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3ac1e9e30 con 0x7fc3ac156490 2023-03-03T10:17:00.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.400+0000 7fc3a3fff700 1 -- 172.21.15.7:0/2359056771 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc3ac1e9e30 con 0x7fc3ac156490 2023-03-03T10:17:00.403 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:17:00.403 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:17:00.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 -- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc38c0682f0 msgr2=0x7fc38c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:00.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc38c0682f0 0x7fc38c06a7a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac0a84c0 tx=0x7fc394008040 comp rx=0 tx=0).stop 2023-03-03T10:17:00.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 -- 172.21.15.7:0/2359056771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 msgr2=0x7fc3ac1496b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:00.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 0x7fc3ac1496b0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4015040 tx=0x7fc3a4004ad0 comp rx=0 tx=0).stop 2023-03-03T10:17:00.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 -- 172.21.15.7:0/2359056771 shutdown_connections 2023-03-03T10:17:00.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc38c0682f0 0x7fc38c06a7a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:00.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3ac156490 0x7fc3ac1496b0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:00.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3ac06a2b0 0x7fc3ac0c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:00.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 --2- 172.21.15.7:0/2359056771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3ac0c5cb0 0x7fc3ac149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:00.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 -- 172.21.15.7:0/2359056771 >> 172.21.15.7:0/2359056771 conn(0x7fc3ac0a9430 msgr2=0x7fc3ac0c2830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:17:00.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 -- 172.21.15.7:0/2359056771 shutdown_connections 2023-03-03T10:17:00.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:00.402+0000 7fc3b211d700 1 -- 172.21.15.7:0/2359056771 wait complete. 2023-03-03T10:17:00.407 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:17:00.417 INFO:tasks.mds_thrash.fs.[b]:waiting for 40 secs before thrashing 2023-03-03T10:17:10.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:17:10.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.190+0000 7f9ad3e1b700 1 Processor -- start 2023-03-03T10:17:10.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.190+0000 7f9ad3e1b700 1 -- start start 2023-03-03T10:17:10.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad3e1b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9acc14f9d0 0x7f9acc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:10.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad3e1b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc0c2f30 con 0x7f9acc14fda0 2023-03-03T10:17:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad3e1b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9acc0c31d0 con 0x7f9acc0abe20 2023-03-03T10:17:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad3e1b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9acc1524a0 con 0x7f9acc14f9d0 2023-03-03T10:17:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad13b6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9acc14fda0 0x7f9acc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53732/0 (socket says 172.21.15.7:53732) 2023-03-03T10:17:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad13b6700 1 -- 172.21.15.7:0/1420260775 learned_addr learned my addr 172.21.15.7:0/1420260775 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:17:10.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180677711 0 0) 0x7f9acc0c2f30 con 0x7f9acc14fda0 2023-03-03T10:17:10.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad1bb7700 1 --2- 172.21.15.7:0/1420260775 >> v2:172.21.15.143:3300/0 conn(0x7f9acc14f9d0 0x7f9acc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:10.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab80035b0 con 0x7f9acc14fda0 2023-03-03T10:17:10.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1058942354 0 0) 0x7f9ab80035b0 con 0x7f9acc14fda0 2023-03-03T10:17:10.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 >> v2:172.21.15.143:3300/0 conn(0x7f9acc14f9d0 msgr2=0x7f9acc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:17:10.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad0bb5700 1 --2- 172.21.15.7:0/1420260775 >> v2:172.21.15.143:3300/0 conn(0x7f9acc14f9d0 0x7f9acc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:10.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 >> v1:172.21.15.143:6789/0 conn(0x7f9acc0abe20 legacy=0x7f9acc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:10.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc0c34b0 con 0x7f9acc14fda0 2023-03-03T10:17:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.191+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f9abc001670 con 0x7f9acc14fda0 2023-03-03T10:17:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 >> v1:172.21.15.7:6789/0 conn(0x7f9acc14fda0 legacy=0x7f9acc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad0bb5700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8003a70 0x7f9ab8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad0bb5700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8004850 0x7f9ab8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad0bb5700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8005320 0x7f9ab800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc1524a0 con 0x7f9ab8005320 2023-03-03T10:17:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab800c230 con 0x7f9ab8003a70 2023-03-03T10:17:10.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab800c3f0 con 0x7f9ab8004850 2023-03-03T10:17:10.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad23b8700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8005320 0x7f9ab800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:10.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad1bb7700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8003a70 0x7f9ab8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:10.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad13b6700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8004850 0x7f9ab8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:10.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad23b8700 1 -- 172.21.15.7:0/1420260775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8004850 msgr2=0x7f9ab8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:17:10.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad23b8700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8004850 0x7f9ab8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:10.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad23b8700 1 -- 172.21.15.7:0/1420260775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8003a70 msgr2=0x7f9ab8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:17:10.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad23b8700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8003a70 0x7f9ab8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:10.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad23b8700 1 -- 172.21.15.7:0/1420260775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab80035b0 con 0x7f9ab8005320 2023-03-03T10:17:10.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad23b8700 1 -- 172.21.15.7:0/1420260775 reap_dead start 2023-03-03T10:17:10.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad23b8700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8005320 0x7f9ab800ba10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400c160 tx=0x7f9ab8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa09bd7542c7b714 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:10.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4001db0 con 0x7f9ab8005320 2023-03-03T10:17:10.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac400ccc0 con 0x7f9ab8005320 2023-03-03T10:17:10.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.192+0000 7f9ad0bb5700 1 -- 172.21.15.7:0/1420260775 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4003c40 con 0x7f9ab8005320 2023-03-03T10:17:10.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.193+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/1420260775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8005320 msgr2=0x7f9ab800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:10.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.193+0000 7f9ad3e1b700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8005320 0x7f9ab800ba10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400c160 tx=0x7f9ab8005880 comp rx=0 tx=0).stop 2023-03-03T10:17:10.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.193+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/1420260775 shutdown_connections 2023-03-03T10:17:10.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.193+0000 7f9ad3e1b700 1 --2- 172.21.15.7:0/1420260775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8005320 0x7f9ab800ba10 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:10.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.193+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/1420260775 >> 172.21.15.7:0/1420260775 conn(0x7f9acc0c8900 msgr2=0x7f9acc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:17:10.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.193+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/1420260775 shutdown_connections 2023-03-03T10:17:10.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.193+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/1420260775 wait complete. 2023-03-03T10:17:10.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.194+0000 7f9ad3e1b700 1 Processor -- start 2023-03-03T10:17:10.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.194+0000 7f9ad3e1b700 1 -- start start 2023-03-03T10:17:10.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad3e1b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8005320 0x7f9acc1ed850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:10.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad3e1b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9acc14c620 0x7f9acc14e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:10.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad3e1b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9acc1e9760 0x7f9acc1e9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:10.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad3e1b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc149b00 con 0x7f9acc14c620 2023-03-03T10:17:10.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad3e1b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc14acc0 con 0x7f9ab8005320 2023-03-03T10:17:10.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad3e1b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc14cb70 con 0x7f9acc1e9760 2023-03-03T10:17:10.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad13b6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9acc14c620 0x7f9acc14e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:10.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad13b6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9acc14c620 0x7f9acc14e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59764/0 (socket says 172.21.15.7:59764) 2023-03-03T10:17:10.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad13b6700 1 -- 172.21.15.7:0/2112333526 learned_addr learned my addr 172.21.15.7:0/2112333526 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:17:10.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad1bb7700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8005320 0x7f9acc1ed850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:10.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad23b8700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9acc1e9760 0x7f9acc1e9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:10.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad13b6700 1 -- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9acc1e9760 msgr2=0x7f9acc1e9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:10.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad13b6700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9acc1e9760 0x7f9acc1e9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:10.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad13b6700 1 -- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8005320 msgr2=0x7f9acc1ed850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:10.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad13b6700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8005320 0x7f9acc1ed850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:10.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.195+0000 7f9ad13b6700 1 -- 172.21.15.7:0/2112333526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc0ac350 con 0x7f9acc14c620 2023-03-03T10:17:10.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.196+0000 7f9ad13b6700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9acc14c620 0x7f9acc14e930 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9abc000f60 tx=0x7f9abc007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:10.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.196+0000 7f9ac27fc700 1 -- 172.21.15.7:0/2112333526 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc00f6c0 con 0x7f9acc14c620 2023-03-03T10:17:10.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.196+0000 7f9ac27fc700 1 -- 172.21.15.7:0/2112333526 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9abc00f860 con 0x7f9acc14c620 2023-03-03T10:17:10.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.196+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/2112333526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9acc0ab630 con 0x7f9acc14c620 2023-03-03T10:17:10.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.196+0000 7f9ac27fc700 1 -- 172.21.15.7:0/2112333526 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc00fc10 con 0x7f9acc14c620 2023-03-03T10:17:10.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.196+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/2112333526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9acc0c3e10 con 0x7f9acc14c620 2023-03-03T10:17:10.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.197+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/2112333526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9acc1ed6a0 con 0x7f9acc14c620 2023-03-03T10:17:10.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.197+0000 7f9ac27fc700 1 -- 172.21.15.7:0/2112333526 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9abc00fd70 con 0x7f9acc14c620 2023-03-03T10:17:10.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.198+0000 7f9ac27fc700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ab00682f0 0x7f9ab006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:10.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.198+0000 7f9ac27fc700 1 -- 172.21.15.7:0/2112333526 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9abc086c10 con 0x7f9acc14c620 2023-03-03T10:17:10.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.198+0000 7f9ad1bb7700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ab00682f0 0x7f9ab006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:10.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.199+0000 7f9ad1bb7700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ab00682f0 0x7f9ab006a7a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009b20 tx=0x7f9ac8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:10.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.202+0000 7f9ac27fc700 1 -- 172.21.15.7:0/2112333526 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9abc052ff0 con 0x7f9acc14c620 2023-03-03T10:17:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.441+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/2112333526 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9acc06fab0 con 0x7f9acc14c620 2023-03-03T10:17:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.442+0000 7f9ac27fc700 1 -- 172.21.15.7:0/2112333526 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9abc056610 con 0x7f9acc14c620 2023-03-03T10:17:10.445 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:17:10.445 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:17:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ab00682f0 msgr2=0x7f9ab006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ab00682f0 0x7f9ab006a7a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009b20 tx=0x7f9ac8008040 comp rx=0 tx=0).stop 2023-03-03T10:17:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/2112333526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9acc14c620 msgr2=0x7f9acc14e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9acc14c620 0x7f9acc14e930 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9abc000f60 tx=0x7f9abc007620 comp rx=0 tx=0).stop 2023-03-03T10:17:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/2112333526 shutdown_connections 2023-03-03T10:17:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9acc1e9760 0x7f9acc1e9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:10.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8005320 0x7f9acc1ed850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:10.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ab00682f0 0x7f9ab006a7a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:10.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 --2- 172.21.15.7:0/2112333526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9acc14c620 0x7f9acc14e930 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:10.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/2112333526 >> 172.21.15.7:0/2112333526 conn(0x7f9acc0c8900 msgr2=0x7f9acc0c6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:17:10.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/2112333526 shutdown_connections 2023-03-03T10:17:10.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:10.444+0000 7f9ad3e1b700 1 -- 172.21.15.7:0/2112333526 wait complete. 2023-03-03T10:17:10.449 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:17:10.458 INFO:tasks.mds_thrash.fs.[a]:waiting for 39 secs before thrashing 2023-03-03T10:17:28.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:28.157 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:40.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:17:40.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.555+0000 7fa47ad66700 1 Processor -- start 2023-03-03T10:17:40.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.555+0000 7fa47ad66700 1 -- start start 2023-03-03T10:17:40.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.555+0000 7fa47ad66700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa474151cd0 0x7fa474154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:40.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.555+0000 7fa47ad66700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa47406c6c0 0x7fa4740736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:40.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.555+0000 7fa47ad66700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa474157ff0 con 0x7fa4740c2e30 2023-03-03T10:17:40.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.555+0000 7fa47ad66700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa47406d640 con 0x7fa47406c6c0 2023-03-03T10:17:40.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.555+0000 7fa47ad66700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47406cda0 con 0x7fa474151cd0 2023-03-03T10:17:40.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa478b02700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa474151cd0 0x7fa474154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:40.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa479303700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa47406c6c0 0x7fa4740736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:40.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa479303700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa47406c6c0 0x7fa4740736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:55564/0 (socket says 172.21.15.7:55564) 2023-03-03T10:17:40.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa473fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fa4740c2e30 0x7fa474158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:57286/0 (socket says 172.21.15.7:57286) 2023-03-03T10:17:40.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa478b02700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa474151cd0 0x7fa474154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:39626/0 (socket says 172.21.15.7:39626) 2023-03-03T10:17:40.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa479303700 1 -- 172.21.15.7:0/2127757189 learned_addr learned my addr 172.21.15.7:0/2127757189 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:17:40.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa479303700 1 -- 172.21.15.7:0/2127757189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa474151cd0 msgr2=0x7fa474154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:17:40.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa479303700 1 --2- 172.21.15.7:0/2127757189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa474151cd0 0x7fa474154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:40.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa479303700 1 -- 172.21.15.7:0/2127757189 >> v1:172.21.15.143:6789/0 conn(0x7fa4740c2e30 legacy=0x7fa474158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:40.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa479303700 1 -- 172.21.15.7:0/2127757189 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47406d080 con 0x7fa47406c6c0 2023-03-03T10:17:40.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa478b02700 1 --2- 172.21.15.7:0/2127757189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa474151cd0 0x7fa474154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:17:40.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.556+0000 7fa479303700 1 --2- 172.21.15.7:0/2127757189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47406c6c0 0x7fa4740736f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009e50 tx=0x7fa46c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a416a84b06b4671 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:40.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa4737fe700 1 -- 172.21.15.7:0/2127757189 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c00e070 con 0x7fa47406c6c0 2023-03-03T10:17:40.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa4737fe700 1 -- 172.21.15.7:0/2127757189 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c00b910 con 0x7fa47406c6c0 2023-03-03T10:17:40.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa4737fe700 1 -- 172.21.15.7:0/2127757189 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c00bcc0 con 0x7fa47406c6c0 2023-03-03T10:17:40.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa47ad66700 1 -- 172.21.15.7:0/2127757189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47406c6c0 msgr2=0x7fa4740736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:40.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa47ad66700 1 --2- 172.21.15.7:0/2127757189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47406c6c0 0x7fa4740736f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009e50 tx=0x7fa46c00b330 comp rx=0 tx=0).stop 2023-03-03T10:17:40.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa47ad66700 1 -- 172.21.15.7:0/2127757189 shutdown_connections 2023-03-03T10:17:40.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa47ad66700 1 --2- 172.21.15.7:0/2127757189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa47406c6c0 0x7fa4740736f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:40.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa47ad66700 1 --2- 172.21.15.7:0/2127757189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa474151cd0 0x7fa474154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:40.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa47ad66700 1 -- 172.21.15.7:0/2127757189 >> 172.21.15.7:0/2127757189 conn(0x7fa4740a94d0 msgr2=0x7fa4740a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:17:40.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa47ad66700 1 -- 172.21.15.7:0/2127757189 shutdown_connections 2023-03-03T10:17:40.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.557+0000 7fa47ad66700 1 -- 172.21.15.7:0/2127757189 wait complete. 2023-03-03T10:17:40.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.558+0000 7fa47ad66700 1 Processor -- start 2023-03-03T10:17:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.558+0000 7fa47ad66700 1 -- start start 2023-03-03T10:17:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.559+0000 7fa47ad66700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa47406c6c0 0x7fa47414d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.559+0000 7fa47ad66700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4740c2e30 0x7fa4741edac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.559+0000 7fa47ad66700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa474151cd0 0x7fa4741ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.559+0000 7fa47ad66700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47414ff80 con 0x7fa474151cd0 2023-03-03T10:17:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.559+0000 7fa47ad66700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa474148d80 con 0x7fa4740c2e30 2023-03-03T10:17:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.559+0000 7fa47ad66700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47414f650 con 0x7fa47406c6c0 2023-03-03T10:17:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.559+0000 7fa479303700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa474151cd0 0x7fa4741ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.559+0000 7fa479303700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa474151cd0 0x7fa4741ee000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59796/0 (socket says 172.21.15.7:59796) 2023-03-03T10:17:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.559+0000 7fa479303700 1 -- 172.21.15.7:0/1866401887 learned_addr learned my addr 172.21.15.7:0/1866401887 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:17:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa473fff700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4740c2e30 0x7fa4741edac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa478b02700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa47406c6c0 0x7fa47414d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa479303700 1 -- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa47406c6c0 msgr2=0x7fa47414d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa479303700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa47406c6c0 0x7fa47414d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa479303700 1 -- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4740c2e30 msgr2=0x7fa4741edac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa479303700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4740c2e30 0x7fa4741edac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa479303700 1 -- 172.21.15.7:0/1866401887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4680094a0 con 0x7fa474151cd0 2023-03-03T10:17:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa473fff700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4740c2e30 0x7fa4741edac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:17:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa479303700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa474151cd0 0x7fa4741ee000 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa4740744a0 tx=0x7fa46c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa4717fa700 1 -- 172.21.15.7:0/1866401887 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c004960 con 0x7fa474151cd0 2023-03-03T10:17:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa4717fa700 1 -- 172.21.15.7:0/1866401887 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c0099b0 con 0x7fa474151cd0 2023-03-03T10:17:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa47ad66700 1 -- 172.21.15.7:0/1866401887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46c009b30 con 0x7fa474151cd0 2023-03-03T10:17:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa4717fa700 1 -- 172.21.15.7:0/1866401887 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c01b670 con 0x7fa474151cd0 2023-03-03T10:17:40.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.560+0000 7fa47ad66700 1 -- 172.21.15.7:0/1866401887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa474157c80 con 0x7fa474151cd0 2023-03-03T10:17:40.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.561+0000 7fa47ad66700 1 -- 172.21.15.7:0/1866401887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47406d360 con 0x7fa474151cd0 2023-03-03T10:17:40.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.562+0000 7fa4717fa700 1 -- 172.21.15.7:0/1866401887 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa46c02b070 con 0x7fa474151cd0 2023-03-03T10:17:40.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.562+0000 7fa4717fa700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4580682f0 0x7fa45806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:40.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.562+0000 7fa4717fa700 1 -- 172.21.15.7:0/1866401887 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa46c087920 con 0x7fa474151cd0 2023-03-03T10:17:40.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.562+0000 7fa478b02700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4580682f0 0x7fa45806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:40.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.563+0000 7fa478b02700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4580682f0 0x7fa45806a7a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa47414a470 tx=0x7fa468008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.567+0000 7fa4717fa700 1 -- 172.21.15.7:0/1866401887 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa46c053cc0 con 0x7fa474151cd0 2023-03-03T10:17:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.804+0000 7fa47ad66700 1 -- 172.21.15.7:0/1866401887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa474154a50 con 0x7fa474151cd0 2023-03-03T10:17:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.805+0000 7fa4717fa700 1 -- 172.21.15.7:0/1866401887 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa46c0572e0 con 0x7fa474151cd0 2023-03-03T10:17:40.808 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:17:40.808 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:17:40.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.806+0000 7fa47ad66700 1 -- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4580682f0 msgr2=0x7fa45806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:40.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.806+0000 7fa47ad66700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4580682f0 0x7fa45806a7a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa47414a470 tx=0x7fa468008040 comp rx=0 tx=0).stop 2023-03-03T10:17:40.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.807+0000 7fa47ad66700 1 -- 172.21.15.7:0/1866401887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa474151cd0 msgr2=0x7fa4741ee000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:40.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.807+0000 7fa47ad66700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa474151cd0 0x7fa4741ee000 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fa4740744a0 tx=0x7fa46c004ad0 comp rx=0 tx=0).stop 2023-03-03T10:17:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.807+0000 7fa47ad66700 1 -- 172.21.15.7:0/1866401887 shutdown_connections 2023-03-03T10:17:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.807+0000 7fa47ad66700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4580682f0 0x7fa45806a7a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.807+0000 7fa47ad66700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa474151cd0 0x7fa4741ee000 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.807+0000 7fa47ad66700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa47406c6c0 0x7fa47414d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.807+0000 7fa47ad66700 1 --2- 172.21.15.7:0/1866401887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4740c2e30 0x7fa4741edac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.807+0000 7fa47ad66700 1 -- 172.21.15.7:0/1866401887 >> 172.21.15.7:0/1866401887 conn(0x7fa4740a94d0 msgr2=0x7fa4740a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:17:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.807+0000 7fa47ad66700 1 -- 172.21.15.7:0/1866401887 shutdown_connections 2023-03-03T10:17:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:40.807+0000 7fa47ad66700 1 -- 172.21.15.7:0/1866401887 wait complete. 2023-03-03T10:17:40.811 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:17:40.821 INFO:tasks.mds_thrash.fs.[b]:waiting for 38 secs before thrashing 2023-03-03T10:17:49.460 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:17:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.591+0000 7ff6377e6700 1 Processor -- start 2023-03-03T10:17:49.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.592+0000 7ff6377e6700 1 -- start start 2023-03-03T10:17:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.592+0000 7ff6377e6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ace30 0x7ff630152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.592+0000 7ff6377e6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6300c26e0 0x7ff630152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.592+0000 7ff6377e6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6300ac900 con 0x7ff6300ac220 2023-03-03T10:17:49.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.592+0000 7ff6377e6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff630071e30 con 0x7ff6300c26e0 2023-03-03T10:17:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.592+0000 7ff6377e6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff630152ed0 con 0x7ff6300ace30 2023-03-03T10:17:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff635d83700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6300c26e0 0x7ff630152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff635d83700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6300c26e0 0x7ff630152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59816/0 (socket says 172.21.15.7:59816) 2023-03-03T10:17:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff635d83700 1 -- 172.21.15.7:0/870461639 learned_addr learned my addr 172.21.15.7:0/870461639 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:17:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff634d81700 1 --2- 172.21.15.7:0/870461639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ace30 0x7ff630152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff635d83700 1 -- 172.21.15.7:0/870461639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ace30 msgr2=0x7ff630152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:17:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff635d83700 1 --2- 172.21.15.7:0/870461639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ace30 0x7ff630152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:49.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff635d83700 1 -- 172.21.15.7:0/870461639 >> v1:172.21.15.143:6789/0 conn(0x7ff6300ac220 legacy=0x7ff6300736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:49.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff635d83700 1 -- 172.21.15.7:0/870461639 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6300acb50 con 0x7ff6300c26e0 2023-03-03T10:17:49.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff634d81700 1 --2- 172.21.15.7:0/870461639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ace30 0x7ff630152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:17:49.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff635d83700 1 --2- 172.21.15.7:0/870461639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300c26e0 0x7ff630152990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff628009e50 tx=0x7ff62800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b64cc229f9a5fe0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:49.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff623fff700 1 -- 172.21.15.7:0/870461639 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62800e070 con 0x7ff6300c26e0 2023-03-03T10:17:49.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff623fff700 1 -- 172.21.15.7:0/870461639 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62800b9c0 con 0x7ff6300c26e0 2023-03-03T10:17:49.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.593+0000 7ff623fff700 1 -- 172.21.15.7:0/870461639 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62800bd70 con 0x7ff6300c26e0 2023-03-03T10:17:49.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.594+0000 7ff6377e6700 1 -- 172.21.15.7:0/870461639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300c26e0 msgr2=0x7ff630152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.594+0000 7ff6377e6700 1 --2- 172.21.15.7:0/870461639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300c26e0 0x7ff630152990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff628009e50 tx=0x7ff62800b330 comp rx=0 tx=0).stop 2023-03-03T10:17:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.594+0000 7ff6377e6700 1 -- 172.21.15.7:0/870461639 shutdown_connections 2023-03-03T10:17:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.594+0000 7ff6377e6700 1 --2- 172.21.15.7:0/870461639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300c26e0 0x7ff630152990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.594+0000 7ff6377e6700 1 --2- 172.21.15.7:0/870461639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ace30 0x7ff630152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.594+0000 7ff6377e6700 1 -- 172.21.15.7:0/870461639 >> 172.21.15.7:0/870461639 conn(0x7ff6300a94d0 msgr2=0x7ff6300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:17:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.594+0000 7ff6377e6700 1 -- 172.21.15.7:0/870461639 shutdown_connections 2023-03-03T10:17:49.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.594+0000 7ff6377e6700 1 -- 172.21.15.7:0/870461639 wait complete. 2023-03-03T10:17:49.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.595+0000 7ff6377e6700 1 Processor -- start 2023-03-03T10:17:49.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.595+0000 7ff6377e6700 1 -- start start 2023-03-03T10:17:49.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff6377e6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ac220 0x7ff6300be430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:49.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff6377e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300ace30 0x7ff6301ea870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:49.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff6377e6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6300c26e0 0x7ff63006cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:49.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff6377e6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63014f7d0 con 0x7ff6300ace30 2023-03-03T10:17:49.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff6377e6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63014ef30 con 0x7ff6300c26e0 2023-03-03T10:17:49.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff6377e6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63014f210 con 0x7ff6300ac220 2023-03-03T10:17:49.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff634d81700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300ace30 0x7ff6301ea870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:49.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff634d81700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300ace30 0x7ff6301ea870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59820/0 (socket says 172.21.15.7:59820) 2023-03-03T10:17:49.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff634d81700 1 -- 172.21.15.7:0/4196029025 learned_addr learned my addr 172.21.15.7:0/4196029025 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:17:49.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff635582700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ac220 0x7ff6300be430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:49.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff635d83700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6300c26e0 0x7ff63006cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:49.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff634d81700 1 -- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ac220 msgr2=0x7ff6300be430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:49.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff634d81700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ac220 0x7ff6300be430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:49.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff634d81700 1 -- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6300c26e0 msgr2=0x7ff63006cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:49.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff634d81700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6300c26e0 0x7ff63006cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff634d81700 1 -- 172.21.15.7:0/4196029025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff628009b30 con 0x7ff6300ace30 2023-03-03T10:17:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff635d83700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6300c26e0 0x7ff63006cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:17:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff635582700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ac220 0x7ff6300be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:17:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff634d81700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300ace30 0x7ff6301ea870 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00e810 tx=0x7ff62c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff621ffb700 1 -- 172.21.15.7:0/4196029025 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c013610 con 0x7ff6300ace30 2023-03-03T10:17:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff621ffb700 1 -- 172.21.15.7:0/4196029025 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62c013770 con 0x7ff6300ace30 2023-03-03T10:17:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff6377e6700 1 -- 172.21.15.7:0/4196029025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff630155ca0 con 0x7ff6300ace30 2023-03-03T10:17:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.596+0000 7ff621ffb700 1 -- 172.21.15.7:0/4196029025 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c013b20 con 0x7ff6300ace30 2023-03-03T10:17:49.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.597+0000 7ff6377e6700 1 -- 172.21.15.7:0/4196029025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6301559c0 con 0x7ff6300ace30 2023-03-03T10:17:49.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.598+0000 7ff6377e6700 1 -- 172.21.15.7:0/4196029025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff630155f80 con 0x7ff6300ace30 2023-03-03T10:17:49.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.598+0000 7ff621ffb700 1 -- 172.21.15.7:0/4196029025 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff62c023070 con 0x7ff6300ace30 2023-03-03T10:17:49.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.598+0000 7ff621ffb700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6140682c0 0x7ff61406a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:17:49.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.598+0000 7ff621ffb700 1 -- 172.21.15.7:0/4196029025 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff62c085e50 con 0x7ff6300ace30 2023-03-03T10:17:49.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.599+0000 7ff635582700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6140682c0 0x7ff61406a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:17:49.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.599+0000 7ff635582700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6140682c0 0x7ff61406a770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff6300c1300 tx=0x7ff624009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:17:49.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.603+0000 7ff621ffb700 1 -- 172.21.15.7:0/4196029025 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff62c0521f0 con 0x7ff6300ace30 2023-03-03T10:17:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.840+0000 7ff6377e6700 1 -- 172.21.15.7:0/4196029025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff630074bb0 con 0x7ff6300ace30 2023-03-03T10:17:49.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.840+0000 7ff621ffb700 1 -- 172.21.15.7:0/4196029025 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff62c055810 con 0x7ff6300ace30 2023-03-03T10:17:49.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:17:49.843 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:17:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 -- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6140682c0 msgr2=0x7ff61406a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6140682c0 0x7ff61406a770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff6300c1300 tx=0x7ff624009040 comp rx=0 tx=0).stop 2023-03-03T10:17:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 -- 172.21.15.7:0/4196029025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300ace30 msgr2=0x7ff6301ea870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:17:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300ace30 0x7ff6301ea870 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00e810 tx=0x7ff62c00ebd0 comp rx=0 tx=0).stop 2023-03-03T10:17:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 -- 172.21.15.7:0/4196029025 shutdown_connections 2023-03-03T10:17:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6300c26e0 0x7ff63006cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6300ac220 0x7ff6300be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:49.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6140682c0 0x7ff61406a770 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:49.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 --2- 172.21.15.7:0/4196029025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6300ace30 0x7ff6301ea870 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:17:49.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 -- 172.21.15.7:0/4196029025 >> 172.21.15.7:0/4196029025 conn(0x7ff6300a94d0 msgr2=0x7ff630154680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:17:49.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 -- 172.21.15.7:0/4196029025 shutdown_connections 2023-03-03T10:17:49.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:17:49.842+0000 7ff6377e6700 1 -- 172.21.15.7:0/4196029025 wait complete. 2023-03-03T10:17:49.847 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:17:49.856 INFO:tasks.mds_thrash.fs.[a]:waiting for 106 secs before thrashing 2023-03-03T10:17:58.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:17:58.227 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:18.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:18:18.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.959+0000 7f04c14bd700 1 Processor -- start 2023-03-03T10:18:18.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.959+0000 7f04c14bd700 1 -- start start 2023-03-03T10:18:18.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04c14bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0caa30 0x7f04bc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04c14bd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04bc154ff0 0x7f04bc0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04c14bd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04bc0c7310 con 0x7f04bc0c00e0 2023-03-03T10:18:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04c14bd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04bc0bfd10 con 0x7f04bc154ff0 2023-03-03T10:18:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04c14bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0bfa30 con 0x7f04bc0caa30 2023-03-03T10:18:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04bb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04bc154ff0 0x7f04bc0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04ba7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f04bc0c00e0 0x7f04bc0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53820/0 (socket says 172.21.15.7:53820) 2023-03-03T10:18:18.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04ba7fc700 1 -- 172.21.15.7:0/3456611772 learned_addr learned my addr 172.21.15.7:0/3456611772 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:18:18.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04baffd700 1 --2- 172.21.15.7:0/3456611772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0caa30 0x7f04bc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:18.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04b9ffb700 1 -- 172.21.15.7:0/3456611772 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762925794 0 0) 0x7f04bc0c7310 con 0x7f04bc0c00e0 2023-03-03T10:18:18.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04b9ffb700 1 -- 172.21.15.7:0/3456611772 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a40035b0 con 0x7f04bc0c00e0 2023-03-03T10:18:18.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04bb7fe700 1 -- 172.21.15.7:0/3456611772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0caa30 msgr2=0x7f04bc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:18:18.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04bb7fe700 1 --2- 172.21.15.7:0/3456611772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0caa30 0x7f04bc0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:18.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04bb7fe700 1 -- 172.21.15.7:0/3456611772 >> v1:172.21.15.7:6789/0 conn(0x7f04bc0c00e0 legacy=0x7f04bc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:18.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04bb7fe700 1 -- 172.21.15.7:0/3456611772 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc154510 con 0x7f04bc154ff0 2023-03-03T10:18:18.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.960+0000 7f04bb7fe700 1 --2- 172.21.15.7:0/3456611772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 0x7f04bc0cae00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009d90 tx=0x7f04b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7bf703a93ce584 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:18:18.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04b9ffb700 1 -- 172.21.15.7:0/3456611772 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b400e070 con 0x7f04bc154ff0 2023-03-03T10:18:18.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04b9ffb700 1 -- 172.21.15.7:0/3456611772 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b400bb20 con 0x7f04bc154ff0 2023-03-03T10:18:18.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04b9ffb700 1 -- 172.21.15.7:0/3456611772 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b400be90 con 0x7f04bc154ff0 2023-03-03T10:18:18.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04c14bd700 1 -- 172.21.15.7:0/3456611772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 msgr2=0x7f04bc0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:18.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04c14bd700 1 --2- 172.21.15.7:0/3456611772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 0x7f04bc0cae00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009d90 tx=0x7f04b400b330 comp rx=0 tx=0).stop 2023-03-03T10:18:18.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04c14bd700 1 -- 172.21.15.7:0/3456611772 shutdown_connections 2023-03-03T10:18:18.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04c14bd700 1 --2- 172.21.15.7:0/3456611772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 0x7f04bc0cae00 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:18.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04c14bd700 1 --2- 172.21.15.7:0/3456611772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0caa30 0x7f04bc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:18.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04c14bd700 1 -- 172.21.15.7:0/3456611772 >> 172.21.15.7:0/3456611772 conn(0x7f04bc066e40 msgr2=0x7f04bc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:18:18.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04c14bd700 1 -- 172.21.15.7:0/3456611772 shutdown_connections 2023-03-03T10:18:18.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.961+0000 7f04c14bd700 1 -- 172.21.15.7:0/3456611772 wait complete. 2023-03-03T10:18:18.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04c14bd700 1 Processor -- start 2023-03-03T10:18:18.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04c14bd700 1 -- start start 2023-03-03T10:18:18.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04c14bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0c00e0 0x7f04bc14c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:18.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04c14bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04bc0caa30 0x7f04bc0c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:18.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04c14bd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 0x7f04bc0c25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:18.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04c14bd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc1edd70 con 0x7f04bc154ff0 2023-03-03T10:18:18.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04c14bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc0c1f20 con 0x7f04bc0caa30 2023-03-03T10:18:18.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04c14bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc14c070 con 0x7f04bc0c00e0 2023-03-03T10:18:18.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04bb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 0x7f04bc0c25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:18.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04bb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 0x7f04bc0c25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59850/0 (socket says 172.21.15.7:59850) 2023-03-03T10:18:18.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.963+0000 7f04bb7fe700 1 -- 172.21.15.7:0/2376685061 learned_addr learned my addr 172.21.15.7:0/2376685061 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:18:18.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04baffd700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0c00e0 0x7f04bc14c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04ba7fc700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04bc0caa30 0x7f04bc0c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04bb7fe700 1 -- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0c00e0 msgr2=0x7f04bc14c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04bb7fe700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0c00e0 0x7f04bc14c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04bb7fe700 1 -- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04bc0caa30 msgr2=0x7f04bc0c19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04bb7fe700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04bc0caa30 0x7f04bc0c19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04bb7fe700 1 -- 172.21.15.7:0/2376685061 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009a70 con 0x7f04bc154ff0 2023-03-03T10:18:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04baffd700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0c00e0 0x7f04bc14c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:18:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04bb7fe700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 0x7f04bc0c25e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f04bc06b270 tx=0x7f04b4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:18:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04a37fe700 1 -- 172.21.15.7:0/2376685061 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40098f0 con 0x7f04bc154ff0 2023-03-03T10:18:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04a37fe700 1 -- 172.21.15.7:0/2376685061 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b4020070 con 0x7f04bc154ff0 2023-03-03T10:18:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04c14bd700 1 -- 172.21.15.7:0/2376685061 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04bc154ad0 con 0x7f04bc154ff0 2023-03-03T10:18:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04a37fe700 1 -- 172.21.15.7:0/2376685061 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40276a0 con 0x7f04bc154ff0 2023-03-03T10:18:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.964+0000 7f04c14bd700 1 -- 172.21.15.7:0/2376685061 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04bc1547f0 con 0x7f04bc154ff0 2023-03-03T10:18:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.965+0000 7f04c14bd700 1 -- 172.21.15.7:0/2376685061 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc0bf650 con 0x7f04bc154ff0 2023-03-03T10:18:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.966+0000 7f04a37fe700 1 -- 172.21.15.7:0/2376685061 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f04b4012430 con 0x7f04bc154ff0 2023-03-03T10:18:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.966+0000 7f04a37fe700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04a80682f0 0x7f04a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:18:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.966+0000 7f04a37fe700 1 -- 172.21.15.7:0/2376685061 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f04b4022070 con 0x7f04bc154ff0 2023-03-03T10:18:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.966+0000 7f04baffd700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04a80682f0 0x7f04a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:18:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.967+0000 7f04baffd700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04a80682f0 0x7f04a806a7a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009c30 tx=0x7f04ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:18:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:18.971+0000 7f04a37fe700 1 -- 172.21.15.7:0/2376685061 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f04b4054860 con 0x7f04bc154ff0 2023-03-03T10:18:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.212+0000 7f04c14bd700 1 -- 172.21.15.7:0/2376685061 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04bc1edf20 con 0x7f04bc154ff0 2023-03-03T10:18:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.212+0000 7f04a37fe700 1 -- 172.21.15.7:0/2376685061 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f04bc1edf20 con 0x7f04bc154ff0 2023-03-03T10:18:19.216 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:18:19.216 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:18:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.214+0000 7f04c14bd700 1 -- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04a80682f0 msgr2=0x7f04a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.214+0000 7f04c14bd700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04a80682f0 0x7f04a806a7a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009c30 tx=0x7f04ac008040 comp rx=0 tx=0).stop 2023-03-03T10:18:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.214+0000 7f04c14bd700 1 -- 172.21.15.7:0/2376685061 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 msgr2=0x7f04bc0c25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:18:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.214+0000 7f04c14bd700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 0x7f04bc0c25e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f04bc06b270 tx=0x7f04b4004ad0 comp rx=0 tx=0).stop 2023-03-03T10:18:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.214+0000 7f04c14bd700 1 -- 172.21.15.7:0/2376685061 shutdown_connections 2023-03-03T10:18:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.214+0000 7f04c14bd700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04a80682f0 0x7f04a806a7a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.214+0000 7f04c14bd700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04bc154ff0 0x7f04bc0c25e0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.214+0000 7f04c14bd700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04bc0c00e0 0x7f04bc14c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.214+0000 7f04c14bd700 1 --2- 172.21.15.7:0/2376685061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04bc0caa30 0x7f04bc0c19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:18:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.214+0000 7f04c14bd700 1 -- 172.21.15.7:0/2376685061 >> 172.21.15.7:0/2376685061 conn(0x7f04bc066e40 msgr2=0x7f04bc0c7690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:18:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.215+0000 7f04c14bd700 1 -- 172.21.15.7:0/2376685061 shutdown_connections 2023-03-03T10:18:19.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:18:19.215+0000 7f04c14bd700 1 -- 172.21.15.7:0/2376685061 wait complete. 2023-03-03T10:18:19.219 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:18:19.228 INFO:tasks.mds_thrash.fs.[b]:waiting for 93 secs before thrashing 2023-03-03T10:18:28.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:28.617 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:58.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:18:58.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:28.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:28.783 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:35.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:19:35.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.993+0000 7f7dc184e700 1 Processor -- start 2023-03-03T10:19:35.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.993+0000 7f7dc184e700 1 -- start start 2023-03-03T10:19:35.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dc184e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7dbc0c00e0 0x7f7dbc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dc184e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7dbc154ff0 0x7f7dbc0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dc184e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc0c7310 con 0x7f7dbc0caa30 2023-03-03T10:19:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dc184e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dbc155430 con 0x7f7dbc154ff0 2023-03-03T10:19:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dc184e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dbc0bfd10 con 0x7f7dbc0c00e0 2023-03-03T10:19:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dbb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7dbc154ff0 0x7f7dbc0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dbaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7dbc0caa30 0x7f7dbc0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53898/0 (socket says 172.21.15.7:53898) 2023-03-03T10:19:35.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dbaffd700 1 -- 172.21.15.7:0/2565093445 learned_addr learned my addr 172.21.15.7:0/2565093445 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:19:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dba7fc700 1 --2- 172.21.15.7:0/2565093445 >> v2:172.21.15.143:3300/0 conn(0x7f7dbc0c00e0 0x7f7dbc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dbb7fe700 1 -- 172.21.15.7:0/2565093445 >> v2:172.21.15.143:3300/0 conn(0x7f7dbc0c00e0 msgr2=0x7f7dbc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:19:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dbb7fe700 1 --2- 172.21.15.7:0/2565093445 >> v2:172.21.15.143:3300/0 conn(0x7f7dbc0c00e0 0x7f7dbc0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dbb7fe700 1 -- 172.21.15.7:0/2565093445 >> v1:172.21.15.7:6789/0 conn(0x7f7dbc0caa30 legacy=0x7f7dbc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7db9ffb700 1 -- 172.21.15.7:0/2565093445 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589367878 0 0) 0x7f7dbc0c7310 con 0x7f7dbc0caa30 2023-03-03T10:19:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.994+0000 7f7dbb7fe700 1 -- 172.21.15.7:0/2565093445 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc0bfa30 con 0x7f7dbc154ff0 2023-03-03T10:19:36.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7dbb7fe700 1 --2- 172.21.15.7:0/2565093445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc154ff0 0x7f7dbc0cae00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0ca910 tx=0x7f7db400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f09b3e34fddc887 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:19:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7db9ffb700 1 -- 172.21.15.7:0/2565093445 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db400e070 con 0x7f7dbc154ff0 2023-03-03T10:19:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7db9ffb700 1 -- 172.21.15.7:0/2565093445 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db400bbd0 con 0x7f7dbc154ff0 2023-03-03T10:19:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7db9ffb700 1 -- 172.21.15.7:0/2565093445 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db4012430 con 0x7f7dbc154ff0 2023-03-03T10:19:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7dc184e700 1 -- 172.21.15.7:0/2565093445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc154ff0 msgr2=0x7f7dbc0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7dc184e700 1 --2- 172.21.15.7:0/2565093445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc154ff0 0x7f7dbc0cae00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0ca910 tx=0x7f7db400b330 comp rx=0 tx=0).stop 2023-03-03T10:19:36.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7dc184e700 1 -- 172.21.15.7:0/2565093445 shutdown_connections 2023-03-03T10:19:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7dc184e700 1 --2- 172.21.15.7:0/2565093445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc154ff0 0x7f7dbc0cae00 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7dc184e700 1 --2- 172.21.15.7:0/2565093445 >> v2:172.21.15.143:3300/0 conn(0x7f7dbc0c00e0 0x7f7dbc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7dc184e700 1 -- 172.21.15.7:0/2565093445 >> 172.21.15.7:0/2565093445 conn(0x7f7dbc066e40 msgr2=0x7f7dbc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:19:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7dc184e700 1 -- 172.21.15.7:0/2565093445 shutdown_connections 2023-03-03T10:19:36.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.995+0000 7f7dc184e700 1 -- 172.21.15.7:0/2565093445 wait complete. 2023-03-03T10:19:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.997+0000 7f7dc184e700 1 Processor -- start 2023-03-03T10:19:36.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.997+0000 7f7dc184e700 1 -- start start 2023-03-03T10:19:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dc184e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0c00e0 0x7f7dbc142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dc184e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7dbc0caa30 0x7f7dbc157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dc184e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dbc154ff0 0x7f7dbc156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dc184e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc0aa940 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dc184e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc158420 con 0x7f7dbc154ff0 2023-03-03T10:19:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dc184e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc1eda10 con 0x7f7dbc0caa30 2023-03-03T10:19:36.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dbaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0c00e0 0x7f7dbc142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dbaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0c00e0 0x7f7dbc142e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59924/0 (socket says 172.21.15.7:59924) 2023-03-03T10:19:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dbaffd700 1 -- 172.21.15.7:0/1364753583 learned_addr learned my addr 172.21.15.7:0/1364753583 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:19:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dbb7fe700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dbc154ff0 0x7f7dbc156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dba7fc700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7dbc0caa30 0x7f7dbc157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dbaffd700 1 -- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7dbc0caa30 msgr2=0x7f7dbc157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dbaffd700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7dbc0caa30 0x7f7dbc157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dbaffd700 1 -- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dbc154ff0 msgr2=0x7f7dbc156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:36.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dbaffd700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dbc154ff0 0x7f7dbc156310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:36.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dbaffd700 1 -- 172.21.15.7:0/1364753583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db4009a70 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.998+0000 7f7dbb7fe700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dbc154ff0 0x7f7dbc156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:19:36.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.999+0000 7f7dbaffd700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0c00e0 0x7f7dbc142e20 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f7db000c750 tx=0x7f7db000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:19:36.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.999+0000 7f7da3fff700 1 -- 172.21.15.7:0/1364753583 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db0004c80 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.999+0000 7f7da3fff700 1 -- 172.21.15.7:0/1364753583 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db0004e00 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.999+0000 7f7dc184e700 1 -- 172.21.15.7:0/1364753583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dbc1547f0 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.999+0000 7f7da3fff700 1 -- 172.21.15.7:0/1364753583 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db0007310 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:35.999+0000 7f7dc184e700 1 -- 172.21.15.7:0/1364753583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dbc154510 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.000+0000 7f7dc184e700 1 -- 172.21.15.7:0/1364753583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dbc154ad0 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.000+0000 7f7da3fff700 1 -- 172.21.15.7:0/1364753583 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7db001a020 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.001+0000 7f7da3fff700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7da80682f0 0x7f7da806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:36.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.001+0000 7f7da3fff700 1 -- 172.21.15.7:0/1364753583 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7db0085e40 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.001+0000 7f7dba7fc700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7da80682f0 0x7f7da806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:36.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.002+0000 7f7dba7fc700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7da80682f0 0x7f7da806a7a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0c00b0 tx=0x7f7dac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:19:36.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.005+0000 7f7da3fff700 1 -- 172.21.15.7:0/1364753583 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7db00520f0 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.245+0000 7f7dc184e700 1 -- 172.21.15.7:0/1364753583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dbc156110 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.245+0000 7f7da3fff700 1 -- 172.21.15.7:0/1364753583 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7db0055710 con 0x7f7dbc0c00e0 2023-03-03T10:19:36.249 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:19:36.249 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:19:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.247+0000 7f7dc184e700 1 -- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7da80682f0 msgr2=0x7f7da806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.247+0000 7f7dc184e700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7da80682f0 0x7f7da806a7a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0c00b0 tx=0x7f7dac008040 comp rx=0 tx=0).stop 2023-03-03T10:19:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.247+0000 7f7dc184e700 1 -- 172.21.15.7:0/1364753583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0c00e0 msgr2=0x7f7dbc142e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.247+0000 7f7dc184e700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0c00e0 0x7f7dbc142e20 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f7db000c750 tx=0x7f7db000cb10 comp rx=0 tx=0).stop 2023-03-03T10:19:36.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.247+0000 7f7dc184e700 1 -- 172.21.15.7:0/1364753583 shutdown_connections 2023-03-03T10:19:36.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.247+0000 7f7dc184e700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7dbc154ff0 0x7f7dbc156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:36.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.247+0000 7f7dc184e700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7da80682f0 0x7f7da806a7a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:36.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.247+0000 7f7dc184e700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dbc0c00e0 0x7f7dbc142e20 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:36.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.247+0000 7f7dc184e700 1 --2- 172.21.15.7:0/1364753583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7dbc0caa30 0x7f7dbc157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:36.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.247+0000 7f7dc184e700 1 -- 172.21.15.7:0/1364753583 >> 172.21.15.7:0/1364753583 conn(0x7f7dbc066e40 msgr2=0x7f7dbc153b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:19:36.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.248+0000 7f7dc184e700 1 -- 172.21.15.7:0/1364753583 shutdown_connections 2023-03-03T10:19:36.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:36.248+0000 7f7dc184e700 1 -- 172.21.15.7:0/1364753583 wait complete. 2023-03-03T10:19:36.253 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:19:36.263 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2023-03-03T10:19:52.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:19:52.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.365+0000 7f1699996700 1 Processor -- start 2023-03-03T10:19:52.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.365+0000 7f1699996700 1 -- start start 2023-03-03T10:19:52.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.365+0000 7f1699996700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f16940c00e0 0x7f16940c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:52.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.365+0000 7f1699996700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1694154ff0 0x7f16940cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:52.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.365+0000 7f1699996700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16940c7310 con 0x7f16940caa30 2023-03-03T10:19:52.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.365+0000 7f1699996700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1694155430 con 0x7f1694154ff0 2023-03-03T10:19:52.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.365+0000 7f1699996700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16940bfd10 con 0x7f16940c00e0 2023-03-03T10:19:52.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f16937fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1694154ff0 0x7f16940cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:52.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f16937fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1694154ff0 0x7f16940cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59942/0 (socket says 172.21.15.7:59942) 2023-03-03T10:19:52.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f16937fe700 1 -- 172.21.15.7:0/2486913529 learned_addr learned my addr 172.21.15.7:0/2486913529 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:19:52.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f16927fc700 1 --2- 172.21.15.7:0/2486913529 >> v2:172.21.15.143:3300/0 conn(0x7f16940c00e0 0x7f16940c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:52.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f16937fe700 1 -- 172.21.15.7:0/2486913529 >> v2:172.21.15.143:3300/0 conn(0x7f16940c00e0 msgr2=0x7f16940c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:19:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f16937fe700 1 --2- 172.21.15.7:0/2486913529 >> v2:172.21.15.143:3300/0 conn(0x7f16940c00e0 0x7f16940c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f16937fe700 1 -- 172.21.15.7:0/2486913529 >> v1:172.21.15.143:6789/0 conn(0x7f16940caa30 legacy=0x7f16940c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f16937fe700 1 -- 172.21.15.7:0/2486913529 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16940bfa30 con 0x7f1694154ff0 2023-03-03T10:19:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f16937fe700 1 --2- 172.21.15.7:0/2486913529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694154ff0 0x7f16940cae00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f16940ca910 tx=0x7f168c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa326c1020055ab server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:19:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f1691ffb700 1 -- 172.21.15.7:0/2486913529 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c00e070 con 0x7f1694154ff0 2023-03-03T10:19:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f1691ffb700 1 -- 172.21.15.7:0/2486913529 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c00b9c0 con 0x7f1694154ff0 2023-03-03T10:19:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.366+0000 7f1691ffb700 1 -- 172.21.15.7:0/2486913529 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c00bd70 con 0x7f1694154ff0 2023-03-03T10:19:52.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.367+0000 7f1699996700 1 -- 172.21.15.7:0/2486913529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694154ff0 msgr2=0x7f16940cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.367+0000 7f1699996700 1 --2- 172.21.15.7:0/2486913529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694154ff0 0x7f16940cae00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f16940ca910 tx=0x7f168c00b330 comp rx=0 tx=0).stop 2023-03-03T10:19:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.367+0000 7f1699996700 1 -- 172.21.15.7:0/2486913529 shutdown_connections 2023-03-03T10:19:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.367+0000 7f1699996700 1 --2- 172.21.15.7:0/2486913529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1694154ff0 0x7f16940cae00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.367+0000 7f1699996700 1 --2- 172.21.15.7:0/2486913529 >> v2:172.21.15.143:3300/0 conn(0x7f16940c00e0 0x7f16940c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.367+0000 7f1699996700 1 -- 172.21.15.7:0/2486913529 >> 172.21.15.7:0/2486913529 conn(0x7f1694066e40 msgr2=0x7f1694071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:19:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.367+0000 7f1699996700 1 -- 172.21.15.7:0/2486913529 shutdown_connections 2023-03-03T10:19:52.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.367+0000 7f1699996700 1 -- 172.21.15.7:0/2486913529 wait complete. 2023-03-03T10:19:52.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.368+0000 7f1699996700 1 Processor -- start 2023-03-03T10:19:52.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.368+0000 7f1699996700 1 -- start start 2023-03-03T10:19:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f1699996700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16940c00e0 0x7f16940c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f1699996700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16940caa30 0x7f1694149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f1699996700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1694154ff0 0x7f169414a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f1699996700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16941deba0 con 0x7f16940caa30 2023-03-03T10:19:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f1699996700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16941dee80 con 0x7f1694154ff0 2023-03-03T10:19:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f1699996700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16941df160 con 0x7f16940c00e0 2023-03-03T10:19:52.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f16927fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16940caa30 0x7f1694149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f16927fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16940caa30 0x7f1694149ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59946/0 (socket says 172.21.15.7:59946) 2023-03-03T10:19:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f16927fc700 1 -- 172.21.15.7:0/561667820 learned_addr learned my addr 172.21.15.7:0/561667820 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:19:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f16937fe700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1694154ff0 0x7f169414a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f1692ffd700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16940c00e0 0x7f16940c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.369+0000 7f16927fc700 1 -- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16940c00e0 msgr2=0x7f16940c1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f16927fc700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16940c00e0 0x7f16940c1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:52.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f16927fc700 1 -- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1694154ff0 msgr2=0x7f169414a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:52.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f16927fc700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1694154ff0 0x7f169414a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:52.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f16927fc700 1 -- 172.21.15.7:0/561667820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c009b30 con 0x7f16940caa30 2023-03-03T10:19:52.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f1692ffd700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16940c00e0 0x7f16940c1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:19:52.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f16927fc700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16940caa30 0x7f1694149ad0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f168400e730 tx=0x7f168400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:19:52.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f167bfff700 1 -- 172.21.15.7:0/561667820 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684013610 con 0x7f16940caa30 2023-03-03T10:19:52.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f167bfff700 1 -- 172.21.15.7:0/561667820 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684013770 con 0x7f16940caa30 2023-03-03T10:19:52.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f167bfff700 1 -- 172.21.15.7:0/561667820 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684013b20 con 0x7f16940caa30 2023-03-03T10:19:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f1699996700 1 -- 172.21.15.7:0/561667820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16941547f0 con 0x7f16940caa30 2023-03-03T10:19:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.370+0000 7f1699996700 1 -- 172.21.15.7:0/561667820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1694154510 con 0x7f16940caa30 2023-03-03T10:19:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.371+0000 7f1699996700 1 -- 172.21.15.7:0/561667820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1694154ad0 con 0x7f16940caa30 2023-03-03T10:19:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.372+0000 7f167bfff700 1 -- 172.21.15.7:0/561667820 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1684019030 con 0x7f16940caa30 2023-03-03T10:19:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.372+0000 7f167bfff700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16800682f0 0x7f168006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:19:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.372+0000 7f167bfff700 1 -- 172.21.15.7:0/561667820 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1684085ac0 con 0x7f16940caa30 2023-03-03T10:19:52.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.372+0000 7f1692ffd700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16800682f0 0x7f168006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:19:52.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.373+0000 7f1692ffd700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16800682f0 0x7f168006a7a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f16941e0ab0 tx=0x7f1688009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:19:52.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.376+0000 7f167bfff700 1 -- 172.21.15.7:0/561667820 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1684051de0 con 0x7f16940caa30 2023-03-03T10:19:52.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.610+0000 7f1699996700 1 -- 172.21.15.7:0/561667820 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1694074bb0 con 0x7f16940caa30 2023-03-03T10:19:52.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.611+0000 7f167bfff700 1 -- 172.21.15.7:0/561667820 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f168400f040 con 0x7f16940caa30 2023-03-03T10:19:52.613 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:19:52.614 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:19:52.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 -- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16800682f0 msgr2=0x7f168006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:52.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16800682f0 0x7f168006a7a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f16941e0ab0 tx=0x7f1688009040 comp rx=0 tx=0).stop 2023-03-03T10:19:52.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 -- 172.21.15.7:0/561667820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16940caa30 msgr2=0x7f1694149ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:19:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16940caa30 0x7f1694149ad0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f168400e730 tx=0x7f168400ea40 comp rx=0 tx=0).stop 2023-03-03T10:19:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 -- 172.21.15.7:0/561667820 shutdown_connections 2023-03-03T10:19:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1694154ff0 0x7f169414a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16940c00e0 0x7f16940c1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16800682f0 0x7f168006a7a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 --2- 172.21.15.7:0/561667820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16940caa30 0x7f1694149ad0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:19:52.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 -- 172.21.15.7:0/561667820 >> 172.21.15.7:0/561667820 conn(0x7f1694066e40 msgr2=0x7f169406b540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:19:52.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 -- 172.21.15.7:0/561667820 shutdown_connections 2023-03-03T10:19:52.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:19:52.613+0000 7f1699996700 1 -- 172.21.15.7:0/561667820 wait complete. 2023-03-03T10:19:52.617 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:19:52.627 INFO:tasks.mds_thrash.fs.[b]:waiting for 105 secs before thrashing 2023-03-03T10:19:58.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:19:58.838 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T10:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T10:20:28.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:28.910 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:44.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:20:44.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.399+0000 7fc7d60ef700 1 Processor -- start 2023-03-03T10:20:44.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.399+0000 7fc7d60ef700 1 -- start start 2023-03-03T10:20:44.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7d60ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d014b060 0x7fc7d006f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:44.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7d60ef700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc7d00c1850 0x7fc7d00c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:44.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7d60ef700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7d00aa170 0x7fc7d00aa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:44.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7d60ef700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d00c4100 con 0x7fc7d00aa170 2023-03-03T10:20:44.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7d60ef700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d006fd40 con 0x7fc7d00c1850 2023-03-03T10:20:44.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7d60ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d0154790 con 0x7fc7d014b060 2023-03-03T10:20:44.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7cffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7d00aa170 0x7fc7d00aa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:44.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7cffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7d00aa170 0x7fc7d00aa5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59984/0 (socket says 172.21.15.7:59984) 2023-03-03T10:20:44.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7cffff700 1 -- 172.21.15.7:0/1413405951 learned_addr learned my addr 172.21.15.7:0/1413405951 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:20:44.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/1413405951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d014b060 0x7fc7d006f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:44.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7ceffd700 1 --2- 172.21.15.7:0/1413405951 >> v2:172.21.15.143:3300/0 conn(0x7fc7d00c1850 0x7fc7d00c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:44.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7cffff700 1 -- 172.21.15.7:0/1413405951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d014b060 msgr2=0x7fc7d006f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:20:44.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7cffff700 1 --2- 172.21.15.7:0/1413405951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d014b060 0x7fc7d006f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7cffff700 1 -- 172.21.15.7:0/1413405951 >> v2:172.21.15.143:3300/0 conn(0x7fc7d00c1850 msgr2=0x7fc7d00c1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:20:44.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7cffff700 1 --2- 172.21.15.7:0/1413405951 >> v2:172.21.15.143:3300/0 conn(0x7fc7d00c1850 0x7fc7d00c1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7cffff700 1 -- 172.21.15.7:0/1413405951 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d0154a70 con 0x7fc7d00aa170 2023-03-03T10:20:44.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.400+0000 7fc7cffff700 1 --2- 172.21.15.7:0/1413405951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00aa170 0x7fc7d00aa5b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00704d0 tx=0x7fc7c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa6c6b48be637da server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:20:44.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7ce7fc700 1 -- 172.21.15.7:0/1413405951 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c800e070 con 0x7fc7d00aa170 2023-03-03T10:20:44.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7ce7fc700 1 -- 172.21.15.7:0/1413405951 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c800b9c0 con 0x7fc7d00aa170 2023-03-03T10:20:44.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7ce7fc700 1 -- 172.21.15.7:0/1413405951 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c800bd70 con 0x7fc7d00aa170 2023-03-03T10:20:44.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7d60ef700 1 -- 172.21.15.7:0/1413405951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00aa170 msgr2=0x7fc7d00aa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:44.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7d60ef700 1 --2- 172.21.15.7:0/1413405951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00aa170 0x7fc7d00aa5b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00704d0 tx=0x7fc7c800b330 comp rx=0 tx=0).stop 2023-03-03T10:20:44.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7d60ef700 1 -- 172.21.15.7:0/1413405951 shutdown_connections 2023-03-03T10:20:44.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7d60ef700 1 --2- 172.21.15.7:0/1413405951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00aa170 0x7fc7d00aa5b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7d60ef700 1 --2- 172.21.15.7:0/1413405951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d014b060 0x7fc7d006f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7d60ef700 1 --2- 172.21.15.7:0/1413405951 >> v2:172.21.15.143:3300/0 conn(0x7fc7d00c1850 0x7fc7d00c1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7d60ef700 1 -- 172.21.15.7:0/1413405951 >> 172.21.15.7:0/1413405951 conn(0x7fc7d00c4f20 msgr2=0x7fc7d00c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:20:44.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7d60ef700 1 -- 172.21.15.7:0/1413405951 shutdown_connections 2023-03-03T10:20:44.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.401+0000 7fc7d60ef700 1 -- 172.21.15.7:0/1413405951 wait complete. 2023-03-03T10:20:44.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.402+0000 7fc7d60ef700 1 Processor -- start 2023-03-03T10:20:44.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.403+0000 7fc7d60ef700 1 -- start start 2023-03-03T10:20:44.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.403+0000 7fc7d60ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00aa170 0x7fc7d00ab950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:44.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.403+0000 7fc7d60ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d00c1850 0x7fc7d0156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:44.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.403+0000 7fc7d60ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014b060 0x7fc7d01570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:44.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.403+0000 7fc7d60ef700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d0156740 con 0x7fc7d014b060 2023-03-03T10:20:44.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.403+0000 7fc7d60ef700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d01587e0 con 0x7fc7d00c1850 2023-03-03T10:20:44.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.403+0000 7fc7d60ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d0158220 con 0x7fc7d00aa170 2023-03-03T10:20:44.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7cffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014b060 0x7fc7d01570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:44.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7cffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014b060 0x7fc7d01570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59990/0 (socket says 172.21.15.7:59990) 2023-03-03T10:20:44.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7cffff700 1 -- 172.21.15.7:0/3047833004 learned_addr learned my addr 172.21.15.7:0/3047833004 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:20:44.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7ceffd700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d00c1850 0x7fc7d0156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:44.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00aa170 0x7fc7d00ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:44.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7cffff700 1 -- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00aa170 msgr2=0x7fc7d00ab950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:44.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7cffff700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00aa170 0x7fc7d00ab950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7cffff700 1 -- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d00c1850 msgr2=0x7fc7d0156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7cffff700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d00c1850 0x7fc7d0156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7cffff700 1 -- 172.21.15.7:0/3047833004 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c8009b30 con 0x7fc7d014b060 2023-03-03T10:20:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7cffff700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014b060 0x7fc7d01570a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00726b0 tx=0x7fc7c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:20:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3047833004 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80099b0 con 0x7fc7d014b060 2023-03-03T10:20:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3047833004 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c8004250 con 0x7fc7d014b060 2023-03-03T10:20:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7d60ef700 1 -- 172.21.15.7:0/3047833004 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d0155030 con 0x7fc7d014b060 2023-03-03T10:20:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3047833004 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8004600 con 0x7fc7d014b060 2023-03-03T10:20:44.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.404+0000 7fc7d60ef700 1 -- 172.21.15.7:0/3047833004 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d0154d50 con 0x7fc7d014b060 2023-03-03T10:20:44.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.405+0000 7fc7d60ef700 1 -- 172.21.15.7:0/3047833004 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d0155310 con 0x7fc7d014b060 2023-03-03T10:20:44.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.406+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3047833004 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc7c802c070 con 0x7fc7d014b060 2023-03-03T10:20:44.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.406+0000 7fc7c7fff700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:20:44.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.406+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3047833004 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc7c8087710 con 0x7fc7d014b060 2023-03-03T10:20:44.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.406+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:20:44.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.407+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc7d01589c0 tx=0x7fc7c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:20:44.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.411+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3047833004 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc7c8053ab0 con 0x7fc7d014b060 2023-03-03T10:20:44.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.647+0000 7fc7d60ef700 1 -- 172.21.15.7:0/3047833004 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7d0156880 con 0x7fc7d014b060 2023-03-03T10:20:44.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.647+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3047833004 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc7c80570d0 con 0x7fc7d014b060 2023-03-03T10:20:44.651 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:20:44.651 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:20:44.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 -- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 msgr2=0x7fc7b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:44.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc7d01589c0 tx=0x7fc7c0008040 comp rx=0 tx=0).stop 2023-03-03T10:20:44.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 -- 172.21.15.7:0/3047833004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014b060 msgr2=0x7fc7d01570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:20:44.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014b060 0x7fc7d01570a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00726b0 tx=0x7fc7c8004ad0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 -- 172.21.15.7:0/3047833004 shutdown_connections 2023-03-03T10:20:44.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014b060 0x7fc7d01570a0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00aa170 0x7fc7d00ab950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 --2- 172.21.15.7:0/3047833004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d00c1850 0x7fc7d0156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:20:44.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 -- 172.21.15.7:0/3047833004 >> 172.21.15.7:0/3047833004 conn(0x7fc7d00c4f20 msgr2=0x7fc7d00aa540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:20:44.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 -- 172.21.15.7:0/3047833004 shutdown_connections 2023-03-03T10:20:44.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:20:44.649+0000 7fc7d60ef700 1 -- 172.21.15.7:0/3047833004 wait complete. 2023-03-03T10:20:44.654 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:20:44.664 INFO:tasks.mds_thrash.fs.[a]:waiting for 24 secs before thrashing 2023-03-03T10:20:58.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:20:58.981 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:08.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:21:08.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.803+0000 7fe8e77a7700 1 Processor -- start 2023-03-03T10:21:08.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.803+0000 7fe8e77a7700 1 -- start start 2023-03-03T10:21:08.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8e77a7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe8e014e900 0x7fe8e015ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:08.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8e77a7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e01510d0 con 0x7fe8e0071e80 2023-03-03T10:21:08.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8e77a7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e00c3740 con 0x7fe8e0072250 2023-03-03T10:21:08.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8e77a7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e0070670 con 0x7fe8e014e900 2023-03-03T10:21:08.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8e5d44700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe8e014e900 0x7fe8e015ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:08.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8e5543700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe8e0071e80 0x7fe8e0151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53986/0 (socket says 172.21.15.7:53986) 2023-03-03T10:21:08.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8e5543700 1 -- 172.21.15.7:0/3403150644 learned_addr learned my addr 172.21.15.7:0/3403150644 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:21:08.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8e5d44700 1 -- 172.21.15.7:0/3403150644 >> v1:172.21.15.143:6789/0 conn(0x7fe8e0072250 legacy=0x7fe8e014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:21:08.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8e5d44700 1 -- 172.21.15.7:0/3403150644 >> v1:172.21.15.7:6789/0 conn(0x7fe8e0071e80 legacy=0x7fe8e0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:08.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8d7fff700 1 -- 172.21.15.7:0/3403150644 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538474943 0 0) 0x7fe8e01510d0 con 0x7fe8e0071e80 2023-03-03T10:21:08.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.804+0000 7fe8e5d44700 1 -- 172.21.15.7:0/3403150644 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e00c39e0 con 0x7fe8e014e900 2023-03-03T10:21:08.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8e5d44700 1 --2- 172.21.15.7:0/3403150644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 0x7fe8e015ca60 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8009d90 tx=0x7fe8d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd41469d9c051996 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:08.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8d7fff700 1 -- 172.21.15.7:0/3403150644 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d800e070 con 0x7fe8e014e900 2023-03-03T10:21:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8d7fff700 1 -- 172.21.15.7:0/3403150644 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d800bbd0 con 0x7fe8e014e900 2023-03-03T10:21:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8d7fff700 1 -- 172.21.15.7:0/3403150644 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8012430 con 0x7fe8e014e900 2023-03-03T10:21:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3403150644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 msgr2=0x7fe8e015ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8e77a7700 1 --2- 172.21.15.7:0/3403150644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 0x7fe8e015ca60 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8009d90 tx=0x7fe8d800b330 comp rx=0 tx=0).stop 2023-03-03T10:21:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3403150644 shutdown_connections 2023-03-03T10:21:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8e77a7700 1 --2- 172.21.15.7:0/3403150644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 0x7fe8e015ca60 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:08.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3403150644 >> 172.21.15.7:0/3403150644 conn(0x7fe8e00a94d0 msgr2=0x7fe8e00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:21:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3403150644 shutdown_connections 2023-03-03T10:21:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.805+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3403150644 wait complete. 2023-03-03T10:21:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e77a7700 1 Processor -- start 2023-03-03T10:21:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e77a7700 1 -- start start 2023-03-03T10:21:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e77a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8e0071e80 0x7fe8e0160c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e77a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8e0072250 0x7fe8e0149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:08.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e77a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 0x7fe8e00bf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e77a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e0160520 con 0x7fe8e014e900 2023-03-03T10:21:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e77a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e01607c0 con 0x7fe8e0072250 2023-03-03T10:21:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e77a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e00be220 con 0x7fe8e0071e80 2023-03-03T10:21:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e5d44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 0x7fe8e00bf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e5d44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 0x7fe8e00bf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60014/0 (socket says 172.21.15.7:60014) 2023-03-03T10:21:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.807+0000 7fe8e5d44700 1 -- 172.21.15.7:0/3229332198 learned_addr learned my addr 172.21.15.7:0/3229332198 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:21:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e5543700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8e0071e80 0x7fe8e0160c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:08.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e4d42700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8e0072250 0x7fe8e0149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e5d44700 1 -- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8e0071e80 msgr2=0x7fe8e0160c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e5d44700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8e0071e80 0x7fe8e0160c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e5d44700 1 -- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8e0072250 msgr2=0x7fe8e0149830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e5d44700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8e0072250 0x7fe8e0149830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e5d44700 1 -- 172.21.15.7:0/3229332198 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d8009a70 con 0x7fe8e014e900 2023-03-03T10:21:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e5543700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8e0071e80 0x7fe8e0160c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:21:08.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e4d42700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8e0072250 0x7fe8e0149830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:21:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e5d44700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 0x7fe8e00bf480 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe8e006b550 tx=0x7fe8d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8d5ffb700 1 -- 172.21.15.7:0/3229332198 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d801b8a0 con 0x7fe8e014e900 2023-03-03T10:21:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8d5ffb700 1 -- 172.21.15.7:0/3229332198 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8d8020070 con 0x7fe8e014e900 2023-03-03T10:21:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3229332198 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e0151f30 con 0x7fe8e014e900 2023-03-03T10:21:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8d5ffb700 1 -- 172.21.15.7:0/3229332198 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d801bc60 con 0x7fe8e014e900 2023-03-03T10:21:08.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.808+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3229332198 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e0151300 con 0x7fe8e014e900 2023-03-03T10:21:08.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.809+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3229332198 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e006c9a0 con 0x7fe8e014e900 2023-03-03T10:21:08.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.810+0000 7fe8d5ffb700 1 -- 172.21.15.7:0/3229332198 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe8d8026400 con 0x7fe8e014e900 2023-03-03T10:21:08.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.810+0000 7fe8d5ffb700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8c40682f0 0x7fe8c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:08.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.810+0000 7fe8d5ffb700 1 -- 172.21.15.7:0/3229332198 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe8d8088270 con 0x7fe8e014e900 2023-03-03T10:21:08.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.810+0000 7fe8e5543700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8c40682f0 0x7fe8c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:08.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.811+0000 7fe8e5543700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8c40682f0 0x7fe8c406a7a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0000f60 tx=0x7fe8d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:08.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:08.814+0000 7fe8d5ffb700 1 -- 172.21.15.7:0/3229332198 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8d80545b0 con 0x7fe8e014e900 2023-03-03T10:21:09.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.051+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3229332198 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8e0160970 con 0x7fe8e014e900 2023-03-03T10:21:09.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.052+0000 7fe8d5ffb700 1 -- 172.21.15.7:0/3229332198 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe8e0160970 con 0x7fe8e014e900 2023-03-03T10:21:09.054 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:21:09.054 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:21:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.053+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8c40682f0 msgr2=0x7fe8c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.053+0000 7fe8e77a7700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8c40682f0 0x7fe8c406a7a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0000f60 tx=0x7fe8d0009040 comp rx=0 tx=0).stop 2023-03-03T10:21:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.053+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3229332198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 msgr2=0x7fe8e00bf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:09.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.053+0000 7fe8e77a7700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 0x7fe8e00bf480 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe8e006b550 tx=0x7fe8d8004ad0 comp rx=0 tx=0).stop 2023-03-03T10:21:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.054+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3229332198 shutdown_connections 2023-03-03T10:21:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.054+0000 7fe8e77a7700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8c40682f0 0x7fe8c406a7a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.054+0000 7fe8e77a7700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8e014e900 0x7fe8e00bf480 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.054+0000 7fe8e77a7700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8e0071e80 0x7fe8e0160c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.054+0000 7fe8e77a7700 1 --2- 172.21.15.7:0/3229332198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8e0072250 0x7fe8e0149830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.054+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3229332198 >> 172.21.15.7:0/3229332198 conn(0x7fe8e00a94d0 msgr2=0x7fe8e00c2940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:21:09.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.054+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3229332198 shutdown_connections 2023-03-03T10:21:09.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:09.054+0000 7fe8e77a7700 1 -- 172.21.15.7:0/3229332198 wait complete. 2023-03-03T10:21:09.058 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:21:09.068 INFO:tasks.mds_thrash.fs.[a]:waiting for 39 secs before thrashing 2023-03-03T10:21:29.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:29.236 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:37.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:21:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.764+0000 7f006c4dd700 1 Processor -- start 2023-03-03T10:21:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.764+0000 7f006c4dd700 1 -- start start 2023-03-03T10:21:37.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.765+0000 7f006c4dd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0064155390 0x7f00640c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:37.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.765+0000 7f006c4dd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f006406e020 0x7f0064071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:37.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.765+0000 7f006c4dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 0x7f0064151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.765+0000 7f006c4dd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0064070b90 con 0x7f0064155390 2023-03-03T10:21:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.765+0000 7f006c4dd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0064151320 con 0x7f006406e020 2023-03-03T10:21:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.765+0000 7f006c4dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00640c4780 con 0x7f0064072520 2023-03-03T10:21:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.765+0000 7f006a279700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0064155390 0x7f00640c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.765+0000 7f006a279700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0064155390 0x7f00640c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60044/0 (socket says 172.21.15.7:60044) 2023-03-03T10:21:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.765+0000 7f006a279700 1 -- 172.21.15.7:0/1679797312 learned_addr learned my addr 172.21.15.7:0/1679797312 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:21:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f006aa7a700 1 --2- 172.21.15.7:0/1679797312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 0x7f0064151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f0069a78700 1 --2- 172.21.15.7:0/1679797312 >> v2:172.21.15.143:3300/0 conn(0x7f006406e020 0x7f0064071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f006a279700 1 -- 172.21.15.7:0/1679797312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 msgr2=0x7f0064151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:21:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f006a279700 1 --2- 172.21.15.7:0/1679797312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 0x7f0064151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f006a279700 1 -- 172.21.15.7:0/1679797312 >> v2:172.21.15.143:3300/0 conn(0x7f006406e020 msgr2=0x7f0064071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:21:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f006a279700 1 --2- 172.21.15.7:0/1679797312 >> v2:172.21.15.143:3300/0 conn(0x7f006406e020 0x7f0064071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f006a279700 1 -- 172.21.15.7:0/1679797312 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0064154910 con 0x7f0064155390 2023-03-03T10:21:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f006aa7a700 1 --2- 172.21.15.7:0/1679797312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 0x7f0064151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:21:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f0069a78700 1 --2- 172.21.15.7:0/1679797312 >> v2:172.21.15.143:3300/0 conn(0x7f006406e020 0x7f0064071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:21:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f006a279700 1 --2- 172.21.15.7:0/1679797312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0064155390 0x7f00640c3e30 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f00640ac9e0 tx=0x7f0054009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8612516424d63f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f0069277700 1 -- 172.21.15.7:0/1679797312 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005400b860 con 0x7f0064155390 2023-03-03T10:21:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f0069277700 1 -- 172.21.15.7:0/1679797312 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005400b9c0 con 0x7f0064155390 2023-03-03T10:21:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.766+0000 7f0069277700 1 -- 172.21.15.7:0/1679797312 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005400bd50 con 0x7f0064155390 2023-03-03T10:21:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.767+0000 7f006c4dd700 1 -- 172.21.15.7:0/1679797312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0064155390 msgr2=0x7f00640c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.767+0000 7f006c4dd700 1 --2- 172.21.15.7:0/1679797312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0064155390 0x7f00640c3e30 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f00640ac9e0 tx=0x7f0054009c00 comp rx=0 tx=0).stop 2023-03-03T10:21:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.767+0000 7f006c4dd700 1 -- 172.21.15.7:0/1679797312 shutdown_connections 2023-03-03T10:21:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.767+0000 7f006c4dd700 1 --2- 172.21.15.7:0/1679797312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 0x7f0064151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:37.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.767+0000 7f006c4dd700 1 --2- 172.21.15.7:0/1679797312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0064155390 0x7f00640c3e30 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:37.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.767+0000 7f006c4dd700 1 --2- 172.21.15.7:0/1679797312 >> v2:172.21.15.143:3300/0 conn(0x7f006406e020 0x7f0064071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:37.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.767+0000 7f006c4dd700 1 -- 172.21.15.7:0/1679797312 >> 172.21.15.7:0/1679797312 conn(0x7f006406c110 msgr2=0x7f006406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:21:37.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.767+0000 7f006c4dd700 1 -- 172.21.15.7:0/1679797312 shutdown_connections 2023-03-03T10:21:37.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.767+0000 7f006c4dd700 1 -- 172.21.15.7:0/1679797312 wait complete. 2023-03-03T10:21:37.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.768+0000 7f006c4dd700 1 Processor -- start 2023-03-03T10:21:37.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.768+0000 7f006c4dd700 1 -- start start 2023-03-03T10:21:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006c4dd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f006406e020 0x7f00641dd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006c4dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 0x7f0064148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006c4dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0064155390 0x7f0064148f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006c4dd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00640c5200 con 0x7f006406e020 2023-03-03T10:21:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006c4dd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00640c54a0 con 0x7f0064155390 2023-03-03T10:21:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006c4dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00641ea7a0 con 0x7f0064072520 2023-03-03T10:21:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006a279700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f006406e020 0x7f00641dd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:37.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006a279700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f006406e020 0x7f00641dd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60050/0 (socket says 172.21.15.7:60050) 2023-03-03T10:21:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006a279700 1 -- 172.21.15.7:0/3532396092 learned_addr learned my addr 172.21.15.7:0/3532396092 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:21:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f0069a78700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 0x7f0064148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006a279700 1 -- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 msgr2=0x7f0064148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006a279700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 0x7f0064148a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006aa7a700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0064155390 0x7f0064148f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006a279700 1 -- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0064155390 msgr2=0x7f0064148f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006a279700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0064155390 0x7f0064148f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:37.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006a279700 1 -- 172.21.15.7:0/3532396092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0054009550 con 0x7f006406e020 2023-03-03T10:21:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006a279700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f006406e020 0x7f00641dd070 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0054000c00 tx=0x7f0054004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f005affd700 1 -- 172.21.15.7:0/3532396092 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00540036a0 con 0x7f006406e020 2023-03-03T10:21:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f005affd700 1 -- 172.21.15.7:0/3532396092 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005401e070 con 0x7f006406e020 2023-03-03T10:21:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.769+0000 7f006c4dd700 1 -- 172.21.15.7:0/3532396092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0064070540 con 0x7f006406e020 2023-03-03T10:21:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.770+0000 7f005affd700 1 -- 172.21.15.7:0/3532396092 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0054003a60 con 0x7f006406e020 2023-03-03T10:21:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.770+0000 7f006c4dd700 1 -- 172.21.15.7:0/3532396092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00640702a0 con 0x7f006406e020 2023-03-03T10:21:37.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.770+0000 7f006c4dd700 1 -- 172.21.15.7:0/3532396092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0064070820 con 0x7f006406e020 2023-03-03T10:21:37.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.772+0000 7f005affd700 1 -- 172.21.15.7:0/3532396092 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f005400f050 con 0x7f006406e020 2023-03-03T10:21:37.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.772+0000 7f005affd700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00480682f0 0x7f004806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:37.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.772+0000 7f005affd700 1 -- 172.21.15.7:0/3532396092 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0054020070 con 0x7f006406e020 2023-03-03T10:21:37.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.772+0000 7f0069a78700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00480682f0 0x7f004806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:37.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.773+0000 7f0069a78700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00480682f0 0x7f004806a7a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f00640c3940 tx=0x7f0060008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:37.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:37.775+0000 7f005affd700 1 -- 172.21.15.7:0/3532396092 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0054053db0 con 0x7f006406e020 2023-03-03T10:21:38.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.014+0000 7f006c4dd700 1 -- 172.21.15.7:0/3532396092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0064074bb0 con 0x7f006406e020 2023-03-03T10:21:38.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.014+0000 7f005affd700 1 -- 172.21.15.7:0/3532396092 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f00540573d0 con 0x7f006406e020 2023-03-03T10:21:38.017 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:21:38.017 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:21:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 -- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00480682f0 msgr2=0x7f004806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00480682f0 0x7f004806a7a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f00640c3940 tx=0x7f0060008040 comp rx=0 tx=0).stop 2023-03-03T10:21:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 -- 172.21.15.7:0/3532396092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f006406e020 msgr2=0x7f00641dd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f006406e020 0x7f00641dd070 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0054000c00 tx=0x7f0054004750 comp rx=0 tx=0).stop 2023-03-03T10:21:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 -- 172.21.15.7:0/3532396092 shutdown_connections 2023-03-03T10:21:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0064155390 0x7f0064148f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00480682f0 0x7f004806a7a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f006406e020 0x7f00641dd070 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 --2- 172.21.15.7:0/3532396092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0064072520 0x7f0064148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:38.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 -- 172.21.15.7:0/3532396092 >> 172.21.15.7:0/3532396092 conn(0x7f006406c110 msgr2=0x7f006414afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:21:38.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 -- 172.21.15.7:0/3532396092 shutdown_connections 2023-03-03T10:21:38.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:38.016+0000 7f006c4dd700 1 -- 172.21.15.7:0/3532396092 wait complete. 2023-03-03T10:21:38.020 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:21:38.030 INFO:tasks.mds_thrash.fs.[b]:waiting for 103 secs before thrashing 2023-03-03T10:21:48.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:21:48.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.204+0000 7f8957efb700 1 Processor -- start 2023-03-03T10:21:48.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.205+0000 7f8957efb700 1 -- start start 2023-03-03T10:21:48.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.205+0000 7f8957efb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8950072250 0x7f895014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:48.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.205+0000 7f8957efb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f89500c1ce0 0x7f895014e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:48.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.205+0000 7f8957efb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89501510d0 con 0x7f8950071e80 2023-03-03T10:21:48.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.205+0000 7f8957efb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89500c3740 con 0x7f8950072250 2023-03-03T10:21:48.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.205+0000 7f8957efb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8950070670 con 0x7f89500c1ce0 2023-03-03T10:21:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.205+0000 7f8955496700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8950072250 0x7f895014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.205+0000 7f8955496700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8950072250 0x7f895014e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60066/0 (socket says 172.21.15.7:60066) 2023-03-03T10:21:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.205+0000 7f8955496700 1 -- 172.21.15.7:0/28867778 learned_addr learned my addr 172.21.15.7:0/28867778 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:21:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.206+0000 7f8956498700 1 --2- 172.21.15.7:0/28867778 >> v2:172.21.15.143:3300/0 conn(0x7f89500c1ce0 0x7f895014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.206+0000 7f8955496700 1 -- 172.21.15.7:0/28867778 >> v2:172.21.15.143:3300/0 conn(0x7f89500c1ce0 msgr2=0x7f895014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:21:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.206+0000 7f8955496700 1 --2- 172.21.15.7:0/28867778 >> v2:172.21.15.143:3300/0 conn(0x7f89500c1ce0 0x7f895014e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.206+0000 7f8955496700 1 -- 172.21.15.7:0/28867778 >> v1:172.21.15.143:6789/0 conn(0x7f8950071e80 legacy=0x7f8950151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:21:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.206+0000 7f8955496700 1 -- 172.21.15.7:0/28867778 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89500c39e0 con 0x7f8950072250 2023-03-03T10:21:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.206+0000 7f8956498700 1 --2- 172.21.15.7:0/28867778 >> v2:172.21.15.143:3300/0 conn(0x7f89500c1ce0 0x7f895014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:21:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.206+0000 7f8955496700 1 --2- 172.21.15.7:0/28867778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950072250 0x7f895014e1f0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8950071090 tx=0x7f894c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51368e4a64cfca9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.206+0000 7f8954c95700 1 -- 172.21.15.7:0/28867778 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c004230 con 0x7f8950072250 2023-03-03T10:21:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.206+0000 7f8954c95700 1 -- 172.21.15.7:0/28867778 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894c004390 con 0x7f8950072250 2023-03-03T10:21:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.206+0000 7f8954c95700 1 -- 172.21.15.7:0/28867778 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c0036a0 con 0x7f8950072250 2023-03-03T10:21:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.207+0000 7f8957efb700 1 -- 172.21.15.7:0/28867778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950072250 msgr2=0x7f895014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.207+0000 7f8957efb700 1 --2- 172.21.15.7:0/28867778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950072250 0x7f895014e1f0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8950071090 tx=0x7f894c009bd0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.207+0000 7f8957efb700 1 -- 172.21.15.7:0/28867778 shutdown_connections 2023-03-03T10:21:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.207+0000 7f8957efb700 1 --2- 172.21.15.7:0/28867778 >> v2:172.21.15.143:3300/0 conn(0x7f89500c1ce0 0x7f895014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.207+0000 7f8957efb700 1 --2- 172.21.15.7:0/28867778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950072250 0x7f895014e1f0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.207+0000 7f8957efb700 1 -- 172.21.15.7:0/28867778 >> 172.21.15.7:0/28867778 conn(0x7f89500a94d0 msgr2=0x7f89500a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:21:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.207+0000 7f8957efb700 1 -- 172.21.15.7:0/28867778 shutdown_connections 2023-03-03T10:21:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.207+0000 7f8957efb700 1 -- 172.21.15.7:0/28867778 wait complete. 2023-03-03T10:21:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.208+0000 7f8957efb700 1 Processor -- start 2023-03-03T10:21:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.208+0000 7f8957efb700 1 -- start start 2023-03-03T10:21:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8957efb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950071e80 0x7f89500aaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8957efb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8950072250 0x7f89501eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8957efb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89500c1ce0 0x7f89501ebb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8957efb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89500ab9c0 con 0x7f8950071e80 2023-03-03T10:21:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8957efb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89501eb160 con 0x7f89500c1ce0 2023-03-03T10:21:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8957efb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89500ace80 con 0x7f8950072250 2023-03-03T10:21:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8955c97700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950071e80 0x7f89500aaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8955c97700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950071e80 0x7f89500aaf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60070/0 (socket says 172.21.15.7:60070) 2023-03-03T10:21:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8955c97700 1 -- 172.21.15.7:0/1924782714 learned_addr learned my addr 172.21.15.7:0/1924782714 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:21:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8956498700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89500c1ce0 0x7f89501ebb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8955496700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8950072250 0x7f89501eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8955c97700 1 -- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8950072250 msgr2=0x7f89501eb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8955c97700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8950072250 0x7f89501eb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8955c97700 1 -- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89500c1ce0 msgr2=0x7f89501ebb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8955c97700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89500c1ce0 0x7f89501ebb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8955c97700 1 -- 172.21.15.7:0/1924782714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c009550 con 0x7f8950071e80 2023-03-03T10:21:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8955c97700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950071e80 0x7f89500aaf50 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f894000c500 tx=0x7f894000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f89467fc700 1 -- 172.21.15.7:0/1924782714 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940011790 con 0x7f8950071e80 2023-03-03T10:21:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f89467fc700 1 -- 172.21.15.7:0/1924782714 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89400118f0 con 0x7f8950071e80 2023-03-03T10:21:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8957efb700 1 -- 172.21.15.7:0/1924782714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8950151f30 con 0x7f8950071e80 2023-03-03T10:21:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f89467fc700 1 -- 172.21.15.7:0/1924782714 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940011ca0 con 0x7f8950071e80 2023-03-03T10:21:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.209+0000 7f8957efb700 1 -- 172.21.15.7:0/1924782714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8950151300 con 0x7f8950071e80 2023-03-03T10:21:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.210+0000 7f8957efb700 1 -- 172.21.15.7:0/1924782714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f895006c9a0 con 0x7f8950071e80 2023-03-03T10:21:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.211+0000 7f89467fc700 1 -- 172.21.15.7:0/1924782714 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8940010050 con 0x7f8950071e80 2023-03-03T10:21:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.211+0000 7f89467fc700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8934068320 0x7f893406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:21:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.212+0000 7f89467fc700 1 -- 172.21.15.7:0/1924782714 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8940085e80 con 0x7f8950071e80 2023-03-03T10:21:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.212+0000 7f8955496700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8934068320 0x7f893406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:21:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.212+0000 7f8955496700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8934068320 0x7f893406a7d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f89500c1780 tx=0x7f894c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:21:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.216+0000 7f89467fc700 1 -- 172.21.15.7:0/1924782714 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8940052220 con 0x7f8950071e80 2023-03-03T10:21:48.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.453+0000 7f8957efb700 1 -- 172.21.15.7:0/1924782714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89500aa460 con 0x7f8950071e80 2023-03-03T10:21:48.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.453+0000 7f89467fc700 1 -- 172.21.15.7:0/1924782714 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8940055840 con 0x7f8950071e80 2023-03-03T10:21:48.457 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:21:48.457 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:21:48.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 -- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8934068320 msgr2=0x7f893406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:48.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8934068320 0x7f893406a7d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f89500c1780 tx=0x7f894c008030 comp rx=0 tx=0).stop 2023-03-03T10:21:48.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 -- 172.21.15.7:0/1924782714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950071e80 msgr2=0x7f89500aaf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:21:48.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950071e80 0x7f89500aaf50 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f894000c500 tx=0x7f894000c8c0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 -- 172.21.15.7:0/1924782714 shutdown_connections 2023-03-03T10:21:48.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89500c1ce0 0x7f89501ebb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8934068320 0x7f893406a7d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8950071e80 0x7f89500aaf50 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 --2- 172.21.15.7:0/1924782714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8950072250 0x7f89501eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:21:48.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 -- 172.21.15.7:0/1924782714 >> 172.21.15.7:0/1924782714 conn(0x7f89500a94d0 msgr2=0x7f89500c2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:21:48.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 -- 172.21.15.7:0/1924782714 shutdown_connections 2023-03-03T10:21:48.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:21:48.455+0000 7f8957efb700 1 -- 172.21.15.7:0/1924782714 wait complete. 2023-03-03T10:21:48.460 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:21:48.470 INFO:tasks.mds_thrash.fs.[a]:waiting for 24 secs before thrashing 2023-03-03T10:21:59.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:21:59.308 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:12.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:22:12.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.608+0000 7f049fc49700 1 Processor -- start 2023-03-03T10:22:12.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.608+0000 7f049fc49700 1 -- start start 2023-03-03T10:22:12.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049fc49700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 0x7f04981592f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:22:12.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049fc49700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f049806e7e0 0x7f049806ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:22:12.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049fc49700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04980c6cb0 con 0x7f04980c64f0 2023-03-03T10:22:12.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049fc49700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04980c7ec0 con 0x7f049806e7e0 2023-03-03T10:22:12.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049fc49700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04980c7100 con 0x7f04981529a0 2023-03-03T10:22:12.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049d9e5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 0x7f04981592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:22:12.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049e1e6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f049806e7e0 0x7f049806ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:22:12.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049d9e5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 0x7f04981592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:39930/0 (socket says 172.21.15.7:39930) 2023-03-03T10:22:12.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049e1e6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f049806e7e0 0x7f049806ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:55866/0 (socket says 172.21.15.7:55866) 2023-03-03T10:22:12.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049d9e5700 1 -- 172.21.15.7:0/431853408 learned_addr learned my addr 172.21.15.7:0/431853408 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:22:12.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.609+0000 7f049d1e4700 1 --1- 172.21.15.7:0/431853408 >> v1:172.21.15.143:6789/0 conn(0x7f04980c64f0 0x7f04980caa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:57590/0 (socket says 172.21.15.7:57590) 2023-03-03T10:22:12.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049c9e3700 1 -- 172.21.15.7:0/431853408 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642782107 0 0) 0x7f04980c6cb0 con 0x7f04980c64f0 2023-03-03T10:22:12.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049c9e3700 1 -- 172.21.15.7:0/431853408 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04840035b0 con 0x7f04980c64f0 2023-03-03T10:22:12.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049e1e6700 1 -- 172.21.15.7:0/431853408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 msgr2=0x7f04981592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:22:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049e1e6700 1 --2- 172.21.15.7:0/431853408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 0x7f04981592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049e1e6700 1 -- 172.21.15.7:0/431853408 >> v1:172.21.15.143:6789/0 conn(0x7f04980c64f0 legacy=0x7f04980caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:22:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049e1e6700 1 -- 172.21.15.7:0/431853408 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04980c73e0 con 0x7f049806e7e0 2023-03-03T10:22:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049d9e5700 1 --2- 172.21.15.7:0/431853408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 0x7f04981592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:22:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049e1e6700 1 --2- 172.21.15.7:0/431853408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f049806e7e0 0x7f049806ec40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0490009d90 tx=0x7f049000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a1cf485dd81aa5c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:22:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049c9e3700 1 -- 172.21.15.7:0/431853408 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049000e070 con 0x7f049806e7e0 2023-03-03T10:22:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049c9e3700 1 -- 172.21.15.7:0/431853408 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049000bb20 con 0x7f049806e7e0 2023-03-03T10:22:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.610+0000 7f049c9e3700 1 -- 172.21.15.7:0/431853408 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490012430 con 0x7f049806e7e0 2023-03-03T10:22:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.611+0000 7f049fc49700 1 -- 172.21.15.7:0/431853408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f049806e7e0 msgr2=0x7f049806ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:22:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.611+0000 7f049fc49700 1 --2- 172.21.15.7:0/431853408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f049806e7e0 0x7f049806ec40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0490009d90 tx=0x7f049000b330 comp rx=0 tx=0).stop 2023-03-03T10:22:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.611+0000 7f049fc49700 1 -- 172.21.15.7:0/431853408 shutdown_connections 2023-03-03T10:22:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.611+0000 7f049fc49700 1 --2- 172.21.15.7:0/431853408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f049806e7e0 0x7f049806ec40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.611+0000 7f049fc49700 1 --2- 172.21.15.7:0/431853408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 0x7f04981592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.611+0000 7f049fc49700 1 -- 172.21.15.7:0/431853408 >> 172.21.15.7:0/431853408 conn(0x7f0498066e40 msgr2=0x7f0498071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:22:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.611+0000 7f049fc49700 1 -- 172.21.15.7:0/431853408 shutdown_connections 2023-03-03T10:22:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.611+0000 7f049fc49700 1 -- 172.21.15.7:0/431853408 wait complete. 2023-03-03T10:22:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.612+0000 7f049fc49700 1 Processor -- start 2023-03-03T10:22:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.612+0000 7f049fc49700 1 -- start start 2023-03-03T10:22:12.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049fc49700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049806e7e0 0x7f04981e4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:22:12.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049fc49700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04980c64f0 0x7f049815bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:22:12.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049fc49700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 0x7f04981eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:22:12.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049fc49700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04980aec20 con 0x7f049806e7e0 2023-03-03T10:22:12.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049fc49700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04981e5150 con 0x7f04980c64f0 2023-03-03T10:22:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049fc49700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04981e42f0 con 0x7f04981529a0 2023-03-03T10:22:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049d9e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049806e7e0 0x7f04981e4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:22:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049d9e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049806e7e0 0x7f04981e4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60094/0 (socket says 172.21.15.7:60094) 2023-03-03T10:22:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049d9e5700 1 -- 172.21.15.7:0/3586284717 learned_addr learned my addr 172.21.15.7:0/3586284717 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:22:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049d1e4700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04980c64f0 0x7f049815bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:22:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049e1e6700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 0x7f04981eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:22:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049d9e5700 1 -- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 msgr2=0x7f04981eeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:22:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049d9e5700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 0x7f04981eeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049d9e5700 1 -- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04980c64f0 msgr2=0x7f049815bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:22:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049d9e5700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04980c64f0 0x7f049815bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.613+0000 7f049d9e5700 1 -- 172.21.15.7:0/3586284717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0494009550 con 0x7f049806e7e0 2023-03-03T10:22:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.614+0000 7f049d9e5700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049806e7e0 0x7f04981e4c10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f049400e9b0 tx=0x7f049400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:22:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.614+0000 7f048e7fc700 1 -- 172.21.15.7:0/3586284717 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049400ee50 con 0x7f049806e7e0 2023-03-03T10:22:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.614+0000 7f048e7fc700 1 -- 172.21.15.7:0/3586284717 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049400c850 con 0x7f049806e7e0 2023-03-03T10:22:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.614+0000 7f049fc49700 1 -- 172.21.15.7:0/3586284717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0490009a70 con 0x7f049806e7e0 2023-03-03T10:22:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.614+0000 7f048e7fc700 1 -- 172.21.15.7:0/3586284717 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049400cbc0 con 0x7f049806e7e0 2023-03-03T10:22:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.614+0000 7f049fc49700 1 -- 172.21.15.7:0/3586284717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04980c79a0 con 0x7f049806e7e0 2023-03-03T10:22:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.615+0000 7f049fc49700 1 -- 172.21.15.7:0/3586284717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04980c76c0 con 0x7f049806e7e0 2023-03-03T10:22:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.615+0000 7f048e7fc700 1 -- 172.21.15.7:0/3586284717 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0494023070 con 0x7f049806e7e0 2023-03-03T10:22:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.616+0000 7f048e7fc700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f047c068320 0x7f047c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:22:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.616+0000 7f048e7fc700 1 -- 172.21.15.7:0/3586284717 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0494086340 con 0x7f049806e7e0 2023-03-03T10:22:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.616+0000 7f049d1e4700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f047c068320 0x7f047c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:22:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.617+0000 7f049d1e4700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f047c068320 0x7f047c06a7d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f04980cb140 tx=0x7f0488007db0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:22:12.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.620+0000 7f048e7fc700 1 -- 172.21.15.7:0/3586284717 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f04940526e0 con 0x7f049806e7e0 2023-03-03T10:22:12.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.857+0000 7f049fc49700 1 -- 172.21.15.7:0/3586284717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04980c6960 con 0x7f049806e7e0 2023-03-03T10:22:12.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.857+0000 7f048e7fc700 1 -- 172.21.15.7:0/3586284717 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0494055d00 con 0x7f049806e7e0 2023-03-03T10:22:12.860 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:22:12.860 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:22:12.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 -- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f047c068320 msgr2=0x7f047c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:22:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f047c068320 0x7f047c06a7d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f04980cb140 tx=0x7f0488007db0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 -- 172.21.15.7:0/3586284717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049806e7e0 msgr2=0x7f04981e4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:22:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049806e7e0 0x7f04981e4c10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f049400e9b0 tx=0x7f049400ecc0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 -- 172.21.15.7:0/3586284717 shutdown_connections 2023-03-03T10:22:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04981529a0 0x7f04981eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f047c068320 0x7f047c06a7d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f049806e7e0 0x7f04981e4c10 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 --2- 172.21.15.7:0/3586284717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04980c64f0 0x7f049815bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:22:12.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 -- 172.21.15.7:0/3586284717 >> 172.21.15.7:0/3586284717 conn(0x7f0498066e40 msgr2=0x7f04980c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:22:12.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 -- 172.21.15.7:0/3586284717 shutdown_connections 2023-03-03T10:22:12.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:22:12.859+0000 7f049fc49700 1 -- 172.21.15.7:0/3586284717 wait complete. 2023-03-03T10:22:12.863 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:22:12.873 INFO:tasks.mds_thrash.fs.[a]:waiting for 108 secs before thrashing 2023-03-03T10:22:29.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:29.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:59.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:22:59.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:21.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:23:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.168+0000 7f68fe704700 1 Processor -- start 2023-03-03T10:23:21.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.168+0000 7f68fe704700 1 -- start start 2023-03-03T10:23:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.168+0000 7f68fe704700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f8151ab0 0x7f68f80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.168+0000 7f68fe704700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68f806c5b0 0x7f68f8157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.168+0000 7f68fe704700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f806d780 con 0x7f68f8073c30 2023-03-03T10:23:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.168+0000 7f68fe704700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f806cb70 con 0x7f68f806c5b0 2023-03-03T10:23:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.168+0000 7f68fe704700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f806ce50 con 0x7f68f8151ab0 2023-03-03T10:23:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68fcca1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68f806c5b0 0x7f68f8157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68f77fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f68f8073c30 0x7f68f806bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54118/0 (socket says 172.21.15.7:54118) 2023-03-03T10:23:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68f77fe700 1 -- 172.21.15.7:0/3834667932 learned_addr learned my addr 172.21.15.7:0/3834667932 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:23:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68f7fff700 1 --2- 172.21.15.7:0/3834667932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f8151ab0 0x7f68f80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68fcca1700 1 -- 172.21.15.7:0/3834667932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f8151ab0 msgr2=0x7f68f80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:23:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68fcca1700 1 --2- 172.21.15.7:0/3834667932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f8151ab0 0x7f68f80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68fcca1700 1 -- 172.21.15.7:0/3834667932 >> v1:172.21.15.7:6789/0 conn(0x7f68f8073c30 legacy=0x7f68f806bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68f6ffd700 1 -- 172.21.15.7:0/3834667932 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273917337 0 0) 0x7f68f806d780 con 0x7f68f8073c30 2023-03-03T10:23:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68fcca1700 1 -- 172.21.15.7:0/3834667932 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f806d130 con 0x7f68f806c5b0 2023-03-03T10:23:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68fcca1700 1 --2- 172.21.15.7:0/3834667932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f806c5b0 0x7f68f8157dc0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f68f0009d90 tx=0x7f68f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbcef4c07114b31a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:23:21.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68f6ffd700 1 -- 172.21.15.7:0/3834667932 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f000e070 con 0x7f68f806c5b0 2023-03-03T10:23:21.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68f6ffd700 1 -- 172.21.15.7:0/3834667932 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f000bb20 con 0x7f68f806c5b0 2023-03-03T10:23:21.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.169+0000 7f68f6ffd700 1 -- 172.21.15.7:0/3834667932 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f0012430 con 0x7f68f806c5b0 2023-03-03T10:23:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.170+0000 7f68fe704700 1 -- 172.21.15.7:0/3834667932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f806c5b0 msgr2=0x7f68f8157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.170+0000 7f68fe704700 1 --2- 172.21.15.7:0/3834667932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f806c5b0 0x7f68f8157dc0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f68f0009d90 tx=0x7f68f000b330 comp rx=0 tx=0).stop 2023-03-03T10:23:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.170+0000 7f68fe704700 1 -- 172.21.15.7:0/3834667932 shutdown_connections 2023-03-03T10:23:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.170+0000 7f68fe704700 1 --2- 172.21.15.7:0/3834667932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f806c5b0 0x7f68f8157dc0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.170+0000 7f68fe704700 1 --2- 172.21.15.7:0/3834667932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f8151ab0 0x7f68f80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.170+0000 7f68fe704700 1 -- 172.21.15.7:0/3834667932 >> 172.21.15.7:0/3834667932 conn(0x7f68f80a94d0 msgr2=0x7f68f80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:23:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.170+0000 7f68fe704700 1 -- 172.21.15.7:0/3834667932 shutdown_connections 2023-03-03T10:23:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.170+0000 7f68fe704700 1 -- 172.21.15.7:0/3834667932 wait complete. 2023-03-03T10:23:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.171+0000 7f68fe704700 1 Processor -- start 2023-03-03T10:23:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.171+0000 7f68fe704700 1 -- start start 2023-03-03T10:23:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68fe704700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f806c5b0 0x7f68f81edbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68fe704700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f8073c30 0x7f68f81ea5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68fe704700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f8151ab0 0x7f68f814a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68fe704700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f81ee120 con 0x7f68f8073c30 2023-03-03T10:23:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68fe704700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f81ed750 con 0x7f68f806c5b0 2023-03-03T10:23:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68fe704700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f81eaaf0 con 0x7f68f8151ab0 2023-03-03T10:23:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68f77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f8073c30 0x7f68f81ea5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68f77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f8073c30 0x7f68f81ea5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60144/0 (socket says 172.21.15.7:60144) 2023-03-03T10:23:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68f77fe700 1 -- 172.21.15.7:0/2678772129 learned_addr learned my addr 172.21.15.7:0/2678772129 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:23:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68fcca1700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f8151ab0 0x7f68f814a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68f7fff700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f806c5b0 0x7f68f81edbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68f77fe700 1 -- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f8151ab0 msgr2=0x7f68f814a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68f77fe700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f8151ab0 0x7f68f814a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68f77fe700 1 -- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f806c5b0 msgr2=0x7f68f81edbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68f77fe700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f806c5b0 0x7f68f81edbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68f77fe700 1 -- 172.21.15.7:0/2678772129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f0009a70 con 0x7f68f8073c30 2023-03-03T10:23:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.172+0000 7f68f7fff700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f806c5b0 0x7f68f81edbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:23:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.173+0000 7f68f77fe700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f8073c30 0x7f68f81ea5b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0011e0 tx=0x7f68ec00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:23:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.173+0000 7f68f4ff9700 1 -- 172.21.15.7:0/2678772129 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec004c80 con 0x7f68f8073c30 2023-03-03T10:23:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.173+0000 7f68f4ff9700 1 -- 172.21.15.7:0/2678772129 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec004e00 con 0x7f68f8073c30 2023-03-03T10:23:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.173+0000 7f68fe704700 1 -- 172.21.15.7:0/2678772129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f8151e80 con 0x7f68f8073c30 2023-03-03T10:23:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.173+0000 7f68f4ff9700 1 -- 172.21.15.7:0/2678772129 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec007310 con 0x7f68f8073c30 2023-03-03T10:23:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.173+0000 7f68fe704700 1 -- 172.21.15.7:0/2678772129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f806d410 con 0x7f68f8073c30 2023-03-03T10:23:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.174+0000 7f68fe704700 1 -- 172.21.15.7:0/2678772129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f8152010 con 0x7f68f8073c30 2023-03-03T10:23:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.174+0000 7f68f4ff9700 1 -- 172.21.15.7:0/2678772129 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f68ec01a020 con 0x7f68f8073c30 2023-03-03T10:23:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.175+0000 7f68f4ff9700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e4068320 0x7f68e406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.175+0000 7f68f4ff9700 1 -- 172.21.15.7:0/2678772129 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f68ec085c40 con 0x7f68f8073c30 2023-03-03T10:23:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.175+0000 7f68f7fff700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e4068320 0x7f68e406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.175+0000 7f68f7fff700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e4068320 0x7f68e406a7d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f68e8009b90 tx=0x7f68e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:23:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.179+0000 7f68f4ff9700 1 -- 172.21.15.7:0/2678772129 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68ec051f80 con 0x7f68f8073c30 2023-03-03T10:23:21.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.417+0000 7f68fe704700 1 -- 172.21.15.7:0/2678772129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68f8074bb0 con 0x7f68f8073c30 2023-03-03T10:23:21.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.418+0000 7f68f4ff9700 1 -- 172.21.15.7:0/2678772129 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f68ec0555a0 con 0x7f68f8073c30 2023-03-03T10:23:21.420 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:23:21.421 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:23:21.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 -- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e4068320 msgr2=0x7f68e406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:21.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e4068320 0x7f68e406a7d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f68e8009b90 tx=0x7f68e8008040 comp rx=0 tx=0).stop 2023-03-03T10:23:21.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 -- 172.21.15.7:0/2678772129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f8073c30 msgr2=0x7f68f81ea5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:21.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f8073c30 0x7f68f81ea5b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0011e0 tx=0x7f68ec00ca30 comp rx=0 tx=0).stop 2023-03-03T10:23:21.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 -- 172.21.15.7:0/2678772129 shutdown_connections 2023-03-03T10:23:21.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f8151ab0 0x7f68f814a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:21.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f806c5b0 0x7f68f81edbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:21.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e4068320 0x7f68e406a7d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:21.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 --2- 172.21.15.7:0/2678772129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f8073c30 0x7f68f81ea5b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:21.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 -- 172.21.15.7:0/2678772129 >> 172.21.15.7:0/2678772129 conn(0x7f68f80a94d0 msgr2=0x7f68f8150dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:23:21.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 -- 172.21.15.7:0/2678772129 shutdown_connections 2023-03-03T10:23:21.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:21.420+0000 7f68fe704700 1 -- 172.21.15.7:0/2678772129 wait complete. 2023-03-03T10:23:21.424 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:23:21.434 INFO:tasks.mds_thrash.fs.[b]:waiting for 33 secs before thrashing 2023-03-03T10:23:29.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:29.523 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:54.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:23:54.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.572+0000 7ff0f03b0700 1 Processor -- start 2023-03-03T10:23:54.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.572+0000 7ff0f03b0700 1 -- start start 2023-03-03T10:23:54.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0f03b0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff0e814f9d0 0x7ff0e8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0f03b0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e80c2f30 con 0x7ff0e80abe20 2023-03-03T10:23:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0f03b0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e80c31d0 con 0x7ff0e814fda0 2023-03-03T10:23:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0f03b0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e81524a0 con 0x7ff0e814f9d0 2023-03-03T10:23:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ee94d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff0e80abe20 0x7ff0e81529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54152/0 (socket says 172.21.15.7:54152) 2023-03-03T10:23:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ee94d700 1 -- 172.21.15.7:0/2813474207 learned_addr learned my addr 172.21.15.7:0/2813474207 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:23:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ee14c700 1 --2- 172.21.15.7:0/2813474207 >> v2:172.21.15.143:3300/0 conn(0x7ff0e814f9d0 0x7ff0e8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456004789 0 0) 0x7ff0e80c2f30 con 0x7ff0e80abe20 2023-03-03T10:23:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d40035b0 con 0x7ff0e80abe20 2023-03-03T10:23:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 294407350 0 0) 0x7ff0d40035b0 con 0x7ff0e80abe20 2023-03-03T10:23:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 >> v2:172.21.15.143:3300/0 conn(0x7ff0e814f9d0 msgr2=0x7ff0e8156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:23:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ed14a700 1 --2- 172.21.15.7:0/2813474207 >> v2:172.21.15.143:3300/0 conn(0x7ff0e814f9d0 0x7ff0e8156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 >> v1:172.21.15.143:6789/0 conn(0x7ff0e814fda0 legacy=0x7ff0e8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e80c34b0 con 0x7ff0e80abe20 2023-03-03T10:23:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.573+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff0e00019f0 con 0x7ff0e80abe20 2023-03-03T10:23:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 >> v1:172.21.15.7:6789/0 conn(0x7ff0e80abe20 legacy=0x7ff0e81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed14a700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d4003a70 0x7ff0d4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed14a700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d4004850 0x7ff0d4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed14a700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d4005320 0x7ff0d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e81524a0 con 0x7ff0d4004850 2023-03-03T10:23:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d400c230 con 0x7ff0d4005320 2023-03-03T10:23:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d400c3f0 con 0x7ff0d4003a70 2023-03-03T10:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed94b700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d4004850 0x7ff0d4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ee94d700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d4005320 0x7ff0d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ee14c700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d4003a70 0x7ff0d4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed94b700 1 -- 172.21.15.7:0/2813474207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d4003a70 msgr2=0x7ff0d4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed94b700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d4003a70 0x7ff0d4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed94b700 1 -- 172.21.15.7:0/2813474207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d4005320 msgr2=0x7ff0d400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed94b700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d4005320 0x7ff0d400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ee94d700 1 -- 172.21.15.7:0/2813474207 reap_dead start 2023-03-03T10:23:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed94b700 1 -- 172.21.15.7:0/2813474207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d40035b0 con 0x7ff0d4004850 2023-03-03T10:23:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed94b700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d4004850 0x7ff0d4004cc0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff0e81501d0 tx=0x7ff0d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26285385aa661f87 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:23:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80052d0 con 0x7ff0d4004850 2023-03-03T10:23:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d8005450 con 0x7ff0d4004850 2023-03-03T10:23:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.574+0000 7ff0ed14a700 1 -- 172.21.15.7:0/2813474207 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80038c0 con 0x7ff0d4004850 2023-03-03T10:23:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.575+0000 7ff0f03b0700 1 -- 172.21.15.7:0/2813474207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d4004850 msgr2=0x7ff0d4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.575+0000 7ff0f03b0700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d4004850 0x7ff0d4004cc0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff0e81501d0 tx=0x7ff0d4005880 comp rx=0 tx=0).stop 2023-03-03T10:23:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.575+0000 7ff0f03b0700 1 -- 172.21.15.7:0/2813474207 shutdown_connections 2023-03-03T10:23:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.575+0000 7ff0f03b0700 1 --2- 172.21.15.7:0/2813474207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d4004850 0x7ff0d4004cc0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.575+0000 7ff0f03b0700 1 -- 172.21.15.7:0/2813474207 >> 172.21.15.7:0/2813474207 conn(0x7ff0e80c8900 msgr2=0x7ff0e80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:23:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.575+0000 7ff0f03b0700 1 -- 172.21.15.7:0/2813474207 shutdown_connections 2023-03-03T10:23:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.575+0000 7ff0f03b0700 1 -- 172.21.15.7:0/2813474207 wait complete. 2023-03-03T10:23:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.576+0000 7ff0f03b0700 1 Processor -- start 2023-03-03T10:23:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.576+0000 7ff0f03b0700 1 -- start start 2023-03-03T10:23:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0f03b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d4004850 0x7ff0e81549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0f03b0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e806a2b0 0x7ff0e806a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0f03b0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e814dfa0 0x7ff0e814e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0f03b0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e81de230 con 0x7ff0e814dfa0 2023-03-03T10:23:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0f03b0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e814f0d0 con 0x7ff0e806a2b0 2023-03-03T10:23:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0f03b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e806c200 con 0x7ff0d4004850 2023-03-03T10:23:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0ee94d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e814dfa0 0x7ff0e814e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0ee94d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e814dfa0 0x7ff0e814e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60184/0 (socket says 172.21.15.7:60184) 2023-03-03T10:23:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0ee94d700 1 -- 172.21.15.7:0/239653568 learned_addr learned my addr 172.21.15.7:0/239653568 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:23:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0ee14c700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d4004850 0x7ff0e81549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0ed94b700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e806a2b0 0x7ff0e806a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0ee94d700 1 -- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d4004850 msgr2=0x7ff0e81549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0ee94d700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d4004850 0x7ff0e81549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0ee94d700 1 -- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e806a2b0 msgr2=0x7ff0e806a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0ee94d700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e806a2b0 0x7ff0e806a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.577+0000 7ff0ee94d700 1 -- 172.21.15.7:0/239653568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e80ac350 con 0x7ff0e814dfa0 2023-03-03T10:23:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.578+0000 7ff0ee94d700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e814dfa0 0x7ff0e814e410 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80ac280 tx=0x7ff0e000a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:23:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.578+0000 7ff0deffd700 1 -- 172.21.15.7:0/239653568 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e000ab20 con 0x7ff0e814dfa0 2023-03-03T10:23:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.578+0000 7ff0deffd700 1 -- 172.21.15.7:0/239653568 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e000aca0 con 0x7ff0e814dfa0 2023-03-03T10:23:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.578+0000 7ff0f03b0700 1 -- 172.21.15.7:0/239653568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e80ab630 con 0x7ff0e814dfa0 2023-03-03T10:23:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.578+0000 7ff0deffd700 1 -- 172.21.15.7:0/239653568 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0013630 con 0x7ff0e814dfa0 2023-03-03T10:23:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.578+0000 7ff0f03b0700 1 -- 172.21.15.7:0/239653568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e80c3e10 con 0x7ff0e814dfa0 2023-03-03T10:23:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.579+0000 7ff0f03b0700 1 -- 172.21.15.7:0/239653568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e806ac60 con 0x7ff0e814dfa0 2023-03-03T10:23:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.579+0000 7ff0deffd700 1 -- 172.21.15.7:0/239653568 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff0e0013790 con 0x7ff0e814dfa0 2023-03-03T10:23:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.580+0000 7ff0deffd700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 0x7ff0cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:23:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.580+0000 7ff0deffd700 1 -- 172.21.15.7:0/239653568 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff0e00869d0 con 0x7ff0e814dfa0 2023-03-03T10:23:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.580+0000 7ff0ee14c700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 0x7ff0cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:23:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.580+0000 7ff0ee14c700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 0x7ff0cc06a7a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80c9590 tx=0x7ff0e4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:23:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.584+0000 7ff0deffd700 1 -- 172.21.15.7:0/239653568 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff0e0052d70 con 0x7ff0e814dfa0 2023-03-03T10:23:54.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.821+0000 7ff0f03b0700 1 -- 172.21.15.7:0/239653568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0e804e6e0 con 0x7ff0e814dfa0 2023-03-03T10:23:54.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.822+0000 7ff0deffd700 1 -- 172.21.15.7:0/239653568 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff0e0056390 con 0x7ff0e814dfa0 2023-03-03T10:23:54.824 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:23:54.825 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:23:54.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 -- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 msgr2=0x7ff0cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:54.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 0x7ff0cc06a7a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80c9590 tx=0x7ff0e4009210 comp rx=0 tx=0).stop 2023-03-03T10:23:54.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 -- 172.21.15.7:0/239653568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e814dfa0 msgr2=0x7ff0e814e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:23:54.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e814dfa0 0x7ff0e814e410 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80ac280 tx=0x7ff0e000a5f0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 -- 172.21.15.7:0/239653568 shutdown_connections 2023-03-03T10:23:54.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 0x7ff0cc06a7a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e814dfa0 0x7ff0e814e410 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d4004850 0x7ff0e81549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 --2- 172.21.15.7:0/239653568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e806a2b0 0x7ff0e806a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:23:54.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 -- 172.21.15.7:0/239653568 >> 172.21.15.7:0/239653568 conn(0x7ff0e80c8900 msgr2=0x7ff0e81519c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:23:54.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 -- 172.21.15.7:0/239653568 shutdown_connections 2023-03-03T10:23:54.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:23:54.824+0000 7ff0f03b0700 1 -- 172.21.15.7:0/239653568 wait complete. 2023-03-03T10:23:54.828 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:23:54.838 INFO:tasks.mds_thrash.fs.[b]:waiting for 32 secs before thrashing 2023-03-03T10:23:59.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:23:59.593 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:00.874 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:24:01.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.009+0000 7f5a2a129700 1 Processor -- start 2023-03-03T10:24:01.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.009+0000 7f5a2a129700 1 -- start start 2023-03-03T10:24:01.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.009+0000 7f5a2a129700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a24155390 0x7f5a240c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:01.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.009+0000 7f5a2a129700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a2406e020 0x7f5a24071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a2a129700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a24154910 con 0x7f5a24072520 2023-03-03T10:24:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a2a129700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a24070b90 con 0x7f5a24155390 2023-03-03T10:24:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a2a129700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a24151320 con 0x7f5a2406e020 2023-03-03T10:24:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a237fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a24155390 0x7f5a240c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a23fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5a24072520 0x7f5a24151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54176/0 (socket says 172.21.15.7:54176) 2023-03-03T10:24:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a23fff700 1 -- 172.21.15.7:0/3318401113 learned_addr learned my addr 172.21.15.7:0/3318401113 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:24:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a22ffd700 1 --2- 172.21.15.7:0/3318401113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a2406e020 0x7f5a24071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a237fe700 1 -- 172.21.15.7:0/3318401113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a2406e020 msgr2=0x7f5a24071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:24:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a237fe700 1 --2- 172.21.15.7:0/3318401113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a2406e020 0x7f5a24071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a237fe700 1 -- 172.21.15.7:0/3318401113 >> v1:172.21.15.7:6789/0 conn(0x7f5a24072520 legacy=0x7f5a24151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a237fe700 1 -- 172.21.15.7:0/3318401113 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240702a0 con 0x7f5a24155390 2023-03-03T10:24:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a237fe700 1 --2- 172.21.15.7:0/3318401113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a24155390 0x7f5a240c3e30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009840 tx=0x7f5a14009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a75b8d5a3aedda server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:01.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.010+0000 7f5a227fc700 1 -- 172.21.15.7:0/3318401113 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a14011830 con 0x7f5a24155390 2023-03-03T10:24:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.011+0000 7f5a227fc700 1 -- 172.21.15.7:0/3318401113 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a14011990 con 0x7f5a24155390 2023-03-03T10:24:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.011+0000 7f5a227fc700 1 -- 172.21.15.7:0/3318401113 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a14011d20 con 0x7f5a24155390 2023-03-03T10:24:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.011+0000 7f5a2a129700 1 -- 172.21.15.7:0/3318401113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a24155390 msgr2=0x7f5a240c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.011+0000 7f5a2a129700 1 --2- 172.21.15.7:0/3318401113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a24155390 0x7f5a240c3e30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009840 tx=0x7f5a14009b50 comp rx=0 tx=0).stop 2023-03-03T10:24:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.011+0000 7f5a2a129700 1 -- 172.21.15.7:0/3318401113 shutdown_connections 2023-03-03T10:24:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.011+0000 7f5a2a129700 1 --2- 172.21.15.7:0/3318401113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a24155390 0x7f5a240c3e30 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.011+0000 7f5a2a129700 1 --2- 172.21.15.7:0/3318401113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a2406e020 0x7f5a24071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.011+0000 7f5a2a129700 1 -- 172.21.15.7:0/3318401113 >> 172.21.15.7:0/3318401113 conn(0x7f5a2406c110 msgr2=0x7f5a2406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:24:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.011+0000 7f5a2a129700 1 -- 172.21.15.7:0/3318401113 shutdown_connections 2023-03-03T10:24:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.011+0000 7f5a2a129700 1 -- 172.21.15.7:0/3318401113 wait complete. 2023-03-03T10:24:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.012+0000 7f5a2a129700 1 Processor -- start 2023-03-03T10:24:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.013+0000 7f5a2a129700 1 -- start start 2023-03-03T10:24:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.013+0000 7f5a2a129700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a2406e020 0x7f5a2414e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.013+0000 7f5a2a129700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a24072520 0x7f5a241dde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:01.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.013+0000 7f5a2a129700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a24155390 0x7f5a241495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:01.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.013+0000 7f5a2a129700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2414cad0 con 0x7f5a2406e020 2023-03-03T10:24:01.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.013+0000 7f5a2a129700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2414dce0 con 0x7f5a24072520 2023-03-03T10:24:01.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.013+0000 7f5a2a129700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2414c690 con 0x7f5a24155390 2023-03-03T10:24:01.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.013+0000 7f5a237fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a2406e020 0x7f5a2414e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:01.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.013+0000 7f5a237fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a2406e020 0x7f5a2414e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60198/0 (socket says 172.21.15.7:60198) 2023-03-03T10:24:01.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.013+0000 7f5a237fe700 1 -- 172.21.15.7:0/3922683902 learned_addr learned my addr 172.21.15.7:0/3922683902 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:24:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a22ffd700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a24072520 0x7f5a241dde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a23fff700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a24155390 0x7f5a241495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a237fe700 1 -- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a24155390 msgr2=0x7f5a241495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a237fe700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a24155390 0x7f5a241495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a237fe700 1 -- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a24072520 msgr2=0x7f5a241dde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a237fe700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a24072520 0x7f5a241dde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a237fe700 1 -- 172.21.15.7:0/3922683902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a140094a0 con 0x7f5a2406e020 2023-03-03T10:24:01.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a22ffd700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a24072520 0x7f5a241dde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:24:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a237fe700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a2406e020 0x7f5a2414e4a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009fd0 tx=0x7f5a14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a0bfff700 1 -- 172.21.15.7:0/3922683902 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a140036a0 con 0x7f5a2406e020 2023-03-03T10:24:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a0bfff700 1 -- 172.21.15.7:0/3922683902 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1401f070 con 0x7f5a2406e020 2023-03-03T10:24:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a2a129700 1 -- 172.21.15.7:0/3922683902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a24070820 con 0x7f5a2406e020 2023-03-03T10:24:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a0bfff700 1 -- 172.21.15.7:0/3922683902 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a14003a60 con 0x7f5a2406e020 2023-03-03T10:24:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.014+0000 7f5a2a129700 1 -- 172.21.15.7:0/3922683902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a24070540 con 0x7f5a2406e020 2023-03-03T10:24:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.015+0000 7f5a2a129700 1 -- 172.21.15.7:0/3922683902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a24155760 con 0x7f5a2406e020 2023-03-03T10:24:01.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.015+0000 7f5a0bfff700 1 -- 172.21.15.7:0/3922683902 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5a1400f050 con 0x7f5a2406e020 2023-03-03T10:24:01.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.016+0000 7f5a0bfff700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a100682f0 0x7f5a1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:01.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.016+0000 7f5a0bfff700 1 -- 172.21.15.7:0/3922683902 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5a14021070 con 0x7f5a2406e020 2023-03-03T10:24:01.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.016+0000 7f5a22ffd700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a100682f0 0x7f5a1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:01.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.017+0000 7f5a22ffd700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a100682f0 0x7f5a1006a7a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5a240c6320 tx=0x7f5a18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:01.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.021+0000 7f5a0bfff700 1 -- 172.21.15.7:0/3922683902 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a14053d20 con 0x7f5a2406e020 2023-03-03T10:24:01.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.257+0000 7f5a2a129700 1 -- 172.21.15.7:0/3922683902 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a24074bb0 con 0x7f5a2406e020 2023-03-03T10:24:01.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.258+0000 7f5a0bfff700 1 -- 172.21.15.7:0/3922683902 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5a14057340 con 0x7f5a2406e020 2023-03-03T10:24:01.262 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:24:01.262 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:24:01.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 -- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a100682f0 msgr2=0x7f5a1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:01.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a100682f0 0x7f5a1006a7a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5a240c6320 tx=0x7f5a18008040 comp rx=0 tx=0).stop 2023-03-03T10:24:01.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 -- 172.21.15.7:0/3922683902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a2406e020 msgr2=0x7f5a2414e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:01.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a2406e020 0x7f5a2414e4a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009fd0 tx=0x7f5a14004750 comp rx=0 tx=0).stop 2023-03-03T10:24:01.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 -- 172.21.15.7:0/3922683902 shutdown_connections 2023-03-03T10:24:01.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a24155390 0x7f5a241495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:01.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a100682f0 0x7f5a1006a7a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:01.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a2406e020 0x7f5a2414e4a0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:01.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 --2- 172.21.15.7:0/3922683902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a24072520 0x7f5a241dde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:01.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 -- 172.21.15.7:0/3922683902 >> 172.21.15.7:0/3922683902 conn(0x7f5a2406c110 msgr2=0x7f5a2414f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:24:01.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 -- 172.21.15.7:0/3922683902 shutdown_connections 2023-03-03T10:24:01.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:01.260+0000 7f5a2a129700 1 -- 172.21.15.7:0/3922683902 wait complete. 2023-03-03T10:24:01.265 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:24:01.275 INFO:tasks.mds_thrash.fs.[a]:waiting for 116 secs before thrashing 2023-03-03T10:24:26.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:24:26.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.975+0000 7fe136a9d700 1 Processor -- start 2023-03-03T10:24:26.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.975+0000 7fe136a9d700 1 -- start start 2023-03-03T10:24:26.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.975+0000 7fe136a9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe13006d870 0x7fe1301550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:26.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe136a9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1300c1da0 0x7fe1300c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe136a9d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1300c3620 con 0x7fe1300c3e30 2023-03-03T10:24:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe136a9d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1300c2af0 con 0x7fe1300c1da0 2023-03-03T10:24:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe136a9d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1301545e0 con 0x7fe13006d870 2023-03-03T10:24:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1300c1da0 0x7fe1300c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1300c1da0 0x7fe1300c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60222/0 (socket says 172.21.15.7:60222) 2023-03-03T10:24:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12ffff700 1 -- 172.21.15.7:0/392009827 learned_addr learned my addr 172.21.15.7:0/392009827 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:24:26.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12ffff700 1 -- 172.21.15.7:0/392009827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe13006d870 msgr2=0x7fe1301550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:24:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe134839700 1 --2- 172.21.15.7:0/392009827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe13006d870 0x7fe1301550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12ffff700 1 --2- 172.21.15.7:0/392009827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe13006d870 0x7fe1301550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12ffff700 1 -- 172.21.15.7:0/392009827 >> v1:172.21.15.143:6789/0 conn(0x7fe1300c3e30 legacy=0x7fe130157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:24:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12ffff700 1 -- 172.21.15.7:0/392009827 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1301548c0 con 0x7fe1300c1da0 2023-03-03T10:24:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12ffff700 1 --2- 172.21.15.7:0/392009827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 0x7fe1300c7350 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0098f0 tx=0x7fe11c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe9945a00730a4d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12f7fe700 1 -- 172.21.15.7:0/392009827 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c004230 con 0x7fe1300c1da0 2023-03-03T10:24:26.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12f7fe700 1 -- 172.21.15.7:0/392009827 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11c004390 con 0x7fe1300c1da0 2023-03-03T10:24:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.976+0000 7fe12f7fe700 1 -- 172.21.15.7:0/392009827 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0036a0 con 0x7fe1300c1da0 2023-03-03T10:24:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.977+0000 7fe136a9d700 1 -- 172.21.15.7:0/392009827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 msgr2=0x7fe1300c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.977+0000 7fe136a9d700 1 --2- 172.21.15.7:0/392009827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 0x7fe1300c7350 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0098f0 tx=0x7fe11c009c00 comp rx=0 tx=0).stop 2023-03-03T10:24:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.977+0000 7fe136a9d700 1 -- 172.21.15.7:0/392009827 shutdown_connections 2023-03-03T10:24:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.977+0000 7fe136a9d700 1 --2- 172.21.15.7:0/392009827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe13006d870 0x7fe1301550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.977+0000 7fe136a9d700 1 --2- 172.21.15.7:0/392009827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 0x7fe1300c7350 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.977+0000 7fe136a9d700 1 -- 172.21.15.7:0/392009827 >> 172.21.15.7:0/392009827 conn(0x7fe13006c110 msgr2=0x7fe13006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:24:26.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.977+0000 7fe136a9d700 1 -- 172.21.15.7:0/392009827 shutdown_connections 2023-03-03T10:24:26.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.977+0000 7fe136a9d700 1 -- 172.21.15.7:0/392009827 wait complete. 2023-03-03T10:24:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.978+0000 7fe136a9d700 1 Processor -- start 2023-03-03T10:24:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.978+0000 7fe136a9d700 1 -- start start 2023-03-03T10:24:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe136a9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe13006d870 0x7fe1301ea030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe136a9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 0x7fe130153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe136a9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1300c3e30 0x7fe130146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe136a9d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1301ea570 con 0x7fe1300c1da0 2023-03-03T10:24:26.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe136a9d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe130156070 con 0x7fe13006d870 2023-03-03T10:24:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe136a9d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe130157400 con 0x7fe1300c3e30 2023-03-03T10:24:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe12ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 0x7fe130153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe12ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 0x7fe130153da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60230/0 (socket says 172.21.15.7:60230) 2023-03-03T10:24:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe12ffff700 1 -- 172.21.15.7:0/1789941241 learned_addr learned my addr 172.21.15.7:0/1789941241 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:24:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe134839700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe13006d870 0x7fe1301ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.979+0000 7fe13503a700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1300c3e30 0x7fe130146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:26.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe12ffff700 1 -- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1300c3e30 msgr2=0x7fe130146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe12ffff700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1300c3e30 0x7fe130146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe12ffff700 1 -- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe13006d870 msgr2=0x7fe1301ea030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe12ffff700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe13006d870 0x7fe1301ea030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe12ffff700 1 -- 172.21.15.7:0/1789941241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c009550 con 0x7fe1300c1da0 2023-03-03T10:24:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe12ffff700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 0x7fe130153da0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009130 tx=0x7fe11c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe12d7fa700 1 -- 172.21.15.7:0/1789941241 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c003cd0 con 0x7fe1300c1da0 2023-03-03T10:24:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe12d7fa700 1 -- 172.21.15.7:0/1789941241 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11c01f070 con 0x7fe1300c1da0 2023-03-03T10:24:26.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe136a9d700 1 -- 172.21.15.7:0/1789941241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe13006dc40 con 0x7fe1300c1da0 2023-03-03T10:24:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe12d7fa700 1 -- 172.21.15.7:0/1789941241 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c00b8a0 con 0x7fe1300c1da0 2023-03-03T10:24:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.980+0000 7fe136a9d700 1 -- 172.21.15.7:0/1789941241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe130154ba0 con 0x7fe1300c1da0 2023-03-03T10:24:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.981+0000 7fe136a9d700 1 -- 172.21.15.7:0/1789941241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1300c38e0 con 0x7fe1300c1da0 2023-03-03T10:24:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.981+0000 7fe12d7fa700 1 -- 172.21.15.7:0/1789941241 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe11c02a020 con 0x7fe1300c1da0 2023-03-03T10:24:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.982+0000 7fe12d7fa700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1140682f0 0x7fe11406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:24:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.982+0000 7fe12d7fa700 1 -- 172.21.15.7:0/1789941241 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe11c0878a0 con 0x7fe1300c1da0 2023-03-03T10:24:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.982+0000 7fe134839700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1140682f0 0x7fe11406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:24:26.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.983+0000 7fe134839700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1140682f0 0x7fe11406a7a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe13014e820 tx=0x7fe124008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:24:26.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:26.986+0000 7fe12d7fa700 1 -- 172.21.15.7:0/1789941241 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe11c053c40 con 0x7fe1300c1da0 2023-03-03T10:24:27.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.224+0000 7fe136a9d700 1 -- 172.21.15.7:0/1789941241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1300c38e0 con 0x7fe1300c1da0 2023-03-03T10:24:27.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.225+0000 7fe12d7fa700 1 -- 172.21.15.7:0/1789941241 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe11c010380 con 0x7fe1300c1da0 2023-03-03T10:24:27.227 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:24:27.228 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:24:27.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 -- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1140682f0 msgr2=0x7fe11406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1140682f0 0x7fe11406a7a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe13014e820 tx=0x7fe124008040 comp rx=0 tx=0).stop 2023-03-03T10:24:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 -- 172.21.15.7:0/1789941241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 msgr2=0x7fe130153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:24:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 0x7fe130153da0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009130 tx=0x7fe11c004750 comp rx=0 tx=0).stop 2023-03-03T10:24:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 -- 172.21.15.7:0/1789941241 shutdown_connections 2023-03-03T10:24:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1300c3e30 0x7fe130146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe13006d870 0x7fe1301ea030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1140682f0 0x7fe11406a7a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:27.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 --2- 172.21.15.7:0/1789941241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1300c1da0 0x7fe130153da0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:24:27.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 -- 172.21.15.7:0/1789941241 >> 172.21.15.7:0/1789941241 conn(0x7fe13006c110 msgr2=0x7fe13014ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:24:27.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 -- 172.21.15.7:0/1789941241 shutdown_connections 2023-03-03T10:24:27.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:24:27.227+0000 7fe136a9d700 1 -- 172.21.15.7:0/1789941241 wait complete. 2023-03-03T10:24:27.231 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:24:27.241 INFO:tasks.mds_thrash.fs.[b]:waiting for 52 secs before thrashing 2023-03-03T10:24:29.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:29.673 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:59.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:24:59.756 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:19.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:25:19.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.378+0000 7fb97750f700 1 Processor -- start 2023-03-03T10:25:19.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.379+0000 7fb97750f700 1 -- start start 2023-03-03T10:25:19.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.379+0000 7fb97750f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb970071e80 0x7fb970151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.379+0000 7fb97750f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb970072250 0x7fb97014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.379+0000 7fb97750f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9701510d0 con 0x7fb9700c1ce0 2023-03-03T10:25:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.379+0000 7fb97750f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9700c3740 con 0x7fb970072250 2023-03-03T10:25:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.379+0000 7fb97750f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb970070670 con 0x7fb970071e80 2023-03-03T10:25:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb974aaa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb970072250 0x7fb97014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb974aaa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb970072250 0x7fb97014e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60270/0 (socket says 172.21.15.7:60270) 2023-03-03T10:25:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb974aaa700 1 -- 172.21.15.7:0/4038008867 learned_addr learned my addr 172.21.15.7:0/4038008867 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:25:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb9752ab700 1 --2- 172.21.15.7:0/4038008867 >> v2:172.21.15.143:3300/0 conn(0x7fb970071e80 0x7fb970151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb974aaa700 1 -- 172.21.15.7:0/4038008867 >> v2:172.21.15.143:3300/0 conn(0x7fb970071e80 msgr2=0x7fb970151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:25:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb974aaa700 1 --2- 172.21.15.7:0/4038008867 >> v2:172.21.15.143:3300/0 conn(0x7fb970071e80 0x7fb970151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb974aaa700 1 -- 172.21.15.7:0/4038008867 >> v1:172.21.15.143:6789/0 conn(0x7fb9700c1ce0 legacy=0x7fb9701546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:25:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb974aaa700 1 -- 172.21.15.7:0/4038008867 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9700c39e0 con 0x7fb970072250 2023-03-03T10:25:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb974aaa700 1 --2- 172.21.15.7:0/4038008867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb970072250 0x7fb97014e1f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb96400bb90 tx=0x7fb96400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8865edac961850d1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:25:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb963fff700 1 -- 172.21.15.7:0/4038008867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96400dd10 con 0x7fb970072250 2023-03-03T10:25:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb963fff700 1 -- 172.21.15.7:0/4038008867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96400de70 con 0x7fb970072250 2023-03-03T10:25:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.380+0000 7fb963fff700 1 -- 172.21.15.7:0/4038008867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964004230 con 0x7fb970072250 2023-03-03T10:25:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.381+0000 7fb97750f700 1 -- 172.21.15.7:0/4038008867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb970072250 msgr2=0x7fb97014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.381+0000 7fb97750f700 1 --2- 172.21.15.7:0/4038008867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb970072250 0x7fb97014e1f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb96400bb90 tx=0x7fb96400bea0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.381+0000 7fb97750f700 1 -- 172.21.15.7:0/4038008867 shutdown_connections 2023-03-03T10:25:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.381+0000 7fb97750f700 1 --2- 172.21.15.7:0/4038008867 >> v2:172.21.15.143:3300/0 conn(0x7fb970071e80 0x7fb970151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.381+0000 7fb97750f700 1 --2- 172.21.15.7:0/4038008867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb970072250 0x7fb97014e1f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.381+0000 7fb97750f700 1 -- 172.21.15.7:0/4038008867 >> 172.21.15.7:0/4038008867 conn(0x7fb9700a94d0 msgr2=0x7fb9700a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:25:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.381+0000 7fb97750f700 1 -- 172.21.15.7:0/4038008867 shutdown_connections 2023-03-03T10:25:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.381+0000 7fb97750f700 1 -- 172.21.15.7:0/4038008867 wait complete. 2023-03-03T10:25:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.382+0000 7fb97750f700 1 Processor -- start 2023-03-03T10:25:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.382+0000 7fb97750f700 1 -- start start 2023-03-03T10:25:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb97750f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb970071e80 0x7fb9701e64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb97750f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb970072250 0x7fb9700aa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb97750f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9700c1ce0 0x7fb9700aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb97750f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97014dce0 con 0x7fb9700c1ce0 2023-03-03T10:25:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb97750f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9701e6a20 con 0x7fb970071e80 2023-03-03T10:25:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb97750f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9700ad080 con 0x7fb970072250 2023-03-03T10:25:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb975aac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9700c1ce0 0x7fb9700aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb975aac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9700c1ce0 0x7fb9700aaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60280/0 (socket says 172.21.15.7:60280) 2023-03-03T10:25:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb975aac700 1 -- 172.21.15.7:0/3622158134 learned_addr learned my addr 172.21.15.7:0/3622158134 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:25:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb9752ab700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb970071e80 0x7fb9701e64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb974aaa700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb970072250 0x7fb9700aa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb975aac700 1 -- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb970072250 msgr2=0x7fb9700aa9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb975aac700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb970072250 0x7fb9700aa9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb975aac700 1 -- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb970071e80 msgr2=0x7fb9701e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb975aac700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb970071e80 0x7fb9701e64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb975aac700 1 -- 172.21.15.7:0/3622158134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96400b7f0 con 0x7fb9700c1ce0 2023-03-03T10:25:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.383+0000 7fb975aac700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9700c1ce0 0x7fb9700aaee0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb96800b9e0 tx=0x7fb96800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:25:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.384+0000 7fb961ffb700 1 -- 172.21.15.7:0/3622158134 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96800d8e0 con 0x7fb9700c1ce0 2023-03-03T10:25:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.384+0000 7fb961ffb700 1 -- 172.21.15.7:0/3622158134 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96800da40 con 0x7fb9700c1ce0 2023-03-03T10:25:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.384+0000 7fb97750f700 1 -- 172.21.15.7:0/3622158134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97006c9a0 con 0x7fb9700c1ce0 2023-03-03T10:25:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.384+0000 7fb961ffb700 1 -- 172.21.15.7:0/3622158134 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96800ddf0 con 0x7fb9700c1ce0 2023-03-03T10:25:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.384+0000 7fb97750f700 1 -- 172.21.15.7:0/3622158134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb970151300 con 0x7fb9700c1ce0 2023-03-03T10:25:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.385+0000 7fb97750f700 1 -- 172.21.15.7:0/3622158134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97006cb30 con 0x7fb9700c1ce0 2023-03-03T10:25:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.385+0000 7fb961ffb700 1 -- 172.21.15.7:0/3622158134 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb968015420 con 0x7fb9700c1ce0 2023-03-03T10:25:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.386+0000 7fb961ffb700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb954068320 0x7fb95406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.386+0000 7fb961ffb700 1 -- 172.21.15.7:0/3622158134 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb968011070 con 0x7fb9700c1ce0 2023-03-03T10:25:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.386+0000 7fb9752ab700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb954068320 0x7fb95406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.387+0000 7fb9752ab700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb954068320 0x7fb95406a7d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00ab60 tx=0x7fb96c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:25:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.390+0000 7fb961ffb700 1 -- 172.21.15.7:0/3622158134 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb96804d5e0 con 0x7fb9700c1ce0 2023-03-03T10:25:19.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.629+0000 7fb97750f700 1 -- 172.21.15.7:0/3622158134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9700c58e0 con 0x7fb9700c1ce0 2023-03-03T10:25:19.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.630+0000 7fb961ffb700 1 -- 172.21.15.7:0/3622158134 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb968052590 con 0x7fb9700c1ce0 2023-03-03T10:25:19.632 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:25:19.632 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:25:19.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 -- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb954068320 msgr2=0x7fb95406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:19.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb954068320 0x7fb95406a7d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00ab60 tx=0x7fb96c008040 comp rx=0 tx=0).stop 2023-03-03T10:25:19.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 -- 172.21.15.7:0/3622158134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9700c1ce0 msgr2=0x7fb9700aaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9700c1ce0 0x7fb9700aaee0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb96800b9e0 tx=0x7fb96800bcf0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 -- 172.21.15.7:0/3622158134 shutdown_connections 2023-03-03T10:25:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb954068320 0x7fb95406a7d0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9700c1ce0 0x7fb9700aaee0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb970071e80 0x7fb9701e64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 --2- 172.21.15.7:0/3622158134 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb970072250 0x7fb9700aa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 -- 172.21.15.7:0/3622158134 >> 172.21.15.7:0/3622158134 conn(0x7fb9700a94d0 msgr2=0x7fb97014dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:25:19.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 -- 172.21.15.7:0/3622158134 shutdown_connections 2023-03-03T10:25:19.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:19.632+0000 7fb97750f700 1 -- 172.21.15.7:0/3622158134 wait complete. 2023-03-03T10:25:19.636 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:25:19.646 INFO:tasks.mds_thrash.fs.[b]:waiting for 13 secs before thrashing 2023-03-03T10:25:30.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:30.092 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:25:32.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:25:32.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.783+0000 7f2847de0700 1 Processor -- start 2023-03-03T10:25:32.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.783+0000 7f2847de0700 1 -- start start 2023-03-03T10:25:32.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.784+0000 7f2847de0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2840151e80 0x7f28400c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:32.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.784+0000 7f2847de0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2840155410 0x7f28400bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:32.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.784+0000 7f2847de0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28400c08d0 con 0x7f28400c6fe0 2023-03-03T10:25:32.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.784+0000 7f2847de0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2840152250 con 0x7f2840151e80 2023-03-03T10:25:32.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.784+0000 7f2847de0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28400c0fb0 con 0x7f2840155410 2023-03-03T10:25:32.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f284537b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2840155410 0x7f28400bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2845b7c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2840151e80 0x7f28400c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f284537b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2840155410 0x7f28400bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:40138/0 (socket says 172.21.15.7:40138) 2023-03-03T10:25:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2845b7c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2840151e80 0x7f28400c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:56070/0 (socket says 172.21.15.7:56070) 2023-03-03T10:25:32.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f284537b700 1 -- 172.21.15.7:0/3419027616 learned_addr learned my addr 172.21.15.7:0/3419027616 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:25:32.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2845b7c700 1 -- 172.21.15.7:0/3419027616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2840155410 msgr2=0x7f28400bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:25:32.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2845b7c700 1 --2- 172.21.15.7:0/3419027616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2840155410 0x7f28400bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:32.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2845b7c700 1 -- 172.21.15.7:0/3419027616 >> v1:172.21.15.143:6789/0 conn(0x7f28400c6fe0 legacy=0x7f28400bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2845b7c700 1 -- 172.21.15.7:0/3419027616 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2840154930 con 0x7f2840151e80 2023-03-03T10:25:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2844b7a700 1 -- 172.21.15.7:0/3419027616 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503653763 0 0) 0x7f28400c08d0 con 0x7f28400c6fe0 2023-03-03T10:25:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f284537b700 1 --2- 172.21.15.7:0/3419027616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2840155410 0x7f28400bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:25:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2845b7c700 1 --2- 172.21.15.7:0/3419027616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2840151e80 0x7f28400c6aa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f283c009870 tx=0x7f283c009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d57088d8c8920527 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:25:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2844b7a700 1 -- 172.21.15.7:0/3419027616 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c011830 con 0x7f2840151e80 2023-03-03T10:25:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2844b7a700 1 -- 172.21.15.7:0/3419027616 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283c011990 con 0x7f2840151e80 2023-03-03T10:25:32.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.785+0000 7f2844b7a700 1 -- 172.21.15.7:0/3419027616 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c011d20 con 0x7f2840151e80 2023-03-03T10:25:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.786+0000 7f2847de0700 1 -- 172.21.15.7:0/3419027616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2840151e80 msgr2=0x7f28400c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.786+0000 7f2847de0700 1 --2- 172.21.15.7:0/3419027616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2840151e80 0x7f28400c6aa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f283c009870 tx=0x7f283c009b80 comp rx=0 tx=0).stop 2023-03-03T10:25:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.786+0000 7f2847de0700 1 -- 172.21.15.7:0/3419027616 shutdown_connections 2023-03-03T10:25:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.786+0000 7f2847de0700 1 --2- 172.21.15.7:0/3419027616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2840151e80 0x7f28400c6aa0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.786+0000 7f2847de0700 1 --2- 172.21.15.7:0/3419027616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2840155410 0x7f28400bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.786+0000 7f2847de0700 1 -- 172.21.15.7:0/3419027616 >> 172.21.15.7:0/3419027616 conn(0x7f2840071630 msgr2=0x7f284006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:25:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.786+0000 7f2847de0700 1 -- 172.21.15.7:0/3419027616 shutdown_connections 2023-03-03T10:25:32.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.786+0000 7f2847de0700 1 -- 172.21.15.7:0/3419027616 wait complete. 2023-03-03T10:25:32.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.788+0000 7f2847de0700 1 Processor -- start 2023-03-03T10:25:32.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.788+0000 7f2847de0700 1 -- start start 2023-03-03T10:25:32.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f2847de0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28400c6fe0 0x7f28401e1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:32.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f2847de0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2840151e80 0x7f284014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:32.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f2847de0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2840155410 0x7f284014c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:32.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f2847de0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2840156390 con 0x7f2840151e80 2023-03-03T10:25:32.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f2847de0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f284014f6d0 con 0x7f2840155410 2023-03-03T10:25:32.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f2847de0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28401573f0 con 0x7f28400c6fe0 2023-03-03T10:25:32.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f284537b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2840151e80 0x7f284014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:32.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f284537b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2840151e80 0x7f284014e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60304/0 (socket says 172.21.15.7:60304) 2023-03-03T10:25:32.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f284537b700 1 -- 172.21.15.7:0/2297244429 learned_addr learned my addr 172.21.15.7:0/2297244429 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:25:32.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f2845b7c700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28400c6fe0 0x7f28401e1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:32.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f284637d700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2840155410 0x7f284014c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:32.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f284537b700 1 -- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28400c6fe0 msgr2=0x7f28401e1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:32.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f284537b700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28400c6fe0 0x7f28401e1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:32.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f284537b700 1 -- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2840155410 msgr2=0x7f284014c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:32.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f284537b700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2840155410 0x7f284014c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:32.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f284537b700 1 -- 172.21.15.7:0/2297244429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2830009550 con 0x7f2840151e80 2023-03-03T10:25:32.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f284537b700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2840151e80 0x7f284014e140 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f283000ec10 tx=0x7f2830009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:25:32.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.789+0000 7f28367fc700 1 -- 172.21.15.7:0/2297244429 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283000c8e0 con 0x7f2840151e80 2023-03-03T10:25:32.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.790+0000 7f28367fc700 1 -- 172.21.15.7:0/2297244429 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283000ca40 con 0x7f2840151e80 2023-03-03T10:25:32.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.790+0000 7f28367fc700 1 -- 172.21.15.7:0/2297244429 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283000cdb0 con 0x7f2840151e80 2023-03-03T10:25:32.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.790+0000 7f2847de0700 1 -- 172.21.15.7:0/2297244429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283c0094a0 con 0x7f2840151e80 2023-03-03T10:25:32.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.790+0000 7f2847de0700 1 -- 172.21.15.7:0/2297244429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2840154ef0 con 0x7f2840151e80 2023-03-03T10:25:32.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.791+0000 7f2847de0700 1 -- 172.21.15.7:0/2297244429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2840154c10 con 0x7f2840151e80 2023-03-03T10:25:32.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.791+0000 7f28367fc700 1 -- 172.21.15.7:0/2297244429 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2830019030 con 0x7f2840151e80 2023-03-03T10:25:32.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.791+0000 7f28367fc700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28240682f0 0x7f282406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:32.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.791+0000 7f28367fc700 1 -- 172.21.15.7:0/2297244429 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f28300861c0 con 0x7f2840151e80 2023-03-03T10:25:32.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.792+0000 7f2845b7c700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28240682f0 0x7f282406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:32.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.792+0000 7f2845b7c700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28240682f0 0x7f282406a7a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f283c0090b0 tx=0x7f283c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:25:32.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:32.796+0000 7f28367fc700 1 -- 172.21.15.7:0/2297244429 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2830052560 con 0x7f2840151e80 2023-03-03T10:25:33.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.033+0000 7f2847de0700 1 -- 172.21.15.7:0/2297244429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28400c5220 con 0x7f2840151e80 2023-03-03T10:25:33.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.034+0000 7f28367fc700 1 -- 172.21.15.7:0/2297244429 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2830017090 con 0x7f2840151e80 2023-03-03T10:25:33.036 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:25:33.037 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:25:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 -- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28240682f0 msgr2=0x7f282406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:33.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28240682f0 0x7f282406a7a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f283c0090b0 tx=0x7f283c01c040 comp rx=0 tx=0).stop 2023-03-03T10:25:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 -- 172.21.15.7:0/2297244429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2840151e80 msgr2=0x7f284014e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2840151e80 0x7f284014e140 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f283000ec10 tx=0x7f2830009e70 comp rx=0 tx=0).stop 2023-03-03T10:25:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 -- 172.21.15.7:0/2297244429 shutdown_connections 2023-03-03T10:25:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2840155410 0x7f284014c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28400c6fe0 0x7f28401e1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28240682f0 0x7f282406a7a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:33.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 --2- 172.21.15.7:0/2297244429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2840151e80 0x7f284014e140 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 -- 172.21.15.7:0/2297244429 >> 172.21.15.7:0/2297244429 conn(0x7f2840071630 msgr2=0x7f28400c2880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:25:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 -- 172.21.15.7:0/2297244429 shutdown_connections 2023-03-03T10:25:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:33.036+0000 7f2847de0700 1 -- 172.21.15.7:0/2297244429 wait complete. 2023-03-03T10:25:33.040 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:25:33.050 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T10:25:57.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:25:57.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.412+0000 7f9767ec2700 1 Processor -- start 2023-03-03T10:25:57.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.412+0000 7f9767ec2700 1 -- start start 2023-03-03T10:25:57.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.413+0000 7f9767ec2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f97600c8790 0x7f97600a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:57.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.413+0000 7f9767ec2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97600aa560 0x7f97600c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:57.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.413+0000 7f9767ec2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97600aa0d0 con 0x7f976006efd0 2023-03-03T10:25:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.413+0000 7f9767ec2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9760071ec0 con 0x7f97600aa560 2023-03-03T10:25:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.413+0000 7f9767ec2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97600a9aa0 con 0x7f97600c8790 2023-03-03T10:25:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.413+0000 7f976645f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97600aa560 0x7f97600c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.413+0000 7f976545d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f976006efd0 0x7f97601557e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54296/0 (socket says 172.21.15.7:54296) 2023-03-03T10:25:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.413+0000 7f976545d700 1 -- 172.21.15.7:0/3431282264 learned_addr learned my addr 172.21.15.7:0/3431282264 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:25:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f9765c5e700 1 --2- 172.21.15.7:0/3431282264 >> v2:172.21.15.143:3300/0 conn(0x7f97600c8790 0x7f97600a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f976645f700 1 -- 172.21.15.7:0/3431282264 >> v2:172.21.15.143:3300/0 conn(0x7f97600c8790 msgr2=0x7f97600a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:25:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f976645f700 1 --2- 172.21.15.7:0/3431282264 >> v2:172.21.15.143:3300/0 conn(0x7f97600c8790 0x7f97600a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f976645f700 1 -- 172.21.15.7:0/3431282264 >> v1:172.21.15.7:6789/0 conn(0x7f976006efd0 legacy=0x7f97601557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f976645f700 1 -- 172.21.15.7:0/3431282264 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97600c2710 con 0x7f97600aa560 2023-03-03T10:25:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f9764c5c700 1 -- 172.21.15.7:0/3431282264 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729752432 0 0) 0x7f97600aa0d0 con 0x7f976006efd0 2023-03-03T10:25:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f976645f700 1 --2- 172.21.15.7:0/3431282264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 0x7f97600c8b60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f97600c8760 tx=0x7f975800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd2a1c653acbbaa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:25:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f9764c5c700 1 -- 172.21.15.7:0/3431282264 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975800e070 con 0x7f97600aa560 2023-03-03T10:25:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f9764c5c700 1 -- 172.21.15.7:0/3431282264 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975800bbd0 con 0x7f97600aa560 2023-03-03T10:25:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f9764c5c700 1 -- 172.21.15.7:0/3431282264 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758012430 con 0x7f97600aa560 2023-03-03T10:25:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f9767ec2700 1 -- 172.21.15.7:0/3431282264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 msgr2=0x7f97600c8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:57.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.414+0000 7f9767ec2700 1 --2- 172.21.15.7:0/3431282264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 0x7f97600c8b60 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f97600c8760 tx=0x7f975800b330 comp rx=0 tx=0).stop 2023-03-03T10:25:57.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.415+0000 7f9767ec2700 1 -- 172.21.15.7:0/3431282264 shutdown_connections 2023-03-03T10:25:57.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.415+0000 7f9767ec2700 1 --2- 172.21.15.7:0/3431282264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 0x7f97600c8b60 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:57.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.415+0000 7f9767ec2700 1 --2- 172.21.15.7:0/3431282264 >> v2:172.21.15.143:3300/0 conn(0x7f97600c8790 0x7f97600a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:57.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.415+0000 7f9767ec2700 1 -- 172.21.15.7:0/3431282264 >> 172.21.15.7:0/3431282264 conn(0x7f97600736f0 msgr2=0x7f9760073af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:25:57.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.415+0000 7f9767ec2700 1 -- 172.21.15.7:0/3431282264 shutdown_connections 2023-03-03T10:25:57.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.415+0000 7f9767ec2700 1 -- 172.21.15.7:0/3431282264 wait complete. 2023-03-03T10:25:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.416+0000 7f9767ec2700 1 Processor -- start 2023-03-03T10:25:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.416+0000 7f9767ec2700 1 -- start start 2023-03-03T10:25:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.416+0000 7f9767ec2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006efd0 0x7f97601502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.416+0000 7f9767ec2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 0x7f97600ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f9767ec2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97600c8790 0x7f9760151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f9767ec2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9760150b70 con 0x7f97600aa560 2023-03-03T10:25:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f9767ec2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97600ab8c0 con 0x7f97600c8790 2023-03-03T10:25:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f9767ec2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9760141e30 con 0x7f976006efd0 2023-03-03T10:25:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f976545d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 0x7f97600ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f976545d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 0x7f97600ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60322/0 (socket says 172.21.15.7:60322) 2023-03-03T10:25:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f976545d700 1 -- 172.21.15.7:0/2786882372 learned_addr learned my addr 172.21.15.7:0/2786882372 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:25:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f976645f700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97600c8790 0x7f9760151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f976545d700 1 -- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006efd0 msgr2=0x7f97601502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f976545d700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006efd0 0x7f97601502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f976545d700 1 -- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97600c8790 msgr2=0x7f9760151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:57.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f976545d700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97600c8790 0x7f9760151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:57.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f976545d700 1 -- 172.21.15.7:0/2786882372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9758009a70 con 0x7f97600aa560 2023-03-03T10:25:57.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f976545d700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 0x7f97600ac710 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f975c00c580 tx=0x7f975c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:25:57.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f97567fc700 1 -- 172.21.15.7:0/2786882372 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c004c80 con 0x7f97600aa560 2023-03-03T10:25:57.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f97567fc700 1 -- 172.21.15.7:0/2786882372 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975c004e00 con 0x7f97600aa560 2023-03-03T10:25:57.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f9767ec2700 1 -- 172.21.15.7:0/2786882372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97600a9d60 con 0x7f97600aa560 2023-03-03T10:25:57.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f97567fc700 1 -- 172.21.15.7:0/2786882372 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c007310 con 0x7f97600aa560 2023-03-03T10:25:57.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.417+0000 7f9767ec2700 1 -- 172.21.15.7:0/2786882372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97600c29f0 con 0x7f97600aa560 2023-03-03T10:25:57.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.418+0000 7f9767ec2700 1 -- 172.21.15.7:0/2786882372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976014c8c0 con 0x7f97600aa560 2023-03-03T10:25:57.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.419+0000 7f97567fc700 1 -- 172.21.15.7:0/2786882372 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f975c01a020 con 0x7f97600aa560 2023-03-03T10:25:57.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.419+0000 7f97567fc700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97440682f0 0x7f974406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:25:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.419+0000 7f97567fc700 1 -- 172.21.15.7:0/2786882372 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f975c085d40 con 0x7f97600aa560 2023-03-03T10:25:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.419+0000 7f9765c5e700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97440682f0 0x7f974406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:25:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.420+0000 7f9765c5e700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97440682f0 0x7f974406a7a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9760151700 tx=0x7f9750008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:25:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.424+0000 7f97567fc700 1 -- 172.21.15.7:0/2786882372 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f975c052080 con 0x7f97600aa560 2023-03-03T10:25:57.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.662+0000 7f9767ec2700 1 -- 172.21.15.7:0/2786882372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97601500f0 con 0x7f97600aa560 2023-03-03T10:25:57.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.663+0000 7f97567fc700 1 -- 172.21.15.7:0/2786882372 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f975c0556a0 con 0x7f97600aa560 2023-03-03T10:25:57.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:25:57.666 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:25:57.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 -- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97440682f0 msgr2=0x7f974406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97440682f0 0x7f974406a7a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9760151700 tx=0x7f9750008040 comp rx=0 tx=0).stop 2023-03-03T10:25:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 -- 172.21.15.7:0/2786882372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 msgr2=0x7f97600ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:25:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 0x7f97600ac710 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f975c00c580 tx=0x7f975c00c940 comp rx=0 tx=0).stop 2023-03-03T10:25:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 -- 172.21.15.7:0/2786882372 shutdown_connections 2023-03-03T10:25:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97600c8790 0x7f9760151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:57.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f976006efd0 0x7f97601502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97440682f0 0x7f974406a7a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 --2- 172.21.15.7:0/2786882372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97600aa560 0x7f97600ac710 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:25:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 -- 172.21.15.7:0/2786882372 >> 172.21.15.7:0/2786882372 conn(0x7f97600736f0 msgr2=0x7f97601462f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:25:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 -- 172.21.15.7:0/2786882372 shutdown_connections 2023-03-03T10:25:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:25:57.665+0000 7f9767ec2700 1 -- 172.21.15.7:0/2786882372 wait complete. 2023-03-03T10:25:57.669 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:25:57.679 INFO:tasks.mds_thrash.fs.[a]:waiting for 77 secs before thrashing 2023-03-03T10:26:00.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:00.180 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:30.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:30.268 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:26:35.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:26:35.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.188+0000 7fe17a359700 1 Processor -- start 2023-03-03T10:26:35.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.188+0000 7fe17a359700 1 -- start start 2023-03-03T10:26:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe17a359700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe17414f9d0 0x7fe174156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe17a359700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe17414fda0 0x7fe174071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe17a359700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1740c2f30 con 0x7fe1740abe20 2023-03-03T10:26:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe17a359700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1740c31d0 con 0x7fe17414fda0 2023-03-03T10:26:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe17a359700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1741524a0 con 0x7fe17414f9d0 2023-03-03T10:26:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe1788f6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe1740abe20 0x7fe1741529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54328/0 (socket says 172.21.15.7:54328) 2023-03-03T10:26:35.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe1788f6700 1 -- 172.21.15.7:0/3445861580 learned_addr learned my addr 172.21.15.7:0/3445861580 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:26:35.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957338102 0 0) 0x7fe1740c2f30 con 0x7fe1740abe20 2023-03-03T10:26:35.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1600035b0 con 0x7fe1740abe20 2023-03-03T10:26:35.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe173fff700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe17414f9d0 0x7fe174156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:35.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe1737fe700 1 --2- 172.21.15.7:0/3445861580 >> v2:172.21.15.143:3300/0 conn(0x7fe17414fda0 0x7fe174071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:35.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351572873 0 0) 0x7fe1600035b0 con 0x7fe1740abe20 2023-03-03T10:26:35.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe17414f9d0 msgr2=0x7fe174156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:26:35.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe172ffd700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe17414f9d0 0x7fe174156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:35.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 >> v2:172.21.15.143:3300/0 conn(0x7fe17414fda0 msgr2=0x7fe174071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:26:35.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe172ffd700 1 --2- 172.21.15.7:0/3445861580 >> v2:172.21.15.143:3300/0 conn(0x7fe17414fda0 0x7fe174071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:35.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1740c34b0 con 0x7fe1740abe20 2023-03-03T10:26:35.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.189+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe16c0019f0 con 0x7fe1740abe20 2023-03-03T10:26:35.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 >> v1:172.21.15.7:6789/0 conn(0x7fe1740abe20 legacy=0x7fe1741529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:35.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe172ffd700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe160003a70 0x7fe160003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:35.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe172ffd700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe17414f9d0 0x7fe160004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:35.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe172ffd700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe160004d90 0x7fe160005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:35.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1740c31d0 con 0x7fe160003a70 2023-03-03T10:26:35.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1741524a0 con 0x7fe160004d90 2023-03-03T10:26:35.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16000baf0 con 0x7fe17414f9d0 2023-03-03T10:26:35.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe173fff700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe160003a70 0x7fe160003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:35.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe1737fe700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe17414f9d0 0x7fe160004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:35.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe1788f6700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe160004d90 0x7fe160005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:35.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe173fff700 1 -- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe17414f9d0 msgr2=0x7fe160004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:26:35.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe173fff700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe17414f9d0 0x7fe160004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe173fff700 1 -- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe160004d90 msgr2=0x7fe160005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:26:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe173fff700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe160004d90 0x7fe160005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe173fff700 1 -- 172.21.15.7:0/3445861580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1600035b0 con 0x7fe160003a70 2023-03-03T10:26:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe1788f6700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe160004d90 0x7fe160005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:26:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe173fff700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe160003a70 0x7fe160003e80 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe164009b60 tx=0x7fe160005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a9c84ac295114a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:26:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16400e6c0 con 0x7fe160003a70 2023-03-03T10:26:35.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16400e820 con 0x7fe160003a70 2023-03-03T10:26:35.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.190+0000 7fe172ffd700 1 -- 172.21.15.7:0/3445861580 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16400eb90 con 0x7fe160003a70 2023-03-03T10:26:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.191+0000 7fe17a359700 1 -- 172.21.15.7:0/3445861580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe160003a70 msgr2=0x7fe160003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.191+0000 7fe17a359700 1 --2- 172.21.15.7:0/3445861580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe160003a70 0x7fe160003e80 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe164009b60 tx=0x7fe160005880 comp rx=0 tx=0).stop 2023-03-03T10:26:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.191+0000 7fe1788f6700 1 -- 172.21.15.7:0/3445861580 reap_dead start 2023-03-03T10:26:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.191+0000 7fe17a359700 1 -- 172.21.15.7:0/3445861580 shutdown_connections 2023-03-03T10:26:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.191+0000 7fe17a359700 1 -- 172.21.15.7:0/3445861580 >> 172.21.15.7:0/3445861580 conn(0x7fe1740c8900 msgr2=0x7fe1740c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:26:35.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.191+0000 7fe17a359700 1 -- 172.21.15.7:0/3445861580 shutdown_connections 2023-03-03T10:26:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.191+0000 7fe17a359700 1 -- 172.21.15.7:0/3445861580 wait complete. 2023-03-03T10:26:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.192+0000 7fe17a359700 1 Processor -- start 2023-03-03T10:26:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.192+0000 7fe17a359700 1 -- start start 2023-03-03T10:26:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe17a359700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1740a9730 0x7fe1741eaa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe17a359700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1740a9b00 0x7fe1741edbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe17a359700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1741ee0f0 0x7fe174145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe17a359700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1741ed720 con 0x7fe1741ee0f0 2023-03-03T10:26:35.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe17a359700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17406d0f0 con 0x7fe1740a9730 2023-03-03T10:26:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe17a359700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1741eaf50 con 0x7fe1740a9b00 2023-03-03T10:26:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe1788f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1741ee0f0 0x7fe174145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe1788f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1741ee0f0 0x7fe174145160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60360/0 (socket says 172.21.15.7:60360) 2023-03-03T10:26:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe1788f6700 1 -- 172.21.15.7:0/227953263 learned_addr learned my addr 172.21.15.7:0/227953263 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:26:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe1737fe700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1740a9b00 0x7fe1741edbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.193+0000 7fe173fff700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1740a9730 0x7fe1741eaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:35.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe1788f6700 1 -- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1740a9b00 msgr2=0x7fe1741edbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe1788f6700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1740a9b00 0x7fe1741edbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe1788f6700 1 -- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1740a9730 msgr2=0x7fe1741eaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe1788f6700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1740a9730 0x7fe1741eaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe1788f6700 1 -- 172.21.15.7:0/227953263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1740ac350 con 0x7fe1741ee0f0 2023-03-03T10:26:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe1737fe700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1740a9b00 0x7fe1741edbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:26:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe1788f6700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1741ee0f0 0x7fe174145160 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0013d0 tx=0x7fe16c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:26:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe170ff9700 1 -- 172.21.15.7:0/227953263 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c017070 con 0x7fe1741ee0f0 2023-03-03T10:26:35.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe170ff9700 1 -- 172.21.15.7:0/227953263 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16c00ebe0 con 0x7fe1741ee0f0 2023-03-03T10:26:35.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe17a359700 1 -- 172.21.15.7:0/227953263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1740ab630 con 0x7fe1741ee0f0 2023-03-03T10:26:35.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe170ff9700 1 -- 172.21.15.7:0/227953263 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c00ed50 con 0x7fe1741ee0f0 2023-03-03T10:26:35.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.194+0000 7fe17a359700 1 -- 172.21.15.7:0/227953263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1740c3e10 con 0x7fe1741ee0f0 2023-03-03T10:26:35.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.195+0000 7fe17a359700 1 -- 172.21.15.7:0/227953263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1741ed8d0 con 0x7fe1741ee0f0 2023-03-03T10:26:35.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.195+0000 7fe170ff9700 1 -- 172.21.15.7:0/227953263 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe16c013420 con 0x7fe1741ee0f0 2023-03-03T10:26:35.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.196+0000 7fe170ff9700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1540682f0 0x7fe15406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:26:35.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.196+0000 7fe170ff9700 1 -- 172.21.15.7:0/227953263 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe16c0861d0 con 0x7fe1741ee0f0 2023-03-03T10:26:35.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.196+0000 7fe173fff700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1540682f0 0x7fe15406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:26:35.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.197+0000 7fe173fff700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1540682f0 0x7fe15406a7a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe1740aaae0 tx=0x7fe16401e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:26:35.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.201+0000 7fe170ff9700 1 -- 172.21.15.7:0/227953263 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe1741ed8d0 con 0x7fe1741ee0f0 2023-03-03T10:26:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.437+0000 7fe17a359700 1 -- 172.21.15.7:0/227953263 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1741ed8d0 con 0x7fe1741ee0f0 2023-03-03T10:26:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.438+0000 7fe170ff9700 1 -- 172.21.15.7:0/227953263 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe1741ed8d0 con 0x7fe1741ee0f0 2023-03-03T10:26:35.440 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:26:35.440 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:26:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.439+0000 7fe17a359700 1 -- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1540682f0 msgr2=0x7fe15406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.439+0000 7fe17a359700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1540682f0 0x7fe15406a7a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe1740aaae0 tx=0x7fe16401e040 comp rx=0 tx=0).stop 2023-03-03T10:26:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.439+0000 7fe17a359700 1 -- 172.21.15.7:0/227953263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1741ee0f0 msgr2=0x7fe174145160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:26:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.439+0000 7fe17a359700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1741ee0f0 0x7fe174145160 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0013d0 tx=0x7fe16c00e440 comp rx=0 tx=0).stop 2023-03-03T10:26:35.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.439+0000 7fe17a359700 1 -- 172.21.15.7:0/227953263 shutdown_connections 2023-03-03T10:26:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.439+0000 7fe17a359700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1540682f0 0x7fe15406a7a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.439+0000 7fe17a359700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1741ee0f0 0x7fe174145160 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.439+0000 7fe17a359700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1740a9730 0x7fe1741eaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.439+0000 7fe17a359700 1 --2- 172.21.15.7:0/227953263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1740a9b00 0x7fe1741edbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:26:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.439+0000 7fe17a359700 1 -- 172.21.15.7:0/227953263 >> 172.21.15.7:0/227953263 conn(0x7fe1740c8900 msgr2=0x7fe174155990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:26:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.440+0000 7fe17a359700 1 -- 172.21.15.7:0/227953263 shutdown_connections 2023-03-03T10:26:35.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:26:35.440+0000 7fe17a359700 1 -- 172.21.15.7:0/227953263 wait complete. 2023-03-03T10:26:35.444 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:26:35.454 INFO:tasks.mds_thrash.fs.[b]:waiting for 102 secs before thrashing 2023-03-03T10:27:00.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:00.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:14.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:27:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.817+0000 7f1635bcc700 1 Processor -- start 2023-03-03T10:27:14.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.817+0000 7f1635bcc700 1 -- start start 2023-03-03T10:27:14.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f1635bcc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16300c2e30 0x7f1630158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f1635bcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 0x7f16300736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f1635bcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1630157ff0 con 0x7f1630151cd0 2023-03-03T10:27:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f1635bcc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f163006d640 con 0x7f16300c2e30 2023-03-03T10:27:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f1635bcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163006cda0 con 0x7f163006c6c0 2023-03-03T10:27:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f162effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16300c2e30 0x7f1630158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f162effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16300c2e30 0x7f1630158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60372/0 (socket says 172.21.15.7:60372) 2023-03-03T10:27:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f162effd700 1 -- 172.21.15.7:0/3724922854 learned_addr learned my addr 172.21.15.7:0/3724922854 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:27:14.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f162ffff700 1 --2- 172.21.15.7:0/3724922854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 0x7f16300736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:14.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f162effd700 1 -- 172.21.15.7:0/3724922854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 msgr2=0x7f16300736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:27:14.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f162effd700 1 --2- 172.21.15.7:0/3724922854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 0x7f16300736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:14.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f162effd700 1 -- 172.21.15.7:0/3724922854 >> v1:172.21.15.143:6789/0 conn(0x7f1630151cd0 legacy=0x7f1630154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:14.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f162effd700 1 -- 172.21.15.7:0/3724922854 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163006d080 con 0x7f16300c2e30 2023-03-03T10:27:14.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.818+0000 7f162ffff700 1 --2- 172.21.15.7:0/3724922854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 0x7f16300736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:27:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f162effd700 1 --2- 172.21.15.7:0/3724922854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 0x7f1630158480 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f16180098f0 tx=0x7f1618009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410ea79a4e0189d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:27:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f162e7fc700 1 -- 172.21.15.7:0/3724922854 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161800b8a0 con 0x7f16300c2e30 2023-03-03T10:27:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f162e7fc700 1 -- 172.21.15.7:0/3724922854 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161800ba00 con 0x7f16300c2e30 2023-03-03T10:27:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f162e7fc700 1 -- 172.21.15.7:0/3724922854 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161800bd90 con 0x7f16300c2e30 2023-03-03T10:27:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f1635bcc700 1 -- 172.21.15.7:0/3724922854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 msgr2=0x7f1630158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f1635bcc700 1 --2- 172.21.15.7:0/3724922854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 0x7f1630158480 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f16180098f0 tx=0x7f1618009c00 comp rx=0 tx=0).stop 2023-03-03T10:27:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f1635bcc700 1 -- 172.21.15.7:0/3724922854 shutdown_connections 2023-03-03T10:27:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f1635bcc700 1 --2- 172.21.15.7:0/3724922854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 0x7f16300736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f1635bcc700 1 --2- 172.21.15.7:0/3724922854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 0x7f1630158480 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f1635bcc700 1 -- 172.21.15.7:0/3724922854 >> 172.21.15.7:0/3724922854 conn(0x7f16300a94d0 msgr2=0x7f16300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:27:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f1635bcc700 1 -- 172.21.15.7:0/3724922854 shutdown_connections 2023-03-03T10:27:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.819+0000 7f1635bcc700 1 -- 172.21.15.7:0/3724922854 wait complete. 2023-03-03T10:27:14.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.821+0000 7f1635bcc700 1 Processor -- start 2023-03-03T10:27:14.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.821+0000 7f1635bcc700 1 -- start start 2023-03-03T10:27:14.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f1635bcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 0x7f16300c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f1635bcc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 0x7f16301ea9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f1635bcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cd0 0x7f16301eaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f1635bcc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1630154080 con 0x7f16300c2e30 2023-03-03T10:27:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f1635bcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1630154360 con 0x7f1630151cd0 2023-03-03T10:27:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f1635bcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16301ea490 con 0x7f163006c6c0 2023-03-03T10:27:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f162effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 0x7f16301ea9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f162effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 0x7f16301ea9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60378/0 (socket says 172.21.15.7:60378) 2023-03-03T10:27:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f162effd700 1 -- 172.21.15.7:0/227259817 learned_addr learned my addr 172.21.15.7:0/227259817 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:27:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f162f7fe700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 0x7f16300c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.822+0000 7f162ffff700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cd0 0x7f16301eaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f162effd700 1 -- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 msgr2=0x7f16300c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f162effd700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 0x7f16300c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f162effd700 1 -- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cd0 msgr2=0x7f16301eaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f162effd700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cd0 0x7f16301eaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f162effd700 1 -- 172.21.15.7:0/227259817 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1618009550 con 0x7f16300c2e30 2023-03-03T10:27:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f162effd700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 0x7f16301ea9b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1618009c40 tx=0x7f1618004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:27:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f1627fff700 1 -- 172.21.15.7:0/227259817 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618004510 con 0x7f16300c2e30 2023-03-03T10:27:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f1627fff700 1 -- 172.21.15.7:0/227259817 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161801e070 con 0x7f16300c2e30 2023-03-03T10:27:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f1635bcc700 1 -- 172.21.15.7:0/227259817 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1630157c80 con 0x7f16300c2e30 2023-03-03T10:27:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f1627fff700 1 -- 172.21.15.7:0/227259817 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16180038e0 con 0x7f16300c2e30 2023-03-03T10:27:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.823+0000 7f1635bcc700 1 -- 172.21.15.7:0/227259817 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163006d360 con 0x7f16300c2e30 2023-03-03T10:27:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.824+0000 7f1635bcc700 1 -- 172.21.15.7:0/227259817 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16301589c0 con 0x7f16300c2e30 2023-03-03T10:27:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.824+0000 7f1627fff700 1 -- 172.21.15.7:0/227259817 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f161800f050 con 0x7f16300c2e30 2023-03-03T10:27:14.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.825+0000 7f1627fff700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 0x7f161006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:27:14.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.825+0000 7f1627fff700 1 -- 172.21.15.7:0/227259817 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1618020070 con 0x7f16300c2e30 2023-03-03T10:27:14.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.825+0000 7f162f7fe700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 0x7f161006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:27:14.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.826+0000 7f162f7fe700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 0x7f161006a7a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1620007fd0 tx=0x7f1620009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:27:14.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:14.829+0000 7f1627fff700 1 -- 172.21.15.7:0/227259817 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1618053840 con 0x7f16300c2e30 2023-03-03T10:27:15.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.065+0000 7f1635bcc700 1 -- 172.21.15.7:0/227259817 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16301541c0 con 0x7f16300c2e30 2023-03-03T10:27:15.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.066+0000 7f1627fff700 1 -- 172.21.15.7:0/227259817 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f16301541c0 con 0x7f16300c2e30 2023-03-03T10:27:15.070 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:27:15.070 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:27:15.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 -- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 msgr2=0x7f161006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:15.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 0x7f161006a7a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1620007fd0 tx=0x7f1620009040 comp rx=0 tx=0).stop 2023-03-03T10:27:15.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 -- 172.21.15.7:0/227259817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 msgr2=0x7f16301ea9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:27:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 0x7f16301ea9b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1618009c40 tx=0x7f1618004750 comp rx=0 tx=0).stop 2023-03-03T10:27:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 -- 172.21.15.7:0/227259817 shutdown_connections 2023-03-03T10:27:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cd0 0x7f16301eaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163006c6c0 0x7f16300c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 0x7f161006a7a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 --2- 172.21.15.7:0/227259817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c2e30 0x7f16301ea9b0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:27:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 -- 172.21.15.7:0/227259817 >> 172.21.15.7:0/227259817 conn(0x7f16300a94d0 msgr2=0x7f163006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:27:15.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 -- 172.21.15.7:0/227259817 shutdown_connections 2023-03-03T10:27:15.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:27:15.068+0000 7f1635bcc700 1 -- 172.21.15.7:0/227259817 wait complete. 2023-03-03T10:27:15.073 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:27:15.083 INFO:tasks.mds_thrash.fs.[a]:waiting for 82 secs before thrashing 2023-03-03T10:27:30.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:27:30.528 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:00.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:00.600 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:17.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:28:17.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.593+0000 7f2bc23db700 1 Processor -- start 2023-03-03T10:28:17.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.593+0000 7f2bc23db700 1 -- start start 2023-03-03T10:28:17.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.593+0000 7f2bc23db700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2bbc072250 0x7f2bbc14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:17.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.593+0000 7f2bc23db700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2bbc0c1ce0 0x7f2bbc14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:17.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.593+0000 7f2bc23db700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc1510d0 con 0x7f2bbc071e80 2023-03-03T10:28:17.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.593+0000 7f2bc23db700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc0c3740 con 0x7f2bbc072250 2023-03-03T10:28:17.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.593+0000 7f2bc23db700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc070670 con 0x7f2bbc0c1ce0 2023-03-03T10:28:17.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2bbc072250 0x7f2bbc14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:17.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2bbc072250 0x7f2bbc14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60420/0 (socket says 172.21.15.7:60420) 2023-03-03T10:28:17.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbb7fe700 1 -- 172.21.15.7:0/3492595921 learned_addr learned my addr 172.21.15.7:0/3492595921 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:28:17.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bc0978700 1 --2- 172.21.15.7:0/3492595921 >> v2:172.21.15.143:3300/0 conn(0x7f2bbc0c1ce0 0x7f2bbc14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:17.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbb7fe700 1 -- 172.21.15.7:0/3492595921 >> v2:172.21.15.143:3300/0 conn(0x7f2bbc0c1ce0 msgr2=0x7f2bbc14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:28:17.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbb7fe700 1 --2- 172.21.15.7:0/3492595921 >> v2:172.21.15.143:3300/0 conn(0x7f2bbc0c1ce0 0x7f2bbc14e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:17.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbb7fe700 1 -- 172.21.15.7:0/3492595921 >> v1:172.21.15.143:6789/0 conn(0x7f2bbc071e80 legacy=0x7f2bbc151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:17.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbb7fe700 1 -- 172.21.15.7:0/3492595921 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc0c39e0 con 0x7f2bbc072250 2023-03-03T10:28:17.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbb7fe700 1 --2- 172.21.15.7:0/3492595921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 0x7f2bbc14e1f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc071090 tx=0x7f2bac00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ed23a619d04de server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:17.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbaffd700 1 -- 172.21.15.7:0/3492595921 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac00dc70 con 0x7f2bbc072250 2023-03-03T10:28:17.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbaffd700 1 -- 172.21.15.7:0/3492595921 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bac00ddd0 con 0x7f2bbc072250 2023-03-03T10:28:17.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.594+0000 7f2bbaffd700 1 -- 172.21.15.7:0/3492595921 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac010690 con 0x7f2bbc072250 2023-03-03T10:28:17.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.595+0000 7f2bc23db700 1 -- 172.21.15.7:0/3492595921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 msgr2=0x7f2bbc14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:17.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.595+0000 7f2bc23db700 1 --2- 172.21.15.7:0/3492595921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 0x7f2bbc14e1f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc071090 tx=0x7f2bac00be70 comp rx=0 tx=0).stop 2023-03-03T10:28:17.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.595+0000 7f2bc23db700 1 -- 172.21.15.7:0/3492595921 shutdown_connections 2023-03-03T10:28:17.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.595+0000 7f2bc23db700 1 --2- 172.21.15.7:0/3492595921 >> v2:172.21.15.143:3300/0 conn(0x7f2bbc0c1ce0 0x7f2bbc14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:17.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.595+0000 7f2bc23db700 1 --2- 172.21.15.7:0/3492595921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 0x7f2bbc14e1f0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:17.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.595+0000 7f2bc23db700 1 -- 172.21.15.7:0/3492595921 >> 172.21.15.7:0/3492595921 conn(0x7f2bbc0a94d0 msgr2=0x7f2bbc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:28:17.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.595+0000 7f2bc23db700 1 -- 172.21.15.7:0/3492595921 shutdown_connections 2023-03-03T10:28:17.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.595+0000 7f2bc23db700 1 -- 172.21.15.7:0/3492595921 wait complete. 2023-03-03T10:28:17.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.596+0000 7f2bc23db700 1 Processor -- start 2023-03-03T10:28:17.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.596+0000 7f2bc23db700 1 -- start start 2023-03-03T10:28:17.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bc23db700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bbc071e80 0x7f2bbc1ed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:17.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bc23db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 0x7f2bbc1ea5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:17.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bc23db700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bbc0c1ce0 0x7f2bbc0be6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:17.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bc23db700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc06df30 con 0x7f2bbc072250 2023-03-03T10:28:17.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bc23db700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc1e9bb0 con 0x7f2bbc0c1ce0 2023-03-03T10:28:17.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bc23db700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc1ea130 con 0x7f2bbc071e80 2023-03-03T10:28:17.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bbb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 0x7f2bbc1ea5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:17.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bbb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 0x7f2bbc1ea5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60426/0 (socket says 172.21.15.7:60426) 2023-03-03T10:28:17.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bbb7fe700 1 -- 172.21.15.7:0/3973239855 learned_addr learned my addr 172.21.15.7:0/3973239855 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:28:17.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bc0978700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bbc0c1ce0 0x7f2bbc0be6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:17.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bbbfff700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bbc071e80 0x7f2bbc1ed1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:17.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bbb7fe700 1 -- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bbc071e80 msgr2=0x7f2bbc1ed1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:17.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bbb7fe700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bbc071e80 0x7f2bbc1ed1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:17.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bbb7fe700 1 -- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bbc0c1ce0 msgr2=0x7f2bbc0be6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:17.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bbb7fe700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bbc0c1ce0 0x7f2bbc0be6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:17.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.597+0000 7f2bbb7fe700 1 -- 172.21.15.7:0/3973239855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac00b7f0 con 0x7f2bbc072250 2023-03-03T10:28:17.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.598+0000 7f2bbb7fe700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 0x7f2bbc1ea5c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00beb0 tx=0x7f2bac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:17.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.598+0000 7f2bb8ff9700 1 -- 172.21.15.7:0/3973239855 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac01d070 con 0x7f2bbc072250 2023-03-03T10:28:17.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.598+0000 7f2bb8ff9700 1 -- 172.21.15.7:0/3973239855 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bac010680 con 0x7f2bbc072250 2023-03-03T10:28:17.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.598+0000 7f2bc23db700 1 -- 172.21.15.7:0/3973239855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc151f30 con 0x7f2bbc072250 2023-03-03T10:28:17.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.598+0000 7f2bb8ff9700 1 -- 172.21.15.7:0/3973239855 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac010aa0 con 0x7f2bbc072250 2023-03-03T10:28:17.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.598+0000 7f2bc23db700 1 -- 172.21.15.7:0/3973239855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bbc151300 con 0x7f2bbc072250 2023-03-03T10:28:17.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.599+0000 7f2bc23db700 1 -- 172.21.15.7:0/3973239855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bbc06c9a0 con 0x7f2bbc072250 2023-03-03T10:28:17.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.599+0000 7f2bb8ff9700 1 -- 172.21.15.7:0/3973239855 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2bac0036a0 con 0x7f2bbc072250 2023-03-03T10:28:17.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.600+0000 7f2bb8ff9700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b9c0682f0 0x7f2b9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:17.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.600+0000 7f2bb8ff9700 1 -- 172.21.15.7:0/3973239855 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2bac087790 con 0x7f2bbc072250 2023-03-03T10:28:17.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.600+0000 7f2bbbfff700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b9c0682f0 0x7f2b9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:17.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.600+0000 7f2bbbfff700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b9c0682f0 0x7f2b9c06a7a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc06e240 tx=0x7f2ba4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:17.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.604+0000 7f2bb8ff9700 1 -- 172.21.15.7:0/3973239855 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2bac053b30 con 0x7f2bbc072250 2023-03-03T10:28:17.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.843+0000 7f2bc23db700 1 -- 172.21.15.7:0/3973239855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bbc06d9d0 con 0x7f2bbc072250 2023-03-03T10:28:17.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.843+0000 7f2bb8ff9700 1 -- 172.21.15.7:0/3973239855 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2bac057150 con 0x7f2bbc072250 2023-03-03T10:28:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:28:17.846 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:28:17.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 -- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b9c0682f0 msgr2=0x7f2b9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:17.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b9c0682f0 0x7f2b9c06a7a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc06e240 tx=0x7f2ba4009040 comp rx=0 tx=0).stop 2023-03-03T10:28:17.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 -- 172.21.15.7:0/3973239855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 msgr2=0x7f2bbc1ea5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:17.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 0x7f2bbc1ea5c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00beb0 tx=0x7f2bac004750 comp rx=0 tx=0).stop 2023-03-03T10:28:17.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 -- 172.21.15.7:0/3973239855 shutdown_connections 2023-03-03T10:28:17.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bbc0c1ce0 0x7f2bbc0be6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:17.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bbc071e80 0x7f2bbc1ed1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:17.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b9c0682f0 0x7f2b9c06a7a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:17.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 --2- 172.21.15.7:0/3973239855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bbc072250 0x7f2bbc1ea5c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:17.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 -- 172.21.15.7:0/3973239855 >> 172.21.15.7:0/3973239855 conn(0x7f2bbc0a94d0 msgr2=0x7f2bbc0c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:28:17.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 -- 172.21.15.7:0/3973239855 shutdown_connections 2023-03-03T10:28:17.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:17.845+0000 7f2bc23db700 1 -- 172.21.15.7:0/3973239855 wait complete. 2023-03-03T10:28:17.849 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:28:17.859 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-03-03T10:28:30.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:30.691 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:28:37.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:28:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.220+0000 7f112a532700 1 Processor -- start 2023-03-03T10:28:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.220+0000 7f112a532700 1 -- start start 2023-03-03T10:28:37.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.221+0000 7f112a532700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11240c00e0 0x7f11240c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:37.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.221+0000 7f112a532700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1124154ff0 0x7f11240cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:37.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.221+0000 7f112a532700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11240c7310 con 0x7f11240caa30 2023-03-03T10:28:37.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.221+0000 7f112a532700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1124155430 con 0x7f11240c00e0 2023-03-03T10:28:37.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.221+0000 7f112a532700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11240bfd10 con 0x7f1124154ff0 2023-03-03T10:28:37.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.221+0000 7f1123fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f11240caa30 0x7f11240c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54426/0 (socket says 172.21.15.7:54426) 2023-03-03T10:28:37.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.221+0000 7f11237fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11240c00e0 0x7f11240c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:37.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.221+0000 7f1123fff700 1 -- 172.21.15.7:0/2886510106 learned_addr learned my addr 172.21.15.7:0/2886510106 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:28:37.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.222+0000 7f1128acf700 1 --2- 172.21.15.7:0/2886510106 >> v2:172.21.15.143:3300/0 conn(0x7f1124154ff0 0x7f11240cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:37.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.222+0000 7f11237fe700 1 -- 172.21.15.7:0/2886510106 >> v2:172.21.15.143:3300/0 conn(0x7f1124154ff0 msgr2=0x7f11240cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:28:37.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.222+0000 7f11237fe700 1 --2- 172.21.15.7:0/2886510106 >> v2:172.21.15.143:3300/0 conn(0x7f1124154ff0 0x7f11240cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:37.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.222+0000 7f11237fe700 1 -- 172.21.15.7:0/2886510106 >> v1:172.21.15.7:6789/0 conn(0x7f11240caa30 legacy=0x7f11240c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:37.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.222+0000 7f1122ffd700 1 -- 172.21.15.7:0/2886510106 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291555766 0 0) 0x7f11240c7310 con 0x7f11240caa30 2023-03-03T10:28:37.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.222+0000 7f11237fe700 1 -- 172.21.15.7:0/2886510106 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11240bfa30 con 0x7f11240c00e0 2023-03-03T10:28:37.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.222+0000 7f11237fe700 1 --2- 172.21.15.7:0/2886510106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240c00e0 0x7f11240c5c60 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f1124071530 tx=0x7f1118009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f5fbc6f8f1c3766 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:37.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.222+0000 7f1122ffd700 1 -- 172.21.15.7:0/2886510106 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118005e30 con 0x7f11240c00e0 2023-03-03T10:28:37.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.222+0000 7f1122ffd700 1 -- 172.21.15.7:0/2886510106 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1118006030 con 0x7f11240c00e0 2023-03-03T10:28:37.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.222+0000 7f1122ffd700 1 -- 172.21.15.7:0/2886510106 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11180063c0 con 0x7f11240c00e0 2023-03-03T10:28:37.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.223+0000 7f112a532700 1 -- 172.21.15.7:0/2886510106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240c00e0 msgr2=0x7f11240c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:37.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.223+0000 7f112a532700 1 --2- 172.21.15.7:0/2886510106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240c00e0 0x7f11240c5c60 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f1124071530 tx=0x7f1118009b00 comp rx=0 tx=0).stop 2023-03-03T10:28:37.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.223+0000 7f112a532700 1 -- 172.21.15.7:0/2886510106 shutdown_connections 2023-03-03T10:28:37.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.223+0000 7f112a532700 1 --2- 172.21.15.7:0/2886510106 >> v2:172.21.15.143:3300/0 conn(0x7f1124154ff0 0x7f11240cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:37.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.223+0000 7f112a532700 1 --2- 172.21.15.7:0/2886510106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240c00e0 0x7f11240c5c60 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:37.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.223+0000 7f112a532700 1 -- 172.21.15.7:0/2886510106 >> 172.21.15.7:0/2886510106 conn(0x7f1124066e40 msgr2=0x7f1124071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:28:37.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.223+0000 7f112a532700 1 -- 172.21.15.7:0/2886510106 shutdown_connections 2023-03-03T10:28:37.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.223+0000 7f112a532700 1 -- 172.21.15.7:0/2886510106 wait complete. 2023-03-03T10:28:37.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.224+0000 7f112a532700 1 Processor -- start 2023-03-03T10:28:37.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.224+0000 7f112a532700 1 -- start start 2023-03-03T10:28:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f112a532700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11240c00e0 0x7f11240c4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f112a532700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240caa30 0x7f1124149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f112a532700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1124154ff0 0x7f11241579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f112a532700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112414a400 con 0x7f11240caa30 2023-03-03T10:28:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f112a532700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11240c2160 con 0x7f11240c00e0 2023-03-03T10:28:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f112a532700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1124149be0 con 0x7f1124154ff0 2023-03-03T10:28:37.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f11237fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240caa30 0x7f1124149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f11237fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240caa30 0x7f1124149ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60452/0 (socket says 172.21.15.7:60452) 2023-03-03T10:28:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f11237fe700 1 -- 172.21.15.7:0/2426050044 learned_addr learned my addr 172.21.15.7:0/2426050044 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:28:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f1128acf700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1124154ff0 0x7f11241579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f1123fff700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11240c00e0 0x7f11240c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f11237fe700 1 -- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1124154ff0 msgr2=0x7f11241579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f11237fe700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1124154ff0 0x7f11241579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f11237fe700 1 -- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11240c00e0 msgr2=0x7f11240c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f11237fe700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11240c00e0 0x7f11240c4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f11237fe700 1 -- 172.21.15.7:0/2426050044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1118009430 con 0x7f11240caa30 2023-03-03T10:28:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.225+0000 7f1123fff700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11240c00e0 0x7f11240c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:28:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.226+0000 7f11237fe700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240caa30 0x7f1124149ec0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1118000c00 tx=0x7f1118004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.226+0000 7f1120ff9700 1 -- 172.21.15.7:0/2426050044 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118004040 con 0x7f11240caa30 2023-03-03T10:28:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.226+0000 7f1120ff9700 1 -- 172.21.15.7:0/2426050044 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1118020070 con 0x7f11240caa30 2023-03-03T10:28:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.226+0000 7f112a532700 1 -- 172.21.15.7:0/2426050044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11241547f0 con 0x7f11240caa30 2023-03-03T10:28:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.226+0000 7f1120ff9700 1 -- 172.21.15.7:0/2426050044 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118004400 con 0x7f11240caa30 2023-03-03T10:28:37.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.226+0000 7f112a532700 1 -- 172.21.15.7:0/2426050044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1124154510 con 0x7f11240caa30 2023-03-03T10:28:37.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.227+0000 7f112a532700 1 -- 172.21.15.7:0/2426050044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1124154ad0 con 0x7f11240caa30 2023-03-03T10:28:37.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.227+0000 7f1120ff9700 1 -- 172.21.15.7:0/2426050044 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f111802b020 con 0x7f11240caa30 2023-03-03T10:28:37.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.227+0000 7f1120ff9700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11100682f0 0x7f111006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:28:37.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.228+0000 7f1120ff9700 1 -- 172.21.15.7:0/2426050044 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1118087700 con 0x7f11240caa30 2023-03-03T10:28:37.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.228+0000 7f1123fff700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11100682f0 0x7f111006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:28:37.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.228+0000 7f1123fff700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11100682f0 0x7f111006a7a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1114000f60 tx=0x7f1114007cc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:28:37.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.232+0000 7f1120ff9700 1 -- 172.21.15.7:0/2426050044 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1118053aa0 con 0x7f11240caa30 2023-03-03T10:28:37.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.471+0000 7f112a532700 1 -- 172.21.15.7:0/2426050044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1124074bb0 con 0x7f11240caa30 2023-03-03T10:28:37.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.472+0000 7f1120ff9700 1 -- 172.21.15.7:0/2426050044 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f11180570c0 con 0x7f11240caa30 2023-03-03T10:28:37.474 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:28:37.475 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:28:37.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 -- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11100682f0 msgr2=0x7f111006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:37.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11100682f0 0x7f111006a7a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1114000f60 tx=0x7f1114007cc0 comp rx=0 tx=0).stop 2023-03-03T10:28:37.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 -- 172.21.15.7:0/2426050044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240caa30 msgr2=0x7f1124149ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:28:37.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240caa30 0x7f1124149ec0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1118000c00 tx=0x7f1118004750 comp rx=0 tx=0).stop 2023-03-03T10:28:37.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 -- 172.21.15.7:0/2426050044 shutdown_connections 2023-03-03T10:28:37.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1124154ff0 0x7f11241579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:37.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11240c00e0 0x7f11240c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:37.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11100682f0 0x7f111006a7a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:37.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 --2- 172.21.15.7:0/2426050044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11240caa30 0x7f1124149ec0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:28:37.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 -- 172.21.15.7:0/2426050044 >> 172.21.15.7:0/2426050044 conn(0x7f1124066e40 msgr2=0x7f1124151230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:28:37.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 -- 172.21.15.7:0/2426050044 shutdown_connections 2023-03-03T10:28:37.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:28:37.474+0000 7f112a532700 1 -- 172.21.15.7:0/2426050044 wait complete. 2023-03-03T10:28:37.478 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:28:37.488 INFO:tasks.mds_thrash.fs.[a]:waiting for 77 secs before thrashing 2023-03-03T10:29:00.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:00.852 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:30.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:30.923 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:29:39.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:29:39.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.997+0000 7f7b0de42700 1 Processor -- start 2023-03-03T10:29:39.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.997+0000 7f7b0de42700 1 -- start start 2023-03-03T10:29:40.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b0de42700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b080c5c60 0x7f7b08154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:40.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b0de42700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080c7310 con 0x7f7b080caa30 2023-03-03T10:29:40.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b0de42700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080bfd10 con 0x7f7b080c00e0 2023-03-03T10:29:40.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b0de42700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b080c60a0 con 0x7f7b080c5c60 2023-03-03T10:29:40.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b07fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b080c5c60 0x7f7b08154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:40.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b077fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7b080caa30 0x7f7b080c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54468/0 (socket says 172.21.15.7:54468) 2023-03-03T10:29:40.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b07fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b080c5c60 0x7f7b08154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60492/0 (socket says 172.21.15.7:60492) 2023-03-03T10:29:40.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b077fe700 1 -- 172.21.15.7:0/4080358630 learned_addr learned my addr 172.21.15.7:0/4080358630 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:29:40.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b07fff700 1 -- 172.21.15.7:0/4080358630 >> v1:172.21.15.143:6789/0 conn(0x7f7b080c00e0 legacy=0x7f7b0815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:40.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b07fff700 1 -- 172.21.15.7:0/4080358630 >> v1:172.21.15.7:6789/0 conn(0x7f7b080caa30 legacy=0x7f7b080c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:40.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b067fc700 1 -- 172.21.15.7:0/4080358630 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909838282 0 0) 0x7f7b080c7310 con 0x7f7b080caa30 2023-03-03T10:29:40.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b07fff700 1 -- 172.21.15.7:0/4080358630 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080bfa30 con 0x7f7b080c5c60 2023-03-03T10:29:40.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b07fff700 1 --2- 172.21.15.7:0/4080358630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c5c60 0x7f7b08154ff0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009d90 tx=0x7f7b0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd354830b55b648f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:29:40.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b067fc700 1 -- 172.21.15.7:0/4080358630 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0000e070 con 0x7f7b080c5c60 2023-03-03T10:29:40.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b067fc700 1 -- 172.21.15.7:0/4080358630 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0000bbd0 con 0x7f7b080c5c60 2023-03-03T10:29:40.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.998+0000 7f7b067fc700 1 -- 172.21.15.7:0/4080358630 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00012430 con 0x7f7b080c5c60 2023-03-03T10:29:40.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.999+0000 7f7b0de42700 1 -- 172.21.15.7:0/4080358630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c5c60 msgr2=0x7f7b08154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:40.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.999+0000 7f7b0de42700 1 --2- 172.21.15.7:0/4080358630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c5c60 0x7f7b08154ff0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009d90 tx=0x7f7b0000b330 comp rx=0 tx=0).stop 2023-03-03T10:29:40.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.999+0000 7f7b0de42700 1 -- 172.21.15.7:0/4080358630 shutdown_connections 2023-03-03T10:29:40.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.999+0000 7f7b0de42700 1 --2- 172.21.15.7:0/4080358630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c5c60 0x7f7b08154ff0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:40.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.999+0000 7f7b0de42700 1 -- 172.21.15.7:0/4080358630 >> 172.21.15.7:0/4080358630 conn(0x7f7b08066e40 msgr2=0x7f7b08071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:29:40.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.999+0000 7f7b0de42700 1 -- 172.21.15.7:0/4080358630 shutdown_connections 2023-03-03T10:29:40.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:39.999+0000 7f7b0de42700 1 -- 172.21.15.7:0/4080358630 wait complete. 2023-03-03T10:29:40.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.000+0000 7f7b0de42700 1 Processor -- start 2023-03-03T10:29:40.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.000+0000 7f7b0de42700 1 -- start start 2023-03-03T10:29:40.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b0de42700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c00e0 0x7f7b080c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:40.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b0de42700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b080c5c60 0x7f7b08156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:40.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b0de42700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b080caa30 0x7f7b0814c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:40.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b0de42700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080c3570 con 0x7f7b080c00e0 2023-03-03T10:29:40.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b0de42700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b081553f0 con 0x7f7b080caa30 2023-03-03T10:29:40.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b0de42700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b081556d0 con 0x7f7b080c5c60 2023-03-03T10:29:40.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b077fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c00e0 0x7f7b080c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:40.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b077fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c00e0 0x7f7b080c3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60494/0 (socket says 172.21.15.7:60494) 2023-03-03T10:29:40.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b077fe700 1 -- 172.21.15.7:0/31482909 learned_addr learned my addr 172.21.15.7:0/31482909 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:29:40.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b06ffd700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b080c5c60 0x7f7b08156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:40.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.001+0000 7f7b07fff700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b080caa30 0x7f7b0814c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:40.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7b077fe700 1 -- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b080c5c60 msgr2=0x7f7b08156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:40.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7b077fe700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b080c5c60 0x7f7b08156570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:40.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7b077fe700 1 -- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b080caa30 msgr2=0x7f7b0814c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:40.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7b077fe700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b080caa30 0x7f7b0814c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:40.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7b077fe700 1 -- 172.21.15.7:0/31482909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00009a70 con 0x7f7b080c00e0 2023-03-03T10:29:40.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7b077fe700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c00e0 0x7f7b080c3fc0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00c520 tx=0x7f7afc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:29:40.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7aeffff700 1 -- 172.21.15.7:0/31482909 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc004c80 con 0x7f7b080c00e0 2023-03-03T10:29:40.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7aeffff700 1 -- 172.21.15.7:0/31482909 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc004de0 con 0x7f7b080c00e0 2023-03-03T10:29:40.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7b0de42700 1 -- 172.21.15.7:0/31482909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b081547f0 con 0x7f7b080c00e0 2023-03-03T10:29:40.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7aeffff700 1 -- 172.21.15.7:0/31482909 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc007310 con 0x7f7b080c00e0 2023-03-03T10:29:40.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.002+0000 7f7b0de42700 1 -- 172.21.15.7:0/31482909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b08154510 con 0x7f7b080c00e0 2023-03-03T10:29:40.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.003+0000 7f7b0de42700 1 -- 172.21.15.7:0/31482909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b08154ad0 con 0x7f7b080c00e0 2023-03-03T10:29:40.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.003+0000 7f7aeffff700 1 -- 172.21.15.7:0/31482909 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7afc01a020 con 0x7f7b080c00e0 2023-03-03T10:29:40.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.004+0000 7f7aeffff700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af40682f0 0x7f7af406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:40.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.004+0000 7f7aeffff700 1 -- 172.21.15.7:0/31482909 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7afc085cc0 con 0x7f7b080c00e0 2023-03-03T10:29:40.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.004+0000 7f7b06ffd700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af40682f0 0x7f7af406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:40.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.005+0000 7f7b06ffd700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af40682f0 0x7f7af406a7a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7b081ee1a0 tx=0x7f7af8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:29:40.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.008+0000 7f7aeffff700 1 -- 172.21.15.7:0/31482909 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7afc08c080 con 0x7f7b080c00e0 2023-03-03T10:29:40.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.246+0000 7f7b0de42700 1 -- 172.21.15.7:0/31482909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b08155890 con 0x7f7b080c00e0 2023-03-03T10:29:40.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.247+0000 7f7aeffff700 1 -- 172.21.15.7:0/31482909 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7afc051f70 con 0x7f7b080c00e0 2023-03-03T10:29:40.249 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:29:40.249 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:29:40.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 -- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af40682f0 msgr2=0x7f7af406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:40.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af40682f0 0x7f7af406a7a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7b081ee1a0 tx=0x7f7af8009040 comp rx=0 tx=0).stop 2023-03-03T10:29:40.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 -- 172.21.15.7:0/31482909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c00e0 msgr2=0x7f7b080c3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c00e0 0x7f7b080c3fc0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00c520 tx=0x7f7afc00c830 comp rx=0 tx=0).stop 2023-03-03T10:29:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 -- 172.21.15.7:0/31482909 shutdown_connections 2023-03-03T10:29:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b080caa30 0x7f7b0814c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af40682f0 0x7f7af406a7a0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b080c00e0 0x7f7b080c3fc0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 --2- 172.21.15.7:0/31482909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b080c5c60 0x7f7b08156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:40.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 -- 172.21.15.7:0/31482909 >> 172.21.15.7:0/31482909 conn(0x7f7b08066e40 msgr2=0x7f7b08065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:29:40.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 -- 172.21.15.7:0/31482909 shutdown_connections 2023-03-03T10:29:40.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:40.249+0000 7f7b0de42700 1 -- 172.21.15.7:0/31482909 wait complete. 2023-03-03T10:29:40.253 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:29:40.263 INFO:tasks.mds_thrash.fs.[b]:waiting for 47 secs before thrashing 2023-03-03T10:29:54.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:29:54.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.625+0000 7f2d81724700 1 Processor -- start 2023-03-03T10:29:54.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.625+0000 7f2d81724700 1 -- start start 2023-03-03T10:29:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.626+0000 7f2d81724700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0c00e0 0x7f2d7c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.626+0000 7f2d81724700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7c0c7310 con 0x7f2d7c0caa30 2023-03-03T10:29:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.626+0000 7f2d81724700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7c0bfd10 con 0x7f2d7c154ff0 2023-03-03T10:29:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.626+0000 7f2d81724700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0bfa30 con 0x7f2d7c0c00e0 2023-03-03T10:29:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.626+0000 7f2d7affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2d7c0caa30 0x7f2d7c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54484/0 (socket says 172.21.15.7:54484) 2023-03-03T10:29:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.626+0000 7f2d7affd700 1 -- 172.21.15.7:0/404345760 learned_addr learned my addr 172.21.15.7:0/404345760 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:29:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.626+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746472265 0 0) 0x7f2d7c0c7310 con 0x7f2d7c0caa30 2023-03-03T10:29:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.626+0000 7f2d7a7fc700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0c00e0 0x7f2d7c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.626+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d640035b0 con 0x7f2d7c0caa30 2023-03-03T10:29:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.626+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674535093 0 0) 0x7f2d640035b0 con 0x7f2d7c0caa30 2023-03-03T10:29:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0c00e0 msgr2=0x7f2d7c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:29:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0c00e0 0x7f2d7c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 >> v1:172.21.15.143:6789/0 conn(0x7f2d7c154ff0 legacy=0x7f2d7c15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c154510 con 0x7f2d7c0caa30 2023-03-03T10:29:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f2d70001670 con 0x7f2d7c0caa30 2023-03-03T10:29:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 >> v1:172.21.15.7:6789/0 conn(0x7f2d7c0caa30 legacy=0x7f2d7c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d64003aa0 0x7f2d64003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d64004850 0x7f2d64004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0c00e0 0x7f2d64005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0bfa30 con 0x7f2d64004850 2023-03-03T10:29:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6400baf0 con 0x7f2d64003aa0 2023-03-03T10:29:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6400bcb0 con 0x7f2d7c0c00e0 2023-03-03T10:29:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d7a7fc700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d64004850 0x7f2d64004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d7affd700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d64003aa0 0x7f2d64003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d7b7fe700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0c00e0 0x7f2d64005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d7a7fc700 1 -- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0c00e0 msgr2=0x7f2d64005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:29:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d7a7fc700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0c00e0 0x7f2d64005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d7a7fc700 1 -- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d64003aa0 msgr2=0x7f2d64003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:29:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d7a7fc700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d64003aa0 0x7f2d64003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d7a7fc700 1 -- 172.21.15.7:0/404345760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d640035b0 con 0x7f2d64004850 2023-03-03T10:29:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d7a7fc700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d64004850 0x7f2d64004cc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009b40 tx=0x7f2d64005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528df485012a18b9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:29:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c005a20 con 0x7f2d64004850 2023-03-03T10:29:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c005b80 con 0x7f2d64004850 2023-03-03T10:29:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.627+0000 7f2d79ffb700 1 -- 172.21.15.7:0/404345760 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c00e6c0 con 0x7f2d64004850 2023-03-03T10:29:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.628+0000 7f2d81724700 1 -- 172.21.15.7:0/404345760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d64004850 msgr2=0x7f2d64004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.628+0000 7f2d81724700 1 --2- 172.21.15.7:0/404345760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d64004850 0x7f2d64004cc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009b40 tx=0x7f2d64005880 comp rx=0 tx=0).stop 2023-03-03T10:29:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.628+0000 7f2d7b7fe700 1 -- 172.21.15.7:0/404345760 reap_dead start 2023-03-03T10:29:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.628+0000 7f2d81724700 1 -- 172.21.15.7:0/404345760 shutdown_connections 2023-03-03T10:29:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.628+0000 7f2d81724700 1 -- 172.21.15.7:0/404345760 >> 172.21.15.7:0/404345760 conn(0x7f2d7c066e40 msgr2=0x7f2d7c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:29:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.628+0000 7f2d81724700 1 -- 172.21.15.7:0/404345760 shutdown_connections 2023-03-03T10:29:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.628+0000 7f2d81724700 1 -- 172.21.15.7:0/404345760 wait complete. 2023-03-03T10:29:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.629+0000 7f2d81724700 1 Processor -- start 2023-03-03T10:29:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.629+0000 7f2d81724700 1 -- start start 2023-03-03T10:29:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d81724700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d64004850 0x7f2d7c14d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d81724700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c1df140 0x7f2d7c157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d81724700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c152de0 0x7f2d7c156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d81724700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c14da70 con 0x7f2d7c152de0 2023-03-03T10:29:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d81724700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c157b50 con 0x7f2d64004850 2023-03-03T10:29:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d81724700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c1de0f0 con 0x7f2d7c1df140 2023-03-03T10:29:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d7b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c152de0 0x7f2d7c156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d7b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c152de0 0x7f2d7c156470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60520/0 (socket says 172.21.15.7:60520) 2023-03-03T10:29:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d7b7fe700 1 -- 172.21.15.7:0/3617481453 learned_addr learned my addr 172.21.15.7:0/3617481453 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:29:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d7a7fc700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c1df140 0x7f2d7c157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.630+0000 7f2d7affd700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d64004850 0x7f2d7c14d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d7b7fe700 1 -- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c1df140 msgr2=0x7f2d7c157610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d7b7fe700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c1df140 0x7f2d7c157610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d7b7fe700 1 -- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d64004850 msgr2=0x7f2d7c14d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d7b7fe700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d64004850 0x7f2d7c14d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d7b7fe700 1 -- 172.21.15.7:0/3617481453 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c154ad0 con 0x7f2d7c152de0 2023-03-03T10:29:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d7a7fc700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c1df140 0x7f2d7c157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:29:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d7b7fe700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c152de0 0x7f2d7c156470 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0c0db0 tx=0x7f2d7400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:29:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d637fe700 1 -- 172.21.15.7:0/3617481453 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74003c10 con 0x7f2d7c152de0 2023-03-03T10:29:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d637fe700 1 -- 172.21.15.7:0/3617481453 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d74003d70 con 0x7f2d7c152de0 2023-03-03T10:29:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d637fe700 1 -- 172.21.15.7:0/3617481453 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74005310 con 0x7f2d7c152de0 2023-03-03T10:29:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d81724700 1 -- 172.21.15.7:0/3617481453 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c1547f0 con 0x7f2d7c152de0 2023-03-03T10:29:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.631+0000 7f2d81724700 1 -- 172.21.15.7:0/3617481453 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c0bf650 con 0x7f2d7c152de0 2023-03-03T10:29:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.632+0000 7f2d81724700 1 -- 172.21.15.7:0/3617481453 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c074bb0 con 0x7f2d7c152de0 2023-03-03T10:29:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.632+0000 7f2d637fe700 1 -- 172.21.15.7:0/3617481453 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2d74005470 con 0x7f2d7c152de0 2023-03-03T10:29:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.633+0000 7f2d637fe700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d680682f0 0x7f2d6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:29:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.633+0000 7f2d637fe700 1 -- 172.21.15.7:0/3617481453 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2d74086310 con 0x7f2d7c152de0 2023-03-03T10:29:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.633+0000 7f2d7affd700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d680682f0 0x7f2d6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:29:54.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.634+0000 7f2d7affd700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d680682f0 0x7f2d6806a7a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2d700010e0 tx=0x7f2d70001c40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:29:54.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.637+0000 7f2d637fe700 1 -- 172.21.15.7:0/3617481453 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2d740526b0 con 0x7f2d7c152de0 2023-03-03T10:29:54.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.881+0000 7f2d81724700 1 -- 172.21.15.7:0/3617481453 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d7c0c5030 con 0x7f2d7c152de0 2023-03-03T10:29:54.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.882+0000 7f2d637fe700 1 -- 172.21.15.7:0/3617481453 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2d74055cd0 con 0x7f2d7c152de0 2023-03-03T10:29:54.885 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:29:54.885 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:29:54.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 -- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d680682f0 msgr2=0x7f2d6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d680682f0 0x7f2d6806a7a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2d700010e0 tx=0x7f2d70001c40 comp rx=0 tx=0).stop 2023-03-03T10:29:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 -- 172.21.15.7:0/3617481453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c152de0 msgr2=0x7f2d7c156470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:29:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c152de0 0x7f2d7c156470 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0c0db0 tx=0x7f2d7400ee50 comp rx=0 tx=0).stop 2023-03-03T10:29:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 -- 172.21.15.7:0/3617481453 shutdown_connections 2023-03-03T10:29:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d680682f0 0x7f2d6806a7a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:54.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c152de0 0x7f2d7c156470 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d64004850 0x7f2d7c14d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 --2- 172.21.15.7:0/3617481453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c1df140 0x7f2d7c157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:29:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 -- 172.21.15.7:0/3617481453 >> 172.21.15.7:0/3617481453 conn(0x7f2d7c066e40 msgr2=0x7f2d7c1512a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:29:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 -- 172.21.15.7:0/3617481453 shutdown_connections 2023-03-03T10:29:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:29:54.884+0000 7f2d81724700 1 -- 172.21.15.7:0/3617481453 wait complete. 2023-03-03T10:29:54.888 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:29:54.899 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T10:30:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T10:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T10:30:01.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:01.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:27.264 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:30:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.400+0000 7fca30778700 1 Processor -- start 2023-03-03T10:30:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.400+0000 7fca30778700 1 -- start start 2023-03-03T10:30:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.401+0000 7fca30778700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2814f9d0 0x7fca28156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.401+0000 7fca30778700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca280c2f30 con 0x7fca280abe20 2023-03-03T10:30:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.401+0000 7fca30778700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca280c31d0 con 0x7fca2814fda0 2023-03-03T10:30:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.401+0000 7fca30778700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca281524a0 con 0x7fca2814f9d0 2023-03-03T10:30:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.401+0000 7fca2ed15700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fca280abe20 0x7fca281529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54522/0 (socket says 172.21.15.7:54522) 2023-03-03T10:30:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.401+0000 7fca2ed15700 1 -- 172.21.15.7:0/421778718 learned_addr learned my addr 172.21.15.7:0/421778718 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:30:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.401+0000 7fca2e514700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2814f9d0 0x7fca28156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.401+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693583168 0 0) 0x7fca280c2f30 con 0x7fca280abe20 2023-03-03T10:30:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.401+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca140035b0 con 0x7fca280abe20 2023-03-03T10:30:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667229929 0 0) 0x7fca280c31d0 con 0x7fca2814fda0 2023-03-03T10:30:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca280c2f30 con 0x7fca2814fda0 2023-03-03T10:30:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 967752907 0 0) 0x7fca140035b0 con 0x7fca280abe20 2023-03-03T10:30:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2814f9d0 msgr2=0x7fca28156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:30:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2814f9d0 0x7fca28156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 >> v1:172.21.15.143:6789/0 conn(0x7fca2814fda0 legacy=0x7fca28071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca280c34b0 con 0x7fca280abe20 2023-03-03T10:30:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fca200019f0 con 0x7fca280abe20 2023-03-03T10:30:27.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2e514700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2814f9d0 0x7fca28156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:30:27.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 >> v1:172.21.15.7:6789/0 conn(0x7fca280abe20 legacy=0x7fca281529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:27.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2814f9d0 0x7fca14003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:27.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca140041c0 0x7fca14004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:27.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca14004d90 0x7fca14005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:27.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca281524a0 con 0x7fca14004d90 2023-03-03T10:30:27.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1400bca0 con 0x7fca140041c0 2023-03-03T10:30:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1400be60 con 0x7fca2814f9d0 2023-03-03T10:30:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2ed15700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca14004d90 0x7fca14005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2e514700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2814f9d0 0x7fca14003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2dd13700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca140041c0 0x7fca14004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2ed15700 1 -- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2814f9d0 msgr2=0x7fca14003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:30:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2ed15700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2814f9d0 0x7fca14003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2ed15700 1 -- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca140041c0 msgr2=0x7fca14004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:30:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2ed15700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca140041c0 0x7fca14004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2ed15700 1 -- 172.21.15.7:0/421778718 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca180094a0 con 0x7fca14004d90 2023-03-03T10:30:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.402+0000 7fca2ed15700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca14004d90 0x7fca14005200 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fca20002230 tx=0x7fca14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b815e7486f9479 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:30:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.403+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca200072d0 con 0x7fca14004d90 2023-03-03T10:30:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.403+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca20007430 con 0x7fca14004d90 2023-03-03T10:30:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.403+0000 7fca2d512700 1 -- 172.21.15.7:0/421778718 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca200077e0 con 0x7fca14004d90 2023-03-03T10:30:27.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.403+0000 7fca30778700 1 -- 172.21.15.7:0/421778718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca14004d90 msgr2=0x7fca14005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:27.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.403+0000 7fca30778700 1 --2- 172.21.15.7:0/421778718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca14004d90 0x7fca14005200 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fca20002230 tx=0x7fca14005880 comp rx=0 tx=0).stop 2023-03-03T10:30:27.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.403+0000 7fca2ed15700 1 -- 172.21.15.7:0/421778718 reap_dead start 2023-03-03T10:30:27.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.403+0000 7fca30778700 1 -- 172.21.15.7:0/421778718 shutdown_connections 2023-03-03T10:30:27.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.403+0000 7fca30778700 1 -- 172.21.15.7:0/421778718 >> 172.21.15.7:0/421778718 conn(0x7fca280c8900 msgr2=0x7fca280c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:30:27.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.403+0000 7fca30778700 1 -- 172.21.15.7:0/421778718 shutdown_connections 2023-03-03T10:30:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.403+0000 7fca30778700 1 -- 172.21.15.7:0/421778718 wait complete. 2023-03-03T10:30:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.404+0000 7fca30778700 1 Processor -- start 2023-03-03T10:30:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.404+0000 7fca30778700 1 -- start start 2023-03-03T10:30:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca30778700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca14004d90 0x7fca280a9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca30778700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca28154040 0x7fca281544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca30778700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca281549f0 0x7fca281de380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca30778700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2814d8c0 con 0x7fca281549f0 2023-03-03T10:30:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca30778700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca281eb800 con 0x7fca14004d90 2023-03-03T10:30:27.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca30778700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca281eac80 con 0x7fca28154040 2023-03-03T10:30:27.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca2ed15700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca281549f0 0x7fca281de380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:27.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca2ed15700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca281549f0 0x7fca281de380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60554/0 (socket says 172.21.15.7:60554) 2023-03-03T10:30:27.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca2ed15700 1 -- 172.21.15.7:0/1140149644 learned_addr learned my addr 172.21.15.7:0/1140149644 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:30:27.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca2dd13700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca28154040 0x7fca281544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:27.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.405+0000 7fca2e514700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca14004d90 0x7fca280a9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:27.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca2ed15700 1 -- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca28154040 msgr2=0x7fca281544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:27.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca2ed15700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca28154040 0x7fca281544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:27.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca2ed15700 1 -- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca14004d90 msgr2=0x7fca280a9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:27.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca2ed15700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca14004d90 0x7fca280a9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:27.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca2ed15700 1 -- 172.21.15.7:0/1140149644 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca280ac350 con 0x7fca281549f0 2023-03-03T10:30:27.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca2ed15700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca281549f0 0x7fca281de380 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fca20011000 tx=0x7fca20001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:30:27.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca1effd700 1 -- 172.21.15.7:0/1140149644 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca20025070 con 0x7fca281549f0 2023-03-03T10:30:27.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca1effd700 1 -- 172.21.15.7:0/1140149644 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca20007b90 con 0x7fca281549f0 2023-03-03T10:30:27.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca30778700 1 -- 172.21.15.7:0/1140149644 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca280ab630 con 0x7fca281549f0 2023-03-03T10:30:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca1effd700 1 -- 172.21.15.7:0/1140149644 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca20012640 con 0x7fca281549f0 2023-03-03T10:30:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.406+0000 7fca30778700 1 -- 172.21.15.7:0/1140149644 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca28071fa0 con 0x7fca281549f0 2023-03-03T10:30:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.407+0000 7fca30778700 1 -- 172.21.15.7:0/1140149644 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca28074bb0 con 0x7fca281549f0 2023-03-03T10:30:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.407+0000 7fca1effd700 1 -- 172.21.15.7:0/1140149644 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fca20010070 con 0x7fca281549f0 2023-03-03T10:30:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.408+0000 7fca1effd700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca0c0682f0 0x7fca0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.408+0000 7fca1effd700 1 -- 172.21.15.7:0/1140149644 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fca20087a30 con 0x7fca281549f0 2023-03-03T10:30:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.408+0000 7fca2e514700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca0c0682f0 0x7fca0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.409+0000 7fca2e514700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca0c0682f0 0x7fca0c06a7a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fca2814f980 tx=0x7fca18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:30:27.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.412+0000 7fca1effd700 1 -- 172.21.15.7:0/1140149644 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fca20053dd0 con 0x7fca281549f0 2023-03-03T10:30:27.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.652+0000 7fca30778700 1 -- 172.21.15.7:0/1140149644 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca28153d60 con 0x7fca281549f0 2023-03-03T10:30:27.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.652+0000 7fca1effd700 1 -- 172.21.15.7:0/1140149644 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fca28153d60 con 0x7fca281549f0 2023-03-03T10:30:27.655 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:30:27.655 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:30:27.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.654+0000 7fca30778700 1 -- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca0c0682f0 msgr2=0x7fca0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:27.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.654+0000 7fca30778700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca0c0682f0 0x7fca0c06a7a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fca2814f980 tx=0x7fca18008040 comp rx=0 tx=0).stop 2023-03-03T10:30:27.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.654+0000 7fca30778700 1 -- 172.21.15.7:0/1140149644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca281549f0 msgr2=0x7fca281de380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:27.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.654+0000 7fca30778700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca281549f0 0x7fca281de380 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fca20011000 tx=0x7fca20001cb0 comp rx=0 tx=0).stop 2023-03-03T10:30:27.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.655+0000 7fca30778700 1 -- 172.21.15.7:0/1140149644 shutdown_connections 2023-03-03T10:30:27.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.655+0000 7fca30778700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca0c0682f0 0x7fca0c06a7a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:27.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.655+0000 7fca30778700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca281549f0 0x7fca281de380 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:27.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.655+0000 7fca30778700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca14004d90 0x7fca280a9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:27.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.655+0000 7fca30778700 1 --2- 172.21.15.7:0/1140149644 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca28154040 0x7fca281544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:27.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.655+0000 7fca30778700 1 -- 172.21.15.7:0/1140149644 >> 172.21.15.7:0/1140149644 conn(0x7fca280c8900 msgr2=0x7fca2814e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:30:27.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.655+0000 7fca30778700 1 -- 172.21.15.7:0/1140149644 shutdown_connections 2023-03-03T10:30:27.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:27.655+0000 7fca30778700 1 -- 172.21.15.7:0/1140149644 wait complete. 2023-03-03T10:30:27.659 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:30:27.669 INFO:tasks.mds_thrash.fs.[b]:waiting for 13 secs before thrashing 2023-03-03T10:30:31.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:31.140 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:30:40.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:30:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.805+0000 7fc33c8f8700 1 Processor -- start 2023-03-03T10:30:40.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.805+0000 7fc33c8f8700 1 -- start start 2023-03-03T10:30:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.806+0000 7fc33c8f8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc338071e80 0x7fc338151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.806+0000 7fc33c8f8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc33814e900 0x7fc33815ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:40.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.806+0000 7fc33c8f8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3381510d0 con 0x7fc338072250 2023-03-03T10:30:40.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.806+0000 7fc33c8f8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3380c3740 con 0x7fc33814e900 2023-03-03T10:30:40.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.806+0000 7fc33c8f8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc338070670 con 0x7fc338071e80 2023-03-03T10:30:40.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.806+0000 7fc335d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc338072250 0x7fc33814e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54544/0 (socket says 172.21.15.7:54544) 2023-03-03T10:30:40.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.806+0000 7fc335d9b700 1 -- 172.21.15.7:0/518729970 learned_addr learned my addr 172.21.15.7:0/518729970 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:30:40.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.806+0000 7fc336d9d700 1 --2- 172.21.15.7:0/518729970 >> v2:172.21.15.7:3300/0 conn(0x7fc33814e900 0x7fc33815ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:40.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc33659c700 1 --2- 172.21.15.7:0/518729970 >> v2:172.21.15.143:3300/0 conn(0x7fc338071e80 0x7fc338151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:40.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc33559a700 1 -- 172.21.15.7:0/518729970 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323895625 0 0) 0x7fc3381510d0 con 0x7fc338072250 2023-03-03T10:30:40.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc33559a700 1 -- 172.21.15.7:0/518729970 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3200035b0 con 0x7fc338072250 2023-03-03T10:30:40.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc336d9d700 1 -- 172.21.15.7:0/518729970 >> v2:172.21.15.143:3300/0 conn(0x7fc338071e80 msgr2=0x7fc338151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:30:40.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc336d9d700 1 --2- 172.21.15.7:0/518729970 >> v2:172.21.15.143:3300/0 conn(0x7fc338071e80 0x7fc338151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:40.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc336d9d700 1 -- 172.21.15.7:0/518729970 >> v1:172.21.15.7:6789/0 conn(0x7fc338072250 legacy=0x7fc33814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:40.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc336d9d700 1 -- 172.21.15.7:0/518729970 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3380c39e0 con 0x7fc33814e900 2023-03-03T10:30:40.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc336d9d700 1 --2- 172.21.15.7:0/518729970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc33814e900 0x7fc33815ba50 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc33806bd10 tx=0x7fc33000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a64306965d744d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:30:40.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc33559a700 1 -- 172.21.15.7:0/518729970 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33000e070 con 0x7fc33814e900 2023-03-03T10:30:40.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc33559a700 1 -- 172.21.15.7:0/518729970 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33000bb20 con 0x7fc33814e900 2023-03-03T10:30:40.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.807+0000 7fc33559a700 1 -- 172.21.15.7:0/518729970 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330012430 con 0x7fc33814e900 2023-03-03T10:30:40.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.808+0000 7fc33c8f8700 1 -- 172.21.15.7:0/518729970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc33814e900 msgr2=0x7fc33815ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:40.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.808+0000 7fc33c8f8700 1 --2- 172.21.15.7:0/518729970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc33814e900 0x7fc33815ba50 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc33806bd10 tx=0x7fc33000b330 comp rx=0 tx=0).stop 2023-03-03T10:30:40.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.808+0000 7fc33c8f8700 1 -- 172.21.15.7:0/518729970 shutdown_connections 2023-03-03T10:30:40.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.808+0000 7fc33c8f8700 1 --2- 172.21.15.7:0/518729970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc33814e900 0x7fc33815ba50 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:40.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.808+0000 7fc33c8f8700 1 --2- 172.21.15.7:0/518729970 >> v2:172.21.15.143:3300/0 conn(0x7fc338071e80 0x7fc338151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:40.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.808+0000 7fc33c8f8700 1 -- 172.21.15.7:0/518729970 >> 172.21.15.7:0/518729970 conn(0x7fc3380a94d0 msgr2=0x7fc3380a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:30:40.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.808+0000 7fc33c8f8700 1 -- 172.21.15.7:0/518729970 shutdown_connections 2023-03-03T10:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.808+0000 7fc33c8f8700 1 -- 172.21.15.7:0/518729970 wait complete. 2023-03-03T10:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.810+0000 7fc33c8f8700 1 Processor -- start 2023-03-03T10:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.810+0000 7fc33c8f8700 1 -- start start 2023-03-03T10:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.810+0000 7fc33c8f8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc338071e80 0x7fc3381ea090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.810+0000 7fc33c8f8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc338072250 0x7fc338149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc33c8f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc33814e900 0x7fc33814a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc33c8f8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381eaaa0 con 0x7fc338072250 2023-03-03T10:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc33c8f8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3380be430 con 0x7fc338071e80 2023-03-03T10:30:40.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc33c8f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381437f0 con 0x7fc33814e900 2023-03-03T10:30:40.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc335d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc338072250 0x7fc338149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:40.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc335d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc338072250 0x7fc338149c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60572/0 (socket says 172.21.15.7:60572) 2023-03-03T10:30:40.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc335d9b700 1 -- 172.21.15.7:0/673498161 learned_addr learned my addr 172.21.15.7:0/673498161 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:30:40.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc33659c700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc338071e80 0x7fc3381ea090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:40.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc336d9d700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc33814e900 0x7fc33814a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:40.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc335d9b700 1 -- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc33814e900 msgr2=0x7fc33814a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:40.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc335d9b700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc33814e900 0x7fc33814a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:40.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc335d9b700 1 -- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc338071e80 msgr2=0x7fc3381ea090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:40.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc335d9b700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc338071e80 0x7fc3381ea090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:40.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc335d9b700 1 -- 172.21.15.7:0/673498161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc330009a70 con 0x7fc338072250 2023-03-03T10:30:40.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc335d9b700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc338072250 0x7fc338149c90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc328005910 tx=0x7fc32800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:30:40.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc326ffd700 1 -- 172.21.15.7:0/673498161 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc328005b40 con 0x7fc338072250 2023-03-03T10:30:40.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc326ffd700 1 -- 172.21.15.7:0/673498161 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc328005ca0 con 0x7fc338072250 2023-03-03T10:30:40.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc33c8f8700 1 -- 172.21.15.7:0/673498161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33806c9a0 con 0x7fc338072250 2023-03-03T10:30:40.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc326ffd700 1 -- 172.21.15.7:0/673498161 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32800ca00 con 0x7fc338072250 2023-03-03T10:30:40.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.811+0000 7fc33c8f8700 1 -- 172.21.15.7:0/673498161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc338151300 con 0x7fc338072250 2023-03-03T10:30:40.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.812+0000 7fc33c8f8700 1 -- 172.21.15.7:0/673498161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33806cb30 con 0x7fc338072250 2023-03-03T10:30:40.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.813+0000 7fc326ffd700 1 -- 172.21.15.7:0/673498161 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc328021070 con 0x7fc338072250 2023-03-03T10:30:40.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.813+0000 7fc326ffd700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3180682f0 0x7fc31806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:30:40.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.813+0000 7fc326ffd700 1 -- 172.21.15.7:0/673498161 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc328088540 con 0x7fc338072250 2023-03-03T10:30:40.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.814+0000 7fc33659c700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3180682f0 0x7fc31806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:30:40.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.814+0000 7fc33659c700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3180682f0 0x7fc31806a7a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc3380c14c0 tx=0x7fc32c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:30:40.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:40.818+0000 7fc326ffd700 1 -- 172.21.15.7:0/673498161 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc3280547f0 con 0x7fc338072250 2023-03-03T10:30:41.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.052+0000 7fc33c8f8700 1 -- 172.21.15.7:0/673498161 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3380be8c0 con 0x7fc338072250 2023-03-03T10:30:41.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.052+0000 7fc326ffd700 1 -- 172.21.15.7:0/673498161 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc3380be8c0 con 0x7fc338072250 2023-03-03T10:30:41.056 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:30:41.056 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:30:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 -- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3180682f0 msgr2=0x7fc31806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3180682f0 0x7fc31806a7a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc3380c14c0 tx=0x7fc32c008040 comp rx=0 tx=0).stop 2023-03-03T10:30:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 -- 172.21.15.7:0/673498161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc338072250 msgr2=0x7fc338149c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:30:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc338072250 0x7fc338149c90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc328005910 tx=0x7fc32800eca0 comp rx=0 tx=0).stop 2023-03-03T10:30:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 -- 172.21.15.7:0/673498161 shutdown_connections 2023-03-03T10:30:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc33814e900 0x7fc33814a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:41.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc338071e80 0x7fc3381ea090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3180682f0 0x7fc31806a7a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 --2- 172.21.15.7:0/673498161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc338072250 0x7fc338149c90 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:30:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 -- 172.21.15.7:0/673498161 >> 172.21.15.7:0/673498161 conn(0x7fc3380a94d0 msgr2=0x7fc33806a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:30:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 -- 172.21.15.7:0/673498161 shutdown_connections 2023-03-03T10:30:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:30:41.054+0000 7fc33c8f8700 1 -- 172.21.15.7:0/673498161 wait complete. 2023-03-03T10:30:41.059 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:30:41.069 INFO:tasks.mds_thrash.fs.[b]:waiting for 35 secs before thrashing 2023-03-03T10:30:59.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:31:00.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.037+0000 7f74c9705700 1 Processor -- start 2023-03-03T10:31:00.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.037+0000 7f74c9705700 1 -- start start 2023-03-03T10:31:00.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.038+0000 7f74c9705700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c40c43d0 0x7f74c40c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.038+0000 7f74c9705700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c4072920 con 0x7f74c414e560 2023-03-03T10:31:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.038+0000 7f74c9705700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c406bef0 con 0x7f74c4154590 2023-03-03T10:31:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.038+0000 7f74c9705700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c4072380 con 0x7f74c40c43d0 2023-03-03T10:31:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.038+0000 7f74c2ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f74c414e560 0x7f74c4071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54564/0 (socket says 172.21.15.7:54564) 2023-03-03T10:31:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.038+0000 7f74c2ffd700 1 -- 172.21.15.7:0/3747535240 learned_addr learned my addr 172.21.15.7:0/3747535240 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:31:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.038+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547187580 0 0) 0x7f74ac001110 con 0x7f74c414e560 2023-03-03T10:31:00.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.038+0000 7f74c27fc700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c40c43d0 0x7f74c40c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74b00035b0 con 0x7f74c414e560 2023-03-03T10:31:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3076309401 0 0) 0x7f74b00035b0 con 0x7f74c414e560 2023-03-03T10:31:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c40c43d0 msgr2=0x7f74c40c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:31:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c40c43d0 0x7f74c40c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 >> v1:172.21.15.143:6789/0 conn(0x7f74c4154590 legacy=0x7f74c41549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c406b910 con 0x7f74c414e560 2023-03-03T10:31:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f74ac001800 con 0x7f74c414e560 2023-03-03T10:31:00.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 >> v1:172.21.15.7:6789/0 conn(0x7f74c414e560 legacy=0x7f74c4071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74b0003aa0 0x7f74b0003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c40c43d0 0x7f74b0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74b0004d90 0x7f74b0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c4072380 con 0x7f74b0003aa0 2023-03-03T10:31:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b000baf0 con 0x7f74b0004d90 2023-03-03T10:31:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b000bcb0 con 0x7f74c40c43d0 2023-03-03T10:31:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c2ffd700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74b0003aa0 0x7f74b0003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c37fe700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74b0004d90 0x7f74b0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c27fc700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c40c43d0 0x7f74b0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c2ffd700 1 -- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c40c43d0 msgr2=0x7f74b0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:31:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c2ffd700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c40c43d0 0x7f74b0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c2ffd700 1 -- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74b0004d90 msgr2=0x7f74b0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:31:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c2ffd700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74b0004d90 0x7f74b0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.039+0000 7f74c2ffd700 1 -- 172.21.15.7:0/3747535240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b00035b0 con 0x7f74b0003aa0 2023-03-03T10:31:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c2ffd700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74b0003aa0 0x7f74b0003eb0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00c240 tx=0x7f74b0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f25feb5d25e9e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:31:00.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac007d80 con 0x7f74b0003aa0 2023-03-03T10:31:00.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac00cda0 con 0x7f74b0003aa0 2023-03-03T10:31:00.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c1ffb700 1 -- 172.21.15.7:0/3747535240 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac005b80 con 0x7f74b0003aa0 2023-03-03T10:31:00.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c9705700 1 -- 172.21.15.7:0/3747535240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74b0003aa0 msgr2=0x7f74b0003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:00.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c9705700 1 --2- 172.21.15.7:0/3747535240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74b0003aa0 0x7f74b0003eb0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00c240 tx=0x7f74b0005880 comp rx=0 tx=0).stop 2023-03-03T10:31:00.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c37fe700 1 -- 172.21.15.7:0/3747535240 reap_dead start 2023-03-03T10:31:00.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c9705700 1 -- 172.21.15.7:0/3747535240 shutdown_connections 2023-03-03T10:31:00.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c9705700 1 -- 172.21.15.7:0/3747535240 >> 172.21.15.7:0/3747535240 conn(0x7f74c40736f0 msgr2=0x7f74c4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:31:00.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c9705700 1 -- 172.21.15.7:0/3747535240 shutdown_connections 2023-03-03T10:31:00.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.040+0000 7f74c9705700 1 -- 172.21.15.7:0/3747535240 wait complete. 2023-03-03T10:31:00.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.042+0000 7f74c9705700 1 Processor -- start 2023-03-03T10:31:00.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.042+0000 7f74c9705700 1 -- start start 2023-03-03T10:31:00.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.042+0000 7f74c9705700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74b0003aa0 0x7f74c40ac4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:00.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.042+0000 7f74c9705700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74c40bf670 0x7f74c4156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:00.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.042+0000 7f74c9705700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c41ea7a0 0x7f74c40c1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:00.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.042+0000 7f74c9705700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40bfc40 con 0x7f74c40bf670 2023-03-03T10:31:00.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.042+0000 7f74c9705700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c40c1800 con 0x7f74b0003aa0 2023-03-03T10:31:00.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.042+0000 7f74c9705700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c41ead10 con 0x7f74c41ea7a0 2023-03-03T10:31:00.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.042+0000 7f74c27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74c40bf670 0x7f74c4156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:00.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74c40bf670 0x7f74c4156050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60600/0 (socket says 172.21.15.7:60600) 2023-03-03T10:31:00.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c27fc700 1 -- 172.21.15.7:0/2996329738 learned_addr learned my addr 172.21.15.7:0/2996329738 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:31:00.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c2ffd700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74b0003aa0 0x7f74c40ac4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:00.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c37fe700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c41ea7a0 0x7f74c40c1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:00.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c27fc700 1 -- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c41ea7a0 msgr2=0x7f74c40c1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:00.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c27fc700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c41ea7a0 0x7f74c40c1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:00.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c27fc700 1 -- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74b0003aa0 msgr2=0x7f74c40ac4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:00.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c27fc700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74b0003aa0 0x7f74c40ac4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:00.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c27fc700 1 -- 172.21.15.7:0/2996329738 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c406bb80 con 0x7f74c40bf670 2023-03-03T10:31:00.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c37fe700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c41ea7a0 0x7f74c40c1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:31:00.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c27fc700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74c40bf670 0x7f74c4156050 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f74b4011b00 tx=0x7f74b4011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:31:00.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74bb7fe700 1 -- 172.21.15.7:0/2996329738 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b400f940 con 0x7f74c40bf670 2023-03-03T10:31:00.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74bb7fe700 1 -- 172.21.15.7:0/2996329738 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b400faa0 con 0x7f74c40bf670 2023-03-03T10:31:00.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c9705700 1 -- 172.21.15.7:0/2996329738 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c40aacc0 con 0x7f74c40bf670 2023-03-03T10:31:00.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74bb7fe700 1 -- 172.21.15.7:0/2996329738 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b400fe50 con 0x7f74c40bf670 2023-03-03T10:31:00.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.043+0000 7f74c9705700 1 -- 172.21.15.7:0/2996329738 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c4155100 con 0x7f74c40bf670 2023-03-03T10:31:00.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.044+0000 7f74c9705700 1 -- 172.21.15.7:0/2996329738 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c40c19b0 con 0x7f74c40bf670 2023-03-03T10:31:00.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.045+0000 7f74bb7fe700 1 -- 172.21.15.7:0/2996329738 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f74b4014440 con 0x7f74c40bf670 2023-03-03T10:31:00.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.045+0000 7f74bb7fe700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74a40682f0 0x7f74a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:00.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.045+0000 7f74bb7fe700 1 -- 172.21.15.7:0/2996329738 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f74b4086a00 con 0x7f74c40bf670 2023-03-03T10:31:00.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.045+0000 7f74c2ffd700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74a40682f0 0x7f74a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:00.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.046+0000 7f74c2ffd700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74a40682f0 0x7f74a406a7a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f74ac008fd0 tx=0x7f74ac00a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:31:00.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.049+0000 7f74bb7fe700 1 -- 172.21.15.7:0/2996329738 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f74c40c19b0 con 0x7f74c40bf670 2023-03-03T10:31:00.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.284+0000 7f74c9705700 1 -- 172.21.15.7:0/2996329738 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74c40c19b0 con 0x7f74c40bf670 2023-03-03T10:31:00.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.284+0000 7f74bb7fe700 1 -- 172.21.15.7:0/2996329738 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f74c40c19b0 con 0x7f74c40bf670 2023-03-03T10:31:00.287 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:31:00.287 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:31:00.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.286+0000 7f74c9705700 1 -- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74a40682f0 msgr2=0x7f74a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:00.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.286+0000 7f74c9705700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74a40682f0 0x7f74a406a7a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f74ac008fd0 tx=0x7f74ac00a000 comp rx=0 tx=0).stop 2023-03-03T10:31:00.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.286+0000 7f74c9705700 1 -- 172.21.15.7:0/2996329738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74c40bf670 msgr2=0x7f74c4156050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:00.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.286+0000 7f74c9705700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74c40bf670 0x7f74c4156050 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f74b4011b00 tx=0x7f74b4011e10 comp rx=0 tx=0).stop 2023-03-03T10:31:00.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.286+0000 7f74c9705700 1 -- 172.21.15.7:0/2996329738 shutdown_connections 2023-03-03T10:31:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.286+0000 7f74c9705700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74c41ea7a0 0x7f74c40c1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.286+0000 7f74c9705700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74b0003aa0 0x7f74c40ac4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.286+0000 7f74c9705700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74a40682f0 0x7f74a406a7a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.286+0000 7f74c9705700 1 --2- 172.21.15.7:0/2996329738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74c40bf670 0x7f74c4156050 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.286+0000 7f74c9705700 1 -- 172.21.15.7:0/2996329738 >> 172.21.15.7:0/2996329738 conn(0x7f74c40736f0 msgr2=0x7f74c4153e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:31:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.287+0000 7f74c9705700 1 -- 172.21.15.7:0/2996329738 shutdown_connections 2023-03-03T10:31:00.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:00.287+0000 7f74c9705700 1 -- 172.21.15.7:0/2996329738 wait complete. 2023-03-03T10:31:00.291 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:31:00.301 INFO:tasks.mds_thrash.fs.[a]:waiting for 81 secs before thrashing 2023-03-03T10:31:01.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:01.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:16.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:31:16.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.207+0000 7fdbb1086700 1 Processor -- start 2023-03-03T10:31:16.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.207+0000 7fdbb1086700 1 -- start start 2023-03-03T10:31:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbb1086700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdbac151b20 0x7fdbac0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbb1086700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdbac0c2950 0x7fdbac157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbb1086700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbac06cb70 con 0x7fdbac073c30 2023-03-03T10:31:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbb1086700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbac06c2f0 con 0x7fdbac151b20 2023-03-03T10:31:16.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbb1086700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbac06ce50 con 0x7fdbac0c2950 2023-03-03T10:31:16.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbaad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdbac151b20 0x7fdbac0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:16.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbab59e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdbac073c30 0x7fdbac158020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54596/0 (socket says 172.21.15.7:54596) 2023-03-03T10:31:16.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbab59e700 1 -- 172.21.15.7:0/3366137648 learned_addr learned my addr 172.21.15.7:0/3366137648 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:31:16.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbaad9d700 1 -- 172.21.15.7:0/3366137648 >> v2:172.21.15.143:3300/0 conn(0x7fdbac0c2950 msgr2=0x7fdbac157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:31:16.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbaa59c700 1 --2- 172.21.15.7:0/3366137648 >> v2:172.21.15.143:3300/0 conn(0x7fdbac0c2950 0x7fdbac157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:16.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbaad9d700 1 --2- 172.21.15.7:0/3366137648 >> v2:172.21.15.143:3300/0 conn(0x7fdbac0c2950 0x7fdbac157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:16.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbaad9d700 1 -- 172.21.15.7:0/3366137648 >> v1:172.21.15.7:6789/0 conn(0x7fdbac073c30 legacy=0x7fdbac158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:16.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdba9d9b700 1 -- 172.21.15.7:0/3366137648 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350718900 0 0) 0x7fdbac06cb70 con 0x7fdbac073c30 2023-03-03T10:31:16.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbaad9d700 1 -- 172.21.15.7:0/3366137648 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac06d130 con 0x7fdbac151b20 2023-03-03T10:31:16.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.208+0000 7fdbaa59c700 1 --2- 172.21.15.7:0/3366137648 >> v2:172.21.15.143:3300/0 conn(0x7fdbac0c2950 0x7fdbac157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:31:16.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdbaad9d700 1 --2- 172.21.15.7:0/3366137648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac151b20 0x7fdbac0736f0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fdbac06b8d0 tx=0x7fdba0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fe5de411968be8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:31:16.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdba9d9b700 1 -- 172.21.15.7:0/3366137648 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0011830 con 0x7fdbac151b20 2023-03-03T10:31:16.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdba9d9b700 1 -- 172.21.15.7:0/3366137648 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba0011990 con 0x7fdbac151b20 2023-03-03T10:31:16.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdba9d9b700 1 -- 172.21.15.7:0/3366137648 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0011d20 con 0x7fdbac151b20 2023-03-03T10:31:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdbb1086700 1 -- 172.21.15.7:0/3366137648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac151b20 msgr2=0x7fdbac0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdbb1086700 1 --2- 172.21.15.7:0/3366137648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac151b20 0x7fdbac0736f0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fdbac06b8d0 tx=0x7fdba0009b20 comp rx=0 tx=0).stop 2023-03-03T10:31:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdbb1086700 1 -- 172.21.15.7:0/3366137648 shutdown_connections 2023-03-03T10:31:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdbb1086700 1 --2- 172.21.15.7:0/3366137648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac151b20 0x7fdbac0736f0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdbb1086700 1 --2- 172.21.15.7:0/3366137648 >> v2:172.21.15.143:3300/0 conn(0x7fdbac0c2950 0x7fdbac157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdbb1086700 1 -- 172.21.15.7:0/3366137648 >> 172.21.15.7:0/3366137648 conn(0x7fdbac0a94d0 msgr2=0x7fdbac0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:31:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdbb1086700 1 -- 172.21.15.7:0/3366137648 shutdown_connections 2023-03-03T10:31:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.209+0000 7fdbb1086700 1 -- 172.21.15.7:0/3366137648 wait complete. 2023-03-03T10:31:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.211+0000 7fdbb1086700 1 Processor -- start 2023-03-03T10:31:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.211+0000 7fdbb1086700 1 -- start start 2023-03-03T10:31:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.212+0000 7fdbb1086700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbac073c30 0x7fdbac0bef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.212+0000 7fdbb1086700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac0c2950 0x7fdbac0bf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.212+0000 7fdbb1086700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbac151b20 0x7fdbac0aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.212+0000 7fdbb1086700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac0bf480 con 0x7fdbac0c2950 2023-03-03T10:31:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.212+0000 7fdbb1086700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac1e97c0 con 0x7fdbac151b20 2023-03-03T10:31:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.212+0000 7fdbb1086700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac1e9a60 con 0x7fdbac073c30 2023-03-03T10:31:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.212+0000 7fdbaa59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac0c2950 0x7fdbac0bf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.212+0000 7fdbaa59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac0c2950 0x7fdbac0bf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60618/0 (socket says 172.21.15.7:60618) 2023-03-03T10:31:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.212+0000 7fdbaa59c700 1 -- 172.21.15.7:0/3823882595 learned_addr learned my addr 172.21.15.7:0/3823882595 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:31:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdbab59e700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbac151b20 0x7fdbac0aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdbaad9d700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbac073c30 0x7fdbac0bef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdbaa59c700 1 -- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbac073c30 msgr2=0x7fdbac0bef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdbaa59c700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbac073c30 0x7fdbac0bef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdbaa59c700 1 -- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbac151b20 msgr2=0x7fdbac0aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdbaa59c700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbac151b20 0x7fdbac0aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdbaa59c700 1 -- 172.21.15.7:0/3823882595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba00094a0 con 0x7fdbac0c2950 2023-03-03T10:31:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdbaa59c700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac0c2950 0x7fdbac0bf610 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00c7f0 tx=0x7fdb9c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:31:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdb937fe700 1 -- 172.21.15.7:0/3823882595 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c011800 con 0x7fdbac0c2950 2023-03-03T10:31:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdb937fe700 1 -- 172.21.15.7:0/3823882595 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c011960 con 0x7fdbac0c2950 2023-03-03T10:31:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdbb1086700 1 -- 172.21.15.7:0/3823882595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbac158730 con 0x7fdbac0c2950 2023-03-03T10:31:16.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdb937fe700 1 -- 172.21.15.7:0/3823882595 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c011cf0 con 0x7fdbac0c2950 2023-03-03T10:31:16.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.213+0000 7fdbb1086700 1 -- 172.21.15.7:0/3823882595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbac06d410 con 0x7fdbac0c2950 2023-03-03T10:31:16.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.214+0000 7fdbb1086700 1 -- 172.21.15.7:0/3823882595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbac151ef0 con 0x7fdbac0c2950 2023-03-03T10:31:16.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.215+0000 7fdb937fe700 1 -- 172.21.15.7:0/3823882595 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdb9c021070 con 0x7fdbac0c2950 2023-03-03T10:31:16.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.215+0000 7fdb937fe700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb980682f0 0x7fdb9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:16.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.215+0000 7fdb937fe700 1 -- 172.21.15.7:0/3823882595 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdb9c085e70 con 0x7fdbac0c2950 2023-03-03T10:31:16.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.215+0000 7fdbaad9d700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb980682f0 0x7fdb9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:16.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.216+0000 7fdbaad9d700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb980682f0 0x7fdb9806a7a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdba00090b0 tx=0x7fdba0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:31:16.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.219+0000 7fdb937fe700 1 -- 172.21.15.7:0/3823882595 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdb9c052210 con 0x7fdbac0c2950 2023-03-03T10:31:16.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.462+0000 7fdbb1086700 1 -- 172.21.15.7:0/3823882595 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbac1e9bd0 con 0x7fdbac0c2950 2023-03-03T10:31:16.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.462+0000 7fdb937fe700 1 -- 172.21.15.7:0/3823882595 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdb9c055830 con 0x7fdbac0c2950 2023-03-03T10:31:16.465 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:31:16.465 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:31:16.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 -- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb980682f0 msgr2=0x7fdb9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:16.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb980682f0 0x7fdb9806a7a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fdba00090b0 tx=0x7fdba0008030 comp rx=0 tx=0).stop 2023-03-03T10:31:16.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 -- 172.21.15.7:0/3823882595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac0c2950 msgr2=0x7fdbac0bf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:16.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac0c2950 0x7fdbac0bf610 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00c7f0 tx=0x7fdb9c00cb00 comp rx=0 tx=0).stop 2023-03-03T10:31:16.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 -- 172.21.15.7:0/3823882595 shutdown_connections 2023-03-03T10:31:16.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbac151b20 0x7fdbac0aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:16.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbac073c30 0x7fdbac0bef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:16.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb980682f0 0x7fdb9806a7a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:16.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 --2- 172.21.15.7:0/3823882595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbac0c2950 0x7fdbac0bf610 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:16.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 -- 172.21.15.7:0/3823882595 >> 172.21.15.7:0/3823882595 conn(0x7fdbac0a94d0 msgr2=0x7fdbac070aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:31:16.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 -- 172.21.15.7:0/3823882595 shutdown_connections 2023-03-03T10:31:16.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:16.464+0000 7fdbb1086700 1 -- 172.21.15.7:0/3823882595 wait complete. 2023-03-03T10:31:16.469 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:31:16.479 INFO:tasks.mds_thrash.fs.[b]:waiting for 22 secs before thrashing 2023-03-03T10:31:31.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:31.272 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:31:38.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:31:38.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.615+0000 7f6c98e86700 1 Processor -- start 2023-03-03T10:31:38.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.615+0000 7f6c98e86700 1 -- start start 2023-03-03T10:31:38.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c98e86700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 0x7f6c94156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:38.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c98e86700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6c9414fda0 0x7f6c94071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:38.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c98e86700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c940c2f30 con 0x7f6c940abe20 2023-03-03T10:31:38.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c98e86700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c940c31d0 con 0x7f6c9414fda0 2023-03-03T10:31:38.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c98e86700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c941524a0 con 0x7f6c9414f9d0 2023-03-03T10:31:38.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c91d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6c9414fda0 0x7f6c94071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:38.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c91d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6c9414fda0 0x7f6c94071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60630/0 (socket says 172.21.15.7:60630) 2023-03-03T10:31:38.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c91d9b700 1 -- 172.21.15.7:0/1318186916 learned_addr learned my addr 172.21.15.7:0/1318186916 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:31:38.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c9259c700 1 --2- 172.21.15.7:0/1318186916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 0x7f6c94156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:38.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c91d9b700 1 -- 172.21.15.7:0/1318186916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 msgr2=0x7f6c94156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:31:38.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c91d9b700 1 --2- 172.21.15.7:0/1318186916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 0x7f6c94156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:38.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c91d9b700 1 -- 172.21.15.7:0/1318186916 >> v1:172.21.15.143:6789/0 conn(0x7f6c940abe20 legacy=0x7f6c941529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:31:38.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c91d9b700 1 -- 172.21.15.7:0/1318186916 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c940c34b0 con 0x7f6c9414fda0 2023-03-03T10:31:38.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.616+0000 7f6c9259c700 1 --2- 172.21.15.7:0/1318186916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 0x7f6c94156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:31:38.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c91d9b700 1 --2- 172.21.15.7:0/1318186916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9414fda0 0x7f6c94071890 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6c840098f0 tx=0x7f6c84009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800b9991f2f22771 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:31:38.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c9159a700 1 -- 172.21.15.7:0/1318186916 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84004230 con 0x7f6c9414fda0 2023-03-03T10:31:38.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c9159a700 1 -- 172.21.15.7:0/1318186916 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c84004390 con 0x7f6c9414fda0 2023-03-03T10:31:38.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c9159a700 1 -- 172.21.15.7:0/1318186916 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c840036a0 con 0x7f6c9414fda0 2023-03-03T10:31:38.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c98e86700 1 -- 172.21.15.7:0/1318186916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9414fda0 msgr2=0x7f6c94071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:38.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c98e86700 1 --2- 172.21.15.7:0/1318186916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9414fda0 0x7f6c94071890 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6c840098f0 tx=0x7f6c84009c00 comp rx=0 tx=0).stop 2023-03-03T10:31:38.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c98e86700 1 -- 172.21.15.7:0/1318186916 shutdown_connections 2023-03-03T10:31:38.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c98e86700 1 --2- 172.21.15.7:0/1318186916 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 0x7f6c94156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:38.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c98e86700 1 --2- 172.21.15.7:0/1318186916 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9414fda0 0x7f6c94071890 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:38.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c98e86700 1 -- 172.21.15.7:0/1318186916 >> 172.21.15.7:0/1318186916 conn(0x7f6c940c8900 msgr2=0x7f6c940c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:31:38.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c98e86700 1 -- 172.21.15.7:0/1318186916 shutdown_connections 2023-03-03T10:31:38.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.617+0000 7f6c98e86700 1 -- 172.21.15.7:0/1318186916 wait complete. 2023-03-03T10:31:38.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.619+0000 7f6c98e86700 1 Processor -- start 2023-03-03T10:31:38.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.619+0000 7f6c98e86700 1 -- start start 2023-03-03T10:31:38.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.619+0000 7f6c98e86700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c940abe20 0x7f6c9406f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:38.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.619+0000 7f6c98e86700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 0x7f6c94154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:38.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c98e86700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9414fda0 0x7f6c9414b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:38.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c98e86700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c941dd5a0 con 0x7f6c940abe20 2023-03-03T10:31:38.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c98e86700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9406f6d0 con 0x7f6c9414fda0 2023-03-03T10:31:38.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c98e86700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9406df80 con 0x7f6c9414f9d0 2023-03-03T10:31:38.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c9259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c940abe20 0x7f6c9406f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:38.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c9259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c940abe20 0x7f6c9406f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60634/0 (socket says 172.21.15.7:60634) 2023-03-03T10:31:38.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c9259c700 1 -- 172.21.15.7:0/1860250327 learned_addr learned my addr 172.21.15.7:0/1860250327 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:31:38.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c91d9b700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 0x7f6c94154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:38.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c92d9d700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9414fda0 0x7f6c9414b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:38.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c9259c700 1 -- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 msgr2=0x7f6c94154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:38.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c9259c700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 0x7f6c94154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:38.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c9259c700 1 -- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9414fda0 msgr2=0x7f6c9414b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:38.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c9259c700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9414fda0 0x7f6c9414b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:38.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c9259c700 1 -- 172.21.15.7:0/1860250327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c84009550 con 0x7f6c940abe20 2023-03-03T10:31:38.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c91d9b700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 0x7f6c94154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:31:38.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c9259c700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c940abe20 0x7f6c9406f040 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800c750 tx=0x7f6c8800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:31:38.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c82ffd700 1 -- 172.21.15.7:0/1860250327 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88011800 con 0x7f6c940abe20 2023-03-03T10:31:38.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c82ffd700 1 -- 172.21.15.7:0/1860250327 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c88011960 con 0x7f6c940abe20 2023-03-03T10:31:38.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c98e86700 1 -- 172.21.15.7:0/1860250327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c940ac350 con 0x7f6c940abe20 2023-03-03T10:31:38.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c82ffd700 1 -- 172.21.15.7:0/1860250327 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88011cd0 con 0x7f6c940abe20 2023-03-03T10:31:38.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.620+0000 7f6c98e86700 1 -- 172.21.15.7:0/1860250327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c940ab630 con 0x7f6c940abe20 2023-03-03T10:31:38.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.621+0000 7f6c98e86700 1 -- 172.21.15.7:0/1860250327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c940c3e10 con 0x7f6c940abe20 2023-03-03T10:31:38.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.622+0000 7f6c82ffd700 1 -- 172.21.15.7:0/1860250327 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c88020070 con 0x7f6c940abe20 2023-03-03T10:31:38.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.622+0000 7f6c82ffd700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c740682f0 0x7f6c7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:31:38.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.622+0000 7f6c82ffd700 1 -- 172.21.15.7:0/1860250327 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6c880860c0 con 0x7f6c940abe20 2023-03-03T10:31:38.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.622+0000 7f6c91d9b700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c740682f0 0x7f6c7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:31:38.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.623+0000 7f6c91d9b700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c740682f0 0x7f6c7406a7a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6c941501d0 tx=0x7f6c84008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:31:38.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.627+0000 7f6c82ffd700 1 -- 172.21.15.7:0/1860250327 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c88052370 con 0x7f6c940abe20 2023-03-03T10:31:38.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.866+0000 7f6c98e86700 1 -- 172.21.15.7:0/1860250327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c94074bb0 con 0x7f6c940abe20 2023-03-03T10:31:38.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.867+0000 7f6c82ffd700 1 -- 172.21.15.7:0/1860250327 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6c88055990 con 0x7f6c940abe20 2023-03-03T10:31:38.870 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:31:38.870 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:31:38.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.868+0000 7f6c98e86700 1 -- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c740682f0 msgr2=0x7f6c7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:38.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.868+0000 7f6c98e86700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c740682f0 0x7f6c7406a7a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6c941501d0 tx=0x7f6c84008030 comp rx=0 tx=0).stop 2023-03-03T10:31:38.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.868+0000 7f6c98e86700 1 -- 172.21.15.7:0/1860250327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c940abe20 msgr2=0x7f6c9406f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:31:38.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.868+0000 7f6c98e86700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c940abe20 0x7f6c9406f040 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800c750 tx=0x7f6c8800cb10 comp rx=0 tx=0).stop 2023-03-03T10:31:38.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.869+0000 7f6c98e86700 1 -- 172.21.15.7:0/1860250327 shutdown_connections 2023-03-03T10:31:38.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.869+0000 7f6c98e86700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9414fda0 0x7f6c9414b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:38.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.869+0000 7f6c98e86700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c740682f0 0x7f6c7406a7a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:38.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.869+0000 7f6c98e86700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c940abe20 0x7f6c9406f040 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:38.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.869+0000 7f6c98e86700 1 --2- 172.21.15.7:0/1860250327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c9414f9d0 0x7f6c94154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:31:38.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.869+0000 7f6c98e86700 1 -- 172.21.15.7:0/1860250327 >> 172.21.15.7:0/1860250327 conn(0x7f6c940c8900 msgr2=0x7f6c940a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:31:38.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.869+0000 7f6c98e86700 1 -- 172.21.15.7:0/1860250327 shutdown_connections 2023-03-03T10:31:38.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:31:38.869+0000 7f6c98e86700 1 -- 172.21.15.7:0/1860250327 wait complete. 2023-03-03T10:31:38.874 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:31:38.884 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2023-03-03T10:32:01.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:01.364 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:12.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:32:13.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.021+0000 7ff523292700 1 Processor -- start 2023-03-03T10:32:13.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.021+0000 7ff523292700 1 -- start start 2023-03-03T10:32:13.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff523292700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff51c14fda0 0x7ff51c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:13.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff523292700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c0abe20 0x7ff51c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:13.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff523292700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51c0c2f30 con 0x7ff51c14f9d0 2023-03-03T10:32:13.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff523292700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51c0c31d0 con 0x7ff51c14fda0 2023-03-03T10:32:13.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff523292700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c1524a0 con 0x7ff51c0abe20 2023-03-03T10:32:13.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff52082d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff51c14fda0 0x7ff51c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:13.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff52082d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff51c14fda0 0x7ff51c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60670/0 (socket says 172.21.15.7:60670) 2023-03-03T10:32:13.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff52082d700 1 -- 172.21.15.7:0/322492710 learned_addr learned my addr 172.21.15.7:0/322492710 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:32:13.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff52182f700 1 --2- 172.21.15.7:0/322492710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c0abe20 0x7ff51c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:13.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff52082d700 1 -- 172.21.15.7:0/322492710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c0abe20 msgr2=0x7ff51c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:32:13.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff52082d700 1 --2- 172.21.15.7:0/322492710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c0abe20 0x7ff51c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:13.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff52082d700 1 -- 172.21.15.7:0/322492710 >> v1:172.21.15.143:6789/0 conn(0x7ff51c14f9d0 legacy=0x7ff51c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:32:13.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.022+0000 7ff52082d700 1 -- 172.21.15.7:0/322492710 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c0c34b0 con 0x7ff51c14fda0 2023-03-03T10:32:13.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.023+0000 7ff52082d700 1 --2- 172.21.15.7:0/322492710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14fda0 0x7ff51c156430 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff5180098f0 tx=0x7ff518009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b631e81a36e88a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:13.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.023+0000 7ff513fff700 1 -- 172.21.15.7:0/322492710 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518004230 con 0x7ff51c14fda0 2023-03-03T10:32:13.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.023+0000 7ff513fff700 1 -- 172.21.15.7:0/322492710 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff518004390 con 0x7ff51c14fda0 2023-03-03T10:32:13.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.023+0000 7ff513fff700 1 -- 172.21.15.7:0/322492710 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5180036a0 con 0x7ff51c14fda0 2023-03-03T10:32:13.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.023+0000 7ff523292700 1 -- 172.21.15.7:0/322492710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14fda0 msgr2=0x7ff51c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:13.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.023+0000 7ff523292700 1 --2- 172.21.15.7:0/322492710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14fda0 0x7ff51c156430 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff5180098f0 tx=0x7ff518009c00 comp rx=0 tx=0).stop 2023-03-03T10:32:13.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.023+0000 7ff523292700 1 -- 172.21.15.7:0/322492710 shutdown_connections 2023-03-03T10:32:13.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.023+0000 7ff523292700 1 --2- 172.21.15.7:0/322492710 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c0abe20 0x7ff51c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:13.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.023+0000 7ff523292700 1 --2- 172.21.15.7:0/322492710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14fda0 0x7ff51c156430 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:13.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.023+0000 7ff523292700 1 -- 172.21.15.7:0/322492710 >> 172.21.15.7:0/322492710 conn(0x7ff51c0c8900 msgr2=0x7ff51c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:32:13.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.024+0000 7ff523292700 1 -- 172.21.15.7:0/322492710 shutdown_connections 2023-03-03T10:32:13.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.024+0000 7ff523292700 1 -- 172.21.15.7:0/322492710 wait complete. 2023-03-03T10:32:13.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.025+0000 7ff523292700 1 Processor -- start 2023-03-03T10:32:13.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.025+0000 7ff523292700 1 -- start start 2023-03-03T10:32:13.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.025+0000 7ff523292700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c0abe20 0x7ff51c1e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:13.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.025+0000 7ff523292700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14f9d0 0x7ff51c06d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:13.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.025+0000 7ff523292700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c14fda0 0x7ff51c06de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:13.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.025+0000 7ff523292700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c0aa830 con 0x7ff51c14f9d0 2023-03-03T10:32:13.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.025+0000 7ff523292700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c1e81d0 con 0x7ff51c0abe20 2023-03-03T10:32:13.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.025+0000 7ff523292700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51c1dc180 con 0x7ff51c14fda0 2023-03-03T10:32:13.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52082d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14f9d0 0x7ff51c06d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:13.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52082d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14f9d0 0x7ff51c06d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60678/0 (socket says 172.21.15.7:60678) 2023-03-03T10:32:13.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52082d700 1 -- 172.21.15.7:0/2721018139 learned_addr learned my addr 172.21.15.7:0/2721018139 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:32:13.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52182f700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c14fda0 0x7ff51c06de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:13.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52102e700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c0abe20 0x7ff51c1e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:13.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52082d700 1 -- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c14fda0 msgr2=0x7ff51c06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:13.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52082d700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c14fda0 0x7ff51c06de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:13.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52082d700 1 -- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c0abe20 msgr2=0x7ff51c1e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:13.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52082d700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c0abe20 0x7ff51c1e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:13.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52082d700 1 -- 172.21.15.7:0/2721018139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff518009550 con 0x7ff51c14f9d0 2023-03-03T10:32:13.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52182f700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c14fda0 0x7ff51c06de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:32:13.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff52082d700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14f9d0 0x7ff51c06d920 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff518004000 tx=0x7ff518004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:13.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff511ffb700 1 -- 172.21.15.7:0/2721018139 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff518003cc0 con 0x7ff51c14f9d0 2023-03-03T10:32:13.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff511ffb700 1 -- 172.21.15.7:0/2721018139 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51801f070 con 0x7ff51c14f9d0 2023-03-03T10:32:13.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff511ffb700 1 -- 172.21.15.7:0/2721018139 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51800b8a0 con 0x7ff51c14f9d0 2023-03-03T10:32:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.026+0000 7ff523292700 1 -- 172.21.15.7:0/2721018139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff51c0ac350 con 0x7ff51c14f9d0 2023-03-03T10:32:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.027+0000 7ff523292700 1 -- 172.21.15.7:0/2721018139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff51c0ab630 con 0x7ff51c14f9d0 2023-03-03T10:32:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.027+0000 7ff523292700 1 -- 172.21.15.7:0/2721018139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51c071fa0 con 0x7ff51c14f9d0 2023-03-03T10:32:13.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.028+0000 7ff511ffb700 1 -- 172.21.15.7:0/2721018139 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff51802a020 con 0x7ff51c14f9d0 2023-03-03T10:32:13.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.028+0000 7ff511ffb700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5000682f0 0x7ff50006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:13.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.028+0000 7ff511ffb700 1 -- 172.21.15.7:0/2721018139 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff518087910 con 0x7ff51c14f9d0 2023-03-03T10:32:13.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.028+0000 7ff52102e700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5000682f0 0x7ff50006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:13.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.029+0000 7ff52102e700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5000682f0 0x7ff50006a7a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff51c150290 tx=0x7ff50c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:13.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.033+0000 7ff511ffb700 1 -- 172.21.15.7:0/2721018139 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff518053bc0 con 0x7ff51c14f9d0 2023-03-03T10:32:13.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.268+0000 7ff523292700 1 -- 172.21.15.7:0/2721018139 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff51c071fa0 con 0x7ff51c14f9d0 2023-03-03T10:32:13.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.269+0000 7ff511ffb700 1 -- 172.21.15.7:0/2721018139 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff5180571e0 con 0x7ff51c14f9d0 2023-03-03T10:32:13.271 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:32:13.271 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:32:13.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 -- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5000682f0 msgr2=0x7ff50006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:13.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5000682f0 0x7ff50006a7a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff51c150290 tx=0x7ff50c009040 comp rx=0 tx=0).stop 2023-03-03T10:32:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 -- 172.21.15.7:0/2721018139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14f9d0 msgr2=0x7ff51c06d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14f9d0 0x7ff51c06d920 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff518004000 tx=0x7ff518004750 comp rx=0 tx=0).stop 2023-03-03T10:32:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 -- 172.21.15.7:0/2721018139 shutdown_connections 2023-03-03T10:32:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff51c14fda0 0x7ff51c06de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff51c0abe20 0x7ff51c1e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5000682f0 0x7ff50006a7a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 --2- 172.21.15.7:0/2721018139 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff51c14f9d0 0x7ff51c06d920 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:13.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 -- 172.21.15.7:0/2721018139 >> 172.21.15.7:0/2721018139 conn(0x7ff51c0c8900 msgr2=0x7ff51c151560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:32:13.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 -- 172.21.15.7:0/2721018139 shutdown_connections 2023-03-03T10:32:13.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:13.271+0000 7ff523292700 1 -- 172.21.15.7:0/2721018139 wait complete. 2023-03-03T10:32:13.275 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:32:13.285 INFO:tasks.mds_thrash.fs.[b]:waiting for 46 secs before thrashing 2023-03-03T10:32:21.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:32:21.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.439+0000 7fa3a387a700 1 Processor -- start 2023-03-03T10:32:21.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.439+0000 7fa3a387a700 1 -- start start 2023-03-03T10:32:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a387a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa39c14fda0 0x7fa39c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a387a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa39c0abe20 0x7fa39c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a387a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa39c0c2f30 con 0x7fa39c14f9d0 2023-03-03T10:32:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a387a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c0c31d0 con 0x7fa39c0abe20 2023-03-03T10:32:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a387a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c1524a0 con 0x7fa39c14fda0 2023-03-03T10:32:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a1e17700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa39c0abe20 0x7fa39c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a1e17700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa39c0abe20 0x7fa39c1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60688/0 (socket says 172.21.15.7:60688) 2023-03-03T10:32:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a1e17700 1 -- 172.21.15.7:0/2188244841 learned_addr learned my addr 172.21.15.7:0/2188244841 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:32:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a0e15700 1 --2- 172.21.15.7:0/2188244841 >> v2:172.21.15.143:3300/0 conn(0x7fa39c14fda0 0x7fa39c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a1e17700 1 -- 172.21.15.7:0/2188244841 >> v2:172.21.15.143:3300/0 conn(0x7fa39c14fda0 msgr2=0x7fa39c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:32:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a1e17700 1 --2- 172.21.15.7:0/2188244841 >> v2:172.21.15.143:3300/0 conn(0x7fa39c14fda0 0x7fa39c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a1e17700 1 -- 172.21.15.7:0/2188244841 >> v1:172.21.15.143:6789/0 conn(0x7fa39c14f9d0 legacy=0x7fa39c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a1e17700 1 -- 172.21.15.7:0/2188244841 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c0c34b0 con 0x7fa39c0abe20 2023-03-03T10:32:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.440+0000 7fa3a1e17700 1 --2- 172.21.15.7:0/2188244841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 0x7fa39c1529c0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0709b0 tx=0x7fa39400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e5a23f2e698468 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa393fff700 1 -- 172.21.15.7:0/2188244841 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39400e070 con 0x7fa39c0abe20 2023-03-03T10:32:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa393fff700 1 -- 172.21.15.7:0/2188244841 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39400b9c0 con 0x7fa39c0abe20 2023-03-03T10:32:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa393fff700 1 -- 172.21.15.7:0/2188244841 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39400bd70 con 0x7fa39c0abe20 2023-03-03T10:32:21.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa3a387a700 1 -- 172.21.15.7:0/2188244841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 msgr2=0x7fa39c1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa3a387a700 1 --2- 172.21.15.7:0/2188244841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 0x7fa39c1529c0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0709b0 tx=0x7fa39400b330 comp rx=0 tx=0).stop 2023-03-03T10:32:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa3a387a700 1 -- 172.21.15.7:0/2188244841 shutdown_connections 2023-03-03T10:32:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa3a387a700 1 --2- 172.21.15.7:0/2188244841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 0x7fa39c1529c0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa3a387a700 1 --2- 172.21.15.7:0/2188244841 >> v2:172.21.15.143:3300/0 conn(0x7fa39c14fda0 0x7fa39c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa3a387a700 1 -- 172.21.15.7:0/2188244841 >> 172.21.15.7:0/2188244841 conn(0x7fa39c0c8900 msgr2=0x7fa39c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:32:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa3a387a700 1 -- 172.21.15.7:0/2188244841 shutdown_connections 2023-03-03T10:32:21.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.441+0000 7fa3a387a700 1 -- 172.21.15.7:0/2188244841 wait complete. 2023-03-03T10:32:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.442+0000 7fa3a387a700 1 Processor -- start 2023-03-03T10:32:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.442+0000 7fa3a387a700 1 -- start start 2023-03-03T10:32:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a387a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 0x7fa39c1de560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a387a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa39c14f9d0 0x7fa39c1555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a387a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa39c14fda0 0x7fa39c1eb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a387a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c1dd470 con 0x7fa39c0abe20 2023-03-03T10:32:21.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a387a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c1df470 con 0x7fa39c14f9d0 2023-03-03T10:32:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a387a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c06db80 con 0x7fa39c14fda0 2023-03-03T10:32:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a1616700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 0x7fa39c1de560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a1616700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 0x7fa39c1de560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60690/0 (socket says 172.21.15.7:60690) 2023-03-03T10:32:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a1616700 1 -- 172.21.15.7:0/1566574132 learned_addr learned my addr 172.21.15.7:0/1566574132 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:32:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a0e15700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa39c14f9d0 0x7fa39c1555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.443+0000 7fa3a1e17700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa39c14fda0 0x7fa39c1eb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:21.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa3a1616700 1 -- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa39c14fda0 msgr2=0x7fa39c1eb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa3a1616700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa39c14fda0 0x7fa39c1eb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa3a1616700 1 -- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa39c14f9d0 msgr2=0x7fa39c1555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa3a1616700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa39c14f9d0 0x7fa39c1555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa3a1616700 1 -- 172.21.15.7:0/1566574132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa394009b30 con 0x7fa39c0abe20 2023-03-03T10:32:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa3a1616700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 0x7fa39c1de560 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fa38c00c510 tx=0x7fa38c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:21.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa391ffb700 1 -- 172.21.15.7:0/1566574132 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c0052d0 con 0x7fa39c0abe20 2023-03-03T10:32:21.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa391ffb700 1 -- 172.21.15.7:0/1566574132 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38c005430 con 0x7fa39c0abe20 2023-03-03T10:32:21.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa3a387a700 1 -- 172.21.15.7:0/1566574132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c0ac350 con 0x7fa39c0abe20 2023-03-03T10:32:21.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa391ffb700 1 -- 172.21.15.7:0/1566574132 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c0038c0 con 0x7fa39c0abe20 2023-03-03T10:32:21.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.444+0000 7fa3a387a700 1 -- 172.21.15.7:0/1566574132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c0ab630 con 0x7fa39c0abe20 2023-03-03T10:32:21.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.445+0000 7fa3a387a700 1 -- 172.21.15.7:0/1566574132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c071fa0 con 0x7fa39c0abe20 2023-03-03T10:32:21.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.446+0000 7fa391ffb700 1 -- 172.21.15.7:0/1566574132 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa38c019030 con 0x7fa39c0abe20 2023-03-03T10:32:21.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.446+0000 7fa391ffb700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3800682f0 0x7fa38006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:21.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.446+0000 7fa391ffb700 1 -- 172.21.15.7:0/1566574132 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa38c085390 con 0x7fa39c0abe20 2023-03-03T10:32:21.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.446+0000 7fa3a0e15700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3800682f0 0x7fa38006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:21.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.447+0000 7fa3a0e15700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3800682f0 0x7fa38006a7a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa39c1df640 tx=0x7fa398008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:21.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.450+0000 7fa391ffb700 1 -- 172.21.15.7:0/1566574132 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa38c051730 con 0x7fa39c0abe20 2023-03-03T10:32:21.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.688+0000 7fa3a387a700 1 -- 172.21.15.7:0/1566574132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa39c074bb0 con 0x7fa39c0abe20 2023-03-03T10:32:21.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.689+0000 7fa391ffb700 1 -- 172.21.15.7:0/1566574132 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa38c054d50 con 0x7fa39c0abe20 2023-03-03T10:32:21.691 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:32:21.691 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:32:21.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 -- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3800682f0 msgr2=0x7fa38006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:21.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3800682f0 0x7fa38006a7a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa39c1df640 tx=0x7fa398008040 comp rx=0 tx=0).stop 2023-03-03T10:32:21.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 -- 172.21.15.7:0/1566574132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 msgr2=0x7fa39c1de560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 0x7fa39c1de560 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fa38c00c510 tx=0x7fa38c00c820 comp rx=0 tx=0).stop 2023-03-03T10:32:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 -- 172.21.15.7:0/1566574132 shutdown_connections 2023-03-03T10:32:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa39c14fda0 0x7fa39c1eb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3800682f0 0x7fa38006a7a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa39c0abe20 0x7fa39c1de560 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:21.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 --2- 172.21.15.7:0/1566574132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa39c14f9d0 0x7fa39c1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:21.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 -- 172.21.15.7:0/1566574132 >> 172.21.15.7:0/1566574132 conn(0x7fa39c0c8900 msgr2=0x7fa39c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:32:21.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 -- 172.21.15.7:0/1566574132 shutdown_connections 2023-03-03T10:32:21.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:21.691+0000 7fa3a387a700 1 -- 172.21.15.7:0/1566574132 wait complete. 2023-03-03T10:32:21.695 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:32:21.705 INFO:tasks.mds_thrash.fs.[a]:waiting for 99 secs before thrashing 2023-03-03T10:32:31.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:31.446 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:32:59.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:32:59.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.423+0000 7fa35b9ae700 1 Processor -- start 2023-03-03T10:32:59.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.423+0000 7fa35b9ae700 1 -- start start 2023-03-03T10:32:59.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa35b9ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa35406a2b0 0x7fa35406a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:59.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa35b9ae700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa3540a6370 0x7fa354155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa35b9ae700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3541558f0 con 0x7fa3540c5cb0 2023-03-03T10:32:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa35b9ae700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35406bab0 con 0x7fa3540a6370 2023-03-03T10:32:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa35b9ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540a6180 con 0x7fa35406a2b0 2023-03-03T10:32:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa358f49700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa3540a6370 0x7fa354155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa359f4b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa3540c5cb0 0x7fa3540c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54704/0 (socket says 172.21.15.7:54704) 2023-03-03T10:32:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa359f4b700 1 -- 172.21.15.7:0/363274782 learned_addr learned my addr 172.21.15.7:0/363274782 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:32:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa35974a700 1 --2- 172.21.15.7:0/363274782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa35406a2b0 0x7fa35406a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa358f49700 1 -- 172.21.15.7:0/363274782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa35406a2b0 msgr2=0x7fa35406a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:32:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa358f49700 1 --2- 172.21.15.7:0/363274782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa35406a2b0 0x7fa35406a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa358f49700 1 -- 172.21.15.7:0/363274782 >> v1:172.21.15.7:6789/0 conn(0x7fa3540c5cb0 legacy=0x7fa3540c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa358f49700 1 -- 172.21.15.7:0/363274782 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3540a5ea0 con 0x7fa3540a6370 2023-03-03T10:32:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa34bfff700 1 -- 172.21.15.7:0/363274782 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388743451 0 0) 0x7fa3541558f0 con 0x7fa3540c5cb0 2023-03-03T10:32:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa358f49700 1 --2- 172.21.15.7:0/363274782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 0x7fa354155d80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa344009840 tx=0x7fa344009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2d50150116b13b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:59.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.424+0000 7fa34bfff700 1 -- 172.21.15.7:0/363274782 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344011830 con 0x7fa3540a6370 2023-03-03T10:32:59.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.425+0000 7fa34bfff700 1 -- 172.21.15.7:0/363274782 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa344011990 con 0x7fa3540a6370 2023-03-03T10:32:59.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.425+0000 7fa34bfff700 1 -- 172.21.15.7:0/363274782 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344011d20 con 0x7fa3540a6370 2023-03-03T10:32:59.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.425+0000 7fa35b9ae700 1 -- 172.21.15.7:0/363274782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 msgr2=0x7fa354155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:59.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.425+0000 7fa35b9ae700 1 --2- 172.21.15.7:0/363274782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 0x7fa354155d80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa344009840 tx=0x7fa344009b50 comp rx=0 tx=0).stop 2023-03-03T10:32:59.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.425+0000 7fa35b9ae700 1 -- 172.21.15.7:0/363274782 shutdown_connections 2023-03-03T10:32:59.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.425+0000 7fa35b9ae700 1 --2- 172.21.15.7:0/363274782 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa35406a2b0 0x7fa35406a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.425+0000 7fa35b9ae700 1 --2- 172.21.15.7:0/363274782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 0x7fa354155d80 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.425+0000 7fa35b9ae700 1 -- 172.21.15.7:0/363274782 >> 172.21.15.7:0/363274782 conn(0x7fa3540a9430 msgr2=0x7fa3540c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:32:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.425+0000 7fa35b9ae700 1 -- 172.21.15.7:0/363274782 shutdown_connections 2023-03-03T10:32:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.425+0000 7fa35b9ae700 1 -- 172.21.15.7:0/363274782 wait complete. 2023-03-03T10:32:59.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.427+0000 7fa35b9ae700 1 Processor -- start 2023-03-03T10:32:59.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.427+0000 7fa35b9ae700 1 -- start start 2023-03-03T10:32:59.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa35b9ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa35406a2b0 0x7fa354154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:59.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa35b9ae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 0x7fa35414cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:59.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa35b9ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3540c5cb0 0x7fa35414d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:59.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa35b9ae700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540723d0 con 0x7fa3540a6370 2023-03-03T10:32:59.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa35b9ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3541dbd60 con 0x7fa3540c5cb0 2023-03-03T10:32:59.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa35b9ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540725e0 con 0x7fa35406a2b0 2023-03-03T10:32:59.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa358f49700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 0x7fa35414cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa358f49700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 0x7fa35414cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60728/0 (socket says 172.21.15.7:60728) 2023-03-03T10:32:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa358f49700 1 -- 172.21.15.7:0/218343110 learned_addr learned my addr 172.21.15.7:0/218343110 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:32:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa35974a700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa35406a2b0 0x7fa354154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa358f49700 1 -- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa35406a2b0 msgr2=0x7fa354154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa359f4b700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3540c5cb0 0x7fa35414d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa358f49700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa35406a2b0 0x7fa354154860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa358f49700 1 -- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3540c5cb0 msgr2=0x7fa35414d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa358f49700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3540c5cb0 0x7fa35414d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa358f49700 1 -- 172.21.15.7:0/218343110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3440094a0 con 0x7fa3540a6370 2023-03-03T10:32:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.428+0000 7fa358f49700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 0x7fa35414cb20 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa344014040 tx=0x7fa344004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.429+0000 7fa349ffb700 1 -- 172.21.15.7:0/218343110 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440038a0 con 0x7fa3540a6370 2023-03-03T10:32:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.429+0000 7fa349ffb700 1 -- 172.21.15.7:0/218343110 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34401e070 con 0x7fa3540a6370 2023-03-03T10:32:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.429+0000 7fa35b9ae700 1 -- 172.21.15.7:0/218343110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa354153160 con 0x7fa3540a6370 2023-03-03T10:32:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.429+0000 7fa349ffb700 1 -- 172.21.15.7:0/218343110 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344003c40 con 0x7fa3540a6370 2023-03-03T10:32:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.429+0000 7fa35b9ae700 1 -- 172.21.15.7:0/218343110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa354152e80 con 0x7fa3540a6370 2023-03-03T10:32:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.430+0000 7fa35b9ae700 1 -- 172.21.15.7:0/218343110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3540c1500 con 0x7fa3540a6370 2023-03-03T10:32:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.430+0000 7fa349ffb700 1 -- 172.21.15.7:0/218343110 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa34400f050 con 0x7fa3540a6370 2023-03-03T10:32:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.431+0000 7fa349ffb700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3380682f0 0x7fa33806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:32:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.431+0000 7fa349ffb700 1 -- 172.21.15.7:0/218343110 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa344087300 con 0x7fa3540a6370 2023-03-03T10:32:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.431+0000 7fa35974a700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3380682f0 0x7fa33806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:32:59.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.431+0000 7fa35974a700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3380682f0 0x7fa33806a7a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa35000b840 tx=0x7fa350008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:32:59.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.435+0000 7fa349ffb700 1 -- 172.21.15.7:0/218343110 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa3440536a0 con 0x7fa3540a6370 2023-03-03T10:32:59.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.670+0000 7fa35b9ae700 1 -- 172.21.15.7:0/218343110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa354074bb0 con 0x7fa3540a6370 2023-03-03T10:32:59.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.671+0000 7fa349ffb700 1 -- 172.21.15.7:0/218343110 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa344020090 con 0x7fa3540a6370 2023-03-03T10:32:59.674 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:32:59.674 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:32:59.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 -- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3380682f0 msgr2=0x7fa33806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:59.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3380682f0 0x7fa33806a7a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa35000b840 tx=0x7fa350008040 comp rx=0 tx=0).stop 2023-03-03T10:32:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 -- 172.21.15.7:0/218343110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 msgr2=0x7fa35414cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:32:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 0x7fa35414cb20 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa344014040 tx=0x7fa344004750 comp rx=0 tx=0).stop 2023-03-03T10:32:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 -- 172.21.15.7:0/218343110 shutdown_connections 2023-03-03T10:32:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3540c5cb0 0x7fa35414d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa35406a2b0 0x7fa354154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3380682f0 0x7fa33806a7a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 --2- 172.21.15.7:0/218343110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3540a6370 0x7fa35414cb20 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:32:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 -- 172.21.15.7:0/218343110 >> 172.21.15.7:0/218343110 conn(0x7fa3540a9430 msgr2=0x7fa3540c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:32:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 -- 172.21.15.7:0/218343110 shutdown_connections 2023-03-03T10:32:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:32:59.673+0000 7fa35b9ae700 1 -- 172.21.15.7:0/218343110 wait complete. 2023-03-03T10:32:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:32:59.687 INFO:tasks.mds_thrash.fs.[b]:waiting for 114 secs before thrashing 2023-03-03T10:33:01.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:01.541 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:31.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:33:31.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:00.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:34:00.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.843+0000 7fa7c1093700 1 Processor -- start 2023-03-03T10:34:00.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.843+0000 7fa7c1093700 1 -- start start 2023-03-03T10:34:00.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7c1093700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 0x7fa7bc157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:34:00.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7c1093700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc0a9c20 con 0x7fa7bc0aa030 2023-03-03T10:34:00.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7c1093700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7bc154f10 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7c1093700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc0ac3c0 con 0x7fa7bc0aad20 2023-03-03T10:34:00.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7bb59e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa7bc0aa030 0x7fa7bc158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54774/0 (socket says 172.21.15.7:54774) 2023-03-03T10:34:00.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7bb59e700 1 -- 172.21.15.7:0/1893499622 learned_addr learned my addr 172.21.15.7:0/1893499622 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:34:00.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7ba59c700 1 --2- 172.21.15.7:0/1893499622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 0x7fa7bc157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:34:00.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7b9d9b700 1 -- 172.21.15.7:0/1893499622 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980984124 0 0) 0x7fa7bc0a9c20 con 0x7fa7bc0aa030 2023-03-03T10:34:00.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7b9d9b700 1 -- 172.21.15.7:0/1893499622 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a40035b0 con 0x7fa7bc0aa030 2023-03-03T10:34:00.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7b9d9b700 1 -- 172.21.15.7:0/1893499622 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112434084 0 0) 0x7fa7bc154f10 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7b9d9b700 1 -- 172.21.15.7:0/1893499622 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7bc0a9c20 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7ba59c700 1 -- 172.21.15.7:0/1893499622 >> v1:172.21.15.143:6789/0 conn(0x7fa7bc14b0d0 legacy=0x7fa7bc071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:00.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7ba59c700 1 -- 172.21.15.7:0/1893499622 >> v1:172.21.15.7:6789/0 conn(0x7fa7bc0aa030 legacy=0x7fa7bc158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:00.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.844+0000 7fa7ba59c700 1 -- 172.21.15.7:0/1893499622 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc06fc70 con 0x7fa7bc0aad20 2023-03-03T10:34:00.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7ba59c700 1 --2- 172.21.15.7:0/1893499622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 0x7fa7bc157df0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009870 tx=0x7fa7b0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dedf999ad84b757 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:34:00.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7b9d9b700 1 -- 172.21.15.7:0/1893499622 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0011830 con 0x7fa7bc0aad20 2023-03-03T10:34:00.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7b9d9b700 1 -- 172.21.15.7:0/1893499622 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b0011990 con 0x7fa7bc0aad20 2023-03-03T10:34:00.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7b9d9b700 1 -- 172.21.15.7:0/1893499622 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0011d00 con 0x7fa7bc0aad20 2023-03-03T10:34:00.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7c1093700 1 -- 172.21.15.7:0/1893499622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 msgr2=0x7fa7bc157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:00.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7c1093700 1 --2- 172.21.15.7:0/1893499622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 0x7fa7bc157df0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009870 tx=0x7fa7b0009b80 comp rx=0 tx=0).stop 2023-03-03T10:34:00.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7c1093700 1 -- 172.21.15.7:0/1893499622 shutdown_connections 2023-03-03T10:34:00.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7c1093700 1 --2- 172.21.15.7:0/1893499622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 0x7fa7bc157df0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:00.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7c1093700 1 -- 172.21.15.7:0/1893499622 >> 172.21.15.7:0/1893499622 conn(0x7fa7bc0c8900 msgr2=0x7fa7bc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:34:00.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7c1093700 1 -- 172.21.15.7:0/1893499622 shutdown_connections 2023-03-03T10:34:00.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.845+0000 7fa7c1093700 1 -- 172.21.15.7:0/1893499622 wait complete. 2023-03-03T10:34:00.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.847+0000 7fa7c1093700 1 Processor -- start 2023-03-03T10:34:00.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.847+0000 7fa7c1093700 1 -- start start 2023-03-03T10:34:00.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7c1093700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7bc0aa030 0x7fa7bc147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:34:00.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7c1093700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 0x7fa7bc1ed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:34:00.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7c1093700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7bc14b0d0 0x7fa7bc156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:34:00.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7c1093700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc155f40 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7c1093700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc1edef0 con 0x7fa7bc0aa030 2023-03-03T10:34:00.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7c1093700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7bc1568a0 con 0x7fa7bc0aad20 2023-03-03T10:34:00.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7bb59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7bc14b0d0 0x7fa7bc156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:34:00.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7bb59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7bc14b0d0 0x7fa7bc156360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60800/0 (socket says 172.21.15.7:60800) 2023-03-03T10:34:00.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7bb59e700 1 -- 172.21.15.7:0/3009548882 learned_addr learned my addr 172.21.15.7:0/3009548882 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:34:00.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7ba59c700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 0x7fa7bc1ed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:34:00.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7bad9d700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7bc0aa030 0x7fa7bc147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:34:00.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7bb59e700 1 -- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 msgr2=0x7fa7bc1ed9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:00.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7bb59e700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 0x7fa7bc1ed9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:00.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7bb59e700 1 -- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7bc0aa030 msgr2=0x7fa7bc147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:00.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7bb59e700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7bc0aa030 0x7fa7bc147420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:00.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7bb59e700 1 -- 172.21.15.7:0/3009548882 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b00094a0 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7bb59e700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7bc14b0d0 0x7fa7bc156360 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400cb50 tx=0x7fa7b400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:34:00.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7a37fe700 1 -- 172.21.15.7:0/3009548882 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b400aa20 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7a37fe700 1 -- 172.21.15.7:0/3009548882 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b400ab80 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7a37fe700 1 -- 172.21.15.7:0/3009548882 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4012400 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.848+0000 7fa7c1093700 1 -- 172.21.15.7:0/3009548882 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7bc06f6b0 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.849+0000 7fa7c1093700 1 -- 172.21.15.7:0/3009548882 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7bc155160 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.849+0000 7fa7c1093700 1 -- 172.21.15.7:0/3009548882 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7bc06f990 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.850+0000 7fa7a37fe700 1 -- 172.21.15.7:0/3009548882 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa7b4019030 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.850+0000 7fa7a37fe700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7a80682f0 0x7fa7a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:34:00.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.850+0000 7fa7a37fe700 1 -- 172.21.15.7:0/3009548882 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa7b4085cc0 con 0x7fa7bc14b0d0 2023-03-03T10:34:00.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.851+0000 7fa7bad9d700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7a80682f0 0x7fa7a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:34:00.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.851+0000 7fa7bad9d700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7a80682f0 0x7fa7a806a7a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc1462a0 tx=0x7fa7ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:34:00.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:00.855+0000 7fa7a37fe700 1 -- 172.21.15.7:0/3009548882 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7b4052060 con 0x7fa7bc14b0d0 2023-03-03T10:34:01.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.093+0000 7fa7c1093700 1 -- 172.21.15.7:0/3009548882 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7bc156080 con 0x7fa7bc14b0d0 2023-03-03T10:34:01.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.093+0000 7fa7a37fe700 1 -- 172.21.15.7:0/3009548882 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa7bc156080 con 0x7fa7bc14b0d0 2023-03-03T10:34:01.097 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:34:01.097 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:34:01.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 -- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7a80682f0 msgr2=0x7fa7a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:01.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7a80682f0 0x7fa7a806a7a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc1462a0 tx=0x7fa7ac009040 comp rx=0 tx=0).stop 2023-03-03T10:34:01.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 -- 172.21.15.7:0/3009548882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7bc14b0d0 msgr2=0x7fa7bc156360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:01.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7bc14b0d0 0x7fa7bc156360 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400cb50 tx=0x7fa7b400ce60 comp rx=0 tx=0).stop 2023-03-03T10:34:01.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 -- 172.21.15.7:0/3009548882 shutdown_connections 2023-03-03T10:34:01.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7a80682f0 0x7fa7a806a7a0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:01.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7bc14b0d0 0x7fa7bc156360 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:01.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7bc0aa030 0x7fa7bc147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:01.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 --2- 172.21.15.7:0/3009548882 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7bc0aad20 0x7fa7bc1ed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:01.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 -- 172.21.15.7:0/3009548882 >> 172.21.15.7:0/3009548882 conn(0x7fa7bc0c8900 msgr2=0x7fa7bc0c4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:34:01.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 -- 172.21.15.7:0/3009548882 shutdown_connections 2023-03-03T10:34:01.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:01.095+0000 7fa7c1093700 1 -- 172.21.15.7:0/3009548882 wait complete. 2023-03-03T10:34:01.100 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:34:01.110 INFO:tasks.mds_thrash.fs.[a]:waiting for 77 secs before thrashing 2023-03-03T10:34:01.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:01.706 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:31.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:31.782 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:34:53.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:34:53.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.824+0000 7f9ed8ad5700 1 Processor -- start 2023-03-03T10:34:53.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.824+0000 7f9ed8ad5700 1 -- start start 2023-03-03T10:34:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed8ad5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9ed41530d0 0x7f9ed4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:34:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed8ad5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed40c2f30 con 0x7f9ed414fda0 2023-03-03T10:34:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed8ad5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed40c31d0 con 0x7f9ed414f9d0 2023-03-03T10:34:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed8ad5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed41524a0 con 0x7f9ed41530d0 2023-03-03T10:34:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed1d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9ed414fda0 0x7f9ed41529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54822/0 (socket says 172.21.15.7:54822) 2023-03-03T10:34:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed1d9b700 1 -- 172.21.15.7:0/3663718274 learned_addr learned my addr 172.21.15.7:0/3663718274 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:34:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed2d9d700 1 --2- 172.21.15.7:0/3663718274 >> v2:172.21.15.7:3300/0 conn(0x7f9ed41530d0 0x7f9ed4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:34:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed159a700 1 -- 172.21.15.7:0/3663718274 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683211678 0 0) 0x7f9ed40c2f30 con 0x7f9ed414fda0 2023-03-03T10:34:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed159a700 1 -- 172.21.15.7:0/3663718274 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ebc0035b0 con 0x7f9ed414fda0 2023-03-03T10:34:53.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed2d9d700 1 -- 172.21.15.7:0/3663718274 >> v1:172.21.15.143:6789/0 conn(0x7f9ed414f9d0 legacy=0x7f9ed4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:34:53.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed2d9d700 1 -- 172.21.15.7:0/3663718274 >> v1:172.21.15.7:6789/0 conn(0x7f9ed414fda0 legacy=0x7f9ed41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:53.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed2d9d700 1 -- 172.21.15.7:0/3663718274 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed40c34b0 con 0x7f9ed41530d0 2023-03-03T10:34:53.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.825+0000 7f9ed2d9d700 1 --2- 172.21.15.7:0/3663718274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 0x7f9ed4156430 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009d90 tx=0x7f9ecc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305abcee68e95d57 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:34:53.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.826+0000 7f9ed159a700 1 -- 172.21.15.7:0/3663718274 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc00e070 con 0x7f9ed41530d0 2023-03-03T10:34:53.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.826+0000 7f9ed159a700 1 -- 172.21.15.7:0/3663718274 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc00bb20 con 0x7f9ed41530d0 2023-03-03T10:34:53.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.826+0000 7f9ed159a700 1 -- 172.21.15.7:0/3663718274 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc012430 con 0x7f9ed41530d0 2023-03-03T10:34:53.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.826+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3663718274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 msgr2=0x7f9ed4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:53.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.826+0000 7f9ed8ad5700 1 --2- 172.21.15.7:0/3663718274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 0x7f9ed4156430 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009d90 tx=0x7f9ecc00b330 comp rx=0 tx=0).stop 2023-03-03T10:34:53.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.826+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3663718274 shutdown_connections 2023-03-03T10:34:53.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.826+0000 7f9ed8ad5700 1 --2- 172.21.15.7:0/3663718274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 0x7f9ed4156430 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:53.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.826+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3663718274 >> 172.21.15.7:0/3663718274 conn(0x7f9ed40c8900 msgr2=0x7f9ed40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:34:53.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.826+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3663718274 shutdown_connections 2023-03-03T10:34:53.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.826+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3663718274 wait complete. 2023-03-03T10:34:53.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.828+0000 7f9ed8ad5700 1 Processor -- start 2023-03-03T10:34:53.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.828+0000 7f9ed8ad5700 1 -- start start 2023-03-03T10:34:53.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.828+0000 7f9ed8ad5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ed414f9d0 0x7f9ed41ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:34:53.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.828+0000 7f9ed8ad5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ed414fda0 0x7f9ed41ed410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:34:53.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.828+0000 7f9ed8ad5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 0x7f9ed40c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:34:53.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.828+0000 7f9ed8ad5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed406a700 con 0x7f9ed41530d0 2023-03-03T10:34:53.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.828+0000 7f9ed8ad5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed41df1f0 con 0x7f9ed414f9d0 2023-03-03T10:34:53.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.828+0000 7f9ed8ad5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed41e9ea0 con 0x7f9ed414fda0 2023-03-03T10:34:53.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 0x7f9ed40c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:34:53.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 0x7f9ed40c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60852/0 (socket says 172.21.15.7:60852) 2023-03-03T10:34:53.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed2d9d700 1 -- 172.21.15.7:0/3844719778 learned_addr learned my addr 172.21.15.7:0/3844719778 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:34:53.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed259c700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ed414f9d0 0x7f9ed41ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:34:53.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed1d9b700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ed414fda0 0x7f9ed41ed410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:34:53.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed2d9d700 1 -- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ed414fda0 msgr2=0x7f9ed41ed410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:53.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed2d9d700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ed414fda0 0x7f9ed41ed410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:53.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed2d9d700 1 -- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ed414f9d0 msgr2=0x7f9ed41ea8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:53.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed2d9d700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ed414f9d0 0x7f9ed41ea8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:53.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed2d9d700 1 -- 172.21.15.7:0/3844719778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc009a70 con 0x7f9ed41530d0 2023-03-03T10:34:53.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed2d9d700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 0x7f9ed40c1a50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc013000 tx=0x7f9ecc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:34:53.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ec2ffd700 1 -- 172.21.15.7:0/3844719778 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc012430 con 0x7f9ed41530d0 2023-03-03T10:34:53.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ec2ffd700 1 -- 172.21.15.7:0/3844719778 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc020070 con 0x7f9ed41530d0 2023-03-03T10:34:53.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3844719778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed40ac350 con 0x7f9ed41530d0 2023-03-03T10:34:53.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ec2ffd700 1 -- 172.21.15.7:0/3844719778 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc004480 con 0x7f9ed41530d0 2023-03-03T10:34:53.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.829+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3844719778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed40ab630 con 0x7f9ed41530d0 2023-03-03T10:34:53.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.830+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3844719778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed4071fa0 con 0x7f9ed41530d0 2023-03-03T10:34:53.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.831+0000 7f9ec2ffd700 1 -- 172.21.15.7:0/3844719778 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ecc02e070 con 0x7f9ed41530d0 2023-03-03T10:34:53.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.831+0000 7f9ec2ffd700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9eb40682f0 0x7f9eb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:34:53.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.831+0000 7f9ec2ffd700 1 -- 172.21.15.7:0/3844719778 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9ecc022070 con 0x7f9ed41530d0 2023-03-03T10:34:53.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.831+0000 7f9ed259c700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9eb40682f0 0x7f9eb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:34:53.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.832+0000 7f9ed259c700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9eb40682f0 0x7f9eb406a7a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4000f60 tx=0x7f9ec4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:34:53.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:53.836+0000 7f9ec2ffd700 1 -- 172.21.15.7:0/3844719778 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ecc08b030 con 0x7f9ed41530d0 2023-03-03T10:34:54.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.074+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3844719778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ed40aa8a0 con 0x7f9ed41530d0 2023-03-03T10:34:54.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.075+0000 7f9ec2ffd700 1 -- 172.21.15.7:0/3844719778 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9ecc054280 con 0x7f9ed41530d0 2023-03-03T10:34:54.078 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:34:54.079 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:34:54.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9eb40682f0 msgr2=0x7f9eb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:54.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9eb40682f0 0x7f9eb406a7a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4000f60 tx=0x7f9ec4009040 comp rx=0 tx=0).stop 2023-03-03T10:34:54.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3844719778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 msgr2=0x7f9ed40c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:34:54.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 0x7f9ed40c1a50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc013000 tx=0x7f9ecc004ad0 comp rx=0 tx=0).stop 2023-03-03T10:34:54.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3844719778 shutdown_connections 2023-03-03T10:34:54.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9eb40682f0 0x7f9eb406a7a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:54.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ed41530d0 0x7f9ed40c1a50 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:54.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ed414f9d0 0x7f9ed41ea8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:54.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 --2- 172.21.15.7:0/3844719778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ed414fda0 0x7f9ed41ed410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:34:54.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3844719778 >> 172.21.15.7:0/3844719778 conn(0x7f9ed40c8900 msgr2=0x7f9ed414eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:34:54.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3844719778 shutdown_connections 2023-03-03T10:34:54.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:34:54.077+0000 7f9ed8ad5700 1 -- 172.21.15.7:0/3844719778 wait complete. 2023-03-03T10:34:54.082 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:34:54.092 INFO:tasks.mds_thrash.fs.[b]:waiting for 80 secs before thrashing 2023-03-03T10:35:01.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:01.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:18.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:35:18.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.247+0000 7fca60382700 1 Processor -- start 2023-03-03T10:35:18.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.247+0000 7fca60382700 1 -- start start 2023-03-03T10:35:18.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca60382700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca5806d4b0 0x7fca580c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:18.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca60382700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca580c0ee0 0x7fca58139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:18.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca60382700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca580c7c20 con 0x7fca5806d880 2023-03-03T10:35:18.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca60382700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca581544c0 con 0x7fca5806d4b0 2023-03-03T10:35:18.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca60382700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca580c7310 con 0x7fca580c0ee0 2023-03-03T10:35:18.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca5e11e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca5806d4b0 0x7fca580c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:18.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca5e11e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca5806d4b0 0x7fca580c1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60874/0 (socket says 172.21.15.7:60874) 2023-03-03T10:35:18.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca5e11e700 1 -- 172.21.15.7:0/3494375083 learned_addr learned my addr 172.21.15.7:0/3494375083 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:35:18.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca5e91f700 1 --2- 172.21.15.7:0/3494375083 >> v2:172.21.15.143:3300/0 conn(0x7fca580c0ee0 0x7fca58139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:18.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca5e11e700 1 -- 172.21.15.7:0/3494375083 >> v2:172.21.15.143:3300/0 conn(0x7fca580c0ee0 msgr2=0x7fca58139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:35:18.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca5e11e700 1 --2- 172.21.15.7:0/3494375083 >> v2:172.21.15.143:3300/0 conn(0x7fca580c0ee0 0x7fca58139ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:18.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca5e11e700 1 -- 172.21.15.7:0/3494375083 >> v1:172.21.15.143:6789/0 conn(0x7fca5806d880 legacy=0x7fca580c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:35:18.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca5e11e700 1 -- 172.21.15.7:0/3494375083 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca58154740 con 0x7fca5806d4b0 2023-03-03T10:35:18.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.248+0000 7fca5e11e700 1 --2- 172.21.15.7:0/3494375083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca5806d4b0 0x7fca580c1da0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fca58074370 tx=0x7fca48009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5337bfbc153bfc4d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:18.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca5d11c700 1 -- 172.21.15.7:0/3494375083 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48004230 con 0x7fca5806d4b0 2023-03-03T10:35:18.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca5d11c700 1 -- 172.21.15.7:0/3494375083 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca48004390 con 0x7fca5806d4b0 2023-03-03T10:35:18.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca5d11c700 1 -- 172.21.15.7:0/3494375083 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca480036a0 con 0x7fca5806d4b0 2023-03-03T10:35:18.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca60382700 1 -- 172.21.15.7:0/3494375083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca5806d4b0 msgr2=0x7fca580c1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:18.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca60382700 1 --2- 172.21.15.7:0/3494375083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca5806d4b0 0x7fca580c1da0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fca58074370 tx=0x7fca48009bd0 comp rx=0 tx=0).stop 2023-03-03T10:35:18.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca60382700 1 -- 172.21.15.7:0/3494375083 shutdown_connections 2023-03-03T10:35:18.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca60382700 1 --2- 172.21.15.7:0/3494375083 >> v2:172.21.15.143:3300/0 conn(0x7fca580c0ee0 0x7fca58139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:18.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca60382700 1 --2- 172.21.15.7:0/3494375083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca5806d4b0 0x7fca580c1da0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:18.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca60382700 1 -- 172.21.15.7:0/3494375083 >> 172.21.15.7:0/3494375083 conn(0x7fca5806c110 msgr2=0x7fca5806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:35:18.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca60382700 1 -- 172.21.15.7:0/3494375083 shutdown_connections 2023-03-03T10:35:18.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.249+0000 7fca60382700 1 -- 172.21.15.7:0/3494375083 wait complete. 2023-03-03T10:35:18.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.250+0000 7fca60382700 1 Processor -- start 2023-03-03T10:35:18.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.250+0000 7fca60382700 1 -- start start 2023-03-03T10:35:18.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca60382700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca5806d4b0 0x7fca580c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:18.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca60382700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca5806d880 0x7fca58150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:18.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca60382700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca580c0ee0 0x7fca58150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:18.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca60382700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca58133ea0 con 0x7fca580c0ee0 2023-03-03T10:35:18.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca60382700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca580aaf50 con 0x7fca5806d4b0 2023-03-03T10:35:18.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca60382700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca58133800 con 0x7fca5806d880 2023-03-03T10:35:18.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca5e91f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca580c0ee0 0x7fca58150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:18.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca5e91f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca580c0ee0 0x7fca58150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60880/0 (socket says 172.21.15.7:60880) 2023-03-03T10:35:18.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca5e91f700 1 -- 172.21.15.7:0/2418992468 learned_addr learned my addr 172.21.15.7:0/2418992468 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:35:18.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca5e11e700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca5806d4b0 0x7fca580c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:18.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.251+0000 7fca5d91d700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca5806d880 0x7fca58150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca5e91f700 1 -- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca5806d880 msgr2=0x7fca58150120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca5e91f700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca5806d880 0x7fca58150120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca5e91f700 1 -- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca5806d4b0 msgr2=0x7fca580c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca5e91f700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca5806d4b0 0x7fca580c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca5e91f700 1 -- 172.21.15.7:0/2418992468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca48009550 con 0x7fca580c0ee0 2023-03-03T10:35:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca5e91f700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca580c0ee0 0x7fca58150660 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fca5000cd60 tx=0x7fca5000a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca4effd700 1 -- 172.21.15.7:0/2418992468 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca50003a20 con 0x7fca580c0ee0 2023-03-03T10:35:18.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca4effd700 1 -- 172.21.15.7:0/2418992468 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca50003ba0 con 0x7fca580c0ee0 2023-03-03T10:35:18.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca60382700 1 -- 172.21.15.7:0/2418992468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca580c78b0 con 0x7fca580c0ee0 2023-03-03T10:35:18.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca4effd700 1 -- 172.21.15.7:0/2418992468 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca50004020 con 0x7fca580c0ee0 2023-03-03T10:35:18.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.252+0000 7fca60382700 1 -- 172.21.15.7:0/2418992468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca580c75d0 con 0x7fca580c0ee0 2023-03-03T10:35:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.253+0000 7fca60382700 1 -- 172.21.15.7:0/2418992468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5806dd30 con 0x7fca580c0ee0 2023-03-03T10:35:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.253+0000 7fca4effd700 1 -- 172.21.15.7:0/2418992468 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fca5001c020 con 0x7fca580c0ee0 2023-03-03T10:35:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.254+0000 7fca4effd700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca3c0682f0 0x7fca3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:35:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.254+0000 7fca4effd700 1 -- 172.21.15.7:0/2418992468 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fca50012070 con 0x7fca580c0ee0 2023-03-03T10:35:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.254+0000 7fca5e11e700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca3c0682f0 0x7fca3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:35:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.255+0000 7fca5e11e700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca3c0682f0 0x7fca3c06a7a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fca48009130 tx=0x7fca48008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:35:18.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.258+0000 7fca4effd700 1 -- 172.21.15.7:0/2418992468 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fca50052bc0 con 0x7fca580c0ee0 2023-03-03T10:35:18.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.496+0000 7fca60382700 1 -- 172.21.15.7:0/2418992468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca5806fa00 con 0x7fca580c0ee0 2023-03-03T10:35:18.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.496+0000 7fca4effd700 1 -- 172.21.15.7:0/2418992468 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fca500561e0 con 0x7fca580c0ee0 2023-03-03T10:35:18.499 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:35:18.499 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:35:18.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 -- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca3c0682f0 msgr2=0x7fca3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:18.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca3c0682f0 0x7fca3c06a7a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fca48009130 tx=0x7fca48008030 comp rx=0 tx=0).stop 2023-03-03T10:35:18.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 -- 172.21.15.7:0/2418992468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca580c0ee0 msgr2=0x7fca58150660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:35:18.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca580c0ee0 0x7fca58150660 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fca5000cd60 tx=0x7fca5000a570 comp rx=0 tx=0).stop 2023-03-03T10:35:18.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 -- 172.21.15.7:0/2418992468 shutdown_connections 2023-03-03T10:35:18.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca3c0682f0 0x7fca3c06a7a0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:18.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca580c0ee0 0x7fca58150660 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:18.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca5806d4b0 0x7fca580c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:18.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 --2- 172.21.15.7:0/2418992468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca5806d880 0x7fca58150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:35:18.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 -- 172.21.15.7:0/2418992468 >> 172.21.15.7:0/2418992468 conn(0x7fca5806c110 msgr2=0x7fca5806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:35:18.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 -- 172.21.15.7:0/2418992468 shutdown_connections 2023-03-03T10:35:18.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:35:18.498+0000 7fca60382700 1 -- 172.21.15.7:0/2418992468 wait complete. 2023-03-03T10:35:18.502 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:35:18.512 INFO:tasks.mds_thrash.fs.[a]:waiting for 72 secs before thrashing 2023-03-03T10:35:31.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:35:31.930 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:02.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:02.147 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:14.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:36:14.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.229+0000 7f9ec8ef2700 1 Processor -- start 2023-03-03T10:36:14.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.229+0000 7f9ec8ef2700 1 -- start start 2023-03-03T10:36:14.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec8ef2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9ec40c17f0 0x7f9ec406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec8ef2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec414ebe0 con 0x7f9ec40c5190 2023-03-03T10:36:14.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec8ef2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec40c2e70 con 0x7f9ec40c6010 2023-03-03T10:36:14.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec8ef2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec4152100 con 0x7f9ec40c17f0 2023-03-03T10:36:14.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec2d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9ec40c5190 0x7f9ec406dbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54882/0 (socket says 172.21.15.7:54882) 2023-03-03T10:36:14.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec2d9d700 1 -- 172.21.15.7:0/1366146607 learned_addr learned my addr 172.21.15.7:0/1366146607 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:36:14.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec1d9b700 1 --2- 172.21.15.7:0/1366146607 >> v2:172.21.15.143:3300/0 conn(0x7f9ec40c17f0 0x7f9ec406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec159a700 1 -- 172.21.15.7:0/1366146607 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126727405 0 0) 0x7f9ec414ebe0 con 0x7f9ec40c5190 2023-03-03T10:36:14.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec159a700 1 -- 172.21.15.7:0/1366146607 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eac0035b0 con 0x7f9ec40c5190 2023-03-03T10:36:14.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec159a700 1 -- 172.21.15.7:0/1366146607 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594411975 0 0) 0x7f9ec40c2e70 con 0x7f9ec40c6010 2023-03-03T10:36:14.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec159a700 1 -- 172.21.15.7:0/1366146607 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec414ebe0 con 0x7f9ec40c6010 2023-03-03T10:36:14.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec159a700 1 -- 172.21.15.7:0/1366146607 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 814409546 0 0) 0x7f9eac0035b0 con 0x7f9ec40c5190 2023-03-03T10:36:14.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec1d9b700 1 -- 172.21.15.7:0/1366146607 >> v1:172.21.15.143:6789/0 conn(0x7f9ec40c6010 legacy=0x7f9ec40c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.230+0000 7f9ec1d9b700 1 -- 172.21.15.7:0/1366146607 >> v1:172.21.15.7:6789/0 conn(0x7f9ec40c5190 legacy=0x7f9ec406dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.231+0000 7f9ec1d9b700 1 -- 172.21.15.7:0/1366146607 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec40aa680 con 0x7f9ec40c17f0 2023-03-03T10:36:14.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.231+0000 7f9ec1d9b700 1 --2- 172.21.15.7:0/1366146607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40c17f0 0x7f9ec406d6a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8009840 tx=0x7f9eb8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e48eb82f19973c4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:14.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.231+0000 7f9ec159a700 1 -- 172.21.15.7:0/1366146607 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc0019f0 con 0x7f9ec40c17f0 2023-03-03T10:36:14.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.231+0000 7f9ec159a700 1 -- 172.21.15.7:0/1366146607 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb8004030 con 0x7f9ec40c17f0 2023-03-03T10:36:14.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.231+0000 7f9ec159a700 1 -- 172.21.15.7:0/1366146607 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb80043c0 con 0x7f9ec40c17f0 2023-03-03T10:36:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.231+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/1366146607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40c17f0 msgr2=0x7f9ec406d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.231+0000 7f9ec8ef2700 1 --2- 172.21.15.7:0/1366146607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40c17f0 0x7f9ec406d6a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8009840 tx=0x7f9eb8009b50 comp rx=0 tx=0).stop 2023-03-03T10:36:14.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.232+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/1366146607 shutdown_connections 2023-03-03T10:36:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.232+0000 7f9ec8ef2700 1 --2- 172.21.15.7:0/1366146607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40c17f0 0x7f9ec406d6a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.232+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/1366146607 >> 172.21.15.7:0/1366146607 conn(0x7f9ec40a9430 msgr2=0x7f9ec40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:36:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.232+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/1366146607 shutdown_connections 2023-03-03T10:36:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.232+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/1366146607 wait complete. 2023-03-03T10:36:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.233+0000 7f9ec8ef2700 1 Processor -- start 2023-03-03T10:36:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.233+0000 7f9ec8ef2700 1 -- start start 2023-03-03T10:36:14.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec8ef2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40c17f0 0x7f9ec406ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec8ef2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ec40c5190 0x7f9ec406fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec8ef2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40c6010 0x7f9ec406ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec8ef2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec41dcdd0 con 0x7f9ec40c5190 2023-03-03T10:36:14.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec8ef2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec41dd020 con 0x7f9ec40c17f0 2023-03-03T10:36:14.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec8ef2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec41dd300 con 0x7f9ec40c6010 2023-03-03T10:36:14.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ec40c5190 0x7f9ec406fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ec40c5190 0x7f9ec406fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60906/0 (socket says 172.21.15.7:60906) 2023-03-03T10:36:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec1d9b700 1 -- 172.21.15.7:0/842417287 learned_addr learned my addr 172.21.15.7:0/842417287 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:36:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec259c700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40c17f0 0x7f9ec406ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec2d9d700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40c6010 0x7f9ec406ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec1d9b700 1 -- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40c6010 msgr2=0x7f9ec406ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec1d9b700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40c6010 0x7f9ec406ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec1d9b700 1 -- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40c17f0 msgr2=0x7f9ec406ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec1d9b700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40c17f0 0x7f9ec406ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec1d9b700 1 -- 172.21.15.7:0/842417287 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb80094a0 con 0x7f9ec40c5190 2023-03-03T10:36:14.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec2d9d700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40c6010 0x7f9ec406ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:36:14.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec1d9b700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ec40c5190 0x7f9ec406fa20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80090b0 tx=0x7f9eb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:14.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9eb2ffd700 1 -- 172.21.15.7:0/842417287 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8011830 con 0x7f9ec40c5190 2023-03-03T10:36:14.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9eb2ffd700 1 -- 172.21.15.7:0/842417287 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb801f070 con 0x7f9ec40c5190 2023-03-03T10:36:14.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/842417287 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec40c2b00 con 0x7f9ec40c5190 2023-03-03T10:36:14.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.234+0000 7f9eb2ffd700 1 -- 172.21.15.7:0/842417287 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8011bf0 con 0x7f9ec40c5190 2023-03-03T10:36:14.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.235+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/842417287 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec40aa960 con 0x7f9ec40c5190 2023-03-03T10:36:14.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.235+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/842417287 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec40c3300 con 0x7f9ec40c5190 2023-03-03T10:36:14.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.236+0000 7f9eb2ffd700 1 -- 172.21.15.7:0/842417287 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9eb8017020 con 0x7f9ec40c5190 2023-03-03T10:36:14.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.236+0000 7f9eb2ffd700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ea40682f0 0x7f9ea406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.236+0000 7f9eb2ffd700 1 -- 172.21.15.7:0/842417287 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9eb8021070 con 0x7f9ec40c5190 2023-03-03T10:36:14.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.236+0000 7f9ec259c700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ea40682f0 0x7f9ea406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.237+0000 7f9ec259c700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ea40682f0 0x7f9ea406a7a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4007fd0 tx=0x7f9eb4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:14.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.241+0000 7f9eb2ffd700 1 -- 172.21.15.7:0/842417287 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9eb8053b50 con 0x7f9ec40c5190 2023-03-03T10:36:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.481+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/842417287 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ec41dc5b0 con 0x7f9ec40c5190 2023-03-03T10:36:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.481+0000 7f9eb2ffd700 1 -- 172.21.15.7:0/842417287 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9ec41dc5b0 con 0x7f9ec40c5190 2023-03-03T10:36:14.484 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:36:14.484 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:36:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ea40682f0 msgr2=0x7f9ea406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ea40682f0 0x7f9ea406a7a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4007fd0 tx=0x7f9eb4009040 comp rx=0 tx=0).stop 2023-03-03T10:36:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/842417287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ec40c5190 msgr2=0x7f9ec406fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ec40c5190 0x7f9ec406fa20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f9eb80090b0 tx=0x7f9eb8004750 comp rx=0 tx=0).stop 2023-03-03T10:36:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/842417287 shutdown_connections 2023-03-03T10:36:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ec40c6010 0x7f9ec406ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ec40c17f0 0x7f9ec406ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ea40682f0 0x7f9ea406a7a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 --2- 172.21.15.7:0/842417287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ec40c5190 0x7f9ec406fa20 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/842417287 >> 172.21.15.7:0/842417287 conn(0x7f9ec40a9430 msgr2=0x7f9ec4065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:36:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/842417287 shutdown_connections 2023-03-03T10:36:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.483+0000 7f9ec8ef2700 1 -- 172.21.15.7:0/842417287 wait complete. 2023-03-03T10:36:14.488 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:36:14.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:36:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.619+0000 7fed8bbdd700 1 Processor -- start 2023-03-03T10:36:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.619+0000 7fed8bbdd700 1 -- start start 2023-03-03T10:36:14.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.620+0000 7fed8bbdd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fed840c1da0 0x7fed84156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.620+0000 7fed8bbdd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed84155840 con 0x7fed8414fb00 2023-03-03T10:36:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.620+0000 7fed8bbdd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed841567d0 con 0x7fed840740f0 2023-03-03T10:36:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.620+0000 7fed8bbdd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed84072940 con 0x7fed840c1da0 2023-03-03T10:36:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.620+0000 7fed89979700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fed8414fb00 0x7fed840c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54892/0 (socket says 172.21.15.7:54892) 2023-03-03T10:36:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.620+0000 7fed89979700 1 -- 172.21.15.7:0/363069976 learned_addr learned my addr 172.21.15.7:0/363069976 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:36:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed89178700 1 --2- 172.21.15.7:0/363069976 >> v2:172.21.15.143:3300/0 conn(0x7fed840c1da0 0x7fed84156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491267911 0 0) 0x7fed84155840 con 0x7fed8414fb00 2023-03-03T10:36:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed700035b0 con 0x7fed8414fb00 2023-03-03T10:36:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885304225 0 0) 0x7fed841567d0 con 0x7fed840740f0 2023-03-03T10:36:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed84155840 con 0x7fed840740f0 2023-03-03T10:36:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2813096347 0 0) 0x7fed700035b0 con 0x7fed8414fb00 2023-03-03T10:36:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 >> v2:172.21.15.143:3300/0 conn(0x7fed840c1da0 msgr2=0x7fed84156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:36:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 --2- 172.21.15.7:0/363069976 >> v2:172.21.15.143:3300/0 conn(0x7fed840c1da0 0x7fed84156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 >> v1:172.21.15.143:6789/0 conn(0x7fed840740f0 legacy=0x7fed8415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed840c2330 con 0x7fed8414fb00 2023-03-03T10:36:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fed74001670 con 0x7fed8414fb00 2023-03-03T10:36:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 >> v1:172.21.15.7:6789/0 conn(0x7fed8414fb00 legacy=0x7fed840c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed70003c00 0x7fed70003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed70004850 0x7fed70004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70005250 0x7fed7000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed84072940 con 0x7fed70004850 2023-03-03T10:36:14.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7000c160 con 0x7fed70005250 2023-03-03T10:36:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7000c320 con 0x7fed70003c00 2023-03-03T10:36:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed89178700 1 --2- 172.21.15.7:0/363069976 >> v2:172.21.15.143:3300/0 conn(0x7fed840c1da0 0x7fed84156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:36:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed89178700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed70004850 0x7fed70004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed89979700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed70003c00 0x7fed70003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed8a17a700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70005250 0x7fed7000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed89178700 1 -- 172.21.15.7:0/363069976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed70003c00 msgr2=0x7fed70003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:36:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed89178700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed70003c00 0x7fed70003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed89178700 1 -- 172.21.15.7:0/363069976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70005250 msgr2=0x7fed7000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:36:14.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed89178700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed70005250 0x7fed7000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.621+0000 7fed89178700 1 -- 172.21.15.7:0/363069976 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed700035b0 con 0x7fed70004850 2023-03-03T10:36:14.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed8a17a700 1 -- 172.21.15.7:0/363069976 reap_dead start 2023-03-03T10:36:14.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed89178700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed70004850 0x7fed70004cc0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fed80009d10 tx=0x7fed70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0a11b401fa0182 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:14.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80009e60 con 0x7fed70004850 2023-03-03T10:36:14.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8000e9a0 con 0x7fed70004850 2023-03-03T10:36:14.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed88977700 1 -- 172.21.15.7:0/363069976 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8000ed30 con 0x7fed70004850 2023-03-03T10:36:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed8bbdd700 1 -- 172.21.15.7:0/363069976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed70004850 msgr2=0x7fed70004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed8bbdd700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed70004850 0x7fed70004cc0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fed80009d10 tx=0x7fed70005880 comp rx=0 tx=0).stop 2023-03-03T10:36:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed8bbdd700 1 -- 172.21.15.7:0/363069976 shutdown_connections 2023-03-03T10:36:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed8bbdd700 1 --2- 172.21.15.7:0/363069976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed70004850 0x7fed70004cc0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed8bbdd700 1 -- 172.21.15.7:0/363069976 >> 172.21.15.7:0/363069976 conn(0x7fed8406c110 msgr2=0x7fed8406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:36:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed8bbdd700 1 -- 172.21.15.7:0/363069976 shutdown_connections 2023-03-03T10:36:14.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.622+0000 7fed8bbdd700 1 -- 172.21.15.7:0/363069976 wait complete. 2023-03-03T10:36:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed8bbdd700 1 Processor -- start 2023-03-03T10:36:14.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed8bbdd700 1 -- start start 2023-03-03T10:36:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed8bbdd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed70004850 0x7fed84147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed8bbdd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed841eb560 0x7fed8414a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed8bbdd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed84150900 0x7fed84148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed8bbdd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed84150470 con 0x7fed841eb560 2023-03-03T10:36:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed8bbdd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed841eb9d0 con 0x7fed84150900 2023-03-03T10:36:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed8bbdd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed84152af0 con 0x7fed70004850 2023-03-03T10:36:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed89178700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed841eb560 0x7fed8414a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed89178700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed841eb560 0x7fed8414a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60926/0 (socket says 172.21.15.7:60926) 2023-03-03T10:36:14.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.624+0000 7fed89178700 1 -- 172.21.15.7:0/3615529977 learned_addr learned my addr 172.21.15.7:0/3615529977 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:36:14.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed8a17a700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed84150900 0x7fed84148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed89979700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed70004850 0x7fed84147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed89178700 1 -- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed70004850 msgr2=0x7fed84147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed89178700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed70004850 0x7fed84147c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed89178700 1 -- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed84150900 msgr2=0x7fed84148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed89178700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed84150900 0x7fed84148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed89178700 1 -- 172.21.15.7:0/3615529977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84153090 con 0x7fed841eb560 2023-03-03T10:36:14.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed8a17a700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed84150900 0x7fed84148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:36:14.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed89178700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed841eb560 0x7fed8414a310 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fed7000c130 tx=0x7fed800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:14.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed7a7fc700 1 -- 172.21.15.7:0/3615529977 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80020070 con 0x7fed841eb560 2023-03-03T10:36:14.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed7a7fc700 1 -- 172.21.15.7:0/3615529977 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80009e60 con 0x7fed841eb560 2023-03-03T10:36:14.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed8bbdd700 1 -- 172.21.15.7:0/3615529977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed84152db0 con 0x7fed841eb560 2023-03-03T10:36:14.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed7a7fc700 1 -- 172.21.15.7:0/3615529977 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed800038e0 con 0x7fed841eb560 2023-03-03T10:36:14.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.625+0000 7fed8bbdd700 1 -- 172.21.15.7:0/3615529977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed84155a90 con 0x7fed841eb560 2023-03-03T10:36:14.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.626+0000 7fed8bbdd700 1 -- 172.21.15.7:0/3615529977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8406e410 con 0x7fed841eb560 2023-03-03T10:36:14.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.627+0000 7fed7a7fc700 1 -- 172.21.15.7:0/3615529977 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed8002d030 con 0x7fed841eb560 2023-03-03T10:36:14.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.627+0000 7fed7a7fc700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed680682f0 0x7fed6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:14.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.627+0000 7fed7a7fc700 1 -- 172.21.15.7:0/3615529977 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fed800873c0 con 0x7fed841eb560 2023-03-03T10:36:14.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.627+0000 7fed89979700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed680682f0 0x7fed6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:14.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.628+0000 7fed89979700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed680682f0 0x7fed6806a7a0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fed74000c00 tx=0x7fed74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:14.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.631+0000 7fed7a7fc700 1 -- 172.21.15.7:0/3615529977 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed80053760 con 0x7fed841eb560 2023-03-03T10:36:14.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.873+0000 7fed8bbdd700 1 -- 172.21.15.7:0/3615529977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed84074bb0 con 0x7fed841eb560 2023-03-03T10:36:14.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.873+0000 7fed7a7fc700 1 -- 172.21.15.7:0/3615529977 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fed80056d80 con 0x7fed841eb560 2023-03-03T10:36:14.876 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:36:14.876 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:36:14.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.875+0000 7fed8bbdd700 1 -- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed680682f0 msgr2=0x7fed6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.875+0000 7fed8bbdd700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed680682f0 0x7fed6806a7a0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fed74000c00 tx=0x7fed74009040 comp rx=0 tx=0).stop 2023-03-03T10:36:14.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.875+0000 7fed8bbdd700 1 -- 172.21.15.7:0/3615529977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed841eb560 msgr2=0x7fed8414a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:14.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.875+0000 7fed8bbdd700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed841eb560 0x7fed8414a310 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fed7000c130 tx=0x7fed800047c0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.875+0000 7fed8bbdd700 1 -- 172.21.15.7:0/3615529977 shutdown_connections 2023-03-03T10:36:14.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.875+0000 7fed8bbdd700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed84150900 0x7fed84148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.875+0000 7fed8bbdd700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed70004850 0x7fed84147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.875+0000 7fed8bbdd700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed680682f0 0x7fed6806a7a0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.875+0000 7fed8bbdd700 1 --2- 172.21.15.7:0/3615529977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed841eb560 0x7fed8414a310 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:14.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.875+0000 7fed8bbdd700 1 -- 172.21.15.7:0/3615529977 >> 172.21.15.7:0/3615529977 conn(0x7fed8406c110 msgr2=0x7fed8414f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:36:14.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.876+0000 7fed8bbdd700 1 -- 172.21.15.7:0/3615529977 shutdown_connections 2023-03-03T10:36:14.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:14.876+0000 7fed8bbdd700 1 -- 172.21.15.7:0/3615529977 wait complete. 2023-03-03T10:36:14.880 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:36:14.890 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2023-03-03T10:36:30.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:36:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.651+0000 7f1700947700 1 Processor -- start 2023-03-03T10:36:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.651+0000 7f1700947700 1 -- start start 2023-03-03T10:36:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.652+0000 7f1700947700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc151770 0x7f16fc151c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.652+0000 7f1700947700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16fc154910 con 0x7f16fc06e020 2023-03-03T10:36:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.652+0000 7f1700947700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16fc0c4540 con 0x7f16fc155390 2023-03-03T10:36:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.652+0000 7f1700947700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc070b90 con 0x7f16fc151770 2023-03-03T10:36:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.652+0000 7f16f9d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f16fc06e020 0x7f16fc071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54916/0 (socket says 172.21.15.7:54916) 2023-03-03T10:36:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.652+0000 7f16f9d9b700 1 -- 172.21.15.7:0/3985632248 learned_addr learned my addr 172.21.15.7:0/3985632248 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:36:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16fad9d700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc151770 0x7f16fc151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859811340 0 0) 0x7f16fc154910 con 0x7f16fc06e020 2023-03-03T10:36:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e40035b0 con 0x7f16fc06e020 2023-03-03T10:36:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601517053 0 0) 0x7f16ec001110 con 0x7f16fc155390 2023-03-03T10:36:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16fc154910 con 0x7f16fc155390 2023-03-03T10:36:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2212112775 0 0) 0x7f16e40035b0 con 0x7f16fc06e020 2023-03-03T10:36:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc151770 msgr2=0x7f16fc151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:36:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc151770 0x7f16fc151c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 >> v1:172.21.15.143:6789/0 conn(0x7f16fc155390 legacy=0x7f16fc0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc0702a0 con 0x7f16fc06e020 2023-03-03T10:36:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f16f0001670 con 0x7f16fc06e020 2023-03-03T10:36:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 >> v1:172.21.15.7:6789/0 conn(0x7f16fc06e020 legacy=0x7f16fc071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16e4003c30 0x7f16e4004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc151770 0x7f16e4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16e4004d90 0x7f16e4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc070b90 con 0x7f16e4004d90 2023-03-03T10:36:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e400ddb0 con 0x7f16e4003c30 2023-03-03T10:36:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e400df40 con 0x7f16fc151770 2023-03-03T10:36:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16fad9d700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16e4004d90 0x7f16e4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:30.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16fa59c700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16e4003c30 0x7f16e4004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:30.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.653+0000 7f16f9d9b700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc151770 0x7f16e4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:30.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.654+0000 7f16fa59c700 1 -- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc151770 msgr2=0x7f16e4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:36:30.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.654+0000 7f16fa59c700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc151770 0x7f16e4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:30.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.654+0000 7f16fa59c700 1 -- 172.21.15.7:0/3985632248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16e4004d90 msgr2=0x7f16e4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:36:30.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.654+0000 7f16fa59c700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16e4004d90 0x7f16e4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:30.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.654+0000 7f16fa59c700 1 -- 172.21.15.7:0/3985632248 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e40035b0 con 0x7f16e4003c30 2023-03-03T10:36:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.654+0000 7f16fad9d700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16e4004d90 0x7f16e4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:36:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.654+0000 7f16fa59c700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16e4003c30 0x7f16e4004020 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f16ec001970 tx=0x7f16e4005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fea00210fb53233b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.654+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec001600 con 0x7f16e4003c30 2023-03-03T10:36:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.654+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ec0052d0 con 0x7f16e4003c30 2023-03-03T10:36:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.654+0000 7f16f959a700 1 -- 172.21.15.7:0/3985632248 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec003680 con 0x7f16e4003c30 2023-03-03T10:36:30.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.655+0000 7f1700947700 1 -- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16e4003c30 msgr2=0x7f16e4004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.655+0000 7f1700947700 1 --2- 172.21.15.7:0/3985632248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16e4003c30 0x7f16e4004020 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f16ec001970 tx=0x7f16e4005880 comp rx=0 tx=0).stop 2023-03-03T10:36:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.655+0000 7f16fad9d700 1 -- 172.21.15.7:0/3985632248 reap_dead start 2023-03-03T10:36:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.655+0000 7f1700947700 1 -- 172.21.15.7:0/3985632248 shutdown_connections 2023-03-03T10:36:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.655+0000 7f1700947700 1 -- 172.21.15.7:0/3985632248 >> 172.21.15.7:0/3985632248 conn(0x7f16fc06c110 msgr2=0x7f16fc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:36:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.655+0000 7f1700947700 1 -- 172.21.15.7:0/3985632248 shutdown_connections 2023-03-03T10:36:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.655+0000 7f1700947700 1 -- 172.21.15.7:0/3985632248 wait complete. 2023-03-03T10:36:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.656+0000 7f1700947700 1 Processor -- start 2023-03-03T10:36:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.656+0000 7f1700947700 1 -- start start 2023-03-03T10:36:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f1700947700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16e4003c30 0x7f16fc153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f1700947700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc14ba50 0x7f16fc149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f1700947700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16fc149fe0 0x7f16fc1e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f1700947700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc14c4f0 con 0x7f16fc149fe0 2023-03-03T10:36:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f1700947700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc1e64a0 con 0x7f16e4003c30 2023-03-03T10:36:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f1700947700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc1dd2e0 con 0x7f16fc14ba50 2023-03-03T10:36:30.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16fad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16fc149fe0 0x7f16fc1e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:30.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16fad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16fc149fe0 0x7f16fc1e8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60950/0 (socket says 172.21.15.7:60950) 2023-03-03T10:36:30.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16fad9d700 1 -- 172.21.15.7:0/1217354250 learned_addr learned my addr 172.21.15.7:0/1217354250 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:36:30.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16fa59c700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16e4003c30 0x7f16fc153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:30.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16f9d9b700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc14ba50 0x7f16fc149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:30.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16fad9d700 1 -- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc14ba50 msgr2=0x7f16fc149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:30.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16fad9d700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc14ba50 0x7f16fc149aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:30.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16fad9d700 1 -- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16e4003c30 msgr2=0x7f16fc153ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:30.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16fad9d700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16e4003c30 0x7f16fc153ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:30.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16fad9d700 1 -- 172.21.15.7:0/1217354250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc070820 con 0x7f16fc149fe0 2023-03-03T10:36:30.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.657+0000 7f16fad9d700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16fc149fe0 0x7f16fc1e8a60 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f16e40047d0 tx=0x7f16f400f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:30.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.658+0000 7f16eaffd700 1 -- 172.21.15.7:0/1217354250 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f400e040 con 0x7f16fc149fe0 2023-03-03T10:36:30.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.658+0000 7f16eaffd700 1 -- 172.21.15.7:0/1217354250 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f400f040 con 0x7f16fc149fe0 2023-03-03T10:36:30.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.658+0000 7f1700947700 1 -- 172.21.15.7:0/1217354250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16fc070540 con 0x7f16fc149fe0 2023-03-03T10:36:30.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.658+0000 7f16eaffd700 1 -- 172.21.15.7:0/1217354250 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4011ec0 con 0x7f16fc149fe0 2023-03-03T10:36:30.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.658+0000 7f1700947700 1 -- 172.21.15.7:0/1217354250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16fc155760 con 0x7f16fc149fe0 2023-03-03T10:36:30.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.659+0000 7f1700947700 1 -- 172.21.15.7:0/1217354250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16fc14bfa0 con 0x7f16fc149fe0 2023-03-03T10:36:30.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.659+0000 7f16eaffd700 1 -- 172.21.15.7:0/1217354250 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f16f4015430 con 0x7f16fc149fe0 2023-03-03T10:36:30.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.660+0000 7f16eaffd700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16dc0682f0 0x7f16dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:36:30.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.660+0000 7f16eaffd700 1 -- 172.21.15.7:0/1217354250 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f16f40864c0 con 0x7f16fc149fe0 2023-03-03T10:36:30.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.660+0000 7f16fa59c700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16dc0682f0 0x7f16dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:36:30.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.660+0000 7f16fa59c700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16dc0682f0 0x7f16dc06a7a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f16fc1dcca0 tx=0x7f16ec00a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:36:30.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.664+0000 7f16eaffd700 1 -- 172.21.15.7:0/1217354250 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f16f4052710 con 0x7f16fc149fe0 2023-03-03T10:36:30.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.909+0000 7f1700947700 1 -- 172.21.15.7:0/1217354250 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16fc074bb0 con 0x7f16fc149fe0 2023-03-03T10:36:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.909+0000 7f16eaffd700 1 -- 172.21.15.7:0/1217354250 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f16f4009c10 con 0x7f16fc149fe0 2023-03-03T10:36:30.912 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:36:30.912 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.911+0000 7f1700947700 1 -- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16dc0682f0 msgr2=0x7f16dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.911+0000 7f1700947700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16dc0682f0 0x7f16dc06a7a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f16fc1dcca0 tx=0x7f16ec00a000 comp rx=0 tx=0).stop 2023-03-03T10:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.911+0000 7f1700947700 1 -- 172.21.15.7:0/1217354250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16fc149fe0 msgr2=0x7f16fc1e8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.911+0000 7f1700947700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16fc149fe0 0x7f16fc1e8a60 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f16e40047d0 tx=0x7f16f400f490 comp rx=0 tx=0).stop 2023-03-03T10:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.911+0000 7f1700947700 1 -- 172.21.15.7:0/1217354250 shutdown_connections 2023-03-03T10:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.911+0000 7f1700947700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16dc0682f0 0x7f16dc06a7a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.911+0000 7f1700947700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16fc149fe0 0x7f16fc1e8a60 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.911+0000 7f1700947700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16e4003c30 0x7f16fc153ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.911+0000 7f1700947700 1 --2- 172.21.15.7:0/1217354250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16fc14ba50 0x7f16fc149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.911+0000 7f1700947700 1 -- 172.21.15.7:0/1217354250 >> 172.21.15.7:0/1217354250 conn(0x7f16fc06c110 msgr2=0x7f16fc0c25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.912+0000 7f1700947700 1 -- 172.21.15.7:0/1217354250 shutdown_connections 2023-03-03T10:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:36:30.912+0000 7f1700947700 1 -- 172.21.15.7:0/1217354250 wait complete. 2023-03-03T10:36:30.916 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:36:30.926 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T10:36:32.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:36:32.209 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:02.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:02.282 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:12.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:37:13.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.061+0000 7f3fe8718700 1 Processor -- start 2023-03-03T10:37:13.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.062+0000 7f3fe8718700 1 -- start start 2023-03-03T10:37:13.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.062+0000 7f3fe8718700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3fe00caa30 0x7f3fe00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:13.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.062+0000 7f3fe8718700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00c00e0 0x7f3fe0154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:13.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.062+0000 7f3fe8718700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe00cb280 con 0x7f3fe00cae00 2023-03-03T10:37:13.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.062+0000 7f3fe8718700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe00c7310 con 0x7f3fe00caa30 2023-03-03T10:37:13.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.062+0000 7f3fe8718700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00bfd10 con 0x7f3fe00c00e0 2023-03-03T10:37:13.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe64b4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3fe00caa30 0x7f3fe00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:13.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe64b4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3fe00caa30 0x7f3fe00c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60976/0 (socket says 172.21.15.7:60976) 2023-03-03T10:37:13.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe64b4700 1 -- 172.21.15.7:0/2552293607 learned_addr learned my addr 172.21.15.7:0/2552293607 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:37:13.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe5cb3700 1 --2- 172.21.15.7:0/2552293607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00c00e0 0x7f3fe0154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:13.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe64b4700 1 -- 172.21.15.7:0/2552293607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00c00e0 msgr2=0x7f3fe0154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:37:13.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe64b4700 1 --2- 172.21.15.7:0/2552293607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00c00e0 0x7f3fe0154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:13.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe64b4700 1 -- 172.21.15.7:0/2552293607 >> v1:172.21.15.143:6789/0 conn(0x7f3fe00cae00 legacy=0x7f3fe00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:13.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe64b4700 1 -- 172.21.15.7:0/2552293607 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe00bfa30 con 0x7f3fe00caa30 2023-03-03T10:37:13.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe64b4700 1 --2- 172.21.15.7:0/2552293607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00caa30 0x7f3fe00c5c60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40098f0 tx=0x7f3fd4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244de1ebd91f791c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:37:13.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe54b2700 1 -- 172.21.15.7:0/2552293607 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd400b8a0 con 0x7f3fe00caa30 2023-03-03T10:37:13.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe54b2700 1 -- 172.21.15.7:0/2552293607 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd400ba00 con 0x7f3fe00caa30 2023-03-03T10:37:13.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.063+0000 7f3fe54b2700 1 -- 172.21.15.7:0/2552293607 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd400bd90 con 0x7f3fe00caa30 2023-03-03T10:37:13.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.064+0000 7f3fe8718700 1 -- 172.21.15.7:0/2552293607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00caa30 msgr2=0x7f3fe00c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:13.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.064+0000 7f3fe8718700 1 --2- 172.21.15.7:0/2552293607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00caa30 0x7f3fe00c5c60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40098f0 tx=0x7f3fd4009c00 comp rx=0 tx=0).stop 2023-03-03T10:37:13.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.064+0000 7f3fe8718700 1 -- 172.21.15.7:0/2552293607 shutdown_connections 2023-03-03T10:37:13.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.064+0000 7f3fe8718700 1 --2- 172.21.15.7:0/2552293607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00caa30 0x7f3fe00c5c60 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:13.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.064+0000 7f3fe8718700 1 --2- 172.21.15.7:0/2552293607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00c00e0 0x7f3fe0154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:13.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.064+0000 7f3fe8718700 1 -- 172.21.15.7:0/2552293607 >> 172.21.15.7:0/2552293607 conn(0x7f3fe0066e40 msgr2=0x7f3fe0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:37:13.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.064+0000 7f3fe8718700 1 -- 172.21.15.7:0/2552293607 shutdown_connections 2023-03-03T10:37:13.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.064+0000 7f3fe8718700 1 -- 172.21.15.7:0/2552293607 wait complete. 2023-03-03T10:37:13.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.065+0000 7f3fe8718700 1 Processor -- start 2023-03-03T10:37:13.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.065+0000 7f3fe8718700 1 -- start start 2023-03-03T10:37:13.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe8718700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe00c00e0 0x7f3fe00c3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:13.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe8718700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00caa30 0x7f3fe00c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:13.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe8718700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00cae00 0x7f3fe01dea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:13.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe8718700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00c2040 con 0x7f3fe00cae00 2023-03-03T10:37:13.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe8718700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe014dcd0 con 0x7f3fe00c00e0 2023-03-03T10:37:13.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe8718700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe014d540 con 0x7f3fe00caa30 2023-03-03T10:37:13.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe6cb5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00cae00 0x7f3fe01dea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:13.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe6cb5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00cae00 0x7f3fe01dea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60986/0 (socket says 172.21.15.7:60986) 2023-03-03T10:37:13.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe6cb5700 1 -- 172.21.15.7:0/4170198438 learned_addr learned my addr 172.21.15.7:0/4170198438 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:37:13.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe64b4700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe00c00e0 0x7f3fe00c3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:13.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe5cb3700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00caa30 0x7f3fe00c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:13.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe6cb5700 1 -- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00caa30 msgr2=0x7f3fe00c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:13.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe6cb5700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00caa30 0x7f3fe00c4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:13.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe6cb5700 1 -- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe00c00e0 msgr2=0x7f3fe00c3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:13.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe6cb5700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe00c00e0 0x7f3fe00c3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:13.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe6cb5700 1 -- 172.21.15.7:0/4170198438 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd4009550 con 0x7f3fe00cae00 2023-03-03T10:37:13.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe5cb3700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00caa30 0x7f3fe00c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:37:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.066+0000 7f3fe6cb5700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00cae00 0x7f3fe01dea90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800cb50 tx=0x7f3fd800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:37:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.067+0000 7f3fd2ffd700 1 -- 172.21.15.7:0/4170198438 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8003c00 con 0x7f3fe00cae00 2023-03-03T10:37:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.067+0000 7f3fd2ffd700 1 -- 172.21.15.7:0/4170198438 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd8003d60 con 0x7f3fe00cae00 2023-03-03T10:37:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.067+0000 7f3fe8718700 1 -- 172.21.15.7:0/4170198438 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe01547f0 con 0x7f3fe00cae00 2023-03-03T10:37:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.067+0000 7f3fd2ffd700 1 -- 172.21.15.7:0/4170198438 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd800aac0 con 0x7f3fe00cae00 2023-03-03T10:37:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.067+0000 7f3fe8718700 1 -- 172.21.15.7:0/4170198438 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe0154510 con 0x7f3fe00cae00 2023-03-03T10:37:13.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.068+0000 7f3fe8718700 1 -- 172.21.15.7:0/4170198438 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe0154ad0 con 0x7f3fe00cae00 2023-03-03T10:37:13.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.068+0000 7f3fd2ffd700 1 -- 172.21.15.7:0/4170198438 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3fd8010050 con 0x7f3fe00cae00 2023-03-03T10:37:13.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.068+0000 7f3fd2ffd700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 0x7f3fc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:13.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.069+0000 7f3fd2ffd700 1 -- 172.21.15.7:0/4170198438 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3fd8085f90 con 0x7f3fe00cae00 2023-03-03T10:37:13.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.069+0000 7f3fe64b4700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 0x7f3fc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:13.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.069+0000 7f3fe64b4700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 0x7f3fc406a7a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4000c00 tx=0x7f3fd4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:37:13.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.073+0000 7f3fd2ffd700 1 -- 172.21.15.7:0/4170198438 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3fd8052330 con 0x7f3fe00cae00 2023-03-03T10:37:13.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.309+0000 7f3fe8718700 1 -- 172.21.15.7:0/4170198438 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fe00c3150 con 0x7f3fe00cae00 2023-03-03T10:37:13.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.309+0000 7f3fd2ffd700 1 -- 172.21.15.7:0/4170198438 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3fd8055950 con 0x7f3fe00cae00 2023-03-03T10:37:13.313 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:37:13.313 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:37:13.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 -- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 msgr2=0x7f3fc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:13.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 0x7f3fc406a7a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4000c00 tx=0x7f3fd4008030 comp rx=0 tx=0).stop 2023-03-03T10:37:13.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 -- 172.21.15.7:0/4170198438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00cae00 msgr2=0x7f3fe01dea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:13.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00cae00 0x7f3fe01dea90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800cb50 tx=0x7f3fd800ce60 comp rx=0 tx=0).stop 2023-03-03T10:37:13.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 -- 172.21.15.7:0/4170198438 shutdown_connections 2023-03-03T10:37:13.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 0x7f3fc406a7a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:13.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe00cae00 0x7f3fe01dea90 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:13.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe00c00e0 0x7f3fe00c3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:13.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 --2- 172.21.15.7:0/4170198438 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00caa30 0x7f3fe00c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:13.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 -- 172.21.15.7:0/4170198438 >> 172.21.15.7:0/4170198438 conn(0x7f3fe0066e40 msgr2=0x7f3fe01513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:37:13.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 -- 172.21.15.7:0/4170198438 shutdown_connections 2023-03-03T10:37:13.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:13.311+0000 7f3fe8718700 1 -- 172.21.15.7:0/4170198438 wait complete. 2023-03-03T10:37:13.316 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:37:13.326 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T10:37:23.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:37:23.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.465+0000 7f2dbf48b700 1 Processor -- start 2023-03-03T10:37:23.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.465+0000 7f2dbf48b700 1 -- start start 2023-03-03T10:37:23.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbf48b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db80c2930 0x7f2db8157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:23.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbf48b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2db806c480 0x7f2db8157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:23.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbf48b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db806d430 con 0x7f2db8151d40 2023-03-03T10:37:23.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbf48b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db806c2a0 con 0x7f2db806c480 2023-03-03T10:37:23.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbf48b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db806cb00 con 0x7f2db80c2930 2023-03-03T10:37:23.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbda28700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2db806c480 0x7f2db8157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:23.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbd227700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2db8151d40 0x7f2db80736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54980/0 (socket says 172.21.15.7:54980) 2023-03-03T10:37:23.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbd227700 1 -- 172.21.15.7:0/2487594133 learned_addr learned my addr 172.21.15.7:0/2487594133 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:37:23.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbca26700 1 --2- 172.21.15.7:0/2487594133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db80c2930 0x7f2db8157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:23.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbda28700 1 -- 172.21.15.7:0/2487594133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db80c2930 msgr2=0x7f2db8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:37:23.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbda28700 1 --2- 172.21.15.7:0/2487594133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db80c2930 0x7f2db8157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:23.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbda28700 1 -- 172.21.15.7:0/2487594133 >> v1:172.21.15.7:6789/0 conn(0x7f2db8151d40 legacy=0x7f2db80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:23.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbda28700 1 -- 172.21.15.7:0/2487594133 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db806cde0 con 0x7f2db806c480 2023-03-03T10:37:23.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2daffff700 1 -- 172.21.15.7:0/2487594133 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581239062 0 0) 0x7f2db806d430 con 0x7f2db8151d40 2023-03-03T10:37:23.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbca26700 1 --2- 172.21.15.7:0/2487594133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db80c2930 0x7f2db8157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:37:23.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2dbda28700 1 --2- 172.21.15.7:0/2487594133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db806c480 0x7f2db8157fd0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2db0009d90 tx=0x7f2db000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7911d4119eae47a3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:37:23.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2daffff700 1 -- 172.21.15.7:0/2487594133 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db000e070 con 0x7f2db806c480 2023-03-03T10:37:23.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2daffff700 1 -- 172.21.15.7:0/2487594133 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db000bbd0 con 0x7f2db806c480 2023-03-03T10:37:23.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.466+0000 7f2daffff700 1 -- 172.21.15.7:0/2487594133 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db0012430 con 0x7f2db806c480 2023-03-03T10:37:23.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.467+0000 7f2dbf48b700 1 -- 172.21.15.7:0/2487594133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db806c480 msgr2=0x7f2db8157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:23.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.467+0000 7f2dbf48b700 1 --2- 172.21.15.7:0/2487594133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db806c480 0x7f2db8157fd0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f2db0009d90 tx=0x7f2db000b330 comp rx=0 tx=0).stop 2023-03-03T10:37:23.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.467+0000 7f2dbf48b700 1 -- 172.21.15.7:0/2487594133 shutdown_connections 2023-03-03T10:37:23.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.467+0000 7f2dbf48b700 1 --2- 172.21.15.7:0/2487594133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db806c480 0x7f2db8157fd0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:23.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.467+0000 7f2dbf48b700 1 --2- 172.21.15.7:0/2487594133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db80c2930 0x7f2db8157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:23.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.467+0000 7f2dbf48b700 1 -- 172.21.15.7:0/2487594133 >> 172.21.15.7:0/2487594133 conn(0x7f2db80a94d0 msgr2=0x7f2db80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:37:23.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.467+0000 7f2dbf48b700 1 -- 172.21.15.7:0/2487594133 shutdown_connections 2023-03-03T10:37:23.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.467+0000 7f2dbf48b700 1 -- 172.21.15.7:0/2487594133 wait complete. 2023-03-03T10:37:23.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.469+0000 7f2dbf48b700 1 Processor -- start 2023-03-03T10:37:23.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.469+0000 7f2dbf48b700 1 -- start start 2023-03-03T10:37:23.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbf48b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db806c480 0x7f2db81edfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:23.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbf48b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2db80c2930 0x7f2db80c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:23.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbf48b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db8151d40 0x7f2db80be430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:23.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbf48b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db80aa1d0 con 0x7f2db8151d40 2023-03-03T10:37:23.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbf48b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db81ed870 con 0x7f2db80c2930 2023-03-03T10:37:23.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbf48b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db81edb50 con 0x7f2db806c480 2023-03-03T10:37:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbda28700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db8151d40 0x7f2db80be430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbda28700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db8151d40 0x7f2db80be430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32778/0 (socket says 172.21.15.7:32778) 2023-03-03T10:37:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbda28700 1 -- 172.21.15.7:0/46942938 learned_addr learned my addr 172.21.15.7:0/46942938 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:37:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbd227700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db806c480 0x7f2db81edfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:23.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbca26700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2db80c2930 0x7f2db80c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:23.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbda28700 1 -- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db806c480 msgr2=0x7f2db81edfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:23.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbda28700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db806c480 0x7f2db81edfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:23.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbda28700 1 -- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2db80c2930 msgr2=0x7f2db80c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:23.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbda28700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2db80c2930 0x7f2db80c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:23.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbda28700 1 -- 172.21.15.7:0/46942938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db0009a70 con 0x7f2db8151d40 2023-03-03T10:37:23.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbda28700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db8151d40 0x7f2db80be430 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2db000b730 tx=0x7f2db0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:37:23.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dadffb700 1 -- 172.21.15.7:0/46942938 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db00098f0 con 0x7f2db8151d40 2023-03-03T10:37:23.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dadffb700 1 -- 172.21.15.7:0/46942938 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db0020070 con 0x7f2db8151d40 2023-03-03T10:37:23.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dadffb700 1 -- 172.21.15.7:0/46942938 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db00276a0 con 0x7f2db8151d40 2023-03-03T10:37:23.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.470+0000 7f2dbf48b700 1 -- 172.21.15.7:0/46942938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db8073e00 con 0x7f2db8151d40 2023-03-03T10:37:23.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.471+0000 7f2dbf48b700 1 -- 172.21.15.7:0/46942938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db806d0c0 con 0x7f2db8151d40 2023-03-03T10:37:23.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.472+0000 7f2dbf48b700 1 -- 172.21.15.7:0/46942938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db806bea0 con 0x7f2db8151d40 2023-03-03T10:37:23.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.472+0000 7f2dadffb700 1 -- 172.21.15.7:0/46942938 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2db002e070 con 0x7f2db8151d40 2023-03-03T10:37:23.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.472+0000 7f2dadffb700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d9c0682c0 0x7f2d9c06a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:23.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.473+0000 7f2dadffb700 1 -- 172.21.15.7:0/46942938 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2db0022070 con 0x7f2db8151d40 2023-03-03T10:37:23.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.473+0000 7f2dbd227700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d9c0682c0 0x7f2d9c06a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:23.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.473+0000 7f2dbd227700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d9c0682c0 0x7f2d9c06a770 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2da8000f60 tx=0x7f2da8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:37:23.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.477+0000 7f2dadffb700 1 -- 172.21.15.7:0/46942938 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2db0054340 con 0x7f2db8151d40 2023-03-03T10:37:23.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.716+0000 7f2dbf48b700 1 -- 172.21.15.7:0/46942938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2db80710d0 con 0x7f2db8151d40 2023-03-03T10:37:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.717+0000 7f2dadffb700 1 -- 172.21.15.7:0/46942938 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2db0057960 con 0x7f2db8151d40 2023-03-03T10:37:23.720 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:37:23.720 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:37:23.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.718+0000 7f2dbf48b700 1 -- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d9c0682c0 msgr2=0x7f2d9c06a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d9c0682c0 0x7f2d9c06a770 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2da8000f60 tx=0x7f2da8009040 comp rx=0 tx=0).stop 2023-03-03T10:37:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 -- 172.21.15.7:0/46942938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db8151d40 msgr2=0x7f2db80be430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:23.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db8151d40 0x7f2db80be430 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2db000b730 tx=0x7f2db0004ad0 comp rx=0 tx=0).stop 2023-03-03T10:37:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 -- 172.21.15.7:0/46942938 shutdown_connections 2023-03-03T10:37:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d9c0682c0 0x7f2d9c06a770 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:23.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2db8151d40 0x7f2db80be430 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2db806c480 0x7f2db81edfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 --2- 172.21.15.7:0/46942938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2db80c2930 0x7f2db80c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 -- 172.21.15.7:0/46942938 >> 172.21.15.7:0/46942938 conn(0x7f2db80a94d0 msgr2=0x7f2db80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:37:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 -- 172.21.15.7:0/46942938 shutdown_connections 2023-03-03T10:37:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:23.719+0000 7f2dbf48b700 1 -- 172.21.15.7:0/46942938 wait complete. 2023-03-03T10:37:23.725 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:37:23.733 INFO:tasks.mds_thrash.fs.[a]:waiting for 82 secs before thrashing 2023-03-03T10:37:32.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:32.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:37:51.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:37:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.027+0000 7f970dda6700 1 Processor -- start 2023-03-03T10:37:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.027+0000 7f970dda6700 1 -- start start 2023-03-03T10:37:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f970dda6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9708151d40 0x7f97080736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f970dda6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f97080c2930 0x7f9708157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f970dda6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970806d430 con 0x7f9708073c30 2023-03-03T10:37:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f970dda6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970806c2a0 con 0x7f97080c2930 2023-03-03T10:37:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f970dda6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97081586e0 con 0x7f9708151d40 2023-03-03T10:37:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f9707fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9708073c30 0x7f9708157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55002/0 (socket says 172.21.15.7:55002) 2023-03-03T10:37:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f9707fff700 1 -- 172.21.15.7:0/394508097 learned_addr learned my addr 172.21.15.7:0/394508097 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:37:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185341921 0 0) 0x7f970806d430 con 0x7f9708073c30 2023-03-03T10:37:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f40035b0 con 0x7f9708073c30 2023-03-03T10:37:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f9706ffd700 1 --2- 172.21.15.7:0/394508097 >> v2:172.21.15.143:3300/0 conn(0x7f97080c2930 0x7f9708157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97077fe700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9708151d40 0x7f97080736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126086631 0 0) 0x7f96f40035b0 con 0x7f9708073c30 2023-03-03T10:37:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9708151d40 msgr2=0x7f97080736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:37:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9708151d40 0x7f97080736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 >> v2:172.21.15.143:3300/0 conn(0x7f97080c2930 msgr2=0x7f9708157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:37:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 --2- 172.21.15.7:0/394508097 >> v2:172.21.15.143:3300/0 conn(0x7f97080c2930 0x7f9708157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970806cb00 con 0x7f9708073c30 2023-03-03T10:37:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f97000019f0 con 0x7f9708073c30 2023-03-03T10:37:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 >> v1:172.21.15.7:6789/0 conn(0x7f9708073c30 legacy=0x7f9708157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9708151d40 0x7f96f4003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.028+0000 7f97067fc700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 0x7f96f4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f97067fc700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96f4004d90 0x7f96f4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970806c2a0 con 0x7f96f4003ff0 2023-03-03T10:37:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97081586e0 con 0x7f96f4004d90 2023-03-03T10:37:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f400bb70 con 0x7f9708151d40 2023-03-03T10:37:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f9706ffd700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 0x7f96f4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f97077fe700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9708151d40 0x7f96f4003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f9707fff700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96f4004d90 0x7f96f4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f9706ffd700 1 -- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9708151d40 msgr2=0x7f96f4003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:37:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f9706ffd700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9708151d40 0x7f96f4003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f9706ffd700 1 -- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96f4004d90 msgr2=0x7f96f4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:37:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f9706ffd700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96f4004d90 0x7f96f4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:52.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f9706ffd700 1 -- 172.21.15.7:0/394508097 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f40035b0 con 0x7f96f4003ff0 2023-03-03T10:37:52.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f9706ffd700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 0x7f96f4004850 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009b30 tx=0x7f96f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a03a425422f0a9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:37:52.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f800e6c0 con 0x7f96f4003ff0 2023-03-03T10:37:52.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f800e820 con 0x7f96f4003ff0 2023-03-03T10:37:52.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.029+0000 7f97067fc700 1 -- 172.21.15.7:0/394508097 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f800eb90 con 0x7f96f4003ff0 2023-03-03T10:37:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.030+0000 7f970dda6700 1 -- 172.21.15.7:0/394508097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 msgr2=0x7f96f4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.030+0000 7f970dda6700 1 --2- 172.21.15.7:0/394508097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 0x7f96f4004850 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009b30 tx=0x7f96f4005880 comp rx=0 tx=0).stop 2023-03-03T10:37:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.030+0000 7f9707fff700 1 -- 172.21.15.7:0/394508097 reap_dead start 2023-03-03T10:37:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.030+0000 7f970dda6700 1 -- 172.21.15.7:0/394508097 shutdown_connections 2023-03-03T10:37:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.030+0000 7f970dda6700 1 -- 172.21.15.7:0/394508097 >> 172.21.15.7:0/394508097 conn(0x7f97080a94d0 msgr2=0x7f97080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:37:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.030+0000 7f970dda6700 1 -- 172.21.15.7:0/394508097 shutdown_connections 2023-03-03T10:37:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.030+0000 7f970dda6700 1 -- 172.21.15.7:0/394508097 wait complete. 2023-03-03T10:37:52.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.031+0000 7f970dda6700 1 Processor -- start 2023-03-03T10:37:52.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.031+0000 7f970dda6700 1 -- start start 2023-03-03T10:37:52.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f970dda6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 0x7f9708153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:52.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f970dda6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f970814e660 0x7f9708154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:52.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f970dda6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080c1140 0x7f97080aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:52.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f970dda6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97081548d0 con 0x7f96f4003ff0 2023-03-03T10:37:52.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f970dda6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97081542c0 con 0x7f970814e660 2023-03-03T10:37:52.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f970dda6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97081552a0 con 0x7f97080c1140 2023-03-03T10:37:52.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f97077fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 0x7f9708153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f97077fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 0x7f9708153a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32800/0 (socket says 172.21.15.7:32800) 2023-03-03T10:37:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f97077fe700 1 -- 172.21.15.7:0/4266253511 learned_addr learned my addr 172.21.15.7:0/4266253511 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:37:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f9706ffd700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f970814e660 0x7f9708154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f9707fff700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080c1140 0x7f97080aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f97077fe700 1 -- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080c1140 msgr2=0x7f97080aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f97077fe700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080c1140 0x7f97080aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f97077fe700 1 -- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f970814e660 msgr2=0x7f9708154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f97077fe700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f970814e660 0x7f9708154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.032+0000 7f97077fe700 1 -- 172.21.15.7:0/4266253511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970806d0c0 con 0x7f96f4003ff0 2023-03-03T10:37:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.033+0000 7f97077fe700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 0x7f9708153a20 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f96f0011660 tx=0x7f96f0011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:37:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.033+0000 7f96fffff700 1 -- 172.21.15.7:0/4266253511 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f0015930 con 0x7f96f4003ff0 2023-03-03T10:37:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.033+0000 7f96fffff700 1 -- 172.21.15.7:0/4266253511 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f0015a90 con 0x7f96f4003ff0 2023-03-03T10:37:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.033+0000 7f970dda6700 1 -- 172.21.15.7:0/4266253511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970806cde0 con 0x7f96f4003ff0 2023-03-03T10:37:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.033+0000 7f96fffff700 1 -- 172.21.15.7:0/4266253511 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f0015e00 con 0x7f96f4003ff0 2023-03-03T10:37:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.033+0000 7f970dda6700 1 -- 172.21.15.7:0/4266253511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970806bea0 con 0x7f96f4003ff0 2023-03-03T10:37:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.034+0000 7f970dda6700 1 -- 172.21.15.7:0/4266253511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9708074bb0 con 0x7f96f4003ff0 2023-03-03T10:37:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.034+0000 7f96fffff700 1 -- 172.21.15.7:0/4266253511 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f96f001b030 con 0x7f96f4003ff0 2023-03-03T10:37:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.035+0000 7f96fffff700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96e80682f0 0x7f96e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:37:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.035+0000 7f96fffff700 1 -- 172.21.15.7:0/4266253511 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f96f00862b0 con 0x7f96f4003ff0 2023-03-03T10:37:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.035+0000 7f9706ffd700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96e80682f0 0x7f96e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:37:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.035+0000 7f9706ffd700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96e80682f0 0x7f96e806a7a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f96f80090b0 tx=0x7f96f8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:37:52.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.039+0000 7f96fffff700 1 -- 172.21.15.7:0/4266253511 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f96f0052650 con 0x7f96f4003ff0 2023-03-03T10:37:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.277+0000 7f970dda6700 1 -- 172.21.15.7:0/4266253511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97080a9fb0 con 0x7f96f4003ff0 2023-03-03T10:37:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.278+0000 7f96fffff700 1 -- 172.21.15.7:0/4266253511 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f96f0055c70 con 0x7f96f4003ff0 2023-03-03T10:37:52.280 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:37:52.281 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:37:52.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 -- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96e80682f0 msgr2=0x7f96e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:52.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96e80682f0 0x7f96e806a7a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f96f80090b0 tx=0x7f96f8008030 comp rx=0 tx=0).stop 2023-03-03T10:37:52.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 -- 172.21.15.7:0/4266253511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 msgr2=0x7f9708153a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:37:52.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 0x7f9708153a20 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f96f0011660 tx=0x7f96f0011970 comp rx=0 tx=0).stop 2023-03-03T10:37:52.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 -- 172.21.15.7:0/4266253511 shutdown_connections 2023-03-03T10:37:52.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97080c1140 0x7f97080aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:52.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96e80682f0 0x7f96e806a7a0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:52.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96f4003ff0 0x7f9708153a20 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:52.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 --2- 172.21.15.7:0/4266253511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f970814e660 0x7f9708154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:37:52.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 -- 172.21.15.7:0/4266253511 >> 172.21.15.7:0/4266253511 conn(0x7f97080a94d0 msgr2=0x7f97080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:37:52.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 -- 172.21.15.7:0/4266253511 shutdown_connections 2023-03-03T10:37:52.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:37:52.280+0000 7f970dda6700 1 -- 172.21.15.7:0/4266253511 wait complete. 2023-03-03T10:37:52.284 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:37:52.294 INFO:tasks.mds_thrash.fs.[b]:waiting for 91 secs before thrashing 2023-03-03T10:38:02.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:02.416 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:32.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:32.502 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:38:45.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:38:45.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.873+0000 7f8cd712a700 1 Processor -- start 2023-03-03T10:38:45.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.873+0000 7f8cd712a700 1 -- start start 2023-03-03T10:38:45.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.873+0000 7f8cd712a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8cd0151b20 0x7f8cd0071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.873+0000 7f8cd712a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8cd0158710 0x7f8cd015ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.873+0000 7f8cd712a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd0151ef0 con 0x7f8cd0154f10 2023-03-03T10:38:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8cd712a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd0157b20 con 0x7f8cd0151b20 2023-03-03T10:38:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8cd712a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd0155410 con 0x7f8cd0158710 2023-03-03T10:38:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8cd4ec6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8cd0151b20 0x7f8cd0071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:45.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8ccffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8cd0154f10 0x7f8cd0158000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55036/0 (socket says 172.21.15.7:55036) 2023-03-03T10:38:45.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8ccffff700 1 -- 172.21.15.7:0/3782498300 learned_addr learned my addr 172.21.15.7:0/3782498300 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:38:45.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8cd56c7700 1 --2- 172.21.15.7:0/3782498300 >> v2:172.21.15.143:3300/0 conn(0x7f8cd0158710 0x7f8cd015ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:45.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8cd4ec6700 1 -- 172.21.15.7:0/3782498300 >> v2:172.21.15.143:3300/0 conn(0x7f8cd0158710 msgr2=0x7f8cd015ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:38:45.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8cd4ec6700 1 --2- 172.21.15.7:0/3782498300 >> v2:172.21.15.143:3300/0 conn(0x7f8cd0158710 0x7f8cd015ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:45.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8cd4ec6700 1 -- 172.21.15.7:0/3782498300 >> v1:172.21.15.7:6789/0 conn(0x7f8cd0154f10 legacy=0x7f8cd0158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:45.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8ccf7fe700 1 -- 172.21.15.7:0/3782498300 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886286752 0 0) 0x7f8cd0151ef0 con 0x7f8cd0154f10 2023-03-03T10:38:45.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8cd4ec6700 1 -- 172.21.15.7:0/3782498300 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd006f640 con 0x7f8cd0151b20 2023-03-03T10:38:45.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8cd4ec6700 1 --2- 172.21.15.7:0/3782498300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 0x7f8cd0071820 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0072800 tx=0x7f8cc4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f6bb53873043f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:38:45.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8ccf7fe700 1 -- 172.21.15.7:0/3782498300 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4011830 con 0x7f8cd0151b20 2023-03-03T10:38:45.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8ccf7fe700 1 -- 172.21.15.7:0/3782498300 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc4011990 con 0x7f8cd0151b20 2023-03-03T10:38:45.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.874+0000 7f8ccf7fe700 1 -- 172.21.15.7:0/3782498300 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4011d20 con 0x7f8cd0151b20 2023-03-03T10:38:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.875+0000 7f8cd712a700 1 -- 172.21.15.7:0/3782498300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 msgr2=0x7f8cd0071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:45.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.875+0000 7f8cd712a700 1 --2- 172.21.15.7:0/3782498300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 0x7f8cd0071820 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0072800 tx=0x7f8cc4009b20 comp rx=0 tx=0).stop 2023-03-03T10:38:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.875+0000 7f8cd712a700 1 -- 172.21.15.7:0/3782498300 shutdown_connections 2023-03-03T10:38:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.875+0000 7f8cd712a700 1 --2- 172.21.15.7:0/3782498300 >> v2:172.21.15.143:3300/0 conn(0x7f8cd0158710 0x7f8cd015ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.875+0000 7f8cd712a700 1 --2- 172.21.15.7:0/3782498300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 0x7f8cd0071820 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.875+0000 7f8cd712a700 1 -- 172.21.15.7:0/3782498300 >> 172.21.15.7:0/3782498300 conn(0x7f8cd00c8900 msgr2=0x7f8cd00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:38:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.875+0000 7f8cd712a700 1 -- 172.21.15.7:0/3782498300 shutdown_connections 2023-03-03T10:38:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.875+0000 7f8cd712a700 1 -- 172.21.15.7:0/3782498300 wait complete. 2023-03-03T10:38:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.876+0000 7f8cd712a700 1 Processor -- start 2023-03-03T10:38:45.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.876+0000 7f8cd712a700 1 -- start start 2023-03-03T10:38:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd712a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 0x7f8cd00c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd712a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cd0154f10 0x7f8cd0155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd712a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cd0158710 0x7f8cd013fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd712a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01e9dc0 con 0x7f8cd0151b20 2023-03-03T10:38:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd712a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01ea600 con 0x7f8cd0158710 2023-03-03T10:38:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd712a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0157780 con 0x7f8cd0154f10 2023-03-03T10:38:45.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd4ec6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 0x7f8cd00c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:45.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd4ec6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 0x7f8cd00c7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32830/0 (socket says 172.21.15.7:32830) 2023-03-03T10:38:45.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd4ec6700 1 -- 172.21.15.7:0/1254088140 learned_addr learned my addr 172.21.15.7:0/1254088140 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:38:45.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8ccffff700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cd0154f10 0x7f8cd0155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:45.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd56c7700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cd0158710 0x7f8cd013fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:45.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd4ec6700 1 -- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cd0154f10 msgr2=0x7f8cd0155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:45.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd4ec6700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cd0154f10 0x7f8cd0155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:45.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd4ec6700 1 -- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cd0158710 msgr2=0x7f8cd013fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:45.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd4ec6700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cd0158710 0x7f8cd013fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.877+0000 7f8cd4ec6700 1 -- 172.21.15.7:0/1254088140 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40094a0 con 0x7f8cd0151b20 2023-03-03T10:38:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.878+0000 7f8cd4ec6700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 0x7f8cd00c7700 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4014040 tx=0x7f8cc4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:38:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.878+0000 7f8ccd7fa700 1 -- 172.21.15.7:0/1254088140 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc40038a0 con 0x7f8cd0151b20 2023-03-03T10:38:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.878+0000 7f8ccd7fa700 1 -- 172.21.15.7:0/1254088140 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc401e070 con 0x7f8cd0151b20 2023-03-03T10:38:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.878+0000 7f8cd712a700 1 -- 172.21.15.7:0/1254088140 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd006fc00 con 0x7f8cd0151b20 2023-03-03T10:38:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.878+0000 7f8ccd7fa700 1 -- 172.21.15.7:0/1254088140 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4003c60 con 0x7f8cd0151b20 2023-03-03T10:38:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.878+0000 7f8cd712a700 1 -- 172.21.15.7:0/1254088140 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd006f920 con 0x7f8cd0151b20 2023-03-03T10:38:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.879+0000 7f8cd712a700 1 -- 172.21.15.7:0/1254088140 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd00aae90 con 0x7f8cd0151b20 2023-03-03T10:38:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.879+0000 7f8ccd7fa700 1 -- 172.21.15.7:0/1254088140 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8cc400f050 con 0x7f8cd0151b20 2023-03-03T10:38:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.880+0000 7f8ccd7fa700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cb40682f0 0x7f8cb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:38:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.880+0000 7f8ccd7fa700 1 -- 172.21.15.7:0/1254088140 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8cc4087280 con 0x7f8cd0151b20 2023-03-03T10:38:45.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.880+0000 7f8ccffff700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cb40682f0 0x7f8cb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:38:45.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.880+0000 7f8ccffff700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cb40682f0 0x7f8cb406a7a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8cd01474d0 tx=0x7f8cbc007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:38:45.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:45.884+0000 7f8ccd7fa700 1 -- 172.21.15.7:0/1254088140 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8cc4053620 con 0x7f8cd0151b20 2023-03-03T10:38:46.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.124+0000 7f8cd712a700 1 -- 172.21.15.7:0/1254088140 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cd0074bb0 con 0x7f8cd0151b20 2023-03-03T10:38:46.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.124+0000 7f8ccd7fa700 1 -- 172.21.15.7:0/1254088140 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8cc4020090 con 0x7f8cd0151b20 2023-03-03T10:38:46.127 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:38:46.127 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:38:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.126+0000 7f8cd712a700 1 -- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cb40682f0 msgr2=0x7f8cb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.126+0000 7f8cd712a700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cb40682f0 0x7f8cb406a7a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8cd01474d0 tx=0x7f8cbc007be0 comp rx=0 tx=0).stop 2023-03-03T10:38:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.126+0000 7f8cd712a700 1 -- 172.21.15.7:0/1254088140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 msgr2=0x7f8cd00c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:38:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.126+0000 7f8cd712a700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 0x7f8cd00c7700 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8cc4014040 tx=0x7f8cc4004750 comp rx=0 tx=0).stop 2023-03-03T10:38:46.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.127+0000 7f8cd712a700 1 -- 172.21.15.7:0/1254088140 shutdown_connections 2023-03-03T10:38:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.127+0000 7f8cd712a700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cd0158710 0x7f8cd013fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.127+0000 7f8cd712a700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cb40682f0 0x7f8cb406a7a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.127+0000 7f8cd712a700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cd0151b20 0x7f8cd00c7700 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.127+0000 7f8cd712a700 1 --2- 172.21.15.7:0/1254088140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cd0154f10 0x7f8cd0155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:38:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.127+0000 7f8cd712a700 1 -- 172.21.15.7:0/1254088140 >> 172.21.15.7:0/1254088140 conn(0x7f8cd00c8900 msgr2=0x7f8cd014a010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:38:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.127+0000 7f8cd712a700 1 -- 172.21.15.7:0/1254088140 shutdown_connections 2023-03-03T10:38:46.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:38:46.127+0000 7f8cd712a700 1 -- 172.21.15.7:0/1254088140 wait complete. 2023-03-03T10:38:46.131 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:38:46.141 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T10:39:02.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:02.574 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:23.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:39:23.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.426+0000 7ff49c19a700 1 Processor -- start 2023-03-03T10:39:23.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.426+0000 7ff49c19a700 1 -- start start 2023-03-03T10:39:23.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.427+0000 7ff49c19a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff49414f9d0 0x7ff494156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:23.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.427+0000 7ff49c19a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff49414fda0 0x7ff494071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:23.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.427+0000 7ff49c19a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4940abe20 0x7ff4941529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:23.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.427+0000 7ff49c19a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4940c2f30 con 0x7ff49414f9d0 2023-03-03T10:39:23.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.427+0000 7ff49c19a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4940c31d0 con 0x7ff49414fda0 2023-03-03T10:39:23.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.427+0000 7ff49c19a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4941524a0 con 0x7ff4940abe20 2023-03-03T10:39:23.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.427+0000 7ff499f36700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff49414f9d0 0x7ff494156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:23.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.427+0000 7ff499f36700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff49414f9d0 0x7ff494156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32868/0 (socket says 172.21.15.7:32868) 2023-03-03T10:39:23.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.427+0000 7ff499f36700 1 -- 172.21.15.7:0/370869041 learned_addr learned my addr 172.21.15.7:0/370869041 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:39:23.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff49a737700 1 --2- 172.21.15.7:0/370869041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4940abe20 0x7ff4941529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff499735700 1 --2- 172.21.15.7:0/370869041 >> v2:172.21.15.143:3300/0 conn(0x7ff49414fda0 0x7ff494071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff499f36700 1 -- 172.21.15.7:0/370869041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4940abe20 msgr2=0x7ff4941529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:39:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff499f36700 1 --2- 172.21.15.7:0/370869041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4940abe20 0x7ff4941529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff499f36700 1 -- 172.21.15.7:0/370869041 >> v2:172.21.15.143:3300/0 conn(0x7ff49414fda0 msgr2=0x7ff494071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:39:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff499f36700 1 --2- 172.21.15.7:0/370869041 >> v2:172.21.15.143:3300/0 conn(0x7ff49414fda0 0x7ff494071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff499f36700 1 -- 172.21.15.7:0/370869041 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4940c34b0 con 0x7ff49414f9d0 2023-03-03T10:39:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff49a737700 1 --2- 172.21.15.7:0/370869041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4940abe20 0x7ff4941529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:39:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff499735700 1 --2- 172.21.15.7:0/370869041 >> v2:172.21.15.143:3300/0 conn(0x7ff49414fda0 0x7ff494071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:39:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff499f36700 1 --2- 172.21.15.7:0/370869041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414f9d0 0x7ff494156430 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ff4940709b0 tx=0x7ff484009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b775f71e6637c5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:39:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff498f34700 1 -- 172.21.15.7:0/370869041 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48400b870 con 0x7ff49414f9d0 2023-03-03T10:39:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff498f34700 1 -- 172.21.15.7:0/370869041 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48400b9d0 con 0x7ff49414f9d0 2023-03-03T10:39:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff498f34700 1 -- 172.21.15.7:0/370869041 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48400bd60 con 0x7ff49414f9d0 2023-03-03T10:39:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff49c19a700 1 -- 172.21.15.7:0/370869041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414f9d0 msgr2=0x7ff494156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.428+0000 7ff49c19a700 1 --2- 172.21.15.7:0/370869041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414f9d0 0x7ff494156430 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ff4940709b0 tx=0x7ff484009bd0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.429+0000 7ff49c19a700 1 -- 172.21.15.7:0/370869041 shutdown_connections 2023-03-03T10:39:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.429+0000 7ff49c19a700 1 --2- 172.21.15.7:0/370869041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4940abe20 0x7ff4941529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.429+0000 7ff49c19a700 1 --2- 172.21.15.7:0/370869041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414f9d0 0x7ff494156430 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.429+0000 7ff49c19a700 1 --2- 172.21.15.7:0/370869041 >> v2:172.21.15.143:3300/0 conn(0x7ff49414fda0 0x7ff494071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.429+0000 7ff49c19a700 1 -- 172.21.15.7:0/370869041 >> 172.21.15.7:0/370869041 conn(0x7ff4940c8900 msgr2=0x7ff4940c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:39:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.429+0000 7ff49c19a700 1 -- 172.21.15.7:0/370869041 shutdown_connections 2023-03-03T10:39:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.429+0000 7ff49c19a700 1 -- 172.21.15.7:0/370869041 wait complete. 2023-03-03T10:39:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.430+0000 7ff49c19a700 1 Processor -- start 2023-03-03T10:39:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.430+0000 7ff49c19a700 1 -- start start 2023-03-03T10:39:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49c19a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4940abe20 0x7ff4941557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49c19a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff49414f9d0 0x7ff49406bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49c19a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414fda0 0x7ff4940c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49c19a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49414cc30 con 0x7ff49414fda0 2023-03-03T10:39:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49c19a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4940c28e0 con 0x7ff4940abe20 2023-03-03T10:39:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49c19a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49414c670 con 0x7ff49414f9d0 2023-03-03T10:39:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49a737700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414fda0 0x7ff4940c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49a737700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414fda0 0x7ff4940c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32876/0 (socket says 172.21.15.7:32876) 2023-03-03T10:39:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49a737700 1 -- 172.21.15.7:0/3566787309 learned_addr learned my addr 172.21.15.7:0/3566787309 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:39:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff499735700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff49414f9d0 0x7ff49406bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff499f36700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4940abe20 0x7ff4941557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49a737700 1 -- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff49414f9d0 msgr2=0x7ff49406bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49a737700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff49414f9d0 0x7ff49406bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49a737700 1 -- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4940abe20 msgr2=0x7ff4941557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49a737700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4940abe20 0x7ff4941557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49a737700 1 -- 172.21.15.7:0/3566787309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484009550 con 0x7ff49414fda0 2023-03-03T10:39:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff499735700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff49414f9d0 0x7ff49406bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:39:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49a737700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414fda0 0x7ff4940c1a50 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00cd80 tx=0x7ff48c00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:39:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff48a7fc700 1 -- 172.21.15.7:0/3566787309 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48c00f480 con 0x7ff49414fda0 2023-03-03T10:39:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff48a7fc700 1 -- 172.21.15.7:0/3566787309 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48c00f600 con 0x7ff49414fda0 2023-03-03T10:39:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49c19a700 1 -- 172.21.15.7:0/3566787309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4940ac350 con 0x7ff49414fda0 2023-03-03T10:39:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff48a7fc700 1 -- 172.21.15.7:0/3566787309 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48c00f9b0 con 0x7ff49414fda0 2023-03-03T10:39:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.431+0000 7ff49c19a700 1 -- 172.21.15.7:0/3566787309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4940ab630 con 0x7ff49414fda0 2023-03-03T10:39:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.432+0000 7ff49c19a700 1 -- 172.21.15.7:0/3566787309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4940c3e10 con 0x7ff49414fda0 2023-03-03T10:39:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.433+0000 7ff48a7fc700 1 -- 172.21.15.7:0/3566787309 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff48c009030 con 0x7ff49414fda0 2023-03-03T10:39:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.433+0000 7ff48a7fc700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4780682f0 0x7ff47806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.433+0000 7ff48a7fc700 1 -- 172.21.15.7:0/3566787309 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff48c086c20 con 0x7ff49414fda0 2023-03-03T10:39:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.433+0000 7ff499f36700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4780682f0 0x7ff47806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.434+0000 7ff499f36700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4780682f0 0x7ff47806a7a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff484009130 tx=0x7ff484008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:39:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.438+0000 7ff48a7fc700 1 -- 172.21.15.7:0/3566787309 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff48c052e70 con 0x7ff49414fda0 2023-03-03T10:39:23.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.676+0000 7ff49c19a700 1 -- 172.21.15.7:0/3566787309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff49414c4d0 con 0x7ff49414fda0 2023-03-03T10:39:23.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.677+0000 7ff48a7fc700 1 -- 172.21.15.7:0/3566787309 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff49414c4d0 con 0x7ff49414fda0 2023-03-03T10:39:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:39:23.681 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:39:23.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 -- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4780682f0 msgr2=0x7ff47806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:23.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4780682f0 0x7ff47806a7a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff484009130 tx=0x7ff484008030 comp rx=0 tx=0).stop 2023-03-03T10:39:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 -- 172.21.15.7:0/3566787309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414fda0 msgr2=0x7ff4940c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414fda0 0x7ff4940c1a50 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff48c00cd80 tx=0x7ff48c00a560 comp rx=0 tx=0).stop 2023-03-03T10:39:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 -- 172.21.15.7:0/3566787309 shutdown_connections 2023-03-03T10:39:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4780682f0 0x7ff47806a7a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff49414fda0 0x7ff4940c1a50 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4940abe20 0x7ff4941557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 --2- 172.21.15.7:0/3566787309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff49414f9d0 0x7ff49406bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 -- 172.21.15.7:0/3566787309 >> 172.21.15.7:0/3566787309 conn(0x7ff4940c8900 msgr2=0x7ff4940c65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:39:23.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 -- 172.21.15.7:0/3566787309 shutdown_connections 2023-03-03T10:39:23.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:23.679+0000 7ff49c19a700 1 -- 172.21.15.7:0/3566787309 wait complete. 2023-03-03T10:39:23.684 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:39:23.694 INFO:tasks.mds_thrash.fs.[b]:waiting for 24 secs before thrashing 2023-03-03T10:39:28.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:39:28.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.279+0000 7f2c53384700 1 Processor -- start 2023-03-03T10:39:28.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.279+0000 7f2c53384700 1 -- start start 2023-03-03T10:39:28.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.280+0000 7f2c53384700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 0x7f2c4c156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:28.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.280+0000 7f2c53384700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2c4c0740f0 0x7f2c4c15cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:28.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.280+0000 7f2c53384700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c4c155840 con 0x7f2c4c14fb00 2023-03-03T10:39:28.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.280+0000 7f2c53384700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4c072940 con 0x7f2c4c0740f0 2023-03-03T10:39:28.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.280+0000 7f2c53384700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c1567d0 con 0x7f2c4c0c1da0 2023-03-03T10:39:28.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.280+0000 7f2c51120700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2c4c14fb00 0x7f2c4c0c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55092/0 (socket says 172.21.15.7:55092) 2023-03-03T10:39:28.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.280+0000 7f2c51120700 1 -- 172.21.15.7:0/3465919759 learned_addr learned my addr 172.21.15.7:0/3465919759 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:39:28.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c51921700 1 --2- 172.21.15.7:0/3465919759 >> v2:172.21.15.143:3300/0 conn(0x7f2c4c0740f0 0x7f2c4c15cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:28.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c5091f700 1 --2- 172.21.15.7:0/3465919759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 0x7f2c4c156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:28.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c3ffff700 1 -- 172.21.15.7:0/3465919759 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572020082 0 0) 0x7f2c4c155840 con 0x7f2c4c14fb00 2023-03-03T10:39:28.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c3ffff700 1 -- 172.21.15.7:0/3465919759 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c380035b0 con 0x7f2c4c14fb00 2023-03-03T10:39:28.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c3ffff700 1 -- 172.21.15.7:0/3465919759 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 209384120 0 0) 0x7f2c380035b0 con 0x7f2c4c14fb00 2023-03-03T10:39:28.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c51921700 1 -- 172.21.15.7:0/3465919759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 msgr2=0x7f2c4c156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:39:28.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c51921700 1 --2- 172.21.15.7:0/3465919759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 0x7f2c4c156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:28.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c51921700 1 -- 172.21.15.7:0/3465919759 >> v1:172.21.15.7:6789/0 conn(0x7f2c4c14fb00 legacy=0x7f2c4c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:28.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c51921700 1 -- 172.21.15.7:0/3465919759 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c0c2330 con 0x7f2c4c0740f0 2023-03-03T10:39:28.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c5091f700 1 --2- 172.21.15.7:0/3465919759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 0x7f2c4c156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:39:28.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c51921700 1 --2- 172.21.15.7:0/3465919759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c0740f0 0x7f2c4c15cbd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009d90 tx=0x7f2c4400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e803ef1c948273dd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:39:28.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c3ffff700 1 -- 172.21.15.7:0/3465919759 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4400e070 con 0x7f2c4c0740f0 2023-03-03T10:39:28.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c3ffff700 1 -- 172.21.15.7:0/3465919759 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c44003a20 con 0x7f2c4c0740f0 2023-03-03T10:39:28.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.281+0000 7f2c3ffff700 1 -- 172.21.15.7:0/3465919759 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48001670 con 0x7f2c4c0740f0 2023-03-03T10:39:28.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.282+0000 7f2c53384700 1 -- 172.21.15.7:0/3465919759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c0740f0 msgr2=0x7f2c4c15cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:28.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.282+0000 7f2c53384700 1 --2- 172.21.15.7:0/3465919759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c0740f0 0x7f2c4c15cbd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009d90 tx=0x7f2c4400b330 comp rx=0 tx=0).stop 2023-03-03T10:39:28.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.282+0000 7f2c53384700 1 -- 172.21.15.7:0/3465919759 shutdown_connections 2023-03-03T10:39:28.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.282+0000 7f2c53384700 1 --2- 172.21.15.7:0/3465919759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c0740f0 0x7f2c4c15cbd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:28.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.282+0000 7f2c53384700 1 --2- 172.21.15.7:0/3465919759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 0x7f2c4c156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:28.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.282+0000 7f2c53384700 1 -- 172.21.15.7:0/3465919759 >> 172.21.15.7:0/3465919759 conn(0x7f2c4c06c110 msgr2=0x7f2c4c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:39:28.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.282+0000 7f2c53384700 1 -- 172.21.15.7:0/3465919759 shutdown_connections 2023-03-03T10:39:28.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.282+0000 7f2c53384700 1 -- 172.21.15.7:0/3465919759 wait complete. 2023-03-03T10:39:28.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.283+0000 7f2c53384700 1 Processor -- start 2023-03-03T10:39:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.283+0000 7f2c53384700 1 -- start start 2023-03-03T10:39:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c53384700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c4c0740f0 0x7f2c4c1dba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c53384700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 0x7f2c4c1e8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c53384700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c14fb00 0x7f2c4c1e91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c53384700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c14af20 con 0x7f2c4c0740f0 2023-03-03T10:39:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c53384700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c1eb5f0 con 0x7f2c4c14fb00 2023-03-03T10:39:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c53384700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c0ab000 con 0x7f2c4c0c1da0 2023-03-03T10:39:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c51120700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c4c0740f0 0x7f2c4c1dba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c51120700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c4c0740f0 0x7f2c4c1dba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32886/0 (socket says 172.21.15.7:32886) 2023-03-03T10:39:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c51120700 1 -- 172.21.15.7:0/287898204 learned_addr learned my addr 172.21.15.7:0/287898204 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:39:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c5091f700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 0x7f2c4c1e8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c51921700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c14fb00 0x7f2c4c1e91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c51120700 1 -- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 msgr2=0x7f2c4c1e8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c51120700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 0x7f2c4c1e8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c51120700 1 -- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c14fb00 msgr2=0x7f2c4c1e91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c51120700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c14fb00 0x7f2c4c1e91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.284+0000 7f2c51120700 1 -- 172.21.15.7:0/287898204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c400094a0 con 0x7f2c4c0740f0 2023-03-03T10:39:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.285+0000 7f2c51120700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c4c0740f0 0x7f2c4c1dba90 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800c750 tx=0x7f2c4800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:39:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.285+0000 7f2c3dffb700 1 -- 172.21.15.7:0/287898204 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48004c80 con 0x7f2c4c0740f0 2023-03-03T10:39:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.285+0000 7f2c3dffb700 1 -- 172.21.15.7:0/287898204 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c48004de0 con 0x7f2c4c0740f0 2023-03-03T10:39:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.285+0000 7f2c53384700 1 -- 172.21.15.7:0/287898204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c44009a70 con 0x7f2c4c0740f0 2023-03-03T10:39:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.285+0000 7f2c3dffb700 1 -- 172.21.15.7:0/287898204 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c480072f0 con 0x7f2c4c0740f0 2023-03-03T10:39:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.285+0000 7f2c53384700 1 -- 172.21.15.7:0/287898204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4c153090 con 0x7f2c4c0740f0 2023-03-03T10:39:28.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.286+0000 7f2c53384700 1 -- 172.21.15.7:0/287898204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4c152db0 con 0x7f2c4c0740f0 2023-03-03T10:39:28.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.286+0000 7f2c3dffb700 1 -- 172.21.15.7:0/287898204 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2c4801a020 con 0x7f2c4c0740f0 2023-03-03T10:39:28.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.287+0000 7f2c3dffb700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c300682f0 0x7f2c3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:28.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.287+0000 7f2c3dffb700 1 -- 172.21.15.7:0/287898204 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2c48085e30 con 0x7f2c4c0740f0 2023-03-03T10:39:28.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.287+0000 7f2c5091f700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c300682f0 0x7f2c3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:28.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.288+0000 7f2c5091f700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c300682f0 0x7f2c3006a7a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0c3300 tx=0x7f2c40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:39:28.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.291+0000 7f2c3dffb700 1 -- 172.21.15.7:0/287898204 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2c48052170 con 0x7f2c4c0740f0 2023-03-03T10:39:28.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.529+0000 7f2c53384700 1 -- 172.21.15.7:0/287898204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c4c06e410 con 0x7f2c4c0740f0 2023-03-03T10:39:28.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.530+0000 7f2c3dffb700 1 -- 172.21.15.7:0/287898204 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2c48055790 con 0x7f2c4c0740f0 2023-03-03T10:39:28.532 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:39:28.533 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:39:28.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 -- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c300682f0 msgr2=0x7f2c3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:28.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c300682f0 0x7f2c3006a7a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c0c3300 tx=0x7f2c40008040 comp rx=0 tx=0).stop 2023-03-03T10:39:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 -- 172.21.15.7:0/287898204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c4c0740f0 msgr2=0x7f2c4c1dba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c4c0740f0 0x7f2c4c1dba90 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2c4800c750 tx=0x7f2c4800cb10 comp rx=0 tx=0).stop 2023-03-03T10:39:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 -- 172.21.15.7:0/287898204 shutdown_connections 2023-03-03T10:39:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c4c14fb00 0x7f2c4c1e91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c300682f0 0x7f2c3006a7a0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c4c0740f0 0x7f2c4c1dba90 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:28.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 --2- 172.21.15.7:0/287898204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c4c0c1da0 0x7f2c4c1e8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 -- 172.21.15.7:0/287898204 >> 172.21.15.7:0/287898204 conn(0x7f2c4c06c110 msgr2=0x7f2c4c14b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:39:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 -- 172.21.15.7:0/287898204 shutdown_connections 2023-03-03T10:39:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:28.532+0000 7f2c53384700 1 -- 172.21.15.7:0/287898204 wait complete. 2023-03-03T10:39:28.536 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:39:28.546 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T10:39:32.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:32.639 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:39:47.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:39:47.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.832+0000 7fa77d900700 1 Processor -- start 2023-03-03T10:39:47.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.832+0000 7fa77d900700 1 -- start start 2023-03-03T10:39:47.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.832+0000 7fa77d900700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7780c7a30 0x7fa77814e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:47.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.832+0000 7fa77d900700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa77806b6f0 0x7fa77814e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:47.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.832+0000 7fa77d900700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa778157bb0 con 0x7fa7780c7e00 2023-03-03T10:39:47.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.832+0000 7fa77d900700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77806ab00 con 0x7fa7780c7a30 2023-03-03T10:39:47.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.832+0000 7fa77d900700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7781545c0 con 0x7fa77806b6f0 2023-03-03T10:39:47.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa776ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7780c7a30 0x7fa77814e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:47.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa776ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7780c7a30 0x7fa77814e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32908/0 (socket says 172.21.15.7:32908) 2023-03-03T10:39:47.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa776ffd700 1 -- 172.21.15.7:0/3995391021 learned_addr learned my addr 172.21.15.7:0/3995391021 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:39:47.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa7777fe700 1 --2- 172.21.15.7:0/3995391021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa77806b6f0 0x7fa77814e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:47.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa776ffd700 1 -- 172.21.15.7:0/3995391021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa77806b6f0 msgr2=0x7fa77814e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:39:47.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa776ffd700 1 --2- 172.21.15.7:0/3995391021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa77806b6f0 0x7fa77814e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:47.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa776ffd700 1 -- 172.21.15.7:0/3995391021 >> v1:172.21.15.143:6789/0 conn(0x7fa7780c7e00 legacy=0x7fa778154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:47.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa776ffd700 1 -- 172.21.15.7:0/3995391021 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77806a2f0 con 0x7fa7780c7a30 2023-03-03T10:39:47.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa7777fe700 1 --2- 172.21.15.7:0/3995391021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa77806b6f0 0x7fa77814e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:39:47.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa776ffd700 1 --2- 172.21.15.7:0/3995391021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7a30 0x7fa77814e160 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa768006fd0 tx=0x7fa768009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c933844f891874 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:39:47.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa775ffb700 1 -- 172.21.15.7:0/3995391021 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa768005e30 con 0x7fa7780c7a30 2023-03-03T10:39:47.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa775ffb700 1 -- 172.21.15.7:0/3995391021 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa768004030 con 0x7fa7780c7a30 2023-03-03T10:39:47.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.833+0000 7fa775ffb700 1 -- 172.21.15.7:0/3995391021 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7680043e0 con 0x7fa7780c7a30 2023-03-03T10:39:47.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.834+0000 7fa77d900700 1 -- 172.21.15.7:0/3995391021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7a30 msgr2=0x7fa77814e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:47.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.834+0000 7fa77d900700 1 --2- 172.21.15.7:0/3995391021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7a30 0x7fa77814e160 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fa768006fd0 tx=0x7fa768009bd0 comp rx=0 tx=0).stop 2023-03-03T10:39:47.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.834+0000 7fa77d900700 1 -- 172.21.15.7:0/3995391021 shutdown_connections 2023-03-03T10:39:47.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.834+0000 7fa77d900700 1 --2- 172.21.15.7:0/3995391021 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa77806b6f0 0x7fa77814e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:47.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.834+0000 7fa77d900700 1 --2- 172.21.15.7:0/3995391021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7a30 0x7fa77814e160 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:47.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.834+0000 7fa77d900700 1 -- 172.21.15.7:0/3995391021 >> 172.21.15.7:0/3995391021 conn(0x7fa778073d20 msgr2=0x7fa7780716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:39:47.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.834+0000 7fa77d900700 1 -- 172.21.15.7:0/3995391021 shutdown_connections 2023-03-03T10:39:47.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.834+0000 7fa77d900700 1 -- 172.21.15.7:0/3995391021 wait complete. 2023-03-03T10:39:47.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.835+0000 7fa77d900700 1 Processor -- start 2023-03-03T10:39:47.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.835+0000 7fa77d900700 1 -- start start 2023-03-03T10:39:47.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa77d900700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7780c7a30 0x7fa7780c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:47.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa77d900700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7e00 0x7fa7780c2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:47.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa77d900700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7780c2640 0x7fa7781ecf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:47.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa77d900700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7780c4e40 con 0x7fa7780c7e00 2023-03-03T10:39:47.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa77d900700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77814cef0 con 0x7fa7780c2640 2023-03-03T10:39:47.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa77d900700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77814d1d0 con 0x7fa7780c7a30 2023-03-03T10:39:47.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa7767fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7e00 0x7fa7780c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:47.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa7767fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7e00 0x7fa7780c2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32916/0 (socket says 172.21.15.7:32916) 2023-03-03T10:39:47.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa7767fc700 1 -- 172.21.15.7:0/2971619015 learned_addr learned my addr 172.21.15.7:0/2971619015 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:39:47.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa776ffd700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7780c7a30 0x7fa7780c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:47.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa7777fe700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7780c2640 0x7fa7781ecf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:47.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa7767fc700 1 -- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7780c7a30 msgr2=0x7fa7780c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:47.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa7767fc700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7780c7a30 0x7fa7780c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:47.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa7767fc700 1 -- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7780c2640 msgr2=0x7fa7781ecf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:47.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa7767fc700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7780c2640 0x7fa7781ecf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:47.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa7767fc700 1 -- 172.21.15.7:0/2971619015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa768009500 con 0x7fa7780c7e00 2023-03-03T10:39:47.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.836+0000 7fa7777fe700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7780c2640 0x7fa7781ecf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:39:47.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.837+0000 7fa7767fc700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7e00 0x7fa7780c2100 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa76000c420 tx=0x7fa76000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:39:47.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.837+0000 7fa76ffff700 1 -- 172.21.15.7:0/2971619015 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7600052d0 con 0x7fa7780c7e00 2023-03-03T10:39:47.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.837+0000 7fa76ffff700 1 -- 172.21.15.7:0/2971619015 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa760005430 con 0x7fa7780c7e00 2023-03-03T10:39:47.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.837+0000 7fa77d900700 1 -- 172.21.15.7:0/2971619015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77806a540 con 0x7fa7780c7e00 2023-03-03T10:39:47.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.837+0000 7fa76ffff700 1 -- 172.21.15.7:0/2971619015 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7600038c0 con 0x7fa7780c7e00 2023-03-03T10:39:47.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.837+0000 7fa77d900700 1 -- 172.21.15.7:0/2971619015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7780bfbb0 con 0x7fa7780c7e00 2023-03-03T10:39:47.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.838+0000 7fa77d900700 1 -- 172.21.15.7:0/2971619015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77806a820 con 0x7fa7780c7e00 2023-03-03T10:39:47.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.838+0000 7fa76ffff700 1 -- 172.21.15.7:0/2971619015 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa760019030 con 0x7fa7780c7e00 2023-03-03T10:39:47.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.839+0000 7fa76ffff700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7580682f0 0x7fa75806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:39:47.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.839+0000 7fa76ffff700 1 -- 172.21.15.7:0/2971619015 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa7600853e0 con 0x7fa7780c7e00 2023-03-03T10:39:47.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.839+0000 7fa776ffd700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7580682f0 0x7fa75806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:39:47.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.840+0000 7fa776ffd700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7580682f0 0x7fa75806a7a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa768000c00 tx=0x7fa768008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:39:47.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:47.843+0000 7fa76ffff700 1 -- 172.21.15.7:0/2971619015 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa760051780 con 0x7fa7780c7e00 2023-03-03T10:39:48.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.081+0000 7fa77d900700 1 -- 172.21.15.7:0/2971619015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa778074bb0 con 0x7fa7780c7e00 2023-03-03T10:39:48.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.082+0000 7fa76ffff700 1 -- 172.21.15.7:0/2971619015 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa760054da0 con 0x7fa7780c7e00 2023-03-03T10:39:48.085 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:39:48.085 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:39:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 -- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7580682f0 msgr2=0x7fa75806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7580682f0 0x7fa75806a7a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa768000c00 tx=0x7fa768008070 comp rx=0 tx=0).stop 2023-03-03T10:39:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 -- 172.21.15.7:0/2971619015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7e00 msgr2=0x7fa7780c2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:39:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7e00 0x7fa7780c2100 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fa76000c420 tx=0x7fa76000c7e0 comp rx=0 tx=0).stop 2023-03-03T10:39:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 -- 172.21.15.7:0/2971619015 shutdown_connections 2023-03-03T10:39:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7780c2640 0x7fa7781ecf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7780c7a30 0x7fa7780c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7580682f0 0x7fa75806a7a0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 --2- 172.21.15.7:0/2971619015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7780c7e00 0x7fa7780c2100 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:39:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 -- 172.21.15.7:0/2971619015 >> 172.21.15.7:0/2971619015 conn(0x7fa77806b6f0 msgr2=0x7fa778152430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:39:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 -- 172.21.15.7:0/2971619015 shutdown_connections 2023-03-03T10:39:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:39:48.084+0000 7fa77d900700 1 -- 172.21.15.7:0/2971619015 wait complete. 2023-03-03T10:39:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:39:48.098 INFO:tasks.mds_thrash.fs.[b]:waiting for 28 secs before thrashing 2023-03-03T10:40:00.002 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T10:39:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T10:40:02.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:02.702 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:16.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:40:16.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.237+0000 7f4efcf82700 1 Processor -- start 2023-03-03T10:40:16.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.237+0000 7f4efcf82700 1 -- start start 2023-03-03T10:40:16.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4efcf82700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ef814f9d0 0x7f4ef8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:16.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4efcf82700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ef80abe20 0x7f4ef81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:16.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4efcf82700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef80c2f30 con 0x7f4ef814fda0 2023-03-03T10:40:16.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4efcf82700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef80c31d0 con 0x7f4ef80abe20 2023-03-03T10:40:16.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4efcf82700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef81524a0 con 0x7f4ef814f9d0 2023-03-03T10:40:16.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4ef6d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ef80abe20 0x7f4ef81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:16.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4ef6d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ef80abe20 0x7f4ef81529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32954/0 (socket says 172.21.15.7:32954) 2023-03-03T10:40:16.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4ef6d9d700 1 -- 172.21.15.7:0/2817843200 learned_addr learned my addr 172.21.15.7:0/2817843200 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:40:16.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4ef659c700 1 --2- 172.21.15.7:0/2817843200 >> v2:172.21.15.143:3300/0 conn(0x7f4ef814f9d0 0x7f4ef8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:16.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4ef6d9d700 1 -- 172.21.15.7:0/2817843200 >> v2:172.21.15.143:3300/0 conn(0x7f4ef814f9d0 msgr2=0x7f4ef8156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:40:16.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4ef6d9d700 1 --2- 172.21.15.7:0/2817843200 >> v2:172.21.15.143:3300/0 conn(0x7f4ef814f9d0 0x7f4ef8156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:16.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4ef6d9d700 1 -- 172.21.15.7:0/2817843200 >> v1:172.21.15.143:6789/0 conn(0x7f4ef814fda0 legacy=0x7f4ef8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:16.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4ef6d9d700 1 -- 172.21.15.7:0/2817843200 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef80c34b0 con 0x7f4ef80abe20 2023-03-03T10:40:16.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.238+0000 7f4ef659c700 1 --2- 172.21.15.7:0/2817843200 >> v2:172.21.15.143:3300/0 conn(0x7f4ef814f9d0 0x7f4ef8156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:40:16.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4ef6d9d700 1 --2- 172.21.15.7:0/2817843200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef80abe20 0x7f4ef81529c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80710a0 tx=0x7f4ef000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edabf07cafc6ff6a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:16.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4ef559a700 1 -- 172.21.15.7:0/2817843200 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef000e070 con 0x7f4ef80abe20 2023-03-03T10:40:16.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4ef559a700 1 -- 172.21.15.7:0/2817843200 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef000b9c0 con 0x7f4ef80abe20 2023-03-03T10:40:16.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4ef559a700 1 -- 172.21.15.7:0/2817843200 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef000bd70 con 0x7f4ef80abe20 2023-03-03T10:40:16.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4efcf82700 1 -- 172.21.15.7:0/2817843200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef80abe20 msgr2=0x7f4ef81529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:16.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4efcf82700 1 --2- 172.21.15.7:0/2817843200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef80abe20 0x7f4ef81529c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80710a0 tx=0x7f4ef000b330 comp rx=0 tx=0).stop 2023-03-03T10:40:16.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4efcf82700 1 -- 172.21.15.7:0/2817843200 shutdown_connections 2023-03-03T10:40:16.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4efcf82700 1 --2- 172.21.15.7:0/2817843200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef80abe20 0x7f4ef81529c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:16.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4efcf82700 1 --2- 172.21.15.7:0/2817843200 >> v2:172.21.15.143:3300/0 conn(0x7f4ef814f9d0 0x7f4ef8156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:16.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4efcf82700 1 -- 172.21.15.7:0/2817843200 >> 172.21.15.7:0/2817843200 conn(0x7f4ef80c8900 msgr2=0x7f4ef80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:40:16.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.239+0000 7f4efcf82700 1 -- 172.21.15.7:0/2817843200 shutdown_connections 2023-03-03T10:40:16.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.240+0000 7f4efcf82700 1 -- 172.21.15.7:0/2817843200 wait complete. 2023-03-03T10:40:16.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.241+0000 7f4efcf82700 1 Processor -- start 2023-03-03T10:40:16.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.241+0000 7f4efcf82700 1 -- start start 2023-03-03T10:40:16.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4efcf82700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef80abe20 0x7f4ef80c2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:16.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4efcf82700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef814f9d0 0x7f4ef80c0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:16.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4efcf82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef814fda0 0x7f4ef81528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:16.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4efcf82700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef81ead90 con 0x7f4ef814f9d0 2023-03-03T10:40:16.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4efcf82700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef81eb030 con 0x7f4ef80abe20 2023-03-03T10:40:16.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4efcf82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef81eb310 con 0x7f4ef814fda0 2023-03-03T10:40:16.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef814f9d0 0x7f4ef80c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:16.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef814f9d0 0x7f4ef80c0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32958/0 (socket says 172.21.15.7:32958) 2023-03-03T10:40:16.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef5d9b700 1 -- 172.21.15.7:0/3740586233 learned_addr learned my addr 172.21.15.7:0/3740586233 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:40:16.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef6d9d700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef814fda0 0x7f4ef81528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:16.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef659c700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef80abe20 0x7f4ef80c2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:16.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef5d9b700 1 -- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef814fda0 msgr2=0x7f4ef81528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef5d9b700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef814fda0 0x7f4ef81528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef5d9b700 1 -- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef80abe20 msgr2=0x7f4ef80c2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef5d9b700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef80abe20 0x7f4ef80c2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef5d9b700 1 -- 172.21.15.7:0/3740586233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef0009b30 con 0x7f4ef814f9d0 2023-03-03T10:40:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef659c700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef80abe20 0x7f4ef80c2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:40:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef6d9d700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef814fda0 0x7f4ef81528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:40:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ef5d9b700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef814f9d0 0x7f4ef80c0580 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00c4b0 tx=0x7f4eec00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ee6ffd700 1 -- 172.21.15.7:0/3740586233 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec0052d0 con 0x7f4ef814f9d0 2023-03-03T10:40:16.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ee6ffd700 1 -- 172.21.15.7:0/3740586233 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eec005430 con 0x7f4ef814f9d0 2023-03-03T10:40:16.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.242+0000 7f4ee6ffd700 1 -- 172.21.15.7:0/3740586233 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec0038c0 con 0x7f4ef814f9d0 2023-03-03T10:40:16.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.243+0000 7f4efcf82700 1 -- 172.21.15.7:0/3740586233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef80ac350 con 0x7f4ef814f9d0 2023-03-03T10:40:16.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.243+0000 7f4efcf82700 1 -- 172.21.15.7:0/3740586233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef80ab630 con 0x7f4ef814f9d0 2023-03-03T10:40:16.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.243+0000 7f4efcf82700 1 -- 172.21.15.7:0/3740586233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef8071fa0 con 0x7f4ef814f9d0 2023-03-03T10:40:16.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.244+0000 7f4ee6ffd700 1 -- 172.21.15.7:0/3740586233 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4eec019030 con 0x7f4ef814f9d0 2023-03-03T10:40:16.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.244+0000 7f4ee6ffd700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 0x7f4ed806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:16.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.245+0000 7f4ee6ffd700 1 -- 172.21.15.7:0/3740586233 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4eec0854f0 con 0x7f4ef814f9d0 2023-03-03T10:40:16.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.245+0000 7f4ef659c700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 0x7f4ed806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:16.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.245+0000 7f4ef659c700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 0x7f4ed806a7a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009a60 tx=0x7f4ee8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.250+0000 7f4ee6ffd700 1 -- 172.21.15.7:0/3740586233 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4eec051890 con 0x7f4ef814f9d0 2023-03-03T10:40:16.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.490+0000 7f4efcf82700 1 -- 172.21.15.7:0/3740586233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ef8074bb0 con 0x7f4ef814f9d0 2023-03-03T10:40:16.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.491+0000 7f4ee6ffd700 1 -- 172.21.15.7:0/3740586233 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4eec054eb0 con 0x7f4ef814f9d0 2023-03-03T10:40:16.493 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:40:16.493 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:40:16.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 -- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 msgr2=0x7f4ed806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:16.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 0x7f4ed806a7a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009a60 tx=0x7f4ee8008040 comp rx=0 tx=0).stop 2023-03-03T10:40:16.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 -- 172.21.15.7:0/3740586233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef814f9d0 msgr2=0x7f4ef80c0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:16.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef814f9d0 0x7f4ef80c0580 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00c4b0 tx=0x7f4eec00c870 comp rx=0 tx=0).stop 2023-03-03T10:40:16.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 -- 172.21.15.7:0/3740586233 shutdown_connections 2023-03-03T10:40:16.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef814fda0 0x7f4ef81528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:16.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef80abe20 0x7f4ef80c2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:16.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 0x7f4ed806a7a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:16.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 --2- 172.21.15.7:0/3740586233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef814f9d0 0x7f4ef80c0580 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:16.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 -- 172.21.15.7:0/3740586233 >> 172.21.15.7:0/3740586233 conn(0x7f4ef80c8900 msgr2=0x7f4ef80c6550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:40:16.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 -- 172.21.15.7:0/3740586233 shutdown_connections 2023-03-03T10:40:16.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:16.493+0000 7f4efcf82700 1 -- 172.21.15.7:0/3740586233 wait complete. 2023-03-03T10:40:16.497 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:40:16.507 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-03-03T10:40:32.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:32.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:40:33.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:40:33.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.684+0000 7febe61b3700 1 Processor -- start 2023-03-03T10:40:33.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.684+0000 7febe61b3700 1 -- start start 2023-03-03T10:40:33.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febe61b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014f9d0 0x7febe0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:33.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febe61b3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe00c2f30 con 0x7febe014fda0 2023-03-03T10:40:33.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febe61b3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe00c31d0 con 0x7febe00abe20 2023-03-03T10:40:33.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febe61b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe01524a0 con 0x7febe014f9d0 2023-03-03T10:40:33.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febdeffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7febe014fda0 0x7febe0071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55190/0 (socket says 172.21.15.7:55190) 2023-03-03T10:40:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febdeffd700 1 -- 172.21.15.7:0/2213324118 learned_addr learned my addr 172.21.15.7:0/2213324118 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:40:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255040826 0 0) 0x7febe00c2f30 con 0x7febe014fda0 2023-03-03T10:40:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febdf7fe700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014f9d0 0x7febe0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc80035b0 con 0x7febe014fda0 2023-03-03T10:40:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2165374731 0 0) 0x7febc80035b0 con 0x7febe014fda0 2023-03-03T10:40:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014f9d0 msgr2=0x7febe0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:40:33.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014f9d0 0x7febe0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 >> v1:172.21.15.143:6789/0 conn(0x7febe00abe20 legacy=0x7febe01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe00c34b0 con 0x7febe014fda0 2023-03-03T10:40:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7febd4001670 con 0x7febe014fda0 2023-03-03T10:40:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 >> v1:172.21.15.7:6789/0 conn(0x7febe014fda0 legacy=0x7febe0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc8003aa0 0x7febc8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014f9d0 0x7febc8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc8004d90 0x7febc8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:33.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe01524a0 con 0x7febc8003aa0 2023-03-03T10:40:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc800bc90 con 0x7febc8004d90 2023-03-03T10:40:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.685+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc800be50 con 0x7febe014f9d0 2023-03-03T10:40:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febdf7fe700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc8003aa0 0x7febc8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febdeffd700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014f9d0 0x7febc8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febdffff700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc8004d90 0x7febc8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febdf7fe700 1 -- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014f9d0 msgr2=0x7febc8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:40:33.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febdf7fe700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014f9d0 0x7febc8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febdf7fe700 1 -- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc8004d90 msgr2=0x7febc8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:40:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febdf7fe700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc8004d90 0x7febc8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febdf7fe700 1 -- 172.21.15.7:0/2213324118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc80035b0 con 0x7febc8003aa0 2023-03-03T10:40:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febdf7fe700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc8003aa0 0x7febc8003eb0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7febd0009b60 tx=0x7febc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261c421179fab7f3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd000e6c0 con 0x7febc8003aa0 2023-03-03T10:40:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd000e820 con 0x7febc8003aa0 2023-03-03T10:40:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.686+0000 7febde7fc700 1 -- 172.21.15.7:0/2213324118 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd000eb90 con 0x7febc8003aa0 2023-03-03T10:40:33.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.687+0000 7febe61b3700 1 -- 172.21.15.7:0/2213324118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc8003aa0 msgr2=0x7febc8003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:33.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.687+0000 7febe61b3700 1 --2- 172.21.15.7:0/2213324118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc8003aa0 0x7febc8003eb0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7febd0009b60 tx=0x7febc8005880 comp rx=0 tx=0).stop 2023-03-03T10:40:33.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.687+0000 7febdffff700 1 -- 172.21.15.7:0/2213324118 reap_dead start 2023-03-03T10:40:33.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.687+0000 7febe61b3700 1 -- 172.21.15.7:0/2213324118 shutdown_connections 2023-03-03T10:40:33.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.687+0000 7febe61b3700 1 -- 172.21.15.7:0/2213324118 >> 172.21.15.7:0/2213324118 conn(0x7febe00c8900 msgr2=0x7febe00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:40:33.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.687+0000 7febe61b3700 1 -- 172.21.15.7:0/2213324118 shutdown_connections 2023-03-03T10:40:33.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.687+0000 7febe61b3700 1 -- 172.21.15.7:0/2213324118 wait complete. 2023-03-03T10:40:33.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.688+0000 7febe61b3700 1 Processor -- start 2023-03-03T10:40:33.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.688+0000 7febe61b3700 1 -- start start 2023-03-03T10:40:33.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febe61b3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc8003aa0 0x7febe014d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:33.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febe61b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febe00c26c0 0x7febe014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:33.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febe61b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014afd0 0x7febe0153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:33.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febe61b3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe01ead00 con 0x7febe00c26c0 2023-03-03T10:40:33.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febe61b3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe01eafe0 con 0x7febc8003aa0 2023-03-03T10:40:33.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febe61b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe01eb2c0 con 0x7febe014afd0 2023-03-03T10:40:33.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febe00c26c0 0x7febe014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:33.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febe00c26c0 0x7febe014aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32990/0 (socket says 172.21.15.7:32990) 2023-03-03T10:40:33.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdeffd700 1 -- 172.21.15.7:0/1604974569 learned_addr learned my addr 172.21.15.7:0/1604974569 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:40:33.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdf7fe700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc8003aa0 0x7febe014d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:33.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdffff700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014afd0 0x7febe0153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:33.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdeffd700 1 -- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014afd0 msgr2=0x7febe0153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:33.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdeffd700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014afd0 0x7febe0153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:33.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdeffd700 1 -- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc8003aa0 msgr2=0x7febe014d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:33.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdeffd700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc8003aa0 0x7febe014d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:33.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdeffd700 1 -- 172.21.15.7:0/1604974569 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe00ac350 con 0x7febe00c26c0 2023-03-03T10:40:33.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.689+0000 7febdeffd700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febe00c26c0 0x7febe014aa90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7febd40072c0 tx=0x7febd40075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:33.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.690+0000 7febc7fff700 1 -- 172.21.15.7:0/1604974569 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4011580 con 0x7febe00c26c0 2023-03-03T10:40:33.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.690+0000 7febc7fff700 1 -- 172.21.15.7:0/1604974569 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd4011700 con 0x7febe00c26c0 2023-03-03T10:40:33.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.690+0000 7febe61b3700 1 -- 172.21.15.7:0/1604974569 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe00ab630 con 0x7febe00c26c0 2023-03-03T10:40:33.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.690+0000 7febc7fff700 1 -- 172.21.15.7:0/1604974569 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4011ab0 con 0x7febe00c26c0 2023-03-03T10:40:33.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.690+0000 7febe61b3700 1 -- 172.21.15.7:0/1604974569 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe0071fa0 con 0x7febe00c26c0 2023-03-03T10:40:33.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.691+0000 7febe61b3700 1 -- 172.21.15.7:0/1604974569 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe0153c80 con 0x7febe00c26c0 2023-03-03T10:40:33.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.691+0000 7febc7fff700 1 -- 172.21.15.7:0/1604974569 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7febd4011c10 con 0x7febe00c26c0 2023-03-03T10:40:33.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.692+0000 7febc7fff700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febcc0682f0 0x7febcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:33.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.692+0000 7febc7fff700 1 -- 172.21.15.7:0/1604974569 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7febd4086150 con 0x7febe00c26c0 2023-03-03T10:40:33.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.692+0000 7febdf7fe700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febcc0682f0 0x7febcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:33.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.692+0000 7febdf7fe700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febcc0682f0 0x7febcc06a7a0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7febd00090b0 tx=0x7febd001f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:33.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.696+0000 7febc7fff700 1 -- 172.21.15.7:0/1604974569 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7febd40524f0 con 0x7febe00c26c0 2023-03-03T10:40:33.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.931+0000 7febe61b3700 1 -- 172.21.15.7:0/1604974569 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febe0074bb0 con 0x7febe00c26c0 2023-03-03T10:40:33.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.932+0000 7febc7fff700 1 -- 172.21.15.7:0/1604974569 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7febd4055b10 con 0x7febe00c26c0 2023-03-03T10:40:33.934 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:40:33.934 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:40:33.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 -- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febcc0682f0 msgr2=0x7febcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febcc0682f0 0x7febcc06a7a0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7febd00090b0 tx=0x7febd001f040 comp rx=0 tx=0).stop 2023-03-03T10:40:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 -- 172.21.15.7:0/1604974569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febe00c26c0 msgr2=0x7febe014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febe00c26c0 0x7febe014aa90 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7febd40072c0 tx=0x7febd40075d0 comp rx=0 tx=0).stop 2023-03-03T10:40:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 -- 172.21.15.7:0/1604974569 shutdown_connections 2023-03-03T10:40:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febe014afd0 0x7febe0153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febc8003aa0 0x7febe014d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:33.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febcc0682f0 0x7febcc06a7a0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 --2- 172.21.15.7:0/1604974569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febe00c26c0 0x7febe014aa90 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 -- 172.21.15.7:0/1604974569 >> 172.21.15.7:0/1604974569 conn(0x7febe00c8900 msgr2=0x7febe0150d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:40:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 -- 172.21.15.7:0/1604974569 shutdown_connections 2023-03-03T10:40:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:33.934+0000 7febe61b3700 1 -- 172.21.15.7:0/1604974569 wait complete. 2023-03-03T10:40:33.938 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:40:33.948 INFO:tasks.mds_thrash.fs.[a]:waiting for 25 secs before thrashing 2023-03-03T10:40:58.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:40:59.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.087+0000 7f9a42387700 1 Processor -- start 2023-03-03T10:40:59.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.087+0000 7f9a42387700 1 -- start start 2023-03-03T10:40:59.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.087+0000 7f9a42387700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a3c0aa030 0x7f9a3c158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:59.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.087+0000 7f9a42387700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c0a9c20 con 0x7f9a3c14b0d0 2023-03-03T10:40:59.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.087+0000 7f9a42387700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c154f10 con 0x7f9a3c0aad20 2023-03-03T10:40:59.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.087+0000 7f9a42387700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c0ac3c0 con 0x7f9a3c0aa030 2023-03-03T10:40:59.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9a3c14b0d0 0x7f9a3c071800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55216/0 (socket says 172.21.15.7:55216) 2023-03-03T10:40:59.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3bfff700 1 -- 172.21.15.7:0/3093368514 learned_addr learned my addr 172.21.15.7:0/3093368514 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:40:59.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135811367 0 0) 0x7f9a3c0a9c20 con 0x7f9a3c14b0d0 2023-03-03T10:40:59.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a40924700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a3c0aa030 0x7f9a3c158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:59.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a240035b0 con 0x7f9a3c14b0d0 2023-03-03T10:40:59.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 623400972 0 0) 0x7f9a240035b0 con 0x7f9a3c14b0d0 2023-03-03T10:40:59.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a3c0aa030 msgr2=0x7f9a3c158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:40:59.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a3c0aa030 0x7f9a3c158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:59.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 >> v1:172.21.15.143:6789/0 conn(0x7f9a3c0aad20 legacy=0x7f9a3c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:59.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c06fc70 con 0x7f9a3c14b0d0 2023-03-03T10:40:59.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f9a2c001670 con 0x7f9a3c14b0d0 2023-03-03T10:40:59.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 >> v1:172.21.15.7:6789/0 conn(0x7f9a3c14b0d0 legacy=0x7f9a3c071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:59.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a24003aa0 0x7f9a24003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:59.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a24004850 0x7f9a24004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:59.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a3c0aa030 0x7f9a24005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:59.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c0ac3c0 con 0x7f9a24003aa0 2023-03-03T10:40:59.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2400bb20 con 0x7f9a24004850 2023-03-03T10:40:59.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2400bce0 con 0x7f9a3c0aa030 2023-03-03T10:40:59.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3bfff700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a24003aa0 0x7f9a24003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:59.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a3b7fe700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a24004850 0x7f9a24004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:59.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.088+0000 7f9a40924700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a3c0aa030 0x7f9a24005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:59.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a3bfff700 1 -- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a3c0aa030 msgr2=0x7f9a24005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:40:59.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a3bfff700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a3c0aa030 0x7f9a24005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:59.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a3bfff700 1 -- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a24004850 msgr2=0x7f9a24004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:40:59.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a3bfff700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a24004850 0x7f9a24004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:59.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a3bfff700 1 -- 172.21.15.7:0/3093368514 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a240035b0 con 0x7f9a24003aa0 2023-03-03T10:40:59.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a3bfff700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a24003aa0 0x7f9a24003e70 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00c240 tx=0x7f9a24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79125a9d63daaa0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:59.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c00cde0 con 0x7f9a24003aa0 2023-03-03T10:40:59.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a2c007ba0 con 0x7f9a24003aa0 2023-03-03T10:40:59.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a3affd700 1 -- 172.21.15.7:0/3093368514 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2c005b80 con 0x7f9a24003aa0 2023-03-03T10:40:59.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a42387700 1 -- 172.21.15.7:0/3093368514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a24003aa0 msgr2=0x7f9a24003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:59.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.089+0000 7f9a42387700 1 --2- 172.21.15.7:0/3093368514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a24003aa0 0x7f9a24003e70 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c00c240 tx=0x7f9a24005880 comp rx=0 tx=0).stop 2023-03-03T10:40:59.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.090+0000 7f9a40924700 1 -- 172.21.15.7:0/3093368514 reap_dead start 2023-03-03T10:40:59.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.090+0000 7f9a42387700 1 -- 172.21.15.7:0/3093368514 shutdown_connections 2023-03-03T10:40:59.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.090+0000 7f9a42387700 1 -- 172.21.15.7:0/3093368514 >> 172.21.15.7:0/3093368514 conn(0x7f9a3c0c8900 msgr2=0x7f9a3c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:40:59.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.090+0000 7f9a42387700 1 -- 172.21.15.7:0/3093368514 shutdown_connections 2023-03-03T10:40:59.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.090+0000 7f9a42387700 1 -- 172.21.15.7:0/3093368514 wait complete. 2023-03-03T10:40:59.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.091+0000 7f9a42387700 1 Processor -- start 2023-03-03T10:40:59.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.091+0000 7f9a42387700 1 -- start start 2023-03-03T10:40:59.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a42387700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a24003aa0 0x7f9a3c1568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:59.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a42387700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a3c156e10 0x7f9a3c1ea5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:59.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a42387700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a3c1eab20 0x7f9a3c146940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:59.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a42387700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c1dff00 con 0x7f9a3c156e10 2023-03-03T10:40:59.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a42387700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c1ea190 con 0x7f9a3c1eab20 2023-03-03T10:40:59.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a42387700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c1dde90 con 0x7f9a24003aa0 2023-03-03T10:40:59.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a3b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a3c156e10 0x7f9a3c1ea5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:59.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a3b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a3c156e10 0x7f9a3c1ea5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33018/0 (socket says 172.21.15.7:33018) 2023-03-03T10:40:59.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a3b7fe700 1 -- 172.21.15.7:0/3053990594 learned_addr learned my addr 172.21.15.7:0/3053990594 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:40:59.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a3bfff700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a24003aa0 0x7f9a3c1568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:59.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a40924700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a3c1eab20 0x7f9a3c146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:59.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a3b7fe700 1 -- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a24003aa0 msgr2=0x7f9a3c1568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:59.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a3b7fe700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a24003aa0 0x7f9a3c1568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:59.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a3b7fe700 1 -- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a3c1eab20 msgr2=0x7f9a3c146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:59.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a3b7fe700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a3c1eab20 0x7f9a3c146940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:59.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a3b7fe700 1 -- 172.21.15.7:0/3053990594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c06f6b0 con 0x7f9a3c156e10 2023-03-03T10:40:59.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a40924700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a3c1eab20 0x7f9a3c146940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:40:59.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a3b7fe700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a3c156e10 0x7f9a3c1ea5e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000f820 tx=0x7f9a3000fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:59.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a38ff9700 1 -- 172.21.15.7:0/3053990594 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300052e0 con 0x7f9a3c156e10 2023-03-03T10:40:59.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a38ff9700 1 -- 172.21.15.7:0/3053990594 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a30005460 con 0x7f9a3c156e10 2023-03-03T10:40:59.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.092+0000 7f9a42387700 1 -- 172.21.15.7:0/3053990594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3c155160 con 0x7f9a3c156e10 2023-03-03T10:40:59.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.093+0000 7f9a38ff9700 1 -- 172.21.15.7:0/3053990594 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300038c0 con 0x7f9a3c156e10 2023-03-03T10:40:59.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.093+0000 7f9a42387700 1 -- 172.21.15.7:0/3053990594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3c06f990 con 0x7f9a3c156e10 2023-03-03T10:40:59.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.093+0000 7f9a42387700 1 -- 172.21.15.7:0/3053990594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3c1ea300 con 0x7f9a3c156e10 2023-03-03T10:40:59.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.094+0000 7f9a38ff9700 1 -- 172.21.15.7:0/3053990594 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a30003a20 con 0x7f9a3c156e10 2023-03-03T10:40:59.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.094+0000 7f9a38ff9700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a280682f0 0x7f9a2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:40:59.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.095+0000 7f9a38ff9700 1 -- 172.21.15.7:0/3053990594 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9a30086080 con 0x7f9a3c156e10 2023-03-03T10:40:59.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.095+0000 7f9a3bfff700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a280682f0 0x7f9a2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:40:59.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.095+0000 7f9a3bfff700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a280682f0 0x7f9a2806a7a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c007d50 tx=0x7f9a2c007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:40:59.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.099+0000 7f9a38ff9700 1 -- 172.21.15.7:0/3053990594 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a3c1ea300 con 0x7f9a3c156e10 2023-03-03T10:40:59.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.335+0000 7f9a42387700 1 -- 172.21.15.7:0/3053990594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a3c1ea300 con 0x7f9a3c156e10 2023-03-03T10:40:59.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.335+0000 7f9a38ff9700 1 -- 172.21.15.7:0/3053990594 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9a3c1ea300 con 0x7f9a3c156e10 2023-03-03T10:40:59.338 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:40:59.339 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:40:59.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 -- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a280682f0 msgr2=0x7f9a2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a280682f0 0x7f9a2806a7a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c007d50 tx=0x7f9a2c007000 comp rx=0 tx=0).stop 2023-03-03T10:40:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 -- 172.21.15.7:0/3053990594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a3c156e10 msgr2=0x7f9a3c1ea5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:40:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a3c156e10 0x7f9a3c1ea5e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f9a3000f820 tx=0x7f9a3000fbe0 comp rx=0 tx=0).stop 2023-03-03T10:40:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 -- 172.21.15.7:0/3053990594 shutdown_connections 2023-03-03T10:40:59.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a3c1eab20 0x7f9a3c146940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:59.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a24003aa0 0x7f9a3c1568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:59.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a280682f0 0x7f9a2806a7a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:59.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 --2- 172.21.15.7:0/3053990594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a3c156e10 0x7f9a3c1ea5e0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:40:59.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 -- 172.21.15.7:0/3053990594 >> 172.21.15.7:0/3053990594 conn(0x7f9a3c0c8900 msgr2=0x7f9a3c0acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:40:59.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 -- 172.21.15.7:0/3053990594 shutdown_connections 2023-03-03T10:40:59.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:40:59.337+0000 7f9a42387700 1 -- 172.21.15.7:0/3053990594 wait complete. 2023-03-03T10:40:59.344 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:40:59.352 INFO:tasks.mds_thrash.fs.[a]:waiting for 113 secs before thrashing 2023-03-03T10:41:02.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:02.937 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:33.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:33.008 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:41:38.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:41:38.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.644+0000 7f6caf0b0700 1 Processor -- start 2023-03-03T10:41:38.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.644+0000 7f6caf0b0700 1 -- start start 2023-03-03T10:41:38.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.644+0000 7f6caf0b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca8151ab0 0x7f6ca80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:38.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.644+0000 7f6caf0b0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6ca806c5b0 0x7f6ca8157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:38.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.644+0000 7f6caf0b0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca806d780 con 0x7f6ca8073c30 2023-03-03T10:41:38.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.644+0000 7f6caf0b0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca806cb70 con 0x7f6ca806c5b0 2023-03-03T10:41:38.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.644+0000 7f6caf0b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca806ce50 con 0x7f6ca8151ab0 2023-03-03T10:41:38.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6cad64d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6ca806c5b0 0x7f6ca8157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:38.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6ca7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6ca8073c30 0x7f6ca806bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55260/0 (socket says 172.21.15.7:55260) 2023-03-03T10:41:38.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6ca7fff700 1 -- 172.21.15.7:0/734913857 learned_addr learned my addr 172.21.15.7:0/734913857 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:41:38.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6cace4c700 1 --2- 172.21.15.7:0/734913857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca8151ab0 0x7f6ca80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:38.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6cad64d700 1 -- 172.21.15.7:0/734913857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca8151ab0 msgr2=0x7f6ca80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:41:38.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6cad64d700 1 --2- 172.21.15.7:0/734913857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca8151ab0 0x7f6ca80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:38.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6cad64d700 1 -- 172.21.15.7:0/734913857 >> v1:172.21.15.7:6789/0 conn(0x7f6ca8073c30 legacy=0x7f6ca806bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:38.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6cad64d700 1 -- 172.21.15.7:0/734913857 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca806d130 con 0x7f6ca806c5b0 2023-03-03T10:41:38.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6ca77fe700 1 -- 172.21.15.7:0/734913857 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500504668 0 0) 0x7f6ca806d780 con 0x7f6ca8073c30 2023-03-03T10:41:38.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6cad64d700 1 --2- 172.21.15.7:0/734913857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 0x7f6ca8157dc0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009d90 tx=0x7f6ca000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1f3a466fdbbc16 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:41:38.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6ca77fe700 1 -- 172.21.15.7:0/734913857 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca000e070 con 0x7f6ca806c5b0 2023-03-03T10:41:38.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6ca77fe700 1 -- 172.21.15.7:0/734913857 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca000bbd0 con 0x7f6ca806c5b0 2023-03-03T10:41:38.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.645+0000 7f6ca77fe700 1 -- 172.21.15.7:0/734913857 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0012430 con 0x7f6ca806c5b0 2023-03-03T10:41:38.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.646+0000 7f6caf0b0700 1 -- 172.21.15.7:0/734913857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 msgr2=0x7f6ca8157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:38.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.646+0000 7f6caf0b0700 1 --2- 172.21.15.7:0/734913857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 0x7f6ca8157dc0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009d90 tx=0x7f6ca000b330 comp rx=0 tx=0).stop 2023-03-03T10:41:38.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.646+0000 7f6caf0b0700 1 -- 172.21.15.7:0/734913857 shutdown_connections 2023-03-03T10:41:38.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.646+0000 7f6caf0b0700 1 --2- 172.21.15.7:0/734913857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 0x7f6ca8157dc0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:38.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.646+0000 7f6caf0b0700 1 --2- 172.21.15.7:0/734913857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca8151ab0 0x7f6ca80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:38.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.646+0000 7f6caf0b0700 1 -- 172.21.15.7:0/734913857 >> 172.21.15.7:0/734913857 conn(0x7f6ca80a94d0 msgr2=0x7f6ca80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:41:38.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.646+0000 7f6caf0b0700 1 -- 172.21.15.7:0/734913857 shutdown_connections 2023-03-03T10:41:38.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.646+0000 7f6caf0b0700 1 -- 172.21.15.7:0/734913857 wait complete. 2023-03-03T10:41:38.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.647+0000 7f6caf0b0700 1 Processor -- start 2023-03-03T10:41:38.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.647+0000 7f6caf0b0700 1 -- start start 2023-03-03T10:41:38.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6caf0b0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 0x7f6ca814ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:38.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6caf0b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca8073c30 0x7f6ca8149140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:38.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6caf0b0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ca8151ab0 0x7f6ca8149680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:38.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6caf0b0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80be6d0 con 0x7f6ca806c5b0 2023-03-03T10:41:38.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6caf0b0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca81eaa00 con 0x7f6ca8151ab0 2023-03-03T10:41:38.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6caf0b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80c6e60 con 0x7f6ca8073c30 2023-03-03T10:41:38.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6cace4c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 0x7f6ca814ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:38.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6cace4c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 0x7f6ca814ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33054/0 (socket says 172.21.15.7:33054) 2023-03-03T10:41:38.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6cace4c700 1 -- 172.21.15.7:0/842717760 learned_addr learned my addr 172.21.15.7:0/842717760 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:41:38.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6ca7fff700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca8073c30 0x7f6ca8149140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:38.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6cad64d700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ca8151ab0 0x7f6ca8149680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:38.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6cace4c700 1 -- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca8073c30 msgr2=0x7f6ca8149140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:38.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6cace4c700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca8073c30 0x7f6ca8149140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:38.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6cace4c700 1 -- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ca8151ab0 msgr2=0x7f6ca8149680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:38.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6cace4c700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ca8151ab0 0x7f6ca8149680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:38.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.648+0000 7f6cace4c700 1 -- 172.21.15.7:0/842717760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca0009a70 con 0x7f6ca806c5b0 2023-03-03T10:41:38.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.649+0000 7f6cace4c700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 0x7f6ca814ba40 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c00d830 tx=0x7f6c9c00db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:41:38.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.649+0000 7f6ca57fa700 1 -- 172.21.15.7:0/842717760 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c0125f0 con 0x7f6ca806c5b0 2023-03-03T10:41:38.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.649+0000 7f6ca57fa700 1 -- 172.21.15.7:0/842717760 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c012750 con 0x7f6ca806c5b0 2023-03-03T10:41:38.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.649+0000 7f6caf0b0700 1 -- 172.21.15.7:0/842717760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca8151e80 con 0x7f6ca806c5b0 2023-03-03T10:41:38.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.649+0000 7f6ca57fa700 1 -- 172.21.15.7:0/842717760 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c012b00 con 0x7f6ca806c5b0 2023-03-03T10:41:38.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.649+0000 7f6caf0b0700 1 -- 172.21.15.7:0/842717760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca806d410 con 0x7f6ca806c5b0 2023-03-03T10:41:38.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.650+0000 7f6caf0b0700 1 -- 172.21.15.7:0/842717760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca8152010 con 0x7f6ca806c5b0 2023-03-03T10:41:38.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.650+0000 7f6ca57fa700 1 -- 172.21.15.7:0/842717760 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c9c012c60 con 0x7f6ca806c5b0 2023-03-03T10:41:38.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.651+0000 7f6ca57fa700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c8c0682f0 0x7f6c8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:41:38.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.651+0000 7f6ca57fa700 1 -- 172.21.15.7:0/842717760 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6c9c086370 con 0x7f6ca806c5b0 2023-03-03T10:41:38.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.651+0000 7f6ca7fff700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c8c0682f0 0x7f6c8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:41:38.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.651+0000 7f6ca7fff700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c8c0682f0 0x7f6c8c06a7a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80bef80 tx=0x7f6c94007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:41:38.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.655+0000 7f6ca57fa700 1 -- 172.21.15.7:0/842717760 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c9c052690 con 0x7f6ca806c5b0 2023-03-03T10:41:38.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.895+0000 7f6caf0b0700 1 -- 172.21.15.7:0/842717760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ca8074bb0 con 0x7f6ca806c5b0 2023-03-03T10:41:38.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.895+0000 7f6ca57fa700 1 -- 172.21.15.7:0/842717760 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6c9c055cb0 con 0x7f6ca806c5b0 2023-03-03T10:41:38.899 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:41:38.899 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:41:38.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 -- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c8c0682f0 msgr2=0x7f6c8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:38.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c8c0682f0 0x7f6c8c06a7a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6ca80bef80 tx=0x7f6c94007de0 comp rx=0 tx=0).stop 2023-03-03T10:41:38.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 -- 172.21.15.7:0/842717760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 msgr2=0x7f6ca814ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:41:38.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 0x7f6ca814ba40 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c00d830 tx=0x7f6c9c00db40 comp rx=0 tx=0).stop 2023-03-03T10:41:38.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 -- 172.21.15.7:0/842717760 shutdown_connections 2023-03-03T10:41:38.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ca8151ab0 0x7f6ca8149680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:38.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c8c0682f0 0x7f6c8c06a7a0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:38.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ca806c5b0 0x7f6ca814ba40 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:38.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 --2- 172.21.15.7:0/842717760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ca8073c30 0x7f6ca8149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:41:38.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 -- 172.21.15.7:0/842717760 >> 172.21.15.7:0/842717760 conn(0x7f6ca80a94d0 msgr2=0x7f6ca81573c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:41:38.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 -- 172.21.15.7:0/842717760 shutdown_connections 2023-03-03T10:41:38.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:41:38.898+0000 7f6caf0b0700 1 -- 172.21.15.7:0/842717760 wait complete. 2023-03-03T10:41:38.902 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:41:38.912 INFO:tasks.mds_thrash.fs.[b]:waiting for 118 secs before thrashing 2023-03-03T10:42:03.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:03.080 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:33.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:33.152 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:42:52.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:42:52.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.489+0000 7f19a6670700 1 Processor -- start 2023-03-03T10:42:52.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.489+0000 7f19a6670700 1 -- start start 2023-03-03T10:42:52.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f19a6670700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f19a014f9d0 0x7f19a0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:52.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f19a6670700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19a014fda0 0x7f19a0071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:52.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f19a6670700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a00c2f30 con 0x7f19a00abe20 2023-03-03T10:42:52.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f19a6670700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a00c31d0 con 0x7f19a014fda0 2023-03-03T10:42:52.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f19a6670700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a01524a0 con 0x7f19a014f9d0 2023-03-03T10:42:52.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f199f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19a014fda0 0x7f19a0071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:52.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f199f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19a014fda0 0x7f19a0071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33130/0 (socket says 172.21.15.7:33130) 2023-03-03T10:42:52.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f199f7fe700 1 -- 172.21.15.7:0/2418881169 learned_addr learned my addr 172.21.15.7:0/2418881169 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:42:52.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f199ffff700 1 --2- 172.21.15.7:0/2418881169 >> v2:172.21.15.143:3300/0 conn(0x7f19a014f9d0 0x7f19a0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:52.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f199f7fe700 1 -- 172.21.15.7:0/2418881169 >> v2:172.21.15.143:3300/0 conn(0x7f19a014f9d0 msgr2=0x7f19a0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:42:52.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f199f7fe700 1 --2- 172.21.15.7:0/2418881169 >> v2:172.21.15.143:3300/0 conn(0x7f19a014f9d0 0x7f19a0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:52.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f199f7fe700 1 -- 172.21.15.7:0/2418881169 >> v1:172.21.15.143:6789/0 conn(0x7f19a00abe20 legacy=0x7f19a01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:52.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f199f7fe700 1 -- 172.21.15.7:0/2418881169 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a00c34b0 con 0x7f19a014fda0 2023-03-03T10:42:52.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.490+0000 7f199ffff700 1 --2- 172.21.15.7:0/2418881169 >> v2:172.21.15.143:3300/0 conn(0x7f19a014f9d0 0x7f19a0156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:42:52.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f199f7fe700 1 --2- 172.21.15.7:0/2418881169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a014fda0 0x7f19a0071890 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f19a00710a0 tx=0x7f1988009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b86b3cd62f48cfa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:42:52.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f199effd700 1 -- 172.21.15.7:0/2418881169 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198800b870 con 0x7f19a014fda0 2023-03-03T10:42:52.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f199effd700 1 -- 172.21.15.7:0/2418881169 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198800b9d0 con 0x7f19a014fda0 2023-03-03T10:42:52.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f199effd700 1 -- 172.21.15.7:0/2418881169 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198800bd60 con 0x7f19a014fda0 2023-03-03T10:42:52.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f19a6670700 1 -- 172.21.15.7:0/2418881169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a014fda0 msgr2=0x7f19a0071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:52.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f19a6670700 1 --2- 172.21.15.7:0/2418881169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a014fda0 0x7f19a0071890 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f19a00710a0 tx=0x7f1988009bd0 comp rx=0 tx=0).stop 2023-03-03T10:42:52.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f19a6670700 1 -- 172.21.15.7:0/2418881169 shutdown_connections 2023-03-03T10:42:52.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f19a6670700 1 --2- 172.21.15.7:0/2418881169 >> v2:172.21.15.143:3300/0 conn(0x7f19a014f9d0 0x7f19a0156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:52.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f19a6670700 1 --2- 172.21.15.7:0/2418881169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a014fda0 0x7f19a0071890 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:52.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f19a6670700 1 -- 172.21.15.7:0/2418881169 >> 172.21.15.7:0/2418881169 conn(0x7f19a00c8900 msgr2=0x7f19a00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:42:52.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f19a6670700 1 -- 172.21.15.7:0/2418881169 shutdown_connections 2023-03-03T10:42:52.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.491+0000 7f19a6670700 1 -- 172.21.15.7:0/2418881169 wait complete. 2023-03-03T10:42:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.492+0000 7f19a6670700 1 Processor -- start 2023-03-03T10:42:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.493+0000 7f19a6670700 1 -- start start 2023-03-03T10:42:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.493+0000 7f19a6670700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a00abe20 0x7f19a01dce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.493+0000 7f19a6670700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19a014f9d0 0x7f19a01ea8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.493+0000 7f19a6670700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a014fda0 0x7f19a0154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:52.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.493+0000 7f19a6670700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00be510 con 0x7f19a00abe20 2023-03-03T10:42:52.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.493+0000 7f19a6670700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a01dd430 con 0x7f19a014f9d0 2023-03-03T10:42:52.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.493+0000 7f19a6670700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a01dd710 con 0x7f19a014fda0 2023-03-03T10:42:52.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.493+0000 7f199ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a00abe20 0x7f19a01dce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:52.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.493+0000 7f199ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a00abe20 0x7f19a01dce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33132/0 (socket says 172.21.15.7:33132) 2023-03-03T10:42:52.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.493+0000 7f199ffff700 1 -- 172.21.15.7:0/2214747841 learned_addr learned my addr 172.21.15.7:0/2214747841 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:42:52.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f199f7fe700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19a014f9d0 0x7f19a01ea8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:52.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f19a4c0d700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a014fda0 0x7f19a0154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:52.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f199ffff700 1 -- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a014fda0 msgr2=0x7f19a0154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:52.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f199ffff700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a014fda0 0x7f19a0154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:52.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f199ffff700 1 -- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19a014f9d0 msgr2=0x7f19a01ea8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:52.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f199ffff700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19a014f9d0 0x7f19a01ea8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:52.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f199ffff700 1 -- 172.21.15.7:0/2214747841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1988009550 con 0x7f19a00abe20 2023-03-03T10:42:52.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f19a4c0d700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a014fda0 0x7f19a0154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:42:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f199ffff700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a00abe20 0x7f19a01dce50 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f199000e810 tx=0x7f199000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:42:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f199cff9700 1 -- 172.21.15.7:0/2214747841 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990013610 con 0x7f19a00abe20 2023-03-03T10:42:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f199cff9700 1 -- 172.21.15.7:0/2214747841 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1990013770 con 0x7f19a00abe20 2023-03-03T10:42:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f19a6670700 1 -- 172.21.15.7:0/2214747841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a00ac350 con 0x7f19a00abe20 2023-03-03T10:42:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f199cff9700 1 -- 172.21.15.7:0/2214747841 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990013b20 con 0x7f19a00abe20 2023-03-03T10:42:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.494+0000 7f19a6670700 1 -- 172.21.15.7:0/2214747841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a00ab630 con 0x7f19a00abe20 2023-03-03T10:42:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.495+0000 7f19a6670700 1 -- 172.21.15.7:0/2214747841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a00c3e10 con 0x7f19a00abe20 2023-03-03T10:42:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.496+0000 7f199cff9700 1 -- 172.21.15.7:0/2214747841 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1990023070 con 0x7f19a00abe20 2023-03-03T10:42:52.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.496+0000 7f199cff9700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 0x7f198006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:42:52.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.496+0000 7f199cff9700 1 -- 172.21.15.7:0/2214747841 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1990085d30 con 0x7f19a00abe20 2023-03-03T10:42:52.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.496+0000 7f199f7fe700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 0x7f198006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:42:52.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.497+0000 7f199f7fe700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 0x7f198006a7a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f19a01501d0 tx=0x7f1988008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:42:52.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.500+0000 7f199cff9700 1 -- 172.21.15.7:0/2214747841 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f19900520d0 con 0x7f19a00abe20 2023-03-03T10:42:52.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.740+0000 7f19a6670700 1 -- 172.21.15.7:0/2214747841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19a01e7aa0 con 0x7f19a00abe20 2023-03-03T10:42:52.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.741+0000 7f199cff9700 1 -- 172.21.15.7:0/2214747841 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f19900556f0 con 0x7f19a00abe20 2023-03-03T10:42:52.743 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:42:52.744 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:42:52.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 -- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 msgr2=0x7f198006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 0x7f198006a7a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f19a01501d0 tx=0x7f1988008030 comp rx=0 tx=0).stop 2023-03-03T10:42:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 -- 172.21.15.7:0/2214747841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a00abe20 msgr2=0x7f19a01dce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:42:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a00abe20 0x7f19a01dce50 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f199000e810 tx=0x7f199000eb20 comp rx=0 tx=0).stop 2023-03-03T10:42:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 -- 172.21.15.7:0/2214747841 shutdown_connections 2023-03-03T10:42:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19a014fda0 0x7f19a0154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 0x7f198006a7a0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:52.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19a00abe20 0x7f19a01dce50 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 --2- 172.21.15.7:0/2214747841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19a014f9d0 0x7f19a01ea8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:42:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 -- 172.21.15.7:0/2214747841 >> 172.21.15.7:0/2214747841 conn(0x7f19a00c8900 msgr2=0x7f19a0150510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:42:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 -- 172.21.15.7:0/2214747841 shutdown_connections 2023-03-03T10:42:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:42:52.743+0000 7f19a6670700 1 -- 172.21.15.7:0/2214747841 wait complete. 2023-03-03T10:42:52.747 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:42:52.757 INFO:tasks.mds_thrash.fs.[a]:waiting for 52 secs before thrashing 2023-03-03T10:43:03.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:03.225 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:33.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:33.387 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:43:36.913 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:43:37.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.049+0000 7f76aad4c700 1 Processor -- start 2023-03-03T10:43:37.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.049+0000 7f76aad4c700 1 -- start start 2023-03-03T10:43:37.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.049+0000 7f76aad4c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76a40c00e0 0x7f76a40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:37.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.049+0000 7f76aad4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76a4154ff0 0x7f76a40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:37.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.049+0000 7f76aad4c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a40c7310 con 0x7f76a40caa30 2023-03-03T10:43:37.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.049+0000 7f76aad4c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a40bfd10 con 0x7f76a40c00e0 2023-03-03T10:43:37.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.049+0000 7f76aad4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a40bfa30 con 0x7f76a4154ff0 2023-03-03T10:43:37.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76a40c00e0 0x7f76a40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:37.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f76a8ae8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f76a40caa30 0x7f76a40c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55414/0 (socket says 172.21.15.7:55414) 2023-03-03T10:43:37.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f76a8ae8700 1 -- 172.21.15.7:0/1434068133 learned_addr learned my addr 172.21.15.7:0/1434068133 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:43:37.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f76a92e9700 1 --2- 172.21.15.7:0/1434068133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76a4154ff0 0x7f76a40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:37.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769b7fe700 1 -- 172.21.15.7:0/1434068133 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986050936 0 0) 0x7f76a40c7310 con 0x7f76a40caa30 2023-03-03T10:43:37.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769b7fe700 1 -- 172.21.15.7:0/1434068133 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c0035b0 con 0x7f76a40caa30 2023-03-03T10:43:37.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769bfff700 1 -- 172.21.15.7:0/1434068133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76a4154ff0 msgr2=0x7f76a40cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:43:37.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769bfff700 1 --2- 172.21.15.7:0/1434068133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76a4154ff0 0x7f76a40cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:37.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769bfff700 1 -- 172.21.15.7:0/1434068133 >> v1:172.21.15.7:6789/0 conn(0x7f76a40caa30 legacy=0x7f76a40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:37.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769bfff700 1 -- 172.21.15.7:0/1434068133 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a4154510 con 0x7f76a40c00e0 2023-03-03T10:43:37.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769bfff700 1 --2- 172.21.15.7:0/1434068133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 0x7f76a40c5c60 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f7694009820 tx=0x7f7694009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b613c1773db8db00 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:37.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769b7fe700 1 -- 172.21.15.7:0/1434068133 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7694005e30 con 0x7f76a40c00e0 2023-03-03T10:43:37.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769b7fe700 1 -- 172.21.15.7:0/1434068133 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7694006030 con 0x7f76a40c00e0 2023-03-03T10:43:37.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.050+0000 7f769b7fe700 1 -- 172.21.15.7:0/1434068133 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76940063c0 con 0x7f76a40c00e0 2023-03-03T10:43:37.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.051+0000 7f76aad4c700 1 -- 172.21.15.7:0/1434068133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 msgr2=0x7f76a40c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:37.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.051+0000 7f76aad4c700 1 --2- 172.21.15.7:0/1434068133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 0x7f76a40c5c60 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f7694009820 tx=0x7f7694009b30 comp rx=0 tx=0).stop 2023-03-03T10:43:37.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.051+0000 7f76aad4c700 1 -- 172.21.15.7:0/1434068133 shutdown_connections 2023-03-03T10:43:37.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.051+0000 7f76aad4c700 1 --2- 172.21.15.7:0/1434068133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76a4154ff0 0x7f76a40cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:37.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.051+0000 7f76aad4c700 1 --2- 172.21.15.7:0/1434068133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 0x7f76a40c5c60 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:37.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.051+0000 7f76aad4c700 1 -- 172.21.15.7:0/1434068133 >> 172.21.15.7:0/1434068133 conn(0x7f76a4066e40 msgr2=0x7f76a4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:43:37.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.051+0000 7f76aad4c700 1 -- 172.21.15.7:0/1434068133 shutdown_connections 2023-03-03T10:43:37.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.051+0000 7f76aad4c700 1 -- 172.21.15.7:0/1434068133 wait complete. 2023-03-03T10:43:37.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.053+0000 7f76aad4c700 1 Processor -- start 2023-03-03T10:43:37.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.053+0000 7f76aad4c700 1 -- start start 2023-03-03T10:43:37.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76aad4c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 0x7f76a40c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:37.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76aad4c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76a40caa30 0x7f76a41ea120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:37.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76aad4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76a4154ff0 0x7f76a41ea660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:37.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76aad4c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a41ee240 con 0x7f76a40c00e0 2023-03-03T10:43:37.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76aad4c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a41ed630 con 0x7f76a40caa30 2023-03-03T10:43:37.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76aad4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a41ed910 con 0x7f76a4154ff0 2023-03-03T10:43:37.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76a8ae8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 0x7f76a40c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:37.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76a8ae8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 0x7f76a40c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33208/0 (socket says 172.21.15.7:33208) 2023-03-03T10:43:37.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76a8ae8700 1 -- 172.21.15.7:0/1851615330 learned_addr learned my addr 172.21.15.7:0/1851615330 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:43:37.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f769bfff700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76a40caa30 0x7f76a41ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:37.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76a8ae8700 1 -- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76a4154ff0 msgr2=0x7f76a41ea660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:37.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76a92e9700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76a4154ff0 0x7f76a41ea660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:37.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76a8ae8700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76a4154ff0 0x7f76a41ea660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:37.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76a8ae8700 1 -- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76a40caa30 msgr2=0x7f76a41ea120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:37.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76a8ae8700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76a40caa30 0x7f76a41ea120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:37.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76a8ae8700 1 -- 172.21.15.7:0/1851615330 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7694009430 con 0x7f76a40c00e0 2023-03-03T10:43:37.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76a8ae8700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 0x7f76a40c8a70 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f76a000c700 tx=0x7f76a000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:37.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.054+0000 7f76997fa700 1 -- 172.21.15.7:0/1851615330 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a0005d70 con 0x7f76a40c00e0 2023-03-03T10:43:37.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.055+0000 7f76997fa700 1 -- 172.21.15.7:0/1851615330 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a00070d0 con 0x7f76a40c00e0 2023-03-03T10:43:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.055+0000 7f76aad4c700 1 -- 172.21.15.7:0/1851615330 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a4154ad0 con 0x7f76a40c00e0 2023-03-03T10:43:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.055+0000 7f76997fa700 1 -- 172.21.15.7:0/1851615330 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a0007460 con 0x7f76a40c00e0 2023-03-03T10:43:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.055+0000 7f76aad4c700 1 -- 172.21.15.7:0/1851615330 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a41547f0 con 0x7f76a40c00e0 2023-03-03T10:43:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.056+0000 7f76aad4c700 1 -- 172.21.15.7:0/1851615330 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a40bf650 con 0x7f76a40c00e0 2023-03-03T10:43:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.056+0000 7f76997fa700 1 -- 172.21.15.7:0/1851615330 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f76a0019030 con 0x7f76a40c00e0 2023-03-03T10:43:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.056+0000 7f76997fa700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76900682f0 0x7f769006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.057+0000 7f76997fa700 1 -- 172.21.15.7:0/1851615330 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f76a00856a0 con 0x7f76a40c00e0 2023-03-03T10:43:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.057+0000 7f769bfff700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76900682f0 0x7f769006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.057+0000 7f769bfff700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76900682f0 0x7f769006a7a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7694013040 tx=0x7f7694008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.061+0000 7f76997fa700 1 -- 172.21.15.7:0/1851615330 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f76a00519c0 con 0x7f76a40c00e0 2023-03-03T10:43:37.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.302+0000 7f76aad4c700 1 -- 172.21.15.7:0/1851615330 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76a4074bb0 con 0x7f76a40c00e0 2023-03-03T10:43:37.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.303+0000 7f76997fa700 1 -- 172.21.15.7:0/1851615330 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f76a0054fe0 con 0x7f76a40c00e0 2023-03-03T10:43:37.306 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:43:37.307 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:43:37.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 -- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76900682f0 msgr2=0x7f769006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:37.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76900682f0 0x7f769006a7a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7694013040 tx=0x7f7694008070 comp rx=0 tx=0).stop 2023-03-03T10:43:37.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 -- 172.21.15.7:0/1851615330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 msgr2=0x7f76a40c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:37.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 0x7f76a40c8a70 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f76a000c700 tx=0x7f76a000cac0 comp rx=0 tx=0).stop 2023-03-03T10:43:37.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 -- 172.21.15.7:0/1851615330 shutdown_connections 2023-03-03T10:43:37.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76a4154ff0 0x7f76a41ea660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:37.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76900682f0 0x7f769006a7a0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:37.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76a40c00e0 0x7f76a40c8a70 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:37.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 --2- 172.21.15.7:0/1851615330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76a40caa30 0x7f76a41ea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:37.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 -- 172.21.15.7:0/1851615330 >> 172.21.15.7:0/1851615330 conn(0x7f76a4066e40 msgr2=0x7f76a4073ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:43:37.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 -- 172.21.15.7:0/1851615330 shutdown_connections 2023-03-03T10:43:37.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:37.305+0000 7f76aad4c700 1 -- 172.21.15.7:0/1851615330 wait complete. 2023-03-03T10:43:37.310 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:43:37.320 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2023-03-03T10:43:44.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:43:44.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.894+0000 7ff3a7e43700 1 Processor -- start 2023-03-03T10:43:44.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.894+0000 7ff3a7e43700 1 -- start start 2023-03-03T10:43:44.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.895+0000 7ff3a7e43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0071c30 0x7ff3a00c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:44.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.895+0000 7ff3a7e43700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff3a0155950 0x7ff3a0155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:44.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.895+0000 7ff3a7e43700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a00713c0 con 0x7ff3a00aac00 2023-03-03T10:43:44.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.895+0000 7ff3a7e43700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a0156310 con 0x7ff3a0155950 2023-03-03T10:43:44.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.895+0000 7ff3a7e43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a0152c70 con 0x7ff3a0071c30 2023-03-03T10:43:44.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.895+0000 7ff3a53de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0071c30 0x7ff3a00c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:44.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.895+0000 7ff3a63e0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff3a0155950 0x7ff3a0155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:44.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.895+0000 7ff3a53de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0071c30 0x7ff3a00c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:41294/0 (socket says 172.21.15.7:41294) 2023-03-03T10:43:44.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.895+0000 7ff3a53de700 1 -- 172.21.15.7:0/3750060099 learned_addr learned my addr 172.21.15.7:0/3750060099 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:43:44.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.896+0000 7ff3a63e0700 1 -- 172.21.15.7:0/3750060099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0071c30 msgr2=0x7ff3a00c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:43:44.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.896+0000 7ff3a63e0700 1 --2- 172.21.15.7:0/3750060099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0071c30 0x7ff3a00c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:44.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.896+0000 7ff3a63e0700 1 -- 172.21.15.7:0/3750060099 >> v1:172.21.15.143:6789/0 conn(0x7ff3a00aac00 legacy=0x7ff3a00beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:44.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.896+0000 7ff3a63e0700 1 -- 172.21.15.7:0/3750060099 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a00c4930 con 0x7ff3a0155950 2023-03-03T10:43:44.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.896+0000 7ff3a4bdd700 1 -- 172.21.15.7:0/3750060099 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479724574 0 0) 0x7ff3a00713c0 con 0x7ff3a00aac00 2023-03-03T10:43:44.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.896+0000 7ff3a63e0700 1 --2- 172.21.15.7:0/3750060099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a0155950 0x7ff3a0155dd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff398009d90 tx=0x7ff39800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=532d38f02f39b52 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:44.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.896+0000 7ff3a4bdd700 1 -- 172.21.15.7:0/3750060099 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39800e070 con 0x7ff3a0155950 2023-03-03T10:43:44.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.896+0000 7ff3a4bdd700 1 -- 172.21.15.7:0/3750060099 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39800bb20 con 0x7ff3a0155950 2023-03-03T10:43:44.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.896+0000 7ff3a4bdd700 1 -- 172.21.15.7:0/3750060099 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398012430 con 0x7ff3a0155950 2023-03-03T10:43:44.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.897+0000 7ff3a7e43700 1 -- 172.21.15.7:0/3750060099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a0155950 msgr2=0x7ff3a0155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:44.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.897+0000 7ff3a7e43700 1 --2- 172.21.15.7:0/3750060099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a0155950 0x7ff3a0155dd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff398009d90 tx=0x7ff39800b330 comp rx=0 tx=0).stop 2023-03-03T10:43:44.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.897+0000 7ff3a7e43700 1 -- 172.21.15.7:0/3750060099 shutdown_connections 2023-03-03T10:43:44.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.897+0000 7ff3a7e43700 1 --2- 172.21.15.7:0/3750060099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a0155950 0x7ff3a0155dd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:44.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.897+0000 7ff3a7e43700 1 --2- 172.21.15.7:0/3750060099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0071c30 0x7ff3a00c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:44.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.897+0000 7ff3a7e43700 1 -- 172.21.15.7:0/3750060099 >> 172.21.15.7:0/3750060099 conn(0x7ff3a00736f0 msgr2=0x7ff3a0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:43:44.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.897+0000 7ff3a7e43700 1 -- 172.21.15.7:0/3750060099 shutdown_connections 2023-03-03T10:43:44.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.897+0000 7ff3a7e43700 1 -- 172.21.15.7:0/3750060099 wait complete. 2023-03-03T10:43:44.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.898+0000 7ff3a7e43700 1 Processor -- start 2023-03-03T10:43:44.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.898+0000 7ff3a7e43700 1 -- start start 2023-03-03T10:43:44.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a7e43700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a0071c30 0x7ff3a00ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:44.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a7e43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a00aac00 0x7ff3a01e8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:44.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a7e43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0155950 0x7ff3a0154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:44.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a7e43700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a01514b0 con 0x7ff3a00aac00 2023-03-03T10:43:44.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a7e43700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a01e8650 con 0x7ff3a0071c30 2023-03-03T10:43:44.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a7e43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a00bf2a0 con 0x7ff3a0155950 2023-03-03T10:43:44.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a53de700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a00aac00 0x7ff3a01e8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:44.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a53de700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a00aac00 0x7ff3a01e8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33226/0 (socket says 172.21.15.7:33226) 2023-03-03T10:43:44.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a53de700 1 -- 172.21.15.7:0/190733574 learned_addr learned my addr 172.21.15.7:0/190733574 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:43:44.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a5bdf700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a0071c30 0x7ff3a00ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:44.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a63e0700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0155950 0x7ff3a0154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:44.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a53de700 1 -- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0155950 msgr2=0x7ff3a0154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:44.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a53de700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0155950 0x7ff3a0154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:44.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a53de700 1 -- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a0071c30 msgr2=0x7ff3a00ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:44.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a53de700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a0071c30 0x7ff3a00ae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:44.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.899+0000 7ff3a53de700 1 -- 172.21.15.7:0/190733574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff398009a70 con 0x7ff3a00aac00 2023-03-03T10:43:44.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.900+0000 7ff3a53de700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a00aac00 0x7ff3a01e8110 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff39c00e910 tx=0x7ff39c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:44.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.900+0000 7ff3967fc700 1 -- 172.21.15.7:0/190733574 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c0096a0 con 0x7ff3a00aac00 2023-03-03T10:43:44.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.900+0000 7ff3967fc700 1 -- 172.21.15.7:0/190733574 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39c013680 con 0x7ff3a00aac00 2023-03-03T10:43:44.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.900+0000 7ff3a7e43700 1 -- 172.21.15.7:0/190733574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a00c4b80 con 0x7ff3a00aac00 2023-03-03T10:43:44.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.900+0000 7ff3967fc700 1 -- 172.21.15.7:0/190733574 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c013a30 con 0x7ff3a00aac00 2023-03-03T10:43:44.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.900+0000 7ff3a7e43700 1 -- 172.21.15.7:0/190733574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a0152360 con 0x7ff3a00aac00 2023-03-03T10:43:44.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.901+0000 7ff3a7e43700 1 -- 172.21.15.7:0/190733574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a0152620 con 0x7ff3a00aac00 2023-03-03T10:43:44.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.901+0000 7ff3967fc700 1 -- 172.21.15.7:0/190733574 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff39c023070 con 0x7ff3a00aac00 2023-03-03T10:43:44.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.902+0000 7ff3967fc700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3840682f0 0x7ff38406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:43:44.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.902+0000 7ff3967fc700 1 -- 172.21.15.7:0/190733574 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff39c086320 con 0x7ff3a00aac00 2023-03-03T10:43:44.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.902+0000 7ff3a5bdf700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3840682f0 0x7ff38406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:43:44.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.902+0000 7ff3a5bdf700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3840682f0 0x7ff38406a7a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff390007fd0 tx=0x7ff390009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:43:44.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:44.906+0000 7ff3967fc700 1 -- 172.21.15.7:0/190733574 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff39c0525d0 con 0x7ff3a00aac00 2023-03-03T10:43:45.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.145+0000 7ff3a7e43700 1 -- 172.21.15.7:0/190733574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3a0074bb0 con 0x7ff3a00aac00 2023-03-03T10:43:45.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.146+0000 7ff3967fc700 1 -- 172.21.15.7:0/190733574 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff39c055bf0 con 0x7ff3a00aac00 2023-03-03T10:43:45.149 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:43:45.149 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:43:45.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 -- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3840682f0 msgr2=0x7ff38406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:45.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3840682f0 0x7ff38406a7a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff390007fd0 tx=0x7ff390009040 comp rx=0 tx=0).stop 2023-03-03T10:43:45.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 -- 172.21.15.7:0/190733574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a00aac00 msgr2=0x7ff3a01e8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:43:45.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a00aac00 0x7ff3a01e8110 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7ff39c00e910 tx=0x7ff39c00ec20 comp rx=0 tx=0).stop 2023-03-03T10:43:45.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 -- 172.21.15.7:0/190733574 shutdown_connections 2023-03-03T10:43:45.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a0155950 0x7ff3a0154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:45.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a0071c30 0x7ff3a00ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:45.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3840682f0 0x7ff38406a7a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:45.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 --2- 172.21.15.7:0/190733574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a00aac00 0x7ff3a01e8110 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:43:45.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 -- 172.21.15.7:0/190733574 >> 172.21.15.7:0/190733574 conn(0x7ff3a00736f0 msgr2=0x7ff3a01dc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:43:45.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 -- 172.21.15.7:0/190733574 shutdown_connections 2023-03-03T10:43:45.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:43:45.148+0000 7ff3a7e43700 1 -- 172.21.15.7:0/190733574 wait complete. 2023-03-03T10:43:45.153 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:43:45.163 INFO:tasks.mds_thrash.fs.[a]:waiting for 63 secs before thrashing 2023-03-03T10:44:03.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:03.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:33.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:33.555 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:44:48.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:44:48.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.301+0000 7fc90c9e1700 1 Processor -- start 2023-03-03T10:44:48.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.301+0000 7fc90c9e1700 1 -- start start 2023-03-03T10:44:48.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90c9e1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9080c8020 0x7fc9080c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:44:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90c9e1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9080c7b90 con 0x7fc90806d4b0 2023-03-03T10:44:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90c9e1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc908154640 con 0x7fc9080c1da0 2023-03-03T10:44:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90c9e1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9080c7330 con 0x7fc9080c8020 2023-03-03T10:44:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc905d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9080c8020 0x7fc9080c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:44:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc90806d4b0 0x7fc90806d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55512/0 (socket says 172.21.15.7:55512) 2023-03-03T10:44:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90659c700 1 -- 172.21.15.7:0/1520375178 learned_addr learned my addr 172.21.15.7:0/1520375178 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:44:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90559a700 1 -- 172.21.15.7:0/1520375178 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21135727 0 0) 0x7fc9080c7b90 con 0x7fc90806d4b0 2023-03-03T10:44:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90559a700 1 -- 172.21.15.7:0/1520375178 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f40035b0 con 0x7fc90806d4b0 2023-03-03T10:44:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc905d9b700 1 -- 172.21.15.7:0/1520375178 >> v1:172.21.15.143:6789/0 conn(0x7fc9080c1da0 legacy=0x7fc908139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:44:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc905d9b700 1 -- 172.21.15.7:0/1520375178 >> v1:172.21.15.7:6789/0 conn(0x7fc90806d4b0 legacy=0x7fc90806d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:44:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc905d9b700 1 -- 172.21.15.7:0/1520375178 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9080c7540 con 0x7fc9080c8020 2023-03-03T10:44:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc905d9b700 1 --2- 172.21.15.7:0/1520375178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 0x7fc9080c07d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0009840 tx=0x7fc8f0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b544c9b0991a65a5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:44:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90559a700 1 -- 172.21.15.7:0/1520375178 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0011830 con 0x7fc9080c8020 2023-03-03T10:44:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90559a700 1 -- 172.21.15.7:0/1520375178 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0011990 con 0x7fc9080c8020 2023-03-03T10:44:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.302+0000 7fc90559a700 1 -- 172.21.15.7:0/1520375178 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0011d20 con 0x7fc9080c8020 2023-03-03T10:44:48.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.303+0000 7fc90c9e1700 1 -- 172.21.15.7:0/1520375178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 msgr2=0x7fc9080c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:44:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.303+0000 7fc90c9e1700 1 --2- 172.21.15.7:0/1520375178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 0x7fc9080c07d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0009840 tx=0x7fc8f0009b50 comp rx=0 tx=0).stop 2023-03-03T10:44:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.303+0000 7fc90c9e1700 1 -- 172.21.15.7:0/1520375178 shutdown_connections 2023-03-03T10:44:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.303+0000 7fc90c9e1700 1 --2- 172.21.15.7:0/1520375178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 0x7fc9080c07d0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:44:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.303+0000 7fc90c9e1700 1 -- 172.21.15.7:0/1520375178 >> 172.21.15.7:0/1520375178 conn(0x7fc90806c110 msgr2=0x7fc90806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:44:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.303+0000 7fc90c9e1700 1 -- 172.21.15.7:0/1520375178 shutdown_connections 2023-03-03T10:44:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.303+0000 7fc90c9e1700 1 -- 172.21.15.7:0/1520375178 wait complete. 2023-03-03T10:44:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.304+0000 7fc90c9e1700 1 Processor -- start 2023-03-03T10:44:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.304+0000 7fc90c9e1700 1 -- start start 2023-03-03T10:44:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc90c9e1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc90806d4b0 0x7fc908137750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:44:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc90c9e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9080c1da0 0x7fc908137c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:44:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc90c9e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 0x7fc90813aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:44:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc90c9e1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9081ea1a0 con 0x7fc9080c8020 2023-03-03T10:44:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc90c9e1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9080ad3a0 con 0x7fc90806d4b0 2023-03-03T10:44:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc90c9e1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90813b6f0 con 0x7fc9080c1da0 2023-03-03T10:44:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc906d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 0x7fc90813aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:44:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc906d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 0x7fc90813aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33310/0 (socket says 172.21.15.7:33310) 2023-03-03T10:44:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc906d9d700 1 -- 172.21.15.7:0/3691117119 learned_addr learned my addr 172.21.15.7:0/3691117119 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:44:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc90659c700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc90806d4b0 0x7fc908137750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:44:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.305+0000 7fc905d9b700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9080c1da0 0x7fc908137c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:44:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc906d9d700 1 -- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9080c1da0 msgr2=0x7fc908137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:44:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc906d9d700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9080c1da0 0x7fc908137c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:44:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc906d9d700 1 -- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc90806d4b0 msgr2=0x7fc908137750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:44:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc906d9d700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc90806d4b0 0x7fc908137750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:44:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc906d9d700 1 -- 172.21.15.7:0/3691117119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f00094a0 con 0x7fc9080c8020 2023-03-03T10:44:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc906d9d700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 0x7fc90813aa90 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc90000cbf0 tx=0x7fc90000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:44:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc8feffd700 1 -- 172.21.15.7:0/3691117119 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc900003c00 con 0x7fc9080c8020 2023-03-03T10:44:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc8feffd700 1 -- 172.21.15.7:0/3691117119 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc900003d60 con 0x7fc9080c8020 2023-03-03T10:44:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc8feffd700 1 -- 172.21.15.7:0/3691117119 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90000aa80 con 0x7fc9080c8020 2023-03-03T10:44:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc90c9e1700 1 -- 172.21.15.7:0/3691117119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90806e210 con 0x7fc9080c8020 2023-03-03T10:44:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.306+0000 7fc90c9e1700 1 -- 172.21.15.7:0/3691117119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9080c7820 con 0x7fc9080c8020 2023-03-03T10:44:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.307+0000 7fc90c9e1700 1 -- 172.21.15.7:0/3691117119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90813a490 con 0x7fc9080c8020 2023-03-03T10:44:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.308+0000 7fc8feffd700 1 -- 172.21.15.7:0/3691117119 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc900010050 con 0x7fc9080c8020 2023-03-03T10:44:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.308+0000 7fc8feffd700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8e8068320 0x7fc8e806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:44:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.308+0000 7fc8feffd700 1 -- 172.21.15.7:0/3691117119 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc900085f10 con 0x7fc9080c8020 2023-03-03T10:44:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.308+0000 7fc90659c700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8e8068320 0x7fc8e806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:44:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.309+0000 7fc90659c700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8e8068320 0x7fc8e806a7d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8007fd0 tx=0x7fc8f8007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:44:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.312+0000 7fc8feffd700 1 -- 172.21.15.7:0/3691117119 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc9000522b0 con 0x7fc9080c8020 2023-03-03T10:44:48.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.548+0000 7fc90c9e1700 1 -- 172.21.15.7:0/3691117119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc90806fa00 con 0x7fc9080c8020 2023-03-03T10:44:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.549+0000 7fc8feffd700 1 -- 172.21.15.7:0/3691117119 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc9000558d0 con 0x7fc9080c8020 2023-03-03T10:44:48.553 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:44:48.554 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:44:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 -- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8e8068320 msgr2=0x7fc8e806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:44:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8e8068320 0x7fc8e806a7d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8007fd0 tx=0x7fc8f8007dc0 comp rx=0 tx=0).stop 2023-03-03T10:44:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 -- 172.21.15.7:0/3691117119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 msgr2=0x7fc90813aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:44:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 0x7fc90813aa90 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fc90000cbf0 tx=0x7fc90000cf00 comp rx=0 tx=0).stop 2023-03-03T10:44:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 -- 172.21.15.7:0/3691117119 shutdown_connections 2023-03-03T10:44:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8e8068320 0x7fc8e806a7d0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:44:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9080c8020 0x7fc90813aa90 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:44:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc90806d4b0 0x7fc908137750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:44:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 --2- 172.21.15.7:0/3691117119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9080c1da0 0x7fc908137c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:44:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 -- 172.21.15.7:0/3691117119 >> 172.21.15.7:0/3691117119 conn(0x7fc90806c110 msgr2=0x7fc908072110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:44:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 -- 172.21.15.7:0/3691117119 shutdown_connections 2023-03-03T10:44:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:44:48.551+0000 7fc90c9e1700 1 -- 172.21.15.7:0/3691117119 wait complete. 2023-03-03T10:44:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:44:48.567 INFO:tasks.mds_thrash.fs.[a]:waiting for 70 secs before thrashing 2023-03-03T10:45:03.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:03.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:14.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:45:14.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.458+0000 7f8a3d9d2700 1 Processor -- start 2023-03-03T10:45:14.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.458+0000 7f8a3d9d2700 1 -- start start 2023-03-03T10:45:14.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a3d9d2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 0x7f8a3814e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:14.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a3d9d2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a380c7e00 0x7f8a3814e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:14.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a3d9d2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a38157bb0 con 0x7f8a3806b6f0 2023-03-03T10:45:14.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a3d9d2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a3806ab00 con 0x7f8a380c7e00 2023-03-03T10:45:14.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a3d9d2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a381545c0 con 0x7f8a380c7a30 2023-03-03T10:45:14.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a367fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a380c7e00 0x7f8a3814e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:14.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a377fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8a3806b6f0 0x7f8a38154a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55542/0 (socket says 172.21.15.7:55542) 2023-03-03T10:45:14.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a377fe700 1 -- 172.21.15.7:0/3863950004 learned_addr learned my addr 172.21.15.7:0/3863950004 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:45:14.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a36ffd700 1 --2- 172.21.15.7:0/3863950004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 0x7f8a3814e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:14.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a367fc700 1 -- 172.21.15.7:0/3863950004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 msgr2=0x7f8a3814e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:45:14.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a367fc700 1 --2- 172.21.15.7:0/3863950004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 0x7f8a3814e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:14.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a367fc700 1 -- 172.21.15.7:0/3863950004 >> v1:172.21.15.7:6789/0 conn(0x7f8a3806b6f0 legacy=0x7f8a38154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:14.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a35ffb700 1 -- 172.21.15.7:0/3863950004 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984899803 0 0) 0x7f8a38157bb0 con 0x7f8a3806b6f0 2023-03-03T10:45:14.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a367fc700 1 -- 172.21.15.7:0/3863950004 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3806a2f0 con 0x7f8a380c7e00 2023-03-03T10:45:14.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a36ffd700 1 --2- 172.21.15.7:0/3863950004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 0x7f8a3814e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:45:14.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.459+0000 7f8a367fc700 1 --2- 172.21.15.7:0/3863950004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c7e00 0x7f8a3814e6a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009840 tx=0x7f8a28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9779bf3b184c04 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:14.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a35ffb700 1 -- 172.21.15.7:0/3863950004 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28011830 con 0x7f8a380c7e00 2023-03-03T10:45:14.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a35ffb700 1 -- 172.21.15.7:0/3863950004 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a28011990 con 0x7f8a380c7e00 2023-03-03T10:45:14.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a35ffb700 1 -- 172.21.15.7:0/3863950004 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28011d20 con 0x7f8a380c7e00 2023-03-03T10:45:14.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/3863950004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c7e00 msgr2=0x7f8a3814e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:14.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a3d9d2700 1 --2- 172.21.15.7:0/3863950004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c7e00 0x7f8a3814e6a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009840 tx=0x7f8a28009b50 comp rx=0 tx=0).stop 2023-03-03T10:45:14.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/3863950004 shutdown_connections 2023-03-03T10:45:14.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a3d9d2700 1 --2- 172.21.15.7:0/3863950004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 0x7f8a3814e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a3d9d2700 1 --2- 172.21.15.7:0/3863950004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c7e00 0x7f8a3814e6a0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/3863950004 >> 172.21.15.7:0/3863950004 conn(0x7f8a38073d20 msgr2=0x7f8a380716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:45:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/3863950004 shutdown_connections 2023-03-03T10:45:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.460+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/3863950004 wait complete. 2023-03-03T10:45:14.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.461+0000 7f8a3d9d2700 1 Processor -- start 2023-03-03T10:45:14.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.462+0000 7f8a3d9d2700 1 -- start start 2023-03-03T10:45:14.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.462+0000 7f8a3d9d2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 0x7f8a3814caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.462+0000 7f8a3d9d2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a380c7e00 0x7f8a3814d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.462+0000 7f8a3d9d2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c49d0 0x7f8a381e9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.462+0000 7f8a3d9d2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a380c4540 con 0x7f8a380c49d0 2023-03-03T10:45:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.462+0000 7f8a3d9d2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a380c1cd0 con 0x7f8a380c7e00 2023-03-03T10:45:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.462+0000 7f8a3d9d2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3814beb0 con 0x7f8a380c7a30 2023-03-03T10:45:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.462+0000 7f8a377fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c49d0 0x7f8a381e9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.462+0000 7f8a377fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c49d0 0x7f8a381e9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33336/0 (socket says 172.21.15.7:33336) 2023-03-03T10:45:14.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.462+0000 7f8a377fe700 1 -- 172.21.15.7:0/1397273253 learned_addr learned my addr 172.21.15.7:0/1397273253 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:45:14.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a36ffd700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 0x7f8a3814caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:14.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a367fc700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a380c7e00 0x7f8a3814d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:14.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a377fe700 1 -- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 msgr2=0x7f8a3814caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:14.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a377fe700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 0x7f8a3814caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:14.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a377fe700 1 -- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a380c7e00 msgr2=0x7f8a3814d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:14.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a377fe700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a380c7e00 0x7f8a3814d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:14.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a377fe700 1 -- 172.21.15.7:0/1397273253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a280094a0 con 0x7f8a380c49d0 2023-03-03T10:45:14.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a377fe700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c49d0 0x7f8a381e9a10 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000cca0 tx=0x7f8a3000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:14.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a1ffff700 1 -- 172.21.15.7:0/1397273253 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3000cea0 con 0x7f8a380c49d0 2023-03-03T10:45:14.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a1ffff700 1 -- 172.21.15.7:0/1397273253 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3000ade0 con 0x7f8a380c49d0 2023-03-03T10:45:14.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/1397273253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3806a540 con 0x7f8a380c49d0 2023-03-03T10:45:14.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a1ffff700 1 -- 172.21.15.7:0/1397273253 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a30007310 con 0x7f8a380c49d0 2023-03-03T10:45:14.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.463+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/1397273253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a380bfbb0 con 0x7f8a380c49d0 2023-03-03T10:45:14.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.464+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/1397273253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3806a820 con 0x7f8a380c49d0 2023-03-03T10:45:14.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.465+0000 7f8a1ffff700 1 -- 172.21.15.7:0/1397273253 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a3001a030 con 0x7f8a380c49d0 2023-03-03T10:45:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.465+0000 7f8a1ffff700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a240682f0 0x7f8a2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.465+0000 7f8a1ffff700 1 -- 172.21.15.7:0/1397273253 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8a30085f30 con 0x7f8a380c49d0 2023-03-03T10:45:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.465+0000 7f8a36ffd700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a240682f0 0x7f8a2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.466+0000 7f8a36ffd700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a240682f0 0x7f8a2406a7a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8a380c9d50 tx=0x7f8a2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.470+0000 7f8a1ffff700 1 -- 172.21.15.7:0/1397273253 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8a300522d0 con 0x7f8a380c49d0 2023-03-03T10:45:14.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.703+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/1397273253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a38074bb0 con 0x7f8a380c49d0 2023-03-03T10:45:14.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.704+0000 7f8a1ffff700 1 -- 172.21.15.7:0/1397273253 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8a30018090 con 0x7f8a380c49d0 2023-03-03T10:45:14.706 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:45:14.706 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:45:14.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a240682f0 msgr2=0x7f8a2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:14.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a240682f0 0x7f8a2406a7a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8a380c9d50 tx=0x7f8a2c008040 comp rx=0 tx=0).stop 2023-03-03T10:45:14.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/1397273253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c49d0 msgr2=0x7f8a381e9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c49d0 0x7f8a381e9a10 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000cca0 tx=0x7f8a3000a340 comp rx=0 tx=0).stop 2023-03-03T10:45:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/1397273253 shutdown_connections 2023-03-03T10:45:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a240682f0 0x7f8a2406a7a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a380c49d0 0x7f8a381e9a10 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a380c7a30 0x7f8a3814caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 --2- 172.21.15.7:0/1397273253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a380c7e00 0x7f8a3814d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:14.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/1397273253 >> 172.21.15.7:0/1397273253 conn(0x7f8a3806b6f0 msgr2=0x7f8a380c5010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:45:14.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/1397273253 shutdown_connections 2023-03-03T10:45:14.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:14.706+0000 7f8a3d9d2700 1 -- 172.21.15.7:0/1397273253 wait complete. 2023-03-03T10:45:14.710 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:45:14.720 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before thrashing 2023-03-03T10:45:16.721 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:45:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.857+0000 7f1f8e0be700 1 Processor -- start 2023-03-03T10:45:16.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.857+0000 7f1f8e0be700 1 -- start start 2023-03-03T10:45:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.858+0000 7f1f8e0be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1f8814fda0 0x7f1f88156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.858+0000 7f1f8e0be700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f880c2f30 con 0x7f1f880abe20 2023-03-03T10:45:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.858+0000 7f1f8e0be700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f880c31d0 con 0x7f1f8814f9d0 2023-03-03T10:45:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.858+0000 7f1f8e0be700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f881524a0 con 0x7f1f8814fda0 2023-03-03T10:45:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.858+0000 7f1f87fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1f880abe20 0x7f1f881529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55556/0 (socket says 172.21.15.7:55556) 2023-03-03T10:45:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.858+0000 7f1f87fff700 1 -- 172.21.15.7:0/1684483770 learned_addr learned my addr 172.21.15.7:0/1684483770 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:45:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f86ffd700 1 --2- 172.21.15.7:0/1684483770 >> v2:172.21.15.143:3300/0 conn(0x7f1f8814fda0 0x7f1f88156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167622138 0 0) 0x7f1f880c2f30 con 0x7f1f880abe20 2023-03-03T10:45:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f740035b0 con 0x7f1f880abe20 2023-03-03T10:45:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2007664726 0 0) 0x7f1f740035b0 con 0x7f1f880abe20 2023-03-03T10:45:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 >> v2:172.21.15.143:3300/0 conn(0x7f1f8814fda0 msgr2=0x7f1f88156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:45:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 --2- 172.21.15.7:0/1684483770 >> v2:172.21.15.143:3300/0 conn(0x7f1f8814fda0 0x7f1f88156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 >> v1:172.21.15.143:6789/0 conn(0x7f1f8814f9d0 legacy=0x7f1f88071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f880c34b0 con 0x7f1f880abe20 2023-03-03T10:45:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f1f800019f0 con 0x7f1f880abe20 2023-03-03T10:45:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 >> v1:172.21.15.7:6789/0 conn(0x7f1f880abe20 legacy=0x7f1f881529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f74003a70 0x7f1f74003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f74004850 0x7f1f74004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f74005320 0x7f1f7400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881524a0 con 0x7f1f74003a70 2023-03-03T10:45:16.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7400c230 con 0x7f1f74005320 2023-03-03T10:45:16.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7400c3f0 con 0x7f1f74004850 2023-03-03T10:45:16.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f877fe700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f74003a70 0x7f1f74003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:16.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f87fff700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f74005320 0x7f1f7400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:16.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f86ffd700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f74004850 0x7f1f74004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:16.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f877fe700 1 -- 172.21.15.7:0/1684483770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f74004850 msgr2=0x7f1f74004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:45:16.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f877fe700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f74004850 0x7f1f74004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f877fe700 1 -- 172.21.15.7:0/1684483770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f74005320 msgr2=0x7f1f7400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:45:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f877fe700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f74005320 0x7f1f7400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f877fe700 1 -- 172.21.15.7:0/1684483770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f740035b0 con 0x7f1f74003a70 2023-03-03T10:45:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f87fff700 1 -- 172.21.15.7:0/1684483770 reap_dead start 2023-03-03T10:45:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.859+0000 7f1f86ffd700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f74004850 0x7f1f74004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:45:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f877fe700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f74003a70 0x7f1f74003e80 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1f700017c0 tx=0x7f1f74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74270bf28e684afc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f700052d0 con 0x7f1f74003a70 2023-03-03T10:45:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f70005450 con 0x7f1f74003a70 2023-03-03T10:45:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f867fc700 1 -- 172.21.15.7:0/1684483770 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f700038c0 con 0x7f1f74003a70 2023-03-03T10:45:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1684483770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f74003a70 msgr2=0x7f1f74003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f8e0be700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f74003a70 0x7f1f74003e80 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1f700017c0 tx=0x7f1f74005880 comp rx=0 tx=0).stop 2023-03-03T10:45:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1684483770 shutdown_connections 2023-03-03T10:45:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f8e0be700 1 --2- 172.21.15.7:0/1684483770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f74003a70 0x7f1f74003e80 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1684483770 >> 172.21.15.7:0/1684483770 conn(0x7f1f880c8900 msgr2=0x7f1f880c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:45:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1684483770 shutdown_connections 2023-03-03T10:45:16.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.860+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1684483770 wait complete. 2023-03-03T10:45:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.862+0000 7f1f8e0be700 1 Processor -- start 2023-03-03T10:45:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.862+0000 7f1f8e0be700 1 -- start start 2023-03-03T10:45:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.862+0000 7f1f8e0be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f74003a70 0x7f1f881e9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.862+0000 7f1f8e0be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f88155060 0x7f1f880c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.862+0000 7f1f8e0be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8806e9a0 0x7f1f8814ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.862+0000 7f1f8e0be700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f881e9c60 con 0x7f1f88155060 2023-03-03T10:45:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.862+0000 7f1f8e0be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8806f790 con 0x7f1f8806e9a0 2023-03-03T10:45:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.862+0000 7f1f8e0be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f880c2ae0 con 0x7f1f74003a70 2023-03-03T10:45:16.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f86ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f88155060 0x7f1f880c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:16.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f86ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f88155060 0x7f1f880c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33354/0 (socket says 172.21.15.7:33354) 2023-03-03T10:45:16.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f86ffd700 1 -- 172.21.15.7:0/1065878348 learned_addr learned my addr 172.21.15.7:0/1065878348 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:45:16.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f877fe700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f74003a70 0x7f1f881e9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:16.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f87fff700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8806e9a0 0x7f1f8814ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:16.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f86ffd700 1 -- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f74003a70 msgr2=0x7f1f881e9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:16.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f86ffd700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f74003a70 0x7f1f881e9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:16.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f86ffd700 1 -- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8806e9a0 msgr2=0x7f1f8814ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:16.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f86ffd700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8806e9a0 0x7f1f8814ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:16.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f86ffd700 1 -- 172.21.15.7:0/1065878348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f880ac350 con 0x7f1f88155060 2023-03-03T10:45:16.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f87fff700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8806e9a0 0x7f1f8814ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:45:16.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f86ffd700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f88155060 0x7f1f880c1a50 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f1f78011580 tx=0x7f1f78011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:16.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f7ffff700 1 -- 172.21.15.7:0/1065878348 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f78004220 con 0x7f1f88155060 2023-03-03T10:45:16.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f7ffff700 1 -- 172.21.15.7:0/1065878348 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f78004380 con 0x7f1f88155060 2023-03-03T10:45:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1065878348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f880ab630 con 0x7f1f88155060 2023-03-03T10:45:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f7ffff700 1 -- 172.21.15.7:0/1065878348 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f780036a0 con 0x7f1f88155060 2023-03-03T10:45:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.863+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1065878348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f880c3e10 con 0x7f1f88155060 2023-03-03T10:45:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.864+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1065878348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f88074bb0 con 0x7f1f88155060 2023-03-03T10:45:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.865+0000 7f1f7ffff700 1 -- 172.21.15.7:0/1065878348 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f78024070 con 0x7f1f88155060 2023-03-03T10:45:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.865+0000 7f1f7ffff700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f68068320 0x7f1f6806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.865+0000 7f1f7ffff700 1 -- 172.21.15.7:0/1065878348 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1f78086180 con 0x7f1f88155060 2023-03-03T10:45:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.865+0000 7f1f877fe700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f68068320 0x7f1f6806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:16.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.866+0000 7f1f877fe700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f68068320 0x7f1f6806a7d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1f70000c00 tx=0x7f1f70009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:16.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:16.870+0000 7f1f7ffff700 1 -- 172.21.15.7:0/1065878348 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f78052520 con 0x7f1f88155060 2023-03-03T10:45:17.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.108+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1065878348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f88151fe0 con 0x7f1f88155060 2023-03-03T10:45:17.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.109+0000 7f1f7ffff700 1 -- 172.21.15.7:0/1065878348 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1f78055b40 con 0x7f1f88155060 2023-03-03T10:45:17.111 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:45:17.112 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:45:17.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f68068320 msgr2=0x7f1f6806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:17.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f68068320 0x7f1f6806a7d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1f70000c00 tx=0x7f1f70009030 comp rx=0 tx=0).stop 2023-03-03T10:45:17.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1065878348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f88155060 msgr2=0x7f1f880c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:17.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f88155060 0x7f1f880c1a50 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f1f78011580 tx=0x7f1f78011890 comp rx=0 tx=0).stop 2023-03-03T10:45:17.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1065878348 shutdown_connections 2023-03-03T10:45:17.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8806e9a0 0x7f1f8814ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:17.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f74003a70 0x7f1f881e9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:17.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f68068320 0x7f1f6806a7d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:17.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 --2- 172.21.15.7:0/1065878348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f88155060 0x7f1f880c1a50 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:17.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1065878348 >> 172.21.15.7:0/1065878348 conn(0x7f1f880c8900 msgr2=0x7f1f880c7280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:45:17.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1065878348 shutdown_connections 2023-03-03T10:45:17.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:17.111+0000 7f1f8e0be700 1 -- 172.21.15.7:0/1065878348 wait complete. 2023-03-03T10:45:17.115 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:45:17.125 INFO:tasks.mds_thrash.fs.[b]:waiting for 83 secs before thrashing 2023-03-03T10:45:33.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:33.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:45:58.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:45:58.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.704+0000 7f8c02148700 1 Processor -- start 2023-03-03T10:45:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.704+0000 7f8c02148700 1 -- start start 2023-03-03T10:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8c02148700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8bfc14f9d0 0x7f8bfc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8c02148700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14fda0 0x7f8bfc071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8c02148700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0c2f30 con 0x7f8bfc0abe20 2023-03-03T10:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8c02148700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bfc0c31d0 con 0x7f8bfc14f9d0 2023-03-03T10:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8c02148700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc1524a0 con 0x7f8bfc14fda0 2023-03-03T10:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfbfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8bfc0abe20 0x7f8bfc1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55612/0 (socket says 172.21.15.7:55612) 2023-03-03T10:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfbfff700 1 -- 172.21.15.7:0/1180869706 learned_addr learned my addr 172.21.15.7:0/1180869706 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfb7fe700 1 --2- 172.21.15.7:0/1180869706 >> v2:172.21.15.143:3300/0 conn(0x7f8bfc14f9d0 0x7f8bfc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfaffd700 1 --2- 172.21.15.7:0/1180869706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14fda0 0x7f8bfc071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfa7fc700 1 -- 172.21.15.7:0/1180869706 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736333149 0 0) 0x7f8bfc0c2f30 con 0x7f8bfc0abe20 2023-03-03T10:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfa7fc700 1 -- 172.21.15.7:0/1180869706 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be40035b0 con 0x7f8bfc0abe20 2023-03-03T10:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfb7fe700 1 -- 172.21.15.7:0/1180869706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14fda0 msgr2=0x7f8bfc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfb7fe700 1 --2- 172.21.15.7:0/1180869706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14fda0 0x7f8bfc071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfb7fe700 1 -- 172.21.15.7:0/1180869706 >> v1:172.21.15.7:6789/0 conn(0x7f8bfc0abe20 legacy=0x7f8bfc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfb7fe700 1 -- 172.21.15.7:0/1180869706 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc0c34b0 con 0x7f8bfc14f9d0 2023-03-03T10:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.705+0000 7f8bfaffd700 1 --2- 172.21.15.7:0/1180869706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14fda0 0x7f8bfc071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8bfb7fe700 1 --2- 172.21.15.7:0/1180869706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc14f9d0 0x7f8bfc156430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009840 tx=0x7f8bec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=439bdc4e4ccae47e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8bfa7fc700 1 -- 172.21.15.7:0/1180869706 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec011830 con 0x7f8bfc14f9d0 2023-03-03T10:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8bfa7fc700 1 -- 172.21.15.7:0/1180869706 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec011990 con 0x7f8bfc14f9d0 2023-03-03T10:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8bfa7fc700 1 -- 172.21.15.7:0/1180869706 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec011d00 con 0x7f8bfc14f9d0 2023-03-03T10:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8c02148700 1 -- 172.21.15.7:0/1180869706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc14f9d0 msgr2=0x7f8bfc156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8c02148700 1 --2- 172.21.15.7:0/1180869706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc14f9d0 0x7f8bfc156430 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009840 tx=0x7f8bec009b50 comp rx=0 tx=0).stop 2023-03-03T10:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8c02148700 1 -- 172.21.15.7:0/1180869706 shutdown_connections 2023-03-03T10:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8c02148700 1 --2- 172.21.15.7:0/1180869706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc14f9d0 0x7f8bfc156430 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8c02148700 1 --2- 172.21.15.7:0/1180869706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14fda0 0x7f8bfc071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8c02148700 1 -- 172.21.15.7:0/1180869706 >> 172.21.15.7:0/1180869706 conn(0x7f8bfc0c8900 msgr2=0x7f8bfc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:45:58.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.706+0000 7f8c02148700 1 -- 172.21.15.7:0/1180869706 shutdown_connections 2023-03-03T10:45:58.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.707+0000 7f8c02148700 1 -- 172.21.15.7:0/1180869706 wait complete. 2023-03-03T10:45:58.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.708+0000 7f8c02148700 1 Processor -- start 2023-03-03T10:45:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.708+0000 7f8c02148700 1 -- start start 2023-03-03T10:45:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8c02148700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0abe20 0x7f8bfc14dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8c02148700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14f9d0 0x7f8bfc06dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8c02148700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc14fda0 0x7f8bfc14b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8c02148700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0c2840 con 0x7f8bfc0abe20 2023-03-03T10:45:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8c02148700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0c2560 con 0x7f8bfc14fda0 2023-03-03T10:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8c02148700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc14d1e0 con 0x7f8bfc14f9d0 2023-03-03T10:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0abe20 0x7f8bfc14dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0abe20 0x7f8bfc14dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33402/0 (socket says 172.21.15.7:33402) 2023-03-03T10:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfb7fe700 1 -- 172.21.15.7:0/201884189 learned_addr learned my addr 172.21.15.7:0/201884189 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfaffd700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14f9d0 0x7f8bfc06dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfbfff700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc14fda0 0x7f8bfc14b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfb7fe700 1 -- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14f9d0 msgr2=0x7f8bfc06dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfb7fe700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14f9d0 0x7f8bfc06dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfb7fe700 1 -- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc14fda0 msgr2=0x7f8bfc14b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfb7fe700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc14fda0 0x7f8bfc14b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfb7fe700 1 -- 172.21.15.7:0/201884189 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf00094a0 con 0x7f8bfc0abe20 2023-03-03T10:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8bfb7fe700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0abe20 0x7f8bfc14dcc0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0090b0 tx=0x7f8bec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8be3fff700 1 -- 172.21.15.7:0/201884189 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec0045d0 con 0x7f8bfc0abe20 2023-03-03T10:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8be3fff700 1 -- 172.21.15.7:0/201884189 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bec01f070 con 0x7f8bfc0abe20 2023-03-03T10:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8c02148700 1 -- 172.21.15.7:0/201884189 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bec0094a0 con 0x7f8bfc0abe20 2023-03-03T10:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8be3fff700 1 -- 172.21.15.7:0/201884189 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bec0038c0 con 0x7f8bfc0abe20 2023-03-03T10:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.709+0000 7f8c02148700 1 -- 172.21.15.7:0/201884189 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc0ac350 con 0x7f8bfc0abe20 2023-03-03T10:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.710+0000 7f8c02148700 1 -- 172.21.15.7:0/201884189 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bfc0ab630 con 0x7f8bfc0abe20 2023-03-03T10:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.711+0000 7f8be3fff700 1 -- 172.21.15.7:0/201884189 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8bec00f050 con 0x7f8bfc0abe20 2023-03-03T10:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.711+0000 7f8be3fff700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be80682f0 0x7f8be806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.711+0000 7f8be3fff700 1 -- 172.21.15.7:0/201884189 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8bec021070 con 0x7f8bfc0abe20 2023-03-03T10:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.712+0000 7f8bfaffd700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be80682f0 0x7f8be806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:45:58.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.712+0000 7f8bfaffd700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be80682f0 0x7f8be806a7a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc14e200 tx=0x7f8bf0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:45:58.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.716+0000 7f8be3fff700 1 -- 172.21.15.7:0/201884189 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8bec053d70 con 0x7f8bfc0abe20 2023-03-03T10:45:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.958+0000 7f8c02148700 1 -- 172.21.15.7:0/201884189 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bfc0c3e10 con 0x7f8bfc0abe20 2023-03-03T10:45:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.959+0000 7f8be3fff700 1 -- 172.21.15.7:0/201884189 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8bec057390 con 0x7f8bfc0abe20 2023-03-03T10:45:58.963 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:45:58.963 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:45:58.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 -- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be80682f0 msgr2=0x7f8be806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:58.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be80682f0 0x7f8be806a7a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc14e200 tx=0x7f8bf0008040 comp rx=0 tx=0).stop 2023-03-03T10:45:58.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 -- 172.21.15.7:0/201884189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0abe20 msgr2=0x7f8bfc14dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:45:58.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0abe20 0x7f8bfc14dcc0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f8bec0090b0 tx=0x7f8bec004750 comp rx=0 tx=0).stop 2023-03-03T10:45:58.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 -- 172.21.15.7:0/201884189 shutdown_connections 2023-03-03T10:45:58.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bfc14fda0 0x7f8bfc14b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8be80682f0 0x7f8be806a7a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8bfc0abe20 0x7f8bfc14dcc0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 --2- 172.21.15.7:0/201884189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8bfc14f9d0 0x7f8bfc06dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:45:58.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 -- 172.21.15.7:0/201884189 >> 172.21.15.7:0/201884189 conn(0x7f8bfc0c8900 msgr2=0x7f8bfc1528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:45:58.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 -- 172.21.15.7:0/201884189 shutdown_connections 2023-03-03T10:45:58.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:45:58.961+0000 7f8c02148700 1 -- 172.21.15.7:0/201884189 wait complete. 2023-03-03T10:45:58.966 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:45:58.976 INFO:tasks.mds_thrash.fs.[a]:waiting for 110 secs before thrashing 2023-03-03T10:46:03.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:03.762 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:33.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:33.834 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:46:40.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:46:40.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.258+0000 7fcdfc3d5700 1 Processor -- start 2023-03-03T10:46:40.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.258+0000 7fcdfc3d5700 1 -- start start 2023-03-03T10:46:40.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.259+0000 7fcdfc3d5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcdf406e020 0x7fcdf4071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:40.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.259+0000 7fcdfc3d5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcdf4072520 0x7fcdf4151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:40.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.259+0000 7fcdfc3d5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf4154910 con 0x7fcdf4155390 2023-03-03T10:46:40.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.259+0000 7fcdfc3d5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf4070b90 con 0x7fcdf4072520 2023-03-03T10:46:40.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.259+0000 7fcdfc3d5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf4151320 con 0x7fcdf406e020 2023-03-03T10:46:40.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.259+0000 7fcdfa972700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcdf4072520 0x7fcdf4151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:40.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.259+0000 7fcdfa972700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcdf4072520 0x7fcdf4151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33454/0 (socket says 172.21.15.7:33454) 2023-03-03T10:46:40.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.259+0000 7fcdfa972700 1 -- 172.21.15.7:0/2743640470 learned_addr learned my addr 172.21.15.7:0/2743640470 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:46:40.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.259+0000 7fcdf9970700 1 --2- 172.21.15.7:0/2743640470 >> v2:172.21.15.143:3300/0 conn(0x7fcdf406e020 0x7fcdf4071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:40.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.260+0000 7fcdfa972700 1 -- 172.21.15.7:0/2743640470 >> v2:172.21.15.143:3300/0 conn(0x7fcdf406e020 msgr2=0x7fcdf4071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:46:40.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.260+0000 7fcdfa972700 1 --2- 172.21.15.7:0/2743640470 >> v2:172.21.15.143:3300/0 conn(0x7fcdf406e020 0x7fcdf4071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.260+0000 7fcdfa972700 1 -- 172.21.15.7:0/2743640470 >> v1:172.21.15.143:6789/0 conn(0x7fcdf4155390 legacy=0x7fcdf40c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.260+0000 7fcdfa972700 1 -- 172.21.15.7:0/2743640470 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf40c4540 con 0x7fcdf4072520 2023-03-03T10:46:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.260+0000 7fcdfa972700 1 --2- 172.21.15.7:0/2743640470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdf4072520 0x7fcdf4151770 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40ac9e0 tx=0x7fcdec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9691975a868b75a6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:46:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.260+0000 7fcdf916f700 1 -- 172.21.15.7:0/2743640470 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec00e070 con 0x7fcdf4072520 2023-03-03T10:46:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.260+0000 7fcdf916f700 1 -- 172.21.15.7:0/2743640470 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec00b910 con 0x7fcdf4072520 2023-03-03T10:46:40.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.260+0000 7fcdf916f700 1 -- 172.21.15.7:0/2743640470 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec00bcc0 con 0x7fcdf4072520 2023-03-03T10:46:40.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.260+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/2743640470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdf4072520 msgr2=0x7fcdf4151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:40.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.260+0000 7fcdfc3d5700 1 --2- 172.21.15.7:0/2743640470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdf4072520 0x7fcdf4151770 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40ac9e0 tx=0x7fcdec00b330 comp rx=0 tx=0).stop 2023-03-03T10:46:40.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.261+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/2743640470 shutdown_connections 2023-03-03T10:46:40.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.261+0000 7fcdfc3d5700 1 --2- 172.21.15.7:0/2743640470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdf4072520 0x7fcdf4151770 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:40.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.261+0000 7fcdfc3d5700 1 --2- 172.21.15.7:0/2743640470 >> v2:172.21.15.143:3300/0 conn(0x7fcdf406e020 0x7fcdf4071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:40.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.261+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/2743640470 >> 172.21.15.7:0/2743640470 conn(0x7fcdf406c110 msgr2=0x7fcdf406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:46:40.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.261+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/2743640470 shutdown_connections 2023-03-03T10:46:40.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.261+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/2743640470 wait complete. 2023-03-03T10:46:40.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.262+0000 7fcdfc3d5700 1 Processor -- start 2023-03-03T10:46:40.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.262+0000 7fcdfc3d5700 1 -- start start 2023-03-03T10:46:40.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.263+0000 7fcdfc3d5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdf406e020 0x7fcdf40c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:40.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.263+0000 7fcdfc3d5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdf4072520 0x7fcdf40c03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:40.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.263+0000 7fcdfc3d5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdf4155390 0x7fcdf40c0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:40.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.263+0000 7fcdfc3d5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40bcf40 con 0x7fcdf406e020 2023-03-03T10:46:40.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.263+0000 7fcdfc3d5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40c1540 con 0x7fcdf4072520 2023-03-03T10:46:40.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.263+0000 7fcdfc3d5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40bc460 con 0x7fcdf4155390 2023-03-03T10:46:40.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdf9970700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdf4072520 0x7fcdf40c03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:40.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdf9970700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdf4072520 0x7fcdf40c03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:57462/0 (socket says 172.21.15.7:57462) 2023-03-03T10:46:40.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdf9970700 1 -- 172.21.15.7:0/3462538258 learned_addr learned my addr 172.21.15.7:0/3462538258 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:46:40.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdfa171700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdf406e020 0x7fcdf40c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:40.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdfa972700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdf4155390 0x7fcdf40c0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:40.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdf9970700 1 -- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdf4155390 msgr2=0x7fcdf40c0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:40.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdf9970700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdf4155390 0x7fcdf40c0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:40.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdf9970700 1 -- 172.21.15.7:0/3462538258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdf406e020 msgr2=0x7fcdf40c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdf9970700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdf406e020 0x7fcdf40c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdf9970700 1 -- 172.21.15.7:0/3462538258 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec009b30 con 0x7fcdf4072520 2023-03-03T10:46:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.264+0000 7fcdfa972700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdf4155390 0x7fcdf40c0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:46:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.265+0000 7fcdf9970700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdf4072520 0x7fcdf40c03f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000ebe0 tx=0x7fcdf000eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:46:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.265+0000 7fcdeaffd700 1 -- 172.21.15.7:0/3462538258 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0009710 con 0x7fcdf4072520 2023-03-03T10:46:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.265+0000 7fcdeaffd700 1 -- 172.21.15.7:0/3462538258 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf0017c60 con 0x7fcdf4072520 2023-03-03T10:46:40.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.265+0000 7fcdeaffd700 1 -- 172.21.15.7:0/3462538258 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf000c640 con 0x7fcdf4072520 2023-03-03T10:46:40.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.265+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/3462538258 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf4070540 con 0x7fcdf4072520 2023-03-03T10:46:40.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.265+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/3462538258 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf40702a0 con 0x7fcdf4072520 2023-03-03T10:46:40.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.266+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/3462538258 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf4070820 con 0x7fcdf4072520 2023-03-03T10:46:40.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.267+0000 7fcdeaffd700 1 -- 172.21.15.7:0/3462538258 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcdf0026030 con 0x7fcdf4072520 2023-03-03T10:46:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.267+0000 7fcdeaffd700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdd80682f0 0x7fcdd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:46:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.267+0000 7fcdeaffd700 1 -- 172.21.15.7:0/3462538258 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcdf008e7b0 con 0x7fcdf4072520 2023-03-03T10:46:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.267+0000 7fcdfa171700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdd80682f0 0x7fcdd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:46:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.268+0000 7fcdfa171700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdd80682f0 0x7fcdd806a7a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcdf406d290 tx=0x7fcde4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:46:40.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.271+0000 7fcdeaffd700 1 -- 172.21.15.7:0/3462538258 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcdf005ab70 con 0x7fcdf4072520 2023-03-03T10:46:40.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.508+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/3462538258 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdf4074bb0 con 0x7fcdf4072520 2023-03-03T10:46:40.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.508+0000 7fcdeaffd700 1 -- 172.21.15.7:0/3462538258 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcdf005e190 con 0x7fcdf4072520 2023-03-03T10:46:40.511 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:46:40.511 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:46:40.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdd80682f0 msgr2=0x7fcdd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:40.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdd80682f0 0x7fcdd806a7a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcdf406d290 tx=0x7fcde4009040 comp rx=0 tx=0).stop 2023-03-03T10:46:40.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdf4072520 msgr2=0x7fcdf40c03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:46:40.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdf4072520 0x7fcdf40c03f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000ebe0 tx=0x7fcdf000eef0 comp rx=0 tx=0).stop 2023-03-03T10:46:40.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/3462538258 shutdown_connections 2023-03-03T10:46:40.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdf4155390 0x7fcdf40c0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:40.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdd80682f0 0x7fcdd806a7a0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:40.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdf406e020 0x7fcdf40c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:40.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 --2- 172.21.15.7:0/3462538258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdf4072520 0x7fcdf40c03f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:46:40.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/3462538258 >> 172.21.15.7:0/3462538258 conn(0x7fcdf406c110 msgr2=0x7fcdf414f190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:46:40.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/3462538258 shutdown_connections 2023-03-03T10:46:40.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:46:40.511+0000 7fcdfc3d5700 1 -- 172.21.15.7:0/3462538258 wait complete. 2023-03-03T10:46:40.515 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:46:40.525 INFO:tasks.mds_thrash.fs.[b]:waiting for 93 secs before thrashing 2023-03-03T10:47:03.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:03.949 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:34.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:34.068 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:47:48.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:47:49.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.113+0000 7fc13c25b700 1 Processor -- start 2023-03-03T10:47:49.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.113+0000 7fc13c25b700 1 -- start start 2023-03-03T10:47:49.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc13c25b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc134151e80 0x7fc1340c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:47:49.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc13c25b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc134155410 0x7fc1340bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:47:49.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc13c25b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1340c08d0 con 0x7fc1340c6fe0 2023-03-03T10:47:49.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc13c25b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc134152250 con 0x7fc134155410 2023-03-03T10:47:49.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc13c25b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1340c0fb0 con 0x7fc134151e80 2023-03-03T10:47:49.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc1397f6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc134155410 0x7fc1340bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:47:49.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc1397f6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc134155410 0x7fc1340bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33504/0 (socket says 172.21.15.7:33504) 2023-03-03T10:47:49.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc1397f6700 1 -- 172.21.15.7:0/3973156057 learned_addr learned my addr 172.21.15.7:0/3973156057 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:47:49.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc1397f6700 1 -- 172.21.15.7:0/3973156057 >> v2:172.21.15.143:3300/0 conn(0x7fc134151e80 msgr2=0x7fc1340c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:47:49.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc139ff7700 1 --2- 172.21.15.7:0/3973156057 >> v2:172.21.15.143:3300/0 conn(0x7fc134151e80 0x7fc1340c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:47:49.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc1397f6700 1 --2- 172.21.15.7:0/3973156057 >> v2:172.21.15.143:3300/0 conn(0x7fc134151e80 0x7fc1340c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:47:49.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc1397f6700 1 -- 172.21.15.7:0/3973156057 >> v1:172.21.15.143:6789/0 conn(0x7fc1340c6fe0 legacy=0x7fc1340bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:47:49.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc1397f6700 1 -- 172.21.15.7:0/3973156057 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc134154930 con 0x7fc134155410 2023-03-03T10:47:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.114+0000 7fc139ff7700 1 --2- 172.21.15.7:0/3973156057 >> v2:172.21.15.143:3300/0 conn(0x7fc134151e80 0x7fc1340c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:47:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.115+0000 7fc1397f6700 1 --2- 172.21.15.7:0/3973156057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc134155410 0x7fc1340bf8e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc1340cb250 tx=0x7fc130009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56470661c8cd5d6d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:47:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.115+0000 7fc138ff5700 1 -- 172.21.15.7:0/3973156057 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130004230 con 0x7fc134155410 2023-03-03T10:47:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.115+0000 7fc138ff5700 1 -- 172.21.15.7:0/3973156057 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc130004390 con 0x7fc134155410 2023-03-03T10:47:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.115+0000 7fc138ff5700 1 -- 172.21.15.7:0/3973156057 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1300036a0 con 0x7fc134155410 2023-03-03T10:47:49.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.115+0000 7fc13c25b700 1 -- 172.21.15.7:0/3973156057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc134155410 msgr2=0x7fc1340bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:47:49.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.115+0000 7fc13c25b700 1 --2- 172.21.15.7:0/3973156057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc134155410 0x7fc1340bf8e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc1340cb250 tx=0x7fc130009bd0 comp rx=0 tx=0).stop 2023-03-03T10:47:49.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.115+0000 7fc13c25b700 1 -- 172.21.15.7:0/3973156057 shutdown_connections 2023-03-03T10:47:49.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.115+0000 7fc13c25b700 1 --2- 172.21.15.7:0/3973156057 >> v2:172.21.15.143:3300/0 conn(0x7fc134151e80 0x7fc1340c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:47:49.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.115+0000 7fc13c25b700 1 --2- 172.21.15.7:0/3973156057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc134155410 0x7fc1340bf8e0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:47:49.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.115+0000 7fc13c25b700 1 -- 172.21.15.7:0/3973156057 >> 172.21.15.7:0/3973156057 conn(0x7fc134071630 msgr2=0x7fc13406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:47:49.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.116+0000 7fc13c25b700 1 -- 172.21.15.7:0/3973156057 shutdown_connections 2023-03-03T10:47:49.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.116+0000 7fc13c25b700 1 -- 172.21.15.7:0/3973156057 wait complete. 2023-03-03T10:47:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.117+0000 7fc13c25b700 1 Processor -- start 2023-03-03T10:47:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.117+0000 7fc13c25b700 1 -- start start 2023-03-03T10:47:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.117+0000 7fc13c25b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1340c6fe0 0x7fc1341ee930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:47:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.117+0000 7fc13c25b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc134151e80 0x7fc1340c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:47:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.117+0000 7fc13c25b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc134155410 0x7fc1341eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:47:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.117+0000 7fc13c25b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340ca3c0 con 0x7fc1340c6fe0 2023-03-03T10:47:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.117+0000 7fc13c25b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1341eee70 con 0x7fc134151e80 2023-03-03T10:47:49.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.117+0000 7fc13c25b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340bff30 con 0x7fc134155410 2023-03-03T10:47:49.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc139ff7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1340c6fe0 0x7fc1341ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:47:49.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc139ff7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1340c6fe0 0x7fc1341ee930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33508/0 (socket says 172.21.15.7:33508) 2023-03-03T10:47:49.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc139ff7700 1 -- 172.21.15.7:0/1102451408 learned_addr learned my addr 172.21.15.7:0/1102451408 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:47:49.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc1397f6700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc134151e80 0x7fc1340c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:47:49.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc13a7f8700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc134155410 0x7fc1341eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:47:49.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc139ff7700 1 -- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc134155410 msgr2=0x7fc1341eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:47:49.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc139ff7700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc134155410 0x7fc1341eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:47:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc139ff7700 1 -- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc134151e80 msgr2=0x7fc1340c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:47:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc139ff7700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc134151e80 0x7fc1340c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:47:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc139ff7700 1 -- 172.21.15.7:0/1102451408 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc130009550 con 0x7fc1340c6fe0 2023-03-03T10:47:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc1397f6700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc134151e80 0x7fc1340c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:47:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc13a7f8700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc134155410 0x7fc1341eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:47:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc139ff7700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1340c6fe0 0x7fc1341ee930 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c750 tx=0x7fc12400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:47:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc12a7fc700 1 -- 172.21.15.7:0/1102451408 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124011800 con 0x7fc1340c6fe0 2023-03-03T10:47:49.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc12a7fc700 1 -- 172.21.15.7:0/1102451408 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc124011960 con 0x7fc1340c6fe0 2023-03-03T10:47:49.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc13c25b700 1 -- 172.21.15.7:0/1102451408 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc134154ef0 con 0x7fc1340c6fe0 2023-03-03T10:47:49.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc12a7fc700 1 -- 172.21.15.7:0/1102451408 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124011cd0 con 0x7fc1340c6fe0 2023-03-03T10:47:49.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.118+0000 7fc13c25b700 1 -- 172.21.15.7:0/1102451408 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc134154c10 con 0x7fc1340c6fe0 2023-03-03T10:47:49.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.119+0000 7fc13c25b700 1 -- 172.21.15.7:0/1102451408 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1340c5220 con 0x7fc1340c6fe0 2023-03-03T10:47:49.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.120+0000 7fc12a7fc700 1 -- 172.21.15.7:0/1102451408 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc124020070 con 0x7fc1340c6fe0 2023-03-03T10:47:49.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.120+0000 7fc12a7fc700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1180682f0 0x7fc11806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:47:49.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.120+0000 7fc12a7fc700 1 -- 172.21.15.7:0/1102451408 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc124085fb0 con 0x7fc1340c6fe0 2023-03-03T10:47:49.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.120+0000 7fc1397f6700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1180682f0 0x7fc11806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:47:49.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.121+0000 7fc1397f6700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1180682f0 0x7fc11806a7a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc1340cb250 tx=0x7fc130008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:47:49.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.125+0000 7fc12a7fc700 1 -- 172.21.15.7:0/1102451408 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc124052350 con 0x7fc1340c6fe0 2023-03-03T10:47:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.362+0000 7fc13c25b700 1 -- 172.21.15.7:0/1102451408 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1340c0450 con 0x7fc1340c6fe0 2023-03-03T10:47:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.363+0000 7fc12a7fc700 1 -- 172.21.15.7:0/1102451408 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc1340c0450 con 0x7fc1340c6fe0 2023-03-03T10:47:49.365 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:47:49.366 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:47:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 -- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1180682f0 msgr2=0x7fc11806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:47:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1180682f0 0x7fc11806a7a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc1340cb250 tx=0x7fc130008030 comp rx=0 tx=0).stop 2023-03-03T10:47:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 -- 172.21.15.7:0/1102451408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1340c6fe0 msgr2=0x7fc1341ee930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:47:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1340c6fe0 0x7fc1341ee930 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c750 tx=0x7fc12400cb10 comp rx=0 tx=0).stop 2023-03-03T10:47:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 -- 172.21.15.7:0/1102451408 shutdown_connections 2023-03-03T10:47:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc134155410 0x7fc1341eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:47:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1180682f0 0x7fc11806a7a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:47:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1340c6fe0 0x7fc1341ee930 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:47:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 --2- 172.21.15.7:0/1102451408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc134151e80 0x7fc1340c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:47:49.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 -- 172.21.15.7:0/1102451408 >> 172.21.15.7:0/1102451408 conn(0x7fc134071630 msgr2=0x7fc1340c3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:47:49.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 -- 172.21.15.7:0/1102451408 shutdown_connections 2023-03-03T10:47:49.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:47:49.365+0000 7fc13c25b700 1 -- 172.21.15.7:0/1102451408 wait complete. 2023-03-03T10:47:49.369 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:47:49.379 INFO:tasks.mds_thrash.fs.[a]:waiting for 41 secs before thrashing 2023-03-03T10:48:04.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:04.149 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:13.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:48:13.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.662+0000 7f7a91bbf700 1 Processor -- start 2023-03-03T10:48:13.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.662+0000 7f7a91bbf700 1 -- start start 2023-03-03T10:48:13.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a91bbf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7a8c0a6370 0x7f7a8c06a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:13.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a91bbf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7a8c0c5cb0 0x7f7a8c15cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:13.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a91bbf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8c1558f0 con 0x7f7a8c06a2b0 2023-03-03T10:48:13.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a91bbf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8c06bab0 con 0x7f7a8c0a6370 2023-03-03T10:48:13.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a91bbf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8c0a6180 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7a8c0a6370 0x7f7a8c06a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:13.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7a8c06a2b0 0x7f7a8c155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55754/0 (socket says 172.21.15.7:55754) 2023-03-03T10:48:13.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8b7fe700 1 -- 172.21.15.7:0/4015543557 learned_addr learned my addr 172.21.15.7:0/4015543557 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:48:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8bfff700 1 --2- 172.21.15.7:0/4015543557 >> v2:172.21.15.143:3300/0 conn(0x7f7a8c0c5cb0 0x7f7a8c15cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8affd700 1 -- 172.21.15.7:0/4015543557 >> v2:172.21.15.143:3300/0 conn(0x7f7a8c0c5cb0 msgr2=0x7f7a8c15cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:48:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8affd700 1 --2- 172.21.15.7:0/4015543557 >> v2:172.21.15.143:3300/0 conn(0x7f7a8c0c5cb0 0x7f7a8c15cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8affd700 1 -- 172.21.15.7:0/4015543557 >> v1:172.21.15.7:6789/0 conn(0x7f7a8c06a2b0 legacy=0x7f7a8c155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8affd700 1 -- 172.21.15.7:0/4015543557 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8c0a5ea0 con 0x7f7a8c0a6370 2023-03-03T10:48:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8affd700 1 --2- 172.21.15.7:0/4015543557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0a6370 0x7f7a8c06a680 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0c7be0 tx=0x7f7a80009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a01961eb449596b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8a7fc700 1 -- 172.21.15.7:0/4015543557 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80011830 con 0x7f7a8c0a6370 2023-03-03T10:48:13.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8a7fc700 1 -- 172.21.15.7:0/4015543557 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a80011990 con 0x7f7a8c0a6370 2023-03-03T10:48:13.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.663+0000 7f7a8a7fc700 1 -- 172.21.15.7:0/4015543557 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80011d20 con 0x7f7a8c0a6370 2023-03-03T10:48:13.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.664+0000 7f7a91bbf700 1 -- 172.21.15.7:0/4015543557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0a6370 msgr2=0x7f7a8c06a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:13.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.664+0000 7f7a91bbf700 1 --2- 172.21.15.7:0/4015543557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0a6370 0x7f7a8c06a680 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0c7be0 tx=0x7f7a80009b20 comp rx=0 tx=0).stop 2023-03-03T10:48:13.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.664+0000 7f7a91bbf700 1 -- 172.21.15.7:0/4015543557 shutdown_connections 2023-03-03T10:48:13.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.664+0000 7f7a91bbf700 1 --2- 172.21.15.7:0/4015543557 >> v2:172.21.15.143:3300/0 conn(0x7f7a8c0c5cb0 0x7f7a8c15cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:13.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.664+0000 7f7a91bbf700 1 --2- 172.21.15.7:0/4015543557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0a6370 0x7f7a8c06a680 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:13.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.664+0000 7f7a91bbf700 1 -- 172.21.15.7:0/4015543557 >> 172.21.15.7:0/4015543557 conn(0x7f7a8c0a9430 msgr2=0x7f7a8c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:48:13.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.664+0000 7f7a91bbf700 1 -- 172.21.15.7:0/4015543557 shutdown_connections 2023-03-03T10:48:13.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.664+0000 7f7a91bbf700 1 -- 172.21.15.7:0/4015543557 wait complete. 2023-03-03T10:48:13.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.665+0000 7f7a91bbf700 1 Processor -- start 2023-03-03T10:48:13.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.665+0000 7f7a91bbf700 1 -- start start 2023-03-03T10:48:13.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a91bbf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a8c06a2b0 0x7f7a8c150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:13.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a91bbf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a8c0a6370 0x7f7a8c1536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:13.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a91bbf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0c5cb0 0x7f7a8c06fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:13.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a91bbf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c151180 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a91bbf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c1521a0 con 0x7f7a8c06a2b0 2023-03-03T10:48:13.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a91bbf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c1523f0 con 0x7f7a8c0a6370 2023-03-03T10:48:13.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a8bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0c5cb0 0x7f7a8c06fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:13.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a8bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0c5cb0 0x7f7a8c06fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33552/0 (socket says 172.21.15.7:33552) 2023-03-03T10:48:13.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a8bfff700 1 -- 172.21.15.7:0/3233232038 learned_addr learned my addr 172.21.15.7:0/3233232038 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:48:13.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a8b7fe700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a8c06a2b0 0x7f7a8c150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:13.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.666+0000 7f7a8affd700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a8c0a6370 0x7f7a8c1536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:13.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a8bfff700 1 -- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a8c0a6370 msgr2=0x7f7a8c1536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:13.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a8bfff700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a8c0a6370 0x7f7a8c1536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:13.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a8bfff700 1 -- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a8c06a2b0 msgr2=0x7f7a8c150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:13.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a8bfff700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a8c06a2b0 0x7f7a8c150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:13.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a8bfff700 1 -- 172.21.15.7:0/3233232038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a800094a0 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a8affd700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a8c0a6370 0x7f7a8c1536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:48:13.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a8bfff700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0c5cb0 0x7f7a8c06fb20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7a84007e60 tx=0x7f7a8400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:13.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a73fff700 1 -- 172.21.15.7:0/3233232038 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8400ede0 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a73fff700 1 -- 172.21.15.7:0/3233232038 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a84014420 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a91bbf700 1 -- 172.21.15.7:0/3233232038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8c153160 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a73fff700 1 -- 172.21.15.7:0/3233232038 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a840147d0 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.667+0000 7f7a91bbf700 1 -- 172.21.15.7:0/3233232038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8c152e80 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.668+0000 7f7a91bbf700 1 -- 172.21.15.7:0/3233232038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8c0c1500 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.668+0000 7f7a73fff700 1 -- 172.21.15.7:0/3233232038 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7a84012050 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.669+0000 7f7a73fff700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a780682f0 0x7f7a7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.669+0000 7f7a73fff700 1 -- 172.21.15.7:0/3233232038 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7a840867d0 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.669+0000 7f7a8b7fe700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a780682f0 0x7f7a7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:13.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.669+0000 7f7a8b7fe700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a780682f0 0x7f7a7806a7a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c005910 tx=0x7f7a7c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:13.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.673+0000 7f7a73fff700 1 -- 172.21.15.7:0/3233232038 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7a84052af0 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.909+0000 7f7a91bbf700 1 -- 172.21.15.7:0/3233232038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a8c074bb0 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.909+0000 7f7a73fff700 1 -- 172.21.15.7:0/3233232038 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7a84056110 con 0x7f7a8c0c5cb0 2023-03-03T10:48:13.912 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:48:13.912 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:48:13.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.911+0000 7f7a91bbf700 1 -- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a780682f0 msgr2=0x7f7a7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:13.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.911+0000 7f7a91bbf700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a780682f0 0x7f7a7806a7a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c005910 tx=0x7f7a7c009040 comp rx=0 tx=0).stop 2023-03-03T10:48:13.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.911+0000 7f7a91bbf700 1 -- 172.21.15.7:0/3233232038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0c5cb0 msgr2=0x7f7a8c06fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:13.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.911+0000 7f7a91bbf700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0c5cb0 0x7f7a8c06fb20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7a84007e60 tx=0x7f7a8400e3c0 comp rx=0 tx=0).stop 2023-03-03T10:48:13.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.911+0000 7f7a91bbf700 1 -- 172.21.15.7:0/3233232038 shutdown_connections 2023-03-03T10:48:13.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.911+0000 7f7a91bbf700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a780682f0 0x7f7a7806a7a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:13.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.911+0000 7f7a91bbf700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0c5cb0 0x7f7a8c06fb20 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:13.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.911+0000 7f7a91bbf700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a8c06a2b0 0x7f7a8c150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:13.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.911+0000 7f7a91bbf700 1 --2- 172.21.15.7:0/3233232038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a8c0a6370 0x7f7a8c1536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:13.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.911+0000 7f7a91bbf700 1 -- 172.21.15.7:0/3233232038 >> 172.21.15.7:0/3233232038 conn(0x7f7a8c0a9430 msgr2=0x7f7a8c0a6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:48:13.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.912+0000 7f7a91bbf700 1 -- 172.21.15.7:0/3233232038 shutdown_connections 2023-03-03T10:48:13.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:13.912+0000 7f7a91bbf700 1 -- 172.21.15.7:0/3233232038 wait complete. 2023-03-03T10:48:13.916 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:48:13.925 INFO:tasks.mds_thrash.fs.[b]:waiting for 36 secs before thrashing 2023-03-03T10:48:30.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:48:30.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.518+0000 7f8ecd5ce700 1 Processor -- start 2023-03-03T10:48:30.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.518+0000 7f8ecd5ce700 1 -- start start 2023-03-03T10:48:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.518+0000 7f8ecd5ce700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ec80caa30 0x7f8ec80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.518+0000 7f8ecd5ce700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec8154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ecd5ce700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec80cb280 con 0x7f8ec80cae00 2023-03-03T10:48:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ecd5ce700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec80c7310 con 0x7f8ec80caa30 2023-03-03T10:48:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ecd5ce700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80bfd10 con 0x7f8ec80c00e0 2023-03-03T10:48:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec6ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ec80caa30 0x7f8ec80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec67fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec8154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec77fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8ec80cae00 0x7f8ec80c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:59300/0 (socket says 172.21.15.7:59300) 2023-03-03T10:48:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec6ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ec80caa30 0x7f8ec80c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:57570/0 (socket says 172.21.15.7:57570) 2023-03-03T10:48:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec77fe700 1 -- 172.21.15.7:0/4185983740 learned_addr learned my addr 172.21.15.7:0/4185983740 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:48:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec67fc700 1 --2- 172.21.15.7:0/4185983740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec8154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:41636/0 (socket says 172.21.15.7:41636) 2023-03-03T10:48:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec5ffb700 1 -- 172.21.15.7:0/4185983740 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462653654 0 0) 0x7f8ec80cb280 con 0x7f8ec80cae00 2023-03-03T10:48:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec5ffb700 1 -- 172.21.15.7:0/4185983740 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb00035b0 con 0x7f8ec80cae00 2023-03-03T10:48:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec6ffd700 1 -- 172.21.15.7:0/4185983740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 msgr2=0x7f8ec8154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:48:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec6ffd700 1 --2- 172.21.15.7:0/4185983740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec8154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec6ffd700 1 -- 172.21.15.7:0/4185983740 >> v1:172.21.15.143:6789/0 conn(0x7f8ec80cae00 legacy=0x7f8ec80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec6ffd700 1 -- 172.21.15.7:0/4185983740 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec80bfa30 con 0x7f8ec80caa30 2023-03-03T10:48:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.519+0000 7f8ec67fc700 1 --2- 172.21.15.7:0/4185983740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec8154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T10:48:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ec6ffd700 1 --2- 172.21.15.7:0/4185983740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ec80caa30 0x7f8ec80c5c60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8009840 tx=0x7f8eb8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93dd01f855e71e8c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ec5ffb700 1 -- 172.21.15.7:0/4185983740 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8011830 con 0x7f8ec80caa30 2023-03-03T10:48:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ec5ffb700 1 -- 172.21.15.7:0/4185983740 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb8011990 con 0x7f8ec80caa30 2023-03-03T10:48:30.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ec5ffb700 1 -- 172.21.15.7:0/4185983740 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8011d20 con 0x7f8ec80caa30 2023-03-03T10:48:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/4185983740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ec80caa30 msgr2=0x7f8ec80c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ecd5ce700 1 --2- 172.21.15.7:0/4185983740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ec80caa30 0x7f8ec80c5c60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8009840 tx=0x7f8eb8009b50 comp rx=0 tx=0).stop 2023-03-03T10:48:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/4185983740 shutdown_connections 2023-03-03T10:48:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ecd5ce700 1 --2- 172.21.15.7:0/4185983740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ec80caa30 0x7f8ec80c5c60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ecd5ce700 1 --2- 172.21.15.7:0/4185983740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec8154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/4185983740 >> 172.21.15.7:0/4185983740 conn(0x7f8ec8066e40 msgr2=0x7f8ec8071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:48:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/4185983740 shutdown_connections 2023-03-03T10:48:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.520+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/4185983740 wait complete. 2023-03-03T10:48:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ecd5ce700 1 Processor -- start 2023-03-03T10:48:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ecd5ce700 1 -- start start 2023-03-03T10:48:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ecd5ce700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec80c2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ecd5ce700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ec80caa30 0x7f8ec814cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ecd5ce700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ec80cae00 0x7f8ec814d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ecd5ce700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec80ae8a0 con 0x7f8ec80caa30 2023-03-03T10:48:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ecd5ce700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec814d7b0 con 0x7f8ec80cae00 2023-03-03T10:48:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ecd5ce700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec81ea2f0 con 0x7f8ec80c00e0 2023-03-03T10:48:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ec67fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ec80caa30 0x7f8ec814cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ec67fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ec80caa30 0x7f8ec814cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33572/0 (socket says 172.21.15.7:33572) 2023-03-03T10:48:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.522+0000 7f8ec67fc700 1 -- 172.21.15.7:0/497766025 learned_addr learned my addr 172.21.15.7:0/497766025 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:48:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ec77fe700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ec80cae00 0x7f8ec814d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ec6ffd700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec80c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ec67fc700 1 -- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 msgr2=0x7f8ec80c2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ec67fc700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec80c2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ec67fc700 1 -- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ec80cae00 msgr2=0x7f8ec814d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ec67fc700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ec80cae00 0x7f8ec814d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ec67fc700 1 -- 172.21.15.7:0/497766025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc009430 con 0x7f8ec80caa30 2023-03-03T10:48:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ec6ffd700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec80c2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:48:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ec67fc700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ec80caa30 0x7f8ec814cd30 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00ebe0 tx=0x7f8ebc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8eaf7fe700 1 -- 172.21.15.7:0/497766025 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc005a20 con 0x7f8ec80caa30 2023-03-03T10:48:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8eaf7fe700 1 -- 172.21.15.7:0/497766025 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc005b80 con 0x7f8ec80caa30 2023-03-03T10:48:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/497766025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb80094a0 con 0x7f8ec80caa30 2023-03-03T10:48:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8eaf7fe700 1 -- 172.21.15.7:0/497766025 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc00c910 con 0x7f8ec80caa30 2023-03-03T10:48:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.523+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/497766025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec81547f0 con 0x7f8ec80caa30 2023-03-03T10:48:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.524+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/497766025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec8154510 con 0x7f8ec80caa30 2023-03-03T10:48:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.525+0000 7f8eaf7fe700 1 -- 172.21.15.7:0/497766025 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8ebc022070 con 0x7f8ec80caa30 2023-03-03T10:48:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.525+0000 7f8eaf7fe700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb40682f0 0x7f8eb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.525+0000 7f8eaf7fe700 1 -- 172.21.15.7:0/497766025 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8ebc0862c0 con 0x7f8ec80caa30 2023-03-03T10:48:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.525+0000 7f8ec6ffd700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb40682f0 0x7f8eb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:30.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.526+0000 7f8ec6ffd700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb40682f0 0x7f8eb406a7a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8009fd0 tx=0x7f8eb8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:30.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.529+0000 7f8eaf7fe700 1 -- 172.21.15.7:0/497766025 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8ebc052660 con 0x7f8ec80caa30 2023-03-03T10:48:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.769+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/497766025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ec8154ad0 con 0x7f8ec80caa30 2023-03-03T10:48:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.770+0000 7f8eaf7fe700 1 -- 172.21.15.7:0/497766025 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8ebc020090 con 0x7f8ec80caa30 2023-03-03T10:48:30.772 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:48:30.772 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:48:30.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb40682f0 msgr2=0x7f8eb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:30.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb40682f0 0x7f8eb406a7a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8009fd0 tx=0x7f8eb8008030 comp rx=0 tx=0).stop 2023-03-03T10:48:30.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/497766025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ec80caa30 msgr2=0x7f8ec814cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:30.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ec80caa30 0x7f8ec814cd30 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00ebe0 tx=0x7f8ebc00eef0 comp rx=0 tx=0).stop 2023-03-03T10:48:30.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/497766025 shutdown_connections 2023-03-03T10:48:30.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ec80cae00 0x7f8ec814d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:30.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ec80c00e0 0x7f8ec80c2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:30.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb40682f0 0x7f8eb406a7a0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:30.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 --2- 172.21.15.7:0/497766025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ec80caa30 0x7f8ec814cd30 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:30.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/497766025 >> 172.21.15.7:0/497766025 conn(0x7f8ec8066e40 msgr2=0x7f8ec80736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:48:30.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/497766025 shutdown_connections 2023-03-03T10:48:30.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:30.772+0000 7f8ecd5ce700 1 -- 172.21.15.7:0/497766025 wait complete. 2023-03-03T10:48:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:48:30.786 INFO:tasks.mds_thrash.fs.[a]:waiting for 83 secs before thrashing 2023-03-03T10:48:34.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:34.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:48:49.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:48:50.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.063+0000 7f76d403c700 1 Processor -- start 2023-03-03T10:48:50.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.064+0000 7f76d403c700 1 -- start start 2023-03-03T10:48:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.064+0000 7f76d403c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76cc0beab0 0x7f76cc155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:50.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.064+0000 7f76d403c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f76cc1562c0 0x7f76cc15cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.064+0000 7f76d403c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc1523b0 con 0x7f76cc152e30 2023-03-03T10:48:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.064+0000 7f76d403c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc072010 con 0x7f76cc0beab0 2023-03-03T10:48:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.064+0000 7f76d403c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0aaaf0 con 0x7f76cc1562c0 2023-03-03T10:48:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.064+0000 7f76d15d7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76cc0beab0 0x7f76cc155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d15d7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76cc0beab0 0x7f76cc155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33584/0 (socket says 172.21.15.7:33584) 2023-03-03T10:48:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d15d7700 1 -- 172.21.15.7:0/176128579 learned_addr learned my addr 172.21.15.7:0/176128579 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:48:50.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d25d9700 1 --2- 172.21.15.7:0/176128579 >> v2:172.21.15.143:3300/0 conn(0x7f76cc1562c0 0x7f76cc15cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d15d7700 1 -- 172.21.15.7:0/176128579 >> v2:172.21.15.143:3300/0 conn(0x7f76cc1562c0 msgr2=0x7f76cc15cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:48:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d15d7700 1 --2- 172.21.15.7:0/176128579 >> v2:172.21.15.143:3300/0 conn(0x7f76cc1562c0 0x7f76cc15cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d15d7700 1 -- 172.21.15.7:0/176128579 >> v1:172.21.15.143:6789/0 conn(0x7f76cc152e30 legacy=0x7f76cc0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:48:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d15d7700 1 -- 172.21.15.7:0/176128579 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc0aadd0 con 0x7f76cc0beab0 2023-03-03T10:48:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d25d9700 1 --2- 172.21.15.7:0/176128579 >> v2:172.21.15.143:3300/0 conn(0x7f76cc1562c0 0x7f76cc15cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:48:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d15d7700 1 --2- 172.21.15.7:0/176128579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc0beab0 0x7f76cc155d80 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0c3530 tx=0x7f76c8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03535de69f5d3e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d0dd6700 1 -- 172.21.15.7:0/176128579 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8004230 con 0x7f76cc0beab0 2023-03-03T10:48:50.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d0dd6700 1 -- 172.21.15.7:0/176128579 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c8004390 con 0x7f76cc0beab0 2023-03-03T10:48:50.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.065+0000 7f76d0dd6700 1 -- 172.21.15.7:0/176128579 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c80036a0 con 0x7f76cc0beab0 2023-03-03T10:48:50.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.066+0000 7f76d403c700 1 -- 172.21.15.7:0/176128579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc0beab0 msgr2=0x7f76cc155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.066+0000 7f76d403c700 1 --2- 172.21.15.7:0/176128579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc0beab0 0x7f76cc155d80 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0c3530 tx=0x7f76c8009bd0 comp rx=0 tx=0).stop 2023-03-03T10:48:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.066+0000 7f76d403c700 1 -- 172.21.15.7:0/176128579 shutdown_connections 2023-03-03T10:48:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.066+0000 7f76d403c700 1 --2- 172.21.15.7:0/176128579 >> v2:172.21.15.143:3300/0 conn(0x7f76cc1562c0 0x7f76cc15cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.066+0000 7f76d403c700 1 --2- 172.21.15.7:0/176128579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc0beab0 0x7f76cc155d80 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.066+0000 7f76d403c700 1 -- 172.21.15.7:0/176128579 >> 172.21.15.7:0/176128579 conn(0x7f76cc0736f0 msgr2=0x7f76cc073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:48:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.066+0000 7f76d403c700 1 -- 172.21.15.7:0/176128579 shutdown_connections 2023-03-03T10:48:50.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.066+0000 7f76d403c700 1 -- 172.21.15.7:0/176128579 wait complete. 2023-03-03T10:48:50.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.067+0000 7f76d403c700 1 Processor -- start 2023-03-03T10:48:50.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.067+0000 7f76d403c700 1 -- start start 2023-03-03T10:48:50.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.068+0000 7f76d403c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc0beab0 0x7f76cc1483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:50.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.068+0000 7f76d403c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc152e30 0x7f76cc154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:50.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.068+0000 7f76d403c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc1562c0 0x7f76cc1e7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:50.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.068+0000 7f76d403c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0adc10 con 0x7f76cc152e30 2023-03-03T10:48:50.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.068+0000 7f76d403c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc1ebf40 con 0x7f76cc0beab0 2023-03-03T10:48:50.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.068+0000 7f76d403c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc148910 con 0x7f76cc1562c0 2023-03-03T10:48:50.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.068+0000 7f76d15d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc152e30 0x7f76cc154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:50.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d15d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc152e30 0x7f76cc154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33590/0 (socket says 172.21.15.7:33590) 2023-03-03T10:48:50.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d15d7700 1 -- 172.21.15.7:0/1991668770 learned_addr learned my addr 172.21.15.7:0/1991668770 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:48:50.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d1dd8700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc0beab0 0x7f76cc1483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:50.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d25d9700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc1562c0 0x7f76cc1e7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:50.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d15d7700 1 -- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc1562c0 msgr2=0x7f76cc1e7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:50.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d15d7700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc1562c0 0x7f76cc1e7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:50.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d15d7700 1 -- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc0beab0 msgr2=0x7f76cc1483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:50.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d15d7700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc0beab0 0x7f76cc1483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:50.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d15d7700 1 -- 172.21.15.7:0/1991668770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c8009550 con 0x7f76cc152e30 2023-03-03T10:48:50.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d15d7700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc152e30 0x7f76cc154a60 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009fd0 tx=0x7f76c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:50.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76c27fc700 1 -- 172.21.15.7:0/1991668770 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8011630 con 0x7f76cc152e30 2023-03-03T10:48:50.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76c27fc700 1 -- 172.21.15.7:0/1991668770 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c8011790 con 0x7f76cc152e30 2023-03-03T10:48:50.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d403c700 1 -- 172.21.15.7:0/1991668770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc153200 con 0x7f76cc152e30 2023-03-03T10:48:50.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76c27fc700 1 -- 172.21.15.7:0/1991668770 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8011b40 con 0x7f76cc152e30 2023-03-03T10:48:50.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.069+0000 7f76d403c700 1 -- 172.21.15.7:0/1991668770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc071c70 con 0x7f76cc152e30 2023-03-03T10:48:50.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.070+0000 7f76d403c700 1 -- 172.21.15.7:0/1991668770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc153390 con 0x7f76cc152e30 2023-03-03T10:48:50.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.071+0000 7f76c27fc700 1 -- 172.21.15.7:0/1991668770 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f76c8017020 con 0x7f76cc152e30 2023-03-03T10:48:50.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.071+0000 7f76c27fc700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 0x7f76b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:48:50.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.071+0000 7f76c27fc700 1 -- 172.21.15.7:0/1991668770 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f76c8087760 con 0x7f76cc152e30 2023-03-03T10:48:50.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.071+0000 7f76d1dd8700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 0x7f76b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:48:50.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.072+0000 7f76d1dd8700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 0x7f76b006a7a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f76bc004f30 tx=0x7f76bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:48:50.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.076+0000 7f76c27fc700 1 -- 172.21.15.7:0/1991668770 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f76c8053a80 con 0x7f76cc152e30 2023-03-03T10:48:50.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.315+0000 7f76d403c700 1 -- 172.21.15.7:0/1991668770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76cc069570 con 0x7f76cc152e30 2023-03-03T10:48:50.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.316+0000 7f76c27fc700 1 -- 172.21.15.7:0/1991668770 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f76c80570a0 con 0x7f76cc152e30 2023-03-03T10:48:50.318 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:48:50.319 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:48:50.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 -- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 msgr2=0x7f76b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 0x7f76b006a7a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f76bc004f30 tx=0x7f76bc009040 comp rx=0 tx=0).stop 2023-03-03T10:48:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 -- 172.21.15.7:0/1991668770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc152e30 msgr2=0x7f76cc154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:48:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc152e30 0x7f76cc154a60 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009fd0 tx=0x7f76c8004750 comp rx=0 tx=0).stop 2023-03-03T10:48:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 -- 172.21.15.7:0/1991668770 shutdown_connections 2023-03-03T10:48:50.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76cc1562c0 0x7f76cc1e7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76cc0beab0 0x7f76cc1483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b00682f0 0x7f76b006a7a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 --2- 172.21.15.7:0/1991668770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76cc152e30 0x7f76cc154a60 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:48:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 -- 172.21.15.7:0/1991668770 >> 172.21.15.7:0/1991668770 conn(0x7f76cc0736f0 msgr2=0x7f76cc14f260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:48:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 -- 172.21.15.7:0/1991668770 shutdown_connections 2023-03-03T10:48:50.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:48:50.318+0000 7f76d403c700 1 -- 172.21.15.7:0/1991668770 wait complete. 2023-03-03T10:48:50.323 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:48:50.332 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T10:49:04.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:04.390 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:34.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:34.457 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:49:53.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:49:53.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.923+0000 7fe75c746700 1 Processor -- start 2023-03-03T10:49:53.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.923+0000 7fe75c746700 1 -- start start 2023-03-03T10:49:53.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75c746700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7540c2930 0x7fe754157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75c746700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe75406c480 0x7fe754157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75c746700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75406d430 con 0x7fe754151d40 2023-03-03T10:49:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75c746700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75406c2a0 con 0x7fe75406c480 2023-03-03T10:49:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75c746700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75406cb00 con 0x7fe7540c2930 2023-03-03T10:49:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75ace3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe75406c480 0x7fe754157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75ace3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe75406c480 0x7fe754157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33614/0 (socket says 172.21.15.7:33614) 2023-03-03T10:49:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75ace3700 1 -- 172.21.15.7:0/123420658 learned_addr learned my addr 172.21.15.7:0/123420658 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:49:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe759ce1700 1 --2- 172.21.15.7:0/123420658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7540c2930 0x7fe754157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75ace3700 1 -- 172.21.15.7:0/123420658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7540c2930 msgr2=0x7fe754157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:49:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75ace3700 1 --2- 172.21.15.7:0/123420658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7540c2930 0x7fe754157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75ace3700 1 -- 172.21.15.7:0/123420658 >> v1:172.21.15.143:6789/0 conn(0x7fe754151d40 legacy=0x7fe7540736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75ace3700 1 -- 172.21.15.7:0/123420658 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75406cde0 con 0x7fe75406c480 2023-03-03T10:49:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe759ce1700 1 --2- 172.21.15.7:0/123420658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7540c2930 0x7fe754157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:49:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.924+0000 7fe75ace3700 1 --2- 172.21.15.7:0/123420658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe75406c480 0x7fe754157fd0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009e50 tx=0x7fe74c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2dbf74983238086 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:49:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe7594e0700 1 -- 172.21.15.7:0/123420658 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c00e070 con 0x7fe75406c480 2023-03-03T10:49:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe7594e0700 1 -- 172.21.15.7:0/123420658 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c00b9c0 con 0x7fe75406c480 2023-03-03T10:49:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe7594e0700 1 -- 172.21.15.7:0/123420658 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c00bd70 con 0x7fe75406c480 2023-03-03T10:49:53.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe75c746700 1 -- 172.21.15.7:0/123420658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe75406c480 msgr2=0x7fe754157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe75c746700 1 --2- 172.21.15.7:0/123420658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe75406c480 0x7fe754157fd0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009e50 tx=0x7fe74c00b330 comp rx=0 tx=0).stop 2023-03-03T10:49:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe75c746700 1 -- 172.21.15.7:0/123420658 shutdown_connections 2023-03-03T10:49:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe75c746700 1 --2- 172.21.15.7:0/123420658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe75406c480 0x7fe754157fd0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe75c746700 1 --2- 172.21.15.7:0/123420658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7540c2930 0x7fe754157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe75c746700 1 -- 172.21.15.7:0/123420658 >> 172.21.15.7:0/123420658 conn(0x7fe7540a94d0 msgr2=0x7fe7540a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:49:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe75c746700 1 -- 172.21.15.7:0/123420658 shutdown_connections 2023-03-03T10:49:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.925+0000 7fe75c746700 1 -- 172.21.15.7:0/123420658 wait complete. 2023-03-03T10:49:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.926+0000 7fe75c746700 1 Processor -- start 2023-03-03T10:49:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.926+0000 7fe75c746700 1 -- start start 2023-03-03T10:49:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe75c746700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe75406c480 0x7fe7541ed740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe75c746700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7540c2930 0x7fe7541ea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe75c746700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe754151d40 0x7fe754150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe75c746700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541e97c0 con 0x7fe754151d40 2023-03-03T10:49:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe75c746700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541edc80 con 0x7fe7540c2930 2023-03-03T10:49:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe75c746700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7541ea6f0 con 0x7fe75406c480 2023-03-03T10:49:53.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe75ace3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe754151d40 0x7fe754150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:53.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe75ace3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe754151d40 0x7fe754150450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33622/0 (socket says 172.21.15.7:33622) 2023-03-03T10:49:53.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe75ace3700 1 -- 172.21.15.7:0/4159069531 learned_addr learned my addr 172.21.15.7:0/4159069531 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:49:53.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe75a4e2700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe75406c480 0x7fe7541ed740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:53.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.927+0000 7fe759ce1700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7540c2930 0x7fe7541ea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:53.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.928+0000 7fe75ace3700 1 -- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe75406c480 msgr2=0x7fe7541ed740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:53.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.928+0000 7fe75ace3700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe75406c480 0x7fe7541ed740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.928+0000 7fe75ace3700 1 -- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7540c2930 msgr2=0x7fe7541ea1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.928+0000 7fe75ace3700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7540c2930 0x7fe7541ea1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.928+0000 7fe75ace3700 1 -- 172.21.15.7:0/4159069531 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c009b30 con 0x7fe754151d40 2023-03-03T10:49:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.928+0000 7fe759ce1700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7540c2930 0x7fe7541ea1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:49:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.928+0000 7fe75ace3700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe754151d40 0x7fe754150450 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe75406bb70 tx=0x7fe74c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:49:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.929+0000 7fe74affd700 1 -- 172.21.15.7:0/4159069531 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c004250 con 0x7fe754151d40 2023-03-03T10:49:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.929+0000 7fe74affd700 1 -- 172.21.15.7:0/4159069531 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c0043b0 con 0x7fe754151d40 2023-03-03T10:49:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.929+0000 7fe75c746700 1 -- 172.21.15.7:0/4159069531 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe754073e00 con 0x7fe754151d40 2023-03-03T10:49:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.929+0000 7fe74affd700 1 -- 172.21.15.7:0/4159069531 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c004760 con 0x7fe754151d40 2023-03-03T10:49:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.929+0000 7fe75c746700 1 -- 172.21.15.7:0/4159069531 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75406d0c0 con 0x7fe754151d40 2023-03-03T10:49:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.930+0000 7fe75c746700 1 -- 172.21.15.7:0/4159069531 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75406bea0 con 0x7fe754151d40 2023-03-03T10:49:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.930+0000 7fe74affd700 1 -- 172.21.15.7:0/4159069531 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe74c02b070 con 0x7fe754151d40 2023-03-03T10:49:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.931+0000 7fe74affd700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7380682f0 0x7fe73806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.931+0000 7fe74affd700 1 -- 172.21.15.7:0/4159069531 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe74c087410 con 0x7fe754151d40 2023-03-03T10:49:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.931+0000 7fe75a4e2700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7380682f0 0x7fe73806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.932+0000 7fe75a4e2700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7380682f0 0x7fe73806a7a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe750007fd0 tx=0x7fe750009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:49:53.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:53.935+0000 7fe74affd700 1 -- 172.21.15.7:0/4159069531 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe74c053860 con 0x7fe754151d40 2023-03-03T10:49:54.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.170+0000 7fe75c746700 1 -- 172.21.15.7:0/4159069531 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe754074bb0 con 0x7fe754151d40 2023-03-03T10:49:54.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.171+0000 7fe74affd700 1 -- 172.21.15.7:0/4159069531 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe74c01f090 con 0x7fe754151d40 2023-03-03T10:49:54.173 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:49:54.173 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:49:54.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 -- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7380682f0 msgr2=0x7fe73806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7380682f0 0x7fe73806a7a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fe750007fd0 tx=0x7fe750009040 comp rx=0 tx=0).stop 2023-03-03T10:49:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 -- 172.21.15.7:0/4159069531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe754151d40 msgr2=0x7fe754150450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe754151d40 0x7fe754150450 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe75406bb70 tx=0x7fe74c004ad0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 -- 172.21.15.7:0/4159069531 shutdown_connections 2023-03-03T10:49:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7380682f0 0x7fe73806a7a0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe754151d40 0x7fe754150450 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe75406c480 0x7fe7541ed740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 --2- 172.21.15.7:0/4159069531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7540c2930 0x7fe7541ea1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 -- 172.21.15.7:0/4159069531 >> 172.21.15.7:0/4159069531 conn(0x7fe7540a94d0 msgr2=0x7fe754158380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:49:54.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 -- 172.21.15.7:0/4159069531 shutdown_connections 2023-03-03T10:49:54.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.173+0000 7fe75c746700 1 -- 172.21.15.7:0/4159069531 wait complete. 2023-03-03T10:49:54.177 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:49:54.187 INFO:tasks.mds_thrash.fs.[a]:waiting for 38 secs before thrashing 2023-03-03T10:49:54.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:49:54.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.456+0000 7f1b8438b700 1 Processor -- start 2023-03-03T10:49:54.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.456+0000 7f1b8438b700 1 -- start start 2023-03-03T10:49:54.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b8438b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1b7c155390 0x7f1b7c0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:54.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b8438b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1b7c06e020 0x7f1b7c071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:54.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b8438b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c154910 con 0x7f1b7c072520 2023-03-03T10:49:54.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b8438b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b7c070b90 con 0x7f1b7c06e020 2023-03-03T10:49:54.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b8438b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b7c151320 con 0x7f1b7c155390 2023-03-03T10:49:54.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b81926700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1b7c06e020 0x7f1b7c071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:54.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b81926700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1b7c06e020 0x7f1b7c071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33626/0 (socket says 172.21.15.7:33626) 2023-03-03T10:49:54.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b81926700 1 -- 172.21.15.7:0/51069548 learned_addr learned my addr 172.21.15.7:0/51069548 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:49:54.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b82127700 1 --2- 172.21.15.7:0/51069548 >> v2:172.21.15.143:3300/0 conn(0x7f1b7c155390 0x7f1b7c0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:54.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b81926700 1 -- 172.21.15.7:0/51069548 >> v2:172.21.15.143:3300/0 conn(0x7f1b7c155390 msgr2=0x7f1b7c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:49:54.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b81926700 1 --2- 172.21.15.7:0/51069548 >> v2:172.21.15.143:3300/0 conn(0x7f1b7c155390 0x7f1b7c0c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b81926700 1 -- 172.21.15.7:0/51069548 >> v1:172.21.15.143:6789/0 conn(0x7f1b7c072520 legacy=0x7f1b7c151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:54.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b81926700 1 -- 172.21.15.7:0/51069548 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c0702a0 con 0x7f1b7c06e020 2023-03-03T10:49:54.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.457+0000 7f1b81926700 1 --2- 172.21.15.7:0/51069548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c06e020 0x7f1b7c071fe0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c06d700 tx=0x7f1b6c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52c14968d4ca0a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:49:54.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b81125700 1 -- 172.21.15.7:0/51069548 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c011830 con 0x7f1b7c06e020 2023-03-03T10:49:54.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b81125700 1 -- 172.21.15.7:0/51069548 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6c011990 con 0x7f1b7c06e020 2023-03-03T10:49:54.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b81125700 1 -- 172.21.15.7:0/51069548 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c011d20 con 0x7f1b7c06e020 2023-03-03T10:49:54.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b8438b700 1 -- 172.21.15.7:0/51069548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c06e020 msgr2=0x7f1b7c071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:54.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b8438b700 1 --2- 172.21.15.7:0/51069548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c06e020 0x7f1b7c071fe0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c06d700 tx=0x7f1b6c009bd0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b8438b700 1 -- 172.21.15.7:0/51069548 shutdown_connections 2023-03-03T10:49:54.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b8438b700 1 --2- 172.21.15.7:0/51069548 >> v2:172.21.15.143:3300/0 conn(0x7f1b7c155390 0x7f1b7c0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b8438b700 1 --2- 172.21.15.7:0/51069548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c06e020 0x7f1b7c071fe0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b8438b700 1 -- 172.21.15.7:0/51069548 >> 172.21.15.7:0/51069548 conn(0x7f1b7c06c110 msgr2=0x7f1b7c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:49:54.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b8438b700 1 -- 172.21.15.7:0/51069548 shutdown_connections 2023-03-03T10:49:54.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.458+0000 7f1b8438b700 1 -- 172.21.15.7:0/51069548 wait complete. 2023-03-03T10:49:54.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.459+0000 7f1b8438b700 1 Processor -- start 2023-03-03T10:49:54.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.459+0000 7f1b8438b700 1 -- start start 2023-03-03T10:49:54.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b8438b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7c06e020 0x7f1b7c0c1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:54.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b8438b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b7c072520 0x7f1b7c149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:54.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b8438b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c155390 0x7f1b7c1475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:54.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b8438b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0c07d0 con 0x7f1b7c155390 2023-03-03T10:49:54.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b8438b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c149300 con 0x7f1b7c072520 2023-03-03T10:49:54.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b8438b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7c0c0a20 con 0x7f1b7c06e020 2023-03-03T10:49:54.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b82928700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c155390 0x7f1b7c1475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:54.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b82928700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c155390 0x7f1b7c1475a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33636/0 (socket says 172.21.15.7:33636) 2023-03-03T10:49:54.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b82928700 1 -- 172.21.15.7:0/2660437321 learned_addr learned my addr 172.21.15.7:0/2660437321 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:49:54.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b82127700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7c06e020 0x7f1b7c0c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:54.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.460+0000 7f1b81926700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b7c072520 0x7f1b7c149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:54.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b82928700 1 -- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7c06e020 msgr2=0x7f1b7c0c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:54.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b82928700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7c06e020 0x7f1b7c0c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b82928700 1 -- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b7c072520 msgr2=0x7f1b7c149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:54.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b82928700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b7c072520 0x7f1b7c149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b82928700 1 -- 172.21.15.7:0/2660437321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c009550 con 0x7f1b7c155390 2023-03-03T10:49:54.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b82928700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c155390 0x7f1b7c1475a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400cc50 tx=0x7f1b7400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:49:54.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b72ffd700 1 -- 172.21.15.7:0/2660437321 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7400ad80 con 0x7f1b7c155390 2023-03-03T10:49:54.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b72ffd700 1 -- 172.21.15.7:0/2660437321 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7400ce50 con 0x7f1b7c155390 2023-03-03T10:49:54.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b8438b700 1 -- 172.21.15.7:0/2660437321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7c070820 con 0x7f1b7c155390 2023-03-03T10:49:54.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b72ffd700 1 -- 172.21.15.7:0/2660437321 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74003c40 con 0x7f1b7c155390 2023-03-03T10:49:54.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.461+0000 7f1b8438b700 1 -- 172.21.15.7:0/2660437321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7c070540 con 0x7f1b7c155390 2023-03-03T10:49:54.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.462+0000 7f1b8438b700 1 -- 172.21.15.7:0/2660437321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7c155760 con 0x7f1b7c155390 2023-03-03T10:49:54.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.462+0000 7f1b72ffd700 1 -- 172.21.15.7:0/2660437321 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b74010050 con 0x7f1b7c155390 2023-03-03T10:49:54.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.463+0000 7f1b72ffd700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b60068320 0x7f1b6006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:49:54.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.463+0000 7f1b72ffd700 1 -- 172.21.15.7:0/2660437321 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1b740860e0 con 0x7f1b7c155390 2023-03-03T10:49:54.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.463+0000 7f1b82127700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b60068320 0x7f1b6006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:49:54.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.463+0000 7f1b82127700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b60068320 0x7f1b6006a7d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0c2b10 tx=0x7f1b78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:49:54.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.467+0000 7f1b72ffd700 1 -- 172.21.15.7:0/2660437321 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b74052480 con 0x7f1b7c155390 2023-03-03T10:49:54.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.704+0000 7f1b8438b700 1 -- 172.21.15.7:0/2660437321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b7c070cf0 con 0x7f1b7c155390 2023-03-03T10:49:54.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.704+0000 7f1b72ffd700 1 -- 172.21.15.7:0/2660437321 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1b74055aa0 con 0x7f1b7c155390 2023-03-03T10:49:54.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:49:54.707 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:49:54.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 -- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b60068320 msgr2=0x7f1b6006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:54.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b60068320 0x7f1b6006a7d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0c2b10 tx=0x7f1b78008040 comp rx=0 tx=0).stop 2023-03-03T10:49:54.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 -- 172.21.15.7:0/2660437321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c155390 msgr2=0x7f1b7c1475a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:49:54.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c155390 0x7f1b7c1475a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400cc50 tx=0x7f1b7400a340 comp rx=0 tx=0).stop 2023-03-03T10:49:54.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 -- 172.21.15.7:0/2660437321 shutdown_connections 2023-03-03T10:49:54.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b60068320 0x7f1b6006a7d0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7c155390 0x7f1b7c1475a0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7c06e020 0x7f1b7c0c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 --2- 172.21.15.7:0/2660437321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b7c072520 0x7f1b7c149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:49:54.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 -- 172.21.15.7:0/2660437321 >> 172.21.15.7:0/2660437321 conn(0x7f1b7c06c110 msgr2=0x7f1b7c15c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:49:54.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 -- 172.21.15.7:0/2660437321 shutdown_connections 2023-03-03T10:49:54.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:49:54.706+0000 7f1b8438b700 1 -- 172.21.15.7:0/2660437321 wait complete. 2023-03-03T10:49:54.711 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:49:54.720 INFO:tasks.mds_thrash.fs.[b]:waiting for 31 secs before thrashing 2023-03-03T10:50:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T10:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T10:50:04.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:04.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:25.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:50:25.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.859+0000 7f3f4336a700 1 Processor -- start 2023-03-03T10:50:25.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.859+0000 7f3f4336a700 1 -- start start 2023-03-03T10:50:25.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f4336a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c14f9d0 0x7f3f3c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:25.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f4336a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f3c14fda0 0x7f3f3c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:25.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f4336a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c0c2f30 con 0x7f3f3c0abe20 2023-03-03T10:50:25.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f4336a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c0c31d0 con 0x7f3f3c14fda0 2023-03-03T10:50:25.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f4336a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c1524a0 con 0x7f3f3c14f9d0 2023-03-03T10:50:25.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f41907700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3f3c0abe20 0x7f3f3c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55874/0 (socket says 172.21.15.7:55874) 2023-03-03T10:50:25.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f41907700 1 -- 172.21.15.7:0/506743343 learned_addr learned my addr 172.21.15.7:0/506743343 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:50:25.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f41106700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c14f9d0 0x7f3f3c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:25.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f40905700 1 --2- 172.21.15.7:0/506743343 >> v2:172.21.15.143:3300/0 conn(0x7f3f3c14fda0 0x7f3f3c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:25.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088108045 0 0) 0x7f3f3c0c2f30 con 0x7f3f3c0abe20 2023-03-03T10:50:25.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f280035b0 con 0x7f3f3c0abe20 2023-03-03T10:50:25.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861254933 0 0) 0x7f3f280035b0 con 0x7f3f3c0abe20 2023-03-03T10:50:25.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c14f9d0 msgr2=0x7f3f3c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:50:25.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f33fff700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c14f9d0 0x7f3f3c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:25.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 >> v2:172.21.15.143:3300/0 conn(0x7f3f3c14fda0 msgr2=0x7f3f3c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:50:25.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f33fff700 1 --2- 172.21.15.7:0/506743343 >> v2:172.21.15.143:3300/0 conn(0x7f3f3c14fda0 0x7f3f3c071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:25.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0c34b0 con 0x7f3f3c0abe20 2023-03-03T10:50:25.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.860+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f3f340019f0 con 0x7f3f3c0abe20 2023-03-03T10:50:25.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 >> v1:172.21.15.7:6789/0 conn(0x7f3f3c0abe20 legacy=0x7f3f3c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:25.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f33fff700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f28003a70 0x7f3f28003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:25.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f33fff700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c14f9d0 0x7f3f28004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:25.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f33fff700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 0x7f3f28005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:25.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0c31d0 con 0x7f3f28004d90 2023-03-03T10:50:25.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c1524a0 con 0x7f3f28003a70 2023-03-03T10:50:25.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2800baf0 con 0x7f3f3c14f9d0 2023-03-03T10:50:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f41907700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 0x7f3f28005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f41106700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f28003a70 0x7f3f28003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f40905700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c14f9d0 0x7f3f28004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f41907700 1 -- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c14f9d0 msgr2=0x7f3f28004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:50:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f41907700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c14f9d0 0x7f3f28004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f41907700 1 -- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f28003a70 msgr2=0x7f3f28003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:50:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f41907700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f28003a70 0x7f3f28003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:25.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f41907700 1 -- 172.21.15.7:0/506743343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f280035b0 con 0x7f3f28004d90 2023-03-03T10:50:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f41907700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 0x7f3f28005200 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400a8d0 tx=0x7f3f28005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639042ab99d2b672 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:50:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3400a340 con 0x7f3f28004d90 2023-03-03T10:50:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f34012600 con 0x7f3f28004d90 2023-03-03T10:50:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.861+0000 7f3f33fff700 1 -- 172.21.15.7:0/506743343 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f340129b0 con 0x7f3f28004d90 2023-03-03T10:50:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.862+0000 7f3f4336a700 1 -- 172.21.15.7:0/506743343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 msgr2=0x7f3f28005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.862+0000 7f3f4336a700 1 --2- 172.21.15.7:0/506743343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 0x7f3f28005200 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400a8d0 tx=0x7f3f28005880 comp rx=0 tx=0).stop 2023-03-03T10:50:25.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.862+0000 7f3f41907700 1 -- 172.21.15.7:0/506743343 reap_dead start 2023-03-03T10:50:25.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.862+0000 7f3f4336a700 1 -- 172.21.15.7:0/506743343 shutdown_connections 2023-03-03T10:50:25.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.862+0000 7f3f4336a700 1 -- 172.21.15.7:0/506743343 >> 172.21.15.7:0/506743343 conn(0x7f3f3c0c8900 msgr2=0x7f3f3c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:50:25.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.862+0000 7f3f4336a700 1 -- 172.21.15.7:0/506743343 shutdown_connections 2023-03-03T10:50:25.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.862+0000 7f3f4336a700 1 -- 172.21.15.7:0/506743343 wait complete. 2023-03-03T10:50:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.863+0000 7f3f4336a700 1 Processor -- start 2023-03-03T10:50:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.863+0000 7f3f4336a700 1 -- start start 2023-03-03T10:50:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f4336a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 0x7f3f3c14efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f4336a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f3c1e85e0 0x7f3f3c1de5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:25.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f4336a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c06c4c0 0x7f3f3c06c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f4336a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0beb40 con 0x7f3f28004d90 2023-03-03T10:50:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f4336a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c06f720 con 0x7f3f3c1e85e0 2023-03-03T10:50:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f4336a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c06fbb0 con 0x7f3f3c06c4c0 2023-03-03T10:50:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f41106700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 0x7f3f3c14efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f41106700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 0x7f3f3c14efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33670/0 (socket says 172.21.15.7:33670) 2023-03-03T10:50:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f41106700 1 -- 172.21.15.7:0/862536842 learned_addr learned my addr 172.21.15.7:0/862536842 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:50:25.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f41907700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c06c4c0 0x7f3f3c06c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f40905700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f3c1e85e0 0x7f3f3c1de5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f41106700 1 -- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c06c4c0 msgr2=0x7f3f3c06c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f41106700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c06c4c0 0x7f3f3c06c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f41106700 1 -- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f3c1e85e0 msgr2=0x7f3f3c1de5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f41106700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f3c1e85e0 0x7f3f3c1de5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.864+0000 7f3f41106700 1 -- 172.21.15.7:0/862536842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0ac350 con 0x7f3f28004d90 2023-03-03T10:50:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.865+0000 7f3f41106700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 0x7f3f3c14efa0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3f380119d0 tx=0x7f3f38011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:50:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.865+0000 7f3f31ffb700 1 -- 172.21.15.7:0/862536842 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38011e60 con 0x7f3f28004d90 2023-03-03T10:50:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.865+0000 7f3f31ffb700 1 -- 172.21.15.7:0/862536842 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3800f8e0 con 0x7f3f28004d90 2023-03-03T10:50:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.865+0000 7f3f4336a700 1 -- 172.21.15.7:0/862536842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c0ab630 con 0x7f3f28004d90 2023-03-03T10:50:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.865+0000 7f3f31ffb700 1 -- 172.21.15.7:0/862536842 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3800fc70 con 0x7f3f28004d90 2023-03-03T10:50:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.865+0000 7f3f4336a700 1 -- 172.21.15.7:0/862536842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c0c3e10 con 0x7f3f28004d90 2023-03-03T10:50:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.866+0000 7f3f4336a700 1 -- 172.21.15.7:0/862536842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c074bb0 con 0x7f3f28004d90 2023-03-03T10:50:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.866+0000 7f3f31ffb700 1 -- 172.21.15.7:0/862536842 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3f38014440 con 0x7f3f28004d90 2023-03-03T10:50:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.867+0000 7f3f31ffb700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f200682f0 0x7f3f2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.867+0000 7f3f31ffb700 1 -- 172.21.15.7:0/862536842 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3f380867f0 con 0x7f3f28004d90 2023-03-03T10:50:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.867+0000 7f3f40905700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f200682f0 0x7f3f2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.867+0000 7f3f40905700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f200682f0 0x7f3f2006a7a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c06eee0 tx=0x7f3f2c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:50:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:25.871+0000 7f3f31ffb700 1 -- 172.21.15.7:0/862536842 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3f38052b30 con 0x7f3f28004d90 2023-03-03T10:50:26.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.108+0000 7f3f4336a700 1 -- 172.21.15.7:0/862536842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f3c0c91f0 con 0x7f3f28004d90 2023-03-03T10:50:26.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.108+0000 7f3f31ffb700 1 -- 172.21.15.7:0/862536842 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3f38056150 con 0x7f3f28004d90 2023-03-03T10:50:26.111 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:50:26.111 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:50:26.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.110+0000 7f3f4336a700 1 -- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f200682f0 msgr2=0x7f3f2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:26.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.110+0000 7f3f4336a700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f200682f0 0x7f3f2006a7a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c06eee0 tx=0x7f3f2c009210 comp rx=0 tx=0).stop 2023-03-03T10:50:26.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.110+0000 7f3f4336a700 1 -- 172.21.15.7:0/862536842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 msgr2=0x7f3f3c14efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:26.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.110+0000 7f3f4336a700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 0x7f3f3c14efa0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3f380119d0 tx=0x7f3f38011ce0 comp rx=0 tx=0).stop 2023-03-03T10:50:26.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.111+0000 7f3f4336a700 1 -- 172.21.15.7:0/862536842 shutdown_connections 2023-03-03T10:50:26.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.111+0000 7f3f4336a700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3c06c4c0 0x7f3f3c06c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:26.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.111+0000 7f3f4336a700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f200682f0 0x7f3f2006a7a0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:26.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.111+0000 7f3f4336a700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f28004d90 0x7f3f3c14efa0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:26.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.111+0000 7f3f4336a700 1 --2- 172.21.15.7:0/862536842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f3c1e85e0 0x7f3f3c1de5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:26.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.111+0000 7f3f4336a700 1 -- 172.21.15.7:0/862536842 >> 172.21.15.7:0/862536842 conn(0x7f3f3c0c8900 msgr2=0x7f3f3c0c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:50:26.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.111+0000 7f3f4336a700 1 -- 172.21.15.7:0/862536842 shutdown_connections 2023-03-03T10:50:26.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:26.111+0000 7f3f4336a700 1 -- 172.21.15.7:0/862536842 wait complete. 2023-03-03T10:50:26.115 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:50:26.125 INFO:tasks.mds_thrash.fs.[b]:waiting for 77 secs before thrashing 2023-03-03T10:50:32.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:50:32.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.326+0000 7f817635c700 1 Processor -- start 2023-03-03T10:50:32.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.326+0000 7f817635c700 1 -- start start 2023-03-03T10:50:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f817635c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81701530d0 0x7f8170156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f817635c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81700c2f30 con 0x7f817014f9d0 2023-03-03T10:50:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f817635c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81700c31d0 con 0x7f817014fda0 2023-03-03T10:50:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f817635c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81701524a0 con 0x7f81701530d0 2023-03-03T10:50:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f81748f9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81701530d0 0x7f8170156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f816ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f817014f9d0 0x7f8170071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55890/0 (socket says 172.21.15.7:55890) 2023-03-03T10:50:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f816ffff700 1 -- 172.21.15.7:0/2908318952 learned_addr learned my addr 172.21.15.7:0/2908318952 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:50:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f81748f9700 1 -- 172.21.15.7:0/2908318952 >> v1:172.21.15.143:6789/0 conn(0x7f817014fda0 legacy=0x7f81701529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f81748f9700 1 -- 172.21.15.7:0/2908318952 >> v1:172.21.15.7:6789/0 conn(0x7f817014f9d0 legacy=0x7f8170071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f816effd700 1 -- 172.21.15.7:0/2908318952 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819096293 0 0) 0x7f81700c2f30 con 0x7f817014f9d0 2023-03-03T10:50:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f81748f9700 1 -- 172.21.15.7:0/2908318952 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700c34b0 con 0x7f81701530d0 2023-03-03T10:50:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f81748f9700 1 --2- 172.21.15.7:0/2908318952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 0x7f8170156430 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f8168009d90 tx=0x7f816800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3809ad6926c4100 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:50:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f816effd700 1 -- 172.21.15.7:0/2908318952 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816800e070 con 0x7f81701530d0 2023-03-03T10:50:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f816effd700 1 -- 172.21.15.7:0/2908318952 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816800bb90 con 0x7f81701530d0 2023-03-03T10:50:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.328+0000 7f816effd700 1 -- 172.21.15.7:0/2908318952 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168003a20 con 0x7f81701530d0 2023-03-03T10:50:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.329+0000 7f817635c700 1 -- 172.21.15.7:0/2908318952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 msgr2=0x7f8170156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.329+0000 7f817635c700 1 --2- 172.21.15.7:0/2908318952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 0x7f8170156430 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f8168009d90 tx=0x7f816800b330 comp rx=0 tx=0).stop 2023-03-03T10:50:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.329+0000 7f817635c700 1 -- 172.21.15.7:0/2908318952 shutdown_connections 2023-03-03T10:50:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.329+0000 7f817635c700 1 --2- 172.21.15.7:0/2908318952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 0x7f8170156430 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.329+0000 7f817635c700 1 -- 172.21.15.7:0/2908318952 >> 172.21.15.7:0/2908318952 conn(0x7f81700c8900 msgr2=0x7f81700c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:50:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.329+0000 7f817635c700 1 -- 172.21.15.7:0/2908318952 shutdown_connections 2023-03-03T10:50:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.329+0000 7f817635c700 1 -- 172.21.15.7:0/2908318952 wait complete. 2023-03-03T10:50:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.331+0000 7f817635c700 1 Processor -- start 2023-03-03T10:50:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.331+0000 7f817635c700 1 -- start start 2023-03-03T10:50:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f817635c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f817014f9d0 0x7f81701ea940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f817635c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f817014fda0 0x7f81700c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f817635c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 0x7f81701e9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f817635c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701dbdd0 con 0x7f81701530d0 2023-03-03T10:50:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f817635c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701eae80 con 0x7f817014f9d0 2023-03-03T10:50:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f817635c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81701db810 con 0x7f817014fda0 2023-03-03T10:50:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f81748f9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 0x7f81701e9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f81748f9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 0x7f81701e9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33688/0 (socket says 172.21.15.7:33688) 2023-03-03T10:50:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f81748f9700 1 -- 172.21.15.7:0/3232103356 learned_addr learned my addr 172.21.15.7:0/3232103356 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:50:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f816ffff700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f817014f9d0 0x7f81701ea940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f816f7fe700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f817014fda0 0x7f81700c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f81748f9700 1 -- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f817014fda0 msgr2=0x7f81700c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:32.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f81748f9700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f817014fda0 0x7f81700c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f81748f9700 1 -- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f817014f9d0 msgr2=0x7f81701ea940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f81748f9700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f817014f9d0 0x7f81701ea940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f81748f9700 1 -- 172.21.15.7:0/3232103356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8168009a70 con 0x7f81701530d0 2023-03-03T10:50:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f81748f9700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 0x7f81701e9e70 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f8170072520 tx=0x7f8168004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:50:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f816cff9700 1 -- 172.21.15.7:0/3232103356 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168003a20 con 0x7f81701530d0 2023-03-03T10:50:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f816cff9700 1 -- 172.21.15.7:0/3232103356 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8168012650 con 0x7f81701530d0 2023-03-03T10:50:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f817635c700 1 -- 172.21.15.7:0/3232103356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81700ac350 con 0x7f81701530d0 2023-03-03T10:50:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f816cff9700 1 -- 172.21.15.7:0/3232103356 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8168012a00 con 0x7f81701530d0 2023-03-03T10:50:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.332+0000 7f817635c700 1 -- 172.21.15.7:0/3232103356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81700ab630 con 0x7f81701530d0 2023-03-03T10:50:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.333+0000 7f817635c700 1 -- 172.21.15.7:0/3232103356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8170071fa0 con 0x7f81701530d0 2023-03-03T10:50:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.334+0000 7f816cff9700 1 -- 172.21.15.7:0/3232103356 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8168012b60 con 0x7f81701530d0 2023-03-03T10:50:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.334+0000 7f816cff9700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81500682f0 0x7f815006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:50:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.335+0000 7f816cff9700 1 -- 172.21.15.7:0/3232103356 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8168088cb0 con 0x7f81701530d0 2023-03-03T10:50:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.335+0000 7f816ffff700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81500682f0 0x7f815006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:50:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.335+0000 7f816ffff700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81500682f0 0x7f815006a7a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8160000f60 tx=0x7f8160007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:50:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.339+0000 7f816cff9700 1 -- 172.21.15.7:0/3232103356 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8168054f60 con 0x7f81701530d0 2023-03-03T10:50:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.574+0000 7f817635c700 1 -- 172.21.15.7:0/3232103356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8170071fa0 con 0x7f81701530d0 2023-03-03T10:50:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.575+0000 7f816cff9700 1 -- 172.21.15.7:0/3232103356 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8168058580 con 0x7f81701530d0 2023-03-03T10:50:32.577 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:50:32.578 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:50:32.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 -- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81500682f0 msgr2=0x7f815006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81500682f0 0x7f815006a7a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8160000f60 tx=0x7f8160007c90 comp rx=0 tx=0).stop 2023-03-03T10:50:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 -- 172.21.15.7:0/3232103356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 msgr2=0x7f81701e9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:50:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 0x7f81701e9e70 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f8170072520 tx=0x7f8168004ad0 comp rx=0 tx=0).stop 2023-03-03T10:50:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 -- 172.21.15.7:0/3232103356 shutdown_connections 2023-03-03T10:50:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81500682f0 0x7f815006a7a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81701530d0 0x7f81701e9e70 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f817014f9d0 0x7f81701ea940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 --2- 172.21.15.7:0/3232103356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f817014fda0 0x7f81700c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:50:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 -- 172.21.15.7:0/3232103356 >> 172.21.15.7:0/3232103356 conn(0x7f81700c8900 msgr2=0x7f817014efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:50:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 -- 172.21.15.7:0/3232103356 shutdown_connections 2023-03-03T10:50:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:50:32.577+0000 7f817635c700 1 -- 172.21.15.7:0/3232103356 wait complete. 2023-03-03T10:50:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:50:32.591 INFO:tasks.mds_thrash.fs.[a]:waiting for 94 secs before thrashing 2023-03-03T10:50:34.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:50:34.763 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:04.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:04.884 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:35.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:35.005 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:51:43.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:51:43.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.262+0000 7f6de2c13700 1 Processor -- start 2023-03-03T10:51:43.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.262+0000 7f6de2c13700 1 -- start start 2023-03-03T10:51:43.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6de2c13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ddc0aad20 0x7f6ddc157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:51:43.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6de2c13700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc0a9c20 con 0x7f6ddc0aa030 2023-03-03T10:51:43.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6de2c13700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc154f10 con 0x7f6ddc14b0d0 2023-03-03T10:51:43.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6de2c13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc0ac3c0 con 0x7f6ddc0aad20 2023-03-03T10:51:43.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6de11b0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6ddc0aa030 0x7f6ddc158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55932/0 (socket says 172.21.15.7:55932) 2023-03-03T10:51:43.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6de11b0700 1 -- 172.21.15.7:0/660863344 learned_addr learned my addr 172.21.15.7:0/660863344 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:51:43.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6ddbfff700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ddc0aad20 0x7f6ddc157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:51:43.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101845324 0 0) 0x7f6ddc0a9c20 con 0x7f6ddc0aa030 2023-03-03T10:51:43.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dcc0035b0 con 0x7f6ddc0aa030 2023-03-03T10:51:43.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1194216794 0 0) 0x7f6dcc0035b0 con 0x7f6ddc0aa030 2023-03-03T10:51:43.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.263+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ddc0aad20 msgr2=0x7f6ddc157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:51:43.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ddc0aad20 0x7f6ddc157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:51:43.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 >> v1:172.21.15.143:6789/0 conn(0x7f6ddc14b0d0 legacy=0x7f6ddc071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:51:43.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc06fc70 con 0x7f6ddc0aa030 2023-03-03T10:51:43.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f6dd40019f0 con 0x7f6ddc0aa030 2023-03-03T10:51:43.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 >> v1:172.21.15.7:6789/0 conn(0x7f6ddc0aa030 legacy=0x7f6ddc158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:51:43.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dcc003aa0 0x7f6dcc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:51:43.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dcc004850 0x7f6dcc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:51:43.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ddc0aad20 0x7f6dcc0052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:51:43.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc0ac3c0 con 0x7f6dcc003aa0 2023-03-03T10:51:43.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc00bba0 con 0x7f6dcc004850 2023-03-03T10:51:43.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc00bd60 con 0x7f6ddc0aad20 2023-03-03T10:51:43.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6de09af700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dcc003aa0 0x7f6dcc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:51:43.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddbfff700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dcc004850 0x7f6dcc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:51:43.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6de11b0700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ddc0aad20 0x7f6dcc0052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:51:43.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6de09af700 1 -- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ddc0aad20 msgr2=0x7f6dcc0052b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:51:43.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6de09af700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ddc0aad20 0x7f6dcc0052b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:51:43.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6de09af700 1 -- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dcc004850 msgr2=0x7f6dcc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:51:43.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6de09af700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6dcc004850 0x7f6dcc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:51:43.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6de09af700 1 -- 172.21.15.7:0/660863344 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc0035b0 con 0x7f6dcc003aa0 2023-03-03T10:51:43.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6de09af700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dcc003aa0 0x7f6dcc003e70 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0007fd0 tx=0x7f6dcc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599f384561d3371e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:51:43.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd000c660 con 0x7f6dcc003aa0 2023-03-03T10:51:43.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd000c7e0 con 0x7f6dcc003aa0 2023-03-03T10:51:43.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.264+0000 7f6ddb7fe700 1 -- 172.21.15.7:0/660863344 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd000cb90 con 0x7f6dcc003aa0 2023-03-03T10:51:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.265+0000 7f6de2c13700 1 -- 172.21.15.7:0/660863344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dcc003aa0 msgr2=0x7f6dcc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:51:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.265+0000 7f6de2c13700 1 --2- 172.21.15.7:0/660863344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6dcc003aa0 0x7f6dcc003e70 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0007fd0 tx=0x7f6dcc005880 comp rx=0 tx=0).stop 2023-03-03T10:51:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.265+0000 7f6de11b0700 1 -- 172.21.15.7:0/660863344 reap_dead start 2023-03-03T10:51:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.265+0000 7f6de2c13700 1 -- 172.21.15.7:0/660863344 shutdown_connections 2023-03-03T10:51:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.265+0000 7f6de2c13700 1 -- 172.21.15.7:0/660863344 >> 172.21.15.7:0/660863344 conn(0x7f6ddc0c8900 msgr2=0x7f6ddc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:51:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.265+0000 7f6de2c13700 1 -- 172.21.15.7:0/660863344 shutdown_connections 2023-03-03T10:51:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.265+0000 7f6de2c13700 1 -- 172.21.15.7:0/660863344 wait complete. 2023-03-03T10:51:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.266+0000 7f6de2c13700 1 Processor -- start 2023-03-03T10:51:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.266+0000 7f6de2c13700 1 -- start start 2023-03-03T10:51:43.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6de2c13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dcc003aa0 0x7f6ddc0c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:51:43.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6de2c13700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ddc155880 0x7f6ddc1450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:51:43.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6de2c13700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc1455e0 0x7f6ddc1e9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:51:43.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6de2c13700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc155e40 con 0x7f6ddc1455e0 2023-03-03T10:51:43.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6de2c13700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc1eda20 con 0x7f6ddc155880 2023-03-03T10:51:43.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6de2c13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc06bde0 con 0x7f6dcc003aa0 2023-03-03T10:51:43.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6de11b0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc1455e0 0x7f6ddc1e9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:51:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6de11b0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc1455e0 0x7f6ddc1e9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33732/0 (socket says 172.21.15.7:33732) 2023-03-03T10:51:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6de11b0700 1 -- 172.21.15.7:0/3561276370 learned_addr learned my addr 172.21.15.7:0/3561276370 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:51:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6de09af700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dcc003aa0 0x7f6ddc0c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:51:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.267+0000 7f6ddbfff700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ddc155880 0x7f6ddc1450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:51:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6de11b0700 1 -- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dcc003aa0 msgr2=0x7f6ddc0c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:51:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6de11b0700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dcc003aa0 0x7f6ddc0c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:51:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6de11b0700 1 -- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ddc155880 msgr2=0x7f6ddc1450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:51:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6de11b0700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ddc155880 0x7f6ddc1450a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:51:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6de11b0700 1 -- 172.21.15.7:0/3561276370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc06f6b0 con 0x7f6ddc1455e0 2023-03-03T10:51:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6de11b0700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc1455e0 0x7f6ddc1e9960 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4002230 tx=0x7f6dd400a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:51:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6dd97fa700 1 -- 172.21.15.7:0/3561276370 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd40019e0 con 0x7f6ddc1455e0 2023-03-03T10:51:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6dd97fa700 1 -- 172.21.15.7:0/3561276370 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd400ab70 con 0x7f6ddc1455e0 2023-03-03T10:51:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6dd97fa700 1 -- 172.21.15.7:0/3561276370 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd40133f0 con 0x7f6ddc1455e0 2023-03-03T10:51:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6de2c13700 1 -- 172.21.15.7:0/3561276370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ddc155160 con 0x7f6ddc1455e0 2023-03-03T10:51:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.268+0000 7f6de2c13700 1 -- 172.21.15.7:0/3561276370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ddc06f990 con 0x7f6ddc1455e0 2023-03-03T10:51:43.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.269+0000 7f6de2c13700 1 -- 172.21.15.7:0/3561276370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ddc1edbd0 con 0x7f6ddc1455e0 2023-03-03T10:51:43.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.270+0000 7f6dd97fa700 1 -- 172.21.15.7:0/3561276370 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6dd4013760 con 0x7f6ddc1455e0 2023-03-03T10:51:43.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.270+0000 7f6dd97fa700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6dc00682f0 0x7f6dc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:51:43.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.270+0000 7f6dd97fa700 1 -- 172.21.15.7:0/3561276370 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6dd4086bd0 con 0x7f6ddc1455e0 2023-03-03T10:51:43.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.270+0000 7f6de09af700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6dc00682f0 0x7f6dc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:51:43.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.271+0000 7f6de09af700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6dc00682f0 0x7f6dc006a7a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0014040 tx=0x7f6dd0005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:51:43.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.274+0000 7f6dd97fa700 1 -- 172.21.15.7:0/3561276370 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6ddc1edbd0 con 0x7f6ddc1455e0 2023-03-03T10:51:43.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.515+0000 7f6de2c13700 1 -- 172.21.15.7:0/3561276370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ddc1edbd0 con 0x7f6ddc1455e0 2023-03-03T10:51:43.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.516+0000 7f6dd97fa700 1 -- 172.21.15.7:0/3561276370 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6ddc1edbd0 con 0x7f6ddc1455e0 2023-03-03T10:51:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:51:43.518 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:51:43.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 -- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6dc00682f0 msgr2=0x7f6dc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:51:43.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6dc00682f0 0x7f6dc006a7a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0014040 tx=0x7f6dd0005040 comp rx=0 tx=0).stop 2023-03-03T10:51:43.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 -- 172.21.15.7:0/3561276370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc1455e0 msgr2=0x7f6ddc1e9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:51:43.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc1455e0 0x7f6ddc1e9960 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4002230 tx=0x7f6dd400a640 comp rx=0 tx=0).stop 2023-03-03T10:51:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 -- 172.21.15.7:0/3561276370 shutdown_connections 2023-03-03T10:51:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6dc00682f0 0x7f6dc006a7a0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:51:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ddc1455e0 0x7f6ddc1e9960 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:51:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6dcc003aa0 0x7f6ddc0c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:51:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 --2- 172.21.15.7:0/3561276370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ddc155880 0x7f6ddc1450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:51:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 -- 172.21.15.7:0/3561276370 >> 172.21.15.7:0/3561276370 conn(0x7f6ddc0c8900 msgr2=0x7f6ddc157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:51:43.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 -- 172.21.15.7:0/3561276370 shutdown_connections 2023-03-03T10:51:43.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:51:43.518+0000 7f6de2c13700 1 -- 172.21.15.7:0/3561276370 wait complete. 2023-03-03T10:51:43.522 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:51:43.532 INFO:tasks.mds_thrash.fs.[b]:waiting for 59 secs before thrashing 2023-03-03T10:52:05.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:05.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:06.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:52:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.729+0000 7fc2fe597700 1 Processor -- start 2023-03-03T10:52:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.729+0000 7fc2fe597700 1 -- start start 2023-03-03T10:52:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2fe597700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc2f814b0d0 0x7fc2f8071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2fe597700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc2f80aad20 0x7fc2f8157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2fe597700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f80a9c20 con 0x7fc2f8071d40 2023-03-03T10:52:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2fe597700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f8154f10 con 0x7fc2f80aad20 2023-03-03T10:52:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2fe597700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f80ac3c0 con 0x7fc2f814b0d0 2023-03-03T10:52:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2fcb34700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc2f8071d40 0x7fc2f8158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55968/0 (socket says 172.21.15.7:55968) 2023-03-03T10:52:06.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2f77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc2f80aad20 0x7fc2f8157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2fcb34700 1 -- 172.21.15.7:0/1499137456 learned_addr learned my addr 172.21.15.7:0/1499137456 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:52:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2f7fff700 1 --2- 172.21.15.7:0/1499137456 >> v2:172.21.15.143:3300/0 conn(0x7fc2f814b0d0 0x7fc2f8071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2f77fe700 1 -- 172.21.15.7:0/1499137456 >> v2:172.21.15.143:3300/0 conn(0x7fc2f814b0d0 msgr2=0x7fc2f8071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:52:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2f77fe700 1 --2- 172.21.15.7:0/1499137456 >> v2:172.21.15.143:3300/0 conn(0x7fc2f814b0d0 0x7fc2f8071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2f77fe700 1 -- 172.21.15.7:0/1499137456 >> v1:172.21.15.7:6789/0 conn(0x7fc2f8071d40 legacy=0x7fc2f8158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2f77fe700 1 -- 172.21.15.7:0/1499137456 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f806fc70 con 0x7fc2f80aad20 2023-03-03T10:52:06.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2f6ffd700 1 -- 172.21.15.7:0/1499137456 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360450330 0 0) 0x7fc2f80a9c20 con 0x7fc2f8071d40 2023-03-03T10:52:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2f7fff700 1 --2- 172.21.15.7:0/1499137456 >> v2:172.21.15.143:3300/0 conn(0x7fc2f814b0d0 0x7fc2f8071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:52:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2f77fe700 1 --2- 172.21.15.7:0/1499137456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f80aad20 0x7fc2f8157df0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8070930 tx=0x7fc2ec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c5cc88cad9b44a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.730+0000 7fc2f6ffd700 1 -- 172.21.15.7:0/1499137456 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec011830 con 0x7fc2f80aad20 2023-03-03T10:52:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.731+0000 7fc2f6ffd700 1 -- 172.21.15.7:0/1499137456 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ec011990 con 0x7fc2f80aad20 2023-03-03T10:52:06.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.731+0000 7fc2f6ffd700 1 -- 172.21.15.7:0/1499137456 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec011d20 con 0x7fc2f80aad20 2023-03-03T10:52:06.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.731+0000 7fc2fe597700 1 -- 172.21.15.7:0/1499137456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f80aad20 msgr2=0x7fc2f8157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:06.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.731+0000 7fc2fe597700 1 --2- 172.21.15.7:0/1499137456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f80aad20 0x7fc2f8157df0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8070930 tx=0x7fc2ec009b20 comp rx=0 tx=0).stop 2023-03-03T10:52:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.731+0000 7fc2fe597700 1 -- 172.21.15.7:0/1499137456 shutdown_connections 2023-03-03T10:52:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.731+0000 7fc2fe597700 1 --2- 172.21.15.7:0/1499137456 >> v2:172.21.15.143:3300/0 conn(0x7fc2f814b0d0 0x7fc2f8071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.731+0000 7fc2fe597700 1 --2- 172.21.15.7:0/1499137456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f80aad20 0x7fc2f8157df0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.731+0000 7fc2fe597700 1 -- 172.21.15.7:0/1499137456 >> 172.21.15.7:0/1499137456 conn(0x7fc2f80c8900 msgr2=0x7fc2f80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:52:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.731+0000 7fc2fe597700 1 -- 172.21.15.7:0/1499137456 shutdown_connections 2023-03-03T10:52:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.731+0000 7fc2fe597700 1 -- 172.21.15.7:0/1499137456 wait complete. 2023-03-03T10:52:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.732+0000 7fc2fe597700 1 Processor -- start 2023-03-03T10:52:06.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.732+0000 7fc2fe597700 1 -- start start 2023-03-03T10:52:06.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2fe597700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f8071d40 0x7fc2f8149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:06.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2fe597700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f80aad20 0x7fc2f806eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:06.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2fe597700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f814b0d0 0x7fc2f806f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:06.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2fe597700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f8146600 con 0x7fc2f8071d40 2023-03-03T10:52:06.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2fe597700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f81eab70 con 0x7fc2f814b0d0 2023-03-03T10:52:06.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2fe597700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f80c80b0 con 0x7fc2f80aad20 2023-03-03T10:52:06.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2f7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f8071d40 0x7fc2f8149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2f7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f8071d40 0x7fc2f8149be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33758/0 (socket says 172.21.15.7:33758) 2023-03-03T10:52:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2f7fff700 1 -- 172.21.15.7:0/1931595457 learned_addr learned my addr 172.21.15.7:0/1931595457 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:52:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2fcb34700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f814b0d0 0x7fc2f806f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.733+0000 7fc2f77fe700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f80aad20 0x7fc2f806eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2f7fff700 1 -- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f80aad20 msgr2=0x7fc2f806eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2f7fff700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f80aad20 0x7fc2f806eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2f7fff700 1 -- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f814b0d0 msgr2=0x7fc2f806f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:06.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2f7fff700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f814b0d0 0x7fc2f806f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:06.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2f7fff700 1 -- 172.21.15.7:0/1931595457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec0094a0 con 0x7fc2f8071d40 2023-03-03T10:52:06.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2f77fe700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f80aad20 0x7fc2f806eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:52:06.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2f7fff700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f8071d40 0x7fc2f8149be0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c780 tx=0x7fc2e800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:06.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2f4ff9700 1 -- 172.21.15.7:0/1931595457 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8011800 con 0x7fc2f8071d40 2023-03-03T10:52:06.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2f4ff9700 1 -- 172.21.15.7:0/1931595457 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e8011960 con 0x7fc2f8071d40 2023-03-03T10:52:06.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2fe597700 1 -- 172.21.15.7:0/1931595457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f806f6b0 con 0x7fc2f8071d40 2023-03-03T10:52:06.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2f4ff9700 1 -- 172.21.15.7:0/1931595457 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8011cd0 con 0x7fc2f8071d40 2023-03-03T10:52:06.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.734+0000 7fc2fe597700 1 -- 172.21.15.7:0/1931595457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f8155160 con 0x7fc2f8071d40 2023-03-03T10:52:06.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.735+0000 7fc2fe597700 1 -- 172.21.15.7:0/1931595457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f806f990 con 0x7fc2f8071d40 2023-03-03T10:52:06.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.735+0000 7fc2f4ff9700 1 -- 172.21.15.7:0/1931595457 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc2e8020070 con 0x7fc2f8071d40 2023-03-03T10:52:06.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.736+0000 7fc2f4ff9700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2e40682f0 0x7fc2e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.736+0000 7fc2f4ff9700 1 -- 172.21.15.7:0/1931595457 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc2e8086070 con 0x7fc2f8071d40 2023-03-03T10:52:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.736+0000 7fc2f77fe700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2e40682f0 0x7fc2e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.737+0000 7fc2f77fe700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2e40682f0 0x7fc2e406a7a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8070930 tx=0x7fc2ec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.740+0000 7fc2f4ff9700 1 -- 172.21.15.7:0/1931595457 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc2e8052390 con 0x7fc2f8071d40 2023-03-03T10:52:06.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.979+0000 7fc2fe597700 1 -- 172.21.15.7:0/1931595457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2f8074bb0 con 0x7fc2f8071d40 2023-03-03T10:52:06.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.980+0000 7fc2f4ff9700 1 -- 172.21.15.7:0/1931595457 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc2e801e090 con 0x7fc2f8071d40 2023-03-03T10:52:06.982 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:52:06.982 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:52:06.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 -- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2e40682f0 msgr2=0x7fc2e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:06.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2e40682f0 0x7fc2e406a7a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8070930 tx=0x7fc2ec008030 comp rx=0 tx=0).stop 2023-03-03T10:52:06.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 -- 172.21.15.7:0/1931595457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f8071d40 msgr2=0x7fc2f8149be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:06.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f8071d40 0x7fc2f8149be0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c780 tx=0x7fc2e800cb40 comp rx=0 tx=0).stop 2023-03-03T10:52:06.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 -- 172.21.15.7:0/1931595457 shutdown_connections 2023-03-03T10:52:06.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2f814b0d0 0x7fc2f806f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:06.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2e40682f0 0x7fc2e406a7a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:06.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2f8071d40 0x7fc2f8149be0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:06.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 --2- 172.21.15.7:0/1931595457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2f80aad20 0x7fc2f806eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:06.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 -- 172.21.15.7:0/1931595457 >> 172.21.15.7:0/1931595457 conn(0x7fc2f80c8900 msgr2=0x7fc2f80c3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:52:06.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 -- 172.21.15.7:0/1931595457 shutdown_connections 2023-03-03T10:52:06.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:06.982+0000 7fc2fe597700 1 -- 172.21.15.7:0/1931595457 wait complete. 2023-03-03T10:52:06.986 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:52:06.996 INFO:tasks.mds_thrash.fs.[a]:waiting for 36 secs before thrashing 2023-03-03T10:52:35.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:35.246 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:52:42.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:52:42.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.670+0000 7ff600dc9700 1 Processor -- start 2023-03-03T10:52:42.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.670+0000 7ff600dc9700 1 -- start start 2023-03-03T10:52:42.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff600dc9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5fc06a2b0 0x7ff5fc06a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:42.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff600dc9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc0a6370 0x7ff5fc155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:42.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff600dc9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc1558f0 con 0x7ff5fc0c5cb0 2023-03-03T10:52:42.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff600dc9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5fc06bab0 con 0x7ff5fc06a2b0 2023-03-03T10:52:42.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff600dc9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc0a6180 con 0x7ff5fc0a6370 2023-03-03T10:52:42.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff5fa59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5fc06a2b0 0x7ff5fc06a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:42.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff5fad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff5fc0c5cb0 0x7ff5fc0c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56008/0 (socket says 172.21.15.7:56008) 2023-03-03T10:52:42.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff5fad9d700 1 -- 172.21.15.7:0/783153499 learned_addr learned my addr 172.21.15.7:0/783153499 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:52:42.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff5f9d9b700 1 --2- 172.21.15.7:0/783153499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc0a6370 0x7ff5fc155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:42.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff5fa59c700 1 -- 172.21.15.7:0/783153499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc0a6370 msgr2=0x7ff5fc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:52:42.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff5fa59c700 1 --2- 172.21.15.7:0/783153499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc0a6370 0x7ff5fc155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:42.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff5fa59c700 1 -- 172.21.15.7:0/783153499 >> v1:172.21.15.7:6789/0 conn(0x7ff5fc0c5cb0 legacy=0x7ff5fc0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:42.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff5fa59c700 1 -- 172.21.15.7:0/783153499 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc0a5ea0 con 0x7ff5fc06a2b0 2023-03-03T10:52:42.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff5f959a700 1 -- 172.21.15.7:0/783153499 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337394293 0 0) 0x7ff5fc1558f0 con 0x7ff5fc0c5cb0 2023-03-03T10:52:42.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.671+0000 7ff5f9d9b700 1 --2- 172.21.15.7:0/783153499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc0a6370 0x7ff5fc155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:52:42.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff5fa59c700 1 --2- 172.21.15.7:0/783153499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc06a2b0 0x7ff5fc06a680 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009840 tx=0x7ff5f0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0d91bb8d8284e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:42.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff5f959a700 1 -- 172.21.15.7:0/783153499 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0011830 con 0x7ff5fc06a2b0 2023-03-03T10:52:42.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff5f959a700 1 -- 172.21.15.7:0/783153499 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f0011990 con 0x7ff5fc06a2b0 2023-03-03T10:52:42.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff5f959a700 1 -- 172.21.15.7:0/783153499 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0011d20 con 0x7ff5fc06a2b0 2023-03-03T10:52:42.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff600dc9700 1 -- 172.21.15.7:0/783153499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc06a2b0 msgr2=0x7ff5fc06a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:42.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff600dc9700 1 --2- 172.21.15.7:0/783153499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc06a2b0 0x7ff5fc06a680 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009840 tx=0x7ff5f0009b50 comp rx=0 tx=0).stop 2023-03-03T10:52:42.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff600dc9700 1 -- 172.21.15.7:0/783153499 shutdown_connections 2023-03-03T10:52:42.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff600dc9700 1 --2- 172.21.15.7:0/783153499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc06a2b0 0x7ff5fc06a680 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:42.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff600dc9700 1 --2- 172.21.15.7:0/783153499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc0a6370 0x7ff5fc155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:42.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff600dc9700 1 -- 172.21.15.7:0/783153499 >> 172.21.15.7:0/783153499 conn(0x7ff5fc0a9430 msgr2=0x7ff5fc0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:52:42.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff600dc9700 1 -- 172.21.15.7:0/783153499 shutdown_connections 2023-03-03T10:52:42.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.672+0000 7ff600dc9700 1 -- 172.21.15.7:0/783153499 wait complete. 2023-03-03T10:52:42.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.674+0000 7ff600dc9700 1 Processor -- start 2023-03-03T10:52:42.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.674+0000 7ff600dc9700 1 -- start start 2023-03-03T10:52:42.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.674+0000 7ff600dc9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc06a2b0 0x7ff5fc0c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:42.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.674+0000 7ff600dc9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc0a6370 0x7ff5fc14c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:42.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.674+0000 7ff600dc9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5fc0c5cb0 0x7ff5fc14cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:42.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.674+0000 7ff600dc9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc06fcd0 con 0x7ff5fc0a6370 2023-03-03T10:52:42.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.674+0000 7ff600dc9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc070fa0 con 0x7ff5fc0c5cb0 2023-03-03T10:52:42.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.674+0000 7ff600dc9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc1dbc20 con 0x7ff5fc06a2b0 2023-03-03T10:52:42.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5f9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc0a6370 0x7ff5fc14c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:42.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5f9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc0a6370 0x7ff5fc14c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33800/0 (socket says 172.21.15.7:33800) 2023-03-03T10:52:42.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5f9d9b700 1 -- 172.21.15.7:0/251671018 learned_addr learned my addr 172.21.15.7:0/251671018 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:52:42.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5fa59c700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc06a2b0 0x7ff5fc0c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:42.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5fad9d700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5fc0c5cb0 0x7ff5fc14cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:42.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5f9d9b700 1 -- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc06a2b0 msgr2=0x7ff5fc0c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:42.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5f9d9b700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc06a2b0 0x7ff5fc0c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:42.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5f9d9b700 1 -- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5fc0c5cb0 msgr2=0x7ff5fc14cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:42.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5f9d9b700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5fc0c5cb0 0x7ff5fc14cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:42.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5f9d9b700 1 -- 172.21.15.7:0/251671018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f00094a0 con 0x7ff5fc0a6370 2023-03-03T10:52:42.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5f9d9b700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc0a6370 0x7ff5fc14c980 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00c7f0 tx=0x7ff5ec00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:42.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5eaffd700 1 -- 172.21.15.7:0/251671018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec011800 con 0x7ff5fc0a6370 2023-03-03T10:52:42.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5eaffd700 1 -- 172.21.15.7:0/251671018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec011960 con 0x7ff5fc0a6370 2023-03-03T10:52:42.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff600dc9700 1 -- 172.21.15.7:0/251671018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5fc153160 con 0x7ff5fc0a6370 2023-03-03T10:52:42.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff5eaffd700 1 -- 172.21.15.7:0/251671018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec011cf0 con 0x7ff5fc0a6370 2023-03-03T10:52:42.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.675+0000 7ff600dc9700 1 -- 172.21.15.7:0/251671018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc152e80 con 0x7ff5fc0a6370 2023-03-03T10:52:42.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.676+0000 7ff600dc9700 1 -- 172.21.15.7:0/251671018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc0c1500 con 0x7ff5fc0a6370 2023-03-03T10:52:42.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.677+0000 7ff5eaffd700 1 -- 172.21.15.7:0/251671018 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff5ec021070 con 0x7ff5fc0a6370 2023-03-03T10:52:42.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.677+0000 7ff5eaffd700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5dc0682f0 0x7ff5dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:42.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.677+0000 7ff5eaffd700 1 -- 172.21.15.7:0/251671018 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff5ec085e00 con 0x7ff5fc0a6370 2023-03-03T10:52:42.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.677+0000 7ff5fa59c700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5dc0682f0 0x7ff5dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:42.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.678+0000 7ff5fa59c700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5dc0682f0 0x7ff5dc06a7a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00090b0 tx=0x7ff5f001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:42.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.682+0000 7ff5eaffd700 1 -- 172.21.15.7:0/251671018 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5ec0521a0 con 0x7ff5fc0a6370 2023-03-03T10:52:42.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.918+0000 7ff600dc9700 1 -- 172.21.15.7:0/251671018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5fc0710e0 con 0x7ff5fc0a6370 2023-03-03T10:52:42.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.919+0000 7ff5eaffd700 1 -- 172.21.15.7:0/251671018 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff5fc0710e0 con 0x7ff5fc0a6370 2023-03-03T10:52:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:52:42.921 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:52:42.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 -- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5dc0682f0 msgr2=0x7ff5dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:42.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5dc0682f0 0x7ff5dc06a7a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00090b0 tx=0x7ff5f001c040 comp rx=0 tx=0).stop 2023-03-03T10:52:42.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 -- 172.21.15.7:0/251671018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc0a6370 msgr2=0x7ff5fc14c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:42.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc0a6370 0x7ff5fc14c980 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00c7f0 tx=0x7ff5ec00cb00 comp rx=0 tx=0).stop 2023-03-03T10:52:42.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 -- 172.21.15.7:0/251671018 shutdown_connections 2023-03-03T10:52:42.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5fc0c5cb0 0x7ff5fc14cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:42.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5fc06a2b0 0x7ff5fc0c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:42.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5dc0682f0 0x7ff5dc06a7a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:42.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 --2- 172.21.15.7:0/251671018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5fc0a6370 0x7ff5fc14c980 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:42.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 -- 172.21.15.7:0/251671018 >> 172.21.15.7:0/251671018 conn(0x7ff5fc0a9430 msgr2=0x7ff5fc06d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:52:42.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 -- 172.21.15.7:0/251671018 shutdown_connections 2023-03-03T10:52:42.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:42.921+0000 7ff600dc9700 1 -- 172.21.15.7:0/251671018 wait complete. 2023-03-03T10:52:42.925 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:52:42.935 INFO:tasks.mds_thrash.fs.[b]:waiting for 15 secs before thrashing 2023-03-03T10:52:42.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:52:43.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.119+0000 7f9cbb835700 1 Processor -- start 2023-03-03T10:52:43.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.120+0000 7f9cbb835700 1 -- start start 2023-03-03T10:52:43.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.120+0000 7f9cbb835700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb40aac00 0x7f9cb40c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:43.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.120+0000 7f9cbb835700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9cb4155950 0x7f9cb4155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:43.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.120+0000 7f9cbb835700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb40713c0 con 0x7f9cb4071c30 2023-03-03T10:52:43.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.120+0000 7f9cbb835700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb4156310 con 0x7f9cb4155950 2023-03-03T10:52:43.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.120+0000 7f9cbb835700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb4152c70 con 0x7f9cb40aac00 2023-03-03T10:52:43.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9cb9dd2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9cb4155950 0x7f9cb4155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:43.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9cb8dd0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9cb4071c30 0x7f9cb40beb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56020/0 (socket says 172.21.15.7:56020) 2023-03-03T10:52:43.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9cb8dd0700 1 -- 172.21.15.7:0/4153063398 learned_addr learned my addr 172.21.15.7:0/4153063398 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:52:43.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9cb95d1700 1 --2- 172.21.15.7:0/4153063398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb40aac00 0x7f9cb40c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:43.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9cb9dd2700 1 -- 172.21.15.7:0/4153063398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb40aac00 msgr2=0x7f9cb40c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:52:43.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9cb9dd2700 1 --2- 172.21.15.7:0/4153063398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb40aac00 0x7f9cb40c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9cb9dd2700 1 -- 172.21.15.7:0/4153063398 >> v1:172.21.15.7:6789/0 conn(0x7f9cb4071c30 legacy=0x7f9cb40beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:43.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9cb9dd2700 1 -- 172.21.15.7:0/4153063398 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb40c4930 con 0x7f9cb4155950 2023-03-03T10:52:43.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9ca7fff700 1 -- 172.21.15.7:0/4153063398 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523093726 0 0) 0x7f9cb40713c0 con 0x7f9cb4071c30 2023-03-03T10:52:43.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9cb9dd2700 1 --2- 172.21.15.7:0/4153063398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4155950 0x7f9cb4155dd0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f9cac009d90 tx=0x7f9cac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd11d160e54e8be1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:43.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9ca7fff700 1 -- 172.21.15.7:0/4153063398 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac00e070 con 0x7f9cb4155950 2023-03-03T10:52:43.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9ca7fff700 1 -- 172.21.15.7:0/4153063398 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cac00bbd0 con 0x7f9cb4155950 2023-03-03T10:52:43.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.121+0000 7f9ca7fff700 1 -- 172.21.15.7:0/4153063398 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac012430 con 0x7f9cb4155950 2023-03-03T10:52:43.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.122+0000 7f9cbb835700 1 -- 172.21.15.7:0/4153063398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4155950 msgr2=0x7f9cb4155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:43.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.122+0000 7f9cbb835700 1 --2- 172.21.15.7:0/4153063398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4155950 0x7f9cb4155dd0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f9cac009d90 tx=0x7f9cac00b330 comp rx=0 tx=0).stop 2023-03-03T10:52:43.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.122+0000 7f9cbb835700 1 -- 172.21.15.7:0/4153063398 shutdown_connections 2023-03-03T10:52:43.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.122+0000 7f9cbb835700 1 --2- 172.21.15.7:0/4153063398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4155950 0x7f9cb4155dd0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.122+0000 7f9cbb835700 1 --2- 172.21.15.7:0/4153063398 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb40aac00 0x7f9cb40c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.122+0000 7f9cbb835700 1 -- 172.21.15.7:0/4153063398 >> 172.21.15.7:0/4153063398 conn(0x7f9cb40736f0 msgr2=0x7f9cb4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:52:43.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.122+0000 7f9cbb835700 1 -- 172.21.15.7:0/4153063398 shutdown_connections 2023-03-03T10:52:43.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.122+0000 7f9cbb835700 1 -- 172.21.15.7:0/4153063398 wait complete. 2023-03-03T10:52:43.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.123+0000 7f9cbb835700 1 Processor -- start 2023-03-03T10:52:43.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.123+0000 7f9cbb835700 1 -- start start 2023-03-03T10:52:43.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cbb835700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4071c30 0x7f9cb40bec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:43.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cbb835700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb40aac00 0x7f9cb40ad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:43.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cbb835700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb4155950 0x7f9cb40adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:43.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cbb835700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40ae230 con 0x7f9cb4071c30 2023-03-03T10:52:43.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cbb835700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb41eb9f0 con 0x7f9cb40aac00 2023-03-03T10:52:43.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cbb835700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40c12d0 con 0x7f9cb4155950 2023-03-03T10:52:43.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb95d1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4071c30 0x7f9cb40bec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:43.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb95d1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4071c30 0x7f9cb40bec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33812/0 (socket says 172.21.15.7:33812) 2023-03-03T10:52:43.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb95d1700 1 -- 172.21.15.7:0/1975187232 learned_addr learned my addr 172.21.15.7:0/1975187232 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:52:43.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb8dd0700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb40aac00 0x7f9cb40ad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb9dd2700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb4155950 0x7f9cb40adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb95d1700 1 -- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb4155950 msgr2=0x7f9cb40adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb95d1700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb4155950 0x7f9cb40adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb95d1700 1 -- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb40aac00 msgr2=0x7f9cb40ad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb95d1700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb40aac00 0x7f9cb40ad7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb95d1700 1 -- 172.21.15.7:0/1975187232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac009a70 con 0x7f9cb4071c30 2023-03-03T10:52:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cb95d1700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4071c30 0x7f9cb40bec30 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800e810 tx=0x7f9ca800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:43.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9ca5ffb700 1 -- 172.21.15.7:0/1975187232 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8013610 con 0x7f9cb4071c30 2023-03-03T10:52:43.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9ca5ffb700 1 -- 172.21.15.7:0/1975187232 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca8013770 con 0x7f9cb4071c30 2023-03-03T10:52:43.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.124+0000 7f9cbb835700 1 -- 172.21.15.7:0/1975187232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb40c4b80 con 0x7f9cb4071c30 2023-03-03T10:52:43.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.125+0000 7f9ca5ffb700 1 -- 172.21.15.7:0/1975187232 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8013b20 con 0x7f9cb4071c30 2023-03-03T10:52:43.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.125+0000 7f9cbb835700 1 -- 172.21.15.7:0/1975187232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb4152360 con 0x7f9cb4071c30 2023-03-03T10:52:43.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.126+0000 7f9cbb835700 1 -- 172.21.15.7:0/1975187232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb4152620 con 0x7f9cb4071c30 2023-03-03T10:52:43.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.126+0000 7f9ca5ffb700 1 -- 172.21.15.7:0/1975187232 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ca8023070 con 0x7f9cb4071c30 2023-03-03T10:52:43.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.126+0000 7f9ca5ffb700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c980682f0 0x7f9c9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:43.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.127+0000 7f9ca5ffb700 1 -- 172.21.15.7:0/1975187232 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9ca8085e80 con 0x7f9cb4071c30 2023-03-03T10:52:43.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.127+0000 7f9cb8dd0700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c980682f0 0x7f9c9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:43.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.127+0000 7f9cb8dd0700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c980682f0 0x7f9c9806a7a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0000f60 tx=0x7f9cb0007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:43.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.131+0000 7f9ca5ffb700 1 -- 172.21.15.7:0/1975187232 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ca80521a0 con 0x7f9cb4071c30 2023-03-03T10:52:43.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.369+0000 7f9cbb835700 1 -- 172.21.15.7:0/1975187232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cb4074bb0 con 0x7f9cb4071c30 2023-03-03T10:52:43.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.369+0000 7f9ca5ffb700 1 -- 172.21.15.7:0/1975187232 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9ca80557c0 con 0x7f9cb4071c30 2023-03-03T10:52:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:52:43.372 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:52:43.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 -- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c980682f0 msgr2=0x7f9c9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:43.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c980682f0 0x7f9c9806a7a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0000f60 tx=0x7f9cb0007be0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 -- 172.21.15.7:0/1975187232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4071c30 msgr2=0x7f9cb40bec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:43.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4071c30 0x7f9cb40bec30 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800e810 tx=0x7f9ca800ebd0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 -- 172.21.15.7:0/1975187232 shutdown_connections 2023-03-03T10:52:43.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cb4155950 0x7f9cb40adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c980682f0 0x7f9c9806a7a0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cb4071c30 0x7f9cb40bec30 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 --2- 172.21.15.7:0/1975187232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cb40aac00 0x7f9cb40ad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:43.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 -- 172.21.15.7:0/1975187232 >> 172.21.15.7:0/1975187232 conn(0x7f9cb40736f0 msgr2=0x7f9cb4153310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:52:43.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 -- 172.21.15.7:0/1975187232 shutdown_connections 2023-03-03T10:52:43.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:43.371+0000 7f9cbb835700 1 -- 172.21.15.7:0/1975187232 wait complete. 2023-03-03T10:52:43.376 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:52:43.385 INFO:tasks.mds_thrash.fs.[a]:waiting for 94 secs before thrashing 2023-03-03T10:52:57.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:52:58.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.073+0000 7f093aca6700 1 Processor -- start 2023-03-03T10:52:58.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.073+0000 7f093aca6700 1 -- start start 2023-03-03T10:52:58.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.073+0000 7f093aca6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340c00e0 0x7f09340c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.073+0000 7f093aca6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09340c7310 con 0x7f09340caa30 2023-03-03T10:52:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.073+0000 7f093aca6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09340bfd10 con 0x7f0934154ff0 2023-03-03T10:52:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.073+0000 7f093aca6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09340bfa30 con 0x7f09340c00e0 2023-03-03T10:52:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f0938a42700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f09340caa30 0x7f09340c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56052/0 (socket says 172.21.15.7:56052) 2023-03-03T10:52:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f0938a42700 1 -- 172.21.15.7:0/3054644582 learned_addr learned my addr 172.21.15.7:0/3054644582 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:52:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f0933fff700 1 --2- 172.21.15.7:0/3054644582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340c00e0 0x7f09340c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f09337fe700 1 -- 172.21.15.7:0/3054644582 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342624187 0 0) 0x7f09340c7310 con 0x7f09340caa30 2023-03-03T10:52:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f09337fe700 1 -- 172.21.15.7:0/3054644582 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09240035b0 con 0x7f09340caa30 2023-03-03T10:52:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f09337fe700 1 -- 172.21.15.7:0/3054644582 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501247147 0 0) 0x7f09340bfd10 con 0x7f0934154ff0 2023-03-03T10:52:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f09337fe700 1 -- 172.21.15.7:0/3054644582 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09340c7310 con 0x7f0934154ff0 2023-03-03T10:52:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f09337fe700 1 -- 172.21.15.7:0/3054644582 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1342837439 0 0) 0x7f09240035b0 con 0x7f09340caa30 2023-03-03T10:52:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f0933fff700 1 -- 172.21.15.7:0/3054644582 >> v1:172.21.15.143:6789/0 conn(0x7f0934154ff0 legacy=0x7f093415ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f0933fff700 1 -- 172.21.15.7:0/3054644582 >> v1:172.21.15.7:6789/0 conn(0x7f09340caa30 legacy=0x7f09340c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f0933fff700 1 -- 172.21.15.7:0/3054644582 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0934154510 con 0x7f09340c00e0 2023-03-03T10:52:58.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.074+0000 7f0933fff700 1 --2- 172.21.15.7:0/3054644582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340c00e0 0x7f09340c5c60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0920009850 tx=0x7f0920009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4d06a646ab3fbce server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:58.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.075+0000 7f09337fe700 1 -- 172.21.15.7:0/3054644582 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928001670 con 0x7f09340c00e0 2023-03-03T10:52:58.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.075+0000 7f09337fe700 1 -- 172.21.15.7:0/3054644582 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0920005c30 con 0x7f09340c00e0 2023-03-03T10:52:58.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.075+0000 7f09337fe700 1 -- 172.21.15.7:0/3054644582 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920004030 con 0x7f09340c00e0 2023-03-03T10:52:58.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.075+0000 7f093aca6700 1 -- 172.21.15.7:0/3054644582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340c00e0 msgr2=0x7f09340c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:58.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.075+0000 7f093aca6700 1 --2- 172.21.15.7:0/3054644582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340c00e0 0x7f09340c5c60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0920009850 tx=0x7f0920009b60 comp rx=0 tx=0).stop 2023-03-03T10:52:58.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.075+0000 7f093aca6700 1 -- 172.21.15.7:0/3054644582 shutdown_connections 2023-03-03T10:52:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.075+0000 7f093aca6700 1 --2- 172.21.15.7:0/3054644582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340c00e0 0x7f09340c5c60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.075+0000 7f093aca6700 1 -- 172.21.15.7:0/3054644582 >> 172.21.15.7:0/3054644582 conn(0x7f0934066e40 msgr2=0x7f0934071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:52:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.075+0000 7f093aca6700 1 -- 172.21.15.7:0/3054644582 shutdown_connections 2023-03-03T10:52:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.075+0000 7f093aca6700 1 -- 172.21.15.7:0/3054644582 wait complete. 2023-03-03T10:52:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.076+0000 7f093aca6700 1 Processor -- start 2023-03-03T10:52:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.076+0000 7f093aca6700 1 -- start start 2023-03-03T10:52:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.077+0000 7f093aca6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09340c00e0 0x7f09341557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.077+0000 7f093aca6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340caa30 0x7f0934149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.077+0000 7f093aca6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0934154ff0 0x7f09340c2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.077+0000 7f093aca6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0934158480 con 0x7f0934154ff0 2023-03-03T10:52:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.077+0000 7f093aca6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0934157f00 con 0x7f09340c00e0 2023-03-03T10:52:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.077+0000 7f093aca6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093414a4b0 con 0x7f09340caa30 2023-03-03T10:52:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.077+0000 7f0939243700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0934154ff0 0x7f09340c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.077+0000 7f0939243700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0934154ff0 0x7f09340c2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33850/0 (socket says 172.21.15.7:33850) 2023-03-03T10:52:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.077+0000 7f0939243700 1 -- 172.21.15.7:0/4081331416 learned_addr learned my addr 172.21.15.7:0/4081331416 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:52:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f0933fff700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340caa30 0x7f0934149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f0938a42700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09340c00e0 0x7f09341557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f0939243700 1 -- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340caa30 msgr2=0x7f0934149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f0939243700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340caa30 0x7f0934149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f0939243700 1 -- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09340c00e0 msgr2=0x7f09341557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f0939243700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09340c00e0 0x7f09341557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f0939243700 1 -- 172.21.15.7:0/4081331416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920009430 con 0x7f0934154ff0 2023-03-03T10:52:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f0938a42700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09340c00e0 0x7f09341557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:52:58.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f0939243700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0934154ff0 0x7f09340c2220 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f092c00cbc0 tx=0x7f092c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:58.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f09317fa700 1 -- 172.21.15.7:0/4081331416 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c003c00 con 0x7f0934154ff0 2023-03-03T10:52:58.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f09317fa700 1 -- 172.21.15.7:0/4081331416 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c003d60 con 0x7f0934154ff0 2023-03-03T10:52:58.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f093aca6700 1 -- 172.21.15.7:0/4081331416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0934154ad0 con 0x7f0934154ff0 2023-03-03T10:52:58.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f09317fa700 1 -- 172.21.15.7:0/4081331416 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c00ab00 con 0x7f0934154ff0 2023-03-03T10:52:58.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.078+0000 7f093aca6700 1 -- 172.21.15.7:0/4081331416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09341547f0 con 0x7f0934154ff0 2023-03-03T10:52:58.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.079+0000 7f093aca6700 1 -- 172.21.15.7:0/4081331416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09340bf650 con 0x7f0934154ff0 2023-03-03T10:52:58.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.080+0000 7f09317fa700 1 -- 172.21.15.7:0/4081331416 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f092c010050 con 0x7f0934154ff0 2023-03-03T10:52:58.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.080+0000 7f09317fa700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09180682f0 0x7f091806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:52:58.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.080+0000 7f09317fa700 1 -- 172.21.15.7:0/4081331416 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f092c086090 con 0x7f0934154ff0 2023-03-03T10:52:58.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.080+0000 7f0938a42700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09180682f0 0x7f091806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:52:58.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.081+0000 7f0938a42700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09180682f0 0x7f091806a7a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0928001eb0 tx=0x7f0928001ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:52:58.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.084+0000 7f09317fa700 1 -- 172.21.15.7:0/4081331416 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f092c052430 con 0x7f0934154ff0 2023-03-03T10:52:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.324+0000 7f093aca6700 1 -- 172.21.15.7:0/4081331416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0934074bb0 con 0x7f0934154ff0 2023-03-03T10:52:58.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.325+0000 7f09317fa700 1 -- 172.21.15.7:0/4081331416 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f092c055a50 con 0x7f0934154ff0 2023-03-03T10:52:58.327 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:52:58.328 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:52:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 -- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09180682f0 msgr2=0x7f091806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09180682f0 0x7f091806a7a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f0928001eb0 tx=0x7f0928001ca0 comp rx=0 tx=0).stop 2023-03-03T10:52:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 -- 172.21.15.7:0/4081331416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0934154ff0 msgr2=0x7f09340c2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:52:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0934154ff0 0x7f09340c2220 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f092c00cbc0 tx=0x7f092c00ced0 comp rx=0 tx=0).stop 2023-03-03T10:52:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 -- 172.21.15.7:0/4081331416 shutdown_connections 2023-03-03T10:52:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09180682f0 0x7f091806a7a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0934154ff0 0x7f09340c2220 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09340c00e0 0x7f09341557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 --2- 172.21.15.7:0/4081331416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09340caa30 0x7f0934149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:52:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 -- 172.21.15.7:0/4081331416 >> 172.21.15.7:0/4081331416 conn(0x7f0934066e40 msgr2=0x7f09340c8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:52:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 -- 172.21.15.7:0/4081331416 shutdown_connections 2023-03-03T10:52:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:52:58.327+0000 7f093aca6700 1 -- 172.21.15.7:0/4081331416 wait complete. 2023-03-03T10:52:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:52:58.341 INFO:tasks.mds_thrash.fs.[b]:waiting for 81 secs before thrashing 2023-03-03T10:53:05.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:05.386 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:35.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:53:35.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:05.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:05.640 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:17.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:54:17.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.523+0000 7fa9fcd2f700 1 Processor -- start 2023-03-03T10:54:17.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.523+0000 7fa9fcd2f700 1 -- start start 2023-03-03T10:54:17.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.524+0000 7fa9fcd2f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa9f8071e80 0x7fa9f8151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:17.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.524+0000 7fa9fcd2f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa9f814e900 0x7fa9f815ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:17.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.524+0000 7fa9fcd2f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9f81510d0 con 0x7fa9f8072250 2023-03-03T10:54:17.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.524+0000 7fa9fcd2f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f80c3740 con 0x7fa9f814e900 2023-03-03T10:54:17.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.524+0000 7fa9fcd2f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f8070670 con 0x7fa9f8071e80 2023-03-03T10:54:17.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.524+0000 7fa9f6d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa9f814e900 0x7fa9f815ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:17.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.524+0000 7fa9f6d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa9f814e900 0x7fa9f815ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33932/0 (socket says 172.21.15.7:33932) 2023-03-03T10:54:17.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.524+0000 7fa9f6d9d700 1 -- 172.21.15.7:0/3636400371 learned_addr learned my addr 172.21.15.7:0/3636400371 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:54:17.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.524+0000 7fa9f659c700 1 --2- 172.21.15.7:0/3636400371 >> v2:172.21.15.143:3300/0 conn(0x7fa9f8071e80 0x7fa9f8151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:17.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.525+0000 7fa9f6d9d700 1 -- 172.21.15.7:0/3636400371 >> v2:172.21.15.143:3300/0 conn(0x7fa9f8071e80 msgr2=0x7fa9f8151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:54:17.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.525+0000 7fa9f6d9d700 1 --2- 172.21.15.7:0/3636400371 >> v2:172.21.15.143:3300/0 conn(0x7fa9f8071e80 0x7fa9f8151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:17.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.525+0000 7fa9f6d9d700 1 -- 172.21.15.7:0/3636400371 >> v1:172.21.15.143:6789/0 conn(0x7fa9f8072250 legacy=0x7fa9f814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:17.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.525+0000 7fa9f6d9d700 1 -- 172.21.15.7:0/3636400371 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80c39e0 con 0x7fa9f814e900 2023-03-03T10:54:17.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.525+0000 7fa9f6d9d700 1 --2- 172.21.15.7:0/3636400371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f814e900 0x7fa9f815ba50 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa9f806bd10 tx=0x7fa9f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad36debd25e9bec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:54:17.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.525+0000 7fa9f559a700 1 -- 172.21.15.7:0/3636400371 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000e070 con 0x7fa9f814e900 2023-03-03T10:54:17.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.525+0000 7fa9f559a700 1 -- 172.21.15.7:0/3636400371 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f000b910 con 0x7fa9f814e900 2023-03-03T10:54:17.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.525+0000 7fa9f559a700 1 -- 172.21.15.7:0/3636400371 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000bcc0 con 0x7fa9f814e900 2023-03-03T10:54:17.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.526+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/3636400371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f814e900 msgr2=0x7fa9f815ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:17.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.526+0000 7fa9fcd2f700 1 --2- 172.21.15.7:0/3636400371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f814e900 0x7fa9f815ba50 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa9f806bd10 tx=0x7fa9f000b330 comp rx=0 tx=0).stop 2023-03-03T10:54:17.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.526+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/3636400371 shutdown_connections 2023-03-03T10:54:17.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.526+0000 7fa9fcd2f700 1 --2- 172.21.15.7:0/3636400371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f814e900 0x7fa9f815ba50 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:17.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.526+0000 7fa9fcd2f700 1 --2- 172.21.15.7:0/3636400371 >> v2:172.21.15.143:3300/0 conn(0x7fa9f8071e80 0x7fa9f8151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:17.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.526+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/3636400371 >> 172.21.15.7:0/3636400371 conn(0x7fa9f80a94d0 msgr2=0x7fa9f80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:54:17.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.526+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/3636400371 shutdown_connections 2023-03-03T10:54:17.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.526+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/3636400371 wait complete. 2023-03-03T10:54:17.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.527+0000 7fa9fcd2f700 1 Processor -- start 2023-03-03T10:54:17.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.527+0000 7fa9fcd2f700 1 -- start start 2023-03-03T10:54:17.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9fcd2f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8071e80 0x7fa9f80c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:17.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9fcd2f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9f8072250 0x7fa9f814bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:17.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9fcd2f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9f814e900 0x7fa9f814c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:17.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9fcd2f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f814cd90 con 0x7fa9f8071e80 2023-03-03T10:54:17.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9fcd2f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f80c6ef0 con 0x7fa9f8072250 2023-03-03T10:54:17.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9fcd2f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f80c63b0 con 0x7fa9f814e900 2023-03-03T10:54:17.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8071e80 0x7fa9f80c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:17.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8071e80 0x7fa9f80c57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33934/0 (socket says 172.21.15.7:33934) 2023-03-03T10:54:17.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f659c700 1 -- 172.21.15.7:0/914745208 learned_addr learned my addr 172.21.15.7:0/914745208 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:54:17.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f6d9d700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9f814e900 0x7fa9f814c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:17.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f5d9b700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9f8072250 0x7fa9f814bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:17.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f659c700 1 -- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9f814e900 msgr2=0x7fa9f814c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:17.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f659c700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9f814e900 0x7fa9f814c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:17.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f659c700 1 -- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9f8072250 msgr2=0x7fa9f814bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:17.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f659c700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9f8072250 0x7fa9f814bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:17.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f659c700 1 -- 172.21.15.7:0/914745208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f0009b30 con 0x7fa9f8071e80 2023-03-03T10:54:17.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.528+0000 7fa9f6d9d700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9f814e900 0x7fa9f814c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:54:17.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.529+0000 7fa9f659c700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8071e80 0x7fa9f80c57e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800e9b0 tx=0x7fa9e800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:54:17.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.529+0000 7fa9eeffd700 1 -- 172.21.15.7:0/914745208 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8009710 con 0x7fa9f8071e80 2023-03-03T10:54:17.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.529+0000 7fa9eeffd700 1 -- 172.21.15.7:0/914745208 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e800c850 con 0x7fa9f8071e80 2023-03-03T10:54:17.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.529+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/914745208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f806c9a0 con 0x7fa9f8071e80 2023-03-03T10:54:17.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.529+0000 7fa9eeffd700 1 -- 172.21.15.7:0/914745208 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800cbc0 con 0x7fa9f8071e80 2023-03-03T10:54:17.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.529+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/914745208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f8151300 con 0x7fa9f8071e80 2023-03-03T10:54:17.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.530+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/914745208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f806cb30 con 0x7fa9f8071e80 2023-03-03T10:54:17.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.530+0000 7fa9eeffd700 1 -- 172.21.15.7:0/914745208 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa9e8023070 con 0x7fa9f8071e80 2023-03-03T10:54:17.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.531+0000 7fa9eeffd700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9d80682f0 0x7fa9d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:17.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.531+0000 7fa9eeffd700 1 -- 172.21.15.7:0/914745208 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa9e8086480 con 0x7fa9f8071e80 2023-03-03T10:54:17.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.531+0000 7fa9f5d9b700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9d80682f0 0x7fa9d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:17.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.531+0000 7fa9f5d9b700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9d80682f0 0x7fa9d806a7a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa9f814de70 tx=0x7fa9e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:54:17.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.535+0000 7fa9eeffd700 1 -- 172.21.15.7:0/914745208 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa9e8052820 con 0x7fa9f8071e80 2023-03-03T10:54:17.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.774+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/914745208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9f806e110 con 0x7fa9f8071e80 2023-03-03T10:54:17.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.774+0000 7fa9eeffd700 1 -- 172.21.15.7:0/914745208 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa9e8055e40 con 0x7fa9f8071e80 2023-03-03T10:54:17.777 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:54:17.777 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:54:17.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9d80682f0 msgr2=0x7fa9d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9d80682f0 0x7fa9d806a7a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa9f814de70 tx=0x7fa9e0009040 comp rx=0 tx=0).stop 2023-03-03T10:54:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/914745208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8071e80 msgr2=0x7fa9f80c57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8071e80 0x7fa9f80c57e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800e9b0 tx=0x7fa9e800ed70 comp rx=0 tx=0).stop 2023-03-03T10:54:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/914745208 shutdown_connections 2023-03-03T10:54:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9f814e900 0x7fa9f814c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9d80682f0 0x7fa9d806a7a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9f8071e80 0x7fa9f80c57e0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 --2- 172.21.15.7:0/914745208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9f8072250 0x7fa9f814bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/914745208 >> 172.21.15.7:0/914745208 conn(0x7fa9f80a94d0 msgr2=0x7fa9f806da00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:54:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/914745208 shutdown_connections 2023-03-03T10:54:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:17.776+0000 7fa9fcd2f700 1 -- 172.21.15.7:0/914745208 wait complete. 2023-03-03T10:54:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:54:17.790 INFO:tasks.mds_thrash.fs.[a]:waiting for 107 secs before thrashing 2023-03-03T10:54:19.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:54:19.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.478+0000 7f8063593700 1 Processor -- start 2023-03-03T10:54:19.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.478+0000 7f8063593700 1 -- start start 2023-03-03T10:54:19.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8063593700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f805c0c1da0 0x7f805c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:19.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8063593700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c0c7a60 con 0x7f805c0c3e30 2023-03-03T10:54:19.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8063593700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c0c3620 con 0x7f805c06d870 2023-03-03T10:54:19.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8063593700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805c0c2af0 con 0x7f805c0c1da0 2023-03-03T10:54:19.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8060b2e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f805c0c1da0 0x7f805c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:19.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8061b30700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f805c0c3e30 0x7f805c15ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56158/0 (socket says 172.21.15.7:56158) 2023-03-03T10:54:19.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8061b30700 1 -- 172.21.15.7:0/569542835 learned_addr learned my addr 172.21.15.7:0/569542835 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:54:19.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8060b2e700 1 -- 172.21.15.7:0/569542835 >> v1:172.21.15.143:6789/0 conn(0x7f805c06d870 legacy=0x7f805c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:19.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8060b2e700 1 -- 172.21.15.7:0/569542835 >> v1:172.21.15.7:6789/0 conn(0x7f805c0c3e30 legacy=0x7f805c15ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:19.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8060b2e700 1 -- 172.21.15.7:0/569542835 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c1545e0 con 0x7f805c0c1da0 2023-03-03T10:54:19.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.479+0000 7f8060b2e700 1 --2- 172.21.15.7:0/569542835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c1da0 0x7f805c1550c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f8058009840 tx=0x7f8058009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9f1435eb54d1e1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:54:19.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.480+0000 7f8053fff700 1 -- 172.21.15.7:0/569542835 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058011830 con 0x7f805c0c1da0 2023-03-03T10:54:19.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.480+0000 7f8053fff700 1 -- 172.21.15.7:0/569542835 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8058011990 con 0x7f805c0c1da0 2023-03-03T10:54:19.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.480+0000 7f8053fff700 1 -- 172.21.15.7:0/569542835 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058011d20 con 0x7f805c0c1da0 2023-03-03T10:54:19.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.480+0000 7f8063593700 1 -- 172.21.15.7:0/569542835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c1da0 msgr2=0x7f805c1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:19.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.480+0000 7f8063593700 1 --2- 172.21.15.7:0/569542835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c1da0 0x7f805c1550c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f8058009840 tx=0x7f8058009b50 comp rx=0 tx=0).stop 2023-03-03T10:54:19.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.480+0000 7f8063593700 1 -- 172.21.15.7:0/569542835 shutdown_connections 2023-03-03T10:54:19.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.480+0000 7f8063593700 1 --2- 172.21.15.7:0/569542835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c1da0 0x7f805c1550c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:19.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.480+0000 7f8063593700 1 -- 172.21.15.7:0/569542835 >> 172.21.15.7:0/569542835 conn(0x7f805c06c110 msgr2=0x7f805c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:54:19.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.480+0000 7f8063593700 1 -- 172.21.15.7:0/569542835 shutdown_connections 2023-03-03T10:54:19.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.480+0000 7f8063593700 1 -- 172.21.15.7:0/569542835 wait complete. 2023-03-03T10:54:19.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.481+0000 7f8063593700 1 Processor -- start 2023-03-03T10:54:19.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.481+0000 7f8063593700 1 -- start start 2023-03-03T10:54:19.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f8063593700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c06d870 0x7f805c136740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:19.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f8063593700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c0c1da0 0x7f805c0ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:19.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f8063593700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c3e30 0x7f805c0ab830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:19.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f8063593700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c1ee320 con 0x7f805c0c3e30 2023-03-03T10:54:19.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f8063593700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c0aae60 con 0x7f805c0c1da0 2023-03-03T10:54:19.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f8063593700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805c1ee040 con 0x7f805c06d870 2023-03-03T10:54:19.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f8061b30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c3e30 0x7f805c0ab830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:19.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f8061b30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c3e30 0x7f805c0ab830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33952/0 (socket says 172.21.15.7:33952) 2023-03-03T10:54:19.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f8061b30700 1 -- 172.21.15.7:0/3314855231 learned_addr learned my addr 172.21.15.7:0/3314855231 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:54:19.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f806132f700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c06d870 0x7f805c136740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:19.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.482+0000 7f8060b2e700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c0c1da0 0x7f805c0ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:19.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8061b30700 1 -- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c06d870 msgr2=0x7f805c136740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:19.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8061b30700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c06d870 0x7f805c136740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:19.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8061b30700 1 -- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c0c1da0 msgr2=0x7f805c0ab2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:19.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8061b30700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c0c1da0 0x7f805c0ab2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:19.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8061b30700 1 -- 172.21.15.7:0/3314855231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80580094a0 con 0x7f805c0c3e30 2023-03-03T10:54:19.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8060b2e700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c0c1da0 0x7f805c0ab2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:54:19.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8061b30700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c3e30 0x7f805c0ab830 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f805400cbc0 tx=0x7f805400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:54:19.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8051ffb700 1 -- 172.21.15.7:0/3314855231 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805400aaa0 con 0x7f805c0c3e30 2023-03-03T10:54:19.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8051ffb700 1 -- 172.21.15.7:0/3314855231 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805400ac00 con 0x7f805c0c3e30 2023-03-03T10:54:19.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8063593700 1 -- 172.21.15.7:0/3314855231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805c154ba0 con 0x7f805c0c3e30 2023-03-03T10:54:19.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8051ffb700 1 -- 172.21.15.7:0/3314855231 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054012400 con 0x7f805c0c3e30 2023-03-03T10:54:19.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.483+0000 7f8063593700 1 -- 172.21.15.7:0/3314855231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805c1548c0 con 0x7f805c0c3e30 2023-03-03T10:54:19.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.484+0000 7f8063593700 1 -- 172.21.15.7:0/3314855231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805c06dc40 con 0x7f805c0c3e30 2023-03-03T10:54:19.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.485+0000 7f8051ffb700 1 -- 172.21.15.7:0/3314855231 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8054019030 con 0x7f805c0c3e30 2023-03-03T10:54:19.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.485+0000 7f8051ffb700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 0x7f804006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:54:19.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.485+0000 7f8051ffb700 1 -- 172.21.15.7:0/3314855231 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8054085b70 con 0x7f805c0c3e30 2023-03-03T10:54:19.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.485+0000 7f806132f700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 0x7f804006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:54:19.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.486+0000 7f806132f700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 0x7f804006a7d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f804c007fd0 tx=0x7f804c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:54:19.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.490+0000 7f8051ffb700 1 -- 172.21.15.7:0/3314855231 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f805404b5e0 con 0x7f805c0c3e30 2023-03-03T10:54:19.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.732+0000 7f8063593700 1 -- 172.21.15.7:0/3314855231 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f805c074bb0 con 0x7f805c0c3e30 2023-03-03T10:54:19.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.732+0000 7f8051ffb700 1 -- 172.21.15.7:0/3314855231 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8054051fc0 con 0x7f805c0c3e30 2023-03-03T10:54:19.735 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:54:19.735 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:54:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 -- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 msgr2=0x7f804006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 0x7f804006a7d0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f804c007fd0 tx=0x7f804c009040 comp rx=0 tx=0).stop 2023-03-03T10:54:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 -- 172.21.15.7:0/3314855231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c3e30 msgr2=0x7f805c0ab830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:54:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c3e30 0x7f805c0ab830 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f805400cbc0 tx=0x7f805400ced0 comp rx=0 tx=0).stop 2023-03-03T10:54:19.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 -- 172.21.15.7:0/3314855231 shutdown_connections 2023-03-03T10:54:19.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 0x7f804006a7d0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:19.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c0c3e30 0x7f805c0ab830 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:19.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c06d870 0x7f805c136740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:19.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 --2- 172.21.15.7:0/3314855231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c0c1da0 0x7f805c0ab2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:54:19.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 -- 172.21.15.7:0/3314855231 >> 172.21.15.7:0/3314855231 conn(0x7f805c06c110 msgr2=0x7f805c070100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:54:19.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 -- 172.21.15.7:0/3314855231 shutdown_connections 2023-03-03T10:54:19.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:54:19.734+0000 7f8063593700 1 -- 172.21.15.7:0/3314855231 wait complete. 2023-03-03T10:54:19.739 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:54:19.748 INFO:tasks.mds_thrash.fs.[b]:waiting for 104 secs before thrashing 2023-03-03T10:54:35.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:54:35.708 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:05.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:05.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:35.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:55:35.953 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:03.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:56:03.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.886+0000 7f0d9cef7700 1 Processor -- start 2023-03-03T10:56:03.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.886+0000 7f0d9cef7700 1 -- start start 2023-03-03T10:56:03.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.887+0000 7f0d9cef7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0d9814f9d0 0x7f0d98156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:03.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.887+0000 7f0d9cef7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 0x7f0d981529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:03.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.887+0000 7f0d9cef7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d980c2f30 con 0x7f0d9814fda0 2023-03-03T10:56:03.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.887+0000 7f0d9cef7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d980c31d0 con 0x7f0d9814f9d0 2023-03-03T10:56:03.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.887+0000 7f0d9cef7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981524a0 con 0x7f0d980abe20 2023-03-03T10:56:03.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.887+0000 7f0d9659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0d9814f9d0 0x7f0d98156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:03.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.887+0000 7f0d9659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0d9814f9d0 0x7f0d98156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34056/0 (socket says 172.21.15.7:34056) 2023-03-03T10:56:03.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.887+0000 7f0d9659c700 1 -- 172.21.15.7:0/499280654 learned_addr learned my addr 172.21.15.7:0/499280654 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:56:03.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.888+0000 7f0d96d9d700 1 --2- 172.21.15.7:0/499280654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 0x7f0d981529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:03.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.888+0000 7f0d9659c700 1 -- 172.21.15.7:0/499280654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 msgr2=0x7f0d981529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:56:03.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.888+0000 7f0d9659c700 1 --2- 172.21.15.7:0/499280654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 0x7f0d981529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:03.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.888+0000 7f0d9659c700 1 -- 172.21.15.7:0/499280654 >> v1:172.21.15.143:6789/0 conn(0x7f0d9814fda0 legacy=0x7f0d98071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:56:03.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.888+0000 7f0d9659c700 1 -- 172.21.15.7:0/499280654 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d980c34b0 con 0x7f0d9814f9d0 2023-03-03T10:56:03.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.888+0000 7f0d96d9d700 1 --2- 172.21.15.7:0/499280654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 0x7f0d981529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:56:03.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.888+0000 7f0d9659c700 1 --2- 172.21.15.7:0/499280654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 0x7f0d98156430 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f0d800098f0 tx=0x7f0d80009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d36264ae7413b5c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:03.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.888+0000 7f0d9559a700 1 -- 172.21.15.7:0/499280654 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80004230 con 0x7f0d9814f9d0 2023-03-03T10:56:03.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.888+0000 7f0d9559a700 1 -- 172.21.15.7:0/499280654 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d80004390 con 0x7f0d9814f9d0 2023-03-03T10:56:03.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.888+0000 7f0d9559a700 1 -- 172.21.15.7:0/499280654 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d800036a0 con 0x7f0d9814f9d0 2023-03-03T10:56:03.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.889+0000 7f0d9cef7700 1 -- 172.21.15.7:0/499280654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 msgr2=0x7f0d98156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:03.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.889+0000 7f0d9cef7700 1 --2- 172.21.15.7:0/499280654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 0x7f0d98156430 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f0d800098f0 tx=0x7f0d80009c00 comp rx=0 tx=0).stop 2023-03-03T10:56:03.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.889+0000 7f0d9cef7700 1 -- 172.21.15.7:0/499280654 shutdown_connections 2023-03-03T10:56:03.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.889+0000 7f0d9cef7700 1 --2- 172.21.15.7:0/499280654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 0x7f0d981529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:03.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.889+0000 7f0d9cef7700 1 --2- 172.21.15.7:0/499280654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 0x7f0d98156430 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:03.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.889+0000 7f0d9cef7700 1 -- 172.21.15.7:0/499280654 >> 172.21.15.7:0/499280654 conn(0x7f0d980c8900 msgr2=0x7f0d980c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:56:03.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.889+0000 7f0d9cef7700 1 -- 172.21.15.7:0/499280654 shutdown_connections 2023-03-03T10:56:03.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.889+0000 7f0d9cef7700 1 -- 172.21.15.7:0/499280654 wait complete. 2023-03-03T10:56:03.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.890+0000 7f0d9cef7700 1 Processor -- start 2023-03-03T10:56:03.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.890+0000 7f0d9cef7700 1 -- start start 2023-03-03T10:56:03.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d9cef7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 0x7f0d981deab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:03.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d9cef7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 0x7f0d981deff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:03.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d9cef7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d9814fda0 0x7f0d9806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:03.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d9cef7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981df530 con 0x7f0d9814f9d0 2023-03-03T10:56:03.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d9cef7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9806e750 con 0x7f0d9814fda0 2023-03-03T10:56:03.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d9cef7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9806ebe0 con 0x7f0d980abe20 2023-03-03T10:56:03.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d95d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 0x7f0d981deff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:03.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d95d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 0x7f0d981deff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34060/0 (socket says 172.21.15.7:34060) 2023-03-03T10:56:03.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d95d9b700 1 -- 172.21.15.7:0/1879829201 learned_addr learned my addr 172.21.15.7:0/1879829201 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:56:03.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d9659c700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 0x7f0d981deab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:03.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d96d9d700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d9814fda0 0x7f0d9806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:03.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d95d9b700 1 -- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 msgr2=0x7f0d981deab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:03.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d95d9b700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 0x7f0d981deab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d95d9b700 1 -- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d9814fda0 msgr2=0x7f0d9806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d95d9b700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d9814fda0 0x7f0d9806c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d95d9b700 1 -- 172.21.15.7:0/1879829201 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80009550 con 0x7f0d9814f9d0 2023-03-03T10:56:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.891+0000 7f0d95d9b700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 0x7f0d981deff0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800c4a0 tx=0x7f0d8800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.892+0000 7f0d8effd700 1 -- 172.21.15.7:0/1879829201 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88011790 con 0x7f0d9814f9d0 2023-03-03T10:56:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.892+0000 7f0d8effd700 1 -- 172.21.15.7:0/1879829201 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d880118f0 con 0x7f0d9814f9d0 2023-03-03T10:56:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.892+0000 7f0d9cef7700 1 -- 172.21.15.7:0/1879829201 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d980ac350 con 0x7f0d9814f9d0 2023-03-03T10:56:03.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.892+0000 7f0d8effd700 1 -- 172.21.15.7:0/1879829201 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88011ca0 con 0x7f0d9814f9d0 2023-03-03T10:56:03.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.892+0000 7f0d9cef7700 1 -- 172.21.15.7:0/1879829201 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d980ab630 con 0x7f0d9814f9d0 2023-03-03T10:56:03.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.892+0000 7f0d9cef7700 1 -- 172.21.15.7:0/1879829201 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d98071fa0 con 0x7f0d9814f9d0 2023-03-03T10:56:03.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.893+0000 7f0d8effd700 1 -- 172.21.15.7:0/1879829201 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0d88010050 con 0x7f0d9814f9d0 2023-03-03T10:56:03.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.893+0000 7f0d8effd700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d78068320 0x7f0d7806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:03.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.894+0000 7f0d8effd700 1 -- 172.21.15.7:0/1879829201 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0d88085df0 con 0x7f0d9814f9d0 2023-03-03T10:56:03.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.894+0000 7f0d9659c700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d78068320 0x7f0d7806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:03.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.894+0000 7f0d9659c700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d78068320 0x7f0d7806a7d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009130 tx=0x7f0d80008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:03.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:03.898+0000 7f0d8effd700 1 -- 172.21.15.7:0/1879829201 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0d88052190 con 0x7f0d9814f9d0 2023-03-03T10:56:04.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.135+0000 7f0d9cef7700 1 -- 172.21.15.7:0/1879829201 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d981de160 con 0x7f0d9814f9d0 2023-03-03T10:56:04.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.135+0000 7f0d8effd700 1 -- 172.21.15.7:0/1879829201 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0d880557b0 con 0x7f0d9814f9d0 2023-03-03T10:56:04.138 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:56:04.138 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:56:04.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 -- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d78068320 msgr2=0x7f0d7806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d78068320 0x7f0d7806a7d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009130 tx=0x7f0d80008030 comp rx=0 tx=0).stop 2023-03-03T10:56:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 -- 172.21.15.7:0/1879829201 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 msgr2=0x7f0d981deff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 0x7f0d981deff0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800c4a0 tx=0x7f0d8800c860 comp rx=0 tx=0).stop 2023-03-03T10:56:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 -- 172.21.15.7:0/1879829201 shutdown_connections 2023-03-03T10:56:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d9814fda0 0x7f0d9806c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d980abe20 0x7f0d981deab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:04.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d78068320 0x7f0d7806a7d0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 --2- 172.21.15.7:0/1879829201 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d9814f9d0 0x7f0d981deff0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 -- 172.21.15.7:0/1879829201 >> 172.21.15.7:0/1879829201 conn(0x7f0d980c8900 msgr2=0x7f0d980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:56:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 -- 172.21.15.7:0/1879829201 shutdown_connections 2023-03-03T10:56:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.137+0000 7f0d9cef7700 1 -- 172.21.15.7:0/1879829201 wait complete. 2023-03-03T10:56:04.141 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:56:04.151 INFO:tasks.mds_thrash.fs.[b]:waiting for 98 secs before thrashing 2023-03-03T10:56:04.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:56:04.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.927+0000 7fbfeefb8700 1 Processor -- start 2023-03-03T10:56:04.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.927+0000 7fbfeefb8700 1 -- start start 2023-03-03T10:56:04.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfeefb8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe806e020 0x7fbfe8071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:04.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfeefb8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe8154910 con 0x7fbfe8155390 2023-03-03T10:56:04.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfeefb8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe80c4540 con 0x7fbfe8072520 2023-03-03T10:56:04.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfeefb8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe8070b90 con 0x7fbfe806e020 2023-03-03T10:56:04.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfecd54700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbfe8155390 0x7fbfe80c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56278/0 (socket says 172.21.15.7:56278) 2023-03-03T10:56:04.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfecd54700 1 -- 172.21.15.7:0/646084994 learned_addr learned my addr 172.21.15.7:0/646084994 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:56:04.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe7fff700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe806e020 0x7fbfe8071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:04.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382540058 0 0) 0x7fbfe8154910 con 0x7fbfe8155390 2023-03-03T10:56:04.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd80035b0 con 0x7fbfe8155390 2023-03-03T10:56:04.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1919593595 0 0) 0x7fbfd80035b0 con 0x7fbfe8155390 2023-03-03T10:56:04.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe806e020 msgr2=0x7fbfe8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:56:04.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe77fe700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe806e020 0x7fbfe8071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:04.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 >> v1:172.21.15.143:6789/0 conn(0x7fbfe8072520 legacy=0x7fbfe8151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:04.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe80702a0 con 0x7fbfe8155390 2023-03-03T10:56:04.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fbfdc001670 con 0x7fbfe8155390 2023-03-03T10:56:04.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 >> v1:172.21.15.7:6789/0 conn(0x7fbfe8155390 legacy=0x7fbfe80c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:04.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.928+0000 7fbfe77fe700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe806e020 0x7fbfd8003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:04.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe77fe700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 0x7fbfd8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:04.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe77fe700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfd8004d90 0x7fbfd8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:04.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe8070b90 con 0x7fbfd8004020 2023-03-03T10:56:04.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd800de10 con 0x7fbfd8004d90 2023-03-03T10:56:04.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd800dfd0 con 0x7fbfe806e020 2023-03-03T10:56:04.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe7fff700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 0x7fbfd8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:04.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfecd54700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe806e020 0x7fbfd8003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:04.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfed555700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfd8004d90 0x7fbfd8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:04.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe7fff700 1 -- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe806e020 msgr2=0x7fbfd8003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:56:04.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe7fff700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe806e020 0x7fbfd8003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:04.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe7fff700 1 -- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfd8004d90 msgr2=0x7fbfd8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:56:04.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe7fff700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfd8004d90 0x7fbfd8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:04.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe7fff700 1 -- 172.21.15.7:0/646084994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd80035b0 con 0x7fbfd8004020 2023-03-03T10:56:04.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfed555700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfd8004d90 0x7fbfd8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:56:04.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe7fff700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 0x7fbfd8004850 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009b30 tx=0x7fbfd8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d8b0d4552343b1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:04.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd400e6c0 con 0x7fbfd8004020 2023-03-03T10:56:04.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd400e820 con 0x7fbfd8004020 2023-03-03T10:56:04.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.929+0000 7fbfe77fe700 1 -- 172.21.15.7:0/646084994 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd400eb90 con 0x7fbfd8004020 2023-03-03T10:56:04.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.930+0000 7fbfeefb8700 1 -- 172.21.15.7:0/646084994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 msgr2=0x7fbfd8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:04.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.930+0000 7fbfeefb8700 1 --2- 172.21.15.7:0/646084994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 0x7fbfd8004850 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009b30 tx=0x7fbfd8005880 comp rx=0 tx=0).stop 2023-03-03T10:56:04.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.930+0000 7fbfed555700 1 -- 172.21.15.7:0/646084994 reap_dead start 2023-03-03T10:56:04.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.930+0000 7fbfeefb8700 1 -- 172.21.15.7:0/646084994 shutdown_connections 2023-03-03T10:56:04.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.930+0000 7fbfeefb8700 1 -- 172.21.15.7:0/646084994 >> 172.21.15.7:0/646084994 conn(0x7fbfe806c110 msgr2=0x7fbfe806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:56:04.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.930+0000 7fbfeefb8700 1 -- 172.21.15.7:0/646084994 shutdown_connections 2023-03-03T10:56:04.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.930+0000 7fbfeefb8700 1 -- 172.21.15.7:0/646084994 wait complete. 2023-03-03T10:56:04.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.931+0000 7fbfeefb8700 1 Processor -- start 2023-03-03T10:56:04.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.932+0000 7fbfeefb8700 1 -- start start 2023-03-03T10:56:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfeefb8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 0x7fbfe80c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfeefb8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe8149d30 0x7fbfe8149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfeefb8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfe814f860 0x7fbfe814fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfeefb8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe80c5890 con 0x7fbfd8004020 2023-03-03T10:56:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfeefb8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe8149b50 con 0x7fbfe814f860 2023-03-03T10:56:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfeefb8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe806c550 con 0x7fbfe8149d30 2023-03-03T10:56:04.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfecd54700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 0x7fbfe80c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:04.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfecd54700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 0x7fbfe80c5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34078/0 (socket says 172.21.15.7:34078) 2023-03-03T10:56:04.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfecd54700 1 -- 172.21.15.7:0/2794178565 learned_addr learned my addr 172.21.15.7:0/2794178565 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:56:04.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfed555700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfe814f860 0x7fbfe814fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:04.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfecd54700 1 -- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe8149d30 msgr2=0x7fbfe8149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:04.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfe7fff700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe8149d30 0x7fbfe8149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:04.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfecd54700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe8149d30 0x7fbfe8149610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:04.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfecd54700 1 -- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfe814f860 msgr2=0x7fbfe814fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:04.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfecd54700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfe814f860 0x7fbfe814fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:04.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfecd54700 1 -- 172.21.15.7:0/2794178565 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe8070820 con 0x7fbfd8004020 2023-03-03T10:56:04.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfed555700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfe814f860 0x7fbfe814fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:56:04.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfecd54700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 0x7fbfe80c5200 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0011e0 tx=0x7fbfdc007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:04.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfe57fa700 1 -- 172.21.15.7:0/2794178565 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00f660 con 0x7fbfd8004020 2023-03-03T10:56:04.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfe57fa700 1 -- 172.21.15.7:0/2794178565 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc00f7c0 con 0x7fbfd8004020 2023-03-03T10:56:04.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfeefb8700 1 -- 172.21.15.7:0/2794178565 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe8070540 con 0x7fbfd8004020 2023-03-03T10:56:04.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.933+0000 7fbfe57fa700 1 -- 172.21.15.7:0/2794178565 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00fb70 con 0x7fbfd8004020 2023-03-03T10:56:04.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.934+0000 7fbfeefb8700 1 -- 172.21.15.7:0/2794178565 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe8155760 con 0x7fbfd8004020 2023-03-03T10:56:04.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.934+0000 7fbfeefb8700 1 -- 172.21.15.7:0/2794178565 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe8074bb0 con 0x7fbfd8004020 2023-03-03T10:56:04.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.935+0000 7fbfe57fa700 1 -- 172.21.15.7:0/2794178565 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbfdc00fcd0 con 0x7fbfd8004020 2023-03-03T10:56:04.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.935+0000 7fbfe57fa700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfcc0682f0 0x7fbfcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:04.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.935+0000 7fbfe57fa700 1 -- 172.21.15.7:0/2794178565 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbfdc086300 con 0x7fbfd8004020 2023-03-03T10:56:04.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.936+0000 7fbfe7fff700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfcc0682f0 0x7fbfcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:04.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.936+0000 7fbfe7fff700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfcc0682f0 0x7fbfcc06a7a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4000c00 tx=0x7fbfd4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:04.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:04.940+0000 7fbfe57fa700 1 -- 172.21.15.7:0/2794178565 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbfdc052670 con 0x7fbfd8004020 2023-03-03T10:56:05.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.179+0000 7fbfeefb8700 1 -- 172.21.15.7:0/2794178565 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfe814e7f0 con 0x7fbfd8004020 2023-03-03T10:56:05.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.180+0000 7fbfe57fa700 1 -- 172.21.15.7:0/2794178565 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbfdc055c90 con 0x7fbfd8004020 2023-03-03T10:56:05.182 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:56:05.182 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:56:05.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 -- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfcc0682f0 msgr2=0x7fbfcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:05.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfcc0682f0 0x7fbfcc06a7a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4000c00 tx=0x7fbfd4008030 comp rx=0 tx=0).stop 2023-03-03T10:56:05.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 -- 172.21.15.7:0/2794178565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 msgr2=0x7fbfe80c5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:05.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 0x7fbfe80c5200 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0011e0 tx=0x7fbfdc007570 comp rx=0 tx=0).stop 2023-03-03T10:56:05.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 -- 172.21.15.7:0/2794178565 shutdown_connections 2023-03-03T10:56:05.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfe814f860 0x7fbfe814fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:05.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfcc0682f0 0x7fbfcc06a7a0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:05.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfd8004020 0x7fbfe80c5200 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:05.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 --2- 172.21.15.7:0/2794178565 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfe8149d30 0x7fbfe8149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:05.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 -- 172.21.15.7:0/2794178565 >> 172.21.15.7:0/2794178565 conn(0x7fbfe806c110 msgr2=0x7fbfe80c2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:56:05.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 -- 172.21.15.7:0/2794178565 shutdown_connections 2023-03-03T10:56:05.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:05.182+0000 7fbfeefb8700 1 -- 172.21.15.7:0/2794178565 wait complete. 2023-03-03T10:56:05.186 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:56:05.196 INFO:tasks.mds_thrash.fs.[a]:waiting for 54 secs before thrashing 2023-03-03T10:56:06.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:06.091 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:36.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:36.154 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:56:59.198 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:56:59.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.334+0000 7fe7fca21700 1 Processor -- start 2023-03-03T10:56:59.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.334+0000 7fe7fca21700 1 -- start start 2023-03-03T10:56:59.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7fca21700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7f8155390 0x7fe7f80c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:59.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7fca21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7f806e020 0x7fe7f8071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:59.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7fca21700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe7f8072520 0x7fe7f8151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:59.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7fca21700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f8070b90 con 0x7fe7f8155390 2023-03-03T10:56:59.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7fca21700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f8151320 con 0x7fe7f8072520 2023-03-03T10:56:59.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7fca21700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f80c4780 con 0x7fe7f806e020 2023-03-03T10:56:59.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7f659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7f8155390 0x7fe7f80c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:59.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7f659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7f8155390 0x7fe7f80c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34176/0 (socket says 172.21.15.7:34176) 2023-03-03T10:56:59.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7f659c700 1 -- 172.21.15.7:0/2024902981 learned_addr learned my addr 172.21.15.7:0/2024902981 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:56:59.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7f5d9b700 1 --2- 172.21.15.7:0/2024902981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7f806e020 0x7fe7f8071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:59.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7f6d9d700 1 --2- 172.21.15.7:0/2024902981 >> v2:172.21.15.143:3300/0 conn(0x7fe7f8072520 0x7fe7f8151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:59.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.335+0000 7fe7f659c700 1 -- 172.21.15.7:0/2024902981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7f806e020 msgr2=0x7fe7f8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:56:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7f659c700 1 --2- 172.21.15.7:0/2024902981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7f806e020 0x7fe7f8071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7f659c700 1 -- 172.21.15.7:0/2024902981 >> v2:172.21.15.143:3300/0 conn(0x7fe7f8072520 msgr2=0x7fe7f8151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:56:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7f659c700 1 --2- 172.21.15.7:0/2024902981 >> v2:172.21.15.143:3300/0 conn(0x7fe7f8072520 0x7fe7f8151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7f659c700 1 -- 172.21.15.7:0/2024902981 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f8154910 con 0x7fe7f8155390 2023-03-03T10:56:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7f6d9d700 1 --2- 172.21.15.7:0/2024902981 >> v2:172.21.15.143:3300/0 conn(0x7fe7f8072520 0x7fe7f8151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:56:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7f659c700 1 --2- 172.21.15.7:0/2024902981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 0x7fe7f80c3e30 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe7f80c3d80 tx=0x7fe7e800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155f1bfbd8d0626b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7f559a700 1 -- 172.21.15.7:0/2024902981 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e800dd10 con 0x7fe7f8155390 2023-03-03T10:56:59.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7f559a700 1 -- 172.21.15.7:0/2024902981 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e800de70 con 0x7fe7f8155390 2023-03-03T10:56:59.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7f559a700 1 -- 172.21.15.7:0/2024902981 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8010670 con 0x7fe7f8155390 2023-03-03T10:56:59.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7fca21700 1 -- 172.21.15.7:0/2024902981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 msgr2=0x7fe7f80c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:59.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.336+0000 7fe7fca21700 1 --2- 172.21.15.7:0/2024902981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 0x7fe7f80c3e30 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe7f80c3d80 tx=0x7fe7e800bea0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.337+0000 7fe7fca21700 1 -- 172.21.15.7:0/2024902981 shutdown_connections 2023-03-03T10:56:59.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.337+0000 7fe7fca21700 1 --2- 172.21.15.7:0/2024902981 >> v2:172.21.15.143:3300/0 conn(0x7fe7f8072520 0x7fe7f8151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.337+0000 7fe7fca21700 1 --2- 172.21.15.7:0/2024902981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 0x7fe7f80c3e30 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.337+0000 7fe7fca21700 1 --2- 172.21.15.7:0/2024902981 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7f806e020 0x7fe7f8071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.337+0000 7fe7fca21700 1 -- 172.21.15.7:0/2024902981 >> 172.21.15.7:0/2024902981 conn(0x7fe7f806c110 msgr2=0x7fe7f806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:56:59.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.337+0000 7fe7fca21700 1 -- 172.21.15.7:0/2024902981 shutdown_connections 2023-03-03T10:56:59.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.337+0000 7fe7fca21700 1 -- 172.21.15.7:0/2024902981 wait complete. 2023-03-03T10:56:59.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.338+0000 7fe7fca21700 1 Processor -- start 2023-03-03T10:56:59.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.338+0000 7fe7fca21700 1 -- start start 2023-03-03T10:56:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7fca21700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7f806e020 0x7fe7f80c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7fca21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7f8072520 0x7fe7f81e98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7fca21700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 0x7fe7f81e9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7fca21700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f814db60 con 0x7fe7f8155390 2023-03-03T10:56:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7fca21700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f814cf70 con 0x7fe7f806e020 2023-03-03T10:56:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7fca21700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f814d230 con 0x7fe7f8072520 2023-03-03T10:56:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 0x7fe7f81e9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:59.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 0x7fe7f81e9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34186/0 (socket says 172.21.15.7:34186) 2023-03-03T10:56:59.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f6d9d700 1 -- 172.21.15.7:0/2373768903 learned_addr learned my addr 172.21.15.7:0/2373768903 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:56:59.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f5d9b700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7f8072520 0x7fe7f81e98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:59.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f659c700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7f806e020 0x7fe7f80c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:59.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f6d9d700 1 -- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7f8072520 msgr2=0x7fe7f81e98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:59.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f6d9d700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7f8072520 0x7fe7f81e98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f6d9d700 1 -- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7f806e020 msgr2=0x7fe7f80c0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:59.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f6d9d700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7f806e020 0x7fe7f80c0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f6d9d700 1 -- 172.21.15.7:0/2373768903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e800b7f0 con 0x7fe7f8155390 2023-03-03T10:56:59.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f659c700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7f806e020 0x7fe7f80c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:56:59.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.339+0000 7fe7f6d9d700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 0x7fe7f81e9e30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0007ef0 tx=0x7fe7f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:59.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.340+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/2373768903 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f000de60 con 0x7fe7f8155390 2023-03-03T10:56:59.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.340+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/2373768903 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f000f040 con 0x7fe7f8155390 2023-03-03T10:56:59.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.340+0000 7fe7fca21700 1 -- 172.21.15.7:0/2373768903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f8070540 con 0x7fe7f8155390 2023-03-03T10:56:59.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.340+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/2373768903 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0003c60 con 0x7fe7f8155390 2023-03-03T10:56:59.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.340+0000 7fe7fca21700 1 -- 172.21.15.7:0/2373768903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f80702a0 con 0x7fe7f8155390 2023-03-03T10:56:59.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.341+0000 7fe7fca21700 1 -- 172.21.15.7:0/2373768903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f8070820 con 0x7fe7f8155390 2023-03-03T10:56:59.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.341+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/2373768903 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe7f001c030 con 0x7fe7f8155390 2023-03-03T10:56:59.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.342+0000 7fe7e6ffd700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d80682f0 0x7fe7d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:56:59.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.342+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/2373768903 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe7f00861d0 con 0x7fe7f8155390 2023-03-03T10:56:59.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.342+0000 7fe7f659c700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d80682f0 0x7fe7d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:56:59.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.342+0000 7fe7f659c700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d80682f0 0x7fe7d806a7a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800bee0 tx=0x7fe7e801b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:56:59.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.346+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/2373768903 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe7f0052570 con 0x7fe7f8155390 2023-03-03T10:56:59.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.581+0000 7fe7fca21700 1 -- 172.21.15.7:0/2373768903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7f8074bb0 con 0x7fe7f8155390 2023-03-03T10:56:59.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.582+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/2373768903 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe7f0055b90 con 0x7fe7f8155390 2023-03-03T10:56:59.585 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:56:59.586 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:56:59.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 -- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d80682f0 msgr2=0x7fe7d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:59.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d80682f0 0x7fe7d806a7a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800bee0 tx=0x7fe7e801b040 comp rx=0 tx=0).stop 2023-03-03T10:56:59.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 -- 172.21.15.7:0/2373768903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 msgr2=0x7fe7f81e9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:56:59.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 0x7fe7f81e9e30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0007ef0 tx=0x7fe7f000d3b0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 -- 172.21.15.7:0/2373768903 shutdown_connections 2023-03-03T10:56:59.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d80682f0 0x7fe7d806a7a0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f8155390 0x7fe7f81e9e30 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7f806e020 0x7fe7f80c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 --2- 172.21.15.7:0/2373768903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7f8072520 0x7fe7f81e98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:56:59.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 -- 172.21.15.7:0/2373768903 >> 172.21.15.7:0/2373768903 conn(0x7fe7f806c110 msgr2=0x7fe7f80c2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:56:59.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 -- 172.21.15.7:0/2373768903 shutdown_connections 2023-03-03T10:56:59.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:56:59.584+0000 7fe7fca21700 1 -- 172.21.15.7:0/2373768903 wait complete. 2023-03-03T10:56:59.589 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:56:59.599 INFO:tasks.mds_thrash.fs.[a]:waiting for 108 secs before thrashing 2023-03-03T10:57:06.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:06.294 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:36.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:36.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:57:42.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:57:42.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.288+0000 7f6d1855f700 1 Processor -- start 2023-03-03T10:57:42.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.288+0000 7f6d1855f700 1 -- start start 2023-03-03T10:57:42.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.289+0000 7f6d1855f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d100c2930 0x7f6d10157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:57:42.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.289+0000 7f6d1855f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d1006c480 0x7f6d10157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:57:42.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.289+0000 7f6d1855f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1006d430 con 0x7f6d10151d40 2023-03-03T10:57:42.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.289+0000 7f6d1855f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1006c2a0 con 0x7f6d1006c480 2023-03-03T10:57:42.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.289+0000 7f6d1855f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1006cb00 con 0x7f6d100c2930 2023-03-03T10:57:42.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.289+0000 7f6d16afc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d1006c480 0x7f6d10157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:57:42.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.289+0000 7f6d162fb700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6d10151d40 0x7f6d100736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56444/0 (socket says 172.21.15.7:56444) 2023-03-03T10:57:42.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d162fb700 1 -- 172.21.15.7:0/3186823047 learned_addr learned my addr 172.21.15.7:0/3186823047 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:57:42.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d15afa700 1 --2- 172.21.15.7:0/3186823047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d100c2930 0x7f6d10157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:57:42.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d16afc700 1 -- 172.21.15.7:0/3186823047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d100c2930 msgr2=0x7f6d10157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:57:42.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d16afc700 1 --2- 172.21.15.7:0/3186823047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d100c2930 0x7f6d10157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:57:42.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d16afc700 1 -- 172.21.15.7:0/3186823047 >> v1:172.21.15.7:6789/0 conn(0x7f6d10151d40 legacy=0x7f6d100736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:57:42.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d16afc700 1 -- 172.21.15.7:0/3186823047 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1006cde0 con 0x7f6d1006c480 2023-03-03T10:57:42.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d152f9700 1 -- 172.21.15.7:0/3186823047 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261092738 0 0) 0x7f6d1006d430 con 0x7f6d10151d40 2023-03-03T10:57:42.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d16afc700 1 --2- 172.21.15.7:0/3186823047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d1006c480 0x7f6d10157fd0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f6d08009d90 tx=0x7f6d0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74fba04f13c68413 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:57:42.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d152f9700 1 -- 172.21.15.7:0/3186823047 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0800e070 con 0x7f6d1006c480 2023-03-03T10:57:42.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d152f9700 1 -- 172.21.15.7:0/3186823047 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0800bbd0 con 0x7f6d1006c480 2023-03-03T10:57:42.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.290+0000 7f6d152f9700 1 -- 172.21.15.7:0/3186823047 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08012430 con 0x7f6d1006c480 2023-03-03T10:57:42.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.291+0000 7f6d1855f700 1 -- 172.21.15.7:0/3186823047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d1006c480 msgr2=0x7f6d10157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:57:42.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.291+0000 7f6d1855f700 1 --2- 172.21.15.7:0/3186823047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d1006c480 0x7f6d10157fd0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f6d08009d90 tx=0x7f6d0800b330 comp rx=0 tx=0).stop 2023-03-03T10:57:42.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.291+0000 7f6d1855f700 1 -- 172.21.15.7:0/3186823047 shutdown_connections 2023-03-03T10:57:42.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.291+0000 7f6d1855f700 1 --2- 172.21.15.7:0/3186823047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d1006c480 0x7f6d10157fd0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:57:42.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.291+0000 7f6d1855f700 1 --2- 172.21.15.7:0/3186823047 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d100c2930 0x7f6d10157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:57:42.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.291+0000 7f6d1855f700 1 -- 172.21.15.7:0/3186823047 >> 172.21.15.7:0/3186823047 conn(0x7f6d100a94d0 msgr2=0x7f6d100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:57:42.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.291+0000 7f6d1855f700 1 -- 172.21.15.7:0/3186823047 shutdown_connections 2023-03-03T10:57:42.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.291+0000 7f6d1855f700 1 -- 172.21.15.7:0/3186823047 wait complete. 2023-03-03T10:57:42.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.292+0000 7f6d1855f700 1 Processor -- start 2023-03-03T10:57:42.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.292+0000 7f6d1855f700 1 -- start start 2023-03-03T10:57:42.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d1855f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d1006c480 0x7f6d10154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:57:42.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d1855f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d100c2930 0x7f6d100c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:57:42.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d1855f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d10151d40 0x7f6d100ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:57:42.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d1855f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100c09a0 con 0x7f6d100c2930 2023-03-03T10:57:42.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d1855f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100c0c10 con 0x7f6d10151d40 2023-03-03T10:57:42.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d1855f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100c0ef0 con 0x7f6d1006c480 2023-03-03T10:57:42.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d15afa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d100c2930 0x7f6d100c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:57:42.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d15afa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d100c2930 0x7f6d100c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34240/0 (socket says 172.21.15.7:34240) 2023-03-03T10:57:42.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d15afa700 1 -- 172.21.15.7:0/4022049369 learned_addr learned my addr 172.21.15.7:0/4022049369 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:57:42.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d162fb700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d1006c480 0x7f6d10154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:57:42.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.293+0000 7f6d16afc700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d10151d40 0x7f6d100ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:57:42.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d15afa700 1 -- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d1006c480 msgr2=0x7f6d10154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:57:42.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d15afa700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d1006c480 0x7f6d10154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:57:42.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d15afa700 1 -- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d10151d40 msgr2=0x7f6d100ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:57:42.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d15afa700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d10151d40 0x7f6d100ab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:57:42.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d15afa700 1 -- 172.21.15.7:0/4022049369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d08009a70 con 0x7f6d100c2930 2023-03-03T10:57:42.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d162fb700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d1006c480 0x7f6d10154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:57:42.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d16afc700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d10151d40 0x7f6d100ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:57:42.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d15afa700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d100c2930 0x7f6d100c6a60 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400e810 tx=0x7f6d0400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:57:42.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d02ffd700 1 -- 172.21.15.7:0/4022049369 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04013610 con 0x7f6d100c2930 2023-03-03T10:57:42.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d02ffd700 1 -- 172.21.15.7:0/4022049369 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d04013770 con 0x7f6d100c2930 2023-03-03T10:57:42.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d02ffd700 1 -- 172.21.15.7:0/4022049369 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04013b20 con 0x7f6d100c2930 2023-03-03T10:57:42.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d1855f700 1 -- 172.21.15.7:0/4022049369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d10073e00 con 0x7f6d100c2930 2023-03-03T10:57:42.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.294+0000 7f6d1855f700 1 -- 172.21.15.7:0/4022049369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1006d0c0 con 0x7f6d100c2930 2023-03-03T10:57:42.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.295+0000 7f6d1855f700 1 -- 172.21.15.7:0/4022049369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1006bea0 con 0x7f6d100c2930 2023-03-03T10:57:42.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.295+0000 7f6d02ffd700 1 -- 172.21.15.7:0/4022049369 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6d04023070 con 0x7f6d100c2930 2023-03-03T10:57:42.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.296+0000 7f6d02ffd700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6cf40682f0 0x7f6cf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:57:42.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.296+0000 7f6d02ffd700 1 -- 172.21.15.7:0/4022049369 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6d04085e70 con 0x7f6d100c2930 2023-03-03T10:57:42.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.296+0000 7f6d162fb700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6cf40682f0 0x7f6cf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:57:42.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.297+0000 7f6d162fb700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6cf40682f0 0x7f6cf406a7a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c000f60 tx=0x7f6d0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:57:42.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.300+0000 7f6d02ffd700 1 -- 172.21.15.7:0/4022049369 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6d04052190 con 0x7f6d100c2930 2023-03-03T10:57:42.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.540+0000 7f6d1855f700 1 -- 172.21.15.7:0/4022049369 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d100ab280 con 0x7f6d100c2930 2023-03-03T10:57:42.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.540+0000 7f6d02ffd700 1 -- 172.21.15.7:0/4022049369 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6d040557b0 con 0x7f6d100c2930 2023-03-03T10:57:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:57:42.543 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:57:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.542+0000 7f6d1855f700 1 -- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6cf40682f0 msgr2=0x7f6cf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:57:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.542+0000 7f6d1855f700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6cf40682f0 0x7f6cf406a7a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c000f60 tx=0x7f6d0c009040 comp rx=0 tx=0).stop 2023-03-03T10:57:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.542+0000 7f6d1855f700 1 -- 172.21.15.7:0/4022049369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d100c2930 msgr2=0x7f6d100c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:57:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.542+0000 7f6d1855f700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d100c2930 0x7f6d100c6a60 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400e810 tx=0x7f6d0400ebd0 comp rx=0 tx=0).stop 2023-03-03T10:57:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.542+0000 7f6d1855f700 1 -- 172.21.15.7:0/4022049369 shutdown_connections 2023-03-03T10:57:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.542+0000 7f6d1855f700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d10151d40 0x7f6d100ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:57:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.542+0000 7f6d1855f700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d1006c480 0x7f6d10154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:57:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.543+0000 7f6d1855f700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6cf40682f0 0x7f6cf406a7a0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:57:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.543+0000 7f6d1855f700 1 --2- 172.21.15.7:0/4022049369 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d100c2930 0x7f6d100c6a60 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:57:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.543+0000 7f6d1855f700 1 -- 172.21.15.7:0/4022049369 >> 172.21.15.7:0/4022049369 conn(0x7f6d100a94d0 msgr2=0x7f6d10158280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:57:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.543+0000 7f6d1855f700 1 -- 172.21.15.7:0/4022049369 shutdown_connections 2023-03-03T10:57:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:57:42.543+0000 7f6d1855f700 1 -- 172.21.15.7:0/4022049369 wait complete. 2023-03-03T10:57:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:57:42.557 INFO:tasks.mds_thrash.fs.[b]:waiting for 57 secs before thrashing 2023-03-03T10:58:06.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:06.453 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:36.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:36.600 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:58:39.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:58:39.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.695+0000 7fbaa87c6700 1 Processor -- start 2023-03-03T10:58:39.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.695+0000 7fbaa87c6700 1 -- start start 2023-03-03T10:58:39.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.696+0000 7fbaa87c6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbaa006e0a0 0x7fbaa00a85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:39.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.696+0000 7fbaa87c6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa0157ad0 con 0x7fbaa00c2ec0 2023-03-03T10:58:39.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.696+0000 7fbaa87c6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00c6340 con 0x7fbaa00a60f0 2023-03-03T10:58:39.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.696+0000 7fbaa87c6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa006c790 con 0x7fbaa006e0a0 2023-03-03T10:58:39.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.696+0000 7fbaa6d63700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbaa00c2ec0 0x7fbaa015ee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56510/0 (socket says 172.21.15.7:56510) 2023-03-03T10:58:39.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.696+0000 7fbaa6d63700 1 -- 172.21.15.7:0/2004831365 learned_addr learned my addr 172.21.15.7:0/2004831365 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:58:39.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.696+0000 7fbaa6562700 1 --2- 172.21.15.7:0/2004831365 >> v2:172.21.15.143:3300/0 conn(0x7fbaa006e0a0 0x7fbaa00a85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:39.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.696+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71677712 0 0) 0x7fbaa0157ad0 con 0x7fbaa00c2ec0 2023-03-03T10:58:39.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.696+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba8c0035b0 con 0x7fbaa00c2ec0 2023-03-03T10:58:39.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92188099 0 0) 0x7fbaa00c6340 con 0x7fbaa00a60f0 2023-03-03T10:58:39.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa0157ad0 con 0x7fbaa00a60f0 2023-03-03T10:58:39.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225719957 0 0) 0x7fba8c0035b0 con 0x7fbaa00c2ec0 2023-03-03T10:58:39.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 >> v2:172.21.15.143:3300/0 conn(0x7fbaa006e0a0 msgr2=0x7fbaa00a85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:58:39.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 --2- 172.21.15.7:0/2004831365 >> v2:172.21.15.143:3300/0 conn(0x7fbaa006e0a0 0x7fbaa00a85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 >> v1:172.21.15.143:6789/0 conn(0x7fbaa00a60f0 legacy=0x7fbaa0157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:39.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa0155040 con 0x7fbaa00c2ec0 2023-03-03T10:58:39.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fba980019f0 con 0x7fbaa00c2ec0 2023-03-03T10:58:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 >> v1:172.21.15.7:6789/0 conn(0x7fbaa00c2ec0 legacy=0x7fbaa015ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fba8c004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8c004850 0x7fba8c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba8c005330 0x7fba8c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c00c250 con 0x7fba8c003c50 2023-03-03T10:58:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c00c3c0 con 0x7fba8c004850 2023-03-03T10:58:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c00c580 con 0x7fba8c005330 2023-03-03T10:58:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa6562700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fba8c004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:39.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5d61700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8c004850 0x7fba8c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:39.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa6d63700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba8c005330 0x7fba8c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:39.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa6562700 1 -- 172.21.15.7:0/2004831365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba8c005330 msgr2=0x7fba8c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:58:39.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa6562700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba8c005330 0x7fba8c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa6562700 1 -- 172.21.15.7:0/2004831365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8c004850 msgr2=0x7fba8c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:58:39.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa6562700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8c004850 0x7fba8c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa6562700 1 -- 172.21.15.7:0/2004831365 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c0035b0 con 0x7fba8c003c50 2023-03-03T10:58:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5d61700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8c004850 0x7fba8c004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:58:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa6d63700 1 -- 172.21.15.7:0/2004831365 reap_dead start 2023-03-03T10:58:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa6562700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fba8c004040 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fba9c009b60 tx=0x7fba8c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc241ae91c38b64c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.697+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c00e6c0 con 0x7fba8c003c50 2023-03-03T10:58:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.698+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9c00e820 con 0x7fba8c003c50 2023-03-03T10:58:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.698+0000 7fbaa5560700 1 -- 172.21.15.7:0/2004831365 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c00eb90 con 0x7fba8c003c50 2023-03-03T10:58:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.698+0000 7fbaa87c6700 1 -- 172.21.15.7:0/2004831365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 msgr2=0x7fba8c004040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.698+0000 7fbaa87c6700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fba8c004040 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fba9c009b60 tx=0x7fba8c005880 comp rx=0 tx=0).stop 2023-03-03T10:58:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.698+0000 7fbaa87c6700 1 -- 172.21.15.7:0/2004831365 shutdown_connections 2023-03-03T10:58:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.698+0000 7fbaa87c6700 1 --2- 172.21.15.7:0/2004831365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fba8c004040 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.698+0000 7fbaa87c6700 1 -- 172.21.15.7:0/2004831365 >> 172.21.15.7:0/2004831365 conn(0x7fbaa006af40 msgr2=0x7fbaa00c6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:58:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.698+0000 7fbaa87c6700 1 -- 172.21.15.7:0/2004831365 shutdown_connections 2023-03-03T10:58:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.698+0000 7fbaa87c6700 1 -- 172.21.15.7:0/2004831365 wait complete. 2023-03-03T10:58:39.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.700+0000 7fbaa87c6700 1 Processor -- start 2023-03-03T10:58:39.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.700+0000 7fbaa87c6700 1 -- start start 2023-03-03T10:58:39.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa87c6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fbaa00aab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa87c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaa00ab0d0 0x7fbaa0156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa87c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaa006fb20 0x7fbaa01e9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa87c6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa014d920 con 0x7fba8c003c50 2023-03-03T10:58:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa87c6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00a7e10 con 0x7fbaa00ab0d0 2023-03-03T10:58:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa87c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa0070d10 con 0x7fbaa006fb20 2023-03-03T10:58:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa6562700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fbaa00aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:39.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa6562700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fbaa00aab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34306/0 (socket says 172.21.15.7:34306) 2023-03-03T10:58:39.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa6562700 1 -- 172.21.15.7:0/495461628 learned_addr learned my addr 172.21.15.7:0/495461628 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:58:39.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa6562700 1 -- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaa006fb20 msgr2=0x7fbaa01e9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:39.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa6d63700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaa006fb20 0x7fbaa01e9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:39.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa5d61700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaa00ab0d0 0x7fbaa0156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:39.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa6562700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaa006fb20 0x7fbaa01e9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa6562700 1 -- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaa00ab0d0 msgr2=0x7fbaa0156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:39.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa6562700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaa00ab0d0 0x7fbaa0156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa6562700 1 -- 172.21.15.7:0/495461628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00c6620 con 0x7fba8c003c50 2023-03-03T10:58:39.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa5d61700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaa00ab0d0 0x7fbaa0156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T10:58:39.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.701+0000 7fbaa6562700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fbaa00aab90 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0090b0 tx=0x7fba9c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:39.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.702+0000 7fba96ffd700 1 -- 172.21.15.7:0/495461628 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c00e6c0 con 0x7fba8c003c50 2023-03-03T10:58:39.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.702+0000 7fba96ffd700 1 -- 172.21.15.7:0/495461628 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9c003ad0 con 0x7fba8c003c50 2023-03-03T10:58:39.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.702+0000 7fbaa87c6700 1 -- 172.21.15.7:0/495461628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa0155320 con 0x7fba8c003c50 2023-03-03T10:58:39.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.702+0000 7fba96ffd700 1 -- 172.21.15.7:0/495461628 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c003e80 con 0x7fba8c003c50 2023-03-03T10:58:39.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.702+0000 7fbaa87c6700 1 -- 172.21.15.7:0/495461628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa006d2c0 con 0x7fba8c003c50 2023-03-03T10:58:39.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.703+0000 7fbaa87c6700 1 -- 172.21.15.7:0/495461628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa006f970 con 0x7fba8c003c50 2023-03-03T10:58:39.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.703+0000 7fba96ffd700 1 -- 172.21.15.7:0/495461628 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fba9c025070 con 0x7fba8c003c50 2023-03-03T10:58:39.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.703+0000 7fba96ffd700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba840682f0 0x7fba8406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:39.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.704+0000 7fba96ffd700 1 -- 172.21.15.7:0/495461628 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fba9c087ae0 con 0x7fba8c003c50 2023-03-03T10:58:39.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.704+0000 7fbaa5d61700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba840682f0 0x7fba8406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:39.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.704+0000 7fbaa5d61700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba840682f0 0x7fba8406a7a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fba90007fd0 tx=0x7fba90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:39.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.708+0000 7fba96ffd700 1 -- 172.21.15.7:0/495461628 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fba9c053e80 con 0x7fba8c003c50 2023-03-03T10:58:39.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.948+0000 7fbaa87c6700 1 -- 172.21.15.7:0/495461628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbaa0074bb0 con 0x7fba8c003c50 2023-03-03T10:58:39.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.949+0000 7fba96ffd700 1 -- 172.21.15.7:0/495461628 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fba9c0574a0 con 0x7fba8c003c50 2023-03-03T10:58:39.951 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:58:39.951 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:58:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 -- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba840682f0 msgr2=0x7fba8406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba840682f0 0x7fba8406a7a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fba90007fd0 tx=0x7fba90009040 comp rx=0 tx=0).stop 2023-03-03T10:58:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 -- 172.21.15.7:0/495461628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 msgr2=0x7fbaa00aab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fbaa00aab90 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0090b0 tx=0x7fba9c0047c0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 -- 172.21.15.7:0/495461628 shutdown_connections 2023-03-03T10:58:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbaa006fb20 0x7fbaa01e9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba840682f0 0x7fba8406a7a0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c003c50 0x7fbaa00aab90 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 --2- 172.21.15.7:0/495461628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbaa00ab0d0 0x7fbaa0156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 -- 172.21.15.7:0/495461628 >> 172.21.15.7:0/495461628 conn(0x7fbaa006af40 msgr2=0x7fbaa00c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:58:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 -- 172.21.15.7:0/495461628 shutdown_connections 2023-03-03T10:58:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:39.951+0000 7fbaa87c6700 1 -- 172.21.15.7:0/495461628 wait complete. 2023-03-03T10:58:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:58:39.965 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T10:58:47.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T10:58:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.736+0000 7ff6bbfcb700 1 Processor -- start 2023-03-03T10:58:47.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.736+0000 7ff6bbfcb700 1 -- start start 2023-03-03T10:58:47.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6bbfcb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff6b406e020 0x7ff6b4071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:47.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6bbfcb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b4154910 con 0x7ff6b4155390 2023-03-03T10:58:47.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6bbfcb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b40c4540 con 0x7ff6b4072520 2023-03-03T10:58:47.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6bbfcb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b4070b90 con 0x7ff6b406e020 2023-03-03T10:58:47.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b9d67700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff6b4155390 0x7ff6b40c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56534/0 (socket says 172.21.15.7:56534) 2023-03-03T10:58:47.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b9d67700 1 -- 172.21.15.7:0/3253589152 learned_addr learned my addr 172.21.15.7:0/3253589152 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T10:58:47.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b9566700 1 --2- 172.21.15.7:0/3253589152 >> v2:172.21.15.143:3300/0 conn(0x7ff6b406e020 0x7ff6b4071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:47.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211980283 0 0) 0x7ff6b4154910 con 0x7ff6b4155390 2023-03-03T10:58:47.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a00035b0 con 0x7ff6b4155390 2023-03-03T10:58:47.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250665799 0 0) 0x7ff6ac001490 con 0x7ff6b4072520 2023-03-03T10:58:47.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b4154910 con 0x7ff6b4072520 2023-03-03T10:58:47.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 465152088 0 0) 0x7ff6a00035b0 con 0x7ff6b4155390 2023-03-03T10:58:47.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 >> v2:172.21.15.143:3300/0 conn(0x7ff6b406e020 msgr2=0x7ff6b4071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:58:47.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 --2- 172.21.15.7:0/3253589152 >> v2:172.21.15.143:3300/0 conn(0x7ff6b406e020 0x7ff6b4071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:47.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 >> v1:172.21.15.143:6789/0 conn(0x7ff6b4072520 legacy=0x7ff6b4151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:47.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b40702a0 con 0x7ff6b4155390 2023-03-03T10:58:47.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff6b0001670 con 0x7ff6b4155390 2023-03-03T10:58:47.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 >> v1:172.21.15.7:6789/0 conn(0x7ff6b4155390 legacy=0x7ff6b40c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6a0003c50 0x7ff6a0004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.737+0000 7ff6b8d65700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6a0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b8d65700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a00052c0 0x7ff6a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4070b90 con 0x7ff6a0004850 2023-03-03T10:58:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a000c1e0 con 0x7ff6a00052c0 2023-03-03T10:58:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a000e5b0 con 0x7ff6a0003c50 2023-03-03T10:58:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b9566700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6a0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b9d67700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6a0003c50 0x7ff6a0004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6ba568700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a00052c0 0x7ff6a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b9566700 1 -- 172.21.15.7:0/3253589152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6a0003c50 msgr2=0x7ff6a0004040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:58:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b9566700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6a0003c50 0x7ff6a0004040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b9566700 1 -- 172.21.15.7:0/3253589152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a00052c0 msgr2=0x7ff6a000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T10:58:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b9566700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6a00052c0 0x7ff6a000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6ba568700 1 -- 172.21.15.7:0/3253589152 reap_dead start 2023-03-03T10:58:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b9566700 1 -- 172.21.15.7:0/3253589152 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a00035b0 con 0x7ff6a0004850 2023-03-03T10:58:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b9d67700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6a0003c50 0x7ff6a0004040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T10:58:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b9566700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6a0004cc0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009b60 tx=0x7ff6a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5efdb7034a37c5e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a400e6c0 con 0x7ff6a0004850 2023-03-03T10:58:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a400e820 con 0x7ff6a0004850 2023-03-03T10:58:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.738+0000 7ff6b8d65700 1 -- 172.21.15.7:0/3253589152 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a400eb90 con 0x7ff6a0004850 2023-03-03T10:58:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.739+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/3253589152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 msgr2=0x7ff6a0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.739+0000 7ff6bbfcb700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6a0004cc0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009b60 tx=0x7ff6a0005880 comp rx=0 tx=0).stop 2023-03-03T10:58:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.739+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/3253589152 shutdown_connections 2023-03-03T10:58:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.739+0000 7ff6bbfcb700 1 --2- 172.21.15.7:0/3253589152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6a0004cc0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.739+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/3253589152 >> 172.21.15.7:0/3253589152 conn(0x7ff6b406c110 msgr2=0x7ff6b406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:58:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.739+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/3253589152 shutdown_connections 2023-03-03T10:58:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.739+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/3253589152 wait complete. 2023-03-03T10:58:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.740+0000 7ff6bbfcb700 1 Processor -- start 2023-03-03T10:58:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.740+0000 7ff6bbfcb700 1 -- start start 2023-03-03T10:58:47.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6bbfcb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6b41e5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:47.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6bbfcb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6b41dab60 0x7ff6b4141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:47.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6bbfcb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6b4142110 0x7ff6b40aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6bbfcb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b41dd6c0 con 0x7ff6a0004850 2023-03-03T10:58:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6bbfcb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b41dcab0 con 0x7ff6b4142110 2023-03-03T10:58:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6bbfcb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b41dcd90 con 0x7ff6b41dab60 2023-03-03T10:58:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6b9d67700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6b41e5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6b9d67700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6b41e5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34334/0 (socket says 172.21.15.7:34334) 2023-03-03T10:58:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6b9d67700 1 -- 172.21.15.7:0/978976952 learned_addr learned my addr 172.21.15.7:0/978976952 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T10:58:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6ba568700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6b4142110 0x7ff6b40aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6b9566700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6b41dab60 0x7ff6b4141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6b9d67700 1 -- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6b41dab60 msgr2=0x7ff6b4141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6b9d67700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6b41dab60 0x7ff6b4141bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6b9d67700 1 -- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6b4142110 msgr2=0x7ff6b40aa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6b9d67700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6b4142110 0x7ff6b40aa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6b9d67700 1 -- 172.21.15.7:0/978976952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b4070820 con 0x7ff6a0004850 2023-03-03T10:58:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6b9d67700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6b41e5530 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0000ff0 tx=0x7ff6b000ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.741+0000 7ff6aa7fc700 1 -- 172.21.15.7:0/978976952 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0014480 con 0x7ff6a0004850 2023-03-03T10:58:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.742+0000 7ff6aa7fc700 1 -- 172.21.15.7:0/978976952 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b00145e0 con 0x7ff6a0004850 2023-03-03T10:58:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.742+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/978976952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b4070540 con 0x7ff6a0004850 2023-03-03T10:58:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.742+0000 7ff6aa7fc700 1 -- 172.21.15.7:0/978976952 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0014970 con 0x7ff6a0004850 2023-03-03T10:58:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.742+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/978976952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b4155760 con 0x7ff6a0004850 2023-03-03T10:58:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.742+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/978976952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b4074bb0 con 0x7ff6a0004850 2023-03-03T10:58:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.743+0000 7ff6aa7fc700 1 -- 172.21.15.7:0/978976952 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff6b001b030 con 0x7ff6a0004850 2023-03-03T10:58:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.743+0000 7ff6aa7fc700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6980682f0 0x7ff69806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T10:58:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.743+0000 7ff6aa7fc700 1 -- 172.21.15.7:0/978976952 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff6b00855d0 con 0x7ff6a0004850 2023-03-03T10:58:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.744+0000 7ff6b9566700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6980682f0 0x7ff69806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T10:58:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.744+0000 7ff6b9566700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6980682f0 0x7ff69806a7a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40090b0 tx=0x7ff6a401f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T10:58:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.748+0000 7ff6aa7fc700 1 -- 172.21.15.7:0/978976952 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff6b004d5e0 con 0x7ff6a0004850 2023-03-03T10:58:47.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.986+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/978976952 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6b40c1da0 con 0x7ff6a0004850 2023-03-03T10:58:47.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.986+0000 7ff6aa7fc700 1 -- 172.21.15.7:0/978976952 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff6b0019090 con 0x7ff6a0004850 2023-03-03T10:58:47.989 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T10:58:47.989 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T10:58:47.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6980682f0 msgr2=0x7ff69806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:47.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6980682f0 0x7ff69806a7a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40090b0 tx=0x7ff6a401f040 comp rx=0 tx=0).stop 2023-03-03T10:58:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/978976952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 msgr2=0x7ff6b41e5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T10:58:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6b41e5530 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0000ff0 tx=0x7ff6b000ea60 comp rx=0 tx=0).stop 2023-03-03T10:58:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/978976952 shutdown_connections 2023-03-03T10:58:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6b4142110 0x7ff6b40aa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6980682f0 0x7ff69806a7a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6a0004850 0x7ff6b41e5530 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 --2- 172.21.15.7:0/978976952 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6b41dab60 0x7ff6b4141bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T10:58:47.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/978976952 >> 172.21.15.7:0/978976952 conn(0x7ff6b406c110 msgr2=0x7ff6b414a430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T10:58:47.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/978976952 shutdown_connections 2023-03-03T10:58:47.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T10:58:47.988+0000 7ff6bbfcb700 1 -- 172.21.15.7:0/978976952 wait complete. 2023-03-03T10:58:47.992 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T10:58:48.002 INFO:tasks.mds_thrash.fs.[a]:waiting for 94 secs before thrashing 2023-03-03T10:59:06.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:06.730 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:36.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T10:59:36.859 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:00.002 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T10:59:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T11:00:07.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:07.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:22.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:00:22.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.140+0000 7f25fbdba700 1 Processor -- start 2023-03-03T11:00:22.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.140+0000 7f25fbdba700 1 -- start start 2023-03-03T11:00:22.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fbdba700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f25f40c0ee0 0x7f25f40c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:22.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fbdba700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f25f406d240 0x7f25f406d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:22.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fbdba700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f40c2ff0 con 0x7f25f413a160 2023-03-03T11:00:22.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fbdba700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f4158410 con 0x7f25f406d240 2023-03-03T11:00:22.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fbdba700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f4154450 con 0x7f25f40c0ee0 2023-03-03T11:00:22.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25f9b56700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f25f413a160 0x7f25f40c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56632/0 (socket says 172.21.15.7:56632) 2023-03-03T11:00:22.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25f9b56700 1 -- 172.21.15.7:0/3347912321 learned_addr learned my addr 172.21.15.7:0/3347912321 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:00:22.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fa357700 1 --2- 172.21.15.7:0/3347912321 >> v2:172.21.15.7:3300/0 conn(0x7f25f406d240 0x7f25f406d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:22.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25f9355700 1 --2- 172.21.15.7:0/3347912321 >> v2:172.21.15.143:3300/0 conn(0x7f25f40c0ee0 0x7f25f40c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:22.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25f8b54700 1 -- 172.21.15.7:0/3347912321 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617723641 0 0) 0x7f25f0001110 con 0x7f25f413a160 2023-03-03T11:00:22.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25f8b54700 1 -- 172.21.15.7:0/3347912321 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e00035b0 con 0x7f25f413a160 2023-03-03T11:00:22.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fa357700 1 -- 172.21.15.7:0/3347912321 >> v2:172.21.15.143:3300/0 conn(0x7f25f40c0ee0 msgr2=0x7f25f40c1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:00:22.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fa357700 1 --2- 172.21.15.7:0/3347912321 >> v2:172.21.15.143:3300/0 conn(0x7f25f40c0ee0 0x7f25f40c1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:22.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fa357700 1 -- 172.21.15.7:0/3347912321 >> v1:172.21.15.7:6789/0 conn(0x7f25f413a160 legacy=0x7f25f40c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:22.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fa357700 1 -- 172.21.15.7:0/3347912321 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f4157ae0 con 0x7f25f406d240 2023-03-03T11:00:22.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.141+0000 7f25fa357700 1 --2- 172.21.15.7:0/3347912321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f406d240 0x7f25f406d610 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f25f406d140 tx=0x7f25ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7f968730e9ee1d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:00:22.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25f8b54700 1 -- 172.21.15.7:0/3347912321 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec00e070 con 0x7f25f406d240 2023-03-03T11:00:22.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25f8b54700 1 -- 172.21.15.7:0/3347912321 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec00bb20 con 0x7f25f406d240 2023-03-03T11:00:22.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25f8b54700 1 -- 172.21.15.7:0/3347912321 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec012430 con 0x7f25f406d240 2023-03-03T11:00:22.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25fbdba700 1 -- 172.21.15.7:0/3347912321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f406d240 msgr2=0x7f25f406d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:22.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25fbdba700 1 --2- 172.21.15.7:0/3347912321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f406d240 0x7f25f406d610 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f25f406d140 tx=0x7f25ec00b330 comp rx=0 tx=0).stop 2023-03-03T11:00:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25fbdba700 1 -- 172.21.15.7:0/3347912321 shutdown_connections 2023-03-03T11:00:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25fbdba700 1 --2- 172.21.15.7:0/3347912321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f406d240 0x7f25f406d610 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25fbdba700 1 --2- 172.21.15.7:0/3347912321 >> v2:172.21.15.143:3300/0 conn(0x7f25f40c0ee0 0x7f25f40c1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25fbdba700 1 -- 172.21.15.7:0/3347912321 >> 172.21.15.7:0/3347912321 conn(0x7f25f406c110 msgr2=0x7f25f406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:00:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25fbdba700 1 -- 172.21.15.7:0/3347912321 shutdown_connections 2023-03-03T11:00:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.142+0000 7f25fbdba700 1 -- 172.21.15.7:0/3347912321 wait complete. 2023-03-03T11:00:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.143+0000 7f25fbdba700 1 Processor -- start 2023-03-03T11:00:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.143+0000 7f25fbdba700 1 -- start start 2023-03-03T11:00:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fbdba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f406d240 0x7f25f4132be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fbdba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f40c0ee0 0x7f25f4071a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fbdba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f413a160 0x7f25f41ed5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fbdba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4134d70 con 0x7f25f413a160 2023-03-03T11:00:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fbdba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4135050 con 0x7f25f40c0ee0 2023-03-03T11:00:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fbdba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4135330 con 0x7f25f406d240 2023-03-03T11:00:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fa357700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f413a160 0x7f25f41ed5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fa357700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f413a160 0x7f25f41ed5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34430/0 (socket says 172.21.15.7:34430) 2023-03-03T11:00:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fa357700 1 -- 172.21.15.7:0/3175518147 learned_addr learned my addr 172.21.15.7:0/3175518147 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:00:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25f9b56700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f406d240 0x7f25f4132be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25f9355700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f40c0ee0 0x7f25f4071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fa357700 1 -- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f406d240 msgr2=0x7f25f4132be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.144+0000 7f25fa357700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f406d240 0x7f25f4132be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.145+0000 7f25fa357700 1 -- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f40c0ee0 msgr2=0x7f25f4071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.145+0000 7f25fa357700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f40c0ee0 0x7f25f4071a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:22.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.145+0000 7f25fa357700 1 -- 172.21.15.7:0/3175518147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec009a70 con 0x7f25f413a160 2023-03-03T11:00:22.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.145+0000 7f25fa357700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f413a160 0x7f25f41ed5a0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f25ec014040 tx=0x7f25ec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:00:22.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.145+0000 7f25ea7fc700 1 -- 172.21.15.7:0/3175518147 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec012a70 con 0x7f25f413a160 2023-03-03T11:00:22.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.145+0000 7f25ea7fc700 1 -- 172.21.15.7:0/3175518147 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec020070 con 0x7f25f413a160 2023-03-03T11:00:22.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.145+0000 7f25fbdba700 1 -- 172.21.15.7:0/3175518147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f41580a0 con 0x7f25f413a160 2023-03-03T11:00:22.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.145+0000 7f25ea7fc700 1 -- 172.21.15.7:0/3175518147 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec012e30 con 0x7f25f413a160 2023-03-03T11:00:22.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.145+0000 7f25fbdba700 1 -- 172.21.15.7:0/3175518147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f4157dc0 con 0x7f25f413a160 2023-03-03T11:00:22.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.146+0000 7f25fbdba700 1 -- 172.21.15.7:0/3175518147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f41588a0 con 0x7f25f413a160 2023-03-03T11:00:22.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.146+0000 7f25ea7fc700 1 -- 172.21.15.7:0/3175518147 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f25ec004040 con 0x7f25f413a160 2023-03-03T11:00:22.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.147+0000 7f25ea7fc700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25d8068320 0x7f25d806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.147+0000 7f25ea7fc700 1 -- 172.21.15.7:0/3175518147 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f25ec022070 con 0x7f25f413a160 2023-03-03T11:00:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.147+0000 7f25f9b56700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25d8068320 0x7f25d806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.148+0000 7f25f9b56700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25d8068320 0x7f25d806a7d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f25f0001270 tx=0x7f25f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:00:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.151+0000 7f25ea7fc700 1 -- 172.21.15.7:0/3175518147 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f25ec0549b0 con 0x7f25f413a160 2023-03-03T11:00:22.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.389+0000 7f25fbdba700 1 -- 172.21.15.7:0/3175518147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25f41ecee0 con 0x7f25f413a160 2023-03-03T11:00:22.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.389+0000 7f25ea7fc700 1 -- 172.21.15.7:0/3175518147 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f25f41ecee0 con 0x7f25f413a160 2023-03-03T11:00:22.391 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:00:22.392 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:00:22.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 -- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25d8068320 msgr2=0x7f25d806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25d8068320 0x7f25d806a7d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f25f0001270 tx=0x7f25f0009040 comp rx=0 tx=0).stop 2023-03-03T11:00:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 -- 172.21.15.7:0/3175518147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f413a160 msgr2=0x7f25f41ed5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f413a160 0x7f25f41ed5a0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f25ec014040 tx=0x7f25ec004ad0 comp rx=0 tx=0).stop 2023-03-03T11:00:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 -- 172.21.15.7:0/3175518147 shutdown_connections 2023-03-03T11:00:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25d8068320 0x7f25d806a7d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f413a160 0x7f25f41ed5a0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f406d240 0x7f25f4132be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 --2- 172.21.15.7:0/3175518147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f40c0ee0 0x7f25f4071a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 -- 172.21.15.7:0/3175518147 >> 172.21.15.7:0/3175518147 conn(0x7f25f406c110 msgr2=0x7f25f406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:00:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 -- 172.21.15.7:0/3175518147 shutdown_connections 2023-03-03T11:00:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:22.391+0000 7f25fbdba700 1 -- 172.21.15.7:0/3175518147 wait complete. 2023-03-03T11:00:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:00:22.405 INFO:tasks.mds_thrash.fs.[a]:waiting for 99 secs before thrashing 2023-03-03T11:00:35.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:00:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.103+0000 7f0897af5700 1 Processor -- start 2023-03-03T11:00:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.103+0000 7f0897af5700 1 -- start start 2023-03-03T11:00:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f0897af5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0890154f10 0x7f0890158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f0897af5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0890151ef0 con 0x7f0890151b20 2023-03-03T11:00:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f0897af5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0890157b20 con 0x7f08900aa010 2023-03-03T11:00:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f0897af5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0890155410 con 0x7f0890154f10 2023-03-03T11:00:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f0895891700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0890151b20 0x7f0890071820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56654/0 (socket says 172.21.15.7:56654) 2023-03-03T11:00:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f0895891700 1 -- 172.21.15.7:0/3146351500 learned_addr learned my addr 172.21.15.7:0/3146351500 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:00:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f0895090700 1 --2- 172.21.15.7:0/3146351500 >> v2:172.21.15.143:3300/0 conn(0x7f0890154f10 0x7f0890158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174948337 0 0) 0x7f0890151ef0 con 0x7f0890151b20 2023-03-03T11:00:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f087c0035b0 con 0x7f0890151b20 2023-03-03T11:00:36.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1119970029 0 0) 0x7f087c0035b0 con 0x7f0890151b20 2023-03-03T11:00:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.104+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 >> v2:172.21.15.143:3300/0 conn(0x7f0890154f10 msgr2=0x7f0890158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:00:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 --2- 172.21.15.7:0/3146351500 >> v2:172.21.15.143:3300/0 conn(0x7f0890154f10 0x7f0890158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 >> v1:172.21.15.143:6789/0 conn(0x7f08900aa010 legacy=0x7f089015ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089006f640 con 0x7f0890151b20 2023-03-03T11:00:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f088c001670 con 0x7f0890151b20 2023-03-03T11:00:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 >> v1:172.21.15.7:6789/0 conn(0x7f0890151b20 legacy=0x7f0890071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f087c003a70 0x7f087c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c004850 0x7f087c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f087c005200 0x7f087c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0890155410 con 0x7f087c005200 2023-03-03T11:00:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c00c0e0 con 0x7f087c004850 2023-03-03T11:00:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c00c2a0 con 0x7f087c003a70 2023-03-03T11:00:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f0896092700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f087c005200 0x7f087c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f0895090700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c004850 0x7f087c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f0895891700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f087c003a70 0x7f087c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f0896092700 1 -- 172.21.15.7:0/3146351500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f087c003a70 msgr2=0x7f087c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:00:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f0896092700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f087c003a70 0x7f087c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f0896092700 1 -- 172.21.15.7:0/3146351500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c004850 msgr2=0x7f087c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:00:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f0896092700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c004850 0x7f087c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f0896092700 1 -- 172.21.15.7:0/3146351500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c0035b0 con 0x7f087c005200 2023-03-03T11:00:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f0896092700 1 -- 172.21.15.7:0/3146351500 reap_dead start 2023-03-03T11:00:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f0896092700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f087c005200 0x7f087c00ba10 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f088800c190 tx=0x7f087c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437e4727a0eb13bf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:00:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.105+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888001db0 con 0x7f087c005200 2023-03-03T11:00:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.106+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088800ccf0 con 0x7f087c005200 2023-03-03T11:00:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.106+0000 7f089488f700 1 -- 172.21.15.7:0/3146351500 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888003c40 con 0x7f087c005200 2023-03-03T11:00:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.106+0000 7f0897af5700 1 -- 172.21.15.7:0/3146351500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f087c005200 msgr2=0x7f087c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.106+0000 7f0897af5700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f087c005200 0x7f087c00ba10 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f088800c190 tx=0x7f087c005880 comp rx=0 tx=0).stop 2023-03-03T11:00:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.106+0000 7f0897af5700 1 -- 172.21.15.7:0/3146351500 shutdown_connections 2023-03-03T11:00:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.106+0000 7f0897af5700 1 --2- 172.21.15.7:0/3146351500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f087c005200 0x7f087c00ba10 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.106+0000 7f0897af5700 1 -- 172.21.15.7:0/3146351500 >> 172.21.15.7:0/3146351500 conn(0x7f08900c8900 msgr2=0x7f08900c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:00:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.106+0000 7f0897af5700 1 -- 172.21.15.7:0/3146351500 shutdown_connections 2023-03-03T11:00:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.106+0000 7f0897af5700 1 -- 172.21.15.7:0/3146351500 wait complete. 2023-03-03T11:00:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.107+0000 7f0897af5700 1 Processor -- start 2023-03-03T11:00:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.107+0000 7f0897af5700 1 -- start start 2023-03-03T11:00:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0897af5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c005200 0x7f0890152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0897af5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0890153010 0x7f0890153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0897af5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0890147dd0 0x7f0890148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0897af5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08901539a0 con 0x7f0890153010 2023-03-03T11:00:36.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0897af5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08900c73f0 con 0x7f087c005200 2023-03-03T11:00:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0897af5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08900c76b0 con 0x7f0890147dd0 2023-03-03T11:00:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0895090700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0890153010 0x7f0890153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0895090700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0890153010 0x7f0890153460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34456/0 (socket says 172.21.15.7:34456) 2023-03-03T11:00:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0895090700 1 -- 172.21.15.7:0/1351862692 learned_addr learned my addr 172.21.15.7:0/1351862692 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:00:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0896092700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0890147dd0 0x7f0890148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0895891700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c005200 0x7f0890152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:36.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0895090700 1 -- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0890147dd0 msgr2=0x7f0890148250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:36.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0895090700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0890147dd0 0x7f0890148250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:36.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0895090700 1 -- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c005200 msgr2=0x7f0890152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:36.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0895090700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c005200 0x7f0890152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:36.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.108+0000 7f0895090700 1 -- 172.21.15.7:0/1351862692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089006fc00 con 0x7f0890153010 2023-03-03T11:00:36.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.109+0000 7f0895090700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0890153010 0x7f0890153460 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0880011940 tx=0x7f0880011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:00:36.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.109+0000 7f08867fc700 1 -- 172.21.15.7:0/1351862692 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880004240 con 0x7f0890153010 2023-03-03T11:00:36.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.109+0000 7f08867fc700 1 -- 172.21.15.7:0/1351862692 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08800043a0 con 0x7f0890153010 2023-03-03T11:00:36.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.109+0000 7f0897af5700 1 -- 172.21.15.7:0/1351862692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089006f920 con 0x7f0890153010 2023-03-03T11:00:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.109+0000 7f08867fc700 1 -- 172.21.15.7:0/1351862692 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088000f950 con 0x7f0890153010 2023-03-03T11:00:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.109+0000 7f0897af5700 1 -- 172.21.15.7:0/1351862692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08900aae90 con 0x7f0890153010 2023-03-03T11:00:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.110+0000 7f0897af5700 1 -- 172.21.15.7:0/1351862692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0890074bb0 con 0x7f0890153010 2023-03-03T11:00:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.110+0000 7f08867fc700 1 -- 172.21.15.7:0/1351862692 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0880025070 con 0x7f0890153010 2023-03-03T11:00:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.111+0000 7f08867fc700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 0x7f087406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:00:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.111+0000 7f08867fc700 1 -- 172.21.15.7:0/1351862692 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0880086580 con 0x7f0890153010 2023-03-03T11:00:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.111+0000 7f0895891700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 0x7f087406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:00:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.112+0000 7f0895891700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 0x7f087406a7a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f088c001050 tx=0x7f088c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:00:36.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.115+0000 7f08867fc700 1 -- 172.21.15.7:0/1351862692 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0880052920 con 0x7f0890153010 2023-03-03T11:00:36.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.356+0000 7f0897af5700 1 -- 172.21.15.7:0/1351862692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08900a9df0 con 0x7f0890153010 2023-03-03T11:00:36.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.356+0000 7f08867fc700 1 -- 172.21.15.7:0/1351862692 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0880055f40 con 0x7f0890153010 2023-03-03T11:00:36.359 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:00:36.359 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:00:36.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 -- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 msgr2=0x7f087406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:36.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 0x7f087406a7a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f088c001050 tx=0x7f088c009040 comp rx=0 tx=0).stop 2023-03-03T11:00:36.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 -- 172.21.15.7:0/1351862692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0890153010 msgr2=0x7f0890153460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:00:36.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0890153010 0x7f0890153460 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0880011940 tx=0x7f0880011d00 comp rx=0 tx=0).stop 2023-03-03T11:00:36.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 -- 172.21.15.7:0/1351862692 shutdown_connections 2023-03-03T11:00:36.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0890147dd0 0x7f0890148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:36.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f087c005200 0x7f0890152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 0x7f087406a7a0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 --2- 172.21.15.7:0/1351862692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0890153010 0x7f0890153460 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:00:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 -- 172.21.15.7:0/1351862692 >> 172.21.15.7:0/1351862692 conn(0x7f08900c8900 msgr2=0x7f089014acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:00:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 -- 172.21.15.7:0/1351862692 shutdown_connections 2023-03-03T11:00:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:00:36.358+0000 7f0897af5700 1 -- 172.21.15.7:0/1351862692 wait complete. 2023-03-03T11:00:36.362 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:00:36.372 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2023-03-03T11:00:37.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:00:37.169 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:03.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:01:03.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.511+0000 7fd541cf4700 1 Processor -- start 2023-03-03T11:01:03.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.512+0000 7fd541cf4700 1 -- start start 2023-03-03T11:01:03.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.512+0000 7fd541cf4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd53c06c560 0x7fd53c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:03.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.512+0000 7fd541cf4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd53c157fe0 0x7fd53c158460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:03.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.512+0000 7fd541cf4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53c157b50 con 0x7fd53c155090 2023-03-03T11:01:03.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.512+0000 7fd541cf4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c06c240 con 0x7fd53c157fe0 2023-03-03T11:01:03.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.512+0000 7fd541cf4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c0c33b0 con 0x7fd53c06c560 2023-03-03T11:01:03.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.512+0000 7fd53bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd53c157fe0 0x7fd53c158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:03.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.512+0000 7fd53b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd53c155090 0x7fd53c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56690/0 (socket says 172.21.15.7:56690) 2023-03-03T11:01:03.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53b7fe700 1 -- 172.21.15.7:0/568016612 learned_addr learned my addr 172.21.15.7:0/568016612 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:01:03.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53affd700 1 --2- 172.21.15.7:0/568016612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd53c06c560 0x7fd53c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:03.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53bfff700 1 -- 172.21.15.7:0/568016612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd53c06c560 msgr2=0x7fd53c06d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:01:03.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53bfff700 1 --2- 172.21.15.7:0/568016612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd53c06c560 0x7fd53c06d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53bfff700 1 -- 172.21.15.7:0/568016612 >> v1:172.21.15.7:6789/0 conn(0x7fd53c155090 legacy=0x7fd53c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:03.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53bfff700 1 -- 172.21.15.7:0/568016612 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c0c3690 con 0x7fd53c157fe0 2023-03-03T11:01:03.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53a7fc700 1 -- 172.21.15.7:0/568016612 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517813094 0 0) 0x7fd53c157b50 con 0x7fd53c155090 2023-03-03T11:01:03.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53bfff700 1 --2- 172.21.15.7:0/568016612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 0x7fd53c158460 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd534009d90 tx=0x7fd53400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b6838c62155033 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:01:03.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53a7fc700 1 -- 172.21.15.7:0/568016612 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53400e070 con 0x7fd53c157fe0 2023-03-03T11:01:03.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53a7fc700 1 -- 172.21.15.7:0/568016612 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53400bbd0 con 0x7fd53c157fe0 2023-03-03T11:01:03.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.513+0000 7fd53a7fc700 1 -- 172.21.15.7:0/568016612 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534012430 con 0x7fd53c157fe0 2023-03-03T11:01:03.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.514+0000 7fd541cf4700 1 -- 172.21.15.7:0/568016612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 msgr2=0x7fd53c158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:03.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.514+0000 7fd541cf4700 1 --2- 172.21.15.7:0/568016612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 0x7fd53c158460 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd534009d90 tx=0x7fd53400b330 comp rx=0 tx=0).stop 2023-03-03T11:01:03.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.514+0000 7fd541cf4700 1 -- 172.21.15.7:0/568016612 shutdown_connections 2023-03-03T11:01:03.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.514+0000 7fd541cf4700 1 --2- 172.21.15.7:0/568016612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 0x7fd53c158460 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.514+0000 7fd541cf4700 1 --2- 172.21.15.7:0/568016612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd53c06c560 0x7fd53c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.514+0000 7fd541cf4700 1 -- 172.21.15.7:0/568016612 >> 172.21.15.7:0/568016612 conn(0x7fd53c0a94d0 msgr2=0x7fd53c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:01:03.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.514+0000 7fd541cf4700 1 -- 172.21.15.7:0/568016612 shutdown_connections 2023-03-03T11:01:03.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.514+0000 7fd541cf4700 1 -- 172.21.15.7:0/568016612 wait complete. 2023-03-03T11:01:03.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.515+0000 7fd541cf4700 1 Processor -- start 2023-03-03T11:01:03.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.515+0000 7fd541cf4700 1 -- start start 2023-03-03T11:01:03.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd541cf4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd53c06c560 0x7fd53c1edbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:03.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd541cf4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd53c155090 0x7fd53c1ea600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:03.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd541cf4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 0x7fd53c0c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:03.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd541cf4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c14e320 con 0x7fd53c157fe0 2023-03-03T11:01:03.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd541cf4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c1ed170 con 0x7fd53c06c560 2023-03-03T11:01:03.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd541cf4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c1ee100 con 0x7fd53c155090 2023-03-03T11:01:03.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd53bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 0x7fd53c0c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:03.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd53bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 0x7fd53c0c3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34488/0 (socket says 172.21.15.7:34488) 2023-03-03T11:01:03.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd53bfff700 1 -- 172.21.15.7:0/691367925 learned_addr learned my addr 172.21.15.7:0/691367925 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:01:03.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd53affd700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd53c155090 0x7fd53c1ea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:03.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd53b7fe700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd53c06c560 0x7fd53c1edbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:03.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd53bfff700 1 -- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd53c155090 msgr2=0x7fd53c1ea600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:03.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd53bfff700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd53c155090 0x7fd53c1ea600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd53bfff700 1 -- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd53c06c560 msgr2=0x7fd53c1edbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:03.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd53bfff700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd53c06c560 0x7fd53c1edbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.516+0000 7fd53bfff700 1 -- 172.21.15.7:0/691367925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd534009a70 con 0x7fd53c157fe0 2023-03-03T11:01:03.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.517+0000 7fd53bfff700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 0x7fd53c0c3fe0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fd53c06bb00 tx=0x7fd534004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:01:03.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.517+0000 7fd523fff700 1 -- 172.21.15.7:0/691367925 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534012430 con 0x7fd53c157fe0 2023-03-03T11:01:03.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.517+0000 7fd523fff700 1 -- 172.21.15.7:0/691367925 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd534020070 con 0x7fd53c157fe0 2023-03-03T11:01:03.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.517+0000 7fd523fff700 1 -- 172.21.15.7:0/691367925 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534004280 con 0x7fd53c157fe0 2023-03-03T11:01:03.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.517+0000 7fd541cf4700 1 -- 172.21.15.7:0/691367925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c1589a0 con 0x7fd53c157fe0 2023-03-03T11:01:03.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.517+0000 7fd541cf4700 1 -- 172.21.15.7:0/691367925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53c0c3970 con 0x7fd53c157fe0 2023-03-03T11:01:03.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.518+0000 7fd541cf4700 1 -- 172.21.15.7:0/691367925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53c074bb0 con 0x7fd53c157fe0 2023-03-03T11:01:03.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.518+0000 7fd523fff700 1 -- 172.21.15.7:0/691367925 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd53402e070 con 0x7fd53c157fe0 2023-03-03T11:01:03.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.519+0000 7fd523fff700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5280682f0 0x7fd52806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:01:03.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.519+0000 7fd523fff700 1 -- 172.21.15.7:0/691367925 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd534022070 con 0x7fd53c157fe0 2023-03-03T11:01:03.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.519+0000 7fd53b7fe700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5280682f0 0x7fd52806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:01:03.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.519+0000 7fd53b7fe700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5280682f0 0x7fd52806a7a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd52c007fd0 tx=0x7fd52c007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:01:03.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.523+0000 7fd523fff700 1 -- 172.21.15.7:0/691367925 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd5340541e0 con 0x7fd53c157fe0 2023-03-03T11:01:03.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.762+0000 7fd541cf4700 1 -- 172.21.15.7:0/691367925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd53c154560 con 0x7fd53c157fe0 2023-03-03T11:01:03.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.763+0000 7fd523fff700 1 -- 172.21.15.7:0/691367925 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd534087070 con 0x7fd53c157fe0 2023-03-03T11:01:03.765 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:01:03.765 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:01:03.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 -- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5280682f0 msgr2=0x7fd52806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:03.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5280682f0 0x7fd52806a7a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd52c007fd0 tx=0x7fd52c007de0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 -- 172.21.15.7:0/691367925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 msgr2=0x7fd53c0c3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:01:03.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 0x7fd53c0c3fe0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fd53c06bb00 tx=0x7fd534004ad0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 -- 172.21.15.7:0/691367925 shutdown_connections 2023-03-03T11:01:03.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5280682f0 0x7fd52806a7a0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd53c157fe0 0x7fd53c0c3fe0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd53c06c560 0x7fd53c1edbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 --2- 172.21.15.7:0/691367925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd53c155090 0x7fd53c1ea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:01:03.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 -- 172.21.15.7:0/691367925 >> 172.21.15.7:0/691367925 conn(0x7fd53c0a94d0 msgr2=0x7fd53c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:01:03.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 -- 172.21.15.7:0/691367925 shutdown_connections 2023-03-03T11:01:03.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:01:03.765+0000 7fd541cf4700 1 -- 172.21.15.7:0/691367925 wait complete. 2023-03-03T11:01:03.769 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:01:03.779 INFO:tasks.mds_thrash.fs.[b]:waiting for 84 secs before thrashing 2023-03-03T11:01:07.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:07.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:37.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:01:37.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:01.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:02:01.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.544+0000 7f413c188700 1 Processor -- start 2023-03-03T11:02:01.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.544+0000 7f413c188700 1 -- start start 2023-03-03T11:02:01.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f413c188700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f41340aad20 0x7f4134157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:01.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f413c188700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41340a9c20 con 0x7f413414b0d0 2023-03-03T11:02:01.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f413c188700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4134154f10 con 0x7f41340aa030 2023-03-03T11:02:01.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f413c188700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41340ac3c0 con 0x7f41340aad20 2023-03-03T11:02:01.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f4139723700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f41340aad20 0x7f4134157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:01.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f4139f24700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f413414b0d0 0x7f4134071800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56734/0 (socket says 172.21.15.7:56734) 2023-03-03T11:02:01.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f4139f24700 1 -- 172.21.15.7:0/237838020 learned_addr learned my addr 172.21.15.7:0/237838020 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:02:01.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f4139723700 1 -- 172.21.15.7:0/237838020 >> v1:172.21.15.143:6789/0 conn(0x7f41340aa030 legacy=0x7f4134158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:02:01.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f4138f22700 1 -- 172.21.15.7:0/237838020 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918868060 0 0) 0x7f41340a9c20 con 0x7f413414b0d0 2023-03-03T11:02:01.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f4139723700 1 -- 172.21.15.7:0/237838020 >> v1:172.21.15.7:6789/0 conn(0x7f413414b0d0 legacy=0x7f4134071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:01.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f4139723700 1 -- 172.21.15.7:0/237838020 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413406fc70 con 0x7f41340aad20 2023-03-03T11:02:01.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.545+0000 7f4139723700 1 --2- 172.21.15.7:0/237838020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 0x7f4134157df0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f4124009840 tx=0x7f4124009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6e72b7baa4d746 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:01.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.546+0000 7f4138f22700 1 -- 172.21.15.7:0/237838020 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124011830 con 0x7f41340aad20 2023-03-03T11:02:01.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.546+0000 7f4138f22700 1 -- 172.21.15.7:0/237838020 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4124011990 con 0x7f41340aad20 2023-03-03T11:02:01.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.546+0000 7f4138f22700 1 -- 172.21.15.7:0/237838020 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124011d20 con 0x7f41340aad20 2023-03-03T11:02:01.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.546+0000 7f413c188700 1 -- 172.21.15.7:0/237838020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 msgr2=0x7f4134157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:01.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.546+0000 7f413c188700 1 --2- 172.21.15.7:0/237838020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 0x7f4134157df0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f4124009840 tx=0x7f4124009b50 comp rx=0 tx=0).stop 2023-03-03T11:02:01.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.546+0000 7f413c188700 1 -- 172.21.15.7:0/237838020 shutdown_connections 2023-03-03T11:02:01.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.546+0000 7f413c188700 1 --2- 172.21.15.7:0/237838020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 0x7f4134157df0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:01.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.546+0000 7f413c188700 1 -- 172.21.15.7:0/237838020 >> 172.21.15.7:0/237838020 conn(0x7f41340c8900 msgr2=0x7f41340c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:02:01.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.546+0000 7f413c188700 1 -- 172.21.15.7:0/237838020 shutdown_connections 2023-03-03T11:02:01.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.546+0000 7f413c188700 1 -- 172.21.15.7:0/237838020 wait complete. 2023-03-03T11:02:01.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f413c188700 1 Processor -- start 2023-03-03T11:02:01.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f413c188700 1 -- start start 2023-03-03T11:02:01.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f413c188700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41340aa030 0x7f41341ed740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:01.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f413c188700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 0x7f41341edc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:01.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f413c188700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413414b0d0 0x7f41341432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:01.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f413c188700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4134153760 con 0x7f41340aad20 2023-03-03T11:02:01.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f413c188700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41341e1c30 con 0x7f41340aa030 2023-03-03T11:02:01.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f413c188700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4134149710 con 0x7f413414b0d0 2023-03-03T11:02:01.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f4139723700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 0x7f41341edc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:01.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f4139723700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 0x7f41341edc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34528/0 (socket says 172.21.15.7:34528) 2023-03-03T11:02:01.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.548+0000 7f4139723700 1 -- 172.21.15.7:0/2477883131 learned_addr learned my addr 172.21.15.7:0/2477883131 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:02:01.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f4139f24700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41340aa030 0x7f41341ed740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:01.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f413a725700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413414b0d0 0x7f41341432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:01.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f4139723700 1 -- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413414b0d0 msgr2=0x7f41341432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:01.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f4139723700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413414b0d0 0x7f41341432f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:01.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f4139723700 1 -- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41340aa030 msgr2=0x7f41341ed740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:01.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f4139723700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41340aa030 0x7f41341ed740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:01.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f4139723700 1 -- 172.21.15.7:0/2477883131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240094a0 con 0x7f41340aad20 2023-03-03T11:02:01.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f4139723700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 0x7f41341edc80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f41240090b0 tx=0x7f4124004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:01.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f412a7fc700 1 -- 172.21.15.7:0/2477883131 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412401a900 con 0x7f41340aad20 2023-03-03T11:02:01.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f412a7fc700 1 -- 172.21.15.7:0/2477883131 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412401aa60 con 0x7f41340aad20 2023-03-03T11:02:01.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f413c188700 1 -- 172.21.15.7:0/2477883131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413406f6b0 con 0x7f41340aad20 2023-03-03T11:02:01.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f412a7fc700 1 -- 172.21.15.7:0/2477883131 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412401add0 con 0x7f41340aad20 2023-03-03T11:02:01.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.549+0000 7f413c188700 1 -- 172.21.15.7:0/2477883131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4134155160 con 0x7f41340aad20 2023-03-03T11:02:01.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.550+0000 7f413c188700 1 -- 172.21.15.7:0/2477883131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413406f990 con 0x7f41340aad20 2023-03-03T11:02:01.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.551+0000 7f412a7fc700 1 -- 172.21.15.7:0/2477883131 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f412402b030 con 0x7f41340aad20 2023-03-03T11:02:01.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.551+0000 7f412a7fc700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4118068320 0x7f411806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:01.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.551+0000 7f412a7fc700 1 -- 172.21.15.7:0/2477883131 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4124087530 con 0x7f41340aad20 2023-03-03T11:02:01.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.551+0000 7f4139f24700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4118068320 0x7f411806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:01.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.552+0000 7f4139f24700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4118068320 0x7f411806a7d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4130007fd0 tx=0x7f4130007c20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:01.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.555+0000 7f412a7fc700 1 -- 172.21.15.7:0/2477883131 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f41240538d0 con 0x7f41340aad20 2023-03-03T11:02:01.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.793+0000 7f413c188700 1 -- 172.21.15.7:0/2477883131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41340c82a0 con 0x7f41340aad20 2023-03-03T11:02:01.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.793+0000 7f412a7fc700 1 -- 172.21.15.7:0/2477883131 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4124056ef0 con 0x7f41340aad20 2023-03-03T11:02:01.796 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:02:01.796 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:02:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 -- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4118068320 msgr2=0x7f411806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4118068320 0x7f411806a7d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4130007fd0 tx=0x7f4130007c20 comp rx=0 tx=0).stop 2023-03-03T11:02:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 -- 172.21.15.7:0/2477883131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 msgr2=0x7f41341edc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 0x7f41341edc80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f41240090b0 tx=0x7f4124004750 comp rx=0 tx=0).stop 2023-03-03T11:02:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 -- 172.21.15.7:0/2477883131 shutdown_connections 2023-03-03T11:02:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413414b0d0 0x7f41341432f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41340aa030 0x7f41341ed740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4118068320 0x7f411806a7d0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 --2- 172.21.15.7:0/2477883131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41340aad20 0x7f41341edc80 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 -- 172.21.15.7:0/2477883131 >> 172.21.15.7:0/2477883131 conn(0x7f41340c8900 msgr2=0x7f41341581d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:02:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 -- 172.21.15.7:0/2477883131 shutdown_connections 2023-03-03T11:02:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:01.796+0000 7f413c188700 1 -- 172.21.15.7:0/2477883131 wait complete. 2023-03-03T11:02:01.800 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:02:01.810 INFO:tasks.mds_thrash.fs.[a]:waiting for 70 secs before thrashing 2023-03-03T11:02:07.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:07.559 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:27.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:02:27.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.915+0000 7f03cceec700 1 Processor -- start 2023-03-03T11:02:27.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.915+0000 7f03cceec700 1 -- start start 2023-03-03T11:02:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03cceec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f03c80c2e30 0x7f03c8158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03cceec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03c806c6c0 0x7f03c80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03cceec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c8157ff0 con 0x7f03c8151cd0 2023-03-03T11:02:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03cceec700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c806d640 con 0x7f03c80c2e30 2023-03-03T11:02:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03cceec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c806cda0 con 0x7f03c806c6c0 2023-03-03T11:02:27.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03c5d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f03c80c2e30 0x7f03c8158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:27.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03c5d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f03c80c2e30 0x7f03c8158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34550/0 (socket says 172.21.15.7:34550) 2023-03-03T11:02:27.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03c5d9b700 1 -- 172.21.15.7:0/2034206733 learned_addr learned my addr 172.21.15.7:0/2034206733 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:02:27.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03c6d9d700 1 --2- 172.21.15.7:0/2034206733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03c806c6c0 0x7f03c80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:27.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03c5d9b700 1 -- 172.21.15.7:0/2034206733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03c806c6c0 msgr2=0x7f03c80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:02:27.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03c5d9b700 1 --2- 172.21.15.7:0/2034206733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03c806c6c0 0x7f03c80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:27.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03c5d9b700 1 -- 172.21.15.7:0/2034206733 >> v1:172.21.15.143:6789/0 conn(0x7f03c8151cd0 legacy=0x7f03c8154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:27.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03c5d9b700 1 -- 172.21.15.7:0/2034206733 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c806d080 con 0x7f03c80c2e30 2023-03-03T11:02:27.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.916+0000 7f03c5d9b700 1 --2- 172.21.15.7:0/2034206733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c80c2e30 0x7f03c8158480 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0098f0 tx=0x7f03bc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc48715efef730c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:27.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03c559a700 1 -- 172.21.15.7:0/2034206733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc011830 con 0x7f03c80c2e30 2023-03-03T11:02:27.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03c559a700 1 -- 172.21.15.7:0/2034206733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc011990 con 0x7f03c80c2e30 2023-03-03T11:02:27.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03c559a700 1 -- 172.21.15.7:0/2034206733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc011d20 con 0x7f03c80c2e30 2023-03-03T11:02:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03cceec700 1 -- 172.21.15.7:0/2034206733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c80c2e30 msgr2=0x7f03c8158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03cceec700 1 --2- 172.21.15.7:0/2034206733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c80c2e30 0x7f03c8158480 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0098f0 tx=0x7f03bc009c00 comp rx=0 tx=0).stop 2023-03-03T11:02:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03cceec700 1 -- 172.21.15.7:0/2034206733 shutdown_connections 2023-03-03T11:02:27.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03cceec700 1 --2- 172.21.15.7:0/2034206733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03c806c6c0 0x7f03c80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03cceec700 1 --2- 172.21.15.7:0/2034206733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c80c2e30 0x7f03c8158480 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03cceec700 1 -- 172.21.15.7:0/2034206733 >> 172.21.15.7:0/2034206733 conn(0x7f03c80a94d0 msgr2=0x7f03c80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:02:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03cceec700 1 -- 172.21.15.7:0/2034206733 shutdown_connections 2023-03-03T11:02:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.917+0000 7f03cceec700 1 -- 172.21.15.7:0/2034206733 wait complete. 2023-03-03T11:02:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.919+0000 7f03cceec700 1 Processor -- start 2023-03-03T11:02:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.919+0000 7f03cceec700 1 -- start start 2023-03-03T11:02:27.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03cceec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03c806c6c0 0x7f03c81ea4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03cceec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03c80c2e30 0x7f03c80c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03cceec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c8151cd0 0x7f03c8155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03cceec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80aa690 con 0x7f03c8151cd0 2023-03-03T11:02:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03cceec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8155a30 con 0x7f03c80c2e30 2023-03-03T11:02:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03cceec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c81eab00 con 0x7f03c806c6c0 2023-03-03T11:02:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03c6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c8151cd0 0x7f03c8155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03c6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c8151cd0 0x7f03c8155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34558/0 (socket says 172.21.15.7:34558) 2023-03-03T11:02:27.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03c6d9d700 1 -- 172.21.15.7:0/1359260436 learned_addr learned my addr 172.21.15.7:0/1359260436 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:02:27.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03c5d9b700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03c80c2e30 0x7f03c80c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:27.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.920+0000 7f03c659c700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03c806c6c0 0x7f03c81ea4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:27.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03c6d9d700 1 -- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03c806c6c0 msgr2=0x7f03c81ea4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:27.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03c6d9d700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03c806c6c0 0x7f03c81ea4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:27.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03c6d9d700 1 -- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03c80c2e30 msgr2=0x7f03c80c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:27.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03c6d9d700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03c80c2e30 0x7f03c80c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:27.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03c6d9d700 1 -- 172.21.15.7:0/1359260436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc009550 con 0x7f03c8151cd0 2023-03-03T11:02:27.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03c6d9d700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c8151cd0 0x7f03c8155ec0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f03c000efc0 tx=0x7f03c000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:27.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03b6ffd700 1 -- 172.21.15.7:0/1359260436 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c000cad0 con 0x7f03c8151cd0 2023-03-03T11:02:27.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03b6ffd700 1 -- 172.21.15.7:0/1359260436 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c000f040 con 0x7f03c8151cd0 2023-03-03T11:02:27.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03cceec700 1 -- 172.21.15.7:0/1359260436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c8157c80 con 0x7f03c8151cd0 2023-03-03T11:02:27.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03b6ffd700 1 -- 172.21.15.7:0/1359260436 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c000ce70 con 0x7f03c8151cd0 2023-03-03T11:02:27.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.921+0000 7f03cceec700 1 -- 172.21.15.7:0/1359260436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c806d360 con 0x7f03c8151cd0 2023-03-03T11:02:27.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.922+0000 7f03cceec700 1 -- 172.21.15.7:0/1359260436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c81589c0 con 0x7f03c8151cd0 2023-03-03T11:02:27.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.922+0000 7f03b6ffd700 1 -- 172.21.15.7:0/1359260436 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f03c0014480 con 0x7f03c8151cd0 2023-03-03T11:02:27.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.923+0000 7f03b6ffd700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03a80682f0 0x7f03a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:27.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.923+0000 7f03b6ffd700 1 -- 172.21.15.7:0/1359260436 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f03c0086cf0 con 0x7f03c8151cd0 2023-03-03T11:02:27.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.923+0000 7f03c659c700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03a80682f0 0x7f03a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:27.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.924+0000 7f03c659c700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03a80682f0 0x7f03a806a7a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f03b8007fd0 tx=0x7f03b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:27.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:27.927+0000 7f03b6ffd700 1 -- 172.21.15.7:0/1359260436 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f03c0053030 con 0x7f03c8151cd0 2023-03-03T11:02:28.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.164+0000 7f03cceec700 1 -- 172.21.15.7:0/1359260436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03c8155ba0 con 0x7f03c8151cd0 2023-03-03T11:02:28.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.165+0000 7f03b6ffd700 1 -- 172.21.15.7:0/1359260436 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f03c0056650 con 0x7f03c8151cd0 2023-03-03T11:02:28.168 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:02:28.169 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:02:28.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 -- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03a80682f0 msgr2=0x7f03a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03a80682f0 0x7f03a806a7a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f03b8007fd0 tx=0x7f03b8009040 comp rx=0 tx=0).stop 2023-03-03T11:02:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 -- 172.21.15.7:0/1359260436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c8151cd0 msgr2=0x7f03c8155ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c8151cd0 0x7f03c8155ec0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f03c000efc0 tx=0x7f03c000c0c0 comp rx=0 tx=0).stop 2023-03-03T11:02:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 -- 172.21.15.7:0/1359260436 shutdown_connections 2023-03-03T11:02:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03a80682f0 0x7f03a806a7a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03c8151cd0 0x7f03c8155ec0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:28.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03c806c6c0 0x7f03c81ea4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 --2- 172.21.15.7:0/1359260436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03c80c2e30 0x7f03c80c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 -- 172.21.15.7:0/1359260436 >> 172.21.15.7:0/1359260436 conn(0x7f03c80a94d0 msgr2=0x7f03c8071260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:02:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 -- 172.21.15.7:0/1359260436 shutdown_connections 2023-03-03T11:02:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:28.167+0000 7f03cceec700 1 -- 172.21.15.7:0/1359260436 wait complete. 2023-03-03T11:02:28.172 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:02:28.182 INFO:tasks.mds_thrash.fs.[b]:waiting for 30 secs before thrashing 2023-03-03T11:02:37.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:37.632 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:02:58.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:02:58.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.320+0000 7f86fc0c2700 1 Processor -- start 2023-03-03T11:02:58.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.320+0000 7f86fc0c2700 1 -- start start 2023-03-03T11:02:58.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86fc0c2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86f414f9d0 0x7f86f4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86fc0c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f40abe20 0x7f86f41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86fc0c2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f40c2f30 con 0x7f86f414fda0 2023-03-03T11:02:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86fc0c2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f40c31d0 con 0x7f86f414f9d0 2023-03-03T11:02:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86fc0c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f41524a0 con 0x7f86f40abe20 2023-03-03T11:02:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86f9e5e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86f414f9d0 0x7f86f4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86fa65f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f40abe20 0x7f86f41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86f9e5e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86f414f9d0 0x7f86f4156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:58588/0 (socket says 172.21.15.7:58588) 2023-03-03T11:02:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86fa65f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f40abe20 0x7f86f41529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:42658/0 (socket says 172.21.15.7:42658) 2023-03-03T11:02:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86f9e5e700 1 -- 172.21.15.7:0/1934330923 learned_addr learned my addr 172.21.15.7:0/1934330923 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:02:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86f9e5e700 1 -- 172.21.15.7:0/1934330923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f40abe20 msgr2=0x7f86f41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:02:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86f9e5e700 1 --2- 172.21.15.7:0/1934330923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f40abe20 0x7f86f41529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86f9e5e700 1 -- 172.21.15.7:0/1934330923 >> v1:172.21.15.143:6789/0 conn(0x7f86f414fda0 legacy=0x7f86f4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.321+0000 7f86f9e5e700 1 -- 172.21.15.7:0/1934330923 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f40c34b0 con 0x7f86f414f9d0 2023-03-03T11:02:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86f9e5e700 1 --2- 172.21.15.7:0/1934330923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86f414f9d0 0x7f86f4156430 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f86e40098f0 tx=0x7f86e4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=723605512857b9cf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86f8e5c700 1 -- 172.21.15.7:0/1934330923 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e4011830 con 0x7f86f414f9d0 2023-03-03T11:02:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86f8e5c700 1 -- 172.21.15.7:0/1934330923 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86e4011990 con 0x7f86f414f9d0 2023-03-03T11:02:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86f8e5c700 1 -- 172.21.15.7:0/1934330923 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86e4011d20 con 0x7f86f414f9d0 2023-03-03T11:02:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86fc0c2700 1 -- 172.21.15.7:0/1934330923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86f414f9d0 msgr2=0x7f86f4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86fc0c2700 1 --2- 172.21.15.7:0/1934330923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86f414f9d0 0x7f86f4156430 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f86e40098f0 tx=0x7f86e4009c00 comp rx=0 tx=0).stop 2023-03-03T11:02:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86fc0c2700 1 -- 172.21.15.7:0/1934330923 shutdown_connections 2023-03-03T11:02:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86fc0c2700 1 --2- 172.21.15.7:0/1934330923 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f40abe20 0x7f86f41529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86fc0c2700 1 --2- 172.21.15.7:0/1934330923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86f414f9d0 0x7f86f4156430 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86fc0c2700 1 -- 172.21.15.7:0/1934330923 >> 172.21.15.7:0/1934330923 conn(0x7f86f40c8900 msgr2=0x7f86f40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:02:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86fc0c2700 1 -- 172.21.15.7:0/1934330923 shutdown_connections 2023-03-03T11:02:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.322+0000 7f86fc0c2700 1 -- 172.21.15.7:0/1934330923 wait complete. 2023-03-03T11:02:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86fc0c2700 1 Processor -- start 2023-03-03T11:02:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86fc0c2700 1 -- start start 2023-03-03T11:02:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86fc0c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86f40abe20 0x7f86f40c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86fc0c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86f414f9d0 0x7f86f414b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86fc0c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f414fda0 0x7f86f414b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86fc0c2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f41df600 con 0x7f86f414f9d0 2023-03-03T11:02:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86fc0c2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f41eaa40 con 0x7f86f40abe20 2023-03-03T11:02:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86fc0c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f414c7d0 con 0x7f86f414fda0 2023-03-03T11:02:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86f965d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86f414f9d0 0x7f86f414b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86f965d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86f414f9d0 0x7f86f414b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34590/0 (socket says 172.21.15.7:34590) 2023-03-03T11:02:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.324+0000 7f86f965d700 1 -- 172.21.15.7:0/3852629406 learned_addr learned my addr 172.21.15.7:0/3852629406 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:02:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86f9e5e700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86f40abe20 0x7f86f40c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86fa65f700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f414fda0 0x7f86f414b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86f965d700 1 -- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f414fda0 msgr2=0x7f86f414b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86f965d700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f414fda0 0x7f86f414b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86f965d700 1 -- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86f40abe20 msgr2=0x7f86f40c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86f965d700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86f40abe20 0x7f86f40c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86f965d700 1 -- 172.21.15.7:0/3852629406 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e4009550 con 0x7f86f414f9d0 2023-03-03T11:02:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86f9e5e700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86f40abe20 0x7f86f40c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:02:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86f965d700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86f414f9d0 0x7f86f414b390 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f86f000c4e0 tx=0x7f86f000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86ea7fc700 1 -- 172.21.15.7:0/3852629406 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f00052d0 con 0x7f86f414f9d0 2023-03-03T11:02:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86fc0c2700 1 -- 172.21.15.7:0/3852629406 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f40ac350 con 0x7f86f414f9d0 2023-03-03T11:02:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86ea7fc700 1 -- 172.21.15.7:0/3852629406 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f0005430 con 0x7f86f414f9d0 2023-03-03T11:02:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86ea7fc700 1 -- 172.21.15.7:0/3852629406 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f00038c0 con 0x7f86f414f9d0 2023-03-03T11:02:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.325+0000 7f86fc0c2700 1 -- 172.21.15.7:0/3852629406 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f40ab630 con 0x7f86f414f9d0 2023-03-03T11:02:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.326+0000 7f86fc0c2700 1 -- 172.21.15.7:0/3852629406 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f4071fa0 con 0x7f86f414f9d0 2023-03-03T11:02:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.327+0000 7f86ea7fc700 1 -- 172.21.15.7:0/3852629406 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f86f0019030 con 0x7f86f414f9d0 2023-03-03T11:02:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.327+0000 7f86ea7fc700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86d80682f0 0x7f86d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:02:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.327+0000 7f86ea7fc700 1 -- 172.21.15.7:0/3852629406 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f86f0085530 con 0x7f86f414f9d0 2023-03-03T11:02:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.327+0000 7f86f9e5e700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86d80682f0 0x7f86d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:02:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.328+0000 7f86f9e5e700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86d80682f0 0x7f86d806a7a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f86e4009840 tx=0x7f86e4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:02:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.332+0000 7f86ea7fc700 1 -- 172.21.15.7:0/3852629406 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f86f0051850 con 0x7f86f414f9d0 2023-03-03T11:02:58.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.570+0000 7f86fc0c2700 1 -- 172.21.15.7:0/3852629406 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86f4074bb0 con 0x7f86f414f9d0 2023-03-03T11:02:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.571+0000 7f86ea7fc700 1 -- 172.21.15.7:0/3852629406 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f86f0054e70 con 0x7f86f414f9d0 2023-03-03T11:02:58.573 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:02:58.574 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:02:58.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 -- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86d80682f0 msgr2=0x7f86d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:58.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86d80682f0 0x7f86d806a7a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f86e4009840 tx=0x7f86e4008030 comp rx=0 tx=0).stop 2023-03-03T11:02:58.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 -- 172.21.15.7:0/3852629406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86f414f9d0 msgr2=0x7f86f414b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:02:58.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86f414f9d0 0x7f86f414b390 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f86f000c4e0 tx=0x7f86f000c8a0 comp rx=0 tx=0).stop 2023-03-03T11:02:58.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 -- 172.21.15.7:0/3852629406 shutdown_connections 2023-03-03T11:02:58.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86f414fda0 0x7f86f414b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:58.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86f40abe20 0x7f86f40c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:58.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86d80682f0 0x7f86d806a7a0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:58.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 --2- 172.21.15.7:0/3852629406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86f414f9d0 0x7f86f414b390 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:02:58.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 -- 172.21.15.7:0/3852629406 >> 172.21.15.7:0/3852629406 conn(0x7f86f40c8900 msgr2=0x7f86f40c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:02:58.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 -- 172.21.15.7:0/3852629406 shutdown_connections 2023-03-03T11:02:58.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:02:58.573+0000 7f86fc0c2700 1 -- 172.21.15.7:0/3852629406 wait complete. 2023-03-03T11:02:58.577 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:02:58.587 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2023-03-03T11:03:07.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:07.802 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:11.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:03:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.948+0000 7f4f6d202700 1 Processor -- start 2023-03-03T11:03:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.949+0000 7f4f6d202700 1 -- start start 2023-03-03T11:03:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.949+0000 7f4f6d202700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f680c5b00 0x7f4f680bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.949+0000 7f4f6d202700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f680c08d0 con 0x7f4f68151e80 2023-03-03T11:03:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.949+0000 7f4f6d202700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f68154930 con 0x7f4f68155410 2023-03-03T11:03:11.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.949+0000 7f4f6d202700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68152250 con 0x7f4f680c5b00 2023-03-03T11:03:11.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.949+0000 7f4f66d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4f68151e80 0x7f4f680c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56820/0 (socket says 172.21.15.7:56820) 2023-03-03T11:03:11.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f66d9d700 1 -- 172.21.15.7:0/258782312 learned_addr learned my addr 172.21.15.7:0/258782312 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:03:11.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537954068 0 0) 0x7f4f680c08d0 con 0x7f4f68151e80 2023-03-03T11:03:11.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f6759e700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f680c5b00 0x7f4f680bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:11.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f500035b0 con 0x7f4f68151e80 2023-03-03T11:03:11.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1634326884 0 0) 0x7f4f500035b0 con 0x7f4f68151e80 2023-03-03T11:03:11.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f680c5b00 msgr2=0x7f4f680bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:03:11.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f680c5b00 0x7f4f680bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:11.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 >> v1:172.21.15.143:6789/0 conn(0x7f4f68155410 legacy=0x7f4f680bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:11.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f68154c10 con 0x7f4f68151e80 2023-03-03T11:03:11.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f4f58001670 con 0x7f4f68151e80 2023-03-03T11:03:11.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 >> v1:172.21.15.7:6789/0 conn(0x7f4f68151e80 legacy=0x7f4f680c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:11.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f680c5b00 0x7f4f50003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f50003fe0 0x7f4f50004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 0x7f4f50005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5000bac0 con 0x7f4f50004d90 2023-03-03T11:03:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5000bc00 con 0x7f4f50003fe0 2023-03-03T11:03:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5000bdc0 con 0x7f4f680c5b00 2023-03-03T11:03:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f6759e700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 0x7f4f50005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f66d9d700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f680c5b00 0x7f4f50003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.950+0000 7f4f6659c700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f50003fe0 0x7f4f50004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6759e700 1 -- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f680c5b00 msgr2=0x7f4f50003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:03:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6759e700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f680c5b00 0x7f4f50003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6759e700 1 -- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f50003fe0 msgr2=0x7f4f50004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:03:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6759e700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f50003fe0 0x7f4f50004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6759e700 1 -- 172.21.15.7:0/258782312 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f500035b0 con 0x7f4f50004d90 2023-03-03T11:03:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6759e700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 0x7f4f50005200 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009f90 tx=0x7f4f50005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223bbd63772f4fad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:03:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6000ec30 con 0x7f4f50004d90 2023-03-03T11:03:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6000f040 con 0x7f4f50004d90 2023-03-03T11:03:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f65d9b700 1 -- 172.21.15.7:0/258782312 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60014480 con 0x7f4f50004d90 2023-03-03T11:03:11.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6d202700 1 -- 172.21.15.7:0/258782312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 msgr2=0x7f4f50005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:11.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6d202700 1 --2- 172.21.15.7:0/258782312 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 0x7f4f50005200 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009f90 tx=0x7f4f50005880 comp rx=0 tx=0).stop 2023-03-03T11:03:11.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6759e700 1 -- 172.21.15.7:0/258782312 reap_dead start 2023-03-03T11:03:11.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6d202700 1 -- 172.21.15.7:0/258782312 shutdown_connections 2023-03-03T11:03:11.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.951+0000 7f4f6d202700 1 -- 172.21.15.7:0/258782312 >> 172.21.15.7:0/258782312 conn(0x7f4f68071630 msgr2=0x7f4f6806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:03:11.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.952+0000 7f4f6d202700 1 -- 172.21.15.7:0/258782312 shutdown_connections 2023-03-03T11:03:11.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.952+0000 7f4f6d202700 1 -- 172.21.15.7:0/258782312 wait complete. 2023-03-03T11:03:11.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.953+0000 7f4f6d202700 1 Processor -- start 2023-03-03T11:03:11.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.954+0000 7f4f6d202700 1 -- start start 2023-03-03T11:03:11.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.954+0000 7f4f6d202700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 0x7f4f681defe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:11.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.954+0000 7f4f6d202700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f68157d80 0x7f4f68145150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:11.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.954+0000 7f4f6d202700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f681560c0 0x7f4f68145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:11.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.954+0000 7f4f6d202700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68145bd0 con 0x7f4f50004d90 2023-03-03T11:03:11.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.954+0000 7f4f6d202700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f68155ab0 con 0x7f4f68157d80 2023-03-03T11:03:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.954+0000 7f4f6d202700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f680c3db0 con 0x7f4f681560c0 2023-03-03T11:03:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f66d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 0x7f4f681defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f66d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 0x7f4f681defe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34616/0 (socket says 172.21.15.7:34616) 2023-03-03T11:03:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f66d9d700 1 -- 172.21.15.7:0/3037114035 learned_addr learned my addr 172.21.15.7:0/3037114035 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:03:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f6759e700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f681560c0 0x7f4f68145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f6659c700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f68157d80 0x7f4f68145150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f66d9d700 1 -- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f681560c0 msgr2=0x7f4f68145690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:11.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f66d9d700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f681560c0 0x7f4f68145690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:11.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f66d9d700 1 -- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f68157d80 msgr2=0x7f4f68145150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:11.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f66d9d700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f68157d80 0x7f4f68145150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:11.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f66d9d700 1 -- 172.21.15.7:0/3037114035 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f680c5220 con 0x7f4f50004d90 2023-03-03T11:03:11.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f66d9d700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 0x7f4f681defe0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f4f58007290 tx=0x7f4f580075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:03:11.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3037114035 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58011580 con 0x7f4f50004d90 2023-03-03T11:03:11.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3037114035 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f58011700 con 0x7f4f50004d90 2023-03-03T11:03:11.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3037114035 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f58011ab0 con 0x7f4f50004d90 2023-03-03T11:03:11.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f6d202700 1 -- 172.21.15.7:0/3037114035 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f68154ef0 con 0x7f4f50004d90 2023-03-03T11:03:11.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.955+0000 7f4f6d202700 1 -- 172.21.15.7:0/3037114035 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f680c71b0 con 0x7f4f50004d90 2023-03-03T11:03:11.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.956+0000 7f4f6d202700 1 -- 172.21.15.7:0/3037114035 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f680c4370 con 0x7f4f50004d90 2023-03-03T11:03:11.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.957+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3037114035 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4f58011c10 con 0x7f4f50004d90 2023-03-03T11:03:11.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.957+0000 7f4f4f7fe700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f540682f0 0x7f4f5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:03:11.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.957+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3037114035 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4f58086170 con 0x7f4f50004d90 2023-03-03T11:03:11.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.957+0000 7f4f6659c700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f540682f0 0x7f4f5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:03:11.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.958+0000 7f4f6659c700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f540682f0 0x7f4f5406a7a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4f68156810 tx=0x7f4f5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:03:11.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:11.962+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3037114035 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4f580525d0 con 0x7f4f50004d90 2023-03-03T11:03:12.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.202+0000 7f4f6d202700 1 -- 172.21.15.7:0/3037114035 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f680c6ba0 con 0x7f4f50004d90 2023-03-03T11:03:12.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.203+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3037114035 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4f58055bf0 con 0x7f4f50004d90 2023-03-03T11:03:12.205 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:03:12.205 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:03:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 -- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f540682f0 msgr2=0x7f4f5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f540682f0 0x7f4f5406a7a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4f68156810 tx=0x7f4f5c009040 comp rx=0 tx=0).stop 2023-03-03T11:03:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 -- 172.21.15.7:0/3037114035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 msgr2=0x7f4f681defe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:03:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 0x7f4f681defe0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f4f58007290 tx=0x7f4f580075a0 comp rx=0 tx=0).stop 2023-03-03T11:03:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 -- 172.21.15.7:0/3037114035 shutdown_connections 2023-03-03T11:03:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f681560c0 0x7f4f68145690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f540682f0 0x7f4f5406a7a0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f50004d90 0x7f4f681defe0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 --2- 172.21.15.7:0/3037114035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f68157d80 0x7f4f68145150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:03:12.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 -- 172.21.15.7:0/3037114035 >> 172.21.15.7:0/3037114035 conn(0x7f4f68071630 msgr2=0x7f4f680c9320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:03:12.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 -- 172.21.15.7:0/3037114035 shutdown_connections 2023-03-03T11:03:12.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:03:12.205+0000 7f4f6d202700 1 -- 172.21.15.7:0/3037114035 wait complete. 2023-03-03T11:03:12.209 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:03:12.219 INFO:tasks.mds_thrash.fs.[a]:waiting for 62 secs before thrashing 2023-03-03T11:03:37.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:03:37.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:08.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:08.031 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:14.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:04:14.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.357+0000 7f01f2972700 1 Processor -- start 2023-03-03T11:04:14.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.357+0000 7f01f2972700 1 -- start start 2023-03-03T11:04:14.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.358+0000 7f01f2972700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f01ec14b020 0x7f01ec0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:14.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.358+0000 7f01f2972700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01ec0aad20 0x7f01ec157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:14.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.358+0000 7f01f2972700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec071d20 0x7f01ec1581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:14.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.358+0000 7f01f2972700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec0728b0 con 0x7f01ec0aad20 2023-03-03T11:04:14.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.358+0000 7f01f2972700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec155070 con 0x7f01ec14b020 2023-03-03T11:04:14.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.358+0000 7f01f2972700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec06f5b0 con 0x7f01ec071d20 2023-03-03T11:04:14.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.358+0000 7f01eb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01ec0aad20 0x7f01ec157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:14.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.358+0000 7f01eb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01ec0aad20 0x7f01ec157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34654/0 (socket says 172.21.15.7:34654) 2023-03-03T11:04:14.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.358+0000 7f01eb7fe700 1 -- 172.21.15.7:0/2641086335 learned_addr learned my addr 172.21.15.7:0/2641086335 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:04:14.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01ebfff700 1 --2- 172.21.15.7:0/2641086335 >> v2:172.21.15.143:3300/0 conn(0x7f01ec14b020 0x7f01ec0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:14.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01f0f0f700 1 --2- 172.21.15.7:0/2641086335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec071d20 0x7f01ec1581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:14.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01eb7fe700 1 -- 172.21.15.7:0/2641086335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec071d20 msgr2=0x7f01ec1581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:04:14.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01eb7fe700 1 --2- 172.21.15.7:0/2641086335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec071d20 0x7f01ec1581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01eb7fe700 1 -- 172.21.15.7:0/2641086335 >> v2:172.21.15.143:3300/0 conn(0x7f01ec14b020 msgr2=0x7f01ec0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:04:14.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01eb7fe700 1 --2- 172.21.15.7:0/2641086335 >> v2:172.21.15.143:3300/0 conn(0x7f01ec14b020 0x7f01ec0717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01eb7fe700 1 -- 172.21.15.7:0/2641086335 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec06f890 con 0x7f01ec0aad20 2023-03-03T11:04:14.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01f0f0f700 1 --2- 172.21.15.7:0/2641086335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec071d20 0x7f01ec1581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:04:14.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01ebfff700 1 --2- 172.21.15.7:0/2641086335 >> v2:172.21.15.143:3300/0 conn(0x7f01ec14b020 0x7f01ec0717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:04:14.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01eb7fe700 1 --2- 172.21.15.7:0/2641086335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 0x7f01ec157ca0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f01ec070940 tx=0x7f01e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da333e64cbc12881 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:14.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01eaffd700 1 -- 172.21.15.7:0/2641086335 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e000b870 con 0x7f01ec0aad20 2023-03-03T11:04:14.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01eaffd700 1 -- 172.21.15.7:0/2641086335 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e000b9d0 con 0x7f01ec0aad20 2023-03-03T11:04:14.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.359+0000 7f01eaffd700 1 -- 172.21.15.7:0/2641086335 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e000bd60 con 0x7f01ec0aad20 2023-03-03T11:04:14.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.360+0000 7f01f2972700 1 -- 172.21.15.7:0/2641086335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 msgr2=0x7f01ec157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:14.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.360+0000 7f01f2972700 1 --2- 172.21.15.7:0/2641086335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 0x7f01ec157ca0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f01ec070940 tx=0x7f01e0009bd0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.360+0000 7f01f2972700 1 -- 172.21.15.7:0/2641086335 shutdown_connections 2023-03-03T11:04:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.360+0000 7f01f2972700 1 --2- 172.21.15.7:0/2641086335 >> v2:172.21.15.143:3300/0 conn(0x7f01ec14b020 0x7f01ec0717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.360+0000 7f01f2972700 1 --2- 172.21.15.7:0/2641086335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec071d20 0x7f01ec1581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.360+0000 7f01f2972700 1 --2- 172.21.15.7:0/2641086335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 0x7f01ec157ca0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.360+0000 7f01f2972700 1 -- 172.21.15.7:0/2641086335 >> 172.21.15.7:0/2641086335 conn(0x7f01ec0c8900 msgr2=0x7f01ec0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:04:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.360+0000 7f01f2972700 1 -- 172.21.15.7:0/2641086335 shutdown_connections 2023-03-03T11:04:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.360+0000 7f01f2972700 1 -- 172.21.15.7:0/2641086335 wait complete. 2023-03-03T11:04:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.361+0000 7f01f2972700 1 Processor -- start 2023-03-03T11:04:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.361+0000 7f01f2972700 1 -- start start 2023-03-03T11:04:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01f2972700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01ec071d20 0x7f01ec143f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01f2972700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 0x7f01ec0c6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01f2972700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec14b020 0x7f01ec0c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01f2972700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0c6ed0 con 0x7f01ec0aad20 2023-03-03T11:04:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01f2972700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec1ede50 con 0x7f01ec071d20 2023-03-03T11:04:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01f2972700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0c7750 con 0x7f01ec14b020 2023-03-03T11:04:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01eb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 0x7f01ec0c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:14.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01eb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 0x7f01ec0c6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34662/0 (socket says 172.21.15.7:34662) 2023-03-03T11:04:14.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01eb7fe700 1 -- 172.21.15.7:0/2223312580 learned_addr learned my addr 172.21.15.7:0/2223312580 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:04:14.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01ebfff700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01ec071d20 0x7f01ec143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:14.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01f0f0f700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec14b020 0x7f01ec0c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:14.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01eb7fe700 1 -- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec14b020 msgr2=0x7f01ec0c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:14.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01eb7fe700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec14b020 0x7f01ec0c7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01eb7fe700 1 -- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01ec071d20 msgr2=0x7f01ec143f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:14.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01eb7fe700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01ec071d20 0x7f01ec143f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01eb7fe700 1 -- 172.21.15.7:0/2223312580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e0009550 con 0x7f01ec0aad20 2023-03-03T11:04:14.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01ebfff700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01ec071d20 0x7f01ec143f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:04:14.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.362+0000 7f01eb7fe700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 0x7f01ec0c6580 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f01e0013000 tx=0x7f01e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:14.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.363+0000 7f01e8ff9700 1 -- 172.21.15.7:0/2223312580 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0004520 con 0x7f01ec0aad20 2023-03-03T11:04:14.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.363+0000 7f01e8ff9700 1 -- 172.21.15.7:0/2223312580 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e001e070 con 0x7f01ec0aad20 2023-03-03T11:04:14.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.363+0000 7f01f2972700 1 -- 172.21.15.7:0/2223312580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec155350 con 0x7f01ec0aad20 2023-03-03T11:04:14.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.363+0000 7f01e8ff9700 1 -- 172.21.15.7:0/2223312580 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e00038e0 con 0x7f01ec0aad20 2023-03-03T11:04:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.363+0000 7f01f2972700 1 -- 172.21.15.7:0/2223312580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec06fb70 con 0x7f01ec0aad20 2023-03-03T11:04:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.364+0000 7f01f2972700 1 -- 172.21.15.7:0/2223312580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ec158720 con 0x7f01ec0aad20 2023-03-03T11:04:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.364+0000 7f01e8ff9700 1 -- 172.21.15.7:0/2223312580 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f01e000f050 con 0x7f01ec0aad20 2023-03-03T11:04:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.364+0000 7f01e8ff9700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01d80682f0 0x7f01d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.365+0000 7f01e8ff9700 1 -- 172.21.15.7:0/2223312580 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f01e0020070 con 0x7f01ec0aad20 2023-03-03T11:04:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.365+0000 7f01ebfff700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01d80682f0 0x7f01d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.365+0000 7f01ebfff700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01d80682f0 0x7f01d806a7a0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f01dc006fd0 tx=0x7f01dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.369+0000 7f01e8ff9700 1 -- 172.21.15.7:0/2223312580 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f01e0053aa0 con 0x7f01ec0aad20 2023-03-03T11:04:14.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.612+0000 7f01f2972700 1 -- 172.21.15.7:0/2223312580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01ec074bb0 con 0x7f01ec0aad20 2023-03-03T11:04:14.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.612+0000 7f01e8ff9700 1 -- 172.21.15.7:0/2223312580 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f01e00570c0 con 0x7f01ec0aad20 2023-03-03T11:04:14.615 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:04:14.615 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:04:14.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.614+0000 7f01f2972700 1 -- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01d80682f0 msgr2=0x7f01d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.614+0000 7f01f2972700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01d80682f0 0x7f01d806a7a0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f01dc006fd0 tx=0x7f01dc008040 comp rx=0 tx=0).stop 2023-03-03T11:04:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.614+0000 7f01f2972700 1 -- 172.21.15.7:0/2223312580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 msgr2=0x7f01ec0c6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.614+0000 7f01f2972700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 0x7f01ec0c6580 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f01e0013000 tx=0x7f01e0004750 comp rx=0 tx=0).stop 2023-03-03T11:04:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.614+0000 7f01f2972700 1 -- 172.21.15.7:0/2223312580 shutdown_connections 2023-03-03T11:04:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.614+0000 7f01f2972700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01ec14b020 0x7f01ec0c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.614+0000 7f01f2972700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01ec071d20 0x7f01ec143f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.614+0000 7f01f2972700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01d80682f0 0x7f01d806a7a0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.614+0000 7f01f2972700 1 --2- 172.21.15.7:0/2223312580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01ec0aad20 0x7f01ec0c6580 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.614+0000 7f01f2972700 1 -- 172.21.15.7:0/2223312580 >> 172.21.15.7:0/2223312580 conn(0x7f01ec0c8900 msgr2=0x7f01ec148ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:04:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.615+0000 7f01f2972700 1 -- 172.21.15.7:0/2223312580 shutdown_connections 2023-03-03T11:04:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:14.615+0000 7f01f2972700 1 -- 172.21.15.7:0/2223312580 wait complete. 2023-03-03T11:04:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:04:14.628 INFO:tasks.mds_thrash.fs.[a]:waiting for 24 secs before thrashing 2023-03-03T11:04:24.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:04:24.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.725+0000 7f88198bd700 1 Processor -- start 2023-03-03T11:04:24.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.725+0000 7f88198bd700 1 -- start start 2023-03-03T11:04:24.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f88198bd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88140c00e0 0x7f88140c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:24.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f88198bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8814154ff0 0x7f88140cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:24.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f88198bd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88140c7310 con 0x7f88140caa30 2023-03-03T11:04:24.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f88198bd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88140bfd10 con 0x7f88140c00e0 2023-03-03T11:04:24.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f88198bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88140bfa30 con 0x7f8814154ff0 2023-03-03T11:04:24.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f8812ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f88140caa30 0x7f88140c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56892/0 (socket says 172.21.15.7:56892) 2023-03-03T11:04:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f8812ffd700 1 -- 172.21.15.7:0/1686743146 learned_addr learned my addr 172.21.15.7:0/1686743146 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:04:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f88137fe700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8814154ff0 0x7f88140cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f88127fc700 1 --2- 172.21.15.7:0/1686743146 >> v2:172.21.15.143:3300/0 conn(0x7f88140c00e0 0x7f88140c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822304565 0 0) 0x7f88140c7310 con 0x7f88140caa30 2023-03-03T11:04:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.726+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87fc0035b0 con 0x7f88140caa30 2023-03-03T11:04:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1799054081 0 0) 0x7f87fc0035b0 con 0x7f88140caa30 2023-03-03T11:04:24.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8814154ff0 msgr2=0x7f88140cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:04:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8814154ff0 0x7f88140cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 >> v2:172.21.15.143:3300/0 conn(0x7f88140c00e0 msgr2=0x7f88140c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:04:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 --2- 172.21.15.7:0/1686743146 >> v2:172.21.15.143:3300/0 conn(0x7f88140c00e0 0x7f88140c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8814154510 con 0x7f88140caa30 2023-03-03T11:04:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8804001670 con 0x7f88140caa30 2023-03-03T11:04:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 >> v1:172.21.15.7:6789/0 conn(0x7f88140caa30 legacy=0x7f88140c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87fc003a70 0x7f87fc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:24.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87fc004850 0x7f87fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:24.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8814154ff0 0x7f87fc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:24.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88140bfd10 con 0x7f87fc003a70 2023-03-03T11:04:24.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88140bfa30 con 0x7f87fc004850 2023-03-03T11:04:24.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc00ba40 con 0x7f8814154ff0 2023-03-03T11:04:24.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8812ffd700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87fc003a70 0x7f87fc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:24.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f88127fc700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87fc004850 0x7f87fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:24.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f88137fe700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8814154ff0 0x7f87fc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8812ffd700 1 -- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8814154ff0 msgr2=0x7f87fc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:04:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8812ffd700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8814154ff0 0x7f87fc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8812ffd700 1 -- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87fc004850 msgr2=0x7f87fc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:04:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8812ffd700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87fc004850 0x7f87fc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8812ffd700 1 -- 172.21.15.7:0/1686743146 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc0035b0 con 0x7f87fc003a70 2023-03-03T11:04:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.727+0000 7f8812ffd700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87fc003a70 0x7f87fc003e80 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f880400c370 tx=0x7f87fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf3a5762c1aee1c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.728+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804007c60 con 0x7f87fc003a70 2023-03-03T11:04:24.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.728+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8804005960 con 0x7f87fc003a70 2023-03-03T11:04:24.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.728+0000 7f8811ffb700 1 -- 172.21.15.7:0/1686743146 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804005d10 con 0x7f87fc003a70 2023-03-03T11:04:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.728+0000 7f88198bd700 1 -- 172.21.15.7:0/1686743146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87fc003a70 msgr2=0x7f87fc003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.728+0000 7f88198bd700 1 --2- 172.21.15.7:0/1686743146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87fc003a70 0x7f87fc003e80 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f880400c370 tx=0x7f87fc005880 comp rx=0 tx=0).stop 2023-03-03T11:04:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.728+0000 7f88137fe700 1 -- 172.21.15.7:0/1686743146 reap_dead start 2023-03-03T11:04:24.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.728+0000 7f88198bd700 1 -- 172.21.15.7:0/1686743146 shutdown_connections 2023-03-03T11:04:24.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.728+0000 7f88198bd700 1 -- 172.21.15.7:0/1686743146 >> 172.21.15.7:0/1686743146 conn(0x7f8814066e40 msgr2=0x7f8814071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:04:24.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.728+0000 7f88198bd700 1 -- 172.21.15.7:0/1686743146 shutdown_connections 2023-03-03T11:04:24.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.728+0000 7f88198bd700 1 -- 172.21.15.7:0/1686743146 wait complete. 2023-03-03T11:04:24.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.729+0000 7f88198bd700 1 Processor -- start 2023-03-03T11:04:24.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.730+0000 7f88198bd700 1 -- start start 2023-03-03T11:04:24.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.730+0000 7f88198bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87fc003a70 0x7f8814156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:24.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.730+0000 7f88198bd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f881414e430 0x7f881413dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:24.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.730+0000 7f88198bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881414e8a0 0x7f881413e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:24.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.730+0000 7f88198bd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88141579a0 con 0x7f881414e430 2023-03-03T11:04:24.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.730+0000 7f88198bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881414d950 con 0x7f881414e8a0 2023-03-03T11:04:24.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.730+0000 7f88198bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88141edd50 con 0x7f87fc003a70 2023-03-03T11:04:24.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88127fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f881414e430 0x7f881413dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:24.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88127fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f881414e430 0x7f881413dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34694/0 (socket says 172.21.15.7:34694) 2023-03-03T11:04:24.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88127fc700 1 -- 172.21.15.7:0/4208024580 learned_addr learned my addr 172.21.15.7:0/4208024580 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:04:24.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f8812ffd700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87fc003a70 0x7f8814156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:24.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88137fe700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881414e8a0 0x7f881413e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:24.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88127fc700 1 -- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87fc003a70 msgr2=0x7f8814156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:24.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88127fc700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87fc003a70 0x7f8814156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:24.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88127fc700 1 -- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881414e8a0 msgr2=0x7f881413e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:24.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88127fc700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881414e8a0 0x7f881413e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:24.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88127fc700 1 -- 172.21.15.7:0/4208024580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8814154ad0 con 0x7f881414e430 2023-03-03T11:04:24.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88127fc700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f881414e430 0x7f881413dc40 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f8808009fd0 tx=0x7f8808011c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:24.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f87fbfff700 1 -- 172.21.15.7:0/4208024580 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808005c30 con 0x7f881414e430 2023-03-03T11:04:24.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f87fbfff700 1 -- 172.21.15.7:0/4208024580 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8808005d90 con 0x7f881414e430 2023-03-03T11:04:24.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88198bd700 1 -- 172.21.15.7:0/4208024580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88141547f0 con 0x7f881414e430 2023-03-03T11:04:24.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f87fbfff700 1 -- 172.21.15.7:0/4208024580 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808006290 con 0x7f881414e430 2023-03-03T11:04:24.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.731+0000 7f88198bd700 1 -- 172.21.15.7:0/4208024580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88140bf650 con 0x7f881414e430 2023-03-03T11:04:24.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.732+0000 7f88198bd700 1 -- 172.21.15.7:0/4208024580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88141557d0 con 0x7f881414e430 2023-03-03T11:04:24.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.733+0000 7f87fbfff700 1 -- 172.21.15.7:0/4208024580 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f88080063f0 con 0x7f881414e430 2023-03-03T11:04:24.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.733+0000 7f87fbfff700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8800068320 0x7f880006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:24.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.733+0000 7f87fbfff700 1 -- 172.21.15.7:0/4208024580 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f88080866c0 con 0x7f881414e430 2023-03-03T11:04:24.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.733+0000 7f8812ffd700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8800068320 0x7f880006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:24.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.734+0000 7f8812ffd700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8800068320 0x7f880006a7d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8804015040 tx=0x7f880400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:24.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.737+0000 7f87fbfff700 1 -- 172.21.15.7:0/4208024580 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f88141557d0 con 0x7f881414e430 2023-03-03T11:04:24.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.974+0000 7f88198bd700 1 -- 172.21.15.7:0/4208024580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88141557d0 con 0x7f881414e430 2023-03-03T11:04:24.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.975+0000 7f87fbfff700 1 -- 172.21.15.7:0/4208024580 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f88141557d0 con 0x7f881414e430 2023-03-03T11:04:24.977 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:04:24.977 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:04:24.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 -- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8800068320 msgr2=0x7f880006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8800068320 0x7f880006a7d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8804015040 tx=0x7f880400a000 comp rx=0 tx=0).stop 2023-03-03T11:04:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 -- 172.21.15.7:0/4208024580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f881414e430 msgr2=0x7f881413dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f881414e430 0x7f881413dc40 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f8808009fd0 tx=0x7f8808011c80 comp rx=0 tx=0).stop 2023-03-03T11:04:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 -- 172.21.15.7:0/4208024580 shutdown_connections 2023-03-03T11:04:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f881414e8a0 0x7f881413e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87fc003a70 0x7f8814156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:24.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8800068320 0x7f880006a7d0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:24.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 --2- 172.21.15.7:0/4208024580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f881414e430 0x7f881413dc40 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:24.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 -- 172.21.15.7:0/4208024580 >> 172.21.15.7:0/4208024580 conn(0x7f8814066e40 msgr2=0x7f88140c8590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:04:24.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 -- 172.21.15.7:0/4208024580 shutdown_connections 2023-03-03T11:04:24.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:24.977+0000 7f88198bd700 1 -- 172.21.15.7:0/4208024580 wait complete. 2023-03-03T11:04:24.981 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:04:24.991 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T11:04:38.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:38.185 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:04:38.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:04:38.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.764+0000 7f82fc535700 1 Processor -- start 2023-03-03T11:04:38.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.765+0000 7f82fc535700 1 -- start start 2023-03-03T11:04:38.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.765+0000 7f82fc535700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82f4152e30 0x7f82f40c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:38.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82fc535700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f82f40c4910 0x7f82f4156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:38.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82fc535700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f41523b0 con 0x7f82f40beab0 2023-03-03T11:04:38.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82fc535700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f4072010 con 0x7f82f40c4910 2023-03-03T11:04:38.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82fc535700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f40aaaf0 con 0x7f82f4152e30 2023-03-03T11:04:38.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82faad2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f82f40c4910 0x7f82f4156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:38.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82faad2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f82f40c4910 0x7f82f4156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34708/0 (socket says 172.21.15.7:34708) 2023-03-03T11:04:38.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82faad2700 1 -- 172.21.15.7:0/773248245 learned_addr learned my addr 172.21.15.7:0/773248245 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:04:38.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82faad2700 1 -- 172.21.15.7:0/773248245 >> v2:172.21.15.143:3300/0 conn(0x7f82f4152e30 msgr2=0x7f82f40c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:04:38.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82faad2700 1 --2- 172.21.15.7:0/773248245 >> v2:172.21.15.143:3300/0 conn(0x7f82f4152e30 0x7f82f40c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:38.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82faad2700 1 -- 172.21.15.7:0/773248245 >> v1:172.21.15.143:6789/0 conn(0x7f82f40beab0 legacy=0x7f82f4155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:04:38.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82faad2700 1 -- 172.21.15.7:0/773248245 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f40aadd0 con 0x7f82f40c4910 2023-03-03T11:04:38.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82faad2700 1 --2- 172.21.15.7:0/773248245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 0x7f82f4156490 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f82f40c60b0 tx=0x7f82ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2d767352c51d4f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:38.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82f92cf700 1 -- 172.21.15.7:0/773248245 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec00e070 con 0x7f82f40c4910 2023-03-03T11:04:38.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82f92cf700 1 -- 172.21.15.7:0/773248245 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec003a20 con 0x7f82f40c4910 2023-03-03T11:04:38.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.766+0000 7f82f92cf700 1 -- 172.21.15.7:0/773248245 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec003dd0 con 0x7f82f40c4910 2023-03-03T11:04:38.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.767+0000 7f82fc535700 1 -- 172.21.15.7:0/773248245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 msgr2=0x7f82f4156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:38.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.767+0000 7f82fc535700 1 --2- 172.21.15.7:0/773248245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 0x7f82f4156490 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f82f40c60b0 tx=0x7f82ec00b330 comp rx=0 tx=0).stop 2023-03-03T11:04:38.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.767+0000 7f82fc535700 1 -- 172.21.15.7:0/773248245 shutdown_connections 2023-03-03T11:04:38.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.767+0000 7f82fc535700 1 --2- 172.21.15.7:0/773248245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 0x7f82f4156490 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:38.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.767+0000 7f82fc535700 1 --2- 172.21.15.7:0/773248245 >> v2:172.21.15.143:3300/0 conn(0x7f82f4152e30 0x7f82f40c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:38.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.767+0000 7f82fc535700 1 -- 172.21.15.7:0/773248245 >> 172.21.15.7:0/773248245 conn(0x7f82f40736f0 msgr2=0x7f82f4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:04:38.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.767+0000 7f82fc535700 1 -- 172.21.15.7:0/773248245 shutdown_connections 2023-03-03T11:04:38.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.767+0000 7f82fc535700 1 -- 172.21.15.7:0/773248245 wait complete. 2023-03-03T11:04:38.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.768+0000 7f82fc535700 1 Processor -- start 2023-03-03T11:04:38.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.768+0000 7f82fc535700 1 -- start start 2023-03-03T11:04:38.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82fc535700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f40beab0 0x7f82f406ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:38.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82fc535700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 0x7f82f41ea410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:38.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82fc535700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f4152e30 0x7f82f41ea950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:38.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82fc535700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40ac310 con 0x7f82f40c4910 2023-03-03T11:04:38.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82fc535700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41eae90 con 0x7f82f4152e30 2023-03-03T11:04:38.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82fc535700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41e9f10 con 0x7f82f40beab0 2023-03-03T11:04:38.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82f9ad0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 0x7f82f41ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:38.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82f9ad0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 0x7f82f41ea410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34712/0 (socket says 172.21.15.7:34712) 2023-03-03T11:04:38.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82f9ad0700 1 -- 172.21.15.7:0/3841329304 learned_addr learned my addr 172.21.15.7:0/3841329304 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:04:38.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82faad2700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f4152e30 0x7f82f41ea950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:38.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82f9ad0700 1 -- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f40beab0 msgr2=0x7f82f406ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:38.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82fa2d1700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f40beab0 0x7f82f406ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:38.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82f9ad0700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f40beab0 0x7f82f406ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:38.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82f9ad0700 1 -- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f4152e30 msgr2=0x7f82f41ea950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:38.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82f9ad0700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f4152e30 0x7f82f41ea950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:38.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.769+0000 7f82f9ad0700 1 -- 172.21.15.7:0/3841329304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec009b30 con 0x7f82f40c4910 2023-03-03T11:04:38.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.770+0000 7f82f9ad0700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 0x7f82f41ea410 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f82e800b470 tx=0x7f82e800b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:38.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.770+0000 7f82e6ffd700 1 -- 172.21.15.7:0/3841329304 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8011830 con 0x7f82f40c4910 2023-03-03T11:04:38.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.770+0000 7f82e6ffd700 1 -- 172.21.15.7:0/3841329304 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e8011990 con 0x7f82f40c4910 2023-03-03T11:04:38.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.770+0000 7f82fc535700 1 -- 172.21.15.7:0/3841329304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f4153200 con 0x7f82f40c4910 2023-03-03T11:04:38.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.770+0000 7f82e6ffd700 1 -- 172.21.15.7:0/3841329304 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8011d40 con 0x7f82f40c4910 2023-03-03T11:04:38.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.770+0000 7f82fc535700 1 -- 172.21.15.7:0/3841329304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f4071c70 con 0x7f82f40c4910 2023-03-03T11:04:38.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.771+0000 7f82fc535700 1 -- 172.21.15.7:0/3841329304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f4153390 con 0x7f82f40c4910 2023-03-03T11:04:38.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.771+0000 7f82e6ffd700 1 -- 172.21.15.7:0/3841329304 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f82e8021070 con 0x7f82f40c4910 2023-03-03T11:04:38.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.772+0000 7f82e6ffd700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 0x7f82d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:04:38.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.772+0000 7f82e6ffd700 1 -- 172.21.15.7:0/3841329304 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f82e80860b0 con 0x7f82f40c4910 2023-03-03T11:04:38.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.772+0000 7f82fa2d1700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 0x7f82d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:04:38.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.773+0000 7f82fa2d1700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 0x7f82d806a7a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f82f40ae450 tx=0x7f82f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:04:38.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:38.776+0000 7f82e6ffd700 1 -- 172.21.15.7:0/3841329304 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f82e8052450 con 0x7f82f40c4910 2023-03-03T11:04:39.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.016+0000 7f82fc535700 1 -- 172.21.15.7:0/3841329304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82f4074bb0 con 0x7f82f40c4910 2023-03-03T11:04:39.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.016+0000 7f82e6ffd700 1 -- 172.21.15.7:0/3841329304 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f82e8015020 con 0x7f82f40c4910 2023-03-03T11:04:39.019 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:04:39.020 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 -- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 msgr2=0x7f82d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 0x7f82d806a7a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f82f40ae450 tx=0x7f82f0008040 comp rx=0 tx=0).stop 2023-03-03T11:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 -- 172.21.15.7:0/3841329304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 msgr2=0x7f82f41ea410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 0x7f82f41ea410 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f82e800b470 tx=0x7f82e800b830 comp rx=0 tx=0).stop 2023-03-03T11:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 -- 172.21.15.7:0/3841329304 shutdown_connections 2023-03-03T11:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82f4152e30 0x7f82f41ea950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82f40beab0 0x7f82f406ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:39.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82d80682f0 0x7f82d806a7a0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:39.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 --2- 172.21.15.7:0/3841329304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82f40c4910 0x7f82f41ea410 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:04:39.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 -- 172.21.15.7:0/3841329304 >> 172.21.15.7:0/3841329304 conn(0x7f82f40736f0 msgr2=0x7f82f40a9950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:04:39.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 -- 172.21.15.7:0/3841329304 shutdown_connections 2023-03-03T11:04:39.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:04:39.018+0000 7f82fc535700 1 -- 172.21.15.7:0/3841329304 wait complete. 2023-03-03T11:04:39.023 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:04:39.033 INFO:tasks.mds_thrash.fs.[a]:waiting for 52 secs before thrashing 2023-03-03T11:05:08.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:08.298 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:28.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:05:29.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.129+0000 7fb3439a8700 1 Processor -- start 2023-03-03T11:05:29.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.129+0000 7fb3439a8700 1 -- start start 2023-03-03T11:05:29.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb3439a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb33c071e80 0x7fb33c151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:29.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb3439a8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb33c072250 0x7fb33c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb3439a8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb33c0c1ce0 0x7fb33c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb3439a8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33c1510d0 con 0x7fb33c072250 2023-03-03T11:05:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb3439a8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33c0c3740 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb3439a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c070670 con 0x7fb33c071e80 2023-03-03T11:05:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb340f43700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb33c072250 0x7fb33c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb340f43700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb33c072250 0x7fb33c14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34744/0 (socket says 172.21.15.7:34744) 2023-03-03T11:05:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb340f43700 1 -- 172.21.15.7:0/1054925816 learned_addr learned my addr 172.21.15.7:0/1054925816 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:05:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb341f45700 1 --2- 172.21.15.7:0/1054925816 >> v2:172.21.15.143:3300/0 conn(0x7fb33c0c1ce0 0x7fb33c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb340f43700 1 -- 172.21.15.7:0/1054925816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb33c071e80 msgr2=0x7fb33c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:05:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb341744700 1 --2- 172.21.15.7:0/1054925816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb33c071e80 0x7fb33c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb340f43700 1 --2- 172.21.15.7:0/1054925816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb33c071e80 0x7fb33c151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb340f43700 1 -- 172.21.15.7:0/1054925816 >> v2:172.21.15.143:3300/0 conn(0x7fb33c0c1ce0 msgr2=0x7fb33c14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:05:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb340f43700 1 --2- 172.21.15.7:0/1054925816 >> v2:172.21.15.143:3300/0 conn(0x7fb33c0c1ce0 0x7fb33c14e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb340f43700 1 -- 172.21.15.7:0/1054925816 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c0c39e0 con 0x7fb33c072250 2023-03-03T11:05:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.130+0000 7fb341f45700 1 --2- 172.21.15.7:0/1054925816 >> v2:172.21.15.143:3300/0 conn(0x7fb33c0c1ce0 0x7fb33c14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:05:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb340f43700 1 --2- 172.21.15.7:0/1054925816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c072250 0x7fb33c14e1f0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0714c0 tx=0x7fb33800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1394b37ac0d4284b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb333fff700 1 -- 172.21.15.7:0/1054925816 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33800dd30 con 0x7fb33c072250 2023-03-03T11:05:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb333fff700 1 -- 172.21.15.7:0/1054925816 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33800de90 con 0x7fb33c072250 2023-03-03T11:05:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb333fff700 1 -- 172.21.15.7:0/1054925816 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338010690 con 0x7fb33c072250 2023-03-03T11:05:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb3439a8700 1 -- 172.21.15.7:0/1054925816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c072250 msgr2=0x7fb33c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb3439a8700 1 --2- 172.21.15.7:0/1054925816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c072250 0x7fb33c14e1f0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0714c0 tx=0x7fb33800be70 comp rx=0 tx=0).stop 2023-03-03T11:05:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb3439a8700 1 -- 172.21.15.7:0/1054925816 shutdown_connections 2023-03-03T11:05:29.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb3439a8700 1 --2- 172.21.15.7:0/1054925816 >> v2:172.21.15.143:3300/0 conn(0x7fb33c0c1ce0 0x7fb33c14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb3439a8700 1 --2- 172.21.15.7:0/1054925816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb33c071e80 0x7fb33c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb3439a8700 1 --2- 172.21.15.7:0/1054925816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c072250 0x7fb33c14e1f0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb3439a8700 1 -- 172.21.15.7:0/1054925816 >> 172.21.15.7:0/1054925816 conn(0x7fb33c0a94d0 msgr2=0x7fb33c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:29.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb3439a8700 1 -- 172.21.15.7:0/1054925816 shutdown_connections 2023-03-03T11:05:29.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.131+0000 7fb3439a8700 1 -- 172.21.15.7:0/1054925816 wait complete. 2023-03-03T11:05:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.133+0000 7fb3439a8700 1 Processor -- start 2023-03-03T11:05:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.133+0000 7fb3439a8700 1 -- start start 2023-03-03T11:05:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.133+0000 7fb3439a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb33c071e80 0x7fb33c0ab330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb3439a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb33c072250 0x7fb33c142620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb3439a8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c0c1ce0 0x7fb33c142b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb3439a8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0abe50 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb3439a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c1e66c0 con 0x7fb33c072250 2023-03-03T11:05:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb3439a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c1e6390 con 0x7fb33c071e80 2023-03-03T11:05:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb341f45700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c0c1ce0 0x7fb33c142b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb341f45700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c0c1ce0 0x7fb33c142b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34754/0 (socket says 172.21.15.7:34754) 2023-03-03T11:05:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb341f45700 1 -- 172.21.15.7:0/622964488 learned_addr learned my addr 172.21.15.7:0/622964488 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:05:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb340f43700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb33c072250 0x7fb33c142620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb341744700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb33c071e80 0x7fb33c0ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb341f45700 1 -- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb33c071e80 msgr2=0x7fb33c0ab330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb341f45700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb33c071e80 0x7fb33c0ab330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb341f45700 1 -- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb33c072250 msgr2=0x7fb33c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb341f45700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb33c072250 0x7fb33c142620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb341f45700 1 -- 172.21.15.7:0/622964488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33800b7f0 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb341f45700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c0c1ce0 0x7fb33c142b60 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb33400cd80 tx=0x7fb33400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb331ffb700 1 -- 172.21.15.7:0/622964488 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33400f480 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb331ffb700 1 -- 172.21.15.7:0/622964488 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33400f600 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb3439a8700 1 -- 172.21.15.7:0/622964488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33c06c9a0 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.134+0000 7fb331ffb700 1 -- 172.21.15.7:0/622964488 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33400f9b0 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.135+0000 7fb3439a8700 1 -- 172.21.15.7:0/622964488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33c151300 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.136+0000 7fb3439a8700 1 -- 172.21.15.7:0/622964488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33c06cb30 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.136+0000 7fb331ffb700 1 -- 172.21.15.7:0/622964488 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb334009030 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.136+0000 7fb331ffb700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3200682f0 0x7fb32006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.136+0000 7fb331ffb700 1 -- 172.21.15.7:0/622964488 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb334086a80 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.137+0000 7fb341744700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3200682f0 0x7fb32006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.137+0000 7fb341744700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3200682f0 0x7fb32006a7a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb33c070cb0 tx=0x7fb32c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:29.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.141+0000 7fb331ffb700 1 -- 172.21.15.7:0/622964488 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb334052d30 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.384+0000 7fb3439a8700 1 -- 172.21.15.7:0/622964488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb33c074bb0 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.384+0000 7fb331ffb700 1 -- 172.21.15.7:0/622964488 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb334056350 con 0x7fb33c0c1ce0 2023-03-03T11:05:29.387 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:05:29.387 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:05:29.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 -- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3200682f0 msgr2=0x7fb32006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:29.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3200682f0 0x7fb32006a7a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb33c070cb0 tx=0x7fb32c008040 comp rx=0 tx=0).stop 2023-03-03T11:05:29.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 -- 172.21.15.7:0/622964488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c0c1ce0 msgr2=0x7fb33c142b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:29.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c0c1ce0 0x7fb33c142b60 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fb33400cd80 tx=0x7fb33400a560 comp rx=0 tx=0).stop 2023-03-03T11:05:29.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 -- 172.21.15.7:0/622964488 shutdown_connections 2023-03-03T11:05:29.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3200682f0 0x7fb32006a7a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb33c0c1ce0 0x7fb33c142b60 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb33c071e80 0x7fb33c0ab330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 --2- 172.21.15.7:0/622964488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb33c072250 0x7fb33c142620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 -- 172.21.15.7:0/622964488 >> 172.21.15.7:0/622964488 conn(0x7fb33c0a94d0 msgr2=0x7fb33c1472c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 -- 172.21.15.7:0/622964488 shutdown_connections 2023-03-03T11:05:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:29.386+0000 7fb3439a8700 1 -- 172.21.15.7:0/622964488 wait complete. 2023-03-03T11:05:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:05:29.400 INFO:tasks.mds_thrash.fs.[b]:waiting for 24 secs before thrashing 2023-03-03T11:05:31.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:05:31.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.171+0000 7f21a3afe700 1 Processor -- start 2023-03-03T11:05:31.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.171+0000 7f21a3afe700 1 -- start start 2023-03-03T11:05:31.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.172+0000 7f21a3afe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f219c14de50 0x7f219c06a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:31.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.172+0000 7f21a3afe700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c0a65b0 con 0x7f219c06a7f0 2023-03-03T11:05:31.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.172+0000 7f21a3afe700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c1541f0 con 0x7f219c0c65c0 2023-03-03T11:05:31.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.172+0000 7f21a3afe700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219c154e00 con 0x7f219c14de50 2023-03-03T11:05:31.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.172+0000 7f21a209b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f219c06a7f0 0x7f219c0c4dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56974/0 (socket says 172.21.15.7:56974) 2023-03-03T11:05:31.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.172+0000 7f21a209b700 1 -- 172.21.15.7:0/2700296323 learned_addr learned my addr 172.21.15.7:0/2700296323 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:05:31.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.172+0000 7f21a189a700 1 --2- 172.21.15.7:0/2700296323 >> v2:172.21.15.143:3300/0 conn(0x7f219c14de50 0x7f219c06a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:31.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.173+0000 7f21a0898700 1 -- 172.21.15.7:0/2700296323 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031001922 0 0) 0x7f219c0a65b0 con 0x7f219c06a7f0 2023-03-03T11:05:31.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.173+0000 7f21a0898700 1 -- 172.21.15.7:0/2700296323 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21880035b0 con 0x7f219c06a7f0 2023-03-03T11:05:31.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.173+0000 7f21a189a700 1 -- 172.21.15.7:0/2700296323 >> v1:172.21.15.143:6789/0 conn(0x7f219c0c65c0 legacy=0x7f219c0c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:31.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.173+0000 7f21a0898700 1 -- 172.21.15.7:0/2700296323 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129066847 0 0) 0x7f219c1541f0 con 0x7f219c0c65c0 2023-03-03T11:05:31.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.173+0000 7f21a189a700 1 -- 172.21.15.7:0/2700296323 >> v1:172.21.15.7:6789/0 conn(0x7f219c06a7f0 legacy=0x7f219c0c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:31.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.173+0000 7f21a189a700 1 -- 172.21.15.7:0/2700296323 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c1544d0 con 0x7f219c14de50 2023-03-03T11:05:31.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.173+0000 7f21a189a700 1 --2- 172.21.15.7:0/2700296323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219c14de50 0x7f219c06a2b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2198009840 tx=0x7f2198009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dea1756ef9c733f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.173+0000 7f21a0898700 1 -- 172.21.15.7:0/2700296323 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198011830 con 0x7f219c14de50 2023-03-03T11:05:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.173+0000 7f21a0898700 1 -- 172.21.15.7:0/2700296323 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2198011990 con 0x7f219c14de50 2023-03-03T11:05:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.173+0000 7f21a0898700 1 -- 172.21.15.7:0/2700296323 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198011d20 con 0x7f219c14de50 2023-03-03T11:05:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.174+0000 7f21a3afe700 1 -- 172.21.15.7:0/2700296323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219c14de50 msgr2=0x7f219c06a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.174+0000 7f21a3afe700 1 --2- 172.21.15.7:0/2700296323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219c14de50 0x7f219c06a2b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2198009840 tx=0x7f2198009b50 comp rx=0 tx=0).stop 2023-03-03T11:05:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.174+0000 7f21a3afe700 1 -- 172.21.15.7:0/2700296323 shutdown_connections 2023-03-03T11:05:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.174+0000 7f21a3afe700 1 --2- 172.21.15.7:0/2700296323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219c14de50 0x7f219c06a2b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.174+0000 7f21a3afe700 1 -- 172.21.15.7:0/2700296323 >> 172.21.15.7:0/2700296323 conn(0x7f219c0a9430 msgr2=0x7f219c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:31.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.174+0000 7f21a3afe700 1 -- 172.21.15.7:0/2700296323 shutdown_connections 2023-03-03T11:05:31.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.174+0000 7f21a3afe700 1 -- 172.21.15.7:0/2700296323 wait complete. 2023-03-03T11:05:31.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.175+0000 7f21a3afe700 1 Processor -- start 2023-03-03T11:05:31.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.175+0000 7f21a3afe700 1 -- start start 2023-03-03T11:05:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.176+0000 7f21a3afe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219c06a7f0 0x7f219c1eacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.176+0000 7f21a3afe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f219c0c65c0 0x7f219c143760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.176+0000 7f21a3afe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219c14de50 0x7f219c0c2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.176+0000 7f21a3afe700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c0c1a90 con 0x7f219c0c65c0 2023-03-03T11:05:31.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.176+0000 7f21a3afe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c1eb1f0 con 0x7f219c06a7f0 2023-03-03T11:05:31.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.176+0000 7f21a3afe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219c152d00 con 0x7f219c14de50 2023-03-03T11:05:31.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.176+0000 7f21a1099700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f219c0c65c0 0x7f219c143760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:31.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a1099700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f219c0c65c0 0x7f219c143760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34766/0 (socket says 172.21.15.7:34766) 2023-03-03T11:05:31.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a1099700 1 -- 172.21.15.7:0/286504626 learned_addr learned my addr 172.21.15.7:0/286504626 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:05:31.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a189a700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219c06a7f0 0x7f219c1eacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:31.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a209b700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219c14de50 0x7f219c0c2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:31.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a1099700 1 -- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219c14de50 msgr2=0x7f219c0c2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:31.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a1099700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219c14de50 0x7f219c0c2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:31.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a1099700 1 -- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219c06a7f0 msgr2=0x7f219c1eacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:31.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a1099700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219c06a7f0 0x7f219c1eacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:31.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a1099700 1 -- 172.21.15.7:0/286504626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21980094a0 con 0x7f219c0c65c0 2023-03-03T11:05:31.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a209b700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219c14de50 0x7f219c0c2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:05:31.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a1099700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f219c0c65c0 0x7f219c143760 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f218c00e960 tx=0x7f218c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:31.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21927fc700 1 -- 172.21.15.7:0/286504626 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c005270 con 0x7f219c0c65c0 2023-03-03T11:05:31.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21927fc700 1 -- 172.21.15.7:0/286504626 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218c0053f0 con 0x7f219c0c65c0 2023-03-03T11:05:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a3afe700 1 -- 172.21.15.7:0/286504626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f219c154a90 con 0x7f219c0c65c0 2023-03-03T11:05:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21927fc700 1 -- 172.21.15.7:0/286504626 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c005560 con 0x7f219c0c65c0 2023-03-03T11:05:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.177+0000 7f21a3afe700 1 -- 172.21.15.7:0/286504626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f219c1547b0 con 0x7f219c0c65c0 2023-03-03T11:05:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.178+0000 7f21a3afe700 1 -- 172.21.15.7:0/286504626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219c0c2920 con 0x7f219c0c65c0 2023-03-03T11:05:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.179+0000 7f21927fc700 1 -- 172.21.15.7:0/286504626 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f218c021070 con 0x7f219c0c65c0 2023-03-03T11:05:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.179+0000 7f21927fc700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21800682f0 0x7f218006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:31.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.179+0000 7f21927fc700 1 -- 172.21.15.7:0/286504626 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f218c088280 con 0x7f219c0c65c0 2023-03-03T11:05:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.179+0000 7f21a189a700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21800682f0 0x7f218006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.180+0000 7f21a189a700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21800682f0 0x7f218006a7a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2198009fd0 tx=0x7f219801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:31.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.183+0000 7f21927fc700 1 -- 172.21.15.7:0/286504626 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f219c0c2920 con 0x7f219c0c65c0 2023-03-03T11:05:31.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.424+0000 7f21a3afe700 1 -- 172.21.15.7:0/286504626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f219c0c2920 con 0x7f219c0c65c0 2023-03-03T11:05:31.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.425+0000 7f21927fc700 1 -- 172.21.15.7:0/286504626 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f219c0c2920 con 0x7f219c0c65c0 2023-03-03T11:05:31.427 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:05:31.428 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:05:31.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 -- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21800682f0 msgr2=0x7f218006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:31.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21800682f0 0x7f218006a7a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2198009fd0 tx=0x7f219801c040 comp rx=0 tx=0).stop 2023-03-03T11:05:31.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 -- 172.21.15.7:0/286504626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f219c0c65c0 msgr2=0x7f219c143760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:31.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f219c0c65c0 0x7f219c143760 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f218c00e960 tx=0x7f218c00ec70 comp rx=0 tx=0).stop 2023-03-03T11:05:31.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 -- 172.21.15.7:0/286504626 shutdown_connections 2023-03-03T11:05:31.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f219c14de50 0x7f219c0c2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:31.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f219c06a7f0 0x7f219c1eacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:31.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21800682f0 0x7f218006a7a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:31.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 --2- 172.21.15.7:0/286504626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f219c0c65c0 0x7f219c143760 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:31.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 -- 172.21.15.7:0/286504626 >> 172.21.15.7:0/286504626 conn(0x7f219c0a9430 msgr2=0x7f219c140f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:31.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 -- 172.21.15.7:0/286504626 shutdown_connections 2023-03-03T11:05:31.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:31.427+0000 7f21a3afe700 1 -- 172.21.15.7:0/286504626 wait complete. 2023-03-03T11:05:31.431 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:05:31.441 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T11:05:38.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:38.419 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:05:41.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:05:41.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.577+0000 7f2f27e31700 1 Processor -- start 2023-03-03T11:05:41.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.577+0000 7f2f27e31700 1 -- start start 2023-03-03T11:05:41.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.578+0000 7f2f27e31700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2f200c2e30 0x7f2f20158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:41.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.578+0000 7f2f27e31700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f20157ff0 con 0x7f2f20151cd0 2023-03-03T11:05:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.578+0000 7f2f27e31700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2006d640 con 0x7f2f2006c6c0 2023-03-03T11:05:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.578+0000 7f2f27e31700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2006cda0 con 0x7f2f200c2e30 2023-03-03T11:05:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.578+0000 7f2f25bcd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2f20151cd0 0x7f2f20154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56984/0 (socket says 172.21.15.7:56984) 2023-03-03T11:05:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.578+0000 7f2f25bcd700 1 -- 172.21.15.7:0/3782761646 learned_addr learned my addr 172.21.15.7:0/3782761646 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:05:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.578+0000 7f2f253cc700 1 --2- 172.21.15.7:0/3782761646 >> v2:172.21.15.143:3300/0 conn(0x7f2f200c2e30 0x7f2f20158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.578+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773860659 0 0) 0x7f2f20157ff0 con 0x7f2f20151cd0 2023-03-03T11:05:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.578+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f0c0035b0 con 0x7f2f20151cd0 2023-03-03T11:05:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2877232068 0 0) 0x7f2f0c0035b0 con 0x7f2f20151cd0 2023-03-03T11:05:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 >> v2:172.21.15.143:3300/0 conn(0x7f2f200c2e30 msgr2=0x7f2f20158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:05:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 --2- 172.21.15.7:0/3782761646 >> v2:172.21.15.143:3300/0 conn(0x7f2f200c2e30 0x7f2f20158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 >> v1:172.21.15.143:6789/0 conn(0x7f2f2006c6c0 legacy=0x7f2f200736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2006d080 con 0x7f2f20151cd0 2023-03-03T11:05:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f2f10001670 con 0x7f2f20151cd0 2023-03-03T11:05:41.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f253cc700 1 --2- 172.21.15.7:0/3782761646 >> v2:172.21.15.143:3300/0 conn(0x7f2f200c2e30 0x7f2f20158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:05:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 >> v1:172.21.15.7:6789/0 conn(0x7f2f20151cd0 legacy=0x7f2f20154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f0c003a70 0x7f2f0c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f0c004850 0x7f2f0c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f0c005250 0x7f2f0c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2006cda0 con 0x7f2f0c003a70 2023-03-03T11:05:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c00c190 con 0x7f2f0c004850 2023-03-03T11:05:41.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0c00c350 con 0x7f2f0c005250 2023-03-03T11:05:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f25bcd700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f0c003a70 0x7f2f0c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f253cc700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f0c004850 0x7f2f0c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f263ce700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f0c005250 0x7f2f0c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f25bcd700 1 -- 172.21.15.7:0/3782761646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f0c005250 msgr2=0x7f2f0c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:05:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f25bcd700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f0c005250 0x7f2f0c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f25bcd700 1 -- 172.21.15.7:0/3782761646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f0c004850 msgr2=0x7f2f0c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:05:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f25bcd700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f0c004850 0x7f2f0c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:41.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f25bcd700 1 -- 172.21.15.7:0/3782761646 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c0035b0 con 0x7f2f0c003a70 2023-03-03T11:05:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f263ce700 1 -- 172.21.15.7:0/3782761646 reap_dead start 2023-03-03T11:05:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.579+0000 7f2f25bcd700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f0c003a70 0x7f2f0c003e80 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000c210 tx=0x7f2f0c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5c2023f5147ab2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.580+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10007c10 con 0x7f2f0c003a70 2023-03-03T11:05:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.580+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1000cdd0 con 0x7f2f0c003a70 2023-03-03T11:05:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.580+0000 7f2f24bcb700 1 -- 172.21.15.7:0/3782761646 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10005ba0 con 0x7f2f0c003a70 2023-03-03T11:05:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.580+0000 7f2f27e31700 1 -- 172.21.15.7:0/3782761646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f0c003a70 msgr2=0x7f2f0c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.580+0000 7f2f27e31700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f0c003a70 0x7f2f0c003e80 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f2f1000c210 tx=0x7f2f0c005880 comp rx=0 tx=0).stop 2023-03-03T11:05:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.580+0000 7f2f27e31700 1 -- 172.21.15.7:0/3782761646 shutdown_connections 2023-03-03T11:05:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.580+0000 7f2f27e31700 1 --2- 172.21.15.7:0/3782761646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f0c003a70 0x7f2f0c003e80 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.580+0000 7f2f27e31700 1 -- 172.21.15.7:0/3782761646 >> 172.21.15.7:0/3782761646 conn(0x7f2f200a94d0 msgr2=0x7f2f200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.580+0000 7f2f27e31700 1 -- 172.21.15.7:0/3782761646 shutdown_connections 2023-03-03T11:05:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.580+0000 7f2f27e31700 1 -- 172.21.15.7:0/3782761646 wait complete. 2023-03-03T11:05:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.581+0000 7f2f27e31700 1 Processor -- start 2023-03-03T11:05:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f27e31700 1 -- start start 2023-03-03T11:05:41.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f27e31700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f0c003a70 0x7f2f20071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:41.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f27e31700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f2014a3d0 0x7f2f201dfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:41.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f27e31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f201e0340 0x7f2f201e9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:41.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f27e31700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f201ed020 con 0x7f2f201e0340 2023-03-03T11:05:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f27e31700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f201ed2c0 con 0x7f2f2014a3d0 2023-03-03T11:05:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f27e31700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f201ed5a0 con 0x7f2f0c003a70 2023-03-03T11:05:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f263ce700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f201e0340 0x7f2f201e9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f263ce700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f201e0340 0x7f2f201e9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34788/0 (socket says 172.21.15.7:34788) 2023-03-03T11:05:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f263ce700 1 -- 172.21.15.7:0/1673345658 learned_addr learned my addr 172.21.15.7:0/1673345658 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:05:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f25bcd700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f0c003a70 0x7f2f20071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.582+0000 7f2f253cc700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f2014a3d0 0x7f2f201dfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f263ce700 1 -- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f0c003a70 msgr2=0x7f2f20071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f263ce700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f0c003a70 0x7f2f20071760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f263ce700 1 -- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f2014a3d0 msgr2=0x7f2f201dfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f263ce700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f2014a3d0 0x7f2f201dfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f263ce700 1 -- 172.21.15.7:0/1673345658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f20157c80 con 0x7f2f201e0340 2023-03-03T11:05:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f263ce700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f201e0340 0x7f2f201e9d70 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f2f18000f80 tx=0x7f2f1800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f167fc700 1 -- 172.21.15.7:0/1673345658 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1800eee0 con 0x7f2f201e0340 2023-03-03T11:05:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f167fc700 1 -- 172.21.15.7:0/1673345658 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f18003c10 con 0x7f2f201e0340 2023-03-03T11:05:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f27e31700 1 -- 172.21.15.7:0/1673345658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2006d360 con 0x7f2f201e0340 2023-03-03T11:05:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f167fc700 1 -- 172.21.15.7:0/1673345658 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f180050d0 con 0x7f2f201e0340 2023-03-03T11:05:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.583+0000 7f2f27e31700 1 -- 172.21.15.7:0/1673345658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f20154510 con 0x7f2f201e0340 2023-03-03T11:05:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.584+0000 7f2f27e31700 1 -- 172.21.15.7:0/1673345658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f201eaa80 con 0x7f2f201e0340 2023-03-03T11:05:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.585+0000 7f2f167fc700 1 -- 172.21.15.7:0/1673345658 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f18005230 con 0x7f2f201e0340 2023-03-03T11:05:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.585+0000 7f2f167fc700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f040682f0 0x7f2f0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.585+0000 7f2f167fc700 1 -- 172.21.15.7:0/1673345658 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2f180861f0 con 0x7f2f201e0340 2023-03-03T11:05:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.585+0000 7f2f25bcd700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f040682f0 0x7f2f0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.586+0000 7f2f25bcd700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f040682f0 0x7f2f0406a7a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2f10000c00 tx=0x7f2f10007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.589+0000 7f2f167fc700 1 -- 172.21.15.7:0/1673345658 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f201eaa80 con 0x7f2f201e0340 2023-03-03T11:05:41.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.827+0000 7f2f27e31700 1 -- 172.21.15.7:0/1673345658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f201eaa80 con 0x7f2f201e0340 2023-03-03T11:05:41.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.827+0000 7f2f167fc700 1 -- 172.21.15.7:0/1673345658 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2f201eaa80 con 0x7f2f201e0340 2023-03-03T11:05:41.831 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:05:41.831 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:05:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 -- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f040682f0 msgr2=0x7f2f0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f040682f0 0x7f2f0406a7a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2f10000c00 tx=0x7f2f10007000 comp rx=0 tx=0).stop 2023-03-03T11:05:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 -- 172.21.15.7:0/1673345658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f201e0340 msgr2=0x7f2f201e9d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f201e0340 0x7f2f201e9d70 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f2f18000f80 tx=0x7f2f1800ed50 comp rx=0 tx=0).stop 2023-03-03T11:05:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 -- 172.21.15.7:0/1673345658 shutdown_connections 2023-03-03T11:05:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f040682f0 0x7f2f0406a7a0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f201e0340 0x7f2f201e9d70 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f0c003a70 0x7f2f20071760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 --2- 172.21.15.7:0/1673345658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f2014a3d0 0x7f2f201dfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 -- 172.21.15.7:0/1673345658 >> 172.21.15.7:0/1673345658 conn(0x7f2f200a94d0 msgr2=0x7f2f2006a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 -- 172.21.15.7:0/1673345658 shutdown_connections 2023-03-03T11:05:41.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:41.830+0000 7f2f27e31700 1 -- 172.21.15.7:0/1673345658 wait complete. 2023-03-03T11:05:41.835 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:05:41.845 INFO:tasks.mds_thrash.fs.[a]:waiting for 108 secs before thrashing 2023-03-03T11:05:53.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:05:53.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.539+0000 7f556843f700 1 Processor -- start 2023-03-03T11:05:53.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.539+0000 7f556843f700 1 -- start start 2023-03-03T11:05:53.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f556843f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f556006d870 0x7f55601550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f556843f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55600c3e30 0x7f55600c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f556843f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55600c7a60 con 0x7f55600c1da0 2023-03-03T11:05:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f556843f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55600c2af0 con 0x7f556006d870 2023-03-03T11:05:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f556843f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55600c3620 con 0x7f55600c3e30 2023-03-03T11:05:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f55659da700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f55600c1da0 0x7f55600c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57010/0 (socket says 172.21.15.7:57010) 2023-03-03T11:05:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f55659da700 1 -- 172.21.15.7:0/3008799796 learned_addr learned my addr 172.21.15.7:0/3008799796 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:05:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f55669dc700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55600c3e30 0x7f55600c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f55661db700 1 --2- 172.21.15.7:0/3008799796 >> v2:172.21.15.143:3300/0 conn(0x7f556006d870 0x7f55601550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674099091 0 0) 0x7f55600c7a60 con 0x7f55600c1da0 2023-03-03T11:05:53.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.540+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f554c0035b0 con 0x7f55600c1da0 2023-03-03T11:05:53.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3442476862 0 0) 0x7f554c0035b0 con 0x7f55600c1da0 2023-03-03T11:05:53.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55600c3e30 msgr2=0x7f55600c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:05:53.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55600c3e30 0x7f55600c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 >> v2:172.21.15.143:3300/0 conn(0x7f556006d870 msgr2=0x7f55601550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:05:53.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 --2- 172.21.15.7:0/3008799796 >> v2:172.21.15.143:3300/0 conn(0x7f556006d870 0x7f55601550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55601545e0 con 0x7f55600c1da0 2023-03-03T11:05:53.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f555c001670 con 0x7f55600c1da0 2023-03-03T11:05:53.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 >> v1:172.21.15.7:6789/0 conn(0x7f55600c1da0 legacy=0x7f55600c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f554c003a70 0x7f554c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c004850 0x7f554c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55600c3e30 0x7f554c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600c3620 con 0x7f554c004850 2023-03-03T11:05:53.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c014220 con 0x7f554c003a70 2023-03-03T11:05:53.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c014390 con 0x7f55600c3e30 2023-03-03T11:05:53.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55659da700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c004850 0x7f554c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55661db700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f554c003a70 0x7f554c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55669dc700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55600c3e30 0x7f554c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55659da700 1 -- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55600c3e30 msgr2=0x7f554c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:05:53.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55659da700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55600c3e30 0x7f554c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55659da700 1 -- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f554c003a70 msgr2=0x7f554c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:05:53.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55659da700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f554c003a70 0x7f554c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55659da700 1 -- 172.21.15.7:0/3008799796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c0035b0 con 0x7f554c004850 2023-03-03T11:05:53.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.541+0000 7f55659da700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c004850 0x7f554c004cc0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f555c00c320 tx=0x7f554c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa192a9d2c3d26d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:53.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.542+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c00ce60 con 0x7f554c004850 2023-03-03T11:05:53.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.542+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c005960 con 0x7f554c004850 2023-03-03T11:05:53.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.542+0000 7f55651d9700 1 -- 172.21.15.7:0/3008799796 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c005d10 con 0x7f554c004850 2023-03-03T11:05:53.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.542+0000 7f556843f700 1 -- 172.21.15.7:0/3008799796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c004850 msgr2=0x7f554c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.542+0000 7f556843f700 1 --2- 172.21.15.7:0/3008799796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f554c004850 0x7f554c004cc0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f555c00c320 tx=0x7f554c005880 comp rx=0 tx=0).stop 2023-03-03T11:05:53.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.542+0000 7f55669dc700 1 -- 172.21.15.7:0/3008799796 reap_dead start 2023-03-03T11:05:53.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.542+0000 7f556843f700 1 -- 172.21.15.7:0/3008799796 shutdown_connections 2023-03-03T11:05:53.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.542+0000 7f556843f700 1 -- 172.21.15.7:0/3008799796 >> 172.21.15.7:0/3008799796 conn(0x7f556006c110 msgr2=0x7f556006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:53.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.542+0000 7f556843f700 1 -- 172.21.15.7:0/3008799796 shutdown_connections 2023-03-03T11:05:53.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.542+0000 7f556843f700 1 -- 172.21.15.7:0/3008799796 wait complete. 2023-03-03T11:05:53.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.543+0000 7f556843f700 1 Processor -- start 2023-03-03T11:05:53.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f556843f700 1 -- start start 2023-03-03T11:05:53.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f556843f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f554c004850 0x7f556013b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f556843f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5560138b00 0x7f55601e6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f556843f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556013bb60 0x7f55601e6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f556843f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55600ad0a0 con 0x7f556013bb60 2023-03-03T11:05:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f556843f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560137d70 con 0x7f554c004850 2023-03-03T11:05:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f556843f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5560138190 con 0x7f5560138b00 2023-03-03T11:05:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f55669dc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556013bb60 0x7f55601e6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f55669dc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556013bb60 0x7f55601e6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34814/0 (socket says 172.21.15.7:34814) 2023-03-03T11:05:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f55669dc700 1 -- 172.21.15.7:0/1644370326 learned_addr learned my addr 172.21.15.7:0/1644370326 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:05:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f55659da700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5560138b00 0x7f55601e6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.544+0000 7f55661db700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f554c004850 0x7f556013b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f55669dc700 1 -- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5560138b00 msgr2=0x7f55601e6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f55669dc700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5560138b00 0x7f55601e6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f55669dc700 1 -- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f554c004850 msgr2=0x7f556013b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f55669dc700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f554c004850 0x7f556013b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f55669dc700 1 -- 172.21.15.7:0/1644370326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5560154ba0 con 0x7f556013bb60 2023-03-03T11:05:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f55669dc700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556013bb60 0x7f55601e6da0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f556006c0a0 tx=0x7f5558011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f5556ffd700 1 -- 172.21.15.7:0/1644370326 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555800e040 con 0x7f556013bb60 2023-03-03T11:05:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f5556ffd700 1 -- 172.21.15.7:0/1644370326 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555800f040 con 0x7f556013bb60 2023-03-03T11:05:53.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f556843f700 1 -- 172.21.15.7:0/1644370326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55601548c0 con 0x7f556013bb60 2023-03-03T11:05:53.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f5556ffd700 1 -- 172.21.15.7:0/1644370326 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555800fd70 con 0x7f556013bb60 2023-03-03T11:05:53.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.545+0000 7f556843f700 1 -- 172.21.15.7:0/1644370326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556006dc40 con 0x7f556013bb60 2023-03-03T11:05:53.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.546+0000 7f556843f700 1 -- 172.21.15.7:0/1644370326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556014b460 con 0x7f556013bb60 2023-03-03T11:05:53.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.546+0000 7f5556ffd700 1 -- 172.21.15.7:0/1644370326 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5558015430 con 0x7f556013bb60 2023-03-03T11:05:53.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.547+0000 7f5556ffd700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55440682f0 0x7f554406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.547+0000 7f5556ffd700 1 -- 172.21.15.7:0/1644370326 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f55580868e0 con 0x7f556013bb60 2023-03-03T11:05:53.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.547+0000 7f55661db700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55440682f0 0x7f554406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.548+0000 7f55661db700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55440682f0 0x7f554406a7a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5550009be0 tx=0x7f5550009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:53.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.551+0000 7f5556ffd700 1 -- 172.21.15.7:0/1644370326 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5558052c20 con 0x7f556013bb60 2023-03-03T11:05:53.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.790+0000 7f556843f700 1 -- 172.21.15.7:0/1644370326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5560074bb0 con 0x7f556013bb60 2023-03-03T11:05:53.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.790+0000 7f5556ffd700 1 -- 172.21.15.7:0/1644370326 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5558056240 con 0x7f556013bb60 2023-03-03T11:05:53.793 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:05:53.793 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:05:53.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 -- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55440682f0 msgr2=0x7f554406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55440682f0 0x7f554406a7a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5550009be0 tx=0x7f5550009210 comp rx=0 tx=0).stop 2023-03-03T11:05:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 -- 172.21.15.7:0/1644370326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556013bb60 msgr2=0x7f55601e6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556013bb60 0x7f55601e6da0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f556006c0a0 tx=0x7f5558011f60 comp rx=0 tx=0).stop 2023-03-03T11:05:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 -- 172.21.15.7:0/1644370326 shutdown_connections 2023-03-03T11:05:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55440682f0 0x7f554406a7a0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f556013bb60 0x7f55601e6da0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f554c004850 0x7f556013b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 --2- 172.21.15.7:0/1644370326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5560138b00 0x7f55601e6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 -- 172.21.15.7:0/1644370326 >> 172.21.15.7:0/1644370326 conn(0x7f556006c110 msgr2=0x7f5560071a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:53.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 -- 172.21.15.7:0/1644370326 shutdown_connections 2023-03-03T11:05:53.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.792+0000 7f556843f700 1 -- 172.21.15.7:0/1644370326 wait complete. 2023-03-03T11:05:53.796 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:05:53.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:05:53.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.928+0000 7f6bf6a9a700 1 Processor -- start 2023-03-03T11:05:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.928+0000 7f6bf6a9a700 1 -- start start 2023-03-03T11:05:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.929+0000 7f6bf6a9a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf006c480 0x7f6bf01581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.929+0000 7f6bf6a9a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf006d430 con 0x7f6bf00c2930 2023-03-03T11:05:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.929+0000 7f6bf6a9a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf01586e0 con 0x7f6bf0151d40 2023-03-03T11:05:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.929+0000 7f6bf6a9a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf006c2a0 con 0x7f6bf006c480 2023-03-03T11:05:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.929+0000 7f6beffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6bf00c2930 0x7f6bf0157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57030/0 (socket says 172.21.15.7:57030) 2023-03-03T11:05:53.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.929+0000 7f6beffff700 1 -- 172.21.15.7:0/2121991469 learned_addr learned my addr 172.21.15.7:0/2121991469 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:05:53.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bf5037700 1 --2- 172.21.15.7:0/2121991469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf006c480 0x7f6bf01581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bef7fe700 1 -- 172.21.15.7:0/2121991469 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340807441 0 0) 0x7f6bf006d430 con 0x7f6bf00c2930 2023-03-03T11:05:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bef7fe700 1 -- 172.21.15.7:0/2121991469 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6be00035b0 con 0x7f6bf00c2930 2023-03-03T11:05:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bef7fe700 1 -- 172.21.15.7:0/2121991469 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 766599109 0 0) 0x7f6be00035b0 con 0x7f6bf00c2930 2023-03-03T11:05:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bf5037700 1 -- 172.21.15.7:0/2121991469 >> v1:172.21.15.143:6789/0 conn(0x7f6bf0151d40 legacy=0x7f6bf00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bf5037700 1 -- 172.21.15.7:0/2121991469 >> v1:172.21.15.7:6789/0 conn(0x7f6bf00c2930 legacy=0x7f6bf0157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bf5037700 1 -- 172.21.15.7:0/2121991469 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf006cb00 con 0x7f6bf006c480 2023-03-03T11:05:53.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bf5037700 1 --2- 172.21.15.7:0/2121991469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf006c480 0x7f6bf01581a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009dc0 tx=0x7f6be800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98bdfa5fde0be7f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bef7fe700 1 -- 172.21.15.7:0/2121991469 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be800e070 con 0x7f6bf006c480 2023-03-03T11:05:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bef7fe700 1 -- 172.21.15.7:0/2121991469 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be8003a20 con 0x7f6bf006c480 2023-03-03T11:05:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.930+0000 7f6bef7fe700 1 -- 172.21.15.7:0/2121991469 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc001670 con 0x7f6bf006c480 2023-03-03T11:05:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.931+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2121991469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf006c480 msgr2=0x7f6bf01581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.931+0000 7f6bf6a9a700 1 --2- 172.21.15.7:0/2121991469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf006c480 0x7f6bf01581a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009dc0 tx=0x7f6be800b330 comp rx=0 tx=0).stop 2023-03-03T11:05:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.931+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2121991469 shutdown_connections 2023-03-03T11:05:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.931+0000 7f6bf6a9a700 1 --2- 172.21.15.7:0/2121991469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf006c480 0x7f6bf01581a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.931+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2121991469 >> 172.21.15.7:0/2121991469 conn(0x7f6bf00a94d0 msgr2=0x7f6bf00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.931+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2121991469 shutdown_connections 2023-03-03T11:05:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.931+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2121991469 wait complete. 2023-03-03T11:05:53.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.932+0000 7f6bf6a9a700 1 Processor -- start 2023-03-03T11:05:53.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.932+0000 7f6bf6a9a700 1 -- start start 2023-03-03T11:05:53.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf6a9a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf006c480 0x7f6bf014bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf6a9a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf00c2930 0x7f6bf014c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf6a9a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf0151d40 0x7f6bf00c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf6a9a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf014c940 con 0x7f6bf0151d40 2023-03-03T11:05:53.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf6a9a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf00be4d0 con 0x7f6bf006c480 2023-03-03T11:05:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf6a9a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf00be7b0 con 0x7f6bf00c2930 2023-03-03T11:05:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf5037700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf0151d40 0x7f6bf00c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf5037700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf0151d40 0x7f6bf00c0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34826/0 (socket says 172.21.15.7:34826) 2023-03-03T11:05:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf5037700 1 -- 172.21.15.7:0/2828661855 learned_addr learned my addr 172.21.15.7:0/2828661855 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:05:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6beffff700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf00c2930 0x7f6bf014c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf4836700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf006c480 0x7f6bf014bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf5037700 1 -- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf00c2930 msgr2=0x7f6bf014c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf5037700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf00c2930 0x7f6bf014c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf5037700 1 -- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf006c480 msgr2=0x7f6bf014bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf5037700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf006c480 0x7f6bf014bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf5037700 1 -- 172.21.15.7:0/2828661855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8009a70 con 0x7f6bf0151d40 2023-03-03T11:05:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6beffff700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf00c2930 0x7f6bf014c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:05:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.933+0000 7f6bf4836700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf006c480 0x7f6bf014bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:05:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.934+0000 7f6bf5037700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf0151d40 0x7f6bf00c0130 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f6be800b370 tx=0x7f6be8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:53.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.934+0000 7f6bed7fa700 1 -- 172.21.15.7:0/2828661855 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be80045c0 con 0x7f6bf0151d40 2023-03-03T11:05:53.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.934+0000 7f6bed7fa700 1 -- 172.21.15.7:0/2828661855 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be8004720 con 0x7f6bf0151d40 2023-03-03T11:05:53.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.934+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2828661855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf006d0c0 con 0x7f6bf0151d40 2023-03-03T11:05:53.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.934+0000 7f6bed7fa700 1 -- 172.21.15.7:0/2828661855 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8012450 con 0x7f6bf0151d40 2023-03-03T11:05:53.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.934+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2828661855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf006cde0 con 0x7f6bf0151d40 2023-03-03T11:05:53.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.935+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2828661855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf0073e00 con 0x7f6bf0151d40 2023-03-03T11:05:53.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.935+0000 7f6bed7fa700 1 -- 172.21.15.7:0/2828661855 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6be8024030 con 0x7f6bf0151d40 2023-03-03T11:05:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.936+0000 7f6bed7fa700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bd40682f0 0x7f6bd406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:05:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.936+0000 7f6bed7fa700 1 -- 172.21.15.7:0/2828661855 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6be80879c0 con 0x7f6bf0151d40 2023-03-03T11:05:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.936+0000 7f6bf4836700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bd40682f0 0x7f6bd406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:05:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.937+0000 7f6bf4836700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bd40682f0 0x7f6bd406a7a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6be4000f60 tx=0x7f6be4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:05:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:53.940+0000 7f6bed7fa700 1 -- 172.21.15.7:0/2828661855 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6be8053c70 con 0x7f6bf0151d40 2023-03-03T11:05:54.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.176+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2828661855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bf00be970 con 0x7f6bf0151d40 2023-03-03T11:05:54.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.177+0000 7f6bed7fa700 1 -- 172.21.15.7:0/2828661855 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6be8057290 con 0x7f6bf0151d40 2023-03-03T11:05:54.179 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:05:54.179 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:05:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bd40682f0 msgr2=0x7f6bd406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bd40682f0 0x7f6bd406a7a0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6be4000f60 tx=0x7f6be4009040 comp rx=0 tx=0).stop 2023-03-03T11:05:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2828661855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf0151d40 msgr2=0x7f6bf00c0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:05:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf0151d40 0x7f6bf00c0130 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f6be800b370 tx=0x7f6be8004ad0 comp rx=0 tx=0).stop 2023-03-03T11:05:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2828661855 shutdown_connections 2023-03-03T11:05:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bd40682f0 0x7f6bd406a7a0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf0151d40 0x7f6bf00c0130 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf006c480 0x7f6bf014bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 --2- 172.21.15.7:0/2828661855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf00c2930 0x7f6bf014c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:05:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2828661855 >> 172.21.15.7:0/2828661855 conn(0x7f6bf00a94d0 msgr2=0x7f6bf01e0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:05:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2828661855 shutdown_connections 2023-03-03T11:05:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:05:54.179+0000 7f6bf6a9a700 1 -- 172.21.15.7:0/2828661855 wait complete. 2023-03-03T11:05:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:05:54.193 INFO:tasks.mds_thrash.fs.[b]:waiting for 49 secs before thrashing 2023-03-03T11:06:08.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:08.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:38.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:38.696 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:06:43.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:06:43.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.331+0000 7fb41d5d6700 1 Processor -- start 2023-03-03T11:06:43.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.331+0000 7fb41d5d6700 1 -- start start 2023-03-03T11:06:43.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.331+0000 7fb41d5d6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180c00e0 0x7fb4180c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:06:43.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.331+0000 7fb41d5d6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb418154ff0 0x7fb4180cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:06:43.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.331+0000 7fb41d5d6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4180c7310 con 0x7fb4180caa30 2023-03-03T11:06:43.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.331+0000 7fb41d5d6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4180bfd10 con 0x7fb418154ff0 2023-03-03T11:06:43.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.331+0000 7fb41d5d6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4180bfa30 con 0x7fb4180c00e0 2023-03-03T11:06:43.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.331+0000 7fb4177fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb418154ff0 0x7fb4180cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:06:43.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb416ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb4180caa30 0x7fb4180c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57060/0 (socket says 172.21.15.7:57060) 2023-03-03T11:06:43.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb4177fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb418154ff0 0x7fb4180cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34852/0 (socket says 172.21.15.7:34852) 2023-03-03T11:06:43.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb416ffd700 1 -- 172.21.15.7:0/512656432 learned_addr learned my addr 172.21.15.7:0/512656432 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:06:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb4167fc700 1 --2- 172.21.15.7:0/512656432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180c00e0 0x7fb4180c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:06:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb4177fe700 1 -- 172.21.15.7:0/512656432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180c00e0 msgr2=0x7fb4180c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:06:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb4177fe700 1 --2- 172.21.15.7:0/512656432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180c00e0 0x7fb4180c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:06:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb4177fe700 1 -- 172.21.15.7:0/512656432 >> v1:172.21.15.7:6789/0 conn(0x7fb4180caa30 legacy=0x7fb4180c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:06:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb4177fe700 1 -- 172.21.15.7:0/512656432 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb418154510 con 0x7fb418154ff0 2023-03-03T11:06:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb415ffb700 1 -- 172.21.15.7:0/512656432 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817687117 0 0) 0x7fb4180c7310 con 0x7fb4180caa30 2023-03-03T11:06:43.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb4167fc700 1 --2- 172.21.15.7:0/512656432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180c00e0 0x7fb4180c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:06:43.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb4177fe700 1 --2- 172.21.15.7:0/512656432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb418154ff0 0x7fb4180cae00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fb410009d90 tx=0x7fb41000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ff04852f09626c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:06:43.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb415ffb700 1 -- 172.21.15.7:0/512656432 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41000e070 con 0x7fb418154ff0 2023-03-03T11:06:43.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb415ffb700 1 -- 172.21.15.7:0/512656432 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41000bbd0 con 0x7fb418154ff0 2023-03-03T11:06:43.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.332+0000 7fb415ffb700 1 -- 172.21.15.7:0/512656432 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb410012430 con 0x7fb418154ff0 2023-03-03T11:06:43.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.333+0000 7fb41d5d6700 1 -- 172.21.15.7:0/512656432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb418154ff0 msgr2=0x7fb4180cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:06:43.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.333+0000 7fb41d5d6700 1 --2- 172.21.15.7:0/512656432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb418154ff0 0x7fb4180cae00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fb410009d90 tx=0x7fb41000b330 comp rx=0 tx=0).stop 2023-03-03T11:06:43.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.333+0000 7fb41d5d6700 1 -- 172.21.15.7:0/512656432 shutdown_connections 2023-03-03T11:06:43.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.333+0000 7fb41d5d6700 1 --2- 172.21.15.7:0/512656432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb418154ff0 0x7fb4180cae00 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:06:43.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.333+0000 7fb41d5d6700 1 --2- 172.21.15.7:0/512656432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4180c00e0 0x7fb4180c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:06:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.333+0000 7fb41d5d6700 1 -- 172.21.15.7:0/512656432 >> 172.21.15.7:0/512656432 conn(0x7fb418066e40 msgr2=0x7fb418071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:06:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.333+0000 7fb41d5d6700 1 -- 172.21.15.7:0/512656432 shutdown_connections 2023-03-03T11:06:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.333+0000 7fb41d5d6700 1 -- 172.21.15.7:0/512656432 wait complete. 2023-03-03T11:06:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.335+0000 7fb41d5d6700 1 Processor -- start 2023-03-03T11:06:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.335+0000 7fb41d5d6700 1 -- start start 2023-03-03T11:06:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb41d5d6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c00e0 0x7fb418158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:06:43.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb41d5d6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180caa30 0x7fb418157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:06:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb41d5d6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb418154ff0 0x7fb4180c2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:06:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb41d5d6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418157eb0 con 0x7fb4180c00e0 2023-03-03T11:06:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb41d5d6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418155ac0 con 0x7fb4180caa30 2023-03-03T11:06:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb41d5d6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4181574e0 con 0x7fb418154ff0 2023-03-03T11:06:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb416ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c00e0 0x7fb418158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:06:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb416ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c00e0 0x7fb418158360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34854/0 (socket says 172.21.15.7:34854) 2023-03-03T11:06:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb416ffd700 1 -- 172.21.15.7:0/2627047787 learned_addr learned my addr 172.21.15.7:0/2627047787 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:06:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb4167fc700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180caa30 0x7fb418157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:06:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb4177fe700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb418154ff0 0x7fb4180c2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:06:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb416ffd700 1 -- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb418154ff0 msgr2=0x7fb4180c2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:06:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb416ffd700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb418154ff0 0x7fb4180c2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:06:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb416ffd700 1 -- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180caa30 msgr2=0x7fb418157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:06:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb416ffd700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180caa30 0x7fb418157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:06:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb416ffd700 1 -- 172.21.15.7:0/2627047787 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb410009a70 con 0x7fb4180c00e0 2023-03-03T11:06:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb4167fc700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180caa30 0x7fb418157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:06:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb4177fe700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb418154ff0 0x7fb4180c2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:06:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb416ffd700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c00e0 0x7fb418158360 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb40800c750 tx=0x7fb40800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:06:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb40f7fe700 1 -- 172.21.15.7:0/2627047787 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408004c80 con 0x7fb4180c00e0 2023-03-03T11:06:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb40f7fe700 1 -- 172.21.15.7:0/2627047787 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb408004e00 con 0x7fb4180c00e0 2023-03-03T11:06:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb41d5d6700 1 -- 172.21.15.7:0/2627047787 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb418154ad0 con 0x7fb4180c00e0 2023-03-03T11:06:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb40f7fe700 1 -- 172.21.15.7:0/2627047787 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408007310 con 0x7fb4180c00e0 2023-03-03T11:06:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.336+0000 7fb41d5d6700 1 -- 172.21.15.7:0/2627047787 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4181547f0 con 0x7fb4180c00e0 2023-03-03T11:06:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.337+0000 7fb41d5d6700 1 -- 172.21.15.7:0/2627047787 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4180bf650 con 0x7fb4180c00e0 2023-03-03T11:06:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.338+0000 7fb40f7fe700 1 -- 172.21.15.7:0/2627047787 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb40801a020 con 0x7fb4180c00e0 2023-03-03T11:06:43.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.338+0000 7fb40f7fe700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3f80682f0 0x7fb3f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:06:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.338+0000 7fb40f7fe700 1 -- 172.21.15.7:0/2627047787 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb408085df0 con 0x7fb4180c00e0 2023-03-03T11:06:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.339+0000 7fb4167fc700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3f80682f0 0x7fb3f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:06:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.339+0000 7fb4167fc700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3f80682f0 0x7fb3f806a7a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb4181df620 tx=0x7fb400008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:06:43.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.343+0000 7fb40f7fe700 1 -- 172.21.15.7:0/2627047787 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb408052190 con 0x7fb4180c00e0 2023-03-03T11:06:43.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.582+0000 7fb41d5d6700 1 -- 172.21.15.7:0/2627047787 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4181577c0 con 0x7fb4180c00e0 2023-03-03T11:06:43.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.583+0000 7fb40f7fe700 1 -- 172.21.15.7:0/2627047787 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb4080557b0 con 0x7fb4180c00e0 2023-03-03T11:06:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:06:43.586 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:06:43.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.584+0000 7fb41d5d6700 1 -- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3f80682f0 msgr2=0x7fb3f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:06:43.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.584+0000 7fb41d5d6700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3f80682f0 0x7fb3f806a7a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb4181df620 tx=0x7fb400008040 comp rx=0 tx=0).stop 2023-03-03T11:06:43.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.584+0000 7fb41d5d6700 1 -- 172.21.15.7:0/2627047787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c00e0 msgr2=0x7fb418158360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:06:43.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.584+0000 7fb41d5d6700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c00e0 0x7fb418158360 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fb40800c750 tx=0x7fb40800cb10 comp rx=0 tx=0).stop 2023-03-03T11:06:43.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.585+0000 7fb41d5d6700 1 -- 172.21.15.7:0/2627047787 shutdown_connections 2023-03-03T11:06:43.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.585+0000 7fb41d5d6700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb418154ff0 0x7fb4180c2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:06:43.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.585+0000 7fb41d5d6700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3f80682f0 0x7fb3f806a7a0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:06:43.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.585+0000 7fb41d5d6700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4180c00e0 0x7fb418158360 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:06:43.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.585+0000 7fb41d5d6700 1 --2- 172.21.15.7:0/2627047787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4180caa30 0x7fb418157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:06:43.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.585+0000 7fb41d5d6700 1 -- 172.21.15.7:0/2627047787 >> 172.21.15.7:0/2627047787 conn(0x7fb418066e40 msgr2=0x7fb418153d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:06:43.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.585+0000 7fb41d5d6700 1 -- 172.21.15.7:0/2627047787 shutdown_connections 2023-03-03T11:06:43.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:06:43.585+0000 7fb41d5d6700 1 -- 172.21.15.7:0/2627047787 wait complete. 2023-03-03T11:06:43.590 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:06:43.600 INFO:tasks.mds_thrash.fs.[b]:waiting for 31 secs before thrashing 2023-03-03T11:07:08.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:08.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:14.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:07:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.737+0000 7f9bb467c700 1 Processor -- start 2023-03-03T11:07:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.738+0000 7f9bb467c700 1 -- start start 2023-03-03T11:07:14.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.738+0000 7f9bb467c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9bac151770 0x7f9bac151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:14.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.738+0000 7f9bb467c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac154910 con 0x7f9bac06e020 2023-03-03T11:07:14.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.738+0000 7f9bb467c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac0c4540 con 0x7f9bac155390 2023-03-03T11:07:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.739+0000 7f9bb467c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bac070b90 con 0x7f9bac151770 2023-03-03T11:07:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.739+0000 7f9bb1c17700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9bac06e020 0x7f9bac071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57110/0 (socket says 172.21.15.7:57110) 2023-03-03T11:07:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.739+0000 7f9bb1c17700 1 -- 172.21.15.7:0/637980142 learned_addr learned my addr 172.21.15.7:0/637980142 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:07:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.739+0000 7f9bb2c19700 1 --2- 172.21.15.7:0/637980142 >> v2:172.21.15.143:3300/0 conn(0x7f9bac151770 0x7f9bac151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.739+0000 7f9bb1416700 1 -- 172.21.15.7:0/637980142 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824465884 0 0) 0x7f9bac154910 con 0x7f9bac06e020 2023-03-03T11:07:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.739+0000 7f9bb1416700 1 -- 172.21.15.7:0/637980142 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b980035b0 con 0x7f9bac06e020 2023-03-03T11:07:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.739+0000 7f9bb2c19700 1 -- 172.21.15.7:0/637980142 >> v1:172.21.15.143:6789/0 conn(0x7f9bac155390 legacy=0x7f9bac0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.739+0000 7f9bb2c19700 1 -- 172.21.15.7:0/637980142 >> v1:172.21.15.7:6789/0 conn(0x7f9bac06e020 legacy=0x7f9bac071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.739+0000 7f9bb2c19700 1 -- 172.21.15.7:0/637980142 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac0702a0 con 0x7f9bac151770 2023-03-03T11:07:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.739+0000 7f9bb2c19700 1 --2- 172.21.15.7:0/637980142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac151770 0x7f9bac151bd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009d90 tx=0x7f9ba400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f69e6bce5b12429 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:14.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.740+0000 7f9bb1416700 1 -- 172.21.15.7:0/637980142 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba400e070 con 0x7f9bac151770 2023-03-03T11:07:14.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.740+0000 7f9bb1416700 1 -- 172.21.15.7:0/637980142 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba400bb20 con 0x7f9bac151770 2023-03-03T11:07:14.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.740+0000 7f9bb1416700 1 -- 172.21.15.7:0/637980142 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4012430 con 0x7f9bac151770 2023-03-03T11:07:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.740+0000 7f9bb467c700 1 -- 172.21.15.7:0/637980142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac151770 msgr2=0x7f9bac151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.740+0000 7f9bb467c700 1 --2- 172.21.15.7:0/637980142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac151770 0x7f9bac151bd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009d90 tx=0x7f9ba400b330 comp rx=0 tx=0).stop 2023-03-03T11:07:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.740+0000 7f9bb467c700 1 -- 172.21.15.7:0/637980142 shutdown_connections 2023-03-03T11:07:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.740+0000 7f9bb467c700 1 --2- 172.21.15.7:0/637980142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac151770 0x7f9bac151bd0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.740+0000 7f9bb467c700 1 -- 172.21.15.7:0/637980142 >> 172.21.15.7:0/637980142 conn(0x7f9bac06c110 msgr2=0x7f9bac06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:07:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.740+0000 7f9bb467c700 1 -- 172.21.15.7:0/637980142 shutdown_connections 2023-03-03T11:07:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.740+0000 7f9bb467c700 1 -- 172.21.15.7:0/637980142 wait complete. 2023-03-03T11:07:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.741+0000 7f9bb467c700 1 Processor -- start 2023-03-03T11:07:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.741+0000 7f9bb467c700 1 -- start start 2023-03-03T11:07:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb467c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac06e020 0x7f9bac148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb467c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac151770 0x7f9bac14df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb467c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac155390 0x7f9bac0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb467c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1e9400 con 0x7f9bac151770 2023-03-03T11:07:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb467c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1ea3d0 con 0x7f9bac06e020 2023-03-03T11:07:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb467c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1492d0 con 0x7f9bac155390 2023-03-03T11:07:14.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb1c17700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac151770 0x7f9bac14df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:14.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb1c17700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac151770 0x7f9bac14df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34904/0 (socket says 172.21.15.7:34904) 2023-03-03T11:07:14.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb1c17700 1 -- 172.21.15.7:0/4049743445 learned_addr learned my addr 172.21.15.7:0/4049743445 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:07:14.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb2418700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac06e020 0x7f9bac148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:14.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.742+0000 7f9bb2c19700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac155390 0x7f9bac0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:14.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9bb1c17700 1 -- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac155390 msgr2=0x7f9bac0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:14.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9bb1c17700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac155390 0x7f9bac0c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:14.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9bb1c17700 1 -- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac06e020 msgr2=0x7f9bac148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:14.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9bb1c17700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac06e020 0x7f9bac148d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:14.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9bb1c17700 1 -- 172.21.15.7:0/4049743445 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba4009a70 con 0x7f9bac151770 2023-03-03T11:07:14.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9bb2c19700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac155390 0x7f9bac0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:07:14.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9bb1c17700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac151770 0x7f9bac14df40 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800c730 tx=0x7f9ba800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:14.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9b9effd700 1 -- 172.21.15.7:0/4049743445 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8005d70 con 0x7f9bac151770 2023-03-03T11:07:14.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9b9effd700 1 -- 172.21.15.7:0/4049743445 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba80070d0 con 0x7f9bac151770 2023-03-03T11:07:14.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9bb467c700 1 -- 172.21.15.7:0/4049743445 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac070820 con 0x7f9bac151770 2023-03-03T11:07:14.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9b9effd700 1 -- 172.21.15.7:0/4049743445 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8007460 con 0x7f9bac151770 2023-03-03T11:07:14.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.743+0000 7f9bb467c700 1 -- 172.21.15.7:0/4049743445 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac070540 con 0x7f9bac151770 2023-03-03T11:07:14.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.744+0000 7f9bb467c700 1 -- 172.21.15.7:0/4049743445 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac155760 con 0x7f9bac151770 2023-03-03T11:07:14.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.744+0000 7f9b9effd700 1 -- 172.21.15.7:0/4049743445 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ba8019030 con 0x7f9bac151770 2023-03-03T11:07:14.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.745+0000 7f9b9effd700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 0x7f9b9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:14.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.745+0000 7f9b9effd700 1 -- 172.21.15.7:0/4049743445 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9ba8085540 con 0x7f9bac151770 2023-03-03T11:07:14.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.745+0000 7f9bb2418700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 0x7f9b9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:14.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.746+0000 7f9bb2418700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 0x7f9b9006a7a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0007fd0 tx=0x7f9ba0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:14.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.749+0000 7f9b9effd700 1 -- 172.21.15.7:0/4049743445 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ba80518e0 con 0x7f9bac151770 2023-03-03T11:07:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.986+0000 7f9bb467c700 1 -- 172.21.15.7:0/4049743445 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bac074bb0 con 0x7f9bac151770 2023-03-03T11:07:14.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.987+0000 7f9b9effd700 1 -- 172.21.15.7:0/4049743445 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9ba8054f00 con 0x7f9bac151770 2023-03-03T11:07:14.989 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:07:14.990 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:07:14.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 -- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 msgr2=0x7f9b9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:14.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 0x7f9b9006a7a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0007fd0 tx=0x7f9ba0009040 comp rx=0 tx=0).stop 2023-03-03T11:07:14.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 -- 172.21.15.7:0/4049743445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac151770 msgr2=0x7f9bac14df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:14.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac151770 0x7f9bac14df40 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800c730 tx=0x7f9ba800ca40 comp rx=0 tx=0).stop 2023-03-03T11:07:14.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 -- 172.21.15.7:0/4049743445 shutdown_connections 2023-03-03T11:07:14.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac155390 0x7f9bac0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:14.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac06e020 0x7f9bac148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:14.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 0x7f9b9006a7a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:14.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 --2- 172.21.15.7:0/4049743445 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac151770 0x7f9bac14df40 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:14.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 -- 172.21.15.7:0/4049743445 >> 172.21.15.7:0/4049743445 conn(0x7f9bac06c110 msgr2=0x7f9bac06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:07:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 -- 172.21.15.7:0/4049743445 shutdown_connections 2023-03-03T11:07:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:14.989+0000 7f9bb467c700 1 -- 172.21.15.7:0/4049743445 wait complete. 2023-03-03T11:07:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:07:15.003 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before thrashing 2023-03-03T11:07:18.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:07:18.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.141+0000 7fe515a43700 1 Processor -- start 2023-03-03T11:07:18.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.141+0000 7fe515a43700 1 -- start start 2023-03-03T11:07:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.141+0000 7fe515a43700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe5100c5c60 0x7fe510154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.141+0000 7fe515a43700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5100c7310 con 0x7fe5100caa30 2023-03-03T11:07:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.141+0000 7fe515a43700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5100bfd10 con 0x7fe5100c00e0 2023-03-03T11:07:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.141+0000 7fe515a43700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5100c60a0 con 0x7fe5100c5c60 2023-03-03T11:07:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.141+0000 7fe50effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe5100caa30 0x7fe5100c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57122/0 (socket says 172.21.15.7:57122) 2023-03-03T11:07:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50effd700 1 -- 172.21.15.7:0/3781849919 learned_addr learned my addr 172.21.15.7:0/3781849919 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:07:18.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170574065 0 0) 0x7fe5100c7310 con 0x7fe5100caa30 2023-03-03T11:07:18.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50f7fe700 1 --2- 172.21.15.7:0/3781849919 >> v2:172.21.15.143:3300/0 conn(0x7fe5100c5c60 0x7fe510154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:18.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc0035b0 con 0x7fe5100caa30 2023-03-03T11:07:18.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 423249194 0 0) 0x7fe4fc0035b0 con 0x7fe5100caa30 2023-03-03T11:07:18.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 >> v2:172.21.15.143:3300/0 conn(0x7fe5100c5c60 msgr2=0x7fe510154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:07:18.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 --2- 172.21.15.7:0/3781849919 >> v2:172.21.15.143:3300/0 conn(0x7fe5100c5c60 0x7fe510154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:18.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 >> v1:172.21.15.143:6789/0 conn(0x7fe5100c00e0 legacy=0x7fe51015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:18.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5100bfa30 con 0x7fe5100caa30 2023-03-03T11:07:18.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe500001670 con 0x7fe5100caa30 2023-03-03T11:07:18.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 >> v1:172.21.15.7:6789/0 conn(0x7fe5100caa30 legacy=0x7fe5100c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:18.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe4fc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:18.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4fc004850 0x7fe4fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:18.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc005250 0x7fe4fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:18.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc00c160 con 0x7fe4fc003a70 2023-03-03T11:07:18.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc00c2d0 con 0x7fe4fc005250 2023-03-03T11:07:18.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc00c490 con 0x7fe4fc004850 2023-03-03T11:07:18.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50effd700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe4fc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:18.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50f7fe700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc005250 0x7fe4fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:18.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.142+0000 7fe50e7fc700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4fc004850 0x7fe4fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:18.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50effd700 1 -- 172.21.15.7:0/3781849919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4fc004850 msgr2=0x7fe4fc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:07:18.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50effd700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4fc004850 0x7fe4fc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:18.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50effd700 1 -- 172.21.15.7:0/3781849919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc005250 msgr2=0x7fe4fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:07:18.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50effd700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4fc005250 0x7fe4fc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:18.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50effd700 1 -- 172.21.15.7:0/3781849919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc0035b0 con 0x7fe4fc003a70 2023-03-03T11:07:18.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50f7fe700 1 -- 172.21.15.7:0/3781849919 reap_dead start 2023-03-03T11:07:18.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50e7fc700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4fc004850 0x7fe4fc004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:07:18.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50effd700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe4fc003e80 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fe50000c270 tx=0x7fe4fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58e35fda0c01161 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:18.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500007c10 con 0x7fe4fc003a70 2023-03-03T11:07:18.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50000cd30 con 0x7fe4fc003a70 2023-03-03T11:07:18.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe50dffb700 1 -- 172.21.15.7:0/3781849919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50000cea0 con 0x7fe4fc003a70 2023-03-03T11:07:18.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe515a43700 1 -- 172.21.15.7:0/3781849919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 msgr2=0x7fe4fc003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:18.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe515a43700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe4fc003e80 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fe50000c270 tx=0x7fe4fc005880 comp rx=0 tx=0).stop 2023-03-03T11:07:18.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe515a43700 1 -- 172.21.15.7:0/3781849919 shutdown_connections 2023-03-03T11:07:18.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe515a43700 1 --2- 172.21.15.7:0/3781849919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe4fc003e80 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:18.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe515a43700 1 -- 172.21.15.7:0/3781849919 >> 172.21.15.7:0/3781849919 conn(0x7fe510066e40 msgr2=0x7fe510071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:07:18.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.143+0000 7fe515a43700 1 -- 172.21.15.7:0/3781849919 shutdown_connections 2023-03-03T11:07:18.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.144+0000 7fe515a43700 1 -- 172.21.15.7:0/3781849919 wait complete. 2023-03-03T11:07:18.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.145+0000 7fe515a43700 1 Processor -- start 2023-03-03T11:07:18.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.145+0000 7fe515a43700 1 -- start start 2023-03-03T11:07:18.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.145+0000 7fe515a43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe5100aa030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:18.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.145+0000 7fe515a43700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5100c46a0 0x7fe51014d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:18.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.145+0000 7fe515a43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5100c4b10 0x7fe51014d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:18.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.145+0000 7fe515a43700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5100aaa90 con 0x7fe4fc003a70 2023-03-03T11:07:18.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.145+0000 7fe515a43700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe510153fd0 con 0x7fe5100c46a0 2023-03-03T11:07:18.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.145+0000 7fe515a43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe510157810 con 0x7fe5100c4b10 2023-03-03T11:07:18.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe5100aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:18.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe5100aa030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34922/0 (socket says 172.21.15.7:34922) 2023-03-03T11:07:18.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50effd700 1 -- 172.21.15.7:0/2159559765 learned_addr learned my addr 172.21.15.7:0/2159559765 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:07:18.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50e7fc700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5100c46a0 0x7fe51014d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:18.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50f7fe700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5100c4b10 0x7fe51014d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:18.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50effd700 1 -- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5100c4b10 msgr2=0x7fe51014d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:18.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50effd700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5100c4b10 0x7fe51014d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:18.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50effd700 1 -- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5100c46a0 msgr2=0x7fe51014d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:18.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50effd700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5100c46a0 0x7fe51014d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:18.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50effd700 1 -- 172.21.15.7:0/2159559765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5101547f0 con 0x7fe4fc003a70 2023-03-03T11:07:18.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe50effd700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe5100aa030 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fe50000c860 tx=0x7fe510154400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:18.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe507fff700 1 -- 172.21.15.7:0/2159559765 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500001630 con 0x7fe4fc003a70 2023-03-03T11:07:18.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe507fff700 1 -- 172.21.15.7:0/2159559765 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50000cea0 con 0x7fe4fc003a70 2023-03-03T11:07:18.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe515a43700 1 -- 172.21.15.7:0/2159559765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe510154510 con 0x7fe4fc003a70 2023-03-03T11:07:18.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.146+0000 7fe507fff700 1 -- 172.21.15.7:0/2159559765 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500007310 con 0x7fe4fc003a70 2023-03-03T11:07:18.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.147+0000 7fe515a43700 1 -- 172.21.15.7:0/2159559765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe510154ad0 con 0x7fe4fc003a70 2023-03-03T11:07:18.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.147+0000 7fe515a43700 1 -- 172.21.15.7:0/2159559765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5100c19e0 con 0x7fe4fc003a70 2023-03-03T11:07:18.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.148+0000 7fe507fff700 1 -- 172.21.15.7:0/2159559765 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe50002e070 con 0x7fe4fc003a70 2023-03-03T11:07:18.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.148+0000 7fe507fff700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4f0068320 0x7fe4f006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:18.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.149+0000 7fe507fff700 1 -- 172.21.15.7:0/2159559765 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe500087690 con 0x7fe4fc003a70 2023-03-03T11:07:18.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.149+0000 7fe50e7fc700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4f0068320 0x7fe4f006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:18.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.149+0000 7fe50e7fc700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4f0068320 0x7fe4f006a7d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8000c00 tx=0x7fe4f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:18.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.153+0000 7fe507fff700 1 -- 172.21.15.7:0/2159559765 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5100c19e0 con 0x7fe4fc003a70 2023-03-03T11:07:18.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.394+0000 7fe515a43700 1 -- 172.21.15.7:0/2159559765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe510074bb0 con 0x7fe4fc003a70 2023-03-03T11:07:18.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.395+0000 7fe507fff700 1 -- 172.21.15.7:0/2159559765 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe500057520 con 0x7fe4fc003a70 2023-03-03T11:07:18.397 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:07:18.397 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:07:18.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 -- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4f0068320 msgr2=0x7fe4f006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:18.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4f0068320 0x7fe4f006a7d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8000c00 tx=0x7fe4f8009040 comp rx=0 tx=0).stop 2023-03-03T11:07:18.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 -- 172.21.15.7:0/2159559765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 msgr2=0x7fe5100aa030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:18.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe5100aa030 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fe50000c860 tx=0x7fe510154400 comp rx=0 tx=0).stop 2023-03-03T11:07:18.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 -- 172.21.15.7:0/2159559765 shutdown_connections 2023-03-03T11:07:18.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5100c4b10 0x7fe51014d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:18.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4f0068320 0x7fe4f006a7d0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:18.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4fc003a70 0x7fe5100aa030 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:18.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 --2- 172.21.15.7:0/2159559765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5100c46a0 0x7fe51014d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:18.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 -- 172.21.15.7:0/2159559765 >> 172.21.15.7:0/2159559765 conn(0x7fe510066e40 msgr2=0x7fe5100c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:07:18.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 -- 172.21.15.7:0/2159559765 shutdown_connections 2023-03-03T11:07:18.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:18.397+0000 7fe515a43700 1 -- 172.21.15.7:0/2159559765 wait complete. 2023-03-03T11:07:18.401 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:07:18.411 INFO:tasks.mds_thrash.fs.[b]:waiting for 17 secs before thrashing 2023-03-03T11:07:29.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:07:29.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.983+0000 7f27f7877700 1 Processor -- start 2023-03-03T11:07:29.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.983+0000 7f27f7877700 1 -- start start 2023-03-03T11:07:29.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27f7877700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014b0d0 0x7f27f0071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:29.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27f7877700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f00a9c20 con 0x7f27f00aad20 2023-03-03T11:07:29.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27f7877700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f0154f10 con 0x7f27f0071d40 2023-03-03T11:07:29.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27f7877700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00ac3c0 con 0x7f27f014b0d0 2023-03-03T11:07:29.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27f4e12700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f27f00aad20 0x7f27f0157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57152/0 (socket says 172.21.15.7:57152) 2023-03-03T11:07:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27f4e12700 1 -- 172.21.15.7:0/2299435116 learned_addr learned my addr 172.21.15.7:0/2299435116 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:07:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27f5613700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014b0d0 0x7f27f0071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796103519 0 0) 0x7f27f00a9c20 con 0x7f27f00aad20 2023-03-03T11:07:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27dc0035b0 con 0x7f27f00aad20 2023-03-03T11:07:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1118078025 0 0) 0x7f27dc0035b0 con 0x7f27f00aad20 2023-03-03T11:07:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014b0d0 msgr2=0x7f27f0071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:07:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27e7fff700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014b0d0 0x7f27f0071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 >> v1:172.21.15.143:6789/0 conn(0x7f27f0071d40 legacy=0x7f27f015ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f006fc70 con 0x7f27f00aad20 2023-03-03T11:07:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f27e0001670 con 0x7f27f00aad20 2023-03-03T11:07:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 >> v1:172.21.15.7:6789/0 conn(0x7f27f00aad20 legacy=0x7f27f0157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.984+0000 7f27e7fff700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014b0d0 0x7f27dc003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27e7fff700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27dc004020 0x7f27dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27e7fff700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27dc004d90 0x7f27dc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00ac3c0 con 0x7f27dc004d90 2023-03-03T11:07:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc00bc00 con 0x7f27dc004020 2023-03-03T11:07:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc00bdc0 con 0x7f27f014b0d0 2023-03-03T11:07:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27f5e14700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27dc004d90 0x7f27dc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27f5613700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014b0d0 0x7f27dc003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27f4e12700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27dc004020 0x7f27dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27f5e14700 1 -- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014b0d0 msgr2=0x7f27dc003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:07:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27f5e14700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014b0d0 0x7f27dc003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27f5e14700 1 -- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27dc004020 msgr2=0x7f27dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:07:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27f5e14700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27dc004020 0x7f27dc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27f5e14700 1 -- 172.21.15.7:0/2299435116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc0035b0 con 0x7f27dc004d90 2023-03-03T11:07:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27f5e14700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27dc004d90 0x7f27dc005200 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f27e800c160 tx=0x7f27dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff93935ce515082d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8001db0 con 0x7f27dc004d90 2023-03-03T11:07:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e800cca0 con 0x7f27dc004d90 2023-03-03T11:07:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.985+0000 7f27e7fff700 1 -- 172.21.15.7:0/2299435116 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8003c40 con 0x7f27dc004d90 2023-03-03T11:07:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.986+0000 7f27f7877700 1 -- 172.21.15.7:0/2299435116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27dc004d90 msgr2=0x7f27dc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.986+0000 7f27f7877700 1 --2- 172.21.15.7:0/2299435116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27dc004d90 0x7f27dc005200 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f27e800c160 tx=0x7f27dc005880 comp rx=0 tx=0).stop 2023-03-03T11:07:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.986+0000 7f27f5e14700 1 -- 172.21.15.7:0/2299435116 reap_dead start 2023-03-03T11:07:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.986+0000 7f27f7877700 1 -- 172.21.15.7:0/2299435116 shutdown_connections 2023-03-03T11:07:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.986+0000 7f27f7877700 1 -- 172.21.15.7:0/2299435116 >> 172.21.15.7:0/2299435116 conn(0x7f27f00c8900 msgr2=0x7f27f00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:07:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.986+0000 7f27f7877700 1 -- 172.21.15.7:0/2299435116 shutdown_connections 2023-03-03T11:07:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.986+0000 7f27f7877700 1 -- 172.21.15.7:0/2299435116 wait complete. 2023-03-03T11:07:29.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.987+0000 7f27f7877700 1 Processor -- start 2023-03-03T11:07:29.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.987+0000 7f27f7877700 1 -- start start 2023-03-03T11:07:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f7877700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014a100 0x7f27f00c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f7877700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f014a4d0 0x7f27f01ed290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f7877700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f0152290 0x7f27f0144f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f7877700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00c8240 con 0x7f27f014a4d0 2023-03-03T11:07:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f7877700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00c6b70 con 0x7f27f0152290 2023-03-03T11:07:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f7877700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00c7fa0 con 0x7f27f014a100 2023-03-03T11:07:29.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f4e12700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f014a4d0 0x7f27f01ed290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f4e12700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f014a4d0 0x7f27f01ed290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34954/0 (socket says 172.21.15.7:34954) 2023-03-03T11:07:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f4e12700 1 -- 172.21.15.7:0/864397282 learned_addr learned my addr 172.21.15.7:0/864397282 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:07:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f5e14700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f0152290 0x7f27f0144f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f5613700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014a100 0x7f27f00c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f4e12700 1 -- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014a100 msgr2=0x7f27f00c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f4e12700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014a100 0x7f27f00c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:29.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f4e12700 1 -- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f0152290 msgr2=0x7f27f0144f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f4e12700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f0152290 0x7f27f0144f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f4e12700 1 -- 172.21.15.7:0/864397282 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f006f6b0 con 0x7f27f014a4d0 2023-03-03T11:07:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f5613700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014a100 0x7f27f00c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:07:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.988+0000 7f27f5e14700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f0152290 0x7f27f0144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:07:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.989+0000 7f27f4e12700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f014a4d0 0x7f27f01ed290 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f27e0007290 tx=0x7f27e0007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.989+0000 7f27e5ffb700 1 -- 172.21.15.7:0/864397282 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e000f850 con 0x7f27f014a4d0 2023-03-03T11:07:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.989+0000 7f27e5ffb700 1 -- 172.21.15.7:0/864397282 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e000f9f0 con 0x7f27f014a4d0 2023-03-03T11:07:29.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.989+0000 7f27f7877700 1 -- 172.21.15.7:0/864397282 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f0155160 con 0x7f27f014a4d0 2023-03-03T11:07:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.989+0000 7f27e5ffb700 1 -- 172.21.15.7:0/864397282 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e000fda0 con 0x7f27f014a4d0 2023-03-03T11:07:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.989+0000 7f27f7877700 1 -- 172.21.15.7:0/864397282 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f006f990 con 0x7f27f014a4d0 2023-03-03T11:07:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.990+0000 7f27f7877700 1 -- 172.21.15.7:0/864397282 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f0152aa0 con 0x7f27f014a4d0 2023-03-03T11:07:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.990+0000 7f27e5ffb700 1 -- 172.21.15.7:0/864397282 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f27e001d460 con 0x7f27f014a4d0 2023-03-03T11:07:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.991+0000 7f27e5ffb700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27d4068320 0x7f27d406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.991+0000 7f27e5ffb700 1 -- 172.21.15.7:0/864397282 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f27e00862e0 con 0x7f27f014a4d0 2023-03-03T11:07:29.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.991+0000 7f27f5613700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27d4068320 0x7f27d406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:30.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.992+0000 7f27f5613700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27d4068320 0x7f27d406a7d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009c60 tx=0x7f27ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:30.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:29.995+0000 7f27e5ffb700 1 -- 172.21.15.7:0/864397282 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f27f0152aa0 con 0x7f27f014a4d0 2023-03-03T11:07:30.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.236+0000 7f27f7877700 1 -- 172.21.15.7:0/864397282 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27f0152aa0 con 0x7f27f014a4d0 2023-03-03T11:07:30.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.236+0000 7f27e5ffb700 1 -- 172.21.15.7:0/864397282 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f27f0152aa0 con 0x7f27f014a4d0 2023-03-03T11:07:30.239 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:07:30.239 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:07:30.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 -- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27d4068320 msgr2=0x7f27d406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27d4068320 0x7f27d406a7d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009c60 tx=0x7f27ec008040 comp rx=0 tx=0).stop 2023-03-03T11:07:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 -- 172.21.15.7:0/864397282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f014a4d0 msgr2=0x7f27f01ed290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f014a4d0 0x7f27f01ed290 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f27e0007290 tx=0x7f27e0007650 comp rx=0 tx=0).stop 2023-03-03T11:07:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 -- 172.21.15.7:0/864397282 shutdown_connections 2023-03-03T11:07:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27f0152290 0x7f27f0144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27f014a100 0x7f27f00c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27d4068320 0x7f27d406a7d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 --2- 172.21.15.7:0/864397282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27f014a4d0 0x7f27f01ed290 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 -- 172.21.15.7:0/864397282 >> 172.21.15.7:0/864397282 conn(0x7f27f00c8900 msgr2=0x7f27f00acae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:07:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 -- 172.21.15.7:0/864397282 shutdown_connections 2023-03-03T11:07:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:30.238+0000 7f27f7877700 1 -- 172.21.15.7:0/864397282 wait complete. 2023-03-03T11:07:30.242 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:07:30.252 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T11:07:35.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:07:35.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.548+0000 7f343f867700 1 Processor -- start 2023-03-03T11:07:35.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.549+0000 7f343f867700 1 -- start start 2023-03-03T11:07:35.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.549+0000 7f343f867700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3438151d40 0x7f34380736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.549+0000 7f343f867700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f34380c2930 0x7f3438157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.549+0000 7f343f867700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343806d430 con 0x7f3438073c30 2023-03-03T11:07:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.549+0000 7f343f867700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343806c2a0 con 0x7f34380c2930 2023-03-03T11:07:35.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.549+0000 7f343f867700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34381586e0 con 0x7f3438151d40 2023-03-03T11:07:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f343de04700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3438073c30 0x7f3438157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57180/0 (socket says 172.21.15.7:57180) 2023-03-03T11:07:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f343de04700 1 -- 172.21.15.7:0/1758795091 learned_addr learned my addr 172.21.15.7:0/1758795091 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:07:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f343ce02700 1 --2- 172.21.15.7:0/1758795091 >> v2:172.21.15.7:3300/0 conn(0x7f34380c2930 0x7f3438157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f343d603700 1 --2- 172.21.15.7:0/1758795091 >> v2:172.21.15.143:3300/0 conn(0x7f3438151d40 0x7f34380736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f342bfff700 1 -- 172.21.15.7:0/1758795091 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024248259 0 0) 0x7f343806d430 con 0x7f3438073c30 2023-03-03T11:07:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f342bfff700 1 -- 172.21.15.7:0/1758795091 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34240035b0 con 0x7f3438073c30 2023-03-03T11:07:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f343ce02700 1 -- 172.21.15.7:0/1758795091 >> v2:172.21.15.143:3300/0 conn(0x7f3438151d40 msgr2=0x7f34380736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:07:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f343ce02700 1 --2- 172.21.15.7:0/1758795091 >> v2:172.21.15.143:3300/0 conn(0x7f3438151d40 0x7f34380736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f343ce02700 1 -- 172.21.15.7:0/1758795091 >> v1:172.21.15.7:6789/0 conn(0x7f3438073c30 legacy=0x7f3438157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f343ce02700 1 -- 172.21.15.7:0/1758795091 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343806cb00 con 0x7f34380c2930 2023-03-03T11:07:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.550+0000 7f343ce02700 1 --2- 172.21.15.7:0/1758795091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34380c2930 0x7f3438157a90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f342c009840 tx=0x7f342c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3edfbab5a937de67 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f342bfff700 1 -- 172.21.15.7:0/1758795091 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c011830 con 0x7f34380c2930 2023-03-03T11:07:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f342bfff700 1 -- 172.21.15.7:0/1758795091 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c011990 con 0x7f34380c2930 2023-03-03T11:07:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f342bfff700 1 -- 172.21.15.7:0/1758795091 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c011d00 con 0x7f34380c2930 2023-03-03T11:07:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f343f867700 1 -- 172.21.15.7:0/1758795091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34380c2930 msgr2=0x7f3438157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f343f867700 1 --2- 172.21.15.7:0/1758795091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34380c2930 0x7f3438157a90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f342c009840 tx=0x7f342c009b50 comp rx=0 tx=0).stop 2023-03-03T11:07:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f343f867700 1 -- 172.21.15.7:0/1758795091 shutdown_connections 2023-03-03T11:07:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f343f867700 1 --2- 172.21.15.7:0/1758795091 >> v2:172.21.15.143:3300/0 conn(0x7f3438151d40 0x7f34380736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f343f867700 1 --2- 172.21.15.7:0/1758795091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34380c2930 0x7f3438157a90 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f343f867700 1 -- 172.21.15.7:0/1758795091 >> 172.21.15.7:0/1758795091 conn(0x7f34380a94d0 msgr2=0x7f34380a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:07:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f343f867700 1 -- 172.21.15.7:0/1758795091 shutdown_connections 2023-03-03T11:07:35.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.551+0000 7f343f867700 1 -- 172.21.15.7:0/1758795091 wait complete. 2023-03-03T11:07:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.553+0000 7f343f867700 1 Processor -- start 2023-03-03T11:07:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.553+0000 7f343f867700 1 -- start start 2023-03-03T11:07:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.553+0000 7f343f867700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3438073c30 0x7f34380abac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.553+0000 7f343f867700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34380c2930 0x7f34380acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.553+0000 7f343f867700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3438151d40 0x7f34381541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.553+0000 7f343f867700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34380ab630 con 0x7f3438151d40 2023-03-03T11:07:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.553+0000 7f343f867700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3438153dc0 con 0x7f3438073c30 2023-03-03T11:07:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.553+0000 7f343f867700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34380ac460 con 0x7f34380c2930 2023-03-03T11:07:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.553+0000 7f343de04700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3438151d40 0x7f34381541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343de04700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3438151d40 0x7f34381541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34972/0 (socket says 172.21.15.7:34972) 2023-03-03T11:07:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343de04700 1 -- 172.21.15.7:0/2455179014 learned_addr learned my addr 172.21.15.7:0/2455179014 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:07:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343d603700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3438073c30 0x7f34380abac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343de04700 1 -- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34380c2930 msgr2=0x7f34380acad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:35.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343ce02700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34380c2930 0x7f34380acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:35.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343de04700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34380c2930 0x7f34380acad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:35.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343de04700 1 -- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3438073c30 msgr2=0x7f34380abac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:35.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343de04700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3438073c30 0x7f34380abac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:35.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343de04700 1 -- 172.21.15.7:0/2455179014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c0094a0 con 0x7f3438151d40 2023-03-03T11:07:35.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343d603700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3438073c30 0x7f34380abac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:07:35.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343de04700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3438151d40 0x7f34381541e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f343000ca70 tx=0x7f343000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:35.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f3429ffb700 1 -- 172.21.15.7:0/2455179014 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343000aaa0 con 0x7f3438151d40 2023-03-03T11:07:35.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f3429ffb700 1 -- 172.21.15.7:0/2455179014 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343000ac00 con 0x7f3438151d40 2023-03-03T11:07:35.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343f867700 1 -- 172.21.15.7:0/2455179014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343806d0c0 con 0x7f3438151d40 2023-03-03T11:07:35.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f3429ffb700 1 -- 172.21.15.7:0/2455179014 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430012400 con 0x7f3438151d40 2023-03-03T11:07:35.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.554+0000 7f343f867700 1 -- 172.21.15.7:0/2455179014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343806cde0 con 0x7f3438151d40 2023-03-03T11:07:35.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.555+0000 7f343f867700 1 -- 172.21.15.7:0/2455179014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343806bea0 con 0x7f3438151d40 2023-03-03T11:07:35.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.556+0000 7f3429ffb700 1 -- 172.21.15.7:0/2455179014 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3430019030 con 0x7f3438151d40 2023-03-03T11:07:35.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.556+0000 7f3429ffb700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 0x7f341c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:07:35.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.556+0000 7f3429ffb700 1 -- 172.21.15.7:0/2455179014 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3430085bd0 con 0x7f3438151d40 2023-03-03T11:07:35.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.556+0000 7f343d603700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 0x7f341c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:07:35.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.557+0000 7f343d603700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 0x7f341c06a7a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3434009b60 tx=0x7f3434008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:07:35.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.560+0000 7f3429ffb700 1 -- 172.21.15.7:0/2455179014 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f343004b5e0 con 0x7f3438151d40 2023-03-03T11:07:35.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.797+0000 7f343f867700 1 -- 172.21.15.7:0/2455179014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3438074bb0 con 0x7f3438151d40 2023-03-03T11:07:35.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.797+0000 7f3429ffb700 1 -- 172.21.15.7:0/2455179014 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3430052020 con 0x7f3438151d40 2023-03-03T11:07:35.800 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:07:35.800 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:07:35.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 -- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 msgr2=0x7f341c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:35.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 0x7f341c06a7a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3434009b60 tx=0x7f3434008040 comp rx=0 tx=0).stop 2023-03-03T11:07:35.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 -- 172.21.15.7:0/2455179014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3438151d40 msgr2=0x7f34381541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:07:35.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3438151d40 0x7f34381541e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f343000ca70 tx=0x7f343000ce30 comp rx=0 tx=0).stop 2023-03-03T11:07:35.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 -- 172.21.15.7:0/2455179014 shutdown_connections 2023-03-03T11:07:35.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 0x7f341c06a7a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:35.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3438151d40 0x7f34381541e0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:35.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3438073c30 0x7f34380abac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:35.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 --2- 172.21.15.7:0/2455179014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34380c2930 0x7f34380acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:07:35.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 -- 172.21.15.7:0/2455179014 >> 172.21.15.7:0/2455179014 conn(0x7f34380a94d0 msgr2=0x7f3438157200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:07:35.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 -- 172.21.15.7:0/2455179014 shutdown_connections 2023-03-03T11:07:35.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:07:35.800+0000 7f343f867700 1 -- 172.21.15.7:0/2455179014 wait complete. 2023-03-03T11:07:35.804 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:07:35.814 INFO:tasks.mds_thrash.fs.[b]:waiting for 93 secs before thrashing 2023-03-03T11:07:39.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:07:39.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:09.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:09.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:19.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:08:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.391+0000 7f8774a76700 1 Processor -- start 2023-03-03T11:08:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.391+0000 7f8774a76700 1 -- start start 2023-03-03T11:08:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f8774a76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701550c0 0x7f877015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f8774a76700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87700c7a60 con 0x7f87700c1da0 2023-03-03T11:08:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f8774a76700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87700c3620 con 0x7f877006d870 2023-03-03T11:08:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f8774a76700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87700c2af0 con 0x7f87701550c0 2023-03-03T11:08:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876dd9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f87700c1da0 0x7f87700c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57220/0 (socket says 172.21.15.7:57220) 2023-03-03T11:08:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876dd9b700 1 -- 172.21.15.7:0/3978899517 learned_addr learned my addr 172.21.15.7:0/3978899517 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:08:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876ed9d700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701550c0 0x7f877015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283694648 0 0) 0x7f87700c7a60 con 0x7f87700c1da0 2023-03-03T11:08:19.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87580035b0 con 0x7f87700c1da0 2023-03-03T11:08:19.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1789161847 0 0) 0x7f87580035b0 con 0x7f87700c1da0 2023-03-03T11:08:19.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701550c0 msgr2=0x7f877015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:08:19.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701550c0 0x7f877015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:19.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 >> v1:172.21.15.143:6789/0 conn(0x7f877006d870 legacy=0x7f87700c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:19.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87701545e0 con 0x7f87700c1da0 2023-03-03T11:08:19.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8760001670 con 0x7f87700c1da0 2023-03-03T11:08:19.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 >> v1:172.21.15.7:6789/0 conn(0x7f87700c1da0 legacy=0x7f87700c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:19.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701550c0 0x7f8758003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:19.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8758004020 0x7f8758004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:19.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 0x7f8758005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:19.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875800bb20 con 0x7f8758004d90 2023-03-03T11:08:19.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875800bc90 con 0x7f8758004020 2023-03-03T11:08:19.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.392+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f875800be50 con 0x7f87701550c0 2023-03-03T11:08:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876e59c700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701550c0 0x7f8758003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876ed9d700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 0x7f8758005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876dd9b700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8758004020 0x7f8758004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876ed9d700 1 -- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701550c0 msgr2=0x7f8758003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:08:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876ed9d700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701550c0 0x7f8758003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876ed9d700 1 -- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8758004020 msgr2=0x7f8758004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:08:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876ed9d700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8758004020 0x7f8758004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:19.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876ed9d700 1 -- 172.21.15.7:0/3978899517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87580035b0 con 0x7f8758004d90 2023-03-03T11:08:19.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876ed9d700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 0x7f8758005200 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8768009fc0 tx=0x7f8758005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520df08eda22111a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:19.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f876800ec30 con 0x7f8758004d90 2023-03-03T11:08:19.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876800f040 con 0x7f8758004d90 2023-03-03T11:08:19.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.393+0000 7f876d59a700 1 -- 172.21.15.7:0/3978899517 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8768014480 con 0x7f8758004d90 2023-03-03T11:08:19.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.394+0000 7f8774a76700 1 -- 172.21.15.7:0/3978899517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 msgr2=0x7f8758005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:19.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.394+0000 7f8774a76700 1 --2- 172.21.15.7:0/3978899517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 0x7f8758005200 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8768009fc0 tx=0x7f8758005880 comp rx=0 tx=0).stop 2023-03-03T11:08:19.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.394+0000 7f876ed9d700 1 -- 172.21.15.7:0/3978899517 reap_dead start 2023-03-03T11:08:19.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.394+0000 7f8774a76700 1 -- 172.21.15.7:0/3978899517 shutdown_connections 2023-03-03T11:08:19.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.394+0000 7f8774a76700 1 -- 172.21.15.7:0/3978899517 >> 172.21.15.7:0/3978899517 conn(0x7f877006c110 msgr2=0x7f877006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:08:19.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.394+0000 7f8774a76700 1 -- 172.21.15.7:0/3978899517 shutdown_connections 2023-03-03T11:08:19.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.394+0000 7f8774a76700 1 -- 172.21.15.7:0/3978899517 wait complete. 2023-03-03T11:08:19.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.395+0000 7f8774a76700 1 Processor -- start 2023-03-03T11:08:19.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.395+0000 7f8774a76700 1 -- start start 2023-03-03T11:08:19.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f8774a76700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 0x7f87701ed390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:19.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f8774a76700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877013cb00 0x7f87701ed8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:19.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f8774a76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701ede10 0x7f87700aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:19.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f8774a76700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877014b430 con 0x7f8758004d90 2023-03-03T11:08:19.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f8774a76700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8770151990 con 0x7f877013cb00 2023-03-03T11:08:19.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f8774a76700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877013cf90 con 0x7f87701ede10 2023-03-03T11:08:19.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f876e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 0x7f87701ed390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:19.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f876e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 0x7f87701ed390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35018/0 (socket says 172.21.15.7:35018) 2023-03-03T11:08:19.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f876e59c700 1 -- 172.21.15.7:0/4178944293 learned_addr learned my addr 172.21.15.7:0/4178944293 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:08:19.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f876ed9d700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701ede10 0x7f87700aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:19.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f876dd9b700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877013cb00 0x7f87701ed8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:19.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f876e59c700 1 -- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701ede10 msgr2=0x7f87700aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:19.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f876e59c700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701ede10 0x7f87700aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:19.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f876e59c700 1 -- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877013cb00 msgr2=0x7f87701ed8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:19.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f876e59c700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877013cb00 0x7f87701ed8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:19.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.396+0000 7f876e59c700 1 -- 172.21.15.7:0/4178944293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8770154ba0 con 0x7f8758004d90 2023-03-03T11:08:19.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.397+0000 7f876e59c700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 0x7f87701ed390 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f876400f820 tx=0x7f876400fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:19.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.397+0000 7f875effd700 1 -- 172.21.15.7:0/4178944293 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640052e0 con 0x7f8758004d90 2023-03-03T11:08:19.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.397+0000 7f875effd700 1 -- 172.21.15.7:0/4178944293 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8764005440 con 0x7f8758004d90 2023-03-03T11:08:19.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.397+0000 7f8774a76700 1 -- 172.21.15.7:0/4178944293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87701548c0 con 0x7f8758004d90 2023-03-03T11:08:19.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.397+0000 7f875effd700 1 -- 172.21.15.7:0/4178944293 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640038c0 con 0x7f8758004d90 2023-03-03T11:08:19.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.397+0000 7f8774a76700 1 -- 172.21.15.7:0/4178944293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877006dc40 con 0x7f8758004d90 2023-03-03T11:08:19.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.398+0000 7f8774a76700 1 -- 172.21.15.7:0/4178944293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8770074bb0 con 0x7f8758004d90 2023-03-03T11:08:19.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.398+0000 7f875effd700 1 -- 172.21.15.7:0/4178944293 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8764003a20 con 0x7f8758004d90 2023-03-03T11:08:19.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.399+0000 7f875effd700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87500682f0 0x7f875006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:19.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.399+0000 7f875effd700 1 -- 172.21.15.7:0/4178944293 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8764086110 con 0x7f8758004d90 2023-03-03T11:08:19.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.399+0000 7f876dd9b700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87500682f0 0x7f875006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:19.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.399+0000 7f876dd9b700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87500682f0 0x7f875006a7a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f87700727f0 tx=0x7f8760009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:19.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.403+0000 7f875effd700 1 -- 172.21.15.7:0/4178944293 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f87640524b0 con 0x7f8758004d90 2023-03-03T11:08:19.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.646+0000 7f8774a76700 1 -- 172.21.15.7:0/4178944293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87700744b0 con 0x7f8758004d90 2023-03-03T11:08:19.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.646+0000 7f875effd700 1 -- 172.21.15.7:0/4178944293 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8764003cb0 con 0x7f8758004d90 2023-03-03T11:08:19.648 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:08:19.649 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:08:19.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 -- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87500682f0 msgr2=0x7f875006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:19.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87500682f0 0x7f875006a7a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f87700727f0 tx=0x7f8760009040 comp rx=0 tx=0).stop 2023-03-03T11:08:19.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 -- 172.21.15.7:0/4178944293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 msgr2=0x7f87701ed390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:19.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 0x7f87701ed390 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f876400f820 tx=0x7f876400fbe0 comp rx=0 tx=0).stop 2023-03-03T11:08:19.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 -- 172.21.15.7:0/4178944293 shutdown_connections 2023-03-03T11:08:19.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87701ede10 0x7f87700aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:19.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87500682f0 0x7f875006a7a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:19.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8758004d90 0x7f87701ed390 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:19.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 --2- 172.21.15.7:0/4178944293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f877013cb00 0x7f87701ed8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:19.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 -- 172.21.15.7:0/4178944293 >> 172.21.15.7:0/4178944293 conn(0x7f877006c110 msgr2=0x7f8770070170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:08:19.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 -- 172.21.15.7:0/4178944293 shutdown_connections 2023-03-03T11:08:19.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:19.648+0000 7f8774a76700 1 -- 172.21.15.7:0/4178944293 wait complete. 2023-03-03T11:08:19.652 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:08:19.662 INFO:tasks.mds_thrash.fs.[a]:waiting for 25 secs before thrashing 2023-03-03T11:08:39.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:39.349 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:08:44.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:08:44.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.799+0000 7f5357e85700 1 Processor -- start 2023-03-03T11:08:44.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.799+0000 7f5357e85700 1 -- start start 2023-03-03T11:08:44.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5357e85700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f53500717a0 0x7f53500bdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:44.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5357e85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f535006e7f0 0x7f5350151c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:44.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5357e85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5350154760 0x7f5350071e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:44.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5357e85700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53500bd1a0 con 0x7f53500717a0 2023-03-03T11:08:44.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5357e85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5350152170 con 0x7f535006e7f0 2023-03-03T11:08:44.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5357e85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53501517a0 con 0x7f5350154760 2023-03-03T11:08:44.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5355c21700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f53500717a0 0x7f53500bdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:44.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5355c21700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f53500717a0 0x7f53500bdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35050/0 (socket says 172.21.15.7:35050) 2023-03-03T11:08:44.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5355c21700 1 -- 172.21.15.7:0/3268864746 learned_addr learned my addr 172.21.15.7:0/3268864746 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:08:44.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5356422700 1 --2- 172.21.15.7:0/3268864746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5350154760 0x7f5350071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:44.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5355420700 1 --2- 172.21.15.7:0/3268864746 >> v2:172.21.15.143:3300/0 conn(0x7f535006e7f0 0x7f5350151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:44.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5355c21700 1 -- 172.21.15.7:0/3268864746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5350154760 msgr2=0x7f5350071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:08:44.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5355c21700 1 --2- 172.21.15.7:0/3268864746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5350154760 0x7f5350071e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:44.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5355c21700 1 -- 172.21.15.7:0/3268864746 >> v2:172.21.15.143:3300/0 conn(0x7f535006e7f0 msgr2=0x7f5350151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:08:44.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5355c21700 1 --2- 172.21.15.7:0/3268864746 >> v2:172.21.15.143:3300/0 conn(0x7f535006e7f0 0x7f5350151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:44.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5355c21700 1 -- 172.21.15.7:0/3268864746 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5350151170 con 0x7f53500717a0 2023-03-03T11:08:44.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5355c21700 1 --2- 172.21.15.7:0/3268864746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53500717a0 0x7f53500bdcd0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f53500c1900 tx=0x7f534c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f01b9903876b1db server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:44.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.800+0000 7f5354c1f700 1 -- 172.21.15.7:0/3268864746 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c00b870 con 0x7f53500717a0 2023-03-03T11:08:44.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5354c1f700 1 -- 172.21.15.7:0/3268864746 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534c00b9d0 con 0x7f53500717a0 2023-03-03T11:08:44.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5354c1f700 1 -- 172.21.15.7:0/3268864746 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c00bd60 con 0x7f53500717a0 2023-03-03T11:08:44.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5357e85700 1 -- 172.21.15.7:0/3268864746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53500717a0 msgr2=0x7f53500bdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:44.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5357e85700 1 --2- 172.21.15.7:0/3268864746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53500717a0 0x7f53500bdcd0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f53500c1900 tx=0x7f534c009bd0 comp rx=0 tx=0).stop 2023-03-03T11:08:44.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5357e85700 1 -- 172.21.15.7:0/3268864746 shutdown_connections 2023-03-03T11:08:44.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5357e85700 1 --2- 172.21.15.7:0/3268864746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5350154760 0x7f5350071e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:44.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5357e85700 1 --2- 172.21.15.7:0/3268864746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53500717a0 0x7f53500bdcd0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:44.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5357e85700 1 --2- 172.21.15.7:0/3268864746 >> v2:172.21.15.143:3300/0 conn(0x7f535006e7f0 0x7f5350151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:44.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5357e85700 1 -- 172.21.15.7:0/3268864746 >> 172.21.15.7:0/3268864746 conn(0x7f53500a94c0 msgr2=0x7f53500a98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:08:44.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5357e85700 1 -- 172.21.15.7:0/3268864746 shutdown_connections 2023-03-03T11:08:44.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.801+0000 7f5357e85700 1 -- 172.21.15.7:0/3268864746 wait complete. 2023-03-03T11:08:44.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.802+0000 7f5357e85700 1 Processor -- start 2023-03-03T11:08:44.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.802+0000 7f5357e85700 1 -- start start 2023-03-03T11:08:44.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5357e85700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535006e7f0 0x7f53501e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:44.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5357e85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53500717a0 0x7f535014f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:44.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5357e85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5350154760 0x7f53501def50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:44.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5357e85700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53500c7bd0 con 0x7f535006e7f0 2023-03-03T11:08:44.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5357e85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5350150da0 con 0x7f5350154760 2023-03-03T11:08:44.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5357e85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53500abb70 con 0x7f53500717a0 2023-03-03T11:08:44.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5355c21700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535006e7f0 0x7f53501e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:44.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5355c21700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535006e7f0 0x7f53501e9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35056/0 (socket says 172.21.15.7:35056) 2023-03-03T11:08:44.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5355c21700 1 -- 172.21.15.7:0/3320120510 learned_addr learned my addr 172.21.15.7:0/3320120510 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:08:44.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5356422700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5350154760 0x7f53501def50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:44.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.803+0000 7f5355420700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53500717a0 0x7f535014f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:44.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f5355c21700 1 -- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53500717a0 msgr2=0x7f535014f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:44.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f5355c21700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53500717a0 0x7f535014f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:44.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f5355c21700 1 -- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5350154760 msgr2=0x7f53501def50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:44.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f5355c21700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5350154760 0x7f53501def50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:44.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f5355c21700 1 -- 172.21.15.7:0/3320120510 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c009550 con 0x7f535006e7f0 2023-03-03T11:08:44.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f5355420700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53500717a0 0x7f535014f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:08:44.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f5356422700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5350154760 0x7f53501def50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:08:44.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f5355c21700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535006e7f0 0x7f53501e9ec0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f534c000c00 tx=0x7f534c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:44.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f53467fc700 1 -- 172.21.15.7:0/3320120510 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c004540 con 0x7f535006e7f0 2023-03-03T11:08:44.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f53467fc700 1 -- 172.21.15.7:0/3320120510 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534c0036a0 con 0x7f535006e7f0 2023-03-03T11:08:44.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f5357e85700 1 -- 172.21.15.7:0/3320120510 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53500bc5d0 con 0x7f535006e7f0 2023-03-03T11:08:44.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f53467fc700 1 -- 172.21.15.7:0/3320120510 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c003a50 con 0x7f535006e7f0 2023-03-03T11:08:44.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.804+0000 7f5357e85700 1 -- 172.21.15.7:0/3320120510 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53500bc2f0 con 0x7f535006e7f0 2023-03-03T11:08:44.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.805+0000 7f5357e85700 1 -- 172.21.15.7:0/3320120510 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5350151430 con 0x7f535006e7f0 2023-03-03T11:08:44.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.805+0000 7f53467fc700 1 -- 172.21.15.7:0/3320120510 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f534c00f050 con 0x7f535006e7f0 2023-03-03T11:08:44.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.806+0000 7f53467fc700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53340682f0 0x7f533406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:44.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.806+0000 7f53467fc700 1 -- 172.21.15.7:0/3320120510 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f534c087a50 con 0x7f535006e7f0 2023-03-03T11:08:44.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.806+0000 7f5355420700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53340682f0 0x7f533406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:44.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.807+0000 7f5355420700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53340682f0 0x7f533406a7a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f53500c4630 tx=0x7f5340008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:44.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:44.810+0000 7f53467fc700 1 -- 172.21.15.7:0/3320120510 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f534c053d00 con 0x7f535006e7f0 2023-03-03T11:08:45.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.046+0000 7f5357e85700 1 -- 172.21.15.7:0/3320120510 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5350074bb0 con 0x7f535006e7f0 2023-03-03T11:08:45.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.046+0000 7f53467fc700 1 -- 172.21.15.7:0/3320120510 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f534c057320 con 0x7f535006e7f0 2023-03-03T11:08:45.048 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:08:45.049 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:08:45.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 -- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53340682f0 msgr2=0x7f533406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:45.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53340682f0 0x7f533406a7a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f53500c4630 tx=0x7f5340008040 comp rx=0 tx=0).stop 2023-03-03T11:08:45.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 -- 172.21.15.7:0/3320120510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535006e7f0 msgr2=0x7f53501e9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:45.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535006e7f0 0x7f53501e9ec0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f534c000c00 tx=0x7f534c004750 comp rx=0 tx=0).stop 2023-03-03T11:08:45.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 -- 172.21.15.7:0/3320120510 shutdown_connections 2023-03-03T11:08:45.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5350154760 0x7f53501def50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:45.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53340682f0 0x7f533406a7a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:45.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535006e7f0 0x7f53501e9ec0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:45.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 --2- 172.21.15.7:0/3320120510 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53500717a0 0x7f535014f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:45.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 -- 172.21.15.7:0/3320120510 >> 172.21.15.7:0/3320120510 conn(0x7f53500a94c0 msgr2=0x7f53500700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:08:45.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 -- 172.21.15.7:0/3320120510 shutdown_connections 2023-03-03T11:08:45.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:45.048+0000 7f5357e85700 1 -- 172.21.15.7:0/3320120510 wait complete. 2023-03-03T11:08:45.052 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:08:45.062 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before thrashing 2023-03-03T11:08:51.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:08:51.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.199+0000 7f0fdc1f7700 1 Processor -- start 2023-03-03T11:08:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.199+0000 7f0fdc1f7700 1 -- start start 2023-03-03T11:08:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.200+0000 7f0fdc1f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 0x7f0fd4158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.200+0000 7f0fdc1f7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd40a9c20 con 0x7f0fd40aad20 2023-03-03T11:08:51.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.200+0000 7f0fdc1f7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd4154f10 con 0x7f0fd414b0d0 2023-03-03T11:08:51.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.200+0000 7f0fdc1f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd40ac3c0 con 0x7f0fd40aa030 2023-03-03T11:08:51.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.200+0000 7f0fd9792700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0fd40aad20 0x7f0fd4157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57288/0 (socket says 172.21.15.7:57288) 2023-03-03T11:08:51.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.200+0000 7f0fd9792700 1 -- 172.21.15.7:0/776869505 learned_addr learned my addr 172.21.15.7:0/776869505 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:08:51.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fda794700 1 --2- 172.21.15.7:0/776869505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 0x7f0fd4158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fd8f91700 1 -- 172.21.15.7:0/776869505 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590271412 0 0) 0x7f0fd40a9c20 con 0x7f0fd40aad20 2023-03-03T11:08:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fd8f91700 1 -- 172.21.15.7:0/776869505 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fc00035b0 con 0x7f0fd40aad20 2023-03-03T11:08:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fd8f91700 1 -- 172.21.15.7:0/776869505 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088564570 0 0) 0x7f0fd4154f10 con 0x7f0fd414b0d0 2023-03-03T11:08:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fd8f91700 1 -- 172.21.15.7:0/776869505 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd40a9c20 con 0x7f0fd414b0d0 2023-03-03T11:08:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fda794700 1 -- 172.21.15.7:0/776869505 >> v1:172.21.15.143:6789/0 conn(0x7f0fd414b0d0 legacy=0x7f0fd4071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fda794700 1 -- 172.21.15.7:0/776869505 >> v1:172.21.15.7:6789/0 conn(0x7f0fd40aad20 legacy=0x7f0fd4157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fda794700 1 -- 172.21.15.7:0/776869505 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd406fc70 con 0x7f0fd40aa030 2023-03-03T11:08:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fda794700 1 --2- 172.21.15.7:0/776869505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 0x7f0fd4158500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc009dc0 tx=0x7f0fcc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63f3912b07b2a633 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fd8f91700 1 -- 172.21.15.7:0/776869505 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc00e070 con 0x7f0fd40aa030 2023-03-03T11:08:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fd8f91700 1 -- 172.21.15.7:0/776869505 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fcc00bb20 con 0x7f0fd40aa030 2023-03-03T11:08:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.201+0000 7f0fd8f91700 1 -- 172.21.15.7:0/776869505 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc012430 con 0x7f0fd40aa030 2023-03-03T11:08:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.202+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/776869505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 msgr2=0x7f0fd4158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.202+0000 7f0fdc1f7700 1 --2- 172.21.15.7:0/776869505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 0x7f0fd4158500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc009dc0 tx=0x7f0fcc00b330 comp rx=0 tx=0).stop 2023-03-03T11:08:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.202+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/776869505 shutdown_connections 2023-03-03T11:08:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.202+0000 7f0fdc1f7700 1 --2- 172.21.15.7:0/776869505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 0x7f0fd4158500 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.202+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/776869505 >> 172.21.15.7:0/776869505 conn(0x7f0fd40c8900 msgr2=0x7f0fd40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:08:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.202+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/776869505 shutdown_connections 2023-03-03T11:08:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.202+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/776869505 wait complete. 2023-03-03T11:08:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.203+0000 7f0fdc1f7700 1 Processor -- start 2023-03-03T11:08:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.203+0000 7f0fdc1f7700 1 -- start start 2023-03-03T11:08:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fdc1f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 0x7f0fd40c7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fdc1f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fd40aad20 0x7f0fd4152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fdc1f7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd414b0d0 0x7f0fd4149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fdc1f7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41534e0 con 0x7f0fd414b0d0 2023-03-03T11:08:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fdc1f7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd40c66f0 con 0x7f0fd40aad20 2023-03-03T11:08:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fdc1f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41ed5b0 con 0x7f0fd40aa030 2023-03-03T11:08:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fda794700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd414b0d0 0x7f0fd4149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fda794700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd414b0d0 0x7f0fd4149cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35082/0 (socket says 172.21.15.7:35082) 2023-03-03T11:08:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fda794700 1 -- 172.21.15.7:0/618606856 learned_addr learned my addr 172.21.15.7:0/618606856 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:08:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fd9792700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fd40aad20 0x7f0fd4152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fd9f93700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 0x7f0fd40c7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fda794700 1 -- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 msgr2=0x7f0fd40c7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:51.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fda794700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 0x7f0fd40c7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:51.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fda794700 1 -- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fd40aad20 msgr2=0x7f0fd4152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:51.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fda794700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fd40aad20 0x7f0fd4152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:51.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fda794700 1 -- 172.21.15.7:0/618606856 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc009a70 con 0x7f0fd414b0d0 2023-03-03T11:08:51.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.204+0000 7f0fd9f93700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 0x7f0fd40c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:08:51.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.205+0000 7f0fda794700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd414b0d0 0x7f0fd4149cd0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00b370 tx=0x7f0fcc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:51.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.205+0000 7f0fc67fc700 1 -- 172.21.15.7:0/618606856 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc01b810 con 0x7f0fd414b0d0 2023-03-03T11:08:51.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.205+0000 7f0fc67fc700 1 -- 172.21.15.7:0/618606856 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fcc020070 con 0x7f0fd414b0d0 2023-03-03T11:08:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.205+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/618606856 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd406f6b0 con 0x7f0fd414b0d0 2023-03-03T11:08:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.205+0000 7f0fc67fc700 1 -- 172.21.15.7:0/618606856 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc01bbf0 con 0x7f0fd414b0d0 2023-03-03T11:08:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.205+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/618606856 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd4155160 con 0x7f0fd414b0d0 2023-03-03T11:08:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.206+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/618606856 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd406f990 con 0x7f0fd414b0d0 2023-03-03T11:08:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.206+0000 7f0fc67fc700 1 -- 172.21.15.7:0/618606856 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0fcc01bd50 con 0x7f0fd414b0d0 2023-03-03T11:08:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.206+0000 7f0fc67fc700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0fb80682f0 0x7f0fb806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:08:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.207+0000 7f0fc67fc700 1 -- 172.21.15.7:0/618606856 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0fcc022070 con 0x7f0fd414b0d0 2023-03-03T11:08:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.207+0000 7f0fd9f93700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0fb80682f0 0x7f0fb806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:08:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.208+0000 7f0fd9f93700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0fb80682f0 0x7f0fb806a7a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8007fd0 tx=0x7f0fc8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:08:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.211+0000 7f0fc67fc700 1 -- 172.21.15.7:0/618606856 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0fcc054b80 con 0x7f0fd414b0d0 2023-03-03T11:08:51.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.448+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/618606856 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fd41ed720 con 0x7f0fd414b0d0 2023-03-03T11:08:51.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.449+0000 7f0fc67fc700 1 -- 172.21.15.7:0/618606856 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0fcc0581a0 con 0x7f0fd414b0d0 2023-03-03T11:08:51.451 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:08:51.451 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:08:51.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0fb80682f0 msgr2=0x7f0fb806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:51.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0fb80682f0 0x7f0fb806a7a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8007fd0 tx=0x7f0fc8009040 comp rx=0 tx=0).stop 2023-03-03T11:08:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/618606856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd414b0d0 msgr2=0x7f0fd4149cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:08:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd414b0d0 0x7f0fd4149cd0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00b370 tx=0x7f0fcc004ad0 comp rx=0 tx=0).stop 2023-03-03T11:08:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/618606856 shutdown_connections 2023-03-03T11:08:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0fb80682f0 0x7f0fb806a7a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fd414b0d0 0x7f0fd4149cd0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fd40aa030 0x7f0fd40c7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 --2- 172.21.15.7:0/618606856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fd40aad20 0x7f0fd4152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:08:51.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/618606856 >> 172.21.15.7:0/618606856 conn(0x7f0fd40c8900 msgr2=0x7f0fd40acb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:08:51.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/618606856 shutdown_connections 2023-03-03T11:08:51.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:08:51.451+0000 7f0fdc1f7700 1 -- 172.21.15.7:0/618606856 wait complete. 2023-03-03T11:08:51.455 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:08:51.465 INFO:tasks.mds_thrash.fs.[a]:waiting for 67 secs before thrashing 2023-03-03T11:09:08.815 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:09:08.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb64cd4700 1 Processor -- start 2023-03-03T11:09:08.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb64cd4700 1 -- start start 2023-03-03T11:09:08.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb64cd4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb60155090 0x7ffb6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:08.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb64cd4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffb60157fe0 0x7ffb60158460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:08.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb64cd4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb60157b50 con 0x7ffb6006c560 2023-03-03T11:09:08.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb64cd4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6006c240 con 0x7ffb60157fe0 2023-03-03T11:09:08.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb64cd4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb600c33b0 con 0x7ffb60155090 2023-03-03T11:09:08.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb5e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb60155090 0x7ffb6006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:08.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb5ed9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffb60157fe0 0x7ffb60158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:08.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb5ed9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffb60157fe0 0x7ffb60158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:59108/0 (socket says 172.21.15.7:59108) 2023-03-03T11:09:08.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb5e59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb60155090 0x7ffb6006d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:43174/0 (socket says 172.21.15.7:43174) 2023-03-03T11:09:08.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb5ed9d700 1 -- 172.21.15.7:0/1127281048 learned_addr learned my addr 172.21.15.7:0/1127281048 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:09:08.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.951+0000 7ffb5dd9b700 1 --1- 172.21.15.7:0/1127281048 >> v1:172.21.15.143:6789/0 conn(0x7ffb6006c560 0x7ffb600736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:60834/0 (socket says 172.21.15.7:60834) 2023-03-03T11:09:08.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.952+0000 7ffb5e59c700 1 -- 172.21.15.7:0/1127281048 >> v2:172.21.15.143:3300/0 conn(0x7ffb60157fe0 msgr2=0x7ffb60158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:09:08.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.952+0000 7ffb5e59c700 1 --2- 172.21.15.7:0/1127281048 >> v2:172.21.15.143:3300/0 conn(0x7ffb60157fe0 0x7ffb60158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:08.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.952+0000 7ffb5e59c700 1 -- 172.21.15.7:0/1127281048 >> v1:172.21.15.143:6789/0 conn(0x7ffb6006c560 legacy=0x7ffb600736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:08.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.952+0000 7ffb5e59c700 1 -- 172.21.15.7:0/1127281048 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb600c3690 con 0x7ffb60155090 2023-03-03T11:09:08.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.952+0000 7ffb5d59a700 1 -- 172.21.15.7:0/1127281048 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519589921 0 0) 0x7ffb60157b50 con 0x7ffb6006c560 2023-03-03T11:09:08.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.952+0000 7ffb5ed9d700 1 --2- 172.21.15.7:0/1127281048 >> v2:172.21.15.143:3300/0 conn(0x7ffb60157fe0 0x7ffb60158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:09:08.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.952+0000 7ffb5e59c700 1 --2- 172.21.15.7:0/1127281048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb60155090 0x7ffb6006d450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009840 tx=0x7ffb54009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e37d4c74e05f8d1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:08.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.952+0000 7ffb5d59a700 1 -- 172.21.15.7:0/1127281048 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54011830 con 0x7ffb60155090 2023-03-03T11:09:08.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.952+0000 7ffb5d59a700 1 -- 172.21.15.7:0/1127281048 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb54011990 con 0x7ffb60155090 2023-03-03T11:09:08.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.952+0000 7ffb5d59a700 1 -- 172.21.15.7:0/1127281048 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb54011d20 con 0x7ffb60155090 2023-03-03T11:09:08.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.953+0000 7ffb64cd4700 1 -- 172.21.15.7:0/1127281048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb60155090 msgr2=0x7ffb6006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:08.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.953+0000 7ffb64cd4700 1 --2- 172.21.15.7:0/1127281048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb60155090 0x7ffb6006d450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb54009840 tx=0x7ffb54009b50 comp rx=0 tx=0).stop 2023-03-03T11:09:08.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.953+0000 7ffb64cd4700 1 -- 172.21.15.7:0/1127281048 shutdown_connections 2023-03-03T11:09:08.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.953+0000 7ffb64cd4700 1 --2- 172.21.15.7:0/1127281048 >> v2:172.21.15.143:3300/0 conn(0x7ffb60157fe0 0x7ffb60158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:08.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.953+0000 7ffb64cd4700 1 --2- 172.21.15.7:0/1127281048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb60155090 0x7ffb6006d450 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:08.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.953+0000 7ffb64cd4700 1 -- 172.21.15.7:0/1127281048 >> 172.21.15.7:0/1127281048 conn(0x7ffb600a94d0 msgr2=0x7ffb600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:09:08.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.953+0000 7ffb64cd4700 1 -- 172.21.15.7:0/1127281048 shutdown_connections 2023-03-03T11:09:08.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.953+0000 7ffb64cd4700 1 -- 172.21.15.7:0/1127281048 wait complete. 2023-03-03T11:09:08.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.954+0000 7ffb64cd4700 1 Processor -- start 2023-03-03T11:09:08.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.954+0000 7ffb64cd4700 1 -- start start 2023-03-03T11:09:08.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb64cd4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb6006c560 0x7ffb60150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:08.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb64cd4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb60155090 0x7ffb6014a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:08.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb64cd4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb60157fe0 0x7ffb601eada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:08.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb64cd4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb601e9930 con 0x7ffb60157fe0 2023-03-03T11:09:08.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb64cd4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb60156060 con 0x7ffb60155090 2023-03-03T11:09:08.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb64cd4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb601edc00 con 0x7ffb6006c560 2023-03-03T11:09:08.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb5ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb60157fe0 0x7ffb601eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:08.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb5ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb60157fe0 0x7ffb601eada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35110/0 (socket says 172.21.15.7:35110) 2023-03-03T11:09:08.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb5ed9d700 1 -- 172.21.15.7:0/3555214006 learned_addr learned my addr 172.21.15.7:0/3555214006 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:09:08.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb5e59c700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb6006c560 0x7ffb60150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:08.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb5dd9b700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb60155090 0x7ffb6014a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:08.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb5ed9d700 1 -- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb6006c560 msgr2=0x7ffb60150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:08.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb5ed9d700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb6006c560 0x7ffb60150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:08.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb5ed9d700 1 -- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb60155090 msgr2=0x7ffb6014a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:08.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb5ed9d700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb60155090 0x7ffb6014a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:08.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.955+0000 7ffb5ed9d700 1 -- 172.21.15.7:0/3555214006 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb58009b30 con 0x7ffb60157fe0 2023-03-03T11:09:08.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.956+0000 7ffb5ed9d700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb60157fe0 0x7ffb601eada0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800efc0 tx=0x7ffb5800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:08.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.956+0000 7ffb4effd700 1 -- 172.21.15.7:0/3555214006 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58018070 con 0x7ffb60157fe0 2023-03-03T11:09:08.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.956+0000 7ffb4effd700 1 -- 172.21.15.7:0/3555214006 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5800f040 con 0x7ffb60157fe0 2023-03-03T11:09:08.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.956+0000 7ffb64cd4700 1 -- 172.21.15.7:0/3555214006 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb540094a0 con 0x7ffb60157fe0 2023-03-03T11:09:08.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.956+0000 7ffb4effd700 1 -- 172.21.15.7:0/3555214006 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58003c40 con 0x7ffb60157fe0 2023-03-03T11:09:08.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.956+0000 7ffb64cd4700 1 -- 172.21.15.7:0/3555214006 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb601589a0 con 0x7ffb60157fe0 2023-03-03T11:09:08.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.957+0000 7ffb64cd4700 1 -- 172.21.15.7:0/3555214006 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb600c3970 con 0x7ffb60157fe0 2023-03-03T11:09:08.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.957+0000 7ffb4effd700 1 -- 172.21.15.7:0/3555214006 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffb5800c880 con 0x7ffb60157fe0 2023-03-03T11:09:08.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.958+0000 7ffb4effd700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb400682f0 0x7ffb4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:08.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.958+0000 7ffb4effd700 1 -- 172.21.15.7:0/3555214006 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ffb580863f0 con 0x7ffb60157fe0 2023-03-03T11:09:08.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.958+0000 7ffb5e59c700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb400682f0 0x7ffb4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:08.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.958+0000 7ffb5e59c700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb400682f0 0x7ffb4006a7a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ffb540090b0 tx=0x7ffb54008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:08.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:08.963+0000 7ffb4effd700 1 -- 172.21.15.7:0/3555214006 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffb58052710 con 0x7ffb60157fe0 2023-03-03T11:09:09.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.201+0000 7ffb64cd4700 1 -- 172.21.15.7:0/3555214006 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb601545e0 con 0x7ffb60157fe0 2023-03-03T11:09:09.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.202+0000 7ffb4effd700 1 -- 172.21.15.7:0/3555214006 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ffb58009ce0 con 0x7ffb60157fe0 2023-03-03T11:09:09.205 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:09:09.205 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:09:09.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 -- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb400682f0 msgr2=0x7ffb4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:09.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb400682f0 0x7ffb4006a7a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ffb540090b0 tx=0x7ffb54008030 comp rx=0 tx=0).stop 2023-03-03T11:09:09.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 -- 172.21.15.7:0/3555214006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb60157fe0 msgr2=0x7ffb601eada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:09.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb60157fe0 0x7ffb601eada0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7ffb5800efc0 tx=0x7ffb5800c3a0 comp rx=0 tx=0).stop 2023-03-03T11:09:09.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 -- 172.21.15.7:0/3555214006 shutdown_connections 2023-03-03T11:09:09.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb400682f0 0x7ffb4006a7a0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:09.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb60157fe0 0x7ffb601eada0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:09.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb6006c560 0x7ffb60150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:09.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 --2- 172.21.15.7:0/3555214006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb60155090 0x7ffb6014a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:09.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 -- 172.21.15.7:0/3555214006 >> 172.21.15.7:0/3555214006 conn(0x7ffb600a94d0 msgr2=0x7ffb600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:09:09.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 -- 172.21.15.7:0/3555214006 shutdown_connections 2023-03-03T11:09:09.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:09.204+0000 7ffb64cd4700 1 -- 172.21.15.7:0/3555214006 wait complete. 2023-03-03T11:09:09.209 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:09:09.219 INFO:tasks.mds_thrash.fs.[b]:waiting for 45 secs before thrashing 2023-03-03T11:09:09.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:09.461 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:39.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:39.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:09:54.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:09:54.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.352+0000 7fdc26d72700 1 Processor -- start 2023-03-03T11:09:54.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.352+0000 7fdc26d72700 1 -- start start 2023-03-03T11:09:54.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.352+0000 7fdc26d72700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdc2014f9d0 0x7fdc20156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:54.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.352+0000 7fdc26d72700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 0x7fdc20071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc26d72700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc200c2f30 con 0x7fdc200abe20 2023-03-03T11:09:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc26d72700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc200c31d0 con 0x7fdc2014f9d0 2023-03-03T11:09:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc26d72700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201524a0 con 0x7fdc2014fda0 2023-03-03T11:09:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc24b0e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdc2014f9d0 0x7fdc20156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc2530f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdc200abe20 0x7fdc201529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57356/0 (socket says 172.21.15.7:57356) 2023-03-03T11:09:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc24b0e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdc2014f9d0 0x7fdc20156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35144/0 (socket says 172.21.15.7:35144) 2023-03-03T11:09:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc2530f700 1 -- 172.21.15.7:0/163532919 learned_addr learned my addr 172.21.15.7:0/163532919 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:09:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc24b0e700 1 -- 172.21.15.7:0/163532919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 msgr2=0x7fdc20071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:09:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc1ffff700 1 --2- 172.21.15.7:0/163532919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 0x7fdc20071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc24b0e700 1 --2- 172.21.15.7:0/163532919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 0x7fdc20071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc24b0e700 1 -- 172.21.15.7:0/163532919 >> v1:172.21.15.7:6789/0 conn(0x7fdc200abe20 legacy=0x7fdc201529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc24b0e700 1 -- 172.21.15.7:0/163532919 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc200c34b0 con 0x7fdc2014f9d0 2023-03-03T11:09:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc1f7fe700 1 -- 172.21.15.7:0/163532919 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534348671 0 0) 0x7fdc200c2f30 con 0x7fdc200abe20 2023-03-03T11:09:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc1ffff700 1 --2- 172.21.15.7:0/163532919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 0x7fdc20071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:09:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc24b0e700 1 --2- 172.21.15.7:0/163532919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014f9d0 0x7fdc20156430 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009840 tx=0x7fdc14009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8f3390c8cae985 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc1f7fe700 1 -- 172.21.15.7:0/163532919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14011830 con 0x7fdc2014f9d0 2023-03-03T11:09:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc1f7fe700 1 -- 172.21.15.7:0/163532919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc14011990 con 0x7fdc2014f9d0 2023-03-03T11:09:54.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.353+0000 7fdc1f7fe700 1 -- 172.21.15.7:0/163532919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14011d20 con 0x7fdc2014f9d0 2023-03-03T11:09:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.354+0000 7fdc26d72700 1 -- 172.21.15.7:0/163532919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014f9d0 msgr2=0x7fdc20156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.354+0000 7fdc26d72700 1 --2- 172.21.15.7:0/163532919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014f9d0 0x7fdc20156430 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009840 tx=0x7fdc14009b50 comp rx=0 tx=0).stop 2023-03-03T11:09:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.354+0000 7fdc26d72700 1 -- 172.21.15.7:0/163532919 shutdown_connections 2023-03-03T11:09:54.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.354+0000 7fdc26d72700 1 --2- 172.21.15.7:0/163532919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014f9d0 0x7fdc20156430 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:54.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.354+0000 7fdc26d72700 1 --2- 172.21.15.7:0/163532919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 0x7fdc20071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:54.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.354+0000 7fdc26d72700 1 -- 172.21.15.7:0/163532919 >> 172.21.15.7:0/163532919 conn(0x7fdc200c8900 msgr2=0x7fdc200c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:09:54.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.354+0000 7fdc26d72700 1 -- 172.21.15.7:0/163532919 shutdown_connections 2023-03-03T11:09:54.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.354+0000 7fdc26d72700 1 -- 172.21.15.7:0/163532919 wait complete. 2023-03-03T11:09:54.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.355+0000 7fdc26d72700 1 Processor -- start 2023-03-03T11:09:54.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.355+0000 7fdc26d72700 1 -- start start 2023-03-03T11:09:54.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc26d72700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc200abe20 0x7fdc20143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc26d72700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc2014f9d0 0x7fdc20144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc26d72700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 0x7fdc200c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc26d72700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201e9310 con 0x7fdc200abe20 2023-03-03T11:09:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc26d72700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201ed640 con 0x7fdc2014f9d0 2023-03-03T11:09:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc26d72700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201499f0 con 0x7fdc2014fda0 2023-03-03T11:09:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc24b0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc200abe20 0x7fdc20143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc24b0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc200abe20 0x7fdc20143db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35148/0 (socket says 172.21.15.7:35148) 2023-03-03T11:09:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc24b0e700 1 -- 172.21.15.7:0/847044971 learned_addr learned my addr 172.21.15.7:0/847044971 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:09:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc2530f700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 0x7fdc200c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc1ffff700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc2014f9d0 0x7fdc20144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc24b0e700 1 -- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 msgr2=0x7fdc200c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.356+0000 7fdc24b0e700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 0x7fdc200c7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.357+0000 7fdc24b0e700 1 -- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc2014f9d0 msgr2=0x7fdc20144560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.357+0000 7fdc24b0e700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc2014f9d0 0x7fdc20144560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:54.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.357+0000 7fdc24b0e700 1 -- 172.21.15.7:0/847044971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc140094a0 con 0x7fdc200abe20 2023-03-03T11:09:54.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.357+0000 7fdc24b0e700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc200abe20 0x7fdc20143db0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fdc14000c00 tx=0x7fdc14004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:54.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.357+0000 7fdc1d7fa700 1 -- 172.21.15.7:0/847044971 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14004480 con 0x7fdc200abe20 2023-03-03T11:09:54.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.357+0000 7fdc1d7fa700 1 -- 172.21.15.7:0/847044971 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1401f070 con 0x7fdc200abe20 2023-03-03T11:09:54.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.357+0000 7fdc26d72700 1 -- 172.21.15.7:0/847044971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc200ac350 con 0x7fdc200abe20 2023-03-03T11:09:54.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.357+0000 7fdc1d7fa700 1 -- 172.21.15.7:0/847044971 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc140038e0 con 0x7fdc200abe20 2023-03-03T11:09:54.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.357+0000 7fdc26d72700 1 -- 172.21.15.7:0/847044971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc200ab630 con 0x7fdc200abe20 2023-03-03T11:09:54.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.358+0000 7fdc26d72700 1 -- 172.21.15.7:0/847044971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc200c3e10 con 0x7fdc200abe20 2023-03-03T11:09:54.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.358+0000 7fdc1d7fa700 1 -- 172.21.15.7:0/847044971 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc1402a020 con 0x7fdc200abe20 2023-03-03T11:09:54.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.359+0000 7fdc1d7fa700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 0x7fdc0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:54.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.359+0000 7fdc1d7fa700 1 -- 172.21.15.7:0/847044971 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdc140875d0 con 0x7fdc200abe20 2023-03-03T11:09:54.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.359+0000 7fdc1ffff700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 0x7fdc0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:54.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.360+0000 7fdc1ffff700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 0x7fdc0406a7a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fdc200c81b0 tx=0x7fdc0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:54.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.363+0000 7fdc1d7fa700 1 -- 172.21.15.7:0/847044971 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc14053970 con 0x7fdc200abe20 2023-03-03T11:09:54.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.602+0000 7fdc26d72700 1 -- 172.21.15.7:0/847044971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc201e9580 con 0x7fdc200abe20 2023-03-03T11:09:54.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.603+0000 7fdc1d7fa700 1 -- 172.21.15.7:0/847044971 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdc14056f90 con 0x7fdc200abe20 2023-03-03T11:09:54.605 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:09:54.606 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:09:54.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 -- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 msgr2=0x7fdc0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:54.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 0x7fdc0406a7a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fdc200c81b0 tx=0x7fdc0c008040 comp rx=0 tx=0).stop 2023-03-03T11:09:54.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 -- 172.21.15.7:0/847044971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc200abe20 msgr2=0x7fdc20143db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc200abe20 0x7fdc20143db0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fdc14000c00 tx=0x7fdc14004750 comp rx=0 tx=0).stop 2023-03-03T11:09:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 -- 172.21.15.7:0/847044971 shutdown_connections 2023-03-03T11:09:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc2014fda0 0x7fdc200c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 0x7fdc0406a7a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc200abe20 0x7fdc20143db0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 --2- 172.21.15.7:0/847044971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc2014f9d0 0x7fdc20144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 -- 172.21.15.7:0/847044971 >> 172.21.15.7:0/847044971 conn(0x7fdc200c8900 msgr2=0x7fdc20150c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:09:54.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 -- 172.21.15.7:0/847044971 shutdown_connections 2023-03-03T11:09:54.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:54.605+0000 7fdc26d72700 1 -- 172.21.15.7:0/847044971 wait complete. 2023-03-03T11:09:54.609 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:09:54.619 INFO:tasks.mds_thrash.fs.[b]:waiting for 76 secs before thrashing 2023-03-03T11:09:58.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:09:58.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.603+0000 7f7cd3a0f700 1 Processor -- start 2023-03-03T11:09:58.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.604+0000 7f7cd3a0f700 1 -- start start 2023-03-03T11:09:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.604+0000 7f7cd3a0f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7ccc14f9d0 0x7f7ccc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:58.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.604+0000 7f7cd3a0f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ccc0abe20 0x7f7ccc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.604+0000 7f7cd3a0f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc0c2f30 con 0x7f7ccc14fda0 2023-03-03T11:09:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.604+0000 7f7cd3a0f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc0c31d0 con 0x7f7ccc0abe20 2023-03-03T11:09:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.604+0000 7f7cd3a0f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc1524a0 con 0x7f7ccc14f9d0 2023-03-03T11:09:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.604+0000 7f7cd0faa700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7ccc14fda0 0x7f7ccc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57368/0 (socket says 172.21.15.7:57368) 2023-03-03T11:09:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.604+0000 7f7cd0faa700 1 -- 172.21.15.7:0/1160095738 learned_addr learned my addr 172.21.15.7:0/1160095738 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:09:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.604+0000 7f7cd1fac700 1 --2- 172.21.15.7:0/1160095738 >> v2:172.21.15.7:3300/0 conn(0x7f7ccc0abe20 0x7f7ccc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:58.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cc3fff700 1 -- 172.21.15.7:0/1160095738 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640821851 0 0) 0x7f7ccc0c2f30 con 0x7f7ccc14fda0 2023-03-03T11:09:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cc3fff700 1 -- 172.21.15.7:0/1160095738 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb80035b0 con 0x7f7ccc14fda0 2023-03-03T11:09:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cd17ab700 1 --2- 172.21.15.7:0/1160095738 >> v2:172.21.15.143:3300/0 conn(0x7f7ccc14f9d0 0x7f7ccc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cd1fac700 1 -- 172.21.15.7:0/1160095738 >> v2:172.21.15.143:3300/0 conn(0x7f7ccc14f9d0 msgr2=0x7f7ccc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:09:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cd1fac700 1 --2- 172.21.15.7:0/1160095738 >> v2:172.21.15.143:3300/0 conn(0x7f7ccc14f9d0 0x7f7ccc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cd1fac700 1 -- 172.21.15.7:0/1160095738 >> v1:172.21.15.7:6789/0 conn(0x7f7ccc14fda0 legacy=0x7f7ccc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cd1fac700 1 -- 172.21.15.7:0/1160095738 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc0c34b0 con 0x7f7ccc0abe20 2023-03-03T11:09:58.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cd17ab700 1 --2- 172.21.15.7:0/1160095738 >> v2:172.21.15.143:3300/0 conn(0x7f7ccc14f9d0 0x7f7ccc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:09:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cd1fac700 1 --2- 172.21.15.7:0/1160095738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc0abe20 0x7f7ccc1529c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0710a0 tx=0x7f7cc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70068382372f44bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cc3fff700 1 -- 172.21.15.7:0/1160095738 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc400e070 con 0x7f7ccc0abe20 2023-03-03T11:09:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cc3fff700 1 -- 172.21.15.7:0/1160095738 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc400bb20 con 0x7f7ccc0abe20 2023-03-03T11:09:58.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.605+0000 7f7cc3fff700 1 -- 172.21.15.7:0/1160095738 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4012430 con 0x7f7ccc0abe20 2023-03-03T11:09:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.606+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/1160095738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc0abe20 msgr2=0x7f7ccc1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.606+0000 7f7cd3a0f700 1 --2- 172.21.15.7:0/1160095738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc0abe20 0x7f7ccc1529c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0710a0 tx=0x7f7cc400b330 comp rx=0 tx=0).stop 2023-03-03T11:09:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.606+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/1160095738 shutdown_connections 2023-03-03T11:09:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.606+0000 7f7cd3a0f700 1 --2- 172.21.15.7:0/1160095738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc0abe20 0x7f7ccc1529c0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:58.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.606+0000 7f7cd3a0f700 1 --2- 172.21.15.7:0/1160095738 >> v2:172.21.15.143:3300/0 conn(0x7f7ccc14f9d0 0x7f7ccc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.606+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/1160095738 >> 172.21.15.7:0/1160095738 conn(0x7f7ccc0c8900 msgr2=0x7f7ccc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:09:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.606+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/1160095738 shutdown_connections 2023-03-03T11:09:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.606+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/1160095738 wait complete. 2023-03-03T11:09:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.607+0000 7f7cd3a0f700 1 Processor -- start 2023-03-03T11:09:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.607+0000 7f7cd3a0f700 1 -- start start 2023-03-03T11:09:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd3a0f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ccc0abe20 0x7f7ccc06f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:58.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd3a0f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14f9d0 0x7f7ccc154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd3a0f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14fda0 0x7f7ccc06e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd3a0f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc06cf70 con 0x7f7ccc14fda0 2023-03-03T11:09:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd3a0f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc1532c0 con 0x7f7ccc0abe20 2023-03-03T11:09:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd3a0f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0bd3d0 con 0x7f7ccc14f9d0 2023-03-03T11:09:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd1fac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14fda0 0x7f7ccc06e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd1fac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14fda0 0x7f7ccc06e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35166/0 (socket says 172.21.15.7:35166) 2023-03-03T11:09:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd1fac700 1 -- 172.21.15.7:0/3592711215 learned_addr learned my addr 172.21.15.7:0/3592711215 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:09:58.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd0faa700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14f9d0 0x7f7ccc154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:58.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd17ab700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ccc0abe20 0x7f7ccc06f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:58.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd1fac700 1 -- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14f9d0 msgr2=0x7f7ccc154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:58.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd1fac700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14f9d0 0x7f7ccc154550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:58.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd1fac700 1 -- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ccc0abe20 msgr2=0x7f7ccc06f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:58.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd1fac700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ccc0abe20 0x7f7ccc06f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:58.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.608+0000 7f7cd1fac700 1 -- 172.21.15.7:0/3592711215 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc4009a70 con 0x7f7ccc14fda0 2023-03-03T11:09:58.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.609+0000 7f7cd1fac700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14fda0 0x7f7ccc06e220 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4014040 tx=0x7f7cc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:58.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.609+0000 7f7cc1ffb700 1 -- 172.21.15.7:0/3592711215 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4012430 con 0x7f7ccc14fda0 2023-03-03T11:09:58.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.609+0000 7f7cc1ffb700 1 -- 172.21.15.7:0/3592711215 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc4020070 con 0x7f7ccc14fda0 2023-03-03T11:09:58.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.609+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/3592711215 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc0ac350 con 0x7f7ccc14fda0 2023-03-03T11:09:58.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.609+0000 7f7cc1ffb700 1 -- 172.21.15.7:0/3592711215 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc40098f0 con 0x7f7ccc14fda0 2023-03-03T11:09:58.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.609+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/3592711215 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc0ab630 con 0x7f7ccc14fda0 2023-03-03T11:09:58.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.610+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/3592711215 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc071fa0 con 0x7f7ccc14fda0 2023-03-03T11:09:58.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.611+0000 7f7cc1ffb700 1 -- 172.21.15.7:0/3592711215 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7cc4004040 con 0x7f7ccc14fda0 2023-03-03T11:09:58.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.611+0000 7f7cc1ffb700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb0068320 0x7f7cb006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:09:58.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.611+0000 7f7cc1ffb700 1 -- 172.21.15.7:0/3592711215 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7cc4022070 con 0x7f7ccc14fda0 2023-03-03T11:09:58.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.611+0000 7f7cd17ab700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb0068320 0x7f7cb006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:09:58.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.612+0000 7f7cd17ab700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb0068320 0x7f7cb006a7d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8006fd0 tx=0x7f7cc8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:09:58.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.616+0000 7f7cc1ffb700 1 -- 172.21.15.7:0/3592711215 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7cc4054710 con 0x7f7ccc14fda0 2023-03-03T11:09:58.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.852+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/3592711215 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ccc074bb0 con 0x7f7ccc14fda0 2023-03-03T11:09:58.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.852+0000 7f7cc1ffb700 1 -- 172.21.15.7:0/3592711215 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7cc4057d30 con 0x7f7ccc14fda0 2023-03-03T11:09:58.854 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:09:58.855 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:09:58.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.855+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb0068320 msgr2=0x7f7cb006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:58.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.855+0000 7f7cd3a0f700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb0068320 0x7f7cb006a7d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8006fd0 tx=0x7f7cc8008040 comp rx=0 tx=0).stop 2023-03-03T11:09:58.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.855+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/3592711215 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14fda0 msgr2=0x7f7ccc06e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:09:58.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.855+0000 7f7cd3a0f700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14fda0 0x7f7ccc06e220 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4014040 tx=0x7f7cc4004ad0 comp rx=0 tx=0).stop 2023-03-03T11:09:58.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.856+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/3592711215 shutdown_connections 2023-03-03T11:09:58.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.856+0000 7f7cd3a0f700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cb0068320 0x7f7cb006a7d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:58.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.856+0000 7f7cd3a0f700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc14fda0 0x7f7ccc06e220 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:58.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.856+0000 7f7cd3a0f700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ccc0abe20 0x7f7ccc06f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:58.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.856+0000 7f7cd3a0f700 1 --2- 172.21.15.7:0/3592711215 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ccc14f9d0 0x7f7ccc154550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:09:58.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.856+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/3592711215 >> 172.21.15.7:0/3592711215 conn(0x7f7ccc0c8900 msgr2=0x7f7ccc151900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:09:58.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.856+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/3592711215 shutdown_connections 2023-03-03T11:09:58.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:09:58.856+0000 7f7cd3a0f700 1 -- 172.21.15.7:0/3592711215 wait complete. 2023-03-03T11:09:58.860 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:09:58.870 INFO:tasks.mds_thrash.fs.[a]:waiting for 83 secs before thrashing 2023-03-03T11:10:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T11:10:00.000+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T11:10:09.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:09.670 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:39.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:10:39.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:09.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:09.890 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:10.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:11:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.756+0000 7fba8ff7e700 1 Processor -- start 2023-03-03T11:11:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.756+0000 7fba8ff7e700 1 -- start start 2023-03-03T11:11:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8ff7e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fba880beab0 0x7fba88155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8ff7e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba881523b0 con 0x7fba881562c0 2023-03-03T11:11:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8ff7e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba88072010 con 0x7fba88152e30 2023-03-03T11:11:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8ff7e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba880aaaf0 con 0x7fba880beab0 2023-03-03T11:11:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8e51b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fba881562c0 0x7fba8815cd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57470/0 (socket says 172.21.15.7:57470) 2023-03-03T11:11:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8e51b700 1 -- 172.21.15.7:0/3120741019 learned_addr learned my addr 172.21.15.7:0/3120741019 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:11:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8d519700 1 --2- 172.21.15.7:0/3120741019 >> v2:172.21.15.143:3300/0 conn(0x7fba880beab0 0x7fba88155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:10.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212301982 0 0) 0x7fba881523b0 con 0x7fba881562c0 2023-03-03T11:11:10.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba740035b0 con 0x7fba881562c0 2023-03-03T11:11:10.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3087080633 0 0) 0x7fba740035b0 con 0x7fba881562c0 2023-03-03T11:11:10.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 >> v2:172.21.15.143:3300/0 conn(0x7fba880beab0 msgr2=0x7fba88155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:11:10.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8cd18700 1 --2- 172.21.15.7:0/3120741019 >> v2:172.21.15.143:3300/0 conn(0x7fba880beab0 0x7fba88155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:10.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.757+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 >> v1:172.21.15.143:6789/0 conn(0x7fba88152e30 legacy=0x7fba880c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:10.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba880aadd0 con 0x7fba881562c0 2023-03-03T11:11:10.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fba800019f0 con 0x7fba881562c0 2023-03-03T11:11:10.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 >> v1:172.21.15.7:6789/0 conn(0x7fba881562c0 legacy=0x7fba8815cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:10.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba74003a70 0x7fba74003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:10.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba74004850 0x7fba74004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba74005250 0x7fba7400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba880aaaf0 con 0x7fba74004850 2023-03-03T11:11:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7400c190 con 0x7fba74005250 2023-03-03T11:11:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7400c350 con 0x7fba74003a70 2023-03-03T11:11:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8d519700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba74004850 0x7fba74004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8e51b700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba74005250 0x7fba7400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:10.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8dd1a700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba74003a70 0x7fba74003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8d519700 1 -- 172.21.15.7:0/3120741019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba74003a70 msgr2=0x7fba74003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:11:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8d519700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba74003a70 0x7fba74003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8d519700 1 -- 172.21.15.7:0/3120741019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba74005250 msgr2=0x7fba7400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:11:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8d519700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba74005250 0x7fba7400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8e51b700 1 -- 172.21.15.7:0/3120741019 reap_dead start 2023-03-03T11:11:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8d519700 1 -- 172.21.15.7:0/3120741019 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba740035b0 con 0x7fba74004850 2023-03-03T11:11:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8d519700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba74004850 0x7fba74004cc0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fba78009b30 tx=0x7fba74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6837dab235100099 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:11:10.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7800e640 con 0x7fba74004850 2023-03-03T11:11:10.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7800e7a0 con 0x7fba74004850 2023-03-03T11:11:10.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.758+0000 7fba8cd18700 1 -- 172.21.15.7:0/3120741019 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7800eb10 con 0x7fba74004850 2023-03-03T11:11:10.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.759+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3120741019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba74004850 msgr2=0x7fba74004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:10.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.759+0000 7fba8ff7e700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba74004850 0x7fba74004cc0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fba78009b30 tx=0x7fba74005880 comp rx=0 tx=0).stop 2023-03-03T11:11:10.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.759+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3120741019 shutdown_connections 2023-03-03T11:11:10.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.759+0000 7fba8ff7e700 1 --2- 172.21.15.7:0/3120741019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba74004850 0x7fba74004cc0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:10.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.759+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3120741019 >> 172.21.15.7:0/3120741019 conn(0x7fba880736f0 msgr2=0x7fba88073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:11:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.759+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3120741019 shutdown_connections 2023-03-03T11:11:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.759+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3120741019 wait complete. 2023-03-03T11:11:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.760+0000 7fba8ff7e700 1 Processor -- start 2023-03-03T11:11:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.760+0000 7fba8ff7e700 1 -- start start 2023-03-03T11:11:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8ff7e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba74004850 0x7fba8814db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8ff7e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8806be70 0x7fba88148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8ff7e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8814bdf0 0x7fba88149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8ff7e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba881527a0 con 0x7fba8806be70 2023-03-03T11:11:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8ff7e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba881edcf0 con 0x7fba8814bdf0 2023-03-03T11:11:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8ff7e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8806b9d0 con 0x7fba74004850 2023-03-03T11:11:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8d519700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8806be70 0x7fba88148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8d519700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8806be70 0x7fba88148d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35268/0 (socket says 172.21.15.7:35268) 2023-03-03T11:11:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8d519700 1 -- 172.21.15.7:0/3993276032 learned_addr learned my addr 172.21.15.7:0/3993276032 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:11:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8e51b700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8814bdf0 0x7fba88149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8dd1a700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba74004850 0x7fba8814db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8d519700 1 -- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba74004850 msgr2=0x7fba8814db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8d519700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba74004850 0x7fba8814db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8d519700 1 -- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8814bdf0 msgr2=0x7fba88149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8d519700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8814bdf0 0x7fba88149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8d519700 1 -- 172.21.15.7:0/3993276032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba88153200 con 0x7fba8806be70 2023-03-03T11:11:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8e51b700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8814bdf0 0x7fba88149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:11:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.761+0000 7fba8dd1a700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba74004850 0x7fba8814db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:11:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.762+0000 7fba8d519700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8806be70 0x7fba88148d50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fba7800e4b0 tx=0x7fba780047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:11:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.762+0000 7fba7e7fc700 1 -- 172.21.15.7:0/3993276032 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7800e640 con 0x7fba8806be70 2023-03-03T11:11:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.762+0000 7fba7e7fc700 1 -- 172.21.15.7:0/3993276032 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba78003ab0 con 0x7fba8806be70 2023-03-03T11:11:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.762+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3993276032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba88071c70 con 0x7fba8806be70 2023-03-03T11:11:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.762+0000 7fba7e7fc700 1 -- 172.21.15.7:0/3993276032 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78003e60 con 0x7fba8806be70 2023-03-03T11:11:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.762+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3993276032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba88153390 con 0x7fba8806be70 2023-03-03T11:11:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.763+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3993276032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8806b500 con 0x7fba8806be70 2023-03-03T11:11:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.763+0000 7fba7e7fc700 1 -- 172.21.15.7:0/3993276032 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fba78019030 con 0x7fba8806be70 2023-03-03T11:11:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.764+0000 7fba7e7fc700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba6c0682f0 0x7fba6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.764+0000 7fba7e7fc700 1 -- 172.21.15.7:0/3993276032 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fba78087820 con 0x7fba8806be70 2023-03-03T11:11:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.764+0000 7fba8dd1a700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba6c0682f0 0x7fba6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.764+0000 7fba8dd1a700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba6c0682f0 0x7fba6c06a7a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fba84000c00 tx=0x7fba84009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:11:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:10.768+0000 7fba7e7fc700 1 -- 172.21.15.7:0/3993276032 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fba8806b500 con 0x7fba8806be70 2023-03-03T11:11:11.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.004+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3993276032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba8806b500 con 0x7fba8806be70 2023-03-03T11:11:11.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.005+0000 7fba7e7fc700 1 -- 172.21.15.7:0/3993276032 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fba8806b500 con 0x7fba8806be70 2023-03-03T11:11:11.008 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:11:11.008 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:11:11.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba6c0682f0 msgr2=0x7fba6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:11.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba6c0682f0 0x7fba6c06a7a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fba84000c00 tx=0x7fba84009040 comp rx=0 tx=0).stop 2023-03-03T11:11:11.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3993276032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8806be70 msgr2=0x7fba88148d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:11.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8806be70 0x7fba88148d50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fba7800e4b0 tx=0x7fba780047c0 comp rx=0 tx=0).stop 2023-03-03T11:11:11.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3993276032 shutdown_connections 2023-03-03T11:11:11.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba8814bdf0 0x7fba88149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:11.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba74004850 0x7fba8814db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:11.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba6c0682f0 0x7fba6c06a7a0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:11.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 --2- 172.21.15.7:0/3993276032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8806be70 0x7fba88148d50 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:11.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3993276032 >> 172.21.15.7:0/3993276032 conn(0x7fba880736f0 msgr2=0x7fba880c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:11:11.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3993276032 shutdown_connections 2023-03-03T11:11:11.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:11.007+0000 7fba8ff7e700 1 -- 172.21.15.7:0/3993276032 wait complete. 2023-03-03T11:11:11.012 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:11:11.022 INFO:tasks.mds_thrash.fs.[b]:waiting for 54 secs before thrashing 2023-03-03T11:11:21.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:11:22.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.006+0000 7efe6c40e700 1 Processor -- start 2023-03-03T11:11:22.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.006+0000 7efe6c40e700 1 -- start start 2023-03-03T11:11:22.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.007+0000 7efe6c40e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efe64072470 0x7efe6406b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:22.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.007+0000 7efe6c40e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe64070360 con 0x7efe64071990 2023-03-03T11:11:22.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.007+0000 7efe6c40e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6406a5e0 con 0x7efe6414b460 2023-03-03T11:11:22.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.007+0000 7efe6c40e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe641521a0 con 0x7efe64072470 2023-03-03T11:11:22.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.007+0000 7efe6a9ab700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efe64072470 0x7efe6406b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:22.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.007+0000 7efe6a9ab700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efe64072470 0x7efe6406b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35288/0 (socket says 172.21.15.7:35288) 2023-03-03T11:11:22.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.007+0000 7efe6a9ab700 1 -- 172.21.15.7:0/2353869961 learned_addr learned my addr 172.21.15.7:0/2353869961 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:11:22.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.008+0000 7efe6a9ab700 1 -- 172.21.15.7:0/2353869961 >> v1:172.21.15.143:6789/0 conn(0x7efe6414b460 legacy=0x7efe6406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:22.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.008+0000 7efe6a9ab700 1 -- 172.21.15.7:0/2353869961 >> v1:172.21.15.7:6789/0 conn(0x7efe64071990 legacy=0x7efe64071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:22.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.008+0000 7efe691a8700 1 -- 172.21.15.7:0/2353869961 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184966407 0 0) 0x7efe64070360 con 0x7efe64071990 2023-03-03T11:11:22.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.008+0000 7efe6a9ab700 1 -- 172.21.15.7:0/2353869961 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe640c38b0 con 0x7efe64072470 2023-03-03T11:11:22.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.008+0000 7efe6a9ab700 1 --2- 172.21.15.7:0/2353869961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 0x7efe6406b180 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009e50 tx=0x7efe5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267bc176f3c67c2c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:11:22.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.008+0000 7efe691a8700 1 -- 172.21.15.7:0/2353869961 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c00e070 con 0x7efe64072470 2023-03-03T11:11:22.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.008+0000 7efe691a8700 1 -- 172.21.15.7:0/2353869961 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5c00b9c0 con 0x7efe64072470 2023-03-03T11:11:22.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.008+0000 7efe691a8700 1 -- 172.21.15.7:0/2353869961 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5c00bd70 con 0x7efe64072470 2023-03-03T11:11:22.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.009+0000 7efe6c40e700 1 -- 172.21.15.7:0/2353869961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 msgr2=0x7efe6406b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:22.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.009+0000 7efe6c40e700 1 --2- 172.21.15.7:0/2353869961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 0x7efe6406b180 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009e50 tx=0x7efe5c00b330 comp rx=0 tx=0).stop 2023-03-03T11:11:22.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.009+0000 7efe6c40e700 1 -- 172.21.15.7:0/2353869961 shutdown_connections 2023-03-03T11:11:22.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.009+0000 7efe6c40e700 1 --2- 172.21.15.7:0/2353869961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 0x7efe6406b180 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:22.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.009+0000 7efe6c40e700 1 -- 172.21.15.7:0/2353869961 >> 172.21.15.7:0/2353869961 conn(0x7efe640a94d0 msgr2=0x7efe640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:11:22.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.009+0000 7efe6c40e700 1 -- 172.21.15.7:0/2353869961 shutdown_connections 2023-03-03T11:11:22.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.009+0000 7efe6c40e700 1 -- 172.21.15.7:0/2353869961 wait complete. 2023-03-03T11:11:22.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.010+0000 7efe6c40e700 1 Processor -- start 2023-03-03T11:11:22.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.010+0000 7efe6c40e700 1 -- start start 2023-03-03T11:11:22.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6c40e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe64071990 0x7efe64141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:22.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6c40e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 0x7efe640acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:22.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6c40e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe640ad060 0x7efe6414ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6c40e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe64139da0 con 0x7efe64072470 2023-03-03T11:11:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6c40e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe640ab880 con 0x7efe64071990 2023-03-03T11:11:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6c40e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6414fa90 con 0x7efe640ad060 2023-03-03T11:11:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe699a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 0x7efe640acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe699a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 0x7efe640acb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35292/0 (socket says 172.21.15.7:35292) 2023-03-03T11:11:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe699a9700 1 -- 172.21.15.7:0/2093703697 learned_addr learned my addr 172.21.15.7:0/2093703697 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:11:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6a9ab700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe640ad060 0x7efe6414ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:22.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6a1aa700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe64071990 0x7efe64141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:22.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe699a9700 1 -- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe640ad060 msgr2=0x7efe6414ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:22.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe699a9700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe640ad060 0x7efe6414ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:22.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe699a9700 1 -- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe64071990 msgr2=0x7efe64141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:22.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe699a9700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe64071990 0x7efe64141bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:22.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe699a9700 1 -- 172.21.15.7:0/2093703697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c009b30 con 0x7efe64072470 2023-03-03T11:11:22.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6a1aa700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe64071990 0x7efe64141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:11:22.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe699a9700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 0x7efe640acb20 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7efe6000c510 tx=0x7efe6000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:11:22.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe5affd700 1 -- 172.21.15.7:0/2093703697 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe600052d0 con 0x7efe64072470 2023-03-03T11:11:22.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe5affd700 1 -- 172.21.15.7:0/2093703697 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60005430 con 0x7efe64072470 2023-03-03T11:11:22.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6c40e700 1 -- 172.21.15.7:0/2093703697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe640c3b50 con 0x7efe64072470 2023-03-03T11:11:22.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe5affd700 1 -- 172.21.15.7:0/2093703697 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe600038c0 con 0x7efe64072470 2023-03-03T11:11:22.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.011+0000 7efe6c40e700 1 -- 172.21.15.7:0/2093703697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6406a2b0 con 0x7efe64072470 2023-03-03T11:11:22.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.012+0000 7efe6c40e700 1 -- 172.21.15.7:0/2093703697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6406b6c0 con 0x7efe64072470 2023-03-03T11:11:22.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.013+0000 7efe5affd700 1 -- 172.21.15.7:0/2093703697 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efe60019030 con 0x7efe64072470 2023-03-03T11:11:22.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.013+0000 7efe5affd700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe480682f0 0x7efe4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:11:22.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.014+0000 7efe5affd700 1 -- 172.21.15.7:0/2093703697 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efe60085520 con 0x7efe64072470 2023-03-03T11:11:22.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.014+0000 7efe6a1aa700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe480682f0 0x7efe4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:11:22.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.014+0000 7efe6a1aa700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe480682f0 0x7efe4806a7a0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7efe640c6280 tx=0x7efe54009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:11:22.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.018+0000 7efe5affd700 1 -- 172.21.15.7:0/2093703697 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efe60051860 con 0x7efe64072470 2023-03-03T11:11:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.255+0000 7efe6c40e700 1 -- 172.21.15.7:0/2093703697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe64150c30 con 0x7efe64072470 2023-03-03T11:11:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.256+0000 7efe5affd700 1 -- 172.21.15.7:0/2093703697 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efe64150c30 con 0x7efe64072470 2023-03-03T11:11:22.259 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:11:22.259 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:11:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 -- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe480682f0 msgr2=0x7efe4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe480682f0 0x7efe4806a7a0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7efe640c6280 tx=0x7efe54009040 comp rx=0 tx=0).stop 2023-03-03T11:11:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 -- 172.21.15.7:0/2093703697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 msgr2=0x7efe640acb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:11:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 0x7efe640acb20 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7efe6000c510 tx=0x7efe6000c8d0 comp rx=0 tx=0).stop 2023-03-03T11:11:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 -- 172.21.15.7:0/2093703697 shutdown_connections 2023-03-03T11:11:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe640ad060 0x7efe6414ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe64071990 0x7efe64141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe480682f0 0x7efe4806a7a0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 --2- 172.21.15.7:0/2093703697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe64072470 0x7efe640acb20 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:11:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 -- 172.21.15.7:0/2093703697 >> 172.21.15.7:0/2093703697 conn(0x7efe640a94d0 msgr2=0x7efe64142a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:11:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 -- 172.21.15.7:0/2093703697 shutdown_connections 2023-03-03T11:11:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:11:22.258+0000 7efe6c40e700 1 -- 172.21.15.7:0/2093703697 wait complete. 2023-03-03T11:11:22.263 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:11:22.272 INFO:tasks.mds_thrash.fs.[a]:waiting for 73 secs before thrashing 2023-03-03T11:11:40.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:11:40.028 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:05.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:12:05.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.159+0000 7f8248109700 1 Processor -- start 2023-03-03T11:12:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.160+0000 7f8248109700 1 -- start start 2023-03-03T11:12:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.160+0000 7f8248109700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82400ac220 0x7f82400736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.160+0000 7f8248109700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8240152b60 0x7f824015ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.160+0000 7f8248109700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82400ac900 con 0x7f8240073c30 2023-03-03T11:12:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.160+0000 7f8248109700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8240071e30 con 0x7f82400ac220 2023-03-03T11:12:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.160+0000 7f8248109700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82400acb50 con 0x7f8240152b60 2023-03-03T11:12:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f82456a4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8240073c30 0x7f8240152450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57532/0 (socket says 172.21.15.7:57532) 2023-03-03T11:12:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f82456a4700 1 -- 172.21.15.7:0/2395438329 learned_addr learned my addr 172.21.15.7:0/2395438329 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:12:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f82466a6700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8240152b60 0x7f824015ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8245ea5700 1 --2- 172.21.15.7:0/2395438329 >> v2:172.21.15.143:3300/0 conn(0x7f82400ac220 0x7f82400736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813199939 0 0) 0x7f82400ac900 con 0x7f8240073c30 2023-03-03T11:12:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822c0035b0 con 0x7f8240073c30 2023-03-03T11:12:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2268033988 0 0) 0x7f822c0035b0 con 0x7f8240073c30 2023-03-03T11:12:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8240152b60 msgr2=0x7f824015ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:12:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8240152b60 0x7f824015ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 >> v2:172.21.15.143:3300/0 conn(0x7f82400ac220 msgr2=0x7f82400736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:12:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 --2- 172.21.15.7:0/2395438329 >> v2:172.21.15.143:3300/0 conn(0x7f82400ac220 0x7f82400736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82401559c0 con 0x7f8240073c30 2023-03-03T11:12:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f823c001670 con 0x7f8240073c30 2023-03-03T11:12:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f82466a6700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8240152b60 0x7f824015ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:12:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8245ea5700 1 --2- 172.21.15.7:0/2395438329 >> v2:172.21.15.143:3300/0 conn(0x7f82400ac220 0x7f82400736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:12:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 >> v1:172.21.15.7:6789/0 conn(0x7f8240073c30 legacy=0x7f8240152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822c003a70 0x7f822c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8240152b60 0x7f822c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f822c004d90 0x7f822c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8240071e30 con 0x7f822c003a70 2023-03-03T11:12:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82400acb50 con 0x7f822c004d90 2023-03-03T11:12:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822c00ba10 con 0x7f8240152b60 2023-03-03T11:12:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f8245ea5700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822c003a70 0x7f822c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f82456a4700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8240152b60 0x7f822c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.161+0000 7f82466a6700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f822c004d90 0x7f822c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8245ea5700 1 -- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8240152b60 msgr2=0x7f822c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:12:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8245ea5700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8240152b60 0x7f822c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8245ea5700 1 -- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f822c004d90 msgr2=0x7f822c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:12:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8245ea5700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f822c004d90 0x7f822c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8245ea5700 1 -- 172.21.15.7:0/2395438329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8238009a70 con 0x7f822c003a70 2023-03-03T11:12:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8245ea5700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822c003a70 0x7f822c003e80 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f8234009d30 tx=0x7f822c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5fe9417418bdcc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234009e80 con 0x7f822c003a70 2023-03-03T11:12:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823400e9a0 con 0x7f822c003a70 2023-03-03T11:12:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8244ea3700 1 -- 172.21.15.7:0/2395438329 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823400ed30 con 0x7f822c003a70 2023-03-03T11:12:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8248109700 1 -- 172.21.15.7:0/2395438329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822c003a70 msgr2=0x7f822c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8248109700 1 --2- 172.21.15.7:0/2395438329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f822c003a70 0x7f822c003e80 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f8234009d30 tx=0x7f822c005880 comp rx=0 tx=0).stop 2023-03-03T11:12:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f82466a6700 1 -- 172.21.15.7:0/2395438329 reap_dead start 2023-03-03T11:12:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.162+0000 7f8248109700 1 -- 172.21.15.7:0/2395438329 shutdown_connections 2023-03-03T11:12:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.163+0000 7f8248109700 1 -- 172.21.15.7:0/2395438329 >> 172.21.15.7:0/2395438329 conn(0x7f82400a94d0 msgr2=0x7f82400a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:12:05.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.163+0000 7f8248109700 1 -- 172.21.15.7:0/2395438329 shutdown_connections 2023-03-03T11:12:05.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.163+0000 7f8248109700 1 -- 172.21.15.7:0/2395438329 wait complete. 2023-03-03T11:12:05.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.164+0000 7f8248109700 1 Processor -- start 2023-03-03T11:12:05.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.164+0000 7f8248109700 1 -- start start 2023-03-03T11:12:05.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.164+0000 7f8248109700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f822c003a70 0x7f82400c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:05.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.164+0000 7f8248109700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f824006d670 0x7f82400bfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:05.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.164+0000 7f8248109700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f824006dac0 0x7f82400bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:05.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.164+0000 7f8248109700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824006cd00 con 0x7f824006dac0 2023-03-03T11:12:05.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.164+0000 7f8248109700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824006c0c0 con 0x7f822c003a70 2023-03-03T11:12:05.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.164+0000 7f8248109700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824006c360 con 0x7f824006d670 2023-03-03T11:12:05.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82466a6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f824006dac0 0x7f82400bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:05.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82466a6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f824006dac0 0x7f82400bffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35334/0 (socket says 172.21.15.7:35334) 2023-03-03T11:12:05.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82466a6700 1 -- 172.21.15.7:0/3922000463 learned_addr learned my addr 172.21.15.7:0/3922000463 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:12:05.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f8245ea5700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f822c003a70 0x7f82400c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:05.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82456a4700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f824006d670 0x7f82400bfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:05.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82466a6700 1 -- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f824006d670 msgr2=0x7f82400bfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:05.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82466a6700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f824006d670 0x7f82400bfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:05.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82466a6700 1 -- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f822c003a70 msgr2=0x7f82400c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:05.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82466a6700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f822c003a70 0x7f82400c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:05.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82466a6700 1 -- 172.21.15.7:0/3922000463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8240155f80 con 0x7f824006dac0 2023-03-03T11:12:05.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82456a4700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f824006d670 0x7f82400bfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:12:05.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82466a6700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f824006dac0 0x7f82400bffa0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f82400c2490 tx=0x7f823800f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:05.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82327fc700 1 -- 172.21.15.7:0/3922000463 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823800e040 con 0x7f824006dac0 2023-03-03T11:12:05.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82327fc700 1 -- 172.21.15.7:0/3922000463 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823800f040 con 0x7f824006dac0 2023-03-03T11:12:05.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f8248109700 1 -- 172.21.15.7:0/3922000463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8240155ca0 con 0x7f824006dac0 2023-03-03T11:12:05.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f82327fc700 1 -- 172.21.15.7:0/3922000463 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238003a20 con 0x7f824006dac0 2023-03-03T11:12:05.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.165+0000 7f8248109700 1 -- 172.21.15.7:0/3922000463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8240156260 con 0x7f824006dac0 2023-03-03T11:12:05.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.166+0000 7f8248109700 1 -- 172.21.15.7:0/3922000463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8240074bb0 con 0x7f824006dac0 2023-03-03T11:12:05.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.167+0000 7f82327fc700 1 -- 172.21.15.7:0/3922000463 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8238015430 con 0x7f824006dac0 2023-03-03T11:12:05.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.167+0000 7f82327fc700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82240682f0 0x7f822406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:05.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.167+0000 7f82327fc700 1 -- 172.21.15.7:0/3922000463 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8238086990 con 0x7f824006dac0 2023-03-03T11:12:05.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.167+0000 7f8245ea5700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82240682f0 0x7f822406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:05.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.168+0000 7f8245ea5700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82240682f0 0x7f822406a7a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8234000c00 tx=0x7f823401f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:05.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.171+0000 7f82327fc700 1 -- 172.21.15.7:0/3922000463 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8238052cd0 con 0x7f824006dac0 2023-03-03T11:12:05.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.411+0000 7f8248109700 1 -- 172.21.15.7:0/3922000463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f824006ce40 con 0x7f824006dac0 2023-03-03T11:12:05.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.411+0000 7f82327fc700 1 -- 172.21.15.7:0/3922000463 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f824006ce40 con 0x7f824006dac0 2023-03-03T11:12:05.414 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:12:05.414 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:12:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 -- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82240682f0 msgr2=0x7f822406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82240682f0 0x7f822406a7a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8234000c00 tx=0x7f823401f040 comp rx=0 tx=0).stop 2023-03-03T11:12:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 -- 172.21.15.7:0/3922000463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f824006dac0 msgr2=0x7f82400bffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f824006dac0 0x7f82400bffa0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f82400c2490 tx=0x7f823800f490 comp rx=0 tx=0).stop 2023-03-03T11:12:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 -- 172.21.15.7:0/3922000463 shutdown_connections 2023-03-03T11:12:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82240682f0 0x7f822406a7a0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f824006dac0 0x7f82400bffa0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f822c003a70 0x7f82400c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 --2- 172.21.15.7:0/3922000463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f824006d670 0x7f82400bfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 -- 172.21.15.7:0/3922000463 >> 172.21.15.7:0/3922000463 conn(0x7f82400a94d0 msgr2=0x7f82400a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:12:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 -- 172.21.15.7:0/3922000463 shutdown_connections 2023-03-03T11:12:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:05.414+0000 7f8248109700 1 -- 172.21.15.7:0/3922000463 wait complete. 2023-03-03T11:12:05.418 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:12:05.428 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before thrashing 2023-03-03T11:12:07.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:12:07.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.564+0000 7f5f445c7700 1 Processor -- start 2023-03-03T11:12:07.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.564+0000 7f5f445c7700 1 -- start start 2023-03-03T11:12:07.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.565+0000 7f5f445c7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f3c14f9d0 0x7f5f3c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.565+0000 7f5f445c7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5f3c14fda0 0x7f5f3c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.565+0000 7f5f445c7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3c0c2f30 con 0x7f5f3c0abe20 2023-03-03T11:12:07.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.565+0000 7f5f445c7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3c0c31d0 con 0x7f5f3c14f9d0 2023-03-03T11:12:07.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.565+0000 7f5f445c7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3c1524a0 con 0x7f5f3c14fda0 2023-03-03T11:12:07.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.565+0000 7f5f42363700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f3c14f9d0 0x7f5f3c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.565+0000 7f5f42363700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f3c14f9d0 0x7f5f3c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35338/0 (socket says 172.21.15.7:35338) 2023-03-03T11:12:07.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.565+0000 7f5f42363700 1 -- 172.21.15.7:0/4157098806 learned_addr learned my addr 172.21.15.7:0/4157098806 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:12:07.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.566+0000 7f5f42363700 1 -- 172.21.15.7:0/4157098806 >> v2:172.21.15.143:3300/0 conn(0x7f5f3c14fda0 msgr2=0x7f5f3c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:12:07.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.566+0000 7f5f41b62700 1 --2- 172.21.15.7:0/4157098806 >> v2:172.21.15.143:3300/0 conn(0x7f5f3c14fda0 0x7f5f3c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.566+0000 7f5f42363700 1 --2- 172.21.15.7:0/4157098806 >> v2:172.21.15.143:3300/0 conn(0x7f5f3c14fda0 0x7f5f3c071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.566+0000 7f5f42363700 1 -- 172.21.15.7:0/4157098806 >> v1:172.21.15.143:6789/0 conn(0x7f5f3c0abe20 legacy=0x7f5f3c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:12:07.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.566+0000 7f5f42363700 1 -- 172.21.15.7:0/4157098806 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c0c34b0 con 0x7f5f3c14f9d0 2023-03-03T11:12:07.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.566+0000 7f5f41b62700 1 --2- 172.21.15.7:0/4157098806 >> v2:172.21.15.143:3300/0 conn(0x7f5f3c14fda0 0x7f5f3c071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:12:07.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.566+0000 7f5f42363700 1 --2- 172.21.15.7:0/4157098806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c14f9d0 0x7f5f3c156430 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0709b0 tx=0x7f5f38009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73b3bbd46b50d03 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:07.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.566+0000 7f5f41361700 1 -- 172.21.15.7:0/4157098806 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38004230 con 0x7f5f3c14f9d0 2023-03-03T11:12:07.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.566+0000 7f5f41361700 1 -- 172.21.15.7:0/4157098806 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f38004390 con 0x7f5f3c14f9d0 2023-03-03T11:12:07.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.566+0000 7f5f41361700 1 -- 172.21.15.7:0/4157098806 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f380036a0 con 0x7f5f3c14f9d0 2023-03-03T11:12:07.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.567+0000 7f5f445c7700 1 -- 172.21.15.7:0/4157098806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c14f9d0 msgr2=0x7f5f3c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:07.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.567+0000 7f5f445c7700 1 --2- 172.21.15.7:0/4157098806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c14f9d0 0x7f5f3c156430 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0709b0 tx=0x7f5f38009bd0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.567+0000 7f5f445c7700 1 -- 172.21.15.7:0/4157098806 shutdown_connections 2023-03-03T11:12:07.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.567+0000 7f5f445c7700 1 --2- 172.21.15.7:0/4157098806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c14f9d0 0x7f5f3c156430 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.567+0000 7f5f445c7700 1 --2- 172.21.15.7:0/4157098806 >> v2:172.21.15.143:3300/0 conn(0x7f5f3c14fda0 0x7f5f3c071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.567+0000 7f5f445c7700 1 -- 172.21.15.7:0/4157098806 >> 172.21.15.7:0/4157098806 conn(0x7f5f3c0c8900 msgr2=0x7f5f3c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:12:07.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.567+0000 7f5f445c7700 1 -- 172.21.15.7:0/4157098806 shutdown_connections 2023-03-03T11:12:07.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.567+0000 7f5f445c7700 1 -- 172.21.15.7:0/4157098806 wait complete. 2023-03-03T11:12:07.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.568+0000 7f5f445c7700 1 Processor -- start 2023-03-03T11:12:07.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.568+0000 7f5f445c7700 1 -- start start 2023-03-03T11:12:07.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f445c7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c0abe20 0x7f5f3c0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f445c7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f3c14f9d0 0x7f5f3c1ea960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f445c7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f3c14fda0 0x7f5f3c1eaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f445c7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c1db6e0 con 0x7f5f3c0abe20 2023-03-03T11:12:07.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f445c7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c1dc540 con 0x7f5f3c14f9d0 2023-03-03T11:12:07.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f445c7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c1db930 con 0x7f5f3c14fda0 2023-03-03T11:12:07.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42363700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c0abe20 0x7f5f3c0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42363700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c0abe20 0x7f5f3c0c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35342/0 (socket says 172.21.15.7:35342) 2023-03-03T11:12:07.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42363700 1 -- 172.21.15.7:0/932063354 learned_addr learned my addr 172.21.15.7:0/932063354 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:12:07.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f41b62700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f3c14f9d0 0x7f5f3c1ea960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42b64700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f3c14fda0 0x7f5f3c1eaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42363700 1 -- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f3c14fda0 msgr2=0x7f5f3c1eaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:07.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42363700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f3c14fda0 0x7f5f3c1eaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42363700 1 -- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f3c14f9d0 msgr2=0x7f5f3c1ea960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:07.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42363700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f3c14f9d0 0x7f5f3c1ea960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42363700 1 -- 172.21.15.7:0/932063354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f38009550 con 0x7f5f3c0abe20 2023-03-03T11:12:07.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42b64700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f3c14fda0 0x7f5f3c1eaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:12:07.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f42363700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c0abe20 0x7f5f3c0c1a50 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009fd0 tx=0x7f5f38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:07.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f2effd700 1 -- 172.21.15.7:0/932063354 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38003d70 con 0x7f5f3c0abe20 2023-03-03T11:12:07.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.569+0000 7f5f2effd700 1 -- 172.21.15.7:0/932063354 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3801f070 con 0x7f5f3c0abe20 2023-03-03T11:12:07.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.570+0000 7f5f445c7700 1 -- 172.21.15.7:0/932063354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3c0ac350 con 0x7f5f3c0abe20 2023-03-03T11:12:07.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.570+0000 7f5f2effd700 1 -- 172.21.15.7:0/932063354 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38011870 con 0x7f5f3c0abe20 2023-03-03T11:12:07.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.570+0000 7f5f445c7700 1 -- 172.21.15.7:0/932063354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f3c0ab630 con 0x7f5f3c0abe20 2023-03-03T11:12:07.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.570+0000 7f5f445c7700 1 -- 172.21.15.7:0/932063354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3c0c3e10 con 0x7f5f3c0abe20 2023-03-03T11:12:07.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.571+0000 7f5f2effd700 1 -- 172.21.15.7:0/932063354 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5f38017020 con 0x7f5f3c0abe20 2023-03-03T11:12:07.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.571+0000 7f5f2effd700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f200682f0 0x7f5f2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:07.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.572+0000 7f5f2effd700 1 -- 172.21.15.7:0/932063354 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5f380879c0 con 0x7f5f3c0abe20 2023-03-03T11:12:07.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.572+0000 7f5f41b62700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f200682f0 0x7f5f2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:07.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.572+0000 7f5f41b62700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f200682f0 0x7f5f2006a7a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c1edb80 tx=0x7f5f30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:07.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.576+0000 7f5f2effd700 1 -- 172.21.15.7:0/932063354 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5f38053c10 con 0x7f5f3c0abe20 2023-03-03T11:12:07.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.812+0000 7f5f445c7700 1 -- 172.21.15.7:0/932063354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f3c074bb0 con 0x7f5f3c0abe20 2023-03-03T11:12:07.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.812+0000 7f5f2effd700 1 -- 172.21.15.7:0/932063354 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5f38057230 con 0x7f5f3c0abe20 2023-03-03T11:12:07.815 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:12:07.815 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:12:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.814+0000 7f5f445c7700 1 -- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f200682f0 msgr2=0x7f5f2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.814+0000 7f5f445c7700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f200682f0 0x7f5f2006a7a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c1edb80 tx=0x7f5f30008040 comp rx=0 tx=0).stop 2023-03-03T11:12:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.814+0000 7f5f445c7700 1 -- 172.21.15.7:0/932063354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c0abe20 msgr2=0x7f5f3c0c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.814+0000 7f5f445c7700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c0abe20 0x7f5f3c0c1a50 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009fd0 tx=0x7f5f38004750 comp rx=0 tx=0).stop 2023-03-03T11:12:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.814+0000 7f5f445c7700 1 -- 172.21.15.7:0/932063354 shutdown_connections 2023-03-03T11:12:07.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.814+0000 7f5f445c7700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f3c14fda0 0x7f5f3c1eaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.814+0000 7f5f445c7700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f200682f0 0x7f5f2006a7a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.814+0000 7f5f445c7700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f3c0abe20 0x7f5f3c0c1a50 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.814+0000 7f5f445c7700 1 --2- 172.21.15.7:0/932063354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f3c14f9d0 0x7f5f3c1ea960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.815+0000 7f5f445c7700 1 -- 172.21.15.7:0/932063354 >> 172.21.15.7:0/932063354 conn(0x7f5f3c0c8900 msgr2=0x7f5f3c0c7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:12:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.815+0000 7f5f445c7700 1 -- 172.21.15.7:0/932063354 shutdown_connections 2023-03-03T11:12:07.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:07.815+0000 7f5f445c7700 1 -- 172.21.15.7:0/932063354 wait complete. 2023-03-03T11:12:07.819 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:12:07.828 INFO:tasks.mds_thrash.fs.[b]:waiting for 88 secs before thrashing 2023-03-03T11:12:10.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:10.158 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:35.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:12:35.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.409+0000 7f591a0d1700 1 Processor -- start 2023-03-03T11:12:35.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.409+0000 7f591a0d1700 1 -- start start 2023-03-03T11:12:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.410+0000 7f591a0d1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140a9af0 0x7f591414de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.410+0000 7f591a0d1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59140a65b0 con 0x7f591406a2b0 2023-03-03T11:12:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.410+0000 7f591a0d1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59141541f0 con 0x7f59140c65c0 2023-03-03T11:12:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.410+0000 7f591a0d1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5914154e00 con 0x7f59140a9af0 2023-03-03T11:12:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.410+0000 7f5913fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f591406a2b0 0x7f59140c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57586/0 (socket says 172.21.15.7:57586) 2023-03-03T11:12:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.410+0000 7f5913fff700 1 -- 172.21.15.7:0/4221416167 learned_addr learned my addr 172.21.15.7:0/4221416167 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:12:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.410+0000 7f5912ffd700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140a9af0 0x7f591414de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.410+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612620790 0 0) 0x7f59140a65b0 con 0x7f591406a2b0 2023-03-03T11:12:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.410+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59000035b0 con 0x7f591406a2b0 2023-03-03T11:12:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595618847 0 0) 0x7f59141541f0 con 0x7f59140c65c0 2023-03-03T11:12:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59140a65b0 con 0x7f59140c65c0 2023-03-03T11:12:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3886071712 0 0) 0x7f59000035b0 con 0x7f591406a2b0 2023-03-03T11:12:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140a9af0 msgr2=0x7f591414de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:12:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140a9af0 0x7f591414de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 >> v1:172.21.15.143:6789/0 conn(0x7f59140c65c0 legacy=0x7f5914066840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59141544d0 con 0x7f591406a2b0 2023-03-03T11:12:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f590c0019f0 con 0x7f591406a2b0 2023-03-03T11:12:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f5912ffd700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140a9af0 0x7f591414de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:12:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 >> v1:172.21.15.7:6789/0 conn(0x7f591406a2b0 legacy=0x7f59140c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5900003c30 0x7f5900004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5900004850 0x7f5900004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140a9af0 0x7f5900005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5914154e00 con 0x7f5900003c30 2023-03-03T11:12:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590000baf0 con 0x7f5900004850 2023-03-03T11:12:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590000bcb0 con 0x7f59140a9af0 2023-03-03T11:12:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59137fe700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5900003c30 0x7f5900004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f5912ffd700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5900004850 0x7f5900004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f5913fff700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140a9af0 0x7f5900005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59137fe700 1 -- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140a9af0 msgr2=0x7f5900005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:12:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59137fe700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140a9af0 0x7f5900005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59137fe700 1 -- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5900004850 msgr2=0x7f5900004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:12:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59137fe700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5900004850 0x7f5900004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59137fe700 1 -- 172.21.15.7:0/4221416167 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58fc0094a0 con 0x7f5900003c30 2023-03-03T11:12:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f5913fff700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140a9af0 0x7f5900005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:12:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.411+0000 7f59137fe700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5900003c30 0x7f5900004020 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5904001940 tx=0x7f5900005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d7be83d8dd0053 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.412+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904001430 con 0x7f5900003c30 2023-03-03T11:12:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.412+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5904001d70 con 0x7f5900003c30 2023-03-03T11:12:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.412+0000 7f59127fc700 1 -- 172.21.15.7:0/4221416167 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904005510 con 0x7f5900003c30 2023-03-03T11:12:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.412+0000 7f591a0d1700 1 -- 172.21.15.7:0/4221416167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5900003c30 msgr2=0x7f5900004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.412+0000 7f591a0d1700 1 --2- 172.21.15.7:0/4221416167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5900003c30 0x7f5900004020 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5904001940 tx=0x7f5900005880 comp rx=0 tx=0).stop 2023-03-03T11:12:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.412+0000 7f5913fff700 1 -- 172.21.15.7:0/4221416167 reap_dead start 2023-03-03T11:12:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.412+0000 7f591a0d1700 1 -- 172.21.15.7:0/4221416167 shutdown_connections 2023-03-03T11:12:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.412+0000 7f591a0d1700 1 -- 172.21.15.7:0/4221416167 >> 172.21.15.7:0/4221416167 conn(0x7f59140a9430 msgr2=0x7f59140c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:12:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.412+0000 7f591a0d1700 1 -- 172.21.15.7:0/4221416167 shutdown_connections 2023-03-03T11:12:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.412+0000 7f591a0d1700 1 -- 172.21.15.7:0/4221416167 wait complete. 2023-03-03T11:12:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.413+0000 7f591a0d1700 1 Processor -- start 2023-03-03T11:12:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f591a0d1700 1 -- start start 2023-03-03T11:12:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f591a0d1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5900003c30 0x7f59141523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f591a0d1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140c3af0 0x7f5914149ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f591a0d1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f591406b680 0x7f591414a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f591a0d1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591414aa70 con 0x7f591406b680 2023-03-03T11:12:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f591a0d1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59140c1a50 con 0x7f5900003c30 2023-03-03T11:12:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f591a0d1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59140724f0 con 0x7f59140c3af0 2023-03-03T11:12:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f5913fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f591406b680 0x7f591414a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f5913fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f591406b680 0x7f591414a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35386/0 (socket says 172.21.15.7:35386) 2023-03-03T11:12:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f5913fff700 1 -- 172.21.15.7:0/4070924704 learned_addr learned my addr 172.21.15.7:0/4070924704 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:12:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f5912ffd700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140c3af0 0x7f5914149ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.414+0000 7f59137fe700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5900003c30 0x7f59141523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f5913fff700 1 -- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140c3af0 msgr2=0x7f5914149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f5913fff700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140c3af0 0x7f5914149ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f5913fff700 1 -- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5900003c30 msgr2=0x7f59141523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f5913fff700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5900003c30 0x7f59141523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f5913fff700 1 -- 172.21.15.7:0/4070924704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5914154a90 con 0x7f591406b680 2023-03-03T11:12:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f5913fff700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f591406b680 0x7f591414a530 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f59140c4700 tx=0x7f590c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f590bfff700 1 -- 172.21.15.7:0/4070924704 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c00ebe0 con 0x7f591406b680 2023-03-03T11:12:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f590bfff700 1 -- 172.21.15.7:0/4070924704 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590c00ed40 con 0x7f591406b680 2023-03-03T11:12:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f591a0d1700 1 -- 172.21.15.7:0/4070924704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59141547b0 con 0x7f591406b680 2023-03-03T11:12:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f590bfff700 1 -- 172.21.15.7:0/4070924704 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c013660 con 0x7f591406b680 2023-03-03T11:12:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.415+0000 7f591a0d1700 1 -- 172.21.15.7:0/4070924704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5914153e80 con 0x7f591406b680 2023-03-03T11:12:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.416+0000 7f591a0d1700 1 -- 172.21.15.7:0/4070924704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5914074bb0 con 0x7f591406b680 2023-03-03T11:12:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.417+0000 7f590bfff700 1 -- 172.21.15.7:0/4070924704 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f590c0137c0 con 0x7f591406b680 2023-03-03T11:12:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.417+0000 7f590bfff700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f4068320 0x7f58f406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:12:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.417+0000 7f590bfff700 1 -- 172.21.15.7:0/4070924704 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f590c0866f0 con 0x7f591406b680 2023-03-03T11:12:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.417+0000 7f59137fe700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f4068320 0x7f58f406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:12:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.418+0000 7f59137fe700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f4068320 0x7f58f406a7d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5904000c00 tx=0x7f590400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:12:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.422+0000 7f590bfff700 1 -- 172.21.15.7:0/4070924704 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f590c052a90 con 0x7f591406b680 2023-03-03T11:12:35.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.658+0000 7f591a0d1700 1 -- 172.21.15.7:0/4070924704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f591406e0b0 con 0x7f591406b680 2023-03-03T11:12:35.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.658+0000 7f590bfff700 1 -- 172.21.15.7:0/4070924704 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f590c0560b0 con 0x7f591406b680 2023-03-03T11:12:35.661 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:12:35.661 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:12:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.660+0000 7f591a0d1700 1 -- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f4068320 msgr2=0x7f58f406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.660+0000 7f591a0d1700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f4068320 0x7f58f406a7d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5904000c00 tx=0x7f590400a000 comp rx=0 tx=0).stop 2023-03-03T11:12:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.660+0000 7f591a0d1700 1 -- 172.21.15.7:0/4070924704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f591406b680 msgr2=0x7f591414a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:12:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.660+0000 7f591a0d1700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f591406b680 0x7f591414a530 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f59140c4700 tx=0x7f590c00e440 comp rx=0 tx=0).stop 2023-03-03T11:12:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.661+0000 7f591a0d1700 1 -- 172.21.15.7:0/4070924704 shutdown_connections 2023-03-03T11:12:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.661+0000 7f591a0d1700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f4068320 0x7f58f406a7d0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.661+0000 7f591a0d1700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f591406b680 0x7f591414a530 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:35.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.661+0000 7f591a0d1700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5900003c30 0x7f59141523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:35.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.661+0000 7f591a0d1700 1 --2- 172.21.15.7:0/4070924704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59140c3af0 0x7f5914149ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:12:35.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.661+0000 7f591a0d1700 1 -- 172.21.15.7:0/4070924704 >> 172.21.15.7:0/4070924704 conn(0x7f59140a9430 msgr2=0x7f59140a7070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:12:35.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.661+0000 7f591a0d1700 1 -- 172.21.15.7:0/4070924704 shutdown_connections 2023-03-03T11:12:35.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:12:35.661+0000 7f591a0d1700 1 -- 172.21.15.7:0/4070924704 wait complete. 2023-03-03T11:12:35.664 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:12:35.674 INFO:tasks.mds_thrash.fs.[a]:waiting for 105 secs before thrashing 2023-03-03T11:12:40.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:12:40.296 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:10.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:10.419 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:35.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:13:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d121be700 1 Processor -- start 2023-03-03T11:13:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d121be700 1 -- start start 2023-03-03T11:13:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d121be700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5d0c0c2e30 0x7f5d0c158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d121be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d0c06c6c0 0x7f5d0c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d121be700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c157ff0 con 0x7f5d0c151cd0 2023-03-03T11:13:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d121be700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0c06d640 con 0x7f5d0c0c2e30 2023-03-03T11:13:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d121be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c06cda0 con 0x7f5d0c06c6c0 2023-03-03T11:13:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5d0c0c2e30 0x7f5d0c158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5d0c0c2e30 0x7f5d0c158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35444/0 (socket says 172.21.15.7:35444) 2023-03-03T11:13:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0affd700 1 -- 172.21.15.7:0/2816463415 learned_addr learned my addr 172.21.15.7:0/2816463415 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:13:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0bfff700 1 --2- 172.21.15.7:0/2816463415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d0c06c6c0 0x7f5d0c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0affd700 1 -- 172.21.15.7:0/2816463415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d0c06c6c0 msgr2=0x7f5d0c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:13:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0affd700 1 --2- 172.21.15.7:0/2816463415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d0c06c6c0 0x7f5d0c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0affd700 1 -- 172.21.15.7:0/2816463415 >> v1:172.21.15.143:6789/0 conn(0x7f5d0c151cd0 legacy=0x7f5d0c154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0affd700 1 -- 172.21.15.7:0/2816463415 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c06d080 con 0x7f5d0c0c2e30 2023-03-03T11:13:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0affd700 1 --2- 172.21.15.7:0/2816463415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c0c2e30 0x7f5d0c158480 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5d000098f0 tx=0x7f5d00009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a9cfbb286314ed server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:13:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0a7fc700 1 -- 172.21.15.7:0/2816463415 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00011830 con 0x7f5d0c0c2e30 2023-03-03T11:13:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0a7fc700 1 -- 172.21.15.7:0/2816463415 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d00011990 con 0x7f5d0c0c2e30 2023-03-03T11:13:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.968+0000 7f5d0a7fc700 1 -- 172.21.15.7:0/2816463415 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00011d20 con 0x7f5d0c0c2e30 2023-03-03T11:13:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.969+0000 7f5d121be700 1 -- 172.21.15.7:0/2816463415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c0c2e30 msgr2=0x7f5d0c158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.969+0000 7f5d121be700 1 --2- 172.21.15.7:0/2816463415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c0c2e30 0x7f5d0c158480 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5d000098f0 tx=0x7f5d00009c00 comp rx=0 tx=0).stop 2023-03-03T11:13:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.969+0000 7f5d121be700 1 -- 172.21.15.7:0/2816463415 shutdown_connections 2023-03-03T11:13:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.969+0000 7f5d121be700 1 --2- 172.21.15.7:0/2816463415 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d0c06c6c0 0x7f5d0c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.969+0000 7f5d121be700 1 --2- 172.21.15.7:0/2816463415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c0c2e30 0x7f5d0c158480 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.969+0000 7f5d121be700 1 -- 172.21.15.7:0/2816463415 >> 172.21.15.7:0/2816463415 conn(0x7f5d0c0a94d0 msgr2=0x7f5d0c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:13:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.969+0000 7f5d121be700 1 -- 172.21.15.7:0/2816463415 shutdown_connections 2023-03-03T11:13:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.969+0000 7f5d121be700 1 -- 172.21.15.7:0/2816463415 wait complete. 2023-03-03T11:13:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.970+0000 7f5d121be700 1 Processor -- start 2023-03-03T11:13:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.970+0000 7f5d121be700 1 -- start start 2023-03-03T11:13:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d121be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d0c06c6c0 0x7f5d0c1ea4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d121be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d0c0c2e30 0x7f5d0c0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d121be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c151cd0 0x7f5d0c155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d121be700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c0aa690 con 0x7f5d0c151cd0 2023-03-03T11:13:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d121be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c155a30 con 0x7f5d0c0c2e30 2023-03-03T11:13:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d121be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c1eab00 con 0x7f5d0c06c6c0 2023-03-03T11:13:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d0bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c151cd0 0x7f5d0c155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d0bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c151cd0 0x7f5d0c155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35452/0 (socket says 172.21.15.7:35452) 2023-03-03T11:13:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d0bfff700 1 -- 172.21.15.7:0/3929766314 learned_addr learned my addr 172.21.15.7:0/3929766314 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:13:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d0b7fe700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d0c06c6c0 0x7f5d0c1ea4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.971+0000 7f5d0affd700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d0c0c2e30 0x7f5d0c0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5d0bfff700 1 -- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d0c06c6c0 msgr2=0x7f5d0c1ea4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5d0bfff700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d0c06c6c0 0x7f5d0c1ea4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5d0bfff700 1 -- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d0c0c2e30 msgr2=0x7f5d0c0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5d0bfff700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d0c0c2e30 0x7f5d0c0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5d0bfff700 1 -- 172.21.15.7:0/3929766314 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d00009550 con 0x7f5d0c151cd0 2023-03-03T11:13:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5d0bfff700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c151cd0 0x7f5d0c155ec0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400efc0 tx=0x7f5d0400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:13:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5cf3fff700 1 -- 172.21.15.7:0/3929766314 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0400cad0 con 0x7f5d0c151cd0 2023-03-03T11:13:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5cf3fff700 1 -- 172.21.15.7:0/3929766314 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0400f040 con 0x7f5d0c151cd0 2023-03-03T11:13:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5d121be700 1 -- 172.21.15.7:0/3929766314 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0c157c80 con 0x7f5d0c151cd0 2023-03-03T11:13:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5cf3fff700 1 -- 172.21.15.7:0/3929766314 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0400ce70 con 0x7f5d0c151cd0 2023-03-03T11:13:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.972+0000 7f5d121be700 1 -- 172.21.15.7:0/3929766314 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0c06d360 con 0x7f5d0c151cd0 2023-03-03T11:13:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.973+0000 7f5d121be700 1 -- 172.21.15.7:0/3929766314 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d0c1589c0 con 0x7f5d0c151cd0 2023-03-03T11:13:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.973+0000 7f5cf3fff700 1 -- 172.21.15.7:0/3929766314 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5d04014480 con 0x7f5d0c151cd0 2023-03-03T11:13:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.974+0000 7f5cf3fff700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cf80682f0 0x7f5cf806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:13:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.974+0000 7f5cf3fff700 1 -- 172.21.15.7:0/3929766314 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5d04086cf0 con 0x7f5d0c151cd0 2023-03-03T11:13:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.974+0000 7f5d0b7fe700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cf80682f0 0x7f5cf806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:13:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.974+0000 7f5d0b7fe700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cf80682f0 0x7f5cf806a7a0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc007fd0 tx=0x7f5cfc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:13:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:35.978+0000 7f5cf3fff700 1 -- 172.21.15.7:0/3929766314 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5d04053030 con 0x7f5d0c151cd0 2023-03-03T11:13:36.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.218+0000 7f5d121be700 1 -- 172.21.15.7:0/3929766314 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d0c155ba0 con 0x7f5d0c151cd0 2023-03-03T11:13:36.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.218+0000 7f5cf3fff700 1 -- 172.21.15.7:0/3929766314 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5d04056650 con 0x7f5d0c151cd0 2023-03-03T11:13:36.221 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:13:36.222 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:13:36.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 -- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cf80682f0 msgr2=0x7f5cf806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:36.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cf80682f0 0x7f5cf806a7a0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc007fd0 tx=0x7f5cfc009040 comp rx=0 tx=0).stop 2023-03-03T11:13:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 -- 172.21.15.7:0/3929766314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c151cd0 msgr2=0x7f5d0c155ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:13:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c151cd0 0x7f5d0c155ec0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400efc0 tx=0x7f5d0400c0c0 comp rx=0 tx=0).stop 2023-03-03T11:13:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 -- 172.21.15.7:0/3929766314 shutdown_connections 2023-03-03T11:13:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cf80682f0 0x7f5cf806a7a0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d0c151cd0 0x7f5d0c155ec0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d0c06c6c0 0x7f5d0c1ea4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 --2- 172.21.15.7:0/3929766314 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d0c0c2e30 0x7f5d0c0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:13:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 -- 172.21.15.7:0/3929766314 >> 172.21.15.7:0/3929766314 conn(0x7f5d0c0a94d0 msgr2=0x7f5d0c071260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:13:36.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 -- 172.21.15.7:0/3929766314 shutdown_connections 2023-03-03T11:13:36.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:13:36.221+0000 7f5d121be700 1 -- 172.21.15.7:0/3929766314 wait complete. 2023-03-03T11:13:36.225 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:13:36.235 INFO:tasks.mds_thrash.fs.[b]:waiting for 88 secs before thrashing 2023-03-03T11:13:40.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:13:40.547 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:10.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:10.621 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:20.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:14:20.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.811+0000 7f70bf67a700 1 Processor -- start 2023-03-03T11:14:20.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.811+0000 7f70bf67a700 1 -- start start 2023-03-03T11:14:20.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.812+0000 7f70bf67a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f70b80aad20 0x7f70b8157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:20.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.812+0000 7f70bf67a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aa030 0x7f70b8158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:20.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.812+0000 7f70bf67a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b80a9c20 con 0x7f70b814b0d0 2023-03-03T11:14:20.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.812+0000 7f70bf67a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b8154f10 con 0x7f70b80aad20 2023-03-03T11:14:20.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.812+0000 7f70bf67a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b80ac3c0 con 0x7f70b80aa030 2023-03-03T11:14:20.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70bcc15700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f70b80aad20 0x7f70b8157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70bdc17700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aa030 0x7f70b8158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70bcc15700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f70b80aad20 0x7f70b8157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:59506/0 (socket says 172.21.15.7:59506) 2023-03-03T11:14:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70bcc15700 1 -- 172.21.15.7:0/1493057065 learned_addr learned my addr 172.21.15.7:0/1493057065 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:14:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70bcc15700 1 -- 172.21.15.7:0/1493057065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aa030 msgr2=0x7f70b8158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:14:20.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70bcc15700 1 --2- 172.21.15.7:0/1493057065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aa030 0x7f70b8158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70bcc15700 1 -- 172.21.15.7:0/1493057065 >> v1:172.21.15.143:6789/0 conn(0x7f70b814b0d0 legacy=0x7f70b8071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70affff700 1 -- 172.21.15.7:0/1493057065 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910138456 0 0) 0x7f70b80a9c20 con 0x7f70b814b0d0 2023-03-03T11:14:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70bcc15700 1 -- 172.21.15.7:0/1493057065 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b806fc70 con 0x7f70b80aad20 2023-03-03T11:14:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70bdc17700 1 --2- 172.21.15.7:0/1493057065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aa030 0x7f70b8158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:14:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70bcc15700 1 --2- 172.21.15.7:0/1493057065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b80aad20 0x7f70b8157df0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f70a80098f0 tx=0x7f70a8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9511f1f6c603bada server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:14:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70affff700 1 -- 172.21.15.7:0/1493057065 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a8011830 con 0x7f70b80aad20 2023-03-03T11:14:20.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70affff700 1 -- 172.21.15.7:0/1493057065 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a8011990 con 0x7f70b80aad20 2023-03-03T11:14:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.813+0000 7f70affff700 1 -- 172.21.15.7:0/1493057065 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a8011d20 con 0x7f70b80aad20 2023-03-03T11:14:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.814+0000 7f70bf67a700 1 -- 172.21.15.7:0/1493057065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b80aad20 msgr2=0x7f70b8157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.814+0000 7f70bf67a700 1 --2- 172.21.15.7:0/1493057065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b80aad20 0x7f70b8157df0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f70a80098f0 tx=0x7f70a8009c00 comp rx=0 tx=0).stop 2023-03-03T11:14:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.814+0000 7f70bf67a700 1 -- 172.21.15.7:0/1493057065 shutdown_connections 2023-03-03T11:14:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.814+0000 7f70bf67a700 1 --2- 172.21.15.7:0/1493057065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aa030 0x7f70b8158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.814+0000 7f70bf67a700 1 --2- 172.21.15.7:0/1493057065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b80aad20 0x7f70b8157df0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:20.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.814+0000 7f70bf67a700 1 -- 172.21.15.7:0/1493057065 >> 172.21.15.7:0/1493057065 conn(0x7f70b80c8900 msgr2=0x7f70b80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:14:20.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.814+0000 7f70bf67a700 1 -- 172.21.15.7:0/1493057065 shutdown_connections 2023-03-03T11:14:20.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.814+0000 7f70bf67a700 1 -- 172.21.15.7:0/1493057065 wait complete. 2023-03-03T11:14:20.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.815+0000 7f70bf67a700 1 Processor -- start 2023-03-03T11:14:20.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.815+0000 7f70bf67a700 1 -- start start 2023-03-03T11:14:20.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bf67a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70b80aa030 0x7f70b81520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:20.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bf67a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aad20 0x7f70b8152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:20.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bf67a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b814b0d0 0x7f70b8143010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:20.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bf67a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8155880 con 0x7f70b80aa030 2023-03-03T11:14:20.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bf67a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b81526d0 con 0x7f70b814b0d0 2023-03-03T11:14:20.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bf67a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8144920 con 0x7f70b80aad20 2023-03-03T11:14:20.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bd416700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70b80aa030 0x7f70b81520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:20.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bd416700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70b80aa030 0x7f70b81520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35504/0 (socket says 172.21.15.7:35504) 2023-03-03T11:14:20.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bd416700 1 -- 172.21.15.7:0/785245472 learned_addr learned my addr 172.21.15.7:0/785245472 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:14:20.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bdc17700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b814b0d0 0x7f70b8143010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:20.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bcc15700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aad20 0x7f70b8152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:20.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bd416700 1 -- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aad20 msgr2=0x7f70b8152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:20.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bd416700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aad20 0x7f70b8152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:20.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bd416700 1 -- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b814b0d0 msgr2=0x7f70b8143010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:20.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bd416700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b814b0d0 0x7f70b8143010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:20.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.816+0000 7f70bd416700 1 -- 172.21.15.7:0/785245472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b0009a70 con 0x7f70b80aa030 2023-03-03T11:14:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.817+0000 7f70bd416700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70b80aa030 0x7f70b81520d0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f70b40011e0 tx=0x7f70b400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:14:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.817+0000 7f70adffb700 1 -- 172.21.15.7:0/785245472 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b40054e0 con 0x7f70b80aa030 2023-03-03T11:14:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.817+0000 7f70adffb700 1 -- 172.21.15.7:0/785245472 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b4003680 con 0x7f70b80aa030 2023-03-03T11:14:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.817+0000 7f70adffb700 1 -- 172.21.15.7:0/785245472 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b4003a30 con 0x7f70b80aa030 2023-03-03T11:14:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.817+0000 7f70bf67a700 1 -- 172.21.15.7:0/785245472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a8009550 con 0x7f70b80aa030 2023-03-03T11:14:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.817+0000 7f70bf67a700 1 -- 172.21.15.7:0/785245472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b806f6b0 con 0x7f70b80aa030 2023-03-03T11:14:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.818+0000 7f70bf67a700 1 -- 172.21.15.7:0/785245472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b8155160 con 0x7f70b80aa030 2023-03-03T11:14:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.818+0000 7f70adffb700 1 -- 172.21.15.7:0/785245472 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f70b4019030 con 0x7f70b80aa030 2023-03-03T11:14:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.819+0000 7f70adffb700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f709c0682f0 0x7f709c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:14:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.819+0000 7f70adffb700 1 -- 172.21.15.7:0/785245472 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f70b4085660 con 0x7f70b80aa030 2023-03-03T11:14:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.819+0000 7f70bcc15700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f709c0682f0 0x7f709c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:14:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.819+0000 7f70bcc15700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f709c0682f0 0x7f709c06a7a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f70a80098f0 tx=0x7f70a8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:14:20.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:20.823+0000 7f70adffb700 1 -- 172.21.15.7:0/785245472 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70b4051910 con 0x7f70b80aa030 2023-03-03T11:14:21.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.059+0000 7f70bf67a700 1 -- 172.21.15.7:0/785245472 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70b806f990 con 0x7f70b80aa030 2023-03-03T11:14:21.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.059+0000 7f70adffb700 1 -- 172.21.15.7:0/785245472 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f70b4017090 con 0x7f70b80aa030 2023-03-03T11:14:21.062 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:14:21.062 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:14:21.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 -- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f709c0682f0 msgr2=0x7f709c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:21.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f709c0682f0 0x7f709c06a7a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f70a80098f0 tx=0x7f70a8008030 comp rx=0 tx=0).stop 2023-03-03T11:14:21.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 -- 172.21.15.7:0/785245472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70b80aa030 msgr2=0x7f70b81520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:14:21.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70b80aa030 0x7f70b81520d0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f70b40011e0 tx=0x7f70b400cae0 comp rx=0 tx=0).stop 2023-03-03T11:14:21.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 -- 172.21.15.7:0/785245472 shutdown_connections 2023-03-03T11:14:21.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b814b0d0 0x7f70b8143010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:21.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f709c0682f0 0x7f709c06a7a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70b80aa030 0x7f70b81520d0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 --2- 172.21.15.7:0/785245472 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70b80aad20 0x7f70b8152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:14:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 -- 172.21.15.7:0/785245472 >> 172.21.15.7:0/785245472 conn(0x7f70b80c8900 msgr2=0x7f70b80c3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:14:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 -- 172.21.15.7:0/785245472 shutdown_connections 2023-03-03T11:14:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:14:21.061+0000 7f70bf67a700 1 -- 172.21.15.7:0/785245472 wait complete. 2023-03-03T11:14:21.065 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:14:21.075 INFO:tasks.mds_thrash.fs.[a]:waiting for 115 secs before thrashing 2023-03-03T11:14:40.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:14:40.697 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:04.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:15:04.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.375+0000 7f11d47e7700 1 Processor -- start 2023-03-03T11:15:04.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.375+0000 7f11d47e7700 1 -- start start 2023-03-03T11:15:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.376+0000 7f11d47e7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 0x7f11cc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:15:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.376+0000 7f11d47e7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f11cc14fda0 0x7f11cc071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:15:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.376+0000 7f11d47e7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11cc0c2f30 con 0x7f11cc0abe20 2023-03-03T11:15:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.376+0000 7f11d47e7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11cc0c31d0 con 0x7f11cc14fda0 2023-03-03T11:15:04.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.376+0000 7f11d47e7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1524a0 con 0x7f11cc14f9d0 2023-03-03T11:15:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d1d82700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f11cc14fda0 0x7f11cc071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:15:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d2583700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 0x7f11cc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:15:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d1d82700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f11cc14fda0 0x7f11cc071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:59568/0 (socket says 172.21.15.7:59568) 2023-03-03T11:15:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d1d82700 1 -- 172.21.15.7:0/4201355532 learned_addr learned my addr 172.21.15.7:0/4201355532 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:15:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d2d84700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f11cc0abe20 0x7f11cc1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:33060/0 (socket says 172.21.15.7:33060) 2023-03-03T11:15:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d2583700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 0x7f11cc156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:43630/0 (socket says 172.21.15.7:43630) 2023-03-03T11:15:04.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d2583700 1 -- 172.21.15.7:0/4201355532 >> v2:172.21.15.143:3300/0 conn(0x7f11cc14fda0 msgr2=0x7f11cc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:15:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d2583700 1 --2- 172.21.15.7:0/4201355532 >> v2:172.21.15.143:3300/0 conn(0x7f11cc14fda0 0x7f11cc071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:15:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d2583700 1 -- 172.21.15.7:0/4201355532 >> v1:172.21.15.143:6789/0 conn(0x7f11cc0abe20 legacy=0x7f11cc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:15:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d1581700 1 -- 172.21.15.7:0/4201355532 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453651347 0 0) 0x7f11cc0c2f30 con 0x7f11cc0abe20 2023-03-03T11:15:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d2583700 1 -- 172.21.15.7:0/4201355532 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc0c34b0 con 0x7f11cc14f9d0 2023-03-03T11:15:04.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d1d82700 1 --2- 172.21.15.7:0/4201355532 >> v2:172.21.15.143:3300/0 conn(0x7f11cc14fda0 0x7f11cc071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:15:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.377+0000 7f11d2583700 1 --2- 172.21.15.7:0/4201355532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 0x7f11cc156430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009840 tx=0x7f11c8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ce480e94808332 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:15:04.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d1581700 1 -- 172.21.15.7:0/4201355532 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8011830 con 0x7f11cc14f9d0 2023-03-03T11:15:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d1581700 1 -- 172.21.15.7:0/4201355532 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c8011990 con 0x7f11cc14f9d0 2023-03-03T11:15:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d1581700 1 -- 172.21.15.7:0/4201355532 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8011d20 con 0x7f11cc14f9d0 2023-03-03T11:15:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d47e7700 1 -- 172.21.15.7:0/4201355532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 msgr2=0x7f11cc156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:15:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d47e7700 1 --2- 172.21.15.7:0/4201355532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 0x7f11cc156430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009840 tx=0x7f11c8009b50 comp rx=0 tx=0).stop 2023-03-03T11:15:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d47e7700 1 -- 172.21.15.7:0/4201355532 shutdown_connections 2023-03-03T11:15:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d47e7700 1 --2- 172.21.15.7:0/4201355532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 0x7f11cc156430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:15:04.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d47e7700 1 --2- 172.21.15.7:0/4201355532 >> v2:172.21.15.143:3300/0 conn(0x7f11cc14fda0 0x7f11cc071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:15:04.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d47e7700 1 -- 172.21.15.7:0/4201355532 >> 172.21.15.7:0/4201355532 conn(0x7f11cc0c8900 msgr2=0x7f11cc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:15:04.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d47e7700 1 -- 172.21.15.7:0/4201355532 shutdown_connections 2023-03-03T11:15:04.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.378+0000 7f11d47e7700 1 -- 172.21.15.7:0/4201355532 wait complete. 2023-03-03T11:15:04.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.380+0000 7f11d47e7700 1 Processor -- start 2023-03-03T11:15:04.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.380+0000 7f11d47e7700 1 -- start start 2023-03-03T11:15:04.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d47e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11cc0abe20 0x7f11cc1e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:15:04.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d47e7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 0x7f11cc0c20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:15:04.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d47e7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11cc14fda0 0x7f11cc14c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:15:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d47e7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1dbc70 con 0x7f11cc0abe20 2023-03-03T11:15:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d47e7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1dbf50 con 0x7f11cc14fda0 2023-03-03T11:15:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d47e7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1dc230 con 0x7f11cc14f9d0 2023-03-03T11:15:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d2583700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11cc0abe20 0x7f11cc1e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:15:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d2583700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11cc0abe20 0x7f11cc1e7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35564/0 (socket says 172.21.15.7:35564) 2023-03-03T11:15:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d2583700 1 -- 172.21.15.7:0/1784203132 learned_addr learned my addr 172.21.15.7:0/1784203132 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:15:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d1d82700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 0x7f11cc0c20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:15:04.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d2583700 1 -- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 msgr2=0x7f11cc0c20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:15:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d2583700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 0x7f11cc0c20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:15:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d2d84700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11cc14fda0 0x7f11cc14c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:15:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d2583700 1 -- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11cc14fda0 msgr2=0x7f11cc14c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:15:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d2583700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11cc14fda0 0x7f11cc14c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:15:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d2583700 1 -- 172.21.15.7:0/1784203132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c0009550 con 0x7f11cc0abe20 2023-03-03T11:15:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.381+0000 7f11d2583700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11cc0abe20 0x7f11cc1e7280 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009fd0 tx=0x7f11c8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:15:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.382+0000 7f11beffd700 1 -- 172.21.15.7:0/1784203132 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c80045b0 con 0x7f11cc0abe20 2023-03-03T11:15:04.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.382+0000 7f11beffd700 1 -- 172.21.15.7:0/1784203132 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c801f070 con 0x7f11cc0abe20 2023-03-03T11:15:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.382+0000 7f11d47e7700 1 -- 172.21.15.7:0/1784203132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c80094a0 con 0x7f11cc0abe20 2023-03-03T11:15:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.382+0000 7f11beffd700 1 -- 172.21.15.7:0/1784203132 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c80038c0 con 0x7f11cc0abe20 2023-03-03T11:15:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.382+0000 7f11d47e7700 1 -- 172.21.15.7:0/1784203132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11cc0ac350 con 0x7f11cc0abe20 2023-03-03T11:15:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.383+0000 7f11d47e7700 1 -- 172.21.15.7:0/1784203132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11cc0ab630 con 0x7f11cc0abe20 2023-03-03T11:15:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.383+0000 7f11beffd700 1 -- 172.21.15.7:0/1784203132 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f11c802a020 con 0x7f11cc0abe20 2023-03-03T11:15:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.383+0000 7f11beffd700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11b00682f0 0x7f11b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:15:04.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.384+0000 7f11beffd700 1 -- 172.21.15.7:0/1784203132 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f11c80875d0 con 0x7f11cc0abe20 2023-03-03T11:15:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.384+0000 7f11d1d82700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11b00682f0 0x7f11b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:15:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.384+0000 7f11d1d82700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11b00682f0 0x7f11b006a7a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f11cc1dee10 tx=0x7f11c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:15:04.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.388+0000 7f11beffd700 1 -- 172.21.15.7:0/1784203132 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11c8053970 con 0x7f11cc0abe20 2023-03-03T11:15:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.626+0000 7f11d47e7700 1 -- 172.21.15.7:0/1784203132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11cc0c3e10 con 0x7f11cc0abe20 2023-03-03T11:15:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.626+0000 7f11beffd700 1 -- 172.21.15.7:0/1784203132 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f11c8056f90 con 0x7f11cc0abe20 2023-03-03T11:15:04.628 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:15:04.629 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:15:04.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.628+0000 7f11d47e7700 1 -- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11b00682f0 msgr2=0x7f11b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:15:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.628+0000 7f11d47e7700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11b00682f0 0x7f11b006a7a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f11cc1dee10 tx=0x7f11c0008040 comp rx=0 tx=0).stop 2023-03-03T11:15:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.628+0000 7f11d47e7700 1 -- 172.21.15.7:0/1784203132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11cc0abe20 msgr2=0x7f11cc1e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:15:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.628+0000 7f11d47e7700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11cc0abe20 0x7f11cc1e7280 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009fd0 tx=0x7f11c8004750 comp rx=0 tx=0).stop 2023-03-03T11:15:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.628+0000 7f11d47e7700 1 -- 172.21.15.7:0/1784203132 shutdown_connections 2023-03-03T11:15:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.628+0000 7f11d47e7700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11cc14fda0 0x7f11cc14c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:15:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.628+0000 7f11d47e7700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11b00682f0 0x7f11b006a7a0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:15:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.628+0000 7f11d47e7700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11cc0abe20 0x7f11cc1e7280 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:15:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.628+0000 7f11d47e7700 1 --2- 172.21.15.7:0/1784203132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11cc14f9d0 0x7f11cc0c20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:15:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.628+0000 7f11d47e7700 1 -- 172.21.15.7:0/1784203132 >> 172.21.15.7:0/1784203132 conn(0x7f11cc0c8900 msgr2=0x7f11cc1528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:15:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.629+0000 7f11d47e7700 1 -- 172.21.15.7:0/1784203132 shutdown_connections 2023-03-03T11:15:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:15:04.629+0000 7f11d47e7700 1 -- 172.21.15.7:0/1784203132 wait complete. 2023-03-03T11:15:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:15:04.642 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2023-03-03T11:15:10.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:10.787 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:40.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:15:40.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:11.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:11.039 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:16.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:16:16.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.213+0000 7ff39cc24700 1 Processor -- start 2023-03-03T11:16:16.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.214+0000 7ff39cc24700 1 -- start start 2023-03-03T11:16:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.214+0000 7ff39cc24700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff398155390 0x7ff3980c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:16.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.214+0000 7ff39cc24700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff398151770 0x7ff398151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.214+0000 7ff39cc24700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff398154910 con 0x7ff39806e020 2023-03-03T11:16:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.214+0000 7ff39cc24700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff398070b90 con 0x7ff398155390 2023-03-03T11:16:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.214+0000 7ff39cc24700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff398151320 con 0x7ff398151770 2023-03-03T11:16:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff398155390 0x7ff3980c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff395d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff39806e020 0x7ff398071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57860/0 (socket says 172.21.15.7:57860) 2023-03-03T11:16:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff398155390 0x7ff3980c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35646/0 (socket says 172.21.15.7:35646) 2023-03-03T11:16:16.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff395d9b700 1 -- 172.21.15.7:0/1611071291 learned_addr learned my addr 172.21.15.7:0/1611071291 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:16:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff396d9d700 1 --2- 172.21.15.7:0/1611071291 >> v2:172.21.15.143:3300/0 conn(0x7ff398151770 0x7ff398151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39659c700 1 -- 172.21.15.7:0/1611071291 >> v2:172.21.15.143:3300/0 conn(0x7ff398151770 msgr2=0x7ff398151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:16:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39659c700 1 --2- 172.21.15.7:0/1611071291 >> v2:172.21.15.143:3300/0 conn(0x7ff398151770 0x7ff398151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39659c700 1 -- 172.21.15.7:0/1611071291 >> v1:172.21.15.7:6789/0 conn(0x7ff39806e020 legacy=0x7ff398071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39559a700 1 -- 172.21.15.7:0/1611071291 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107336649 0 0) 0x7ff398154910 con 0x7ff39806e020 2023-03-03T11:16:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39659c700 1 -- 172.21.15.7:0/1611071291 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3980702a0 con 0x7ff398155390 2023-03-03T11:16:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39659c700 1 --2- 172.21.15.7:0/1611071291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398155390 0x7ff3980c3e30 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff39806ffe0 tx=0x7ff38c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c64b96e201e2c7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:16.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39559a700 1 -- 172.21.15.7:0/1611071291 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c011830 con 0x7ff398155390 2023-03-03T11:16:16.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39559a700 1 -- 172.21.15.7:0/1611071291 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c011990 con 0x7ff398155390 2023-03-03T11:16:16.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.215+0000 7ff39559a700 1 -- 172.21.15.7:0/1611071291 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c011d20 con 0x7ff398155390 2023-03-03T11:16:16.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.216+0000 7ff39cc24700 1 -- 172.21.15.7:0/1611071291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398155390 msgr2=0x7ff3980c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:16.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.216+0000 7ff39cc24700 1 --2- 172.21.15.7:0/1611071291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398155390 0x7ff3980c3e30 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff39806ffe0 tx=0x7ff38c009b20 comp rx=0 tx=0).stop 2023-03-03T11:16:16.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.216+0000 7ff39cc24700 1 -- 172.21.15.7:0/1611071291 shutdown_connections 2023-03-03T11:16:16.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.216+0000 7ff39cc24700 1 --2- 172.21.15.7:0/1611071291 >> v2:172.21.15.143:3300/0 conn(0x7ff398151770 0x7ff398151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:16.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.216+0000 7ff39cc24700 1 --2- 172.21.15.7:0/1611071291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398155390 0x7ff3980c3e30 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:16.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.216+0000 7ff39cc24700 1 -- 172.21.15.7:0/1611071291 >> 172.21.15.7:0/1611071291 conn(0x7ff39806c110 msgr2=0x7ff39806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:16:16.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.216+0000 7ff39cc24700 1 -- 172.21.15.7:0/1611071291 shutdown_connections 2023-03-03T11:16:16.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.216+0000 7ff39cc24700 1 -- 172.21.15.7:0/1611071291 wait complete. 2023-03-03T11:16:16.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.217+0000 7ff39cc24700 1 Processor -- start 2023-03-03T11:16:16.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.217+0000 7ff39cc24700 1 -- start start 2023-03-03T11:16:16.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff39cc24700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39806e020 0x7ff398145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:16.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff39cc24700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398151770 0x7ff3981db9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff39cc24700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff398155390 0x7ff398153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff39cc24700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff398152db0 con 0x7ff398151770 2023-03-03T11:16:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff39cc24700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3981dc7f0 con 0x7ff39806e020 2023-03-03T11:16:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff39cc24700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3980c1940 con 0x7ff398155390 2023-03-03T11:16:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff395d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398151770 0x7ff3981db9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff395d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398151770 0x7ff3981db9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35654/0 (socket says 172.21.15.7:35654) 2023-03-03T11:16:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff395d9b700 1 -- 172.21.15.7:0/2785294879 learned_addr learned my addr 172.21.15.7:0/2785294879 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:16:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff39659c700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39806e020 0x7ff398145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:16.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.218+0000 7ff396d9d700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff398155390 0x7ff398153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:16.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff395d9b700 1 -- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff398155390 msgr2=0x7ff398153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:16.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff395d9b700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff398155390 0x7ff398153ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:16.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff395d9b700 1 -- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39806e020 msgr2=0x7ff398145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:16.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff395d9b700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39806e020 0x7ff398145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:16.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff395d9b700 1 -- 172.21.15.7:0/2785294879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c0094a0 con 0x7ff398151770 2023-03-03T11:16:16.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff39659c700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39806e020 0x7ff398145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:16:16.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff395d9b700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398151770 0x7ff3981db9d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff3880011e0 tx=0x7ff38800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:16.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff386ffd700 1 -- 172.21.15.7:0/2785294879 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388004c80 con 0x7ff398151770 2023-03-03T11:16:16.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff386ffd700 1 -- 172.21.15.7:0/2785294879 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff388004e00 con 0x7ff398151770 2023-03-03T11:16:16.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff39cc24700 1 -- 172.21.15.7:0/2785294879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff398070820 con 0x7ff398151770 2023-03-03T11:16:16.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff386ffd700 1 -- 172.21.15.7:0/2785294879 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff388007310 con 0x7ff398151770 2023-03-03T11:16:16.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.219+0000 7ff39cc24700 1 -- 172.21.15.7:0/2785294879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff398070540 con 0x7ff398151770 2023-03-03T11:16:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.220+0000 7ff39cc24700 1 -- 172.21.15.7:0/2785294879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff398155760 con 0x7ff398151770 2023-03-03T11:16:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.221+0000 7ff386ffd700 1 -- 172.21.15.7:0/2785294879 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff38801a020 con 0x7ff398151770 2023-03-03T11:16:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.221+0000 7ff386ffd700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff378068320 0x7ff37806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.221+0000 7ff386ffd700 1 -- 172.21.15.7:0/2785294879 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff388085db0 con 0x7ff398151770 2023-03-03T11:16:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.221+0000 7ff39659c700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff378068320 0x7ff37806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.222+0000 7ff39659c700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff378068320 0x7ff37806a7d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff38c0090b0 tx=0x7ff38c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.226+0000 7ff386ffd700 1 -- 172.21.15.7:0/2785294879 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff388052150 con 0x7ff398151770 2023-03-03T11:16:16.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.467+0000 7ff39cc24700 1 -- 172.21.15.7:0/2785294879 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff398145d70 con 0x7ff398151770 2023-03-03T11:16:16.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.467+0000 7ff386ffd700 1 -- 172.21.15.7:0/2785294879 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff388055770 con 0x7ff398151770 2023-03-03T11:16:16.470 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:16:16.470 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:16:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 -- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff378068320 msgr2=0x7ff37806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff378068320 0x7ff37806a7d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff38c0090b0 tx=0x7ff38c01c040 comp rx=0 tx=0).stop 2023-03-03T11:16:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 -- 172.21.15.7:0/2785294879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398151770 msgr2=0x7ff3981db9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398151770 0x7ff3981db9d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7ff3880011e0 tx=0x7ff38800cae0 comp rx=0 tx=0).stop 2023-03-03T11:16:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 -- 172.21.15.7:0/2785294879 shutdown_connections 2023-03-03T11:16:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff398155390 0x7ff398153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff39806e020 0x7ff398145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff378068320 0x7ff37806a7d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 --2- 172.21.15.7:0/2785294879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff398151770 0x7ff3981db9d0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 -- 172.21.15.7:0/2785294879 >> 172.21.15.7:0/2785294879 conn(0x7ff39806c110 msgr2=0x7ff3980c7370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:16:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 -- 172.21.15.7:0/2785294879 shutdown_connections 2023-03-03T11:16:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:16.470+0000 7ff39cc24700 1 -- 172.21.15.7:0/2785294879 wait complete. 2023-03-03T11:16:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:16:16.484 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-03-03T11:16:41.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:41.176 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:16:41.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:16:41.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.779+0000 7fc758f03700 1 Processor -- start 2023-03-03T11:16:41.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.779+0000 7fc758f03700 1 -- start start 2023-03-03T11:16:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.780+0000 7fc758f03700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc754152b60 0x7fc75415dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.780+0000 7fc758f03700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7540ac900 con 0x7fc7540ace30 2023-03-03T11:16:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.780+0000 7fc758f03700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7540acb50 con 0x7fc7540ac220 2023-03-03T11:16:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.780+0000 7fc758f03700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc754071e30 con 0x7fc754152b60 2023-03-03T11:16:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.780+0000 7fc751d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc7540ace30 0x7fc754152450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57892/0 (socket says 172.21.15.7:57892) 2023-03-03T11:16:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.780+0000 7fc751d9b700 1 -- 172.21.15.7:0/2304420731 learned_addr learned my addr 172.21.15.7:0/2304420731 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:16:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc752d9d700 1 --2- 172.21.15.7:0/2304420731 >> v2:172.21.15.143:3300/0 conn(0x7fc754152b60 0x7fc75415dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143470858 0 0) 0x7fc7540ac900 con 0x7fc7540ace30 2023-03-03T11:16:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7400035b0 con 0x7fc7540ace30 2023-03-03T11:16:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129246704 0 0) 0x7fc7400035b0 con 0x7fc7540ace30 2023-03-03T11:16:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 >> v2:172.21.15.143:3300/0 conn(0x7fc754152b60 msgr2=0x7fc75415dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:16:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 --2- 172.21.15.7:0/2304420731 >> v2:172.21.15.143:3300/0 conn(0x7fc754152b60 0x7fc75415dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 >> v1:172.21.15.143:6789/0 conn(0x7fc7540ac220 legacy=0x7fc7540736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7541559c0 con 0x7fc7540ace30 2023-03-03T11:16:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc73c001670 con 0x7fc7540ace30 2023-03-03T11:16:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 >> v1:172.21.15.7:6789/0 conn(0x7fc7540ace30 legacy=0x7fc754152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc740003a70 0x7fc740003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740004850 0x7fc740004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc740005250 0x7fc74000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc754071e30 con 0x7fc740003a70 2023-03-03T11:16:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74000c190 con 0x7fc740004850 2023-03-03T11:16:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74000c350 con 0x7fc740005250 2023-03-03T11:16:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc75259c700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc740003a70 0x7fc740003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc751d9b700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740004850 0x7fc740004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.781+0000 7fc752d9d700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc740005250 0x7fc74000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc75259c700 1 -- 172.21.15.7:0/2304420731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc740005250 msgr2=0x7fc74000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:16:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc75259c700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc740005250 0x7fc74000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc75259c700 1 -- 172.21.15.7:0/2304420731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740004850 msgr2=0x7fc740004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:16:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc75259c700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc740004850 0x7fc740004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc752d9d700 1 -- 172.21.15.7:0/2304420731 reap_dead start 2023-03-03T11:16:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc75259c700 1 -- 172.21.15.7:0/2304420731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7400035b0 con 0x7fc740003a70 2023-03-03T11:16:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc75259c700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc740003a70 0x7fc740003e80 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc744007fd0 tx=0x7fc740005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a0d73a7ca43ad7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7440052d0 con 0x7fc740003a70 2023-03-03T11:16:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc744005430 con 0x7fc740003a70 2023-03-03T11:16:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc75159a700 1 -- 172.21.15.7:0/2304420731 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7440038c0 con 0x7fc740003a70 2023-03-03T11:16:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc758f03700 1 -- 172.21.15.7:0/2304420731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc740003a70 msgr2=0x7fc740003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc758f03700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc740003a70 0x7fc740003e80 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fc744007fd0 tx=0x7fc740005880 comp rx=0 tx=0).stop 2023-03-03T11:16:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc758f03700 1 -- 172.21.15.7:0/2304420731 shutdown_connections 2023-03-03T11:16:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc758f03700 1 --2- 172.21.15.7:0/2304420731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc740003a70 0x7fc740003e80 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.782+0000 7fc758f03700 1 -- 172.21.15.7:0/2304420731 >> 172.21.15.7:0/2304420731 conn(0x7fc7540a94d0 msgr2=0x7fc7540a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:16:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.783+0000 7fc758f03700 1 -- 172.21.15.7:0/2304420731 shutdown_connections 2023-03-03T11:16:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.783+0000 7fc758f03700 1 -- 172.21.15.7:0/2304420731 wait complete. 2023-03-03T11:16:41.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.784+0000 7fc758f03700 1 Processor -- start 2023-03-03T11:16:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.784+0000 7fc758f03700 1 -- start start 2023-03-03T11:16:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc758f03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc740003a70 0x7fc7541ea340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc758f03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7540c62e0 0x7fc7540c6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc758f03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7541ec2b0 0x7fc7541ec720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc758f03700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7541dbdb0 con 0x7fc7540c62e0 2023-03-03T11:16:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc758f03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7540c4800 con 0x7fc7541ec2b0 2023-03-03T11:16:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc758f03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7541ecc60 con 0x7fc740003a70 2023-03-03T11:16:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc751d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7540c62e0 0x7fc7540c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc751d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7540c62e0 0x7fc7540c6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35692/0 (socket says 172.21.15.7:35692) 2023-03-03T11:16:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc751d9b700 1 -- 172.21.15.7:0/733783329 learned_addr learned my addr 172.21.15.7:0/733783329 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:16:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc752d9d700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7541ec2b0 0x7fc7541ec720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc75259c700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc740003a70 0x7fc7541ea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc751d9b700 1 -- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc740003a70 msgr2=0x7fc7541ea340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc751d9b700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc740003a70 0x7fc7541ea340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc751d9b700 1 -- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7541ec2b0 msgr2=0x7fc7541ec720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc751d9b700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7541ec2b0 0x7fc7541ec720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc751d9b700 1 -- 172.21.15.7:0/733783329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc754155f80 con 0x7fc7540c62e0 2023-03-03T11:16:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc75259c700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc740003a70 0x7fc7541ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:16:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc751d9b700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7540c62e0 0x7fc7540c6750 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc73c007290 tx=0x7fc73c0075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc74affd700 1 -- 172.21.15.7:0/733783329 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c011580 con 0x7fc7540c62e0 2023-03-03T11:16:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc74affd700 1 -- 172.21.15.7:0/733783329 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c0116e0 con 0x7fc7540c62e0 2023-03-03T11:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc758f03700 1 -- 172.21.15.7:0/733783329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc754155ca0 con 0x7fc7540c62e0 2023-03-03T11:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc74affd700 1 -- 172.21.15.7:0/733783329 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c011a50 con 0x7fc7540c62e0 2023-03-03T11:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.785+0000 7fc758f03700 1 -- 172.21.15.7:0/733783329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc754156260 con 0x7fc7540c62e0 2023-03-03T11:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.786+0000 7fc758f03700 1 -- 172.21.15.7:0/733783329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc754074bb0 con 0x7fc7540c62e0 2023-03-03T11:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.787+0000 7fc74affd700 1 -- 172.21.15.7:0/733783329 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc73c011bb0 con 0x7fc7540c62e0 2023-03-03T11:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.787+0000 7fc74affd700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7340682f0 0x7fc73406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.787+0000 7fc74affd700 1 -- 172.21.15.7:0/733783329 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc73c086260 con 0x7fc7540c62e0 2023-03-03T11:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.787+0000 7fc75259c700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7340682f0 0x7fc73406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:41.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.788+0000 7fc75259c700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7340682f0 0x7fc73406a7a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc744015040 tx=0x7fc744009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:41.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:41.792+0000 7fc74affd700 1 -- 172.21.15.7:0/733783329 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc73c0525a0 con 0x7fc7540c62e0 2023-03-03T11:16:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.028+0000 7fc758f03700 1 -- 172.21.15.7:0/733783329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7540a9fb0 con 0x7fc7540c62e0 2023-03-03T11:16:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.029+0000 7fc74affd700 1 -- 172.21.15.7:0/733783329 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc73c055bc0 con 0x7fc7540c62e0 2023-03-03T11:16:42.032 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:16:42.032 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:16:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 -- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7340682f0 msgr2=0x7fc73406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7340682f0 0x7fc73406a7a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc744015040 tx=0x7fc744009030 comp rx=0 tx=0).stop 2023-03-03T11:16:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 -- 172.21.15.7:0/733783329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7540c62e0 msgr2=0x7fc7540c6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7540c62e0 0x7fc7540c6750 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fc73c007290 tx=0x7fc73c0075a0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 -- 172.21.15.7:0/733783329 shutdown_connections 2023-03-03T11:16:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7541ec2b0 0x7fc7541ec720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc740003a70 0x7fc7541ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7340682f0 0x7fc73406a7a0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 --2- 172.21.15.7:0/733783329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7540c62e0 0x7fc7540c6750 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 -- 172.21.15.7:0/733783329 >> 172.21.15.7:0/733783329 conn(0x7fc7540a94d0 msgr2=0x7fc7540c2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:16:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 -- 172.21.15.7:0/733783329 shutdown_connections 2023-03-03T11:16:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.031+0000 7fc758f03700 1 -- 172.21.15.7:0/733783329 wait complete. 2023-03-03T11:16:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:16:42.046 INFO:tasks.mds_thrash.fs.[b]:waiting for 58 secs before thrashing 2023-03-03T11:16:42.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:16:42.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.609+0000 7f86188de700 1 Processor -- start 2023-03-03T11:16:42.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.609+0000 7f86188de700 1 -- start start 2023-03-03T11:16:42.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f86188de700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f86140c6010 0x7f86140c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:42.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f86188de700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f86140c17f0 0x7f86140c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:42.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f86188de700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86140c5500 con 0x7f861406d6a0 2023-03-03T11:16:42.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f86188de700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8614152100 con 0x7f86140c17f0 2023-03-03T11:16:42.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f86188de700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861414ebe0 con 0x7f86140c6010 2023-03-03T11:16:42.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f8611d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f86140c17f0 0x7f86140c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:42.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f8612d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f861406d6a0 0x7f861406dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57914/0 (socket says 172.21.15.7:57914) 2023-03-03T11:16:42.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f8612d9d700 1 -- 172.21.15.7:0/2509888125 learned_addr learned my addr 172.21.15.7:0/2509888125 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:16:42.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f861259c700 1 --2- 172.21.15.7:0/2509888125 >> v2:172.21.15.143:3300/0 conn(0x7f86140c6010 0x7f86140c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:42.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f8611d9b700 1 -- 172.21.15.7:0/2509888125 >> v2:172.21.15.143:3300/0 conn(0x7f86140c6010 msgr2=0x7f86140c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:16:42.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f8611d9b700 1 --2- 172.21.15.7:0/2509888125 >> v2:172.21.15.143:3300/0 conn(0x7f86140c6010 0x7f86140c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f8611d9b700 1 -- 172.21.15.7:0/2509888125 >> v1:172.21.15.7:6789/0 conn(0x7f861406d6a0 legacy=0x7f861406dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:42.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f8611d9b700 1 -- 172.21.15.7:0/2509888125 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86140c2e70 con 0x7f86140c17f0 2023-03-03T11:16:42.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.610+0000 7f861159a700 1 -- 172.21.15.7:0/2509888125 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484615072 0 0) 0x7f86140c5500 con 0x7f861406d6a0 2023-03-03T11:16:42.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f8611d9b700 1 --2- 172.21.15.7:0/2509888125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86140c17f0 0x7f86140c4fc0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f86140c6530 tx=0x7f860400bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c237ffd6cd3d88 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:42.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f861159a700 1 -- 172.21.15.7:0/2509888125 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860400dc70 con 0x7f86140c17f0 2023-03-03T11:16:42.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f861159a700 1 -- 172.21.15.7:0/2509888125 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860400ddd0 con 0x7f86140c17f0 2023-03-03T11:16:42.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f861159a700 1 -- 172.21.15.7:0/2509888125 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8604010690 con 0x7f86140c17f0 2023-03-03T11:16:42.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f86188de700 1 -- 172.21.15.7:0/2509888125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86140c17f0 msgr2=0x7f86140c4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:42.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f86188de700 1 --2- 172.21.15.7:0/2509888125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86140c17f0 0x7f86140c4fc0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f86140c6530 tx=0x7f860400bdc0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f86188de700 1 -- 172.21.15.7:0/2509888125 shutdown_connections 2023-03-03T11:16:42.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f86188de700 1 --2- 172.21.15.7:0/2509888125 >> v2:172.21.15.143:3300/0 conn(0x7f86140c6010 0x7f86140c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f86188de700 1 --2- 172.21.15.7:0/2509888125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86140c17f0 0x7f86140c4fc0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f86188de700 1 -- 172.21.15.7:0/2509888125 >> 172.21.15.7:0/2509888125 conn(0x7f86140a9430 msgr2=0x7f86140c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:16:42.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f86188de700 1 -- 172.21.15.7:0/2509888125 shutdown_connections 2023-03-03T11:16:42.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.611+0000 7f86188de700 1 -- 172.21.15.7:0/2509888125 wait complete. 2023-03-03T11:16:42.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.613+0000 7f86188de700 1 Processor -- start 2023-03-03T11:16:42.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.613+0000 7f86188de700 1 -- start start 2023-03-03T11:16:42.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.613+0000 7f86188de700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406d6a0 0x7f8614071200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:42.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.613+0000 7f86188de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86140c17f0 0x7f8614071740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:42.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.613+0000 7f86188de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86140c6010 0x7f86140c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:42.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.613+0000 7f86188de700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861414ab30 con 0x7f861406d6a0 2023-03-03T11:16:42.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.613+0000 7f86188de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86141ed5a0 con 0x7f86140c17f0 2023-03-03T11:16:42.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.613+0000 7f86188de700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86140a5eb0 con 0x7f86140c6010 2023-03-03T11:16:42.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f861259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406d6a0 0x7f8614071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:42.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f861259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406d6a0 0x7f8614071200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35704/0 (socket says 172.21.15.7:35704) 2023-03-03T11:16:42.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f861259c700 1 -- 172.21.15.7:0/2881260653 learned_addr learned my addr 172.21.15.7:0/2881260653 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:16:42.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f8611d9b700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86140c17f0 0x7f8614071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:42.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f8612d9d700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86140c6010 0x7f86140c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:42.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f861259c700 1 -- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86140c6010 msgr2=0x7f86140c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:42.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f861259c700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86140c6010 0x7f86140c00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f861259c700 1 -- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86140c17f0 msgr2=0x7f8614071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:42.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f861259c700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86140c17f0 0x7f8614071740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f861259c700 1 -- 172.21.15.7:0/2881260653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860400b740 con 0x7f861406d6a0 2023-03-03T11:16:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f861259c700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406d6a0 0x7f8614071200 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f860800d800 tx=0x7f860800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f8602ffd700 1 -- 172.21.15.7:0/2881260653 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86080125f0 con 0x7f861406d6a0 2023-03-03T11:16:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f8602ffd700 1 -- 172.21.15.7:0/2881260653 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8608012750 con 0x7f861406d6a0 2023-03-03T11:16:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f86188de700 1 -- 172.21.15.7:0/2881260653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86140aa960 con 0x7f861406d6a0 2023-03-03T11:16:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f8602ffd700 1 -- 172.21.15.7:0/2881260653 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8608012b00 con 0x7f861406d6a0 2023-03-03T11:16:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.614+0000 7f86188de700 1 -- 172.21.15.7:0/2881260653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86140aa680 con 0x7f861406d6a0 2023-03-03T11:16:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.615+0000 7f86188de700 1 -- 172.21.15.7:0/2881260653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86140c2b00 con 0x7f861406d6a0 2023-03-03T11:16:42.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.616+0000 7f8602ffd700 1 -- 172.21.15.7:0/2881260653 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8608012c60 con 0x7f861406d6a0 2023-03-03T11:16:42.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.616+0000 7f8602ffd700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f40682f0 0x7f85f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:16:42.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.616+0000 7f8602ffd700 1 -- 172.21.15.7:0/2881260653 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8608086290 con 0x7f861406d6a0 2023-03-03T11:16:42.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.616+0000 7f8611d9b700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f40682f0 0x7f85f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:16:42.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.617+0000 7f8611d9b700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f40682f0 0x7f85f406a7a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f860400be00 tx=0x7f8604008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:16:42.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.620+0000 7f8602ffd700 1 -- 172.21.15.7:0/2881260653 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8608052630 con 0x7f861406d6a0 2023-03-03T11:16:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.856+0000 7f86188de700 1 -- 172.21.15.7:0/2881260653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8614071c80 con 0x7f861406d6a0 2023-03-03T11:16:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.857+0000 7f8602ffd700 1 -- 172.21.15.7:0/2881260653 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8608055c50 con 0x7f861406d6a0 2023-03-03T11:16:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:16:42.861 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:16:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 -- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f40682f0 msgr2=0x7f85f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f40682f0 0x7f85f406a7a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f860400be00 tx=0x7f8604008040 comp rx=0 tx=0).stop 2023-03-03T11:16:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 -- 172.21.15.7:0/2881260653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406d6a0 msgr2=0x7f8614071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:16:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406d6a0 0x7f8614071200 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f860800d800 tx=0x7f860800db10 comp rx=0 tx=0).stop 2023-03-03T11:16:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 -- 172.21.15.7:0/2881260653 shutdown_connections 2023-03-03T11:16:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86140c6010 0x7f86140c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85f40682f0 0x7f85f406a7a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f861406d6a0 0x7f8614071200 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 --2- 172.21.15.7:0/2881260653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86140c17f0 0x7f8614071740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:16:42.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 -- 172.21.15.7:0/2881260653 >> 172.21.15.7:0/2881260653 conn(0x7f86140a9430 msgr2=0x7f861406dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:16:42.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 -- 172.21.15.7:0/2881260653 shutdown_connections 2023-03-03T11:16:42.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:16:42.859+0000 7f86188de700 1 -- 172.21.15.7:0/2881260653 wait complete. 2023-03-03T11:16:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:16:42.874 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2023-03-03T11:17:11.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:11.248 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:25.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:17:26.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.012+0000 7f11e13ea700 1 Processor -- start 2023-03-03T11:17:26.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.012+0000 7f11e13ea700 1 -- start start 2023-03-03T11:17:26.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11e13ea700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f11dc06bea0 0x7f11dc157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:26.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11e13ea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11dc0c28e0 0x7f11dc1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:26.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11e13ea700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0c3ae0 con 0x7f11dc151dd0 2023-03-03T11:17:26.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11e13ea700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11dc06d970 con 0x7f11dc0c28e0 2023-03-03T11:17:26.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11e13ea700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11dc06c3c0 con 0x7f11dc06bea0 2023-03-03T11:17:26.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11db7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11dc0c28e0 0x7f11dc1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:26.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11daffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f11dc151dd0 0x7f11dc0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57948/0 (socket says 172.21.15.7:57948) 2023-03-03T11:17:26.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11daffd700 1 -- 172.21.15.7:0/3978508814 learned_addr learned my addr 172.21.15.7:0/3978508814 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:17:26.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11da7fc700 1 --2- 172.21.15.7:0/3978508814 >> v2:172.21.15.143:3300/0 conn(0x7f11dc06bea0 0x7f11dc157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:26.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11db7fe700 1 -- 172.21.15.7:0/3978508814 >> v2:172.21.15.143:3300/0 conn(0x7f11dc06bea0 msgr2=0x7f11dc157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:17:26.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11db7fe700 1 --2- 172.21.15.7:0/3978508814 >> v2:172.21.15.143:3300/0 conn(0x7f11dc06bea0 0x7f11dc157cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:26.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11db7fe700 1 -- 172.21.15.7:0/3978508814 >> v1:172.21.15.7:6789/0 conn(0x7f11dc151dd0 legacy=0x7f11dc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:26.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.013+0000 7f11d9ffb700 1 -- 172.21.15.7:0/3978508814 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234091108 0 0) 0x7f11dc0c3ae0 con 0x7f11dc151dd0 2023-03-03T11:17:26.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11db7fe700 1 -- 172.21.15.7:0/3978508814 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc06cab0 con 0x7f11dc0c28e0 2023-03-03T11:17:26.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11db7fe700 1 --2- 172.21.15.7:0/3978508814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc0c28e0 0x7f11dc1581f0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f11dc1521a0 tx=0x7f11d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422734ecaf6b9c51 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:26.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11d9ffb700 1 -- 172.21.15.7:0/3978508814 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d400e070 con 0x7f11dc0c28e0 2023-03-03T11:17:26.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11d9ffb700 1 -- 172.21.15.7:0/3978508814 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d400bbd0 con 0x7f11dc0c28e0 2023-03-03T11:17:26.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11d9ffb700 1 -- 172.21.15.7:0/3978508814 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4012430 con 0x7f11dc0c28e0 2023-03-03T11:17:26.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11e13ea700 1 -- 172.21.15.7:0/3978508814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc0c28e0 msgr2=0x7f11dc1581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:26.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11e13ea700 1 --2- 172.21.15.7:0/3978508814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc0c28e0 0x7f11dc1581f0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f11dc1521a0 tx=0x7f11d400b330 comp rx=0 tx=0).stop 2023-03-03T11:17:26.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11e13ea700 1 -- 172.21.15.7:0/3978508814 shutdown_connections 2023-03-03T11:17:26.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11e13ea700 1 --2- 172.21.15.7:0/3978508814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc0c28e0 0x7f11dc1581f0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:26.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11e13ea700 1 --2- 172.21.15.7:0/3978508814 >> v2:172.21.15.143:3300/0 conn(0x7f11dc06bea0 0x7f11dc157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:26.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11e13ea700 1 -- 172.21.15.7:0/3978508814 >> 172.21.15.7:0/3978508814 conn(0x7f11dc0a94d0 msgr2=0x7f11dc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:17:26.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.014+0000 7f11e13ea700 1 -- 172.21.15.7:0/3978508814 shutdown_connections 2023-03-03T11:17:26.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.015+0000 7f11e13ea700 1 -- 172.21.15.7:0/3978508814 wait complete. 2023-03-03T11:17:26.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.016+0000 7f11e13ea700 1 Processor -- start 2023-03-03T11:17:26.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.016+0000 7f11e13ea700 1 -- start start 2023-03-03T11:17:26.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11e13ea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11dc06bea0 0x7f11dc14fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:26.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11e13ea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11dc0c28e0 0x7f11dc1ea110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:26.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11e13ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc151dd0 0x7f11dc1ea650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:26.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11e13ea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc1ee290 con 0x7f11dc151dd0 2023-03-03T11:17:26.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11e13ea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc1ed680 con 0x7f11dc06bea0 2023-03-03T11:17:26.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11e13ea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc1ed960 con 0x7f11dc0c28e0 2023-03-03T11:17:26.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11db7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc151dd0 0x7f11dc1ea650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:26.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11db7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc151dd0 0x7f11dc1ea650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35742/0 (socket says 172.21.15.7:35742) 2023-03-03T11:17:26.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11db7fe700 1 -- 172.21.15.7:0/410138581 learned_addr learned my addr 172.21.15.7:0/410138581 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:17:26.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11da7fc700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11dc0c28e0 0x7f11dc1ea110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:26.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11daffd700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11dc06bea0 0x7f11dc14fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:26.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11db7fe700 1 -- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11dc0c28e0 msgr2=0x7f11dc1ea110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:26.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11db7fe700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11dc0c28e0 0x7f11dc1ea110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:26.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11db7fe700 1 -- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11dc06bea0 msgr2=0x7f11dc14fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:26.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11db7fe700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11dc06bea0 0x7f11dc14fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:26.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11db7fe700 1 -- 172.21.15.7:0/410138581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d4009a70 con 0x7f11dc151dd0 2023-03-03T11:17:26.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11db7fe700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc151dd0 0x7f11dc1ea650 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f11dc06bae0 tx=0x7f11d4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:26.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11c37fe700 1 -- 172.21.15.7:0/410138581 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4012430 con 0x7f11dc151dd0 2023-03-03T11:17:26.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11c37fe700 1 -- 172.21.15.7:0/410138581 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d4020070 con 0x7f11dc151dd0 2023-03-03T11:17:26.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11e13ea700 1 -- 172.21.15.7:0/410138581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11dc0c3280 con 0x7f11dc151dd0 2023-03-03T11:17:26.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11c37fe700 1 -- 172.21.15.7:0/410138581 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4004280 con 0x7f11dc151dd0 2023-03-03T11:17:26.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.017+0000 7f11e13ea700 1 -- 172.21.15.7:0/410138581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc06c610 con 0x7f11dc151dd0 2023-03-03T11:17:26.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.018+0000 7f11e13ea700 1 -- 172.21.15.7:0/410138581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc0c3560 con 0x7f11dc151dd0 2023-03-03T11:17:26.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.019+0000 7f11c37fe700 1 -- 172.21.15.7:0/410138581 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f11d40043e0 con 0x7f11dc151dd0 2023-03-03T11:17:26.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.019+0000 7f11c37fe700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c80682f0 0x7f11c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:26.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.019+0000 7f11c37fe700 1 -- 172.21.15.7:0/410138581 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f11d4022070 con 0x7f11dc151dd0 2023-03-03T11:17:26.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.019+0000 7f11daffd700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c80682f0 0x7f11c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:26.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.020+0000 7f11daffd700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c80682f0 0x7f11c806a7a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f11d0000f60 tx=0x7f11d0007c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:26.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.024+0000 7f11c37fe700 1 -- 172.21.15.7:0/410138581 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11d4054cc0 con 0x7f11dc151dd0 2023-03-03T11:17:26.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.259+0000 7f11e13ea700 1 -- 172.21.15.7:0/410138581 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11dc074bb0 con 0x7f11dc151dd0 2023-03-03T11:17:26.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.260+0000 7f11c37fe700 1 -- 172.21.15.7:0/410138581 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f11d40582e0 con 0x7f11dc151dd0 2023-03-03T11:17:26.262 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:17:26.262 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:17:26.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.261+0000 7f11e13ea700 1 -- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c80682f0 msgr2=0x7f11c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:26.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.261+0000 7f11e13ea700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c80682f0 0x7f11c806a7a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f11d0000f60 tx=0x7f11d0007c00 comp rx=0 tx=0).stop 2023-03-03T11:17:26.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.261+0000 7f11e13ea700 1 -- 172.21.15.7:0/410138581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc151dd0 msgr2=0x7f11dc1ea650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:26.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.261+0000 7f11e13ea700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc151dd0 0x7f11dc1ea650 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f11dc06bae0 tx=0x7f11d4004ad0 comp rx=0 tx=0).stop 2023-03-03T11:17:26.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.261+0000 7f11e13ea700 1 -- 172.21.15.7:0/410138581 shutdown_connections 2023-03-03T11:17:26.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.261+0000 7f11e13ea700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11c80682f0 0x7f11c806a7a0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:26.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.261+0000 7f11e13ea700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11dc151dd0 0x7f11dc1ea650 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:26.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.261+0000 7f11e13ea700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11dc06bea0 0x7f11dc14fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:26.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.261+0000 7f11e13ea700 1 --2- 172.21.15.7:0/410138581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11dc0c28e0 0x7f11dc1ea110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:26.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.261+0000 7f11e13ea700 1 -- 172.21.15.7:0/410138581 >> 172.21.15.7:0/410138581 conn(0x7f11dc0a94d0 msgr2=0x7f11dc071630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:17:26.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.262+0000 7f11e13ea700 1 -- 172.21.15.7:0/410138581 shutdown_connections 2023-03-03T11:17:26.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:26.262+0000 7f11e13ea700 1 -- 172.21.15.7:0/410138581 wait complete. 2023-03-03T11:17:26.265 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:17:26.275 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-03-03T11:17:40.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:17:40.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.183+0000 7f9738939700 1 Processor -- start 2023-03-03T11:17:40.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.183+0000 7f9738939700 1 -- start start 2023-03-03T11:17:40.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.184+0000 7f9738939700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f973414f9d0 0x7f9734156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:40.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.184+0000 7f9738939700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f973414fda0 0x7f9734071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:40.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.184+0000 7f9738939700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97340c2f30 con 0x7f97340abe20 2023-03-03T11:17:40.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.184+0000 7f9738939700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97340c31d0 con 0x7f973414fda0 2023-03-03T11:17:40.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.184+0000 7f9738939700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97341524a0 con 0x7f973414f9d0 2023-03-03T11:17:40.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.184+0000 7f9731d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f973414fda0 0x7f9734071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:40.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.184+0000 7f9731d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f973414fda0 0x7f9734071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35766/0 (socket says 172.21.15.7:35766) 2023-03-03T11:17:40.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.184+0000 7f9731d9b700 1 -- 172.21.15.7:0/1775257372 learned_addr learned my addr 172.21.15.7:0/1775257372 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:17:40.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.185+0000 7f973259c700 1 --2- 172.21.15.7:0/1775257372 >> v2:172.21.15.143:3300/0 conn(0x7f973414f9d0 0x7f9734156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:40.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.185+0000 7f9731d9b700 1 -- 172.21.15.7:0/1775257372 >> v2:172.21.15.143:3300/0 conn(0x7f973414f9d0 msgr2=0x7f9734156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:17:40.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.185+0000 7f9731d9b700 1 --2- 172.21.15.7:0/1775257372 >> v2:172.21.15.143:3300/0 conn(0x7f973414f9d0 0x7f9734156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:40.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.185+0000 7f9731d9b700 1 -- 172.21.15.7:0/1775257372 >> v1:172.21.15.143:6789/0 conn(0x7f97340abe20 legacy=0x7f97341529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:17:40.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.185+0000 7f9731d9b700 1 -- 172.21.15.7:0/1775257372 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97340c34b0 con 0x7f973414fda0 2023-03-03T11:17:40.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.185+0000 7f973259c700 1 --2- 172.21.15.7:0/1775257372 >> v2:172.21.15.143:3300/0 conn(0x7f973414f9d0 0x7f9734156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:17:40.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.185+0000 7f9731d9b700 1 --2- 172.21.15.7:0/1775257372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f973414fda0 0x7f9734071890 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f97340710a0 tx=0x7f9728009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b5e1262cca8513 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:40.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.185+0000 7f973159a700 1 -- 172.21.15.7:0/1775257372 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9728004230 con 0x7f973414fda0 2023-03-03T11:17:40.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.185+0000 7f973159a700 1 -- 172.21.15.7:0/1775257372 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9728004390 con 0x7f973414fda0 2023-03-03T11:17:40.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.185+0000 7f973159a700 1 -- 172.21.15.7:0/1775257372 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97280036a0 con 0x7f973414fda0 2023-03-03T11:17:40.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.186+0000 7f9738939700 1 -- 172.21.15.7:0/1775257372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f973414fda0 msgr2=0x7f9734071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:40.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.186+0000 7f9738939700 1 --2- 172.21.15.7:0/1775257372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f973414fda0 0x7f9734071890 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f97340710a0 tx=0x7f9728009bd0 comp rx=0 tx=0).stop 2023-03-03T11:17:40.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.186+0000 7f9738939700 1 -- 172.21.15.7:0/1775257372 shutdown_connections 2023-03-03T11:17:40.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.186+0000 7f9738939700 1 --2- 172.21.15.7:0/1775257372 >> v2:172.21.15.143:3300/0 conn(0x7f973414f9d0 0x7f9734156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:40.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.186+0000 7f9738939700 1 --2- 172.21.15.7:0/1775257372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f973414fda0 0x7f9734071890 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:40.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.186+0000 7f9738939700 1 -- 172.21.15.7:0/1775257372 >> 172.21.15.7:0/1775257372 conn(0x7f97340c8900 msgr2=0x7f97340c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:17:40.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.186+0000 7f9738939700 1 -- 172.21.15.7:0/1775257372 shutdown_connections 2023-03-03T11:17:40.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.186+0000 7f9738939700 1 -- 172.21.15.7:0/1775257372 wait complete. 2023-03-03T11:17:40.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.187+0000 7f9738939700 1 Processor -- start 2023-03-03T11:17:40.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.187+0000 7f9738939700 1 -- start start 2023-03-03T11:17:40.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f9738939700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97340abe20 0x7f97341df170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:40.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f9738939700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f973414f9d0 0x7f97340c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:40.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f9738939700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f973414fda0 0x7f973414aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:40.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f9738939700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973406fb90 con 0x7f97340abe20 2023-03-03T11:17:40.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f9738939700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97341edcd0 con 0x7f973414f9d0 2023-03-03T11:17:40.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f9738939700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97341ee200 con 0x7f973414fda0 2023-03-03T11:17:40.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f973259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97340abe20 0x7f97341df170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:40.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f973259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97340abe20 0x7f97341df170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35770/0 (socket says 172.21.15.7:35770) 2023-03-03T11:17:40.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f973259c700 1 -- 172.21.15.7:0/4232031960 learned_addr learned my addr 172.21.15.7:0/4232031960 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:17:40.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f9731d9b700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f973414f9d0 0x7f97340c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:40.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f9732d9d700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f973414fda0 0x7f973414aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f973259c700 1 -- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f973414fda0 msgr2=0x7f973414aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f973259c700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f973414fda0 0x7f973414aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f973259c700 1 -- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f973414f9d0 msgr2=0x7f97340c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f973259c700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f973414f9d0 0x7f97340c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.188+0000 7f973259c700 1 -- 172.21.15.7:0/4232031960 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9728009550 con 0x7f97340abe20 2023-03-03T11:17:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.189+0000 7f973259c700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97340abe20 0x7f97341df170 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f972400c780 tx=0x7f972400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.189+0000 7f9722ffd700 1 -- 172.21.15.7:0/4232031960 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9724011800 con 0x7f97340abe20 2023-03-03T11:17:40.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.189+0000 7f9722ffd700 1 -- 172.21.15.7:0/4232031960 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9724011960 con 0x7f97340abe20 2023-03-03T11:17:40.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.189+0000 7f9738939700 1 -- 172.21.15.7:0/4232031960 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97340ac350 con 0x7f97340abe20 2023-03-03T11:17:40.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.189+0000 7f9722ffd700 1 -- 172.21.15.7:0/4232031960 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9724011cd0 con 0x7f97340abe20 2023-03-03T11:17:40.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.189+0000 7f9738939700 1 -- 172.21.15.7:0/4232031960 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97340ab630 con 0x7f97340abe20 2023-03-03T11:17:40.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.190+0000 7f9738939700 1 -- 172.21.15.7:0/4232031960 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97340c3e10 con 0x7f97340abe20 2023-03-03T11:17:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.190+0000 7f9722ffd700 1 -- 172.21.15.7:0/4232031960 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9724020070 con 0x7f97340abe20 2023-03-03T11:17:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.191+0000 7f9722ffd700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97140682f0 0x7f971406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.191+0000 7f9722ffd700 1 -- 172.21.15.7:0/4232031960 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9724086010 con 0x7f97340abe20 2023-03-03T11:17:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.191+0000 7f9731d9b700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97140682f0 0x7f971406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.191+0000 7f9731d9b700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97140682f0 0x7f971406a7a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f97341501d0 tx=0x7f9728008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.195+0000 7f9722ffd700 1 -- 172.21.15.7:0/4232031960 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9724052330 con 0x7f97340abe20 2023-03-03T11:17:40.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.433+0000 7f9738939700 1 -- 172.21.15.7:0/4232031960 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f973404e6e0 con 0x7f97340abe20 2023-03-03T11:17:40.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.433+0000 7f9722ffd700 1 -- 172.21.15.7:0/4232031960 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f972401e090 con 0x7f97340abe20 2023-03-03T11:17:40.436 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:17:40.436 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:17:40.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.435+0000 7f9738939700 1 -- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97140682f0 msgr2=0x7f971406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:40.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.435+0000 7f9738939700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97140682f0 0x7f971406a7a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f97341501d0 tx=0x7f9728008030 comp rx=0 tx=0).stop 2023-03-03T11:17:40.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.435+0000 7f9738939700 1 -- 172.21.15.7:0/4232031960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97340abe20 msgr2=0x7f97341df170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:40.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.435+0000 7f9738939700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97340abe20 0x7f97341df170 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f972400c780 tx=0x7f972400cb40 comp rx=0 tx=0).stop 2023-03-03T11:17:40.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.435+0000 7f9738939700 1 -- 172.21.15.7:0/4232031960 shutdown_connections 2023-03-03T11:17:40.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.435+0000 7f9738939700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f973414fda0 0x7f973414aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:40.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.435+0000 7f9738939700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97140682f0 0x7f971406a7a0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:40.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.435+0000 7f9738939700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97340abe20 0x7f97341df170 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:40.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.435+0000 7f9738939700 1 --2- 172.21.15.7:0/4232031960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f973414f9d0 0x7f97340c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:40.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.435+0000 7f9738939700 1 -- 172.21.15.7:0/4232031960 >> 172.21.15.7:0/4232031960 conn(0x7f97340c8900 msgr2=0x7f97340c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:17:40.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.436+0000 7f9738939700 1 -- 172.21.15.7:0/4232031960 shutdown_connections 2023-03-03T11:17:40.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:40.436+0000 7f9738939700 1 -- 172.21.15.7:0/4232031960 wait complete. 2023-03-03T11:17:40.440 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:17:40.449 INFO:tasks.mds_thrash.fs.[b]:waiting for 70 secs before thrashing 2023-03-03T11:17:41.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:41.395 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:17:52.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:17:52.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.414+0000 7f7f1e0fe700 1 Processor -- start 2023-03-03T11:17:52.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.414+0000 7f7f1e0fe700 1 -- start start 2023-03-03T11:17:52.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.415+0000 7f7f1e0fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7f18071c30 0x7f7f180c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:52.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.415+0000 7f7f1e0fe700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f180713c0 con 0x7f7f18155950 2023-03-03T11:17:52.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.415+0000 7f7f1e0fe700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f18152c70 con 0x7f7f180aac00 2023-03-03T11:17:52.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.415+0000 7f7f1e0fe700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f180c4930 con 0x7f7f18071c30 2023-03-03T11:17:52.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.415+0000 7f7f16ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7f18071c30 0x7f7f180c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:52.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.415+0000 7f7f16ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7f18071c30 0x7f7f180c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35788/0 (socket says 172.21.15.7:35788) 2023-03-03T11:17:52.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.415+0000 7f7f16ffd700 1 -- 172.21.15.7:0/1736866650 learned_addr learned my addr 172.21.15.7:0/1736866650 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:17:52.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.415+0000 7f7f17fff700 1 --1- 172.21.15.7:0/1736866650 >> v1:172.21.15.7:6789/0 conn(0x7f7f18155950 0x7f7f1815cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58002/0 (socket says 172.21.15.7:58002) 2023-03-03T11:17:52.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.416+0000 7f7f16ffd700 1 -- 172.21.15.7:0/1736866650 >> v1:172.21.15.143:6789/0 conn(0x7f7f180aac00 legacy=0x7f7f180beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:52.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.416+0000 7f7f16ffd700 1 -- 172.21.15.7:0/1736866650 >> v1:172.21.15.7:6789/0 conn(0x7f7f18155950 legacy=0x7f7f1815cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:52.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.416+0000 7f7f167fc700 1 -- 172.21.15.7:0/1736866650 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170812902 0 0) 0x7f7f180713c0 con 0x7f7f18155950 2023-03-03T11:17:52.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.416+0000 7f7f16ffd700 1 -- 172.21.15.7:0/1736866650 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f18152360 con 0x7f7f18071c30 2023-03-03T11:17:52.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.416+0000 7f7f16ffd700 1 --2- 172.21.15.7:0/1736866650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18071c30 0x7f7f180c43f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f7f080098f0 tx=0x7f7f08009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b209ba47f12574a0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:52.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.416+0000 7f7f167fc700 1 -- 172.21.15.7:0/1736866650 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800b8a0 con 0x7f7f18071c30 2023-03-03T11:17:52.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.416+0000 7f7f167fc700 1 -- 172.21.15.7:0/1736866650 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0800ba00 con 0x7f7f18071c30 2023-03-03T11:17:52.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.416+0000 7f7f167fc700 1 -- 172.21.15.7:0/1736866650 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800bd90 con 0x7f7f18071c30 2023-03-03T11:17:52.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.416+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/1736866650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18071c30 msgr2=0x7f7f180c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:52.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.416+0000 7f7f1e0fe700 1 --2- 172.21.15.7:0/1736866650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18071c30 0x7f7f180c43f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f7f080098f0 tx=0x7f7f08009c00 comp rx=0 tx=0).stop 2023-03-03T11:17:52.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.417+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/1736866650 shutdown_connections 2023-03-03T11:17:52.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.417+0000 7f7f1e0fe700 1 --2- 172.21.15.7:0/1736866650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18071c30 0x7f7f180c43f0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:52.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.417+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/1736866650 >> 172.21.15.7:0/1736866650 conn(0x7f7f180736f0 msgr2=0x7f7f18073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:17:52.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.417+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/1736866650 shutdown_connections 2023-03-03T11:17:52.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.417+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/1736866650 wait complete. 2023-03-03T11:17:52.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.418+0000 7f7f1e0fe700 1 Processor -- start 2023-03-03T11:17:52.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.418+0000 7f7f1e0fe700 1 -- start start 2023-03-03T11:17:52.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f1e0fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f18071c30 0x7f7f181ed830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:52.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f1e0fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f180aac00 0x7f7f1806bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:52.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f1e0fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18155950 0x7f7f1814a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:52.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f1e0fe700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f181ea2f0 con 0x7f7f18155950 2023-03-03T11:17:52.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f1e0fe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f181ecd50 con 0x7f7f180aac00 2023-03-03T11:17:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f1e0fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f181ed030 con 0x7f7f18071c30 2023-03-03T11:17:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f17fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18155950 0x7f7f1814a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f17fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18155950 0x7f7f1814a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35796/0 (socket says 172.21.15.7:35796) 2023-03-03T11:17:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f17fff700 1 -- 172.21.15.7:0/3041356639 learned_addr learned my addr 172.21.15.7:0/3041356639 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:17:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f16ffd700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f180aac00 0x7f7f1806bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f177fe700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f18071c30 0x7f7f181ed830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f17fff700 1 -- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f18071c30 msgr2=0x7f7f181ed830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f17fff700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f18071c30 0x7f7f181ed830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f17fff700 1 -- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f180aac00 msgr2=0x7f7f1806bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f17fff700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f180aac00 0x7f7f1806bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f17fff700 1 -- 172.21.15.7:0/3041356639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f08009550 con 0x7f7f18155950 2023-03-03T11:17:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f17fff700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18155950 0x7f7f1814a860 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000cba0 tx=0x7f7f1000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f0ffff700 1 -- 172.21.15.7:0/3041356639 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1000ab80 con 0x7f7f18155950 2023-03-03T11:17:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f0ffff700 1 -- 172.21.15.7:0/3041356639 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1000ad00 con 0x7f7f18155950 2023-03-03T11:17:52.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/3041356639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f18152620 con 0x7f7f18155950 2023-03-03T11:17:52.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.419+0000 7f7f0ffff700 1 -- 172.21.15.7:0/3041356639 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f10007310 con 0x7f7f18155950 2023-03-03T11:17:52.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.420+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/3041356639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f180c4b80 con 0x7f7f18155950 2023-03-03T11:17:52.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.420+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/3041356639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f18152900 con 0x7f7f18155950 2023-03-03T11:17:52.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.421+0000 7f7f0ffff700 1 -- 172.21.15.7:0/3041356639 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7f1001a030 con 0x7f7f18155950 2023-03-03T11:17:52.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.422+0000 7f7f0ffff700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef8068320 0x7f7ef806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:17:52.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.422+0000 7f7f0ffff700 1 -- 172.21.15.7:0/3041356639 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7f10086180 con 0x7f7f18155950 2023-03-03T11:17:52.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.422+0000 7f7f177fe700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef8068320 0x7f7ef806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:17:52.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.423+0000 7f7f177fe700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef8068320 0x7f7ef806a7d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f7f00000f60 tx=0x7f7f00009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:17:52.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.425+0000 7f7f0ffff700 1 -- 172.21.15.7:0/3041356639 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7f10052560 con 0x7f7f18155950 2023-03-03T11:17:52.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.660+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/3041356639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f18074bb0 con 0x7f7f18155950 2023-03-03T11:17:52.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.661+0000 7f7f0ffff700 1 -- 172.21.15.7:0/3041356639 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7f10018020 con 0x7f7f18155950 2023-03-03T11:17:52.664 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:17:52.664 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:17:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef8068320 msgr2=0x7f7ef806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef8068320 0x7f7ef806a7d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f7f00000f60 tx=0x7f7f00009040 comp rx=0 tx=0).stop 2023-03-03T11:17:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/3041356639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18155950 msgr2=0x7f7f1814a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:17:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18155950 0x7f7f1814a860 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000cba0 tx=0x7f7f1000cf60 comp rx=0 tx=0).stop 2023-03-03T11:17:52.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/3041356639 shutdown_connections 2023-03-03T11:17:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef8068320 0x7f7ef806a7d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f18155950 0x7f7f1814a860 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f18071c30 0x7f7f181ed830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 --2- 172.21.15.7:0/3041356639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f180aac00 0x7f7f1806bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:17:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/3041356639 >> 172.21.15.7:0/3041356639 conn(0x7f7f180736f0 msgr2=0x7f7f18152f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:17:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/3041356639 shutdown_connections 2023-03-03T11:17:52.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:17:52.663+0000 7f7f1e0fe700 1 -- 172.21.15.7:0/3041356639 wait complete. 2023-03-03T11:17:52.668 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:17:52.678 INFO:tasks.mds_thrash.fs.[a]:waiting for 94 secs before thrashing 2023-03-03T11:18:11.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:11.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:41.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:41.548 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:18:50.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:18:50.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.588+0000 7f02c8a5e700 1 Processor -- start 2023-03-03T11:18:50.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.588+0000 7f02c8a5e700 1 -- start start 2023-03-03T11:18:50.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.588+0000 7f02c8a5e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02c40c2930 0x7f02c4157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:50.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c8a5e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c406c480 0x7f02c4157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:50.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c8a5e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c406d430 con 0x7f02c4151d40 2023-03-03T11:18:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c8a5e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c406c2a0 con 0x7f02c40c2930 2023-03-03T11:18:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c8a5e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c406cb00 con 0x7f02c406c480 2023-03-03T11:18:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c1d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02c40c2930 0x7f02c4157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c1d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02c40c2930 0x7f02c4157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35832/0 (socket says 172.21.15.7:35832) 2023-03-03T11:18:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c1d9b700 1 -- 172.21.15.7:0/4188091813 learned_addr learned my addr 172.21.15.7:0/4188091813 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:18:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c2d9d700 1 --2- 172.21.15.7:0/4188091813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c406c480 0x7f02c4157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:50.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c1d9b700 1 -- 172.21.15.7:0/4188091813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c406c480 msgr2=0x7f02c4157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:18:50.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c1d9b700 1 --2- 172.21.15.7:0/4188091813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c406c480 0x7f02c4157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:50.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c1d9b700 1 -- 172.21.15.7:0/4188091813 >> v1:172.21.15.143:6789/0 conn(0x7f02c4151d40 legacy=0x7f02c40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:50.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c1d9b700 1 -- 172.21.15.7:0/4188091813 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c406cde0 con 0x7f02c40c2930 2023-03-03T11:18:50.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c2d9d700 1 --2- 172.21.15.7:0/4188091813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c406c480 0x7f02c4157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:18:50.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c1d9b700 1 --2- 172.21.15.7:0/4188091813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c40c2930 0x7f02c4157a90 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f02b80098f0 tx=0x7f02b8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80b982be23beed1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:18:50.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c159a700 1 -- 172.21.15.7:0/4188091813 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b800b8a0 con 0x7f02c40c2930 2023-03-03T11:18:50.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c159a700 1 -- 172.21.15.7:0/4188091813 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b800ba00 con 0x7f02c40c2930 2023-03-03T11:18:50.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.589+0000 7f02c159a700 1 -- 172.21.15.7:0/4188091813 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b800bd90 con 0x7f02c40c2930 2023-03-03T11:18:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.590+0000 7f02c8a5e700 1 -- 172.21.15.7:0/4188091813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c40c2930 msgr2=0x7f02c4157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.590+0000 7f02c8a5e700 1 --2- 172.21.15.7:0/4188091813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c40c2930 0x7f02c4157a90 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f02b80098f0 tx=0x7f02b8009c00 comp rx=0 tx=0).stop 2023-03-03T11:18:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.590+0000 7f02c8a5e700 1 -- 172.21.15.7:0/4188091813 shutdown_connections 2023-03-03T11:18:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.590+0000 7f02c8a5e700 1 --2- 172.21.15.7:0/4188091813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c406c480 0x7f02c4157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.590+0000 7f02c8a5e700 1 --2- 172.21.15.7:0/4188091813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c40c2930 0x7f02c4157a90 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.590+0000 7f02c8a5e700 1 -- 172.21.15.7:0/4188091813 >> 172.21.15.7:0/4188091813 conn(0x7f02c40a94d0 msgr2=0x7f02c40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:18:50.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.590+0000 7f02c8a5e700 1 -- 172.21.15.7:0/4188091813 shutdown_connections 2023-03-03T11:18:50.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.590+0000 7f02c8a5e700 1 -- 172.21.15.7:0/4188091813 wait complete. 2023-03-03T11:18:50.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.591+0000 7f02c8a5e700 1 Processor -- start 2023-03-03T11:18:50.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.591+0000 7f02c8a5e700 1 -- start start 2023-03-03T11:18:50.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c8a5e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02c406c480 0x7f02c41ddf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:50.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c8a5e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c40c2930 0x7f02c41de470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:50.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c8a5e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c4151d40 0x7f02c4148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:50.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c8a5e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40c1450 con 0x7f02c4151d40 2023-03-03T11:18:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c8a5e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c41de9b0 con 0x7f02c406c480 2023-03-03T11:18:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c8a5e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c40bfed0 con 0x7f02c40c2930 2023-03-03T11:18:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c4151d40 0x7f02c4148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c4151d40 0x7f02c4148080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35840/0 (socket says 172.21.15.7:35840) 2023-03-03T11:18:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c2d9d700 1 -- 172.21.15.7:0/3697590099 learned_addr learned my addr 172.21.15.7:0/3697590099 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:18:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c1d9b700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c40c2930 0x7f02c41de470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.592+0000 7f02c259c700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02c406c480 0x7f02c41ddf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:50.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02c2d9d700 1 -- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c40c2930 msgr2=0x7f02c41de470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02c2d9d700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c40c2930 0x7f02c41de470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02c2d9d700 1 -- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02c406c480 msgr2=0x7f02c41ddf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02c2d9d700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02c406c480 0x7f02c41ddf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02c2d9d700 1 -- 172.21.15.7:0/3697590099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b8009550 con 0x7f02c4151d40 2023-03-03T11:18:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02c259c700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02c406c480 0x7f02c41ddf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:18:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02c2d9d700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c4151d40 0x7f02c4148080 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f02bc007e70 tx=0x7f02bc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:18:50.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02b2ffd700 1 -- 172.21.15.7:0/3697590099 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc00ddd0 con 0x7f02c4151d40 2023-03-03T11:18:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02b2ffd700 1 -- 172.21.15.7:0/3697590099 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02bc00f040 con 0x7f02c4151d40 2023-03-03T11:18:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02c8a5e700 1 -- 172.21.15.7:0/3697590099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c4073e00 con 0x7f02c4151d40 2023-03-03T11:18:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02b2ffd700 1 -- 172.21.15.7:0/3697590099 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc003c60 con 0x7f02c4151d40 2023-03-03T11:18:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.593+0000 7f02c8a5e700 1 -- 172.21.15.7:0/3697590099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c406d0c0 con 0x7f02c4151d40 2023-03-03T11:18:50.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.594+0000 7f02c8a5e700 1 -- 172.21.15.7:0/3697590099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c406bea0 con 0x7f02c4151d40 2023-03-03T11:18:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.594+0000 7f02b2ffd700 1 -- 172.21.15.7:0/3697590099 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f02bc01c030 con 0x7f02c4151d40 2023-03-03T11:18:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.595+0000 7f02b2ffd700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02a40682f0 0x7f02a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:18:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.595+0000 7f02b2ffd700 1 -- 172.21.15.7:0/3697590099 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f02bc086080 con 0x7f02c4151d40 2023-03-03T11:18:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.595+0000 7f02c259c700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02a40682f0 0x7f02a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:18:50.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.596+0000 7f02c259c700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02a40682f0 0x7f02a406a7a0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f02b4000f60 tx=0x7f02b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:18:50.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.599+0000 7f02b2ffd700 1 -- 172.21.15.7:0/3697590099 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f02bc052420 con 0x7f02c4151d40 2023-03-03T11:18:50.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.835+0000 7f02c8a5e700 1 -- 172.21.15.7:0/3697590099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02c4074bb0 con 0x7f02c4151d40 2023-03-03T11:18:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.836+0000 7f02b2ffd700 1 -- 172.21.15.7:0/3697590099 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f02bc01a090 con 0x7f02c4151d40 2023-03-03T11:18:50.838 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:18:50.838 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:18:50.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 -- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02a40682f0 msgr2=0x7f02a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:50.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02a40682f0 0x7f02a406a7a0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f02b4000f60 tx=0x7f02b4009040 comp rx=0 tx=0).stop 2023-03-03T11:18:50.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 -- 172.21.15.7:0/3697590099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c4151d40 msgr2=0x7f02c4148080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:18:50.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c4151d40 0x7f02c4148080 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f02bc007e70 tx=0x7f02bc00d3b0 comp rx=0 tx=0).stop 2023-03-03T11:18:50.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 -- 172.21.15.7:0/3697590099 shutdown_connections 2023-03-03T11:18:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02a40682f0 0x7f02a406a7a0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02c4151d40 0x7f02c4148080 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02c406c480 0x7f02c41ddf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 --2- 172.21.15.7:0/3697590099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02c40c2930 0x7f02c41de470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:18:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 -- 172.21.15.7:0/3697590099 >> 172.21.15.7:0/3697590099 conn(0x7f02c40a94d0 msgr2=0x7f02c40c45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:18:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 -- 172.21.15.7:0/3697590099 shutdown_connections 2023-03-03T11:18:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:18:50.838+0000 7f02c8a5e700 1 -- 172.21.15.7:0/3697590099 wait complete. 2023-03-03T11:18:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:18:50.852 INFO:tasks.mds_thrash.fs.[b]:waiting for 15 secs before thrashing 2023-03-03T11:19:05.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:19:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.989+0000 7f075c874700 1 Processor -- start 2023-03-03T11:19:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.989+0000 7f075c874700 1 -- start start 2023-03-03T11:19:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.990+0000 7f075c874700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f075806bea0 0x7f075806c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.990+0000 7f075c874700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0758157dc0 0x7f0758158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.990+0000 7f075c874700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075806d780 con 0x7f0758151ab0 2023-03-03T11:19:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.990+0000 7f075c874700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075806c830 con 0x7f075806bea0 2023-03-03T11:19:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.990+0000 7f075c874700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075806cb70 con 0x7f0758157dc0 2023-03-03T11:19:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.990+0000 7f0755d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f075806bea0 0x7f075806c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.990+0000 7f0756d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0758157dc0 0x7f0758158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.990+0000 7f075659c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f0758151ab0 0x7f07580736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:33352/0 (socket says 172.21.15.7:33352) 2023-03-03T11:19:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.990+0000 7f075659c700 1 -- 172.21.15.7:0/3353463523 learned_addr learned my addr 172.21.15.7:0/3353463523 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:19:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f075559a700 1 -- 172.21.15.7:0/3353463523 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032378485 0 0) 0x7f075806d780 con 0x7f0758151ab0 2023-03-03T11:19:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f075559a700 1 -- 172.21.15.7:0/3353463523 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07400035b0 con 0x7f0758151ab0 2023-03-03T11:19:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f0756d9d700 1 -- 172.21.15.7:0/3353463523 >> v2:172.21.15.143:3300/0 conn(0x7f075806bea0 msgr2=0x7f075806c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:19:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f0756d9d700 1 --2- 172.21.15.7:0/3353463523 >> v2:172.21.15.143:3300/0 conn(0x7f075806bea0 0x7f075806c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f0756d9d700 1 -- 172.21.15.7:0/3353463523 >> v1:172.21.15.143:6789/0 conn(0x7f0758151ab0 legacy=0x7f07580736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f0756d9d700 1 -- 172.21.15.7:0/3353463523 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075806ce50 con 0x7f0758157dc0 2023-03-03T11:19:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f0755d9b700 1 --2- 172.21.15.7:0/3353463523 >> v2:172.21.15.143:3300/0 conn(0x7f075806bea0 0x7f075806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:19:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f0756d9d700 1 --2- 172.21.15.7:0/3353463523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0758157dc0 0x7f0758158240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0750009d90 tx=0x7f075000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1662c1a301a509a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f075559a700 1 -- 172.21.15.7:0/3353463523 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075000e070 con 0x7f0758157dc0 2023-03-03T11:19:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f075559a700 1 -- 172.21.15.7:0/3353463523 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075000bb20 con 0x7f0758157dc0 2023-03-03T11:19:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.991+0000 7f075559a700 1 -- 172.21.15.7:0/3353463523 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750012430 con 0x7f0758157dc0 2023-03-03T11:19:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.992+0000 7f075c874700 1 -- 172.21.15.7:0/3353463523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0758157dc0 msgr2=0x7f0758158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.992+0000 7f075c874700 1 --2- 172.21.15.7:0/3353463523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0758157dc0 0x7f0758158240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0750009d90 tx=0x7f075000b330 comp rx=0 tx=0).stop 2023-03-03T11:19:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.992+0000 7f075c874700 1 -- 172.21.15.7:0/3353463523 shutdown_connections 2023-03-03T11:19:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.992+0000 7f075c874700 1 --2- 172.21.15.7:0/3353463523 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0758157dc0 0x7f0758158240 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.992+0000 7f075c874700 1 --2- 172.21.15.7:0/3353463523 >> v2:172.21.15.143:3300/0 conn(0x7f075806bea0 0x7f075806c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.992+0000 7f075c874700 1 -- 172.21.15.7:0/3353463523 >> 172.21.15.7:0/3353463523 conn(0x7f07580a94d0 msgr2=0x7f07580a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:19:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.992+0000 7f075c874700 1 -- 172.21.15.7:0/3353463523 shutdown_connections 2023-03-03T11:19:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.992+0000 7f075c874700 1 -- 172.21.15.7:0/3353463523 wait complete. 2023-03-03T11:19:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.993+0000 7f075c874700 1 Processor -- start 2023-03-03T11:19:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.993+0000 7f075c874700 1 -- start start 2023-03-03T11:19:06.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.994+0000 7f075c874700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075806bea0 0x7f07580c0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:06.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.994+0000 7f075c874700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0758151ab0 0x7f075814be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:06.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.994+0000 7f075c874700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0758157dc0 0x7f075814d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:06.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.994+0000 7f075c874700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07580c07c0 con 0x7f0758157dc0 2023-03-03T11:19:06.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.994+0000 7f075c874700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07580c1380 con 0x7f0758151ab0 2023-03-03T11:19:06.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.994+0000 7f075c874700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0758153b20 con 0x7f075806bea0 2023-03-03T11:19:06.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.994+0000 7f0756d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0758157dc0 0x7f075814d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:06.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.994+0000 7f0756d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0758157dc0 0x7f075814d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35862/0 (socket says 172.21.15.7:35862) 2023-03-03T11:19:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.994+0000 7f0756d9d700 1 -- 172.21.15.7:0/1451744190 learned_addr learned my addr 172.21.15.7:0/1451744190 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:19:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.994+0000 7f0755d9b700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0758151ab0 0x7f075814be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f075659c700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075806bea0 0x7f07580c0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f0756d9d700 1 -- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075806bea0 msgr2=0x7f07580c0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f0756d9d700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075806bea0 0x7f07580c0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f0756d9d700 1 -- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0758151ab0 msgr2=0x7f075814be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f0756d9d700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0758151ab0 0x7f075814be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:06.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f0756d9d700 1 -- 172.21.15.7:0/1451744190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07480094a0 con 0x7f0758157dc0 2023-03-03T11:19:06.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f075659c700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075806bea0 0x7f07580c0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:19:06.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f0755d9b700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0758151ab0 0x7f075814be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:19:06.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f0756d9d700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0758157dc0 0x7f075814d9b0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f075000b680 tx=0x7f0750004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:06.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f0746ffd700 1 -- 172.21.15.7:0/1451744190 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750012a70 con 0x7f0758157dc0 2023-03-03T11:19:06.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f0746ffd700 1 -- 172.21.15.7:0/1451744190 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0750020070 con 0x7f0758157dc0 2023-03-03T11:19:06.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f075c874700 1 -- 172.21.15.7:0/1451744190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0750009a70 con 0x7f0758157dc0 2023-03-03T11:19:06.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f0746ffd700 1 -- 172.21.15.7:0/1451744190 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750012e30 con 0x7f0758157dc0 2023-03-03T11:19:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.995+0000 7f075c874700 1 -- 172.21.15.7:0/1451744190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075806d410 con 0x7f0758157dc0 2023-03-03T11:19:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.996+0000 7f075c874700 1 -- 172.21.15.7:0/1451744190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075806d130 con 0x7f0758157dc0 2023-03-03T11:19:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.996+0000 7f0746ffd700 1 -- 172.21.15.7:0/1451744190 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f075002e070 con 0x7f0758157dc0 2023-03-03T11:19:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.997+0000 7f0746ffd700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 0x7f073806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.997+0000 7f0746ffd700 1 -- 172.21.15.7:0/1451744190 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0750022070 con 0x7f0758157dc0 2023-03-03T11:19:06.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.997+0000 7f075659c700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 0x7f073806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:06.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:05.998+0000 7f075659c700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 0x7f073806a7a0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f074c0011e0 tx=0x7f074c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:06.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.001+0000 7f0746ffd700 1 -- 172.21.15.7:0/1451744190 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f075008b030 con 0x7f0758157dc0 2023-03-03T11:19:06.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.238+0000 7f075c874700 1 -- 172.21.15.7:0/1451744190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0758073e00 con 0x7f0758157dc0 2023-03-03T11:19:06.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.238+0000 7f0746ffd700 1 -- 172.21.15.7:0/1451744190 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f075002c090 con 0x7f0758157dc0 2023-03-03T11:19:06.242 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:19:06.242 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:19:06.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 -- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 msgr2=0x7f073806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:06.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 0x7f073806a7a0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f074c0011e0 tx=0x7f074c009040 comp rx=0 tx=0).stop 2023-03-03T11:19:06.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 -- 172.21.15.7:0/1451744190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0758157dc0 msgr2=0x7f075814d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:06.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0758157dc0 0x7f075814d9b0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f075000b680 tx=0x7f0750004ad0 comp rx=0 tx=0).stop 2023-03-03T11:19:06.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 -- 172.21.15.7:0/1451744190 shutdown_connections 2023-03-03T11:19:06.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 0x7f073806a7a0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:06.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0758157dc0 0x7f075814d9b0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:06.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075806bea0 0x7f07580c0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:06.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 --2- 172.21.15.7:0/1451744190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0758151ab0 0x7f075814be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:06.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 -- 172.21.15.7:0/1451744190 >> 172.21.15.7:0/1451744190 conn(0x7f07580a94d0 msgr2=0x7f07581581c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:19:06.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 -- 172.21.15.7:0/1451744190 shutdown_connections 2023-03-03T11:19:06.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:06.240+0000 7f075c874700 1 -- 172.21.15.7:0/1451744190 wait complete. 2023-03-03T11:19:06.245 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:19:06.255 INFO:tasks.mds_thrash.fs.[b]:waiting for 23 secs before thrashing 2023-03-03T11:19:11.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:11.669 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:26.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:19:26.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.815+0000 7f294693e700 1 Processor -- start 2023-03-03T11:19:26.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.815+0000 7f294693e700 1 -- start start 2023-03-03T11:19:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f294693e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2940151b20 0x7f29400736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f294693e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2940073c30 0x7f29401581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f294693e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294006cb70 con 0x7f29400c2950 2023-03-03T11:19:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f294693e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294006c2f0 con 0x7f2940151b20 2023-03-03T11:19:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f294693e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294006ce50 con 0x7f2940073c30 2023-03-03T11:19:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f293ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2940151b20 0x7f29400736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f293ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2940151b20 0x7f29400736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35872/0 (socket says 172.21.15.7:35872) 2023-03-03T11:19:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f293ffff700 1 -- 172.21.15.7:0/1697655946 learned_addr learned my addr 172.21.15.7:0/1697655946 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:19:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f2944edb700 1 --2- 172.21.15.7:0/1697655946 >> v2:172.21.15.143:3300/0 conn(0x7f2940073c30 0x7f29401581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f293ffff700 1 -- 172.21.15.7:0/1697655946 >> v2:172.21.15.143:3300/0 conn(0x7f2940073c30 msgr2=0x7f29401581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:19:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f293ffff700 1 --2- 172.21.15.7:0/1697655946 >> v2:172.21.15.143:3300/0 conn(0x7f2940073c30 0x7f29401581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:26.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f293ffff700 1 -- 172.21.15.7:0/1697655946 >> v1:172.21.15.143:6789/0 conn(0x7f29400c2950 legacy=0x7f2940157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:19:26.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f293ffff700 1 -- 172.21.15.7:0/1697655946 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294006d130 con 0x7f2940151b20 2023-03-03T11:19:26.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.816+0000 7f293ffff700 1 --2- 172.21.15.7:0/1697655946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940151b20 0x7f29400736f0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f294006e380 tx=0x7f2930009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28bb3d0a965cb19d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:26.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f293effd700 1 -- 172.21.15.7:0/1697655946 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930004230 con 0x7f2940151b20 2023-03-03T11:19:26.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f293effd700 1 -- 172.21.15.7:0/1697655946 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2930004390 con 0x7f2940151b20 2023-03-03T11:19:26.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f293effd700 1 -- 172.21.15.7:0/1697655946 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29300036a0 con 0x7f2940151b20 2023-03-03T11:19:26.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f294693e700 1 -- 172.21.15.7:0/1697655946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940151b20 msgr2=0x7f29400736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:26.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f294693e700 1 --2- 172.21.15.7:0/1697655946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940151b20 0x7f29400736f0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f294006e380 tx=0x7f2930009bd0 comp rx=0 tx=0).stop 2023-03-03T11:19:26.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f294693e700 1 -- 172.21.15.7:0/1697655946 shutdown_connections 2023-03-03T11:19:26.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f294693e700 1 --2- 172.21.15.7:0/1697655946 >> v2:172.21.15.143:3300/0 conn(0x7f2940073c30 0x7f29401581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:26.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f294693e700 1 --2- 172.21.15.7:0/1697655946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2940151b20 0x7f29400736f0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:26.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f294693e700 1 -- 172.21.15.7:0/1697655946 >> 172.21.15.7:0/1697655946 conn(0x7f29400a94d0 msgr2=0x7f29400a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:19:26.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f294693e700 1 -- 172.21.15.7:0/1697655946 shutdown_connections 2023-03-03T11:19:26.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.817+0000 7f294693e700 1 -- 172.21.15.7:0/1697655946 wait complete. 2023-03-03T11:19:26.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.819+0000 7f294693e700 1 Processor -- start 2023-03-03T11:19:26.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.819+0000 7f294693e700 1 -- start start 2023-03-03T11:19:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f294693e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2940073c30 0x7f294014de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f294693e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c2950 0x7f29401543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f294693e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2940151b20 0x7f2940154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f294693e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400c04b0 con 0x7f29400c2950 2023-03-03T11:19:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f294693e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400c1560 con 0x7f2940151b20 2023-03-03T11:19:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f294693e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29401eae60 con 0x7f2940073c30 2023-03-03T11:19:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f293f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c2950 0x7f29401543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f293f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c2950 0x7f29401543e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35878/0 (socket says 172.21.15.7:35878) 2023-03-03T11:19:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f293f7fe700 1 -- 172.21.15.7:0/1545252145 learned_addr learned my addr 172.21.15.7:0/1545252145 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:19:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f2944edb700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2940151b20 0x7f2940154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f293ffff700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2940073c30 0x7f294014de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f293f7fe700 1 -- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2940073c30 msgr2=0x7f294014de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f293f7fe700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2940073c30 0x7f294014de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f293f7fe700 1 -- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2940151b20 msgr2=0x7f2940154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f293f7fe700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2940151b20 0x7f2940154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.820+0000 7f293f7fe700 1 -- 172.21.15.7:0/1545252145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2930009550 con 0x7f29400c2950 2023-03-03T11:19:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.821+0000 7f293f7fe700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c2950 0x7f29401543e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f292800b4a0 tx=0x7f292800b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.821+0000 7f293cff9700 1 -- 172.21.15.7:0/1545252145 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928011830 con 0x7f29400c2950 2023-03-03T11:19:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.821+0000 7f293cff9700 1 -- 172.21.15.7:0/1545252145 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2928011990 con 0x7f29400c2950 2023-03-03T11:19:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.821+0000 7f294693e700 1 -- 172.21.15.7:0/1545252145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2940158730 con 0x7f29400c2950 2023-03-03T11:19:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.821+0000 7f293cff9700 1 -- 172.21.15.7:0/1545252145 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928011d40 con 0x7f29400c2950 2023-03-03T11:19:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.821+0000 7f294693e700 1 -- 172.21.15.7:0/1545252145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294006d410 con 0x7f29400c2950 2023-03-03T11:19:26.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.822+0000 7f294693e700 1 -- 172.21.15.7:0/1545252145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2940151ef0 con 0x7f29400c2950 2023-03-03T11:19:26.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.823+0000 7f293cff9700 1 -- 172.21.15.7:0/1545252145 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f292800f050 con 0x7f29400c2950 2023-03-03T11:19:26.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.823+0000 7f293cff9700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2920068320 0x7f292006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:26.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.823+0000 7f293cff9700 1 -- 172.21.15.7:0/1545252145 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2928085f90 con 0x7f29400c2950 2023-03-03T11:19:26.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.823+0000 7f293ffff700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2920068320 0x7f292006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:26.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.824+0000 7f293ffff700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2920068320 0x7f292006a7d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2930009130 tx=0x7f2930008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:26.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:26.827+0000 7f293cff9700 1 -- 172.21.15.7:0/1545252145 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f29280523d0 con 0x7f29400c2950 2023-03-03T11:19:27.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.069+0000 7f294693e700 1 -- 172.21.15.7:0/1545252145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2940074bb0 con 0x7f29400c2950 2023-03-03T11:19:27.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.070+0000 7f293cff9700 1 -- 172.21.15.7:0/1545252145 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f29280559f0 con 0x7f29400c2950 2023-03-03T11:19:27.072 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:19:27.072 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:19:27.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 -- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2920068320 msgr2=0x7f292006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:27.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2920068320 0x7f292006a7d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2930009130 tx=0x7f2930008030 comp rx=0 tx=0).stop 2023-03-03T11:19:27.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 -- 172.21.15.7:0/1545252145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c2950 msgr2=0x7f29401543e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:27.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c2950 0x7f29401543e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f292800b4a0 tx=0x7f292800b7b0 comp rx=0 tx=0).stop 2023-03-03T11:19:27.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 -- 172.21.15.7:0/1545252145 shutdown_connections 2023-03-03T11:19:27.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2940151b20 0x7f2940154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:27.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2940073c30 0x7f294014de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:27.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2920068320 0x7f292006a7d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:27.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 --2- 172.21.15.7:0/1545252145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29400c2950 0x7f29401543e0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:27.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 -- 172.21.15.7:0/1545252145 >> 172.21.15.7:0/1545252145 conn(0x7f29400a94d0 msgr2=0x7f2940150190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:19:27.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 -- 172.21.15.7:0/1545252145 shutdown_connections 2023-03-03T11:19:27.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:27.072+0000 7f294693e700 1 -- 172.21.15.7:0/1545252145 wait complete. 2023-03-03T11:19:27.076 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:19:27.086 INFO:tasks.mds_thrash.fs.[a]:waiting for 116 secs before thrashing 2023-03-03T11:19:29.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:19:29.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.393+0000 7fef95d5d700 1 Processor -- start 2023-03-03T11:19:29.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.393+0000 7fef95d5d700 1 -- start start 2023-03-03T11:19:29.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef95d5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef900aac00 0x7fef900c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:29.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef95d5d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef90155950 0x7fef90155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:29.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef95d5d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef900713c0 con 0x7fef90071c30 2023-03-03T11:19:29.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef95d5d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef90156310 con 0x7fef90155950 2023-03-03T11:19:29.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef95d5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef90152c70 con 0x7fef900aac00 2023-03-03T11:19:29.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fef90071c30 0x7fef900beb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58098/0 (socket says 172.21.15.7:58098) 2023-03-03T11:19:29.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8effd700 1 -- 172.21.15.7:0/4294648899 learned_addr learned my addr 172.21.15.7:0/4294648899 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:19:29.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8f7fe700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef900aac00 0x7fef900c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:29.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8ffff700 1 --2- 172.21.15.7:0/4294648899 >> v2:172.21.15.143:3300/0 conn(0x7fef90155950 0x7fef90155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:29.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107256808 0 0) 0x7fef900713c0 con 0x7fef90071c30 2023-03-03T11:19:29.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef780035b0 con 0x7fef90071c30 2023-03-03T11:19:29.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2964902984 0 0) 0x7fef780035b0 con 0x7fef90071c30 2023-03-03T11:19:29.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef900aac00 msgr2=0x7fef900c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:19:29.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef900aac00 0x7fef900c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 >> v2:172.21.15.143:3300/0 conn(0x7fef90155950 msgr2=0x7fef90155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:19:29.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 --2- 172.21.15.7:0/4294648899 >> v2:172.21.15.143:3300/0 conn(0x7fef90155950 0x7fef90155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef900c4930 con 0x7fef90071c30 2023-03-03T11:19:29.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fef84001670 con 0x7fef90071c30 2023-03-03T11:19:29.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 >> v1:172.21.15.7:6789/0 conn(0x7fef90071c30 legacy=0x7fef900beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:29.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 0x7fef78003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:29.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef78004850 0x7fef78004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:29.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef900aac00 0x7fef78005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:29.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef90156310 con 0x7fef78003a70 2023-03-03T11:19:29.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef90152c70 con 0x7fef78004850 2023-03-03T11:19:29.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.394+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7800ba40 con 0x7fef900aac00 2023-03-03T11:19:29.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8ffff700 1 --2- 172.21.15.7:0/4294648899 >> v2:172.21.15.143:3300/0 conn(0x7fef90155950 0x7fef90155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:19:29.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8f7fe700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 0x7fef78003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:29.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8effd700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef78004850 0x7fef78004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:29.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8ffff700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef900aac00 0x7fef78005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:29.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8f7fe700 1 -- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef900aac00 msgr2=0x7fef78005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:19:29.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8f7fe700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef900aac00 0x7fef78005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8f7fe700 1 -- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef78004850 msgr2=0x7fef78004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:19:29.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8f7fe700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef78004850 0x7fef78004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8f7fe700 1 -- 172.21.15.7:0/4294648899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef780035b0 con 0x7fef78003a70 2023-03-03T11:19:29.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8f7fe700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 0x7fef78003e80 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fef80009b60 tx=0x7fef78005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78abf3e8dfc139e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:29.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8000e6c0 con 0x7fef78003a70 2023-03-03T11:19:29.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8000e820 con 0x7fef78003a70 2023-03-03T11:19:29.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.395+0000 7fef8e7fc700 1 -- 172.21.15.7:0/4294648899 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8000eb90 con 0x7fef78003a70 2023-03-03T11:19:29.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.396+0000 7fef95d5d700 1 -- 172.21.15.7:0/4294648899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 msgr2=0x7fef78003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:29.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.396+0000 7fef95d5d700 1 --2- 172.21.15.7:0/4294648899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 0x7fef78003e80 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fef80009b60 tx=0x7fef78005880 comp rx=0 tx=0).stop 2023-03-03T11:19:29.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.396+0000 7fef8ffff700 1 -- 172.21.15.7:0/4294648899 reap_dead start 2023-03-03T11:19:29.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.396+0000 7fef95d5d700 1 -- 172.21.15.7:0/4294648899 shutdown_connections 2023-03-03T11:19:29.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.396+0000 7fef95d5d700 1 -- 172.21.15.7:0/4294648899 >> 172.21.15.7:0/4294648899 conn(0x7fef900736f0 msgr2=0x7fef90073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:19:29.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.396+0000 7fef95d5d700 1 -- 172.21.15.7:0/4294648899 shutdown_connections 2023-03-03T11:19:29.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.396+0000 7fef95d5d700 1 -- 172.21.15.7:0/4294648899 wait complete. 2023-03-03T11:19:29.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.397+0000 7fef95d5d700 1 Processor -- start 2023-03-03T11:19:29.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.397+0000 7fef95d5d700 1 -- start start 2023-03-03T11:19:29.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef95d5d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 0x7fef900ad900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:29.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef95d5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef901eb190 0x7fef9014a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:29.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef95d5d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef900c78f0 0x7fef900ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:29.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef95d5d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef901551a0 con 0x7fef78003a70 2023-03-03T11:19:29.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef95d5d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef90154520 con 0x7fef900c78f0 2023-03-03T11:19:29.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef95d5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef90154800 con 0x7fef901eb190 2023-03-03T11:19:29.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 0x7fef900ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:29.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 0x7fef900ad900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35898/0 (socket says 172.21.15.7:35898) 2023-03-03T11:19:29.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8f7fe700 1 -- 172.21.15.7:0/1523338748 learned_addr learned my addr 172.21.15.7:0/1523338748 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:19:29.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8effd700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef901eb190 0x7fef9014a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:29.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8ffff700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef900c78f0 0x7fef900ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:29.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8f7fe700 1 -- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef901eb190 msgr2=0x7fef9014a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:29.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8f7fe700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef901eb190 0x7fef9014a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8f7fe700 1 -- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef900c78f0 msgr2=0x7fef900ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:29.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8f7fe700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef900c78f0 0x7fef900ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8f7fe700 1 -- 172.21.15.7:0/1523338748 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef900c4b80 con 0x7fef78003a70 2023-03-03T11:19:29.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.398+0000 7fef8ffff700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef900c78f0 0x7fef900ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:19:29.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.399+0000 7fef8f7fe700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 0x7fef900ad900 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fef78004780 tx=0x7fef800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:29.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.399+0000 7fef77fff700 1 -- 172.21.15.7:0/1523338748 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80021070 con 0x7fef78003a70 2023-03-03T11:19:29.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.399+0000 7fef77fff700 1 -- 172.21.15.7:0/1523338748 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef800045d0 con 0x7fef78003a70 2023-03-03T11:19:29.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.399+0000 7fef95d5d700 1 -- 172.21.15.7:0/1523338748 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef90152360 con 0x7fef78003a70 2023-03-03T11:19:29.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.399+0000 7fef77fff700 1 -- 172.21.15.7:0/1523338748 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8000e6c0 con 0x7fef78003a70 2023-03-03T11:19:29.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.399+0000 7fef95d5d700 1 -- 172.21.15.7:0/1523338748 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef90152620 con 0x7fef78003a70 2023-03-03T11:19:29.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.400+0000 7fef95d5d700 1 -- 172.21.15.7:0/1523338748 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef90074570 con 0x7fef78003a70 2023-03-03T11:19:29.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.400+0000 7fef77fff700 1 -- 172.21.15.7:0/1523338748 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fef8002e030 con 0x7fef78003a70 2023-03-03T11:19:29.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.401+0000 7fef77fff700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef7c0682f0 0x7fef7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:19:29.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.401+0000 7fef77fff700 1 -- 172.21.15.7:0/1523338748 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fef80087510 con 0x7fef78003a70 2023-03-03T11:19:29.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.401+0000 7fef8effd700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef7c0682f0 0x7fef7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:19:29.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.401+0000 7fef8effd700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef7c0682f0 0x7fef7c06a7a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fef901eb580 tx=0x7fef84009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:19:29.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.405+0000 7fef77fff700 1 -- 172.21.15.7:0/1523338748 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fef800538b0 con 0x7fef78003a70 2023-03-03T11:19:29.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.649+0000 7fef95d5d700 1 -- 172.21.15.7:0/1523338748 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef90071a10 con 0x7fef78003a70 2023-03-03T11:19:29.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.649+0000 7fef77fff700 1 -- 172.21.15.7:0/1523338748 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fef80056ed0 con 0x7fef78003a70 2023-03-03T11:19:29.651 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:19:29.652 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:19:29.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 -- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef7c0682f0 msgr2=0x7fef7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:29.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef7c0682f0 0x7fef7c06a7a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fef901eb580 tx=0x7fef84009040 comp rx=0 tx=0).stop 2023-03-03T11:19:29.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 -- 172.21.15.7:0/1523338748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 msgr2=0x7fef900ad900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:19:29.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 0x7fef900ad900 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fef78004780 tx=0x7fef800047c0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 -- 172.21.15.7:0/1523338748 shutdown_connections 2023-03-03T11:19:29.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef900c78f0 0x7fef900ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef7c0682f0 0x7fef7c06a7a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef78003a70 0x7fef900ad900 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 --2- 172.21.15.7:0/1523338748 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef901eb190 0x7fef9014a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:19:29.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 -- 172.21.15.7:0/1523338748 >> 172.21.15.7:0/1523338748 conn(0x7fef900736f0 msgr2=0x7fef90073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:19:29.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 -- 172.21.15.7:0/1523338748 shutdown_connections 2023-03-03T11:19:29.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:19:29.651+0000 7fef95d5d700 1 -- 172.21.15.7:0/1523338748 wait complete. 2023-03-03T11:19:29.655 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:19:29.665 INFO:tasks.mds_thrash.fs.[b]:waiting for 115 secs before thrashing 2023-03-03T11:19:41.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:19:41.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T11:19:59.998+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T11:20:11.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:11.887 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:41.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:20:41.983 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:12.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:12.098 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:23.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:21:23.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.222+0000 7fc5c04d2700 1 Processor -- start 2023-03-03T11:21:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.222+0000 7fc5c04d2700 1 -- start start 2023-03-03T11:21:23.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.223+0000 7fc5c04d2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc5b814f990 0x7fc5b8152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:23.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.223+0000 7fc5c04d2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80ab5e0 0x7fc5b80aba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.223+0000 7fc5c04d2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b814fd60 con 0x7fc5b81562b0 2023-03-03T11:21:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.223+0000 7fc5c04d2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b806c030 con 0x7fc5b814f990 2023-03-03T11:21:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.223+0000 7fc5c04d2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c29c0 con 0x7fc5b80ab5e0 2023-03-03T11:21:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bda6d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc5b81562b0 0x7fc5b806b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58188/0 (socket says 172.21.15.7:58188) 2023-03-03T11:21:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bda6d700 1 -- 172.21.15.7:0/2956777942 learned_addr learned my addr 172.21.15.7:0/2956777942 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:21:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bea6f700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80ab5e0 0x7fc5b80aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5be26e700 1 --2- 172.21.15.7:0/2956777942 >> v2:172.21.15.143:3300/0 conn(0x7fc5b814f990 0x7fc5b8152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456543930 0 0) 0x7fc5b814fd60 con 0x7fc5b81562b0 2023-03-03T11:21:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a40035b0 con 0x7fc5b81562b0 2023-03-03T11:21:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868371950 0 0) 0x7fc5a40035b0 con 0x7fc5b81562b0 2023-03-03T11:21:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80ab5e0 msgr2=0x7fc5b80aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:21:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80ab5e0 0x7fc5b80aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 >> v2:172.21.15.143:3300/0 conn(0x7fc5b814f990 msgr2=0x7fc5b8152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:21:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 --2- 172.21.15.7:0/2956777942 >> v2:172.21.15.143:3300/0 conn(0x7fc5b814f990 0x7fc5b8152ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80c2400 con 0x7fc5b81562b0 2023-03-03T11:21:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc5ac001670 con 0x7fc5b81562b0 2023-03-03T11:21:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 >> v1:172.21.15.7:6789/0 conn(0x7fc5b81562b0 legacy=0x7fc5b806b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5a4003a70 0x7fc5a4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5a4004850 0x7fc5a4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80ab5e0 0x7fc5a4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b806c030 con 0x7fc5a4003a70 2023-03-03T11:21:23.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80c29c0 con 0x7fc5a4004850 2023-03-03T11:21:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a400bb40 con 0x7fc5b80ab5e0 2023-03-03T11:21:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5be26e700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5a4003a70 0x7fc5a4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.224+0000 7fc5bda6d700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5a4004850 0x7fc5a4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5bea6f700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80ab5e0 0x7fc5a4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5be26e700 1 -- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80ab5e0 msgr2=0x7fc5a4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:21:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5be26e700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80ab5e0 0x7fc5a4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5be26e700 1 -- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5a4004850 msgr2=0x7fc5a4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:21:23.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5be26e700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5a4004850 0x7fc5a4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5be26e700 1 -- 172.21.15.7:0/2956777942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a40035b0 con 0x7fc5a4003a70 2023-03-03T11:21:23.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5bea6f700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5b80ab5e0 0x7fc5a4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:21:23.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5be26e700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5a4003a70 0x7fc5a4003e80 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009b60 tx=0x7fc5a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12e21c0929eaa33 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:23.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400e6c0 con 0x7fc5a4003a70 2023-03-03T11:21:23.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b400e820 con 0x7fc5a4003a70 2023-03-03T11:21:23.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5bd26c700 1 -- 172.21.15.7:0/2956777942 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400eb90 con 0x7fc5a4003a70 2023-03-03T11:21:23.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5c04d2700 1 -- 172.21.15.7:0/2956777942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5a4003a70 msgr2=0x7fc5a4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:23.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5c04d2700 1 --2- 172.21.15.7:0/2956777942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5a4003a70 0x7fc5a4003e80 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009b60 tx=0x7fc5a4005880 comp rx=0 tx=0).stop 2023-03-03T11:21:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.225+0000 7fc5bea6f700 1 -- 172.21.15.7:0/2956777942 reap_dead start 2023-03-03T11:21:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.226+0000 7fc5c04d2700 1 -- 172.21.15.7:0/2956777942 shutdown_connections 2023-03-03T11:21:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.226+0000 7fc5c04d2700 1 -- 172.21.15.7:0/2956777942 >> 172.21.15.7:0/2956777942 conn(0x7fc5b80c62e0 msgr2=0x7fc5b80c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:21:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.226+0000 7fc5c04d2700 1 -- 172.21.15.7:0/2956777942 shutdown_connections 2023-03-03T11:21:23.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.226+0000 7fc5c04d2700 1 -- 172.21.15.7:0/2956777942 wait complete. 2023-03-03T11:21:23.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.227+0000 7fc5c04d2700 1 Processor -- start 2023-03-03T11:21:23.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.227+0000 7fc5c04d2700 1 -- start start 2023-03-03T11:21:23.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5c04d2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5a4003a70 0x7fc5b8151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:23.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5c04d2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b81e75b0 0x7fc5b8147130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5c04d2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b8147670 0x7fc5b8149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5c04d2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b814bce0 con 0x7fc5b8147670 2023-03-03T11:21:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5c04d2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b8147ae0 con 0x7fc5b81e75b0 2023-03-03T11:21:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5c04d2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b81489b0 con 0x7fc5a4003a70 2023-03-03T11:21:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5bea6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b8147670 0x7fc5b8149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5bea6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b8147670 0x7fc5b8149c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35990/0 (socket says 172.21.15.7:35990) 2023-03-03T11:21:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5bea6f700 1 -- 172.21.15.7:0/3080333617 learned_addr learned my addr 172.21.15.7:0/3080333617 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:21:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5bda6d700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b81e75b0 0x7fc5b8147130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.228+0000 7fc5be26e700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5a4003a70 0x7fc5b8151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5bea6f700 1 -- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5a4003a70 msgr2=0x7fc5b8151380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5bea6f700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5a4003a70 0x7fc5b8151380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5bea6f700 1 -- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b81e75b0 msgr2=0x7fc5b8147130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5bea6f700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b81e75b0 0x7fc5b8147130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5bea6f700 1 -- 172.21.15.7:0/3080333617 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b81523f0 con 0x7fc5b8147670 2023-03-03T11:21:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5bea6f700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b8147670 0x7fc5b8149c30 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fc5b806ad90 tx=0x7fc5b00104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5aaffd700 1 -- 172.21.15.7:0/3080333617 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000e050 con 0x7fc5b8147670 2023-03-03T11:21:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5aaffd700 1 -- 172.21.15.7:0/3080333617 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b000f040 con 0x7fc5b8147670 2023-03-03T11:21:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5c04d2700 1 -- 172.21.15.7:0/3080333617 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b80c26e0 con 0x7fc5b8147670 2023-03-03T11:21:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5aaffd700 1 -- 172.21.15.7:0/3080333617 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0010d40 con 0x7fc5b8147670 2023-03-03T11:21:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.229+0000 7fc5c04d2700 1 -- 172.21.15.7:0/3080333617 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b81526d0 con 0x7fc5b8147670 2023-03-03T11:21:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.230+0000 7fc5c04d2700 1 -- 172.21.15.7:0/3080333617 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b81dbf70 con 0x7fc5b8147670 2023-03-03T11:21:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.231+0000 7fc5aaffd700 1 -- 172.21.15.7:0/3080333617 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc5b0003a20 con 0x7fc5b8147670 2023-03-03T11:21:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.231+0000 7fc5aaffd700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc59c0682f0 0x7fc59c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:23.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.231+0000 7fc5aaffd700 1 -- 172.21.15.7:0/3080333617 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc5b0086990 con 0x7fc5b8147670 2023-03-03T11:21:23.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.231+0000 7fc5be26e700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc59c0682f0 0x7fc59c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:23.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.232+0000 7fc5be26e700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc59c0682f0 0x7fc59c06a7a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400a010 tx=0x7fc5b4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:23.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.235+0000 7fc5aaffd700 1 -- 172.21.15.7:0/3080333617 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5b0052d30 con 0x7fc5b8147670 2023-03-03T11:21:23.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.472+0000 7fc5c04d2700 1 -- 172.21.15.7:0/3080333617 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5b8074bb0 con 0x7fc5b8147670 2023-03-03T11:21:23.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.472+0000 7fc5aaffd700 1 -- 172.21.15.7:0/3080333617 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc5b0010ea0 con 0x7fc5b8147670 2023-03-03T11:21:23.475 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:21:23.475 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:21:23.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 -- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc59c0682f0 msgr2=0x7fc59c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:23.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc59c0682f0 0x7fc59c06a7a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400a010 tx=0x7fc5b4008030 comp rx=0 tx=0).stop 2023-03-03T11:21:23.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 -- 172.21.15.7:0/3080333617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b8147670 msgr2=0x7fc5b8149c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:23.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b8147670 0x7fc5b8149c30 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fc5b806ad90 tx=0x7fc5b00104a0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 -- 172.21.15.7:0/3080333617 shutdown_connections 2023-03-03T11:21:23.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc59c0682f0 0x7fc59c06a7a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5b8147670 0x7fc5b8149c30 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5a4003a70 0x7fc5b8151380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 --2- 172.21.15.7:0/3080333617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5b81e75b0 0x7fc5b8147130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:23.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 -- 172.21.15.7:0/3080333617 >> 172.21.15.7:0/3080333617 conn(0x7fc5b80c62e0 msgr2=0x7fc5b80aba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:21:23.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 -- 172.21.15.7:0/3080333617 shutdown_connections 2023-03-03T11:21:23.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:23.475+0000 7fc5c04d2700 1 -- 172.21.15.7:0/3080333617 wait complete. 2023-03-03T11:21:23.479 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:21:23.488 INFO:tasks.mds_thrash.fs.[a]:waiting for 52 secs before thrashing 2023-03-03T11:21:24.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:21:24.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.803+0000 7f1b30888700 1 Processor -- start 2023-03-03T11:21:24.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.803+0000 7f1b30888700 1 -- start start 2023-03-03T11:21:24.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.804+0000 7f1b30888700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1b2c072250 0x7f1b2c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:24.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.804+0000 7f1b30888700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c1510d0 con 0x7f1b2c0c1ce0 2023-03-03T11:21:24.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.804+0000 7f1b30888700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c0c3740 con 0x7f1b2c071e80 2023-03-03T11:21:24.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.804+0000 7f1b30888700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2c070670 con 0x7f1b2c072250 2023-03-03T11:21:24.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.804+0000 7f1b2ad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1b2c0c1ce0 0x7f1b2c15ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58210/0 (socket says 172.21.15.7:58210) 2023-03-03T11:21:24.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.804+0000 7f1b2ad9d700 1 -- 172.21.15.7:0/1995680208 learned_addr learned my addr 172.21.15.7:0/1995680208 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:21:24.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.804+0000 7f1b29d9b700 1 --2- 172.21.15.7:0/1995680208 >> v2:172.21.15.143:3300/0 conn(0x7f1b2c072250 0x7f1b2c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:24.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.804+0000 7f1b2959a700 1 -- 172.21.15.7:0/1995680208 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611614582 0 0) 0x7f1b2c1510d0 con 0x7f1b2c0c1ce0 2023-03-03T11:21:24.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.804+0000 7f1b2959a700 1 -- 172.21.15.7:0/1995680208 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b140035b0 con 0x7f1b2c0c1ce0 2023-03-03T11:21:24.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.804+0000 7f1b2959a700 1 -- 172.21.15.7:0/1995680208 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423603122 0 0) 0x7f1b2c0c3740 con 0x7f1b2c071e80 2023-03-03T11:21:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.805+0000 7f1b2959a700 1 -- 172.21.15.7:0/1995680208 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b2c1510d0 con 0x7f1b2c071e80 2023-03-03T11:21:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.805+0000 7f1b2959a700 1 -- 172.21.15.7:0/1995680208 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1234086888 0 0) 0x7f1b140035b0 con 0x7f1b2c0c1ce0 2023-03-03T11:21:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.805+0000 7f1b29d9b700 1 -- 172.21.15.7:0/1995680208 >> v1:172.21.15.143:6789/0 conn(0x7f1b2c071e80 legacy=0x7f1b2c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.805+0000 7f1b29d9b700 1 -- 172.21.15.7:0/1995680208 >> v1:172.21.15.7:6789/0 conn(0x7f1b2c0c1ce0 legacy=0x7f1b2c15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.805+0000 7f1b29d9b700 1 -- 172.21.15.7:0/1995680208 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c0c39e0 con 0x7f1b2c072250 2023-03-03T11:21:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.805+0000 7f1b29d9b700 1 --2- 172.21.15.7:0/1995680208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b2c072250 0x7f1b2c14e1f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1b20009870 tx=0x7f1b20009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96cc08397f323358 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:24.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.805+0000 7f1b2959a700 1 -- 172.21.15.7:0/1995680208 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240019f0 con 0x7f1b2c072250 2023-03-03T11:21:24.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.805+0000 7f1b2959a700 1 -- 172.21.15.7:0/1995680208 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b20004030 con 0x7f1b2c072250 2023-03-03T11:21:24.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.805+0000 7f1b2959a700 1 -- 172.21.15.7:0/1995680208 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b200043c0 con 0x7f1b2c072250 2023-03-03T11:21:24.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.806+0000 7f1b30888700 1 -- 172.21.15.7:0/1995680208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b2c072250 msgr2=0x7f1b2c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:24.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.806+0000 7f1b30888700 1 --2- 172.21.15.7:0/1995680208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b2c072250 0x7f1b2c14e1f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1b20009870 tx=0x7f1b20009b80 comp rx=0 tx=0).stop 2023-03-03T11:21:24.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.806+0000 7f1b30888700 1 -- 172.21.15.7:0/1995680208 shutdown_connections 2023-03-03T11:21:24.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.806+0000 7f1b30888700 1 --2- 172.21.15.7:0/1995680208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b2c072250 0x7f1b2c14e1f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:24.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.806+0000 7f1b30888700 1 -- 172.21.15.7:0/1995680208 >> 172.21.15.7:0/1995680208 conn(0x7f1b2c0a94d0 msgr2=0x7f1b2c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:21:24.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.806+0000 7f1b30888700 1 -- 172.21.15.7:0/1995680208 shutdown_connections 2023-03-03T11:21:24.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.806+0000 7f1b30888700 1 -- 172.21.15.7:0/1995680208 wait complete. 2023-03-03T11:21:24.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.807+0000 7f1b30888700 1 Processor -- start 2023-03-03T11:21:24.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.807+0000 7f1b30888700 1 -- start start 2023-03-03T11:21:24.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b30888700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b2c071e80 0x7f1b2c1479a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:24.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b30888700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b2c072250 0x7f1b2c1491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:24.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b30888700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b2c0c1ce0 0x7f1b2c149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:24.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b30888700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c14a570 con 0x7f1b2c072250 2023-03-03T11:21:24.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b30888700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c14ca50 con 0x7f1b2c071e80 2023-03-03T11:21:24.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b30888700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c14cd30 con 0x7f1b2c0c1ce0 2023-03-03T11:21:24.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b29d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b2c072250 0x7f1b2c1491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:24.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b29d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b2c072250 0x7f1b2c1491e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36002/0 (socket says 172.21.15.7:36002) 2023-03-03T11:21:24.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b29d9b700 1 -- 172.21.15.7:0/542094425 learned_addr learned my addr 172.21.15.7:0/542094425 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:21:24.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b2a59c700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b2c071e80 0x7f1b2c1479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:24.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b2ad9d700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b2c0c1ce0 0x7f1b2c149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:24.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b29d9b700 1 -- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b2c0c1ce0 msgr2=0x7f1b2c149720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:24.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b29d9b700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b2c0c1ce0 0x7f1b2c149720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b29d9b700 1 -- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b2c071e80 msgr2=0x7f1b2c1479a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b29d9b700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b2c071e80 0x7f1b2c1479a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b29d9b700 1 -- 172.21.15.7:0/542094425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b200094a0 con 0x7f1b2c072250 2023-03-03T11:21:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b2a59c700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b2c071e80 0x7f1b2c1479a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:21:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b2ad9d700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b2c0c1ce0 0x7f1b2c149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:21:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.808+0000 7f1b29d9b700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b2c072250 0x7f1b2c1491e0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f1b200090b0 tx=0x7f1b20004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.809+0000 7f1b1affd700 1 -- 172.21.15.7:0/542094425 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b200045b0 con 0x7f1b2c072250 2023-03-03T11:21:24.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.809+0000 7f1b1affd700 1 -- 172.21.15.7:0/542094425 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2001e070 con 0x7f1b2c072250 2023-03-03T11:21:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.809+0000 7f1b1affd700 1 -- 172.21.15.7:0/542094425 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b20011870 con 0x7f1b2c072250 2023-03-03T11:21:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.809+0000 7f1b30888700 1 -- 172.21.15.7:0/542094425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2c151f30 con 0x7f1b2c072250 2023-03-03T11:21:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.809+0000 7f1b30888700 1 -- 172.21.15.7:0/542094425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2c151300 con 0x7f1b2c072250 2023-03-03T11:21:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.810+0000 7f1b30888700 1 -- 172.21.15.7:0/542094425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2c06c9a0 con 0x7f1b2c072250 2023-03-03T11:21:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.810+0000 7f1b1affd700 1 -- 172.21.15.7:0/542094425 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b20017020 con 0x7f1b2c072250 2023-03-03T11:21:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.810+0000 7f1b1affd700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b0c0682f0 0x7f1b0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:21:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.811+0000 7f1b1affd700 1 -- 172.21.15.7:0/542094425 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1b200871d0 con 0x7f1b2c072250 2023-03-03T11:21:24.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.811+0000 7f1b2a59c700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b0c0682f0 0x7f1b0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:21:24.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.811+0000 7f1b2a59c700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b0c0682f0 0x7f1b0c06a7a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0011e0 tx=0x7f1b1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:21:24.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:24.815+0000 7f1b1affd700 1 -- 172.21.15.7:0/542094425 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b20053570 con 0x7f1b2c072250 2023-03-03T11:21:25.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.055+0000 7f1b30888700 1 -- 172.21.15.7:0/542094425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b2c074bb0 con 0x7f1b2c072250 2023-03-03T11:21:25.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.056+0000 7f1b1affd700 1 -- 172.21.15.7:0/542094425 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1b20020090 con 0x7f1b2c072250 2023-03-03T11:21:25.058 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:21:25.058 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:21:25.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 -- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b0c0682f0 msgr2=0x7f1b0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:25.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b0c0682f0 0x7f1b0c06a7a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0011e0 tx=0x7f1b1c009040 comp rx=0 tx=0).stop 2023-03-03T11:21:25.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 -- 172.21.15.7:0/542094425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b2c072250 msgr2=0x7f1b2c1491e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:21:25.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b2c072250 0x7f1b2c1491e0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f1b200090b0 tx=0x7f1b20004750 comp rx=0 tx=0).stop 2023-03-03T11:21:25.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 -- 172.21.15.7:0/542094425 shutdown_connections 2023-03-03T11:21:25.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b2c0c1ce0 0x7f1b2c149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:25.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b2c071e80 0x7f1b2c1479a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:25.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b0c0682f0 0x7f1b0c06a7a0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:25.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 --2- 172.21.15.7:0/542094425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b2c072250 0x7f1b2c1491e0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:21:25.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 -- 172.21.15.7:0/542094425 >> 172.21.15.7:0/542094425 conn(0x7f1b2c0a94d0 msgr2=0x7f1b2c0c2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:21:25.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 -- 172.21.15.7:0/542094425 shutdown_connections 2023-03-03T11:21:25.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:21:25.058+0000 7f1b30888700 1 -- 172.21.15.7:0/542094425 wait complete. 2023-03-03T11:21:25.062 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:21:25.072 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T11:21:42.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:21:42.272 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:12.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:12.392 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:15.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:22:15.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.625+0000 7ff074ba1700 1 Processor -- start 2023-03-03T11:22:15.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.625+0000 7ff074ba1700 1 -- start start 2023-03-03T11:22:15.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.626+0000 7ff074ba1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff07006c560 0x7ff07006d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:15.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.626+0000 7ff074ba1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 0x7ff070158460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:15.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.626+0000 7ff074ba1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff070157b50 con 0x7ff070155090 2023-03-03T11:22:15.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.626+0000 7ff074ba1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff07006c240 con 0x7ff07006c560 2023-03-03T11:22:15.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.626+0000 7ff074ba1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700c33b0 con 0x7ff070157fe0 2023-03-03T11:22:15.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06dd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff07006c560 0x7ff07006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:15.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 0x7ff070158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:15.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06dd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff07006c560 0x7ff07006d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:60040/0 (socket says 172.21.15.7:60040) 2023-03-03T11:22:15.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 0x7ff070158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:44108/0 (socket says 172.21.15.7:44108) 2023-03-03T11:22:15.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06e59c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ff070155090 0x7ff0700736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:33534/0 (socket says 172.21.15.7:33534) 2023-03-03T11:22:15.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06dd9b700 1 -- 172.21.15.7:0/2475355899 learned_addr learned my addr 172.21.15.7:0/2475355899 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:22:15.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06dd9b700 1 -- 172.21.15.7:0/2475355899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 msgr2=0x7ff070158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:22:15.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06dd9b700 1 --2- 172.21.15.7:0/2475355899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 0x7ff070158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06dd9b700 1 -- 172.21.15.7:0/2475355899 >> v1:172.21.15.143:6789/0 conn(0x7ff070155090 legacy=0x7ff0700736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06d59a700 1 -- 172.21.15.7:0/2475355899 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973183461 0 0) 0x7ff070157b50 con 0x7ff070155090 2023-03-03T11:22:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06dd9b700 1 -- 172.21.15.7:0/2475355899 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0700c3690 con 0x7ff07006c560 2023-03-03T11:22:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06ed9d700 1 --2- 172.21.15.7:0/2475355899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 0x7ff070158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:22:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.627+0000 7ff06dd9b700 1 --2- 172.21.15.7:0/2475355899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff07006c560 0x7ff07006d450 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff0640098f0 tx=0x7ff064009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b53f80836321379e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:22:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff06d59a700 1 -- 172.21.15.7:0/2475355899 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064011830 con 0x7ff07006c560 2023-03-03T11:22:15.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff06d59a700 1 -- 172.21.15.7:0/2475355899 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff064011990 con 0x7ff07006c560 2023-03-03T11:22:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff06d59a700 1 -- 172.21.15.7:0/2475355899 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064011d20 con 0x7ff07006c560 2023-03-03T11:22:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff074ba1700 1 -- 172.21.15.7:0/2475355899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff07006c560 msgr2=0x7ff07006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff074ba1700 1 --2- 172.21.15.7:0/2475355899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff07006c560 0x7ff07006d450 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff0640098f0 tx=0x7ff064009c00 comp rx=0 tx=0).stop 2023-03-03T11:22:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff074ba1700 1 -- 172.21.15.7:0/2475355899 shutdown_connections 2023-03-03T11:22:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff074ba1700 1 --2- 172.21.15.7:0/2475355899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 0x7ff070158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff074ba1700 1 --2- 172.21.15.7:0/2475355899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff07006c560 0x7ff07006d450 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:15.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff074ba1700 1 -- 172.21.15.7:0/2475355899 >> 172.21.15.7:0/2475355899 conn(0x7ff0700a94d0 msgr2=0x7ff0700a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:22:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff074ba1700 1 -- 172.21.15.7:0/2475355899 shutdown_connections 2023-03-03T11:22:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.628+0000 7ff074ba1700 1 -- 172.21.15.7:0/2475355899 wait complete. 2023-03-03T11:22:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.630+0000 7ff074ba1700 1 Processor -- start 2023-03-03T11:22:15.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.630+0000 7ff074ba1700 1 -- start start 2023-03-03T11:22:15.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.630+0000 7ff074ba1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff07006c560 0x7ff0700bff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:15.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.630+0000 7ff074ba1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff070155090 0x7ff0700c0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:15.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.630+0000 7ff074ba1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 0x7ff0701eb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:15.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.630+0000 7ff074ba1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700c0a70 con 0x7ff070155090 2023-03-03T11:22:15.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.630+0000 7ff074ba1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0700bf650 con 0x7ff07006c560 2023-03-03T11:22:15.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.630+0000 7ff074ba1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff070156500 con 0x7ff070157fe0 2023-03-03T11:22:15.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.630+0000 7ff06dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff070155090 0x7ff0700c0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:15.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff06dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff070155090 0x7ff0700c0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36038/0 (socket says 172.21.15.7:36038) 2023-03-03T11:22:15.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff06dd9b700 1 -- 172.21.15.7:0/413983122 learned_addr learned my addr 172.21.15.7:0/413983122 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:22:15.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff06e59c700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff07006c560 0x7ff0700bff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:15.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff06ed9d700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 0x7ff0701eb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:15.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff06dd9b700 1 -- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 msgr2=0x7ff0701eb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:15.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff06dd9b700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 0x7ff0701eb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:15.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff06dd9b700 1 -- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff07006c560 msgr2=0x7ff0700bff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:15.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff06dd9b700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff07006c560 0x7ff0700bff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:15.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff06dd9b700 1 -- 172.21.15.7:0/413983122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff068009a70 con 0x7ff070155090 2023-03-03T11:22:15.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff06dd9b700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff070155090 0x7ff0700c0450 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7ff07006bb00 tx=0x7ff064004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:22:15.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff05effd700 1 -- 172.21.15.7:0/413983122 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0640042c0 con 0x7ff070155090 2023-03-03T11:22:15.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff05effd700 1 -- 172.21.15.7:0/413983122 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06401f070 con 0x7ff070155090 2023-03-03T11:22:15.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff05effd700 1 -- 172.21.15.7:0/413983122 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0640036a0 con 0x7ff070155090 2023-03-03T11:22:15.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff074ba1700 1 -- 172.21.15.7:0/413983122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff064009550 con 0x7ff070155090 2023-03-03T11:22:15.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.631+0000 7ff074ba1700 1 -- 172.21.15.7:0/413983122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0701589a0 con 0x7ff070155090 2023-03-03T11:22:15.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.632+0000 7ff074ba1700 1 -- 172.21.15.7:0/413983122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0700c3970 con 0x7ff070155090 2023-03-03T11:22:15.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.633+0000 7ff05effd700 1 -- 172.21.15.7:0/413983122 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff06402a020 con 0x7ff070155090 2023-03-03T11:22:15.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.633+0000 7ff05effd700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0500682f0 0x7ff05006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:15.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.633+0000 7ff05effd700 1 -- 172.21.15.7:0/413983122 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff0640876c0 con 0x7ff070155090 2023-03-03T11:22:15.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.633+0000 7ff06e59c700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0500682f0 0x7ff05006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:15.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.634+0000 7ff06e59c700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0500682f0 0x7ff05006a7a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff060007fd0 tx=0x7ff060007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:22:15.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.638+0000 7ff05effd700 1 -- 172.21.15.7:0/413983122 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff064053970 con 0x7ff070155090 2023-03-03T11:22:15.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.873+0000 7ff074ba1700 1 -- 172.21.15.7:0/413983122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0701545e0 con 0x7ff070155090 2023-03-03T11:22:15.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.873+0000 7ff05effd700 1 -- 172.21.15.7:0/413983122 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff064006d80 con 0x7ff070155090 2023-03-03T11:22:15.876 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:22:15.876 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:22:15.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 -- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0500682f0 msgr2=0x7ff05006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:15.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0500682f0 0x7ff05006a7a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff060007fd0 tx=0x7ff060007de0 comp rx=0 tx=0).stop 2023-03-03T11:22:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 -- 172.21.15.7:0/413983122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff070155090 msgr2=0x7ff0700c0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff070155090 0x7ff0700c0450 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7ff07006bb00 tx=0x7ff064004750 comp rx=0 tx=0).stop 2023-03-03T11:22:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 -- 172.21.15.7:0/413983122 shutdown_connections 2023-03-03T11:22:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff070157fe0 0x7ff0701eb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff07006c560 0x7ff0700bff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0500682f0 0x7ff05006a7a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:15.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 --2- 172.21.15.7:0/413983122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff070155090 0x7ff0700c0450 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:15.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 -- 172.21.15.7:0/413983122 >> 172.21.15.7:0/413983122 conn(0x7ff0700a94d0 msgr2=0x7ff0700c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:22:15.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 -- 172.21.15.7:0/413983122 shutdown_connections 2023-03-03T11:22:15.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:15.875+0000 7ff074ba1700 1 -- 172.21.15.7:0/413983122 wait complete. 2023-03-03T11:22:15.880 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:22:15.890 INFO:tasks.mds_thrash.fs.[a]:waiting for 91 secs before thrashing 2023-03-03T11:22:27.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:22:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.209+0000 7f94c470f700 1 Processor -- start 2023-03-03T11:22:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.210+0000 7f94c470f700 1 -- start start 2023-03-03T11:22:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.210+0000 7f94c470f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c6010 0x7f94bc0c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.210+0000 7f94c470f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94bc0c17f0 0x7f94bc0c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.210+0000 7f94c470f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94bc0c5500 con 0x7f94bc06d6a0 2023-03-03T11:22:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.210+0000 7f94c470f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94bc152100 con 0x7f94bc0c17f0 2023-03-03T11:22:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.210+0000 7f94c470f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc14ebe0 con 0x7f94bc0c6010 2023-03-03T11:22:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c1caa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94bc0c17f0 0x7f94bc0c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c1caa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94bc0c17f0 0x7f94bc0c4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36050/0 (socket says 172.21.15.7:36050) 2023-03-03T11:22:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c1caa700 1 -- 172.21.15.7:0/1016006573 learned_addr learned my addr 172.21.15.7:0/1016006573 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:22:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c24ab700 1 --2- 172.21.15.7:0/1016006573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c6010 0x7f94bc0c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c1caa700 1 -- 172.21.15.7:0/1016006573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c6010 msgr2=0x7f94bc0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:22:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c1caa700 1 --2- 172.21.15.7:0/1016006573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c6010 0x7f94bc0c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c1caa700 1 -- 172.21.15.7:0/1016006573 >> v1:172.21.15.143:6789/0 conn(0x7f94bc06d6a0 legacy=0x7f94bc06db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c1caa700 1 -- 172.21.15.7:0/1016006573 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94bc0c2e70 con 0x7f94bc0c17f0 2023-03-03T11:22:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c24ab700 1 --2- 172.21.15.7:0/1016006573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c6010 0x7f94bc0c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:22:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c1caa700 1 --2- 172.21.15.7:0/1016006573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc0c17f0 0x7f94bc0c4fc0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f94b800bb90 tx=0x7f94b800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1c35b9b9a7d068 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:22:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c14a9700 1 -- 172.21.15.7:0/1016006573 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b800dd50 con 0x7f94bc0c17f0 2023-03-03T11:22:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c14a9700 1 -- 172.21.15.7:0/1016006573 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b8010470 con 0x7f94bc0c17f0 2023-03-03T11:22:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.211+0000 7f94c14a9700 1 -- 172.21.15.7:0/1016006573 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b8010800 con 0x7f94bc0c17f0 2023-03-03T11:22:27.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.212+0000 7f94c470f700 1 -- 172.21.15.7:0/1016006573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc0c17f0 msgr2=0x7f94bc0c4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:27.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.212+0000 7f94c470f700 1 --2- 172.21.15.7:0/1016006573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc0c17f0 0x7f94bc0c4fc0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f94b800bb90 tx=0x7f94b800bea0 comp rx=0 tx=0).stop 2023-03-03T11:22:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.212+0000 7f94c470f700 1 -- 172.21.15.7:0/1016006573 shutdown_connections 2023-03-03T11:22:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.212+0000 7f94c470f700 1 --2- 172.21.15.7:0/1016006573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c6010 0x7f94bc0c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.212+0000 7f94c470f700 1 --2- 172.21.15.7:0/1016006573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc0c17f0 0x7f94bc0c4fc0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.212+0000 7f94c470f700 1 -- 172.21.15.7:0/1016006573 >> 172.21.15.7:0/1016006573 conn(0x7f94bc0a9430 msgr2=0x7f94bc0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:22:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.212+0000 7f94c470f700 1 -- 172.21.15.7:0/1016006573 shutdown_connections 2023-03-03T11:22:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.212+0000 7f94c470f700 1 -- 172.21.15.7:0/1016006573 wait complete. 2023-03-03T11:22:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.213+0000 7f94c470f700 1 Processor -- start 2023-03-03T11:22:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.213+0000 7f94c470f700 1 -- start start 2023-03-03T11:22:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c470f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc06d6a0 0x7f94bc1ed810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c470f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c17f0 0x7f94bc1505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c470f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94bc0c6010 0x7f94bc154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c470f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc152ff0 con 0x7f94bc06d6a0 2023-03-03T11:22:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c470f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0c00a0 con 0x7f94bc0c6010 2023-03-03T11:22:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c470f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94bc0c0380 con 0x7f94bc0c17f0 2023-03-03T11:22:27.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c24ab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc06d6a0 0x7f94bc1ed810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:27.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c24ab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc06d6a0 0x7f94bc1ed810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36054/0 (socket says 172.21.15.7:36054) 2023-03-03T11:22:27.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c24ab700 1 -- 172.21.15.7:0/881122451 learned_addr learned my addr 172.21.15.7:0/881122451 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:22:27.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c1caa700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c17f0 0x7f94bc1505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:27.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c2cac700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94bc0c6010 0x7f94bc154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:27.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c24ab700 1 -- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c17f0 msgr2=0x7f94bc1505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:27.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c24ab700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c17f0 0x7f94bc1505c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:27.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c24ab700 1 -- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94bc0c6010 msgr2=0x7f94bc154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:27.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c24ab700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94bc0c6010 0x7f94bc154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:27.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.214+0000 7f94c24ab700 1 -- 172.21.15.7:0/881122451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b800b7f0 con 0x7f94bc06d6a0 2023-03-03T11:22:27.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.215+0000 7f94c24ab700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc06d6a0 0x7f94bc1ed810 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f94b000e760 tx=0x7f94b000ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:22:27.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.215+0000 7f94aeffd700 1 -- 172.21.15.7:0/881122451 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b000c070 con 0x7f94bc06d6a0 2023-03-03T11:22:27.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.215+0000 7f94aeffd700 1 -- 172.21.15.7:0/881122451 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0013610 con 0x7f94bc06d6a0 2023-03-03T11:22:27.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.215+0000 7f94aeffd700 1 -- 172.21.15.7:0/881122451 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b00139c0 con 0x7f94bc06d6a0 2023-03-03T11:22:27.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.215+0000 7f94c470f700 1 -- 172.21.15.7:0/881122451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94bc0aa960 con 0x7f94bc06d6a0 2023-03-03T11:22:27.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.215+0000 7f94c470f700 1 -- 172.21.15.7:0/881122451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94bc0aa680 con 0x7f94bc06d6a0 2023-03-03T11:22:27.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.216+0000 7f94c470f700 1 -- 172.21.15.7:0/881122451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94bc0c2b00 con 0x7f94bc06d6a0 2023-03-03T11:22:27.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.216+0000 7f94aeffd700 1 -- 172.21.15.7:0/881122451 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f94b0023070 con 0x7f94bc06d6a0 2023-03-03T11:22:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.217+0000 7f94aeffd700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94a00682f0 0x7f94a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:22:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.217+0000 7f94aeffd700 1 -- 172.21.15.7:0/881122451 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f94b0085d80 con 0x7f94bc06d6a0 2023-03-03T11:22:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.217+0000 7f94c1caa700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94a00682f0 0x7f94a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:22:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.218+0000 7f94c1caa700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94a00682f0 0x7f94a006a7a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f94b800bee0 tx=0x7f94b8013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:22:27.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.221+0000 7f94aeffd700 1 -- 172.21.15.7:0/881122451 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f94b0052120 con 0x7f94bc06d6a0 2023-03-03T11:22:27.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.459+0000 7f94c470f700 1 -- 172.21.15.7:0/881122451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94bc06fa20 con 0x7f94bc06d6a0 2023-03-03T11:22:27.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.460+0000 7f94aeffd700 1 -- 172.21.15.7:0/881122451 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f94b0059680 con 0x7f94bc06d6a0 2023-03-03T11:22:27.462 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:22:27.462 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:22:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.461+0000 7f94c470f700 1 -- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94a00682f0 msgr2=0x7f94a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.461+0000 7f94c470f700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94a00682f0 0x7f94a006a7a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f94b800bee0 tx=0x7f94b8013000 comp rx=0 tx=0).stop 2023-03-03T11:22:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.461+0000 7f94c470f700 1 -- 172.21.15.7:0/881122451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc06d6a0 msgr2=0x7f94bc1ed810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:22:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.462+0000 7f94c470f700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc06d6a0 0x7f94bc1ed810 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f94b000e760 tx=0x7f94b000ea70 comp rx=0 tx=0).stop 2023-03-03T11:22:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.462+0000 7f94c470f700 1 -- 172.21.15.7:0/881122451 shutdown_connections 2023-03-03T11:22:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.462+0000 7f94c470f700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94bc0c6010 0x7f94bc154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:27.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.462+0000 7f94c470f700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94a00682f0 0x7f94a006a7a0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:27.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.462+0000 7f94c470f700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94bc06d6a0 0x7f94bc1ed810 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:27.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.462+0000 7f94c470f700 1 --2- 172.21.15.7:0/881122451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94bc0c17f0 0x7f94bc1505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:22:27.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.462+0000 7f94c470f700 1 -- 172.21.15.7:0/881122451 >> 172.21.15.7:0/881122451 conn(0x7f94bc0a9430 msgr2=0x7f94bc0a9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:22:27.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.462+0000 7f94c470f700 1 -- 172.21.15.7:0/881122451 shutdown_connections 2023-03-03T11:22:27.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:22:27.462+0000 7f94c470f700 1 -- 172.21.15.7:0/881122451 wait complete. 2023-03-03T11:22:27.466 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:22:27.475 INFO:tasks.mds_thrash.fs.[b]:waiting for 37 secs before thrashing 2023-03-03T11:22:42.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:22:42.515 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:04.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:23:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.615+0000 7f6e18fb0700 1 Processor -- start 2023-03-03T11:23:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.615+0000 7f6e18fb0700 1 -- start start 2023-03-03T11:23:04.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e18fb0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6e14151b20 0x7f6e140736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:04.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e18fb0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e14073c30 0x7f6e141581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:04.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e18fb0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1406cb70 con 0x7f6e140c2950 2023-03-03T11:23:04.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e18fb0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1406c2f0 con 0x7f6e14073c30 2023-03-03T11:23:04.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e18fb0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1406ce50 con 0x7f6e14151b20 2023-03-03T11:23:04.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e12d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e14073c30 0x7f6e141581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:04.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e11d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6e140c2950 0x7f6e14157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58318/0 (socket says 172.21.15.7:58318) 2023-03-03T11:23:04.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e11d9b700 1 -- 172.21.15.7:0/348064540 learned_addr learned my addr 172.21.15.7:0/348064540 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:23:04.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e1259c700 1 --2- 172.21.15.7:0/348064540 >> v2:172.21.15.143:3300/0 conn(0x7f6e14151b20 0x7f6e140736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:04.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e12d9d700 1 -- 172.21.15.7:0/348064540 >> v2:172.21.15.143:3300/0 conn(0x7f6e14151b20 msgr2=0x7f6e140736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:23:04.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e12d9d700 1 --2- 172.21.15.7:0/348064540 >> v2:172.21.15.143:3300/0 conn(0x7f6e14151b20 0x7f6e140736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:04.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e12d9d700 1 -- 172.21.15.7:0/348064540 >> v1:172.21.15.7:6789/0 conn(0x7f6e140c2950 legacy=0x7f6e14157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:04.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e1159a700 1 -- 172.21.15.7:0/348064540 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093867079 0 0) 0x7f6e1406cb70 con 0x7f6e140c2950 2023-03-03T11:23:04.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e12d9d700 1 -- 172.21.15.7:0/348064540 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1406d130 con 0x7f6e14073c30 2023-03-03T11:23:04.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.616+0000 7f6e1259c700 1 --2- 172.21.15.7:0/348064540 >> v2:172.21.15.143:3300/0 conn(0x7f6e14151b20 0x7f6e140736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:23:04.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.617+0000 7f6e12d9d700 1 --2- 172.21.15.7:0/348064540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14073c30 0x7f6e141581f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f6e1406b9c0 tx=0x7f6e0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75dc3dadc562586 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:23:04.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.617+0000 7f6e1159a700 1 -- 172.21.15.7:0/348064540 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c00e070 con 0x7f6e14073c30 2023-03-03T11:23:04.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.617+0000 7f6e1159a700 1 -- 172.21.15.7:0/348064540 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c00bbd0 con 0x7f6e14073c30 2023-03-03T11:23:04.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.617+0000 7f6e1159a700 1 -- 172.21.15.7:0/348064540 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c012430 con 0x7f6e14073c30 2023-03-03T11:23:04.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.617+0000 7f6e18fb0700 1 -- 172.21.15.7:0/348064540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14073c30 msgr2=0x7f6e141581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:04.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.617+0000 7f6e18fb0700 1 --2- 172.21.15.7:0/348064540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14073c30 0x7f6e141581f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f6e1406b9c0 tx=0x7f6e0c00b330 comp rx=0 tx=0).stop 2023-03-03T11:23:04.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.617+0000 7f6e18fb0700 1 -- 172.21.15.7:0/348064540 shutdown_connections 2023-03-03T11:23:04.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.617+0000 7f6e18fb0700 1 --2- 172.21.15.7:0/348064540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14073c30 0x7f6e141581f0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:04.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.617+0000 7f6e18fb0700 1 --2- 172.21.15.7:0/348064540 >> v2:172.21.15.143:3300/0 conn(0x7f6e14151b20 0x7f6e140736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:04.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.617+0000 7f6e18fb0700 1 -- 172.21.15.7:0/348064540 >> 172.21.15.7:0/348064540 conn(0x7f6e140a94d0 msgr2=0x7f6e140a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:23:04.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.618+0000 7f6e18fb0700 1 -- 172.21.15.7:0/348064540 shutdown_connections 2023-03-03T11:23:04.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.618+0000 7f6e18fb0700 1 -- 172.21.15.7:0/348064540 wait complete. 2023-03-03T11:23:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.619+0000 7f6e18fb0700 1 Processor -- start 2023-03-03T11:23:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.619+0000 7f6e18fb0700 1 -- start start 2023-03-03T11:23:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.620+0000 7f6e18fb0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e14073c30 0x7f6e141ed5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.620+0000 7f6e18fb0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e140c2950 0x7f6e141edae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.620+0000 7f6e18fb0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14151b20 0x7f6e140bee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.620+0000 7f6e18fb0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e141ee270 con 0x7f6e14151b20 2023-03-03T11:23:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.620+0000 7f6e18fb0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e140c0060 con 0x7f6e140c2950 2023-03-03T11:23:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.620+0000 7f6e18fb0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e141ee020 con 0x7f6e14073c30 2023-03-03T11:23:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.620+0000 7f6e12d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14151b20 0x7f6e140bee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.620+0000 7f6e12d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14151b20 0x7f6e140bee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36114/0 (socket says 172.21.15.7:36114) 2023-03-03T11:23:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.620+0000 7f6e12d9d700 1 -- 172.21.15.7:0/2342728849 learned_addr learned my addr 172.21.15.7:0/2342728849 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:23:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e11d9b700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e140c2950 0x7f6e141edae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:04.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e1259c700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e14073c30 0x7f6e141ed5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e12d9d700 1 -- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e14073c30 msgr2=0x7f6e141ed5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e12d9d700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e14073c30 0x7f6e141ed5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e12d9d700 1 -- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e140c2950 msgr2=0x7f6e141edae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e12d9d700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e140c2950 0x7f6e141edae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e12d9d700 1 -- 172.21.15.7:0/2342728849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c009a70 con 0x7f6e14151b20 2023-03-03T11:23:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e11d9b700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e140c2950 0x7f6e141edae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:23:04.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e12d9d700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14151b20 0x7f6e140bee40 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c00b730 tx=0x7f6e0c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:23:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e02ffd700 1 -- 172.21.15.7:0/2342728849 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c012430 con 0x7f6e14151b20 2023-03-03T11:23:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e02ffd700 1 -- 172.21.15.7:0/2342728849 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0c020070 con 0x7f6e14151b20 2023-03-03T11:23:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e02ffd700 1 -- 172.21.15.7:0/2342728849 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c004280 con 0x7f6e14151b20 2023-03-03T11:23:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e18fb0700 1 -- 172.21.15.7:0/2342728849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e14158730 con 0x7f6e14151b20 2023-03-03T11:23:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.621+0000 7f6e18fb0700 1 -- 172.21.15.7:0/2342728849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1406d410 con 0x7f6e14151b20 2023-03-03T11:23:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.622+0000 7f6e18fb0700 1 -- 172.21.15.7:0/2342728849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e14151ef0 con 0x7f6e14151b20 2023-03-03T11:23:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.623+0000 7f6e02ffd700 1 -- 172.21.15.7:0/2342728849 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6e0c02e070 con 0x7f6e14151b20 2023-03-03T11:23:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.623+0000 7f6e02ffd700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df4068320 0x7f6df406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.623+0000 7f6e02ffd700 1 -- 172.21.15.7:0/2342728849 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6e0c022070 con 0x7f6e14151b20 2023-03-03T11:23:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.623+0000 7f6e1259c700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df4068320 0x7f6df406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.624+0000 7f6e1259c700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df4068320 0x7f6df406a7d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6e08006fd0 tx=0x7f6e08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:23:04.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.628+0000 7f6e02ffd700 1 -- 172.21.15.7:0/2342728849 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6e0c054350 con 0x7f6e14151b20 2023-03-03T11:23:04.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.867+0000 7f6e18fb0700 1 -- 172.21.15.7:0/2342728849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e140acdc0 con 0x7f6e14151b20 2023-03-03T11:23:04.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.867+0000 7f6e02ffd700 1 -- 172.21.15.7:0/2342728849 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6e0c057970 con 0x7f6e14151b20 2023-03-03T11:23:04.869 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:23:04.870 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:23:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 -- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df4068320 msgr2=0x7f6df406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df4068320 0x7f6df406a7d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6e08006fd0 tx=0x7f6e08008040 comp rx=0 tx=0).stop 2023-03-03T11:23:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 -- 172.21.15.7:0/2342728849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14151b20 msgr2=0x7f6e140bee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14151b20 0x7f6e140bee40 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c00b730 tx=0x7f6e0c004ad0 comp rx=0 tx=0).stop 2023-03-03T11:23:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 -- 172.21.15.7:0/2342728849 shutdown_connections 2023-03-03T11:23:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df4068320 0x7f6df406a7d0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14151b20 0x7f6e140bee40 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e14073c30 0x7f6e141ed5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 --2- 172.21.15.7:0/2342728849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e140c2950 0x7f6e141edae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 -- 172.21.15.7:0/2342728849 >> 172.21.15.7:0/2342728849 conn(0x7f6e140a94d0 msgr2=0x7f6e1406c430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:23:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 -- 172.21.15.7:0/2342728849 shutdown_connections 2023-03-03T11:23:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:04.869+0000 7f6e18fb0700 1 -- 172.21.15.7:0/2342728849 wait complete. 2023-03-03T11:23:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:23:04.883 INFO:tasks.mds_thrash.fs.[b]:waiting for 117 secs before thrashing 2023-03-03T11:23:12.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:12.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:42.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:42.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:23:46.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:23:47.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.029+0000 7f148ee9e700 1 Processor -- start 2023-03-03T11:23:47.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.029+0000 7f148ee9e700 1 -- start start 2023-03-03T11:23:47.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.030+0000 7f148ee9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14880a6370 0x7f148806a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:47.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.030+0000 7f148ee9e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f14880c5cb0 0x7f148815cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:47.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.030+0000 7f148ee9e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14881558f0 con 0x7f148806a2b0 2023-03-03T11:23:47.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.030+0000 7f148ee9e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148806bab0 con 0x7f14880c5cb0 2023-03-03T11:23:47.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.030+0000 7f148ee9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880a6180 con 0x7f14880a6370 2023-03-03T11:23:47.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.030+0000 7f148d43b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f14880c5cb0 0x7f148815cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:47.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.030+0000 7f148cc3a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f148806a2b0 0x7f1488155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58356/0 (socket says 172.21.15.7:58356) 2023-03-03T11:23:47.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.030+0000 7f148d43b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f14880c5cb0 0x7f148815cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36148/0 (socket says 172.21.15.7:36148) 2023-03-03T11:23:47.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.030+0000 7f148cc3a700 1 -- 172.21.15.7:0/1342476241 learned_addr learned my addr 172.21.15.7:0/1342476241 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:23:47.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.031+0000 7f1487fff700 1 --2- 172.21.15.7:0/1342476241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14880a6370 0x7f148806a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:47.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.031+0000 7f148d43b700 1 -- 172.21.15.7:0/1342476241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14880a6370 msgr2=0x7f148806a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:23:47.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.031+0000 7f148d43b700 1 --2- 172.21.15.7:0/1342476241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14880a6370 0x7f148806a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:47.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.031+0000 7f148d43b700 1 -- 172.21.15.7:0/1342476241 >> v1:172.21.15.7:6789/0 conn(0x7f148806a2b0 legacy=0x7f1488155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:47.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.031+0000 7f14877fe700 1 -- 172.21.15.7:0/1342476241 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066469865 0 0) 0x7f14881558f0 con 0x7f148806a2b0 2023-03-03T11:23:47.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.031+0000 7f148d43b700 1 -- 172.21.15.7:0/1342476241 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880a5ea0 con 0x7f14880c5cb0 2023-03-03T11:23:47.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.031+0000 7f148d43b700 1 --2- 172.21.15.7:0/1342476241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 0x7f148815cbd0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f1480009d90 tx=0x7f148000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8d48342ccb329f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:23:47.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.031+0000 7f14877fe700 1 -- 172.21.15.7:0/1342476241 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148000e070 con 0x7f14880c5cb0 2023-03-03T11:23:47.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.031+0000 7f14877fe700 1 -- 172.21.15.7:0/1342476241 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148000bbd0 con 0x7f14880c5cb0 2023-03-03T11:23:47.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.031+0000 7f14877fe700 1 -- 172.21.15.7:0/1342476241 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480012430 con 0x7f14880c5cb0 2023-03-03T11:23:47.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.032+0000 7f148ee9e700 1 -- 172.21.15.7:0/1342476241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 msgr2=0x7f148815cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:47.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.032+0000 7f148ee9e700 1 --2- 172.21.15.7:0/1342476241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 0x7f148815cbd0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f1480009d90 tx=0x7f148000b330 comp rx=0 tx=0).stop 2023-03-03T11:23:47.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.032+0000 7f148ee9e700 1 -- 172.21.15.7:0/1342476241 shutdown_connections 2023-03-03T11:23:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.032+0000 7f148ee9e700 1 --2- 172.21.15.7:0/1342476241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 0x7f148815cbd0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.032+0000 7f148ee9e700 1 --2- 172.21.15.7:0/1342476241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f14880a6370 0x7f148806a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.032+0000 7f148ee9e700 1 -- 172.21.15.7:0/1342476241 >> 172.21.15.7:0/1342476241 conn(0x7f14880a9430 msgr2=0x7f14880c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:23:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.032+0000 7f148ee9e700 1 -- 172.21.15.7:0/1342476241 shutdown_connections 2023-03-03T11:23:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.032+0000 7f148ee9e700 1 -- 172.21.15.7:0/1342476241 wait complete. 2023-03-03T11:23:47.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.033+0000 7f148ee9e700 1 Processor -- start 2023-03-03T11:23:47.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.033+0000 7f148ee9e700 1 -- start start 2023-03-03T11:23:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148ee9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f148806a2b0 0x7f14881e9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148ee9e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14880a6370 0x7f1488151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148ee9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 0x7f148814e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148ee9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1488072150 con 0x7f14880c5cb0 2023-03-03T11:23:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148ee9e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14881edce0 con 0x7f14880a6370 2023-03-03T11:23:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148ee9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14881ea2a0 con 0x7f148806a2b0 2023-03-03T11:23:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148d43b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 0x7f148814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:47.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148d43b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 0x7f148814e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36154/0 (socket says 172.21.15.7:36154) 2023-03-03T11:23:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148d43b700 1 -- 172.21.15.7:0/89136822 learned_addr learned my addr 172.21.15.7:0/89136822 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:23:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148cc3a700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f148806a2b0 0x7f14881e9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f1487fff700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14880a6370 0x7f1488151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148d43b700 1 -- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f148806a2b0 msgr2=0x7f14881e9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148d43b700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f148806a2b0 0x7f14881e9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148d43b700 1 -- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14880a6370 msgr2=0x7f1488151950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148d43b700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14880a6370 0x7f1488151950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148d43b700 1 -- 172.21.15.7:0/89136822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1480009a70 con 0x7f14880c5cb0 2023-03-03T11:23:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148cc3a700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f148806a2b0 0x7f14881e9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:23:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.034+0000 7f148d43b700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 0x7f148814e460 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f1480014040 tx=0x7f1480004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:23:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.035+0000 7f14857fa700 1 -- 172.21.15.7:0/89136822 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480012430 con 0x7f14880c5cb0 2023-03-03T11:23:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.035+0000 7f14857fa700 1 -- 172.21.15.7:0/89136822 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1480020070 con 0x7f14880c5cb0 2023-03-03T11:23:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.035+0000 7f148ee9e700 1 -- 172.21.15.7:0/89136822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1488153160 con 0x7f14880c5cb0 2023-03-03T11:23:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.035+0000 7f14857fa700 1 -- 172.21.15.7:0/89136822 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480004260 con 0x7f14880c5cb0 2023-03-03T11:23:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.035+0000 7f148ee9e700 1 -- 172.21.15.7:0/89136822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1488152e80 con 0x7f14880c5cb0 2023-03-03T11:23:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.036+0000 7f148ee9e700 1 -- 172.21.15.7:0/89136822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14880c1500 con 0x7f14880c5cb0 2023-03-03T11:23:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.036+0000 7f14857fa700 1 -- 172.21.15.7:0/89136822 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f148002e070 con 0x7f14880c5cb0 2023-03-03T11:23:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.036+0000 7f14857fa700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f146c0682f0 0x7f146c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:23:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.037+0000 7f14857fa700 1 -- 172.21.15.7:0/89136822 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1480022070 con 0x7f14880c5cb0 2023-03-03T11:23:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.037+0000 7f148cc3a700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f146c0682f0 0x7f146c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:23:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.037+0000 7f148cc3a700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f146c0682f0 0x7f146c06a7a0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f147c000f60 tx=0x7f147c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:23:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.041+0000 7f14857fa700 1 -- 172.21.15.7:0/89136822 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1480054300 con 0x7f14880c5cb0 2023-03-03T11:23:47.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.279+0000 7f148ee9e700 1 -- 172.21.15.7:0/89136822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1488072340 con 0x7f14880c5cb0 2023-03-03T11:23:47.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.279+0000 7f14857fa700 1 -- 172.21.15.7:0/89136822 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f148002c090 con 0x7f14880c5cb0 2023-03-03T11:23:47.281 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:23:47.282 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:23:47.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.281+0000 7f148ee9e700 1 -- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f146c0682f0 msgr2=0x7f146c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:47.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.281+0000 7f148ee9e700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f146c0682f0 0x7f146c06a7a0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f147c000f60 tx=0x7f147c009040 comp rx=0 tx=0).stop 2023-03-03T11:23:47.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.281+0000 7f148ee9e700 1 -- 172.21.15.7:0/89136822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 msgr2=0x7f148814e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:23:47.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.281+0000 7f148ee9e700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 0x7f148814e460 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f1480014040 tx=0x7f1480004ad0 comp rx=0 tx=0).stop 2023-03-03T11:23:47.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.281+0000 7f148ee9e700 1 -- 172.21.15.7:0/89136822 shutdown_connections 2023-03-03T11:23:47.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.281+0000 7f148ee9e700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f146c0682f0 0x7f146c06a7a0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:47.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.281+0000 7f148ee9e700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f14880c5cb0 0x7f148814e460 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.281+0000 7f148ee9e700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f148806a2b0 0x7f14881e9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.281+0000 7f148ee9e700 1 --2- 172.21.15.7:0/89136822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14880a6370 0x7f1488151950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:23:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.281+0000 7f148ee9e700 1 -- 172.21.15.7:0/89136822 >> 172.21.15.7:0/89136822 conn(0x7f14880a9430 msgr2=0x7f14880c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:23:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.282+0000 7f148ee9e700 1 -- 172.21.15.7:0/89136822 shutdown_connections 2023-03-03T11:23:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:23:47.282+0000 7f148ee9e700 1 -- 172.21.15.7:0/89136822 wait complete. 2023-03-03T11:23:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:23:47.296 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T11:24:12.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:12.880 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:43.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:43.028 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:24:53.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:24:53.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.433+0000 7fdd82a38700 1 Processor -- start 2023-03-03T11:24:53.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.433+0000 7fdd82a38700 1 -- start start 2023-03-03T11:24:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd82a38700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdd7c0ace30 0x7fdd7c152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:53.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd82a38700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdd7c0c26e0 0x7fdd7c152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:53.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd82a38700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c0ac900 con 0x7fdd7c0ac220 2023-03-03T11:24:53.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd82a38700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7c071e30 con 0x7fdd7c0c26e0 2023-03-03T11:24:53.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd82a38700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7c152ed0 con 0x7fdd7c0ace30 2023-03-03T11:24:53.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd80fd5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdd7c0c26e0 0x7fdd7c152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:53.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd80fd5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdd7c0c26e0 0x7fdd7c152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36198/0 (socket says 172.21.15.7:36198) 2023-03-03T11:24:53.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd80fd5700 1 -- 172.21.15.7:0/2134296597 learned_addr learned my addr 172.21.15.7:0/2134296597 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:24:53.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd7b7fe700 1 --2- 172.21.15.7:0/2134296597 >> v2:172.21.15.143:3300/0 conn(0x7fdd7c0ace30 0x7fdd7c152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:53.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd80fd5700 1 -- 172.21.15.7:0/2134296597 >> v2:172.21.15.143:3300/0 conn(0x7fdd7c0ace30 msgr2=0x7fdd7c152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:24:53.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd80fd5700 1 --2- 172.21.15.7:0/2134296597 >> v2:172.21.15.143:3300/0 conn(0x7fdd7c0ace30 0x7fdd7c152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:53.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd80fd5700 1 -- 172.21.15.7:0/2134296597 >> v1:172.21.15.143:6789/0 conn(0x7fdd7c0ac220 legacy=0x7fdd7c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:24:53.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.434+0000 7fdd80fd5700 1 -- 172.21.15.7:0/2134296597 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c0acb50 con 0x7fdd7c0c26e0 2023-03-03T11:24:53.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd80fd5700 1 --2- 172.21.15.7:0/2134296597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0c26e0 0x7fdd7c152990 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0c4eb0 tx=0x7fdd7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37eac40ecc3115fe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:24:53.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd7affd700 1 -- 172.21.15.7:0/2134296597 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7400e070 con 0x7fdd7c0c26e0 2023-03-03T11:24:53.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd7affd700 1 -- 172.21.15.7:0/2134296597 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd74003a20 con 0x7fdd7c0c26e0 2023-03-03T11:24:53.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd7affd700 1 -- 172.21.15.7:0/2134296597 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74003dd0 con 0x7fdd7c0c26e0 2023-03-03T11:24:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd82a38700 1 -- 172.21.15.7:0/2134296597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0c26e0 msgr2=0x7fdd7c152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd82a38700 1 --2- 172.21.15.7:0/2134296597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0c26e0 0x7fdd7c152990 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0c4eb0 tx=0x7fdd7400b330 comp rx=0 tx=0).stop 2023-03-03T11:24:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd82a38700 1 -- 172.21.15.7:0/2134296597 shutdown_connections 2023-03-03T11:24:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd82a38700 1 --2- 172.21.15.7:0/2134296597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0c26e0 0x7fdd7c152990 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd82a38700 1 --2- 172.21.15.7:0/2134296597 >> v2:172.21.15.143:3300/0 conn(0x7fdd7c0ace30 0x7fdd7c152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd82a38700 1 -- 172.21.15.7:0/2134296597 >> 172.21.15.7:0/2134296597 conn(0x7fdd7c0a94d0 msgr2=0x7fdd7c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:24:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd82a38700 1 -- 172.21.15.7:0/2134296597 shutdown_connections 2023-03-03T11:24:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.435+0000 7fdd82a38700 1 -- 172.21.15.7:0/2134296597 wait complete. 2023-03-03T11:24:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.437+0000 7fdd82a38700 1 Processor -- start 2023-03-03T11:24:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.437+0000 7fdd82a38700 1 -- start start 2023-03-03T11:24:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.437+0000 7fdd82a38700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd7c0ac220 0x7fdd7c0ba050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.437+0000 7fdd82a38700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0ace30 0x7fdd7c151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.437+0000 7fdd82a38700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd7c0c26e0 0x7fdd7c0c0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.437+0000 7fdd82a38700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c06e170 con 0x7fdd7c0ace30 2023-03-03T11:24:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.437+0000 7fdd82a38700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c14d310 con 0x7fdd7c0ac220 2023-03-03T11:24:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.437+0000 7fdd82a38700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c1ee020 con 0x7fdd7c0c26e0 2023-03-03T11:24:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0ace30 0x7fdd7c151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0ace30 0x7fdd7c151400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36202/0 (socket says 172.21.15.7:36202) 2023-03-03T11:24:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7b7fe700 1 -- 172.21.15.7:0/2748553081 learned_addr learned my addr 172.21.15.7:0/2748553081 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:24:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7bfff700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd7c0ac220 0x7fdd7c0ba050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd80fd5700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd7c0c26e0 0x7fdd7c0c0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7b7fe700 1 -- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd7c0c26e0 msgr2=0x7fdd7c0c0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7b7fe700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd7c0c26e0 0x7fdd7c0c0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7b7fe700 1 -- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd7c0ac220 msgr2=0x7fdd7c0ba050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7b7fe700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd7c0ac220 0x7fdd7c0ba050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:53.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7b7fe700 1 -- 172.21.15.7:0/2748553081 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd74009b30 con 0x7fdd7c0ace30 2023-03-03T11:24:53.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7bfff700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd7c0ac220 0x7fdd7c0ba050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:24:53.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd7b7fe700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0ace30 0x7fdd7c151400 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400e810 tx=0x7fdd6400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:24:53.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd78ff9700 1 -- 172.21.15.7:0/2748553081 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64013610 con 0x7fdd7c0ace30 2023-03-03T11:24:53.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd78ff9700 1 -- 172.21.15.7:0/2748553081 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd64013770 con 0x7fdd7c0ace30 2023-03-03T11:24:53.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd82a38700 1 -- 172.21.15.7:0/2748553081 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7c155ca0 con 0x7fdd7c0ace30 2023-03-03T11:24:53.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd78ff9700 1 -- 172.21.15.7:0/2748553081 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64013b20 con 0x7fdd7c0ace30 2023-03-03T11:24:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.438+0000 7fdd82a38700 1 -- 172.21.15.7:0/2748553081 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7c1559c0 con 0x7fdd7c0ace30 2023-03-03T11:24:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.439+0000 7fdd82a38700 1 -- 172.21.15.7:0/2748553081 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7c155f80 con 0x7fdd7c0ace30 2023-03-03T11:24:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.440+0000 7fdd78ff9700 1 -- 172.21.15.7:0/2748553081 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdd64023070 con 0x7fdd7c0ace30 2023-03-03T11:24:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.440+0000 7fdd78ff9700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd5c0682f0 0x7fdd5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:24:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.440+0000 7fdd78ff9700 1 -- 172.21.15.7:0/2748553081 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdd64085ea0 con 0x7fdd7c0ace30 2023-03-03T11:24:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.440+0000 7fdd7bfff700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd5c0682f0 0x7fdd5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:24:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.441+0000 7fdd7bfff700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd5c0682f0 0x7fdd5c06a7a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c003fc0 tx=0x7fdd6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:24:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.445+0000 7fdd78ff9700 1 -- 172.21.15.7:0/2748553081 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdd640521c0 con 0x7fdd7c0ace30 2023-03-03T11:24:53.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.680+0000 7fdd82a38700 1 -- 172.21.15.7:0/2748553081 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd7c074bb0 con 0x7fdd7c0ace30 2023-03-03T11:24:53.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.681+0000 7fdd78ff9700 1 -- 172.21.15.7:0/2748553081 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdd640557e0 con 0x7fdd7c0ace30 2023-03-03T11:24:53.683 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:24:53.683 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:24:53.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 -- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd5c0682f0 msgr2=0x7fdd5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:53.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd5c0682f0 0x7fdd5c06a7a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c003fc0 tx=0x7fdd6c009040 comp rx=0 tx=0).stop 2023-03-03T11:24:53.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 -- 172.21.15.7:0/2748553081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0ace30 msgr2=0x7fdd7c151400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:24:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0ace30 0x7fdd7c151400 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400e810 tx=0x7fdd6400ebd0 comp rx=0 tx=0).stop 2023-03-03T11:24:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 -- 172.21.15.7:0/2748553081 shutdown_connections 2023-03-03T11:24:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd7c0c26e0 0x7fdd7c0c0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd7c0ac220 0x7fdd7c0ba050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdd5c0682f0 0x7fdd5c06a7a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 --2- 172.21.15.7:0/2748553081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd7c0ace30 0x7fdd7c151400 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:24:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 -- 172.21.15.7:0/2748553081 >> 172.21.15.7:0/2748553081 conn(0x7fdd7c0a94d0 msgr2=0x7fdd7c073e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:24:53.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 -- 172.21.15.7:0/2748553081 shutdown_connections 2023-03-03T11:24:53.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:24:53.683+0000 7fdd82a38700 1 -- 172.21.15.7:0/2748553081 wait complete. 2023-03-03T11:24:53.687 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:24:53.697 INFO:tasks.mds_thrash.fs.[a]:waiting for 111 secs before thrashing 2023-03-03T11:25:01.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:25:02.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.021+0000 7fa598354700 1 Processor -- start 2023-03-03T11:25:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.021+0000 7fa598354700 1 -- start start 2023-03-03T11:25:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa598354700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa590151ab0 0x7fa5900736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa598354700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa590073c30 0x7fa59006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa598354700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa59006c3e0 0x7fa590157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa598354700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa59006d780 con 0x7fa590073c30 2023-03-03T11:25:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa598354700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa59006cb70 con 0x7fa590151ab0 2023-03-03T11:25:02.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa598354700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59006ce50 con 0x7fa59006c3e0 2023-03-03T11:25:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa5958ef700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa590073c30 0x7fa59006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa5958ef700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa590073c30 0x7fa59006bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36214/0 (socket says 172.21.15.7:36214) 2023-03-03T11:25:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa5958ef700 1 -- 172.21.15.7:0/1570483999 learned_addr learned my addr 172.21.15.7:0/1570483999 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:25:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa5960f0700 1 --2- 172.21.15.7:0/1570483999 >> v2:172.21.15.143:3300/0 conn(0x7fa590151ab0 0x7fa5900736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.022+0000 7fa5968f1700 1 --2- 172.21.15.7:0/1570483999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa59006c3e0 0x7fa590157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.023+0000 7fa5958ef700 1 -- 172.21.15.7:0/1570483999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa59006c3e0 msgr2=0x7fa590157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:25:02.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.023+0000 7fa5958ef700 1 --2- 172.21.15.7:0/1570483999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa59006c3e0 0x7fa590157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.023+0000 7fa5958ef700 1 -- 172.21.15.7:0/1570483999 >> v2:172.21.15.143:3300/0 conn(0x7fa590151ab0 msgr2=0x7fa5900736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:25:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.023+0000 7fa5958ef700 1 --2- 172.21.15.7:0/1570483999 >> v2:172.21.15.143:3300/0 conn(0x7fa590151ab0 0x7fa5900736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.023+0000 7fa5958ef700 1 -- 172.21.15.7:0/1570483999 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59006d130 con 0x7fa590073c30 2023-03-03T11:25:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.023+0000 7fa5958ef700 1 --2- 172.21.15.7:0/1570483999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 0x7fa59006bea0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fa59006bb20 tx=0x7fa58c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a80f7b4280a5b1c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:25:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.023+0000 7fa5950ee700 1 -- 172.21.15.7:0/1570483999 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c011830 con 0x7fa590073c30 2023-03-03T11:25:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.023+0000 7fa5950ee700 1 -- 172.21.15.7:0/1570483999 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58c011990 con 0x7fa590073c30 2023-03-03T11:25:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.023+0000 7fa5950ee700 1 -- 172.21.15.7:0/1570483999 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c011d20 con 0x7fa590073c30 2023-03-03T11:25:02.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.024+0000 7fa598354700 1 -- 172.21.15.7:0/1570483999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 msgr2=0x7fa59006bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.024+0000 7fa598354700 1 --2- 172.21.15.7:0/1570483999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 0x7fa59006bea0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fa59006bb20 tx=0x7fa58c009bd0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.024+0000 7fa598354700 1 -- 172.21.15.7:0/1570483999 shutdown_connections 2023-03-03T11:25:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.024+0000 7fa598354700 1 --2- 172.21.15.7:0/1570483999 >> v2:172.21.15.143:3300/0 conn(0x7fa590151ab0 0x7fa5900736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.024+0000 7fa598354700 1 --2- 172.21.15.7:0/1570483999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa59006c3e0 0x7fa590157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.024+0000 7fa598354700 1 --2- 172.21.15.7:0/1570483999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 0x7fa59006bea0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.024+0000 7fa598354700 1 -- 172.21.15.7:0/1570483999 >> 172.21.15.7:0/1570483999 conn(0x7fa5900a94d0 msgr2=0x7fa5900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:25:02.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.024+0000 7fa598354700 1 -- 172.21.15.7:0/1570483999 shutdown_connections 2023-03-03T11:25:02.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.024+0000 7fa598354700 1 -- 172.21.15.7:0/1570483999 wait complete. 2023-03-03T11:25:02.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.025+0000 7fa598354700 1 Processor -- start 2023-03-03T11:25:02.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.025+0000 7fa598354700 1 -- start start 2023-03-03T11:25:02.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.026+0000 7fa598354700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa59006c3e0 0x7fa5901ed040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:02.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.026+0000 7fa598354700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 0x7fa590146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:02.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.026+0000 7fa598354700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa590151ab0 0x7fa590146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:02.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.026+0000 7fa598354700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5901eaf40 con 0x7fa590073c30 2023-03-03T11:25:02.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.026+0000 7fa598354700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa590146b90 con 0x7fa590151ab0 2023-03-03T11:25:02.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.026+0000 7fa598354700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5900be430 con 0x7fa59006c3e0 2023-03-03T11:25:02.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.026+0000 7fa5958ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 0x7fa590146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:02.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa5958ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 0x7fa590146110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36222/0 (socket says 172.21.15.7:36222) 2023-03-03T11:25:02.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa5958ef700 1 -- 172.21.15.7:0/571426805 learned_addr learned my addr 172.21.15.7:0/571426805 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:25:02.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa5960f0700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa59006c3e0 0x7fa5901ed040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:02.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa5968f1700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa590151ab0 0x7fa590146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:02.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa5958ef700 1 -- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa59006c3e0 msgr2=0x7fa5901ed040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:02.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa5958ef700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa59006c3e0 0x7fa5901ed040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa5958ef700 1 -- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa590151ab0 msgr2=0x7fa590146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:02.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa5958ef700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa590151ab0 0x7fa590146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa5958ef700 1 -- 172.21.15.7:0/571426805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c009550 con 0x7fa590073c30 2023-03-03T11:25:02.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa5958ef700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 0x7fa590146110 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fa58c000c00 tx=0x7fa58c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:25:02.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa586ffd700 1 -- 172.21.15.7:0/571426805 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c004240 con 0x7fa590073c30 2023-03-03T11:25:02.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa586ffd700 1 -- 172.21.15.7:0/571426805 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58c01f070 con 0x7fa590073c30 2023-03-03T11:25:02.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa598354700 1 -- 172.21.15.7:0/571426805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa590151e80 con 0x7fa590073c30 2023-03-03T11:25:02.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa586ffd700 1 -- 172.21.15.7:0/571426805 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c0036a0 con 0x7fa590073c30 2023-03-03T11:25:02.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.027+0000 7fa598354700 1 -- 172.21.15.7:0/571426805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59006d410 con 0x7fa590073c30 2023-03-03T11:25:02.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.028+0000 7fa598354700 1 -- 172.21.15.7:0/571426805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa590152010 con 0x7fa590073c30 2023-03-03T11:25:02.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.029+0000 7fa586ffd700 1 -- 172.21.15.7:0/571426805 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa58c02a020 con 0x7fa590073c30 2023-03-03T11:25:02.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.029+0000 7fa586ffd700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5740682f0 0x7fa57406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:02.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.029+0000 7fa586ffd700 1 -- 172.21.15.7:0/571426805 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa58c087440 con 0x7fa590073c30 2023-03-03T11:25:02.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.029+0000 7fa5960f0700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5740682f0 0x7fa57406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:02.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.030+0000 7fa5960f0700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5740682f0 0x7fa57406a7a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa590155ea0 tx=0x7fa580009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:25:02.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.033+0000 7fa586ffd700 1 -- 172.21.15.7:0/571426805 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa58c053760 con 0x7fa590073c30 2023-03-03T11:25:02.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.270+0000 7fa598354700 1 -- 172.21.15.7:0/571426805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5901eb0f0 con 0x7fa590073c30 2023-03-03T11:25:02.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.271+0000 7fa586ffd700 1 -- 172.21.15.7:0/571426805 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa58c015090 con 0x7fa590073c30 2023-03-03T11:25:02.273 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:25:02.273 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:25:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 -- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5740682f0 msgr2=0x7fa57406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5740682f0 0x7fa57406a7a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa590155ea0 tx=0x7fa580009210 comp rx=0 tx=0).stop 2023-03-03T11:25:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 -- 172.21.15.7:0/571426805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 msgr2=0x7fa590146110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 0x7fa590146110 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fa58c000c00 tx=0x7fa58c004750 comp rx=0 tx=0).stop 2023-03-03T11:25:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 -- 172.21.15.7:0/571426805 shutdown_connections 2023-03-03T11:25:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa590151ab0 0x7fa590146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa59006c3e0 0x7fa5901ed040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5740682f0 0x7fa57406a7a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 --2- 172.21.15.7:0/571426805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa590073c30 0x7fa590146110 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 -- 172.21.15.7:0/571426805 >> 172.21.15.7:0/571426805 conn(0x7fa5900a94d0 msgr2=0x7fa5900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:25:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 -- 172.21.15.7:0/571426805 shutdown_connections 2023-03-03T11:25:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:02.273+0000 7fa598354700 1 -- 172.21.15.7:0/571426805 wait complete. 2023-03-03T11:25:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:25:02.287 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before thrashing 2023-03-03T11:25:11.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:25:11.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.425+0000 7f7484ded700 1 Processor -- start 2023-03-03T11:25:11.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.425+0000 7f7484ded700 1 -- start start 2023-03-03T11:25:11.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.426+0000 7f7484ded700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7480151cd0 0x7f7480154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:11.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.426+0000 7f7484ded700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f748006c6c0 0x7f74800736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:11.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.426+0000 7f7484ded700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7480157ff0 con 0x7f74800c2e30 2023-03-03T11:25:11.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.426+0000 7f7484ded700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748006d640 con 0x7f748006c6c0 2023-03-03T11:25:11.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.426+0000 7f7484ded700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748006cda0 con 0x7f7480151cd0 2023-03-03T11:25:11.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.426+0000 7f747ed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f748006c6c0 0x7f74800736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:11.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.426+0000 7f747dd9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f74800c2e30 0x7f7480158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58446/0 (socket says 172.21.15.7:58446) 2023-03-03T11:25:11.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.426+0000 7f747ed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f748006c6c0 0x7f74800736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36234/0 (socket says 172.21.15.7:36234) 2023-03-03T11:25:11.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.426+0000 7f747dd9b700 1 -- 172.21.15.7:0/1004261676 learned_addr learned my addr 172.21.15.7:0/1004261676 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:25:11.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747e59c700 1 --2- 172.21.15.7:0/1004261676 >> v2:172.21.15.143:3300/0 conn(0x7f7480151cd0 0x7f7480154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:11.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747ed9d700 1 -- 172.21.15.7:0/1004261676 >> v2:172.21.15.143:3300/0 conn(0x7f7480151cd0 msgr2=0x7f7480154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:25:11.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747ed9d700 1 --2- 172.21.15.7:0/1004261676 >> v2:172.21.15.143:3300/0 conn(0x7f7480151cd0 0x7f7480154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:11.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747ed9d700 1 -- 172.21.15.7:0/1004261676 >> v1:172.21.15.7:6789/0 conn(0x7f74800c2e30 legacy=0x7f7480158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:11.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747d59a700 1 -- 172.21.15.7:0/1004261676 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581771930 0 0) 0x7f7480157ff0 con 0x7f74800c2e30 2023-03-03T11:25:11.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747ed9d700 1 -- 172.21.15.7:0/1004261676 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748006d080 con 0x7f748006c6c0 2023-03-03T11:25:11.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747e59c700 1 --2- 172.21.15.7:0/1004261676 >> v2:172.21.15.143:3300/0 conn(0x7f7480151cd0 0x7f7480154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:25:11.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747ed9d700 1 --2- 172.21.15.7:0/1004261676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f748006c6c0 0x7f74800736f0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f748006f340 tx=0x7f747800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0172ef1440b68 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:25:11.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747d59a700 1 -- 172.21.15.7:0/1004261676 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747800e070 con 0x7f748006c6c0 2023-03-03T11:25:11.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747d59a700 1 -- 172.21.15.7:0/1004261676 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747800bbd0 con 0x7f748006c6c0 2023-03-03T11:25:11.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.427+0000 7f747d59a700 1 -- 172.21.15.7:0/1004261676 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478012430 con 0x7f748006c6c0 2023-03-03T11:25:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.428+0000 7f7484ded700 1 -- 172.21.15.7:0/1004261676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f748006c6c0 msgr2=0x7f74800736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.428+0000 7f7484ded700 1 --2- 172.21.15.7:0/1004261676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f748006c6c0 0x7f74800736f0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f748006f340 tx=0x7f747800b330 comp rx=0 tx=0).stop 2023-03-03T11:25:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.428+0000 7f7484ded700 1 -- 172.21.15.7:0/1004261676 shutdown_connections 2023-03-03T11:25:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.428+0000 7f7484ded700 1 --2- 172.21.15.7:0/1004261676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f748006c6c0 0x7f74800736f0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.428+0000 7f7484ded700 1 --2- 172.21.15.7:0/1004261676 >> v2:172.21.15.143:3300/0 conn(0x7f7480151cd0 0x7f7480154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.428+0000 7f7484ded700 1 -- 172.21.15.7:0/1004261676 >> 172.21.15.7:0/1004261676 conn(0x7f74800a94d0 msgr2=0x7f74800a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:25:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.428+0000 7f7484ded700 1 -- 172.21.15.7:0/1004261676 shutdown_connections 2023-03-03T11:25:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.428+0000 7f7484ded700 1 -- 172.21.15.7:0/1004261676 wait complete. 2023-03-03T11:25:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.429+0000 7f7484ded700 1 Processor -- start 2023-03-03T11:25:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.429+0000 7f7484ded700 1 -- start start 2023-03-03T11:25:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f7484ded700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748006c6c0 0x7f74801e0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f7484ded700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74800c2e30 0x7f748014b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f7484ded700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7480151cd0 0x7f748014be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f7484ded700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74801e1d00 con 0x7f7480151cd0 2023-03-03T11:25:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f7484ded700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748014c790 con 0x7f74800c2e30 2023-03-03T11:25:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f7484ded700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748014aa60 con 0x7f748006c6c0 2023-03-03T11:25:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7480151cd0 0x7f748014be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7480151cd0 0x7f748014be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36242/0 (socket says 172.21.15.7:36242) 2023-03-03T11:25:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747ed9d700 1 -- 172.21.15.7:0/1612531361 learned_addr learned my addr 172.21.15.7:0/1612531361 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:25:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747dd9b700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74800c2e30 0x7f748014b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747e59c700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748006c6c0 0x7f74801e0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747ed9d700 1 -- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748006c6c0 msgr2=0x7f74801e0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747ed9d700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748006c6c0 0x7f74801e0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747ed9d700 1 -- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74800c2e30 msgr2=0x7f748014b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747ed9d700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74800c2e30 0x7f748014b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747ed9d700 1 -- 172.21.15.7:0/1612531361 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7478009a70 con 0x7f7480151cd0 2023-03-03T11:25:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.430+0000 7f747e59c700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748006c6c0 0x7f74801e0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:25:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.431+0000 7f747ed9d700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7480151cd0 0x7f748014be80 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f747800b370 tx=0x7f7478004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:25:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.431+0000 7f746effd700 1 -- 172.21.15.7:0/1612531361 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478012430 con 0x7f7480151cd0 2023-03-03T11:25:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.431+0000 7f746effd700 1 -- 172.21.15.7:0/1612531361 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7478020070 con 0x7f7480151cd0 2023-03-03T11:25:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.431+0000 7f746effd700 1 -- 172.21.15.7:0/1612531361 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7478004040 con 0x7f7480151cd0 2023-03-03T11:25:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.431+0000 7f7484ded700 1 -- 172.21.15.7:0/1612531361 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7480157c80 con 0x7f7480151cd0 2023-03-03T11:25:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.431+0000 7f7484ded700 1 -- 172.21.15.7:0/1612531361 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748006d360 con 0x7f7480151cd0 2023-03-03T11:25:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.432+0000 7f7484ded700 1 -- 172.21.15.7:0/1612531361 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7480154a50 con 0x7f7480151cd0 2023-03-03T11:25:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.432+0000 7f746effd700 1 -- 172.21.15.7:0/1612531361 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f747802e070 con 0x7f7480151cd0 2023-03-03T11:25:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.433+0000 7f746effd700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7460068320 0x7f746006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:25:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.433+0000 7f746effd700 1 -- 172.21.15.7:0/1612531361 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7478022070 con 0x7f7480151cd0 2023-03-03T11:25:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.433+0000 7f747e59c700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7460068320 0x7f746006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:25:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.434+0000 7f747e59c700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7460068320 0x7f746006a7d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f747000aa20 tx=0x7f7470008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:25:11.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.437+0000 7f746effd700 1 -- 172.21.15.7:0/1612531361 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7478054280 con 0x7f7480151cd0 2023-03-03T11:25:11.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.677+0000 7f7484ded700 1 -- 172.21.15.7:0/1612531361 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f748014b660 con 0x7f7480151cd0 2023-03-03T11:25:11.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.678+0000 7f746effd700 1 -- 172.21.15.7:0/1612531361 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f748014b660 con 0x7f7480151cd0 2023-03-03T11:25:11.680 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:25:11.680 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:25:11.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 -- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7460068320 msgr2=0x7f746006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:11.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7460068320 0x7f746006a7d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f747000aa20 tx=0x7f7470008040 comp rx=0 tx=0).stop 2023-03-03T11:25:11.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 -- 172.21.15.7:0/1612531361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7480151cd0 msgr2=0x7f748014be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:25:11.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7480151cd0 0x7f748014be80 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f747800b370 tx=0x7f7478004ad0 comp rx=0 tx=0).stop 2023-03-03T11:25:11.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 -- 172.21.15.7:0/1612531361 shutdown_connections 2023-03-03T11:25:11.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7460068320 0x7f746006a7d0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:11.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7480151cd0 0x7f748014be80 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:11.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748006c6c0 0x7f74801e0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:11.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 --2- 172.21.15.7:0/1612531361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74800c2e30 0x7f748014b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:25:11.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 -- 172.21.15.7:0/1612531361 >> 172.21.15.7:0/1612531361 conn(0x7f74800a94d0 msgr2=0x7f74800736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:25:11.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 -- 172.21.15.7:0/1612531361 shutdown_connections 2023-03-03T11:25:11.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:25:11.680+0000 7f7484ded700 1 -- 172.21.15.7:0/1612531361 wait complete. 2023-03-03T11:25:11.684 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:25:11.694 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T11:25:13.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:13.099 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:43.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:25:43.238 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:13.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:13.484 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:43.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:43.572 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:26:44.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:26:44.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.834+0000 7f26d2049700 1 Processor -- start 2023-03-03T11:26:44.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.834+0000 7f26d2049700 1 -- start start 2023-03-03T11:26:44.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.835+0000 7f26d2049700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc06c480 0x7f26cc1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:26:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.835+0000 7f26d2049700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26cc06d430 con 0x7f26cc151d40 2023-03-03T11:26:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.835+0000 7f26d2049700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26cc1586e0 con 0x7f26cc0c2930 2023-03-03T11:26:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.835+0000 7f26d2049700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc06c2a0 con 0x7f26cc06c480 2023-03-03T11:26:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.835+0000 7f26cb7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f26cc151d40 0x7f26cc0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58530/0 (socket says 172.21.15.7:58530) 2023-03-03T11:26:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.835+0000 7f26cb7fe700 1 -- 172.21.15.7:0/2002476088 learned_addr learned my addr 172.21.15.7:0/2002476088 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:26:44.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26cbfff700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc06c480 0x7f26cc1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:26:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407547368 0 0) 0x7f26cc06d430 con 0x7f26cc151d40 2023-03-03T11:26:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26b80035b0 con 0x7f26cc151d40 2023-03-03T11:26:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2889981882 0 0) 0x7f26b80035b0 con 0x7f26cc151d40 2023-03-03T11:26:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc06c480 msgr2=0x7f26cc1581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:26:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc06c480 0x7f26cc1581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:26:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 >> v1:172.21.15.143:6789/0 conn(0x7f26cc0c2930 legacy=0x7f26cc157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:26:44.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc06cb00 con 0x7f26cc151d40 2023-03-03T11:26:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f26b4001670 con 0x7f26cc151d40 2023-03-03T11:26:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 >> v1:172.21.15.7:6789/0 conn(0x7f26cc151d40 legacy=0x7f26cc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:26:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 0x7f26b8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:26:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b8004850 0x7f26b8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:26:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc06c480 0x7f26b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:26:44.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc06c2a0 con 0x7f26b8003aa0 2023-03-03T11:26:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8005560 con 0x7f26b8004850 2023-03-03T11:26:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b8005720 con 0x7f26cc06c480 2023-03-03T11:26:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26cb7fe700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 0x7f26b8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:26:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26cbfff700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc06c480 0x7f26b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:26:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26caffd700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b8004850 0x7f26b8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:26:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26cb7fe700 1 -- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc06c480 msgr2=0x7f26b800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:26:44.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26cb7fe700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc06c480 0x7f26b800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:26:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26cb7fe700 1 -- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b8004850 msgr2=0x7f26b8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:26:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26cb7fe700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b8004850 0x7f26b8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:26:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26cb7fe700 1 -- 172.21.15.7:0/2002476088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b80035b0 con 0x7f26b8003aa0 2023-03-03T11:26:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.836+0000 7f26caffd700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26b8004850 0x7f26b8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:26:44.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26cb7fe700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 0x7f26b8003eb0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f26b400c240 tx=0x7f26b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37fe63fb5cec859 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:26:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b4007c10 con 0x7f26b8003aa0 2023-03-03T11:26:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b400ce00 con 0x7f26b8003aa0 2023-03-03T11:26:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26ca7fc700 1 -- 172.21.15.7:0/2002476088 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b4005ba0 con 0x7f26b8003aa0 2023-03-03T11:26:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26d2049700 1 -- 172.21.15.7:0/2002476088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 msgr2=0x7f26b8003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:26:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26d2049700 1 --2- 172.21.15.7:0/2002476088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 0x7f26b8003eb0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f26b400c240 tx=0x7f26b8005880 comp rx=0 tx=0).stop 2023-03-03T11:26:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26cbfff700 1 -- 172.21.15.7:0/2002476088 reap_dead start 2023-03-03T11:26:44.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26d2049700 1 -- 172.21.15.7:0/2002476088 shutdown_connections 2023-03-03T11:26:44.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26d2049700 1 -- 172.21.15.7:0/2002476088 >> 172.21.15.7:0/2002476088 conn(0x7f26cc0a94d0 msgr2=0x7f26cc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:26:44.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26d2049700 1 -- 172.21.15.7:0/2002476088 shutdown_connections 2023-03-03T11:26:44.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.837+0000 7f26d2049700 1 -- 172.21.15.7:0/2002476088 wait complete. 2023-03-03T11:26:44.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.839+0000 7f26d2049700 1 Processor -- start 2023-03-03T11:26:44.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.839+0000 7f26d2049700 1 -- start start 2023-03-03T11:26:44.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26d2049700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 0x7f26cc0bb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:26:44.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26d2049700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26cc14e3c0 0x7f26cc0bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:26:44.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26d2049700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc14d3b0 0x7f26cc14d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:26:44.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26d2049700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc14e870 con 0x7f26b8003aa0 2023-03-03T11:26:44.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26d2049700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc1523e0 con 0x7f26cc14e3c0 2023-03-03T11:26:44.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26d2049700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc0bbe30 con 0x7f26cc14d3b0 2023-03-03T11:26:44.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26cb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 0x7f26cc0bb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:26:44.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26cb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 0x7f26cc0bb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36330/0 (socket says 172.21.15.7:36330) 2023-03-03T11:26:44.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26cb7fe700 1 -- 172.21.15.7:0/410350711 learned_addr learned my addr 172.21.15.7:0/410350711 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:26:44.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26cb7fe700 1 -- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc14d3b0 msgr2=0x7f26cc14d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:26:44.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26caffd700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26cc14e3c0 0x7f26cc0bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:26:44.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26cbfff700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc14d3b0 0x7f26cc14d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:26:44.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26cb7fe700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc14d3b0 0x7f26cc14d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:26:44.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26cb7fe700 1 -- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26cc14e3c0 msgr2=0x7f26cc0bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:26:44.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26cb7fe700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26cc14e3c0 0x7f26cc0bfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:26:44.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.840+0000 7f26cb7fe700 1 -- 172.21.15.7:0/410350711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc06d0c0 con 0x7f26b8003aa0 2023-03-03T11:26:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.841+0000 7f26cb7fe700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 0x7f26cc0bb8f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f26b400c830 tx=0x7f26b4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:26:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.841+0000 7f26c3fff700 1 -- 172.21.15.7:0/410350711 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b4001630 con 0x7f26b8003aa0 2023-03-03T11:26:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.841+0000 7f26c3fff700 1 -- 172.21.15.7:0/410350711 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b4007bb0 con 0x7f26b8003aa0 2023-03-03T11:26:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.841+0000 7f26d2049700 1 -- 172.21.15.7:0/410350711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26cc06cde0 con 0x7f26b8003aa0 2023-03-03T11:26:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.841+0000 7f26c3fff700 1 -- 172.21.15.7:0/410350711 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b4007310 con 0x7f26b8003aa0 2023-03-03T11:26:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.841+0000 7f26d2049700 1 -- 172.21.15.7:0/410350711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26cc073e00 con 0x7f26b8003aa0 2023-03-03T11:26:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.842+0000 7f26d2049700 1 -- 172.21.15.7:0/410350711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26cc152960 con 0x7f26b8003aa0 2023-03-03T11:26:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.842+0000 7f26c3fff700 1 -- 172.21.15.7:0/410350711 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f26b4010070 con 0x7f26b8003aa0 2023-03-03T11:26:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.843+0000 7f26c3fff700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26ac0682f0 0x7f26ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:26:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.843+0000 7f26c3fff700 1 -- 172.21.15.7:0/410350711 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f26b40877b0 con 0x7f26b8003aa0 2023-03-03T11:26:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.843+0000 7f26caffd700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26ac0682f0 0x7f26ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:26:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.844+0000 7f26caffd700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26ac0682f0 0x7f26ac06a7a0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0ad470 tx=0x7f26bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:26:44.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:44.847+0000 7f26c3fff700 1 -- 172.21.15.7:0/410350711 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f26b4053b50 con 0x7f26b8003aa0 2023-03-03T11:26:45.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.084+0000 7f26d2049700 1 -- 172.21.15.7:0/410350711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26cc14dd70 con 0x7f26b8003aa0 2023-03-03T11:26:45.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.084+0000 7f26c3fff700 1 -- 172.21.15.7:0/410350711 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f26b4057170 con 0x7f26b8003aa0 2023-03-03T11:26:45.088 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:26:45.088 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:26:45.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 -- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26ac0682f0 msgr2=0x7f26ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:26:45.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26ac0682f0 0x7f26ac06a7a0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0ad470 tx=0x7f26bc009040 comp rx=0 tx=0).stop 2023-03-03T11:26:45.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 -- 172.21.15.7:0/410350711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 msgr2=0x7f26cc0bb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:26:45.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 0x7f26cc0bb8f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f26b400c830 tx=0x7f26b4001a00 comp rx=0 tx=0).stop 2023-03-03T11:26:45.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 -- 172.21.15.7:0/410350711 shutdown_connections 2023-03-03T11:26:45.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26cc14d3b0 0x7f26cc14d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:26:45.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26ac0682f0 0x7f26ac06a7a0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:26:45.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26b8003aa0 0x7f26cc0bb8f0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:26:45.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 --2- 172.21.15.7:0/410350711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26cc14e3c0 0x7f26cc0bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:26:45.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 -- 172.21.15.7:0/410350711 >> 172.21.15.7:0/410350711 conn(0x7f26cc0a94d0 msgr2=0x7f26cc0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:26:45.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.086+0000 7f26d2049700 1 -- 172.21.15.7:0/410350711 shutdown_connections 2023-03-03T11:26:45.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:26:45.087+0000 7f26d2049700 1 -- 172.21.15.7:0/410350711 wait complete. 2023-03-03T11:26:45.092 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:26:45.101 INFO:tasks.mds_thrash.fs.[a]:waiting for 87 secs before thrashing 2023-03-03T11:27:07.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:27:07.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.831+0000 7f3e1b387700 1 Processor -- start 2023-03-03T11:27:07.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.831+0000 7f3e1b387700 1 -- start start 2023-03-03T11:27:07.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.832+0000 7f3e1b387700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3e1414f9d0 0x7f3e14156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:07.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.832+0000 7f3e1b387700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e140abe20 0x7f3e141529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.832+0000 7f3e1b387700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e140c2f30 con 0x7f3e1414fda0 2023-03-03T11:27:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.832+0000 7f3e1b387700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e140c31d0 con 0x7f3e1414f9d0 2023-03-03T11:27:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.832+0000 7f3e1b387700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e141524a0 con 0x7f3e140abe20 2023-03-03T11:27:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.832+0000 7f3e18922700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3e1414fda0 0x7f3e14071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58558/0 (socket says 172.21.15.7:58558) 2023-03-03T11:27:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.832+0000 7f3e18922700 1 -- 172.21.15.7:0/1565261350 learned_addr learned my addr 172.21.15.7:0/1565261350 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:27:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e19924700 1 --2- 172.21.15.7:0/1565261350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e140abe20 0x7f3e141529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e19123700 1 --2- 172.21.15.7:0/1565261350 >> v2:172.21.15.143:3300/0 conn(0x7f3e1414f9d0 0x7f3e14156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e0bfff700 1 -- 172.21.15.7:0/1565261350 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254271491 0 0) 0x7f3e140c2f30 con 0x7f3e1414fda0 2023-03-03T11:27:07.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e0bfff700 1 -- 172.21.15.7:0/1565261350 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e000035b0 con 0x7f3e1414fda0 2023-03-03T11:27:07.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e19924700 1 -- 172.21.15.7:0/1565261350 >> v2:172.21.15.143:3300/0 conn(0x7f3e1414f9d0 msgr2=0x7f3e14156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:27:07.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e19924700 1 --2- 172.21.15.7:0/1565261350 >> v2:172.21.15.143:3300/0 conn(0x7f3e1414f9d0 0x7f3e14156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:07.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e0bfff700 1 -- 172.21.15.7:0/1565261350 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3700831029 0 0) 0x7f3e000035b0 con 0x7f3e1414fda0 2023-03-03T11:27:07.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e19924700 1 -- 172.21.15.7:0/1565261350 >> v1:172.21.15.7:6789/0 conn(0x7f3e1414fda0 legacy=0x7f3e14071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:07.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e19924700 1 -- 172.21.15.7:0/1565261350 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e140c34b0 con 0x7f3e140abe20 2023-03-03T11:27:07.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e19123700 1 --2- 172.21.15.7:0/1565261350 >> v2:172.21.15.143:3300/0 conn(0x7f3e1414f9d0 0x7f3e14156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:27:07.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e19924700 1 --2- 172.21.15.7:0/1565261350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e140abe20 0x7f3e141529c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009d90 tx=0x7f3e0c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f29066c00255be72 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:27:07.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e0bfff700 1 -- 172.21.15.7:0/1565261350 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c00e070 con 0x7f3e140abe20 2023-03-03T11:27:07.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e0bfff700 1 -- 172.21.15.7:0/1565261350 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0c003a20 con 0x7f3e140abe20 2023-03-03T11:27:07.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.833+0000 7f3e0bfff700 1 -- 172.21.15.7:0/1565261350 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e10001670 con 0x7f3e140abe20 2023-03-03T11:27:07.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.834+0000 7f3e1b387700 1 -- 172.21.15.7:0/1565261350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e140abe20 msgr2=0x7f3e141529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.834+0000 7f3e1b387700 1 --2- 172.21.15.7:0/1565261350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e140abe20 0x7f3e141529c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009d90 tx=0x7f3e0c00b330 comp rx=0 tx=0).stop 2023-03-03T11:27:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.834+0000 7f3e1b387700 1 -- 172.21.15.7:0/1565261350 shutdown_connections 2023-03-03T11:27:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.834+0000 7f3e1b387700 1 --2- 172.21.15.7:0/1565261350 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e140abe20 0x7f3e141529c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.834+0000 7f3e1b387700 1 --2- 172.21.15.7:0/1565261350 >> v2:172.21.15.143:3300/0 conn(0x7f3e1414f9d0 0x7f3e14156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.834+0000 7f3e1b387700 1 -- 172.21.15.7:0/1565261350 >> 172.21.15.7:0/1565261350 conn(0x7f3e140c8900 msgr2=0x7f3e140c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:27:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.834+0000 7f3e1b387700 1 -- 172.21.15.7:0/1565261350 shutdown_connections 2023-03-03T11:27:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.834+0000 7f3e1b387700 1 -- 172.21.15.7:0/1565261350 wait complete. 2023-03-03T11:27:07.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.835+0000 7f3e1b387700 1 Processor -- start 2023-03-03T11:27:07.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.835+0000 7f3e1b387700 1 -- start start 2023-03-03T11:27:07.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e1b387700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e140abe20 0x7f3e1406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:07.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e1b387700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e1414f9d0 0x7f3e140be400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:07.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e1b387700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e1414fda0 0x7f3e1414c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:07.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e1b387700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1406f840 con 0x7f3e140abe20 2023-03-03T11:27:07.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e1b387700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e141eac50 con 0x7f3e1414fda0 2023-03-03T11:27:07.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e1b387700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e140bfae0 con 0x7f3e1414f9d0 2023-03-03T11:27:07.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e19123700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e140abe20 0x7f3e1406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:07.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e19123700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e140abe20 0x7f3e1406f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36350/0 (socket says 172.21.15.7:36350) 2023-03-03T11:27:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e19123700 1 -- 172.21.15.7:0/2426934529 learned_addr learned my addr 172.21.15.7:0/2426934529 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:27:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e18922700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e1414f9d0 0x7f3e140be400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e19924700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e1414fda0 0x7f3e1414c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e19123700 1 -- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e1414f9d0 msgr2=0x7f3e140be400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e19123700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e1414f9d0 0x7f3e140be400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e19123700 1 -- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e1414fda0 msgr2=0x7f3e1414c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e19123700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e1414fda0 0x7f3e1414c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:07.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e19123700 1 -- 172.21.15.7:0/2426934529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e040094a0 con 0x7f3e140abe20 2023-03-03T11:27:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.836+0000 7f3e19123700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e140abe20 0x7f3e1406f1b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400ebe0 tx=0x7f3e0400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:27:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.837+0000 7f3e09ffb700 1 -- 172.21.15.7:0/2426934529 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0400c850 con 0x7f3e140abe20 2023-03-03T11:27:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.837+0000 7f3e09ffb700 1 -- 172.21.15.7:0/2426934529 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0400c9b0 con 0x7f3e140abe20 2023-03-03T11:27:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.837+0000 7f3e1b387700 1 -- 172.21.15.7:0/2426934529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0c009a70 con 0x7f3e140abe20 2023-03-03T11:27:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.837+0000 7f3e09ffb700 1 -- 172.21.15.7:0/2426934529 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0400cd40 con 0x7f3e140abe20 2023-03-03T11:27:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.837+0000 7f3e1b387700 1 -- 172.21.15.7:0/2426934529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e140ac350 con 0x7f3e140abe20 2023-03-03T11:27:07.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.838+0000 7f3e1b387700 1 -- 172.21.15.7:0/2426934529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e140ab630 con 0x7f3e140abe20 2023-03-03T11:27:07.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.838+0000 7f3e09ffb700 1 -- 172.21.15.7:0/2426934529 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3e04019030 con 0x7f3e140abe20 2023-03-03T11:27:07.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.839+0000 7f3e09ffb700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3df80682f0 0x7f3df806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:27:07.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.839+0000 7f3e09ffb700 1 -- 172.21.15.7:0/2426934529 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3e040860e0 con 0x7f3e140abe20 2023-03-03T11:27:07.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.839+0000 7f3e18922700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3df80682f0 0x7f3df806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:27:07.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.839+0000 7f3e18922700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3df80682f0 0x7f3df806a7a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3e141566b0 tx=0x7f3e10001230 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:27:07.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:07.843+0000 7f3e09ffb700 1 -- 172.21.15.7:0/2426934529 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3e04052480 con 0x7f3e140abe20 2023-03-03T11:27:08.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.082+0000 7f3e1b387700 1 -- 172.21.15.7:0/2426934529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e14071fa0 con 0x7f3e140abe20 2023-03-03T11:27:08.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.083+0000 7f3e09ffb700 1 -- 172.21.15.7:0/2426934529 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3e0400cea0 con 0x7f3e140abe20 2023-03-03T11:27:08.085 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:27:08.085 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:27:08.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.085+0000 7f3e1b387700 1 -- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3df80682f0 msgr2=0x7f3df806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:08.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.085+0000 7f3e1b387700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3df80682f0 0x7f3df806a7a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3e141566b0 tx=0x7f3e10001230 comp rx=0 tx=0).stop 2023-03-03T11:27:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.085+0000 7f3e1b387700 1 -- 172.21.15.7:0/2426934529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e140abe20 msgr2=0x7f3e1406f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:27:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.085+0000 7f3e1b387700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e140abe20 0x7f3e1406f1b0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400ebe0 tx=0x7f3e0400eef0 comp rx=0 tx=0).stop 2023-03-03T11:27:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.085+0000 7f3e1b387700 1 -- 172.21.15.7:0/2426934529 shutdown_connections 2023-03-03T11:27:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.085+0000 7f3e1b387700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e1414fda0 0x7f3e1414c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.085+0000 7f3e1b387700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3df80682f0 0x7f3df806a7a0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.085+0000 7f3e1b387700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e140abe20 0x7f3e1406f1b0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.085+0000 7f3e1b387700 1 --2- 172.21.15.7:0/2426934529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3e1414f9d0 0x7f3e140be400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:27:08.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.085+0000 7f3e1b387700 1 -- 172.21.15.7:0/2426934529 >> 172.21.15.7:0/2426934529 conn(0x7f3e140c8900 msgr2=0x7f3e140c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:27:08.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.086+0000 7f3e1b387700 1 -- 172.21.15.7:0/2426934529 shutdown_connections 2023-03-03T11:27:08.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:27:08.086+0000 7f3e1b387700 1 -- 172.21.15.7:0/2426934529 wait complete. 2023-03-03T11:27:08.089 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:27:08.099 INFO:tasks.mds_thrash.fs.[b]:waiting for 84 secs before thrashing 2023-03-03T11:27:13.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:13.710 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:43.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:27:43.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:12.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:28:12.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.240+0000 7f132ecba700 1 Processor -- start 2023-03-03T11:28:12.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.240+0000 7f132ecba700 1 -- start start 2023-03-03T11:28:12.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ecba700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f132814b0d0 0x7f1328071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:12.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ecba700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f13280aad20 0x7f1328157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:12.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ecba700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13280a9c20 con 0x7f1328071d40 2023-03-03T11:28:12.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ecba700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1328154f10 con 0x7f132814b0d0 2023-03-03T11:28:12.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ecba700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13280ac3c0 con 0x7f13280aad20 2023-03-03T11:28:12.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ca56700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f132814b0d0 0x7f1328071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:12.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ca56700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f132814b0d0 0x7f1328071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36408/0 (socket says 172.21.15.7:36408) 2023-03-03T11:28:12.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ca56700 1 -- 172.21.15.7:0/251998254 learned_addr learned my addr 172.21.15.7:0/251998254 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:28:12.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f1327fff700 1 --2- 172.21.15.7:0/251998254 >> v2:172.21.15.143:3300/0 conn(0x7f13280aad20 0x7f1328157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:12.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ca56700 1 -- 172.21.15.7:0/251998254 >> v2:172.21.15.143:3300/0 conn(0x7f13280aad20 msgr2=0x7f1328157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:28:12.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ca56700 1 --2- 172.21.15.7:0/251998254 >> v2:172.21.15.143:3300/0 conn(0x7f13280aad20 0x7f1328157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ca56700 1 -- 172.21.15.7:0/251998254 >> v1:172.21.15.143:6789/0 conn(0x7f1328071d40 legacy=0x7f1328158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:28:12.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ca56700 1 -- 172.21.15.7:0/251998254 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132806fc70 con 0x7f132814b0d0 2023-03-03T11:28:12.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f1327fff700 1 --2- 172.21.15.7:0/251998254 >> v2:172.21.15.143:3300/0 conn(0x7f13280aad20 0x7f1328157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:28:12.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f132ca56700 1 --2- 172.21.15.7:0/251998254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f132814b0d0 0x7f1328071800 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f13280c2860 tx=0x7f131c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d04ce86394f5a7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:28:12.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.241+0000 7f13277fe700 1 -- 172.21.15.7:0/251998254 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c004230 con 0x7f132814b0d0 2023-03-03T11:28:12.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.242+0000 7f13277fe700 1 -- 172.21.15.7:0/251998254 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c004390 con 0x7f132814b0d0 2023-03-03T11:28:12.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.242+0000 7f13277fe700 1 -- 172.21.15.7:0/251998254 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c0036a0 con 0x7f132814b0d0 2023-03-03T11:28:12.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.242+0000 7f132ecba700 1 -- 172.21.15.7:0/251998254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f132814b0d0 msgr2=0x7f1328071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:12.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.242+0000 7f132ecba700 1 --2- 172.21.15.7:0/251998254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f132814b0d0 0x7f1328071800 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f13280c2860 tx=0x7f131c009bd0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.242+0000 7f132ecba700 1 -- 172.21.15.7:0/251998254 shutdown_connections 2023-03-03T11:28:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.242+0000 7f132ecba700 1 --2- 172.21.15.7:0/251998254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f132814b0d0 0x7f1328071800 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.242+0000 7f132ecba700 1 --2- 172.21.15.7:0/251998254 >> v2:172.21.15.143:3300/0 conn(0x7f13280aad20 0x7f1328157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.242+0000 7f132ecba700 1 -- 172.21.15.7:0/251998254 >> 172.21.15.7:0/251998254 conn(0x7f13280c8900 msgr2=0x7f13280c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:28:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.242+0000 7f132ecba700 1 -- 172.21.15.7:0/251998254 shutdown_connections 2023-03-03T11:28:12.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.242+0000 7f132ecba700 1 -- 172.21.15.7:0/251998254 wait complete. 2023-03-03T11:28:12.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.244+0000 7f132ecba700 1 Processor -- start 2023-03-03T11:28:12.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.244+0000 7f132ecba700 1 -- start start 2023-03-03T11:28:12.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f132ecba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1328071d40 0x7f13281de4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:12.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f132ecba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13280aad20 0x7f1328155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:12.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f132ecba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f132814b0d0 0x7f1328145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:12.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f132ecba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13281ded80 con 0x7f13280aad20 2023-03-03T11:28:12.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f132ecba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1328155960 con 0x7f1328071d40 2023-03-03T11:28:12.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f132ecba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13281e0570 con 0x7f132814b0d0 2023-03-03T11:28:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f1327fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13280aad20 0x7f1328155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f1327fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13280aad20 0x7f1328155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36414/0 (socket says 172.21.15.7:36414) 2023-03-03T11:28:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f1327fff700 1 -- 172.21.15.7:0/227220785 learned_addr learned my addr 172.21.15.7:0/227220785 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:28:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f132ca56700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1328071d40 0x7f13281de4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f132d257700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f132814b0d0 0x7f1328145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f1327fff700 1 -- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f132814b0d0 msgr2=0x7f1328145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f1327fff700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f132814b0d0 0x7f1328145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f1327fff700 1 -- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1328071d40 msgr2=0x7f13281de4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:12.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f1327fff700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1328071d40 0x7f13281de4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f1327fff700 1 -- 172.21.15.7:0/227220785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c009550 con 0x7f13280aad20 2023-03-03T11:28:12.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f132ca56700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1328071d40 0x7f13281de4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:28:12.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.245+0000 7f1327fff700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13280aad20 0x7f1328155ba0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f131400c7f0 tx=0x7f131400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:28:12.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.246+0000 7f13257fa700 1 -- 172.21.15.7:0/227220785 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314004220 con 0x7f13280aad20 2023-03-03T11:28:12.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.246+0000 7f13257fa700 1 -- 172.21.15.7:0/227220785 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13140043a0 con 0x7f13280aad20 2023-03-03T11:28:12.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.246+0000 7f132ecba700 1 -- 172.21.15.7:0/227220785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f132806f6b0 con 0x7f13280aad20 2023-03-03T11:28:12.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.246+0000 7f13257fa700 1 -- 172.21.15.7:0/227220785 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13140038e0 con 0x7f13280aad20 2023-03-03T11:28:12.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.246+0000 7f132ecba700 1 -- 172.21.15.7:0/227220785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1328155160 con 0x7f13280aad20 2023-03-03T11:28:12.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.247+0000 7f132ecba700 1 -- 172.21.15.7:0/227220785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132806f990 con 0x7f13280aad20 2023-03-03T11:28:12.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.247+0000 7f13257fa700 1 -- 172.21.15.7:0/227220785 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1314020070 con 0x7f13280aad20 2023-03-03T11:28:12.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.247+0000 7f13257fa700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f130c0682f0 0x7f130c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:12.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.247+0000 7f13257fa700 1 -- 172.21.15.7:0/227220785 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1314085c70 con 0x7f13280aad20 2023-03-03T11:28:12.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.248+0000 7f132ca56700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f130c0682f0 0x7f130c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:12.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.248+0000 7f132ca56700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f130c0682f0 0x7f130c06a7a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f131c009c10 tx=0x7f131c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:28:12.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.252+0000 7f13257fa700 1 -- 172.21.15.7:0/227220785 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1314052010 con 0x7f13280aad20 2023-03-03T11:28:12.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.492+0000 7f132ecba700 1 -- 172.21.15.7:0/227220785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13280c60e0 con 0x7f13280aad20 2023-03-03T11:28:12.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.493+0000 7f13257fa700 1 -- 172.21.15.7:0/227220785 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1314055630 con 0x7f13280aad20 2023-03-03T11:28:12.495 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:28:12.495 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:28:12.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 -- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f130c0682f0 msgr2=0x7f130c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:12.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f130c0682f0 0x7f130c06a7a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f131c009c10 tx=0x7f131c008030 comp rx=0 tx=0).stop 2023-03-03T11:28:12.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 -- 172.21.15.7:0/227220785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13280aad20 msgr2=0x7f1328155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:12.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13280aad20 0x7f1328155ba0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f131400c7f0 tx=0x7f131400cbb0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 -- 172.21.15.7:0/227220785 shutdown_connections 2023-03-03T11:28:12.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f132814b0d0 0x7f1328145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1328071d40 0x7f13281de4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f130c0682f0 0x7f130c06a7a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 --2- 172.21.15.7:0/227220785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13280aad20 0x7f1328155ba0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:12.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 -- 172.21.15.7:0/227220785 >> 172.21.15.7:0/227220785 conn(0x7f13280c8900 msgr2=0x7f13280c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:28:12.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 -- 172.21.15.7:0/227220785 shutdown_connections 2023-03-03T11:28:12.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:12.495+0000 7f132ecba700 1 -- 172.21.15.7:0/227220785 wait complete. 2023-03-03T11:28:12.499 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:28:12.509 INFO:tasks.mds_thrash.fs.[a]:waiting for 51 secs before thrashing 2023-03-03T11:28:13.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:13.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:32.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:28:32.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.238+0000 7f901d0cb700 1 Processor -- start 2023-03-03T11:28:32.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.238+0000 7f901d0cb700 1 -- start start 2023-03-03T11:28:32.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.239+0000 7f901d0cb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f90181530d0 0x7f9018156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:32.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.239+0000 7f901d0cb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90180c2f30 con 0x7f901814f9d0 2023-03-03T11:28:32.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.239+0000 7f901d0cb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90180c31d0 con 0x7f901814fda0 2023-03-03T11:28:32.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.239+0000 7f901d0cb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90181524a0 con 0x7f90181530d0 2023-03-03T11:28:32.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.239+0000 7f901759e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f90181530d0 0x7f9018156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:32.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.239+0000 7f9016d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f901814f9d0 0x7f9018071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58678/0 (socket says 172.21.15.7:58678) 2023-03-03T11:28:32.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.239+0000 7f9016d9d700 1 -- 172.21.15.7:0/258451181 learned_addr learned my addr 172.21.15.7:0/258451181 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:28:32.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.240+0000 7f901759e700 1 -- 172.21.15.7:0/258451181 >> v1:172.21.15.143:6789/0 conn(0x7f901814fda0 legacy=0x7f90181529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:28:32.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.240+0000 7f901759e700 1 -- 172.21.15.7:0/258451181 >> v1:172.21.15.7:6789/0 conn(0x7f901814f9d0 legacy=0x7f9018071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:32.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.240+0000 7f901759e700 1 -- 172.21.15.7:0/258451181 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90180c34b0 con 0x7f90181530d0 2023-03-03T11:28:32.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.240+0000 7f9015d9b700 1 -- 172.21.15.7:0/258451181 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486345544 0 0) 0x7f90180c2f30 con 0x7f901814f9d0 2023-03-03T11:28:32.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.240+0000 7f901759e700 1 --2- 172.21.15.7:0/258451181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90181530d0 0x7f9018156430 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f9010009d90 tx=0x7f901000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5aec92575771ce4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:28:32.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.240+0000 7f9015d9b700 1 -- 172.21.15.7:0/258451181 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901000e070 con 0x7f90181530d0 2023-03-03T11:28:32.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.240+0000 7f9015d9b700 1 -- 172.21.15.7:0/258451181 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f901000bbd0 con 0x7f90181530d0 2023-03-03T11:28:32.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.240+0000 7f9015d9b700 1 -- 172.21.15.7:0/258451181 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9010012430 con 0x7f90181530d0 2023-03-03T11:28:32.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.241+0000 7f901d0cb700 1 -- 172.21.15.7:0/258451181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90181530d0 msgr2=0x7f9018156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:32.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.241+0000 7f901d0cb700 1 --2- 172.21.15.7:0/258451181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90181530d0 0x7f9018156430 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f9010009d90 tx=0x7f901000b330 comp rx=0 tx=0).stop 2023-03-03T11:28:32.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.241+0000 7f901d0cb700 1 -- 172.21.15.7:0/258451181 shutdown_connections 2023-03-03T11:28:32.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.241+0000 7f901d0cb700 1 --2- 172.21.15.7:0/258451181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90181530d0 0x7f9018156430 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:32.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.241+0000 7f901d0cb700 1 -- 172.21.15.7:0/258451181 >> 172.21.15.7:0/258451181 conn(0x7f90180c8900 msgr2=0x7f90180c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:28:32.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.241+0000 7f901d0cb700 1 -- 172.21.15.7:0/258451181 shutdown_connections 2023-03-03T11:28:32.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.241+0000 7f901d0cb700 1 -- 172.21.15.7:0/258451181 wait complete. 2023-03-03T11:28:32.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.243+0000 7f901d0cb700 1 Processor -- start 2023-03-03T11:28:32.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.243+0000 7f901d0cb700 1 -- start start 2023-03-03T11:28:32.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f901d0cb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901814f9d0 0x7f90181eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:32.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f901d0cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f901814fda0 0x7f9018151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f901d0cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90181530d0 0x7f90181deb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f901d0cb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901806d7f0 con 0x7f901814f9d0 2023-03-03T11:28:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f901d0cb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90181de3d0 con 0x7f90181530d0 2023-03-03T11:28:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f901d0cb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901806e400 con 0x7f901814fda0 2023-03-03T11:28:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f9016d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901814f9d0 0x7f90181eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f9016d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901814f9d0 0x7f90181eb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36472/0 (socket says 172.21.15.7:36472) 2023-03-03T11:28:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f9016d9d700 1 -- 172.21.15.7:0/816003435 learned_addr learned my addr 172.21.15.7:0/816003435 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:28:32.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f901759e700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90181530d0 0x7f90181deb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:32.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f901659c700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f901814fda0 0x7f9018151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:32.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f9016d9d700 1 -- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f901814fda0 msgr2=0x7f9018151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:32.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f9016d9d700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f901814fda0 0x7f9018151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:32.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f9016d9d700 1 -- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90181530d0 msgr2=0x7f90181deb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:32.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f9016d9d700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90181530d0 0x7f90181deb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:32.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f9016d9d700 1 -- 172.21.15.7:0/816003435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9010009a70 con 0x7f901814f9d0 2023-03-03T11:28:32.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f901759e700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90181530d0 0x7f90181deb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:28:32.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f9016d9d700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901814f9d0 0x7f90181eb330 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f900800c550 tx=0x7f900800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:28:32.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f900f7fe700 1 -- 172.21.15.7:0/816003435 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008004c80 con 0x7f901814f9d0 2023-03-03T11:28:32.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f900f7fe700 1 -- 172.21.15.7:0/816003435 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9008004de0 con 0x7f901814f9d0 2023-03-03T11:28:32.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.244+0000 7f900f7fe700 1 -- 172.21.15.7:0/816003435 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008007310 con 0x7f901814f9d0 2023-03-03T11:28:32.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.245+0000 7f901d0cb700 1 -- 172.21.15.7:0/816003435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90180ac350 con 0x7f901814f9d0 2023-03-03T11:28:32.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.245+0000 7f901d0cb700 1 -- 172.21.15.7:0/816003435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90180ab630 con 0x7f901814f9d0 2023-03-03T11:28:32.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.245+0000 7f901d0cb700 1 -- 172.21.15.7:0/816003435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9018071fa0 con 0x7f901814f9d0 2023-03-03T11:28:32.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.246+0000 7f900f7fe700 1 -- 172.21.15.7:0/816003435 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f900801a020 con 0x7f901814f9d0 2023-03-03T11:28:32.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.246+0000 7f900f7fe700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ff80682f0 0x7f8ff806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:28:32.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.247+0000 7f900f7fe700 1 -- 172.21.15.7:0/816003435 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9008085e50 con 0x7f901814f9d0 2023-03-03T11:28:32.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.247+0000 7f901659c700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ff80682f0 0x7f8ff806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:28:32.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.247+0000 7f901659c700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ff80682f0 0x7f8ff806a7a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f90181df6f0 tx=0x7f9000009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:28:32.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.251+0000 7f900f7fe700 1 -- 172.21.15.7:0/816003435 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9008052070 con 0x7f901814f9d0 2023-03-03T11:28:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.488+0000 7f901d0cb700 1 -- 172.21.15.7:0/816003435 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9018074bb0 con 0x7f901814f9d0 2023-03-03T11:28:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.489+0000 7f900f7fe700 1 -- 172.21.15.7:0/816003435 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9008087020 con 0x7f901814f9d0 2023-03-03T11:28:32.491 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:28:32.491 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:28:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 -- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ff80682f0 msgr2=0x7f8ff806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ff80682f0 0x7f8ff806a7a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f90181df6f0 tx=0x7f9000009040 comp rx=0 tx=0).stop 2023-03-03T11:28:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 -- 172.21.15.7:0/816003435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901814f9d0 msgr2=0x7f90181eb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:28:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901814f9d0 0x7f90181eb330 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f900800c550 tx=0x7f900800c910 comp rx=0 tx=0).stop 2023-03-03T11:28:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 -- 172.21.15.7:0/816003435 shutdown_connections 2023-03-03T11:28:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90181530d0 0x7f90181deb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8ff80682f0 0x7f8ff806a7a0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f901814f9d0 0x7f90181eb330 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 --2- 172.21.15.7:0/816003435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f901814fda0 0x7f9018151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:28:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 -- 172.21.15.7:0/816003435 >> 172.21.15.7:0/816003435 conn(0x7f90180c8900 msgr2=0x7f90180c7920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:28:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 -- 172.21.15.7:0/816003435 shutdown_connections 2023-03-03T11:28:32.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:28:32.491+0000 7f901d0cb700 1 -- 172.21.15.7:0/816003435 wait complete. 2023-03-03T11:28:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:28:32.505 INFO:tasks.mds_thrash.fs.[b]:waiting for 87 secs before thrashing 2023-03-03T11:28:44.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:28:44.050 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:03.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:29:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.646+0000 7fbd8fc9e700 1 Processor -- start 2023-03-03T11:29:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.646+0000 7fbd8fc9e700 1 -- start start 2023-03-03T11:29:03.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8fc9e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbd88152e30 0x7fbd880c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:03.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8fc9e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbd880c4910 0x7fbd88156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:03.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8fc9e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd881523b0 con 0x7fbd880beab0 2023-03-03T11:29:03.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8fc9e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd88072010 con 0x7fbd880c4910 2023-03-03T11:29:03.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8fc9e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd880aaaf0 con 0x7fbd88152e30 2023-03-03T11:29:03.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8e23b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbd880c4910 0x7fbd88156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:03.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8e23b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbd880c4910 0x7fbd88156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36502/0 (socket says 172.21.15.7:36502) 2023-03-03T11:29:03.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8e23b700 1 -- 172.21.15.7:0/1969277844 learned_addr learned my addr 172.21.15.7:0/1969277844 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:29:03.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8da3a700 1 --2- 172.21.15.7:0/1969277844 >> v2:172.21.15.143:3300/0 conn(0x7fbd88152e30 0x7fbd880c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:03.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8e23b700 1 -- 172.21.15.7:0/1969277844 >> v2:172.21.15.143:3300/0 conn(0x7fbd88152e30 msgr2=0x7fbd880c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:29:03.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8e23b700 1 --2- 172.21.15.7:0/1969277844 >> v2:172.21.15.143:3300/0 conn(0x7fbd88152e30 0x7fbd880c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:03.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8e23b700 1 -- 172.21.15.7:0/1969277844 >> v1:172.21.15.143:6789/0 conn(0x7fbd880beab0 legacy=0x7fbd88155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:03.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8e23b700 1 -- 172.21.15.7:0/1969277844 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd880aadd0 con 0x7fbd880c4910 2023-03-03T11:29:03.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.647+0000 7fbd8e23b700 1 --2- 172.21.15.7:0/1969277844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 0x7fbd88156490 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fbd880c60b0 tx=0x7fbd8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6028065c4735c7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:29:03.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8ca38700 1 -- 172.21.15.7:0/1969277844 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8000e070 con 0x7fbd880c4910 2023-03-03T11:29:03.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8ca38700 1 -- 172.21.15.7:0/1969277844 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8000b9c0 con 0x7fbd880c4910 2023-03-03T11:29:03.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8ca38700 1 -- 172.21.15.7:0/1969277844 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8000bd70 con 0x7fbd880c4910 2023-03-03T11:29:03.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1969277844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 msgr2=0x7fbd88156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:03.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8fc9e700 1 --2- 172.21.15.7:0/1969277844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 0x7fbd88156490 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fbd880c60b0 tx=0x7fbd8000b330 comp rx=0 tx=0).stop 2023-03-03T11:29:03.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1969277844 shutdown_connections 2023-03-03T11:29:03.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8fc9e700 1 --2- 172.21.15.7:0/1969277844 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 0x7fbd88156490 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:03.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8fc9e700 1 --2- 172.21.15.7:0/1969277844 >> v2:172.21.15.143:3300/0 conn(0x7fbd88152e30 0x7fbd880c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:03.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1969277844 >> 172.21.15.7:0/1969277844 conn(0x7fbd880736f0 msgr2=0x7fbd88073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:29:03.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1969277844 shutdown_connections 2023-03-03T11:29:03.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.648+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1969277844 wait complete. 2023-03-03T11:29:03.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8fc9e700 1 Processor -- start 2023-03-03T11:29:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8fc9e700 1 -- start start 2023-03-03T11:29:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8fc9e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd880beab0 0x7fbd8814dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8fc9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 0x7fbd881f1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8fc9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd88152e30 0x7fbd881f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8fc9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd880c7f30 con 0x7fbd880c4910 2023-03-03T11:29:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8fc9e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8806bad0 con 0x7fbd880beab0 2023-03-03T11:29:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8fc9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8806b510 con 0x7fbd88152e30 2023-03-03T11:29:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8d239700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 0x7fbd881f1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8d239700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 0x7fbd881f1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36506/0 (socket says 172.21.15.7:36506) 2023-03-03T11:29:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8d239700 1 -- 172.21.15.7:0/1684557326 learned_addr learned my addr 172.21.15.7:0/1684557326 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:29:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.650+0000 7fbd8da3a700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd880beab0 0x7fbd8814dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd8e23b700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd88152e30 0x7fbd881f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd8d239700 1 -- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd88152e30 msgr2=0x7fbd881f18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd8d239700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd88152e30 0x7fbd881f18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd8d239700 1 -- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd880beab0 msgr2=0x7fbd8814dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd8d239700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd880beab0 0x7fbd8814dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd8d239700 1 -- 172.21.15.7:0/1684557326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80009b30 con 0x7fbd880c4910 2023-03-03T11:29:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd8d239700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 0x7fbd881f1360 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800c450 tx=0x7fbd7800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:29:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd7e7fc700 1 -- 172.21.15.7:0/1684557326 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd780052d0 con 0x7fbd880c4910 2023-03-03T11:29:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd7e7fc700 1 -- 172.21.15.7:0/1684557326 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd78005430 con 0x7fbd880c4910 2023-03-03T11:29:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1684557326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd88153200 con 0x7fbd880c4910 2023-03-03T11:29:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd7e7fc700 1 -- 172.21.15.7:0/1684557326 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd780038c0 con 0x7fbd880c4910 2023-03-03T11:29:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.651+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1684557326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd88071c70 con 0x7fbd880c4910 2023-03-03T11:29:03.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.652+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1684557326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd88153390 con 0x7fbd880c4910 2023-03-03T11:29:03.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.653+0000 7fbd7e7fc700 1 -- 172.21.15.7:0/1684557326 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd78019030 con 0x7fbd880c4910 2023-03-03T11:29:03.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.653+0000 7fbd7e7fc700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd6c0682f0 0x7fbd6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:03.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.653+0000 7fbd7e7fc700 1 -- 172.21.15.7:0/1684557326 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbd78085320 con 0x7fbd880c4910 2023-03-03T11:29:03.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.653+0000 7fbd8da3a700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd6c0682f0 0x7fbd6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:03.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.654+0000 7fbd8da3a700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd6c0682f0 0x7fbd6c06a7a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fbd840099d0 tx=0x7fbd84008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:29:03.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.658+0000 7fbd7e7fc700 1 -- 172.21.15.7:0/1684557326 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbd780516c0 con 0x7fbd880c4910 2023-03-03T11:29:03.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.893+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1684557326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd88074bb0 con 0x7fbd880c4910 2023-03-03T11:29:03.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.893+0000 7fbd7e7fc700 1 -- 172.21.15.7:0/1684557326 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbd7800ce60 con 0x7fbd880c4910 2023-03-03T11:29:03.896 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:29:03.897 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:29:03.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.895+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd6c0682f0 msgr2=0x7fbd6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:03.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.895+0000 7fbd8fc9e700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd6c0682f0 0x7fbd6c06a7a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fbd840099d0 tx=0x7fbd84008040 comp rx=0 tx=0).stop 2023-03-03T11:29:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.896+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1684557326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 msgr2=0x7fbd881f1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.896+0000 7fbd8fc9e700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 0x7fbd881f1360 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800c450 tx=0x7fbd7800c760 comp rx=0 tx=0).stop 2023-03-03T11:29:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.896+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1684557326 shutdown_connections 2023-03-03T11:29:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.896+0000 7fbd8fc9e700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd88152e30 0x7fbd881f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.896+0000 7fbd8fc9e700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd880beab0 0x7fbd8814dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.896+0000 7fbd8fc9e700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd6c0682f0 0x7fbd6c06a7a0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:03.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.896+0000 7fbd8fc9e700 1 --2- 172.21.15.7:0/1684557326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd880c4910 0x7fbd881f1360 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:03.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.896+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1684557326 >> 172.21.15.7:0/1684557326 conn(0x7fbd880736f0 msgr2=0x7fbd8814f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:29:03.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.896+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1684557326 shutdown_connections 2023-03-03T11:29:03.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:03.896+0000 7fbd8fc9e700 1 -- 172.21.15.7:0/1684557326 wait complete. 2023-03-03T11:29:03.900 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:29:03.910 INFO:tasks.mds_thrash.fs.[a]:waiting for 46 secs before thrashing 2023-03-03T11:29:14.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:14.196 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:44.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:44.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:29:49.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:29:50.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.047+0000 7f433cc2b700 1 Processor -- start 2023-03-03T11:29:50.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.047+0000 7f433cc2b700 1 -- start start 2023-03-03T11:29:50.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.048+0000 7f433cc2b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 0x7f43380c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:50.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.048+0000 7f433cc2b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f433806e020 0x7f4338071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:50.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.048+0000 7f433cc2b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4338072520 0x7f4338151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:50.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.048+0000 7f433cc2b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4338070b90 con 0x7f4338072520 2023-03-03T11:29:50.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.048+0000 7f433cc2b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4338151320 con 0x7f433806e020 2023-03-03T11:29:50.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.048+0000 7f433cc2b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43380c4780 con 0x7f4338155390 2023-03-03T11:29:50.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.048+0000 7f4336d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4338072520 0x7f4338151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:50.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.048+0000 7f4336d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4338072520 0x7f4338151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36536/0 (socket says 172.21.15.7:36536) 2023-03-03T11:29:50.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.048+0000 7f4336d9d700 1 -- 172.21.15.7:0/3746554669 learned_addr learned my addr 172.21.15.7:0/3746554669 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:29:50.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f433659c700 1 --2- 172.21.15.7:0/3746554669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 0x7f43380c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:50.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f4335d9b700 1 --2- 172.21.15.7:0/3746554669 >> v2:172.21.15.143:3300/0 conn(0x7f433806e020 0x7f4338071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:50.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f4336d9d700 1 -- 172.21.15.7:0/3746554669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 msgr2=0x7f43380c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:29:50.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f4336d9d700 1 --2- 172.21.15.7:0/3746554669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 0x7f43380c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f4336d9d700 1 -- 172.21.15.7:0/3746554669 >> v2:172.21.15.143:3300/0 conn(0x7f433806e020 msgr2=0x7f4338071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:29:50.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f4336d9d700 1 --2- 172.21.15.7:0/3746554669 >> v2:172.21.15.143:3300/0 conn(0x7f433806e020 0x7f4338071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f4336d9d700 1 -- 172.21.15.7:0/3746554669 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4338154910 con 0x7f4338072520 2023-03-03T11:29:50.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f4336d9d700 1 --2- 172.21.15.7:0/3746554669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 0x7f4338151770 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f433806f090 tx=0x7f433000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60b4225921a7242a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:29:50.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f433559a700 1 -- 172.21.15.7:0/3746554669 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433000e070 con 0x7f4338072520 2023-03-03T11:29:50.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f433559a700 1 -- 172.21.15.7:0/3746554669 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433000b9c0 con 0x7f4338072520 2023-03-03T11:29:50.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.049+0000 7f433559a700 1 -- 172.21.15.7:0/3746554669 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433000bd70 con 0x7f4338072520 2023-03-03T11:29:50.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.050+0000 7f433cc2b700 1 -- 172.21.15.7:0/3746554669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 msgr2=0x7f4338151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:50.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.050+0000 7f433cc2b700 1 --2- 172.21.15.7:0/3746554669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 0x7f4338151770 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f433806f090 tx=0x7f433000b330 comp rx=0 tx=0).stop 2023-03-03T11:29:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.050+0000 7f433cc2b700 1 -- 172.21.15.7:0/3746554669 shutdown_connections 2023-03-03T11:29:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.050+0000 7f433cc2b700 1 --2- 172.21.15.7:0/3746554669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 0x7f4338151770 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.050+0000 7f433cc2b700 1 --2- 172.21.15.7:0/3746554669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 0x7f43380c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.050+0000 7f433cc2b700 1 --2- 172.21.15.7:0/3746554669 >> v2:172.21.15.143:3300/0 conn(0x7f433806e020 0x7f4338071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.050+0000 7f433cc2b700 1 -- 172.21.15.7:0/3746554669 >> 172.21.15.7:0/3746554669 conn(0x7f433806c110 msgr2=0x7f433806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:29:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.050+0000 7f433cc2b700 1 -- 172.21.15.7:0/3746554669 shutdown_connections 2023-03-03T11:29:50.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.050+0000 7f433cc2b700 1 -- 172.21.15.7:0/3746554669 wait complete. 2023-03-03T11:29:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.051+0000 7f433cc2b700 1 Processor -- start 2023-03-03T11:29:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.051+0000 7f433cc2b700 1 -- start start 2023-03-03T11:29:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f433cc2b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f433806e020 0x7f433814fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f433cc2b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 0x7f43380aad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f433cc2b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 0x7f43380c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f433cc2b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380ab840 con 0x7f4338072520 2023-03-03T11:29:50.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f433cc2b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433814f3b0 con 0x7f433806e020 2023-03-03T11:29:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f433cc2b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4338142c10 con 0x7f4338155390 2023-03-03T11:29:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4335d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 0x7f43380aad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4335d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 0x7f43380aad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36540/0 (socket says 172.21.15.7:36540) 2023-03-03T11:29:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4335d9b700 1 -- 172.21.15.7:0/1125786309 learned_addr learned my addr 172.21.15.7:0/1125786309 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:29:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f433659c700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f433806e020 0x7f433814fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4336d9d700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 0x7f43380c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4335d9b700 1 -- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 msgr2=0x7f43380c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:50.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4335d9b700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 0x7f43380c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4335d9b700 1 -- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f433806e020 msgr2=0x7f433814fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:50.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4335d9b700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f433806e020 0x7f433814fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4335d9b700 1 -- 172.21.15.7:0/1125786309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4330009b30 con 0x7f4338072520 2023-03-03T11:29:50.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4336d9d700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 0x7f43380c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:29:50.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f433659c700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f433806e020 0x7f433814fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:29:50.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.052+0000 7f4335d9b700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 0x7f43380aad20 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f432000e810 tx=0x7f432000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:29:50.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.053+0000 7f432effd700 1 -- 172.21.15.7:0/1125786309 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320013610 con 0x7f4338072520 2023-03-03T11:29:50.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.053+0000 7f432effd700 1 -- 172.21.15.7:0/1125786309 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4320013770 con 0x7f4338072520 2023-03-03T11:29:50.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.053+0000 7f433cc2b700 1 -- 172.21.15.7:0/1125786309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4338070540 con 0x7f4338072520 2023-03-03T11:29:50.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.053+0000 7f432effd700 1 -- 172.21.15.7:0/1125786309 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320013b00 con 0x7f4338072520 2023-03-03T11:29:50.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.053+0000 7f433cc2b700 1 -- 172.21.15.7:0/1125786309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43380702a0 con 0x7f4338072520 2023-03-03T11:29:50.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.054+0000 7f433cc2b700 1 -- 172.21.15.7:0/1125786309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4338070820 con 0x7f4338072520 2023-03-03T11:29:50.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.054+0000 7f432effd700 1 -- 172.21.15.7:0/1125786309 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4320023070 con 0x7f4338072520 2023-03-03T11:29:50.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.054+0000 7f432effd700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43180682f0 0x7f431806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:50.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.055+0000 7f432effd700 1 -- 172.21.15.7:0/1125786309 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4320085dd0 con 0x7f4338072520 2023-03-03T11:29:50.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.055+0000 7f433659c700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43180682f0 0x7f431806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:50.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.055+0000 7f433659c700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43180682f0 0x7f431806a7a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4328009820 tx=0x7f4328008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:29:50.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.059+0000 7f432effd700 1 -- 172.21.15.7:0/1125786309 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4320052170 con 0x7f4338072520 2023-03-03T11:29:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.298+0000 7f433cc2b700 1 -- 172.21.15.7:0/1125786309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f433814fcc0 con 0x7f4338072520 2023-03-03T11:29:50.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.298+0000 7f432effd700 1 -- 172.21.15.7:0/1125786309 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f432000f040 con 0x7f4338072520 2023-03-03T11:29:50.300 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:29:50.300 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:29:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 -- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43180682f0 msgr2=0x7f431806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43180682f0 0x7f431806a7a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4328009820 tx=0x7f4328008040 comp rx=0 tx=0).stop 2023-03-03T11:29:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 -- 172.21.15.7:0/1125786309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 msgr2=0x7f43380aad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 0x7f43380aad20 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f432000e810 tx=0x7f432000ebd0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 -- 172.21.15.7:0/1125786309 shutdown_connections 2023-03-03T11:29:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4338155390 0x7f43380c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f433806e020 0x7f433814fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43180682f0 0x7f431806a7a0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 --2- 172.21.15.7:0/1125786309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4338072520 0x7f43380aad20 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 -- 172.21.15.7:0/1125786309 >> 172.21.15.7:0/1125786309 conn(0x7f433806c110 msgr2=0x7f43380c2160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:29:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 -- 172.21.15.7:0/1125786309 shutdown_connections 2023-03-03T11:29:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:50.300+0000 7f433cc2b700 1 -- 172.21.15.7:0/1125786309 wait complete. 2023-03-03T11:29:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:29:50.314 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T11:29:59.505 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:29:59.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.641+0000 7f86410da700 1 Processor -- start 2023-03-03T11:29:59.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.641+0000 7f86410da700 1 -- start start 2023-03-03T11:29:59.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.641+0000 7f86410da700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f863c0ac220 0x7f863c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:59.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.641+0000 7f86410da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 0x7f863c15ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.641+0000 7f86410da700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863c0ac900 con 0x7f863c073c30 2023-03-03T11:29:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.641+0000 7f86410da700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f863c071e30 con 0x7f863c0ac220 2023-03-03T11:29:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.641+0000 7f86410da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863c0acb50 con 0x7f863c152b60 2023-03-03T11:29:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.642+0000 7f863ad9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f863c0ac220 0x7f863c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.642+0000 7f863b59e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 0x7f863c15ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.642+0000 7f863a59c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f863c073c30 0x7f863c152450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:34056/0 (socket says 172.21.15.7:34056) 2023-03-03T11:29:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.642+0000 7f863a59c700 1 -- 172.21.15.7:0/252794223 learned_addr learned my addr 172.21.15.7:0/252794223 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:29:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.642+0000 7f863b59e700 1 -- 172.21.15.7:0/252794223 >> v2:172.21.15.143:3300/0 conn(0x7f863c0ac220 msgr2=0x7f863c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:29:59.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.642+0000 7f863b59e700 1 --2- 172.21.15.7:0/252794223 >> v2:172.21.15.143:3300/0 conn(0x7f863c0ac220 0x7f863c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:59.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.642+0000 7f863b59e700 1 -- 172.21.15.7:0/252794223 >> v1:172.21.15.143:6789/0 conn(0x7f863c073c30 legacy=0x7f863c152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:59.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.642+0000 7f863b59e700 1 -- 172.21.15.7:0/252794223 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863c1559c0 con 0x7f863c152b60 2023-03-03T11:29:59.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.642+0000 7f863ad9d700 1 --2- 172.21.15.7:0/252794223 >> v2:172.21.15.143:3300/0 conn(0x7f863c0ac220 0x7f863c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:29:59.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.642+0000 7f863b59e700 1 --2- 172.21.15.7:0/252794223 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 0x7f863c15ccf0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8634009d90 tx=0x7f863400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=228d83fa6771bb01 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:29:59.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f8639d9b700 1 -- 172.21.15.7:0/252794223 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863400e070 con 0x7f863c152b60 2023-03-03T11:29:59.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f8639d9b700 1 -- 172.21.15.7:0/252794223 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863400bb20 con 0x7f863c152b60 2023-03-03T11:29:59.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f8639d9b700 1 -- 172.21.15.7:0/252794223 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8634012430 con 0x7f863c152b60 2023-03-03T11:29:59.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f86410da700 1 -- 172.21.15.7:0/252794223 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 msgr2=0x7f863c15ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f86410da700 1 --2- 172.21.15.7:0/252794223 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 0x7f863c15ccf0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8634009d90 tx=0x7f863400b330 comp rx=0 tx=0).stop 2023-03-03T11:29:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f86410da700 1 -- 172.21.15.7:0/252794223 shutdown_connections 2023-03-03T11:29:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f86410da700 1 --2- 172.21.15.7:0/252794223 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 0x7f863c15ccf0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f86410da700 1 --2- 172.21.15.7:0/252794223 >> v2:172.21.15.143:3300/0 conn(0x7f863c0ac220 0x7f863c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f86410da700 1 -- 172.21.15.7:0/252794223 >> 172.21.15.7:0/252794223 conn(0x7f863c0a94d0 msgr2=0x7f863c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:29:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f86410da700 1 -- 172.21.15.7:0/252794223 shutdown_connections 2023-03-03T11:29:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.643+0000 7f86410da700 1 -- 172.21.15.7:0/252794223 wait complete. 2023-03-03T11:29:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.645+0000 7f86410da700 1 Processor -- start 2023-03-03T11:29:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.645+0000 7f86410da700 1 -- start start 2023-03-03T11:29:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.645+0000 7f86410da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f863c073c30 0x7f863c0c5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.645+0000 7f86410da700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f863c0ac220 0x7f863c1513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.645+0000 7f86410da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 0x7f863c14b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f86410da700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c14dea0 con 0x7f863c073c30 2023-03-03T11:29:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f86410da700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c1ec470 con 0x7f863c0ac220 2023-03-03T11:29:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f86410da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863c14f380 con 0x7f863c152b60 2023-03-03T11:29:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f863c073c30 0x7f863c0c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f863c073c30 0x7f863c0c5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36564/0 (socket says 172.21.15.7:36564) 2023-03-03T11:29:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863ad9d700 1 -- 172.21.15.7:0/1113492137 learned_addr learned my addr 172.21.15.7:0/1113492137 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:29:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863a59c700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f863c0ac220 0x7f863c1513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863b59e700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 0x7f863c14b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863ad9d700 1 -- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 msgr2=0x7f863c14b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863ad9d700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 0x7f863c14b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863ad9d700 1 -- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f863c0ac220 msgr2=0x7f863c1513d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863ad9d700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f863c0ac220 0x7f863c1513d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863ad9d700 1 -- 172.21.15.7:0/1113492137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c0094a0 con 0x7f863c073c30 2023-03-03T11:29:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f863ad9d700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f863c073c30 0x7f863c0c5830 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f862c00ec10 tx=0x7f862c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:29:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f86237fe700 1 -- 172.21.15.7:0/1113492137 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c009760 con 0x7f863c073c30 2023-03-03T11:29:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f86237fe700 1 -- 172.21.15.7:0/1113492137 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862c00c850 con 0x7f863c073c30 2023-03-03T11:29:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f86410da700 1 -- 172.21.15.7:0/1113492137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8634009a70 con 0x7f863c073c30 2023-03-03T11:29:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.646+0000 7f86237fe700 1 -- 172.21.15.7:0/1113492137 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c00cbe0 con 0x7f863c073c30 2023-03-03T11:29:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.647+0000 7f86410da700 1 -- 172.21.15.7:0/1113492137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863c155f80 con 0x7f863c073c30 2023-03-03T11:29:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.647+0000 7f86410da700 1 -- 172.21.15.7:0/1113492137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863c155ca0 con 0x7f863c073c30 2023-03-03T11:29:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.648+0000 7f86237fe700 1 -- 172.21.15.7:0/1113492137 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f862c019030 con 0x7f863c073c30 2023-03-03T11:29:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.648+0000 7f86237fe700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86280682f0 0x7f862806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:29:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.648+0000 7f86237fe700 1 -- 172.21.15.7:0/1113492137 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f862c085f40 con 0x7f863c073c30 2023-03-03T11:29:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.649+0000 7f863a59c700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86280682f0 0x7f862806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:29:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.649+0000 7f863a59c700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86280682f0 0x7f862806a7a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f863c070740 tx=0x7f8630009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:29:59.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.653+0000 7f86237fe700 1 -- 172.21.15.7:0/1113492137 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f862c0522e0 con 0x7f863c073c30 2023-03-03T11:29:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.890+0000 7f86410da700 1 -- 172.21.15.7:0/1113492137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f863c156260 con 0x7f863c073c30 2023-03-03T11:29:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.890+0000 7f86237fe700 1 -- 172.21.15.7:0/1113492137 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f862c017090 con 0x7f863c073c30 2023-03-03T11:29:59.894 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:29:59.894 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:29:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.892+0000 7f86410da700 1 -- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86280682f0 msgr2=0x7f862806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.892+0000 7f86410da700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86280682f0 0x7f862806a7a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f863c070740 tx=0x7f8630009040 comp rx=0 tx=0).stop 2023-03-03T11:29:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.892+0000 7f86410da700 1 -- 172.21.15.7:0/1113492137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f863c073c30 msgr2=0x7f863c0c5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:29:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.892+0000 7f86410da700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f863c073c30 0x7f863c0c5830 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f862c00ec10 tx=0x7f862c00ef20 comp rx=0 tx=0).stop 2023-03-03T11:29:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.892+0000 7f86410da700 1 -- 172.21.15.7:0/1113492137 shutdown_connections 2023-03-03T11:29:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.892+0000 7f86410da700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f863c152b60 0x7f863c14b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.892+0000 7f86410da700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f86280682f0 0x7f862806a7a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.892+0000 7f86410da700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f863c073c30 0x7f863c0c5830 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.892+0000 7f86410da700 1 --2- 172.21.15.7:0/1113492137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f863c0ac220 0x7f863c1513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:29:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.893+0000 7f86410da700 1 -- 172.21.15.7:0/1113492137 >> 172.21.15.7:0/1113492137 conn(0x7f863c0a94d0 msgr2=0x7f863c0acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:29:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.893+0000 7f86410da700 1 -- 172.21.15.7:0/1113492137 shutdown_connections 2023-03-03T11:29:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:29:59.893+0000 7f86410da700 1 -- 172.21.15.7:0/1113492137 wait complete. 2023-03-03T11:29:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:29:59.907 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T11:30:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T11:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T11:30:10.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:30:11.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.047+0000 7ffb335d5700 1 Processor -- start 2023-03-03T11:30:11.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.047+0000 7ffb335d5700 1 -- start start 2023-03-03T11:30:11.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb335d5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffb2c0c00e0 0x7ffb2c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:11.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb335d5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffb2c154ff0 0x7ffb2c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:11.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb335d5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c0c7310 con 0x7ffb2c0caa30 2023-03-03T11:30:11.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb335d5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb2c155430 con 0x7ffb2c154ff0 2023-03-03T11:30:11.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb335d5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb2c0bfd10 con 0x7ffb2c0c00e0 2023-03-03T11:30:11.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb31b72700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffb2c154ff0 0x7ffb2c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:11.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb31b72700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffb2c154ff0 0x7ffb2c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36578/0 (socket says 172.21.15.7:36578) 2023-03-03T11:30:11.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb31b72700 1 -- 172.21.15.7:0/2261422106 learned_addr learned my addr 172.21.15.7:0/2261422106 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:30:11.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb30b70700 1 --2- 172.21.15.7:0/2261422106 >> v2:172.21.15.143:3300/0 conn(0x7ffb2c0c00e0 0x7ffb2c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:11.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb31b72700 1 -- 172.21.15.7:0/2261422106 >> v2:172.21.15.143:3300/0 conn(0x7ffb2c0c00e0 msgr2=0x7ffb2c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:30:11.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb31b72700 1 --2- 172.21.15.7:0/2261422106 >> v2:172.21.15.143:3300/0 conn(0x7ffb2c0c00e0 0x7ffb2c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb31b72700 1 -- 172.21.15.7:0/2261422106 >> v1:172.21.15.143:6789/0 conn(0x7ffb2c0caa30 legacy=0x7ffb2c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:30:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb31b72700 1 -- 172.21.15.7:0/2261422106 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2c0bfa30 con 0x7ffb2c154ff0 2023-03-03T11:30:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb31b72700 1 --2- 172.21.15.7:0/2261422106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 0x7ffb2c0cae00 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c0ca910 tx=0x7ffb2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961237e5b3e13ff7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb23fff700 1 -- 172.21.15.7:0/2261422106 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2400e070 con 0x7ffb2c154ff0 2023-03-03T11:30:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb23fff700 1 -- 172.21.15.7:0/2261422106 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2400b9c0 con 0x7ffb2c154ff0 2023-03-03T11:30:11.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.048+0000 7ffb23fff700 1 -- 172.21.15.7:0/2261422106 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2400bd70 con 0x7ffb2c154ff0 2023-03-03T11:30:11.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.049+0000 7ffb335d5700 1 -- 172.21.15.7:0/2261422106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 msgr2=0x7ffb2c0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:11.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.049+0000 7ffb335d5700 1 --2- 172.21.15.7:0/2261422106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 0x7ffb2c0cae00 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c0ca910 tx=0x7ffb2400b330 comp rx=0 tx=0).stop 2023-03-03T11:30:11.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.049+0000 7ffb335d5700 1 -- 172.21.15.7:0/2261422106 shutdown_connections 2023-03-03T11:30:11.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.049+0000 7ffb335d5700 1 --2- 172.21.15.7:0/2261422106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 0x7ffb2c0cae00 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:11.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.049+0000 7ffb335d5700 1 --2- 172.21.15.7:0/2261422106 >> v2:172.21.15.143:3300/0 conn(0x7ffb2c0c00e0 0x7ffb2c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:11.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.049+0000 7ffb335d5700 1 -- 172.21.15.7:0/2261422106 >> 172.21.15.7:0/2261422106 conn(0x7ffb2c066e40 msgr2=0x7ffb2c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:30:11.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.049+0000 7ffb335d5700 1 -- 172.21.15.7:0/2261422106 shutdown_connections 2023-03-03T11:30:11.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.049+0000 7ffb335d5700 1 -- 172.21.15.7:0/2261422106 wait complete. 2023-03-03T11:30:11.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.050+0000 7ffb335d5700 1 Processor -- start 2023-03-03T11:30:11.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.050+0000 7ffb335d5700 1 -- start start 2023-03-03T11:30:11.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb335d5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb2c0c00e0 0x7ffb2c1ed560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:11.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb335d5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2c0caa30 0x7ffb2c156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:11.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb335d5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 0x7ffb2c156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:11.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb335d5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c156ed0 con 0x7ffb2c154ff0 2023-03-03T11:30:11.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb335d5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c14c480 con 0x7ffb2c0caa30 2023-03-03T11:30:11.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb335d5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c0c1b60 con 0x7ffb2c0c00e0 2023-03-03T11:30:11.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb31b72700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 0x7ffb2c156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:11.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb31b72700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 0x7ffb2c156990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36584/0 (socket says 172.21.15.7:36584) 2023-03-03T11:30:11.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb31b72700 1 -- 172.21.15.7:0/3771592505 learned_addr learned my addr 172.21.15.7:0/3771592505 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:30:11.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb30b70700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2c0caa30 0x7ffb2c156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:11.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb31371700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb2c0c00e0 0x7ffb2c1ed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:11.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb31b72700 1 -- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb2c0c00e0 msgr2=0x7ffb2c1ed560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:11.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb31b72700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb2c0c00e0 0x7ffb2c1ed560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:11.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb31b72700 1 -- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2c0caa30 msgr2=0x7ffb2c156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:11.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb31b72700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2c0caa30 0x7ffb2c156450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:11.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb31b72700 1 -- 172.21.15.7:0/3771592505 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb24009b30 con 0x7ffb2c154ff0 2023-03-03T11:30:11.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb30b70700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2c0caa30 0x7ffb2c156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:30:11.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.051+0000 7ffb31371700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb2c0c00e0 0x7ffb2c1ed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:30:11.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.052+0000 7ffb31b72700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 0x7ffb2c156990 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c06b270 tx=0x7ffb24004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:11.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.052+0000 7ffb21ffb700 1 -- 172.21.15.7:0/3771592505 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb240099b0 con 0x7ffb2c154ff0 2023-03-03T11:30:11.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.052+0000 7ffb21ffb700 1 -- 172.21.15.7:0/3771592505 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb240092e0 con 0x7ffb2c154ff0 2023-03-03T11:30:11.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.052+0000 7ffb335d5700 1 -- 172.21.15.7:0/3771592505 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2c1547f0 con 0x7ffb2c154ff0 2023-03-03T11:30:11.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.052+0000 7ffb21ffb700 1 -- 172.21.15.7:0/3771592505 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2401b630 con 0x7ffb2c154ff0 2023-03-03T11:30:11.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.052+0000 7ffb335d5700 1 -- 172.21.15.7:0/3771592505 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2c154510 con 0x7ffb2c154ff0 2023-03-03T11:30:11.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.053+0000 7ffb335d5700 1 -- 172.21.15.7:0/3771592505 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2c154ad0 con 0x7ffb2c154ff0 2023-03-03T11:30:11.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.053+0000 7ffb21ffb700 1 -- 172.21.15.7:0/3771592505 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffb2402b070 con 0x7ffb2c154ff0 2023-03-03T11:30:11.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.054+0000 7ffb21ffb700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb100682f0 0x7ffb1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:11.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.054+0000 7ffb21ffb700 1 -- 172.21.15.7:0/3771592505 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ffb24087910 con 0x7ffb2c154ff0 2023-03-03T11:30:11.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.054+0000 7ffb31371700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb100682f0 0x7ffb1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:11.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.055+0000 7ffb31371700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb100682f0 0x7ffb1006a7a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ffb28000f60 tx=0x7ffb28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:11.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.059+0000 7ffb21ffb700 1 -- 172.21.15.7:0/3771592505 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffb24053cb0 con 0x7ffb2c154ff0 2023-03-03T11:30:11.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.294+0000 7ffb335d5700 1 -- 172.21.15.7:0/3771592505 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb2c074bb0 con 0x7ffb2c154ff0 2023-03-03T11:30:11.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.295+0000 7ffb21ffb700 1 -- 172.21.15.7:0/3771592505 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ffb240572d0 con 0x7ffb2c154ff0 2023-03-03T11:30:11.298 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:30:11.298 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:30:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 -- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb100682f0 msgr2=0x7ffb1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb100682f0 0x7ffb1006a7a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ffb28000f60 tx=0x7ffb28009040 comp rx=0 tx=0).stop 2023-03-03T11:30:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 -- 172.21.15.7:0/3771592505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 msgr2=0x7ffb2c156990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 0x7ffb2c156990 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c06b270 tx=0x7ffb24004ad0 comp rx=0 tx=0).stop 2023-03-03T11:30:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 -- 172.21.15.7:0/3771592505 shutdown_connections 2023-03-03T11:30:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffb100682f0 0x7ffb1006a7a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb2c154ff0 0x7ffb2c156990 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb2c0c00e0 0x7ffb2c1ed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 --2- 172.21.15.7:0/3771592505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb2c0caa30 0x7ffb2c156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 -- 172.21.15.7:0/3771592505 >> 172.21.15.7:0/3771592505 conn(0x7ffb2c066e40 msgr2=0x7ffb2c06b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:30:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 -- 172.21.15.7:0/3771592505 shutdown_connections 2023-03-03T11:30:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:11.297+0000 7ffb335d5700 1 -- 172.21.15.7:0/3771592505 wait complete. 2023-03-03T11:30:11.302 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:30:11.312 INFO:tasks.mds_thrash.fs.[b]:waiting for 25 secs before thrashing 2023-03-03T11:30:14.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:14.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:36.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:30:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.448+0000 7f92c8e18700 1 Processor -- start 2023-03-03T11:30:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.448+0000 7f92c8e18700 1 -- start start 2023-03-03T11:30:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.449+0000 7f92c8e18700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f92c414f9d0 0x7f92c4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.449+0000 7f92c8e18700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c40abe20 0x7f92c41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.449+0000 7f92c8e18700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c40c2f30 con 0x7f92c414fda0 2023-03-03T11:30:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.449+0000 7f92c8e18700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c40c31d0 con 0x7f92c414f9d0 2023-03-03T11:30:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.449+0000 7f92c8e18700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c41524a0 con 0x7f92c40abe20 2023-03-03T11:30:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.449+0000 7f92c259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f92c414f9d0 0x7f92c4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f92c414f9d0 0x7f92c4156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36610/0 (socket says 172.21.15.7:36610) 2023-03-03T11:30:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c259c700 1 -- 172.21.15.7:0/2988273778 learned_addr learned my addr 172.21.15.7:0/2988273778 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:30:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c2d9d700 1 --2- 172.21.15.7:0/2988273778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c40abe20 0x7f92c41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c259c700 1 -- 172.21.15.7:0/2988273778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c40abe20 msgr2=0x7f92c41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:30:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c259c700 1 --2- 172.21.15.7:0/2988273778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c40abe20 0x7f92c41529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c259c700 1 -- 172.21.15.7:0/2988273778 >> v1:172.21.15.143:6789/0 conn(0x7f92c414fda0 legacy=0x7f92c4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c259c700 1 -- 172.21.15.7:0/2988273778 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c40c34b0 con 0x7f92c414f9d0 2023-03-03T11:30:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c259c700 1 --2- 172.21.15.7:0/2988273778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414f9d0 0x7f92c4156430 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f92b40098f0 tx=0x7f92b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77e402ba558e0b9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c159a700 1 -- 172.21.15.7:0/2988273778 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b400b8a0 con 0x7f92c414f9d0 2023-03-03T11:30:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c159a700 1 -- 172.21.15.7:0/2988273778 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b400ba00 con 0x7f92c414f9d0 2023-03-03T11:30:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.450+0000 7f92c159a700 1 -- 172.21.15.7:0/2988273778 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b400bd90 con 0x7f92c414f9d0 2023-03-03T11:30:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.451+0000 7f92c8e18700 1 -- 172.21.15.7:0/2988273778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414f9d0 msgr2=0x7f92c4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.451+0000 7f92c8e18700 1 --2- 172.21.15.7:0/2988273778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414f9d0 0x7f92c4156430 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f92b40098f0 tx=0x7f92b4009c00 comp rx=0 tx=0).stop 2023-03-03T11:30:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.451+0000 7f92c8e18700 1 -- 172.21.15.7:0/2988273778 shutdown_connections 2023-03-03T11:30:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.451+0000 7f92c8e18700 1 --2- 172.21.15.7:0/2988273778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c40abe20 0x7f92c41529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.451+0000 7f92c8e18700 1 --2- 172.21.15.7:0/2988273778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414f9d0 0x7f92c4156430 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.451+0000 7f92c8e18700 1 -- 172.21.15.7:0/2988273778 >> 172.21.15.7:0/2988273778 conn(0x7f92c40c8900 msgr2=0x7f92c40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:30:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.451+0000 7f92c8e18700 1 -- 172.21.15.7:0/2988273778 shutdown_connections 2023-03-03T11:30:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.451+0000 7f92c8e18700 1 -- 172.21.15.7:0/2988273778 wait complete. 2023-03-03T11:30:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.453+0000 7f92c8e18700 1 Processor -- start 2023-03-03T11:30:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.453+0000 7f92c8e18700 1 -- start start 2023-03-03T11:30:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.453+0000 7f92c8e18700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c40abe20 0x7f92c40c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.453+0000 7f92c8e18700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c414f9d0 0x7f92c41ed790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.453+0000 7f92c8e18700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414fda0 0x7f92c41edcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.453+0000 7f92c8e18700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c41dd340 con 0x7f92c414fda0 2023-03-03T11:30:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c8e18700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c41ed300 con 0x7f92c40abe20 2023-03-03T11:30:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c8e18700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c414cf60 con 0x7f92c414f9d0 2023-03-03T11:30:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414fda0 0x7f92c41edcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414fda0 0x7f92c41edcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36620/0 (socket says 172.21.15.7:36620) 2023-03-03T11:30:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c2d9d700 1 -- 172.21.15.7:0/3218174688 learned_addr learned my addr 172.21.15.7:0/3218174688 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:30:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c259c700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c40abe20 0x7f92c40c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c1d9b700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c414f9d0 0x7f92c41ed790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c2d9d700 1 -- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c414f9d0 msgr2=0x7f92c41ed790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c2d9d700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c414f9d0 0x7f92c41ed790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c2d9d700 1 -- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c40abe20 msgr2=0x7f92c40c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c2d9d700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c40abe20 0x7f92c40c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c2d9d700 1 -- 172.21.15.7:0/3218174688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b4009550 con 0x7f92c414fda0 2023-03-03T11:30:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92c2d9d700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414fda0 0x7f92c41edcd0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009fd0 tx=0x7f92bc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.454+0000 7f92b2ffd700 1 -- 172.21.15.7:0/3218174688 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc007ea0 con 0x7f92c414fda0 2023-03-03T11:30:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.455+0000 7f92b2ffd700 1 -- 172.21.15.7:0/3218174688 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc00ed30 con 0x7f92c414fda0 2023-03-03T11:30:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.455+0000 7f92c8e18700 1 -- 172.21.15.7:0/3218174688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c40ac350 con 0x7f92c414fda0 2023-03-03T11:30:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.455+0000 7f92b2ffd700 1 -- 172.21.15.7:0/3218174688 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc00eea0 con 0x7f92c414fda0 2023-03-03T11:30:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.455+0000 7f92c8e18700 1 -- 172.21.15.7:0/3218174688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c40ab630 con 0x7f92c414fda0 2023-03-03T11:30:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.456+0000 7f92c8e18700 1 -- 172.21.15.7:0/3218174688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c4071fa0 con 0x7f92c414fda0 2023-03-03T11:30:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.456+0000 7f92b2ffd700 1 -- 172.21.15.7:0/3218174688 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f92bc012050 con 0x7f92c414fda0 2023-03-03T11:30:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.456+0000 7f92b2ffd700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92a40682f0 0x7f92a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.456+0000 7f92b2ffd700 1 -- 172.21.15.7:0/3218174688 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f92bc0865d0 con 0x7f92c414fda0 2023-03-03T11:30:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.456+0000 7f92c259c700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92a40682f0 0x7f92a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.457+0000 7f92c259c700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92a40682f0 0x7f92a406a7a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009840 tx=0x7f92b4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:36.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.461+0000 7f92b2ffd700 1 -- 172.21.15.7:0/3218174688 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f92bc052970 con 0x7f92c414fda0 2023-03-03T11:30:36.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.699+0000 7f92c8e18700 1 -- 172.21.15.7:0/3218174688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92c41dcb00 con 0x7f92c414fda0 2023-03-03T11:30:36.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.700+0000 7f92b2ffd700 1 -- 172.21.15.7:0/3218174688 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f92bc055f90 con 0x7f92c414fda0 2023-03-03T11:30:36.702 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:30:36.702 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:30:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 -- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92a40682f0 msgr2=0x7f92a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92a40682f0 0x7f92a406a7a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009840 tx=0x7f92b4008030 comp rx=0 tx=0).stop 2023-03-03T11:30:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 -- 172.21.15.7:0/3218174688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414fda0 msgr2=0x7f92c41edcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414fda0 0x7f92c41edcd0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009fd0 tx=0x7f92bc00e3c0 comp rx=0 tx=0).stop 2023-03-03T11:30:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 -- 172.21.15.7:0/3218174688 shutdown_connections 2023-03-03T11:30:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92a40682f0 0x7f92a406a7a0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c414fda0 0x7f92c41edcd0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92c40abe20 0x7f92c40c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 --2- 172.21.15.7:0/3218174688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92c414f9d0 0x7f92c41ed790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 -- 172.21.15.7:0/3218174688 >> 172.21.15.7:0/3218174688 conn(0x7f92c40c8900 msgr2=0x7f92c40c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:30:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 -- 172.21.15.7:0/3218174688 shutdown_connections 2023-03-03T11:30:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:36.702+0000 7f92c8e18700 1 -- 172.21.15.7:0/3218174688 wait complete. 2023-03-03T11:30:36.706 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:30:36.716 INFO:tasks.mds_thrash.fs.[b]:waiting for 103 secs before thrashing 2023-03-03T11:30:39.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:30:39.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.452+0000 7f5da5134700 1 Processor -- start 2023-03-03T11:30:39.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.452+0000 7f5da5134700 1 -- start start 2023-03-03T11:30:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.452+0000 7f5da5134700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5da01586c0 0x7f5da006d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.452+0000 7f5da5134700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 0x7f5da006d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.452+0000 7f5da5134700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da0154560 con 0x7f5da013a0f0 2023-03-03T11:30:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.452+0000 7f5da5134700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da0154840 con 0x7f5da01586c0 2023-03-03T11:30:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.452+0000 7f5da5134700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da0157be0 con 0x7f5da00c1da0 2023-03-03T11:30:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9e59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5da01586c0 0x7f5da006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9ed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5da013a0f0 0x7f5da00c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58844/0 (socket says 172.21.15.7:58844) 2023-03-03T11:30:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9e59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5da01586c0 0x7f5da006d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36634/0 (socket says 172.21.15.7:36634) 2023-03-03T11:30:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9ed9d700 1 -- 172.21.15.7:0/2352293707 learned_addr learned my addr 172.21.15.7:0/2352293707 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:30:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9f59e700 1 --2- 172.21.15.7:0/2352293707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 0x7f5da006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9e59c700 1 -- 172.21.15.7:0/2352293707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 msgr2=0x7f5da006d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:30:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9e59c700 1 --2- 172.21.15.7:0/2352293707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 0x7f5da006d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9e59c700 1 -- 172.21.15.7:0/2352293707 >> v1:172.21.15.7:6789/0 conn(0x7f5da013a0f0 legacy=0x7f5da00c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9e59c700 1 -- 172.21.15.7:0/2352293707 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da0157ec0 con 0x7f5da01586c0 2023-03-03T11:30:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9dd9b700 1 -- 172.21.15.7:0/2352293707 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207641959 0 0) 0x7f5da0154560 con 0x7f5da013a0f0 2023-03-03T11:30:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9e59c700 1 --2- 172.21.15.7:0/2352293707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 0x7f5da006d240 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009840 tx=0x7f5d90009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f8edb83b9c8636 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9dd9b700 1 -- 172.21.15.7:0/2352293707 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d90011830 con 0x7f5da01586c0 2023-03-03T11:30:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9dd9b700 1 -- 172.21.15.7:0/2352293707 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d90011990 con 0x7f5da01586c0 2023-03-03T11:30:39.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.453+0000 7f5d9dd9b700 1 -- 172.21.15.7:0/2352293707 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d90011d20 con 0x7f5da01586c0 2023-03-03T11:30:39.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.454+0000 7f5da5134700 1 -- 172.21.15.7:0/2352293707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 msgr2=0x7f5da006d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:39.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.454+0000 7f5da5134700 1 --2- 172.21.15.7:0/2352293707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 0x7f5da006d240 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009840 tx=0x7f5d90009b50 comp rx=0 tx=0).stop 2023-03-03T11:30:39.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.454+0000 7f5da5134700 1 -- 172.21.15.7:0/2352293707 shutdown_connections 2023-03-03T11:30:39.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.454+0000 7f5da5134700 1 --2- 172.21.15.7:0/2352293707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 0x7f5da006d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.454+0000 7f5da5134700 1 --2- 172.21.15.7:0/2352293707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 0x7f5da006d240 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.454+0000 7f5da5134700 1 -- 172.21.15.7:0/2352293707 >> 172.21.15.7:0/2352293707 conn(0x7f5da006c110 msgr2=0x7f5da006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:30:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.454+0000 7f5da5134700 1 -- 172.21.15.7:0/2352293707 shutdown_connections 2023-03-03T11:30:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.454+0000 7f5da5134700 1 -- 172.21.15.7:0/2352293707 wait complete. 2023-03-03T11:30:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.455+0000 7f5da5134700 1 Processor -- start 2023-03-03T11:30:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.455+0000 7f5da5134700 1 -- start start 2023-03-03T11:30:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5da5134700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 0x7f5da0156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5da5134700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5da013a0f0 0x7f5da01ed130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:39.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5da5134700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 0x7f5da01ed670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5da5134700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da00c6330 con 0x7f5da01586c0 2023-03-03T11:30:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5da5134700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da00c5890 con 0x7f5da013a0f0 2023-03-03T11:30:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5da5134700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da00acc40 con 0x7f5da00c1da0 2023-03-03T11:30:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5d9f59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 0x7f5da01ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5d9f59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 0x7f5da01ed670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36642/0 (socket says 172.21.15.7:36642) 2023-03-03T11:30:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5d9f59e700 1 -- 172.21.15.7:0/2235539622 learned_addr learned my addr 172.21.15.7:0/2235539622 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:30:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5d9ed9d700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 0x7f5da0156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:39.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.456+0000 7f5d9e59c700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5da013a0f0 0x7f5da01ed130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:39.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d9f59e700 1 -- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 msgr2=0x7f5da0156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:39.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d9f59e700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 0x7f5da0156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d9f59e700 1 -- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5da013a0f0 msgr2=0x7f5da01ed130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:39.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d9f59e700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5da013a0f0 0x7f5da01ed130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d9f59e700 1 -- 172.21.15.7:0/2235539622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d900094a0 con 0x7f5da01586c0 2023-03-03T11:30:39.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d9ed9d700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 0x7f5da0156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:30:39.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d9e59c700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5da013a0f0 0x7f5da01ed130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:30:39.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d9f59e700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 0x7f5da01ed670 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5d98007e40 tx=0x7f5d9800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:39.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d977fe700 1 -- 172.21.15.7:0/2235539622 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9800eea0 con 0x7f5da01586c0 2023-03-03T11:30:39.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d977fe700 1 -- 172.21.15.7:0/2235539622 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d98014420 con 0x7f5da01586c0 2023-03-03T11:30:39.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5da5134700 1 -- 172.21.15.7:0/2235539622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da006e1a0 con 0x7f5da01586c0 2023-03-03T11:30:39.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5d977fe700 1 -- 172.21.15.7:0/2235539622 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d980147b0 con 0x7f5da01586c0 2023-03-03T11:30:39.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.457+0000 7f5da5134700 1 -- 172.21.15.7:0/2235539622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da01581a0 con 0x7f5da01586c0 2023-03-03T11:30:39.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.458+0000 7f5da5134700 1 -- 172.21.15.7:0/2235539622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da006e330 con 0x7f5da01586c0 2023-03-03T11:30:39.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.458+0000 7f5d977fe700 1 -- 172.21.15.7:0/2235539622 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5d98012050 con 0x7f5da01586c0 2023-03-03T11:30:39.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.459+0000 7f5d977fe700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d800682f0 0x7f5d8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:30:39.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.459+0000 7f5d977fe700 1 -- 172.21.15.7:0/2235539622 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5d98086820 con 0x7f5da01586c0 2023-03-03T11:30:39.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.459+0000 7f5d9ed9d700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d800682f0 0x7f5d8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:30:39.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.460+0000 7f5d9ed9d700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d800682f0 0x7f5d8006a7a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5d88000f60 tx=0x7f5d88007cc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:30:39.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.463+0000 7f5d977fe700 1 -- 172.21.15.7:0/2235539622 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5d98052bc0 con 0x7f5da01586c0 2023-03-03T11:30:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.701+0000 7f5da5134700 1 -- 172.21.15.7:0/2235539622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5da0074bb0 con 0x7f5da01586c0 2023-03-03T11:30:39.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.702+0000 7f5d977fe700 1 -- 172.21.15.7:0/2235539622 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5d980561e0 con 0x7f5da01586c0 2023-03-03T11:30:39.704 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:30:39.704 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:30:39.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.703+0000 7f5da5134700 1 -- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d800682f0 msgr2=0x7f5d8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d800682f0 0x7f5d8006a7a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f5d88000f60 tx=0x7f5d88007cc0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 -- 172.21.15.7:0/2235539622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 msgr2=0x7f5da01ed670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:30:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 0x7f5da01ed670 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5d98007e40 tx=0x7f5d9800e3c0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 -- 172.21.15.7:0/2235539622 shutdown_connections 2023-03-03T11:30:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d800682f0 0x7f5d8006a7a0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5da01586c0 0x7f5da01ed670 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5da00c1da0 0x7f5da0156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 --2- 172.21.15.7:0/2235539622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5da013a0f0 0x7f5da01ed130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:30:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 -- 172.21.15.7:0/2235539622 >> 172.21.15.7:0/2235539622 conn(0x7f5da006c110 msgr2=0x7f5da006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:30:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 -- 172.21.15.7:0/2235539622 shutdown_connections 2023-03-03T11:30:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:30:39.704+0000 7f5da5134700 1 -- 172.21.15.7:0/2235539622 wait complete. 2023-03-03T11:30:39.707 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:30:39.717 INFO:tasks.mds_thrash.fs.[a]:waiting for 114 secs before thrashing 2023-03-03T11:30:44.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:30:44.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:14.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:14.791 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:44.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:31:44.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:15.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:15.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:19.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:32:19.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.853+0000 7f748a5a2700 1 Processor -- start 2023-03-03T11:32:19.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.853+0000 7f748a5a2700 1 -- start start 2023-03-03T11:32:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f748a5a2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f74841530d0 0x7f7484156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f748a5a2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74840c2f30 con 0x7f748414f9d0 2023-03-03T11:32:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f748a5a2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74840c31d0 con 0x7f748414fda0 2023-03-03T11:32:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f748a5a2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74841524a0 con 0x7f74841530d0 2023-03-03T11:32:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7483fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f748414f9d0 0x7f7484071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58922/0 (socket says 172.21.15.7:58922) 2023-03-03T11:32:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7483fff700 1 -- 172.21.15.7:0/573295645 learned_addr learned my addr 172.21.15.7:0/573295645 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:32:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526804 0 0) 0x7f74840c2f30 con 0x7f748414f9d0 2023-03-03T11:32:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7488b3f700 1 --2- 172.21.15.7:0/573295645 >> v2:172.21.15.143:3300/0 conn(0x7f74841530d0 0x7f7484156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f746c0035b0 con 0x7f748414f9d0 2023-03-03T11:32:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3188762756 0 0) 0x7f746c0035b0 con 0x7f748414f9d0 2023-03-03T11:32:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 >> v2:172.21.15.143:3300/0 conn(0x7f74841530d0 msgr2=0x7f7484156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:32:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7482ffd700 1 --2- 172.21.15.7:0/573295645 >> v2:172.21.15.143:3300/0 conn(0x7f74841530d0 0x7f7484156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:19.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 >> v1:172.21.15.143:6789/0 conn(0x7f748414fda0 legacy=0x7f74841529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:19.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74840c34b0 con 0x7f748414f9d0 2023-03-03T11:32:19.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.854+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7474001670 con 0x7f748414f9d0 2023-03-03T11:32:19.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 >> v1:172.21.15.7:6789/0 conn(0x7f748414f9d0 legacy=0x7f7484071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:19.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7482ffd700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f746c003a70 0x7f746c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:19.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7482ffd700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f746c004850 0x7f746c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:19.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7482ffd700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f746c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:19.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74841524a0 con 0x7f746c005320 2023-03-03T11:32:19.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746c00c1e0 con 0x7f746c003a70 2023-03-03T11:32:19.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746c00c3a0 con 0x7f746c004850 2023-03-03T11:32:19.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7488b3f700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f746c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:19.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7483fff700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f746c003a70 0x7f746c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:19.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f74837fe700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f746c004850 0x7f746c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:19.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7488b3f700 1 -- 172.21.15.7:0/573295645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f746c004850 msgr2=0x7f746c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:32:19.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7488b3f700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f746c004850 0x7f746c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:19.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7488b3f700 1 -- 172.21.15.7:0/573295645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f746c003a70 msgr2=0x7f746c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:32:19.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7488b3f700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f746c003a70 0x7f746c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:19.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7488b3f700 1 -- 172.21.15.7:0/573295645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746c0035b0 con 0x7f746c005320 2023-03-03T11:32:19.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7488b3f700 1 -- 172.21.15.7:0/573295645 reap_dead start 2023-03-03T11:32:19.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7488b3f700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f746c00ba10 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f7484071070 tx=0x7f746c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b20ad2b7038e79 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:19.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c018070 con 0x7f746c005320 2023-03-03T11:32:19.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747c00f040 con 0x7f746c005320 2023-03-03T11:32:19.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.855+0000 7f7482ffd700 1 -- 172.21.15.7:0/573295645 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c00ebd0 con 0x7f746c005320 2023-03-03T11:32:19.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.856+0000 7f748a5a2700 1 -- 172.21.15.7:0/573295645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 msgr2=0x7f746c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:19.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.856+0000 7f748a5a2700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f746c00ba10 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f7484071070 tx=0x7f746c005880 comp rx=0 tx=0).stop 2023-03-03T11:32:19.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.856+0000 7f748a5a2700 1 -- 172.21.15.7:0/573295645 shutdown_connections 2023-03-03T11:32:19.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.856+0000 7f748a5a2700 1 --2- 172.21.15.7:0/573295645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f746c00ba10 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:19.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.856+0000 7f748a5a2700 1 -- 172.21.15.7:0/573295645 >> 172.21.15.7:0/573295645 conn(0x7f74840c8900 msgr2=0x7f74840c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:32:19.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.856+0000 7f748a5a2700 1 -- 172.21.15.7:0/573295645 shutdown_connections 2023-03-03T11:32:19.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.856+0000 7f748a5a2700 1 -- 172.21.15.7:0/573295645 wait complete. 2023-03-03T11:32:19.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.858+0000 7f748a5a2700 1 Processor -- start 2023-03-03T11:32:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.858+0000 7f748a5a2700 1 -- start start 2023-03-03T11:32:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f748a5a2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f7484154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f748a5a2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74841ded80 0x7f7484148f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f748a5a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748414c610 0x7f7484149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f748a5a2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74841dcb50 con 0x7f746c005320 2023-03-03T11:32:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f748a5a2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74840c1dd0 con 0x7f74841ded80 2023-03-03T11:32:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f748a5a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74841dbb20 con 0x7f748414c610 2023-03-03T11:32:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7483fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f7484154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7483fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f7484154ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36722/0 (socket says 172.21.15.7:36722) 2023-03-03T11:32:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7483fff700 1 -- 172.21.15.7:0/4169432303 learned_addr learned my addr 172.21.15.7:0/4169432303 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:32:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f74837fe700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74841ded80 0x7f7484148f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7488b3f700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748414c610 0x7f7484149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7483fff700 1 -- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748414c610 msgr2=0x7f7484149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7483fff700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748414c610 0x7f7484149490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7483fff700 1 -- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74841ded80 msgr2=0x7f7484148f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7483fff700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74841ded80 0x7f7484148f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7483fff700 1 -- 172.21.15.7:0/4169432303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74840ac350 con 0x7f746c005320 2023-03-03T11:32:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f74837fe700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74841ded80 0x7f7484148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:32:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7483fff700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f7484154ee0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f7474007310 tx=0x7f7474007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7480ff9700 1 -- 172.21.15.7:0/4169432303 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474001630 con 0x7f746c005320 2023-03-03T11:32:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7480ff9700 1 -- 172.21.15.7:0/4169432303 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747400f6c0 con 0x7f746c005320 2023-03-03T11:32:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f748a5a2700 1 -- 172.21.15.7:0/4169432303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74840ab630 con 0x7f746c005320 2023-03-03T11:32:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.859+0000 7f7480ff9700 1 -- 172.21.15.7:0/4169432303 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747400fa70 con 0x7f746c005320 2023-03-03T11:32:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.860+0000 7f748a5a2700 1 -- 172.21.15.7:0/4169432303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74840c3e10 con 0x7f746c005320 2023-03-03T11:32:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.860+0000 7f748a5a2700 1 -- 172.21.15.7:0/4169432303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74841dc410 con 0x7f746c005320 2023-03-03T11:32:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.861+0000 7f7480ff9700 1 -- 172.21.15.7:0/4169432303 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f747400fbd0 con 0x7f746c005320 2023-03-03T11:32:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.861+0000 7f7480ff9700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 0x7f747006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.861+0000 7f7480ff9700 1 -- 172.21.15.7:0/4169432303 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f74740861c0 con 0x7f746c005320 2023-03-03T11:32:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.862+0000 7f74837fe700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 0x7f747006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.862+0000 7f74837fe700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 0x7f747006a7a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7484150b30 tx=0x7f7478009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:19.866+0000 7f7480ff9700 1 -- 172.21.15.7:0/4169432303 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7474052560 con 0x7f746c005320 2023-03-03T11:32:20.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.107+0000 7f748a5a2700 1 -- 172.21.15.7:0/4169432303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74840aa8a0 con 0x7f746c005320 2023-03-03T11:32:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.108+0000 7f7480ff9700 1 -- 172.21.15.7:0/4169432303 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7474055b80 con 0x7f746c005320 2023-03-03T11:32:20.111 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:32:20.111 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:32:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 -- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 msgr2=0x7f747006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 0x7f747006a7a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7484150b30 tx=0x7f7478009040 comp rx=0 tx=0).stop 2023-03-03T11:32:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 -- 172.21.15.7:0/4169432303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 msgr2=0x7f7484154ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f7484154ee0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f7474007310 tx=0x7f7474007620 comp rx=0 tx=0).stop 2023-03-03T11:32:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 -- 172.21.15.7:0/4169432303 shutdown_connections 2023-03-03T11:32:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f748414c610 0x7f7484149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74700682f0 0x7f747006a7a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f746c005320 0x7f7484154ee0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 --2- 172.21.15.7:0/4169432303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74841ded80 0x7f7484148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 -- 172.21.15.7:0/4169432303 >> 172.21.15.7:0/4169432303 conn(0x7f74840c8900 msgr2=0x7f74840c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:32:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 -- 172.21.15.7:0/4169432303 shutdown_connections 2023-03-03T11:32:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:20.110+0000 7f748a5a2700 1 -- 172.21.15.7:0/4169432303 wait complete. 2023-03-03T11:32:20.115 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:32:20.125 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T11:32:31.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:32:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.263+0000 7f423eedc700 1 Processor -- start 2023-03-03T11:32:31.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.263+0000 7f423eedc700 1 -- start start 2023-03-03T11:32:31.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.263+0000 7f423eedc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f423806d240 0x7f4238156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:31.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f423eedc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42380c24b0 con 0x7f42380740f0 2023-03-03T11:32:31.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f423eedc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4238152ef0 con 0x7f42380c3e30 2023-03-03T11:32:31.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f423eedc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4238155960 con 0x7f423806d240 2023-03-03T11:32:31.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f423d479700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f42380740f0 0x7f423815cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58954/0 (socket says 172.21.15.7:58954) 2023-03-03T11:32:31.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f423d479700 1 -- 172.21.15.7:0/902548184 learned_addr learned my addr 172.21.15.7:0/902548184 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:32:31.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255104135 0 0) 0x7f42380c24b0 con 0x7f42380740f0 2023-03-03T11:32:31.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f4237fff700 1 --2- 172.21.15.7:0/902548184 >> v2:172.21.15.143:3300/0 conn(0x7f423806d240 0x7f4238156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:31.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42280035b0 con 0x7f42380740f0 2023-03-03T11:32:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2162169344 0 0) 0x7f42280035b0 con 0x7f42380740f0 2023-03-03T11:32:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 >> v2:172.21.15.143:3300/0 conn(0x7f423806d240 msgr2=0x7f4238156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:32:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 --2- 172.21.15.7:0/902548184 >> v2:172.21.15.143:3300/0 conn(0x7f423806d240 0x7f4238156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 >> v1:172.21.15.143:6789/0 conn(0x7f42380c3e30 legacy=0x7f42380c1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4238155c40 con 0x7f42380740f0 2023-03-03T11:32:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f42300019f0 con 0x7f42380740f0 2023-03-03T11:32:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 >> v1:172.21.15.7:6789/0 conn(0x7f42380740f0 legacy=0x7f423815cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:31.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4228003a70 0x7f4228003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:31.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4228004850 0x7f4228004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:31.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4228005320 0x7f422800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:31.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4238155960 con 0x7f4228005320 2023-03-03T11:32:31.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422800c230 con 0x7f4228004850 2023-03-03T11:32:31.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.264+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422800c3f0 con 0x7f4228003a70 2023-03-03T11:32:31.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f423d479700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4228005320 0x7f422800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:31.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f4237fff700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4228004850 0x7f4228004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f423cc78700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4228003a70 0x7f4228003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f423d479700 1 -- 172.21.15.7:0/902548184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4228003a70 msgr2=0x7f4228003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:32:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f423d479700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4228003a70 0x7f4228003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f423d479700 1 -- 172.21.15.7:0/902548184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4228004850 msgr2=0x7f4228004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:32:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f423d479700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4228004850 0x7f4228004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f423d479700 1 -- 172.21.15.7:0/902548184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42280035b0 con 0x7f4228005320 2023-03-03T11:32:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f4237fff700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4228004850 0x7f4228004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:32:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f423d479700 1 -- 172.21.15.7:0/902548184 reap_dead start 2023-03-03T11:32:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f423d479700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4228005320 0x7f422800ba10 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f423000a830 tx=0x7f4228005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a551af73966788a7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423000a2f0 con 0x7f4228005320 2023-03-03T11:32:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4230012620 con 0x7f4228005320 2023-03-03T11:32:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.265+0000 7f42377fe700 1 -- 172.21.15.7:0/902548184 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42300129d0 con 0x7f4228005320 2023-03-03T11:32:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.266+0000 7f423eedc700 1 -- 172.21.15.7:0/902548184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4228005320 msgr2=0x7f422800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.266+0000 7f423eedc700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4228005320 0x7f422800ba10 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f423000a830 tx=0x7f4228005880 comp rx=0 tx=0).stop 2023-03-03T11:32:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.266+0000 7f423eedc700 1 -- 172.21.15.7:0/902548184 shutdown_connections 2023-03-03T11:32:31.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.266+0000 7f423eedc700 1 --2- 172.21.15.7:0/902548184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4228005320 0x7f422800ba10 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:31.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.266+0000 7f423eedc700 1 -- 172.21.15.7:0/902548184 >> 172.21.15.7:0/902548184 conn(0x7f423806c110 msgr2=0x7f423806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:32:31.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.266+0000 7f423eedc700 1 -- 172.21.15.7:0/902548184 shutdown_connections 2023-03-03T11:32:31.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.266+0000 7f423eedc700 1 -- 172.21.15.7:0/902548184 wait complete. 2023-03-03T11:32:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.267+0000 7f423eedc700 1 Processor -- start 2023-03-03T11:32:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.267+0000 7f423eedc700 1 -- start start 2023-03-03T11:32:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f423eedc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4228005320 0x7f4238151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f423eedc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f423814add0 0x7f42380c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:31.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f423eedc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42381de7b0 0x7f42381dec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:31.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f423eedc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4238151d30 con 0x7f423814add0 2023-03-03T11:32:31.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f423eedc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42381de3d0 con 0x7f42381de7b0 2023-03-03T11:32:31.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f423eedc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42381ebdf0 con 0x7f4228005320 2023-03-03T11:32:31.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f4237fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f423814add0 0x7f42380c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:31.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f4237fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f423814add0 0x7f42380c64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36754/0 (socket says 172.21.15.7:36754) 2023-03-03T11:32:31.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f4237fff700 1 -- 172.21.15.7:0/3842680672 learned_addr learned my addr 172.21.15.7:0/3842680672 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:32:31.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f423cc78700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4228005320 0x7f4238151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f423d479700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42381de7b0 0x7f42381dec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f4237fff700 1 -- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4228005320 msgr2=0x7f4238151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f4237fff700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4228005320 0x7f4238151330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f4237fff700 1 -- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42381de7b0 msgr2=0x7f42381dec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:31.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f4237fff700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42381de7b0 0x7f42381dec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:31.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f4237fff700 1 -- 172.21.15.7:0/3842680672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423806e150 con 0x7f423814add0 2023-03-03T11:32:31.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f423cc78700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4228005320 0x7f4238151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:32:31.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f423d479700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42381de7b0 0x7f42381dec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:32:31.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.268+0000 7f4237fff700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f423814add0 0x7f42380c64c0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f4224011ca0 tx=0x7f422400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:31.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.269+0000 7f42357fa700 1 -- 172.21.15.7:0/3842680672 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422400fb30 con 0x7f423814add0 2023-03-03T11:32:31.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.269+0000 7f42357fa700 1 -- 172.21.15.7:0/3842680672 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422400fc90 con 0x7f423814add0 2023-03-03T11:32:31.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.269+0000 7f423eedc700 1 -- 172.21.15.7:0/3842680672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4238155f20 con 0x7f423814add0 2023-03-03T11:32:31.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.269+0000 7f42357fa700 1 -- 172.21.15.7:0/3842680672 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4224004020 con 0x7f423814add0 2023-03-03T11:32:31.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.269+0000 7f423eedc700 1 -- 172.21.15.7:0/3842680672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42381531d0 con 0x7f423814add0 2023-03-03T11:32:31.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.270+0000 7f423eedc700 1 -- 172.21.15.7:0/3842680672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423806e410 con 0x7f423814add0 2023-03-03T11:32:31.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.270+0000 7f42357fa700 1 -- 172.21.15.7:0/3842680672 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4224024070 con 0x7f423814add0 2023-03-03T11:32:31.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.271+0000 7f42357fa700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f421c0682f0 0x7f421c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:31.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.271+0000 7f42357fa700 1 -- 172.21.15.7:0/3842680672 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f42240865c0 con 0x7f423814add0 2023-03-03T11:32:31.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.271+0000 7f423cc78700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f421c0682f0 0x7f421c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:31.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.272+0000 7f423cc78700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f421c0682f0 0x7f421c06a7a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f422c000c00 tx=0x7f422c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:31.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.275+0000 7f42357fa700 1 -- 172.21.15.7:0/3842680672 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4224052960 con 0x7f423814add0 2023-03-03T11:32:31.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.513+0000 7f423eedc700 1 -- 172.21.15.7:0/3842680672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f423806c4e0 con 0x7f423814add0 2023-03-03T11:32:31.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.513+0000 7f42357fa700 1 -- 172.21.15.7:0/3842680672 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4224055f80 con 0x7f423814add0 2023-03-03T11:32:31.515 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:32:31.516 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:32:31.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 -- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f421c0682f0 msgr2=0x7f421c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:31.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f421c0682f0 0x7f421c06a7a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f422c000c00 tx=0x7f422c009040 comp rx=0 tx=0).stop 2023-03-03T11:32:31.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 -- 172.21.15.7:0/3842680672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f423814add0 msgr2=0x7f42380c64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:31.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f423814add0 0x7f42380c64c0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f4224011ca0 tx=0x7f422400f520 comp rx=0 tx=0).stop 2023-03-03T11:32:31.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 -- 172.21.15.7:0/3842680672 shutdown_connections 2023-03-03T11:32:31.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42381de7b0 0x7f42381dec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:31.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4228005320 0x7f4238151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:31.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f421c0682f0 0x7f421c06a7a0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:31.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 --2- 172.21.15.7:0/3842680672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f423814add0 0x7f42380c64c0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:31.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 -- 172.21.15.7:0/3842680672 >> 172.21.15.7:0/3842680672 conn(0x7f423806c110 msgr2=0x7f423814da20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:32:31.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 -- 172.21.15.7:0/3842680672 shutdown_connections 2023-03-03T11:32:31.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:31.516+0000 7f423eedc700 1 -- 172.21.15.7:0/3842680672 wait complete. 2023-03-03T11:32:31.520 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:32:31.529 INFO:tasks.mds_thrash.fs.[b]:waiting for 83 secs before thrashing 2023-03-03T11:32:33.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:32:33.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.851+0000 7f25ab65a700 1 Processor -- start 2023-03-03T11:32:33.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.851+0000 7f25ab65a700 1 -- start start 2023-03-03T11:32:33.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25ab65a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f25a4151e80 0x7f25a40c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:33.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25ab65a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f25a4155410 0x7f25a40bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:33.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25ab65a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a40c08d0 con 0x7f25a40c6fe0 2023-03-03T11:32:33.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25ab65a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a4152250 con 0x7f25a4155410 2023-03-03T11:32:33.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25ab65a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a40c0fb0 con 0x7f25a4151e80 2023-03-03T11:32:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25a8bf5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f25a4155410 0x7f25a40bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25a8bf5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f25a4155410 0x7f25a40bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36760/0 (socket says 172.21.15.7:36760) 2023-03-03T11:32:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25a8bf5700 1 -- 172.21.15.7:0/3465230884 learned_addr learned my addr 172.21.15.7:0/3465230884 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:32:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25a8bf5700 1 -- 172.21.15.7:0/3465230884 >> v2:172.21.15.143:3300/0 conn(0x7f25a4151e80 msgr2=0x7f25a40c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:32:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25a8bf5700 1 --2- 172.21.15.7:0/3465230884 >> v2:172.21.15.143:3300/0 conn(0x7f25a4151e80 0x7f25a40c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25a8bf5700 1 -- 172.21.15.7:0/3465230884 >> v1:172.21.15.143:6789/0 conn(0x7f25a40c6fe0 legacy=0x7f25a40bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:32:33.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25a8bf5700 1 -- 172.21.15.7:0/3465230884 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a4154930 con 0x7f25a4155410 2023-03-03T11:32:33.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.852+0000 7f25a8bf5700 1 --2- 172.21.15.7:0/3465230884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a4155410 0x7f25a40bf8e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f25a40cb250 tx=0x7f2594009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb04636bee2861f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f259bfff700 1 -- 172.21.15.7:0/3465230884 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594004230 con 0x7f25a4155410 2023-03-03T11:32:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f259bfff700 1 -- 172.21.15.7:0/3465230884 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2594004390 con 0x7f25a4155410 2023-03-03T11:32:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f259bfff700 1 -- 172.21.15.7:0/3465230884 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940036a0 con 0x7f25a4155410 2023-03-03T11:32:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f25ab65a700 1 -- 172.21.15.7:0/3465230884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a4155410 msgr2=0x7f25a40bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f25ab65a700 1 --2- 172.21.15.7:0/3465230884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a4155410 0x7f25a40bf8e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f25a40cb250 tx=0x7f2594009bd0 comp rx=0 tx=0).stop 2023-03-03T11:32:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f25ab65a700 1 -- 172.21.15.7:0/3465230884 shutdown_connections 2023-03-03T11:32:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f25ab65a700 1 --2- 172.21.15.7:0/3465230884 >> v2:172.21.15.143:3300/0 conn(0x7f25a4151e80 0x7f25a40c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f25ab65a700 1 --2- 172.21.15.7:0/3465230884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a4155410 0x7f25a40bf8e0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f25ab65a700 1 -- 172.21.15.7:0/3465230884 >> 172.21.15.7:0/3465230884 conn(0x7f25a4071630 msgr2=0x7f25a406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:32:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f25ab65a700 1 -- 172.21.15.7:0/3465230884 shutdown_connections 2023-03-03T11:32:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.853+0000 7f25ab65a700 1 -- 172.21.15.7:0/3465230884 wait complete. 2023-03-03T11:32:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.854+0000 7f25ab65a700 1 Processor -- start 2023-03-03T11:32:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.855+0000 7f25ab65a700 1 -- start start 2023-03-03T11:32:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.855+0000 7f25ab65a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a40c6fe0 0x7f25a4144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.855+0000 7f25ab65a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a4151e80 0x7f25a414fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.855+0000 7f25ab65a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a4155410 0x7f25a41ed450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.855+0000 7f25ab65a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a40cab20 con 0x7f25a40c6fe0 2023-03-03T11:32:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.855+0000 7f25ab65a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a40c8d30 con 0x7f25a4151e80 2023-03-03T11:32:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.855+0000 7f25ab65a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a40c8a50 con 0x7f25a4155410 2023-03-03T11:32:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.855+0000 7f25a93f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a40c6fe0 0x7f25a4144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.855+0000 7f25a93f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a40c6fe0 0x7f25a4144420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36764/0 (socket says 172.21.15.7:36764) 2023-03-03T11:32:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.855+0000 7f25a93f6700 1 -- 172.21.15.7:0/586995913 learned_addr learned my addr 172.21.15.7:0/586995913 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:32:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25a8bf5700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a4151e80 0x7f25a414fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25a9bf7700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a4155410 0x7f25a41ed450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25a93f6700 1 -- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a4155410 msgr2=0x7f25a41ed450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25a93f6700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a4155410 0x7f25a41ed450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25a93f6700 1 -- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a4151e80 msgr2=0x7f25a414fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:33.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25a93f6700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a4151e80 0x7f25a414fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25a93f6700 1 -- 172.21.15.7:0/586995913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2594009550 con 0x7f25a40c6fe0 2023-03-03T11:32:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25a8bf5700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a4151e80 0x7f25a414fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:32:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25a93f6700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a40c6fe0 0x7f25a4144420 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f25a000b530 tx=0x7f25a000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f2599ffb700 1 -- 172.21.15.7:0/586995913 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0011830 con 0x7f25a40c6fe0 2023-03-03T11:32:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f2599ffb700 1 -- 172.21.15.7:0/586995913 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a00119b0 con 0x7f25a40c6fe0 2023-03-03T11:32:33.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25ab65a700 1 -- 172.21.15.7:0/586995913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a4154ef0 con 0x7f25a40c6fe0 2023-03-03T11:32:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f2599ffb700 1 -- 172.21.15.7:0/586995913 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0011d60 con 0x7f25a40c6fe0 2023-03-03T11:32:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.856+0000 7f25ab65a700 1 -- 172.21.15.7:0/586995913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a4154c10 con 0x7f25a40c6fe0 2023-03-03T11:32:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.857+0000 7f25ab65a700 1 -- 172.21.15.7:0/586995913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a40c5220 con 0x7f25a40c6fe0 2023-03-03T11:32:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.858+0000 7f2599ffb700 1 -- 172.21.15.7:0/586995913 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f25a0021070 con 0x7f25a40c6fe0 2023-03-03T11:32:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.858+0000 7f2599ffb700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25880682f0 0x7f258806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:32:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.858+0000 7f2599ffb700 1 -- 172.21.15.7:0/586995913 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f25a00860d0 con 0x7f25a40c6fe0 2023-03-03T11:32:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.858+0000 7f25a8bf5700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25880682f0 0x7f258806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:32:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.859+0000 7f25a8bf5700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25880682f0 0x7f258806a7a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f25a40cb250 tx=0x7f2594008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:32:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:33.862+0000 7f2599ffb700 1 -- 172.21.15.7:0/586995913 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f25a0052470 con 0x7f25a40c6fe0 2023-03-03T11:32:34.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.098+0000 7f25ab65a700 1 -- 172.21.15.7:0/586995913 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25a40cad90 con 0x7f25a40c6fe0 2023-03-03T11:32:34.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.099+0000 7f2599ffb700 1 -- 172.21.15.7:0/586995913 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f25a0015090 con 0x7f25a40c6fe0 2023-03-03T11:32:34.101 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:32:34.101 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:32:34.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 -- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25880682f0 msgr2=0x7f258806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:34.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25880682f0 0x7f258806a7a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f25a40cb250 tx=0x7f2594008030 comp rx=0 tx=0).stop 2023-03-03T11:32:34.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 -- 172.21.15.7:0/586995913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a40c6fe0 msgr2=0x7f25a4144420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:32:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a40c6fe0 0x7f25a4144420 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f25a000b530 tx=0x7f25a000b8f0 comp rx=0 tx=0).stop 2023-03-03T11:32:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 -- 172.21.15.7:0/586995913 shutdown_connections 2023-03-03T11:32:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25a4155410 0x7f25a41ed450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25880682f0 0x7f258806a7a0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25a40c6fe0 0x7f25a4144420 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 --2- 172.21.15.7:0/586995913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25a4151e80 0x7f25a414fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:32:34.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 -- 172.21.15.7:0/586995913 >> 172.21.15.7:0/586995913 conn(0x7f25a4071630 msgr2=0x7f25a40bff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:32:34.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 -- 172.21.15.7:0/586995913 shutdown_connections 2023-03-03T11:32:34.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:32:34.101+0000 7f25ab65a700 1 -- 172.21.15.7:0/586995913 wait complete. 2023-03-03T11:32:34.105 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:32:34.115 INFO:tasks.mds_thrash.fs.[a]:waiting for 48 secs before thrashing 2023-03-03T11:32:45.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:32:45.131 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:15.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:15.278 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:22.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:33:22.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.251+0000 7fb7421ea700 1 Processor -- start 2023-03-03T11:33:22.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.251+0000 7fb7421ea700 1 -- start start 2023-03-03T11:33:22.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb7421ea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb73c14f9d0 0x7fb73c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb7421ea700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb73c0abe20 0x7fb73c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb7421ea700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c0c2f30 con 0x7fb73c14fda0 2023-03-03T11:33:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb7421ea700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73c0c31d0 con 0x7fb73c14f9d0 2023-03-03T11:33:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb7421ea700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73c1524a0 con 0x7fb73c0abe20 2023-03-03T11:33:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb73b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb73c14f9d0 0x7fb73c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb73affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb73c14fda0 0x7fb73c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59008/0 (socket says 172.21.15.7:59008) 2023-03-03T11:33:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb73affd700 1 -- 172.21.15.7:0/1928504720 learned_addr learned my addr 172.21.15.7:0/1928504720 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:33:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb73b7fe700 1 --2- 172.21.15.7:0/1928504720 >> v2:172.21.15.7:3300/0 conn(0x7fb73c14f9d0 0x7fb73c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36796/0 (socket says 172.21.15.7:36796) 2023-03-03T11:33:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb73bfff700 1 --2- 172.21.15.7:0/1928504720 >> v2:172.21.15.143:3300/0 conn(0x7fb73c0abe20 0x7fb73c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.252+0000 7fb73b7fe700 1 -- 172.21.15.7:0/1928504720 >> v2:172.21.15.143:3300/0 conn(0x7fb73c0abe20 msgr2=0x7fb73c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:33:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb73b7fe700 1 --2- 172.21.15.7:0/1928504720 >> v2:172.21.15.143:3300/0 conn(0x7fb73c0abe20 0x7fb73c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb73b7fe700 1 -- 172.21.15.7:0/1928504720 >> v1:172.21.15.7:6789/0 conn(0x7fb73c14fda0 legacy=0x7fb73c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb73a7fc700 1 -- 172.21.15.7:0/1928504720 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943096970 0 0) 0x7fb73c0c2f30 con 0x7fb73c14fda0 2023-03-03T11:33:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb73b7fe700 1 -- 172.21.15.7:0/1928504720 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c0c34b0 con 0x7fb73c14f9d0 2023-03-03T11:33:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb73b7fe700 1 --2- 172.21.15.7:0/1928504720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14f9d0 0x7fb73c156430 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fb73c071230 tx=0x7fb72c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a4fa6ba3d59b92e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb73a7fc700 1 -- 172.21.15.7:0/1928504720 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c011830 con 0x7fb73c14f9d0 2023-03-03T11:33:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb73a7fc700 1 -- 172.21.15.7:0/1928504720 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c011990 con 0x7fb73c14f9d0 2023-03-03T11:33:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb73a7fc700 1 -- 172.21.15.7:0/1928504720 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c011d20 con 0x7fb73c14f9d0 2023-03-03T11:33:22.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb7421ea700 1 -- 172.21.15.7:0/1928504720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14f9d0 msgr2=0x7fb73c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:22.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb7421ea700 1 --2- 172.21.15.7:0/1928504720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14f9d0 0x7fb73c156430 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fb73c071230 tx=0x7fb72c009b20 comp rx=0 tx=0).stop 2023-03-03T11:33:22.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb7421ea700 1 -- 172.21.15.7:0/1928504720 shutdown_connections 2023-03-03T11:33:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb7421ea700 1 --2- 172.21.15.7:0/1928504720 >> v2:172.21.15.143:3300/0 conn(0x7fb73c0abe20 0x7fb73c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb7421ea700 1 --2- 172.21.15.7:0/1928504720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14f9d0 0x7fb73c156430 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.253+0000 7fb7421ea700 1 -- 172.21.15.7:0/1928504720 >> 172.21.15.7:0/1928504720 conn(0x7fb73c0c8900 msgr2=0x7fb73c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:33:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.254+0000 7fb7421ea700 1 -- 172.21.15.7:0/1928504720 shutdown_connections 2023-03-03T11:33:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.254+0000 7fb7421ea700 1 -- 172.21.15.7:0/1928504720 wait complete. 2023-03-03T11:33:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.255+0000 7fb7421ea700 1 Processor -- start 2023-03-03T11:33:22.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.255+0000 7fb7421ea700 1 -- start start 2023-03-03T11:33:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb7421ea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb73c0abe20 0x7fb73c06f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb7421ea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb73c14f9d0 0x7fb73c1dba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb7421ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14fda0 0x7fb73c1dbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb7421ea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c150390 con 0x7fb73c14fda0 2023-03-03T11:33:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb7421ea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c06e380 con 0x7fb73c14f9d0 2023-03-03T11:33:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb7421ea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c06eb90 con 0x7fb73c0abe20 2023-03-03T11:33:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14fda0 0x7fb73c1dbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:22.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14fda0 0x7fb73c1dbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36806/0 (socket says 172.21.15.7:36806) 2023-03-03T11:33:22.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73bfff700 1 -- 172.21.15.7:0/2240741074 learned_addr learned my addr 172.21.15.7:0/2240741074 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:33:22.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73b7fe700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb73c0abe20 0x7fb73c06f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:22.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73affd700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb73c14f9d0 0x7fb73c1dba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:22.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73bfff700 1 -- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb73c0abe20 msgr2=0x7fb73c06f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:22.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73bfff700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb73c0abe20 0x7fb73c06f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:22.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73bfff700 1 -- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb73c14f9d0 msgr2=0x7fb73c1dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:22.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73bfff700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb73c14f9d0 0x7fb73c1dba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:22.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73bfff700 1 -- 172.21.15.7:0/2240741074 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0094a0 con 0x7fb73c14fda0 2023-03-03T11:33:22.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73affd700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb73c14f9d0 0x7fb73c1dba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:33:22.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73b7fe700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb73c0abe20 0x7fb73c06f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:33:22.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb73bfff700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14fda0 0x7fb73c1dbf90 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb734009fd0 tx=0x7fb73400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:22.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb733fff700 1 -- 172.21.15.7:0/2240741074 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73400ede0 con 0x7fb73c14fda0 2023-03-03T11:33:22.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb733fff700 1 -- 172.21.15.7:0/2240741074 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb734014420 con 0x7fb73c14fda0 2023-03-03T11:33:22.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.256+0000 7fb7421ea700 1 -- 172.21.15.7:0/2240741074 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73c0ac350 con 0x7fb73c14fda0 2023-03-03T11:33:22.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.257+0000 7fb733fff700 1 -- 172.21.15.7:0/2240741074 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7340147d0 con 0x7fb73c14fda0 2023-03-03T11:33:22.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.257+0000 7fb7421ea700 1 -- 172.21.15.7:0/2240741074 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73c0ab630 con 0x7fb73c14fda0 2023-03-03T11:33:22.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.258+0000 7fb7421ea700 1 -- 172.21.15.7:0/2240741074 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73c071fa0 con 0x7fb73c14fda0 2023-03-03T11:33:22.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.258+0000 7fb733fff700 1 -- 172.21.15.7:0/2240741074 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb734012050 con 0x7fb73c14fda0 2023-03-03T11:33:22.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.258+0000 7fb733fff700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb71c0682f0 0x7fb71c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:22.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.259+0000 7fb733fff700 1 -- 172.21.15.7:0/2240741074 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb734086760 con 0x7fb73c14fda0 2023-03-03T11:33:22.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.259+0000 7fb73b7fe700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb71c0682f0 0x7fb71c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:22.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.259+0000 7fb73b7fe700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb71c0682f0 0x7fb71c06a7a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0090b0 tx=0x7fb72c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:22.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.263+0000 7fb733fff700 1 -- 172.21.15.7:0/2240741074 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb734052b00 con 0x7fb73c14fda0 2023-03-03T11:33:22.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.498+0000 7fb7421ea700 1 -- 172.21.15.7:0/2240741074 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb73c1db770 con 0x7fb73c14fda0 2023-03-03T11:33:22.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.499+0000 7fb733fff700 1 -- 172.21.15.7:0/2240741074 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb73c1db770 con 0x7fb73c14fda0 2023-03-03T11:33:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:33:22.502 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:33:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 -- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb71c0682f0 msgr2=0x7fb71c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb71c0682f0 0x7fb71c06a7a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0090b0 tx=0x7fb72c008030 comp rx=0 tx=0).stop 2023-03-03T11:33:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 -- 172.21.15.7:0/2240741074 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14fda0 msgr2=0x7fb73c1dbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14fda0 0x7fb73c1dbf90 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fb734009fd0 tx=0x7fb73400e3c0 comp rx=0 tx=0).stop 2023-03-03T11:33:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 -- 172.21.15.7:0/2240741074 shutdown_connections 2023-03-03T11:33:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb71c0682f0 0x7fb71c06a7a0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb73c14fda0 0x7fb73c1dbf90 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb73c0abe20 0x7fb73c06f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 --2- 172.21.15.7:0/2240741074 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb73c14f9d0 0x7fb73c1dba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 -- 172.21.15.7:0/2240741074 >> 172.21.15.7:0/2240741074 conn(0x7fb73c0c8900 msgr2=0x7fb73c0c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:33:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 -- 172.21.15.7:0/2240741074 shutdown_connections 2023-03-03T11:33:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:22.501+0000 7fb7421ea700 1 -- 172.21.15.7:0/2240741074 wait complete. 2023-03-03T11:33:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:33:22.515 INFO:tasks.mds_thrash.fs.[a]:waiting for 33 secs before thrashing 2023-03-03T11:33:45.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:45.367 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:33:54.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:33:54.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.663+0000 7f7e3bf1a700 1 Processor -- start 2023-03-03T11:33:54.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.664+0000 7f7e3bf1a700 1 -- start start 2023-03-03T11:33:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.664+0000 7f7e3bf1a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e3414f9d0 0x7f7e34156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.664+0000 7f7e3bf1a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7e340abe20 0x7f7e341529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.664+0000 7f7e3bf1a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e340c2f30 con 0x7f7e3414fda0 2023-03-03T11:33:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.664+0000 7f7e3bf1a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e340c31d0 con 0x7f7e3414f9d0 2023-03-03T11:33:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.664+0000 7f7e3bf1a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e341524a0 con 0x7f7e340abe20 2023-03-03T11:33:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.664+0000 7f7e39cb6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e3414f9d0 0x7f7e34156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:54.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.664+0000 7f7e39cb6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e3414f9d0 0x7f7e34156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36814/0 (socket says 172.21.15.7:36814) 2023-03-03T11:33:54.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.664+0000 7f7e39cb6700 1 -- 172.21.15.7:0/3752351676 learned_addr learned my addr 172.21.15.7:0/3752351676 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:33:54.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.664+0000 7f7e394b5700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7e3414fda0 0x7f7e34071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59026/0 (socket says 172.21.15.7:59026) 2023-03-03T11:33:54.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.665+0000 7f7e39cb6700 1 -- 172.21.15.7:0/3752351676 >> v2:172.21.15.143:3300/0 conn(0x7f7e340abe20 msgr2=0x7f7e341529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:33:54.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.665+0000 7f7e39cb6700 1 --2- 172.21.15.7:0/3752351676 >> v2:172.21.15.143:3300/0 conn(0x7f7e340abe20 0x7f7e341529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:54.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.665+0000 7f7e39cb6700 1 -- 172.21.15.7:0/3752351676 >> v1:172.21.15.7:6789/0 conn(0x7f7e3414fda0 legacy=0x7f7e34071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:54.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.665+0000 7f7e39cb6700 1 -- 172.21.15.7:0/3752351676 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e340c34b0 con 0x7f7e3414f9d0 2023-03-03T11:33:54.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.665+0000 7f7e38cb4700 1 -- 172.21.15.7:0/3752351676 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252868142 0 0) 0x7f7e340c2f30 con 0x7f7e3414fda0 2023-03-03T11:33:54.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.665+0000 7f7e39cb6700 1 --2- 172.21.15.7:0/3752351676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e3414f9d0 0x7f7e34156430 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f7e34071230 tx=0x7f7e30009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2cee904f14ba06 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:54.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.665+0000 7f7e38cb4700 1 -- 172.21.15.7:0/3752351676 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3000b870 con 0x7f7e3414f9d0 2023-03-03T11:33:54.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.665+0000 7f7e38cb4700 1 -- 172.21.15.7:0/3752351676 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3000b9d0 con 0x7f7e3414f9d0 2023-03-03T11:33:54.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.665+0000 7f7e38cb4700 1 -- 172.21.15.7:0/3752351676 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3000bd60 con 0x7f7e3414f9d0 2023-03-03T11:33:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.666+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3752351676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e3414f9d0 msgr2=0x7f7e34156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.666+0000 7f7e3bf1a700 1 --2- 172.21.15.7:0/3752351676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e3414f9d0 0x7f7e34156430 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f7e34071230 tx=0x7f7e30009bd0 comp rx=0 tx=0).stop 2023-03-03T11:33:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.666+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3752351676 shutdown_connections 2023-03-03T11:33:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.666+0000 7f7e3bf1a700 1 --2- 172.21.15.7:0/3752351676 >> v2:172.21.15.143:3300/0 conn(0x7f7e340abe20 0x7f7e341529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.666+0000 7f7e3bf1a700 1 --2- 172.21.15.7:0/3752351676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e3414f9d0 0x7f7e34156430 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.666+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3752351676 >> 172.21.15.7:0/3752351676 conn(0x7f7e340c8900 msgr2=0x7f7e340c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:33:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.666+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3752351676 shutdown_connections 2023-03-03T11:33:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.666+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3752351676 wait complete. 2023-03-03T11:33:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.667+0000 7f7e3bf1a700 1 Processor -- start 2023-03-03T11:33:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.667+0000 7f7e3bf1a700 1 -- start start 2023-03-03T11:33:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e3bf1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e340abe20 0x7f7e340c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e3bf1a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e3414f9d0 0x7f7e341ecc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e3bf1a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e3414fda0 0x7f7e341ed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e3bf1a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e341e9540 con 0x7f7e340abe20 2023-03-03T11:33:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e3bf1a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e340c1b80 con 0x7f7e3414f9d0 2023-03-03T11:33:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e3bf1a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e341e9750 con 0x7f7e3414fda0 2023-03-03T11:33:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e39cb6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e340abe20 0x7f7e340c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e39cb6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e340abe20 0x7f7e340c7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36820/0 (socket says 172.21.15.7:36820) 2023-03-03T11:33:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e39cb6700 1 -- 172.21.15.7:0/3959564019 learned_addr learned my addr 172.21.15.7:0/3959564019 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:33:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e394b5700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e3414f9d0 0x7f7e341ecc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e3a4b7700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e3414fda0 0x7f7e341ed1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e39cb6700 1 -- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e3414fda0 msgr2=0x7f7e341ed1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e39cb6700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e3414fda0 0x7f7e341ed1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e39cb6700 1 -- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e3414f9d0 msgr2=0x7f7e341ecc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e39cb6700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e3414f9d0 0x7f7e341ecc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:54.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e39cb6700 1 -- 172.21.15.7:0/3959564019 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e30009550 con 0x7f7e340abe20 2023-03-03T11:33:54.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e3a4b7700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e3414fda0 0x7f7e341ed1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:33:54.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e39cb6700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e340abe20 0x7f7e340c7850 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009c10 tx=0x7f7e30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:54.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e267fc700 1 -- 172.21.15.7:0/3959564019 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e30004310 con 0x7f7e340abe20 2023-03-03T11:33:54.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e267fc700 1 -- 172.21.15.7:0/3959564019 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3001f070 con 0x7f7e340abe20 2023-03-03T11:33:54.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3959564019 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e340ac350 con 0x7f7e340abe20 2023-03-03T11:33:54.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e267fc700 1 -- 172.21.15.7:0/3959564019 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e300036a0 con 0x7f7e340abe20 2023-03-03T11:33:54.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.668+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3959564019 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e340ab630 con 0x7f7e340abe20 2023-03-03T11:33:54.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.669+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3959564019 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e34071fa0 con 0x7f7e340abe20 2023-03-03T11:33:54.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.670+0000 7f7e267fc700 1 -- 172.21.15.7:0/3959564019 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7e3002a020 con 0x7f7e340abe20 2023-03-03T11:33:54.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.670+0000 7f7e267fc700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e180682f0 0x7f7e1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:54.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.670+0000 7f7e267fc700 1 -- 172.21.15.7:0/3959564019 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7e300877a0 con 0x7f7e340abe20 2023-03-03T11:33:54.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.671+0000 7f7e394b5700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e180682f0 0x7f7e1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:54.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.671+0000 7f7e394b5700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e180682f0 0x7f7e1806a7a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7e34151650 tx=0x7f7e28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:54.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.675+0000 7f7e267fc700 1 -- 172.21.15.7:0/3959564019 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7e30053b40 con 0x7f7e340abe20 2023-03-03T11:33:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.909+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3959564019 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e341ecab0 con 0x7f7e340abe20 2023-03-03T11:33:54.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.910+0000 7f7e267fc700 1 -- 172.21.15.7:0/3959564019 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7e30057160 con 0x7f7e340abe20 2023-03-03T11:33:54.912 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:33:54.912 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:33:54.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e180682f0 msgr2=0x7f7e1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:54.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e180682f0 0x7f7e1806a7a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7e34151650 tx=0x7f7e28009040 comp rx=0 tx=0).stop 2023-03-03T11:33:54.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3959564019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e340abe20 msgr2=0x7f7e340c7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:54.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e340abe20 0x7f7e340c7850 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009c10 tx=0x7f7e30004750 comp rx=0 tx=0).stop 2023-03-03T11:33:54.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3959564019 shutdown_connections 2023-03-03T11:33:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e3414fda0 0x7f7e341ed1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e180682f0 0x7f7e1806a7a0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e340abe20 0x7f7e340c7850 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 --2- 172.21.15.7:0/3959564019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e3414f9d0 0x7f7e341ecc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3959564019 >> 172.21.15.7:0/3959564019 conn(0x7f7e340c8900 msgr2=0x7f7e34151900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:33:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3959564019 shutdown_connections 2023-03-03T11:33:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:54.912+0000 7f7e3bf1a700 1 -- 172.21.15.7:0/3959564019 wait complete. 2023-03-03T11:33:54.916 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:33:54.926 INFO:tasks.mds_thrash.fs.[b]:waiting for 15 secs before thrashing 2023-03-03T11:33:55.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:33:55.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.650+0000 7f81ce187700 1 Processor -- start 2023-03-03T11:33:55.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.650+0000 7f81ce187700 1 -- start start 2023-03-03T11:33:55.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81ce187700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81c80c17f0 0x7f81c806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:55.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81ce187700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f81c80c5190 0x7f81c806dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:55.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81ce187700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c814ebe0 con 0x7f81c80c6010 2023-03-03T11:33:55.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81ce187700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c8152100 con 0x7f81c80c17f0 2023-03-03T11:33:55.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81ce187700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c80c2e70 con 0x7f81c80c5190 2023-03-03T11:33:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81c6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81c80c17f0 0x7f81c806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81c6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81c80c17f0 0x7f81c806d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36830/0 (socket says 172.21.15.7:36830) 2023-03-03T11:33:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81c77fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f81c80c6010 0x7f81c80c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59040/0 (socket says 172.21.15.7:59040) 2023-03-03T11:33:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81c6ffd700 1 -- 172.21.15.7:0/784716258 learned_addr learned my addr 172.21.15.7:0/784716258 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:33:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81c7fff700 1 --2- 172.21.15.7:0/784716258 >> v2:172.21.15.143:3300/0 conn(0x7f81c80c5190 0x7f81c806dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81c6ffd700 1 -- 172.21.15.7:0/784716258 >> v2:172.21.15.143:3300/0 conn(0x7f81c80c5190 msgr2=0x7f81c806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:33:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81c6ffd700 1 --2- 172.21.15.7:0/784716258 >> v2:172.21.15.143:3300/0 conn(0x7f81c80c5190 0x7f81c806dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81c6ffd700 1 -- 172.21.15.7:0/784716258 >> v1:172.21.15.7:6789/0 conn(0x7f81c80c6010 legacy=0x7f81c80c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81c6ffd700 1 -- 172.21.15.7:0/784716258 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c80aa680 con 0x7f81c80c17f0 2023-03-03T11:33:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.651+0000 7f81c6ffd700 1 --2- 172.21.15.7:0/784716258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c17f0 0x7f81c806d6a0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f81c806f490 tx=0x7f81b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f20fff122845af5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81c67fc700 1 -- 172.21.15.7:0/784716258 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b000b870 con 0x7f81c80c17f0 2023-03-03T11:33:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81c67fc700 1 -- 172.21.15.7:0/784716258 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b000b9d0 con 0x7f81c80c17f0 2023-03-03T11:33:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81c67fc700 1 -- 172.21.15.7:0/784716258 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b000bd60 con 0x7f81c80c17f0 2023-03-03T11:33:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81ce187700 1 -- 172.21.15.7:0/784716258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c17f0 msgr2=0x7f81c806d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81ce187700 1 --2- 172.21.15.7:0/784716258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c17f0 0x7f81c806d6a0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f81c806f490 tx=0x7f81b0009bd0 comp rx=0 tx=0).stop 2023-03-03T11:33:55.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81ce187700 1 -- 172.21.15.7:0/784716258 shutdown_connections 2023-03-03T11:33:55.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81ce187700 1 --2- 172.21.15.7:0/784716258 >> v2:172.21.15.143:3300/0 conn(0x7f81c80c5190 0x7f81c806dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:55.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81ce187700 1 --2- 172.21.15.7:0/784716258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c17f0 0x7f81c806d6a0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:55.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81ce187700 1 -- 172.21.15.7:0/784716258 >> 172.21.15.7:0/784716258 conn(0x7f81c80a9430 msgr2=0x7f81c80c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:33:55.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81ce187700 1 -- 172.21.15.7:0/784716258 shutdown_connections 2023-03-03T11:33:55.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.652+0000 7f81ce187700 1 -- 172.21.15.7:0/784716258 wait complete. 2023-03-03T11:33:55.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.653+0000 7f81ce187700 1 Processor -- start 2023-03-03T11:33:55.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.654+0000 7f81ce187700 1 -- start start 2023-03-03T11:33:55.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.654+0000 7f81ce187700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81c80c17f0 0x7f81c814e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:55.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.654+0000 7f81ce187700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c5190 0x7f81c8070c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:55.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.654+0000 7f81ce187700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81c80c6010 0x7f81c8071160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:55.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.654+0000 7f81ce187700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8152cc0 con 0x7f81c80c5190 2023-03-03T11:33:55.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.654+0000 7f81ce187700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c814b8e0 con 0x7f81c80c6010 2023-03-03T11:33:55.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.654+0000 7f81ce187700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c81529e0 con 0x7f81c80c17f0 2023-03-03T11:33:55.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.654+0000 7f81c6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c5190 0x7f81c8070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:55.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.654+0000 7f81c6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c5190 0x7f81c8070c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36836/0 (socket says 172.21.15.7:36836) 2023-03-03T11:33:55.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.654+0000 7f81c6ffd700 1 -- 172.21.15.7:0/3703188497 learned_addr learned my addr 172.21.15.7:0/3703188497 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:33:55.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81c77fe700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81c80c17f0 0x7f81c814e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:55.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81c7fff700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81c80c6010 0x7f81c8071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:55.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81c6ffd700 1 -- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81c80c17f0 msgr2=0x7f81c814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:55.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81c6ffd700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81c80c17f0 0x7f81c814e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:55.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81c6ffd700 1 -- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81c80c6010 msgr2=0x7f81c8071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:55.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81c6ffd700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81c80c6010 0x7f81c8071160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:55.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81c6ffd700 1 -- 172.21.15.7:0/3703188497 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b0009550 con 0x7f81c80c5190 2023-03-03T11:33:55.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81c6ffd700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c5190 0x7f81c8070c20 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f81b0000c00 tx=0x7f81b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:55.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81bffff700 1 -- 172.21.15.7:0/3703188497 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b0004520 con 0x7f81c80c5190 2023-03-03T11:33:55.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81bffff700 1 -- 172.21.15.7:0/3703188497 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b001e070 con 0x7f81c80c5190 2023-03-03T11:33:55.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81ce187700 1 -- 172.21.15.7:0/3703188497 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c80c2b00 con 0x7f81c80c5190 2023-03-03T11:33:55.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81bffff700 1 -- 172.21.15.7:0/3703188497 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b00038a0 con 0x7f81c80c5190 2023-03-03T11:33:55.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.655+0000 7f81ce187700 1 -- 172.21.15.7:0/3703188497 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c80aa960 con 0x7f81c80c5190 2023-03-03T11:33:55.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.656+0000 7f81ce187700 1 -- 172.21.15.7:0/3703188497 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c80c32e0 con 0x7f81c80c5190 2023-03-03T11:33:55.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.657+0000 7f81bffff700 1 -- 172.21.15.7:0/3703188497 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f81b000f050 con 0x7f81c80c5190 2023-03-03T11:33:55.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.657+0000 7f81bffff700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81a80682f0 0x7f81a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:33:55.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.657+0000 7f81bffff700 1 -- 172.21.15.7:0/3703188497 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f81b0020070 con 0x7f81c80c5190 2023-03-03T11:33:55.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.657+0000 7f81c77fe700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81a80682f0 0x7f81a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:33:55.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.658+0000 7f81c77fe700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81a80682f0 0x7f81a806a7a0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f81b8000f60 tx=0x7f81b8007d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:33:55.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.662+0000 7f81bffff700 1 -- 172.21.15.7:0/3703188497 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81b0053ac0 con 0x7f81c80c5190 2023-03-03T11:33:55.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.900+0000 7f81ce187700 1 -- 172.21.15.7:0/3703188497 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81c8070a70 con 0x7f81c80c5190 2023-03-03T11:33:55.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.901+0000 7f81bffff700 1 -- 172.21.15.7:0/3703188497 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f81b00570e0 con 0x7f81c80c5190 2023-03-03T11:33:55.902 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:33:55.903 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:33:55.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 -- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81a80682f0 msgr2=0x7f81a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:55.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81a80682f0 0x7f81a806a7a0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f81b8000f60 tx=0x7f81b8007d10 comp rx=0 tx=0).stop 2023-03-03T11:33:55.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 -- 172.21.15.7:0/3703188497 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c5190 msgr2=0x7f81c8070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:33:55.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c5190 0x7f81c8070c20 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f81b0000c00 tx=0x7f81b0004750 comp rx=0 tx=0).stop 2023-03-03T11:33:55.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 -- 172.21.15.7:0/3703188497 shutdown_connections 2023-03-03T11:33:55.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81c80c6010 0x7f81c8071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:55.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81c80c17f0 0x7f81c814e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:55.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81a80682f0 0x7f81a806a7a0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:55.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 --2- 172.21.15.7:0/3703188497 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81c80c5190 0x7f81c8070c20 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:33:55.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 -- 172.21.15.7:0/3703188497 >> 172.21.15.7:0/3703188497 conn(0x7f81c80a9430 msgr2=0x7f81c80a6740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:33:55.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 -- 172.21.15.7:0/3703188497 shutdown_connections 2023-03-03T11:33:55.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:33:55.903+0000 7f81ce187700 1 -- 172.21.15.7:0/3703188497 wait complete. 2023-03-03T11:33:55.907 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:33:55.916 INFO:tasks.mds_thrash.fs.[a]:waiting for 116 secs before thrashing 2023-03-03T11:34:09.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:34:10.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.063+0000 7eff56dcc700 1 Processor -- start 2023-03-03T11:34:10.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.063+0000 7eff56dcc700 1 -- start start 2023-03-03T11:34:10.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.063+0000 7eff56dcc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7eff50155390 0x7eff500c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:10.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.063+0000 7eff56dcc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff5006e020 0x7eff50071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:10.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.063+0000 7eff56dcc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff50154910 con 0x7eff50072520 2023-03-03T11:34:10.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff56dcc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff50070b90 con 0x7eff5006e020 2023-03-03T11:34:10.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff56dcc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff50151320 con 0x7eff50155390 2023-03-03T11:34:10.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff5006e020 0x7eff50071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:10.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff5006e020 0x7eff50071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36854/0 (socket says 172.21.15.7:36854) 2023-03-03T11:34:10.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4ffff700 1 -- 172.21.15.7:0/762144831 learned_addr learned my addr 172.21.15.7:0/762144831 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:34:10.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4ffff700 1 -- 172.21.15.7:0/762144831 >> v2:172.21.15.143:3300/0 conn(0x7eff50155390 msgr2=0x7eff500c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:34:10.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff54b68700 1 --2- 172.21.15.7:0/762144831 >> v2:172.21.15.143:3300/0 conn(0x7eff50155390 0x7eff500c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:10.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4ffff700 1 --2- 172.21.15.7:0/762144831 >> v2:172.21.15.143:3300/0 conn(0x7eff50155390 0x7eff500c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4ffff700 1 -- 172.21.15.7:0/762144831 >> v1:172.21.15.143:6789/0 conn(0x7eff50072520 legacy=0x7eff50151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:34:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4ffff700 1 -- 172.21.15.7:0/762144831 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff500702a0 con 0x7eff5006e020 2023-03-03T11:34:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff54b68700 1 --2- 172.21.15.7:0/762144831 >> v2:172.21.15.143:3300/0 conn(0x7eff50155390 0x7eff500c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:34:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4ffff700 1 --2- 172.21.15.7:0/762144831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 0x7eff50071fe0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7eff5006d700 tx=0x7eff3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435ab3d962415369 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:34:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4f7fe700 1 -- 172.21.15.7:0/762144831 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3c004230 con 0x7eff5006e020 2023-03-03T11:34:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4f7fe700 1 -- 172.21.15.7:0/762144831 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3c004390 con 0x7eff5006e020 2023-03-03T11:34:10.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.064+0000 7eff4f7fe700 1 -- 172.21.15.7:0/762144831 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3c0036a0 con 0x7eff5006e020 2023-03-03T11:34:10.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.065+0000 7eff56dcc700 1 -- 172.21.15.7:0/762144831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 msgr2=0x7eff50071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:10.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.065+0000 7eff56dcc700 1 --2- 172.21.15.7:0/762144831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 0x7eff50071fe0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7eff5006d700 tx=0x7eff3c009bd0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.065+0000 7eff56dcc700 1 -- 172.21.15.7:0/762144831 shutdown_connections 2023-03-03T11:34:10.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.065+0000 7eff56dcc700 1 --2- 172.21.15.7:0/762144831 >> v2:172.21.15.143:3300/0 conn(0x7eff50155390 0x7eff500c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.065+0000 7eff56dcc700 1 --2- 172.21.15.7:0/762144831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 0x7eff50071fe0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.065+0000 7eff56dcc700 1 -- 172.21.15.7:0/762144831 >> 172.21.15.7:0/762144831 conn(0x7eff5006c110 msgr2=0x7eff5006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:34:10.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.065+0000 7eff56dcc700 1 -- 172.21.15.7:0/762144831 shutdown_connections 2023-03-03T11:34:10.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.065+0000 7eff56dcc700 1 -- 172.21.15.7:0/762144831 wait complete. 2023-03-03T11:34:10.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.066+0000 7eff56dcc700 1 Processor -- start 2023-03-03T11:34:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.066+0000 7eff56dcc700 1 -- start start 2023-03-03T11:34:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff56dcc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 0x7eff50150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff56dcc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff50072520 0x7eff500c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff56dcc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff50155390 0x7eff5013e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff56dcc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff501e8790 con 0x7eff5006e020 2023-03-03T11:34:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff56dcc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff501e8cc0 con 0x7eff50072520 2023-03-03T11:34:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff56dcc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff50140ba0 con 0x7eff50155390 2023-03-03T11:34:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff54b68700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 0x7eff50150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff54b68700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 0x7eff50150a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36860/0 (socket says 172.21.15.7:36860) 2023-03-03T11:34:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff54b68700 1 -- 172.21.15.7:0/2262164036 learned_addr learned my addr 172.21.15.7:0/2262164036 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:34:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff55369700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff50155390 0x7eff5013e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff54b68700 1 -- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff50155390 msgr2=0x7eff5013e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff4ffff700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff50072520 0x7eff500c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff54b68700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff50155390 0x7eff5013e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff54b68700 1 -- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff50072520 msgr2=0x7eff500c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:10.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.067+0000 7eff54b68700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff50072520 0x7eff500c5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.068+0000 7eff54b68700 1 -- 172.21.15.7:0/2262164036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c009550 con 0x7eff5006e020 2023-03-03T11:34:10.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.068+0000 7eff54b68700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 0x7eff50150a80 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7eff4400c820 tx=0x7eff4400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:34:10.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.068+0000 7eff4d7fa700 1 -- 172.21.15.7:0/2262164036 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff44011800 con 0x7eff5006e020 2023-03-03T11:34:10.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.068+0000 7eff4d7fa700 1 -- 172.21.15.7:0/2262164036 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff44011960 con 0x7eff5006e020 2023-03-03T11:34:10.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.068+0000 7eff56dcc700 1 -- 172.21.15.7:0/2262164036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff50070820 con 0x7eff5006e020 2023-03-03T11:34:10.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.068+0000 7eff4d7fa700 1 -- 172.21.15.7:0/2262164036 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff44011d10 con 0x7eff5006e020 2023-03-03T11:34:10.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.068+0000 7eff56dcc700 1 -- 172.21.15.7:0/2262164036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff50070540 con 0x7eff5006e020 2023-03-03T11:34:10.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.069+0000 7eff56dcc700 1 -- 172.21.15.7:0/2262164036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff50155760 con 0x7eff5006e020 2023-03-03T11:34:10.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.069+0000 7eff4d7fa700 1 -- 172.21.15.7:0/2262164036 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7eff44021070 con 0x7eff5006e020 2023-03-03T11:34:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.070+0000 7eff4d7fa700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff340682f0 0x7eff3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.070+0000 7eff4d7fa700 1 -- 172.21.15.7:0/2262164036 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7eff44086050 con 0x7eff5006e020 2023-03-03T11:34:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.070+0000 7eff4ffff700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff340682f0 0x7eff3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.070+0000 7eff4ffff700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff340682f0 0x7eff3406a7a0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7eff501429c0 tx=0x7eff3c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:34:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.074+0000 7eff4d7fa700 1 -- 172.21.15.7:0/2262164036 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7eff44052370 con 0x7eff5006e020 2023-03-03T11:34:10.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.310+0000 7eff56dcc700 1 -- 172.21.15.7:0/2262164036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff500718d0 con 0x7eff5006e020 2023-03-03T11:34:10.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.311+0000 7eff4d7fa700 1 -- 172.21.15.7:0/2262164036 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7eff44055990 con 0x7eff5006e020 2023-03-03T11:34:10.313 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:34:10.313 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:34:10.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 -- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff340682f0 msgr2=0x7eff3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:10.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff340682f0 0x7eff3406a7a0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7eff501429c0 tx=0x7eff3c008030 comp rx=0 tx=0).stop 2023-03-03T11:34:10.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 -- 172.21.15.7:0/2262164036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 msgr2=0x7eff50150a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:10.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 0x7eff50150a80 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7eff4400c820 tx=0x7eff4400cbe0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 -- 172.21.15.7:0/2262164036 shutdown_connections 2023-03-03T11:34:10.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff50155390 0x7eff5013e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff340682f0 0x7eff3406a7a0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff5006e020 0x7eff50150a80 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 --2- 172.21.15.7:0/2262164036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff50072520 0x7eff500c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:10.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 -- 172.21.15.7:0/2262164036 >> 172.21.15.7:0/2262164036 conn(0x7eff5006c110 msgr2=0x7eff5015c100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:34:10.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 -- 172.21.15.7:0/2262164036 shutdown_connections 2023-03-03T11:34:10.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:10.313+0000 7eff56dcc700 1 -- 172.21.15.7:0/2262164036 wait complete. 2023-03-03T11:34:10.317 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:34:10.327 INFO:tasks.mds_thrash.fs.[b]:waiting for 32 secs before thrashing 2023-03-03T11:34:15.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:15.496 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:42.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.464+0000 7fcbaf91d700 1 Processor -- start 2023-03-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.464+0000 7fcbaf91d700 1 -- start start 2023-03-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.465+0000 7fcbaf91d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 0x7fcba8071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.465+0000 7fcbaf91d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcba8072520 0x7fcba8151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.465+0000 7fcbaf91d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba8154910 con 0x7fcba8155390 2023-03-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.465+0000 7fcbaf91d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba8070b90 con 0x7fcba8072520 2023-03-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.465+0000 7fcbaf91d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba8151320 con 0x7fcba806e020 2023-03-03T11:34:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcbadeba700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcba8072520 0x7fcba8151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcbaceb8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 0x7fcba8071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcbad6b9700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fcba8155390 0x7fcba80c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:34380/0 (socket says 172.21.15.7:34380) 2023-03-03T11:34:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcbad6b9700 1 -- 172.21.15.7:0/515458437 learned_addr learned my addr 172.21.15.7:0/515458437 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:34:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcb9ffff700 1 -- 172.21.15.7:0/515458437 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366294594 0 0) 0x7fcba8154910 con 0x7fcba8155390 2023-03-03T11:34:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcb9ffff700 1 -- 172.21.15.7:0/515458437 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb940035b0 con 0x7fcba8155390 2023-03-03T11:34:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcbaceb8700 1 -- 172.21.15.7:0/515458437 >> v2:172.21.15.143:3300/0 conn(0x7fcba8072520 msgr2=0x7fcba8151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:34:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcbaceb8700 1 --2- 172.21.15.7:0/515458437 >> v2:172.21.15.143:3300/0 conn(0x7fcba8072520 0x7fcba8151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcbaceb8700 1 -- 172.21.15.7:0/515458437 >> v1:172.21.15.143:6789/0 conn(0x7fcba8155390 legacy=0x7fcba80c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcbaceb8700 1 -- 172.21.15.7:0/515458437 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba80c4540 con 0x7fcba806e020 2023-03-03T11:34:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcbadeba700 1 --2- 172.21.15.7:0/515458437 >> v2:172.21.15.143:3300/0 conn(0x7fcba8072520 0x7fcba8151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:34:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcbaceb8700 1 --2- 172.21.15.7:0/515458437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 0x7fcba8071fe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009840 tx=0x7fcba4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1de091350d38ba44 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:34:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcb9ffff700 1 -- 172.21.15.7:0/515458437 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4011830 con 0x7fcba806e020 2023-03-03T11:34:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcb9ffff700 1 -- 172.21.15.7:0/515458437 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba4011990 con 0x7fcba806e020 2023-03-03T11:34:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.466+0000 7fcb9ffff700 1 -- 172.21.15.7:0/515458437 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4011d20 con 0x7fcba806e020 2023-03-03T11:34:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.467+0000 7fcbaf91d700 1 -- 172.21.15.7:0/515458437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 msgr2=0x7fcba8071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.467+0000 7fcbaf91d700 1 --2- 172.21.15.7:0/515458437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 0x7fcba8071fe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009840 tx=0x7fcba4009b50 comp rx=0 tx=0).stop 2023-03-03T11:34:42.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.467+0000 7fcbaf91d700 1 -- 172.21.15.7:0/515458437 shutdown_connections 2023-03-03T11:34:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.467+0000 7fcbaf91d700 1 --2- 172.21.15.7:0/515458437 >> v2:172.21.15.143:3300/0 conn(0x7fcba8072520 0x7fcba8151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.467+0000 7fcbaf91d700 1 --2- 172.21.15.7:0/515458437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 0x7fcba8071fe0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.467+0000 7fcbaf91d700 1 -- 172.21.15.7:0/515458437 >> 172.21.15.7:0/515458437 conn(0x7fcba806c110 msgr2=0x7fcba806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:34:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.467+0000 7fcbaf91d700 1 -- 172.21.15.7:0/515458437 shutdown_connections 2023-03-03T11:34:42.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.467+0000 7fcbaf91d700 1 -- 172.21.15.7:0/515458437 wait complete. 2023-03-03T11:34:42.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.468+0000 7fcbaf91d700 1 Processor -- start 2023-03-03T11:34:42.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.468+0000 7fcbaf91d700 1 -- start start 2023-03-03T11:34:42.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbaf91d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 0x7fcba8152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbaf91d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcba8072520 0x7fcba81ed590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbaf91d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcba8155390 0x7fcba80c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbaf91d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba81e9960 con 0x7fcba8072520 2023-03-03T11:34:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbaf91d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba81e9610 con 0x7fcba8155390 2023-03-03T11:34:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbaf91d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba80c5990 con 0x7fcba806e020 2023-03-03T11:34:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbaceb8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcba8072520 0x7fcba81ed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:42.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbaceb8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcba8072520 0x7fcba81ed590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36890/0 (socket says 172.21.15.7:36890) 2023-03-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbaceb8700 1 -- 172.21.15.7:0/3890123715 learned_addr learned my addr 172.21.15.7:0/3890123715 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbad6b9700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 0x7fcba8152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.469+0000 7fcbadeba700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcba8155390 0x7fcba80c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcbaceb8700 1 -- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 msgr2=0x7fcba8152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcbaceb8700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 0x7fcba8152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcbaceb8700 1 -- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcba8155390 msgr2=0x7fcba80c0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcbaceb8700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcba8155390 0x7fcba80c0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcbaceb8700 1 -- 172.21.15.7:0/3890123715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba0009a70 con 0x7fcba8072520 2023-03-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcbaceb8700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcba8072520 0x7fcba81ed590 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009fd0 tx=0x7fcba4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcb9dffb700 1 -- 172.21.15.7:0/3890123715 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4004550 con 0x7fcba8072520 2023-03-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcb9dffb700 1 -- 172.21.15.7:0/3890123715 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba401e070 con 0x7fcba8072520 2023-03-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcbaf91d700 1 -- 172.21.15.7:0/3890123715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba40094a0 con 0x7fcba8072520 2023-03-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcb9dffb700 1 -- 172.21.15.7:0/3890123715 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba40038e0 con 0x7fcba8072520 2023-03-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.470+0000 7fcbaf91d700 1 -- 172.21.15.7:0/3890123715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba8070540 con 0x7fcba8072520 2023-03-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.471+0000 7fcbaf91d700 1 -- 172.21.15.7:0/3890123715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba80702a0 con 0x7fcba8072520 2023-03-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.472+0000 7fcb9dffb700 1 -- 172.21.15.7:0/3890123715 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcba400f050 con 0x7fcba8072520 2023-03-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.472+0000 7fcb9dffb700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb8c0682f0 0x7fcb8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.472+0000 7fcb9dffb700 1 -- 172.21.15.7:0/3890123715 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcba4086d40 con 0x7fcba8072520 2023-03-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.472+0000 7fcbad6b9700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb8c0682f0 0x7fcb8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.473+0000 7fcbad6b9700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb8c0682f0 0x7fcb8c06a7a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fcb98007fd0 tx=0x7fcb98007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.476+0000 7fcb9dffb700 1 -- 172.21.15.7:0/3890123715 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcba40530e0 con 0x7fcba8072520 2023-03-03T11:34:42.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.716+0000 7fcbaf91d700 1 -- 172.21.15.7:0/3890123715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcba8070820 con 0x7fcba8072520 2023-03-03T11:34:42.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.717+0000 7fcb9dffb700 1 -- 172.21.15.7:0/3890123715 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcba4056700 con 0x7fcba8072520 2023-03-03T11:34:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:34:42.720 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:34:42.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 -- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb8c0682f0 msgr2=0x7fcb8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:42.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb8c0682f0 0x7fcb8c06a7a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fcb98007fd0 tx=0x7fcb98007dc0 comp rx=0 tx=0).stop 2023-03-03T11:34:42.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 -- 172.21.15.7:0/3890123715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcba8072520 msgr2=0x7fcba81ed590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:34:42.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcba8072520 0x7fcba81ed590 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009fd0 tx=0x7fcba4004750 comp rx=0 tx=0).stop 2023-03-03T11:34:42.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 -- 172.21.15.7:0/3890123715 shutdown_connections 2023-03-03T11:34:42.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcba8155390 0x7fcba80c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:42.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcba806e020 0x7fcba8152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:42.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb8c0682f0 0x7fcb8c06a7a0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:42.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 --2- 172.21.15.7:0/3890123715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcba8072520 0x7fcba81ed590 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:34:42.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 -- 172.21.15.7:0/3890123715 >> 172.21.15.7:0/3890123715 conn(0x7fcba806c110 msgr2=0x7fcba8152080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:34:42.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 -- 172.21.15.7:0/3890123715 shutdown_connections 2023-03-03T11:34:42.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:34:42.719+0000 7fcbaf91d700 1 -- 172.21.15.7:0/3890123715 wait complete. 2023-03-03T11:34:42.724 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:34:42.733 INFO:tasks.mds_thrash.fs.[b]:waiting for 31 secs before thrashing 2023-03-03T11:34:45.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:34:45.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:13.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:35:13.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.871+0000 7fc92bbb9700 1 Processor -- start 2023-03-03T11:35:13.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.871+0000 7fc92bbb9700 1 -- start start 2023-03-03T11:35:13.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc92bbb9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9240ace30 0x7fc924152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:13.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc92bbb9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9240c26e0 0x7fc924152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:13.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc92bbb9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9240ac900 con 0x7fc9240ac220 2023-03-03T11:35:13.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc92bbb9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc924071e30 con 0x7fc9240ace30 2023-03-03T11:35:13.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc92bbb9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc924152ed0 con 0x7fc9240c26e0 2023-03-03T11:35:13.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc929154700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9240ace30 0x7fc924152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:13.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc929955700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc9240ac220 0x7fc9240736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59128/0 (socket says 172.21.15.7:59128) 2023-03-03T11:35:13.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc929955700 1 -- 172.21.15.7:0/4177626347 learned_addr learned my addr 172.21.15.7:0/4177626347 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:35:13.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc929154700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9240ace30 0x7fc924152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36918/0 (socket says 172.21.15.7:36918) 2023-03-03T11:35:13.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc92a156700 1 --2- 172.21.15.7:0/4177626347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9240c26e0 0x7fc924152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc929154700 1 -- 172.21.15.7:0/4177626347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9240c26e0 msgr2=0x7fc924152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:35:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc929154700 1 --2- 172.21.15.7:0/4177626347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9240c26e0 0x7fc924152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc929154700 1 -- 172.21.15.7:0/4177626347 >> v1:172.21.15.7:6789/0 conn(0x7fc9240ac220 legacy=0x7fc9240736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc929154700 1 -- 172.21.15.7:0/4177626347 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9240acb50 con 0x7fc9240ace30 2023-03-03T11:35:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc928953700 1 -- 172.21.15.7:0/4177626347 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021895619 0 0) 0x7fc9240ac900 con 0x7fc9240ac220 2023-03-03T11:35:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.872+0000 7fc929154700 1 --2- 172.21.15.7:0/4177626347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ace30 0x7fc924152450 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc914009840 tx=0x7fc914009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b27971f1bfd88a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:35:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc928953700 1 -- 172.21.15.7:0/4177626347 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914011830 con 0x7fc9240ace30 2023-03-03T11:35:13.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc928953700 1 -- 172.21.15.7:0/4177626347 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc914011990 con 0x7fc9240ace30 2023-03-03T11:35:13.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc928953700 1 -- 172.21.15.7:0/4177626347 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914011d20 con 0x7fc9240ace30 2023-03-03T11:35:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc92bbb9700 1 -- 172.21.15.7:0/4177626347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ace30 msgr2=0x7fc924152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc92bbb9700 1 --2- 172.21.15.7:0/4177626347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ace30 0x7fc924152450 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc914009840 tx=0x7fc914009b50 comp rx=0 tx=0).stop 2023-03-03T11:35:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc92bbb9700 1 -- 172.21.15.7:0/4177626347 shutdown_connections 2023-03-03T11:35:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc92bbb9700 1 --2- 172.21.15.7:0/4177626347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9240c26e0 0x7fc924152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc92bbb9700 1 --2- 172.21.15.7:0/4177626347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ace30 0x7fc924152450 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc92bbb9700 1 -- 172.21.15.7:0/4177626347 >> 172.21.15.7:0/4177626347 conn(0x7fc9240a94d0 msgr2=0x7fc9240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:35:13.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc92bbb9700 1 -- 172.21.15.7:0/4177626347 shutdown_connections 2023-03-03T11:35:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.873+0000 7fc92bbb9700 1 -- 172.21.15.7:0/4177626347 wait complete. 2023-03-03T11:35:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.874+0000 7fc92bbb9700 1 Processor -- start 2023-03-03T11:35:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.875+0000 7fc92bbb9700 1 -- start start 2023-03-03T11:35:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.875+0000 7fc92bbb9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ac220 0x7fc9241e9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.875+0000 7fc92bbb9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9240ace30 0x7fc9240ba1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.875+0000 7fc92bbb9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9240c26e0 0x7fc924154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.875+0000 7fc92bbb9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241ddb30 con 0x7fc9240ac220 2023-03-03T11:35:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.875+0000 7fc92bbb9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241de7f0 con 0x7fc9240c26e0 2023-03-03T11:35:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.875+0000 7fc92bbb9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9240ba710 con 0x7fc9240ace30 2023-03-03T11:35:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.875+0000 7fc929955700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ac220 0x7fc9241e9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.875+0000 7fc929955700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ac220 0x7fc9241e9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36922/0 (socket says 172.21.15.7:36922) 2023-03-03T11:35:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.875+0000 7fc929955700 1 -- 172.21.15.7:0/3058411947 learned_addr learned my addr 172.21.15.7:0/3058411947 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:35:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc929955700 1 -- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9240ace30 msgr2=0x7fc9240ba1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc929955700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9240ace30 0x7fc9240ba1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc929955700 1 -- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9240c26e0 msgr2=0x7fc924154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:13.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc929955700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9240c26e0 0x7fc924154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:13.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc929955700 1 -- 172.21.15.7:0/3058411947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9140094a0 con 0x7fc9240ac220 2023-03-03T11:35:13.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc929955700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ac220 0x7fc9241e9980 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc92000c5b0 tx=0x7fc92000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:35:13.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc91a7fc700 1 -- 172.21.15.7:0/3058411947 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920004c80 con 0x7fc9240ac220 2023-03-03T11:35:13.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc91a7fc700 1 -- 172.21.15.7:0/3058411947 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920004de0 con 0x7fc9240ac220 2023-03-03T11:35:13.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc91a7fc700 1 -- 172.21.15.7:0/3058411947 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920007310 con 0x7fc9240ac220 2023-03-03T11:35:13.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc92bbb9700 1 -- 172.21.15.7:0/3058411947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc924155ca0 con 0x7fc9240ac220 2023-03-03T11:35:13.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.876+0000 7fc92bbb9700 1 -- 172.21.15.7:0/3058411947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9241559c0 con 0x7fc9240ac220 2023-03-03T11:35:13.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.877+0000 7fc92bbb9700 1 -- 172.21.15.7:0/3058411947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc924155f80 con 0x7fc9240ac220 2023-03-03T11:35:13.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.878+0000 7fc91a7fc700 1 -- 172.21.15.7:0/3058411947 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc92001a020 con 0x7fc9240ac220 2023-03-03T11:35:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.878+0000 7fc91a7fc700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9080682f0 0x7fc90806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.878+0000 7fc91a7fc700 1 -- 172.21.15.7:0/3058411947 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc920085d00 con 0x7fc9240ac220 2023-03-03T11:35:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.878+0000 7fc929154700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9080682f0 0x7fc90806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.879+0000 7fc929154700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9080682f0 0x7fc90806a7a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc924073fb0 tx=0x7fc914010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:35:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:13.882+0000 7fc91a7fc700 1 -- 172.21.15.7:0/3058411947 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc920089030 con 0x7fc9240ac220 2023-03-03T11:35:14.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.118+0000 7fc92bbb9700 1 -- 172.21.15.7:0/3058411947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc92406cf30 con 0x7fc9240ac220 2023-03-03T11:35:14.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.119+0000 7fc91a7fc700 1 -- 172.21.15.7:0/3058411947 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc92406cf30 con 0x7fc9240ac220 2023-03-03T11:35:14.120 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:35:14.121 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:35:14.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.120+0000 7fc92bbb9700 1 -- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9080682f0 msgr2=0x7fc90806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:14.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.120+0000 7fc92bbb9700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9080682f0 0x7fc90806a7a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc924073fb0 tx=0x7fc914010000 comp rx=0 tx=0).stop 2023-03-03T11:35:14.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.121+0000 7fc92bbb9700 1 -- 172.21.15.7:0/3058411947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ac220 msgr2=0x7fc9241e9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:14.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.121+0000 7fc92bbb9700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ac220 0x7fc9241e9980 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc92000c5b0 tx=0x7fc92000c970 comp rx=0 tx=0).stop 2023-03-03T11:35:14.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.121+0000 7fc92bbb9700 1 -- 172.21.15.7:0/3058411947 shutdown_connections 2023-03-03T11:35:14.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.121+0000 7fc92bbb9700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9240c26e0 0x7fc924154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:14.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.121+0000 7fc92bbb9700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9080682f0 0x7fc90806a7a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:14.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.121+0000 7fc92bbb9700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9240ac220 0x7fc9241e9980 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:14.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.121+0000 7fc92bbb9700 1 --2- 172.21.15.7:0/3058411947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc9240ace30 0x7fc9240ba1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:14.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.121+0000 7fc92bbb9700 1 -- 172.21.15.7:0/3058411947 >> 172.21.15.7:0/3058411947 conn(0x7fc9240a94d0 msgr2=0x7fc9240ab1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:35:14.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.121+0000 7fc92bbb9700 1 -- 172.21.15.7:0/3058411947 shutdown_connections 2023-03-03T11:35:14.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:14.121+0000 7fc92bbb9700 1 -- 172.21.15.7:0/3058411947 wait complete. 2023-03-03T11:35:14.124 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:35:14.134 INFO:tasks.mds_thrash.fs.[b]:waiting for 77 secs before thrashing 2023-03-03T11:35:15.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:15.723 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:45.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:45.795 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:35:51.917 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:35:52.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50e1259700 1 Processor -- start 2023-03-03T11:35:52.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50e1259700 1 -- start start 2023-03-03T11:35:52.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50e1259700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0caa30 0x7f50dc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:52.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50e1259700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f50dc154ff0 0x7f50dc0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:52.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50e1259700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50dc0c7310 con 0x7f50dc0c00e0 2023-03-03T11:35:52.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50e1259700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50dc0bfd10 con 0x7f50dc154ff0 2023-03-03T11:35:52.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50e1259700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc0bfa30 con 0x7f50dc0caa30 2023-03-03T11:35:52.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50da59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f50dc0c00e0 0x7f50dc0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59168/0 (socket says 172.21.15.7:59168) 2023-03-03T11:35:52.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50da59c700 1 -- 172.21.15.7:0/1575461433 learned_addr learned my addr 172.21.15.7:0/1575461433 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:35:52.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439773708 0 0) 0x7f50dc0c7310 con 0x7f50dc0c00e0 2023-03-03T11:35:52.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c40035b0 con 0x7f50dc0c00e0 2023-03-03T11:35:52.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50dad9d700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0caa30 0x7f50dc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:52.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50db59e700 1 --2- 172.21.15.7:0/1575461433 >> v2:172.21.15.143:3300/0 conn(0x7f50dc154ff0 0x7f50dc0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:52.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2316877225 0 0) 0x7f50c40035b0 con 0x7f50dc0c00e0 2023-03-03T11:35:52.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0caa30 msgr2=0x7f50dc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:35:52.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0caa30 0x7f50dc0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:52.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 >> v2:172.21.15.143:3300/0 conn(0x7f50dc154ff0 msgr2=0x7f50dc0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:35:52.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 --2- 172.21.15.7:0/1575461433 >> v2:172.21.15.143:3300/0 conn(0x7f50dc154ff0 0x7f50dc0cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:52.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc154510 con 0x7f50dc0c00e0 2023-03-03T11:35:52.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f50d00017f0 con 0x7f50dc0c00e0 2023-03-03T11:35:52.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50dad9d700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0caa30 0x7f50dc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:35:52.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 >> v1:172.21.15.7:6789/0 conn(0x7f50dc0c00e0 legacy=0x7f50dc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:52.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50c4003a70 0x7f50c4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:52.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0caa30 0x7f50c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:52.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50c4004d90 0x7f50c4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:52.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc0bfd10 con 0x7f50c4004d90 2023-03-03T11:35:52.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc0bfa30 con 0x7f50c4003a70 2023-03-03T11:35:52.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c400bac0 con 0x7f50dc0caa30 2023-03-03T11:35:52.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50db59e700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50c4004d90 0x7f50c4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:52.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.055+0000 7f50dad9d700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50c4003a70 0x7f50c4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:52.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50da59c700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0caa30 0x7f50c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:52.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50db59e700 1 -- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0caa30 msgr2=0x7f50c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:35:52.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50db59e700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0caa30 0x7f50c4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:52.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50db59e700 1 -- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50c4003a70 msgr2=0x7f50c4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:35:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50db59e700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50c4003a70 0x7f50c4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50db59e700 1 -- 172.21.15.7:0/1575461433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c40035b0 con 0x7f50c4004d90 2023-03-03T11:35:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50db59e700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50c4004d90 0x7f50c4005200 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009fc0 tx=0x7f50c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dac1abd656f75d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:35:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d400ec30 con 0x7f50c4004d90 2023-03-03T11:35:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d400f040 con 0x7f50c4004d90 2023-03-03T11:35:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50d9d9b700 1 -- 172.21.15.7:0/1575461433 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4014480 con 0x7f50c4004d90 2023-03-03T11:35:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50e1259700 1 -- 172.21.15.7:0/1575461433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50c4004d90 msgr2=0x7f50c4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.056+0000 7f50e1259700 1 --2- 172.21.15.7:0/1575461433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50c4004d90 0x7f50c4005200 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009fc0 tx=0x7f50c4005880 comp rx=0 tx=0).stop 2023-03-03T11:35:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.057+0000 7f50db59e700 1 -- 172.21.15.7:0/1575461433 reap_dead start 2023-03-03T11:35:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.057+0000 7f50e1259700 1 -- 172.21.15.7:0/1575461433 shutdown_connections 2023-03-03T11:35:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.057+0000 7f50e1259700 1 -- 172.21.15.7:0/1575461433 >> 172.21.15.7:0/1575461433 conn(0x7f50dc066e40 msgr2=0x7f50dc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:35:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.057+0000 7f50e1259700 1 -- 172.21.15.7:0/1575461433 shutdown_connections 2023-03-03T11:35:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.057+0000 7f50e1259700 1 -- 172.21.15.7:0/1575461433 wait complete. 2023-03-03T11:35:52.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.058+0000 7f50e1259700 1 Processor -- start 2023-03-03T11:35:52.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.058+0000 7f50e1259700 1 -- start start 2023-03-03T11:35:52.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50e1259700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50dc0c2d10 0x7f50dc157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:52.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50e1259700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0c30e0 0x7f50dc0c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:52.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50e1259700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50dc0c45f0 0x7f50dc149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:52.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50e1259700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc0c3c60 con 0x7f50dc0c2d10 2023-03-03T11:35:52.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50e1259700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc1561d0 con 0x7f50dc0c45f0 2023-03-03T11:35:52.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50e1259700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc155610 con 0x7f50dc0c30e0 2023-03-03T11:35:52.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50dad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50dc0c2d10 0x7f50dc157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:52.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50dad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50dc0c2d10 0x7f50dc157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36968/0 (socket says 172.21.15.7:36968) 2023-03-03T11:35:52.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50dad9d700 1 -- 172.21.15.7:0/3051569522 learned_addr learned my addr 172.21.15.7:0/3051569522 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:35:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50db59e700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50dc0c45f0 0x7f50dc149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50dad9d700 1 -- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0c30e0 msgr2=0x7f50dc0c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50dad9d700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0c30e0 0x7f50dc0c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50dad9d700 1 -- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50dc0c45f0 msgr2=0x7f50dc149a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50dad9d700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50dc0c45f0 0x7f50dc149a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.059+0000 7f50dad9d700 1 -- 172.21.15.7:0/3051569522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc154ad0 con 0x7f50dc0c2d10 2023-03-03T11:35:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.060+0000 7f50dad9d700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50dc0c2d10 0x7f50dc157970 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0098b0 tx=0x7f50cc009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:35:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.060+0000 7f50c37fe700 1 -- 172.21.15.7:0/3051569522 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc010880 con 0x7f50dc0c2d10 2023-03-03T11:35:52.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.060+0000 7f50c37fe700 1 -- 172.21.15.7:0/3051569522 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50cc0109e0 con 0x7f50dc0c2d10 2023-03-03T11:35:52.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.060+0000 7f50e1259700 1 -- 172.21.15.7:0/3051569522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50dc1547f0 con 0x7f50dc0c2d10 2023-03-03T11:35:52.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.060+0000 7f50c37fe700 1 -- 172.21.15.7:0/3051569522 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc010d70 con 0x7f50dc0c2d10 2023-03-03T11:35:52.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.060+0000 7f50e1259700 1 -- 172.21.15.7:0/3051569522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50dc0bf650 con 0x7f50dc0c2d10 2023-03-03T11:35:52.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.061+0000 7f50e1259700 1 -- 172.21.15.7:0/3051569522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50dc0c3f00 con 0x7f50dc0c2d10 2023-03-03T11:35:52.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.061+0000 7f50c37fe700 1 -- 172.21.15.7:0/3051569522 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f50cc025070 con 0x7f50dc0c2d10 2023-03-03T11:35:52.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.062+0000 7f50c37fe700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50c80682f0 0x7f50c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:35:52.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.062+0000 7f50c37fe700 1 -- 172.21.15.7:0/3051569522 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f50cc0868b0 con 0x7f50dc0c2d10 2023-03-03T11:35:52.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.062+0000 7f50da59c700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50c80682f0 0x7f50c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:35:52.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.062+0000 7f50da59c700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50c80682f0 0x7f50c806a7a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f50d00011d0 tx=0x7f50d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:35:52.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.067+0000 7f50c37fe700 1 -- 172.21.15.7:0/3051569522 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f50cc052c50 con 0x7f50dc0c2d10 2023-03-03T11:35:52.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.303+0000 7f50e1259700 1 -- 172.21.15.7:0/3051569522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50dc155470 con 0x7f50dc0c2d10 2023-03-03T11:35:52.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.304+0000 7f50c37fe700 1 -- 172.21.15.7:0/3051569522 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f50dc155470 con 0x7f50dc0c2d10 2023-03-03T11:35:52.307 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:35:52.307 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:35:52.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 -- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50c80682f0 msgr2=0x7f50c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:52.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50c80682f0 0x7f50c806a7a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f50d00011d0 tx=0x7f50d0009040 comp rx=0 tx=0).stop 2023-03-03T11:35:52.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 -- 172.21.15.7:0/3051569522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50dc0c2d10 msgr2=0x7f50dc157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:35:52.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50dc0c2d10 0x7f50dc157970 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0098b0 tx=0x7f50cc009c70 comp rx=0 tx=0).stop 2023-03-03T11:35:52.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 -- 172.21.15.7:0/3051569522 shutdown_connections 2023-03-03T11:35:52.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50dc0c45f0 0x7f50dc149a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:52.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50c80682f0 0x7f50c806a7a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:52.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50dc0c2d10 0x7f50dc157970 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:52.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 --2- 172.21.15.7:0/3051569522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50dc0c30e0 0x7f50dc0c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:35:52.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 -- 172.21.15.7:0/3051569522 >> 172.21.15.7:0/3051569522 conn(0x7f50dc066e40 msgr2=0x7f50dc0c68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:35:52.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 -- 172.21.15.7:0/3051569522 shutdown_connections 2023-03-03T11:35:52.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:35:52.306+0000 7f50e1259700 1 -- 172.21.15.7:0/3051569522 wait complete. 2023-03-03T11:35:52.311 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:35:52.321 INFO:tasks.mds_thrash.fs.[a]:waiting for 19 secs before thrashing 2023-03-03T11:36:11.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:36:11.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.461+0000 7f08f274a700 1 Processor -- start 2023-03-03T11:36:11.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.461+0000 7f08f274a700 1 -- start start 2023-03-03T11:36:11.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08f274a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec0aa030 0x7f08ec158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:11.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08f274a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec0a9c20 con 0x7f08ec0aad20 2023-03-03T11:36:11.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08f274a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec154f10 con 0x7f08ec14b0d0 2023-03-03T11:36:11.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08f274a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0ac3c0 con 0x7f08ec0aa030 2023-03-03T11:36:11.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08eb7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f08ec0aad20 0x7f08ec157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59198/0 (socket says 172.21.15.7:59198) 2023-03-03T11:36:11.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08eb7fe700 1 -- 172.21.15.7:0/1855957135 learned_addr learned my addr 172.21.15.7:0/1855957135 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:36:11.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08f0ce7700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec0aa030 0x7f08ec158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:11.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424924083 0 0) 0x7f08ec0a9c20 con 0x7f08ec0aad20 2023-03-03T11:36:11.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d80035b0 con 0x7f08ec0aad20 2023-03-03T11:36:11.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224788941 0 0) 0x7f08ec154f10 con 0x7f08ec14b0d0 2023-03-03T11:36:11.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ec0a9c20 con 0x7f08ec14b0d0 2023-03-03T11:36:11.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2790094750 0 0) 0x7f08d80035b0 con 0x7f08ec0aad20 2023-03-03T11:36:11.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec0aa030 msgr2=0x7f08ec158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:36:11.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08eaffd700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec0aa030 0x7f08ec158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:11.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.462+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 >> v1:172.21.15.143:6789/0 conn(0x7f08ec14b0d0 legacy=0x7f08ec071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:11.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec06fc70 con 0x7f08ec0aad20 2023-03-03T11:36:11.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f08d4001670 con 0x7f08ec0aad20 2023-03-03T11:36:11.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 >> v1:172.21.15.7:6789/0 conn(0x7f08ec0aad20 legacy=0x7f08ec157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:11.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d8003c30 0x7f08d8004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:11.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 0x7f08d8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:11.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec0aa030 0x7f08d80052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:11.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0ac3c0 con 0x7f08d8004850 2023-03-03T11:36:11.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d800bc00 con 0x7f08d8003c30 2023-03-03T11:36:11.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d800bdc0 con 0x7f08ec0aa030 2023-03-03T11:36:11.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eb7fe700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 0x7f08d8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:11.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08ebfff700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d8003c30 0x7f08d8004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:11.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08f0ce7700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec0aa030 0x7f08d80052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:11.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eb7fe700 1 -- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec0aa030 msgr2=0x7f08d80052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:36:11.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eb7fe700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec0aa030 0x7f08d80052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:11.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eb7fe700 1 -- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d8003c30 msgr2=0x7f08d8004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:36:11.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eb7fe700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d8003c30 0x7f08d8004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:11.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eb7fe700 1 -- 172.21.15.7:0/1855957135 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d80035b0 con 0x7f08d8004850 2023-03-03T11:36:11.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eb7fe700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 0x7f08d8004cc0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0ab1a0 tx=0x7f08d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0518e99f38f6cb0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:11.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d400cc50 con 0x7f08d8004850 2023-03-03T11:36:11.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d400cdd0 con 0x7f08d8004850 2023-03-03T11:36:11.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.463+0000 7f08eaffd700 1 -- 172.21.15.7:0/1855957135 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4005ba0 con 0x7f08d8004850 2023-03-03T11:36:11.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.464+0000 7f08f274a700 1 -- 172.21.15.7:0/1855957135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 msgr2=0x7f08d8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:11.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.464+0000 7f08f274a700 1 --2- 172.21.15.7:0/1855957135 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 0x7f08d8004cc0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0ab1a0 tx=0x7f08d8005880 comp rx=0 tx=0).stop 2023-03-03T11:36:11.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.464+0000 7f08f0ce7700 1 -- 172.21.15.7:0/1855957135 reap_dead start 2023-03-03T11:36:11.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.464+0000 7f08f274a700 1 -- 172.21.15.7:0/1855957135 shutdown_connections 2023-03-03T11:36:11.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.464+0000 7f08f274a700 1 -- 172.21.15.7:0/1855957135 >> 172.21.15.7:0/1855957135 conn(0x7f08ec0c8900 msgr2=0x7f08ec0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:36:11.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.464+0000 7f08f274a700 1 -- 172.21.15.7:0/1855957135 shutdown_connections 2023-03-03T11:36:11.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.464+0000 7f08f274a700 1 -- 172.21.15.7:0/1855957135 wait complete. 2023-03-03T11:36:11.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.466+0000 7f08f274a700 1 Processor -- start 2023-03-03T11:36:11.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.466+0000 7f08f274a700 1 -- start start 2023-03-03T11:36:11.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08f274a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 0x7f08ec0c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:11.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08f274a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08ec0c6310 0x7f08ec143040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:11.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08f274a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec143580 0x7f08ec1ed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:11.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08f274a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec145cd0 con 0x7f08d8004850 2023-03-03T11:36:11.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08f274a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec146570 con 0x7f08ec0c6310 2023-03-03T11:36:11.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08f274a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec143d20 con 0x7f08ec143580 2023-03-03T11:36:11.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08ebfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 0x7f08ec0c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:11.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08ebfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 0x7f08ec0c7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36996/0 (socket says 172.21.15.7:36996) 2023-03-03T11:36:11.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08ebfff700 1 -- 172.21.15.7:0/2385023432 learned_addr learned my addr 172.21.15.7:0/2385023432 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:36:11.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08eb7fe700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08ec0c6310 0x7f08ec143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:11.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08f0ce7700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec143580 0x7f08ec1ed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:11.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08ebfff700 1 -- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec143580 msgr2=0x7f08ec1ed3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:11.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08ebfff700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec143580 0x7f08ec1ed3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:11.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08ebfff700 1 -- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08ec0c6310 msgr2=0x7f08ec143040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:11.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08ebfff700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08ec0c6310 0x7f08ec143040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:11.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08ebfff700 1 -- 172.21.15.7:0/2385023432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec06f6b0 con 0x7f08d8004850 2023-03-03T11:36:11.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08ebfff700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 0x7f08ec0c7c70 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00f810 tx=0x7f08dc00fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:11.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.467+0000 7f08e8ff9700 1 -- 172.21.15.7:0/2385023432 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc005320 con 0x7f08d8004850 2023-03-03T11:36:11.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.468+0000 7f08e8ff9700 1 -- 172.21.15.7:0/2385023432 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08dc0054a0 con 0x7f08d8004850 2023-03-03T11:36:11.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.468+0000 7f08f274a700 1 -- 172.21.15.7:0/2385023432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec155160 con 0x7f08d8004850 2023-03-03T11:36:11.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.468+0000 7f08e8ff9700 1 -- 172.21.15.7:0/2385023432 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc006aa0 con 0x7f08d8004850 2023-03-03T11:36:11.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.468+0000 7f08f274a700 1 -- 172.21.15.7:0/2385023432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec06f990 con 0x7f08d8004850 2023-03-03T11:36:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.468+0000 7f08f274a700 1 -- 172.21.15.7:0/2385023432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec074bb0 con 0x7f08d8004850 2023-03-03T11:36:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.470+0000 7f08e8ff9700 1 -- 172.21.15.7:0/2385023432 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f08dc003680 con 0x7f08d8004850 2023-03-03T11:36:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.470+0000 7f08e8ff9700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08cc0682f0 0x7f08cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.470+0000 7f08e8ff9700 1 -- 172.21.15.7:0/2385023432 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f08dc085b90 con 0x7f08d8004850 2023-03-03T11:36:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.470+0000 7f08eb7fe700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08cc0682f0 0x7f08cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.471+0000 7f08eb7fe700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08cc0682f0 0x7f08cc06a7a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f08d4015040 tx=0x7f08d4007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:11.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.474+0000 7f08e8ff9700 1 -- 172.21.15.7:0/2385023432 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08dc051fd0 con 0x7f08d8004850 2023-03-03T11:36:11.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.714+0000 7f08f274a700 1 -- 172.21.15.7:0/2385023432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08ec0aa540 con 0x7f08d8004850 2023-03-03T11:36:11.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.715+0000 7f08e8ff9700 1 -- 172.21.15.7:0/2385023432 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f08dc0555f0 con 0x7f08d8004850 2023-03-03T11:36:11.717 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:36:11.717 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:36:11.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 -- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08cc0682f0 msgr2=0x7f08cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:11.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08cc0682f0 0x7f08cc06a7a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f08d4015040 tx=0x7f08d4007000 comp rx=0 tx=0).stop 2023-03-03T11:36:11.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 -- 172.21.15.7:0/2385023432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 msgr2=0x7f08ec0c7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:11.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 0x7f08ec0c7c70 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00f810 tx=0x7f08dc00fbd0 comp rx=0 tx=0).stop 2023-03-03T11:36:11.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 -- 172.21.15.7:0/2385023432 shutdown_connections 2023-03-03T11:36:11.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08ec143580 0x7f08ec1ed3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:11.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08cc0682f0 0x7f08cc06a7a0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:11.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d8004850 0x7f08ec0c7c70 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:11.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 --2- 172.21.15.7:0/2385023432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08ec0c6310 0x7f08ec143040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:11.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 -- 172.21.15.7:0/2385023432 >> 172.21.15.7:0/2385023432 conn(0x7f08ec0c8900 msgr2=0x7f08ec0c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:36:11.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 -- 172.21.15.7:0/2385023432 shutdown_connections 2023-03-03T11:36:11.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:11.717+0000 7f08f274a700 1 -- 172.21.15.7:0/2385023432 wait complete. 2023-03-03T11:36:11.720 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:36:11.730 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T11:36:15.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:15.940 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:31.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:36:31.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.271+0000 7f6d84452700 1 Processor -- start 2023-03-03T11:36:31.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.271+0000 7f6d84452700 1 -- start start 2023-03-03T11:36:31.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.272+0000 7f6d84452700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14fda0 0x7f6d7c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:31.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.272+0000 7f6d84452700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7c0c2f30 con 0x7f6d7c0abe20 2023-03-03T11:36:31.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.272+0000 7f6d84452700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7c0c31d0 con 0x7f6d7c14f9d0 2023-03-03T11:36:31.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.272+0000 7f6d84452700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c1524a0 con 0x7f6d7c14fda0 2023-03-03T11:36:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d829ef700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6d7c0abe20 0x7f6d7c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59228/0 (socket says 172.21.15.7:59228) 2023-03-03T11:36:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d829ef700 1 -- 172.21.15.7:0/1070638128 learned_addr learned my addr 172.21.15.7:0/1070638128 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:36:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d819ed700 1 --2- 172.21.15.7:0/1070638128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14fda0 0x7f6d7c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d811ec700 1 -- 172.21.15.7:0/1070638128 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131761350 0 0) 0x7f6d7c0c2f30 con 0x7f6d7c0abe20 2023-03-03T11:36:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d811ec700 1 -- 172.21.15.7:0/1070638128 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d680035b0 con 0x7f6d7c0abe20 2023-03-03T11:36:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d811ec700 1 -- 172.21.15.7:0/1070638128 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957985021 0 0) 0x7f6d7c0c31d0 con 0x7f6d7c14f9d0 2023-03-03T11:36:31.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d811ec700 1 -- 172.21.15.7:0/1070638128 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c0c2f30 con 0x7f6d7c14f9d0 2023-03-03T11:36:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d811ec700 1 -- 172.21.15.7:0/1070638128 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1101425632 0 0) 0x7f6d680035b0 con 0x7f6d7c0abe20 2023-03-03T11:36:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d819ed700 1 -- 172.21.15.7:0/1070638128 >> v1:172.21.15.143:6789/0 conn(0x7f6d7c14f9d0 legacy=0x7f6d7c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d819ed700 1 -- 172.21.15.7:0/1070638128 >> v1:172.21.15.7:6789/0 conn(0x7f6d7c0abe20 legacy=0x7f6d7c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d819ed700 1 -- 172.21.15.7:0/1070638128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c0c34b0 con 0x7f6d7c14fda0 2023-03-03T11:36:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.273+0000 7f6d819ed700 1 --2- 172.21.15.7:0/1070638128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14fda0 0x7f6d7c156430 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009870 tx=0x7f6d70009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffc8d91cb72c71e4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.274+0000 7f6d811ec700 1 -- 172.21.15.7:0/1070638128 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d740019f0 con 0x7f6d7c14fda0 2023-03-03T11:36:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.274+0000 7f6d811ec700 1 -- 172.21.15.7:0/1070638128 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d70004030 con 0x7f6d7c14fda0 2023-03-03T11:36:31.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.274+0000 7f6d811ec700 1 -- 172.21.15.7:0/1070638128 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d700043c0 con 0x7f6d7c14fda0 2023-03-03T11:36:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.274+0000 7f6d84452700 1 -- 172.21.15.7:0/1070638128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14fda0 msgr2=0x7f6d7c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.274+0000 7f6d84452700 1 --2- 172.21.15.7:0/1070638128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14fda0 0x7f6d7c156430 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009870 tx=0x7f6d70009b80 comp rx=0 tx=0).stop 2023-03-03T11:36:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.274+0000 7f6d84452700 1 -- 172.21.15.7:0/1070638128 shutdown_connections 2023-03-03T11:36:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.274+0000 7f6d84452700 1 --2- 172.21.15.7:0/1070638128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14fda0 0x7f6d7c156430 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.274+0000 7f6d84452700 1 -- 172.21.15.7:0/1070638128 >> 172.21.15.7:0/1070638128 conn(0x7f6d7c0c8900 msgr2=0x7f6d7c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:36:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.274+0000 7f6d84452700 1 -- 172.21.15.7:0/1070638128 shutdown_connections 2023-03-03T11:36:31.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.274+0000 7f6d84452700 1 -- 172.21.15.7:0/1070638128 wait complete. 2023-03-03T11:36:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.275+0000 7f6d84452700 1 Processor -- start 2023-03-03T11:36:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.276+0000 7f6d84452700 1 -- start start 2023-03-03T11:36:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.276+0000 7f6d84452700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d7c0abe20 0x7f6d7c06f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.276+0000 7f6d84452700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14f9d0 0x7f6d7c1e9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.276+0000 7f6d84452700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c14fda0 0x7f6d7c1ea1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.276+0000 7f6d84452700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c0bdac0 con 0x7f6d7c14fda0 2023-03-03T11:36:31.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.276+0000 7f6d84452700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c0be9d0 con 0x7f6d7c0abe20 2023-03-03T11:36:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.276+0000 7f6d84452700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c1e7e50 con 0x7f6d7c14f9d0 2023-03-03T11:36:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.276+0000 7f6d829ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c14fda0 0x7f6d7c1ea1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.276+0000 7f6d829ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c14fda0 0x7f6d7c1ea1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37022/0 (socket says 172.21.15.7:37022) 2023-03-03T11:36:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.276+0000 7f6d829ef700 1 -- 172.21.15.7:0/1945881975 learned_addr learned my addr 172.21.15.7:0/1945881975 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:36:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d821ee700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d7c0abe20 0x7f6d7c06f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d819ed700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14f9d0 0x7f6d7c1e9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d829ef700 1 -- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14f9d0 msgr2=0x7f6d7c1e9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:31.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d829ef700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14f9d0 0x7f6d7c1e9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d829ef700 1 -- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d7c0abe20 msgr2=0x7f6d7c06f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d829ef700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d7c0abe20 0x7f6d7c06f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d829ef700 1 -- 172.21.15.7:0/1945881975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d700094a0 con 0x7f6d7c14fda0 2023-03-03T11:36:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d821ee700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d7c0abe20 0x7f6d7c06f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:36:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d819ed700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14f9d0 0x7f6d7c1e9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:36:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d829ef700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c14fda0 0x7f6d7c1ea1c0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400cc20 tx=0x7f6d7400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d6effd700 1 -- 172.21.15.7:0/1945881975 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7400ac20 con 0x7f6d7c14fda0 2023-03-03T11:36:31.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d6effd700 1 -- 172.21.15.7:0/1945881975 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7400ad80 con 0x7f6d7c14fda0 2023-03-03T11:36:31.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d84452700 1 -- 172.21.15.7:0/1945881975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7c0ac350 con 0x7f6d7c14fda0 2023-03-03T11:36:31.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d6effd700 1 -- 172.21.15.7:0/1945881975 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d74007310 con 0x7f6d7c14fda0 2023-03-03T11:36:31.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.277+0000 7f6d84452700 1 -- 172.21.15.7:0/1945881975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7c0ab630 con 0x7f6d7c14fda0 2023-03-03T11:36:31.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.278+0000 7f6d84452700 1 -- 172.21.15.7:0/1945881975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d7c071fa0 con 0x7f6d7c14fda0 2023-03-03T11:36:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.279+0000 7f6d6effd700 1 -- 172.21.15.7:0/1945881975 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6d7401a030 con 0x7f6d7c14fda0 2023-03-03T11:36:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.279+0000 7f6d6effd700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6d600682f0 0x7f6d6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.279+0000 7f6d6effd700 1 -- 172.21.15.7:0/1945881975 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6d74085f40 con 0x7f6d7c14fda0 2023-03-03T11:36:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.279+0000 7f6d821ee700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6d600682f0 0x7f6d6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.280+0000 7f6d821ee700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6d600682f0 0x7f6d6006a7a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6d780011e0 tx=0x7f6d78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:31.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.283+0000 7f6d6effd700 1 -- 172.21.15.7:0/1945881975 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6d740522e0 con 0x7f6d7c14fda0 2023-03-03T11:36:31.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.519+0000 7f6d84452700 1 -- 172.21.15.7:0/1945881975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d7c1e8730 con 0x7f6d7c14fda0 2023-03-03T11:36:31.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.520+0000 7f6d6effd700 1 -- 172.21.15.7:0/1945881975 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6d7c1e8730 con 0x7f6d7c14fda0 2023-03-03T11:36:31.522 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:36:31.522 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:36:31.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 -- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6d600682f0 msgr2=0x7f6d6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:31.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6d600682f0 0x7f6d6006a7a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6d780011e0 tx=0x7f6d78009040 comp rx=0 tx=0).stop 2023-03-03T11:36:31.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 -- 172.21.15.7:0/1945881975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c14fda0 msgr2=0x7f6d7c1ea1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:31.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c14fda0 0x7f6d7c1ea1c0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400cc20 tx=0x7f6d7400a060 comp rx=0 tx=0).stop 2023-03-03T11:36:31.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 -- 172.21.15.7:0/1945881975 shutdown_connections 2023-03-03T11:36:31.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6d600682f0 0x7f6d6006a7a0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d7c14fda0 0x7f6d7c1ea1c0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d7c0abe20 0x7f6d7c06f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 --2- 172.21.15.7:0/1945881975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d7c14f9d0 0x7f6d7c1e9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 -- 172.21.15.7:0/1945881975 >> 172.21.15.7:0/1945881975 conn(0x7f6d7c0c8900 msgr2=0x7f6d7c152c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:36:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 -- 172.21.15.7:0/1945881975 shutdown_connections 2023-03-03T11:36:31.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:31.522+0000 7f6d84452700 1 -- 172.21.15.7:0/1945881975 wait complete. 2023-03-03T11:36:31.526 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:36:31.536 INFO:tasks.mds_thrash.fs.[b]:waiting for 13 secs before thrashing 2023-03-03T11:36:44.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:36:44.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.673+0000 7f9c39298700 1 Processor -- start 2023-03-03T11:36:44.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.673+0000 7f9c39298700 1 -- start start 2023-03-03T11:36:44.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c39298700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c3414b340 0x7f9c34152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:44.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c39298700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c340c2790 0x7f9c34152b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:44.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c39298700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c3406c920 con 0x7f9c34151140 2023-03-03T11:36:44.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c39298700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c340abd90 con 0x7f9c3414b340 2023-03-03T11:36:44.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c39298700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c340c1300 con 0x7f9c340c2790 2023-03-03T11:36:44.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c32ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c3414b340 0x7f9c34152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:44.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c32ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c3414b340 0x7f9c34152630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37032/0 (socket says 172.21.15.7:37032) 2023-03-03T11:36:44.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c32ffd700 1 -- 172.21.15.7:0/1383772017 learned_addr learned my addr 172.21.15.7:0/1383772017 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:36:44.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c32ffd700 1 -- 172.21.15.7:0/1383772017 >> v2:172.21.15.143:3300/0 conn(0x7f9c340c2790 msgr2=0x7f9c34152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:36:44.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c327fc700 1 --2- 172.21.15.7:0/1383772017 >> v2:172.21.15.143:3300/0 conn(0x7f9c340c2790 0x7f9c34152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:44.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c32ffd700 1 --2- 172.21.15.7:0/1383772017 >> v2:172.21.15.143:3300/0 conn(0x7f9c340c2790 0x7f9c34152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:44.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c32ffd700 1 -- 172.21.15.7:0/1383772017 >> v1:172.21.15.143:6789/0 conn(0x7f9c34151140 legacy=0x7f9c341515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:36:44.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c32ffd700 1 -- 172.21.15.7:0/1383772017 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3406bd50 con 0x7f9c3414b340 2023-03-03T11:36:44.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c327fc700 1 --2- 172.21.15.7:0/1383772017 >> v2:172.21.15.143:3300/0 conn(0x7f9c340c2790 0x7f9c34152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:36:44.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.674+0000 7f9c32ffd700 1 --2- 172.21.15.7:0/1383772017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 0x7f9c34152630 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9c340c2c90 tx=0x7f9c2800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83b2e3909c551a4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:44.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c31ffb700 1 -- 172.21.15.7:0/1383772017 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2800dcf0 con 0x7f9c3414b340 2023-03-03T11:36:44.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c31ffb700 1 -- 172.21.15.7:0/1383772017 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2800de50 con 0x7f9c3414b340 2023-03-03T11:36:44.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c31ffb700 1 -- 172.21.15.7:0/1383772017 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28004250 con 0x7f9c3414b340 2023-03-03T11:36:44.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c39298700 1 -- 172.21.15.7:0/1383772017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 msgr2=0x7f9c34152630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:44.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c39298700 1 --2- 172.21.15.7:0/1383772017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 0x7f9c34152630 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9c340c2c90 tx=0x7f9c2800be70 comp rx=0 tx=0).stop 2023-03-03T11:36:44.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c39298700 1 -- 172.21.15.7:0/1383772017 shutdown_connections 2023-03-03T11:36:44.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c39298700 1 --2- 172.21.15.7:0/1383772017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 0x7f9c34152630 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:44.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c39298700 1 --2- 172.21.15.7:0/1383772017 >> v2:172.21.15.143:3300/0 conn(0x7f9c340c2790 0x7f9c34152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:44.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c39298700 1 -- 172.21.15.7:0/1383772017 >> 172.21.15.7:0/1383772017 conn(0x7f9c340c62e0 msgr2=0x7f9c340c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:36:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c39298700 1 -- 172.21.15.7:0/1383772017 shutdown_connections 2023-03-03T11:36:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.675+0000 7f9c39298700 1 -- 172.21.15.7:0/1383772017 wait complete. 2023-03-03T11:36:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.677+0000 7f9c39298700 1 Processor -- start 2023-03-03T11:36:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.677+0000 7f9c39298700 1 -- start start 2023-03-03T11:36:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c39298700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c340c2790 0x7f9c341dc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c39298700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 0x7f9c341dd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:44.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c39298700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34151140 0x7f9c340c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c39298700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340722d0 con 0x7f9c3414b340 2023-03-03T11:36:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c39298700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c341e5a60 con 0x7f9c340c2790 2023-03-03T11:36:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c39298700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340c5740 con 0x7f9c34151140 2023-03-03T11:36:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c327fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 0x7f9c341dd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c327fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 0x7f9c341dd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37040/0 (socket says 172.21.15.7:37040) 2023-03-03T11:36:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c327fc700 1 -- 172.21.15.7:0/367824192 learned_addr learned my addr 172.21.15.7:0/367824192 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:36:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c32ffd700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c340c2790 0x7f9c341dc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:44.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c337fe700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34151140 0x7f9c340c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c327fc700 1 -- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34151140 msgr2=0x7f9c340c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c327fc700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34151140 0x7f9c340c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c327fc700 1 -- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c340c2790 msgr2=0x7f9c341dc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c327fc700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c340c2790 0x7f9c341dc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c327fc700 1 -- 172.21.15.7:0/367824192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2800b7f0 con 0x7f9c3414b340 2023-03-03T11:36:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.678+0000 7f9c337fe700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34151140 0x7f9c340c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:36:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.679+0000 7f9c327fc700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 0x7f9c341dd010 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400c7f0 tx=0x7f9c2400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:44.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.679+0000 7f9c1b7fe700 1 -- 172.21.15.7:0/367824192 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24011800 con 0x7f9c3414b340 2023-03-03T11:36:44.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.679+0000 7f9c1b7fe700 1 -- 172.21.15.7:0/367824192 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c24011960 con 0x7f9c3414b340 2023-03-03T11:36:44.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.679+0000 7f9c1b7fe700 1 -- 172.21.15.7:0/367824192 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24011cf0 con 0x7f9c3414b340 2023-03-03T11:36:44.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.679+0000 7f9c39298700 1 -- 172.21.15.7:0/367824192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c3406c2d0 con 0x7f9c3414b340 2023-03-03T11:36:44.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.679+0000 7f9c39298700 1 -- 172.21.15.7:0/367824192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3406bff0 con 0x7f9c3414b340 2023-03-03T11:36:44.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.680+0000 7f9c39298700 1 -- 172.21.15.7:0/367824192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3406c5b0 con 0x7f9c3414b340 2023-03-03T11:36:44.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.680+0000 7f9c1b7fe700 1 -- 172.21.15.7:0/367824192 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9c24021070 con 0x7f9c3414b340 2023-03-03T11:36:44.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.681+0000 7f9c1b7fe700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c200682f0 0x7f9c2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:36:44.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.681+0000 7f9c1b7fe700 1 -- 172.21.15.7:0/367824192 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9c24085f50 con 0x7f9c3414b340 2023-03-03T11:36:44.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.681+0000 7f9c32ffd700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c200682f0 0x7f9c2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:36:44.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.681+0000 7f9c32ffd700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c200682f0 0x7f9c2006a7a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800b340 tx=0x7f9c2801d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:36:44.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.685+0000 7f9c1b7fe700 1 -- 172.21.15.7:0/367824192 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9c24052270 con 0x7f9c3414b340 2023-03-03T11:36:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.922+0000 7f9c39298700 1 -- 172.21.15.7:0/367824192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c34074bb0 con 0x7f9c3414b340 2023-03-03T11:36:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.923+0000 7f9c1b7fe700 1 -- 172.21.15.7:0/367824192 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9c24055890 con 0x7f9c3414b340 2023-03-03T11:36:44.925 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:36:44.925 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:36:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 -- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c200682f0 msgr2=0x7f9c2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c200682f0 0x7f9c2006a7a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800b340 tx=0x7f9c2801d040 comp rx=0 tx=0).stop 2023-03-03T11:36:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 -- 172.21.15.7:0/367824192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 msgr2=0x7f9c341dd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:36:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 0x7f9c341dd010 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400c7f0 tx=0x7f9c2400cb00 comp rx=0 tx=0).stop 2023-03-03T11:36:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 -- 172.21.15.7:0/367824192 shutdown_connections 2023-03-03T11:36:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c34151140 0x7f9c340c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c340c2790 0x7f9c341dc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c200682f0 0x7f9c2006a7a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 --2- 172.21.15.7:0/367824192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c3414b340 0x7f9c341dd010 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:36:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 -- 172.21.15.7:0/367824192 >> 172.21.15.7:0/367824192 conn(0x7f9c340c62e0 msgr2=0x7f9c34152bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:36:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 -- 172.21.15.7:0/367824192 shutdown_connections 2023-03-03T11:36:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:36:44.925+0000 7f9c39298700 1 -- 172.21.15.7:0/367824192 wait complete. 2023-03-03T11:36:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:36:44.938 INFO:tasks.mds_thrash.fs.[b]:waiting for 98 secs before thrashing 2023-03-03T11:36:46.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:36:46.004 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:00.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:37:00.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.868+0000 7faa943a7700 1 Processor -- start 2023-03-03T11:37:00.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.868+0000 7faa943a7700 1 -- start start 2023-03-03T11:37:00.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.869+0000 7faa943a7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faa8c14b0d0 0x7faa8c071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:00.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.869+0000 7faa943a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c0aad20 0x7faa8c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:00.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.869+0000 7faa943a7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0a9c20 con 0x7faa8c071d40 2023-03-03T11:37:00.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.869+0000 7faa943a7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c154f10 con 0x7faa8c14b0d0 2023-03-03T11:37:00.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.869+0000 7faa943a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0ac3c0 con 0x7faa8c0aad20 2023-03-03T11:37:00.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.869+0000 7faa92944700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faa8c071d40 0x7faa8c158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59278/0 (socket says 172.21.15.7:59278) 2023-03-03T11:37:00.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.869+0000 7faa92944700 1 -- 172.21.15.7:0/3273521137 learned_addr learned my addr 172.21.15.7:0/3273521137 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:37:00.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa91942700 1 --2- 172.21.15.7:0/3273521137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c0aad20 0x7faa8c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:00.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa92143700 1 --2- 172.21.15.7:0/3273521137 >> v2:172.21.15.143:3300/0 conn(0x7faa8c14b0d0 0x7faa8c071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:00.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa91141700 1 -- 172.21.15.7:0/3273521137 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317992685 0 0) 0x7faa8c0a9c20 con 0x7faa8c071d40 2023-03-03T11:37:00.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa91141700 1 -- 172.21.15.7:0/3273521137 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa780035b0 con 0x7faa8c071d40 2023-03-03T11:37:00.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa91141700 1 -- 172.21.15.7:0/3273521137 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2219962763 0 0) 0x7faa780035b0 con 0x7faa8c071d40 2023-03-03T11:37:00.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa91942700 1 -- 172.21.15.7:0/3273521137 >> v2:172.21.15.143:3300/0 conn(0x7faa8c14b0d0 msgr2=0x7faa8c071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:37:00.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa91942700 1 --2- 172.21.15.7:0/3273521137 >> v2:172.21.15.143:3300/0 conn(0x7faa8c14b0d0 0x7faa8c071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:00.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa91942700 1 -- 172.21.15.7:0/3273521137 >> v1:172.21.15.7:6789/0 conn(0x7faa8c071d40 legacy=0x7faa8c158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:00.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa91942700 1 -- 172.21.15.7:0/3273521137 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c06fc70 con 0x7faa8c0aad20 2023-03-03T11:37:00.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa92143700 1 --2- 172.21.15.7:0/3273521137 >> v2:172.21.15.143:3300/0 conn(0x7faa8c14b0d0 0x7faa8c071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:37:00.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.870+0000 7faa91942700 1 --2- 172.21.15.7:0/3273521137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c0aad20 0x7faa8c157df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faa80009840 tx=0x7faa80009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e7926242d8bd017 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:37:00.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa91141700 1 -- 172.21.15.7:0/3273521137 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840019f0 con 0x7faa8c0aad20 2023-03-03T11:37:00.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa91141700 1 -- 172.21.15.7:0/3273521137 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa80004030 con 0x7faa8c0aad20 2023-03-03T11:37:00.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa91141700 1 -- 172.21.15.7:0/3273521137 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa800043a0 con 0x7faa8c0aad20 2023-03-03T11:37:00.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa943a7700 1 -- 172.21.15.7:0/3273521137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c0aad20 msgr2=0x7faa8c157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:00.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa943a7700 1 --2- 172.21.15.7:0/3273521137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c0aad20 0x7faa8c157df0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faa80009840 tx=0x7faa80009b50 comp rx=0 tx=0).stop 2023-03-03T11:37:00.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa943a7700 1 -- 172.21.15.7:0/3273521137 shutdown_connections 2023-03-03T11:37:00.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa943a7700 1 --2- 172.21.15.7:0/3273521137 >> v2:172.21.15.143:3300/0 conn(0x7faa8c14b0d0 0x7faa8c071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:00.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa943a7700 1 --2- 172.21.15.7:0/3273521137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c0aad20 0x7faa8c157df0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:00.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa943a7700 1 -- 172.21.15.7:0/3273521137 >> 172.21.15.7:0/3273521137 conn(0x7faa8c0c8900 msgr2=0x7faa8c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:37:00.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa943a7700 1 -- 172.21.15.7:0/3273521137 shutdown_connections 2023-03-03T11:37:00.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.871+0000 7faa943a7700 1 -- 172.21.15.7:0/3273521137 wait complete. 2023-03-03T11:37:00.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.873+0000 7faa943a7700 1 Processor -- start 2023-03-03T11:37:00.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.873+0000 7faa943a7700 1 -- start start 2023-03-03T11:37:00.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa943a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa8c071d40 0x7faa8c1e9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:00.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa943a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa8c0aad20 0x7faa8c1ed6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:00.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa943a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c14b0d0 0x7faa8c1e6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:00.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa943a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1ecf80 con 0x7faa8c071d40 2023-03-03T11:37:00.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa943a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1ed260 con 0x7faa8c0aad20 2023-03-03T11:37:00.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa943a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c153c60 con 0x7faa8c14b0d0 2023-03-03T11:37:00.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92143700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa8c071d40 0x7faa8c1e9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:00.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92143700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa8c071d40 0x7faa8c1e9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37068/0 (socket says 172.21.15.7:37068) 2023-03-03T11:37:00.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92143700 1 -- 172.21.15.7:0/4131599329 learned_addr learned my addr 172.21.15.7:0/4131599329 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:37:00.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92944700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c14b0d0 0x7faa8c1e6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:00.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa91942700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa8c0aad20 0x7faa8c1ed6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:00.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92143700 1 -- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c14b0d0 msgr2=0x7faa8c1e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:00.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92143700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c14b0d0 0x7faa8c1e6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:00.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92143700 1 -- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa8c0aad20 msgr2=0x7faa8c1ed6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:00.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92143700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa8c0aad20 0x7faa8c1ed6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:00.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92143700 1 -- 172.21.15.7:0/4131599329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa880094a0 con 0x7faa8c071d40 2023-03-03T11:37:00.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92944700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c14b0d0 0x7faa8c1e6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:37:00.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.874+0000 7faa92143700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa8c071d40 0x7faa8c1e9a10 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7faa8800ec10 tx=0x7faa88009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:37:00.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.875+0000 7faa7effd700 1 -- 172.21.15.7:0/4131599329 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8800c940 con 0x7faa8c071d40 2023-03-03T11:37:00.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.875+0000 7faa7effd700 1 -- 172.21.15.7:0/4131599329 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8800caa0 con 0x7faa8c071d40 2023-03-03T11:37:00.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.875+0000 7faa943a7700 1 -- 172.21.15.7:0/4131599329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa800094a0 con 0x7faa8c071d40 2023-03-03T11:37:00.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.875+0000 7faa7effd700 1 -- 172.21.15.7:0/4131599329 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8800ce10 con 0x7faa8c071d40 2023-03-03T11:37:00.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.875+0000 7faa943a7700 1 -- 172.21.15.7:0/4131599329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c06f6b0 con 0x7faa8c071d40 2023-03-03T11:37:00.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.875+0000 7faa943a7700 1 -- 172.21.15.7:0/4131599329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c155160 con 0x7faa8c071d40 2023-03-03T11:37:00.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.877+0000 7faa7effd700 1 -- 172.21.15.7:0/4131599329 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faa88019030 con 0x7faa8c071d40 2023-03-03T11:37:00.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.878+0000 7faa7effd700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa700682f0 0x7faa7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:37:00.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.878+0000 7faa7effd700 1 -- 172.21.15.7:0/4131599329 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faa88086b20 con 0x7faa8c071d40 2023-03-03T11:37:00.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.878+0000 7faa91942700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa700682f0 0x7faa7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:37:00.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.878+0000 7faa91942700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa700682f0 0x7faa7006a7a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7faa800090b0 tx=0x7faa80013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:37:00.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:00.881+0000 7faa7effd700 1 -- 172.21.15.7:0/4131599329 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faa88052d70 con 0x7faa8c071d40 2023-03-03T11:37:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.119+0000 7faa943a7700 1 -- 172.21.15.7:0/4131599329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa8c06f990 con 0x7faa8c071d40 2023-03-03T11:37:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.119+0000 7faa7effd700 1 -- 172.21.15.7:0/4131599329 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faa88056390 con 0x7faa8c071d40 2023-03-03T11:37:01.122 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:37:01.122 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:37:01.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 -- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa700682f0 msgr2=0x7faa7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:01.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa700682f0 0x7faa7006a7a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7faa800090b0 tx=0x7faa80013040 comp rx=0 tx=0).stop 2023-03-03T11:37:01.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 -- 172.21.15.7:0/4131599329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa8c071d40 msgr2=0x7faa8c1e9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:37:01.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa8c071d40 0x7faa8c1e9a10 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7faa8800ec10 tx=0x7faa88009ec0 comp rx=0 tx=0).stop 2023-03-03T11:37:01.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 -- 172.21.15.7:0/4131599329 shutdown_connections 2023-03-03T11:37:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa8c14b0d0 0x7faa8c1e6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa700682f0 0x7faa7006a7a0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa8c071d40 0x7faa8c1e9a10 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 --2- 172.21.15.7:0/4131599329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa8c0aad20 0x7faa8c1ed6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:37:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 -- 172.21.15.7:0/4131599329 >> 172.21.15.7:0/4131599329 conn(0x7faa8c0c8900 msgr2=0x7faa8c157df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:37:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 -- 172.21.15.7:0/4131599329 shutdown_connections 2023-03-03T11:37:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:37:01.121+0000 7faa943a7700 1 -- 172.21.15.7:0/4131599329 wait complete. 2023-03-03T11:37:01.125 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:37:01.135 INFO:tasks.mds_thrash.fs.[a]:waiting for 112 secs before thrashing 2023-03-03T11:37:16.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:16.184 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:46.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:37:46.340 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:16.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:16.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:22.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:38:23.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.076+0000 7ff7e7ebd700 1 Processor -- start 2023-03-03T11:38:23.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.076+0000 7ff7e7ebd700 1 -- start start 2023-03-03T11:38:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e7ebd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff7e00c2930 0x7ff7e0157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e7ebd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e006c480 0x7ff7e0157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e7ebd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e006d430 con 0x7ff7e0151d40 2023-03-03T11:38:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e7ebd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7e006c2a0 con 0x7ff7e00c2930 2023-03-03T11:38:23.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e7ebd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e006cb00 con 0x7ff7e006c480 2023-03-03T11:38:23.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e5458700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff7e00c2930 0x7ff7e0157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:23.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e645a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e006c480 0x7ff7e0157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:23.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e5458700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff7e00c2930 0x7ff7e0157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:32900/0 (socket says 172.21.15.7:32900) 2023-03-03T11:38:23.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e5458700 1 -- 172.21.15.7:0/1006714772 learned_addr learned my addr 172.21.15.7:0/1006714772 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:38:23.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e645a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e006c480 0x7ff7e0157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:45200/0 (socket says 172.21.15.7:45200) 2023-03-03T11:38:23.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.077+0000 7ff7e5c59700 1 --1- 172.21.15.7:0/1006714772 >> v1:172.21.15.143:6789/0 conn(0x7ff7e0151d40 0x7ff7e00736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:34624/0 (socket says 172.21.15.7:34624) 2023-03-03T11:38:23.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.078+0000 7ff7e5458700 1 -- 172.21.15.7:0/1006714772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e006c480 msgr2=0x7ff7e0157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:38:23.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.078+0000 7ff7e5458700 1 --2- 172.21.15.7:0/1006714772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e006c480 0x7ff7e0157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:23.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.078+0000 7ff7e5458700 1 -- 172.21.15.7:0/1006714772 >> v1:172.21.15.143:6789/0 conn(0x7ff7e0151d40 legacy=0x7ff7e00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:23.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.078+0000 7ff7e5458700 1 -- 172.21.15.7:0/1006714772 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e006cde0 con 0x7ff7e00c2930 2023-03-03T11:38:23.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.078+0000 7ff7e645a700 1 --2- 172.21.15.7:0/1006714772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e006c480 0x7ff7e0157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:38:23.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.078+0000 7ff7e5458700 1 --2- 172.21.15.7:0/1006714772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e00c2930 0x7ff7e0157a90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0098f0 tx=0x7ff7dc009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe3cdf5a23e7b695 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:38:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.078+0000 7ff7e4c57700 1 -- 172.21.15.7:0/1006714772 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc011830 con 0x7ff7e00c2930 2023-03-03T11:38:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.078+0000 7ff7e4c57700 1 -- 172.21.15.7:0/1006714772 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7dc011990 con 0x7ff7e00c2930 2023-03-03T11:38:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.078+0000 7ff7e4c57700 1 -- 172.21.15.7:0/1006714772 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc011d20 con 0x7ff7e00c2930 2023-03-03T11:38:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.079+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/1006714772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e00c2930 msgr2=0x7ff7e0157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.079+0000 7ff7e7ebd700 1 --2- 172.21.15.7:0/1006714772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e00c2930 0x7ff7e0157a90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0098f0 tx=0x7ff7dc009c00 comp rx=0 tx=0).stop 2023-03-03T11:38:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.079+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/1006714772 shutdown_connections 2023-03-03T11:38:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.079+0000 7ff7e7ebd700 1 --2- 172.21.15.7:0/1006714772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e006c480 0x7ff7e0157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:23.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.079+0000 7ff7e7ebd700 1 --2- 172.21.15.7:0/1006714772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e00c2930 0x7ff7e0157a90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:23.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.079+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/1006714772 >> 172.21.15.7:0/1006714772 conn(0x7ff7e00a94d0 msgr2=0x7ff7e00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:38:23.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.079+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/1006714772 shutdown_connections 2023-03-03T11:38:23.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.079+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/1006714772 wait complete. 2023-03-03T11:38:23.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.080+0000 7ff7e7ebd700 1 Processor -- start 2023-03-03T11:38:23.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.080+0000 7ff7e7ebd700 1 -- start start 2023-03-03T11:38:23.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e7ebd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7e006c480 0x7ff7e0148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:23.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e7ebd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e00c2930 0x7ff7e01485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:23.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e7ebd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e0151d40 0x7ff7e01562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:23.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e7ebd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e0148ae0 con 0x7ff7e006c480 2023-03-03T11:38:23.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e7ebd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e00c1440 con 0x7ff7e0151d40 2023-03-03T11:38:23.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e7ebd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e01ee0a0 con 0x7ff7e00c2930 2023-03-03T11:38:23.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e5c59700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7e006c480 0x7ff7e0148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:23.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e5c59700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7e006c480 0x7ff7e0148060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37130/0 (socket says 172.21.15.7:37130) 2023-03-03T11:38:23.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e5c59700 1 -- 172.21.15.7:0/3983821650 learned_addr learned my addr 172.21.15.7:0/3983821650 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:38:23.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e5458700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e00c2930 0x7ff7e01485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:23.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e645a700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e0151d40 0x7ff7e01562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:23.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e5c59700 1 -- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e00c2930 msgr2=0x7ff7e01485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:23.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e5c59700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e00c2930 0x7ff7e01485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:23.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e5c59700 1 -- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e0151d40 msgr2=0x7ff7e01562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:23.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e5c59700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e0151d40 0x7ff7e01562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:23.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.081+0000 7ff7e5c59700 1 -- 172.21.15.7:0/3983821650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d8009a70 con 0x7ff7e006c480 2023-03-03T11:38:23.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.082+0000 7ff7e5c59700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7e006c480 0x7ff7e0148060 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ff7d000c550 tx=0x7ff7d000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:38:23.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.082+0000 7ff7d67fc700 1 -- 172.21.15.7:0/3983821650 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d0005cb0 con 0x7ff7e006c480 2023-03-03T11:38:23.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.082+0000 7ff7d67fc700 1 -- 172.21.15.7:0/3983821650 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d0005e10 con 0x7ff7e006c480 2023-03-03T11:38:23.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.082+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/3983821650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7dc009550 con 0x7ff7e006c480 2023-03-03T11:38:23.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.082+0000 7ff7d67fc700 1 -- 172.21.15.7:0/3983821650 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d00072d0 con 0x7ff7e006c480 2023-03-03T11:38:23.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.082+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/3983821650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e0073e00 con 0x7ff7e006c480 2023-03-03T11:38:23.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.083+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/3983821650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e006d0c0 con 0x7ff7e006c480 2023-03-03T11:38:23.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.083+0000 7ff7d67fc700 1 -- 172.21.15.7:0/3983821650 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff7d0019030 con 0x7ff7e006c480 2023-03-03T11:38:23.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.084+0000 7ff7d67fc700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7c40682f0 0x7ff7c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:23.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.084+0000 7ff7d67fc700 1 -- 172.21.15.7:0/3983821650 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff7d0085270 con 0x7ff7e006c480 2023-03-03T11:38:23.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.084+0000 7ff7e5458700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7c40682f0 0x7ff7c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:23.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.084+0000 7ff7e5458700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7c40682f0 0x7ff7c406a7a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009c40 tx=0x7ff7dc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:38:23.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.088+0000 7ff7d67fc700 1 -- 172.21.15.7:0/3983821650 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff7d004b5e0 con 0x7ff7e006c480 2023-03-03T11:38:23.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.328+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/3983821650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7e006bea0 con 0x7ff7e006c480 2023-03-03T11:38:23.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.328+0000 7ff7d67fc700 1 -- 172.21.15.7:0/3983821650 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff7d0017090 con 0x7ff7e006c480 2023-03-03T11:38:23.330 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:38:23.330 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:38:23.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7c40682f0 msgr2=0x7ff7c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:23.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7c40682f0 0x7ff7c406a7a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009c40 tx=0x7ff7dc008030 comp rx=0 tx=0).stop 2023-03-03T11:38:23.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/3983821650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7e006c480 msgr2=0x7ff7e0148060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:23.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7e006c480 0x7ff7e0148060 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ff7d000c550 tx=0x7ff7d000c860 comp rx=0 tx=0).stop 2023-03-03T11:38:23.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/3983821650 shutdown_connections 2023-03-03T11:38:23.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7e0151d40 0x7ff7e01562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:23.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7c40682f0 0x7ff7c406a7a0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:23.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7e006c480 0x7ff7e0148060 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:23.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 --2- 172.21.15.7:0/3983821650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7e00c2930 0x7ff7e01485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:23.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/3983821650 >> 172.21.15.7:0/3983821650 conn(0x7ff7e00a94d0 msgr2=0x7ff7e00c45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:38:23.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/3983821650 shutdown_connections 2023-03-03T11:38:23.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:23.330+0000 7ff7e7ebd700 1 -- 172.21.15.7:0/3983821650 wait complete. 2023-03-03T11:38:23.334 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:38:23.344 INFO:tasks.mds_thrash.fs.[b]:waiting for 96 secs before thrashing 2023-03-03T11:38:46.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:46.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:38:53.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:38:53.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.273+0000 7fe2f7bb0700 1 Processor -- start 2023-03-03T11:38:53.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.273+0000 7fe2f7bb0700 1 -- start start 2023-03-03T11:38:53.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f7bb0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f014f9d0 0x7fe2f0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:53.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f7bb0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2f00abe20 0x7fe2f01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:53.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f7bb0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f00c2f30 con 0x7fe2f014fda0 2023-03-03T11:38:53.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f7bb0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f00c31d0 con 0x7fe2f00abe20 2023-03-03T11:38:53.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f7bb0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f01524a0 con 0x7fe2f014f9d0 2023-03-03T11:38:53.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f614d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2f00abe20 0x7fe2f01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:53.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f614d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2f00abe20 0x7fe2f01529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37156/0 (socket says 172.21.15.7:37156) 2023-03-03T11:38:53.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f614d700 1 -- 172.21.15.7:0/4004763596 learned_addr learned my addr 172.21.15.7:0/4004763596 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:38:53.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f594c700 1 --2- 172.21.15.7:0/4004763596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f014f9d0 0x7fe2f0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:53.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f614d700 1 -- 172.21.15.7:0/4004763596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f014f9d0 msgr2=0x7fe2f0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:38:53.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f614d700 1 --2- 172.21.15.7:0/4004763596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f014f9d0 0x7fe2f0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:53.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f614d700 1 -- 172.21.15.7:0/4004763596 >> v1:172.21.15.143:6789/0 conn(0x7fe2f014fda0 legacy=0x7fe2f0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:53.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f614d700 1 -- 172.21.15.7:0/4004763596 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f00c34b0 con 0x7fe2f00abe20 2023-03-03T11:38:53.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.274+0000 7fe2f614d700 1 --2- 172.21.15.7:0/4004763596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 0x7fe2f01529c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009e50 tx=0x7fe2e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ad4f363da328c3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:38:53.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f494a700 1 -- 172.21.15.7:0/4004763596 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800e070 con 0x7fe2f00abe20 2023-03-03T11:38:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f494a700 1 -- 172.21.15.7:0/4004763596 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e800b910 con 0x7fe2f00abe20 2023-03-03T11:38:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f494a700 1 -- 172.21.15.7:0/4004763596 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e800bcc0 con 0x7fe2f00abe20 2023-03-03T11:38:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/4004763596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 msgr2=0x7fe2f01529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f7bb0700 1 --2- 172.21.15.7:0/4004763596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 0x7fe2f01529c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009e50 tx=0x7fe2e800b330 comp rx=0 tx=0).stop 2023-03-03T11:38:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/4004763596 shutdown_connections 2023-03-03T11:38:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f7bb0700 1 --2- 172.21.15.7:0/4004763596 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 0x7fe2f01529c0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f7bb0700 1 --2- 172.21.15.7:0/4004763596 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f014f9d0 0x7fe2f0156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/4004763596 >> 172.21.15.7:0/4004763596 conn(0x7fe2f00c8900 msgr2=0x7fe2f00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:38:53.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/4004763596 shutdown_connections 2023-03-03T11:38:53.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.275+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/4004763596 wait complete. 2023-03-03T11:38:53.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.277+0000 7fe2f7bb0700 1 Processor -- start 2023-03-03T11:38:53.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.277+0000 7fe2f7bb0700 1 -- start start 2023-03-03T11:38:53.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f7bb0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 0x7fe2f00aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:53.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f7bb0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f014f9d0 0x7fe2f006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:53.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f7bb0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f014fda0 0x7fe2f014c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:53.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f7bb0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f006f380 con 0x7fe2f00abe20 2023-03-03T11:38:53.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f7bb0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f006fb90 con 0x7fe2f014f9d0 2023-03-03T11:38:53.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f7bb0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f01ebbc0 con 0x7fe2f014fda0 2023-03-03T11:38:53.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f594c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 0x7fe2f00aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:53.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f594c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 0x7fe2f00aa2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37162/0 (socket says 172.21.15.7:37162) 2023-03-03T11:38:53.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f594c700 1 -- 172.21.15.7:0/2722164551 learned_addr learned my addr 172.21.15.7:0/2722164551 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:38:53.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f514b700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f014f9d0 0x7fe2f006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:53.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f614d700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f014fda0 0x7fe2f014c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:53.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f594c700 1 -- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f014fda0 msgr2=0x7fe2f014c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:53.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f594c700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f014fda0 0x7fe2f014c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:53.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f594c700 1 -- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f014f9d0 msgr2=0x7fe2f006e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:53.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f594c700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f014f9d0 0x7fe2f006e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:53.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f594c700 1 -- 172.21.15.7:0/2722164551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e8009b30 con 0x7fe2f00abe20 2023-03-03T11:38:53.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.278+0000 7fe2f594c700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 0x7fe2f00aa2b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00e9e0 tx=0x7fe2ec00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:38:53.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.279+0000 7fe2e67fc700 1 -- 172.21.15.7:0/2722164551 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec009710 con 0x7fe2f00abe20 2023-03-03T11:38:53.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.279+0000 7fe2e67fc700 1 -- 172.21.15.7:0/2722164551 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ec00c850 con 0x7fe2f00abe20 2023-03-03T11:38:53.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.279+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/2722164551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f00ac350 con 0x7fe2f00abe20 2023-03-03T11:38:53.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.279+0000 7fe2e67fc700 1 -- 172.21.15.7:0/2722164551 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec00cbc0 con 0x7fe2f00abe20 2023-03-03T11:38:53.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.279+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/2722164551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f00ab630 con 0x7fe2f00abe20 2023-03-03T11:38:53.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.280+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/2722164551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f0071fa0 con 0x7fe2f00abe20 2023-03-03T11:38:53.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.280+0000 7fe2e67fc700 1 -- 172.21.15.7:0/2722164551 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe2ec023070 con 0x7fe2f00abe20 2023-03-03T11:38:53.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.280+0000 7fe2e67fc700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2d4068320 0x7fe2d406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:38:53.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.281+0000 7fe2e67fc700 1 -- 172.21.15.7:0/2722164551 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe2ec0864d0 con 0x7fe2f00abe20 2023-03-03T11:38:53.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.281+0000 7fe2f514b700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2d4068320 0x7fe2d406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:38:53.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.281+0000 7fe2f514b700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2d4068320 0x7fe2d406a7d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00c42c0 tx=0x7fe2e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:38:53.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.285+0000 7fe2e67fc700 1 -- 172.21.15.7:0/2722164551 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2ec0527f0 con 0x7fe2f00abe20 2023-03-03T11:38:53.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.526+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/2722164551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2f0074bb0 con 0x7fe2f00abe20 2023-03-03T11:38:53.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.526+0000 7fe2e67fc700 1 -- 172.21.15.7:0/2722164551 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe2ec055e10 con 0x7fe2f00abe20 2023-03-03T11:38:53.528 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:38:53.528 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:38:53.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2d4068320 msgr2=0x7fe2d406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:53.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2d4068320 0x7fe2d406a7d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00c42c0 tx=0x7fe2e0009040 comp rx=0 tx=0).stop 2023-03-03T11:38:53.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/2722164551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 msgr2=0x7fe2f00aa2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:38:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 0x7fe2f00aa2b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00e9e0 tx=0x7fe2ec00eda0 comp rx=0 tx=0).stop 2023-03-03T11:38:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/2722164551 shutdown_connections 2023-03-03T11:38:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f014fda0 0x7fe2f014c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2d4068320 0x7fe2d406a7d0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2f00abe20 0x7fe2f00aa2b0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 --2- 172.21.15.7:0/2722164551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f014f9d0 0x7fe2f006e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:38:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/2722164551 >> 172.21.15.7:0/2722164551 conn(0x7fe2f00c8900 msgr2=0x7fe2f0151a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:38:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/2722164551 shutdown_connections 2023-03-03T11:38:53.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:38:53.528+0000 7fe2f7bb0700 1 -- 172.21.15.7:0/2722164551 wait complete. 2023-03-03T11:38:53.532 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:38:53.542 INFO:tasks.mds_thrash.fs.[a]:waiting for 23 secs before thrashing 2023-03-03T11:39:16.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:39:16.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.681+0000 7f4f1e56a700 1 Processor -- start 2023-03-03T11:39:16.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.681+0000 7f4f1e56a700 1 -- start start 2023-03-03T11:39:16.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.682+0000 7f4f1e56a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4f1814e900 0x7f4f1815ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:16.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.682+0000 7f4f1e56a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f181510d0 con 0x7f4f18071e80 2023-03-03T11:39:16.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.682+0000 7f4f1e56a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f180c3740 con 0x7f4f18072250 2023-03-03T11:39:16.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.682+0000 7f4f1e56a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f18070670 con 0x7f4f1814e900 2023-03-03T11:39:16.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.682+0000 7f4f17fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4f18071e80 0x7f4f18151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59396/0 (socket says 172.21.15.7:59396) 2023-03-03T11:39:16.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.682+0000 7f4f17fff700 1 -- 172.21.15.7:0/1703417328 learned_addr learned my addr 172.21.15.7:0/1703417328 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:39:16.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584272511 0 0) 0x7f4f181510d0 con 0x7f4f18071e80 2023-03-03T11:39:16.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f000035b0 con 0x7f4f18071e80 2023-03-03T11:39:16.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f1cb07700 1 --2- 172.21.15.7:0/1703417328 >> v2:172.21.15.143:3300/0 conn(0x7f4f1814e900 0x7f4f1815ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:16.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3056430578 0 0) 0x7f4f000035b0 con 0x7f4f18071e80 2023-03-03T11:39:16.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 >> v2:172.21.15.143:3300/0 conn(0x7f4f1814e900 msgr2=0x7f4f1815ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:39:16.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 --2- 172.21.15.7:0/1703417328 >> v2:172.21.15.143:3300/0 conn(0x7f4f1814e900 0x7f4f1815ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 >> v1:172.21.15.143:6789/0 conn(0x7f4f18072250 legacy=0x7f4f1814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:16.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f180c39e0 con 0x7f4f18071e80 2023-03-03T11:39:16.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f4f08001670 con 0x7f4f18071e80 2023-03-03T11:39:16.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f1cb07700 1 --2- 172.21.15.7:0/1703417328 >> v2:172.21.15.143:3300/0 conn(0x7f4f1814e900 0x7f4f1815ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:39:16.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 >> v1:172.21.15.7:6789/0 conn(0x7f4f18071e80 legacy=0x7f4f18151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:16.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f00003a70 0x7f4f00003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:16.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f00004850 0x7f4f00004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:16.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f0000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:16.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18070670 con 0x7f4f00005300 2023-03-03T11:39:16.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0000c230 con 0x7f4f00003a70 2023-03-03T11:39:16.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0000c3f0 con 0x7f4f00004850 2023-03-03T11:39:16.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f1cb07700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f0000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:16.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f17fff700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f00003a70 0x7f4f00003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:16.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f177fe700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f00004850 0x7f4f00004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:16.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f1cb07700 1 -- 172.21.15.7:0/1703417328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f00004850 msgr2=0x7f4f00004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:39:16.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f1cb07700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f00004850 0x7f4f00004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f1cb07700 1 -- 172.21.15.7:0/1703417328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f00003a70 msgr2=0x7f4f00003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:39:16.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f1cb07700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f00003a70 0x7f4f00003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.683+0000 7f4f1cb07700 1 -- 172.21.15.7:0/1703417328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f000035b0 con 0x7f4f00005300 2023-03-03T11:39:16.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f1cb07700 1 -- 172.21.15.7:0/1703417328 reap_dead start 2023-03-03T11:39:16.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f1cb07700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f0000ba10 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f4f100073b0 tx=0x7f4f00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40cc6a4155b23e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:39:16.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f10009490 con 0x7f4f00005300 2023-03-03T11:39:16.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f10014400 con 0x7f4f00005300 2023-03-03T11:39:16.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f16ffd700 1 -- 172.21.15.7:0/1703417328 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f10014790 con 0x7f4f00005300 2023-03-03T11:39:16.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1703417328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 msgr2=0x7f4f0000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:16.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f1e56a700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f0000ba10 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f4f100073b0 tx=0x7f4f00005880 comp rx=0 tx=0).stop 2023-03-03T11:39:16.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1703417328 shutdown_connections 2023-03-03T11:39:16.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f1e56a700 1 --2- 172.21.15.7:0/1703417328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f0000ba10 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1703417328 >> 172.21.15.7:0/1703417328 conn(0x7f4f180a94d0 msgr2=0x7f4f180a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:39:16.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1703417328 shutdown_connections 2023-03-03T11:39:16.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.684+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1703417328 wait complete. 2023-03-03T11:39:16.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.686+0000 7f4f1e56a700 1 Processor -- start 2023-03-03T11:39:16.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.686+0000 7f4f1e56a700 1 -- start start 2023-03-03T11:39:16.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f1e56a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f1814a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:16.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f1e56a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f18148700 0x7f4f180c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:16.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f1e56a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f181474c0 0x7f4f18141af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:16.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f1e56a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180c7150 con 0x7f4f00005300 2023-03-03T11:39:16.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f1e56a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f180bab80 con 0x7f4f18148700 2023-03-03T11:39:16.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f1e56a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1814a820 con 0x7f4f181474c0 2023-03-03T11:39:16.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f17fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f1814a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:16.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f17fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f1814a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37196/0 (socket says 172.21.15.7:37196) 2023-03-03T11:39:16.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f17fff700 1 -- 172.21.15.7:0/1383804218 learned_addr learned my addr 172.21.15.7:0/1383804218 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:39:16.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f1cb07700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f181474c0 0x7f4f18141af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:16.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f177fe700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f18148700 0x7f4f180c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:16.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f17fff700 1 -- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f181474c0 msgr2=0x7f4f18141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:16.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f17fff700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f181474c0 0x7f4f18141af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f17fff700 1 -- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f18148700 msgr2=0x7f4f180c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:16.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f17fff700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f18148700 0x7f4f180c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f17fff700 1 -- 172.21.15.7:0/1383804218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1806c9a0 con 0x7f4f00005300 2023-03-03T11:39:16.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f177fe700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f18148700 0x7f4f180c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:39:16.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f17fff700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f1814a2e0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f4f08007290 tx=0x7f4f080075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:39:16.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f14ff9700 1 -- 172.21.15.7:0/1383804218 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08011580 con 0x7f4f00005300 2023-03-03T11:39:16.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f14ff9700 1 -- 172.21.15.7:0/1383804218 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f08011700 con 0x7f4f00005300 2023-03-03T11:39:16.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1383804218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f18151300 con 0x7f4f00005300 2023-03-03T11:39:16.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.687+0000 7f4f14ff9700 1 -- 172.21.15.7:0/1383804218 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08011ab0 con 0x7f4f00005300 2023-03-03T11:39:16.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.688+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1383804218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1806cb30 con 0x7f4f00005300 2023-03-03T11:39:16.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.689+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1383804218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f180c6bc0 con 0x7f4f00005300 2023-03-03T11:39:16.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.689+0000 7f4f14ff9700 1 -- 172.21.15.7:0/1383804218 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4f08011c10 con 0x7f4f00005300 2023-03-03T11:39:16.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.689+0000 7f4f14ff9700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f04068320 0x7f4f0406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:16.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.689+0000 7f4f14ff9700 1 -- 172.21.15.7:0/1383804218 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4f080862e0 con 0x7f4f00005300 2023-03-03T11:39:16.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.690+0000 7f4f177fe700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f04068320 0x7f4f0406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:16.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.690+0000 7f4f177fe700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f04068320 0x7f4f0406a7d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4f180bada0 tx=0x7f4f0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:39:16.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:16.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.694+0000 7f4f14ff9700 1 -- 172.21.15.7:0/1383804218 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4f080525f0 con 0x7f4f00005300 2023-03-03T11:39:16.701 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:16.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.933+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1383804218 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f18074bb0 con 0x7f4f00005300 2023-03-03T11:39:16.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.933+0000 7f4f14ff9700 1 -- 172.21.15.7:0/1383804218 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4f08055c10 con 0x7f4f00005300 2023-03-03T11:39:16.935 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:39:16.935 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:39:16.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f04068320 msgr2=0x7f4f0406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:16.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f04068320 0x7f4f0406a7d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4f180bada0 tx=0x7f4f0c009040 comp rx=0 tx=0).stop 2023-03-03T11:39:16.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1383804218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 msgr2=0x7f4f1814a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:16.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f1814a2e0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f4f08007290 tx=0x7f4f080075a0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1383804218 shutdown_connections 2023-03-03T11:39:16.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f181474c0 0x7f4f18141af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f04068320 0x7f4f0406a7d0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f00005300 0x7f4f1814a2e0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 --2- 172.21.15.7:0/1383804218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f18148700 0x7f4f180c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:16.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1383804218 >> 172.21.15.7:0/1383804218 conn(0x7f4f180a94d0 msgr2=0x7f4f18151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:39:16.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1383804218 shutdown_connections 2023-03-03T11:39:16.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:16.935+0000 7f4f1e56a700 1 -- 172.21.15.7:0/1383804218 wait complete. 2023-03-03T11:39:16.938 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:39:16.948 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-03-03T11:39:42.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:39:43.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.086+0000 7f59ebfb4700 1 Processor -- start 2023-03-03T11:39:43.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.087+0000 7f59ebfb4700 1 -- start start 2023-03-03T11:39:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.087+0000 7f59ebfb4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59e406d240 0x7f59e40c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.087+0000 7f59ebfb4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e40c3e30 0x7f59e40c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.087+0000 7f59ebfb4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e40ad2b0 con 0x7f59e414fc50 2023-03-03T11:39:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.087+0000 7f59ebfb4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e40c30c0 con 0x7f59e406d240 2023-03-03T11:39:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.087+0000 7f59ebfb4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4156420 con 0x7f59e40c3e30 2023-03-03T11:39:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e954f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59e406d240 0x7f59e40c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e9d50700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f59e414fc50 0x7f59e40c1f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59428/0 (socket says 172.21.15.7:59428) 2023-03-03T11:39:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e9d50700 1 -- 172.21.15.7:0/4006197090 learned_addr learned my addr 172.21.15.7:0/4006197090 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:39:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59ea551700 1 --2- 172.21.15.7:0/4006197090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e40c3e30 0x7f59e40c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e954f700 1 -- 172.21.15.7:0/4006197090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e40c3e30 msgr2=0x7f59e40c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:39:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e954f700 1 --2- 172.21.15.7:0/4006197090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e40c3e30 0x7f59e40c42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e954f700 1 -- 172.21.15.7:0/4006197090 >> v1:172.21.15.7:6789/0 conn(0x7f59e414fc50 legacy=0x7f59e40c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e8d4e700 1 -- 172.21.15.7:0/4006197090 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491888654 0 0) 0x7f59e40ad2b0 con 0x7f59e414fc50 2023-03-03T11:39:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e954f700 1 -- 172.21.15.7:0/4006197090 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e40ad520 con 0x7f59e406d240 2023-03-03T11:39:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e954f700 1 --2- 172.21.15.7:0/4006197090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 0x7f59e40c27b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009840 tx=0x7f59e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c5a7c3effc22a4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:39:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e8d4e700 1 -- 172.21.15.7:0/4006197090 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0011830 con 0x7f59e406d240 2023-03-03T11:39:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e8d4e700 1 -- 172.21.15.7:0/4006197090 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e0011990 con 0x7f59e406d240 2023-03-03T11:39:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.088+0000 7f59e8d4e700 1 -- 172.21.15.7:0/4006197090 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0011d20 con 0x7f59e406d240 2023-03-03T11:39:43.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.089+0000 7f59ebfb4700 1 -- 172.21.15.7:0/4006197090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 msgr2=0x7f59e40c27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:43.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.089+0000 7f59ebfb4700 1 --2- 172.21.15.7:0/4006197090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 0x7f59e40c27b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f59e0009840 tx=0x7f59e0009b50 comp rx=0 tx=0).stop 2023-03-03T11:39:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.089+0000 7f59ebfb4700 1 -- 172.21.15.7:0/4006197090 shutdown_connections 2023-03-03T11:39:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.089+0000 7f59ebfb4700 1 --2- 172.21.15.7:0/4006197090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e40c3e30 0x7f59e40c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.089+0000 7f59ebfb4700 1 --2- 172.21.15.7:0/4006197090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 0x7f59e40c27b0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.089+0000 7f59ebfb4700 1 -- 172.21.15.7:0/4006197090 >> 172.21.15.7:0/4006197090 conn(0x7f59e406c110 msgr2=0x7f59e406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:39:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.089+0000 7f59ebfb4700 1 -- 172.21.15.7:0/4006197090 shutdown_connections 2023-03-03T11:39:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.089+0000 7f59ebfb4700 1 -- 172.21.15.7:0/4006197090 wait complete. 2023-03-03T11:39:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.090+0000 7f59ebfb4700 1 Processor -- start 2023-03-03T11:39:43.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.090+0000 7f59ebfb4700 1 -- start start 2023-03-03T11:39:43.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59ebfb4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 0x7f59e4147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:43.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59ebfb4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e40c3e30 0x7f59e40c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:43.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59ebfb4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e414fc50 0x7f59e40c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:43.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59ebfb4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40c6a40 con 0x7f59e406d240 2023-03-03T11:39:43.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59ebfb4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4148a90 con 0x7f59e40c3e30 2023-03-03T11:39:43.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59ebfb4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40c5c10 con 0x7f59e414fc50 2023-03-03T11:39:43.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59e9d50700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 0x7f59e4147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59e9d50700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 0x7f59e4147730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37224/0 (socket says 172.21.15.7:37224) 2023-03-03T11:39:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59e9d50700 1 -- 172.21.15.7:0/3553217828 learned_addr learned my addr 172.21.15.7:0/3553217828 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:39:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59ea551700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e414fc50 0x7f59e40c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59e954f700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e40c3e30 0x7f59e40c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59e9d50700 1 -- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e414fc50 msgr2=0x7f59e40c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59e9d50700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e414fc50 0x7f59e40c7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59e9d50700 1 -- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e40c3e30 msgr2=0x7f59e40c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59e9d50700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e40c3e30 0x7f59e40c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59e9d50700 1 -- 172.21.15.7:0/3553217828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e00094a0 con 0x7f59e406d240 2023-03-03T11:39:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.091+0000 7f59e9d50700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 0x7f59e4147730 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f59d40011e0 tx=0x7f59d400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:39:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.092+0000 7f59da7fc700 1 -- 172.21.15.7:0/3553217828 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4004c80 con 0x7f59e406d240 2023-03-03T11:39:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.092+0000 7f59da7fc700 1 -- 172.21.15.7:0/3553217828 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d4004e00 con 0x7f59e406d240 2023-03-03T11:39:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.092+0000 7f59ebfb4700 1 -- 172.21.15.7:0/3553217828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e4074220 con 0x7f59e406d240 2023-03-03T11:39:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.092+0000 7f59da7fc700 1 -- 172.21.15.7:0/3553217828 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d4007310 con 0x7f59e406d240 2023-03-03T11:39:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.092+0000 7f59ebfb4700 1 -- 172.21.15.7:0/3553217828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e40c2cf0 con 0x7f59e406d240 2023-03-03T11:39:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.093+0000 7f59ebfb4700 1 -- 172.21.15.7:0/3553217828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e4156690 con 0x7f59e406d240 2023-03-03T11:39:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.093+0000 7f59da7fc700 1 -- 172.21.15.7:0/3553217828 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f59d401a020 con 0x7f59e406d240 2023-03-03T11:39:43.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.093+0000 7f59da7fc700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59c80682f0 0x7f59c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:43.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.094+0000 7f59da7fc700 1 -- 172.21.15.7:0/3553217828 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f59d4085e90 con 0x7f59e406d240 2023-03-03T11:39:43.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.094+0000 7f59e954f700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59c80682f0 0x7f59c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:43.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.094+0000 7f59e954f700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59c80682f0 0x7f59c806a7a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f59e0000c00 tx=0x7f59e0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:39:43.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.098+0000 7f59da7fc700 1 -- 172.21.15.7:0/3553217828 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f59d4052140 con 0x7f59e406d240 2023-03-03T11:39:43.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.338+0000 7f59ebfb4700 1 -- 172.21.15.7:0/3553217828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59e41e1980 con 0x7f59e406d240 2023-03-03T11:39:43.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.338+0000 7f59da7fc700 1 -- 172.21.15.7:0/3553217828 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f59e41e1980 con 0x7f59e406d240 2023-03-03T11:39:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:39:43.340 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:39:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.340+0000 7f59ebfb4700 1 -- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59c80682f0 msgr2=0x7f59c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:43.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.340+0000 7f59ebfb4700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59c80682f0 0x7f59c806a7a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f59e0000c00 tx=0x7f59e0008030 comp rx=0 tx=0).stop 2023-03-03T11:39:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.340+0000 7f59ebfb4700 1 -- 172.21.15.7:0/3553217828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 msgr2=0x7f59e4147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.340+0000 7f59ebfb4700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 0x7f59e4147730 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f59d40011e0 tx=0x7f59d400cae0 comp rx=0 tx=0).stop 2023-03-03T11:39:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.340+0000 7f59ebfb4700 1 -- 172.21.15.7:0/3553217828 shutdown_connections 2023-03-03T11:39:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.340+0000 7f59ebfb4700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59e414fc50 0x7f59e40c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.340+0000 7f59ebfb4700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59c80682f0 0x7f59c806a7a0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.340+0000 7f59ebfb4700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59e406d240 0x7f59e4147730 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.340+0000 7f59ebfb4700 1 --2- 172.21.15.7:0/3553217828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59e40c3e30 0x7f59e40c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:43.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.340+0000 7f59ebfb4700 1 -- 172.21.15.7:0/3553217828 >> 172.21.15.7:0/3553217828 conn(0x7f59e406c110 msgr2=0x7f59e406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:39:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.341+0000 7f59ebfb4700 1 -- 172.21.15.7:0/3553217828 shutdown_connections 2023-03-03T11:39:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:43.341+0000 7f59ebfb4700 1 -- 172.21.15.7:0/3553217828 wait complete. 2023-03-03T11:39:43.344 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:39:43.354 INFO:tasks.mds_thrash.fs.[a]:waiting for 91 secs before thrashing 2023-03-03T11:39:46.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:46.759 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:39:59.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:39:59.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.482+0000 7fab7641b700 1 Processor -- start 2023-03-03T11:39:59.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.482+0000 7fab7641b700 1 -- start start 2023-03-03T11:39:59.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab7641b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab70151cd0 0x7fab70154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:59.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab7641b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab700c2e30 0x7fab70158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab7641b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fab7006c6c0 0x7fab700736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab7641b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab70157ff0 con 0x7fab70151cd0 2023-03-03T11:39:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab7641b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7006d640 con 0x7fab7006c6c0 2023-03-03T11:39:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab7641b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7006cda0 con 0x7fab700c2e30 2023-03-03T11:39:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab6ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab70151cd0 0x7fab70154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab6ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab70151cd0 0x7fab70154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37250/0 (socket says 172.21.15.7:37250) 2023-03-03T11:39:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab6ffff700 1 -- 172.21.15.7:0/1678500713 learned_addr learned my addr 172.21.15.7:0/1678500713 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:39:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab749b8700 1 --2- 172.21.15.7:0/1678500713 >> v2:172.21.15.143:3300/0 conn(0x7fab7006c6c0 0x7fab700736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab6f7fe700 1 --2- 172.21.15.7:0/1678500713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab700c2e30 0x7fab70158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab6ffff700 1 -- 172.21.15.7:0/1678500713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab700c2e30 msgr2=0x7fab70158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:39:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab6ffff700 1 --2- 172.21.15.7:0/1678500713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab700c2e30 0x7fab70158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab6ffff700 1 -- 172.21.15.7:0/1678500713 >> v2:172.21.15.143:3300/0 conn(0x7fab7006c6c0 msgr2=0x7fab700736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:39:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab6ffff700 1 --2- 172.21.15.7:0/1678500713 >> v2:172.21.15.143:3300/0 conn(0x7fab7006c6c0 0x7fab700736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.483+0000 7fab6ffff700 1 -- 172.21.15.7:0/1678500713 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7006d080 con 0x7fab70151cd0 2023-03-03T11:39:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab6ffff700 1 --2- 172.21.15.7:0/1678500713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab70151cd0 0x7fab70154510 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fab7006ffb0 tx=0x7fab60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ed80babd874247 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:39:59.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab6effd700 1 -- 172.21.15.7:0/1678500713 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6000b870 con 0x7fab70151cd0 2023-03-03T11:39:59.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab6effd700 1 -- 172.21.15.7:0/1678500713 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6000b9d0 con 0x7fab70151cd0 2023-03-03T11:39:59.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab6effd700 1 -- 172.21.15.7:0/1678500713 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6000bd60 con 0x7fab70151cd0 2023-03-03T11:39:59.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab7641b700 1 -- 172.21.15.7:0/1678500713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab70151cd0 msgr2=0x7fab70154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:59.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab7641b700 1 --2- 172.21.15.7:0/1678500713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab70151cd0 0x7fab70154510 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fab7006ffb0 tx=0x7fab60009bd0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab7641b700 1 -- 172.21.15.7:0/1678500713 shutdown_connections 2023-03-03T11:39:59.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab7641b700 1 --2- 172.21.15.7:0/1678500713 >> v2:172.21.15.143:3300/0 conn(0x7fab7006c6c0 0x7fab700736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab7641b700 1 --2- 172.21.15.7:0/1678500713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab70151cd0 0x7fab70154510 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab7641b700 1 --2- 172.21.15.7:0/1678500713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab700c2e30 0x7fab70158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab7641b700 1 -- 172.21.15.7:0/1678500713 >> 172.21.15.7:0/1678500713 conn(0x7fab700a94d0 msgr2=0x7fab700a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:39:59.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab7641b700 1 -- 172.21.15.7:0/1678500713 shutdown_connections 2023-03-03T11:39:59.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.484+0000 7fab7641b700 1 -- 172.21.15.7:0/1678500713 wait complete. 2023-03-03T11:39:59.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.486+0000 7fab7641b700 1 Processor -- start 2023-03-03T11:39:59.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.486+0000 7fab7641b700 1 -- start start 2023-03-03T11:39:59.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.486+0000 7fab7641b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab7006c6c0 0x7fab701e1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:59.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.486+0000 7fab7641b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab700c2e30 0x7fab70156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:59.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.486+0000 7fab7641b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab70151cd0 0x7fab70070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:59.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.486+0000 7fab7641b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7014a960 con 0x7fab7006c6c0 2023-03-03T11:39:59.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.486+0000 7fab7641b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab70071b30 con 0x7fab700c2e30 2023-03-03T11:39:59.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.486+0000 7fab7641b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab700720b0 con 0x7fab70151cd0 2023-03-03T11:39:59.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab7006c6c0 0x7fab701e1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:59.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab7006c6c0 0x7fab701e1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37256/0 (socket says 172.21.15.7:37256) 2023-03-03T11:39:59.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6ffff700 1 -- 172.21.15.7:0/3210283568 learned_addr learned my addr 172.21.15.7:0/3210283568 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:39:59.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab749b8700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab70151cd0 0x7fab70070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:59.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6f7fe700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab700c2e30 0x7fab70156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:59.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6ffff700 1 -- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab70151cd0 msgr2=0x7fab70070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:59.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6ffff700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab70151cd0 0x7fab70070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6ffff700 1 -- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab700c2e30 msgr2=0x7fab70156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:59.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6ffff700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab700c2e30 0x7fab70156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6ffff700 1 -- 172.21.15.7:0/3210283568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60009550 con 0x7fab7006c6c0 2023-03-03T11:39:59.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6ffff700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab7006c6c0 0x7fab701e1190 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fab700744a0 tx=0x7fab60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:39:59.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6cff9700 1 -- 172.21.15.7:0/3210283568 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60004540 con 0x7fab7006c6c0 2023-03-03T11:39:59.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6cff9700 1 -- 172.21.15.7:0/3210283568 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab600036a0 con 0x7fab7006c6c0 2023-03-03T11:39:59.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab7641b700 1 -- 172.21.15.7:0/3210283568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab70157c80 con 0x7fab7006c6c0 2023-03-03T11:39:59.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab6cff9700 1 -- 172.21.15.7:0/3210283568 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60003a50 con 0x7fab7006c6c0 2023-03-03T11:39:59.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.487+0000 7fab7641b700 1 -- 172.21.15.7:0/3210283568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7006d360 con 0x7fab7006c6c0 2023-03-03T11:39:59.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.488+0000 7fab7641b700 1 -- 172.21.15.7:0/3210283568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab70154a50 con 0x7fab7006c6c0 2023-03-03T11:39:59.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.489+0000 7fab6cff9700 1 -- 172.21.15.7:0/3210283568 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fab6000f050 con 0x7fab7006c6c0 2023-03-03T11:39:59.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.489+0000 7fab6cff9700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab500682f0 0x7fab5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:39:59.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.489+0000 7fab6cff9700 1 -- 172.21.15.7:0/3210283568 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fab60087910 con 0x7fab7006c6c0 2023-03-03T11:39:59.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.489+0000 7fab6f7fe700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab500682f0 0x7fab5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:39:59.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.490+0000 7fab6f7fe700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab500682f0 0x7fab5006a7a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fab7014acf0 tx=0x7fab58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:39:59.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.494+0000 7fab6cff9700 1 -- 172.21.15.7:0/3210283568 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab60053cb0 con 0x7fab7006c6c0 2023-03-03T11:39:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.733+0000 7fab7641b700 1 -- 172.21.15.7:0/3210283568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab701534c0 con 0x7fab7006c6c0 2023-03-03T11:39:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.733+0000 7fab6cff9700 1 -- 172.21.15.7:0/3210283568 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fab600572d0 con 0x7fab7006c6c0 2023-03-03T11:39:59.735 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:39:59.735 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:39:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 -- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab500682f0 msgr2=0x7fab5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab500682f0 0x7fab5006a7a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fab7014acf0 tx=0x7fab58008040 comp rx=0 tx=0).stop 2023-03-03T11:39:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 -- 172.21.15.7:0/3210283568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab7006c6c0 msgr2=0x7fab701e1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:39:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab7006c6c0 0x7fab701e1190 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fab700744a0 tx=0x7fab60004750 comp rx=0 tx=0).stop 2023-03-03T11:39:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 -- 172.21.15.7:0/3210283568 shutdown_connections 2023-03-03T11:39:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab70151cd0 0x7fab70070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab500682f0 0x7fab5006a7a0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab7006c6c0 0x7fab701e1190 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 --2- 172.21.15.7:0/3210283568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab700c2e30 0x7fab70156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:39:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 -- 172.21.15.7:0/3210283568 >> 172.21.15.7:0/3210283568 conn(0x7fab700a94d0 msgr2=0x7fab70065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:39:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 -- 172.21.15.7:0/3210283568 shutdown_connections 2023-03-03T11:39:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:39:59.735+0000 7fab7641b700 1 -- 172.21.15.7:0/3210283568 wait complete. 2023-03-03T11:39:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:39:59.749 INFO:tasks.mds_thrash.fs.[b]:waiting for 111 secs before thrashing 2023-03-03T11:40:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T11:40:00.000+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T11:40:16.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:16.922 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:47.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:40:47.094 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:14.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:41:14.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.492+0000 7f8f5864b700 1 Processor -- start 2023-03-03T11:41:14.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.492+0000 7f8f5864b700 1 -- start start 2023-03-03T11:41:14.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f5864b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8f50151ab0 0x7f8f500736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:14.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f5864b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f50073c30 0x7f8f5006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:14.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f5864b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8f5006c3e0 0x7f8f50157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:14.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f5864b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f5006d780 con 0x7f8f5006c3e0 2023-03-03T11:41:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f5864b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f5006cb70 con 0x7f8f50151ab0 2023-03-03T11:41:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f5864b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5006ce50 con 0x7f8f50073c30 2023-03-03T11:41:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f56be8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8f5006c3e0 0x7f8f50157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f56be8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8f5006c3e0 0x7f8f50157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37324/0 (socket says 172.21.15.7:37324) 2023-03-03T11:41:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f56be8700 1 -- 172.21.15.7:0/629129696 learned_addr learned my addr 172.21.15.7:0/629129696 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:41:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f563e7700 1 --2- 172.21.15.7:0/629129696 >> v2:172.21.15.143:3300/0 conn(0x7f8f50151ab0 0x7f8f500736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f55be6700 1 --2- 172.21.15.7:0/629129696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f50073c30 0x7f8f5006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:14.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f56be8700 1 -- 172.21.15.7:0/629129696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f50073c30 msgr2=0x7f8f5006bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:41:14.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f56be8700 1 --2- 172.21.15.7:0/629129696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f50073c30 0x7f8f5006bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f56be8700 1 -- 172.21.15.7:0/629129696 >> v2:172.21.15.143:3300/0 conn(0x7f8f50151ab0 msgr2=0x7f8f500736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:41:14.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f56be8700 1 --2- 172.21.15.7:0/629129696 >> v2:172.21.15.143:3300/0 conn(0x7f8f50151ab0 0x7f8f500736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f56be8700 1 -- 172.21.15.7:0/629129696 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5006d130 con 0x7f8f5006c3e0 2023-03-03T11:41:14.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.493+0000 7f8f56be8700 1 --2- 172.21.15.7:0/629129696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f5006c3e0 0x7f8f50157dc0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f8f5006bb20 tx=0x7f8f4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1def65e8a85a10b6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:14.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f553e5700 1 -- 172.21.15.7:0/629129696 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4800e070 con 0x7f8f5006c3e0 2023-03-03T11:41:14.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f553e5700 1 -- 172.21.15.7:0/629129696 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4800b910 con 0x7f8f5006c3e0 2023-03-03T11:41:14.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f553e5700 1 -- 172.21.15.7:0/629129696 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4800bcc0 con 0x7f8f5006c3e0 2023-03-03T11:41:14.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f5864b700 1 -- 172.21.15.7:0/629129696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f5006c3e0 msgr2=0x7f8f50157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:14.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f5864b700 1 --2- 172.21.15.7:0/629129696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f5006c3e0 0x7f8f50157dc0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f8f5006bb20 tx=0x7f8f4800b330 comp rx=0 tx=0).stop 2023-03-03T11:41:14.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f5864b700 1 -- 172.21.15.7:0/629129696 shutdown_connections 2023-03-03T11:41:14.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f5864b700 1 --2- 172.21.15.7:0/629129696 >> v2:172.21.15.143:3300/0 conn(0x7f8f50151ab0 0x7f8f500736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f5864b700 1 --2- 172.21.15.7:0/629129696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f5006c3e0 0x7f8f50157dc0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f5864b700 1 --2- 172.21.15.7:0/629129696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f50073c30 0x7f8f5006bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f5864b700 1 -- 172.21.15.7:0/629129696 >> 172.21.15.7:0/629129696 conn(0x7f8f500a94d0 msgr2=0x7f8f500a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:41:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f5864b700 1 -- 172.21.15.7:0/629129696 shutdown_connections 2023-03-03T11:41:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.494+0000 7f8f5864b700 1 -- 172.21.15.7:0/629129696 wait complete. 2023-03-03T11:41:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.496+0000 7f8f5864b700 1 Processor -- start 2023-03-03T11:41:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.496+0000 7f8f5864b700 1 -- start start 2023-03-03T11:41:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.496+0000 7f8f5864b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f5006c3e0 0x7f8f50146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:14.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.496+0000 7f8f5864b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f50073c30 0x7f8f500abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:14.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.496+0000 7f8f5864b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f50151ab0 0x7f8f5014bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:14.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.496+0000 7f8f5864b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f50146c80 con 0x7f8f50073c30 2023-03-03T11:41:14.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.496+0000 7f8f5864b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f500aa1d0 con 0x7f8f50151ab0 2023-03-03T11:41:14.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.496+0000 7f8f5864b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f501ea920 con 0x7f8f5006c3e0 2023-03-03T11:41:14.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.496+0000 7f8f55be6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f50073c30 0x7f8f500abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:14.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f55be6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f50073c30 0x7f8f500abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37328/0 (socket says 172.21.15.7:37328) 2023-03-03T11:41:14.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f55be6700 1 -- 172.21.15.7:0/2687443156 learned_addr learned my addr 172.21.15.7:0/2687443156 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:41:14.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f56be8700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f50151ab0 0x7f8f5014bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f563e7700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f5006c3e0 0x7f8f50146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f55be6700 1 -- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f5006c3e0 msgr2=0x7f8f50146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f55be6700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f5006c3e0 0x7f8f50146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f55be6700 1 -- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f50151ab0 msgr2=0x7f8f5014bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f55be6700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f50151ab0 0x7f8f5014bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f55be6700 1 -- 172.21.15.7:0/2687443156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f48009b30 con 0x7f8f50073c30 2023-03-03T11:41:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f563e7700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f5006c3e0 0x7f8f50146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:41:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f56be8700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f50151ab0 0x7f8f5014bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:41:14.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f55be6700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f50073c30 0x7f8f500abd00 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000e9e0 tx=0x7f8f4000eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:14.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f46ffd700 1 -- 172.21.15.7:0/2687443156 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f40009710 con 0x7f8f50073c30 2023-03-03T11:41:14.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f46ffd700 1 -- 172.21.15.7:0/2687443156 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4000c7f0 con 0x7f8f50073c30 2023-03-03T11:41:14.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f5864b700 1 -- 172.21.15.7:0/2687443156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f50151e80 con 0x7f8f50073c30 2023-03-03T11:41:14.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f46ffd700 1 -- 172.21.15.7:0/2687443156 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4000cb60 con 0x7f8f50073c30 2023-03-03T11:41:14.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.497+0000 7f8f5864b700 1 -- 172.21.15.7:0/2687443156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5006d410 con 0x7f8f50073c30 2023-03-03T11:41:14.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.498+0000 7f8f5864b700 1 -- 172.21.15.7:0/2687443156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f50152010 con 0x7f8f50073c30 2023-03-03T11:41:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.499+0000 7f8f46ffd700 1 -- 172.21.15.7:0/2687443156 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8f40019030 con 0x7f8f50073c30 2023-03-03T11:41:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.499+0000 7f8f46ffd700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f340682f0 0x7f8f3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:14.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.499+0000 7f8f46ffd700 1 -- 172.21.15.7:0/2687443156 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8f40086110 con 0x7f8f50073c30 2023-03-03T11:41:14.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.499+0000 7f8f563e7700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f340682f0 0x7f8f3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:14.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.500+0000 7f8f563e7700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f340682f0 0x7f8f3406a7a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8f501e1be0 tx=0x7f8f4c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:14.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.503+0000 7f8f46ffd700 1 -- 172.21.15.7:0/2687443156 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8f400524b0 con 0x7f8f50073c30 2023-03-03T11:41:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.740+0000 7f8f5864b700 1 -- 172.21.15.7:0/2687443156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f50074bb0 con 0x7f8f50073c30 2023-03-03T11:41:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.740+0000 7f8f46ffd700 1 -- 172.21.15.7:0/2687443156 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8f40017090 con 0x7f8f50073c30 2023-03-03T11:41:14.743 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:41:14.743 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:41:14.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 -- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f340682f0 msgr2=0x7f8f3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:14.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f340682f0 0x7f8f3406a7a0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8f501e1be0 tx=0x7f8f4c008040 comp rx=0 tx=0).stop 2023-03-03T11:41:14.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 -- 172.21.15.7:0/2687443156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f50073c30 msgr2=0x7f8f500abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:14.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f50073c30 0x7f8f500abd00 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000e9e0 tx=0x7f8f4000eda0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 -- 172.21.15.7:0/2687443156 shutdown_connections 2023-03-03T11:41:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f50151ab0 0x7f8f5014bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f5006c3e0 0x7f8f50146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f340682f0 0x7f8f3406a7a0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 --2- 172.21.15.7:0/2687443156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f50073c30 0x7f8f500abd00 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 -- 172.21.15.7:0/2687443156 >> 172.21.15.7:0/2687443156 conn(0x7f8f500a94d0 msgr2=0x7f8f500a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:41:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 -- 172.21.15.7:0/2687443156 shutdown_connections 2023-03-03T11:41:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:14.742+0000 7f8f5864b700 1 -- 172.21.15.7:0/2687443156 wait complete. 2023-03-03T11:41:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:41:14.757 INFO:tasks.mds_thrash.fs.[a]:waiting for 102 secs before thrashing 2023-03-03T11:41:17.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:17.249 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:47.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:47.331 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:41:50.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:41:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.887+0000 7f0f72aa5700 1 Processor -- start 2023-03-03T11:41:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.887+0000 7f0f72aa5700 1 -- start start 2023-03-03T11:41:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.887+0000 7f0f72aa5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f6c06d870 0x7f0f6c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.887+0000 7f0f72aa5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0f6c0c1da0 0x7f0f6c0c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.887+0000 7f0f72aa5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c0c3620 con 0x7f0f6c0c3e30 2023-03-03T11:41:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.887+0000 7f0f72aa5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6c0c2af0 con 0x7f0f6c06d870 2023-03-03T11:41:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.887+0000 7f0f72aa5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f6c1545e0 con 0x7f0f6c0c1da0 2023-03-03T11:41:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f70841700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f6c06d870 0x7f0f6c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f71042700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0f6c0c3e30 0x7f0f6c157ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59598/0 (socket says 172.21.15.7:59598) 2023-03-03T11:41:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f71042700 1 -- 172.21.15.7:0/1421005607 learned_addr learned my addr 172.21.15.7:0/1421005607 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:41:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f70841700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f6c06d870 0x7f0f6c1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37384/0 (socket says 172.21.15.7:37384) 2023-03-03T11:41:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f6bfff700 1 --2- 172.21.15.7:0/1421005607 >> v2:172.21.15.143:3300/0 conn(0x7f0f6c0c1da0 0x7f0f6c0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f70841700 1 -- 172.21.15.7:0/1421005607 >> v2:172.21.15.143:3300/0 conn(0x7f0f6c0c1da0 msgr2=0x7f0f6c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:41:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f70841700 1 --2- 172.21.15.7:0/1421005607 >> v2:172.21.15.143:3300/0 conn(0x7f0f6c0c1da0 0x7f0f6c0c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f70841700 1 -- 172.21.15.7:0/1421005607 >> v1:172.21.15.7:6789/0 conn(0x7f0f6c0c3e30 legacy=0x7f0f6c157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f6b7fe700 1 -- 172.21.15.7:0/1421005607 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184161214 0 0) 0x7f0f6c0c3620 con 0x7f0f6c0c3e30 2023-03-03T11:41:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f70841700 1 -- 172.21.15.7:0/1421005607 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c1548c0 con 0x7f0f6c06d870 2023-03-03T11:41:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f70841700 1 --2- 172.21.15.7:0/1421005607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 0x7f0f6c1550c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0c33a0 tx=0x7f0f60009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a75e337d98ef98 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f6b7fe700 1 -- 172.21.15.7:0/1421005607 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60011830 con 0x7f0f6c06d870 2023-03-03T11:41:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f6b7fe700 1 -- 172.21.15.7:0/1421005607 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f60011990 con 0x7f0f6c06d870 2023-03-03T11:41:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.888+0000 7f0f6b7fe700 1 -- 172.21.15.7:0/1421005607 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60011d20 con 0x7f0f6c06d870 2023-03-03T11:41:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.889+0000 7f0f72aa5700 1 -- 172.21.15.7:0/1421005607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 msgr2=0x7f0f6c1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.889+0000 7f0f72aa5700 1 --2- 172.21.15.7:0/1421005607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 0x7f0f6c1550c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0c33a0 tx=0x7f0f60009b20 comp rx=0 tx=0).stop 2023-03-03T11:41:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.889+0000 7f0f72aa5700 1 -- 172.21.15.7:0/1421005607 shutdown_connections 2023-03-03T11:41:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.889+0000 7f0f72aa5700 1 --2- 172.21.15.7:0/1421005607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 0x7f0f6c1550c0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.889+0000 7f0f72aa5700 1 --2- 172.21.15.7:0/1421005607 >> v2:172.21.15.143:3300/0 conn(0x7f0f6c0c1da0 0x7f0f6c0c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.889+0000 7f0f72aa5700 1 -- 172.21.15.7:0/1421005607 >> 172.21.15.7:0/1421005607 conn(0x7f0f6c06c110 msgr2=0x7f0f6c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:41:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.889+0000 7f0f72aa5700 1 -- 172.21.15.7:0/1421005607 shutdown_connections 2023-03-03T11:41:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.889+0000 7f0f72aa5700 1 -- 172.21.15.7:0/1421005607 wait complete. 2023-03-03T11:41:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.890+0000 7f0f72aa5700 1 Processor -- start 2023-03-03T11:41:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.890+0000 7f0f72aa5700 1 -- start start 2023-03-03T11:41:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f72aa5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 0x7f0f6c1ed730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f72aa5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f6c0c1da0 0x7f0f6c0c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f72aa5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6c0c3e30 0x7f0f6c142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f72aa5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c14e450 con 0x7f0f6c06d870 2023-03-03T11:41:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f72aa5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0acba0 con 0x7f0f6c0c1da0 2023-03-03T11:41:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f72aa5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0ace80 con 0x7f0f6c0c3e30 2023-03-03T11:41:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f70841700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 0x7f0f6c1ed730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f70841700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 0x7f0f6c1ed730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37388/0 (socket says 172.21.15.7:37388) 2023-03-03T11:41:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f70841700 1 -- 172.21.15.7:0/14974673 learned_addr learned my addr 172.21.15.7:0/14974673 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:41:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f71042700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6c0c3e30 0x7f0f6c142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f6bfff700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f6c0c1da0 0x7f0f6c0c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f70841700 1 -- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6c0c3e30 msgr2=0x7f0f6c142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f70841700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6c0c3e30 0x7f0f6c142a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f70841700 1 -- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f6c0c1da0 msgr2=0x7f0f6c0c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f70841700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f6c0c1da0 0x7f0f6c0c0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f70841700 1 -- 172.21.15.7:0/14974673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f600094a0 con 0x7f0f6c06d870 2023-03-03T11:41:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f6bfff700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f6c0c1da0 0x7f0f6c0c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:41:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.891+0000 7f0f71042700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6c0c3e30 0x7f0f6c142a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:41:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.892+0000 7f0f70841700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 0x7f0f6c1ed730 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f0f600090b0 tx=0x7f0f60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.892+0000 7f0f697fa700 1 -- 172.21.15.7:0/14974673 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f600038a0 con 0x7f0f6c06d870 2023-03-03T11:41:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.892+0000 7f0f697fa700 1 -- 172.21.15.7:0/14974673 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6001f070 con 0x7f0f6c06d870 2023-03-03T11:41:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.892+0000 7f0f72aa5700 1 -- 172.21.15.7:0/14974673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6c06dc40 con 0x7f0f6c06d870 2023-03-03T11:41:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.892+0000 7f0f697fa700 1 -- 172.21.15.7:0/14974673 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60003c20 con 0x7f0f6c06d870 2023-03-03T11:41:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.892+0000 7f0f72aa5700 1 -- 172.21.15.7:0/14974673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6c154ba0 con 0x7f0f6c06d870 2023-03-03T11:41:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.893+0000 7f0f72aa5700 1 -- 172.21.15.7:0/14974673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6c0c38e0 con 0x7f0f6c06d870 2023-03-03T11:41:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.893+0000 7f0f697fa700 1 -- 172.21.15.7:0/14974673 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0f6000f050 con 0x7f0f6c06d870 2023-03-03T11:41:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.894+0000 7f0f697fa700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f500682f0 0x7f0f5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:41:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.894+0000 7f0f697fa700 1 -- 172.21.15.7:0/14974673 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0f60021070 con 0x7f0f6c06d870 2023-03-03T11:41:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.894+0000 7f0f6bfff700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f500682f0 0x7f0f5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:41:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.894+0000 7f0f6bfff700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f500682f0 0x7f0f5006a7a0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0ac560 tx=0x7f0f58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:41:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:50.898+0000 7f0f697fa700 1 -- 172.21.15.7:0/14974673 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0f60053e40 con 0x7f0f6c06d870 2023-03-03T11:41:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.138+0000 7f0f72aa5700 1 -- 172.21.15.7:0/14974673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f6c074bb0 con 0x7f0f6c06d870 2023-03-03T11:41:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.138+0000 7f0f697fa700 1 -- 172.21.15.7:0/14974673 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0f60057460 con 0x7f0f6c06d870 2023-03-03T11:41:51.140 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:41:51.140 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:41:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 -- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f500682f0 msgr2=0x7f0f5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f500682f0 0x7f0f5006a7a0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0ac560 tx=0x7f0f58008040 comp rx=0 tx=0).stop 2023-03-03T11:41:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 -- 172.21.15.7:0/14974673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 msgr2=0x7f0f6c1ed730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:41:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 0x7f0f6c1ed730 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f0f600090b0 tx=0x7f0f60004750 comp rx=0 tx=0).stop 2023-03-03T11:41:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 -- 172.21.15.7:0/14974673 shutdown_connections 2023-03-03T11:41:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f6c0c3e30 0x7f0f6c142a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f500682f0 0x7f0f5006a7a0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f6c06d870 0x7f0f6c1ed730 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 --2- 172.21.15.7:0/14974673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f6c0c1da0 0x7f0f6c0c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:41:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 -- 172.21.15.7:0/14974673 >> 172.21.15.7:0/14974673 conn(0x7f0f6c06c110 msgr2=0x7f0f6c153e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:41:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 -- 172.21.15.7:0/14974673 shutdown_connections 2023-03-03T11:41:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:41:51.140+0000 7f0f72aa5700 1 -- 172.21.15.7:0/14974673 wait complete. 2023-03-03T11:41:51.144 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:41:51.154 INFO:tasks.mds_thrash.fs.[b]:waiting for 24 secs before thrashing 2023-03-03T11:42:15.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:42:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.292+0000 7f4896102700 1 Processor -- start 2023-03-03T11:42:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.292+0000 7f4896102700 1 -- start start 2023-03-03T11:42:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f4896102700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4890151d40 0x7f48900736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f4896102700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489006d430 con 0x7f4890073c30 2023-03-03T11:42:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f4896102700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489006cb00 con 0x7f48900c2930 2023-03-03T11:42:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f4896102700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489006c2a0 con 0x7f4890151d40 2023-03-03T11:42:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f488f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4890151d40 0x7f48900736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f488ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4890073c30 0x7f48901581a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59616/0 (socket says 172.21.15.7:59616) 2023-03-03T11:42:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f488ffff700 1 -- 172.21.15.7:0/4135266035 learned_addr learned my addr 172.21.15.7:0/4135266035 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:42:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f488f7fe700 1 -- 172.21.15.7:0/4135266035 >> v1:172.21.15.143:6789/0 conn(0x7f48900c2930 legacy=0x7f4890157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:42:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f488f7fe700 1 -- 172.21.15.7:0/4135266035 >> v1:172.21.15.7:6789/0 conn(0x7f4890073c30 legacy=0x7f48901581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f488f7fe700 1 -- 172.21.15.7:0/4135266035 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489006cde0 con 0x7f4890151d40 2023-03-03T11:42:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f488e7fc700 1 -- 172.21.15.7:0/4135266035 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822603796 0 0) 0x7f489006d430 con 0x7f4890073c30 2023-03-03T11:42:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f488f7fe700 1 --2- 172.21.15.7:0/4135266035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4890151d40 0x7f48900736f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f4878009870 tx=0x7f4878009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b02f55fd26c71b4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:42:15.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.293+0000 7f488e7fc700 1 -- 172.21.15.7:0/4135266035 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4878004230 con 0x7f4890151d40 2023-03-03T11:42:15.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.294+0000 7f488e7fc700 1 -- 172.21.15.7:0/4135266035 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4878004390 con 0x7f4890151d40 2023-03-03T11:42:15.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.294+0000 7f488e7fc700 1 -- 172.21.15.7:0/4135266035 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48780036a0 con 0x7f4890151d40 2023-03-03T11:42:15.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.294+0000 7f4896102700 1 -- 172.21.15.7:0/4135266035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4890151d40 msgr2=0x7f48900736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:15.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.294+0000 7f4896102700 1 --2- 172.21.15.7:0/4135266035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4890151d40 0x7f48900736f0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f4878009870 tx=0x7f4878009b80 comp rx=0 tx=0).stop 2023-03-03T11:42:15.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.294+0000 7f4896102700 1 -- 172.21.15.7:0/4135266035 shutdown_connections 2023-03-03T11:42:15.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.294+0000 7f4896102700 1 --2- 172.21.15.7:0/4135266035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4890151d40 0x7f48900736f0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:15.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.294+0000 7f4896102700 1 -- 172.21.15.7:0/4135266035 >> 172.21.15.7:0/4135266035 conn(0x7f48900a94d0 msgr2=0x7f48900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:42:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.294+0000 7f4896102700 1 -- 172.21.15.7:0/4135266035 shutdown_connections 2023-03-03T11:42:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.294+0000 7f4896102700 1 -- 172.21.15.7:0/4135266035 wait complete. 2023-03-03T11:42:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.296+0000 7f4896102700 1 Processor -- start 2023-03-03T11:42:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.296+0000 7f4896102700 1 -- start start 2023-03-03T11:42:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f4896102700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4890073c30 0x7f48901562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f4896102700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48900c2930 0x7f48900c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f4896102700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4890151d40 0x7f4890145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f4896102700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890157850 con 0x7f48900c2930 2023-03-03T11:42:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f4896102700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48900c6210 con 0x7f4890151d40 2023-03-03T11:42:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f4896102700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48901edfe0 con 0x7f4890073c30 2023-03-03T11:42:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48900c2930 0x7f48900c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:15.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48900c2930 0x7f48900c6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37408/0 (socket says 172.21.15.7:37408) 2023-03-03T11:42:15.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488effd700 1 -- 172.21.15.7:0/3684934334 learned_addr learned my addr 172.21.15.7:0/3684934334 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:42:15.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488f7fe700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4890073c30 0x7f48901562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:15.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488ffff700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4890151d40 0x7f4890145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:15.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488effd700 1 -- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4890073c30 msgr2=0x7f48901562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:15.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488effd700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4890073c30 0x7f48901562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:15.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488effd700 1 -- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4890151d40 msgr2=0x7f4890145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:15.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488effd700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4890151d40 0x7f4890145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:15.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488effd700 1 -- 172.21.15.7:0/3684934334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48780094a0 con 0x7f48900c2930 2023-03-03T11:42:15.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488ffff700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4890151d40 0x7f4890145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:42:15.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.297+0000 7f488effd700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48900c2930 0x7f48900c6b10 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f488000c4d0 tx=0x7f488000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:42:15.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.298+0000 7f4887fff700 1 -- 172.21.15.7:0/3684934334 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880011790 con 0x7f48900c2930 2023-03-03T11:42:15.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.298+0000 7f4887fff700 1 -- 172.21.15.7:0/3684934334 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48800118f0 con 0x7f48900c2930 2023-03-03T11:42:15.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.298+0000 7f4887fff700 1 -- 172.21.15.7:0/3684934334 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880011ca0 con 0x7f48900c2930 2023-03-03T11:42:15.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.298+0000 7f4896102700 1 -- 172.21.15.7:0/3684934334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489006bea0 con 0x7f48900c2930 2023-03-03T11:42:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.298+0000 7f4896102700 1 -- 172.21.15.7:0/3684934334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489006d0c0 con 0x7f48900c2930 2023-03-03T11:42:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.299+0000 7f4896102700 1 -- 172.21.15.7:0/3684934334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489006c030 con 0x7f48900c2930 2023-03-03T11:42:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.299+0000 7f4887fff700 1 -- 172.21.15.7:0/3684934334 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4880010050 con 0x7f48900c2930 2023-03-03T11:42:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.299+0000 7f4887fff700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48700682f0 0x7f487006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.300+0000 7f4887fff700 1 -- 172.21.15.7:0/3684934334 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4880085d90 con 0x7f48900c2930 2023-03-03T11:42:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.300+0000 7f488f7fe700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48700682f0 0x7f487006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.300+0000 7f488f7fe700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48700682f0 0x7f487006a7a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4878009fd0 tx=0x7f4878008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:42:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.304+0000 7f4887fff700 1 -- 172.21.15.7:0/3684934334 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4880052130 con 0x7f48900c2930 2023-03-03T11:42:15.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.540+0000 7f4896102700 1 -- 172.21.15.7:0/3684934334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48900c66b0 con 0x7f48900c2930 2023-03-03T11:42:15.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.541+0000 7f4887fff700 1 -- 172.21.15.7:0/3684934334 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4880055750 con 0x7f48900c2930 2023-03-03T11:42:15.543 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:42:15.543 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:42:15.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 -- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48700682f0 msgr2=0x7f487006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:15.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48700682f0 0x7f487006a7a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4878009fd0 tx=0x7f4878008030 comp rx=0 tx=0).stop 2023-03-03T11:42:15.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 -- 172.21.15.7:0/3684934334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48900c2930 msgr2=0x7f48900c6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:15.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48900c2930 0x7f48900c6b10 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f488000c4d0 tx=0x7f488000c7e0 comp rx=0 tx=0).stop 2023-03-03T11:42:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 -- 172.21.15.7:0/3684934334 shutdown_connections 2023-03-03T11:42:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4890151d40 0x7f4890145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4890073c30 0x7f48901562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48700682f0 0x7f487006a7a0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 --2- 172.21.15.7:0/3684934334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48900c2930 0x7f48900c6b10 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 -- 172.21.15.7:0/3684934334 >> 172.21.15.7:0/3684934334 conn(0x7f48900a94d0 msgr2=0x7f48900c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:42:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 -- 172.21.15.7:0/3684934334 shutdown_connections 2023-03-03T11:42:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:15.543+0000 7f4896102700 1 -- 172.21.15.7:0/3684934334 wait complete. 2023-03-03T11:42:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:42:15.557 INFO:tasks.mds_thrash.fs.[b]:waiting for 93 secs before thrashing 2023-03-03T11:42:17.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:17.485 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:47.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:47.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:42:56.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:42:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.894+0000 7f1e99500700 1 Processor -- start 2023-03-03T11:42:56.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.894+0000 7f1e99500700 1 -- start start 2023-03-03T11:42:56.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.895+0000 7f1e99500700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e94151cd0 0x7f1e94154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:56.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.895+0000 7f1e99500700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1e9406c6c0 0x7f1e940736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:56.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.895+0000 7f1e99500700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e94157ff0 con 0x7f1e940c2e30 2023-03-03T11:42:56.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.895+0000 7f1e99500700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9406d640 con 0x7f1e9406c6c0 2023-03-03T11:42:56.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.895+0000 7f1e99500700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9406cda0 con 0x7f1e94151cd0 2023-03-03T11:42:56.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.895+0000 7f1e927fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1e940c2e30 0x7f1e94158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59676/0 (socket says 172.21.15.7:59676) 2023-03-03T11:42:56.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.895+0000 7f1e927fc700 1 -- 172.21.15.7:0/2610610632 learned_addr learned my addr 172.21.15.7:0/2610610632 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:42:56.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053755689 0 0) 0x7f1e94157ff0 con 0x7f1e940c2e30 2023-03-03T11:42:56.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e92ffd700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e94151cd0 0x7f1e94154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:56.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e937fe700 1 --2- 172.21.15.7:0/2610610632 >> v2:172.21.15.143:3300/0 conn(0x7f1e9406c6c0 0x7f1e940736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:56.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e7c0035b0 con 0x7f1e940c2e30 2023-03-03T11:42:56.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948364382 0 0) 0x7f1e7c0035b0 con 0x7f1e940c2e30 2023-03-03T11:42:56.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e94151cd0 msgr2=0x7f1e94154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:42:56.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e94151cd0 0x7f1e94154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 >> v2:172.21.15.143:3300/0 conn(0x7f1e9406c6c0 msgr2=0x7f1e940736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:42:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 --2- 172.21.15.7:0/2610610632 >> v2:172.21.15.143:3300/0 conn(0x7f1e9406c6c0 0x7f1e940736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9406d080 con 0x7f1e940c2e30 2023-03-03T11:42:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f1e84001670 con 0x7f1e940c2e30 2023-03-03T11:42:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 >> v1:172.21.15.7:6789/0 conn(0x7f1e940c2e30 legacy=0x7f1e94158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 0x7f1e7c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e7c004850 0x7f1e7c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:56.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e7c005200 0x7f1e7c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:56.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9406d640 con 0x7f1e7c003a70 2023-03-03T11:42:56.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9406cda0 con 0x7f1e7c004850 2023-03-03T11:42:56.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c00c110 con 0x7f1e7c005200 2023-03-03T11:42:56.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e92ffd700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e94151cd0 0x7f1e94154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:42:56.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e92ffd700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 0x7f1e7c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:56.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e927fc700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e7c004850 0x7f1e7c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:56.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.896+0000 7f1e937fe700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e7c005200 0x7f1e7c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:56.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e92ffd700 1 -- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e7c005200 msgr2=0x7f1e7c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:42:56.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e92ffd700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e7c005200 0x7f1e7c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:56.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e92ffd700 1 -- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e7c004850 msgr2=0x7f1e7c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:42:56.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e92ffd700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e7c004850 0x7f1e7c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:56.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e92ffd700 1 -- 172.21.15.7:0/2610610632 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c0035b0 con 0x7f1e7c003a70 2023-03-03T11:42:56.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e92ffd700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 0x7f1e7c003e80 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f1e94074470 tx=0x7f1e7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=badd5cbcca745280 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:42:56.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8800c070 con 0x7f1e7c003a70 2023-03-03T11:42:56.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e88013630 con 0x7f1e7c003a70 2023-03-03T11:42:56.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e91ffb700 1 -- 172.21.15.7:0/2610610632 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e880093d0 con 0x7f1e7c003a70 2023-03-03T11:42:56.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e99500700 1 -- 172.21.15.7:0/2610610632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 msgr2=0x7f1e7c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:56.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e99500700 1 --2- 172.21.15.7:0/2610610632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 0x7f1e7c003e80 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f1e94074470 tx=0x7f1e7c005880 comp rx=0 tx=0).stop 2023-03-03T11:42:56.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e937fe700 1 -- 172.21.15.7:0/2610610632 reap_dead start 2023-03-03T11:42:56.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e99500700 1 -- 172.21.15.7:0/2610610632 shutdown_connections 2023-03-03T11:42:56.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.897+0000 7f1e99500700 1 -- 172.21.15.7:0/2610610632 >> 172.21.15.7:0/2610610632 conn(0x7f1e940a94d0 msgr2=0x7f1e940a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:42:56.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.898+0000 7f1e99500700 1 -- 172.21.15.7:0/2610610632 shutdown_connections 2023-03-03T11:42:56.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.898+0000 7f1e99500700 1 -- 172.21.15.7:0/2610610632 wait complete. 2023-03-03T11:42:56.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.899+0000 7f1e99500700 1 Processor -- start 2023-03-03T11:42:56.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.899+0000 7f1e99500700 1 -- start start 2023-03-03T11:42:56.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e99500700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 0x7f1e9414bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:56.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e99500700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e941e0140 0x7f1e9414f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:56.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e99500700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e941edfd0 0x7f1e941dee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:56.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e99500700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e941edb40 con 0x7f1e7c003a70 2023-03-03T11:42:56.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e99500700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e941e0530 con 0x7f1e941e0140 2023-03-03T11:42:56.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e99500700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9414b6e0 con 0x7f1e941edfd0 2023-03-03T11:42:56.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e92ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 0x7f1e9414bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:56.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e92ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 0x7f1e9414bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37476/0 (socket says 172.21.15.7:37476) 2023-03-03T11:42:56.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e92ffd700 1 -- 172.21.15.7:0/230509264 learned_addr learned my addr 172.21.15.7:0/230509264 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:42:56.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e927fc700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e941e0140 0x7f1e9414f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:56.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e937fe700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e941edfd0 0x7f1e941dee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:56.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e92ffd700 1 -- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e941edfd0 msgr2=0x7f1e941dee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:56.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e92ffd700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e941edfd0 0x7f1e941dee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:56.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e92ffd700 1 -- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e941e0140 msgr2=0x7f1e9414f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:56.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e92ffd700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e941e0140 0x7f1e9414f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:56.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e92ffd700 1 -- 172.21.15.7:0/230509264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e94157c80 con 0x7f1e7c003a70 2023-03-03T11:42:56.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e937fe700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e941edfd0 0x7f1e941dee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:42:56.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e92ffd700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 0x7f1e9414bc70 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00c060 tx=0x7f1e880047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:42:56.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e7b7fe700 1 -- 172.21.15.7:0/230509264 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8801d810 con 0x7f1e7c003a70 2023-03-03T11:42:56.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e7b7fe700 1 -- 172.21.15.7:0/230509264 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e8801d970 con 0x7f1e7c003a70 2023-03-03T11:42:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e99500700 1 -- 172.21.15.7:0/230509264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9406d360 con 0x7f1e7c003a70 2023-03-03T11:42:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e7b7fe700 1 -- 172.21.15.7:0/230509264 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8801dd20 con 0x7f1e7c003a70 2023-03-03T11:42:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.900+0000 7f1e99500700 1 -- 172.21.15.7:0/230509264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e94073c30 con 0x7f1e7c003a70 2023-03-03T11:42:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.902+0000 7f1e99500700 1 -- 172.21.15.7:0/230509264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9406a850 con 0x7f1e7c003a70 2023-03-03T11:42:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.902+0000 7f1e7b7fe700 1 -- 172.21.15.7:0/230509264 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1e8802f070 con 0x7f1e7c003a70 2023-03-03T11:42:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.902+0000 7f1e7b7fe700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e800682f0 0x7f1e8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:42:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.902+0000 7f1e7b7fe700 1 -- 172.21.15.7:0/230509264 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1e88023030 con 0x7f1e7c003a70 2023-03-03T11:42:56.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.902+0000 7f1e927fc700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e800682f0 0x7f1e8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:42:56.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.903+0000 7f1e927fc700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e800682f0 0x7f1e8006a7a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f1e84001120 tx=0x7f1e84007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:42:56.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:56.907+0000 7f1e7b7fe700 1 -- 172.21.15.7:0/230509264 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1e88053830 con 0x7f1e7c003a70 2023-03-03T11:42:57.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.146+0000 7f1e99500700 1 -- 172.21.15.7:0/230509264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e941ed9b0 con 0x7f1e7c003a70 2023-03-03T11:42:57.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.147+0000 7f1e7b7fe700 1 -- 172.21.15.7:0/230509264 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1e88056e50 con 0x7f1e7c003a70 2023-03-03T11:42:57.148 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:42:57.148 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:42:57.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 -- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e800682f0 msgr2=0x7f1e8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:57.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e800682f0 0x7f1e8006a7a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f1e84001120 tx=0x7f1e84007980 comp rx=0 tx=0).stop 2023-03-03T11:42:57.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 -- 172.21.15.7:0/230509264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 msgr2=0x7f1e9414bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:42:57.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 0x7f1e9414bc70 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00c060 tx=0x7f1e880047c0 comp rx=0 tx=0).stop 2023-03-03T11:42:57.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 -- 172.21.15.7:0/230509264 shutdown_connections 2023-03-03T11:42:57.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1e941edfd0 0x7f1e941dee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:57.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e800682f0 0x7f1e8006a7a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:57.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e7c003a70 0x7f1e9414bc70 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:57.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 --2- 172.21.15.7:0/230509264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1e941e0140 0x7f1e9414f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:42:57.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 -- 172.21.15.7:0/230509264 >> 172.21.15.7:0/230509264 conn(0x7f1e940a94d0 msgr2=0x7f1e9406dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:42:57.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 -- 172.21.15.7:0/230509264 shutdown_connections 2023-03-03T11:42:57.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:42:57.149+0000 7f1e99500700 1 -- 172.21.15.7:0/230509264 wait complete. 2023-03-03T11:42:57.152 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:42:57.162 INFO:tasks.mds_thrash.fs.[a]:waiting for 116 secs before thrashing 2023-03-03T11:43:17.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:17.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:47.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:47.818 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:43:48.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:43:48.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.695+0000 7fdbd8f22700 1 Processor -- start 2023-03-03T11:43:48.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.695+0000 7fdbd8f22700 1 -- start start 2023-03-03T11:43:48.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.696+0000 7fdbd8f22700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbd414f9d0 0x7fdbd4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:48.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.696+0000 7fdbd8f22700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40c2f30 con 0x7fdbd414fda0 2023-03-03T11:43:48.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.696+0000 7fdbd8f22700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40c31d0 con 0x7fdbd40abe20 2023-03-03T11:43:48.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.696+0000 7fdbd8f22700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd41524a0 con 0x7fdbd414f9d0 2023-03-03T11:43:48.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.696+0000 7fdbd1d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdbd414fda0 0x7fdbd4071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59744/0 (socket says 172.21.15.7:59744) 2023-03-03T11:43:48.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.696+0000 7fdbd1d9b700 1 -- 172.21.15.7:0/3916756594 learned_addr learned my addr 172.21.15.7:0/3916756594 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:43:48.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd259c700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbd414f9d0 0x7fdbd4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:48.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197258864 0 0) 0x7fdbd40c2f30 con 0x7fdbd414fda0 2023-03-03T11:43:48.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc00035b0 con 0x7fdbd414fda0 2023-03-03T11:43:48.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445475564 0 0) 0x7fdbc00035b0 con 0x7fdbd414fda0 2023-03-03T11:43:48.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbd414f9d0 msgr2=0x7fdbd4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:43:48.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbd414f9d0 0x7fdbd4156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:48.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 >> v1:172.21.15.143:6789/0 conn(0x7fdbd40abe20 legacy=0x7fdbd41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:48.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd40c34b0 con 0x7fdbd414fda0 2023-03-03T11:43:48.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fdbbc001670 con 0x7fdbd414fda0 2023-03-03T11:43:48.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 >> v1:172.21.15.7:6789/0 conn(0x7fdbd414fda0 legacy=0x7fdbd4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:48.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbc0003aa0 0x7fdbc0003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:48.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbd414f9d0 0x7fdbc0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:48.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbc0004d90 0x7fdbc0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:48.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd41524a0 con 0x7fdbc0004d90 2023-03-03T11:43:48.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc000bc90 con 0x7fdbc0003aa0 2023-03-03T11:43:48.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc000be50 con 0x7fdbd414f9d0 2023-03-03T11:43:48.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd2d9d700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbc0004d90 0x7fdbc0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:48.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd1d9b700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbd414f9d0 0x7fdbc0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:48.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.697+0000 7fdbd259c700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbc0003aa0 0x7fdbc0003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:48.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd2d9d700 1 -- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbd414f9d0 msgr2=0x7fdbc0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:43:48.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd2d9d700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbd414f9d0 0x7fdbc0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:48.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd2d9d700 1 -- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbc0003aa0 msgr2=0x7fdbc0003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:43:48.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd2d9d700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbc0003aa0 0x7fdbc0003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:48.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd2d9d700 1 -- 172.21.15.7:0/3916756594 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00035b0 con 0x7fdbc0004d90 2023-03-03T11:43:48.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd2d9d700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbc0004d90 0x7fdbc0005200 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00c190 tx=0x7fdbc0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54340c3cf9501d9a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:48.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc001db0 con 0x7fdbc0004d90 2023-03-03T11:43:48.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbcc00ccd0 con 0x7fdbc0004d90 2023-03-03T11:43:48.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd159a700 1 -- 172.21.15.7:0/3916756594 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc003c40 con 0x7fdbc0004d90 2023-03-03T11:43:48.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd8f22700 1 -- 172.21.15.7:0/3916756594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbc0004d90 msgr2=0x7fdbc0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:48.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd8f22700 1 --2- 172.21.15.7:0/3916756594 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbc0004d90 0x7fdbc0005200 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00c190 tx=0x7fdbc0005880 comp rx=0 tx=0).stop 2023-03-03T11:43:48.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd2d9d700 1 -- 172.21.15.7:0/3916756594 reap_dead start 2023-03-03T11:43:48.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd8f22700 1 -- 172.21.15.7:0/3916756594 shutdown_connections 2023-03-03T11:43:48.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.698+0000 7fdbd8f22700 1 -- 172.21.15.7:0/3916756594 >> 172.21.15.7:0/3916756594 conn(0x7fdbd40c8900 msgr2=0x7fdbd40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:43:48.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.699+0000 7fdbd8f22700 1 -- 172.21.15.7:0/3916756594 shutdown_connections 2023-03-03T11:43:48.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.699+0000 7fdbd8f22700 1 -- 172.21.15.7:0/3916756594 wait complete. 2023-03-03T11:43:48.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.700+0000 7fdbd8f22700 1 Processor -- start 2023-03-03T11:43:48.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.700+0000 7fdbd8f22700 1 -- start start 2023-03-03T11:43:48.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd8f22700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbc0004d90 0x7fdbd4150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:48.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd8f22700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbd4151a80 0x7fdbd41550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:48.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd8f22700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd41e8750 0x7fdbd4155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:48.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd8f22700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd406cf20 con 0x7fdbd41e8750 2023-03-03T11:43:48.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd8f22700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd41e8c50 con 0x7fdbd4151a80 2023-03-03T11:43:48.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd8f22700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd414b540 con 0x7fdbc0004d90 2023-03-03T11:43:48.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd41e8750 0x7fdbd4155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:48.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd41e8750 0x7fdbd4155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37546/0 (socket says 172.21.15.7:37546) 2023-03-03T11:43:48.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd2d9d700 1 -- 172.21.15.7:0/2413875493 learned_addr learned my addr 172.21.15.7:0/2413875493 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:43:48.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd1d9b700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbd4151a80 0x7fdbd41550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:48.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd259c700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbc0004d90 0x7fdbd4150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:48.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd2d9d700 1 -- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbc0004d90 msgr2=0x7fdbd4150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:48.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd2d9d700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbc0004d90 0x7fdbd4150520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:48.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd2d9d700 1 -- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbd4151a80 msgr2=0x7fdbd41550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:48.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd2d9d700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbd4151a80 0x7fdbd41550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:48.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd2d9d700 1 -- 172.21.15.7:0/2413875493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd40ac350 con 0x7fdbd41e8750 2023-03-03T11:43:48.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbd2d9d700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd41e8750 0x7fdbd4155600 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc004000 tx=0x7fdbcc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:48.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbcaffd700 1 -- 172.21.15.7:0/2413875493 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc0050d0 con 0x7fdbd41e8750 2023-03-03T11:43:48.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbcaffd700 1 -- 172.21.15.7:0/2413875493 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbcc005250 con 0x7fdbd41e8750 2023-03-03T11:43:48.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.701+0000 7fdbcaffd700 1 -- 172.21.15.7:0/2413875493 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc005400 con 0x7fdbd41e8750 2023-03-03T11:43:48.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.702+0000 7fdbd8f22700 1 -- 172.21.15.7:0/2413875493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd40ab630 con 0x7fdbd41e8750 2023-03-03T11:43:48.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.702+0000 7fdbd8f22700 1 -- 172.21.15.7:0/2413875493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd4071fa0 con 0x7fdbd41e8750 2023-03-03T11:43:48.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.702+0000 7fdbd8f22700 1 -- 172.21.15.7:0/2413875493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd40c5e50 con 0x7fdbd41e8750 2023-03-03T11:43:48.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.703+0000 7fdbcaffd700 1 -- 172.21.15.7:0/2413875493 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdbcc010050 con 0x7fdbd41e8750 2023-03-03T11:43:48.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.703+0000 7fdbcaffd700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbb40682f0 0x7fdbb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:43:48.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.703+0000 7fdbcaffd700 1 -- 172.21.15.7:0/2413875493 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdbcc087680 con 0x7fdbd41e8750 2023-03-03T11:43:48.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.703+0000 7fdbd259c700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbb40682f0 0x7fdbb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:43:48.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.704+0000 7fdbd259c700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbb40682f0 0x7fdbb406a7a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009c60 tx=0x7fdbc4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:43:48.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.708+0000 7fdbcaffd700 1 -- 172.21.15.7:0/2413875493 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdbcc053ae0 con 0x7fdbd41e8750 2023-03-03T11:43:48.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.945+0000 7fdbd8f22700 1 -- 172.21.15.7:0/2413875493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbd40acf80 con 0x7fdbd41e8750 2023-03-03T11:43:48.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.945+0000 7fdbcaffd700 1 -- 172.21.15.7:0/2413875493 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdbcc057100 con 0x7fdbd41e8750 2023-03-03T11:43:48.947 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:43:48.948 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:43:48.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 -- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbb40682f0 msgr2=0x7fdbb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:48.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbb40682f0 0x7fdbb406a7a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009c60 tx=0x7fdbc4009210 comp rx=0 tx=0).stop 2023-03-03T11:43:48.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 -- 172.21.15.7:0/2413875493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd41e8750 msgr2=0x7fdbd4155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:43:48.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd41e8750 0x7fdbd4155600 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc004000 tx=0x7fdbcc0014a0 comp rx=0 tx=0).stop 2023-03-03T11:43:48.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 -- 172.21.15.7:0/2413875493 shutdown_connections 2023-03-03T11:43:48.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbb40682f0 0x7fdbb406a7a0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:48.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbd41e8750 0x7fdbd4155600 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:48.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbc0004d90 0x7fdbd4150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:48.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 --2- 172.21.15.7:0/2413875493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbd4151a80 0x7fdbd41550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:43:48.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 -- 172.21.15.7:0/2413875493 >> 172.21.15.7:0/2413875493 conn(0x7fdbd40c8900 msgr2=0x7fdbd41528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:43:48.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 -- 172.21.15.7:0/2413875493 shutdown_connections 2023-03-03T11:43:48.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:43:48.947+0000 7fdbd8f22700 1 -- 172.21.15.7:0/2413875493 wait complete. 2023-03-03T11:43:48.951 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:43:48.961 INFO:tasks.mds_thrash.fs.[b]:waiting for 79 secs before thrashing 2023-03-03T11:44:17.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:17.931 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:48.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:48.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:44:53.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:44:53.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.300+0000 7fcb6d1d7700 1 Processor -- start 2023-03-03T11:44:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.300+0000 7fcb6d1d7700 1 -- start start 2023-03-03T11:44:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6d1d7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcb68155410 0x7fcb680bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6d1d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 0x7fcb680bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6d1d7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb680c08d0 con 0x7fcb68151e80 2023-03-03T11:44:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6d1d7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb68152250 con 0x7fcb68155410 2023-03-03T11:44:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6d1d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb680c0fb0 con 0x7fcb680c5b00 2023-03-03T11:44:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcb68155410 0x7fcb680bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcb68155410 0x7fcb680bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37602/0 (socket says 172.21.15.7:37602) 2023-03-03T11:44:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6659c700 1 -- 172.21.15.7:0/585769998 learned_addr learned my addr 172.21.15.7:0/585769998 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:44:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6759e700 1 --2- 172.21.15.7:0/585769998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 0x7fcb680bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6659c700 1 -- 172.21.15.7:0/585769998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 msgr2=0x7fcb680bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:44:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6659c700 1 --2- 172.21.15.7:0/585769998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 0x7fcb680bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:53.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6659c700 1 -- 172.21.15.7:0/585769998 >> v1:172.21.15.143:6789/0 conn(0x7fcb68151e80 legacy=0x7fcb680c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:53.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6659c700 1 -- 172.21.15.7:0/585769998 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb68154930 con 0x7fcb68155410 2023-03-03T11:44:53.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb6659c700 1 --2- 172.21.15.7:0/585769998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 0x7fcb680bf8e0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0098f0 tx=0x7fcb5c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff6fcd1a0c5f3bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:44:53.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb65d9b700 1 -- 172.21.15.7:0/585769998 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c00b8a0 con 0x7fcb68155410 2023-03-03T11:44:53.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb65d9b700 1 -- 172.21.15.7:0/585769998 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c00ba00 con 0x7fcb68155410 2023-03-03T11:44:53.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.301+0000 7fcb65d9b700 1 -- 172.21.15.7:0/585769998 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c00bd90 con 0x7fcb68155410 2023-03-03T11:44:53.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.302+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/585769998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 msgr2=0x7fcb680bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:53.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.302+0000 7fcb6d1d7700 1 --2- 172.21.15.7:0/585769998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 0x7fcb680bf8e0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0098f0 tx=0x7fcb5c009c00 comp rx=0 tx=0).stop 2023-03-03T11:44:53.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.302+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/585769998 shutdown_connections 2023-03-03T11:44:53.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.302+0000 7fcb6d1d7700 1 --2- 172.21.15.7:0/585769998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 0x7fcb680bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:53.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.302+0000 7fcb6d1d7700 1 --2- 172.21.15.7:0/585769998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 0x7fcb680bf8e0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:53.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.302+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/585769998 >> 172.21.15.7:0/585769998 conn(0x7fcb68071630 msgr2=0x7fcb6806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:44:53.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.302+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/585769998 shutdown_connections 2023-03-03T11:44:53.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.302+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/585769998 wait complete. 2023-03-03T11:44:53.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6d1d7700 1 Processor -- start 2023-03-03T11:44:53.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6d1d7700 1 -- start start 2023-03-03T11:44:53.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6d1d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 0x7fcb680bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:53.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6d1d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68151e80 0x7fcb681ea930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:53.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6d1d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 0x7fcb681eae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:53.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6d1d7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6814dc30 con 0x7fcb68155410 2023-03-03T11:44:53.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6d1d7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6814f960 con 0x7fcb68151e80 2023-03-03T11:44:53.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6d1d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6814f680 con 0x7fcb680c5b00 2023-03-03T11:44:53.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6759e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 0x7fcb681eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:53.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6759e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 0x7fcb681eae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37610/0 (socket says 172.21.15.7:37610) 2023-03-03T11:44:53.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.304+0000 7fcb6759e700 1 -- 172.21.15.7:0/3253267542 learned_addr learned my addr 172.21.15.7:0/3253267542 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:44:53.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb66d9d700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 0x7fcb680bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:53.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb6759e700 1 -- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 msgr2=0x7fcb680bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:53.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb6759e700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 0x7fcb680bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:53.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb6659c700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68151e80 0x7fcb681ea930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:53.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb6759e700 1 -- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68151e80 msgr2=0x7fcb681ea930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:53.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb6759e700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68151e80 0x7fcb681ea930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:53.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb6759e700 1 -- 172.21.15.7:0/3253267542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c009550 con 0x7fcb68155410 2023-03-03T11:44:53.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb6659c700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68151e80 0x7fcb681ea930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:44:53.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb66d9d700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 0x7fcb680bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:44:53.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb6759e700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 0x7fcb681eae70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fcb60007e60 tx=0x7fcb6000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:44:53.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb4f7fe700 1 -- 172.21.15.7:0/3253267542 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6000ede0 con 0x7fcb68155410 2023-03-03T11:44:53.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb4f7fe700 1 -- 172.21.15.7:0/3253267542 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb60014420 con 0x7fcb68155410 2023-03-03T11:44:53.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/3253267542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb68154ef0 con 0x7fcb68155410 2023-03-03T11:44:53.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb4f7fe700 1 -- 172.21.15.7:0/3253267542 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb600147d0 con 0x7fcb68155410 2023-03-03T11:44:53.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.305+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/3253267542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb68154c10 con 0x7fcb68155410 2023-03-03T11:44:53.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.306+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/3253267542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb680c5220 con 0x7fcb68155410 2023-03-03T11:44:53.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.307+0000 7fcb4f7fe700 1 -- 172.21.15.7:0/3253267542 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcb60012050 con 0x7fcb68155410 2023-03-03T11:44:53.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.307+0000 7fcb4f7fe700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb540682f0 0x7fcb5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:44:53.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.307+0000 7fcb4f7fe700 1 -- 172.21.15.7:0/3253267542 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcb60086840 con 0x7fcb68155410 2023-03-03T11:44:53.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.307+0000 7fcb66d9d700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb540682f0 0x7fcb5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:44:53.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.308+0000 7fcb66d9d700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb540682f0 0x7fcb5406a7a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcb58000f60 tx=0x7fcb58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:44:53.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.311+0000 7fcb4f7fe700 1 -- 172.21.15.7:0/3253267542 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcb60052b60 con 0x7fcb68155410 2023-03-03T11:44:53.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.549+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/3253267542 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb68074bb0 con 0x7fcb68155410 2023-03-03T11:44:53.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.549+0000 7fcb4f7fe700 1 -- 172.21.15.7:0/3253267542 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcb60056180 con 0x7fcb68155410 2023-03-03T11:44:53.551 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:44:53.551 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:44:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb540682f0 msgr2=0x7fcb5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb540682f0 0x7fcb5406a7a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcb58000f60 tx=0x7fcb58009040 comp rx=0 tx=0).stop 2023-03-03T11:44:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/3253267542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 msgr2=0x7fcb681eae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:44:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 0x7fcb681eae70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fcb60007e60 tx=0x7fcb6000e3c0 comp rx=0 tx=0).stop 2023-03-03T11:44:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/3253267542 shutdown_connections 2023-03-03T11:44:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb540682f0 0x7fcb5406a7a0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb68155410 0x7fcb681eae70 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb680c5b00 0x7fcb680bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 --2- 172.21.15.7:0/3253267542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68151e80 0x7fcb681ea930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:44:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/3253267542 >> 172.21.15.7:0/3253267542 conn(0x7fcb68071630 msgr2=0x7fcb68152e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:44:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/3253267542 shutdown_connections 2023-03-03T11:44:53.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:44:53.551+0000 7fcb6d1d7700 1 -- 172.21.15.7:0/3253267542 wait complete. 2023-03-03T11:44:53.555 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:44:53.564 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T11:45:07.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:45:08.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.098+0000 7f825d12e700 1 Processor -- start 2023-03-03T11:45:08.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.099+0000 7f825d12e700 1 -- start start 2023-03-03T11:45:08.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.099+0000 7f825d12e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82580aad20 0x7f8258157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:08.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.099+0000 7f825d12e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82580aa030 0x7f82581581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:08.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.099+0000 7f825d12e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82580728b0 con 0x7f825814b020 2023-03-03T11:45:08.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.099+0000 7f825d12e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8258155070 con 0x7f82580aa030 2023-03-03T11:45:08.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.099+0000 7f825d12e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825806f5b0 con 0x7f82580aad20 2023-03-03T11:45:08.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.099+0000 7f8256d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f825814b020 0x7f82580717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59842/0 (socket says 172.21.15.7:59842) 2023-03-03T11:45:08.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.099+0000 7f8256d9d700 1 -- 172.21.15.7:0/3302545467 learned_addr learned my addr 172.21.15.7:0/3302545467 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:45:08.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f825759e700 1 --2- 172.21.15.7:0/3302545467 >> v2:172.21.15.143:3300/0 conn(0x7f82580aa030 0x7f82581581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:08.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f825659c700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82580aad20 0x7f8258157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:08.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041041152 0 0) 0x7f82580728b0 con 0x7f825814b020 2023-03-03T11:45:08.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82440035b0 con 0x7f825814b020 2023-03-03T11:45:08.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2431436908 0 0) 0x7f82440035b0 con 0x7f825814b020 2023-03-03T11:45:08.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82580aad20 msgr2=0x7f8258157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:45:08.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82580aad20 0x7f8258157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:08.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 >> v2:172.21.15.143:3300/0 conn(0x7f82580aa030 msgr2=0x7f82581581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:45:08.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 --2- 172.21.15.7:0/3302545467 >> v2:172.21.15.143:3300/0 conn(0x7f82580aa030 0x7f82581581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:08.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825806f890 con 0x7f825814b020 2023-03-03T11:45:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8248001670 con 0x7f825814b020 2023-03-03T11:45:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 >> v1:172.21.15.7:6789/0 conn(0x7f825814b020 legacy=0x7f82580717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244003a70 0x7f8244003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8244004850 0x7f8244004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8244005200 0x7f824400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8258155070 con 0x7f8244005200 2023-03-03T11:45:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825806f5b0 con 0x7f8244004850 2023-03-03T11:45:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824400bfd0 con 0x7f8244003a70 2023-03-03T11:45:08.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f825659c700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82580aad20 0x7f8258157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:45:08.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f825759e700 1 --2- 172.21.15.7:0/3302545467 >> v2:172.21.15.143:3300/0 conn(0x7f82580aa030 0x7f82581581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:45:08.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f8256d9d700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244003a70 0x7f8244003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:08.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f825759e700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8244005200 0x7f824400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:08.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.100+0000 7f825659c700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8244004850 0x7f8244004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:08.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825759e700 1 -- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244003a70 msgr2=0x7f8244003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:45:08.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825759e700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244003a70 0x7f8244003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:08.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825759e700 1 -- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8244004850 msgr2=0x7f8244004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:45:08.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825759e700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8244004850 0x7f8244004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:08.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825759e700 1 -- 172.21.15.7:0/3302545467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82440035b0 con 0x7f8244005200 2023-03-03T11:45:08.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825659c700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8244004850 0x7f8244004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:45:08.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f8256d9d700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244003a70 0x7f8244003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:45:08.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825759e700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8244005200 0x7f824400ba10 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f825000e350 tx=0x7f8244005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c788025c4a6571 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:45:08.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825000ee10 con 0x7f8244005200 2023-03-03T11:45:08.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825000f040 con 0x7f8244005200 2023-03-03T11:45:08.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f8255d9b700 1 -- 172.21.15.7:0/3302545467 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8250003c40 con 0x7f8244005200 2023-03-03T11:45:08.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825d12e700 1 -- 172.21.15.7:0/3302545467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8244005200 msgr2=0x7f824400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:08.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825d12e700 1 --2- 172.21.15.7:0/3302545467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8244005200 0x7f824400ba10 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f825000e350 tx=0x7f8244005880 comp rx=0 tx=0).stop 2023-03-03T11:45:08.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825759e700 1 -- 172.21.15.7:0/3302545467 reap_dead start 2023-03-03T11:45:08.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825d12e700 1 -- 172.21.15.7:0/3302545467 shutdown_connections 2023-03-03T11:45:08.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.101+0000 7f825d12e700 1 -- 172.21.15.7:0/3302545467 >> 172.21.15.7:0/3302545467 conn(0x7f82580c8900 msgr2=0x7f82580c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:45:08.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.102+0000 7f825d12e700 1 -- 172.21.15.7:0/3302545467 shutdown_connections 2023-03-03T11:45:08.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.102+0000 7f825d12e700 1 -- 172.21.15.7:0/3302545467 wait complete. 2023-03-03T11:45:08.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.103+0000 7f825d12e700 1 Processor -- start 2023-03-03T11:45:08.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.103+0000 7f825d12e700 1 -- start start 2023-03-03T11:45:08.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825d12e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244005200 0x7f82581ed610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:08.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825d12e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258151b80 0x7f825814f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:08.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825d12e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258152b80 0x7f825814b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:08.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825d12e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825814a8b0 con 0x7f8258151b80 2023-03-03T11:45:08.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825d12e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825814fb50 con 0x7f8258152b80 2023-03-03T11:45:08.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825d12e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825814f870 con 0x7f8244005200 2023-03-03T11:45:08.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258151b80 0x7f825814f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:08.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258151b80 0x7f825814f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37644/0 (socket says 172.21.15.7:37644) 2023-03-03T11:45:08.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825659c700 1 -- 172.21.15.7:0/740072109 learned_addr learned my addr 172.21.15.7:0/740072109 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:45:08.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f8256d9d700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244005200 0x7f82581ed610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:08.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825759e700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258152b80 0x7f825814b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:08.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825659c700 1 -- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244005200 msgr2=0x7f82581ed610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:08.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825659c700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244005200 0x7f82581ed610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:08.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825659c700 1 -- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258152b80 msgr2=0x7f825814b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:08.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825659c700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258152b80 0x7f825814b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:08.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825659c700 1 -- 172.21.15.7:0/740072109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8258155350 con 0x7f8258151b80 2023-03-03T11:45:08.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f8256d9d700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244005200 0x7f82581ed610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:45:08.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825759e700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258152b80 0x7f825814b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:45:08.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.104+0000 7f825659c700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258151b80 0x7f825814f200 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f82400118b0 tx=0x7f8240011c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:45:08.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.105+0000 7f824f7fe700 1 -- 172.21.15.7:0/740072109 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824000fae0 con 0x7f8258151b80 2023-03-03T11:45:08.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.105+0000 7f824f7fe700 1 -- 172.21.15.7:0/740072109 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824000fc40 con 0x7f8258151b80 2023-03-03T11:45:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.105+0000 7f825d12e700 1 -- 172.21.15.7:0/740072109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825806fb70 con 0x7f8258151b80 2023-03-03T11:45:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.105+0000 7f824f7fe700 1 -- 172.21.15.7:0/740072109 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8240011e50 con 0x7f8258151b80 2023-03-03T11:45:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.105+0000 7f825d12e700 1 -- 172.21.15.7:0/740072109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8258071ef0 con 0x7f8258151b80 2023-03-03T11:45:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.106+0000 7f825d12e700 1 -- 172.21.15.7:0/740072109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82581ed460 con 0x7f8258151b80 2023-03-03T11:45:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.107+0000 7f824f7fe700 1 -- 172.21.15.7:0/740072109 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8240012040 con 0x7f8258151b80 2023-03-03T11:45:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.107+0000 7f824f7fe700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82380682f0 0x7f823806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.107+0000 7f824f7fe700 1 -- 172.21.15.7:0/740072109 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8240086a40 con 0x7f8258151b80 2023-03-03T11:45:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.107+0000 7f8256d9d700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82380682f0 0x7f823806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.108+0000 7f8256d9d700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82380682f0 0x7f823806a7a0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f82480015d0 tx=0x7f8248009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:45:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.111+0000 7f824f7fe700 1 -- 172.21.15.7:0/740072109 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8240052e80 con 0x7f8258151b80 2023-03-03T11:45:08.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.350+0000 7f825d12e700 1 -- 172.21.15.7:0/740072109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8258074bb0 con 0x7f8258151b80 2023-03-03T11:45:08.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.350+0000 7f824f7fe700 1 -- 172.21.15.7:0/740072109 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f82400564a0 con 0x7f8258151b80 2023-03-03T11:45:08.352 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:45:08.352 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:45:08.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.352+0000 7f825d12e700 1 -- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82380682f0 msgr2=0x7f823806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:08.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.352+0000 7f825d12e700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82380682f0 0x7f823806a7a0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f82480015d0 tx=0x7f8248009040 comp rx=0 tx=0).stop 2023-03-03T11:45:08.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.352+0000 7f825d12e700 1 -- 172.21.15.7:0/740072109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258151b80 msgr2=0x7f825814f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.352+0000 7f825d12e700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258151b80 0x7f825814f200 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f82400118b0 tx=0x7f8240011c70 comp rx=0 tx=0).stop 2023-03-03T11:45:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.352+0000 7f825d12e700 1 -- 172.21.15.7:0/740072109 shutdown_connections 2023-03-03T11:45:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.352+0000 7f825d12e700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8258152b80 0x7f825814b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.352+0000 7f825d12e700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8244005200 0x7f82581ed610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.352+0000 7f825d12e700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82380682f0 0x7f823806a7a0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.352+0000 7f825d12e700 1 --2- 172.21.15.7:0/740072109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258151b80 0x7f825814f200 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.352+0000 7f825d12e700 1 -- 172.21.15.7:0/740072109 >> 172.21.15.7:0/740072109 conn(0x7f82580c8900 msgr2=0x7f82580c3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:45:08.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.353+0000 7f825d12e700 1 -- 172.21.15.7:0/740072109 shutdown_connections 2023-03-03T11:45:08.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:08.353+0000 7f825d12e700 1 -- 172.21.15.7:0/740072109 wait complete. 2023-03-03T11:45:08.356 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:45:08.366 INFO:tasks.mds_thrash.fs.[b]:waiting for 91 secs before thrashing 2023-03-03T11:45:18.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:18.199 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:48.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:48.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:45:58.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:45:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.704+0000 7f34bd1c9700 1 Processor -- start 2023-03-03T11:45:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.704+0000 7f34bd1c9700 1 -- start start 2023-03-03T11:45:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.704+0000 7f34bd1c9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f34b81530d0 0x7f34b8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34bd1c9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b80c2f30 con 0x7f34b814f9d0 2023-03-03T11:45:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34bd1c9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b80c31d0 con 0x7f34b814fda0 2023-03-03T11:45:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34bd1c9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b81524a0 con 0x7f34b81530d0 2023-03-03T11:45:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b6d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f34b814f9d0 0x7f34b8071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59898/0 (socket says 172.21.15.7:59898) 2023-03-03T11:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b6d9d700 1 -- 172.21.15.7:0/1385318340 learned_addr learned my addr 172.21.15.7:0/1385318340 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b759e700 1 --2- 172.21.15.7:0/1385318340 >> v2:172.21.15.143:3300/0 conn(0x7f34b81530d0 0x7f34b8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243882487 0 0) 0x7f34b80c2f30 con 0x7f34b814f9d0 2023-03-03T11:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a00035b0 con 0x7f34b814f9d0 2023-03-03T11:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2846459 0 0) 0x7f34a00035b0 con 0x7f34b814f9d0 2023-03-03T11:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 >> v2:172.21.15.143:3300/0 conn(0x7f34b81530d0 msgr2=0x7f34b8156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:45:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 --2- 172.21.15.7:0/1385318340 >> v2:172.21.15.143:3300/0 conn(0x7f34b81530d0 0x7f34b8156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 >> v1:172.21.15.143:6789/0 conn(0x7f34b814fda0 legacy=0x7f34b81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b80c34b0 con 0x7f34b814f9d0 2023-03-03T11:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f34a8001670 con 0x7f34b814f9d0 2023-03-03T11:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 >> v1:172.21.15.7:6789/0 conn(0x7f34b814f9d0 legacy=0x7f34b8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34a0003a70 0x7f34a0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34a0004850 0x7f34a0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a0005270 0x7f34a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b81524a0 con 0x7f34a0005270 2023-03-03T11:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a000c1b0 con 0x7f34a0003a70 2023-03-03T11:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.705+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a000c370 con 0x7f34a0004850 2023-03-03T11:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b759e700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a0005270 0x7f34a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b6d9d700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34a0003a70 0x7f34a0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b659c700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34a0004850 0x7f34a0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b759e700 1 -- 172.21.15.7:0/1385318340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34a0004850 msgr2=0x7f34a0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:45:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b759e700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34a0004850 0x7f34a0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b759e700 1 -- 172.21.15.7:0/1385318340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34a0003a70 msgr2=0x7f34a0003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b759e700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34a0003a70 0x7f34a0003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b759e700 1 -- 172.21.15.7:0/1385318340 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a00035b0 con 0x7f34a0005270 2023-03-03T11:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b6d9d700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34a0003a70 0x7f34a0003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b759e700 1 -- 172.21.15.7:0/1385318340 reap_dead start 2023-03-03T11:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b759e700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a0005270 0x7f34a000ba10 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f34b8071070 tx=0x7f34a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c60d84882ac5fb0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b0018070 con 0x7f34a0005270 2023-03-03T11:45:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b000f040 con 0x7f34a0005270 2023-03-03T11:45:58.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.706+0000 7f34b5d9b700 1 -- 172.21.15.7:0/1385318340 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b000eb10 con 0x7f34a0005270 2023-03-03T11:45:58.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.707+0000 7f34bd1c9700 1 -- 172.21.15.7:0/1385318340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a0005270 msgr2=0x7f34a000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:58.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.707+0000 7f34bd1c9700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a0005270 0x7f34a000ba10 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f34b8071070 tx=0x7f34a0005880 comp rx=0 tx=0).stop 2023-03-03T11:45:58.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.707+0000 7f34bd1c9700 1 -- 172.21.15.7:0/1385318340 shutdown_connections 2023-03-03T11:45:58.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.707+0000 7f34bd1c9700 1 --2- 172.21.15.7:0/1385318340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a0005270 0x7f34a000ba10 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:58.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.707+0000 7f34bd1c9700 1 -- 172.21.15.7:0/1385318340 >> 172.21.15.7:0/1385318340 conn(0x7f34b80c8900 msgr2=0x7f34b80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:45:58.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.707+0000 7f34bd1c9700 1 -- 172.21.15.7:0/1385318340 shutdown_connections 2023-03-03T11:45:58.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.707+0000 7f34bd1c9700 1 -- 172.21.15.7:0/1385318340 wait complete. 2023-03-03T11:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.708+0000 7f34bd1c9700 1 Processor -- start 2023-03-03T11:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.708+0000 7f34bd1c9700 1 -- start start 2023-03-03T11:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34bd1c9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34a0005270 0x7f34b80bd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34bd1c9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34b814eef0 0x7f34b80c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34bd1c9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34b81dd340 0x7f34b81dd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34bd1c9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b814f360 con 0x7f34b814eef0 2023-03-03T11:45:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34bd1c9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b81dc910 con 0x7f34a0005270 2023-03-03T11:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34bd1c9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b806f600 con 0x7f34b81dd340 2023-03-03T11:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34b659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34b814eef0 0x7f34b80c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34b659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34b814eef0 0x7f34b80c6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37698/0 (socket says 172.21.15.7:37698) 2023-03-03T11:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34b659c700 1 -- 172.21.15.7:0/2372446853 learned_addr learned my addr 172.21.15.7:0/2372446853 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34b759e700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34b81dd340 0x7f34b81dd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.709+0000 7f34b6d9d700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34a0005270 0x7f34b80bd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f34b659c700 1 -- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34b81dd340 msgr2=0x7f34b81dd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f34b659c700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34b81dd340 0x7f34b81dd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f34b659c700 1 -- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34a0005270 msgr2=0x7f34b80bd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f34b659c700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34a0005270 0x7f34b80bd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f34b659c700 1 -- 172.21.15.7:0/2372446853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b80ac350 con 0x7f34b814eef0 2023-03-03T11:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f34b659c700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34b814eef0 0x7f34b80c6b40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00f870 tx=0x7f34ac00fb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f349f7fe700 1 -- 172.21.15.7:0/2372446853 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac0053e0 con 0x7f34b814eef0 2023-03-03T11:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f349f7fe700 1 -- 172.21.15.7:0/2372446853 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ac005560 con 0x7f34b814eef0 2023-03-03T11:45:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f34bd1c9700 1 -- 172.21.15.7:0/2372446853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b80ab630 con 0x7f34b814eef0 2023-03-03T11:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f349f7fe700 1 -- 172.21.15.7:0/2372446853 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac006aa0 con 0x7f34b814eef0 2023-03-03T11:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.710+0000 7f34bd1c9700 1 -- 172.21.15.7:0/2372446853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b80c3e10 con 0x7f34b814eef0 2023-03-03T11:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.711+0000 7f34bd1c9700 1 -- 172.21.15.7:0/2372446853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b8074bb0 con 0x7f34b814eef0 2023-03-03T11:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.711+0000 7f349f7fe700 1 -- 172.21.15.7:0/2372446853 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f34ac01c030 con 0x7f34b814eef0 2023-03-03T11:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.712+0000 7f349f7fe700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34a40682f0 0x7f34a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.712+0000 7f349f7fe700 1 -- 172.21.15.7:0/2372446853 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f34ac085720 con 0x7f34b814eef0 2023-03-03T11:45:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.712+0000 7f34b6d9d700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34a40682f0 0x7f34a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:45:58.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.713+0000 7f34b6d9d700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34a40682f0 0x7f34a406a7a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f34a8001200 tx=0x7f34a8001b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:45:58.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.716+0000 7f349f7fe700 1 -- 172.21.15.7:0/2372446853 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f34ac051ac0 con 0x7f34b814eef0 2023-03-03T11:45:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.954+0000 7f34bd1c9700 1 -- 172.21.15.7:0/2372446853 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34b81dca80 con 0x7f34b814eef0 2023-03-03T11:45:58.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.954+0000 7f349f7fe700 1 -- 172.21.15.7:0/2372446853 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f34ac0550e0 con 0x7f34b814eef0 2023-03-03T11:45:58.956 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:45:58.956 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:45:58.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.956+0000 7f34bd1c9700 1 -- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34a40682f0 msgr2=0x7f34a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:58.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.956+0000 7f34bd1c9700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34a40682f0 0x7f34a406a7a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f34a8001200 tx=0x7f34a8001b60 comp rx=0 tx=0).stop 2023-03-03T11:45:58.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.956+0000 7f34bd1c9700 1 -- 172.21.15.7:0/2372446853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34b814eef0 msgr2=0x7f34b80c6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.956+0000 7f34bd1c9700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34b814eef0 0x7f34b80c6b40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00f870 tx=0x7f34ac00fb80 comp rx=0 tx=0).stop 2023-03-03T11:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.956+0000 7f34bd1c9700 1 -- 172.21.15.7:0/2372446853 shutdown_connections 2023-03-03T11:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.956+0000 7f34bd1c9700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34b81dd340 0x7f34b81dd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.956+0000 7f34bd1c9700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34a0005270 0x7f34b80bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.956+0000 7f34bd1c9700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34a40682f0 0x7f34a406a7a0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.956+0000 7f34bd1c9700 1 --2- 172.21.15.7:0/2372446853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34b814eef0 0x7f34b80c6b40 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.956+0000 7f34bd1c9700 1 -- 172.21.15.7:0/2372446853 >> 172.21.15.7:0/2372446853 conn(0x7f34b80c8900 msgr2=0x7f34b80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:45:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.957+0000 7f34bd1c9700 1 -- 172.21.15.7:0/2372446853 shutdown_connections 2023-03-03T11:45:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:45:58.957+0000 7f34bd1c9700 1 -- 172.21.15.7:0/2372446853 wait complete. 2023-03-03T11:45:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:45:58.970 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before thrashing 2023-03-03T11:46:03.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:46:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.108+0000 7f71592a7700 1 Processor -- start 2023-03-03T11:46:04.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.108+0000 7f71592a7700 1 -- start start 2023-03-03T11:46:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.108+0000 7f71592a7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f71540c1ca0 0x7f71540aa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.108+0000 7f71592a7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71540c20a0 con 0x7f71540c1850 2023-03-03T11:46:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.108+0000 7f71592a7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7154154790 con 0x7f715414b060 2023-03-03T11:46:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.108+0000 7f71592a7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71540c4100 con 0x7f71540c1ca0 2023-03-03T11:46:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f71537fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f71540c1ca0 0x7f71540aa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:04.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f71527fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f71540c1850 0x7f71540aa170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59920/0 (socket says 172.21.15.7:59920) 2023-03-03T11:46:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f71527fc700 1 -- 172.21.15.7:0/429675884 learned_addr learned my addr 172.21.15.7:0/429675884 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:46:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f71537fe700 1 -- 172.21.15.7:0/429675884 >> v1:172.21.15.143:6789/0 conn(0x7f715414b060 legacy=0x7f715406f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:46:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f71537fe700 1 -- 172.21.15.7:0/429675884 >> v1:172.21.15.7:6789/0 conn(0x7f71540c1850 legacy=0x7f71540aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f71537fe700 1 -- 172.21.15.7:0/429675884 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7154154a70 con 0x7f71540c1ca0 2023-03-03T11:46:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f7151ffb700 1 -- 172.21.15.7:0/429675884 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192196236 0 0) 0x7f71540c20a0 con 0x7f71540c1850 2023-03-03T11:46:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f71537fe700 1 --2- 172.21.15.7:0/429675884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71540c1ca0 0x7f71540aa880 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f714c009d90 tx=0x7f714c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c812a79561294f6e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f7151ffb700 1 -- 172.21.15.7:0/429675884 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c00e070 con 0x7f71540c1ca0 2023-03-03T11:46:04.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f7151ffb700 1 -- 172.21.15.7:0/429675884 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c00bb90 con 0x7f71540c1ca0 2023-03-03T11:46:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.109+0000 7f7151ffb700 1 -- 172.21.15.7:0/429675884 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c003a20 con 0x7f71540c1ca0 2023-03-03T11:46:04.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.110+0000 7f71592a7700 1 -- 172.21.15.7:0/429675884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71540c1ca0 msgr2=0x7f71540aa880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.110+0000 7f71592a7700 1 --2- 172.21.15.7:0/429675884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71540c1ca0 0x7f71540aa880 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f714c009d90 tx=0x7f714c00b330 comp rx=0 tx=0).stop 2023-03-03T11:46:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.110+0000 7f71592a7700 1 -- 172.21.15.7:0/429675884 shutdown_connections 2023-03-03T11:46:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.110+0000 7f71592a7700 1 --2- 172.21.15.7:0/429675884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71540c1ca0 0x7f71540aa880 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.110+0000 7f71592a7700 1 -- 172.21.15.7:0/429675884 >> 172.21.15.7:0/429675884 conn(0x7f71540c4f20 msgr2=0x7f71540c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:46:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.110+0000 7f71592a7700 1 -- 172.21.15.7:0/429675884 shutdown_connections 2023-03-03T11:46:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.110+0000 7f71592a7700 1 -- 172.21.15.7:0/429675884 wait complete. 2023-03-03T11:46:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.112+0000 7f71592a7700 1 Processor -- start 2023-03-03T11:46:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.112+0000 7f71592a7700 1 -- start start 2023-03-03T11:46:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71592a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71540c1850 0x7f7154152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71592a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71540c1ca0 0x7f71541ea640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71592a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715414b060 0x7f7154146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71592a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71541e9f30 con 0x7f715414b060 2023-03-03T11:46:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71592a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71541edf50 con 0x7f71540c1850 2023-03-03T11:46:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71592a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71541ee230 con 0x7f71540c1ca0 2023-03-03T11:46:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71537fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715414b060 0x7f7154146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71537fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715414b060 0x7f7154146520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37716/0 (socket says 172.21.15.7:37716) 2023-03-03T11:46:04.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71537fe700 1 -- 172.21.15.7:0/2080498963 learned_addr learned my addr 172.21.15.7:0/2080498963 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:46:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71527fc700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71540c1ca0 0x7f71541ea640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f7152ffd700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71540c1850 0x7f7154152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71537fe700 1 -- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71540c1ca0 msgr2=0x7f71541ea640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71537fe700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71540c1ca0 0x7f71541ea640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71537fe700 1 -- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71540c1850 msgr2=0x7f7154152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71537fe700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71540c1850 0x7f7154152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71537fe700 1 -- 172.21.15.7:0/2080498963 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c009a70 con 0x7f715414b060 2023-03-03T11:46:04.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f7152ffd700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71540c1850 0x7f7154152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:46:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71537fe700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715414b060 0x7f7154146520 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f71540c2230 tx=0x7f714c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f713b7fe700 1 -- 172.21.15.7:0/2080498963 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c012630 con 0x7f715414b060 2023-03-03T11:46:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f713b7fe700 1 -- 172.21.15.7:0/2080498963 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c0127b0 con 0x7f715414b060 2023-03-03T11:46:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f713b7fe700 1 -- 172.21.15.7:0/2080498963 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c012b60 con 0x7f715414b060 2023-03-03T11:46:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.113+0000 7f71592a7700 1 -- 172.21.15.7:0/2080498963 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7154155030 con 0x7f715414b060 2023-03-03T11:46:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.114+0000 7f71592a7700 1 -- 172.21.15.7:0/2080498963 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7154154d50 con 0x7f715414b060 2023-03-03T11:46:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.114+0000 7f71592a7700 1 -- 172.21.15.7:0/2080498963 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7154155310 con 0x7f715414b060 2023-03-03T11:46:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.115+0000 7f713b7fe700 1 -- 172.21.15.7:0/2080498963 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f714c02c070 con 0x7f715414b060 2023-03-03T11:46:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.115+0000 7f713b7fe700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71400682f0 0x7f714006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.115+0000 7f713b7fe700 1 -- 172.21.15.7:0/2080498963 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f714c088350 con 0x7f715414b060 2023-03-03T11:46:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.116+0000 7f7152ffd700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71400682f0 0x7f714006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.116+0000 7f7152ffd700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71400682f0 0x7f714006a7a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f715414e680 tx=0x7f7144008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.120+0000 7f713b7fe700 1 -- 172.21.15.7:0/2080498963 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f714c0546f0 con 0x7f715414b060 2023-03-03T11:46:04.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.359+0000 7f71592a7700 1 -- 172.21.15.7:0/2080498963 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71541ea330 con 0x7f715414b060 2023-03-03T11:46:04.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.360+0000 7f713b7fe700 1 -- 172.21.15.7:0/2080498963 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f714c057d10 con 0x7f715414b060 2023-03-03T11:46:04.361 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:46:04.361 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:46:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 -- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71400682f0 msgr2=0x7f714006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71400682f0 0x7f714006a7a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f715414e680 tx=0x7f7144008040 comp rx=0 tx=0).stop 2023-03-03T11:46:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 -- 172.21.15.7:0/2080498963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715414b060 msgr2=0x7f7154146520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715414b060 0x7f7154146520 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f71540c2230 tx=0x7f714c004ad0 comp rx=0 tx=0).stop 2023-03-03T11:46:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 -- 172.21.15.7:0/2080498963 shutdown_connections 2023-03-03T11:46:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71400682f0 0x7f714006a7a0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f715414b060 0x7f7154146520 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71540c1850 0x7f7154152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 --2- 172.21.15.7:0/2080498963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71540c1ca0 0x7f71541ea640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:04.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 -- 172.21.15.7:0/2080498963 >> 172.21.15.7:0/2080498963 conn(0x7f71540c4f20 msgr2=0x7f7154153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:46:04.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 -- 172.21.15.7:0/2080498963 shutdown_connections 2023-03-03T11:46:04.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:04.362+0000 7f71592a7700 1 -- 172.21.15.7:0/2080498963 wait complete. 2023-03-03T11:46:04.365 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:46:04.375 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T11:46:18.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:18.475 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:39.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:46:39.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.504+0000 7f9b45c7b700 1 Processor -- start 2023-03-03T11:46:39.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.504+0000 7f9b45c7b700 1 -- start start 2023-03-03T11:46:39.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b45c7b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b4013a0f0 0x7f9b400c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:39.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b45c7b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b400c1da0 0x7f9b4006d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:39.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b45c7b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b4006de90 con 0x7f9b401586c0 2023-03-03T11:46:39.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b45c7b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b40154840 con 0x7f9b400c1da0 2023-03-03T11:46:39.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b45c7b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40154560 con 0x7f9b4013a0f0 2023-03-03T11:46:39.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b3ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b400c1da0 0x7f9b4006d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:39.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b3ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b400c1da0 0x7f9b4006d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37732/0 (socket says 172.21.15.7:37732) 2023-03-03T11:46:39.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b3ffff700 1 -- 172.21.15.7:0/1968895348 learned_addr learned my addr 172.21.15.7:0/1968895348 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:46:39.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b3f7fe700 1 --2- 172.21.15.7:0/1968895348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b4013a0f0 0x7f9b400c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:39.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b3ffff700 1 -- 172.21.15.7:0/1968895348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b4013a0f0 msgr2=0x7f9b400c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:46:39.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b3ffff700 1 --2- 172.21.15.7:0/1968895348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b4013a0f0 0x7f9b400c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:39.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b3ffff700 1 -- 172.21.15.7:0/1968895348 >> v1:172.21.15.143:6789/0 conn(0x7f9b401586c0 legacy=0x7f9b4006d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:39.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.505+0000 7f9b3ffff700 1 -- 172.21.15.7:0/1968895348 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b40157be0 con 0x7f9b400c1da0 2023-03-03T11:46:39.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b3ffff700 1 --2- 172.21.15.7:0/1968895348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b400c1da0 0x7f9b4006d950 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009e50 tx=0x7f9b3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe585933bb333c10 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:39.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b3e7fc700 1 -- 172.21.15.7:0/1968895348 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3800e070 con 0x7f9b400c1da0 2023-03-03T11:46:39.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b3e7fc700 1 -- 172.21.15.7:0/1968895348 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3800b9c0 con 0x7f9b400c1da0 2023-03-03T11:46:39.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b3e7fc700 1 -- 172.21.15.7:0/1968895348 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3800bd70 con 0x7f9b400c1da0 2023-03-03T11:46:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b45c7b700 1 -- 172.21.15.7:0/1968895348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b400c1da0 msgr2=0x7f9b4006d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b45c7b700 1 --2- 172.21.15.7:0/1968895348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b400c1da0 0x7f9b4006d950 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009e50 tx=0x7f9b3800b330 comp rx=0 tx=0).stop 2023-03-03T11:46:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b45c7b700 1 -- 172.21.15.7:0/1968895348 shutdown_connections 2023-03-03T11:46:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b45c7b700 1 --2- 172.21.15.7:0/1968895348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b400c1da0 0x7f9b4006d950 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b45c7b700 1 --2- 172.21.15.7:0/1968895348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b4013a0f0 0x7f9b400c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b45c7b700 1 -- 172.21.15.7:0/1968895348 >> 172.21.15.7:0/1968895348 conn(0x7f9b4006c110 msgr2=0x7f9b4006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:46:39.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b45c7b700 1 -- 172.21.15.7:0/1968895348 shutdown_connections 2023-03-03T11:46:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.506+0000 7f9b45c7b700 1 -- 172.21.15.7:0/1968895348 wait complete. 2023-03-03T11:46:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.508+0000 7f9b45c7b700 1 Processor -- start 2023-03-03T11:46:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.508+0000 7f9b45c7b700 1 -- start start 2023-03-03T11:46:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.508+0000 7f9b45c7b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b400c1da0 0x7f9b401e0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.508+0000 7f9b45c7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b4013a0f0 0x7f9b401511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.508+0000 7f9b45c7b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b401586c0 0x7f9b40151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:39.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.508+0000 7f9b45c7b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40156040 con 0x7f9b4013a0f0 2023-03-03T11:46:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b45c7b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40155900 con 0x7f9b400c1da0 2023-03-03T11:46:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b45c7b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b40070bc0 con 0x7f9b401586c0 2023-03-03T11:46:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b4013a0f0 0x7f9b401511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b4013a0f0 0x7f9b401511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37736/0 (socket says 172.21.15.7:37736) 2023-03-03T11:46:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3effd700 1 -- 172.21.15.7:0/2754430018 learned_addr learned my addr 172.21.15.7:0/2754430018 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:46:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3f7fe700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b400c1da0 0x7f9b401e0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3ffff700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b401586c0 0x7f9b40151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:39.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3effd700 1 -- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b401586c0 msgr2=0x7f9b40151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3effd700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b401586c0 0x7f9b40151700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3effd700 1 -- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b400c1da0 msgr2=0x7f9b401e0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3effd700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b400c1da0 0x7f9b401e0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3effd700 1 -- 172.21.15.7:0/2754430018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b38009b30 con 0x7f9b4013a0f0 2023-03-03T11:46:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b3effd700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b4013a0f0 0x7f9b401511c0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400c4e0 tx=0x7f9b3400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b27fff700 1 -- 172.21.15.7:0/2754430018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b340052d0 con 0x7f9b4013a0f0 2023-03-03T11:46:39.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b27fff700 1 -- 172.21.15.7:0/2754430018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b34005430 con 0x7f9b4013a0f0 2023-03-03T11:46:39.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b45c7b700 1 -- 172.21.15.7:0/2754430018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b401581a0 con 0x7f9b4013a0f0 2023-03-03T11:46:39.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b27fff700 1 -- 172.21.15.7:0/2754430018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b340038c0 con 0x7f9b4013a0f0 2023-03-03T11:46:39.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.509+0000 7f9b45c7b700 1 -- 172.21.15.7:0/2754430018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b40157ec0 con 0x7f9b4013a0f0 2023-03-03T11:46:39.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.510+0000 7f9b45c7b700 1 -- 172.21.15.7:0/2754430018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4006e1a0 con 0x7f9b4013a0f0 2023-03-03T11:46:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.511+0000 7f9b27fff700 1 -- 172.21.15.7:0/2754430018 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9b34019030 con 0x7f9b4013a0f0 2023-03-03T11:46:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.511+0000 7f9b27fff700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b2c0682f0 0x7f9b2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.511+0000 7f9b27fff700 1 -- 172.21.15.7:0/2754430018 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9b34085300 con 0x7f9b4013a0f0 2023-03-03T11:46:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.511+0000 7f9b3f7fe700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b2c0682f0 0x7f9b2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.512+0000 7f9b3f7fe700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b2c0682f0 0x7f9b2c06a7a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009770 tx=0x7f9b30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:39.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.515+0000 7f9b27fff700 1 -- 172.21.15.7:0/2754430018 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9b340516a0 con 0x7f9b4013a0f0 2023-03-03T11:46:39.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.752+0000 7f9b45c7b700 1 -- 172.21.15.7:0/2754430018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b4006de90 con 0x7f9b4013a0f0 2023-03-03T11:46:39.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.752+0000 7f9b27fff700 1 -- 172.21.15.7:0/2754430018 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9b4006de90 con 0x7f9b4013a0f0 2023-03-03T11:46:39.755 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:46:39.755 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:46:39.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.754+0000 7f9b45c7b700 1 -- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b2c0682f0 msgr2=0x7f9b2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:39.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.754+0000 7f9b45c7b700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b2c0682f0 0x7f9b2c06a7a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009770 tx=0x7f9b30008040 comp rx=0 tx=0).stop 2023-03-03T11:46:39.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.754+0000 7f9b45c7b700 1 -- 172.21.15.7:0/2754430018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b4013a0f0 msgr2=0x7f9b401511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:39.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.754+0000 7f9b45c7b700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b4013a0f0 0x7f9b401511c0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400c4e0 tx=0x7f9b3400c8a0 comp rx=0 tx=0).stop 2023-03-03T11:46:39.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.754+0000 7f9b45c7b700 1 -- 172.21.15.7:0/2754430018 shutdown_connections 2023-03-03T11:46:39.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.754+0000 7f9b45c7b700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b401586c0 0x7f9b40151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:39.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.754+0000 7f9b45c7b700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b400c1da0 0x7f9b401e0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:39.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.754+0000 7f9b45c7b700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b2c0682f0 0x7f9b2c06a7a0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:39.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.754+0000 7f9b45c7b700 1 --2- 172.21.15.7:0/2754430018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b4013a0f0 0x7f9b401511c0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:39.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.754+0000 7f9b45c7b700 1 -- 172.21.15.7:0/2754430018 >> 172.21.15.7:0/2754430018 conn(0x7f9b4006c110 msgr2=0x7f9b40137150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:46:39.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.755+0000 7f9b45c7b700 1 -- 172.21.15.7:0/2754430018 shutdown_connections 2023-03-03T11:46:39.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:39.755+0000 7f9b45c7b700 1 -- 172.21.15.7:0/2754430018 wait complete. 2023-03-03T11:46:39.758 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:46:39.768 INFO:tasks.mds_thrash.fs.[b]:waiting for 66 secs before thrashing 2023-03-03T11:46:46.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:46:46.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.512+0000 7fcb73dcf700 1 Processor -- start 2023-03-03T11:46:46.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.512+0000 7fcb73dcf700 1 -- start start 2023-03-03T11:46:46.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb73dcf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb6c14e560 0x7fcb6c071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:46.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb73dcf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcb6c0c4840 0x7fcb6c154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:46.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb73dcf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c072920 con 0x7fcb6c0c43d0 2023-03-03T11:46:46.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb73dcf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6c06bef0 con 0x7fcb6c0c4840 2023-03-03T11:46:46.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb73dcf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c1551e0 con 0x7fcb6c14e560 2023-03-03T11:46:46.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb7236c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcb6c0c4840 0x7fcb6c154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:46.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb7236c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcb6c0c4840 0x7fcb6c154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37748/0 (socket says 172.21.15.7:37748) 2023-03-03T11:46:46.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb7236c700 1 -- 172.21.15.7:0/2065810951 learned_addr learned my addr 172.21.15.7:0/2065810951 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:46:46.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb71b6b700 1 --2- 172.21.15.7:0/2065810951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb6c14e560 0x7fcb6c071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:46.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb7236c700 1 -- 172.21.15.7:0/2065810951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb6c14e560 msgr2=0x7fcb6c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:46:46.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb7236c700 1 --2- 172.21.15.7:0/2065810951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb6c14e560 0x7fcb6c071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:46.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb7236c700 1 -- 172.21.15.7:0/2065810951 >> v1:172.21.15.143:6789/0 conn(0x7fcb6c0c43d0 legacy=0x7fcb6c154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:46.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb7236c700 1 -- 172.21.15.7:0/2065810951 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c06b910 con 0x7fcb6c0c4840 2023-03-03T11:46:46.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.513+0000 7fcb7236c700 1 --2- 172.21.15.7:0/2065810951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 0x7fcb6c154ca0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009e50 tx=0x7fcb6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd9b0e43275e406 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:46.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb70b69700 1 -- 172.21.15.7:0/2065810951 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6400e070 con 0x7fcb6c0c4840 2023-03-03T11:46:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb70b69700 1 -- 172.21.15.7:0/2065810951 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6400b9c0 con 0x7fcb6c0c4840 2023-03-03T11:46:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb70b69700 1 -- 172.21.15.7:0/2065810951 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6400bd70 con 0x7fcb6c0c4840 2023-03-03T11:46:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb73dcf700 1 -- 172.21.15.7:0/2065810951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 msgr2=0x7fcb6c154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb73dcf700 1 --2- 172.21.15.7:0/2065810951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 0x7fcb6c154ca0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009e50 tx=0x7fcb6400b330 comp rx=0 tx=0).stop 2023-03-03T11:46:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb73dcf700 1 -- 172.21.15.7:0/2065810951 shutdown_connections 2023-03-03T11:46:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb73dcf700 1 --2- 172.21.15.7:0/2065810951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 0x7fcb6c154ca0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb73dcf700 1 --2- 172.21.15.7:0/2065810951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb6c14e560 0x7fcb6c071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:46.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb73dcf700 1 -- 172.21.15.7:0/2065810951 >> 172.21.15.7:0/2065810951 conn(0x7fcb6c0736f0 msgr2=0x7fcb6c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:46:46.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb73dcf700 1 -- 172.21.15.7:0/2065810951 shutdown_connections 2023-03-03T11:46:46.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.514+0000 7fcb73dcf700 1 -- 172.21.15.7:0/2065810951 wait complete. 2023-03-03T11:46:46.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.515+0000 7fcb73dcf700 1 Processor -- start 2023-03-03T11:46:46.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.515+0000 7fcb73dcf700 1 -- start start 2023-03-03T11:46:46.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb73dcf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb6c0c43d0 0x7fcb6c14ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:46.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb73dcf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 0x7fcb6c1ed5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:46.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb73dcf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb6c14e560 0x7fcb6c1edb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:46.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb73dcf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0ade10 con 0x7fcb6c0c4840 2023-03-03T11:46:46.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb73dcf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c1ed160 con 0x7fcb6c14e560 2023-03-03T11:46:46.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb73dcf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0adb30 con 0x7fcb6c0c43d0 2023-03-03T11:46:46.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb7136a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 0x7fcb6c1ed5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:46.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb7136a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 0x7fcb6c1ed5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37752/0 (socket says 172.21.15.7:37752) 2023-03-03T11:46:46.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb7136a700 1 -- 172.21.15.7:0/4107544499 learned_addr learned my addr 172.21.15.7:0/4107544499 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:46:46.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb7236c700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb6c14e560 0x7fcb6c1edb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:46.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.516+0000 7fcb71b6b700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb6c0c43d0 0x7fcb6c14ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:46.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb7136a700 1 -- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb6c0c43d0 msgr2=0x7fcb6c14ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:46.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb7136a700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb6c0c43d0 0x7fcb6c14ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:46.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb7136a700 1 -- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb6c14e560 msgr2=0x7fcb6c1edb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:46.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb7136a700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb6c14e560 0x7fcb6c1edb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:46.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb7136a700 1 -- 172.21.15.7:0/4107544499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb64009b30 con 0x7fcb6c0c4840 2023-03-03T11:46:46.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb7136a700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 0x7fcb6c1ed5f0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c4e0 tx=0x7fcb5c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:46.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb627fc700 1 -- 172.21.15.7:0/4107544499 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0052d0 con 0x7fcb6c0c4840 2023-03-03T11:46:46.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb627fc700 1 -- 172.21.15.7:0/4107544499 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c005430 con 0x7fcb6c0c4840 2023-03-03T11:46:46.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb73dcf700 1 -- 172.21.15.7:0/4107544499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c0aacc0 con 0x7fcb6c0c4840 2023-03-03T11:46:46.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb627fc700 1 -- 172.21.15.7:0/4107544499 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0038c0 con 0x7fcb6c0c4840 2023-03-03T11:46:46.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.517+0000 7fcb73dcf700 1 -- 172.21.15.7:0/4107544499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c0c4ca0 con 0x7fcb6c0c4840 2023-03-03T11:46:46.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.518+0000 7fcb73dcf700 1 -- 172.21.15.7:0/4107544499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c06bb80 con 0x7fcb6c0c4840 2023-03-03T11:46:46.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.518+0000 7fcb627fc700 1 -- 172.21.15.7:0/4107544499 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcb5c019030 con 0x7fcb6c0c4840 2023-03-03T11:46:46.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.519+0000 7fcb627fc700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb500682f0 0x7fcb5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:46:46.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.519+0000 7fcb627fc700 1 -- 172.21.15.7:0/4107544499 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcb5c0851f0 con 0x7fcb6c0c4840 2023-03-03T11:46:46.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.519+0000 7fcb71b6b700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb500682f0 0x7fcb5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:46:46.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.520+0000 7fcb71b6b700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb500682f0 0x7fcb5006a7a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c152120 tx=0x7fcb68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:46:46.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.523+0000 7fcb627fc700 1 -- 172.21.15.7:0/4107544499 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcb5c051530 con 0x7fcb6c0c4840 2023-03-03T11:46:46.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.760+0000 7fcb73dcf700 1 -- 172.21.15.7:0/4107544499 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb6c14f220 con 0x7fcb6c0c4840 2023-03-03T11:46:46.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.761+0000 7fcb627fc700 1 -- 172.21.15.7:0/4107544499 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcb6c14f220 con 0x7fcb6c0c4840 2023-03-03T11:46:46.763 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:46:46.764 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:46:46.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.762+0000 7fcb73dcf700 1 -- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb500682f0 msgr2=0x7fcb5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:46.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.762+0000 7fcb73dcf700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb500682f0 0x7fcb5006a7a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c152120 tx=0x7fcb68008040 comp rx=0 tx=0).stop 2023-03-03T11:46:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.762+0000 7fcb73dcf700 1 -- 172.21.15.7:0/4107544499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 msgr2=0x7fcb6c1ed5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:46:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.762+0000 7fcb73dcf700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 0x7fcb6c1ed5f0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00c4e0 tx=0x7fcb5c00c7f0 comp rx=0 tx=0).stop 2023-03-03T11:46:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.762+0000 7fcb73dcf700 1 -- 172.21.15.7:0/4107544499 shutdown_connections 2023-03-03T11:46:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.762+0000 7fcb73dcf700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb6c14e560 0x7fcb6c1edb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.762+0000 7fcb73dcf700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb6c0c43d0 0x7fcb6c14ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.762+0000 7fcb73dcf700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb500682f0 0x7fcb5006a7a0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.762+0000 7fcb73dcf700 1 --2- 172.21.15.7:0/4107544499 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb6c0c4840 0x7fcb6c1ed5f0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:46:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.762+0000 7fcb73dcf700 1 -- 172.21.15.7:0/4107544499 >> 172.21.15.7:0/4107544499 conn(0x7fcb6c0736f0 msgr2=0x7fcb6c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:46:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.763+0000 7fcb73dcf700 1 -- 172.21.15.7:0/4107544499 shutdown_connections 2023-03-03T11:46:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:46:46.763+0000 7fcb73dcf700 1 -- 172.21.15.7:0/4107544499 wait complete. 2023-03-03T11:46:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:46:46.777 INFO:tasks.mds_thrash.fs.[a]:waiting for 90 secs before thrashing 2023-03-03T11:46:48.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:46:48.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:18.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:18.735 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:45.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:47:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.907+0000 7ff581d31700 1 Processor -- start 2023-03-03T11:47:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.907+0000 7ff581d31700 1 -- start start 2023-03-03T11:47:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.907+0000 7ff581d31700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 0x7ff57c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.907+0000 7ff581d31700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff57c073c30 0x7ff57c15cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.907+0000 7ff581d31700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c0c2730 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.907+0000 7ff581d31700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57c0c22b0 con 0x7ff57c073c30 2023-03-03T11:47:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.907+0000 7ff581d31700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0c2980 con 0x7ff57c14fa20 2023-03-03T11:47:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff57c0ac9e0 0x7ff57c1524e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60012/0 (socket says 172.21.15.7:60012) 2023-03-03T11:47:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57affd700 1 -- 172.21.15.7:0/2905073248 learned_addr learned my addr 172.21.15.7:0/2905073248 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:47:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57bfff700 1 --2- 172.21.15.7:0/2905073248 >> v2:172.21.15.143:3300/0 conn(0x7ff57c073c30 0x7ff57c15cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57b7fe700 1 --2- 172.21.15.7:0/2905073248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 0x7ff57c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57a7fc700 1 -- 172.21.15.7:0/2905073248 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168360757 0 0) 0x7ff57c0c2730 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57a7fc700 1 -- 172.21.15.7:0/2905073248 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5680035b0 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57b7fe700 1 -- 172.21.15.7:0/2905073248 >> v2:172.21.15.143:3300/0 conn(0x7ff57c073c30 msgr2=0x7ff57c15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:47:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57b7fe700 1 --2- 172.21.15.7:0/2905073248 >> v2:172.21.15.143:3300/0 conn(0x7ff57c073c30 0x7ff57c15cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57a7fc700 1 -- 172.21.15.7:0/2905073248 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3469365176 0 0) 0x7ff5680035b0 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57b7fe700 1 -- 172.21.15.7:0/2905073248 >> v1:172.21.15.7:6789/0 conn(0x7ff57c0ac9e0 legacy=0x7ff57c1524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57b7fe700 1 -- 172.21.15.7:0/2905073248 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c155a50 con 0x7ff57c14fa20 2023-03-03T11:47:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57bfff700 1 --2- 172.21.15.7:0/2905073248 >> v2:172.21.15.143:3300/0 conn(0x7ff57c073c30 0x7ff57c15cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:47:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.908+0000 7ff57b7fe700 1 --2- 172.21.15.7:0/2905073248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 0x7ff57c0736f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009840 tx=0x7ff56c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfaae744dbc1fabd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:47:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff57a7fc700 1 -- 172.21.15.7:0/2905073248 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564001670 con 0x7ff57c14fa20 2023-03-03T11:47:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff57a7fc700 1 -- 172.21.15.7:0/2905073248 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c004030 con 0x7ff57c14fa20 2023-03-03T11:47:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff57a7fc700 1 -- 172.21.15.7:0/2905073248 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c0043c0 con 0x7ff57c14fa20 2023-03-03T11:47:45.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff581d31700 1 -- 172.21.15.7:0/2905073248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 msgr2=0x7ff57c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:45.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff581d31700 1 --2- 172.21.15.7:0/2905073248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 0x7ff57c0736f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009840 tx=0x7ff56c009b50 comp rx=0 tx=0).stop 2023-03-03T11:47:45.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff581d31700 1 -- 172.21.15.7:0/2905073248 shutdown_connections 2023-03-03T11:47:45.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff581d31700 1 --2- 172.21.15.7:0/2905073248 >> v2:172.21.15.143:3300/0 conn(0x7ff57c073c30 0x7ff57c15cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:45.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff581d31700 1 --2- 172.21.15.7:0/2905073248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 0x7ff57c0736f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:45.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff581d31700 1 -- 172.21.15.7:0/2905073248 >> 172.21.15.7:0/2905073248 conn(0x7ff57c0a94d0 msgr2=0x7ff57c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:47:45.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff581d31700 1 -- 172.21.15.7:0/2905073248 shutdown_connections 2023-03-03T11:47:45.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.909+0000 7ff581d31700 1 -- 172.21.15.7:0/2905073248 wait complete. 2023-03-03T11:47:45.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.910+0000 7ff581d31700 1 Processor -- start 2023-03-03T11:47:45.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff581d31700 1 -- start start 2023-03-03T11:47:45.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff581d31700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff57c073c30 0x7ff57c1df2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:45.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff581d31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff57c0ac9e0 0x7ff57c14a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:45.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff581d31700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 0x7ff57c14aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:45.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff581d31700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c14b020 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff581d31700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c1e7ba0 con 0x7ff57c073c30 2023-03-03T11:47:45.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff581d31700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c1e7e80 con 0x7ff57c14fa20 2023-03-03T11:47:45.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff57affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff57c0ac9e0 0x7ff57c14a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:45.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff57affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff57c0ac9e0 0x7ff57c14a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37806/0 (socket says 172.21.15.7:37806) 2023-03-03T11:47:45.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff57b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff57c073c30 0x7ff57c1df2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:45.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff57affd700 1 -- 172.21.15.7:0/4245010733 learned_addr learned my addr 172.21.15.7:0/4245010733 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:47:45.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.911+0000 7ff57bfff700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 0x7ff57c14aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:45.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff57affd700 1 -- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 msgr2=0x7ff57c14aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:45.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff57affd700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 0x7ff57c14aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:45.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff57affd700 1 -- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff57c073c30 msgr2=0x7ff57c1df2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:45.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff57affd700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff57c073c30 0x7ff57c1df2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:45.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff57affd700 1 -- 172.21.15.7:0/4245010733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff574009a70 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff57b7fe700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff57c073c30 0x7ff57c1df2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:47:45.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff57affd700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff57c0ac9e0 0x7ff57c14a5a0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7ff56400c780 tx=0x7ff56400ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:47:45.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff573fff700 1 -- 172.21.15.7:0/4245010733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff564004c80 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff573fff700 1 -- 172.21.15.7:0/4245010733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff564004de0 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff573fff700 1 -- 172.21.15.7:0/4245010733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5640072f0 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff581d31700 1 -- 172.21.15.7:0/4245010733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff56c0094a0 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.912+0000 7ff581d31700 1 -- 172.21.15.7:0/4245010733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57c156010 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.913+0000 7ff581d31700 1 -- 172.21.15.7:0/4245010733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57c155d30 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.914+0000 7ff573fff700 1 -- 172.21.15.7:0/4245010733 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff564010070 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.914+0000 7ff573fff700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff55c0682f0 0x7ff55c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:47:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.914+0000 7ff573fff700 1 -- 172.21.15.7:0/4245010733 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff564085000 con 0x7ff57c0ac9e0 2023-03-03T11:47:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.914+0000 7ff57b7fe700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff55c0682f0 0x7ff55c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:47:45.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.915+0000 7ff57b7fe700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff55c0682f0 0x7ff55c06a7a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00a010 tx=0x7ff56c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:47:45.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:45.918+0000 7ff573fff700 1 -- 172.21.15.7:0/4245010733 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5640513a0 con 0x7ff57c0ac9e0 2023-03-03T11:47:46.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.157+0000 7ff581d31700 1 -- 172.21.15.7:0/4245010733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff57c1562f0 con 0x7ff57c0ac9e0 2023-03-03T11:47:46.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.157+0000 7ff573fff700 1 -- 172.21.15.7:0/4245010733 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff564018090 con 0x7ff57c0ac9e0 2023-03-03T11:47:46.159 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:47:46.159 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:47:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 -- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff55c0682f0 msgr2=0x7ff55c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff55c0682f0 0x7ff55c06a7a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff56c00a010 tx=0x7ff56c008030 comp rx=0 tx=0).stop 2023-03-03T11:47:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 -- 172.21.15.7:0/4245010733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff57c0ac9e0 msgr2=0x7ff57c14a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:47:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff57c0ac9e0 0x7ff57c14a5a0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7ff56400c780 tx=0x7ff56400ca90 comp rx=0 tx=0).stop 2023-03-03T11:47:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 -- 172.21.15.7:0/4245010733 shutdown_connections 2023-03-03T11:47:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff57c14fa20 0x7ff57c14aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff57c073c30 0x7ff57c1df2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff55c0682f0 0x7ff55c06a7a0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 --2- 172.21.15.7:0/4245010733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff57c0ac9e0 0x7ff57c14a5a0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:47:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 -- 172.21.15.7:0/4245010733 >> 172.21.15.7:0/4245010733 conn(0x7ff57c0a94d0 msgr2=0x7ff57c070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:47:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 -- 172.21.15.7:0/4245010733 shutdown_connections 2023-03-03T11:47:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:47:46.159+0000 7ff581d31700 1 -- 172.21.15.7:0/4245010733 wait complete. 2023-03-03T11:47:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:47:46.173 INFO:tasks.mds_thrash.fs.[b]:waiting for 55 secs before thrashing 2023-03-03T11:47:48.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:47:48.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:16.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:48:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.912+0000 7f01c4ad7700 1 Processor -- start 2023-03-03T11:48:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.913+0000 7f01c4ad7700 1 -- start start 2023-03-03T11:48:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.913+0000 7f01c4ad7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01c00c2790 0x7f01c0152d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.913+0000 7f01c4ad7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01c01546f0 0x7f01c0154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.913+0000 7f01c4ad7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c006c920 con 0x7f01c014b340 2023-03-03T11:48:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.913+0000 7f01c4ad7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c00abd90 con 0x7f01c01546f0 2023-03-03T11:48:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.913+0000 7f01c4ad7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00c1300 con 0x7f01c00c2790 2023-03-03T11:48:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01c01546f0 0x7f01c0154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01c01546f0 0x7f01c0154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37832/0 (socket says 172.21.15.7:37832) 2023-03-03T11:48:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bed9d700 1 -- 172.21.15.7:0/2579799460 learned_addr learned my addr 172.21.15.7:0/2579799460 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:48:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bdd9b700 1 --2- 172.21.15.7:0/2579799460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01c00c2790 0x7f01c0152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bed9d700 1 -- 172.21.15.7:0/2579799460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01c00c2790 msgr2=0x7f01c0152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:48:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bed9d700 1 --2- 172.21.15.7:0/2579799460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01c00c2790 0x7f01c0152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bed9d700 1 -- 172.21.15.7:0/2579799460 >> v1:172.21.15.143:6789/0 conn(0x7f01c014b340 legacy=0x7f01c0152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:48:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bed9d700 1 -- 172.21.15.7:0/2579799460 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c0153280 con 0x7f01c01546f0 2023-03-03T11:48:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bed9d700 1 --2- 172.21.15.7:0/2579799460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 0x7f01c0154b40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009e50 tx=0x7f01b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e9c229b3ee58be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bd59a700 1 -- 172.21.15.7:0/2579799460 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b800e070 con 0x7f01c01546f0 2023-03-03T11:48:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bd59a700 1 -- 172.21.15.7:0/2579799460 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b8003a20 con 0x7f01c01546f0 2023-03-03T11:48:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.914+0000 7f01bd59a700 1 -- 172.21.15.7:0/2579799460 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b8003dd0 con 0x7f01c01546f0 2023-03-03T11:48:16.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.915+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2579799460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 msgr2=0x7f01c0154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:16.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.915+0000 7f01c4ad7700 1 --2- 172.21.15.7:0/2579799460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 0x7f01c0154b40 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009e50 tx=0x7f01b800b330 comp rx=0 tx=0).stop 2023-03-03T11:48:16.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.915+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2579799460 shutdown_connections 2023-03-03T11:48:16.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.915+0000 7f01c4ad7700 1 --2- 172.21.15.7:0/2579799460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 0x7f01c0154b40 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:16.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.915+0000 7f01c4ad7700 1 --2- 172.21.15.7:0/2579799460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01c00c2790 0x7f01c0152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:16.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.915+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2579799460 >> 172.21.15.7:0/2579799460 conn(0x7f01c00c62e0 msgr2=0x7f01c00c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:48:16.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.915+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2579799460 shutdown_connections 2023-03-03T11:48:16.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.915+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2579799460 wait complete. 2023-03-03T11:48:16.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.916+0000 7f01c4ad7700 1 Processor -- start 2023-03-03T11:48:16.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.916+0000 7f01c4ad7700 1 -- start start 2023-03-03T11:48:16.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01c4ad7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01c00c2790 0x7f01c0147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:16.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01c4ad7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01c014b340 0x7f01c01dee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:16.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01c4ad7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 0x7f01c0140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:16.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01c4ad7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c0141e80 con 0x7f01c01546f0 2023-03-03T11:48:16.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01c4ad7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01ed4e0 con 0x7f01c014b340 2023-03-03T11:48:16.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01c4ad7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01480c0 con 0x7f01c00c2790 2023-03-03T11:48:16.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01bed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 0x7f01c0140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:16.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01bed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 0x7f01c0140ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37838/0 (socket says 172.21.15.7:37838) 2023-03-03T11:48:16.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01bed9d700 1 -- 172.21.15.7:0/2468428197 learned_addr learned my addr 172.21.15.7:0/2468428197 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:48:16.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01be59c700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01c00c2790 0x7f01c0147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:16.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01bdd9b700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01c014b340 0x7f01c01dee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:16.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.917+0000 7f01bed9d700 1 -- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01c00c2790 msgr2=0x7f01c0147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:16.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01bed9d700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01c00c2790 0x7f01c0147b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:16.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01bed9d700 1 -- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01c014b340 msgr2=0x7f01c01dee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:16.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01bed9d700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01c014b340 0x7f01c01dee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:16.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01bed9d700 1 -- 172.21.15.7:0/2468428197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b8009b30 con 0x7f01c01546f0 2023-03-03T11:48:16.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01bdd9b700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01c014b340 0x7f01c01dee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:48:16.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01bed9d700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 0x7f01c0140ae0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f01c006d780 tx=0x7f01b8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:16.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01b6ffd700 1 -- 172.21.15.7:0/2468428197 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b80092e0 con 0x7f01c01546f0 2023-03-03T11:48:16.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01b6ffd700 1 -- 172.21.15.7:0/2468428197 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b8004240 con 0x7f01c01546f0 2023-03-03T11:48:16.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2468428197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c006bff0 con 0x7f01c01546f0 2023-03-03T11:48:16.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01b6ffd700 1 -- 172.21.15.7:0/2468428197 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b80045f0 con 0x7f01c01546f0 2023-03-03T11:48:16.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.918+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2468428197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c006bd50 con 0x7f01c01546f0 2023-03-03T11:48:16.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.919+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2468428197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c006c2d0 con 0x7f01c01546f0 2023-03-03T11:48:16.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.919+0000 7f01b6ffd700 1 -- 172.21.15.7:0/2468428197 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f01b8004750 con 0x7f01c01546f0 2023-03-03T11:48:16.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.920+0000 7f01b6ffd700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 0x7f01a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:16.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.920+0000 7f01b6ffd700 1 -- 172.21.15.7:0/2468428197 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f01b8087500 con 0x7f01c01546f0 2023-03-03T11:48:16.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.920+0000 7f01be59c700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 0x7f01a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:16.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.920+0000 7f01be59c700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 0x7f01a006a7a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f01a8009870 tx=0x7f01a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:16.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:16.924+0000 7f01b6ffd700 1 -- 172.21.15.7:0/2468428197 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f01b8053820 con 0x7f01c01546f0 2023-03-03T11:48:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.164+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2468428197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01c0074bb0 con 0x7f01c01546f0 2023-03-03T11:48:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.164+0000 7f01b6ffd700 1 -- 172.21.15.7:0/2468428197 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f01b8007310 con 0x7f01c01546f0 2023-03-03T11:48:17.166 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:48:17.166 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:48:17.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 msgr2=0x7f01a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:17.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 0x7f01a006a7a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f01a8009870 tx=0x7f01a8008040 comp rx=0 tx=0).stop 2023-03-03T11:48:17.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2468428197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 msgr2=0x7f01c0140ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:17.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 0x7f01c0140ae0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f01c006d780 tx=0x7f01b8004ad0 comp rx=0 tx=0).stop 2023-03-03T11:48:17.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2468428197 shutdown_connections 2023-03-03T11:48:17.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 0x7f01a006a7a0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:17.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01c01546f0 0x7f01c0140ae0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:17.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01c00c2790 0x7f01c0147b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:17.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 --2- 172.21.15.7:0/2468428197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01c014b340 0x7f01c01dee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:17.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2468428197 >> 172.21.15.7:0/2468428197 conn(0x7f01c00c62e0 msgr2=0x7f01c00c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:48:17.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2468428197 shutdown_connections 2023-03-03T11:48:17.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:17.167+0000 7f01c4ad7700 1 -- 172.21.15.7:0/2468428197 wait complete. 2023-03-03T11:48:17.170 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:48:17.180 INFO:tasks.mds_thrash.fs.[a]:waiting for 24 secs before thrashing 2023-03-03T11:48:18.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:18.970 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:41.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:48:41.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:48:41.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.313+0000 7fabd1cf2700 1 Processor -- start 2023-03-03T11:48:41.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.313+0000 7fabd1cf2700 1 -- start start 2023-03-03T11:48:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabd1cf2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fabcc0caa30 0x7fabcc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabd1cf2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabcc154ff0 0x7fabcc0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabd1cf2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc0c7310 con 0x7fabcc0c00e0 2023-03-03T11:48:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabd1cf2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabcc155430 con 0x7fabcc154ff0 2023-03-03T11:48:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabd1cf2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabcc0bfd10 con 0x7fabcc0caa30 2023-03-03T11:48:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabcbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabcc154ff0 0x7fabcc0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabcbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabcc154ff0 0x7fabcc0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37858/0 (socket says 172.21.15.7:37858) 2023-03-03T11:48:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabcbfff700 1 -- 172.21.15.7:0/369052243 learned_addr learned my addr 172.21.15.7:0/369052243 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:48:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabcaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fabcc0c00e0 0x7fabcc0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60066/0 (socket says 172.21.15.7:60066) 2023-03-03T11:48:41.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabcb7fe700 1 --2- 172.21.15.7:0/369052243 >> v2:172.21.15.143:3300/0 conn(0x7fabcc0caa30 0x7fabcc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabcbfff700 1 -- 172.21.15.7:0/369052243 >> v2:172.21.15.143:3300/0 conn(0x7fabcc0caa30 msgr2=0x7fabcc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:48:41.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabcbfff700 1 --2- 172.21.15.7:0/369052243 >> v2:172.21.15.143:3300/0 conn(0x7fabcc0caa30 0x7fabcc0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabcbfff700 1 -- 172.21.15.7:0/369052243 >> v1:172.21.15.7:6789/0 conn(0x7fabcc0c00e0 legacy=0x7fabcc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabcbfff700 1 -- 172.21.15.7:0/369052243 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc0bfa30 con 0x7fabcc154ff0 2023-03-03T11:48:41.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.314+0000 7fabca7fc700 1 -- 172.21.15.7:0/369052243 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560422621 0 0) 0x7fabcc0c7310 con 0x7fabcc0c00e0 2023-03-03T11:48:41.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.315+0000 7fabcbfff700 1 --2- 172.21.15.7:0/369052243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 0x7fabcc0cae00 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0c81b0 tx=0x7fabc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4de999fb0f91590 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:41.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.315+0000 7fabca7fc700 1 -- 172.21.15.7:0/369052243 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc400e070 con 0x7fabcc154ff0 2023-03-03T11:48:41.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.315+0000 7fabca7fc700 1 -- 172.21.15.7:0/369052243 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc400b910 con 0x7fabcc154ff0 2023-03-03T11:48:41.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.315+0000 7fabca7fc700 1 -- 172.21.15.7:0/369052243 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc400bcc0 con 0x7fabcc154ff0 2023-03-03T11:48:41.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.315+0000 7fabd1cf2700 1 -- 172.21.15.7:0/369052243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 msgr2=0x7fabcc0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.315+0000 7fabd1cf2700 1 --2- 172.21.15.7:0/369052243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 0x7fabcc0cae00 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0c81b0 tx=0x7fabc400b330 comp rx=0 tx=0).stop 2023-03-03T11:48:41.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.315+0000 7fabd1cf2700 1 -- 172.21.15.7:0/369052243 shutdown_connections 2023-03-03T11:48:41.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.315+0000 7fabd1cf2700 1 --2- 172.21.15.7:0/369052243 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 0x7fabcc0cae00 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.315+0000 7fabd1cf2700 1 --2- 172.21.15.7:0/369052243 >> v2:172.21.15.143:3300/0 conn(0x7fabcc0caa30 0x7fabcc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.315+0000 7fabd1cf2700 1 -- 172.21.15.7:0/369052243 >> 172.21.15.7:0/369052243 conn(0x7fabcc066e40 msgr2=0x7fabcc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:48:41.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.316+0000 7fabd1cf2700 1 -- 172.21.15.7:0/369052243 shutdown_connections 2023-03-03T11:48:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.316+0000 7fabd1cf2700 1 -- 172.21.15.7:0/369052243 wait complete. 2023-03-03T11:48:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.317+0000 7fabd1cf2700 1 Processor -- start 2023-03-03T11:48:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.317+0000 7fabd1cf2700 1 -- start start 2023-03-03T11:48:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.318+0000 7fabd1cf2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabcc0c00e0 0x7fabcc156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.318+0000 7fabd1cf2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabcc0caa30 0x7fabcc1ed990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.318+0000 7fabd1cf2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 0x7fabcc1e6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.318+0000 7fabd1cf2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc14de00 con 0x7fabcc154ff0 2023-03-03T11:48:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.318+0000 7fabd1cf2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc157400 con 0x7fabcc0c00e0 2023-03-03T11:48:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.318+0000 7fabd1cf2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc0a9f60 con 0x7fabcc0caa30 2023-03-03T11:48:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.318+0000 7fabcbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 0x7fabcc1e6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.318+0000 7fabcbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 0x7fabcc1e6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37864/0 (socket says 172.21.15.7:37864) 2023-03-03T11:48:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.318+0000 7fabcbfff700 1 -- 172.21.15.7:0/956022709 learned_addr learned my addr 172.21.15.7:0/956022709 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:48:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabcb7fe700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabcc0c00e0 0x7fabcc156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabcaffd700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabcc0caa30 0x7fabcc1ed990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabcbfff700 1 -- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabcc0caa30 msgr2=0x7fabcc1ed990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabcbfff700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabcc0caa30 0x7fabcc1ed990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabcbfff700 1 -- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabcc0c00e0 msgr2=0x7fabcc156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabcbfff700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabcc0c00e0 0x7fabcc156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabcbfff700 1 -- 172.21.15.7:0/956022709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc4009b30 con 0x7fabcc154ff0 2023-03-03T11:48:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabcaffd700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabcc0caa30 0x7fabcc1ed990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:48:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabcbfff700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 0x7fabcc1e6d00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fabcc06b270 tx=0x7fabc4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabc3fff700 1 -- 172.21.15.7:0/956022709 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc4004930 con 0x7fabcc154ff0 2023-03-03T11:48:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabc3fff700 1 -- 172.21.15.7:0/956022709 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc40099b0 con 0x7fabcc154ff0 2023-03-03T11:48:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabd1cf2700 1 -- 172.21.15.7:0/956022709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabcc1547f0 con 0x7fabcc154ff0 2023-03-03T11:48:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabc3fff700 1 -- 172.21.15.7:0/956022709 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc401b650 con 0x7fabcc154ff0 2023-03-03T11:48:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.319+0000 7fabd1cf2700 1 -- 172.21.15.7:0/956022709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabcc154510 con 0x7fabcc154ff0 2023-03-03T11:48:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.320+0000 7fabd1cf2700 1 -- 172.21.15.7:0/956022709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabcc154ad0 con 0x7fabcc154ff0 2023-03-03T11:48:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.321+0000 7fabc3fff700 1 -- 172.21.15.7:0/956022709 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fabc4021030 con 0x7fabcc154ff0 2023-03-03T11:48:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.322+0000 7fabc3fff700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabac0682f0 0x7fabac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.322+0000 7fabc3fff700 1 -- 172.21.15.7:0/956022709 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fabc40874f0 con 0x7fabcc154ff0 2023-03-03T11:48:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.322+0000 7fabcb7fe700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabac0682f0 0x7fabac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.323+0000 7fabcb7fe700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabac0682f0 0x7fabac06a7a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009be0 tx=0x7fabbc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.326+0000 7fabc3fff700 1 -- 172.21.15.7:0/956022709 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fabc4053890 con 0x7fabcc154ff0 2023-03-03T11:48:41.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.351+0000 7f0201c2b700 1 Processor -- start 2023-03-03T11:48:41.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.351+0000 7f0201c2b700 1 -- start start 2023-03-03T11:48:41.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f0201c2b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01fc0caa30 0x7f01fc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f0201c2b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f01fc0c00e0 0x7f01fc154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f0201c2b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01fc0cae00 0x7f01fc0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f0201c2b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01fc0cb280 con 0x7f01fc0caa30 2023-03-03T11:48:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f0201c2b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01fc0bfd10 con 0x7f01fc0c00e0 2023-03-03T11:48:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f0201c2b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc0bfa30 con 0x7f01fc0cae00 2023-03-03T11:48:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f01fb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01fc0caa30 0x7f01fc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f01fb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01fc0caa30 0x7f01fc0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37868/0 (socket says 172.21.15.7:37868) 2023-03-03T11:48:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f01fb7fe700 1 -- 172.21.15.7:0/847048526 learned_addr learned my addr 172.21.15.7:0/847048526 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:48:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f01faffd700 1 --2- 172.21.15.7:0/847048526 >> v2:172.21.15.143:3300/0 conn(0x7f01fc0c00e0 0x7f01fc154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f01fbfff700 1 --2- 172.21.15.7:0/847048526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01fc0cae00 0x7f01fc0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f01fb7fe700 1 -- 172.21.15.7:0/847048526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01fc0cae00 msgr2=0x7f01fc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:48:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f01fb7fe700 1 --2- 172.21.15.7:0/847048526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01fc0cae00 0x7f01fc0c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f01fb7fe700 1 -- 172.21.15.7:0/847048526 >> v2:172.21.15.143:3300/0 conn(0x7f01fc0c00e0 msgr2=0x7f01fc154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:48:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f01fb7fe700 1 --2- 172.21.15.7:0/847048526 >> v2:172.21.15.143:3300/0 conn(0x7f01fc0c00e0 0x7f01fc154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.352+0000 7f01fb7fe700 1 -- 172.21.15.7:0/847048526 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc154510 con 0x7f01fc0caa30 2023-03-03T11:48:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f01fb7fe700 1 --2- 172.21.15.7:0/847048526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 0x7f01fc0c5c60 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f01fc0ca910 tx=0x7f01f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260a8b546458ec99 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f01fa7fc700 1 -- 172.21.15.7:0/847048526 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f400e070 con 0x7f01fc0caa30 2023-03-03T11:48:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f01fa7fc700 1 -- 172.21.15.7:0/847048526 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f400b9c0 con 0x7f01fc0caa30 2023-03-03T11:48:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f01fa7fc700 1 -- 172.21.15.7:0/847048526 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f400bd70 con 0x7f01fc0caa30 2023-03-03T11:48:41.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f0201c2b700 1 -- 172.21.15.7:0/847048526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 msgr2=0x7f01fc0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f0201c2b700 1 --2- 172.21.15.7:0/847048526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 0x7f01fc0c5c60 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f01fc0ca910 tx=0x7f01f400b330 comp rx=0 tx=0).stop 2023-03-03T11:48:41.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f0201c2b700 1 -- 172.21.15.7:0/847048526 shutdown_connections 2023-03-03T11:48:41.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f0201c2b700 1 --2- 172.21.15.7:0/847048526 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01fc0cae00 0x7f01fc0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f0201c2b700 1 --2- 172.21.15.7:0/847048526 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 0x7f01fc0c5c60 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f0201c2b700 1 --2- 172.21.15.7:0/847048526 >> v2:172.21.15.143:3300/0 conn(0x7f01fc0c00e0 0x7f01fc154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f0201c2b700 1 -- 172.21.15.7:0/847048526 >> 172.21.15.7:0/847048526 conn(0x7f01fc066e40 msgr2=0x7f01fc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:48:41.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f0201c2b700 1 -- 172.21.15.7:0/847048526 shutdown_connections 2023-03-03T11:48:41.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.353+0000 7f0201c2b700 1 -- 172.21.15.7:0/847048526 wait complete. 2023-03-03T11:48:41.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.355+0000 7f0201c2b700 1 Processor -- start 2023-03-03T11:48:41.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.355+0000 7f0201c2b700 1 -- start start 2023-03-03T11:48:41.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f0201c2b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01fc0c00e0 0x7f01fc1df080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f0201c2b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 0x7f01fc1ddcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f0201c2b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01fc0cae00 0x7f01fc0c2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f0201c2b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc158910 con 0x7f01fc0caa30 2023-03-03T11:48:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f0201c2b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc1df5c0 con 0x7f01fc0cae00 2023-03-03T11:48:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f0201c2b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01fc156900 con 0x7f01fc0c00e0 2023-03-03T11:48:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f01faffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 0x7f01fc1ddcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f01faffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 0x7f01fc1ddcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37876/0 (socket says 172.21.15.7:37876) 2023-03-03T11:48:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f01faffd700 1 -- 172.21.15.7:0/1768232927 learned_addr learned my addr 172.21.15.7:0/1768232927 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:48:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f01fb7fe700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01fc0c00e0 0x7f01fc1df080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.356+0000 7f01fbfff700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01fc0cae00 0x7f01fc0c2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f01faffd700 1 -- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01fc0c00e0 msgr2=0x7f01fc1df080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f01faffd700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01fc0c00e0 0x7f01fc1df080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f01faffd700 1 -- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01fc0cae00 msgr2=0x7f01fc0c2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f01faffd700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01fc0cae00 0x7f01fc0c2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f01faffd700 1 -- 172.21.15.7:0/1768232927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f4009b30 con 0x7f01fc0caa30 2023-03-03T11:48:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f01faffd700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 0x7f01fc1ddcb0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f01e400d800 tx=0x7f01e400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f01f3fff700 1 -- 172.21.15.7:0/1768232927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40125f0 con 0x7f01fc0caa30 2023-03-03T11:48:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f01f3fff700 1 -- 172.21.15.7:0/1768232927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e4012750 con 0x7f01fc0caa30 2023-03-03T11:48:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f0201c2b700 1 -- 172.21.15.7:0/1768232927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01fc154ad0 con 0x7f01fc0caa30 2023-03-03T11:48:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f01f3fff700 1 -- 172.21.15.7:0/1768232927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e4012b00 con 0x7f01fc0caa30 2023-03-03T11:48:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.357+0000 7f0201c2b700 1 -- 172.21.15.7:0/1768232927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01fc1547f0 con 0x7f01fc0caa30 2023-03-03T11:48:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.358+0000 7f0201c2b700 1 -- 172.21.15.7:0/1768232927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01fc0bf650 con 0x7f01fc0caa30 2023-03-03T11:48:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.358+0000 7f01f3fff700 1 -- 172.21.15.7:0/1768232927 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f01e4012c60 con 0x7f01fc0caa30 2023-03-03T11:48:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.359+0000 7f01f3fff700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01dc0682f0 0x7f01dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.359+0000 7f01f3fff700 1 -- 172.21.15.7:0/1768232927 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f01e4086080 con 0x7f01fc0caa30 2023-03-03T11:48:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.359+0000 7f01fb7fe700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01dc0682f0 0x7f01dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.360+0000 7f01fb7fe700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01dc0682f0 0x7f01dc06a7a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f01f400b370 tx=0x7f01f4009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.363+0000 7f01f3fff700 1 -- 172.21.15.7:0/1768232927 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f01e4052420 con 0x7f01fc0caa30 2023-03-03T11:48:41.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.565+0000 7fabd1cf2700 1 -- 172.21.15.7:0/956022709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabcc074bb0 con 0x7fabcc154ff0 2023-03-03T11:48:41.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.566+0000 7fabc3fff700 1 -- 172.21.15.7:0/956022709 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fabc4056eb0 con 0x7fabcc154ff0 2023-03-03T11:48:41.568 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:48:41.568 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:48:41.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 -- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabac0682f0 msgr2=0x7fabac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabac0682f0 0x7fabac06a7a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009be0 tx=0x7fabbc009210 comp rx=0 tx=0).stop 2023-03-03T11:48:41.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 -- 172.21.15.7:0/956022709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 msgr2=0x7fabcc1e6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 0x7fabcc1e6d00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fabcc06b270 tx=0x7fabc4004ad0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 -- 172.21.15.7:0/956022709 shutdown_connections 2023-03-03T11:48:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fabac0682f0 0x7fabac06a7a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabcc154ff0 0x7fabcc1e6d00 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabcc0c00e0 0x7fabcc156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 --2- 172.21.15.7:0/956022709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabcc0caa30 0x7fabcc1ed990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 -- 172.21.15.7:0/956022709 >> 172.21.15.7:0/956022709 conn(0x7fabcc066e40 msgr2=0x7fabcc1530e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:48:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 -- 172.21.15.7:0/956022709 shutdown_connections 2023-03-03T11:48:41.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.568+0000 7fabd1cf2700 1 -- 172.21.15.7:0/956022709 wait complete. 2023-03-03T11:48:41.572 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:48:41.583 INFO:tasks.mds_thrash.fs.[b]:waiting for 102 secs before thrashing 2023-03-03T11:48:41.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.601+0000 7f0201c2b700 1 -- 172.21.15.7:0/1768232927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01fc156ab0 con 0x7f01fc0caa30 2023-03-03T11:48:41.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.602+0000 7f01f3fff700 1 -- 172.21.15.7:0/1768232927 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f01fc156ab0 con 0x7f01fc0caa30 2023-03-03T11:48:41.604 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:48:41.604 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:48:41.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 -- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01dc0682f0 msgr2=0x7f01dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01dc0682f0 0x7f01dc06a7a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f01f400b370 tx=0x7f01f4009000 comp rx=0 tx=0).stop 2023-03-03T11:48:41.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 -- 172.21.15.7:0/1768232927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 msgr2=0x7f01fc1ddcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 0x7f01fc1ddcb0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f01e400d800 tx=0x7f01e400db10 comp rx=0 tx=0).stop 2023-03-03T11:48:41.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 -- 172.21.15.7:0/1768232927 shutdown_connections 2023-03-03T11:48:41.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01fc0cae00 0x7f01fc0c2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01fc0c00e0 0x7f01fc1df080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01dc0682f0 0x7f01dc06a7a0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 --2- 172.21.15.7:0/1768232927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01fc0caa30 0x7f01fc1ddcb0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 -- 172.21.15.7:0/1768232927 >> 172.21.15.7:0/1768232927 conn(0x7f01fc066e40 msgr2=0x7f01fc0c8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:48:41.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 -- 172.21.15.7:0/1768232927 shutdown_connections 2023-03-03T11:48:41.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.604+0000 7f0201c2b700 1 -- 172.21.15.7:0/1768232927 wait complete. 2023-03-03T11:48:41.608 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:48:41.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:48:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.781+0000 7ff8f772e700 1 Processor -- start 2023-03-03T11:48:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.781+0000 7ff8f772e700 1 -- start start 2023-03-03T11:48:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.782+0000 7ff8f772e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f00c5140 0x7ff8f0157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.782+0000 7ff8f772e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f006dad0 con 0x7ff8f01580e0 2023-03-03T11:48:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.782+0000 7ff8f772e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f006ddb0 con 0x7ff8f0151de0 2023-03-03T11:48:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.782+0000 7ff8f772e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f006e370 con 0x7ff8f00c5140 2023-03-03T11:48:41.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.782+0000 7ff8f5ccb700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff8f01580e0 0x7ff8f006a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60098/0 (socket says 172.21.15.7:60098) 2023-03-03T11:48:41.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.782+0000 7ff8f5ccb700 1 -- 172.21.15.7:0/3768320713 learned_addr learned my addr 172.21.15.7:0/3768320713 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:48:41.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.782+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939466631 0 0) 0x7ff8f006dad0 con 0x7ff8f01580e0 2023-03-03T11:48:41.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.782+0000 7ff8f4cc9700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f00c5140 0x7ff8f0157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc0035b0 con 0x7ff8f01580e0 2023-03-03T11:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705135690 0 0) 0x7ff8dc0035b0 con 0x7ff8f01580e0 2023-03-03T11:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f00c5140 msgr2=0x7ff8f0157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f00c5140 0x7ff8f0157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 >> v1:172.21.15.143:6789/0 conn(0x7ff8f0151de0 legacy=0x7ff8f00c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f006e090 con 0x7ff8f01580e0 2023-03-03T11:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff8e80019f0 con 0x7ff8f01580e0 2023-03-03T11:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 >> v1:172.21.15.7:6789/0 conn(0x7ff8f01580e0 legacy=0x7ff8f006a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8dc003aa0 0x7ff8dc003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f00c5140 0x7ff8dc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8dc004d90 0x7ff8dc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f006e370 con 0x7ff8dc004d90 2023-03-03T11:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc00baf0 con 0x7ff8dc003aa0 2023-03-03T11:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc00bcb0 con 0x7ff8f00c5140 2023-03-03T11:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8f54ca700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8dc003aa0 0x7ff8dc003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8f5ccb700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8dc004d90 0x7ff8dc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8f4cc9700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f00c5140 0x7ff8dc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8f5ccb700 1 -- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f00c5140 msgr2=0x7ff8dc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8f5ccb700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f00c5140 0x7ff8dc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8f5ccb700 1 -- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8dc003aa0 msgr2=0x7ff8dc003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:48:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8f5ccb700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8dc003aa0 0x7ff8dc003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8f5ccb700 1 -- 172.21.15.7:0/3768320713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc0035b0 con 0x7ff8dc004d90 2023-03-03T11:48:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.783+0000 7ff8f5ccb700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8dc004d90 0x7ff8dc005200 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800a830 tx=0x7ff8dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c6065f9e81633a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.784+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e800a2f0 con 0x7ff8dc004d90 2023-03-03T11:48:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.784+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e8012600 con 0x7ff8dc004d90 2023-03-03T11:48:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.784+0000 7ff8e7fff700 1 -- 172.21.15.7:0/3768320713 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80129b0 con 0x7ff8dc004d90 2023-03-03T11:48:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.784+0000 7ff8f772e700 1 -- 172.21.15.7:0/3768320713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8dc004d90 msgr2=0x7ff8dc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.784+0000 7ff8f772e700 1 --2- 172.21.15.7:0/3768320713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8dc004d90 0x7ff8dc005200 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800a830 tx=0x7ff8dc005880 comp rx=0 tx=0).stop 2023-03-03T11:48:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.784+0000 7ff8f5ccb700 1 -- 172.21.15.7:0/3768320713 reap_dead start 2023-03-03T11:48:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.784+0000 7ff8f772e700 1 -- 172.21.15.7:0/3768320713 shutdown_connections 2023-03-03T11:48:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.784+0000 7ff8f772e700 1 -- 172.21.15.7:0/3768320713 >> 172.21.15.7:0/3768320713 conn(0x7ff8f00a9430 msgr2=0x7ff8f00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:48:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.784+0000 7ff8f772e700 1 -- 172.21.15.7:0/3768320713 shutdown_connections 2023-03-03T11:48:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.784+0000 7ff8f772e700 1 -- 172.21.15.7:0/3768320713 wait complete. 2023-03-03T11:48:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.785+0000 7ff8f772e700 1 Processor -- start 2023-03-03T11:48:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f772e700 1 -- start start 2023-03-03T11:48:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f772e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f014a4d0 0x7ff8f0147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f772e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f014a8a0 0x7ff8f01ed020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f772e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f01ed560 0x7ff8f006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f772e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f014d0b0 con 0x7ff8f014a8a0 2023-03-03T11:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f772e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f014ca80 con 0x7ff8f014a4d0 2023-03-03T11:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f772e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f014c220 con 0x7ff8f01ed560 2023-03-03T11:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f4cc9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f014a8a0 0x7ff8f01ed020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f4cc9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f014a8a0 0x7ff8f01ed020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37896/0 (socket says 172.21.15.7:37896) 2023-03-03T11:48:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f4cc9700 1 -- 172.21.15.7:0/2054191018 learned_addr learned my addr 172.21.15.7:0/2054191018 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f5ccb700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f01ed560 0x7ff8f006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.786+0000 7ff8f54ca700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f014a4d0 0x7ff8f0147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8f4cc9700 1 -- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f01ed560 msgr2=0x7ff8f006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8f4cc9700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f01ed560 0x7ff8f006c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8f4cc9700 1 -- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f014a4d0 msgr2=0x7ff8f0147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8f4cc9700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f014a4d0 0x7ff8f0147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8f4cc9700 1 -- 172.21.15.7:0/2054191018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00c1c10 con 0x7ff8f014a8a0 2023-03-03T11:48:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8f4cc9700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f014a8a0 0x7ff8f01ed020 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0011aa0 tx=0x7ff8e0011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8e5ffb700 1 -- 172.21.15.7:0/2054191018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0011ee0 con 0x7ff8f014a8a0 2023-03-03T11:48:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8e5ffb700 1 -- 172.21.15.7:0/2054191018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e000f940 con 0x7ff8f014a8a0 2023-03-03T11:48:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8f772e700 1 -- 172.21.15.7:0/2054191018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f00c4c20 con 0x7ff8f014a8a0 2023-03-03T11:48:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8e5ffb700 1 -- 172.21.15.7:0/2054191018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e000fcf0 con 0x7ff8f014a8a0 2023-03-03T11:48:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.787+0000 7ff8f772e700 1 -- 172.21.15.7:0/2054191018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f0066de0 con 0x7ff8f014a8a0 2023-03-03T11:48:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.788+0000 7ff8f772e700 1 -- 172.21.15.7:0/2054191018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f0074bb0 con 0x7ff8f014a8a0 2023-03-03T11:48:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.789+0000 7ff8e5ffb700 1 -- 172.21.15.7:0/2054191018 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff8e0014440 con 0x7ff8f014a8a0 2023-03-03T11:48:41.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.789+0000 7ff8e5ffb700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8d40682f0 0x7ff8d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:48:41.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.789+0000 7ff8e5ffb700 1 -- 172.21.15.7:0/2054191018 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff8e00868d0 con 0x7ff8f014a8a0 2023-03-03T11:48:41.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.789+0000 7ff8f54ca700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8d40682f0 0x7ff8d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:48:41.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.790+0000 7ff8f54ca700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8d40682f0 0x7ff8d406a7a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec007fd0 tx=0x7ff8ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:48:41.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:41.793+0000 7ff8e5ffb700 1 -- 172.21.15.7:0/2054191018 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff8e0052cd0 con 0x7ff8f014a8a0 2023-03-03T11:48:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.031+0000 7ff8f772e700 1 -- 172.21.15.7:0/2054191018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8f0066bc0 con 0x7ff8f014a8a0 2023-03-03T11:48:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.031+0000 7ff8e5ffb700 1 -- 172.21.15.7:0/2054191018 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff8e00562f0 con 0x7ff8f014a8a0 2023-03-03T11:48:42.034 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:48:42.034 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:48:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 -- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8d40682f0 msgr2=0x7ff8d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8d40682f0 0x7ff8d406a7a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec007fd0 tx=0x7ff8ec009040 comp rx=0 tx=0).stop 2023-03-03T11:48:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 -- 172.21.15.7:0/2054191018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f014a8a0 msgr2=0x7ff8f01ed020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:48:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f014a8a0 0x7ff8f01ed020 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0011aa0 tx=0x7ff8e0011db0 comp rx=0 tx=0).stop 2023-03-03T11:48:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 -- 172.21.15.7:0/2054191018 shutdown_connections 2023-03-03T11:48:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8f01ed560 0x7ff8f006c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8f014a4d0 0x7ff8f0147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8d40682f0 0x7ff8d406a7a0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 --2- 172.21.15.7:0/2054191018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f014a8a0 0x7ff8f01ed020 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:48:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 -- 172.21.15.7:0/2054191018 >> 172.21.15.7:0/2054191018 conn(0x7ff8f00a9430 msgr2=0x7ff8f00a5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:48:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 -- 172.21.15.7:0/2054191018 shutdown_connections 2023-03-03T11:48:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:48:42.034+0000 7ff8f772e700 1 -- 172.21.15.7:0/2054191018 wait complete. 2023-03-03T11:48:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:48:42.047 INFO:tasks.mds_thrash.fs.[a]:waiting for 51 secs before thrashing 2023-03-03T11:48:49.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:48:49.118 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:19.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:19.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:33.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:49:33.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.184+0000 7f87cca76700 1 Processor -- start 2023-03-03T11:49:33.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.184+0000 7f87cca76700 1 -- start start 2023-03-03T11:49:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.185+0000 7f87cca76700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f87c80caa30 0x7f87c80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.185+0000 7f87cca76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80c00e0 0x7f87c8154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.185+0000 7f87cca76700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c80cb280 con 0x7f87c80cae00 2023-03-03T11:49:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.185+0000 7f87cca76700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c80c7310 con 0x7f87c80caa30 2023-03-03T11:49:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.185+0000 7f87cca76700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c80bfd10 con 0x7f87c80c00e0 2023-03-03T11:49:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c5d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80c00e0 0x7f87c8154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c659c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f87c80caa30 0x7f87c80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c5d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80c00e0 0x7f87c8154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:46000/0 (socket says 172.21.15.7:46000) 2023-03-03T11:49:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c6d9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f87c80cae00 0x7f87c80c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:35428/0 (socket says 172.21.15.7:35428) 2023-03-03T11:49:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c5d9b700 1 -- 172.21.15.7:0/1800591689 learned_addr learned my addr 172.21.15.7:0/1800591689 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:49:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c5d9b700 1 -- 172.21.15.7:0/1800591689 >> v2:172.21.15.143:3300/0 conn(0x7f87c80caa30 msgr2=0x7f87c80c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:49:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c5d9b700 1 --2- 172.21.15.7:0/1800591689 >> v2:172.21.15.143:3300/0 conn(0x7f87c80caa30 0x7f87c80c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:33.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c5d9b700 1 -- 172.21.15.7:0/1800591689 >> v1:172.21.15.143:6789/0 conn(0x7f87c80cae00 legacy=0x7f87c80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:33.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c5d9b700 1 -- 172.21.15.7:0/1800591689 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c80bfa30 con 0x7f87c80c00e0 2023-03-03T11:49:33.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c659c700 1 --2- 172.21.15.7:0/1800591689 >> v2:172.21.15.143:3300/0 conn(0x7f87c80caa30 0x7f87c80c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:49:33.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c5d9b700 1 --2- 172.21.15.7:0/1800591689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80c00e0 0x7f87c8154ff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f87b8006fd0 tx=0x7f87b8009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a274e116c4fa47d5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:49:33.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.186+0000 7f87c559a700 1 -- 172.21.15.7:0/1800591689 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8005e30 con 0x7f87c80c00e0 2023-03-03T11:49:33.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.187+0000 7f87c559a700 1 -- 172.21.15.7:0/1800591689 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b8004030 con 0x7f87c80c00e0 2023-03-03T11:49:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.187+0000 7f87c559a700 1 -- 172.21.15.7:0/1800591689 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b80043e0 con 0x7f87c80c00e0 2023-03-03T11:49:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.187+0000 7f87cca76700 1 -- 172.21.15.7:0/1800591689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80c00e0 msgr2=0x7f87c8154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.187+0000 7f87cca76700 1 --2- 172.21.15.7:0/1800591689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80c00e0 0x7f87c8154ff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f87b8006fd0 tx=0x7f87b8009bd0 comp rx=0 tx=0).stop 2023-03-03T11:49:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.187+0000 7f87cca76700 1 -- 172.21.15.7:0/1800591689 shutdown_connections 2023-03-03T11:49:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.187+0000 7f87cca76700 1 --2- 172.21.15.7:0/1800591689 >> v2:172.21.15.143:3300/0 conn(0x7f87c80caa30 0x7f87c80c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.187+0000 7f87cca76700 1 --2- 172.21.15.7:0/1800591689 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80c00e0 0x7f87c8154ff0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.187+0000 7f87cca76700 1 -- 172.21.15.7:0/1800591689 >> 172.21.15.7:0/1800591689 conn(0x7f87c8066e40 msgr2=0x7f87c8071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:49:33.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.187+0000 7f87cca76700 1 -- 172.21.15.7:0/1800591689 shutdown_connections 2023-03-03T11:49:33.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.187+0000 7f87cca76700 1 -- 172.21.15.7:0/1800591689 wait complete. 2023-03-03T11:49:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.188+0000 7f87cca76700 1 Processor -- start 2023-03-03T11:49:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87cca76700 1 -- start start 2023-03-03T11:49:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87cca76700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c80c00e0 0x7f87c814c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87cca76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80caa30 0x7f87c80c9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87cca76700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87c80cae00 0x7f87c8156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87cca76700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c8155e80 con 0x7f87c80cae00 2023-03-03T11:49:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87cca76700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c8155870 con 0x7f87c80c00e0 2023-03-03T11:49:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87cca76700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c81edc50 con 0x7f87c80caa30 2023-03-03T11:49:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87c659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c80c00e0 0x7f87c814c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87c6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87c80cae00 0x7f87c8156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87c659c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c80c00e0 0x7f87c814c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:33706/0 (socket says 172.21.15.7:33706) 2023-03-03T11:49:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87c5d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80caa30 0x7f87c80c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87c659c700 1 -- 172.21.15.7:0/3763445444 learned_addr learned my addr 172.21.15.7:0/3763445444 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:49:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.189+0000 7f87c6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87c80cae00 0x7f87c8156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37936/0 (socket says 172.21.15.7:37936) 2023-03-03T11:49:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87c6d9d700 1 -- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80caa30 msgr2=0x7f87c80c9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87c6d9d700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80caa30 0x7f87c80c9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87c6d9d700 1 -- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c80c00e0 msgr2=0x7f87c814c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87c6d9d700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c80c00e0 0x7f87c814c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87c6d9d700 1 -- 172.21.15.7:0/3763445444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87bc0094a0 con 0x7f87c80cae00 2023-03-03T11:49:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87c6d9d700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87c80cae00 0x7f87c8156ac0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f87c000ca70 tx=0x7f87c000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:49:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87b6ffd700 1 -- 172.21.15.7:0/3763445444 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c000a9e0 con 0x7f87c80cae00 2023-03-03T11:49:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87b6ffd700 1 -- 172.21.15.7:0/3763445444 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c000ab40 con 0x7f87c80cae00 2023-03-03T11:49:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87cca76700 1 -- 172.21.15.7:0/3763445444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b8009500 con 0x7f87c80cae00 2023-03-03T11:49:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87b6ffd700 1 -- 172.21.15.7:0/3763445444 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c0007070 con 0x7f87c80cae00 2023-03-03T11:49:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.190+0000 7f87cca76700 1 -- 172.21.15.7:0/3763445444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c81547f0 con 0x7f87c80cae00 2023-03-03T11:49:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.191+0000 7f87cca76700 1 -- 172.21.15.7:0/3763445444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c8154510 con 0x7f87c80cae00 2023-03-03T11:49:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.191+0000 7f87b6ffd700 1 -- 172.21.15.7:0/3763445444 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f87c0019030 con 0x7f87c80cae00 2023-03-03T11:49:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.192+0000 7f87b6ffd700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87a80682f0 0x7f87a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:49:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.192+0000 7f87b6ffd700 1 -- 172.21.15.7:0/3763445444 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f87c0085c30 con 0x7f87c80cae00 2023-03-03T11:49:33.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.192+0000 7f87c659c700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87a80682f0 0x7f87a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:49:33.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.193+0000 7f87c659c700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87a80682f0 0x7f87a806a7a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f87c80ca180 tx=0x7f87bc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:49:33.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.196+0000 7f87b6ffd700 1 -- 172.21.15.7:0/3763445444 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f87c004b5e0 con 0x7f87c80cae00 2023-03-03T11:49:33.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.432+0000 7f87cca76700 1 -- 172.21.15.7:0/3763445444 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87c8154ad0 con 0x7f87c80cae00 2023-03-03T11:49:33.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.433+0000 7f87b6ffd700 1 -- 172.21.15.7:0/3763445444 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f87c0017090 con 0x7f87c80cae00 2023-03-03T11:49:33.435 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:49:33.435 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:49:33.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 -- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87a80682f0 msgr2=0x7f87a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:33.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87a80682f0 0x7f87a806a7a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f87c80ca180 tx=0x7f87bc009210 comp rx=0 tx=0).stop 2023-03-03T11:49:33.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 -- 172.21.15.7:0/3763445444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87c80cae00 msgr2=0x7f87c8156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:49:33.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87c80cae00 0x7f87c8156ac0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f87c000ca70 tx=0x7f87c000cd80 comp rx=0 tx=0).stop 2023-03-03T11:49:33.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 -- 172.21.15.7:0/3763445444 shutdown_connections 2023-03-03T11:49:33.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87a80682f0 0x7f87a806a7a0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:33.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87c80cae00 0x7f87c8156ac0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:33.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87c80c00e0 0x7f87c814c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:33.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 --2- 172.21.15.7:0/3763445444 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87c80caa30 0x7f87c80c9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:49:33.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 -- 172.21.15.7:0/3763445444 >> 172.21.15.7:0/3763445444 conn(0x7f87c8066e40 msgr2=0x7f87c814fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:49:33.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 -- 172.21.15.7:0/3763445444 shutdown_connections 2023-03-03T11:49:33.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:49:33.435+0000 7f87cca76700 1 -- 172.21.15.7:0/3763445444 wait complete. 2023-03-03T11:49:33.439 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:49:33.449 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2023-03-03T11:49:49.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:49:49.318 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T11:50:00.000+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T11:50:19.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:19.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:23.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:50:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.722+0000 7fb2f4b82700 1 Processor -- start 2023-03-03T11:50:23.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.722+0000 7fb2f4b82700 1 -- start start 2023-03-03T11:50:23.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2f4b82700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb2f0071e80 0x7fb2f0151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2f4b82700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2f014e900 0x7fb2f015ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2f4b82700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f01510d0 con 0x7fb2f0072250 2023-03-03T11:50:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2f4b82700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f00c3740 con 0x7fb2f014e900 2023-03-03T11:50:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2f4b82700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f0070670 con 0x7fb2f0071e80 2023-03-03T11:50:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2eed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2f014e900 0x7fb2f015ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2eed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2f014e900 0x7fb2f015ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37972/0 (socket says 172.21.15.7:37972) 2023-03-03T11:50:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2eed9d700 1 -- 172.21.15.7:0/2579884294 learned_addr learned my addr 172.21.15.7:0/2579884294 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:50:23.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2ee59c700 1 --2- 172.21.15.7:0/2579884294 >> v2:172.21.15.143:3300/0 conn(0x7fb2f0071e80 0x7fb2f0151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2eed9d700 1 -- 172.21.15.7:0/2579884294 >> v2:172.21.15.143:3300/0 conn(0x7fb2f0071e80 msgr2=0x7fb2f0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:50:23.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2eed9d700 1 --2- 172.21.15.7:0/2579884294 >> v2:172.21.15.143:3300/0 conn(0x7fb2f0071e80 0x7fb2f0151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2eed9d700 1 -- 172.21.15.7:0/2579884294 >> v1:172.21.15.7:6789/0 conn(0x7fb2f0072250 legacy=0x7fb2f014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2eed9d700 1 -- 172.21.15.7:0/2579884294 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f00c39e0 con 0x7fb2f014e900 2023-03-03T11:50:23.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.723+0000 7fb2ed59a700 1 -- 172.21.15.7:0/2579884294 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566690513 0 0) 0x7fb2f01510d0 con 0x7fb2f0072250 2023-03-03T11:50:23.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.724+0000 7fb2eed9d700 1 --2- 172.21.15.7:0/2579884294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f014e900 0x7fb2f015ba50 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fb2f006bd10 tx=0x7fb2e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323281625acf84c3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.724+0000 7fb2ed59a700 1 -- 172.21.15.7:0/2579884294 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e800e070 con 0x7fb2f014e900 2023-03-03T11:50:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.724+0000 7fb2ed59a700 1 -- 172.21.15.7:0/2579884294 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e800b910 con 0x7fb2f014e900 2023-03-03T11:50:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.724+0000 7fb2ed59a700 1 -- 172.21.15.7:0/2579884294 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e800bcc0 con 0x7fb2f014e900 2023-03-03T11:50:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.724+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2579884294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f014e900 msgr2=0x7fb2f015ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.724+0000 7fb2f4b82700 1 --2- 172.21.15.7:0/2579884294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f014e900 0x7fb2f015ba50 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fb2f006bd10 tx=0x7fb2e800b330 comp rx=0 tx=0).stop 2023-03-03T11:50:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.724+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2579884294 shutdown_connections 2023-03-03T11:50:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.724+0000 7fb2f4b82700 1 --2- 172.21.15.7:0/2579884294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f014e900 0x7fb2f015ba50 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.724+0000 7fb2f4b82700 1 --2- 172.21.15.7:0/2579884294 >> v2:172.21.15.143:3300/0 conn(0x7fb2f0071e80 0x7fb2f0151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.724+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2579884294 >> 172.21.15.7:0/2579884294 conn(0x7fb2f00a94d0 msgr2=0x7fb2f00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:23.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.725+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2579884294 shutdown_connections 2023-03-03T11:50:23.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.725+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2579884294 wait complete. 2023-03-03T11:50:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.726+0000 7fb2f4b82700 1 Processor -- start 2023-03-03T11:50:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.726+0000 7fb2f4b82700 1 -- start start 2023-03-03T11:50:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2f4b82700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f0071e80 0x7fb2f014bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2f4b82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f0072250 0x7fb2f01ed210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2f4b82700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f014e900 0x7fb2f01e6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2f4b82700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f0150250 con 0x7fb2f0071e80 2023-03-03T11:50:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2f4b82700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f01510f0 con 0x7fb2f014e900 2023-03-03T11:50:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2f4b82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f01ea780 con 0x7fb2f0072250 2023-03-03T11:50:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2ee59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f0071e80 0x7fb2f014bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2ee59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f0071e80 0x7fb2f014bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37974/0 (socket says 172.21.15.7:37974) 2023-03-03T11:50:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2ee59c700 1 -- 172.21.15.7:0/2503785584 learned_addr learned my addr 172.21.15.7:0/2503785584 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:50:23.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2eed9d700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f014e900 0x7fb2f01e6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2edd9b700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f0072250 0x7fb2f01ed210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2ee59c700 1 -- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f0072250 msgr2=0x7fb2f01ed210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2ee59c700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f0072250 0x7fb2f01ed210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2ee59c700 1 -- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f014e900 msgr2=0x7fb2f01e6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2ee59c700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f014e900 0x7fb2f01e6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.727+0000 7fb2ee59c700 1 -- 172.21.15.7:0/2503785584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e8009b30 con 0x7fb2f0071e80 2023-03-03T11:50:23.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.728+0000 7fb2ee59c700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f0071e80 0x7fb2f014bef0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fb2e400e870 tx=0x7fb2e400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.728+0000 7fb2deffd700 1 -- 172.21.15.7:0/2503785584 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4009590 con 0x7fb2f0071e80 2023-03-03T11:50:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.728+0000 7fb2deffd700 1 -- 172.21.15.7:0/2503785584 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e4013680 con 0x7fb2f0071e80 2023-03-03T11:50:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.728+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2503785584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f006c9a0 con 0x7fb2f0071e80 2023-03-03T11:50:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.728+0000 7fb2deffd700 1 -- 172.21.15.7:0/2503785584 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e40139f0 con 0x7fb2f0071e80 2023-03-03T11:50:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.728+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2503785584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f0151300 con 0x7fb2f0071e80 2023-03-03T11:50:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.729+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2503785584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f006cb30 con 0x7fb2f0071e80 2023-03-03T11:50:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.730+0000 7fb2deffd700 1 -- 172.21.15.7:0/2503785584 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb2e4019030 con 0x7fb2f0071e80 2023-03-03T11:50:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.730+0000 7fb2deffd700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2d00682f0 0x7fb2d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.730+0000 7fb2deffd700 1 -- 172.21.15.7:0/2503785584 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb2e4085de0 con 0x7fb2f0071e80 2023-03-03T11:50:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.730+0000 7fb2edd9b700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2d00682f0 0x7fb2d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.731+0000 7fb2edd9b700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2d00682f0 0x7fb2d006a7a0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0145260 tx=0x7fb2e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.734+0000 7fb2deffd700 1 -- 172.21.15.7:0/2503785584 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb2e4052180 con 0x7fb2f0071e80 2023-03-03T11:50:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.977+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2503785584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2f0074bb0 con 0x7fb2f0071e80 2023-03-03T11:50:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.978+0000 7fb2deffd700 1 -- 172.21.15.7:0/2503785584 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb2e40557a0 con 0x7fb2f0071e80 2023-03-03T11:50:23.980 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:50:23.980 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:50:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2d00682f0 msgr2=0x7fb2d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2d00682f0 0x7fb2d006a7a0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0145260 tx=0x7fb2e0009040 comp rx=0 tx=0).stop 2023-03-03T11:50:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2503785584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f0071e80 msgr2=0x7fb2f014bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f0071e80 0x7fb2f014bef0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fb2e400e870 tx=0x7fb2e400eb80 comp rx=0 tx=0).stop 2023-03-03T11:50:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2503785584 shutdown_connections 2023-03-03T11:50:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2f014e900 0x7fb2f01e6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2d00682f0 0x7fb2d006a7a0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2f0071e80 0x7fb2f014bef0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 --2- 172.21.15.7:0/2503785584 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2f0072250 0x7fb2f01ed210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2503785584 >> 172.21.15.7:0/2503785584 conn(0x7fb2f00a94d0 msgr2=0x7fb2f0151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2503785584 shutdown_connections 2023-03-03T11:50:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:23.980+0000 7fb2f4b82700 1 -- 172.21.15.7:0/2503785584 wait complete. 2023-03-03T11:50:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:50:23.993 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T11:50:34.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:50:35.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.131+0000 7f7b73565700 1 Processor -- start 2023-03-03T11:50:35.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.131+0000 7f7b73565700 1 -- start start 2023-03-03T11:50:35.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.132+0000 7f7b73565700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7b6c151ab0 0x7f7b6c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:35.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.132+0000 7f7b73565700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b6c06c5b0 0x7f7b6c157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:35.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.132+0000 7f7b73565700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c06d780 con 0x7f7b6c073c30 2023-03-03T11:50:35.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.132+0000 7f7b73565700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6c06cb70 con 0x7f7b6c06c5b0 2023-03-03T11:50:35.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.132+0000 7f7b73565700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b6c06ce50 con 0x7f7b6c151ab0 2023-03-03T11:50:35.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.132+0000 7f7b71b02700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b6c06c5b0 0x7f7b6c157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:35.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.132+0000 7f7b71b02700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b6c06c5b0 0x7f7b6c157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37996/0 (socket says 172.21.15.7:37996) 2023-03-03T11:50:35.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.132+0000 7f7b71b02700 1 -- 172.21.15.7:0/2138109402 learned_addr learned my addr 172.21.15.7:0/2138109402 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:50:35.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.132+0000 7f7b70b00700 1 --1- 172.21.15.7:0/2138109402 >> v1:172.21.15.7:6789/0 conn(0x7f7b6c073c30 0x7f7b6c06bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60204/0 (socket says 172.21.15.7:60204) 2023-03-03T11:50:35.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b71301700 1 --2- 172.21.15.7:0/2138109402 >> v2:172.21.15.143:3300/0 conn(0x7f7b6c151ab0 0x7f7b6c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:35.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b71b02700 1 -- 172.21.15.7:0/2138109402 >> v2:172.21.15.143:3300/0 conn(0x7f7b6c151ab0 msgr2=0x7f7b6c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:50:35.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b71b02700 1 --2- 172.21.15.7:0/2138109402 >> v2:172.21.15.143:3300/0 conn(0x7f7b6c151ab0 0x7f7b6c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:35.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b71b02700 1 -- 172.21.15.7:0/2138109402 >> v1:172.21.15.7:6789/0 conn(0x7f7b6c073c30 legacy=0x7f7b6c06bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:35.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b71b02700 1 -- 172.21.15.7:0/2138109402 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c06d130 con 0x7f7b6c06c5b0 2023-03-03T11:50:35.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b63fff700 1 -- 172.21.15.7:0/2138109402 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319235853 0 0) 0x7f7b6c06d780 con 0x7f7b6c073c30 2023-03-03T11:50:35.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b71301700 1 --2- 172.21.15.7:0/2138109402 >> v2:172.21.15.143:3300/0 conn(0x7f7b6c151ab0 0x7f7b6c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T11:50:35.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b71b02700 1 --2- 172.21.15.7:0/2138109402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c06c5b0 0x7f7b6c157dc0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c06bb20 tx=0x7f7b6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9af09245e49b16b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:35.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b63fff700 1 -- 172.21.15.7:0/2138109402 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6400e070 con 0x7f7b6c06c5b0 2023-03-03T11:50:35.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b63fff700 1 -- 172.21.15.7:0/2138109402 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6400b9c0 con 0x7f7b6c06c5b0 2023-03-03T11:50:35.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.133+0000 7f7b63fff700 1 -- 172.21.15.7:0/2138109402 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6400bd70 con 0x7f7b6c06c5b0 2023-03-03T11:50:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.134+0000 7f7b73565700 1 -- 172.21.15.7:0/2138109402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c06c5b0 msgr2=0x7f7b6c157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.134+0000 7f7b73565700 1 --2- 172.21.15.7:0/2138109402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c06c5b0 0x7f7b6c157dc0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c06bb20 tx=0x7f7b6400b330 comp rx=0 tx=0).stop 2023-03-03T11:50:35.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.134+0000 7f7b73565700 1 -- 172.21.15.7:0/2138109402 shutdown_connections 2023-03-03T11:50:35.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.134+0000 7f7b73565700 1 --2- 172.21.15.7:0/2138109402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c06c5b0 0x7f7b6c157dc0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:35.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.134+0000 7f7b73565700 1 --2- 172.21.15.7:0/2138109402 >> v2:172.21.15.143:3300/0 conn(0x7f7b6c151ab0 0x7f7b6c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:35.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.134+0000 7f7b73565700 1 -- 172.21.15.7:0/2138109402 >> 172.21.15.7:0/2138109402 conn(0x7f7b6c0a94d0 msgr2=0x7f7b6c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:35.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.134+0000 7f7b73565700 1 -- 172.21.15.7:0/2138109402 shutdown_connections 2023-03-03T11:50:35.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.134+0000 7f7b73565700 1 -- 172.21.15.7:0/2138109402 wait complete. 2023-03-03T11:50:35.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.135+0000 7f7b73565700 1 Processor -- start 2023-03-03T11:50:35.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.135+0000 7f7b73565700 1 -- start start 2023-03-03T11:50:35.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b73565700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b6c06c5b0 0x7f7b6c14e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:35.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b73565700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b6c073c30 0x7f7b6c148680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:35.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b73565700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c151ab0 0x7f7b6c148bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:35.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b73565700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c0c0bc0 con 0x7f7b6c151ab0 2023-03-03T11:50:35.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b73565700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c153a50 con 0x7f7b6c073c30 2023-03-03T11:50:35.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b73565700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c0bffb0 con 0x7f7b6c06c5b0 2023-03-03T11:50:35.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b71b02700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c151ab0 0x7f7b6c148bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:35.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b71b02700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c151ab0 0x7f7b6c148bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38002/0 (socket says 172.21.15.7:38002) 2023-03-03T11:50:35.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b71b02700 1 -- 172.21.15.7:0/65254730 learned_addr learned my addr 172.21.15.7:0/65254730 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:50:35.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b70b00700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b6c073c30 0x7f7b6c148680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:35.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b71301700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b6c06c5b0 0x7f7b6c14e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:35.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b71b02700 1 -- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b6c06c5b0 msgr2=0x7f7b6c14e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:35.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b71b02700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b6c06c5b0 0x7f7b6c14e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:35.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b71b02700 1 -- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b6c073c30 msgr2=0x7f7b6c148680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:35.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b71b02700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b6c073c30 0x7f7b6c148680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:35.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b71b02700 1 -- 172.21.15.7:0/65254730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b64009b30 con 0x7f7b6c151ab0 2023-03-03T11:50:35.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b71b02700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c151ab0 0x7f7b6c148bc0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400b730 tx=0x7f7b64004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:35.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b61ffb700 1 -- 172.21.15.7:0/65254730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b640099b0 con 0x7f7b6c151ab0 2023-03-03T11:50:35.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b61ffb700 1 -- 172.21.15.7:0/65254730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b640092e0 con 0x7f7b6c151ab0 2023-03-03T11:50:35.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b73565700 1 -- 172.21.15.7:0/65254730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c151e80 con 0x7f7b6c151ab0 2023-03-03T11:50:35.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.136+0000 7f7b61ffb700 1 -- 172.21.15.7:0/65254730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6401c640 con 0x7f7b6c151ab0 2023-03-03T11:50:35.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.137+0000 7f7b73565700 1 -- 172.21.15.7:0/65254730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c06d410 con 0x7f7b6c151ab0 2023-03-03T11:50:35.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.138+0000 7f7b73565700 1 -- 172.21.15.7:0/65254730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c152010 con 0x7f7b6c151ab0 2023-03-03T11:50:35.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.138+0000 7f7b61ffb700 1 -- 172.21.15.7:0/65254730 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7b64022030 con 0x7f7b6c151ab0 2023-03-03T11:50:35.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.139+0000 7f7b61ffb700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b500682f0 0x7f7b5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.139+0000 7f7b61ffb700 1 -- 172.21.15.7:0/65254730 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7b64087ab0 con 0x7f7b6c151ab0 2023-03-03T11:50:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.139+0000 7f7b71301700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b500682f0 0x7f7b5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.139+0000 7f7b71301700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b500682f0 0x7f7b5006a7a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c006fd0 tx=0x7f7b5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:35.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.143+0000 7f7b61ffb700 1 -- 172.21.15.7:0/65254730 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7b64053e50 con 0x7f7b6c151ab0 2023-03-03T11:50:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.378+0000 7f7b73565700 1 -- 172.21.15.7:0/65254730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b6c074bb0 con 0x7f7b6c151ab0 2023-03-03T11:50:35.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.379+0000 7f7b61ffb700 1 -- 172.21.15.7:0/65254730 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7b64057470 con 0x7f7b6c151ab0 2023-03-03T11:50:35.381 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:50:35.381 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:50:35.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 -- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b500682f0 msgr2=0x7f7b5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b500682f0 0x7f7b5006a7a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c006fd0 tx=0x7f7b5c008040 comp rx=0 tx=0).stop 2023-03-03T11:50:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 -- 172.21.15.7:0/65254730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c151ab0 msgr2=0x7f7b6c148bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c151ab0 0x7f7b6c148bc0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400b730 tx=0x7f7b64004ad0 comp rx=0 tx=0).stop 2023-03-03T11:50:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 -- 172.21.15.7:0/65254730 shutdown_connections 2023-03-03T11:50:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b500682f0 0x7f7b5006a7a0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6c151ab0 0x7f7b6c148bc0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b6c06c5b0 0x7f7b6c14e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 --2- 172.21.15.7:0/65254730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b6c073c30 0x7f7b6c148680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 -- 172.21.15.7:0/65254730 >> 172.21.15.7:0/65254730 conn(0x7f7b6c0a94d0 msgr2=0x7f7b6c0c3070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 -- 172.21.15.7:0/65254730 shutdown_connections 2023-03-03T11:50:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:35.381+0000 7f7b73565700 1 -- 172.21.15.7:0/65254730 wait complete. 2023-03-03T11:50:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:50:35.394 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-03-03T11:50:41.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:50:41.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.587+0000 7f4cba752700 1 Processor -- start 2023-03-03T11:50:41.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.587+0000 7f4cba752700 1 -- start start 2023-03-03T11:50:41.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.588+0000 7f4cba752700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4cb4071e80 0x7f4cb4151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.588+0000 7f4cba752700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb4072250 0x7f4cb414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.588+0000 7f4cba752700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb41510d0 con 0x7f4cb40c1ce0 2023-03-03T11:50:41.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.588+0000 7f4cba752700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb40c3740 con 0x7f4cb4071e80 2023-03-03T11:50:41.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.588+0000 7f4cba752700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb4070670 con 0x7f4cb4072250 2023-03-03T11:50:41.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.588+0000 7f4cb37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb4072250 0x7f4cb414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.588+0000 7f4cb3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4cb4071e80 0x7f4cb4151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.588+0000 7f4cb3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4cb4071e80 0x7f4cb4151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:33788/0 (socket says 172.21.15.7:33788) 2023-03-03T11:50:41.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.588+0000 7f4cb8cef700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f4cb40c1ce0 0x7f4cb41546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:35516/0 (socket says 172.21.15.7:35516) 2023-03-03T11:50:41.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.588+0000 7f4cb3fff700 1 -- 172.21.15.7:0/47037697 learned_addr learned my addr 172.21.15.7:0/47037697 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:50:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.589+0000 7f4cb37fe700 1 -- 172.21.15.7:0/47037697 >> v2:172.21.15.143:3300/0 conn(0x7f4cb4071e80 msgr2=0x7f4cb4151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:50:41.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.589+0000 7f4cb37fe700 1 --2- 172.21.15.7:0/47037697 >> v2:172.21.15.143:3300/0 conn(0x7f4cb4071e80 0x7f4cb4151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.589+0000 7f4cb37fe700 1 -- 172.21.15.7:0/47037697 >> v1:172.21.15.143:6789/0 conn(0x7f4cb40c1ce0 legacy=0x7f4cb41546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.589+0000 7f4cb2ffd700 1 -- 172.21.15.7:0/47037697 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018355850 0 0) 0x7f4cb41510d0 con 0x7f4cb40c1ce0 2023-03-03T11:50:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.589+0000 7f4cb37fe700 1 -- 172.21.15.7:0/47037697 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb40c39e0 con 0x7f4cb4072250 2023-03-03T11:50:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.589+0000 7f4cb3fff700 1 --2- 172.21.15.7:0/47037697 >> v2:172.21.15.143:3300/0 conn(0x7f4cb4071e80 0x7f4cb4151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:50:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.589+0000 7f4cb37fe700 1 --2- 172.21.15.7:0/47037697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb4072250 0x7f4cb414e1f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009840 tx=0x7f4c9c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a91753b8509815d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.589+0000 7f4cb2ffd700 1 -- 172.21.15.7:0/47037697 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c011830 con 0x7f4cb4072250 2023-03-03T11:50:41.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.589+0000 7f4cb2ffd700 1 -- 172.21.15.7:0/47037697 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c011990 con 0x7f4cb4072250 2023-03-03T11:50:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.589+0000 7f4cb2ffd700 1 -- 172.21.15.7:0/47037697 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c011d20 con 0x7f4cb4072250 2023-03-03T11:50:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.590+0000 7f4cba752700 1 -- 172.21.15.7:0/47037697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb4072250 msgr2=0x7f4cb414e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.590+0000 7f4cba752700 1 --2- 172.21.15.7:0/47037697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb4072250 0x7f4cb414e1f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009840 tx=0x7f4c9c009b50 comp rx=0 tx=0).stop 2023-03-03T11:50:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.590+0000 7f4cba752700 1 -- 172.21.15.7:0/47037697 shutdown_connections 2023-03-03T11:50:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.590+0000 7f4cba752700 1 --2- 172.21.15.7:0/47037697 >> v2:172.21.15.143:3300/0 conn(0x7f4cb4071e80 0x7f4cb4151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.590+0000 7f4cba752700 1 --2- 172.21.15.7:0/47037697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb4072250 0x7f4cb414e1f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.590+0000 7f4cba752700 1 -- 172.21.15.7:0/47037697 >> 172.21.15.7:0/47037697 conn(0x7f4cb40a94d0 msgr2=0x7f4cb40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.590+0000 7f4cba752700 1 -- 172.21.15.7:0/47037697 shutdown_connections 2023-03-03T11:50:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.590+0000 7f4cba752700 1 -- 172.21.15.7:0/47037697 wait complete. 2023-03-03T11:50:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.591+0000 7f4cba752700 1 Processor -- start 2023-03-03T11:50:41.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.591+0000 7f4cba752700 1 -- start start 2023-03-03T11:50:41.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cba752700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cb4071e80 0x7f4cb40aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cba752700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb4072250 0x7f4cb40aa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cba752700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb40c1ce0 0x7f4cb414ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cba752700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb414b7f0 con 0x7f4cb4072250 2023-03-03T11:50:41.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cba752700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb41dc560 con 0x7f4cb4071e80 2023-03-03T11:50:41.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cba752700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb40c3030 con 0x7f4cb40c1ce0 2023-03-03T11:50:41.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb4072250 0x7f4cb40aa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb4072250 0x7f4cb40aa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38020/0 (socket says 172.21.15.7:38020) 2023-03-03T11:50:41.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb37fe700 1 -- 172.21.15.7:0/2950917762 learned_addr learned my addr 172.21.15.7:0/2950917762 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:50:41.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb3fff700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cb4071e80 0x7f4cb40aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb8cef700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb40c1ce0 0x7f4cb414ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb37fe700 1 -- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb40c1ce0 msgr2=0x7f4cb414ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb37fe700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb40c1ce0 0x7f4cb414ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb37fe700 1 -- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cb4071e80 msgr2=0x7f4cb40aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb37fe700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cb4071e80 0x7f4cb40aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb37fe700 1 -- 172.21.15.7:0/2950917762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca400b7f0 con 0x7f4cb4072250 2023-03-03T11:50:41.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb8cef700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb40c1ce0 0x7f4cb414ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:50:41.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.592+0000 7f4cb37fe700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb4072250 0x7f4cb40aa710 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009fd0 tx=0x7f4c9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:41.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.593+0000 7f4cb0ff9700 1 -- 172.21.15.7:0/2950917762 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0038a0 con 0x7f4cb4072250 2023-03-03T11:50:41.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.593+0000 7f4cb0ff9700 1 -- 172.21.15.7:0/2950917762 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c01f070 con 0x7f4cb4072250 2023-03-03T11:50:41.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.593+0000 7f4cba752700 1 -- 172.21.15.7:0/2950917762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9c0094a0 con 0x7f4cb4072250 2023-03-03T11:50:41.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.593+0000 7f4cb0ff9700 1 -- 172.21.15.7:0/2950917762 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c003c40 con 0x7f4cb4072250 2023-03-03T11:50:41.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.593+0000 7f4cba752700 1 -- 172.21.15.7:0/2950917762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb406c9a0 con 0x7f4cb4072250 2023-03-03T11:50:41.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.594+0000 7f4cba752700 1 -- 172.21.15.7:0/2950917762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb4151300 con 0x7f4cb4072250 2023-03-03T11:50:41.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.594+0000 7f4cb0ff9700 1 -- 172.21.15.7:0/2950917762 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4c9c00f050 con 0x7f4cb4072250 2023-03-03T11:50:41.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.595+0000 7f4cb0ff9700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 0x7f4c9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.595+0000 7f4cb0ff9700 1 -- 172.21.15.7:0/2950917762 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4c9c021070 con 0x7f4cb4072250 2023-03-03T11:50:41.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.595+0000 7f4cb3fff700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 0x7f4c9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.595+0000 7f4cb3fff700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 0x7f4c9406a7a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400be70 tx=0x7f4ca400b4d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:41.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.599+0000 7f4cb0ff9700 1 -- 172.21.15.7:0/2950917762 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c9c053e30 con 0x7f4cb4072250 2023-03-03T11:50:41.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.835+0000 7f4cba752700 1 -- 172.21.15.7:0/2950917762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cb406cb30 con 0x7f4cb4072250 2023-03-03T11:50:41.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.836+0000 7f4cb0ff9700 1 -- 172.21.15.7:0/2950917762 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4c9c057450 con 0x7f4cb4072250 2023-03-03T11:50:41.838 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:50:41.839 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:50:41.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 -- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 msgr2=0x7f4c9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 0x7f4c9406a7a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400be70 tx=0x7f4ca400b4d0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 -- 172.21.15.7:0/2950917762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb4072250 msgr2=0x7f4cb40aa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb4072250 0x7f4cb40aa710 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009fd0 tx=0x7f4c9c004750 comp rx=0 tx=0).stop 2023-03-03T11:50:41.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 -- 172.21.15.7:0/2950917762 shutdown_connections 2023-03-03T11:50:41.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cb40c1ce0 0x7f4cb414ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cb4071e80 0x7f4cb40aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c940682f0 0x7f4c9406a7a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 --2- 172.21.15.7:0/2950917762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb4072250 0x7f4cb40aa710 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 -- 172.21.15.7:0/2950917762 >> 172.21.15.7:0/2950917762 conn(0x7f4cb40a94d0 msgr2=0x7f4cb4151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:41.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 -- 172.21.15.7:0/2950917762 shutdown_connections 2023-03-03T11:50:41.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.838+0000 7f4cba752700 1 -- 172.21.15.7:0/2950917762 wait complete. 2023-03-03T11:50:41.842 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:50:41.852 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:50:41.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.974+0000 7f7f1dfee700 1 Processor -- start 2023-03-03T11:50:41.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.974+0000 7f7f1dfee700 1 -- start start 2023-03-03T11:50:41.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.974+0000 7f7f1dfee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f180c5190 0x7f7f1806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.974+0000 7f7f1dfee700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1814ebe0 con 0x7f7f180c6010 2023-03-03T11:50:41.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.974+0000 7f7f1dfee700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f180c2e70 con 0x7f7f180c17f0 2023-03-03T11:50:41.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.974+0000 7f7f1dfee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18152100 con 0x7f7f180c5190 2023-03-03T11:50:41.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f177fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7f180c6010 0x7f7f180c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60238/0 (socket says 172.21.15.7:60238) 2023-03-03T11:50:41.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f177fe700 1 -- 172.21.15.7:0/2037883719 learned_addr learned my addr 172.21.15.7:0/2037883719 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:50:41.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f17fff700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f180c5190 0x7f7f1806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743604583 0 0) 0x7f7f1814ebe0 con 0x7f7f180c6010 2023-03-03T11:50:41.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f000035b0 con 0x7f7f180c6010 2023-03-03T11:50:41.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2331929797 0 0) 0x7f7f000035b0 con 0x7f7f180c6010 2023-03-03T11:50:41.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f180c5190 msgr2=0x7f7f1806ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:50:41.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f180c5190 0x7f7f1806ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 >> v1:172.21.15.143:6789/0 conn(0x7f7f180c17f0 legacy=0x7f7f1806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f180aa680 con 0x7f7f180c6010 2023-03-03T11:50:41.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7f08001670 con 0x7f7f180c6010 2023-03-03T11:50:41.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 >> v1:172.21.15.7:6789/0 conn(0x7f7f180c6010 legacy=0x7f7f180c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f180c5190 0x7f7f00003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 0x7f7f00004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f00004d90 0x7f7f00005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18152100 con 0x7f7f00003fe0 2023-03-03T11:50:41.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0000bba0 con 0x7f7f00004d90 2023-03-03T11:50:41.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0000bd30 con 0x7f7f180c5190 2023-03-03T11:50:41.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.975+0000 7f7f16ffd700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 0x7f7f00004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f177fe700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f180c5190 0x7f7f00003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f17fff700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f00004d90 0x7f7f00005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f16ffd700 1 -- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f180c5190 msgr2=0x7f7f00003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:50:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f16ffd700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f180c5190 0x7f7f00003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f16ffd700 1 -- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f00004d90 msgr2=0x7f7f00005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:50:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f16ffd700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f00004d90 0x7f7f00005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f16ffd700 1 -- 172.21.15.7:0/2037883719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f000035b0 con 0x7f7f00003fe0 2023-03-03T11:50:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f16ffd700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 0x7f7f00004850 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0018d0 tx=0x7f7f00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed5a6bcbe17581a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:41.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c00c650 con 0x7f7f00003fe0 2023-03-03T11:50:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0c00c7b0 con 0x7f7f00003fe0 2023-03-03T11:50:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f167fc700 1 -- 172.21.15.7:0/2037883719 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c00cb60 con 0x7f7f00003fe0 2023-03-03T11:50:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f1dfee700 1 -- 172.21.15.7:0/2037883719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 msgr2=0x7f7f00004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f1dfee700 1 --2- 172.21.15.7:0/2037883719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 0x7f7f00004850 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0018d0 tx=0x7f7f00005880 comp rx=0 tx=0).stop 2023-03-03T11:50:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.976+0000 7f7f17fff700 1 -- 172.21.15.7:0/2037883719 reap_dead start 2023-03-03T11:50:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.977+0000 7f7f1dfee700 1 -- 172.21.15.7:0/2037883719 shutdown_connections 2023-03-03T11:50:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.977+0000 7f7f1dfee700 1 -- 172.21.15.7:0/2037883719 >> 172.21.15.7:0/2037883719 conn(0x7f7f180a9430 msgr2=0x7f7f180c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:41.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.977+0000 7f7f1dfee700 1 -- 172.21.15.7:0/2037883719 shutdown_connections 2023-03-03T11:50:41.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.977+0000 7f7f1dfee700 1 -- 172.21.15.7:0/2037883719 wait complete. 2023-03-03T11:50:41.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.978+0000 7f7f1dfee700 1 Processor -- start 2023-03-03T11:50:41.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.978+0000 7f7f1dfee700 1 -- start start 2023-03-03T11:50:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f1dfee700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 0x7f7f180bbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f1dfee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f180c0530 0x7f7f181537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f1dfee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f1814dfc0 0x7f7f1814e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f1dfee700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f180be3d0 con 0x7f7f00003fe0 2023-03-03T11:50:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f1dfee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f18153350 con 0x7f7f180c0530 2023-03-03T11:50:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f1dfee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f181551e0 con 0x7f7f1814dfc0 2023-03-03T11:50:41.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f177fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 0x7f7f180bbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f177fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 0x7f7f180bbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38038/0 (socket says 172.21.15.7:38038) 2023-03-03T11:50:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f177fe700 1 -- 172.21.15.7:0/316450589 learned_addr learned my addr 172.21.15.7:0/316450589 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:50:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f177fe700 1 -- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f1814dfc0 msgr2=0x7f7f1814e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f177fe700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f1814dfc0 0x7f7f1814e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f177fe700 1 -- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f180c0530 msgr2=0x7f7f181537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f177fe700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f180c0530 0x7f7f181537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:41.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f177fe700 1 -- 172.21.15.7:0/316450589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f180c2b00 con 0x7f7f00003fe0 2023-03-03T11:50:41.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f177fe700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 0x7f7f180bbfb0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f7f080072c0 tx=0x7f7f08007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:41.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7efffff700 1 -- 172.21.15.7:0/316450589 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08001090 con 0x7f7f00003fe0 2023-03-03T11:50:41.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7efffff700 1 -- 172.21.15.7:0/316450589 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f080015e0 con 0x7f7f00003fe0 2023-03-03T11:50:41.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f1dfee700 1 -- 172.21.15.7:0/316450589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f180aa960 con 0x7f7f00003fe0 2023-03-03T11:50:41.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7efffff700 1 -- 172.21.15.7:0/316450589 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800fa90 con 0x7f7f00003fe0 2023-03-03T11:50:41.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.979+0000 7f7f1dfee700 1 -- 172.21.15.7:0/316450589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f180c3300 con 0x7f7f00003fe0 2023-03-03T11:50:41.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.980+0000 7f7f1dfee700 1 -- 172.21.15.7:0/316450589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f18153630 con 0x7f7f00003fe0 2023-03-03T11:50:41.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.981+0000 7f7efffff700 1 -- 172.21.15.7:0/316450589 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7f0800fbf0 con 0x7f7f00003fe0 2023-03-03T11:50:41.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.981+0000 7f7efffff700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f040682f0 0x7f7f0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:50:41.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.981+0000 7f7efffff700 1 -- 172.21.15.7:0/316450589 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7f08086140 con 0x7f7f00003fe0 2023-03-03T11:50:41.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.981+0000 7f7f16ffd700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f040682f0 0x7f7f0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:50:41.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.982+0000 7f7f16ffd700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f040682f0 0x7f7f0406a7a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7f181557c0 tx=0x7f7f0c009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:50:41.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:41.985+0000 7f7efffff700 1 -- 172.21.15.7:0/316450589 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7f080524e0 con 0x7f7f00003fe0 2023-03-03T11:50:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.220+0000 7f7f1dfee700 1 -- 172.21.15.7:0/316450589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f18151e80 con 0x7f7f00003fe0 2023-03-03T11:50:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.221+0000 7f7efffff700 1 -- 172.21.15.7:0/316450589 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7f08055b00 con 0x7f7f00003fe0 2023-03-03T11:50:42.223 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:50:42.223 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 -- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f040682f0 msgr2=0x7f7f0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f040682f0 0x7f7f0406a7a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7f181557c0 tx=0x7f7f0c009030 comp rx=0 tx=0).stop 2023-03-03T11:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 -- 172.21.15.7:0/316450589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 msgr2=0x7f7f180bbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 0x7f7f180bbfb0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f7f080072c0 tx=0x7f7f08007680 comp rx=0 tx=0).stop 2023-03-03T11:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 -- 172.21.15.7:0/316450589 shutdown_connections 2023-03-03T11:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f1814dfc0 0x7f7f1814e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f040682f0 0x7f7f0406a7a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f00003fe0 0x7f7f180bbfb0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 --2- 172.21.15.7:0/316450589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f180c0530 0x7f7f181537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:50:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 -- 172.21.15.7:0/316450589 >> 172.21.15.7:0/316450589 conn(0x7f7f180a9430 msgr2=0x7f7f18150f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:50:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 -- 172.21.15.7:0/316450589 shutdown_connections 2023-03-03T11:50:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:50:42.223+0000 7f7f1dfee700 1 -- 172.21.15.7:0/316450589 wait complete. 2023-03-03T11:50:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:50:42.236 INFO:tasks.mds_thrash.fs.[a]:waiting for 88 secs before thrashing 2023-03-03T11:50:49.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:50:49.564 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:19.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:19.718 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:35.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:51:35.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.532+0000 7fb44ce88700 1 Processor -- start 2023-03-03T11:51:35.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.532+0000 7fb44ce88700 1 -- start start 2023-03-03T11:51:35.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb44ce88700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480aa030 0x7fb448158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:35.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb44ce88700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4480a9c20 con 0x7fb4480aad20 2023-03-03T11:51:35.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb44ce88700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb448154f10 con 0x7fb44814b0d0 2023-03-03T11:51:35.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb44ce88700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480ac3c0 con 0x7fb4480aa030 2023-03-03T11:51:35.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb445d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb4480aad20 0x7fb448157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60308/0 (socket says 172.21.15.7:60308) 2023-03-03T11:51:35.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb445d9b700 1 -- 172.21.15.7:0/172940735 learned_addr learned my addr 172.21.15.7:0/172940735 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:51:35.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb446d9d700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480aa030 0x7fb448158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:35.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481267282 0 0) 0x7fb4480a9c20 con 0x7fb4480aad20 2023-03-03T11:51:35.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4300035b0 con 0x7fb4480aad20 2023-03-03T11:51:35.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1054187311 0 0) 0x7fb4300035b0 con 0x7fb4480aad20 2023-03-03T11:51:35.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480aa030 msgr2=0x7fb448158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:51:35.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb44559a700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480aa030 0x7fb448158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:35.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.533+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 >> v1:172.21.15.143:6789/0 conn(0x7fb44814b0d0 legacy=0x7fb448071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44806fc70 con 0x7fb4480aad20 2023-03-03T11:51:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb43c001670 con 0x7fb4480aad20 2023-03-03T11:51:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 >> v1:172.21.15.7:6789/0 conn(0x7fb4480aad20 legacy=0x7fb448157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480aa030 0x7fb430003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb430003fe0 0x7fb430004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430004d90 0x7fb430005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480ac3c0 con 0x7fb430003fe0 2023-03-03T11:51:35.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43000bba0 con 0x7fb430004d90 2023-03-03T11:51:35.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43000bd30 con 0x7fb4480aa030 2023-03-03T11:51:35.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb445d9b700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb430003fe0 0x7fb430004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:35.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44659c700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480aa030 0x7fb430003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:35.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb446d9d700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430004d90 0x7fb430005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:35.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb445d9b700 1 -- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480aa030 msgr2=0x7fb430003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:51:35.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb445d9b700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4480aa030 0x7fb430003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:35.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb445d9b700 1 -- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430004d90 msgr2=0x7fb430005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:51:35.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb445d9b700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430004d90 0x7fb430005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:35.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb445d9b700 1 -- 172.21.15.7:0/172940735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4300035b0 con 0x7fb430003fe0 2023-03-03T11:51:35.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb446d9d700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430004d90 0x7fb430005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:51:35.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb445d9b700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb430003fe0 0x7fb430004850 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00c210 tx=0x7fb430005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540e7bbbb582552a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:51:35.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c00cd70 con 0x7fb430003fe0 2023-03-03T11:51:35.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43c007ba0 con 0x7fb430003fe0 2023-03-03T11:51:35.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.534+0000 7fb44559a700 1 -- 172.21.15.7:0/172940735 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c005ba0 con 0x7fb430003fe0 2023-03-03T11:51:35.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.535+0000 7fb44ce88700 1 -- 172.21.15.7:0/172940735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb430003fe0 msgr2=0x7fb430004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.535+0000 7fb44ce88700 1 --2- 172.21.15.7:0/172940735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb430003fe0 0x7fb430004850 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00c210 tx=0x7fb430005880 comp rx=0 tx=0).stop 2023-03-03T11:51:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.535+0000 7fb446d9d700 1 -- 172.21.15.7:0/172940735 reap_dead start 2023-03-03T11:51:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.535+0000 7fb44ce88700 1 -- 172.21.15.7:0/172940735 shutdown_connections 2023-03-03T11:51:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.535+0000 7fb44ce88700 1 -- 172.21.15.7:0/172940735 >> 172.21.15.7:0/172940735 conn(0x7fb4480c8900 msgr2=0x7fb4480c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:51:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.535+0000 7fb44ce88700 1 -- 172.21.15.7:0/172940735 shutdown_connections 2023-03-03T11:51:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.535+0000 7fb44ce88700 1 -- 172.21.15.7:0/172940735 wait complete. 2023-03-03T11:51:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.536+0000 7fb44ce88700 1 Processor -- start 2023-03-03T11:51:35.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.537+0000 7fb44ce88700 1 -- start start 2023-03-03T11:51:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.537+0000 7fb44ce88700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb430003fe0 0x7fb448151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:35.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.537+0000 7fb44ce88700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb448153400 0x7fb44814fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.537+0000 7fb44ce88700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4481539c0 0x7fb448152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.537+0000 7fb44ce88700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb448154b30 con 0x7fb448153400 2023-03-03T11:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.537+0000 7fb44ce88700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44814f2f0 con 0x7fb4481539c0 2023-03-03T11:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.537+0000 7fb44ce88700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb448154570 con 0x7fb430003fe0 2023-03-03T11:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.537+0000 7fb445d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb448153400 0x7fb44814fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.537+0000 7fb445d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb448153400 0x7fb44814fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38110/0 (socket says 172.21.15.7:38110) 2023-03-03T11:51:35.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.537+0000 7fb445d9b700 1 -- 172.21.15.7:0/503085928 learned_addr learned my addr 172.21.15.7:0/503085928 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb446d9d700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4481539c0 0x7fb448152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb44659c700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb430003fe0 0x7fb448151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb445d9b700 1 -- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb430003fe0 msgr2=0x7fb448151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb445d9b700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb430003fe0 0x7fb448151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb445d9b700 1 -- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4481539c0 msgr2=0x7fb448152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb445d9b700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4481539c0 0x7fb448152020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb445d9b700 1 -- 172.21.15.7:0/503085928 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44806f6b0 con 0x7fb448153400 2023-03-03T11:51:35.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb445d9b700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb448153400 0x7fb44814fbf0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fb430004790 tx=0x7fb44814ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb436ffd700 1 -- 172.21.15.7:0/503085928 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c004c80 con 0x7fb448153400 2023-03-03T11:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb436ffd700 1 -- 172.21.15.7:0/503085928 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43c004de0 con 0x7fb448153400 2023-03-03T11:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb44ce88700 1 -- 172.21.15.7:0/503085928 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb448155160 con 0x7fb448153400 2023-03-03T11:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb436ffd700 1 -- 172.21.15.7:0/503085928 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c0072f0 con 0x7fb448153400 2023-03-03T11:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.538+0000 7fb44ce88700 1 -- 172.21.15.7:0/503085928 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44806f990 con 0x7fb448153400 2023-03-03T11:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.539+0000 7fb44ce88700 1 -- 172.21.15.7:0/503085928 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb448074bb0 con 0x7fb448153400 2023-03-03T11:51:35.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.540+0000 7fb436ffd700 1 -- 172.21.15.7:0/503085928 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb43c010070 con 0x7fb448153400 2023-03-03T11:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.540+0000 7fb436ffd700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb428068320 0x7fb42806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.540+0000 7fb436ffd700 1 -- 172.21.15.7:0/503085928 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb43c0871f0 con 0x7fb448153400 2023-03-03T11:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.540+0000 7fb44659c700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb428068320 0x7fb42806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.541+0000 7fb44659c700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb428068320 0x7fb42806a7d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb438000c00 tx=0x7fb438009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:51:35.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.545+0000 7fb436ffd700 1 -- 172.21.15.7:0/503085928 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb43c053590 con 0x7fb448153400 2023-03-03T11:51:35.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.786+0000 7fb44ce88700 1 -- 172.21.15.7:0/503085928 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4480aa540 con 0x7fb448153400 2023-03-03T11:51:35.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.787+0000 7fb436ffd700 1 -- 172.21.15.7:0/503085928 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb43c01aea0 con 0x7fb448153400 2023-03-03T11:51:35.789 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:51:35.789 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:51:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 -- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb428068320 msgr2=0x7fb42806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb428068320 0x7fb42806a7d0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb438000c00 tx=0x7fb438009040 comp rx=0 tx=0).stop 2023-03-03T11:51:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 -- 172.21.15.7:0/503085928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb448153400 msgr2=0x7fb44814fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:51:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb448153400 0x7fb44814fbf0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fb430004790 tx=0x7fb44814ea50 comp rx=0 tx=0).stop 2023-03-03T11:51:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 -- 172.21.15.7:0/503085928 shutdown_connections 2023-03-03T11:51:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4481539c0 0x7fb448152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:35.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb430003fe0 0x7fb448151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb428068320 0x7fb42806a7d0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 --2- 172.21.15.7:0/503085928 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb448153400 0x7fb44814fbf0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:51:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 -- 172.21.15.7:0/503085928 >> 172.21.15.7:0/503085928 conn(0x7fb4480c8900 msgr2=0x7fb4480ac590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:51:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 -- 172.21.15.7:0/503085928 shutdown_connections 2023-03-03T11:51:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:51:35.789+0000 7fb44ce88700 1 -- 172.21.15.7:0/503085928 wait complete. 2023-03-03T11:51:35.792 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:51:35.802 INFO:tasks.mds_thrash.fs.[b]:waiting for 72 secs before thrashing 2023-03-03T11:51:49.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:51:49.838 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:10.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:52:10.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.375+0000 7facec2e6700 1 Processor -- start 2023-03-03T11:52:10.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.375+0000 7facec2e6700 1 -- start start 2023-03-03T11:52:10.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facec2e6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7face40c00e0 0x7face40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:10.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facec2e6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7face4154ff0 0x7face40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:10.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facec2e6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face40c7310 con 0x7face40caa30 2023-03-03T11:52:10.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facec2e6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face4155430 con 0x7face4154ff0 2023-03-03T11:52:10.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facec2e6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face40bfd10 con 0x7face40c00e0 2023-03-03T11:52:10.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facea883700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7face4154ff0 0x7face40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:10.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facea082700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7face40caa30 0x7face40c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60360/0 (socket says 172.21.15.7:60360) 2023-03-03T11:52:10.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facea082700 1 -- 172.21.15.7:0/2601725828 learned_addr learned my addr 172.21.15.7:0/2601725828 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:52:10.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7face9881700 1 --2- 172.21.15.7:0/2601725828 >> v2:172.21.15.143:3300/0 conn(0x7face40c00e0 0x7face40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:10.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facea883700 1 -- 172.21.15.7:0/2601725828 >> v2:172.21.15.143:3300/0 conn(0x7face40c00e0 msgr2=0x7face40c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:52:10.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facea883700 1 --2- 172.21.15.7:0/2601725828 >> v2:172.21.15.143:3300/0 conn(0x7face40c00e0 0x7face40c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facea883700 1 -- 172.21.15.7:0/2601725828 >> v1:172.21.15.7:6789/0 conn(0x7face40caa30 legacy=0x7face40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facea883700 1 -- 172.21.15.7:0/2601725828 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face40bfa30 con 0x7face4154ff0 2023-03-03T11:52:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7face9080700 1 -- 172.21.15.7:0/2601725828 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298129633 0 0) 0x7face40c7310 con 0x7face40caa30 2023-03-03T11:52:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.376+0000 7facea883700 1 --2- 172.21.15.7:0/2601725828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 0x7face40cae00 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7face40ca910 tx=0x7facdc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f198da910e217a5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:52:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7face9080700 1 -- 172.21.15.7:0/2601725828 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc00e070 con 0x7face4154ff0 2023-03-03T11:52:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7face9080700 1 -- 172.21.15.7:0/2601725828 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc00bbd0 con 0x7face4154ff0 2023-03-03T11:52:10.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7face9080700 1 -- 172.21.15.7:0/2601725828 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc012430 con 0x7face4154ff0 2023-03-03T11:52:10.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7facec2e6700 1 -- 172.21.15.7:0/2601725828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 msgr2=0x7face40cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7facec2e6700 1 --2- 172.21.15.7:0/2601725828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 0x7face40cae00 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7face40ca910 tx=0x7facdc00b330 comp rx=0 tx=0).stop 2023-03-03T11:52:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7facec2e6700 1 -- 172.21.15.7:0/2601725828 shutdown_connections 2023-03-03T11:52:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7facec2e6700 1 --2- 172.21.15.7:0/2601725828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 0x7face40cae00 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7facec2e6700 1 --2- 172.21.15.7:0/2601725828 >> v2:172.21.15.143:3300/0 conn(0x7face40c00e0 0x7face40c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7facec2e6700 1 -- 172.21.15.7:0/2601725828 >> 172.21.15.7:0/2601725828 conn(0x7face4066e40 msgr2=0x7face4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:52:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7facec2e6700 1 -- 172.21.15.7:0/2601725828 shutdown_connections 2023-03-03T11:52:10.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.377+0000 7facec2e6700 1 -- 172.21.15.7:0/2601725828 wait complete. 2023-03-03T11:52:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.378+0000 7facec2e6700 1 Processor -- start 2023-03-03T11:52:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.378+0000 7facec2e6700 1 -- start start 2023-03-03T11:52:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7facec2e6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face40c00e0 0x7face4142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7facec2e6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40caa30 0x7face4157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7facec2e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 0x7face4156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7facec2e6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face40aa940 con 0x7face4154ff0 2023-03-03T11:52:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7facec2e6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face4158420 con 0x7face40c00e0 2023-03-03T11:52:10.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7facec2e6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face41eda10 con 0x7face40caa30 2023-03-03T11:52:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7facea883700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 0x7face4156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7facea883700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 0x7face4156310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38158/0 (socket says 172.21.15.7:38158) 2023-03-03T11:52:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7facea883700 1 -- 172.21.15.7:0/4071500122 learned_addr learned my addr 172.21.15.7:0/4071500122 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:52:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7face9881700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40caa30 0x7face4157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.379+0000 7facea082700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face40c00e0 0x7face4142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facea883700 1 -- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40caa30 msgr2=0x7face4157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:10.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facea883700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40caa30 0x7face4157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facea883700 1 -- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face40c00e0 msgr2=0x7face4142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facea883700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face40c00e0 0x7face4142e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facea883700 1 -- 172.21.15.7:0/4071500122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc009a70 con 0x7face4154ff0 2023-03-03T11:52:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facea883700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 0x7face4156310 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7facdc00b730 tx=0x7facdc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:52:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facdaffd700 1 -- 172.21.15.7:0/4071500122 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc012430 con 0x7face4154ff0 2023-03-03T11:52:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facdaffd700 1 -- 172.21.15.7:0/4071500122 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc020070 con 0x7face4154ff0 2023-03-03T11:52:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facec2e6700 1 -- 172.21.15.7:0/4071500122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face41547f0 con 0x7face4154ff0 2023-03-03T11:52:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facdaffd700 1 -- 172.21.15.7:0/4071500122 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc004280 con 0x7face4154ff0 2023-03-03T11:52:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.380+0000 7facec2e6700 1 -- 172.21.15.7:0/4071500122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face4154510 con 0x7face4154ff0 2023-03-03T11:52:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.381+0000 7facec2e6700 1 -- 172.21.15.7:0/4071500122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face4154ad0 con 0x7face4154ff0 2023-03-03T11:52:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.382+0000 7facdaffd700 1 -- 172.21.15.7:0/4071500122 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7facdc0043e0 con 0x7face4154ff0 2023-03-03T11:52:10.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.382+0000 7facdaffd700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facc80682f0 0x7facc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:10.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.382+0000 7facdaffd700 1 -- 172.21.15.7:0/4071500122 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7facdc022070 con 0x7face4154ff0 2023-03-03T11:52:10.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.382+0000 7facea082700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facc80682f0 0x7facc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:10.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.383+0000 7facea082700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facc80682f0 0x7facc806a7a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7face0007fd0 tx=0x7face0007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:52:10.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.386+0000 7facdaffd700 1 -- 172.21.15.7:0/4071500122 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7facdc054810 con 0x7face4154ff0 2023-03-03T11:52:10.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.622+0000 7facec2e6700 1 -- 172.21.15.7:0/4071500122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7face4074bb0 con 0x7face4154ff0 2023-03-03T11:52:10.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.622+0000 7facdaffd700 1 -- 172.21.15.7:0/4071500122 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7facdc057e30 con 0x7face4154ff0 2023-03-03T11:52:10.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:52:10.624 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:52:10.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 -- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facc80682f0 msgr2=0x7facc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:10.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facc80682f0 0x7facc806a7a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7face0007fd0 tx=0x7face0007de0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 -- 172.21.15.7:0/4071500122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 msgr2=0x7face4156310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:10.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 0x7face4156310 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7facdc00b730 tx=0x7facdc004ad0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 -- 172.21.15.7:0/4071500122 shutdown_connections 2023-03-03T11:52:10.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facc80682f0 0x7facc806a7a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4154ff0 0x7face4156310 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face40c00e0 0x7face4142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 --2- 172.21.15.7:0/4071500122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40caa30 0x7face4157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:10.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 -- 172.21.15.7:0/4071500122 >> 172.21.15.7:0/4071500122 conn(0x7face4066e40 msgr2=0x7face4153b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:52:10.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 -- 172.21.15.7:0/4071500122 shutdown_connections 2023-03-03T11:52:10.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:10.624+0000 7facec2e6700 1 -- 172.21.15.7:0/4071500122 wait complete. 2023-03-03T11:52:10.628 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:52:10.638 INFO:tasks.mds_thrash.fs.[a]:waiting for 35 secs before thrashing 2023-03-03T11:52:19.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:19.969 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:45.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:52:45.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.777+0000 7f06e26c6700 1 Processor -- start 2023-03-03T11:52:45.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.777+0000 7f06e26c6700 1 -- start start 2023-03-03T11:52:45.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.778+0000 7f06e26c6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06dc1586c0 0x7f06dc06d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:45.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.778+0000 7f06e26c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 0x7f06dc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:45.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.778+0000 7f06e26c6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06dc154560 con 0x7f06dc13a0f0 2023-03-03T11:52:45.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.778+0000 7f06e26c6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06dc154840 con 0x7f06dc1586c0 2023-03-03T11:52:45.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.778+0000 7f06e26c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc157be0 con 0x7f06dc0c1da0 2023-03-03T11:52:45.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.778+0000 7f06db7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06dc1586c0 0x7f06dc06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:45.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.778+0000 7f06db7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06dc1586c0 0x7f06dc06d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38188/0 (socket says 172.21.15.7:38188) 2023-03-03T11:52:45.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.778+0000 7f06db7fe700 1 -- 172.21.15.7:0/2517951004 learned_addr learned my addr 172.21.15.7:0/2517951004 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:52:45.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.779+0000 7f06e0c63700 1 --2- 172.21.15.7:0/2517951004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 0x7f06dc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:45.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.779+0000 7f06db7fe700 1 -- 172.21.15.7:0/2517951004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 msgr2=0x7f06dc06d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:52:45.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.779+0000 7f06db7fe700 1 --2- 172.21.15.7:0/2517951004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 0x7f06dc06d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:45.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.779+0000 7f06db7fe700 1 -- 172.21.15.7:0/2517951004 >> v1:172.21.15.143:6789/0 conn(0x7f06dc13a0f0 legacy=0x7f06dc0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:45.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.779+0000 7f06db7fe700 1 -- 172.21.15.7:0/2517951004 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc157ec0 con 0x7f06dc1586c0 2023-03-03T11:52:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.779+0000 7f06e0c63700 1 --2- 172.21.15.7:0/2517951004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 0x7f06dc06d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:52:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.779+0000 7f06db7fe700 1 --2- 172.21.15.7:0/2517951004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 0x7f06dc06d240 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f06c40098f0 tx=0x7f06c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84990a10a498f79 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:52:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.779+0000 7f06daffd700 1 -- 172.21.15.7:0/2517951004 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c400b8a0 con 0x7f06dc1586c0 2023-03-03T11:52:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.779+0000 7f06daffd700 1 -- 172.21.15.7:0/2517951004 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c400ba00 con 0x7f06dc1586c0 2023-03-03T11:52:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.779+0000 7f06daffd700 1 -- 172.21.15.7:0/2517951004 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c400bd90 con 0x7f06dc1586c0 2023-03-03T11:52:45.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.780+0000 7f06e26c6700 1 -- 172.21.15.7:0/2517951004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 msgr2=0x7f06dc06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:45.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.780+0000 7f06e26c6700 1 --2- 172.21.15.7:0/2517951004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 0x7f06dc06d240 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f06c40098f0 tx=0x7f06c4009c00 comp rx=0 tx=0).stop 2023-03-03T11:52:45.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.780+0000 7f06e26c6700 1 -- 172.21.15.7:0/2517951004 shutdown_connections 2023-03-03T11:52:45.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.780+0000 7f06e26c6700 1 --2- 172.21.15.7:0/2517951004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 0x7f06dc06d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:45.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.780+0000 7f06e26c6700 1 --2- 172.21.15.7:0/2517951004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 0x7f06dc06d240 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:45.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.780+0000 7f06e26c6700 1 -- 172.21.15.7:0/2517951004 >> 172.21.15.7:0/2517951004 conn(0x7f06dc06c110 msgr2=0x7f06dc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:52:45.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.780+0000 7f06e26c6700 1 -- 172.21.15.7:0/2517951004 shutdown_connections 2023-03-03T11:52:45.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.780+0000 7f06e26c6700 1 -- 172.21.15.7:0/2517951004 wait complete. 2023-03-03T11:52:45.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.781+0000 7f06e26c6700 1 Processor -- start 2023-03-03T11:52:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.781+0000 7f06e26c6700 1 -- start start 2023-03-03T11:52:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e26c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 0x7f06dc1ed360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e26c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06dc13a0f0 0x7f06dc0ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e26c6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 0x7f06dc1ea590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e26c6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc1e9df0 con 0x7f06dc1586c0 2023-03-03T11:52:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e26c6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0aa450 con 0x7f06dc13a0f0 2023-03-03T11:52:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e26c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0aa730 con 0x7f06dc0c1da0 2023-03-03T11:52:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e0c63700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 0x7f06dc1ea590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e0c63700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 0x7f06dc1ea590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38196/0 (socket says 172.21.15.7:38196) 2023-03-03T11:52:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e0c63700 1 -- 172.21.15.7:0/2871333502 learned_addr learned my addr 172.21.15.7:0/2871333502 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:52:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06dbfff700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 0x7f06dc1ed360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06db7fe700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06dc13a0f0 0x7f06dc0ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e0c63700 1 -- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 msgr2=0x7f06dc1ed360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e0c63700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 0x7f06dc1ed360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e0c63700 1 -- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06dc13a0f0 msgr2=0x7f06dc0ab5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e0c63700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06dc13a0f0 0x7f06dc0ab5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e0c63700 1 -- 172.21.15.7:0/2871333502 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c4009550 con 0x7f06dc1586c0 2023-03-03T11:52:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06db7fe700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06dc13a0f0 0x7f06dc0ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:52:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.782+0000 7f06e0c63700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 0x7f06dc1ea590 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f06d4009fa0 tx=0x7f06d400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:52:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.783+0000 7f06d8ff9700 1 -- 172.21.15.7:0/2871333502 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d400de60 con 0x7f06dc1586c0 2023-03-03T11:52:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.783+0000 7f06d8ff9700 1 -- 172.21.15.7:0/2871333502 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d400f040 con 0x7f06dc1586c0 2023-03-03T11:52:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.783+0000 7f06e26c6700 1 -- 172.21.15.7:0/2871333502 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06dc06e1a0 con 0x7f06dc1586c0 2023-03-03T11:52:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.783+0000 7f06d8ff9700 1 -- 172.21.15.7:0/2871333502 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d4003c80 con 0x7f06dc1586c0 2023-03-03T11:52:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.783+0000 7f06e26c6700 1 -- 172.21.15.7:0/2871333502 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06dc1581a0 con 0x7f06dc1586c0 2023-03-03T11:52:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.784+0000 7f06e26c6700 1 -- 172.21.15.7:0/2871333502 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06dc06e330 con 0x7f06dc1586c0 2023-03-03T11:52:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.784+0000 7f06d8ff9700 1 -- 172.21.15.7:0/2871333502 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f06d401c030 con 0x7f06dc1586c0 2023-03-03T11:52:45.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.784+0000 7f06d8ff9700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06bc0682f0 0x7f06bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:45.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.785+0000 7f06d8ff9700 1 -- 172.21.15.7:0/2871333502 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f06d4086120 con 0x7f06dc1586c0 2023-03-03T11:52:45.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.785+0000 7f06dbfff700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06bc0682f0 0x7f06bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:45.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.785+0000 7f06dbfff700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06bc0682f0 0x7f06bc06a7a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f06cc007fd0 tx=0x7f06cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:52:45.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:45.789+0000 7f06d8ff9700 1 -- 172.21.15.7:0/2871333502 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f06d4052460 con 0x7f06dc1586c0 2023-03-03T11:52:46.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.032+0000 7f06e26c6700 1 -- 172.21.15.7:0/2871333502 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06dc074bb0 con 0x7f06dc1586c0 2023-03-03T11:52:46.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.033+0000 7f06d8ff9700 1 -- 172.21.15.7:0/2871333502 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f06d4055a80 con 0x7f06dc1586c0 2023-03-03T11:52:46.034 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:52:46.035 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:52:46.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.034+0000 7f06e26c6700 1 -- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06bc0682f0 msgr2=0x7f06bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:46.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.034+0000 7f06e26c6700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06bc0682f0 0x7f06bc06a7a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f06cc007fd0 tx=0x7f06cc009040 comp rx=0 tx=0).stop 2023-03-03T11:52:46.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.034+0000 7f06e26c6700 1 -- 172.21.15.7:0/2871333502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 msgr2=0x7f06dc1ea590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:46.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.034+0000 7f06e26c6700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 0x7f06dc1ea590 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f06d4009fa0 tx=0x7f06d400d3b0 comp rx=0 tx=0).stop 2023-03-03T11:52:46.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.034+0000 7f06e26c6700 1 -- 172.21.15.7:0/2871333502 shutdown_connections 2023-03-03T11:52:46.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.034+0000 7f06e26c6700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06bc0682f0 0x7f06bc06a7a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:46.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.034+0000 7f06e26c6700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06dc1586c0 0x7f06dc1ea590 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:46.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.034+0000 7f06e26c6700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06dc0c1da0 0x7f06dc1ed360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:46.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.034+0000 7f06e26c6700 1 --2- 172.21.15.7:0/2871333502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06dc13a0f0 0x7f06dc0ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:46.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.035+0000 7f06e26c6700 1 -- 172.21.15.7:0/2871333502 >> 172.21.15.7:0/2871333502 conn(0x7f06dc06c110 msgr2=0x7f06dc138ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:52:46.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.035+0000 7f06e26c6700 1 -- 172.21.15.7:0/2871333502 shutdown_connections 2023-03-03T11:52:46.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:46.035+0000 7f06e26c6700 1 -- 172.21.15.7:0/2871333502 wait complete. 2023-03-03T11:52:46.038 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:52:46.048 INFO:tasks.mds_thrash.fs.[a]:waiting for 112 secs before thrashing 2023-03-03T11:52:47.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:52:47.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.939+0000 7fe199b5f700 1 Processor -- start 2023-03-03T11:52:47.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.939+0000 7fe199b5f700 1 -- start start 2023-03-03T11:52:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe199b5f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe19414f860 0x7fe1941562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe199b5f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe1940c20d0 0x7fe194152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe199b5f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1940c27c0 0x7fe1940736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe199b5f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19406d350 con 0x7fe19414f860 2023-03-03T11:52:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe199b5f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1941558e0 con 0x7fe1940c20d0 2023-03-03T11:52:47.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe199b5f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe194156830 con 0x7fe1940c27c0 2023-03-03T11:52:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe1937fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe19414f860 0x7fe1941562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe1937fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe19414f860 0x7fe1941562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38200/0 (socket says 172.21.15.7:38200) 2023-03-03T11:52:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe1937fe700 1 -- 172.21.15.7:0/2090478178 learned_addr learned my addr 172.21.15.7:0/2090478178 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:52:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe192ffd700 1 --2- 172.21.15.7:0/2090478178 >> v2:172.21.15.143:3300/0 conn(0x7fe1940c20d0 0x7fe194152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe193fff700 1 --2- 172.21.15.7:0/2090478178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1940c27c0 0x7fe1940736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe1937fe700 1 -- 172.21.15.7:0/2090478178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1940c27c0 msgr2=0x7fe1940736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:52:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe1937fe700 1 --2- 172.21.15.7:0/2090478178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1940c27c0 0x7fe1940736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:47.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe1937fe700 1 -- 172.21.15.7:0/2090478178 >> v2:172.21.15.143:3300/0 conn(0x7fe1940c20d0 msgr2=0x7fe194152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:52:47.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe1937fe700 1 --2- 172.21.15.7:0/2090478178 >> v2:172.21.15.143:3300/0 conn(0x7fe1940c20d0 0x7fe194152d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:47.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe1937fe700 1 -- 172.21.15.7:0/2090478178 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe194155e60 con 0x7fe19414f860 2023-03-03T11:52:47.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.940+0000 7fe1937fe700 1 --2- 172.21.15.7:0/2090478178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 0x7fe1941562f0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fe194072520 tx=0x7fe184009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430ee47f813c96bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:52:47.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe1927fc700 1 -- 172.21.15.7:0/2090478178 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18400b860 con 0x7fe19414f860 2023-03-03T11:52:47.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe1927fc700 1 -- 172.21.15.7:0/2090478178 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18400b9c0 con 0x7fe19414f860 2023-03-03T11:52:47.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe1927fc700 1 -- 172.21.15.7:0/2090478178 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18400bd50 con 0x7fe19414f860 2023-03-03T11:52:47.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe199b5f700 1 -- 172.21.15.7:0/2090478178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 msgr2=0x7fe1941562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe199b5f700 1 --2- 172.21.15.7:0/2090478178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 0x7fe1941562f0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fe194072520 tx=0x7fe184009c00 comp rx=0 tx=0).stop 2023-03-03T11:52:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe199b5f700 1 -- 172.21.15.7:0/2090478178 shutdown_connections 2023-03-03T11:52:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe199b5f700 1 --2- 172.21.15.7:0/2090478178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1940c27c0 0x7fe1940736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe199b5f700 1 --2- 172.21.15.7:0/2090478178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 0x7fe1941562f0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe199b5f700 1 --2- 172.21.15.7:0/2090478178 >> v2:172.21.15.143:3300/0 conn(0x7fe1940c20d0 0x7fe194152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe199b5f700 1 -- 172.21.15.7:0/2090478178 >> 172.21.15.7:0/2090478178 conn(0x7fe1940a94d0 msgr2=0x7fe1940a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:52:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe199b5f700 1 -- 172.21.15.7:0/2090478178 shutdown_connections 2023-03-03T11:52:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.941+0000 7fe199b5f700 1 -- 172.21.15.7:0/2090478178 wait complete. 2023-03-03T11:52:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.942+0000 7fe199b5f700 1 Processor -- start 2023-03-03T11:52:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe199b5f700 1 -- start start 2023-03-03T11:52:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe199b5f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1940c20d0 0x7fe1940aa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe199b5f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1940c27c0 0x7fe1940ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe199b5f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 0x7fe1940ab730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe199b5f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe194071b60 con 0x7fe19414f860 2023-03-03T11:52:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe199b5f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19414ced0 con 0x7fe1940c20d0 2023-03-03T11:52:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe199b5f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1940712c0 con 0x7fe1940c27c0 2023-03-03T11:52:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe193fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 0x7fe1940ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe193fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 0x7fe1940ab730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38210/0 (socket says 172.21.15.7:38210) 2023-03-03T11:52:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe193fff700 1 -- 172.21.15.7:0/4118266938 learned_addr learned my addr 172.21.15.7:0/4118266938 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:52:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe1937fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1940c20d0 0x7fe1940aa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.943+0000 7fe192ffd700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1940c27c0 0x7fe1940ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe193fff700 1 -- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1940c27c0 msgr2=0x7fe1940ab1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe193fff700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1940c27c0 0x7fe1940ab1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe193fff700 1 -- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1940c20d0 msgr2=0x7fe1940aa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe193fff700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1940c20d0 0x7fe1940aa930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe193fff700 1 -- 172.21.15.7:0/4118266938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe184009550 con 0x7fe19414f860 2023-03-03T11:52:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe193fff700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 0x7fe1940ab730 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00cd70 tx=0x7fe18c00a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:52:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe18bfff700 1 -- 172.21.15.7:0/4118266938 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c003a20 con 0x7fe19414f860 2023-03-03T11:52:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe18bfff700 1 -- 172.21.15.7:0/4118266938 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c003b80 con 0x7fe19414f860 2023-03-03T11:52:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe199b5f700 1 -- 172.21.15.7:0/4118266938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19406cd90 con 0x7fe19414f860 2023-03-03T11:52:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe18bfff700 1 -- 172.21.15.7:0/4118266938 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c004020 con 0x7fe19414f860 2023-03-03T11:52:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.944+0000 7fe199b5f700 1 -- 172.21.15.7:0/4118266938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1941532b0 con 0x7fe19414f860 2023-03-03T11:52:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.945+0000 7fe199b5f700 1 -- 172.21.15.7:0/4118266938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19406d070 con 0x7fe19414f860 2023-03-03T11:52:47.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.946+0000 7fe18bfff700 1 -- 172.21.15.7:0/4118266938 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe18c01c020 con 0x7fe19414f860 2023-03-03T11:52:47.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.946+0000 7fe18bfff700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe174068320 0x7fe17406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:52:47.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.946+0000 7fe18bfff700 1 -- 172.21.15.7:0/4118266938 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe18c012070 con 0x7fe19414f860 2023-03-03T11:52:47.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.946+0000 7fe1937fe700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe174068320 0x7fe17406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:52:47.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.947+0000 7fe1937fe700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe174068320 0x7fe17406a7d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe184000c00 tx=0x7fe184008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:52:47.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:47.950+0000 7fe18bfff700 1 -- 172.21.15.7:0/4118266938 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe18c052b00 con 0x7fe19414f860 2023-03-03T11:52:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.192+0000 7fe199b5f700 1 -- 172.21.15.7:0/4118266938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe194074bb0 con 0x7fe19414f860 2023-03-03T11:52:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.193+0000 7fe18bfff700 1 -- 172.21.15.7:0/4118266938 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe18c056120 con 0x7fe19414f860 2023-03-03T11:52:48.194 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:52:48.195 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:52:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 -- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe174068320 msgr2=0x7fe17406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe174068320 0x7fe17406a7d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe184000c00 tx=0x7fe184008030 comp rx=0 tx=0).stop 2023-03-03T11:52:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 -- 172.21.15.7:0/4118266938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 msgr2=0x7fe1940ab730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:52:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 0x7fe1940ab730 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00cd70 tx=0x7fe18c00a570 comp rx=0 tx=0).stop 2023-03-03T11:52:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 -- 172.21.15.7:0/4118266938 shutdown_connections 2023-03-03T11:52:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe174068320 0x7fe17406a7d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe19414f860 0x7fe1940ab730 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1940c20d0 0x7fe1940aa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 --2- 172.21.15.7:0/4118266938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1940c27c0 0x7fe1940ab1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:52:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 -- 172.21.15.7:0/4118266938 >> 172.21.15.7:0/4118266938 conn(0x7fe1940a94d0 msgr2=0x7fe194154390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:52:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 -- 172.21.15.7:0/4118266938 shutdown_connections 2023-03-03T11:52:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:52:48.195+0000 7fe199b5f700 1 -- 172.21.15.7:0/4118266938 wait complete. 2023-03-03T11:52:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:52:48.208 INFO:tasks.mds_thrash.fs.[b]:waiting for 41 secs before thrashing 2023-03-03T11:52:50.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:52:50.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:20.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:20.204 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:29.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:53:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.345+0000 7f10faee5700 1 Processor -- start 2023-03-03T11:53:29.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.345+0000 7f10faee5700 1 -- start start 2023-03-03T11:53:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.346+0000 7f10faee5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f10f40c65c0 0x7f10f40c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.346+0000 7f10faee5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40a9af0 0x7f10f40c2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.346+0000 7f10faee5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f40a65b0 con 0x7f10f414de50 2023-03-03T11:53:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.346+0000 7f10faee5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10f4154e00 con 0x7f10f40c65c0 2023-03-03T11:53:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.346+0000 7f10faee5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f41541f0 con 0x7f10f40a9af0 2023-03-03T11:53:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.346+0000 7f10f3fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f10f40c65c0 0x7f10f40c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.346+0000 7f10f9482700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40a9af0 0x7f10f40c2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.346+0000 7f10f8c81700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f10f414de50 0x7f10f406a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:35730/0 (socket says 172.21.15.7:35730) 2023-03-03T11:53:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.346+0000 7f10f8c81700 1 -- 172.21.15.7:0/275867339 learned_addr learned my addr 172.21.15.7:0/275867339 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:53:29.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f37fe700 1 -- 172.21.15.7:0/275867339 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155418291 0 0) 0x7f10f40a65b0 con 0x7f10f414de50 2023-03-03T11:53:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f37fe700 1 -- 172.21.15.7:0/275867339 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e40034d0 con 0x7f10f414de50 2023-03-03T11:53:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f9482700 1 -- 172.21.15.7:0/275867339 >> v2:172.21.15.143:3300/0 conn(0x7f10f40c65c0 msgr2=0x7f10f40c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:53:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f9482700 1 --2- 172.21.15.7:0/275867339 >> v2:172.21.15.143:3300/0 conn(0x7f10f40c65c0 0x7f10f40c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f9482700 1 -- 172.21.15.7:0/275867339 >> v1:172.21.15.143:6789/0 conn(0x7f10f414de50 legacy=0x7f10f406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f9482700 1 -- 172.21.15.7:0/275867339 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f41544d0 con 0x7f10f40a9af0 2023-03-03T11:53:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f3fff700 1 --2- 172.21.15.7:0/275867339 >> v2:172.21.15.143:3300/0 conn(0x7f10f40c65c0 0x7f10f40c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:53:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f9482700 1 --2- 172.21.15.7:0/275867339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40a9af0 0x7f10f40c2c60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009d90 tx=0x7f10ec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c27da32482eb1b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:53:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f37fe700 1 -- 172.21.15.7:0/275867339 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec00e070 con 0x7f10f40a9af0 2023-03-03T11:53:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f37fe700 1 -- 172.21.15.7:0/275867339 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ec00bb20 con 0x7f10f40a9af0 2023-03-03T11:53:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.347+0000 7f10f37fe700 1 -- 172.21.15.7:0/275867339 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec012430 con 0x7f10f40a9af0 2023-03-03T11:53:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.348+0000 7f10faee5700 1 -- 172.21.15.7:0/275867339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40a9af0 msgr2=0x7f10f40c2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.348+0000 7f10faee5700 1 --2- 172.21.15.7:0/275867339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40a9af0 0x7f10f40c2c60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009d90 tx=0x7f10ec00b330 comp rx=0 tx=0).stop 2023-03-03T11:53:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.348+0000 7f10faee5700 1 -- 172.21.15.7:0/275867339 shutdown_connections 2023-03-03T11:53:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.348+0000 7f10faee5700 1 --2- 172.21.15.7:0/275867339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40a9af0 0x7f10f40c2c60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.348+0000 7f10faee5700 1 --2- 172.21.15.7:0/275867339 >> v2:172.21.15.143:3300/0 conn(0x7f10f40c65c0 0x7f10f40c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.348+0000 7f10faee5700 1 -- 172.21.15.7:0/275867339 >> 172.21.15.7:0/275867339 conn(0x7f10f40a9430 msgr2=0x7f10f40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:53:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.348+0000 7f10faee5700 1 -- 172.21.15.7:0/275867339 shutdown_connections 2023-03-03T11:53:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.348+0000 7f10faee5700 1 -- 172.21.15.7:0/275867339 wait complete. 2023-03-03T11:53:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.349+0000 7f10faee5700 1 Processor -- start 2023-03-03T11:53:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.349+0000 7f10faee5700 1 -- start start 2023-03-03T11:53:29.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10faee5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10f40a9af0 0x7f10f40c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:29.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10faee5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40c65c0 0x7f10f4147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:29.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10faee5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10f414de50 0x7f10f4148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:29.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10faee5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f4151400 con 0x7f10f414de50 2023-03-03T11:53:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10faee5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f406da20 con 0x7f10f40a9af0 2023-03-03T11:53:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10faee5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f40c1400 con 0x7f10f40c65c0 2023-03-03T11:53:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10f9482700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10f414de50 0x7f10f4148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10f9482700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10f414de50 0x7f10f4148270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38240/0 (socket says 172.21.15.7:38240) 2023-03-03T11:53:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10f9482700 1 -- 172.21.15.7:0/122343891 learned_addr learned my addr 172.21.15.7:0/122343891 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:53:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10f8c81700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10f40a9af0 0x7f10f40c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:29.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10f3fff700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40c65c0 0x7f10f4147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10f9482700 1 -- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40c65c0 msgr2=0x7f10f4147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10f9482700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40c65c0 0x7f10f4147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10f9482700 1 -- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10f40a9af0 msgr2=0x7f10f40c4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10f9482700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10f40a9af0 0x7f10f40c4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.350+0000 7f10f9482700 1 -- 172.21.15.7:0/122343891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00094a0 con 0x7f10f414de50 2023-03-03T11:53:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.351+0000 7f10f9482700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10f414de50 0x7f10f4148270 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f10ec014040 tx=0x7f10ec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:53:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.351+0000 7f10f17fa700 1 -- 172.21.15.7:0/122343891 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0129f0 con 0x7f10f414de50 2023-03-03T11:53:29.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.351+0000 7f10f17fa700 1 -- 172.21.15.7:0/122343891 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ec020070 con 0x7f10f414de50 2023-03-03T11:53:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.351+0000 7f10faee5700 1 -- 172.21.15.7:0/122343891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ec009a70 con 0x7f10f414de50 2023-03-03T11:53:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.351+0000 7f10f17fa700 1 -- 172.21.15.7:0/122343891 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec012db0 con 0x7f10f414de50 2023-03-03T11:53:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.351+0000 7f10faee5700 1 -- 172.21.15.7:0/122343891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f4154a90 con 0x7f10f414de50 2023-03-03T11:53:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.352+0000 7f10faee5700 1 -- 172.21.15.7:0/122343891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f41547b0 con 0x7f10f414de50 2023-03-03T11:53:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.352+0000 7f10f17fa700 1 -- 172.21.15.7:0/122343891 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f10ec004040 con 0x7f10f414de50 2023-03-03T11:53:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.353+0000 7f10f17fa700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10d80682f0 0x7f10d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:53:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.353+0000 7f10f17fa700 1 -- 172.21.15.7:0/122343891 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f10ec022070 con 0x7f10f414de50 2023-03-03T11:53:29.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.353+0000 7f10f8c81700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10d80682f0 0x7f10d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:53:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.354+0000 7f10f8c81700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10d80682f0 0x7f10d806a7a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f10e80011e0 tx=0x7f10e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:53:29.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.357+0000 7f10f17fa700 1 -- 172.21.15.7:0/122343891 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f10ec0548b0 con 0x7f10f414de50 2023-03-03T11:53:29.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.594+0000 7f10faee5700 1 -- 172.21.15.7:0/122343891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10f4153e80 con 0x7f10f414de50 2023-03-03T11:53:29.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.594+0000 7f10f17fa700 1 -- 172.21.15.7:0/122343891 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f10ec057ed0 con 0x7f10f414de50 2023-03-03T11:53:29.597 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:53:29.598 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:53:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 -- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10d80682f0 msgr2=0x7f10d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10d80682f0 0x7f10d806a7a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f10e80011e0 tx=0x7f10e8009040 comp rx=0 tx=0).stop 2023-03-03T11:53:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 -- 172.21.15.7:0/122343891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10f414de50 msgr2=0x7f10f4148270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:53:29.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10f414de50 0x7f10f4148270 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f10ec014040 tx=0x7f10ec004ad0 comp rx=0 tx=0).stop 2023-03-03T11:53:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 -- 172.21.15.7:0/122343891 shutdown_connections 2023-03-03T11:53:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10d80682f0 0x7f10d806a7a0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10f414de50 0x7f10f4148270 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10f40a9af0 0x7f10f40c4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 --2- 172.21.15.7:0/122343891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10f40c65c0 0x7f10f4147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:53:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 -- 172.21.15.7:0/122343891 >> 172.21.15.7:0/122343891 conn(0x7f10f40a9430 msgr2=0x7f10f41523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:53:29.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 -- 172.21.15.7:0/122343891 shutdown_connections 2023-03-03T11:53:29.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:53:29.596+0000 7f10faee5700 1 -- 172.21.15.7:0/122343891 wait complete. 2023-03-03T11:53:29.601 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:53:29.611 INFO:tasks.mds_thrash.fs.[b]:waiting for 47 secs before thrashing 2023-03-03T11:53:50.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:53:50.366 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:16.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:54:16.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.748+0000 7fb9f4d82700 1 Processor -- start 2023-03-03T11:54:16.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.748+0000 7fb9f4d82700 1 -- start start 2023-03-03T11:54:16.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.749+0000 7fb9f4d82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9f00719e0 0x7fb9f0150930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:16.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.749+0000 7fb9f4d82700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f00aae60 con 0x7fb9f0151e20 2023-03-03T11:54:16.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.749+0000 7fb9f4d82700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f0152270 con 0x7fb9f00c14f0 2023-03-03T11:54:16.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.749+0000 7fb9f4d82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f00c2760 con 0x7fb9f00719e0 2023-03-03T11:54:16.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.749+0000 7fb9edd9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb9f0151e20 0x7fb9f00aa750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60460/0 (socket says 172.21.15.7:60460) 2023-03-03T11:54:16.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.749+0000 7fb9edd9b700 1 -- 172.21.15.7:0/1196656908 learned_addr learned my addr 172.21.15.7:0/1196656908 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:54:16.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875048019 0 0) 0x7fb9d8001110 con 0x7fb9f0151e20 2023-03-03T11:54:16.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ee59c700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9f00719e0 0x7fb9f0150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:16.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc0046c0 con 0x7fb9f0151e20 2023-03-03T11:54:16.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2864694006 0 0) 0x7fb9dc0046c0 con 0x7fb9f0151e20 2023-03-03T11:54:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9f00719e0 msgr2=0x7fb9f0150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:54:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9f00719e0 0x7fb9f0150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 >> v1:172.21.15.143:6789/0 conn(0x7fb9f00c14f0 legacy=0x7fb9f015b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f00a93d0 con 0x7fb9f0151e20 2023-03-03T11:54:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb9d8001800 con 0x7fb9f0151e20 2023-03-03T11:54:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 >> v1:172.21.15.7:6789/0 conn(0x7fb9f0151e20 legacy=0x7fb9f00aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9dc004bb0 0x7fb9dc004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 0x7fb9dc005dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9dc006430 0x7fb9dc00cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f00c2760 con 0x7fb9dc005960 2023-03-03T11:54:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc00d2c0 con 0x7fb9dc006430 2023-03-03T11:54:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc003550 con 0x7fb9dc004bb0 2023-03-03T11:54:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ee59c700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9f00719e0 0x7fb9f0150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:54:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9edd9b700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 0x7fb9dc005dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9eed9d700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9dc006430 0x7fb9dc00cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.750+0000 7fb9ee59c700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9dc004bb0 0x7fb9dc004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9edd9b700 1 -- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9dc004bb0 msgr2=0x7fb9dc004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:54:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9edd9b700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9dc004bb0 0x7fb9dc004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9edd9b700 1 -- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9dc006430 msgr2=0x7fb9dc00cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:54:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9edd9b700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9dc006430 0x7fb9dc00cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9edd9b700 1 -- 172.21.15.7:0/1196656908 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0046c0 con 0x7fb9dc005960 2023-03-03T11:54:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9ee59c700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9dc004bb0 0x7fb9dc004fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:54:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9edd9b700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 0x7fb9dc005dd0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800c240 tx=0x7fb9dc006990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b7226d859054f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8007db0 con 0x7fb9dc005960 2023-03-03T11:54:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d800ce00 con 0x7fb9dc005960 2023-03-03T11:54:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9ed59a700 1 -- 172.21.15.7:0/1196656908 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8005ba0 con 0x7fb9dc005960 2023-03-03T11:54:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9f4d82700 1 -- 172.21.15.7:0/1196656908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 msgr2=0x7fb9dc005dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9f4d82700 1 --2- 172.21.15.7:0/1196656908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 0x7fb9dc005dd0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800c240 tx=0x7fb9dc006990 comp rx=0 tx=0).stop 2023-03-03T11:54:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.751+0000 7fb9eed9d700 1 -- 172.21.15.7:0/1196656908 reap_dead start 2023-03-03T11:54:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.752+0000 7fb9f4d82700 1 -- 172.21.15.7:0/1196656908 shutdown_connections 2023-03-03T11:54:16.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.752+0000 7fb9f4d82700 1 -- 172.21.15.7:0/1196656908 >> 172.21.15.7:0/1196656908 conn(0x7fb9f00c62e0 msgr2=0x7fb9f00c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:54:16.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.752+0000 7fb9f4d82700 1 -- 172.21.15.7:0/1196656908 shutdown_connections 2023-03-03T11:54:16.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.752+0000 7fb9f4d82700 1 -- 172.21.15.7:0/1196656908 wait complete. 2023-03-03T11:54:16.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.753+0000 7fb9f4d82700 1 Processor -- start 2023-03-03T11:54:16.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.753+0000 7fb9f4d82700 1 -- start start 2023-03-03T11:54:16.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9f4d82700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 0x7fb9f0151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:16.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9f4d82700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9f0146a20 0x7fb9f00c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:16.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9f4d82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9f014fb30 0x7fb9f01457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:16.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9f4d82700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f0146840 con 0x7fb9dc005960 2023-03-03T11:54:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9f4d82700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f0151780 con 0x7fb9f0146a20 2023-03-03T11:54:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9f4d82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f0150db0 con 0x7fb9f014fb30 2023-03-03T11:54:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9ee59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 0x7fb9f0151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9ee59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 0x7fb9f0151240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38260/0 (socket says 172.21.15.7:38260) 2023-03-03T11:54:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9ee59c700 1 -- 172.21.15.7:0/3657987251 learned_addr learned my addr 172.21.15.7:0/3657987251 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:54:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9ee59c700 1 -- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9f014fb30 msgr2=0x7fb9f01457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9ee59c700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9f014fb30 0x7fb9f01457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:16.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9ee59c700 1 -- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9f0146a20 msgr2=0x7fb9f00c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9ee59c700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9f0146a20 0x7fb9f00c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9ee59c700 1 -- 172.21.15.7:0/3657987251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f00a9990 con 0x7fb9dc005960 2023-03-03T11:54:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9ee59c700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 0x7fb9f0151240 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000f4c0 tx=0x7fb9e000f880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9e6ffd700 1 -- 172.21.15.7:0/3657987251 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e000c070 con 0x7fb9dc005960 2023-03-03T11:54:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9e6ffd700 1 -- 172.21.15.7:0/3657987251 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e000fed0 con 0x7fb9dc005960 2023-03-03T11:54:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9f4d82700 1 -- 172.21.15.7:0/3657987251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f00a96b0 con 0x7fb9dc005960 2023-03-03T11:54:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.754+0000 7fb9e6ffd700 1 -- 172.21.15.7:0/3657987251 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0014a90 con 0x7fb9dc005960 2023-03-03T11:54:16.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.755+0000 7fb9f4d82700 1 -- 172.21.15.7:0/3657987251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f00a9c70 con 0x7fb9dc005960 2023-03-03T11:54:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.755+0000 7fb9f4d82700 1 -- 172.21.15.7:0/3657987251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f0074bb0 con 0x7fb9dc005960 2023-03-03T11:54:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.756+0000 7fb9e6ffd700 1 -- 172.21.15.7:0/3657987251 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb9e0024070 con 0x7fb9dc005960 2023-03-03T11:54:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.756+0000 7fb9e6ffd700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9d00682f0 0x7fb9d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.756+0000 7fb9e6ffd700 1 -- 172.21.15.7:0/3657987251 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb9e0086410 con 0x7fb9dc005960 2023-03-03T11:54:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.756+0000 7fb9edd9b700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9d00682f0 0x7fb9d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:16.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.757+0000 7fb9edd9b700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9d00682f0 0x7fb9d006a7a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb9f01538a0 tx=0x7fb9d800a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:16.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.761+0000 7fb9e6ffd700 1 -- 172.21.15.7:0/3657987251 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb9e00526c0 con 0x7fb9dc005960 2023-03-03T11:54:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.998+0000 7fb9f4d82700 1 -- 172.21.15.7:0/3657987251 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9f00c29c0 con 0x7fb9dc005960 2023-03-03T11:54:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:16.999+0000 7fb9e6ffd700 1 -- 172.21.15.7:0/3657987251 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb9e0055ce0 con 0x7fb9dc005960 2023-03-03T11:54:17.001 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:54:17.001 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:54:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 -- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9d00682f0 msgr2=0x7fb9d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9d00682f0 0x7fb9d006a7a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb9f01538a0 tx=0x7fb9d800a000 comp rx=0 tx=0).stop 2023-03-03T11:54:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 -- 172.21.15.7:0/3657987251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 msgr2=0x7fb9f0151240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 0x7fb9f0151240 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000f4c0 tx=0x7fb9e000f880 comp rx=0 tx=0).stop 2023-03-03T11:54:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 -- 172.21.15.7:0/3657987251 shutdown_connections 2023-03-03T11:54:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9f014fb30 0x7fb9f01457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9d00682f0 0x7fb9d006a7a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9dc005960 0x7fb9f0151240 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 --2- 172.21.15.7:0/3657987251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9f0146a20 0x7fb9f00c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 -- 172.21.15.7:0/3657987251 >> 172.21.15.7:0/3657987251 conn(0x7fb9f00c62e0 msgr2=0x7fb9f00c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:54:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 -- 172.21.15.7:0/3657987251 shutdown_connections 2023-03-03T11:54:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:17.001+0000 7fb9f4d82700 1 -- 172.21.15.7:0/3657987251 wait complete. 2023-03-03T11:54:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:54:17.014 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before thrashing 2023-03-03T11:54:20.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:20.445 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:23.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:54:23.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.147+0000 7f4ce5b45700 1 Processor -- start 2023-03-03T11:54:23.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.147+0000 7f4ce5b45700 1 -- start start 2023-03-03T11:54:23.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4ce5b45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 0x7f4ce0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:23.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4ce5b45700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ce00abe20 0x7f4ce01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:23.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4ce5b45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce00c2f30 con 0x7f4ce014f9d0 2023-03-03T11:54:23.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4ce5b45700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce00c31d0 con 0x7f4ce00abe20 2023-03-03T11:54:23.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4ce5b45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce01524a0 con 0x7f4ce014fda0 2023-03-03T11:54:23.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4cdeffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 0x7f4ce0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:23.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4cdffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ce00abe20 0x7f4ce01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4cdffff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ce00abe20 0x7f4ce01529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:34054/0 (socket says 172.21.15.7:34054) 2023-03-03T11:54:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4cdeffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 0x7f4ce0156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:46350/0 (socket says 172.21.15.7:46350) 2023-03-03T11:54:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4cdf7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f4ce014f9d0 0x7f4ce0071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:35776/0 (socket says 172.21.15.7:35776) 2023-03-03T11:54:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.148+0000 7f4cdffff700 1 -- 172.21.15.7:0/1235127013 learned_addr learned my addr 172.21.15.7:0/1235127013 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:54:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4cdeffd700 1 -- 172.21.15.7:0/1235127013 >> v2:172.21.15.143:3300/0 conn(0x7f4ce00abe20 msgr2=0x7f4ce01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:54:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4cdeffd700 1 --2- 172.21.15.7:0/1235127013 >> v2:172.21.15.143:3300/0 conn(0x7f4ce00abe20 0x7f4ce01529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:23.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4cdeffd700 1 -- 172.21.15.7:0/1235127013 >> v1:172.21.15.143:6789/0 conn(0x7f4ce014f9d0 legacy=0x7f4ce0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4cde7fc700 1 -- 172.21.15.7:0/1235127013 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654973789 0 0) 0x7f4ce00c2f30 con 0x7f4ce014f9d0 2023-03-03T11:54:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4cdeffd700 1 -- 172.21.15.7:0/1235127013 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce00c34b0 con 0x7f4ce014fda0 2023-03-03T11:54:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4cdffff700 1 --2- 172.21.15.7:0/1235127013 >> v2:172.21.15.143:3300/0 conn(0x7f4ce00abe20 0x7f4ce01529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:54:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4cdeffd700 1 --2- 172.21.15.7:0/1235127013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 0x7f4ce0156430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009840 tx=0x7f4cd0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c550861d3deaf644 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4cde7fc700 1 -- 172.21.15.7:0/1235127013 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0011830 con 0x7f4ce014fda0 2023-03-03T11:54:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4cde7fc700 1 -- 172.21.15.7:0/1235127013 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd0011990 con 0x7f4ce014fda0 2023-03-03T11:54:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4cde7fc700 1 -- 172.21.15.7:0/1235127013 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0011d20 con 0x7f4ce014fda0 2023-03-03T11:54:23.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4ce5b45700 1 -- 172.21.15.7:0/1235127013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 msgr2=0x7f4ce0156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:23.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4ce5b45700 1 --2- 172.21.15.7:0/1235127013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 0x7f4ce0156430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009840 tx=0x7f4cd0009b50 comp rx=0 tx=0).stop 2023-03-03T11:54:23.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4ce5b45700 1 -- 172.21.15.7:0/1235127013 shutdown_connections 2023-03-03T11:54:23.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4ce5b45700 1 --2- 172.21.15.7:0/1235127013 >> v2:172.21.15.143:3300/0 conn(0x7f4ce00abe20 0x7f4ce01529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:23.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.149+0000 7f4ce5b45700 1 --2- 172.21.15.7:0/1235127013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 0x7f4ce0156430 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:23.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.150+0000 7f4ce5b45700 1 -- 172.21.15.7:0/1235127013 >> 172.21.15.7:0/1235127013 conn(0x7f4ce00c8900 msgr2=0x7f4ce00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:54:23.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.150+0000 7f4ce5b45700 1 -- 172.21.15.7:0/1235127013 shutdown_connections 2023-03-03T11:54:23.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.150+0000 7f4ce5b45700 1 -- 172.21.15.7:0/1235127013 wait complete. 2023-03-03T11:54:23.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.151+0000 7f4ce5b45700 1 Processor -- start 2023-03-03T11:54:23.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.151+0000 7f4ce5b45700 1 -- start start 2023-03-03T11:54:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4ce5b45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce00abe20 0x7f4ce006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4ce5b45700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ce014f9d0 0x7f4ce01ed880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4ce5b45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 0x7f4ce01ea100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4ce5b45700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce01ea640 con 0x7f4ce014f9d0 2023-03-03T11:54:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4ce5b45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce006ab70 con 0x7f4ce00abe20 2023-03-03T11:54:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4ce5b45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce006a890 con 0x7f4ce014fda0 2023-03-03T11:54:23.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ce014f9d0 0x7f4ce01ed880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ce014f9d0 0x7f4ce01ed880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38284/0 (socket says 172.21.15.7:38284) 2023-03-03T11:54:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdeffd700 1 -- 172.21.15.7:0/2262590434 learned_addr learned my addr 172.21.15.7:0/2262590434 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:54:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdffff700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 0x7f4ce01ea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdf7fe700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce00abe20 0x7f4ce006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdeffd700 1 -- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 msgr2=0x7f4ce01ea100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdeffd700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 0x7f4ce01ea100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdeffd700 1 -- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce00abe20 msgr2=0x7f4ce006a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:23.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdeffd700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce00abe20 0x7f4ce006a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdeffd700 1 -- 172.21.15.7:0/2262590434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd8009b30 con 0x7f4ce014f9d0 2023-03-03T11:54:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cdeffd700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ce014f9d0 0x7f4ce01ed880 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00090b0 tx=0x7f4cd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cc7fff700 1 -- 172.21.15.7:0/2262590434 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0004450 con 0x7f4ce014f9d0 2023-03-03T11:54:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cc7fff700 1 -- 172.21.15.7:0/2262590434 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd00045b0 con 0x7f4ce014f9d0 2023-03-03T11:54:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4ce5b45700 1 -- 172.21.15.7:0/2262590434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd00094a0 con 0x7f4ce014f9d0 2023-03-03T11:54:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.152+0000 7f4cc7fff700 1 -- 172.21.15.7:0/2262590434 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd00038c0 con 0x7f4ce014f9d0 2023-03-03T11:54:23.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.153+0000 7f4ce5b45700 1 -- 172.21.15.7:0/2262590434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce00ac350 con 0x7f4ce014f9d0 2023-03-03T11:54:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.153+0000 7f4ce5b45700 1 -- 172.21.15.7:0/2262590434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce00ab630 con 0x7f4ce014f9d0 2023-03-03T11:54:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.154+0000 7f4cc7fff700 1 -- 172.21.15.7:0/2262590434 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4cd000f050 con 0x7f4ce014f9d0 2023-03-03T11:54:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.154+0000 7f4cc7fff700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ccc0682f0 0x7f4ccc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.154+0000 7f4cc7fff700 1 -- 172.21.15.7:0/2262590434 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4cd0087370 con 0x7f4ce014f9d0 2023-03-03T11:54:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.155+0000 7f4cdf7fe700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ccc0682f0 0x7f4ccc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:23.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.155+0000 7f4cdf7fe700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ccc0682f0 0x7f4ccc06a7a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4007fd0 tx=0x7f4cd4007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:23.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.159+0000 7f4cc7fff700 1 -- 172.21.15.7:0/2262590434 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4cd0053710 con 0x7f4ce014f9d0 2023-03-03T11:54:23.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.396+0000 7f4ce5b45700 1 -- 172.21.15.7:0/2262590434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ce0071fa0 con 0x7f4ce014f9d0 2023-03-03T11:54:23.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.397+0000 7f4cc7fff700 1 -- 172.21.15.7:0/2262590434 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4cd0056d30 con 0x7f4ce014f9d0 2023-03-03T11:54:23.399 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:54:23.399 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:54:23.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 -- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ccc0682f0 msgr2=0x7f4ccc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:23.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ccc0682f0 0x7f4ccc06a7a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4007fd0 tx=0x7f4cd4007de0 comp rx=0 tx=0).stop 2023-03-03T11:54:23.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 -- 172.21.15.7:0/2262590434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ce014f9d0 msgr2=0x7f4ce01ed880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:23.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ce014f9d0 0x7f4ce01ed880 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00090b0 tx=0x7f4cd0004750 comp rx=0 tx=0).stop 2023-03-03T11:54:23.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 -- 172.21.15.7:0/2262590434 shutdown_connections 2023-03-03T11:54:23.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ce014fda0 0x7f4ce01ea100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:23.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ce00abe20 0x7f4ce006a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:23.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ccc0682f0 0x7f4ccc06a7a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:23.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 --2- 172.21.15.7:0/2262590434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ce014f9d0 0x7f4ce01ed880 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:23.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 -- 172.21.15.7:0/2262590434 >> 172.21.15.7:0/2262590434 conn(0x7f4ce00c8900 msgr2=0x7f4ce00c8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:54:23.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 -- 172.21.15.7:0/2262590434 shutdown_connections 2023-03-03T11:54:23.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:23.399+0000 7f4ce5b45700 1 -- 172.21.15.7:0/2262590434 wait complete. 2023-03-03T11:54:23.402 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:54:23.412 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T11:54:34.414 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:54:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.549+0000 7f4990223700 1 Processor -- start 2023-03-03T11:54:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.550+0000 7f4990223700 1 -- start start 2023-03-03T11:54:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.550+0000 7f4990223700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4988151d40 0x7f49880736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.550+0000 7f4990223700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498806d430 con 0x7f49880c2930 2023-03-03T11:54:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.550+0000 7f4990223700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498806cb00 con 0x7f4988073c30 2023-03-03T11:54:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.550+0000 7f4990223700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498806c2a0 con 0x7f4988151d40 2023-03-03T11:54:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498d7be700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f49880c2930 0x7f4988157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60508/0 (socket says 172.21.15.7:60508) 2023-03-03T11:54:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498d7be700 1 -- 172.21.15.7:0/3599038558 learned_addr learned my addr 172.21.15.7:0/3599038558 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:54:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498dfbf700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4988151d40 0x7f49880736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109823329 0 0) 0x7f498806d430 con 0x7f49880c2930 2023-03-03T11:54:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49740035b0 con 0x7f49880c2930 2023-03-03T11:54:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1225479328 0 0) 0x7f49740035b0 con 0x7f49880c2930 2023-03-03T11:54:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4988151d40 msgr2=0x7f49880736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:54:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4988151d40 0x7f49880736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 >> v1:172.21.15.143:6789/0 conn(0x7f4988073c30 legacy=0x7f49881581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498806cde0 con 0x7f49880c2930 2023-03-03T11:54:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f497c001670 con 0x7f49880c2930 2023-03-03T11:54:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 >> v1:172.21.15.7:6789/0 conn(0x7f49880c2930 legacy=0x7f4988157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4974003aa0 0x7f4974003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4974004850 0x7f4974004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4988151d40 0x7f497400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498806c2a0 con 0x7f4974003aa0 2023-03-03T11:54:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4974005530 con 0x7f4974004850 2023-03-03T11:54:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49740056f0 con 0x7f4988151d40 2023-03-03T11:54:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498dfbf700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4974003aa0 0x7f4974003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498d7be700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4974004850 0x7f4974004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.551+0000 7f498e7c0700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4988151d40 0x7f497400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f498dfbf700 1 -- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4988151d40 msgr2=0x7f497400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f498dfbf700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4988151d40 0x7f497400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f498dfbf700 1 -- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4974004850 msgr2=0x7f4974004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f498dfbf700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4974004850 0x7f4974004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f498dfbf700 1 -- 172.21.15.7:0/3599038558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49740035b0 con 0x7f4974003aa0 2023-03-03T11:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f498dfbf700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4974003aa0 0x7f4974003eb0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f4984009b60 tx=0x7f4974005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd28d09dd0a49ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498400e6c0 con 0x7f4974003aa0 2023-03-03T11:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498400e820 con 0x7f4974003aa0 2023-03-03T11:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f498cfbd700 1 -- 172.21.15.7:0/3599038558 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498400eb90 con 0x7f4974003aa0 2023-03-03T11:54:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f4990223700 1 -- 172.21.15.7:0/3599038558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4974003aa0 msgr2=0x7f4974003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f4990223700 1 --2- 172.21.15.7:0/3599038558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4974003aa0 0x7f4974003eb0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f4984009b60 tx=0x7f4974005880 comp rx=0 tx=0).stop 2023-03-03T11:54:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.552+0000 7f498e7c0700 1 -- 172.21.15.7:0/3599038558 reap_dead start 2023-03-03T11:54:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.553+0000 7f4990223700 1 -- 172.21.15.7:0/3599038558 shutdown_connections 2023-03-03T11:54:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.553+0000 7f4990223700 1 -- 172.21.15.7:0/3599038558 >> 172.21.15.7:0/3599038558 conn(0x7f49880a94d0 msgr2=0x7f49880a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:54:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.553+0000 7f4990223700 1 -- 172.21.15.7:0/3599038558 shutdown_connections 2023-03-03T11:54:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.553+0000 7f4990223700 1 -- 172.21.15.7:0/3599038558 wait complete. 2023-03-03T11:54:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.554+0000 7f4990223700 1 Processor -- start 2023-03-03T11:54:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.554+0000 7f4990223700 1 -- start start 2023-03-03T11:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f4990223700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4974003aa0 0x7f49880c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f4990223700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49880c6a70 0x7f4988154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f4990223700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f498806ab80 0x7f498806b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f4990223700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4988155180 con 0x7f49880c6a70 2023-03-03T11:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f4990223700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49880c60b0 con 0x7f4974003aa0 2023-03-03T11:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f4990223700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49881eaf90 con 0x7f498806ab80 2023-03-03T11:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f498d7be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49880c6a70 0x7f4988154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f498d7be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49880c6a70 0x7f4988154730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38308/0 (socket says 172.21.15.7:38308) 2023-03-03T11:54:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f498d7be700 1 -- 172.21.15.7:0/1929039982 learned_addr learned my addr 172.21.15.7:0/1929039982 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:54:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f498dfbf700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4974003aa0 0x7f49880c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.555+0000 7f498e7c0700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f498806ab80 0x7f498806b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f498d7be700 1 -- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f498806ab80 msgr2=0x7f498806b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f498d7be700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f498806ab80 0x7f498806b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f498d7be700 1 -- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4974003aa0 msgr2=0x7f49880c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f498d7be700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4974003aa0 0x7f49880c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f498d7be700 1 -- 172.21.15.7:0/1929039982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498806bea0 con 0x7f49880c6a70 2023-03-03T11:54:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f498d7be700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49880c6a70 0x7f4988154730 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f497c007290 tx=0x7f497c0075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f497a7fc700 1 -- 172.21.15.7:0/1929039982 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c012560 con 0x7f49880c6a70 2023-03-03T11:54:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f497a7fc700 1 -- 172.21.15.7:0/1929039982 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497c0126e0 con 0x7f49880c6a70 2023-03-03T11:54:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f4990223700 1 -- 172.21.15.7:0/1929039982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498806d0c0 con 0x7f49880c6a70 2023-03-03T11:54:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f497a7fc700 1 -- 172.21.15.7:0/1929039982 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c012a90 con 0x7f49880c6a70 2023-03-03T11:54:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.556+0000 7f4990223700 1 -- 172.21.15.7:0/1929039982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498806c030 con 0x7f49880c6a70 2023-03-03T11:54:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.557+0000 7f4990223700 1 -- 172.21.15.7:0/1929039982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4988074bb0 con 0x7f49880c6a70 2023-03-03T11:54:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.557+0000 7f497a7fc700 1 -- 172.21.15.7:0/1929039982 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f497c012bf0 con 0x7f49880c6a70 2023-03-03T11:54:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.558+0000 7f497a7fc700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f496c068320 0x7f496c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.558+0000 7f497a7fc700 1 -- 172.21.15.7:0/1929039982 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f497c086290 con 0x7f49880c6a70 2023-03-03T11:54:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.558+0000 7f498dfbf700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f496c068320 0x7f496c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.559+0000 7f498dfbf700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f496c068320 0x7f496c06a7d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f49840090b0 tx=0x7f498401f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.563+0000 7f497a7fc700 1 -- 172.21.15.7:0/1929039982 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f497c0525d0 con 0x7f49880c6a70 2023-03-03T11:54:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.800+0000 7f4990223700 1 -- 172.21.15.7:0/1929039982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49880a9fb0 con 0x7f49880c6a70 2023-03-03T11:54:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.801+0000 7f497a7fc700 1 -- 172.21.15.7:0/1929039982 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f497c055bf0 con 0x7f49880c6a70 2023-03-03T11:54:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:54:34.804 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:54:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 -- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f496c068320 msgr2=0x7f496c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f496c068320 0x7f496c06a7d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f49840090b0 tx=0x7f498401f040 comp rx=0 tx=0).stop 2023-03-03T11:54:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 -- 172.21.15.7:0/1929039982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49880c6a70 msgr2=0x7f4988154730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49880c6a70 0x7f4988154730 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f497c007290 tx=0x7f497c0075a0 comp rx=0 tx=0).stop 2023-03-03T11:54:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 -- 172.21.15.7:0/1929039982 shutdown_connections 2023-03-03T11:54:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f498806ab80 0x7f498806b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4974003aa0 0x7f49880c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f496c068320 0x7f496c06a7d0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 --2- 172.21.15.7:0/1929039982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49880c6a70 0x7f4988154730 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 -- 172.21.15.7:0/1929039982 >> 172.21.15.7:0/1929039982 conn(0x7f49880a94d0 msgr2=0x7f4988158160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:54:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 -- 172.21.15.7:0/1929039982 shutdown_connections 2023-03-03T11:54:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:34.803+0000 7f4990223700 1 -- 172.21.15.7:0/1929039982 wait complete. 2023-03-03T11:54:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:54:34.817 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-03-03T11:54:38.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:54:38.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.185+0000 7fcdecf23700 1 Processor -- start 2023-03-03T11:54:38.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.185+0000 7fcdecf23700 1 -- start start 2023-03-03T11:54:38.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcdecf23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcde8151770 0x7fcde8151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:38.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcdecf23700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde8154910 con 0x7fcde8155390 2023-03-03T11:54:38.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcdecf23700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80c4540 con 0x7fcde806e020 2023-03-03T11:54:38.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcdecf23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde8070b90 con 0x7fcde8151770 2023-03-03T11:54:38.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcde659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcde8155390 0x7fcde80c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60534/0 (socket says 172.21.15.7:60534) 2023-03-03T11:54:38.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcde659c700 1 -- 172.21.15.7:0/2517644288 learned_addr learned my addr 172.21.15.7:0/2517644288 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:54:38.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcde6d9d700 1 --2- 172.21.15.7:0/2517644288 >> v2:172.21.15.143:3300/0 conn(0x7fcde8151770 0x7fcde8151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcde559a700 1 -- 172.21.15.7:0/2517644288 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279583237 0 0) 0x7fcde8154910 con 0x7fcde8155390 2023-03-03T11:54:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcde559a700 1 -- 172.21.15.7:0/2517644288 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd00035b0 con 0x7fcde8155390 2023-03-03T11:54:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcde559a700 1 -- 172.21.15.7:0/2517644288 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796527283 0 0) 0x7fcddc001110 con 0x7fcde806e020 2023-03-03T11:54:38.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.186+0000 7fcde559a700 1 -- 172.21.15.7:0/2517644288 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde8154910 con 0x7fcde806e020 2023-03-03T11:54:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.187+0000 7fcde6d9d700 1 -- 172.21.15.7:0/2517644288 >> v1:172.21.15.143:6789/0 conn(0x7fcde806e020 legacy=0x7fcde8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.187+0000 7fcde559a700 1 -- 172.21.15.7:0/2517644288 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2687213436 0 0) 0x7fcdd00035b0 con 0x7fcde8155390 2023-03-03T11:54:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.187+0000 7fcde6d9d700 1 -- 172.21.15.7:0/2517644288 >> v1:172.21.15.7:6789/0 conn(0x7fcde8155390 legacy=0x7fcde80c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:38.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.187+0000 7fcde6d9d700 1 -- 172.21.15.7:0/2517644288 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80702a0 con 0x7fcde8151770 2023-03-03T11:54:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.187+0000 7fcde6d9d700 1 --2- 172.21.15.7:0/2517644288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde8151770 0x7fcde8151bd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009d90 tx=0x7fcde000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e49fc43057970285 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.187+0000 7fcde559a700 1 -- 172.21.15.7:0/2517644288 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde000e070 con 0x7fcde8151770 2023-03-03T11:54:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.187+0000 7fcde559a700 1 -- 172.21.15.7:0/2517644288 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde0003a20 con 0x7fcde8151770 2023-03-03T11:54:38.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.187+0000 7fcde559a700 1 -- 172.21.15.7:0/2517644288 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8001670 con 0x7fcde8151770 2023-03-03T11:54:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.187+0000 7fcdecf23700 1 -- 172.21.15.7:0/2517644288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde8151770 msgr2=0x7fcde8151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.187+0000 7fcdecf23700 1 --2- 172.21.15.7:0/2517644288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde8151770 0x7fcde8151bd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009d90 tx=0x7fcde000b330 comp rx=0 tx=0).stop 2023-03-03T11:54:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.188+0000 7fcdecf23700 1 -- 172.21.15.7:0/2517644288 shutdown_connections 2023-03-03T11:54:38.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.188+0000 7fcdecf23700 1 --2- 172.21.15.7:0/2517644288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde8151770 0x7fcde8151bd0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.188+0000 7fcdecf23700 1 -- 172.21.15.7:0/2517644288 >> 172.21.15.7:0/2517644288 conn(0x7fcde806c110 msgr2=0x7fcde806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:54:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.188+0000 7fcdecf23700 1 -- 172.21.15.7:0/2517644288 shutdown_connections 2023-03-03T11:54:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.188+0000 7fcdecf23700 1 -- 172.21.15.7:0/2517644288 wait complete. 2023-03-03T11:54:38.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.189+0000 7fcdecf23700 1 Processor -- start 2023-03-03T11:54:38.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.189+0000 7fcdecf23700 1 -- start start 2023-03-03T11:54:38.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcdecf23700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcde806e020 0x7fcde81ebb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcdecf23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcde8151770 0x7fcde81e9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcdecf23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde8155390 0x7fcde80c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:38.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcdecf23700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde81eb6e0 con 0x7fcde806e020 2023-03-03T11:54:38.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcdecf23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde814b9e0 con 0x7fcde8155390 2023-03-03T11:54:38.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcdecf23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde81ec040 con 0x7fcde8151770 2023-03-03T11:54:38.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcde806e020 0x7fcde81ebb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:38.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcde806e020 0x7fcde81ebb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38328/0 (socket says 172.21.15.7:38328) 2023-03-03T11:54:38.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde659c700 1 -- 172.21.15.7:0/252303068 learned_addr learned my addr 172.21.15.7:0/252303068 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:54:38.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde5d9b700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcde8151770 0x7fcde81e9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:38.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde6d9d700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde8155390 0x7fcde80c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:38.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde659c700 1 -- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcde8151770 msgr2=0x7fcde81e9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde659c700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcde8151770 0x7fcde81e9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde659c700 1 -- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde8155390 msgr2=0x7fcde80c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde659c700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde8155390 0x7fcde80c2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:38.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde659c700 1 -- 172.21.15.7:0/252303068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde0009a70 con 0x7fcde806e020 2023-03-03T11:54:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcde659c700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcde806e020 0x7fcde81ebb00 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800c700 tx=0x7fcdd800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.190+0000 7fcdd6ffd700 1 -- 172.21.15.7:0/252303068 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8004c80 con 0x7fcde806e020 2023-03-03T11:54:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.191+0000 7fcdd6ffd700 1 -- 172.21.15.7:0/252303068 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd8004de0 con 0x7fcde806e020 2023-03-03T11:54:38.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.191+0000 7fcdecf23700 1 -- 172.21.15.7:0/252303068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde8070820 con 0x7fcde806e020 2023-03-03T11:54:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.191+0000 7fcdd6ffd700 1 -- 172.21.15.7:0/252303068 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8007310 con 0x7fcde806e020 2023-03-03T11:54:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.191+0000 7fcdecf23700 1 -- 172.21.15.7:0/252303068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde8070540 con 0x7fcde806e020 2023-03-03T11:54:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.191+0000 7fcdecf23700 1 -- 172.21.15.7:0/252303068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde8155760 con 0x7fcde806e020 2023-03-03T11:54:38.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.192+0000 7fcdd6ffd700 1 -- 172.21.15.7:0/252303068 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcdd801a020 con 0x7fcde806e020 2023-03-03T11:54:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.193+0000 7fcdd6ffd700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc80682f0 0x7fcdc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:54:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.193+0000 7fcdd6ffd700 1 -- 172.21.15.7:0/252303068 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcdd8085ee0 con 0x7fcde806e020 2023-03-03T11:54:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.193+0000 7fcde5d9b700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc80682f0 0x7fcdc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:54:38.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.193+0000 7fcde5d9b700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc80682f0 0x7fcdc806a7a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fcddc001270 tx=0x7fcddc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:54:38.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.197+0000 7fcdd6ffd700 1 -- 172.21.15.7:0/252303068 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcdd8052190 con 0x7fcde806e020 2023-03-03T11:54:38.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.436+0000 7fcdecf23700 1 -- 172.21.15.7:0/252303068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcde81eb950 con 0x7fcde806e020 2023-03-03T11:54:38.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.436+0000 7fcdd6ffd700 1 -- 172.21.15.7:0/252303068 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcdd80557b0 con 0x7fcde806e020 2023-03-03T11:54:38.439 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:54:38.439 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:54:38.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.438+0000 7fcdecf23700 1 -- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc80682f0 msgr2=0x7fcdc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:38.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.438+0000 7fcdecf23700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc80682f0 0x7fcdc806a7a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fcddc001270 tx=0x7fcddc009040 comp rx=0 tx=0).stop 2023-03-03T11:54:38.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.438+0000 7fcdecf23700 1 -- 172.21.15.7:0/252303068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcde806e020 msgr2=0x7fcde81ebb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:54:38.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.438+0000 7fcdecf23700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcde806e020 0x7fcde81ebb00 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800c700 tx=0x7fcdd800cac0 comp rx=0 tx=0).stop 2023-03-03T11:54:38.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.438+0000 7fcdecf23700 1 -- 172.21.15.7:0/252303068 shutdown_connections 2023-03-03T11:54:38.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.438+0000 7fcdecf23700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcde8155390 0x7fcde80c2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:38.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.438+0000 7fcdecf23700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc80682f0 0x7fcdc806a7a0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:38.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.438+0000 7fcdecf23700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcde806e020 0x7fcde81ebb00 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:38.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.438+0000 7fcdecf23700 1 --2- 172.21.15.7:0/252303068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcde8151770 0x7fcde81e9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:54:38.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.438+0000 7fcdecf23700 1 -- 172.21.15.7:0/252303068 >> 172.21.15.7:0/252303068 conn(0x7fcde806c110 msgr2=0x7fcde80740c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:54:38.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.439+0000 7fcdecf23700 1 -- 172.21.15.7:0/252303068 shutdown_connections 2023-03-03T11:54:38.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:54:38.439+0000 7fcdecf23700 1 -- 172.21.15.7:0/252303068 wait complete. 2023-03-03T11:54:38.443 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:54:38.454 INFO:tasks.mds_thrash.fs.[a]:waiting for 90 secs before thrashing 2023-03-03T11:54:50.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:54:50.576 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:20.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:20.710 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:51.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:55:51.029 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:08.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:56:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.589+0000 7f459536a700 1 Processor -- start 2023-03-03T11:56:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.589+0000 7f459536a700 1 -- start start 2023-03-03T11:56:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f459536a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f45900c00e0 0x7f45900c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f459536a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45900c7310 con 0x7f45900caa30 2023-03-03T11:56:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f459536a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45900bfd10 con 0x7f4590154ff0 2023-03-03T11:56:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f459536a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4590155430 con 0x7f45900c00e0 2023-03-03T11:56:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f458e7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f45900c00e0 0x7f45900c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f458effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f45900caa30 0x7f45900c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60588/0 (socket says 172.21.15.7:60588) 2023-03-03T11:56:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f458effd700 1 -- 172.21.15.7:0/2935921590 learned_addr learned my addr 172.21.15.7:0/2935921590 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:56:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f458e7fc700 1 -- 172.21.15.7:0/2935921590 >> v1:172.21.15.143:6789/0 conn(0x7f4590154ff0 legacy=0x7f459015ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:56:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f458e7fc700 1 -- 172.21.15.7:0/2935921590 >> v1:172.21.15.7:6789/0 conn(0x7f45900caa30 legacy=0x7f45900c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f458e7fc700 1 -- 172.21.15.7:0/2935921590 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45900bfa30 con 0x7f45900c00e0 2023-03-03T11:56:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.590+0000 7f458dffb700 1 -- 172.21.15.7:0/2935921590 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202106874 0 0) 0x7f45900c7310 con 0x7f45900caa30 2023-03-03T11:56:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f458e7fc700 1 --2- 172.21.15.7:0/2935921590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900c00e0 0x7f45900c5c60 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f4580009820 tx=0x7f4580009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f658734a789a697 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f458dffb700 1 -- 172.21.15.7:0/2935921590 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580005e30 con 0x7f45900c00e0 2023-03-03T11:56:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f458dffb700 1 -- 172.21.15.7:0/2935921590 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4580004030 con 0x7f45900c00e0 2023-03-03T11:56:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f458dffb700 1 -- 172.21.15.7:0/2935921590 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45800043e0 con 0x7f45900c00e0 2023-03-03T11:56:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f459536a700 1 -- 172.21.15.7:0/2935921590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900c00e0 msgr2=0x7f45900c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f459536a700 1 --2- 172.21.15.7:0/2935921590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900c00e0 0x7f45900c5c60 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f4580009820 tx=0x7f4580009b30 comp rx=0 tx=0).stop 2023-03-03T11:56:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f459536a700 1 -- 172.21.15.7:0/2935921590 shutdown_connections 2023-03-03T11:56:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f459536a700 1 --2- 172.21.15.7:0/2935921590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900c00e0 0x7f45900c5c60 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f459536a700 1 -- 172.21.15.7:0/2935921590 >> 172.21.15.7:0/2935921590 conn(0x7f4590066e40 msgr2=0x7f4590071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:56:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f459536a700 1 -- 172.21.15.7:0/2935921590 shutdown_connections 2023-03-03T11:56:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.591+0000 7f459536a700 1 -- 172.21.15.7:0/2935921590 wait complete. 2023-03-03T11:56:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.593+0000 7f459536a700 1 Processor -- start 2023-03-03T11:56:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.593+0000 7f459536a700 1 -- start start 2023-03-03T11:56:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.593+0000 7f459536a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45900c00e0 0x7f4590157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.593+0000 7f459536a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900caa30 0x7f459014adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.593+0000 7f459536a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4590154ff0 0x7f45901ed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.593+0000 7f459536a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45901ee0e0 con 0x7f45900caa30 2023-03-03T11:56:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.593+0000 7f459536a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459014a5c0 con 0x7f45900c00e0 2023-03-03T11:56:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.593+0000 7f459536a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459014d9f0 con 0x7f4590154ff0 2023-03-03T11:56:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.593+0000 7f458e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900caa30 0x7f459014adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f458e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900caa30 0x7f459014adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38384/0 (socket says 172.21.15.7:38384) 2023-03-03T11:56:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f458e7fc700 1 -- 172.21.15.7:0/3092577454 learned_addr learned my addr 172.21.15.7:0/3092577454 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:56:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f458effd700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45900c00e0 0x7f4590157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f458f7fe700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4590154ff0 0x7f45901ed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f458e7fc700 1 -- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4590154ff0 msgr2=0x7f45901ed3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f458e7fc700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4590154ff0 0x7f45901ed3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f458e7fc700 1 -- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45900c00e0 msgr2=0x7f4590157ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f458e7fc700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45900c00e0 0x7f4590157ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f458e7fc700 1 -- 172.21.15.7:0/3092577454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4580009430 con 0x7f45900caa30 2023-03-03T11:56:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f458e7fc700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900caa30 0x7f459014adb0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f4580009b70 tx=0x7f4580004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f45877fe700 1 -- 172.21.15.7:0/3092577454 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458001d630 con 0x7f45900caa30 2023-03-03T11:56:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f45877fe700 1 -- 172.21.15.7:0/3092577454 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458001d7d0 con 0x7f45900caa30 2023-03-03T11:56:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f459536a700 1 -- 172.21.15.7:0/3092577454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45901547f0 con 0x7f45900caa30 2023-03-03T11:56:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f45877fe700 1 -- 172.21.15.7:0/3092577454 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458001db80 con 0x7f45900caa30 2023-03-03T11:56:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.594+0000 7f459536a700 1 -- 172.21.15.7:0/3092577454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4590154510 con 0x7f45900caa30 2023-03-03T11:56:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.595+0000 7f459536a700 1 -- 172.21.15.7:0/3092577454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4590154ad0 con 0x7f45900caa30 2023-03-03T11:56:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.596+0000 7f45877fe700 1 -- 172.21.15.7:0/3092577454 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f458002d070 con 0x7f45900caa30 2023-03-03T11:56:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.596+0000 7f45877fe700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45700682f0 0x7f457006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:08.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.596+0000 7f45877fe700 1 -- 172.21.15.7:0/3092577454 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f45800877c0 con 0x7f45900caa30 2023-03-03T11:56:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.596+0000 7f458effd700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45700682f0 0x7f457006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.597+0000 7f458effd700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45700682f0 0x7f457006a7a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4578000f60 tx=0x7f4578007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:08.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.601+0000 7f45877fe700 1 -- 172.21.15.7:0/3092577454 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4580053b60 con 0x7f45900caa30 2023-03-03T11:56:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.839+0000 7f459536a700 1 -- 172.21.15.7:0/3092577454 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4590157ff0 con 0x7f45900caa30 2023-03-03T11:56:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.840+0000 7f45877fe700 1 -- 172.21.15.7:0/3092577454 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4580057180 con 0x7f45900caa30 2023-03-03T11:56:08.841 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:56:08.842 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:56:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 -- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45700682f0 msgr2=0x7f457006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45700682f0 0x7f457006a7a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4578000f60 tx=0x7f4578007be0 comp rx=0 tx=0).stop 2023-03-03T11:56:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 -- 172.21.15.7:0/3092577454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900caa30 msgr2=0x7f459014adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900caa30 0x7f459014adb0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f4580009b70 tx=0x7f4580004750 comp rx=0 tx=0).stop 2023-03-03T11:56:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 -- 172.21.15.7:0/3092577454 shutdown_connections 2023-03-03T11:56:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4590154ff0 0x7f45901ed3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45900c00e0 0x7f4590157ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45700682f0 0x7f457006a7a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 --2- 172.21.15.7:0/3092577454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45900caa30 0x7f459014adb0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 -- 172.21.15.7:0/3092577454 >> 172.21.15.7:0/3092577454 conn(0x7f4590066e40 msgr2=0x7f4590153340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:56:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 -- 172.21.15.7:0/3092577454 shutdown_connections 2023-03-03T11:56:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:08.842+0000 7f459536a700 1 -- 172.21.15.7:0/3092577454 wait complete. 2023-03-03T11:56:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:56:08.856 INFO:tasks.mds_thrash.fs.[a]:waiting for 29 secs before thrashing 2023-03-03T11:56:21.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:21.092 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:23.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:56:23.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.950+0000 7ff6efb69700 1 Processor -- start 2023-03-03T11:56:23.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.950+0000 7ff6efb69700 1 -- start start 2023-03-03T11:56:23.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.951+0000 7ff6efb69700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6e80aad90 0x7ff6e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.951+0000 7ff6efb69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6e806b920 0x7ff6e806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.951+0000 7ff6efb69700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e806eda0 con 0x7ff6e806bfe0 2023-03-03T11:56:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.951+0000 7ff6efb69700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e806f080 con 0x7ff6e80aad90 2023-03-03T11:56:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.951+0000 7ff6efb69700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e81584b0 con 0x7ff6e806b920 2023-03-03T11:56:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.951+0000 7ff6ed905700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6e80aad90 0x7ff6e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.951+0000 7ff6ee106700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff6e806bfe0 0x7ff6e806a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60610/0 (socket says 172.21.15.7:60610) 2023-03-03T11:56:23.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.951+0000 7ff6ed905700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6e80aad90 0x7ff6e806e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38394/0 (socket says 172.21.15.7:38394) 2023-03-03T11:56:23.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.951+0000 7ff6ee106700 1 -- 172.21.15.7:0/3422630364 learned_addr learned my addr 172.21.15.7:0/3422630364 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:56:23.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.952+0000 7ff6ed104700 1 --2- 172.21.15.7:0/3422630364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6e806b920 0x7ff6e806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:23.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.952+0000 7ff6ed905700 1 -- 172.21.15.7:0/3422630364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6e806b920 msgr2=0x7ff6e806a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:56:23.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.952+0000 7ff6ed905700 1 --2- 172.21.15.7:0/3422630364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6e806b920 0x7ff6e806a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:23.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.952+0000 7ff6ed905700 1 -- 172.21.15.7:0/3422630364 >> v1:172.21.15.7:6789/0 conn(0x7ff6e806bfe0 legacy=0x7ff6e806a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:23.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.952+0000 7ff6ed905700 1 -- 172.21.15.7:0/3422630364 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e81586a0 con 0x7ff6e80aad90 2023-03-03T11:56:23.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.952+0000 7ff6ec903700 1 -- 172.21.15.7:0/3422630364 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059081772 0 0) 0x7ff6e806eda0 con 0x7ff6e806bfe0 2023-03-03T11:56:23.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.952+0000 7ff6ed905700 1 --2- 172.21.15.7:0/3422630364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 0x7ff6e806e2f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009840 tx=0x7ff6dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d23381e11e6c7a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:23.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.952+0000 7ff6ec903700 1 -- 172.21.15.7:0/3422630364 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc011830 con 0x7ff6e80aad90 2023-03-03T11:56:23.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.952+0000 7ff6ec903700 1 -- 172.21.15.7:0/3422630364 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc011990 con 0x7ff6e80aad90 2023-03-03T11:56:23.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.952+0000 7ff6ec903700 1 -- 172.21.15.7:0/3422630364 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc011d20 con 0x7ff6e80aad90 2023-03-03T11:56:23.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.953+0000 7ff6efb69700 1 -- 172.21.15.7:0/3422630364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 msgr2=0x7ff6e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:23.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.953+0000 7ff6efb69700 1 --2- 172.21.15.7:0/3422630364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 0x7ff6e806e2f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009840 tx=0x7ff6dc009b50 comp rx=0 tx=0).stop 2023-03-03T11:56:23.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.953+0000 7ff6efb69700 1 -- 172.21.15.7:0/3422630364 shutdown_connections 2023-03-03T11:56:23.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.953+0000 7ff6efb69700 1 --2- 172.21.15.7:0/3422630364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 0x7ff6e806e2f0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.953+0000 7ff6efb69700 1 --2- 172.21.15.7:0/3422630364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6e806b920 0x7ff6e806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.953+0000 7ff6efb69700 1 -- 172.21.15.7:0/3422630364 >> 172.21.15.7:0/3422630364 conn(0x7ff6e80736f0 msgr2=0x7ff6e8073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:56:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.953+0000 7ff6efb69700 1 -- 172.21.15.7:0/3422630364 shutdown_connections 2023-03-03T11:56:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.953+0000 7ff6efb69700 1 -- 172.21.15.7:0/3422630364 wait complete. 2023-03-03T11:56:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.954+0000 7ff6efb69700 1 Processor -- start 2023-03-03T11:56:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.954+0000 7ff6efb69700 1 -- start start 2023-03-03T11:56:23.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6efb69700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6e806b920 0x7ff6e80ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6efb69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6e806bfe0 0x7ff6e81ed8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6efb69700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 0x7ff6e81ede10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6efb69700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e81ee350 con 0x7ff6e80aad90 2023-03-03T11:56:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6efb69700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e80c90d0 con 0x7ff6e806b920 2023-03-03T11:56:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6efb69700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e80adc10 con 0x7ff6e806bfe0 2023-03-03T11:56:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ee106700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 0x7ff6e81ede10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ee106700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 0x7ff6e81ede10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38404/0 (socket says 172.21.15.7:38404) 2023-03-03T11:56:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ee106700 1 -- 172.21.15.7:0/3506981049 learned_addr learned my addr 172.21.15.7:0/3506981049 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:56:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ed905700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6e806b920 0x7ff6e80ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ed104700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6e806bfe0 0x7ff6e81ed8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ee106700 1 -- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6e806bfe0 msgr2=0x7ff6e81ed8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ee106700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6e806bfe0 0x7ff6e81ed8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ee106700 1 -- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6e806b920 msgr2=0x7ff6e80ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ee106700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6e806b920 0x7ff6e80ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ee106700 1 -- 172.21.15.7:0/3506981049 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc0094a0 con 0x7ff6e80aad90 2023-03-03T11:56:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6ee106700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 0x7ff6e81ede10 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000cb30 tx=0x7ff6e000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6da7fc700 1 -- 172.21.15.7:0/3506981049 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e000ab50 con 0x7ff6e80aad90 2023-03-03T11:56:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6da7fc700 1 -- 172.21.15.7:0/3506981049 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e000acb0 con 0x7ff6e80aad90 2023-03-03T11:56:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6efb69700 1 -- 172.21.15.7:0/3506981049 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e8071cd0 con 0x7ff6e80aad90 2023-03-03T11:56:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6da7fc700 1 -- 172.21.15.7:0/3506981049 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0007310 con 0x7ff6e80aad90 2023-03-03T11:56:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.955+0000 7ff6efb69700 1 -- 172.21.15.7:0/3506981049 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e8158980 con 0x7ff6e80aad90 2023-03-03T11:56:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.956+0000 7ff6efb69700 1 -- 172.21.15.7:0/3506981049 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e8071270 con 0x7ff6e80aad90 2023-03-03T11:56:23.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.957+0000 7ff6da7fc700 1 -- 172.21.15.7:0/3506981049 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff6e001a030 con 0x7ff6e80aad90 2023-03-03T11:56:23.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.957+0000 7ff6da7fc700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6cc0682f0 0x7ff6cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:23.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.957+0000 7ff6da7fc700 1 -- 172.21.15.7:0/3506981049 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff6e0085ec0 con 0x7ff6e80aad90 2023-03-03T11:56:23.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.957+0000 7ff6ed905700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6cc0682f0 0x7ff6cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:23.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.958+0000 7ff6ed905700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6cc0682f0 0x7ff6cc06a7a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc000c00 tx=0x7ff6dc01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:23.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:23.962+0000 7ff6da7fc700 1 -- 172.21.15.7:0/3506981049 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff6e0052110 con 0x7ff6e80aad90 2023-03-03T11:56:24.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.201+0000 7ff6efb69700 1 -- 172.21.15.7:0/3506981049 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6e8074bb0 con 0x7ff6e80aad90 2023-03-03T11:56:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.201+0000 7ff6da7fc700 1 -- 172.21.15.7:0/3506981049 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff6e0018090 con 0x7ff6e80aad90 2023-03-03T11:56:24.203 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:56:24.203 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:56:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 -- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6cc0682f0 msgr2=0x7ff6cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6cc0682f0 0x7ff6cc06a7a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc000c00 tx=0x7ff6dc01c040 comp rx=0 tx=0).stop 2023-03-03T11:56:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 -- 172.21.15.7:0/3506981049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 msgr2=0x7ff6e81ede10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 0x7ff6e81ede10 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000cb30 tx=0x7ff6e000cef0 comp rx=0 tx=0).stop 2023-03-03T11:56:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 -- 172.21.15.7:0/3506981049 shutdown_connections 2023-03-03T11:56:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6cc0682f0 0x7ff6cc06a7a0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e80aad90 0x7ff6e81ede10 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6e806b920 0x7ff6e80ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 --2- 172.21.15.7:0/3506981049 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6e806bfe0 0x7ff6e81ed8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 -- 172.21.15.7:0/3506981049 >> 172.21.15.7:0/3506981049 conn(0x7ff6e80736f0 msgr2=0x7ff6e8157d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:56:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 -- 172.21.15.7:0/3506981049 shutdown_connections 2023-03-03T11:56:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:24.203+0000 7ff6efb69700 1 -- 172.21.15.7:0/3506981049 wait complete. 2023-03-03T11:56:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:56:24.217 INFO:tasks.mds_thrash.fs.[b]:waiting for 21 secs before thrashing 2023-03-03T11:56:37.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:56:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.992+0000 7f769be3b700 1 Processor -- start 2023-03-03T11:56:37.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.992+0000 7f769be3b700 1 -- start start 2023-03-03T11:56:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.993+0000 7f769be3b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f769414f9d0 0x7f7694156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:37.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.993+0000 7f769be3b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76940c2f30 con 0x7f76940abe20 2023-03-03T11:56:37.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.993+0000 7f769be3b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76940c31d0 con 0x7f769414fda0 2023-03-03T11:56:37.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.993+0000 7f769be3b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76941524a0 con 0x7f769414f9d0 2023-03-03T11:56:37.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.993+0000 7f769a3d8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f76940abe20 0x7f76941529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60628/0 (socket says 172.21.15.7:60628) 2023-03-03T11:56:37.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.993+0000 7f769a3d8700 1 -- 172.21.15.7:0/3966836688 learned_addr learned my addr 172.21.15.7:0/3966836688 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:56:37.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7699bd7700 1 --2- 172.21.15.7:0/3966836688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f769414f9d0 0x7f7694156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:37.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7698bd5700 1 -- 172.21.15.7:0/3966836688 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619804034 0 0) 0x7f76940c2f30 con 0x7f76940abe20 2023-03-03T11:56:37.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7698bd5700 1 -- 172.21.15.7:0/3966836688 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76800035b0 con 0x7f76940abe20 2023-03-03T11:56:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7698bd5700 1 -- 172.21.15.7:0/3966836688 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612913292 0 0) 0x7f76940c31d0 con 0x7f769414fda0 2023-03-03T11:56:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7698bd5700 1 -- 172.21.15.7:0/3966836688 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76940c2f30 con 0x7f769414fda0 2023-03-03T11:56:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7698bd5700 1 -- 172.21.15.7:0/3966836688 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2838844715 0 0) 0x7f76800035b0 con 0x7f76940abe20 2023-03-03T11:56:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7699bd7700 1 -- 172.21.15.7:0/3966836688 >> v1:172.21.15.143:6789/0 conn(0x7f769414fda0 legacy=0x7f7694071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7699bd7700 1 -- 172.21.15.7:0/3966836688 >> v1:172.21.15.7:6789/0 conn(0x7f76940abe20 legacy=0x7f76941529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:37.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7699bd7700 1 -- 172.21.15.7:0/3966836688 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76940c34b0 con 0x7f769414f9d0 2023-03-03T11:56:37.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7699bd7700 1 --2- 172.21.15.7:0/3966836688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f769414f9d0 0x7f7694156430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7684009870 tx=0x7f7684009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d2882f9c8bc3a3d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:38.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7698bd5700 1 -- 172.21.15.7:0/3966836688 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c0019f0 con 0x7f769414f9d0 2023-03-03T11:56:38.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7698bd5700 1 -- 172.21.15.7:0/3966836688 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7684004030 con 0x7f769414f9d0 2023-03-03T11:56:38.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.994+0000 7f7698bd5700 1 -- 172.21.15.7:0/3966836688 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76840043c0 con 0x7f769414f9d0 2023-03-03T11:56:38.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.995+0000 7f769be3b700 1 -- 172.21.15.7:0/3966836688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f769414f9d0 msgr2=0x7f7694156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:38.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.995+0000 7f769be3b700 1 --2- 172.21.15.7:0/3966836688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f769414f9d0 0x7f7694156430 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7684009870 tx=0x7f7684009b80 comp rx=0 tx=0).stop 2023-03-03T11:56:38.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.995+0000 7f769be3b700 1 -- 172.21.15.7:0/3966836688 shutdown_connections 2023-03-03T11:56:38.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.995+0000 7f769be3b700 1 --2- 172.21.15.7:0/3966836688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f769414f9d0 0x7f7694156430 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:38.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.995+0000 7f769be3b700 1 -- 172.21.15.7:0/3966836688 >> 172.21.15.7:0/3966836688 conn(0x7f76940c8900 msgr2=0x7f76940c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:56:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.995+0000 7f769be3b700 1 -- 172.21.15.7:0/3966836688 shutdown_connections 2023-03-03T11:56:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.995+0000 7f769be3b700 1 -- 172.21.15.7:0/3966836688 wait complete. 2023-03-03T11:56:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.997+0000 7f769be3b700 1 Processor -- start 2023-03-03T11:56:38.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.997+0000 7f769be3b700 1 -- start start 2023-03-03T11:56:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769be3b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76940abe20 0x7f76941eb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769be3b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f769414f9d0 0x7f76941549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769be3b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f769414fda0 0x7f769414b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769be3b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76941de4e0 con 0x7f769414fda0 2023-03-03T11:56:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769be3b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76941ebd90 con 0x7f769414f9d0 2023-03-03T11:56:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769be3b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76941dc440 con 0x7f76940abe20 2023-03-03T11:56:38.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769a3d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f769414fda0 0x7f769414b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769a3d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f769414fda0 0x7f769414b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38422/0 (socket says 172.21.15.7:38422) 2023-03-03T11:56:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769a3d8700 1 -- 172.21.15.7:0/3621679461 learned_addr learned my addr 172.21.15.7:0/3621679461 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:56:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f7699bd7700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76940abe20 0x7f76941eb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f76993d6700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f769414f9d0 0x7f76941549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769a3d8700 1 -- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76940abe20 msgr2=0x7f76941eb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769a3d8700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76940abe20 0x7f76941eb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769a3d8700 1 -- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f769414f9d0 msgr2=0x7f76941549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:38.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769a3d8700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f769414f9d0 0x7f76941549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f769a3d8700 1 -- 172.21.15.7:0/3621679461 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76840094a0 con 0x7f769414fda0 2023-03-03T11:56:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f7699bd7700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76940abe20 0x7f76941eb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:56:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.998+0000 7f76993d6700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f769414f9d0 0x7f76941549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:56:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.999+0000 7f769a3d8700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f769414fda0 0x7f769414b550 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f768c00cc00 tx=0x7f768c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.999+0000 7f768a7fc700 1 -- 172.21.15.7:0/3621679461 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c00abe0 con 0x7f769414fda0 2023-03-03T11:56:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.999+0000 7f768a7fc700 1 -- 172.21.15.7:0/3621679461 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c00ad40 con 0x7f769414fda0 2023-03-03T11:56:38.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.999+0000 7f769be3b700 1 -- 172.21.15.7:0/3621679461 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76940ac350 con 0x7f769414fda0 2023-03-03T11:56:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.999+0000 7f768a7fc700 1 -- 172.21.15.7:0/3621679461 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c007310 con 0x7f769414fda0 2023-03-03T11:56:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:37.999+0000 7f769be3b700 1 -- 172.21.15.7:0/3621679461 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76940ab630 con 0x7f769414fda0 2023-03-03T11:56:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.000+0000 7f769be3b700 1 -- 172.21.15.7:0/3621679461 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7694071fa0 con 0x7f769414fda0 2023-03-03T11:56:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.000+0000 7f768a7fc700 1 -- 172.21.15.7:0/3621679461 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f768c010070 con 0x7f769414fda0 2023-03-03T11:56:38.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.001+0000 7f768a7fc700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76780682f0 0x7f767806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:38.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.001+0000 7f768a7fc700 1 -- 172.21.15.7:0/3621679461 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f768c085c00 con 0x7f769414fda0 2023-03-03T11:56:38.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.001+0000 7f7699bd7700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76780682f0 0x7f767806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:38.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.002+0000 7f7699bd7700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76780682f0 0x7f767806a7a0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f76840097c0 tx=0x7f7684013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:38.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.005+0000 7f768a7fc700 1 -- 172.21.15.7:0/3621679461 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f768c08b020 con 0x7f769414fda0 2023-03-03T11:56:38.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.243+0000 7f769be3b700 1 -- 172.21.15.7:0/3621679461 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76940c60e0 con 0x7f769414fda0 2023-03-03T11:56:38.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.244+0000 7f768a7fc700 1 -- 172.21.15.7:0/3621679461 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f768c051f20 con 0x7f769414fda0 2023-03-03T11:56:38.247 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:56:38.247 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:56:38.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 -- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76780682f0 msgr2=0x7f767806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:38.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76780682f0 0x7f767806a7a0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f76840097c0 tx=0x7f7684013040 comp rx=0 tx=0).stop 2023-03-03T11:56:38.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 -- 172.21.15.7:0/3621679461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f769414fda0 msgr2=0x7f769414b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:38.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f769414fda0 0x7f769414b550 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f768c00cc00 tx=0x7f768c00a060 comp rx=0 tx=0).stop 2023-03-03T11:56:38.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 -- 172.21.15.7:0/3621679461 shutdown_connections 2023-03-03T11:56:38.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76780682f0 0x7f767806a7a0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:38.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f769414fda0 0x7f769414b550 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:38.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76940abe20 0x7f76941eb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:38.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 --2- 172.21.15.7:0/3621679461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f769414f9d0 0x7f76941549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:38.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 -- 172.21.15.7:0/3621679461 >> 172.21.15.7:0/3621679461 conn(0x7f76940c8900 msgr2=0x7f76941519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:56:38.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 -- 172.21.15.7:0/3621679461 shutdown_connections 2023-03-03T11:56:38.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:38.246+0000 7f769be3b700 1 -- 172.21.15.7:0/3621679461 wait complete. 2023-03-03T11:56:38.250 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:56:38.260 INFO:tasks.mds_thrash.fs.[a]:waiting for 23 secs before thrashing 2023-03-03T11:56:45.218 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:56:45.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.354+0000 7fb267743700 1 Processor -- start 2023-03-03T11:56:45.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.354+0000 7fb267743700 1 -- start start 2023-03-03T11:56:45.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.355+0000 7fb267743700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb260155390 0x7fb2600c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:45.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.355+0000 7fb267743700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb260151770 0x7fb260151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:45.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.355+0000 7fb267743700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb260154910 con 0x7fb26006e020 2023-03-03T11:56:45.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.355+0000 7fb267743700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb260070b90 con 0x7fb260151770 2023-03-03T11:56:45.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.355+0000 7fb267743700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb260151320 con 0x7fb260155390 2023-03-03T11:56:45.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.355+0000 7fb265ce0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb260151770 0x7fb260151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:45.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.355+0000 7fb265ce0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb260151770 0x7fb260151bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38430/0 (socket says 172.21.15.7:38430) 2023-03-03T11:56:45.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.355+0000 7fb265ce0700 1 -- 172.21.15.7:0/3981564832 learned_addr learned my addr 172.21.15.7:0/3981564832 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:56:45.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.355+0000 7fb2654df700 1 --2- 172.21.15.7:0/3981564832 >> v2:172.21.15.143:3300/0 conn(0x7fb260155390 0x7fb2600c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:45.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb265ce0700 1 -- 172.21.15.7:0/3981564832 >> v2:172.21.15.143:3300/0 conn(0x7fb260155390 msgr2=0x7fb2600c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:56:45.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb265ce0700 1 --2- 172.21.15.7:0/3981564832 >> v2:172.21.15.143:3300/0 conn(0x7fb260155390 0x7fb2600c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:45.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb265ce0700 1 -- 172.21.15.7:0/3981564832 >> v1:172.21.15.143:6789/0 conn(0x7fb26006e020 legacy=0x7fb260071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:45.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb265ce0700 1 -- 172.21.15.7:0/3981564832 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2600702a0 con 0x7fb260151770 2023-03-03T11:56:45.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb2654df700 1 --2- 172.21.15.7:0/3981564832 >> v2:172.21.15.143:3300/0 conn(0x7fb260155390 0x7fb2600c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:56:45.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb265ce0700 1 --2- 172.21.15.7:0/3981564832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb260151770 0x7fb260151bd0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fb26006d700 tx=0x7fb25800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d6c52bf6fc43d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:45.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb257fff700 1 -- 172.21.15.7:0/3981564832 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258010070 con 0x7fb260151770 2023-03-03T11:56:45.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb257fff700 1 -- 172.21.15.7:0/3981564832 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25800dd90 con 0x7fb260151770 2023-03-03T11:56:45.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb257fff700 1 -- 172.21.15.7:0/3981564832 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25800b310 con 0x7fb260151770 2023-03-03T11:56:45.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb267743700 1 -- 172.21.15.7:0/3981564832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb260151770 msgr2=0x7fb260151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:45.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb267743700 1 --2- 172.21.15.7:0/3981564832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb260151770 0x7fb260151bd0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fb26006d700 tx=0x7fb25800d430 comp rx=0 tx=0).stop 2023-03-03T11:56:45.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb267743700 1 -- 172.21.15.7:0/3981564832 shutdown_connections 2023-03-03T11:56:45.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb267743700 1 --2- 172.21.15.7:0/3981564832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb260151770 0x7fb260151bd0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:45.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.356+0000 7fb267743700 1 --2- 172.21.15.7:0/3981564832 >> v2:172.21.15.143:3300/0 conn(0x7fb260155390 0x7fb2600c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:45.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.357+0000 7fb267743700 1 -- 172.21.15.7:0/3981564832 >> 172.21.15.7:0/3981564832 conn(0x7fb26006c110 msgr2=0x7fb26006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:56:45.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.357+0000 7fb267743700 1 -- 172.21.15.7:0/3981564832 shutdown_connections 2023-03-03T11:56:45.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.357+0000 7fb267743700 1 -- 172.21.15.7:0/3981564832 wait complete. 2023-03-03T11:56:45.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.358+0000 7fb267743700 1 Processor -- start 2023-03-03T11:56:45.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.358+0000 7fb267743700 1 -- start start 2023-03-03T11:56:45.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.358+0000 7fb267743700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb26006e020 0x7fb2600ac070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:45.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.358+0000 7fb267743700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb260151770 0x7fb26014b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:45.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.358+0000 7fb267743700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb260155390 0x7fb2600721e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:45.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.358+0000 7fb267743700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600ac700 con 0x7fb26006e020 2023-03-03T11:56:45.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.358+0000 7fb267743700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600ab240 con 0x7fb260151770 2023-03-03T11:56:45.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.358+0000 7fb267743700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600ab520 con 0x7fb260155390 2023-03-03T11:56:45.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb2654df700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb26006e020 0x7fb2600ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:45.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb2654df700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb26006e020 0x7fb2600ac070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38432/0 (socket says 172.21.15.7:38432) 2023-03-03T11:56:45.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb2654df700 1 -- 172.21.15.7:0/3509177676 learned_addr learned my addr 172.21.15.7:0/3509177676 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:56:45.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb264cde700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb260151770 0x7fb26014b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:45.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb265ce0700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb260155390 0x7fb2600721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:45.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb2654df700 1 -- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb260155390 msgr2=0x7fb2600721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:45.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb2654df700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb260155390 0x7fb2600721e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:45.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb2654df700 1 -- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb260151770 msgr2=0x7fb26014b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:45.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb2654df700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb260151770 0x7fb26014b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:45.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb2654df700 1 -- 172.21.15.7:0/3509177676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25800bd40 con 0x7fb26006e020 2023-03-03T11:56:45.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb264cde700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb260151770 0x7fb26014b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:56:45.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb265ce0700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb260155390 0x7fb2600721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:56:45.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb2654df700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb26006e020 0x7fb2600ac070 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb25000e810 tx=0x7fb25000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:45.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb255ffb700 1 -- 172.21.15.7:0/3509177676 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25000c020 con 0x7fb26006e020 2023-03-03T11:56:45.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb255ffb700 1 -- 172.21.15.7:0/3509177676 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb250004250 con 0x7fb26006e020 2023-03-03T11:56:45.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb267743700 1 -- 172.21.15.7:0/3509177676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb260070820 con 0x7fb26006e020 2023-03-03T11:56:45.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.359+0000 7fb255ffb700 1 -- 172.21.15.7:0/3509177676 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2500036a0 con 0x7fb26006e020 2023-03-03T11:56:45.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.360+0000 7fb267743700 1 -- 172.21.15.7:0/3509177676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb260070540 con 0x7fb26006e020 2023-03-03T11:56:45.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.361+0000 7fb267743700 1 -- 172.21.15.7:0/3509177676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb260155760 con 0x7fb26006e020 2023-03-03T11:56:45.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.361+0000 7fb255ffb700 1 -- 172.21.15.7:0/3509177676 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb250022070 con 0x7fb26006e020 2023-03-03T11:56:45.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.361+0000 7fb255ffb700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2440682f0 0x7fb24406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:56:45.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.361+0000 7fb255ffb700 1 -- 172.21.15.7:0/3509177676 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb250085ee0 con 0x7fb26006e020 2023-03-03T11:56:45.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.361+0000 7fb264cde700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2440682f0 0x7fb24406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:56:45.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.362+0000 7fb264cde700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2440682f0 0x7fb24406a7a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb2601548b0 tx=0x7fb25c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:56:45.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.365+0000 7fb255ffb700 1 -- 172.21.15.7:0/3509177676 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb250052280 con 0x7fb26006e020 2023-03-03T11:56:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.601+0000 7fb267743700 1 -- 172.21.15.7:0/3509177676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb260074bb0 con 0x7fb26006e020 2023-03-03T11:56:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.602+0000 7fb255ffb700 1 -- 172.21.15.7:0/3509177676 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb25000f040 con 0x7fb26006e020 2023-03-03T11:56:45.604 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:56:45.605 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:56:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 -- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2440682f0 msgr2=0x7fb24406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2440682f0 0x7fb24406a7a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fb2601548b0 tx=0x7fb25c009040 comp rx=0 tx=0).stop 2023-03-03T11:56:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 -- 172.21.15.7:0/3509177676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb26006e020 msgr2=0x7fb2600ac070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:56:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb26006e020 0x7fb2600ac070 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb25000e810 tx=0x7fb25000ebd0 comp rx=0 tx=0).stop 2023-03-03T11:56:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 -- 172.21.15.7:0/3509177676 shutdown_connections 2023-03-03T11:56:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb260155390 0x7fb2600721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2440682f0 0x7fb24406a7a0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb26006e020 0x7fb2600ac070 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 --2- 172.21.15.7:0/3509177676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb260151770 0x7fb26014b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:56:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 -- 172.21.15.7:0/3509177676 >> 172.21.15.7:0/3509177676 conn(0x7fb26006c110 msgr2=0x7fb26006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:56:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 -- 172.21.15.7:0/3509177676 shutdown_connections 2023-03-03T11:56:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:56:45.604+0000 7fb267743700 1 -- 172.21.15.7:0/3509177676 wait complete. 2023-03-03T11:56:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:56:45.618 INFO:tasks.mds_thrash.fs.[b]:waiting for 63 secs before thrashing 2023-03-03T11:56:51.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:56:51.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:01.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:57:01.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.398+0000 7f136ae9a700 1 Processor -- start 2023-03-03T11:57:01.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.399+0000 7f136ae9a700 1 -- start start 2023-03-03T11:57:01.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.399+0000 7f136ae9a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f136406bea0 0x7f136406c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:01.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.399+0000 7f136ae9a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364157dc0 0x7f1364158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:01.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.399+0000 7f136ae9a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136406d780 con 0x7f1364151ab0 2023-03-03T11:57:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.399+0000 7f136ae9a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136406c830 con 0x7f136406bea0 2023-03-03T11:57:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.399+0000 7f136ae9a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136406cb70 con 0x7f1364157dc0 2023-03-03T11:57:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.399+0000 7f1368c36700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1364151ab0 0x7f13640736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60658/0 (socket says 172.21.15.7:60658) 2023-03-03T11:57:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.399+0000 7f1368c36700 1 -- 172.21.15.7:0/216768735 learned_addr learned my addr 172.21.15.7:0/216768735 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:57:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f1369437700 1 --2- 172.21.15.7:0/216768735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364157dc0 0x7f1364158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f1363fff700 1 --2- 172.21.15.7:0/216768735 >> v2:172.21.15.143:3300/0 conn(0x7f136406bea0 0x7f136406c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f13637fe700 1 -- 172.21.15.7:0/216768735 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886115837 0 0) 0x7f136406d780 con 0x7f1364151ab0 2023-03-03T11:57:01.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f13637fe700 1 -- 172.21.15.7:0/216768735 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13540035b0 con 0x7f1364151ab0 2023-03-03T11:57:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f13637fe700 1 -- 172.21.15.7:0/216768735 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3178184239 0 0) 0x7f13540035b0 con 0x7f1364151ab0 2023-03-03T11:57:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f1363fff700 1 -- 172.21.15.7:0/216768735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364157dc0 msgr2=0x7f1364158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:57:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f1363fff700 1 --2- 172.21.15.7:0/216768735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364157dc0 0x7f1364158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f1363fff700 1 -- 172.21.15.7:0/216768735 >> v1:172.21.15.7:6789/0 conn(0x7f1364151ab0 legacy=0x7f13640736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f1363fff700 1 -- 172.21.15.7:0/216768735 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136406ce50 con 0x7f136406bea0 2023-03-03T11:57:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f1369437700 1 --2- 172.21.15.7:0/216768735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364157dc0 0x7f1364158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T11:57:01.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f1363fff700 1 --2- 172.21.15.7:0/216768735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f136406bea0 0x7f136406c2f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1350009840 tx=0x7f1350009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cdadfc757c92c03 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:57:01.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.400+0000 7f13637fe700 1 -- 172.21.15.7:0/216768735 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358001670 con 0x7f136406bea0 2023-03-03T11:57:01.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.401+0000 7f13637fe700 1 -- 172.21.15.7:0/216768735 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1350004030 con 0x7f136406bea0 2023-03-03T11:57:01.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.401+0000 7f13637fe700 1 -- 172.21.15.7:0/216768735 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500043a0 con 0x7f136406bea0 2023-03-03T11:57:01.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.401+0000 7f136ae9a700 1 -- 172.21.15.7:0/216768735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f136406bea0 msgr2=0x7f136406c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:01.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.401+0000 7f136ae9a700 1 --2- 172.21.15.7:0/216768735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f136406bea0 0x7f136406c2f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1350009840 tx=0x7f1350009b50 comp rx=0 tx=0).stop 2023-03-03T11:57:01.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.401+0000 7f136ae9a700 1 -- 172.21.15.7:0/216768735 shutdown_connections 2023-03-03T11:57:01.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.401+0000 7f136ae9a700 1 --2- 172.21.15.7:0/216768735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364157dc0 0x7f1364158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:01.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.401+0000 7f136ae9a700 1 --2- 172.21.15.7:0/216768735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f136406bea0 0x7f136406c2f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:01.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.401+0000 7f136ae9a700 1 -- 172.21.15.7:0/216768735 >> 172.21.15.7:0/216768735 conn(0x7f13640a94d0 msgr2=0x7f13640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:57:01.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.401+0000 7f136ae9a700 1 -- 172.21.15.7:0/216768735 shutdown_connections 2023-03-03T11:57:01.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.401+0000 7f136ae9a700 1 -- 172.21.15.7:0/216768735 wait complete. 2023-03-03T11:57:01.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.402+0000 7f136ae9a700 1 Processor -- start 2023-03-03T11:57:01.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.402+0000 7f136ae9a700 1 -- start start 2023-03-03T11:57:01.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f136ae9a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f136406bea0 0x7f13640c1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:01.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f136ae9a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364151ab0 0x7f13641edc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:01.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f136ae9a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1364157dc0 0x7f136406a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:01.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f136ae9a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13640aa360 con 0x7f136406bea0 2023-03-03T11:57:01.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f136ae9a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13641ed7d0 con 0x7f1364157dc0 2023-03-03T11:57:01.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f136ae9a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13640bed60 con 0x7f1364151ab0 2023-03-03T11:57:01.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f1368c36700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f136406bea0 0x7f13640c1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:01.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f1368c36700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f136406bea0 0x7f13640c1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38452/0 (socket says 172.21.15.7:38452) 2023-03-03T11:57:01.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f1368c36700 1 -- 172.21.15.7:0/61451508 learned_addr learned my addr 172.21.15.7:0/61451508 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:57:01.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f1363fff700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364151ab0 0x7f13641edc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:01.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.403+0000 7f1369437700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1364157dc0 0x7f136406a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:01.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f1368c36700 1 -- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364151ab0 msgr2=0x7f13641edc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:01.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f1368c36700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364151ab0 0x7f13641edc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:01.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f1368c36700 1 -- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1364157dc0 msgr2=0x7f136406a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:01.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f1368c36700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1364157dc0 0x7f136406a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:01.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f1368c36700 1 -- 172.21.15.7:0/61451508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c009a70 con 0x7f136406bea0 2023-03-03T11:57:01.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f1368c36700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f136406bea0 0x7f13640c1130 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f13580011e0 tx=0x7f135800ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:57:01.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f13617fa700 1 -- 172.21.15.7:0/61451508 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358004c80 con 0x7f136406bea0 2023-03-03T11:57:01.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f13617fa700 1 -- 172.21.15.7:0/61451508 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1358004de0 con 0x7f136406bea0 2023-03-03T11:57:01.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f136ae9a700 1 -- 172.21.15.7:0/61451508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13500094a0 con 0x7f136406bea0 2023-03-03T11:57:01.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f13617fa700 1 -- 172.21.15.7:0/61451508 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13580072f0 con 0x7f136406bea0 2023-03-03T11:57:01.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.404+0000 7f136ae9a700 1 -- 172.21.15.7:0/61451508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136406d410 con 0x7f136406bea0 2023-03-03T11:57:01.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.405+0000 7f136ae9a700 1 -- 172.21.15.7:0/61451508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136406d130 con 0x7f136406bea0 2023-03-03T11:57:01.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.405+0000 7f13617fa700 1 -- 172.21.15.7:0/61451508 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f135801a020 con 0x7f136406bea0 2023-03-03T11:57:01.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.406+0000 7f13617fa700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13480682f0 0x7f134806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:01.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.406+0000 7f13617fa700 1 -- 172.21.15.7:0/61451508 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1358085d50 con 0x7f136406bea0 2023-03-03T11:57:01.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.406+0000 7f1363fff700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13480682f0 0x7f134806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:01.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.407+0000 7f1363fff700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13480682f0 0x7f134806a7a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1350000c00 tx=0x7f1350008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:57:01.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.410+0000 7f13617fa700 1 -- 172.21.15.7:0/61451508 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13580520f0 con 0x7f136406bea0 2023-03-03T11:57:01.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.650+0000 7f136ae9a700 1 -- 172.21.15.7:0/61451508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13641edab0 con 0x7f136406bea0 2023-03-03T11:57:01.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.651+0000 7f13617fa700 1 -- 172.21.15.7:0/61451508 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1358055710 con 0x7f136406bea0 2023-03-03T11:57:01.652 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:57:01.652 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:57:01.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 -- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13480682f0 msgr2=0x7f134806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13480682f0 0x7f134806a7a0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f1350000c00 tx=0x7f1350008030 comp rx=0 tx=0).stop 2023-03-03T11:57:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 -- 172.21.15.7:0/61451508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f136406bea0 msgr2=0x7f13640c1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f136406bea0 0x7f13640c1130 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f13580011e0 tx=0x7f135800ca50 comp rx=0 tx=0).stop 2023-03-03T11:57:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 -- 172.21.15.7:0/61451508 shutdown_connections 2023-03-03T11:57:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1364157dc0 0x7f136406a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13480682f0 0x7f134806a7a0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f136406bea0 0x7f13640c1130 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 --2- 172.21.15.7:0/61451508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1364151ab0 0x7f13641edc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 -- 172.21.15.7:0/61451508 >> 172.21.15.7:0/61451508 conn(0x7f13640a94d0 msgr2=0x7f13641559e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:57:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 -- 172.21.15.7:0/61451508 shutdown_connections 2023-03-03T11:57:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:01.653+0000 7f136ae9a700 1 -- 172.21.15.7:0/61451508 wait complete. 2023-03-03T11:57:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:57:01.666 INFO:tasks.mds_thrash.fs.[a]:waiting for 57 secs before thrashing 2023-03-03T11:57:21.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:21.301 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:48.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:57:48.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.755+0000 7f9bcd570700 1 Processor -- start 2023-03-03T11:57:48.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.755+0000 7f9bcd570700 1 -- start start 2023-03-03T11:57:48.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.756+0000 7f9bcd570700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9bc80c7e00 0x7f9bc8154a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:48.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.756+0000 7f9bcd570700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9bc806b6f0 0x7f9bc8154f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:48.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.756+0000 7f9bcd570700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc8157bb0 con 0x7f9bc80c7a30 2023-03-03T11:57:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.756+0000 7f9bcd570700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc806ab00 con 0x7f9bc806b6f0 2023-03-03T11:57:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.756+0000 7f9bcd570700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc81545c0 con 0x7f9bc80c7e00 2023-03-03T11:57:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.756+0000 7f9bc77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9bc806b6f0 0x7f9bc8154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9bc806b6f0 0x7f9bc8154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38504/0 (socket says 172.21.15.7:38504) 2023-03-03T11:57:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc77fe700 1 -- 172.21.15.7:0/4158024604 learned_addr learned my addr 172.21.15.7:0/4158024604 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:57:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc67fc700 1 --2- 172.21.15.7:0/4158024604 >> v2:172.21.15.143:3300/0 conn(0x7f9bc80c7e00 0x7f9bc8154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc77fe700 1 -- 172.21.15.7:0/4158024604 >> v2:172.21.15.143:3300/0 conn(0x7f9bc80c7e00 msgr2=0x7f9bc8154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:57:48.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc77fe700 1 --2- 172.21.15.7:0/4158024604 >> v2:172.21.15.143:3300/0 conn(0x7f9bc80c7e00 0x7f9bc8154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:48.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc77fe700 1 -- 172.21.15.7:0/4158024604 >> v1:172.21.15.143:6789/0 conn(0x7f9bc80c7a30 legacy=0x7f9bc814e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:48.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc77fe700 1 -- 172.21.15.7:0/4158024604 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc806a2f0 con 0x7f9bc806b6f0 2023-03-03T11:57:48.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc77fe700 1 --2- 172.21.15.7:0/4158024604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc806b6f0 0x7f9bc8154f50 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f9bc808c770 tx=0x7f9bc000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ecce96ce30353 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:57:48.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc5ffb700 1 -- 172.21.15.7:0/4158024604 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc000e070 con 0x7f9bc806b6f0 2023-03-03T11:57:48.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc5ffb700 1 -- 172.21.15.7:0/4158024604 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc000b9c0 con 0x7f9bc806b6f0 2023-03-03T11:57:48.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.757+0000 7f9bc5ffb700 1 -- 172.21.15.7:0/4158024604 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc000bd70 con 0x7f9bc806b6f0 2023-03-03T11:57:48.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.758+0000 7f9bcd570700 1 -- 172.21.15.7:0/4158024604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc806b6f0 msgr2=0x7f9bc8154f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:48.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.758+0000 7f9bcd570700 1 --2- 172.21.15.7:0/4158024604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc806b6f0 0x7f9bc8154f50 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f9bc808c770 tx=0x7f9bc000b330 comp rx=0 tx=0).stop 2023-03-03T11:57:48.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.758+0000 7f9bcd570700 1 -- 172.21.15.7:0/4158024604 shutdown_connections 2023-03-03T11:57:48.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.758+0000 7f9bcd570700 1 --2- 172.21.15.7:0/4158024604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc806b6f0 0x7f9bc8154f50 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.758+0000 7f9bcd570700 1 --2- 172.21.15.7:0/4158024604 >> v2:172.21.15.143:3300/0 conn(0x7f9bc80c7e00 0x7f9bc8154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.758+0000 7f9bcd570700 1 -- 172.21.15.7:0/4158024604 >> 172.21.15.7:0/4158024604 conn(0x7f9bc8073d20 msgr2=0x7f9bc80716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:57:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.758+0000 7f9bcd570700 1 -- 172.21.15.7:0/4158024604 shutdown_connections 2023-03-03T11:57:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.758+0000 7f9bcd570700 1 -- 172.21.15.7:0/4158024604 wait complete. 2023-03-03T11:57:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.759+0000 7f9bcd570700 1 Processor -- start 2023-03-03T11:57:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.759+0000 7f9bcd570700 1 -- start start 2023-03-03T11:57:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bcd570700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bc80c7a30 0x7f9bc81ed120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:48.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bcd570700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bc80c7e00 0x7f9bc80c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bcd570700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc81574f0 0x7f9bc80c3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bcd570700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81de050 con 0x7f9bc81574f0 2023-03-03T11:57:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bcd570700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81ed960 con 0x7f9bc80c7e00 2023-03-03T11:57:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bcd570700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81edc40 con 0x7f9bc80c7a30 2023-03-03T11:57:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc81574f0 0x7f9bc80c3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc81574f0 0x7f9bc80c3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38510/0 (socket says 172.21.15.7:38510) 2023-03-03T11:57:48.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc77fe700 1 -- 172.21.15.7:0/2863625386 learned_addr learned my addr 172.21.15.7:0/2863625386 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:57:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc67fc700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bc80c7e00 0x7f9bc80c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc6ffd700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bc80c7a30 0x7f9bc81ed120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc77fe700 1 -- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bc80c7a30 msgr2=0x7f9bc81ed120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc77fe700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bc80c7a30 0x7f9bc81ed120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc77fe700 1 -- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bc80c7e00 msgr2=0x7f9bc80c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:48.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc77fe700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bc80c7e00 0x7f9bc80c2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:48.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc77fe700 1 -- 172.21.15.7:0/2863625386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc0009b30 con 0x7f9bc81574f0 2023-03-03T11:57:48.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bc77fe700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc81574f0 0x7f9bc80c3370 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0012000 tx=0x7f9bc0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:57:48.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.760+0000 7f9bbf7fe700 1 -- 172.21.15.7:0/2863625386 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc00099b0 con 0x7f9bc81574f0 2023-03-03T11:57:48.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.761+0000 7f9bbf7fe700 1 -- 172.21.15.7:0/2863625386 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc00092e0 con 0x7f9bc81574f0 2023-03-03T11:57:48.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.761+0000 7f9bcd570700 1 -- 172.21.15.7:0/2863625386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc806a540 con 0x7f9bc81574f0 2023-03-03T11:57:48.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.761+0000 7f9bbf7fe700 1 -- 172.21.15.7:0/2863625386 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc001c640 con 0x7f9bc81574f0 2023-03-03T11:57:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.761+0000 7f9bcd570700 1 -- 172.21.15.7:0/2863625386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc80bfbb0 con 0x7f9bc81574f0 2023-03-03T11:57:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.762+0000 7f9bcd570700 1 -- 172.21.15.7:0/2863625386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc806a820 con 0x7f9bc81574f0 2023-03-03T11:57:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.762+0000 7f9bbf7fe700 1 -- 172.21.15.7:0/2863625386 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9bc0022030 con 0x7f9bc81574f0 2023-03-03T11:57:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.762+0000 7f9bbf7fe700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ba80682f0 0x7f9ba806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.763+0000 7f9bbf7fe700 1 -- 172.21.15.7:0/2863625386 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9bc0087a20 con 0x7f9bc81574f0 2023-03-03T11:57:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.763+0000 7f9bc6ffd700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ba80682f0 0x7f9ba806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.763+0000 7f9bc6ffd700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ba80682f0 0x7f9ba806a7a0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8000fb0 tx=0x7f9bb8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:57:48.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:48.767+0000 7f9bbf7fe700 1 -- 172.21.15.7:0/2863625386 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9bc0053dc0 con 0x7f9bc81574f0 2023-03-03T11:57:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.007+0000 7f9bcd570700 1 -- 172.21.15.7:0/2863625386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bc80c2b50 con 0x7f9bc81574f0 2023-03-03T11:57:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.008+0000 7f9bbf7fe700 1 -- 172.21.15.7:0/2863625386 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9bc80c2b50 con 0x7f9bc81574f0 2023-03-03T11:57:49.013 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:57:49.013 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:57:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 -- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ba80682f0 msgr2=0x7f9ba806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ba80682f0 0x7f9ba806a7a0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8000fb0 tx=0x7f9bb8009040 comp rx=0 tx=0).stop 2023-03-03T11:57:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 -- 172.21.15.7:0/2863625386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc81574f0 msgr2=0x7f9bc80c3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc81574f0 0x7f9bc80c3370 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0012000 tx=0x7f9bc0004ad0 comp rx=0 tx=0).stop 2023-03-03T11:57:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 -- 172.21.15.7:0/2863625386 shutdown_connections 2023-03-03T11:57:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ba80682f0 0x7f9ba806a7a0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bc81574f0 0x7f9bc80c3370 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bc80c7a30 0x7f9bc81ed120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 --2- 172.21.15.7:0/2863625386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bc80c7e00 0x7f9bc80c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 -- 172.21.15.7:0/2863625386 >> 172.21.15.7:0/2863625386 conn(0x7f9bc806b6f0 msgr2=0x7f9bc8153e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:57:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 -- 172.21.15.7:0/2863625386 shutdown_connections 2023-03-03T11:57:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:49.010+0000 7f9bcd570700 1 -- 172.21.15.7:0/2863625386 wait complete. 2023-03-03T11:57:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:57:49.025 INFO:tasks.mds_thrash.fs.[b]:waiting for 98 secs before thrashing 2023-03-03T11:57:51.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:51.449 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:57:58.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:57:58.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.804+0000 7f3fe7559700 1 Processor -- start 2023-03-03T11:57:58.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.805+0000 7f3fe7559700 1 -- start start 2023-03-03T11:57:58.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.805+0000 7f3fe7559700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3fe0151d40 0x7f3fe00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:58.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.805+0000 7f3fe7559700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe006d430 con 0x7f3fe0073c30 2023-03-03T11:57:58.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.805+0000 7f3fe7559700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe006cb00 con 0x7f3fe00c2930 2023-03-03T11:57:58.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.805+0000 7f3fe7559700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe006c2a0 con 0x7f3fe0151d40 2023-03-03T11:57:58.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fe52f5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3fe0151d40 0x7f3fe00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:58.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fe5af6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3fe0073c30 0x7f3fe01581a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60734/0 (socket says 172.21.15.7:60734) 2023-03-03T11:57:58.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fe5af6700 1 -- 172.21.15.7:0/3060095990 learned_addr learned my addr 172.21.15.7:0/3060095990 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:57:58.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fe52f5700 1 -- 172.21.15.7:0/3060095990 >> v1:172.21.15.143:6789/0 conn(0x7f3fe00c2930 legacy=0x7f3fe0157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:57:58.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fe52f5700 1 -- 172.21.15.7:0/3060095990 >> v1:172.21.15.7:6789/0 conn(0x7f3fe0073c30 legacy=0x7f3fe01581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:58.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fd3fff700 1 -- 172.21.15.7:0/3060095990 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888656911 0 0) 0x7f3fe006d430 con 0x7f3fe0073c30 2023-03-03T11:57:58.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fe52f5700 1 -- 172.21.15.7:0/3060095990 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe006cde0 con 0x7f3fe0151d40 2023-03-03T11:57:58.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fe52f5700 1 --2- 172.21.15.7:0/3060095990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0151d40 0x7f3fe00736f0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009870 tx=0x7f3fdc009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbcf34d651f1d757 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:57:58.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fd3fff700 1 -- 172.21.15.7:0/3060095990 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc004230 con 0x7f3fe0151d40 2023-03-03T11:57:58.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fd3fff700 1 -- 172.21.15.7:0/3060095990 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fdc004390 con 0x7f3fe0151d40 2023-03-03T11:57:58.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.806+0000 7f3fd3fff700 1 -- 172.21.15.7:0/3060095990 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc0036a0 con 0x7f3fe0151d40 2023-03-03T11:57:58.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.807+0000 7f3fe7559700 1 -- 172.21.15.7:0/3060095990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0151d40 msgr2=0x7f3fe00736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.807+0000 7f3fe7559700 1 --2- 172.21.15.7:0/3060095990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0151d40 0x7f3fe00736f0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009870 tx=0x7f3fdc009b80 comp rx=0 tx=0).stop 2023-03-03T11:57:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.807+0000 7f3fe7559700 1 -- 172.21.15.7:0/3060095990 shutdown_connections 2023-03-03T11:57:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.807+0000 7f3fe7559700 1 --2- 172.21.15.7:0/3060095990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0151d40 0x7f3fe00736f0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.807+0000 7f3fe7559700 1 -- 172.21.15.7:0/3060095990 >> 172.21.15.7:0/3060095990 conn(0x7f3fe00a94d0 msgr2=0x7f3fe00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:57:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.807+0000 7f3fe7559700 1 -- 172.21.15.7:0/3060095990 shutdown_connections 2023-03-03T11:57:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.807+0000 7f3fe7559700 1 -- 172.21.15.7:0/3060095990 wait complete. 2023-03-03T11:57:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.808+0000 7f3fe7559700 1 Processor -- start 2023-03-03T11:57:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.808+0000 7f3fe7559700 1 -- start start 2023-03-03T11:57:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe7559700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0073c30 0x7f3fe014e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe7559700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00c2930 0x7f3fe01ea720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe7559700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe0151d40 0x7f3fe01de410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe7559700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00beba0 con 0x7f3fe0073c30 2023-03-03T11:57:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe7559700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe01e9ff0 con 0x7f3fe0151d40 2023-03-03T11:57:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe7559700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00bf8a0 con 0x7f3fe00c2930 2023-03-03T11:57:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe52f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0073c30 0x7f3fe014e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe52f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0073c30 0x7f3fe014e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38524/0 (socket says 172.21.15.7:38524) 2023-03-03T11:57:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe52f5700 1 -- 172.21.15.7:0/2587424054 learned_addr learned my addr 172.21.15.7:0/2587424054 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:57:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe5af6700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe0151d40 0x7f3fe01de410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe4af4700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00c2930 0x7f3fe01ea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe52f5700 1 -- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00c2930 msgr2=0x7f3fe01ea720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe52f5700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00c2930 0x7f3fe01ea720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:58.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe52f5700 1 -- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe0151d40 msgr2=0x7f3fe01de410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:58.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe52f5700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe0151d40 0x7f3fe01de410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:58.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe52f5700 1 -- 172.21.15.7:0/2587424054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc0094a0 con 0x7f3fe0073c30 2023-03-03T11:57:58.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe5af6700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe0151d40 0x7f3fe01de410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T11:57:58.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.809+0000 7f3fe52f5700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0073c30 0x7f3fe014e050 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0090b0 tx=0x7f3fdc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:57:58.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.810+0000 7f3fd1ffb700 1 -- 172.21.15.7:0/2587424054 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc011830 con 0x7f3fe0073c30 2023-03-03T11:57:58.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.810+0000 7f3fd1ffb700 1 -- 172.21.15.7:0/2587424054 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fdc011990 con 0x7f3fe0073c30 2023-03-03T11:57:58.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.810+0000 7f3fe7559700 1 -- 172.21.15.7:0/2587424054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe006bea0 con 0x7f3fe0073c30 2023-03-03T11:57:58.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.810+0000 7f3fd1ffb700 1 -- 172.21.15.7:0/2587424054 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc011d40 con 0x7f3fe0073c30 2023-03-03T11:57:58.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.810+0000 7f3fe7559700 1 -- 172.21.15.7:0/2587424054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe006d0c0 con 0x7f3fe0073c30 2023-03-03T11:57:58.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.811+0000 7f3fe7559700 1 -- 172.21.15.7:0/2587424054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe006c030 con 0x7f3fe0073c30 2023-03-03T11:57:58.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.811+0000 7f3fd1ffb700 1 -- 172.21.15.7:0/2587424054 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3fdc017020 con 0x7f3fe0073c30 2023-03-03T11:57:58.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.812+0000 7f3fd1ffb700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 0x7f3fc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:57:58.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.812+0000 7f3fd1ffb700 1 -- 172.21.15.7:0/2587424054 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3fdc087820 con 0x7f3fe0073c30 2023-03-03T11:57:58.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.812+0000 7f3fe4af4700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 0x7f3fc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:57:58.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.813+0000 7f3fe4af4700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 0x7f3fc406a7a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0145750 tx=0x7f3fd4007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:57:58.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:58.816+0000 7f3fd1ffb700 1 -- 172.21.15.7:0/2587424054 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3fdc053bc0 con 0x7f3fe0073c30 2023-03-03T11:57:59.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.059+0000 7f3fe7559700 1 -- 172.21.15.7:0/2587424054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fe0074bb0 con 0x7f3fe0073c30 2023-03-03T11:57:59.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.060+0000 7f3fd1ffb700 1 -- 172.21.15.7:0/2587424054 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3fdc0571e0 con 0x7f3fe0073c30 2023-03-03T11:57:59.062 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:57:59.062 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:57:59.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 -- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 msgr2=0x7f3fc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:59.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 0x7f3fc406a7a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0145750 tx=0x7f3fd4007ec0 comp rx=0 tx=0).stop 2023-03-03T11:57:59.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 -- 172.21.15.7:0/2587424054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0073c30 msgr2=0x7f3fe014e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:57:59.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0073c30 0x7f3fe014e050 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0090b0 tx=0x7f3fdc0047c0 comp rx=0 tx=0).stop 2023-03-03T11:57:59.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 -- 172.21.15.7:0/2587424054 shutdown_connections 2023-03-03T11:57:59.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe0151d40 0x7f3fe01de410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:59.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fc40682f0 0x7f3fc406a7a0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:59.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0073c30 0x7f3fe014e050 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:59.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 --2- 172.21.15.7:0/2587424054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe00c2930 0x7f3fe01ea720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:57:59.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 -- 172.21.15.7:0/2587424054 >> 172.21.15.7:0/2587424054 conn(0x7f3fe00a94d0 msgr2=0x7f3fe00c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:57:59.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 -- 172.21.15.7:0/2587424054 shutdown_connections 2023-03-03T11:57:59.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:57:59.062+0000 7f3fe7559700 1 -- 172.21.15.7:0/2587424054 wait complete. 2023-03-03T11:57:59.066 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:57:59.075 INFO:tasks.mds_thrash.fs.[a]:waiting for 119 secs before thrashing 2023-03-03T11:58:21.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:21.512 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:51.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:58:51.598 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:21.752 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:21.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:27.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:59:27.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.162+0000 7f5e4417b700 1 Processor -- start 2023-03-03T11:59:27.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.162+0000 7f5e4417b700 1 -- start start 2023-03-03T11:59:27.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.163+0000 7f5e4417b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5e3c151d40 0x7f5e3c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:27.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.163+0000 7f5e4417b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5e3c073c30 0x7f5e3c1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:27.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.163+0000 7f5e4417b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c06d430 con 0x7f5e3c0c2930 2023-03-03T11:59:27.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.163+0000 7f5e4417b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c06c2a0 con 0x7f5e3c151d40 2023-03-03T11:59:27.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.163+0000 7f5e4417b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c1586e0 con 0x7f5e3c073c30 2023-03-03T11:59:27.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.163+0000 7f5e41f17700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5e3c151d40 0x7f5e3c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:27.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.163+0000 7f5e41716700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5e3c0c2930 0x7f5e3c157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60794/0 (socket says 172.21.15.7:60794) 2023-03-03T11:59:27.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.163+0000 7f5e41716700 1 -- 172.21.15.7:0/271900394 learned_addr learned my addr 172.21.15.7:0/271900394 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T11:59:27.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.164+0000 7f5e42718700 1 --2- 172.21.15.7:0/271900394 >> v2:172.21.15.143:3300/0 conn(0x7f5e3c073c30 0x7f5e3c1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:27.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.164+0000 7f5e41f17700 1 -- 172.21.15.7:0/271900394 >> v2:172.21.15.143:3300/0 conn(0x7f5e3c073c30 msgr2=0x7f5e3c1581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:59:27.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.164+0000 7f5e41f17700 1 --2- 172.21.15.7:0/271900394 >> v2:172.21.15.143:3300/0 conn(0x7f5e3c073c30 0x7f5e3c1581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:27.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.164+0000 7f5e41f17700 1 -- 172.21.15.7:0/271900394 >> v1:172.21.15.7:6789/0 conn(0x7f5e3c0c2930 legacy=0x7f5e3c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:27.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.164+0000 7f5e40f15700 1 -- 172.21.15.7:0/271900394 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995283781 0 0) 0x7f5e3c06d430 con 0x7f5e3c0c2930 2023-03-03T11:59:27.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.164+0000 7f5e41f17700 1 -- 172.21.15.7:0/271900394 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c06cb00 con 0x7f5e3c151d40 2023-03-03T11:59:27.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.164+0000 7f5e41f17700 1 --2- 172.21.15.7:0/271900394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c151d40 0x7f5e3c0736f0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009840 tx=0x7f5e2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d7deb5915fa73c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:59:27.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.164+0000 7f5e40f15700 1 -- 172.21.15.7:0/271900394 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c011830 con 0x7f5e3c151d40 2023-03-03T11:59:27.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.164+0000 7f5e40f15700 1 -- 172.21.15.7:0/271900394 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c011990 con 0x7f5e3c151d40 2023-03-03T11:59:27.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.164+0000 7f5e40f15700 1 -- 172.21.15.7:0/271900394 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c011d20 con 0x7f5e3c151d40 2023-03-03T11:59:27.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.165+0000 7f5e4417b700 1 -- 172.21.15.7:0/271900394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c151d40 msgr2=0x7f5e3c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:27.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.165+0000 7f5e4417b700 1 --2- 172.21.15.7:0/271900394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c151d40 0x7f5e3c0736f0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009840 tx=0x7f5e2c009b50 comp rx=0 tx=0).stop 2023-03-03T11:59:27.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.165+0000 7f5e4417b700 1 -- 172.21.15.7:0/271900394 shutdown_connections 2023-03-03T11:59:27.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.165+0000 7f5e4417b700 1 --2- 172.21.15.7:0/271900394 >> v2:172.21.15.143:3300/0 conn(0x7f5e3c073c30 0x7f5e3c1581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:27.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.165+0000 7f5e4417b700 1 --2- 172.21.15.7:0/271900394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c151d40 0x7f5e3c0736f0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:27.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.165+0000 7f5e4417b700 1 -- 172.21.15.7:0/271900394 >> 172.21.15.7:0/271900394 conn(0x7f5e3c0a94d0 msgr2=0x7f5e3c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:59:27.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.165+0000 7f5e4417b700 1 -- 172.21.15.7:0/271900394 shutdown_connections 2023-03-03T11:59:27.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.165+0000 7f5e4417b700 1 -- 172.21.15.7:0/271900394 wait complete. 2023-03-03T11:59:27.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.166+0000 7f5e4417b700 1 Processor -- start 2023-03-03T11:59:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.166+0000 7f5e4417b700 1 -- start start 2023-03-03T11:59:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e4417b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c073c30 0x7f5e3c0c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e4417b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e3c0c2930 0x7f5e3c14bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e4417b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e3c151d40 0x7f5e3c14c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e4417b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0ad210 con 0x7f5e3c073c30 2023-03-03T11:59:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e4417b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c155480 con 0x7f5e3c0c2930 2023-03-03T11:59:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e4417b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0acc50 con 0x7f5e3c151d40 2023-03-03T11:59:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e41f17700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c073c30 0x7f5e3c0c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e41f17700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c073c30 0x7f5e3c0c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38588/0 (socket says 172.21.15.7:38588) 2023-03-03T11:59:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e41f17700 1 -- 172.21.15.7:0/27656545 learned_addr learned my addr 172.21.15.7:0/27656545 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:59:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e41716700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e3c0c2930 0x7f5e3c14bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e42718700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e3c151d40 0x7f5e3c14c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e41f17700 1 -- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e3c151d40 msgr2=0x7f5e3c14c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e41f17700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e3c151d40 0x7f5e3c14c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e41f17700 1 -- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e3c0c2930 msgr2=0x7f5e3c14bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e41f17700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e3c0c2930 0x7f5e3c14bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e41f17700 1 -- 172.21.15.7:0/27656545 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c0094a0 con 0x7f5e3c073c30 2023-03-03T11:59:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e41f17700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c073c30 0x7f5e3c0c69c0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0090b0 tx=0x7f5e2c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:59:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e327fc700 1 -- 172.21.15.7:0/27656545 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0036a0 con 0x7f5e3c073c30 2023-03-03T11:59:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e327fc700 1 -- 172.21.15.7:0/27656545 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c01f070 con 0x7f5e3c073c30 2023-03-03T11:59:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e4417b700 1 -- 172.21.15.7:0/27656545 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c06d0c0 con 0x7f5e3c073c30 2023-03-03T11:59:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.167+0000 7f5e327fc700 1 -- 172.21.15.7:0/27656545 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c003a20 con 0x7f5e3c073c30 2023-03-03T11:59:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.168+0000 7f5e4417b700 1 -- 172.21.15.7:0/27656545 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c06cde0 con 0x7f5e3c073c30 2023-03-03T11:59:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.169+0000 7f5e4417b700 1 -- 172.21.15.7:0/27656545 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c06bea0 con 0x7f5e3c073c30 2023-03-03T11:59:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.169+0000 7f5e327fc700 1 -- 172.21.15.7:0/27656545 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5e2c02a020 con 0x7f5e3c073c30 2023-03-03T11:59:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.169+0000 7f5e327fc700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e200682f0 0x7f5e2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.169+0000 7f5e327fc700 1 -- 172.21.15.7:0/27656545 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5e2c087700 con 0x7f5e3c073c30 2023-03-03T11:59:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.170+0000 7f5e41716700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e200682f0 0x7f5e2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.170+0000 7f5e41716700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e200682f0 0x7f5e2006a7a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c158160 tx=0x7f5e38009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:59:27.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.174+0000 7f5e327fc700 1 -- 172.21.15.7:0/27656545 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5e2c053aa0 con 0x7f5e3c073c30 2023-03-03T11:59:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.412+0000 7f5e4417b700 1 -- 172.21.15.7:0/27656545 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e3c074bb0 con 0x7f5e3c073c30 2023-03-03T11:59:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.413+0000 7f5e327fc700 1 -- 172.21.15.7:0/27656545 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5e2c0570c0 con 0x7f5e3c073c30 2023-03-03T11:59:27.415 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:59:27.415 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:59:27.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 -- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e200682f0 msgr2=0x7f5e2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e200682f0 0x7f5e2006a7a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c158160 tx=0x7f5e38009040 comp rx=0 tx=0).stop 2023-03-03T11:59:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 -- 172.21.15.7:0/27656545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c073c30 msgr2=0x7f5e3c0c69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c073c30 0x7f5e3c0c69c0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0090b0 tx=0x7f5e2c004750 comp rx=0 tx=0).stop 2023-03-03T11:59:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 -- 172.21.15.7:0/27656545 shutdown_connections 2023-03-03T11:59:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e3c151d40 0x7f5e3c14c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:27.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e200682f0 0x7f5e2006a7a0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:27.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e3c073c30 0x7f5e3c0c69c0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:27.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 --2- 172.21.15.7:0/27656545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e3c0c2930 0x7f5e3c14bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:27.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 -- 172.21.15.7:0/27656545 >> 172.21.15.7:0/27656545 conn(0x7f5e3c0a94d0 msgr2=0x7f5e3c071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:59:27.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 -- 172.21.15.7:0/27656545 shutdown_connections 2023-03-03T11:59:27.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:27.415+0000 7f5e4417b700 1 -- 172.21.15.7:0/27656545 wait complete. 2023-03-03T11:59:27.419 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:59:27.428 INFO:tasks.mds_thrash.fs.[b]:waiting for 79 secs before thrashing 2023-03-03T11:59:51.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:51.894 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T11:59:58.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T11:59:58.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.213+0000 7f9366aec700 1 Processor -- start 2023-03-03T11:59:58.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.213+0000 7f9366aec700 1 -- start start 2023-03-03T11:59:58.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.213+0000 7f9366aec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9360072250 0x7f936014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:58.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.213+0000 7f9366aec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600c1ce0 0x7f936014e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:58.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.213+0000 7f9366aec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93601510d0 con 0x7f9360071e80 2023-03-03T11:59:58.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.213+0000 7f9366aec700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93600c3740 con 0x7f9360072250 2023-03-03T11:59:58.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.213+0000 7f9366aec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9360070670 con 0x7f93600c1ce0 2023-03-03T11:59:58.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9360072250 0x7f936014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:58.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9360072250 0x7f936014e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38610/0 (socket says 172.21.15.7:38610) 2023-03-03T11:59:58.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935ffff700 1 -- 172.21.15.7:0/2541482482 learned_addr learned my addr 172.21.15.7:0/2541482482 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:59:58.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f9365089700 1 --2- 172.21.15.7:0/2541482482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600c1ce0 0x7f936014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:58.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935ffff700 1 -- 172.21.15.7:0/2541482482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600c1ce0 msgr2=0x7f936014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T11:59:58.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935ffff700 1 --2- 172.21.15.7:0/2541482482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600c1ce0 0x7f936014e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:58.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935ffff700 1 -- 172.21.15.7:0/2541482482 >> v1:172.21.15.143:6789/0 conn(0x7f9360071e80 legacy=0x7f9360151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:58.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935ffff700 1 -- 172.21.15.7:0/2541482482 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93600c39e0 con 0x7f9360072250 2023-03-03T11:59:58.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935ffff700 1 --2- 172.21.15.7:0/2541482482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 0x7f936014e1f0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f934c00bb90 tx=0x7f934c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f74b0690e0bbfb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:59:58.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935f7fe700 1 -- 172.21.15.7:0/2541482482 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c00dd50 con 0x7f9360072250 2023-03-03T11:59:58.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935f7fe700 1 -- 172.21.15.7:0/2541482482 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c010470 con 0x7f9360072250 2023-03-03T11:59:58.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.214+0000 7f935f7fe700 1 -- 172.21.15.7:0/2541482482 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c010800 con 0x7f9360072250 2023-03-03T11:59:58.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.215+0000 7f9366aec700 1 -- 172.21.15.7:0/2541482482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 msgr2=0x7f936014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:58.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.215+0000 7f9366aec700 1 --2- 172.21.15.7:0/2541482482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 0x7f936014e1f0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f934c00bb90 tx=0x7f934c00bea0 comp rx=0 tx=0).stop 2023-03-03T11:59:58.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.215+0000 7f9366aec700 1 -- 172.21.15.7:0/2541482482 shutdown_connections 2023-03-03T11:59:58.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.215+0000 7f9366aec700 1 --2- 172.21.15.7:0/2541482482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600c1ce0 0x7f936014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:58.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.215+0000 7f9366aec700 1 --2- 172.21.15.7:0/2541482482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 0x7f936014e1f0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:58.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.215+0000 7f9366aec700 1 -- 172.21.15.7:0/2541482482 >> 172.21.15.7:0/2541482482 conn(0x7f93600a94d0 msgr2=0x7f93600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:59:58.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.215+0000 7f9366aec700 1 -- 172.21.15.7:0/2541482482 shutdown_connections 2023-03-03T11:59:58.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.215+0000 7f9366aec700 1 -- 172.21.15.7:0/2541482482 wait complete. 2023-03-03T11:59:58.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.216+0000 7f9366aec700 1 Processor -- start 2023-03-03T11:59:58.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.216+0000 7f9366aec700 1 -- start start 2023-03-03T11:59:58.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f9366aec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9360071e80 0x7f9360149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:58.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f9366aec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 0x7f936014ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:58.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f9366aec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600c1ce0 0x7f936014f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:58.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f9366aec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93601e5fa0 con 0x7f9360072250 2023-03-03T11:59:58.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f9366aec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936014d4d0 con 0x7f9360071e80 2023-03-03T11:59:58.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f9366aec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936014c000 con 0x7f93600c1ce0 2023-03-03T11:59:58.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f935ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 0x7f936014ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:58.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f935ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 0x7f936014ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38616/0 (socket says 172.21.15.7:38616) 2023-03-03T11:59:58.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f935ffff700 1 -- 172.21.15.7:0/2380677859 learned_addr learned my addr 172.21.15.7:0/2380677859 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T11:59:58.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f9364888700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9360071e80 0x7f9360149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:58.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.217+0000 7f9365089700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600c1ce0 0x7f936014f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:58.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f935ffff700 1 -- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600c1ce0 msgr2=0x7f936014f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:58.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f935ffff700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600c1ce0 0x7f936014f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:58.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f935ffff700 1 -- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9360071e80 msgr2=0x7f9360149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:58.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f935ffff700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9360071e80 0x7f9360149bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:58.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f935ffff700 1 -- 172.21.15.7:0/2380677859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c00b7f0 con 0x7f9360072250 2023-03-03T11:59:58.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f935ffff700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 0x7f936014ef40 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f934c00bee0 tx=0x7f934c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:59:58.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f935d7fa700 1 -- 172.21.15.7:0/2380677859 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c01f070 con 0x7f9360072250 2023-03-03T11:59:58.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f935d7fa700 1 -- 172.21.15.7:0/2380677859 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c0038b0 con 0x7f9360072250 2023-03-03T11:59:58.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f935d7fa700 1 -- 172.21.15.7:0/2380677859 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c003c60 con 0x7f9360072250 2023-03-03T11:59:58.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f9366aec700 1 -- 172.21.15.7:0/2380677859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9360151f30 con 0x7f9360072250 2023-03-03T11:59:58.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.218+0000 7f9366aec700 1 -- 172.21.15.7:0/2380677859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9360151300 con 0x7f9360072250 2023-03-03T11:59:58.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.219+0000 7f9366aec700 1 -- 172.21.15.7:0/2380677859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936006c9a0 con 0x7f9360072250 2023-03-03T11:59:58.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.219+0000 7f935d7fa700 1 -- 172.21.15.7:0/2380677859 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f934c019400 con 0x7f9360072250 2023-03-03T11:59:58.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.220+0000 7f935d7fa700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93440682f0 0x7f934406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T11:59:58.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.220+0000 7f935d7fa700 1 -- 172.21.15.7:0/2380677859 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f934c089ef0 con 0x7f9360072250 2023-03-03T11:59:58.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.220+0000 7f9364888700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93440682f0 0x7f934406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T11:59:58.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.220+0000 7f9364888700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93440682f0 0x7f934406a7a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f9354000f60 tx=0x7f9354009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T11:59:58.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.224+0000 7f935d7fa700 1 -- 172.21.15.7:0/2380677859 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f934c0562e0 con 0x7f9360072250 2023-03-03T11:59:58.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.464+0000 7f9366aec700 1 -- 172.21.15.7:0/2380677859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9360074bb0 con 0x7f9360072250 2023-03-03T11:59:58.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.464+0000 7f935d7fa700 1 -- 172.21.15.7:0/2380677859 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f934c059900 con 0x7f9360072250 2023-03-03T11:59:58.467 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T11:59:58.468 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T11:59:58.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 -- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93440682f0 msgr2=0x7f934406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:58.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93440682f0 0x7f934406a7a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f9354000f60 tx=0x7f9354009040 comp rx=0 tx=0).stop 2023-03-03T11:59:58.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 -- 172.21.15.7:0/2380677859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 msgr2=0x7f936014ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T11:59:58.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 0x7f936014ef40 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f934c00bee0 tx=0x7f934c004750 comp rx=0 tx=0).stop 2023-03-03T11:59:58.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 -- 172.21.15.7:0/2380677859 shutdown_connections 2023-03-03T11:59:58.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f93600c1ce0 0x7f936014f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:58.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9360071e80 0x7f9360149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:58.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93440682f0 0x7f934406a7a0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:58.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 --2- 172.21.15.7:0/2380677859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9360072250 0x7f936014ef40 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T11:59:58.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 -- 172.21.15.7:0/2380677859 >> 172.21.15.7:0/2380677859 conn(0x7f93600a94d0 msgr2=0x7f936006d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T11:59:58.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 -- 172.21.15.7:0/2380677859 shutdown_connections 2023-03-03T11:59:58.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T11:59:58.467+0000 7f9366aec700 1 -- 172.21.15.7:0/2380677859 wait complete. 2023-03-03T11:59:58.472 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T11:59:58.481 INFO:tasks.mds_thrash.fs.[a]:waiting for 44 secs before thrashing 2023-03-03T12:00:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T11:59:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T12:00:21.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:21.948 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:42.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:00:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.620+0000 7f535f0bb700 1 Processor -- start 2023-03-03T12:00:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.620+0000 7f535f0bb700 1 -- start start 2023-03-03T12:00:42.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f535f0bb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 0x7f5358154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:42.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f535f0bb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f53580c2e30 0x7f5358158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:42.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f535f0bb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f535806c6c0 0x7f53580736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:42.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f535f0bb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5358157ff0 con 0x7f535806c6c0 2023-03-03T12:00:42.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f535f0bb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f535806d640 con 0x7f53580c2e30 2023-03-03T12:00:42.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f535f0bb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f535806cda0 con 0x7f5358151cd0 2023-03-03T12:00:42.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f535d658700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f535806c6c0 0x7f53580736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:42.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f535d658700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f535806c6c0 0x7f53580736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38664/0 (socket says 172.21.15.7:38664) 2023-03-03T12:00:42.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f535d658700 1 -- 172.21.15.7:0/1954338900 learned_addr learned my addr 172.21.15.7:0/1954338900 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:00:42.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f5357fff700 1 --2- 172.21.15.7:0/1954338900 >> v2:172.21.15.143:3300/0 conn(0x7f53580c2e30 0x7f5358158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:42.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.621+0000 7f535ce57700 1 --2- 172.21.15.7:0/1954338900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 0x7f5358154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:42.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535d658700 1 -- 172.21.15.7:0/1954338900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 msgr2=0x7f5358154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:00:42.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535d658700 1 --2- 172.21.15.7:0/1954338900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 0x7f5358154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535d658700 1 -- 172.21.15.7:0/1954338900 >> v2:172.21.15.143:3300/0 conn(0x7f53580c2e30 msgr2=0x7f5358158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:00:42.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535d658700 1 --2- 172.21.15.7:0/1954338900 >> v2:172.21.15.143:3300/0 conn(0x7f53580c2e30 0x7f5358158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535d658700 1 -- 172.21.15.7:0/1954338900 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535806d080 con 0x7f535806c6c0 2023-03-03T12:00:42.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535ce57700 1 --2- 172.21.15.7:0/1954338900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 0x7f5358154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:00:42.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535d658700 1 --2- 172.21.15.7:0/1954338900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 0x7f53580736f0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f5358074150 tx=0x7f535000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca984a7ffe28694d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:00:42.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f53577fe700 1 -- 172.21.15.7:0/1954338900 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535000e070 con 0x7f535806c6c0 2023-03-03T12:00:42.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f53577fe700 1 -- 172.21.15.7:0/1954338900 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535000b910 con 0x7f535806c6c0 2023-03-03T12:00:42.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f53577fe700 1 -- 172.21.15.7:0/1954338900 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535000bcc0 con 0x7f535806c6c0 2023-03-03T12:00:42.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535f0bb700 1 -- 172.21.15.7:0/1954338900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 msgr2=0x7f53580736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:00:42.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535f0bb700 1 --2- 172.21.15.7:0/1954338900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 0x7f53580736f0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f5358074150 tx=0x7f535000b330 comp rx=0 tx=0).stop 2023-03-03T12:00:42.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535f0bb700 1 -- 172.21.15.7:0/1954338900 shutdown_connections 2023-03-03T12:00:42.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535f0bb700 1 --2- 172.21.15.7:0/1954338900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 0x7f53580736f0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535f0bb700 1 --2- 172.21.15.7:0/1954338900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 0x7f5358154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535f0bb700 1 --2- 172.21.15.7:0/1954338900 >> v2:172.21.15.143:3300/0 conn(0x7f53580c2e30 0x7f5358158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535f0bb700 1 -- 172.21.15.7:0/1954338900 >> 172.21.15.7:0/1954338900 conn(0x7f53580a94d0 msgr2=0x7f53580a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:00:42.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.622+0000 7f535f0bb700 1 -- 172.21.15.7:0/1954338900 shutdown_connections 2023-03-03T12:00:42.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.623+0000 7f535f0bb700 1 -- 172.21.15.7:0/1954338900 wait complete. 2023-03-03T12:00:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.624+0000 7f535f0bb700 1 Processor -- start 2023-03-03T12:00:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.624+0000 7f535f0bb700 1 -- start start 2023-03-03T12:00:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f535f0bb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 0x7f5358156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f535f0bb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53580c2e30 0x7f53580c5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f535f0bb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 0x7f53580c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f535f0bb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53580c67f0 con 0x7f535806c6c0 2023-03-03T12:00:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f535f0bb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53580c5370 con 0x7f53580c2e30 2023-03-03T12:00:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f535f0bb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53580c65a0 con 0x7f5358151cd0 2023-03-03T12:00:42.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f535ce57700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 0x7f5358156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:42.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f535ce57700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 0x7f5358156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38666/0 (socket says 172.21.15.7:38666) 2023-03-03T12:00:42.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f535ce57700 1 -- 172.21.15.7:0/3509658972 learned_addr learned my addr 172.21.15.7:0/3509658972 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:00:42.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f5357fff700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53580c2e30 0x7f53580c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:42.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.625+0000 7f535d658700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 0x7f53580c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:42.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f535ce57700 1 -- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 msgr2=0x7f53580c5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:00:42.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f535ce57700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 0x7f53580c5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f535ce57700 1 -- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53580c2e30 msgr2=0x7f53580c5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:00:42.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f535ce57700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53580c2e30 0x7f53580c5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f535ce57700 1 -- 172.21.15.7:0/3509658972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5350009b30 con 0x7f535806c6c0 2023-03-03T12:00:42.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f535ce57700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 0x7f5358156bc0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f534c00e810 tx=0x7f534c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:00:42.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f53557fa700 1 -- 172.21.15.7:0/3509658972 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c00c070 con 0x7f535806c6c0 2023-03-03T12:00:42.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f53557fa700 1 -- 172.21.15.7:0/3509658972 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534c004230 con 0x7f535806c6c0 2023-03-03T12:00:42.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f53557fa700 1 -- 172.21.15.7:0/3509658972 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534c0036a0 con 0x7f535806c6c0 2023-03-03T12:00:42.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f535f0bb700 1 -- 172.21.15.7:0/3509658972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5358157c80 con 0x7f535806c6c0 2023-03-03T12:00:42.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.626+0000 7f535f0bb700 1 -- 172.21.15.7:0/3509658972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535806d360 con 0x7f535806c6c0 2023-03-03T12:00:42.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.627+0000 7f535f0bb700 1 -- 172.21.15.7:0/3509658972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5358154a50 con 0x7f535806c6c0 2023-03-03T12:00:42.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.628+0000 7f53557fa700 1 -- 172.21.15.7:0/3509658972 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f534c019020 con 0x7f535806c6c0 2023-03-03T12:00:42.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.628+0000 7f53557fa700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 0x7f533c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:42.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.628+0000 7f53557fa700 1 -- 172.21.15.7:0/3509658972 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f534c085dc0 con 0x7f535806c6c0 2023-03-03T12:00:42.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.628+0000 7f5357fff700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 0x7f533c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:42.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.629+0000 7f5357fff700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 0x7f533c06a7a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f53580c4e70 tx=0x7f5344009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:00:42.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.632+0000 7f53557fa700 1 -- 172.21.15.7:0/3509658972 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f534c052160 con 0x7f535806c6c0 2023-03-03T12:00:42.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.869+0000 7f535f0bb700 1 -- 172.21.15.7:0/3509658972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53580c51d0 con 0x7f535806c6c0 2023-03-03T12:00:42.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.870+0000 7f53557fa700 1 -- 172.21.15.7:0/3509658972 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f53580c51d0 con 0x7f535806c6c0 2023-03-03T12:00:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:00:42.872 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:00:42.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 -- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 msgr2=0x7f533c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:00:42.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 0x7f533c06a7a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f53580c4e70 tx=0x7f5344009210 comp rx=0 tx=0).stop 2023-03-03T12:00:42.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 -- 172.21.15.7:0/3509658972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 msgr2=0x7f5358156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:00:42.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 0x7f5358156bc0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f534c00e810 tx=0x7f534c00eb20 comp rx=0 tx=0).stop 2023-03-03T12:00:42.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 -- 172.21.15.7:0/3509658972 shutdown_connections 2023-03-03T12:00:42.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5358151cd0 0x7f53580c5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 0x7f533c06a7a0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f535806c6c0 0x7f5358156bc0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 --2- 172.21.15.7:0/3509658972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53580c2e30 0x7f53580c5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 -- 172.21.15.7:0/3509658972 >> 172.21.15.7:0/3509658972 conn(0x7f53580a94d0 msgr2=0x7f5358070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:00:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 -- 172.21.15.7:0/3509658972 shutdown_connections 2023-03-03T12:00:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:42.872+0000 7f535f0bb700 1 -- 172.21.15.7:0/3509658972 wait complete. 2023-03-03T12:00:42.875 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:00:42.885 INFO:tasks.mds_thrash.fs.[a]:waiting for 87 secs before thrashing 2023-03-03T12:00:46.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:00:46.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.566+0000 7f615cce1700 1 Processor -- start 2023-03-03T12:00:46.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.566+0000 7f615cce1700 1 -- start start 2023-03-03T12:00:46.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.567+0000 7f615cce1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6158071990 0x7f6158071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:46.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.567+0000 7f615cce1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f615814b460 0x7f61580722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:46.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.567+0000 7f615cce1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6158151c00 0x7f615806aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:46.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.567+0000 7f615cce1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61581521a0 con 0x7f6158151c00 2023-03-03T12:00:46.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.567+0000 7f615cce1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6158070360 con 0x7f615814b460 2023-03-03T12:00:46.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.567+0000 7f615cce1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615806a5e0 con 0x7f6158071990 2023-03-03T12:00:46.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.567+0000 7f6156d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6158151c00 0x7f615806aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:46.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f6156d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6158151c00 0x7f615806aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38678/0 (socket says 172.21.15.7:38678) 2023-03-03T12:00:46.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f6156d9d700 1 -- 172.21.15.7:0/2402164144 learned_addr learned my addr 172.21.15.7:0/2402164144 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:00:46.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f615659c700 1 --2- 172.21.15.7:0/2402164144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6158071990 0x7f6158071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:46.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f6155d9b700 1 --2- 172.21.15.7:0/2402164144 >> v2:172.21.15.143:3300/0 conn(0x7f615814b460 0x7f61580722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:46.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f6156d9d700 1 -- 172.21.15.7:0/2402164144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6158071990 msgr2=0x7f6158071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:00:46.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f6156d9d700 1 --2- 172.21.15.7:0/2402164144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6158071990 0x7f6158071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f6156d9d700 1 -- 172.21.15.7:0/2402164144 >> v2:172.21.15.143:3300/0 conn(0x7f615814b460 msgr2=0x7f61580722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:00:46.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f6156d9d700 1 --2- 172.21.15.7:0/2402164144 >> v2:172.21.15.143:3300/0 conn(0x7f615814b460 0x7f61580722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f6156d9d700 1 -- 172.21.15.7:0/2402164144 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61580c38b0 con 0x7f6158151c00 2023-03-03T12:00:46.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f6156d9d700 1 --2- 172.21.15.7:0/2402164144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 0x7f615806aa70 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f615806bd20 tx=0x7f615000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b72d61952c19fbd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:00:46.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f615559a700 1 -- 172.21.15.7:0/2402164144 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615000e070 con 0x7f6158151c00 2023-03-03T12:00:46.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f615559a700 1 -- 172.21.15.7:0/2402164144 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615000b910 con 0x7f6158151c00 2023-03-03T12:00:46.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.568+0000 7f615559a700 1 -- 172.21.15.7:0/2402164144 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615000bcc0 con 0x7f6158151c00 2023-03-03T12:00:46.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.569+0000 7f615cce1700 1 -- 172.21.15.7:0/2402164144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 msgr2=0x7f615806aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:00:46.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.569+0000 7f615cce1700 1 --2- 172.21.15.7:0/2402164144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 0x7f615806aa70 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f615806bd20 tx=0x7f615000b330 comp rx=0 tx=0).stop 2023-03-03T12:00:46.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.569+0000 7f615cce1700 1 -- 172.21.15.7:0/2402164144 shutdown_connections 2023-03-03T12:00:46.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.569+0000 7f615cce1700 1 --2- 172.21.15.7:0/2402164144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 0x7f615806aa70 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.569+0000 7f615cce1700 1 --2- 172.21.15.7:0/2402164144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6158071990 0x7f6158071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.569+0000 7f615cce1700 1 --2- 172.21.15.7:0/2402164144 >> v2:172.21.15.143:3300/0 conn(0x7f615814b460 0x7f61580722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.569+0000 7f615cce1700 1 -- 172.21.15.7:0/2402164144 >> 172.21.15.7:0/2402164144 conn(0x7f61580a94d0 msgr2=0x7f61580a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:00:46.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.569+0000 7f615cce1700 1 -- 172.21.15.7:0/2402164144 shutdown_connections 2023-03-03T12:00:46.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.569+0000 7f615cce1700 1 -- 172.21.15.7:0/2402164144 wait complete. 2023-03-03T12:00:46.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.570+0000 7f615cce1700 1 Processor -- start 2023-03-03T12:00:46.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.570+0000 7f615cce1700 1 -- start start 2023-03-03T12:00:46.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f615cce1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6158071990 0x7f61580aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:46.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f615cce1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 0x7f61581ddbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:46.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f615cce1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61581de120 0x7f61580ab680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:46.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f615cce1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61580ab330 con 0x7f6158151c00 2023-03-03T12:00:46.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f615cce1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61581501d0 con 0x7f6158071990 2023-03-03T12:00:46.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f615cce1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6158150660 con 0x7f61581de120 2023-03-03T12:00:46.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f6155d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 0x7f61581ddbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:46.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f6155d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 0x7f61581ddbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38682/0 (socket says 172.21.15.7:38682) 2023-03-03T12:00:46.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f6155d9b700 1 -- 172.21.15.7:0/3903995769 learned_addr learned my addr 172.21.15.7:0/3903995769 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:00:46.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f615659c700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6158071990 0x7f61580aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:46.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.571+0000 7f6156d9d700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61581de120 0x7f61580ab680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f6155d9b700 1 -- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61581de120 msgr2=0x7f61580ab680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:00:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f6155d9b700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61581de120 0x7f61580ab680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f6155d9b700 1 -- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6158071990 msgr2=0x7f61580aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:00:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f6155d9b700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6158071990 0x7f61580aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f6155d9b700 1 -- 172.21.15.7:0/3903995769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6150009b30 con 0x7f6158151c00 2023-03-03T12:00:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f6156d9d700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61581de120 0x7f61580ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:00:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f6155d9b700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 0x7f61581ddbe0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f614000e9b0 tx=0x7f614000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:00:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f614effd700 1 -- 172.21.15.7:0/3903995769 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140009710 con 0x7f6158151c00 2023-03-03T12:00:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f614effd700 1 -- 172.21.15.7:0/3903995769 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614000c850 con 0x7f6158151c00 2023-03-03T12:00:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f614effd700 1 -- 172.21.15.7:0/3903995769 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614000cbc0 con 0x7f6158151c00 2023-03-03T12:00:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f615cce1700 1 -- 172.21.15.7:0/3903995769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61580c3b50 con 0x7f6158151c00 2023-03-03T12:00:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.572+0000 7f615cce1700 1 -- 172.21.15.7:0/3903995769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615806a2b0 con 0x7f6158151c00 2023-03-03T12:00:46.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.573+0000 7f615cce1700 1 -- 172.21.15.7:0/3903995769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615806afb0 con 0x7f6158151c00 2023-03-03T12:00:46.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.574+0000 7f614effd700 1 -- 172.21.15.7:0/3903995769 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6140023070 con 0x7f6158151c00 2023-03-03T12:00:46.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.575+0000 7f614effd700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6138068320 0x7f613806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:00:46.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.575+0000 7f614effd700 1 -- 172.21.15.7:0/3903995769 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f61400862e0 con 0x7f6158151c00 2023-03-03T12:00:46.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.575+0000 7f615659c700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6138068320 0x7f613806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:00:46.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.575+0000 7f615659c700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6138068320 0x7f613806a7d0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6148009be0 tx=0x7f6148009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:00:46.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.579+0000 7f614effd700 1 -- 172.21.15.7:0/3903995769 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6140052680 con 0x7f6158151c00 2023-03-03T12:00:46.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.816+0000 7f615cce1700 1 -- 172.21.15.7:0/3903995769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6158150340 con 0x7f6158151c00 2023-03-03T12:00:46.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.817+0000 7f614effd700 1 -- 172.21.15.7:0/3903995769 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6140055ca0 con 0x7f6158151c00 2023-03-03T12:00:46.819 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:00:46.819 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:00:46.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 -- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6138068320 msgr2=0x7f613806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:00:46.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6138068320 0x7f613806a7d0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6148009be0 tx=0x7f6148009210 comp rx=0 tx=0).stop 2023-03-03T12:00:46.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 -- 172.21.15.7:0/3903995769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 msgr2=0x7f61581ddbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:00:46.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 0x7f61581ddbe0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f614000e9b0 tx=0x7f614000ecc0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 -- 172.21.15.7:0/3903995769 shutdown_connections 2023-03-03T12:00:46.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61581de120 0x7f61580ab680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6158071990 0x7f61580aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6138068320 0x7f613806a7d0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 --2- 172.21.15.7:0/3903995769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6158151c00 0x7f61581ddbe0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:00:46.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 -- 172.21.15.7:0/3903995769 >> 172.21.15.7:0/3903995769 conn(0x7f61580a94d0 msgr2=0x7f6158065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:00:46.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 -- 172.21.15.7:0/3903995769 shutdown_connections 2023-03-03T12:00:46.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:00:46.819+0000 7f615cce1700 1 -- 172.21.15.7:0/3903995769 wait complete. 2023-03-03T12:00:46.823 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:00:46.832 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-03-03T12:00:52.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:00:52.021 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:22.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:22.168 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:52.269 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:01:52.272 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:09.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:02:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.024+0000 7f7d3ece5700 1 Processor -- start 2023-03-03T12:02:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.024+0000 7f7d3ece5700 1 -- start start 2023-03-03T12:02:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3ece5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3814f860 0x7f7d381562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3ece5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7d380c20d0 0x7f7d38152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3ece5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7d380c27c0 0x7f7d380736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3ece5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3806d350 con 0x7f7d380c27c0 2023-03-03T12:02:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3ece5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d381558e0 con 0x7f7d380c20d0 2023-03-03T12:02:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3ece5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d38156830 con 0x7f7d3814f860 2023-03-03T12:02:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3d282700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7d380c27c0 0x7f7d380736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3d282700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7d380c27c0 0x7f7d380736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38794/0 (socket says 172.21.15.7:38794) 2023-03-03T12:02:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3d282700 1 -- 172.21.15.7:0/3362457151 learned_addr learned my addr 172.21.15.7:0/3362457151 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:02:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3ca81700 1 --2- 172.21.15.7:0/3362457151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3814f860 0x7f7d381562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d37fff700 1 --2- 172.21.15.7:0/3362457151 >> v2:172.21.15.143:3300/0 conn(0x7f7d380c20d0 0x7f7d38152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3d282700 1 -- 172.21.15.7:0/3362457151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3814f860 msgr2=0x7f7d381562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:02:10.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3d282700 1 --2- 172.21.15.7:0/3362457151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3814f860 0x7f7d381562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3d282700 1 -- 172.21.15.7:0/3362457151 >> v2:172.21.15.143:3300/0 conn(0x7f7d380c20d0 msgr2=0x7f7d38152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:02:10.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.025+0000 7f7d3d282700 1 --2- 172.21.15.7:0/3362457151 >> v2:172.21.15.143:3300/0 conn(0x7f7d380c20d0 0x7f7d38152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d3d282700 1 -- 172.21.15.7:0/3362457151 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38155e60 con 0x7f7d380c27c0 2023-03-03T12:02:10.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d37fff700 1 --2- 172.21.15.7:0/3362457151 >> v2:172.21.15.143:3300/0 conn(0x7f7d380c20d0 0x7f7d38152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:02:10.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d3d282700 1 --2- 172.21.15.7:0/3362457151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d380c27c0 0x7f7d380736f0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7d38072430 tx=0x7f7d3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c61b611746b5d63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:10.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d377fe700 1 -- 172.21.15.7:0/3362457151 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3000e070 con 0x7f7d380c27c0 2023-03-03T12:02:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d377fe700 1 -- 172.21.15.7:0/3362457151 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3000b980 con 0x7f7d380c27c0 2023-03-03T12:02:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d377fe700 1 -- 172.21.15.7:0/3362457151 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3000bd30 con 0x7f7d380c27c0 2023-03-03T12:02:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d3ece5700 1 -- 172.21.15.7:0/3362457151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d380c27c0 msgr2=0x7f7d380736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d3ece5700 1 --2- 172.21.15.7:0/3362457151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d380c27c0 0x7f7d380736f0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7d38072430 tx=0x7f7d3000b330 comp rx=0 tx=0).stop 2023-03-03T12:02:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d3ece5700 1 -- 172.21.15.7:0/3362457151 shutdown_connections 2023-03-03T12:02:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d3ece5700 1 --2- 172.21.15.7:0/3362457151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d380c27c0 0x7f7d380736f0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d3ece5700 1 --2- 172.21.15.7:0/3362457151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3814f860 0x7f7d381562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d3ece5700 1 --2- 172.21.15.7:0/3362457151 >> v2:172.21.15.143:3300/0 conn(0x7f7d380c20d0 0x7f7d38152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.026+0000 7f7d3ece5700 1 -- 172.21.15.7:0/3362457151 >> 172.21.15.7:0/3362457151 conn(0x7f7d380a94d0 msgr2=0x7f7d380a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:02:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.027+0000 7f7d3ece5700 1 -- 172.21.15.7:0/3362457151 shutdown_connections 2023-03-03T12:02:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.027+0000 7f7d3ece5700 1 -- 172.21.15.7:0/3362457151 wait complete. 2023-03-03T12:02:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.028+0000 7f7d3ece5700 1 Processor -- start 2023-03-03T12:02:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.028+0000 7f7d3ece5700 1 -- start start 2023-03-03T12:02:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3ece5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d380c20d0 0x7f7d381ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3ece5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d380c27c0 0x7f7d380ac430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:10.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3ece5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3814f860 0x7f7d38070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3ece5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d38070520 con 0x7f7d3814f860 2023-03-03T12:02:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3ece5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380c08a0 con 0x7f7d380c27c0 2023-03-03T12:02:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3ece5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d380701d0 con 0x7f7d380c20d0 2023-03-03T12:02:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3d282700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3814f860 0x7f7d38070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3d282700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3814f860 0x7f7d38070940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38800/0 (socket says 172.21.15.7:38800) 2023-03-03T12:02:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3d282700 1 -- 172.21.15.7:0/1463579958 learned_addr learned my addr 172.21.15.7:0/1463579958 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:02:10.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3ca81700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d380c20d0 0x7f7d381ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:10.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d37fff700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d380c27c0 0x7f7d380ac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:10.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3d282700 1 -- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d380c20d0 msgr2=0x7f7d381ea8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:10.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3d282700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d380c20d0 0x7f7d381ea8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3d282700 1 -- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d380c27c0 msgr2=0x7f7d380ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:10.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3d282700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d380c27c0 0x7f7d380ac430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3d282700 1 -- 172.21.15.7:0/1463579958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d30009b30 con 0x7f7d3814f860 2023-03-03T12:02:10.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d3d282700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3814f860 0x7f7d38070940 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000b730 tx=0x7f7d30004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:10.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.029+0000 7f7d357fa700 1 -- 172.21.15.7:0/1463579958 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30004450 con 0x7f7d3814f860 2023-03-03T12:02:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.030+0000 7f7d357fa700 1 -- 172.21.15.7:0/1463579958 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d300045b0 con 0x7f7d3814f860 2023-03-03T12:02:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.030+0000 7f7d3ece5700 1 -- 172.21.15.7:0/1463579958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3806cd90 con 0x7f7d3814f860 2023-03-03T12:02:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.030+0000 7f7d357fa700 1 -- 172.21.15.7:0/1463579958 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30004960 con 0x7f7d3814f860 2023-03-03T12:02:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.030+0000 7f7d3ece5700 1 -- 172.21.15.7:0/1463579958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d381532b0 con 0x7f7d3814f860 2023-03-03T12:02:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.031+0000 7f7d3ece5700 1 -- 172.21.15.7:0/1463579958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3806d070 con 0x7f7d3814f860 2023-03-03T12:02:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.031+0000 7f7d357fa700 1 -- 172.21.15.7:0/1463579958 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7d3002c070 con 0x7f7d3814f860 2023-03-03T12:02:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.031+0000 7f7d357fa700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d1c0682f0 0x7f7d1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:10.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.031+0000 7f7d357fa700 1 -- 172.21.15.7:0/1463579958 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7d30087710 con 0x7f7d3814f860 2023-03-03T12:02:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.032+0000 7f7d3ca81700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d1c0682f0 0x7f7d1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.032+0000 7f7d3ca81700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d1c0682f0 0x7f7d1c06a7a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c006b20 tx=0x7f7d2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:10.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.036+0000 7f7d357fa700 1 -- 172.21.15.7:0/1463579958 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7d30053ab0 con 0x7f7d3814f860 2023-03-03T12:02:10.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.270+0000 7f7d3ece5700 1 -- 172.21.15.7:0/1463579958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d38070660 con 0x7f7d3814f860 2023-03-03T12:02:10.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.270+0000 7f7d357fa700 1 -- 172.21.15.7:0/1463579958 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7d38070660 con 0x7f7d3814f860 2023-03-03T12:02:10.273 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:02:10.273 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:02:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 -- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d1c0682f0 msgr2=0x7f7d1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d1c0682f0 0x7f7d1c06a7a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c006b20 tx=0x7f7d2c008040 comp rx=0 tx=0).stop 2023-03-03T12:02:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 -- 172.21.15.7:0/1463579958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3814f860 msgr2=0x7f7d38070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3814f860 0x7f7d38070940 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000b730 tx=0x7f7d30004ad0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 -- 172.21.15.7:0/1463579958 shutdown_connections 2023-03-03T12:02:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d1c0682f0 0x7f7d1c06a7a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3814f860 0x7f7d38070940 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d380c20d0 0x7f7d381ea8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 --2- 172.21.15.7:0/1463579958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d380c27c0 0x7f7d380ac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 -- 172.21.15.7:0/1463579958 >> 172.21.15.7:0/1463579958 conn(0x7f7d380a94d0 msgr2=0x7f7d3806a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:02:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 -- 172.21.15.7:0/1463579958 shutdown_connections 2023-03-03T12:02:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:10.272+0000 7f7d3ece5700 1 -- 172.21.15.7:0/1463579958 wait complete. 2023-03-03T12:02:10.277 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:02:10.287 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T12:02:22.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:22.413 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:35.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:02:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.971+0000 7fd0d3b1d700 1 Processor -- start 2023-03-03T12:02:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.971+0000 7fd0d3b1d700 1 -- start start 2023-03-03T12:02:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d3b1d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd0cc0c2e30 0x7fd0cc158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d3b1d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0cc157ff0 con 0x7fd0cc06c6c0 2023-03-03T12:02:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d3b1d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0cc06d640 con 0x7fd0cc151cd0 2023-03-03T12:02:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d3b1d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0cc06cda0 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d10b8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd0cc0c2e30 0x7fd0cc158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d20ba700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd0cc06c6c0 0x7fd0cc0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:32806/0 (socket says 172.21.15.7:32806) 2023-03-03T12:02:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d20ba700 1 -- 172.21.15.7:0/4140090424 learned_addr learned my addr 172.21.15.7:0/4140090424 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:02:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d10b8700 1 -- 172.21.15.7:0/4140090424 >> v1:172.21.15.143:6789/0 conn(0x7fd0cc151cd0 legacy=0x7fd0cc154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d10b8700 1 -- 172.21.15.7:0/4140090424 >> v1:172.21.15.7:6789/0 conn(0x7fd0cc06c6c0 legacy=0x7fd0cc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d08b7700 1 -- 172.21.15.7:0/4140090424 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33590206 0 0) 0x7fd0cc157ff0 con 0x7fd0cc06c6c0 2023-03-03T12:02:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d10b8700 1 -- 172.21.15.7:0/4140090424 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc06d080 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.972+0000 7fd0d10b8700 1 --2- 172.21.15.7:0/4140090424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 0x7fd0cc158480 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009840 tx=0x7fd0bc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6e56e4f85eba39 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.973+0000 7fd0d08b7700 1 -- 172.21.15.7:0/4140090424 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc011830 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.973+0000 7fd0d08b7700 1 -- 172.21.15.7:0/4140090424 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc011990 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.973+0000 7fd0d08b7700 1 -- 172.21.15.7:0/4140090424 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc011d20 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.973+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/4140090424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 msgr2=0x7fd0cc158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.973+0000 7fd0d3b1d700 1 --2- 172.21.15.7:0/4140090424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 0x7fd0cc158480 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009840 tx=0x7fd0bc009b50 comp rx=0 tx=0).stop 2023-03-03T12:02:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.973+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/4140090424 shutdown_connections 2023-03-03T12:02:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.973+0000 7fd0d3b1d700 1 --2- 172.21.15.7:0/4140090424 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 0x7fd0cc158480 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.973+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/4140090424 >> 172.21.15.7:0/4140090424 conn(0x7fd0cc0a94d0 msgr2=0x7fd0cc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:02:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.973+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/4140090424 shutdown_connections 2023-03-03T12:02:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.973+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/4140090424 wait complete. 2023-03-03T12:02:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.974+0000 7fd0d3b1d700 1 Processor -- start 2023-03-03T12:02:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.975+0000 7fd0d3b1d700 1 -- start start 2023-03-03T12:02:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.975+0000 7fd0d3b1d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0cc06c6c0 0x7fd0cc0c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.975+0000 7fd0d3b1d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 0x7fd0cc155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.975+0000 7fd0d3b1d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0cc151cd0 0x7fd0cc155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.975+0000 7fd0d3b1d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc152f70 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.975+0000 7fd0d3b1d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc152230 con 0x7fd0cc06c6c0 2023-03-03T12:02:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.975+0000 7fd0d3b1d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc152510 con 0x7fd0cc151cd0 2023-03-03T12:02:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.975+0000 7fd0d10b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 0x7fd0cc155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.975+0000 7fd0d10b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 0x7fd0cc155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38830/0 (socket says 172.21.15.7:38830) 2023-03-03T12:02:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.975+0000 7fd0d10b8700 1 -- 172.21.15.7:0/826093802 learned_addr learned my addr 172.21.15.7:0/826093802 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:02:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d20ba700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0cc151cd0 0x7fd0cc155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d18b9700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0cc06c6c0 0x7fd0cc0c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d10b8700 1 -- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0cc151cd0 msgr2=0x7fd0cc155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d10b8700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0cc151cd0 0x7fd0cc155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d10b8700 1 -- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0cc06c6c0 msgr2=0x7fd0cc0c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d10b8700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0cc06c6c0 0x7fd0cc0c6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d10b8700 1 -- 172.21.15.7:0/826093802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc0094a0 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d20ba700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0cc151cd0 0x7fd0cc155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:02:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d10b8700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 0x7fd0cc155900 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc014040 tx=0x7fd0bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0c27fc700 1 -- 172.21.15.7:0/826093802 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc01a900 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0c27fc700 1 -- 172.21.15.7:0/826093802 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc01aa60 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/826093802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0cc157c80 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0c27fc700 1 -- 172.21.15.7:0/826093802 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc01add0 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.976+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/826093802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0cc06d360 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.977+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/826093802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc1589c0 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.978+0000 7fd0c27fc700 1 -- 172.21.15.7:0/826093802 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd0bc02b030 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.978+0000 7fd0c27fc700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0b0068320 0x7fd0b006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.978+0000 7fd0c27fc700 1 -- 172.21.15.7:0/826093802 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd0bc087580 con 0x7fd0cc0c2e30 2023-03-03T12:02:35.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.978+0000 7fd0d18b9700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0b0068320 0x7fd0b006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.979+0000 7fd0d18b9700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0b0068320 0x7fd0b006a7d0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8000f60 tx=0x7fd0c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:35.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:35.982+0000 7fd0c27fc700 1 -- 172.21.15.7:0/826093802 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd0bc053920 con 0x7fd0cc0c2e30 2023-03-03T12:02:36.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.219+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/826093802 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0cc074bb0 con 0x7fd0cc0c2e30 2023-03-03T12:02:36.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.220+0000 7fd0c27fc700 1 -- 172.21.15.7:0/826093802 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd0bc056f40 con 0x7fd0cc0c2e30 2023-03-03T12:02:36.222 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:02:36.222 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:02:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0b0068320 msgr2=0x7fd0b006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0b0068320 0x7fd0b006a7d0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8000f60 tx=0x7fd0c8009040 comp rx=0 tx=0).stop 2023-03-03T12:02:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/826093802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 msgr2=0x7fd0cc155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 0x7fd0cc155900 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc014040 tx=0x7fd0bc004750 comp rx=0 tx=0).stop 2023-03-03T12:02:36.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/826093802 shutdown_connections 2023-03-03T12:02:36.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0cc151cd0 0x7fd0cc155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:36.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0cc06c6c0 0x7fd0cc0c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:36.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0b0068320 0x7fd0b006a7d0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:36.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 --2- 172.21.15.7:0/826093802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0cc0c2e30 0x7fd0cc155900 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:36.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/826093802 >> 172.21.15.7:0/826093802 conn(0x7fd0cc0a94d0 msgr2=0x7fd0cc0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:02:36.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/826093802 shutdown_connections 2023-03-03T12:02:36.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:36.222+0000 7fd0d3b1d700 1 -- 172.21.15.7:0/826093802 wait complete. 2023-03-03T12:02:36.226 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:02:36.235 INFO:tasks.mds_thrash.fs.[b]:waiting for 21 secs before thrashing 2023-03-03T12:02:52.545 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:52.549 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:02:57.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:02:57.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.373+0000 7fba80219700 1 Processor -- start 2023-03-03T12:02:57.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.373+0000 7fba80219700 1 -- start start 2023-03-03T12:02:57.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.374+0000 7fba80219700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba78071e80 0x7fba78151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:57.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.374+0000 7fba80219700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba78072250 0x7fba7814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:57.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.374+0000 7fba80219700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba781510d0 con 0x7fba780c1ce0 2023-03-03T12:02:57.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.374+0000 7fba80219700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba780c3740 con 0x7fba78071e80 2023-03-03T12:02:57.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.374+0000 7fba80219700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba78070670 con 0x7fba78072250 2023-03-03T12:02:57.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.374+0000 7fba7dfb5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba78071e80 0x7fba78151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:57.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.374+0000 7fba7dfb5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba78071e80 0x7fba78151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38870/0 (socket says 172.21.15.7:38870) 2023-03-03T12:02:57.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.374+0000 7fba7dfb5700 1 -- 172.21.15.7:0/1825116857 learned_addr learned my addr 172.21.15.7:0/1825116857 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:02:57.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.374+0000 7fba7d7b4700 1 --2- 172.21.15.7:0/1825116857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba78072250 0x7fba7814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:57.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.374+0000 7fba7dfb5700 1 -- 172.21.15.7:0/1825116857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba78072250 msgr2=0x7fba7814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:02:57.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba7dfb5700 1 --2- 172.21.15.7:0/1825116857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba78072250 0x7fba7814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:57.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba7dfb5700 1 -- 172.21.15.7:0/1825116857 >> v1:172.21.15.143:6789/0 conn(0x7fba780c1ce0 legacy=0x7fba781546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:02:57.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba7dfb5700 1 -- 172.21.15.7:0/1825116857 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba780c39e0 con 0x7fba78071e80 2023-03-03T12:02:57.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba7d7b4700 1 --2- 172.21.15.7:0/1825116857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba78072250 0x7fba7814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:02:57.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba7dfb5700 1 --2- 172.21.15.7:0/1825116857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78071e80 0x7fba78151820 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fba680098f0 tx=0x7fba68009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac94c5ce0455faa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:57.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba7cfb3700 1 -- 172.21.15.7:0/1825116857 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68004230 con 0x7fba78071e80 2023-03-03T12:02:57.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba7cfb3700 1 -- 172.21.15.7:0/1825116857 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba68004390 con 0x7fba78071e80 2023-03-03T12:02:57.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba7cfb3700 1 -- 172.21.15.7:0/1825116857 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba680036a0 con 0x7fba78071e80 2023-03-03T12:02:57.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba80219700 1 -- 172.21.15.7:0/1825116857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78071e80 msgr2=0x7fba78151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:57.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba80219700 1 --2- 172.21.15.7:0/1825116857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78071e80 0x7fba78151820 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fba680098f0 tx=0x7fba68009c00 comp rx=0 tx=0).stop 2023-03-03T12:02:57.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba80219700 1 -- 172.21.15.7:0/1825116857 shutdown_connections 2023-03-03T12:02:57.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.375+0000 7fba80219700 1 --2- 172.21.15.7:0/1825116857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78071e80 0x7fba78151820 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:57.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.376+0000 7fba80219700 1 --2- 172.21.15.7:0/1825116857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba78072250 0x7fba7814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:57.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.376+0000 7fba80219700 1 -- 172.21.15.7:0/1825116857 >> 172.21.15.7:0/1825116857 conn(0x7fba780a94d0 msgr2=0x7fba780a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:02:57.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.376+0000 7fba80219700 1 -- 172.21.15.7:0/1825116857 shutdown_connections 2023-03-03T12:02:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.376+0000 7fba80219700 1 -- 172.21.15.7:0/1825116857 wait complete. 2023-03-03T12:02:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.377+0000 7fba80219700 1 Processor -- start 2023-03-03T12:02:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.377+0000 7fba80219700 1 -- start start 2023-03-03T12:02:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.377+0000 7fba80219700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba78071e80 0x7fba7814a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba80219700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78072250 0x7fba7815da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba80219700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba780c1ce0 0x7fba7815dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba80219700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba781eda90 con 0x7fba78072250 2023-03-03T12:02:57.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba80219700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7814abd0 con 0x7fba78071e80 2023-03-03T12:02:57.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba80219700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7815c680 con 0x7fba780c1ce0 2023-03-03T12:02:57.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7d7b4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78072250 0x7fba7815da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:57.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7d7b4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78072250 0x7fba7815da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38878/0 (socket says 172.21.15.7:38878) 2023-03-03T12:02:57.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7d7b4700 1 -- 172.21.15.7:0/951317128 learned_addr learned my addr 172.21.15.7:0/951317128 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:02:57.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7e7b6700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba780c1ce0 0x7fba7815dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:57.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7d7b4700 1 -- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba780c1ce0 msgr2=0x7fba7815dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:57.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7d7b4700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba780c1ce0 0x7fba7815dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:57.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7dfb5700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba78071e80 0x7fba7814a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:57.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7d7b4700 1 -- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba78071e80 msgr2=0x7fba7814a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:57.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7d7b4700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba78071e80 0x7fba7814a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:57.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7d7b4700 1 -- 172.21.15.7:0/951317128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba68009550 con 0x7fba78072250 2023-03-03T12:02:57.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba7d7b4700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78072250 0x7fba7815da70 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fba7400c740 tx=0x7fba7400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:57.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba6e7fc700 1 -- 172.21.15.7:0/951317128 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74004220 con 0x7fba78072250 2023-03-03T12:02:57.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba6e7fc700 1 -- 172.21.15.7:0/951317128 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba74004380 con 0x7fba78072250 2023-03-03T12:02:57.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba80219700 1 -- 172.21.15.7:0/951317128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7806c9a0 con 0x7fba78072250 2023-03-03T12:02:57.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba6e7fc700 1 -- 172.21.15.7:0/951317128 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba740036a0 con 0x7fba78072250 2023-03-03T12:02:57.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.378+0000 7fba80219700 1 -- 172.21.15.7:0/951317128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba78151300 con 0x7fba78072250 2023-03-03T12:02:57.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.379+0000 7fba80219700 1 -- 172.21.15.7:0/951317128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7806cb30 con 0x7fba78072250 2023-03-03T12:02:57.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.380+0000 7fba6e7fc700 1 -- 172.21.15.7:0/951317128 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fba74020070 con 0x7fba78072250 2023-03-03T12:02:57.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.380+0000 7fba6e7fc700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba5c0682f0 0x7fba5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:57.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.380+0000 7fba6e7fc700 1 -- 172.21.15.7:0/951317128 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fba740863b0 con 0x7fba78072250 2023-03-03T12:02:57.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.381+0000 7fba7dfb5700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba5c0682f0 0x7fba5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:57.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.381+0000 7fba7dfb5700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba5c0682f0 0x7fba5c06a7a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fba68009c40 tx=0x7fba68008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:57.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.385+0000 7fba6e7fc700 1 -- 172.21.15.7:0/951317128 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fba740526d0 con 0x7fba78072250 2023-03-03T12:02:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.626+0000 7fba80219700 1 -- 172.21.15.7:0/951317128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba78074bb0 con 0x7fba78072250 2023-03-03T12:02:57.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.626+0000 7fba6e7fc700 1 -- 172.21.15.7:0/951317128 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fba74055cf0 con 0x7fba78072250 2023-03-03T12:02:57.628 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:02:57.628 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:02:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 -- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba5c0682f0 msgr2=0x7fba5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba5c0682f0 0x7fba5c06a7a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fba68009c40 tx=0x7fba68008030 comp rx=0 tx=0).stop 2023-03-03T12:02:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 -- 172.21.15.7:0/951317128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78072250 msgr2=0x7fba7815da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78072250 0x7fba7815da70 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fba7400c740 tx=0x7fba7400cb00 comp rx=0 tx=0).stop 2023-03-03T12:02:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 -- 172.21.15.7:0/951317128 shutdown_connections 2023-03-03T12:02:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba780c1ce0 0x7fba7815dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba78071e80 0x7fba7814a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba5c0682f0 0x7fba5c06a7a0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 --2- 172.21.15.7:0/951317128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba78072250 0x7fba7815da70 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 -- 172.21.15.7:0/951317128 >> 172.21.15.7:0/951317128 conn(0x7fba780a94d0 msgr2=0x7fba7815d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:02:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 -- 172.21.15.7:0/951317128 shutdown_connections 2023-03-03T12:02:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:57.628+0000 7fba80219700 1 -- 172.21.15.7:0/951317128 wait complete. 2023-03-03T12:02:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:02:57.642 INFO:tasks.mds_thrash.fs.[b]:waiting for 37 secs before thrashing 2023-03-03T12:02:59.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:02:59.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.425+0000 7f3d51cb6700 1 Processor -- start 2023-03-03T12:02:59.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.425+0000 7f3d51cb6700 1 -- start start 2023-03-03T12:02:59.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.426+0000 7f3d51cb6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0aae30 0x7f3d4c0beab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.426+0000 7f3d51cb6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3d4c0beff0 0x7f3d4c0c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.426+0000 7f3d51cb6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4c0c3020 con 0x7f3d4c0c49e0 2023-03-03T12:02:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.426+0000 7f3d51cb6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d4c0c4e00 con 0x7f3d4c0beff0 2023-03-03T12:02:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.426+0000 7f3d51cb6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c1552d0 con 0x7f3d4c0aae30 2023-03-03T12:02:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.426+0000 7f3d4affd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3d4c0beff0 0x7f3d4c0c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.426+0000 7f3d4b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0aae30 0x7f3d4c0beab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.426+0000 7f3d4bfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f3d4c0c49e0 0x7f3d4c06e430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:36388/0 (socket says 172.21.15.7:36388) 2023-03-03T12:02:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.426+0000 7f3d4bfff700 1 -- 172.21.15.7:0/682331559 learned_addr learned my addr 172.21.15.7:0/682331559 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:02:59.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.427+0000 7f3d4b7fe700 1 -- 172.21.15.7:0/682331559 >> v2:172.21.15.143:3300/0 conn(0x7f3d4c0beff0 msgr2=0x7f3d4c0c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:02:59.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.427+0000 7f3d4b7fe700 1 --2- 172.21.15.7:0/682331559 >> v2:172.21.15.143:3300/0 conn(0x7f3d4c0beff0 0x7f3d4c0c44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.427+0000 7f3d4b7fe700 1 -- 172.21.15.7:0/682331559 >> v1:172.21.15.143:6789/0 conn(0x7f3d4c0c49e0 legacy=0x7f3d4c06e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.427+0000 7f3d4a7fc700 1 -- 172.21.15.7:0/682331559 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536434695 0 0) 0x7f3d4c0c3020 con 0x7f3d4c0c49e0 2023-03-03T12:02:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.427+0000 7f3d4b7fe700 1 -- 172.21.15.7:0/682331559 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c1546e0 con 0x7f3d4c0aae30 2023-03-03T12:02:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.427+0000 7f3d4affd700 1 --2- 172.21.15.7:0/682331559 >> v2:172.21.15.143:3300/0 conn(0x7f3d4c0beff0 0x7f3d4c0c44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:02:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.427+0000 7f3d4b7fe700 1 --2- 172.21.15.7:0/682331559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0aae30 0x7f3d4c0beab0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009840 tx=0x7f3d3c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f59a49dd72698097 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.427+0000 7f3d4a7fc700 1 -- 172.21.15.7:0/682331559 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c011830 con 0x7f3d4c0aae30 2023-03-03T12:02:59.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.427+0000 7f3d4a7fc700 1 -- 172.21.15.7:0/682331559 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3c011990 con 0x7f3d4c0aae30 2023-03-03T12:02:59.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.427+0000 7f3d4a7fc700 1 -- 172.21.15.7:0/682331559 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c011d20 con 0x7f3d4c0aae30 2023-03-03T12:02:59.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.428+0000 7f3d51cb6700 1 -- 172.21.15.7:0/682331559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0aae30 msgr2=0x7f3d4c0beab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:59.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.428+0000 7f3d51cb6700 1 --2- 172.21.15.7:0/682331559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0aae30 0x7f3d4c0beab0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009840 tx=0x7f3d3c009b50 comp rx=0 tx=0).stop 2023-03-03T12:02:59.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.428+0000 7f3d51cb6700 1 -- 172.21.15.7:0/682331559 shutdown_connections 2023-03-03T12:02:59.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.428+0000 7f3d51cb6700 1 --2- 172.21.15.7:0/682331559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0aae30 0x7f3d4c0beab0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:59.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.428+0000 7f3d51cb6700 1 --2- 172.21.15.7:0/682331559 >> v2:172.21.15.143:3300/0 conn(0x7f3d4c0beff0 0x7f3d4c0c44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:59.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.428+0000 7f3d51cb6700 1 -- 172.21.15.7:0/682331559 >> 172.21.15.7:0/682331559 conn(0x7f3d4c0c8510 msgr2=0x7f3d4c0c8910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:02:59.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.428+0000 7f3d51cb6700 1 -- 172.21.15.7:0/682331559 shutdown_connections 2023-03-03T12:02:59.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.428+0000 7f3d51cb6700 1 -- 172.21.15.7:0/682331559 wait complete. 2023-03-03T12:02:59.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.429+0000 7f3d51cb6700 1 Processor -- start 2023-03-03T12:02:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.429+0000 7f3d51cb6700 1 -- start start 2023-03-03T12:02:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d51cb6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d4c0aae30 0x7f3d4c155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d51cb6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0beff0 0x7f3d4c06b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:59.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d51cb6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4c0c49e0 0x7f3d4c156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d51cb6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c156b50 con 0x7f3d4c0aae30 2023-03-03T12:02:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d51cb6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c0c62f0 con 0x7f3d4c0c49e0 2023-03-03T12:02:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d51cb6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c156050 con 0x7f3d4c0beff0 2023-03-03T12:02:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d4c0aae30 0x7f3d4c155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d4c0aae30 0x7f3d4c155b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38890/0 (socket says 172.21.15.7:38890) 2023-03-03T12:02:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4b7fe700 1 -- 172.21.15.7:0/2084827990 learned_addr learned my addr 172.21.15.7:0/2084827990 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:02:59.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4bfff700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4c0c49e0 0x7f3d4c156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4affd700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0beff0 0x7f3d4c06b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4b7fe700 1 -- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0beff0 msgr2=0x7f3d4c06b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4b7fe700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0beff0 0x7f3d4c06b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4b7fe700 1 -- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4c0c49e0 msgr2=0x7f3d4c156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4b7fe700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4c0c49e0 0x7f3d4c156fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4b7fe700 1 -- 172.21.15.7:0/2084827990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d400094a0 con 0x7f3d4c0aae30 2023-03-03T12:02:59.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4affd700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0beff0 0x7f3d4c06b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:02:59.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.430+0000 7f3d4b7fe700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d4c0aae30 0x7f3d4c155b10 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0090b0 tx=0x7f3d3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:59.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.431+0000 7f3d33fff700 1 -- 172.21.15.7:0/2084827990 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c0038a0 con 0x7f3d4c0aae30 2023-03-03T12:02:59.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.431+0000 7f3d33fff700 1 -- 172.21.15.7:0/2084827990 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3c01f070 con 0x7f3d4c0aae30 2023-03-03T12:02:59.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.431+0000 7f3d51cb6700 1 -- 172.21.15.7:0/2084827990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c0094a0 con 0x7f3d4c0aae30 2023-03-03T12:02:59.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.431+0000 7f3d33fff700 1 -- 172.21.15.7:0/2084827990 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c003c40 con 0x7f3d4c0aae30 2023-03-03T12:02:59.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.431+0000 7f3d51cb6700 1 -- 172.21.15.7:0/2084827990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c154c80 con 0x7f3d4c0aae30 2023-03-03T12:02:59.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.432+0000 7f3d51cb6700 1 -- 172.21.15.7:0/2084827990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c1549a0 con 0x7f3d4c0aae30 2023-03-03T12:02:59.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.432+0000 7f3d33fff700 1 -- 172.21.15.7:0/2084827990 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d3c00f050 con 0x7f3d4c0aae30 2023-03-03T12:02:59.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.433+0000 7f3d33fff700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d380682f0 0x7f3d3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:02:59.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.433+0000 7f3d33fff700 1 -- 172.21.15.7:0/2084827990 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3d3c021070 con 0x7f3d4c0aae30 2023-03-03T12:02:59.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.433+0000 7f3d4affd700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d380682f0 0x7f3d3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:02:59.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.433+0000 7f3d4affd700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d380682f0 0x7f3d3806a7a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0c2f40 tx=0x7f3d40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:02:59.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.437+0000 7f3d33fff700 1 -- 172.21.15.7:0/2084827990 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d3c053dd0 con 0x7f3d4c0aae30 2023-03-03T12:02:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.675+0000 7f3d51cb6700 1 -- 172.21.15.7:0/2084827990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d4c156e30 con 0x7f3d4c0aae30 2023-03-03T12:02:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.676+0000 7f3d33fff700 1 -- 172.21.15.7:0/2084827990 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3d3c015090 con 0x7f3d4c0aae30 2023-03-03T12:02:59.678 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:02:59.678 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:02:59.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 -- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d380682f0 msgr2=0x7f3d3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:59.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d380682f0 0x7f3d3806a7a0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0c2f40 tx=0x7f3d40008040 comp rx=0 tx=0).stop 2023-03-03T12:02:59.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 -- 172.21.15.7:0/2084827990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d4c0aae30 msgr2=0x7f3d4c155b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:02:59.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d4c0aae30 0x7f3d4c155b10 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0090b0 tx=0x7f3d3c004750 comp rx=0 tx=0).stop 2023-03-03T12:02:59.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 -- 172.21.15.7:0/2084827990 shutdown_connections 2023-03-03T12:02:59.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4c0c49e0 0x7f3d4c156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:59.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d380682f0 0x7f3d3806a7a0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d4c0aae30 0x7f3d4c155b10 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 --2- 172.21.15.7:0/2084827990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4c0beff0 0x7f3d4c06b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:02:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 -- 172.21.15.7:0/2084827990 >> 172.21.15.7:0/2084827990 conn(0x7f3d4c0c8510 msgr2=0x7f3d4c0be680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:02:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 -- 172.21.15.7:0/2084827990 shutdown_connections 2023-03-03T12:02:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:02:59.678+0000 7f3d51cb6700 1 -- 172.21.15.7:0/2084827990 wait complete. 2023-03-03T12:02:59.682 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:02:59.691 INFO:tasks.mds_thrash.fs.[a]:waiting for 81 secs before thrashing 2023-03-03T12:03:22.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:22.714 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:34.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:03:34.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.781+0000 7f0dbf7de700 1 Processor -- start 2023-03-03T12:03:34.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.782+0000 7f0dbf7de700 1 -- start start 2023-03-03T12:03:34.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.782+0000 7f0dbf7de700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0db813a0f0 0x7f0db80c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:34.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.782+0000 7f0dbf7de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db80c1da0 0x7f0db806d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:34.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.782+0000 7f0dbf7de700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db806de90 con 0x7f0db81586c0 2023-03-03T12:03:34.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.782+0000 7f0dbf7de700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db8154840 con 0x7f0db813a0f0 2023-03-03T12:03:34.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.782+0000 7f0dbf7de700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8154560 con 0x7f0db80c1da0 2023-03-03T12:03:34.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0dbcd79700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0db81586c0 0x7f0db806d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:32892/0 (socket says 172.21.15.7:32892) 2023-03-03T12:03:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0dbcd79700 1 -- 172.21.15.7:0/1961070572 learned_addr learned my addr 172.21.15.7:0/1961070572 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:03:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0dbdd7b700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db80c1da0 0x7f0db806d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0dbd57a700 1 --2- 172.21.15.7:0/1961070572 >> v2:172.21.15.143:3300/0 conn(0x7f0db813a0f0 0x7f0db80c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821732899 0 0) 0x7f0da8001110 con 0x7f0db81586c0 2023-03-03T12:03:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da40035b0 con 0x7f0db81586c0 2023-03-03T12:03:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1702379270 0 0) 0x7f0da40035b0 con 0x7f0db81586c0 2023-03-03T12:03:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db80c1da0 msgr2=0x7f0db806d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:03:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db80c1da0 0x7f0db806d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 >> v2:172.21.15.143:3300/0 conn(0x7f0db813a0f0 msgr2=0x7f0db80c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:03:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 --2- 172.21.15.7:0/1961070572 >> v2:172.21.15.143:3300/0 conn(0x7f0db813a0f0 0x7f0db80c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db8157be0 con 0x7f0db81586c0 2023-03-03T12:03:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0da8001800 con 0x7f0db81586c0 2023-03-03T12:03:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 >> v1:172.21.15.7:6789/0 conn(0x7f0db81586c0 legacy=0x7f0db806d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0da4003a70 0x7f0da4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db80c1da0 0x7f0da4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da4004d90 0x7f0da4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8154840 con 0x7f0da4003a70 2023-03-03T12:03:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8154560 con 0x7f0da4004d90 2023-03-03T12:03:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da400bb40 con 0x7f0db80c1da0 2023-03-03T12:03:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.783+0000 7f0dbd57a700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0da4003a70 0x7f0da4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0dbcd79700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db80c1da0 0x7f0da4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0dbdd7b700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da4004d90 0x7f0da4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0dbd57a700 1 -- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db80c1da0 msgr2=0x7f0da4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:03:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0dbd57a700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db80c1da0 0x7f0da4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0dbd57a700 1 -- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da4004d90 msgr2=0x7f0da4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:03:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0dbd57a700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0da4004d90 0x7f0da4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0dbd57a700 1 -- 172.21.15.7:0/1961070572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da40035b0 con 0x7f0da4003a70 2023-03-03T12:03:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0dbd57a700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0da4003a70 0x7f0da4003e80 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009b60 tx=0x7f0da4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622cda6154babee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:03:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db400e6c0 con 0x7f0da4003a70 2023-03-03T12:03:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db400e820 con 0x7f0da4003a70 2023-03-03T12:03:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.784+0000 7f0daffff700 1 -- 172.21.15.7:0/1961070572 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db400eb90 con 0x7f0da4003a70 2023-03-03T12:03:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.785+0000 7f0dbf7de700 1 -- 172.21.15.7:0/1961070572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0da4003a70 msgr2=0x7f0da4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.785+0000 7f0dbf7de700 1 --2- 172.21.15.7:0/1961070572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0da4003a70 0x7f0da4003e80 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009b60 tx=0x7f0da4005880 comp rx=0 tx=0).stop 2023-03-03T12:03:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.785+0000 7f0dbdd7b700 1 -- 172.21.15.7:0/1961070572 reap_dead start 2023-03-03T12:03:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.785+0000 7f0dbf7de700 1 -- 172.21.15.7:0/1961070572 shutdown_connections 2023-03-03T12:03:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.785+0000 7f0dbf7de700 1 -- 172.21.15.7:0/1961070572 >> 172.21.15.7:0/1961070572 conn(0x7f0db806c110 msgr2=0x7f0db806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:03:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.785+0000 7f0dbf7de700 1 -- 172.21.15.7:0/1961070572 shutdown_connections 2023-03-03T12:03:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.785+0000 7f0dbf7de700 1 -- 172.21.15.7:0/1961070572 wait complete. 2023-03-03T12:03:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.786+0000 7f0dbf7de700 1 Processor -- start 2023-03-03T12:03:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.786+0000 7f0dbf7de700 1 -- start start 2023-03-03T12:03:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbf7de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da4003a70 0x7f0db8134490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbf7de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db81ed3a0 0x7f0db8151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbf7de700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db81ed810 0x7f0db8151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbf7de700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8152000 con 0x7f0db81ed810 2023-03-03T12:03:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbf7de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80ab570 con 0x7f0db81ed3a0 2023-03-03T12:03:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbf7de700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db813a750 con 0x7f0da4003a70 2023-03-03T12:03:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbdd7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db81ed810 0x7f0db8151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbdd7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db81ed810 0x7f0db8151ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38926/0 (socket says 172.21.15.7:38926) 2023-03-03T12:03:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbdd7b700 1 -- 172.21.15.7:0/3918692943 learned_addr learned my addr 172.21.15.7:0/3918692943 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:03:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbd57a700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da4003a70 0x7f0db8134490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbcd79700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db81ed3a0 0x7f0db8151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbdd7b700 1 -- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da4003a70 msgr2=0x7f0db8134490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbdd7b700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da4003a70 0x7f0db8134490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbdd7b700 1 -- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db81ed3a0 msgr2=0x7f0db8151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbdd7b700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db81ed3a0 0x7f0db8151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.787+0000 7f0dbdd7b700 1 -- 172.21.15.7:0/3918692943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db81581a0 con 0x7f0db81ed810 2023-03-03T12:03:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.788+0000 7f0dbdd7b700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db81ed810 0x7f0db8151ac0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f0db80c1890 tx=0x7f0db0011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:03:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.788+0000 7f0dadffb700 1 -- 172.21.15.7:0/3918692943 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db000e040 con 0x7f0db81ed810 2023-03-03T12:03:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.788+0000 7f0dadffb700 1 -- 172.21.15.7:0/3918692943 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db000f040 con 0x7f0db81ed810 2023-03-03T12:03:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.788+0000 7f0dbf7de700 1 -- 172.21.15.7:0/3918692943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db8157ec0 con 0x7f0db81ed810 2023-03-03T12:03:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.788+0000 7f0dadffb700 1 -- 172.21.15.7:0/3918692943 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db000fd50 con 0x7f0db81ed810 2023-03-03T12:03:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.788+0000 7f0dbf7de700 1 -- 172.21.15.7:0/3918692943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db806e1a0 con 0x7f0db81ed810 2023-03-03T12:03:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.789+0000 7f0dbf7de700 1 -- 172.21.15.7:0/3918692943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db8074bb0 con 0x7f0db81ed810 2023-03-03T12:03:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.789+0000 7f0dadffb700 1 -- 172.21.15.7:0/3918692943 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0db0015430 con 0x7f0db81ed810 2023-03-03T12:03:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.790+0000 7f0dadffb700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d9c0682f0 0x7f0d9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:03:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.790+0000 7f0dadffb700 1 -- 172.21.15.7:0/3918692943 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0db0086f10 con 0x7f0db81ed810 2023-03-03T12:03:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.790+0000 7f0dbd57a700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d9c0682f0 0x7f0d9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:03:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.791+0000 7f0dbd57a700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d9c0682f0 0x7f0d9c06a7a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0db40090b0 tx=0x7f0db4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:03:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:34.794+0000 7f0dadffb700 1 -- 172.21.15.7:0/3918692943 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0db0053250 con 0x7f0db81ed810 2023-03-03T12:03:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.035+0000 7f0dbf7de700 1 -- 172.21.15.7:0/3918692943 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0db8138880 con 0x7f0db81ed810 2023-03-03T12:03:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.036+0000 7f0dadffb700 1 -- 172.21.15.7:0/3918692943 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0db0056870 con 0x7f0db81ed810 2023-03-03T12:03:35.038 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:03:35.038 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:03:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 -- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d9c0682f0 msgr2=0x7f0d9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d9c0682f0 0x7f0d9c06a7a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0db40090b0 tx=0x7f0db4008030 comp rx=0 tx=0).stop 2023-03-03T12:03:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 -- 172.21.15.7:0/3918692943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db81ed810 msgr2=0x7f0db8151ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:03:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db81ed810 0x7f0db8151ac0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f0db80c1890 tx=0x7f0db0011f80 comp rx=0 tx=0).stop 2023-03-03T12:03:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 -- 172.21.15.7:0/3918692943 shutdown_connections 2023-03-03T12:03:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d9c0682f0 0x7f0d9c06a7a0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db81ed810 0x7f0db8151ac0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0da4003a70 0x7f0db8134490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 --2- 172.21.15.7:0/3918692943 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db81ed3a0 0x7f0db8151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:03:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 -- 172.21.15.7:0/3918692943 >> 172.21.15.7:0/3918692943 conn(0x7f0db806c110 msgr2=0x7f0db80c3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:03:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 -- 172.21.15.7:0/3918692943 shutdown_connections 2023-03-03T12:03:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:03:35.038+0000 7f0dbf7de700 1 -- 172.21.15.7:0/3918692943 wait complete. 2023-03-03T12:03:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:03:35.052 INFO:tasks.mds_thrash.fs.[b]:waiting for 91 secs before thrashing 2023-03-03T12:03:52.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:03:52.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:20.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:04:20.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.831+0000 7f4101465700 1 Processor -- start 2023-03-03T12:04:20.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.831+0000 7f4101465700 1 -- start start 2023-03-03T12:04:20.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.831+0000 7f4101465700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40fc0caa30 0x7f40fc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:04:20.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.831+0000 7f4101465700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40fc154ff0 0x7f40fc0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:04:20.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.831+0000 7f4101465700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40fc0c7310 con 0x7f40fc0c00e0 2023-03-03T12:04:20.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.831+0000 7f4101465700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40fc155430 con 0x7f40fc154ff0 2023-03-03T12:04:20.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.831+0000 7f4101465700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40fc0bfd10 con 0x7f40fc0caa30 2023-03-03T12:04:20.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40fb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40fc154ff0 0x7f40fc0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:04:20.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40fb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40fc154ff0 0x7f40fc0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38976/0 (socket says 172.21.15.7:38976) 2023-03-03T12:04:20.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40fb7fe700 1 -- 172.21.15.7:0/3682272936 learned_addr learned my addr 172.21.15.7:0/3682272936 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:04:20.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40faffd700 1 --2- 172.21.15.7:0/3682272936 >> v2:172.21.15.143:3300/0 conn(0x7f40fc0caa30 0x7f40fc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:04:20.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40fb7fe700 1 -- 172.21.15.7:0/3682272936 >> v2:172.21.15.143:3300/0 conn(0x7f40fc0caa30 msgr2=0x7f40fc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:04:20.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40fb7fe700 1 --2- 172.21.15.7:0/3682272936 >> v2:172.21.15.143:3300/0 conn(0x7f40fc0caa30 0x7f40fc0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:04:20.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40fb7fe700 1 -- 172.21.15.7:0/3682272936 >> v1:172.21.15.143:6789/0 conn(0x7f40fc0c00e0 legacy=0x7f40fc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:04:20.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40fb7fe700 1 -- 172.21.15.7:0/3682272936 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40fc0bfa30 con 0x7f40fc154ff0 2023-03-03T12:04:20.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40faffd700 1 --2- 172.21.15.7:0/3682272936 >> v2:172.21.15.143:3300/0 conn(0x7f40fc0caa30 0x7f40fc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:04:20.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40fb7fe700 1 --2- 172.21.15.7:0/3682272936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc154ff0 0x7f40fc0cae00 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0c81b0 tx=0x7f40f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3ba632db4be95e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:04:20.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40f9ffb700 1 -- 172.21.15.7:0/3682272936 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f400e070 con 0x7f40fc154ff0 2023-03-03T12:04:20.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40f9ffb700 1 -- 172.21.15.7:0/3682272936 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f400b9c0 con 0x7f40fc154ff0 2023-03-03T12:04:20.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.832+0000 7f40f9ffb700 1 -- 172.21.15.7:0/3682272936 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f400bd70 con 0x7f40fc154ff0 2023-03-03T12:04:20.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.833+0000 7f4101465700 1 -- 172.21.15.7:0/3682272936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc154ff0 msgr2=0x7f40fc0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:04:20.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.833+0000 7f4101465700 1 --2- 172.21.15.7:0/3682272936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc154ff0 0x7f40fc0cae00 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0c81b0 tx=0x7f40f400b330 comp rx=0 tx=0).stop 2023-03-03T12:04:20.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.833+0000 7f4101465700 1 -- 172.21.15.7:0/3682272936 shutdown_connections 2023-03-03T12:04:20.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.833+0000 7f4101465700 1 --2- 172.21.15.7:0/3682272936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc154ff0 0x7f40fc0cae00 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:04:20.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.833+0000 7f4101465700 1 --2- 172.21.15.7:0/3682272936 >> v2:172.21.15.143:3300/0 conn(0x7f40fc0caa30 0x7f40fc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:04:20.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.833+0000 7f4101465700 1 -- 172.21.15.7:0/3682272936 >> 172.21.15.7:0/3682272936 conn(0x7f40fc066e40 msgr2=0x7f40fc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:04:20.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.833+0000 7f4101465700 1 -- 172.21.15.7:0/3682272936 shutdown_connections 2023-03-03T12:04:20.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.833+0000 7f4101465700 1 -- 172.21.15.7:0/3682272936 wait complete. 2023-03-03T12:04:20.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.834+0000 7f4101465700 1 Processor -- start 2023-03-03T12:04:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.834+0000 7f4101465700 1 -- start start 2023-03-03T12:04:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f4101465700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc0c00e0 0x7f40fc1e1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:04:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f4101465700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40fc0caa30 0x7f40fc0c2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:04:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f4101465700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40fc154ff0 0x7f40fc0c2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:04:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f4101465700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1eadc0 con 0x7f40fc0c00e0 2023-03-03T12:04:20.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f4101465700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1e9e80 con 0x7f40fc0caa30 2023-03-03T12:04:20.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f4101465700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1ea140 con 0x7f40fc154ff0 2023-03-03T12:04:20.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f40faffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc0c00e0 0x7f40fc1e1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:04:20.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f40faffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc0c00e0 0x7f40fc1e1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38978/0 (socket says 172.21.15.7:38978) 2023-03-03T12:04:20.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f40faffd700 1 -- 172.21.15.7:0/814429843 learned_addr learned my addr 172.21.15.7:0/814429843 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:04:20.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f40fb7fe700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40fc154ff0 0x7f40fc0c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:04:20.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.835+0000 7f40fa7fc700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40fc0caa30 0x7f40fc0c2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:04:20.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f40faffd700 1 -- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40fc154ff0 msgr2=0x7f40fc0c2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:04:20.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f40faffd700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40fc154ff0 0x7f40fc0c2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:04:20.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f40faffd700 1 -- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40fc0caa30 msgr2=0x7f40fc0c2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:04:20.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f40faffd700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40fc0caa30 0x7f40fc0c2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:04:20.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f40faffd700 1 -- 172.21.15.7:0/814429843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4009b30 con 0x7f40fc0c00e0 2023-03-03T12:04:20.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f40faffd700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc0c00e0 0x7f40fc1e1070 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00e810 tx=0x7f40ec00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:04:20.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f40e37fe700 1 -- 172.21.15.7:0/814429843 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec013610 con 0x7f40fc0c00e0 2023-03-03T12:04:20.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f40e37fe700 1 -- 172.21.15.7:0/814429843 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec013770 con 0x7f40fc0c00e0 2023-03-03T12:04:20.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f40e37fe700 1 -- 172.21.15.7:0/814429843 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec013b20 con 0x7f40fc0c00e0 2023-03-03T12:04:20.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f4101465700 1 -- 172.21.15.7:0/814429843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40fc1547f0 con 0x7f40fc0c00e0 2023-03-03T12:04:20.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.836+0000 7f4101465700 1 -- 172.21.15.7:0/814429843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40fc154510 con 0x7f40fc0c00e0 2023-03-03T12:04:20.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.837+0000 7f4101465700 1 -- 172.21.15.7:0/814429843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40fc154ad0 con 0x7f40fc0c00e0 2023-03-03T12:04:20.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.837+0000 7f40e37fe700 1 -- 172.21.15.7:0/814429843 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f40ec023070 con 0x7f40fc0c00e0 2023-03-03T12:04:20.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.838+0000 7f40e37fe700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40e80682f0 0x7f40e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:04:20.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.838+0000 7f40e37fe700 1 -- 172.21.15.7:0/814429843 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f40ec085c90 con 0x7f40fc0c00e0 2023-03-03T12:04:20.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.838+0000 7f40fa7fc700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40e80682f0 0x7f40e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:04:20.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.839+0000 7f40fa7fc700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40e80682f0 0x7f40e806a7a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0c2010 tx=0x7f40f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:04:20.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:20.842+0000 7f40e37fe700 1 -- 172.21.15.7:0/814429843 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f40ec052030 con 0x7f40fc0c00e0 2023-03-03T12:04:21.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.078+0000 7f4101465700 1 -- 172.21.15.7:0/814429843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40fc074bb0 con 0x7f40fc0c00e0 2023-03-03T12:04:21.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.078+0000 7f40e37fe700 1 -- 172.21.15.7:0/814429843 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f40ec055650 con 0x7f40fc0c00e0 2023-03-03T12:04:21.081 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:04:21.081 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:04:21.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.080+0000 7f4101465700 1 -- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40e80682f0 msgr2=0x7f40e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:04:21.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.080+0000 7f4101465700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40e80682f0 0x7f40e806a7a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0c2010 tx=0x7f40f0009040 comp rx=0 tx=0).stop 2023-03-03T12:04:21.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.080+0000 7f4101465700 1 -- 172.21.15.7:0/814429843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc0c00e0 msgr2=0x7f40fc1e1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:04:21.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.080+0000 7f4101465700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc0c00e0 0x7f40fc1e1070 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00e810 tx=0x7f40ec00eb20 comp rx=0 tx=0).stop 2023-03-03T12:04:21.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.080+0000 7f4101465700 1 -- 172.21.15.7:0/814429843 shutdown_connections 2023-03-03T12:04:21.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.080+0000 7f4101465700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40fc154ff0 0x7f40fc0c2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:04:21.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.080+0000 7f4101465700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40e80682f0 0x7f40e806a7a0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:04:21.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.080+0000 7f4101465700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40fc0c00e0 0x7f40fc1e1070 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:04:21.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.080+0000 7f4101465700 1 --2- 172.21.15.7:0/814429843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40fc0caa30 0x7f40fc0c2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:04:21.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.080+0000 7f4101465700 1 -- 172.21.15.7:0/814429843 >> 172.21.15.7:0/814429843 conn(0x7f40fc066e40 msgr2=0x7f40fc0c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:04:21.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.081+0000 7f4101465700 1 -- 172.21.15.7:0/814429843 shutdown_connections 2023-03-03T12:04:21.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:04:21.081+0000 7f4101465700 1 -- 172.21.15.7:0/814429843 wait complete. 2023-03-03T12:04:21.084 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:04:21.094 INFO:tasks.mds_thrash.fs.[a]:waiting for 93 secs before thrashing 2023-03-03T12:04:22.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:22.996 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:53.141 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:04:53.144 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:06.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:05:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.191+0000 7fa0eada1700 1 Processor -- start 2023-03-03T12:05:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.191+0000 7fa0eada1700 1 -- start start 2023-03-03T12:05:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.192+0000 7fa0eada1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0e40a6370 0x7fa0e406a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.192+0000 7fa0eada1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa0e40c5cb0 0x7fa0e415cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.192+0000 7fa0eada1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e41558f0 con 0x7fa0e406a2b0 2023-03-03T12:05:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.192+0000 7fa0eada1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e406bab0 con 0x7fa0e40a6370 2023-03-03T12:05:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.192+0000 7fa0eada1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e40a6180 con 0x7fa0e40c5cb0 2023-03-03T12:05:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.192+0000 7fa0e3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0e40a6370 0x7fa0e406a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.192+0000 7fa0e8b3d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa0e406a2b0 0x7fa0e4155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33002/0 (socket says 172.21.15.7:33002) 2023-03-03T12:05:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.192+0000 7fa0e8b3d700 1 -- 172.21.15.7:0/1251758565 learned_addr learned my addr 172.21.15.7:0/1251758565 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:05:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.192+0000 7fa0e3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0e40a6370 0x7fa0e406a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39024/0 (socket says 172.21.15.7:39024) 2023-03-03T12:05:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.192+0000 7fa0e933e700 1 --2- 172.21.15.7:0/1251758565 >> v2:172.21.15.143:3300/0 conn(0x7fa0e40c5cb0 0x7fa0e415cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0e3fff700 1 -- 172.21.15.7:0/1251758565 >> v2:172.21.15.143:3300/0 conn(0x7fa0e40c5cb0 msgr2=0x7fa0e415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:05:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0e3fff700 1 --2- 172.21.15.7:0/1251758565 >> v2:172.21.15.143:3300/0 conn(0x7fa0e40c5cb0 0x7fa0e415cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0e3fff700 1 -- 172.21.15.7:0/1251758565 >> v1:172.21.15.7:6789/0 conn(0x7fa0e406a2b0 legacy=0x7fa0e4155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0e3fff700 1 -- 172.21.15.7:0/1251758565 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e40a5ea0 con 0x7fa0e40a6370 2023-03-03T12:05:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0e37fe700 1 -- 172.21.15.7:0/1251758565 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533711433 0 0) 0x7fa0e41558f0 con 0x7fa0e406a2b0 2023-03-03T12:05:06.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0e3fff700 1 --2- 172.21.15.7:0/1251758565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e40a6370 0x7fa0e406a680 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40c7be0 tx=0x7fa0d0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af86dfba417b9b00 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:06.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0e37fe700 1 -- 172.21.15.7:0/1251758565 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0011830 con 0x7fa0e40a6370 2023-03-03T12:05:06.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0e37fe700 1 -- 172.21.15.7:0/1251758565 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d0011990 con 0x7fa0e40a6370 2023-03-03T12:05:06.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0e37fe700 1 -- 172.21.15.7:0/1251758565 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0011d20 con 0x7fa0e40a6370 2023-03-03T12:05:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0eada1700 1 -- 172.21.15.7:0/1251758565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e40a6370 msgr2=0x7fa0e406a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.193+0000 7fa0eada1700 1 --2- 172.21.15.7:0/1251758565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e40a6370 0x7fa0e406a680 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40c7be0 tx=0x7fa0d0009b20 comp rx=0 tx=0).stop 2023-03-03T12:05:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.194+0000 7fa0eada1700 1 -- 172.21.15.7:0/1251758565 shutdown_connections 2023-03-03T12:05:06.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.194+0000 7fa0eada1700 1 --2- 172.21.15.7:0/1251758565 >> v2:172.21.15.143:3300/0 conn(0x7fa0e40c5cb0 0x7fa0e415cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.194+0000 7fa0eada1700 1 --2- 172.21.15.7:0/1251758565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e40a6370 0x7fa0e406a680 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.194+0000 7fa0eada1700 1 -- 172.21.15.7:0/1251758565 >> 172.21.15.7:0/1251758565 conn(0x7fa0e40a9430 msgr2=0x7fa0e40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:05:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.194+0000 7fa0eada1700 1 -- 172.21.15.7:0/1251758565 shutdown_connections 2023-03-03T12:05:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.194+0000 7fa0eada1700 1 -- 172.21.15.7:0/1251758565 wait complete. 2023-03-03T12:05:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.195+0000 7fa0eada1700 1 Processor -- start 2023-03-03T12:05:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.195+0000 7fa0eada1700 1 -- start start 2023-03-03T12:05:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0eada1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e406a2b0 0x7fa0e41dc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:06.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0eada1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e40a6370 0x7fa0e41eb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0eada1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c5cb0 0x7fa0e41e9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0eada1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e41ebee0 con 0x7fa0e406a2b0 2023-03-03T12:05:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0eada1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4070110 con 0x7fa0e40a6370 2023-03-03T12:05:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0eada1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e41ebc00 con 0x7fa0e40c5cb0 2023-03-03T12:05:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e8b3d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e406a2b0 0x7fa0e41dc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e8b3d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e406a2b0 0x7fa0e41dc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39028/0 (socket says 172.21.15.7:39028) 2023-03-03T12:05:06.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e8b3d700 1 -- 172.21.15.7:0/2183495972 learned_addr learned my addr 172.21.15.7:0/2183495972 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:05:06.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e3fff700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e40a6370 0x7fa0e41eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:06.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e933e700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c5cb0 0x7fa0e41e9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:06.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e8b3d700 1 -- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c5cb0 msgr2=0x7fa0e41e9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:06.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e8b3d700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c5cb0 0x7fa0e41e9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:06.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e8b3d700 1 -- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e40a6370 msgr2=0x7fa0e41eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:06.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e8b3d700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e40a6370 0x7fa0e41eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:06.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e8b3d700 1 -- 172.21.15.7:0/2183495972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d00094a0 con 0x7fa0e406a2b0 2023-03-03T12:05:06.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e3fff700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e40a6370 0x7fa0e41eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:05:06.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e933e700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c5cb0 0x7fa0e41e9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:05:06.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e8b3d700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e406a2b0 0x7fa0e41dc2f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800c750 tx=0x7fa0d800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:06.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e17fa700 1 -- 172.21.15.7:0/2183495972 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8004c80 con 0x7fa0e406a2b0 2023-03-03T12:05:06.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.196+0000 7fa0e17fa700 1 -- 172.21.15.7:0/2183495972 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d8004e00 con 0x7fa0e406a2b0 2023-03-03T12:05:06.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.197+0000 7fa0eada1700 1 -- 172.21.15.7:0/2183495972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e4153160 con 0x7fa0e406a2b0 2023-03-03T12:05:06.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.197+0000 7fa0e17fa700 1 -- 172.21.15.7:0/2183495972 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8007310 con 0x7fa0e406a2b0 2023-03-03T12:05:06.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.197+0000 7fa0eada1700 1 -- 172.21.15.7:0/2183495972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e4152e80 con 0x7fa0e406a2b0 2023-03-03T12:05:06.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.197+0000 7fa0eada1700 1 -- 172.21.15.7:0/2183495972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e40c1500 con 0x7fa0e406a2b0 2023-03-03T12:05:06.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.198+0000 7fa0e17fa700 1 -- 172.21.15.7:0/2183495972 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa0d801a020 con 0x7fa0e406a2b0 2023-03-03T12:05:06.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.198+0000 7fa0e17fa700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 0x7fa0c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:06.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.199+0000 7fa0e17fa700 1 -- 172.21.15.7:0/2183495972 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa0d8085de0 con 0x7fa0e406a2b0 2023-03-03T12:05:06.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.199+0000 7fa0e3fff700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 0x7fa0c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:06.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.199+0000 7fa0e3fff700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 0x7fa0c806a7a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00090b0 tx=0x7fa0d0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:06.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.203+0000 7fa0e17fa700 1 -- 172.21.15.7:0/2183495972 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa0d8052120 con 0x7fa0e406a2b0 2023-03-03T12:05:06.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.439+0000 7fa0eada1700 1 -- 172.21.15.7:0/2183495972 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0e41e9040 con 0x7fa0e406a2b0 2023-03-03T12:05:06.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.439+0000 7fa0e17fa700 1 -- 172.21.15.7:0/2183495972 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa0d8055740 con 0x7fa0e406a2b0 2023-03-03T12:05:06.442 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:05:06.442 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:05:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 -- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 msgr2=0x7fa0c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:06.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 0x7fa0c806a7a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00090b0 tx=0x7fa0d0008030 comp rx=0 tx=0).stop 2023-03-03T12:05:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 -- 172.21.15.7:0/2183495972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e406a2b0 msgr2=0x7fa0e41dc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e406a2b0 0x7fa0e41dc2f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800c750 tx=0x7fa0d800cb10 comp rx=0 tx=0).stop 2023-03-03T12:05:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 -- 172.21.15.7:0/2183495972 shutdown_connections 2023-03-03T12:05:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c5cb0 0x7fa0e41e9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 0x7fa0c806a7a0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e406a2b0 0x7fa0e41dc2f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:06.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 --2- 172.21.15.7:0/2183495972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e40a6370 0x7fa0e41eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 -- 172.21.15.7:0/2183495972 >> 172.21.15.7:0/2183495972 conn(0x7fa0e40a9430 msgr2=0x7fa0e406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:05:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 -- 172.21.15.7:0/2183495972 shutdown_connections 2023-03-03T12:05:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:06.442+0000 7fa0eada1700 1 -- 172.21.15.7:0/2183495972 wait complete. 2023-03-03T12:05:06.445 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:05:06.455 INFO:tasks.mds_thrash.fs.[b]:waiting for 26 secs before thrashing 2023-03-03T12:05:23.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:23.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:32.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:05:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.593+0000 7f62ba60b700 1 Processor -- start 2023-03-03T12:05:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.593+0000 7f62ba60b700 1 -- start start 2023-03-03T12:05:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.594+0000 7f62ba60b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f62b4152e30 0x7f62b40c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.594+0000 7f62ba60b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b40beab0 0x7f62b40c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.594+0000 7f62ba60b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b4156910 con 0x7f62b4155d80 2023-03-03T12:05:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.594+0000 7f62ba60b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b41523b0 con 0x7f62b40beab0 2023-03-03T12:05:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.594+0000 7f62ba60b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b4072010 con 0x7f62b4152e30 2023-03-03T12:05:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.594+0000 7f62b37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b40beab0 0x7f62b40c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.594+0000 7f62b37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f62b40beab0 0x7f62b40c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39048/0 (socket says 172.21.15.7:39048) 2023-03-03T12:05:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.594+0000 7f62b37fe700 1 -- 172.21.15.7:0/1054268187 learned_addr learned my addr 172.21.15.7:0/1054268187 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:05:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.594+0000 7f62b3fff700 1 --2- 172.21.15.7:0/1054268187 >> v2:172.21.15.143:3300/0 conn(0x7f62b4152e30 0x7f62b40c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.595+0000 7f62b37fe700 1 -- 172.21.15.7:0/1054268187 >> v2:172.21.15.143:3300/0 conn(0x7f62b4152e30 msgr2=0x7f62b40c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:05:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.595+0000 7f62b37fe700 1 --2- 172.21.15.7:0/1054268187 >> v2:172.21.15.143:3300/0 conn(0x7f62b4152e30 0x7f62b40c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.595+0000 7f62b37fe700 1 -- 172.21.15.7:0/1054268187 >> v1:172.21.15.143:6789/0 conn(0x7f62b4155d80 legacy=0x7f62b4156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.595+0000 7f62b37fe700 1 -- 172.21.15.7:0/1054268187 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b40aaaf0 con 0x7f62b40beab0 2023-03-03T12:05:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.595+0000 7f62b37fe700 1 --2- 172.21.15.7:0/1054268187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b40beab0 0x7f62b40c4910 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f62b40c60b0 tx=0x7f62a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2828fd8595954595 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:32.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.595+0000 7f62b2ffd700 1 -- 172.21.15.7:0/1054268187 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a800b870 con 0x7f62b40beab0 2023-03-03T12:05:32.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.595+0000 7f62b2ffd700 1 -- 172.21.15.7:0/1054268187 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a800b9d0 con 0x7f62b40beab0 2023-03-03T12:05:32.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.595+0000 7f62b2ffd700 1 -- 172.21.15.7:0/1054268187 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a800bd60 con 0x7f62b40beab0 2023-03-03T12:05:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.596+0000 7f62ba60b700 1 -- 172.21.15.7:0/1054268187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b40beab0 msgr2=0x7f62b40c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.596+0000 7f62ba60b700 1 --2- 172.21.15.7:0/1054268187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b40beab0 0x7f62b40c4910 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f62b40c60b0 tx=0x7f62a8009bd0 comp rx=0 tx=0).stop 2023-03-03T12:05:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.596+0000 7f62ba60b700 1 -- 172.21.15.7:0/1054268187 shutdown_connections 2023-03-03T12:05:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.596+0000 7f62ba60b700 1 --2- 172.21.15.7:0/1054268187 >> v2:172.21.15.143:3300/0 conn(0x7f62b4152e30 0x7f62b40c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.596+0000 7f62ba60b700 1 --2- 172.21.15.7:0/1054268187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b40beab0 0x7f62b40c4910 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.596+0000 7f62ba60b700 1 -- 172.21.15.7:0/1054268187 >> 172.21.15.7:0/1054268187 conn(0x7f62b40736f0 msgr2=0x7f62b4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:05:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.596+0000 7f62ba60b700 1 -- 172.21.15.7:0/1054268187 shutdown_connections 2023-03-03T12:05:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.596+0000 7f62ba60b700 1 -- 172.21.15.7:0/1054268187 wait complete. 2023-03-03T12:05:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.597+0000 7f62ba60b700 1 Processor -- start 2023-03-03T12:05:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.597+0000 7f62ba60b700 1 -- start start 2023-03-03T12:05:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62ba60b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62b40beab0 0x7f62b406ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62ba60b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62b4152e30 0x7f62b414b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62ba60b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b4155d80 0x7f62b414aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62ba60b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b41eabb0 con 0x7f62b4155d80 2023-03-03T12:05:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62ba60b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b41e9fa0 con 0x7f62b4152e30 2023-03-03T12:05:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62ba60b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b41ea280 con 0x7f62b40beab0 2023-03-03T12:05:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b8ba8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b4155d80 0x7f62b414aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b8ba8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b4155d80 0x7f62b414aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39056/0 (socket says 172.21.15.7:39056) 2023-03-03T12:05:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b8ba8700 1 -- 172.21.15.7:0/1207004620 learned_addr learned my addr 172.21.15.7:0/1207004620 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:05:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b3fff700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62b40beab0 0x7f62b406ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b37fe700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62b4152e30 0x7f62b414b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b8ba8700 1 -- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62b40beab0 msgr2=0x7f62b406ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b8ba8700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62b40beab0 0x7f62b406ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b8ba8700 1 -- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62b4152e30 msgr2=0x7f62b414b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b8ba8700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62b4152e30 0x7f62b414b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b8ba8700 1 -- 172.21.15.7:0/1207004620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a8009550 con 0x7f62b4155d80 2023-03-03T12:05:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.598+0000 7f62b8ba8700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b4155d80 0x7f62b414aa30 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00cc50 tx=0x7f62ac00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.599+0000 7f62b0ff9700 1 -- 172.21.15.7:0/1207004620 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac003c00 con 0x7f62b4155d80 2023-03-03T12:05:32.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.599+0000 7f62b0ff9700 1 -- 172.21.15.7:0/1207004620 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ac003d60 con 0x7f62b4155d80 2023-03-03T12:05:32.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.599+0000 7f62ba60b700 1 -- 172.21.15.7:0/1207004620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b4071c70 con 0x7f62b4155d80 2023-03-03T12:05:32.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.599+0000 7f62b0ff9700 1 -- 172.21.15.7:0/1207004620 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac00aab0 con 0x7f62b4155d80 2023-03-03T12:05:32.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.599+0000 7f62ba60b700 1 -- 172.21.15.7:0/1207004620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b40aadd0 con 0x7f62b4155d80 2023-03-03T12:05:32.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.600+0000 7f62ba60b700 1 -- 172.21.15.7:0/1207004620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b4153200 con 0x7f62b4155d80 2023-03-03T12:05:32.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.600+0000 7f62b0ff9700 1 -- 172.21.15.7:0/1207004620 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f62ac010050 con 0x7f62b4155d80 2023-03-03T12:05:32.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.601+0000 7f62b0ff9700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62a00682f0 0x7f62a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:32.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.601+0000 7f62b0ff9700 1 -- 172.21.15.7:0/1207004620 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f62ac085f80 con 0x7f62b4155d80 2023-03-03T12:05:32.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.601+0000 7f62b3fff700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62a00682f0 0x7f62a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:32.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.601+0000 7f62b3fff700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62a00682f0 0x7f62a006a7a0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f62a40099d0 tx=0x7f62a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:32.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.605+0000 7f62b0ff9700 1 -- 172.21.15.7:0/1207004620 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f62ac052320 con 0x7f62b4155d80 2023-03-03T12:05:32.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.843+0000 7f62ba60b700 1 -- 172.21.15.7:0/1207004620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62b4073ff0 con 0x7f62b4155d80 2023-03-03T12:05:32.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.844+0000 7f62b0ff9700 1 -- 172.21.15.7:0/1207004620 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f62ac055940 con 0x7f62b4155d80 2023-03-03T12:05:32.847 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:05:32.848 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:05:32.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 -- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62a00682f0 msgr2=0x7f62a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:32.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62a00682f0 0x7f62a006a7a0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f62a40099d0 tx=0x7f62a4008040 comp rx=0 tx=0).stop 2023-03-03T12:05:32.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 -- 172.21.15.7:0/1207004620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b4155d80 msgr2=0x7f62b414aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:32.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b4155d80 0x7f62b414aa30 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00cc50 tx=0x7f62ac00cf60 comp rx=0 tx=0).stop 2023-03-03T12:05:32.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 -- 172.21.15.7:0/1207004620 shutdown_connections 2023-03-03T12:05:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62a00682f0 0x7f62a006a7a0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62b4155d80 0x7f62b414aa30 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62b40beab0 0x7f62b406ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 --2- 172.21.15.7:0/1207004620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62b4152e30 0x7f62b414b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 -- 172.21.15.7:0/1207004620 >> 172.21.15.7:0/1207004620 conn(0x7f62b40736f0 msgr2=0x7f62b40c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:05:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 -- 172.21.15.7:0/1207004620 shutdown_connections 2023-03-03T12:05:32.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:32.846+0000 7f62ba60b700 1 -- 172.21.15.7:0/1207004620 wait complete. 2023-03-03T12:05:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:05:32.861 INFO:tasks.mds_thrash.fs.[b]:waiting for 52 secs before thrashing 2023-03-03T12:05:53.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:53.326 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:05:54.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:05:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.232+0000 7fee33bc1700 1 Processor -- start 2023-03-03T12:05:54.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.232+0000 7fee33bc1700 1 -- start start 2023-03-03T12:05:54.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.233+0000 7fee33bc1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c06d240 0x7fee2c156440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:54.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.233+0000 7fee33bc1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fee2c0740f0 0x7fee2c0c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:54.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.233+0000 7fee33bc1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c0c24b0 con 0x7fee2c0c3e30 2023-03-03T12:05:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.233+0000 7fee33bc1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee2c152ef0 con 0x7fee2c0740f0 2023-03-03T12:05:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.233+0000 7fee33bc1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c155960 con 0x7fee2c06d240 2023-03-03T12:05:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.233+0000 7fee3115c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c06d240 0x7fee2c156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.233+0000 7fee3215e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fee2c0740f0 0x7fee2c0c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.233+0000 7fee3115c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c06d240 0x7fee2c156440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:47150/0 (socket says 172.21.15.7:47150) 2023-03-03T12:05:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.233+0000 7fee3115c700 1 -- 172.21.15.7:0/2839851585 learned_addr learned my addr 172.21.15.7:0/2839851585 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:05:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.234+0000 7fee3115c700 1 -- 172.21.15.7:0/2839851585 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0740f0 msgr2=0x7fee2c0c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:05:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.234+0000 7fee3115c700 1 --2- 172.21.15.7:0/2839851585 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0740f0 0x7fee2c0c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.234+0000 7fee3115c700 1 -- 172.21.15.7:0/2839851585 >> v1:172.21.15.143:6789/0 conn(0x7fee2c0c3e30 legacy=0x7fee2c0c1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.234+0000 7fee3115c700 1 -- 172.21.15.7:0/2839851585 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c155c40 con 0x7fee2c06d240 2023-03-03T12:05:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.234+0000 7fee3215e700 1 --2- 172.21.15.7:0/2839851585 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0740f0 0x7fee2c0c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:05:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.234+0000 7fee3115c700 1 --2- 172.21.15.7:0/2839851585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c06d240 0x7fee2c156440 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0098f0 tx=0x7fee1c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5681acb2dcdce413 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.234+0000 7fee3095b700 1 -- 172.21.15.7:0/2839851585 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c011830 con 0x7fee2c06d240 2023-03-03T12:05:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.234+0000 7fee3095b700 1 -- 172.21.15.7:0/2839851585 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c011990 con 0x7fee2c06d240 2023-03-03T12:05:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.234+0000 7fee3095b700 1 -- 172.21.15.7:0/2839851585 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c011d20 con 0x7fee2c06d240 2023-03-03T12:05:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.234+0000 7fee33bc1700 1 -- 172.21.15.7:0/2839851585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c06d240 msgr2=0x7fee2c156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.235+0000 7fee33bc1700 1 --2- 172.21.15.7:0/2839851585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c06d240 0x7fee2c156440 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0098f0 tx=0x7fee1c009c00 comp rx=0 tx=0).stop 2023-03-03T12:05:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.235+0000 7fee33bc1700 1 -- 172.21.15.7:0/2839851585 shutdown_connections 2023-03-03T12:05:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.235+0000 7fee33bc1700 1 --2- 172.21.15.7:0/2839851585 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0740f0 0x7fee2c0c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.235+0000 7fee33bc1700 1 --2- 172.21.15.7:0/2839851585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c06d240 0x7fee2c156440 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.235+0000 7fee33bc1700 1 -- 172.21.15.7:0/2839851585 >> 172.21.15.7:0/2839851585 conn(0x7fee2c06c110 msgr2=0x7fee2c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:05:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.235+0000 7fee33bc1700 1 -- 172.21.15.7:0/2839851585 shutdown_connections 2023-03-03T12:05:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.235+0000 7fee33bc1700 1 -- 172.21.15.7:0/2839851585 wait complete. 2023-03-03T12:05:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.236+0000 7fee33bc1700 1 Processor -- start 2023-03-03T12:05:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.236+0000 7fee33bc1700 1 -- start start 2023-03-03T12:05:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee33bc1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c06d240 0x7fee2c1df0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee33bc1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c0740f0 0x7fee2c14c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee33bc1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c0c3e30 0x7fee2c14cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee33bc1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c14d040 con 0x7fee2c06d240 2023-03-03T12:05:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee33bc1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c14d250 con 0x7fee2c0c3e30 2023-03-03T12:05:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee33bc1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c1eb4f0 con 0x7fee2c0740f0 2023-03-03T12:05:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3195d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c06d240 0x7fee2c1df0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3195d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c06d240 0x7fee2c1df0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39084/0 (socket says 172.21.15.7:39084) 2023-03-03T12:05:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3195d700 1 -- 172.21.15.7:0/1295329549 learned_addr learned my addr 172.21.15.7:0/1295329549 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:05:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3115c700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c0740f0 0x7fee2c14c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3215e700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c0c3e30 0x7fee2c14cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3195d700 1 -- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c0740f0 msgr2=0x7fee2c14c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3195d700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c0740f0 0x7fee2c14c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3195d700 1 -- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c0c3e30 msgr2=0x7fee2c14cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3195d700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c0c3e30 0x7fee2c14cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3195d700 1 -- 172.21.15.7:0/1295329549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee24009a70 con 0x7fee2c06d240 2023-03-03T12:05:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee3195d700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c06d240 0x7fee2c1df0e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fee2800c450 tx=0x7fee2800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee227fc700 1 -- 172.21.15.7:0/1295329549 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee280052d0 con 0x7fee2c06d240 2023-03-03T12:05:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.237+0000 7fee227fc700 1 -- 172.21.15.7:0/1295329549 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee28005430 con 0x7fee2c06d240 2023-03-03T12:05:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.238+0000 7fee33bc1700 1 -- 172.21.15.7:0/1295329549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c009550 con 0x7fee2c06d240 2023-03-03T12:05:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.238+0000 7fee227fc700 1 -- 172.21.15.7:0/1295329549 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee280038c0 con 0x7fee2c06d240 2023-03-03T12:05:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.238+0000 7fee33bc1700 1 -- 172.21.15.7:0/1295329549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c06e150 con 0x7fee2c06d240 2023-03-03T12:05:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.238+0000 7fee33bc1700 1 -- 172.21.15.7:0/1295329549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c155f20 con 0x7fee2c06d240 2023-03-03T12:05:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.239+0000 7fee227fc700 1 -- 172.21.15.7:0/1295329549 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fee28019030 con 0x7fee2c06d240 2023-03-03T12:05:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.239+0000 7fee227fc700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 0x7fee1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:05:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.239+0000 7fee227fc700 1 -- 172.21.15.7:0/1295329549 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fee28085340 con 0x7fee2c06d240 2023-03-03T12:05:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.240+0000 7fee3115c700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 0x7fee1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:05:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.240+0000 7fee3115c700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 0x7fee1006a7a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fee2c06d610 tx=0x7fee1c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:05:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.244+0000 7fee227fc700 1 -- 172.21.15.7:0/1295329549 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fee280516e0 con 0x7fee2c06d240 2023-03-03T12:05:54.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.481+0000 7fee33bc1700 1 -- 172.21.15.7:0/1295329549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee2c06e410 con 0x7fee2c06d240 2023-03-03T12:05:54.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.481+0000 7fee227fc700 1 -- 172.21.15.7:0/1295329549 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fee28054d00 con 0x7fee2c06d240 2023-03-03T12:05:54.483 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:05:54.483 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:05:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 -- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 msgr2=0x7fee1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 0x7fee1006a7a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fee2c06d610 tx=0x7fee1c008030 comp rx=0 tx=0).stop 2023-03-03T12:05:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 -- 172.21.15.7:0/1295329549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c06d240 msgr2=0x7fee2c1df0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:05:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c06d240 0x7fee2c1df0e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fee2800c450 tx=0x7fee2800c810 comp rx=0 tx=0).stop 2023-03-03T12:05:54.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 -- 172.21.15.7:0/1295329549 shutdown_connections 2023-03-03T12:05:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c0c3e30 0x7fee2c14cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 0x7fee1006a7a0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c06d240 0x7fee2c1df0e0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 --2- 172.21.15.7:0/1295329549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee2c0740f0 0x7fee2c14c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:05:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 -- 172.21.15.7:0/1295329549 >> 172.21.15.7:0/1295329549 conn(0x7fee2c06c110 msgr2=0x7fee2c06ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:05:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 -- 172.21.15.7:0/1295329549 shutdown_connections 2023-03-03T12:05:54.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:05:54.483+0000 7fee33bc1700 1 -- 172.21.15.7:0/1295329549 wait complete. 2023-03-03T12:05:54.487 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:05:54.496 INFO:tasks.mds_thrash.fs.[a]:waiting for 23 secs before thrashing 2023-03-03T12:06:17.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:06:17.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.635+0000 7ff35ae67700 1 Processor -- start 2023-03-03T12:06:17.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.635+0000 7ff35ae67700 1 -- start start 2023-03-03T12:06:17.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff35ae67700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540c00e0 0x7ff3540c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:17.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff35ae67700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff354154ff0 0x7ff3540cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:17.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff35ae67700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3540c7310 con 0x7ff3540caa30 2023-03-03T12:06:17.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff35ae67700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3540bfd10 con 0x7ff354154ff0 2023-03-03T12:06:17.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff35ae67700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540bfa30 con 0x7ff3540c00e0 2023-03-03T12:06:17.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff358c03700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff3540caa30 0x7ff3540c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33096/0 (socket says 172.21.15.7:33096) 2023-03-03T12:06:17.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff358c03700 1 -- 172.21.15.7:0/360037375 learned_addr learned my addr 172.21.15.7:0/360037375 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:06:17.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff359404700 1 --2- 172.21.15.7:0/360037375 >> v2:172.21.15.143:3300/0 conn(0x7ff354154ff0 0x7ff3540cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:17.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff353fff700 1 --2- 172.21.15.7:0/360037375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540c00e0 0x7ff3540c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:17.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff3537fe700 1 -- 172.21.15.7:0/360037375 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787033653 0 0) 0x7ff3540c7310 con 0x7ff3540caa30 2023-03-03T12:06:17.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff3537fe700 1 -- 172.21.15.7:0/360037375 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3440035b0 con 0x7ff3540caa30 2023-03-03T12:06:17.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff353fff700 1 -- 172.21.15.7:0/360037375 >> v2:172.21.15.143:3300/0 conn(0x7ff354154ff0 msgr2=0x7ff3540cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:06:17.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.636+0000 7ff353fff700 1 --2- 172.21.15.7:0/360037375 >> v2:172.21.15.143:3300/0 conn(0x7ff354154ff0 0x7ff3540cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:17.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.637+0000 7ff353fff700 1 -- 172.21.15.7:0/360037375 >> v1:172.21.15.7:6789/0 conn(0x7ff3540caa30 legacy=0x7ff3540c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:17.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.637+0000 7ff3537fe700 1 -- 172.21.15.7:0/360037375 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 234290815 0 0) 0x7ff3440035b0 con 0x7ff3540caa30 2023-03-03T12:06:17.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.637+0000 7ff353fff700 1 -- 172.21.15.7:0/360037375 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff354154510 con 0x7ff3540c00e0 2023-03-03T12:06:17.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.637+0000 7ff359404700 1 --2- 172.21.15.7:0/360037375 >> v2:172.21.15.143:3300/0 conn(0x7ff354154ff0 0x7ff3540cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:06:17.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.637+0000 7ff353fff700 1 --2- 172.21.15.7:0/360037375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540c00e0 0x7ff3540c5c60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff340009820 tx=0x7ff340009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46331769efc1513f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:06:17.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.637+0000 7ff3537fe700 1 -- 172.21.15.7:0/360037375 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348001670 con 0x7ff3540c00e0 2023-03-03T12:06:17.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.637+0000 7ff3537fe700 1 -- 172.21.15.7:0/360037375 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff340005c30 con 0x7ff3540c00e0 2023-03-03T12:06:17.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.637+0000 7ff3537fe700 1 -- 172.21.15.7:0/360037375 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340004030 con 0x7ff3540c00e0 2023-03-03T12:06:17.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.638+0000 7ff35ae67700 1 -- 172.21.15.7:0/360037375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540c00e0 msgr2=0x7ff3540c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:17.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.638+0000 7ff35ae67700 1 --2- 172.21.15.7:0/360037375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540c00e0 0x7ff3540c5c60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff340009820 tx=0x7ff340009b30 comp rx=0 tx=0).stop 2023-03-03T12:06:17.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.638+0000 7ff35ae67700 1 -- 172.21.15.7:0/360037375 shutdown_connections 2023-03-03T12:06:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.638+0000 7ff35ae67700 1 --2- 172.21.15.7:0/360037375 >> v2:172.21.15.143:3300/0 conn(0x7ff354154ff0 0x7ff3540cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.638+0000 7ff35ae67700 1 --2- 172.21.15.7:0/360037375 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540c00e0 0x7ff3540c5c60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.638+0000 7ff35ae67700 1 -- 172.21.15.7:0/360037375 >> 172.21.15.7:0/360037375 conn(0x7ff354066e40 msgr2=0x7ff354071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:06:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.638+0000 7ff35ae67700 1 -- 172.21.15.7:0/360037375 shutdown_connections 2023-03-03T12:06:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.638+0000 7ff35ae67700 1 -- 172.21.15.7:0/360037375 wait complete. 2023-03-03T12:06:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.639+0000 7ff35ae67700 1 Processor -- start 2023-03-03T12:06:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.639+0000 7ff35ae67700 1 -- start start 2023-03-03T12:06:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff35ae67700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3540c00e0 0x7ff354149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff35ae67700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540caa30 0x7ff354156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff35ae67700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff354154ff0 0x7ff354156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff35ae67700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540c3160 con 0x7ff3540c00e0 2023-03-03T12:06:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff35ae67700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540c35f0 con 0x7ff354154ff0 2023-03-03T12:06:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff35ae67700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3541573c0 con 0x7ff3540caa30 2023-03-03T12:06:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff358c03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3540c00e0 0x7ff354149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff358c03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3540c00e0 0x7ff354149120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39122/0 (socket says 172.21.15.7:39122) 2023-03-03T12:06:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff358c03700 1 -- 172.21.15.7:0/489450946 learned_addr learned my addr 172.21.15.7:0/489450946 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:06:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff359404700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff354154ff0 0x7ff354156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff353fff700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540caa30 0x7ff354156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff358c03700 1 -- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540caa30 msgr2=0x7ff354156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff358c03700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540caa30 0x7ff354156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff358c03700 1 -- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff354154ff0 msgr2=0x7ff354156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff358c03700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff354154ff0 0x7ff354156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.640+0000 7ff358c03700 1 -- 172.21.15.7:0/489450946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c009a70 con 0x7ff3540c00e0 2023-03-03T12:06:17.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.641+0000 7ff358c03700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3540c00e0 0x7ff354149120 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff3480011e0 tx=0x7ff34800ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:06:17.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.641+0000 7ff3517fa700 1 -- 172.21.15.7:0/489450946 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348004c80 con 0x7ff3540c00e0 2023-03-03T12:06:17.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.641+0000 7ff3517fa700 1 -- 172.21.15.7:0/489450946 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff348004de0 con 0x7ff3540c00e0 2023-03-03T12:06:17.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.641+0000 7ff3517fa700 1 -- 172.21.15.7:0/489450946 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3480072f0 con 0x7ff3540c00e0 2023-03-03T12:06:17.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.641+0000 7ff35ae67700 1 -- 172.21.15.7:0/489450946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff340009430 con 0x7ff3540c00e0 2023-03-03T12:06:17.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.641+0000 7ff35ae67700 1 -- 172.21.15.7:0/489450946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff354154ad0 con 0x7ff3540c00e0 2023-03-03T12:06:17.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.642+0000 7ff35ae67700 1 -- 172.21.15.7:0/489450946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3541547f0 con 0x7ff3540c00e0 2023-03-03T12:06:17.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.642+0000 7ff3517fa700 1 -- 172.21.15.7:0/489450946 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff34801a020 con 0x7ff3540c00e0 2023-03-03T12:06:17.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.643+0000 7ff3517fa700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3380682f0 0x7ff33806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:17.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.643+0000 7ff3517fa700 1 -- 172.21.15.7:0/489450946 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff348085cf0 con 0x7ff3540c00e0 2023-03-03T12:06:17.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.643+0000 7ff353fff700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3380682f0 0x7ff33806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:17.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.643+0000 7ff353fff700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3380682f0 0x7ff33806a7a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff340009b70 tx=0x7ff340008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:06:17.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.647+0000 7ff3517fa700 1 -- 172.21.15.7:0/489450946 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff348052090 con 0x7ff3540c00e0 2023-03-03T12:06:17.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.889+0000 7ff35ae67700 1 -- 172.21.15.7:0/489450946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3540c3440 con 0x7ff3540c00e0 2023-03-03T12:06:17.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.889+0000 7ff3517fa700 1 -- 172.21.15.7:0/489450946 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff3480556b0 con 0x7ff3540c00e0 2023-03-03T12:06:17.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:06:17.892 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:06:17.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 -- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3380682f0 msgr2=0x7ff33806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:17.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3380682f0 0x7ff33806a7a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff340009b70 tx=0x7ff340008070 comp rx=0 tx=0).stop 2023-03-03T12:06:17.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 -- 172.21.15.7:0/489450946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3540c00e0 msgr2=0x7ff354149120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:17.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3540c00e0 0x7ff354149120 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff3480011e0 tx=0x7ff34800ca50 comp rx=0 tx=0).stop 2023-03-03T12:06:17.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 -- 172.21.15.7:0/489450946 shutdown_connections 2023-03-03T12:06:17.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff354154ff0 0x7ff354156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:17.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3380682f0 0x7ff33806a7a0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:17.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3540c00e0 0x7ff354149120 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:17.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 --2- 172.21.15.7:0/489450946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3540caa30 0x7ff354156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:17.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 -- 172.21.15.7:0/489450946 >> 172.21.15.7:0/489450946 conn(0x7ff354066e40 msgr2=0x7ff354151ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:06:17.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 -- 172.21.15.7:0/489450946 shutdown_connections 2023-03-03T12:06:17.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:17.892+0000 7ff35ae67700 1 -- 172.21.15.7:0/489450946 wait complete. 2023-03-03T12:06:17.895 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:06:17.905 INFO:tasks.mds_thrash.fs.[a]:waiting for 45 secs before thrashing 2023-03-03T12:06:23.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:23.483 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:24.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:06:25.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:24.999+0000 7fc560326700 1 Processor -- start 2023-03-03T12:06:25.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:24.999+0000 7fc560326700 1 -- start start 2023-03-03T12:06:25.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.000+0000 7fc560326700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc558071e80 0x7fc558151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:25.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.000+0000 7fc560326700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5581510d0 con 0x7fc558072250 2023-03-03T12:06:25.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.000+0000 7fc560326700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5580c3740 con 0x7fc55814e900 2023-03-03T12:06:25.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.000+0000 7fc560326700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc558070670 con 0x7fc558071e80 2023-03-03T12:06:25.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.000+0000 7fc55d8c1700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc558072250 0x7fc55814e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33114/0 (socket says 172.21.15.7:33114) 2023-03-03T12:06:25.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.000+0000 7fc55d8c1700 1 -- 172.21.15.7:0/1764721486 learned_addr learned my addr 172.21.15.7:0/1764721486 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:06:25.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.000+0000 7fc55e0c2700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc558071e80 0x7fc558151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:25.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.000+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668907810 0 0) 0x7fc5581510d0 con 0x7fc558072250 2023-03-03T12:06:25.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.000+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5440035b0 con 0x7fc558072250 2023-03-03T12:06:25.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1490561715 0 0) 0x7fc5440035b0 con 0x7fc558072250 2023-03-03T12:06:25.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc558071e80 msgr2=0x7fc558151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:06:25.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc558071e80 0x7fc558151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:25.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 >> v1:172.21.15.143:6789/0 conn(0x7fc55814e900 legacy=0x7fc55815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:25.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5580c39e0 con 0x7fc558072250 2023-03-03T12:06:25.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc554001670 con 0x7fc558072250 2023-03-03T12:06:25.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 >> v1:172.21.15.7:6789/0 conn(0x7fc558072250 legacy=0x7fc55814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:25.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc544003aa0 0x7fc544003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:25.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc544004850 0x7fc544004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:25.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc558071e80 0x7fc544005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:25.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc558070670 con 0x7fc544004850 2023-03-03T12:06:25.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54400bba0 con 0x7fc544003aa0 2023-03-03T12:06:25.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54400df70 con 0x7fc558071e80 2023-03-03T12:06:25.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d8c1700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc544004850 0x7fc544004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:25.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55e0c2700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc544003aa0 0x7fc544003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:25.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55e8c3700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc558071e80 0x7fc544005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:25.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d8c1700 1 -- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc558071e80 msgr2=0x7fc544005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:06:25.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d8c1700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc558071e80 0x7fc544005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:25.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d8c1700 1 -- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc544003aa0 msgr2=0x7fc544003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:06:25.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d8c1700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc544003aa0 0x7fc544003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:25.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d8c1700 1 -- 172.21.15.7:0/1764721486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5440035b0 con 0x7fc544004850 2023-03-03T12:06:25.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.001+0000 7fc55d8c1700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc544004850 0x7fc544004cc0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fc55400c210 tx=0x7fc544005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e0c9d00644991 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:06:25.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.002+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554007c10 con 0x7fc544004850 2023-03-03T12:06:25.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.002+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55400cdd0 con 0x7fc544004850 2023-03-03T12:06:25.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.002+0000 7fc55d0c0700 1 -- 172.21.15.7:0/1764721486 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554005ba0 con 0x7fc544004850 2023-03-03T12:06:25.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.002+0000 7fc560326700 1 -- 172.21.15.7:0/1764721486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc544004850 msgr2=0x7fc544004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:25.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.002+0000 7fc560326700 1 --2- 172.21.15.7:0/1764721486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc544004850 0x7fc544004cc0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fc55400c210 tx=0x7fc544005880 comp rx=0 tx=0).stop 2023-03-03T12:06:25.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.002+0000 7fc55e8c3700 1 -- 172.21.15.7:0/1764721486 reap_dead start 2023-03-03T12:06:25.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.002+0000 7fc560326700 1 -- 172.21.15.7:0/1764721486 shutdown_connections 2023-03-03T12:06:25.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.002+0000 7fc560326700 1 -- 172.21.15.7:0/1764721486 >> 172.21.15.7:0/1764721486 conn(0x7fc5580a94d0 msgr2=0x7fc5580a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:06:25.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.002+0000 7fc560326700 1 -- 172.21.15.7:0/1764721486 shutdown_connections 2023-03-03T12:06:25.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.002+0000 7fc560326700 1 -- 172.21.15.7:0/1764721486 wait complete. 2023-03-03T12:06:25.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.003+0000 7fc560326700 1 Processor -- start 2023-03-03T12:06:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.003+0000 7fc560326700 1 -- start start 2023-03-03T12:06:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc560326700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc544004850 0x7fc5581e9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc560326700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc558147440 0x7fc5580c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc560326700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc558147990 0x7fc5580c1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc560326700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580be430 con 0x7fc558147990 2023-03-03T12:06:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc560326700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580be960 con 0x7fc558147440 2023-03-03T12:06:25.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc560326700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580be680 con 0x7fc544004850 2023-03-03T12:06:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc55e8c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc558147990 0x7fc5580c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc55e8c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc558147990 0x7fc5580c1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39150/0 (socket says 172.21.15.7:39150) 2023-03-03T12:06:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc55e8c3700 1 -- 172.21.15.7:0/1787959640 learned_addr learned my addr 172.21.15.7:0/1787959640 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:06:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc55e0c2700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc544004850 0x7fc5581e9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.004+0000 7fc55d8c1700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc558147440 0x7fc5580c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc55e8c3700 1 -- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc544004850 msgr2=0x7fc5581e9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc55e8c3700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc544004850 0x7fc5581e9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc55e8c3700 1 -- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc558147440 msgr2=0x7fc5580c0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc55e8c3700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc558147440 0x7fc5580c0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc55e8c3700 1 -- 172.21.15.7:0/1787959640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55806c9a0 con 0x7fc558147990 2023-03-03T12:06:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc55e8c3700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc558147990 0x7fc5580c1080 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fc5580713a0 tx=0x7fc55000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:06:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc54effd700 1 -- 172.21.15.7:0/1787959640 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550003a00 con 0x7fc558147990 2023-03-03T12:06:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc54effd700 1 -- 172.21.15.7:0/1787959640 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc550003b60 con 0x7fc558147990 2023-03-03T12:06:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc54effd700 1 -- 172.21.15.7:0/1787959640 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55000d930 con 0x7fc558147990 2023-03-03T12:06:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc560326700 1 -- 172.21.15.7:0/1787959640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc558151300 con 0x7fc558147990 2023-03-03T12:06:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.005+0000 7fc560326700 1 -- 172.21.15.7:0/1787959640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55806cb30 con 0x7fc558147990 2023-03-03T12:06:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.006+0000 7fc560326700 1 -- 172.21.15.7:0/1787959640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc558074bb0 con 0x7fc558147990 2023-03-03T12:06:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.006+0000 7fc54effd700 1 -- 172.21.15.7:0/1787959640 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc55000da90 con 0x7fc558147990 2023-03-03T12:06:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.007+0000 7fc54effd700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc53c0682f0 0x7fc53c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:06:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.007+0000 7fc54effd700 1 -- 172.21.15.7:0/1787959640 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc550086390 con 0x7fc558147990 2023-03-03T12:06:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.007+0000 7fc55e0c2700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc53c0682f0 0x7fc53c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:06:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.008+0000 7fc55e0c2700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc53c0682f0 0x7fc53c06a7a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc548009c60 tx=0x7fc548009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:06:25.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.011+0000 7fc54effd700 1 -- 172.21.15.7:0/1787959640 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc550052730 con 0x7fc558147990 2023-03-03T12:06:25.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.248+0000 7fc560326700 1 -- 172.21.15.7:0/1787959640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5580a9fb0 con 0x7fc558147990 2023-03-03T12:06:25.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.249+0000 7fc54effd700 1 -- 172.21.15.7:0/1787959640 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc550055d50 con 0x7fc558147990 2023-03-03T12:06:25.251 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:06:25.251 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:06:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 -- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc53c0682f0 msgr2=0x7fc53c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc53c0682f0 0x7fc53c06a7a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc548009c60 tx=0x7fc548009210 comp rx=0 tx=0).stop 2023-03-03T12:06:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 -- 172.21.15.7:0/1787959640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc558147990 msgr2=0x7fc5580c1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:06:25.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc558147990 0x7fc5580c1080 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fc5580713a0 tx=0x7fc55000d430 comp rx=0 tx=0).stop 2023-03-03T12:06:25.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 -- 172.21.15.7:0/1787959640 shutdown_connections 2023-03-03T12:06:25.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc53c0682f0 0x7fc53c06a7a0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:25.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc558147990 0x7fc5580c1080 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:25.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc544004850 0x7fc5581e9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:25.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 --2- 172.21.15.7:0/1787959640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc558147440 0x7fc5580c0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:06:25.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 -- 172.21.15.7:0/1787959640 >> 172.21.15.7:0/1787959640 conn(0x7fc5580a94d0 msgr2=0x7fc55814b140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:06:25.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 -- 172.21.15.7:0/1787959640 shutdown_connections 2023-03-03T12:06:25.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:06:25.251+0000 7fc560326700 1 -- 172.21.15.7:0/1787959640 wait complete. 2023-03-03T12:06:25.255 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:06:25.264 INFO:tasks.mds_thrash.fs.[b]:waiting for 87 secs before thrashing 2023-03-03T12:06:53.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:06:53.630 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:02.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:07:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.046+0000 7f1f22973700 1 Processor -- start 2023-03-03T12:07:03.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.046+0000 7f1f22973700 1 -- start start 2023-03-03T12:07:03.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.046+0000 7f1f22973700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c0ac220 0x7f1f1c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f22973700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1f1c152b60 0x7f1f1c15ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f22973700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c0ac900 con 0x7f1f1c073c30 2023-03-03T12:07:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f22973700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f1c071e30 con 0x7f1f1c152b60 2023-03-03T12:07:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f22973700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0acb50 con 0x7f1f1c0ac220 2023-03-03T12:07:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f1bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c0ac220 0x7f1f1c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f20f10700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1f1c152b60 0x7f1f1c15ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f20f10700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1f1c152b60 0x7f1f1c15ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:34976/0 (socket says 172.21.15.7:34976) 2023-03-03T12:07:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f1bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c0ac220 0x7f1f1c0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:47270/0 (socket says 172.21.15.7:47270) 2023-03-03T12:07:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f20f10700 1 -- 172.21.15.7:0/3947411634 learned_addr learned my addr 172.21.15.7:0/3947411634 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:07:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f1b7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1f1c073c30 0x7f1f1c152450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:36698/0 (socket says 172.21.15.7:36698) 2023-03-03T12:07:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f1bfff700 1 -- 172.21.15.7:0/3947411634 >> v2:172.21.15.143:3300/0 conn(0x7f1f1c152b60 msgr2=0x7f1f1c15ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:07:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/3947411634 >> v2:172.21.15.143:3300/0 conn(0x7f1f1c152b60 0x7f1f1c15ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f1bfff700 1 -- 172.21.15.7:0/3947411634 >> v1:172.21.15.143:6789/0 conn(0x7f1f1c073c30 legacy=0x7f1f1c152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f1affd700 1 -- 172.21.15.7:0/3947411634 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426906902 0 0) 0x7f1f1c0ac900 con 0x7f1f1c073c30 2023-03-03T12:07:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f1bfff700 1 -- 172.21.15.7:0/3947411634 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c1559c0 con 0x7f1f1c0ac220 2023-03-03T12:07:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.047+0000 7f1f20f10700 1 --2- 172.21.15.7:0/3947411634 >> v2:172.21.15.143:3300/0 conn(0x7f1f1c152b60 0x7f1f1c15ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:07:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/3947411634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c0ac220 0x7f1f1c0736f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009840 tx=0x7f1f0c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ff0d8d39a94a2b4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:07:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f1affd700 1 -- 172.21.15.7:0/3947411634 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c011830 con 0x7f1f1c0ac220 2023-03-03T12:07:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f1affd700 1 -- 172.21.15.7:0/3947411634 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c011990 con 0x7f1f1c0ac220 2023-03-03T12:07:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f1affd700 1 -- 172.21.15.7:0/3947411634 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c011d20 con 0x7f1f1c0ac220 2023-03-03T12:07:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f22973700 1 -- 172.21.15.7:0/3947411634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c0ac220 msgr2=0x7f1f1c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f22973700 1 --2- 172.21.15.7:0/3947411634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c0ac220 0x7f1f1c0736f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009840 tx=0x7f1f0c009b50 comp rx=0 tx=0).stop 2023-03-03T12:07:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f22973700 1 -- 172.21.15.7:0/3947411634 shutdown_connections 2023-03-03T12:07:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f22973700 1 --2- 172.21.15.7:0/3947411634 >> v2:172.21.15.143:3300/0 conn(0x7f1f1c152b60 0x7f1f1c15ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f22973700 1 --2- 172.21.15.7:0/3947411634 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c0ac220 0x7f1f1c0736f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f22973700 1 -- 172.21.15.7:0/3947411634 >> 172.21.15.7:0/3947411634 conn(0x7f1f1c0a94d0 msgr2=0x7f1f1c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:07:03.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f22973700 1 -- 172.21.15.7:0/3947411634 shutdown_connections 2023-03-03T12:07:03.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.048+0000 7f1f22973700 1 -- 172.21.15.7:0/3947411634 wait complete. 2023-03-03T12:07:03.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.050+0000 7f1f22973700 1 Processor -- start 2023-03-03T12:07:03.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.050+0000 7f1f22973700 1 -- start start 2023-03-03T12:07:03.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.050+0000 7f1f22973700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c073c30 0x7f1f1c0c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:03.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.050+0000 7f1f22973700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ac220 0x7f1f1c1ebe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:03.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.050+0000 7f1f22973700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c152b60 0x7f1f1c151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:03.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.050+0000 7f1f22973700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c1483a0 con 0x7f1f1c0ac220 2023-03-03T12:07:03.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.050+0000 7f1f22973700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c1eb970 con 0x7f1f1c073c30 2023-03-03T12:07:03.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.050+0000 7f1f22973700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c1488d0 con 0x7f1f1c152b60 2023-03-03T12:07:03.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f1b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ac220 0x7f1f1c1ebe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:03.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f1b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ac220 0x7f1f1c1ebe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39206/0 (socket says 172.21.15.7:39206) 2023-03-03T12:07:03.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f20f10700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c152b60 0x7f1f1c151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:03.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f1b7fe700 1 -- 172.21.15.7:0/2220029811 learned_addr learned my addr 172.21.15.7:0/2220029811 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:07:03.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c073c30 0x7f1f1c0c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:03.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f1b7fe700 1 -- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c152b60 msgr2=0x7f1f1c151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:03.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f1b7fe700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c152b60 0x7f1f1c151310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:03.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f1b7fe700 1 -- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c073c30 msgr2=0x7f1f1c0c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:03.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f1b7fe700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c073c30 0x7f1f1c0c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:03.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f1b7fe700 1 -- 172.21.15.7:0/2220029811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f14009b30 con 0x7f1f1c0ac220 2023-03-03T12:07:03.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f20f10700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c152b60 0x7f1f1c151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:07:03.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f1b7fe700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ac220 0x7f1f1c1ebe00 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000c550 tx=0x7f1f1000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:07:03.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f18ff9700 1 -- 172.21.15.7:0/2220029811 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10004c80 con 0x7f1f1c0ac220 2023-03-03T12:07:03.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f18ff9700 1 -- 172.21.15.7:0/2220029811 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10004e00 con 0x7f1f1c0ac220 2023-03-03T12:07:03.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f22973700 1 -- 172.21.15.7:0/2220029811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0c0094a0 con 0x7f1f1c0ac220 2023-03-03T12:07:03.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f18ff9700 1 -- 172.21.15.7:0/2220029811 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10007310 con 0x7f1f1c0ac220 2023-03-03T12:07:03.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.051+0000 7f1f22973700 1 -- 172.21.15.7:0/2220029811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c155f80 con 0x7f1f1c0ac220 2023-03-03T12:07:03.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.052+0000 7f1f22973700 1 -- 172.21.15.7:0/2220029811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c155ca0 con 0x7f1f1c0ac220 2023-03-03T12:07:03.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.053+0000 7f1f18ff9700 1 -- 172.21.15.7:0/2220029811 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f1001a020 con 0x7f1f1c0ac220 2023-03-03T12:07:03.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.053+0000 7f1f18ff9700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f080682f0 0x7f1f0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:03.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.053+0000 7f1f18ff9700 1 -- 172.21.15.7:0/2220029811 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1f10085c20 con 0x7f1f1c0ac220 2023-03-03T12:07:03.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.053+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f080682f0 0x7f1f0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:03.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.054+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f080682f0 0x7f1f0806a7a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c000c00 tx=0x7f1f0c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:07:03.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.058+0000 7f1f18ff9700 1 -- 172.21.15.7:0/2220029811 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f10051f40 con 0x7f1f1c0ac220 2023-03-03T12:07:03.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.295+0000 7f1f22973700 1 -- 172.21.15.7:0/2220029811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f1c156260 con 0x7f1f1c0ac220 2023-03-03T12:07:03.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.295+0000 7f1f18ff9700 1 -- 172.21.15.7:0/2220029811 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1f10055560 con 0x7f1f1c0ac220 2023-03-03T12:07:03.297 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:07:03.297 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:07:03.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 -- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f080682f0 msgr2=0x7f1f0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f080682f0 0x7f1f0806a7a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c000c00 tx=0x7f1f0c01c040 comp rx=0 tx=0).stop 2023-03-03T12:07:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 -- 172.21.15.7:0/2220029811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ac220 msgr2=0x7f1f1c1ebe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ac220 0x7f1f1c1ebe00 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000c550 tx=0x7f1f1000c860 comp rx=0 tx=0).stop 2023-03-03T12:07:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 -- 172.21.15.7:0/2220029811 shutdown_connections 2023-03-03T12:07:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f1c152b60 0x7f1f1c151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c073c30 0x7f1f1c0c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:03.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f080682f0 0x7f1f0806a7a0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:03.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 --2- 172.21.15.7:0/2220029811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ac220 0x7f1f1c1ebe00 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:03.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 -- 172.21.15.7:0/2220029811 >> 172.21.15.7:0/2220029811 conn(0x7f1f1c0a94d0 msgr2=0x7f1f1c152450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:07:03.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 -- 172.21.15.7:0/2220029811 shutdown_connections 2023-03-03T12:07:03.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:03.297+0000 7f1f22973700 1 -- 172.21.15.7:0/2220029811 wait complete. 2023-03-03T12:07:03.300 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:07:03.310 INFO:tasks.mds_thrash.fs.[a]:waiting for 117 secs before thrashing 2023-03-03T12:07:23.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:23.776 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:52.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:07:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.402+0000 7fae1da00700 1 Processor -- start 2023-03-03T12:07:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.402+0000 7fae1da00700 1 -- start start 2023-03-03T12:07:52.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae1da00700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae180caa30 0x7fae180c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:52.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae1da00700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae180c00e0 0x7fae18154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:52.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae1da00700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae180cae00 0x7fae180c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:52.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae1da00700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae180cb280 con 0x7fae180c00e0 2023-03-03T12:07:52.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae1da00700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae180bfd10 con 0x7fae180cae00 2023-03-03T12:07:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae1da00700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae180bfa30 con 0x7fae180caa30 2023-03-03T12:07:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae167fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae180c00e0 0x7fae18154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae167fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae180c00e0 0x7fae18154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39248/0 (socket says 172.21.15.7:39248) 2023-03-03T12:07:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae167fc700 1 -- 172.21.15.7:0/299491990 learned_addr learned my addr 172.21.15.7:0/299491990 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:07:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae16ffd700 1 --2- 172.21.15.7:0/299491990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae180caa30 0x7fae180c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae177fe700 1 --2- 172.21.15.7:0/299491990 >> v2:172.21.15.143:3300/0 conn(0x7fae180cae00 0x7fae180c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae167fc700 1 -- 172.21.15.7:0/299491990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae180caa30 msgr2=0x7fae180c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:07:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae167fc700 1 --2- 172.21.15.7:0/299491990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae180caa30 0x7fae180c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae167fc700 1 -- 172.21.15.7:0/299491990 >> v2:172.21.15.143:3300/0 conn(0x7fae180cae00 msgr2=0x7fae180c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:07:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae167fc700 1 --2- 172.21.15.7:0/299491990 >> v2:172.21.15.143:3300/0 conn(0x7fae180cae00 0x7fae180c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.403+0000 7fae167fc700 1 -- 172.21.15.7:0/299491990 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae18154510 con 0x7fae180c00e0 2023-03-03T12:07:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae167fc700 1 --2- 172.21.15.7:0/299491990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180c00e0 0x7fae18154ff0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fae18158a70 tx=0x7fae08009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f3066668880000 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:07:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae15ffb700 1 -- 172.21.15.7:0/299491990 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae08005e30 con 0x7fae180c00e0 2023-03-03T12:07:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae15ffb700 1 -- 172.21.15.7:0/299491990 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae08004030 con 0x7fae180c00e0 2023-03-03T12:07:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae15ffb700 1 -- 172.21.15.7:0/299491990 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae080043e0 con 0x7fae180c00e0 2023-03-03T12:07:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae1da00700 1 -- 172.21.15.7:0/299491990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180c00e0 msgr2=0x7fae18154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae1da00700 1 --2- 172.21.15.7:0/299491990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180c00e0 0x7fae18154ff0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fae18158a70 tx=0x7fae08009bd0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae1da00700 1 -- 172.21.15.7:0/299491990 shutdown_connections 2023-03-03T12:07:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae1da00700 1 --2- 172.21.15.7:0/299491990 >> v2:172.21.15.143:3300/0 conn(0x7fae180cae00 0x7fae180c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae1da00700 1 --2- 172.21.15.7:0/299491990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae180caa30 0x7fae180c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae1da00700 1 --2- 172.21.15.7:0/299491990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180c00e0 0x7fae18154ff0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae1da00700 1 -- 172.21.15.7:0/299491990 >> 172.21.15.7:0/299491990 conn(0x7fae18066e40 msgr2=0x7fae18071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:07:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae1da00700 1 -- 172.21.15.7:0/299491990 shutdown_connections 2023-03-03T12:07:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.404+0000 7fae1da00700 1 -- 172.21.15.7:0/299491990 wait complete. 2023-03-03T12:07:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.406+0000 7fae1da00700 1 Processor -- start 2023-03-03T12:07:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.406+0000 7fae1da00700 1 -- start start 2023-03-03T12:07:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae1da00700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae180c00e0 0x7fae180c28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae1da00700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae180caa30 0x7fae18156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae1da00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180cae00 0x7fae181570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae1da00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180c48b0 con 0x7fae180cae00 2023-03-03T12:07:52.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae1da00700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180c3ca0 con 0x7fae180c00e0 2023-03-03T12:07:52.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae1da00700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae180c3f80 con 0x7fae180caa30 2023-03-03T12:07:52.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae177fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180cae00 0x7fae181570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:52.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae177fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180cae00 0x7fae181570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39254/0 (socket says 172.21.15.7:39254) 2023-03-03T12:07:52.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae177fe700 1 -- 172.21.15.7:0/2191092940 learned_addr learned my addr 172.21.15.7:0/2191092940 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:07:52.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae167fc700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae180caa30 0x7fae18156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:52.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae16ffd700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae180c00e0 0x7fae180c28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:52.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae177fe700 1 -- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae180caa30 msgr2=0x7fae18156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:52.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae177fe700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae180caa30 0x7fae18156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae177fe700 1 -- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae180c00e0 msgr2=0x7fae180c28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:52.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae177fe700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae180c00e0 0x7fae180c28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae177fe700 1 -- 172.21.15.7:0/2191092940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae08009500 con 0x7fae180cae00 2023-03-03T12:07:52.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fae177fe700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180cae00 0x7fae181570a0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fae10007e40 tx=0x7fae1000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:07:52.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.407+0000 7fadfffff700 1 -- 172.21.15.7:0/2191092940 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1000eea0 con 0x7fae180cae00 2023-03-03T12:07:52.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.408+0000 7fadfffff700 1 -- 172.21.15.7:0/2191092940 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae10014420 con 0x7fae180cae00 2023-03-03T12:07:52.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.408+0000 7fae1da00700 1 -- 172.21.15.7:0/2191092940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae18154ad0 con 0x7fae180cae00 2023-03-03T12:07:52.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.408+0000 7fadfffff700 1 -- 172.21.15.7:0/2191092940 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae100147b0 con 0x7fae180cae00 2023-03-03T12:07:52.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.408+0000 7fae1da00700 1 -- 172.21.15.7:0/2191092940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae181547f0 con 0x7fae180cae00 2023-03-03T12:07:52.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.409+0000 7fae1da00700 1 -- 172.21.15.7:0/2191092940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae180bf650 con 0x7fae180cae00 2023-03-03T12:07:52.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.409+0000 7fadfffff700 1 -- 172.21.15.7:0/2191092940 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae10012050 con 0x7fae180cae00 2023-03-03T12:07:52.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.409+0000 7fadfffff700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae040682f0 0x7fae0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:07:52.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.410+0000 7fadfffff700 1 -- 172.21.15.7:0/2191092940 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fae100867b0 con 0x7fae180cae00 2023-03-03T12:07:52.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.410+0000 7fae16ffd700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae040682f0 0x7fae0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:07:52.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.410+0000 7fae16ffd700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae040682f0 0x7fae0406a7a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009930 tx=0x7fae0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:07:52.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.414+0000 7fadfffff700 1 -- 172.21.15.7:0/2191092940 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae10052b50 con 0x7fae180cae00 2023-03-03T12:07:52.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.652+0000 7fae1da00700 1 -- 172.21.15.7:0/2191092940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae1804e6e0 con 0x7fae180cae00 2023-03-03T12:07:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.653+0000 7fadfffff700 1 -- 172.21.15.7:0/2191092940 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fae10056170 con 0x7fae180cae00 2023-03-03T12:07:52.656 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:07:52.656 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:07:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 -- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae040682f0 msgr2=0x7fae0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae040682f0 0x7fae0406a7a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009930 tx=0x7fae0c008040 comp rx=0 tx=0).stop 2023-03-03T12:07:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 -- 172.21.15.7:0/2191092940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180cae00 msgr2=0x7fae181570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:07:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180cae00 0x7fae181570a0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fae10007e40 tx=0x7fae1000e3c0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 -- 172.21.15.7:0/2191092940 shutdown_connections 2023-03-03T12:07:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae040682f0 0x7fae0406a7a0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae180cae00 0x7fae181570a0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae180c00e0 0x7fae180c28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 --2- 172.21.15.7:0/2191092940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae180caa30 0x7fae18156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:07:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 -- 172.21.15.7:0/2191092940 >> 172.21.15.7:0/2191092940 conn(0x7fae18066e40 msgr2=0x7fae180c7720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:07:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 -- 172.21.15.7:0/2191092940 shutdown_connections 2023-03-03T12:07:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:07:52.655+0000 7fae1da00700 1 -- 172.21.15.7:0/2191092940 wait complete. 2023-03-03T12:07:52.660 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:07:52.670 INFO:tasks.mds_thrash.fs.[b]:waiting for 33 secs before thrashing 2023-03-03T12:07:53.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:07:53.846 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:23.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:23.979 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:25.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:08:25.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.807+0000 7f12589b7700 1 Processor -- start 2023-03-03T12:08:25.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.807+0000 7f12589b7700 1 -- start start 2023-03-03T12:08:25.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.808+0000 7f12589b7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f12540c00e0 0x7f12540c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:25.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.808+0000 7f12589b7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 0x7f12540cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:25.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.808+0000 7f12589b7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12540c7310 con 0x7f12540caa30 2023-03-03T12:08:25.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.808+0000 7f12589b7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12540bfd10 con 0x7f12540c00e0 2023-03-03T12:08:25.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.808+0000 7f12589b7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12540bfa30 con 0x7f1254154ff0 2023-03-03T12:08:25.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1252d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 0x7f12540cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:25.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1251d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f12540c00e0 0x7f12540c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:25.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1251d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f12540c00e0 0x7f12540c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:35046/0 (socket says 172.21.15.7:35046) 2023-03-03T12:08:25.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1252d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 0x7f12540cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:47346/0 (socket says 172.21.15.7:47346) 2023-03-03T12:08:25.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1251d9b700 1 -- 172.21.15.7:0/960324340 learned_addr learned my addr 172.21.15.7:0/960324340 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:08:25.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1252d9d700 1 -- 172.21.15.7:0/960324340 >> v2:172.21.15.143:3300/0 conn(0x7f12540c00e0 msgr2=0x7f12540c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:08:25.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1252d9d700 1 --2- 172.21.15.7:0/960324340 >> v2:172.21.15.143:3300/0 conn(0x7f12540c00e0 0x7f12540c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:25.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1252d9d700 1 -- 172.21.15.7:0/960324340 >> v1:172.21.15.143:6789/0 conn(0x7f12540caa30 legacy=0x7f12540c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:25.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1252d9d700 1 -- 172.21.15.7:0/960324340 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1254154510 con 0x7f1254154ff0 2023-03-03T12:08:25.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1251d9b700 1 --2- 172.21.15.7:0/960324340 >> v2:172.21.15.143:3300/0 conn(0x7f12540c00e0 0x7f12540c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:08:25.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.809+0000 7f1252d9d700 1 --2- 172.21.15.7:0/960324340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 0x7f12540cae00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f124c009d90 tx=0x7f124c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88eb9ce0256450bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:08:25.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f125159a700 1 -- 172.21.15.7:0/960324340 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c00e070 con 0x7f1254154ff0 2023-03-03T12:08:25.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f125159a700 1 -- 172.21.15.7:0/960324340 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124c003a20 con 0x7f1254154ff0 2023-03-03T12:08:25.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f125159a700 1 -- 172.21.15.7:0/960324340 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c003dd0 con 0x7f1254154ff0 2023-03-03T12:08:25.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f12589b7700 1 -- 172.21.15.7:0/960324340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 msgr2=0x7f12540cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:25.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f12589b7700 1 --2- 172.21.15.7:0/960324340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 0x7f12540cae00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f124c009d90 tx=0x7f124c00b330 comp rx=0 tx=0).stop 2023-03-03T12:08:25.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f12589b7700 1 -- 172.21.15.7:0/960324340 shutdown_connections 2023-03-03T12:08:25.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f12589b7700 1 --2- 172.21.15.7:0/960324340 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 0x7f12540cae00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:25.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f12589b7700 1 --2- 172.21.15.7:0/960324340 >> v2:172.21.15.143:3300/0 conn(0x7f12540c00e0 0x7f12540c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:25.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f12589b7700 1 -- 172.21.15.7:0/960324340 >> 172.21.15.7:0/960324340 conn(0x7f1254066e40 msgr2=0x7f1254071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:08:25.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f12589b7700 1 -- 172.21.15.7:0/960324340 shutdown_connections 2023-03-03T12:08:25.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.810+0000 7f12589b7700 1 -- 172.21.15.7:0/960324340 wait complete. 2023-03-03T12:08:25.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f12589b7700 1 Processor -- start 2023-03-03T12:08:25.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f12589b7700 1 -- start start 2023-03-03T12:08:25.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f12589b7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12540c00e0 0x7f12541dda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:25.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f12589b7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12540caa30 0x7f125414c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:25.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f12589b7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 0x7f125414cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:25.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f12589b7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125414d160 con 0x7f12540c00e0 2023-03-03T12:08:25.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f12589b7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125414da90 con 0x7f12540caa30 2023-03-03T12:08:25.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f12589b7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12541de3b0 con 0x7f1254154ff0 2023-03-03T12:08:25.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f125259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12540c00e0 0x7f12541dda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:25.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f125259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12540c00e0 0x7f12541dda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39276/0 (socket says 172.21.15.7:39276) 2023-03-03T12:08:25.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f125259c700 1 -- 172.21.15.7:0/673863925 learned_addr learned my addr 172.21.15.7:0/673863925 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:08:25.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f1252d9d700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 0x7f125414cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:25.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.812+0000 7f1251d9b700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12540caa30 0x7f125414c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:25.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f125259c700 1 -- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 msgr2=0x7f125414cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:25.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f125259c700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 0x7f125414cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:25.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f125259c700 1 -- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12540caa30 msgr2=0x7f125414c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:25.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f125259c700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12540caa30 0x7f125414c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:25.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f125259c700 1 -- 172.21.15.7:0/673863925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1244009500 con 0x7f12540c00e0 2023-03-03T12:08:25.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f125259c700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12540c00e0 0x7f12541dda40 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f124800c780 tx=0x7f124800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:08:25.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f1242ffd700 1 -- 172.21.15.7:0/673863925 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12480054e0 con 0x7f12540c00e0 2023-03-03T12:08:25.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f1242ffd700 1 -- 172.21.15.7:0/673863925 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1248003680 con 0x7f12540c00e0 2023-03-03T12:08:25.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f1242ffd700 1 -- 172.21.15.7:0/673863925 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248003a30 con 0x7f12540c00e0 2023-03-03T12:08:25.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f12589b7700 1 -- 172.21.15.7:0/673863925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124c009a70 con 0x7f12540c00e0 2023-03-03T12:08:25.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.813+0000 7f12589b7700 1 -- 172.21.15.7:0/673863925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1254154ad0 con 0x7f12540c00e0 2023-03-03T12:08:25.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.814+0000 7f12589b7700 1 -- 172.21.15.7:0/673863925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12541547f0 con 0x7f12540c00e0 2023-03-03T12:08:25.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.815+0000 7f1242ffd700 1 -- 172.21.15.7:0/673863925 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1248019030 con 0x7f12540c00e0 2023-03-03T12:08:25.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.815+0000 7f1242ffd700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12340682f0 0x7f123406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:08:25.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.815+0000 7f1242ffd700 1 -- 172.21.15.7:0/673863925 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1248085590 con 0x7f12540c00e0 2023-03-03T12:08:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.815+0000 7f1251d9b700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12340682f0 0x7f123406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:08:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.816+0000 7f1251d9b700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12340682f0 0x7f123406a7a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f12540c45b0 tx=0x7f1244009290 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:08:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:25.820+0000 7f1242ffd700 1 -- 172.21.15.7:0/673863925 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f12480518b0 con 0x7f12540c00e0 2023-03-03T12:08:26.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.058+0000 7f12589b7700 1 -- 172.21.15.7:0/673863925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12540bf650 con 0x7f12540c00e0 2023-03-03T12:08:26.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.059+0000 7f1242ffd700 1 -- 172.21.15.7:0/673863925 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1248054ed0 con 0x7f12540c00e0 2023-03-03T12:08:26.061 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:08:26.061 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:08:26.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 -- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12340682f0 msgr2=0x7f123406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:26.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12340682f0 0x7f123406a7a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f12540c45b0 tx=0x7f1244009290 comp rx=0 tx=0).stop 2023-03-03T12:08:26.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 -- 172.21.15.7:0/673863925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12540c00e0 msgr2=0x7f12541dda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:08:26.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12540c00e0 0x7f12541dda40 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f124800c780 tx=0x7f124800ca90 comp rx=0 tx=0).stop 2023-03-03T12:08:26.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 -- 172.21.15.7:0/673863925 shutdown_connections 2023-03-03T12:08:26.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1254154ff0 0x7f125414cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:26.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12340682f0 0x7f123406a7a0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:26.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12540c00e0 0x7f12541dda40 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:26.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 --2- 172.21.15.7:0/673863925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12540caa30 0x7f125414c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:08:26.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 -- 172.21.15.7:0/673863925 >> 172.21.15.7:0/673863925 conn(0x7f1254066e40 msgr2=0x7f1254152100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:08:26.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 -- 172.21.15.7:0/673863925 shutdown_connections 2023-03-03T12:08:26.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:08:26.060+0000 7f12589b7700 1 -- 172.21.15.7:0/673863925 wait complete. 2023-03-03T12:08:26.064 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:08:26.075 INFO:tasks.mds_thrash.fs.[b]:waiting for 79 secs before thrashing 2023-03-03T12:08:54.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:08:54.082 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:00.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:09:00.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.448+0000 7f8509ee7700 1 Processor -- start 2023-03-03T12:09:00.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.448+0000 7f8509ee7700 1 -- start start 2023-03-03T12:09:00.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.449+0000 7f8509ee7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414f9d0 0x7f8504156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:00.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.449+0000 7f8509ee7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f85040abe20 0x7f85041529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:00.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.449+0000 7f8509ee7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85040c2f30 con 0x7f850414fda0 2023-03-03T12:09:00.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.449+0000 7f8509ee7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85040c31d0 con 0x7f85040abe20 2023-03-03T12:09:00.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.449+0000 7f8509ee7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85041524a0 con 0x7f850414f9d0 2023-03-03T12:09:00.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.449+0000 7f8503fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f85040abe20 0x7f85041529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:00.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.449+0000 7f8503fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f85040abe20 0x7f85041529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:35084/0 (socket says 172.21.15.7:35084) 2023-03-03T12:09:00.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.449+0000 7f8502ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f850414fda0 0x7f8504071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:36806/0 (socket says 172.21.15.7:36806) 2023-03-03T12:09:00.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.449+0000 7f8503fff700 1 -- 172.21.15.7:0/1527240682 learned_addr learned my addr 172.21.15.7:0/1527240682 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:09:00.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.449+0000 7f85037fe700 1 --2- 172.21.15.7:0/1527240682 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414f9d0 0x7f8504156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:00.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.450+0000 7f85037fe700 1 -- 172.21.15.7:0/1527240682 >> v2:172.21.15.143:3300/0 conn(0x7f85040abe20 msgr2=0x7f85041529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:09:00.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.450+0000 7f85037fe700 1 --2- 172.21.15.7:0/1527240682 >> v2:172.21.15.143:3300/0 conn(0x7f85040abe20 0x7f85041529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:00.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.450+0000 7f85037fe700 1 -- 172.21.15.7:0/1527240682 >> v1:172.21.15.143:6789/0 conn(0x7f850414fda0 legacy=0x7f8504071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:00.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.450+0000 7f85037fe700 1 -- 172.21.15.7:0/1527240682 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85040c34b0 con 0x7f850414f9d0 2023-03-03T12:09:00.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.450+0000 7f8503fff700 1 --2- 172.21.15.7:0/1527240682 >> v2:172.21.15.143:3300/0 conn(0x7f85040abe20 0x7f85041529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:09:00.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.450+0000 7f85037fe700 1 --2- 172.21.15.7:0/1527240682 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414f9d0 0x7f8504156430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009840 tx=0x7f84ec009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b492a3e83c46211f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:00.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.450+0000 7f85027fc700 1 -- 172.21.15.7:0/1527240682 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec011830 con 0x7f850414f9d0 2023-03-03T12:09:00.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.450+0000 7f85027fc700 1 -- 172.21.15.7:0/1527240682 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec011990 con 0x7f850414f9d0 2023-03-03T12:09:00.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.450+0000 7f85027fc700 1 -- 172.21.15.7:0/1527240682 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec011d20 con 0x7f850414f9d0 2023-03-03T12:09:00.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.451+0000 7f8509ee7700 1 -- 172.21.15.7:0/1527240682 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414f9d0 msgr2=0x7f8504156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:00.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.451+0000 7f8509ee7700 1 --2- 172.21.15.7:0/1527240682 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414f9d0 0x7f8504156430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009840 tx=0x7f84ec009b50 comp rx=0 tx=0).stop 2023-03-03T12:09:00.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.451+0000 7f8509ee7700 1 -- 172.21.15.7:0/1527240682 shutdown_connections 2023-03-03T12:09:00.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.451+0000 7f8509ee7700 1 --2- 172.21.15.7:0/1527240682 >> v2:172.21.15.143:3300/0 conn(0x7f85040abe20 0x7f85041529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:00.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.451+0000 7f8509ee7700 1 --2- 172.21.15.7:0/1527240682 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414f9d0 0x7f8504156430 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:00.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.451+0000 7f8509ee7700 1 -- 172.21.15.7:0/1527240682 >> 172.21.15.7:0/1527240682 conn(0x7f85040c8900 msgr2=0x7f85040c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:09:00.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.451+0000 7f8509ee7700 1 -- 172.21.15.7:0/1527240682 shutdown_connections 2023-03-03T12:09:00.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.451+0000 7f8509ee7700 1 -- 172.21.15.7:0/1527240682 wait complete. 2023-03-03T12:09:00.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.452+0000 7f8509ee7700 1 Processor -- start 2023-03-03T12:09:00.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.452+0000 7f8509ee7700 1 -- start start 2023-03-03T12:09:00.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f8509ee7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040abe20 0x7f85040c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:00.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f8509ee7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850414f9d0 0x7f850414dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:00.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f8509ee7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414fda0 0x7f850414e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:00.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f8509ee7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85041dd5d0 con 0x7f85040abe20 2023-03-03T12:09:00.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f8509ee7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85040be420 con 0x7f850414f9d0 2023-03-03T12:09:00.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f8509ee7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85040c1e90 con 0x7f850414fda0 2023-03-03T12:09:00.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f85037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040abe20 0x7f85040c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:00.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f85037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040abe20 0x7f85040c3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39312/0 (socket says 172.21.15.7:39312) 2023-03-03T12:09:00.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f85037fe700 1 -- 172.21.15.7:0/2246634700 learned_addr learned my addr 172.21.15.7:0/2246634700 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:09:00.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f8503fff700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414fda0 0x7f850414e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:00.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f8502ffd700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850414f9d0 0x7f850414dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:00.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f85037fe700 1 -- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414fda0 msgr2=0x7f850414e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:00.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f85037fe700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414fda0 0x7f850414e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:00.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f85037fe700 1 -- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850414f9d0 msgr2=0x7f850414dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:00.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f85037fe700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850414f9d0 0x7f850414dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f85037fe700 1 -- 172.21.15.7:0/2246634700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc009b30 con 0x7f85040abe20 2023-03-03T12:09:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f8502ffd700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850414f9d0 0x7f850414dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:09:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.453+0000 7f85037fe700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040abe20 0x7f85040c3fc0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009790 tx=0x7f84ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.454+0000 7f84fbfff700 1 -- 172.21.15.7:0/2246634700 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec004570 con 0x7f85040abe20 2023-03-03T12:09:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.454+0000 7f84fbfff700 1 -- 172.21.15.7:0/2246634700 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec01f070 con 0x7f85040abe20 2023-03-03T12:09:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.454+0000 7f8509ee7700 1 -- 172.21.15.7:0/2246634700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ec0094a0 con 0x7f85040abe20 2023-03-03T12:09:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.454+0000 7f84fbfff700 1 -- 172.21.15.7:0/2246634700 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec0038c0 con 0x7f85040abe20 2023-03-03T12:09:00.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.454+0000 7f8509ee7700 1 -- 172.21.15.7:0/2246634700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85040ac350 con 0x7f85040abe20 2023-03-03T12:09:00.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.455+0000 7f8509ee7700 1 -- 172.21.15.7:0/2246634700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85040ab630 con 0x7f85040abe20 2023-03-03T12:09:00.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.455+0000 7f84fbfff700 1 -- 172.21.15.7:0/2246634700 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f84ec02a020 con 0x7f85040abe20 2023-03-03T12:09:00.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.456+0000 7f84fbfff700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 0x7f84e406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:00.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.456+0000 7f84fbfff700 1 -- 172.21.15.7:0/2246634700 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f84ec0876a0 con 0x7f85040abe20 2023-03-03T12:09:00.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.456+0000 7f8502ffd700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 0x7f84e406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:00.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.456+0000 7f8502ffd700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 0x7f84e406a7d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f85040ad150 tx=0x7f84f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:00.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.460+0000 7f84fbfff700 1 -- 172.21.15.7:0/2246634700 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f84ec053a40 con 0x7f85040abe20 2023-03-03T12:09:00.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.700+0000 7f8509ee7700 1 -- 172.21.15.7:0/2246634700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8504071fa0 con 0x7f85040abe20 2023-03-03T12:09:00.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.701+0000 7f84fbfff700 1 -- 172.21.15.7:0/2246634700 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f84ec057060 con 0x7f85040abe20 2023-03-03T12:09:00.703 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:09:00.704 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:09:00.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 -- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 msgr2=0x7f84e406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:00.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 0x7f84e406a7d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f85040ad150 tx=0x7f84f4009040 comp rx=0 tx=0).stop 2023-03-03T12:09:00.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 -- 172.21.15.7:0/2246634700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040abe20 msgr2=0x7f85040c3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:00.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040abe20 0x7f85040c3fc0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f84ec009790 tx=0x7f84ec004750 comp rx=0 tx=0).stop 2023-03-03T12:09:00.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 -- 172.21.15.7:0/2246634700 shutdown_connections 2023-03-03T12:09:00.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f850414fda0 0x7f850414e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:00.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 0x7f84e406a7d0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:00.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040abe20 0x7f85040c3fc0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:00.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 --2- 172.21.15.7:0/2246634700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850414f9d0 0x7f850414dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:00.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 -- 172.21.15.7:0/2246634700 >> 172.21.15.7:0/2246634700 conn(0x7f85040c8900 msgr2=0x7f8504150200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:09:00.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 -- 172.21.15.7:0/2246634700 shutdown_connections 2023-03-03T12:09:00.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:00.703+0000 7f8509ee7700 1 -- 172.21.15.7:0/2246634700 wait complete. 2023-03-03T12:09:00.707 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:09:00.717 INFO:tasks.mds_thrash.fs.[a]:waiting for 57 secs before thrashing 2023-03-03T12:09:24.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:24.204 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:45.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:09:45.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.214+0000 7f0edf6fb700 1 Processor -- start 2023-03-03T12:09:45.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.214+0000 7f0edf6fb700 1 -- start start 2023-03-03T12:09:45.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0edf6fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ed8155390 0x7f0ed80c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:45.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0edf6fb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0ed806e020 0x7f0ed8071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:45.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0edf6fb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0ed8072520 0x7f0ed8151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:45.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0edf6fb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed8070b90 con 0x7f0ed8072520 2023-03-03T12:09:45.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0edf6fb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed8151320 con 0x7f0ed806e020 2023-03-03T12:09:45.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0edf6fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed80c4780 con 0x7f0ed8155390 2023-03-03T12:09:45.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0eddc98700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0ed8072520 0x7f0ed8151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:45.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0eddc98700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0ed8072520 0x7f0ed8151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39350/0 (socket says 172.21.15.7:39350) 2023-03-03T12:09:45.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0eddc98700 1 -- 172.21.15.7:0/2486428849 learned_addr learned my addr 172.21.15.7:0/2486428849 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:09:45.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0edd497700 1 --2- 172.21.15.7:0/2486428849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ed8155390 0x7f0ed80c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:45.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0edcc96700 1 --2- 172.21.15.7:0/2486428849 >> v2:172.21.15.143:3300/0 conn(0x7f0ed806e020 0x7f0ed8071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:45.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0eddc98700 1 -- 172.21.15.7:0/2486428849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ed8155390 msgr2=0x7f0ed80c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:09:45.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0eddc98700 1 --2- 172.21.15.7:0/2486428849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ed8155390 0x7f0ed80c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0eddc98700 1 -- 172.21.15.7:0/2486428849 >> v2:172.21.15.143:3300/0 conn(0x7f0ed806e020 msgr2=0x7f0ed8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:09:45.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0eddc98700 1 --2- 172.21.15.7:0/2486428849 >> v2:172.21.15.143:3300/0 conn(0x7f0ed806e020 0x7f0ed8071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.215+0000 7f0eddc98700 1 -- 172.21.15.7:0/2486428849 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed8154910 con 0x7f0ed8072520 2023-03-03T12:09:45.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0eddc98700 1 --2- 172.21.15.7:0/2486428849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 0x7f0ed8151770 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f0ed806f090 tx=0x7f0ed000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd312fd7dbb62bb1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:45.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0ecffff700 1 -- 172.21.15.7:0/2486428849 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000e070 con 0x7f0ed8072520 2023-03-03T12:09:45.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0ecffff700 1 -- 172.21.15.7:0/2486428849 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed000b910 con 0x7f0ed8072520 2023-03-03T12:09:45.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0ecffff700 1 -- 172.21.15.7:0/2486428849 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000bcc0 con 0x7f0ed8072520 2023-03-03T12:09:45.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0edf6fb700 1 -- 172.21.15.7:0/2486428849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 msgr2=0x7f0ed8151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:45.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0edf6fb700 1 --2- 172.21.15.7:0/2486428849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 0x7f0ed8151770 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f0ed806f090 tx=0x7f0ed000b330 comp rx=0 tx=0).stop 2023-03-03T12:09:45.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0edf6fb700 1 -- 172.21.15.7:0/2486428849 shutdown_connections 2023-03-03T12:09:45.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0edf6fb700 1 --2- 172.21.15.7:0/2486428849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 0x7f0ed8151770 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0edf6fb700 1 --2- 172.21.15.7:0/2486428849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ed8155390 0x7f0ed80c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0edf6fb700 1 --2- 172.21.15.7:0/2486428849 >> v2:172.21.15.143:3300/0 conn(0x7f0ed806e020 0x7f0ed8071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.216+0000 7f0edf6fb700 1 -- 172.21.15.7:0/2486428849 >> 172.21.15.7:0/2486428849 conn(0x7f0ed806c110 msgr2=0x7f0ed806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:09:45.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.217+0000 7f0edf6fb700 1 -- 172.21.15.7:0/2486428849 shutdown_connections 2023-03-03T12:09:45.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.217+0000 7f0edf6fb700 1 -- 172.21.15.7:0/2486428849 wait complete. 2023-03-03T12:09:45.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.218+0000 7f0edf6fb700 1 Processor -- start 2023-03-03T12:09:45.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.218+0000 7f0edf6fb700 1 -- start start 2023-03-03T12:09:45.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.218+0000 7f0edf6fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ed806e020 0x7f0ed814dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:45.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.218+0000 7f0edf6fb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 0x7f0ed81e9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:45.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.218+0000 7f0edf6fb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ed8155390 0x7f0ed814f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:45.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.218+0000 7f0edf6fb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed814fdb0 con 0x7f0ed8072520 2023-03-03T12:09:45.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.218+0000 7f0edf6fb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed81e9cc0 con 0x7f0ed8155390 2023-03-03T12:09:45.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.218+0000 7f0edf6fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed81e7410 con 0x7f0ed806e020 2023-03-03T12:09:45.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edcc96700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 0x7f0ed81e9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:45.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edcc96700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 0x7f0ed81e9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39354/0 (socket says 172.21.15.7:39354) 2023-03-03T12:09:45.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edcc96700 1 -- 172.21.15.7:0/16358240 learned_addr learned my addr 172.21.15.7:0/16358240 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:09:45.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edd497700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ed806e020 0x7f0ed814dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:45.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0eddc98700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ed8155390 0x7f0ed814f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:45.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edcc96700 1 -- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ed806e020 msgr2=0x7f0ed814dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:45.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edcc96700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ed806e020 0x7f0ed814dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edcc96700 1 -- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ed8155390 msgr2=0x7f0ed814f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:45.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edcc96700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ed8155390 0x7f0ed814f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edcc96700 1 -- 172.21.15.7:0/16358240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed0009b30 con 0x7f0ed8072520 2023-03-03T12:09:45.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edcc96700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 0x7f0ed81e9780 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400e930 tx=0x7f0ed400ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:45.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0ecdffb700 1 -- 172.21.15.7:0/16358240 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4009690 con 0x7f0ed8072520 2023-03-03T12:09:45.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0ecdffb700 1 -- 172.21.15.7:0/16358240 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed400c850 con 0x7f0ed8072520 2023-03-03T12:09:45.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0edf6fb700 1 -- 172.21.15.7:0/16358240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed8070540 con 0x7f0ed8072520 2023-03-03T12:09:45.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.219+0000 7f0ecdffb700 1 -- 172.21.15.7:0/16358240 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed400cbe0 con 0x7f0ed8072520 2023-03-03T12:09:45.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.220+0000 7f0edf6fb700 1 -- 172.21.15.7:0/16358240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed80702a0 con 0x7f0ed8072520 2023-03-03T12:09:45.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.220+0000 7f0edf6fb700 1 -- 172.21.15.7:0/16358240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed8070820 con 0x7f0ed8072520 2023-03-03T12:09:45.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.221+0000 7f0ecdffb700 1 -- 172.21.15.7:0/16358240 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0ed4019030 con 0x7f0ed8072520 2023-03-03T12:09:45.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.221+0000 7f0ecdffb700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ebc0682f0 0x7f0ebc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:45.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.221+0000 7f0ecdffb700 1 -- 172.21.15.7:0/16358240 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0ed4086100 con 0x7f0ed8072520 2023-03-03T12:09:45.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.222+0000 7f0edd497700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ebc0682f0 0x7f0ebc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:45.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.222+0000 7f0edd497700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ebc0682f0 0x7f0ebc06a7a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800b240 tx=0x7f0ec800b420 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:45.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.226+0000 7f0ecdffb700 1 -- 172.21.15.7:0/16358240 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0ed4052440 con 0x7f0ed8072520 2023-03-03T12:09:45.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.464+0000 7f0edf6fb700 1 -- 172.21.15.7:0/16358240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ed8074bb0 con 0x7f0ed8072520 2023-03-03T12:09:45.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.465+0000 7f0ecdffb700 1 -- 172.21.15.7:0/16358240 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0ed4055a60 con 0x7f0ed8072520 2023-03-03T12:09:45.467 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:09:45.467 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:09:45.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 -- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ebc0682f0 msgr2=0x7f0ebc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:45.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ebc0682f0 0x7f0ebc06a7a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800b240 tx=0x7f0ec800b420 comp rx=0 tx=0).stop 2023-03-03T12:09:45.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 -- 172.21.15.7:0/16358240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 msgr2=0x7f0ed81e9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:45.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 0x7f0ed81e9780 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400e930 tx=0x7f0ed400ecf0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 -- 172.21.15.7:0/16358240 shutdown_connections 2023-03-03T12:09:45.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ed8155390 0x7f0ed814f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ed806e020 0x7f0ed814dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ebc0682f0 0x7f0ebc06a7a0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 --2- 172.21.15.7:0/16358240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ed8072520 0x7f0ed81e9780 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 -- 172.21.15.7:0/16358240 >> 172.21.15.7:0/16358240 conn(0x7f0ed806c110 msgr2=0x7f0ed80c2340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:09:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 -- 172.21.15.7:0/16358240 shutdown_connections 2023-03-03T12:09:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:45.467+0000 7f0edf6fb700 1 -- 172.21.15.7:0/16358240 wait complete. 2023-03-03T12:09:45.471 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:09:45.480 INFO:tasks.mds_thrash.fs.[b]:waiting for 21 secs before thrashing 2023-03-03T12:09:54.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:54.351 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:09:57.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:09:57.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.854+0000 7fc5f01e1700 1 Processor -- start 2023-03-03T12:09:57.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.854+0000 7fc5f01e1700 1 -- start start 2023-03-03T12:09:57.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.855+0000 7fc5f01e1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc5e814fb40 0x7fc5e80c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:57.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.855+0000 7fc5f01e1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e80c73b0 con 0x7fc5e80c3e30 2023-03-03T12:09:57.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.855+0000 7fc5f01e1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e8152fb0 con 0x7fc5e806d240 2023-03-03T12:09:57.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.855+0000 7fc5f01e1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e80c8280 con 0x7fc5e814fb40 2023-03-03T12:09:57.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.855+0000 7fc5edf7d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc5e814fb40 0x7fc5e80c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:57.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.855+0000 7fc5ee77e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc5e80c3e30 0x7fc5e815cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33350/0 (socket says 172.21.15.7:33350) 2023-03-03T12:09:57.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.855+0000 7fc5ee77e700 1 -- 172.21.15.7:0/344062091 learned_addr learned my addr 172.21.15.7:0/344062091 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:09:57.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.856+0000 7fc5edf7d700 1 -- 172.21.15.7:0/344062091 >> v1:172.21.15.143:6789/0 conn(0x7fc5e806d240 legacy=0x7fc5e80c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:57.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.856+0000 7fc5edf7d700 1 -- 172.21.15.7:0/344062091 >> v1:172.21.15.7:6789/0 conn(0x7fc5e80c3e30 legacy=0x7fc5e815cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:57.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.856+0000 7fc5ecf7b700 1 -- 172.21.15.7:0/344062091 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127488568 0 0) 0x7fc5e80c73b0 con 0x7fc5e80c3e30 2023-03-03T12:09:57.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.856+0000 7fc5edf7d700 1 -- 172.21.15.7:0/344062091 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e80c7650 con 0x7fc5e814fb40 2023-03-03T12:09:57.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.856+0000 7fc5edf7d700 1 --2- 172.21.15.7:0/344062091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e814fb40 0x7fc5e80c1da0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009840 tx=0x7fc5d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3938b0f18cef784d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:57.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.856+0000 7fc5ecf7b700 1 -- 172.21.15.7:0/344062091 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8011830 con 0x7fc5e814fb40 2023-03-03T12:09:57.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.856+0000 7fc5ecf7b700 1 -- 172.21.15.7:0/344062091 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d8011990 con 0x7fc5e814fb40 2023-03-03T12:09:57.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.856+0000 7fc5ecf7b700 1 -- 172.21.15.7:0/344062091 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8011d20 con 0x7fc5e814fb40 2023-03-03T12:09:57.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.857+0000 7fc5f01e1700 1 -- 172.21.15.7:0/344062091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e814fb40 msgr2=0x7fc5e80c1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:57.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.857+0000 7fc5f01e1700 1 --2- 172.21.15.7:0/344062091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e814fb40 0x7fc5e80c1da0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009840 tx=0x7fc5d8009b50 comp rx=0 tx=0).stop 2023-03-03T12:09:57.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.857+0000 7fc5f01e1700 1 -- 172.21.15.7:0/344062091 shutdown_connections 2023-03-03T12:09:57.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.857+0000 7fc5f01e1700 1 --2- 172.21.15.7:0/344062091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e814fb40 0x7fc5e80c1da0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:57.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.857+0000 7fc5f01e1700 1 -- 172.21.15.7:0/344062091 >> 172.21.15.7:0/344062091 conn(0x7fc5e806c110 msgr2=0x7fc5e806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:09:57.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.857+0000 7fc5f01e1700 1 -- 172.21.15.7:0/344062091 shutdown_connections 2023-03-03T12:09:57.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.857+0000 7fc5f01e1700 1 -- 172.21.15.7:0/344062091 wait complete. 2023-03-03T12:09:57.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.858+0000 7fc5f01e1700 1 Processor -- start 2023-03-03T12:09:57.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.858+0000 7fc5f01e1700 1 -- start start 2023-03-03T12:09:57.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5f01e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e806d240 0x7fc5e81eb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:57.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5f01e1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5e80c3e30 0x7fc5e8147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:57.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5f01e1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5e814fb40 0x7fc5e8147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:57.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5f01e1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e81ebd80 con 0x7fc5e806d240 2023-03-03T12:09:57.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5f01e1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e814df40 con 0x7fc5e814fb40 2023-03-03T12:09:57.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5f01e1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e814b620 con 0x7fc5e80c3e30 2023-03-03T12:09:57.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5edf7d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e806d240 0x7fc5e81eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:57.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5edf7d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e806d240 0x7fc5e81eb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39372/0 (socket says 172.21.15.7:39372) 2023-03-03T12:09:57.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5edf7d700 1 -- 172.21.15.7:0/2796630713 learned_addr learned my addr 172.21.15.7:0/2796630713 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:09:57.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5edf7d700 1 -- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5e80c3e30 msgr2=0x7fc5e8147960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:57.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5edf7d700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5e80c3e30 0x7fc5e8147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5edf7d700 1 -- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5e814fb40 msgr2=0x7fc5e8147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5edf7d700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5e814fb40 0x7fc5e8147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5edf7d700 1 -- 172.21.15.7:0/2796630713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d80094a0 con 0x7fc5e806d240 2023-03-03T12:09:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5edf7d700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e806d240 0x7fc5e81eb300 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8014040 tx=0x7fc5d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5de7fc700 1 -- 172.21.15.7:0/2796630713 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d801a900 con 0x7fc5e806d240 2023-03-03T12:09:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5de7fc700 1 -- 172.21.15.7:0/2796630713 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d801aa60 con 0x7fc5e806d240 2023-03-03T12:09:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5de7fc700 1 -- 172.21.15.7:0/2796630713 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d801add0 con 0x7fc5e806d240 2023-03-03T12:09:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.859+0000 7fc5f01e1700 1 -- 172.21.15.7:0/2796630713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e80740f0 con 0x7fc5e806d240 2023-03-03T12:09:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.860+0000 7fc5f01e1700 1 -- 172.21.15.7:0/2796630713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e8153290 con 0x7fc5e806d240 2023-03-03T12:09:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.860+0000 7fc5f01e1700 1 -- 172.21.15.7:0/2796630713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e8074280 con 0x7fc5e806d240 2023-03-03T12:09:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.861+0000 7fc5de7fc700 1 -- 172.21.15.7:0/2796630713 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc5d800f050 con 0x7fc5e806d240 2023-03-03T12:09:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.861+0000 7fc5de7fc700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5cc0682f0 0x7fc5cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:09:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.861+0000 7fc5de7fc700 1 -- 172.21.15.7:0/2796630713 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc5d8087160 con 0x7fc5e806d240 2023-03-03T12:09:57.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.861+0000 7fc5ed77c700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5cc0682f0 0x7fc5cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:09:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.862+0000 7fc5ed77c700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5cc0682f0 0x7fc5cc06a7a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4000f60 tx=0x7fc5e4007a40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:09:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:57.866+0000 7fc5de7fc700 1 -- 172.21.15.7:0/2796630713 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5d80534c0 con 0x7fc5e806d240 2023-03-03T12:09:58.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.110+0000 7fc5f01e1700 1 -- 172.21.15.7:0/2796630713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5e814b480 con 0x7fc5e806d240 2023-03-03T12:09:58.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.111+0000 7fc5de7fc700 1 -- 172.21.15.7:0/2796630713 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc5e814b480 con 0x7fc5e806d240 2023-03-03T12:09:58.114 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:09:58.114 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:09:58.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 -- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5cc0682f0 msgr2=0x7fc5cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:58.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5cc0682f0 0x7fc5cc06a7a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4000f60 tx=0x7fc5e4007a40 comp rx=0 tx=0).stop 2023-03-03T12:09:58.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 -- 172.21.15.7:0/2796630713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e806d240 msgr2=0x7fc5e81eb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:09:58.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e806d240 0x7fc5e81eb300 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8014040 tx=0x7fc5d8004750 comp rx=0 tx=0).stop 2023-03-03T12:09:58.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 -- 172.21.15.7:0/2796630713 shutdown_connections 2023-03-03T12:09:58.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5e814fb40 0x7fc5e8147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:58.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5cc0682f0 0x7fc5cc06a7a0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:58.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5e806d240 0x7fc5e81eb300 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:58.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 --2- 172.21.15.7:0/2796630713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5e80c3e30 0x7fc5e8147960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:09:58.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 -- 172.21.15.7:0/2796630713 >> 172.21.15.7:0/2796630713 conn(0x7fc5e806c110 msgr2=0x7fc5e806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:09:58.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 -- 172.21.15.7:0/2796630713 shutdown_connections 2023-03-03T12:09:58.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:09:58.113+0000 7fc5f01e1700 1 -- 172.21.15.7:0/2796630713 wait complete. 2023-03-03T12:09:58.117 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:09:58.127 INFO:tasks.mds_thrash.fs.[a]:waiting for 88 secs before thrashing 2023-03-03T12:10:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T12:09:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T12:10:06.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:10:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.620+0000 7fa492c88700 1 Processor -- start 2023-03-03T12:10:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.620+0000 7fa492c88700 1 -- start start 2023-03-03T12:10:06.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa492c88700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa48c14f9d0 0x7fa48c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:06.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa492c88700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c14fda0 0x7fa48c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:06.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa492c88700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48c0c2f30 con 0x7fa48c0abe20 2023-03-03T12:10:06.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa492c88700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48c0c31d0 con 0x7fa48c14f9d0 2023-03-03T12:10:06.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa492c88700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c1524a0 con 0x7fa48c14fda0 2023-03-03T12:10:06.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa490a24700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa48c14f9d0 0x7fa48c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa48bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c14fda0 0x7fa48c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa490a24700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa48c14f9d0 0x7fa48c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:35156/0 (socket says 172.21.15.7:35156) 2023-03-03T12:10:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa48bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c14fda0 0x7fa48c071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:47456/0 (socket says 172.21.15.7:47456) 2023-03-03T12:10:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa490a24700 1 -- 172.21.15.7:0/244340995 learned_addr learned my addr 172.21.15.7:0/244340995 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:10:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.621+0000 7fa491225700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fa48c0abe20 0x7fa48c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:36886/0 (socket says 172.21.15.7:36886) 2023-03-03T12:10:06.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.622+0000 7fa490a24700 1 -- 172.21.15.7:0/244340995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c14fda0 msgr2=0x7fa48c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:10:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.622+0000 7fa490a24700 1 --2- 172.21.15.7:0/244340995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c14fda0 0x7fa48c071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.622+0000 7fa490a24700 1 -- 172.21.15.7:0/244340995 >> v1:172.21.15.143:6789/0 conn(0x7fa48c0abe20 legacy=0x7fa48c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.622+0000 7fa490a24700 1 -- 172.21.15.7:0/244340995 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c0c34b0 con 0x7fa48c14f9d0 2023-03-03T12:10:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.622+0000 7fa48bfff700 1 --2- 172.21.15.7:0/244340995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c14fda0 0x7fa48c071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:10:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.622+0000 7fa490a24700 1 --2- 172.21.15.7:0/244340995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c14f9d0 0x7fa48c156430 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa4800098f0 tx=0x7fa480009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d0b6a499cb1ab5f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:10:06.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.622+0000 7fa48b7fe700 1 -- 172.21.15.7:0/244340995 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480011830 con 0x7fa48c14f9d0 2023-03-03T12:10:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.622+0000 7fa48b7fe700 1 -- 172.21.15.7:0/244340995 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480011990 con 0x7fa48c14f9d0 2023-03-03T12:10:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.622+0000 7fa48b7fe700 1 -- 172.21.15.7:0/244340995 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480011d20 con 0x7fa48c14f9d0 2023-03-03T12:10:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.623+0000 7fa492c88700 1 -- 172.21.15.7:0/244340995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c14f9d0 msgr2=0x7fa48c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.623+0000 7fa492c88700 1 --2- 172.21.15.7:0/244340995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c14f9d0 0x7fa48c156430 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa4800098f0 tx=0x7fa480009c00 comp rx=0 tx=0).stop 2023-03-03T12:10:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.623+0000 7fa492c88700 1 -- 172.21.15.7:0/244340995 shutdown_connections 2023-03-03T12:10:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.623+0000 7fa492c88700 1 --2- 172.21.15.7:0/244340995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c14f9d0 0x7fa48c156430 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:06.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.623+0000 7fa492c88700 1 --2- 172.21.15.7:0/244340995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c14fda0 0x7fa48c071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:06.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.623+0000 7fa492c88700 1 -- 172.21.15.7:0/244340995 >> 172.21.15.7:0/244340995 conn(0x7fa48c0c8900 msgr2=0x7fa48c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:10:06.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.623+0000 7fa492c88700 1 -- 172.21.15.7:0/244340995 shutdown_connections 2023-03-03T12:10:06.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.623+0000 7fa492c88700 1 -- 172.21.15.7:0/244340995 wait complete. 2023-03-03T12:10:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.624+0000 7fa492c88700 1 Processor -- start 2023-03-03T12:10:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.624+0000 7fa492c88700 1 -- start start 2023-03-03T12:10:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa492c88700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c0abe20 0x7fa48c0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa492c88700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c14f9d0 0x7fa48c143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:06.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa492c88700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa48c14fda0 0x7fa48c144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:06.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa492c88700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c0c2740 con 0x7fa48c14fda0 2023-03-03T12:10:06.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa492c88700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c06a450 con 0x7fa48c14f9d0 2023-03-03T12:10:06.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa492c88700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c06a730 con 0x7fa48c0abe20 2023-03-03T12:10:06.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa491225700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa48c14fda0 0x7fa48c144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:06.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa491225700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa48c14fda0 0x7fa48c144080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39390/0 (socket says 172.21.15.7:39390) 2023-03-03T12:10:06.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa491225700 1 -- 172.21.15.7:0/2920413305 learned_addr learned my addr 172.21.15.7:0/2920413305 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:10:06.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa48bfff700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c14f9d0 0x7fa48c143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:06.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.625+0000 7fa490a24700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c0abe20 0x7fa48c0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:06.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa491225700 1 -- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c0abe20 msgr2=0x7fa48c0c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:06.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa491225700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c0abe20 0x7fa48c0c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:06.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa491225700 1 -- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c14f9d0 msgr2=0x7fa48c143b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:06.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa491225700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c14f9d0 0x7fa48c143b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:06.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa491225700 1 -- 172.21.15.7:0/2920413305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4780094a0 con 0x7fa48c14fda0 2023-03-03T12:10:06.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa48bfff700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c14f9d0 0x7fa48c143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:10:06.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa491225700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa48c14fda0 0x7fa48c144080 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fa48400cbc0 tx=0x7fa48400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:10:06.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa4897fa700 1 -- 172.21.15.7:0/2920413305 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48400a9e0 con 0x7fa48c14fda0 2023-03-03T12:10:06.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa4897fa700 1 -- 172.21.15.7:0/2920413305 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48400ab40 con 0x7fa48c14fda0 2023-03-03T12:10:06.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa4897fa700 1 -- 172.21.15.7:0/2920413305 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa484007070 con 0x7fa48c14fda0 2023-03-03T12:10:06.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa492c88700 1 -- 172.21.15.7:0/2920413305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa480009550 con 0x7fa48c14fda0 2023-03-03T12:10:06.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.626+0000 7fa492c88700 1 -- 172.21.15.7:0/2920413305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48c0ac350 con 0x7fa48c14fda0 2023-03-03T12:10:06.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.627+0000 7fa492c88700 1 -- 172.21.15.7:0/2920413305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48c0ab630 con 0x7fa48c14fda0 2023-03-03T12:10:06.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.627+0000 7fa4897fa700 1 -- 172.21.15.7:0/2920413305 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa484019030 con 0x7fa48c14fda0 2023-03-03T12:10:06.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.628+0000 7fa4897fa700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4700682f0 0x7fa47006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:10:06.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.628+0000 7fa4897fa700 1 -- 172.21.15.7:0/2920413305 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa484085b90 con 0x7fa48c14fda0 2023-03-03T12:10:06.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.628+0000 7fa490a24700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4700682f0 0x7fa47006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:10:06.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.629+0000 7fa490a24700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4700682f0 0x7fa47006a7a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fa480009c40 tx=0x7fa480008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:10:06.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.632+0000 7fa4897fa700 1 -- 172.21.15.7:0/2920413305 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa48404b5e0 con 0x7fa48c14fda0 2023-03-03T12:10:06.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.870+0000 7fa492c88700 1 -- 172.21.15.7:0/2920413305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa48c0c3e10 con 0x7fa48c14fda0 2023-03-03T12:10:06.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.871+0000 7fa4897fa700 1 -- 172.21.15.7:0/2920413305 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa484017090 con 0x7fa48c14fda0 2023-03-03T12:10:06.874 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:10:06.874 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:10:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 -- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4700682f0 msgr2=0x7fa47006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:06.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4700682f0 0x7fa47006a7a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fa480009c40 tx=0x7fa480008030 comp rx=0 tx=0).stop 2023-03-03T12:10:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 -- 172.21.15.7:0/2920413305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa48c14fda0 msgr2=0x7fa48c144080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:10:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa48c14fda0 0x7fa48c144080 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fa48400cbc0 tx=0x7fa48400ced0 comp rx=0 tx=0).stop 2023-03-03T12:10:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 -- 172.21.15.7:0/2920413305 shutdown_connections 2023-03-03T12:10:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4700682f0 0x7fa47006a7a0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa48c14fda0 0x7fa48c144080 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa48c0abe20 0x7fa48c0c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:06.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 --2- 172.21.15.7:0/2920413305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa48c14f9d0 0x7fa48c143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:10:06.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 -- 172.21.15.7:0/2920413305 >> 172.21.15.7:0/2920413305 conn(0x7fa48c0c8900 msgr2=0x7fa48c0c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:10:06.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 -- 172.21.15.7:0/2920413305 shutdown_connections 2023-03-03T12:10:06.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:10:06.873+0000 7fa492c88700 1 -- 172.21.15.7:0/2920413305 wait complete. 2023-03-03T12:10:06.880 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:10:06.888 INFO:tasks.mds_thrash.fs.[b]:waiting for 80 secs before thrashing 2023-03-03T12:10:24.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:24.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:54.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:10:54.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:24.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:24.717 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:26.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:11:26.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.265+0000 7f8e91bda700 1 Processor -- start 2023-03-03T12:11:26.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.265+0000 7f8e91bda700 1 -- start start 2023-03-03T12:11:26.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.266+0000 7f8e91bda700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e8c072250 0x7f8e8c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:26.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.266+0000 7f8e91bda700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e8c0c1ce0 0x7f8e8c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:26.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.266+0000 7f8e91bda700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c1510d0 con 0x7f8e8c071e80 2023-03-03T12:11:26.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.266+0000 7f8e91bda700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8c0c3740 con 0x7f8e8c0c1ce0 2023-03-03T12:11:26.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.266+0000 7f8e91bda700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8c070670 con 0x7f8e8c072250 2023-03-03T12:11:26.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.266+0000 7f8e8b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8e8c071e80 0x7f8e8c151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33436/0 (socket says 172.21.15.7:33436) 2023-03-03T12:11:26.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.266+0000 7f8e8bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e8c0c1ce0 0x7f8e8c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:26.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.266+0000 7f8e8b7fe700 1 -- 172.21.15.7:0/425263905 learned_addr learned my addr 172.21.15.7:0/425263905 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:11:26.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8affd700 1 --2- 172.21.15.7:0/425263905 >> v2:172.21.15.143:3300/0 conn(0x7f8e8c072250 0x7f8e8c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:26.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8a7fc700 1 -- 172.21.15.7:0/425263905 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271378255 0 0) 0x7f8e8c1510d0 con 0x7f8e8c071e80 2023-03-03T12:11:26.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8a7fc700 1 -- 172.21.15.7:0/425263905 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e740035b0 con 0x7f8e8c071e80 2023-03-03T12:11:26.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8bfff700 1 -- 172.21.15.7:0/425263905 >> v2:172.21.15.143:3300/0 conn(0x7f8e8c072250 msgr2=0x7f8e8c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:11:26.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8bfff700 1 --2- 172.21.15.7:0/425263905 >> v2:172.21.15.143:3300/0 conn(0x7f8e8c072250 0x7f8e8c14e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:26.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8bfff700 1 -- 172.21.15.7:0/425263905 >> v1:172.21.15.7:6789/0 conn(0x7f8e8c071e80 legacy=0x7f8e8c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:26.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8bfff700 1 -- 172.21.15.7:0/425263905 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c0c39e0 con 0x7f8e8c0c1ce0 2023-03-03T12:11:26.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8bfff700 1 --2- 172.21.15.7:0/425263905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c0c1ce0 0x7f8e8c14e730 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c06b520 tx=0x7f8e8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91c821df8d5b154 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:26.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8a7fc700 1 -- 172.21.15.7:0/425263905 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8400e070 con 0x7f8e8c0c1ce0 2023-03-03T12:11:26.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8a7fc700 1 -- 172.21.15.7:0/425263905 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8400bb20 con 0x7f8e8c0c1ce0 2023-03-03T12:11:26.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.267+0000 7f8e8a7fc700 1 -- 172.21.15.7:0/425263905 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8400be90 con 0x7f8e8c0c1ce0 2023-03-03T12:11:26.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.268+0000 7f8e91bda700 1 -- 172.21.15.7:0/425263905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c0c1ce0 msgr2=0x7f8e8c14e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:26.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.268+0000 7f8e91bda700 1 --2- 172.21.15.7:0/425263905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c0c1ce0 0x7f8e8c14e730 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c06b520 tx=0x7f8e8400b330 comp rx=0 tx=0).stop 2023-03-03T12:11:26.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.268+0000 7f8e91bda700 1 -- 172.21.15.7:0/425263905 shutdown_connections 2023-03-03T12:11:26.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.268+0000 7f8e91bda700 1 --2- 172.21.15.7:0/425263905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c0c1ce0 0x7f8e8c14e730 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:26.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.268+0000 7f8e91bda700 1 --2- 172.21.15.7:0/425263905 >> v2:172.21.15.143:3300/0 conn(0x7f8e8c072250 0x7f8e8c14e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:26.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.268+0000 7f8e91bda700 1 -- 172.21.15.7:0/425263905 >> 172.21.15.7:0/425263905 conn(0x7f8e8c0a94d0 msgr2=0x7f8e8c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:11:26.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.268+0000 7f8e91bda700 1 -- 172.21.15.7:0/425263905 shutdown_connections 2023-03-03T12:11:26.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.268+0000 7f8e91bda700 1 -- 172.21.15.7:0/425263905 wait complete. 2023-03-03T12:11:26.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.269+0000 7f8e91bda700 1 Processor -- start 2023-03-03T12:11:26.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.269+0000 7f8e91bda700 1 -- start start 2023-03-03T12:11:26.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e91bda700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c071e80 0x7f8e8c14c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:26.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e91bda700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c072250 0x7f8e8c1e7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:26.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e91bda700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c0c1ce0 0x7f8e8c1e83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:26.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e91bda700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c1ea8a0 con 0x7f8e8c071e80 2023-03-03T12:11:26.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e91bda700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c145390 con 0x7f8e8c072250 2023-03-03T12:11:26.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e91bda700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c1455e0 con 0x7f8e8c0c1ce0 2023-03-03T12:11:26.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c071e80 0x7f8e8c14c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:26.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c071e80 0x7f8e8c14c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39458/0 (socket says 172.21.15.7:39458) 2023-03-03T12:11:26.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8b7fe700 1 -- 172.21.15.7:0/809533672 learned_addr learned my addr 172.21.15.7:0/809533672 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:11:26.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8bfff700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c0c1ce0 0x7f8e8c1e83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:26.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8affd700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c072250 0x7f8e8c1e7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:26.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8b7fe700 1 -- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c0c1ce0 msgr2=0x7f8e8c1e83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:26.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8b7fe700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c0c1ce0 0x7f8e8c1e83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:26.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8b7fe700 1 -- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c072250 msgr2=0x7f8e8c1e7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:26.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8b7fe700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c072250 0x7f8e8c1e7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:26.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8b7fe700 1 -- 172.21.15.7:0/809533672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e84009a70 con 0x7f8e8c071e80 2023-03-03T12:11:26.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8affd700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c072250 0x7f8e8c1e7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:11:26.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.270+0000 7f8e8b7fe700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c071e80 0x7f8e8c14c290 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00c4d0 tx=0x7f8e7c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:26.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.271+0000 7f8e73fff700 1 -- 172.21.15.7:0/809533672 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c005d70 con 0x7f8e8c071e80 2023-03-03T12:11:26.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.271+0000 7f8e73fff700 1 -- 172.21.15.7:0/809533672 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c0070d0 con 0x7f8e8c071e80 2023-03-03T12:11:26.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.271+0000 7f8e91bda700 1 -- 172.21.15.7:0/809533672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c151f30 con 0x7f8e8c071e80 2023-03-03T12:11:26.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.271+0000 7f8e73fff700 1 -- 172.21.15.7:0/809533672 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c007440 con 0x7f8e8c071e80 2023-03-03T12:11:26.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.271+0000 7f8e91bda700 1 -- 172.21.15.7:0/809533672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c151300 con 0x7f8e8c071e80 2023-03-03T12:11:26.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.272+0000 7f8e91bda700 1 -- 172.21.15.7:0/809533672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8c06c9a0 con 0x7f8e8c071e80 2023-03-03T12:11:26.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.272+0000 7f8e73fff700 1 -- 172.21.15.7:0/809533672 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e7c019030 con 0x7f8e8c071e80 2023-03-03T12:11:26.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.272+0000 7f8e73fff700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 0x7f8e7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:26.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.272+0000 7f8e73fff700 1 -- 172.21.15.7:0/809533672 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8e7c0853b0 con 0x7f8e8c071e80 2023-03-03T12:11:26.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.273+0000 7f8e8affd700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 0x7f8e7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:26.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.273+0000 7f8e8affd700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 0x7f8e7806a7a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c06a8a0 tx=0x7f8e80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:26.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.277+0000 7f8e73fff700 1 -- 172.21.15.7:0/809533672 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e7c04b5e0 con 0x7f8e8c071e80 2023-03-03T12:11:26.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.515+0000 7f8e91bda700 1 -- 172.21.15.7:0/809533672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e8c074bb0 con 0x7f8e8c071e80 2023-03-03T12:11:26.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.516+0000 7f8e73fff700 1 -- 172.21.15.7:0/809533672 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8e7c017090 con 0x7f8e8c071e80 2023-03-03T12:11:26.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:11:26.518 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:11:26.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 -- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 msgr2=0x7f8e7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:26.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 0x7f8e7806a7a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c06a8a0 tx=0x7f8e80008040 comp rx=0 tx=0).stop 2023-03-03T12:11:26.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 -- 172.21.15.7:0/809533672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c071e80 msgr2=0x7f8e8c14c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:26.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c071e80 0x7f8e8c14c290 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00c4d0 tx=0x7f8e7c00c890 comp rx=0 tx=0).stop 2023-03-03T12:11:26.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 -- 172.21.15.7:0/809533672 shutdown_connections 2023-03-03T12:11:26.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e8c0c1ce0 0x7f8e8c1e83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:26.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 0x7f8e7806a7a0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:26.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e8c071e80 0x7f8e8c14c290 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 --2- 172.21.15.7:0/809533672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e8c072250 0x7f8e8c1e7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 -- 172.21.15.7:0/809533672 >> 172.21.15.7:0/809533672 conn(0x7f8e8c0a94d0 msgr2=0x7f8e8c0c4510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:11:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 -- 172.21.15.7:0/809533672 shutdown_connections 2023-03-03T12:11:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:26.518+0000 7f8e91bda700 1 -- 172.21.15.7:0/809533672 wait complete. 2023-03-03T12:11:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:11:26.532 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2023-03-03T12:11:26.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:11:27.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.016+0000 7efd95ba5700 1 Processor -- start 2023-03-03T12:11:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.016+0000 7efd95ba5700 1 -- start start 2023-03-03T12:11:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd95ba5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efd90151cd0 0x7efd90154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd95ba5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9006c6c0 0x7efd900736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd95ba5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd90157ff0 con 0x7efd900c2e30 2023-03-03T12:11:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd95ba5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9006d640 con 0x7efd90151cd0 2023-03-03T12:11:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd95ba5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9006cda0 con 0x7efd9006c6c0 2023-03-03T12:11:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7efd900c2e30 0x7efd90158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33446/0 (socket says 172.21.15.7:33446) 2023-03-03T12:11:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8effd700 1 -- 172.21.15.7:0/2245659758 learned_addr learned my addr 172.21.15.7:0/2245659758 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:11:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8ffff700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9006c6c0 0x7efd900736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8f7fe700 1 --2- 172.21.15.7:0/2245659758 >> v2:172.21.15.143:3300/0 conn(0x7efd90151cd0 0x7efd90154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438057000 0 0) 0x7efd90157ff0 con 0x7efd900c2e30 2023-03-03T12:11:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd7c0035b0 con 0x7efd900c2e30 2023-03-03T12:11:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3718139448 0 0) 0x7efd7c0035b0 con 0x7efd900c2e30 2023-03-03T12:11:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9006c6c0 msgr2=0x7efd900736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:11:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8e7fc700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9006c6c0 0x7efd900736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 >> v2:172.21.15.143:3300/0 conn(0x7efd90151cd0 msgr2=0x7efd90154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:11:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8e7fc700 1 --2- 172.21.15.7:0/2245659758 >> v2:172.21.15.143:3300/0 conn(0x7efd90151cd0 0x7efd90154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9006d080 con 0x7efd900c2e30 2023-03-03T12:11:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7efd78001670 con 0x7efd900c2e30 2023-03-03T12:11:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.017+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 >> v1:172.21.15.7:6789/0 conn(0x7efd900c2e30 legacy=0x7efd90158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8e7fc700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c003a70 0x7efd7c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8e7fc700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9006c6c0 0x7efd7c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8e7fc700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd7c004d90 0x7efd7c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:27.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9006d640 con 0x7efd7c004d90 2023-03-03T12:11:27.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9006cda0 con 0x7efd7c003a70 2023-03-03T12:11:27.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c00baf0 con 0x7efd9006c6c0 2023-03-03T12:11:27.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8ffff700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd7c004d90 0x7efd7c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:27.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8effd700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9006c6c0 0x7efd7c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:27.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8f7fe700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c003a70 0x7efd7c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:27.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8ffff700 1 -- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9006c6c0 msgr2=0x7efd7c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:11:27.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8ffff700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9006c6c0 0x7efd7c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:27.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8ffff700 1 -- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c003a70 msgr2=0x7efd7c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:11:27.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8ffff700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd7c003a70 0x7efd7c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8ffff700 1 -- 172.21.15.7:0/2245659758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c0035b0 con 0x7efd7c004d90 2023-03-03T12:11:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8ffff700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd7c004d90 0x7efd7c005200 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7efd88009fc0 tx=0x7efd7c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf0dc4bf45e561f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8800ec30 con 0x7efd7c004d90 2023-03-03T12:11:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8800f040 con 0x7efd7c004d90 2023-03-03T12:11:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.018+0000 7efd8e7fc700 1 -- 172.21.15.7:0/2245659758 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88014480 con 0x7efd7c004d90 2023-03-03T12:11:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.019+0000 7efd95ba5700 1 -- 172.21.15.7:0/2245659758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd7c004d90 msgr2=0x7efd7c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.019+0000 7efd95ba5700 1 --2- 172.21.15.7:0/2245659758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd7c004d90 0x7efd7c005200 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7efd88009fc0 tx=0x7efd7c005880 comp rx=0 tx=0).stop 2023-03-03T12:11:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.019+0000 7efd8ffff700 1 -- 172.21.15.7:0/2245659758 reap_dead start 2023-03-03T12:11:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.019+0000 7efd95ba5700 1 -- 172.21.15.7:0/2245659758 shutdown_connections 2023-03-03T12:11:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.019+0000 7efd95ba5700 1 -- 172.21.15.7:0/2245659758 >> 172.21.15.7:0/2245659758 conn(0x7efd900a94d0 msgr2=0x7efd900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:11:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.019+0000 7efd95ba5700 1 -- 172.21.15.7:0/2245659758 shutdown_connections 2023-03-03T12:11:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.019+0000 7efd95ba5700 1 -- 172.21.15.7:0/2245659758 wait complete. 2023-03-03T12:11:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.020+0000 7efd95ba5700 1 Processor -- start 2023-03-03T12:11:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.020+0000 7efd95ba5700 1 -- start start 2023-03-03T12:11:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd95ba5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9014da60 0x7efd900c6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:27.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd95ba5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd9014de30 0x7efd900c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:27.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd95ba5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd901eb6a0 0x7efd90149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:27.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd95ba5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd900c5590 con 0x7efd9014de30 2023-03-03T12:11:27.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd95ba5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd900c63f0 con 0x7efd901eb6a0 2023-03-03T12:11:27.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd95ba5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9014e730 con 0x7efd9014da60 2023-03-03T12:11:27.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd8effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd9014de30 0x7efd900c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:27.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd8effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd9014de30 0x7efd900c6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39476/0 (socket says 172.21.15.7:39476) 2023-03-03T12:11:27.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd8effd700 1 -- 172.21.15.7:0/3639497500 learned_addr learned my addr 172.21.15.7:0/3639497500 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:11:27.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd8f7fe700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9014da60 0x7efd900c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:27.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd8ffff700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd901eb6a0 0x7efd90149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:27.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd8effd700 1 -- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9014da60 msgr2=0x7efd900c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:27.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd8effd700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9014da60 0x7efd900c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:27.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd8effd700 1 -- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd901eb6a0 msgr2=0x7efd90149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:27.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd8effd700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd901eb6a0 0x7efd90149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:27.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.021+0000 7efd8effd700 1 -- 172.21.15.7:0/3639497500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd90157c80 con 0x7efd9014de30 2023-03-03T12:11:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.022+0000 7efd8effd700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd9014de30 0x7efd900c6dc0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7efd78007340 tx=0x7efd78007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.022+0000 7efd87fff700 1 -- 172.21.15.7:0/3639497500 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78000f80 con 0x7efd9014de30 2023-03-03T12:11:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.022+0000 7efd87fff700 1 -- 172.21.15.7:0/3639497500 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7800f8d0 con 0x7efd9014de30 2023-03-03T12:11:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.022+0000 7efd95ba5700 1 -- 172.21.15.7:0/3639497500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9006d360 con 0x7efd9014de30 2023-03-03T12:11:27.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.022+0000 7efd87fff700 1 -- 172.21.15.7:0/3639497500 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7800fc80 con 0x7efd9014de30 2023-03-03T12:11:27.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.022+0000 7efd95ba5700 1 -- 172.21.15.7:0/3639497500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd90154a50 con 0x7efd9014de30 2023-03-03T12:11:27.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.023+0000 7efd95ba5700 1 -- 172.21.15.7:0/3639497500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd90074bb0 con 0x7efd9014de30 2023-03-03T12:11:27.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.023+0000 7efd87fff700 1 -- 172.21.15.7:0/3639497500 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efd7801c030 con 0x7efd9014de30 2023-03-03T12:11:27.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.024+0000 7efd87fff700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd700682f0 0x7efd7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:27.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.024+0000 7efd87fff700 1 -- 172.21.15.7:0/3639497500 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efd78085cf0 con 0x7efd9014de30 2023-03-03T12:11:27.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.024+0000 7efd8f7fe700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd700682f0 0x7efd7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:27.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.024+0000 7efd8f7fe700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd700682f0 0x7efd7006a7a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7efd80009ba0 tx=0x7efd80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:27.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.028+0000 7efd87fff700 1 -- 172.21.15.7:0/3639497500 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efd78052030 con 0x7efd9014de30 2023-03-03T12:11:27.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.267+0000 7efd95ba5700 1 -- 172.21.15.7:0/3639497500 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd900c6530 con 0x7efd9014de30 2023-03-03T12:11:27.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.267+0000 7efd87fff700 1 -- 172.21.15.7:0/3639497500 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efd7801a090 con 0x7efd9014de30 2023-03-03T12:11:27.269 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:11:27.269 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:11:27.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.269+0000 7efd95ba5700 1 -- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd700682f0 msgr2=0x7efd7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:27.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.269+0000 7efd95ba5700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd700682f0 0x7efd7006a7a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7efd80009ba0 tx=0x7efd80008040 comp rx=0 tx=0).stop 2023-03-03T12:11:27.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.269+0000 7efd95ba5700 1 -- 172.21.15.7:0/3639497500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd9014de30 msgr2=0x7efd900c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:27.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.269+0000 7efd95ba5700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd9014de30 0x7efd900c6dc0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7efd78007340 tx=0x7efd78007700 comp rx=0 tx=0).stop 2023-03-03T12:11:27.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.270+0000 7efd95ba5700 1 -- 172.21.15.7:0/3639497500 shutdown_connections 2023-03-03T12:11:27.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.270+0000 7efd95ba5700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd901eb6a0 0x7efd90149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:27.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.270+0000 7efd95ba5700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd9014da60 0x7efd900c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:27.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.270+0000 7efd95ba5700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd700682f0 0x7efd7006a7a0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:27.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.270+0000 7efd95ba5700 1 --2- 172.21.15.7:0/3639497500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd9014de30 0x7efd900c6dc0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:27.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.270+0000 7efd95ba5700 1 -- 172.21.15.7:0/3639497500 >> 172.21.15.7:0/3639497500 conn(0x7efd900a94d0 msgr2=0x7efd9006b280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:11:27.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.270+0000 7efd95ba5700 1 -- 172.21.15.7:0/3639497500 shutdown_connections 2023-03-03T12:11:27.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:27.270+0000 7efd95ba5700 1 -- 172.21.15.7:0/3639497500 wait complete. 2023-03-03T12:11:27.273 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:11:27.283 INFO:tasks.mds_thrash.fs.[b]:waiting for 80 secs before thrashing 2023-03-03T12:11:41.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:11:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.670+0000 7f5fc9059700 1 Processor -- start 2023-03-03T12:11:41.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.670+0000 7f5fc9059700 1 -- start start 2023-03-03T12:11:41.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.671+0000 7f5fc9059700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5fc414fda0 0x7f5fc4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:41.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.671+0000 7f5fc9059700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc40c2f30 con 0x7f5fc40abe20 2023-03-03T12:11:41.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.671+0000 7f5fc9059700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc40c31d0 con 0x7f5fc414f9d0 2023-03-03T12:11:41.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.671+0000 7f5fc9059700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc41524a0 con 0x7f5fc414fda0 2023-03-03T12:11:41.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.671+0000 7f5fc359e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5fc40abe20 0x7f5fc41529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33468/0 (socket says 172.21.15.7:33468) 2023-03-03T12:11:41.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.671+0000 7f5fc359e700 1 -- 172.21.15.7:0/2868813508 learned_addr learned my addr 172.21.15.7:0/2868813508 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:11:41.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.671+0000 7f5fc259c700 1 --2- 172.21.15.7:0/2868813508 >> v2:172.21.15.143:3300/0 conn(0x7f5fc414fda0 0x7f5fc4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:41.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.671+0000 7f5fc1d9b700 1 -- 172.21.15.7:0/2868813508 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346281008 0 0) 0x7f5fc40c2f30 con 0x7f5fc40abe20 2023-03-03T12:11:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.671+0000 7f5fc1d9b700 1 -- 172.21.15.7:0/2868813508 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fac0035b0 con 0x7f5fc40abe20 2023-03-03T12:11:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.672+0000 7f5fc1d9b700 1 -- 172.21.15.7:0/2868813508 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1309605857 0 0) 0x7f5fac0035b0 con 0x7f5fc40abe20 2023-03-03T12:11:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.672+0000 7f5fc259c700 1 -- 172.21.15.7:0/2868813508 >> v1:172.21.15.143:6789/0 conn(0x7f5fc414f9d0 legacy=0x7f5fc4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.672+0000 7f5fc259c700 1 -- 172.21.15.7:0/2868813508 >> v1:172.21.15.7:6789/0 conn(0x7f5fc40abe20 legacy=0x7f5fc41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.672+0000 7f5fc259c700 1 -- 172.21.15.7:0/2868813508 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc40c34b0 con 0x7f5fc414fda0 2023-03-03T12:11:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.672+0000 7f5fc259c700 1 --2- 172.21.15.7:0/2868813508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc414fda0 0x7f5fc4156430 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009840 tx=0x7f5fb4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80c444a027dc05e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:41.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.672+0000 7f5fc1d9b700 1 -- 172.21.15.7:0/2868813508 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc0019f0 con 0x7f5fc414fda0 2023-03-03T12:11:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.672+0000 7f5fc1d9b700 1 -- 172.21.15.7:0/2868813508 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb4004030 con 0x7f5fc414fda0 2023-03-03T12:11:41.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.672+0000 7f5fc1d9b700 1 -- 172.21.15.7:0/2868813508 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb40043c0 con 0x7f5fc414fda0 2023-03-03T12:11:41.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.672+0000 7f5fc9059700 1 -- 172.21.15.7:0/2868813508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc414fda0 msgr2=0x7f5fc4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:41.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.673+0000 7f5fc9059700 1 --2- 172.21.15.7:0/2868813508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc414fda0 0x7f5fc4156430 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009840 tx=0x7f5fb4009b50 comp rx=0 tx=0).stop 2023-03-03T12:11:41.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.673+0000 7f5fc9059700 1 -- 172.21.15.7:0/2868813508 shutdown_connections 2023-03-03T12:11:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.673+0000 7f5fc9059700 1 --2- 172.21.15.7:0/2868813508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc414fda0 0x7f5fc4156430 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.673+0000 7f5fc9059700 1 -- 172.21.15.7:0/2868813508 >> 172.21.15.7:0/2868813508 conn(0x7f5fc40c8900 msgr2=0x7f5fc40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:11:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.673+0000 7f5fc9059700 1 -- 172.21.15.7:0/2868813508 shutdown_connections 2023-03-03T12:11:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.673+0000 7f5fc9059700 1 -- 172.21.15.7:0/2868813508 wait complete. 2023-03-03T12:11:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.674+0000 7f5fc9059700 1 Processor -- start 2023-03-03T12:11:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.674+0000 7f5fc9059700 1 -- start start 2023-03-03T12:11:41.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc9059700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fc40abe20 0x7f5fc41dcd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc9059700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc414f9d0 0x7f5fc414dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc9059700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fc414fda0 0x7f5fc40a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc9059700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40c2090 con 0x7f5fc40abe20 2023-03-03T12:11:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc9059700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40c2370 con 0x7f5fc414f9d0 2023-03-03T12:11:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc9059700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40c2650 con 0x7f5fc414fda0 2023-03-03T12:11:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fc40abe20 0x7f5fc41dcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fc40abe20 0x7f5fc41dcd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39490/0 (socket says 172.21.15.7:39490) 2023-03-03T12:11:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc2d9d700 1 -- 172.21.15.7:0/1113669761 learned_addr learned my addr 172.21.15.7:0/1113669761 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:11:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc359e700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fc414fda0 0x7f5fc40a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc259c700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc414f9d0 0x7f5fc414dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc2d9d700 1 -- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fc414fda0 msgr2=0x7f5fc40a9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc2d9d700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fc414fda0 0x7f5fc40a9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc2d9d700 1 -- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc414f9d0 msgr2=0x7f5fc414dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc2d9d700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc414f9d0 0x7f5fc414dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc2d9d700 1 -- 172.21.15.7:0/1113669761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb40094a0 con 0x7f5fc40abe20 2023-03-03T12:11:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.675+0000 7f5fc2d9d700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fc40abe20 0x7f5fc41dcd80 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800c780 tx=0x7f5fb800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.676+0000 7f5fab7fe700 1 -- 172.21.15.7:0/1113669761 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8005480 con 0x7f5fc40abe20 2023-03-03T12:11:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.676+0000 7f5fab7fe700 1 -- 172.21.15.7:0/1113669761 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb8003680 con 0x7f5fc40abe20 2023-03-03T12:11:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.676+0000 7f5fab7fe700 1 -- 172.21.15.7:0/1113669761 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8003a30 con 0x7f5fc40abe20 2023-03-03T12:11:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.676+0000 7f5fc9059700 1 -- 172.21.15.7:0/1113669761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc40ac350 con 0x7f5fc40abe20 2023-03-03T12:11:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.676+0000 7f5fc9059700 1 -- 172.21.15.7:0/1113669761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc40ab630 con 0x7f5fc40abe20 2023-03-03T12:11:41.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.677+0000 7f5fc9059700 1 -- 172.21.15.7:0/1113669761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc4071fa0 con 0x7f5fc40abe20 2023-03-03T12:11:41.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.677+0000 7f5fab7fe700 1 -- 172.21.15.7:0/1113669761 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5fb8010070 con 0x7f5fc40abe20 2023-03-03T12:11:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.678+0000 7f5fab7fe700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5fb00682f0 0x7f5fb006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:11:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.678+0000 7f5fab7fe700 1 -- 172.21.15.7:0/1113669761 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5fb8085b70 con 0x7f5fc40abe20 2023-03-03T12:11:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.678+0000 7f5fc259c700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5fb00682f0 0x7f5fb006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:11:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.678+0000 7f5fc259c700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5fb00682f0 0x7f5fb006a7a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5fc41501d0 tx=0x7f5fb4008000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:11:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.682+0000 7f5fab7fe700 1 -- 172.21.15.7:0/1113669761 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5fb8051f10 con 0x7f5fc40abe20 2023-03-03T12:11:41.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.920+0000 7f5fc9059700 1 -- 172.21.15.7:0/1113669761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fc40ac670 con 0x7f5fc40abe20 2023-03-03T12:11:41.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.921+0000 7f5fab7fe700 1 -- 172.21.15.7:0/1113669761 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5fb8055530 con 0x7f5fc40abe20 2023-03-03T12:11:41.922 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:11:41.923 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:11:41.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 -- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5fb00682f0 msgr2=0x7f5fb006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:41.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5fb00682f0 0x7f5fb006a7a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5fc41501d0 tx=0x7f5fb4008000 comp rx=0 tx=0).stop 2023-03-03T12:11:41.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 -- 172.21.15.7:0/1113669761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fc40abe20 msgr2=0x7f5fc41dcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:11:41.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fc40abe20 0x7f5fc41dcd80 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800c780 tx=0x7f5fb800cb40 comp rx=0 tx=0).stop 2023-03-03T12:11:41.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 -- 172.21.15.7:0/1113669761 shutdown_connections 2023-03-03T12:11:41.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5fc414fda0 0x7f5fc40a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5fb00682f0 0x7f5fb006a7a0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fc40abe20 0x7f5fc41dcd80 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 --2- 172.21.15.7:0/1113669761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc414f9d0 0x7f5fc414dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:11:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 -- 172.21.15.7:0/1113669761 >> 172.21.15.7:0/1113669761 conn(0x7f5fc40c8900 msgr2=0x7f5fc40c7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:11:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 -- 172.21.15.7:0/1113669761 shutdown_connections 2023-03-03T12:11:41.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:11:41.923+0000 7f5fc9059700 1 -- 172.21.15.7:0/1113669761 wait complete. 2023-03-03T12:11:41.927 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:11:41.936 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2023-03-03T12:11:54.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:11:54.871 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:25.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:25.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:47.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:12:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.421+0000 7ff83fb07700 1 Processor -- start 2023-03-03T12:12:47.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.421+0000 7ff83fb07700 1 -- start start 2023-03-03T12:12:47.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.422+0000 7ff83fb07700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff838151cc0 0x7ff8380ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:47.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.422+0000 7ff83fb07700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8380b00b0 0x7ff8380c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:47.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.422+0000 7ff83fb07700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8380cac10 0x7ff8380c7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:47.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.422+0000 7ff83fb07700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff838152090 con 0x7ff838151cc0 2023-03-03T12:12:47.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.422+0000 7ff83fb07700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8380c6cf0 con 0x7ff8380cac10 2023-03-03T12:12:47.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.422+0000 7ff83fb07700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8381548f0 con 0x7ff8380b00b0 2023-03-03T12:12:47.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.422+0000 7ff83d8a3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff838151cc0 0x7ff8380ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:47.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.422+0000 7ff83d8a3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff838151cc0 0x7ff8380ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39554/0 (socket says 172.21.15.7:39554) 2023-03-03T12:12:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.422+0000 7ff83d8a3700 1 -- 172.21.15.7:0/3260531281 learned_addr learned my addr 172.21.15.7:0/3260531281 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:12:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.422+0000 7ff83d0a2700 1 --2- 172.21.15.7:0/3260531281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8380b00b0 0x7ff8380c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83d8a3700 1 -- 172.21.15.7:0/3260531281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8380b00b0 msgr2=0x7ff8380c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:12:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83e0a4700 1 --2- 172.21.15.7:0/3260531281 >> v2:172.21.15.143:3300/0 conn(0x7ff8380cac10 0x7ff8380c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83d8a3700 1 --2- 172.21.15.7:0/3260531281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8380b00b0 0x7ff8380c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83d8a3700 1 -- 172.21.15.7:0/3260531281 >> v2:172.21.15.143:3300/0 conn(0x7ff8380cac10 msgr2=0x7ff8380c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:12:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83d8a3700 1 --2- 172.21.15.7:0/3260531281 >> v2:172.21.15.143:3300/0 conn(0x7ff8380cac10 0x7ff8380c7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83d8a3700 1 -- 172.21.15.7:0/3260531281 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff838154bd0 con 0x7ff838151cc0 2023-03-03T12:12:47.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83d0a2700 1 --2- 172.21.15.7:0/3260531281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8380b00b0 0x7ff8380c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:12:47.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83d8a3700 1 --2- 172.21.15.7:0/3260531281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838151cc0 0x7ff8380ca6d0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7ff8380c2430 tx=0x7ff828009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ad28d7cdf76aeb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:12:47.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83c8a1700 1 -- 172.21.15.7:0/3260531281 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82800b870 con 0x7ff838151cc0 2023-03-03T12:12:47.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83c8a1700 1 -- 172.21.15.7:0/3260531281 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82800b9d0 con 0x7ff838151cc0 2023-03-03T12:12:47.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83c8a1700 1 -- 172.21.15.7:0/3260531281 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82800bd60 con 0x7ff838151cc0 2023-03-03T12:12:47.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83fb07700 1 -- 172.21.15.7:0/3260531281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838151cc0 msgr2=0x7ff8380ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.423+0000 7ff83fb07700 1 --2- 172.21.15.7:0/3260531281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838151cc0 0x7ff8380ca6d0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7ff8380c2430 tx=0x7ff828009bd0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.424+0000 7ff83fb07700 1 -- 172.21.15.7:0/3260531281 shutdown_connections 2023-03-03T12:12:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.424+0000 7ff83fb07700 1 --2- 172.21.15.7:0/3260531281 >> v2:172.21.15.143:3300/0 conn(0x7ff8380cac10 0x7ff8380c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.424+0000 7ff83fb07700 1 --2- 172.21.15.7:0/3260531281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838151cc0 0x7ff8380ca6d0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.424+0000 7ff83fb07700 1 --2- 172.21.15.7:0/3260531281 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8380b00b0 0x7ff8380c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.424+0000 7ff83fb07700 1 -- 172.21.15.7:0/3260531281 >> 172.21.15.7:0/3260531281 conn(0x7ff83806b980 msgr2=0x7ff838071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:12:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.424+0000 7ff83fb07700 1 -- 172.21.15.7:0/3260531281 shutdown_connections 2023-03-03T12:12:47.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.424+0000 7ff83fb07700 1 -- 172.21.15.7:0/3260531281 wait complete. 2023-03-03T12:12:47.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.425+0000 7ff83fb07700 1 Processor -- start 2023-03-03T12:12:47.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.425+0000 7ff83fb07700 1 -- start start 2023-03-03T12:12:47.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.425+0000 7ff83fb07700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8380b00b0 0x7ff838149060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:47.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.425+0000 7ff83fb07700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8380cac10 0x7ff838156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:47.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.425+0000 7ff83fb07700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff838151cc0 0x7ff8381ed640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:47.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.425+0000 7ff83fb07700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff838156dd0 con 0x7ff8380b00b0 2023-03-03T12:12:47.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.425+0000 7ff83fb07700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8380c2890 con 0x7ff8380cac10 2023-03-03T12:12:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.425+0000 7ff83fb07700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff838148d80 con 0x7ff838151cc0 2023-03-03T12:12:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83d8a3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8380b00b0 0x7ff838149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83d8a3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8380b00b0 0x7ff838149060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39560/0 (socket says 172.21.15.7:39560) 2023-03-03T12:12:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83d8a3700 1 -- 172.21.15.7:0/3855443700 learned_addr learned my addr 172.21.15.7:0/3855443700 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:12:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83e0a4700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff838151cc0 0x7ff8381ed640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83d0a2700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8380cac10 0x7ff838156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83d8a3700 1 -- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff838151cc0 msgr2=0x7ff8381ed640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83d8a3700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff838151cc0 0x7ff8381ed640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83d8a3700 1 -- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8380cac10 msgr2=0x7ff838156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83d8a3700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8380cac10 0x7ff838156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83d8a3700 1 -- 172.21.15.7:0/3855443700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff828009550 con 0x7ff8380b00b0 2023-03-03T12:12:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83d8a3700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8380b00b0 0x7ff838149060 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7ff828014040 tx=0x7ff828004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:12:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff82e7fc700 1 -- 172.21.15.7:0/3855443700 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828004520 con 0x7ff8380b00b0 2023-03-03T12:12:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff82e7fc700 1 -- 172.21.15.7:0/3855443700 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82801e070 con 0x7ff8380b00b0 2023-03-03T12:12:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff82e7fc700 1 -- 172.21.15.7:0/3855443700 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8280038e0 con 0x7ff8380b00b0 2023-03-03T12:12:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83fb07700 1 -- 172.21.15.7:0/3855443700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff838155190 con 0x7ff8380b00b0 2023-03-03T12:12:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.426+0000 7ff83fb07700 1 -- 172.21.15.7:0/3855443700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff838154eb0 con 0x7ff8380b00b0 2023-03-03T12:12:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.427+0000 7ff83fb07700 1 -- 172.21.15.7:0/3855443700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8381545a0 con 0x7ff8380b00b0 2023-03-03T12:12:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.428+0000 7ff82e7fc700 1 -- 172.21.15.7:0/3855443700 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff82800f050 con 0x7ff8380b00b0 2023-03-03T12:12:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.428+0000 7ff82e7fc700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff81c0682f0 0x7ff81c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:12:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.428+0000 7ff82e7fc700 1 -- 172.21.15.7:0/3855443700 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff828020070 con 0x7ff8380b00b0 2023-03-03T12:12:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.428+0000 7ff83d0a2700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff81c0682f0 0x7ff81c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:12:47.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.429+0000 7ff83d0a2700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff81c0682f0 0x7ff81c06a7a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff8381eaf90 tx=0x7ff834008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:12:47.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.432+0000 7ff82e7fc700 1 -- 172.21.15.7:0/3855443700 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff828053b80 con 0x7ff8380b00b0 2023-03-03T12:12:47.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.669+0000 7ff83fb07700 1 -- 172.21.15.7:0/3855443700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8380c0620 con 0x7ff8380b00b0 2023-03-03T12:12:47.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.670+0000 7ff82e7fc700 1 -- 172.21.15.7:0/3855443700 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff828015090 con 0x7ff8380b00b0 2023-03-03T12:12:47.671 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:12:47.672 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:12:47.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 -- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff81c0682f0 msgr2=0x7ff81c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:47.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff81c0682f0 0x7ff81c06a7a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff8381eaf90 tx=0x7ff834008040 comp rx=0 tx=0).stop 2023-03-03T12:12:47.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 -- 172.21.15.7:0/3855443700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8380b00b0 msgr2=0x7ff838149060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:12:47.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8380b00b0 0x7ff838149060 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7ff828014040 tx=0x7ff828004750 comp rx=0 tx=0).stop 2023-03-03T12:12:47.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 -- 172.21.15.7:0/3855443700 shutdown_connections 2023-03-03T12:12:47.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff838151cc0 0x7ff8381ed640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff81c0682f0 0x7ff81c06a7a0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8380b00b0 0x7ff838149060 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 --2- 172.21.15.7:0/3855443700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8380cac10 0x7ff838156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:12:47.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 -- 172.21.15.7:0/3855443700 >> 172.21.15.7:0/3855443700 conn(0x7ff83806b980 msgr2=0x7ff8380c5310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:12:47.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 -- 172.21.15.7:0/3855443700 shutdown_connections 2023-03-03T12:12:47.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:12:47.672+0000 7ff83fb07700 1 -- 172.21.15.7:0/3855443700 wait complete. 2023-03-03T12:12:47.675 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:12:47.685 INFO:tasks.mds_thrash.fs.[b]:waiting for 40 secs before thrashing 2023-03-03T12:12:55.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:12:55.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:00.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:13:01.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.074+0000 7f0e4fc6c700 1 Processor -- start 2023-03-03T12:13:01.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.075+0000 7f0e4fc6c700 1 -- start start 2023-03-03T12:13:01.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4fc6c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 0x7f0e48156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:01.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4fc6c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0e480abe20 0x7f0e481529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:01.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4fc6c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e480c2f30 con 0x7f0e4814fda0 2023-03-03T12:13:01.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4fc6c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e480c31d0 con 0x7f0e480abe20 2023-03-03T12:13:01.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4fc6c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e481524a0 con 0x7f0e4814f9d0 2023-03-03T12:13:01.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4e209700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0e480abe20 0x7f0e481529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:01.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4da08700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 0x7f0e48156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:01.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4e209700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0e480abe20 0x7f0e481529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:35346/0 (socket says 172.21.15.7:35346) 2023-03-03T12:13:01.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4e209700 1 -- 172.21.15.7:0/2399282892 learned_addr learned my addr 172.21.15.7:0/2399282892 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:13:01.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4e209700 1 -- 172.21.15.7:0/2399282892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 msgr2=0x7f0e48156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:13:01.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4e209700 1 --2- 172.21.15.7:0/2399282892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 0x7f0e48156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:01.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4e209700 1 -- 172.21.15.7:0/2399282892 >> v1:172.21.15.143:6789/0 conn(0x7f0e4814fda0 legacy=0x7f0e48071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:01.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4e209700 1 -- 172.21.15.7:0/2399282892 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e480c34b0 con 0x7f0e480abe20 2023-03-03T12:13:01.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.076+0000 7f0e4da08700 1 --2- 172.21.15.7:0/2399282892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 0x7f0e48156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:13:01.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.077+0000 7f0e4e209700 1 --2- 172.21.15.7:0/2399282892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e480abe20 0x7f0e481529c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0e40009e50 tx=0x7f0e4000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d89004ca4c1dd35 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:01.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.077+0000 7f0e4ca06700 1 -- 172.21.15.7:0/2399282892 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4000e070 con 0x7f0e480abe20 2023-03-03T12:13:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.077+0000 7f0e4ca06700 1 -- 172.21.15.7:0/2399282892 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4000b910 con 0x7f0e480abe20 2023-03-03T12:13:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.077+0000 7f0e4ca06700 1 -- 172.21.15.7:0/2399282892 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4000bcc0 con 0x7f0e480abe20 2023-03-03T12:13:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.077+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2399282892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e480abe20 msgr2=0x7f0e481529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.077+0000 7f0e4fc6c700 1 --2- 172.21.15.7:0/2399282892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e480abe20 0x7f0e481529c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0e40009e50 tx=0x7f0e4000b330 comp rx=0 tx=0).stop 2023-03-03T12:13:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.077+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2399282892 shutdown_connections 2023-03-03T12:13:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.077+0000 7f0e4fc6c700 1 --2- 172.21.15.7:0/2399282892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e480abe20 0x7f0e481529c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.077+0000 7f0e4fc6c700 1 --2- 172.21.15.7:0/2399282892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 0x7f0e48156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:01.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.077+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2399282892 >> 172.21.15.7:0/2399282892 conn(0x7f0e480c8900 msgr2=0x7f0e480c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:13:01.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.078+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2399282892 shutdown_connections 2023-03-03T12:13:01.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.078+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2399282892 wait complete. 2023-03-03T12:13:01.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.079+0000 7f0e4fc6c700 1 Processor -- start 2023-03-03T12:13:01.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.079+0000 7f0e4fc6c700 1 -- start start 2023-03-03T12:13:01.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.079+0000 7f0e4fc6c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e480abe20 0x7f0e4806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:01.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.079+0000 7f0e4fc6c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 0x7f0e48152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:01.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.079+0000 7f0e4fc6c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e4814fda0 0x7f0e481de0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:01.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.079+0000 7f0e4fc6c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4806f3f0 con 0x7f0e4814fda0 2023-03-03T12:13:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.079+0000 7f0e4fc6c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e481db7a0 con 0x7f0e480abe20 2023-03-03T12:13:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.079+0000 7f0e4fc6c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e481dba80 con 0x7f0e4814f9d0 2023-03-03T12:13:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4e209700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e4814fda0 0x7f0e481de0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4e209700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e4814fda0 0x7f0e481de0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39578/0 (socket says 172.21.15.7:39578) 2023-03-03T12:13:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4e209700 1 -- 172.21.15.7:0/2028333837 learned_addr learned my addr 172.21.15.7:0/2028333837 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:13:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4da08700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e480abe20 0x7f0e4806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:01.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4d207700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 0x7f0e48152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4e209700 1 -- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 msgr2=0x7f0e48152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4e209700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 0x7f0e48152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4e209700 1 -- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e480abe20 msgr2=0x7f0e4806ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4e209700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e480abe20 0x7f0e4806ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4e209700 1 -- 172.21.15.7:0/2028333837 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e440094a0 con 0x7f0e4814fda0 2023-03-03T12:13:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4da08700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e480abe20 0x7f0e4806ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:13:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4e209700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e4814fda0 0x7f0e481de0a0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0e48072520 tx=0x7f0e40004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:01.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e3a7fc700 1 -- 172.21.15.7:0/2028333837 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40004800 con 0x7f0e4814fda0 2023-03-03T12:13:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e3a7fc700 1 -- 172.21.15.7:0/2028333837 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e400092e0 con 0x7f0e4814fda0 2023-03-03T12:13:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2028333837 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e40009b30 con 0x7f0e4814fda0 2023-03-03T12:13:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e3a7fc700 1 -- 172.21.15.7:0/2028333837 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4001b670 con 0x7f0e4814fda0 2023-03-03T12:13:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.080+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2028333837 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e480ac350 con 0x7f0e4814fda0 2023-03-03T12:13:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.081+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2028333837 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e480ab630 con 0x7f0e4814fda0 2023-03-03T12:13:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.082+0000 7f0e3a7fc700 1 -- 172.21.15.7:0/2028333837 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0e40021030 con 0x7f0e4814fda0 2023-03-03T12:13:01.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.082+0000 7f0e3a7fc700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e2c0682f0 0x7f0e2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:01.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.082+0000 7f0e3a7fc700 1 -- 172.21.15.7:0/2028333837 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0e40087590 con 0x7f0e4814fda0 2023-03-03T12:13:01.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.082+0000 7f0e4da08700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e2c0682f0 0x7f0e2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:01.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.083+0000 7f0e4da08700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e2c0682f0 0x7f0e2c06a7a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0e440114b0 tx=0x7f0e44009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:01.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.087+0000 7f0e3a7fc700 1 -- 172.21.15.7:0/2028333837 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0e40053840 con 0x7f0e4814fda0 2023-03-03T12:13:01.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.327+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2028333837 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e48071fa0 con 0x7f0e4814fda0 2023-03-03T12:13:01.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.328+0000 7f0e3a7fc700 1 -- 172.21.15.7:0/2028333837 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0e4001f070 con 0x7f0e4814fda0 2023-03-03T12:13:01.330 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:13:01.330 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:13:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e2c0682f0 msgr2=0x7f0e2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e2c0682f0 0x7f0e2c06a7a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0e440114b0 tx=0x7f0e44009210 comp rx=0 tx=0).stop 2023-03-03T12:13:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2028333837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e4814fda0 msgr2=0x7f0e481de0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e4814fda0 0x7f0e481de0a0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0e48072520 tx=0x7f0e40004ad0 comp rx=0 tx=0).stop 2023-03-03T12:13:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2028333837 shutdown_connections 2023-03-03T12:13:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e2c0682f0 0x7f0e2c06a7a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0e4814fda0 0x7f0e481de0a0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0e480abe20 0x7f0e4806ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 --2- 172.21.15.7:0/2028333837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0e4814f9d0 0x7f0e48152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2028333837 >> 172.21.15.7:0/2028333837 conn(0x7f0e480c8900 msgr2=0x7f0e48151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:13:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2028333837 shutdown_connections 2023-03-03T12:13:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:01.330+0000 7f0e4fc6c700 1 -- 172.21.15.7:0/2028333837 wait complete. 2023-03-03T12:13:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:13:01.343 INFO:tasks.mds_thrash.fs.[a]:waiting for 44 secs before thrashing 2023-03-03T12:13:25.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:25.320 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:27.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:13:27.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.823+0000 7fd31176b700 1 Processor -- start 2023-03-03T12:13:27.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.823+0000 7fd31176b700 1 -- start start 2023-03-03T12:13:27.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.823+0000 7fd31176b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd30c0caa30 0x7fd30c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:27.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.823+0000 7fd31176b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 0x7fd30c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:27.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.823+0000 7fd31176b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd30c0cae00 0x7fd30c0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:27.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.823+0000 7fd31176b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd30c0cb280 con 0x7fd30c0caa30 2023-03-03T12:13:27.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.823+0000 7fd31176b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd30c0bfd10 con 0x7fd30c0cae00 2023-03-03T12:13:27.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.823+0000 7fd31176b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c0bfa30 con 0x7fd30c0c00e0 2023-03-03T12:13:27.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd30c0caa30 0x7fd30c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:27.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd30c0caa30 0x7fd30c0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39612/0 (socket says 172.21.15.7:39612) 2023-03-03T12:13:27.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30affd700 1 -- 172.21.15.7:0/3118440219 learned_addr learned my addr 172.21.15.7:0/3118440219 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:13:27.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30a7fc700 1 --2- 172.21.15.7:0/3118440219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 0x7fd30c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:27.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30b7fe700 1 --2- 172.21.15.7:0/3118440219 >> v2:172.21.15.143:3300/0 conn(0x7fd30c0cae00 0x7fd30c0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:27.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30affd700 1 -- 172.21.15.7:0/3118440219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 msgr2=0x7fd30c154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:13:27.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30affd700 1 --2- 172.21.15.7:0/3118440219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 0x7fd30c154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:27.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30affd700 1 -- 172.21.15.7:0/3118440219 >> v2:172.21.15.143:3300/0 conn(0x7fd30c0cae00 msgr2=0x7fd30c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:13:27.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30affd700 1 --2- 172.21.15.7:0/3118440219 >> v2:172.21.15.143:3300/0 conn(0x7fd30c0cae00 0x7fd30c0c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:27.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30affd700 1 -- 172.21.15.7:0/3118440219 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c154510 con 0x7fd30c0caa30 2023-03-03T12:13:27.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30b7fe700 1 --2- 172.21.15.7:0/3118440219 >> v2:172.21.15.143:3300/0 conn(0x7fd30c0cae00 0x7fd30c0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:13:27.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30a7fc700 1 --2- 172.21.15.7:0/3118440219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 0x7fd30c154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:13:27.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd30affd700 1 --2- 172.21.15.7:0/3118440219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 0x7fd30c0c5c60 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd30c158a70 tx=0x7fd2fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3cfe3893b8995d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:27.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd309ffb700 1 -- 172.21.15.7:0/3118440219 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc00b870 con 0x7fd30c0caa30 2023-03-03T12:13:27.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd309ffb700 1 -- 172.21.15.7:0/3118440219 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2fc00b9d0 con 0x7fd30c0caa30 2023-03-03T12:13:27.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.824+0000 7fd309ffb700 1 -- 172.21.15.7:0/3118440219 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc00bd60 con 0x7fd30c0caa30 2023-03-03T12:13:27.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.825+0000 7fd31176b700 1 -- 172.21.15.7:0/3118440219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 msgr2=0x7fd30c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:27.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.825+0000 7fd31176b700 1 --2- 172.21.15.7:0/3118440219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 0x7fd30c0c5c60 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd30c158a70 tx=0x7fd2fc009bd0 comp rx=0 tx=0).stop 2023-03-03T12:13:27.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.825+0000 7fd31176b700 1 -- 172.21.15.7:0/3118440219 shutdown_connections 2023-03-03T12:13:27.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.825+0000 7fd31176b700 1 --2- 172.21.15.7:0/3118440219 >> v2:172.21.15.143:3300/0 conn(0x7fd30c0cae00 0x7fd30c0c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:27.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.825+0000 7fd31176b700 1 --2- 172.21.15.7:0/3118440219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 0x7fd30c0c5c60 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:27.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.825+0000 7fd31176b700 1 --2- 172.21.15.7:0/3118440219 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 0x7fd30c154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:27.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.825+0000 7fd31176b700 1 -- 172.21.15.7:0/3118440219 >> 172.21.15.7:0/3118440219 conn(0x7fd30c066e40 msgr2=0x7fd30c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:13:27.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.825+0000 7fd31176b700 1 -- 172.21.15.7:0/3118440219 shutdown_connections 2023-03-03T12:13:27.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.825+0000 7fd31176b700 1 -- 172.21.15.7:0/3118440219 wait complete. 2023-03-03T12:13:27.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.826+0000 7fd31176b700 1 Processor -- start 2023-03-03T12:13:27.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.826+0000 7fd31176b700 1 -- start start 2023-03-03T12:13:27.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd31176b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 0x7fd30c1e9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:27.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd31176b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 0x7fd30c0c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:27.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd31176b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd30c0cae00 0x7fd30c1edc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:27.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd31176b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c0c3330 con 0x7fd30c0caa30 2023-03-03T12:13:27.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd31176b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c1ea2c0 con 0x7fd30c0cae00 2023-03-03T12:13:27.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd31176b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c0c2d70 con 0x7fd30c0c00e0 2023-03-03T12:13:27.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd30a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 0x7fd30c0c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:27.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd30a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 0x7fd30c0c6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39618/0 (socket says 172.21.15.7:39618) 2023-03-03T12:13:27.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd30a7fc700 1 -- 172.21.15.7:0/336358249 learned_addr learned my addr 172.21.15.7:0/336358249 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:13:27.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd30affd700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 0x7fd30c1e9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:27.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.827+0000 7fd30b7fe700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd30c0cae00 0x7fd30c1edc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:27.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd30a7fc700 1 -- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 msgr2=0x7fd30c1e9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:27.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd30a7fc700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 0x7fd30c1e9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:27.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd30a7fc700 1 -- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd30c0cae00 msgr2=0x7fd30c1edc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:27.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd30a7fc700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd30c0cae00 0x7fd30c1edc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:27.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd30a7fc700 1 -- 172.21.15.7:0/336358249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc009550 con 0x7fd30c0caa30 2023-03-03T12:13:27.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd30a7fc700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 0x7fd30c0c6a00 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400e810 tx=0x7fd2f400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:27.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd3037fe700 1 -- 172.21.15.7:0/336358249 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4005c20 con 0x7fd30c0caa30 2023-03-03T12:13:27.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd3037fe700 1 -- 172.21.15.7:0/336358249 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f4005d80 con 0x7fd30c0caa30 2023-03-03T12:13:27.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd3037fe700 1 -- 172.21.15.7:0/336358249 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4004250 con 0x7fd30c0caa30 2023-03-03T12:13:27.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd31176b700 1 -- 172.21.15.7:0/336358249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30c154ad0 con 0x7fd30c0caa30 2023-03-03T12:13:27.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.828+0000 7fd31176b700 1 -- 172.21.15.7:0/336358249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30c1547f0 con 0x7fd30c0caa30 2023-03-03T12:13:27.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.829+0000 7fd31176b700 1 -- 172.21.15.7:0/336358249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd30c0bf650 con 0x7fd30c0caa30 2023-03-03T12:13:27.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.829+0000 7fd3037fe700 1 -- 172.21.15.7:0/336358249 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd2f4022070 con 0x7fd30c0caa30 2023-03-03T12:13:27.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.830+0000 7fd3037fe700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2ec0682f0 0x7fd2ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:27.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.830+0000 7fd3037fe700 1 -- 172.21.15.7:0/336358249 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd2f4085e60 con 0x7fd30c0caa30 2023-03-03T12:13:27.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.830+0000 7fd30affd700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2ec0682f0 0x7fd2ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:27.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.831+0000 7fd30affd700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2ec0682f0 0x7fd2ec06a7a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc000c00 tx=0x7fd2fc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:27.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:27.834+0000 7fd3037fe700 1 -- 172.21.15.7:0/336358249 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd2f4052200 con 0x7fd30c0caa30 2023-03-03T12:13:28.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.077+0000 7fd31176b700 1 -- 172.21.15.7:0/336358249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd30c0c1520 con 0x7fd30c0caa30 2023-03-03T12:13:28.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.078+0000 7fd3037fe700 1 -- 172.21.15.7:0/336358249 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd2f400f040 con 0x7fd30c0caa30 2023-03-03T12:13:28.079 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:13:28.079 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:13:28.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.079+0000 7fd31176b700 1 -- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2ec0682f0 msgr2=0x7fd2ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:28.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.079+0000 7fd31176b700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2ec0682f0 0x7fd2ec06a7a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc000c00 tx=0x7fd2fc008030 comp rx=0 tx=0).stop 2023-03-03T12:13:28.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.079+0000 7fd31176b700 1 -- 172.21.15.7:0/336358249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 msgr2=0x7fd30c0c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:28.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.079+0000 7fd31176b700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 0x7fd30c0c6a00 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400e810 tx=0x7fd2f400ebd0 comp rx=0 tx=0).stop 2023-03-03T12:13:28.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.079+0000 7fd31176b700 1 -- 172.21.15.7:0/336358249 shutdown_connections 2023-03-03T12:13:28.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.079+0000 7fd31176b700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd30c0cae00 0x7fd30c1edc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:28.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.079+0000 7fd31176b700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd30c0c00e0 0x7fd30c1e9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:28.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.079+0000 7fd31176b700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2ec0682f0 0x7fd2ec06a7a0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:28.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.079+0000 7fd31176b700 1 --2- 172.21.15.7:0/336358249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0caa30 0x7fd30c0c6a00 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:28.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.079+0000 7fd31176b700 1 -- 172.21.15.7:0/336358249 >> 172.21.15.7:0/336358249 conn(0x7fd30c066e40 msgr2=0x7fd30c152cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:13:28.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.080+0000 7fd31176b700 1 -- 172.21.15.7:0/336358249 shutdown_connections 2023-03-03T12:13:28.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:28.080+0000 7fd31176b700 1 -- 172.21.15.7:0/336358249 wait complete. 2023-03-03T12:13:28.083 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:13:28.093 INFO:tasks.mds_thrash.fs.[b]:waiting for 105 secs before thrashing 2023-03-03T12:13:45.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:13:45.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.482+0000 7f6ae8c8d700 1 Processor -- start 2023-03-03T12:13:45.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.482+0000 7f6ae8c8d700 1 -- start start 2023-03-03T12:13:45.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae8c8d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6ae406a2b0 0x7f6ae406a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:45.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae8c8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae40a6370 0x7f6ae4155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:45.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae8c8d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6ae40c5cb0 0x7f6ae41562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:45.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae8c8d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae406bab0 con 0x7f6ae406a2b0 2023-03-03T12:13:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae8c8d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae4156800 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae8c8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae41558f0 con 0x7f6ae40a6370 2023-03-03T12:13:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6ae406a2b0 0x7f6ae406a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6ae406a2b0 0x7f6ae406a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39630/0 (socket says 172.21.15.7:39630) 2023-03-03T12:13:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae259c700 1 -- 172.21.15.7:0/4151216178 learned_addr learned my addr 172.21.15.7:0/4151216178 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:13:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae1d9b700 1 --2- 172.21.15.7:0/4151216178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae40a6370 0x7f6ae4155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae2d9d700 1 --2- 172.21.15.7:0/4151216178 >> v2:172.21.15.143:3300/0 conn(0x7f6ae40c5cb0 0x7f6ae41562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae259c700 1 -- 172.21.15.7:0/4151216178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae40a6370 msgr2=0x7f6ae4155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:13:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae259c700 1 --2- 172.21.15.7:0/4151216178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae40a6370 0x7f6ae4155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae259c700 1 -- 172.21.15.7:0/4151216178 >> v2:172.21.15.143:3300/0 conn(0x7f6ae40c5cb0 msgr2=0x7f6ae41562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:13:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae259c700 1 --2- 172.21.15.7:0/4151216178 >> v2:172.21.15.143:3300/0 conn(0x7f6ae40c5cb0 0x7f6ae41562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae259c700 1 -- 172.21.15.7:0/4151216178 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae40a6180 con 0x7f6ae406a2b0 2023-03-03T12:13:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae1d9b700 1 --2- 172.21.15.7:0/4151216178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae40a6370 0x7f6ae4155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:13:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.483+0000 7f6ae259c700 1 --2- 172.21.15.7:0/4151216178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae406a2b0 0x7f6ae406a680 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f6ae406b710 tx=0x7f6ad4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fb873fb8ed41fe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:45.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae159a700 1 -- 172.21.15.7:0/4151216178 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad400b870 con 0x7f6ae406a2b0 2023-03-03T12:13:45.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae159a700 1 -- 172.21.15.7:0/4151216178 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad400b9d0 con 0x7f6ae406a2b0 2023-03-03T12:13:45.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae159a700 1 -- 172.21.15.7:0/4151216178 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad400bd60 con 0x7f6ae406a2b0 2023-03-03T12:13:45.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/4151216178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae406a2b0 msgr2=0x7f6ae406a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:45.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae8c8d700 1 --2- 172.21.15.7:0/4151216178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae406a2b0 0x7f6ae406a680 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f6ae406b710 tx=0x7f6ad4009bd0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/4151216178 shutdown_connections 2023-03-03T12:13:45.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae8c8d700 1 --2- 172.21.15.7:0/4151216178 >> v2:172.21.15.143:3300/0 conn(0x7f6ae40c5cb0 0x7f6ae41562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae8c8d700 1 --2- 172.21.15.7:0/4151216178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae406a2b0 0x7f6ae406a680 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae8c8d700 1 --2- 172.21.15.7:0/4151216178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae40a6370 0x7f6ae4155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/4151216178 >> 172.21.15.7:0/4151216178 conn(0x7f6ae40a9430 msgr2=0x7f6ae40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:13:45.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/4151216178 shutdown_connections 2023-03-03T12:13:45.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.484+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/4151216178 wait complete. 2023-03-03T12:13:45.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae8c8d700 1 Processor -- start 2023-03-03T12:13:45.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae8c8d700 1 -- start start 2023-03-03T12:13:45.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae8c8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae406a2b0 0x7f6ae41552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae8c8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ae40a6370 0x7f6ae41ed230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae8c8d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae40c5cb0 0x7f6ae4150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae8c8d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae40c1100 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae8c8d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae41ea870 con 0x7f6ae40a6370 2023-03-03T12:13:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae8c8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae41eab50 con 0x7f6ae406a2b0 2023-03-03T12:13:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae40c5cb0 0x7f6ae4150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae40c5cb0 0x7f6ae4150090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39640/0 (socket says 172.21.15.7:39640) 2023-03-03T12:13:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae2d9d700 1 -- 172.21.15.7:0/3032479791 learned_addr learned my addr 172.21.15.7:0/3032479791 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:13:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae259c700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae406a2b0 0x7f6ae41552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.486+0000 7f6ae1d9b700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ae40a6370 0x7f6ae41ed230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ae2d9d700 1 -- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae406a2b0 msgr2=0x7f6ae41552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ae2d9d700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae406a2b0 0x7f6ae41552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ae2d9d700 1 -- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ae40a6370 msgr2=0x7f6ae41ed230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ae2d9d700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ae40a6370 0x7f6ae41ed230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ae2d9d700 1 -- 172.21.15.7:0/3032479791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad4009550 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ae2d9d700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae40c5cb0 0x7f6ae4150090 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f6adc007e40 tx=0x7f6adc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ad2ffd700 1 -- 172.21.15.7:0/3032479791 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc00ed30 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ad2ffd700 1 -- 172.21.15.7:0/3032479791 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc014420 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/3032479791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae4152e80 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ad2ffd700 1 -- 172.21.15.7:0/3032479791 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0147d0 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.487+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/3032479791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae40a5ea0 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.488+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/3032479791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae4153160 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.489+0000 7f6ad2ffd700 1 -- 172.21.15.7:0/3032479791 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6adc012050 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.489+0000 7f6ad2ffd700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ac40682f0 0x7f6ac406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:13:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.489+0000 7f6ad2ffd700 1 -- 172.21.15.7:0/3032479791 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6adc086630 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.489+0000 7f6ae259c700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ac40682f0 0x7f6ac406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:13:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.490+0000 7f6ae259c700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ac40682f0 0x7f6ac406a7a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4014040 tx=0x7f6ad4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:13:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.493+0000 7f6ad2ffd700 1 -- 172.21.15.7:0/3032479791 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6adc0529d0 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.730+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/3032479791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ae4074bb0 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.731+0000 7f6ad2ffd700 1 -- 172.21.15.7:0/3032479791 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6adc055ff0 con 0x7f6ae40c5cb0 2023-03-03T12:13:45.733 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:13:45.733 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:13:45.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ac40682f0 msgr2=0x7f6ac406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:45.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ac40682f0 0x7f6ac406a7a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4014040 tx=0x7f6ad4008030 comp rx=0 tx=0).stop 2023-03-03T12:13:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/3032479791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae40c5cb0 msgr2=0x7f6ae4150090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:13:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae40c5cb0 0x7f6ae4150090 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f6adc007e40 tx=0x7f6adc00e3c0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/3032479791 shutdown_connections 2023-03-03T12:13:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ac40682f0 0x7f6ac406a7a0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ae40c5cb0 0x7f6ae4150090 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6ae406a2b0 0x7f6ae41552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 --2- 172.21.15.7:0/3032479791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ae40a6370 0x7f6ae41ed230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:13:45.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/3032479791 >> 172.21.15.7:0/3032479791 conn(0x7f6ae40a9430 msgr2=0x7f6ae414e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:13:45.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/3032479791 shutdown_connections 2023-03-03T12:13:45.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:13:45.733+0000 7f6ae8c8d700 1 -- 172.21.15.7:0/3032479791 wait complete. 2023-03-03T12:13:45.736 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:13:45.746 INFO:tasks.mds_thrash.fs.[a]:waiting for 63 secs before thrashing 2023-03-03T12:13:55.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:13:55.482 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:25.543 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:25.546 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:48.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:14:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.884+0000 7fb2c4710700 1 Processor -- start 2023-03-03T12:14:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.884+0000 7fb2c4710700 1 -- start start 2023-03-03T12:14:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.885+0000 7fb2c4710700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb2bc071e80 0x7fb2bc151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.885+0000 7fb2c4710700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2bc14e900 0x7fb2bc15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.885+0000 7fb2c4710700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc1510d0 con 0x7fb2bc072250 2023-03-03T12:14:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.885+0000 7fb2c4710700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2bc0c3740 con 0x7fb2bc071e80 2023-03-03T12:14:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.885+0000 7fb2c4710700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc070670 con 0x7fb2bc14e900 2023-03-03T12:14:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.885+0000 7fb2c1cab700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb2bc072250 0x7fb2bc14e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33690/0 (socket says 172.21.15.7:33690) 2023-03-03T12:14:48.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.885+0000 7fb2c1cab700 1 -- 172.21.15.7:0/1184936023 learned_addr learned my addr 172.21.15.7:0/1184936023 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:14:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c2cad700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2bc14e900 0x7fb2bc15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c24ac700 1 --2- 172.21.15.7:0/1184936023 >> v2:172.21.15.143:3300/0 conn(0x7fb2bc071e80 0x7fb2bc151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509835601 0 0) 0x7fb2bc1510d0 con 0x7fb2bc072250 2023-03-03T12:14:48.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a80035b0 con 0x7fb2bc072250 2023-03-03T12:14:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3385203093 0 0) 0x7fb2a80035b0 con 0x7fb2bc072250 2023-03-03T12:14:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2bc14e900 msgr2=0x7fb2bc15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:14:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2bc14e900 0x7fb2bc15ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 >> v2:172.21.15.143:3300/0 conn(0x7fb2bc071e80 msgr2=0x7fb2bc151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:14:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 --2- 172.21.15.7:0/1184936023 >> v2:172.21.15.143:3300/0 conn(0x7fb2bc071e80 0x7fb2bc151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc0c39e0 con 0x7fb2bc072250 2023-03-03T12:14:48.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb2b0001670 con 0x7fb2bc072250 2023-03-03T12:14:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 >> v1:172.21.15.7:6789/0 conn(0x7fb2bc072250 legacy=0x7fb2bc14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2a8003a70 0x7fb2a8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2a8004850 0x7fb2a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8005200 0x7fb2a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc0c3740 con 0x7fb2a8003a70 2023-03-03T12:14:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc070670 con 0x7fb2a8004850 2023-03-03T12:14:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a800bfd0 con 0x7fb2a8005200 2023-03-03T12:14:48.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c2cad700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2bc14e900 0x7fb2bc15ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:14:48.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c24ac700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2a8003a70 0x7fb2a8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:48.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c1cab700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2a8004850 0x7fb2a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:48.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.886+0000 7fb2c2cad700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8005200 0x7fb2a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:48.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c24ac700 1 -- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8005200 msgr2=0x7fb2a800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:14:48.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c24ac700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8005200 0x7fb2a800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:48.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c24ac700 1 -- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2a8004850 msgr2=0x7fb2a8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:14:48.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c24ac700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2a8004850 0x7fb2a8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:48.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c24ac700 1 -- 172.21.15.7:0/1184936023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a80035b0 con 0x7fb2a8003a70 2023-03-03T12:14:48.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c2cad700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8005200 0x7fb2a800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:14:48.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c24ac700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2a8003a70 0x7fb2a8003e80 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009b60 tx=0x7fb2a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166b0eee4396b517 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:14:48.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b800e6c0 con 0x7fb2a8003a70 2023-03-03T12:14:48.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b800e820 con 0x7fb2a8003a70 2023-03-03T12:14:48.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c14aa700 1 -- 172.21.15.7:0/1184936023 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b800eb90 con 0x7fb2a8003a70 2023-03-03T12:14:48.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c4710700 1 -- 172.21.15.7:0/1184936023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2a8003a70 msgr2=0x7fb2a8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:48.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.887+0000 7fb2c4710700 1 --2- 172.21.15.7:0/1184936023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2a8003a70 0x7fb2a8003e80 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009b60 tx=0x7fb2a8005880 comp rx=0 tx=0).stop 2023-03-03T12:14:48.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.888+0000 7fb2c2cad700 1 -- 172.21.15.7:0/1184936023 reap_dead start 2023-03-03T12:14:48.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.888+0000 7fb2c4710700 1 -- 172.21.15.7:0/1184936023 shutdown_connections 2023-03-03T12:14:48.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.888+0000 7fb2c4710700 1 -- 172.21.15.7:0/1184936023 >> 172.21.15.7:0/1184936023 conn(0x7fb2bc0a94d0 msgr2=0x7fb2bc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:14:48.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.888+0000 7fb2c4710700 1 -- 172.21.15.7:0/1184936023 shutdown_connections 2023-03-03T12:14:48.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.888+0000 7fb2c4710700 1 -- 172.21.15.7:0/1184936023 wait complete. 2023-03-03T12:14:48.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.889+0000 7fb2c4710700 1 Processor -- start 2023-03-03T12:14:48.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.889+0000 7fb2c4710700 1 -- start start 2023-03-03T12:14:48.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.890+0000 7fb2c4710700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8003a70 0x7fb2bc1eb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:48.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.890+0000 7fb2c4710700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc1ed220 0x7fb2bc1ea520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:48.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.890+0000 7fb2c4710700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2bc0c62e0 0x7fb2bc0c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:48.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.890+0000 7fb2c4710700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc1eda70 con 0x7fb2bc0c62e0 2023-03-03T12:14:48.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.890+0000 7fb2c4710700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc1edd50 con 0x7fb2bc1ed220 2023-03-03T12:14:48.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.890+0000 7fb2c4710700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc1ee030 con 0x7fb2a8003a70 2023-03-03T12:14:48.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.890+0000 7fb2c2cad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2bc0c62e0 0x7fb2bc0c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:48.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.890+0000 7fb2c2cad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2bc0c62e0 0x7fb2bc0c6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39724/0 (socket says 172.21.15.7:39724) 2023-03-03T12:14:48.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.890+0000 7fb2c2cad700 1 -- 172.21.15.7:0/322507053 learned_addr learned my addr 172.21.15.7:0/322507053 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:14:48.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.890+0000 7fb2c24ac700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8003a70 0x7fb2bc1eb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:48.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c1cab700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc1ed220 0x7fb2bc1ea520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:48.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c2cad700 1 -- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8003a70 msgr2=0x7fb2bc1eb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:48.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c2cad700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8003a70 0x7fb2bc1eb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:48.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c2cad700 1 -- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc1ed220 msgr2=0x7fb2bc1ea520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:48.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c2cad700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc1ed220 0x7fb2bc1ea520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:48.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c2cad700 1 -- 172.21.15.7:0/322507053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc06c9a0 con 0x7fb2bc0c62e0 2023-03-03T12:14:48.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c24ac700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8003a70 0x7fb2bc1eb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:14:48.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c1cab700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc1ed220 0x7fb2bc1ea520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:14:48.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c2cad700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2bc0c62e0 0x7fb2bc0c6760 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc072670 tx=0x7fb2b40114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:14:48.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2aeffd700 1 -- 172.21.15.7:0/322507053 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b400e070 con 0x7fb2bc0c62e0 2023-03-03T12:14:48.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2aeffd700 1 -- 172.21.15.7:0/322507053 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b4011d40 con 0x7fb2bc0c62e0 2023-03-03T12:14:48.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c4710700 1 -- 172.21.15.7:0/322507053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2bc151300 con 0x7fb2bc0c62e0 2023-03-03T12:14:48.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2aeffd700 1 -- 172.21.15.7:0/322507053 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4011eb0 con 0x7fb2bc0c62e0 2023-03-03T12:14:48.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.891+0000 7fb2c4710700 1 -- 172.21.15.7:0/322507053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc06cb30 con 0x7fb2bc0c62e0 2023-03-03T12:14:48.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.892+0000 7fb2c4710700 1 -- 172.21.15.7:0/322507053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2bc074bb0 con 0x7fb2bc0c62e0 2023-03-03T12:14:48.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.893+0000 7fb2aeffd700 1 -- 172.21.15.7:0/322507053 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb2b4003a20 con 0x7fb2bc0c62e0 2023-03-03T12:14:48.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.893+0000 7fb2aeffd700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2a00682f0 0x7fb2a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:14:48.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.893+0000 7fb2aeffd700 1 -- 172.21.15.7:0/322507053 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb2b4086b00 con 0x7fb2bc0c62e0 2023-03-03T12:14:48.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.893+0000 7fb2c24ac700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2a00682f0 0x7fb2a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:14:48.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.894+0000 7fb2c24ac700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2a00682f0 0x7fb2a006a7a0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80090b0 tx=0x7fb2b8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:14:48.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:48.897+0000 7fb2aeffd700 1 -- 172.21.15.7:0/322507053 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb2b4052e40 con 0x7fb2bc0c62e0 2023-03-03T12:14:49.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.138+0000 7fb2c4710700 1 -- 172.21.15.7:0/322507053 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2bc1ea210 con 0x7fb2bc0c62e0 2023-03-03T12:14:49.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.138+0000 7fb2aeffd700 1 -- 172.21.15.7:0/322507053 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb2b4003cb0 con 0x7fb2bc0c62e0 2023-03-03T12:14:49.140 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:14:49.140 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:14:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 -- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2a00682f0 msgr2=0x7fb2a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2a00682f0 0x7fb2a006a7a0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80090b0 tx=0x7fb2b8008030 comp rx=0 tx=0).stop 2023-03-03T12:14:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 -- 172.21.15.7:0/322507053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2bc0c62e0 msgr2=0x7fb2bc0c6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:14:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2bc0c62e0 0x7fb2bc0c6760 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc072670 tx=0x7fb2b40114b0 comp rx=0 tx=0).stop 2023-03-03T12:14:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 -- 172.21.15.7:0/322507053 shutdown_connections 2023-03-03T12:14:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2a00682f0 0x7fb2a006a7a0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2bc0c62e0 0x7fb2bc0c6760 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2a8003a70 0x7fb2bc1eb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 --2- 172.21.15.7:0/322507053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2bc1ed220 0x7fb2bc1ea520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:14:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 -- 172.21.15.7:0/322507053 >> 172.21.15.7:0/322507053 conn(0x7fb2bc0a94d0 msgr2=0x7fb2bc06a860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:14:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 -- 172.21.15.7:0/322507053 shutdown_connections 2023-03-03T12:14:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:14:49.140+0000 7fb2c4710700 1 -- 172.21.15.7:0/322507053 wait complete. 2023-03-03T12:14:49.144 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:14:49.154 INFO:tasks.mds_thrash.fs.[a]:waiting for 61 secs before thrashing 2023-03-03T12:14:55.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:14:55.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:13.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:15:13.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.231+0000 7fb424ef7700 1 Processor -- start 2023-03-03T12:15:13.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.232+0000 7fb424ef7700 1 -- start start 2023-03-03T12:15:13.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.232+0000 7fb424ef7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb420151d40 0x7fb4200736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:13.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.232+0000 7fb424ef7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb4200c2930 0x7fb420157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:13.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.232+0000 7fb424ef7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb42006d430 con 0x7fb420073c30 2023-03-03T12:15:13.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.232+0000 7fb424ef7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42006c2a0 con 0x7fb4200c2930 2023-03-03T12:15:13.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.232+0000 7fb424ef7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4201586e0 con 0x7fb420151d40 2023-03-03T12:15:13.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41dd9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb4200c2930 0x7fb420157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:13.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41ed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb420073c30 0x7fb420157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33720/0 (socket says 172.21.15.7:33720) 2023-03-03T12:15:13.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41ed9d700 1 -- 172.21.15.7:0/3590119269 learned_addr learned my addr 172.21.15.7:0/3590119269 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:15:13.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41e59c700 1 --2- 172.21.15.7:0/3590119269 >> v2:172.21.15.143:3300/0 conn(0x7fb420151d40 0x7fb4200736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:13.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41dd9b700 1 -- 172.21.15.7:0/3590119269 >> v2:172.21.15.143:3300/0 conn(0x7fb420151d40 msgr2=0x7fb4200736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:15:13.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41dd9b700 1 --2- 172.21.15.7:0/3590119269 >> v2:172.21.15.143:3300/0 conn(0x7fb420151d40 0x7fb4200736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:13.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41dd9b700 1 -- 172.21.15.7:0/3590119269 >> v1:172.21.15.7:6789/0 conn(0x7fb420073c30 legacy=0x7fb420157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:13.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41dd9b700 1 -- 172.21.15.7:0/3590119269 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42006cb00 con 0x7fb4200c2930 2023-03-03T12:15:13.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41dd9b700 1 --2- 172.21.15.7:0/3590119269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 0x7fb420157a90 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fb408009840 tx=0x7fb408009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d9522e7de346be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:13.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41d59a700 1 -- 172.21.15.7:0/3590119269 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408011830 con 0x7fb4200c2930 2023-03-03T12:15:13.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41d59a700 1 -- 172.21.15.7:0/3590119269 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb408011990 con 0x7fb4200c2930 2023-03-03T12:15:13.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.233+0000 7fb41d59a700 1 -- 172.21.15.7:0/3590119269 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408011d20 con 0x7fb4200c2930 2023-03-03T12:15:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.234+0000 7fb424ef7700 1 -- 172.21.15.7:0/3590119269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 msgr2=0x7fb420157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.234+0000 7fb424ef7700 1 --2- 172.21.15.7:0/3590119269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 0x7fb420157a90 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fb408009840 tx=0x7fb408009b50 comp rx=0 tx=0).stop 2023-03-03T12:15:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.234+0000 7fb424ef7700 1 -- 172.21.15.7:0/3590119269 shutdown_connections 2023-03-03T12:15:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.234+0000 7fb424ef7700 1 --2- 172.21.15.7:0/3590119269 >> v2:172.21.15.143:3300/0 conn(0x7fb420151d40 0x7fb4200736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.234+0000 7fb424ef7700 1 --2- 172.21.15.7:0/3590119269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 0x7fb420157a90 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.234+0000 7fb424ef7700 1 -- 172.21.15.7:0/3590119269 >> 172.21.15.7:0/3590119269 conn(0x7fb4200a94d0 msgr2=0x7fb4200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:15:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.234+0000 7fb424ef7700 1 -- 172.21.15.7:0/3590119269 shutdown_connections 2023-03-03T12:15:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.234+0000 7fb424ef7700 1 -- 172.21.15.7:0/3590119269 wait complete. 2023-03-03T12:15:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.235+0000 7fb424ef7700 1 Processor -- start 2023-03-03T12:15:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.235+0000 7fb424ef7700 1 -- start start 2023-03-03T12:15:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb424ef7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb420073c30 0x7fb4200aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb424ef7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 0x7fb420156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb424ef7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb420151d40 0x7fb4201eb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb424ef7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42014c580 con 0x7fb4200c2930 2023-03-03T12:15:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb424ef7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4200ac8d0 con 0x7fb420073c30 2023-03-03T12:15:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb424ef7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb420156800 con 0x7fb420151d40 2023-03-03T12:15:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb41dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 0x7fb420156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb41dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 0x7fb420156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39748/0 (socket says 172.21.15.7:39748) 2023-03-03T12:15:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb41dd9b700 1 -- 172.21.15.7:0/1989642719 learned_addr learned my addr 172.21.15.7:0/1989642719 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:15:13.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb41e59c700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb420073c30 0x7fb4200aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:13.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb41ed9d700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb420151d40 0x7fb4201eb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:13.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb41dd9b700 1 -- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb420151d40 msgr2=0x7fb4201eb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:13.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.236+0000 7fb41dd9b700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb420151d40 0x7fb4201eb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:13.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.237+0000 7fb41dd9b700 1 -- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb420073c30 msgr2=0x7fb4200aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:13.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.237+0000 7fb41dd9b700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb420073c30 0x7fb4200aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:13.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.237+0000 7fb41dd9b700 1 -- 172.21.15.7:0/1989642719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4080094a0 con 0x7fb4200c2930 2023-03-03T12:15:13.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.237+0000 7fb41dd9b700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 0x7fb420156c90 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fb4080090b0 tx=0x7fb408004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:13.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.237+0000 7fb416ffd700 1 -- 172.21.15.7:0/1989642719 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4080038a0 con 0x7fb4200c2930 2023-03-03T12:15:13.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.237+0000 7fb416ffd700 1 -- 172.21.15.7:0/1989642719 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40801e070 con 0x7fb4200c2930 2023-03-03T12:15:13.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.237+0000 7fb416ffd700 1 -- 172.21.15.7:0/1989642719 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408003c20 con 0x7fb4200c2930 2023-03-03T12:15:13.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.237+0000 7fb424ef7700 1 -- 172.21.15.7:0/1989642719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42006d0c0 con 0x7fb4200c2930 2023-03-03T12:15:13.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.237+0000 7fb424ef7700 1 -- 172.21.15.7:0/1989642719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb42006cde0 con 0x7fb4200c2930 2023-03-03T12:15:13.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.238+0000 7fb424ef7700 1 -- 172.21.15.7:0/1989642719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42006bea0 con 0x7fb4200c2930 2023-03-03T12:15:13.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.239+0000 7fb416ffd700 1 -- 172.21.15.7:0/1989642719 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb40800f050 con 0x7fb4200c2930 2023-03-03T12:15:13.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.239+0000 7fb416ffd700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4000682f0 0x7fb40006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:13.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.239+0000 7fb416ffd700 1 -- 172.21.15.7:0/1989642719 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb4080871d0 con 0x7fb4200c2930 2023-03-03T12:15:13.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.239+0000 7fb41e59c700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4000682f0 0x7fb40006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:13.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.240+0000 7fb41e59c700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4000682f0 0x7fb40006a7a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb410009930 tx=0x7fb410008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:13.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.243+0000 7fb416ffd700 1 -- 172.21.15.7:0/1989642719 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb408053570 con 0x7fb4200c2930 2023-03-03T12:15:13.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.481+0000 7fb424ef7700 1 -- 172.21.15.7:0/1989642719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb420156ae0 con 0x7fb4200c2930 2023-03-03T12:15:13.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.482+0000 7fb416ffd700 1 -- 172.21.15.7:0/1989642719 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb408020090 con 0x7fb4200c2930 2023-03-03T12:15:13.484 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:15:13.484 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:15:13.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 -- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4000682f0 msgr2=0x7fb40006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:13.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4000682f0 0x7fb40006a7a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb410009930 tx=0x7fb410008040 comp rx=0 tx=0).stop 2023-03-03T12:15:13.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 -- 172.21.15.7:0/1989642719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 msgr2=0x7fb420156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:13.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 0x7fb420156c90 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fb4080090b0 tx=0x7fb408004750 comp rx=0 tx=0).stop 2023-03-03T12:15:13.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 -- 172.21.15.7:0/1989642719 shutdown_connections 2023-03-03T12:15:13.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb420151d40 0x7fb4201eb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:13.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb420073c30 0x7fb4200aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:13.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4000682f0 0x7fb40006a7a0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:13.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 --2- 172.21.15.7:0/1989642719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4200c2930 0x7fb420156c90 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:13.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 -- 172.21.15.7:0/1989642719 >> 172.21.15.7:0/1989642719 conn(0x7fb4200a94d0 msgr2=0x7fb4200c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:15:13.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 -- 172.21.15.7:0/1989642719 shutdown_connections 2023-03-03T12:15:13.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:13.484+0000 7fb424ef7700 1 -- 172.21.15.7:0/1989642719 wait complete. 2023-03-03T12:15:13.488 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:15:13.497 INFO:tasks.mds_thrash.fs.[b]:waiting for 71 secs before thrashing 2023-03-03T12:15:25.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:25.681 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:50.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:15:50.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.293+0000 7fc4f6765700 1 Processor -- start 2023-03-03T12:15:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.293+0000 7fc4f6765700 1 -- start start 2023-03-03T12:15:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4f6765700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc4f014f9d0 0x7fc4f0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:50.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4f6765700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f00c2f30 con 0x7fc4f00abe20 2023-03-03T12:15:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4f6765700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f00c31d0 con 0x7fc4f014fda0 2023-03-03T12:15:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4f6765700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f01524a0 con 0x7fc4f014f9d0 2023-03-03T12:15:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4effff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc4f014f9d0 0x7fc4f0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4f4d02700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc4f00abe20 0x7fc4f01529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33758/0 (socket says 172.21.15.7:33758) 2023-03-03T12:15:50.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4f4d02700 1 -- 172.21.15.7:0/3128459778 learned_addr learned my addr 172.21.15.7:0/3128459778 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:15:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4effff700 1 -- 172.21.15.7:0/3128459778 >> v1:172.21.15.143:6789/0 conn(0x7fc4f014fda0 legacy=0x7fc4f0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:50.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4effff700 1 -- 172.21.15.7:0/3128459778 >> v1:172.21.15.7:6789/0 conn(0x7fc4f00abe20 legacy=0x7fc4f01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4effff700 1 -- 172.21.15.7:0/3128459778 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f00c34b0 con 0x7fc4f014f9d0 2023-03-03T12:15:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.294+0000 7fc4eeffd700 1 -- 172.21.15.7:0/3128459778 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554134029 0 0) 0x7fc4f00c2f30 con 0x7fc4f00abe20 2023-03-03T12:15:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4effff700 1 --2- 172.21.15.7:0/3128459778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f014f9d0 0x7fc4f0156430 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009840 tx=0x7fc4d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef3e0167e2b0de0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4eeffd700 1 -- 172.21.15.7:0/3128459778 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8011830 con 0x7fc4f014f9d0 2023-03-03T12:15:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4eeffd700 1 -- 172.21.15.7:0/3128459778 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d8011990 con 0x7fc4f014f9d0 2023-03-03T12:15:50.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4eeffd700 1 -- 172.21.15.7:0/3128459778 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8011d20 con 0x7fc4f014f9d0 2023-03-03T12:15:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4f6765700 1 -- 172.21.15.7:0/3128459778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f014f9d0 msgr2=0x7fc4f0156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4f6765700 1 --2- 172.21.15.7:0/3128459778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f014f9d0 0x7fc4f0156430 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009840 tx=0x7fc4d8009b50 comp rx=0 tx=0).stop 2023-03-03T12:15:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4f6765700 1 -- 172.21.15.7:0/3128459778 shutdown_connections 2023-03-03T12:15:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4f6765700 1 --2- 172.21.15.7:0/3128459778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f014f9d0 0x7fc4f0156430 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4f6765700 1 -- 172.21.15.7:0/3128459778 >> 172.21.15.7:0/3128459778 conn(0x7fc4f00c8900 msgr2=0x7fc4f00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:15:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4f6765700 1 -- 172.21.15.7:0/3128459778 shutdown_connections 2023-03-03T12:15:50.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.295+0000 7fc4f6765700 1 -- 172.21.15.7:0/3128459778 wait complete. 2023-03-03T12:15:50.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.296+0000 7fc4f6765700 1 Processor -- start 2023-03-03T12:15:50.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.297+0000 7fc4f6765700 1 -- start start 2023-03-03T12:15:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.297+0000 7fc4f6765700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f00abe20 0x7fc4f014cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.297+0000 7fc4f6765700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f014f9d0 0x7fc4f00a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.297+0000 7fc4f6765700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4f014fda0 0x7fc4f014bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.297+0000 7fc4f6765700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f00c21a0 con 0x7fc4f00abe20 2023-03-03T12:15:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.297+0000 7fc4f6765700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f014d010 con 0x7fc4f014f9d0 2023-03-03T12:15:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.297+0000 7fc4f6765700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f014c7f0 con 0x7fc4f014fda0 2023-03-03T12:15:50.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4effff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f00abe20 0x7fc4f014cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4effff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f00abe20 0x7fc4f014cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39780/0 (socket says 172.21.15.7:39780) 2023-03-03T12:15:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4effff700 1 -- 172.21.15.7:0/40118183 learned_addr learned my addr 172.21.15.7:0/40118183 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:15:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4ef7fe700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f014f9d0 0x7fc4f00a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4f4d02700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4f014fda0 0x7fc4f014bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4effff700 1 -- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4f014fda0 msgr2=0x7fc4f014bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4effff700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4f014fda0 0x7fc4f014bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:50.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4effff700 1 -- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f014f9d0 msgr2=0x7fc4f00a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4effff700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f014f9d0 0x7fc4f00a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4effff700 1 -- 172.21.15.7:0/40118183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d80094a0 con 0x7fc4f00abe20 2023-03-03T12:15:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4ef7fe700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f014f9d0 0x7fc4f00a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:15:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4effff700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f00abe20 0x7fc4f014cad0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80090b0 tx=0x7fc4d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4ecff9700 1 -- 172.21.15.7:0/40118183 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d801a980 con 0x7fc4f00abe20 2023-03-03T12:15:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4ecff9700 1 -- 172.21.15.7:0/40118183 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d801ab00 con 0x7fc4f00abe20 2023-03-03T12:15:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4f6765700 1 -- 172.21.15.7:0/40118183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f00ac350 con 0x7fc4f00abe20 2023-03-03T12:15:50.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4ecff9700 1 -- 172.21.15.7:0/40118183 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8024420 con 0x7fc4f00abe20 2023-03-03T12:15:50.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.298+0000 7fc4f6765700 1 -- 172.21.15.7:0/40118183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f00ab630 con 0x7fc4f00abe20 2023-03-03T12:15:50.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.299+0000 7fc4f6765700 1 -- 172.21.15.7:0/40118183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f0071fa0 con 0x7fc4f00abe20 2023-03-03T12:15:50.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.300+0000 7fc4ecff9700 1 -- 172.21.15.7:0/40118183 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc4d802b030 con 0x7fc4f00abe20 2023-03-03T12:15:50.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.300+0000 7fc4ecff9700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4d00682f0 0x7fc4d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:15:50.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.300+0000 7fc4ecff9700 1 -- 172.21.15.7:0/40118183 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc4d8087680 con 0x7fc4f00abe20 2023-03-03T12:15:50.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.300+0000 7fc4ef7fe700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4d00682f0 0x7fc4d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:15:50.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.301+0000 7fc4ef7fe700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4d00682f0 0x7fc4d006a7a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc4f006ed60 tx=0x7fc4e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:15:50.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.304+0000 7fc4ecff9700 1 -- 172.21.15.7:0/40118183 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc4d80539a0 con 0x7fc4f00abe20 2023-03-03T12:15:50.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.539+0000 7fc4f6765700 1 -- 172.21.15.7:0/40118183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4f01df030 con 0x7fc4f00abe20 2023-03-03T12:15:50.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.540+0000 7fc4ecff9700 1 -- 172.21.15.7:0/40118183 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc4f01df030 con 0x7fc4f00abe20 2023-03-03T12:15:50.541 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:15:50.542 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:15:50.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.541+0000 7fc4f6765700 1 -- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4d00682f0 msgr2=0x7fc4d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:50.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.541+0000 7fc4f6765700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4d00682f0 0x7fc4d006a7a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc4f006ed60 tx=0x7fc4e0009040 comp rx=0 tx=0).stop 2023-03-03T12:15:50.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.542+0000 7fc4f6765700 1 -- 172.21.15.7:0/40118183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f00abe20 msgr2=0x7fc4f014cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:15:50.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.542+0000 7fc4f6765700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f00abe20 0x7fc4f014cad0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80090b0 tx=0x7fc4d8004750 comp rx=0 tx=0).stop 2023-03-03T12:15:50.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.542+0000 7fc4f6765700 1 -- 172.21.15.7:0/40118183 shutdown_connections 2023-03-03T12:15:50.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.542+0000 7fc4f6765700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4f014fda0 0x7fc4f014bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:50.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.542+0000 7fc4f6765700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4d00682f0 0x7fc4d006a7a0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:50.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.542+0000 7fc4f6765700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4f00abe20 0x7fc4f014cad0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:50.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.542+0000 7fc4f6765700 1 --2- 172.21.15.7:0/40118183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4f014f9d0 0x7fc4f00a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:15:50.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.542+0000 7fc4f6765700 1 -- 172.21.15.7:0/40118183 >> 172.21.15.7:0/40118183 conn(0x7fc4f00c8900 msgr2=0x7fc4f014e340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:15:50.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.542+0000 7fc4f6765700 1 -- 172.21.15.7:0/40118183 shutdown_connections 2023-03-03T12:15:50.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:15:50.542+0000 7fc4f6765700 1 -- 172.21.15.7:0/40118183 wait complete. 2023-03-03T12:15:50.545 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:15:50.555 INFO:tasks.mds_thrash.fs.[a]:waiting for 76 secs before thrashing 2023-03-03T12:15:55.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:15:55.754 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:24.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:16:24.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.637+0000 7f65ed2a8700 1 Processor -- start 2023-03-03T12:16:24.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.637+0000 7f65ed2a8700 1 -- start start 2023-03-03T12:16:24.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.638+0000 7f65ed2a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e80caa30 0x7f65e80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:16:24.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.638+0000 7f65ed2a8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e80c7310 con 0x7f65e80c00e0 2023-03-03T12:16:24.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.638+0000 7f65ed2a8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e80bfd10 con 0x7f65e8154ff0 2023-03-03T12:16:24.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.638+0000 7f65ed2a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80bfa30 con 0x7f65e80caa30 2023-03-03T12:16:24.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.638+0000 7f65e67fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f65e80c00e0 0x7f65e80c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33782/0 (socket says 172.21.15.7:33782) 2023-03-03T12:16:24.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.638+0000 7f65e67fc700 1 -- 172.21.15.7:0/1390431867 learned_addr learned my addr 172.21.15.7:0/1390431867 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:16:24.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698644065 0 0) 0x7f65e80c7310 con 0x7f65e80c00e0 2023-03-03T12:16:24.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e6ffd700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e80caa30 0x7f65e80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:16:24.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d00035b0 con 0x7f65e80c00e0 2023-03-03T12:16:24.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1855038807 0 0) 0x7f65d00035b0 con 0x7f65e80c00e0 2023-03-03T12:16:24.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e80caa30 msgr2=0x7f65e80c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:16:24.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e80caa30 0x7f65e80c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:16:24.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 >> v1:172.21.15.143:6789/0 conn(0x7f65e8154ff0 legacy=0x7f65e815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:16:24.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e8154510 con 0x7f65e80c00e0 2023-03-03T12:16:24.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f65d80017f0 con 0x7f65e80c00e0 2023-03-03T12:16:24.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 >> v1:172.21.15.7:6789/0 conn(0x7f65e80c00e0 legacy=0x7f65e80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:16:24.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0003aa0 0x7f65d0003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:16:24.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d0004850 0x7f65d0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:16:24.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e80caa30 0x7f65d0005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:16:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80bfa30 con 0x7f65d0003aa0 2023-03-03T12:16:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d000bba0 con 0x7f65d0004850 2023-03-03T12:16:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d000bd60 con 0x7f65e80caa30 2023-03-03T12:16:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e6ffd700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0003aa0 0x7f65d0003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:16:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e67fc700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d0004850 0x7f65d0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:16:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e77fe700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e80caa30 0x7f65d0005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:16:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e6ffd700 1 -- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e80caa30 msgr2=0x7f65d0005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:16:24.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e6ffd700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e80caa30 0x7f65d0005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:16:24.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e6ffd700 1 -- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d0004850 msgr2=0x7f65d0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:16:24.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e6ffd700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d0004850 0x7f65d0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:16:24.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e6ffd700 1 -- 172.21.15.7:0/1390431867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00035b0 con 0x7f65d0003aa0 2023-03-03T12:16:24.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e67fc700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d0004850 0x7f65d0004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:16:24.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.639+0000 7f65e77fe700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e80caa30 0x7f65d0005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:16:24.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65e6ffd700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0003aa0 0x7f65d0003eb0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009b60 tx=0x7f65d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6447fbb1d695d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:16:24.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc00e6c0 con 0x7f65d0003aa0 2023-03-03T12:16:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc00e820 con 0x7f65d0003aa0 2023-03-03T12:16:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65e5ffb700 1 -- 172.21.15.7:0/1390431867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc00eb90 con 0x7f65d0003aa0 2023-03-03T12:16:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1390431867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0003aa0 msgr2=0x7f65d0003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:16:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65ed2a8700 1 --2- 172.21.15.7:0/1390431867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0003aa0 0x7f65d0003eb0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009b60 tx=0x7f65d0005880 comp rx=0 tx=0).stop 2023-03-03T12:16:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65e77fe700 1 -- 172.21.15.7:0/1390431867 reap_dead start 2023-03-03T12:16:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1390431867 shutdown_connections 2023-03-03T12:16:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1390431867 >> 172.21.15.7:0/1390431867 conn(0x7f65e8066e40 msgr2=0x7f65e8071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:16:24.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1390431867 shutdown_connections 2023-03-03T12:16:24.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.640+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1390431867 wait complete. 2023-03-03T12:16:24.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65ed2a8700 1 Processor -- start 2023-03-03T12:16:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65ed2a8700 1 -- start start 2023-03-03T12:16:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65ed2a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d0003aa0 0x7f65e814ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:16:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65ed2a8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e8157dd0 0x7f65e814c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:16:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65ed2a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e81ea5c0 0x7f65e81eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:16:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65ed2a8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e81506d0 con 0x7f65e8157dd0 2023-03-03T12:16:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65ed2a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e814b140 con 0x7f65d0003aa0 2023-03-03T12:16:24.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65ed2a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e81ea100 con 0x7f65e81ea5c0 2023-03-03T12:16:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65e67fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e8157dd0 0x7f65e814c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:16:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65e67fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e8157dd0 0x7f65e814c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39816/0 (socket says 172.21.15.7:39816) 2023-03-03T12:16:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.642+0000 7f65e67fc700 1 -- 172.21.15.7:0/1118187776 learned_addr learned my addr 172.21.15.7:0/1118187776 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:16:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65e6ffd700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d0003aa0 0x7f65e814ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:16:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65e77fe700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e81ea5c0 0x7f65e81eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:16:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65e67fc700 1 -- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e81ea5c0 msgr2=0x7f65e81eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:16:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65e67fc700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e81ea5c0 0x7f65e81eaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:16:24.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65e67fc700 1 -- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d0003aa0 msgr2=0x7f65e814ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:16:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65e67fc700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d0003aa0 0x7f65e814ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:16:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65e67fc700 1 -- 172.21.15.7:0/1118187776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e8154ad0 con 0x7f65e8157dd0 2023-03-03T12:16:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65e77fe700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e81ea5c0 0x7f65e81eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:16:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65e67fc700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e8157dd0 0x7f65e814c6b0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f65d800eaf0 tx=0x7f65d800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:16:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65cf7fe700 1 -- 172.21.15.7:0/1118187776 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8007e70 con 0x7f65e8157dd0 2023-03-03T12:16:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65cf7fe700 1 -- 172.21.15.7:0/1118187776 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d80072e0 con 0x7f65e8157dd0 2023-03-03T12:16:24.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65cf7fe700 1 -- 172.21.15.7:0/1118187776 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8007670 con 0x7f65e8157dd0 2023-03-03T12:16:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1118187776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e81547f0 con 0x7f65e8157dd0 2023-03-03T12:16:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.643+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1118187776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e80bf650 con 0x7f65e8157dd0 2023-03-03T12:16:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.644+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1118187776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e80c7f90 con 0x7f65e8157dd0 2023-03-03T12:16:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.645+0000 7f65cf7fe700 1 -- 172.21.15.7:0/1118187776 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f65d8005830 con 0x7f65e8157dd0 2023-03-03T12:16:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.645+0000 7f65cf7fe700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65d40682f0 0x7f65d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:16:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.645+0000 7f65cf7fe700 1 -- 172.21.15.7:0/1118187776 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f65d8086120 con 0x7f65e8157dd0 2023-03-03T12:16:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.645+0000 7f65e6ffd700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65d40682f0 0x7f65d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:16:24.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.646+0000 7f65e6ffd700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65d40682f0 0x7f65d406a7a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0090b0 tx=0x7f65dc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:16:24.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.649+0000 7f65cf7fe700 1 -- 172.21.15.7:0/1118187776 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f65d8052580 con 0x7f65e8157dd0 2023-03-03T12:16:24.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.889+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1118187776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65e80c1210 con 0x7f65e8157dd0 2023-03-03T12:16:24.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.889+0000 7f65cf7fe700 1 -- 172.21.15.7:0/1118187776 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f65d8055ba0 con 0x7f65e8157dd0 2023-03-03T12:16:24.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:16:24.891 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:16:24.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.891+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65d40682f0 msgr2=0x7f65d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:16:24.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.891+0000 7f65ed2a8700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65d40682f0 0x7f65d406a7a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0090b0 tx=0x7f65dc008030 comp rx=0 tx=0).stop 2023-03-03T12:16:24.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.891+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1118187776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e8157dd0 msgr2=0x7f65e814c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:16:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.891+0000 7f65ed2a8700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e8157dd0 0x7f65e814c6b0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f65d800eaf0 tx=0x7f65d800ee00 comp rx=0 tx=0).stop 2023-03-03T12:16:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.891+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1118187776 shutdown_connections 2023-03-03T12:16:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.891+0000 7f65ed2a8700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65e81ea5c0 0x7f65e81eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:16:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.891+0000 7f65ed2a8700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d0003aa0 0x7f65e814ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:16:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.891+0000 7f65ed2a8700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65d40682f0 0x7f65d406a7a0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:16:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.891+0000 7f65ed2a8700 1 --2- 172.21.15.7:0/1118187776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65e8157dd0 0x7f65e814c6b0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:16:24.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.891+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1118187776 >> 172.21.15.7:0/1118187776 conn(0x7f65e8066e40 msgr2=0x7f65e80c77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:16:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.892+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1118187776 shutdown_connections 2023-03-03T12:16:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:16:24.892+0000 7f65ed2a8700 1 -- 172.21.15.7:0/1118187776 wait complete. 2023-03-03T12:16:24.895 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:16:24.905 INFO:tasks.mds_thrash.fs.[b]:waiting for 72 secs before thrashing 2023-03-03T12:16:25.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:25.824 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:55.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:16:55.896 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:06.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:17:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.694+0000 7f49d191b700 1 Processor -- start 2023-03-03T12:17:06.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.694+0000 7f49d191b700 1 -- start start 2023-03-03T12:17:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49d191b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc151b00 0x7f49cc06e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49d191b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f49cc154420 0x7f49cc154870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49d191b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49cc06e730 con 0x7f49cc0c4630 2023-03-03T12:17:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49d191b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49cc1583e0 con 0x7f49cc154420 2023-03-03T12:17:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49d191b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc157ab0 con 0x7f49cc151b00 2023-03-03T12:17:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49ca7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f49cc0c4630 0x7f49cc0a94c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33832/0 (socket says 172.21.15.7:33832) 2023-03-03T12:17:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49ca7fc700 1 -- 172.21.15.7:0/877589370 learned_addr learned my addr 172.21.15.7:0/877589370 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:17:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49cb7fe700 1 --2- 172.21.15.7:0/877589370 >> v2:172.21.15.143:3300/0 conn(0x7f49cc154420 0x7f49cc154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450283869 0 0) 0x7f49cc06e730 con 0x7f49cc0c4630 2023-03-03T12:17:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b40035b0 con 0x7f49cc0c4630 2023-03-03T12:17:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49caffd700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc151b00 0x7f49cc06e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1916023073 0 0) 0x7f49b40035b0 con 0x7f49cc0c4630 2023-03-03T12:17:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc151b00 msgr2=0x7f49cc06e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:17:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc151b00 0x7f49cc06e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 >> v2:172.21.15.143:3300/0 conn(0x7f49cc154420 msgr2=0x7f49cc154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:17:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 --2- 172.21.15.7:0/877589370 >> v2:172.21.15.143:3300/0 conn(0x7f49cc154420 0x7f49cc154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc157d90 con 0x7f49cc0c4630 2023-03-03T12:17:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f49bc001670 con 0x7f49cc0c4630 2023-03-03T12:17:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 >> v1:172.21.15.7:6789/0 conn(0x7f49cc0c4630 legacy=0x7f49cc0a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.695+0000 7f49c9ffb700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b4003a70 0x7f49b4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49c9ffb700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc151b00 0x7f49b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49c9ffb700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b4004d90 0x7f49b4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc1583e0 con 0x7f49b4004d90 2023-03-03T12:17:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc157ab0 con 0x7f49b4003a70 2023-03-03T12:17:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b400ba60 con 0x7f49cc151b00 2023-03-03T12:17:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49cb7fe700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b4004d90 0x7f49b4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49caffd700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b4003a70 0x7f49b4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49ca7fc700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc151b00 0x7f49b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49cb7fe700 1 -- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc151b00 msgr2=0x7f49b4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:17:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49cb7fe700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc151b00 0x7f49b4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49cb7fe700 1 -- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b4003a70 msgr2=0x7f49b4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:17:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49cb7fe700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49b4003a70 0x7f49b4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49cb7fe700 1 -- 172.21.15.7:0/877589370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b40035b0 con 0x7f49b4004d90 2023-03-03T12:17:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49cb7fe700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b4004d90 0x7f49b4005200 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f49c400e170 tx=0x7f49b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7fb92788aaf5df4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c400ec90 con 0x7f49b4004d90 2023-03-03T12:17:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c400f040 con 0x7f49b4004d90 2023-03-03T12:17:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.696+0000 7f49c9ffb700 1 -- 172.21.15.7:0/877589370 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4014480 con 0x7f49b4004d90 2023-03-03T12:17:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.697+0000 7f49d191b700 1 -- 172.21.15.7:0/877589370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b4004d90 msgr2=0x7f49b4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.697+0000 7f49d191b700 1 --2- 172.21.15.7:0/877589370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49b4004d90 0x7f49b4005200 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f49c400e170 tx=0x7f49b4005880 comp rx=0 tx=0).stop 2023-03-03T12:17:06.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.697+0000 7f49cb7fe700 1 -- 172.21.15.7:0/877589370 reap_dead start 2023-03-03T12:17:06.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.697+0000 7f49d191b700 1 -- 172.21.15.7:0/877589370 shutdown_connections 2023-03-03T12:17:06.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.697+0000 7f49d191b700 1 -- 172.21.15.7:0/877589370 >> 172.21.15.7:0/877589370 conn(0x7f49cc0c84d0 msgr2=0x7f49cc0c88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:17:06.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.697+0000 7f49d191b700 1 -- 172.21.15.7:0/877589370 shutdown_connections 2023-03-03T12:17:06.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.697+0000 7f49d191b700 1 -- 172.21.15.7:0/877589370 wait complete. 2023-03-03T12:17:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.698+0000 7f49d191b700 1 Processor -- start 2023-03-03T12:17:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.698+0000 7f49d191b700 1 -- start start 2023-03-03T12:17:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49d191b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49cc149350 0x7f49cc14fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49d191b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc149720 0x7f49cc152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49d191b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49cc06ec70 0x7f49cc06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49d191b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc14de60 con 0x7f49cc149350 2023-03-03T12:17:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49d191b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc1540e0 con 0x7f49cc06ec70 2023-03-03T12:17:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49d191b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc14ff40 con 0x7f49cc149720 2023-03-03T12:17:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49caffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49cc149350 0x7f49cc14fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49caffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49cc149350 0x7f49cc14fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39860/0 (socket says 172.21.15.7:39860) 2023-03-03T12:17:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49caffd700 1 -- 172.21.15.7:0/729513939 learned_addr learned my addr 172.21.15.7:0/729513939 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:17:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49ca7fc700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc149720 0x7f49cc152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49cb7fe700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49cc06ec70 0x7f49cc06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49caffd700 1 -- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc149720 msgr2=0x7f49cc152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49caffd700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc149720 0x7f49cc152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49caffd700 1 -- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49cc06ec70 msgr2=0x7f49cc06f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49caffd700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49cc06ec70 0x7f49cc06f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.699+0000 7f49caffd700 1 -- 172.21.15.7:0/729513939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc158870 con 0x7f49cc149350 2023-03-03T12:17:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.700+0000 7f49caffd700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49cc149350 0x7f49cc14fa00 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f49c0011ad0 tx=0x7f49c0011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.700+0000 7f49b3fff700 1 -- 172.21.15.7:0/729513939 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000f940 con 0x7f49cc149350 2023-03-03T12:17:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.700+0000 7f49b3fff700 1 -- 172.21.15.7:0/729513939 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c000faa0 con 0x7f49cc149350 2023-03-03T12:17:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.700+0000 7f49b3fff700 1 -- 172.21.15.7:0/729513939 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000fe50 con 0x7f49cc149350 2023-03-03T12:17:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.700+0000 7f49d191b700 1 -- 172.21.15.7:0/729513939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49cc158070 con 0x7f49cc149350 2023-03-03T12:17:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.700+0000 7f49d191b700 1 -- 172.21.15.7:0/729513939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49cc0c3c30 con 0x7f49cc149350 2023-03-03T12:17:06.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.701+0000 7f49d191b700 1 -- 172.21.15.7:0/729513939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49cc074bb0 con 0x7f49cc149350 2023-03-03T12:17:06.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.701+0000 7f49b3fff700 1 -- 172.21.15.7:0/729513939 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f49c0014440 con 0x7f49cc149350 2023-03-03T12:17:06.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.702+0000 7f49b3fff700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49b80682f0 0x7f49b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:06.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.702+0000 7f49b3fff700 1 -- 172.21.15.7:0/729513939 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f49c0086970 con 0x7f49cc149350 2023-03-03T12:17:06.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.702+0000 7f49ca7fc700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49b80682f0 0x7f49b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:06.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.702+0000 7f49ca7fc700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49b80682f0 0x7f49b806a7a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0c8420 tx=0x7f49bc007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:06.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.706+0000 7f49b3fff700 1 -- 172.21.15.7:0/729513939 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f49c0052d70 con 0x7f49cc149350 2023-03-03T12:17:06.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.946+0000 7f49d191b700 1 -- 172.21.15.7:0/729513939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49cc071f80 con 0x7f49cc149350 2023-03-03T12:17:06.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.947+0000 7f49b3fff700 1 -- 172.21.15.7:0/729513939 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f49c0056390 con 0x7f49cc149350 2023-03-03T12:17:06.948 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:17:06.948 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:17:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 -- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49b80682f0 msgr2=0x7f49b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49b80682f0 0x7f49b806a7a0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f49cc0c8420 tx=0x7f49bc007980 comp rx=0 tx=0).stop 2023-03-03T12:17:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 -- 172.21.15.7:0/729513939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49cc149350 msgr2=0x7f49cc14fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49cc149350 0x7f49cc14fa00 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f49c0011ad0 tx=0x7f49c0011de0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 -- 172.21.15.7:0/729513939 shutdown_connections 2023-03-03T12:17:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49cc06ec70 0x7f49cc06f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49b80682f0 0x7f49b806a7a0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49cc149350 0x7f49cc14fa00 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 --2- 172.21.15.7:0/729513939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49cc149720 0x7f49cc152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 -- 172.21.15.7:0/729513939 >> 172.21.15.7:0/729513939 conn(0x7f49cc0c84d0 msgr2=0x7f49cc0c50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:17:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 -- 172.21.15.7:0/729513939 shutdown_connections 2023-03-03T12:17:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:06.949+0000 7f49d191b700 1 -- 172.21.15.7:0/729513939 wait complete. 2023-03-03T12:17:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:17:06.962 INFO:tasks.mds_thrash.fs.[a]:waiting for 37 secs before thrashing 2023-03-03T12:17:25.957 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:25.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:36.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:17:37.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.044+0000 7f47824fb700 1 Processor -- start 2023-03-03T12:17:37.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.044+0000 7f47824fb700 1 -- start start 2023-03-03T12:17:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.044+0000 7f47824fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f477c0c43d0 0x7f477c0c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.044+0000 7f47824fb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477c072920 con 0x7f477c14e560 2023-03-03T12:17:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.044+0000 7f47824fb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477c06bef0 con 0x7f477c154590 2023-03-03T12:17:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.044+0000 7f47824fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477c072380 con 0x7f477c0c43d0 2023-03-03T12:17:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f477c14e560 0x7f477c071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33860/0 (socket says 172.21.15.7:33860) 2023-03-03T12:17:37.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477bfff700 1 -- 172.21.15.7:0/203587440 learned_addr learned my addr 172.21.15.7:0/203587440 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:17:37.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035932048 0 0) 0x7f476c001110 con 0x7f477c14e560 2023-03-03T12:17:37.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47640035b0 con 0x7f477c14e560 2023-03-03T12:17:37.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477b7fe700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f477c0c43d0 0x7f477c0c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:37.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3145846682 0 0) 0x7f47640035b0 con 0x7f477c14e560 2023-03-03T12:17:37.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f477c0c43d0 msgr2=0x7f477c0c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:17:37.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f477c0c43d0 0x7f477c0c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:37.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 >> v1:172.21.15.143:6789/0 conn(0x7f477c154590 legacy=0x7f477c1549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:37.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c06b910 con 0x7f477c14e560 2023-03-03T12:17:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f476c001800 con 0x7f477c14e560 2023-03-03T12:17:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 >> v1:172.21.15.7:6789/0 conn(0x7f477c14e560 legacy=0x7f477c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f477c0c43d0 0x7f4764003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4764004020 0x7f4764004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4764004d90 0x7f4764005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c072380 con 0x7f4764004020 2023-03-03T12:17:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476400bb50 con 0x7f4764004d90 2023-03-03T12:17:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f476400bd10 con 0x7f477c0c43d0 2023-03-03T12:17:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477b7fe700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4764004020 0x7f4764004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f477bfff700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f477c0c43d0 0x7f4764003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.045+0000 7f4780a98700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4764004d90 0x7f4764005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f477b7fe700 1 -- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f477c0c43d0 msgr2=0x7f4764003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:17:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f477b7fe700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f477c0c43d0 0x7f4764003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f477b7fe700 1 -- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4764004d90 msgr2=0x7f4764005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:17:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f477b7fe700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4764004d90 0x7f4764005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f477b7fe700 1 -- 172.21.15.7:0/203587440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47640035b0 con 0x7f4764004020 2023-03-03T12:17:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f477b7fe700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4764004020 0x7f4764004850 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f4770009b60 tx=0x7f4764005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0463e08660f09f6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000e6c0 con 0x7f4764004020 2023-03-03T12:17:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477000e820 con 0x7f4764004020 2023-03-03T12:17:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f477affd700 1 -- 172.21.15.7:0/203587440 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000eb90 con 0x7f4764004020 2023-03-03T12:17:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f47824fb700 1 -- 172.21.15.7:0/203587440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4764004020 msgr2=0x7f4764004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f47824fb700 1 --2- 172.21.15.7:0/203587440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4764004020 0x7f4764004850 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f4770009b60 tx=0x7f4764005880 comp rx=0 tx=0).stop 2023-03-03T12:17:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.046+0000 7f4780a98700 1 -- 172.21.15.7:0/203587440 reap_dead start 2023-03-03T12:17:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.047+0000 7f47824fb700 1 -- 172.21.15.7:0/203587440 shutdown_connections 2023-03-03T12:17:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.047+0000 7f47824fb700 1 -- 172.21.15.7:0/203587440 >> 172.21.15.7:0/203587440 conn(0x7f477c0736f0 msgr2=0x7f477c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:17:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.047+0000 7f47824fb700 1 -- 172.21.15.7:0/203587440 shutdown_connections 2023-03-03T12:17:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.047+0000 7f47824fb700 1 -- 172.21.15.7:0/203587440 wait complete. 2023-03-03T12:17:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.048+0000 7f47824fb700 1 Processor -- start 2023-03-03T12:17:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.048+0000 7f47824fb700 1 -- start start 2023-03-03T12:17:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f47824fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4764004020 0x7f477c1ee090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f47824fb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f477c14d7a0 0x7f477c144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f47824fb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f477c0c7070 0x7f477c14af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f47824fb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c14c6e0 con 0x7f477c0c7070 2023-03-03T12:17:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f47824fb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c1edc00 con 0x7f477c14d7a0 2023-03-03T12:17:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f47824fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477c0bedd0 con 0x7f4764004020 2023-03-03T12:17:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4780a98700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f477c0c7070 0x7f477c14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4780a98700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f477c0c7070 0x7f477c14af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39896/0 (socket says 172.21.15.7:39896) 2023-03-03T12:17:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4780a98700 1 -- 172.21.15.7:0/2997058451 learned_addr learned my addr 172.21.15.7:0/2997058451 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:17:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f477b7fe700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f477c14d7a0 0x7f477c144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4780a98700 1 -- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4764004020 msgr2=0x7f477c1ee090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f477bfff700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4764004020 0x7f477c1ee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4780a98700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4764004020 0x7f477c1ee090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4780a98700 1 -- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f477c14d7a0 msgr2=0x7f477c144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4780a98700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f477c14d7a0 0x7f477c144d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4780a98700 1 -- 172.21.15.7:0/2997058451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c06bb80 con 0x7f477c0c7070 2023-03-03T12:17:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4780a98700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f477c0c7070 0x7f477c14af10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f477c0c6040 tx=0x7f477400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4778ff9700 1 -- 172.21.15.7:0/2997058451 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774003c10 con 0x7f477c0c7070 2023-03-03T12:17:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4778ff9700 1 -- 172.21.15.7:0/2997058451 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4774003d90 con 0x7f477c0c7070 2023-03-03T12:17:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f4778ff9700 1 -- 172.21.15.7:0/2997058451 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477400db70 con 0x7f477c0c7070 2023-03-03T12:17:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.049+0000 7f47824fb700 1 -- 172.21.15.7:0/2997058451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477c0aacc0 con 0x7f477c0c7070 2023-03-03T12:17:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.050+0000 7f47824fb700 1 -- 172.21.15.7:0/2997058451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477c155100 con 0x7f477c0c7070 2023-03-03T12:17:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.050+0000 7f47824fb700 1 -- 172.21.15.7:0/2997058451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477c074bb0 con 0x7f477c0c7070 2023-03-03T12:17:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.051+0000 7f4778ff9700 1 -- 172.21.15.7:0/2997058451 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f477400dcd0 con 0x7f477c0c7070 2023-03-03T12:17:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.051+0000 7f4778ff9700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4768068320 0x7f476806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.051+0000 7f4778ff9700 1 -- 172.21.15.7:0/2997058451 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4774086470 con 0x7f477c0c7070 2023-03-03T12:17:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.052+0000 7f477bfff700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4768068320 0x7f476806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.052+0000 7f477bfff700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4768068320 0x7f476806a7d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f476c000fb0 tx=0x7f476c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:37.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.056+0000 7f4778ff9700 1 -- 172.21.15.7:0/2997058451 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4774052810 con 0x7f477c0c7070 2023-03-03T12:17:37.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.294+0000 7f47824fb700 1 -- 172.21.15.7:0/2997058451 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f477c1edd40 con 0x7f477c0c7070 2023-03-03T12:17:37.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.295+0000 7f4778ff9700 1 -- 172.21.15.7:0/2997058451 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4774055e30 con 0x7f477c0c7070 2023-03-03T12:17:37.297 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:17:37.297 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:17:37.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 -- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4768068320 msgr2=0x7f476806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:37.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4768068320 0x7f476806a7d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f476c000fb0 tx=0x7f476c009040 comp rx=0 tx=0).stop 2023-03-03T12:17:37.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 -- 172.21.15.7:0/2997058451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f477c0c7070 msgr2=0x7f477c14af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:37.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f477c0c7070 0x7f477c14af10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f477c0c6040 tx=0x7f477400d430 comp rx=0 tx=0).stop 2023-03-03T12:17:37.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 -- 172.21.15.7:0/2997058451 shutdown_connections 2023-03-03T12:17:37.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4768068320 0x7f476806a7d0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:37.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f477c0c7070 0x7f477c14af10 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:37.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4764004020 0x7f477c1ee090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:37.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 --2- 172.21.15.7:0/2997058451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f477c14d7a0 0x7f477c144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:37.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 -- 172.21.15.7:0/2997058451 >> 172.21.15.7:0/2997058451 conn(0x7f477c0736f0 msgr2=0x7f477c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:17:37.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 -- 172.21.15.7:0/2997058451 shutdown_connections 2023-03-03T12:17:37.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:37.297+0000 7f47824fb700 1 -- 172.21.15.7:0/2997058451 wait complete. 2023-03-03T12:17:37.300 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:17:37.310 INFO:tasks.mds_thrash.fs.[b]:waiting for 50 secs before thrashing 2023-03-03T12:17:43.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:17:44.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.100+0000 7fd68af36700 1 Processor -- start 2023-03-03T12:17:44.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.100+0000 7fd68af36700 1 -- start start 2023-03-03T12:17:44.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd68af36700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6841586c0 0x7fd68406d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:44.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd68af36700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd6840c1da0 0x7fd68406d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:44.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd68af36700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd684154560 con 0x7fd68413a0f0 2023-03-03T12:17:44.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd68af36700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd684154840 con 0x7fd6840c1da0 2023-03-03T12:17:44.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd68af36700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd684157be0 con 0x7fd6841586c0 2023-03-03T12:17:44.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd688cd2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd68413a0f0 0x7fd6840c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33886/0 (socket says 172.21.15.7:33886) 2023-03-03T12:17:44.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd688cd2700 1 -- 172.21.15.7:0/516472306 learned_addr learned my addr 172.21.15.7:0/516472306 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:17:44.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd683fff700 1 --2- 172.21.15.7:0/516472306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6841586c0 0x7fd68406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:44.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd6894d3700 1 --2- 172.21.15.7:0/516472306 >> v2:172.21.15.143:3300/0 conn(0x7fd6840c1da0 0x7fd68406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:44.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd6837fe700 1 -- 172.21.15.7:0/516472306 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794110997 0 0) 0x7fd684154560 con 0x7fd68413a0f0 2023-03-03T12:17:44.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd6837fe700 1 -- 172.21.15.7:0/516472306 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6740035b0 con 0x7fd68413a0f0 2023-03-03T12:17:44.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd6837fe700 1 -- 172.21.15.7:0/516472306 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 830213702 0 0) 0x7fd6740035b0 con 0x7fd68413a0f0 2023-03-03T12:17:44.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd6894d3700 1 -- 172.21.15.7:0/516472306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6841586c0 msgr2=0x7fd68406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:17:44.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd6894d3700 1 --2- 172.21.15.7:0/516472306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6841586c0 0x7fd68406d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:44.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd6894d3700 1 -- 172.21.15.7:0/516472306 >> v1:172.21.15.7:6789/0 conn(0x7fd68413a0f0 legacy=0x7fd6840c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:44.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.101+0000 7fd6894d3700 1 -- 172.21.15.7:0/516472306 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd684157ec0 con 0x7fd6840c1da0 2023-03-03T12:17:44.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd6894d3700 1 --2- 172.21.15.7:0/516472306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6840c1da0 0x7fd68406d780 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009d90 tx=0x7fd67c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb7e6071dde60b0f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:44.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd6837fe700 1 -- 172.21.15.7:0/516472306 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c00e070 con 0x7fd6840c1da0 2023-03-03T12:17:44.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd6837fe700 1 -- 172.21.15.7:0/516472306 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67c003a20 con 0x7fd6840c1da0 2023-03-03T12:17:44.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd6837fe700 1 -- 172.21.15.7:0/516472306 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678001670 con 0x7fd6840c1da0 2023-03-03T12:17:44.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd68af36700 1 -- 172.21.15.7:0/516472306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6840c1da0 msgr2=0x7fd68406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:44.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd68af36700 1 --2- 172.21.15.7:0/516472306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6840c1da0 0x7fd68406d780 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009d90 tx=0x7fd67c00b330 comp rx=0 tx=0).stop 2023-03-03T12:17:44.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd68af36700 1 -- 172.21.15.7:0/516472306 shutdown_connections 2023-03-03T12:17:44.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd68af36700 1 --2- 172.21.15.7:0/516472306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6840c1da0 0x7fd68406d780 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:44.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd68af36700 1 --2- 172.21.15.7:0/516472306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6841586c0 0x7fd68406d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:44.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd68af36700 1 -- 172.21.15.7:0/516472306 >> 172.21.15.7:0/516472306 conn(0x7fd68406c110 msgr2=0x7fd68406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:17:44.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd68af36700 1 -- 172.21.15.7:0/516472306 shutdown_connections 2023-03-03T12:17:44.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.102+0000 7fd68af36700 1 -- 172.21.15.7:0/516472306 wait complete. 2023-03-03T12:17:44.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.104+0000 7fd68af36700 1 Processor -- start 2023-03-03T12:17:44.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.104+0000 7fd68af36700 1 -- start start 2023-03-03T12:17:44.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd68af36700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6840c1da0 0x7fd684156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:44.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd68af36700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd68413a0f0 0x7fd684070c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:44.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd68af36700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6841586c0 0x7fd684150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:44.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd68af36700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6840ace70 con 0x7fd6840c1da0 2023-03-03T12:17:44.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd68af36700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6840707f0 con 0x7fd6841586c0 2023-03-03T12:17:44.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd68af36700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6840701a0 con 0x7fd68413a0f0 2023-03-03T12:17:44.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd688cd2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6840c1da0 0x7fd684156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:44.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd688cd2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6840c1da0 0x7fd684156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39912/0 (socket says 172.21.15.7:39912) 2023-03-03T12:17:44.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd688cd2700 1 -- 172.21.15.7:0/3010589937 learned_addr learned my addr 172.21.15.7:0/3010589937 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:17:44.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd6894d3700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6841586c0 0x7fd684150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:44.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.105+0000 7fd683fff700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd68413a0f0 0x7fd684070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:44.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd688cd2700 1 -- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd68413a0f0 msgr2=0x7fd684070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:44.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd688cd2700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd68413a0f0 0x7fd684070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:44.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd688cd2700 1 -- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6841586c0 msgr2=0x7fd684150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:44.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd688cd2700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6841586c0 0x7fd684150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:44.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd688cd2700 1 -- 172.21.15.7:0/3010589937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c009a70 con 0x7fd6840c1da0 2023-03-03T12:17:44.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd688cd2700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6840c1da0 0x7fd684156b00 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fd67800c780 tx=0x7fd67800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:44.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd6817fa700 1 -- 172.21.15.7:0/3010589937 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678004c80 con 0x7fd6840c1da0 2023-03-03T12:17:44.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd6817fa700 1 -- 172.21.15.7:0/3010589937 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd678004de0 con 0x7fd6840c1da0 2023-03-03T12:17:44.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd68af36700 1 -- 172.21.15.7:0/3010589937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68406e1a0 con 0x7fd6840c1da0 2023-03-03T12:17:44.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd6817fa700 1 -- 172.21.15.7:0/3010589937 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6780072f0 con 0x7fd6840c1da0 2023-03-03T12:17:44.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.106+0000 7fd68af36700 1 -- 172.21.15.7:0/3010589937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6841581a0 con 0x7fd6840c1da0 2023-03-03T12:17:44.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.107+0000 7fd68af36700 1 -- 172.21.15.7:0/3010589937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68406e330 con 0x7fd6840c1da0 2023-03-03T12:17:44.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.107+0000 7fd6817fa700 1 -- 172.21.15.7:0/3010589937 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd678010070 con 0x7fd6840c1da0 2023-03-03T12:17:44.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.108+0000 7fd6817fa700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 0x7fd66806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:17:44.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.108+0000 7fd6817fa700 1 -- 172.21.15.7:0/3010589937 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd678085290 con 0x7fd6840c1da0 2023-03-03T12:17:44.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.108+0000 7fd683fff700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 0x7fd66806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:17:44.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.109+0000 7fd683fff700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 0x7fd66806a7a0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fd6840c16a0 tx=0x7fd670008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:17:44.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.112+0000 7fd6817fa700 1 -- 172.21.15.7:0/3010589937 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd6780515d0 con 0x7fd6840c1da0 2023-03-03T12:17:44.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.350+0000 7fd68af36700 1 -- 172.21.15.7:0/3010589937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd684070ad0 con 0x7fd6840c1da0 2023-03-03T12:17:44.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.351+0000 7fd6817fa700 1 -- 172.21.15.7:0/3010589937 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd678018090 con 0x7fd6840c1da0 2023-03-03T12:17:44.353 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:17:44.354 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:17:44.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 -- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 msgr2=0x7fd66806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:44.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 0x7fd66806a7a0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fd6840c16a0 tx=0x7fd670008040 comp rx=0 tx=0).stop 2023-03-03T12:17:44.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 -- 172.21.15.7:0/3010589937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6840c1da0 msgr2=0x7fd684156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:17:44.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6840c1da0 0x7fd684156b00 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fd67800c780 tx=0x7fd67800cb40 comp rx=0 tx=0).stop 2023-03-03T12:17:44.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 -- 172.21.15.7:0/3010589937 shutdown_connections 2023-03-03T12:17:44.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6841586c0 0x7fd684150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:44.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 0x7fd66806a7a0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:44.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6840c1da0 0x7fd684156b00 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:44.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 --2- 172.21.15.7:0/3010589937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd68413a0f0 0x7fd684070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:17:44.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 -- 172.21.15.7:0/3010589937 >> 172.21.15.7:0/3010589937 conn(0x7fd68406c110 msgr2=0x7fd6840c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:17:44.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 -- 172.21.15.7:0/3010589937 shutdown_connections 2023-03-03T12:17:44.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:17:44.353+0000 7fd68af36700 1 -- 172.21.15.7:0/3010589937 wait complete. 2023-03-03T12:17:44.358 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:17:44.367 INFO:tasks.mds_thrash.fs.[a]:waiting for 116 secs before thrashing 2023-03-03T12:17:56.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:17:56.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:26.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:26.179 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:27.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:18:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.448+0000 7f9c59e07700 1 Processor -- start 2023-03-03T12:18:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.448+0000 7f9c59e07700 1 -- start start 2023-03-03T12:18:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.449+0000 7f9c59e07700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c5414f9d0 0x7f9c54156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:18:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.449+0000 7f9c59e07700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c5414fda0 0x7f9c54071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:18:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.449+0000 7f9c59e07700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c540c2f30 con 0x7f9c540abe20 2023-03-03T12:18:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.449+0000 7f9c59e07700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c540c31d0 con 0x7f9c5414fda0 2023-03-03T12:18:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.449+0000 7f9c59e07700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c541524a0 con 0x7f9c5414f9d0 2023-03-03T12:18:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.449+0000 7f9c52ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c5414fda0 0x7f9c54071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:18:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.449+0000 7f9c52ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c5414fda0 0x7f9c54071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39942/0 (socket says 172.21.15.7:39942) 2023-03-03T12:18:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.449+0000 7f9c52ffd700 1 -- 172.21.15.7:0/2493473397 learned_addr learned my addr 172.21.15.7:0/2493473397 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:18:27.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.450+0000 7f9c537fe700 1 --2- 172.21.15.7:0/2493473397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c5414f9d0 0x7f9c54156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:18:27.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.450+0000 7f9c52ffd700 1 -- 172.21.15.7:0/2493473397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c5414f9d0 msgr2=0x7f9c54156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:18:27.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.450+0000 7f9c52ffd700 1 --2- 172.21.15.7:0/2493473397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c5414f9d0 0x7f9c54156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:18:27.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.450+0000 7f9c52ffd700 1 -- 172.21.15.7:0/2493473397 >> v1:172.21.15.143:6789/0 conn(0x7f9c540abe20 legacy=0x7f9c541529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:18:27.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.450+0000 7f9c52ffd700 1 -- 172.21.15.7:0/2493473397 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c540c34b0 con 0x7f9c5414fda0 2023-03-03T12:18:27.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.450+0000 7f9c52ffd700 1 --2- 172.21.15.7:0/2493473397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414fda0 0x7f9c54071890 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f9c480098f0 tx=0x7f9c48009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136c037ae3014821 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:18:27.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.450+0000 7f9c527fc700 1 -- 172.21.15.7:0/2493473397 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4800b8a0 con 0x7f9c5414fda0 2023-03-03T12:18:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.450+0000 7f9c527fc700 1 -- 172.21.15.7:0/2493473397 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4800ba00 con 0x7f9c5414fda0 2023-03-03T12:18:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.450+0000 7f9c527fc700 1 -- 172.21.15.7:0/2493473397 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4800bd90 con 0x7f9c5414fda0 2023-03-03T12:18:27.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.451+0000 7f9c59e07700 1 -- 172.21.15.7:0/2493473397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414fda0 msgr2=0x7f9c54071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:18:27.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.451+0000 7f9c59e07700 1 --2- 172.21.15.7:0/2493473397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414fda0 0x7f9c54071890 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f9c480098f0 tx=0x7f9c48009c00 comp rx=0 tx=0).stop 2023-03-03T12:18:27.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.451+0000 7f9c59e07700 1 -- 172.21.15.7:0/2493473397 shutdown_connections 2023-03-03T12:18:27.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.451+0000 7f9c59e07700 1 --2- 172.21.15.7:0/2493473397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c5414f9d0 0x7f9c54156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:18:27.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.451+0000 7f9c59e07700 1 --2- 172.21.15.7:0/2493473397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414fda0 0x7f9c54071890 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:18:27.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.451+0000 7f9c59e07700 1 -- 172.21.15.7:0/2493473397 >> 172.21.15.7:0/2493473397 conn(0x7f9c540c8900 msgr2=0x7f9c540c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:18:27.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.451+0000 7f9c59e07700 1 -- 172.21.15.7:0/2493473397 shutdown_connections 2023-03-03T12:18:27.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.451+0000 7f9c59e07700 1 -- 172.21.15.7:0/2493473397 wait complete. 2023-03-03T12:18:27.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.452+0000 7f9c59e07700 1 Processor -- start 2023-03-03T12:18:27.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.452+0000 7f9c59e07700 1 -- start start 2023-03-03T12:18:27.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c59e07700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c540abe20 0x7f9c540bd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:18:27.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c59e07700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414f9d0 0x7f9c540c23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:18:27.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c59e07700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c541549a0 0x7f9c54146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:18:27.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c59e07700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c541de220 con 0x7f9c5414f9d0 2023-03-03T12:18:27.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c59e07700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5414f800 con 0x7f9c540abe20 2023-03-03T12:18:27.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c59e07700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c54153230 con 0x7f9c541549a0 2023-03-03T12:18:27.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c52ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414f9d0 0x7f9c540c23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:18:27.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c52ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414f9d0 0x7f9c540c23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39946/0 (socket says 172.21.15.7:39946) 2023-03-03T12:18:27.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c52ffd700 1 -- 172.21.15.7:0/4276277894 learned_addr learned my addr 172.21.15.7:0/4276277894 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:18:27.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c52ffd700 1 -- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c541549a0 msgr2=0x7f9c54146e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:18:27.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c53fff700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c541549a0 0x7f9c54146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:18:27.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c537fe700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c540abe20 0x7f9c540bd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:18:27.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c52ffd700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c541549a0 0x7f9c54146e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:18:27.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c52ffd700 1 -- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c540abe20 msgr2=0x7f9c540bd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:18:27.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c52ffd700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c540abe20 0x7f9c540bd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:18:27.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c52ffd700 1 -- 172.21.15.7:0/4276277894 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c48009550 con 0x7f9c5414f9d0 2023-03-03T12:18:27.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c53fff700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c541549a0 0x7f9c54146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:18:27.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.453+0000 7f9c52ffd700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414f9d0 0x7f9c540c23f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f9c48000c00 tx=0x7f9c48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:18:27.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.454+0000 7f9c3bfff700 1 -- 172.21.15.7:0/4276277894 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c480043c0 con 0x7f9c5414f9d0 2023-03-03T12:18:27.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.454+0000 7f9c3bfff700 1 -- 172.21.15.7:0/4276277894 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4801f070 con 0x7f9c5414f9d0 2023-03-03T12:18:27.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.454+0000 7f9c59e07700 1 -- 172.21.15.7:0/4276277894 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c540ac350 con 0x7f9c5414f9d0 2023-03-03T12:18:27.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.454+0000 7f9c3bfff700 1 -- 172.21.15.7:0/4276277894 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c480036a0 con 0x7f9c5414f9d0 2023-03-03T12:18:27.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.454+0000 7f9c59e07700 1 -- 172.21.15.7:0/4276277894 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c540ab630 con 0x7f9c5414f9d0 2023-03-03T12:18:27.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.455+0000 7f9c59e07700 1 -- 172.21.15.7:0/4276277894 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c540c3e10 con 0x7f9c5414f9d0 2023-03-03T12:18:27.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.455+0000 7f9c3bfff700 1 -- 172.21.15.7:0/4276277894 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9c4802a020 con 0x7f9c5414f9d0 2023-03-03T12:18:27.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.456+0000 7f9c3bfff700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c400682f0 0x7f9c4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:18:27.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.456+0000 7f9c3bfff700 1 -- 172.21.15.7:0/4276277894 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9c480877a0 con 0x7f9c5414f9d0 2023-03-03T12:18:27.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.456+0000 7f9c537fe700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c400682f0 0x7f9c4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:18:27.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.456+0000 7f9c537fe700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c400682f0 0x7f9c4006a7a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009950 tx=0x7f9c44008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:18:27.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.460+0000 7f9c3bfff700 1 -- 172.21.15.7:0/4276277894 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9c48053ac0 con 0x7f9c5414f9d0 2023-03-03T12:18:27.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.699+0000 7f9c59e07700 1 -- 172.21.15.7:0/4276277894 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c54074bb0 con 0x7f9c5414f9d0 2023-03-03T12:18:27.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.699+0000 7f9c3bfff700 1 -- 172.21.15.7:0/4276277894 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9c480570e0 con 0x7f9c5414f9d0 2023-03-03T12:18:27.701 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:18:27.701 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:18:27.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 -- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c400682f0 msgr2=0x7f9c4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:18:27.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c400682f0 0x7f9c4006a7a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009950 tx=0x7f9c44008040 comp rx=0 tx=0).stop 2023-03-03T12:18:27.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 -- 172.21.15.7:0/4276277894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414f9d0 msgr2=0x7f9c540c23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:18:27.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414f9d0 0x7f9c540c23f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f9c48000c00 tx=0x7f9c48004750 comp rx=0 tx=0).stop 2023-03-03T12:18:27.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 -- 172.21.15.7:0/4276277894 shutdown_connections 2023-03-03T12:18:27.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c541549a0 0x7f9c54146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:18:27.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c540abe20 0x7f9c540bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:18:27.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c400682f0 0x7f9c4006a7a0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:18:27.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 --2- 172.21.15.7:0/4276277894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c5414f9d0 0x7f9c540c23f0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:18:27.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 -- 172.21.15.7:0/4276277894 >> 172.21.15.7:0/4276277894 conn(0x7f9c540c8900 msgr2=0x7f9c540c6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:18:27.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 -- 172.21.15.7:0/4276277894 shutdown_connections 2023-03-03T12:18:27.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:18:27.701+0000 7f9c59e07700 1 -- 172.21.15.7:0/4276277894 wait complete. 2023-03-03T12:18:27.705 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:18:27.715 INFO:tasks.mds_thrash.fs.[b]:waiting for 48 secs before thrashing 2023-03-03T12:18:56.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:18:56.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:15.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:19:15.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.853+0000 7f13f99c0700 1 Processor -- start 2023-03-03T12:19:15.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.853+0000 7f13f99c0700 1 -- start start 2023-03-03T12:19:15.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f99c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40caa30 0x7f13f40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:15.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f99c0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f40c7310 con 0x7f13f40c00e0 2023-03-03T12:19:15.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f99c0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f40bfd10 con 0x7f13f4154ff0 2023-03-03T12:19:15.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f99c0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f40bfa30 con 0x7f13f40caa30 2023-03-03T12:19:15.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f27fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f13f40c00e0 0x7f13f40c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33962/0 (socket says 172.21.15.7:33962) 2023-03-03T12:19:15.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f27fc700 1 -- 172.21.15.7:0/3092606504 learned_addr learned my addr 172.21.15.7:0/3092606504 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:19:15.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f2ffd700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40caa30 0x7f13f40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:15.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337040052 0 0) 0x7f13f40c7310 con 0x7f13f40c00e0 2023-03-03T12:19:15.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13dc0035b0 con 0x7f13f40c00e0 2023-03-03T12:19:15.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3916762588 0 0) 0x7f13dc0035b0 con 0x7f13f40c00e0 2023-03-03T12:19:15.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40caa30 msgr2=0x7f13f40c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:19:15.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f1ffb700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40caa30 0x7f13f40c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:15.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 >> v1:172.21.15.143:6789/0 conn(0x7f13f4154ff0 legacy=0x7f13f415ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:15.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.854+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f4154510 con 0x7f13f40c00e0 2023-03-03T12:19:15.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f13e40017f0 con 0x7f13f40c00e0 2023-03-03T12:19:15.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 >> v1:172.21.15.7:6789/0 conn(0x7f13f40c00e0 legacy=0x7f13f40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:15.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 0x7f13dc003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:15.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13dc004850 0x7f13dc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:15.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40caa30 0x7f13dc005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:15.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f40bfa30 con 0x7f13dc003aa0 2023-03-03T12:19:15.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc00bba0 con 0x7f13dc004850 2023-03-03T12:19:15.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13dc00bd60 con 0x7f13f40caa30 2023-03-03T12:19:15.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f2ffd700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 0x7f13dc003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:15.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f27fc700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13dc004850 0x7f13dc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:15.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f37fe700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40caa30 0x7f13dc005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:15.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f2ffd700 1 -- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40caa30 msgr2=0x7f13dc005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:19:15.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f2ffd700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40caa30 0x7f13dc005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:15.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f2ffd700 1 -- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13dc004850 msgr2=0x7f13dc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:19:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f2ffd700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13dc004850 0x7f13dc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f2ffd700 1 -- 172.21.15.7:0/3092606504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc0035b0 con 0x7f13dc003aa0 2023-03-03T12:19:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f2ffd700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 0x7f13dc003eb0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009b60 tx=0x7f13dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb3804e7c90d1b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e800e6c0 con 0x7f13dc003aa0 2023-03-03T12:19:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e800e820 con 0x7f13dc003aa0 2023-03-03T12:19:15.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.855+0000 7f13f1ffb700 1 -- 172.21.15.7:0/3092606504 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e800eb90 con 0x7f13dc003aa0 2023-03-03T12:19:15.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.856+0000 7f13f99c0700 1 -- 172.21.15.7:0/3092606504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 msgr2=0x7f13dc003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:15.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.856+0000 7f13f99c0700 1 --2- 172.21.15.7:0/3092606504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 0x7f13dc003eb0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009b60 tx=0x7f13dc005880 comp rx=0 tx=0).stop 2023-03-03T12:19:15.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.856+0000 7f13f37fe700 1 -- 172.21.15.7:0/3092606504 reap_dead start 2023-03-03T12:19:15.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.856+0000 7f13f99c0700 1 -- 172.21.15.7:0/3092606504 shutdown_connections 2023-03-03T12:19:15.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.856+0000 7f13f99c0700 1 -- 172.21.15.7:0/3092606504 >> 172.21.15.7:0/3092606504 conn(0x7f13f4066e40 msgr2=0x7f13f4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:19:15.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.856+0000 7f13f99c0700 1 -- 172.21.15.7:0/3092606504 shutdown_connections 2023-03-03T12:19:15.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.856+0000 7f13f99c0700 1 -- 172.21.15.7:0/3092606504 wait complete. 2023-03-03T12:19:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.857+0000 7f13f99c0700 1 Processor -- start 2023-03-03T12:19:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.857+0000 7f13f99c0700 1 -- start start 2023-03-03T12:19:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f99c0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 0x7f13f414bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f99c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40c31b0 0x7f13f41ed870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f99c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f414c150 0x7f13f41eddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f99c0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f414b7f0 con 0x7f13dc003aa0 2023-03-03T12:19:15.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f99c0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f40c2580 con 0x7f13f414c150 2023-03-03T12:19:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f99c0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f40c1fc0 con 0x7f13f40c31b0 2023-03-03T12:19:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 0x7f13f414bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 0x7f13f414bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39992/0 (socket says 172.21.15.7:39992) 2023-03-03T12:19:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f2ffd700 1 -- 172.21.15.7:0/4170348412 learned_addr learned my addr 172.21.15.7:0/4170348412 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:19:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f27fc700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40c31b0 0x7f13f41ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f37fe700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f414c150 0x7f13f41eddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f2ffd700 1 -- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40c31b0 msgr2=0x7f13f41ed870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:15.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f2ffd700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40c31b0 0x7f13f41ed870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f2ffd700 1 -- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f414c150 msgr2=0x7f13f41eddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f2ffd700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f414c150 0x7f13f41eddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f2ffd700 1 -- 172.21.15.7:0/4170348412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f4154ad0 con 0x7f13dc003aa0 2023-03-03T12:19:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f27fc700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40c31b0 0x7f13f41ed870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:19:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.858+0000 7f13f37fe700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f414c150 0x7f13f41eddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:19:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.859+0000 7f13f2ffd700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 0x7f13f414bc10 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f13dc005810 tx=0x7f13e80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:15.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.859+0000 7f13dbfff700 1 -- 172.21.15.7:0/4170348412 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e8021070 con 0x7f13dc003aa0 2023-03-03T12:19:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.859+0000 7f13dbfff700 1 -- 172.21.15.7:0/4170348412 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e80045d0 con 0x7f13dc003aa0 2023-03-03T12:19:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.859+0000 7f13f99c0700 1 -- 172.21.15.7:0/4170348412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f41547f0 con 0x7f13dc003aa0 2023-03-03T12:19:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.859+0000 7f13dbfff700 1 -- 172.21.15.7:0/4170348412 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e800e6c0 con 0x7f13dc003aa0 2023-03-03T12:19:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.859+0000 7f13f99c0700 1 -- 172.21.15.7:0/4170348412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f40bf650 con 0x7f13dc003aa0 2023-03-03T12:19:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.860+0000 7f13f99c0700 1 -- 172.21.15.7:0/4170348412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f40af330 con 0x7f13dc003aa0 2023-03-03T12:19:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.860+0000 7f13dbfff700 1 -- 172.21.15.7:0/4170348412 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f13e802e030 con 0x7f13dc003aa0 2023-03-03T12:19:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.860+0000 7f13dbfff700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13e00682f0 0x7f13e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:15.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.861+0000 7f13dbfff700 1 -- 172.21.15.7:0/4170348412 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f13e80874a0 con 0x7f13dc003aa0 2023-03-03T12:19:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.861+0000 7f13f27fc700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13e00682f0 0x7f13e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.861+0000 7f13f27fc700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13e00682f0 0x7f13e006a7a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f13e40011d0 tx=0x7f13e4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:15.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:15.865+0000 7f13dbfff700 1 -- 172.21.15.7:0/4170348412 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13e8053840 con 0x7f13dc003aa0 2023-03-03T12:19:16.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.104+0000 7f13f99c0700 1 -- 172.21.15.7:0/4170348412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13f40afa20 con 0x7f13dc003aa0 2023-03-03T12:19:16.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.104+0000 7f13dbfff700 1 -- 172.21.15.7:0/4170348412 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f13e8056e60 con 0x7f13dc003aa0 2023-03-03T12:19:16.106 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:19:16.106 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:19:16.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 -- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13e00682f0 msgr2=0x7f13e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:16.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13e00682f0 0x7f13e006a7a0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f13e40011d0 tx=0x7f13e4009040 comp rx=0 tx=0).stop 2023-03-03T12:19:16.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 -- 172.21.15.7:0/4170348412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 msgr2=0x7f13f414bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:16.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 0x7f13f414bc10 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f13dc005810 tx=0x7f13e80047c0 comp rx=0 tx=0).stop 2023-03-03T12:19:16.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 -- 172.21.15.7:0/4170348412 shutdown_connections 2023-03-03T12:19:16.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f414c150 0x7f13f41eddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:16.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13e00682f0 0x7f13e006a7a0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:16.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc003aa0 0x7f13f414bc10 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:16.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 --2- 172.21.15.7:0/4170348412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13f40c31b0 0x7f13f41ed870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:16.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 -- 172.21.15.7:0/4170348412 >> 172.21.15.7:0/4170348412 conn(0x7f13f4066e40 msgr2=0x7f13f40c68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:19:16.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 -- 172.21.15.7:0/4170348412 shutdown_connections 2023-03-03T12:19:16.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:16.107+0000 7f13f99c0700 1 -- 172.21.15.7:0/4170348412 wait complete. 2023-03-03T12:19:16.110 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:19:16.120 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2023-03-03T12:19:26.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:26.455 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:40.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:19:40.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.505+0000 7fc1c4749700 1 Processor -- start 2023-03-03T12:19:40.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.505+0000 7fc1c4749700 1 -- start start 2023-03-03T12:19:40.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.506+0000 7fc1c4749700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc1bc151b20 0x7fc1bc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:40.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.506+0000 7fc1c4749700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc073c30 0x7fc1bc1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:40.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.506+0000 7fc1c4749700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1bc06cb70 con 0x7fc1bc0c2950 2023-03-03T12:19:40.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.506+0000 7fc1c4749700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1bc06c2f0 con 0x7fc1bc151b20 2023-03-03T12:19:40.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.506+0000 7fc1c4749700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc06ce50 con 0x7fc1bc073c30 2023-03-03T12:19:40.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.506+0000 7fc1c24e5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc1bc151b20 0x7fc1bc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:40.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c2ce6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc073c30 0x7fc1bc1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:40.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c24e5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc1bc151b20 0x7fc1bc0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:35782/0 (socket says 172.21.15.7:35782) 2023-03-03T12:19:40.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c24e5700 1 -- 172.21.15.7:0/4036805884 learned_addr learned my addr 172.21.15.7:0/4036805884 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:19:40.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c1ce4700 1 --1- 172.21.15.7:0/4036805884 >> v1:172.21.15.143:6789/0 conn(0x7fc1bc0c2950 0x7fc1bc157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:37508/0 (socket says 172.21.15.7:37508) 2023-03-03T12:19:40.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c2ce6700 1 -- 172.21.15.7:0/4036805884 >> v2:172.21.15.143:3300/0 conn(0x7fc1bc151b20 msgr2=0x7fc1bc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:19:40.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c2ce6700 1 --2- 172.21.15.7:0/4036805884 >> v2:172.21.15.143:3300/0 conn(0x7fc1bc151b20 0x7fc1bc0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:40.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c2ce6700 1 -- 172.21.15.7:0/4036805884 >> v1:172.21.15.143:6789/0 conn(0x7fc1bc0c2950 legacy=0x7fc1bc157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:40.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c2ce6700 1 -- 172.21.15.7:0/4036805884 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc06d130 con 0x7fc1bc073c30 2023-03-03T12:19:40.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c14e3700 1 -- 172.21.15.7:0/4036805884 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350422154 0 0) 0x7fc1bc06cb70 con 0x7fc1bc0c2950 2023-03-03T12:19:40.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c24e5700 1 --2- 172.21.15.7:0/4036805884 >> v2:172.21.15.143:3300/0 conn(0x7fc1bc151b20 0x7fc1bc0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:19:40.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c2ce6700 1 --2- 172.21.15.7:0/4036805884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc073c30 0x7fc1bc1581f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009d90 tx=0x7fc1b400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8354f23d6e858ee3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:40.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c14e3700 1 -- 172.21.15.7:0/4036805884 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b400e070 con 0x7fc1bc073c30 2023-03-03T12:19:40.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c14e3700 1 -- 172.21.15.7:0/4036805884 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b400bb20 con 0x7fc1bc073c30 2023-03-03T12:19:40.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.507+0000 7fc1c14e3700 1 -- 172.21.15.7:0/4036805884 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4012430 con 0x7fc1bc073c30 2023-03-03T12:19:40.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.508+0000 7fc1c4749700 1 -- 172.21.15.7:0/4036805884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc073c30 msgr2=0x7fc1bc1581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:40.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.508+0000 7fc1c4749700 1 --2- 172.21.15.7:0/4036805884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc073c30 0x7fc1bc1581f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4009d90 tx=0x7fc1b400b330 comp rx=0 tx=0).stop 2023-03-03T12:19:40.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.508+0000 7fc1c4749700 1 -- 172.21.15.7:0/4036805884 shutdown_connections 2023-03-03T12:19:40.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.508+0000 7fc1c4749700 1 --2- 172.21.15.7:0/4036805884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc073c30 0x7fc1bc1581f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:40.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.508+0000 7fc1c4749700 1 --2- 172.21.15.7:0/4036805884 >> v2:172.21.15.143:3300/0 conn(0x7fc1bc151b20 0x7fc1bc0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:40.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.508+0000 7fc1c4749700 1 -- 172.21.15.7:0/4036805884 >> 172.21.15.7:0/4036805884 conn(0x7fc1bc0a94d0 msgr2=0x7fc1bc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:19:40.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.508+0000 7fc1c4749700 1 -- 172.21.15.7:0/4036805884 shutdown_connections 2023-03-03T12:19:40.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.508+0000 7fc1c4749700 1 -- 172.21.15.7:0/4036805884 wait complete. 2023-03-03T12:19:40.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.509+0000 7fc1c4749700 1 Processor -- start 2023-03-03T12:19:40.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.509+0000 7fc1c4749700 1 -- start start 2023-03-03T12:19:40.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c4749700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1bc073c30 0x7fc1bc156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:40.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c4749700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1bc0c2950 0x7fc1bc0c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:40.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c4749700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc151b20 0x7fc1bc0c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:40.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c4749700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc1e0430 con 0x7fc1bc073c30 2023-03-03T12:19:40.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c4749700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc1e0170 con 0x7fc1bc0c2950 2023-03-03T12:19:40.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c4749700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc1dec60 con 0x7fc1bc151b20 2023-03-03T12:19:40.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c24e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1bc073c30 0x7fc1bc156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:40.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c24e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1bc073c30 0x7fc1bc156360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40012/0 (socket says 172.21.15.7:40012) 2023-03-03T12:19:40.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c24e5700 1 -- 172.21.15.7:0/3015121917 learned_addr learned my addr 172.21.15.7:0/3015121917 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:19:40.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c1ce4700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1bc0c2950 0x7fc1bc0c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:40.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c2ce6700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc151b20 0x7fc1bc0c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:40.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c24e5700 1 -- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc151b20 msgr2=0x7fc1bc0c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:40.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c24e5700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc151b20 0x7fc1bc0c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:40.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c24e5700 1 -- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1bc0c2950 msgr2=0x7fc1bc0c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:40.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c24e5700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1bc0c2950 0x7fc1bc0c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:40.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.510+0000 7fc1c24e5700 1 -- 172.21.15.7:0/3015121917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac009550 con 0x7fc1bc073c30 2023-03-03T12:19:40.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.511+0000 7fc1c24e5700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1bc073c30 0x7fc1bc156360 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac00e9e0 tx=0x7fc1ac00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:40.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.511+0000 7fc1b2ffd700 1 -- 172.21.15.7:0/3015121917 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00ee80 con 0x7fc1bc073c30 2023-03-03T12:19:40.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.511+0000 7fc1b2ffd700 1 -- 172.21.15.7:0/3015121917 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac00c850 con 0x7fc1bc073c30 2023-03-03T12:19:40.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.511+0000 7fc1c4749700 1 -- 172.21.15.7:0/3015121917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b4009a70 con 0x7fc1bc073c30 2023-03-03T12:19:40.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.511+0000 7fc1b2ffd700 1 -- 172.21.15.7:0/3015121917 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00cbc0 con 0x7fc1bc073c30 2023-03-03T12:19:40.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.511+0000 7fc1c4749700 1 -- 172.21.15.7:0/3015121917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1bc158730 con 0x7fc1bc073c30 2023-03-03T12:19:40.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.512+0000 7fc1c4749700 1 -- 172.21.15.7:0/3015121917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1bc06d410 con 0x7fc1bc073c30 2023-03-03T12:19:40.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.512+0000 7fc1b2ffd700 1 -- 172.21.15.7:0/3015121917 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc1ac019030 con 0x7fc1bc073c30 2023-03-03T12:19:40.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.513+0000 7fc1b2ffd700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 0x7fc1a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:19:40.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.513+0000 7fc1b2ffd700 1 -- 172.21.15.7:0/3015121917 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc1ac086010 con 0x7fc1bc073c30 2023-03-03T12:19:40.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.513+0000 7fc1c1ce4700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 0x7fc1a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:19:40.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.513+0000 7fc1c1ce4700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 0x7fc1a006a7a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc1554e0 tx=0x7fc1b8007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:19:40.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.517+0000 7fc1b2ffd700 1 -- 172.21.15.7:0/3015121917 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc1ac0523b0 con 0x7fc1bc073c30 2023-03-03T12:19:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.752+0000 7fc1c4749700 1 -- 172.21.15.7:0/3015121917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1bc151ef0 con 0x7fc1bc073c30 2023-03-03T12:19:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.753+0000 7fc1b2ffd700 1 -- 172.21.15.7:0/3015121917 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc1ac017090 con 0x7fc1bc073c30 2023-03-03T12:19:40.754 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:19:40.754 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:19:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 -- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 msgr2=0x7fc1a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:40.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 0x7fc1a006a7a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc1554e0 tx=0x7fc1b8007e70 comp rx=0 tx=0).stop 2023-03-03T12:19:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 -- 172.21.15.7:0/3015121917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1bc073c30 msgr2=0x7fc1bc156360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:19:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1bc073c30 0x7fc1bc156360 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac00e9e0 tx=0x7fc1ac00ecf0 comp rx=0 tx=0).stop 2023-03-03T12:19:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 -- 172.21.15.7:0/3015121917 shutdown_connections 2023-03-03T12:19:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1bc151b20 0x7fc1bc0c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 0x7fc1a006a7a0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1bc073c30 0x7fc1bc156360 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:40.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 --2- 172.21.15.7:0/3015121917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1bc0c2950 0x7fc1bc0c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:19:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 -- 172.21.15.7:0/3015121917 >> 172.21.15.7:0/3015121917 conn(0x7fc1bc0a94d0 msgr2=0x7fc1bc070aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:19:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 -- 172.21.15.7:0/3015121917 shutdown_connections 2023-03-03T12:19:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:19:40.755+0000 7fc1c4749700 1 -- 172.21.15.7:0/3015121917 wait complete. 2023-03-03T12:19:40.758 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:19:40.768 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2023-03-03T12:19:56.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:19:56.551 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T12:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T12:20:07.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:20:07.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.906+0000 7fb24f3e9700 1 Processor -- start 2023-03-03T12:20:07.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.907+0000 7fb24f3e9700 1 -- start start 2023-03-03T12:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.907+0000 7fb24f3e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb248151d40 0x7fb2480736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.907+0000 7fb24f3e9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb2480c2930 0x7fb248157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.907+0000 7fb24f3e9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24806d430 con 0x7fb248073c30 2023-03-03T12:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.907+0000 7fb24f3e9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24806c2a0 con 0x7fb2480c2930 2023-03-03T12:20:07.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.907+0000 7fb24f3e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481586e0 con 0x7fb248151d40 2023-03-03T12:20:07.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.908+0000 7fb24c984700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb2480c2930 0x7fb248157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:07.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.908+0000 7fb24d185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb248151d40 0x7fb2480736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:07.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.908+0000 7fb24c984700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb2480c2930 0x7fb248157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:35804/0 (socket says 172.21.15.7:35804) 2023-03-03T12:20:07.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.908+0000 7fb24d986700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb248073c30 0x7fb248157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:37528/0 (socket says 172.21.15.7:37528) 2023-03-03T12:20:07.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.908+0000 7fb24c984700 1 -- 172.21.15.7:0/416550165 learned_addr learned my addr 172.21.15.7:0/416550165 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:20:07.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.908+0000 7fb24c984700 1 -- 172.21.15.7:0/416550165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb248151d40 msgr2=0x7fb2480736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:20:07.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.908+0000 7fb24c984700 1 --2- 172.21.15.7:0/416550165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb248151d40 0x7fb2480736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:07.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.908+0000 7fb24c984700 1 -- 172.21.15.7:0/416550165 >> v1:172.21.15.143:6789/0 conn(0x7fb248073c30 legacy=0x7fb248157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:07.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.908+0000 7fb24c984700 1 -- 172.21.15.7:0/416550165 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24806cb00 con 0x7fb2480c2930 2023-03-03T12:20:07.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.908+0000 7fb24c984700 1 --2- 172.21.15.7:0/416550165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2480c2930 0x7fb248157a90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb2440098f0 tx=0x7fb244009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e39ce35b13b504ea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:20:07.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb23ffff700 1 -- 172.21.15.7:0/416550165 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244011830 con 0x7fb2480c2930 2023-03-03T12:20:07.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb23ffff700 1 -- 172.21.15.7:0/416550165 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb244011990 con 0x7fb2480c2930 2023-03-03T12:20:07.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb23ffff700 1 -- 172.21.15.7:0/416550165 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244011d20 con 0x7fb2480c2930 2023-03-03T12:20:07.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb24f3e9700 1 -- 172.21.15.7:0/416550165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2480c2930 msgr2=0x7fb248157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:07.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb24f3e9700 1 --2- 172.21.15.7:0/416550165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2480c2930 0x7fb248157a90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb2440098f0 tx=0x7fb244009c00 comp rx=0 tx=0).stop 2023-03-03T12:20:07.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb24f3e9700 1 -- 172.21.15.7:0/416550165 shutdown_connections 2023-03-03T12:20:07.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb24f3e9700 1 --2- 172.21.15.7:0/416550165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb248151d40 0x7fb2480736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:07.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb24f3e9700 1 --2- 172.21.15.7:0/416550165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2480c2930 0x7fb248157a90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:07.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb24f3e9700 1 -- 172.21.15.7:0/416550165 >> 172.21.15.7:0/416550165 conn(0x7fb2480a94d0 msgr2=0x7fb2480a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:20:07.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb24f3e9700 1 -- 172.21.15.7:0/416550165 shutdown_connections 2023-03-03T12:20:07.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.909+0000 7fb24f3e9700 1 -- 172.21.15.7:0/416550165 wait complete. 2023-03-03T12:20:07.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.911+0000 7fb24f3e9700 1 Processor -- start 2023-03-03T12:20:07.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.911+0000 7fb24f3e9700 1 -- start start 2023-03-03T12:20:07.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.911+0000 7fb24f3e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb248073c30 0x7fb2480ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:07.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.911+0000 7fb24f3e9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2480c2930 0x7fb2481edbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:07.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.911+0000 7fb24f3e9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb248151d40 0x7fb2480c0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:07.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.911+0000 7fb24f3e9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481ed720 con 0x7fb2480c2930 2023-03-03T12:20:07.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.911+0000 7fb24f3e9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481ed3d0 con 0x7fb248151d40 2023-03-03T12:20:07.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.911+0000 7fb24f3e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480c6550 con 0x7fb248073c30 2023-03-03T12:20:07.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24c984700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2480c2930 0x7fb2481edbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:07.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24d185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb248073c30 0x7fb2480ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:07.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24c984700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2480c2930 0x7fb2481edbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40036/0 (socket says 172.21.15.7:40036) 2023-03-03T12:20:07.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24c984700 1 -- 172.21.15.7:0/2365640758 learned_addr learned my addr 172.21.15.7:0/2365640758 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:20:07.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24d986700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb248151d40 0x7fb2480c0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:07.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24c984700 1 -- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb248073c30 msgr2=0x7fb2480ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:07.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24c984700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb248073c30 0x7fb2480ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:07.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24c984700 1 -- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb248151d40 msgr2=0x7fb2480c0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:07.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24c984700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb248151d40 0x7fb2480c0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:07.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24c984700 1 -- 172.21.15.7:0/2365640758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb244009550 con 0x7fb2480c2930 2023-03-03T12:20:07.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24c984700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2480c2930 0x7fb2481edbb0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fb244009c40 tx=0x7fb244004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:20:07.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb23dffb700 1 -- 172.21.15.7:0/2365640758 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244004510 con 0x7fb2480c2930 2023-03-03T12:20:07.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb23dffb700 1 -- 172.21.15.7:0/2365640758 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24401e070 con 0x7fb2480c2930 2023-03-03T12:20:07.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb24f3e9700 1 -- 172.21.15.7:0/2365640758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24806d0c0 con 0x7fb2480c2930 2023-03-03T12:20:07.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.912+0000 7fb23dffb700 1 -- 172.21.15.7:0/2365640758 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2440038e0 con 0x7fb2480c2930 2023-03-03T12:20:07.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.913+0000 7fb24f3e9700 1 -- 172.21.15.7:0/2365640758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24806cde0 con 0x7fb2480c2930 2023-03-03T12:20:07.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.914+0000 7fb24f3e9700 1 -- 172.21.15.7:0/2365640758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24806bea0 con 0x7fb2480c2930 2023-03-03T12:20:07.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.914+0000 7fb23dffb700 1 -- 172.21.15.7:0/2365640758 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb24400f050 con 0x7fb2480c2930 2023-03-03T12:20:07.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.914+0000 7fb23dffb700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb22c0682f0 0x7fb22c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:07.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.915+0000 7fb23dffb700 1 -- 172.21.15.7:0/2365640758 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb244086eb0 con 0x7fb2480c2930 2023-03-03T12:20:07.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.915+0000 7fb24d185700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb22c0682f0 0x7fb22c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.915+0000 7fb24d185700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb22c0682f0 0x7fb22c06a7a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fb248151a50 tx=0x7fb238009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:20:07.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:07.919+0000 7fb23dffb700 1 -- 172.21.15.7:0/2365640758 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb244053300 con 0x7fb2480c2930 2023-03-03T12:20:08.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.157+0000 7fb24f3e9700 1 -- 172.21.15.7:0/2365640758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2480c6a70 con 0x7fb2480c2930 2023-03-03T12:20:08.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.158+0000 7fb23dffb700 1 -- 172.21.15.7:0/2365640758 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb244020090 con 0x7fb2480c2930 2023-03-03T12:20:08.159 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:20:08.159 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:20:08.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 -- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb22c0682f0 msgr2=0x7fb22c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:08.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb22c0682f0 0x7fb22c06a7a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fb248151a50 tx=0x7fb238009210 comp rx=0 tx=0).stop 2023-03-03T12:20:08.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 -- 172.21.15.7:0/2365640758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2480c2930 msgr2=0x7fb2481edbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:08.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2480c2930 0x7fb2481edbb0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fb244009c40 tx=0x7fb244004750 comp rx=0 tx=0).stop 2023-03-03T12:20:08.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 -- 172.21.15.7:0/2365640758 shutdown_connections 2023-03-03T12:20:08.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb248151d40 0x7fb2480c0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:08.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb248073c30 0x7fb2480ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:08.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb22c0682f0 0x7fb22c06a7a0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:08.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 --2- 172.21.15.7:0/2365640758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2480c2930 0x7fb2481edbb0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:08.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 -- 172.21.15.7:0/2365640758 >> 172.21.15.7:0/2365640758 conn(0x7fb2480a94d0 msgr2=0x7fb24814f410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:20:08.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 -- 172.21.15.7:0/2365640758 shutdown_connections 2023-03-03T12:20:08.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:08.160+0000 7fb24f3e9700 1 -- 172.21.15.7:0/2365640758 wait complete. 2023-03-03T12:20:08.163 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:20:08.173 INFO:tasks.mds_thrash.fs.[a]:waiting for 105 secs before thrashing 2023-03-03T12:20:26.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:26.730 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:42.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:20:42.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.258+0000 7f2ad1a6b700 1 Processor -- start 2023-03-03T12:20:42.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.259+0000 7f2ad1a6b700 1 -- start start 2023-03-03T12:20:42.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.259+0000 7f2ad1a6b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2acc14fda0 0x7f2acc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:42.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.259+0000 7f2ad1a6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 0x7f2acc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:42.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.259+0000 7f2ad1a6b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc0c2f30 con 0x7f2acc14f9d0 2023-03-03T12:20:42.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.259+0000 7f2ad1a6b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2acc0c31d0 con 0x7f2acc14fda0 2023-03-03T12:20:42.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.259+0000 7f2ad1a6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc1524a0 con 0x7f2acc0abe20 2023-03-03T12:20:42.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 0x7f2acc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:42.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acaffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2acc14fda0 0x7f2acc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:42.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acaffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2acc14fda0 0x7f2acc156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:35832/0 (socket says 172.21.15.7:35832) 2023-03-03T12:20:42.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acbfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 0x7f2acc1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:48132/0 (socket says 172.21.15.7:48132) 2023-03-03T12:20:42.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acaffd700 1 -- 172.21.15.7:0/3912652263 learned_addr learned my addr 172.21.15.7:0/3912652263 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:20:42.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acb7fe700 1 --1- 172.21.15.7:0/3912652263 >> v1:172.21.15.143:6789/0 conn(0x7f2acc14f9d0 0x7f2acc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:37556/0 (socket says 172.21.15.7:37556) 2023-03-03T12:20:42.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acaffd700 1 -- 172.21.15.7:0/3912652263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 msgr2=0x7f2acc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:20:42.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acaffd700 1 --2- 172.21.15.7:0/3912652263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 0x7f2acc1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:42.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acaffd700 1 -- 172.21.15.7:0/3912652263 >> v1:172.21.15.143:6789/0 conn(0x7f2acc14f9d0 legacy=0x7f2acc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:42.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acaffd700 1 -- 172.21.15.7:0/3912652263 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc0c34b0 con 0x7f2acc14fda0 2023-03-03T12:20:42.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2aca7fc700 1 -- 172.21.15.7:0/3912652263 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426085030 0 0) 0x7f2acc0c2f30 con 0x7f2acc14f9d0 2023-03-03T12:20:42.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2acaffd700 1 --2- 172.21.15.7:0/3912652263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14fda0 0x7f2acc156430 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00098f0 tx=0x7f2ac0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a9677d14ecf4465 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:20:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2aca7fc700 1 -- 172.21.15.7:0/3912652263 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0011830 con 0x7f2acc14fda0 2023-03-03T12:20:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2aca7fc700 1 -- 172.21.15.7:0/3912652263 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac0011990 con 0x7f2acc14fda0 2023-03-03T12:20:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.260+0000 7f2aca7fc700 1 -- 172.21.15.7:0/3912652263 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0011d20 con 0x7f2acc14fda0 2023-03-03T12:20:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.261+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/3912652263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14fda0 msgr2=0x7f2acc156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.261+0000 7f2ad1a6b700 1 --2- 172.21.15.7:0/3912652263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14fda0 0x7f2acc156430 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00098f0 tx=0x7f2ac0009c00 comp rx=0 tx=0).stop 2023-03-03T12:20:42.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.261+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/3912652263 shutdown_connections 2023-03-03T12:20:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.261+0000 7f2ad1a6b700 1 --2- 172.21.15.7:0/3912652263 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 0x7f2acc1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.261+0000 7f2ad1a6b700 1 --2- 172.21.15.7:0/3912652263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14fda0 0x7f2acc156430 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.261+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/3912652263 >> 172.21.15.7:0/3912652263 conn(0x7f2acc0c8900 msgr2=0x7f2acc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:20:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.261+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/3912652263 shutdown_connections 2023-03-03T12:20:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.261+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/3912652263 wait complete. 2023-03-03T12:20:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.262+0000 7f2ad1a6b700 1 Processor -- start 2023-03-03T12:20:42.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.262+0000 7f2ad1a6b700 1 -- start start 2023-03-03T12:20:42.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2ad1a6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 0x7f2acc149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:42.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2ad1a6b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14f9d0 0x7f2acc14ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:42.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2ad1a6b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14fda0 0x7f2acc14c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:42.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2ad1a6b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc154290 con 0x7f2acc14f9d0 2023-03-03T12:20:42.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2ad1a6b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc14d540 con 0x7f2acc14fda0 2023-03-03T12:20:42.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2ad1a6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc14c390 con 0x7f2acc0abe20 2023-03-03T12:20:42.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2acaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14f9d0 0x7f2acc14ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:42.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2acaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14f9d0 0x7f2acc14ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40064/0 (socket says 172.21.15.7:40064) 2023-03-03T12:20:42.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2acaffd700 1 -- 172.21.15.7:0/1619153317 learned_addr learned my addr 172.21.15.7:0/1619153317 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:20:42.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2acb7fe700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 0x7f2acc149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:42.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.263+0000 7f2acbfff700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14fda0 0x7f2acc14c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:42.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2acaffd700 1 -- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 msgr2=0x7f2acc149070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:42.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2acaffd700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 0x7f2acc149070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:42.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2acaffd700 1 -- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14fda0 msgr2=0x7f2acc14c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:42.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2acaffd700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14fda0 0x7f2acc14c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:42.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2acaffd700 1 -- 172.21.15.7:0/1619153317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac0009550 con 0x7f2acc14f9d0 2023-03-03T12:20:42.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2acaffd700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14f9d0 0x7f2acc14ac00 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0013000 tx=0x7f2ac0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:20:42.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2ab3fff700 1 -- 172.21.15.7:0/1619153317 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0004550 con 0x7f2acc14f9d0 2023-03-03T12:20:42.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2ab3fff700 1 -- 172.21.15.7:0/1619153317 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac00036a0 con 0x7f2acc14f9d0 2023-03-03T12:20:42.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/1619153317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc0ac350 con 0x7f2acc14f9d0 2023-03-03T12:20:42.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2ab3fff700 1 -- 172.21.15.7:0/1619153317 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0003a30 con 0x7f2acc14f9d0 2023-03-03T12:20:42.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.264+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/1619153317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc0ab630 con 0x7f2acc14f9d0 2023-03-03T12:20:42.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.265+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/1619153317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc071fa0 con 0x7f2acc14f9d0 2023-03-03T12:20:42.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.266+0000 7f2ab3fff700 1 -- 172.21.15.7:0/1619153317 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ac000f050 con 0x7f2acc14f9d0 2023-03-03T12:20:42.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.266+0000 7f2ab3fff700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab80682f0 0x7f2ab806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:20:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.266+0000 7f2ab3fff700 1 -- 172.21.15.7:0/1619153317 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2ac0087740 con 0x7f2acc14f9d0 2023-03-03T12:20:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.266+0000 7f2acb7fe700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab80682f0 0x7f2ab806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:20:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.267+0000 7f2acb7fe700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab80682f0 0x7f2ab806a7a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f2abc007fd0 tx=0x7f2abc007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:20:42.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.270+0000 7f2ab3fff700 1 -- 172.21.15.7:0/1619153317 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ac0053ae0 con 0x7f2acc14f9d0 2023-03-03T12:20:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.509+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/1619153317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2acc1543d0 con 0x7f2acc14f9d0 2023-03-03T12:20:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.509+0000 7f2ab3fff700 1 -- 172.21.15.7:0/1619153317 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2ac0057100 con 0x7f2acc14f9d0 2023-03-03T12:20:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:20:42.511 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:20:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab80682f0 msgr2=0x7f2ab806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab80682f0 0x7f2ab806a7a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f2abc007fd0 tx=0x7f2abc007de0 comp rx=0 tx=0).stop 2023-03-03T12:20:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/1619153317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14f9d0 msgr2=0x7f2acc14ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:20:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14f9d0 0x7f2acc14ac00 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0013000 tx=0x7f2ac0004750 comp rx=0 tx=0).stop 2023-03-03T12:20:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/1619153317 shutdown_connections 2023-03-03T12:20:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2acc14fda0 0x7f2acc14c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2acc0abe20 0x7f2acc149070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ab80682f0 0x7f2ab806a7a0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 --2- 172.21.15.7:0/1619153317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2acc14f9d0 0x7f2acc14ac00 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:20:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/1619153317 >> 172.21.15.7:0/1619153317 conn(0x7f2acc0c8900 msgr2=0x7f2acc0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:20:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/1619153317 shutdown_connections 2023-03-03T12:20:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:20:42.511+0000 7f2ad1a6b700 1 -- 172.21.15.7:0/1619153317 wait complete. 2023-03-03T12:20:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:20:42.525 INFO:tasks.mds_thrash.fs.[b]:waiting for 78 secs before thrashing 2023-03-03T12:20:56.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:20:56.854 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:26.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:26.935 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:53.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:21:53.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.311+0000 7fcd9cf1a700 1 Processor -- start 2023-03-03T12:21:53.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.311+0000 7fcd9cf1a700 1 -- start start 2023-03-03T12:21:53.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.312+0000 7fcd9cf1a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcd9814f9d0 0x7fcd98156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:53.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.312+0000 7fcd9cf1a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcd980abe20 0x7fcd981529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:53.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.312+0000 7fcd9cf1a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd980c2f30 con 0x7fcd9814fda0 2023-03-03T12:21:53.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.312+0000 7fcd9cf1a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd980c31d0 con 0x7fcd980abe20 2023-03-03T12:21:53.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.312+0000 7fcd9cf1a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd981524a0 con 0x7fcd9814f9d0 2023-03-03T12:21:53.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.312+0000 7fcd96d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcd980abe20 0x7fcd981529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:53.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd96d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcd980abe20 0x7fcd981529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40118/0 (socket says 172.21.15.7:40118) 2023-03-03T12:21:53.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd96d9d700 1 -- 172.21.15.7:0/3338343233 learned_addr learned my addr 172.21.15.7:0/3338343233 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:21:53.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd9659c700 1 --2- 172.21.15.7:0/3338343233 >> v2:172.21.15.143:3300/0 conn(0x7fcd9814f9d0 0x7fcd98156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:53.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd96d9d700 1 -- 172.21.15.7:0/3338343233 >> v2:172.21.15.143:3300/0 conn(0x7fcd9814f9d0 msgr2=0x7fcd98156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:21:53.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd96d9d700 1 --2- 172.21.15.7:0/3338343233 >> v2:172.21.15.143:3300/0 conn(0x7fcd9814f9d0 0x7fcd98156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:53.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd96d9d700 1 -- 172.21.15.7:0/3338343233 >> v1:172.21.15.143:6789/0 conn(0x7fcd9814fda0 legacy=0x7fcd98071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:21:53.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd96d9d700 1 -- 172.21.15.7:0/3338343233 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd980c34b0 con 0x7fcd980abe20 2023-03-03T12:21:53.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd9659c700 1 --2- 172.21.15.7:0/3338343233 >> v2:172.21.15.143:3300/0 conn(0x7fcd9814f9d0 0x7fcd98156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:21:53.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd96d9d700 1 --2- 172.21.15.7:0/3338343233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 0x7fcd981529c0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fcd980710a0 tx=0x7fcd9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee03faa8566e747d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:21:53.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd9559a700 1 -- 172.21.15.7:0/3338343233 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9000e070 con 0x7fcd980abe20 2023-03-03T12:21:53.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd9559a700 1 -- 172.21.15.7:0/3338343233 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9000b9c0 con 0x7fcd980abe20 2023-03-03T12:21:53.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.313+0000 7fcd9559a700 1 -- 172.21.15.7:0/3338343233 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9000bd70 con 0x7fcd980abe20 2023-03-03T12:21:53.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.314+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/3338343233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 msgr2=0x7fcd981529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:53.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.314+0000 7fcd9cf1a700 1 --2- 172.21.15.7:0/3338343233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 0x7fcd981529c0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fcd980710a0 tx=0x7fcd9000b330 comp rx=0 tx=0).stop 2023-03-03T12:21:53.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.314+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/3338343233 shutdown_connections 2023-03-03T12:21:53.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.314+0000 7fcd9cf1a700 1 --2- 172.21.15.7:0/3338343233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 0x7fcd981529c0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:53.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.314+0000 7fcd9cf1a700 1 --2- 172.21.15.7:0/3338343233 >> v2:172.21.15.143:3300/0 conn(0x7fcd9814f9d0 0x7fcd98156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:53.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.314+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/3338343233 >> 172.21.15.7:0/3338343233 conn(0x7fcd980c8900 msgr2=0x7fcd980c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:21:53.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.314+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/3338343233 shutdown_connections 2023-03-03T12:21:53.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.314+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/3338343233 wait complete. 2023-03-03T12:21:53.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.315+0000 7fcd9cf1a700 1 Processor -- start 2023-03-03T12:21:53.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.315+0000 7fcd9cf1a700 1 -- start start 2023-03-03T12:21:53.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9cf1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 0x7fcd98145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:53.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9cf1a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd9814f9d0 0x7fcd980c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:53.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9cf1a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd9814fda0 0x7fcd981e9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:53.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9cf1a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd98145c20 con 0x7fcd980abe20 2023-03-03T12:21:53.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9cf1a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd981ecdd0 con 0x7fcd9814f9d0 2023-03-03T12:21:53.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9cf1a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd981ed0b0 con 0x7fcd9814fda0 2023-03-03T12:21:53.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 0x7fcd98145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:53.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 0x7fcd98145590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40120/0 (socket says 172.21.15.7:40120) 2023-03-03T12:21:53.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9659c700 1 -- 172.21.15.7:0/1888361024 learned_addr learned my addr 172.21.15.7:0/1888361024 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:21:53.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd95d9b700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd9814f9d0 0x7fcd980c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:53.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd96d9d700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd9814fda0 0x7fcd981e9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:53.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9659c700 1 -- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd9814fda0 msgr2=0x7fcd981e9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:53.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9659c700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd9814fda0 0x7fcd981e9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:53.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9659c700 1 -- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd9814f9d0 msgr2=0x7fcd980c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:53.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9659c700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd9814f9d0 0x7fcd980c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:53.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.316+0000 7fcd9659c700 1 -- 172.21.15.7:0/1888361024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd90009b30 con 0x7fcd980abe20 2023-03-03T12:21:53.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.317+0000 7fcd9659c700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 0x7fcd98145590 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800c750 tx=0x7fcd8800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:21:53.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.317+0000 7fcd8effd700 1 -- 172.21.15.7:0/1888361024 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88011800 con 0x7fcd980abe20 2023-03-03T12:21:53.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.317+0000 7fcd8effd700 1 -- 172.21.15.7:0/1888361024 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd88011960 con 0x7fcd980abe20 2023-03-03T12:21:53.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.317+0000 7fcd8effd700 1 -- 172.21.15.7:0/1888361024 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88011cd0 con 0x7fcd980abe20 2023-03-03T12:21:53.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.317+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/1888361024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd980ac350 con 0x7fcd980abe20 2023-03-03T12:21:53.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.317+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/1888361024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd980ab630 con 0x7fcd980abe20 2023-03-03T12:21:53.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.318+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/1888361024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd98071fa0 con 0x7fcd980abe20 2023-03-03T12:21:53.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.318+0000 7fcd8effd700 1 -- 172.21.15.7:0/1888361024 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcd88020070 con 0x7fcd980abe20 2023-03-03T12:21:53.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.319+0000 7fcd8effd700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd780682f0 0x7fcd7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:21:53.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.319+0000 7fcd8effd700 1 -- 172.21.15.7:0/1888361024 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcd88086090 con 0x7fcd980abe20 2023-03-03T12:21:53.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.319+0000 7fcd95d9b700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd780682f0 0x7fcd7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:21:53.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.320+0000 7fcd95d9b700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd780682f0 0x7fcd7806a7a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcd9806a480 tx=0x7fcd80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:21:53.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.323+0000 7fcd8effd700 1 -- 172.21.15.7:0/1888361024 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcd88052340 con 0x7fcd980abe20 2023-03-03T12:21:53.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.563+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/1888361024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd981ecc30 con 0x7fcd980abe20 2023-03-03T12:21:53.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.563+0000 7fcd8effd700 1 -- 172.21.15.7:0/1888361024 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcd981ecc30 con 0x7fcd980abe20 2023-03-03T12:21:53.565 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:21:53.565 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:21:53.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.565+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd780682f0 msgr2=0x7fcd7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:53.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.565+0000 7fcd9cf1a700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd780682f0 0x7fcd7806a7a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcd9806a480 tx=0x7fcd80009040 comp rx=0 tx=0).stop 2023-03-03T12:21:53.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.565+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/1888361024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 msgr2=0x7fcd98145590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:21:53.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.565+0000 7fcd9cf1a700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 0x7fcd98145590 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800c750 tx=0x7fcd8800cb10 comp rx=0 tx=0).stop 2023-03-03T12:21:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.565+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/1888361024 shutdown_connections 2023-03-03T12:21:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.565+0000 7fcd9cf1a700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd9814fda0 0x7fcd981e9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.565+0000 7fcd9cf1a700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd780682f0 0x7fcd7806a7a0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.565+0000 7fcd9cf1a700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd980abe20 0x7fcd98145590 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.565+0000 7fcd9cf1a700 1 --2- 172.21.15.7:0/1888361024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd9814f9d0 0x7fcd980c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:21:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.565+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/1888361024 >> 172.21.15.7:0/1888361024 conn(0x7fcd980c8900 msgr2=0x7fcd980c6550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:21:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.566+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/1888361024 shutdown_connections 2023-03-03T12:21:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:21:53.566+0000 7fcd9cf1a700 1 -- 172.21.15.7:0/1888361024 wait complete. 2023-03-03T12:21:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:21:53.579 INFO:tasks.mds_thrash.fs.[a]:waiting for 107 secs before thrashing 2023-03-03T12:21:57.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:21:57.011 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:00.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:22:00.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.664+0000 7f1f931ed700 1 Processor -- start 2023-03-03T12:22:00.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.664+0000 7f1f931ed700 1 -- start start 2023-03-03T12:22:00.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f931ed700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f8c06d4b0 0x7f1f8c06d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:00.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f931ed700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1f8c0c1da0 0x7f1f8c139d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:00.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f931ed700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c0c7b90 con 0x7f1f8c0c8020 2023-03-03T12:22:00.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f931ed700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8c0c7330 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f931ed700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8c154420 con 0x7f1f8c0c1da0 2023-03-03T12:22:00.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f90f89700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f8c06d4b0 0x7f1f8c06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:00.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f90f89700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f8c06d4b0 0x7f1f8c06d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40130/0 (socket says 172.21.15.7:40130) 2023-03-03T12:22:00.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f90f89700 1 -- 172.21.15.7:0/569590902 learned_addr learned my addr 172.21.15.7:0/569590902 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:22:00.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f9178a700 1 --2- 172.21.15.7:0/569590902 >> v2:172.21.15.143:3300/0 conn(0x7f1f8c0c1da0 0x7f1f8c139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:00.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f90f89700 1 -- 172.21.15.7:0/569590902 >> v2:172.21.15.143:3300/0 conn(0x7f1f8c0c1da0 msgr2=0x7f1f8c139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:22:00.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f90f89700 1 --2- 172.21.15.7:0/569590902 >> v2:172.21.15.143:3300/0 conn(0x7f1f8c0c1da0 0x7f1f8c139d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:00.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f90f89700 1 -- 172.21.15.7:0/569590902 >> v1:172.21.15.143:6789/0 conn(0x7f1f8c0c8020 legacy=0x7f1f8c0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:00.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f90f89700 1 -- 172.21.15.7:0/569590902 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c154640 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f90f89700 1 --2- 172.21.15.7:0/569590902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 0x7f1f8c06d880 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0c18b0 tx=0x7f1f80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9617f83da6f79fc7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:22:00.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.665+0000 7f1f8b7fe700 1 -- 172.21.15.7:0/569590902 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8000b870 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.666+0000 7f1f8b7fe700 1 -- 172.21.15.7:0/569590902 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8000b9d0 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.666+0000 7f1f8b7fe700 1 -- 172.21.15.7:0/569590902 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8000bd60 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.666+0000 7f1f931ed700 1 -- 172.21.15.7:0/569590902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 msgr2=0x7f1f8c06d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:00.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.666+0000 7f1f931ed700 1 --2- 172.21.15.7:0/569590902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 0x7f1f8c06d880 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0c18b0 tx=0x7f1f80009bd0 comp rx=0 tx=0).stop 2023-03-03T12:22:00.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.666+0000 7f1f931ed700 1 -- 172.21.15.7:0/569590902 shutdown_connections 2023-03-03T12:22:00.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.666+0000 7f1f931ed700 1 --2- 172.21.15.7:0/569590902 >> v2:172.21.15.143:3300/0 conn(0x7f1f8c0c1da0 0x7f1f8c139d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:00.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.666+0000 7f1f931ed700 1 --2- 172.21.15.7:0/569590902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 0x7f1f8c06d880 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:00.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.666+0000 7f1f931ed700 1 -- 172.21.15.7:0/569590902 >> 172.21.15.7:0/569590902 conn(0x7f1f8c06c110 msgr2=0x7f1f8c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:22:00.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.666+0000 7f1f931ed700 1 -- 172.21.15.7:0/569590902 shutdown_connections 2023-03-03T12:22:00.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.666+0000 7f1f931ed700 1 -- 172.21.15.7:0/569590902 wait complete. 2023-03-03T12:22:00.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.667+0000 7f1f931ed700 1 Processor -- start 2023-03-03T12:22:00.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.667+0000 7f1f931ed700 1 -- start start 2023-03-03T12:22:00.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f931ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 0x7f1f8c1df8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:00.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f931ed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8c0c1da0 0x7f1f8c0abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:00.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f931ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f8c0c8020 0x7f1f8c1ecd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:00.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f931ed700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c0ad290 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f931ed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c0accd0 con 0x7f1f8c0c1da0 2023-03-03T12:22:00.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f931ed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c1dace0 con 0x7f1f8c0c8020 2023-03-03T12:22:00.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f90f89700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 0x7f1f8c1df8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f90f89700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 0x7f1f8c1df8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40136/0 (socket says 172.21.15.7:40136) 2023-03-03T12:22:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f90f89700 1 -- 172.21.15.7:0/530520780 learned_addr learned my addr 172.21.15.7:0/530520780 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:22:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f8bfff700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8c0c1da0 0x7f1f8c0abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.668+0000 7f1f9178a700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f8c0c8020 0x7f1f8c1ecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f90f89700 1 -- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f8c0c8020 msgr2=0x7f1f8c1ecd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f90f89700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f8c0c8020 0x7f1f8c1ecd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:00.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f90f89700 1 -- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8c0c1da0 msgr2=0x7f1f8c0abbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f90f89700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8c0c1da0 0x7f1f8c0abbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f90f89700 1 -- 172.21.15.7:0/530520780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f80009550 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f8bfff700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8c0c1da0 0x7f1f8c0abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:22:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f90f89700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 0x7f1f8c1df8f0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f1f80000c00 tx=0x7f1f80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:22:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f897fa700 1 -- 172.21.15.7:0/530520780 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80004520 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f897fa700 1 -- 172.21.15.7:0/530520780 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8001e070 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f931ed700 1 -- 172.21.15.7:0/530520780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8c0c7820 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f897fa700 1 -- 172.21.15.7:0/530520780 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f800038a0 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.669+0000 7f1f931ed700 1 -- 172.21.15.7:0/530520780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8c0c7540 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.670+0000 7f1f931ed700 1 -- 172.21.15.7:0/530520780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8c06ddc0 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.671+0000 7f1f897fa700 1 -- 172.21.15.7:0/530520780 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f8000f050 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.671+0000 7f1f897fa700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f700682f0 0x7f1f7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.671+0000 7f1f897fa700 1 -- 172.21.15.7:0/530520780 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1f80020070 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.671+0000 7f1f8bfff700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f700682f0 0x7f1f7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.672+0000 7f1f8bfff700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f700682f0 0x7f1f7006a7a0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c155170 tx=0x7f1f78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:22:00.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.676+0000 7f1f897fa700 1 -- 172.21.15.7:0/530520780 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f80053990 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.914+0000 7f1f931ed700 1 -- 172.21.15.7:0/530520780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f8c074bb0 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.915+0000 7f1f897fa700 1 -- 172.21.15.7:0/530520780 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1f80056fb0 con 0x7f1f8c06d4b0 2023-03-03T12:22:00.917 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:22:00.917 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:22:00.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 -- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f700682f0 msgr2=0x7f1f7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:00.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f700682f0 0x7f1f7006a7a0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c155170 tx=0x7f1f78009040 comp rx=0 tx=0).stop 2023-03-03T12:22:00.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 -- 172.21.15.7:0/530520780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 msgr2=0x7f1f8c1df8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:00.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 0x7f1f8c1df8f0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f1f80000c00 tx=0x7f1f80004750 comp rx=0 tx=0).stop 2023-03-03T12:22:00.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 -- 172.21.15.7:0/530520780 shutdown_connections 2023-03-03T12:22:00.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f8c0c8020 0x7f1f8c1ecd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:00.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f700682f0 0x7f1f7006a7a0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:00.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f8c06d4b0 0x7f1f8c1df8f0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:00.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 --2- 172.21.15.7:0/530520780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f8c0c1da0 0x7f1f8c0abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:00.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 -- 172.21.15.7:0/530520780 >> 172.21.15.7:0/530520780 conn(0x7f1f8c06c110 msgr2=0x7f1f8c1549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:22:00.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 -- 172.21.15.7:0/530520780 shutdown_connections 2023-03-03T12:22:00.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:00.917+0000 7f1f931ed700 1 -- 172.21.15.7:0/530520780 wait complete. 2023-03-03T12:22:00.920 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:22:00.930 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2023-03-03T12:22:27.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:27.152 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:34.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:22:35.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.070+0000 7f7acec53700 1 Processor -- start 2023-03-03T12:22:35.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.070+0000 7f7acec53700 1 -- start start 2023-03-03T12:22:35.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7acec53700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ac814b020 0x7f7ac80717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7acec53700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac80728b0 con 0x7f7ac80aad20 2023-03-03T12:22:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7acec53700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac8155070 con 0x7f7ac8071d20 2023-03-03T12:22:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7acec53700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac806f5b0 con 0x7f7ac814b020 2023-03-03T12:22:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7acc9ef700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ac814b020 0x7f7ac80717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7ac7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7ac80aad20 0x7f7ac8157ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34144/0 (socket says 172.21.15.7:34144) 2023-03-03T12:22:35.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7ac7fff700 1 -- 172.21.15.7:0/1230342566 learned_addr learned my addr 172.21.15.7:0/1230342566 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:22:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7acc9ef700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ac814b020 0x7f7ac80717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40162/0 (socket says 172.21.15.7:40162) 2023-03-03T12:22:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7acc9ef700 1 -- 172.21.15.7:0/1230342566 >> v1:172.21.15.143:6789/0 conn(0x7f7ac8071d20 legacy=0x7f7ac81583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:22:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7acc9ef700 1 -- 172.21.15.7:0/1230342566 >> v1:172.21.15.7:6789/0 conn(0x7f7ac80aad20 legacy=0x7f7ac8157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.071+0000 7f7ac77fe700 1 -- 172.21.15.7:0/1230342566 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911187263 0 0) 0x7f7ac80728b0 con 0x7f7ac80aad20 2023-03-03T12:22:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.072+0000 7f7acc9ef700 1 -- 172.21.15.7:0/1230342566 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac806f890 con 0x7f7ac814b020 2023-03-03T12:22:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.072+0000 7f7acc9ef700 1 --2- 172.21.15.7:0/1230342566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac814b020 0x7f7ac80717e0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009840 tx=0x7f7abc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6547873ee1616c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:22:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.072+0000 7f7ac77fe700 1 -- 172.21.15.7:0/1230342566 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc004230 con 0x7f7ac814b020 2023-03-03T12:22:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.072+0000 7f7ac77fe700 1 -- 172.21.15.7:0/1230342566 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc004390 con 0x7f7ac814b020 2023-03-03T12:22:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.072+0000 7f7ac77fe700 1 -- 172.21.15.7:0/1230342566 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0036a0 con 0x7f7ac814b020 2023-03-03T12:22:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.072+0000 7f7acec53700 1 -- 172.21.15.7:0/1230342566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac814b020 msgr2=0x7f7ac80717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.072+0000 7f7acec53700 1 --2- 172.21.15.7:0/1230342566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac814b020 0x7f7ac80717e0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009840 tx=0x7f7abc009b50 comp rx=0 tx=0).stop 2023-03-03T12:22:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.072+0000 7f7acec53700 1 -- 172.21.15.7:0/1230342566 shutdown_connections 2023-03-03T12:22:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.072+0000 7f7acec53700 1 --2- 172.21.15.7:0/1230342566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac814b020 0x7f7ac80717e0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.072+0000 7f7acec53700 1 -- 172.21.15.7:0/1230342566 >> 172.21.15.7:0/1230342566 conn(0x7f7ac80c8900 msgr2=0x7f7ac80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:22:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.073+0000 7f7acec53700 1 -- 172.21.15.7:0/1230342566 shutdown_connections 2023-03-03T12:22:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.073+0000 7f7acec53700 1 -- 172.21.15.7:0/1230342566 wait complete. 2023-03-03T12:22:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.074+0000 7f7acec53700 1 Processor -- start 2023-03-03T12:22:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.074+0000 7f7acec53700 1 -- start start 2023-03-03T12:22:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.074+0000 7f7acec53700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac8071d20 0x7f7ac80c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.074+0000 7f7acec53700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ac80aad20 0x7f7ac81e9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acec53700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ac814b020 0x7f7ac81ea090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acec53700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac81e0ed0 con 0x7f7ac8071d20 2023-03-03T12:22:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acec53700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac81e1930 con 0x7f7ac814b020 2023-03-03T12:22:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acec53700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac81ea5d0 con 0x7f7ac80aad20 2023-03-03T12:22:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acc9ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac8071d20 0x7f7ac80c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acc9ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac8071d20 0x7f7ac80c7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40168/0 (socket says 172.21.15.7:40168) 2023-03-03T12:22:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acc9ef700 1 -- 172.21.15.7:0/3181461152 learned_addr learned my addr 172.21.15.7:0/3181461152 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:22:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acc9ef700 1 -- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ac80aad20 msgr2=0x7f7ac81e9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acc9ef700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ac80aad20 0x7f7ac81e9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acc9ef700 1 -- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ac814b020 msgr2=0x7f7ac81ea090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acd1f0700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ac814b020 0x7f7ac81ea090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acc9ef700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ac814b020 0x7f7ac81ea090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acc9ef700 1 -- 172.21.15.7:0/3181461152 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc0094a0 con 0x7f7ac8071d20 2023-03-03T12:22:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acc9ef700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac8071d20 0x7f7ac80c7700 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0090b0 tx=0x7f7abc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:22:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7ac57fa700 1 -- 172.21.15.7:0/3181461152 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc011830 con 0x7f7ac8071d20 2023-03-03T12:22:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7ac57fa700 1 -- 172.21.15.7:0/3181461152 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc011990 con 0x7f7ac8071d20 2023-03-03T12:22:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acec53700 1 -- 172.21.15.7:0/3181461152 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac8155350 con 0x7f7ac8071d20 2023-03-03T12:22:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7ac57fa700 1 -- 172.21.15.7:0/3181461152 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc011d40 con 0x7f7ac8071d20 2023-03-03T12:22:35.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.075+0000 7f7acec53700 1 -- 172.21.15.7:0/3181461152 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac806fb70 con 0x7f7ac8071d20 2023-03-03T12:22:35.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.076+0000 7f7acec53700 1 -- 172.21.15.7:0/3181461152 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac81e99a0 con 0x7f7ac8071d20 2023-03-03T12:22:35.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.077+0000 7f7ac57fa700 1 -- 172.21.15.7:0/3181461152 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7abc017020 con 0x7f7ac8071d20 2023-03-03T12:22:35.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.077+0000 7f7ac57fa700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7aac0682f0 0x7f7aac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:22:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.077+0000 7f7ac57fa700 1 -- 172.21.15.7:0/3181461152 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7abc087a60 con 0x7f7ac8071d20 2023-03-03T12:22:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.078+0000 7f7ac7fff700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7aac0682f0 0x7f7aac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:22:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.078+0000 7f7ac7fff700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7aac0682f0 0x7f7aac06a7a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7ab40011e0 tx=0x7f7ab40079b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:22:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.082+0000 7f7ac57fa700 1 -- 172.21.15.7:0/3181461152 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7abc053d10 con 0x7f7ac8071d20 2023-03-03T12:22:35.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.317+0000 7f7acec53700 1 -- 172.21.15.7:0/3181461152 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ac8074bb0 con 0x7f7ac8071d20 2023-03-03T12:22:35.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.317+0000 7f7ac57fa700 1 -- 172.21.15.7:0/3181461152 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7abc057330 con 0x7f7ac8071d20 2023-03-03T12:22:35.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:22:35.319 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:22:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 -- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7aac0682f0 msgr2=0x7f7aac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7aac0682f0 0x7f7aac06a7a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7ab40011e0 tx=0x7f7ab40079b0 comp rx=0 tx=0).stop 2023-03-03T12:22:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 -- 172.21.15.7:0/3181461152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac8071d20 msgr2=0x7f7ac80c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:22:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac8071d20 0x7f7ac80c7700 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0090b0 tx=0x7f7abc0047c0 comp rx=0 tx=0).stop 2023-03-03T12:22:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 -- 172.21.15.7:0/3181461152 shutdown_connections 2023-03-03T12:22:35.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ac814b020 0x7f7ac81ea090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:35.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7aac0682f0 0x7f7aac06a7a0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:35.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ac8071d20 0x7f7ac80c7700 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:35.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 --2- 172.21.15.7:0/3181461152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ac80aad20 0x7f7ac81e9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:22:35.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 -- 172.21.15.7:0/3181461152 >> 172.21.15.7:0/3181461152 conn(0x7f7ac80c8900 msgr2=0x7f7ac8156680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:22:35.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 -- 172.21.15.7:0/3181461152 shutdown_connections 2023-03-03T12:22:35.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:22:35.319+0000 7f7acec53700 1 -- 172.21.15.7:0/3181461152 wait complete. 2023-03-03T12:22:35.323 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:22:35.333 INFO:tasks.mds_thrash.fs.[b]:waiting for 94 secs before thrashing 2023-03-03T12:22:57.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:22:57.274 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:27.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:27.329 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:40.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:23:40.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.718+0000 7fd73d0fb700 1 Processor -- start 2023-03-03T12:23:40.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.718+0000 7fd73d0fb700 1 -- start start 2023-03-03T12:23:40.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.719+0000 7fd73d0fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7380c00e0 0x7fd7380c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:23:40.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.719+0000 7fd73d0fb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7380c7310 con 0x7fd7380caa30 2023-03-03T12:23:40.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.719+0000 7fd73d0fb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7380bfd10 con 0x7fd738154ff0 2023-03-03T12:23:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.719+0000 7fd73d0fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380bfa30 con 0x7fd7380c00e0 2023-03-03T12:23:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.719+0000 7fd736d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd7380caa30 0x7fd7380c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34222/0 (socket says 172.21.15.7:34222) 2023-03-03T12:23:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.719+0000 7fd736d9d700 1 -- 172.21.15.7:0/457992404 learned_addr learned my addr 172.21.15.7:0/457992404 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:23:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.719+0000 7fd73659c700 1 --2- 172.21.15.7:0/457992404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7380c00e0 0x7fd7380c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:23:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.719+0000 7fd735d9b700 1 -- 172.21.15.7:0/457992404 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708726725 0 0) 0x7fd7380c7310 con 0x7fd7380caa30 2023-03-03T12:23:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.719+0000 7fd735d9b700 1 -- 172.21.15.7:0/457992404 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7200035b0 con 0x7fd7380caa30 2023-03-03T12:23:40.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.719+0000 7fd735d9b700 1 -- 172.21.15.7:0/457992404 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027354511 0 0) 0x7fd7380bfd10 con 0x7fd738154ff0 2023-03-03T12:23:40.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.720+0000 7fd735d9b700 1 -- 172.21.15.7:0/457992404 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7380c7310 con 0x7fd738154ff0 2023-03-03T12:23:40.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.720+0000 7fd73659c700 1 -- 172.21.15.7:0/457992404 >> v1:172.21.15.143:6789/0 conn(0x7fd738154ff0 legacy=0x7fd73815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:23:40.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.720+0000 7fd73659c700 1 -- 172.21.15.7:0/457992404 >> v1:172.21.15.7:6789/0 conn(0x7fd7380caa30 legacy=0x7fd7380c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:23:40.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.720+0000 7fd735d9b700 1 -- 172.21.15.7:0/457992404 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2691068416 0 0) 0x7fd7200035b0 con 0x7fd7380caa30 2023-03-03T12:23:40.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.720+0000 7fd73659c700 1 -- 172.21.15.7:0/457992404 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd738154510 con 0x7fd7380c00e0 2023-03-03T12:23:40.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.720+0000 7fd73659c700 1 --2- 172.21.15.7:0/457992404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7380c00e0 0x7fd7380c5c60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd728009850 tx=0x7fd728009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a233efdf799df4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:23:40.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.720+0000 7fd735d9b700 1 -- 172.21.15.7:0/457992404 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c001670 con 0x7fd7380c00e0 2023-03-03T12:23:40.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.720+0000 7fd735d9b700 1 -- 172.21.15.7:0/457992404 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd728005c30 con 0x7fd7380c00e0 2023-03-03T12:23:40.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.720+0000 7fd735d9b700 1 -- 172.21.15.7:0/457992404 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728004030 con 0x7fd7380c00e0 2023-03-03T12:23:40.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.721+0000 7fd73d0fb700 1 -- 172.21.15.7:0/457992404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7380c00e0 msgr2=0x7fd7380c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:23:40.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.721+0000 7fd73d0fb700 1 --2- 172.21.15.7:0/457992404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7380c00e0 0x7fd7380c5c60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd728009850 tx=0x7fd728009b60 comp rx=0 tx=0).stop 2023-03-03T12:23:40.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.721+0000 7fd73d0fb700 1 -- 172.21.15.7:0/457992404 shutdown_connections 2023-03-03T12:23:40.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.721+0000 7fd73d0fb700 1 --2- 172.21.15.7:0/457992404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7380c00e0 0x7fd7380c5c60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:23:40.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.721+0000 7fd73d0fb700 1 -- 172.21.15.7:0/457992404 >> 172.21.15.7:0/457992404 conn(0x7fd738066e40 msgr2=0x7fd738071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:23:40.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.721+0000 7fd73d0fb700 1 -- 172.21.15.7:0/457992404 shutdown_connections 2023-03-03T12:23:40.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.721+0000 7fd73d0fb700 1 -- 172.21.15.7:0/457992404 wait complete. 2023-03-03T12:23:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.722+0000 7fd73d0fb700 1 Processor -- start 2023-03-03T12:23:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.722+0000 7fd73d0fb700 1 -- start start 2023-03-03T12:23:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73d0fb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7380c00e0 0x7fd738149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:23:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73d0fb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7380caa30 0x7fd73814a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:23:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73d0fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd738154ff0 0x7fd738145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:23:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73d0fb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380c2330 con 0x7fd7380caa30 2023-03-03T12:23:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73d0fb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380c4620 con 0x7fd7380c00e0 2023-03-03T12:23:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73d0fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd738156140 con 0x7fd738154ff0 2023-03-03T12:23:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7380caa30 0x7fd73814a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:23:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7380caa30 0x7fd73814a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40250/0 (socket says 172.21.15.7:40250) 2023-03-03T12:23:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73659c700 1 -- 172.21.15.7:0/2500591457 learned_addr learned my addr 172.21.15.7:0/2500591457 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:23:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd736d9d700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7380c00e0 0x7fd738149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:23:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73759e700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd738154ff0 0x7fd738145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:23:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73659c700 1 -- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd738154ff0 msgr2=0x7fd738145a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:23:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73659c700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd738154ff0 0x7fd738145a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:23:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73659c700 1 -- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7380c00e0 msgr2=0x7fd738149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:23:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73659c700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7380c00e0 0x7fd738149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:23:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73659c700 1 -- 172.21.15.7:0/2500591457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd728009430 con 0x7fd7380caa30 2023-03-03T12:23:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd736d9d700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7380c00e0 0x7fd738149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:23:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.723+0000 7fd73759e700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd738154ff0 0x7fd738145a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:23:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.724+0000 7fd73659c700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7380caa30 0x7fd73814a1a0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fd728004000 tx=0x7fd728004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:23:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.724+0000 7fd71f7fe700 1 -- 172.21.15.7:0/2500591457 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7280039b0 con 0x7fd7380caa30 2023-03-03T12:23:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.724+0000 7fd71f7fe700 1 -- 172.21.15.7:0/2500591457 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd728003b10 con 0x7fd7380caa30 2023-03-03T12:23:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.724+0000 7fd73d0fb700 1 -- 172.21.15.7:0/2500591457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd738154ad0 con 0x7fd7380caa30 2023-03-03T12:23:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.724+0000 7fd71f7fe700 1 -- 172.21.15.7:0/2500591457 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728011420 con 0x7fd7380caa30 2023-03-03T12:23:40.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.724+0000 7fd73d0fb700 1 -- 172.21.15.7:0/2500591457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7381547f0 con 0x7fd7380caa30 2023-03-03T12:23:40.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.725+0000 7fd73d0fb700 1 -- 172.21.15.7:0/2500591457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7380bf650 con 0x7fd7380caa30 2023-03-03T12:23:40.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.725+0000 7fd71f7fe700 1 -- 172.21.15.7:0/2500591457 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd72800f050 con 0x7fd7380caa30 2023-03-03T12:23:40.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.726+0000 7fd71f7fe700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7240682f0 0x7fd72406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:23:40.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.726+0000 7fd71f7fe700 1 -- 172.21.15.7:0/2500591457 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd7280876b0 con 0x7fd7380caa30 2023-03-03T12:23:40.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.726+0000 7fd736d9d700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7240682f0 0x7fd72406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:23:40.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.727+0000 7fd736d9d700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7240682f0 0x7fd72406a7a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0011e0 tx=0x7fd72c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:23:40.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.730+0000 7fd71f7fe700 1 -- 172.21.15.7:0/2500591457 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7280539d0 con 0x7fd7380caa30 2023-03-03T12:23:40.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.968+0000 7fd73d0fb700 1 -- 172.21.15.7:0/2500591457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7381562b0 con 0x7fd7380caa30 2023-03-03T12:23:40.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.968+0000 7fd71f7fe700 1 -- 172.21.15.7:0/2500591457 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd728056ff0 con 0x7fd7380caa30 2023-03-03T12:23:40.970 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:23:40.970 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:23:40.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.970+0000 7fd73d0fb700 1 -- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7240682f0 msgr2=0x7fd72406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:23:40.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.970+0000 7fd73d0fb700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7240682f0 0x7fd72406a7a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd72c0011e0 tx=0x7fd72c009040 comp rx=0 tx=0).stop 2023-03-03T12:23:40.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.970+0000 7fd73d0fb700 1 -- 172.21.15.7:0/2500591457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7380caa30 msgr2=0x7fd73814a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:23:40.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.970+0000 7fd73d0fb700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7380caa30 0x7fd73814a1a0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fd728004000 tx=0x7fd728004750 comp rx=0 tx=0).stop 2023-03-03T12:23:40.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.970+0000 7fd73d0fb700 1 -- 172.21.15.7:0/2500591457 shutdown_connections 2023-03-03T12:23:40.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.970+0000 7fd73d0fb700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd738154ff0 0x7fd738145a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:23:40.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.970+0000 7fd73d0fb700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7380c00e0 0x7fd738149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:23:40.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.970+0000 7fd73d0fb700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7240682f0 0x7fd72406a7a0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:23:40.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.970+0000 7fd73d0fb700 1 --2- 172.21.15.7:0/2500591457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7380caa30 0x7fd73814a1a0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:23:40.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.971+0000 7fd73d0fb700 1 -- 172.21.15.7:0/2500591457 >> 172.21.15.7:0/2500591457 conn(0x7fd738066e40 msgr2=0x7fd7380c62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:23:40.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.971+0000 7fd73d0fb700 1 -- 172.21.15.7:0/2500591457 shutdown_connections 2023-03-03T12:23:40.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:23:40.971+0000 7fd73d0fb700 1 -- 172.21.15.7:0/2500591457 wait complete. 2023-03-03T12:23:40.974 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:23:40.984 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T12:23:57.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:23:57.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:09.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:24:09.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.466+0000 7fcfc51ad700 1 Processor -- start 2023-03-03T12:24:09.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.466+0000 7fcfc51ad700 1 -- start start 2023-03-03T12:24:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.466+0000 7fcfc51ad700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcfc006e020 0x7fcfc0071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfc51ad700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcfc0072520 0x7fcfc0151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfc51ad700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc0154910 con 0x7fcfc0155390 2023-03-03T12:24:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfc51ad700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc0070b90 con 0x7fcfc0072520 2023-03-03T12:24:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfc51ad700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc0151320 con 0x7fcfc006e020 2023-03-03T12:24:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbf59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcfc0072520 0x7fcfc0151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcfc0155390 0x7fcfc00c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34258/0 (socket says 172.21.15.7:34258) 2023-03-03T12:24:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbed9d700 1 -- 172.21.15.7:0/3190285451 learned_addr learned my addr 172.21.15.7:0/3190285451 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:24:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbe59c700 1 --2- 172.21.15.7:0/3190285451 >> v2:172.21.15.143:3300/0 conn(0x7fcfc006e020 0x7fcfc0071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbf59e700 1 -- 172.21.15.7:0/3190285451 >> v2:172.21.15.143:3300/0 conn(0x7fcfc006e020 msgr2=0x7fcfc0071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:24:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbf59e700 1 --2- 172.21.15.7:0/3190285451 >> v2:172.21.15.143:3300/0 conn(0x7fcfc006e020 0x7fcfc0071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbf59e700 1 -- 172.21.15.7:0/3190285451 >> v1:172.21.15.7:6789/0 conn(0x7fcfc0155390 legacy=0x7fcfc00c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbf59e700 1 -- 172.21.15.7:0/3190285451 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00c4540 con 0x7fcfc0072520 2023-03-03T12:24:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbdd9b700 1 -- 172.21.15.7:0/3190285451 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738324615 0 0) 0x7fcfc0154910 con 0x7fcfc0155390 2023-03-03T12:24:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbe59c700 1 --2- 172.21.15.7:0/3190285451 >> v2:172.21.15.143:3300/0 conn(0x7fcfc006e020 0x7fcfc0071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:24:09.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbf59e700 1 --2- 172.21.15.7:0/3190285451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc0072520 0x7fcfc0151770 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00ac9e0 tx=0x7fcfb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ef8461dffbd562 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:24:09.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbdd9b700 1 -- 172.21.15.7:0/3190285451 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb800e070 con 0x7fcfc0072520 2023-03-03T12:24:09.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbdd9b700 1 -- 172.21.15.7:0/3190285451 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb800bbd0 con 0x7fcfc0072520 2023-03-03T12:24:09.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.467+0000 7fcfbdd9b700 1 -- 172.21.15.7:0/3190285451 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8012430 con 0x7fcfc0072520 2023-03-03T12:24:09.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.468+0000 7fcfc51ad700 1 -- 172.21.15.7:0/3190285451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc0072520 msgr2=0x7fcfc0151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:09.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.468+0000 7fcfc51ad700 1 --2- 172.21.15.7:0/3190285451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc0072520 0x7fcfc0151770 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00ac9e0 tx=0x7fcfb800b330 comp rx=0 tx=0).stop 2023-03-03T12:24:09.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.468+0000 7fcfc51ad700 1 -- 172.21.15.7:0/3190285451 shutdown_connections 2023-03-03T12:24:09.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.468+0000 7fcfc51ad700 1 --2- 172.21.15.7:0/3190285451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc0072520 0x7fcfc0151770 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:09.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.468+0000 7fcfc51ad700 1 --2- 172.21.15.7:0/3190285451 >> v2:172.21.15.143:3300/0 conn(0x7fcfc006e020 0x7fcfc0071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:09.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.468+0000 7fcfc51ad700 1 -- 172.21.15.7:0/3190285451 >> 172.21.15.7:0/3190285451 conn(0x7fcfc006c110 msgr2=0x7fcfc006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:24:09.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.468+0000 7fcfc51ad700 1 -- 172.21.15.7:0/3190285451 shutdown_connections 2023-03-03T12:24:09.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.468+0000 7fcfc51ad700 1 -- 172.21.15.7:0/3190285451 wait complete. 2023-03-03T12:24:09.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.469+0000 7fcfc51ad700 1 Processor -- start 2023-03-03T12:24:09.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.469+0000 7fcfc51ad700 1 -- start start 2023-03-03T12:24:09.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfc51ad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc006e020 0x7fcfc0073960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:09.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfc51ad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc0072520 0x7fcfc01e9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:09.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfc51ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc0155390 0x7fcfc014a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:09.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfc51ad700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc01e96d0 con 0x7fcfc006e020 2023-03-03T12:24:09.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfc51ad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc01548b0 con 0x7fcfc0072520 2023-03-03T12:24:09.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfc51ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc00c07d0 con 0x7fcfc0155390 2023-03-03T12:24:09.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfbed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc006e020 0x7fcfc0073960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:09.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfbed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc006e020 0x7fcfc0073960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40284/0 (socket says 172.21.15.7:40284) 2023-03-03T12:24:09.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfbed9d700 1 -- 172.21.15.7:0/1113486130 learned_addr learned my addr 172.21.15.7:0/1113486130 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:24:09.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfbe59c700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc0072520 0x7fcfc01e9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:09.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfbed9d700 1 -- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc0155390 msgr2=0x7fcfc014a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:09.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfbed9d700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc0155390 0x7fcfc014a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:09.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfbed9d700 1 -- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc0072520 msgr2=0x7fcfc01e9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:09.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.470+0000 7fcfbed9d700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc0072520 0x7fcfc01e9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:09.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.471+0000 7fcfbed9d700 1 -- 172.21.15.7:0/1113486130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb8009a70 con 0x7fcfc006e020 2023-03-03T12:24:09.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.471+0000 7fcfbed9d700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc006e020 0x7fcfc0073960 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400c550 tx=0x7fcfb400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:24:09.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.471+0000 7fcfa77fe700 1 -- 172.21.15.7:0/1113486130 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4004c80 con 0x7fcfc006e020 2023-03-03T12:24:09.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.471+0000 7fcfa77fe700 1 -- 172.21.15.7:0/1113486130 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb4004e00 con 0x7fcfc006e020 2023-03-03T12:24:09.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.471+0000 7fcfa77fe700 1 -- 172.21.15.7:0/1113486130 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4007310 con 0x7fcfc006e020 2023-03-03T12:24:09.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.471+0000 7fcfc51ad700 1 -- 172.21.15.7:0/1113486130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc0070540 con 0x7fcfc006e020 2023-03-03T12:24:09.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.471+0000 7fcfc51ad700 1 -- 172.21.15.7:0/1113486130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc00702a0 con 0x7fcfc006e020 2023-03-03T12:24:09.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.472+0000 7fcfc51ad700 1 -- 172.21.15.7:0/1113486130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc0070820 con 0x7fcfc006e020 2023-03-03T12:24:09.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.473+0000 7fcfa77fe700 1 -- 172.21.15.7:0/1113486130 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcfb401a020 con 0x7fcfc006e020 2023-03-03T12:24:09.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.473+0000 7fcfa77fe700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 0x7fcfac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:09.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.473+0000 7fcfa77fe700 1 -- 172.21.15.7:0/1113486130 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcfb4085e30 con 0x7fcfc006e020 2023-03-03T12:24:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.473+0000 7fcfbe59c700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 0x7fcfac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.474+0000 7fcfbe59c700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 0x7fcfac06a7a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0006fd0 tx=0x7fcfb0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:24:09.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.477+0000 7fcfa77fe700 1 -- 172.21.15.7:0/1113486130 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcfb4052220 con 0x7fcfc006e020 2023-03-03T12:24:09.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.716+0000 7fcfc51ad700 1 -- 172.21.15.7:0/1113486130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfc01e9810 con 0x7fcfc006e020 2023-03-03T12:24:09.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.717+0000 7fcfa77fe700 1 -- 172.21.15.7:0/1113486130 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcfc01e9810 con 0x7fcfc006e020 2023-03-03T12:24:09.718 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:24:09.719 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:24:09.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 -- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 msgr2=0x7fcfac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:09.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 0x7fcfac06a7a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0006fd0 tx=0x7fcfb0008040 comp rx=0 tx=0).stop 2023-03-03T12:24:09.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 -- 172.21.15.7:0/1113486130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc006e020 msgr2=0x7fcfc0073960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:09.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc006e020 0x7fcfc0073960 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400c550 tx=0x7fcfb400c910 comp rx=0 tx=0).stop 2023-03-03T12:24:09.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 -- 172.21.15.7:0/1113486130 shutdown_connections 2023-03-03T12:24:09.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcfc0155390 0x7fcfc014a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:09.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfac0682f0 0x7fcfac06a7a0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:09.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcfc006e020 0x7fcfc0073960 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:09.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 --2- 172.21.15.7:0/1113486130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcfc0072520 0x7fcfc01e9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:09.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 -- 172.21.15.7:0/1113486130 >> 172.21.15.7:0/1113486130 conn(0x7fcfc006c110 msgr2=0x7fcfc014e230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:24:09.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 -- 172.21.15.7:0/1113486130 shutdown_connections 2023-03-03T12:24:09.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:09.719+0000 7fcfc51ad700 1 -- 172.21.15.7:0/1113486130 wait complete. 2023-03-03T12:24:09.722 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:24:09.732 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T12:24:27.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:27.572 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:46.985 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:24:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.123+0000 7f7dfbd28700 1 Processor -- start 2023-03-03T12:24:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.123+0000 7f7dfbd28700 1 -- start start 2023-03-03T12:24:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.124+0000 7f7dfbd28700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7df414f9d0 0x7f7df4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.124+0000 7f7dfbd28700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df40c2f30 con 0x7f7df40abe20 2023-03-03T12:24:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.124+0000 7f7dfbd28700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df40c31d0 con 0x7f7df414fda0 2023-03-03T12:24:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.124+0000 7f7dfbd28700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df41524a0 con 0x7f7df414f9d0 2023-03-03T12:24:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.124+0000 7f7df9ac4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7df414f9d0 0x7f7df4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.124+0000 7f7dfa2c5700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7df40abe20 0x7f7df41529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34304/0 (socket says 172.21.15.7:34304) 2023-03-03T12:24:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.124+0000 7f7dfa2c5700 1 -- 172.21.15.7:0/908658532 learned_addr learned my addr 172.21.15.7:0/908658532 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:24:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.125+0000 7f7df9ac4700 1 -- 172.21.15.7:0/908658532 >> v1:172.21.15.143:6789/0 conn(0x7f7df414fda0 legacy=0x7f7df4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:24:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.125+0000 7f7df9ac4700 1 -- 172.21.15.7:0/908658532 >> v1:172.21.15.7:6789/0 conn(0x7f7df40abe20 legacy=0x7f7df41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.125+0000 7f7df9ac4700 1 -- 172.21.15.7:0/908658532 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df40c34b0 con 0x7f7df414f9d0 2023-03-03T12:24:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.125+0000 7f7df9ac4700 1 --2- 172.21.15.7:0/908658532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414f9d0 0x7f7df4156430 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7de4009840 tx=0x7f7de4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5588f0f9cd2065e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:24:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.125+0000 7f7df8ac2700 1 -- 172.21.15.7:0/908658532 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de4004230 con 0x7f7df414f9d0 2023-03-03T12:24:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.125+0000 7f7df8ac2700 1 -- 172.21.15.7:0/908658532 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de4004390 con 0x7f7df414f9d0 2023-03-03T12:24:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.125+0000 7f7df8ac2700 1 -- 172.21.15.7:0/908658532 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de40036a0 con 0x7f7df414f9d0 2023-03-03T12:24:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.126+0000 7f7dfbd28700 1 -- 172.21.15.7:0/908658532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414f9d0 msgr2=0x7f7df4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.126+0000 7f7dfbd28700 1 --2- 172.21.15.7:0/908658532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414f9d0 0x7f7df4156430 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7de4009840 tx=0x7f7de4009b50 comp rx=0 tx=0).stop 2023-03-03T12:24:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.126+0000 7f7dfbd28700 1 -- 172.21.15.7:0/908658532 shutdown_connections 2023-03-03T12:24:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.126+0000 7f7dfbd28700 1 --2- 172.21.15.7:0/908658532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414f9d0 0x7f7df4156430 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.126+0000 7f7dfbd28700 1 -- 172.21.15.7:0/908658532 >> 172.21.15.7:0/908658532 conn(0x7f7df40c8900 msgr2=0x7f7df40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:24:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.126+0000 7f7dfbd28700 1 -- 172.21.15.7:0/908658532 shutdown_connections 2023-03-03T12:24:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.126+0000 7f7dfbd28700 1 -- 172.21.15.7:0/908658532 wait complete. 2023-03-03T12:24:47.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.127+0000 7f7dfbd28700 1 Processor -- start 2023-03-03T12:24:47.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.127+0000 7f7dfbd28700 1 -- start start 2023-03-03T12:24:47.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfbd28700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df40abe20 0x7f7df41491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:47.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfbd28700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7df414f9d0 0x7f7df4154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:47.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfbd28700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414fda0 0x7f7df414a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:47.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfbd28700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df4149710 con 0x7f7df414fda0 2023-03-03T12:24:47.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfbd28700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df41dba50 con 0x7f7df40abe20 2023-03-03T12:24:47.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfbd28700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df41dbd30 con 0x7f7df414f9d0 2023-03-03T12:24:47.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfa2c5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414fda0 0x7f7df414a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:47.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfa2c5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414fda0 0x7f7df414a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40332/0 (socket says 172.21.15.7:40332) 2023-03-03T12:24:47.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfa2c5700 1 -- 172.21.15.7:0/2799232010 learned_addr learned my addr 172.21.15.7:0/2799232010 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:24:47.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7df9ac4700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df40abe20 0x7f7df41491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:47.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7df92c3700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7df414f9d0 0x7f7df4154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:47.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfa2c5700 1 -- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7df414f9d0 msgr2=0x7f7df4154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:47.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfa2c5700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7df414f9d0 0x7f7df4154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:47.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfa2c5700 1 -- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df40abe20 msgr2=0x7f7df41491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:47.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfa2c5700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df40abe20 0x7f7df41491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfa2c5700 1 -- 172.21.15.7:0/2799232010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de40094a0 con 0x7f7df414fda0 2023-03-03T12:24:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7df9ac4700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df40abe20 0x7f7df41491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:24:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dfa2c5700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414fda0 0x7f7df414a080 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00caa0 tx=0x7f7dec00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:24:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dea7fc700 1 -- 172.21.15.7:0/2799232010 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec00a9e0 con 0x7f7df414fda0 2023-03-03T12:24:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.128+0000 7f7dea7fc700 1 -- 172.21.15.7:0/2799232010 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dec00ab40 con 0x7f7df414fda0 2023-03-03T12:24:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.129+0000 7f7dfbd28700 1 -- 172.21.15.7:0/2799232010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df40ac350 con 0x7f7df414fda0 2023-03-03T12:24:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.129+0000 7f7dea7fc700 1 -- 172.21.15.7:0/2799232010 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec012400 con 0x7f7df414fda0 2023-03-03T12:24:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.129+0000 7f7dfbd28700 1 -- 172.21.15.7:0/2799232010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df40ab630 con 0x7f7df414fda0 2023-03-03T12:24:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.130+0000 7f7dfbd28700 1 -- 172.21.15.7:0/2799232010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df4071fa0 con 0x7f7df414fda0 2023-03-03T12:24:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.130+0000 7f7dea7fc700 1 -- 172.21.15.7:0/2799232010 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7dec019030 con 0x7f7df414fda0 2023-03-03T12:24:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.130+0000 7f7dea7fc700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dd8068320 0x7f7dd806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:24:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.131+0000 7f7dea7fc700 1 -- 172.21.15.7:0/2799232010 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7dec085b60 con 0x7f7df414fda0 2023-03-03T12:24:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.131+0000 7f7df9ac4700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dd8068320 0x7f7dd806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:24:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.131+0000 7f7df9ac4700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dd8068320 0x7f7dd806a7d0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f7de4000c00 tx=0x7f7de401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:24:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.135+0000 7f7dea7fc700 1 -- 172.21.15.7:0/2799232010 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7dec04b5e0 con 0x7f7df414fda0 2023-03-03T12:24:47.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.375+0000 7f7dfbd28700 1 -- 172.21.15.7:0/2799232010 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7df4074bb0 con 0x7f7df414fda0 2023-03-03T12:24:47.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.375+0000 7f7dea7fc700 1 -- 172.21.15.7:0/2799232010 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7dec051fb0 con 0x7f7df414fda0 2023-03-03T12:24:47.377 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:24:47.377 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:24:47.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 -- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dd8068320 msgr2=0x7f7dd806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:47.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dd8068320 0x7f7dd806a7d0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f7de4000c00 tx=0x7f7de401c040 comp rx=0 tx=0).stop 2023-03-03T12:24:47.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 -- 172.21.15.7:0/2799232010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414fda0 msgr2=0x7f7df414a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:24:47.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414fda0 0x7f7df414a080 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00caa0 tx=0x7f7dec00cdb0 comp rx=0 tx=0).stop 2023-03-03T12:24:47.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 -- 172.21.15.7:0/2799232010 shutdown_connections 2023-03-03T12:24:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dd8068320 0x7f7dd806a7d0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7df414fda0 0x7f7df414a080 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7df40abe20 0x7f7df41491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 --2- 172.21.15.7:0/2799232010 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7df414f9d0 0x7f7df4154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:24:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 -- 172.21.15.7:0/2799232010 >> 172.21.15.7:0/2799232010 conn(0x7f7df40c8900 msgr2=0x7f7df40c6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:24:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 -- 172.21.15.7:0/2799232010 shutdown_connections 2023-03-03T12:24:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:24:47.377+0000 7f7dfbd28700 1 -- 172.21.15.7:0/2799232010 wait complete. 2023-03-03T12:24:47.381 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:24:47.390 INFO:tasks.mds_thrash.fs.[a]:waiting for 28 secs before thrashing 2023-03-03T12:24:57.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:24:57.727 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:15.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:25:15.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.529+0000 7f5951fbb700 1 Processor -- start 2023-03-03T12:25:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.529+0000 7f5951fbb700 1 -- start start 2023-03-03T12:25:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f5951fbb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f594c071e80 0x7f594c151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f5951fbb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f594c072250 0x7f594c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f5951fbb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c1510d0 con 0x7f594c0c1ce0 2023-03-03T12:25:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f5951fbb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c0c3740 con 0x7f594c072250 2023-03-03T12:25:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f5951fbb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c070670 con 0x7f594c071e80 2023-03-03T12:25:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f594c072250 0x7f594c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f594c0c1ce0 0x7f594c1546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34334/0 (socket says 172.21.15.7:34334) 2023-03-03T12:25:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594bfff700 1 -- 172.21.15.7:0/3146675330 learned_addr learned my addr 172.21.15.7:0/3146675330 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:25:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594b7fe700 1 --2- 172.21.15.7:0/3146675330 >> v2:172.21.15.143:3300/0 conn(0x7f594c071e80 0x7f594c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594affd700 1 -- 172.21.15.7:0/3146675330 >> v2:172.21.15.143:3300/0 conn(0x7f594c071e80 msgr2=0x7f594c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:25:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594affd700 1 --2- 172.21.15.7:0/3146675330 >> v2:172.21.15.143:3300/0 conn(0x7f594c071e80 0x7f594c151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594affd700 1 -- 172.21.15.7:0/3146675330 >> v1:172.21.15.7:6789/0 conn(0x7f594c0c1ce0 legacy=0x7f594c1546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594a7fc700 1 -- 172.21.15.7:0/3146675330 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127918357 0 0) 0x7f594c1510d0 con 0x7f594c0c1ce0 2023-03-03T12:25:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594affd700 1 -- 172.21.15.7:0/3146675330 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c0c39e0 con 0x7f594c072250 2023-03-03T12:25:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594b7fe700 1 --2- 172.21.15.7:0/3146675330 >> v2:172.21.15.143:3300/0 conn(0x7f594c071e80 0x7f594c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:25:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.530+0000 7f594affd700 1 --2- 172.21.15.7:0/3146675330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c072250 0x7f594c14e1f0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f593c009840 tx=0x7f593c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac311530640aaf56 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:25:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f594a7fc700 1 -- 172.21.15.7:0/3146675330 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c011830 con 0x7f594c072250 2023-03-03T12:25:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f594a7fc700 1 -- 172.21.15.7:0/3146675330 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c011990 con 0x7f594c072250 2023-03-03T12:25:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f594a7fc700 1 -- 172.21.15.7:0/3146675330 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c011d20 con 0x7f594c072250 2023-03-03T12:25:15.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f5951fbb700 1 -- 172.21.15.7:0/3146675330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c072250 msgr2=0x7f594c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:15.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f5951fbb700 1 --2- 172.21.15.7:0/3146675330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c072250 0x7f594c14e1f0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f593c009840 tx=0x7f593c009b50 comp rx=0 tx=0).stop 2023-03-03T12:25:15.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f5951fbb700 1 -- 172.21.15.7:0/3146675330 shutdown_connections 2023-03-03T12:25:15.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f5951fbb700 1 --2- 172.21.15.7:0/3146675330 >> v2:172.21.15.143:3300/0 conn(0x7f594c071e80 0x7f594c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:15.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f5951fbb700 1 --2- 172.21.15.7:0/3146675330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c072250 0x7f594c14e1f0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:15.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f5951fbb700 1 -- 172.21.15.7:0/3146675330 >> 172.21.15.7:0/3146675330 conn(0x7f594c0a94d0 msgr2=0x7f594c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:25:15.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f5951fbb700 1 -- 172.21.15.7:0/3146675330 shutdown_connections 2023-03-03T12:25:15.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.531+0000 7f5951fbb700 1 -- 172.21.15.7:0/3146675330 wait complete. 2023-03-03T12:25:15.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.533+0000 7f5951fbb700 1 Processor -- start 2023-03-03T12:25:15.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.533+0000 7f5951fbb700 1 -- start start 2023-03-03T12:25:15.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f5951fbb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f594c071e80 0x7f594c0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:15.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f5951fbb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f594c072250 0x7f594c13f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:15.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f5951fbb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c0c1ce0 0x7f594c13fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:15.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f5951fbb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c13ff70 con 0x7f594c0c1ce0 2023-03-03T12:25:15.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f5951fbb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c14ef40 con 0x7f594c071e80 2023-03-03T12:25:15.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f5951fbb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c14fac0 con 0x7f594c072250 2023-03-03T12:25:15.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c0c1ce0 0x7f594c13fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:15.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c0c1ce0 0x7f594c13fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40360/0 (socket says 172.21.15.7:40360) 2023-03-03T12:25:15.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594bfff700 1 -- 172.21.15.7:0/2444662333 learned_addr learned my addr 172.21.15.7:0/2444662333 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:25:15.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594b7fe700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f594c071e80 0x7f594c0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:15.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594affd700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f594c072250 0x7f594c13f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:15.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594bfff700 1 -- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f594c072250 msgr2=0x7f594c13f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:15.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594bfff700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f594c072250 0x7f594c13f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:15.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594bfff700 1 -- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f594c071e80 msgr2=0x7f594c0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:15.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594bfff700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f594c071e80 0x7f594c0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:15.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594bfff700 1 -- 172.21.15.7:0/2444662333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0094a0 con 0x7f594c0c1ce0 2023-03-03T12:25:15.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594b7fe700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f594c071e80 0x7f594c0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:25:15.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.534+0000 7f594affd700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f594c072250 0x7f594c13f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:25:15.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.535+0000 7f594bfff700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c0c1ce0 0x7f594c13fa30 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f594400cc40 tx=0x7f59440021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:25:15.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.535+0000 7f5933fff700 1 -- 172.21.15.7:0/2444662333 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594400ab40 con 0x7f594c0c1ce0 2023-03-03T12:25:15.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.535+0000 7f5933fff700 1 -- 172.21.15.7:0/2444662333 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594400acc0 con 0x7f594c0c1ce0 2023-03-03T12:25:15.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.535+0000 7f5951fbb700 1 -- 172.21.15.7:0/2444662333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c06c9a0 con 0x7f594c0c1ce0 2023-03-03T12:25:15.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.535+0000 7f5933fff700 1 -- 172.21.15.7:0/2444662333 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5944007310 con 0x7f594c0c1ce0 2023-03-03T12:25:15.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.535+0000 7f5951fbb700 1 -- 172.21.15.7:0/2444662333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c151300 con 0x7f594c0c1ce0 2023-03-03T12:25:15.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.536+0000 7f5951fbb700 1 -- 172.21.15.7:0/2444662333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594c06cb30 con 0x7f594c0c1ce0 2023-03-03T12:25:15.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.536+0000 7f5933fff700 1 -- 172.21.15.7:0/2444662333 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f594401a030 con 0x7f594c0c1ce0 2023-03-03T12:25:15.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.537+0000 7f5933fff700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59380682f0 0x7f593806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:15.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.537+0000 7f5933fff700 1 -- 172.21.15.7:0/2444662333 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5944085de0 con 0x7f594c0c1ce0 2023-03-03T12:25:15.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.537+0000 7f594b7fe700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59380682f0 0x7f593806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:15.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.538+0000 7f594b7fe700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59380682f0 0x7f593806a7a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f5940009a00 tx=0x7f5940008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:25:15.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.541+0000 7f5933fff700 1 -- 172.21.15.7:0/2444662333 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5944052180 con 0x7f594c0c1ce0 2023-03-03T12:25:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.781+0000 7f5951fbb700 1 -- 172.21.15.7:0/2444662333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f594c074bb0 con 0x7f594c0c1ce0 2023-03-03T12:25:15.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.781+0000 7f5933fff700 1 -- 172.21.15.7:0/2444662333 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5944018090 con 0x7f594c0c1ce0 2023-03-03T12:25:15.783 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:25:15.783 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:25:15.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 -- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59380682f0 msgr2=0x7f593806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:15.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59380682f0 0x7f593806a7a0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f5940009a00 tx=0x7f5940008040 comp rx=0 tx=0).stop 2023-03-03T12:25:15.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 -- 172.21.15.7:0/2444662333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c0c1ce0 msgr2=0x7f594c13fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:15.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c0c1ce0 0x7f594c13fa30 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f594400cc40 tx=0x7f59440021c0 comp rx=0 tx=0).stop 2023-03-03T12:25:15.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 -- 172.21.15.7:0/2444662333 shutdown_connections 2023-03-03T12:25:15.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59380682f0 0x7f593806a7a0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:15.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f594c0c1ce0 0x7f594c13fa30 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:15.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f594c071e80 0x7f594c0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:15.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 --2- 172.21.15.7:0/2444662333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f594c072250 0x7f594c13f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:15.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 -- 172.21.15.7:0/2444662333 >> 172.21.15.7:0/2444662333 conn(0x7f594c0a94d0 msgr2=0x7f594c151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:25:15.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.783+0000 7f5951fbb700 1 -- 172.21.15.7:0/2444662333 shutdown_connections 2023-03-03T12:25:15.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:15.784+0000 7f5951fbb700 1 -- 172.21.15.7:0/2444662333 wait complete. 2023-03-03T12:25:15.787 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:25:15.797 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T12:25:27.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:27.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:57.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:25:57.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.936+0000 7f0c79436700 1 Processor -- start 2023-03-03T12:25:57.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.936+0000 7f0c79436700 1 -- start start 2023-03-03T12:25:57.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.936+0000 7f0c79436700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c740caa30 0x7f0c740c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.936+0000 7f0c79436700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c740c00e0 0x7f0c74154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.936+0000 7f0c79436700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0c740cae00 0x7f0c740c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.936+0000 7f0c79436700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c740cb280 con 0x7f0c740caa30 2023-03-03T12:25:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.936+0000 7f0c79436700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c740bfd10 con 0x7f0c740cae00 2023-03-03T12:25:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.936+0000 7f0c79436700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740bfa30 con 0x7f0c740c00e0 2023-03-03T12:25:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c72ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c740caa30 0x7f0c740c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c72ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c740caa30 0x7f0c740c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40392/0 (socket says 172.21.15.7:40392) 2023-03-03T12:25:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c72ffd700 1 -- 172.21.15.7:0/880268326 learned_addr learned my addr 172.21.15.7:0/880268326 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:25:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c737fe700 1 --2- 172.21.15.7:0/880268326 >> v2:172.21.15.143:3300/0 conn(0x7f0c740cae00 0x7f0c740c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c727fc700 1 --2- 172.21.15.7:0/880268326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c740c00e0 0x7f0c74154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c72ffd700 1 -- 172.21.15.7:0/880268326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c740c00e0 msgr2=0x7f0c74154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:25:57.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c72ffd700 1 --2- 172.21.15.7:0/880268326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c740c00e0 0x7f0c74154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c72ffd700 1 -- 172.21.15.7:0/880268326 >> v2:172.21.15.143:3300/0 conn(0x7f0c740cae00 msgr2=0x7f0c740c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:25:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c72ffd700 1 --2- 172.21.15.7:0/880268326 >> v2:172.21.15.143:3300/0 conn(0x7f0c740cae00 0x7f0c740c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c72ffd700 1 -- 172.21.15.7:0/880268326 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c74154510 con 0x7f0c740caa30 2023-03-03T12:25:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c72ffd700 1 --2- 172.21.15.7:0/880268326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740caa30 0x7f0c740c5c60 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f0c74158a70 tx=0x7f0c68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46812570f77ef495 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:25:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c71ffb700 1 -- 172.21.15.7:0/880268326 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800b870 con 0x7f0c740caa30 2023-03-03T12:25:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c71ffb700 1 -- 172.21.15.7:0/880268326 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6800b9d0 con 0x7f0c740caa30 2023-03-03T12:25:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.937+0000 7f0c71ffb700 1 -- 172.21.15.7:0/880268326 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800bd60 con 0x7f0c740caa30 2023-03-03T12:25:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.938+0000 7f0c79436700 1 -- 172.21.15.7:0/880268326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740caa30 msgr2=0x7f0c740c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.938+0000 7f0c79436700 1 --2- 172.21.15.7:0/880268326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740caa30 0x7f0c740c5c60 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f0c74158a70 tx=0x7f0c68009bd0 comp rx=0 tx=0).stop 2023-03-03T12:25:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.938+0000 7f0c79436700 1 -- 172.21.15.7:0/880268326 shutdown_connections 2023-03-03T12:25:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.938+0000 7f0c79436700 1 --2- 172.21.15.7:0/880268326 >> v2:172.21.15.143:3300/0 conn(0x7f0c740cae00 0x7f0c740c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.938+0000 7f0c79436700 1 --2- 172.21.15.7:0/880268326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740caa30 0x7f0c740c5c60 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.938+0000 7f0c79436700 1 --2- 172.21.15.7:0/880268326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c740c00e0 0x7f0c74154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.938+0000 7f0c79436700 1 -- 172.21.15.7:0/880268326 >> 172.21.15.7:0/880268326 conn(0x7f0c74066e40 msgr2=0x7f0c74071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:25:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.938+0000 7f0c79436700 1 -- 172.21.15.7:0/880268326 shutdown_connections 2023-03-03T12:25:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.938+0000 7f0c79436700 1 -- 172.21.15.7:0/880268326 wait complete. 2023-03-03T12:25:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.939+0000 7f0c79436700 1 Processor -- start 2023-03-03T12:25:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.939+0000 7f0c79436700 1 -- start start 2023-03-03T12:25:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c79436700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740c00e0 0x7f0c741de000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c79436700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c740caa30 0x7f0c74158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c79436700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c740cae00 0x7f0c741ed280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:57.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c79436700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741de8f0 con 0x7f0c740c00e0 2023-03-03T12:25:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c79436700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741e1900 con 0x7f0c740caa30 2023-03-03T12:25:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c79436700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c74156990 con 0x7f0c740cae00 2023-03-03T12:25:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c72ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740c00e0 0x7f0c741de000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c72ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740c00e0 0x7f0c741de000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40398/0 (socket says 172.21.15.7:40398) 2023-03-03T12:25:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c72ffd700 1 -- 172.21.15.7:0/3284059397 learned_addr learned my addr 172.21.15.7:0/3284059397 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:25:57.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c727fc700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c740caa30 0x7f0c74158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c737fe700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c740cae00 0x7f0c741ed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c72ffd700 1 -- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c740cae00 msgr2=0x7f0c741ed280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c72ffd700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c740cae00 0x7f0c741ed280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c72ffd700 1 -- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c740caa30 msgr2=0x7f0c74158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c72ffd700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c740caa30 0x7f0c74158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.940+0000 7f0c72ffd700 1 -- 172.21.15.7:0/3284059397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c68009550 con 0x7f0c740c00e0 2023-03-03T12:25:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.941+0000 7f0c72ffd700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740c00e0 0x7f0c741de000 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0c7406b270 tx=0x7f0c68004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:25:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.941+0000 7f0c5b7fe700 1 -- 172.21.15.7:0/3284059397 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68004540 con 0x7f0c740c00e0 2023-03-03T12:25:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.941+0000 7f0c5b7fe700 1 -- 172.21.15.7:0/3284059397 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c680036a0 con 0x7f0c740c00e0 2023-03-03T12:25:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.941+0000 7f0c79436700 1 -- 172.21.15.7:0/3284059397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c74154ad0 con 0x7f0c740c00e0 2023-03-03T12:25:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.941+0000 7f0c5b7fe700 1 -- 172.21.15.7:0/3284059397 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68003a50 con 0x7f0c740c00e0 2023-03-03T12:25:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.941+0000 7f0c79436700 1 -- 172.21.15.7:0/3284059397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c741547f0 con 0x7f0c740c00e0 2023-03-03T12:25:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.942+0000 7f0c79436700 1 -- 172.21.15.7:0/3284059397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c740bf650 con 0x7f0c740c00e0 2023-03-03T12:25:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.942+0000 7f0c5b7fe700 1 -- 172.21.15.7:0/3284059397 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0c6800f050 con 0x7f0c740c00e0 2023-03-03T12:25:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.943+0000 7f0c5b7fe700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c600682f0 0x7f0c6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:25:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.943+0000 7f0c5b7fe700 1 -- 172.21.15.7:0/3284059397 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0c68087990 con 0x7f0c740c00e0 2023-03-03T12:25:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.943+0000 7f0c727fc700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c600682f0 0x7f0c6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:25:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.944+0000 7f0c727fc700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c600682f0 0x7f0c6006a7a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f0c74071da0 tx=0x7f0c64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:25:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:57.947+0000 7f0c5b7fe700 1 -- 172.21.15.7:0/3284059397 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0c68053d30 con 0x7f0c740c00e0 2023-03-03T12:25:57.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:57.993 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:25:58.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.195+0000 7f0c79436700 1 -- 172.21.15.7:0/3284059397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c740afa20 con 0x7f0c740c00e0 2023-03-03T12:25:58.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.195+0000 7f0c5b7fe700 1 -- 172.21.15.7:0/3284059397 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0c68057350 con 0x7f0c740c00e0 2023-03-03T12:25:58.197 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:25:58.197 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:25:58.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 -- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c600682f0 msgr2=0x7f0c6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:58.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c600682f0 0x7f0c6006a7a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f0c74071da0 tx=0x7f0c64008040 comp rx=0 tx=0).stop 2023-03-03T12:25:58.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 -- 172.21.15.7:0/3284059397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740c00e0 msgr2=0x7f0c741de000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:25:58.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740c00e0 0x7f0c741de000 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0c7406b270 tx=0x7f0c68004750 comp rx=0 tx=0).stop 2023-03-03T12:25:58.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 -- 172.21.15.7:0/3284059397 shutdown_connections 2023-03-03T12:25:58.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c740cae00 0x7f0c741ed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:58.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c600682f0 0x7f0c6006a7a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:58.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c740c00e0 0x7f0c741de000 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:58.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 --2- 172.21.15.7:0/3284059397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c740caa30 0x7f0c74158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:25:58.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 -- 172.21.15.7:0/3284059397 >> 172.21.15.7:0/3284059397 conn(0x7f0c74066e40 msgr2=0x7f0c74071560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:25:58.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 -- 172.21.15.7:0/3284059397 shutdown_connections 2023-03-03T12:25:58.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:25:58.197+0000 7f0c79436700 1 -- 172.21.15.7:0/3284059397 wait complete. 2023-03-03T12:25:58.201 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:25:58.211 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T12:26:05.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:26:05.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.870+0000 7fa89e427700 1 Processor -- start 2023-03-03T12:26:05.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.870+0000 7fa89e427700 1 -- start start 2023-03-03T12:26:05.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa89e427700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa898151e80 0x7fa8980c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:05.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa89e427700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8980c08d0 con 0x7fa8980c6fe0 2023-03-03T12:26:05.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa89e427700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa898154930 con 0x7fa898155410 2023-03-03T12:26:05.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa89e427700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa898152250 con 0x7fa898151e80 2023-03-03T12:26:05.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa89c9c4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa8980c6fe0 0x7fa8980bfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34396/0 (socket says 172.21.15.7:34396) 2023-03-03T12:26:05.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa89c9c4700 1 -- 172.21.15.7:0/2055830930 learned_addr learned my addr 172.21.15.7:0/2055830930 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:26:05.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa897fff700 1 --2- 172.21.15.7:0/2055830930 >> v2:172.21.15.143:3300/0 conn(0x7fa898151e80 0x7fa8980c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:05.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789544456 0 0) 0x7fa8980c08d0 con 0x7fa8980c6fe0 2023-03-03T12:26:05.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8840035b0 con 0x7fa8980c6fe0 2023-03-03T12:26:05.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2727730588 0 0) 0x7fa8840035b0 con 0x7fa8980c6fe0 2023-03-03T12:26:05.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 >> v2:172.21.15.143:3300/0 conn(0x7fa898151e80 msgr2=0x7fa8980c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:26:05.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.871+0000 7fa896ffd700 1 --2- 172.21.15.7:0/2055830930 >> v2:172.21.15.143:3300/0 conn(0x7fa898151e80 0x7fa8980c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:05.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 >> v1:172.21.15.143:6789/0 conn(0x7fa898155410 legacy=0x7fa8980bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:05.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa898154c10 con 0x7fa8980c6fe0 2023-03-03T12:26:05.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa8900019f0 con 0x7fa8980c6fe0 2023-03-03T12:26:05.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 >> v1:172.21.15.7:6789/0 conn(0x7fa8980c6fe0 legacy=0x7fa8980bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:05.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa884003a70 0x7fa884003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:05.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa884004850 0x7fa884004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:05.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa88400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:05.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88400c0e0 con 0x7fa884005200 2023-03-03T12:26:05.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88400c220 con 0x7fa884004850 2023-03-03T12:26:05.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88400c3e0 con 0x7fa884003a70 2023-03-03T12:26:05.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa89c9c4700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa88400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:05.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa8977fe700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa884004850 0x7fa884004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:05.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa897fff700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa884003a70 0x7fa884003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:05.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa89c9c4700 1 -- 172.21.15.7:0/2055830930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa884003a70 msgr2=0x7fa884003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:26:05.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa89c9c4700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa884003a70 0x7fa884003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:05.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa89c9c4700 1 -- 172.21.15.7:0/2055830930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa884004850 msgr2=0x7fa884004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:26:05.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa89c9c4700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa884004850 0x7fa884004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:05.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa89c9c4700 1 -- 172.21.15.7:0/2055830930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8840035b0 con 0x7fa884005200 2023-03-03T12:26:05.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa89c9c4700 1 -- 172.21.15.7:0/2055830930 reap_dead start 2023-03-03T12:26:05.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa89c9c4700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa88400ba10 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fa89000a830 tx=0x7fa884005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c444c5dbed23ddc2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:26:05.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89000a2f0 con 0x7fa884005200 2023-03-03T12:26:05.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa890012600 con 0x7fa884005200 2023-03-03T12:26:05.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.872+0000 7fa896ffd700 1 -- 172.21.15.7:0/2055830930 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8900129b0 con 0x7fa884005200 2023-03-03T12:26:05.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.873+0000 7fa89e427700 1 -- 172.21.15.7:0/2055830930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 msgr2=0x7fa88400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:05.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.873+0000 7fa89e427700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa88400ba10 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fa89000a830 tx=0x7fa884005880 comp rx=0 tx=0).stop 2023-03-03T12:26:05.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.873+0000 7fa89e427700 1 -- 172.21.15.7:0/2055830930 shutdown_connections 2023-03-03T12:26:05.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.873+0000 7fa89e427700 1 --2- 172.21.15.7:0/2055830930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa88400ba10 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:05.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.873+0000 7fa89e427700 1 -- 172.21.15.7:0/2055830930 >> 172.21.15.7:0/2055830930 conn(0x7fa898071630 msgr2=0x7fa89806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:26:05.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.873+0000 7fa89e427700 1 -- 172.21.15.7:0/2055830930 shutdown_connections 2023-03-03T12:26:05.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.873+0000 7fa89e427700 1 -- 172.21.15.7:0/2055830930 wait complete. 2023-03-03T12:26:05.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.874+0000 7fa89e427700 1 Processor -- start 2023-03-03T12:26:05.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.874+0000 7fa89e427700 1 -- start start 2023-03-03T12:26:05.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa89e427700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa8981462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:05.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa89e427700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa898155ab0 0x7fa8981e10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:05.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa89e427700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa89814df40 0x7fa89814e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:05.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa89e427700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8981571e0 con 0x7fa884005200 2023-03-03T12:26:05.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa89e427700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8981eae30 con 0x7fa898155ab0 2023-03-03T12:26:05.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa89e427700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8981eb360 con 0x7fa89814df40 2023-03-03T12:26:05.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa897fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa8981462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:05.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa897fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa8981462c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40424/0 (socket says 172.21.15.7:40424) 2023-03-03T12:26:05.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa897fff700 1 -- 172.21.15.7:0/4239881843 learned_addr learned my addr 172.21.15.7:0/4239881843 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:26:05.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa8977fe700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa898155ab0 0x7fa8981e10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:05.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.875+0000 7fa89c9c4700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa89814df40 0x7fa89814e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:05.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa897fff700 1 -- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa89814df40 msgr2=0x7fa89814e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:05.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa897fff700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa89814df40 0x7fa89814e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:05.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa897fff700 1 -- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa898155ab0 msgr2=0x7fa8981e10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:05.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa897fff700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa898155ab0 0x7fa8981e10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:05.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa897fff700 1 -- 172.21.15.7:0/4239881843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8980c5220 con 0x7fa884005200 2023-03-03T12:26:05.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa897fff700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa8981462c0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fa8880119d0 tx=0x7fa888011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:26:05.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa894ff9700 1 -- 172.21.15.7:0/4239881843 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888011e70 con 0x7fa884005200 2023-03-03T12:26:05.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa894ff9700 1 -- 172.21.15.7:0/4239881843 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88800f940 con 0x7fa884005200 2023-03-03T12:26:05.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa894ff9700 1 -- 172.21.15.7:0/4239881843 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88800fcd0 con 0x7fa884005200 2023-03-03T12:26:05.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa89e427700 1 -- 172.21.15.7:0/4239881843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa898154ef0 con 0x7fa884005200 2023-03-03T12:26:05.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.876+0000 7fa89e427700 1 -- 172.21.15.7:0/4239881843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8980c5b00 con 0x7fa884005200 2023-03-03T12:26:05.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.877+0000 7fa89e427700 1 -- 172.21.15.7:0/4239881843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa898074bb0 con 0x7fa884005200 2023-03-03T12:26:05.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.877+0000 7fa894ff9700 1 -- 172.21.15.7:0/4239881843 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa888014440 con 0x7fa884005200 2023-03-03T12:26:05.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.878+0000 7fa894ff9700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8780682f0 0x7fa87806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:26:05.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.878+0000 7fa894ff9700 1 -- 172.21.15.7:0/4239881843 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa888086970 con 0x7fa884005200 2023-03-03T12:26:05.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.878+0000 7fa8977fe700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8780682f0 0x7fa87806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:26:05.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.879+0000 7fa8977fe700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8780682f0 0x7fa87806a7a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa898155ec0 tx=0x7fa880009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:26:05.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:05.882+0000 7fa894ff9700 1 -- 172.21.15.7:0/4239881843 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa888052c70 con 0x7fa884005200 2023-03-03T12:26:06.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.122+0000 7fa89e427700 1 -- 172.21.15.7:0/4239881843 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8980c6860 con 0x7fa884005200 2023-03-03T12:26:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.123+0000 7fa894ff9700 1 -- 172.21.15.7:0/4239881843 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa888056290 con 0x7fa884005200 2023-03-03T12:26:06.125 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:26:06.125 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:26:06.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 -- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8780682f0 msgr2=0x7fa87806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8780682f0 0x7fa87806a7a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa898155ec0 tx=0x7fa880009040 comp rx=0 tx=0).stop 2023-03-03T12:26:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 -- 172.21.15.7:0/4239881843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 msgr2=0x7fa8981462c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:26:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa8981462c0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fa8880119d0 tx=0x7fa888011ce0 comp rx=0 tx=0).stop 2023-03-03T12:26:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 -- 172.21.15.7:0/4239881843 shutdown_connections 2023-03-03T12:26:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa89814df40 0x7fa89814e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8780682f0 0x7fa87806a7a0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa884005200 0x7fa8981462c0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 --2- 172.21.15.7:0/4239881843 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa898155ab0 0x7fa8981e10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:26:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 -- 172.21.15.7:0/4239881843 >> 172.21.15.7:0/4239881843 conn(0x7fa898071630 msgr2=0x7fa89814fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:26:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 -- 172.21.15.7:0/4239881843 shutdown_connections 2023-03-03T12:26:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:26:06.125+0000 7fa89e427700 1 -- 172.21.15.7:0/4239881843 wait complete. 2023-03-03T12:26:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:26:06.138 INFO:tasks.mds_thrash.fs.[b]:waiting for 79 secs before thrashing 2023-03-03T12:26:28.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:28.133 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:58.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:26:58.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:03.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:27:03.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.349+0000 7fbbc11f7700 1 Processor -- start 2023-03-03T12:27:03.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.349+0000 7fbbc11f7700 1 -- start start 2023-03-03T12:27:03.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.350+0000 7fbbc11f7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbbbc14f9d0 0x7fbbbc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:03.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.350+0000 7fbbc11f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14fda0 0x7fbbbc071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:03.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.350+0000 7fbbc11f7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbbbc0abe20 0x7fbbbc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:03.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.350+0000 7fbbc11f7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbbc0c2f30 con 0x7fbbbc0abe20 2023-03-03T12:27:03.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.350+0000 7fbbc11f7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbbc0c31d0 con 0x7fbbbc14f9d0 2023-03-03T12:27:03.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.350+0000 7fbbc11f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc1524a0 con 0x7fbbbc14fda0 2023-03-03T12:27:03.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.350+0000 7fbbbb59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbbbc0abe20 0x7fbbbc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:03.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.350+0000 7fbbbb59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbbbc0abe20 0x7fbbbc1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40466/0 (socket says 172.21.15.7:40466) 2023-03-03T12:27:03.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.350+0000 7fbbbb59e700 1 -- 172.21.15.7:0/1656431250 learned_addr learned my addr 172.21.15.7:0/1656431250 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:27:03.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbba59c700 1 --2- 172.21.15.7:0/1656431250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14fda0 0x7fbbbc071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:03.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbbad9d700 1 --2- 172.21.15.7:0/1656431250 >> v2:172.21.15.143:3300/0 conn(0x7fbbbc14f9d0 0x7fbbbc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:03.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbbb59e700 1 -- 172.21.15.7:0/1656431250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14fda0 msgr2=0x7fbbbc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:27:03.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbbb59e700 1 --2- 172.21.15.7:0/1656431250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14fda0 0x7fbbbc071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbbb59e700 1 -- 172.21.15.7:0/1656431250 >> v2:172.21.15.143:3300/0 conn(0x7fbbbc14f9d0 msgr2=0x7fbbbc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:27:03.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbbb59e700 1 --2- 172.21.15.7:0/1656431250 >> v2:172.21.15.143:3300/0 conn(0x7fbbbc14f9d0 0x7fbbbc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbbb59e700 1 -- 172.21.15.7:0/1656431250 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc0c34b0 con 0x7fbbbc0abe20 2023-03-03T12:27:03.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbbad9d700 1 --2- 172.21.15.7:0/1656431250 >> v2:172.21.15.143:3300/0 conn(0x7fbbbc14f9d0 0x7fbbbc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:27:03.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbba59c700 1 --2- 172.21.15.7:0/1656431250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14fda0 0x7fbbbc071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:27:03.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbbb59e700 1 --2- 172.21.15.7:0/1656431250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc0abe20 0x7fbbbc1529c0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0710a0 tx=0x7fbbb400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9904d20032dd5e26 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:03.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbb9d9b700 1 -- 172.21.15.7:0/1656431250 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb400e070 con 0x7fbbbc0abe20 2023-03-03T12:27:03.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbb9d9b700 1 -- 172.21.15.7:0/1656431250 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb400b9c0 con 0x7fbbbc0abe20 2023-03-03T12:27:03.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbb9d9b700 1 -- 172.21.15.7:0/1656431250 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb400bd70 con 0x7fbbbc0abe20 2023-03-03T12:27:03.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.351+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1656431250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc0abe20 msgr2=0x7fbbbc1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:03.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.352+0000 7fbbc11f7700 1 --2- 172.21.15.7:0/1656431250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc0abe20 0x7fbbbc1529c0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0710a0 tx=0x7fbbb400b330 comp rx=0 tx=0).stop 2023-03-03T12:27:03.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.352+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1656431250 shutdown_connections 2023-03-03T12:27:03.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.352+0000 7fbbc11f7700 1 --2- 172.21.15.7:0/1656431250 >> v2:172.21.15.143:3300/0 conn(0x7fbbbc14f9d0 0x7fbbbc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.352+0000 7fbbc11f7700 1 --2- 172.21.15.7:0/1656431250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc0abe20 0x7fbbbc1529c0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.352+0000 7fbbc11f7700 1 --2- 172.21.15.7:0/1656431250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14fda0 0x7fbbbc071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.352+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1656431250 >> 172.21.15.7:0/1656431250 conn(0x7fbbbc0c8900 msgr2=0x7fbbbc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:27:03.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.352+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1656431250 shutdown_connections 2023-03-03T12:27:03.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.352+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1656431250 wait complete. 2023-03-03T12:27:03.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.353+0000 7fbbc11f7700 1 Processor -- start 2023-03-03T12:27:03.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.353+0000 7fbbc11f7700 1 -- start start 2023-03-03T12:27:03.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbc11f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc0abe20 0x7fbbbc0c2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:03.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbc11f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14f9d0 0x7fbbbc1e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:03.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbc11f7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc14fda0 0x7fbbbc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:03.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbc11f7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc0bf7e0 con 0x7fbbbc14fda0 2023-03-03T12:27:03.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbc11f7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc0c2670 con 0x7fbbbc0abe20 2023-03-03T12:27:03.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbc11f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc06d920 con 0x7fbbbc14f9d0 2023-03-03T12:27:03.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbb59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc14fda0 0x7fbbbc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:03.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbb59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc14fda0 0x7fbbbc06ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40474/0 (socket says 172.21.15.7:40474) 2023-03-03T12:27:03.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbb59e700 1 -- 172.21.15.7:0/1780441776 learned_addr learned my addr 172.21.15.7:0/1780441776 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:27:03.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbba59c700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14f9d0 0x7fbbbc1e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:03.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbad9d700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc0abe20 0x7fbbbc0c2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:03.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbb59e700 1 -- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14f9d0 msgr2=0x7fbbbc1e8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:03.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbb59e700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14f9d0 0x7fbbbc1e8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbb59e700 1 -- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc0abe20 msgr2=0x7fbbbc0c2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:03.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbb59e700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc0abe20 0x7fbbbc0c2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbb59e700 1 -- 172.21.15.7:0/1780441776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb4009b30 con 0x7fbbbc14fda0 2023-03-03T12:27:03.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbad9d700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc0abe20 0x7fbbbc0c2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:27:03.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbba59c700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14f9d0 0x7fbbbc1e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:27:03.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.354+0000 7fbbbb59e700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc14fda0 0x7fbbbc06ddb0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400b370 tx=0x7fbbb4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.355+0000 7fbbb37fe700 1 -- 172.21.15.7:0/1780441776 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4004310 con 0x7fbbbc14fda0 2023-03-03T12:27:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.355+0000 7fbbb37fe700 1 -- 172.21.15.7:0/1780441776 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb4004470 con 0x7fbbbc14fda0 2023-03-03T12:27:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.355+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1780441776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbbc0ac350 con 0x7fbbbc14fda0 2023-03-03T12:27:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.355+0000 7fbbb37fe700 1 -- 172.21.15.7:0/1780441776 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4004820 con 0x7fbbbc14fda0 2023-03-03T12:27:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.355+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1780441776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbbc0ab630 con 0x7fbbbc14fda0 2023-03-03T12:27:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.356+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1780441776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbbc0c3e10 con 0x7fbbbc14fda0 2023-03-03T12:27:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.356+0000 7fbbb37fe700 1 -- 172.21.15.7:0/1780441776 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbbb402b070 con 0x7fbbbc14fda0 2023-03-03T12:27:03.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.357+0000 7fbbb37fe700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb9c0682f0 0x7fbb9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:03.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.357+0000 7fbbb37fe700 1 -- 172.21.15.7:0/1780441776 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbbb4087500 con 0x7fbbbc14fda0 2023-03-03T12:27:03.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.357+0000 7fbbbad9d700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb9c0682f0 0x7fbb9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:03.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.357+0000 7fbbbad9d700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb9c0682f0 0x7fbb9c06a7a0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0c2a30 tx=0x7fbbac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:03.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.361+0000 7fbbb37fe700 1 -- 172.21.15.7:0/1780441776 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbbb4053950 con 0x7fbbbc14fda0 2023-03-03T12:27:03.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.603+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1780441776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbbc1e87b0 con 0x7fbbbc14fda0 2023-03-03T12:27:03.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.603+0000 7fbbb37fe700 1 -- 172.21.15.7:0/1780441776 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbbb4056f70 con 0x7fbbbc14fda0 2023-03-03T12:27:03.605 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:27:03.605 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:27:03.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb9c0682f0 msgr2=0x7fbb9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb9c0682f0 0x7fbb9c06a7a0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0c2a30 tx=0x7fbbac008040 comp rx=0 tx=0).stop 2023-03-03T12:27:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1780441776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc14fda0 msgr2=0x7fbbbc06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc14fda0 0x7fbbbc06ddb0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400b370 tx=0x7fbbb4004ad0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1780441776 shutdown_connections 2023-03-03T12:27:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb9c0682f0 0x7fbb9c06a7a0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbbbc14fda0 0x7fbbbc06ddb0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbbc0abe20 0x7fbbbc0c2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 --2- 172.21.15.7:0/1780441776 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbbbc14f9d0 0x7fbbbc1e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:03.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1780441776 >> 172.21.15.7:0/1780441776 conn(0x7fbbbc0c8900 msgr2=0x7fbbbc0c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:27:03.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1780441776 shutdown_connections 2023-03-03T12:27:03.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:03.605+0000 7fbbc11f7700 1 -- 172.21.15.7:0/1780441776 wait complete. 2023-03-03T12:27:03.608 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:27:03.618 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2023-03-03T12:27:25.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:27:25.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.277+0000 7ff9c0017700 1 Processor -- start 2023-03-03T12:27:25.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.277+0000 7ff9c0017700 1 -- start start 2023-03-03T12:27:25.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9c0017700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9b8151ab0 0x7ff9b80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:25.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9c0017700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff9b8073c30 0x7ff9b806bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:25.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9c0017700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b806c3e0 0x7ff9b8157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:25.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9c0017700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b806d780 con 0x7ff9b8151ab0 2023-03-03T12:27:25.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9c0017700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b806cb70 con 0x7ff9b8073c30 2023-03-03T12:27:25.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9c0017700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b806ce50 con 0x7ff9b806c3e0 2023-03-03T12:27:25.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9bddb3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9b8151ab0 0x7ff9b80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:25.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9bddb3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9b8151ab0 0x7ff9b80736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40496/0 (socket says 172.21.15.7:40496) 2023-03-03T12:27:25.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9bddb3700 1 -- 172.21.15.7:0/521753728 learned_addr learned my addr 172.21.15.7:0/521753728 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:27:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9bd5b2700 1 --2- 172.21.15.7:0/521753728 >> v2:172.21.15.143:3300/0 conn(0x7ff9b8073c30 0x7ff9b806bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.278+0000 7ff9be5b4700 1 --2- 172.21.15.7:0/521753728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b806c3e0 0x7ff9b8157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9bddb3700 1 -- 172.21.15.7:0/521753728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b806c3e0 msgr2=0x7ff9b8157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:27:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9bddb3700 1 --2- 172.21.15.7:0/521753728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b806c3e0 0x7ff9b8157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9bddb3700 1 -- 172.21.15.7:0/521753728 >> v2:172.21.15.143:3300/0 conn(0x7ff9b8073c30 msgr2=0x7ff9b806bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:27:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9bddb3700 1 --2- 172.21.15.7:0/521753728 >> v2:172.21.15.143:3300/0 conn(0x7ff9b8073c30 0x7ff9b806bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9bddb3700 1 -- 172.21.15.7:0/521753728 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b806d130 con 0x7ff9b8151ab0 2023-03-03T12:27:25.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9bd5b2700 1 --2- 172.21.15.7:0/521753728 >> v2:172.21.15.143:3300/0 conn(0x7ff9b8073c30 0x7ff9b806bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:27:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9bddb3700 1 --2- 172.21.15.7:0/521753728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8151ab0 0x7ff9b80736f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80701f0 tx=0x7ff9a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41b462964cd825 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9bcdb1700 1 -- 172.21.15.7:0/521753728 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a800b870 con 0x7ff9b8151ab0 2023-03-03T12:27:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9bcdb1700 1 -- 172.21.15.7:0/521753728 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a800b9d0 con 0x7ff9b8151ab0 2023-03-03T12:27:25.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9bcdb1700 1 -- 172.21.15.7:0/521753728 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a800bd60 con 0x7ff9b8151ab0 2023-03-03T12:27:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9c0017700 1 -- 172.21.15.7:0/521753728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8151ab0 msgr2=0x7ff9b80736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9c0017700 1 --2- 172.21.15.7:0/521753728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8151ab0 0x7ff9b80736f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80701f0 tx=0x7ff9a8009bd0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9c0017700 1 -- 172.21.15.7:0/521753728 shutdown_connections 2023-03-03T12:27:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9c0017700 1 --2- 172.21.15.7:0/521753728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b806c3e0 0x7ff9b8157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9c0017700 1 --2- 172.21.15.7:0/521753728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8151ab0 0x7ff9b80736f0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9c0017700 1 --2- 172.21.15.7:0/521753728 >> v2:172.21.15.143:3300/0 conn(0x7ff9b8073c30 0x7ff9b806bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.279+0000 7ff9c0017700 1 -- 172.21.15.7:0/521753728 >> 172.21.15.7:0/521753728 conn(0x7ff9b80a94d0 msgr2=0x7ff9b80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:27:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.280+0000 7ff9c0017700 1 -- 172.21.15.7:0/521753728 shutdown_connections 2023-03-03T12:27:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.280+0000 7ff9c0017700 1 -- 172.21.15.7:0/521753728 wait complete. 2023-03-03T12:27:25.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.281+0000 7ff9c0017700 1 Processor -- start 2023-03-03T12:27:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.281+0000 7ff9c0017700 1 -- start start 2023-03-03T12:27:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9c0017700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b806c3e0 0x7ff9b81dea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9c0017700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 0x7ff9b814e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9c0017700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8151ab0 0x7ff9b81df190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9c0017700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b814f640 con 0x7ff9b8073c30 2023-03-03T12:27:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9c0017700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b81ddc90 con 0x7ff9b806c3e0 2023-03-03T12:27:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9c0017700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b81ddf70 con 0x7ff9b8151ab0 2023-03-03T12:27:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9bd5b2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 0x7ff9b814e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9bd5b2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 0x7ff9b814e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40504/0 (socket says 172.21.15.7:40504) 2023-03-03T12:27:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9bd5b2700 1 -- 172.21.15.7:0/1455475631 learned_addr learned my addr 172.21.15.7:0/1455475631 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:27:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9be5b4700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8151ab0 0x7ff9b81df190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9bd5b2700 1 -- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8151ab0 msgr2=0x7ff9b81df190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9bd5b2700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8151ab0 0x7ff9b81df190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9bd5b2700 1 -- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b806c3e0 msgr2=0x7ff9b81dea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9bddb3700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b806c3e0 0x7ff9b81dea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9bd5b2700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b806c3e0 0x7ff9b81dea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9bd5b2700 1 -- 172.21.15.7:0/1455475631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a8009550 con 0x7ff9b8073c30 2023-03-03T12:27:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9bd5b2700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 0x7ff9b814e5a0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400e810 tx=0x7ff9b400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/1455475631 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4013610 con 0x7ff9b8073c30 2023-03-03T12:27:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/1455475631 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b4013770 con 0x7ff9b8073c30 2023-03-03T12:27:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9c0017700 1 -- 172.21.15.7:0/1455475631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b8151e80 con 0x7ff9b8073c30 2023-03-03T12:27:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/1455475631 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4013b20 con 0x7ff9b8073c30 2023-03-03T12:27:25.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.282+0000 7ff9c0017700 1 -- 172.21.15.7:0/1455475631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b806d410 con 0x7ff9b8073c30 2023-03-03T12:27:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.283+0000 7ff9c0017700 1 -- 172.21.15.7:0/1455475631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b8152010 con 0x7ff9b8073c30 2023-03-03T12:27:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.284+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/1455475631 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff9b4023070 con 0x7ff9b8073c30 2023-03-03T12:27:25.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.284+0000 7ff9ae7fc700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 0x7ff99c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.284+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/1455475631 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff9b4085dd0 con 0x7ff9b8073c30 2023-03-03T12:27:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.284+0000 7ff9bddb3700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 0x7ff99c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.285+0000 7ff9bddb3700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 0x7ff99c06a7a0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff9b81df0c0 tx=0x7ff9a8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:25.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.289+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/1455475631 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff9b4052170 con 0x7ff9b8073c30 2023-03-03T12:27:25.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.524+0000 7ff9c0017700 1 -- 172.21.15.7:0/1455475631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9b8074bb0 con 0x7ff9b8073c30 2023-03-03T12:27:25.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.525+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/1455475631 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff9b4055790 con 0x7ff9b8073c30 2023-03-03T12:27:25.527 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:27:25.527 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:27:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 -- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 msgr2=0x7ff99c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 0x7ff99c06a7a0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff9b81df0c0 tx=0x7ff9a8008030 comp rx=0 tx=0).stop 2023-03-03T12:27:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 -- 172.21.15.7:0/1455475631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 msgr2=0x7ff9b814e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 0x7ff9b814e5a0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400e810 tx=0x7ff9b400ebd0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 -- 172.21.15.7:0/1455475631 shutdown_connections 2023-03-03T12:27:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8151ab0 0x7ff9b81df190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b806c3e0 0x7ff9b81dea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 0x7ff99c06a7a0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 --2- 172.21.15.7:0/1455475631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 0x7ff9b814e5a0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 -- 172.21.15.7:0/1455475631 >> 172.21.15.7:0/1455475631 conn(0x7ff9b80a94d0 msgr2=0x7ff9b8071310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:27:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 -- 172.21.15.7:0/1455475631 shutdown_connections 2023-03-03T12:27:25.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:25.527+0000 7ff9c0017700 1 -- 172.21.15.7:0/1455475631 wait complete. 2023-03-03T12:27:25.531 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:27:25.540 INFO:tasks.mds_thrash.fs.[b]:waiting for 94 secs before thrashing 2023-03-03T12:27:28.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:28.368 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:46.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:27:46.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.756+0000 7fbee1edc700 1 Processor -- start 2023-03-03T12:27:46.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.756+0000 7fbee1edc700 1 -- start start 2023-03-03T12:27:46.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbee1edc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbedc0c6010 0x7fbedc0c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:46.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbee1edc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbedc0c4fc0 0x7fbedc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:46.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbee1edc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbedc14ebe0 con 0x7fbedc0c17f0 2023-03-03T12:27:46.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbee1edc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbedc152100 con 0x7fbedc0c6010 2023-03-03T12:27:46.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbee1edc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbedc0c2e70 con 0x7fbedc0c4fc0 2023-03-03T12:27:46.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbedb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbedc0c6010 0x7fbedc0c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:46.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbedaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbedc0c17f0 0x7fbedc06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34504/0 (socket says 172.21.15.7:34504) 2023-03-03T12:27:46.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbedaffd700 1 -- 172.21.15.7:0/2949599837 learned_addr learned my addr 172.21.15.7:0/2949599837 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:27:46.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbedbfff700 1 --2- 172.21.15.7:0/2949599837 >> v2:172.21.15.143:3300/0 conn(0x7fbedc0c4fc0 0x7fbedc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:46.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbedb7fe700 1 -- 172.21.15.7:0/2949599837 >> v2:172.21.15.143:3300/0 conn(0x7fbedc0c4fc0 msgr2=0x7fbedc06ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:27:46.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbedb7fe700 1 --2- 172.21.15.7:0/2949599837 >> v2:172.21.15.143:3300/0 conn(0x7fbedc0c4fc0 0x7fbedc06ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:46.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbedb7fe700 1 -- 172.21.15.7:0/2949599837 >> v1:172.21.15.7:6789/0 conn(0x7fbedc0c17f0 legacy=0x7fbedc06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:46.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbedb7fe700 1 -- 172.21.15.7:0/2949599837 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbedc0aa680 con 0x7fbedc0c6010 2023-03-03T12:27:46.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbeda7fc700 1 -- 172.21.15.7:0/2949599837 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083257446 0 0) 0x7fbedc14ebe0 con 0x7fbedc0c17f0 2023-03-03T12:27:46.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.757+0000 7fbedbfff700 1 --2- 172.21.15.7:0/2949599837 >> v2:172.21.15.143:3300/0 conn(0x7fbedc0c4fc0 0x7fbedc06ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:27:46.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbedb7fe700 1 --2- 172.21.15.7:0/2949599837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c6010 0x7fbedc0c4a80 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fbedc06f490 tx=0x7fbecc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c995cad1ef9c8a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:46.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbeda7fc700 1 -- 172.21.15.7:0/2949599837 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc011830 con 0x7fbedc0c6010 2023-03-03T12:27:46.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbeda7fc700 1 -- 172.21.15.7:0/2949599837 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbecc011990 con 0x7fbedc0c6010 2023-03-03T12:27:46.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbeda7fc700 1 -- 172.21.15.7:0/2949599837 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc011d20 con 0x7fbedc0c6010 2023-03-03T12:27:46.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbee1edc700 1 -- 172.21.15.7:0/2949599837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c6010 msgr2=0x7fbedc0c4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:46.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbee1edc700 1 --2- 172.21.15.7:0/2949599837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c6010 0x7fbedc0c4a80 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fbedc06f490 tx=0x7fbecc009b20 comp rx=0 tx=0).stop 2023-03-03T12:27:46.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbee1edc700 1 -- 172.21.15.7:0/2949599837 shutdown_connections 2023-03-03T12:27:46.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbee1edc700 1 --2- 172.21.15.7:0/2949599837 >> v2:172.21.15.143:3300/0 conn(0x7fbedc0c4fc0 0x7fbedc06ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:46.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbee1edc700 1 --2- 172.21.15.7:0/2949599837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c6010 0x7fbedc0c4a80 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:46.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbee1edc700 1 -- 172.21.15.7:0/2949599837 >> 172.21.15.7:0/2949599837 conn(0x7fbedc0a9430 msgr2=0x7fbedc0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:27:46.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbee1edc700 1 -- 172.21.15.7:0/2949599837 shutdown_connections 2023-03-03T12:27:46.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.758+0000 7fbee1edc700 1 -- 172.21.15.7:0/2949599837 wait complete. 2023-03-03T12:27:46.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.760+0000 7fbee1edc700 1 Processor -- start 2023-03-03T12:27:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.760+0000 7fbee1edc700 1 -- start start 2023-03-03T12:27:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.760+0000 7fbee1edc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c17f0 0x7fbedc155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbee1edc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbedc0c4fc0 0x7fbedc14c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbee1edc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc0c6010 0x7fbedc14c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbee1edc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc153c30 con 0x7fbedc0c17f0 2023-03-03T12:27:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbee1edc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc1ea5b0 con 0x7fbedc0c6010 2023-03-03T12:27:46.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbee1edc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc1e9ca0 con 0x7fbedc0c4fc0 2023-03-03T12:27:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c17f0 0x7fbedc155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c17f0 0x7fbedc155000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40528/0 (socket says 172.21.15.7:40528) 2023-03-03T12:27:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedb7fe700 1 -- 172.21.15.7:0/1967066087 learned_addr learned my addr 172.21.15.7:0/1967066087 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:27:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedbfff700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc0c6010 0x7fbedc14c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedaffd700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbedc0c4fc0 0x7fbedc14c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedb7fe700 1 -- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbedc0c4fc0 msgr2=0x7fbedc14c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedb7fe700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbedc0c4fc0 0x7fbedc14c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:46.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedb7fe700 1 -- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc0c6010 msgr2=0x7fbedc14c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:46.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedb7fe700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc0c6010 0x7fbedc14c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:46.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedb7fe700 1 -- 172.21.15.7:0/1967066087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc0094a0 con 0x7fbedc0c17f0 2023-03-03T12:27:46.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedaffd700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbedc0c4fc0 0x7fbedc14c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:27:46.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbedb7fe700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c17f0 0x7fbedc155000 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fbecc014040 tx=0x7fbecc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:46.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbec3fff700 1 -- 172.21.15.7:0/1967066087 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc0038a0 con 0x7fbedc0c17f0 2023-03-03T12:27:46.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbec3fff700 1 -- 172.21.15.7:0/1967066087 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbecc01e070 con 0x7fbedc0c17f0 2023-03-03T12:27:46.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbee1edc700 1 -- 172.21.15.7:0/1967066087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbedc0c2b00 con 0x7fbedc0c17f0 2023-03-03T12:27:46.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbec3fff700 1 -- 172.21.15.7:0/1967066087 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc003c60 con 0x7fbedc0c17f0 2023-03-03T12:27:46.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.761+0000 7fbee1edc700 1 -- 172.21.15.7:0/1967066087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbedc0aa960 con 0x7fbedc0c17f0 2023-03-03T12:27:46.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.762+0000 7fbee1edc700 1 -- 172.21.15.7:0/1967066087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbedc0c3300 con 0x7fbedc0c17f0 2023-03-03T12:27:46.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.763+0000 7fbec3fff700 1 -- 172.21.15.7:0/1967066087 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbecc00f050 con 0x7fbedc0c17f0 2023-03-03T12:27:46.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.763+0000 7fbec3fff700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbec80682f0 0x7fbec806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:27:46.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.763+0000 7fbec3fff700 1 -- 172.21.15.7:0/1967066087 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbecc087310 con 0x7fbedc0c17f0 2023-03-03T12:27:46.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.763+0000 7fbedaffd700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbec80682f0 0x7fbec806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:27:46.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.764+0000 7fbedaffd700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbec80682f0 0x7fbec806a7a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0a8640 tx=0x7fbed0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:27:46.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:46.768+0000 7fbec3fff700 1 -- 172.21.15.7:0/1967066087 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbecc053630 con 0x7fbedc0c17f0 2023-03-03T12:27:47.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.007+0000 7fbee1edc700 1 -- 172.21.15.7:0/1967066087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbedc074bb0 con 0x7fbedc0c17f0 2023-03-03T12:27:47.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.008+0000 7fbec3fff700 1 -- 172.21.15.7:0/1967066087 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbecc020090 con 0x7fbedc0c17f0 2023-03-03T12:27:47.010 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:27:47.010 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:27:47.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 -- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbec80682f0 msgr2=0x7fbec806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:47.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbec80682f0 0x7fbec806a7a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0a8640 tx=0x7fbed0009040 comp rx=0 tx=0).stop 2023-03-03T12:27:47.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 -- 172.21.15.7:0/1967066087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c17f0 msgr2=0x7fbedc155000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:27:47.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c17f0 0x7fbedc155000 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fbecc014040 tx=0x7fbecc004750 comp rx=0 tx=0).stop 2023-03-03T12:27:47.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 -- 172.21.15.7:0/1967066087 shutdown_connections 2023-03-03T12:27:47.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc0c6010 0x7fbedc14c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:47.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbec80682f0 0x7fbec806a7a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:47.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbedc0c17f0 0x7fbedc155000 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:47.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 --2- 172.21.15.7:0/1967066087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbedc0c4fc0 0x7fbedc14c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:27:47.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 -- 172.21.15.7:0/1967066087 >> 172.21.15.7:0/1967066087 conn(0x7fbedc0a9430 msgr2=0x7fbedc14d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:27:47.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 -- 172.21.15.7:0/1967066087 shutdown_connections 2023-03-03T12:27:47.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:27:47.010+0000 7fbee1edc700 1 -- 172.21.15.7:0/1967066087 wait complete. 2023-03-03T12:27:47.013 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:27:47.023 INFO:tasks.mds_thrash.fs.[a]:waiting for 57 secs before thrashing 2023-03-03T12:27:58.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:27:58.439 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:28.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:28.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:44.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:28:44.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.161+0000 7f3962d8c700 1 Processor -- start 2023-03-03T12:28:44.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.161+0000 7f3962d8c700 1 -- start start 2023-03-03T12:28:44.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f3962d8c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f395c14fda0 0x7f395c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:44.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f3962d8c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f395c0abe20 0x7f395c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:44.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f3962d8c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f395c0c2f30 con 0x7f395c14f9d0 2023-03-03T12:28:44.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f3962d8c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395c0c31d0 con 0x7f395c14fda0 2023-03-03T12:28:44.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f3962d8c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395c1524a0 con 0x7f395c0abe20 2023-03-03T12:28:44.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f395bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f395c14fda0 0x7f395c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:44.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f395bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f395c14fda0 0x7f395c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40598/0 (socket says 172.21.15.7:40598) 2023-03-03T12:28:44.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f395bfff700 1 -- 172.21.15.7:0/901706055 learned_addr learned my addr 172.21.15.7:0/901706055 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:28:44.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f3961329700 1 --2- 172.21.15.7:0/901706055 >> v2:172.21.15.143:3300/0 conn(0x7f395c0abe20 0x7f395c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:44.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f395bfff700 1 -- 172.21.15.7:0/901706055 >> v2:172.21.15.143:3300/0 conn(0x7f395c0abe20 msgr2=0x7f395c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:28:44.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f395bfff700 1 --2- 172.21.15.7:0/901706055 >> v2:172.21.15.143:3300/0 conn(0x7f395c0abe20 0x7f395c1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:44.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f395bfff700 1 -- 172.21.15.7:0/901706055 >> v1:172.21.15.143:6789/0 conn(0x7f395c14f9d0 legacy=0x7f395c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:28:44.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f395bfff700 1 -- 172.21.15.7:0/901706055 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395c0c34b0 con 0x7f395c14fda0 2023-03-03T12:28:44.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.162+0000 7f395bfff700 1 --2- 172.21.15.7:0/901706055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14fda0 0x7f395c156430 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f395c071230 tx=0x7f3948009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e666a2fb1f1a2b6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:28:44.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f395b7fe700 1 -- 172.21.15.7:0/901706055 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948004230 con 0x7f395c14fda0 2023-03-03T12:28:44.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f395b7fe700 1 -- 172.21.15.7:0/901706055 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3948004390 con 0x7f395c14fda0 2023-03-03T12:28:44.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f395b7fe700 1 -- 172.21.15.7:0/901706055 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39480036a0 con 0x7f395c14fda0 2023-03-03T12:28:44.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f3962d8c700 1 -- 172.21.15.7:0/901706055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14fda0 msgr2=0x7f395c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:44.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f3962d8c700 1 --2- 172.21.15.7:0/901706055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14fda0 0x7f395c156430 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f395c071230 tx=0x7f3948009bd0 comp rx=0 tx=0).stop 2023-03-03T12:28:44.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f3962d8c700 1 -- 172.21.15.7:0/901706055 shutdown_connections 2023-03-03T12:28:44.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f3962d8c700 1 --2- 172.21.15.7:0/901706055 >> v2:172.21.15.143:3300/0 conn(0x7f395c0abe20 0x7f395c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:44.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f3962d8c700 1 --2- 172.21.15.7:0/901706055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14fda0 0x7f395c156430 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:44.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f3962d8c700 1 -- 172.21.15.7:0/901706055 >> 172.21.15.7:0/901706055 conn(0x7f395c0c8900 msgr2=0x7f395c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:28:44.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f3962d8c700 1 -- 172.21.15.7:0/901706055 shutdown_connections 2023-03-03T12:28:44.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.163+0000 7f3962d8c700 1 -- 172.21.15.7:0/901706055 wait complete. 2023-03-03T12:28:44.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.164+0000 7f3962d8c700 1 Processor -- start 2023-03-03T12:28:44.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.164+0000 7f3962d8c700 1 -- start start 2023-03-03T12:28:44.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.165+0000 7f3962d8c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f395c0abe20 0x7f395c06bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:44.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.165+0000 7f3962d8c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14f9d0 0x7f395c1edaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:44.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.165+0000 7f3962d8c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f395c14fda0 0x7f395c0c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:44.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.165+0000 7f3962d8c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c06ba90 con 0x7f395c14f9d0 2023-03-03T12:28:44.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.165+0000 7f3962d8c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395c1ed660 con 0x7f395c14fda0 2023-03-03T12:28:44.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.165+0000 7f3962d8c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395c1ddb50 con 0x7f395c0abe20 2023-03-03T12:28:44.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.165+0000 7f395bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14f9d0 0x7f395c1edaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:44.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.165+0000 7f395bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14f9d0 0x7f395c1edaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40606/0 (socket says 172.21.15.7:40606) 2023-03-03T12:28:44.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.165+0000 7f395bfff700 1 -- 172.21.15.7:0/838846124 learned_addr learned my addr 172.21.15.7:0/838846124 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:28:44.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f3960b28700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f395c0abe20 0x7f395c06bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:44.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f395bfff700 1 -- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f395c0abe20 msgr2=0x7f395c06bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:44.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f395bfff700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f395c0abe20 0x7f395c06bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:44.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f395bfff700 1 -- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f395c14fda0 msgr2=0x7f395c0c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:44.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f395bfff700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f395c14fda0 0x7f395c0c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:44.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f395bfff700 1 -- 172.21.15.7:0/838846124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3948009550 con 0x7f395c14f9d0 2023-03-03T12:28:44.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f395bfff700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14f9d0 0x7f395c1edaf0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f3948004000 tx=0x7f3948004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:28:44.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f39597fa700 1 -- 172.21.15.7:0/838846124 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948003d80 con 0x7f395c14f9d0 2023-03-03T12:28:44.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f39597fa700 1 -- 172.21.15.7:0/838846124 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394801f070 con 0x7f395c14f9d0 2023-03-03T12:28:44.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f3962d8c700 1 -- 172.21.15.7:0/838846124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f395c0ac350 con 0x7f395c14f9d0 2023-03-03T12:28:44.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f39597fa700 1 -- 172.21.15.7:0/838846124 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948011870 con 0x7f395c14f9d0 2023-03-03T12:28:44.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.166+0000 7f3962d8c700 1 -- 172.21.15.7:0/838846124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f395c0ab630 con 0x7f395c14f9d0 2023-03-03T12:28:44.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.167+0000 7f3962d8c700 1 -- 172.21.15.7:0/838846124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f395c071fa0 con 0x7f395c14f9d0 2023-03-03T12:28:44.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.167+0000 7f39597fa700 1 -- 172.21.15.7:0/838846124 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3948017020 con 0x7f395c14f9d0 2023-03-03T12:28:44.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.168+0000 7f39597fa700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39400682f0 0x7f394006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:44.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.168+0000 7f39597fa700 1 -- 172.21.15.7:0/838846124 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3948087870 con 0x7f395c14f9d0 2023-03-03T12:28:44.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.168+0000 7f3960b28700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39400682f0 0x7f394006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:44.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.169+0000 7f3960b28700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39400682f0 0x7f394006a7a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3950001670 tx=0x7f3950009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:28:44.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.172+0000 7f39597fa700 1 -- 172.21.15.7:0/838846124 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3948053c10 con 0x7f395c14f9d0 2023-03-03T12:28:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.411+0000 7f3962d8c700 1 -- 172.21.15.7:0/838846124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f395c1ed810 con 0x7f395c14f9d0 2023-03-03T12:28:44.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.412+0000 7f39597fa700 1 -- 172.21.15.7:0/838846124 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f395c1ed810 con 0x7f395c14f9d0 2023-03-03T12:28:44.414 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:28:44.414 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:28:44.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 -- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39400682f0 msgr2=0x7f394006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:44.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39400682f0 0x7f394006a7a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3950001670 tx=0x7f3950009040 comp rx=0 tx=0).stop 2023-03-03T12:28:44.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 -- 172.21.15.7:0/838846124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14f9d0 msgr2=0x7f395c1edaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:44.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14f9d0 0x7f395c1edaf0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f3948004000 tx=0x7f3948004750 comp rx=0 tx=0).stop 2023-03-03T12:28:44.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 -- 172.21.15.7:0/838846124 shutdown_connections 2023-03-03T12:28:44.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f395c14fda0 0x7f395c0c17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:44.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f395c0abe20 0x7f395c06bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:44.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39400682f0 0x7f394006a7a0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:44.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 --2- 172.21.15.7:0/838846124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f395c14f9d0 0x7f395c1edaf0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:44.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 -- 172.21.15.7:0/838846124 >> 172.21.15.7:0/838846124 conn(0x7f395c0c8900 msgr2=0x7f395c0c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:28:44.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 -- 172.21.15.7:0/838846124 shutdown_connections 2023-03-03T12:28:44.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:44.414+0000 7f3962d8c700 1 -- 172.21.15.7:0/838846124 wait complete. 2023-03-03T12:28:44.418 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:28:44.428 INFO:tasks.mds_thrash.fs.[a]:waiting for 94 secs before thrashing 2023-03-03T12:28:58.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:58.735 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:28:59.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:28:59.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.678+0000 7fe8f62be700 1 Processor -- start 2023-03-03T12:28:59.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.678+0000 7fe8f62be700 1 -- start start 2023-03-03T12:28:59.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.679+0000 7fe8f62be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe8f006a2b0 0x7fe8f006a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.679+0000 7fe8f62be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f0158030 0x7fe8f01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.679+0000 7fe8f62be700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f00c1c10 con 0x7fe8f0151c60 2023-03-03T12:28:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.679+0000 7fe8f62be700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f00c26b0 con 0x7fe8f006a2b0 2023-03-03T12:28:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.679+0000 7fe8f62be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f01555c0 con 0x7fe8f0158030 2023-03-03T12:28:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.679+0000 7fe8f485b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f0158030 0x7fe8f01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:59.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.679+0000 7fe8ef7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe8f006a2b0 0x7fe8f006a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:59.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.679+0000 7fe8f485b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f0158030 0x7fe8f01584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:48688/0 (socket says 172.21.15.7:48688) 2023-03-03T12:28:59.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.679+0000 7fe8f485b700 1 -- 172.21.15.7:0/2578662293 learned_addr learned my addr 172.21.15.7:0/2578662293 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:28:59.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f485b700 1 -- 172.21.15.7:0/2578662293 >> v2:172.21.15.143:3300/0 conn(0x7fe8f006a2b0 msgr2=0x7fe8f006a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:28:59.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f485b700 1 --2- 172.21.15.7:0/2578662293 >> v2:172.21.15.143:3300/0 conn(0x7fe8f006a2b0 0x7fe8f006a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:59.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f485b700 1 -- 172.21.15.7:0/2578662293 >> v1:172.21.15.143:6789/0 conn(0x7fe8f0151c60 legacy=0x7fe8f00c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:59.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f485b700 1 -- 172.21.15.7:0/2578662293 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f0152030 con 0x7fe8f0158030 2023-03-03T12:28:59.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8ef7fe700 1 --2- 172.21.15.7:0/2578662293 >> v2:172.21.15.143:3300/0 conn(0x7fe8f006a2b0 0x7fe8f006a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:28:59.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f485b700 1 --2- 172.21.15.7:0/2578662293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f0158030 0x7fe8f01584b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8009e50 tx=0x7fe8e800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d354337cacb300bf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:28:59.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8eeffd700 1 -- 172.21.15.7:0/2578662293 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e800e070 con 0x7fe8f0158030 2023-03-03T12:28:59.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8eeffd700 1 -- 172.21.15.7:0/2578662293 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e800b910 con 0x7fe8f0158030 2023-03-03T12:28:59.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8eeffd700 1 -- 172.21.15.7:0/2578662293 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e800bcc0 con 0x7fe8f0158030 2023-03-03T12:28:59.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f62be700 1 -- 172.21.15.7:0/2578662293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f0158030 msgr2=0x7fe8f01584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:59.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f62be700 1 --2- 172.21.15.7:0/2578662293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f0158030 0x7fe8f01584b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8009e50 tx=0x7fe8e800b330 comp rx=0 tx=0).stop 2023-03-03T12:28:59.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f62be700 1 -- 172.21.15.7:0/2578662293 shutdown_connections 2023-03-03T12:28:59.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f62be700 1 --2- 172.21.15.7:0/2578662293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f0158030 0x7fe8f01584b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:59.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f62be700 1 --2- 172.21.15.7:0/2578662293 >> v2:172.21.15.143:3300/0 conn(0x7fe8f006a2b0 0x7fe8f006a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:59.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.680+0000 7fe8f62be700 1 -- 172.21.15.7:0/2578662293 >> 172.21.15.7:0/2578662293 conn(0x7fe8f00a9430 msgr2=0x7fe8f00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:28:59.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.681+0000 7fe8f62be700 1 -- 172.21.15.7:0/2578662293 shutdown_connections 2023-03-03T12:28:59.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.681+0000 7fe8f62be700 1 -- 172.21.15.7:0/2578662293 wait complete. 2023-03-03T12:28:59.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.682+0000 7fe8f62be700 1 Processor -- start 2023-03-03T12:28:59.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.682+0000 7fe8f62be700 1 -- start start 2023-03-03T12:28:59.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f62be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f006a2b0 0x7fe8f006fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:59.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f62be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8f0151c60 0x7fe8f01ed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:59.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f62be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f0158030 0x7fe8f0153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:59.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f62be700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f01524c0 con 0x7fe8f0158030 2023-03-03T12:28:59.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f62be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f0152a80 con 0x7fe8f0151c60 2023-03-03T12:28:59.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f62be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f01edce0 con 0x7fe8f006a2b0 2023-03-03T12:28:59.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f485b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f0158030 0x7fe8f0153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:59.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f485b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f0158030 0x7fe8f0153f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40626/0 (socket says 172.21.15.7:40626) 2023-03-03T12:28:59.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f485b700 1 -- 172.21.15.7:0/3151716169 learned_addr learned my addr 172.21.15.7:0/3151716169 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:28:59.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8ef7fe700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8f0151c60 0x7fe8f01ed4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:59.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8effff700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f006a2b0 0x7fe8f006fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:59.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f485b700 1 -- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f006a2b0 msgr2=0x7fe8f006fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:59.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f485b700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f006a2b0 0x7fe8f006fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:59.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f485b700 1 -- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8f0151c60 msgr2=0x7fe8f01ed4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:59.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f485b700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8f0151c60 0x7fe8f01ed4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:59.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f485b700 1 -- 172.21.15.7:0/3151716169 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e00094a0 con 0x7fe8f0158030 2023-03-03T12:28:59.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f485b700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f0158030 0x7fe8f0153f30 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fe8f00668d0 tx=0x7fe8e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:28:59.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8ecff9700 1 -- 172.21.15.7:0/3151716169 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8004960 con 0x7fe8f0158030 2023-03-03T12:28:59.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8ecff9700 1 -- 172.21.15.7:0/3151716169 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e80099b0 con 0x7fe8f0158030 2023-03-03T12:28:59.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8f62be700 1 -- 172.21.15.7:0/3151716169 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e8009b30 con 0x7fe8f0158030 2023-03-03T12:28:59.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.683+0000 7fe8ecff9700 1 -- 172.21.15.7:0/3151716169 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e801c680 con 0x7fe8f0158030 2023-03-03T12:28:59.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.684+0000 7fe8f62be700 1 -- 172.21.15.7:0/3151716169 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f01589f0 con 0x7fe8f0158030 2023-03-03T12:28:59.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.684+0000 7fe8f62be700 1 -- 172.21.15.7:0/3151716169 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f0157b30 con 0x7fe8f0158030 2023-03-03T12:28:59.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.685+0000 7fe8ecff9700 1 -- 172.21.15.7:0/3151716169 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe8e802c070 con 0x7fe8f0158030 2023-03-03T12:28:59.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.685+0000 7fe8ecff9700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8d00682f0 0x7fe8d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:28:59.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.685+0000 7fe8ecff9700 1 -- 172.21.15.7:0/3151716169 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe8e8087c90 con 0x7fe8f0158030 2023-03-03T12:28:59.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.685+0000 7fe8effff700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8d00682f0 0x7fe8d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:28:59.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.686+0000 7fe8effff700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8d00682f0 0x7fe8d006a7a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fe8f01572b0 tx=0x7fe8d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:28:59.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.690+0000 7fe8ecff9700 1 -- 172.21.15.7:0/3151716169 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8e8054030 con 0x7fe8f0158030 2023-03-03T12:28:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.929+0000 7fe8f62be700 1 -- 172.21.15.7:0/3151716169 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8f00c4a80 con 0x7fe8f0158030 2023-03-03T12:28:59.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.930+0000 7fe8ecff9700 1 -- 172.21.15.7:0/3151716169 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe8e8057650 con 0x7fe8f0158030 2023-03-03T12:28:59.932 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:28:59.932 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:28:59.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.931+0000 7fe8f62be700 1 -- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8d00682f0 msgr2=0x7fe8d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:59.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.931+0000 7fe8f62be700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8d00682f0 0x7fe8d006a7a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fe8f01572b0 tx=0x7fe8d8009040 comp rx=0 tx=0).stop 2023-03-03T12:28:59.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.932+0000 7fe8f62be700 1 -- 172.21.15.7:0/3151716169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f0158030 msgr2=0x7fe8f0153f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:28:59.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.932+0000 7fe8f62be700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f0158030 0x7fe8f0153f30 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fe8f00668d0 tx=0x7fe8e8004ad0 comp rx=0 tx=0).stop 2023-03-03T12:28:59.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.932+0000 7fe8f62be700 1 -- 172.21.15.7:0/3151716169 shutdown_connections 2023-03-03T12:28:59.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.932+0000 7fe8f62be700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8d00682f0 0x7fe8d006a7a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:59.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.932+0000 7fe8f62be700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f0158030 0x7fe8f0153f30 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:59.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.932+0000 7fe8f62be700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8f006a2b0 0x7fe8f006fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:59.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.932+0000 7fe8f62be700 1 --2- 172.21.15.7:0/3151716169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8f0151c60 0x7fe8f01ed4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:28:59.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.932+0000 7fe8f62be700 1 -- 172.21.15.7:0/3151716169 >> 172.21.15.7:0/3151716169 conn(0x7fe8f00a9430 msgr2=0x7fe8f00a9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:28:59.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.932+0000 7fe8f62be700 1 -- 172.21.15.7:0/3151716169 shutdown_connections 2023-03-03T12:28:59.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:28:59.932+0000 7fe8f62be700 1 -- 172.21.15.7:0/3151716169 wait complete. 2023-03-03T12:28:59.935 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:28:59.945 INFO:tasks.mds_thrash.fs.[b]:waiting for 94 secs before thrashing 2023-03-03T12:29:28.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:28.863 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:58.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:29:59.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T12:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T12:30:18.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:30:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.566+0000 7f90c3cf1700 1 Processor -- start 2023-03-03T12:30:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.567+0000 7f90c3cf1700 1 -- start start 2023-03-03T12:30:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.567+0000 7f90c3cf1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f90bc155410 0x7f90bc0bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.567+0000 7f90c3cf1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f90bc0c5b00 0x7f90bc0bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.567+0000 7f90c3cf1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90bc0c08d0 con 0x7f90bc151e80 2023-03-03T12:30:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.567+0000 7f90c3cf1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc152250 con 0x7f90bc0c5b00 2023-03-03T12:30:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.567+0000 7f90c3cf1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc0c0fb0 con 0x7f90bc155410 2023-03-03T12:30:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.567+0000 7f90c228e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f90bc0c5b00 0x7f90bc0bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.567+0000 7f90c1a8d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f90bc151e80 0x7f90bc0c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34690/0 (socket says 172.21.15.7:34690) 2023-03-03T12:30:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c1a8d700 1 -- 172.21.15.7:0/687545534 learned_addr learned my addr 172.21.15.7:0/687545534 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:30:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c128c700 1 --2- 172.21.15.7:0/687545534 >> v2:172.21.15.143:3300/0 conn(0x7f90bc155410 0x7f90bc0bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c0a8b700 1 -- 172.21.15.7:0/687545534 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328049199 0 0) 0x7f90bc0c08d0 con 0x7f90bc151e80 2023-03-03T12:30:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c0a8b700 1 -- 172.21.15.7:0/687545534 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a80035b0 con 0x7f90bc151e80 2023-03-03T12:30:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c228e700 1 -- 172.21.15.7:0/687545534 >> v2:172.21.15.143:3300/0 conn(0x7f90bc155410 msgr2=0x7f90bc0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:30:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c228e700 1 --2- 172.21.15.7:0/687545534 >> v2:172.21.15.143:3300/0 conn(0x7f90bc155410 0x7f90bc0bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c228e700 1 -- 172.21.15.7:0/687545534 >> v1:172.21.15.7:6789/0 conn(0x7f90bc151e80 legacy=0x7f90bc0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c228e700 1 -- 172.21.15.7:0/687545534 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc154930 con 0x7f90bc0c5b00 2023-03-03T12:30:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c228e700 1 --2- 172.21.15.7:0/687545534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 0x7f90bc0bfe20 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f90bc06c380 tx=0x7f90b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa51fd53308e4fb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c0a8b700 1 -- 172.21.15.7:0/687545534 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b400e070 con 0x7f90bc0c5b00 2023-03-03T12:30:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c0a8b700 1 -- 172.21.15.7:0/687545534 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b400bb20 con 0x7f90bc0c5b00 2023-03-03T12:30:18.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.568+0000 7f90c0a8b700 1 -- 172.21.15.7:0/687545534 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4012430 con 0x7f90bc0c5b00 2023-03-03T12:30:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.569+0000 7f90c3cf1700 1 -- 172.21.15.7:0/687545534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 msgr2=0x7f90bc0bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.569+0000 7f90c3cf1700 1 --2- 172.21.15.7:0/687545534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 0x7f90bc0bfe20 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f90bc06c380 tx=0x7f90b400b330 comp rx=0 tx=0).stop 2023-03-03T12:30:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.569+0000 7f90c3cf1700 1 -- 172.21.15.7:0/687545534 shutdown_connections 2023-03-03T12:30:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.569+0000 7f90c3cf1700 1 --2- 172.21.15.7:0/687545534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 0x7f90bc0bfe20 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.569+0000 7f90c3cf1700 1 --2- 172.21.15.7:0/687545534 >> v2:172.21.15.143:3300/0 conn(0x7f90bc155410 0x7f90bc0bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.569+0000 7f90c3cf1700 1 -- 172.21.15.7:0/687545534 >> 172.21.15.7:0/687545534 conn(0x7f90bc071630 msgr2=0x7f90bc06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:30:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.569+0000 7f90c3cf1700 1 -- 172.21.15.7:0/687545534 shutdown_connections 2023-03-03T12:30:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.569+0000 7f90c3cf1700 1 -- 172.21.15.7:0/687545534 wait complete. 2023-03-03T12:30:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.570+0000 7f90c3cf1700 1 Processor -- start 2023-03-03T12:30:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.570+0000 7f90c3cf1700 1 -- start start 2023-03-03T12:30:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c3cf1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 0x7f90bc1de4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c3cf1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90bc151e80 0x7f90bc0c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c3cf1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc155410 0x7f90bc1e0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c3cf1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc1dea00 con 0x7f90bc0c5b00 2023-03-03T12:30:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c3cf1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc1e0c30 con 0x7f90bc155410 2023-03-03T12:30:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c3cf1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc0ca240 con 0x7f90bc151e80 2023-03-03T12:30:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c1a8d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 0x7f90bc1de4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c1a8d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 0x7f90bc1de4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40716/0 (socket says 172.21.15.7:40716) 2023-03-03T12:30:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c1a8d700 1 -- 172.21.15.7:0/172697379 learned_addr learned my addr 172.21.15.7:0/172697379 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:30:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c228e700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc155410 0x7f90bc1e0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c128c700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90bc151e80 0x7f90bc0c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c1a8d700 1 -- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90bc151e80 msgr2=0x7f90bc0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c1a8d700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90bc151e80 0x7f90bc0c8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c1a8d700 1 -- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc155410 msgr2=0x7f90bc1e0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c1a8d700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc155410 0x7f90bc1e0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c1a8d700 1 -- 172.21.15.7:0/172697379 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b4009a70 con 0x7f90bc0c5b00 2023-03-03T12:30:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c228e700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc155410 0x7f90bc1e0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:30:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.571+0000 7f90c1a8d700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 0x7f90bc1de4c0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f90b800c700 tx=0x7f90b800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.572+0000 7f90ae7fc700 1 -- 172.21.15.7:0/172697379 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b8005d70 con 0x7f90bc0c5b00 2023-03-03T12:30:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.572+0000 7f90ae7fc700 1 -- 172.21.15.7:0/172697379 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b80070d0 con 0x7f90bc0c5b00 2023-03-03T12:30:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.572+0000 7f90ae7fc700 1 -- 172.21.15.7:0/172697379 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b8007460 con 0x7f90bc0c5b00 2023-03-03T12:30:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.572+0000 7f90c3cf1700 1 -- 172.21.15.7:0/172697379 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90bc154ef0 con 0x7f90bc0c5b00 2023-03-03T12:30:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.572+0000 7f90c3cf1700 1 -- 172.21.15.7:0/172697379 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90bc154c10 con 0x7f90bc0c5b00 2023-03-03T12:30:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.573+0000 7f90c3cf1700 1 -- 172.21.15.7:0/172697379 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc0c5220 con 0x7f90bc0c5b00 2023-03-03T12:30:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.573+0000 7f90ae7fc700 1 -- 172.21.15.7:0/172697379 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f90b8019030 con 0x7f90bc0c5b00 2023-03-03T12:30:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.573+0000 7f90ae7fc700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a00682f0 0x7f90a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.574+0000 7f90ae7fc700 1 -- 172.21.15.7:0/172697379 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f90b8085690 con 0x7f90bc0c5b00 2023-03-03T12:30:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.574+0000 7f90c128c700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a00682f0 0x7f90a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.574+0000 7f90c128c700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a00682f0 0x7f90a006a7a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f90bc14d650 tx=0x7f90b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.578+0000 7f90ae7fc700 1 -- 172.21.15.7:0/172697379 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f90b80519b0 con 0x7f90bc0c5b00 2023-03-03T12:30:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.821+0000 7f90c3cf1700 1 -- 172.21.15.7:0/172697379 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90bc074bb0 con 0x7f90bc0c5b00 2023-03-03T12:30:18.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.822+0000 7f90ae7fc700 1 -- 172.21.15.7:0/172697379 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f90b8054fd0 con 0x7f90bc0c5b00 2023-03-03T12:30:18.824 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:30:18.824 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:30:18.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 -- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a00682f0 msgr2=0x7f90a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:18.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a00682f0 0x7f90a006a7a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f90bc14d650 tx=0x7f90b0008040 comp rx=0 tx=0).stop 2023-03-03T12:30:18.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 -- 172.21.15.7:0/172697379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 msgr2=0x7f90bc1de4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:18.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 0x7f90bc1de4c0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f90b800c700 tx=0x7f90b800cac0 comp rx=0 tx=0).stop 2023-03-03T12:30:18.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 -- 172.21.15.7:0/172697379 shutdown_connections 2023-03-03T12:30:18.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90bc155410 0x7f90bc1e0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:18.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90a00682f0 0x7f90a006a7a0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90bc0c5b00 0x7f90bc1de4c0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 --2- 172.21.15.7:0/172697379 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90bc151e80 0x7f90bc0c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 -- 172.21.15.7:0/172697379 >> 172.21.15.7:0/172697379 conn(0x7f90bc071630 msgr2=0x7f90bc0c3100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:30:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 -- 172.21.15.7:0/172697379 shutdown_connections 2023-03-03T12:30:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:18.824+0000 7f90c3cf1700 1 -- 172.21.15.7:0/172697379 wait complete. 2023-03-03T12:30:18.827 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:30:18.837 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before thrashing 2023-03-03T12:30:23.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:30:23.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.975+0000 7f4a46f9e700 1 Processor -- start 2023-03-03T12:30:23.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.975+0000 7f4a46f9e700 1 -- start start 2023-03-03T12:30:23.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a46f9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a400ab0c0 0x7f4a400c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:23.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a46f9e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4a4006a2b0 0x7f4a4006a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a46f9e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4a40157a40 0x7f4a40157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a46f9e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a40154530 con 0x7f4a4006a2b0 2023-03-03T12:30:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a46f9e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a40154810 con 0x7f4a40157a40 2023-03-03T12:30:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a46f9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a40154af0 con 0x7f4a400ab0c0 2023-03-03T12:30:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a3ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4a4006a2b0 0x7f4a4006a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a3ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4a4006a2b0 0x7f4a4006a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40726/0 (socket says 172.21.15.7:40726) 2023-03-03T12:30:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a3ffff700 1 -- 172.21.15.7:0/1203544797 learned_addr learned my addr 172.21.15.7:0/1203544797 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:30:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a4553b700 1 --2- 172.21.15.7:0/1203544797 >> v2:172.21.15.143:3300/0 conn(0x7f4a40157a40 0x7f4a40157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a3ffff700 1 -- 172.21.15.7:0/1203544797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a400ab0c0 msgr2=0x7f4a400c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:30:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a44d3a700 1 --2- 172.21.15.7:0/1203544797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a400ab0c0 0x7f4a400c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a3ffff700 1 --2- 172.21.15.7:0/1203544797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a400ab0c0 0x7f4a400c1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a3ffff700 1 -- 172.21.15.7:0/1203544797 >> v2:172.21.15.143:3300/0 conn(0x7f4a40157a40 msgr2=0x7f4a40157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:30:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a3ffff700 1 --2- 172.21.15.7:0/1203544797 >> v2:172.21.15.143:3300/0 conn(0x7f4a40157a40 0x7f4a40157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a3ffff700 1 -- 172.21.15.7:0/1203544797 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a40154dd0 con 0x7f4a4006a2b0 2023-03-03T12:30:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a4553b700 1 --2- 172.21.15.7:0/1203544797 >> v2:172.21.15.143:3300/0 conn(0x7f4a40157a40 0x7f4a40157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:30:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.976+0000 7f4a3ffff700 1 --2- 172.21.15.7:0/1203544797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 0x7f4a4006a720 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f4a400c2540 tx=0x7f4a2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29058f89b44bcc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a3f7fe700 1 -- 172.21.15.7:0/1203544797 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00b870 con 0x7f4a4006a2b0 2023-03-03T12:30:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a3f7fe700 1 -- 172.21.15.7:0/1203544797 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c00b9d0 con 0x7f4a4006a2b0 2023-03-03T12:30:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a3f7fe700 1 -- 172.21.15.7:0/1203544797 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00bd60 con 0x7f4a4006a2b0 2023-03-03T12:30:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a46f9e700 1 -- 172.21.15.7:0/1203544797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 msgr2=0x7f4a4006a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a46f9e700 1 --2- 172.21.15.7:0/1203544797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 0x7f4a4006a720 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f4a400c2540 tx=0x7f4a2c009bd0 comp rx=0 tx=0).stop 2023-03-03T12:30:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a46f9e700 1 -- 172.21.15.7:0/1203544797 shutdown_connections 2023-03-03T12:30:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a46f9e700 1 --2- 172.21.15.7:0/1203544797 >> v2:172.21.15.143:3300/0 conn(0x7f4a40157a40 0x7f4a40157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a46f9e700 1 --2- 172.21.15.7:0/1203544797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a400ab0c0 0x7f4a400c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a46f9e700 1 --2- 172.21.15.7:0/1203544797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 0x7f4a4006a720 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a46f9e700 1 -- 172.21.15.7:0/1203544797 >> 172.21.15.7:0/1203544797 conn(0x7f4a400a9430 msgr2=0x7f4a400c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:30:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a46f9e700 1 -- 172.21.15.7:0/1203544797 shutdown_connections 2023-03-03T12:30:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.977+0000 7f4a46f9e700 1 -- 172.21.15.7:0/1203544797 wait complete. 2023-03-03T12:30:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.978+0000 7f4a46f9e700 1 Processor -- start 2023-03-03T12:30:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.978+0000 7f4a46f9e700 1 -- start start 2023-03-03T12:30:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.979+0000 7f4a46f9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 0x7f4a400a77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.979+0000 7f4a46f9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a400ab0c0 0x7f4a400c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.979+0000 7f4a46f9e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a40157a40 0x7f4a400c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.979+0000 7f4a46f9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4006fe70 con 0x7f4a4006a2b0 2023-03-03T12:30:23.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.979+0000 7f4a46f9e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a401560e0 con 0x7f4a40157a40 2023-03-03T12:30:23.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.979+0000 7f4a46f9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a40156d50 con 0x7f4a400ab0c0 2023-03-03T12:30:23.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.979+0000 7f4a44d3a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 0x7f4a400a77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:23.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.979+0000 7f4a44d3a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 0x7f4a400a77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40732/0 (socket says 172.21.15.7:40732) 2023-03-03T12:30:23.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.979+0000 7f4a44d3a700 1 -- 172.21.15.7:0/3869083313 learned_addr learned my addr 172.21.15.7:0/3869083313 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:30:23.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a4553b700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a40157a40 0x7f4a400c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:23.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a3ffff700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a400ab0c0 0x7f4a400c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:23.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a44d3a700 1 -- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a400ab0c0 msgr2=0x7f4a400c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a44d3a700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a400ab0c0 0x7f4a400c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a44d3a700 1 -- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a40157a40 msgr2=0x7f4a400c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a44d3a700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a40157a40 0x7f4a400c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a44d3a700 1 -- 172.21.15.7:0/3869083313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c009550 con 0x7f4a4006a2b0 2023-03-03T12:30:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a4553b700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a40157a40 0x7f4a400c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:30:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a44d3a700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 0x7f4a400a77e0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400c7f0 tx=0x7f4a3400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a3d7fa700 1 -- 172.21.15.7:0/3869083313 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34011800 con 0x7f4a4006a2b0 2023-03-03T12:30:23.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a3d7fa700 1 -- 172.21.15.7:0/3869083313 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a34011960 con 0x7f4a4006a2b0 2023-03-03T12:30:23.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a46f9e700 1 -- 172.21.15.7:0/3869083313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a400ab490 con 0x7f4a4006a2b0 2023-03-03T12:30:23.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a3d7fa700 1 -- 172.21.15.7:0/3869083313 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34011d10 con 0x7f4a4006a2b0 2023-03-03T12:30:23.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.980+0000 7f4a46f9e700 1 -- 172.21.15.7:0/3869083313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a401550b0 con 0x7f4a4006a2b0 2023-03-03T12:30:23.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.981+0000 7f4a46f9e700 1 -- 172.21.15.7:0/3869083313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a400c31c0 con 0x7f4a4006a2b0 2023-03-03T12:30:23.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.982+0000 7f4a3d7fa700 1 -- 172.21.15.7:0/3869083313 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4a34021070 con 0x7f4a4006a2b0 2023-03-03T12:30:23.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.982+0000 7f4a3d7fa700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a240682f0 0x7f4a2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:23.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.982+0000 7f4a3d7fa700 1 -- 172.21.15.7:0/3869083313 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4a34086050 con 0x7f4a4006a2b0 2023-03-03T12:30:23.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.982+0000 7f4a3ffff700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a240682f0 0x7f4a2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:23.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.983+0000 7f4a3ffff700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a240682f0 0x7f4a2406a7a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c014040 tx=0x7f4a2c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:23.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:23.987+0000 7f4a3d7fa700 1 -- 172.21.15.7:0/3869083313 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4a340523f0 con 0x7f4a4006a2b0 2023-03-03T12:30:24.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.222+0000 7f4a46f9e700 1 -- 172.21.15.7:0/3869083313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a40074bb0 con 0x7f4a4006a2b0 2023-03-03T12:30:24.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.222+0000 7f4a3d7fa700 1 -- 172.21.15.7:0/3869083313 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4a34055a10 con 0x7f4a4006a2b0 2023-03-03T12:30:24.225 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:30:24.225 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:30:24.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 -- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a240682f0 msgr2=0x7f4a2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a240682f0 0x7f4a2406a7a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c014040 tx=0x7f4a2c008030 comp rx=0 tx=0).stop 2023-03-03T12:30:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 -- 172.21.15.7:0/3869083313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 msgr2=0x7f4a400a77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 0x7f4a400a77e0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400c7f0 tx=0x7f4a3400cbb0 comp rx=0 tx=0).stop 2023-03-03T12:30:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 -- 172.21.15.7:0/3869083313 shutdown_connections 2023-03-03T12:30:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a40157a40 0x7f4a400c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a240682f0 0x7f4a2406a7a0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a4006a2b0 0x7f4a400a77e0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 --2- 172.21.15.7:0/3869083313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a400ab0c0 0x7f4a400c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 -- 172.21.15.7:0/3869083313 >> 172.21.15.7:0/3869083313 conn(0x7f4a400a9430 msgr2=0x7f4a400c1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:30:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 -- 172.21.15.7:0/3869083313 shutdown_connections 2023-03-03T12:30:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:24.225+0000 7f4a46f9e700 1 -- 172.21.15.7:0/3869083313 wait complete. 2023-03-03T12:30:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:30:24.238 INFO:tasks.mds_thrash.fs.[a]:waiting for 117 secs before thrashing 2023-03-03T12:30:29.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:29.156 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:33.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:30:34.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.082+0000 7ffbe86a3700 1 Processor -- start 2023-03-03T12:30:34.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.082+0000 7ffbe86a3700 1 -- start start 2023-03-03T12:30:34.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.083+0000 7ffbe86a3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe014fda0 0x7ffbe0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:34.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.083+0000 7ffbe86a3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe00c2f30 con 0x7ffbe014f9d0 2023-03-03T12:30:34.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.083+0000 7ffbe86a3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe00c31d0 con 0x7ffbe00abe20 2023-03-03T12:30:34.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.083+0000 7ffbe86a3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe01524a0 con 0x7ffbe014fda0 2023-03-03T12:30:34.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.083+0000 7ffbe643f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ffbe014f9d0 0x7ffbe0071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34734/0 (socket says 172.21.15.7:34734) 2023-03-03T12:30:34.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.083+0000 7ffbe643f700 1 -- 172.21.15.7:0/3058175211 learned_addr learned my addr 172.21.15.7:0/3058175211 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:30:34.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.083+0000 7ffbe5c3e700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe014fda0 0x7ffbe0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:34.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.083+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685614045 0 0) 0x7ffbe00c2f30 con 0x7ffbe014f9d0 2023-03-03T12:30:34.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.083+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc0035b0 con 0x7ffbe014f9d0 2023-03-03T12:30:34.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639029936 0 0) 0x7ffbe00c31d0 con 0x7ffbe00abe20 2023-03-03T12:30:34.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbe00c2f30 con 0x7ffbe00abe20 2023-03-03T12:30:34.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1956200045 0 0) 0x7ffbcc0035b0 con 0x7ffbe014f9d0 2023-03-03T12:30:34.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe014fda0 msgr2=0x7ffbe0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:30:34.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe014fda0 0x7ffbe0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:34.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 >> v1:172.21.15.143:6789/0 conn(0x7ffbe00abe20 legacy=0x7ffbe01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:34.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe00c34b0 con 0x7ffbe014f9d0 2023-03-03T12:30:34.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ffbd0001670 con 0x7ffbe014f9d0 2023-03-03T12:30:34.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 >> v1:172.21.15.7:6789/0 conn(0x7ffbe014f9d0 legacy=0x7ffbe0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:34.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 0x7ffbcc004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:34.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe014fda0 0x7ffbcc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:34.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbcc004d90 0x7ffbcc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:34.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe01524a0 con 0x7ffbcc003c30 2023-03-03T12:30:34.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc00bc20 con 0x7ffbcc004d90 2023-03-03T12:30:34.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc00bde0 con 0x7ffbe014fda0 2023-03-03T12:30:34.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe643f700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 0x7ffbcc004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:34.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe5c3e700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe014fda0 0x7ffbcc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:34.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe6c40700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbcc004d90 0x7ffbcc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:34.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe643f700 1 -- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe014fda0 msgr2=0x7ffbcc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:30:34.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe643f700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe014fda0 0x7ffbcc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:34.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe643f700 1 -- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbcc004d90 msgr2=0x7ffbcc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:30:34.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe643f700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbcc004d90 0x7ffbcc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:34.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe643f700 1 -- 172.21.15.7:0/3058175211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc0035b0 con 0x7ffbcc003c30 2023-03-03T12:30:34.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe6c40700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbcc004d90 0x7ffbcc005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:30:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe643f700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 0x7ffbcc004020 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0001eb0 tx=0x7ffbcc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a371110a1a8b996 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.084+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd000cd80 con 0x7ffbcc003c30 2023-03-03T12:30:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.085+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd0001af0 con 0x7ffbcc003c30 2023-03-03T12:30:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.085+0000 7ffbe543d700 1 -- 172.21.15.7:0/3058175211 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0005ba0 con 0x7ffbcc003c30 2023-03-03T12:30:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.085+0000 7ffbe86a3700 1 -- 172.21.15.7:0/3058175211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 msgr2=0x7ffbcc004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.085+0000 7ffbe86a3700 1 --2- 172.21.15.7:0/3058175211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 0x7ffbcc004020 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0001eb0 tx=0x7ffbcc005880 comp rx=0 tx=0).stop 2023-03-03T12:30:34.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.085+0000 7ffbe6c40700 1 -- 172.21.15.7:0/3058175211 reap_dead start 2023-03-03T12:30:34.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.085+0000 7ffbe86a3700 1 -- 172.21.15.7:0/3058175211 shutdown_connections 2023-03-03T12:30:34.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.085+0000 7ffbe86a3700 1 -- 172.21.15.7:0/3058175211 >> 172.21.15.7:0/3058175211 conn(0x7ffbe00c8900 msgr2=0x7ffbe00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:30:34.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.085+0000 7ffbe86a3700 1 -- 172.21.15.7:0/3058175211 shutdown_connections 2023-03-03T12:30:34.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.085+0000 7ffbe86a3700 1 -- 172.21.15.7:0/3058175211 wait complete. 2023-03-03T12:30:34.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.086+0000 7ffbe86a3700 1 Processor -- start 2023-03-03T12:30:34.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.086+0000 7ffbe86a3700 1 -- start start 2023-03-03T12:30:34.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe86a3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 0x7ffbe01dee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:34.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe86a3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe0152b30 0x7ffbe01ed270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:34.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe86a3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbe01ed7b0 0x7ffbe006c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:34.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe86a3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe006cf00 con 0x7ffbcc003c30 2023-03-03T12:30:34.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe86a3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe01e7390 con 0x7ffbe01ed7b0 2023-03-03T12:30:34.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe86a3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe006a2b0 con 0x7ffbe0152b30 2023-03-03T12:30:34.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe643f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 0x7ffbe01dee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:34.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe643f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 0x7ffbe01dee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40766/0 (socket says 172.21.15.7:40766) 2023-03-03T12:30:34.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe643f700 1 -- 172.21.15.7:0/2368431355 learned_addr learned my addr 172.21.15.7:0/2368431355 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:30:34.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe5c3e700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe0152b30 0x7ffbe01ed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:34.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe6c40700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbe01ed7b0 0x7ffbe006c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:34.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe643f700 1 -- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe0152b30 msgr2=0x7ffbe01ed270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:34.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe643f700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe0152b30 0x7ffbe01ed270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:34.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe643f700 1 -- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbe01ed7b0 msgr2=0x7ffbe006c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:34.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe643f700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbe01ed7b0 0x7ffbe006c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:34.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.087+0000 7ffbe643f700 1 -- 172.21.15.7:0/2368431355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe00ac350 con 0x7ffbcc003c30 2023-03-03T12:30:34.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.088+0000 7ffbe643f700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 0x7ffbe01dee90 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000c7d0 tx=0x7ffbd0001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:34.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.088+0000 7ffbd6ffd700 1 -- 172.21.15.7:0/2368431355 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0004c80 con 0x7ffbcc003c30 2023-03-03T12:30:34.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.088+0000 7ffbd6ffd700 1 -- 172.21.15.7:0/2368431355 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd0004de0 con 0x7ffbcc003c30 2023-03-03T12:30:34.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.088+0000 7ffbe86a3700 1 -- 172.21.15.7:0/2368431355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe00ab630 con 0x7ffbcc003c30 2023-03-03T12:30:34.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.088+0000 7ffbd6ffd700 1 -- 172.21.15.7:0/2368431355 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd00072f0 con 0x7ffbcc003c30 2023-03-03T12:30:34.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.088+0000 7ffbe86a3700 1 -- 172.21.15.7:0/2368431355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe0071fa0 con 0x7ffbcc003c30 2023-03-03T12:30:34.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.089+0000 7ffbe86a3700 1 -- 172.21.15.7:0/2368431355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe01edc50 con 0x7ffbcc003c30 2023-03-03T12:30:34.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.089+0000 7ffbd6ffd700 1 -- 172.21.15.7:0/2368431355 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffbd002e070 con 0x7ffbcc003c30 2023-03-03T12:30:34.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.090+0000 7ffbd6ffd700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbc40682f0 0x7ffbc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:30:34.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.090+0000 7ffbd6ffd700 1 -- 172.21.15.7:0/2368431355 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ffbd0087150 con 0x7ffbcc003c30 2023-03-03T12:30:34.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.090+0000 7ffbe5c3e700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbc40682f0 0x7ffbc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:30:34.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.090+0000 7ffbe5c3e700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbc40682f0 0x7ffbc406a7a0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00c4900 tx=0x7ffbdc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:30:34.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.094+0000 7ffbd6ffd700 1 -- 172.21.15.7:0/2368431355 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffbd00535b0 con 0x7ffbcc003c30 2023-03-03T12:30:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.331+0000 7ffbe86a3700 1 -- 172.21.15.7:0/2368431355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbe0074bb0 con 0x7ffbcc003c30 2023-03-03T12:30:34.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.331+0000 7ffbd6ffd700 1 -- 172.21.15.7:0/2368431355 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ffbd0056bd0 con 0x7ffbcc003c30 2023-03-03T12:30:34.335 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:30:34.335 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:30:34.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 -- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbc40682f0 msgr2=0x7ffbc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:34.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbc40682f0 0x7ffbc406a7a0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00c4900 tx=0x7ffbdc009210 comp rx=0 tx=0).stop 2023-03-03T12:30:34.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 -- 172.21.15.7:0/2368431355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 msgr2=0x7ffbe01dee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:30:34.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 0x7ffbe01dee90 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000c7d0 tx=0x7ffbd0001980 comp rx=0 tx=0).stop 2023-03-03T12:30:34.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 -- 172.21.15.7:0/2368431355 shutdown_connections 2023-03-03T12:30:34.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbe01ed7b0 0x7ffbe006c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:34.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffbc40682f0 0x7ffbc406a7a0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:34.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbcc003c30 0x7ffbe01dee90 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:34.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 --2- 172.21.15.7:0/2368431355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffbe0152b30 0x7ffbe01ed270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:30:34.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 -- 172.21.15.7:0/2368431355 >> 172.21.15.7:0/2368431355 conn(0x7ffbe00c8900 msgr2=0x7ffbe00c6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:30:34.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 -- 172.21.15.7:0/2368431355 shutdown_connections 2023-03-03T12:30:34.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:30:34.333+0000 7ffbe86a3700 1 -- 172.21.15.7:0/2368431355 wait complete. 2023-03-03T12:30:34.338 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:30:34.348 INFO:tasks.mds_thrash.fs.[b]:waiting for 59 secs before thrashing 2023-03-03T12:30:59.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:30:59.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:29.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:29.400 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:33.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:31:33.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.484+0000 7f296a6f2700 1 Processor -- start 2023-03-03T12:31:33.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.485+0000 7f296a6f2700 1 -- start start 2023-03-03T12:31:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.485+0000 7f296a6f2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2964151950 0x7f2964154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.485+0000 7f296a6f2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29640c5dc0 0x7f29640bf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.485+0000 7f296a6f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 0x7f29640ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:33.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.485+0000 7f296a6f2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29640c5940 con 0x7f29640c5dc0 2023-03-03T12:31:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.485+0000 7f296a6f2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29641584a0 con 0x7f2964151950 2023-03-03T12:31:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.485+0000 7f296a6f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2964158780 con 0x7f29640b0490 2023-03-03T12:31:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f29637fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29640c5dc0 0x7f29640bf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f29637fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f29640c5dc0 0x7f29640bf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40818/0 (socket says 172.21.15.7:40818) 2023-03-03T12:31:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f29637fe700 1 -- 172.21.15.7:0/4024903606 learned_addr learned my addr 172.21.15.7:0/4024903606 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:31:33.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f2963fff700 1 --2- 172.21.15.7:0/4024903606 >> v2:172.21.15.143:3300/0 conn(0x7f2964151950 0x7f2964154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f2968c8f700 1 --2- 172.21.15.7:0/4024903606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 0x7f29640ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f29637fe700 1 -- 172.21.15.7:0/4024903606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 msgr2=0x7f29640ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:31:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f29637fe700 1 --2- 172.21.15.7:0/4024903606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 0x7f29640ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f29637fe700 1 -- 172.21.15.7:0/4024903606 >> v2:172.21.15.143:3300/0 conn(0x7f2964151950 msgr2=0x7f2964154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:31:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f29637fe700 1 --2- 172.21.15.7:0/4024903606 >> v2:172.21.15.143:3300/0 conn(0x7f2964151950 0x7f2964154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f29637fe700 1 -- 172.21.15.7:0/4024903606 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29641546c0 con 0x7f29640c5dc0 2023-03-03T12:31:33.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f2968c8f700 1 --2- 172.21.15.7:0/4024903606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 0x7f29640ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:31:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f29637fe700 1 --2- 172.21.15.7:0/4024903606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 0x7f29640bf600 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2964066f60 tx=0x7f2958009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ec70e10a1f2eef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:31:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f2962ffd700 1 -- 172.21.15.7:0/4024903606 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2958011830 con 0x7f29640c5dc0 2023-03-03T12:31:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f2962ffd700 1 -- 172.21.15.7:0/4024903606 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2958011990 con 0x7f29640c5dc0 2023-03-03T12:31:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.486+0000 7f2962ffd700 1 -- 172.21.15.7:0/4024903606 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2958011d20 con 0x7f29640c5dc0 2023-03-03T12:31:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.487+0000 7f296a6f2700 1 -- 172.21.15.7:0/4024903606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 msgr2=0x7f29640bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.487+0000 7f296a6f2700 1 --2- 172.21.15.7:0/4024903606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 0x7f29640bf600 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2964066f60 tx=0x7f2958009bd0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.487+0000 7f296a6f2700 1 -- 172.21.15.7:0/4024903606 shutdown_connections 2023-03-03T12:31:33.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.487+0000 7f296a6f2700 1 --2- 172.21.15.7:0/4024903606 >> v2:172.21.15.143:3300/0 conn(0x7f2964151950 0x7f2964154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.487+0000 7f296a6f2700 1 --2- 172.21.15.7:0/4024903606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 0x7f29640ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.487+0000 7f296a6f2700 1 --2- 172.21.15.7:0/4024903606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 0x7f29640bf600 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.487+0000 7f296a6f2700 1 -- 172.21.15.7:0/4024903606 >> 172.21.15.7:0/4024903606 conn(0x7f2964066840 msgr2=0x7f296406ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:31:33.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.487+0000 7f296a6f2700 1 -- 172.21.15.7:0/4024903606 shutdown_connections 2023-03-03T12:31:33.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.487+0000 7f296a6f2700 1 -- 172.21.15.7:0/4024903606 wait complete. 2023-03-03T12:31:33.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.488+0000 7f296a6f2700 1 Processor -- start 2023-03-03T12:31:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.488+0000 7f296a6f2700 1 -- start start 2023-03-03T12:31:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f296a6f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 0x7f2964156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f296a6f2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 0x7f29641ed6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f296a6f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964151950 0x7f29641edc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f296a6f2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29641511b0 con 0x7f29640c5dc0 2023-03-03T12:31:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f296a6f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29641eaa80 con 0x7f2964151950 2023-03-03T12:31:33.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f296a6f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2964156fc0 con 0x7f29640b0490 2023-03-03T12:31:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f29637fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 0x7f29641ed6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f29637fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 0x7f29641ed6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40824/0 (socket says 172.21.15.7:40824) 2023-03-03T12:31:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f29637fe700 1 -- 172.21.15.7:0/1062193113 learned_addr learned my addr 172.21.15.7:0/1062193113 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:31:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f2963fff700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 0x7f2964156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f2968c8f700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964151950 0x7f29641edc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:33.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f29637fe700 1 -- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 msgr2=0x7f2964156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f29637fe700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 0x7f2964156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f29637fe700 1 -- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964151950 msgr2=0x7f29641edc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f29637fe700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964151950 0x7f29641edc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.489+0000 7f29637fe700 1 -- 172.21.15.7:0/1062193113 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2958009550 con 0x7f29640c5dc0 2023-03-03T12:31:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.490+0000 7f29637fe700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 0x7f29641ed6e0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f2958000c00 tx=0x7f2958004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:31:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.490+0000 7f2960ff9700 1 -- 172.21.15.7:0/1062193113 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2958004240 con 0x7f29640c5dc0 2023-03-03T12:31:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.490+0000 7f2960ff9700 1 -- 172.21.15.7:0/1062193113 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295801f070 con 0x7f29640c5dc0 2023-03-03T12:31:33.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.490+0000 7f296a6f2700 1 -- 172.21.15.7:0/1062193113 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2964151d20 con 0x7f29640c5dc0 2023-03-03T12:31:33.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.490+0000 7f2960ff9700 1 -- 172.21.15.7:0/1062193113 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29580036a0 con 0x7f29640c5dc0 2023-03-03T12:31:33.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.490+0000 7f296a6f2700 1 -- 172.21.15.7:0/1062193113 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29641549a0 con 0x7f29640c5dc0 2023-03-03T12:31:33.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.491+0000 7f296a6f2700 1 -- 172.21.15.7:0/1062193113 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2964154370 con 0x7f29640c5dc0 2023-03-03T12:31:33.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.491+0000 7f2960ff9700 1 -- 172.21.15.7:0/1062193113 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f295802a020 con 0x7f29640c5dc0 2023-03-03T12:31:33.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.492+0000 7f2960ff9700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29500682f0 0x7f295006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:31:33.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.492+0000 7f2960ff9700 1 -- 172.21.15.7:0/1062193113 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f29580875a0 con 0x7f29640c5dc0 2023-03-03T12:31:33.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.492+0000 7f2963fff700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29500682f0 0x7f295006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:31:33.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.493+0000 7f2963fff700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29500682f0 0x7f295006a7a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f29641571f0 tx=0x7f2954009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:31:33.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.496+0000 7f2960ff9700 1 -- 172.21.15.7:0/1062193113 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2958053850 con 0x7f29640c5dc0 2023-03-03T12:31:33.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.732+0000 7f296a6f2700 1 -- 172.21.15.7:0/1062193113 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f296404e6e0 con 0x7f29640c5dc0 2023-03-03T12:31:33.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.732+0000 7f2960ff9700 1 -- 172.21.15.7:0/1062193113 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2958056e70 con 0x7f29640c5dc0 2023-03-03T12:31:33.736 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:31:33.736 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:31:33.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 -- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29500682f0 msgr2=0x7f295006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:33.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29500682f0 0x7f295006a7a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f29641571f0 tx=0x7f2954009210 comp rx=0 tx=0).stop 2023-03-03T12:31:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 -- 172.21.15.7:0/1062193113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 msgr2=0x7f29641ed6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:31:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 0x7f29641ed6e0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f2958000c00 tx=0x7f2958004750 comp rx=0 tx=0).stop 2023-03-03T12:31:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 -- 172.21.15.7:0/1062193113 shutdown_connections 2023-03-03T12:31:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964151950 0x7f29641edc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29640b0490 0x7f2964156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29500682f0 0x7f295006a7a0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 --2- 172.21.15.7:0/1062193113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29640c5dc0 0x7f29641ed6e0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:31:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 -- 172.21.15.7:0/1062193113 >> 172.21.15.7:0/1062193113 conn(0x7f2964066840 msgr2=0x7f2964154ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:31:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 -- 172.21.15.7:0/1062193113 shutdown_connections 2023-03-03T12:31:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:31:33.734+0000 7f296a6f2700 1 -- 172.21.15.7:0/1062193113 wait complete. 2023-03-03T12:31:33.739 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:31:33.749 INFO:tasks.mds_thrash.fs.[b]:waiting for 53 secs before thrashing 2023-03-03T12:31:59.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:31:59.538 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:21.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:32:21.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.377+0000 7f069fd72700 1 Processor -- start 2023-03-03T12:32:21.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.378+0000 7f069fd72700 1 -- start start 2023-03-03T12:32:21.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.378+0000 7f069fd72700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f069814f9d0 0x7f0698156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:21.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.378+0000 7f069fd72700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06980abe20 0x7f06981529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:21.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.378+0000 7f069fd72700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06980c2f30 con 0x7f069814fda0 2023-03-03T12:32:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.378+0000 7f069fd72700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06980c31d0 con 0x7f069814f9d0 2023-03-03T12:32:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.378+0000 7f069fd72700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06981524a0 con 0x7f06980abe20 2023-03-03T12:32:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069db0e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f069814f9d0 0x7f0698156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069d30d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f069814fda0 0x7f0698071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34838/0 (socket says 172.21.15.7:34838) 2023-03-03T12:32:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069d30d700 1 -- 172.21.15.7:0/4203150453 learned_addr learned my addr 172.21.15.7:0/4203150453 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:32:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069e30f700 1 --2- 172.21.15.7:0/4203150453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06980abe20 0x7f06981529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069db0e700 1 -- 172.21.15.7:0/4203150453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06980abe20 msgr2=0x7f06981529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:32:21.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069db0e700 1 --2- 172.21.15.7:0/4203150453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06980abe20 0x7f06981529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:21.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069db0e700 1 -- 172.21.15.7:0/4203150453 >> v1:172.21.15.7:6789/0 conn(0x7f069814fda0 legacy=0x7f0698071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:21.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069cb0c700 1 -- 172.21.15.7:0/4203150453 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740896780 0 0) 0x7f06980c2f30 con 0x7f069814fda0 2023-03-03T12:32:21.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069db0e700 1 -- 172.21.15.7:0/4203150453 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06980c34b0 con 0x7f069814f9d0 2023-03-03T12:32:21.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069db0e700 1 --2- 172.21.15.7:0/4203150453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f069814f9d0 0x7f0698156430 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f0694009840 tx=0x7f0694009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdcc94f5ad60130d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:32:21.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069cb0c700 1 -- 172.21.15.7:0/4203150453 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694011830 con 0x7f069814f9d0 2023-03-03T12:32:21.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069cb0c700 1 -- 172.21.15.7:0/4203150453 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0694011990 con 0x7f069814f9d0 2023-03-03T12:32:21.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.379+0000 7f069cb0c700 1 -- 172.21.15.7:0/4203150453 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694011d20 con 0x7f069814f9d0 2023-03-03T12:32:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.380+0000 7f069fd72700 1 -- 172.21.15.7:0/4203150453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f069814f9d0 msgr2=0x7f0698156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.380+0000 7f069fd72700 1 --2- 172.21.15.7:0/4203150453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f069814f9d0 0x7f0698156430 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f0694009840 tx=0x7f0694009b50 comp rx=0 tx=0).stop 2023-03-03T12:32:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.380+0000 7f069fd72700 1 -- 172.21.15.7:0/4203150453 shutdown_connections 2023-03-03T12:32:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.380+0000 7f069fd72700 1 --2- 172.21.15.7:0/4203150453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06980abe20 0x7f06981529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.380+0000 7f069fd72700 1 --2- 172.21.15.7:0/4203150453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f069814f9d0 0x7f0698156430 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.380+0000 7f069fd72700 1 -- 172.21.15.7:0/4203150453 >> 172.21.15.7:0/4203150453 conn(0x7f06980c8900 msgr2=0x7f06980c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:32:21.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.380+0000 7f069fd72700 1 -- 172.21.15.7:0/4203150453 shutdown_connections 2023-03-03T12:32:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.380+0000 7f069fd72700 1 -- 172.21.15.7:0/4203150453 wait complete. 2023-03-03T12:32:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.381+0000 7f069fd72700 1 Processor -- start 2023-03-03T12:32:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.381+0000 7f069fd72700 1 -- start start 2023-03-03T12:32:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069fd72700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06980abe20 0x7f06981eb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069fd72700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f069814f9d0 0x7f0698154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069fd72700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f069814fda0 0x7f069806d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:21.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069fd72700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069806ee50 con 0x7f06980abe20 2023-03-03T12:32:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069fd72700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069806f0f0 con 0x7f069814fda0 2023-03-03T12:32:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069fd72700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06981eb140 con 0x7f069814f9d0 2023-03-03T12:32:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069db0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06980abe20 0x7f06981eb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069db0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06980abe20 0x7f06981eb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40864/0 (socket says 172.21.15.7:40864) 2023-03-03T12:32:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069db0e700 1 -- 172.21.15.7:0/2589867426 learned_addr learned my addr 172.21.15.7:0/2589867426 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:32:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069d30d700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f069814f9d0 0x7f0698154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069e30f700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f069814fda0 0x7f069806d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:21.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069db0e700 1 -- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f069814f9d0 msgr2=0x7f0698154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069db0e700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f069814f9d0 0x7f0698154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069db0e700 1 -- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f069814fda0 msgr2=0x7f069806d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069db0e700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f069814fda0 0x7f069806d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.382+0000 7f069db0e700 1 -- 172.21.15.7:0/2589867426 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06940094a0 con 0x7f06980abe20 2023-03-03T12:32:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.383+0000 7f069db0e700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06980abe20 0x7f06981eb940 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f06940090b0 tx=0x7f0694004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:32:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.383+0000 7f068a7fc700 1 -- 172.21.15.7:0/2589867426 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06940038a0 con 0x7f06980abe20 2023-03-03T12:32:21.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.383+0000 7f068a7fc700 1 -- 172.21.15.7:0/2589867426 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069401f070 con 0x7f06980abe20 2023-03-03T12:32:21.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.383+0000 7f069fd72700 1 -- 172.21.15.7:0/2589867426 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06980ac350 con 0x7f06980abe20 2023-03-03T12:32:21.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.383+0000 7f068a7fc700 1 -- 172.21.15.7:0/2589867426 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694003c20 con 0x7f06980abe20 2023-03-03T12:32:21.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.383+0000 7f069fd72700 1 -- 172.21.15.7:0/2589867426 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06980ab630 con 0x7f06980abe20 2023-03-03T12:32:21.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.384+0000 7f069fd72700 1 -- 172.21.15.7:0/2589867426 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0698071fa0 con 0x7f06980abe20 2023-03-03T12:32:21.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.384+0000 7f068a7fc700 1 -- 172.21.15.7:0/2589867426 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f069400f050 con 0x7f06980abe20 2023-03-03T12:32:21.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.385+0000 7f068a7fc700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f067c0682f0 0x7f067c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:21.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.385+0000 7f068a7fc700 1 -- 172.21.15.7:0/2589867426 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0694021070 con 0x7f06980abe20 2023-03-03T12:32:21.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.385+0000 7f069d30d700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f067c0682f0 0x7f067c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:21.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.385+0000 7f069d30d700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f067c0682f0 0x7f067c06a7a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f0698150820 tx=0x7f068c007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:32:21.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.389+0000 7f068a7fc700 1 -- 172.21.15.7:0/2589867426 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0694053ef0 con 0x7f06980abe20 2023-03-03T12:32:21.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.628+0000 7f069fd72700 1 -- 172.21.15.7:0/2589867426 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06981eb660 con 0x7f06980abe20 2023-03-03T12:32:21.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.628+0000 7f068a7fc700 1 -- 172.21.15.7:0/2589867426 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f06981eb660 con 0x7f06980abe20 2023-03-03T12:32:21.631 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:32:21.631 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:32:21.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.630+0000 7f069fd72700 1 -- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f067c0682f0 msgr2=0x7f067c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.630+0000 7f069fd72700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f067c0682f0 0x7f067c06a7a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f0698150820 tx=0x7f068c007be0 comp rx=0 tx=0).stop 2023-03-03T12:32:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.630+0000 7f069fd72700 1 -- 172.21.15.7:0/2589867426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06980abe20 msgr2=0x7f06981eb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.630+0000 7f069fd72700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06980abe20 0x7f06981eb940 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f06940090b0 tx=0x7f0694004750 comp rx=0 tx=0).stop 2023-03-03T12:32:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.631+0000 7f069fd72700 1 -- 172.21.15.7:0/2589867426 shutdown_connections 2023-03-03T12:32:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.631+0000 7f069fd72700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f069814fda0 0x7f069806d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.631+0000 7f069fd72700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f067c0682f0 0x7f067c06a7a0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:21.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.631+0000 7f069fd72700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06980abe20 0x7f06981eb940 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:21.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.631+0000 7f069fd72700 1 --2- 172.21.15.7:0/2589867426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f069814f9d0 0x7f0698154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:21.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.631+0000 7f069fd72700 1 -- 172.21.15.7:0/2589867426 >> 172.21.15.7:0/2589867426 conn(0x7f06980c8900 msgr2=0x7f06980c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:32:21.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.631+0000 7f069fd72700 1 -- 172.21.15.7:0/2589867426 shutdown_connections 2023-03-03T12:32:21.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:21.631+0000 7f069fd72700 1 -- 172.21.15.7:0/2589867426 wait complete. 2023-03-03T12:32:21.634 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:32:21.644 INFO:tasks.mds_thrash.fs.[a]:waiting for 73 secs before thrashing 2023-03-03T12:32:26.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:32:26.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.888+0000 7f25cccbb700 1 Processor -- start 2023-03-03T12:32:26.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.888+0000 7f25cccbb700 1 -- start start 2023-03-03T12:32:26.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.888+0000 7f25cccbb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f25c814b0d0 0x7f25c8071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:26.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.888+0000 7f25cccbb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25c80aad20 0x7f25c8157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:26.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25cccbb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c80a9c20 con 0x7f25c8071d40 2023-03-03T12:32:26.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25cccbb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c8154f10 con 0x7f25c814b0d0 2023-03-03T12:32:26.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25cccbb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80ac3c0 con 0x7f25c80aad20 2023-03-03T12:32:26.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c6d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f25c8071d40 0x7f25c8158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34860/0 (socket says 172.21.15.7:34860) 2023-03-03T12:32:26.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c6d9d700 1 -- 172.21.15.7:0/3343746828 learned_addr learned my addr 172.21.15.7:0/3343746828 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:32:26.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c5d9b700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25c80aad20 0x7f25c8157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:26.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c659c700 1 --2- 172.21.15.7:0/3343746828 >> v2:172.21.15.143:3300/0 conn(0x7f25c814b0d0 0x7f25c8071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:26.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125843231 0 0) 0x7f25c80a9c20 con 0x7f25c8071d40 2023-03-03T12:32:26.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b40035b0 con 0x7f25c8071d40 2023-03-03T12:32:26.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4082954114 0 0) 0x7f25b40035b0 con 0x7f25c8071d40 2023-03-03T12:32:26.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25c80aad20 msgr2=0x7f25c8157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:32:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25c80aad20 0x7f25c8157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 >> v2:172.21.15.143:3300/0 conn(0x7f25c814b0d0 msgr2=0x7f25c8071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:32:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 --2- 172.21.15.7:0/3343746828 >> v2:172.21.15.143:3300/0 conn(0x7f25c814b0d0 0x7f25c8071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c806fc70 con 0x7f25c8071d40 2023-03-03T12:32:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f25c00019f0 con 0x7f25c8071d40 2023-03-03T12:32:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 >> v1:172.21.15.7:6789/0 conn(0x7f25c8071d40 legacy=0x7f25c8158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:26.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25b4003a70 0x7f25b4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25c80aad20 0x7f25b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25b4004d90 0x7f25b4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c8154f10 con 0x7f25b4004d90 2023-03-03T12:32:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.889+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80ac3c0 con 0x7f25b4003a70 2023-03-03T12:32:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b400ba60 con 0x7f25c80aad20 2023-03-03T12:32:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c6d9d700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25b4004d90 0x7f25b4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:26.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c659c700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25b4003a70 0x7f25b4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c5d9b700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25c80aad20 0x7f25b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c6d9d700 1 -- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25c80aad20 msgr2=0x7f25b4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:32:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c6d9d700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25c80aad20 0x7f25b4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c6d9d700 1 -- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25b4003a70 msgr2=0x7f25b4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:32:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c6d9d700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25b4003a70 0x7f25b4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c6d9d700 1 -- 172.21.15.7:0/3343746828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b40035b0 con 0x7f25b4004d90 2023-03-03T12:32:26.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c6d9d700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25b4004d90 0x7f25b4005200 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f25c000a900 tx=0x7f25b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4aca72653746f45 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:32:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c000a340 con 0x7f25b4004d90 2023-03-03T12:32:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c0012620 con 0x7f25b4004d90 2023-03-03T12:32:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.890+0000 7f25c559a700 1 -- 172.21.15.7:0/3343746828 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c00129d0 con 0x7f25b4004d90 2023-03-03T12:32:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.891+0000 7f25cccbb700 1 -- 172.21.15.7:0/3343746828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25b4004d90 msgr2=0x7f25b4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.891+0000 7f25cccbb700 1 --2- 172.21.15.7:0/3343746828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25b4004d90 0x7f25b4005200 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f25c000a900 tx=0x7f25b4005880 comp rx=0 tx=0).stop 2023-03-03T12:32:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.891+0000 7f25c6d9d700 1 -- 172.21.15.7:0/3343746828 reap_dead start 2023-03-03T12:32:26.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.891+0000 7f25cccbb700 1 -- 172.21.15.7:0/3343746828 shutdown_connections 2023-03-03T12:32:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.891+0000 7f25cccbb700 1 -- 172.21.15.7:0/3343746828 >> 172.21.15.7:0/3343746828 conn(0x7f25c80c8900 msgr2=0x7f25c80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:32:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.891+0000 7f25cccbb700 1 -- 172.21.15.7:0/3343746828 shutdown_connections 2023-03-03T12:32:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.891+0000 7f25cccbb700 1 -- 172.21.15.7:0/3343746828 wait complete. 2023-03-03T12:32:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.892+0000 7f25cccbb700 1 Processor -- start 2023-03-03T12:32:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.892+0000 7f25cccbb700 1 -- start start 2023-03-03T12:32:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25cccbb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25b4004d90 0x7f25c81ed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25cccbb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25c8158670 0x7f25c81523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25cccbb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25c81ed900 0x7f25c81528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25cccbb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c81e0ee0 con 0x7f25c81ed900 2023-03-03T12:32:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25cccbb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c8152e20 con 0x7f25c8158670 2023-03-03T12:32:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25cccbb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c81edff0 con 0x7f25b4004d90 2023-03-03T12:32:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25c6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25c81ed900 0x7f25c81528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25c6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25c81ed900 0x7f25c81528e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40890/0 (socket says 172.21.15.7:40890) 2023-03-03T12:32:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25c6d9d700 1 -- 172.21.15.7:0/43643521 learned_addr learned my addr 172.21.15.7:0/43643521 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:32:26.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25c5d9b700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25c8158670 0x7f25c81523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:26.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.893+0000 7f25c659c700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25b4004d90 0x7f25c81ed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:26.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25c6d9d700 1 -- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25b4004d90 msgr2=0x7f25c81ed3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:26.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25c6d9d700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25b4004d90 0x7f25c81ed3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:26.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25c6d9d700 1 -- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25c8158670 msgr2=0x7f25c81523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:26.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25c6d9d700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25c8158670 0x7f25c81523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:26.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25c6d9d700 1 -- 172.21.15.7:0/43643521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c806f6b0 con 0x7f25c81ed900 2023-03-03T12:32:26.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25c6d9d700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25c81ed900 0x7f25c81528e0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f25c0012000 tx=0x7f25c0001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:32:26.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25beffd700 1 -- 172.21.15.7:0/43643521 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c000a340 con 0x7f25c81ed900 2023-03-03T12:32:26.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25beffd700 1 -- 172.21.15.7:0/43643521 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c0001df0 con 0x7f25c81ed900 2023-03-03T12:32:26.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25cccbb700 1 -- 172.21.15.7:0/43643521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c8155160 con 0x7f25c81ed900 2023-03-03T12:32:26.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25beffd700 1 -- 172.21.15.7:0/43643521 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0026640 con 0x7f25c81ed900 2023-03-03T12:32:26.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.894+0000 7f25cccbb700 1 -- 172.21.15.7:0/43643521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c806f990 con 0x7f25c81ed900 2023-03-03T12:32:26.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.895+0000 7f25cccbb700 1 -- 172.21.15.7:0/43643521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c80aa9f0 con 0x7f25c81ed900 2023-03-03T12:32:26.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.895+0000 7f25beffd700 1 -- 172.21.15.7:0/43643521 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f25c0010070 con 0x7f25c81ed900 2023-03-03T12:32:26.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.896+0000 7f25beffd700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25a8068320 0x7f25a806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:32:26.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.896+0000 7f25beffd700 1 -- 172.21.15.7:0/43643521 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f25c0087dd0 con 0x7f25c81ed900 2023-03-03T12:32:26.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.896+0000 7f25c659c700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25a8068320 0x7f25a806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:32:26.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.897+0000 7f25c659c700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25a8068320 0x7f25a806a7d0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f25b8009c60 tx=0x7f25b8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:32:26.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:26.900+0000 7f25beffd700 1 -- 172.21.15.7:0/43643521 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f25c00542e0 con 0x7f25c81ed900 2023-03-03T12:32:27.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.138+0000 7f25cccbb700 1 -- 172.21.15.7:0/43643521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25c81edd10 con 0x7f25c81ed900 2023-03-03T12:32:27.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.139+0000 7f25beffd700 1 -- 172.21.15.7:0/43643521 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f25c81edd10 con 0x7f25c81ed900 2023-03-03T12:32:27.141 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:32:27.141 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:32:27.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 -- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25a8068320 msgr2=0x7f25a806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:27.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25a8068320 0x7f25a806a7d0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f25b8009c60 tx=0x7f25b8009210 comp rx=0 tx=0).stop 2023-03-03T12:32:27.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 -- 172.21.15.7:0/43643521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25c81ed900 msgr2=0x7f25c81528e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:32:27.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25c81ed900 0x7f25c81528e0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f25c0012000 tx=0x7f25c0001360 comp rx=0 tx=0).stop 2023-03-03T12:32:27.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 -- 172.21.15.7:0/43643521 shutdown_connections 2023-03-03T12:32:27.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25a8068320 0x7f25a806a7d0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:27.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25c81ed900 0x7f25c81528e0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:27.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25b4004d90 0x7f25c81ed3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:27.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 --2- 172.21.15.7:0/43643521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25c8158670 0x7f25c81523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:32:27.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 -- 172.21.15.7:0/43643521 >> 172.21.15.7:0/43643521 conn(0x7f25c80c8900 msgr2=0x7f25c80ac5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:32:27.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 -- 172.21.15.7:0/43643521 shutdown_connections 2023-03-03T12:32:27.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:32:27.141+0000 7f25cccbb700 1 -- 172.21.15.7:0/43643521 wait complete. 2023-03-03T12:32:27.144 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:32:27.155 INFO:tasks.mds_thrash.fs.[b]:waiting for 118 secs before thrashing 2023-03-03T12:32:29.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:29.600 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:59.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:32:59.740 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:29.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:29.869 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:34.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:33:34.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.784+0000 7f4ba0694700 1 Processor -- start 2023-03-03T12:33:34.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.784+0000 7f4ba0694700 1 -- start start 2023-03-03T12:33:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4ba0694700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4b980ace30 0x7f4b98152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4ba0694700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4b980c26e0 0x7f4b98152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4ba0694700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b980ac900 con 0x7f4b980ac220 2023-03-03T12:33:34.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4ba0694700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b98071e30 con 0x7f4b980c26e0 2023-03-03T12:33:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4ba0694700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b98152ed0 con 0x7f4b980ace30 2023-03-03T12:33:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4b9ec31700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4b980c26e0 0x7f4b98152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4b9ec31700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4b980c26e0 0x7f4b98152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40922/0 (socket says 172.21.15.7:40922) 2023-03-03T12:33:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4b9ec31700 1 -- 172.21.15.7:0/3679422295 learned_addr learned my addr 172.21.15.7:0/3679422295 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:33:34.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4b9dc2f700 1 --2- 172.21.15.7:0/3679422295 >> v2:172.21.15.143:3300/0 conn(0x7f4b980ace30 0x7f4b98152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4b9ec31700 1 -- 172.21.15.7:0/3679422295 >> v2:172.21.15.143:3300/0 conn(0x7f4b980ace30 msgr2=0x7f4b98152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:33:34.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4b9ec31700 1 --2- 172.21.15.7:0/3679422295 >> v2:172.21.15.143:3300/0 conn(0x7f4b980ace30 0x7f4b98152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4b9ec31700 1 -- 172.21.15.7:0/3679422295 >> v1:172.21.15.143:6789/0 conn(0x7f4b980ac220 legacy=0x7f4b980736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.785+0000 7f4b9ec31700 1 -- 172.21.15.7:0/3679422295 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b980acb50 con 0x7f4b980c26e0 2023-03-03T12:33:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.786+0000 7f4b9ec31700 1 --2- 172.21.15.7:0/3679422295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 0x7f4b98152990 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f4b980c4eb0 tx=0x7f4b9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952d3e5dc60085bc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:33:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.786+0000 7f4b9d42e700 1 -- 172.21.15.7:0/3679422295 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9000e070 con 0x7f4b980c26e0 2023-03-03T12:33:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.786+0000 7f4b9d42e700 1 -- 172.21.15.7:0/3679422295 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9000b910 con 0x7f4b980c26e0 2023-03-03T12:33:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.786+0000 7f4b9d42e700 1 -- 172.21.15.7:0/3679422295 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9000bcc0 con 0x7f4b980c26e0 2023-03-03T12:33:34.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.786+0000 7f4ba0694700 1 -- 172.21.15.7:0/3679422295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 msgr2=0x7f4b98152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.786+0000 7f4ba0694700 1 --2- 172.21.15.7:0/3679422295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 0x7f4b98152990 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f4b980c4eb0 tx=0x7f4b9000b330 comp rx=0 tx=0).stop 2023-03-03T12:33:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.786+0000 7f4ba0694700 1 -- 172.21.15.7:0/3679422295 shutdown_connections 2023-03-03T12:33:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.786+0000 7f4ba0694700 1 --2- 172.21.15.7:0/3679422295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 0x7f4b98152990 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.786+0000 7f4ba0694700 1 --2- 172.21.15.7:0/3679422295 >> v2:172.21.15.143:3300/0 conn(0x7f4b980ace30 0x7f4b98152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.786+0000 7f4ba0694700 1 -- 172.21.15.7:0/3679422295 >> 172.21.15.7:0/3679422295 conn(0x7f4b980a94d0 msgr2=0x7f4b980a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:33:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.787+0000 7f4ba0694700 1 -- 172.21.15.7:0/3679422295 shutdown_connections 2023-03-03T12:33:34.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.787+0000 7f4ba0694700 1 -- 172.21.15.7:0/3679422295 wait complete. 2023-03-03T12:33:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.788+0000 7f4ba0694700 1 Processor -- start 2023-03-03T12:33:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.788+0000 7f4ba0694700 1 -- start start 2023-03-03T12:33:34.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.788+0000 7f4ba0694700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b980ac220 0x7f4b98151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.788+0000 7f4ba0694700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b980ace30 0x7f4b980be960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.788+0000 7f4ba0694700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 0x7f4b980beea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.788+0000 7f4ba0694700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b980bfce0 con 0x7f4b980c26e0 2023-03-03T12:33:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.788+0000 7f4ba0694700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b981eb9c0 con 0x7f4b980ace30 2023-03-03T12:33:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.788+0000 7f4ba0694700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b980c15d0 con 0x7f4b980ac220 2023-03-03T12:33:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9ec31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 0x7f4b980beea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9ec31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 0x7f4b980beea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40928/0 (socket says 172.21.15.7:40928) 2023-03-03T12:33:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9ec31700 1 -- 172.21.15.7:0/1321515149 learned_addr learned my addr 172.21.15.7:0/1321515149 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:33:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9dc2f700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b980ace30 0x7f4b980be960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9e430700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b980ac220 0x7f4b98151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9ec31700 1 -- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b980ac220 msgr2=0x7f4b98151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9ec31700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b980ac220 0x7f4b98151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9ec31700 1 -- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b980ace30 msgr2=0x7f4b980be960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9ec31700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b980ace30 0x7f4b980be960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9ec31700 1 -- 172.21.15.7:0/1321515149 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b90009b30 con 0x7f4b980c26e0 2023-03-03T12:33:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b9ec31700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 0x7f4b980beea0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f4b9806b980 tx=0x7f4b90004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:33:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b8effd700 1 -- 172.21.15.7:0/1321515149 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90004930 con 0x7f4b980c26e0 2023-03-03T12:33:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b8effd700 1 -- 172.21.15.7:0/1321515149 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b900099b0 con 0x7f4b980c26e0 2023-03-03T12:33:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4b8effd700 1 -- 172.21.15.7:0/1321515149 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9001b670 con 0x7f4b980c26e0 2023-03-03T12:33:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4ba0694700 1 -- 172.21.15.7:0/1321515149 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b98155ca0 con 0x7f4b980c26e0 2023-03-03T12:33:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.789+0000 7f4ba0694700 1 -- 172.21.15.7:0/1321515149 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b981559c0 con 0x7f4b980c26e0 2023-03-03T12:33:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.790+0000 7f4ba0694700 1 -- 172.21.15.7:0/1321515149 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b98155f80 con 0x7f4b980c26e0 2023-03-03T12:33:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.791+0000 7f4b8effd700 1 -- 172.21.15.7:0/1321515149 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4b9002b070 con 0x7f4b980c26e0 2023-03-03T12:33:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.791+0000 7f4b8effd700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b7c0682f0 0x7f4b7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.791+0000 7f4b8effd700 1 -- 172.21.15.7:0/1321515149 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4b900878d0 con 0x7f4b980c26e0 2023-03-03T12:33:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.791+0000 7f4b9e430700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b7c0682f0 0x7f4b7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.792+0000 7f4b9e430700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b7c0682f0 0x7f4b7c06a7a0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4b94000f60 tx=0x7f4b94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:33:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:34.795+0000 7f4b8effd700 1 -- 172.21.15.7:0/1321515149 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b90053c70 con 0x7f4b980c26e0 2023-03-03T12:33:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.037+0000 7f4ba0694700 1 -- 172.21.15.7:0/1321515149 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b981ebb30 con 0x7f4b980c26e0 2023-03-03T12:33:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.037+0000 7f4b8effd700 1 -- 172.21.15.7:0/1321515149 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4b90057290 con 0x7f4b980c26e0 2023-03-03T12:33:35.040 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:33:35.040 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:33:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.039+0000 7f4ba0694700 1 -- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b7c0682f0 msgr2=0x7f4b7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.039+0000 7f4ba0694700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b7c0682f0 0x7f4b7c06a7a0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f4b94000f60 tx=0x7f4b94009040 comp rx=0 tx=0).stop 2023-03-03T12:33:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.040+0000 7f4ba0694700 1 -- 172.21.15.7:0/1321515149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 msgr2=0x7f4b980beea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.040+0000 7f4ba0694700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 0x7f4b980beea0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f4b9806b980 tx=0x7f4b90004ad0 comp rx=0 tx=0).stop 2023-03-03T12:33:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.040+0000 7f4ba0694700 1 -- 172.21.15.7:0/1321515149 shutdown_connections 2023-03-03T12:33:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.040+0000 7f4ba0694700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b7c0682f0 0x7f4b7c06a7a0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.040+0000 7f4ba0694700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b980c26e0 0x7f4b980beea0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.040+0000 7f4ba0694700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b980ac220 0x7f4b98151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.040+0000 7f4ba0694700 1 --2- 172.21.15.7:0/1321515149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b980ace30 0x7f4b980be960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.040+0000 7f4ba0694700 1 -- 172.21.15.7:0/1321515149 >> 172.21.15.7:0/1321515149 conn(0x7f4b980a94d0 msgr2=0x7f4b98073e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:33:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.040+0000 7f4ba0694700 1 -- 172.21.15.7:0/1321515149 shutdown_connections 2023-03-03T12:33:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:35.040+0000 7f4ba0694700 1 -- 172.21.15.7:0/1321515149 wait complete. 2023-03-03T12:33:35.044 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:33:35.054 INFO:tasks.mds_thrash.fs.[a]:waiting for 17 secs before thrashing 2023-03-03T12:33:52.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:33:52.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.192+0000 7f945bd6e700 1 Processor -- start 2023-03-03T12:33:52.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.192+0000 7f945bd6e700 1 -- start start 2023-03-03T12:33:52.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94540aac00 0x7f94540c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:52.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9454071c30 0x7f94540beb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:52.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94540713c0 con 0x7f94540bf0a0 2023-03-03T12:33:52.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9454152c70 con 0x7f94540aac00 2023-03-03T12:33:52.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94540c4930 con 0x7f9454071c30 2023-03-03T12:33:52.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9459b0a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94540aac00 0x7f94540c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:52.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9459b0a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94540aac00 0x7f94540c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40944/0 (socket says 172.21.15.7:40944) 2023-03-03T12:33:52.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9459b0a700 1 -- 172.21.15.7:0/1251791615 learned_addr learned my addr 172.21.15.7:0/1251791615 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:33:52.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9459309700 1 --2- 172.21.15.7:0/1251791615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9454071c30 0x7f94540beb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:52.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9459b0a700 1 -- 172.21.15.7:0/1251791615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9454071c30 msgr2=0x7f94540beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:33:52.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9459b0a700 1 --2- 172.21.15.7:0/1251791615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9454071c30 0x7f94540beb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:52.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9459b0a700 1 -- 172.21.15.7:0/1251791615 >> v1:172.21.15.143:6789/0 conn(0x7f94540bf0a0 legacy=0x7f9454155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:33:52.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9459b0a700 1 -- 172.21.15.7:0/1251791615 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9454152360 con 0x7f94540aac00 2023-03-03T12:33:52.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9459b0a700 1 --2- 172.21.15.7:0/1251791615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94540aac00 0x7f94540c43f0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f94440098f0 tx=0x7f9444009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd297da96977633 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:33:52.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9458b08700 1 -- 172.21.15.7:0/1251791615 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444004230 con 0x7f94540aac00 2023-03-03T12:33:52.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9458b08700 1 -- 172.21.15.7:0/1251791615 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9444004390 con 0x7f94540aac00 2023-03-03T12:33:52.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f9458b08700 1 -- 172.21.15.7:0/1251791615 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94440036a0 con 0x7f94540aac00 2023-03-03T12:33:52.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 -- 172.21.15.7:0/1251791615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94540aac00 msgr2=0x7f94540c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:52.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 --2- 172.21.15.7:0/1251791615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94540aac00 0x7f94540c43f0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f94440098f0 tx=0x7f9444009c00 comp rx=0 tx=0).stop 2023-03-03T12:33:52.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 -- 172.21.15.7:0/1251791615 shutdown_connections 2023-03-03T12:33:52.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 --2- 172.21.15.7:0/1251791615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94540aac00 0x7f94540c43f0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:52.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 --2- 172.21.15.7:0/1251791615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9454071c30 0x7f94540beb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:52.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 -- 172.21.15.7:0/1251791615 >> 172.21.15.7:0/1251791615 conn(0x7f94540736f0 msgr2=0x7f9454073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:33:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 -- 172.21.15.7:0/1251791615 shutdown_connections 2023-03-03T12:33:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.194+0000 7f945bd6e700 1 -- 172.21.15.7:0/1251791615 wait complete. 2023-03-03T12:33:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f945bd6e700 1 Processor -- start 2023-03-03T12:33:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f945bd6e700 1 -- start start 2023-03-03T12:33:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f945bd6e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9454071c30 0x7f94541eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f945bd6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94540aac00 0x7f94541547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f945bd6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94540bf0a0 0x7f94540c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f945bd6e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94541555f0 con 0x7f9454071c30 2023-03-03T12:33:52.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f945bd6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94541e7620 con 0x7f94540aac00 2023-03-03T12:33:52.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f945bd6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94541eba50 con 0x7f94540bf0a0 2023-03-03T12:33:52.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f9459b0a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9454071c30 0x7f94541eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:52.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f9459b0a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9454071c30 0x7f94541eb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40950/0 (socket says 172.21.15.7:40950) 2023-03-03T12:33:52.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.196+0000 7f9459b0a700 1 -- 172.21.15.7:0/454397566 learned_addr learned my addr 172.21.15.7:0/454397566 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:33:52.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f9459309700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94540aac00 0x7f94541547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:52.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f945a30b700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94540bf0a0 0x7f94540c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:52.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f9459b0a700 1 -- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94540bf0a0 msgr2=0x7f94540c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:52.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f9459b0a700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94540bf0a0 0x7f94540c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:52.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f9459b0a700 1 -- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94540aac00 msgr2=0x7f94541547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:52.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f9459b0a700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94540aac00 0x7f94541547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:52.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f9459b0a700 1 -- 172.21.15.7:0/454397566 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9444009550 con 0x7f9454071c30 2023-03-03T12:33:52.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f9459b0a700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9454071c30 0x7f94541eb510 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f9454074310 tx=0x7f9444004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:33:52.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f944a7fc700 1 -- 172.21.15.7:0/454397566 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444003d70 con 0x7f9454071c30 2023-03-03T12:33:52.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f944a7fc700 1 -- 172.21.15.7:0/454397566 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944400b660 con 0x7f9454071c30 2023-03-03T12:33:52.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f944a7fc700 1 -- 172.21.15.7:0/454397566 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944400ba10 con 0x7f9454071c30 2023-03-03T12:33:52.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f945bd6e700 1 -- 172.21.15.7:0/454397566 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9454152620 con 0x7f9454071c30 2023-03-03T12:33:52.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.197+0000 7f945bd6e700 1 -- 172.21.15.7:0/454397566 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94540c4b80 con 0x7f9454071c30 2023-03-03T12:33:52.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.198+0000 7f945bd6e700 1 -- 172.21.15.7:0/454397566 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9454152900 con 0x7f9454071c30 2023-03-03T12:33:52.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.199+0000 7f944a7fc700 1 -- 172.21.15.7:0/454397566 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9444017020 con 0x7f9454071c30 2023-03-03T12:33:52.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.199+0000 7f944a7fc700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94380682f0 0x7f943806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:33:52.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.199+0000 7f944a7fc700 1 -- 172.21.15.7:0/454397566 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9444087780 con 0x7f9454071c30 2023-03-03T12:33:52.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.199+0000 7f9459309700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94380682f0 0x7f943806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:33:52.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.200+0000 7f9459309700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94380682f0 0x7f943806a7a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f94540c7aa0 tx=0x7f9450008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:33:52.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.204+0000 7f944a7fc700 1 -- 172.21.15.7:0/454397566 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9444053ac0 con 0x7f9454071c30 2023-03-03T12:33:52.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.442+0000 7f945bd6e700 1 -- 172.21.15.7:0/454397566 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94540c7b10 con 0x7f9454071c30 2023-03-03T12:33:52.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.443+0000 7f944a7fc700 1 -- 172.21.15.7:0/454397566 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f94440570e0 con 0x7f9454071c30 2023-03-03T12:33:52.446 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:33:52.447 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:33:52.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 -- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94380682f0 msgr2=0x7f943806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:52.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94380682f0 0x7f943806a7a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f94540c7aa0 tx=0x7f9450008040 comp rx=0 tx=0).stop 2023-03-03T12:33:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 -- 172.21.15.7:0/454397566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9454071c30 msgr2=0x7f94541eb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:33:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9454071c30 0x7f94541eb510 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f9454074310 tx=0x7f9444004750 comp rx=0 tx=0).stop 2023-03-03T12:33:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 -- 172.21.15.7:0/454397566 shutdown_connections 2023-03-03T12:33:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94540bf0a0 0x7f94540c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94380682f0 0x7f943806a7a0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9454071c30 0x7f94541eb510 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:52.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 --2- 172.21.15.7:0/454397566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94540aac00 0x7f94541547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:33:52.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 -- 172.21.15.7:0/454397566 >> 172.21.15.7:0/454397566 conn(0x7f94540736f0 msgr2=0x7f945406eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:33:52.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 -- 172.21.15.7:0/454397566 shutdown_connections 2023-03-03T12:33:52.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:33:52.445+0000 7f945bd6e700 1 -- 172.21.15.7:0/454397566 wait complete. 2023-03-03T12:33:52.450 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:33:52.460 INFO:tasks.mds_thrash.fs.[a]:waiting for 100 secs before thrashing 2023-03-03T12:33:59.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:33:59.949 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:25.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:34:25.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.292+0000 7fa7d0edc700 1 Processor -- start 2023-03-03T12:34:25.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.292+0000 7fa7d0edc700 1 -- start start 2023-03-03T12:34:25.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.293+0000 7fa7d0edc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7cc154f10 0x7fa7cc158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:34:25.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.293+0000 7fa7d0edc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc151ef0 con 0x7fa7cc0aa010 2023-03-03T12:34:25.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.293+0000 7fa7d0edc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc157b20 con 0x7fa7cc151b20 2023-03-03T12:34:25.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.293+0000 7fa7d0edc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7cc155410 con 0x7fa7cc154f10 2023-03-03T12:34:25.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7c9d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7cc154f10 0x7fa7cc158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:34:25.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7cad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa7cc0aa010 0x7fa7cc15ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34964/0 (socket says 172.21.15.7:34964) 2023-03-03T12:34:25.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7c9d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7cc154f10 0x7fa7cc158000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40982/0 (socket says 172.21.15.7:40982) 2023-03-03T12:34:25.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7cad9d700 1 -- 172.21.15.7:0/3923443077 learned_addr learned my addr 172.21.15.7:0/3923443077 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:34:25.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7c9d9b700 1 -- 172.21.15.7:0/3923443077 >> v1:172.21.15.143:6789/0 conn(0x7fa7cc151b20 legacy=0x7fa7cc071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:25.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7c9d9b700 1 -- 172.21.15.7:0/3923443077 >> v1:172.21.15.7:6789/0 conn(0x7fa7cc0aa010 legacy=0x7fa7cc15ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:25.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7c9d9b700 1 -- 172.21.15.7:0/3923443077 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc06f640 con 0x7fa7cc154f10 2023-03-03T12:34:25.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7c959a700 1 -- 172.21.15.7:0/3923443077 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293147550 0 0) 0x7fa7cc151ef0 con 0x7fa7cc0aa010 2023-03-03T12:34:25.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7c9d9b700 1 --2- 172.21.15.7:0/3923443077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc154f10 0x7fa7cc158000 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009840 tx=0x7fa7c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120f8c95ad6f77e4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:34:25.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7c959a700 1 -- 172.21.15.7:0/3923443077 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0011830 con 0x7fa7cc154f10 2023-03-03T12:34:25.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7c959a700 1 -- 172.21.15.7:0/3923443077 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c0011990 con 0x7fa7cc154f10 2023-03-03T12:34:25.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.294+0000 7fa7c959a700 1 -- 172.21.15.7:0/3923443077 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0011d20 con 0x7fa7cc154f10 2023-03-03T12:34:25.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.295+0000 7fa7d0edc700 1 -- 172.21.15.7:0/3923443077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc154f10 msgr2=0x7fa7cc158000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:25.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.295+0000 7fa7d0edc700 1 --2- 172.21.15.7:0/3923443077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc154f10 0x7fa7cc158000 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009840 tx=0x7fa7c0009b50 comp rx=0 tx=0).stop 2023-03-03T12:34:25.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.295+0000 7fa7d0edc700 1 -- 172.21.15.7:0/3923443077 shutdown_connections 2023-03-03T12:34:25.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.295+0000 7fa7d0edc700 1 --2- 172.21.15.7:0/3923443077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc154f10 0x7fa7cc158000 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:25.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.295+0000 7fa7d0edc700 1 -- 172.21.15.7:0/3923443077 >> 172.21.15.7:0/3923443077 conn(0x7fa7cc0c8900 msgr2=0x7fa7cc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:34:25.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.295+0000 7fa7d0edc700 1 -- 172.21.15.7:0/3923443077 shutdown_connections 2023-03-03T12:34:25.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.295+0000 7fa7d0edc700 1 -- 172.21.15.7:0/3923443077 wait complete. 2023-03-03T12:34:25.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.296+0000 7fa7d0edc700 1 Processor -- start 2023-03-03T12:34:25.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.296+0000 7fa7d0edc700 1 -- start start 2023-03-03T12:34:25.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7d0edc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7cc0aa010 0x7fa7cc0c6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:34:25.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7d0edc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc151b20 0x7fa7cc145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:34:25.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7d0edc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7cc154f10 0x7fa7cc146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:34:25.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7d0edc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc1469c0 con 0x7fa7cc151b20 2023-03-03T12:34:25.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7d0edc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc152920 con 0x7fa7cc154f10 2023-03-03T12:34:25.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7d0edc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc152c00 con 0x7fa7cc0aa010 2023-03-03T12:34:25.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7c9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc151b20 0x7fa7cc145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:34:25.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7c9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc151b20 0x7fa7cc145f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40988/0 (socket says 172.21.15.7:40988) 2023-03-03T12:34:25.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7c9d9b700 1 -- 172.21.15.7:0/2144904562 learned_addr learned my addr 172.21.15.7:0/2144904562 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:34:25.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7cad9d700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7cc154f10 0x7fa7cc146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:34:25.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7ca59c700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7cc0aa010 0x7fa7cc0c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:34:25.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7c9d9b700 1 -- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7cc0aa010 msgr2=0x7fa7cc0c6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:25.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7c9d9b700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7cc0aa010 0x7fa7cc0c6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:25.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7c9d9b700 1 -- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7cc154f10 msgr2=0x7fa7cc146480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:25.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7c9d9b700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7cc154f10 0x7fa7cc146480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:25.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.297+0000 7fa7c9d9b700 1 -- 172.21.15.7:0/2144904562 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c00094a0 con 0x7fa7cc151b20 2023-03-03T12:34:25.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.298+0000 7fa7cad9d700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7cc154f10 0x7fa7cc146480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:34:25.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.298+0000 7fa7c9d9b700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc151b20 0x7fa7cc145f40 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fa7c00090b0 tx=0x7fa7c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:34:25.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.298+0000 7fa7baffd700 1 -- 172.21.15.7:0/2144904562 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c001a980 con 0x7fa7cc151b20 2023-03-03T12:34:25.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.298+0000 7fa7baffd700 1 -- 172.21.15.7:0/2144904562 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c001aae0 con 0x7fa7cc151b20 2023-03-03T12:34:25.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.298+0000 7fa7d0edc700 1 -- 172.21.15.7:0/2144904562 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7cc06fc00 con 0x7fa7cc151b20 2023-03-03T12:34:25.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.298+0000 7fa7baffd700 1 -- 172.21.15.7:0/2144904562 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c001ae70 con 0x7fa7cc151b20 2023-03-03T12:34:25.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.298+0000 7fa7d0edc700 1 -- 172.21.15.7:0/2144904562 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7cc06f920 con 0x7fa7cc151b20 2023-03-03T12:34:25.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.299+0000 7fa7d0edc700 1 -- 172.21.15.7:0/2144904562 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7cc071f30 con 0x7fa7cc151b20 2023-03-03T12:34:25.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.299+0000 7fa7baffd700 1 -- 172.21.15.7:0/2144904562 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa7c002b030 con 0x7fa7cc151b20 2023-03-03T12:34:25.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.300+0000 7fa7baffd700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7ac0682f0 0x7fa7ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:34:25.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.300+0000 7fa7baffd700 1 -- 172.21.15.7:0/2144904562 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa7c0087580 con 0x7fa7cc151b20 2023-03-03T12:34:25.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.300+0000 7fa7ca59c700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7ac0682f0 0x7fa7ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:34:25.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.301+0000 7fa7ca59c700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7ac0682f0 0x7fa7ac06a7a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000f60 tx=0x7fa7bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:34:25.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.304+0000 7fa7baffd700 1 -- 172.21.15.7:0/2144904562 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7c0053920 con 0x7fa7cc151b20 2023-03-03T12:34:25.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.546+0000 7fa7d0edc700 1 -- 172.21.15.7:0/2144904562 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7cc074bb0 con 0x7fa7cc151b20 2023-03-03T12:34:25.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.547+0000 7fa7baffd700 1 -- 172.21.15.7:0/2144904562 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa7c0056f40 con 0x7fa7cc151b20 2023-03-03T12:34:25.549 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:34:25.549 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:34:25.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 -- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7ac0682f0 msgr2=0x7fa7ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:25.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7ac0682f0 0x7fa7ac06a7a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000f60 tx=0x7fa7bc009040 comp rx=0 tx=0).stop 2023-03-03T12:34:25.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 -- 172.21.15.7:0/2144904562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc151b20 msgr2=0x7fa7cc145f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:34:25.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc151b20 0x7fa7cc145f40 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fa7c00090b0 tx=0x7fa7c0004750 comp rx=0 tx=0).stop 2023-03-03T12:34:25.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 -- 172.21.15.7:0/2144904562 shutdown_connections 2023-03-03T12:34:25.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7cc154f10 0x7fa7cc146480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:25.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7cc0aa010 0x7fa7cc0c6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:25.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7ac0682f0 0x7fa7ac06a7a0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:25.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 --2- 172.21.15.7:0/2144904562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7cc151b20 0x7fa7cc145f40 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:34:25.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 -- 172.21.15.7:0/2144904562 >> 172.21.15.7:0/2144904562 conn(0x7fa7cc0c8900 msgr2=0x7fa7cc0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:34:25.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 -- 172.21.15.7:0/2144904562 shutdown_connections 2023-03-03T12:34:25.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:34:25.549+0000 7fa7d0edc700 1 -- 172.21.15.7:0/2144904562 wait complete. 2023-03-03T12:34:25.553 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:34:25.563 INFO:tasks.mds_thrash.fs.[b]:waiting for 63 secs before thrashing 2023-03-03T12:34:30.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:34:30.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:00.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:00.217 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:28.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:35:28.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.701+0000 7fa24eef2700 1 Processor -- start 2023-03-03T12:35:28.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.701+0000 7fa24eef2700 1 -- start start 2023-03-03T12:35:28.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.701+0000 7fa24eef2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa24814f9d0 0x7fa248156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:28.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.701+0000 7fa24eef2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa2480abe20 0x7fa2481529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:28.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.701+0000 7fa24eef2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2480c2f30 con 0x7fa24814fda0 2023-03-03T12:35:28.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24eef2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2480c31d0 con 0x7fa24814f9d0 2023-03-03T12:35:28.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24eef2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2481524a0 con 0x7fa2480abe20 2023-03-03T12:35:28.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24cc8e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa24814f9d0 0x7fa248156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:28.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24cc8e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa24814f9d0 0x7fa248156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41038/0 (socket says 172.21.15.7:41038) 2023-03-03T12:35:28.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24cc8e700 1 -- 172.21.15.7:0/125394728 learned_addr learned my addr 172.21.15.7:0/125394728 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:35:28.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24d48f700 1 --2- 172.21.15.7:0/125394728 >> v2:172.21.15.143:3300/0 conn(0x7fa2480abe20 0x7fa2481529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:28.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24cc8e700 1 -- 172.21.15.7:0/125394728 >> v2:172.21.15.143:3300/0 conn(0x7fa2480abe20 msgr2=0x7fa2481529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:35:28.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24cc8e700 1 --2- 172.21.15.7:0/125394728 >> v2:172.21.15.143:3300/0 conn(0x7fa2480abe20 0x7fa2481529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:28.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24cc8e700 1 -- 172.21.15.7:0/125394728 >> v1:172.21.15.143:6789/0 conn(0x7fa24814fda0 legacy=0x7fa248071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:35:28.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24cc8e700 1 -- 172.21.15.7:0/125394728 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2480c34b0 con 0x7fa24814f9d0 2023-03-03T12:35:28.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa24cc8e700 1 --2- 172.21.15.7:0/125394728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24814f9d0 0x7fa248156430 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fa248071230 tx=0x7fa23c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ab1251e5d70bbb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:35:28.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa2477fe700 1 -- 172.21.15.7:0/125394728 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c004230 con 0x7fa24814f9d0 2023-03-03T12:35:28.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa2477fe700 1 -- 172.21.15.7:0/125394728 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c004390 con 0x7fa24814f9d0 2023-03-03T12:35:28.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.702+0000 7fa2477fe700 1 -- 172.21.15.7:0/125394728 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c0036a0 con 0x7fa24814f9d0 2023-03-03T12:35:28.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.703+0000 7fa24eef2700 1 -- 172.21.15.7:0/125394728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24814f9d0 msgr2=0x7fa248156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:28.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.703+0000 7fa24eef2700 1 --2- 172.21.15.7:0/125394728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24814f9d0 0x7fa248156430 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fa248071230 tx=0x7fa23c009bd0 comp rx=0 tx=0).stop 2023-03-03T12:35:28.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.703+0000 7fa24eef2700 1 -- 172.21.15.7:0/125394728 shutdown_connections 2023-03-03T12:35:28.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.703+0000 7fa24eef2700 1 --2- 172.21.15.7:0/125394728 >> v2:172.21.15.143:3300/0 conn(0x7fa2480abe20 0x7fa2481529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:28.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.703+0000 7fa24eef2700 1 --2- 172.21.15.7:0/125394728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24814f9d0 0x7fa248156430 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:28.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.703+0000 7fa24eef2700 1 -- 172.21.15.7:0/125394728 >> 172.21.15.7:0/125394728 conn(0x7fa2480c8900 msgr2=0x7fa2480c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:35:28.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.703+0000 7fa24eef2700 1 -- 172.21.15.7:0/125394728 shutdown_connections 2023-03-03T12:35:28.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.703+0000 7fa24eef2700 1 -- 172.21.15.7:0/125394728 wait complete. 2023-03-03T12:35:28.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.704+0000 7fa24eef2700 1 Processor -- start 2023-03-03T12:35:28.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.704+0000 7fa24eef2700 1 -- start start 2023-03-03T12:35:28.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24eef2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2480abe20 0x7fa248150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:28.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24eef2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24814f9d0 0x7fa24813dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:28.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24eef2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa24814fda0 0x7fa24813e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:28.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24eef2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2481dd430 con 0x7fa2480abe20 2023-03-03T12:35:28.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24eef2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24806dc00 con 0x7fa24814f9d0 2023-03-03T12:35:28.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24eef2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24813e710 con 0x7fa24814fda0 2023-03-03T12:35:28.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24cc8e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2480abe20 0x7fa248150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:28.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24cc8e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2480abe20 0x7fa248150e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41044/0 (socket says 172.21.15.7:41044) 2023-03-03T12:35:28.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24cc8e700 1 -- 172.21.15.7:0/2306337196 learned_addr learned my addr 172.21.15.7:0/2306337196 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:35:28.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa247fff700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24814f9d0 0x7fa24813dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:28.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24d48f700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa24814fda0 0x7fa24813e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:28.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24cc8e700 1 -- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa24814fda0 msgr2=0x7fa24813e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:28.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24cc8e700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa24814fda0 0x7fa24813e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:28.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24cc8e700 1 -- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24814f9d0 msgr2=0x7fa24813dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:28.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24cc8e700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24814f9d0 0x7fa24813dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:28.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa24cc8e700 1 -- 172.21.15.7:0/2306337196 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c009550 con 0x7fa2480abe20 2023-03-03T12:35:28.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.705+0000 7fa247fff700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24814f9d0 0x7fa24813dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:35:28.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.706+0000 7fa24cc8e700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2480abe20 0x7fa248150e80 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fa23c009130 tx=0x7fa23c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:35:28.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.706+0000 7fa2457fa700 1 -- 172.21.15.7:0/2306337196 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c011630 con 0x7fa2480abe20 2023-03-03T12:35:28.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.706+0000 7fa2457fa700 1 -- 172.21.15.7:0/2306337196 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c011790 con 0x7fa2480abe20 2023-03-03T12:35:28.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.706+0000 7fa24eef2700 1 -- 172.21.15.7:0/2306337196 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2480ac350 con 0x7fa2480abe20 2023-03-03T12:35:28.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.706+0000 7fa2457fa700 1 -- 172.21.15.7:0/2306337196 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c011b40 con 0x7fa2480abe20 2023-03-03T12:35:28.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.706+0000 7fa24eef2700 1 -- 172.21.15.7:0/2306337196 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2480ab630 con 0x7fa2480abe20 2023-03-03T12:35:28.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.707+0000 7fa24eef2700 1 -- 172.21.15.7:0/2306337196 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa248071fa0 con 0x7fa2480abe20 2023-03-03T12:35:28.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.707+0000 7fa2457fa700 1 -- 172.21.15.7:0/2306337196 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa23c017020 con 0x7fa2480abe20 2023-03-03T12:35:28.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.708+0000 7fa2457fa700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa22c0682f0 0x7fa22c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:28.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.708+0000 7fa2457fa700 1 -- 172.21.15.7:0/2306337196 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa23c087ad0 con 0x7fa2480abe20 2023-03-03T12:35:28.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.708+0000 7fa247fff700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa22c0682f0 0x7fa22c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:28.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.709+0000 7fa247fff700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa22c0682f0 0x7fa22c06a7a0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa2481501d0 tx=0x7fa234009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:35:28.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.712+0000 7fa2457fa700 1 -- 172.21.15.7:0/2306337196 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa23c053d80 con 0x7fa2480abe20 2023-03-03T12:35:28.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.950+0000 7fa24eef2700 1 -- 172.21.15.7:0/2306337196 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa24804e6e0 con 0x7fa2480abe20 2023-03-03T12:35:28.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.951+0000 7fa2457fa700 1 -- 172.21.15.7:0/2306337196 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa23c0573a0 con 0x7fa2480abe20 2023-03-03T12:35:28.953 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:35:28.953 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:35:28.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 -- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa22c0682f0 msgr2=0x7fa22c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:28.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa22c0682f0 0x7fa22c06a7a0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa2481501d0 tx=0x7fa234009040 comp rx=0 tx=0).stop 2023-03-03T12:35:28.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 -- 172.21.15.7:0/2306337196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2480abe20 msgr2=0x7fa248150e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:28.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2480abe20 0x7fa248150e80 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fa23c009130 tx=0x7fa23c004750 comp rx=0 tx=0).stop 2023-03-03T12:35:28.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 -- 172.21.15.7:0/2306337196 shutdown_connections 2023-03-03T12:35:28.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa24814fda0 0x7fa24813e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:28.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa22c0682f0 0x7fa22c06a7a0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:28.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2480abe20 0x7fa248150e80 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:28.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 --2- 172.21.15.7:0/2306337196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa24814f9d0 0x7fa24813dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:28.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 -- 172.21.15.7:0/2306337196 >> 172.21.15.7:0/2306337196 conn(0x7fa2480c8900 msgr2=0x7fa2480c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:35:28.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 -- 172.21.15.7:0/2306337196 shutdown_connections 2023-03-03T12:35:28.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:28.953+0000 7fa24eef2700 1 -- 172.21.15.7:0/2306337196 wait complete. 2023-03-03T12:35:28.957 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:35:28.967 INFO:tasks.mds_thrash.fs.[b]:waiting for 30 secs before thrashing 2023-03-03T12:35:30.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:30.355 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:35:32.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:35:32.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.598+0000 7f6d0e5be700 1 Processor -- start 2023-03-03T12:35:32.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.598+0000 7f6d0e5be700 1 -- start start 2023-03-03T12:35:32.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d0e5be700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d0814f9d0 0x7f6d08156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:32.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d0e5be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6d0814fda0 0x7f6d08071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:32.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d0e5be700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d080c2f30 con 0x7f6d080abe20 2023-03-03T12:35:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d0e5be700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d080c31d0 con 0x7f6d0814f9d0 2023-03-03T12:35:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d0e5be700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d081524a0 con 0x7f6d0814fda0 2023-03-03T12:35:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d07fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d0814f9d0 0x7f6d08156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d0cb5b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6d080abe20 0x7f6d081529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35034/0 (socket says 172.21.15.7:35034) 2023-03-03T12:35:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d0cb5b700 1 -- 172.21.15.7:0/3343067387 learned_addr learned my addr 172.21.15.7:0/3343067387 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:35:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d077fe700 1 --2- 172.21.15.7:0/3343067387 >> v2:172.21.15.143:3300/0 conn(0x7f6d0814fda0 0x7f6d08071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d07fff700 1 -- 172.21.15.7:0/3343067387 >> v2:172.21.15.143:3300/0 conn(0x7f6d0814fda0 msgr2=0x7f6d08071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:35:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d07fff700 1 --2- 172.21.15.7:0/3343067387 >> v2:172.21.15.143:3300/0 conn(0x7f6d0814fda0 0x7f6d08071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d07fff700 1 -- 172.21.15.7:0/3343067387 >> v1:172.21.15.7:6789/0 conn(0x7f6d080abe20 legacy=0x7f6d081529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d07fff700 1 -- 172.21.15.7:0/3343067387 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d080c34b0 con 0x7f6d0814f9d0 2023-03-03T12:35:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.599+0000 7f6d06ffd700 1 -- 172.21.15.7:0/3343067387 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453597444 0 0) 0x7f6d080c2f30 con 0x7f6d080abe20 2023-03-03T12:35:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d07fff700 1 --2- 172.21.15.7:0/3343067387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 0x7f6d08156430 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6d080709b0 tx=0x7f6cf8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2945b3da996305b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:35:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d06ffd700 1 -- 172.21.15.7:0/3343067387 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8011830 con 0x7f6d0814f9d0 2023-03-03T12:35:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d06ffd700 1 -- 172.21.15.7:0/3343067387 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf8011990 con 0x7f6d0814f9d0 2023-03-03T12:35:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d06ffd700 1 -- 172.21.15.7:0/3343067387 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8011d20 con 0x7f6d0814f9d0 2023-03-03T12:35:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d0e5be700 1 -- 172.21.15.7:0/3343067387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 msgr2=0x7f6d08156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d0e5be700 1 --2- 172.21.15.7:0/3343067387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 0x7f6d08156430 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6d080709b0 tx=0x7f6cf8009b20 comp rx=0 tx=0).stop 2023-03-03T12:35:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d0e5be700 1 -- 172.21.15.7:0/3343067387 shutdown_connections 2023-03-03T12:35:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d0e5be700 1 --2- 172.21.15.7:0/3343067387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 0x7f6d08156430 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d0e5be700 1 --2- 172.21.15.7:0/3343067387 >> v2:172.21.15.143:3300/0 conn(0x7f6d0814fda0 0x7f6d08071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d0e5be700 1 -- 172.21.15.7:0/3343067387 >> 172.21.15.7:0/3343067387 conn(0x7f6d080c8900 msgr2=0x7f6d080c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:35:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d0e5be700 1 -- 172.21.15.7:0/3343067387 shutdown_connections 2023-03-03T12:35:32.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.600+0000 7f6d0e5be700 1 -- 172.21.15.7:0/3343067387 wait complete. 2023-03-03T12:35:32.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.602+0000 7f6d0e5be700 1 Processor -- start 2023-03-03T12:35:32.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.602+0000 7f6d0e5be700 1 -- start start 2023-03-03T12:35:32.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d0e5be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d080abe20 0x7f6d081e7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:32.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d0e5be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 0x7f6d080bd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:32.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d0e5be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d0814fda0 0x7f6d080bfa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:32.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d0e5be700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080c07a0 con 0x7f6d0814f9d0 2023-03-03T12:35:32.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d0e5be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0806d510 con 0x7f6d080abe20 2023-03-03T12:35:32.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d0e5be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080bde50 con 0x7f6d0814fda0 2023-03-03T12:35:32.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d077fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 0x7f6d080bd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:32.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d077fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 0x7f6d080bd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41058/0 (socket says 172.21.15.7:41058) 2023-03-03T12:35:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d077fe700 1 -- 172.21.15.7:0/4260946175 learned_addr learned my addr 172.21.15.7:0/4260946175 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:35:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d07fff700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d080abe20 0x7f6d081e7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d0cb5b700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d0814fda0 0x7f6d080bfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d077fe700 1 -- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d0814fda0 msgr2=0x7f6d080bfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d077fe700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d0814fda0 0x7f6d080bfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d077fe700 1 -- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d080abe20 msgr2=0x7f6d081e7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.603+0000 7f6d077fe700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d080abe20 0x7f6d081e7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.604+0000 7f6d077fe700 1 -- 172.21.15.7:0/4260946175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf80094a0 con 0x7f6d0814f9d0 2023-03-03T12:35:32.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.604+0000 7f6d0cb5b700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d0814fda0 0x7f6d080bfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:35:32.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.604+0000 7f6d077fe700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 0x7f6d080bd640 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000e810 tx=0x7f6cf000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:35:32.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.604+0000 7f6d04ff9700 1 -- 172.21.15.7:0/4260946175 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0013610 con 0x7f6d0814f9d0 2023-03-03T12:35:32.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.604+0000 7f6d04ff9700 1 -- 172.21.15.7:0/4260946175 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf0013770 con 0x7f6d0814f9d0 2023-03-03T12:35:32.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.604+0000 7f6d0e5be700 1 -- 172.21.15.7:0/4260946175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d080ac350 con 0x7f6d0814f9d0 2023-03-03T12:35:32.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.604+0000 7f6d04ff9700 1 -- 172.21.15.7:0/4260946175 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0013b00 con 0x7f6d0814f9d0 2023-03-03T12:35:32.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.604+0000 7f6d0e5be700 1 -- 172.21.15.7:0/4260946175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d080ab630 con 0x7f6d0814f9d0 2023-03-03T12:35:32.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.605+0000 7f6d0e5be700 1 -- 172.21.15.7:0/4260946175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d080c3e10 con 0x7f6d0814f9d0 2023-03-03T12:35:32.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.605+0000 7f6d04ff9700 1 -- 172.21.15.7:0/4260946175 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6cf0023070 con 0x7f6d0814f9d0 2023-03-03T12:35:32.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.606+0000 7f6d04ff9700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ce80682f0 0x7f6ce806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:32.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.606+0000 7f6d04ff9700 1 -- 172.21.15.7:0/4260946175 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6cf0085ee0 con 0x7f6d0814f9d0 2023-03-03T12:35:32.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.606+0000 7f6d07fff700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ce80682f0 0x7f6ce806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:32.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.607+0000 7f6d07fff700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ce80682f0 0x7f6ce806a7a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80090b0 tx=0x7f6cf8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:35:32.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.611+0000 7f6d04ff9700 1 -- 172.21.15.7:0/4260946175 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6cf0052190 con 0x7f6d0814f9d0 2023-03-03T12:35:32.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.847+0000 7f6d0e5be700 1 -- 172.21.15.7:0/4260946175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d08074bb0 con 0x7f6d0814f9d0 2023-03-03T12:35:32.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.847+0000 7f6d04ff9700 1 -- 172.21.15.7:0/4260946175 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6cf00557b0 con 0x7f6d0814f9d0 2023-03-03T12:35:32.849 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:35:32.849 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:35:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 -- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ce80682f0 msgr2=0x7f6ce806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ce80682f0 0x7f6ce806a7a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80090b0 tx=0x7f6cf8008030 comp rx=0 tx=0).stop 2023-03-03T12:35:32.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 -- 172.21.15.7:0/4260946175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 msgr2=0x7f6d080bd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:32.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 0x7f6d080bd640 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000e810 tx=0x7f6cf000ebd0 comp rx=0 tx=0).stop 2023-03-03T12:35:32.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 -- 172.21.15.7:0/4260946175 shutdown_connections 2023-03-03T12:35:32.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6d0814fda0 0x7f6d080bfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:32.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6d080abe20 0x7f6d081e7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:32.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ce80682f0 0x7f6ce806a7a0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:32.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 --2- 172.21.15.7:0/4260946175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d0814f9d0 0x7f6d080bd640 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:32.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 -- 172.21.15.7:0/4260946175 >> 172.21.15.7:0/4260946175 conn(0x7f6d080c8900 msgr2=0x7f6d081515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:35:32.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 -- 172.21.15.7:0/4260946175 shutdown_connections 2023-03-03T12:35:32.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:32.849+0000 7f6d0e5be700 1 -- 172.21.15.7:0/4260946175 wait complete. 2023-03-03T12:35:32.853 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:35:32.863 INFO:tasks.mds_thrash.fs.[a]:waiting for 107 secs before thrashing 2023-03-03T12:35:58.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:35:59.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.106+0000 7f5a0a91a700 1 Processor -- start 2023-03-03T12:35:59.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.106+0000 7f5a0a91a700 1 -- start start 2023-03-03T12:35:59.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a0a91a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5a0414ffa0 0x7f5a0415dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:59.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a0a91a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a040acb10 con 0x7f5a04073960 2023-03-03T12:35:59.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a0a91a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a04155bd0 con 0x7f5a04073d30 2023-03-03T12:35:59.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a0a91a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a040c2b70 con 0x7f5a0414ffa0 2023-03-03T12:35:59.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a03fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5a04073960 0x7f5a041524e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35066/0 (socket says 172.21.15.7:35066) 2023-03-03T12:35:59.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a03fff700 1 -- 172.21.15.7:0/3889551029 learned_addr learned my addr 172.21.15.7:0/3889551029 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:35:59.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a08eb7700 1 --2- 172.21.15.7:0/3889551029 >> v2:172.21.15.143:3300/0 conn(0x7f5a0414ffa0 0x7f5a0415dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:59.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3889551029 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830646750 0 0) 0x7f5a040acb10 con 0x7f5a04073960 2023-03-03T12:35:59.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3889551029 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f00035b0 con 0x7f5a04073960 2023-03-03T12:35:59.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3889551029 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377190250 0 0) 0x7f5a04155bd0 con 0x7f5a04073d30 2023-03-03T12:35:59.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.107+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3889551029 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a040acb10 con 0x7f5a04073d30 2023-03-03T12:35:59.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.108+0000 7f5a08eb7700 1 -- 172.21.15.7:0/3889551029 >> v1:172.21.15.143:6789/0 conn(0x7f5a04073d30 legacy=0x7f5a0414f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:59.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.108+0000 7f5a08eb7700 1 -- 172.21.15.7:0/3889551029 >> v1:172.21.15.7:6789/0 conn(0x7f5a04073960 legacy=0x7f5a041524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:59.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.108+0000 7f5a08eb7700 1 -- 172.21.15.7:0/3889551029 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a04155eb0 con 0x7f5a0414ffa0 2023-03-03T12:35:59.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.108+0000 7f5a08eb7700 1 --2- 172.21.15.7:0/3889551029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0414ffa0 0x7f5a0415dd90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009d90 tx=0x7f59fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa787bc4ee2c51ff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:35:59.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.108+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3889551029 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc00e070 con 0x7f5a0414ffa0 2023-03-03T12:35:59.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.108+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3889551029 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc00bb20 con 0x7f5a0414ffa0 2023-03-03T12:35:59.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.108+0000 7f5a02ffd700 1 -- 172.21.15.7:0/3889551029 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc012430 con 0x7f5a0414ffa0 2023-03-03T12:35:59.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.108+0000 7f5a0a91a700 1 -- 172.21.15.7:0/3889551029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0414ffa0 msgr2=0x7f5a0415dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:59.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.108+0000 7f5a0a91a700 1 --2- 172.21.15.7:0/3889551029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0414ffa0 0x7f5a0415dd90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009d90 tx=0x7f59fc00b330 comp rx=0 tx=0).stop 2023-03-03T12:35:59.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.109+0000 7f5a0a91a700 1 -- 172.21.15.7:0/3889551029 shutdown_connections 2023-03-03T12:35:59.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.109+0000 7f5a0a91a700 1 --2- 172.21.15.7:0/3889551029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a0414ffa0 0x7f5a0415dd90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:59.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.109+0000 7f5a0a91a700 1 -- 172.21.15.7:0/3889551029 >> 172.21.15.7:0/3889551029 conn(0x7f5a040a94d0 msgr2=0x7f5a040a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:35:59.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.109+0000 7f5a0a91a700 1 -- 172.21.15.7:0/3889551029 shutdown_connections 2023-03-03T12:35:59.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.109+0000 7f5a0a91a700 1 -- 172.21.15.7:0/3889551029 wait complete. 2023-03-03T12:35:59.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.110+0000 7f5a0a91a700 1 Processor -- start 2023-03-03T12:35:59.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.110+0000 7f5a0a91a700 1 -- start start 2023-03-03T12:35:59.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a0a91a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a04073960 0x7f5a041eadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:59.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a0a91a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a04073d30 0x7f5a041e8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:59.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a0a91a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0414ffa0 0x7f5a041e8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:59.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a0a91a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a041eb930 con 0x7f5a04073960 2023-03-03T12:35:59.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a0a91a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a041eb690 con 0x7f5a04073d30 2023-03-03T12:35:59.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a0a91a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a040c0160 con 0x7f5a0414ffa0 2023-03-03T12:35:59.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a03fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a04073960 0x7f5a041eadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:59.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a03fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a04073960 0x7f5a041eadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41092/0 (socket says 172.21.15.7:41092) 2023-03-03T12:35:59.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a03fff700 1 -- 172.21.15.7:0/418226867 learned_addr learned my addr 172.21.15.7:0/418226867 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:35:59.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a037fe700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a04073d30 0x7f5a041e8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:59.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a08eb7700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0414ffa0 0x7f5a041e8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:59.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a03fff700 1 -- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0414ffa0 msgr2=0x7f5a041e8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:59.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a03fff700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0414ffa0 0x7f5a041e8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:59.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a03fff700 1 -- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a04073d30 msgr2=0x7f5a041e8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:59.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a03fff700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a04073d30 0x7f5a041e8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:59.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a03fff700 1 -- 172.21.15.7:0/418226867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc009a70 con 0x7f5a04073960 2023-03-03T12:35:59.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a03fff700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a04073960 0x7f5a041eadc0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f59f400c700 tx=0x7f59f400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:35:59.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a00ff9700 1 -- 172.21.15.7:0/418226867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4005d70 con 0x7f5a04073960 2023-03-03T12:35:59.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a00ff9700 1 -- 172.21.15.7:0/418226867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f40070d0 con 0x7f5a04073960 2023-03-03T12:35:59.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.111+0000 7f5a0a91a700 1 -- 172.21.15.7:0/418226867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a040c1ce0 con 0x7f5a04073960 2023-03-03T12:35:59.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.112+0000 7f5a00ff9700 1 -- 172.21.15.7:0/418226867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4007460 con 0x7f5a04073960 2023-03-03T12:35:59.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.112+0000 7f5a0a91a700 1 -- 172.21.15.7:0/418226867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a04156190 con 0x7f5a04073960 2023-03-03T12:35:59.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.112+0000 7f5a0a91a700 1 -- 172.21.15.7:0/418226867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a040c1e70 con 0x7f5a04073960 2023-03-03T12:35:59.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.113+0000 7f5a00ff9700 1 -- 172.21.15.7:0/418226867 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f59f4019030 con 0x7f5a04073960 2023-03-03T12:35:59.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.113+0000 7f5a00ff9700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59e40682f0 0x7f59e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:35:59.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.113+0000 7f5a00ff9700 1 -- 172.21.15.7:0/418226867 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f59f4085470 con 0x7f5a04073960 2023-03-03T12:35:59.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.114+0000 7f5a037fe700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59e40682f0 0x7f59e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:35:59.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.114+0000 7f5a037fe700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59e40682f0 0x7f59e406a7a0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f5a040c3a70 tx=0x7f59ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:35:59.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.118+0000 7f5a00ff9700 1 -- 172.21.15.7:0/418226867 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f59f4051790 con 0x7f5a04073960 2023-03-03T12:35:59.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.360+0000 7f5a0a91a700 1 -- 172.21.15.7:0/418226867 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a040bffc0 con 0x7f5a04073960 2023-03-03T12:35:59.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.361+0000 7f5a00ff9700 1 -- 172.21.15.7:0/418226867 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5a040bffc0 con 0x7f5a04073960 2023-03-03T12:35:59.364 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:35:59.364 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:35:59.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 -- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59e40682f0 msgr2=0x7f59e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:59.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59e40682f0 0x7f59e406a7a0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f5a040c3a70 tx=0x7f59ec009040 comp rx=0 tx=0).stop 2023-03-03T12:35:59.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 -- 172.21.15.7:0/418226867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a04073960 msgr2=0x7f5a041eadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:35:59.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a04073960 0x7f5a041eadc0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f59f400c700 tx=0x7f59f400ca10 comp rx=0 tx=0).stop 2023-03-03T12:35:59.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 -- 172.21.15.7:0/418226867 shutdown_connections 2023-03-03T12:35:59.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a0414ffa0 0x7f5a041e8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:59.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59e40682f0 0x7f59e406a7a0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:59.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a04073960 0x7f5a041eadc0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:59.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 --2- 172.21.15.7:0/418226867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a04073d30 0x7f5a041e8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:35:59.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 -- 172.21.15.7:0/418226867 >> 172.21.15.7:0/418226867 conn(0x7f5a040a94d0 msgr2=0x7f5a0414e860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:35:59.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 -- 172.21.15.7:0/418226867 shutdown_connections 2023-03-03T12:35:59.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:35:59.363+0000 7f5a0a91a700 1 -- 172.21.15.7:0/418226867 wait complete. 2023-03-03T12:35:59.368 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:35:59.378 INFO:tasks.mds_thrash.fs.[b]:waiting for 87 secs before thrashing 2023-03-03T12:36:00.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:00.478 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:30.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:36:30.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:00.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:00.689 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:19.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:37:20.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.001+0000 7f560c47d700 1 Processor -- start 2023-03-03T12:37:20.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.001+0000 7f560c47d700 1 -- start start 2023-03-03T12:37:20.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.002+0000 7f560c47d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5604152e30 0x7f56040c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:20.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.002+0000 7f560c47d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 0x7f56040c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:20.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.002+0000 7f560c47d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5604155d80 0x7f5604156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:20.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.002+0000 7f560c47d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56041523b0 con 0x7f5604155d80 2023-03-03T12:37:20.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.002+0000 7f560c47d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5604072010 con 0x7f5604152e30 2023-03-03T12:37:20.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.002+0000 7f560c47d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56040aaaf0 con 0x7f56040beab0 2023-03-03T12:37:20.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f560aa1a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5604155d80 0x7f5604156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:20.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f560aa1a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5604155d80 0x7f5604156220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41132/0 (socket says 172.21.15.7:41132) 2023-03-03T12:37:20.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f560aa1a700 1 -- 172.21.15.7:0/374796619 learned_addr learned my addr 172.21.15.7:0/374796619 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:37:20.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f560a219700 1 --2- 172.21.15.7:0/374796619 >> v2:172.21.15.143:3300/0 conn(0x7f5604152e30 0x7f56040c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:20.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f5609a18700 1 --2- 172.21.15.7:0/374796619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 0x7f56040c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:20.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f560aa1a700 1 -- 172.21.15.7:0/374796619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 msgr2=0x7f56040c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:37:20.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f560aa1a700 1 --2- 172.21.15.7:0/374796619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 0x7f56040c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f560aa1a700 1 -- 172.21.15.7:0/374796619 >> v2:172.21.15.143:3300/0 conn(0x7f5604152e30 msgr2=0x7f56040c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:37:20.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f560aa1a700 1 --2- 172.21.15.7:0/374796619 >> v2:172.21.15.143:3300/0 conn(0x7f5604152e30 0x7f56040c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f560aa1a700 1 -- 172.21.15.7:0/374796619 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56040aadd0 con 0x7f5604155d80 2023-03-03T12:37:20.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f560aa1a700 1 --2- 172.21.15.7:0/374796619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604155d80 0x7f5604156220 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f56040c60b0 tx=0x7f55fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311557811bf303ef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:20.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f5609217700 1 -- 172.21.15.7:0/374796619 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc00e070 con 0x7f5604155d80 2023-03-03T12:37:20.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f5609217700 1 -- 172.21.15.7:0/374796619 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55fc00b9c0 con 0x7f5604155d80 2023-03-03T12:37:20.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.003+0000 7f5609217700 1 -- 172.21.15.7:0/374796619 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc00bd70 con 0x7f5604155d80 2023-03-03T12:37:20.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.004+0000 7f560c47d700 1 -- 172.21.15.7:0/374796619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604155d80 msgr2=0x7f5604156220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:20.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.004+0000 7f560c47d700 1 --2- 172.21.15.7:0/374796619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604155d80 0x7f5604156220 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f56040c60b0 tx=0x7f55fc00b330 comp rx=0 tx=0).stop 2023-03-03T12:37:20.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.004+0000 7f560c47d700 1 -- 172.21.15.7:0/374796619 shutdown_connections 2023-03-03T12:37:20.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.004+0000 7f560c47d700 1 --2- 172.21.15.7:0/374796619 >> v2:172.21.15.143:3300/0 conn(0x7f5604152e30 0x7f56040c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.004+0000 7f560c47d700 1 --2- 172.21.15.7:0/374796619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604155d80 0x7f5604156220 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.004+0000 7f560c47d700 1 --2- 172.21.15.7:0/374796619 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 0x7f56040c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.004+0000 7f560c47d700 1 -- 172.21.15.7:0/374796619 >> 172.21.15.7:0/374796619 conn(0x7f56040736f0 msgr2=0x7f5604073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:37:20.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.004+0000 7f560c47d700 1 -- 172.21.15.7:0/374796619 shutdown_connections 2023-03-03T12:37:20.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.004+0000 7f560c47d700 1 -- 172.21.15.7:0/374796619 wait complete. 2023-03-03T12:37:20.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.005+0000 7f560c47d700 1 Processor -- start 2023-03-03T12:37:20.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.005+0000 7f560c47d700 1 -- start start 2023-03-03T12:37:20.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f560c47d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 0x7f56041e7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:20.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f560c47d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604152e30 0x7f56040c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:20.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f560c47d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5604155d80 0x7f5604146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:20.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f560c47d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56040ad020 con 0x7f5604152e30 2023-03-03T12:37:20.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f560c47d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56040ac780 con 0x7f5604155d80 2023-03-03T12:37:20.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f560c47d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56040aca60 con 0x7f56040beab0 2023-03-03T12:37:20.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f5609a18700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604152e30 0x7f56040c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:20.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f5609a18700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604152e30 0x7f56040c75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41136/0 (socket says 172.21.15.7:41136) 2023-03-03T12:37:20.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f5609a18700 1 -- 172.21.15.7:0/1600832178 learned_addr learned my addr 172.21.15.7:0/1600832178 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:37:20.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f560a219700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 0x7f56041e7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:20.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f560aa1a700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5604155d80 0x7f5604146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:20.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f5609a18700 1 -- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 msgr2=0x7f56041e7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:20.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f5609a18700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 0x7f56041e7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f5609a18700 1 -- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5604155d80 msgr2=0x7f5604146b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:20.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f5609a18700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5604155d80 0x7f5604146b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f5609a18700 1 -- 172.21.15.7:0/1600832178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc009b30 con 0x7f5604152e30 2023-03-03T12:37:20.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f560a219700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 0x7f56041e7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:37:20.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.006+0000 7f5609a18700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604152e30 0x7f56040c75f0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f560000d800 tx=0x7f560000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:20.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.007+0000 7f55f6ffd700 1 -- 172.21.15.7:0/1600832178 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56000125f0 con 0x7f5604152e30 2023-03-03T12:37:20.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.007+0000 7f55f6ffd700 1 -- 172.21.15.7:0/1600832178 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5600012750 con 0x7f5604152e30 2023-03-03T12:37:20.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.007+0000 7f560c47d700 1 -- 172.21.15.7:0/1600832178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5604156760 con 0x7f5604152e30 2023-03-03T12:37:20.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.007+0000 7f55f6ffd700 1 -- 172.21.15.7:0/1600832178 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600012ae0 con 0x7f5604152e30 2023-03-03T12:37:20.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.007+0000 7f560c47d700 1 -- 172.21.15.7:0/1600832178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5604071c70 con 0x7f5604152e30 2023-03-03T12:37:20.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.008+0000 7f560c47d700 1 -- 172.21.15.7:0/1600832178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56041568f0 con 0x7f5604152e30 2023-03-03T12:37:20.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.008+0000 7f55f6ffd700 1 -- 172.21.15.7:0/1600832178 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5600012c40 con 0x7f5604152e30 2023-03-03T12:37:20.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.008+0000 7f55f6ffd700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55e80682f0 0x7f55e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:20.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.009+0000 7f55f6ffd700 1 -- 172.21.15.7:0/1600832178 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5600086330 con 0x7f5604152e30 2023-03-03T12:37:20.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.009+0000 7f560a219700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55e80682f0 0x7f55e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:20.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.009+0000 7f560a219700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55e80682f0 0x7f55e806a7a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f55f80099f0 tx=0x7f55f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:20.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.013+0000 7f55f6ffd700 1 -- 172.21.15.7:0/1600832178 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5600052650 con 0x7f5604152e30 2023-03-03T12:37:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.254+0000 7f560c47d700 1 -- 172.21.15.7:0/1600832178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5604074bb0 con 0x7f5604152e30 2023-03-03T12:37:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.254+0000 7f55f6ffd700 1 -- 172.21.15.7:0/1600832178 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5600055c70 con 0x7f5604152e30 2023-03-03T12:37:20.256 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:37:20.257 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:37:20.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 -- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55e80682f0 msgr2=0x7f55e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:20.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55e80682f0 0x7f55e806a7a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f55f80099f0 tx=0x7f55f8008040 comp rx=0 tx=0).stop 2023-03-03T12:37:20.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 -- 172.21.15.7:0/1600832178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604152e30 msgr2=0x7f56040c75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:20.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604152e30 0x7f56040c75f0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f560000d800 tx=0x7f560000db10 comp rx=0 tx=0).stop 2023-03-03T12:37:20.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 -- 172.21.15.7:0/1600832178 shutdown_connections 2023-03-03T12:37:20.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5604155d80 0x7f5604146b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56040beab0 0x7f56041e7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55e80682f0 0x7f55e806a7a0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 --2- 172.21.15.7:0/1600832178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604152e30 0x7f56040c75f0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:20.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 -- 172.21.15.7:0/1600832178 >> 172.21.15.7:0/1600832178 conn(0x7f56040736f0 msgr2=0x7f56040a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:37:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 -- 172.21.15.7:0/1600832178 shutdown_connections 2023-03-03T12:37:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:20.256+0000 7f560c47d700 1 -- 172.21.15.7:0/1600832178 wait complete. 2023-03-03T12:37:20.260 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:37:20.270 INFO:tasks.mds_thrash.fs.[a]:waiting for 20 secs before thrashing 2023-03-03T12:37:26.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:37:26.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.517+0000 7f6e0df13700 1 Processor -- start 2023-03-03T12:37:26.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.517+0000 7f6e0df13700 1 -- start start 2023-03-03T12:37:26.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e0df13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e08152e30 0x7f6e080c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:26.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e0df13700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e080c4910 0x7f6e08156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e0df13700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e081523b0 con 0x7f6e080beab0 2023-03-03T12:37:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e0df13700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e08072010 con 0x7f6e080c4910 2023-03-03T12:37:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e0df13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080aaaf0 con 0x7f6e08152e30 2023-03-03T12:37:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e07fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e080c4910 0x7f6e08156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e06ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6e080beab0 0x7f6e08155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35124/0 (socket says 172.21.15.7:35124) 2023-03-03T12:37:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e06ffd700 1 -- 172.21.15.7:0/2510208657 learned_addr learned my addr 172.21.15.7:0/2510208657 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:37:26.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e077fe700 1 --2- 172.21.15.7:0/2510208657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e08152e30 0x7f6e080c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:26.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e07fff700 1 -- 172.21.15.7:0/2510208657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e08152e30 msgr2=0x7f6e080c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:37:26.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e07fff700 1 --2- 172.21.15.7:0/2510208657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e08152e30 0x7f6e080c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:26.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e07fff700 1 -- 172.21.15.7:0/2510208657 >> v1:172.21.15.7:6789/0 conn(0x7f6e080beab0 legacy=0x7f6e08155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:26.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e067fc700 1 -- 172.21.15.7:0/2510208657 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612169297 0 0) 0x7f6e081523b0 con 0x7f6e080beab0 2023-03-03T12:37:26.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e07fff700 1 -- 172.21.15.7:0/2510208657 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e080aadd0 con 0x7f6e080c4910 2023-03-03T12:37:26.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e07fff700 1 --2- 172.21.15.7:0/2510208657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080c4910 0x7f6e08156490 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009d90 tx=0x7f6e0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b48eb4a13b6ad1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:26.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e067fc700 1 -- 172.21.15.7:0/2510208657 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0000e070 con 0x7f6e080c4910 2023-03-03T12:37:26.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e067fc700 1 -- 172.21.15.7:0/2510208657 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e0000bbd0 con 0x7f6e080c4910 2023-03-03T12:37:26.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.518+0000 7f6e067fc700 1 -- 172.21.15.7:0/2510208657 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00012430 con 0x7f6e080c4910 2023-03-03T12:37:26.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.519+0000 7f6e0df13700 1 -- 172.21.15.7:0/2510208657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080c4910 msgr2=0x7f6e08156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:26.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.519+0000 7f6e0df13700 1 --2- 172.21.15.7:0/2510208657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080c4910 0x7f6e08156490 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009d90 tx=0x7f6e0000b330 comp rx=0 tx=0).stop 2023-03-03T12:37:26.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.519+0000 7f6e0df13700 1 -- 172.21.15.7:0/2510208657 shutdown_connections 2023-03-03T12:37:26.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.519+0000 7f6e0df13700 1 --2- 172.21.15.7:0/2510208657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080c4910 0x7f6e08156490 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:26.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.519+0000 7f6e0df13700 1 --2- 172.21.15.7:0/2510208657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e08152e30 0x7f6e080c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:26.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.519+0000 7f6e0df13700 1 -- 172.21.15.7:0/2510208657 >> 172.21.15.7:0/2510208657 conn(0x7f6e080736f0 msgr2=0x7f6e08073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:37:26.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.519+0000 7f6e0df13700 1 -- 172.21.15.7:0/2510208657 shutdown_connections 2023-03-03T12:37:26.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.519+0000 7f6e0df13700 1 -- 172.21.15.7:0/2510208657 wait complete. 2023-03-03T12:37:26.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.520+0000 7f6e0df13700 1 Processor -- start 2023-03-03T12:37:26.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.520+0000 7f6e0df13700 1 -- start start 2023-03-03T12:37:26.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e0df13700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080beab0 0x7f6e080ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:26.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e0df13700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e080c4910 0x7f6e08154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:26.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e0df13700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e08152e30 0x7f6e081e8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:26.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e0df13700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e081514c0 con 0x7f6e080beab0 2023-03-03T12:37:26.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e0df13700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e081e7d70 con 0x7f6e08152e30 2023-03-03T12:37:26.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e0df13700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e081e7a40 con 0x7f6e080c4910 2023-03-03T12:37:26.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e077fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080beab0 0x7f6e080ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:26.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e077fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080beab0 0x7f6e080ab460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41150/0 (socket says 172.21.15.7:41150) 2023-03-03T12:37:26.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e077fe700 1 -- 172.21.15.7:0/3731442057 learned_addr learned my addr 172.21.15.7:0/3731442057 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:37:26.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e06ffd700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e080c4910 0x7f6e08154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:26.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.521+0000 7f6e07fff700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e08152e30 0x7f6e081e8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:26.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6e077fe700 1 -- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e080c4910 msgr2=0x7f6e08154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:26.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6e077fe700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e080c4910 0x7f6e08154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:26.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6e077fe700 1 -- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e08152e30 msgr2=0x7f6e081e8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:26.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6e077fe700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e08152e30 0x7f6e081e8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:26.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6e077fe700 1 -- 172.21.15.7:0/3731442057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e00009a70 con 0x7f6e080beab0 2023-03-03T12:37:26.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6e06ffd700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e080c4910 0x7f6e08154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:37:26.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6e077fe700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080beab0 0x7f6e080ab460 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f6df800e810 tx=0x7f6df800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:26.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6deffff700 1 -- 172.21.15.7:0/3731442057 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8013610 con 0x7f6e080beab0 2023-03-03T12:37:26.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6deffff700 1 -- 172.21.15.7:0/3731442057 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df8013770 con 0x7f6e080beab0 2023-03-03T12:37:26.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6e0df13700 1 -- 172.21.15.7:0/3731442057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e08153200 con 0x7f6e080beab0 2023-03-03T12:37:26.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6deffff700 1 -- 172.21.15.7:0/3731442057 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8013b20 con 0x7f6e080beab0 2023-03-03T12:37:26.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.522+0000 7f6e0df13700 1 -- 172.21.15.7:0/3731442057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e08071c70 con 0x7f6e080beab0 2023-03-03T12:37:26.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.523+0000 7f6e0df13700 1 -- 172.21.15.7:0/3731442057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e08153390 con 0x7f6e080beab0 2023-03-03T12:37:26.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.523+0000 7f6deffff700 1 -- 172.21.15.7:0/3731442057 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6df8023070 con 0x7f6e080beab0 2023-03-03T12:37:26.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.524+0000 7f6deffff700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df40682f0 0x7f6df406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:26.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.524+0000 7f6deffff700 1 -- 172.21.15.7:0/3731442057 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6df8085e50 con 0x7f6e080beab0 2023-03-03T12:37:26.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.524+0000 7f6e06ffd700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df40682f0 0x7f6df406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:26.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.524+0000 7f6e06ffd700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df40682f0 0x7f6df406a7a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6e080c5910 tx=0x7f6dfc007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:26.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.528+0000 7f6deffff700 1 -- 172.21.15.7:0/3731442057 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6df8052170 con 0x7f6e080beab0 2023-03-03T12:37:26.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.767+0000 7f6e0df13700 1 -- 172.21.15.7:0/3731442057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e081e8050 con 0x7f6e080beab0 2023-03-03T12:37:26.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.768+0000 7f6deffff700 1 -- 172.21.15.7:0/3731442057 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6df8055790 con 0x7f6e080beab0 2023-03-03T12:37:26.770 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:37:26.770 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:37:26.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 -- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df40682f0 msgr2=0x7f6df406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:26.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df40682f0 0x7f6df406a7a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6e080c5910 tx=0x7f6dfc007d30 comp rx=0 tx=0).stop 2023-03-03T12:37:26.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 -- 172.21.15.7:0/3731442057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080beab0 msgr2=0x7f6e080ab460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:26.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080beab0 0x7f6e080ab460 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f6df800e810 tx=0x7f6df800ebd0 comp rx=0 tx=0).stop 2023-03-03T12:37:26.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 -- 172.21.15.7:0/3731442057 shutdown_connections 2023-03-03T12:37:26.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e08152e30 0x7f6e081e8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:26.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6df40682f0 0x7f6df406a7a0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:26.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e080beab0 0x7f6e080ab460 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:26.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 --2- 172.21.15.7:0/3731442057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e080c4910 0x7f6e08154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:26.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 -- 172.21.15.7:0/3731442057 >> 172.21.15.7:0/3731442057 conn(0x7f6e080736f0 msgr2=0x7f6e08073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:37:26.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 -- 172.21.15.7:0/3731442057 shutdown_connections 2023-03-03T12:37:26.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:26.770+0000 7f6e0df13700 1 -- 172.21.15.7:0/3731442057 wait complete. 2023-03-03T12:37:26.774 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:37:26.784 INFO:tasks.mds_thrash.fs.[b]:waiting for 58 secs before thrashing 2023-03-03T12:37:30.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:30.759 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:37:40.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:37:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.410+0000 7f252aaf8700 1 Processor -- start 2023-03-03T12:37:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.410+0000 7f252aaf8700 1 -- start start 2023-03-03T12:37:40.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f252aaf8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f252414f9d0 0x7f2524156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:40.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f252aaf8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f252414fda0 0x7f2524071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:40.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f252aaf8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25240abe20 0x7f25241529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:40.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f252aaf8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25240c2f30 con 0x7f252414f9d0 2023-03-03T12:37:40.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f252aaf8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25240c31d0 con 0x7f252414fda0 2023-03-03T12:37:40.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f252aaf8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25241524a0 con 0x7f25240abe20 2023-03-03T12:37:40.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2528894700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f252414f9d0 0x7f2524156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:40.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2528894700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f252414f9d0 0x7f2524156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41180/0 (socket says 172.21.15.7:41180) 2023-03-03T12:37:40.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2528894700 1 -- 172.21.15.7:0/2207036655 learned_addr learned my addr 172.21.15.7:0/2207036655 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:37:40.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2523fff700 1 --2- 172.21.15.7:0/2207036655 >> v2:172.21.15.143:3300/0 conn(0x7f252414fda0 0x7f2524071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:40.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2529095700 1 --2- 172.21.15.7:0/2207036655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25240abe20 0x7f25241529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:40.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2528894700 1 -- 172.21.15.7:0/2207036655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25240abe20 msgr2=0x7f25241529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:37:40.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2528894700 1 --2- 172.21.15.7:0/2207036655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25240abe20 0x7f25241529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2528894700 1 -- 172.21.15.7:0/2207036655 >> v2:172.21.15.143:3300/0 conn(0x7f252414fda0 msgr2=0x7f2524071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:37:40.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2528894700 1 --2- 172.21.15.7:0/2207036655 >> v2:172.21.15.143:3300/0 conn(0x7f252414fda0 0x7f2524071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2528894700 1 -- 172.21.15.7:0/2207036655 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25240c34b0 con 0x7f252414f9d0 2023-03-03T12:37:40.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2523fff700 1 --2- 172.21.15.7:0/2207036655 >> v2:172.21.15.143:3300/0 conn(0x7f252414fda0 0x7f2524071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:37:40.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2529095700 1 --2- 172.21.15.7:0/2207036655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25240abe20 0x7f25241529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:37:40.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.411+0000 7f2528894700 1 --2- 172.21.15.7:0/2207036655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414f9d0 0x7f2524156430 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f25240709b0 tx=0x7f2518009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3df1b3fc98b14e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:40.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f25237fe700 1 -- 172.21.15.7:0/2207036655 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251800b870 con 0x7f252414f9d0 2023-03-03T12:37:40.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f25237fe700 1 -- 172.21.15.7:0/2207036655 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251800b9d0 con 0x7f252414f9d0 2023-03-03T12:37:40.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f25237fe700 1 -- 172.21.15.7:0/2207036655 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251800bd60 con 0x7f252414f9d0 2023-03-03T12:37:40.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f252aaf8700 1 -- 172.21.15.7:0/2207036655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414f9d0 msgr2=0x7f2524156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:40.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f252aaf8700 1 --2- 172.21.15.7:0/2207036655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414f9d0 0x7f2524156430 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f25240709b0 tx=0x7f2518009bd0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f252aaf8700 1 -- 172.21.15.7:0/2207036655 shutdown_connections 2023-03-03T12:37:40.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f252aaf8700 1 --2- 172.21.15.7:0/2207036655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25240abe20 0x7f25241529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f252aaf8700 1 --2- 172.21.15.7:0/2207036655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414f9d0 0x7f2524156430 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f252aaf8700 1 --2- 172.21.15.7:0/2207036655 >> v2:172.21.15.143:3300/0 conn(0x7f252414fda0 0x7f2524071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f252aaf8700 1 -- 172.21.15.7:0/2207036655 >> 172.21.15.7:0/2207036655 conn(0x7f25240c8900 msgr2=0x7f25240c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:37:40.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f252aaf8700 1 -- 172.21.15.7:0/2207036655 shutdown_connections 2023-03-03T12:37:40.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.412+0000 7f252aaf8700 1 -- 172.21.15.7:0/2207036655 wait complete. 2023-03-03T12:37:40.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.413+0000 7f252aaf8700 1 Processor -- start 2023-03-03T12:37:40.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.414+0000 7f252aaf8700 1 -- start start 2023-03-03T12:37:40.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.414+0000 7f252aaf8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25240abe20 0x7f25241dc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:40.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.414+0000 7f252aaf8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f252414f9d0 0x7f25241ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:40.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.414+0000 7f252aaf8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414fda0 0x7f25241edba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:40.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.414+0000 7f252aaf8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252406a500 con 0x7f252414fda0 2023-03-03T12:37:40.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.414+0000 7f252aaf8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25241ee0e0 con 0x7f25240abe20 2023-03-03T12:37:40.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.414+0000 7f252aaf8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25241dc8e0 con 0x7f252414f9d0 2023-03-03T12:37:40.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2529095700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414fda0 0x7f25241edba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:40.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2529095700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414fda0 0x7f25241edba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41188/0 (socket says 172.21.15.7:41188) 2023-03-03T12:37:40.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2529095700 1 -- 172.21.15.7:0/2714186253 learned_addr learned my addr 172.21.15.7:0/2714186253 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:37:40.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2523fff700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f252414f9d0 0x7f25241ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:40.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2528894700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25240abe20 0x7f25241dc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:40.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2529095700 1 -- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f252414f9d0 msgr2=0x7f25241ea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:40.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2529095700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f252414f9d0 0x7f25241ea7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2529095700 1 -- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25240abe20 msgr2=0x7f25241dc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:40.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2529095700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25240abe20 0x7f25241dc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2529095700 1 -- 172.21.15.7:0/2714186253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2518009550 con 0x7f252414fda0 2023-03-03T12:37:40.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2528894700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25240abe20 0x7f25241dc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:37:40.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f2529095700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414fda0 0x7f25241edba0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f251c00cdb0 tx=0x7f251c00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:40.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f25217fa700 1 -- 172.21.15.7:0/2714186253 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c00f480 con 0x7f252414fda0 2023-03-03T12:37:40.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f25217fa700 1 -- 172.21.15.7:0/2714186253 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251c00f600 con 0x7f252414fda0 2023-03-03T12:37:40.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f252aaf8700 1 -- 172.21.15.7:0/2714186253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25240ac350 con 0x7f252414fda0 2023-03-03T12:37:40.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f25217fa700 1 -- 172.21.15.7:0/2714186253 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c00f9b0 con 0x7f252414fda0 2023-03-03T12:37:40.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.415+0000 7f252aaf8700 1 -- 172.21.15.7:0/2714186253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25240ab630 con 0x7f252414fda0 2023-03-03T12:37:40.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.416+0000 7f252aaf8700 1 -- 172.21.15.7:0/2714186253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25240c3e10 con 0x7f252414fda0 2023-03-03T12:37:40.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.417+0000 7f25217fa700 1 -- 172.21.15.7:0/2714186253 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f251c009030 con 0x7f252414fda0 2023-03-03T12:37:40.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.417+0000 7f25217fa700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25080682f0 0x7f250806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:37:40.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.417+0000 7f25217fa700 1 -- 172.21.15.7:0/2714186253 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f251c086a70 con 0x7f252414fda0 2023-03-03T12:37:40.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.418+0000 7f2528894700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25080682f0 0x7f250806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:37:40.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.418+0000 7f2528894700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25080682f0 0x7f250806a7a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2518000c00 tx=0x7f2518008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:37:40.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.422+0000 7f25217fa700 1 -- 172.21.15.7:0/2714186253 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f251c052e10 con 0x7f252414fda0 2023-03-03T12:37:40.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.659+0000 7f252aaf8700 1 -- 172.21.15.7:0/2714186253 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2524074bb0 con 0x7f252414fda0 2023-03-03T12:37:40.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.660+0000 7f25217fa700 1 -- 172.21.15.7:0/2714186253 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f251c057e60 con 0x7f252414fda0 2023-03-03T12:37:40.662 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:37:40.662 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:37:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 -- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25080682f0 msgr2=0x7f250806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25080682f0 0x7f250806a7a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2518000c00 tx=0x7f2518008030 comp rx=0 tx=0).stop 2023-03-03T12:37:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 -- 172.21.15.7:0/2714186253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414fda0 msgr2=0x7f25241edba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:37:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414fda0 0x7f25241edba0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f251c00cdb0 tx=0x7f251c00a560 comp rx=0 tx=0).stop 2023-03-03T12:37:40.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 -- 172.21.15.7:0/2714186253 shutdown_connections 2023-03-03T12:37:40.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25080682f0 0x7f250806a7a0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f252414fda0 0x7f25241edba0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25240abe20 0x7f25241dc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 --2- 172.21.15.7:0/2714186253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f252414f9d0 0x7f25241ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:37:40.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 -- 172.21.15.7:0/2714186253 >> 172.21.15.7:0/2714186253 conn(0x7f25240c8900 msgr2=0x7f25240c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:37:40.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 -- 172.21.15.7:0/2714186253 shutdown_connections 2023-03-03T12:37:40.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:37:40.662+0000 7f252aaf8700 1 -- 172.21.15.7:0/2714186253 wait complete. 2023-03-03T12:37:40.666 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:37:40.676 INFO:tasks.mds_thrash.fs.[a]:waiting for 31 secs before thrashing 2023-03-03T12:38:00.833 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:00.837 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:11.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:38:11.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.809+0000 7f2092036700 1 Processor -- start 2023-03-03T12:38:11.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.809+0000 7f2092036700 1 -- start start 2023-03-03T12:38:11.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f2092036700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f208c06c480 0x7f208c1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:11.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f2092036700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c06cb70 con 0x7f208c151b20 2023-03-03T12:38:11.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f2092036700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c06ce50 con 0x7f208c0c2950 2023-03-03T12:38:11.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f2092036700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c06c2f0 con 0x7f208c06c480 2023-03-03T12:38:11.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f208bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f208c06c480 0x7f208c1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:11.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f208b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f208c151b20 0x7f208c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35204/0 (socket says 172.21.15.7:35204) 2023-03-03T12:38:11.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f208b7fe700 1 -- 172.21.15.7:0/3752515277 learned_addr learned my addr 172.21.15.7:0/3752515277 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:38:11.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f208bfff700 1 -- 172.21.15.7:0/3752515277 >> v1:172.21.15.143:6789/0 conn(0x7f208c0c2950 legacy=0x7f208c157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:38:11.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f208bfff700 1 -- 172.21.15.7:0/3752515277 >> v1:172.21.15.7:6789/0 conn(0x7f208c151b20 legacy=0x7f208c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:11.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f208a7fc700 1 -- 172.21.15.7:0/3752515277 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885858654 0 0) 0x7f208c06cb70 con 0x7f208c151b20 2023-03-03T12:38:11.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.810+0000 7f208bfff700 1 -- 172.21.15.7:0/3752515277 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c06d130 con 0x7f208c06c480 2023-03-03T12:38:11.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f208bfff700 1 --2- 172.21.15.7:0/3752515277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 0x7f208c1581f0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f2084009d90 tx=0x7f208400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37da9148fd8716b1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:11.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f208a7fc700 1 -- 172.21.15.7:0/3752515277 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208400e070 con 0x7f208c06c480 2023-03-03T12:38:11.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f208a7fc700 1 -- 172.21.15.7:0/3752515277 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208400bb90 con 0x7f208c06c480 2023-03-03T12:38:11.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f208a7fc700 1 -- 172.21.15.7:0/3752515277 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2084003a20 con 0x7f208c06c480 2023-03-03T12:38:11.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f2092036700 1 -- 172.21.15.7:0/3752515277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 msgr2=0x7f208c1581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:11.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f2092036700 1 --2- 172.21.15.7:0/3752515277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 0x7f208c1581f0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f2084009d90 tx=0x7f208400b330 comp rx=0 tx=0).stop 2023-03-03T12:38:11.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f2092036700 1 -- 172.21.15.7:0/3752515277 shutdown_connections 2023-03-03T12:38:11.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f2092036700 1 --2- 172.21.15.7:0/3752515277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 0x7f208c1581f0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:11.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f2092036700 1 -- 172.21.15.7:0/3752515277 >> 172.21.15.7:0/3752515277 conn(0x7f208c0a94d0 msgr2=0x7f208c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:38:11.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f2092036700 1 -- 172.21.15.7:0/3752515277 shutdown_connections 2023-03-03T12:38:11.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.811+0000 7f2092036700 1 -- 172.21.15.7:0/3752515277 wait complete. 2023-03-03T12:38:11.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.813+0000 7f2092036700 1 Processor -- start 2023-03-03T12:38:11.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.813+0000 7f2092036700 1 -- start start 2023-03-03T12:38:11.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.813+0000 7f2092036700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 0x7f208c154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:11.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.813+0000 7f2092036700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f208c0c2950 0x7f208c1498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:11.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.813+0000 7f2092036700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f208c151b20 0x7f208c149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:11.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f2092036700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c154360 con 0x7f208c06c480 2023-03-03T12:38:11.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f2092036700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c152320 con 0x7f208c0c2950 2023-03-03T12:38:11.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f2092036700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c152600 con 0x7f208c151b20 2023-03-03T12:38:11.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f208b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 0x7f208c154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:11.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f208b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 0x7f208c154810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41230/0 (socket says 172.21.15.7:41230) 2023-03-03T12:38:11.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f208b7fe700 1 -- 172.21.15.7:0/4025515624 learned_addr learned my addr 172.21.15.7:0/4025515624 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:38:11.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f208b7fe700 1 -- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f208c151b20 msgr2=0x7f208c149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:11.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f208b7fe700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f208c151b20 0x7f208c149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:11.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f208b7fe700 1 -- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f208c0c2950 msgr2=0x7f208c1498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:11.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f208b7fe700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f208c0c2950 0x7f208c1498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:11.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f208b7fe700 1 -- 172.21.15.7:0/4025515624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084009a70 con 0x7f208c06c480 2023-03-03T12:38:11.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f208b7fe700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 0x7f208c154810 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f207c00c550 tx=0x7f207c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:11.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f2083fff700 1 -- 172.21.15.7:0/4025515624 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c004c80 con 0x7f208c06c480 2023-03-03T12:38:11.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f2083fff700 1 -- 172.21.15.7:0/4025515624 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c004e00 con 0x7f208c06c480 2023-03-03T12:38:11.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f2092036700 1 -- 172.21.15.7:0/4025515624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c158730 con 0x7f208c06c480 2023-03-03T12:38:11.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f2083fff700 1 -- 172.21.15.7:0/4025515624 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c007310 con 0x7f208c06c480 2023-03-03T12:38:11.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.814+0000 7f2092036700 1 -- 172.21.15.7:0/4025515624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c06d410 con 0x7f208c06c480 2023-03-03T12:38:11.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.815+0000 7f2092036700 1 -- 172.21.15.7:0/4025515624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c151ef0 con 0x7f208c06c480 2023-03-03T12:38:11.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.816+0000 7f2083fff700 1 -- 172.21.15.7:0/4025515624 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f207c01a020 con 0x7f208c06c480 2023-03-03T12:38:11.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.816+0000 7f2083fff700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f206c0682f0 0x7f206c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:11.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.816+0000 7f2083fff700 1 -- 172.21.15.7:0/4025515624 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f207c085e00 con 0x7f208c06c480 2023-03-03T12:38:11.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.817+0000 7f208affd700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f206c0682f0 0x7f206c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:11.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.817+0000 7f208affd700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f206c0682f0 0x7f206c06a7a0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f208c1551b0 tx=0x7f2074006c50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:11.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:11.821+0000 7f2083fff700 1 -- 172.21.15.7:0/4025515624 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f207c0521f0 con 0x7f208c06c480 2023-03-03T12:38:12.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.058+0000 7f2092036700 1 -- 172.21.15.7:0/4025515624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f208c074bb0 con 0x7f208c06c480 2023-03-03T12:38:12.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.059+0000 7f2083fff700 1 -- 172.21.15.7:0/4025515624 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f207c055810 con 0x7f208c06c480 2023-03-03T12:38:12.062 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:38:12.062 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:38:12.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 -- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f206c0682f0 msgr2=0x7f206c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:12.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f206c0682f0 0x7f206c06a7a0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f208c1551b0 tx=0x7f2074006c50 comp rx=0 tx=0).stop 2023-03-03T12:38:12.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 -- 172.21.15.7:0/4025515624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 msgr2=0x7f208c154810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:12.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 0x7f208c154810 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f207c00c550 tx=0x7f207c00c910 comp rx=0 tx=0).stop 2023-03-03T12:38:12.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 -- 172.21.15.7:0/4025515624 shutdown_connections 2023-03-03T12:38:12.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f208c151b20 0x7f208c149e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:12.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f206c0682f0 0x7f206c06a7a0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:12.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f208c06c480 0x7f208c154810 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:12.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 --2- 172.21.15.7:0/4025515624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f208c0c2950 0x7f208c1498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:12.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 -- 172.21.15.7:0/4025515624 >> 172.21.15.7:0/4025515624 conn(0x7f208c0a94d0 msgr2=0x7f208c0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:38:12.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 -- 172.21.15.7:0/4025515624 shutdown_connections 2023-03-03T12:38:12.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:12.061+0000 7f2092036700 1 -- 172.21.15.7:0/4025515624 wait complete. 2023-03-03T12:38:12.066 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:38:12.076 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2023-03-03T12:38:24.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:38:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.922+0000 7fa6ecf70700 1 Processor -- start 2023-03-03T12:38:24.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.922+0000 7fa6ecf70700 1 -- start start 2023-03-03T12:38:24.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.922+0000 7fa6ecf70700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa6e814b020 0x7fa6e80717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:24.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6ecf70700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e8071d20 0x7fa6e81583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:24.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6ecf70700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e81588f0 con 0x7fa6e80aad20 2023-03-03T12:38:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6ecf70700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6e80728b0 con 0x7fa6e814b020 2023-03-03T12:38:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6ecf70700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e8155070 con 0x7fa6e8071d20 2023-03-03T12:38:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa6e814b020 0x7fa6e80717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e5d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa6e80aad20 0x7fa6e8157ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35228/0 (socket says 172.21.15.7:35228) 2023-03-03T12:38:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e5d9b700 1 -- 172.21.15.7:0/343501762 learned_addr learned my addr 172.21.15.7:0/343501762 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:38:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e6d9d700 1 --2- 172.21.15.7:0/343501762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e8071d20 0x7fa6e81583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e659c700 1 -- 172.21.15.7:0/343501762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e8071d20 msgr2=0x7fa6e81583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:38:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e659c700 1 --2- 172.21.15.7:0/343501762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e8071d20 0x7fa6e81583b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e659c700 1 -- 172.21.15.7:0/343501762 >> v1:172.21.15.7:6789/0 conn(0x7fa6e80aad20 legacy=0x7fa6e8157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e659c700 1 -- 172.21.15.7:0/343501762 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e806f5b0 con 0x7fa6e814b020 2023-03-03T12:38:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e559a700 1 -- 172.21.15.7:0/343501762 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044496577 0 0) 0x7fa6e81588f0 con 0x7fa6e80aad20 2023-03-03T12:38:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e659c700 1 --2- 172.21.15.7:0/343501762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e814b020 0x7fa6e80717e0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009840 tx=0x7fa6d8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef171fb571ecd259 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e559a700 1 -- 172.21.15.7:0/343501762 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8011830 con 0x7fa6e814b020 2023-03-03T12:38:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e559a700 1 -- 172.21.15.7:0/343501762 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d8011990 con 0x7fa6e814b020 2023-03-03T12:38:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.923+0000 7fa6e559a700 1 -- 172.21.15.7:0/343501762 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8011d20 con 0x7fa6e814b020 2023-03-03T12:38:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.924+0000 7fa6ecf70700 1 -- 172.21.15.7:0/343501762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e814b020 msgr2=0x7fa6e80717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.924+0000 7fa6ecf70700 1 --2- 172.21.15.7:0/343501762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e814b020 0x7fa6e80717e0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009840 tx=0x7fa6d8009b50 comp rx=0 tx=0).stop 2023-03-03T12:38:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.924+0000 7fa6ecf70700 1 -- 172.21.15.7:0/343501762 shutdown_connections 2023-03-03T12:38:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.924+0000 7fa6ecf70700 1 --2- 172.21.15.7:0/343501762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e8071d20 0x7fa6e81583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.924+0000 7fa6ecf70700 1 --2- 172.21.15.7:0/343501762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e814b020 0x7fa6e80717e0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.924+0000 7fa6ecf70700 1 -- 172.21.15.7:0/343501762 >> 172.21.15.7:0/343501762 conn(0x7fa6e80c8900 msgr2=0x7fa6e80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:38:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.924+0000 7fa6ecf70700 1 -- 172.21.15.7:0/343501762 shutdown_connections 2023-03-03T12:38:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.924+0000 7fa6ecf70700 1 -- 172.21.15.7:0/343501762 wait complete. 2023-03-03T12:38:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.925+0000 7fa6ecf70700 1 Processor -- start 2023-03-03T12:38:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.925+0000 7fa6ecf70700 1 -- start start 2023-03-03T12:38:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6ecf70700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8071d20 0x7fa6e81ddef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6ecf70700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e80aad20 0x7fa6e81e1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6ecf70700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e814b020 0x7fa6e814c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6ecf70700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e81dff70 con 0x7fa6e80aad20 2023-03-03T12:38:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6ecf70700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e814c160 con 0x7fa6e8071d20 2023-03-03T12:38:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6ecf70700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e80ad7c0 con 0x7fa6e814b020 2023-03-03T12:38:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e80aad20 0x7fa6e81e1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e80aad20 0x7fa6e81e1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41254/0 (socket says 172.21.15.7:41254) 2023-03-03T12:38:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e5d9b700 1 -- 172.21.15.7:0/2183861357 learned_addr learned my addr 172.21.15.7:0/2183861357 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:38:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e6d9d700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e814b020 0x7fa6e814c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e659c700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8071d20 0x7fa6e81ddef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:24.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e5d9b700 1 -- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e814b020 msgr2=0x7fa6e814c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e5d9b700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e814b020 0x7fa6e814c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e5d9b700 1 -- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8071d20 msgr2=0x7fa6e81ddef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e5d9b700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8071d20 0x7fa6e81ddef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e5d9b700 1 -- 172.21.15.7:0/2183861357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d80094a0 con 0x7fa6e80aad20 2023-03-03T12:38:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e6d9d700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e814b020 0x7fa6e814c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:38:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.926+0000 7fa6e659c700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8071d20 0x7fa6e81ddef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:38:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.927+0000 7fa6e5d9b700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e80aad20 0x7fa6e81e1350 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00c520 tx=0x7fa6dc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:24.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.927+0000 7fa6d6ffd700 1 -- 172.21.15.7:0/2183861357 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc004c80 con 0x7fa6e80aad20 2023-03-03T12:38:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.927+0000 7fa6d6ffd700 1 -- 172.21.15.7:0/2183861357 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc004de0 con 0x7fa6e80aad20 2023-03-03T12:38:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.927+0000 7fa6ecf70700 1 -- 172.21.15.7:0/2183861357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e806fb70 con 0x7fa6e80aad20 2023-03-03T12:38:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.927+0000 7fa6d6ffd700 1 -- 172.21.15.7:0/2183861357 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc0072d0 con 0x7fa6e80aad20 2023-03-03T12:38:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.927+0000 7fa6ecf70700 1 -- 172.21.15.7:0/2183861357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e806f890 con 0x7fa6e80aad20 2023-03-03T12:38:24.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.928+0000 7fa6ecf70700 1 -- 172.21.15.7:0/2183861357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e8155350 con 0x7fa6e80aad20 2023-03-03T12:38:24.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.928+0000 7fa6d6ffd700 1 -- 172.21.15.7:0/2183861357 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa6dc01a020 con 0x7fa6e80aad20 2023-03-03T12:38:24.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.929+0000 7fa6d6ffd700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6c80682f0 0x7fa6c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:24.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.929+0000 7fa6d6ffd700 1 -- 172.21.15.7:0/2183861357 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa6dc085cf0 con 0x7fa6e80aad20 2023-03-03T12:38:24.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.929+0000 7fa6e659c700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6c80682f0 0x7fa6c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:24.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.929+0000 7fa6e659c700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6c80682f0 0x7fa6c806a7a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80090b0 tx=0x7fa6d8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:24.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:24.933+0000 7fa6d6ffd700 1 -- 172.21.15.7:0/2183861357 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa6dc052140 con 0x7fa6e80aad20 2023-03-03T12:38:25.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.172+0000 7fa6ecf70700 1 -- 172.21.15.7:0/2183861357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6e814c310 con 0x7fa6e80aad20 2023-03-03T12:38:25.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.172+0000 7fa6d6ffd700 1 -- 172.21.15.7:0/2183861357 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa6e814c310 con 0x7fa6e80aad20 2023-03-03T12:38:25.174 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:38:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:38:25.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.174+0000 7fa6ecf70700 1 -- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6c80682f0 msgr2=0x7fa6c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:25.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.174+0000 7fa6ecf70700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6c80682f0 0x7fa6c806a7a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80090b0 tx=0x7fa6d8008030 comp rx=0 tx=0).stop 2023-03-03T12:38:25.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.174+0000 7fa6ecf70700 1 -- 172.21.15.7:0/2183861357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e80aad20 msgr2=0x7fa6e81e1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:25.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.174+0000 7fa6ecf70700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e80aad20 0x7fa6e81e1350 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00c520 tx=0x7fa6dc00c8e0 comp rx=0 tx=0).stop 2023-03-03T12:38:25.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.174+0000 7fa6ecf70700 1 -- 172.21.15.7:0/2183861357 shutdown_connections 2023-03-03T12:38:25.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.174+0000 7fa6ecf70700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6e814b020 0x7fa6e814c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:25.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.174+0000 7fa6ecf70700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6e8071d20 0x7fa6e81ddef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:25.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.174+0000 7fa6ecf70700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6c80682f0 0x7fa6c806a7a0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:25.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.174+0000 7fa6ecf70700 1 --2- 172.21.15.7:0/2183861357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6e80aad20 0x7fa6e81e1350 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:25.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.174+0000 7fa6ecf70700 1 -- 172.21.15.7:0/2183861357 >> 172.21.15.7:0/2183861357 conn(0x7fa6e80c8900 msgr2=0x7fa6e8155f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:38:25.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.175+0000 7fa6ecf70700 1 -- 172.21.15.7:0/2183861357 shutdown_connections 2023-03-03T12:38:25.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:25.175+0000 7fa6ecf70700 1 -- 172.21.15.7:0/2183861357 wait complete. 2023-03-03T12:38:25.178 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:38:25.188 INFO:tasks.mds_thrash.fs.[b]:waiting for 12 secs before thrashing 2023-03-03T12:38:30.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:30.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:38:37.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:38:37.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.322+0000 7f0df987d700 1 Processor -- start 2023-03-03T12:38:37.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.322+0000 7f0df987d700 1 -- start start 2023-03-03T12:38:37.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df987d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0df414e560 0x7f0df4071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:37.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df987d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0df40c43d0 0x7f0df40c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:37.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df987d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df4072920 con 0x7f0df4154590 2023-03-03T12:38:37.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df987d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df406bef0 con 0x7f0df414e560 2023-03-03T12:38:37.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df987d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df406b910 con 0x7f0df40c43d0 2023-03-03T12:38:37.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df2ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0df414e560 0x7f0df4071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:37.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df2ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0df414e560 0x7f0df4071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41268/0 (socket says 172.21.15.7:41268) 2023-03-03T12:38:37.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df2ffd700 1 -- 172.21.15.7:0/3614853879 learned_addr learned my addr 172.21.15.7:0/3614853879 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:38:37.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df27fc700 1 --2- 172.21.15.7:0/3614853879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0df40c43d0 0x7f0df40c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:37.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df2ffd700 1 -- 172.21.15.7:0/3614853879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0df40c43d0 msgr2=0x7f0df40c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:38:37.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df2ffd700 1 --2- 172.21.15.7:0/3614853879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0df40c43d0 0x7f0df40c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:37.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df2ffd700 1 -- 172.21.15.7:0/3614853879 >> v1:172.21.15.143:6789/0 conn(0x7f0df4154590 legacy=0x7f0df41549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:37.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.323+0000 7f0df2ffd700 1 -- 172.21.15.7:0/3614853879 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df40aacc0 con 0x7f0df414e560 2023-03-03T12:38:37.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df2ffd700 1 --2- 172.21.15.7:0/3614853879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df414e560 0x7f0df4071c70 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f0de80098f0 tx=0x7f0de8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3100a2c97d7fb28 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:37.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df1ffb700 1 -- 172.21.15.7:0/3614853879 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de800b8a0 con 0x7f0df414e560 2023-03-03T12:38:37.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df1ffb700 1 -- 172.21.15.7:0/3614853879 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de800ba00 con 0x7f0df414e560 2023-03-03T12:38:37.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df1ffb700 1 -- 172.21.15.7:0/3614853879 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de800bd90 con 0x7f0df414e560 2023-03-03T12:38:37.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df987d700 1 -- 172.21.15.7:0/3614853879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df414e560 msgr2=0x7f0df4071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:37.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df987d700 1 --2- 172.21.15.7:0/3614853879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df414e560 0x7f0df4071c70 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f0de80098f0 tx=0x7f0de8009c00 comp rx=0 tx=0).stop 2023-03-03T12:38:37.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df987d700 1 -- 172.21.15.7:0/3614853879 shutdown_connections 2023-03-03T12:38:37.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df987d700 1 --2- 172.21.15.7:0/3614853879 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df414e560 0x7f0df4071c70 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:37.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df987d700 1 --2- 172.21.15.7:0/3614853879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0df40c43d0 0x7f0df40c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:37.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df987d700 1 -- 172.21.15.7:0/3614853879 >> 172.21.15.7:0/3614853879 conn(0x7f0df40736f0 msgr2=0x7f0df4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:38:37.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df987d700 1 -- 172.21.15.7:0/3614853879 shutdown_connections 2023-03-03T12:38:37.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.324+0000 7f0df987d700 1 -- 172.21.15.7:0/3614853879 wait complete. 2023-03-03T12:38:37.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.325+0000 7f0df987d700 1 Processor -- start 2023-03-03T12:38:37.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.326+0000 7f0df987d700 1 -- start start 2023-03-03T12:38:37.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.326+0000 7f0df987d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df40c43d0 0x7f0df40c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:37.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.326+0000 7f0df987d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df414e560 0x7f0df4148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:37.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.326+0000 7f0df987d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0df4154590 0x7f0df4148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:37.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.326+0000 7f0df987d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df4157f90 con 0x7f0df40c43d0 2023-03-03T12:38:37.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.326+0000 7f0df987d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df414dc10 con 0x7f0df414e560 2023-03-03T12:38:37.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.326+0000 7f0df987d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df4158230 con 0x7f0df4154590 2023-03-03T12:38:37.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.326+0000 7f0df2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df40c43d0 0x7f0df40c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:37.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.326+0000 7f0df2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df40c43d0 0x7f0df40c7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41272/0 (socket says 172.21.15.7:41272) 2023-03-03T12:38:37.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.326+0000 7f0df2ffd700 1 -- 172.21.15.7:0/3571146529 learned_addr learned my addr 172.21.15.7:0/3571146529 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:38:37.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df27fc700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df414e560 0x7f0df4148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:37.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df37fe700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0df4154590 0x7f0df4148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:37.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df2ffd700 1 -- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0df4154590 msgr2=0x7f0df4148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:37.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df2ffd700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0df4154590 0x7f0df4148870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:37.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df2ffd700 1 -- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df414e560 msgr2=0x7f0df4148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:37.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df2ffd700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df414e560 0x7f0df4148330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:37.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df2ffd700 1 -- 172.21.15.7:0/3571146529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de8009550 con 0x7f0df40c43d0 2023-03-03T12:38:37.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df27fc700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df414e560 0x7f0df4148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:38:37.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df2ffd700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df40c43d0 0x7f0df40c7720 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009c40 tx=0x7f0de8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:37.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0ddbfff700 1 -- 172.21.15.7:0/3571146529 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de80036a0 con 0x7f0df40c43d0 2023-03-03T12:38:37.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0ddbfff700 1 -- 172.21.15.7:0/3571146529 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de801f070 con 0x7f0df40c43d0 2023-03-03T12:38:37.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df987d700 1 -- 172.21.15.7:0/3571146529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df4155100 con 0x7f0df40c43d0 2023-03-03T12:38:37.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0ddbfff700 1 -- 172.21.15.7:0/3571146529 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8003a60 con 0x7f0df40c43d0 2023-03-03T12:38:37.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.327+0000 7f0df987d700 1 -- 172.21.15.7:0/3571146529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df406bb80 con 0x7f0df40c43d0 2023-03-03T12:38:37.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.328+0000 7f0df987d700 1 -- 172.21.15.7:0/3571146529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df41ed320 con 0x7f0df40c43d0 2023-03-03T12:38:37.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.329+0000 7f0ddbfff700 1 -- 172.21.15.7:0/3571146529 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0de800f050 con 0x7f0df40c43d0 2023-03-03T12:38:37.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.329+0000 7f0ddbfff700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0de00682f0 0x7f0de006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:38:37.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.329+0000 7f0ddbfff700 1 -- 172.21.15.7:0/3571146529 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0de8021070 con 0x7f0df40c43d0 2023-03-03T12:38:37.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.329+0000 7f0df27fc700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0de00682f0 0x7f0de006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:38:37.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.330+0000 7f0df27fc700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0de00682f0 0x7f0de006a7a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f0df414d540 tx=0x7f0de4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:38:37.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.334+0000 7f0ddbfff700 1 -- 172.21.15.7:0/3571146529 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0de8053e40 con 0x7f0df40c43d0 2023-03-03T12:38:37.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.572+0000 7f0df987d700 1 -- 172.21.15.7:0/3571146529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0df4074bb0 con 0x7f0df40c43d0 2023-03-03T12:38:37.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.573+0000 7f0ddbfff700 1 -- 172.21.15.7:0/3571146529 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0de8057460 con 0x7f0df40c43d0 2023-03-03T12:38:37.575 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:38:37.575 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:38:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 -- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0de00682f0 msgr2=0x7f0de006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0de00682f0 0x7f0de006a7a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f0df414d540 tx=0x7f0de4008040 comp rx=0 tx=0).stop 2023-03-03T12:38:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 -- 172.21.15.7:0/3571146529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df40c43d0 msgr2=0x7f0df40c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:38:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df40c43d0 0x7f0df40c7720 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009c40 tx=0x7f0de8004750 comp rx=0 tx=0).stop 2023-03-03T12:38:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 -- 172.21.15.7:0/3571146529 shutdown_connections 2023-03-03T12:38:37.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0df4154590 0x7f0df4148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:37.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0de00682f0 0x7f0de006a7a0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:37.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0df40c43d0 0x7f0df40c7720 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:37.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 --2- 172.21.15.7:0/3571146529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0df414e560 0x7f0df4148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:38:37.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 -- 172.21.15.7:0/3571146529 >> 172.21.15.7:0/3571146529 conn(0x7f0df40736f0 msgr2=0x7f0df41549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:38:37.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 -- 172.21.15.7:0/3571146529 shutdown_connections 2023-03-03T12:38:37.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:38:37.575+0000 7f0df987d700 1 -- 172.21.15.7:0/3571146529 wait complete. 2023-03-03T12:38:37.579 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:38:37.589 INFO:tasks.mds_thrash.fs.[b]:waiting for 38 secs before thrashing 2023-03-03T12:39:01.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:01.108 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:15.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:39:15.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.726+0000 7f9968c4b700 1 Processor -- start 2023-03-03T12:39:15.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.726+0000 7f9968c4b700 1 -- start start 2023-03-03T12:39:15.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f9968c4b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f996414e900 0x7f996415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:15.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f9968c4b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99641510d0 con 0x7f9964072250 2023-03-03T12:39:15.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f9968c4b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99640c3740 con 0x7f9964071e80 2023-03-03T12:39:15.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f9968c4b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9964070670 con 0x7f996414e900 2023-03-03T12:39:15.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f9961d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9964072250 0x7f996414e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35292/0 (socket says 172.21.15.7:35292) 2023-03-03T12:39:15.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f9961d9b700 1 -- 172.21.15.7:0/407260246 learned_addr learned my addr 172.21.15.7:0/407260246 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:39:15.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f9962d9d700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f996414e900 0x7f996415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:15.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160324493 0 0) 0x7f99641510d0 con 0x7f9964072250 2023-03-03T12:39:15.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c0035b0 con 0x7f9964072250 2023-03-03T12:39:15.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913003292 0 0) 0x7f99640c3740 con 0x7f9964071e80 2023-03-03T12:39:15.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99641510d0 con 0x7f9964071e80 2023-03-03T12:39:15.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3996854237 0 0) 0x7f994c0035b0 con 0x7f9964072250 2023-03-03T12:39:15.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f996414e900 msgr2=0x7f996415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:39:15.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f996414e900 0x7f996415ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:15.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 >> v1:172.21.15.143:6789/0 conn(0x7f9964071e80 legacy=0x7f9964151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:15.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99640c39e0 con 0x7f9964072250 2023-03-03T12:39:15.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f9958001670 con 0x7f9964072250 2023-03-03T12:39:15.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.727+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 >> v1:172.21.15.7:6789/0 conn(0x7f9964072250 legacy=0x7f996414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996159a700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f996414e900 0x7f994c003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996159a700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 0x7f994c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996159a700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994c004d90 0x7f994c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964070670 con 0x7f994c0041e0 2023-03-03T12:39:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c00bc20 con 0x7f994c004d90 2023-03-03T12:39:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c00bde0 con 0x7f996414e900 2023-03-03T12:39:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f9961d9b700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 0x7f994c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f9962d9d700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994c004d90 0x7f994c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996259c700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f996414e900 0x7f994c003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f9961d9b700 1 -- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f996414e900 msgr2=0x7f994c003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:39:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f9961d9b700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f996414e900 0x7f994c003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f9961d9b700 1 -- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994c004d90 msgr2=0x7f994c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:39:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f9961d9b700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f994c004d90 0x7f994c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f9961d9b700 1 -- 172.21.15.7:0/407260246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0035b0 con 0x7f994c0041e0 2023-03-03T12:39:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996259c700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f996414e900 0x7f994c003c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:39:15.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f9961d9b700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 0x7f994c004850 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f9958001eb0 tx=0x7f994c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aec88b612556659 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:15.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995800cc40 con 0x7f994c0041e0 2023-03-03T12:39:15.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995800cdc0 con 0x7f994c0041e0 2023-03-03T12:39:15.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.728+0000 7f996159a700 1 -- 172.21.15.7:0/407260246 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958005ba0 con 0x7f994c0041e0 2023-03-03T12:39:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.729+0000 7f9968c4b700 1 -- 172.21.15.7:0/407260246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 msgr2=0x7f994c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.729+0000 7f9968c4b700 1 --2- 172.21.15.7:0/407260246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 0x7f994c004850 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f9958001eb0 tx=0x7f994c005880 comp rx=0 tx=0).stop 2023-03-03T12:39:15.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.729+0000 7f9962d9d700 1 -- 172.21.15.7:0/407260246 reap_dead start 2023-03-03T12:39:15.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.729+0000 7f9968c4b700 1 -- 172.21.15.7:0/407260246 shutdown_connections 2023-03-03T12:39:15.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.729+0000 7f9968c4b700 1 -- 172.21.15.7:0/407260246 >> 172.21.15.7:0/407260246 conn(0x7f99640a94d0 msgr2=0x7f99640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:39:15.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.729+0000 7f9968c4b700 1 -- 172.21.15.7:0/407260246 shutdown_connections 2023-03-03T12:39:15.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.729+0000 7f9968c4b700 1 -- 172.21.15.7:0/407260246 wait complete. 2023-03-03T12:39:15.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.730+0000 7f9968c4b700 1 Processor -- start 2023-03-03T12:39:15.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.730+0000 7f9968c4b700 1 -- start start 2023-03-03T12:39:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f9968c4b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 0x7f99641e0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f9968c4b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99640ac050 0x7f99640ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f9968c4b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99640aca00 0x7f99641ed1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f9968c4b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964149770 con 0x7f994c0041e0 2023-03-03T12:39:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f9968c4b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99640ab660 con 0x7f99640aca00 2023-03-03T12:39:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f9968c4b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99640ab890 con 0x7f99640ac050 2023-03-03T12:39:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f996259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 0x7f99641e0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f996259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 0x7f99641e0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41322/0 (socket says 172.21.15.7:41322) 2023-03-03T12:39:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f996259c700 1 -- 172.21.15.7:0/3426547656 learned_addr learned my addr 172.21.15.7:0/3426547656 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:39:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f9962d9d700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99640aca00 0x7f99641ed1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f9961d9b700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99640ac050 0x7f99640ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f996259c700 1 -- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99640ac050 msgr2=0x7f99640ac4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f996259c700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99640ac050 0x7f99640ac4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f996259c700 1 -- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99640aca00 msgr2=0x7f99641ed1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:15.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f996259c700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99640aca00 0x7f99641ed1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:15.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.731+0000 7f996259c700 1 -- 172.21.15.7:0/3426547656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9964151f30 con 0x7f994c0041e0 2023-03-03T12:39:15.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.732+0000 7f996259c700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 0x7f99641e0b70 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f995400e7e0 tx=0x7f995400eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:15.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.732+0000 7f9952ffd700 1 -- 172.21.15.7:0/3426547656 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954017070 con 0x7f994c0041e0 2023-03-03T12:39:15.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.732+0000 7f9952ffd700 1 -- 172.21.15.7:0/3426547656 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99540052e0 con 0x7f994c0041e0 2023-03-03T12:39:15.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.732+0000 7f9968c4b700 1 -- 172.21.15.7:0/3426547656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9964151300 con 0x7f994c0041e0 2023-03-03T12:39:15.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.732+0000 7f9952ffd700 1 -- 172.21.15.7:0/3426547656 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954005450 con 0x7f994c0041e0 2023-03-03T12:39:15.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.732+0000 7f9968c4b700 1 -- 172.21.15.7:0/3426547656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996406c9a0 con 0x7f994c0041e0 2023-03-03T12:39:15.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.733+0000 7f9968c4b700 1 -- 172.21.15.7:0/3426547656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99640abea0 con 0x7f994c0041e0 2023-03-03T12:39:15.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.733+0000 7f9952ffd700 1 -- 172.21.15.7:0/3426547656 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9954006860 con 0x7f994c0041e0 2023-03-03T12:39:15.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.734+0000 7f9952ffd700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99440682f0 0x7f994406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:15.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.734+0000 7f9952ffd700 1 -- 172.21.15.7:0/3426547656 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f99540859a0 con 0x7f994c0041e0 2023-03-03T12:39:15.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.734+0000 7f9961d9b700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99440682f0 0x7f994406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:15.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.734+0000 7f9961d9b700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99440682f0 0x7f994406a7a0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9958015040 tx=0x7f9958007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:15.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.738+0000 7f9952ffd700 1 -- 172.21.15.7:0/3426547656 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9954051d40 con 0x7f994c0041e0 2023-03-03T12:39:15.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.975+0000 7f9968c4b700 1 -- 172.21.15.7:0/3426547656 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f996406db10 con 0x7f994c0041e0 2023-03-03T12:39:15.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.975+0000 7f9952ffd700 1 -- 172.21.15.7:0/3426547656 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9954055360 con 0x7f994c0041e0 2023-03-03T12:39:15.979 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:39:15.979 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:39:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 -- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99440682f0 msgr2=0x7f994406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99440682f0 0x7f994406a7a0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9958015040 tx=0x7f9958007000 comp rx=0 tx=0).stop 2023-03-03T12:39:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 -- 172.21.15.7:0/3426547656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 msgr2=0x7f99641e0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 0x7f99641e0b70 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f995400e7e0 tx=0x7f995400eba0 comp rx=0 tx=0).stop 2023-03-03T12:39:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 -- 172.21.15.7:0/3426547656 shutdown_connections 2023-03-03T12:39:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99640aca00 0x7f99641ed1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99440682f0 0x7f994406a7a0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f994c0041e0 0x7f99641e0b70 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 --2- 172.21.15.7:0/3426547656 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99640ac050 0x7f99640ac4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 -- 172.21.15.7:0/3426547656 >> 172.21.15.7:0/3426547656 conn(0x7f99640a94d0 msgr2=0x7f996414e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:39:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 -- 172.21.15.7:0/3426547656 shutdown_connections 2023-03-03T12:39:15.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:15.978+0000 7f9968c4b700 1 -- 172.21.15.7:0/3426547656 wait complete. 2023-03-03T12:39:15.983 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:39:15.993 INFO:tasks.mds_thrash.fs.[b]:waiting for 104 secs before thrashing 2023-03-03T12:39:20.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:39:20.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.213+0000 7fa61e4a7700 1 Processor -- start 2023-03-03T12:39:20.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.213+0000 7fa61e4a7700 1 -- start start 2023-03-03T12:39:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.214+0000 7fa61e4a7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa6180aad20 0x7fa618157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.214+0000 7fa61e4a7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6181588f0 con 0x7fa61814b020 2023-03-03T12:39:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.214+0000 7fa61e4a7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6180728b0 con 0x7fa6180aa030 2023-03-03T12:39:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.214+0000 7fa61e4a7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa618155070 con 0x7fa6180aad20 2023-03-03T12:39:20.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.214+0000 7fa6177fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa6180aad20 0x7fa618157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:20.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.214+0000 7fa617fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa61814b020 0x7fa6180717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35318/0 (socket says 172.21.15.7:35318) 2023-03-03T12:39:20.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.214+0000 7fa617fff700 1 -- 172.21.15.7:0/1120058520 learned_addr learned my addr 172.21.15.7:0/1120058520 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:39:20.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa6177fe700 1 -- 172.21.15.7:0/1120058520 >> v1:172.21.15.143:6789/0 conn(0x7fa6180aa030 legacy=0x7fa6181581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:39:20.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa6177fe700 1 -- 172.21.15.7:0/1120058520 >> v1:172.21.15.7:6789/0 conn(0x7fa61814b020 legacy=0x7fa6180717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:20.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa6177fe700 1 -- 172.21.15.7:0/1120058520 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61806f5b0 con 0x7fa6180aad20 2023-03-03T12:39:20.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa6177fe700 1 --2- 172.21.15.7:0/1120058520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6180aad20 0x7fa618157ca0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fa60c009840 tx=0x7fa60c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42298a9f61f2b96c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:20.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa616ffd700 1 -- 172.21.15.7:0/1120058520 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c004230 con 0x7fa6180aad20 2023-03-03T12:39:20.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa616ffd700 1 -- 172.21.15.7:0/1120058520 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c004390 con 0x7fa6180aad20 2023-03-03T12:39:20.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa616ffd700 1 -- 172.21.15.7:0/1120058520 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c0036a0 con 0x7fa6180aad20 2023-03-03T12:39:20.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa61e4a7700 1 -- 172.21.15.7:0/1120058520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6180aad20 msgr2=0x7fa618157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:20.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa61e4a7700 1 --2- 172.21.15.7:0/1120058520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6180aad20 0x7fa618157ca0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fa60c009840 tx=0x7fa60c009b50 comp rx=0 tx=0).stop 2023-03-03T12:39:20.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa61e4a7700 1 -- 172.21.15.7:0/1120058520 shutdown_connections 2023-03-03T12:39:20.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.215+0000 7fa61e4a7700 1 --2- 172.21.15.7:0/1120058520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6180aad20 0x7fa618157ca0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:20.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.216+0000 7fa61e4a7700 1 -- 172.21.15.7:0/1120058520 >> 172.21.15.7:0/1120058520 conn(0x7fa6180c8900 msgr2=0x7fa6180c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:39:20.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.216+0000 7fa61e4a7700 1 -- 172.21.15.7:0/1120058520 shutdown_connections 2023-03-03T12:39:20.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.216+0000 7fa61e4a7700 1 -- 172.21.15.7:0/1120058520 wait complete. 2023-03-03T12:39:20.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.217+0000 7fa61e4a7700 1 Processor -- start 2023-03-03T12:39:20.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.217+0000 7fa61e4a7700 1 -- start start 2023-03-03T12:39:20.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61e4a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6180aa030 0x7fa618142900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:20.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61e4a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6180aad20 0x7fa618148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:20.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61e4a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa61814b020 0x7fa6181ed480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:20.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61e4a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa618147ca0 con 0x7fa61814b020 2023-03-03T12:39:20.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61e4a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c6540 con 0x7fa6180aad20 2023-03-03T12:39:20.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61e4a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa618147990 con 0x7fa6180aa030 2023-03-03T12:39:20.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61ca44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa61814b020 0x7fa6181ed480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:20.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61ca44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa61814b020 0x7fa6181ed480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41348/0 (socket says 172.21.15.7:41348) 2023-03-03T12:39:20.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61ca44700 1 -- 172.21.15.7:0/699733607 learned_addr learned my addr 172.21.15.7:0/699733607 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:39:20.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa6177fe700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6180aad20 0x7fa618148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:20.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa617fff700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6180aa030 0x7fa618142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:20.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61ca44700 1 -- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6180aa030 msgr2=0x7fa618142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:20.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61ca44700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6180aa030 0x7fa618142900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:20.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61ca44700 1 -- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6180aad20 msgr2=0x7fa618148dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:20.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61ca44700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6180aad20 0x7fa618148dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:20.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61ca44700 1 -- 172.21.15.7:0/699733607 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c0094a0 con 0x7fa61814b020 2023-03-03T12:39:20.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa617fff700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6180aa030 0x7fa618142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:39:20.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa6177fe700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6180aad20 0x7fa618148dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:39:20.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa61ca44700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa61814b020 0x7fa6181ed480 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fa61000b970 tx=0x7fa61000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:20.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.218+0000 7fa614ff9700 1 -- 172.21.15.7:0/699733607 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61000c9b0 con 0x7fa61814b020 2023-03-03T12:39:20.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.219+0000 7fa614ff9700 1 -- 172.21.15.7:0/699733607 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61000cb10 con 0x7fa61814b020 2023-03-03T12:39:20.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.219+0000 7fa614ff9700 1 -- 172.21.15.7:0/699733607 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610014440 con 0x7fa61814b020 2023-03-03T12:39:20.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.219+0000 7fa61e4a7700 1 -- 172.21.15.7:0/699733607 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61806fb70 con 0x7fa61814b020 2023-03-03T12:39:20.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.219+0000 7fa61e4a7700 1 -- 172.21.15.7:0/699733607 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61806f890 con 0x7fa61814b020 2023-03-03T12:39:20.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.220+0000 7fa61e4a7700 1 -- 172.21.15.7:0/699733607 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa618143b70 con 0x7fa61814b020 2023-03-03T12:39:20.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.220+0000 7fa614ff9700 1 -- 172.21.15.7:0/699733607 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa61001b030 con 0x7fa61814b020 2023-03-03T12:39:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.220+0000 7fa614ff9700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6040682f0 0x7fa60406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:39:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.221+0000 7fa614ff9700 1 -- 172.21.15.7:0/699733607 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa610085c70 con 0x7fa61814b020 2023-03-03T12:39:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.221+0000 7fa617fff700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6040682f0 0x7fa60406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:39:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.221+0000 7fa617fff700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6040682f0 0x7fa60406a7a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa608000f60 tx=0x7fa608009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:39:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.225+0000 7fa614ff9700 1 -- 172.21.15.7:0/699733607 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa610052010 con 0x7fa61814b020 2023-03-03T12:39:20.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.464+0000 7fa61e4a7700 1 -- 172.21.15.7:0/699733607 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa618155350 con 0x7fa61814b020 2023-03-03T12:39:20.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.465+0000 7fa614ff9700 1 -- 172.21.15.7:0/699733607 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa610055630 con 0x7fa61814b020 2023-03-03T12:39:20.468 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:39:20.468 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:39:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 -- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6040682f0 msgr2=0x7fa60406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6040682f0 0x7fa60406a7a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa608000f60 tx=0x7fa608009040 comp rx=0 tx=0).stop 2023-03-03T12:39:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 -- 172.21.15.7:0/699733607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa61814b020 msgr2=0x7fa6181ed480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:39:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa61814b020 0x7fa6181ed480 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fa61000b970 tx=0x7fa61000bd30 comp rx=0 tx=0).stop 2023-03-03T12:39:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 -- 172.21.15.7:0/699733607 shutdown_connections 2023-03-03T12:39:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6040682f0 0x7fa60406a7a0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa61814b020 0x7fa6181ed480 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6180aa030 0x7fa618142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 --2- 172.21.15.7:0/699733607 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6180aad20 0x7fa618148dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:39:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 -- 172.21.15.7:0/699733607 >> 172.21.15.7:0/699733607 conn(0x7fa6180c8900 msgr2=0x7fa6180ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:39:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 -- 172.21.15.7:0/699733607 shutdown_connections 2023-03-03T12:39:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:39:20.467+0000 7fa61e4a7700 1 -- 172.21.15.7:0/699733607 wait complete. 2023-03-03T12:39:20.472 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:39:20.481 INFO:tasks.mds_thrash.fs.[a]:waiting for 44 secs before thrashing 2023-03-03T12:39:31.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:39:31.246 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T12:39:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T12:40:01.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:01.384 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:04.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:40:04.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.619+0000 7fe488276700 1 Processor -- start 2023-03-03T12:40:04.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.619+0000 7fe488276700 1 -- start start 2023-03-03T12:40:04.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.620+0000 7fe488276700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4800ac340 0x7fe4801510b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:04.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.620+0000 7fe488276700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe4800ab520 0x7fe4800c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:04.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.620+0000 7fe488276700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4801515f0 con 0x7fe4800c7820 2023-03-03T12:40:04.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.620+0000 7fe488276700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48006d9c0 con 0x7fe4800ac340 2023-03-03T12:40:04.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.620+0000 7fe488276700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe480151830 con 0x7fe4800ab520 2023-03-03T12:40:04.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.620+0000 7fe486012700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4800ac340 0x7fe4801510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:04.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.620+0000 7fe486012700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4800ac340 0x7fe4801510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41384/0 (socket says 172.21.15.7:41384) 2023-03-03T12:40:04.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.620+0000 7fe486012700 1 -- 172.21.15.7:0/801139087 learned_addr learned my addr 172.21.15.7:0/801139087 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:40:04.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe485811700 1 --2- 172.21.15.7:0/801139087 >> v2:172.21.15.143:3300/0 conn(0x7fe4800ab520 0x7fe4800c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:04.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe486012700 1 -- 172.21.15.7:0/801139087 >> v2:172.21.15.143:3300/0 conn(0x7fe4800ab520 msgr2=0x7fe4800c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:40:04.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe486012700 1 --2- 172.21.15.7:0/801139087 >> v2:172.21.15.143:3300/0 conn(0x7fe4800ab520 0x7fe4800c72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe486012700 1 -- 172.21.15.7:0/801139087 >> v1:172.21.15.143:6789/0 conn(0x7fe4800c7820 legacy=0x7fe4800c4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:40:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe486012700 1 -- 172.21.15.7:0/801139087 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe480151b10 con 0x7fe4800ac340 2023-03-03T12:40:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe485811700 1 --2- 172.21.15.7:0/801139087 >> v2:172.21.15.143:3300/0 conn(0x7fe4800ab520 0x7fe4800c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:40:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe486012700 1 --2- 172.21.15.7:0/801139087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ac340 0x7fe4801510b0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fe4800c4ed0 tx=0x7fe47000be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ba17b10feccc29 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe485010700 1 -- 172.21.15.7:0/801139087 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47000dcf0 con 0x7fe4800ac340 2023-03-03T12:40:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe485010700 1 -- 172.21.15.7:0/801139087 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47000de50 con 0x7fe4800ac340 2023-03-03T12:40:04.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe485010700 1 -- 172.21.15.7:0/801139087 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470004250 con 0x7fe4800ac340 2023-03-03T12:40:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe488276700 1 -- 172.21.15.7:0/801139087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ac340 msgr2=0x7fe4801510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.621+0000 7fe488276700 1 --2- 172.21.15.7:0/801139087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ac340 0x7fe4801510b0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fe4800c4ed0 tx=0x7fe47000be70 comp rx=0 tx=0).stop 2023-03-03T12:40:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.622+0000 7fe488276700 1 -- 172.21.15.7:0/801139087 shutdown_connections 2023-03-03T12:40:04.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.622+0000 7fe488276700 1 --2- 172.21.15.7:0/801139087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ac340 0x7fe4801510b0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.622+0000 7fe488276700 1 --2- 172.21.15.7:0/801139087 >> v2:172.21.15.143:3300/0 conn(0x7fe4800ab520 0x7fe4800c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.622+0000 7fe488276700 1 -- 172.21.15.7:0/801139087 >> 172.21.15.7:0/801139087 conn(0x7fe48006c060 msgr2=0x7fe48006c460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:40:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.622+0000 7fe488276700 1 -- 172.21.15.7:0/801139087 shutdown_connections 2023-03-03T12:40:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.622+0000 7fe488276700 1 -- 172.21.15.7:0/801139087 wait complete. 2023-03-03T12:40:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.623+0000 7fe488276700 1 Processor -- start 2023-03-03T12:40:04.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.623+0000 7fe488276700 1 -- start start 2023-03-03T12:40:04.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe488276700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ab520 0x7fe4801ea4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:04.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe488276700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4800ac340 0x7fe4800c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:04.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe488276700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4800c7820 0x7fe4801ecfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:04.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe488276700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4800c0660 con 0x7fe4800ab520 2023-03-03T12:40:04.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe488276700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801ea020 con 0x7fe4800c7820 2023-03-03T12:40:04.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe488276700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801e99f0 con 0x7fe4800ac340 2023-03-03T12:40:04.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486012700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ab520 0x7fe4801ea4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486012700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ab520 0x7fe4801ea4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41390/0 (socket says 172.21.15.7:41390) 2023-03-03T12:40:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486012700 1 -- 172.21.15.7:0/3101952335 learned_addr learned my addr 172.21.15.7:0/3101952335 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:40:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe485811700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4800ac340 0x7fe4800c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486012700 1 -- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4800ac340 msgr2=0x7fe4800c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486012700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4800ac340 0x7fe4800c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486012700 1 -- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4800c7820 msgr2=0x7fe4801ecfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:04.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486813700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4800c7820 0x7fe4801ecfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486012700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4800c7820 0x7fe4801ecfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486012700 1 -- 172.21.15.7:0/3101952335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47000b7f0 con 0x7fe4800ab520 2023-03-03T12:40:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe485811700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4800ac340 0x7fe4800c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:40:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486813700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4800c7820 0x7fe4801ecfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:40:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe486012700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ab520 0x7fe4801ea4b0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fe47000beb0 tx=0x7fe470004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe476ffd700 1 -- 172.21.15.7:0/3101952335 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470010670 con 0x7fe4800ab520 2023-03-03T12:40:04.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe476ffd700 1 -- 172.21.15.7:0/3101952335 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe470004240 con 0x7fe4800ab520 2023-03-03T12:40:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe488276700 1 -- 172.21.15.7:0/3101952335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4800c3e40 con 0x7fe4800ab520 2023-03-03T12:40:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe476ffd700 1 -- 172.21.15.7:0/3101952335 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470010a10 con 0x7fe4800ab520 2023-03-03T12:40:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.624+0000 7fe488276700 1 -- 172.21.15.7:0/3101952335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe480151df0 con 0x7fe4800ab520 2023-03-03T12:40:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.625+0000 7fe488276700 1 -- 172.21.15.7:0/3101952335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4800c3b00 con 0x7fe4800ab520 2023-03-03T12:40:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.626+0000 7fe476ffd700 1 -- 172.21.15.7:0/3101952335 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe47002c430 con 0x7fe4800ab520 2023-03-03T12:40:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.626+0000 7fe476ffd700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 0x7fe46406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:04.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.626+0000 7fe476ffd700 1 -- 172.21.15.7:0/3101952335 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe470088530 con 0x7fe4800ab520 2023-03-03T12:40:04.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.626+0000 7fe485811700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 0x7fe46406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:04.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.627+0000 7fe485811700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 0x7fe46406a7a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe48014d380 tx=0x7fe47c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:04.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.631+0000 7fe476ffd700 1 -- 172.21.15.7:0/3101952335 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe470054870 con 0x7fe4800ab520 2023-03-03T12:40:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.872+0000 7fe488276700 1 -- 172.21.15.7:0/3101952335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4801ea300 con 0x7fe4800ab520 2023-03-03T12:40:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.873+0000 7fe476ffd700 1 -- 172.21.15.7:0/3101952335 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe470057e90 con 0x7fe4800ab520 2023-03-03T12:40:04.875 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:40:04.875 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:40:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 -- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 msgr2=0x7fe46406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 0x7fe46406a7a0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fe48014d380 tx=0x7fe47c008040 comp rx=0 tx=0).stop 2023-03-03T12:40:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 -- 172.21.15.7:0/3101952335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ab520 msgr2=0x7fe4801ea4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ab520 0x7fe4801ea4b0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fe47000beb0 tx=0x7fe470004750 comp rx=0 tx=0).stop 2023-03-03T12:40:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 -- 172.21.15.7:0/3101952335 shutdown_connections 2023-03-03T12:40:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4800c7820 0x7fe4801ecfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 0x7fe46406a7a0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4800ab520 0x7fe4801ea4b0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 --2- 172.21.15.7:0/3101952335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4800ac340 0x7fe4800c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 -- 172.21.15.7:0/3101952335 >> 172.21.15.7:0/3101952335 conn(0x7fe48006c060 msgr2=0x7fe4800c4260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:40:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 -- 172.21.15.7:0/3101952335 shutdown_connections 2023-03-03T12:40:04.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:04.875+0000 7fe488276700 1 -- 172.21.15.7:0/3101952335 wait complete. 2023-03-03T12:40:04.879 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:40:04.889 INFO:tasks.mds_thrash.fs.[a]:waiting for 12 secs before thrashing 2023-03-03T12:40:16.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:40:17.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.025+0000 7f6aa8b2a700 1 Processor -- start 2023-03-03T12:40:17.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.025+0000 7f6aa8b2a700 1 -- start start 2023-03-03T12:40:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.026+0000 7f6aa8b2a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6aa414f9d0 0x7f6aa4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.026+0000 7f6aa8b2a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 0x7f6aa41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.026+0000 7f6aa8b2a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa40c2f30 con 0x7f6aa414fda0 2023-03-03T12:40:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.026+0000 7f6aa8b2a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa40c31d0 con 0x7f6aa414f9d0 2023-03-03T12:40:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.026+0000 7f6aa8b2a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa41524a0 con 0x7f6aa40abe20 2023-03-03T12:40:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa259c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6aa414f9d0 0x7f6aa4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa2d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 0x7f6aa41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa2d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 0x7f6aa41529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:49480/0 (socket says 172.21.15.7:49480) 2023-03-03T12:40:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa259c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6aa414f9d0 0x7f6aa4156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:37180/0 (socket says 172.21.15.7:37180) 2023-03-03T12:40:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa2d9d700 1 -- 172.21.15.7:0/3934130885 learned_addr learned my addr 172.21.15.7:0/3934130885 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:40:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa1d9b700 1 --1- 172.21.15.7:0/3934130885 >> v1:172.21.15.143:6789/0 conn(0x7f6aa414fda0 0x7f6aa4071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:38906/0 (socket says 172.21.15.7:38906) 2023-03-03T12:40:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa2d9d700 1 -- 172.21.15.7:0/3934130885 >> v2:172.21.15.143:3300/0 conn(0x7f6aa414f9d0 msgr2=0x7f6aa4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:40:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa2d9d700 1 --2- 172.21.15.7:0/3934130885 >> v2:172.21.15.143:3300/0 conn(0x7f6aa414f9d0 0x7f6aa4156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa2d9d700 1 -- 172.21.15.7:0/3934130885 >> v1:172.21.15.143:6789/0 conn(0x7f6aa414fda0 legacy=0x7f6aa4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa2d9d700 1 -- 172.21.15.7:0/3934130885 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa40c34b0 con 0x7f6aa40abe20 2023-03-03T12:40:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa259c700 1 --2- 172.21.15.7:0/3934130885 >> v2:172.21.15.143:3300/0 conn(0x7f6aa414f9d0 0x7f6aa4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:40:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa159a700 1 -- 172.21.15.7:0/3934130885 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033793891 0 0) 0x7f6aa40c2f30 con 0x7f6aa414fda0 2023-03-03T12:40:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa2d9d700 1 --2- 172.21.15.7:0/3934130885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 0x7f6aa41529c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009e50 tx=0x7f6a9c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d7995a25cba695 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa159a700 1 -- 172.21.15.7:0/3934130885 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c00e070 con 0x7f6aa40abe20 2023-03-03T12:40:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa159a700 1 -- 172.21.15.7:0/3934130885 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9c00b910 con 0x7f6aa40abe20 2023-03-03T12:40:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.027+0000 7f6aa159a700 1 -- 172.21.15.7:0/3934130885 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c00bcc0 con 0x7f6aa40abe20 2023-03-03T12:40:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.028+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/3934130885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 msgr2=0x7f6aa41529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.028+0000 7f6aa8b2a700 1 --2- 172.21.15.7:0/3934130885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 0x7f6aa41529c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009e50 tx=0x7f6a9c00b330 comp rx=0 tx=0).stop 2023-03-03T12:40:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.028+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/3934130885 shutdown_connections 2023-03-03T12:40:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.028+0000 7f6aa8b2a700 1 --2- 172.21.15.7:0/3934130885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 0x7f6aa41529c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.028+0000 7f6aa8b2a700 1 --2- 172.21.15.7:0/3934130885 >> v2:172.21.15.143:3300/0 conn(0x7f6aa414f9d0 0x7f6aa4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:17.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.028+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/3934130885 >> 172.21.15.7:0/3934130885 conn(0x7f6aa40c8900 msgr2=0x7f6aa40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:40:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.028+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/3934130885 shutdown_connections 2023-03-03T12:40:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.028+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/3934130885 wait complete. 2023-03-03T12:40:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.029+0000 7f6aa8b2a700 1 Processor -- start 2023-03-03T12:40:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.029+0000 7f6aa8b2a700 1 -- start start 2023-03-03T12:40:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa8b2a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 0x7f6aa41549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa8b2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6aa414f9d0 0x7f6aa414b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:17.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa8b2a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6aa414fda0 0x7f6aa414b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa8b2a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa406e250 con 0x7f6aa414fda0 2023-03-03T12:40:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa8b2a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa406d860 con 0x7f6aa414f9d0 2023-03-03T12:40:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa8b2a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa406df20 con 0x7f6aa40abe20 2023-03-03T12:40:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6aa414fda0 0x7f6aa414b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6aa414fda0 0x7f6aa414b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41414/0 (socket says 172.21.15.7:41414) 2023-03-03T12:40:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa2d9d700 1 -- 172.21.15.7:0/1349948937 learned_addr learned my addr 172.21.15.7:0/1349948937 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:40:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa259c700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 0x7f6aa41549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:17.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.030+0000 7f6aa1d9b700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6aa414f9d0 0x7f6aa414b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6aa2d9d700 1 -- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 msgr2=0x7f6aa41549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6aa2d9d700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 0x7f6aa41549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6aa2d9d700 1 -- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6aa414f9d0 msgr2=0x7f6aa414b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6aa2d9d700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6aa414f9d0 0x7f6aa414b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6aa2d9d700 1 -- 172.21.15.7:0/1349948937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a980094a0 con 0x7f6aa414fda0 2023-03-03T12:40:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6aa2d9d700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6aa414fda0 0x7f6aa414b960 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f6aa41525e0 tx=0x7f6a9c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:17.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6a92ffd700 1 -- 172.21.15.7:0/1349948937 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c004880 con 0x7f6aa414fda0 2023-03-03T12:40:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6a92ffd700 1 -- 172.21.15.7:0/1349948937 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9c0099b0 con 0x7f6aa414fda0 2023-03-03T12:40:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/1349948937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9c009b30 con 0x7f6aa414fda0 2023-03-03T12:40:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6a92ffd700 1 -- 172.21.15.7:0/1349948937 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c01b670 con 0x7f6aa414fda0 2023-03-03T12:40:17.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.031+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/1349948937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa40ac350 con 0x7f6aa414fda0 2023-03-03T12:40:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.032+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/1349948937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa40ab630 con 0x7f6aa414fda0 2023-03-03T12:40:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.033+0000 7f6a92ffd700 1 -- 172.21.15.7:0/1349948937 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6a9c021030 con 0x7f6aa414fda0 2023-03-03T12:40:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.033+0000 7f6a92ffd700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a84068320 0x7f6a8406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.033+0000 7f6a92ffd700 1 -- 172.21.15.7:0/1349948937 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6a9c0873e0 con 0x7f6aa414fda0 2023-03-03T12:40:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.033+0000 7f6aa259c700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a84068320 0x7f6a8406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:17.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.034+0000 7f6aa259c700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a84068320 0x7f6a8406a7d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009d70 tx=0x7f6a98009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:17.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.037+0000 7f6a92ffd700 1 -- 172.21.15.7:0/1349948937 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a9c053780 con 0x7f6aa414fda0 2023-03-03T12:40:17.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.276+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/1349948937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6aa4071fa0 con 0x7f6aa414fda0 2023-03-03T12:40:17.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.276+0000 7f6a92ffd700 1 -- 172.21.15.7:0/1349948937 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6a9c056da0 con 0x7f6aa414fda0 2023-03-03T12:40:17.279 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:40:17.280 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:40:17.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a84068320 msgr2=0x7f6a8406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a84068320 0x7f6a8406a7d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009d70 tx=0x7f6a98009210 comp rx=0 tx=0).stop 2023-03-03T12:40:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/1349948937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6aa414fda0 msgr2=0x7f6aa414b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6aa414fda0 0x7f6aa414b960 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f6aa41525e0 tx=0x7f6a9c004ad0 comp rx=0 tx=0).stop 2023-03-03T12:40:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/1349948937 shutdown_connections 2023-03-03T12:40:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a84068320 0x7f6a8406a7d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:17.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6aa414fda0 0x7f6aa414b960 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:17.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6aa40abe20 0x7f6aa41549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:17.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 --2- 172.21.15.7:0/1349948937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6aa414f9d0 0x7f6aa414b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:17.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/1349948937 >> 172.21.15.7:0/1349948937 conn(0x7f6aa40c8900 msgr2=0x7f6aa4151360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:40:17.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/1349948937 shutdown_connections 2023-03-03T12:40:17.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:17.279+0000 7f6aa8b2a700 1 -- 172.21.15.7:0/1349948937 wait complete. 2023-03-03T12:40:17.283 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:40:17.293 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before thrashing 2023-03-03T12:40:26.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:40:26.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.430+0000 7f91fdbfc700 1 Processor -- start 2023-03-03T12:40:26.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.430+0000 7f91fdbfc700 1 -- start start 2023-03-03T12:40:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91fdbfc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91f814b020 0x7f91f80717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91fdbfc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f80728b0 con 0x7f91f80aad20 2023-03-03T12:40:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91fdbfc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f8155070 con 0x7f91f8071d20 2023-03-03T12:40:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91fdbfc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f806f5b0 con 0x7f91f814b020 2023-03-03T12:40:26.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91f77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91f814b020 0x7f91f80717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:26.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91f6ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f91f80aad20 0x7f91f8157ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35412/0 (socket says 172.21.15.7:35412) 2023-03-03T12:40:26.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91f77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91f814b020 0x7f91f80717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41432/0 (socket says 172.21.15.7:41432) 2023-03-03T12:40:26.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91f6ffd700 1 -- 172.21.15.7:0/1383382880 learned_addr learned my addr 172.21.15.7:0/1383382880 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:40:26.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91f77fe700 1 -- 172.21.15.7:0/1383382880 >> v1:172.21.15.143:6789/0 conn(0x7f91f8071d20 legacy=0x7f91f81583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:40:26.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91f77fe700 1 -- 172.21.15.7:0/1383382880 >> v1:172.21.15.7:6789/0 conn(0x7f91f80aad20 legacy=0x7f91f8157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:26.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.431+0000 7f91f77fe700 1 -- 172.21.15.7:0/1383382880 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f806f890 con 0x7f91f814b020 2023-03-03T12:40:26.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91f77fe700 1 --2- 172.21.15.7:0/1383382880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f814b020 0x7f91f80717e0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f91e8009840 tx=0x7f91e8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95faee3bc273f6bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:26.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91f67fc700 1 -- 172.21.15.7:0/1383382880 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e8004230 con 0x7f91f814b020 2023-03-03T12:40:26.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91f67fc700 1 -- 172.21.15.7:0/1383382880 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e8004390 con 0x7f91f814b020 2023-03-03T12:40:26.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91f67fc700 1 -- 172.21.15.7:0/1383382880 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e80036a0 con 0x7f91f814b020 2023-03-03T12:40:26.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91fdbfc700 1 -- 172.21.15.7:0/1383382880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f814b020 msgr2=0x7f91f80717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:26.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91fdbfc700 1 --2- 172.21.15.7:0/1383382880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f814b020 0x7f91f80717e0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f91e8009840 tx=0x7f91e8009b50 comp rx=0 tx=0).stop 2023-03-03T12:40:26.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91fdbfc700 1 -- 172.21.15.7:0/1383382880 shutdown_connections 2023-03-03T12:40:26.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91fdbfc700 1 --2- 172.21.15.7:0/1383382880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f814b020 0x7f91f80717e0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:26.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91fdbfc700 1 -- 172.21.15.7:0/1383382880 >> 172.21.15.7:0/1383382880 conn(0x7f91f80c8900 msgr2=0x7f91f80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:40:26.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91fdbfc700 1 -- 172.21.15.7:0/1383382880 shutdown_connections 2023-03-03T12:40:26.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.432+0000 7f91fdbfc700 1 -- 172.21.15.7:0/1383382880 wait complete. 2023-03-03T12:40:26.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.433+0000 7f91fdbfc700 1 Processor -- start 2023-03-03T12:40:26.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.434+0000 7f91fdbfc700 1 -- start start 2023-03-03T12:40:26.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.434+0000 7f91fdbfc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f8071d20 0x7f91f80c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:26.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.434+0000 7f91fdbfc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80aad20 0x7f91f8143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:26.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.434+0000 7f91fdbfc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f814b020 0x7f91f8143e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:26.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.434+0000 7f91fdbfc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f8153ce0 con 0x7f91f80aad20 2023-03-03T12:40:26.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.434+0000 7f91fdbfc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f81e14e0 con 0x7f91f8071d20 2023-03-03T12:40:26.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.434+0000 7f91fdbfc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f8144390 con 0x7f91f814b020 2023-03-03T12:40:26.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80aad20 0x7f91f8143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:26.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80aad20 0x7f91f8143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41440/0 (socket says 172.21.15.7:41440) 2023-03-03T12:40:26.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f6ffd700 1 -- 172.21.15.7:0/3065346417 learned_addr learned my addr 172.21.15.7:0/3065346417 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:40:26.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f77fe700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f8071d20 0x7f91f80c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:26.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f7fff700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f814b020 0x7f91f8143e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:26.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f6ffd700 1 -- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f814b020 msgr2=0x7f91f8143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:26.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f6ffd700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f814b020 0x7f91f8143e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:26.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f6ffd700 1 -- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f8071d20 msgr2=0x7f91f80c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:26.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f6ffd700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f8071d20 0x7f91f80c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:26.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f6ffd700 1 -- 172.21.15.7:0/3065346417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e80094a0 con 0x7f91f80aad20 2023-03-03T12:40:26.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f77fe700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f8071d20 0x7f91f80c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:40:26.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91f6ffd700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80aad20 0x7f91f8143910 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f91e000c530 tx=0x7f91e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:26.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91effff700 1 -- 172.21.15.7:0/3065346417 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0005d70 con 0x7f91f80aad20 2023-03-03T12:40:26.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91effff700 1 -- 172.21.15.7:0/3065346417 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e00070d0 con 0x7f91f80aad20 2023-03-03T12:40:26.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91fdbfc700 1 -- 172.21.15.7:0/3065346417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f8155350 con 0x7f91f80aad20 2023-03-03T12:40:26.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91effff700 1 -- 172.21.15.7:0/3065346417 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0007460 con 0x7f91f80aad20 2023-03-03T12:40:26.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.435+0000 7f91fdbfc700 1 -- 172.21.15.7:0/3065346417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f806fb70 con 0x7f91f80aad20 2023-03-03T12:40:26.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.436+0000 7f91fdbfc700 1 -- 172.21.15.7:0/3065346417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f8145570 con 0x7f91f80aad20 2023-03-03T12:40:26.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.437+0000 7f91effff700 1 -- 172.21.15.7:0/3065346417 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f91e0019030 con 0x7f91f80aad20 2023-03-03T12:40:26.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.437+0000 7f91effff700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 0x7f91d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:26.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.437+0000 7f91effff700 1 -- 172.21.15.7:0/3065346417 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f91e0085580 con 0x7f91f80aad20 2023-03-03T12:40:26.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.437+0000 7f91f77fe700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 0x7f91d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:26.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.438+0000 7f91f77fe700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 0x7f91d806a7a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f91e80090b0 tx=0x7f91e8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:26.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.441+0000 7f91effff700 1 -- 172.21.15.7:0/3065346417 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f91e0051890 con 0x7f91f80aad20 2023-03-03T12:40:26.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.679+0000 7f91fdbfc700 1 -- 172.21.15.7:0/3065346417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91f8074bb0 con 0x7f91f80aad20 2023-03-03T12:40:26.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.679+0000 7f91effff700 1 -- 172.21.15.7:0/3065346417 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f91e0017020 con 0x7f91f80aad20 2023-03-03T12:40:26.682 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:40:26.682 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:40:26.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 -- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 msgr2=0x7f91d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:26.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 0x7f91d806a7a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f91e80090b0 tx=0x7f91e8008030 comp rx=0 tx=0).stop 2023-03-03T12:40:26.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 -- 172.21.15.7:0/3065346417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80aad20 msgr2=0x7f91f8143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:26.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80aad20 0x7f91f8143910 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f91e000c530 tx=0x7f91e000c8f0 comp rx=0 tx=0).stop 2023-03-03T12:40:26.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 -- 172.21.15.7:0/3065346417 shutdown_connections 2023-03-03T12:40:26.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f814b020 0x7f91f8143e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:26.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f8071d20 0x7f91f80c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:26.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 0x7f91d806a7a0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:26.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 --2- 172.21.15.7:0/3065346417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80aad20 0x7f91f8143910 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:26.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 -- 172.21.15.7:0/3065346417 >> 172.21.15.7:0/3065346417 conn(0x7f91f80c8900 msgr2=0x7f91f80c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:40:26.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 -- 172.21.15.7:0/3065346417 shutdown_connections 2023-03-03T12:40:26.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:26.681+0000 7f91fdbfc700 1 -- 172.21.15.7:0/3065346417 wait complete. 2023-03-03T12:40:26.686 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:40:26.696 INFO:tasks.mds_thrash.fs.[a]:waiting for 16 secs before thrashing 2023-03-03T12:40:31.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:31.481 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:40:42.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:40:42.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.834+0000 7f16c5fb4700 1 Processor -- start 2023-03-03T12:40:42.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.834+0000 7f16c5fb4700 1 -- start start 2023-03-03T12:40:42.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16c5fb4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f16c014e560 0x7f16c0071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:42.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16c5fb4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00c43d0 0x7f16c00c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:42.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16c5fb4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c0072920 con 0x7f16c0154590 2023-03-03T12:40:42.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16c5fb4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c006bef0 con 0x7f16c014e560 2023-03-03T12:40:42.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16c5fb4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c006b910 con 0x7f16c00c43d0 2023-03-03T12:40:42.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16bffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f16c0154590 0x7f16c01549f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35426/0 (socket says 172.21.15.7:35426) 2023-03-03T12:40:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16bffff700 1 -- 172.21.15.7:0/3678931522 learned_addr learned my addr 172.21.15.7:0/3678931522 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:40:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16bf7fe700 1 --2- 172.21.15.7:0/3678931522 >> v2:172.21.15.143:3300/0 conn(0x7f16c014e560 0x7f16c0071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16b7fff700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00c43d0 0x7f16c00c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491077812 0 0) 0x7f16b8001490 con 0x7f16c0154590 2023-03-03T12:40:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ac0035b0 con 0x7f16c0154590 2023-03-03T12:40:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 24021785 0 0) 0x7f16ac0035b0 con 0x7f16c0154590 2023-03-03T12:40:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00c43d0 msgr2=0x7f16c00c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:40:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00c43d0 0x7f16c00c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 >> v2:172.21.15.143:3300/0 conn(0x7f16c014e560 msgr2=0x7f16c0071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:40:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 --2- 172.21.15.7:0/3678931522 >> v2:172.21.15.143:3300/0 conn(0x7f16c014e560 0x7f16c0071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c00aacc0 con 0x7f16c0154590 2023-03-03T12:40:42.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f16b8001b80 con 0x7f16c0154590 2023-03-03T12:40:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 >> v1:172.21.15.7:6789/0 conn(0x7f16c0154590 legacy=0x7f16c01549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.835+0000 7f16beffd700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16ac003a70 0x7f16ac003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16beffd700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00c43d0 0x7f16ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16beffd700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16ac004d90 0x7f16ac005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c006bef0 con 0x7f16ac003a70 2023-03-03T12:40:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c006b910 con 0x7f16ac004d90 2023-03-03T12:40:42.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac00ba10 con 0x7f16c00c43d0 2023-03-03T12:40:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16bf7fe700 1 --2- 172.21.15.7:0/3678931522 >> v2:172.21.15.143:3300/0 conn(0x7f16c014e560 0x7f16c0071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:40:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16bffff700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16ac004d90 0x7f16ac005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16bf7fe700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16ac003a70 0x7f16ac003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16b7fff700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00c43d0 0x7f16ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16bf7fe700 1 -- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00c43d0 msgr2=0x7f16ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:40:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16bf7fe700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00c43d0 0x7f16ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16bf7fe700 1 -- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16ac004d90 msgr2=0x7f16ac005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:40:42.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16bf7fe700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16ac004d90 0x7f16ac005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16bf7fe700 1 -- 172.21.15.7:0/3678931522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac0035b0 con 0x7f16ac003a70 2023-03-03T12:40:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16bffff700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16ac004d90 0x7f16ac005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:40:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16b7fff700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00c43d0 0x7f16ac004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:40:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16bf7fe700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16ac003a70 0x7f16ac003e80 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009d40 tx=0x7f16ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac511cbdd12a46de server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0009e90 con 0x7f16ac003a70 2023-03-03T12:40:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b0004030 con 0x7f16ac003a70 2023-03-03T12:40:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.836+0000 7f16beffd700 1 -- 172.21.15.7:0/3678931522 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b00043a0 con 0x7f16ac003a70 2023-03-03T12:40:42.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.837+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3678931522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16ac003a70 msgr2=0x7f16ac003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.837+0000 7f16c5fb4700 1 --2- 172.21.15.7:0/3678931522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16ac003a70 0x7f16ac003e80 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009d40 tx=0x7f16ac005880 comp rx=0 tx=0).stop 2023-03-03T12:40:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.837+0000 7f16bffff700 1 -- 172.21.15.7:0/3678931522 reap_dead start 2023-03-03T12:40:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.837+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3678931522 shutdown_connections 2023-03-03T12:40:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.837+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3678931522 >> 172.21.15.7:0/3678931522 conn(0x7f16c00736f0 msgr2=0x7f16c0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:40:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.837+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3678931522 shutdown_connections 2023-03-03T12:40:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.837+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3678931522 wait complete. 2023-03-03T12:40:42.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.838+0000 7f16c5fb4700 1 Processor -- start 2023-03-03T12:40:42.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.838+0000 7f16c5fb4700 1 -- start start 2023-03-03T12:40:42.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16c5fb4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16ac003a70 0x7f16c01e9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:42.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16c5fb4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c0153560 0x7f16c00adac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:42.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16c5fb4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00ae000 0x7f16c014ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:42.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16c5fb4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c00ab9e0 con 0x7f16c0153560 2023-03-03T12:40:42.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16c5fb4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c01edcc0 con 0x7f16ac003a70 2023-03-03T12:40:42.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16c5fb4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c00ae480 con 0x7f16c00ae000 2023-03-03T12:40:42.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16b7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c0153560 0x7f16c00adac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16b7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c0153560 0x7f16c00adac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41460/0 (socket says 172.21.15.7:41460) 2023-03-03T12:40:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16b7fff700 1 -- 172.21.15.7:0/3083355872 learned_addr learned my addr 172.21.15.7:0/3083355872 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:40:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16bf7fe700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16ac003a70 0x7f16c01e9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16bffff700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00ae000 0x7f16c014ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16b7fff700 1 -- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00ae000 msgr2=0x7f16c014ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16b7fff700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00ae000 0x7f16c014ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16b7fff700 1 -- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16ac003a70 msgr2=0x7f16c01e9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:42.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16b7fff700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16ac003a70 0x7f16c01e9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.839+0000 7f16b7fff700 1 -- 172.21.15.7:0/3083355872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c0155100 con 0x7f16c0153560 2023-03-03T12:40:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.840+0000 7f16b7fff700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c0153560 0x7f16c00adac0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f16a8011cd0 tx=0x7f16a8009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.840+0000 7f16bcff9700 1 -- 172.21.15.7:0/3083355872 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a800f9e0 con 0x7f16c0153560 2023-03-03T12:40:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.840+0000 7f16bcff9700 1 -- 172.21.15.7:0/3083355872 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a800fb40 con 0x7f16c0153560 2023-03-03T12:40:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.840+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3083355872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c006bb80 con 0x7f16c0153560 2023-03-03T12:40:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.840+0000 7f16bcff9700 1 -- 172.21.15.7:0/3083355872 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8004020 con 0x7f16c0153560 2023-03-03T12:40:42.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.840+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3083355872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c00aa970 con 0x7f16c0153560 2023-03-03T12:40:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.841+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3083355872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c014eda0 con 0x7f16c0153560 2023-03-03T12:40:42.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.841+0000 7f16bcff9700 1 -- 172.21.15.7:0/3083355872 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f16a8024070 con 0x7f16c0153560 2023-03-03T12:40:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.842+0000 7f16bcff9700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16a00682f0 0x7f16a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:40:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.842+0000 7f16bcff9700 1 -- 172.21.15.7:0/3083355872 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f16a8086580 con 0x7f16c0153560 2023-03-03T12:40:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.842+0000 7f16bf7fe700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16a00682f0 0x7f16a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:40:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.842+0000 7f16bf7fe700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16a00682f0 0x7f16a006a7a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009ce0 tx=0x7f16b001d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:40:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:42.846+0000 7f16bcff9700 1 -- 172.21.15.7:0/3083355872 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f16a8052920 con 0x7f16c0153560 2023-03-03T12:40:43.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.085+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3083355872 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16c0071a50 con 0x7f16c0153560 2023-03-03T12:40:43.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.085+0000 7f16bcff9700 1 -- 172.21.15.7:0/3083355872 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f16a8022090 con 0x7f16c0153560 2023-03-03T12:40:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:40:43.088 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:40:43.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16a00682f0 msgr2=0x7f16a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:43.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16a00682f0 0x7f16a006a7a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009ce0 tx=0x7f16b001d040 comp rx=0 tx=0).stop 2023-03-03T12:40:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3083355872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c0153560 msgr2=0x7f16c00adac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:40:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c0153560 0x7f16c00adac0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f16a8011cd0 tx=0x7f16a8009f10 comp rx=0 tx=0).stop 2023-03-03T12:40:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3083355872 shutdown_connections 2023-03-03T12:40:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16c00ae000 0x7f16c014ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16ac003a70 0x7f16c01e9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16a00682f0 0x7f16a006a7a0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 --2- 172.21.15.7:0/3083355872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c0153560 0x7f16c00adac0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:40:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3083355872 >> 172.21.15.7:0/3083355872 conn(0x7f16c00736f0 msgr2=0x7f16c01527e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:40:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3083355872 shutdown_connections 2023-03-03T12:40:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:40:43.087+0000 7f16c5fb4700 1 -- 172.21.15.7:0/3083355872 wait complete. 2023-03-03T12:40:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:40:43.101 INFO:tasks.mds_thrash.fs.[a]:waiting for 67 secs before thrashing 2023-03-03T12:40:59.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:41:00.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.131+0000 7f0853144700 1 Processor -- start 2023-03-03T12:41:00.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.131+0000 7f0853144700 1 -- start start 2023-03-03T12:41:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.131+0000 7f0853144700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f084c1530d0 0x7f084c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.131+0000 7f0853144700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084c0c2f30 con 0x7f084c14f9d0 2023-03-03T12:41:00.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.131+0000 7f0853144700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084c0c31d0 con 0x7f084c14fda0 2023-03-03T12:41:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.131+0000 7f0853144700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084c1524a0 con 0x7f084c1530d0 2023-03-03T12:41:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.131+0000 7f08516e1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f084c1530d0 0x7f084c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.132+0000 7f0850ee0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f084c14f9d0 0x7f084c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35460/0 (socket says 172.21.15.7:35460) 2023-03-03T12:41:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.132+0000 7f0850ee0700 1 -- 172.21.15.7:0/1492348810 learned_addr learned my addr 172.21.15.7:0/1492348810 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:41:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.132+0000 7f08516e1700 1 -- 172.21.15.7:0/1492348810 >> v1:172.21.15.143:6789/0 conn(0x7f084c14fda0 legacy=0x7f084c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.132+0000 7f08516e1700 1 -- 172.21.15.7:0/1492348810 >> v1:172.21.15.7:6789/0 conn(0x7f084c14f9d0 legacy=0x7f084c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.132+0000 7f084b7fe700 1 -- 172.21.15.7:0/1492348810 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912375744 0 0) 0x7f084c0c2f30 con 0x7f084c14f9d0 2023-03-03T12:41:00.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.132+0000 7f08516e1700 1 -- 172.21.15.7:0/1492348810 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c0c34b0 con 0x7f084c1530d0 2023-03-03T12:41:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.132+0000 7f08516e1700 1 --2- 172.21.15.7:0/1492348810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 0x7f084c156430 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f0844009d90 tx=0x7f084400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35dcc33389b481ef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.132+0000 7f084b7fe700 1 -- 172.21.15.7:0/1492348810 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084400e070 con 0x7f084c1530d0 2023-03-03T12:41:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.132+0000 7f084b7fe700 1 -- 172.21.15.7:0/1492348810 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084400bbd0 con 0x7f084c1530d0 2023-03-03T12:41:00.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.132+0000 7f084b7fe700 1 -- 172.21.15.7:0/1492348810 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844012430 con 0x7f084c1530d0 2023-03-03T12:41:00.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.133+0000 7f0853144700 1 -- 172.21.15.7:0/1492348810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 msgr2=0x7f084c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:00.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.133+0000 7f0853144700 1 --2- 172.21.15.7:0/1492348810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 0x7f084c156430 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f0844009d90 tx=0x7f084400b330 comp rx=0 tx=0).stop 2023-03-03T12:41:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.133+0000 7f0853144700 1 -- 172.21.15.7:0/1492348810 shutdown_connections 2023-03-03T12:41:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.133+0000 7f0853144700 1 --2- 172.21.15.7:0/1492348810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 0x7f084c156430 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.133+0000 7f0853144700 1 -- 172.21.15.7:0/1492348810 >> 172.21.15.7:0/1492348810 conn(0x7f084c0c8900 msgr2=0x7f084c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:41:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.133+0000 7f0853144700 1 -- 172.21.15.7:0/1492348810 shutdown_connections 2023-03-03T12:41:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.133+0000 7f0853144700 1 -- 172.21.15.7:0/1492348810 wait complete. 2023-03-03T12:41:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.134+0000 7f0853144700 1 Processor -- start 2023-03-03T12:41:00.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.134+0000 7f0853144700 1 -- start start 2023-03-03T12:41:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f0853144700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f084c14f9d0 0x7f084c1eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f0853144700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c14fda0 0x7f084c151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f0853144700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 0x7f084c1deb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f0853144700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c06d7f0 con 0x7f084c1530d0 2023-03-03T12:41:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f0853144700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1de3d0 con 0x7f084c14fda0 2023-03-03T12:41:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f0853144700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c06e400 con 0x7f084c14f9d0 2023-03-03T12:41:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f08516e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 0x7f084c1deb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:00.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f08516e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 0x7f084c1deb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41490/0 (socket says 172.21.15.7:41490) 2023-03-03T12:41:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f08516e1700 1 -- 172.21.15.7:0/644882667 learned_addr learned my addr 172.21.15.7:0/644882667 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:41:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f0850ee0700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f084c14f9d0 0x7f084c1eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f084bfff700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c14fda0 0x7f084c151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f08516e1700 1 -- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f084c14f9d0 msgr2=0x7f084c1eb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f08516e1700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f084c14f9d0 0x7f084c1eb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f08516e1700 1 -- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c14fda0 msgr2=0x7f084c151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:00.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f08516e1700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c14fda0 0x7f084c151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.135+0000 7f08516e1700 1 -- 172.21.15.7:0/644882667 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0844009a70 con 0x7f084c1530d0 2023-03-03T12:41:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.136+0000 7f08516e1700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 0x7f084c1deb20 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f0844013000 tx=0x7f0844004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.136+0000 7f08497fa700 1 -- 172.21.15.7:0/644882667 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844012430 con 0x7f084c1530d0 2023-03-03T12:41:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.136+0000 7f08497fa700 1 -- 172.21.15.7:0/644882667 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0844020070 con 0x7f084c1530d0 2023-03-03T12:41:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.136+0000 7f0853144700 1 -- 172.21.15.7:0/644882667 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084c0ac350 con 0x7f084c1530d0 2023-03-03T12:41:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.136+0000 7f08497fa700 1 -- 172.21.15.7:0/644882667 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844007000 con 0x7f084c1530d0 2023-03-03T12:41:00.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.136+0000 7f0853144700 1 -- 172.21.15.7:0/644882667 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c0ab630 con 0x7f084c1530d0 2023-03-03T12:41:00.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.137+0000 7f0853144700 1 -- 172.21.15.7:0/644882667 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084c071fa0 con 0x7f084c1530d0 2023-03-03T12:41:00.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.137+0000 7f08497fa700 1 -- 172.21.15.7:0/644882667 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0844004280 con 0x7f084c1530d0 2023-03-03T12:41:00.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.138+0000 7f08497fa700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08300682f0 0x7f083006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:00.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.138+0000 7f08497fa700 1 -- 172.21.15.7:0/644882667 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0844022070 con 0x7f084c1530d0 2023-03-03T12:41:00.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.138+0000 7f0850ee0700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08300682f0 0x7f083006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:00.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.138+0000 7f0850ee0700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08300682f0 0x7f083006a7a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f0840000f60 tx=0x7f0840004a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:00.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.142+0000 7f08497fa700 1 -- 172.21.15.7:0/644882667 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0844054990 con 0x7f084c1530d0 2023-03-03T12:41:00.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.380+0000 7f0853144700 1 -- 172.21.15.7:0/644882667 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f084c074bb0 con 0x7f084c1530d0 2023-03-03T12:41:00.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.381+0000 7f08497fa700 1 -- 172.21.15.7:0/644882667 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0844057fb0 con 0x7f084c1530d0 2023-03-03T12:41:00.383 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:41:00.383 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:41:00.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 -- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08300682f0 msgr2=0x7f083006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:00.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08300682f0 0x7f083006a7a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f0840000f60 tx=0x7f0840004a50 comp rx=0 tx=0).stop 2023-03-03T12:41:00.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 -- 172.21.15.7:0/644882667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 msgr2=0x7f084c1deb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:00.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 0x7f084c1deb20 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f0844013000 tx=0x7f0844004ad0 comp rx=0 tx=0).stop 2023-03-03T12:41:00.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 -- 172.21.15.7:0/644882667 shutdown_connections 2023-03-03T12:41:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08300682f0 0x7f083006a7a0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1530d0 0x7f084c1deb20 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f084c14f9d0 0x7f084c1eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 --2- 172.21.15.7:0/644882667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f084c14fda0 0x7f084c151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 -- 172.21.15.7:0/644882667 >> 172.21.15.7:0/644882667 conn(0x7f084c0c8900 msgr2=0x7f084c0c7920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:41:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 -- 172.21.15.7:0/644882667 shutdown_connections 2023-03-03T12:41:00.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:00.383+0000 7f0853144700 1 -- 172.21.15.7:0/644882667 wait complete. 2023-03-03T12:41:00.387 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:41:00.397 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T12:41:01.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:01.628 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:18.398 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:41:18.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.533+0000 7f65a66e8700 1 Processor -- start 2023-03-03T12:41:18.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.533+0000 7f65a66e8700 1 -- start start 2023-03-03T12:41:18.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.533+0000 7f65a66e8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65a00c8790 0x7f65a00a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:18.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.533+0000 7f65a66e8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f65a00aa560 0x7f65a00c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:18.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.533+0000 7f65a66e8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a00aa0d0 con 0x7f65a006efd0 2023-03-03T12:41:18.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f65a66e8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a0071ec0 con 0x7f65a00c8790 2023-03-03T12:41:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f65a66e8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a00a9aa0 con 0x7f65a00aa560 2023-03-03T12:41:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65a00c8790 0x7f65a00a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65a00c8790 0x7f65a00a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41500/0 (socket says 172.21.15.7:41500) 2023-03-03T12:41:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659ffff700 1 -- 172.21.15.7:0/550868117 learned_addr learned my addr 172.21.15.7:0/550868117 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:41:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659ffff700 1 -- 172.21.15.7:0/550868117 >> v2:172.21.15.143:3300/0 conn(0x7f65a00aa560 msgr2=0x7f65a00c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:41:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f65a4c85700 1 --2- 172.21.15.7:0/550868117 >> v2:172.21.15.143:3300/0 conn(0x7f65a00aa560 0x7f65a00c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659ffff700 1 --2- 172.21.15.7:0/550868117 >> v2:172.21.15.143:3300/0 conn(0x7f65a00aa560 0x7f65a00c8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659ffff700 1 -- 172.21.15.7:0/550868117 >> v1:172.21.15.143:6789/0 conn(0x7f65a006efd0 legacy=0x7f65a01557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:41:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659ffff700 1 -- 172.21.15.7:0/550868117 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a00c2710 con 0x7f65a00c8790 2023-03-03T12:41:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f65a4c85700 1 --2- 172.21.15.7:0/550868117 >> v2:172.21.15.143:3300/0 conn(0x7f65a00aa560 0x7f65a00c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:41:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659ffff700 1 --2- 172.21.15.7:0/550868117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00c8790 0x7f65a00a94c0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f65a0071190 tx=0x7f6590009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4faaabaddd31029d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659effd700 1 -- 172.21.15.7:0/550868117 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590004230 con 0x7f65a00c8790 2023-03-03T12:41:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659effd700 1 -- 172.21.15.7:0/550868117 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6590004390 con 0x7f65a00c8790 2023-03-03T12:41:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.534+0000 7f659effd700 1 -- 172.21.15.7:0/550868117 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65900036a0 con 0x7f65a00c8790 2023-03-03T12:41:18.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.535+0000 7f65a66e8700 1 -- 172.21.15.7:0/550868117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00c8790 msgr2=0x7f65a00a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:18.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.535+0000 7f65a66e8700 1 --2- 172.21.15.7:0/550868117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00c8790 0x7f65a00a94c0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f65a0071190 tx=0x7f6590009bd0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.535+0000 7f65a66e8700 1 -- 172.21.15.7:0/550868117 shutdown_connections 2023-03-03T12:41:18.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.535+0000 7f65a66e8700 1 --2- 172.21.15.7:0/550868117 >> v2:172.21.15.143:3300/0 conn(0x7f65a00aa560 0x7f65a00c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.535+0000 7f65a66e8700 1 --2- 172.21.15.7:0/550868117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00c8790 0x7f65a00a94c0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.535+0000 7f65a66e8700 1 -- 172.21.15.7:0/550868117 >> 172.21.15.7:0/550868117 conn(0x7f65a00736f0 msgr2=0x7f65a0073af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:41:18.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.535+0000 7f65a66e8700 1 -- 172.21.15.7:0/550868117 shutdown_connections 2023-03-03T12:41:18.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.535+0000 7f65a66e8700 1 -- 172.21.15.7:0/550868117 wait complete. 2023-03-03T12:41:18.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.536+0000 7f65a66e8700 1 Processor -- start 2023-03-03T12:41:18.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.536+0000 7f65a66e8700 1 -- start start 2023-03-03T12:41:18.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f65a66e8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65a006efd0 0x7f65a00c1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:18.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f65a66e8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00aa560 0x7f65a0140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:18.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f65a66e8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65a00c8790 0x7f65a0141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:18.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f65a66e8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00ad310 con 0x7f65a00aa560 2023-03-03T12:41:18.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f65a66e8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a0141f60 con 0x7f65a00c8790 2023-03-03T12:41:18.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f65a66e8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00ab640 con 0x7f65a006efd0 2023-03-03T12:41:18.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f659f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00aa560 0x7f65a0140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:18.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f659f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00aa560 0x7f65a0140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41506/0 (socket says 172.21.15.7:41506) 2023-03-03T12:41:18.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f659f7fe700 1 -- 172.21.15.7:0/3267494337 learned_addr learned my addr 172.21.15.7:0/3267494337 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:41:18.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f65a4c85700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65a00c8790 0x7f65a0141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:18.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f659ffff700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65a006efd0 0x7f65a00c1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:18.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f659f7fe700 1 -- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65a006efd0 msgr2=0x7f65a00c1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:18.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f659f7fe700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65a006efd0 0x7f65a00c1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f659f7fe700 1 -- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65a00c8790 msgr2=0x7f65a0141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:18.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f659f7fe700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65a00c8790 0x7f65a0141490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.537+0000 7f659f7fe700 1 -- 172.21.15.7:0/3267494337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6590009550 con 0x7f65a00aa560 2023-03-03T12:41:18.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.538+0000 7f659f7fe700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00aa560 0x7f65a0140f50 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f659400b4a0 tx=0x7f659400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:18.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.538+0000 7f659cff9700 1 -- 172.21.15.7:0/3267494337 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594011830 con 0x7f65a00aa560 2023-03-03T12:41:18.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.538+0000 7f659cff9700 1 -- 172.21.15.7:0/3267494337 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6594011990 con 0x7f65a00aa560 2023-03-03T12:41:18.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.538+0000 7f65a66e8700 1 -- 172.21.15.7:0/3267494337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a00a9d60 con 0x7f65a00aa560 2023-03-03T12:41:18.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.538+0000 7f659cff9700 1 -- 172.21.15.7:0/3267494337 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594011d40 con 0x7f65a00aa560 2023-03-03T12:41:18.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.538+0000 7f65a66e8700 1 -- 172.21.15.7:0/3267494337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a00c29f0 con 0x7f65a00aa560 2023-03-03T12:41:18.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.539+0000 7f65a66e8700 1 -- 172.21.15.7:0/3267494337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a014c8c0 con 0x7f65a00aa560 2023-03-03T12:41:18.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.539+0000 7f659cff9700 1 -- 172.21.15.7:0/3267494337 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f659400f050 con 0x7f65a00aa560 2023-03-03T12:41:18.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.540+0000 7f659cff9700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f658c0682f0 0x7f658c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:18.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.540+0000 7f659cff9700 1 -- 172.21.15.7:0/3267494337 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6594085bc0 con 0x7f65a00aa560 2023-03-03T12:41:18.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.540+0000 7f659ffff700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f658c0682f0 0x7f658c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:18.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.540+0000 7f659ffff700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f658c0682f0 0x7f658c06a7a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6590009130 tx=0x7f6590008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:18.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.544+0000 7f659cff9700 1 -- 172.21.15.7:0/3267494337 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6594051f60 con 0x7f65a00aa560 2023-03-03T12:41:18.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.784+0000 7f65a66e8700 1 -- 172.21.15.7:0/3267494337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65a0074bb0 con 0x7f65a00aa560 2023-03-03T12:41:18.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.784+0000 7f659cff9700 1 -- 172.21.15.7:0/3267494337 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6594055580 con 0x7f65a00aa560 2023-03-03T12:41:18.787 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:41:18.787 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:41:18.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 -- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f658c0682f0 msgr2=0x7f658c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:18.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f658c0682f0 0x7f658c06a7a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6590009130 tx=0x7f6590008030 comp rx=0 tx=0).stop 2023-03-03T12:41:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 -- 172.21.15.7:0/3267494337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00aa560 msgr2=0x7f65a0140f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00aa560 0x7f65a0140f50 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f659400b4a0 tx=0x7f659400b7b0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 -- 172.21.15.7:0/3267494337 shutdown_connections 2023-03-03T12:41:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65a00c8790 0x7f65a0141490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65a006efd0 0x7f65a00c1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f658c0682f0 0x7f658c06a7a0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 --2- 172.21.15.7:0/3267494337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65a00aa560 0x7f65a0140f50 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 -- 172.21.15.7:0/3267494337 >> 172.21.15.7:0/3267494337 conn(0x7f65a00736f0 msgr2=0x7f65a014a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:41:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 -- 172.21.15.7:0/3267494337 shutdown_connections 2023-03-03T12:41:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:18.786+0000 7f65a66e8700 1 -- 172.21.15.7:0/3267494337 wait complete. 2023-03-03T12:41:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:41:18.800 INFO:tasks.mds_thrash.fs.[b]:waiting for 94 secs before thrashing 2023-03-03T12:41:31.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:31.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:41:50.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:41:50.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.238+0000 7fee338b8700 1 Processor -- start 2023-03-03T12:41:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.238+0000 7fee338b8700 1 -- start start 2023-03-03T12:41:50.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.239+0000 7fee338b8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fee2c0c5b00 0x7fee2c0bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:50.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.239+0000 7fee338b8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c0c08d0 con 0x7fee2c155410 2023-03-03T12:41:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.239+0000 7fee338b8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c154930 con 0x7fee2c151e80 2023-03-03T12:41:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.239+0000 7fee338b8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee2c152250 con 0x7fee2c0c5b00 2023-03-03T12:41:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.239+0000 7fee30e53700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fee2c155410 0x7fee2c0bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35502/0 (socket says 172.21.15.7:35502) 2023-03-03T12:41:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.239+0000 7fee30e53700 1 -- 172.21.15.7:0/3654951351 learned_addr learned my addr 172.21.15.7:0/3654951351 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:41:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee31e55700 1 --2- 172.21.15.7:0/3654951351 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0c5b00 0x7fee2c0bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989049255 0 0) 0x7fee2c0c08d0 con 0x7fee2c155410 2023-03-03T12:41:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee180035b0 con 0x7fee2c155410 2023-03-03T12:41:50.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1419507529 0 0) 0x7fee180035b0 con 0x7fee2c155410 2023-03-03T12:41:50.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0c5b00 msgr2=0x7fee2c0bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:41:50.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 --2- 172.21.15.7:0/3654951351 >> v2:172.21.15.143:3300/0 conn(0x7fee2c0c5b00 0x7fee2c0bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:50.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 >> v1:172.21.15.143:6789/0 conn(0x7fee2c151e80 legacy=0x7fee2c0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:50.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c154c10 con 0x7fee2c155410 2023-03-03T12:41:50.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fee28001670 con 0x7fee2c155410 2023-03-03T12:41:50.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 >> v1:172.21.15.7:6789/0 conn(0x7fee2c155410 legacy=0x7fee2c0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:50.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee18003a70 0x7fee18003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:50.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee18004850 0x7fee18004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:50.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18005200 0x7fee1800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:50.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1800c0e0 con 0x7fee18003a70 2023-03-03T12:41:50.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1800c220 con 0x7fee18004850 2023-03-03T12:41:50.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1800c3e0 con 0x7fee18005200 2023-03-03T12:41:50.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee31654700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee18003a70 0x7fee18003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:50.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee30e53700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee18004850 0x7fee18004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:50.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee31e55700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18005200 0x7fee1800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:50.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee31654700 1 -- 172.21.15.7:0/3654951351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18005200 msgr2=0x7fee1800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:41:50.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee31654700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18005200 0x7fee1800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:50.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee31654700 1 -- 172.21.15.7:0/3654951351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee18004850 msgr2=0x7fee18004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:41:50.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee31654700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee18004850 0x7fee18004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:50.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee31654700 1 -- 172.21.15.7:0/3654951351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee180035b0 con 0x7fee18003a70 2023-03-03T12:41:50.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.240+0000 7fee31e55700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18005200 0x7fee1800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:41:50.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee31e55700 1 -- 172.21.15.7:0/3654951351 reap_dead start 2023-03-03T12:41:50.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee31654700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee18003a70 0x7fee18003e80 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0017c0 tx=0x7fee18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e83ee26b526cf3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:50.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0052d0 con 0x7fee18003a70 2023-03-03T12:41:50.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c005430 con 0x7fee18003a70 2023-03-03T12:41:50.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee23fff700 1 -- 172.21.15.7:0/3654951351 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0038c0 con 0x7fee18003a70 2023-03-03T12:41:50.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee338b8700 1 -- 172.21.15.7:0/3654951351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee18003a70 msgr2=0x7fee18003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:50.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee338b8700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee18003a70 0x7fee18003e80 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0017c0 tx=0x7fee18005880 comp rx=0 tx=0).stop 2023-03-03T12:41:50.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee338b8700 1 -- 172.21.15.7:0/3654951351 shutdown_connections 2023-03-03T12:41:50.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee338b8700 1 --2- 172.21.15.7:0/3654951351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee18003a70 0x7fee18003e80 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:50.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee338b8700 1 -- 172.21.15.7:0/3654951351 >> 172.21.15.7:0/3654951351 conn(0x7fee2c071630 msgr2=0x7fee2c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:41:50.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee338b8700 1 -- 172.21.15.7:0/3654951351 shutdown_connections 2023-03-03T12:41:50.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.241+0000 7fee338b8700 1 -- 172.21.15.7:0/3654951351 wait complete. 2023-03-03T12:41:50.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.243+0000 7fee338b8700 1 Processor -- start 2023-03-03T12:41:50.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.243+0000 7fee338b8700 1 -- start start 2023-03-03T12:41:50.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.243+0000 7fee338b8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18003a70 0x7fee2c1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.243+0000 7fee338b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c155ab0 0x7fee2c158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.243+0000 7fee338b8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c145650 0x7fee2c145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.243+0000 7fee338b8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c1df7f0 con 0x7fee2c155ab0 2023-03-03T12:41:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.243+0000 7fee338b8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c157dd0 con 0x7fee2c145650 2023-03-03T12:41:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.243+0000 7fee338b8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c146030 con 0x7fee18003a70 2023-03-03T12:41:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee30e53700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c155ab0 0x7fee2c158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee30e53700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c155ab0 0x7fee2c158730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41534/0 (socket says 172.21.15.7:41534) 2023-03-03T12:41:50.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee30e53700 1 -- 172.21.15.7:0/3046796072 learned_addr learned my addr 172.21.15.7:0/3046796072 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:41:50.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee31e55700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c145650 0x7fee2c145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:50.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee31654700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18003a70 0x7fee2c1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:50.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee30e53700 1 -- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18003a70 msgr2=0x7fee2c1581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:50.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee30e53700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18003a70 0x7fee2c1581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:50.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee30e53700 1 -- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c145650 msgr2=0x7fee2c145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee30e53700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c145650 0x7fee2c145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee30e53700 1 -- 172.21.15.7:0/3046796072 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c0c5220 con 0x7fee2c155ab0 2023-03-03T12:41:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee31654700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18003a70 0x7fee2c1581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:41:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee30e53700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c155ab0 0x7fee2c158730 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fee28001050 tx=0x7fee28007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee21ffb700 1 -- 172.21.15.7:0/3046796072 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28012560 con 0x7fee2c155ab0 2023-03-03T12:41:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee21ffb700 1 -- 172.21.15.7:0/3046796072 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee280126e0 con 0x7fee2c155ab0 2023-03-03T12:41:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee338b8700 1 -- 172.21.15.7:0/3046796072 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2c154ef0 con 0x7fee2c155ab0 2023-03-03T12:41:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee21ffb700 1 -- 172.21.15.7:0/3046796072 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28012a90 con 0x7fee2c155ab0 2023-03-03T12:41:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.244+0000 7fee338b8700 1 -- 172.21.15.7:0/3046796072 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c0c6aa0 con 0x7fee2c155ab0 2023-03-03T12:41:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.245+0000 7fee338b8700 1 -- 172.21.15.7:0/3046796072 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c156ce0 con 0x7fee2c155ab0 2023-03-03T12:41:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.246+0000 7fee21ffb700 1 -- 172.21.15.7:0/3046796072 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fee28012bf0 con 0x7fee2c155ab0 2023-03-03T12:41:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.246+0000 7fee21ffb700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 0x7fee1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.246+0000 7fee21ffb700 1 -- 172.21.15.7:0/3046796072 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fee28086260 con 0x7fee2c155ab0 2023-03-03T12:41:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.247+0000 7fee31654700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 0x7fee1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:50.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.247+0000 7fee31654700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 0x7fee1006a7a0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fee1c015040 tx=0x7fee1c009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:50.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.251+0000 7fee21ffb700 1 -- 172.21.15.7:0/3046796072 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fee280525a0 con 0x7fee2c155ab0 2023-03-03T12:41:50.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.489+0000 7fee338b8700 1 -- 172.21.15.7:0/3046796072 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee2c157f10 con 0x7fee2c155ab0 2023-03-03T12:41:50.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.489+0000 7fee21ffb700 1 -- 172.21.15.7:0/3046796072 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fee2c157f10 con 0x7fee2c155ab0 2023-03-03T12:41:50.492 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:41:50.492 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:41:50.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.491+0000 7fee338b8700 1 -- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 msgr2=0x7fee1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:50.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.491+0000 7fee338b8700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 0x7fee1006a7a0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fee1c015040 tx=0x7fee1c009030 comp rx=0 tx=0).stop 2023-03-03T12:41:50.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.492+0000 7fee338b8700 1 -- 172.21.15.7:0/3046796072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c155ab0 msgr2=0x7fee2c158730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:50.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.492+0000 7fee338b8700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c155ab0 0x7fee2c158730 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fee28001050 tx=0x7fee28007570 comp rx=0 tx=0).stop 2023-03-03T12:41:50.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.492+0000 7fee338b8700 1 -- 172.21.15.7:0/3046796072 shutdown_connections 2023-03-03T12:41:50.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.492+0000 7fee338b8700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee2c145650 0x7fee2c145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:50.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.492+0000 7fee338b8700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fee18003a70 0x7fee2c1581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:50.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.492+0000 7fee338b8700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fee100682f0 0x7fee1006a7a0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:50.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.492+0000 7fee338b8700 1 --2- 172.21.15.7:0/3046796072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fee2c155ab0 0x7fee2c158730 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:50.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.492+0000 7fee338b8700 1 -- 172.21.15.7:0/3046796072 >> 172.21.15.7:0/3046796072 conn(0x7fee2c071630 msgr2=0x7fee2c0c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:41:50.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.492+0000 7fee338b8700 1 -- 172.21.15.7:0/3046796072 shutdown_connections 2023-03-03T12:41:50.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:50.492+0000 7fee338b8700 1 -- 172.21.15.7:0/3046796072 wait complete. 2023-03-03T12:41:50.496 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:41:50.505 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before thrashing 2023-03-03T12:41:59.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:41:59.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.644+0000 7f1184785700 1 Processor -- start 2023-03-03T12:41:59.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.644+0000 7f1184785700 1 -- start start 2023-03-03T12:41:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f1184785700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f117c14fcb0 0x7f117c070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f1184785700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c155b90 0x7f117c155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f1184785700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c0c2eb0 con 0x7f117c0c2730 2023-03-03T12:41:59.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f1184785700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117c071ea0 con 0x7f117c14fcb0 2023-03-03T12:41:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f1184785700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0c3100 con 0x7f117c155b90 2023-03-03T12:41:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f1181d20700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f117c0c2730 0x7f117c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35520/0 (socket says 172.21.15.7:35520) 2023-03-03T12:41:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f1181d20700 1 -- 172.21.15.7:0/119800181 learned_addr learned my addr 172.21.15.7:0/119800181 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:41:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240367435 0 0) 0x7f117c0c2eb0 con 0x7f117c0c2730 2023-03-03T12:41:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11680035b0 con 0x7f117c0c2730 2023-03-03T12:41:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f1182521700 1 --2- 172.21.15.7:0/119800181 >> v2:172.21.15.143:3300/0 conn(0x7f117c14fcb0 0x7f117c070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:59.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f1182d22700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c155b90 0x7f117c155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.646+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2776662145 0 0) 0x7f11680035b0 con 0x7f117c0c2730 2023-03-03T12:41:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c155b90 msgr2=0x7f117c155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:41:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c155b90 0x7f117c155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 >> v2:172.21.15.143:3300/0 conn(0x7f117c14fcb0 msgr2=0x7f117c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:41:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 --2- 172.21.15.7:0/119800181 >> v2:172.21.15.143:3300/0 conn(0x7f117c14fcb0 0x7f117c070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0c33e0 con 0x7f117c0c2730 2023-03-03T12:41:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f1178001670 con 0x7f117c0c2730 2023-03-03T12:41:59.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 >> v1:172.21.15.7:6789/0 conn(0x7f117c0c2730 legacy=0x7f117c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 0x7f1168003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1168004850 0x7f1168004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c155b90 0x7f1168005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c071ea0 con 0x7f1168003a70 2023-03-03T12:41:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0c3100 con 0x7f1168004850 2023-03-03T12:41:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116800bb10 con 0x7f117c155b90 2023-03-03T12:41:59.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1182521700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 0x7f1168003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1181d20700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1168004850 0x7f1168004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1182d22700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c155b90 0x7f1168005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1182521700 1 -- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c155b90 msgr2=0x7f1168005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1182521700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c155b90 0x7f1168005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1182521700 1 -- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1168004850 msgr2=0x7f1168004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1182521700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1168004850 0x7f1168004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1182521700 1 -- 172.21.15.7:0/119800181 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11680035b0 con 0x7f1168003a70 2023-03-03T12:41:59.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1182d22700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c155b90 0x7f1168005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:41:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1181d20700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1168004850 0x7f1168004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:41:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f1182521700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 0x7f1168003e80 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f1170009b90 tx=0x7f1168005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7422d9856b16b7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117000e6c0 con 0x7f1168003a70 2023-03-03T12:41:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117000e820 con 0x7f1168003a70 2023-03-03T12:41:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.647+0000 7f118151f700 1 -- 172.21.15.7:0/119800181 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117000eb90 con 0x7f1168003a70 2023-03-03T12:41:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.648+0000 7f1184785700 1 -- 172.21.15.7:0/119800181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 msgr2=0x7f1168003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:59.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.648+0000 7f1184785700 1 --2- 172.21.15.7:0/119800181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 0x7f1168003e80 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f1170009b90 tx=0x7f1168005880 comp rx=0 tx=0).stop 2023-03-03T12:41:59.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.648+0000 7f1182d22700 1 -- 172.21.15.7:0/119800181 reap_dead start 2023-03-03T12:41:59.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.648+0000 7f1184785700 1 -- 172.21.15.7:0/119800181 shutdown_connections 2023-03-03T12:41:59.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.648+0000 7f1184785700 1 -- 172.21.15.7:0/119800181 >> 172.21.15.7:0/119800181 conn(0x7f117c0a94d0 msgr2=0x7f117c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:41:59.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.648+0000 7f1184785700 1 -- 172.21.15.7:0/119800181 shutdown_connections 2023-03-03T12:41:59.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.648+0000 7f1184785700 1 -- 172.21.15.7:0/119800181 wait complete. 2023-03-03T12:41:59.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.649+0000 7f1184785700 1 Processor -- start 2023-03-03T12:41:59.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.649+0000 7f1184785700 1 -- start start 2023-03-03T12:41:59.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1184785700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 0x7f117c0be430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:59.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1184785700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117c1eb1c0 0x7f117c1eb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:59.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1184785700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c1518f0 0x7f117c151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:59.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1184785700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c149d90 con 0x7f1168003a70 2023-03-03T12:41:59.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1184785700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c150d00 con 0x7f117c1eb1c0 2023-03-03T12:41:59.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1184785700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c14a2c0 con 0x7f117c1518f0 2023-03-03T12:41:59.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1182521700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 0x7f117c0be430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:59.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1182521700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 0x7f117c0be430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41552/0 (socket says 172.21.15.7:41552) 2023-03-03T12:41:59.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1182521700 1 -- 172.21.15.7:0/3321886110 learned_addr learned my addr 172.21.15.7:0/3321886110 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:41:59.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1181d20700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117c1eb1c0 0x7f117c1eb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:59.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1182d22700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c1518f0 0x7f117c151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:59.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1182521700 1 -- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c1518f0 msgr2=0x7f117c151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:59.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1182521700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c1518f0 0x7f117c151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1182521700 1 -- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117c1eb1c0 msgr2=0x7f117c1eb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:59.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1182521700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117c1eb1c0 0x7f117c1eb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1182521700 1 -- 172.21.15.7:0/3321886110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0c39a0 con 0x7f1168003a70 2023-03-03T12:41:59.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1181d20700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117c1eb1c0 0x7f117c1eb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:41:59.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.650+0000 7f1182d22700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c1518f0 0x7f117c151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:41:59.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.651+0000 7f1182521700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 0x7f117c0be430 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f117000e4b0 tx=0x7f11700047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:59.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.651+0000 7f116effd700 1 -- 172.21.15.7:0/3321886110 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1170021070 con 0x7f1168003a70 2023-03-03T12:41:59.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.651+0000 7f116effd700 1 -- 172.21.15.7:0/3321886110 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1170004540 con 0x7f1168003a70 2023-03-03T12:41:59.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.651+0000 7f116effd700 1 -- 172.21.15.7:0/3321886110 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117000e6c0 con 0x7f1168003a70 2023-03-03T12:41:59.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.651+0000 7f1184785700 1 -- 172.21.15.7:0/3321886110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c0c36c0 con 0x7f1168003a70 2023-03-03T12:41:59.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.651+0000 7f1184785700 1 -- 172.21.15.7:0/3321886110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c073e00 con 0x7f1168003a70 2023-03-03T12:41:59.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.652+0000 7f1184785700 1 -- 172.21.15.7:0/3321886110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c153750 con 0x7f1168003a70 2023-03-03T12:41:59.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.652+0000 7f116effd700 1 -- 172.21.15.7:0/3321886110 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f117002e030 con 0x7f1168003a70 2023-03-03T12:41:59.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.653+0000 7f116effd700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11600682f0 0x7f116006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:41:59.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.653+0000 7f116effd700 1 -- 172.21.15.7:0/3321886110 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1170087410 con 0x7f1168003a70 2023-03-03T12:41:59.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.653+0000 7f1181d20700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11600682f0 0x7f116006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:41:59.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.653+0000 7f1181d20700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11600682f0 0x7f116006a7a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1178001eb0 tx=0x7f11780010c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:41:59.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.657+0000 7f116effd700 1 -- 172.21.15.7:0/3321886110 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11700537b0 con 0x7f1168003a70 2023-03-03T12:41:59.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.897+0000 7f1184785700 1 -- 172.21.15.7:0/3321886110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f117c149d90 con 0x7f1168003a70 2023-03-03T12:41:59.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.897+0000 7f116effd700 1 -- 172.21.15.7:0/3321886110 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1170056dd0 con 0x7f1168003a70 2023-03-03T12:41:59.900 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:41:59.900 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:41:59.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 -- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11600682f0 msgr2=0x7f116006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:59.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11600682f0 0x7f116006a7a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1178001eb0 tx=0x7f11780010c0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 -- 172.21.15.7:0/3321886110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 msgr2=0x7f117c0be430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:41:59.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 0x7f117c0be430 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f117000e4b0 tx=0x7f11700047c0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 -- 172.21.15.7:0/3321886110 shutdown_connections 2023-03-03T12:41:59.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f117c1518f0 0x7f117c151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11600682f0 0x7f116006a7a0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1168003a70 0x7f117c0be430 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 --2- 172.21.15.7:0/3321886110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f117c1eb1c0 0x7f117c1eb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:41:59.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 -- 172.21.15.7:0/3321886110 >> 172.21.15.7:0/3321886110 conn(0x7f117c0a94d0 msgr2=0x7f117c070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:41:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 -- 172.21.15.7:0/3321886110 shutdown_connections 2023-03-03T12:41:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:41:59.899+0000 7f1184785700 1 -- 172.21.15.7:0/3321886110 wait complete. 2023-03-03T12:41:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:41:59.913 INFO:tasks.mds_thrash.fs.[a]:waiting for 104 secs before thrashing 2023-03-03T12:42:02.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:02.020 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:32.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:32.101 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:42:52.801 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:42:52.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.939+0000 7fadf1ec7700 1 Processor -- start 2023-03-03T12:42:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.939+0000 7fadf1ec7700 1 -- start start 2023-03-03T12:42:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.939+0000 7fadf1ec7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec14b0d0 0x7fadec071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.939+0000 7fadf1ec7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadec071d40 0x7fadec158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.939+0000 7fadf1ec7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec0a9c20 con 0x7fadec0aad20 2023-03-03T12:42:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.939+0000 7fadf1ec7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadec154f10 con 0x7fadec071d40 2023-03-03T12:42:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.939+0000 7fadf1ec7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadec0ac3c0 con 0x7fadec14b0d0 2023-03-03T12:42:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadebfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadec071d40 0x7fadec158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadeaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fadec0aad20 0x7fadec157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35568/0 (socket says 172.21.15.7:35568) 2023-03-03T12:42:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadeaffd700 1 -- 172.21.15.7:0/1661612032 learned_addr learned my addr 172.21.15.7:0/1661612032 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:42:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadebfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fadec071d40 0x7fadec158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41590/0 (socket says 172.21.15.7:41590) 2023-03-03T12:42:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadeb7fe700 1 --2- 172.21.15.7:0/1661612032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec14b0d0 0x7fadec071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadebfff700 1 -- 172.21.15.7:0/1661612032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec14b0d0 msgr2=0x7fadec071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:42:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadebfff700 1 --2- 172.21.15.7:0/1661612032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec14b0d0 0x7fadec071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadebfff700 1 -- 172.21.15.7:0/1661612032 >> v1:172.21.15.7:6789/0 conn(0x7fadec0aad20 legacy=0x7fadec157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadebfff700 1 -- 172.21.15.7:0/1661612032 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec06fc70 con 0x7fadec071d40 2023-03-03T12:42:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadea7fc700 1 -- 172.21.15.7:0/1661612032 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681828032 0 0) 0x7fadec0a9c20 con 0x7fadec0aad20 2023-03-03T12:42:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadebfff700 1 --2- 172.21.15.7:0/1661612032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec071d40 0x7fadec158500 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fade4009840 tx=0x7fade4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b28186afe178617 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:42:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadea7fc700 1 -- 172.21.15.7:0/1661612032 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4011830 con 0x7fadec071d40 2023-03-03T12:42:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadea7fc700 1 -- 172.21.15.7:0/1661612032 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade4011990 con 0x7fadec071d40 2023-03-03T12:42:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.940+0000 7fadea7fc700 1 -- 172.21.15.7:0/1661612032 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4011d20 con 0x7fadec071d40 2023-03-03T12:42:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.941+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1661612032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec071d40 msgr2=0x7fadec158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.941+0000 7fadf1ec7700 1 --2- 172.21.15.7:0/1661612032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec071d40 0x7fadec158500 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fade4009840 tx=0x7fade4009b50 comp rx=0 tx=0).stop 2023-03-03T12:42:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.941+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1661612032 shutdown_connections 2023-03-03T12:42:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.941+0000 7fadf1ec7700 1 --2- 172.21.15.7:0/1661612032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec071d40 0x7fadec158500 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.941+0000 7fadf1ec7700 1 --2- 172.21.15.7:0/1661612032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec14b0d0 0x7fadec071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.941+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1661612032 >> 172.21.15.7:0/1661612032 conn(0x7fadec0c8900 msgr2=0x7fadec0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:42:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.941+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1661612032 shutdown_connections 2023-03-03T12:42:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.941+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1661612032 wait complete. 2023-03-03T12:42:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.942+0000 7fadf1ec7700 1 Processor -- start 2023-03-03T12:42:52.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.942+0000 7fadf1ec7700 1 -- start start 2023-03-03T12:42:52.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadf1ec7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadec071d40 0x7fadec152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:52.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadf1ec7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec0aad20 0x7fadec1570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:52.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadf1ec7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec14b0d0 0x7fadec1ed600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:52.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadf1ec7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec1ece90 con 0x7fadec14b0d0 2023-03-03T12:42:52.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadf1ec7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec06ae40 con 0x7fadec071d40 2023-03-03T12:42:52.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadf1ec7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadec153ff0 con 0x7fadec0aad20 2023-03-03T12:42:52.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadebfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec14b0d0 0x7fadec1ed600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:52.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadebfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec14b0d0 0x7fadec1ed600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41596/0 (socket says 172.21.15.7:41596) 2023-03-03T12:42:52.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadebfff700 1 -- 172.21.15.7:0/1054082292 learned_addr learned my addr 172.21.15.7:0/1054082292 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:42:52.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadeb7fe700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadec071d40 0x7fadec152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:52.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.943+0000 7fadeaffd700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec0aad20 0x7fadec1570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:52.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadebfff700 1 -- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec0aad20 msgr2=0x7fadec1570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:52.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadebfff700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec0aad20 0x7fadec1570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:52.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadebfff700 1 -- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadec071d40 msgr2=0x7fadec152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:52.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadebfff700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadec071d40 0x7fadec152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:52.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadebfff700 1 -- 172.21.15.7:0/1054082292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade40094a0 con 0x7fadec14b0d0 2023-03-03T12:42:52.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadeaffd700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec0aad20 0x7fadec1570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:42:52.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadebfff700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec14b0d0 0x7fadec1ed600 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fade4014040 tx=0x7fade4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:42:52.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadd3fff700 1 -- 172.21.15.7:0/1054082292 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade40038a0 con 0x7fadec14b0d0 2023-03-03T12:42:52.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadd3fff700 1 -- 172.21.15.7:0/1054082292 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade401e070 con 0x7fadec14b0d0 2023-03-03T12:42:52.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1054082292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadec06f6b0 con 0x7fadec14b0d0 2023-03-03T12:42:52.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadd3fff700 1 -- 172.21.15.7:0/1054082292 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4003c20 con 0x7fadec14b0d0 2023-03-03T12:42:52.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.944+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1054082292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadec155160 con 0x7fadec14b0d0 2023-03-03T12:42:52.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.945+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1054082292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadec06f990 con 0x7fadec14b0d0 2023-03-03T12:42:52.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.945+0000 7fadd3fff700 1 -- 172.21.15.7:0/1054082292 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fade400f050 con 0x7fadec14b0d0 2023-03-03T12:42:52.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.946+0000 7fadd3fff700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadd80682f0 0x7fadd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:42:52.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.946+0000 7fadd3fff700 1 -- 172.21.15.7:0/1054082292 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fade40871c0 con 0x7fadec14b0d0 2023-03-03T12:42:52.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.946+0000 7fadeb7fe700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadd80682f0 0x7fadd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:42:52.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.947+0000 7fadeb7fe700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadd80682f0 0x7fadd806a7a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7faddc009b00 tx=0x7faddc003a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:42:52.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:52.950+0000 7fadd3fff700 1 -- 172.21.15.7:0/1054082292 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fade4053560 con 0x7fadec14b0d0 2023-03-03T12:42:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.189+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1054082292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadec074bb0 con 0x7fadec14b0d0 2023-03-03T12:42:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.190+0000 7fadd3fff700 1 -- 172.21.15.7:0/1054082292 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fade4015090 con 0x7fadec14b0d0 2023-03-03T12:42:53.192 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:42:53.192 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:42:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadd80682f0 msgr2=0x7fadd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadd80682f0 0x7fadd806a7a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7faddc009b00 tx=0x7faddc003a20 comp rx=0 tx=0).stop 2023-03-03T12:42:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1054082292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec14b0d0 msgr2=0x7fadec1ed600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:42:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec14b0d0 0x7fadec1ed600 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fade4014040 tx=0x7fade4004750 comp rx=0 tx=0).stop 2023-03-03T12:42:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1054082292 shutdown_connections 2023-03-03T12:42:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadd80682f0 0x7fadd806a7a0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fadec14b0d0 0x7fadec1ed600 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:53.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fadec071d40 0x7fadec152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:53.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 --2- 172.21.15.7:0/1054082292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fadec0aad20 0x7fadec1570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:42:53.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1054082292 >> 172.21.15.7:0/1054082292 conn(0x7fadec0c8900 msgr2=0x7fadec0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:42:53.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1054082292 shutdown_connections 2023-03-03T12:42:53.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:42:53.192+0000 7fadf1ec7700 1 -- 172.21.15.7:0/1054082292 wait complete. 2023-03-03T12:42:53.195 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:42:53.206 INFO:tasks.mds_thrash.fs.[b]:waiting for 33 secs before thrashing 2023-03-03T12:43:02.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:02.172 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:26.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:43:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.343+0000 7f3bf141b700 1 Processor -- start 2023-03-03T12:43:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.344+0000 7f3bf141b700 1 -- start start 2023-03-03T12:43:26.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.344+0000 7f3bf141b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3bec0caa30 0x7f3bec0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.344+0000 7f3bf141b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec154ff0 0x7f3bec0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.344+0000 7f3bf141b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bec0c7310 con 0x7f3bec0c00e0 2023-03-03T12:43:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.344+0000 7f3bf141b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bec0bfd10 con 0x7f3bec0caa30 2023-03-03T12:43:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.344+0000 7f3bf141b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec0bfa30 con 0x7f3bec154ff0 2023-03-03T12:43:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.345+0000 7f3beb7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec154ff0 0x7f3bec0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.345+0000 7f3beaffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3bec0caa30 0x7f3bec0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:26.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.345+0000 7f3beb7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec154ff0 0x7f3bec0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:49690/0 (socket says 172.21.15.7:49690) 2023-03-03T12:43:26.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.345+0000 7f3beb7fe700 1 -- 172.21.15.7:0/3321064099 learned_addr learned my addr 172.21.15.7:0/3321064099 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:43:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.345+0000 7f3beb7fe700 1 -- 172.21.15.7:0/3321064099 >> v2:172.21.15.143:3300/0 conn(0x7f3bec0caa30 msgr2=0x7f3bec0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:43:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.345+0000 7f3beb7fe700 1 --2- 172.21.15.7:0/3321064099 >> v2:172.21.15.143:3300/0 conn(0x7f3bec0caa30 0x7f3bec0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.345+0000 7f3beb7fe700 1 -- 172.21.15.7:0/3321064099 >> v1:172.21.15.143:6789/0 conn(0x7f3bec0c00e0 legacy=0x7f3bec0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.345+0000 7f3beb7fe700 1 -- 172.21.15.7:0/3321064099 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec154510 con 0x7f3bec154ff0 2023-03-03T12:43:26.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.345+0000 7f3beb7fe700 1 --2- 172.21.15.7:0/3321064099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec154ff0 0x7f3bec0cae00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009e50 tx=0x7f3be400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9addb2ce3697c597 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3be9ffb700 1 -- 172.21.15.7:0/3321064099 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be400e070 con 0x7f3bec154ff0 2023-03-03T12:43:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3be9ffb700 1 -- 172.21.15.7:0/3321064099 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be400b910 con 0x7f3bec154ff0 2023-03-03T12:43:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3be9ffb700 1 -- 172.21.15.7:0/3321064099 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be400bcc0 con 0x7f3bec154ff0 2023-03-03T12:43:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3bf141b700 1 -- 172.21.15.7:0/3321064099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec154ff0 msgr2=0x7f3bec0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3bf141b700 1 --2- 172.21.15.7:0/3321064099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec154ff0 0x7f3bec0cae00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009e50 tx=0x7f3be400b330 comp rx=0 tx=0).stop 2023-03-03T12:43:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3bf141b700 1 -- 172.21.15.7:0/3321064099 shutdown_connections 2023-03-03T12:43:26.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3bf141b700 1 --2- 172.21.15.7:0/3321064099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec154ff0 0x7f3bec0cae00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3bf141b700 1 --2- 172.21.15.7:0/3321064099 >> v2:172.21.15.143:3300/0 conn(0x7f3bec0caa30 0x7f3bec0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3bf141b700 1 -- 172.21.15.7:0/3321064099 >> 172.21.15.7:0/3321064099 conn(0x7f3bec066e40 msgr2=0x7f3bec071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:43:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3bf141b700 1 -- 172.21.15.7:0/3321064099 shutdown_connections 2023-03-03T12:43:26.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.346+0000 7f3bf141b700 1 -- 172.21.15.7:0/3321064099 wait complete. 2023-03-03T12:43:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.348+0000 7f3bf141b700 1 Processor -- start 2023-03-03T12:43:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.348+0000 7f3bf141b700 1 -- start start 2023-03-03T12:43:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.348+0000 7f3bf141b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec0c00e0 0x7f3bec156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.348+0000 7f3bf141b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec0caa30 0x7f3bec1ed230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.348+0000 7f3bf141b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bec154ff0 0x7f3bec157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:26.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.348+0000 7f3bf141b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec156850 con 0x7f3bec154ff0 2023-03-03T12:43:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.348+0000 7f3bf141b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec156670 con 0x7f3bec0caa30 2023-03-03T12:43:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.348+0000 7f3bf141b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec0c1db0 con 0x7f3bec0c00e0 2023-03-03T12:43:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3beb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bec154ff0 0x7f3bec157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3beb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bec154ff0 0x7f3bec157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41626/0 (socket says 172.21.15.7:41626) 2023-03-03T12:43:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3beb7fe700 1 -- 172.21.15.7:0/2718747529 learned_addr learned my addr 172.21.15.7:0/2718747529 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:43:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3beaffd700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec0c00e0 0x7f3bec156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:26.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3bea7fc700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec0caa30 0x7f3bec1ed230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3beb7fe700 1 -- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec0c00e0 msgr2=0x7f3bec156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3beb7fe700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec0c00e0 0x7f3bec156130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3beb7fe700 1 -- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec0caa30 msgr2=0x7f3bec1ed230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3beb7fe700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec0caa30 0x7f3bec1ed230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3beb7fe700 1 -- 172.21.15.7:0/2718747529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be4009b30 con 0x7f3bec154ff0 2023-03-03T12:43:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3beb7fe700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bec154ff0 0x7f3bec157c70 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0bfb70 tx=0x7f3be4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3be37fe700 1 -- 172.21.15.7:0/2718747529 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4004960 con 0x7f3bec154ff0 2023-03-03T12:43:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3be37fe700 1 -- 172.21.15.7:0/2718747529 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be40099b0 con 0x7f3bec154ff0 2023-03-03T12:43:26.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3bf141b700 1 -- 172.21.15.7:0/2718747529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bec154ad0 con 0x7f3bec154ff0 2023-03-03T12:43:26.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3be37fe700 1 -- 172.21.15.7:0/2718747529 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be401c680 con 0x7f3bec154ff0 2023-03-03T12:43:26.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.349+0000 7f3bf141b700 1 -- 172.21.15.7:0/2718747529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bec1547f0 con 0x7f3bec154ff0 2023-03-03T12:43:26.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.350+0000 7f3bf141b700 1 -- 172.21.15.7:0/2718747529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bec0bf650 con 0x7f3bec154ff0 2023-03-03T12:43:26.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.351+0000 7f3be37fe700 1 -- 172.21.15.7:0/2718747529 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3be402c070 con 0x7f3bec154ff0 2023-03-03T12:43:26.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.351+0000 7f3be37fe700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc0682f0 0x7f3bcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:26.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.351+0000 7f3be37fe700 1 -- 172.21.15.7:0/2718747529 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3be4087cb0 con 0x7f3bec154ff0 2023-03-03T12:43:26.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.351+0000 7f3beaffd700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc0682f0 0x7f3bcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:26.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.352+0000 7f3beaffd700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc0682f0 0x7f3bcc06a7a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009b70 tx=0x7f3bdc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:26.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.356+0000 7f3be37fe700 1 -- 172.21.15.7:0/2718747529 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3be4054050 con 0x7f3bec154ff0 2023-03-03T12:43:26.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.598+0000 7f3bf141b700 1 -- 172.21.15.7:0/2718747529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bec0c1f60 con 0x7f3bec154ff0 2023-03-03T12:43:26.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.598+0000 7f3be37fe700 1 -- 172.21.15.7:0/2718747529 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3bec0c1f60 con 0x7f3bec154ff0 2023-03-03T12:43:26.601 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:43:26.601 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:43:26.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.600+0000 7f3bf141b700 1 -- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc0682f0 msgr2=0x7f3bcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.600+0000 7f3bf141b700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc0682f0 0x7f3bcc06a7a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009b70 tx=0x7f3bdc008040 comp rx=0 tx=0).stop 2023-03-03T12:43:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.600+0000 7f3bf141b700 1 -- 172.21.15.7:0/2718747529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bec154ff0 msgr2=0x7f3bec157c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.600+0000 7f3bf141b700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bec154ff0 0x7f3bec157c70 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0bfb70 tx=0x7f3be4004ad0 comp rx=0 tx=0).stop 2023-03-03T12:43:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.601+0000 7f3bf141b700 1 -- 172.21.15.7:0/2718747529 shutdown_connections 2023-03-03T12:43:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.601+0000 7f3bf141b700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc0682f0 0x7f3bcc06a7a0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.601+0000 7f3bf141b700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bec154ff0 0x7f3bec157c70 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:26.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.601+0000 7f3bf141b700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bec0c00e0 0x7f3bec156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.601+0000 7f3bf141b700 1 --2- 172.21.15.7:0/2718747529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bec0caa30 0x7f3bec1ed230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.601+0000 7f3bf141b700 1 -- 172.21.15.7:0/2718747529 >> 172.21.15.7:0/2718747529 conn(0x7f3bec066e40 msgr2=0x7f3bec150460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:43:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.601+0000 7f3bf141b700 1 -- 172.21.15.7:0/2718747529 shutdown_connections 2023-03-03T12:43:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:26.601+0000 7f3bf141b700 1 -- 172.21.15.7:0/2718747529 wait complete. 2023-03-03T12:43:26.604 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:43:26.614 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before thrashing 2023-03-03T12:43:31.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:43:31.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.753+0000 7f013ae99700 1 Processor -- start 2023-03-03T12:43:31.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.754+0000 7f013ae99700 1 -- start start 2023-03-03T12:43:31.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.754+0000 7f013ae99700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01340a5d20 0x7f0134152e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:31.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.754+0000 7f013ae99700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01340c25e0 0x7f01340c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:31.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.754+0000 7f013ae99700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01340ba3d0 con 0x7f013414fbc0 2023-03-03T12:43:31.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.754+0000 7f013ae99700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01340c2f80 con 0x7f01340a5d20 2023-03-03T12:43:31.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.754+0000 7f013ae99700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01340c1540 con 0x7f01340c25e0 2023-03-03T12:43:31.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.754+0000 7f0133fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01340a5d20 0x7f0134152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:31.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.754+0000 7f0138c35700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f013414fbc0 0x7f01340a85a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35616/0 (socket says 172.21.15.7:35616) 2023-03-03T12:43:31.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.754+0000 7f0138c35700 1 -- 172.21.15.7:0/176042620 learned_addr learned my addr 172.21.15.7:0/176042620 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:43:31.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f0139436700 1 --2- 172.21.15.7:0/176042620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01340c25e0 0x7f01340c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:31.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f01337fe700 1 -- 172.21.15.7:0/176042620 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595585935 0 0) 0x7f01340ba3d0 con 0x7f013414fbc0 2023-03-03T12:43:31.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f01337fe700 1 -- 172.21.15.7:0/176042620 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01240035b0 con 0x7f013414fbc0 2023-03-03T12:43:31.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f0133fff700 1 -- 172.21.15.7:0/176042620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01340c25e0 msgr2=0x7f01340c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:43:31.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f0133fff700 1 --2- 172.21.15.7:0/176042620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01340c25e0 0x7f01340c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:31.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f0133fff700 1 -- 172.21.15.7:0/176042620 >> v1:172.21.15.7:6789/0 conn(0x7f013414fbc0 legacy=0x7f01340a85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:31.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f0133fff700 1 -- 172.21.15.7:0/176042620 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01340c1820 con 0x7f01340a5d20 2023-03-03T12:43:31.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f0133fff700 1 --2- 172.21.15.7:0/176042620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01340a5d20 0x7f0134152e10 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f0120009840 tx=0x7f0120009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbff3483efaca93 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:31.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f01337fe700 1 -- 172.21.15.7:0/176042620 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120011830 con 0x7f01340a5d20 2023-03-03T12:43:31.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f01337fe700 1 -- 172.21.15.7:0/176042620 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0120011990 con 0x7f01340a5d20 2023-03-03T12:43:31.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.755+0000 7f01337fe700 1 -- 172.21.15.7:0/176042620 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120011d20 con 0x7f01340a5d20 2023-03-03T12:43:31.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.756+0000 7f013ae99700 1 -- 172.21.15.7:0/176042620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01340a5d20 msgr2=0x7f0134152e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:31.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.756+0000 7f013ae99700 1 --2- 172.21.15.7:0/176042620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01340a5d20 0x7f0134152e10 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f0120009840 tx=0x7f0120009b50 comp rx=0 tx=0).stop 2023-03-03T12:43:31.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.756+0000 7f013ae99700 1 -- 172.21.15.7:0/176042620 shutdown_connections 2023-03-03T12:43:31.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.756+0000 7f013ae99700 1 --2- 172.21.15.7:0/176042620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01340c25e0 0x7f01340c2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:31.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.756+0000 7f013ae99700 1 --2- 172.21.15.7:0/176042620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01340a5d20 0x7f0134152e10 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:31.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.756+0000 7f013ae99700 1 -- 172.21.15.7:0/176042620 >> 172.21.15.7:0/176042620 conn(0x7f013406af40 msgr2=0x7f01340c6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:43:31.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.756+0000 7f013ae99700 1 -- 172.21.15.7:0/176042620 shutdown_connections 2023-03-03T12:43:31.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.756+0000 7f013ae99700 1 -- 172.21.15.7:0/176042620 wait complete. 2023-03-03T12:43:31.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.757+0000 7f013ae99700 1 Processor -- start 2023-03-03T12:43:31.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.757+0000 7f013ae99700 1 -- start start 2023-03-03T12:43:31.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f013ae99700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01340a5d20 0x7f01340c0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:31.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f013ae99700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01340c25e0 0x7f01340c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:31.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f013ae99700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f013414fbc0 0x7f01341eade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:31.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f013ae99700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341ebab0 con 0x7f013414fbc0 2023-03-03T12:43:31.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f013ae99700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341eb7d0 con 0x7f01340a5d20 2023-03-03T12:43:31.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f013ae99700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01340bfe40 con 0x7f01340c25e0 2023-03-03T12:43:31.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0139436700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f013414fbc0 0x7f01341eade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:31.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0139436700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f013414fbc0 0x7f01341eade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41644/0 (socket says 172.21.15.7:41644) 2023-03-03T12:43:31.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0139436700 1 -- 172.21.15.7:0/3875585003 learned_addr learned my addr 172.21.15.7:0/3875585003 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:43:31.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0133fff700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01340c25e0 0x7f01340c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0138c35700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01340a5d20 0x7f01340c0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0139436700 1 -- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01340c25e0 msgr2=0x7f01340c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0139436700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01340c25e0 0x7f01340c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0139436700 1 -- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01340a5d20 msgr2=0x7f01340c0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0139436700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01340a5d20 0x7f01340c0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0139436700 1 -- 172.21.15.7:0/3875585003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01200094a0 con 0x7f013414fbc0 2023-03-03T12:43:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0138c35700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01340a5d20 0x7f01340c0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:43:31.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.758+0000 7f0139436700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f013414fbc0 0x7f01341eade0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f012c007e40 tx=0x7f012c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.759+0000 7f01317fa700 1 -- 172.21.15.7:0/3875585003 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c00eea0 con 0x7f013414fbc0 2023-03-03T12:43:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.759+0000 7f01317fa700 1 -- 172.21.15.7:0/3875585003 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012c014420 con 0x7f013414fbc0 2023-03-03T12:43:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.759+0000 7f01317fa700 1 -- 172.21.15.7:0/3875585003 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c0147b0 con 0x7f013414fbc0 2023-03-03T12:43:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.759+0000 7f013ae99700 1 -- 172.21.15.7:0/3875585003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01340c1de0 con 0x7f013414fbc0 2023-03-03T12:43:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.759+0000 7f013ae99700 1 -- 172.21.15.7:0/3875585003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01340c1b00 con 0x7f013414fbc0 2023-03-03T12:43:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.760+0000 7f013ae99700 1 -- 172.21.15.7:0/3875585003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01340c20c0 con 0x7f013414fbc0 2023-03-03T12:43:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.761+0000 7f01317fa700 1 -- 172.21.15.7:0/3875585003 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f012c012050 con 0x7f013414fbc0 2023-03-03T12:43:31.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.761+0000 7f01317fa700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01180682f0 0x7f011806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:31.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.761+0000 7f01317fa700 1 -- 172.21.15.7:0/3875585003 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f012c087680 con 0x7f013414fbc0 2023-03-03T12:43:31.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.761+0000 7f0138c35700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01180682f0 0x7f011806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:31.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.762+0000 7f0138c35700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01180682f0 0x7f011806a7a0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f01280010e0 tx=0x7f0128009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:31.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:31.765+0000 7f01317fa700 1 -- 172.21.15.7:0/3875585003 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f012c053ac0 con 0x7f013414fbc0 2023-03-03T12:43:32.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.002+0000 7f013ae99700 1 -- 172.21.15.7:0/3875585003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0134074bb0 con 0x7f013414fbc0 2023-03-03T12:43:32.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.002+0000 7f01317fa700 1 -- 172.21.15.7:0/3875585003 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f012c0570e0 con 0x7f013414fbc0 2023-03-03T12:43:32.006 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:43:32.006 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:43:32.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.004+0000 7f013ae99700 1 -- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01180682f0 msgr2=0x7f011806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.004+0000 7f013ae99700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01180682f0 0x7f011806a7a0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f01280010e0 tx=0x7f0128009040 comp rx=0 tx=0).stop 2023-03-03T12:43:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.004+0000 7f013ae99700 1 -- 172.21.15.7:0/3875585003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f013414fbc0 msgr2=0x7f01341eade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.004+0000 7f013ae99700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f013414fbc0 0x7f01341eade0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f012c007e40 tx=0x7f012c00e3c0 comp rx=0 tx=0).stop 2023-03-03T12:43:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.004+0000 7f013ae99700 1 -- 172.21.15.7:0/3875585003 shutdown_connections 2023-03-03T12:43:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.004+0000 7f013ae99700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01180682f0 0x7f011806a7a0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.004+0000 7f013ae99700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f013414fbc0 0x7f01341eade0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.004+0000 7f013ae99700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01340a5d20 0x7f01340c0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.004+0000 7f013ae99700 1 --2- 172.21.15.7:0/3875585003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01340c25e0 0x7f01340c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:32.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.004+0000 7f013ae99700 1 -- 172.21.15.7:0/3875585003 >> 172.21.15.7:0/3875585003 conn(0x7f013406af40 msgr2=0x7f0134150f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:43:32.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.005+0000 7f013ae99700 1 -- 172.21.15.7:0/3875585003 shutdown_connections 2023-03-03T12:43:32.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:32.005+0000 7f013ae99700 1 -- 172.21.15.7:0/3875585003 wait complete. 2023-03-03T12:43:32.009 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:43:32.019 INFO:tasks.mds_thrash.fs.[b]:waiting for 48 secs before thrashing 2023-03-03T12:43:32.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:32.293 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:43:43.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:43:44.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.050+0000 7fd876319700 1 Processor -- start 2023-03-03T12:43:44.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.050+0000 7fd876319700 1 -- start start 2023-03-03T12:43:44.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd876319700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd8700c1ac0 0x7fd8700c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:44.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd876319700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8700aa150 con 0x7fd8700aabc0 2023-03-03T12:43:44.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd876319700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87006f5b0 con 0x7fd87014ae20 2023-03-03T12:43:44.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd876319700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87006f890 con 0x7fd8700c1ac0 2023-03-03T12:43:44.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd86f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd8700aabc0 0x7fd870071840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35638/0 (socket says 172.21.15.7:35638) 2023-03-03T12:43:44.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd86f7fe700 1 -- 172.21.15.7:0/30658192 learned_addr learned my addr 172.21.15.7:0/30658192 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:43:44.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544234623 0 0) 0x7fd8700aa150 con 0x7fd8700aabc0 2023-03-03T12:43:44.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd85c0035b0 con 0x7fd8700aabc0 2023-03-03T12:43:44.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd8748b6700 1 --2- 172.21.15.7:0/30658192 >> v2:172.21.15.143:3300/0 conn(0x7fd8700c1ac0 0x7fd8700c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:44.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2796941640 0 0) 0x7fd85c0035b0 con 0x7fd8700aabc0 2023-03-03T12:43:44.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 >> v2:172.21.15.143:3300/0 conn(0x7fd8700c1ac0 msgr2=0x7fd8700c1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:43:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd86effd700 1 --2- 172.21.15.7:0/30658192 >> v2:172.21.15.143:3300/0 conn(0x7fd8700c1ac0 0x7fd8700c1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 >> v1:172.21.15.143:6789/0 conn(0x7fd87014ae20 legacy=0x7fd870154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87006fb70 con 0x7fd8700aabc0 2023-03-03T12:43:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.051+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd858001670 con 0x7fd8700aabc0 2023-03-03T12:43:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 >> v1:172.21.15.7:6789/0 conn(0x7fd8700aabc0 legacy=0x7fd870071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86effd700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd85c003a70 0x7fd85c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86effd700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd85c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86effd700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd85c005200 0x7fd85c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:44.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87006f890 con 0x7fd85c004850 2023-03-03T12:43:44.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c00c0e0 con 0x7fd85c005200 2023-03-03T12:43:44.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c00c2a0 con 0x7fd85c003a70 2023-03-03T12:43:44.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86f7fe700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd85c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:44.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd8748b6700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd85c005200 0x7fd85c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:44.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86ffff700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd85c003a70 0x7fd85c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:44.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86f7fe700 1 -- 172.21.15.7:0/30658192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd85c003a70 msgr2=0x7fd85c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:43:44.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86f7fe700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd85c003a70 0x7fd85c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86f7fe700 1 -- 172.21.15.7:0/30658192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd85c005200 msgr2=0x7fd85c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:43:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86f7fe700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd85c005200 0x7fd85c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd8748b6700 1 -- 172.21.15.7:0/30658192 reap_dead start 2023-03-03T12:43:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86f7fe700 1 -- 172.21.15.7:0/30658192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c0035b0 con 0x7fd85c004850 2023-03-03T12:43:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86f7fe700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd85c004cc0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fd85800c240 tx=0x7fd85c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd031a6b796e18c4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858007c10 con 0x7fd85c004850 2023-03-03T12:43:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85800cd60 con 0x7fd85c004850 2023-03-03T12:43:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.052+0000 7fd86effd700 1 -- 172.21.15.7:0/30658192 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858005ba0 con 0x7fd85c004850 2023-03-03T12:43:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.053+0000 7fd876319700 1 -- 172.21.15.7:0/30658192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 msgr2=0x7fd85c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.053+0000 7fd876319700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd85c004cc0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fd85800c240 tx=0x7fd85c005880 comp rx=0 tx=0).stop 2023-03-03T12:43:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.053+0000 7fd876319700 1 -- 172.21.15.7:0/30658192 shutdown_connections 2023-03-03T12:43:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.053+0000 7fd876319700 1 --2- 172.21.15.7:0/30658192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd85c004cc0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.053+0000 7fd876319700 1 -- 172.21.15.7:0/30658192 >> 172.21.15.7:0/30658192 conn(0x7fd8700c8900 msgr2=0x7fd8700c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:43:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.053+0000 7fd876319700 1 -- 172.21.15.7:0/30658192 shutdown_connections 2023-03-03T12:43:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.053+0000 7fd876319700 1 -- 172.21.15.7:0/30658192 wait complete. 2023-03-03T12:43:44.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.054+0000 7fd876319700 1 Processor -- start 2023-03-03T12:43:44.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.054+0000 7fd876319700 1 -- start start 2023-03-03T12:43:44.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd876319700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd87014d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:44.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd876319700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd870156d30 0x7fd8701ed5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:44.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd876319700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd87006c4a0 0x7fd8701edb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:44.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd876319700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87006b760 con 0x7fd85c004850 2023-03-03T12:43:44.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd876319700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701460d0 con 0x7fd87006c4a0 2023-03-03T12:43:44.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd876319700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8701ed150 con 0x7fd870156d30 2023-03-03T12:43:44.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd86ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd87014d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:44.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd86ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd87014d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41668/0 (socket says 172.21.15.7:41668) 2023-03-03T12:43:44.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd86ffff700 1 -- 172.21.15.7:0/3291805851 learned_addr learned my addr 172.21.15.7:0/3291805851 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:43:44.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd86ffff700 1 -- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd870156d30 msgr2=0x7fd8701ed5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:44.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd86f7fe700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd870156d30 0x7fd8701ed5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:44.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.055+0000 7fd8748b6700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd87006c4a0 0x7fd8701edb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:44.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd86ffff700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd870156d30 0x7fd8701ed5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:44.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd86ffff700 1 -- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd87006c4a0 msgr2=0x7fd8701edb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:44.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd86ffff700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd87006c4a0 0x7fd8701edb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:44.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd86ffff700 1 -- 172.21.15.7:0/3291805851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87014b1f0 con 0x7fd85c004850 2023-03-03T12:43:44.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd8748b6700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd87006c4a0 0x7fd8701edb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:43:44.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd86ffff700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd87014d730 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fd86000f850 tx=0x7fd86000fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:44.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd86cff9700 1 -- 172.21.15.7:0/3291805851 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8600050d0 con 0x7fd85c004850 2023-03-03T12:43:44.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd86cff9700 1 -- 172.21.15.7:0/3291805851 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd860005230 con 0x7fd85c004850 2023-03-03T12:43:44.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd86cff9700 1 -- 172.21.15.7:0/3291805851 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860003680 con 0x7fd85c004850 2023-03-03T12:43:44.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd876319700 1 -- 172.21.15.7:0/3291805851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8700aa430 con 0x7fd85c004850 2023-03-03T12:43:44.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.056+0000 7fd876319700 1 -- 172.21.15.7:0/3291805851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd870154e70 con 0x7fd85c004850 2023-03-03T12:43:44.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.057+0000 7fd876319700 1 -- 172.21.15.7:0/3291805851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd870074bb0 con 0x7fd85c004850 2023-03-03T12:43:44.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.057+0000 7fd86cff9700 1 -- 172.21.15.7:0/3291805851 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd8600037e0 con 0x7fd85c004850 2023-03-03T12:43:44.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.058+0000 7fd86cff9700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8500682f0 0x7fd85006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:43:44.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.058+0000 7fd86cff9700 1 -- 172.21.15.7:0/3291805851 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd860086140 con 0x7fd85c004850 2023-03-03T12:43:44.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.058+0000 7fd86f7fe700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8500682f0 0x7fd85006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:43:44.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.059+0000 7fd86f7fe700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8500682f0 0x7fd85006a7a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fd858015040 tx=0x7fd858007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:43:44.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.062+0000 7fd86cff9700 1 -- 172.21.15.7:0/3291805851 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd8600523f0 con 0x7fd85c004850 2023-03-03T12:43:44.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.302+0000 7fd876319700 1 -- 172.21.15.7:0/3291805851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8700a9db0 con 0x7fd85c004850 2023-03-03T12:43:44.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.302+0000 7fd86cff9700 1 -- 172.21.15.7:0/3291805851 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd860055a10 con 0x7fd85c004850 2023-03-03T12:43:44.306 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:43:44.306 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:43:44.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.304+0000 7fd876319700 1 -- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8500682f0 msgr2=0x7fd85006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:44.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.304+0000 7fd876319700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8500682f0 0x7fd85006a7a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fd858015040 tx=0x7fd858007000 comp rx=0 tx=0).stop 2023-03-03T12:43:44.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.304+0000 7fd876319700 1 -- 172.21.15.7:0/3291805851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 msgr2=0x7fd87014d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:43:44.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.304+0000 7fd876319700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd87014d730 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fd86000f850 tx=0x7fd86000fc10 comp rx=0 tx=0).stop 2023-03-03T12:43:44.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.304+0000 7fd876319700 1 -- 172.21.15.7:0/3291805851 shutdown_connections 2023-03-03T12:43:44.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.304+0000 7fd876319700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd87006c4a0 0x7fd8701edb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:44.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.304+0000 7fd876319700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8500682f0 0x7fd85006a7a0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:44.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.304+0000 7fd876319700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd85c004850 0x7fd87014d730 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:44.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.304+0000 7fd876319700 1 --2- 172.21.15.7:0/3291805851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd870156d30 0x7fd8701ed5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:43:44.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.304+0000 7fd876319700 1 -- 172.21.15.7:0/3291805851 >> 172.21.15.7:0/3291805851 conn(0x7fd8700c8900 msgr2=0x7fd8700acd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:43:44.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.305+0000 7fd876319700 1 -- 172.21.15.7:0/3291805851 shutdown_connections 2023-03-03T12:43:44.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:43:44.305+0000 7fd876319700 1 -- 172.21.15.7:0/3291805851 wait complete. 2023-03-03T12:43:44.309 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:43:44.319 INFO:tasks.mds_thrash.fs.[a]:waiting for 116 secs before thrashing 2023-03-03T12:44:02.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:02.407 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:20.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:44:20.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.157+0000 7f3512149700 1 Processor -- start 2023-03-03T12:44:20.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.157+0000 7f3512149700 1 -- start start 2023-03-03T12:44:20.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f3512149700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f350c14f9d0 0x7f350c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:20.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f3512149700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f350c0abe20 0x7f350c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:20.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f3512149700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c0c2f30 con 0x7f350c14fda0 2023-03-03T12:44:20.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f3512149700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c0c31d0 con 0x7f350c0abe20 2023-03-03T12:44:20.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f3512149700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c1524a0 con 0x7f350c14f9d0 2023-03-03T12:44:20.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f350bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f350c0abe20 0x7f350c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:20.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f350affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f350c14fda0 0x7f350c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35678/0 (socket says 172.21.15.7:35678) 2023-03-03T12:44:20.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f350affd700 1 -- 172.21.15.7:0/1611083289 learned_addr learned my addr 172.21.15.7:0/1611083289 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:44:20.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f350b7fe700 1 --2- 172.21.15.7:0/1611083289 >> v2:172.21.15.143:3300/0 conn(0x7f350c14f9d0 0x7f350c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:20.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f350bfff700 1 -- 172.21.15.7:0/1611083289 >> v2:172.21.15.143:3300/0 conn(0x7f350c14f9d0 msgr2=0x7f350c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:44:20.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.158+0000 7f350bfff700 1 --2- 172.21.15.7:0/1611083289 >> v2:172.21.15.143:3300/0 conn(0x7f350c14f9d0 0x7f350c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:20.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f350bfff700 1 -- 172.21.15.7:0/1611083289 >> v1:172.21.15.7:6789/0 conn(0x7f350c14fda0 legacy=0x7f350c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:20.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f350a7fc700 1 -- 172.21.15.7:0/1611083289 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532296572 0 0) 0x7f350c0c2f30 con 0x7f350c14fda0 2023-03-03T12:44:20.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f350bfff700 1 -- 172.21.15.7:0/1611083289 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c0c34b0 con 0x7f350c0abe20 2023-03-03T12:44:20.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f350b7fe700 1 --2- 172.21.15.7:0/1611083289 >> v2:172.21.15.143:3300/0 conn(0x7f350c14f9d0 0x7f350c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:44:20.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f350bfff700 1 --2- 172.21.15.7:0/1611083289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c0abe20 0x7f350c1529c0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f350c0710a0 tx=0x7f350400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a6c548c12ac4a9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:44:20.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f350a7fc700 1 -- 172.21.15.7:0/1611083289 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350400e070 con 0x7f350c0abe20 2023-03-03T12:44:20.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f350a7fc700 1 -- 172.21.15.7:0/1611083289 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350400bbd0 con 0x7f350c0abe20 2023-03-03T12:44:20.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f350a7fc700 1 -- 172.21.15.7:0/1611083289 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3504012430 con 0x7f350c0abe20 2023-03-03T12:44:20.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f3512149700 1 -- 172.21.15.7:0/1611083289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c0abe20 msgr2=0x7f350c1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:20.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f3512149700 1 --2- 172.21.15.7:0/1611083289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c0abe20 0x7f350c1529c0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f350c0710a0 tx=0x7f350400b330 comp rx=0 tx=0).stop 2023-03-03T12:44:20.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f3512149700 1 -- 172.21.15.7:0/1611083289 shutdown_connections 2023-03-03T12:44:20.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f3512149700 1 --2- 172.21.15.7:0/1611083289 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c0abe20 0x7f350c1529c0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:20.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f3512149700 1 --2- 172.21.15.7:0/1611083289 >> v2:172.21.15.143:3300/0 conn(0x7f350c14f9d0 0x7f350c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:20.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f3512149700 1 -- 172.21.15.7:0/1611083289 >> 172.21.15.7:0/1611083289 conn(0x7f350c0c8900 msgr2=0x7f350c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:44:20.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f3512149700 1 -- 172.21.15.7:0/1611083289 shutdown_connections 2023-03-03T12:44:20.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.159+0000 7f3512149700 1 -- 172.21.15.7:0/1611083289 wait complete. 2023-03-03T12:44:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.161+0000 7f3512149700 1 Processor -- start 2023-03-03T12:44:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.161+0000 7f3512149700 1 -- start start 2023-03-03T12:44:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.161+0000 7f3512149700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f350c0abe20 0x7f350c153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.161+0000 7f3512149700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c14f9d0 0x7f350c0c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:20.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f3512149700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f350c14fda0 0x7f350c0c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f3512149700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c14ad40 con 0x7f350c14f9d0 2023-03-03T12:44:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f3512149700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c0c6820 con 0x7f350c0abe20 2023-03-03T12:44:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f3512149700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350c1ea400 con 0x7f350c14fda0 2023-03-03T12:44:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c14f9d0 0x7f350c0c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c14f9d0 0x7f350c0c79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41706/0 (socket says 172.21.15.7:41706) 2023-03-03T12:44:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350affd700 1 -- 172.21.15.7:0/1423137666 learned_addr learned my addr 172.21.15.7:0/1423137666 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:44:20.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350b7fe700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f350c0abe20 0x7f350c153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350bfff700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f350c14fda0 0x7f350c0c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350affd700 1 -- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f350c14fda0 msgr2=0x7f350c0c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350affd700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f350c14fda0 0x7f350c0c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350affd700 1 -- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f350c0abe20 msgr2=0x7f350c153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350affd700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f350c0abe20 0x7f350c153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350affd700 1 -- 172.21.15.7:0/1423137666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504009a70 con 0x7f350c14f9d0 2023-03-03T12:44:20.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350bfff700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f350c14fda0 0x7f350c0c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:44:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f350affd700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c14f9d0 0x7f350c0c79d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f34f400c540 tx=0x7f34f400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:44:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f3503fff700 1 -- 172.21.15.7:0/1423137666 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f4004c80 con 0x7f350c14f9d0 2023-03-03T12:44:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f3503fff700 1 -- 172.21.15.7:0/1423137666 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f4004de0 con 0x7f350c14f9d0 2023-03-03T12:44:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f3512149700 1 -- 172.21.15.7:0/1423137666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c0ac350 con 0x7f350c14f9d0 2023-03-03T12:44:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f3503fff700 1 -- 172.21.15.7:0/1423137666 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f4007310 con 0x7f350c14f9d0 2023-03-03T12:44:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.162+0000 7f3512149700 1 -- 172.21.15.7:0/1423137666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c0ab630 con 0x7f350c14f9d0 2023-03-03T12:44:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.163+0000 7f3512149700 1 -- 172.21.15.7:0/1423137666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350c071fa0 con 0x7f350c14f9d0 2023-03-03T12:44:20.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.164+0000 7f3503fff700 1 -- 172.21.15.7:0/1423137666 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f34f401a020 con 0x7f350c14f9d0 2023-03-03T12:44:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.164+0000 7f3503fff700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34ec0682f0 0x7f34ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:44:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.164+0000 7f3503fff700 1 -- 172.21.15.7:0/1423137666 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f34f4085e80 con 0x7f350c14f9d0 2023-03-03T12:44:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.164+0000 7f350b7fe700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34ec0682f0 0x7f34ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:44:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.165+0000 7f350b7fe700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34ec0682f0 0x7f34ec06a7a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f34fc006fd0 tx=0x7f34fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:44:20.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.169+0000 7f3503fff700 1 -- 172.21.15.7:0/1423137666 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f34f40521e0 con 0x7f350c14f9d0 2023-03-03T12:44:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.408+0000 7f3512149700 1 -- 172.21.15.7:0/1423137666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f350c0c6960 con 0x7f350c14f9d0 2023-03-03T12:44:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.408+0000 7f3503fff700 1 -- 172.21.15.7:0/1423137666 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f350c0c6960 con 0x7f350c14f9d0 2023-03-03T12:44:20.410 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:44:20.410 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:44:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 -- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34ec0682f0 msgr2=0x7f34ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34ec0682f0 0x7f34ec06a7a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f34fc006fd0 tx=0x7f34fc008040 comp rx=0 tx=0).stop 2023-03-03T12:44:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 -- 172.21.15.7:0/1423137666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c14f9d0 msgr2=0x7f350c0c79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:44:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c14f9d0 0x7f350c0c79d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f34f400c540 tx=0x7f34f400c900 comp rx=0 tx=0).stop 2023-03-03T12:44:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 -- 172.21.15.7:0/1423137666 shutdown_connections 2023-03-03T12:44:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f350c14fda0 0x7f350c0c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f350c0abe20 0x7f350c153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34ec0682f0 0x7f34ec06a7a0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:20.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 --2- 172.21.15.7:0/1423137666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f350c14f9d0 0x7f350c0c79d0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:44:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 -- 172.21.15.7:0/1423137666 >> 172.21.15.7:0/1423137666 conn(0x7f350c0c8900 msgr2=0x7f350c0c8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:44:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 -- 172.21.15.7:0/1423137666 shutdown_connections 2023-03-03T12:44:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:44:20.410+0000 7f3512149700 1 -- 172.21.15.7:0/1423137666 wait complete. 2023-03-03T12:44:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:44:20.424 INFO:tasks.mds_thrash.fs.[b]:waiting for 42 secs before thrashing 2023-03-03T12:44:32.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:44:32.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:02.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:45:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.563+0000 7f1c4a88a700 1 Processor -- start 2023-03-03T12:45:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.563+0000 7f1c4a88a700 1 -- start start 2023-03-03T12:45:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.563+0000 7f1c4a88a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1c440caa30 0x7f1c440c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.563+0000 7f1c4a88a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1c440c00e0 0x7f1c44154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.563+0000 7f1c4a88a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440c7310 con 0x7f1c440cae00 2023-03-03T12:45:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.563+0000 7f1c4a88a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c440cb240 con 0x7f1c440c00e0 2023-03-03T12:45:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.563+0000 7f1c4a88a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c440bfd10 con 0x7f1c440caa30 2023-03-03T12:45:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c437fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1c440c00e0 0x7f1c44154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c437fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1c440c00e0 0x7f1c44154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41732/0 (socket says 172.21.15.7:41732) 2023-03-03T12:45:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c437fe700 1 -- 172.21.15.7:0/3218058413 learned_addr learned my addr 172.21.15.7:0/3218058413 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:45:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c43fff700 1 --2- 172.21.15.7:0/3218058413 >> v2:172.21.15.143:3300/0 conn(0x7f1c440caa30 0x7f1c440c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c437fe700 1 -- 172.21.15.7:0/3218058413 >> v2:172.21.15.143:3300/0 conn(0x7f1c440caa30 msgr2=0x7f1c440c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:45:02.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c437fe700 1 --2- 172.21.15.7:0/3218058413 >> v2:172.21.15.143:3300/0 conn(0x7f1c440caa30 0x7f1c440c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:02.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c437fe700 1 -- 172.21.15.7:0/3218058413 >> v1:172.21.15.143:6789/0 conn(0x7f1c440cae00 legacy=0x7f1c440c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:45:02.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c437fe700 1 -- 172.21.15.7:0/3218058413 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c440bfa30 con 0x7f1c440c00e0 2023-03-03T12:45:02.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c437fe700 1 --2- 172.21.15.7:0/3218058413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 0x7f1c44154ff0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f1c440c81b0 tx=0x7f1c38009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c95fae4c2bfbef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:45:02.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c42ffd700 1 -- 172.21.15.7:0/3218058413 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c38005e30 con 0x7f1c440c00e0 2023-03-03T12:45:02.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c42ffd700 1 -- 172.21.15.7:0/3218058413 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c38006030 con 0x7f1c440c00e0 2023-03-03T12:45:02.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.564+0000 7f1c42ffd700 1 -- 172.21.15.7:0/3218058413 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c380063e0 con 0x7f1c440c00e0 2023-03-03T12:45:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.565+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3218058413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 msgr2=0x7f1c44154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.565+0000 7f1c4a88a700 1 --2- 172.21.15.7:0/3218058413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 0x7f1c44154ff0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f1c440c81b0 tx=0x7f1c38009bd0 comp rx=0 tx=0).stop 2023-03-03T12:45:02.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.565+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3218058413 shutdown_connections 2023-03-03T12:45:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.565+0000 7f1c4a88a700 1 --2- 172.21.15.7:0/3218058413 >> v2:172.21.15.143:3300/0 conn(0x7f1c440caa30 0x7f1c440c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.565+0000 7f1c4a88a700 1 --2- 172.21.15.7:0/3218058413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 0x7f1c44154ff0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.565+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3218058413 >> 172.21.15.7:0/3218058413 conn(0x7f1c44066e40 msgr2=0x7f1c44071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:45:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.565+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3218058413 shutdown_connections 2023-03-03T12:45:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.565+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3218058413 wait complete. 2023-03-03T12:45:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.566+0000 7f1c4a88a700 1 Processor -- start 2023-03-03T12:45:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.566+0000 7f1c4a88a700 1 -- start start 2023-03-03T12:45:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c4a88a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 0x7f1c441edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c4a88a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c440caa30 0x7f1c440c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c4a88a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c440cae00 0x7f1c4414c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c4a88a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c44155950 con 0x7f1c440c00e0 2023-03-03T12:45:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c4a88a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c44155c10 con 0x7f1c440caa30 2023-03-03T12:45:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c4a88a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c44155ef0 con 0x7f1c440cae00 2023-03-03T12:45:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c43fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 0x7f1c441edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c43fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 0x7f1c441edb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41736/0 (socket says 172.21.15.7:41736) 2023-03-03T12:45:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c43fff700 1 -- 172.21.15.7:0/3095058840 learned_addr learned my addr 172.21.15.7:0/3095058840 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:45:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c437fe700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c440caa30 0x7f1c440c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.567+0000 7f1c48e27700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c440cae00 0x7f1c4414c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c43fff700 1 -- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c440cae00 msgr2=0x7f1c4414c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c43fff700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c440cae00 0x7f1c4414c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c43fff700 1 -- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c440caa30 msgr2=0x7f1c440c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:02.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c43fff700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c440caa30 0x7f1c440c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c43fff700 1 -- 172.21.15.7:0/3095058840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c38009500 con 0x7f1c440c00e0 2023-03-03T12:45:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c48e27700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c440cae00 0x7f1c4414c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:45:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c43fff700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 0x7f1c441edb40 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f1c3400e810 tx=0x7f1c3400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:45:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c40ff9700 1 -- 172.21.15.7:0/3095058840 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c34013610 con 0x7f1c440c00e0 2023-03-03T12:45:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c40ff9700 1 -- 172.21.15.7:0/3095058840 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c34013770 con 0x7f1c440c00e0 2023-03-03T12:45:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c40ff9700 1 -- 172.21.15.7:0/3095058840 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c34013b20 con 0x7f1c440c00e0 2023-03-03T12:45:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3095058840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c441547f0 con 0x7f1c440c00e0 2023-03-03T12:45:02.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.568+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3095058840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c44154510 con 0x7f1c440c00e0 2023-03-03T12:45:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.569+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3095058840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c44154ad0 con 0x7f1c440c00e0 2023-03-03T12:45:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.569+0000 7f1c40ff9700 1 -- 172.21.15.7:0/3095058840 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1c34023070 con 0x7f1c440c00e0 2023-03-03T12:45:02.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.570+0000 7f1c40ff9700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c300682f0 0x7f1c3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:02.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.570+0000 7f1c40ff9700 1 -- 172.21.15.7:0/3095058840 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1c34085de0 con 0x7f1c440c00e0 2023-03-03T12:45:02.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.570+0000 7f1c437fe700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c300682f0 0x7f1c3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:02.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.571+0000 7f1c437fe700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c300682f0 0x7f1c3006a7a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1c440c81b0 tx=0x7f1c38008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:45:02.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.574+0000 7f1c40ff9700 1 -- 172.21.15.7:0/3095058840 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1c34052100 con 0x7f1c440c00e0 2023-03-03T12:45:02.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:02.674 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:02.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.813+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3095058840 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c44074bb0 con 0x7f1c440c00e0 2023-03-03T12:45:02.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.813+0000 7f1c40ff9700 1 -- 172.21.15.7:0/3095058840 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1c34055720 con 0x7f1c440c00e0 2023-03-03T12:45:02.815 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:45:02.816 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:45:02.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.815+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c300682f0 msgr2=0x7f1c3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.815+0000 7f1c4a88a700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c300682f0 0x7f1c3006a7a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1c440c81b0 tx=0x7f1c38008070 comp rx=0 tx=0).stop 2023-03-03T12:45:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.815+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3095058840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 msgr2=0x7f1c441edb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.815+0000 7f1c4a88a700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 0x7f1c441edb40 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f1c3400e810 tx=0x7f1c3400eb20 comp rx=0 tx=0).stop 2023-03-03T12:45:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.815+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3095058840 shutdown_connections 2023-03-03T12:45:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.815+0000 7f1c4a88a700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c440cae00 0x7f1c4414c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.816+0000 7f1c4a88a700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c300682f0 0x7f1c3006a7a0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:02.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.816+0000 7f1c4a88a700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c440c00e0 0x7f1c441edb40 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:02.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.816+0000 7f1c4a88a700 1 --2- 172.21.15.7:0/3095058840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c440caa30 0x7f1c440c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:02.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.816+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3095058840 >> 172.21.15.7:0/3095058840 conn(0x7f1c44066e40 msgr2=0x7f1c44153c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:45:02.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.816+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3095058840 shutdown_connections 2023-03-03T12:45:02.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:02.816+0000 7f1c4a88a700 1 -- 172.21.15.7:0/3095058840 wait complete. 2023-03-03T12:45:02.819 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:45:02.829 INFO:tasks.mds_thrash.fs.[b]:waiting for 54 secs before thrashing 2023-03-03T12:45:32.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:32.798 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:45:40.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:45:40.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.458+0000 7fb12d214700 1 Processor -- start 2023-03-03T12:45:40.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.458+0000 7fb12d214700 1 -- start start 2023-03-03T12:45:40.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb12d214700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb128071e80 0x7fb128151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:40.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb12d214700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb12814e900 0x7fb12815ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:40.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb12d214700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1281510d0 con 0x7fb128072250 2023-03-03T12:45:40.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb12d214700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1280c3740 con 0x7fb128071e80 2023-03-03T12:45:40.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb12d214700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb128070670 con 0x7fb12814e900 2023-03-03T12:45:40.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb126d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb128071e80 0x7fb128151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:40.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb126d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb128071e80 0x7fb128151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41770/0 (socket says 172.21.15.7:41770) 2023-03-03T12:45:40.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb126d9d700 1 -- 172.21.15.7:0/59884474 learned_addr learned my addr 172.21.15.7:0/59884474 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:45:40.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb12759e700 1 --2- 172.21.15.7:0/59884474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb12814e900 0x7fb12815ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:40.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb126d9d700 1 -- 172.21.15.7:0/59884474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb12814e900 msgr2=0x7fb12815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:45:40.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb126d9d700 1 --2- 172.21.15.7:0/59884474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb12814e900 0x7fb12815ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:40.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb126d9d700 1 -- 172.21.15.7:0/59884474 >> v1:172.21.15.143:6789/0 conn(0x7fb128072250 legacy=0x7fb12814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:45:40.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb126d9d700 1 -- 172.21.15.7:0/59884474 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1280c39e0 con 0x7fb128071e80 2023-03-03T12:45:40.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb12759e700 1 --2- 172.21.15.7:0/59884474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb12814e900 0x7fb12815ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:45:40.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb126d9d700 1 --2- 172.21.15.7:0/59884474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 0x7fb128151820 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fb11c0098f0 tx=0x7fb11c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725023d80d83a1f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:45:40.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.459+0000 7fb125d9b700 1 -- 172.21.15.7:0/59884474 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c004230 con 0x7fb128071e80 2023-03-03T12:45:40.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.460+0000 7fb125d9b700 1 -- 172.21.15.7:0/59884474 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c004390 con 0x7fb128071e80 2023-03-03T12:45:40.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.460+0000 7fb125d9b700 1 -- 172.21.15.7:0/59884474 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c0036a0 con 0x7fb128071e80 2023-03-03T12:45:40.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.460+0000 7fb12d214700 1 -- 172.21.15.7:0/59884474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 msgr2=0x7fb128151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:40.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.460+0000 7fb12d214700 1 --2- 172.21.15.7:0/59884474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 0x7fb128151820 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fb11c0098f0 tx=0x7fb11c009c00 comp rx=0 tx=0).stop 2023-03-03T12:45:40.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.460+0000 7fb12d214700 1 -- 172.21.15.7:0/59884474 shutdown_connections 2023-03-03T12:45:40.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.460+0000 7fb12d214700 1 --2- 172.21.15.7:0/59884474 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb12814e900 0x7fb12815ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:40.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.460+0000 7fb12d214700 1 --2- 172.21.15.7:0/59884474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 0x7fb128151820 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:40.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.460+0000 7fb12d214700 1 -- 172.21.15.7:0/59884474 >> 172.21.15.7:0/59884474 conn(0x7fb1280a94d0 msgr2=0x7fb1280a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:45:40.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.460+0000 7fb12d214700 1 -- 172.21.15.7:0/59884474 shutdown_connections 2023-03-03T12:45:40.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.460+0000 7fb12d214700 1 -- 172.21.15.7:0/59884474 wait complete. 2023-03-03T12:45:40.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.462+0000 7fb12d214700 1 Processor -- start 2023-03-03T12:45:40.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.462+0000 7fb12d214700 1 -- start start 2023-03-03T12:45:40.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.462+0000 7fb12d214700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 0x7fb1280ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:40.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.462+0000 7fb12d214700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb128072250 0x7fb1281eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:40.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.462+0000 7fb12d214700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb12814e900 0x7fb1281eb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:40.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.462+0000 7fb12d214700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12814ce00 con 0x7fb128071e80 2023-03-03T12:45:40.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.462+0000 7fb12d214700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12814b040 con 0x7fb12814e900 2023-03-03T12:45:40.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.462+0000 7fb12d214700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb12814fee0 con 0x7fb128072250 2023-03-03T12:45:40.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.462+0000 7fb126d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 0x7fb1280ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:40.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb126d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 0x7fb1280ac650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41776/0 (socket says 172.21.15.7:41776) 2023-03-03T12:45:40.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb126d9d700 1 -- 172.21.15.7:0/1059189366 learned_addr learned my addr 172.21.15.7:0/1059189366 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:45:40.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb12759e700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb12814e900 0x7fb1281eb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:40.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb126d9d700 1 -- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb128072250 msgr2=0x7fb1281eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:40.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb126d9d700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb128072250 0x7fb1281eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:40.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb126d9d700 1 -- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb12814e900 msgr2=0x7fb1281eb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:40.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb126d9d700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb12814e900 0x7fb1281eb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:40.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb126d9d700 1 -- 172.21.15.7:0/1059189366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c009550 con 0x7fb128071e80 2023-03-03T12:45:40.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb126d9d700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 0x7fb1280ac650 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fb12806b550 tx=0x7fb11c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:45:40.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb10f7fe700 1 -- 172.21.15.7:0/1059189366 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c003e70 con 0x7fb128071e80 2023-03-03T12:45:40.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb10f7fe700 1 -- 172.21.15.7:0/1059189366 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c01f070 con 0x7fb128071e80 2023-03-03T12:45:40.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb12d214700 1 -- 172.21.15.7:0/1059189366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12806c9a0 con 0x7fb128071e80 2023-03-03T12:45:40.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb10f7fe700 1 -- 172.21.15.7:0/1059189366 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c00b8c0 con 0x7fb128071e80 2023-03-03T12:45:40.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.463+0000 7fb12d214700 1 -- 172.21.15.7:0/1059189366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb128151300 con 0x7fb128071e80 2023-03-03T12:45:40.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.464+0000 7fb12d214700 1 -- 172.21.15.7:0/1059189366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12806cb30 con 0x7fb128071e80 2023-03-03T12:45:40.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.465+0000 7fb10f7fe700 1 -- 172.21.15.7:0/1059189366 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb11c02a020 con 0x7fb128071e80 2023-03-03T12:45:40.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.465+0000 7fb10f7fe700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1140682f0 0x7fb11406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:40.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.465+0000 7fb10f7fe700 1 -- 172.21.15.7:0/1059189366 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb11c087a30 con 0x7fb128071e80 2023-03-03T12:45:40.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.465+0000 7fb12659c700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1140682f0 0x7fb11406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:40.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.466+0000 7fb12659c700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1140682f0 0x7fb11406a7a0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb1280c1780 tx=0x7fb118007a40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:45:40.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.470+0000 7fb10f7fe700 1 -- 172.21.15.7:0/1059189366 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb11c053dd0 con 0x7fb128071e80 2023-03-03T12:45:40.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.707+0000 7fb12d214700 1 -- 172.21.15.7:0/1059189366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb12806cb30 con 0x7fb128071e80 2023-03-03T12:45:40.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.708+0000 7fb10f7fe700 1 -- 172.21.15.7:0/1059189366 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb11c0573f0 con 0x7fb128071e80 2023-03-03T12:45:40.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:45:40.711 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:45:40.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 -- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1140682f0 msgr2=0x7fb11406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1140682f0 0x7fb11406a7a0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb1280c1780 tx=0x7fb118007a40 comp rx=0 tx=0).stop 2023-03-03T12:45:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 -- 172.21.15.7:0/1059189366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 msgr2=0x7fb1280ac650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 0x7fb1280ac650 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fb12806b550 tx=0x7fb11c004750 comp rx=0 tx=0).stop 2023-03-03T12:45:40.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 -- 172.21.15.7:0/1059189366 shutdown_connections 2023-03-03T12:45:40.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb12814e900 0x7fb1281eb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:40.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1140682f0 0x7fb11406a7a0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:40.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb128071e80 0x7fb1280ac650 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:40.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 --2- 172.21.15.7:0/1059189366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb128072250 0x7fb1281eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:40.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 -- 172.21.15.7:0/1059189366 >> 172.21.15.7:0/1059189366 conn(0x7fb1280a94d0 msgr2=0x7fb1280c2960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:45:40.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 -- 172.21.15.7:0/1059189366 shutdown_connections 2023-03-03T12:45:40.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:40.710+0000 7fb12d214700 1 -- 172.21.15.7:0/1059189366 wait complete. 2023-03-03T12:45:40.715 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:45:40.724 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T12:45:56.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:45:56.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ef1a30700 1 Processor -- start 2023-03-03T12:45:56.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ef1a30700 1 -- start start 2023-03-03T12:45:56.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ef1a30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0c00e0 0x7f4eec0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:56.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ef1a30700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eec0c7310 con 0x7f4eec154ff0 2023-03-03T12:45:56.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ef1a30700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eec0bfd10 con 0x7f4eec0caa30 2023-03-03T12:45:56.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ef1a30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec0bfa30 con 0x7f4eec0c00e0 2023-03-03T12:45:56.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4eeb7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4eec154ff0 0x7f4eec15ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35788/0 (socket says 172.21.15.7:35788) 2023-03-03T12:45:56.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4eeb7fe700 1 -- 172.21.15.7:0/3289408072 learned_addr learned my addr 172.21.15.7:0/3289408072 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:45:56.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4eea7fc700 1 --2- 172.21.15.7:0/3289408072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0c00e0 0x7f4eec0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:56.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ee9ffb700 1 -- 172.21.15.7:0/3289408072 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898833544 0 0) 0x7f4eec0c7310 con 0x7f4eec154ff0 2023-03-03T12:45:56.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ee9ffb700 1 -- 172.21.15.7:0/3289408072 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed40035b0 con 0x7f4eec154ff0 2023-03-03T12:45:56.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ee9ffb700 1 -- 172.21.15.7:0/3289408072 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199685374 0 0) 0x7f4eec0bfd10 con 0x7f4eec0caa30 2023-03-03T12:45:56.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ee9ffb700 1 -- 172.21.15.7:0/3289408072 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eec0c7310 con 0x7f4eec0caa30 2023-03-03T12:45:56.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4ee9ffb700 1 -- 172.21.15.7:0/3289408072 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 662269934 0 0) 0x7f4ed40035b0 con 0x7f4eec154ff0 2023-03-03T12:45:56.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4eea7fc700 1 -- 172.21.15.7:0/3289408072 >> v1:172.21.15.143:6789/0 conn(0x7f4eec0caa30 legacy=0x7f4eec0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:56.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4eea7fc700 1 -- 172.21.15.7:0/3289408072 >> v1:172.21.15.7:6789/0 conn(0x7f4eec154ff0 legacy=0x7f4eec15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:56.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4eea7fc700 1 -- 172.21.15.7:0/3289408072 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eec154510 con 0x7f4eec0c00e0 2023-03-03T12:45:56.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.967+0000 7f4eea7fc700 1 --2- 172.21.15.7:0/3289408072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0c00e0 0x7f4eec0c5c60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009850 tx=0x7f4ee0009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56b4dba49cae0a96 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:45:56.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.968+0000 7f4ee9ffb700 1 -- 172.21.15.7:0/3289408072 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40019f0 con 0x7f4eec0c00e0 2023-03-03T12:45:56.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.968+0000 7f4ee9ffb700 1 -- 172.21.15.7:0/3289408072 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee0005c30 con 0x7f4eec0c00e0 2023-03-03T12:45:56.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.968+0000 7f4ee9ffb700 1 -- 172.21.15.7:0/3289408072 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0004030 con 0x7f4eec0c00e0 2023-03-03T12:45:56.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.968+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3289408072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0c00e0 msgr2=0x7f4eec0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:56.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.968+0000 7f4ef1a30700 1 --2- 172.21.15.7:0/3289408072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0c00e0 0x7f4eec0c5c60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009850 tx=0x7f4ee0009b60 comp rx=0 tx=0).stop 2023-03-03T12:45:56.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.968+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3289408072 shutdown_connections 2023-03-03T12:45:56.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.968+0000 7f4ef1a30700 1 --2- 172.21.15.7:0/3289408072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0c00e0 0x7f4eec0c5c60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:56.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.968+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3289408072 >> 172.21.15.7:0/3289408072 conn(0x7f4eec066e40 msgr2=0x7f4eec071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:45:56.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.968+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3289408072 shutdown_connections 2023-03-03T12:45:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.968+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3289408072 wait complete. 2023-03-03T12:45:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.969+0000 7f4ef1a30700 1 Processor -- start 2023-03-03T12:45:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.970+0000 7f4ef1a30700 1 -- start start 2023-03-03T12:45:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.970+0000 7f4ef1a30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec0c00e0 0x7f4eec0c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.970+0000 7f4ef1a30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0caa30 0x7f4eec156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.970+0000 7f4ef1a30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eec154ff0 0x7f4eec1ed100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.970+0000 7f4ef1a30700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec0c3b60 con 0x7f4eec154ff0 2023-03-03T12:45:56.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.970+0000 7f4ef1a30700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec0c34e0 con 0x7f4eec0c00e0 2023-03-03T12:45:56.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.970+0000 7f4ef1a30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec155430 con 0x7f4eec0caa30 2023-03-03T12:45:56.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.970+0000 7f4eeb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eec154ff0 0x7f4eec1ed100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:56.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4eeb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eec154ff0 0x7f4eec1ed100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41814/0 (socket says 172.21.15.7:41814) 2023-03-03T12:45:56.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4eeb7fe700 1 -- 172.21.15.7:0/3785679777 learned_addr learned my addr 172.21.15.7:0/3785679777 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:45:56.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4eeaffd700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec0c00e0 0x7f4eec0c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:56.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4eea7fc700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0caa30 0x7f4eec156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:56.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4eeb7fe700 1 -- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0caa30 msgr2=0x7f4eec156750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:56.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4eeb7fe700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0caa30 0x7f4eec156750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:56.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4eeb7fe700 1 -- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec0c00e0 msgr2=0x7f4eec0c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:56.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4eeb7fe700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec0c00e0 0x7f4eec0c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:56.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4eeb7fe700 1 -- 172.21.15.7:0/3785679777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee0009430 con 0x7f4eec154ff0 2023-03-03T12:45:56.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4eeb7fe700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eec154ff0 0x7f4eec1ed100 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400cc00 tx=0x7f4ee400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:45:56.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4ed3fff700 1 -- 172.21.15.7:0/3785679777 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee400aa80 con 0x7f4eec154ff0 2023-03-03T12:45:56.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4ed3fff700 1 -- 172.21.15.7:0/3785679777 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee400abe0 con 0x7f4eec154ff0 2023-03-03T12:45:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3785679777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eec154ad0 con 0x7f4eec154ff0 2023-03-03T12:45:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4ed3fff700 1 -- 172.21.15.7:0/3785679777 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40070d0 con 0x7f4eec154ff0 2023-03-03T12:45:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.971+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3785679777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eec1547f0 con 0x7f4eec154ff0 2023-03-03T12:45:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.972+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3785679777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eec0bf650 con 0x7f4eec154ff0 2023-03-03T12:45:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.973+0000 7f4ed3fff700 1 -- 172.21.15.7:0/3785679777 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ee401a030 con 0x7f4eec154ff0 2023-03-03T12:45:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.973+0000 7f4ed3fff700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 0x7f4ed806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:45:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.973+0000 7f4ed3fff700 1 -- 172.21.15.7:0/3785679777 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4ee4085dd0 con 0x7f4eec154ff0 2023-03-03T12:45:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.973+0000 7f4eeaffd700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 0x7f4ed806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:45:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.974+0000 7f4eeaffd700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 0x7f4ed806a7a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0011e0 tx=0x7f4edc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:45:56.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:56.977+0000 7f4ed3fff700 1 -- 172.21.15.7:0/3785679777 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ee4052170 con 0x7f4eec154ff0 2023-03-03T12:45:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.215+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3785679777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4eec0c3ca0 con 0x7f4eec154ff0 2023-03-03T12:45:57.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.215+0000 7f4ed3fff700 1 -- 172.21.15.7:0/3785679777 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4eec0c3ca0 con 0x7f4eec154ff0 2023-03-03T12:45:57.219 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:45:57.219 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:45:57.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.217+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 msgr2=0x7f4ed806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:57.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.217+0000 7f4ef1a30700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 0x7f4ed806a7a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0011e0 tx=0x7f4edc009040 comp rx=0 tx=0).stop 2023-03-03T12:45:57.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.217+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3785679777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eec154ff0 msgr2=0x7f4eec1ed100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:45:57.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.217+0000 7f4ef1a30700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eec154ff0 0x7f4eec1ed100 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400cc00 tx=0x7f4ee400cf10 comp rx=0 tx=0).stop 2023-03-03T12:45:57.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.217+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3785679777 shutdown_connections 2023-03-03T12:45:57.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.217+0000 7f4ef1a30700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ed80682f0 0x7f4ed806a7a0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:57.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.217+0000 7f4ef1a30700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4eec154ff0 0x7f4eec1ed100 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:57.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.218+0000 7f4ef1a30700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec0c00e0 0x7f4eec0c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:57.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.218+0000 7f4ef1a30700 1 --2- 172.21.15.7:0/3785679777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4eec0caa30 0x7f4eec156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:45:57.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.218+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3785679777 >> 172.21.15.7:0/3785679777 conn(0x7f4eec066e40 msgr2=0x7f4eec153d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:45:57.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.218+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3785679777 shutdown_connections 2023-03-03T12:45:57.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:45:57.218+0000 7f4ef1a30700 1 -- 172.21.15.7:0/3785679777 wait complete. 2023-03-03T12:45:57.222 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:45:57.232 INFO:tasks.mds_thrash.fs.[b]:waiting for 54 secs before thrashing 2023-03-03T12:46:02.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:02.860 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:29.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:46:29.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.862+0000 7f8cf46a7700 1 Processor -- start 2023-03-03T12:46:29.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.862+0000 7f8cf46a7700 1 -- start start 2023-03-03T12:46:29.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf46a7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8cec14f9d0 0x7f8cec156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:29.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf46a7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8cec14fda0 0x7f8cec071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:29.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf46a7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec0c2f30 con 0x7f8cec0abe20 2023-03-03T12:46:29.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf46a7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec0c31d0 con 0x7f8cec14f9d0 2023-03-03T12:46:29.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf46a7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cec1524a0 con 0x7f8cec14fda0 2023-03-03T12:46:29.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf2443700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8cec14f9d0 0x7f8cec156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:29.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf2c44700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8cec0abe20 0x7f8cec1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35830/0 (socket says 172.21.15.7:35830) 2023-03-03T12:46:29.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf2c44700 1 -- 172.21.15.7:0/1784764056 learned_addr learned my addr 172.21.15.7:0/1784764056 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:46:29.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf1c42700 1 --2- 172.21.15.7:0/1784764056 >> v2:172.21.15.143:3300/0 conn(0x7f8cec14fda0 0x7f8cec071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:29.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf2443700 1 -- 172.21.15.7:0/1784764056 >> v2:172.21.15.143:3300/0 conn(0x7f8cec14fda0 msgr2=0x7f8cec071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:46:29.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf2443700 1 --2- 172.21.15.7:0/1784764056 >> v2:172.21.15.143:3300/0 conn(0x7f8cec14fda0 0x7f8cec071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:29.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.863+0000 7f8cf2443700 1 -- 172.21.15.7:0/1784764056 >> v1:172.21.15.7:6789/0 conn(0x7f8cec0abe20 legacy=0x7f8cec1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:29.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf2443700 1 -- 172.21.15.7:0/1784764056 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec0c34b0 con 0x7f8cec14f9d0 2023-03-03T12:46:29.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf1441700 1 -- 172.21.15.7:0/1784764056 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479608711 0 0) 0x7f8cec0c2f30 con 0x7f8cec0abe20 2023-03-03T12:46:29.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf2443700 1 --2- 172.21.15.7:0/1784764056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec14f9d0 0x7f8cec156430 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0709b0 tx=0x7f8cd800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613e0cd8dc842c12 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:29.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf1441700 1 -- 172.21.15.7:0/1784764056 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd800e070 con 0x7f8cec14f9d0 2023-03-03T12:46:29.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf1441700 1 -- 172.21.15.7:0/1784764056 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd800bb20 con 0x7f8cec14f9d0 2023-03-03T12:46:29.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf1441700 1 -- 172.21.15.7:0/1784764056 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8012430 con 0x7f8cec14f9d0 2023-03-03T12:46:29.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf46a7700 1 -- 172.21.15.7:0/1784764056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec14f9d0 msgr2=0x7f8cec156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:29.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf46a7700 1 --2- 172.21.15.7:0/1784764056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec14f9d0 0x7f8cec156430 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0709b0 tx=0x7f8cd800b330 comp rx=0 tx=0).stop 2023-03-03T12:46:29.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf46a7700 1 -- 172.21.15.7:0/1784764056 shutdown_connections 2023-03-03T12:46:29.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf46a7700 1 --2- 172.21.15.7:0/1784764056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec14f9d0 0x7f8cec156430 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:29.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf46a7700 1 --2- 172.21.15.7:0/1784764056 >> v2:172.21.15.143:3300/0 conn(0x7f8cec14fda0 0x7f8cec071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:29.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.864+0000 7f8cf46a7700 1 -- 172.21.15.7:0/1784764056 >> 172.21.15.7:0/1784764056 conn(0x7f8cec0c8900 msgr2=0x7f8cec0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:46:29.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.865+0000 7f8cf46a7700 1 -- 172.21.15.7:0/1784764056 shutdown_connections 2023-03-03T12:46:29.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.865+0000 7f8cf46a7700 1 -- 172.21.15.7:0/1784764056 wait complete. 2023-03-03T12:46:29.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf46a7700 1 Processor -- start 2023-03-03T12:46:29.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf46a7700 1 -- start start 2023-03-03T12:46:29.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf46a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec0abe20 0x7f8cec1e7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:29.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf46a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cec14f9d0 0x7f8cec0bd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:29.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf46a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cec14fda0 0x7f8cec0bfa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:29.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf46a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0c07a0 con 0x7f8cec0abe20 2023-03-03T12:46:29.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf46a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec06d510 con 0x7f8cec14fda0 2023-03-03T12:46:29.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf46a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec0bde50 con 0x7f8cec14f9d0 2023-03-03T12:46:29.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf2443700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec0abe20 0x7f8cec1e7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:29.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf2443700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec0abe20 0x7f8cec1e7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41856/0 (socket says 172.21.15.7:41856) 2023-03-03T12:46:29.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.866+0000 7f8cf2443700 1 -- 172.21.15.7:0/615806124 learned_addr learned my addr 172.21.15.7:0/615806124 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:46:29.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8cf2443700 1 -- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cec14f9d0 msgr2=0x7f8cec0bd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:29.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8cf2443700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cec14f9d0 0x7f8cec0bd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:29.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8cf2c44700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cec14fda0 0x7f8cec0bfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:29.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8cf2443700 1 -- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cec14fda0 msgr2=0x7f8cec0bfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:29.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8cf2443700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cec14fda0 0x7f8cec0bfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:29.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8cf2443700 1 -- 172.21.15.7:0/615806124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd8009a70 con 0x7f8cec0abe20 2023-03-03T12:46:29.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8cf2443700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec0abe20 0x7f8cec1e7b50 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8014040 tx=0x7f8cd8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:29.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8ceaffd700 1 -- 172.21.15.7:0/615806124 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8012430 con 0x7f8cec0abe20 2023-03-03T12:46:29.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8ceaffd700 1 -- 172.21.15.7:0/615806124 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd8020070 con 0x7f8cec0abe20 2023-03-03T12:46:29.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8ceaffd700 1 -- 172.21.15.7:0/615806124 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8004280 con 0x7f8cec0abe20 2023-03-03T12:46:29.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8cf46a7700 1 -- 172.21.15.7:0/615806124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec0ac350 con 0x7f8cec0abe20 2023-03-03T12:46:29.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.867+0000 7f8cf46a7700 1 -- 172.21.15.7:0/615806124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec0ab630 con 0x7f8cec0abe20 2023-03-03T12:46:29.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.868+0000 7f8cf46a7700 1 -- 172.21.15.7:0/615806124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec0c3e10 con 0x7f8cec0abe20 2023-03-03T12:46:29.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.869+0000 7f8ceaffd700 1 -- 172.21.15.7:0/615806124 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8cd802e070 con 0x7f8cec0abe20 2023-03-03T12:46:29.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.869+0000 7f8ceaffd700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd00682f0 0x7f8cd006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:29.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.869+0000 7f8ceaffd700 1 -- 172.21.15.7:0/615806124 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8cd8022070 con 0x7f8cec0abe20 2023-03-03T12:46:29.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.869+0000 7f8cf1c42700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd00682f0 0x7f8cd006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:29.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.870+0000 7f8cf1c42700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd00682f0 0x7f8cd006a7a0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0006b20 tx=0x7f8ce0006ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:29.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:29.874+0000 7f8ceaffd700 1 -- 172.21.15.7:0/615806124 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8cd808c070 con 0x7f8cec0abe20 2023-03-03T12:46:30.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.109+0000 7f8cf46a7700 1 -- 172.21.15.7:0/615806124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cec074bb0 con 0x7f8cec0abe20 2023-03-03T12:46:30.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.109+0000 7f8ceaffd700 1 -- 172.21.15.7:0/615806124 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8cd8054200 con 0x7f8cec0abe20 2023-03-03T12:46:30.111 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:46:30.112 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:46:30.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.111+0000 7f8cf46a7700 1 -- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd00682f0 msgr2=0x7f8cd006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:30.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.111+0000 7f8cf46a7700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd00682f0 0x7f8cd006a7a0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0006b20 tx=0x7f8ce0006ab0 comp rx=0 tx=0).stop 2023-03-03T12:46:30.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.111+0000 7f8cf46a7700 1 -- 172.21.15.7:0/615806124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec0abe20 msgr2=0x7f8cec1e7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:30.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.111+0000 7f8cf46a7700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec0abe20 0x7f8cec1e7b50 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8014040 tx=0x7f8cd8004ad0 comp rx=0 tx=0).stop 2023-03-03T12:46:30.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.112+0000 7f8cf46a7700 1 -- 172.21.15.7:0/615806124 shutdown_connections 2023-03-03T12:46:30.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.112+0000 7f8cf46a7700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cec14fda0 0x7f8cec0bfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:30.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.112+0000 7f8cf46a7700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8cd00682f0 0x7f8cd006a7a0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:30.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.112+0000 7f8cf46a7700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8cec0abe20 0x7f8cec1e7b50 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:30.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.112+0000 7f8cf46a7700 1 --2- 172.21.15.7:0/615806124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8cec14f9d0 0x7f8cec0bd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:30.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.112+0000 7f8cf46a7700 1 -- 172.21.15.7:0/615806124 >> 172.21.15.7:0/615806124 conn(0x7f8cec0c8900 msgr2=0x7f8cec1515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:46:30.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.112+0000 7f8cf46a7700 1 -- 172.21.15.7:0/615806124 shutdown_connections 2023-03-03T12:46:30.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:30.112+0000 7f8cf46a7700 1 -- 172.21.15.7:0/615806124 wait complete. 2023-03-03T12:46:30.115 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:46:30.125 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before thrashing 2023-03-03T12:46:32.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:46:32.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.264+0000 7fa4b2740700 1 Processor -- start 2023-03-03T12:46:32.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.265+0000 7fa4b2740700 1 -- start start 2023-03-03T12:46:32.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.265+0000 7fa4b2740700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac155090 0x7fa4ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:32.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.265+0000 7fa4b2740700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4ac157fe0 0x7fa4ac158460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:32.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.265+0000 7fa4b2740700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ac157b50 con 0x7fa4ac06c560 2023-03-03T12:46:32.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.265+0000 7fa4b2740700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ac06c240 con 0x7fa4ac157fe0 2023-03-03T12:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.265+0000 7fa4b2740700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0c33b0 con 0x7fa4ac155090 2023-03-03T12:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.265+0000 7fa4b0cdd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4ac157fe0 0x7fa4ac158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4b0cdd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4ac157fe0 0x7fa4ac158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41874/0 (socket says 172.21.15.7:41874) 2023-03-03T12:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4b0cdd700 1 -- 172.21.15.7:0/1847322963 learned_addr learned my addr 172.21.15.7:0/1847322963 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4abfff700 1 --2- 172.21.15.7:0/1847322963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac155090 0x7fa4ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4b0cdd700 1 -- 172.21.15.7:0/1847322963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac155090 msgr2=0x7fa4ac06d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4b0cdd700 1 --2- 172.21.15.7:0/1847322963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac155090 0x7fa4ac06d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4b0cdd700 1 -- 172.21.15.7:0/1847322963 >> v1:172.21.15.143:6789/0 conn(0x7fa4ac06c560 legacy=0x7fa4ac0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4b0cdd700 1 -- 172.21.15.7:0/1847322963 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac0c3690 con 0x7fa4ac157fe0 2023-03-03T12:46:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4abfff700 1 --2- 172.21.15.7:0/1847322963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac155090 0x7fa4ac06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:46:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4b0cdd700 1 --2- 172.21.15.7:0/1847322963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac157fe0 0x7fa4ac158460 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009e50 tx=0x7fa4a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85b7a1099ea8490 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4aaffd700 1 -- 172.21.15.7:0/1847322963 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a400e070 con 0x7fa4ac157fe0 2023-03-03T12:46:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4aaffd700 1 -- 172.21.15.7:0/1847322963 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a400b9c0 con 0x7fa4ac157fe0 2023-03-03T12:46:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.266+0000 7fa4aaffd700 1 -- 172.21.15.7:0/1847322963 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a400bd70 con 0x7fa4ac157fe0 2023-03-03T12:46:32.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.267+0000 7fa4b2740700 1 -- 172.21.15.7:0/1847322963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac157fe0 msgr2=0x7fa4ac158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.267+0000 7fa4b2740700 1 --2- 172.21.15.7:0/1847322963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac157fe0 0x7fa4ac158460 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009e50 tx=0x7fa4a400b330 comp rx=0 tx=0).stop 2023-03-03T12:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.267+0000 7fa4b2740700 1 -- 172.21.15.7:0/1847322963 shutdown_connections 2023-03-03T12:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.267+0000 7fa4b2740700 1 --2- 172.21.15.7:0/1847322963 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac157fe0 0x7fa4ac158460 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.267+0000 7fa4b2740700 1 --2- 172.21.15.7:0/1847322963 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac155090 0x7fa4ac06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.267+0000 7fa4b2740700 1 -- 172.21.15.7:0/1847322963 >> 172.21.15.7:0/1847322963 conn(0x7fa4ac0a94d0 msgr2=0x7fa4ac0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.267+0000 7fa4b2740700 1 -- 172.21.15.7:0/1847322963 shutdown_connections 2023-03-03T12:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.267+0000 7fa4b2740700 1 -- 172.21.15.7:0/1847322963 wait complete. 2023-03-03T12:46:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.268+0000 7fa4b2740700 1 Processor -- start 2023-03-03T12:46:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.269+0000 7fa4b2740700 1 -- start start 2023-03-03T12:46:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.269+0000 7fa4b2740700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac06c560 0x7fa4ac155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.269+0000 7fa4b2740700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac155090 0x7fa4ac0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.269+0000 7fa4b2740700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ac157fe0 0x7fa4ac1ea7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.269+0000 7fa4b2740700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac1e02c0 con 0x7fa4ac155090 2023-03-03T12:46:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.269+0000 7fa4b2740700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac1ea340 con 0x7fa4ac157fe0 2023-03-03T12:46:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.269+0000 7fa4b2740700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac1e9ff0 con 0x7fa4ac06c560 2023-03-03T12:46:32.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.269+0000 7fa4ab7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac155090 0x7fa4ac0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:32.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.269+0000 7fa4ab7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac155090 0x7fa4ac0c6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41878/0 (socket says 172.21.15.7:41878) 2023-03-03T12:46:32.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.269+0000 7fa4ab7fe700 1 -- 172.21.15.7:0/2462802064 learned_addr learned my addr 172.21.15.7:0/2462802064 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:46:32.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4b0cdd700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ac157fe0 0x7fa4ac1ea7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:32.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4abfff700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac06c560 0x7fa4ac155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:32.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4ab7fe700 1 -- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac06c560 msgr2=0x7fa4ac155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:32.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4ab7fe700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac06c560 0x7fa4ac155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:32.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4ab7fe700 1 -- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ac157fe0 msgr2=0x7fa4ac1ea7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4ab7fe700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ac157fe0 0x7fa4ac1ea7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4ab7fe700 1 -- 172.21.15.7:0/2462802064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4009b30 con 0x7fa4ac155090 2023-03-03T12:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4b0cdd700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ac157fe0 0x7fa4ac1ea7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4abfff700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac06c560 0x7fa4ac155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4ab7fe700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac155090 0x7fa4ac0c6420 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fa49400c4e0 tx=0x7fa49400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4a8ff9700 1 -- 172.21.15.7:0/2462802064 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940052d0 con 0x7fa4ac155090 2023-03-03T12:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4a8ff9700 1 -- 172.21.15.7:0/2462802064 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa494005430 con 0x7fa4ac155090 2023-03-03T12:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4b2740700 1 -- 172.21.15.7:0/2462802064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac1589a0 con 0x7fa4ac155090 2023-03-03T12:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4a8ff9700 1 -- 172.21.15.7:0/2462802064 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940038c0 con 0x7fa4ac155090 2023-03-03T12:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.270+0000 7fa4b2740700 1 -- 172.21.15.7:0/2462802064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac0c3970 con 0x7fa4ac155090 2023-03-03T12:46:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.271+0000 7fa4b2740700 1 -- 172.21.15.7:0/2462802064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac1545e0 con 0x7fa4ac155090 2023-03-03T12:46:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.272+0000 7fa4a8ff9700 1 -- 172.21.15.7:0/2462802064 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa494019030 con 0x7fa4ac155090 2023-03-03T12:46:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.272+0000 7fa4a8ff9700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa48c0682f0 0x7fa48c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.272+0000 7fa4a8ff9700 1 -- 172.21.15.7:0/2462802064 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa494085500 con 0x7fa4ac155090 2023-03-03T12:46:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.272+0000 7fa4abfff700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa48c0682f0 0x7fa48c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.273+0000 7fa4abfff700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa48c0682f0 0x7fa48c06a7a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009a60 tx=0x7fa49c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.276+0000 7fa4a8ff9700 1 -- 172.21.15.7:0/2462802064 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa4940518a0 con 0x7fa4ac155090 2023-03-03T12:46:32.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.515+0000 7fa4b2740700 1 -- 172.21.15.7:0/2462802064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4ac074bb0 con 0x7fa4ac155090 2023-03-03T12:46:32.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.515+0000 7fa4a8ff9700 1 -- 172.21.15.7:0/2462802064 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa494054ec0 con 0x7fa4ac155090 2023-03-03T12:46:32.517 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:46:32.518 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:46:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 -- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa48c0682f0 msgr2=0x7fa48c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa48c0682f0 0x7fa48c06a7a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009a60 tx=0x7fa49c008040 comp rx=0 tx=0).stop 2023-03-03T12:46:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 -- 172.21.15.7:0/2462802064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac155090 msgr2=0x7fa4ac0c6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac155090 0x7fa4ac0c6420 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fa49400c4e0 tx=0x7fa49400c8a0 comp rx=0 tx=0).stop 2023-03-03T12:46:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 -- 172.21.15.7:0/2462802064 shutdown_connections 2023-03-03T12:46:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ac157fe0 0x7fa4ac1ea7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4ac06c560 0x7fa4ac155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa48c0682f0 0x7fa48c06a7a0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 --2- 172.21.15.7:0/2462802064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4ac155090 0x7fa4ac0c6420 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 -- 172.21.15.7:0/2462802064 >> 172.21.15.7:0/2462802064 conn(0x7fa4ac0a94d0 msgr2=0x7fa4ac071660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:46:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 -- 172.21.15.7:0/2462802064 shutdown_connections 2023-03-03T12:46:32.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:32.517+0000 7fa4b2740700 1 -- 172.21.15.7:0/2462802064 wait complete. 2023-03-03T12:46:32.521 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:46:32.531 INFO:tasks.mds_thrash.fs.[a]:waiting for 11 secs before thrashing 2023-03-03T12:46:33.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:33.007 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:46:43.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:46:43.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.669+0000 7fc81ffda700 1 Processor -- start 2023-03-03T12:46:43.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.669+0000 7fc81ffda700 1 -- start start 2023-03-03T12:46:43.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81ffda700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc8180caa30 0x7fc8180c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:43.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81ffda700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc818154ff0 0x7fc8180cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:43.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81ffda700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8180c7310 con 0x7fc8180c00e0 2023-03-03T12:46:43.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81ffda700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc818155430 con 0x7fc8180caa30 2023-03-03T12:46:43.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81ffda700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8180bfd10 con 0x7fc818154ff0 2023-03-03T12:46:43.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81dd76700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc8180caa30 0x7fc8180c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:43.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81dd76700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc8180caa30 0x7fc8180c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41890/0 (socket says 172.21.15.7:41890) 2023-03-03T12:46:43.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81dd76700 1 -- 172.21.15.7:0/2603058323 learned_addr learned my addr 172.21.15.7:0/2603058323 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:46:43.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81e577700 1 --2- 172.21.15.7:0/2603058323 >> v2:172.21.15.143:3300/0 conn(0x7fc818154ff0 0x7fc8180cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:43.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81dd76700 1 -- 172.21.15.7:0/2603058323 >> v2:172.21.15.143:3300/0 conn(0x7fc818154ff0 msgr2=0x7fc8180cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:46:43.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81dd76700 1 --2- 172.21.15.7:0/2603058323 >> v2:172.21.15.143:3300/0 conn(0x7fc818154ff0 0x7fc8180cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81dd76700 1 -- 172.21.15.7:0/2603058323 >> v1:172.21.15.143:6789/0 conn(0x7fc8180c00e0 legacy=0x7fc8180c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:43.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81dd76700 1 -- 172.21.15.7:0/2603058323 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8180bfa30 con 0x7fc8180caa30 2023-03-03T12:46:43.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81e577700 1 --2- 172.21.15.7:0/2603058323 >> v2:172.21.15.143:3300/0 conn(0x7fc818154ff0 0x7fc8180cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:46:43.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.670+0000 7fc81dd76700 1 --2- 172.21.15.7:0/2603058323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8180caa30 0x7fc8180c5c60 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fc818071530 tx=0x7fc808009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e9d026731d56d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:43.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81cd74700 1 -- 172.21.15.7:0/2603058323 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80800b870 con 0x7fc8180caa30 2023-03-03T12:46:43.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81cd74700 1 -- 172.21.15.7:0/2603058323 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc80800b9d0 con 0x7fc8180caa30 2023-03-03T12:46:43.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81cd74700 1 -- 172.21.15.7:0/2603058323 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc80800bd60 con 0x7fc8180caa30 2023-03-03T12:46:43.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81ffda700 1 -- 172.21.15.7:0/2603058323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8180caa30 msgr2=0x7fc8180c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:43.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81ffda700 1 --2- 172.21.15.7:0/2603058323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8180caa30 0x7fc8180c5c60 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fc818071530 tx=0x7fc808009bd0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81ffda700 1 -- 172.21.15.7:0/2603058323 shutdown_connections 2023-03-03T12:46:43.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81ffda700 1 --2- 172.21.15.7:0/2603058323 >> v2:172.21.15.143:3300/0 conn(0x7fc818154ff0 0x7fc8180cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81ffda700 1 --2- 172.21.15.7:0/2603058323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8180caa30 0x7fc8180c5c60 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81ffda700 1 -- 172.21.15.7:0/2603058323 >> 172.21.15.7:0/2603058323 conn(0x7fc818066e40 msgr2=0x7fc818071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:46:43.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81ffda700 1 -- 172.21.15.7:0/2603058323 shutdown_connections 2023-03-03T12:46:43.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.671+0000 7fc81ffda700 1 -- 172.21.15.7:0/2603058323 wait complete. 2023-03-03T12:46:43.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.673+0000 7fc81ffda700 1 Processor -- start 2023-03-03T12:46:43.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.673+0000 7fc81ffda700 1 -- start start 2023-03-03T12:46:43.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81ffda700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8180c00e0 0x7fc818156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:43.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81ffda700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8180caa30 0x7fc8180aa030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:43.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81ffda700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc818154ff0 0x7fc8180c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:43.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81ffda700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8180c3c40 con 0x7fc818154ff0 2023-03-03T12:46:43.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81ffda700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8181ea3c0 con 0x7fc8180c00e0 2023-03-03T12:46:43.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81ffda700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8181ea6a0 con 0x7fc8180caa30 2023-03-03T12:46:43.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81e577700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc818154ff0 0x7fc8180c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:43.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81e577700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc818154ff0 0x7fc8180c4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41896/0 (socket says 172.21.15.7:41896) 2023-03-03T12:46:43.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81e577700 1 -- 172.21.15.7:0/3240119122 learned_addr learned my addr 172.21.15.7:0/3240119122 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:46:43.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81dd76700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8180c00e0 0x7fc818156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:43.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81d575700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8180caa30 0x7fc8180aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:43.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81e577700 1 -- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8180caa30 msgr2=0x7fc8180aa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:43.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81e577700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8180caa30 0x7fc8180aa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81e577700 1 -- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8180c00e0 msgr2=0x7fc818156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:43.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81e577700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8180c00e0 0x7fc818156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81e577700 1 -- 172.21.15.7:0/3240119122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc808009550 con 0x7fc818154ff0 2023-03-03T12:46:43.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.674+0000 7fc81dd76700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8180c00e0 0x7fc818156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:46:43.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.675+0000 7fc81e577700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc818154ff0 0x7fc8180c4690 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fc810007e70 tx=0x7fc81000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:43.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.675+0000 7fc80e7fc700 1 -- 172.21.15.7:0/3240119122 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81000de40 con 0x7fc818154ff0 2023-03-03T12:46:43.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.675+0000 7fc80e7fc700 1 -- 172.21.15.7:0/3240119122 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81000f040 con 0x7fc818154ff0 2023-03-03T12:46:43.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.675+0000 7fc81ffda700 1 -- 172.21.15.7:0/3240119122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8181547f0 con 0x7fc818154ff0 2023-03-03T12:46:43.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.675+0000 7fc80e7fc700 1 -- 172.21.15.7:0/3240119122 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc810003c60 con 0x7fc818154ff0 2023-03-03T12:46:43.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.675+0000 7fc81ffda700 1 -- 172.21.15.7:0/3240119122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc818154510 con 0x7fc818154ff0 2023-03-03T12:46:43.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.676+0000 7fc81ffda700 1 -- 172.21.15.7:0/3240119122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc818154ad0 con 0x7fc818154ff0 2023-03-03T12:46:43.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.676+0000 7fc80e7fc700 1 -- 172.21.15.7:0/3240119122 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc81001c030 con 0x7fc818154ff0 2023-03-03T12:46:43.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.677+0000 7fc80e7fc700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7fc0682f0 0x7fc7fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:43.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.677+0000 7fc80e7fc700 1 -- 172.21.15.7:0/3240119122 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc810086140 con 0x7fc818154ff0 2023-03-03T12:46:43.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.677+0000 7fc81dd76700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7fc0682f0 0x7fc7fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:43.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.678+0000 7fc81dd76700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7fc0682f0 0x7fc7fc06a7a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc808000c00 tx=0x7fc808008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:43.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.681+0000 7fc80e7fc700 1 -- 172.21.15.7:0/3240119122 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc8100524e0 con 0x7fc818154ff0 2023-03-03T12:46:43.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.919+0000 7fc81ffda700 1 -- 172.21.15.7:0/3240119122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc818074bb0 con 0x7fc818154ff0 2023-03-03T12:46:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.919+0000 7fc80e7fc700 1 -- 172.21.15.7:0/3240119122 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc810055b00 con 0x7fc818154ff0 2023-03-03T12:46:43.921 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:46:43.921 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:46:43.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 -- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7fc0682f0 msgr2=0x7fc7fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:43.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7fc0682f0 0x7fc7fc06a7a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc808000c00 tx=0x7fc808008030 comp rx=0 tx=0).stop 2023-03-03T12:46:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 -- 172.21.15.7:0/3240119122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc818154ff0 msgr2=0x7fc8180c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc818154ff0 0x7fc8180c4690 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fc810007e70 tx=0x7fc81000d3b0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 -- 172.21.15.7:0/3240119122 shutdown_connections 2023-03-03T12:46:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7fc0682f0 0x7fc7fc06a7a0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc818154ff0 0x7fc8180c4690 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8180c00e0 0x7fc818156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 --2- 172.21.15.7:0/3240119122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8180caa30 0x7fc8180aa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 -- 172.21.15.7:0/3240119122 >> 172.21.15.7:0/3240119122 conn(0x7fc818066e40 msgr2=0x7fc8181520a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:46:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 -- 172.21.15.7:0/3240119122 shutdown_connections 2023-03-03T12:46:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:43.921+0000 7fc81ffda700 1 -- 172.21.15.7:0/3240119122 wait complete. 2023-03-03T12:46:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:46:43.935 INFO:tasks.mds_thrash.fs.[a]:waiting for 82 secs before thrashing 2023-03-03T12:46:51.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:46:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.370+0000 7efc8550e700 1 Processor -- start 2023-03-03T12:46:51.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.370+0000 7efc8550e700 1 -- start start 2023-03-03T12:46:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc8550e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efc800c00e0 0x7efc800c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc8550e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc80154ff0 0x7efc800cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc8550e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc800c7310 con 0x7efc800caa30 2023-03-03T12:46:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc8550e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc80155430 con 0x7efc80154ff0 2023-03-03T12:46:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc8550e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc800bfd10 con 0x7efc800c00e0 2023-03-03T12:46:51.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc7f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc80154ff0 0x7efc800cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc7f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efc80154ff0 0x7efc800cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41918/0 (socket says 172.21.15.7:41918) 2023-03-03T12:46:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc7f7fe700 1 -- 172.21.15.7:0/3906231897 learned_addr learned my addr 172.21.15.7:0/3906231897 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:46:51.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc7e7fc700 1 --2- 172.21.15.7:0/3906231897 >> v2:172.21.15.143:3300/0 conn(0x7efc800c00e0 0x7efc800c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc7f7fe700 1 -- 172.21.15.7:0/3906231897 >> v2:172.21.15.143:3300/0 conn(0x7efc800c00e0 msgr2=0x7efc800c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:46:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc7f7fe700 1 --2- 172.21.15.7:0/3906231897 >> v2:172.21.15.143:3300/0 conn(0x7efc800c00e0 0x7efc800c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc7f7fe700 1 -- 172.21.15.7:0/3906231897 >> v1:172.21.15.143:6789/0 conn(0x7efc800caa30 legacy=0x7efc800c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc7f7fe700 1 -- 172.21.15.7:0/3906231897 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc800bfa30 con 0x7efc80154ff0 2023-03-03T12:46:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.371+0000 7efc7e7fc700 1 --2- 172.21.15.7:0/3906231897 >> v2:172.21.15.143:3300/0 conn(0x7efc800c00e0 0x7efc800c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:46:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc7f7fe700 1 --2- 172.21.15.7:0/3906231897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 0x7efc800cae00 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7efc800ca910 tx=0x7efc7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7def8e57fe80c7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc7dffb700 1 -- 172.21.15.7:0/3906231897 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7800e070 con 0x7efc80154ff0 2023-03-03T12:46:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc7dffb700 1 -- 172.21.15.7:0/3906231897 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7800b9c0 con 0x7efc80154ff0 2023-03-03T12:46:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc7dffb700 1 -- 172.21.15.7:0/3906231897 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7800bd70 con 0x7efc80154ff0 2023-03-03T12:46:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc8550e700 1 -- 172.21.15.7:0/3906231897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 msgr2=0x7efc800cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc8550e700 1 --2- 172.21.15.7:0/3906231897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 0x7efc800cae00 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7efc800ca910 tx=0x7efc7800b330 comp rx=0 tx=0).stop 2023-03-03T12:46:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc8550e700 1 -- 172.21.15.7:0/3906231897 shutdown_connections 2023-03-03T12:46:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc8550e700 1 --2- 172.21.15.7:0/3906231897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 0x7efc800cae00 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc8550e700 1 --2- 172.21.15.7:0/3906231897 >> v2:172.21.15.143:3300/0 conn(0x7efc800c00e0 0x7efc800c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc8550e700 1 -- 172.21.15.7:0/3906231897 >> 172.21.15.7:0/3906231897 conn(0x7efc80066e40 msgr2=0x7efc80071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:46:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc8550e700 1 -- 172.21.15.7:0/3906231897 shutdown_connections 2023-03-03T12:46:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.372+0000 7efc8550e700 1 -- 172.21.15.7:0/3906231897 wait complete. 2023-03-03T12:46:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.374+0000 7efc8550e700 1 Processor -- start 2023-03-03T12:46:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.374+0000 7efc8550e700 1 -- start start 2023-03-03T12:46:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.374+0000 7efc8550e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc800c00e0 0x7efc801575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.374+0000 7efc8550e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc800caa30 0x7efc801ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc8550e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 0x7efc8014cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc8550e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc800c45e0 con 0x7efc80154ff0 2023-03-03T12:46:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc8550e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc800c48c0 con 0x7efc800c00e0 2023-03-03T12:46:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc8550e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8014db30 con 0x7efc800caa30 2023-03-03T12:46:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 0x7efc8014cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 0x7efc8014cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41924/0 (socket says 172.21.15.7:41924) 2023-03-03T12:46:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7f7fe700 1 -- 172.21.15.7:0/1791827892 learned_addr learned my addr 172.21.15.7:0/1791827892 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:46:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7effd700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc800c00e0 0x7efc801575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7e7fc700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc800caa30 0x7efc801ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7f7fe700 1 -- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc800caa30 msgr2=0x7efc801ed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7f7fe700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc800caa30 0x7efc801ed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7f7fe700 1 -- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc800c00e0 msgr2=0x7efc801575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7f7fe700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc800c00e0 0x7efc801575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7f7fe700 1 -- 172.21.15.7:0/1791827892 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc78009b30 con 0x7efc80154ff0 2023-03-03T12:46:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc7f7fe700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 0x7efc8014cd30 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7efc7800b730 tx=0x7efc78004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc677fe700 1 -- 172.21.15.7:0/1791827892 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc78004870 con 0x7efc80154ff0 2023-03-03T12:46:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc677fe700 1 -- 172.21.15.7:0/1791827892 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc780099b0 con 0x7efc80154ff0 2023-03-03T12:46:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc8550e700 1 -- 172.21.15.7:0/1791827892 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc801547f0 con 0x7efc80154ff0 2023-03-03T12:46:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc677fe700 1 -- 172.21.15.7:0/1791827892 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7801b670 con 0x7efc80154ff0 2023-03-03T12:46:51.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.375+0000 7efc8550e700 1 -- 172.21.15.7:0/1791827892 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc80154510 con 0x7efc80154ff0 2023-03-03T12:46:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.376+0000 7efc8550e700 1 -- 172.21.15.7:0/1791827892 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc80154ad0 con 0x7efc80154ff0 2023-03-03T12:46:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.377+0000 7efc677fe700 1 -- 172.21.15.7:0/1791827892 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efc78021030 con 0x7efc80154ff0 2023-03-03T12:46:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.377+0000 7efc677fe700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc6c0682f0 0x7efc6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:46:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.377+0000 7efc677fe700 1 -- 172.21.15.7:0/1791827892 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efc780875c0 con 0x7efc80154ff0 2023-03-03T12:46:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.377+0000 7efc7effd700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc6c0682f0 0x7efc6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:46:51.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.378+0000 7efc7effd700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc6c0682f0 0x7efc6c06a7a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7efc800af540 tx=0x7efc74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:46:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.382+0000 7efc677fe700 1 -- 172.21.15.7:0/1791827892 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efc78053870 con 0x7efc80154ff0 2023-03-03T12:46:51.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.622+0000 7efc8550e700 1 -- 172.21.15.7:0/1791827892 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc800c7990 con 0x7efc80154ff0 2023-03-03T12:46:51.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.622+0000 7efc677fe700 1 -- 172.21.15.7:0/1791827892 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efc7801f020 con 0x7efc80154ff0 2023-03-03T12:46:51.625 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:46:51.625 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:46:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 -- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc6c0682f0 msgr2=0x7efc6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc6c0682f0 0x7efc6c06a7a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7efc800af540 tx=0x7efc74009040 comp rx=0 tx=0).stop 2023-03-03T12:46:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 -- 172.21.15.7:0/1791827892 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 msgr2=0x7efc8014cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:46:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 0x7efc8014cd30 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7efc7800b730 tx=0x7efc78004ad0 comp rx=0 tx=0).stop 2023-03-03T12:46:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 -- 172.21.15.7:0/1791827892 shutdown_connections 2023-03-03T12:46:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc6c0682f0 0x7efc6c06a7a0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc80154ff0 0x7efc8014cd30 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc800c00e0 0x7efc801575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 --2- 172.21.15.7:0/1791827892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc800caa30 0x7efc801ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:46:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 -- 172.21.15.7:0/1791827892 >> 172.21.15.7:0/1791827892 conn(0x7efc80066e40 msgr2=0x7efc80073f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:46:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 -- 172.21.15.7:0/1791827892 shutdown_connections 2023-03-03T12:46:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:46:51.624+0000 7efc8550e700 1 -- 172.21.15.7:0/1791827892 wait complete. 2023-03-03T12:46:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:46:51.639 INFO:tasks.mds_thrash.fs.[b]:waiting for 50 secs before thrashing 2023-03-03T12:47:03.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:03.170 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:33.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:33.309 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:47:41.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:47:41.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.777+0000 7f5616da1700 1 Processor -- start 2023-03-03T12:47:41.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.777+0000 7f5616da1700 1 -- start start 2023-03-03T12:47:41.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5616da1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f561014f860 0x7f56101562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:41.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5616da1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f56100c20d0 0x7f5610152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:41.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5616da1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5610155e60 con 0x7f56100c27c0 2023-03-03T12:47:41.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5616da1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f561006d350 con 0x7f561014f860 2023-03-03T12:47:41.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5616da1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56101558e0 con 0x7f56100c20d0 2023-03-03T12:47:41.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5614b3d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f561014f860 0x7f56101562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:41.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5614b3d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f561014f860 0x7f56101562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41950/0 (socket says 172.21.15.7:41950) 2023-03-03T12:47:41.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5614b3d700 1 -- 172.21.15.7:0/2099046621 learned_addr learned my addr 172.21.15.7:0/2099046621 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:47:41.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f560ffff700 1 --2- 172.21.15.7:0/2099046621 >> v2:172.21.15.143:3300/0 conn(0x7f56100c20d0 0x7f5610152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:41.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5614b3d700 1 -- 172.21.15.7:0/2099046621 >> v2:172.21.15.143:3300/0 conn(0x7f56100c20d0 msgr2=0x7f5610152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:47:41.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5614b3d700 1 --2- 172.21.15.7:0/2099046621 >> v2:172.21.15.143:3300/0 conn(0x7f56100c20d0 0x7f5610152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:41.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.778+0000 7f5614b3d700 1 -- 172.21.15.7:0/2099046621 >> v1:172.21.15.143:6789/0 conn(0x7f56100c27c0 legacy=0x7f56100736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:47:41.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f5614b3d700 1 -- 172.21.15.7:0/2099046621 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56101532b0 con 0x7f561014f860 2023-03-03T12:47:41.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f560ffff700 1 --2- 172.21.15.7:0/2099046621 >> v2:172.21.15.143:3300/0 conn(0x7f56100c20d0 0x7f5610152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:47:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f5614b3d700 1 --2- 172.21.15.7:0/2099046621 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f561014f860 0x7f56101562f0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f5610072520 tx=0x7f5604009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f38a35021fcdfc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:47:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f560f7fe700 1 -- 172.21.15.7:0/2099046621 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5604004230 con 0x7f561014f860 2023-03-03T12:47:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f560f7fe700 1 -- 172.21.15.7:0/2099046621 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5604004390 con 0x7f561014f860 2023-03-03T12:47:41.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f560f7fe700 1 -- 172.21.15.7:0/2099046621 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56040036a0 con 0x7f561014f860 2023-03-03T12:47:41.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f5616da1700 1 -- 172.21.15.7:0/2099046621 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f561014f860 msgr2=0x7f56101562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:41.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f5616da1700 1 --2- 172.21.15.7:0/2099046621 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f561014f860 0x7f56101562f0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f5610072520 tx=0x7f5604009bd0 comp rx=0 tx=0).stop 2023-03-03T12:47:41.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f5616da1700 1 -- 172.21.15.7:0/2099046621 shutdown_connections 2023-03-03T12:47:41.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f5616da1700 1 --2- 172.21.15.7:0/2099046621 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f561014f860 0x7f56101562f0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.779+0000 7f5616da1700 1 --2- 172.21.15.7:0/2099046621 >> v2:172.21.15.143:3300/0 conn(0x7f56100c20d0 0x7f5610152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.780+0000 7f5616da1700 1 -- 172.21.15.7:0/2099046621 >> 172.21.15.7:0/2099046621 conn(0x7f56100a94d0 msgr2=0x7f56100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:47:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.780+0000 7f5616da1700 1 -- 172.21.15.7:0/2099046621 shutdown_connections 2023-03-03T12:47:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.780+0000 7f5616da1700 1 -- 172.21.15.7:0/2099046621 wait complete. 2023-03-03T12:47:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.781+0000 7f5616da1700 1 Processor -- start 2023-03-03T12:47:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.781+0000 7f5616da1700 1 -- start start 2023-03-03T12:47:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.782+0000 7f5616da1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56100c20d0 0x7f5610154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:41.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.782+0000 7f5616da1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56100c27c0 0x7f56101def00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.782+0000 7f5616da1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f561014f860 0x7f56100c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.782+0000 7f5616da1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5610153c20 con 0x7f56100c27c0 2023-03-03T12:47:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.782+0000 7f5616da1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5610153fb0 con 0x7f561014f860 2023-03-03T12:47:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.782+0000 7f5616da1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561014c400 con 0x7f56100c20d0 2023-03-03T12:47:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.782+0000 7f560ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56100c27c0 0x7f56101def00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.782+0000 7f560ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56100c27c0 0x7f56101def00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41956/0 (socket says 172.21.15.7:41956) 2023-03-03T12:47:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.782+0000 7f560ffff700 1 -- 172.21.15.7:0/2209805699 learned_addr learned my addr 172.21.15.7:0/2209805699 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:47:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f561533e700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f561014f860 0x7f56100c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f5614b3d700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56100c20d0 0x7f5610154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f560ffff700 1 -- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56100c20d0 msgr2=0x7f5610154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f560ffff700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56100c20d0 0x7f5610154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f560ffff700 1 -- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f561014f860 msgr2=0x7f56100c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f560ffff700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f561014f860 0x7f56100c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f560ffff700 1 -- 172.21.15.7:0/2209805699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5604009550 con 0x7f56100c27c0 2023-03-03T12:47:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f560ffff700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56100c27c0 0x7f56101def00 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00c780 tx=0x7f55fc00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:47:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f560d7fa700 1 -- 172.21.15.7:0/2209805699 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc011800 con 0x7f56100c27c0 2023-03-03T12:47:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f560d7fa700 1 -- 172.21.15.7:0/2209805699 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55fc011960 con 0x7f56100c27c0 2023-03-03T12:47:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f5616da1700 1 -- 172.21.15.7:0/2209805699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561006d070 con 0x7f56100c27c0 2023-03-03T12:47:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f560d7fa700 1 -- 172.21.15.7:0/2209805699 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55fc011cd0 con 0x7f56100c27c0 2023-03-03T12:47:41.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.783+0000 7f5616da1700 1 -- 172.21.15.7:0/2209805699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561006cd90 con 0x7f56100c27c0 2023-03-03T12:47:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.784+0000 7f5616da1700 1 -- 172.21.15.7:0/2209805699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5610155af0 con 0x7f56100c27c0 2023-03-03T12:47:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.785+0000 7f560d7fa700 1 -- 172.21.15.7:0/2209805699 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f55fc020070 con 0x7f56100c27c0 2023-03-03T12:47:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.786+0000 7f560d7fa700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f40682f0 0x7f55f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:47:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.786+0000 7f560d7fa700 1 -- 172.21.15.7:0/2209805699 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f55fc086410 con 0x7f56100c27c0 2023-03-03T12:47:41.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.786+0000 7f5614b3d700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f40682f0 0x7f55f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:47:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.786+0000 7f5614b3d700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f40682f0 0x7f55f406a7a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5604009130 tx=0x7f5604008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:47:41.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:41.789+0000 7f560d7fa700 1 -- 172.21.15.7:0/2209805699 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f55fc0527b0 con 0x7f56100c27c0 2023-03-03T12:47:42.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.029+0000 7f5616da1700 1 -- 172.21.15.7:0/2209805699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5610074bb0 con 0x7f56100c27c0 2023-03-03T12:47:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.029+0000 7f560d7fa700 1 -- 172.21.15.7:0/2209805699 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f55fc055dd0 con 0x7f56100c27c0 2023-03-03T12:47:42.031 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:47:42.031 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:47:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 -- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f40682f0 msgr2=0x7f55f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f40682f0 0x7f55f406a7a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5604009130 tx=0x7f5604008030 comp rx=0 tx=0).stop 2023-03-03T12:47:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 -- 172.21.15.7:0/2209805699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56100c27c0 msgr2=0x7f56101def00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:47:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56100c27c0 0x7f56101def00 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00c780 tx=0x7f55fc00cb40 comp rx=0 tx=0).stop 2023-03-03T12:47:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 -- 172.21.15.7:0/2209805699 shutdown_connections 2023-03-03T12:47:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f561014f860 0x7f56100c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56100c20d0 0x7f5610154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f40682f0 0x7f55f406a7a0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 --2- 172.21.15.7:0/2209805699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56100c27c0 0x7f56101def00 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:47:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 -- 172.21.15.7:0/2209805699 >> 172.21.15.7:0/2209805699 conn(0x7f56100a94d0 msgr2=0x7f56100c3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:47:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 -- 172.21.15.7:0/2209805699 shutdown_connections 2023-03-03T12:47:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:47:42.031+0000 7f5616da1700 1 -- 172.21.15.7:0/2209805699 wait complete. 2023-03-03T12:47:42.035 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:47:42.045 INFO:tasks.mds_thrash.fs.[b]:waiting for 85 secs before thrashing 2023-03-03T12:48:03.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:03.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:05.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:48:06.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.071+0000 7f8406d5f700 1 Processor -- start 2023-03-03T12:48:06.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.071+0000 7f8406d5f700 1 -- start start 2023-03-03T12:48:06.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f8406d5f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f840014de50 0x7f840006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f8406d5f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f84000c65c0 0x7f84000c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f8406d5f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84000a65b0 con 0x7f840006a7f0 2023-03-03T12:48:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f8406d5f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8400154e00 con 0x7f84000c65c0 2023-03-03T12:48:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f8406d5f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84000c3370 con 0x7f840014de50 2023-03-03T12:48:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f83fffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f84000c65c0 0x7f84000c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f84052fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f840006a7f0 0x7f84000c2c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35960/0 (socket says 172.21.15.7:35960) 2023-03-03T12:48:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f84052fc700 1 -- 172.21.15.7:0/1172739337 learned_addr learned my addr 172.21.15.7:0/1172739337 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:48:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f8404afb700 1 --2- 172.21.15.7:0/1172739337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f840014de50 0x7f840006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f83ff7fe700 1 -- 172.21.15.7:0/1172739337 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081541986 0 0) 0x7f84000a65b0 con 0x7f840006a7f0 2023-03-03T12:48:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f83ff7fe700 1 -- 172.21.15.7:0/1172739337 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f00035b0 con 0x7f840006a7f0 2023-03-03T12:48:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f83fffff700 1 -- 172.21.15.7:0/1172739337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f840014de50 msgr2=0x7f840006a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:48:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.072+0000 7f83fffff700 1 --2- 172.21.15.7:0/1172739337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f840014de50 0x7f840006a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f83fffff700 1 -- 172.21.15.7:0/1172739337 >> v1:172.21.15.7:6789/0 conn(0x7f840006a7f0 legacy=0x7f84000c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f83fffff700 1 -- 172.21.15.7:0/1172739337 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84001541f0 con 0x7f84000c65c0 2023-03-03T12:48:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f8404afb700 1 --2- 172.21.15.7:0/1172739337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f840014de50 0x7f840006a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:48:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f83fffff700 1 --2- 172.21.15.7:0/1172739337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84000c65c0 0x7f84000c2720 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f83ec009840 tx=0x7f83ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d745cbfb71921681 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:48:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f83ff7fe700 1 -- 172.21.15.7:0/1172739337 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ec011830 con 0x7f84000c65c0 2023-03-03T12:48:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f83ff7fe700 1 -- 172.21.15.7:0/1172739337 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ec011990 con 0x7f84000c65c0 2023-03-03T12:48:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f83ff7fe700 1 -- 172.21.15.7:0/1172739337 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ec011d20 con 0x7f84000c65c0 2023-03-03T12:48:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f8406d5f700 1 -- 172.21.15.7:0/1172739337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84000c65c0 msgr2=0x7f84000c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f8406d5f700 1 --2- 172.21.15.7:0/1172739337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84000c65c0 0x7f84000c2720 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f83ec009840 tx=0x7f83ec009b50 comp rx=0 tx=0).stop 2023-03-03T12:48:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f8406d5f700 1 -- 172.21.15.7:0/1172739337 shutdown_connections 2023-03-03T12:48:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f8406d5f700 1 --2- 172.21.15.7:0/1172739337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f840014de50 0x7f840006a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f8406d5f700 1 --2- 172.21.15.7:0/1172739337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84000c65c0 0x7f84000c2720 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f8406d5f700 1 -- 172.21.15.7:0/1172739337 >> 172.21.15.7:0/1172739337 conn(0x7f84000a9430 msgr2=0x7f84000c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:48:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.073+0000 7f8406d5f700 1 -- 172.21.15.7:0/1172739337 shutdown_connections 2023-03-03T12:48:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.074+0000 7f8406d5f700 1 -- 172.21.15.7:0/1172739337 wait complete. 2023-03-03T12:48:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.075+0000 7f8406d5f700 1 Processor -- start 2023-03-03T12:48:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.075+0000 7f8406d5f700 1 -- start start 2023-03-03T12:48:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.075+0000 7f8406d5f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f840006a7f0 0x7f840006fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.075+0000 7f8406d5f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84000c65c0 0x7f840014fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.075+0000 7f8406d5f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f840014de50 0x7f8400150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.075+0000 7f8406d5f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8400153680 con 0x7f840014de50 2023-03-03T12:48:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.075+0000 7f8406d5f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840014f620 con 0x7f840006a7f0 2023-03-03T12:48:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.075+0000 7f8406d5f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8400152a70 con 0x7f84000c65c0 2023-03-03T12:48:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f84052fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f840014de50 0x7f8400150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f84052fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f840014de50 0x7f8400150180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41986/0 (socket says 172.21.15.7:41986) 2023-03-03T12:48:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f84052fc700 1 -- 172.21.15.7:0/2208947701 learned_addr learned my addr 172.21.15.7:0/2208947701 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:48:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f83fffff700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84000c65c0 0x7f840014fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f8404afb700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f840006a7f0 0x7f840006fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f84052fc700 1 -- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84000c65c0 msgr2=0x7f840014fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f84052fc700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84000c65c0 0x7f840014fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f84052fc700 1 -- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f840006a7f0 msgr2=0x7f840006fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f84052fc700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f840006a7f0 0x7f840006fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f84052fc700 1 -- 172.21.15.7:0/2208947701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ec0094a0 con 0x7f840014de50 2023-03-03T12:48:06.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f8404afb700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f840006a7f0 0x7f840006fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:48:06.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f84052fc700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f840014de50 0x7f8400150180 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f83f800caa0 tx=0x7f83f800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:48:06.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f83fd7fa700 1 -- 172.21.15.7:0/2208947701 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f800aa80 con 0x7f840014de50 2023-03-03T12:48:06.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f83fd7fa700 1 -- 172.21.15.7:0/2208947701 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f800abe0 con 0x7f840014de50 2023-03-03T12:48:06.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f8406d5f700 1 -- 172.21.15.7:0/2208947701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84001547b0 con 0x7f840014de50 2023-03-03T12:48:06.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f83fd7fa700 1 -- 172.21.15.7:0/2208947701 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8012400 con 0x7f840014de50 2023-03-03T12:48:06.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.076+0000 7f8406d5f700 1 -- 172.21.15.7:0/2208947701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84001544d0 con 0x7f840014de50 2023-03-03T12:48:06.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.077+0000 7f8406d5f700 1 -- 172.21.15.7:0/2208947701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8400154a90 con 0x7f840014de50 2023-03-03T12:48:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.078+0000 7f83fd7fa700 1 -- 172.21.15.7:0/2208947701 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f83f8019030 con 0x7f840014de50 2023-03-03T12:48:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.078+0000 7f83fd7fa700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 0x7f83e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:48:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.078+0000 7f83fd7fa700 1 -- 172.21.15.7:0/2208947701 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f83f8085d80 con 0x7f840014de50 2023-03-03T12:48:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.079+0000 7f8404afb700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 0x7f83e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:48:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.079+0000 7f8404afb700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 0x7f83e406a7a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f84001ded10 tx=0x7f83f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:48:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.083+0000 7f83fd7fa700 1 -- 172.21.15.7:0/2208947701 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f83f8052120 con 0x7f840014de50 2023-03-03T12:48:06.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.323+0000 7f8406d5f700 1 -- 172.21.15.7:0/2208947701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8400074bb0 con 0x7f840014de50 2023-03-03T12:48:06.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.324+0000 7f83fd7fa700 1 -- 172.21.15.7:0/2208947701 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f83f8055740 con 0x7f840014de50 2023-03-03T12:48:06.327 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:48:06.327 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:48:06.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 -- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 msgr2=0x7f83e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:06.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 0x7f83e406a7a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f84001ded10 tx=0x7f83f4008040 comp rx=0 tx=0).stop 2023-03-03T12:48:06.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 -- 172.21.15.7:0/2208947701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f840014de50 msgr2=0x7f8400150180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:48:06.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f840014de50 0x7f8400150180 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f83f800caa0 tx=0x7f83f800ce60 comp rx=0 tx=0).stop 2023-03-03T12:48:06.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 -- 172.21.15.7:0/2208947701 shutdown_connections 2023-03-03T12:48:06.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 0x7f83e406a7a0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:06.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f840014de50 0x7f8400150180 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:06.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f840006a7f0 0x7f840006fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:06.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 --2- 172.21.15.7:0/2208947701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84000c65c0 0x7f840014fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:48:06.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 -- 172.21.15.7:0/2208947701 >> 172.21.15.7:0/2208947701 conn(0x7f84000a9430 msgr2=0x7f8400066870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:48:06.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 -- 172.21.15.7:0/2208947701 shutdown_connections 2023-03-03T12:48:06.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:48:06.326+0000 7f8406d5f700 1 -- 172.21.15.7:0/2208947701 wait complete. 2023-03-03T12:48:06.331 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:48:06.341 INFO:tasks.mds_thrash.fs.[a]:waiting for 60 secs before thrashing 2023-03-03T12:48:33.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:48:33.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:03.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:03.606 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:06.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:49:06.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.477+0000 7f4b4cf2a700 1 Processor -- start 2023-03-03T12:49:06.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.477+0000 7f4b4cf2a700 1 -- start start 2023-03-03T12:49:06.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4cf2a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4b480c00e0 0x7f4b480c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:06.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4cf2a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b480c7310 con 0x7f4b480caa30 2023-03-03T12:49:06.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4cf2a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b480bfd10 con 0x7f4b48154ff0 2023-03-03T12:49:06.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4cf2a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b48155430 con 0x7f4b480c00e0 2023-03-03T12:49:06.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4b480caa30 0x7f4b480c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36016/0 (socket says 172.21.15.7:36016) 2023-03-03T12:49:06.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4659c700 1 -- 172.21.15.7:0/1648849515 learned_addr learned my addr 172.21.15.7:0/1648849515 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:49:06.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b45d9b700 1 --2- 172.21.15.7:0/1648849515 >> v2:172.21.15.143:3300/0 conn(0x7f4b480c00e0 0x7f4b480c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:06.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4559a700 1 -- 172.21.15.7:0/1648849515 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475543944 0 0) 0x7f4b480c7310 con 0x7f4b480caa30 2023-03-03T12:49:06.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4559a700 1 -- 172.21.15.7:0/1648849515 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b300035b0 con 0x7f4b480caa30 2023-03-03T12:49:06.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4559a700 1 -- 172.21.15.7:0/1648849515 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172033707 0 0) 0x7f4b480bfd10 con 0x7f4b48154ff0 2023-03-03T12:49:06.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4559a700 1 -- 172.21.15.7:0/1648849515 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b480c7310 con 0x7f4b48154ff0 2023-03-03T12:49:06.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b4559a700 1 -- 172.21.15.7:0/1648849515 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3004882478 0 0) 0x7f4b300035b0 con 0x7f4b480caa30 2023-03-03T12:49:06.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b45d9b700 1 -- 172.21.15.7:0/1648849515 >> v1:172.21.15.143:6789/0 conn(0x7f4b48154ff0 legacy=0x7f4b4815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:06.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b45d9b700 1 -- 172.21.15.7:0/1648849515 >> v1:172.21.15.7:6789/0 conn(0x7f4b480caa30 legacy=0x7f4b480c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:06.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.478+0000 7f4b45d9b700 1 -- 172.21.15.7:0/1648849515 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b480bfa30 con 0x7f4b480c00e0 2023-03-03T12:49:06.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.479+0000 7f4b45d9b700 1 --2- 172.21.15.7:0/1648849515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b480c00e0 0x7f4b480c5c60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009820 tx=0x7f4b3c009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c67c348e6ecf78c7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:49:06.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.479+0000 7f4b4559a700 1 -- 172.21.15.7:0/1648849515 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b38001670 con 0x7f4b480c00e0 2023-03-03T12:49:06.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.479+0000 7f4b4559a700 1 -- 172.21.15.7:0/1648849515 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c005c30 con 0x7f4b480c00e0 2023-03-03T12:49:06.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.479+0000 7f4b4559a700 1 -- 172.21.15.7:0/1648849515 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c004030 con 0x7f4b480c00e0 2023-03-03T12:49:06.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.479+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/1648849515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b480c00e0 msgr2=0x7f4b480c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:06.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.479+0000 7f4b4cf2a700 1 --2- 172.21.15.7:0/1648849515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b480c00e0 0x7f4b480c5c60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c009820 tx=0x7f4b3c009b30 comp rx=0 tx=0).stop 2023-03-03T12:49:06.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.479+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/1648849515 shutdown_connections 2023-03-03T12:49:06.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.479+0000 7f4b4cf2a700 1 --2- 172.21.15.7:0/1648849515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b480c00e0 0x7f4b480c5c60 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:06.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.479+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/1648849515 >> 172.21.15.7:0/1648849515 conn(0x7f4b48066e40 msgr2=0x7f4b48071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:49:06.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.480+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/1648849515 shutdown_connections 2023-03-03T12:49:06.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.480+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/1648849515 wait complete. 2023-03-03T12:49:06.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.481+0000 7f4b4cf2a700 1 Processor -- start 2023-03-03T12:49:06.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.481+0000 7f4b4cf2a700 1 -- start start 2023-03-03T12:49:06.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b4cf2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b480c00e0 0x7f4b48149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:06.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b4cf2a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b480caa30 0x7f4b48157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:06.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b4cf2a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b48154ff0 0x7f4b481578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:06.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b4cf2a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4814ac10 con 0x7f4b48154ff0 2023-03-03T12:49:06.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b4cf2a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4814aef0 con 0x7f4b480c00e0 2023-03-03T12:49:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b4cf2a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b481eda40 con 0x7f4b480caa30 2023-03-03T12:49:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b46d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b48154ff0 0x7f4b481578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b46d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b48154ff0 0x7f4b481578b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42046/0 (socket says 172.21.15.7:42046) 2023-03-03T12:49:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b46d9d700 1 -- 172.21.15.7:0/3792205536 learned_addr learned my addr 172.21.15.7:0/3792205536 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:49:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b4659c700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b480c00e0 0x7f4b48149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b45d9b700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b480caa30 0x7f4b48157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:06.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b46d9d700 1 -- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b480caa30 msgr2=0x7f4b48157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b46d9d700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b480caa30 0x7f4b48157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b46d9d700 1 -- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b480c00e0 msgr2=0x7f4b48149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b46d9d700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b480c00e0 0x7f4b48149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b46d9d700 1 -- 172.21.15.7:0/3792205536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c009430 con 0x7f4b48154ff0 2023-03-03T12:49:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b45d9b700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b480caa30 0x7f4b48157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:49:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b46d9d700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b48154ff0 0x7f4b481578b0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000cc20 tx=0x7f4b4000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:49:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.482+0000 7f4b36ffd700 1 -- 172.21.15.7:0/3792205536 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b40003c00 con 0x7f4b48154ff0 2023-03-03T12:49:06.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.483+0000 7f4b36ffd700 1 -- 172.21.15.7:0/3792205536 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b40003d60 con 0x7f4b48154ff0 2023-03-03T12:49:06.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.483+0000 7f4b36ffd700 1 -- 172.21.15.7:0/3792205536 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4000ab90 con 0x7f4b48154ff0 2023-03-03T12:49:06.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.483+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/3792205536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b481547f0 con 0x7f4b48154ff0 2023-03-03T12:49:06.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.483+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/3792205536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b48154510 con 0x7f4b48154ff0 2023-03-03T12:49:06.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.483+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/3792205536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b48154ad0 con 0x7f4b48154ff0 2023-03-03T12:49:06.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.484+0000 7f4b36ffd700 1 -- 172.21.15.7:0/3792205536 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4b40010050 con 0x7f4b48154ff0 2023-03-03T12:49:06.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.484+0000 7f4b36ffd700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b280682f0 0x7f4b2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.485+0000 7f4b36ffd700 1 -- 172.21.15.7:0/3792205536 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4b40086110 con 0x7f4b48154ff0 2023-03-03T12:49:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.485+0000 7f4b4659c700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b280682f0 0x7f4b2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.485+0000 7f4b4659c700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b280682f0 0x7f4b2806a7a0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4b38000f60 tx=0x7f4b38000bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:49:06.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.489+0000 7f4b36ffd700 1 -- 172.21.15.7:0/3792205536 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b400524b0 con 0x7f4b48154ff0 2023-03-03T12:49:06.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.725+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/3792205536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b48074bb0 con 0x7f4b48154ff0 2023-03-03T12:49:06.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.725+0000 7f4b36ffd700 1 -- 172.21.15.7:0/3792205536 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4b40055ad0 con 0x7f4b48154ff0 2023-03-03T12:49:06.728 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:49:06.728 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:49:06.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b280682f0 msgr2=0x7f4b2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:06.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b280682f0 0x7f4b2806a7a0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4b38000f60 tx=0x7f4b38000bc0 comp rx=0 tx=0).stop 2023-03-03T12:49:06.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/3792205536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b48154ff0 msgr2=0x7f4b481578b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b48154ff0 0x7f4b481578b0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000cc20 tx=0x7f4b4000cf30 comp rx=0 tx=0).stop 2023-03-03T12:49:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/3792205536 shutdown_connections 2023-03-03T12:49:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b280682f0 0x7f4b2806a7a0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b48154ff0 0x7f4b481578b0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b480c00e0 0x7f4b48149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 --2- 172.21.15.7:0/3792205536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b480caa30 0x7f4b48157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:06.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/3792205536 >> 172.21.15.7:0/3792205536 conn(0x7f4b48066e40 msgr2=0x7f4b481539a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:49:06.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/3792205536 shutdown_connections 2023-03-03T12:49:06.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:06.727+0000 7f4b4cf2a700 1 -- 172.21.15.7:0/3792205536 wait complete. 2023-03-03T12:49:06.732 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:49:06.742 INFO:tasks.mds_thrash.fs.[a]:waiting for 95 secs before thrashing 2023-03-03T12:49:07.046 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:49:07.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.168+0000 7f317c3f3700 1 Processor -- start 2023-03-03T12:49:07.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.168+0000 7f317c3f3700 1 -- start start 2023-03-03T12:49:07.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317c3f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3174151e80 0x7f31740c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:07.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317c3f3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3174155410 0x7f31740bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:07.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317c3f3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f31740c6fe0 0x7f31740bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:07.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317c3f3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3174152250 con 0x7f3174155410 2023-03-03T12:49:07.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317c3f3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31740c0fb0 con 0x7f31740c6fe0 2023-03-03T12:49:07.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317c3f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3174154740 con 0x7f3174151e80 2023-03-03T12:49:07.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317998e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3174155410 0x7f31740bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:07.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317998e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3174155410 0x7f31740bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42052/0 (socket says 172.21.15.7:42052) 2023-03-03T12:49:07.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317998e700 1 -- 172.21.15.7:0/2040596973 learned_addr learned my addr 172.21.15.7:0/2040596973 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:49:07.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317a990700 1 --2- 172.21.15.7:0/2040596973 >> v2:172.21.15.143:3300/0 conn(0x7f31740c6fe0 0x7f31740bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:07.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317998e700 1 -- 172.21.15.7:0/2040596973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3174151e80 msgr2=0x7f31740c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:49:07.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317998e700 1 --2- 172.21.15.7:0/2040596973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3174151e80 0x7f31740c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317998e700 1 -- 172.21.15.7:0/2040596973 >> v2:172.21.15.143:3300/0 conn(0x7f31740c6fe0 msgr2=0x7f31740bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:49:07.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317998e700 1 --2- 172.21.15.7:0/2040596973 >> v2:172.21.15.143:3300/0 conn(0x7f31740c6fe0 0x7f31740bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.169+0000 7f317998e700 1 -- 172.21.15.7:0/2040596973 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31740c08d0 con 0x7f3174155410 2023-03-03T12:49:07.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317998e700 1 --2- 172.21.15.7:0/2040596973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174155410 0x7f31740bf8e0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f317408c770 tx=0x7f3170009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b905e9a482478207 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:49:07.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317918d700 1 -- 172.21.15.7:0/2040596973 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317000b870 con 0x7f3174155410 2023-03-03T12:49:07.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317918d700 1 -- 172.21.15.7:0/2040596973 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317000b9d0 con 0x7f3174155410 2023-03-03T12:49:07.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317918d700 1 -- 172.21.15.7:0/2040596973 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317000bd60 con 0x7f3174155410 2023-03-03T12:49:07.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317c3f3700 1 -- 172.21.15.7:0/2040596973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174155410 msgr2=0x7f31740bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:07.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317c3f3700 1 --2- 172.21.15.7:0/2040596973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174155410 0x7f31740bf8e0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f317408c770 tx=0x7f3170009bd0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317c3f3700 1 -- 172.21.15.7:0/2040596973 shutdown_connections 2023-03-03T12:49:07.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317c3f3700 1 --2- 172.21.15.7:0/2040596973 >> v2:172.21.15.143:3300/0 conn(0x7f31740c6fe0 0x7f31740bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317c3f3700 1 --2- 172.21.15.7:0/2040596973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3174151e80 0x7f31740c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317c3f3700 1 --2- 172.21.15.7:0/2040596973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174155410 0x7f31740bf8e0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317c3f3700 1 -- 172.21.15.7:0/2040596973 >> 172.21.15.7:0/2040596973 conn(0x7f3174071630 msgr2=0x7f317406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:49:07.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317c3f3700 1 -- 172.21.15.7:0/2040596973 shutdown_connections 2023-03-03T12:49:07.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.170+0000 7f317c3f3700 1 -- 172.21.15.7:0/2040596973 wait complete. 2023-03-03T12:49:07.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317c3f3700 1 Processor -- start 2023-03-03T12:49:07.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317c3f3700 1 -- start start 2023-03-03T12:49:07.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317c3f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31740c6fe0 0x7f31740c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:07.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317c3f3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174151e80 0x7f3174143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:07.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317c3f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3174155410 0x7f317414c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:07.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317c3f3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317406bfa0 con 0x7f3174151e80 2023-03-03T12:49:07.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317c3f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317414c1d0 con 0x7f31740c6fe0 2023-03-03T12:49:07.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317c3f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3174143c40 con 0x7f3174155410 2023-03-03T12:49:07.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317998e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174151e80 0x7f3174143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:07.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317998e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174151e80 0x7f3174143070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42058/0 (socket says 172.21.15.7:42058) 2023-03-03T12:49:07.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.172+0000 7f317998e700 1 -- 172.21.15.7:0/3646043486 learned_addr learned my addr 172.21.15.7:0/3646043486 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:49:07.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317a18f700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31740c6fe0 0x7f31740c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:07.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317998e700 1 -- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3174155410 msgr2=0x7f317414c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:07.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317a990700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3174155410 0x7f317414c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:07.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317998e700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3174155410 0x7f317414c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317998e700 1 -- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31740c6fe0 msgr2=0x7f31740c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:07.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317998e700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31740c6fe0 0x7f31740c87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317998e700 1 -- 172.21.15.7:0/3646043486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3170009550 con 0x7f3174151e80 2023-03-03T12:49:07.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317a18f700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31740c6fe0 0x7f31740c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:49:07.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317998e700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174151e80 0x7f3174143070 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f31740cb4b0 tx=0x7f3170004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:49:07.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f316affd700 1 -- 172.21.15.7:0/3646043486 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3170004310 con 0x7f3174151e80 2023-03-03T12:49:07.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f316affd700 1 -- 172.21.15.7:0/3646043486 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317001f070 con 0x7f3174151e80 2023-03-03T12:49:07.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317c3f3700 1 -- 172.21.15.7:0/3646043486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3174154c10 con 0x7f3174151e80 2023-03-03T12:49:07.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f316affd700 1 -- 172.21.15.7:0/3646043486 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31700036a0 con 0x7f3174151e80 2023-03-03T12:49:07.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.173+0000 7f317c3f3700 1 -- 172.21.15.7:0/3646043486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3174154930 con 0x7f3174151e80 2023-03-03T12:49:07.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.174+0000 7f317c3f3700 1 -- 172.21.15.7:0/3646043486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3174154ef0 con 0x7f3174151e80 2023-03-03T12:49:07.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.174+0000 7f316affd700 1 -- 172.21.15.7:0/3646043486 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f317002a020 con 0x7f3174151e80 2023-03-03T12:49:07.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.175+0000 7f316affd700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31580682f0 0x7f315806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:49:07.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.175+0000 7f316affd700 1 -- 172.21.15.7:0/3646043486 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f31700877a0 con 0x7f3174151e80 2023-03-03T12:49:07.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.175+0000 7f317a18f700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31580682f0 0x7f315806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:49:07.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.176+0000 7f317a18f700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31580682f0 0x7f315806a7a0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f316400a8f0 tx=0x7f3164008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:49:07.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.179+0000 7f316affd700 1 -- 172.21.15.7:0/3646043486 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3170053b40 con 0x7f3174151e80 2023-03-03T12:49:07.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.418+0000 7f317c3f3700 1 -- 172.21.15.7:0/3646043486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3174074bb0 con 0x7f3174151e80 2023-03-03T12:49:07.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.419+0000 7f316affd700 1 -- 172.21.15.7:0/3646043486 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3170057160 con 0x7f3174151e80 2023-03-03T12:49:07.422 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:49:07.422 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:49:07.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 -- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31580682f0 msgr2=0x7f315806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:07.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31580682f0 0x7f315806a7a0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f316400a8f0 tx=0x7f3164008040 comp rx=0 tx=0).stop 2023-03-03T12:49:07.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 -- 172.21.15.7:0/3646043486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174151e80 msgr2=0x7f3174143070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:49:07.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174151e80 0x7f3174143070 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f31740cb4b0 tx=0x7f3170004750 comp rx=0 tx=0).stop 2023-03-03T12:49:07.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 -- 172.21.15.7:0/3646043486 shutdown_connections 2023-03-03T12:49:07.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3174155410 0x7f317414c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31740c6fe0 0x7f31740c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31580682f0 0x7f315806a7a0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 --2- 172.21.15.7:0/3646043486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3174151e80 0x7f3174143070 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:49:07.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 -- 172.21.15.7:0/3646043486 >> 172.21.15.7:0/3646043486 conn(0x7f3174071630 msgr2=0x7f31740c3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:49:07.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 -- 172.21.15.7:0/3646043486 shutdown_connections 2023-03-03T12:49:07.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:49:07.421+0000 7f317c3f3700 1 -- 172.21.15.7:0/3646043486 wait complete. 2023-03-03T12:49:07.426 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:49:07.435 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-03-03T12:49:33.752 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:49:33.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T12:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T12:50:03.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:03.832 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:07.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:50:07.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.574+0000 7fefeca55700 1 Processor -- start 2023-03-03T12:50:07.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.574+0000 7fefeca55700 1 -- start start 2023-03-03T12:50:07.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.575+0000 7fefeca55700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fefe8151b20 0x7fefe80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:07.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.575+0000 7fefeca55700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 0x7fefe81581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:07.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.575+0000 7fefeca55700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe806cb70 con 0x7fefe80c2950 2023-03-03T12:50:07.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.575+0000 7fefeca55700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe806c2f0 con 0x7fefe8151b20 2023-03-03T12:50:07.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.575+0000 7fefeca55700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe806ce50 con 0x7fefe8073c30 2023-03-03T12:50:07.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 0x7fefe81581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:07.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe659c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fefe8151b20 0x7fefe80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:07.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe6d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 0x7fefe81581f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:50202/0 (socket says 172.21.15.7:50202) 2023-03-03T12:50:07.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe6d9d700 1 -- 172.21.15.7:0/1452426383 learned_addr learned my addr 172.21.15.7:0/1452426383 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:50:07.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe5d9b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fefe80c2950 0x7fefe8157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:39630/0 (socket says 172.21.15.7:39630) 2023-03-03T12:50:07.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe6d9d700 1 -- 172.21.15.7:0/1452426383 >> v2:172.21.15.143:3300/0 conn(0x7fefe8151b20 msgr2=0x7fefe80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:50:07.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe6d9d700 1 --2- 172.21.15.7:0/1452426383 >> v2:172.21.15.143:3300/0 conn(0x7fefe8151b20 0x7fefe80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:07.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe6d9d700 1 -- 172.21.15.7:0/1452426383 >> v1:172.21.15.143:6789/0 conn(0x7fefe80c2950 legacy=0x7fefe8157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:07.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe6d9d700 1 -- 172.21.15.7:0/1452426383 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe806d130 con 0x7fefe8073c30 2023-03-03T12:50:07.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe6d9d700 1 --2- 172.21.15.7:0/1452426383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 0x7fefe81581f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009e50 tx=0x7fefe000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a28e47034b79c3cc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:50:07.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe559a700 1 -- 172.21.15.7:0/1452426383 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe000e070 con 0x7fefe8073c30 2023-03-03T12:50:07.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe559a700 1 -- 172.21.15.7:0/1452426383 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe000b910 con 0x7fefe8073c30 2023-03-03T12:50:07.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.576+0000 7fefe559a700 1 -- 172.21.15.7:0/1452426383 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe000bcc0 con 0x7fefe8073c30 2023-03-03T12:50:07.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.577+0000 7fefeca55700 1 -- 172.21.15.7:0/1452426383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 msgr2=0x7fefe81581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:07.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.577+0000 7fefeca55700 1 --2- 172.21.15.7:0/1452426383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 0x7fefe81581f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009e50 tx=0x7fefe000b330 comp rx=0 tx=0).stop 2023-03-03T12:50:07.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.577+0000 7fefeca55700 1 -- 172.21.15.7:0/1452426383 shutdown_connections 2023-03-03T12:50:07.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.577+0000 7fefeca55700 1 --2- 172.21.15.7:0/1452426383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 0x7fefe81581f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:07.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.577+0000 7fefeca55700 1 --2- 172.21.15.7:0/1452426383 >> v2:172.21.15.143:3300/0 conn(0x7fefe8151b20 0x7fefe80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:07.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.577+0000 7fefeca55700 1 -- 172.21.15.7:0/1452426383 >> 172.21.15.7:0/1452426383 conn(0x7fefe80a94d0 msgr2=0x7fefe80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:50:07.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.577+0000 7fefeca55700 1 -- 172.21.15.7:0/1452426383 shutdown_connections 2023-03-03T12:50:07.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.577+0000 7fefeca55700 1 -- 172.21.15.7:0/1452426383 wait complete. 2023-03-03T12:50:07.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.578+0000 7fefeca55700 1 Processor -- start 2023-03-03T12:50:07.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.578+0000 7fefeca55700 1 -- start start 2023-03-03T12:50:07.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefeca55700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 0x7fefe8156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:07.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefeca55700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefe80c2950 0x7fefe8148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:07.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefeca55700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe8151b20 0x7fefe8148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:07.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefeca55700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe8149180 con 0x7fefe80c2950 2023-03-03T12:50:07.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefeca55700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe814c710 con 0x7fefe8151b20 2023-03-03T12:50:07.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefeca55700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe81dfaa0 con 0x7fefe8073c30 2023-03-03T12:50:07.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefe5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefe80c2950 0x7fefe8148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefe5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefe80c2950 0x7fefe8148700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42136/0 (socket says 172.21.15.7:42136) 2023-03-03T12:50:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefe5d9b700 1 -- 172.21.15.7:0/1197543827 learned_addr learned my addr 172.21.15.7:0/1197543827 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:50:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefe659c700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 0x7fefe8156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.579+0000 7fefe6d9d700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe8151b20 0x7fefe8148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefe5d9b700 1 -- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 msgr2=0x7fefe8156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefe5d9b700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 0x7fefe8156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefe5d9b700 1 -- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe8151b20 msgr2=0x7fefe8148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefe5d9b700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe8151b20 0x7fefe8148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefe5d9b700 1 -- 172.21.15.7:0/1197543827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe0009b30 con 0x7fefe80c2950 2023-03-03T12:50:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefe5d9b700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefe80c2950 0x7fefe8148700 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fefd000c530 tx=0x7fefd000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:50:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefdeffd700 1 -- 172.21.15.7:0/1197543827 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd000cea0 con 0x7fefe80c2950 2023-03-03T12:50:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefdeffd700 1 -- 172.21.15.7:0/1197543827 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd0005cb0 con 0x7fefe80c2950 2023-03-03T12:50:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefdeffd700 1 -- 172.21.15.7:0/1197543827 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00072d0 con 0x7fefe80c2950 2023-03-03T12:50:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefeca55700 1 -- 172.21.15.7:0/1197543827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe8158730 con 0x7fefe80c2950 2023-03-03T12:50:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.580+0000 7fefeca55700 1 -- 172.21.15.7:0/1197543827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe806d410 con 0x7fefe80c2950 2023-03-03T12:50:07.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.581+0000 7fefeca55700 1 -- 172.21.15.7:0/1197543827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe8151ef0 con 0x7fefe80c2950 2023-03-03T12:50:07.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.582+0000 7fefdeffd700 1 -- 172.21.15.7:0/1197543827 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fefd0019030 con 0x7fefe80c2950 2023-03-03T12:50:07.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.582+0000 7fefdeffd700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefc80682f0 0x7fefc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:07.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.582+0000 7fefdeffd700 1 -- 172.21.15.7:0/1197543827 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fefd0085190 con 0x7fefe80c2950 2023-03-03T12:50:07.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.582+0000 7fefe659c700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefc80682f0 0x7fefc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:07.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.583+0000 7fefe659c700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefc80682f0 0x7fefc806a7a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fefd8009b40 tx=0x7fefd8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:50:07.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.586+0000 7fefdeffd700 1 -- 172.21.15.7:0/1197543827 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fefd004b5e0 con 0x7fefe80c2950 2023-03-03T12:50:07.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.829+0000 7fefeca55700 1 -- 172.21.15.7:0/1197543827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefe80c54a0 con 0x7fefe80c2950 2023-03-03T12:50:07.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.830+0000 7fefdeffd700 1 -- 172.21.15.7:0/1197543827 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fefd0017090 con 0x7fefe80c2950 2023-03-03T12:50:07.832 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:50:07.832 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:50:07.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 -- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefc80682f0 msgr2=0x7fefc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:07.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefc80682f0 0x7fefc806a7a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fefd8009b40 tx=0x7fefd8009210 comp rx=0 tx=0).stop 2023-03-03T12:50:07.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 -- 172.21.15.7:0/1197543827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefe80c2950 msgr2=0x7fefe8148700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefe80c2950 0x7fefe8148700 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fefd000c530 tx=0x7fefd000c840 comp rx=0 tx=0).stop 2023-03-03T12:50:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 -- 172.21.15.7:0/1197543827 shutdown_connections 2023-03-03T12:50:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefe8151b20 0x7fefe8148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fefe8073c30 0x7fefe8156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefc80682f0 0x7fefc806a7a0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 --2- 172.21.15.7:0/1197543827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefe80c2950 0x7fefe8148700 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:07.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 -- 172.21.15.7:0/1197543827 >> 172.21.15.7:0/1197543827 conn(0x7fefe80a94d0 msgr2=0x7fefe806c430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:50:07.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 -- 172.21.15.7:0/1197543827 shutdown_connections 2023-03-03T12:50:07.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:07.832+0000 7fefeca55700 1 -- 172.21.15.7:0/1197543827 wait complete. 2023-03-03T12:50:07.836 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:50:07.846 INFO:tasks.mds_thrash.fs.[b]:waiting for 20 secs before thrashing 2023-03-03T12:50:27.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:50:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.985+0000 7f7afbf07700 1 Processor -- start 2023-03-03T12:50:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.986+0000 7f7afbf07700 1 -- start start 2023-03-03T12:50:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.986+0000 7f7afbf07700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7af4071c30 0x7f7af40c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.986+0000 7f7afbf07700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7af4155950 0x7f7af4155d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.986+0000 7f7afbf07700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af40713c0 con 0x7f7af40aac00 2023-03-03T12:50:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.986+0000 7f7afbf07700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af41562d0 con 0x7f7af4071c30 2023-03-03T12:50:27.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.986+0000 7f7afbf07700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af4152c70 con 0x7f7af4155950 2023-03-03T12:50:27.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.986+0000 7f7af94a2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7af4071c30 0x7f7af40c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:27.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.986+0000 7f7af9ca3700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7af40aac00 0x7f7af40beb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36126/0 (socket says 172.21.15.7:36126) 2023-03-03T12:50:27.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.986+0000 7f7af9ca3700 1 -- 172.21.15.7:0/3884760612 learned_addr learned my addr 172.21.15.7:0/3884760612 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:50:27.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.987+0000 7f7afa4a4700 1 --2- 172.21.15.7:0/3884760612 >> v2:172.21.15.143:3300/0 conn(0x7f7af4155950 0x7f7af4155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:27.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.987+0000 7f7af94a2700 1 -- 172.21.15.7:0/3884760612 >> v2:172.21.15.143:3300/0 conn(0x7f7af4155950 msgr2=0x7f7af4155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:50:27.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.987+0000 7f7af94a2700 1 --2- 172.21.15.7:0/3884760612 >> v2:172.21.15.143:3300/0 conn(0x7f7af4155950 0x7f7af4155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:27.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.987+0000 7f7af94a2700 1 -- 172.21.15.7:0/3884760612 >> v1:172.21.15.7:6789/0 conn(0x7f7af40aac00 legacy=0x7f7af40beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.987+0000 7f7af94a2700 1 -- 172.21.15.7:0/3884760612 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af40c4930 con 0x7f7af4071c30 2023-03-03T12:50:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.987+0000 7f7af8ca1700 1 -- 172.21.15.7:0/3884760612 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590868062 0 0) 0x7f7af40713c0 con 0x7f7af40aac00 2023-03-03T12:50:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.987+0000 7f7af94a2700 1 --2- 172.21.15.7:0/3884760612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4071c30 0x7f7af40c43f0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f7af4071720 tx=0x7f7ae4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a132f593af1186 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:50:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.987+0000 7f7af8ca1700 1 -- 172.21.15.7:0/3884760612 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4011830 con 0x7f7af4071c30 2023-03-03T12:50:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.987+0000 7f7af8ca1700 1 -- 172.21.15.7:0/3884760612 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae4011990 con 0x7f7af4071c30 2023-03-03T12:50:27.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.987+0000 7f7af8ca1700 1 -- 172.21.15.7:0/3884760612 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4011d20 con 0x7f7af4071c30 2023-03-03T12:50:27.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.988+0000 7f7afbf07700 1 -- 172.21.15.7:0/3884760612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4071c30 msgr2=0x7f7af40c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:27.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.988+0000 7f7afbf07700 1 --2- 172.21.15.7:0/3884760612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4071c30 0x7f7af40c43f0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f7af4071720 tx=0x7f7ae4009b20 comp rx=0 tx=0).stop 2023-03-03T12:50:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.988+0000 7f7afbf07700 1 -- 172.21.15.7:0/3884760612 shutdown_connections 2023-03-03T12:50:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.988+0000 7f7afbf07700 1 --2- 172.21.15.7:0/3884760612 >> v2:172.21.15.143:3300/0 conn(0x7f7af4155950 0x7f7af4155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.988+0000 7f7afbf07700 1 --2- 172.21.15.7:0/3884760612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4071c30 0x7f7af40c43f0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.988+0000 7f7afbf07700 1 -- 172.21.15.7:0/3884760612 >> 172.21.15.7:0/3884760612 conn(0x7f7af40736f0 msgr2=0x7f7af4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:50:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.988+0000 7f7afbf07700 1 -- 172.21.15.7:0/3884760612 shutdown_connections 2023-03-03T12:50:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.988+0000 7f7afbf07700 1 -- 172.21.15.7:0/3884760612 wait complete. 2023-03-03T12:50:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.989+0000 7f7afbf07700 1 Processor -- start 2023-03-03T12:50:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.990+0000 7f7afbf07700 1 -- start start 2023-03-03T12:50:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.990+0000 7f7afbf07700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7af4071c30 0x7f7af414e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.990+0000 7f7afbf07700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7af40aac00 0x7f7af41eb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.990+0000 7f7afbf07700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4155950 0x7f7af414d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.990+0000 7f7afbf07700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af41eb9b0 con 0x7f7af4155950 2023-03-03T12:50:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.990+0000 7f7afbf07700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af414cbc0 con 0x7f7af40aac00 2023-03-03T12:50:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.990+0000 7f7afbf07700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af40c7c60 con 0x7f7af4071c30 2023-03-03T12:50:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afa4a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4155950 0x7f7af414d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afa4a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4155950 0x7f7af414d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42156/0 (socket says 172.21.15.7:42156) 2023-03-03T12:50:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afa4a4700 1 -- 172.21.15.7:0/3011648661 learned_addr learned my addr 172.21.15.7:0/3011648661 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:50:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7af94a2700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7af40aac00 0x7f7af41eb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7af9ca3700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7af4071c30 0x7f7af414e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afa4a4700 1 -- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7af4071c30 msgr2=0x7f7af414e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afa4a4700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7af4071c30 0x7f7af414e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afa4a4700 1 -- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7af40aac00 msgr2=0x7f7af41eb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:27.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afa4a4700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7af40aac00 0x7f7af41eb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afa4a4700 1 -- 172.21.15.7:0/3011648661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae40094a0 con 0x7f7af4155950 2023-03-03T12:50:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7af94a2700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7af40aac00 0x7f7af41eb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:50:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afa4a4700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4155950 0x7f7af414d380 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f7aec007e60 tx=0x7f7aec00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:50:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7aea7fc700 1 -- 172.21.15.7:0/3011648661 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00ede0 con 0x7f7af4155950 2023-03-03T12:50:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7aea7fc700 1 -- 172.21.15.7:0/3011648661 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec014420 con 0x7f7af4155950 2023-03-03T12:50:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afbf07700 1 -- 172.21.15.7:0/3011648661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af40c4b80 con 0x7f7af4155950 2023-03-03T12:50:27.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7aea7fc700 1 -- 172.21.15.7:0/3011648661 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec0147d0 con 0x7f7af4155950 2023-03-03T12:50:27.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.991+0000 7f7afbf07700 1 -- 172.21.15.7:0/3011648661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af4152360 con 0x7f7af4155950 2023-03-03T12:50:27.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.992+0000 7f7afbf07700 1 -- 172.21.15.7:0/3011648661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af4152620 con 0x7f7af4155950 2023-03-03T12:50:27.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.993+0000 7f7aea7fc700 1 -- 172.21.15.7:0/3011648661 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7aec012050 con 0x7f7af4155950 2023-03-03T12:50:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.993+0000 7f7aea7fc700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ad8068320 0x7f7ad806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.993+0000 7f7aea7fc700 1 -- 172.21.15.7:0/3011648661 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7aec086750 con 0x7f7af4155950 2023-03-03T12:50:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.993+0000 7f7af9ca3700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ad8068320 0x7f7ad806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.994+0000 7f7af9ca3700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ad8068320 0x7f7ad806a7d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7af0000f60 tx=0x7f7af0007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:50:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:27.998+0000 7f7aea7fc700 1 -- 172.21.15.7:0/3011648661 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7aec052af0 con 0x7f7af4155950 2023-03-03T12:50:28.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.234+0000 7f7afbf07700 1 -- 172.21.15.7:0/3011648661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7af414ca20 con 0x7f7af4155950 2023-03-03T12:50:28.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.234+0000 7f7aea7fc700 1 -- 172.21.15.7:0/3011648661 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7af414ca20 con 0x7f7af4155950 2023-03-03T12:50:28.236 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:50:28.237 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:50:28.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.236+0000 7f7afbf07700 1 -- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ad8068320 msgr2=0x7f7ad806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:28.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.236+0000 7f7afbf07700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ad8068320 0x7f7ad806a7d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7af0000f60 tx=0x7f7af0007be0 comp rx=0 tx=0).stop 2023-03-03T12:50:28.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.236+0000 7f7afbf07700 1 -- 172.21.15.7:0/3011648661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4155950 msgr2=0x7f7af414d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:28.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.236+0000 7f7afbf07700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4155950 0x7f7af414d380 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f7aec007e60 tx=0x7f7aec00e3c0 comp rx=0 tx=0).stop 2023-03-03T12:50:28.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.236+0000 7f7afbf07700 1 -- 172.21.15.7:0/3011648661 shutdown_connections 2023-03-03T12:50:28.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.236+0000 7f7afbf07700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ad8068320 0x7f7ad806a7d0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:28.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.236+0000 7f7afbf07700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7af4155950 0x7f7af414d380 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:28.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.237+0000 7f7afbf07700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7af4071c30 0x7f7af414e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:28.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.237+0000 7f7afbf07700 1 --2- 172.21.15.7:0/3011648661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7af40aac00 0x7f7af41eb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:28.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.237+0000 7f7afbf07700 1 -- 172.21.15.7:0/3011648661 >> 172.21.15.7:0/3011648661 conn(0x7f7af40736f0 msgr2=0x7f7af414c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:50:28.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.237+0000 7f7afbf07700 1 -- 172.21.15.7:0/3011648661 shutdown_connections 2023-03-03T12:50:28.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:28.237+0000 7f7afbf07700 1 -- 172.21.15.7:0/3011648661 wait complete. 2023-03-03T12:50:28.240 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:50:28.250 INFO:tasks.mds_thrash.fs.[b]:waiting for 89 secs before thrashing 2023-03-03T12:50:33.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:33.896 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:50:41.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:50:41.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.880+0000 7f6f0c87f700 1 Processor -- start 2023-03-03T12:50:41.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.880+0000 7f6f0c87f700 1 -- start start 2023-03-03T12:50:41.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.881+0000 7f6f0c87f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f0814de50 0x7f6f0806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:41.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.881+0000 7f6f0c87f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6f0806a7f0 0x7f6f080c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:41.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.881+0000 7f6f0c87f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080a65b0 con 0x7f6f080c65c0 2023-03-03T12:50:41.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.881+0000 7f6f0c87f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f08154e00 con 0x7f6f0806a7f0 2023-03-03T12:50:41.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.881+0000 7f6f0c87f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f080c3370 con 0x7f6f0814de50 2023-03-03T12:50:41.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.881+0000 7f6f05d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6f080c65c0 0x7f6f080c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36146/0 (socket says 172.21.15.7:36146) 2023-03-03T12:50:41.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.881+0000 7f6f05d9b700 1 -- 172.21.15.7:0/2084895435 learned_addr learned my addr 172.21.15.7:0/2084895435 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:50:41.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f06d9d700 1 --2- 172.21.15.7:0/2084895435 >> v2:172.21.15.143:3300/0 conn(0x7f6f0806a7f0 0x7f6f080c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:41.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f0659c700 1 --2- 172.21.15.7:0/2084895435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f0814de50 0x7f6f0806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:41.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f0559a700 1 -- 172.21.15.7:0/2084895435 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533485320 0 0) 0x7f6f080a65b0 con 0x7f6f080c65c0 2023-03-03T12:50:41.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f0559a700 1 -- 172.21.15.7:0/2084895435 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef00035b0 con 0x7f6f080c65c0 2023-03-03T12:50:41.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f0659c700 1 -- 172.21.15.7:0/2084895435 >> v2:172.21.15.143:3300/0 conn(0x7f6f0806a7f0 msgr2=0x7f6f080c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:50:41.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f0659c700 1 --2- 172.21.15.7:0/2084895435 >> v2:172.21.15.143:3300/0 conn(0x7f6f0806a7f0 0x7f6f080c2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:41.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f0659c700 1 -- 172.21.15.7:0/2084895435 >> v1:172.21.15.7:6789/0 conn(0x7f6f080c65c0 legacy=0x7f6f080c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:41.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f0559a700 1 -- 172.21.15.7:0/2084895435 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3845601218 0 0) 0x7f6ef00035b0 con 0x7f6f080c65c0 2023-03-03T12:50:41.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f0659c700 1 -- 172.21.15.7:0/2084895435 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f081541f0 con 0x7f6f0814de50 2023-03-03T12:50:41.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f06d9d700 1 --2- 172.21.15.7:0/2084895435 >> v2:172.21.15.143:3300/0 conn(0x7f6f0806a7f0 0x7f6f080c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:50:41.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.882+0000 7f6f0659c700 1 --2- 172.21.15.7:0/2084895435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f0814de50 0x7f6f0806a2b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009840 tx=0x7f6efc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d60ef06c855c70ec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:50:41.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0559a700 1 -- 172.21.15.7:0/2084895435 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8001670 con 0x7f6f0814de50 2023-03-03T12:50:41.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0559a700 1 -- 172.21.15.7:0/2084895435 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc004030 con 0x7f6f0814de50 2023-03-03T12:50:41.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0559a700 1 -- 172.21.15.7:0/2084895435 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc0043c0 con 0x7f6f0814de50 2023-03-03T12:50:41.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0c87f700 1 -- 172.21.15.7:0/2084895435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f0814de50 msgr2=0x7f6f0806a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0c87f700 1 --2- 172.21.15.7:0/2084895435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f0814de50 0x7f6f0806a2b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009840 tx=0x7f6efc009b50 comp rx=0 tx=0).stop 2023-03-03T12:50:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0c87f700 1 -- 172.21.15.7:0/2084895435 shutdown_connections 2023-03-03T12:50:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0c87f700 1 --2- 172.21.15.7:0/2084895435 >> v2:172.21.15.143:3300/0 conn(0x7f6f0806a7f0 0x7f6f080c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0c87f700 1 --2- 172.21.15.7:0/2084895435 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f0814de50 0x7f6f0806a2b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0c87f700 1 -- 172.21.15.7:0/2084895435 >> 172.21.15.7:0/2084895435 conn(0x7f6f080a9430 msgr2=0x7f6f080c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:50:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0c87f700 1 -- 172.21.15.7:0/2084895435 shutdown_connections 2023-03-03T12:50:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.883+0000 7f6f0c87f700 1 -- 172.21.15.7:0/2084895435 wait complete. 2023-03-03T12:50:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.885+0000 7f6f0c87f700 1 Processor -- start 2023-03-03T12:50:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.885+0000 7f6f0c87f700 1 -- start start 2023-03-03T12:50:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.885+0000 7f6f0c87f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f0806a7f0 0x7f6f08152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.885+0000 7f6f0c87f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f080c65c0 0x7f6f08148c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.885+0000 7f6f0c87f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f0814de50 0x7f6f08149140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.885+0000 7f6f0c87f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f08149680 con 0x7f6f0814de50 2023-03-03T12:50:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.885+0000 7f6f0c87f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f081505b0 con 0x7f6f0806a7f0 2023-03-03T12:50:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.885+0000 7f6f0c87f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f08151130 con 0x7f6f080c65c0 2023-03-03T12:50:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f06d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f0814de50 0x7f6f08149140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f06d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f0814de50 0x7f6f08149140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42176/0 (socket says 172.21.15.7:42176) 2023-03-03T12:50:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f06d9d700 1 -- 172.21.15.7:0/3375949797 learned_addr learned my addr 172.21.15.7:0/3375949797 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:50:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f05d9b700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f080c65c0 0x7f6f08148c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f0659c700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f0806a7f0 0x7f6f08152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f06d9d700 1 -- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f080c65c0 msgr2=0x7f6f08148c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f06d9d700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f080c65c0 0x7f6f08148c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f06d9d700 1 -- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f0806a7f0 msgr2=0x7f6f08152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f06d9d700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f0806a7f0 0x7f6f08152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f06d9d700 1 -- 172.21.15.7:0/3375949797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00009a70 con 0x7f6f0814de50 2023-03-03T12:50:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f06d9d700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f0814de50 0x7f6f08149140 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000efc0 tx=0x7f6f0000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:50:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6ef6ffd700 1 -- 172.21.15.7:0/3375949797 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0000ca90 con 0x7f6f0814de50 2023-03-03T12:50:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6ef6ffd700 1 -- 172.21.15.7:0/3375949797 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0000f040 con 0x7f6f0814de50 2023-03-03T12:50:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6ef6ffd700 1 -- 172.21.15.7:0/3375949797 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0000ce30 con 0x7f6f0814de50 2023-03-03T12:50:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f0c87f700 1 -- 172.21.15.7:0/3375949797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6efc0094a0 con 0x7f6f0814de50 2023-03-03T12:50:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.886+0000 7f6f0c87f700 1 -- 172.21.15.7:0/3375949797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f081547b0 con 0x7f6f0814de50 2023-03-03T12:50:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.887+0000 7f6f0c87f700 1 -- 172.21.15.7:0/3375949797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f081544d0 con 0x7f6f0814de50 2023-03-03T12:50:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.888+0000 7f6ef6ffd700 1 -- 172.21.15.7:0/3375949797 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6f00014480 con 0x7f6f0814de50 2023-03-03T12:50:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.888+0000 7f6ef6ffd700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee80682f0 0x7f6ee806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:50:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.888+0000 7f6ef6ffd700 1 -- 172.21.15.7:0/3375949797 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6f000868e0 con 0x7f6f0814de50 2023-03-03T12:50:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.888+0000 7f6f0659c700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee80682f0 0x7f6ee806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:50:41.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.889+0000 7f6f0659c700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee80682f0 0x7f6ee806a7a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0090b0 tx=0x7f6efc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:50:41.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:41.893+0000 7f6ef6ffd700 1 -- 172.21.15.7:0/3375949797 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6f00052c20 con 0x7f6f0814de50 2023-03-03T12:50:42.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.130+0000 7f6f0c87f700 1 -- 172.21.15.7:0/3375949797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f08154a90 con 0x7f6f0814de50 2023-03-03T12:50:42.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.131+0000 7f6ef6ffd700 1 -- 172.21.15.7:0/3375949797 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6f00009cd0 con 0x7f6f0814de50 2023-03-03T12:50:42.133 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:50:42.133 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:50:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 -- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee80682f0 msgr2=0x7f6ee806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee80682f0 0x7f6ee806a7a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0090b0 tx=0x7f6efc008030 comp rx=0 tx=0).stop 2023-03-03T12:50:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 -- 172.21.15.7:0/3375949797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f0814de50 msgr2=0x7f6f08149140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:50:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f0814de50 0x7f6f08149140 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000efc0 tx=0x7f6f0000c3a0 comp rx=0 tx=0).stop 2023-03-03T12:50:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 -- 172.21.15.7:0/3375949797 shutdown_connections 2023-03-03T12:50:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee80682f0 0x7f6ee806a7a0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f0814de50 0x7f6f08149140 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f0806a7f0 0x7f6f08152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 --2- 172.21.15.7:0/3375949797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f080c65c0 0x7f6f08148c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:50:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 -- 172.21.15.7:0/3375949797 >> 172.21.15.7:0/3375949797 conn(0x7f6f080a9430 msgr2=0x7f6f080a76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:50:42.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 -- 172.21.15.7:0/3375949797 shutdown_connections 2023-03-03T12:50:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:50:42.133+0000 7f6f0c87f700 1 -- 172.21.15.7:0/3375949797 wait complete. 2023-03-03T12:50:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:50:42.147 INFO:tasks.mds_thrash.fs.[a]:waiting for 47 secs before thrashing 2023-03-03T12:51:04.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:04.025 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:29.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:51:29.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.285+0000 7f8925686700 1 Processor -- start 2023-03-03T12:51:29.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.285+0000 7f8925686700 1 -- start start 2023-03-03T12:51:29.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.285+0000 7f8925686700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89200c00e0 0x7f89200c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.285+0000 7f8925686700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89200c7310 con 0x7f8920154ff0 2023-03-03T12:51:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.285+0000 7f8925686700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89200bfd10 con 0x7f89200caa30 2023-03-03T12:51:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.285+0000 7f8925686700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8920155430 con 0x7f89200c00e0 2023-03-03T12:51:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.286+0000 7f891e7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89200c00e0 0x7f89200c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.286+0000 7f891f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8920154ff0 0x7f892015ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36204/0 (socket says 172.21.15.7:36204) 2023-03-03T12:51:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.286+0000 7f891f7fe700 1 -- 172.21.15.7:0/1368722431 learned_addr learned my addr 172.21.15.7:0/1368722431 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:51:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.286+0000 7f891e7fc700 1 -- 172.21.15.7:0/1368722431 >> v1:172.21.15.143:6789/0 conn(0x7f89200caa30 legacy=0x7f89200c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:29.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.286+0000 7f891e7fc700 1 -- 172.21.15.7:0/1368722431 >> v1:172.21.15.7:6789/0 conn(0x7f8920154ff0 legacy=0x7f892015ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:29.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.286+0000 7f891e7fc700 1 -- 172.21.15.7:0/1368722431 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200bfa30 con 0x7f89200c00e0 2023-03-03T12:51:29.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.286+0000 7f891e7fc700 1 --2- 172.21.15.7:0/1368722431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 0x7f89200c5c60 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f8908009820 tx=0x7f8908009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1189f29829043b94 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:51:29.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.286+0000 7f891dffb700 1 -- 172.21.15.7:0/1368722431 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908005e30 con 0x7f89200c00e0 2023-03-03T12:51:29.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.286+0000 7f891dffb700 1 -- 172.21.15.7:0/1368722431 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8908006030 con 0x7f89200c00e0 2023-03-03T12:51:29.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.286+0000 7f891dffb700 1 -- 172.21.15.7:0/1368722431 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89080063e0 con 0x7f89200c00e0 2023-03-03T12:51:29.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.287+0000 7f8925686700 1 -- 172.21.15.7:0/1368722431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 msgr2=0x7f89200c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.287+0000 7f8925686700 1 --2- 172.21.15.7:0/1368722431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 0x7f89200c5c60 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f8908009820 tx=0x7f8908009b30 comp rx=0 tx=0).stop 2023-03-03T12:51:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.287+0000 7f8925686700 1 -- 172.21.15.7:0/1368722431 shutdown_connections 2023-03-03T12:51:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.287+0000 7f8925686700 1 --2- 172.21.15.7:0/1368722431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 0x7f89200c5c60 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.287+0000 7f8925686700 1 -- 172.21.15.7:0/1368722431 >> 172.21.15.7:0/1368722431 conn(0x7f8920066e40 msgr2=0x7f8920071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:51:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.287+0000 7f8925686700 1 -- 172.21.15.7:0/1368722431 shutdown_connections 2023-03-03T12:51:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.287+0000 7f8925686700 1 -- 172.21.15.7:0/1368722431 wait complete. 2023-03-03T12:51:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.288+0000 7f8925686700 1 Processor -- start 2023-03-03T12:51:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.288+0000 7f8925686700 1 -- start start 2023-03-03T12:51:29.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f8925686700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 0x7f8920148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:29.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f8925686700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89200caa30 0x7f89201ed200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:29.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f8925686700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8920154ff0 0x7f8920147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:29.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f8925686700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89200c1fa0 con 0x7f89200c00e0 2023-03-03T12:51:29.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f8925686700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89201493f0 con 0x7f8920154ff0 2023-03-03T12:51:29.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f8925686700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8920148bd0 con 0x7f89200caa30 2023-03-03T12:51:29.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 0x7f8920148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:29.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 0x7f8920148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42226/0 (socket says 172.21.15.7:42226) 2023-03-03T12:51:29.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891effd700 1 -- 172.21.15.7:0/4113012975 learned_addr learned my addr 172.21.15.7:0/4113012975 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:51:29.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891f7fe700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8920154ff0 0x7f8920147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:29.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891e7fc700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89200caa30 0x7f89201ed200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:29.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891effd700 1 -- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89200caa30 msgr2=0x7f89201ed200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:29.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891effd700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89200caa30 0x7f89201ed200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:29.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891effd700 1 -- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8920154ff0 msgr2=0x7f8920147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:29.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891effd700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8920154ff0 0x7f8920147ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:29.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891effd700 1 -- 172.21.15.7:0/4113012975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8908009430 con 0x7f89200c00e0 2023-03-03T12:51:29.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891f7fe700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8920154ff0 0x7f8920147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:51:29.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891e7fc700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89200caa30 0x7f89201ed200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:51:29.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.289+0000 7f891effd700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 0x7f8920148eb0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f891000c700 tx=0x7f891000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:51:29.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.290+0000 7f89177fe700 1 -- 172.21.15.7:0/4113012975 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89100052d0 con 0x7f89200c00e0 2023-03-03T12:51:29.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.290+0000 7f89177fe700 1 -- 172.21.15.7:0/4113012975 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8910005430 con 0x7f89200c00e0 2023-03-03T12:51:29.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.290+0000 7f8925686700 1 -- 172.21.15.7:0/4113012975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89201547f0 con 0x7f89200c00e0 2023-03-03T12:51:29.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.290+0000 7f89177fe700 1 -- 172.21.15.7:0/4113012975 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8910003880 con 0x7f89200c00e0 2023-03-03T12:51:29.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.290+0000 7f8925686700 1 -- 172.21.15.7:0/4113012975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8920154510 con 0x7f89200c00e0 2023-03-03T12:51:29.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.291+0000 7f8925686700 1 -- 172.21.15.7:0/4113012975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8920154ad0 con 0x7f89200c00e0 2023-03-03T12:51:29.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.291+0000 7f89177fe700 1 -- 172.21.15.7:0/4113012975 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8910019030 con 0x7f89200c00e0 2023-03-03T12:51:29.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.291+0000 7f89177fe700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89000682f0 0x7f890006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:29.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.292+0000 7f89177fe700 1 -- 172.21.15.7:0/4113012975 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8910085460 con 0x7f89200c00e0 2023-03-03T12:51:29.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.292+0000 7f891e7fc700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89000682f0 0x7f890006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:29.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.292+0000 7f891e7fc700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89000682f0 0x7f890006a7a0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f8908013040 tx=0x7f8908008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:51:29.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.297+0000 7f89177fe700 1 -- 172.21.15.7:0/4113012975 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f891004b5e0 con 0x7f89200c00e0 2023-03-03T12:51:29.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.535+0000 7f8925686700 1 -- 172.21.15.7:0/4113012975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8920074bb0 con 0x7f89200c00e0 2023-03-03T12:51:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.536+0000 7f89177fe700 1 -- 172.21.15.7:0/4113012975 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f89100517d0 con 0x7f89200c00e0 2023-03-03T12:51:29.538 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:51:29.538 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:51:29.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 -- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89000682f0 msgr2=0x7f890006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:29.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89000682f0 0x7f890006a7a0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f8908013040 tx=0x7f8908008070 comp rx=0 tx=0).stop 2023-03-03T12:51:29.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 -- 172.21.15.7:0/4113012975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 msgr2=0x7f8920148eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:29.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 0x7f8920148eb0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f891000c700 tx=0x7f891000cac0 comp rx=0 tx=0).stop 2023-03-03T12:51:29.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 -- 172.21.15.7:0/4113012975 shutdown_connections 2023-03-03T12:51:29.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8920154ff0 0x7f8920147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89000682f0 0x7f890006a7a0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89200c00e0 0x7f8920148eb0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 --2- 172.21.15.7:0/4113012975 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89200caa30 0x7f89201ed200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 -- 172.21.15.7:0/4113012975 >> 172.21.15.7:0/4113012975 conn(0x7f8920066e40 msgr2=0x7f8920153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:51:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 -- 172.21.15.7:0/4113012975 shutdown_connections 2023-03-03T12:51:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:29.538+0000 7f8925686700 1 -- 172.21.15.7:0/4113012975 wait complete. 2023-03-03T12:51:29.542 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:51:29.552 INFO:tasks.mds_thrash.fs.[a]:waiting for 50 secs before thrashing 2023-03-03T12:51:34.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:34.147 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:51:57.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:51:57.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.388+0000 7f18f091d700 1 Processor -- start 2023-03-03T12:51:57.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.388+0000 7f18f091d700 1 -- start start 2023-03-03T12:51:57.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.389+0000 7f18f091d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec0aad20 0x7f18ec157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:57.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.389+0000 7f18f091d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f18ec0aa030 0x7f18ec1581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:57.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.389+0000 7f18f091d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ec0728b0 con 0x7f18ec14b020 2023-03-03T12:51:57.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.389+0000 7f18f091d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18ec155070 con 0x7f18ec0aa030 2023-03-03T12:51:57.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.389+0000 7f18f091d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ec06f5b0 con 0x7f18ec0aad20 2023-03-03T12:51:57.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.389+0000 7f18e9d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec0aad20 0x7f18ec157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:57.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.389+0000 7f18ead9d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f18ec0aa030 0x7f18ec1581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:57.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.389+0000 7f18ea59c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f18ec14b020 0x7f18ec0717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:39748/0 (socket says 172.21.15.7:39748) 2023-03-03T12:51:57.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.389+0000 7f18ea59c700 1 -- 172.21.15.7:0/2636382001 learned_addr learned my addr 172.21.15.7:0/2636382001 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:51:57.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.390+0000 7f18ead9d700 1 -- 172.21.15.7:0/2636382001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec0aad20 msgr2=0x7f18ec157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:51:57.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.390+0000 7f18ead9d700 1 --2- 172.21.15.7:0/2636382001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec0aad20 0x7f18ec157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:57.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.390+0000 7f18ead9d700 1 -- 172.21.15.7:0/2636382001 >> v1:172.21.15.143:6789/0 conn(0x7f18ec14b020 legacy=0x7f18ec0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:57.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.390+0000 7f18e959a700 1 -- 172.21.15.7:0/2636382001 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041302932 0 0) 0x7f18ec0728b0 con 0x7f18ec14b020 2023-03-03T12:51:57.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.390+0000 7f18ead9d700 1 -- 172.21.15.7:0/2636382001 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec06f890 con 0x7f18ec0aa030 2023-03-03T12:51:57.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.390+0000 7f18e9d9b700 1 --2- 172.21.15.7:0/2636382001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec0aad20 0x7f18ec157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T12:51:57.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.390+0000 7f18ead9d700 1 --2- 172.21.15.7:0/2636382001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18ec0aa030 0x7f18ec1581e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009d90 tx=0x7f18e400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=278650efa83e9f92 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:51:57.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.390+0000 7f18e959a700 1 -- 172.21.15.7:0/2636382001 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e400e070 con 0x7f18ec0aa030 2023-03-03T12:51:57.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.390+0000 7f18e959a700 1 -- 172.21.15.7:0/2636382001 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e400bb20 con 0x7f18ec0aa030 2023-03-03T12:51:57.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.390+0000 7f18e959a700 1 -- 172.21.15.7:0/2636382001 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4012430 con 0x7f18ec0aa030 2023-03-03T12:51:57.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.391+0000 7f18f091d700 1 -- 172.21.15.7:0/2636382001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18ec0aa030 msgr2=0x7f18ec1581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:57.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.391+0000 7f18f091d700 1 --2- 172.21.15.7:0/2636382001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18ec0aa030 0x7f18ec1581e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009d90 tx=0x7f18e400b330 comp rx=0 tx=0).stop 2023-03-03T12:51:57.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.391+0000 7f18f091d700 1 -- 172.21.15.7:0/2636382001 shutdown_connections 2023-03-03T12:51:57.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.391+0000 7f18f091d700 1 --2- 172.21.15.7:0/2636382001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18ec0aa030 0x7f18ec1581e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:57.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.391+0000 7f18f091d700 1 --2- 172.21.15.7:0/2636382001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec0aad20 0x7f18ec157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:57.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.391+0000 7f18f091d700 1 -- 172.21.15.7:0/2636382001 >> 172.21.15.7:0/2636382001 conn(0x7f18ec0c8900 msgr2=0x7f18ec0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:51:57.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.391+0000 7f18f091d700 1 -- 172.21.15.7:0/2636382001 shutdown_connections 2023-03-03T12:51:57.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.391+0000 7f18f091d700 1 -- 172.21.15.7:0/2636382001 wait complete. 2023-03-03T12:51:57.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.392+0000 7f18f091d700 1 Processor -- start 2023-03-03T12:51:57.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.392+0000 7f18f091d700 1 -- start start 2023-03-03T12:51:57.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18f091d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18ec0aa030 0x7f18ec0c3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:57.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18f091d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18ec0aad20 0x7f18ec145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:57.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18f091d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec14b020 0x7f18ec1464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:57.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18f091d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ec150bc0 con 0x7f18ec0aa030 2023-03-03T12:51:57.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18f091d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ec1ece40 con 0x7f18ec0aad20 2023-03-03T12:51:57.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18f091d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18ec146a30 con 0x7f18ec14b020 2023-03-03T12:51:57.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18ea59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18ec0aa030 0x7f18ec0c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:57.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18ea59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18ec0aa030 0x7f18ec0c3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42256/0 (socket says 172.21.15.7:42256) 2023-03-03T12:51:57.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18ea59c700 1 -- 172.21.15.7:0/4258745418 learned_addr learned my addr 172.21.15.7:0/4258745418 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:51:57.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18e9d9b700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18ec0aad20 0x7f18ec145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:57.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18ead9d700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec14b020 0x7f18ec1464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:57.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18ea59c700 1 -- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec14b020 msgr2=0x7f18ec1464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:57.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18ea59c700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec14b020 0x7f18ec1464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:57.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18ea59c700 1 -- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18ec0aad20 msgr2=0x7f18ec145fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:57.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18ea59c700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18ec0aad20 0x7f18ec145fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:57.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18ea59c700 1 -- 172.21.15.7:0/4258745418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc0094a0 con 0x7f18ec0aa030 2023-03-03T12:51:57.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.393+0000 7f18ea59c700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18ec0aa030 0x7f18ec0c3000 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f18e000c550 tx=0x7f18e000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:51:57.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.394+0000 7f18daffd700 1 -- 172.21.15.7:0/4258745418 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0004c80 con 0x7f18ec0aa030 2023-03-03T12:51:57.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.394+0000 7f18daffd700 1 -- 172.21.15.7:0/4258745418 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e0004e00 con 0x7f18ec0aa030 2023-03-03T12:51:57.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.394+0000 7f18f091d700 1 -- 172.21.15.7:0/4258745418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e4009a70 con 0x7f18ec0aa030 2023-03-03T12:51:57.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.394+0000 7f18daffd700 1 -- 172.21.15.7:0/4258745418 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0007310 con 0x7f18ec0aa030 2023-03-03T12:51:57.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.394+0000 7f18f091d700 1 -- 172.21.15.7:0/4258745418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ec155350 con 0x7f18ec0aa030 2023-03-03T12:51:57.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.395+0000 7f18f091d700 1 -- 172.21.15.7:0/4258745418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ec06fb70 con 0x7f18ec0aa030 2023-03-03T12:51:57.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.395+0000 7f18daffd700 1 -- 172.21.15.7:0/4258745418 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f18e001a020 con 0x7f18ec0aa030 2023-03-03T12:51:57.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.395+0000 7f18daffd700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18cc0682f0 0x7f18cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:51:57.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.396+0000 7f18daffd700 1 -- 172.21.15.7:0/4258745418 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f18e0085bb0 con 0x7f18ec0aa030 2023-03-03T12:51:57.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.396+0000 7f18e9d9b700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18cc0682f0 0x7f18cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:51:57.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.396+0000 7f18e9d9b700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18cc0682f0 0x7f18cc06a7a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0c62c0 tx=0x7f18dc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:51:57.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.400+0000 7f18daffd700 1 -- 172.21.15.7:0/4258745418 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18e008c080 con 0x7f18ec0aa030 2023-03-03T12:51:57.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.635+0000 7f18f091d700 1 -- 172.21.15.7:0/4258745418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18ec071ef0 con 0x7f18ec0aa030 2023-03-03T12:51:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.635+0000 7f18daffd700 1 -- 172.21.15.7:0/4258745418 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f18e0051f50 con 0x7f18ec0aa030 2023-03-03T12:51:57.637 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:51:57.637 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:51:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 -- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18cc0682f0 msgr2=0x7f18cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18cc0682f0 0x7f18cc06a7a0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f18ec0c62c0 tx=0x7f18dc009210 comp rx=0 tx=0).stop 2023-03-03T12:51:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 -- 172.21.15.7:0/4258745418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18ec0aa030 msgr2=0x7f18ec0c3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:51:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18ec0aa030 0x7f18ec0c3000 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f18e000c550 tx=0x7f18e000c860 comp rx=0 tx=0).stop 2023-03-03T12:51:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 -- 172.21.15.7:0/4258745418 shutdown_connections 2023-03-03T12:51:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18ec14b020 0x7f18ec1464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18cc0682f0 0x7f18cc06a7a0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18ec0aa030 0x7f18ec0c3000 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 --2- 172.21.15.7:0/4258745418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18ec0aad20 0x7f18ec145fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:51:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 -- 172.21.15.7:0/4258745418 >> 172.21.15.7:0/4258745418 conn(0x7f18ec0c8900 msgr2=0x7f18ec0717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:51:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 -- 172.21.15.7:0/4258745418 shutdown_connections 2023-03-03T12:51:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:51:57.637+0000 7f18f091d700 1 -- 172.21.15.7:0/4258745418 wait complete. 2023-03-03T12:51:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:51:57.651 INFO:tasks.mds_thrash.fs.[b]:waiting for 92 secs before thrashing 2023-03-03T12:52:04.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:04.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:19.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:52:19.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.689+0000 7f883d8e3700 1 Processor -- start 2023-03-03T12:52:19.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.689+0000 7f883d8e3700 1 -- start start 2023-03-03T12:52:19.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.690+0000 7f883d8e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88380c17f0 0x7f883806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:19.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.690+0000 7f883d8e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88380c5190 0x7f883806dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:19.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.690+0000 7f883d8e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883814ebe0 con 0x7f88380c6010 2023-03-03T12:52:19.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.690+0000 7f883d8e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8838152100 con 0x7f88380c17f0 2023-03-03T12:52:19.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.690+0000 7f883d8e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88380c2e70 con 0x7f88380c5190 2023-03-03T12:52:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.690+0000 7f88377fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88380c5190 0x7f883806dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.690+0000 7f88367fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88380c17f0 0x7f883806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.690+0000 7f8836ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f88380c6010 0x7f88380c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:39768/0 (socket says 172.21.15.7:39768) 2023-03-03T12:52:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.690+0000 7f88367fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88380c17f0 0x7f883806d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:38044/0 (socket says 172.21.15.7:38044) 2023-03-03T12:52:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.690+0000 7f8836ffd700 1 -- 172.21.15.7:0/3861983319 learned_addr learned my addr 172.21.15.7:0/3861983319 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:52:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.691+0000 7f88367fc700 1 -- 172.21.15.7:0/3861983319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88380c5190 msgr2=0x7f883806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:52:19.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.691+0000 7f88367fc700 1 --2- 172.21.15.7:0/3861983319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88380c5190 0x7f883806dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.691+0000 7f88367fc700 1 -- 172.21.15.7:0/3861983319 >> v1:172.21.15.143:6789/0 conn(0x7f88380c6010 legacy=0x7f88380c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.691+0000 7f88367fc700 1 -- 172.21.15.7:0/3861983319 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88380aa680 con 0x7f88380c17f0 2023-03-03T12:52:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.691+0000 7f88367fc700 1 --2- 172.21.15.7:0/3861983319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88380c17f0 0x7f883806d6a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8828009840 tx=0x7f8828009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65f3a486a5a01333 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:52:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.691+0000 7f8835ffb700 1 -- 172.21.15.7:0/3861983319 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8828011830 con 0x7f88380c17f0 2023-03-03T12:52:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.691+0000 7f8835ffb700 1 -- 172.21.15.7:0/3861983319 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8828011990 con 0x7f88380c17f0 2023-03-03T12:52:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.691+0000 7f8835ffb700 1 -- 172.21.15.7:0/3861983319 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8828011d20 con 0x7f88380c17f0 2023-03-03T12:52:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.692+0000 7f883d8e3700 1 -- 172.21.15.7:0/3861983319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88380c17f0 msgr2=0x7f883806d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.692+0000 7f883d8e3700 1 --2- 172.21.15.7:0/3861983319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88380c17f0 0x7f883806d6a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8828009840 tx=0x7f8828009b50 comp rx=0 tx=0).stop 2023-03-03T12:52:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.692+0000 7f883d8e3700 1 -- 172.21.15.7:0/3861983319 shutdown_connections 2023-03-03T12:52:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.692+0000 7f883d8e3700 1 --2- 172.21.15.7:0/3861983319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88380c5190 0x7f883806dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.692+0000 7f883d8e3700 1 --2- 172.21.15.7:0/3861983319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88380c17f0 0x7f883806d6a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.692+0000 7f883d8e3700 1 -- 172.21.15.7:0/3861983319 >> 172.21.15.7:0/3861983319 conn(0x7f88380a9430 msgr2=0x7f88380c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:52:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.692+0000 7f883d8e3700 1 -- 172.21.15.7:0/3861983319 shutdown_connections 2023-03-03T12:52:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.692+0000 7f883d8e3700 1 -- 172.21.15.7:0/3861983319 wait complete. 2023-03-03T12:52:19.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.693+0000 7f883d8e3700 1 Processor -- start 2023-03-03T12:52:19.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.693+0000 7f883d8e3700 1 -- start start 2023-03-03T12:52:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f883d8e3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88380c17f0 0x7f88381dcc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f883d8e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88380c5190 0x7f883814d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f883d8e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88380c6010 0x7f8838154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f883d8e3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88381dbac0 con 0x7f88380c17f0 2023-03-03T12:52:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f883d8e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88381db680 con 0x7f88380c5190 2023-03-03T12:52:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f883d8e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8838154940 con 0x7f88380c6010 2023-03-03T12:52:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f8836ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88380c17f0 0x7f88381dcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f8836ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88380c17f0 0x7f88381dcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42274/0 (socket says 172.21.15.7:42274) 2023-03-03T12:52:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f8836ffd700 1 -- 172.21.15.7:0/1569292909 learned_addr learned my addr 172.21.15.7:0/1569292909 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:52:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f88377fe700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88380c6010 0x7f8838154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f88367fc700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88380c5190 0x7f883814d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f8836ffd700 1 -- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88380c6010 msgr2=0x7f8838154dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f8836ffd700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88380c6010 0x7f8838154dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f8836ffd700 1 -- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88380c5190 msgr2=0x7f883814d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f8836ffd700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88380c5190 0x7f883814d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f8836ffd700 1 -- 172.21.15.7:0/1569292909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88280094a0 con 0x7f88380c17f0 2023-03-03T12:52:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.694+0000 7f8836ffd700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88380c17f0 0x7f88381dcc10 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f882000c550 tx=0x7f882000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:52:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.695+0000 7f882ffff700 1 -- 172.21.15.7:0/1569292909 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820005d70 con 0x7f88380c17f0 2023-03-03T12:52:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.695+0000 7f882ffff700 1 -- 172.21.15.7:0/1569292909 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88200070d0 con 0x7f88380c17f0 2023-03-03T12:52:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.695+0000 7f883d8e3700 1 -- 172.21.15.7:0/1569292909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88380c2b00 con 0x7f88380c17f0 2023-03-03T12:52:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.695+0000 7f882ffff700 1 -- 172.21.15.7:0/1569292909 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820007440 con 0x7f88380c17f0 2023-03-03T12:52:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.695+0000 7f883d8e3700 1 -- 172.21.15.7:0/1569292909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88380aa960 con 0x7f88380c17f0 2023-03-03T12:52:19.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.696+0000 7f883d8e3700 1 -- 172.21.15.7:0/1569292909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88380c3300 con 0x7f88380c17f0 2023-03-03T12:52:19.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.696+0000 7f882ffff700 1 -- 172.21.15.7:0/1569292909 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8820019030 con 0x7f88380c17f0 2023-03-03T12:52:19.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.696+0000 7f882ffff700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88180682f0 0x7f881806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:19.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.696+0000 7f882ffff700 1 -- 172.21.15.7:0/1569292909 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f88200852a0 con 0x7f88380c17f0 2023-03-03T12:52:19.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.697+0000 7f88367fc700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88180682f0 0x7f881806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:19.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.697+0000 7f88367fc700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88180682f0 0x7f881806a7a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8828000c00 tx=0x7f882801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:52:19.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.701+0000 7f882ffff700 1 -- 172.21.15.7:0/1569292909 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f882004b5e0 con 0x7f88380c17f0 2023-03-03T12:52:19.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.946+0000 7f883d8e3700 1 -- 172.21.15.7:0/1569292909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8838154c20 con 0x7f88380c17f0 2023-03-03T12:52:19.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.946+0000 7f882ffff700 1 -- 172.21.15.7:0/1569292909 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f88200516f0 con 0x7f88380c17f0 2023-03-03T12:52:19.948 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:52:19.948 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:52:19.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 -- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88180682f0 msgr2=0x7f881806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:19.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88180682f0 0x7f881806a7a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f8828000c00 tx=0x7f882801c040 comp rx=0 tx=0).stop 2023-03-03T12:52:19.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 -- 172.21.15.7:0/1569292909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88380c17f0 msgr2=0x7f88381dcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:19.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88380c17f0 0x7f88381dcc10 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f882000c550 tx=0x7f882000c860 comp rx=0 tx=0).stop 2023-03-03T12:52:19.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 -- 172.21.15.7:0/1569292909 shutdown_connections 2023-03-03T12:52:19.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88380c6010 0x7f8838154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:19.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88180682f0 0x7f881806a7a0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:19.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88380c17f0 0x7f88381dcc10 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:19.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 --2- 172.21.15.7:0/1569292909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88380c5190 0x7f883814d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:19.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 -- 172.21.15.7:0/1569292909 >> 172.21.15.7:0/1569292909 conn(0x7f88380a9430 msgr2=0x7f88380a7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:52:19.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 -- 172.21.15.7:0/1569292909 shutdown_connections 2023-03-03T12:52:19.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:19.948+0000 7f883d8e3700 1 -- 172.21.15.7:0/1569292909 wait complete. 2023-03-03T12:52:19.952 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:52:19.962 INFO:tasks.mds_thrash.fs.[a]:waiting for 36 secs before thrashing 2023-03-03T12:52:34.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:34.416 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:52:55.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:52:56.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.100+0000 7f7c07e30700 1 Processor -- start 2023-03-03T12:52:56.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.100+0000 7f7c07e30700 1 -- start start 2023-03-03T12:52:56.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c07e30700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c00072250 0x7f7c0014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:56.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c07e30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c000c1ce0 0x7f7c0014e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:56.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c07e30700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c001510d0 con 0x7f7c00071e80 2023-03-03T12:52:56.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c07e30700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c000c3740 con 0x7f7c00072250 2023-03-03T12:52:56.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c07e30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c00070670 con 0x7f7c000c1ce0 2023-03-03T12:52:56.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c053cb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c00072250 0x7f7c0014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:56.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c05bcc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7c00071e80 0x7f7c00151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36278/0 (socket says 172.21.15.7:36278) 2023-03-03T12:52:56.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c05bcc700 1 -- 172.21.15.7:0/3087519971 learned_addr learned my addr 172.21.15.7:0/3087519971 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:52:56.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c063cd700 1 --2- 172.21.15.7:0/3087519971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c000c1ce0 0x7f7c0014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:56.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c053cb700 1 -- 172.21.15.7:0/3087519971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c000c1ce0 msgr2=0x7f7c0014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:52:56.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c053cb700 1 --2- 172.21.15.7:0/3087519971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c000c1ce0 0x7f7c0014e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:56.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c053cb700 1 -- 172.21.15.7:0/3087519971 >> v1:172.21.15.7:6789/0 conn(0x7f7c00071e80 legacy=0x7f7c00151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:56.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c053cb700 1 -- 172.21.15.7:0/3087519971 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c000c39e0 con 0x7f7c00072250 2023-03-03T12:52:56.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c063cd700 1 --2- 172.21.15.7:0/3087519971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c000c1ce0 0x7f7c0014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:52:56.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.101+0000 7f7c053cb700 1 --2- 172.21.15.7:0/3087519971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 0x7f7c0014e1f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009840 tx=0x7f7bf0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b2233cc588c7a0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:52:56.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c04bca700 1 -- 172.21.15.7:0/3087519971 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0011830 con 0x7f7c00072250 2023-03-03T12:52:56.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c04bca700 1 -- 172.21.15.7:0/3087519971 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf0011990 con 0x7f7c00072250 2023-03-03T12:52:56.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c04bca700 1 -- 172.21.15.7:0/3087519971 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0011d20 con 0x7f7c00072250 2023-03-03T12:52:56.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c07e30700 1 -- 172.21.15.7:0/3087519971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 msgr2=0x7f7c0014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:56.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c07e30700 1 --2- 172.21.15.7:0/3087519971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 0x7f7c0014e1f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009840 tx=0x7f7bf0009b50 comp rx=0 tx=0).stop 2023-03-03T12:52:56.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c07e30700 1 -- 172.21.15.7:0/3087519971 shutdown_connections 2023-03-03T12:52:56.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c07e30700 1 --2- 172.21.15.7:0/3087519971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c000c1ce0 0x7f7c0014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:56.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c07e30700 1 --2- 172.21.15.7:0/3087519971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 0x7f7c0014e1f0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:56.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c07e30700 1 -- 172.21.15.7:0/3087519971 >> 172.21.15.7:0/3087519971 conn(0x7f7c000a94d0 msgr2=0x7f7c000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:52:56.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c07e30700 1 -- 172.21.15.7:0/3087519971 shutdown_connections 2023-03-03T12:52:56.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.102+0000 7f7c07e30700 1 -- 172.21.15.7:0/3087519971 wait complete. 2023-03-03T12:52:56.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.103+0000 7f7c07e30700 1 Processor -- start 2023-03-03T12:52:56.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.103+0000 7f7c07e30700 1 -- start start 2023-03-03T12:52:56.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c07e30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c00071e80 0x7f7c001f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:56.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c07e30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 0x7f7c001eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:56.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c07e30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c000c1ce0 0x7f7c00146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:56.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c07e30700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c000c6e90 con 0x7f7c00072250 2023-03-03T12:52:56.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c07e30700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c001462b0 con 0x7f7c000c1ce0 2023-03-03T12:52:56.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c07e30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c001f21f0 con 0x7f7c00071e80 2023-03-03T12:52:56.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c053cb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 0x7f7c001eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:56.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c053cb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 0x7f7c001eb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42306/0 (socket says 172.21.15.7:42306) 2023-03-03T12:52:56.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c053cb700 1 -- 172.21.15.7:0/438599234 learned_addr learned my addr 172.21.15.7:0/438599234 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:52:56.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c05bcc700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c00071e80 0x7f7c001f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:56.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.104+0000 7f7c063cd700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c000c1ce0 0x7f7c00146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:56.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7c053cb700 1 -- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c00071e80 msgr2=0x7f7c001f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:56.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7c053cb700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c00071e80 0x7f7c001f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:56.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7c053cb700 1 -- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c000c1ce0 msgr2=0x7f7c00146ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:56.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7c053cb700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c000c1ce0 0x7f7c00146ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:56.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7c053cb700 1 -- 172.21.15.7:0/438599234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf00094a0 con 0x7f7c00072250 2023-03-03T12:52:56.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7c063cd700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c000c1ce0 0x7f7c00146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:52:56.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7c053cb700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 0x7f7c001eb4f0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00090b0 tx=0x7f7bf0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:52:56.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7bf67fc700 1 -- 172.21.15.7:0/438599234 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00038a0 con 0x7f7c00072250 2023-03-03T12:52:56.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7bf67fc700 1 -- 172.21.15.7:0/438599234 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf001f070 con 0x7f7c00072250 2023-03-03T12:52:56.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7c07e30700 1 -- 172.21.15.7:0/438599234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c00151f30 con 0x7f7c00072250 2023-03-03T12:52:56.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7bf67fc700 1 -- 172.21.15.7:0/438599234 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0003c60 con 0x7f7c00072250 2023-03-03T12:52:56.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.105+0000 7f7c07e30700 1 -- 172.21.15.7:0/438599234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c00151300 con 0x7f7c00072250 2023-03-03T12:52:56.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.106+0000 7f7c07e30700 1 -- 172.21.15.7:0/438599234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c000c60e0 con 0x7f7c00072250 2023-03-03T12:52:56.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.106+0000 7f7bf67fc700 1 -- 172.21.15.7:0/438599234 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7bf000f050 con 0x7f7c00072250 2023-03-03T12:52:56.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.107+0000 7f7bf67fc700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7be40682f0 0x7f7be406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:52:56.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.107+0000 7f7bf67fc700 1 -- 172.21.15.7:0/438599234 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7bf0021070 con 0x7f7c00072250 2023-03-03T12:52:56.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.107+0000 7f7c05bcc700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7be40682f0 0x7f7be406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:52:56.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.108+0000 7f7c05bcc700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7be40682f0 0x7f7be406a7a0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc005910 tx=0x7f7bfc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:52:56.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.111+0000 7f7bf67fc700 1 -- 172.21.15.7:0/438599234 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7bf0053f30 con 0x7f7c00072250 2023-03-03T12:52:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.349+0000 7f7c07e30700 1 -- 172.21.15.7:0/438599234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c0006c9a0 con 0x7f7c00072250 2023-03-03T12:52:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.350+0000 7f7bf67fc700 1 -- 172.21.15.7:0/438599234 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7bf0057550 con 0x7f7c00072250 2023-03-03T12:52:56.352 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:52:56.352 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:52:56.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 -- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7be40682f0 msgr2=0x7f7be406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:56.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7be40682f0 0x7f7be406a7a0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc005910 tx=0x7f7bfc009040 comp rx=0 tx=0).stop 2023-03-03T12:52:56.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 -- 172.21.15.7:0/438599234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 msgr2=0x7f7c001eb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:52:56.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 0x7f7c001eb4f0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00090b0 tx=0x7f7bf0004750 comp rx=0 tx=0).stop 2023-03-03T12:52:56.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 -- 172.21.15.7:0/438599234 shutdown_connections 2023-03-03T12:52:56.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c000c1ce0 0x7f7c00146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:56.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c00071e80 0x7f7c001f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:56.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7be40682f0 0x7f7be406a7a0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:56.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 --2- 172.21.15.7:0/438599234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c00072250 0x7f7c001eb4f0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:52:56.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 -- 172.21.15.7:0/438599234 >> 172.21.15.7:0/438599234 conn(0x7f7c000a94d0 msgr2=0x7f7c0006a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:52:56.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 -- 172.21.15.7:0/438599234 shutdown_connections 2023-03-03T12:52:56.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:52:56.352+0000 7f7c07e30700 1 -- 172.21.15.7:0/438599234 wait complete. 2023-03-03T12:52:56.356 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:52:56.365 INFO:tasks.mds_thrash.fs.[a]:waiting for 30 secs before thrashing 2023-03-03T12:53:04.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:04.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:26.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:53:26.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.503+0000 7fa2080ed700 1 Processor -- start 2023-03-03T12:53:26.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.503+0000 7fa2080ed700 1 -- start start 2023-03-03T12:53:26.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa2080ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa20006a2b0 0x7fa20006a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:26.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa2080ed700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa2000a6370 0x7fa200155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:26.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa2080ed700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa2000c5cb0 0x7fa2001562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:26.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa2080ed700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa20006bab0 con 0x7fa2000c5cb0 2023-03-03T12:53:26.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa2080ed700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa200156800 con 0x7fa2000a6370 2023-03-03T12:53:26.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa2080ed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001558f0 con 0x7fa20006a2b0 2023-03-03T12:53:26.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa20668a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa2000c5cb0 0x7fa2001562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa20668a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa2000c5cb0 0x7fa2001562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42334/0 (socket says 172.21.15.7:42334) 2023-03-03T12:53:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa20668a700 1 -- 172.21.15.7:0/155704970 learned_addr learned my addr 172.21.15.7:0/155704970 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:53:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa205688700 1 --2- 172.21.15.7:0/155704970 >> v2:172.21.15.143:3300/0 conn(0x7fa2000a6370 0x7fa200155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa205e89700 1 --2- 172.21.15.7:0/155704970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa20006a2b0 0x7fa20006a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa20668a700 1 -- 172.21.15.7:0/155704970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa20006a2b0 msgr2=0x7fa20006a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:53:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa20668a700 1 --2- 172.21.15.7:0/155704970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa20006a2b0 0x7fa20006a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa20668a700 1 -- 172.21.15.7:0/155704970 >> v2:172.21.15.143:3300/0 conn(0x7fa2000a6370 msgr2=0x7fa200155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:53:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa20668a700 1 --2- 172.21.15.7:0/155704970 >> v2:172.21.15.143:3300/0 conn(0x7fa2000a6370 0x7fa200155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.504+0000 7fa20668a700 1 -- 172.21.15.7:0/155704970 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2000a6180 con 0x7fa2000c5cb0 2023-03-03T12:53:26.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa20668a700 1 --2- 172.21.15.7:0/155704970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000c5cb0 0x7fa2001562c0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fa2000ac060 tx=0x7fa1f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6381657dfcc6e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:53:26.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa204e87700 1 -- 172.21.15.7:0/155704970 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f800e070 con 0x7fa2000c5cb0 2023-03-03T12:53:26.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa204e87700 1 -- 172.21.15.7:0/155704970 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f800b9c0 con 0x7fa2000c5cb0 2023-03-03T12:53:26.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa204e87700 1 -- 172.21.15.7:0/155704970 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f800bd70 con 0x7fa2000c5cb0 2023-03-03T12:53:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa2080ed700 1 -- 172.21.15.7:0/155704970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000c5cb0 msgr2=0x7fa2001562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:26.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa2080ed700 1 --2- 172.21.15.7:0/155704970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000c5cb0 0x7fa2001562c0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fa2000ac060 tx=0x7fa1f800b330 comp rx=0 tx=0).stop 2023-03-03T12:53:26.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa2080ed700 1 -- 172.21.15.7:0/155704970 shutdown_connections 2023-03-03T12:53:26.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa2080ed700 1 --2- 172.21.15.7:0/155704970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000c5cb0 0x7fa2001562c0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa2080ed700 1 --2- 172.21.15.7:0/155704970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa20006a2b0 0x7fa20006a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa2080ed700 1 --2- 172.21.15.7:0/155704970 >> v2:172.21.15.143:3300/0 conn(0x7fa2000a6370 0x7fa200155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa2080ed700 1 -- 172.21.15.7:0/155704970 >> 172.21.15.7:0/155704970 conn(0x7fa2000a9430 msgr2=0x7fa2000c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:53:26.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa2080ed700 1 -- 172.21.15.7:0/155704970 shutdown_connections 2023-03-03T12:53:26.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.505+0000 7fa2080ed700 1 -- 172.21.15.7:0/155704970 wait complete. 2023-03-03T12:53:26.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.507+0000 7fa2080ed700 1 Processor -- start 2023-03-03T12:53:26.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.507+0000 7fa2080ed700 1 -- start start 2023-03-03T12:53:26.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.507+0000 7fa2080ed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa20006a2b0 0x7fa2001eb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:26.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.507+0000 7fa2080ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000a6370 0x7fa200147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:26.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.507+0000 7fa2080ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2000c5cb0 0x7fa20014de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:26.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.507+0000 7fa2080ed700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa200155660 con 0x7fa2000a6370 2023-03-03T12:53:26.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.507+0000 7fa2080ed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001eb310 con 0x7fa20006a2b0 2023-03-03T12:53:26.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.507+0000 7fa2080ed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2001550e0 con 0x7fa2000c5cb0 2023-03-03T12:53:26.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205688700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000a6370 0x7fa200147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:26.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205688700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000a6370 0x7fa200147f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42338/0 (socket says 172.21.15.7:42338) 2023-03-03T12:53:26.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205688700 1 -- 172.21.15.7:0/432393245 learned_addr learned my addr 172.21.15.7:0/432393245 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:53:26.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205e89700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa20006a2b0 0x7fa2001eb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:26.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa20668a700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2000c5cb0 0x7fa20014de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:26.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205688700 1 -- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2000c5cb0 msgr2=0x7fa20014de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:26.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205688700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2000c5cb0 0x7fa20014de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205688700 1 -- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa20006a2b0 msgr2=0x7fa2001eb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:26.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205688700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa20006a2b0 0x7fa2001eb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205688700 1 -- 172.21.15.7:0/432393245 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f8009b30 con 0x7fa2000a6370 2023-03-03T12:53:26.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205e89700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa20006a2b0 0x7fa2001eb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:53:26.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa20668a700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2000c5cb0 0x7fa20014de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:53:26.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa205688700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000a6370 0x7fa200147f10 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00e810 tx=0x7fa1fc00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:53:26.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa1f67fc700 1 -- 172.21.15.7:0/432393245 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc013610 con 0x7fa2000a6370 2023-03-03T12:53:26.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa1f67fc700 1 -- 172.21.15.7:0/432393245 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc013770 con 0x7fa2000a6370 2023-03-03T12:53:26.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa1f67fc700 1 -- 172.21.15.7:0/432393245 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc013b20 con 0x7fa2000a6370 2023-03-03T12:53:26.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa2080ed700 1 -- 172.21.15.7:0/432393245 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa200152e80 con 0x7fa2000a6370 2023-03-03T12:53:26.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.508+0000 7fa2080ed700 1 -- 172.21.15.7:0/432393245 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2000a5ea0 con 0x7fa2000a6370 2023-03-03T12:53:26.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.509+0000 7fa2080ed700 1 -- 172.21.15.7:0/432393245 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa200153160 con 0x7fa2000a6370 2023-03-03T12:53:26.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.510+0000 7fa1f67fc700 1 -- 172.21.15.7:0/432393245 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa1fc023070 con 0x7fa2000a6370 2023-03-03T12:53:26.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.510+0000 7fa1f67fc700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1e40682f0 0x7fa1e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:26.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.510+0000 7fa1f67fc700 1 -- 172.21.15.7:0/432393245 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa1fc085e80 con 0x7fa2000a6370 2023-03-03T12:53:26.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.510+0000 7fa205e89700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1e40682f0 0x7fa1e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:26.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.511+0000 7fa205e89700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1e40682f0 0x7fa1e406a7a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa20014d080 tx=0x7fa1f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:53:26.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.514+0000 7fa1f67fc700 1 -- 172.21.15.7:0/432393245 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa1fc0521a0 con 0x7fa2000a6370 2023-03-03T12:53:26.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.756+0000 7fa2080ed700 1 -- 172.21.15.7:0/432393245 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2001eb480 con 0x7fa2000a6370 2023-03-03T12:53:26.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.757+0000 7fa1f67fc700 1 -- 172.21.15.7:0/432393245 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa1fc0557c0 con 0x7fa2000a6370 2023-03-03T12:53:26.759 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:53:26.759 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:53:26.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 -- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1e40682f0 msgr2=0x7fa1e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:26.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1e40682f0 0x7fa1e406a7a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fa20014d080 tx=0x7fa1f0008040 comp rx=0 tx=0).stop 2023-03-03T12:53:26.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 -- 172.21.15.7:0/432393245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000a6370 msgr2=0x7fa200147f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:26.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000a6370 0x7fa200147f10 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00e810 tx=0x7fa1fc00ebd0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 -- 172.21.15.7:0/432393245 shutdown_connections 2023-03-03T12:53:26.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2000c5cb0 0x7fa20014de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa20006a2b0 0x7fa2001eb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1e40682f0 0x7fa1e406a7a0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 --2- 172.21.15.7:0/432393245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2000a6370 0x7fa200147f10 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:26.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 -- 172.21.15.7:0/432393245 >> 172.21.15.7:0/432393245 conn(0x7fa2000a9430 msgr2=0x7fa2000c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:53:26.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 -- 172.21.15.7:0/432393245 shutdown_connections 2023-03-03T12:53:26.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:26.759+0000 7fa2080ed700 1 -- 172.21.15.7:0/432393245 wait complete. 2023-03-03T12:53:26.763 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:53:26.773 INFO:tasks.mds_thrash.fs.[a]:waiting for 85 secs before thrashing 2023-03-03T12:53:29.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:53:29.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.789+0000 7f51fa55d700 1 Processor -- start 2023-03-03T12:53:29.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.789+0000 7f51fa55d700 1 -- start start 2023-03-03T12:53:29.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.790+0000 7f51fa55d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f51f414fc50 0x7f51f40c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:29.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.790+0000 7f51fa55d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 0x7f51f40c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:29.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.790+0000 7f51fa55d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f40ad2b0 con 0x7f51f40c3e30 2023-03-03T12:53:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.790+0000 7f51fa55d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f40c30c0 con 0x7f51f414fc50 2023-03-03T12:53:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.790+0000 7f51fa55d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f4156420 con 0x7f51f406d240 2023-03-03T12:53:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.790+0000 7f51f3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f51f414fc50 0x7f51f40c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.790+0000 7f51f8afa700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f51f40c3e30 0x7f51f40c7390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36334/0 (socket says 172.21.15.7:36334) 2023-03-03T12:53:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.790+0000 7f51f8afa700 1 -- 172.21.15.7:0/2871148264 learned_addr learned my addr 172.21.15.7:0/2871148264 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:53:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.790+0000 7f51f37fe700 1 --2- 172.21.15.7:0/2871148264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 0x7f51f40c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.790+0000 7f51f3fff700 1 -- 172.21.15.7:0/2871148264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 msgr2=0x7f51f40c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:53:29.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51f3fff700 1 --2- 172.21.15.7:0/2871148264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 0x7f51f40c27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:29.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51f3fff700 1 -- 172.21.15.7:0/2871148264 >> v1:172.21.15.7:6789/0 conn(0x7f51f40c3e30 legacy=0x7f51f40c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:29.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51f3fff700 1 -- 172.21.15.7:0/2871148264 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f40ad520 con 0x7f51f414fc50 2023-03-03T12:53:29.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51f2ffd700 1 -- 172.21.15.7:0/2871148264 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380436287 0 0) 0x7f51f40ad2b0 con 0x7f51f40c3e30 2023-03-03T12:53:29.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51f3fff700 1 --2- 172.21.15.7:0/2871148264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 0x7f51f40c1f50 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009840 tx=0x7f51dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bb032ca8b05bd4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:53:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51f2ffd700 1 -- 172.21.15.7:0/2871148264 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc011830 con 0x7f51f414fc50 2023-03-03T12:53:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51f2ffd700 1 -- 172.21.15.7:0/2871148264 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc011990 con 0x7f51f414fc50 2023-03-03T12:53:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51f2ffd700 1 -- 172.21.15.7:0/2871148264 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc011d20 con 0x7f51f414fc50 2023-03-03T12:53:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51fa55d700 1 -- 172.21.15.7:0/2871148264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 msgr2=0x7f51f40c1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51fa55d700 1 --2- 172.21.15.7:0/2871148264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 0x7f51f40c1f50 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009840 tx=0x7f51dc009b50 comp rx=0 tx=0).stop 2023-03-03T12:53:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51fa55d700 1 -- 172.21.15.7:0/2871148264 shutdown_connections 2023-03-03T12:53:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51fa55d700 1 --2- 172.21.15.7:0/2871148264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 0x7f51f40c1f50 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51fa55d700 1 --2- 172.21.15.7:0/2871148264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 0x7f51f40c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51fa55d700 1 -- 172.21.15.7:0/2871148264 >> 172.21.15.7:0/2871148264 conn(0x7f51f406c110 msgr2=0x7f51f406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:53:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.791+0000 7f51fa55d700 1 -- 172.21.15.7:0/2871148264 shutdown_connections 2023-03-03T12:53:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.792+0000 7f51fa55d700 1 -- 172.21.15.7:0/2871148264 wait complete. 2023-03-03T12:53:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.793+0000 7f51fa55d700 1 Processor -- start 2023-03-03T12:53:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.793+0000 7f51fa55d700 1 -- start start 2023-03-03T12:53:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.793+0000 7f51fa55d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 0x7f51f41eb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51fa55d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51f40c3e30 0x7f51f40c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51fa55d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 0x7f51f40c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51fa55d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f41ec200 con 0x7f51f414fc50 2023-03-03T12:53:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51fa55d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f40c5bb0 con 0x7f51f40c3e30 2023-03-03T12:53:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51fa55d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f41ebf20 con 0x7f51f406d240 2023-03-03T12:53:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f8afa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 0x7f51f40c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f8afa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 0x7f51f40c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42360/0 (socket says 172.21.15.7:42360) 2023-03-03T12:53:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f8afa700 1 -- 172.21.15.7:0/3914034522 learned_addr learned my addr 172.21.15.7:0/3914034522 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:53:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f3fff700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 0x7f51f41eb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f37fe700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51f40c3e30 0x7f51f40c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:29.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f8afa700 1 -- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 msgr2=0x7f51f41eb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:29.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f8afa700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 0x7f51f41eb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:29.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f8afa700 1 -- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51f40c3e30 msgr2=0x7f51f40c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:29.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f8afa700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51f40c3e30 0x7f51f40c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:29.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f8afa700 1 -- 172.21.15.7:0/3914034522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc0094a0 con 0x7f51f414fc50 2023-03-03T12:53:29.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f37fe700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51f40c3e30 0x7f51f40c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:53:29.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f3fff700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 0x7f51f41eb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:53:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f8afa700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 0x7f51f40c7960 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00cba0 tx=0x7f51ec00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:53:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f0ff9700 1 -- 172.21.15.7:0/3914034522 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec00ab80 con 0x7f51f414fc50 2023-03-03T12:53:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f0ff9700 1 -- 172.21.15.7:0/3914034522 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec00ad00 con 0x7f51f414fc50 2023-03-03T12:53:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51fa55d700 1 -- 172.21.15.7:0/3914034522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f40c2cf0 con 0x7f51f414fc50 2023-03-03T12:53:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51f0ff9700 1 -- 172.21.15.7:0/3914034522 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec007310 con 0x7f51f414fc50 2023-03-03T12:53:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.794+0000 7f51fa55d700 1 -- 172.21.15.7:0/3914034522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f40c2490 con 0x7f51f414fc50 2023-03-03T12:53:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.796+0000 7f51fa55d700 1 -- 172.21.15.7:0/3914034522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f4074220 con 0x7f51f414fc50 2023-03-03T12:53:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.796+0000 7f51f0ff9700 1 -- 172.21.15.7:0/3914034522 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f51ec01a030 con 0x7f51f414fc50 2023-03-03T12:53:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.796+0000 7f51f0ff9700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51d40682f0 0x7f51d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:53:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.796+0000 7f51f0ff9700 1 -- 172.21.15.7:0/3914034522 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f51ec085e80 con 0x7f51f414fc50 2023-03-03T12:53:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.797+0000 7f51f3fff700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51d40682f0 0x7f51d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:53:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.797+0000 7f51f3fff700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51d40682f0 0x7f51d406a7a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f51f414c7c0 tx=0x7f51dc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:53:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:29.801+0000 7f51f0ff9700 1 -- 172.21.15.7:0/3914034522 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f51ec052220 con 0x7f51f414fc50 2023-03-03T12:53:30.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.038+0000 7f51fa55d700 1 -- 172.21.15.7:0/3914034522 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51f40c5d60 con 0x7f51f414fc50 2023-03-03T12:53:30.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.038+0000 7f51f0ff9700 1 -- 172.21.15.7:0/3914034522 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f51f40c5d60 con 0x7f51f414fc50 2023-03-03T12:53:30.041 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:53:30.041 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:53:30.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.040+0000 7f51fa55d700 1 -- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51d40682f0 msgr2=0x7f51d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:30.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.040+0000 7f51fa55d700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51d40682f0 0x7f51d406a7a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f51f414c7c0 tx=0x7f51dc008030 comp rx=0 tx=0).stop 2023-03-03T12:53:30.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.040+0000 7f51fa55d700 1 -- 172.21.15.7:0/3914034522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 msgr2=0x7f51f40c7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:53:30.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.040+0000 7f51fa55d700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 0x7f51f40c7960 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00cba0 tx=0x7f51ec00cf60 comp rx=0 tx=0).stop 2023-03-03T12:53:30.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.040+0000 7f51fa55d700 1 -- 172.21.15.7:0/3914034522 shutdown_connections 2023-03-03T12:53:30.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.040+0000 7f51fa55d700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51d40682f0 0x7f51d406a7a0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:30.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.040+0000 7f51fa55d700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f414fc50 0x7f51f40c7960 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:30.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.040+0000 7f51fa55d700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f406d240 0x7f51f41eb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:30.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.040+0000 7f51fa55d700 1 --2- 172.21.15.7:0/3914034522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51f40c3e30 0x7f51f40c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:53:30.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.040+0000 7f51fa55d700 1 -- 172.21.15.7:0/3914034522 >> 172.21.15.7:0/3914034522 conn(0x7f51f406c110 msgr2=0x7f51f406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:53:30.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.041+0000 7f51fa55d700 1 -- 172.21.15.7:0/3914034522 shutdown_connections 2023-03-03T12:53:30.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:53:30.041+0000 7f51fa55d700 1 -- 172.21.15.7:0/3914034522 wait complete. 2023-03-03T12:53:30.045 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:53:30.055 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T12:53:34.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:53:34.683 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:04.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:04.872 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:34.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:34.977 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:54:51.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:54:51.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.910+0000 7f5bf1815700 1 Processor -- start 2023-03-03T12:54:51.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.910+0000 7f5bf1815700 1 -- start start 2023-03-03T12:54:51.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.911+0000 7f5bf1815700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5bec0a5d20 0x7f5bec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:54:51.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.911+0000 7f5bf1815700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec157f10 0x7f5bec158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:54:51.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.911+0000 7f5bf1815700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bec1555c0 con 0x7f5bec151e30 2023-03-03T12:54:51.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.911+0000 7f5bf1815700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bec1549b0 con 0x7f5bec0a5d20 2023-03-03T12:54:51.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.911+0000 7f5bf1815700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec154c90 con 0x7f5bec157f10 2023-03-03T12:54:51.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.911+0000 7f5bea7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5bec0a5d20 0x7f5bec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:54:51.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.911+0000 7f5bea7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5bec0a5d20 0x7f5bec0a6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42422/0 (socket says 172.21.15.7:42422) 2023-03-03T12:54:51.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.911+0000 7f5bea7fc700 1 -- 172.21.15.7:0/2317142463 learned_addr learned my addr 172.21.15.7:0/2317142463 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:54:51.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.912+0000 7f5bea7fc700 1 -- 172.21.15.7:0/2317142463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec157f10 msgr2=0x7f5bec158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:54:51.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.912+0000 7f5beb7fe700 1 --2- 172.21.15.7:0/2317142463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec157f10 0x7f5bec158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:54:51.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.912+0000 7f5bea7fc700 1 --2- 172.21.15.7:0/2317142463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec157f10 0x7f5bec158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:54:51.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.912+0000 7f5bea7fc700 1 -- 172.21.15.7:0/2317142463 >> v1:172.21.15.143:6789/0 conn(0x7f5bec151e30 legacy=0x7f5bec0c6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:54:51.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.912+0000 7f5bea7fc700 1 -- 172.21.15.7:0/2317142463 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec154f70 con 0x7f5bec0a5d20 2023-03-03T12:54:51.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.912+0000 7f5beb7fe700 1 --2- 172.21.15.7:0/2317142463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec157f10 0x7f5bec158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:54:51.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.912+0000 7f5bea7fc700 1 --2- 172.21.15.7:0/2317142463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec0a5d20 0x7f5bec0a6190 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0098f0 tx=0x7f5bdc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bccbd12adef6c9a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:54:51.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.912+0000 7f5be9ffb700 1 -- 172.21.15.7:0/2317142463 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc00b8a0 con 0x7f5bec0a5d20 2023-03-03T12:54:51.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.912+0000 7f5be9ffb700 1 -- 172.21.15.7:0/2317142463 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc00ba00 con 0x7f5bec0a5d20 2023-03-03T12:54:51.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.912+0000 7f5be9ffb700 1 -- 172.21.15.7:0/2317142463 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc00bd90 con 0x7f5bec0a5d20 2023-03-03T12:54:51.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.913+0000 7f5bf1815700 1 -- 172.21.15.7:0/2317142463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec0a5d20 msgr2=0x7f5bec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:54:51.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.913+0000 7f5bf1815700 1 --2- 172.21.15.7:0/2317142463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec0a5d20 0x7f5bec0a6190 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0098f0 tx=0x7f5bdc009c00 comp rx=0 tx=0).stop 2023-03-03T12:54:51.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.913+0000 7f5bf1815700 1 -- 172.21.15.7:0/2317142463 shutdown_connections 2023-03-03T12:54:51.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.913+0000 7f5bf1815700 1 --2- 172.21.15.7:0/2317142463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec157f10 0x7f5bec158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:54:51.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.913+0000 7f5bf1815700 1 --2- 172.21.15.7:0/2317142463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec0a5d20 0x7f5bec0a6190 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:54:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.913+0000 7f5bf1815700 1 -- 172.21.15.7:0/2317142463 >> 172.21.15.7:0/2317142463 conn(0x7f5bec06a490 msgr2=0x7f5bec06a860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:54:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.913+0000 7f5bf1815700 1 -- 172.21.15.7:0/2317142463 shutdown_connections 2023-03-03T12:54:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.913+0000 7f5bf1815700 1 -- 172.21.15.7:0/2317142463 wait complete. 2023-03-03T12:54:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.914+0000 7f5bf1815700 1 Processor -- start 2023-03-03T12:54:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.914+0000 7f5bf1815700 1 -- start start 2023-03-03T12:54:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bf1815700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec0a5d20 0x7f5bec150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:54:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bf1815700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec151e30 0x7f5bec149960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:54:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bf1815700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bec157f10 0x7f5bec1562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:54:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bf1815700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec150d10 con 0x7f5bec151e30 2023-03-03T12:54:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bf1815700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec150ff0 con 0x7f5bec157f10 2023-03-03T12:54:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bf1815700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec1512d0 con 0x7f5bec0a5d20 2023-03-03T12:54:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bea7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec151e30 0x7f5bec149960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:54:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bea7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec151e30 0x7f5bec149960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42428/0 (socket says 172.21.15.7:42428) 2023-03-03T12:54:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bea7fc700 1 -- 172.21.15.7:0/80897018 learned_addr learned my addr 172.21.15.7:0/80897018 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:54:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5beb7fe700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bec157f10 0x7f5bec1562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:54:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5beaffd700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec0a5d20 0x7f5bec150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:54:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bea7fc700 1 -- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec0a5d20 msgr2=0x7f5bec150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:54:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bea7fc700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec0a5d20 0x7f5bec150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:54:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bea7fc700 1 -- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bec157f10 msgr2=0x7f5bec1562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:54:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bea7fc700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bec157f10 0x7f5bec1562b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:54:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bea7fc700 1 -- 172.21.15.7:0/80897018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc009550 con 0x7f5bec151e30 2023-03-03T12:54:51.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.915+0000 7f5bea7fc700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec151e30 0x7f5bec149960 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009130 tx=0x7f5bdc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:54:51.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.916+0000 7f5bd3fff700 1 -- 172.21.15.7:0/80897018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc0036a0 con 0x7f5bec151e30 2023-03-03T12:54:51.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.916+0000 7f5bd3fff700 1 -- 172.21.15.7:0/80897018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc01e070 con 0x7f5bec151e30 2023-03-03T12:54:51.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.916+0000 7f5bf1815700 1 -- 172.21.15.7:0/80897018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bec154680 con 0x7f5bec151e30 2023-03-03T12:54:51.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.916+0000 7f5bd3fff700 1 -- 172.21.15.7:0/80897018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc003a40 con 0x7f5bec151e30 2023-03-03T12:54:51.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.916+0000 7f5bf1815700 1 -- 172.21.15.7:0/80897018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bec155250 con 0x7f5bec151e30 2023-03-03T12:54:51.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.917+0000 7f5bf1815700 1 -- 172.21.15.7:0/80897018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bec0c5770 con 0x7f5bec151e30 2023-03-03T12:54:51.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.917+0000 7f5bd3fff700 1 -- 172.21.15.7:0/80897018 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5bdc00f050 con 0x7f5bec151e30 2023-03-03T12:54:51.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.918+0000 7f5bd3fff700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd80682f0 0x7f5bd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:54:51.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.918+0000 7f5bd3fff700 1 -- 172.21.15.7:0/80897018 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5bdc020070 con 0x7f5bec151e30 2023-03-03T12:54:51.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.918+0000 7f5beaffd700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd80682f0 0x7f5bd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:54:51.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.918+0000 7f5beaffd700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd80682f0 0x7f5bd806a7a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f5bec157d10 tx=0x7f5be0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:54:51.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:51.922+0000 7f5bd3fff700 1 -- 172.21.15.7:0/80897018 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5bdc0539f0 con 0x7f5bec151e30 2023-03-03T12:54:52.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.165+0000 7f5bf1815700 1 -- 172.21.15.7:0/80897018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bec074bb0 con 0x7f5bec151e30 2023-03-03T12:54:52.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.166+0000 7f5bd3fff700 1 -- 172.21.15.7:0/80897018 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5bdc057010 con 0x7f5bec151e30 2023-03-03T12:54:52.170 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:54:52.170 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:54:52.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.167+0000 7f5bf1815700 1 -- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd80682f0 msgr2=0x7f5bd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:54:52.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.167+0000 7f5bf1815700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd80682f0 0x7f5bd806a7a0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f5bec157d10 tx=0x7f5be0009040 comp rx=0 tx=0).stop 2023-03-03T12:54:52.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.167+0000 7f5bf1815700 1 -- 172.21.15.7:0/80897018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec151e30 msgr2=0x7f5bec149960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:54:52.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.167+0000 7f5bf1815700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec151e30 0x7f5bec149960 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009130 tx=0x7f5bdc004750 comp rx=0 tx=0).stop 2023-03-03T12:54:52.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.168+0000 7f5bf1815700 1 -- 172.21.15.7:0/80897018 shutdown_connections 2023-03-03T12:54:52.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.168+0000 7f5bf1815700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bec157f10 0x7f5bec1562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:54:52.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.168+0000 7f5bf1815700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5bec0a5d20 0x7f5bec150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:54:52.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.168+0000 7f5bf1815700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5bd80682f0 0x7f5bd806a7a0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:54:52.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.168+0000 7f5bf1815700 1 --2- 172.21.15.7:0/80897018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bec151e30 0x7f5bec149960 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:54:52.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.168+0000 7f5bf1815700 1 -- 172.21.15.7:0/80897018 >> 172.21.15.7:0/80897018 conn(0x7f5bec06a490 msgr2=0x7f5bec0c13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:54:52.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.168+0000 7f5bf1815700 1 -- 172.21.15.7:0/80897018 shutdown_connections 2023-03-03T12:54:52.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:54:52.168+0000 7f5bf1815700 1 -- 172.21.15.7:0/80897018 wait complete. 2023-03-03T12:54:52.173 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:54:52.181 INFO:tasks.mds_thrash.fs.[a]:waiting for 101 secs before thrashing 2023-03-03T12:55:05.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:05.048 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:26.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:55:26.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.193+0000 7fede7681700 1 Processor -- start 2023-03-03T12:55:26.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.194+0000 7fede7681700 1 -- start start 2023-03-03T12:55:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.194+0000 7fede7681700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fede0151cd0 0x7fede0154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.194+0000 7fede7681700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede006c6c0 0x7fede00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.194+0000 7fede7681700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede0157ff0 con 0x7fede00c2e30 2023-03-03T12:55:26.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.194+0000 7fede7681700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede006d640 con 0x7fede0151cd0 2023-03-03T12:55:26.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.194+0000 7fede7681700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede006cda0 con 0x7fede006c6c0 2023-03-03T12:55:26.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede541d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fede0151cd0 0x7fede0154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:26.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede4c1c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fede00c2e30 0x7fede0158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36442/0 (socket says 172.21.15.7:36442) 2023-03-03T12:55:26.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede4c1c700 1 -- 172.21.15.7:0/802033616 learned_addr learned my addr 172.21.15.7:0/802033616 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:55:26.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede541d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fede0151cd0 0x7fede0154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42460/0 (socket says 172.21.15.7:42460) 2023-03-03T12:55:26.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede5c1e700 1 --2- 172.21.15.7:0/802033616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede006c6c0 0x7fede00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:26.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede541d700 1 -- 172.21.15.7:0/802033616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede006c6c0 msgr2=0x7fede00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:55:26.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede541d700 1 --2- 172.21.15.7:0/802033616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede006c6c0 0x7fede00736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:26.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede541d700 1 -- 172.21.15.7:0/802033616 >> v1:172.21.15.7:6789/0 conn(0x7fede00c2e30 legacy=0x7fede0158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:26.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede541d700 1 -- 172.21.15.7:0/802033616 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede006d080 con 0x7fede0151cd0 2023-03-03T12:55:26.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fedd3fff700 1 -- 172.21.15.7:0/802033616 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525873134 0 0) 0x7fede0157ff0 con 0x7fede00c2e30 2023-03-03T12:55:26.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede5c1e700 1 --2- 172.21.15.7:0/802033616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede006c6c0 0x7fede00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:55:26.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fede541d700 1 --2- 172.21.15.7:0/802033616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede0151cd0 0x7fede0154510 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7feddc009840 tx=0x7feddc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a5adafa7c91610 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:55:26.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fedd3fff700 1 -- 172.21.15.7:0/802033616 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc011830 con 0x7fede0151cd0 2023-03-03T12:55:26.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fedd3fff700 1 -- 172.21.15.7:0/802033616 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feddc011990 con 0x7fede0151cd0 2023-03-03T12:55:26.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.195+0000 7fedd3fff700 1 -- 172.21.15.7:0/802033616 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc011d20 con 0x7fede0151cd0 2023-03-03T12:55:26.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.196+0000 7fede7681700 1 -- 172.21.15.7:0/802033616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede0151cd0 msgr2=0x7fede0154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:26.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.196+0000 7fede7681700 1 --2- 172.21.15.7:0/802033616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede0151cd0 0x7fede0154510 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7feddc009840 tx=0x7feddc009b50 comp rx=0 tx=0).stop 2023-03-03T12:55:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.196+0000 7fede7681700 1 -- 172.21.15.7:0/802033616 shutdown_connections 2023-03-03T12:55:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.196+0000 7fede7681700 1 --2- 172.21.15.7:0/802033616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede006c6c0 0x7fede00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.196+0000 7fede7681700 1 --2- 172.21.15.7:0/802033616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede0151cd0 0x7fede0154510 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.196+0000 7fede7681700 1 -- 172.21.15.7:0/802033616 >> 172.21.15.7:0/802033616 conn(0x7fede00a94d0 msgr2=0x7fede00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:55:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.196+0000 7fede7681700 1 -- 172.21.15.7:0/802033616 shutdown_connections 2023-03-03T12:55:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.196+0000 7fede7681700 1 -- 172.21.15.7:0/802033616 wait complete. 2023-03-03T12:55:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.198+0000 7fede7681700 1 Processor -- start 2023-03-03T12:55:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.198+0000 7fede7681700 1 -- start start 2023-03-03T12:55:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.198+0000 7fede7681700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede006c6c0 0x7fede01570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.198+0000 7fede7681700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede00c2e30 0x7fede01edb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede7681700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fede0151cd0 0x7fede01ee090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede7681700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede01ed6c0 con 0x7fede006c6c0 2023-03-03T12:55:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede7681700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede01507c0 con 0x7fede0151cd0 2023-03-03T12:55:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede7681700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede0150aa0 con 0x7fede00c2e30 2023-03-03T12:55:26.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede541d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede006c6c0 0x7fede01570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:26.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede541d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede006c6c0 0x7fede01570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42466/0 (socket says 172.21.15.7:42466) 2023-03-03T12:55:26.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede541d700 1 -- 172.21.15.7:0/2160493900 learned_addr learned my addr 172.21.15.7:0/2160493900 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:55:26.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede541d700 1 -- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede00c2e30 msgr2=0x7fede01edb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:26.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede4c1c700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede00c2e30 0x7fede01edb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:26.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede541d700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede00c2e30 0x7fede01edb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:26.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede541d700 1 -- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fede0151cd0 msgr2=0x7fede01ee090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:26.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede5c1e700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fede0151cd0 0x7fede01ee090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:26.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede541d700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fede0151cd0 0x7fede01ee090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:26.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede541d700 1 -- 172.21.15.7:0/2160493900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc0094a0 con 0x7fede006c6c0 2023-03-03T12:55:26.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede541d700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede006c6c0 0x7fede01570d0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7feddc000c00 tx=0x7feddc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:55:26.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fedd1ffb700 1 -- 172.21.15.7:0/2160493900 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc0038a0 con 0x7fede006c6c0 2023-03-03T12:55:26.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fedd1ffb700 1 -- 172.21.15.7:0/2160493900 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feddc01e070 con 0x7fede006c6c0 2023-03-03T12:55:26.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fede7681700 1 -- 172.21.15.7:0/2160493900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede0157c80 con 0x7fede006c6c0 2023-03-03T12:55:26.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.199+0000 7fedd1ffb700 1 -- 172.21.15.7:0/2160493900 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc003c60 con 0x7fede006c6c0 2023-03-03T12:55:26.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.200+0000 7fede7681700 1 -- 172.21.15.7:0/2160493900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede006d360 con 0x7fede006c6c0 2023-03-03T12:55:26.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.200+0000 7fede7681700 1 -- 172.21.15.7:0/2160493900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede0154a50 con 0x7fede006c6c0 2023-03-03T12:55:26.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.201+0000 7fedd1ffb700 1 -- 172.21.15.7:0/2160493900 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feddc00f050 con 0x7fede006c6c0 2023-03-03T12:55:26.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.201+0000 7fedd1ffb700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fedc40682f0 0x7fedc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:55:26.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.201+0000 7fedd1ffb700 1 -- 172.21.15.7:0/2160493900 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7feddc0872b0 con 0x7fede006c6c0 2023-03-03T12:55:26.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.202+0000 7fede4c1c700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fedc40682f0 0x7fedc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:55:26.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.202+0000 7fede4c1c700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fedc40682f0 0x7fedc406a7a0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fede014ec70 tx=0x7fedd4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:55:26.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.206+0000 7fedd1ffb700 1 -- 172.21.15.7:0/2160493900 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feddc053650 con 0x7fede006c6c0 2023-03-03T12:55:26.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.442+0000 7fede7681700 1 -- 172.21.15.7:0/2160493900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fede01ed9a0 con 0x7fede006c6c0 2023-03-03T12:55:26.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.442+0000 7fedd1ffb700 1 -- 172.21.15.7:0/2160493900 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7feddc020090 con 0x7fede006c6c0 2023-03-03T12:55:26.445 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:55:26.445 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:55:26.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 -- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fedc40682f0 msgr2=0x7fedc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:26.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fedc40682f0 0x7fedc406a7a0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fede014ec70 tx=0x7fedd4009040 comp rx=0 tx=0).stop 2023-03-03T12:55:26.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 -- 172.21.15.7:0/2160493900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede006c6c0 msgr2=0x7fede01570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:55:26.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede006c6c0 0x7fede01570d0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7feddc000c00 tx=0x7feddc004750 comp rx=0 tx=0).stop 2023-03-03T12:55:26.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 -- 172.21.15.7:0/2160493900 shutdown_connections 2023-03-03T12:55:26.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fede0151cd0 0x7fede01ee090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:26.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fedc40682f0 0x7fedc406a7a0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:26.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fede006c6c0 0x7fede01570d0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:26.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 --2- 172.21.15.7:0/2160493900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fede00c2e30 0x7fede01edb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:55:26.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 -- 172.21.15.7:0/2160493900 >> 172.21.15.7:0/2160493900 conn(0x7fede00a94d0 msgr2=0x7fede00736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:55:26.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 -- 172.21.15.7:0/2160493900 shutdown_connections 2023-03-03T12:55:26.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:55:26.444+0000 7fede7681700 1 -- 172.21.15.7:0/2160493900 wait complete. 2023-03-03T12:55:26.448 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:55:26.458 INFO:tasks.mds_thrash.fs.[b]:waiting for 54 secs before thrashing 2023-03-03T12:55:35.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:55:35.119 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:05.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:05.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:20.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:56:20.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.596+0000 7f91cdd21700 1 Processor -- start 2023-03-03T12:56:20.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.596+0000 7f91cdd21700 1 -- start start 2023-03-03T12:56:20.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.597+0000 7f91cdd21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91c80c20d0 0x7f91c81562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:20.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.597+0000 7f91cdd21700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91c80c27c0 0x7f91c80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:20.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.597+0000 7f91cdd21700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c8155e60 con 0x7f91c814f860 2023-03-03T12:56:20.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.597+0000 7f91cdd21700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c806d350 con 0x7f91c80c27c0 2023-03-03T12:56:20.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.597+0000 7f91cdd21700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c81558e0 con 0x7f91c80c20d0 2023-03-03T12:56:20.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.597+0000 7f91c7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91c80c27c0 0x7f91c80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.597+0000 7f91c77fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f91c814f860 0x7f91c8152d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36492/0 (socket says 172.21.15.7:36492) 2023-03-03T12:56:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.597+0000 7f91c77fe700 1 -- 172.21.15.7:0/1420395283 learned_addr learned my addr 172.21.15.7:0/1420395283 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:56:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.598+0000 7f91c6ffd700 1 --2- 172.21.15.7:0/1420395283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91c80c20d0 0x7f91c81562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.598+0000 7f91c7fff700 1 -- 172.21.15.7:0/1420395283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91c80c20d0 msgr2=0x7f91c81562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:56:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.598+0000 7f91c7fff700 1 --2- 172.21.15.7:0/1420395283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91c80c20d0 0x7f91c81562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.598+0000 7f91c7fff700 1 -- 172.21.15.7:0/1420395283 >> v1:172.21.15.7:6789/0 conn(0x7f91c814f860 legacy=0x7f91c8152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.598+0000 7f91c67fc700 1 -- 172.21.15.7:0/1420395283 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306271568 0 0) 0x7f91c8155e60 con 0x7f91c814f860 2023-03-03T12:56:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.598+0000 7f91c7fff700 1 -- 172.21.15.7:0/1420395283 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c806cd90 con 0x7f91c80c27c0 2023-03-03T12:56:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.598+0000 7f91c7fff700 1 --2- 172.21.15.7:0/1420395283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c80c27c0 0x7f91c80736f0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f91c0009d90 tx=0x7f91c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0c4b98643fa454 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.598+0000 7f91c67fc700 1 -- 172.21.15.7:0/1420395283 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c000e070 con 0x7f91c80c27c0 2023-03-03T12:56:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.598+0000 7f91c67fc700 1 -- 172.21.15.7:0/1420395283 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c000bbd0 con 0x7f91c80c27c0 2023-03-03T12:56:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.598+0000 7f91c67fc700 1 -- 172.21.15.7:0/1420395283 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c0012430 con 0x7f91c80c27c0 2023-03-03T12:56:20.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.599+0000 7f91cdd21700 1 -- 172.21.15.7:0/1420395283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c80c27c0 msgr2=0x7f91c80736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:20.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.599+0000 7f91cdd21700 1 --2- 172.21.15.7:0/1420395283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c80c27c0 0x7f91c80736f0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f91c0009d90 tx=0x7f91c000b330 comp rx=0 tx=0).stop 2023-03-03T12:56:20.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.599+0000 7f91cdd21700 1 -- 172.21.15.7:0/1420395283 shutdown_connections 2023-03-03T12:56:20.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.599+0000 7f91cdd21700 1 --2- 172.21.15.7:0/1420395283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c80c27c0 0x7f91c80736f0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:20.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.599+0000 7f91cdd21700 1 --2- 172.21.15.7:0/1420395283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91c80c20d0 0x7f91c81562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:20.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.599+0000 7f91cdd21700 1 -- 172.21.15.7:0/1420395283 >> 172.21.15.7:0/1420395283 conn(0x7f91c80a94d0 msgr2=0x7f91c80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:56:20.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.599+0000 7f91cdd21700 1 -- 172.21.15.7:0/1420395283 shutdown_connections 2023-03-03T12:56:20.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.599+0000 7f91cdd21700 1 -- 172.21.15.7:0/1420395283 wait complete. 2023-03-03T12:56:20.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.600+0000 7f91cdd21700 1 Processor -- start 2023-03-03T12:56:20.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.600+0000 7f91cdd21700 1 -- start start 2023-03-03T12:56:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91cdd21700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91c80c20d0 0x7f91c81eded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91cdd21700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91c80c27c0 0x7f91c8154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91cdd21700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c814f860 0x7f91c81db950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91cdd21700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c80ab950 con 0x7f91c814f860 2023-03-03T12:56:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91cdd21700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c80ab5b0 con 0x7f91c80c20d0 2023-03-03T12:56:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91cdd21700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c80ac4f0 con 0x7f91c80c27c0 2023-03-03T12:56:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c814f860 0x7f91c81db950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c814f860 0x7f91c81db950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42520/0 (socket says 172.21.15.7:42520) 2023-03-03T12:56:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c7fff700 1 -- 172.21.15.7:0/1979740426 learned_addr learned my addr 172.21.15.7:0/1979740426 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:56:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c6ffd700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91c80c27c0 0x7f91c8154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c77fe700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91c80c20d0 0x7f91c81eded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c7fff700 1 -- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91c80c27c0 msgr2=0x7f91c8154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c7fff700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91c80c27c0 0x7f91c8154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c7fff700 1 -- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91c80c20d0 msgr2=0x7f91c81eded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c7fff700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91c80c20d0 0x7f91c81eded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:20.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c7fff700 1 -- 172.21.15.7:0/1979740426 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c0009a70 con 0x7f91c814f860 2023-03-03T12:56:20.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.601+0000 7f91c7fff700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c814f860 0x7f91c81db950 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f91c000b370 tx=0x7f91c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:20.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.602+0000 7f91bffff700 1 -- 172.21.15.7:0/1979740426 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c0012430 con 0x7f91c814f860 2023-03-03T12:56:20.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.602+0000 7f91bffff700 1 -- 172.21.15.7:0/1979740426 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91c0020070 con 0x7f91c814f860 2023-03-03T12:56:20.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.602+0000 7f91cdd21700 1 -- 172.21.15.7:0/1979740426 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c8155af0 con 0x7f91c814f860 2023-03-03T12:56:20.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.602+0000 7f91bffff700 1 -- 172.21.15.7:0/1979740426 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c0004280 con 0x7f91c814f860 2023-03-03T12:56:20.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.602+0000 7f91cdd21700 1 -- 172.21.15.7:0/1979740426 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c806d070 con 0x7f91c814f860 2023-03-03T12:56:20.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.603+0000 7f91cdd21700 1 -- 172.21.15.7:0/1979740426 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c814fc30 con 0x7f91c814f860 2023-03-03T12:56:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.603+0000 7f91bffff700 1 -- 172.21.15.7:0/1979740426 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f91c00043e0 con 0x7f91c814f860 2023-03-03T12:56:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.603+0000 7f91bffff700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91a80682c0 0x7f91a806a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.604+0000 7f91bffff700 1 -- 172.21.15.7:0/1979740426 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f91c0022070 con 0x7f91c814f860 2023-03-03T12:56:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.604+0000 7f91c77fe700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91a80682c0 0x7f91a806a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.604+0000 7f91c77fe700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91a80682c0 0x7f91a806a770 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f91b8000f60 tx=0x7f91b8007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.608+0000 7f91bffff700 1 -- 172.21.15.7:0/1979740426 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f91c0054ab0 con 0x7f91c814f860 2023-03-03T12:56:20.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.847+0000 7f91cdd21700 1 -- 172.21.15.7:0/1979740426 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91c8074bb0 con 0x7f91c814f860 2023-03-03T12:56:20.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.847+0000 7f91bffff700 1 -- 172.21.15.7:0/1979740426 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f91c00580d0 con 0x7f91c814f860 2023-03-03T12:56:20.849 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:56:20.850 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:56:20.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.849+0000 7f91cdd21700 1 -- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91a80682c0 msgr2=0x7f91a806a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:20.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.849+0000 7f91cdd21700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91a80682c0 0x7f91a806a770 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f91b8000f60 tx=0x7f91b8007c90 comp rx=0 tx=0).stop 2023-03-03T12:56:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.849+0000 7f91cdd21700 1 -- 172.21.15.7:0/1979740426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c814f860 msgr2=0x7f91c81db950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.849+0000 7f91cdd21700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c814f860 0x7f91c81db950 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f91c000b370 tx=0x7f91c0004ad0 comp rx=0 tx=0).stop 2023-03-03T12:56:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.849+0000 7f91cdd21700 1 -- 172.21.15.7:0/1979740426 shutdown_connections 2023-03-03T12:56:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.849+0000 7f91cdd21700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91a80682c0 0x7f91a806a770 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.849+0000 7f91cdd21700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91c814f860 0x7f91c81db950 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.849+0000 7f91cdd21700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91c80c20d0 0x7f91c81eded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:20.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.849+0000 7f91cdd21700 1 --2- 172.21.15.7:0/1979740426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91c80c27c0 0x7f91c8154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:20.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.849+0000 7f91cdd21700 1 -- 172.21.15.7:0/1979740426 >> 172.21.15.7:0/1979740426 conn(0x7f91c80a94d0 msgr2=0x7f91c81536e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:56:20.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.850+0000 7f91cdd21700 1 -- 172.21.15.7:0/1979740426 shutdown_connections 2023-03-03T12:56:20.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:20.850+0000 7f91cdd21700 1 -- 172.21.15.7:0/1979740426 wait complete. 2023-03-03T12:56:20.853 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:56:20.863 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2023-03-03T12:56:33.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:56:33.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.320+0000 7fbdefdaa700 1 Processor -- start 2023-03-03T12:56:33.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.320+0000 7fbdefdaa700 1 -- start start 2023-03-03T12:56:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdefdaa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbde814f9d0 0x7fbde8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdefdaa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde814fda0 0x7fbde8071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdefdaa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde80c2f30 con 0x7fbde80abe20 2023-03-03T12:56:33.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdefdaa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde80c31d0 con 0x7fbde814f9d0 2023-03-03T12:56:33.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdefdaa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde81524a0 con 0x7fbde814fda0 2023-03-03T12:56:33.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdedb46700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbde814f9d0 0x7fbde8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:33.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdee347700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbde80abe20 0x7fbde81529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36520/0 (socket says 172.21.15.7:36520) 2023-03-03T12:56:33.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdee347700 1 -- 172.21.15.7:0/1112692132 learned_addr learned my addr 172.21.15.7:0/1112692132 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:56:33.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbded345700 1 --2- 172.21.15.7:0/1112692132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde814fda0 0x7fbde8071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:33.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdecb44700 1 -- 172.21.15.7:0/1112692132 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105786338 0 0) 0x7fbde80c2f30 con 0x7fbde80abe20 2023-03-03T12:56:33.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdecb44700 1 -- 172.21.15.7:0/1112692132 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd40035b0 con 0x7fbde80abe20 2023-03-03T12:56:33.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdedb46700 1 -- 172.21.15.7:0/1112692132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde814fda0 msgr2=0x7fbde8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:56:33.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdedb46700 1 --2- 172.21.15.7:0/1112692132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde814fda0 0x7fbde8071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:33.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdedb46700 1 -- 172.21.15.7:0/1112692132 >> v1:172.21.15.7:6789/0 conn(0x7fbde80abe20 legacy=0x7fbde81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:33.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdedb46700 1 -- 172.21.15.7:0/1112692132 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde80c34b0 con 0x7fbde814f9d0 2023-03-03T12:56:33.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.321+0000 7fbdedb46700 1 --2- 172.21.15.7:0/1112692132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 0x7fbde8156430 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009840 tx=0x7fbde4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999a3440cf52c6e4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:33.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdecb44700 1 -- 172.21.15.7:0/1112692132 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4011830 con 0x7fbde814f9d0 2023-03-03T12:56:33.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdecb44700 1 -- 172.21.15.7:0/1112692132 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde4011990 con 0x7fbde814f9d0 2023-03-03T12:56:33.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdecb44700 1 -- 172.21.15.7:0/1112692132 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4011d20 con 0x7fbde814f9d0 2023-03-03T12:56:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdefdaa700 1 -- 172.21.15.7:0/1112692132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 msgr2=0x7fbde8156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdefdaa700 1 --2- 172.21.15.7:0/1112692132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 0x7fbde8156430 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009840 tx=0x7fbde4009b50 comp rx=0 tx=0).stop 2023-03-03T12:56:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdefdaa700 1 -- 172.21.15.7:0/1112692132 shutdown_connections 2023-03-03T12:56:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdefdaa700 1 --2- 172.21.15.7:0/1112692132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 0x7fbde8156430 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdefdaa700 1 --2- 172.21.15.7:0/1112692132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde814fda0 0x7fbde8071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdefdaa700 1 -- 172.21.15.7:0/1112692132 >> 172.21.15.7:0/1112692132 conn(0x7fbde80c8900 msgr2=0x7fbde80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:56:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdefdaa700 1 -- 172.21.15.7:0/1112692132 shutdown_connections 2023-03-03T12:56:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.322+0000 7fbdefdaa700 1 -- 172.21.15.7:0/1112692132 wait complete. 2023-03-03T12:56:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.324+0000 7fbdefdaa700 1 Processor -- start 2023-03-03T12:56:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.324+0000 7fbdefdaa700 1 -- start start 2023-03-03T12:56:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbdefdaa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbde80abe20 0x7fbde806f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbdefdaa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 0x7fbde806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbdefdaa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde814fda0 0x7fbde814b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbdefdaa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde806eaf0 con 0x7fbde814f9d0 2023-03-03T12:56:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbdefdaa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde806e810 con 0x7fbde80abe20 2023-03-03T12:56:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbdefdaa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde81e7570 con 0x7fbde814fda0 2023-03-03T12:56:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbded345700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 0x7fbde806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbded345700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 0x7fbde806c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42546/0 (socket says 172.21.15.7:42546) 2023-03-03T12:56:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbded345700 1 -- 172.21.15.7:0/269964109 learned_addr learned my addr 172.21.15.7:0/269964109 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:56:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbdee347700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde814fda0 0x7fbde814b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbdedb46700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbde80abe20 0x7fbde806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbded345700 1 -- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde814fda0 msgr2=0x7fbde814b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbded345700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde814fda0 0x7fbde814b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:33.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbded345700 1 -- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbde80abe20 msgr2=0x7fbde806f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:33.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbded345700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbde80abe20 0x7fbde806f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:33.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbded345700 1 -- 172.21.15.7:0/269964109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde40094a0 con 0x7fbde814f9d0 2023-03-03T12:56:33.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.325+0000 7fbded345700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 0x7fbde806c4c0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800e980 tx=0x7fbdd800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:33.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.326+0000 7fbdde7fc700 1 -- 172.21.15.7:0/269964109 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8009710 con 0x7fbde814f9d0 2023-03-03T12:56:33.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.326+0000 7fbdde7fc700 1 -- 172.21.15.7:0/269964109 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd800c850 con 0x7fbde814f9d0 2023-03-03T12:56:33.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.326+0000 7fbdefdaa700 1 -- 172.21.15.7:0/269964109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde80ac350 con 0x7fbde814f9d0 2023-03-03T12:56:33.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.326+0000 7fbdde7fc700 1 -- 172.21.15.7:0/269964109 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd800cc00 con 0x7fbde814f9d0 2023-03-03T12:56:33.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.326+0000 7fbdefdaa700 1 -- 172.21.15.7:0/269964109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde80ab630 con 0x7fbde814f9d0 2023-03-03T12:56:33.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.327+0000 7fbdefdaa700 1 -- 172.21.15.7:0/269964109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde80c3e10 con 0x7fbde814f9d0 2023-03-03T12:56:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.327+0000 7fbdde7fc700 1 -- 172.21.15.7:0/269964109 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbdd8019030 con 0x7fbde814f9d0 2023-03-03T12:56:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.327+0000 7fbdde7fc700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdcc0682f0 0x7fbdcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:56:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.327+0000 7fbdde7fc700 1 -- 172.21.15.7:0/269964109 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbdd8086120 con 0x7fbde814f9d0 2023-03-03T12:56:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.328+0000 7fbdedb46700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdcc0682f0 0x7fbdcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:56:33.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.328+0000 7fbdedb46700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdcc0682f0 0x7fbdcc06a7a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fbde40090b0 tx=0x7fbde4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:56:33.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.332+0000 7fbdde7fc700 1 -- 172.21.15.7:0/269964109 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbdd8052580 con 0x7fbde814f9d0 2023-03-03T12:56:33.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.571+0000 7fbdefdaa700 1 -- 172.21.15.7:0/269964109 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbde81e73d0 con 0x7fbde814f9d0 2023-03-03T12:56:33.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.572+0000 7fbdde7fc700 1 -- 172.21.15.7:0/269964109 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbde81e73d0 con 0x7fbde814f9d0 2023-03-03T12:56:33.573 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:56:33.574 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:56:33.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 -- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdcc0682f0 msgr2=0x7fbdcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdcc0682f0 0x7fbdcc06a7a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fbde40090b0 tx=0x7fbde4008030 comp rx=0 tx=0).stop 2023-03-03T12:56:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 -- 172.21.15.7:0/269964109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 msgr2=0x7fbde806c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:56:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 0x7fbde806c4c0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800e980 tx=0x7fbdd800ed40 comp rx=0 tx=0).stop 2023-03-03T12:56:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 -- 172.21.15.7:0/269964109 shutdown_connections 2023-03-03T12:56:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde814fda0 0x7fbde814b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbde80abe20 0x7fbde806f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:33.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdcc0682f0 0x7fbdcc06a7a0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:33.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 --2- 172.21.15.7:0/269964109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde814f9d0 0x7fbde806c4c0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:56:33.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 -- 172.21.15.7:0/269964109 >> 172.21.15.7:0/269964109 conn(0x7fbde80c8900 msgr2=0x7fbde8151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:56:33.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 -- 172.21.15.7:0/269964109 shutdown_connections 2023-03-03T12:56:33.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:56:33.574+0000 7fbdefdaa700 1 -- 172.21.15.7:0/269964109 wait complete. 2023-03-03T12:56:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:56:33.587 INFO:tasks.mds_thrash.fs.[a]:waiting for 112 secs before thrashing 2023-03-03T12:56:35.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:56:35.371 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:05.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:05.451 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:35.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:35.573 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:57:57.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:57:58.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.000+0000 7f9a1a1ea700 1 Processor -- start 2023-03-03T12:57:58.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.000+0000 7f9a1a1ea700 1 -- start start 2023-03-03T12:57:58.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a1a1ea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a140c00e0 0x7f9a140c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:57:58.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a1a1ea700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a140c7310 con 0x7f9a14154ff0 2023-03-03T12:57:58.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a1a1ea700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a140bfd10 con 0x7f9a140caa30 2023-03-03T12:57:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a1a1ea700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a14155430 con 0x7f9a140c00e0 2023-03-03T12:57:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a13fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9a14154ff0 0x7f9a1415ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36596/0 (socket says 172.21.15.7:36596) 2023-03-03T12:57:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a12ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a140c00e0 0x7f9a140c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:57:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a13fff700 1 -- 172.21.15.7:0/3893405363 learned_addr learned my addr 172.21.15.7:0/3893405363 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:57:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a12ffd700 1 -- 172.21.15.7:0/3893405363 >> v1:172.21.15.143:6789/0 conn(0x7f9a140caa30 legacy=0x7f9a140c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:57:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a12ffd700 1 -- 172.21.15.7:0/3893405363 >> v1:172.21.15.7:6789/0 conn(0x7f9a14154ff0 legacy=0x7f9a1415ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:57:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a12ffd700 1 -- 172.21.15.7:0/3893405363 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a140bfa30 con 0x7f9a140c00e0 2023-03-03T12:57:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.001+0000 7f9a127fc700 1 -- 172.21.15.7:0/3893405363 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292511517 0 0) 0x7f9a140c7310 con 0x7f9a14154ff0 2023-03-03T12:57:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a12ffd700 1 --2- 172.21.15.7:0/3893405363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 0x7f9a140c5c60 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f99fc009820 tx=0x7f99fc009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c8f9f34e00e872 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:57:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a127fc700 1 -- 172.21.15.7:0/3893405363 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc005e30 con 0x7f9a140c00e0 2023-03-03T12:57:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a127fc700 1 -- 172.21.15.7:0/3893405363 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99fc006030 con 0x7f9a140c00e0 2023-03-03T12:57:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a127fc700 1 -- 172.21.15.7:0/3893405363 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc0063e0 con 0x7f9a140c00e0 2023-03-03T12:57:58.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/3893405363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 msgr2=0x7f9a140c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:57:58.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a1a1ea700 1 --2- 172.21.15.7:0/3893405363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 0x7f9a140c5c60 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f99fc009820 tx=0x7f99fc009b30 comp rx=0 tx=0).stop 2023-03-03T12:57:58.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/3893405363 shutdown_connections 2023-03-03T12:57:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a1a1ea700 1 --2- 172.21.15.7:0/3893405363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 0x7f9a140c5c60 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:57:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/3893405363 >> 172.21.15.7:0/3893405363 conn(0x7f9a14066e40 msgr2=0x7f9a14071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:57:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/3893405363 shutdown_connections 2023-03-03T12:57:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.002+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/3893405363 wait complete. 2023-03-03T12:57:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.004+0000 7f9a1a1ea700 1 Processor -- start 2023-03-03T12:57:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.004+0000 7f9a1a1ea700 1 -- start start 2023-03-03T12:57:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a1a1ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 0x7f9a140c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:57:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a1a1ea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a140caa30 0x7f9a1413ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:57:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a1a1ea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a14154ff0 0x7f9a1413f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:57:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a1a1ea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a141584f0 con 0x7f9a140c00e0 2023-03-03T12:57:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a1a1ea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1413f860 con 0x7f9a14154ff0 2023-03-03T12:57:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a1a1ea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a141578e0 con 0x7f9a140caa30 2023-03-03T12:57:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 0x7f9a140c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:57:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 0x7f9a140c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42618/0 (socket says 172.21.15.7:42618) 2023-03-03T12:57:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a137fe700 1 -- 172.21.15.7:0/1378390784 learned_addr learned my addr 172.21.15.7:0/1378390784 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:57:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a13fff700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a14154ff0 0x7f9a1413f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:57:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a12ffd700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a140caa30 0x7f9a1413ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:57:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a137fe700 1 -- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a140caa30 msgr2=0x7f9a1413ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:57:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a137fe700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a140caa30 0x7f9a1413ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:57:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a137fe700 1 -- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a14154ff0 msgr2=0x7f9a1413f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:57:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a137fe700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a14154ff0 0x7f9a1413f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:57:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.005+0000 7f9a137fe700 1 -- 172.21.15.7:0/1378390784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc009430 con 0x7f9a140c00e0 2023-03-03T12:57:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.006+0000 7f9a137fe700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 0x7f9a140c3e20 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400c700 tx=0x7f9a0400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:57:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.006+0000 7f9a0bfff700 1 -- 172.21.15.7:0/1378390784 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a040052d0 con 0x7f9a140c00e0 2023-03-03T12:57:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.006+0000 7f9a0bfff700 1 -- 172.21.15.7:0/1378390784 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a04005430 con 0x7f9a140c00e0 2023-03-03T12:57:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.006+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/1378390784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a141547f0 con 0x7f9a140c00e0 2023-03-03T12:57:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.006+0000 7f9a0bfff700 1 -- 172.21.15.7:0/1378390784 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a04003880 con 0x7f9a140c00e0 2023-03-03T12:57:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.006+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/1378390784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a14154510 con 0x7f9a140c00e0 2023-03-03T12:57:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.007+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/1378390784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a14154ad0 con 0x7f9a140c00e0 2023-03-03T12:57:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.007+0000 7f9a0bfff700 1 -- 172.21.15.7:0/1378390784 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a04019030 con 0x7f9a140c00e0 2023-03-03T12:57:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.008+0000 7f9a0bfff700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99f40682f0 0x7f99f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:57:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.008+0000 7f9a0bfff700 1 -- 172.21.15.7:0/1378390784 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9a04085460 con 0x7f9a140c00e0 2023-03-03T12:57:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.008+0000 7f9a12ffd700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99f40682f0 0x7f99f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:57:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.009+0000 7f9a12ffd700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99f40682f0 0x7f99f406a7a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f99fc013040 tx=0x7f99fc008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:57:58.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.012+0000 7f9a0bfff700 1 -- 172.21.15.7:0/1378390784 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a0404b5e0 con 0x7f9a140c00e0 2023-03-03T12:57:58.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.248+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/1378390784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a14074bb0 con 0x7f9a140c00e0 2023-03-03T12:57:58.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.249+0000 7f9a0bfff700 1 -- 172.21.15.7:0/1378390784 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9a040517d0 con 0x7f9a140c00e0 2023-03-03T12:57:58.252 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:57:58.252 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:57:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99f40682f0 msgr2=0x7f99f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:57:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99f40682f0 0x7f99f406a7a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f99fc013040 tx=0x7f99fc008070 comp rx=0 tx=0).stop 2023-03-03T12:57:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/1378390784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 msgr2=0x7f9a140c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:57:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 0x7f9a140c3e20 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400c700 tx=0x7f9a0400cac0 comp rx=0 tx=0).stop 2023-03-03T12:57:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/1378390784 shutdown_connections 2023-03-03T12:57:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a14154ff0 0x7f9a1413f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:57:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99f40682f0 0x7f99f406a7a0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:57:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a140c00e0 0x7f9a140c3e20 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:57:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 --2- 172.21.15.7:0/1378390784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a140caa30 0x7f9a1413ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:57:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/1378390784 >> 172.21.15.7:0/1378390784 conn(0x7f9a14066e40 msgr2=0x7f9a14071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:57:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/1378390784 shutdown_connections 2023-03-03T12:57:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:57:58.251+0000 7f9a1a1ea700 1 -- 172.21.15.7:0/1378390784 wait complete. 2023-03-03T12:57:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:57:58.266 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-03-03T12:58:05.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:05.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:25.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:58:25.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.725+0000 7f2610baa700 1 Processor -- start 2023-03-03T12:58:25.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.725+0000 7f2610baa700 1 -- start start 2023-03-03T12:58:25.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.726+0000 7f2610baa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f260c0beab0 0x7f260c155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:25.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.726+0000 7f2610baa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f260c1562c0 0x7f260c15cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.726+0000 7f2610baa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f260c1523b0 con 0x7f260c152e30 2023-03-03T12:58:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.726+0000 7f2610baa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f260c072010 con 0x7f260c0beab0 2023-03-03T12:58:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.726+0000 7f2610baa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f260c0aaaf0 con 0x7f260c1562c0 2023-03-03T12:58:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.726+0000 7f2609d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f260c0beab0 0x7f260c155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.726+0000 7f2609d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f260c0beab0 0x7f260c155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42648/0 (socket says 172.21.15.7:42648) 2023-03-03T12:58:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.726+0000 7f2609d9b700 1 -- 172.21.15.7:0/4107818823 learned_addr learned my addr 172.21.15.7:0/4107818823 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:58:25.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.726+0000 7f260ad9d700 1 --2- 172.21.15.7:0/4107818823 >> v2:172.21.15.143:3300/0 conn(0x7f260c1562c0 0x7f260c15cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:25.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.727+0000 7f2609d9b700 1 -- 172.21.15.7:0/4107818823 >> v2:172.21.15.143:3300/0 conn(0x7f260c1562c0 msgr2=0x7f260c15cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:58:25.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.727+0000 7f2609d9b700 1 --2- 172.21.15.7:0/4107818823 >> v2:172.21.15.143:3300/0 conn(0x7f260c1562c0 0x7f260c15cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:25.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.727+0000 7f2609d9b700 1 -- 172.21.15.7:0/4107818823 >> v1:172.21.15.143:6789/0 conn(0x7f260c152e30 legacy=0x7f260c0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:25.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.727+0000 7f2609d9b700 1 -- 172.21.15.7:0/4107818823 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f260c0aadd0 con 0x7f260c0beab0 2023-03-03T12:58:25.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.727+0000 7f260ad9d700 1 --2- 172.21.15.7:0/4107818823 >> v2:172.21.15.143:3300/0 conn(0x7f260c1562c0 0x7f260c15cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:58:25.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.727+0000 7f2609d9b700 1 --2- 172.21.15.7:0/4107818823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c0beab0 0x7f260c155d80 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f260c0c3530 tx=0x7f2600009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a016f95a0861ad1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:58:25.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.727+0000 7f260959a700 1 -- 172.21.15.7:0/4107818823 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260000b870 con 0x7f260c0beab0 2023-03-03T12:58:25.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.727+0000 7f260959a700 1 -- 172.21.15.7:0/4107818823 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f260000b9d0 con 0x7f260c0beab0 2023-03-03T12:58:25.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.727+0000 7f260959a700 1 -- 172.21.15.7:0/4107818823 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260000bd60 con 0x7f260c0beab0 2023-03-03T12:58:25.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.728+0000 7f2610baa700 1 -- 172.21.15.7:0/4107818823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c0beab0 msgr2=0x7f260c155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:25.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.728+0000 7f2610baa700 1 --2- 172.21.15.7:0/4107818823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c0beab0 0x7f260c155d80 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f260c0c3530 tx=0x7f2600009bd0 comp rx=0 tx=0).stop 2023-03-03T12:58:25.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.728+0000 7f2610baa700 1 -- 172.21.15.7:0/4107818823 shutdown_connections 2023-03-03T12:58:25.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.728+0000 7f2610baa700 1 --2- 172.21.15.7:0/4107818823 >> v2:172.21.15.143:3300/0 conn(0x7f260c1562c0 0x7f260c15cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.728+0000 7f2610baa700 1 --2- 172.21.15.7:0/4107818823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c0beab0 0x7f260c155d80 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.728+0000 7f2610baa700 1 -- 172.21.15.7:0/4107818823 >> 172.21.15.7:0/4107818823 conn(0x7f260c0736f0 msgr2=0x7f260c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:58:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.728+0000 7f2610baa700 1 -- 172.21.15.7:0/4107818823 shutdown_connections 2023-03-03T12:58:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.728+0000 7f2610baa700 1 -- 172.21.15.7:0/4107818823 wait complete. 2023-03-03T12:58:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.729+0000 7f2610baa700 1 Processor -- start 2023-03-03T12:58:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.729+0000 7f2610baa700 1 -- start start 2023-03-03T12:58:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2610baa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f260c0beab0 0x7f260c0c1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:25.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2610baa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c152e30 0x7f260c0c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:25.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2610baa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1562c0 0x7f260c1ea920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:25.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2610baa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c0ad650 con 0x7f260c152e30 2023-03-03T12:58:25.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2610baa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c155a00 con 0x7f260c1562c0 2023-03-03T12:58:25.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2610baa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260c1e7e80 con 0x7f260c0beab0 2023-03-03T12:58:25.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2609d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c152e30 0x7f260c0c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:25.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2609d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c152e30 0x7f260c0c4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42654/0 (socket says 172.21.15.7:42654) 2023-03-03T12:58:25.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2609d9b700 1 -- 172.21.15.7:0/2061753158 learned_addr learned my addr 172.21.15.7:0/2061753158 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:58:25.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f260a59c700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f260c0beab0 0x7f260c0c1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f260ad9d700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1562c0 0x7f260c1ea920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2609d9b700 1 -- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f260c0beab0 msgr2=0x7f260c0c1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2609d9b700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f260c0beab0 0x7f260c0c1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2609d9b700 1 -- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1562c0 msgr2=0x7f260c1ea920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2609d9b700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1562c0 0x7f260c1ea920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f2609d9b700 1 -- 172.21.15.7:0/2061753158 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2600009550 con 0x7f260c152e30 2023-03-03T12:58:25.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f260a59c700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f260c0beab0 0x7f260c0c1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:58:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.730+0000 7f260ad9d700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1562c0 0x7f260c1ea920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:58:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.731+0000 7f2609d9b700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c152e30 0x7f260c0c4360 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f2600000c00 tx=0x7f2600004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:58:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.731+0000 7f25faffd700 1 -- 172.21.15.7:0/2061753158 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2600004520 con 0x7f260c152e30 2023-03-03T12:58:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.731+0000 7f25faffd700 1 -- 172.21.15.7:0/2061753158 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f260001e070 con 0x7f260c152e30 2023-03-03T12:58:25.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.731+0000 7f2610baa700 1 -- 172.21.15.7:0/2061753158 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260c153200 con 0x7f260c152e30 2023-03-03T12:58:25.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.731+0000 7f25faffd700 1 -- 172.21.15.7:0/2061753158 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26000038a0 con 0x7f260c152e30 2023-03-03T12:58:25.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.731+0000 7f2610baa700 1 -- 172.21.15.7:0/2061753158 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260c071c70 con 0x7f260c152e30 2023-03-03T12:58:25.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.732+0000 7f2610baa700 1 -- 172.21.15.7:0/2061753158 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f260c153390 con 0x7f260c152e30 2023-03-03T12:58:25.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.732+0000 7f25faffd700 1 -- 172.21.15.7:0/2061753158 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f260000f050 con 0x7f260c152e30 2023-03-03T12:58:25.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.733+0000 7f25faffd700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25ec0682f0 0x7f25ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:58:25.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.733+0000 7f25faffd700 1 -- 172.21.15.7:0/2061753158 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2600020070 con 0x7f260c152e30 2023-03-03T12:58:25.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.733+0000 7f260a59c700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25ec0682f0 0x7f25ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:58:25.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.733+0000 7f260a59c700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25ec0682f0 0x7f25ec06a7a0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f25fc000f60 tx=0x7f25fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:58:25.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.737+0000 7f25faffd700 1 -- 172.21.15.7:0/2061753158 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2600053aa0 con 0x7f260c152e30 2023-03-03T12:58:25.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.975+0000 7f2610baa700 1 -- 172.21.15.7:0/2061753158 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f260c1e7ff0 con 0x7f260c152e30 2023-03-03T12:58:25.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.975+0000 7f25faffd700 1 -- 172.21.15.7:0/2061753158 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f26000570c0 con 0x7f260c152e30 2023-03-03T12:58:25.977 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:58:25.977 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:58:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 -- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25ec0682f0 msgr2=0x7f25ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25ec0682f0 0x7f25ec06a7a0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f25fc000f60 tx=0x7f25fc009040 comp rx=0 tx=0).stop 2023-03-03T12:58:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 -- 172.21.15.7:0/2061753158 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c152e30 msgr2=0x7f260c0c4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:58:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c152e30 0x7f260c0c4360 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f2600000c00 tx=0x7f2600004750 comp rx=0 tx=0).stop 2023-03-03T12:58:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 -- 172.21.15.7:0/2061753158 shutdown_connections 2023-03-03T12:58:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1562c0 0x7f260c1ea920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f260c0beab0 0x7f260c0c1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25ec0682f0 0x7f25ec06a7a0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 --2- 172.21.15.7:0/2061753158 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f260c152e30 0x7f260c0c4360 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:58:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 -- 172.21.15.7:0/2061753158 >> 172.21.15.7:0/2061753158 conn(0x7f260c0736f0 msgr2=0x7f260c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:58:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 -- 172.21.15.7:0/2061753158 shutdown_connections 2023-03-03T12:58:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:58:25.977+0000 7f2610baa700 1 -- 172.21.15.7:0/2061753158 wait complete. 2023-03-03T12:58:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:58:25.991 INFO:tasks.mds_thrash.fs.[a]:waiting for 73 secs before thrashing 2023-03-03T12:58:35.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:58:35.790 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:05.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:05.929 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:36.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:36.085 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T12:59:38.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:59:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.130+0000 7f4ba85ed700 1 Processor -- start 2023-03-03T12:59:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.130+0000 7f4ba85ed700 1 -- start start 2023-03-03T12:59:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.131+0000 7f4ba85ed700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ba00c17f0 0x7f4ba006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.131+0000 7f4ba85ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c5190 0x7f4ba006dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.131+0000 7f4ba85ed700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba014ebe0 con 0x7f4ba00c6010 2023-03-03T12:59:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.131+0000 7f4ba85ed700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba0152100 con 0x7f4ba00c17f0 2023-03-03T12:59:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.131+0000 7f4ba85ed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00c2e70 con 0x7f4ba00c5190 2023-03-03T12:59:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.131+0000 7f4ba5b88700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ba00c17f0 0x7f4ba006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5b88700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4ba00c17f0 0x7f4ba006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42692/0 (socket says 172.21.15.7:42692) 2023-03-03T12:59:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5b88700 1 -- 172.21.15.7:0/3387675404 learned_addr learned my addr 172.21.15.7:0/3387675404 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:59:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba6b8a700 1 --2- 172.21.15.7:0/3387675404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c5190 0x7f4ba006dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5b88700 1 -- 172.21.15.7:0/3387675404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c5190 msgr2=0x7f4ba006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:59:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5b88700 1 --2- 172.21.15.7:0/3387675404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c5190 0x7f4ba006dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5b88700 1 -- 172.21.15.7:0/3387675404 >> v1:172.21.15.143:6789/0 conn(0x7f4ba00c6010 legacy=0x7f4ba00c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5387700 1 -- 172.21.15.7:0/3387675404 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919001312 0 0) 0x7f4ba014ebe0 con 0x7f4ba00c6010 2023-03-03T12:59:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5b88700 1 -- 172.21.15.7:0/3387675404 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba00aa680 con 0x7f4ba00c17f0 2023-03-03T12:59:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5b88700 1 --2- 172.21.15.7:0/3387675404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c17f0 0x7f4ba006d6a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f4b940098f0 tx=0x7f4b94009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28108b828c2cc79e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5387700 1 -- 172.21.15.7:0/3387675404 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94011830 con 0x7f4ba00c17f0 2023-03-03T12:59:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5387700 1 -- 172.21.15.7:0/3387675404 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b94011990 con 0x7f4ba00c17f0 2023-03-03T12:59:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.132+0000 7f4ba5387700 1 -- 172.21.15.7:0/3387675404 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94011d20 con 0x7f4ba00c17f0 2023-03-03T12:59:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.133+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3387675404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c17f0 msgr2=0x7f4ba006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.133+0000 7f4ba85ed700 1 --2- 172.21.15.7:0/3387675404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c17f0 0x7f4ba006d6a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f4b940098f0 tx=0x7f4b94009c00 comp rx=0 tx=0).stop 2023-03-03T12:59:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.133+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3387675404 shutdown_connections 2023-03-03T12:59:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.133+0000 7f4ba85ed700 1 --2- 172.21.15.7:0/3387675404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c5190 0x7f4ba006dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.133+0000 7f4ba85ed700 1 --2- 172.21.15.7:0/3387675404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c17f0 0x7f4ba006d6a0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.133+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3387675404 >> 172.21.15.7:0/3387675404 conn(0x7f4ba00a9430 msgr2=0x7f4ba00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:59:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.133+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3387675404 shutdown_connections 2023-03-03T12:59:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.133+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3387675404 wait complete. 2023-03-03T12:59:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.134+0000 7f4ba85ed700 1 Processor -- start 2023-03-03T12:59:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.134+0000 7f4ba85ed700 1 -- start start 2023-03-03T12:59:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba85ed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c17f0 0x7f4ba0153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba85ed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ba00c5190 0x7f4ba00bc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba85ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c6010 0x7f4ba0071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba85ed700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba01ebc70 con 0x7f4ba00c6010 2023-03-03T12:59:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba85ed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba01529f0 con 0x7f4ba00c5190 2023-03-03T12:59:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba85ed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba0152430 con 0x7f4ba00c17f0 2023-03-03T12:59:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba6b8a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c6010 0x7f4ba0071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba6b8a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c6010 0x7f4ba0071ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42700/0 (socket says 172.21.15.7:42700) 2023-03-03T12:59:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba6b8a700 1 -- 172.21.15.7:0/3479779735 learned_addr learned my addr 172.21.15.7:0/3479779735 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:59:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba5b88700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ba00c5190 0x7f4ba00bc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba6389700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c17f0 0x7f4ba0153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba6b8a700 1 -- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c17f0 msgr2=0x7f4ba0153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba6b8a700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c17f0 0x7f4ba0153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba6b8a700 1 -- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ba00c5190 msgr2=0x7f4ba00bc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba6b8a700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ba00c5190 0x7f4ba00bc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba6b8a700 1 -- 172.21.15.7:0/3479779735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b94009550 con 0x7f4ba00c6010 2023-03-03T12:59:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba5b88700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ba00c5190 0x7f4ba00bc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:59:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.135+0000 7f4ba6389700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c17f0 0x7f4ba0153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:59:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.136+0000 7f4ba6b8a700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c6010 0x7f4ba0071ad0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800efc0 tx=0x7f4b9800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.136+0000 7f4b92ffd700 1 -- 172.21.15.7:0/3479779735 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9800cd40 con 0x7f4ba00c6010 2023-03-03T12:59:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.136+0000 7f4b92ffd700 1 -- 172.21.15.7:0/3479779735 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9800f040 con 0x7f4ba00c6010 2023-03-03T12:59:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.136+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3479779735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba00c2b00 con 0x7f4ba00c6010 2023-03-03T12:59:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.136+0000 7f4b92ffd700 1 -- 172.21.15.7:0/3479779735 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b980146c0 con 0x7f4ba00c6010 2023-03-03T12:59:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.136+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3479779735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba00aa960 con 0x7f4ba00c6010 2023-03-03T12:59:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.137+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3479779735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba00c3300 con 0x7f4ba00c6010 2023-03-03T12:59:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.137+0000 7f4b92ffd700 1 -- 172.21.15.7:0/3479779735 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4b98014820 con 0x7f4ba00c6010 2023-03-03T12:59:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.138+0000 7f4b92ffd700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b840682f0 0x7f4b8406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.138+0000 7f4b92ffd700 1 -- 172.21.15.7:0/3479779735 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4b98086bc0 con 0x7f4ba00c6010 2023-03-03T12:59:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.138+0000 7f4ba6389700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b840682f0 0x7f4b8406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.139+0000 7f4ba6389700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b840682f0 0x7f4b8406a7a0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c000f60 tx=0x7f4b9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.142+0000 7f4b92ffd700 1 -- 172.21.15.7:0/3479779735 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b98052f60 con 0x7f4ba00c6010 2023-03-03T12:59:39.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.380+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3479779735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ba00a6e50 con 0x7f4ba00c6010 2023-03-03T12:59:39.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.381+0000 7f4b92ffd700 1 -- 172.21.15.7:0/3479779735 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4b9800cea0 con 0x7f4ba00c6010 2023-03-03T12:59:39.383 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:59:39.383 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:59:39.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b840682f0 msgr2=0x7f4b8406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:39.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b840682f0 0x7f4b8406a7a0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c000f60 tx=0x7f4b9c009040 comp rx=0 tx=0).stop 2023-03-03T12:59:39.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3479779735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c6010 msgr2=0x7f4ba0071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:39.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c6010 0x7f4ba0071ad0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800efc0 tx=0x7f4b9800c3a0 comp rx=0 tx=0).stop 2023-03-03T12:59:39.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3479779735 shutdown_connections 2023-03-03T12:59:39.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b840682f0 0x7f4b8406a7a0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ba00c6010 0x7f4ba0071ad0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ba00c17f0 0x7f4ba0153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 --2- 172.21.15.7:0/3479779735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ba00c5190 0x7f4ba00bc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3479779735 >> 172.21.15.7:0/3479779735 conn(0x7f4ba00a9430 msgr2=0x7f4ba00a6740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:59:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3479779735 shutdown_connections 2023-03-03T12:59:39.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:39.383+0000 7f4ba85ed700 1 -- 172.21.15.7:0/3479779735 wait complete. 2023-03-03T12:59:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:59:39.396 INFO:tasks.mds_thrash.fs.[a]:waiting for 12 secs before thrashing 2023-03-03T12:59:47.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:59:47.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.404+0000 7fbf16386700 1 Processor -- start 2023-03-03T12:59:47.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.404+0000 7fbf16386700 1 -- start start 2023-03-03T12:59:47.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.404+0000 7fbf16386700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 0x7fbf101550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:47.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.404+0000 7fbf16386700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbf100c3e30 0x7fbf100c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:47.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.404+0000 7fbf16386700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf100c7a60 con 0x7fbf100c1da0 2023-03-03T12:59:47.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.404+0000 7fbf16386700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf100c2af0 con 0x7fbf100c3e30 2023-03-03T12:59:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf16386700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100c3620 con 0x7fbf1006d870 2023-03-03T12:59:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf14923700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbf100c3e30 0x7fbf100c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf0f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbf100c1da0 0x7fbf100c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36690/0 (socket says 172.21.15.7:36690) 2023-03-03T12:59:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf0f7fe700 1 -- 172.21.15.7:0/2365093708 learned_addr learned my addr 172.21.15.7:0/2365093708 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:59:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf14923700 1 -- 172.21.15.7:0/2365093708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 msgr2=0x7fbf101550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:59:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf0ffff700 1 --2- 172.21.15.7:0/2365093708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 0x7fbf101550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf14923700 1 --2- 172.21.15.7:0/2365093708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 0x7fbf101550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf14923700 1 -- 172.21.15.7:0/2365093708 >> v1:172.21.15.7:6789/0 conn(0x7fbf100c1da0 legacy=0x7fbf100c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf0effd700 1 -- 172.21.15.7:0/2365093708 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237820876 0 0) 0x7fbf100c7a60 con 0x7fbf100c1da0 2023-03-03T12:59:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf14923700 1 -- 172.21.15.7:0/2365093708 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf101545e0 con 0x7fbf100c3e30 2023-03-03T12:59:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf0ffff700 1 --2- 172.21.15.7:0/2365093708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 0x7fbf101550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T12:59:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf14923700 1 --2- 172.21.15.7:0/2365093708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 0x7fbf100c4290 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009d90 tx=0x7fbf0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9704bcff1451bb6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf0effd700 1 -- 172.21.15.7:0/2365093708 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0800e070 con 0x7fbf100c3e30 2023-03-03T12:59:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf0effd700 1 -- 172.21.15.7:0/2365093708 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0800bbd0 con 0x7fbf100c3e30 2023-03-03T12:59:47.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.405+0000 7fbf0effd700 1 -- 172.21.15.7:0/2365093708 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf08012430 con 0x7fbf100c3e30 2023-03-03T12:59:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.406+0000 7fbf16386700 1 -- 172.21.15.7:0/2365093708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 msgr2=0x7fbf100c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.406+0000 7fbf16386700 1 --2- 172.21.15.7:0/2365093708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 0x7fbf100c4290 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009d90 tx=0x7fbf0800b330 comp rx=0 tx=0).stop 2023-03-03T12:59:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.406+0000 7fbf16386700 1 -- 172.21.15.7:0/2365093708 shutdown_connections 2023-03-03T12:59:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.406+0000 7fbf16386700 1 --2- 172.21.15.7:0/2365093708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 0x7fbf100c4290 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.406+0000 7fbf16386700 1 --2- 172.21.15.7:0/2365093708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 0x7fbf101550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.406+0000 7fbf16386700 1 -- 172.21.15.7:0/2365093708 >> 172.21.15.7:0/2365093708 conn(0x7fbf1006c110 msgr2=0x7fbf1006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:59:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.406+0000 7fbf16386700 1 -- 172.21.15.7:0/2365093708 shutdown_connections 2023-03-03T12:59:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.406+0000 7fbf16386700 1 -- 172.21.15.7:0/2365093708 wait complete. 2023-03-03T12:59:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.408+0000 7fbf16386700 1 Processor -- start 2023-03-03T12:59:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.408+0000 7fbf16386700 1 -- start start 2023-03-03T12:59:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf16386700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 0x7fbf100c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf16386700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf100c1da0 0x7fbf10150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf16386700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 0x7fbf101513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf16386700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10157770 con 0x7fbf100c3e30 2023-03-03T12:59:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf16386700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10155ad0 con 0x7fbf100c1da0 2023-03-03T12:59:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf16386700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100acc90 con 0x7fbf1006d870 2023-03-03T12:59:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf14923700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 0x7fbf101513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf14923700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 0x7fbf101513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42718/0 (socket says 172.21.15.7:42718) 2023-03-03T12:59:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf14923700 1 -- 172.21.15.7:0/4124684744 learned_addr learned my addr 172.21.15.7:0/4124684744 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:59:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf0ffff700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 0x7fbf100c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf0f7fe700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf100c1da0 0x7fbf10150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf14923700 1 -- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 msgr2=0x7fbf100c0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf14923700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 0x7fbf100c0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf14923700 1 -- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf100c1da0 msgr2=0x7fbf10150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf14923700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf100c1da0 0x7fbf10150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf14923700 1 -- 172.21.15.7:0/4124684744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf08009a70 con 0x7fbf100c3e30 2023-03-03T12:59:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf0ffff700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 0x7fbf100c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:59:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf0f7fe700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf100c1da0 0x7fbf10150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:59:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.409+0000 7fbf14923700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 0x7fbf101513b0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800b730 tx=0x7fbf08004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:47.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.410+0000 7fbf0cff9700 1 -- 172.21.15.7:0/4124684744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf08003ea0 con 0x7fbf100c3e30 2023-03-03T12:59:47.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.410+0000 7fbf0cff9700 1 -- 172.21.15.7:0/4124684744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf08020070 con 0x7fbf100c3e30 2023-03-03T12:59:47.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.410+0000 7fbf16386700 1 -- 172.21.15.7:0/4124684744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf10154ba0 con 0x7fbf100c3e30 2023-03-03T12:59:47.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.410+0000 7fbf0cff9700 1 -- 172.21.15.7:0/4124684744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf08004280 con 0x7fbf100c3e30 2023-03-03T12:59:47.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.410+0000 7fbf16386700 1 -- 172.21.15.7:0/4124684744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf101548c0 con 0x7fbf100c3e30 2023-03-03T12:59:47.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.411+0000 7fbf16386700 1 -- 172.21.15.7:0/4124684744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1006dc40 con 0x7fbf100c3e30 2023-03-03T12:59:47.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.411+0000 7fbf0cff9700 1 -- 172.21.15.7:0/4124684744 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbf080043e0 con 0x7fbf100c3e30 2023-03-03T12:59:47.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.411+0000 7fbf0cff9700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbefc068320 0x7fbefc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:47.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.412+0000 7fbf0cff9700 1 -- 172.21.15.7:0/4124684744 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbf08022070 con 0x7fbf100c3e30 2023-03-03T12:59:47.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.412+0000 7fbf0ffff700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbefc068320 0x7fbefc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:47.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.412+0000 7fbf0ffff700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbefc068320 0x7fbefc06a7d0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fbf00006fd0 tx=0x7fbf00008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:47.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.416+0000 7fbf0cff9700 1 -- 172.21.15.7:0/4124684744 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbf08054a80 con 0x7fbf100c3e30 2023-03-03T12:59:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.657+0000 7fbf16386700 1 -- 172.21.15.7:0/4124684744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf10155c40 con 0x7fbf100c3e30 2023-03-03T12:59:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.658+0000 7fbf0cff9700 1 -- 172.21.15.7:0/4124684744 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbf10155c40 con 0x7fbf100c3e30 2023-03-03T12:59:47.659 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:59:47.660 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:59:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 -- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbefc068320 msgr2=0x7fbefc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbefc068320 0x7fbefc06a7d0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fbf00006fd0 tx=0x7fbf00008040 comp rx=0 tx=0).stop 2023-03-03T12:59:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 -- 172.21.15.7:0/4124684744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 msgr2=0x7fbf101513b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 0x7fbf101513b0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800b730 tx=0x7fbf08004ad0 comp rx=0 tx=0).stop 2023-03-03T12:59:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 -- 172.21.15.7:0/4124684744 shutdown_connections 2023-03-03T12:59:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbefc068320 0x7fbefc06a7d0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf100c3e30 0x7fbf101513b0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf1006d870 0x7fbf100c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 --2- 172.21.15.7:0/4124684744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf100c1da0 0x7fbf10150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 -- 172.21.15.7:0/4124684744 >> 172.21.15.7:0/4124684744 conn(0x7fbf1006c110 msgr2=0x7fbf1006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:59:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 -- 172.21.15.7:0/4124684744 shutdown_connections 2023-03-03T12:59:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:47.660+0000 7fbf16386700 1 -- 172.21.15.7:0/4124684744 wait complete. 2023-03-03T12:59:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:59:47.673 INFO:tasks.mds_thrash.fs.[b]:waiting for 35 secs before thrashing 2023-03-03T12:59:51.398 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T12:59:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.535+0000 7fa550dd4700 1 Processor -- start 2023-03-03T12:59:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.535+0000 7fa550dd4700 1 -- start start 2023-03-03T12:59:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa550dd4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa54c14fda0 0x7fa54c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa550dd4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c0c2f30 con 0x7fa54c14f9d0 2023-03-03T12:59:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa550dd4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c0c31d0 con 0x7fa54c0abe20 2023-03-03T12:59:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa550dd4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54c1524a0 con 0x7fa54c14fda0 2023-03-03T12:59:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa549d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa54c14fda0 0x7fa54c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa54a59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa54c14f9d0 0x7fa54c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36704/0 (socket says 172.21.15.7:36704) 2023-03-03T12:59:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa54a59c700 1 -- 172.21.15.7:0/941409779 learned_addr learned my addr 172.21.15.7:0/941409779 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T12:59:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa549d9b700 1 -- 172.21.15.7:0/941409779 >> v1:172.21.15.143:6789/0 conn(0x7fa54c0abe20 legacy=0x7fa54c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T12:59:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa549d9b700 1 -- 172.21.15.7:0/941409779 >> v1:172.21.15.7:6789/0 conn(0x7fa54c14f9d0 legacy=0x7fa54c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa549d9b700 1 -- 172.21.15.7:0/941409779 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c0c34b0 con 0x7fa54c14fda0 2023-03-03T12:59:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.536+0000 7fa549d9b700 1 --2- 172.21.15.7:0/941409779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14fda0 0x7fa54c156430 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fa540009840 tx=0x7fa540009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c7c878ff5683ed server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.537+0000 7fa54959a700 1 -- 172.21.15.7:0/941409779 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540004230 con 0x7fa54c14fda0 2023-03-03T12:59:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.537+0000 7fa54959a700 1 -- 172.21.15.7:0/941409779 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa540004390 con 0x7fa54c14fda0 2023-03-03T12:59:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.537+0000 7fa54959a700 1 -- 172.21.15.7:0/941409779 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5400036a0 con 0x7fa54c14fda0 2023-03-03T12:59:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.537+0000 7fa550dd4700 1 -- 172.21.15.7:0/941409779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14fda0 msgr2=0x7fa54c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.537+0000 7fa550dd4700 1 --2- 172.21.15.7:0/941409779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14fda0 0x7fa54c156430 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fa540009840 tx=0x7fa540009b50 comp rx=0 tx=0).stop 2023-03-03T12:59:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.537+0000 7fa550dd4700 1 -- 172.21.15.7:0/941409779 shutdown_connections 2023-03-03T12:59:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.537+0000 7fa550dd4700 1 --2- 172.21.15.7:0/941409779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14fda0 0x7fa54c156430 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.537+0000 7fa550dd4700 1 -- 172.21.15.7:0/941409779 >> 172.21.15.7:0/941409779 conn(0x7fa54c0c8900 msgr2=0x7fa54c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:59:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.537+0000 7fa550dd4700 1 -- 172.21.15.7:0/941409779 shutdown_connections 2023-03-03T12:59:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.537+0000 7fa550dd4700 1 -- 172.21.15.7:0/941409779 wait complete. 2023-03-03T12:59:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.539+0000 7fa550dd4700 1 Processor -- start 2023-03-03T12:59:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.539+0000 7fa550dd4700 1 -- start start 2023-03-03T12:59:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa550dd4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa54c0abe20 0x7fa54c155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa550dd4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14f9d0 0x7fa54c149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa550dd4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa54c14fda0 0x7fa54c14c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa550dd4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c14bdc0 con 0x7fa54c14f9d0 2023-03-03T12:59:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa550dd4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c1e80d0 con 0x7fa54c14fda0 2023-03-03T12:59:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa550dd4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c14c060 con 0x7fa54c0abe20 2023-03-03T12:59:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa549d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14f9d0 0x7fa54c149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa549d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14f9d0 0x7fa54c149e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42732/0 (socket says 172.21.15.7:42732) 2023-03-03T12:59:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa549d9b700 1 -- 172.21.15.7:0/2220556804 learned_addr learned my addr 172.21.15.7:0/2220556804 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T12:59:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa54ad9d700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa54c14fda0 0x7fa54c14c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa54a59c700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa54c0abe20 0x7fa54c155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa549d9b700 1 -- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa54c0abe20 msgr2=0x7fa54c155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa549d9b700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa54c0abe20 0x7fa54c155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa549d9b700 1 -- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa54c14fda0 msgr2=0x7fa54c14c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa549d9b700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa54c14fda0 0x7fa54c14c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa549d9b700 1 -- 172.21.15.7:0/2220556804 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400094a0 con 0x7fa54c14f9d0 2023-03-03T12:59:51.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.540+0000 7fa54ad9d700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa54c14fda0 0x7fa54c14c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T12:59:51.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.541+0000 7fa549d9b700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14f9d0 0x7fa54c149e70 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fa5400090b0 tx=0x7fa540004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:51.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.541+0000 7fa53affd700 1 -- 172.21.15.7:0/2220556804 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540011830 con 0x7fa54c14f9d0 2023-03-03T12:59:51.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.541+0000 7fa53affd700 1 -- 172.21.15.7:0/2220556804 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54001e070 con 0x7fa54c14f9d0 2023-03-03T12:59:51.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.541+0000 7fa550dd4700 1 -- 172.21.15.7:0/2220556804 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c0ac350 con 0x7fa54c14f9d0 2023-03-03T12:59:51.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.541+0000 7fa53affd700 1 -- 172.21.15.7:0/2220556804 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540011bf0 con 0x7fa54c14f9d0 2023-03-03T12:59:51.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.541+0000 7fa550dd4700 1 -- 172.21.15.7:0/2220556804 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c0ab630 con 0x7fa54c14f9d0 2023-03-03T12:59:51.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.542+0000 7fa550dd4700 1 -- 172.21.15.7:0/2220556804 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54c071fa0 con 0x7fa54c14f9d0 2023-03-03T12:59:51.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.542+0000 7fa53affd700 1 -- 172.21.15.7:0/2220556804 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa540017020 con 0x7fa54c14f9d0 2023-03-03T12:59:51.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.543+0000 7fa53affd700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa52c0682f0 0x7fa52c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T12:59:51.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.543+0000 7fa53affd700 1 -- 172.21.15.7:0/2220556804 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa540087440 con 0x7fa54c14f9d0 2023-03-03T12:59:51.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.543+0000 7fa54a59c700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa52c0682f0 0x7fa52c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T12:59:51.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.543+0000 7fa54a59c700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa52c0682f0 0x7fa52c06a7a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fa54c06f710 tx=0x7fa53c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T12:59:51.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.547+0000 7fa53affd700 1 -- 172.21.15.7:0/2220556804 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa5400537e0 con 0x7fa54c14f9d0 2023-03-03T12:59:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.787+0000 7fa550dd4700 1 -- 172.21.15.7:0/2220556804 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa54c074bb0 con 0x7fa54c14f9d0 2023-03-03T12:59:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.788+0000 7fa53affd700 1 -- 172.21.15.7:0/2220556804 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa540056e00 con 0x7fa54c14f9d0 2023-03-03T12:59:51.789 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T12:59:51.790 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T12:59:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 -- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa52c0682f0 msgr2=0x7fa52c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa52c0682f0 0x7fa52c06a7a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fa54c06f710 tx=0x7fa53c009040 comp rx=0 tx=0).stop 2023-03-03T12:59:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 -- 172.21.15.7:0/2220556804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14f9d0 msgr2=0x7fa54c149e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T12:59:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14f9d0 0x7fa54c149e70 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fa5400090b0 tx=0x7fa540004750 comp rx=0 tx=0).stop 2023-03-03T12:59:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 -- 172.21.15.7:0/2220556804 shutdown_connections 2023-03-03T12:59:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa54c14fda0 0x7fa54c14c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa54c0abe20 0x7fa54c155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa52c0682f0 0x7fa52c06a7a0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 --2- 172.21.15.7:0/2220556804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa54c14f9d0 0x7fa54c149e70 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T12:59:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 -- 172.21.15.7:0/2220556804 >> 172.21.15.7:0/2220556804 conn(0x7fa54c0c8900 msgr2=0x7fa54c1518c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T12:59:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 -- 172.21.15.7:0/2220556804 shutdown_connections 2023-03-03T12:59:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T12:59:51.790+0000 7fa550dd4700 1 -- 172.21.15.7:0/2220556804 wait complete. 2023-03-03T12:59:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T12:59:51.803 INFO:tasks.mds_thrash.fs.[a]:waiting for 75 secs before thrashing 2023-03-03T13:00:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T12:59:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T13:00:06.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:06.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:22.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:00:22.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.813+0000 7fbda49db700 1 Processor -- start 2023-03-03T13:00:22.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.813+0000 7fbda49db700 1 -- start start 2023-03-03T13:00:22.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbda00ac220 0x7fbda00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:22.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda0073c30 0x7fbda0152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:22.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbda00ace30 0x7fbda0152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:22.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda0071e30 con 0x7fbda00ace30 2023-03-03T13:00:22.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda0152ed0 con 0x7fbda00ac220 2023-03-03T13:00:22.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00ac900 con 0x7fbda0073c30 2023-03-03T13:00:22.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9ed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbda00ace30 0x7fbda0152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:22.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9ed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbda00ace30 0x7fbda0152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42760/0 (socket says 172.21.15.7:42760) 2023-03-03T13:00:22.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9ed9d700 1 -- 172.21.15.7:0/4029014921 learned_addr learned my addr 172.21.15.7:0/4029014921 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:00:22.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9e59c700 1 --2- 172.21.15.7:0/4029014921 >> v2:172.21.15.143:3300/0 conn(0x7fbda00ac220 0x7fbda00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:22.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9dd9b700 1 --2- 172.21.15.7:0/4029014921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda0073c30 0x7fbda0152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:22.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9ed9d700 1 -- 172.21.15.7:0/4029014921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda0073c30 msgr2=0x7fbda0152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:00:22.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9ed9d700 1 --2- 172.21.15.7:0/4029014921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda0073c30 0x7fbda0152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:22.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9ed9d700 1 -- 172.21.15.7:0/4029014921 >> v2:172.21.15.143:3300/0 conn(0x7fbda00ac220 msgr2=0x7fbda00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:00:22.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9ed9d700 1 --2- 172.21.15.7:0/4029014921 >> v2:172.21.15.143:3300/0 conn(0x7fbda00ac220 0x7fbda00736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:22.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9ed9d700 1 -- 172.21.15.7:0/4029014921 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda00acb50 con 0x7fbda00ace30 2023-03-03T13:00:22.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9ed9d700 1 --2- 172.21.15.7:0/4029014921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda00ace30 0x7fbda0152990 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fbda006fbf0 tx=0x7fbd9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a61439e05b91b09 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:00:22.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9d59a700 1 -- 172.21.15.7:0/4029014921 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9800e070 con 0x7fbda00ace30 2023-03-03T13:00:22.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9d59a700 1 -- 172.21.15.7:0/4029014921 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9800b9c0 con 0x7fbda00ace30 2023-03-03T13:00:22.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbd9d59a700 1 -- 172.21.15.7:0/4029014921 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9800bd70 con 0x7fbda00ace30 2023-03-03T13:00:22.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 -- 172.21.15.7:0/4029014921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda00ace30 msgr2=0x7fbda0152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:22.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 --2- 172.21.15.7:0/4029014921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda00ace30 0x7fbda0152990 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7fbda006fbf0 tx=0x7fbd9800b330 comp rx=0 tx=0).stop 2023-03-03T13:00:22.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 -- 172.21.15.7:0/4029014921 shutdown_connections 2023-03-03T13:00:22.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 --2- 172.21.15.7:0/4029014921 >> v2:172.21.15.143:3300/0 conn(0x7fbda00ac220 0x7fbda00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:22.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 --2- 172.21.15.7:0/4029014921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda00ace30 0x7fbda0152990 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:22.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 --2- 172.21.15.7:0/4029014921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda0073c30 0x7fbda0152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:22.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 -- 172.21.15.7:0/4029014921 >> 172.21.15.7:0/4029014921 conn(0x7fbda00a94d0 msgr2=0x7fbda00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:00:22.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 -- 172.21.15.7:0/4029014921 shutdown_connections 2023-03-03T13:00:22.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.814+0000 7fbda49db700 1 -- 172.21.15.7:0/4029014921 wait complete. 2023-03-03T13:00:22.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.815+0000 7fbda49db700 1 Processor -- start 2023-03-03T13:00:22.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.815+0000 7fbda49db700 1 -- start start 2023-03-03T13:00:22.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbda49db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda0073c30 0x7fbda00c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:22.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbda49db700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbda00ac220 0x7fbda0150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:22.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbda49db700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda00ace30 0x7fbda014b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:22.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbda49db700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00c5f70 con 0x7fbda0073c30 2023-03-03T13:00:22.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbda49db700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00c1450 con 0x7fbda00ac220 2023-03-03T13:00:22.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbda49db700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00c0e90 con 0x7fbda00ace30 2023-03-03T13:00:22.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbd9e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda0073c30 0x7fbda00c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:22.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbd9e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda0073c30 0x7fbda00c5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42764/0 (socket says 172.21.15.7:42764) 2023-03-03T13:00:22.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbd9e59c700 1 -- 172.21.15.7:0/1000638846 learned_addr learned my addr 172.21.15.7:0/1000638846 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:00:22.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbd9dd9b700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbda00ac220 0x7fbda0150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.816+0000 7fbd9ed9d700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda00ace30 0x7fbda014b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbd9e59c700 1 -- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda00ace30 msgr2=0x7fbda014b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbd9e59c700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda00ace30 0x7fbda014b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbd9e59c700 1 -- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbda00ac220 msgr2=0x7fbda0150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbd9e59c700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbda00ac220 0x7fbda0150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbd9e59c700 1 -- 172.21.15.7:0/1000638846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd98009b30 con 0x7fbda0073c30 2023-03-03T13:00:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbd9ed9d700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda00ace30 0x7fbda014b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:00:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbd9e59c700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda0073c30 0x7fbda00c5a30 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000d800 tx=0x7fbd9000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:00:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbd8effd700 1 -- 172.21.15.7:0/1000638846 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd900125f0 con 0x7fbda0073c30 2023-03-03T13:00:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbd8effd700 1 -- 172.21.15.7:0/1000638846 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd90012750 con 0x7fbda0073c30 2023-03-03T13:00:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbda49db700 1 -- 172.21.15.7:0/1000638846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda0155ca0 con 0x7fbda0073c30 2023-03-03T13:00:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbd8effd700 1 -- 172.21.15.7:0/1000638846 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90012b00 con 0x7fbda0073c30 2023-03-03T13:00:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.817+0000 7fbda49db700 1 -- 172.21.15.7:0/1000638846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda01559c0 con 0x7fbda0073c30 2023-03-03T13:00:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.818+0000 7fbda49db700 1 -- 172.21.15.7:0/1000638846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda0155f80 con 0x7fbda0073c30 2023-03-03T13:00:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.819+0000 7fbd8effd700 1 -- 172.21.15.7:0/1000638846 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd90012c60 con 0x7fbda0073c30 2023-03-03T13:00:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.819+0000 7fbd8effd700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd800682f0 0x7fbd8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:00:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.819+0000 7fbd8effd700 1 -- 172.21.15.7:0/1000638846 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbd900865b0 con 0x7fbda0073c30 2023-03-03T13:00:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.819+0000 7fbd9dd9b700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd800682f0 0x7fbd8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:00:22.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.820+0000 7fbd9dd9b700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd800682f0 0x7fbd8006a7a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fbda00ab540 tx=0x7fbd94008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:00:22.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:22.824+0000 7fbd8effd700 1 -- 172.21.15.7:0/1000638846 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbd90052950 con 0x7fbda0073c30 2023-03-03T13:00:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.060+0000 7fbda49db700 1 -- 172.21.15.7:0/1000638846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbda0074bb0 con 0x7fbda0073c30 2023-03-03T13:00:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.060+0000 7fbd8effd700 1 -- 172.21.15.7:0/1000638846 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbd90055f70 con 0x7fbda0073c30 2023-03-03T13:00:23.063 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:00:23.063 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:00:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 -- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd800682f0 msgr2=0x7fbd8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd800682f0 0x7fbd8006a7a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fbda00ab540 tx=0x7fbd94008040 comp rx=0 tx=0).stop 2023-03-03T13:00:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 -- 172.21.15.7:0/1000638846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda0073c30 msgr2=0x7fbda00c5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:00:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda0073c30 0x7fbda00c5a30 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000d800 tx=0x7fbd9000dbc0 comp rx=0 tx=0).stop 2023-03-03T13:00:23.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 -- 172.21.15.7:0/1000638846 shutdown_connections 2023-03-03T13:00:23.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbda00ace30 0x7fbda014b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:23.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd800682f0 0x7fbd8006a7a0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:23.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbda0073c30 0x7fbda00c5a30 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:23.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 --2- 172.21.15.7:0/1000638846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbda00ac220 0x7fbda0150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:00:23.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 -- 172.21.15.7:0/1000638846 >> 172.21.15.7:0/1000638846 conn(0x7fbda00a94d0 msgr2=0x7fbda00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:00:23.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 -- 172.21.15.7:0/1000638846 shutdown_connections 2023-03-03T13:00:23.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:00:23.062+0000 7fbda49db700 1 -- 172.21.15.7:0/1000638846 wait complete. 2023-03-03T13:00:23.067 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:00:23.077 INFO:tasks.mds_thrash.fs.[b]:waiting for 59 secs before thrashing 2023-03-03T13:00:36.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:00:36.269 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:06.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:06.417 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:06.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:01:06.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.942+0000 7fcf8e5f8700 1 Processor -- start 2023-03-03T13:01:06.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.942+0000 7fcf8e5f8700 1 -- start start 2023-03-03T13:01:06.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf8e5f8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcf88155390 0x7fcf880c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:06.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf8e5f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf8806e020 0x7fcf88071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:06.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf8e5f8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf88154910 con 0x7fcf88072520 2023-03-03T13:01:06.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf8e5f8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf88070b90 con 0x7fcf88155390 2023-03-03T13:01:06.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf8e5f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88151320 con 0x7fcf8806e020 2023-03-03T13:01:06.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf8cb95700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcf88072520 0x7fcf88151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36776/0 (socket says 172.21.15.7:36776) 2023-03-03T13:01:06.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf8cb95700 1 -- 172.21.15.7:0/2950011511 learned_addr learned my addr 172.21.15.7:0/2950011511 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:01:06.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf877fe700 1 --2- 172.21.15.7:0/2950011511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf8806e020 0x7fcf88071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:06.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf87fff700 1 --2- 172.21.15.7:0/2950011511 >> v2:172.21.15.143:3300/0 conn(0x7fcf88155390 0x7fcf880c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:06.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf86ffd700 1 -- 172.21.15.7:0/2950011511 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991083356 0 0) 0x7fcf88154910 con 0x7fcf88072520 2023-03-03T13:01:06.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.943+0000 7fcf86ffd700 1 -- 172.21.15.7:0/2950011511 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf740035b0 con 0x7fcf88072520 2023-03-03T13:01:06.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf87fff700 1 -- 172.21.15.7:0/2950011511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf8806e020 msgr2=0x7fcf88071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:01:06.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf87fff700 1 --2- 172.21.15.7:0/2950011511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf8806e020 0x7fcf88071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:06.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf87fff700 1 -- 172.21.15.7:0/2950011511 >> v1:172.21.15.7:6789/0 conn(0x7fcf88072520 legacy=0x7fcf88151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:06.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf87fff700 1 -- 172.21.15.7:0/2950011511 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf880702a0 con 0x7fcf88155390 2023-03-03T13:01:06.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf877fe700 1 --2- 172.21.15.7:0/2950011511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf8806e020 0x7fcf88071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:01:06.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf87fff700 1 --2- 172.21.15.7:0/2950011511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf88155390 0x7fcf880c3e30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009840 tx=0x7fcf78009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3d51d95e9c0c2f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:01:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf86ffd700 1 -- 172.21.15.7:0/2950011511 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78011830 con 0x7fcf88155390 2023-03-03T13:01:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf86ffd700 1 -- 172.21.15.7:0/2950011511 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf78011990 con 0x7fcf88155390 2023-03-03T13:01:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf86ffd700 1 -- 172.21.15.7:0/2950011511 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78011d20 con 0x7fcf88155390 2023-03-03T13:01:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2950011511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf88155390 msgr2=0x7fcf880c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.944+0000 7fcf8e5f8700 1 --2- 172.21.15.7:0/2950011511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf88155390 0x7fcf880c3e30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009840 tx=0x7fcf78009b50 comp rx=0 tx=0).stop 2023-03-03T13:01:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.945+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2950011511 shutdown_connections 2023-03-03T13:01:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.945+0000 7fcf8e5f8700 1 --2- 172.21.15.7:0/2950011511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf88155390 0x7fcf880c3e30 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.945+0000 7fcf8e5f8700 1 --2- 172.21.15.7:0/2950011511 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf8806e020 0x7fcf88071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.945+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2950011511 >> 172.21.15.7:0/2950011511 conn(0x7fcf8806c110 msgr2=0x7fcf8806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:01:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.945+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2950011511 shutdown_connections 2023-03-03T13:01:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.945+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2950011511 wait complete. 2023-03-03T13:01:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.946+0000 7fcf8e5f8700 1 Processor -- start 2023-03-03T13:01:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.946+0000 7fcf8e5f8700 1 -- start start 2023-03-03T13:01:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf8e5f8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf8806e020 0x7fcf8814a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf8e5f8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88072520 0x7fcf880c0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf8e5f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf88155390 0x7fcf88073960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf8e5f8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880c07d0 con 0x7fcf88072520 2023-03-03T13:01:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf8e5f8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881eabd0 con 0x7fcf8806e020 2023-03-03T13:01:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf8e5f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881eb100 con 0x7fcf88155390 2023-03-03T13:01:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf877fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88072520 0x7fcf880c0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:06.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf877fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88072520 0x7fcf880c0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42800/0 (socket says 172.21.15.7:42800) 2023-03-03T13:01:06.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf877fe700 1 -- 172.21.15.7:0/2553636031 learned_addr learned my addr 172.21.15.7:0/2553636031 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:01:06.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf87fff700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf8806e020 0x7fcf8814a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:06.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf8cb95700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf88155390 0x7fcf88073960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:06.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf877fe700 1 -- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf88155390 msgr2=0x7fcf88073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:06.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf877fe700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf88155390 0x7fcf88073960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:06.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf877fe700 1 -- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf8806e020 msgr2=0x7fcf8814a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:06.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf877fe700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf8806e020 0x7fcf8814a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:06.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf877fe700 1 -- 172.21.15.7:0/2553636031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf700094a0 con 0x7fcf88072520 2023-03-03T13:01:06.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf87fff700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf8806e020 0x7fcf8814a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:01:06.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.947+0000 7fcf877fe700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88072520 0x7fcf880c0e10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fcf7000ec10 tx=0x7fcf70009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:01:06.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.948+0000 7fcf84ff9700 1 -- 172.21.15.7:0/2553636031 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7000c8e0 con 0x7fcf88072520 2023-03-03T13:01:06.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.948+0000 7fcf84ff9700 1 -- 172.21.15.7:0/2553636031 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7000ca40 con 0x7fcf88072520 2023-03-03T13:01:06.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.948+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2553636031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf780094a0 con 0x7fcf88072520 2023-03-03T13:01:06.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.948+0000 7fcf84ff9700 1 -- 172.21.15.7:0/2553636031 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7000cdd0 con 0x7fcf88072520 2023-03-03T13:01:06.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.948+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2553636031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf88070820 con 0x7fcf88072520 2023-03-03T13:01:06.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.949+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2553636031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf88070540 con 0x7fcf88072520 2023-03-03T13:01:06.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.949+0000 7fcf84ff9700 1 -- 172.21.15.7:0/2553636031 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcf70019030 con 0x7fcf88072520 2023-03-03T13:01:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.949+0000 7fcf84ff9700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf680682f0 0x7fcf6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.950+0000 7fcf84ff9700 1 -- 172.21.15.7:0/2553636031 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcf70086130 con 0x7fcf88072520 2023-03-03T13:01:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.950+0000 7fcf87fff700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf680682f0 0x7fcf6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.950+0000 7fcf87fff700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf680682f0 0x7fcf6806a7a0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fcf78000c00 tx=0x7fcf78008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:01:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:06.954+0000 7fcf84ff9700 1 -- 172.21.15.7:0/2553636031 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcf700524d0 con 0x7fcf88072520 2023-03-03T13:01:07.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.192+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2553636031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf88155760 con 0x7fcf88072520 2023-03-03T13:01:07.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.192+0000 7fcf84ff9700 1 -- 172.21.15.7:0/2553636031 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcf70055af0 con 0x7fcf88072520 2023-03-03T13:01:07.194 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:01:07.194 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:01:07.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf680682f0 msgr2=0x7fcf6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:07.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf680682f0 0x7fcf6806a7a0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fcf78000c00 tx=0x7fcf78008030 comp rx=0 tx=0).stop 2023-03-03T13:01:07.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2553636031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88072520 msgr2=0x7fcf880c0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:07.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88072520 0x7fcf880c0e10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fcf7000ec10 tx=0x7fcf70009e20 comp rx=0 tx=0).stop 2023-03-03T13:01:07.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2553636031 shutdown_connections 2023-03-03T13:01:07.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcf88155390 0x7fcf88073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:07.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf8806e020 0x7fcf8814a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:07.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcf680682f0 0x7fcf6806a7a0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:07.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 --2- 172.21.15.7:0/2553636031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88072520 0x7fcf880c0e10 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:07.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2553636031 >> 172.21.15.7:0/2553636031 conn(0x7fcf8806c110 msgr2=0x7fcf8814ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:01:07.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2553636031 shutdown_connections 2023-03-03T13:01:07.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:07.194+0000 7fcf8e5f8700 1 -- 172.21.15.7:0/2553636031 wait complete. 2023-03-03T13:01:07.198 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:01:07.208 INFO:tasks.mds_thrash.fs.[a]:waiting for 112 secs before thrashing 2023-03-03T13:01:22.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:01:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.216+0000 7fc5370a2700 1 Processor -- start 2023-03-03T13:01:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.216+0000 7fc5370a2700 1 -- start start 2023-03-03T13:01:22.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.217+0000 7fc5370a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5300c5b00 0x7fc5300bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:22.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.217+0000 7fc5370a2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5300c08d0 con 0x7fc530155410 2023-03-03T13:01:22.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.217+0000 7fc5370a2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc530154930 con 0x7fc530151e80 2023-03-03T13:01:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.217+0000 7fc5370a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc530152250 con 0x7fc5300c5b00 2023-03-03T13:01:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.217+0000 7fc52ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc530155410 0x7fc5300bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36798/0 (socket says 172.21.15.7:36798) 2023-03-03T13:01:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.217+0000 7fc52ffff700 1 -- 172.21.15.7:0/2459022979 learned_addr learned my addr 172.21.15.7:0/2459022979 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:01:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.217+0000 7fc53563f700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5300c5b00 0x7fc5300bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688844291 0 0) 0x7fc5300c08d0 con 0x7fc530155410 2023-03-03T13:01:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5200035b0 con 0x7fc530155410 2023-03-03T13:01:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288101178 0 0) 0x7fc530154930 con 0x7fc530151e80 2023-03-03T13:01:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5300c08d0 con 0x7fc530151e80 2023-03-03T13:01:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1498125454 0 0) 0x7fc5200035b0 con 0x7fc530155410 2023-03-03T13:01:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5300c5b00 msgr2=0x7fc5300bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:01:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5300c5b00 0x7fc5300bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 >> v1:172.21.15.143:6789/0 conn(0x7fc530151e80 legacy=0x7fc5300c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc530154c10 con 0x7fc530155410 2023-03-03T13:01:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc51c001670 con 0x7fc530155410 2023-03-03T13:01:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc53563f700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5300c5b00 0x7fc5300bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:01:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 >> v1:172.21.15.7:6789/0 conn(0x7fc530155410 legacy=0x7fc5300bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5300c5b00 0x7fc520003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5200041c0 0x7fc520004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc520004d90 0x7fc520005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52000bb20 con 0x7fc520004d90 2023-03-03T13:01:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52000bc90 con 0x7fc5200041c0 2023-03-03T13:01:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52000be50 con 0x7fc5300c5b00 2023-03-03T13:01:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc53563f700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc520004d90 0x7fc520005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc534e3e700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5300c5b00 0x7fc520003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc52ffff700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5200041c0 0x7fc520004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc53563f700 1 -- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5300c5b00 msgr2=0x7fc520003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:01:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc53563f700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5300c5b00 0x7fc520003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc53563f700 1 -- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5200041c0 msgr2=0x7fc520004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:01:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc53563f700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5200041c0 0x7fc520004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.218+0000 7fc53563f700 1 -- 172.21.15.7:0/2459022979 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc528009a70 con 0x7fc520004d90 2023-03-03T13:01:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc53563f700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc520004d90 0x7fc520005200 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fc52800e350 tx=0x7fc520005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254d9821fd480964 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:01:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52800ee10 con 0x7fc520004d90 2023-03-03T13:01:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52800f040 con 0x7fc520004d90 2023-03-03T13:01:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc52f7fe700 1 -- 172.21.15.7:0/2459022979 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc528003c60 con 0x7fc520004d90 2023-03-03T13:01:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc5370a2700 1 -- 172.21.15.7:0/2459022979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc520004d90 msgr2=0x7fc520005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc5370a2700 1 --2- 172.21.15.7:0/2459022979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc520004d90 0x7fc520005200 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fc52800e350 tx=0x7fc520005880 comp rx=0 tx=0).stop 2023-03-03T13:01:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc53563f700 1 -- 172.21.15.7:0/2459022979 reap_dead start 2023-03-03T13:01:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc5370a2700 1 -- 172.21.15.7:0/2459022979 shutdown_connections 2023-03-03T13:01:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc5370a2700 1 -- 172.21.15.7:0/2459022979 >> 172.21.15.7:0/2459022979 conn(0x7fc530071630 msgr2=0x7fc53006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:01:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc5370a2700 1 -- 172.21.15.7:0/2459022979 shutdown_connections 2023-03-03T13:01:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.219+0000 7fc5370a2700 1 -- 172.21.15.7:0/2459022979 wait complete. 2023-03-03T13:01:22.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.221+0000 7fc5370a2700 1 Processor -- start 2023-03-03T13:01:22.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.221+0000 7fc5370a2700 1 -- start start 2023-03-03T13:01:22.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc5370a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc520004d90 0x7fc5301ea980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:22.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc5370a2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc53014ee70 0x7fc530156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:22.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc5370a2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5301ddf90 0x7fc53014e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:22.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc5370a2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53014da90 con 0x7fc5301ddf90 2023-03-03T13:01:22.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc5370a2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53014e010 con 0x7fc53014ee70 2023-03-03T13:01:22.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc5370a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301eaec0 con 0x7fc520004d90 2023-03-03T13:01:22.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc53563f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5301ddf90 0x7fc53014e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:22.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc53563f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5301ddf90 0x7fc53014e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42832/0 (socket says 172.21.15.7:42832) 2023-03-03T13:01:22.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc53563f700 1 -- 172.21.15.7:0/125132119 learned_addr learned my addr 172.21.15.7:0/125132119 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:01:22.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc534e3e700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc520004d90 0x7fc5301ea980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:22.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc52ffff700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc53014ee70 0x7fc530156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:22.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc53563f700 1 -- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc520004d90 msgr2=0x7fc5301ea980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:22.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc53563f700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc520004d90 0x7fc5301ea980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:22.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc53563f700 1 -- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc53014ee70 msgr2=0x7fc530156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:22.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc53563f700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc53014ee70 0x7fc530156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:22.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc53563f700 1 -- 172.21.15.7:0/125132119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5300c5220 con 0x7fc5301ddf90 2023-03-03T13:01:22.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc52ffff700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc53014ee70 0x7fc530156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:01:22.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc53563f700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5301ddf90 0x7fc53014e4a0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc52800e940 tx=0x7fc528009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:01:22.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc52d7fa700 1 -- 172.21.15.7:0/125132119 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc528007000 con 0x7fc5301ddf90 2023-03-03T13:01:22.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc52d7fa700 1 -- 172.21.15.7:0/125132119 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc528004240 con 0x7fc5301ddf90 2023-03-03T13:01:22.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc5370a2700 1 -- 172.21.15.7:0/125132119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc530154ef0 con 0x7fc5301ddf90 2023-03-03T13:01:22.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.222+0000 7fc52d7fa700 1 -- 172.21.15.7:0/125132119 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5280045d0 con 0x7fc5301ddf90 2023-03-03T13:01:22.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.223+0000 7fc5370a2700 1 -- 172.21.15.7:0/125132119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5300c71b0 con 0x7fc5301ddf90 2023-03-03T13:01:22.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.224+0000 7fc5370a2700 1 -- 172.21.15.7:0/125132119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc530074bb0 con 0x7fc5301ddf90 2023-03-03T13:01:22.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.224+0000 7fc52d7fa700 1 -- 172.21.15.7:0/125132119 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc528029030 con 0x7fc5301ddf90 2023-03-03T13:01:22.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.224+0000 7fc52d7fa700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5140682f0 0x7fc51406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:01:22.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.224+0000 7fc52d7fa700 1 -- 172.21.15.7:0/125132119 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc5280872e0 con 0x7fc5301ddf90 2023-03-03T13:01:22.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.224+0000 7fc534e3e700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5140682f0 0x7fc51406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:01:22.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.225+0000 7fc534e3e700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5140682f0 0x7fc51406a7a0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc524007fd0 tx=0x7fc524009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:01:22.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.229+0000 7fc52d7fa700 1 -- 172.21.15.7:0/125132119 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc528053680 con 0x7fc5301ddf90 2023-03-03T13:01:22.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.467+0000 7fc5370a2700 1 -- 172.21.15.7:0/125132119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc53014e150 con 0x7fc5301ddf90 2023-03-03T13:01:22.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.467+0000 7fc52d7fa700 1 -- 172.21.15.7:0/125132119 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc528009dc0 con 0x7fc5301ddf90 2023-03-03T13:01:22.470 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:01:22.470 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:01:22.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.469+0000 7fc5370a2700 1 -- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5140682f0 msgr2=0x7fc51406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:22.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5140682f0 0x7fc51406a7a0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc524007fd0 tx=0x7fc524009040 comp rx=0 tx=0).stop 2023-03-03T13:01:22.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 -- 172.21.15.7:0/125132119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5301ddf90 msgr2=0x7fc53014e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:01:22.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5301ddf90 0x7fc53014e4a0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc52800e940 tx=0x7fc528009750 comp rx=0 tx=0).stop 2023-03-03T13:01:22.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 -- 172.21.15.7:0/125132119 shutdown_connections 2023-03-03T13:01:22.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5140682f0 0x7fc51406a7a0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:22.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5301ddf90 0x7fc53014e4a0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:22.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc520004d90 0x7fc5301ea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:22.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 --2- 172.21.15.7:0/125132119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc53014ee70 0x7fc530156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:01:22.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 -- 172.21.15.7:0/125132119 >> 172.21.15.7:0/125132119 conn(0x7fc530071630 msgr2=0x7fc5301525b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:01:22.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 -- 172.21.15.7:0/125132119 shutdown_connections 2023-03-03T13:01:22.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:01:22.470+0000 7fc5370a2700 1 -- 172.21.15.7:0/125132119 wait complete. 2023-03-03T13:01:22.474 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:01:22.484 INFO:tasks.mds_thrash.fs.[b]:waiting for 39 secs before thrashing 2023-03-03T13:01:36.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:01:36.479 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:01.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:02:01.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.625+0000 7f953089e700 1 Processor -- start 2023-03-03T13:02:01.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.625+0000 7f953089e700 1 -- start start 2023-03-03T13:02:01.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f953089e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f952c0719e0 0x7f952c150930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:01.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f953089e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0c14f0 0x7f952c0c1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:01.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f953089e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952c0aae60 con 0x7f952c151e20 2023-03-03T13:02:01.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f953089e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c152270 con 0x7f952c0719e0 2023-03-03T13:02:01.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f953089e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c0c2760 con 0x7f952c0c14f0 2023-03-03T13:02:01.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f9529d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f952c151e20 0x7f952c0aa750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36848/0 (socket says 172.21.15.7:36848) 2023-03-03T13:02:01.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f9529d9b700 1 -- 172.21.15.7:0/213786232 learned_addr learned my addr 172.21.15.7:0/213786232 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:02:01.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f952a59c700 1 --2- 172.21.15.7:0/213786232 >> v2:172.21.15.143:3300/0 conn(0x7f952c0719e0 0x7f952c150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:01.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f952ad9d700 1 --2- 172.21.15.7:0/213786232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0c14f0 0x7f952c0c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:01.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f952959a700 1 -- 172.21.15.7:0/213786232 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182270360 0 0) 0x7f951c001110 con 0x7f952c151e20 2023-03-03T13:02:01.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.626+0000 7f952959a700 1 -- 172.21.15.7:0/213786232 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95140035b0 con 0x7f952c151e20 2023-03-03T13:02:01.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.627+0000 7f952a59c700 1 -- 172.21.15.7:0/213786232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0c14f0 msgr2=0x7f952c0c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:02:01.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.627+0000 7f952a59c700 1 --2- 172.21.15.7:0/213786232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0c14f0 0x7f952c0c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:01.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.627+0000 7f952a59c700 1 -- 172.21.15.7:0/213786232 >> v1:172.21.15.7:6789/0 conn(0x7f952c151e20 legacy=0x7f952c0aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:01.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.627+0000 7f952959a700 1 -- 172.21.15.7:0/213786232 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3906005507 0 0) 0x7f95140035b0 con 0x7f952c151e20 2023-03-03T13:02:01.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.627+0000 7f952a59c700 1 -- 172.21.15.7:0/213786232 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c0a93d0 con 0x7f952c0719e0 2023-03-03T13:02:01.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.627+0000 7f952a59c700 1 --2- 172.21.15.7:0/213786232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 0x7f952c150930 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9520009840 tx=0x7f9520009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f99e68163724e98 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:02:01.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.627+0000 7f952959a700 1 -- 172.21.15.7:0/213786232 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c001800 con 0x7f952c0719e0 2023-03-03T13:02:01.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.627+0000 7f952959a700 1 -- 172.21.15.7:0/213786232 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9520004030 con 0x7f952c0719e0 2023-03-03T13:02:01.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.627+0000 7f952959a700 1 -- 172.21.15.7:0/213786232 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95200043a0 con 0x7f952c0719e0 2023-03-03T13:02:01.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.628+0000 7f953089e700 1 -- 172.21.15.7:0/213786232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 msgr2=0x7f952c150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:01.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.628+0000 7f953089e700 1 --2- 172.21.15.7:0/213786232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 0x7f952c150930 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9520009840 tx=0x7f9520009b50 comp rx=0 tx=0).stop 2023-03-03T13:02:01.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.628+0000 7f953089e700 1 -- 172.21.15.7:0/213786232 shutdown_connections 2023-03-03T13:02:01.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.628+0000 7f953089e700 1 --2- 172.21.15.7:0/213786232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0c14f0 0x7f952c0c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:01.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.628+0000 7f953089e700 1 --2- 172.21.15.7:0/213786232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 0x7f952c150930 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:01.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.628+0000 7f953089e700 1 -- 172.21.15.7:0/213786232 >> 172.21.15.7:0/213786232 conn(0x7f952c0c62e0 msgr2=0x7f952c0c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:02:01.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.628+0000 7f953089e700 1 -- 172.21.15.7:0/213786232 shutdown_connections 2023-03-03T13:02:01.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.628+0000 7f953089e700 1 -- 172.21.15.7:0/213786232 wait complete. 2023-03-03T13:02:01.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.629+0000 7f953089e700 1 Processor -- start 2023-03-03T13:02:01.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.629+0000 7f953089e700 1 -- start start 2023-03-03T13:02:01.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f953089e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 0x7f952c143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:01.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f953089e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0c14f0 0x7f952c14c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:01.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f953089e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f952c151e20 0x7f952c14b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:01.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f953089e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c14b3b0 con 0x7f952c151e20 2023-03-03T13:02:01.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f953089e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c14ec20 con 0x7f952c0719e0 2023-03-03T13:02:01.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f953089e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c1e7860 con 0x7f952c0c14f0 2023-03-03T13:02:01.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f952ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f952c151e20 0x7f952c14b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:01.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f952a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 0x7f952c143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:01.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f952ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f952c151e20 0x7f952c14b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42876/0 (socket says 172.21.15.7:42876) 2023-03-03T13:02:01.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f952a59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 0x7f952c143070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:38646/0 (socket says 172.21.15.7:38646) 2023-03-03T13:02:01.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f952ad9d700 1 -- 172.21.15.7:0/2052171463 learned_addr learned my addr 172.21.15.7:0/2052171463 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:02:01.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.630+0000 7f9529d9b700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0c14f0 0x7f952c14c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:01.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f952ad9d700 1 -- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0c14f0 msgr2=0x7f952c14c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:01.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f952ad9d700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0c14f0 0x7f952c14c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:01.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f952ad9d700 1 -- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 msgr2=0x7f952c143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:01.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f952ad9d700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 0x7f952c143070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:01.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f952ad9d700 1 -- 172.21.15.7:0/2052171463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95200094a0 con 0x7f952c151e20 2023-03-03T13:02:01.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f952a59c700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 0x7f952c143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:02:01.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f952ad9d700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f952c151e20 0x7f952c14b7d0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f9524009d30 tx=0x7f952400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:02:01.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f951affd700 1 -- 172.21.15.7:0/2052171463 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952400cc90 con 0x7f952c151e20 2023-03-03T13:02:01.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f951affd700 1 -- 172.21.15.7:0/2052171463 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952400f040 con 0x7f952c151e20 2023-03-03T13:02:01.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f953089e700 1 -- 172.21.15.7:0/2052171463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c0a9990 con 0x7f952c151e20 2023-03-03T13:02:01.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f951affd700 1 -- 172.21.15.7:0/2052171463 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524009e80 con 0x7f952c151e20 2023-03-03T13:02:01.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.631+0000 7f953089e700 1 -- 172.21.15.7:0/2052171463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c0a96b0 con 0x7f952c151e20 2023-03-03T13:02:01.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.632+0000 7f953089e700 1 -- 172.21.15.7:0/2052171463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c0a9c70 con 0x7f952c151e20 2023-03-03T13:02:01.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.632+0000 7f951affd700 1 -- 172.21.15.7:0/2052171463 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9524014480 con 0x7f952c151e20 2023-03-03T13:02:01.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.633+0000 7f951affd700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f950c0682f0 0x7f950c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:01.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.633+0000 7f951affd700 1 -- 172.21.15.7:0/2052171463 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f95240869f0 con 0x7f952c151e20 2023-03-03T13:02:01.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.633+0000 7f952a59c700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f950c0682f0 0x7f950c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:01.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.634+0000 7f952a59c700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f950c0682f0 0x7f950c06a7a0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f95200090b0 tx=0x7f9520008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:02:01.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.637+0000 7f951affd700 1 -- 172.21.15.7:0/2052171463 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9524088e60 con 0x7f952c151e20 2023-03-03T13:02:01.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.879+0000 7f953089e700 1 -- 172.21.15.7:0/2052171463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f952c14b4f0 con 0x7f952c151e20 2023-03-03T13:02:01.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.879+0000 7f951affd700 1 -- 172.21.15.7:0/2052171463 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f952c14b4f0 con 0x7f952c151e20 2023-03-03T13:02:01.881 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:02:01.882 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:02:01.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 -- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f950c0682f0 msgr2=0x7f950c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:01.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f950c0682f0 0x7f950c06a7a0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f95200090b0 tx=0x7f9520008030 comp rx=0 tx=0).stop 2023-03-03T13:02:01.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 -- 172.21.15.7:0/2052171463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f952c151e20 msgr2=0x7f952c14b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:01.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f952c151e20 0x7f952c14b7d0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f9524009d30 tx=0x7f952400c3a0 comp rx=0 tx=0).stop 2023-03-03T13:02:01.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 -- 172.21.15.7:0/2052171463 shutdown_connections 2023-03-03T13:02:01.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f950c0682f0 0x7f950c06a7a0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:01.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f952c151e20 0x7f952c14b7d0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:01.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f952c0719e0 0x7f952c143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:01.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 --2- 172.21.15.7:0/2052171463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f952c0c14f0 0x7f952c14c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:01.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 -- 172.21.15.7:0/2052171463 >> 172.21.15.7:0/2052171463 conn(0x7f952c0c62e0 msgr2=0x7f952c0aa750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:02:01.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 -- 172.21.15.7:0/2052171463 shutdown_connections 2023-03-03T13:02:01.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:01.882+0000 7f953089e700 1 -- 172.21.15.7:0/2052171463 wait complete. 2023-03-03T13:02:01.885 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:02:01.895 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T13:02:06.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:06.684 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:36.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:36.823 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:02:59.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:02:59.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.346+0000 7f8284052700 1 Processor -- start 2023-03-03T13:02:59.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.346+0000 7f8284052700 1 -- start start 2023-03-03T13:02:59.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.346+0000 7f8284052700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c0719e0 0x7f827c150930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:59.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8284052700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c0aae60 con 0x7f827c151e20 2023-03-03T13:02:59.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8284052700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c152270 con 0x7f827c0c0c60 2023-03-03T13:02:59.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8284052700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0c2760 con 0x7f827c0719e0 2023-03-03T13:02:59.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f82815ed700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f827c151e20 0x7f827c0aa750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36898/0 (socket says 172.21.15.7:36898) 2023-03-03T13:02:59.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f82815ed700 1 -- 172.21.15.7:0/3685057715 learned_addr learned my addr 172.21.15.7:0/3685057715 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:02:59.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434863978 0 0) 0x7f826c001110 con 0x7f827c151e20 2023-03-03T13:02:59.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82680035b0 con 0x7f827c151e20 2023-03-03T13:02:59.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8281dee700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c0719e0 0x7f827c150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:59.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415918671 0 0) 0x7f82680035b0 con 0x7f827c151e20 2023-03-03T13:02:59.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c0719e0 msgr2=0x7f827c150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:02:59.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c0719e0 0x7f827c150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:59.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 >> v1:172.21.15.143:6789/0 conn(0x7f827c0c0c60 legacy=0x7f827c0c10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:59.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c0a93d0 con 0x7f827c151e20 2023-03-03T13:02:59.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f826c001800 con 0x7f827c151e20 2023-03-03T13:02:59.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 >> v1:172.21.15.7:6789/0 conn(0x7f827c151e20 legacy=0x7f827c0aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:59.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8268003aa0 0x7f8268003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:59.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c0719e0 0x7f8268004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:59.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8268004d90 0x7f8268005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:59.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0c2760 con 0x7f8268004d90 2023-03-03T13:02:59.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826800bc40 con 0x7f8268003aa0 2023-03-03T13:02:59.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.347+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826800e010 con 0x7f827c0719e0 2023-03-03T13:02:59.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f82825ef700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8268004d90 0x7f8268005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:59.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f8281dee700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8268003aa0 0x7f8268003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:59.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f82815ed700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c0719e0 0x7f8268004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:59.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f82825ef700 1 -- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c0719e0 msgr2=0x7f8268004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:02:59.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f82825ef700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c0719e0 0x7f8268004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:59.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f82825ef700 1 -- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8268003aa0 msgr2=0x7f8268003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:02:59.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f82825ef700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8268003aa0 0x7f8268003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:59.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f82825ef700 1 -- 172.21.15.7:0/3685057715 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82680035b0 con 0x7f8268004d90 2023-03-03T13:02:59.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f82825ef700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8268004d90 0x7f8268005200 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f827400c190 tx=0x7f8268005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3a11b8f3dd3852 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:02:59.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274001db0 con 0x7f8268004d90 2023-03-03T13:02:59.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827400ccd0 con 0x7f8268004d90 2023-03-03T13:02:59.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.348+0000 7f8280dec700 1 -- 172.21.15.7:0/3685057715 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274003c40 con 0x7f8268004d90 2023-03-03T13:02:59.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.349+0000 7f8284052700 1 -- 172.21.15.7:0/3685057715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8268004d90 msgr2=0x7f8268005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:59.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.349+0000 7f8284052700 1 --2- 172.21.15.7:0/3685057715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8268004d90 0x7f8268005200 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f827400c190 tx=0x7f8268005880 comp rx=0 tx=0).stop 2023-03-03T13:02:59.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.349+0000 7f82825ef700 1 -- 172.21.15.7:0/3685057715 reap_dead start 2023-03-03T13:02:59.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.349+0000 7f8284052700 1 -- 172.21.15.7:0/3685057715 shutdown_connections 2023-03-03T13:02:59.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.349+0000 7f8284052700 1 -- 172.21.15.7:0/3685057715 >> 172.21.15.7:0/3685057715 conn(0x7f827c0c62e0 msgr2=0x7f827c0c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:02:59.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.349+0000 7f8284052700 1 -- 172.21.15.7:0/3685057715 shutdown_connections 2023-03-03T13:02:59.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.349+0000 7f8284052700 1 -- 172.21.15.7:0/3685057715 wait complete. 2023-03-03T13:02:59.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.350+0000 7f8284052700 1 Processor -- start 2023-03-03T13:02:59.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.350+0000 7f8284052700 1 -- start start 2023-03-03T13:02:59.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f8284052700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f827c152f20 0x7f827c14f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:59.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f8284052700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c1532f0 0x7f827c1e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:59.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f8284052700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f827c14f380 0x7f827c1e9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:59.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f8284052700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c1e6d20 con 0x7f827c14f380 2023-03-03T13:02:59.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f8284052700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c14e8d0 con 0x7f827c152f20 2023-03-03T13:02:59.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f8284052700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c14ebb0 con 0x7f827c1532f0 2023-03-03T13:02:59.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82825ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f827c14f380 0x7f827c1e9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:59.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82825ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f827c14f380 0x7f827c1e9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42932/0 (socket says 172.21.15.7:42932) 2023-03-03T13:02:59.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82825ef700 1 -- 172.21.15.7:0/2110126170 learned_addr learned my addr 172.21.15.7:0/2110126170 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:02:59.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f8281dee700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f827c152f20 0x7f827c14f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:59.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82815ed700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c1532f0 0x7f827c1e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:59.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82825ef700 1 -- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c1532f0 msgr2=0x7f827c1e67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:59.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82825ef700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c1532f0 0x7f827c1e67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:59.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82825ef700 1 -- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f827c152f20 msgr2=0x7f827c14f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:59.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82825ef700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f827c152f20 0x7f827c14f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:59.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82825ef700 1 -- 172.21.15.7:0/2110126170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c0a9990 con 0x7f827c14f380 2023-03-03T13:02:59.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82825ef700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f827c14f380 0x7f827c1e9ad0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f8274015040 tx=0x7f82740014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:02:59.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82727fc700 1 -- 172.21.15.7:0/2110126170 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274001240 con 0x7f827c14f380 2023-03-03T13:02:59.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82727fc700 1 -- 172.21.15.7:0/2110126170 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82740050d0 con 0x7f827c14f380 2023-03-03T13:02:59.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.351+0000 7f82727fc700 1 -- 172.21.15.7:0/2110126170 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274005260 con 0x7f827c14f380 2023-03-03T13:02:59.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.352+0000 7f8284052700 1 -- 172.21.15.7:0/2110126170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827c0a96b0 con 0x7f827c14f380 2023-03-03T13:02:59.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.352+0000 7f8284052700 1 -- 172.21.15.7:0/2110126170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827c0a9c70 con 0x7f827c14f380 2023-03-03T13:02:59.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.352+0000 7f8284052700 1 -- 172.21.15.7:0/2110126170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827c074bb0 con 0x7f827c14f380 2023-03-03T13:02:59.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.353+0000 7f82727fc700 1 -- 172.21.15.7:0/2110126170 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f827402e070 con 0x7f827c14f380 2023-03-03T13:02:59.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.353+0000 7f82727fc700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82600682f0 0x7f826006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:02:59.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.353+0000 7f82727fc700 1 -- 172.21.15.7:0/2110126170 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8274087260 con 0x7f827c14f380 2023-03-03T13:02:59.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.354+0000 7f8281dee700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82600682f0 0x7f826006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:02:59.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.354+0000 7f8281dee700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82600682f0 0x7f826006a7a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8278009950 tx=0x7f8278008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:02:59.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.358+0000 7f82727fc700 1 -- 172.21.15.7:0/2110126170 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8274053600 con 0x7f827c14f380 2023-03-03T13:02:59.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.598+0000 7f8284052700 1 -- 172.21.15.7:0/2110126170 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f827c0c29c0 con 0x7f827c14f380 2023-03-03T13:02:59.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.598+0000 7f82727fc700 1 -- 172.21.15.7:0/2110126170 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8274056c20 con 0x7f827c14f380 2023-03-03T13:02:59.600 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:02:59.600 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:02:59.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 -- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82600682f0 msgr2=0x7f826006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:59.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82600682f0 0x7f826006a7a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8278009950 tx=0x7f8278008040 comp rx=0 tx=0).stop 2023-03-03T13:02:59.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 -- 172.21.15.7:0/2110126170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f827c14f380 msgr2=0x7f827c1e9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:02:59.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f827c14f380 0x7f827c1e9ad0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f8274015040 tx=0x7f82740014a0 comp rx=0 tx=0).stop 2023-03-03T13:02:59.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 -- 172.21.15.7:0/2110126170 shutdown_connections 2023-03-03T13:02:59.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82600682f0 0x7f826006a7a0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:59.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f827c14f380 0x7f827c1e9ad0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:59.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f827c152f20 0x7f827c14f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:59.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 --2- 172.21.15.7:0/2110126170 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f827c1532f0 0x7f827c1e67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:02:59.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 -- 172.21.15.7:0/2110126170 >> 172.21.15.7:0/2110126170 conn(0x7f827c0c62e0 msgr2=0x7f827c06be10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:02:59.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 -- 172.21.15.7:0/2110126170 shutdown_connections 2023-03-03T13:02:59.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:02:59.600+0000 7f8284052700 1 -- 172.21.15.7:0/2110126170 wait complete. 2023-03-03T13:02:59.604 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:02:59.614 INFO:tasks.mds_thrash.fs.[a]:waiting for 61 secs before thrashing 2023-03-03T13:03:03.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:03:04.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.034+0000 7fda3861f700 1 Processor -- start 2023-03-03T13:03:04.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.034+0000 7fda3861f700 1 -- start start 2023-03-03T13:03:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.035+0000 7fda3861f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda30151d40 0x7fda300736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.035+0000 7fda3861f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda300c2930 0x7fda30157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.035+0000 7fda3861f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda30073c30 0x7fda30157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.035+0000 7fda3861f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3006c2a0 con 0x7fda30151d40 2023-03-03T13:03:04.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.035+0000 7fda3861f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3006d430 con 0x7fda30073c30 2023-03-03T13:03:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.035+0000 7fda3861f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3006cb00 con 0x7fda300c2930 2023-03-03T13:03:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.035+0000 7fda363bb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda30151d40 0x7fda300736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda363bb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda30151d40 0x7fda300736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42940/0 (socket says 172.21.15.7:42940) 2023-03-03T13:03:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda363bb700 1 -- 172.21.15.7:0/2756745943 learned_addr learned my addr 172.21.15.7:0/2756745943 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:03:04.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda35bba700 1 --2- 172.21.15.7:0/2756745943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda300c2930 0x7fda30157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda36bbc700 1 --2- 172.21.15.7:0/2756745943 >> v2:172.21.15.143:3300/0 conn(0x7fda30073c30 0x7fda30157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda363bb700 1 -- 172.21.15.7:0/2756745943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda300c2930 msgr2=0x7fda30157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:03:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda363bb700 1 --2- 172.21.15.7:0/2756745943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda300c2930 0x7fda30157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda363bb700 1 -- 172.21.15.7:0/2756745943 >> v2:172.21.15.143:3300/0 conn(0x7fda30073c30 msgr2=0x7fda30157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:03:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda363bb700 1 --2- 172.21.15.7:0/2756745943 >> v2:172.21.15.143:3300/0 conn(0x7fda30073c30 0x7fda30157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda363bb700 1 -- 172.21.15.7:0/2756745943 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3006cde0 con 0x7fda30151d40 2023-03-03T13:03:04.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda36bbc700 1 --2- 172.21.15.7:0/2756745943 >> v2:172.21.15.143:3300/0 conn(0x7fda30073c30 0x7fda30157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:03:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda363bb700 1 --2- 172.21.15.7:0/2756745943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 0x7fda300736f0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fda300c3db0 tx=0x7fda2c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ca0bbbe66afc82 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:03:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda353b9700 1 -- 172.21.15.7:0/2756745943 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c011830 con 0x7fda30151d40 2023-03-03T13:03:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda353b9700 1 -- 172.21.15.7:0/2756745943 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c011990 con 0x7fda30151d40 2023-03-03T13:03:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.036+0000 7fda353b9700 1 -- 172.21.15.7:0/2756745943 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c011d20 con 0x7fda30151d40 2023-03-03T13:03:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.037+0000 7fda3861f700 1 -- 172.21.15.7:0/2756745943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 msgr2=0x7fda300736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.037+0000 7fda3861f700 1 --2- 172.21.15.7:0/2756745943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 0x7fda300736f0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fda300c3db0 tx=0x7fda2c009bd0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.037+0000 7fda3861f700 1 -- 172.21.15.7:0/2756745943 shutdown_connections 2023-03-03T13:03:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.037+0000 7fda3861f700 1 --2- 172.21.15.7:0/2756745943 >> v2:172.21.15.143:3300/0 conn(0x7fda30073c30 0x7fda30157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.037+0000 7fda3861f700 1 --2- 172.21.15.7:0/2756745943 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 0x7fda300736f0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.037+0000 7fda3861f700 1 --2- 172.21.15.7:0/2756745943 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda300c2930 0x7fda30157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.037+0000 7fda3861f700 1 -- 172.21.15.7:0/2756745943 >> 172.21.15.7:0/2756745943 conn(0x7fda300a94d0 msgr2=0x7fda300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:03:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.037+0000 7fda3861f700 1 -- 172.21.15.7:0/2756745943 shutdown_connections 2023-03-03T13:03:04.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.037+0000 7fda3861f700 1 -- 172.21.15.7:0/2756745943 wait complete. 2023-03-03T13:03:04.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.038+0000 7fda3861f700 1 Processor -- start 2023-03-03T13:03:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.039+0000 7fda3861f700 1 -- start start 2023-03-03T13:03:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.039+0000 7fda3861f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda30073c30 0x7fda300c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.039+0000 7fda3861f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300c2930 0x7fda301ed430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.039+0000 7fda3861f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 0x7fda301ed970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.039+0000 7fda3861f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300bf6e0 con 0x7fda30151d40 2023-03-03T13:03:04.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.039+0000 7fda3861f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300aa600 con 0x7fda300c2930 2023-03-03T13:03:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.039+0000 7fda3861f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda300ab3d0 con 0x7fda30073c30 2023-03-03T13:03:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda36bbc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 0x7fda301ed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda36bbc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 0x7fda301ed970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42950/0 (socket says 172.21.15.7:42950) 2023-03-03T13:03:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda36bbc700 1 -- 172.21.15.7:0/292671692 learned_addr learned my addr 172.21.15.7:0/292671692 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:03:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda363bb700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda30073c30 0x7fda300c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:04.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda35bba700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300c2930 0x7fda301ed430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda36bbc700 1 -- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda30073c30 msgr2=0x7fda300c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda36bbc700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda30073c30 0x7fda300c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda36bbc700 1 -- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300c2930 msgr2=0x7fda301ed430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda36bbc700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300c2930 0x7fda301ed430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda36bbc700 1 -- 172.21.15.7:0/292671692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c009550 con 0x7fda30151d40 2023-03-03T13:03:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda363bb700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda30073c30 0x7fda300c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:03:04.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda35bba700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300c2930 0x7fda301ed430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:03:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda36bbc700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 0x7fda301ed970 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fda28009fa0 tx=0x7fda2800d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:03:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda22ffd700 1 -- 172.21.15.7:0/292671692 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28003a20 con 0x7fda30151d40 2023-03-03T13:03:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda22ffd700 1 -- 172.21.15.7:0/292671692 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2800f040 con 0x7fda30151d40 2023-03-03T13:03:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda3861f700 1 -- 172.21.15.7:0/292671692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3006bea0 con 0x7fda30151d40 2023-03-03T13:03:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda22ffd700 1 -- 172.21.15.7:0/292671692 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28003de0 con 0x7fda30151d40 2023-03-03T13:03:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.040+0000 7fda3861f700 1 -- 172.21.15.7:0/292671692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3006d0c0 con 0x7fda30151d40 2023-03-03T13:03:04.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.041+0000 7fda3861f700 1 -- 172.21.15.7:0/292671692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3006c030 con 0x7fda30151d40 2023-03-03T13:03:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.042+0000 7fda22ffd700 1 -- 172.21.15.7:0/292671692 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda2801c030 con 0x7fda30151d40 2023-03-03T13:03:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.042+0000 7fda22ffd700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda140682f0 0x7fda1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.042+0000 7fda22ffd700 1 -- 172.21.15.7:0/292671692 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fda280861f0 con 0x7fda30151d40 2023-03-03T13:03:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.042+0000 7fda363bb700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda140682f0 0x7fda1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:04.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.043+0000 7fda363bb700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda140682f0 0x7fda1406a7a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009840 tx=0x7fda2c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:03:04.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.046+0000 7fda22ffd700 1 -- 172.21.15.7:0/292671692 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda28052530 con 0x7fda30151d40 2023-03-03T13:03:04.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.290+0000 7fda3861f700 1 -- 172.21.15.7:0/292671692 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda301ed100 con 0x7fda30151d40 2023-03-03T13:03:04.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.290+0000 7fda22ffd700 1 -- 172.21.15.7:0/292671692 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fda28055b50 con 0x7fda30151d40 2023-03-03T13:03:04.292 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:03:04.292 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:03:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 -- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda140682f0 msgr2=0x7fda1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:04.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda140682f0 0x7fda1406a7a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009840 tx=0x7fda2c008030 comp rx=0 tx=0).stop 2023-03-03T13:03:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 -- 172.21.15.7:0/292671692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 msgr2=0x7fda301ed970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 0x7fda301ed970 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fda28009fa0 tx=0x7fda2800d460 comp rx=0 tx=0).stop 2023-03-03T13:03:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 -- 172.21.15.7:0/292671692 shutdown_connections 2023-03-03T13:03:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda140682f0 0x7fda1406a7a0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda30151d40 0x7fda301ed970 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda30073c30 0x7fda300c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 --2- 172.21.15.7:0/292671692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda300c2930 0x7fda301ed430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 -- 172.21.15.7:0/292671692 >> 172.21.15.7:0/292671692 conn(0x7fda300a94d0 msgr2=0x7fda30158280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:03:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 -- 172.21.15.7:0/292671692 shutdown_connections 2023-03-03T13:03:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:04.292+0000 7fda3861f700 1 -- 172.21.15.7:0/292671692 wait complete. 2023-03-03T13:03:04.296 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:03:04.306 INFO:tasks.mds_thrash.fs.[b]:waiting for 15 secs before thrashing 2023-03-03T13:03:06.891 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:06.894 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:19.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:03:19.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.443+0000 7f0db7282700 1 Processor -- start 2023-03-03T13:03:19.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.444+0000 7f0db7282700 1 -- start start 2023-03-03T13:03:19.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.444+0000 7f0db7282700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0db00c5b00 0x7f0db00bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:19.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.444+0000 7f0db7282700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db00c08d0 con 0x7f0db0155410 2023-03-03T13:03:19.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0db7282700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db0154930 con 0x7f0db0151e80 2023-03-03T13:03:19.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0db7282700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db0152250 con 0x7f0db00c5b00 2023-03-03T13:03:19.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0db481d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0db0155410 0x7f0db00bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36940/0 (socket says 172.21.15.7:36940) 2023-03-03T13:03:19.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0db481d700 1 -- 172.21.15.7:0/2771974781 learned_addr learned my addr 172.21.15.7:0/2771974781 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:03:19.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0db581f700 1 --2- 172.21.15.7:0/2771974781 >> v2:172.21.15.143:3300/0 conn(0x7f0db00c5b00 0x7f0db00bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:19.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0da3fff700 1 -- 172.21.15.7:0/2771974781 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494659908 0 0) 0x7f0db00c08d0 con 0x7f0db0155410 2023-03-03T13:03:19.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0da3fff700 1 -- 172.21.15.7:0/2771974781 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c0035b0 con 0x7f0db0155410 2023-03-03T13:03:19.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0db581f700 1 -- 172.21.15.7:0/2771974781 >> v1:172.21.15.143:6789/0 conn(0x7f0db0151e80 legacy=0x7f0db00c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:19.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0db581f700 1 -- 172.21.15.7:0/2771974781 >> v1:172.21.15.7:6789/0 conn(0x7f0db0155410 legacy=0x7f0db00bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:19.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0db581f700 1 -- 172.21.15.7:0/2771974781 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0154c10 con 0x7f0db00c5b00 2023-03-03T13:03:19.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.445+0000 7f0db581f700 1 --2- 172.21.15.7:0/2771974781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db00c5b00 0x7f0db00bfff0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009d90 tx=0x7f0da800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43077f80366aa65a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:03:19.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.446+0000 7f0da3fff700 1 -- 172.21.15.7:0/2771974781 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da800e070 con 0x7f0db00c5b00 2023-03-03T13:03:19.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.446+0000 7f0da3fff700 1 -- 172.21.15.7:0/2771974781 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da800bb20 con 0x7f0db00c5b00 2023-03-03T13:03:19.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.446+0000 7f0da3fff700 1 -- 172.21.15.7:0/2771974781 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8012430 con 0x7f0db00c5b00 2023-03-03T13:03:19.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.446+0000 7f0db7282700 1 -- 172.21.15.7:0/2771974781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db00c5b00 msgr2=0x7f0db00bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:19.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.446+0000 7f0db7282700 1 --2- 172.21.15.7:0/2771974781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db00c5b00 0x7f0db00bfff0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009d90 tx=0x7f0da800b330 comp rx=0 tx=0).stop 2023-03-03T13:03:19.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.446+0000 7f0db7282700 1 -- 172.21.15.7:0/2771974781 shutdown_connections 2023-03-03T13:03:19.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.446+0000 7f0db7282700 1 --2- 172.21.15.7:0/2771974781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db00c5b00 0x7f0db00bfff0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:19.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.446+0000 7f0db7282700 1 -- 172.21.15.7:0/2771974781 >> 172.21.15.7:0/2771974781 conn(0x7f0db0071630 msgr2=0x7f0db006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:03:19.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.446+0000 7f0db7282700 1 -- 172.21.15.7:0/2771974781 shutdown_connections 2023-03-03T13:03:19.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.446+0000 7f0db7282700 1 -- 172.21.15.7:0/2771974781 wait complete. 2023-03-03T13:03:19.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.447+0000 7f0db7282700 1 Processor -- start 2023-03-03T13:03:19.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.447+0000 7f0db7282700 1 -- start start 2023-03-03T13:03:19.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db7282700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db00c5b00 0x7f0db0153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:19.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db7282700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db0151e80 0x7f0db01461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:19.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db7282700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0155410 0x7f0db0146710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:19.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db7282700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db0146e80 con 0x7f0db00c5b00 2023-03-03T13:03:19.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db7282700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db00c9670 con 0x7f0db0155410 2023-03-03T13:03:19.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db7282700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db0146c50 con 0x7f0db0151e80 2023-03-03T13:03:19.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db501e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db00c5b00 0x7f0db0153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:19.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db501e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db00c5b00 0x7f0db0153a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42966/0 (socket says 172.21.15.7:42966) 2023-03-03T13:03:19.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db501e700 1 -- 172.21.15.7:0/2057200114 learned_addr learned my addr 172.21.15.7:0/2057200114 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:03:19.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db581f700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0155410 0x7f0db0146710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:19.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.448+0000 7f0db481d700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db0151e80 0x7f0db01461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0db501e700 1 -- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db0151e80 msgr2=0x7f0db01461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0db501e700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db0151e80 0x7f0db01461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0db501e700 1 -- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0155410 msgr2=0x7f0db0146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0db501e700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0155410 0x7f0db0146710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0db501e700 1 -- 172.21.15.7:0/2057200114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da8009a70 con 0x7f0db00c5b00 2023-03-03T13:03:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0db501e700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db00c5b00 0x7f0db0153a60 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00c4e0 tx=0x7f0dac00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:03:19.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0da1ffb700 1 -- 172.21.15.7:0/2057200114 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac0052d0 con 0x7f0db00c5b00 2023-03-03T13:03:19.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0da1ffb700 1 -- 172.21.15.7:0/2057200114 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dac005430 con 0x7f0db00c5b00 2023-03-03T13:03:19.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0db7282700 1 -- 172.21.15.7:0/2057200114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db00c5220 con 0x7f0db00c5b00 2023-03-03T13:03:19.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0da1ffb700 1 -- 172.21.15.7:0/2057200114 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dac0038c0 con 0x7f0db00c5b00 2023-03-03T13:03:19.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.449+0000 7f0db7282700 1 -- 172.21.15.7:0/2057200114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db0154ef0 con 0x7f0db00c5b00 2023-03-03T13:03:19.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.450+0000 7f0db7282700 1 -- 172.21.15.7:0/2057200114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db00c0530 con 0x7f0db00c5b00 2023-03-03T13:03:19.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.451+0000 7f0da1ffb700 1 -- 172.21.15.7:0/2057200114 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0dac019030 con 0x7f0db00c5b00 2023-03-03T13:03:19.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.451+0000 7f0da1ffb700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d94068320 0x7f0d9406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:03:19.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.451+0000 7f0da1ffb700 1 -- 172.21.15.7:0/2057200114 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0dac085450 con 0x7f0db00c5b00 2023-03-03T13:03:19.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.451+0000 7f0db481d700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d94068320 0x7f0d9406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:03:19.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.452+0000 7f0db481d700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d94068320 0x7f0d9406a7d0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0db0152ae0 tx=0x7f0da4007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:03:19.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.455+0000 7f0da1ffb700 1 -- 172.21.15.7:0/2057200114 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0dac051770 con 0x7f0db00c5b00 2023-03-03T13:03:19.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.694+0000 7f0db7282700 1 -- 172.21.15.7:0/2057200114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0db0074bb0 con 0x7f0db00c5b00 2023-03-03T13:03:19.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.694+0000 7f0da1ffb700 1 -- 172.21.15.7:0/2057200114 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0dac054d90 con 0x7f0db00c5b00 2023-03-03T13:03:19.696 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:03:19.696 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:03:19.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 -- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d94068320 msgr2=0x7f0d9406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:19.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d94068320 0x7f0d9406a7d0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0db0152ae0 tx=0x7f0da4007dc0 comp rx=0 tx=0).stop 2023-03-03T13:03:19.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 -- 172.21.15.7:0/2057200114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db00c5b00 msgr2=0x7f0db0153a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:03:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db00c5b00 0x7f0db0153a60 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f0dac00c4e0 tx=0x7f0dac00c7f0 comp rx=0 tx=0).stop 2023-03-03T13:03:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 -- 172.21.15.7:0/2057200114 shutdown_connections 2023-03-03T13:03:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0db0155410 0x7f0db0146710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d94068320 0x7f0d9406a7d0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0db00c5b00 0x7f0db0153a60 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 --2- 172.21.15.7:0/2057200114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0db0151e80 0x7f0db01461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:03:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 -- 172.21.15.7:0/2057200114 >> 172.21.15.7:0/2057200114 conn(0x7f0db0071630 msgr2=0x7f0db014fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:03:19.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 -- 172.21.15.7:0/2057200114 shutdown_connections 2023-03-03T13:03:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:03:19.696+0000 7f0db7282700 1 -- 172.21.15.7:0/2057200114 wait complete. 2023-03-03T13:03:19.700 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:03:19.710 INFO:tasks.mds_thrash.fs.[b]:waiting for 67 secs before thrashing 2023-03-03T13:03:36.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:03:36.965 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:00.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:04:00.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.751+0000 7fa4490a9700 1 Processor -- start 2023-03-03T13:04:00.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.751+0000 7fa4490a9700 1 -- start start 2023-03-03T13:04:00.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.752+0000 7fa4490a9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa444155410 0x7fa4440bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:00.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.752+0000 7fa4490a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 0x7fa4440bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:00.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.752+0000 7fa4490a9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4440c08d0 con 0x7fa444151e80 2023-03-03T13:04:00.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.752+0000 7fa4490a9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa444152250 con 0x7fa444155410 2023-03-03T13:04:00.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.752+0000 7fa4490a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4440c0fb0 con 0x7fa4440c5b00 2023-03-03T13:04:00.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.752+0000 7fa44259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa444155410 0x7fa4440bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:00.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.752+0000 7fa442d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa444151e80 0x7fa4440c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36966/0 (socket says 172.21.15.7:36966) 2023-03-03T13:04:00.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.752+0000 7fa442d9d700 1 -- 172.21.15.7:0/1510462728 learned_addr learned my addr 172.21.15.7:0/1510462728 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:04:00.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa44359e700 1 --2- 172.21.15.7:0/1510462728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 0x7fa4440bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:00.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa441d9b700 1 -- 172.21.15.7:0/1510462728 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927564835 0 0) 0x7fa4440c08d0 con 0x7fa444151e80 2023-03-03T13:04:00.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa441d9b700 1 -- 172.21.15.7:0/1510462728 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4300035b0 con 0x7fa444151e80 2023-03-03T13:04:00.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa44259c700 1 -- 172.21.15.7:0/1510462728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 msgr2=0x7fa4440bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:04:00.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa44259c700 1 --2- 172.21.15.7:0/1510462728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 0x7fa4440bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:00.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa44259c700 1 -- 172.21.15.7:0/1510462728 >> v1:172.21.15.7:6789/0 conn(0x7fa444151e80 legacy=0x7fa4440c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:00.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa44259c700 1 -- 172.21.15.7:0/1510462728 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa444154930 con 0x7fa444155410 2023-03-03T13:04:00.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa44359e700 1 --2- 172.21.15.7:0/1510462728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 0x7fa4440bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:04:00.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa44259c700 1 --2- 172.21.15.7:0/1510462728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444155410 0x7fa4440bf8e0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009840 tx=0x7fa42c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2974751896edce server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:04:00.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa441d9b700 1 -- 172.21.15.7:0/1510462728 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c011830 con 0x7fa444155410 2023-03-03T13:04:00.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa441d9b700 1 -- 172.21.15.7:0/1510462728 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42c011990 con 0x7fa444155410 2023-03-03T13:04:00.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.753+0000 7fa441d9b700 1 -- 172.21.15.7:0/1510462728 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c011d20 con 0x7fa444155410 2023-03-03T13:04:00.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.754+0000 7fa4490a9700 1 -- 172.21.15.7:0/1510462728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444155410 msgr2=0x7fa4440bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:00.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.754+0000 7fa4490a9700 1 --2- 172.21.15.7:0/1510462728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444155410 0x7fa4440bf8e0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009840 tx=0x7fa42c009b50 comp rx=0 tx=0).stop 2023-03-03T13:04:00.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.754+0000 7fa4490a9700 1 -- 172.21.15.7:0/1510462728 shutdown_connections 2023-03-03T13:04:00.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.754+0000 7fa4490a9700 1 --2- 172.21.15.7:0/1510462728 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 0x7fa4440bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:00.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.754+0000 7fa4490a9700 1 --2- 172.21.15.7:0/1510462728 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444155410 0x7fa4440bf8e0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:00.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.754+0000 7fa4490a9700 1 -- 172.21.15.7:0/1510462728 >> 172.21.15.7:0/1510462728 conn(0x7fa444071630 msgr2=0x7fa44406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:04:00.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.754+0000 7fa4490a9700 1 -- 172.21.15.7:0/1510462728 shutdown_connections 2023-03-03T13:04:00.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.754+0000 7fa4490a9700 1 -- 172.21.15.7:0/1510462728 wait complete. 2023-03-03T13:04:00.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.755+0000 7fa4490a9700 1 Processor -- start 2023-03-03T13:04:00.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.755+0000 7fa4490a9700 1 -- start start 2023-03-03T13:04:00.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa4490a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 0x7fa44414f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:00.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa4490a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444151e80 0x7fa4441459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:00.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa4490a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa444155410 0x7fa44414de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:00.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa4490a9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa444150720 con 0x7fa444151e80 2023-03-03T13:04:00.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa4490a9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4441ea980 con 0x7fa444155410 2023-03-03T13:04:00.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa4490a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4441eac60 con 0x7fa4440c5b00 2023-03-03T13:04:00.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa44259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444151e80 0x7fa4441459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:00.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa44259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444151e80 0x7fa4441459a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42994/0 (socket says 172.21.15.7:42994) 2023-03-03T13:04:00.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa44259c700 1 -- 172.21.15.7:0/1547040572 learned_addr learned my addr 172.21.15.7:0/1547040572 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:04:00.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa442d9d700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 0x7fa44414f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:00.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa44359e700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa444155410 0x7fa44414de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:00.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa44259c700 1 -- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 msgr2=0x7fa44414f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:00.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa44259c700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 0x7fa44414f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:00.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa44259c700 1 -- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa444155410 msgr2=0x7fa44414de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:00.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa44259c700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa444155410 0x7fa44414de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:00.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa44259c700 1 -- 172.21.15.7:0/1547040572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c0094a0 con 0x7fa444151e80 2023-03-03T13:04:00.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa442d9d700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 0x7fa44414f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:04:00.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.756+0000 7fa44359e700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa444155410 0x7fa44414de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:04:00.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.757+0000 7fa44259c700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444151e80 0x7fa4441459a0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fa42c0090b0 tx=0x7fa42c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:04:00.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.757+0000 7fa43b7fe700 1 -- 172.21.15.7:0/1547040572 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c0036a0 con 0x7fa444151e80 2023-03-03T13:04:00.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.757+0000 7fa43b7fe700 1 -- 172.21.15.7:0/1547040572 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42c01f070 con 0x7fa444151e80 2023-03-03T13:04:00.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.757+0000 7fa4490a9700 1 -- 172.21.15.7:0/1547040572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa444154ef0 con 0x7fa444151e80 2023-03-03T13:04:00.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.757+0000 7fa43b7fe700 1 -- 172.21.15.7:0/1547040572 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c003a20 con 0x7fa444151e80 2023-03-03T13:04:00.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.757+0000 7fa4490a9700 1 -- 172.21.15.7:0/1547040572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa444154c10 con 0x7fa444151e80 2023-03-03T13:04:00.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.758+0000 7fa4490a9700 1 -- 172.21.15.7:0/1547040572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4440c5220 con 0x7fa444151e80 2023-03-03T13:04:00.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.758+0000 7fa43b7fe700 1 -- 172.21.15.7:0/1547040572 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa42c00f050 con 0x7fa444151e80 2023-03-03T13:04:00.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.759+0000 7fa43b7fe700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4240682f0 0x7fa42406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:00.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.759+0000 7fa43b7fe700 1 -- 172.21.15.7:0/1547040572 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa42c021070 con 0x7fa444151e80 2023-03-03T13:04:00.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.759+0000 7fa442d9d700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4240682f0 0x7fa42406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:00.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.760+0000 7fa442d9d700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4240682f0 0x7fa42406a7a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa4340011e0 tx=0x7fa434009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:04:00.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:00.763+0000 7fa43b7fe700 1 -- 172.21.15.7:0/1547040572 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa42c053e90 con 0x7fa444151e80 2023-03-03T13:04:01.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.003+0000 7fa4490a9700 1 -- 172.21.15.7:0/1547040572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4441523a0 con 0x7fa444151e80 2023-03-03T13:04:01.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.003+0000 7fa43b7fe700 1 -- 172.21.15.7:0/1547040572 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa42c0574b0 con 0x7fa444151e80 2023-03-03T13:04:01.005 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:04:01.005 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:04:01.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 -- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4240682f0 msgr2=0x7fa42406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:01.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4240682f0 0x7fa42406a7a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa4340011e0 tx=0x7fa434009040 comp rx=0 tx=0).stop 2023-03-03T13:04:01.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 -- 172.21.15.7:0/1547040572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444151e80 msgr2=0x7fa4441459a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:01.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444151e80 0x7fa4441459a0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fa42c0090b0 tx=0x7fa42c004750 comp rx=0 tx=0).stop 2023-03-03T13:04:01.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 -- 172.21.15.7:0/1547040572 shutdown_connections 2023-03-03T13:04:01.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa444155410 0x7fa44414de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:01.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4440c5b00 0x7fa44414f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:01.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4240682f0 0x7fa42406a7a0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:01.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 --2- 172.21.15.7:0/1547040572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa444151e80 0x7fa4441459a0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:01.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 -- 172.21.15.7:0/1547040572 >> 172.21.15.7:0/1547040572 conn(0x7fa444071630 msgr2=0x7fa444153750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:04:01.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 -- 172.21.15.7:0/1547040572 shutdown_connections 2023-03-03T13:04:01.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:01.006+0000 7fa4490a9700 1 -- 172.21.15.7:0/1547040572 wait complete. 2023-03-03T13:04:01.009 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:04:01.019 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T13:04:07.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:07.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:26.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:04:26.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.849+0000 7f15a15a2700 1 Processor -- start 2023-03-03T13:04:26.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.849+0000 7f15a15a2700 1 -- start start 2023-03-03T13:04:26.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f15a15a2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f159c06b9b0 0x7f159c155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:26.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f15a15a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c157b40 0x7f159c0c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:26.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f15a15a2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c0ab060 con 0x7f159c0aac90 2023-03-03T13:04:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f15a15a2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159c0c35f0 con 0x7f159c06b9b0 2023-03-03T13:04:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f15a15a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159c06bf50 con 0x7f159c157b40 2023-03-03T13:04:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f159a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f159c06b9b0 0x7f159c155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f159a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f159c06b9b0 0x7f159c155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43036/0 (socket says 172.21.15.7:43036) 2023-03-03T13:04:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f159a7fc700 1 -- 172.21.15.7:0/1796174164 learned_addr learned my addr 172.21.15.7:0/1796174164 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:04:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f159b7fe700 1 --2- 172.21.15.7:0/1796174164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c157b40 0x7f159c0c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f159a7fc700 1 -- 172.21.15.7:0/1796174164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c157b40 msgr2=0x7f159c0c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:04:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f159a7fc700 1 --2- 172.21.15.7:0/1796174164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c157b40 0x7f159c0c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f159a7fc700 1 -- 172.21.15.7:0/1796174164 >> v1:172.21.15.143:6789/0 conn(0x7f159c0aac90 legacy=0x7f159c071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f159a7fc700 1 -- 172.21.15.7:0/1796174164 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c06c230 con 0x7f159c06b9b0 2023-03-03T13:04:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.850+0000 7f159a7fc700 1 --2- 172.21.15.7:0/1796174164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c06b9b0 0x7f159c155180 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f15900098f0 tx=0x7f1590009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca051bc202b1abca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:04:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f1599ffb700 1 -- 172.21.15.7:0/1796174164 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590011830 con 0x7f159c06b9b0 2023-03-03T13:04:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f1599ffb700 1 -- 172.21.15.7:0/1796174164 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1590011990 con 0x7f159c06b9b0 2023-03-03T13:04:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f1599ffb700 1 -- 172.21.15.7:0/1796174164 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590011d20 con 0x7f159c06b9b0 2023-03-03T13:04:26.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f15a15a2700 1 -- 172.21.15.7:0/1796174164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c06b9b0 msgr2=0x7f159c155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f15a15a2700 1 --2- 172.21.15.7:0/1796174164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c06b9b0 0x7f159c155180 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f15900098f0 tx=0x7f1590009c00 comp rx=0 tx=0).stop 2023-03-03T13:04:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f15a15a2700 1 -- 172.21.15.7:0/1796174164 shutdown_connections 2023-03-03T13:04:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f15a15a2700 1 --2- 172.21.15.7:0/1796174164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c157b40 0x7f159c0c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f15a15a2700 1 --2- 172.21.15.7:0/1796174164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c06b9b0 0x7f159c155180 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f15a15a2700 1 -- 172.21.15.7:0/1796174164 >> 172.21.15.7:0/1796174164 conn(0x7f159c0736f0 msgr2=0x7f159c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:04:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f15a15a2700 1 -- 172.21.15.7:0/1796174164 shutdown_connections 2023-03-03T13:04:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.851+0000 7f15a15a2700 1 -- 172.21.15.7:0/1796174164 wait complete. 2023-03-03T13:04:26.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.852+0000 7f15a15a2700 1 Processor -- start 2023-03-03T13:04:26.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.852+0000 7f15a15a2700 1 -- start start 2023-03-03T13:04:26.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f15a15a2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f159c06b9b0 0x7f159c0bec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:26.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f15a15a2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c0aac90 0x7f159c0c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:26.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f15a15a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c157b40 0x7f159c0c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:26.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f15a15a2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c0ac330 con 0x7f159c0aac90 2023-03-03T13:04:26.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f15a15a2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f159c0ab460 con 0x7f159c06b9b0 2023-03-03T13:04:26.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f15a15a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159c1edf10 con 0x7f159c157b40 2023-03-03T13:04:26.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f159a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c0aac90 0x7f159c0c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:26.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f159a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c0aac90 0x7f159c0c8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43042/0 (socket says 172.21.15.7:43042) 2023-03-03T13:04:26.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f159a7fc700 1 -- 172.21.15.7:0/2473920773 learned_addr learned my addr 172.21.15.7:0/2473920773 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:04:26.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f159affd700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f159c06b9b0 0x7f159c0bec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:26.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.853+0000 7f159b7fe700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c157b40 0x7f159c0c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:26.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f159a7fc700 1 -- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c157b40 msgr2=0x7f159c0c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:26.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f159a7fc700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c157b40 0x7f159c0c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:26.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f159a7fc700 1 -- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f159c06b9b0 msgr2=0x7f159c0bec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:26.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f159a7fc700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f159c06b9b0 0x7f159c0bec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:26.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f159a7fc700 1 -- 172.21.15.7:0/2473920773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1590009550 con 0x7f159c0aac90 2023-03-03T13:04:26.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f159a7fc700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c0aac90 0x7f159c0c8670 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f1590009c40 tx=0x7f1590004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:04:26.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f15837fe700 1 -- 172.21.15.7:0/2473920773 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590004510 con 0x7f159c0aac90 2023-03-03T13:04:26.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f15837fe700 1 -- 172.21.15.7:0/2473920773 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159001e070 con 0x7f159c0aac90 2023-03-03T13:04:26.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f15837fe700 1 -- 172.21.15.7:0/2473920773 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15900038e0 con 0x7f159c0aac90 2023-03-03T13:04:26.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f15a15a2700 1 -- 172.21.15.7:0/2473920773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159c154980 con 0x7f159c0aac90 2023-03-03T13:04:26.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.854+0000 7f15a15a2700 1 -- 172.21.15.7:0/2473920773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f159c1546a0 con 0x7f159c0aac90 2023-03-03T13:04:26.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.855+0000 7f15a15a2700 1 -- 172.21.15.7:0/2473920773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159c154c60 con 0x7f159c0aac90 2023-03-03T13:04:26.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.855+0000 7f15837fe700 1 -- 172.21.15.7:0/2473920773 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f159000f050 con 0x7f159c0aac90 2023-03-03T13:04:26.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.856+0000 7f15837fe700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15880682f0 0x7f158806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:04:26.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.856+0000 7f15837fe700 1 -- 172.21.15.7:0/2473920773 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1590086fe0 con 0x7f159c0aac90 2023-03-03T13:04:26.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.856+0000 7f159affd700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15880682f0 0x7f158806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:04:26.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.857+0000 7f159affd700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15880682f0 0x7f158806a7a0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f158c007fd0 tx=0x7f158c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:04:26.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:26.860+0000 7f15837fe700 1 -- 172.21.15.7:0/2473920773 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1590053300 con 0x7f159c0aac90 2023-03-03T13:04:27.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.095+0000 7f15a15a2700 1 -- 172.21.15.7:0/2473920773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f159c0ac5d0 con 0x7f159c0aac90 2023-03-03T13:04:27.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.096+0000 7f15837fe700 1 -- 172.21.15.7:0/2473920773 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1590020090 con 0x7f159c0aac90 2023-03-03T13:04:27.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:04:27.098 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:04:27.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 -- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15880682f0 msgr2=0x7f158806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:27.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15880682f0 0x7f158806a7a0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f158c007fd0 tx=0x7f158c009040 comp rx=0 tx=0).stop 2023-03-03T13:04:27.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 -- 172.21.15.7:0/2473920773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c0aac90 msgr2=0x7f159c0c8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:04:27.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c0aac90 0x7f159c0c8670 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f1590009c40 tx=0x7f1590004750 comp rx=0 tx=0).stop 2023-03-03T13:04:27.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 -- 172.21.15.7:0/2473920773 shutdown_connections 2023-03-03T13:04:27.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f159c157b40 0x7f159c0c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:27.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f159c06b9b0 0x7f159c0bec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:27.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15880682f0 0x7f158806a7a0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:27.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 --2- 172.21.15.7:0/2473920773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c0aac90 0x7f159c0c8670 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:04:27.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 -- 172.21.15.7:0/2473920773 >> 172.21.15.7:0/2473920773 conn(0x7f159c0736f0 msgr2=0x7f159c151650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:04:27.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 -- 172.21.15.7:0/2473920773 shutdown_connections 2023-03-03T13:04:27.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:04:27.098+0000 7f15a15a2700 1 -- 172.21.15.7:0/2473920773 wait complete. 2023-03-03T13:04:27.102 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:04:27.112 INFO:tasks.mds_thrash.fs.[b]:waiting for 71 secs before thrashing 2023-03-03T13:04:37.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:04:37.226 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:05.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:05:05.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.158+0000 7fbd59304700 1 Processor -- start 2023-03-03T13:05:05.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.158+0000 7fbd59304700 1 -- start start 2023-03-03T13:05:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.159+0000 7fbd59304700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbd540c1da0 0x7fbd54156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.159+0000 7fbd59304700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbd540740f0 0x7fbd5415cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:05.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.159+0000 7fbd59304700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd54155840 con 0x7fbd5414fb00 2023-03-03T13:05:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.159+0000 7fbd59304700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd54072940 con 0x7fbd540c1da0 2023-03-03T13:05:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.159+0000 7fbd59304700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd541567d0 con 0x7fbd540740f0 2023-03-03T13:05:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.159+0000 7fbd527fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbd540c1da0 0x7fbd54156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.159+0000 7fbd52ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbd5414fb00 0x7fbd540c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37062/0 (socket says 172.21.15.7:37062) 2023-03-03T13:05:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.159+0000 7fbd52ffd700 1 -- 172.21.15.7:0/232203041 learned_addr learned my addr 172.21.15.7:0/232203041 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:05:05.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.159+0000 7fbd537fe700 1 --2- 172.21.15.7:0/232203041 >> v2:172.21.15.143:3300/0 conn(0x7fbd540740f0 0x7fbd5415cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd527fc700 1 -- 172.21.15.7:0/232203041 >> v2:172.21.15.143:3300/0 conn(0x7fbd540740f0 msgr2=0x7fbd5415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:05:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd527fc700 1 --2- 172.21.15.7:0/232203041 >> v2:172.21.15.143:3300/0 conn(0x7fbd540740f0 0x7fbd5415cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd527fc700 1 -- 172.21.15.7:0/232203041 >> v1:172.21.15.7:6789/0 conn(0x7fbd5414fb00 legacy=0x7fbd540c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd527fc700 1 -- 172.21.15.7:0/232203041 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd540c2330 con 0x7fbd540c1da0 2023-03-03T13:05:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd51ffb700 1 -- 172.21.15.7:0/232203041 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860477732 0 0) 0x7fbd54155840 con 0x7fbd5414fb00 2023-03-03T13:05:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd527fc700 1 --2- 172.21.15.7:0/232203041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c1da0 0x7fbd54156290 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fbd5406d0b0 tx=0x7fbd48009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbdf9e4efcc2402 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:05.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd51ffb700 1 -- 172.21.15.7:0/232203041 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd48011830 con 0x7fbd540c1da0 2023-03-03T13:05:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd51ffb700 1 -- 172.21.15.7:0/232203041 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd48011990 con 0x7fbd540c1da0 2023-03-03T13:05:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd51ffb700 1 -- 172.21.15.7:0/232203041 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd48011d20 con 0x7fbd540c1da0 2023-03-03T13:05:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd59304700 1 -- 172.21.15.7:0/232203041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c1da0 msgr2=0x7fbd54156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.160+0000 7fbd59304700 1 --2- 172.21.15.7:0/232203041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c1da0 0x7fbd54156290 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fbd5406d0b0 tx=0x7fbd48009b20 comp rx=0 tx=0).stop 2023-03-03T13:05:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.161+0000 7fbd59304700 1 -- 172.21.15.7:0/232203041 shutdown_connections 2023-03-03T13:05:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.161+0000 7fbd59304700 1 --2- 172.21.15.7:0/232203041 >> v2:172.21.15.143:3300/0 conn(0x7fbd540740f0 0x7fbd5415cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:05.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.161+0000 7fbd59304700 1 --2- 172.21.15.7:0/232203041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c1da0 0x7fbd54156290 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.161+0000 7fbd59304700 1 -- 172.21.15.7:0/232203041 >> 172.21.15.7:0/232203041 conn(0x7fbd5406c110 msgr2=0x7fbd5406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:05:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.161+0000 7fbd59304700 1 -- 172.21.15.7:0/232203041 shutdown_connections 2023-03-03T13:05:05.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.161+0000 7fbd59304700 1 -- 172.21.15.7:0/232203041 wait complete. 2023-03-03T13:05:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.162+0000 7fbd59304700 1 Processor -- start 2023-03-03T13:05:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.162+0000 7fbd59304700 1 -- start start 2023-03-03T13:05:05.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd59304700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540740f0 0x7fbd541eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd59304700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd540c1da0 0x7fbd5414aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd59304700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5414fb00 0x7fbd54153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd59304700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd541de990 con 0x7fbd540740f0 2023-03-03T13:05:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd59304700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5414af40 con 0x7fbd540c1da0 2023-03-03T13:05:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd59304700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540c7c10 con 0x7fbd5414fb00 2023-03-03T13:05:05.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd52ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540740f0 0x7fbd541eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd52ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540740f0 0x7fbd541eaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43088/0 (socket says 172.21.15.7:43088) 2023-03-03T13:05:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd52ffd700 1 -- 172.21.15.7:0/2262198563 learned_addr learned my addr 172.21.15.7:0/2262198563 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:05:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd527fc700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd540c1da0 0x7fbd5414aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd537fe700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5414fb00 0x7fbd54153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd52ffd700 1 -- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5414fb00 msgr2=0x7fbd54153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd52ffd700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5414fb00 0x7fbd54153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:05.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd52ffd700 1 -- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd540c1da0 msgr2=0x7fbd5414aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd52ffd700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd540c1da0 0x7fbd5414aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd52ffd700 1 -- 172.21.15.7:0/2262198563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd480094a0 con 0x7fbd540740f0 2023-03-03T13:05:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd52ffd700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540740f0 0x7fbd541eaf80 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c001f80 tx=0x7fbd4c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.163+0000 7fbd3b7fe700 1 -- 172.21.15.7:0/2262198563 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c00acd0 con 0x7fbd540740f0 2023-03-03T13:05:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.164+0000 7fbd3b7fe700 1 -- 172.21.15.7:0/2262198563 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c00ae50 con 0x7fbd540740f0 2023-03-03T13:05:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.164+0000 7fbd59304700 1 -- 172.21.15.7:0/2262198563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd54153090 con 0x7fbd540740f0 2023-03-03T13:05:05.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.164+0000 7fbd3b7fe700 1 -- 172.21.15.7:0/2262198563 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c007310 con 0x7fbd540740f0 2023-03-03T13:05:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.164+0000 7fbd59304700 1 -- 172.21.15.7:0/2262198563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd54152db0 con 0x7fbd540740f0 2023-03-03T13:05:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.165+0000 7fbd59304700 1 -- 172.21.15.7:0/2262198563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd54155a90 con 0x7fbd540740f0 2023-03-03T13:05:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.165+0000 7fbd3b7fe700 1 -- 172.21.15.7:0/2262198563 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd4c01a030 con 0x7fbd540740f0 2023-03-03T13:05:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.166+0000 7fbd3b7fe700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.166+0000 7fbd3b7fe700 1 -- 172.21.15.7:0/2262198563 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbd4c085e30 con 0x7fbd540740f0 2023-03-03T13:05:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.166+0000 7fbd527fc700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:05.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.166+0000 7fbd527fc700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fbd480090b0 tx=0x7fbd4801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:05.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.170+0000 7fbd3b7fe700 1 -- 172.21.15.7:0/2262198563 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbd4c0521d0 con 0x7fbd540740f0 2023-03-03T13:05:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.409+0000 7fbd59304700 1 -- 172.21.15.7:0/2262198563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd5406e410 con 0x7fbd540740f0 2023-03-03T13:05:05.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.409+0000 7fbd3b7fe700 1 -- 172.21.15.7:0/2262198563 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbd4c018090 con 0x7fbd540740f0 2023-03-03T13:05:05.411 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:05:05.411 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:05:05.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 -- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 msgr2=0x7fbd4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:05.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fbd480090b0 tx=0x7fbd4801c040 comp rx=0 tx=0).stop 2023-03-03T13:05:05.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 -- 172.21.15.7:0/2262198563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540740f0 msgr2=0x7fbd541eaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:05.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540740f0 0x7fbd541eaf80 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c001f80 tx=0x7fbd4c00a340 comp rx=0 tx=0).stop 2023-03-03T13:05:05.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 -- 172.21.15.7:0/2262198563 shutdown_connections 2023-03-03T13:05:05.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5414fb00 0x7fbd54153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:05.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:05.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540740f0 0x7fbd541eaf80 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:05.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 --2- 172.21.15.7:0/2262198563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd540c1da0 0x7fbd5414aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:05.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 -- 172.21.15.7:0/2262198563 >> 172.21.15.7:0/2262198563 conn(0x7fbd5406c110 msgr2=0x7fbd540ab770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:05:05.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 -- 172.21.15.7:0/2262198563 shutdown_connections 2023-03-03T13:05:05.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:05.411+0000 7fbd59304700 1 -- 172.21.15.7:0/2262198563 wait complete. 2023-03-03T13:05:05.415 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:05:05.425 INFO:tasks.mds_thrash.fs.[a]:waiting for 70 secs before thrashing 2023-03-03T13:05:07.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:07.289 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:37.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:37.378 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:05:38.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:05:38.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.250+0000 7f3d73860700 1 Processor -- start 2023-03-03T13:05:38.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.250+0000 7f3d73860700 1 -- start start 2023-03-03T13:05:38.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d73860700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3d6c155410 0x7f3d6c0bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:38.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d73860700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d6c0c5b00 0x7f3d6c0bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:38.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d73860700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c0c08d0 con 0x7f3d6c151e80 2023-03-03T13:05:38.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d73860700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6c152250 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d73860700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6c0c0fb0 con 0x7f3d6c155410 2023-03-03T13:05:38.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d71dfd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d6c0c5b00 0x7f3d6c0bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:38.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d71dfd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d6c0c5b00 0x7f3d6c0bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43112/0 (socket says 172.21.15.7:43112) 2023-03-03T13:05:38.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d71dfd700 1 -- 172.21.15.7:0/3620653599 learned_addr learned my addr 172.21.15.7:0/3620653599 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:05:38.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d71dfd700 1 -- 172.21.15.7:0/3620653599 >> v2:172.21.15.143:3300/0 conn(0x7f3d6c155410 msgr2=0x7f3d6c0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:05:38.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d70dfb700 1 --2- 172.21.15.7:0/3620653599 >> v2:172.21.15.143:3300/0 conn(0x7f3d6c155410 0x7f3d6c0bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:38.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d71dfd700 1 --2- 172.21.15.7:0/3620653599 >> v2:172.21.15.143:3300/0 conn(0x7f3d6c155410 0x7f3d6c0bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:38.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d71dfd700 1 -- 172.21.15.7:0/3620653599 >> v1:172.21.15.143:6789/0 conn(0x7f3d6c151e80 legacy=0x7f3d6c0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:05:38.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d71dfd700 1 -- 172.21.15.7:0/3620653599 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c154930 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d70dfb700 1 --2- 172.21.15.7:0/3620653599 >> v2:172.21.15.143:3300/0 conn(0x7f3d6c155410 0x7f3d6c0bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:05:38.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.251+0000 7f3d71dfd700 1 --2- 172.21.15.7:0/3620653599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 0x7f3d6c0bfe20 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c06c380 tx=0x7f3d6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33a81cb1f571a4d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:38.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d63fff700 1 -- 172.21.15.7:0/3620653599 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6400e070 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d63fff700 1 -- 172.21.15.7:0/3620653599 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d64003a20 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d63fff700 1 -- 172.21.15.7:0/3620653599 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64003dd0 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d73860700 1 -- 172.21.15.7:0/3620653599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 msgr2=0x7f3d6c0bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:38.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d73860700 1 --2- 172.21.15.7:0/3620653599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 0x7f3d6c0bfe20 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c06c380 tx=0x7f3d6400b330 comp rx=0 tx=0).stop 2023-03-03T13:05:38.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d73860700 1 -- 172.21.15.7:0/3620653599 shutdown_connections 2023-03-03T13:05:38.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d73860700 1 --2- 172.21.15.7:0/3620653599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 0x7f3d6c0bfe20 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:38.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d73860700 1 --2- 172.21.15.7:0/3620653599 >> v2:172.21.15.143:3300/0 conn(0x7f3d6c155410 0x7f3d6c0bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:38.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d73860700 1 -- 172.21.15.7:0/3620653599 >> 172.21.15.7:0/3620653599 conn(0x7f3d6c071630 msgr2=0x7f3d6c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:05:38.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d73860700 1 -- 172.21.15.7:0/3620653599 shutdown_connections 2023-03-03T13:05:38.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.252+0000 7f3d73860700 1 -- 172.21.15.7:0/3620653599 wait complete. 2023-03-03T13:05:38.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.253+0000 7f3d73860700 1 Processor -- start 2023-03-03T13:05:38.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.253+0000 7f3d73860700 1 -- start start 2023-03-03T13:05:38.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d73860700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 0x7f3d6c0c3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:38.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d73860700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d6c151e80 0x7f3d6c1ed740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:38.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d73860700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d6c155410 0x7f3d6c1edc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:38.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d73860700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c157fe0 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d73860700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c155fb0 con 0x7f3d6c151e80 2023-03-03T13:05:38.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d73860700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c1576b0 con 0x7f3d6c155410 2023-03-03T13:05:38.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d715fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 0x7f3d6c0c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:38.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d715fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 0x7f3d6c0c3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43114/0 (socket says 172.21.15.7:43114) 2023-03-03T13:05:38.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d715fc700 1 -- 172.21.15.7:0/2591111574 learned_addr learned my addr 172.21.15.7:0/2591111574 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:05:38.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d70dfb700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d6c151e80 0x7f3d6c1ed740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:38.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.254+0000 7f3d71dfd700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d6c155410 0x7f3d6c1edc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:38.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d715fc700 1 -- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d6c155410 msgr2=0x7f3d6c1edc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:38.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d715fc700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d6c155410 0x7f3d6c1edc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:38.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d715fc700 1 -- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d6c151e80 msgr2=0x7f3d6c1ed740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:38.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d715fc700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d6c151e80 0x7f3d6c1ed740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:38.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d715fc700 1 -- 172.21.15.7:0/2591111574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d64009b30 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d715fc700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 0x7f3d6c0c3580 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c00c4d0 tx=0x7f3d5c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:38.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d61ffb700 1 -- 172.21.15.7:0/2591111574 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c011790 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d61ffb700 1 -- 172.21.15.7:0/2591111574 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5c0118f0 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d73860700 1 -- 172.21.15.7:0/2591111574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6c154ef0 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d61ffb700 1 -- 172.21.15.7:0/2591111574 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c011ca0 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.255+0000 7f3d73860700 1 -- 172.21.15.7:0/2591111574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6c154c10 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.256+0000 7f3d73860700 1 -- 172.21.15.7:0/2591111574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6c0c5220 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.257+0000 7f3d61ffb700 1 -- 172.21.15.7:0/2591111574 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d5c010050 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.257+0000 7f3d61ffb700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d500682f0 0x7f3d5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:38.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.257+0000 7f3d61ffb700 1 -- 172.21.15.7:0/2591111574 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3d5c085d30 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.257+0000 7f3d70dfb700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d500682f0 0x7f3d5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.258+0000 7f3d70dfb700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d500682f0 0x7f3d5006a7a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0c9060 tx=0x7f3d68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:38.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.261+0000 7f3d61ffb700 1 -- 172.21.15.7:0/2591111574 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d5c0520d0 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.501+0000 7f3d73860700 1 -- 172.21.15.7:0/2591111574 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d6c074bb0 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.502+0000 7f3d61ffb700 1 -- 172.21.15.7:0/2591111574 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3d5c0556f0 con 0x7f3d6c0c5b00 2023-03-03T13:05:38.504 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:05:38.504 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:05:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 -- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d500682f0 msgr2=0x7f3d5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d500682f0 0x7f3d5006a7a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0c9060 tx=0x7f3d68008040 comp rx=0 tx=0).stop 2023-03-03T13:05:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 -- 172.21.15.7:0/2591111574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 msgr2=0x7f3d6c0c3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 0x7f3d6c0c3580 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c00c4d0 tx=0x7f3d5c00c7e0 comp rx=0 tx=0).stop 2023-03-03T13:05:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 -- 172.21.15.7:0/2591111574 shutdown_connections 2023-03-03T13:05:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d6c155410 0x7f3d6c1edc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d500682f0 0x7f3d5006a7a0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d6c0c5b00 0x7f3d6c0c3580 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 --2- 172.21.15.7:0/2591111574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d6c151e80 0x7f3d6c1ed740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 -- 172.21.15.7:0/2591111574 >> 172.21.15.7:0/2591111574 conn(0x7f3d6c071630 msgr2=0x7f3d6c1525b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:05:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 -- 172.21.15.7:0/2591111574 shutdown_connections 2023-03-03T13:05:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:38.504+0000 7f3d73860700 1 -- 172.21.15.7:0/2591111574 wait complete. 2023-03-03T13:05:38.508 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:05:38.518 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T13:05:49.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:05:49.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.656+0000 7f7890a6b700 1 Processor -- start 2023-03-03T13:05:49.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.656+0000 7f7890a6b700 1 -- start start 2023-03-03T13:05:49.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7890a6b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f788c072250 0x7f788c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:49.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7890a6b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f788c0c1ce0 0x7f788c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:49.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7890a6b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c1510d0 con 0x7f788c071e80 2023-03-03T13:05:49.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7890a6b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c0c3740 con 0x7f788c072250 2023-03-03T13:05:49.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7890a6b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c070670 con 0x7f788c0c1ce0 2023-03-03T13:05:49.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7889d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f788c072250 0x7f788c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:49.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7889d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f788c072250 0x7f788c14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43134/0 (socket says 172.21.15.7:43134) 2023-03-03T13:05:49.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7889d9b700 1 -- 172.21.15.7:0/2333018416 learned_addr learned my addr 172.21.15.7:0/2333018416 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:05:49.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f788ad9d700 1 --2- 172.21.15.7:0/2333018416 >> v2:172.21.15.143:3300/0 conn(0x7f788c0c1ce0 0x7f788c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:49.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7889d9b700 1 -- 172.21.15.7:0/2333018416 >> v2:172.21.15.143:3300/0 conn(0x7f788c0c1ce0 msgr2=0x7f788c14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:05:49.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7889d9b700 1 --2- 172.21.15.7:0/2333018416 >> v2:172.21.15.143:3300/0 conn(0x7f788c0c1ce0 0x7f788c14e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:49.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7889d9b700 1 -- 172.21.15.7:0/2333018416 >> v1:172.21.15.143:6789/0 conn(0x7f788c071e80 legacy=0x7f788c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:49.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7889d9b700 1 -- 172.21.15.7:0/2333018416 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c0c39e0 con 0x7f788c072250 2023-03-03T13:05:49.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f7889d9b700 1 --2- 172.21.15.7:0/2333018416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 0x7f788c14e1f0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f788c071090 tx=0x7f7880009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ff46dd3f0e3a14 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:49.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.657+0000 7f788959a700 1 -- 172.21.15.7:0/2333018416 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880011830 con 0x7f788c072250 2023-03-03T13:05:49.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.658+0000 7f788959a700 1 -- 172.21.15.7:0/2333018416 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7880011990 con 0x7f788c072250 2023-03-03T13:05:49.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.658+0000 7f788959a700 1 -- 172.21.15.7:0/2333018416 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880011d20 con 0x7f788c072250 2023-03-03T13:05:49.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.658+0000 7f7890a6b700 1 -- 172.21.15.7:0/2333018416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 msgr2=0x7f788c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:49.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.658+0000 7f7890a6b700 1 --2- 172.21.15.7:0/2333018416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 0x7f788c14e1f0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f788c071090 tx=0x7f7880009bd0 comp rx=0 tx=0).stop 2023-03-03T13:05:49.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.658+0000 7f7890a6b700 1 -- 172.21.15.7:0/2333018416 shutdown_connections 2023-03-03T13:05:49.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.658+0000 7f7890a6b700 1 --2- 172.21.15.7:0/2333018416 >> v2:172.21.15.143:3300/0 conn(0x7f788c0c1ce0 0x7f788c14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:49.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.658+0000 7f7890a6b700 1 --2- 172.21.15.7:0/2333018416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 0x7f788c14e1f0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:49.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.658+0000 7f7890a6b700 1 -- 172.21.15.7:0/2333018416 >> 172.21.15.7:0/2333018416 conn(0x7f788c0a94d0 msgr2=0x7f788c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:05:49.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.658+0000 7f7890a6b700 1 -- 172.21.15.7:0/2333018416 shutdown_connections 2023-03-03T13:05:49.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.658+0000 7f7890a6b700 1 -- 172.21.15.7:0/2333018416 wait complete. 2023-03-03T13:05:49.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.659+0000 7f7890a6b700 1 Processor -- start 2023-03-03T13:05:49.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.659+0000 7f7890a6b700 1 -- start start 2023-03-03T13:05:49.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f7890a6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f788c071e80 0x7f788c14d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:49.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f7890a6b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 0x7f788c1459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:49.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f7890a6b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c0c1ce0 0x7f788c145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:49.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f7890a6b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c144b20 con 0x7f788c072250 2023-03-03T13:05:49.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f7890a6b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c1dd340 con 0x7f788c0c1ce0 2023-03-03T13:05:49.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f7890a6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0c5160 con 0x7f788c071e80 2023-03-03T13:05:49.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f7889d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 0x7f788c1459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:49.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f7889d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 0x7f788c1459a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43140/0 (socket says 172.21.15.7:43140) 2023-03-03T13:05:49.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f7889d9b700 1 -- 172.21.15.7:0/1758662209 learned_addr learned my addr 172.21.15.7:0/1758662209 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:05:49.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f788a59c700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f788c071e80 0x7f788c14d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:49.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.660+0000 7f788ad9d700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c0c1ce0 0x7f788c145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:49.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f7889d9b700 1 -- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f788c071e80 msgr2=0x7f788c14d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:49.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f7889d9b700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f788c071e80 0x7f788c14d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:49.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f7889d9b700 1 -- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c0c1ce0 msgr2=0x7f788c145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:49.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f7889d9b700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c0c1ce0 0x7f788c145ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:49.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f7889d9b700 1 -- 172.21.15.7:0/1758662209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7880009550 con 0x7f788c072250 2023-03-03T13:05:49.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f7889d9b700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 0x7f788c1459a0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f7880000c00 tx=0x7f7880004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:49.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f787affd700 1 -- 172.21.15.7:0/1758662209 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880004590 con 0x7f788c072250 2023-03-03T13:05:49.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f7890a6b700 1 -- 172.21.15.7:0/1758662209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788c151f30 con 0x7f788c072250 2023-03-03T13:05:49.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f787affd700 1 -- 172.21.15.7:0/1758662209 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78800036a0 con 0x7f788c072250 2023-03-03T13:05:49.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f787affd700 1 -- 172.21.15.7:0/1758662209 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880003a50 con 0x7f788c072250 2023-03-03T13:05:49.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.661+0000 7f7890a6b700 1 -- 172.21.15.7:0/1758662209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788c151300 con 0x7f788c072250 2023-03-03T13:05:49.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.662+0000 7f7890a6b700 1 -- 172.21.15.7:0/1758662209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788c06c9a0 con 0x7f788c072250 2023-03-03T13:05:49.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.662+0000 7f787affd700 1 -- 172.21.15.7:0/1758662209 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f788000f050 con 0x7f788c072250 2023-03-03T13:05:49.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.663+0000 7f787affd700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f786c0682f0 0x7f786c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:05:49.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.663+0000 7f787affd700 1 -- 172.21.15.7:0/1758662209 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f78800877c0 con 0x7f788c072250 2023-03-03T13:05:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.663+0000 7f788a59c700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f786c0682f0 0x7f786c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:05:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.664+0000 7f788a59c700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f786c0682f0 0x7f786c06a7a0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f787c007fd0 tx=0x7f787c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:05:49.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.667+0000 7f787affd700 1 -- 172.21.15.7:0/1758662209 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7880053ae0 con 0x7f788c072250 2023-03-03T13:05:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.906+0000 7f7890a6b700 1 -- 172.21.15.7:0/1758662209 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f788c074bb0 con 0x7f788c072250 2023-03-03T13:05:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.906+0000 7f787affd700 1 -- 172.21.15.7:0/1758662209 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7880057100 con 0x7f788c072250 2023-03-03T13:05:49.909 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:05:49.909 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:05:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.908+0000 7f7890a6b700 1 -- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f786c0682f0 msgr2=0x7f786c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.908+0000 7f7890a6b700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f786c0682f0 0x7f786c06a7a0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f787c007fd0 tx=0x7f787c009040 comp rx=0 tx=0).stop 2023-03-03T13:05:49.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.908+0000 7f7890a6b700 1 -- 172.21.15.7:0/1758662209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 msgr2=0x7f788c1459a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:05:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.908+0000 7f7890a6b700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 0x7f788c1459a0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f7880000c00 tx=0x7f7880004750 comp rx=0 tx=0).stop 2023-03-03T13:05:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.908+0000 7f7890a6b700 1 -- 172.21.15.7:0/1758662209 shutdown_connections 2023-03-03T13:05:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.908+0000 7f7890a6b700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c0c1ce0 0x7f788c145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.908+0000 7f7890a6b700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f788c071e80 0x7f788c14d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.908+0000 7f7890a6b700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f786c0682f0 0x7f786c06a7a0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.908+0000 7f7890a6b700 1 --2- 172.21.15.7:0/1758662209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c072250 0x7f788c1459a0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:05:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.908+0000 7f7890a6b700 1 -- 172.21.15.7:0/1758662209 >> 172.21.15.7:0/1758662209 conn(0x7f788c0a94d0 msgr2=0x7f788c06a850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:05:49.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.909+0000 7f7890a6b700 1 -- 172.21.15.7:0/1758662209 shutdown_connections 2023-03-03T13:05:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:05:49.909+0000 7f7890a6b700 1 -- 172.21.15.7:0/1758662209 wait complete. 2023-03-03T13:05:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:05:49.923 INFO:tasks.mds_thrash.fs.[b]:waiting for 57 secs before thrashing 2023-03-03T13:06:07.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:07.476 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:15.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:06:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.563+0000 7f7a243d4700 1 Processor -- start 2023-03-03T13:06:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.563+0000 7f7a243d4700 1 -- start start 2023-03-03T13:06:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.564+0000 7f7a243d4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7a1c14b0d0 0x7f7a1c071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.564+0000 7f7a243d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 0x7f7a1c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.564+0000 7f7a243d4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1c0a9c20 con 0x7f7a1c071d40 2023-03-03T13:06:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.564+0000 7f7a243d4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1c154f10 con 0x7f7a1c14b0d0 2023-03-03T13:06:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a243d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0ac3c0 con 0x7f7a1c0aad20 2023-03-03T13:06:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a22170700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7a1c14b0d0 0x7f7a1c071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a22971700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7a1c071d40 0x7f7a1c158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37148/0 (socket says 172.21.15.7:37148) 2023-03-03T13:06:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a22971700 1 -- 172.21.15.7:0/1429843953 learned_addr learned my addr 172.21.15.7:0/1429843953 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:06:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a2196f700 1 --2- 172.21.15.7:0/1429843953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 0x7f7a1c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a22170700 1 -- 172.21.15.7:0/1429843953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 msgr2=0x7f7a1c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:06:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a22170700 1 --2- 172.21.15.7:0/1429843953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 0x7f7a1c157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a22170700 1 -- 172.21.15.7:0/1429843953 >> v1:172.21.15.7:6789/0 conn(0x7f7a1c071d40 legacy=0x7f7a1c158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a2116e700 1 -- 172.21.15.7:0/1429843953 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905347964 0 0) 0x7f7a1c0a9c20 con 0x7f7a1c071d40 2023-03-03T13:06:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a22170700 1 -- 172.21.15.7:0/1429843953 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c06fc70 con 0x7f7a1c14b0d0 2023-03-03T13:06:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a2196f700 1 --2- 172.21.15.7:0/1429843953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 0x7f7a1c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:06:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a22170700 1 --2- 172.21.15.7:0/1429843953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c14b0d0 0x7f7a1c071800 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009840 tx=0x7f7a18009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bddd98fd475ac1b9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:06:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a2116e700 1 -- 172.21.15.7:0/1429843953 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18011830 con 0x7f7a1c14b0d0 2023-03-03T13:06:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a2116e700 1 -- 172.21.15.7:0/1429843953 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a18011990 con 0x7f7a1c14b0d0 2023-03-03T13:06:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.565+0000 7f7a2116e700 1 -- 172.21.15.7:0/1429843953 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18011d20 con 0x7f7a1c14b0d0 2023-03-03T13:06:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.566+0000 7f7a243d4700 1 -- 172.21.15.7:0/1429843953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c14b0d0 msgr2=0x7f7a1c071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.566+0000 7f7a243d4700 1 --2- 172.21.15.7:0/1429843953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c14b0d0 0x7f7a1c071800 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009840 tx=0x7f7a18009b50 comp rx=0 tx=0).stop 2023-03-03T13:06:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.566+0000 7f7a243d4700 1 -- 172.21.15.7:0/1429843953 shutdown_connections 2023-03-03T13:06:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.566+0000 7f7a243d4700 1 --2- 172.21.15.7:0/1429843953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c14b0d0 0x7f7a1c071800 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.566+0000 7f7a243d4700 1 --2- 172.21.15.7:0/1429843953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 0x7f7a1c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.566+0000 7f7a243d4700 1 -- 172.21.15.7:0/1429843953 >> 172.21.15.7:0/1429843953 conn(0x7f7a1c0c8900 msgr2=0x7f7a1c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:06:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.566+0000 7f7a243d4700 1 -- 172.21.15.7:0/1429843953 shutdown_connections 2023-03-03T13:06:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.566+0000 7f7a243d4700 1 -- 172.21.15.7:0/1429843953 wait complete. 2023-03-03T13:06:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.567+0000 7f7a243d4700 1 Processor -- start 2023-03-03T13:06:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.567+0000 7f7a243d4700 1 -- start start 2023-03-03T13:06:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a243d4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c071d40 0x7f7a1c0c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a243d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 0x7f7a1c1ea3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a243d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a1c14b0d0 0x7f7a1c1ed900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a243d4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c154500 con 0x7f7a1c071d40 2023-03-03T13:06:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a243d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c1e9910 con 0x7f7a1c14b0d0 2023-03-03T13:06:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a243d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0c1ea0 con 0x7f7a1c0aad20 2023-03-03T13:06:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22170700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c071d40 0x7f7a1c0c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22170700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c071d40 0x7f7a1c0c17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43170/0 (socket says 172.21.15.7:43170) 2023-03-03T13:06:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22170700 1 -- 172.21.15.7:0/1134663749 learned_addr learned my addr 172.21.15.7:0/1134663749 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:06:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a2196f700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 0x7f7a1c1ea3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22971700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a1c14b0d0 0x7f7a1c1ed900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22170700 1 -- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 msgr2=0x7f7a1c1ea3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22170700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 0x7f7a1c1ea3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22170700 1 -- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a1c14b0d0 msgr2=0x7f7a1c1ed900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22170700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a1c14b0d0 0x7f7a1c1ed900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22170700 1 -- 172.21.15.7:0/1134663749 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a180094a0 con 0x7f7a1c071d40 2023-03-03T13:06:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22971700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a1c14b0d0 0x7f7a1c1ed900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:06:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.568+0000 7f7a22170700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c071d40 0x7f7a1c0c17e0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f7a18000c00 tx=0x7f7a18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:06:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.569+0000 7f7a0effd700 1 -- 172.21.15.7:0/1134663749 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180036a0 con 0x7f7a1c071d40 2023-03-03T13:06:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.569+0000 7f7a0effd700 1 -- 172.21.15.7:0/1134663749 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1801f070 con 0x7f7a1c071d40 2023-03-03T13:06:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.569+0000 7f7a243d4700 1 -- 172.21.15.7:0/1134663749 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1c06f6b0 con 0x7f7a1c071d40 2023-03-03T13:06:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.569+0000 7f7a0effd700 1 -- 172.21.15.7:0/1134663749 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18003a60 con 0x7f7a1c071d40 2023-03-03T13:06:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.569+0000 7f7a243d4700 1 -- 172.21.15.7:0/1134663749 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1c155160 con 0x7f7a1c071d40 2023-03-03T13:06:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.570+0000 7f7a243d4700 1 -- 172.21.15.7:0/1134663749 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1c06f990 con 0x7f7a1c071d40 2023-03-03T13:06:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.570+0000 7f7a0effd700 1 -- 172.21.15.7:0/1134663749 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7a1800f050 con 0x7f7a1c071d40 2023-03-03T13:06:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.571+0000 7f7a0effd700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a000682f0 0x7f7a0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.571+0000 7f7a0effd700 1 -- 172.21.15.7:0/1134663749 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7a18021070 con 0x7f7a1c071d40 2023-03-03T13:06:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.571+0000 7f7a2196f700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a000682f0 0x7f7a0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.571+0000 7f7a2196f700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a000682f0 0x7f7a0006a7a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0c7e00 tx=0x7f7a10008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:06:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.575+0000 7f7a0effd700 1 -- 172.21.15.7:0/1134663749 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7a18053f10 con 0x7f7a1c071d40 2023-03-03T13:06:15.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.811+0000 7f7a243d4700 1 -- 172.21.15.7:0/1134663749 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a1c074bb0 con 0x7f7a1c071d40 2023-03-03T13:06:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.811+0000 7f7a0effd700 1 -- 172.21.15.7:0/1134663749 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7a18057530 con 0x7f7a1c071d40 2023-03-03T13:06:15.813 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:06:15.813 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:06:15.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 -- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a000682f0 msgr2=0x7f7a0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:15.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a000682f0 0x7f7a0006a7a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0c7e00 tx=0x7f7a10008040 comp rx=0 tx=0).stop 2023-03-03T13:06:15.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 -- 172.21.15.7:0/1134663749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c071d40 msgr2=0x7f7a1c0c17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:15.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c071d40 0x7f7a1c0c17e0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f7a18000c00 tx=0x7f7a18004750 comp rx=0 tx=0).stop 2023-03-03T13:06:15.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 -- 172.21.15.7:0/1134663749 shutdown_connections 2023-03-03T13:06:15.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a1c14b0d0 0x7f7a1c1ed900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:15.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a000682f0 0x7f7a0006a7a0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:15.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a1c071d40 0x7f7a1c0c17e0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:15.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 --2- 172.21.15.7:0/1134663749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a1c0aad20 0x7f7a1c1ea3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:15.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 -- 172.21.15.7:0/1134663749 >> 172.21.15.7:0/1134663749 conn(0x7f7a1c0c8900 msgr2=0x7f7a1c14a390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:06:15.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 -- 172.21.15.7:0/1134663749 shutdown_connections 2023-03-03T13:06:15.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:15.813+0000 7f7a243d4700 1 -- 172.21.15.7:0/1134663749 wait complete. 2023-03-03T13:06:15.817 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:06:15.827 INFO:tasks.mds_thrash.fs.[a]:waiting for 99 secs before thrashing 2023-03-03T13:06:37.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:37.556 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:06:46.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:06:47.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.060+0000 7fa6ad220700 1 Processor -- start 2023-03-03T13:06:47.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.060+0000 7fa6ad220700 1 -- start start 2023-03-03T13:06:47.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.061+0000 7fa6ad220700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa6a8151d40 0x7fa6a80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:47.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.061+0000 7fa6ad220700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a806d430 con 0x7fa6a8073c30 2023-03-03T13:06:47.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.061+0000 7fa6ad220700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a806cb00 con 0x7fa6a80c2930 2023-03-03T13:06:47.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.061+0000 7fa6ad220700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a806c2a0 con 0x7fa6a8151d40 2023-03-03T13:06:47.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.061+0000 7fa6a759e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa6a8073c30 0x7fa6a81581a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37182/0 (socket says 172.21.15.7:37182) 2023-03-03T13:06:47.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.061+0000 7fa6a759e700 1 -- 172.21.15.7:0/599829937 learned_addr learned my addr 172.21.15.7:0/599829937 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:06:47.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a6d9d700 1 --2- 172.21.15.7:0/599829937 >> v2:172.21.15.143:3300/0 conn(0x7fa6a8151d40 0x7fa6a80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:47.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231470649 0 0) 0x7fa6a806d430 con 0x7fa6a8073c30 2023-03-03T13:06:47.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6940035b0 con 0x7fa6a8073c30 2023-03-03T13:06:47.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2751814701 0 0) 0x7fa6940035b0 con 0x7fa6a8073c30 2023-03-03T13:06:47.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 >> v2:172.21.15.143:3300/0 conn(0x7fa6a8151d40 msgr2=0x7fa6a80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:06:47.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 --2- 172.21.15.7:0/599829937 >> v2:172.21.15.143:3300/0 conn(0x7fa6a8151d40 0x7fa6a80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 >> v1:172.21.15.143:6789/0 conn(0x7fa6a80c2930 legacy=0x7fa6a8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:47.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a806cde0 con 0x7fa6a8073c30 2023-03-03T13:06:47.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa6a00019f0 con 0x7fa6a8073c30 2023-03-03T13:06:47.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 >> v1:172.21.15.7:6789/0 conn(0x7fa6a8073c30 legacy=0x7fa6a81581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:47.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa694003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:47.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694004850 0x7fa694004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:47.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa694005320 0x7fa69400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:47.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a806c2a0 con 0x7fa694003a70 2023-03-03T13:06:47.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69400c250 con 0x7fa694005320 2023-03-03T13:06:47.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69400c410 con 0x7fa694004850 2023-03-03T13:06:47.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a6d9d700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa694003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:47.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a759e700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa694005320 0x7fa69400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:47.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a659c700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694004850 0x7fa694004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.062+0000 7fa6a6d9d700 1 -- 172.21.15.7:0/599829937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694004850 msgr2=0x7fa694004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:06:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6a6d9d700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694004850 0x7fa694004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6a6d9d700 1 -- 172.21.15.7:0/599829937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa694005320 msgr2=0x7fa69400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:06:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6a6d9d700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa694005320 0x7fa69400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6a759e700 1 -- 172.21.15.7:0/599829937 reap_dead start 2023-03-03T13:06:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6a6d9d700 1 -- 172.21.15.7:0/599829937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6940035b0 con 0x7fa694003a70 2023-03-03T13:06:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6a659c700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694004850 0x7fa694004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:06:47.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6a6d9d700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa694003e80 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fa698009b60 tx=0x7fa694005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70beb61a2e84b58 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:06:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800e6c0 con 0x7fa694003a70 2023-03-03T13:06:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69800e820 con 0x7fa694003a70 2023-03-03T13:06:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6a5d9b700 1 -- 172.21.15.7:0/599829937 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800eb90 con 0x7fa694003a70 2023-03-03T13:06:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6ad220700 1 -- 172.21.15.7:0/599829937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 msgr2=0x7fa694003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6ad220700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa694003e80 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fa698009b60 tx=0x7fa694005880 comp rx=0 tx=0).stop 2023-03-03T13:06:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6ad220700 1 -- 172.21.15.7:0/599829937 shutdown_connections 2023-03-03T13:06:47.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6ad220700 1 --2- 172.21.15.7:0/599829937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa694003e80 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6ad220700 1 -- 172.21.15.7:0/599829937 >> 172.21.15.7:0/599829937 conn(0x7fa6a80a94d0 msgr2=0x7fa6a80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:06:47.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.063+0000 7fa6ad220700 1 -- 172.21.15.7:0/599829937 shutdown_connections 2023-03-03T13:06:47.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.064+0000 7fa6ad220700 1 -- 172.21.15.7:0/599829937 wait complete. 2023-03-03T13:06:47.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.065+0000 7fa6ad220700 1 Processor -- start 2023-03-03T13:06:47.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.065+0000 7fa6ad220700 1 -- start start 2023-03-03T13:06:47.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6ad220700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa6a814c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:47.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6ad220700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6a80bf9f0 0x7fa6a8154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:47.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6ad220700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6a814b5b0 0x7fa6a814ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:47.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6ad220700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a80c01d0 con 0x7fa694003a70 2023-03-03T13:06:47.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6ad220700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a814dad0 con 0x7fa6a80bf9f0 2023-03-03T13:06:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6ad220700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a814ddb0 con 0x7fa6a814b5b0 2023-03-03T13:06:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa6a814c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa6a814c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43210/0 (socket says 172.21.15.7:43210) 2023-03-03T13:06:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a6d9d700 1 -- 172.21.15.7:0/3215755713 learned_addr learned my addr 172.21.15.7:0/3215755713 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:06:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a659c700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6a80bf9f0 0x7fa6a8154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a759e700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6a814b5b0 0x7fa6a814ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:47.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a6d9d700 1 -- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6a814b5b0 msgr2=0x7fa6a814ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a6d9d700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6a814b5b0 0x7fa6a814ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a6d9d700 1 -- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6a80bf9f0 msgr2=0x7fa6a8154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a6d9d700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6a80bf9f0 0x7fa6a8154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a6d9d700 1 -- 172.21.15.7:0/3215755713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a806bea0 con 0x7fa694003a70 2023-03-03T13:06:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a659c700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6a80bf9f0 0x7fa6a8154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:06:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a759e700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6a814b5b0 0x7fa6a814ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:06:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.066+0000 7fa6a6d9d700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa6a814c200 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fa69400c220 tx=0x7fa6980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:06:47.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.067+0000 7fa69f7fe700 1 -- 172.21.15.7:0/3215755713 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800e6c0 con 0x7fa694003a70 2023-03-03T13:06:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.067+0000 7fa69f7fe700 1 -- 172.21.15.7:0/3215755713 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa698003ad0 con 0x7fa694003a70 2023-03-03T13:06:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.067+0000 7fa6ad220700 1 -- 172.21.15.7:0/3215755713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a806d0c0 con 0x7fa694003a70 2023-03-03T13:06:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.067+0000 7fa69f7fe700 1 -- 172.21.15.7:0/3215755713 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa698003e80 con 0x7fa694003a70 2023-03-03T13:06:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.067+0000 7fa6ad220700 1 -- 172.21.15.7:0/3215755713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a806c030 con 0x7fa694003a70 2023-03-03T13:06:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.068+0000 7fa6ad220700 1 -- 172.21.15.7:0/3215755713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a8074bb0 con 0x7fa694003a70 2023-03-03T13:06:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.068+0000 7fa69f7fe700 1 -- 172.21.15.7:0/3215755713 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa698019030 con 0x7fa694003a70 2023-03-03T13:06:47.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.068+0000 7fa69f7fe700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6880682f0 0x7fa68806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:06:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.068+0000 7fa69f7fe700 1 -- 172.21.15.7:0/3215755713 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa698087770 con 0x7fa694003a70 2023-03-03T13:06:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.069+0000 7fa6a659c700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6880682f0 0x7fa68806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:06:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.069+0000 7fa6a659c700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6880682f0 0x7fa68806a7a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fa690000c00 tx=0x7fa690009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:06:47.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.073+0000 7fa69f7fe700 1 -- 172.21.15.7:0/3215755713 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa698053b10 con 0x7fa694003a70 2023-03-03T13:06:47.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.312+0000 7fa6ad220700 1 -- 172.21.15.7:0/3215755713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6a80a9fb0 con 0x7fa694003a70 2023-03-03T13:06:47.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.313+0000 7fa69f7fe700 1 -- 172.21.15.7:0/3215755713 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa698057130 con 0x7fa694003a70 2023-03-03T13:06:47.315 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:06:47.315 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:06:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 -- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6880682f0 msgr2=0x7fa68806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6880682f0 0x7fa68806a7a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fa690000c00 tx=0x7fa690009040 comp rx=0 tx=0).stop 2023-03-03T13:06:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 -- 172.21.15.7:0/3215755713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 msgr2=0x7fa6a814c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:06:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa6a814c200 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fa69400c220 tx=0x7fa6980047c0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 -- 172.21.15.7:0/3215755713 shutdown_connections 2023-03-03T13:06:47.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6a814b5b0 0x7fa6a814ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6880682f0 0x7fa68806a7a0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa694003a70 0x7fa6a814c200 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 --2- 172.21.15.7:0/3215755713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6a80bf9f0 0x7fa6a8154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:06:47.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 -- 172.21.15.7:0/3215755713 >> 172.21.15.7:0/3215755713 conn(0x7fa6a80a94d0 msgr2=0x7fa6a80c34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:06:47.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 -- 172.21.15.7:0/3215755713 shutdown_connections 2023-03-03T13:06:47.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:06:47.315+0000 7fa6ad220700 1 -- 172.21.15.7:0/3215755713 wait complete. 2023-03-03T13:06:47.319 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:06:47.329 INFO:tasks.mds_thrash.fs.[b]:waiting for 89 secs before thrashing 2023-03-03T13:07:07.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:07.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:37.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:37.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:07:54.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:07:54.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.965+0000 7f1c178eb700 1 Processor -- start 2023-03-03T13:07:54.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.965+0000 7f1c178eb700 1 -- start start 2023-03-03T13:07:54.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.966+0000 7f1c178eb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1c10151d40 0x7f1c100736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:54.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.966+0000 7f1c178eb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1006d430 con 0x7f1c10073c30 2023-03-03T13:07:54.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.966+0000 7f1c178eb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1006cb00 con 0x7f1c100c2930 2023-03-03T13:07:54.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.966+0000 7f1c178eb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1006c2a0 con 0x7f1c10151d40 2023-03-03T13:07:54.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.966+0000 7f1c15687700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1c10151d40 0x7f1c100736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:54.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.966+0000 7f1c15e88700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1c10073c30 0x7f1c101581a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37266/0 (socket says 172.21.15.7:37266) 2023-03-03T13:07:54.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.966+0000 7f1c15e88700 1 -- 172.21.15.7:0/3791329540 learned_addr learned my addr 172.21.15.7:0/3791329540 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:07:54.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c15687700 1 -- 172.21.15.7:0/3791329540 >> v1:172.21.15.143:6789/0 conn(0x7f1c100c2930 legacy=0x7f1c10157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:54.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c15687700 1 -- 172.21.15.7:0/3791329540 >> v1:172.21.15.7:6789/0 conn(0x7f1c10073c30 legacy=0x7f1c101581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:54.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c15687700 1 -- 172.21.15.7:0/3791329540 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1006cde0 con 0x7f1c10151d40 2023-03-03T13:07:54.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c07fff700 1 -- 172.21.15.7:0/3791329540 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356411847 0 0) 0x7f1c1006d430 con 0x7f1c10073c30 2023-03-03T13:07:54.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c15687700 1 --2- 172.21.15.7:0/3791329540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10151d40 0x7f1c100736f0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009870 tx=0x7f1c00009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22107d5ac8ea31e6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:07:54.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c07fff700 1 -- 172.21.15.7:0/3791329540 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00011830 con 0x7f1c10151d40 2023-03-03T13:07:54.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c07fff700 1 -- 172.21.15.7:0/3791329540 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c00011990 con 0x7f1c10151d40 2023-03-03T13:07:54.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c07fff700 1 -- 172.21.15.7:0/3791329540 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00011d20 con 0x7f1c10151d40 2023-03-03T13:07:54.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c178eb700 1 -- 172.21.15.7:0/3791329540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10151d40 msgr2=0x7f1c100736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:54.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c178eb700 1 --2- 172.21.15.7:0/3791329540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10151d40 0x7f1c100736f0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009870 tx=0x7f1c00009b80 comp rx=0 tx=0).stop 2023-03-03T13:07:54.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.967+0000 7f1c178eb700 1 -- 172.21.15.7:0/3791329540 shutdown_connections 2023-03-03T13:07:54.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.968+0000 7f1c178eb700 1 --2- 172.21.15.7:0/3791329540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10151d40 0x7f1c100736f0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:54.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.968+0000 7f1c178eb700 1 -- 172.21.15.7:0/3791329540 >> 172.21.15.7:0/3791329540 conn(0x7f1c100a94d0 msgr2=0x7f1c100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:07:54.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.968+0000 7f1c178eb700 1 -- 172.21.15.7:0/3791329540 shutdown_connections 2023-03-03T13:07:54.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.968+0000 7f1c178eb700 1 -- 172.21.15.7:0/3791329540 wait complete. 2023-03-03T13:07:54.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.969+0000 7f1c178eb700 1 Processor -- start 2023-03-03T13:07:54.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.969+0000 7f1c178eb700 1 -- start start 2023-03-03T13:07:54.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.969+0000 7f1c178eb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10073c30 0x7f1c10156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:54.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c178eb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c100c2930 0x7f1c100aa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:54.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c178eb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c10151d40 0x7f1c100aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:54.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c178eb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100ab210 con 0x7f1c10073c30 2023-03-03T13:07:54.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c178eb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100c0b90 con 0x7f1c100c2930 2023-03-03T13:07:54.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c178eb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100c0770 con 0x7f1c10151d40 2023-03-03T13:07:54.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c15687700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10073c30 0x7f1c10156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:54.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c15687700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10073c30 0x7f1c10156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43288/0 (socket says 172.21.15.7:43288) 2023-03-03T13:07:54.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c15687700 1 -- 172.21.15.7:0/2759875734 learned_addr learned my addr 172.21.15.7:0/2759875734 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:07:54.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c14e86700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c100c2930 0x7f1c100aa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:54.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c15e88700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c10151d40 0x7f1c100aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:54.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c15687700 1 -- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c10151d40 msgr2=0x7f1c100aacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:54.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c15687700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c10151d40 0x7f1c100aacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:54.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c15687700 1 -- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c100c2930 msgr2=0x7f1c100aa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:54.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c15687700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c100c2930 0x7f1c100aa790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:54.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c15687700 1 -- 172.21.15.7:0/2759875734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c000094a0 con 0x7f1c10073c30 2023-03-03T13:07:54.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c15687700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10073c30 0x7f1c10156ba0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f1c000090b0 tx=0x7f1c00004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:07:54.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c05ffb700 1 -- 172.21.15.7:0/2759875734 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c000038a0 con 0x7f1c10073c30 2023-03-03T13:07:54.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c05ffb700 1 -- 172.21.15.7:0/2759875734 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0001e070 con 0x7f1c10073c30 2023-03-03T13:07:54.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c178eb700 1 -- 172.21.15.7:0/2759875734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1006bea0 con 0x7f1c10073c30 2023-03-03T13:07:54.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.970+0000 7f1c05ffb700 1 -- 172.21.15.7:0/2759875734 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c00003c60 con 0x7f1c10073c30 2023-03-03T13:07:54.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.971+0000 7f1c178eb700 1 -- 172.21.15.7:0/2759875734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1006d0c0 con 0x7f1c10073c30 2023-03-03T13:07:54.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.971+0000 7f1c178eb700 1 -- 172.21.15.7:0/2759875734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1006c030 con 0x7f1c10073c30 2023-03-03T13:07:54.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.973+0000 7f1c05ffb700 1 -- 172.21.15.7:0/2759875734 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1c0000f050 con 0x7f1c10073c30 2023-03-03T13:07:54.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.973+0000 7f1c05ffb700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bf40682f0 0x7f1bf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:07:54.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.973+0000 7f1c05ffb700 1 -- 172.21.15.7:0/2759875734 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1c00087720 con 0x7f1c10073c30 2023-03-03T13:07:54.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.973+0000 7f1c14e86700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bf40682f0 0x7f1bf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:07:54.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.974+0000 7f1c14e86700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bf40682f0 0x7f1bf406a7a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1c100c3760 tx=0x7f1c0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:07:54.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:54.977+0000 7f1c05ffb700 1 -- 172.21.15.7:0/2759875734 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1c00053b60 con 0x7f1c10073c30 2023-03-03T13:07:55.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.214+0000 7f1c178eb700 1 -- 172.21.15.7:0/2759875734 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c10074bb0 con 0x7f1c10073c30 2023-03-03T13:07:55.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.215+0000 7f1c05ffb700 1 -- 172.21.15.7:0/2759875734 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1c00004570 con 0x7f1c10073c30 2023-03-03T13:07:55.217 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:07:55.218 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:07:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 -- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bf40682f0 msgr2=0x7f1bf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:55.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bf40682f0 0x7f1bf406a7a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1c100c3760 tx=0x7f1c0c009040 comp rx=0 tx=0).stop 2023-03-03T13:07:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 -- 172.21.15.7:0/2759875734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10073c30 msgr2=0x7f1c10156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:07:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10073c30 0x7f1c10156ba0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f1c000090b0 tx=0x7f1c00004750 comp rx=0 tx=0).stop 2023-03-03T13:07:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 -- 172.21.15.7:0/2759875734 shutdown_connections 2023-03-03T13:07:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c10151d40 0x7f1c100aacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1bf40682f0 0x7f1bf406a7a0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c10073c30 0x7f1c10156ba0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:55.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 --2- 172.21.15.7:0/2759875734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c100c2930 0x7f1c100aa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:07:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 -- 172.21.15.7:0/2759875734 >> 172.21.15.7:0/2759875734 conn(0x7f1c100a94d0 msgr2=0x7f1c100c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:07:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 -- 172.21.15.7:0/2759875734 shutdown_connections 2023-03-03T13:07:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:07:55.217+0000 7f1c178eb700 1 -- 172.21.15.7:0/2759875734 wait complete. 2023-03-03T13:07:55.221 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:07:55.231 INFO:tasks.mds_thrash.fs.[a]:waiting for 20 secs before thrashing 2023-03-03T13:08:07.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:07.778 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:15.233 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:08:15.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.369+0000 7fc996d6f700 1 Processor -- start 2023-03-03T13:08:15.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.369+0000 7fc996d6f700 1 -- start start 2023-03-03T13:08:15.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc996d6f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc99014b460 0x7fc99006aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:15.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc996d6f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc990072470 0x7fc99006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:15.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc996d6f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc990070360 con 0x7fc990071990 2023-03-03T13:08:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc996d6f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9901521a0 con 0x7fc99014b460 2023-03-03T13:08:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc996d6f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99006a5e0 con 0x7fc990072470 2023-03-03T13:08:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc994b0b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc990071990 0x7fc990071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37294/0 (socket says 172.21.15.7:37294) 2023-03-03T13:08:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc994b0b700 1 -- 172.21.15.7:0/3878461769 learned_addr learned my addr 172.21.15.7:0/3878461769 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:08:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc98ffff700 1 --2- 172.21.15.7:0/3878461769 >> v2:172.21.15.143:3300/0 conn(0x7fc99014b460 0x7fc99006aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048160984 0 0) 0x7fc990070360 con 0x7fc990071990 2023-03-03T13:08:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc99530c700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc990072470 0x7fc99006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.370+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9800035b0 con 0x7fc990071990 2023-03-03T13:08:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4094330344 0 0) 0x7fc9800035b0 con 0x7fc990071990 2023-03-03T13:08:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc990072470 msgr2=0x7fc99006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc990072470 0x7fc99006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 >> v2:172.21.15.143:3300/0 conn(0x7fc99014b460 msgr2=0x7fc99006aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 --2- 172.21.15.7:0/3878461769 >> v2:172.21.15.143:3300/0 conn(0x7fc99014b460 0x7fc99006aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9900c38b0 con 0x7fc990071990 2023-03-03T13:08:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc984001670 con 0x7fc990071990 2023-03-03T13:08:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 >> v1:172.21.15.7:6789/0 conn(0x7fc990071990 legacy=0x7fc990071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc980003a70 0x7fc980003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc980004850 0x7fc980004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc990072470 0x7fc980005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901521a0 con 0x7fc980003a70 2023-03-03T13:08:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99006a5e0 con 0x7fc980004850 2023-03-03T13:08:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98000ba40 con 0x7fc990072470 2023-03-03T13:08:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc994b0b700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc980003a70 0x7fc980003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98ffff700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc980004850 0x7fc980004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc99530c700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc990072470 0x7fc980005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc994b0b700 1 -- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc990072470 msgr2=0x7fc980005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc994b0b700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc990072470 0x7fc980005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc994b0b700 1 -- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc980004850 msgr2=0x7fc980004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc994b0b700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc980004850 0x7fc980004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc994b0b700 1 -- 172.21.15.7:0/3878461769 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9800035b0 con 0x7fc980003a70 2023-03-03T13:08:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc98ffff700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc980004850 0x7fc980004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:08:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc99530c700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc990072470 0x7fc980005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:08:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.371+0000 7fc994b0b700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc980003a70 0x7fc980003e80 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fc98400c370 tx=0x7fc980005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e58df1ee00402d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.372+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc984007c60 con 0x7fc980003a70 2023-03-03T13:08:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.372+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc984005960 con 0x7fc980003a70 2023-03-03T13:08:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.372+0000 7fc98f7fe700 1 -- 172.21.15.7:0/3878461769 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc984005d10 con 0x7fc980003a70 2023-03-03T13:08:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.372+0000 7fc996d6f700 1 -- 172.21.15.7:0/3878461769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc980003a70 msgr2=0x7fc980003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.372+0000 7fc996d6f700 1 --2- 172.21.15.7:0/3878461769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc980003a70 0x7fc980003e80 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fc98400c370 tx=0x7fc980005880 comp rx=0 tx=0).stop 2023-03-03T13:08:15.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.372+0000 7fc99530c700 1 -- 172.21.15.7:0/3878461769 reap_dead start 2023-03-03T13:08:15.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.372+0000 7fc996d6f700 1 -- 172.21.15.7:0/3878461769 shutdown_connections 2023-03-03T13:08:15.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.372+0000 7fc996d6f700 1 -- 172.21.15.7:0/3878461769 >> 172.21.15.7:0/3878461769 conn(0x7fc9900a94d0 msgr2=0x7fc9900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:08:15.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.372+0000 7fc996d6f700 1 -- 172.21.15.7:0/3878461769 shutdown_connections 2023-03-03T13:08:15.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.372+0000 7fc996d6f700 1 -- 172.21.15.7:0/3878461769 wait complete. 2023-03-03T13:08:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.374+0000 7fc996d6f700 1 Processor -- start 2023-03-03T13:08:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.374+0000 7fc996d6f700 1 -- start start 2023-03-03T13:08:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc996d6f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc980003a70 0x7fc99006cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc996d6f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc990146580 0x7fc9901e6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc996d6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9901dc5e0 0x7fc990149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc996d6f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901dd670 con 0x7fc9901dc5e0 2023-03-03T13:08:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc996d6f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901dd390 con 0x7fc990146580 2023-03-03T13:08:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc996d6f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901eb010 con 0x7fc980003a70 2023-03-03T13:08:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc99530c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9901dc5e0 0x7fc990149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc99530c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9901dc5e0 0x7fc990149570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43330/0 (socket says 172.21.15.7:43330) 2023-03-03T13:08:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc99530c700 1 -- 172.21.15.7:0/3310391059 learned_addr learned my addr 172.21.15.7:0/3310391059 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:08:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc98ffff700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc990146580 0x7fc9901e6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc994b0b700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc980003a70 0x7fc99006cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc99530c700 1 -- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc980003a70 msgr2=0x7fc99006cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc99530c700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc980003a70 0x7fc99006cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc99530c700 1 -- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc990146580 msgr2=0x7fc9901e6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc99530c700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc990146580 0x7fc9901e6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.375+0000 7fc99530c700 1 -- 172.21.15.7:0/3310391059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9900c3b50 con 0x7fc9901dc5e0 2023-03-03T13:08:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.376+0000 7fc99530c700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9901dc5e0 0x7fc990149570 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fc9900be320 tx=0x7fc988011640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.376+0000 7fc98d7fa700 1 -- 172.21.15.7:0/3310391059 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc988014650 con 0x7fc9901dc5e0 2023-03-03T13:08:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.376+0000 7fc98d7fa700 1 -- 172.21.15.7:0/3310391059 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9880147b0 con 0x7fc9901dc5e0 2023-03-03T13:08:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.376+0000 7fc98d7fa700 1 -- 172.21.15.7:0/3310391059 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc988014b60 con 0x7fc9901dc5e0 2023-03-03T13:08:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.376+0000 7fc996d6f700 1 -- 172.21.15.7:0/3310391059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99006a2b0 con 0x7fc9901dc5e0 2023-03-03T13:08:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.376+0000 7fc996d6f700 1 -- 172.21.15.7:0/3310391059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc990151c00 con 0x7fc9901dc5e0 2023-03-03T13:08:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.377+0000 7fc996d6f700 1 -- 172.21.15.7:0/3310391059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc990074bb0 con 0x7fc9901dc5e0 2023-03-03T13:08:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.377+0000 7fc98d7fa700 1 -- 172.21.15.7:0/3310391059 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc988023030 con 0x7fc9901dc5e0 2023-03-03T13:08:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.378+0000 7fc98d7fa700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9740682f0 0x7fc97406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.378+0000 7fc98d7fa700 1 -- 172.21.15.7:0/3310391059 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc988088c70 con 0x7fc9901dc5e0 2023-03-03T13:08:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.378+0000 7fc994b0b700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9740682f0 0x7fc97406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.378+0000 7fc994b0b700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9740682f0 0x7fc97406a7a0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fc984015040 tx=0x7fc98400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.382+0000 7fc98d7fa700 1 -- 172.21.15.7:0/3310391059 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc988055010 con 0x7fc9901dc5e0 2023-03-03T13:08:15.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.622+0000 7fc996d6f700 1 -- 172.21.15.7:0/3310391059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9901dd7b0 con 0x7fc9901dc5e0 2023-03-03T13:08:15.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.623+0000 7fc98d7fa700 1 -- 172.21.15.7:0/3310391059 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc9901dd7b0 con 0x7fc9901dc5e0 2023-03-03T13:08:15.625 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:08:15.625 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:08:15.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 -- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9740682f0 msgr2=0x7fc97406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:15.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9740682f0 0x7fc97406a7a0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fc984015040 tx=0x7fc98400a000 comp rx=0 tx=0).stop 2023-03-03T13:08:15.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 -- 172.21.15.7:0/3310391059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9901dc5e0 msgr2=0x7fc990149570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:15.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9901dc5e0 0x7fc990149570 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fc9900be320 tx=0x7fc988011640 comp rx=0 tx=0).stop 2023-03-03T13:08:15.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 -- 172.21.15.7:0/3310391059 shutdown_connections 2023-03-03T13:08:15.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9740682f0 0x7fc97406a7a0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:15.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9901dc5e0 0x7fc990149570 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:15.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc980003a70 0x7fc99006cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:15.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 --2- 172.21.15.7:0/3310391059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc990146580 0x7fc9901e6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:15.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 -- 172.21.15.7:0/3310391059 >> 172.21.15.7:0/3310391059 conn(0x7fc9900a94d0 msgr2=0x7fc9900c5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:08:15.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 -- 172.21.15.7:0/3310391059 shutdown_connections 2023-03-03T13:08:15.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:15.625+0000 7fc996d6f700 1 -- 172.21.15.7:0/3310391059 wait complete. 2023-03-03T13:08:15.629 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:08:15.639 INFO:tasks.mds_thrash.fs.[a]:waiting for 31 secs before thrashing 2023-03-03T13:08:16.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:08:16.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.466+0000 7fa34a8d3700 1 Processor -- start 2023-03-03T13:08:16.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.466+0000 7fa34a8d3700 1 -- start start 2023-03-03T13:08:16.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.466+0000 7fa34a8d3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa344151b20 0x7fa3440736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:16.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.466+0000 7fa34a8d3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34406cb70 con 0x7fa3440c2950 2023-03-03T13:08:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.466+0000 7fa34a8d3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34406ce50 con 0x7fa344073c30 2023-03-03T13:08:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.466+0000 7fa34a8d3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34406c2f0 con 0x7fa344151b20 2023-03-03T13:08:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa3437fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa3440c2950 0x7fa344157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37314/0 (socket says 172.21.15.7:37314) 2023-03-03T13:08:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa3437fe700 1 -- 172.21.15.7:0/2449529015 learned_addr learned my addr 172.21.15.7:0/2449529015 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:08:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa343fff700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa344151b20 0x7fa3440736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331289980 0 0) 0x7fa34406cb70 con 0x7fa3440c2950 2023-03-03T13:08:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa32c0035b0 con 0x7fa3440c2950 2023-03-03T13:08:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1293425827 0 0) 0x7fa32c0035b0 con 0x7fa3440c2950 2023-03-03T13:08:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa344151b20 msgr2=0x7fa3440736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa344151b20 0x7fa3440736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 >> v1:172.21.15.143:6789/0 conn(0x7fa344073c30 legacy=0x7fa3441581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34406d130 con 0x7fa3440c2950 2023-03-03T13:08:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa338001670 con 0x7fa3440c2950 2023-03-03T13:08:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa343fff700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa344151b20 0x7fa3440736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:08:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 >> v1:172.21.15.7:6789/0 conn(0x7fa3440c2950 legacy=0x7fa344157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa32c003aa0 0x7fa32c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa344151b20 0x7fa32c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa32c004e00 0x7fa32c005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34406c2f0 con 0x7fa32c004e00 2023-03-03T13:08:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c00bcb0 con 0x7fa32c003aa0 2023-03-03T13:08:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c00be70 con 0x7fa344151b20 2023-03-03T13:08:16.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa348e70700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa32c004e00 0x7fa32c005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa343fff700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa32c003aa0 0x7fa32c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.467+0000 7fa3437fe700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa344151b20 0x7fa32c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa348e70700 1 -- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa344151b20 msgr2=0x7fa32c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa348e70700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa344151b20 0x7fa32c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa348e70700 1 -- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa32c003aa0 msgr2=0x7fa32c003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa348e70700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa32c003aa0 0x7fa32c003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa348e70700 1 -- 172.21.15.7:0/2449529015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c0035b0 con 0x7fa32c004e00 2023-03-03T13:08:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa348e70700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa32c004e00 0x7fa32c005270 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00c240 tx=0x7fa32c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53781b8916d06cd1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c001df0 con 0x7fa32c004e00 2023-03-03T13:08:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33c00cce0 con 0x7fa32c004e00 2023-03-03T13:08:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa342ffd700 1 -- 172.21.15.7:0/2449529015 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c003c40 con 0x7fa32c004e00 2023-03-03T13:08:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2449529015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa32c004e00 msgr2=0x7fa32c005270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa34a8d3700 1 --2- 172.21.15.7:0/2449529015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa32c004e00 0x7fa32c005270 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00c240 tx=0x7fa32c005880 comp rx=0 tx=0).stop 2023-03-03T13:08:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa348e70700 1 -- 172.21.15.7:0/2449529015 reap_dead start 2023-03-03T13:08:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2449529015 shutdown_connections 2023-03-03T13:08:16.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.468+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2449529015 >> 172.21.15.7:0/2449529015 conn(0x7fa3440a94d0 msgr2=0x7fa3440a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:08:16.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.469+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2449529015 shutdown_connections 2023-03-03T13:08:16.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.469+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2449529015 wait complete. 2023-03-03T13:08:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.470+0000 7fa34a8d3700 1 Processor -- start 2023-03-03T13:08:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.470+0000 7fa34a8d3700 1 -- start start 2023-03-03T13:08:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa34a8d3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa34414ded0 0x7fa34414c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa34a8d3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa34414e2a0 0x7fa34414b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa34a8d3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa34414b6a0 0x7fa3440c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa34a8d3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440c7080 con 0x7fa34414e2a0 2023-03-03T13:08:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa34a8d3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3441ddbd0 con 0x7fa34414b6a0 2023-03-03T13:08:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa34a8d3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34414bb10 con 0x7fa34414ded0 2023-03-03T13:08:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa34414e2a0 0x7fa34414b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa34414e2a0 0x7fa34414b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43348/0 (socket says 172.21.15.7:43348) 2023-03-03T13:08:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa3437fe700 1 -- 172.21.15.7:0/2349232733 learned_addr learned my addr 172.21.15.7:0/2349232733 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:08:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa348e70700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa34414b6a0 0x7fa3440c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa343fff700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa34414ded0 0x7fa34414c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa3437fe700 1 -- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa34414ded0 msgr2=0x7fa34414c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa3437fe700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa34414ded0 0x7fa34414c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa3437fe700 1 -- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa34414b6a0 msgr2=0x7fa3440c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa3437fe700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa34414b6a0 0x7fa3440c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa3437fe700 1 -- 172.21.15.7:0/2349232733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa344151ef0 con 0x7fa34414e2a0 2023-03-03T13:08:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa3437fe700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa34414e2a0 0x7fa34414b160 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fa3380072f0 tx=0x7fa338007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa340ff9700 1 -- 172.21.15.7:0/2349232733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338011580 con 0x7fa34414e2a0 2023-03-03T13:08:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.471+0000 7fa340ff9700 1 -- 172.21.15.7:0/2349232733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3380116e0 con 0x7fa34414e2a0 2023-03-03T13:08:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.472+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2349232733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34406d410 con 0x7fa34414e2a0 2023-03-03T13:08:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.472+0000 7fa340ff9700 1 -- 172.21.15.7:0/2349232733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338011a90 con 0x7fa34414e2a0 2023-03-03T13:08:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.472+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2349232733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34406bea0 con 0x7fa34414e2a0 2023-03-03T13:08:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.472+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2349232733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa344074bb0 con 0x7fa34414e2a0 2023-03-03T13:08:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.473+0000 7fa340ff9700 1 -- 172.21.15.7:0/2349232733 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa33801c030 con 0x7fa34414e2a0 2023-03-03T13:08:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.473+0000 7fa340ff9700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3300682f0 0x7fa33006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.473+0000 7fa340ff9700 1 -- 172.21.15.7:0/2349232733 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa3380858d0 con 0x7fa34414e2a0 2023-03-03T13:08:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.474+0000 7fa343fff700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3300682f0 0x7fa33006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.474+0000 7fa343fff700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3300682f0 0x7fa33006a7a0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa3340045a0 tx=0x7fa334008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:16.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.478+0000 7fa340ff9700 1 -- 172.21.15.7:0/2349232733 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa338051c70 con 0x7fa34414e2a0 2023-03-03T13:08:16.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.717+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2349232733 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3440a9fb0 con 0x7fa34414e2a0 2023-03-03T13:08:16.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.718+0000 7fa340ff9700 1 -- 172.21.15.7:0/2349232733 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa338055290 con 0x7fa34414e2a0 2023-03-03T13:08:16.720 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:08:16.720 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:08:16.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3300682f0 msgr2=0x7fa33006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:16.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3300682f0 0x7fa33006a7a0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa3340045a0 tx=0x7fa334008040 comp rx=0 tx=0).stop 2023-03-03T13:08:16.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2349232733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa34414e2a0 msgr2=0x7fa34414b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:16.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa34414e2a0 0x7fa34414b160 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fa3380072f0 tx=0x7fa338007600 comp rx=0 tx=0).stop 2023-03-03T13:08:16.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2349232733 shutdown_connections 2023-03-03T13:08:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa34414b6a0 0x7fa3440c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa34414ded0 0x7fa34414c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3300682f0 0x7fa33006a7a0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 --2- 172.21.15.7:0/2349232733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa34414e2a0 0x7fa34414b160 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2349232733 >> 172.21.15.7:0/2349232733 conn(0x7fa3440a94d0 msgr2=0x7fa344071420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:08:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2349232733 shutdown_connections 2023-03-03T13:08:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:16.720+0000 7fa34a8d3700 1 -- 172.21.15.7:0/2349232733 wait complete. 2023-03-03T13:08:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:08:16.734 INFO:tasks.mds_thrash.fs.[b]:waiting for 13 secs before thrashing 2023-03-03T13:08:29.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:08:29.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.873+0000 7ff60b327700 1 Processor -- start 2023-03-03T13:08:29.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.873+0000 7ff60b327700 1 -- start start 2023-03-03T13:08:29.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff60b327700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff604071e80 0x7ff604151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:29.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff60b327700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff604072250 0x7ff60414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:29.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff60b327700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6040c1ce0 0x7ff60414e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:29.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff60b327700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6041510d0 con 0x7ff604071e80 2023-03-03T13:08:29.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff60b327700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6040c3740 con 0x7ff604072250 2023-03-03T13:08:29.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff60b327700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff604070670 con 0x7ff6040c1ce0 2023-03-03T13:08:29.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6090c3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff604071e80 0x7ff604151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:29.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6090c3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff604071e80 0x7ff604151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43366/0 (socket says 172.21.15.7:43366) 2023-03-03T13:08:29.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6090c3700 1 -- 172.21.15.7:0/1080514780 learned_addr learned my addr 172.21.15.7:0/1080514780 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:08:29.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6088c2700 1 --2- 172.21.15.7:0/1080514780 >> v2:172.21.15.143:3300/0 conn(0x7ff604072250 0x7ff60414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:29.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6098c4700 1 --2- 172.21.15.7:0/1080514780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6040c1ce0 0x7ff60414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:29.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6090c3700 1 -- 172.21.15.7:0/1080514780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6040c1ce0 msgr2=0x7ff60414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:29.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6090c3700 1 --2- 172.21.15.7:0/1080514780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6040c1ce0 0x7ff60414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:29.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6090c3700 1 -- 172.21.15.7:0/1080514780 >> v2:172.21.15.143:3300/0 conn(0x7ff604072250 msgr2=0x7ff60414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:29.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6090c3700 1 --2- 172.21.15.7:0/1080514780 >> v2:172.21.15.143:3300/0 conn(0x7ff604072250 0x7ff60414e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:29.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6090c3700 1 -- 172.21.15.7:0/1080514780 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6040c39e0 con 0x7ff604071e80 2023-03-03T13:08:29.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6098c4700 1 --2- 172.21.15.7:0/1080514780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6040c1ce0 0x7ff60414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:08:29.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.874+0000 7ff6090c3700 1 --2- 172.21.15.7:0/1080514780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 0x7ff604151820 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7ff60406b520 tx=0x7ff600009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2c1cf34b946d45 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:29.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff5fbfff700 1 -- 172.21.15.7:0/1080514780 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60000b870 con 0x7ff604071e80 2023-03-03T13:08:29.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff5fbfff700 1 -- 172.21.15.7:0/1080514780 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60000b9d0 con 0x7ff604071e80 2023-03-03T13:08:29.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff5fbfff700 1 -- 172.21.15.7:0/1080514780 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60000bd60 con 0x7ff604071e80 2023-03-03T13:08:29.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff60b327700 1 -- 172.21.15.7:0/1080514780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 msgr2=0x7ff604151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:29.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff60b327700 1 --2- 172.21.15.7:0/1080514780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 0x7ff604151820 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7ff60406b520 tx=0x7ff600009bd0 comp rx=0 tx=0).stop 2023-03-03T13:08:29.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff60b327700 1 -- 172.21.15.7:0/1080514780 shutdown_connections 2023-03-03T13:08:29.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff60b327700 1 --2- 172.21.15.7:0/1080514780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6040c1ce0 0x7ff60414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:29.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff60b327700 1 --2- 172.21.15.7:0/1080514780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 0x7ff604151820 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:29.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff60b327700 1 --2- 172.21.15.7:0/1080514780 >> v2:172.21.15.143:3300/0 conn(0x7ff604072250 0x7ff60414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:29.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff60b327700 1 -- 172.21.15.7:0/1080514780 >> 172.21.15.7:0/1080514780 conn(0x7ff6040a94d0 msgr2=0x7ff6040a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:08:29.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff60b327700 1 -- 172.21.15.7:0/1080514780 shutdown_connections 2023-03-03T13:08:29.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.875+0000 7ff60b327700 1 -- 172.21.15.7:0/1080514780 wait complete. 2023-03-03T13:08:29.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff60b327700 1 Processor -- start 2023-03-03T13:08:29.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff60b327700 1 -- start start 2023-03-03T13:08:29.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff60b327700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 0x7ff6041dd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:29.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff60b327700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff604072250 0x7ff6041ea850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:29.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff60b327700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6040c1ce0 0x7ff604146f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff60b327700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6041e9e60 con 0x7ff604071e80 2023-03-03T13:08:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff60b327700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6041ddd90 con 0x7ff6040c1ce0 2023-03-03T13:08:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff60b327700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6041ead90 con 0x7ff604072250 2023-03-03T13:08:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff6090c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 0x7ff6041dd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff6090c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 0x7ff6041dd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43372/0 (socket says 172.21.15.7:43372) 2023-03-03T13:08:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.877+0000 7ff6090c3700 1 -- 172.21.15.7:0/2028864194 learned_addr learned my addr 172.21.15.7:0/2028864194 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:08:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff6098c4700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6040c1ce0 0x7ff604146f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff6088c2700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff604072250 0x7ff6041ea850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff6090c3700 1 -- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff604072250 msgr2=0x7ff6041ea850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff6090c3700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff604072250 0x7ff6041ea850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff6090c3700 1 -- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6040c1ce0 msgr2=0x7ff604146f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff6090c3700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6040c1ce0 0x7ff604146f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff6090c3700 1 -- 172.21.15.7:0/2028864194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600009550 con 0x7ff604071e80 2023-03-03T13:08:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff6088c2700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff604072250 0x7ff6041ea850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:08:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff6098c4700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6040c1ce0 0x7ff604146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:08:29.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff6090c3700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 0x7ff6041dd850 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7ff600000c00 tx=0x7ff600004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:29.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff5f9ffb700 1 -- 172.21.15.7:0/2028864194 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600004540 con 0x7ff604071e80 2023-03-03T13:08:29.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff5f9ffb700 1 -- 172.21.15.7:0/2028864194 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6000036a0 con 0x7ff604071e80 2023-03-03T13:08:29.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff60b327700 1 -- 172.21.15.7:0/2028864194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60406c9a0 con 0x7ff604071e80 2023-03-03T13:08:29.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff5f9ffb700 1 -- 172.21.15.7:0/2028864194 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600003a50 con 0x7ff604071e80 2023-03-03T13:08:29.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.878+0000 7ff60b327700 1 -- 172.21.15.7:0/2028864194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff604151300 con 0x7ff604071e80 2023-03-03T13:08:29.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.879+0000 7ff60b327700 1 -- 172.21.15.7:0/2028864194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60406cb30 con 0x7ff604071e80 2023-03-03T13:08:29.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.880+0000 7ff5f9ffb700 1 -- 172.21.15.7:0/2028864194 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff60000f050 con 0x7ff604071e80 2023-03-03T13:08:29.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.880+0000 7ff5f9ffb700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5e80682f0 0x7ff5e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:29.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.880+0000 7ff5f9ffb700 1 -- 172.21.15.7:0/2028864194 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff600087a10 con 0x7ff604071e80 2023-03-03T13:08:29.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.880+0000 7ff6088c2700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5e80682f0 0x7ff5e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:29.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.881+0000 7ff6088c2700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5e80682f0 0x7ff5e806a7a0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7ff6041554c0 tx=0x7ff5f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:29.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:29.885+0000 7ff5f9ffb700 1 -- 172.21.15.7:0/2028864194 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff600053db0 con 0x7ff604071e80 2023-03-03T13:08:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.118+0000 7ff60b327700 1 -- 172.21.15.7:0/2028864194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6041ea6a0 con 0x7ff604071e80 2023-03-03T13:08:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.119+0000 7ff5f9ffb700 1 -- 172.21.15.7:0/2028864194 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff6000573d0 con 0x7ff604071e80 2023-03-03T13:08:30.121 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:08:30.121 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:08:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 -- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5e80682f0 msgr2=0x7ff5e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5e80682f0 0x7ff5e806a7a0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7ff6041554c0 tx=0x7ff5f4008040 comp rx=0 tx=0).stop 2023-03-03T13:08:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 -- 172.21.15.7:0/2028864194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 msgr2=0x7ff6041dd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 0x7ff6041dd850 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7ff600000c00 tx=0x7ff600004750 comp rx=0 tx=0).stop 2023-03-03T13:08:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 -- 172.21.15.7:0/2028864194 shutdown_connections 2023-03-03T13:08:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6040c1ce0 0x7ff604146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5e80682f0 0x7ff5e806a7a0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff604071e80 0x7ff6041dd850 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 --2- 172.21.15.7:0/2028864194 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff604072250 0x7ff6041ea850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 -- 172.21.15.7:0/2028864194 >> 172.21.15.7:0/2028864194 conn(0x7ff6040a94d0 msgr2=0x7ff60406d580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:08:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 -- 172.21.15.7:0/2028864194 shutdown_connections 2023-03-03T13:08:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:30.121+0000 7ff60b327700 1 -- 172.21.15.7:0/2028864194 wait complete. 2023-03-03T13:08:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:08:30.134 INFO:tasks.mds_thrash.fs.[b]:waiting for 75 secs before thrashing 2023-03-03T13:08:37.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:37.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:08:46.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:08:46.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.778+0000 7f146b7b5700 1 Processor -- start 2023-03-03T13:08:46.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.778+0000 7f146b7b5700 1 -- start start 2023-03-03T13:08:46.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f146b7b5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1464151ab0 0x7f14640736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:46.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f146b7b5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 0x7f1464157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:46.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f146b7b5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146406d780 con 0x7f1464073c30 2023-03-03T13:08:46.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f146b7b5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f146406cb70 con 0x7f1464151ab0 2023-03-03T13:08:46.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f146b7b5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f146406ce50 con 0x7f146406c5b0 2023-03-03T13:08:46.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f1469d52700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 0x7f1464157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:46.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f1469551700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1464151ab0 0x7f14640736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:46.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f1469d52700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 0x7f1464157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:51472/0 (socket says 172.21.15.7:51472) 2023-03-03T13:08:46.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f1469d52700 1 -- 172.21.15.7:0/2810431684 learned_addr learned my addr 172.21.15.7:0/2810431684 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:08:46.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f1469d52700 1 -- 172.21.15.7:0/2810431684 >> v2:172.21.15.143:3300/0 conn(0x7f1464151ab0 msgr2=0x7f14640736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:08:46.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f1469d52700 1 --2- 172.21.15.7:0/2810431684 >> v2:172.21.15.143:3300/0 conn(0x7f1464151ab0 0x7f14640736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:46.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f1469d52700 1 -- 172.21.15.7:0/2810431684 >> v1:172.21.15.143:6789/0 conn(0x7f1464073c30 legacy=0x7f146406bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:46.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.779+0000 7f1469d52700 1 -- 172.21.15.7:0/2810431684 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146406d130 con 0x7f146406c5b0 2023-03-03T13:08:46.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f1469551700 1 --2- 172.21.15.7:0/2810431684 >> v2:172.21.15.143:3300/0 conn(0x7f1464151ab0 0x7f14640736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:08:46.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f1469d52700 1 --2- 172.21.15.7:0/2810431684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 0x7f1464157dc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f145c009e50 tx=0x7f145c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=983afc63303646f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:46.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f145bfff700 1 -- 172.21.15.7:0/2810431684 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c00e070 con 0x7f146406c5b0 2023-03-03T13:08:46.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f145bfff700 1 -- 172.21.15.7:0/2810431684 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c00b910 con 0x7f146406c5b0 2023-03-03T13:08:46.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f145bfff700 1 -- 172.21.15.7:0/2810431684 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c00bcc0 con 0x7f146406c5b0 2023-03-03T13:08:46.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f146b7b5700 1 -- 172.21.15.7:0/2810431684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 msgr2=0x7f1464157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:46.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f146b7b5700 1 --2- 172.21.15.7:0/2810431684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 0x7f1464157dc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f145c009e50 tx=0x7f145c00b330 comp rx=0 tx=0).stop 2023-03-03T13:08:46.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f146b7b5700 1 -- 172.21.15.7:0/2810431684 shutdown_connections 2023-03-03T13:08:46.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f146b7b5700 1 --2- 172.21.15.7:0/2810431684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 0x7f1464157dc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:46.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f146b7b5700 1 --2- 172.21.15.7:0/2810431684 >> v2:172.21.15.143:3300/0 conn(0x7f1464151ab0 0x7f14640736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:46.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.780+0000 7f146b7b5700 1 -- 172.21.15.7:0/2810431684 >> 172.21.15.7:0/2810431684 conn(0x7f14640a94d0 msgr2=0x7f14640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:08:46.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.781+0000 7f146b7b5700 1 -- 172.21.15.7:0/2810431684 shutdown_connections 2023-03-03T13:08:46.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.781+0000 7f146b7b5700 1 -- 172.21.15.7:0/2810431684 wait complete. 2023-03-03T13:08:46.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.782+0000 7f146b7b5700 1 Processor -- start 2023-03-03T13:08:46.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.782+0000 7f146b7b5700 1 -- start start 2023-03-03T13:08:46.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.782+0000 7f146b7b5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 0x7f146414cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:46.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.782+0000 7f146b7b5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1464073c30 0x7f14640bf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:46.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.782+0000 7f146b7b5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1464151ab0 0x7f14640be5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:46.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f146b7b5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14641ea220 con 0x7f1464151ab0 2023-03-03T13:08:46.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f146b7b5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14640c1300 con 0x7f1464073c30 2023-03-03T13:08:46.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f146b7b5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14641ea4a0 con 0x7f146406c5b0 2023-03-03T13:08:46.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1469d52700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1464151ab0 0x7f14640be5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:46.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1469d52700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1464151ab0 0x7f14640be5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43406/0 (socket says 172.21.15.7:43406) 2023-03-03T13:08:46.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1469d52700 1 -- 172.21.15.7:0/1684252680 learned_addr learned my addr 172.21.15.7:0/1684252680 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:08:46.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1468d50700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1464073c30 0x7f14640bf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:46.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1469551700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 0x7f146414cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:46.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1469d52700 1 -- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 msgr2=0x7f146414cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:46.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1469d52700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 0x7f146414cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:46.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1469d52700 1 -- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1464073c30 msgr2=0x7f14640bf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:46.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1469d52700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1464073c30 0x7f14640bf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:46.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1469d52700 1 -- 172.21.15.7:0/1684252680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14540094a0 con 0x7f1464151ab0 2023-03-03T13:08:46.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1468d50700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1464073c30 0x7f14640bf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:08:46.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1469d52700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1464151ab0 0x7f14640be5f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f145c012000 tx=0x7f145c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:46.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1459ffb700 1 -- 172.21.15.7:0/1684252680 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c004960 con 0x7f1464151ab0 2023-03-03T13:08:46.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1459ffb700 1 -- 172.21.15.7:0/1684252680 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145c0092e0 con 0x7f1464151ab0 2023-03-03T13:08:46.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f1459ffb700 1 -- 172.21.15.7:0/1684252680 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c01c660 con 0x7f1464151ab0 2023-03-03T13:08:46.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.783+0000 7f146b7b5700 1 -- 172.21.15.7:0/1684252680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f145c009b30 con 0x7f1464151ab0 2023-03-03T13:08:46.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.784+0000 7f146b7b5700 1 -- 172.21.15.7:0/1684252680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1464151e80 con 0x7f1464151ab0 2023-03-03T13:08:46.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.785+0000 7f146b7b5700 1 -- 172.21.15.7:0/1684252680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146406d410 con 0x7f1464151ab0 2023-03-03T13:08:46.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.785+0000 7f1459ffb700 1 -- 172.21.15.7:0/1684252680 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f145c022030 con 0x7f1464151ab0 2023-03-03T13:08:46.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.785+0000 7f1459ffb700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14480682f0 0x7f144806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:08:46.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.786+0000 7f1459ffb700 1 -- 172.21.15.7:0/1684252680 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f145c087a00 con 0x7f1464151ab0 2023-03-03T13:08:46.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.786+0000 7f1469551700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14480682f0 0x7f144806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:08:46.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.786+0000 7f1469551700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14480682f0 0x7f144806a7a0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f1464150290 tx=0x7f1454008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:08:46.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:46.790+0000 7f1459ffb700 1 -- 172.21.15.7:0/1684252680 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f145c053da0 con 0x7f1464151ab0 2023-03-03T13:08:47.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.027+0000 7f146b7b5700 1 -- 172.21.15.7:0/1684252680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1464152010 con 0x7f1464151ab0 2023-03-03T13:08:47.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.028+0000 7f1459ffb700 1 -- 172.21.15.7:0/1684252680 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f145c0573c0 con 0x7f1464151ab0 2023-03-03T13:08:47.030 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:08:47.030 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:08:47.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 -- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14480682f0 msgr2=0x7f144806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:47.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14480682f0 0x7f144806a7a0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f1464150290 tx=0x7f1454008040 comp rx=0 tx=0).stop 2023-03-03T13:08:47.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 -- 172.21.15.7:0/1684252680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1464151ab0 msgr2=0x7f14640be5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:08:47.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1464151ab0 0x7f14640be5f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f145c012000 tx=0x7f145c004ad0 comp rx=0 tx=0).stop 2023-03-03T13:08:47.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 -- 172.21.15.7:0/1684252680 shutdown_connections 2023-03-03T13:08:47.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14480682f0 0x7f144806a7a0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:47.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1464151ab0 0x7f14640be5f0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:47.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f146406c5b0 0x7f146414cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:47.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 --2- 172.21.15.7:0/1684252680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1464073c30 0x7f14640bf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:08:47.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 -- 172.21.15.7:0/1684252680 >> 172.21.15.7:0/1684252680 conn(0x7f14640a94d0 msgr2=0x7f1464156a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:08:47.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 -- 172.21.15.7:0/1684252680 shutdown_connections 2023-03-03T13:08:47.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:08:47.030+0000 7f146b7b5700 1 -- 172.21.15.7:0/1684252680 wait complete. 2023-03-03T13:08:47.034 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:08:47.043 INFO:tasks.mds_thrash.fs.[a]:waiting for 41 secs before thrashing 2023-03-03T13:09:07.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:07.930 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:28.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:09:28.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.181+0000 7f0fb57cf700 1 Processor -- start 2023-03-03T13:09:28.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.182+0000 7f0fb57cf700 1 -- start start 2023-03-03T13:09:28.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.182+0000 7f0fb57cf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0fb00c5c60 0x7f0fb0154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:28.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.182+0000 7f0fb57cf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb00c7310 con 0x7f0fb00c00e0 2023-03-03T13:09:28.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.182+0000 7f0fb57cf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb00bfd10 con 0x7f0fb00caa30 2023-03-03T13:09:28.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.182+0000 7f0fb57cf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb00c60a0 con 0x7f0fb00c5c60 2023-03-03T13:09:28.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.182+0000 7f0faf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0fb00c5c60 0x7f0fb0154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:28.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.182+0000 7f0fae7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0fb00c00e0 0x7f0fb015cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37422/0 (socket says 172.21.15.7:37422) 2023-03-03T13:09:28.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.182+0000 7f0fae7fc700 1 -- 172.21.15.7:0/986876867 learned_addr learned my addr 172.21.15.7:0/986876867 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:09:28.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.183+0000 7f0faf7fe700 1 -- 172.21.15.7:0/986876867 >> v1:172.21.15.143:6789/0 conn(0x7f0fb00caa30 legacy=0x7f0fb00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:09:28.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.183+0000 7f0faf7fe700 1 -- 172.21.15.7:0/986876867 >> v1:172.21.15.7:6789/0 conn(0x7f0fb00c00e0 legacy=0x7f0fb015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:28.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.183+0000 7f0faf7fe700 1 -- 172.21.15.7:0/986876867 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb00bfa30 con 0x7f0fb00c5c60 2023-03-03T13:09:28.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.183+0000 7f0fadffb700 1 -- 172.21.15.7:0/986876867 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597809870 0 0) 0x7f0fb00c7310 con 0x7f0fb00c00e0 2023-03-03T13:09:28.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.183+0000 7f0faf7fe700 1 --2- 172.21.15.7:0/986876867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c5c60 0x7f0fb0154ff0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009d90 tx=0x7f0fa800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c94cb05e6e8274c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:28.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.183+0000 7f0fadffb700 1 -- 172.21.15.7:0/986876867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa800e070 con 0x7f0fb00c5c60 2023-03-03T13:09:28.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.183+0000 7f0fadffb700 1 -- 172.21.15.7:0/986876867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa800bb90 con 0x7f0fb00c5c60 2023-03-03T13:09:28.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.183+0000 7f0fadffb700 1 -- 172.21.15.7:0/986876867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8003a20 con 0x7f0fb00c5c60 2023-03-03T13:09:28.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.184+0000 7f0fb57cf700 1 -- 172.21.15.7:0/986876867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c5c60 msgr2=0x7f0fb0154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:28.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.184+0000 7f0fb57cf700 1 --2- 172.21.15.7:0/986876867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c5c60 0x7f0fb0154ff0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009d90 tx=0x7f0fa800b330 comp rx=0 tx=0).stop 2023-03-03T13:09:28.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.184+0000 7f0fb57cf700 1 -- 172.21.15.7:0/986876867 shutdown_connections 2023-03-03T13:09:28.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.184+0000 7f0fb57cf700 1 --2- 172.21.15.7:0/986876867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c5c60 0x7f0fb0154ff0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:28.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.184+0000 7f0fb57cf700 1 -- 172.21.15.7:0/986876867 >> 172.21.15.7:0/986876867 conn(0x7f0fb0066e40 msgr2=0x7f0fb0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:28.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.184+0000 7f0fb57cf700 1 -- 172.21.15.7:0/986876867 shutdown_connections 2023-03-03T13:09:28.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.184+0000 7f0fb57cf700 1 -- 172.21.15.7:0/986876867 wait complete. 2023-03-03T13:09:28.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.185+0000 7f0fb57cf700 1 Processor -- start 2023-03-03T13:09:28.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.185+0000 7f0fb57cf700 1 -- start start 2023-03-03T13:09:28.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0fb57cf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c00e0 0x7f0fb01495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:28.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0fb57cf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fb00c5c60 0x7f0fb0148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:28.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0fb57cf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fb00caa30 0x7f0fb00c2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:28.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0fb57cf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb014d060 con 0x7f0fb00c00e0 2023-03-03T13:09:28.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0fb57cf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb01edc30 con 0x7f0fb00c5c60 2023-03-03T13:09:28.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0fb57cf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb006b8d0 con 0x7f0fb00caa30 2023-03-03T13:09:28.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0faeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c00e0 0x7f0fb01495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:28.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0faeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c00e0 0x7f0fb01495f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43446/0 (socket says 172.21.15.7:43446) 2023-03-03T13:09:28.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0faeffd700 1 -- 172.21.15.7:0/1170699242 learned_addr learned my addr 172.21.15.7:0/1170699242 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:09:28.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0fae7fc700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fb00c5c60 0x7f0fb0148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:28.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0faf7fe700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fb00caa30 0x7f0fb00c2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:28.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0faeffd700 1 -- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fb00caa30 msgr2=0x7f0fb00c2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:28.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0faeffd700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fb00caa30 0x7f0fb00c2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:28.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0faeffd700 1 -- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fb00c5c60 msgr2=0x7f0fb0148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:28.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0faeffd700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fb00c5c60 0x7f0fb0148100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:28.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0faeffd700 1 -- 172.21.15.7:0/1170699242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa8009a70 con 0x7f0fb00c00e0 2023-03-03T13:09:28.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.186+0000 7f0faf7fe700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fb00caa30 0x7f0fb00c2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:09:28.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.187+0000 7f0faeffd700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c00e0 0x7f0fb01495f0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f0fa000c4d0 tx=0x7f0fa000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:28.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.187+0000 7f0fa77fe700 1 -- 172.21.15.7:0/1170699242 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0011790 con 0x7f0fb00c00e0 2023-03-03T13:09:28.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.187+0000 7f0fa77fe700 1 -- 172.21.15.7:0/1170699242 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa00118f0 con 0x7f0fb00c00e0 2023-03-03T13:09:28.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.187+0000 7f0fb57cf700 1 -- 172.21.15.7:0/1170699242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb01547f0 con 0x7f0fb00c00e0 2023-03-03T13:09:28.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.187+0000 7f0fa77fe700 1 -- 172.21.15.7:0/1170699242 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0011ca0 con 0x7f0fb00c00e0 2023-03-03T13:09:28.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.187+0000 7f0fb57cf700 1 -- 172.21.15.7:0/1170699242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb0154510 con 0x7f0fb00c00e0 2023-03-03T13:09:28.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.188+0000 7f0fb57cf700 1 -- 172.21.15.7:0/1170699242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb0154ad0 con 0x7f0fb00c00e0 2023-03-03T13:09:28.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.188+0000 7f0fa77fe700 1 -- 172.21.15.7:0/1170699242 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0fa0010050 con 0x7f0fb00c00e0 2023-03-03T13:09:28.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.189+0000 7f0fa77fe700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f900682f0 0x7f0f9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:28.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.189+0000 7f0fa77fe700 1 -- 172.21.15.7:0/1170699242 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0fa0085c00 con 0x7f0fb00c00e0 2023-03-03T13:09:28.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.189+0000 7f0fae7fc700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f900682f0 0x7f0f9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:28.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.189+0000 7f0fae7fc700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f900682f0 0x7f0f9006a7a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0fb01508d0 tx=0x7f0f98009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:28.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.193+0000 7f0fa77fe700 1 -- 172.21.15.7:0/1170699242 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0fa008b020 con 0x7f0fb00c00e0 2023-03-03T13:09:28.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.436+0000 7f0fb57cf700 1 -- 172.21.15.7:0/1170699242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fb0074bb0 con 0x7f0fb00c00e0 2023-03-03T13:09:28.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.437+0000 7f0fa77fe700 1 -- 172.21.15.7:0/1170699242 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0fa0051f40 con 0x7f0fb00c00e0 2023-03-03T13:09:28.439 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:09:28.439 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:09:28.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 -- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f900682f0 msgr2=0x7f0f9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:28.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f900682f0 0x7f0f9006a7a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0fb01508d0 tx=0x7f0f98009040 comp rx=0 tx=0).stop 2023-03-03T13:09:28.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 -- 172.21.15.7:0/1170699242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c00e0 msgr2=0x7f0fb01495f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:28.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c00e0 0x7f0fb01495f0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f0fa000c4d0 tx=0x7f0fa000c890 comp rx=0 tx=0).stop 2023-03-03T13:09:28.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 -- 172.21.15.7:0/1170699242 shutdown_connections 2023-03-03T13:09:28.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0fb00caa30 0x7f0fb00c2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:28.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f900682f0 0x7f0f9006a7a0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:28.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0fb00c00e0 0x7f0fb01495f0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:28.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 --2- 172.21.15.7:0/1170699242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fb00c5c60 0x7f0fb0148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:28.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 -- 172.21.15.7:0/1170699242 >> 172.21.15.7:0/1170699242 conn(0x7f0fb0066e40 msgr2=0x7f0fb00c8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:28.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 -- 172.21.15.7:0/1170699242 shutdown_connections 2023-03-03T13:09:28.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:28.439+0000 7f0fb57cf700 1 -- 172.21.15.7:0/1170699242 wait complete. 2023-03-03T13:09:28.442 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:09:28.452 INFO:tasks.mds_thrash.fs.[a]:waiting for 20 secs before thrashing 2023-03-03T13:09:37.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:37.993 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:09:45.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:09:45.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.273+0000 7f6b2f827700 1 Processor -- start 2023-03-03T13:09:45.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.274+0000 7f6b2f827700 1 -- start start 2023-03-03T13:09:45.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.274+0000 7f6b2f827700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b28071e80 0x7f6b28151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:45.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.274+0000 7f6b2f827700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6b28072250 0x7f6b2814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:45.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.274+0000 7f6b2f827700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b281510d0 con 0x7f6b280c1ce0 2023-03-03T13:09:45.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.274+0000 7f6b2f827700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b280c3740 con 0x7f6b28072250 2023-03-03T13:09:45.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.274+0000 7f6b2f827700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28070670 con 0x7f6b28071e80 2023-03-03T13:09:45.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b2ddc4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6b280c1ce0 0x7f6b281546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37440/0 (socket says 172.21.15.7:37440) 2023-03-03T13:09:45.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b2ddc4700 1 -- 172.21.15.7:0/4179488781 learned_addr learned my addr 172.21.15.7:0/4179488781 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:09:45.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b2d5c3700 1 --2- 172.21.15.7:0/4179488781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b28071e80 0x7f6b28151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:45.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b2cdc2700 1 --2- 172.21.15.7:0/4179488781 >> v2:172.21.15.143:3300/0 conn(0x7f6b28072250 0x7f6b2814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:45.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b1bfff700 1 -- 172.21.15.7:0/4179488781 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263551759 0 0) 0x7f6b281510d0 con 0x7f6b280c1ce0 2023-03-03T13:09:45.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b1bfff700 1 -- 172.21.15.7:0/4179488781 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b140035b0 con 0x7f6b280c1ce0 2023-03-03T13:09:45.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b2d5c3700 1 -- 172.21.15.7:0/4179488781 >> v2:172.21.15.143:3300/0 conn(0x7f6b28072250 msgr2=0x7f6b2814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:09:45.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b2d5c3700 1 --2- 172.21.15.7:0/4179488781 >> v2:172.21.15.143:3300/0 conn(0x7f6b28072250 0x7f6b2814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b2d5c3700 1 -- 172.21.15.7:0/4179488781 >> v1:172.21.15.7:6789/0 conn(0x7f6b280c1ce0 legacy=0x7f6b281546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:45.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b2d5c3700 1 -- 172.21.15.7:0/4179488781 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b280c39e0 con 0x7f6b28071e80 2023-03-03T13:09:45.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b2cdc2700 1 --2- 172.21.15.7:0/4179488781 >> v2:172.21.15.143:3300/0 conn(0x7f6b28072250 0x7f6b2814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:09:45.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b2d5c3700 1 --2- 172.21.15.7:0/4179488781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b28071e80 0x7f6b28151820 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400bae0 tx=0x7f6b2400bdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20a326fb6c1c5a54 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:45.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b1bfff700 1 -- 172.21.15.7:0/4179488781 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2400dc70 con 0x7f6b28071e80 2023-03-03T13:09:45.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.275+0000 7f6b1bfff700 1 -- 172.21.15.7:0/4179488781 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2400ddd0 con 0x7f6b28071e80 2023-03-03T13:09:45.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.276+0000 7f6b1bfff700 1 -- 172.21.15.7:0/4179488781 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24010690 con 0x7f6b28071e80 2023-03-03T13:09:45.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.276+0000 7f6b2f827700 1 -- 172.21.15.7:0/4179488781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b28071e80 msgr2=0x7f6b28151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:45.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.276+0000 7f6b2f827700 1 --2- 172.21.15.7:0/4179488781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b28071e80 0x7f6b28151820 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400bae0 tx=0x7f6b2400bdf0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.276+0000 7f6b2f827700 1 -- 172.21.15.7:0/4179488781 shutdown_connections 2023-03-03T13:09:45.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.276+0000 7f6b2f827700 1 --2- 172.21.15.7:0/4179488781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b28071e80 0x7f6b28151820 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.276+0000 7f6b2f827700 1 --2- 172.21.15.7:0/4179488781 >> v2:172.21.15.143:3300/0 conn(0x7f6b28072250 0x7f6b2814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.276+0000 7f6b2f827700 1 -- 172.21.15.7:0/4179488781 >> 172.21.15.7:0/4179488781 conn(0x7f6b280a94d0 msgr2=0x7f6b280a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:45.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.276+0000 7f6b2f827700 1 -- 172.21.15.7:0/4179488781 shutdown_connections 2023-03-03T13:09:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.276+0000 7f6b2f827700 1 -- 172.21.15.7:0/4179488781 wait complete. 2023-03-03T13:09:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.278+0000 7f6b2f827700 1 Processor -- start 2023-03-03T13:09:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.278+0000 7f6b2f827700 1 -- start start 2023-03-03T13:09:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2f827700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b28071e80 0x7f6b281e9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2f827700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b28072250 0x7f6b280ac690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2f827700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b280c1ce0 0x7f6b280c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2f827700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28149810 con 0x7f6b28072250 2023-03-03T13:09:45.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2f827700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b280c5800 con 0x7f6b28071e80 2023-03-03T13:09:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2f827700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b280c6420 con 0x7f6b280c1ce0 2023-03-03T13:09:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2cdc2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b28072250 0x7f6b280ac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2cdc2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b28072250 0x7f6b280ac690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43464/0 (socket says 172.21.15.7:43464) 2023-03-03T13:09:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2cdc2700 1 -- 172.21.15.7:0/1431025399 learned_addr learned my addr 172.21.15.7:0/1431025399 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:09:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2d5c3700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b28071e80 0x7f6b281e9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2ddc4700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b280c1ce0 0x7f6b280c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2cdc2700 1 -- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b280c1ce0 msgr2=0x7f6b280c52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:45.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2cdc2700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b280c1ce0 0x7f6b280c52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2cdc2700 1 -- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b28071e80 msgr2=0x7f6b281e9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:45.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2cdc2700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b28071e80 0x7f6b281e9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2cdc2700 1 -- 172.21.15.7:0/1431025399 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c0094a0 con 0x7f6b28072250 2023-03-03T13:09:45.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b2cdc2700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b28072250 0x7f6b280ac690 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00ebb0 tx=0x7f6b1c00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:45.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.279+0000 7f6b19ffb700 1 -- 172.21.15.7:0/1431025399 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c00c850 con 0x7f6b28072250 2023-03-03T13:09:45.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.280+0000 7f6b19ffb700 1 -- 172.21.15.7:0/1431025399 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c00c9b0 con 0x7f6b28072250 2023-03-03T13:09:45.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.280+0000 7f6b2f827700 1 -- 172.21.15.7:0/1431025399 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2400b740 con 0x7f6b28072250 2023-03-03T13:09:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.280+0000 7f6b19ffb700 1 -- 172.21.15.7:0/1431025399 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c00cd40 con 0x7f6b28072250 2023-03-03T13:09:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.280+0000 7f6b2f827700 1 -- 172.21.15.7:0/1431025399 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2806c9a0 con 0x7f6b28072250 2023-03-03T13:09:45.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.281+0000 7f6b2f827700 1 -- 172.21.15.7:0/1431025399 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b28151300 con 0x7f6b28072250 2023-03-03T13:09:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.281+0000 7f6b19ffb700 1 -- 172.21.15.7:0/1431025399 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6b1c023070 con 0x7f6b28072250 2023-03-03T13:09:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.282+0000 7f6b19ffb700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b0c0682f0 0x7f6b0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.282+0000 7f6b19ffb700 1 -- 172.21.15.7:0/1431025399 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6b1c086400 con 0x7f6b28072250 2023-03-03T13:09:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.282+0000 7f6b2d5c3700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b0c0682f0 0x7f6b0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:45.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.282+0000 7f6b2d5c3700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b0c0682f0 0x7f6b0c06a7a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400be30 tx=0x7f6b24013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.286+0000 7f6b19ffb700 1 -- 172.21.15.7:0/1431025399 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b1c0527a0 con 0x7f6b28072250 2023-03-03T13:09:45.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.526+0000 7f6b2f827700 1 -- 172.21.15.7:0/1431025399 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b2806cb30 con 0x7f6b28072250 2023-03-03T13:09:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.526+0000 7f6b19ffb700 1 -- 172.21.15.7:0/1431025399 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6b1c055dc0 con 0x7f6b28072250 2023-03-03T13:09:45.528 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:09:45.529 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:09:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.528+0000 7f6b2f827700 1 -- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b0c0682f0 msgr2=0x7f6b0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:45.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.528+0000 7f6b2f827700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b0c0682f0 0x7f6b0c06a7a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400be30 tx=0x7f6b24013000 comp rx=0 tx=0).stop 2023-03-03T13:09:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.529+0000 7f6b2f827700 1 -- 172.21.15.7:0/1431025399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b28072250 msgr2=0x7f6b280ac690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.529+0000 7f6b2f827700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b28072250 0x7f6b280ac690 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00ebb0 tx=0x7f6b1c00eec0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.529+0000 7f6b2f827700 1 -- 172.21.15.7:0/1431025399 shutdown_connections 2023-03-03T13:09:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.529+0000 7f6b2f827700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b280c1ce0 0x7f6b280c52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.529+0000 7f6b2f827700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b28071e80 0x7f6b281e9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.529+0000 7f6b2f827700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b0c0682f0 0x7f6b0c06a7a0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.529+0000 7f6b2f827700 1 --2- 172.21.15.7:0/1431025399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b28072250 0x7f6b280ac690 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:45.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.529+0000 7f6b2f827700 1 -- 172.21.15.7:0/1431025399 >> 172.21.15.7:0/1431025399 conn(0x7f6b280a94d0 msgr2=0x7f6b281474e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:45.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.529+0000 7f6b2f827700 1 -- 172.21.15.7:0/1431025399 shutdown_connections 2023-03-03T13:09:45.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:45.529+0000 7f6b2f827700 1 -- 172.21.15.7:0/1431025399 wait complete. 2023-03-03T13:09:45.532 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:09:45.542 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before thrashing 2023-03-03T13:09:48.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:09:48.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.591+0000 7f31109fc700 1 Processor -- start 2023-03-03T13:09:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.591+0000 7f31109fc700 1 -- start start 2023-03-03T13:09:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.592+0000 7f31109fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f310c1582b0 0x7f310c0c1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.592+0000 7f31109fc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c06e2d0 con 0x7f310c0c1500 2023-03-03T13:09:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.592+0000 7f31109fc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0c53b0 con 0x7f310c151e70 2023-03-03T13:09:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.592+0000 7f31109fc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310c0c2720 con 0x7f310c1582b0 2023-03-03T13:09:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.592+0000 7f3109d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f310c0c1500 0x7f310c157ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37450/0 (socket says 172.21.15.7:37450) 2023-03-03T13:09:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.592+0000 7f3109d9b700 1 -- 172.21.15.7:0/917992646 learned_addr learned my addr 172.21.15.7:0/917992646 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:09:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.592+0000 7f310ad9d700 1 --2- 172.21.15.7:0/917992646 >> v2:172.21.15.143:3300/0 conn(0x7f310c1582b0 0x7f310c0c1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.592+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063316040 0 0) 0x7f310c06e2d0 con 0x7f310c0c1500 2023-03-03T13:09:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.592+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f40035b0 con 0x7f310c0c1500 2023-03-03T13:09:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 776101940 0 0) 0x7f30f40035b0 con 0x7f310c0c1500 2023-03-03T13:09:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 >> v2:172.21.15.143:3300/0 conn(0x7f310c1582b0 msgr2=0x7f310c0c1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:09:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 --2- 172.21.15.7:0/917992646 >> v2:172.21.15.143:3300/0 conn(0x7f310c1582b0 0x7f310c0c1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 >> v1:172.21.15.143:6789/0 conn(0x7f310c151e70 legacy=0x7f310c06a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c06da30 con 0x7f310c0c1500 2023-03-03T13:09:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f30fc001670 con 0x7f310c0c1500 2023-03-03T13:09:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 >> v1:172.21.15.7:6789/0 conn(0x7f310c0c1500 legacy=0x7f310c157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f30f4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30f4004850 0x7f30f4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30f4005200 0x7f30f400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f400c0e0 con 0x7f30f4003a70 2023-03-03T13:09:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f400c250 con 0x7f30f4005200 2023-03-03T13:09:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f400c410 con 0x7f30f4004850 2023-03-03T13:09:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310a59c700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f30f4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f3109d9b700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30f4004850 0x7f30f4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310ad9d700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30f4005200 0x7f30f400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310a59c700 1 -- 172.21.15.7:0/917992646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30f4004850 msgr2=0x7f30f4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:09:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310a59c700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30f4004850 0x7f30f4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310a59c700 1 -- 172.21.15.7:0/917992646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30f4005200 msgr2=0x7f30f400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:09:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310a59c700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30f4005200 0x7f30f400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310a59c700 1 -- 172.21.15.7:0/917992646 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f40035b0 con 0x7f30f4003a70 2023-03-03T13:09:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310ad9d700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30f4005200 0x7f30f400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:09:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310ad9d700 1 -- 172.21.15.7:0/917992646 reap_dead start 2023-03-03T13:09:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.593+0000 7f310a59c700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f30f4003e40 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f3100001880 tx=0x7f30f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc92e889cf793839 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.594+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100001370 con 0x7f30f4003a70 2023-03-03T13:09:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.594+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31000052f0 con 0x7f30f4003a70 2023-03-03T13:09:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.594+0000 7f310959a700 1 -- 172.21.15.7:0/917992646 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100003680 con 0x7f30f4003a70 2023-03-03T13:09:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.594+0000 7f31109fc700 1 -- 172.21.15.7:0/917992646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 msgr2=0x7f30f4003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.594+0000 7f31109fc700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f30f4003e40 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f3100001880 tx=0x7f30f4005880 comp rx=0 tx=0).stop 2023-03-03T13:09:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.594+0000 7f31109fc700 1 -- 172.21.15.7:0/917992646 shutdown_connections 2023-03-03T13:09:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.594+0000 7f31109fc700 1 --2- 172.21.15.7:0/917992646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f30f4003e40 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.594+0000 7f31109fc700 1 -- 172.21.15.7:0/917992646 >> 172.21.15.7:0/917992646 conn(0x7f310c0a9430 msgr2=0x7f310c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.594+0000 7f31109fc700 1 -- 172.21.15.7:0/917992646 shutdown_connections 2023-03-03T13:09:48.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.594+0000 7f31109fc700 1 -- 172.21.15.7:0/917992646 wait complete. 2023-03-03T13:09:48.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f31109fc700 1 Processor -- start 2023-03-03T13:09:48.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f31109fc700 1 -- start start 2023-03-03T13:09:48.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f31109fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f310c072320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:48.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f31109fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c1578c0 0x7f310c0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:48.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f31109fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f310c14e130 0x7f310c0717f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:48.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f31109fc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c071d30 con 0x7f30f4003a70 2023-03-03T13:09:48.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f31109fc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c155770 con 0x7f310c14e130 2023-03-03T13:09:48.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f31109fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c1e9cc0 con 0x7f310c1578c0 2023-03-03T13:09:48.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f310a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f310c072320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:48.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f310a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f310c072320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43482/0 (socket says 172.21.15.7:43482) 2023-03-03T13:09:48.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f310a59c700 1 -- 172.21.15.7:0/2954520256 learned_addr learned my addr 172.21.15.7:0/2954520256 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:09:48.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f3109d9b700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c1578c0 0x7f310c0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:48.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.596+0000 7f310ad9d700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f310c14e130 0x7f310c0717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:48.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f310a59c700 1 -- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c1578c0 msgr2=0x7f310c0712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:48.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f310a59c700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c1578c0 0x7f310c0712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f310a59c700 1 -- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f310c14e130 msgr2=0x7f310c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f310a59c700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f310c14e130 0x7f310c0717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f310a59c700 1 -- 172.21.15.7:0/2954520256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c06dff0 con 0x7f30f4003a70 2023-03-03T13:09:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f310a59c700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f310c072320 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f31000011a0 tx=0x7f3100001110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f30faffd700 1 -- 172.21.15.7:0/2954520256 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100005460 con 0x7f30f4003a70 2023-03-03T13:09:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f30faffd700 1 -- 172.21.15.7:0/2954520256 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3100006860 con 0x7f30f4003a70 2023-03-03T13:09:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f30faffd700 1 -- 172.21.15.7:0/2954520256 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3100006c10 con 0x7f30f4003a70 2023-03-03T13:09:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f31109fc700 1 -- 172.21.15.7:0/2954520256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310c06dd10 con 0x7f30f4003a70 2023-03-03T13:09:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.597+0000 7f31109fc700 1 -- 172.21.15.7:0/2954520256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310c0c4bb0 con 0x7f30f4003a70 2023-03-03T13:09:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.598+0000 7f31109fc700 1 -- 172.21.15.7:0/2954520256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310c1558e0 con 0x7f30f4003a70 2023-03-03T13:09:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.599+0000 7f30faffd700 1 -- 172.21.15.7:0/2954520256 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3100010070 con 0x7f30f4003a70 2023-03-03T13:09:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.599+0000 7f30faffd700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30ec068320 0x7f30ec06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.599+0000 7f30faffd700 1 -- 172.21.15.7:0/2954520256 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3100086880 con 0x7f30f4003a70 2023-03-03T13:09:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.599+0000 7f3109d9b700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30ec068320 0x7f30ec06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.600+0000 7f3109d9b700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30ec068320 0x7f30ec06a7d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f310c150c20 tx=0x7f30fc007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.603+0000 7f30faffd700 1 -- 172.21.15.7:0/2954520256 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3100052c20 con 0x7f30f4003a70 2023-03-03T13:09:48.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.845+0000 7f31109fc700 1 -- 172.21.15.7:0/2954520256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f310c074bb0 con 0x7f30f4003a70 2023-03-03T13:09:48.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.845+0000 7f30faffd700 1 -- 172.21.15.7:0/2954520256 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3100056240 con 0x7f30f4003a70 2023-03-03T13:09:48.847 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:09:48.848 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:09:48.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 -- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30ec068320 msgr2=0x7f30ec06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30ec068320 0x7f30ec06a7d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f310c150c20 tx=0x7f30fc007040 comp rx=0 tx=0).stop 2023-03-03T13:09:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 -- 172.21.15.7:0/2954520256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 msgr2=0x7f310c072320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f310c072320 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f31000011a0 tx=0x7f3100001110 comp rx=0 tx=0).stop 2023-03-03T13:09:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 -- 172.21.15.7:0/2954520256 shutdown_connections 2023-03-03T13:09:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f310c14e130 0x7f310c0717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30ec068320 0x7f30ec06a7d0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:48.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30f4003a70 0x7f310c072320 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:48.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 --2- 172.21.15.7:0/2954520256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f310c1578c0 0x7f310c0712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:48.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 -- 172.21.15.7:0/2954520256 >> 172.21.15.7:0/2954520256 conn(0x7f310c0a9430 msgr2=0x7f310c14fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:48.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 -- 172.21.15.7:0/2954520256 shutdown_connections 2023-03-03T13:09:48.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:48.848+0000 7f31109fc700 1 -- 172.21.15.7:0/2954520256 wait complete. 2023-03-03T13:09:48.851 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:09:48.861 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T13:09:53.544 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:09:53.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.681+0000 7f67a57d7700 1 Processor -- start 2023-03-03T13:09:53.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.681+0000 7f67a57d7700 1 -- start start 2023-03-03T13:09:53.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f67a57d7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f67a00c2e30 0x7f67a0158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:53.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f67a57d7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a0157ff0 con 0x7f67a0151cd0 2023-03-03T13:09:53.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f67a57d7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a006d640 con 0x7f67a006c6c0 2023-03-03T13:09:53.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f67a57d7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a006cda0 con 0x7f67a00c2e30 2023-03-03T13:09:53.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f679e7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f67a00c2e30 0x7f67a0158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:53.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f679effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f67a0151cd0 0x7f67a0154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37470/0 (socket says 172.21.15.7:37470) 2023-03-03T13:09:53.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f679effd700 1 -- 172.21.15.7:0/2780801030 learned_addr learned my addr 172.21.15.7:0/2780801030 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:09:53.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f679dffb700 1 -- 172.21.15.7:0/2780801030 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223662939 0 0) 0x7f67a0157ff0 con 0x7f67a0151cd0 2023-03-03T13:09:53.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f679dffb700 1 -- 172.21.15.7:0/2780801030 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67880035b0 con 0x7f67a0151cd0 2023-03-03T13:09:53.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f679e7fc700 1 -- 172.21.15.7:0/2780801030 >> v1:172.21.15.143:6789/0 conn(0x7f67a006c6c0 legacy=0x7f67a00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:53.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f679e7fc700 1 -- 172.21.15.7:0/2780801030 >> v1:172.21.15.7:6789/0 conn(0x7f67a0151cd0 legacy=0x7f67a0154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:53.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f679e7fc700 1 -- 172.21.15.7:0/2780801030 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a006d080 con 0x7f67a00c2e30 2023-03-03T13:09:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.682+0000 7f679e7fc700 1 --2- 172.21.15.7:0/2780801030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a00c2e30 0x7f67a0158480 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f6794009840 tx=0x7f6794009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c6591002eb3f49 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.683+0000 7f679dffb700 1 -- 172.21.15.7:0/2780801030 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794011830 con 0x7f67a00c2e30 2023-03-03T13:09:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.683+0000 7f679dffb700 1 -- 172.21.15.7:0/2780801030 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6794011990 con 0x7f67a00c2e30 2023-03-03T13:09:53.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.683+0000 7f679dffb700 1 -- 172.21.15.7:0/2780801030 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794011d20 con 0x7f67a00c2e30 2023-03-03T13:09:53.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.683+0000 7f67a57d7700 1 -- 172.21.15.7:0/2780801030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a00c2e30 msgr2=0x7f67a0158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:53.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.683+0000 7f67a57d7700 1 --2- 172.21.15.7:0/2780801030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a00c2e30 0x7f67a0158480 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f6794009840 tx=0x7f6794009b50 comp rx=0 tx=0).stop 2023-03-03T13:09:53.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.683+0000 7f67a57d7700 1 -- 172.21.15.7:0/2780801030 shutdown_connections 2023-03-03T13:09:53.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.683+0000 7f67a57d7700 1 --2- 172.21.15.7:0/2780801030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a00c2e30 0x7f67a0158480 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:53.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.683+0000 7f67a57d7700 1 -- 172.21.15.7:0/2780801030 >> 172.21.15.7:0/2780801030 conn(0x7f67a00a94d0 msgr2=0x7f67a00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:53.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.683+0000 7f67a57d7700 1 -- 172.21.15.7:0/2780801030 shutdown_connections 2023-03-03T13:09:53.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.683+0000 7f67a57d7700 1 -- 172.21.15.7:0/2780801030 wait complete. 2023-03-03T13:09:53.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.684+0000 7f67a57d7700 1 Processor -- start 2023-03-03T13:09:53.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f67a57d7700 1 -- start start 2023-03-03T13:09:53.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f67a57d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67a006c6c0 0x7f67a00c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:53.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f67a57d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67a00c2e30 0x7f67a00aa470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:53.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f67a57d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a0151cd0 0x7f67a014d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:53.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f67a57d7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a00aceb0 con 0x7f67a0151cd0 2023-03-03T13:09:53.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f67a57d7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a01ed4b0 con 0x7f67a006c6c0 2023-03-03T13:09:53.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f67a57d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a014e220 con 0x7f67a00c2e30 2023-03-03T13:09:53.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f679f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a0151cd0 0x7f67a014d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:53.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f679f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a0151cd0 0x7f67a014d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43500/0 (socket says 172.21.15.7:43500) 2023-03-03T13:09:53.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f679f7fe700 1 -- 172.21.15.7:0/328152378 learned_addr learned my addr 172.21.15.7:0/328152378 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:09:53.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f679effd700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67a006c6c0 0x7f67a00c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:53.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.685+0000 7f679e7fc700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67a00c2e30 0x7f67a00aa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:53.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f679f7fe700 1 -- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67a00c2e30 msgr2=0x7f67a00aa470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:53.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f679f7fe700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67a00c2e30 0x7f67a00aa470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:53.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f679f7fe700 1 -- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67a006c6c0 msgr2=0x7f67a00c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:53.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f679f7fe700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67a006c6c0 0x7f67a00c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:53.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f679f7fe700 1 -- 172.21.15.7:0/328152378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67940094a0 con 0x7f67a0151cd0 2023-03-03T13:09:53.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f679effd700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67a006c6c0 0x7f67a00c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:09:53.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f679f7fe700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a0151cd0 0x7f67a014d440 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f679800cba0 tx=0x7f679800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:53.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f67877fe700 1 -- 172.21.15.7:0/328152378 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6798003c00 con 0x7f67a0151cd0 2023-03-03T13:09:53.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f67877fe700 1 -- 172.21.15.7:0/328152378 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6798003d60 con 0x7f67a0151cd0 2023-03-03T13:09:53.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f67a57d7700 1 -- 172.21.15.7:0/328152378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a0157c80 con 0x7f67a0151cd0 2023-03-03T13:09:53.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f67877fe700 1 -- 172.21.15.7:0/328152378 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679800aba0 con 0x7f67a0151cd0 2023-03-03T13:09:53.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.686+0000 7f67a57d7700 1 -- 172.21.15.7:0/328152378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a006d360 con 0x7f67a0151cd0 2023-03-03T13:09:53.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.687+0000 7f67a57d7700 1 -- 172.21.15.7:0/328152378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a01589c0 con 0x7f67a0151cd0 2023-03-03T13:09:53.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.687+0000 7f67877fe700 1 -- 172.21.15.7:0/328152378 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6798010050 con 0x7f67a0151cd0 2023-03-03T13:09:53.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.688+0000 7f67877fe700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f678c0682f0 0x7f678c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:53.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.688+0000 7f67877fe700 1 -- 172.21.15.7:0/328152378 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6798086070 con 0x7f67a0151cd0 2023-03-03T13:09:53.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.688+0000 7f679effd700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f678c0682f0 0x7f678c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:53.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.689+0000 7f679effd700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f678c0682f0 0x7f678c06a7a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f67900011e0 tx=0x7f6790009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:53.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.692+0000 7f67877fe700 1 -- 172.21.15.7:0/328152378 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f67980523b0 con 0x7f67a0151cd0 2023-03-03T13:09:53.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.935+0000 7f67a57d7700 1 -- 172.21.15.7:0/328152378 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67a00ad440 con 0x7f67a0151cd0 2023-03-03T13:09:53.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.935+0000 7f67877fe700 1 -- 172.21.15.7:0/328152378 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f67980559d0 con 0x7f67a0151cd0 2023-03-03T13:09:53.937 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:09:53.937 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:09:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 -- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f678c0682f0 msgr2=0x7f678c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f678c0682f0 0x7f678c06a7a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f67900011e0 tx=0x7f6790009040 comp rx=0 tx=0).stop 2023-03-03T13:09:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 -- 172.21.15.7:0/328152378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a0151cd0 msgr2=0x7f67a014d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a0151cd0 0x7f67a014d440 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f679800cba0 tx=0x7f679800cf60 comp rx=0 tx=0).stop 2023-03-03T13:09:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 -- 172.21.15.7:0/328152378 shutdown_connections 2023-03-03T13:09:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f678c0682f0 0x7f678c06a7a0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67a0151cd0 0x7f67a014d440 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67a006c6c0 0x7f67a00c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 --2- 172.21.15.7:0/328152378 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67a00c2e30 0x7f67a00aa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 -- 172.21.15.7:0/328152378 >> 172.21.15.7:0/328152378 conn(0x7f67a00a94d0 msgr2=0x7f67a0070fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 -- 172.21.15.7:0/328152378 shutdown_connections 2023-03-03T13:09:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:53.937+0000 7f67a57d7700 1 -- 172.21.15.7:0/328152378 wait complete. 2023-03-03T13:09:53.941 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:09:53.951 INFO:tasks.mds_thrash.fs.[b]:waiting for 70 secs before thrashing 2023-03-03T13:09:58.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:09:59.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.000+0000 7f5456035700 1 Processor -- start 2023-03-03T13:09:59.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.000+0000 7f5456035700 1 -- start start 2023-03-03T13:09:59.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f5456035700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54500a6370 0x7f545006a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:59.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f5456035700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54501558f0 con 0x7f54500c5cb0 2023-03-03T13:09:59.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f5456035700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500a6180 con 0x7f545006a2b0 2023-03-03T13:09:59.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f5456035700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f545006bab0 con 0x7f54500a6370 2023-03-03T13:09:59.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f544effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54500a6370 0x7f545006a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:59.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f544ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f54500c5cb0 0x7f545015cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37488/0 (socket says 172.21.15.7:37488) 2023-03-03T13:09:59.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f544ffff700 1 -- 172.21.15.7:0/3595356560 learned_addr learned my addr 172.21.15.7:0/3595356560 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:09:59.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f544effd700 1 -- 172.21.15.7:0/3595356560 >> v1:172.21.15.143:6789/0 conn(0x7f545006a2b0 legacy=0x7f5450155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:59.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f544effd700 1 -- 172.21.15.7:0/3595356560 >> v1:172.21.15.7:6789/0 conn(0x7f54500c5cb0 legacy=0x7f545015cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:59.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f544effd700 1 -- 172.21.15.7:0/3595356560 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500a5ea0 con 0x7f54500a6370 2023-03-03T13:09:59.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.001+0000 7f544e7fc700 1 -- 172.21.15.7:0/3595356560 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272073054 0 0) 0x7f54501558f0 con 0x7f54500c5cb0 2023-03-03T13:09:59.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f544effd700 1 --2- 172.21.15.7:0/3595356560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 0x7f545006a680 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f5440009840 tx=0x7f5440009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8885601636bb5ff0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:59.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f544e7fc700 1 -- 172.21.15.7:0/3595356560 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440011830 con 0x7f54500a6370 2023-03-03T13:09:59.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f544e7fc700 1 -- 172.21.15.7:0/3595356560 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5440011990 con 0x7f54500a6370 2023-03-03T13:09:59.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f544e7fc700 1 -- 172.21.15.7:0/3595356560 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5440011d20 con 0x7f54500a6370 2023-03-03T13:09:59.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f5456035700 1 -- 172.21.15.7:0/3595356560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 msgr2=0x7f545006a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:59.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f5456035700 1 --2- 172.21.15.7:0/3595356560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 0x7f545006a680 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f5440009840 tx=0x7f5440009b50 comp rx=0 tx=0).stop 2023-03-03T13:09:59.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f5456035700 1 -- 172.21.15.7:0/3595356560 shutdown_connections 2023-03-03T13:09:59.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f5456035700 1 --2- 172.21.15.7:0/3595356560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 0x7f545006a680 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:59.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f5456035700 1 -- 172.21.15.7:0/3595356560 >> 172.21.15.7:0/3595356560 conn(0x7f54500a9430 msgr2=0x7f54500c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:59.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f5456035700 1 -- 172.21.15.7:0/3595356560 shutdown_connections 2023-03-03T13:09:59.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.002+0000 7f5456035700 1 -- 172.21.15.7:0/3595356560 wait complete. 2023-03-03T13:09:59.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.003+0000 7f5456035700 1 Processor -- start 2023-03-03T13:09:59.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.004+0000 7f5456035700 1 -- start start 2023-03-03T13:09:59.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.004+0000 7f5456035700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545006a2b0 0x7f54501db8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:59.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.004+0000 7f5456035700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 0x7f545014a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:59.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.004+0000 7f5456035700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54500c5cb0 0x7f54500c2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:59.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.004+0000 7f5456035700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54501dd950 con 0x7f54500a6370 2023-03-03T13:09:59.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.004+0000 7f5456035700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545014aec0 con 0x7f54500c5cb0 2023-03-03T13:09:59.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.004+0000 7f5456035700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54501ddbf0 con 0x7f545006a2b0 2023-03-03T13:09:59.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.004+0000 7f544effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 0x7f545014a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:59.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.004+0000 7f544effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 0x7f545014a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43510/0 (socket says 172.21.15.7:43510) 2023-03-03T13:09:59.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f544effd700 1 -- 172.21.15.7:0/3560550113 learned_addr learned my addr 172.21.15.7:0/3560550113 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:09:59.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f544ffff700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54500c5cb0 0x7f54500c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:59.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f544f7fe700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545006a2b0 0x7f54501db8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:59.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f544effd700 1 -- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545006a2b0 msgr2=0x7f54501db8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:59.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f544effd700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545006a2b0 0x7f54501db8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:59.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f544effd700 1 -- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54500c5cb0 msgr2=0x7f54500c2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:59.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f544effd700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54500c5cb0 0x7f54500c2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:59.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f544effd700 1 -- 172.21.15.7:0/3560550113 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54400094a0 con 0x7f54500a6370 2023-03-03T13:09:59.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f544effd700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 0x7f545014a980 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f54400090b0 tx=0x7f5440004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:59.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f5437fff700 1 -- 172.21.15.7:0/3560550113 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544001a7e0 con 0x7f54500a6370 2023-03-03T13:09:59.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f5437fff700 1 -- 172.21.15.7:0/3560550113 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544001a940 con 0x7f54500a6370 2023-03-03T13:09:59.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f5456035700 1 -- 172.21.15.7:0/3560550113 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5450153160 con 0x7f54500a6370 2023-03-03T13:09:59.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f5437fff700 1 -- 172.21.15.7:0/3560550113 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544001acb0 con 0x7f54500a6370 2023-03-03T13:09:59.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.005+0000 7f5456035700 1 -- 172.21.15.7:0/3560550113 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5450152e80 con 0x7f54500a6370 2023-03-03T13:09:59.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.006+0000 7f5456035700 1 -- 172.21.15.7:0/3560550113 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54500c1500 con 0x7f54500a6370 2023-03-03T13:09:59.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.007+0000 7f5437fff700 1 -- 172.21.15.7:0/3560550113 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f544002b030 con 0x7f54500a6370 2023-03-03T13:09:59.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.007+0000 7f5437fff700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f543c0682f0 0x7f543c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:09:59.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.007+0000 7f5437fff700 1 -- 172.21.15.7:0/3560550113 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5440087360 con 0x7f54500a6370 2023-03-03T13:09:59.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.007+0000 7f544f7fe700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f543c0682f0 0x7f543c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:09:59.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.008+0000 7f544f7fe700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f543c0682f0 0x7f543c06a7a0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f5444007fd0 tx=0x7f5444009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:09:59.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.012+0000 7f5437fff700 1 -- 172.21.15.7:0/3560550113 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f54400537b0 con 0x7f54500a6370 2023-03-03T13:09:59.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.250+0000 7f5456035700 1 -- 172.21.15.7:0/3560550113 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5450074bb0 con 0x7f54500a6370 2023-03-03T13:09:59.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.251+0000 7f5437fff700 1 -- 172.21.15.7:0/3560550113 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5440015090 con 0x7f54500a6370 2023-03-03T13:09:59.253 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:09:59.253 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:09:59.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 -- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f543c0682f0 msgr2=0x7f543c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:59.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f543c0682f0 0x7f543c06a7a0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f5444007fd0 tx=0x7f5444009040 comp rx=0 tx=0).stop 2023-03-03T13:09:59.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 -- 172.21.15.7:0/3560550113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 msgr2=0x7f545014a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:09:59.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 0x7f545014a980 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f54400090b0 tx=0x7f5440004750 comp rx=0 tx=0).stop 2023-03-03T13:09:59.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 -- 172.21.15.7:0/3560550113 shutdown_connections 2023-03-03T13:09:59.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54500c5cb0 0x7f54500c2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:59.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f545006a2b0 0x7f54501db8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:59.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f543c0682f0 0x7f543c06a7a0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 --2- 172.21.15.7:0/3560550113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54500a6370 0x7f545014a980 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:09:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 -- 172.21.15.7:0/3560550113 >> 172.21.15.7:0/3560550113 conn(0x7f54500a9430 msgr2=0x7f54500a78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:09:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 -- 172.21.15.7:0/3560550113 shutdown_connections 2023-03-03T13:09:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:09:59.253+0000 7f5456035700 1 -- 172.21.15.7:0/3560550113 wait complete. 2023-03-03T13:09:59.256 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:09:59.266 INFO:tasks.mds_thrash.fs.[a]:waiting for 21 secs before thrashing 2023-03-03T13:10:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T13:09:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T13:10:08.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:08.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:20.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:10:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.405+0000 7fab64c23700 1 Processor -- start 2023-03-03T13:10:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.405+0000 7fab64c23700 1 -- start start 2023-03-03T13:10:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab64c23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fab60151ab0 0x7fab600736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab64c23700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab60073c30 0x7fab6006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab64c23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab6006c3e0 0x7fab60157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab64c23700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6006d780 con 0x7fab60073c30 2023-03-03T13:10:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab64c23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6006cb70 con 0x7fab60151ab0 2023-03-03T13:10:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab64c23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6006ce50 con 0x7fab6006c3e0 2023-03-03T13:10:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab5dd9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab60073c30 0x7fab6006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab5dd9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab60073c30 0x7fab6006bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43548/0 (socket says 172.21.15.7:43548) 2023-03-03T13:10:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab5dd9b700 1 -- 172.21.15.7:0/4178301874 learned_addr learned my addr 172.21.15.7:0/4178301874 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:10:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab5e59c700 1 --2- 172.21.15.7:0/4178301874 >> v2:172.21.15.143:3300/0 conn(0x7fab60151ab0 0x7fab600736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.406+0000 7fab5ed9d700 1 --2- 172.21.15.7:0/4178301874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab6006c3e0 0x7fab60157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.407+0000 7fab5dd9b700 1 -- 172.21.15.7:0/4178301874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab6006c3e0 msgr2=0x7fab60157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:10:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.407+0000 7fab5dd9b700 1 --2- 172.21.15.7:0/4178301874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab6006c3e0 0x7fab60157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.407+0000 7fab5dd9b700 1 -- 172.21.15.7:0/4178301874 >> v2:172.21.15.143:3300/0 conn(0x7fab60151ab0 msgr2=0x7fab600736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:10:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.407+0000 7fab5dd9b700 1 --2- 172.21.15.7:0/4178301874 >> v2:172.21.15.143:3300/0 conn(0x7fab60151ab0 0x7fab600736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.407+0000 7fab5dd9b700 1 -- 172.21.15.7:0/4178301874 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6006d130 con 0x7fab60073c30 2023-03-03T13:10:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.407+0000 7fab5dd9b700 1 --2- 172.21.15.7:0/4178301874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60073c30 0x7fab6006bea0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fab6006bb20 tx=0x7fab48009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa93f09922e5ec0a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.407+0000 7fab5d59a700 1 -- 172.21.15.7:0/4178301874 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4800b870 con 0x7fab60073c30 2023-03-03T13:10:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.407+0000 7fab5d59a700 1 -- 172.21.15.7:0/4178301874 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4800b9d0 con 0x7fab60073c30 2023-03-03T13:10:20.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.407+0000 7fab5d59a700 1 -- 172.21.15.7:0/4178301874 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4800bd60 con 0x7fab60073c30 2023-03-03T13:10:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.408+0000 7fab64c23700 1 -- 172.21.15.7:0/4178301874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60073c30 msgr2=0x7fab6006bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.408+0000 7fab64c23700 1 --2- 172.21.15.7:0/4178301874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60073c30 0x7fab6006bea0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fab6006bb20 tx=0x7fab48009bd0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.408+0000 7fab64c23700 1 -- 172.21.15.7:0/4178301874 shutdown_connections 2023-03-03T13:10:20.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.408+0000 7fab64c23700 1 --2- 172.21.15.7:0/4178301874 >> v2:172.21.15.143:3300/0 conn(0x7fab60151ab0 0x7fab600736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.408+0000 7fab64c23700 1 --2- 172.21.15.7:0/4178301874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab6006c3e0 0x7fab60157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.408+0000 7fab64c23700 1 --2- 172.21.15.7:0/4178301874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60073c30 0x7fab6006bea0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.408+0000 7fab64c23700 1 -- 172.21.15.7:0/4178301874 >> 172.21.15.7:0/4178301874 conn(0x7fab600a94d0 msgr2=0x7fab600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:10:20.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.408+0000 7fab64c23700 1 -- 172.21.15.7:0/4178301874 shutdown_connections 2023-03-03T13:10:20.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.408+0000 7fab64c23700 1 -- 172.21.15.7:0/4178301874 wait complete. 2023-03-03T13:10:20.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.409+0000 7fab64c23700 1 Processor -- start 2023-03-03T13:10:20.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.409+0000 7fab64c23700 1 -- start start 2023-03-03T13:10:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab64c23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab6006c3e0 0x7fab601ed040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab64c23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab60073c30 0x7fab60146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab64c23700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60151ab0 0x7fab60146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab64c23700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab601eaf40 con 0x7fab60151ab0 2023-03-03T13:10:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab64c23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab60146b90 con 0x7fab6006c3e0 2023-03-03T13:10:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab64c23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab600be430 con 0x7fab60073c30 2023-03-03T13:10:20.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60151ab0 0x7fab60146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60151ab0 0x7fab60146650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43556/0 (socket says 172.21.15.7:43556) 2023-03-03T13:10:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5ed9d700 1 -- 172.21.15.7:0/902421126 learned_addr learned my addr 172.21.15.7:0/902421126 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:10:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5e59c700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab6006c3e0 0x7fab601ed040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5dd9b700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab60073c30 0x7fab60146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5ed9d700 1 -- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab60073c30 msgr2=0x7fab60146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5ed9d700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab60073c30 0x7fab60146110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5ed9d700 1 -- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab6006c3e0 msgr2=0x7fab601ed040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:20.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5ed9d700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab6006c3e0 0x7fab601ed040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5ed9d700 1 -- 172.21.15.7:0/902421126 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab48009550 con 0x7fab60151ab0 2023-03-03T13:10:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.410+0000 7fab5dd9b700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab60073c30 0x7fab60146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:10:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.411+0000 7fab5ed9d700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60151ab0 0x7fab60146650 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fab58007e40 tx=0x7fab5800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.411+0000 7fab56ffd700 1 -- 172.21.15.7:0/902421126 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5800eea0 con 0x7fab60151ab0 2023-03-03T13:10:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.411+0000 7fab56ffd700 1 -- 172.21.15.7:0/902421126 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab58014420 con 0x7fab60151ab0 2023-03-03T13:10:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.411+0000 7fab64c23700 1 -- 172.21.15.7:0/902421126 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab60151e80 con 0x7fab60151ab0 2023-03-03T13:10:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.411+0000 7fab56ffd700 1 -- 172.21.15.7:0/902421126 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab580147b0 con 0x7fab60151ab0 2023-03-03T13:10:20.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.411+0000 7fab64c23700 1 -- 172.21.15.7:0/902421126 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6006d410 con 0x7fab60151ab0 2023-03-03T13:10:20.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.412+0000 7fab64c23700 1 -- 172.21.15.7:0/902421126 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab60152010 con 0x7fab60151ab0 2023-03-03T13:10:20.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.412+0000 7fab56ffd700 1 -- 172.21.15.7:0/902421126 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fab58012050 con 0x7fab60151ab0 2023-03-03T13:10:20.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.413+0000 7fab56ffd700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab400682f0 0x7fab4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:10:20.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.413+0000 7fab56ffd700 1 -- 172.21.15.7:0/902421126 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fab580867f0 con 0x7fab60151ab0 2023-03-03T13:10:20.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.413+0000 7fab5e59c700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab400682f0 0x7fab4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:10:20.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.413+0000 7fab5e59c700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab400682f0 0x7fab4006a7a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fab60155ea0 tx=0x7fab50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:10:20.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.417+0000 7fab56ffd700 1 -- 172.21.15.7:0/902421126 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab58052b90 con 0x7fab60151ab0 2023-03-03T13:10:20.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.653+0000 7fab64c23700 1 -- 172.21.15.7:0/902421126 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab601eb0f0 con 0x7fab60151ab0 2023-03-03T13:10:20.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.654+0000 7fab56ffd700 1 -- 172.21.15.7:0/902421126 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fab601eb0f0 con 0x7fab60151ab0 2023-03-03T13:10:20.657 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:10:20.657 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:10:20.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 -- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab400682f0 msgr2=0x7fab4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:20.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab400682f0 0x7fab4006a7a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fab60155ea0 tx=0x7fab50008040 comp rx=0 tx=0).stop 2023-03-03T13:10:20.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 -- 172.21.15.7:0/902421126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60151ab0 msgr2=0x7fab60146650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:10:20.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60151ab0 0x7fab60146650 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fab58007e40 tx=0x7fab5800e3c0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 -- 172.21.15.7:0/902421126 shutdown_connections 2023-03-03T13:10:20.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab400682f0 0x7fab4006a7a0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab60151ab0 0x7fab60146650 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab6006c3e0 0x7fab601ed040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 --2- 172.21.15.7:0/902421126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab60073c30 0x7fab60146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:10:20.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 -- 172.21.15.7:0/902421126 >> 172.21.15.7:0/902421126 conn(0x7fab600a94d0 msgr2=0x7fab600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:10:20.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 -- 172.21.15.7:0/902421126 shutdown_connections 2023-03-03T13:10:20.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:10:20.656+0000 7fab64c23700 1 -- 172.21.15.7:0/902421126 wait complete. 2023-03-03T13:10:20.660 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:10:20.670 INFO:tasks.mds_thrash.fs.[a]:waiting for 54 secs before thrashing 2023-03-03T13:10:38.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:10:38.138 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:03.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:11:04.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.090+0000 7fd37041b700 1 Processor -- start 2023-03-03T13:11:04.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.090+0000 7fd37041b700 1 -- start start 2023-03-03T13:11:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd37041b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd36814fda0 0x7fd368156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd37041b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3680abe20 0x7fd3681529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:04.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd37041b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3680c2f30 con 0x7fd36814f9d0 2023-03-03T13:11:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd37041b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3680c31d0 con 0x7fd36814fda0 2023-03-03T13:11:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd37041b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3681524a0 con 0x7fd3680abe20 2023-03-03T13:11:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd36d9b6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd36814fda0 0x7fd368156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd36d9b6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd36814fda0 0x7fd368156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43590/0 (socket says 172.21.15.7:43590) 2023-03-03T13:11:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd36d9b6700 1 -- 172.21.15.7:0/3048075347 learned_addr learned my addr 172.21.15.7:0/3048075347 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:11:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd36d9b6700 1 -- 172.21.15.7:0/3048075347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3680abe20 msgr2=0x7fd3681529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:11:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd36e9b8700 1 --2- 172.21.15.7:0/3048075347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3680abe20 0x7fd3681529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:04.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd36d9b6700 1 --2- 172.21.15.7:0/3048075347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3680abe20 0x7fd3681529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd36d9b6700 1 -- 172.21.15.7:0/3048075347 >> v1:172.21.15.143:6789/0 conn(0x7fd36814f9d0 legacy=0x7fd368071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:11:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.091+0000 7fd36d9b6700 1 -- 172.21.15.7:0/3048075347 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3680c34b0 con 0x7fd36814fda0 2023-03-03T13:11:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd36d9b6700 1 --2- 172.21.15.7:0/3048075347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd36814fda0 0x7fd368156430 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fd3580098f0 tx=0x7fd358009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8106dfc0d07e7f09 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd36d1b5700 1 -- 172.21.15.7:0/3048075347 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd358004230 con 0x7fd36814fda0 2023-03-03T13:11:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd36d1b5700 1 -- 172.21.15.7:0/3048075347 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd358004390 con 0x7fd36814fda0 2023-03-03T13:11:04.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd36d1b5700 1 -- 172.21.15.7:0/3048075347 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3580036a0 con 0x7fd36814fda0 2023-03-03T13:11:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd37041b700 1 -- 172.21.15.7:0/3048075347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd36814fda0 msgr2=0x7fd368156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:04.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd37041b700 1 --2- 172.21.15.7:0/3048075347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd36814fda0 0x7fd368156430 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fd3580098f0 tx=0x7fd358009c00 comp rx=0 tx=0).stop 2023-03-03T13:11:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd37041b700 1 -- 172.21.15.7:0/3048075347 shutdown_connections 2023-03-03T13:11:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd37041b700 1 --2- 172.21.15.7:0/3048075347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3680abe20 0x7fd3681529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd37041b700 1 --2- 172.21.15.7:0/3048075347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd36814fda0 0x7fd368156430 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd37041b700 1 -- 172.21.15.7:0/3048075347 >> 172.21.15.7:0/3048075347 conn(0x7fd3680c8900 msgr2=0x7fd3680c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd37041b700 1 -- 172.21.15.7:0/3048075347 shutdown_connections 2023-03-03T13:11:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.092+0000 7fd37041b700 1 -- 172.21.15.7:0/3048075347 wait complete. 2023-03-03T13:11:04.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd37041b700 1 Processor -- start 2023-03-03T13:11:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd37041b700 1 -- start start 2023-03-03T13:11:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd37041b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3680abe20 0x7fd3680c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd37041b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd36814f9d0 0x7fd36806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd37041b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36814fda0 0x7fd36814b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd37041b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3681dbb10 con 0x7fd3680abe20 2023-03-03T13:11:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd37041b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3681dc6a0 con 0x7fd36814f9d0 2023-03-03T13:11:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd37041b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3681dc980 con 0x7fd36814fda0 2023-03-03T13:11:04.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd36e1b7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3680abe20 0x7fd3680c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd36e1b7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3680abe20 0x7fd3680c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43596/0 (socket says 172.21.15.7:43596) 2023-03-03T13:11:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd36e1b7700 1 -- 172.21.15.7:0/811441182 learned_addr learned my addr 172.21.15.7:0/811441182 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:11:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd36d9b6700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd36814f9d0 0x7fd36806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.094+0000 7fd36e9b8700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36814fda0 0x7fd36814b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd36e1b7700 1 -- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36814fda0 msgr2=0x7fd36814b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd36e1b7700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36814fda0 0x7fd36814b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:04.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd36e1b7700 1 -- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd36814f9d0 msgr2=0x7fd36806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd36e1b7700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd36814f9d0 0x7fd36806e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd36e1b7700 1 -- 172.21.15.7:0/811441182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd358009550 con 0x7fd3680abe20 2023-03-03T13:11:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd36e1b7700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3680abe20 0x7fd3680c68f0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fd36400c4d0 tx=0x7fd36400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd35effd700 1 -- 172.21.15.7:0/811441182 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd364011790 con 0x7fd3680abe20 2023-03-03T13:11:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd35effd700 1 -- 172.21.15.7:0/811441182 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3640118f0 con 0x7fd3680abe20 2023-03-03T13:11:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd37041b700 1 -- 172.21.15.7:0/811441182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3680ac350 con 0x7fd3680abe20 2023-03-03T13:11:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd35effd700 1 -- 172.21.15.7:0/811441182 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd364011ca0 con 0x7fd3680abe20 2023-03-03T13:11:04.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.095+0000 7fd37041b700 1 -- 172.21.15.7:0/811441182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3680ab630 con 0x7fd3680abe20 2023-03-03T13:11:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.096+0000 7fd37041b700 1 -- 172.21.15.7:0/811441182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd368071fa0 con 0x7fd3680abe20 2023-03-03T13:11:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.097+0000 7fd35effd700 1 -- 172.21.15.7:0/811441182 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd364010050 con 0x7fd3680abe20 2023-03-03T13:11:04.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.097+0000 7fd35effd700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd34c0682f0 0x7fd34c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:04.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.097+0000 7fd35effd700 1 -- 172.21.15.7:0/811441182 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd364085d80 con 0x7fd3680abe20 2023-03-03T13:11:04.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.097+0000 7fd36d9b6700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd34c0682f0 0x7fd34c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:04.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.098+0000 7fd36d9b6700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd34c0682f0 0x7fd34c06a7a0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd3681501d0 tx=0x7fd358008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:04.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.101+0000 7fd35effd700 1 -- 172.21.15.7:0/811441182 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd3640520c0 con 0x7fd3680abe20 2023-03-03T13:11:04.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.340+0000 7fd37041b700 1 -- 172.21.15.7:0/811441182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3681dbdf0 con 0x7fd3680abe20 2023-03-03T13:11:04.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.340+0000 7fd35effd700 1 -- 172.21.15.7:0/811441182 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd364058090 con 0x7fd3680abe20 2023-03-03T13:11:04.342 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:11:04.342 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:11:04.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 -- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd34c0682f0 msgr2=0x7fd34c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:04.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd34c0682f0 0x7fd34c06a7a0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd3681501d0 tx=0x7fd358008030 comp rx=0 tx=0).stop 2023-03-03T13:11:04.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 -- 172.21.15.7:0/811441182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3680abe20 msgr2=0x7fd3680c68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:04.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3680abe20 0x7fd3680c68f0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fd36400c4d0 tx=0x7fd36400c7e0 comp rx=0 tx=0).stop 2023-03-03T13:11:04.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 -- 172.21.15.7:0/811441182 shutdown_connections 2023-03-03T13:11:04.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd36814fda0 0x7fd36814b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:04.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd34c0682f0 0x7fd34c06a7a0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:04.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3680abe20 0x7fd3680c68f0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:04.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 --2- 172.21.15.7:0/811441182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd36814f9d0 0x7fd36806e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:04.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 -- 172.21.15.7:0/811441182 >> 172.21.15.7:0/811441182 conn(0x7fd3680c8900 msgr2=0x7fd3680c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:04.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 -- 172.21.15.7:0/811441182 shutdown_connections 2023-03-03T13:11:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:04.342+0000 7fd37041b700 1 -- 172.21.15.7:0/811441182 wait complete. 2023-03-03T13:11:04.346 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:11:04.356 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before thrashing 2023-03-03T13:11:06.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:11:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.495+0000 7f962f942700 1 Processor -- start 2023-03-03T13:11:06.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.495+0000 7f962f942700 1 -- start start 2023-03-03T13:11:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962f942700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9628151d40 0x7f96280736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962f942700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9628073c30 0x7f96281581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962f942700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962806d430 con 0x7f96280c2930 2023-03-03T13:11:06.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962f942700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962806c2a0 con 0x7f9628073c30 2023-03-03T13:11:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962f942700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96281586e0 con 0x7f9628151d40 2023-03-03T13:11:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962dedf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9628073c30 0x7f96281581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962cedd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f96280c2930 0x7f9628157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37590/0 (socket says 172.21.15.7:37590) 2023-03-03T13:11:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962cedd700 1 -- 172.21.15.7:0/1805428352 learned_addr learned my addr 172.21.15.7:0/1805428352 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:11:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962d6de700 1 --2- 172.21.15.7:0/1805428352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9628151d40 0x7f96280736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962dedf700 1 -- 172.21.15.7:0/1805428352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9628151d40 msgr2=0x7f96280736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:11:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962dedf700 1 --2- 172.21.15.7:0/1805428352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9628151d40 0x7f96280736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:06.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962dedf700 1 -- 172.21.15.7:0/1805428352 >> v1:172.21.15.7:6789/0 conn(0x7f96280c2930 legacy=0x7f9628157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962dedf700 1 -- 172.21.15.7:0/1805428352 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962806cb00 con 0x7f9628073c30 2023-03-03T13:11:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f961ffff700 1 -- 172.21.15.7:0/1805428352 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229281955 0 0) 0x7f962806d430 con 0x7f96280c2930 2023-03-03T13:11:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.496+0000 7f962dedf700 1 --2- 172.21.15.7:0/1805428352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9628073c30 0x7f96281581a0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f9620009d90 tx=0x7f962000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989bebbe66b708a6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f961ffff700 1 -- 172.21.15.7:0/1805428352 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962000e070 con 0x7f9628073c30 2023-03-03T13:11:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f961ffff700 1 -- 172.21.15.7:0/1805428352 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962000bbd0 con 0x7f9628073c30 2023-03-03T13:11:06.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f961ffff700 1 -- 172.21.15.7:0/1805428352 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620012430 con 0x7f9628073c30 2023-03-03T13:11:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f962f942700 1 -- 172.21.15.7:0/1805428352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9628073c30 msgr2=0x7f96281581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f962f942700 1 --2- 172.21.15.7:0/1805428352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9628073c30 0x7f96281581a0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f9620009d90 tx=0x7f962000b330 comp rx=0 tx=0).stop 2023-03-03T13:11:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f962f942700 1 -- 172.21.15.7:0/1805428352 shutdown_connections 2023-03-03T13:11:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f962f942700 1 --2- 172.21.15.7:0/1805428352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9628073c30 0x7f96281581a0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f962f942700 1 --2- 172.21.15.7:0/1805428352 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9628151d40 0x7f96280736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f962f942700 1 -- 172.21.15.7:0/1805428352 >> 172.21.15.7:0/1805428352 conn(0x7f96280a94d0 msgr2=0x7f96280a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:06.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f962f942700 1 -- 172.21.15.7:0/1805428352 shutdown_connections 2023-03-03T13:11:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.497+0000 7f962f942700 1 -- 172.21.15.7:0/1805428352 wait complete. 2023-03-03T13:11:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.498+0000 7f962f942700 1 Processor -- start 2023-03-03T13:11:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.498+0000 7f962f942700 1 -- start start 2023-03-03T13:11:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962f942700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9628073c30 0x7f9628158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962f942700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96280c2930 0x7f96281e0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962f942700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9628151d40 0x7f96281eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962f942700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96281dfcd0 con 0x7f96280c2930 2023-03-03T13:11:06.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962f942700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96281edd30 con 0x7f9628073c30 2023-03-03T13:11:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962f942700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96281eb3d0 con 0x7f9628151d40 2023-03-03T13:11:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962cedd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96280c2930 0x7f96281e0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962cedd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96280c2930 0x7f96281e0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43618/0 (socket says 172.21.15.7:43618) 2023-03-03T13:11:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962cedd700 1 -- 172.21.15.7:0/695292092 learned_addr learned my addr 172.21.15.7:0/695292092 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:11:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962dedf700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9628151d40 0x7f96281eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962d6de700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9628073c30 0x7f9628158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:06.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962cedd700 1 -- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9628151d40 msgr2=0x7f96281eacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:06.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962cedd700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9628151d40 0x7f96281eacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:06.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962cedd700 1 -- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9628073c30 msgr2=0x7f9628158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:06.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962cedd700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9628073c30 0x7f9628158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:06.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.499+0000 7f962cedd700 1 -- 172.21.15.7:0/695292092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9620009a70 con 0x7f96280c2930 2023-03-03T13:11:06.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.500+0000 7f962d6de700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9628073c30 0x7f9628158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:11:06.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.500+0000 7f962cedd700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96280c2930 0x7f96281e0160 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f961800c550 tx=0x7f961800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:06.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.500+0000 7f961dffb700 1 -- 172.21.15.7:0/695292092 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618004c80 con 0x7f96280c2930 2023-03-03T13:11:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.500+0000 7f961dffb700 1 -- 172.21.15.7:0/695292092 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9618004e00 con 0x7f96280c2930 2023-03-03T13:11:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.500+0000 7f962f942700 1 -- 172.21.15.7:0/695292092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962806d0c0 con 0x7f96280c2930 2023-03-03T13:11:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.500+0000 7f961dffb700 1 -- 172.21.15.7:0/695292092 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618007310 con 0x7f96280c2930 2023-03-03T13:11:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.500+0000 7f962f942700 1 -- 172.21.15.7:0/695292092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962806cde0 con 0x7f96280c2930 2023-03-03T13:11:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.501+0000 7f962f942700 1 -- 172.21.15.7:0/695292092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962806bea0 con 0x7f96280c2930 2023-03-03T13:11:06.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.501+0000 7f961dffb700 1 -- 172.21.15.7:0/695292092 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f961801a020 con 0x7f96280c2930 2023-03-03T13:11:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.502+0000 7f961dffb700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f960c0682f0 0x7f960c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.502+0000 7f961dffb700 1 -- 172.21.15.7:0/695292092 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9618085e30 con 0x7f96280c2930 2023-03-03T13:11:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.502+0000 7f962d6de700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f960c0682f0 0x7f960c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.502+0000 7f962d6de700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f960c0682f0 0x7f960c06a7a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f9624006fd0 tx=0x7f9624008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:06.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.506+0000 7f961dffb700 1 -- 172.21.15.7:0/695292092 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f96180521f0 con 0x7f96280c2930 2023-03-03T13:11:06.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.744+0000 7f962f942700 1 -- 172.21.15.7:0/695292092 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9628074bb0 con 0x7f96280c2930 2023-03-03T13:11:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.744+0000 7f961dffb700 1 -- 172.21.15.7:0/695292092 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9618055810 con 0x7f96280c2930 2023-03-03T13:11:06.746 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:11:06.746 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:11:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 -- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f960c0682f0 msgr2=0x7f960c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f960c0682f0 0x7f960c06a7a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f9624006fd0 tx=0x7f9624008040 comp rx=0 tx=0).stop 2023-03-03T13:11:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 -- 172.21.15.7:0/695292092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96280c2930 msgr2=0x7f96281e0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96280c2930 0x7f96281e0160 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f961800c550 tx=0x7f961800c910 comp rx=0 tx=0).stop 2023-03-03T13:11:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 -- 172.21.15.7:0/695292092 shutdown_connections 2023-03-03T13:11:06.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9628151d40 0x7f96281eacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:06.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9628073c30 0x7f9628158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:06.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f960c0682f0 0x7f960c06a7a0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:06.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 --2- 172.21.15.7:0/695292092 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f96280c2930 0x7f96281e0160 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:06.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 -- 172.21.15.7:0/695292092 >> 172.21.15.7:0/695292092 conn(0x7f96280a94d0 msgr2=0x7f96280a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:06.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 -- 172.21.15.7:0/695292092 shutdown_connections 2023-03-03T13:11:06.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:06.746+0000 7f962f942700 1 -- 172.21.15.7:0/695292092 wait complete. 2023-03-03T13:11:06.750 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:11:06.760 INFO:tasks.mds_thrash.fs.[b]:waiting for 53 secs before thrashing 2023-03-03T13:11:08.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:08.268 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:14.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:11:14.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.808+0000 7f5a1d63b700 1 Processor -- start 2023-03-03T13:11:14.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.808+0000 7f5a1d63b700 1 -- start start 2023-03-03T13:11:14.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.809+0000 7f5a1d63b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a1806e020 0x7f5a18071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:14.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.809+0000 7f5a1d63b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 0x7f5a18151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:14.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.809+0000 7f5a1d63b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a18154910 con 0x7f5a18155390 2023-03-03T13:11:14.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.809+0000 7f5a1d63b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a18070b90 con 0x7f5a1806e020 2023-03-03T13:11:14.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.809+0000 7f5a1d63b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a18151320 con 0x7f5a18072520 2023-03-03T13:11:14.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.809+0000 7f5a167fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a1806e020 0x7f5a18071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:14.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.809+0000 7f5a16ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5a18155390 0x7f5a180c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37610/0 (socket says 172.21.15.7:37610) 2023-03-03T13:11:14.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.809+0000 7f5a16ffd700 1 -- 172.21.15.7:0/4183130116 learned_addr learned my addr 172.21.15.7:0/4183130116 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:11:14.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.809+0000 7f5a177fe700 1 --2- 172.21.15.7:0/4183130116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 0x7f5a18151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:14.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a167fc700 1 -- 172.21.15.7:0/4183130116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 msgr2=0x7f5a18151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:11:14.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a167fc700 1 --2- 172.21.15.7:0/4183130116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 0x7f5a18151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:14.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a167fc700 1 -- 172.21.15.7:0/4183130116 >> v1:172.21.15.7:6789/0 conn(0x7f5a18155390 legacy=0x7f5a180c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:14.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a167fc700 1 -- 172.21.15.7:0/4183130116 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180c4540 con 0x7f5a1806e020 2023-03-03T13:11:14.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a177fe700 1 --2- 172.21.15.7:0/4183130116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 0x7f5a18151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:11:14.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a167fc700 1 --2- 172.21.15.7:0/4183130116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 0x7f5a18071fe0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f5a08009840 tx=0x7f5a08009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616988a629701e26 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:14.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a15ffb700 1 -- 172.21.15.7:0/4183130116 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08011830 con 0x7f5a1806e020 2023-03-03T13:11:14.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a15ffb700 1 -- 172.21.15.7:0/4183130116 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a08011990 con 0x7f5a1806e020 2023-03-03T13:11:14.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a15ffb700 1 -- 172.21.15.7:0/4183130116 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08011d20 con 0x7f5a1806e020 2023-03-03T13:11:14.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a1d63b700 1 -- 172.21.15.7:0/4183130116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 msgr2=0x7f5a18071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:14.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.810+0000 7f5a1d63b700 1 --2- 172.21.15.7:0/4183130116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 0x7f5a18071fe0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f5a08009840 tx=0x7f5a08009b50 comp rx=0 tx=0).stop 2023-03-03T13:11:14.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.811+0000 7f5a1d63b700 1 -- 172.21.15.7:0/4183130116 shutdown_connections 2023-03-03T13:11:14.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.811+0000 7f5a1d63b700 1 --2- 172.21.15.7:0/4183130116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 0x7f5a18151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:14.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.811+0000 7f5a1d63b700 1 --2- 172.21.15.7:0/4183130116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 0x7f5a18071fe0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:14.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.811+0000 7f5a1d63b700 1 -- 172.21.15.7:0/4183130116 >> 172.21.15.7:0/4183130116 conn(0x7f5a1806c110 msgr2=0x7f5a1806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:14.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.811+0000 7f5a1d63b700 1 -- 172.21.15.7:0/4183130116 shutdown_connections 2023-03-03T13:11:14.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.811+0000 7f5a1d63b700 1 -- 172.21.15.7:0/4183130116 wait complete. 2023-03-03T13:11:14.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.812+0000 7f5a1d63b700 1 Processor -- start 2023-03-03T13:11:14.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.812+0000 7f5a1d63b700 1 -- start start 2023-03-03T13:11:14.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a1d63b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 0x7f5a1814dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a1d63b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 0x7f5a18139ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a1d63b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a18155390 0x7f5a1813a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a1d63b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1814c3b0 con 0x7f5a1806e020 2023-03-03T13:11:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a1d63b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181542c0 con 0x7f5a18155390 2023-03-03T13:11:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a1d63b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181e89f0 con 0x7f5a18072520 2023-03-03T13:11:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a16ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 0x7f5a1814dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a16ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 0x7f5a1814dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43636/0 (socket says 172.21.15.7:43636) 2023-03-03T13:11:14.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a16ffd700 1 -- 172.21.15.7:0/1658433286 learned_addr learned my addr 172.21.15.7:0/1658433286 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:11:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a167fc700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 0x7f5a18139ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a177fe700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a18155390 0x7f5a1813a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a16ffd700 1 -- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 msgr2=0x7f5a18139ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a16ffd700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 0x7f5a18139ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a16ffd700 1 -- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a18155390 msgr2=0x7f5a1813a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a16ffd700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a18155390 0x7f5a1813a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:14.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a16ffd700 1 -- 172.21.15.7:0/1658433286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a080094a0 con 0x7f5a1806e020 2023-03-03T13:11:14.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a167fc700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 0x7f5a18139ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:11:14.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a16ffd700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 0x7f5a1814dca0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000c480 tx=0x7f5a0000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:14.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.813+0000 7f5a0f7fe700 1 -- 172.21.15.7:0/1658433286 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00005d70 con 0x7f5a1806e020 2023-03-03T13:11:14.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.814+0000 7f5a0f7fe700 1 -- 172.21.15.7:0/1658433286 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a000070d0 con 0x7f5a1806e020 2023-03-03T13:11:14.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.814+0000 7f5a0f7fe700 1 -- 172.21.15.7:0/1658433286 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00007440 con 0x7f5a1806e020 2023-03-03T13:11:14.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.814+0000 7f5a1d63b700 1 -- 172.21.15.7:0/1658433286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a18070540 con 0x7f5a1806e020 2023-03-03T13:11:14.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.814+0000 7f5a1d63b700 1 -- 172.21.15.7:0/1658433286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a180702a0 con 0x7f5a1806e020 2023-03-03T13:11:14.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.815+0000 7f5a1d63b700 1 -- 172.21.15.7:0/1658433286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18070820 con 0x7f5a1806e020 2023-03-03T13:11:14.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.815+0000 7f5a0f7fe700 1 -- 172.21.15.7:0/1658433286 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5a00019030 con 0x7f5a1806e020 2023-03-03T13:11:14.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.815+0000 7f5a0f7fe700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f80682f0 0x7f59f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.816+0000 7f5a0f7fe700 1 -- 172.21.15.7:0/1658433286 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5a00085430 con 0x7f5a1806e020 2023-03-03T13:11:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.816+0000 7f5a167fc700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f80682f0 0x7f59f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.816+0000 7f5a167fc700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f80682f0 0x7f59f806a7a0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f5a18074730 tx=0x7f5a08008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:14.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:14.820+0000 7f5a0f7fe700 1 -- 172.21.15.7:0/1658433286 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a0004b5e0 con 0x7f5a1806e020 2023-03-03T13:11:15.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.056+0000 7f5a1d63b700 1 -- 172.21.15.7:0/1658433286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a18074bb0 con 0x7f5a1806e020 2023-03-03T13:11:15.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.057+0000 7f5a0f7fe700 1 -- 172.21.15.7:0/1658433286 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5a00051880 con 0x7f5a1806e020 2023-03-03T13:11:15.060 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:11:15.060 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:11:15.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 -- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f80682f0 msgr2=0x7f59f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:15.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f80682f0 0x7f59f806a7a0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f5a18074730 tx=0x7f5a08008030 comp rx=0 tx=0).stop 2023-03-03T13:11:15.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 -- 172.21.15.7:0/1658433286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 msgr2=0x7f5a1814dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:15.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 0x7f5a1814dca0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000c480 tx=0x7f5a0000c840 comp rx=0 tx=0).stop 2023-03-03T13:11:15.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 -- 172.21.15.7:0/1658433286 shutdown_connections 2023-03-03T13:11:15.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a18155390 0x7f5a1813a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:15.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59f80682f0 0x7f59f806a7a0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:15.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1806e020 0x7f5a1814dca0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:15.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 --2- 172.21.15.7:0/1658433286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a18072520 0x7f5a18139ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:15.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 -- 172.21.15.7:0/1658433286 >> 172.21.15.7:0/1658433286 conn(0x7f5a1806c110 msgr2=0x7f5a1814b170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:15.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 -- 172.21.15.7:0/1658433286 shutdown_connections 2023-03-03T13:11:15.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:15.059+0000 7f5a1d63b700 1 -- 172.21.15.7:0/1658433286 wait complete. 2023-03-03T13:11:15.063 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:11:15.073 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before thrashing 2023-03-03T13:11:24.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:11:24.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.211+0000 7f439b92f700 1 Processor -- start 2023-03-03T13:11:24.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.211+0000 7f439b92f700 1 -- start start 2023-03-03T13:11:24.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f439b92f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43940ab0c0 0x7f43940c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:24.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f439b92f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4394154530 con 0x7f439406a2b0 2023-03-03T13:11:24.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f439b92f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4394154810 con 0x7f439406a720 2023-03-03T13:11:24.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f439b92f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4394154af0 con 0x7f43940ab0c0 2023-03-03T13:11:24.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f4398eca700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f439406a2b0 0x7f4394157a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37626/0 (socket says 172.21.15.7:37626) 2023-03-03T13:11:24.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f4398eca700 1 -- 172.21.15.7:0/3451597809 learned_addr learned my addr 172.21.15.7:0/3451597809 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:11:24.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f43996cb700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43940ab0c0 0x7f43940c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:24.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072612660 0 0) 0x7f4394154530 con 0x7f439406a2b0 2023-03-03T13:11:24.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43800035b0 con 0x7f439406a2b0 2023-03-03T13:11:24.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398912512 0 0) 0x7f4394154810 con 0x7f439406a720 2023-03-03T13:11:24.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4394154530 con 0x7f439406a720 2023-03-03T13:11:24.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2204699636 0 0) 0x7f43800035b0 con 0x7f439406a2b0 2023-03-03T13:11:24.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43940ab0c0 msgr2=0x7f43940c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:11:24.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43940ab0c0 0x7f43940c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:24.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 >> v1:172.21.15.143:6789/0 conn(0x7f439406a720 legacy=0x7f4394158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:24.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4394154dd0 con 0x7f439406a2b0 2023-03-03T13:11:24.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f4384001670 con 0x7f439406a2b0 2023-03-03T13:11:24.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 >> v1:172.21.15.7:6789/0 conn(0x7f439406a2b0 legacy=0x7f4394157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:24.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4380003c30 0x7f4380004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:24.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43940ab0c0 0x7f4380004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:24.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.212+0000 7f438bfff700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4380004d90 0x7f4380005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:24.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4394154af0 con 0x7f4380003c30 2023-03-03T13:11:24.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438000baf0 con 0x7f4380004d90 2023-03-03T13:11:24.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438000bcb0 con 0x7f43940ab0c0 2023-03-03T13:11:24.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f43996cb700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4380003c30 0x7f4380004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:24.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f4398eca700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43940ab0c0 0x7f4380004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:24.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f4399ecc700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4380004d90 0x7f4380005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:24.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f43996cb700 1 -- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43940ab0c0 msgr2=0x7f4380004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:11:24.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f43996cb700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43940ab0c0 0x7f4380004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:24.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f43996cb700 1 -- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4380004d90 msgr2=0x7f4380005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:11:24.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f43996cb700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4380004d90 0x7f4380005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:24.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f43996cb700 1 -- 172.21.15.7:0/3451597809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43800035b0 con 0x7f4380003c30 2023-03-03T13:11:24.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f43996cb700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4380003c30 0x7f4380004020 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f4390009b40 tx=0x7f4380005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e148c28d5a9a1cfa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:24.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439000e6c0 con 0x7f4380003c30 2023-03-03T13:11:24.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439000e820 con 0x7f4380003c30 2023-03-03T13:11:24.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.213+0000 7f438bfff700 1 -- 172.21.15.7:0/3451597809 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439000ebd0 con 0x7f4380003c30 2023-03-03T13:11:24.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.214+0000 7f439b92f700 1 -- 172.21.15.7:0/3451597809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4380003c30 msgr2=0x7f4380004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:24.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.214+0000 7f439b92f700 1 --2- 172.21.15.7:0/3451597809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4380003c30 0x7f4380004020 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f4390009b40 tx=0x7f4380005880 comp rx=0 tx=0).stop 2023-03-03T13:11:24.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.214+0000 7f4399ecc700 1 -- 172.21.15.7:0/3451597809 reap_dead start 2023-03-03T13:11:24.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.214+0000 7f439b92f700 1 -- 172.21.15.7:0/3451597809 shutdown_connections 2023-03-03T13:11:24.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.214+0000 7f439b92f700 1 -- 172.21.15.7:0/3451597809 >> 172.21.15.7:0/3451597809 conn(0x7f43940a9430 msgr2=0x7f43940c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:24.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.214+0000 7f439b92f700 1 -- 172.21.15.7:0/3451597809 shutdown_connections 2023-03-03T13:11:24.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.214+0000 7f439b92f700 1 -- 172.21.15.7:0/3451597809 wait complete. 2023-03-03T13:11:24.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.215+0000 7f439b92f700 1 Processor -- start 2023-03-03T13:11:24.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.215+0000 7f439b92f700 1 -- start start 2023-03-03T13:11:24.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f439b92f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4380003c30 0x7f43941ed790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:24.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f439b92f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43941edcd0 0x7f43941df140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:24.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f439b92f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43941df680 0x7f43941509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:24.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f439b92f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4394070770 con 0x7f43941edcd0 2023-03-03T13:11:24.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f439b92f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439406fbe0 con 0x7f4380003c30 2023-03-03T13:11:24.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f439b92f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43940725c0 con 0x7f43941df680 2023-03-03T13:11:24.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f4398eca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43941edcd0 0x7f43941df140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:24.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f4398eca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43941edcd0 0x7f43941df140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43658/0 (socket says 172.21.15.7:43658) 2023-03-03T13:11:24.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f4398eca700 1 -- 172.21.15.7:0/1148247637 learned_addr learned my addr 172.21.15.7:0/1148247637 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:11:24.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f43996cb700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4380003c30 0x7f43941ed790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:24.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f4399ecc700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43941df680 0x7f43941509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:24.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f4398eca700 1 -- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43941df680 msgr2=0x7f43941509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f4398eca700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43941df680 0x7f43941509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f4398eca700 1 -- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4380003c30 msgr2=0x7f43941ed790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f4398eca700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4380003c30 0x7f43941ed790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f4398eca700 1 -- 172.21.15.7:0/1148247637 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43940ab490 con 0x7f43941edcd0 2023-03-03T13:11:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.216+0000 7f4398eca700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43941edcd0 0x7f43941df140 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f4384007310 tx=0x7f4384007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.217+0000 7f4389ffb700 1 -- 172.21.15.7:0/1148247637 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4384012560 con 0x7f43941edcd0 2023-03-03T13:11:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.217+0000 7f4389ffb700 1 -- 172.21.15.7:0/1148247637 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43840126c0 con 0x7f43941edcd0 2023-03-03T13:11:24.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.217+0000 7f439b92f700 1 -- 172.21.15.7:0/1148247637 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43941550b0 con 0x7f43941edcd0 2023-03-03T13:11:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.217+0000 7f4389ffb700 1 -- 172.21.15.7:0/1148247637 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4384012a30 con 0x7f43941edcd0 2023-03-03T13:11:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.217+0000 7f439b92f700 1 -- 172.21.15.7:0/1148247637 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43940c31c0 con 0x7f43941edcd0 2023-03-03T13:11:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.218+0000 7f439b92f700 1 -- 172.21.15.7:0/1148247637 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4394074bb0 con 0x7f43941edcd0 2023-03-03T13:11:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.218+0000 7f4389ffb700 1 -- 172.21.15.7:0/1148247637 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4384012b90 con 0x7f43941edcd0 2023-03-03T13:11:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.219+0000 7f4389ffb700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43780682f0 0x7f437806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.219+0000 7f4389ffb700 1 -- 172.21.15.7:0/1148247637 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f43840860f0 con 0x7f43941edcd0 2023-03-03T13:11:24.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.219+0000 7f43996cb700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43780682f0 0x7f437806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:24.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.220+0000 7f43996cb700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43780682f0 0x7f437806a7a0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f43900090b0 tx=0x7f4390008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:24.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.223+0000 7f4389ffb700 1 -- 172.21.15.7:0/1148247637 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4384052540 con 0x7f43941edcd0 2023-03-03T13:11:24.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.466+0000 7f439b92f700 1 -- 172.21.15.7:0/1148247637 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43940c7070 con 0x7f43941edcd0 2023-03-03T13:11:24.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.466+0000 7f4389ffb700 1 -- 172.21.15.7:0/1148247637 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4384012e20 con 0x7f43941edcd0 2023-03-03T13:11:24.469 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:11:24.469 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:11:24.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 -- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43780682f0 msgr2=0x7f437806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:24.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43780682f0 0x7f437806a7a0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f43900090b0 tx=0x7f4390008030 comp rx=0 tx=0).stop 2023-03-03T13:11:24.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 -- 172.21.15.7:0/1148247637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43941edcd0 msgr2=0x7f43941df140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:24.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43941edcd0 0x7f43941df140 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f4384007310 tx=0x7f4384007620 comp rx=0 tx=0).stop 2023-03-03T13:11:24.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 -- 172.21.15.7:0/1148247637 shutdown_connections 2023-03-03T13:11:24.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43941df680 0x7f43941509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:24.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4380003c30 0x7f43941ed790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:24.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43780682f0 0x7f437806a7a0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:24.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 --2- 172.21.15.7:0/1148247637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43941edcd0 0x7f43941df140 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:24.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 -- 172.21.15.7:0/1148247637 >> 172.21.15.7:0/1148247637 conn(0x7f43940a9430 msgr2=0x7f4394153890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:24.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 -- 172.21.15.7:0/1148247637 shutdown_connections 2023-03-03T13:11:24.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:24.468+0000 7f439b92f700 1 -- 172.21.15.7:0/1148247637 wait complete. 2023-03-03T13:11:24.473 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:11:24.483 INFO:tasks.mds_thrash.fs.[a]:waiting for 73 secs before thrashing 2023-03-03T13:11:38.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:38.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:11:59.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:11:59.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.898+0000 7fddd9ef6700 1 Processor -- start 2023-03-03T13:11:59.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.898+0000 7fddd9ef6700 1 -- start start 2023-03-03T13:11:59.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd9ef6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40caa30 0x7fddd40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:59.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd9ef6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fddd40c00e0 0x7fddd4154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:59.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd9ef6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fddd40cae00 0x7fddd40c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd9ef6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd40cb280 con 0x7fddd40c00e0 2023-03-03T13:11:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd9ef6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd40bfd10 con 0x7fddd40cae00 2023-03-03T13:11:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd9ef6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd40bfa30 con 0x7fddd40caa30 2023-03-03T13:11:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd2ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fddd40c00e0 0x7fddd4154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd2ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fddd40c00e0 0x7fddd4154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43680/0 (socket says 172.21.15.7:43680) 2023-03-03T13:11:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd2ffd700 1 -- 172.21.15.7:0/921179216 learned_addr learned my addr 172.21.15.7:0/921179216 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:11:59.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd37fe700 1 --2- 172.21.15.7:0/921179216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40caa30 0x7fddd40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:59.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd3fff700 1 --2- 172.21.15.7:0/921179216 >> v2:172.21.15.143:3300/0 conn(0x7fddd40cae00 0x7fddd40c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:59.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd2ffd700 1 -- 172.21.15.7:0/921179216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40caa30 msgr2=0x7fddd40c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:11:59.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd2ffd700 1 --2- 172.21.15.7:0/921179216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40caa30 0x7fddd40c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:59.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd2ffd700 1 -- 172.21.15.7:0/921179216 >> v2:172.21.15.143:3300/0 conn(0x7fddd40cae00 msgr2=0x7fddd40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:11:59.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd2ffd700 1 --2- 172.21.15.7:0/921179216 >> v2:172.21.15.143:3300/0 conn(0x7fddd40cae00 0x7fddd40c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:59.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd2ffd700 1 -- 172.21.15.7:0/921179216 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd4154510 con 0x7fddd40c00e0 2023-03-03T13:11:59.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.899+0000 7fddd2ffd700 1 --2- 172.21.15.7:0/921179216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40c00e0 0x7fddd4154ff0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fddd4158a70 tx=0x7fddbc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0b0788ec0d0e08 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:59.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd27fc700 1 -- 172.21.15.7:0/921179216 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc005e30 con 0x7fddd40c00e0 2023-03-03T13:11:59.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd27fc700 1 -- 172.21.15.7:0/921179216 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddbc004030 con 0x7fddd40c00e0 2023-03-03T13:11:59.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd27fc700 1 -- 172.21.15.7:0/921179216 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc0043e0 con 0x7fddd40c00e0 2023-03-03T13:11:59.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd9ef6700 1 -- 172.21.15.7:0/921179216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40c00e0 msgr2=0x7fddd4154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:59.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd9ef6700 1 --2- 172.21.15.7:0/921179216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40c00e0 0x7fddd4154ff0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fddd4158a70 tx=0x7fddbc009bd0 comp rx=0 tx=0).stop 2023-03-03T13:11:59.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd9ef6700 1 -- 172.21.15.7:0/921179216 shutdown_connections 2023-03-03T13:11:59.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd9ef6700 1 --2- 172.21.15.7:0/921179216 >> v2:172.21.15.143:3300/0 conn(0x7fddd40cae00 0x7fddd40c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:59.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd9ef6700 1 --2- 172.21.15.7:0/921179216 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40caa30 0x7fddd40c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:59.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd9ef6700 1 --2- 172.21.15.7:0/921179216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40c00e0 0x7fddd4154ff0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:59.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd9ef6700 1 -- 172.21.15.7:0/921179216 >> 172.21.15.7:0/921179216 conn(0x7fddd4066e40 msgr2=0x7fddd4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:11:59.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd9ef6700 1 -- 172.21.15.7:0/921179216 shutdown_connections 2023-03-03T13:11:59.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.900+0000 7fddd9ef6700 1 -- 172.21.15.7:0/921179216 wait complete. 2023-03-03T13:11:59.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.901+0000 7fddd9ef6700 1 Processor -- start 2023-03-03T13:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.902+0000 7fddd9ef6700 1 -- start start 2023-03-03T13:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.902+0000 7fddd9ef6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40c00e0 0x7fddd40c28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.902+0000 7fddd9ef6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd40caa30 0x7fddd4156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.902+0000 7fddd9ef6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40cae00 0x7fddd41570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.902+0000 7fddd9ef6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd40c48b0 con 0x7fddd40cae00 2023-03-03T13:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.902+0000 7fddd9ef6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd40c3ca0 con 0x7fddd40caa30 2023-03-03T13:11:59.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.902+0000 7fddd9ef6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd40c3f80 con 0x7fddd40c00e0 2023-03-03T13:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40cae00 0x7fddd41570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40cae00 0x7fddd41570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43688/0 (socket says 172.21.15.7:43688) 2023-03-03T13:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd3fff700 1 -- 172.21.15.7:0/3707788575 learned_addr learned my addr 172.21.15.7:0/3707788575 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd37fe700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40c00e0 0x7fddd40c28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd2ffd700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd40caa30 0x7fddd4156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd3fff700 1 -- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40c00e0 msgr2=0x7fddd40c28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:59.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd3fff700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40c00e0 0x7fddd40c28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:59.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd3fff700 1 -- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd40caa30 msgr2=0x7fddd4156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:11:59.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd3fff700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd40caa30 0x7fddd4156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:11:59.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd3fff700 1 -- 172.21.15.7:0/3707788575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc009500 con 0x7fddd40cae00 2023-03-03T13:11:59.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd37fe700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40c00e0 0x7fddd40c28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:11:59.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd3fff700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40cae00 0x7fddd41570a0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fddcc007e40 tx=0x7fddcc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:59.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddcbfff700 1 -- 172.21.15.7:0/3707788575 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc00eea0 con 0x7fddd40cae00 2023-03-03T13:11:59.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddcbfff700 1 -- 172.21.15.7:0/3707788575 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc014420 con 0x7fddd40cae00 2023-03-03T13:11:59.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd9ef6700 1 -- 172.21.15.7:0/3707788575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd4154ad0 con 0x7fddd40cae00 2023-03-03T13:11:59.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddcbfff700 1 -- 172.21.15.7:0/3707788575 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc0147b0 con 0x7fddd40cae00 2023-03-03T13:11:59.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.903+0000 7fddd9ef6700 1 -- 172.21.15.7:0/3707788575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd41547f0 con 0x7fddd40cae00 2023-03-03T13:11:59.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.904+0000 7fddd9ef6700 1 -- 172.21.15.7:0/3707788575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd40bf650 con 0x7fddd40cae00 2023-03-03T13:11:59.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.905+0000 7fddcbfff700 1 -- 172.21.15.7:0/3707788575 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fddcc012050 con 0x7fddd40cae00 2023-03-03T13:11:59.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.905+0000 7fddcbfff700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddb40682f0 0x7fddb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:11:59.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.905+0000 7fddcbfff700 1 -- 172.21.15.7:0/3707788575 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fddcc0867a0 con 0x7fddd40cae00 2023-03-03T13:11:59.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.905+0000 7fddd37fe700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddb40682f0 0x7fddb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:11:59.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.906+0000 7fddd37fe700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddb40682f0 0x7fddb406a7a0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009a00 tx=0x7fddc4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:11:59.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:11:59.910+0000 7fddcbfff700 1 -- 172.21.15.7:0/3707788575 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fddcc052b40 con 0x7fddd40cae00 2023-03-03T13:12:00.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.148+0000 7fddd9ef6700 1 -- 172.21.15.7:0/3707788575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddd404e6e0 con 0x7fddd40cae00 2023-03-03T13:12:00.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.149+0000 7fddcbfff700 1 -- 172.21.15.7:0/3707788575 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fddcc056160 con 0x7fddd40cae00 2023-03-03T13:12:00.150 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:12:00.150 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:12:00.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.150+0000 7fddd9ef6700 1 -- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddb40682f0 msgr2=0x7fddb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:00.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.150+0000 7fddd9ef6700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddb40682f0 0x7fddb406a7a0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009a00 tx=0x7fddc4008040 comp rx=0 tx=0).stop 2023-03-03T13:12:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.150+0000 7fddd9ef6700 1 -- 172.21.15.7:0/3707788575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40cae00 msgr2=0x7fddd41570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.150+0000 7fddd9ef6700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40cae00 0x7fddd41570a0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fddcc007e40 tx=0x7fddcc00e3c0 comp rx=0 tx=0).stop 2023-03-03T13:12:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.151+0000 7fddd9ef6700 1 -- 172.21.15.7:0/3707788575 shutdown_connections 2023-03-03T13:12:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.151+0000 7fddd9ef6700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddb40682f0 0x7fddb406a7a0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.151+0000 7fddd9ef6700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd40cae00 0x7fddd41570a0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.151+0000 7fddd9ef6700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd40c00e0 0x7fddd40c28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.151+0000 7fddd9ef6700 1 --2- 172.21.15.7:0/3707788575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd40caa30 0x7fddd4156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.151+0000 7fddd9ef6700 1 -- 172.21.15.7:0/3707788575 >> 172.21.15.7:0/3707788575 conn(0x7fddd4066e40 msgr2=0x7fddd40c7720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:12:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.151+0000 7fddd9ef6700 1 -- 172.21.15.7:0/3707788575 shutdown_connections 2023-03-03T13:12:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:00.151+0000 7fddd9ef6700 1 -- 172.21.15.7:0/3707788575 wait complete. 2023-03-03T13:12:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:12:00.164 INFO:tasks.mds_thrash.fs.[b]:waiting for 95 secs before thrashing 2023-03-03T13:12:08.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:08.553 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:37.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:12:37.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.620+0000 7fd53785b700 1 Processor -- start 2023-03-03T13:12:37.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.621+0000 7fd53785b700 1 -- start start 2023-03-03T13:12:37.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.621+0000 7fd53785b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd530151e80 0x7fd5300c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:37.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.621+0000 7fd53785b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 0x7fd5300bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:37.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.621+0000 7fd53785b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300c08d0 con 0x7fd5300c6fe0 2023-03-03T13:12:37.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.621+0000 7fd53785b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd530152250 con 0x7fd530151e80 2023-03-03T13:12:37.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.621+0000 7fd53785b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300c0fb0 con 0x7fd530155410 2023-03-03T13:12:37.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.621+0000 7fd5355f7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd530151e80 0x7fd5300c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:37.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd5355f7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd530151e80 0x7fd5300c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43716/0 (socket says 172.21.15.7:43716) 2023-03-03T13:12:37.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd5355f7700 1 -- 172.21.15.7:0/2486376891 learned_addr learned my addr 172.21.15.7:0/2486376891 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:12:37.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd534df6700 1 --2- 172.21.15.7:0/2486376891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 0x7fd5300bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:37.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd5355f7700 1 -- 172.21.15.7:0/2486376891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 msgr2=0x7fd5300bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:12:37.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd5355f7700 1 --2- 172.21.15.7:0/2486376891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 0x7fd5300bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:37.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd5355f7700 1 -- 172.21.15.7:0/2486376891 >> v1:172.21.15.143:6789/0 conn(0x7fd5300c6fe0 legacy=0x7fd5300bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:37.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd5355f7700 1 -- 172.21.15.7:0/2486376891 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530154930 con 0x7fd530151e80 2023-03-03T13:12:37.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd534df6700 1 --2- 172.21.15.7:0/2486376891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 0x7fd5300bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:12:37.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd5355f7700 1 --2- 172.21.15.7:0/2486376891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 0x7fd5300c6aa0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fd520009920 tx=0x7fd520009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9abb6ebea405f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:12:37.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd527fff700 1 -- 172.21.15.7:0/2486376891 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52000ba20 con 0x7fd530151e80 2023-03-03T13:12:37.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd527fff700 1 -- 172.21.15.7:0/2486376891 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52000bb80 con 0x7fd530151e80 2023-03-03T13:12:37.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.622+0000 7fd527fff700 1 -- 172.21.15.7:0/2486376891 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520009e70 con 0x7fd530151e80 2023-03-03T13:12:37.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.623+0000 7fd53785b700 1 -- 172.21.15.7:0/2486376891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 msgr2=0x7fd5300c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:37.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.623+0000 7fd53785b700 1 --2- 172.21.15.7:0/2486376891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 0x7fd5300c6aa0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fd520009920 tx=0x7fd520009c30 comp rx=0 tx=0).stop 2023-03-03T13:12:37.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.623+0000 7fd53785b700 1 -- 172.21.15.7:0/2486376891 shutdown_connections 2023-03-03T13:12:37.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.623+0000 7fd53785b700 1 --2- 172.21.15.7:0/2486376891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 0x7fd5300c6aa0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:37.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.623+0000 7fd53785b700 1 --2- 172.21.15.7:0/2486376891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 0x7fd5300bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:37.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.623+0000 7fd53785b700 1 -- 172.21.15.7:0/2486376891 >> 172.21.15.7:0/2486376891 conn(0x7fd530071630 msgr2=0x7fd53006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:12:37.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.623+0000 7fd53785b700 1 -- 172.21.15.7:0/2486376891 shutdown_connections 2023-03-03T13:12:37.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.623+0000 7fd53785b700 1 -- 172.21.15.7:0/2486376891 wait complete. 2023-03-03T13:12:37.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.624+0000 7fd53785b700 1 Processor -- start 2023-03-03T13:12:37.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.624+0000 7fd53785b700 1 -- start start 2023-03-03T13:12:37.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd53785b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5300c6fe0 0x7fd53014d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:37.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd53785b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 0x7fd5300bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:37.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd53785b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 0x7fd530157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:37.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd53785b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53014f960 con 0x7fd530151e80 2023-03-03T13:12:37.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd53785b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd530157270 con 0x7fd5300c6fe0 2023-03-03T13:12:37.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd53785b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd530156f40 con 0x7fd530155410 2023-03-03T13:12:37.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd534df6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 0x7fd5300bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:37.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd534df6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 0x7fd5300bf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43724/0 (socket says 172.21.15.7:43724) 2023-03-03T13:12:37.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd534df6700 1 -- 172.21.15.7:0/1192978803 learned_addr learned my addr 172.21.15.7:0/1192978803 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:12:37.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd534df6700 1 -- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 msgr2=0x7fd530157690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:37.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd5355f7700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5300c6fe0 0x7fd53014d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:37.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd535df8700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 0x7fd530157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:37.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd534df6700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 0x7fd530157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:37.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd534df6700 1 -- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5300c6fe0 msgr2=0x7fd53014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:37.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd534df6700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5300c6fe0 0x7fd53014d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:37.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd534df6700 1 -- 172.21.15.7:0/1192978803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd520009550 con 0x7fd530151e80 2023-03-03T13:12:37.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd534df6700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 0x7fd5300bf940 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00e840 tx=0x7fd52c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:12:37.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.625+0000 7fd525ffb700 1 -- 172.21.15.7:0/1192978803 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c013610 con 0x7fd530151e80 2023-03-03T13:12:37.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.626+0000 7fd525ffb700 1 -- 172.21.15.7:0/1192978803 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c013770 con 0x7fd530151e80 2023-03-03T13:12:37.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.626+0000 7fd53785b700 1 -- 172.21.15.7:0/1192978803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd530154ef0 con 0x7fd530151e80 2023-03-03T13:12:37.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.626+0000 7fd525ffb700 1 -- 172.21.15.7:0/1192978803 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c013b20 con 0x7fd530151e80 2023-03-03T13:12:37.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.626+0000 7fd53785b700 1 -- 172.21.15.7:0/1192978803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd530154c10 con 0x7fd530151e80 2023-03-03T13:12:37.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.627+0000 7fd53785b700 1 -- 172.21.15.7:0/1192978803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5300c5220 con 0x7fd530151e80 2023-03-03T13:12:37.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.627+0000 7fd525ffb700 1 -- 172.21.15.7:0/1192978803 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd52c023070 con 0x7fd530151e80 2023-03-03T13:12:37.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.628+0000 7fd525ffb700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5140682f0 0x7fd51406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:12:37.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.628+0000 7fd525ffb700 1 -- 172.21.15.7:0/1192978803 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd52c085de0 con 0x7fd530151e80 2023-03-03T13:12:37.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.628+0000 7fd5355f7700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5140682f0 0x7fd51406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:12:37.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.628+0000 7fd5355f7700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5140682f0 0x7fd51406a7a0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fd5300c84b0 tx=0x7fd520008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:12:37.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.632+0000 7fd525ffb700 1 -- 172.21.15.7:0/1192978803 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd52c052180 con 0x7fd530151e80 2023-03-03T13:12:37.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.870+0000 7fd53785b700 1 -- 172.21.15.7:0/1192978803 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5301573b0 con 0x7fd530151e80 2023-03-03T13:12:37.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.870+0000 7fd525ffb700 1 -- 172.21.15.7:0/1192978803 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd5301573b0 con 0x7fd530151e80 2023-03-03T13:12:37.872 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:12:37.872 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:12:37.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 -- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5140682f0 msgr2=0x7fd51406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:37.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5140682f0 0x7fd51406a7a0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fd5300c84b0 tx=0x7fd520008030 comp rx=0 tx=0).stop 2023-03-03T13:12:37.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 -- 172.21.15.7:0/1192978803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 msgr2=0x7fd5300bf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:12:37.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 0x7fd5300bf940 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00e840 tx=0x7fd52c00ec00 comp rx=0 tx=0).stop 2023-03-03T13:12:37.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 -- 172.21.15.7:0/1192978803 shutdown_connections 2023-03-03T13:12:37.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd530155410 0x7fd530157690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:37.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5300c6fe0 0x7fd53014d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:37.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5140682f0 0x7fd51406a7a0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:37.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 --2- 172.21.15.7:0/1192978803 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd530151e80 0x7fd5300bf940 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:12:37.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 -- 172.21.15.7:0/1192978803 >> 172.21.15.7:0/1192978803 conn(0x7fd530071630 msgr2=0x7fd5300c40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:12:37.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 -- 172.21.15.7:0/1192978803 shutdown_connections 2023-03-03T13:12:37.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:12:37.873+0000 7fd53785b700 1 -- 172.21.15.7:0/1192978803 wait complete. 2023-03-03T13:12:37.876 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:12:37.886 INFO:tasks.mds_thrash.fs.[a]:waiting for 119 secs before thrashing 2023-03-03T13:12:38.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:12:38.625 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:08.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:08.762 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:35.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:13:35.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.303+0000 7f852199f700 1 Processor -- start 2023-03-03T13:13:35.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.303+0000 7f852199f700 1 -- start start 2023-03-03T13:13:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.303+0000 7f852199f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f851c0c00e0 0x7f851c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.303+0000 7f852199f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f851c154ff0 0x7f851c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.303+0000 7f852199f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851c0c7310 con 0x7f851c0caa30 2023-03-03T13:13:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.303+0000 7f852199f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f851c155430 con 0x7f851c0c00e0 2023-03-03T13:13:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.303+0000 7f852199f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f851c0bfd10 con 0x7f851c154ff0 2023-03-03T13:13:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f851c0c00e0 0x7f851c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f851c0caa30 0x7f851c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37746/0 (socket says 172.21.15.7:37746) 2023-03-03T13:13:35.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f851c0c00e0 0x7f851c0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43768/0 (socket says 172.21.15.7:43768) 2023-03-03T13:13:35.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851affd700 1 -- 172.21.15.7:0/2468348291 learned_addr learned my addr 172.21.15.7:0/2468348291 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:13:35.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851b7fe700 1 --2- 172.21.15.7:0/2468348291 >> v2:172.21.15.143:3300/0 conn(0x7f851c154ff0 0x7f851c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:35.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851a7fc700 1 -- 172.21.15.7:0/2468348291 >> v2:172.21.15.143:3300/0 conn(0x7f851c154ff0 msgr2=0x7f851c0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:13:35.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851a7fc700 1 --2- 172.21.15.7:0/2468348291 >> v2:172.21.15.143:3300/0 conn(0x7f851c154ff0 0x7f851c0cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:35.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851a7fc700 1 -- 172.21.15.7:0/2468348291 >> v1:172.21.15.7:6789/0 conn(0x7f851c0caa30 legacy=0x7f851c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:35.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851a7fc700 1 -- 172.21.15.7:0/2468348291 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c0bfa30 con 0x7f851c0c00e0 2023-03-03T13:13:35.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f8519ffb700 1 -- 172.21.15.7:0/2468348291 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480845943 0 0) 0x7f851c0c7310 con 0x7f851c0caa30 2023-03-03T13:13:35.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851b7fe700 1 --2- 172.21.15.7:0/2468348291 >> v2:172.21.15.143:3300/0 conn(0x7f851c154ff0 0x7f851c0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:13:35.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f851a7fc700 1 --2- 172.21.15.7:0/2468348291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 0x7f851c0c5c60 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f851c071530 tx=0x7f8510009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8a5393b4c4b7e9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:13:35.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f8519ffb700 1 -- 172.21.15.7:0/2468348291 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8510005e30 con 0x7f851c0c00e0 2023-03-03T13:13:35.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f8519ffb700 1 -- 172.21.15.7:0/2468348291 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8510006030 con 0x7f851c0c00e0 2023-03-03T13:13:35.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.304+0000 7f8519ffb700 1 -- 172.21.15.7:0/2468348291 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85100063c0 con 0x7f851c0c00e0 2023-03-03T13:13:35.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.305+0000 7f852199f700 1 -- 172.21.15.7:0/2468348291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 msgr2=0x7f851c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:35.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.305+0000 7f852199f700 1 --2- 172.21.15.7:0/2468348291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 0x7f851c0c5c60 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f851c071530 tx=0x7f8510009b00 comp rx=0 tx=0).stop 2023-03-03T13:13:35.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.305+0000 7f852199f700 1 -- 172.21.15.7:0/2468348291 shutdown_connections 2023-03-03T13:13:35.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.305+0000 7f852199f700 1 --2- 172.21.15.7:0/2468348291 >> v2:172.21.15.143:3300/0 conn(0x7f851c154ff0 0x7f851c0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:35.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.305+0000 7f852199f700 1 --2- 172.21.15.7:0/2468348291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 0x7f851c0c5c60 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:35.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.305+0000 7f852199f700 1 -- 172.21.15.7:0/2468348291 >> 172.21.15.7:0/2468348291 conn(0x7f851c066e40 msgr2=0x7f851c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:13:35.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.305+0000 7f852199f700 1 -- 172.21.15.7:0/2468348291 shutdown_connections 2023-03-03T13:13:35.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.305+0000 7f852199f700 1 -- 172.21.15.7:0/2468348291 wait complete. 2023-03-03T13:13:35.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.306+0000 7f852199f700 1 Processor -- start 2023-03-03T13:13:35.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.306+0000 7f852199f700 1 -- start start 2023-03-03T13:13:35.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f852199f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 0x7f851c1dda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:35.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f852199f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f851c0caa30 0x7f851c1ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:35.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f852199f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851c154ff0 0x7f851c1ed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:35.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f852199f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c0c23d0 con 0x7f851c0c00e0 2023-03-03T13:13:35.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f852199f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c14d480 con 0x7f851c154ff0 2023-03-03T13:13:35.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f852199f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851c14d760 con 0x7f851c0caa30 2023-03-03T13:13:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 0x7f851c1dda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 0x7f851c1dda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43772/0 (socket says 172.21.15.7:43772) 2023-03-03T13:13:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851affd700 1 -- 172.21.15.7:0/510561014 learned_addr learned my addr 172.21.15.7:0/510561014 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:13:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851b7fe700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851c154ff0 0x7f851c1ed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851affd700 1 -- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f851c0caa30 msgr2=0x7f851c1ed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851affd700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f851c0caa30 0x7f851c1ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851affd700 1 -- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851c154ff0 msgr2=0x7f851c1ed7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:35.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851affd700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851c154ff0 0x7f851c1ed7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851affd700 1 -- 172.21.15.7:0/510561014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8510009430 con 0x7f851c0c00e0 2023-03-03T13:13:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851b7fe700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851c154ff0 0x7f851c1ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:13:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.307+0000 7f851affd700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 0x7f851c1dda40 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f850c0011e0 tx=0x7f850c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:13:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.308+0000 7f8503fff700 1 -- 172.21.15.7:0/510561014 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c004c80 con 0x7f851c0c00e0 2023-03-03T13:13:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.308+0000 7f8503fff700 1 -- 172.21.15.7:0/510561014 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c004e00 con 0x7f851c0c00e0 2023-03-03T13:13:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.308+0000 7f852199f700 1 -- 172.21.15.7:0/510561014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851c1547f0 con 0x7f851c0c00e0 2023-03-03T13:13:35.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.308+0000 7f8503fff700 1 -- 172.21.15.7:0/510561014 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c007310 con 0x7f851c0c00e0 2023-03-03T13:13:35.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.308+0000 7f852199f700 1 -- 172.21.15.7:0/510561014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851c154510 con 0x7f851c0c00e0 2023-03-03T13:13:35.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.309+0000 7f852199f700 1 -- 172.21.15.7:0/510561014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851c154ad0 con 0x7f851c0c00e0 2023-03-03T13:13:35.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.309+0000 7f8503fff700 1 -- 172.21.15.7:0/510561014 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f850c01a020 con 0x7f851c0c00e0 2023-03-03T13:13:35.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.310+0000 7f8503fff700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85080682f0 0x7f850806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:13:35.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.310+0000 7f8503fff700 1 -- 172.21.15.7:0/510561014 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f850c085e20 con 0x7f851c0c00e0 2023-03-03T13:13:35.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.310+0000 7f851a7fc700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85080682f0 0x7f850806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:13:35.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.311+0000 7f851a7fc700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85080682f0 0x7f850806a7a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f851c158a40 tx=0x7f8510010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:13:35.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.314+0000 7f8503fff700 1 -- 172.21.15.7:0/510561014 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f850c0521c0 con 0x7f851c0c00e0 2023-03-03T13:13:35.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.552+0000 7f852199f700 1 -- 172.21.15.7:0/510561014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f851c0bfe50 con 0x7f851c0c00e0 2023-03-03T13:13:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.552+0000 7f8503fff700 1 -- 172.21.15.7:0/510561014 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f850c0557e0 con 0x7f851c0c00e0 2023-03-03T13:13:35.555 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:13:35.555 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:13:35.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 -- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85080682f0 msgr2=0x7f850806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:35.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85080682f0 0x7f850806a7a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f851c158a40 tx=0x7f8510010000 comp rx=0 tx=0).stop 2023-03-03T13:13:35.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 -- 172.21.15.7:0/510561014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 msgr2=0x7f851c1dda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:13:35.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 0x7f851c1dda40 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f850c0011e0 tx=0x7f850c00cae0 comp rx=0 tx=0).stop 2023-03-03T13:13:35.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 -- 172.21.15.7:0/510561014 shutdown_connections 2023-03-03T13:13:35.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f851c154ff0 0x7f851c1ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:35.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85080682f0 0x7f850806a7a0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:35.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f851c0c00e0 0x7f851c1dda40 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:35.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 --2- 172.21.15.7:0/510561014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f851c0caa30 0x7f851c1ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:13:35.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 -- 172.21.15.7:0/510561014 >> 172.21.15.7:0/510561014 conn(0x7f851c066e40 msgr2=0x7f851c153920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:13:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 -- 172.21.15.7:0/510561014 shutdown_connections 2023-03-03T13:13:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:13:35.554+0000 7f852199f700 1 -- 172.21.15.7:0/510561014 wait complete. 2023-03-03T13:13:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:13:35.569 INFO:tasks.mds_thrash.fs.[b]:waiting for 106 secs before thrashing 2023-03-03T13:13:38.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:13:38.917 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:09.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:09.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:36.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:14:37.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.023+0000 7fcdaa6ec700 1 Processor -- start 2023-03-03T13:14:37.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.023+0000 7fcdaa6ec700 1 -- start start 2023-03-03T13:14:37.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.024+0000 7fcdaa6ec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcda4072250 0x7fcda414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:37.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.024+0000 7fcdaa6ec700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda41510d0 con 0x7fcda4071e80 2023-03-03T13:14:37.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.024+0000 7fcdaa6ec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda40c3740 con 0x7fcda40c1ce0 2023-03-03T13:14:37.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.024+0000 7fcdaa6ec700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda4070670 con 0x7fcda4072250 2023-03-03T13:14:37.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.024+0000 7fcda3fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcda4071e80 0x7fcda4151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37790/0 (socket says 172.21.15.7:37790) 2023-03-03T13:14:37.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.024+0000 7fcda37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcda4072250 0x7fcda414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:37.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.024+0000 7fcda3fff700 1 -- 172.21.15.7:0/1180158088 learned_addr learned my addr 172.21.15.7:0/1180158088 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:14:37.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.024+0000 7fcda37fe700 1 -- 172.21.15.7:0/1180158088 >> v1:172.21.15.143:6789/0 conn(0x7fcda40c1ce0 legacy=0x7fcda415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:14:37.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.024+0000 7fcda37fe700 1 -- 172.21.15.7:0/1180158088 >> v1:172.21.15.7:6789/0 conn(0x7fcda4071e80 legacy=0x7fcda4151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:37.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.024+0000 7fcda37fe700 1 -- 172.21.15.7:0/1180158088 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda40c39e0 con 0x7fcda4072250 2023-03-03T13:14:37.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcda37fe700 1 --2- 172.21.15.7:0/1180158088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda4072250 0x7fcda414e1f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fcd98009870 tx=0x7fcd98009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6dd5c04224cd20 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:14:37.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcda2ffd700 1 -- 172.21.15.7:0/1180158088 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd98004230 con 0x7fcda4072250 2023-03-03T13:14:37.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcda2ffd700 1 -- 172.21.15.7:0/1180158088 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd98004390 con 0x7fcda4072250 2023-03-03T13:14:37.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcda2ffd700 1 -- 172.21.15.7:0/1180158088 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd980036a0 con 0x7fcda4072250 2023-03-03T13:14:37.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1180158088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda4072250 msgr2=0x7fcda414e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:37.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcdaa6ec700 1 --2- 172.21.15.7:0/1180158088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda4072250 0x7fcda414e1f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fcd98009870 tx=0x7fcd98009b80 comp rx=0 tx=0).stop 2023-03-03T13:14:37.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1180158088 shutdown_connections 2023-03-03T13:14:37.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcdaa6ec700 1 --2- 172.21.15.7:0/1180158088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda4072250 0x7fcda414e1f0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:37.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1180158088 >> 172.21.15.7:0/1180158088 conn(0x7fcda40a94d0 msgr2=0x7fcda40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:14:37.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1180158088 shutdown_connections 2023-03-03T13:14:37.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.025+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1180158088 wait complete. 2023-03-03T13:14:37.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.027+0000 7fcdaa6ec700 1 Processor -- start 2023-03-03T13:14:37.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.027+0000 7fcdaa6ec700 1 -- start start 2023-03-03T13:14:37.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcdaa6ec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda4071e80 0x7fcda41400b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcdaa6ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda4072250 0x7fcda414f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcdaa6ec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda40c1ce0 0x7fcda41e7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcdaa6ec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda4145bc0 con 0x7fcda40c1ce0 2023-03-03T13:14:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcdaa6ec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda41e7870 con 0x7fcda4071e80 2023-03-03T13:14:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcdaa6ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda40bee20 con 0x7fcda4072250 2023-03-03T13:14:37.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcda8c89700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda40c1ce0 0x7fcda41e7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcda8c89700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda40c1ce0 0x7fcda41e7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43818/0 (socket says 172.21.15.7:43818) 2023-03-03T13:14:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcda8c89700 1 -- 172.21.15.7:0/1124994697 learned_addr learned my addr 172.21.15.7:0/1124994697 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:14:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcda3fff700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda4071e80 0x7fcda41400b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.028+0000 7fcda37fe700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda4072250 0x7fcda414f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcda8c89700 1 -- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda4072250 msgr2=0x7fcda414f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcda8c89700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda4072250 0x7fcda414f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcda8c89700 1 -- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda4071e80 msgr2=0x7fcda41400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:37.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcda8c89700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda4071e80 0x7fcda41400b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:37.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcda8c89700 1 -- 172.21.15.7:0/1124994697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd980094a0 con 0x7fcda40c1ce0 2023-03-03T13:14:37.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcda8c89700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda40c1ce0 0x7fcda41e7cc0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00b9e0 tx=0x7fcd9c00bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:14:37.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcda0ff9700 1 -- 172.21.15.7:0/1124994697 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c003c00 con 0x7fcda40c1ce0 2023-03-03T13:14:37.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcda0ff9700 1 -- 172.21.15.7:0/1124994697 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c003da0 con 0x7fcda40c1ce0 2023-03-03T13:14:37.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1124994697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda4151f30 con 0x7fcda40c1ce0 2023-03-03T13:14:37.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcda0ff9700 1 -- 172.21.15.7:0/1124994697 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c00d920 con 0x7fcda40c1ce0 2023-03-03T13:14:37.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.029+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1124994697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda4151300 con 0x7fcda40c1ce0 2023-03-03T13:14:37.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.030+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1124994697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda406c9a0 con 0x7fcda40c1ce0 2023-03-03T13:14:37.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.030+0000 7fcda0ff9700 1 -- 172.21.15.7:0/1124994697 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcd9c01b020 con 0x7fcda40c1ce0 2023-03-03T13:14:37.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.031+0000 7fcda0ff9700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd900682f0 0x7fcd9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:14:37.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.031+0000 7fcda0ff9700 1 -- 172.21.15.7:0/1124994697 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcd9c011070 con 0x7fcda40c1ce0 2023-03-03T13:14:37.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.031+0000 7fcda3fff700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd900682f0 0x7fcd9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:14:37.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.032+0000 7fcda3fff700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd900682f0 0x7fcd9006a7a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005910 tx=0x7fcd94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:14:37.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.035+0000 7fcda0ff9700 1 -- 172.21.15.7:0/1124994697 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcd9c052340 con 0x7fcda40c1ce0 2023-03-03T13:14:37.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.273+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1124994697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcda4074bb0 con 0x7fcda40c1ce0 2023-03-03T13:14:37.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.274+0000 7fcda0ff9700 1 -- 172.21.15.7:0/1124994697 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcd9c055960 con 0x7fcda40c1ce0 2023-03-03T13:14:37.277 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:14:37.277 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:14:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd900682f0 msgr2=0x7fcd9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd900682f0 0x7fcd9006a7a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005910 tx=0x7fcd94009040 comp rx=0 tx=0).stop 2023-03-03T13:14:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1124994697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda40c1ce0 msgr2=0x7fcda41e7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:14:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda40c1ce0 0x7fcda41e7cc0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00b9e0 tx=0x7fcd9c00bcf0 comp rx=0 tx=0).stop 2023-03-03T13:14:37.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1124994697 shutdown_connections 2023-03-03T13:14:37.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd900682f0 0x7fcd9006a7a0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:37.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcda40c1ce0 0x7fcda41e7cc0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:37.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcda4071e80 0x7fcda41400b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:37.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 --2- 172.21.15.7:0/1124994697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcda4072250 0x7fcda414f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:14:37.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1124994697 >> 172.21.15.7:0/1124994697 conn(0x7fcda40a94d0 msgr2=0x7fcda4146760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:14:37.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1124994697 shutdown_connections 2023-03-03T13:14:37.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:14:37.276+0000 7fcdaa6ec700 1 -- 172.21.15.7:0/1124994697 wait complete. 2023-03-03T13:14:37.281 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:14:37.291 INFO:tasks.mds_thrash.fs.[a]:waiting for 94 secs before thrashing 2023-03-03T13:14:39.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:14:39.077 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:09.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:09.225 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:21.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:15:21.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.707+0000 7fe8653b1700 1 Processor -- start 2023-03-03T13:15:21.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.708+0000 7fe8653b1700 1 -- start start 2023-03-03T13:15:21.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.708+0000 7fe8653b1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8600c00e0 0x7fe8600c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:21.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.708+0000 7fe8653b1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe860154ff0 0x7fe8600cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:21.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.708+0000 7fe8653b1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8600c7310 con 0x7fe8600caa30 2023-03-03T13:15:21.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.708+0000 7fe8653b1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8600bfd10 con 0x7fe860154ff0 2023-03-03T13:15:21.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.708+0000 7fe8653b1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600bfa30 con 0x7fe8600c00e0 2023-03-03T13:15:21.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.708+0000 7fe85effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe8600caa30 0x7fe8600c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37828/0 (socket says 172.21.15.7:37828) 2023-03-03T13:15:21.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.708+0000 7fe85effd700 1 -- 172.21.15.7:0/4109160571 learned_addr learned my addr 172.21.15.7:0/4109160571 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:15:21.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85f7fe700 1 --2- 172.21.15.7:0/4109160571 >> v2:172.21.15.143:3300/0 conn(0x7fe860154ff0 0x7fe8600cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:21.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85e7fc700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8600c00e0 0x7fe8600c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:21.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547345513 0 0) 0x7fe8600c7310 con 0x7fe8600caa30 2023-03-03T13:15:21.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8480035b0 con 0x7fe8600caa30 2023-03-03T13:15:21.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3915530864 0 0) 0x7fe8480035b0 con 0x7fe8600caa30 2023-03-03T13:15:21.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8600c00e0 msgr2=0x7fe8600c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:15:21.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8600c00e0 0x7fe8600c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:21.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 >> v2:172.21.15.143:3300/0 conn(0x7fe860154ff0 msgr2=0x7fe8600cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:15:21.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 --2- 172.21.15.7:0/4109160571 >> v2:172.21.15.143:3300/0 conn(0x7fe860154ff0 0x7fe8600cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:21.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe860154510 con 0x7fe8600caa30 2023-03-03T13:15:21.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe854001670 con 0x7fe8600caa30 2023-03-03T13:15:21.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 >> v1:172.21.15.7:6789/0 conn(0x7fe8600caa30 legacy=0x7fe8600c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:21.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe848003a70 0x7fe848003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:21.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe848004850 0x7fe848004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:21.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8600c00e0 0x7fe848005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:21.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600bfd10 con 0x7fe848004850 2023-03-03T13:15:21.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600bfa30 con 0x7fe848003a70 2023-03-03T13:15:21.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84800ba40 con 0x7fe8600c00e0 2023-03-03T13:15:21.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85e7fc700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe848004850 0x7fe848004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:21.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85effd700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe848003a70 0x7fe848003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:21.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.709+0000 7fe85f7fe700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8600c00e0 0x7fe848005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:21.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.710+0000 7fe85e7fc700 1 -- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8600c00e0 msgr2=0x7fe848005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:15:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.710+0000 7fe85e7fc700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8600c00e0 0x7fe848005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.710+0000 7fe85e7fc700 1 -- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe848003a70 msgr2=0x7fe848003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:15:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.710+0000 7fe85e7fc700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe848003a70 0x7fe848003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.710+0000 7fe85e7fc700 1 -- 172.21.15.7:0/4109160571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8480035b0 con 0x7fe848004850 2023-03-03T13:15:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.710+0000 7fe85effd700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe848003a70 0x7fe848003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:15:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.710+0000 7fe85e7fc700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe848004850 0x7fe848004cc0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fe850009b40 tx=0x7fe848005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd79140e5a81547 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:15:21.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.710+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe850005a20 con 0x7fe848004850 2023-03-03T13:15:21.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.710+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe850005b80 con 0x7fe848004850 2023-03-03T13:15:21.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.710+0000 7fe85dffb700 1 -- 172.21.15.7:0/4109160571 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85000e6c0 con 0x7fe848004850 2023-03-03T13:15:21.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.711+0000 7fe8653b1700 1 -- 172.21.15.7:0/4109160571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe848004850 msgr2=0x7fe848004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:21.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.711+0000 7fe8653b1700 1 --2- 172.21.15.7:0/4109160571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe848004850 0x7fe848004cc0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fe850009b40 tx=0x7fe848005880 comp rx=0 tx=0).stop 2023-03-03T13:15:21.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.711+0000 7fe85f7fe700 1 -- 172.21.15.7:0/4109160571 reap_dead start 2023-03-03T13:15:21.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.711+0000 7fe8653b1700 1 -- 172.21.15.7:0/4109160571 shutdown_connections 2023-03-03T13:15:21.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.711+0000 7fe8653b1700 1 -- 172.21.15.7:0/4109160571 >> 172.21.15.7:0/4109160571 conn(0x7fe860066e40 msgr2=0x7fe860071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:15:21.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.711+0000 7fe8653b1700 1 -- 172.21.15.7:0/4109160571 shutdown_connections 2023-03-03T13:15:21.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.711+0000 7fe8653b1700 1 -- 172.21.15.7:0/4109160571 wait complete. 2023-03-03T13:15:21.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.712+0000 7fe8653b1700 1 Processor -- start 2023-03-03T13:15:21.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.712+0000 7fe8653b1700 1 -- start start 2023-03-03T13:15:21.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe8653b1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe86006b550 0x7fe8601eb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:21.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe8653b1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86006b920 0x7fe8601eb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:21.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe8653b1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8601ebe40 0x7fe860156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:21.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe8653b1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8601eda40 con 0x7fe86006b550 2023-03-03T13:15:21.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe8653b1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8601edf70 con 0x7fe86006b920 2023-03-03T13:15:21.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe8653b1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86014a000 con 0x7fe8601ebe40 2023-03-03T13:15:21.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe86006b550 0x7fe8601eb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:21.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe86006b550 0x7fe8601eb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43860/0 (socket says 172.21.15.7:43860) 2023-03-03T13:15:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85effd700 1 -- 172.21.15.7:0/2387779673 learned_addr learned my addr 172.21.15.7:0/2387779673 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:15:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85f7fe700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8601ebe40 0x7fe860156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85e7fc700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86006b920 0x7fe8601eb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85effd700 1 -- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8601ebe40 msgr2=0x7fe860156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85effd700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8601ebe40 0x7fe860156630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85effd700 1 -- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86006b920 msgr2=0x7fe8601eb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:21.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85effd700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86006b920 0x7fe8601eb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85effd700 1 -- 172.21.15.7:0/2387779673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe860154ad0 con 0x7fe86006b550 2023-03-03T13:15:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85f7fe700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8601ebe40 0x7fe860156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:15:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85e7fc700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86006b920 0x7fe8601eb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:15:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.713+0000 7fe85effd700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe86006b550 0x7fe8601eb3c0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fe854007340 tx=0x7fe854007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:15:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.714+0000 7fe8477fe700 1 -- 172.21.15.7:0/2387779673 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85400f870 con 0x7fe86006b550 2023-03-03T13:15:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.714+0000 7fe8477fe700 1 -- 172.21.15.7:0/2387779673 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85400f9d0 con 0x7fe86006b550 2023-03-03T13:15:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.714+0000 7fe8653b1700 1 -- 172.21.15.7:0/2387779673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8601547f0 con 0x7fe86006b550 2023-03-03T13:15:21.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.714+0000 7fe8477fe700 1 -- 172.21.15.7:0/2387779673 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85400fd40 con 0x7fe86006b550 2023-03-03T13:15:21.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.714+0000 7fe8653b1700 1 -- 172.21.15.7:0/2387779673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8600bf650 con 0x7fe86006b550 2023-03-03T13:15:21.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.715+0000 7fe8653b1700 1 -- 172.21.15.7:0/2387779673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8601eb210 con 0x7fe86006b550 2023-03-03T13:15:21.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.715+0000 7fe8477fe700 1 -- 172.21.15.7:0/2387779673 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe85401d460 con 0x7fe86006b550 2023-03-03T13:15:21.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.715+0000 7fe8477fe700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe84c0682f0 0x7fe84c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:15:21.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.716+0000 7fe8477fe700 1 -- 172.21.15.7:0/2387779673 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe8540864a0 con 0x7fe86006b550 2023-03-03T13:15:21.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.716+0000 7fe85e7fc700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe84c0682f0 0x7fe84c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:15:21.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.717+0000 7fe85e7fc700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe84c0682f0 0x7fe84c06a7a0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe86004f920 tx=0x7fe850008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:15:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.720+0000 7fe8477fe700 1 -- 172.21.15.7:0/2387779673 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8540527e0 con 0x7fe86006b550 2023-03-03T13:15:21.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.958+0000 7fe8653b1700 1 -- 172.21.15.7:0/2387779673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe860074bb0 con 0x7fe86006b550 2023-03-03T13:15:21.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.959+0000 7fe8477fe700 1 -- 172.21.15.7:0/2387779673 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe85400fea0 con 0x7fe86006b550 2023-03-03T13:15:21.961 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:15:21.961 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:15:21.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.960+0000 7fe8653b1700 1 -- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe84c0682f0 msgr2=0x7fe84c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:21.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.960+0000 7fe8653b1700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe84c0682f0 0x7fe84c06a7a0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe86004f920 tx=0x7fe850008030 comp rx=0 tx=0).stop 2023-03-03T13:15:21.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.960+0000 7fe8653b1700 1 -- 172.21.15.7:0/2387779673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe86006b550 msgr2=0x7fe8601eb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:15:21.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.960+0000 7fe8653b1700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe86006b550 0x7fe8601eb3c0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fe854007340 tx=0x7fe854007700 comp rx=0 tx=0).stop 2023-03-03T13:15:21.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.960+0000 7fe8653b1700 1 -- 172.21.15.7:0/2387779673 shutdown_connections 2023-03-03T13:15:21.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.960+0000 7fe8653b1700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8601ebe40 0x7fe860156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:21.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.960+0000 7fe8653b1700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe84c0682f0 0x7fe84c06a7a0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:21.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.960+0000 7fe8653b1700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe86006b550 0x7fe8601eb3c0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:21.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.961+0000 7fe8653b1700 1 --2- 172.21.15.7:0/2387779673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86006b920 0x7fe8601eb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:15:21.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.961+0000 7fe8653b1700 1 -- 172.21.15.7:0/2387779673 >> 172.21.15.7:0/2387779673 conn(0x7fe860066e40 msgr2=0x7fe8600c91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:15:21.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.961+0000 7fe8653b1700 1 -- 172.21.15.7:0/2387779673 shutdown_connections 2023-03-03T13:15:21.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:15:21.961+0000 7fe8653b1700 1 -- 172.21.15.7:0/2387779673 wait complete. 2023-03-03T13:15:21.964 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:15:21.974 INFO:tasks.mds_thrash.fs.[b]:waiting for 102 secs before thrashing 2023-03-03T13:15:39.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:15:39.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:09.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:09.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:11.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:16:11.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.430+0000 7ff4eb3d7700 1 Processor -- start 2023-03-03T13:16:11.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.430+0000 7ff4eb3d7700 1 -- start start 2023-03-03T13:16:11.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4eb3d7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff4e4151e80 0x7ff4e40c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:11.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4eb3d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4e4155410 0x7ff4e40bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4eb3d7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff4e40c6fe0 0x7ff4e40bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4eb3d7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e4152250 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4eb3d7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e40c0fb0 con 0x7ff4e4151e80 2023-03-03T13:16:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4eb3d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4154740 con 0x7ff4e4155410 2023-03-03T13:16:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4e9974700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff4e40c6fe0 0x7ff4e40bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4e9974700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff4e40c6fe0 0x7ff4e40bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43902/0 (socket says 172.21.15.7:43902) 2023-03-03T13:16:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4e9974700 1 -- 172.21.15.7:0/2528264842 learned_addr learned my addr 172.21.15.7:0/2528264842 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:16:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4e9173700 1 --2- 172.21.15.7:0/2528264842 >> v2:172.21.15.143:3300/0 conn(0x7ff4e4151e80 0x7ff4e40c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4e8972700 1 --2- 172.21.15.7:0/2528264842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4e4155410 0x7ff4e40bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4e9974700 1 -- 172.21.15.7:0/2528264842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4e4155410 msgr2=0x7ff4e40bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:16:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4e9974700 1 --2- 172.21.15.7:0/2528264842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4e4155410 0x7ff4e40bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.431+0000 7ff4e9974700 1 -- 172.21.15.7:0/2528264842 >> v2:172.21.15.143:3300/0 conn(0x7ff4e4151e80 msgr2=0x7ff4e40c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:16:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4e9974700 1 --2- 172.21.15.7:0/2528264842 >> v2:172.21.15.143:3300/0 conn(0x7ff4e4151e80 0x7ff4e40c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4e9974700 1 -- 172.21.15.7:0/2528264842 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e40c08d0 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4e9974700 1 --2- 172.21.15.7:0/2528264842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 0x7ff4e40bfe20 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40cb250 tx=0x7ff4dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa8bd2f7f9766f3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:16:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4dbfff700 1 -- 172.21.15.7:0/2528264842 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc00e070 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4dbfff700 1 -- 172.21.15.7:0/2528264842 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc00b910 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4dbfff700 1 -- 172.21.15.7:0/2528264842 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc00bcc0 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/2528264842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 msgr2=0x7ff4e40bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4eb3d7700 1 --2- 172.21.15.7:0/2528264842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 0x7ff4e40bfe20 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40cb250 tx=0x7ff4dc00b330 comp rx=0 tx=0).stop 2023-03-03T13:16:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/2528264842 shutdown_connections 2023-03-03T13:16:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4eb3d7700 1 --2- 172.21.15.7:0/2528264842 >> v2:172.21.15.143:3300/0 conn(0x7ff4e4151e80 0x7ff4e40c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4eb3d7700 1 --2- 172.21.15.7:0/2528264842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 0x7ff4e40bfe20 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4eb3d7700 1 --2- 172.21.15.7:0/2528264842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4e4155410 0x7ff4e40bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.432+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/2528264842 >> 172.21.15.7:0/2528264842 conn(0x7ff4e4071630 msgr2=0x7ff4e406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:16:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.433+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/2528264842 shutdown_connections 2023-03-03T13:16:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.433+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/2528264842 wait complete. 2023-03-03T13:16:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.434+0000 7ff4eb3d7700 1 Processor -- start 2023-03-03T13:16:11.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.434+0000 7ff4eb3d7700 1 -- start start 2023-03-03T13:16:11.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.434+0000 7ff4eb3d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 0x7ff4e41537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:11.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.434+0000 7ff4eb3d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4e4151e80 0x7ff4e414a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:11.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.434+0000 7ff4eb3d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e4155410 0x7ff4e414a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:11.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.434+0000 7ff4eb3d7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e414ad90 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.434+0000 7ff4eb3d7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e41567a0 con 0x7ff4e4155410 2023-03-03T13:16:11.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.434+0000 7ff4eb3d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e40c8840 con 0x7ff4e4151e80 2023-03-03T13:16:11.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e9173700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 0x7ff4e41537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e9173700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 0x7ff4e41537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43904/0 (socket says 172.21.15.7:43904) 2023-03-03T13:16:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e9173700 1 -- 172.21.15.7:0/525161801 learned_addr learned my addr 172.21.15.7:0/525161801 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:16:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e8972700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4e4151e80 0x7ff4e414a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e9974700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e4155410 0x7ff4e414a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e9173700 1 -- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4e4151e80 msgr2=0x7ff4e414a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e9173700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4e4151e80 0x7ff4e414a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e9173700 1 -- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e4155410 msgr2=0x7ff4e414a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:11.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e9173700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e4155410 0x7ff4e414a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e9173700 1 -- 172.21.15.7:0/525161801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc009b30 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4e9173700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 0x7ff4e41537f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400e870 tx=0x7ff4d400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:16:11.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4d9ffb700 1 -- 172.21.15.7:0/525161801 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4009590 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4d9ffb700 1 -- 172.21.15.7:0/525161801 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d4013680 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4d9ffb700 1 -- 172.21.15.7:0/525161801 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d40139f0 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.435+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/525161801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e4154c10 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.436+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/525161801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e4154930 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.436+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/525161801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e4154ef0 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.437+0000 7ff4d9ffb700 1 -- 172.21.15.7:0/525161801 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff4d4019030 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.437+0000 7ff4d9ffb700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c80682f0 0x7ff4c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:16:11.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.437+0000 7ff4d9ffb700 1 -- 172.21.15.7:0/525161801 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff4d4085f00 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.437+0000 7ff4e8972700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c80682f0 0x7ff4c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:16:11.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.438+0000 7ff4e8972700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c80682f0 0x7ff4c806a7a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40c8600 tx=0x7ff4e0009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:16:11.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.442+0000 7ff4d9ffb700 1 -- 172.21.15.7:0/525161801 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff4d40521b0 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.680+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/525161801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4e40c8670 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.681+0000 7ff4d9ffb700 1 -- 172.21.15.7:0/525161801 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff4d40557d0 con 0x7ff4e40c6fe0 2023-03-03T13:16:11.682 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:16:11.683 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:16:11.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.682+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c80682f0 msgr2=0x7ff4c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:11.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.682+0000 7ff4eb3d7700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c80682f0 0x7ff4c806a7a0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40c8600 tx=0x7ff4e0009210 comp rx=0 tx=0).stop 2023-03-03T13:16:11.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.682+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/525161801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 msgr2=0x7ff4e41537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:16:11.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.682+0000 7ff4eb3d7700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 0x7ff4e41537f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400e870 tx=0x7ff4d400eb80 comp rx=0 tx=0).stop 2023-03-03T13:16:11.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.683+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/525161801 shutdown_connections 2023-03-03T13:16:11.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.683+0000 7ff4eb3d7700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4e4155410 0x7ff4e414a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.683+0000 7ff4eb3d7700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c80682f0 0x7ff4c806a7a0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.683+0000 7ff4eb3d7700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4e40c6fe0 0x7ff4e41537f0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.683+0000 7ff4eb3d7700 1 --2- 172.21.15.7:0/525161801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4e4151e80 0x7ff4e414a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:16:11.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.683+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/525161801 >> 172.21.15.7:0/525161801 conn(0x7ff4e4071630 msgr2=0x7ff4e40c32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:16:11.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.683+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/525161801 shutdown_connections 2023-03-03T13:16:11.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:16:11.683+0000 7ff4eb3d7700 1 -- 172.21.15.7:0/525161801 wait complete. 2023-03-03T13:16:11.686 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:16:11.696 INFO:tasks.mds_thrash.fs.[a]:waiting for 85 secs before thrashing 2023-03-03T13:16:39.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:16:39.665 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:03.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:17:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.113+0000 7f2251301700 1 Processor -- start 2023-03-03T13:17:04.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.113+0000 7f2251301700 1 -- start start 2023-03-03T13:17:04.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.114+0000 7f2251301700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c0caa30 0x7f224c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.114+0000 7f2251301700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224c0c7310 con 0x7f224c154ff0 2023-03-03T13:17:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.114+0000 7f2251301700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224c0bfd10 con 0x7f224c0c00e0 2023-03-03T13:17:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.114+0000 7f2251301700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0bfa30 con 0x7f224c0caa30 2023-03-03T13:17:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.114+0000 7f224b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f224c154ff0 0x7f224c15ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37916/0 (socket says 172.21.15.7:37916) 2023-03-03T13:17:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.114+0000 7f224b7fe700 1 -- 172.21.15.7:0/1018323306 learned_addr learned my addr 172.21.15.7:0/1018323306 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:17:04.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.114+0000 7f224affd700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c0caa30 0x7f224c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.114+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250181324 0 0) 0x7f224c0c7310 con 0x7f224c154ff0 2023-03-03T13:17:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22340035b0 con 0x7f224c154ff0 2023-03-03T13:17:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524582498 0 0) 0x7f224c0bfd10 con 0x7f224c0c00e0 2023-03-03T13:17:04.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f224c0c7310 con 0x7f224c0c00e0 2023-03-03T13:17:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095564773 0 0) 0x7f22340035b0 con 0x7f224c154ff0 2023-03-03T13:17:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c0caa30 msgr2=0x7f224c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:17:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c0caa30 0x7f224c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 >> v1:172.21.15.143:6789/0 conn(0x7f224c0c00e0 legacy=0x7f224c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c154510 con 0x7f224c154ff0 2023-03-03T13:17:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f22440019f0 con 0x7f224c154ff0 2023-03-03T13:17:04.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f224affd700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c0caa30 0x7f224c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:17:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 >> v1:172.21.15.7:6789/0 conn(0x7f224c154ff0 legacy=0x7f224c15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2234003c30 0x7f2234004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2234004850 0x7f2234004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c0caa30 0x7f2234005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0bfa30 con 0x7f2234003c30 2023-03-03T13:17:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223400bba0 con 0x7f2234004850 2023-03-03T13:17:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f223400bd60 con 0x7f224c0caa30 2023-03-03T13:17:04.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f224affd700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2234003c30 0x7f2234004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f224a7fc700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2234004850 0x7f2234004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f224b7fe700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c0caa30 0x7f2234005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f224affd700 1 -- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c0caa30 msgr2=0x7f2234005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:17:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f224affd700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c0caa30 0x7f2234005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f224affd700 1 -- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2234004850 msgr2=0x7f2234004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:17:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f224affd700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2234004850 0x7f2234004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:04.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f224affd700 1 -- 172.21.15.7:0/1018323306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c0094a0 con 0x7f2234003c30 2023-03-03T13:17:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.115+0000 7f224b7fe700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c0caa30 0x7f2234005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:17:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f224affd700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2234003c30 0x7f2234004020 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f223c009d30 tx=0x7f2234005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b853199e4e244d94 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c009e80 con 0x7f2234003c30 2023-03-03T13:17:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223c00e9a0 con 0x7f2234003c30 2023-03-03T13:17:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f2249ffb700 1 -- 172.21.15.7:0/1018323306 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c00ed10 con 0x7f2234003c30 2023-03-03T13:17:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f2251301700 1 -- 172.21.15.7:0/1018323306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2234003c30 msgr2=0x7f2234004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f2251301700 1 --2- 172.21.15.7:0/1018323306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2234003c30 0x7f2234004020 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f223c009d30 tx=0x7f2234005880 comp rx=0 tx=0).stop 2023-03-03T13:17:04.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f224b7fe700 1 -- 172.21.15.7:0/1018323306 reap_dead start 2023-03-03T13:17:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f2251301700 1 -- 172.21.15.7:0/1018323306 shutdown_connections 2023-03-03T13:17:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f2251301700 1 -- 172.21.15.7:0/1018323306 >> 172.21.15.7:0/1018323306 conn(0x7f224c066e40 msgr2=0x7f224c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:17:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f2251301700 1 -- 172.21.15.7:0/1018323306 shutdown_connections 2023-03-03T13:17:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.116+0000 7f2251301700 1 -- 172.21.15.7:0/1018323306 wait complete. 2023-03-03T13:17:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f2251301700 1 Processor -- start 2023-03-03T13:17:04.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f2251301700 1 -- start start 2023-03-03T13:17:04.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f2251301700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2234003c30 0x7f224c153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:04.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f2251301700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f224c150740 0x7f224c148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:04.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f2251301700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c1493e0 0x7f224c0c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:04.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f2251301700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c1ea730 con 0x7f224c150740 2023-03-03T13:17:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f2251301700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c149880 con 0x7f2234003c30 2023-03-03T13:17:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f2251301700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c1e73a0 con 0x7f224c1493e0 2023-03-03T13:17:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f224a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f224c150740 0x7f224c148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f224a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f224c150740 0x7f224c148ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43946/0 (socket says 172.21.15.7:43946) 2023-03-03T13:17:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f224a7fc700 1 -- 172.21.15.7:0/2090008604 learned_addr learned my addr 172.21.15.7:0/2090008604 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:17:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f224affd700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2234003c30 0x7f224c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:04.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.118+0000 7f224b7fe700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c1493e0 0x7f224c0c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f224a7fc700 1 -- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c1493e0 msgr2=0x7f224c0c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f224a7fc700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c1493e0 0x7f224c0c3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f224a7fc700 1 -- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2234003c30 msgr2=0x7f224c153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f224a7fc700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2234003c30 0x7f224c153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f224a7fc700 1 -- 172.21.15.7:0/2090008604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c154ad0 con 0x7f224c150740 2023-03-03T13:17:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f224a7fc700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f224c150740 0x7f224c148ea0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f224000f7c0 tx=0x7f224000fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f22337fe700 1 -- 172.21.15.7:0/2090008604 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22400052e0 con 0x7f224c150740 2023-03-03T13:17:04.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f22337fe700 1 -- 172.21.15.7:0/2090008604 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2240005440 con 0x7f224c150740 2023-03-03T13:17:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f22337fe700 1 -- 172.21.15.7:0/2090008604 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240003910 con 0x7f224c150740 2023-03-03T13:17:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f2251301700 1 -- 172.21.15.7:0/2090008604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c1547f0 con 0x7f224c150740 2023-03-03T13:17:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.119+0000 7f2251301700 1 -- 172.21.15.7:0/2090008604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c0bf650 con 0x7f224c150740 2023-03-03T13:17:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.120+0000 7f2251301700 1 -- 172.21.15.7:0/2090008604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c0c3850 con 0x7f224c150740 2023-03-03T13:17:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.120+0000 7f22337fe700 1 -- 172.21.15.7:0/2090008604 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2240003a70 con 0x7f224c150740 2023-03-03T13:17:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.121+0000 7f22337fe700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22380682f0 0x7f223806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:04.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.121+0000 7f22337fe700 1 -- 172.21.15.7:0/2090008604 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2240086040 con 0x7f224c150740 2023-03-03T13:17:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.121+0000 7f224affd700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22380682f0 0x7f223806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.122+0000 7f224affd700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22380682f0 0x7f223806a7a0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f223c0090b0 tx=0x7f223c01e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:04.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.125+0000 7f22337fe700 1 -- 172.21.15.7:0/2090008604 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f22400523e0 con 0x7f224c150740 2023-03-03T13:17:04.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.364+0000 7f2251301700 1 -- 172.21.15.7:0/2090008604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f224c074bb0 con 0x7f224c150740 2023-03-03T13:17:04.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.364+0000 7f22337fe700 1 -- 172.21.15.7:0/2090008604 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2240055a00 con 0x7f224c150740 2023-03-03T13:17:04.366 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:17:04.366 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:17:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 -- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22380682f0 msgr2=0x7f223806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22380682f0 0x7f223806a7a0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f223c0090b0 tx=0x7f223c01e040 comp rx=0 tx=0).stop 2023-03-03T13:17:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 -- 172.21.15.7:0/2090008604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f224c150740 msgr2=0x7f224c148ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f224c150740 0x7f224c148ea0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f224000f7c0 tx=0x7f224000fad0 comp rx=0 tx=0).stop 2023-03-03T13:17:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 -- 172.21.15.7:0/2090008604 shutdown_connections 2023-03-03T13:17:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f224c1493e0 0x7f224c0c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2234003c30 0x7f224c153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22380682f0 0x7f223806a7a0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 --2- 172.21.15.7:0/2090008604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f224c150740 0x7f224c148ea0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 -- 172.21.15.7:0/2090008604 >> 172.21.15.7:0/2090008604 conn(0x7f224c066e40 msgr2=0x7f224c0c7500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:17:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 -- 172.21.15.7:0/2090008604 shutdown_connections 2023-03-03T13:17:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:04.366+0000 7f2251301700 1 -- 172.21.15.7:0/2090008604 wait complete. 2023-03-03T13:17:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:17:04.380 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2023-03-03T13:17:09.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:09.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:36.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:17:36.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.835+0000 7f6bd91a1700 1 Processor -- start 2023-03-03T13:17:36.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.835+0000 7f6bd91a1700 1 -- start start 2023-03-03T13:17:36.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd91a1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bd40c5cb0 0x7f6bd406a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd91a1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd41558f0 con 0x7f6bd4156490 2023-03-03T13:17:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd91a1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd40a6180 con 0x7f6bd406a2b0 2023-03-03T13:17:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd91a1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd40c3290 con 0x7f6bd40c5cb0 2023-03-03T13:17:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bd40c5cb0 0x7f6bd406a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd359e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6bd4156490 0x7f6bd415cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37964/0 (socket says 172.21.15.7:37964) 2023-03-03T13:17:36.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd359e700 1 -- 172.21.15.7:0/2195903688 learned_addr learned my addr 172.21.15.7:0/2195903688 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:17:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd259c700 1 -- 172.21.15.7:0/2195903688 >> v1:172.21.15.143:6789/0 conn(0x7f6bd406a2b0 legacy=0x7f6bd4155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd259c700 1 -- 172.21.15.7:0/2195903688 >> v1:172.21.15.7:6789/0 conn(0x7f6bd4156490 legacy=0x7f6bd415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd259c700 1 -- 172.21.15.7:0/2195903688 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd40a5ea0 con 0x7f6bd40c5cb0 2023-03-03T13:17:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd1d9b700 1 -- 172.21.15.7:0/2195903688 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177208537 0 0) 0x7f6bd41558f0 con 0x7f6bd4156490 2023-03-03T13:17:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd259c700 1 --2- 172.21.15.7:0/2195903688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd40c5cb0 0x7f6bd406a680 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009840 tx=0x7f6bc8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcbf27c6209f85cd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd1d9b700 1 -- 172.21.15.7:0/2195903688 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8011830 con 0x7f6bd40c5cb0 2023-03-03T13:17:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd1d9b700 1 -- 172.21.15.7:0/2195903688 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc8011990 con 0x7f6bd40c5cb0 2023-03-03T13:17:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.836+0000 7f6bd1d9b700 1 -- 172.21.15.7:0/2195903688 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8011d20 con 0x7f6bd40c5cb0 2023-03-03T13:17:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.837+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2195903688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd40c5cb0 msgr2=0x7f6bd406a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.837+0000 7f6bd91a1700 1 --2- 172.21.15.7:0/2195903688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd40c5cb0 0x7f6bd406a680 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009840 tx=0x7f6bc8009b50 comp rx=0 tx=0).stop 2023-03-03T13:17:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.837+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2195903688 shutdown_connections 2023-03-03T13:17:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.837+0000 7f6bd91a1700 1 --2- 172.21.15.7:0/2195903688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd40c5cb0 0x7f6bd406a680 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.837+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2195903688 >> 172.21.15.7:0/2195903688 conn(0x7f6bd40a9430 msgr2=0x7f6bd40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:17:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.837+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2195903688 shutdown_connections 2023-03-03T13:17:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.837+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2195903688 wait complete. 2023-03-03T13:17:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.838+0000 7f6bd91a1700 1 Processor -- start 2023-03-03T13:17:36.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.839+0000 7f6bd91a1700 1 -- start start 2023-03-03T13:17:36.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.839+0000 7f6bd91a1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd406a2b0 0x7f6bd41ed1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.839+0000 7f6bd91a1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd40c5cb0 0x7f6bd406c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.839+0000 7f6bd91a1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd4156490 0x7f6bd41ea440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.839+0000 7f6bd91a1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd41ea020 con 0x7f6bd4156490 2023-03-03T13:17:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.839+0000 7f6bd91a1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40ab500 con 0x7f6bd40c5cb0 2023-03-03T13:17:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.839+0000 7f6bd91a1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40ab7e0 con 0x7f6bd406a2b0 2023-03-03T13:17:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.839+0000 7f6bd359e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd4156490 0x7f6bd41ea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.839+0000 7f6bd359e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd4156490 0x7f6bd41ea440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43988/0 (socket says 172.21.15.7:43988) 2023-03-03T13:17:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.839+0000 7f6bd359e700 1 -- 172.21.15.7:0/2939568349 learned_addr learned my addr 172.21.15.7:0/2939568349 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:17:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd259c700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd40c5cb0 0x7f6bd406c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd2d9d700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd406a2b0 0x7f6bd41ed1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd359e700 1 -- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd406a2b0 msgr2=0x7f6bd41ed1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd359e700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd406a2b0 0x7f6bd41ed1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd359e700 1 -- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd40c5cb0 msgr2=0x7f6bd406c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd359e700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd40c5cb0 0x7f6bd406c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd359e700 1 -- 172.21.15.7:0/2939568349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc80094a0 con 0x7f6bd4156490 2023-03-03T13:17:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd2d9d700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd406a2b0 0x7f6bd41ed1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:17:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd359e700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd4156490 0x7f6bd41ea440 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00cba0 tx=0x7f6bcc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bbb7fe700 1 -- 172.21.15.7:0/2939568349 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00ab80 con 0x7f6bd4156490 2023-03-03T13:17:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bbb7fe700 1 -- 172.21.15.7:0/2939568349 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc00ad00 con 0x7f6bd4156490 2023-03-03T13:17:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2939568349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd4153160 con 0x7f6bd4156490 2023-03-03T13:17:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bbb7fe700 1 -- 172.21.15.7:0/2939568349 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc007310 con 0x7f6bd4156490 2023-03-03T13:17:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.840+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2939568349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd4152e80 con 0x7f6bd4156490 2023-03-03T13:17:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.841+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2939568349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd40c68e0 con 0x7f6bd4156490 2023-03-03T13:17:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.842+0000 7f6bbb7fe700 1 -- 172.21.15.7:0/2939568349 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6bcc01a030 con 0x7f6bd4156490 2023-03-03T13:17:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.842+0000 7f6bbb7fe700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bc00682f0 0x7f6bc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.842+0000 7f6bbb7fe700 1 -- 172.21.15.7:0/2939568349 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6bcc085ea0 con 0x7f6bd4156490 2023-03-03T13:17:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.842+0000 7f6bd2d9d700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bc00682f0 0x7f6bc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.843+0000 7f6bd2d9d700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bc00682f0 0x7f6bc006a7a0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4000f60 tx=0x7f6bc4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:36.847+0000 7f6bbb7fe700 1 -- 172.21.15.7:0/2939568349 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6bcc0521e0 con 0x7f6bd4156490 2023-03-03T13:17:37.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.084+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2939568349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bd41ea160 con 0x7f6bd4156490 2023-03-03T13:17:37.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.084+0000 7f6bbb7fe700 1 -- 172.21.15.7:0/2939568349 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6bd41ea160 con 0x7f6bd4156490 2023-03-03T13:17:37.086 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:17:37.086 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:17:37.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bc00682f0 msgr2=0x7f6bc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:37.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bc00682f0 0x7f6bc006a7a0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4000f60 tx=0x7f6bc4009040 comp rx=0 tx=0).stop 2023-03-03T13:17:37.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2939568349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd4156490 msgr2=0x7f6bd41ea440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:37.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd4156490 0x7f6bd41ea440 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00cba0 tx=0x7f6bcc00cf60 comp rx=0 tx=0).stop 2023-03-03T13:17:37.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2939568349 shutdown_connections 2023-03-03T13:17:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6bc00682f0 0x7f6bc006a7a0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd4156490 0x7f6bd41ea440 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bd406a2b0 0x7f6bd41ed1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 --2- 172.21.15.7:0/2939568349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bd40c5cb0 0x7f6bd406c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2939568349 >> 172.21.15.7:0/2939568349 conn(0x7f6bd40a9430 msgr2=0x7f6bd41545c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:17:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2939568349 shutdown_connections 2023-03-03T13:17:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:37.086+0000 7f6bd91a1700 1 -- 172.21.15.7:0/2939568349 wait complete. 2023-03-03T13:17:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:17:37.100 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before thrashing 2023-03-03T13:17:39.101 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:17:39.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.238+0000 7fa594252700 1 Processor -- start 2023-03-03T13:17:39.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.238+0000 7fa594252700 1 -- start start 2023-03-03T13:17:39.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa594252700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa58c14b020 0x7fa58c0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa594252700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa58c071d20 0x7fa58c1583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa594252700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58c1588f0 con 0x7fa58c0aad20 2023-03-03T13:17:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa594252700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa58c0728b0 con 0x7fa58c14b020 2023-03-03T13:17:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa594252700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c155070 con 0x7fa58c071d20 2023-03-03T13:17:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa591fee700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa58c14b020 0x7fa58c0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:39.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa591fee700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa58c14b020 0x7fa58c0717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43996/0 (socket says 172.21.15.7:43996) 2023-03-03T13:17:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa591fee700 1 -- 172.21.15.7:0/968885970 learned_addr learned my addr 172.21.15.7:0/968885970 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:17:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa591fee700 1 -- 172.21.15.7:0/968885970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa58c071d20 msgr2=0x7fa58c1583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:17:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa591fee700 1 --2- 172.21.15.7:0/968885970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa58c071d20 0x7fa58c1583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa591fee700 1 -- 172.21.15.7:0/968885970 >> v1:172.21.15.143:6789/0 conn(0x7fa58c0aad20 legacy=0x7fa58c157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:17:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa591fee700 1 -- 172.21.15.7:0/968885970 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c06f5b0 con 0x7fa58c14b020 2023-03-03T13:17:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa591fee700 1 --2- 172.21.15.7:0/968885970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c14b020 0x7fa58c0717e0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fa5880098f0 tx=0x7fa588009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5ee9d9711db853 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:39.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.239+0000 7fa590fec700 1 -- 172.21.15.7:0/968885970 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588004230 con 0x7fa58c14b020 2023-03-03T13:17:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.240+0000 7fa590fec700 1 -- 172.21.15.7:0/968885970 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa588004390 con 0x7fa58c14b020 2023-03-03T13:17:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.240+0000 7fa590fec700 1 -- 172.21.15.7:0/968885970 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5880036a0 con 0x7fa58c14b020 2023-03-03T13:17:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.240+0000 7fa594252700 1 -- 172.21.15.7:0/968885970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c14b020 msgr2=0x7fa58c0717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.240+0000 7fa594252700 1 --2- 172.21.15.7:0/968885970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c14b020 0x7fa58c0717e0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fa5880098f0 tx=0x7fa588009c00 comp rx=0 tx=0).stop 2023-03-03T13:17:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.240+0000 7fa594252700 1 -- 172.21.15.7:0/968885970 shutdown_connections 2023-03-03T13:17:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.240+0000 7fa594252700 1 --2- 172.21.15.7:0/968885970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa58c071d20 0x7fa58c1583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.240+0000 7fa594252700 1 --2- 172.21.15.7:0/968885970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c14b020 0x7fa58c0717e0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.240+0000 7fa594252700 1 -- 172.21.15.7:0/968885970 >> 172.21.15.7:0/968885970 conn(0x7fa58c0c8900 msgr2=0x7fa58c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:17:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.240+0000 7fa594252700 1 -- 172.21.15.7:0/968885970 shutdown_connections 2023-03-03T13:17:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.240+0000 7fa594252700 1 -- 172.21.15.7:0/968885970 wait complete. 2023-03-03T13:17:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.241+0000 7fa594252700 1 Processor -- start 2023-03-03T13:17:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.241+0000 7fa594252700 1 -- start start 2023-03-03T13:17:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa594252700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa58c071d20 0x7fa58c1ea1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:39.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa594252700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c0aad20 0x7fa58c06eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa594252700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa58c14b020 0x7fa58c14de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa594252700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c14c510 con 0x7fa58c0aad20 2023-03-03T13:17:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa594252700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c1ea730 con 0x7fa58c071d20 2023-03-03T13:17:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa594252700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c14bf50 con 0x7fa58c14b020 2023-03-03T13:17:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa5917ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c0aad20 0x7fa58c06eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa5917ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c0aad20 0x7fa58c06eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44006/0 (socket says 172.21.15.7:44006) 2023-03-03T13:17:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa5917ed700 1 -- 172.21.15.7:0/2620872275 learned_addr learned my addr 172.21.15.7:0/2620872275 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:17:39.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa591fee700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa58c071d20 0x7fa58c1ea1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.242+0000 7fa5927ef700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa58c14b020 0x7fa58c14de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa5917ed700 1 -- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa58c14b020 msgr2=0x7fa58c14de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa5917ed700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa58c14b020 0x7fa58c14de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa5917ed700 1 -- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa58c071d20 msgr2=0x7fa58c1ea1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa5917ed700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa58c071d20 0x7fa58c1ea1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa5917ed700 1 -- 172.21.15.7:0/2620872275 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa588009550 con 0x7fa58c0aad20 2023-03-03T13:17:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa5917ed700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c0aad20 0x7fa58c06eae0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00b480 tx=0x7fa57c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa5827fc700 1 -- 172.21.15.7:0/2620872275 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c011830 con 0x7fa58c0aad20 2023-03-03T13:17:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa5827fc700 1 -- 172.21.15.7:0/2620872275 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c011990 con 0x7fa58c0aad20 2023-03-03T13:17:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa594252700 1 -- 172.21.15.7:0/2620872275 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58c06fb70 con 0x7fa58c0aad20 2023-03-03T13:17:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa5827fc700 1 -- 172.21.15.7:0/2620872275 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c011d40 con 0x7fa58c0aad20 2023-03-03T13:17:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.243+0000 7fa594252700 1 -- 172.21.15.7:0/2620872275 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58c06f890 con 0x7fa58c0aad20 2023-03-03T13:17:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.244+0000 7fa594252700 1 -- 172.21.15.7:0/2620872275 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58c155350 con 0x7fa58c0aad20 2023-03-03T13:17:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.244+0000 7fa5827fc700 1 -- 172.21.15.7:0/2620872275 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa57c00f050 con 0x7fa58c0aad20 2023-03-03T13:17:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.245+0000 7fa5827fc700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5700682f0 0x7fa57006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:17:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.245+0000 7fa5827fc700 1 -- 172.21.15.7:0/2620872275 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa57c085b50 con 0x7fa58c0aad20 2023-03-03T13:17:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.245+0000 7fa591fee700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5700682f0 0x7fa57006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:17:39.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.246+0000 7fa591fee700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5700682f0 0x7fa57006a7a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa588009130 tx=0x7fa588008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:17:39.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.250+0000 7fa5827fc700 1 -- 172.21.15.7:0/2620872275 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa57c051ef0 con 0x7fa58c0aad20 2023-03-03T13:17:39.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.489+0000 7fa594252700 1 -- 172.21.15.7:0/2620872275 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa58c1df2a0 con 0x7fa58c0aad20 2023-03-03T13:17:39.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.489+0000 7fa5827fc700 1 -- 172.21.15.7:0/2620872275 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa57c055510 con 0x7fa58c0aad20 2023-03-03T13:17:39.491 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:17:39.491 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:17:39.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.491+0000 7fa594252700 1 -- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5700682f0 msgr2=0x7fa57006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:39.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.491+0000 7fa594252700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5700682f0 0x7fa57006a7a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa588009130 tx=0x7fa588008030 comp rx=0 tx=0).stop 2023-03-03T13:17:39.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.491+0000 7fa594252700 1 -- 172.21.15.7:0/2620872275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c0aad20 msgr2=0x7fa58c06eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:17:39.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.491+0000 7fa594252700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c0aad20 0x7fa58c06eae0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00b480 tx=0x7fa57c00b790 comp rx=0 tx=0).stop 2023-03-03T13:17:39.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.492+0000 7fa594252700 1 -- 172.21.15.7:0/2620872275 shutdown_connections 2023-03-03T13:17:39.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.492+0000 7fa594252700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa58c14b020 0x7fa58c14de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:39.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.492+0000 7fa594252700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa58c071d20 0x7fa58c1ea1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:39.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.492+0000 7fa594252700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5700682f0 0x7fa57006a7a0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:39.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.492+0000 7fa594252700 1 --2- 172.21.15.7:0/2620872275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa58c0aad20 0x7fa58c06eae0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:17:39.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.492+0000 7fa594252700 1 -- 172.21.15.7:0/2620872275 >> 172.21.15.7:0/2620872275 conn(0x7fa58c0c8900 msgr2=0x7fa58c1555a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:17:39.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.492+0000 7fa594252700 1 -- 172.21.15.7:0/2620872275 shutdown_connections 2023-03-03T13:17:39.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:17:39.492+0000 7fa594252700 1 -- 172.21.15.7:0/2620872275 wait complete. 2023-03-03T13:17:39.495 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:17:39.505 INFO:tasks.mds_thrash.fs.[a]:waiting for 36 secs before thrashing 2023-03-03T13:17:39.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:17:39.923 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:10.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:10.012 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:15.506 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:18:15.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.643+0000 7f8138ab4700 1 Processor -- start 2023-03-03T13:18:15.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.643+0000 7f8138ab4700 1 -- start start 2023-03-03T13:18:15.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.644+0000 7f8138ab4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8134072250 0x7f813414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:15.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.644+0000 7f8138ab4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81340c1ce0 0x7f813414e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:15.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.644+0000 7f8138ab4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81341510d0 con 0x7f8134071e80 2023-03-03T13:18:15.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.644+0000 7f8138ab4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81340c3740 con 0x7f81340c1ce0 2023-03-03T13:18:15.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.644+0000 7f8138ab4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8134070670 con 0x7f8134072250 2023-03-03T13:18:15.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f8132d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81340c1ce0 0x7f813414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:15.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f813259c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8134071e80 0x7f8134151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37996/0 (socket says 172.21.15.7:37996) 2023-03-03T13:18:15.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f813259c700 1 -- 172.21.15.7:0/2452538663 learned_addr learned my addr 172.21.15.7:0/2452538663 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:18:15.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f8131d9b700 1 --2- 172.21.15.7:0/2452538663 >> v2:172.21.15.143:3300/0 conn(0x7f8134072250 0x7f813414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:15.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f8132d9d700 1 -- 172.21.15.7:0/2452538663 >> v2:172.21.15.143:3300/0 conn(0x7f8134072250 msgr2=0x7f813414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:18:15.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f8132d9d700 1 --2- 172.21.15.7:0/2452538663 >> v2:172.21.15.143:3300/0 conn(0x7f8134072250 0x7f813414e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f8132d9d700 1 -- 172.21.15.7:0/2452538663 >> v1:172.21.15.7:6789/0 conn(0x7f8134071e80 legacy=0x7f8134151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:15.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f8132d9d700 1 -- 172.21.15.7:0/2452538663 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81340c39e0 con 0x7f81340c1ce0 2023-03-03T13:18:15.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f8131d9b700 1 --2- 172.21.15.7:0/2452538663 >> v2:172.21.15.143:3300/0 conn(0x7f8134072250 0x7f813414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:18:15.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f813159a700 1 -- 172.21.15.7:0/2452538663 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814445973 0 0) 0x7f81341510d0 con 0x7f8134071e80 2023-03-03T13:18:15.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f8132d9d700 1 --2- 172.21.15.7:0/2452538663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81340c1ce0 0x7f813414e730 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f813406b520 tx=0x7f812c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563c1ae6af255ad4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:15.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f813159a700 1 -- 172.21.15.7:0/2452538663 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00e070 con 0x7f81340c1ce0 2023-03-03T13:18:15.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f813159a700 1 -- 172.21.15.7:0/2452538663 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c00bbd0 con 0x7f81340c1ce0 2023-03-03T13:18:15.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.645+0000 7f813159a700 1 -- 172.21.15.7:0/2452538663 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c012430 con 0x7f81340c1ce0 2023-03-03T13:18:15.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.646+0000 7f8138ab4700 1 -- 172.21.15.7:0/2452538663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81340c1ce0 msgr2=0x7f813414e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:15.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.646+0000 7f8138ab4700 1 --2- 172.21.15.7:0/2452538663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81340c1ce0 0x7f813414e730 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f813406b520 tx=0x7f812c00b330 comp rx=0 tx=0).stop 2023-03-03T13:18:15.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.646+0000 7f8138ab4700 1 -- 172.21.15.7:0/2452538663 shutdown_connections 2023-03-03T13:18:15.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.646+0000 7f8138ab4700 1 --2- 172.21.15.7:0/2452538663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81340c1ce0 0x7f813414e730 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.646+0000 7f8138ab4700 1 --2- 172.21.15.7:0/2452538663 >> v2:172.21.15.143:3300/0 conn(0x7f8134072250 0x7f813414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.646+0000 7f8138ab4700 1 -- 172.21.15.7:0/2452538663 >> 172.21.15.7:0/2452538663 conn(0x7f81340a94d0 msgr2=0x7f81340a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:18:15.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.646+0000 7f8138ab4700 1 -- 172.21.15.7:0/2452538663 shutdown_connections 2023-03-03T13:18:15.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.646+0000 7f8138ab4700 1 -- 172.21.15.7:0/2452538663 wait complete. 2023-03-03T13:18:15.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.647+0000 7f8138ab4700 1 Processor -- start 2023-03-03T13:18:15.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.647+0000 7f8138ab4700 1 -- start start 2023-03-03T13:18:15.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8138ab4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8134071e80 0x7f813415a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:15.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8138ab4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8134072250 0x7f8134141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:15.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8138ab4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81340c1ce0 0x7f81341419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:15.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8138ab4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81340c7180 con 0x7f8134072250 2023-03-03T13:18:15.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8138ab4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8134150770 con 0x7f8134071e80 2023-03-03T13:18:15.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8138ab4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8134150d10 con 0x7f81340c1ce0 2023-03-03T13:18:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8131d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8134072250 0x7f8134141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8131d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8134072250 0x7f8134141480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44024/0 (socket says 172.21.15.7:44024) 2023-03-03T13:18:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8131d9b700 1 -- 172.21.15.7:0/778793030 learned_addr learned my addr 172.21.15.7:0/778793030 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:18:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f813259c700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8134071e80 0x7f813415a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8132d9d700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81340c1ce0 0x7f81341419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8131d9b700 1 -- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81340c1ce0 msgr2=0x7f81341419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8131d9b700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81340c1ce0 0x7f81341419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8131d9b700 1 -- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8134071e80 msgr2=0x7f813415a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:15.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8131d9b700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8134071e80 0x7f813415a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.648+0000 7f8131d9b700 1 -- 172.21.15.7:0/778793030 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c009a70 con 0x7f8134072250 2023-03-03T13:18:15.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.649+0000 7f8132d9d700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81340c1ce0 0x7f81341419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:18:15.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.649+0000 7f813259c700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8134071e80 0x7f813415a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:18:15.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.649+0000 7f8131d9b700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8134072250 0x7f8134141480 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f812800e810 tx=0x7f812800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:15.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.649+0000 7f8122ffd700 1 -- 172.21.15.7:0/778793030 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128004230 con 0x7f8134072250 2023-03-03T13:18:15.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.649+0000 7f8122ffd700 1 -- 172.21.15.7:0/778793030 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8128004390 con 0x7f8134072250 2023-03-03T13:18:15.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.649+0000 7f8138ab4700 1 -- 172.21.15.7:0/778793030 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8134151f30 con 0x7f8134072250 2023-03-03T13:18:15.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.649+0000 7f8122ffd700 1 -- 172.21.15.7:0/778793030 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81280036a0 con 0x7f8134072250 2023-03-03T13:18:15.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.649+0000 7f8138ab4700 1 -- 172.21.15.7:0/778793030 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8134151300 con 0x7f8134072250 2023-03-03T13:18:15.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.650+0000 7f8138ab4700 1 -- 172.21.15.7:0/778793030 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813406c9a0 con 0x7f8134072250 2023-03-03T13:18:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.650+0000 7f8122ffd700 1 -- 172.21.15.7:0/778793030 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8128022070 con 0x7f8134072250 2023-03-03T13:18:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.651+0000 7f8122ffd700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81140682f0 0x7f811406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.651+0000 7f8122ffd700 1 -- 172.21.15.7:0/778793030 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8128085ba0 con 0x7f8134072250 2023-03-03T13:18:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.651+0000 7f813259c700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81140682f0 0x7f811406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.652+0000 7f813259c700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81140682f0 0x7f811406a7a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8124000f60 tx=0x7f8124007cc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.655+0000 7f8122ffd700 1 -- 172.21.15.7:0/778793030 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8128051f40 con 0x7f8134072250 2023-03-03T13:18:15.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.892+0000 7f8138ab4700 1 -- 172.21.15.7:0/778793030 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8134150e50 con 0x7f8134072250 2023-03-03T13:18:15.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.893+0000 7f8122ffd700 1 -- 172.21.15.7:0/778793030 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8134150e50 con 0x7f8134072250 2023-03-03T13:18:15.894 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:18:15.895 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:18:15.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 -- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81140682f0 msgr2=0x7f811406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:15.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81140682f0 0x7f811406a7a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8124000f60 tx=0x7f8124007cc0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 -- 172.21.15.7:0/778793030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8134072250 msgr2=0x7f8134141480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:15.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8134072250 0x7f8134141480 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f812800e810 tx=0x7f812800ebd0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 -- 172.21.15.7:0/778793030 shutdown_connections 2023-03-03T13:18:15.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81340c1ce0 0x7f81341419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8134071e80 0x7f813415a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81140682f0 0x7f811406a7a0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 --2- 172.21.15.7:0/778793030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8134072250 0x7f8134141480 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:15.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 -- 172.21.15.7:0/778793030 >> 172.21.15.7:0/778793030 conn(0x7f81340a94d0 msgr2=0x7f81340c25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:18:15.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 -- 172.21.15.7:0/778793030 shutdown_connections 2023-03-03T13:18:15.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:15.895+0000 7f8138ab4700 1 -- 172.21.15.7:0/778793030 wait complete. 2023-03-03T13:18:15.898 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:18:15.908 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2023-03-03T13:18:30.381 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:18:30.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.518+0000 7f1912606700 1 Processor -- start 2023-03-03T13:18:30.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.518+0000 7f1912606700 1 -- start start 2023-03-03T13:18:30.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f1912606700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f190c071e80 0x7f190c151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f1912606700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f190c072250 0x7f190c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f1912606700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 0x7f190c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f1912606700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190c1510d0 con 0x7f190c072250 2023-03-03T13:18:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f1912606700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190c0c3740 con 0x7f190c071e80 2023-03-03T13:18:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f1912606700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c070670 con 0x7f190c0c1ce0 2023-03-03T13:18:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f190b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f190c072250 0x7f190c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:30.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f190b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f190c072250 0x7f190c14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44036/0 (socket says 172.21.15.7:44036) 2023-03-03T13:18:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f190b7fe700 1 -- 172.21.15.7:0/3264266575 learned_addr learned my addr 172.21.15.7:0/3264266575 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:18:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f190bfff700 1 --2- 172.21.15.7:0/3264266575 >> v2:172.21.15.143:3300/0 conn(0x7f190c071e80 0x7f190c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.519+0000 7f1910ba3700 1 --2- 172.21.15.7:0/3264266575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 0x7f190c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.520+0000 7f190b7fe700 1 -- 172.21.15.7:0/3264266575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 msgr2=0x7f190c14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:18:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.520+0000 7f190b7fe700 1 --2- 172.21.15.7:0/3264266575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 0x7f190c14e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.520+0000 7f190b7fe700 1 -- 172.21.15.7:0/3264266575 >> v2:172.21.15.143:3300/0 conn(0x7f190c071e80 msgr2=0x7f190c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:18:30.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.520+0000 7f190b7fe700 1 --2- 172.21.15.7:0/3264266575 >> v2:172.21.15.143:3300/0 conn(0x7f190c071e80 0x7f190c151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.520+0000 7f190b7fe700 1 -- 172.21.15.7:0/3264266575 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190c0c39e0 con 0x7f190c072250 2023-03-03T13:18:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.520+0000 7f190b7fe700 1 --2- 172.21.15.7:0/3264266575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 0x7f190c14e1f0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f190c06bd10 tx=0x7f18f400be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155988f0fcf54037 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.520+0000 7f190affd700 1 -- 172.21.15.7:0/3264266575 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f400dc70 con 0x7f190c072250 2023-03-03T13:18:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.520+0000 7f190affd700 1 -- 172.21.15.7:0/3264266575 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f400ddd0 con 0x7f190c072250 2023-03-03T13:18:30.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.520+0000 7f190affd700 1 -- 172.21.15.7:0/3264266575 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4010690 con 0x7f190c072250 2023-03-03T13:18:30.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.521+0000 7f1912606700 1 -- 172.21.15.7:0/3264266575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 msgr2=0x7f190c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:30.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.521+0000 7f1912606700 1 --2- 172.21.15.7:0/3264266575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 0x7f190c14e1f0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f190c06bd10 tx=0x7f18f400be70 comp rx=0 tx=0).stop 2023-03-03T13:18:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.521+0000 7f1912606700 1 -- 172.21.15.7:0/3264266575 shutdown_connections 2023-03-03T13:18:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.521+0000 7f1912606700 1 --2- 172.21.15.7:0/3264266575 >> v2:172.21.15.143:3300/0 conn(0x7f190c071e80 0x7f190c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.521+0000 7f1912606700 1 --2- 172.21.15.7:0/3264266575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 0x7f190c14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.521+0000 7f1912606700 1 --2- 172.21.15.7:0/3264266575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 0x7f190c14e1f0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.521+0000 7f1912606700 1 -- 172.21.15.7:0/3264266575 >> 172.21.15.7:0/3264266575 conn(0x7f190c0a94d0 msgr2=0x7f190c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:18:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.521+0000 7f1912606700 1 -- 172.21.15.7:0/3264266575 shutdown_connections 2023-03-03T13:18:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.521+0000 7f1912606700 1 -- 172.21.15.7:0/3264266575 wait complete. 2023-03-03T13:18:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.522+0000 7f1912606700 1 Processor -- start 2023-03-03T13:18:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.522+0000 7f1912606700 1 -- start start 2023-03-03T13:18:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f1912606700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f190c071e80 0x7f190c1e9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f1912606700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 0x7f190c152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f1912606700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 0x7f190c154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f1912606700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c06a850 con 0x7f190c072250 2023-03-03T13:18:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f1912606700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0c6880 con 0x7f190c071e80 2023-03-03T13:18:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f1912606700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190c0c6b60 con 0x7f190c0c1ce0 2023-03-03T13:18:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f190b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 0x7f190c152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f190b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 0x7f190c152f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44042/0 (socket says 172.21.15.7:44042) 2023-03-03T13:18:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f190b7fe700 1 -- 172.21.15.7:0/238223178 learned_addr learned my addr 172.21.15.7:0/238223178 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:18:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f190bfff700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f190c071e80 0x7f190c1e9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f1910ba3700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 0x7f190c154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f190b7fe700 1 -- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 msgr2=0x7f190c154b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f190b7fe700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 0x7f190c154b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f190b7fe700 1 -- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f190c071e80 msgr2=0x7f190c1e9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f190b7fe700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f190c071e80 0x7f190c1e9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f190b7fe700 1 -- 172.21.15.7:0/238223178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f400b7f0 con 0x7f190c072250 2023-03-03T13:18:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f1910ba3700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 0x7f190c154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:18:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.523+0000 7f190b7fe700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 0x7f190c152f20 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f18f400beb0 tx=0x7f18f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.524+0000 7f1908ff9700 1 -- 172.21.15.7:0/238223178 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f401f070 con 0x7f190c072250 2023-03-03T13:18:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.524+0000 7f1908ff9700 1 -- 172.21.15.7:0/238223178 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f40038b0 con 0x7f190c072250 2023-03-03T13:18:30.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.524+0000 7f1912606700 1 -- 172.21.15.7:0/238223178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190c06c9a0 con 0x7f190c072250 2023-03-03T13:18:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.524+0000 7f1908ff9700 1 -- 172.21.15.7:0/238223178 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4003c60 con 0x7f190c072250 2023-03-03T13:18:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.524+0000 7f1912606700 1 -- 172.21.15.7:0/238223178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190c151300 con 0x7f190c072250 2023-03-03T13:18:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.525+0000 7f1912606700 1 -- 172.21.15.7:0/238223178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190c06cb30 con 0x7f190c072250 2023-03-03T13:18:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.525+0000 7f1908ff9700 1 -- 172.21.15.7:0/238223178 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f18f4019400 con 0x7f190c072250 2023-03-03T13:18:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.526+0000 7f1908ff9700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18ec0682f0 0x7f18ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.526+0000 7f1908ff9700 1 -- 172.21.15.7:0/238223178 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f18f4089400 con 0x7f190c072250 2023-03-03T13:18:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.526+0000 7f190bfff700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18ec0682f0 0x7f18ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.526+0000 7f190bfff700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18ec0682f0 0x7f18ec06a7a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f190c0c2f80 tx=0x7f18fc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:30.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.530+0000 7f1908ff9700 1 -- 172.21.15.7:0/238223178 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18f4055740 con 0x7f190c072250 2023-03-03T13:18:30.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.773+0000 7f1912606700 1 -- 172.21.15.7:0/238223178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f190c074bb0 con 0x7f190c072250 2023-03-03T13:18:30.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.773+0000 7f1908ff9700 1 -- 172.21.15.7:0/238223178 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f18f4058d60 con 0x7f190c072250 2023-03-03T13:18:30.775 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:18:30.775 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:18:30.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.775+0000 7f1912606700 1 -- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18ec0682f0 msgr2=0x7f18ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:30.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.775+0000 7f1912606700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18ec0682f0 0x7f18ec06a7a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f190c0c2f80 tx=0x7f18fc009210 comp rx=0 tx=0).stop 2023-03-03T13:18:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.776+0000 7f1912606700 1 -- 172.21.15.7:0/238223178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 msgr2=0x7f190c152f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.776+0000 7f1912606700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 0x7f190c152f20 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f18f400beb0 tx=0x7f18f4004750 comp rx=0 tx=0).stop 2023-03-03T13:18:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.776+0000 7f1912606700 1 -- 172.21.15.7:0/238223178 shutdown_connections 2023-03-03T13:18:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.776+0000 7f1912606700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f190c0c1ce0 0x7f190c154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.776+0000 7f1912606700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f190c071e80 0x7f190c1e9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.776+0000 7f1912606700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18ec0682f0 0x7f18ec06a7a0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.776+0000 7f1912606700 1 --2- 172.21.15.7:0/238223178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f190c072250 0x7f190c152f20 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:30.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.776+0000 7f1912606700 1 -- 172.21.15.7:0/238223178 >> 172.21.15.7:0/238223178 conn(0x7f190c0a94d0 msgr2=0x7f190c06d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:18:30.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.776+0000 7f1912606700 1 -- 172.21.15.7:0/238223178 shutdown_connections 2023-03-03T13:18:30.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:30.776+0000 7f1912606700 1 -- 172.21.15.7:0/238223178 wait complete. 2023-03-03T13:18:30.779 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:18:30.789 INFO:tasks.mds_thrash.fs.[b]:waiting for 36 secs before thrashing 2023-03-03T13:18:30.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:18:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.032+0000 7f62a3a6d700 1 Processor -- start 2023-03-03T13:18:31.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.033+0000 7f62a3a6d700 1 -- start start 2023-03-03T13:18:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.033+0000 7f62a3a6d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c151d40 0x7f629c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.033+0000 7f62a3a6d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f629c0c2930 0x7f629c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.033+0000 7f62a3a6d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f629c073c30 0x7f629c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.033+0000 7f62a3a6d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f629c06c2a0 con 0x7f629c0c2930 2023-03-03T13:18:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.033+0000 7f62a3a6d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f629c06d430 con 0x7f629c073c30 2023-03-03T13:18:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.033+0000 7f62a3a6d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629c06cb00 con 0x7f629c151d40 2023-03-03T13:18:31.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.033+0000 7f62a1008700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f629c0c2930 0x7f629c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.033+0000 7f62a1008700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f629c0c2930 0x7f629c157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44050/0 (socket says 172.21.15.7:44050) 2023-03-03T13:18:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.033+0000 7f62a1008700 1 -- 172.21.15.7:0/2955540030 learned_addr learned my addr 172.21.15.7:0/2955540030 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:18:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a200a700 1 --2- 172.21.15.7:0/2955540030 >> v2:172.21.15.143:3300/0 conn(0x7f629c073c30 0x7f629c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a1008700 1 -- 172.21.15.7:0/2955540030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c151d40 msgr2=0x7f629c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:18:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a1809700 1 --2- 172.21.15.7:0/2955540030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c151d40 0x7f629c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a1008700 1 --2- 172.21.15.7:0/2955540030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c151d40 0x7f629c0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a1008700 1 -- 172.21.15.7:0/2955540030 >> v2:172.21.15.143:3300/0 conn(0x7f629c073c30 msgr2=0x7f629c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:18:31.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a1008700 1 --2- 172.21.15.7:0/2955540030 >> v2:172.21.15.143:3300/0 conn(0x7f629c073c30 0x7f629c157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a1008700 1 -- 172.21.15.7:0/2955540030 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629c06cde0 con 0x7f629c0c2930 2023-03-03T13:18:31.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a200a700 1 --2- 172.21.15.7:0/2955540030 >> v2:172.21.15.143:3300/0 conn(0x7f629c073c30 0x7f629c157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:18:31.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a1008700 1 --2- 172.21.15.7:0/2955540030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 0x7f629c157a90 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f629c0c3db0 tx=0x7f6290009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa453d63c67619d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:31.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a0807700 1 -- 172.21.15.7:0/2955540030 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629000b870 con 0x7f629c0c2930 2023-03-03T13:18:31.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a0807700 1 -- 172.21.15.7:0/2955540030 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629000b9d0 con 0x7f629c0c2930 2023-03-03T13:18:31.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.034+0000 7f62a0807700 1 -- 172.21.15.7:0/2955540030 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629000bd60 con 0x7f629c0c2930 2023-03-03T13:18:31.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.035+0000 7f62a3a6d700 1 -- 172.21.15.7:0/2955540030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 msgr2=0x7f629c157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:31.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.035+0000 7f62a3a6d700 1 --2- 172.21.15.7:0/2955540030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 0x7f629c157a90 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f629c0c3db0 tx=0x7f6290009bd0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.035+0000 7f62a3a6d700 1 -- 172.21.15.7:0/2955540030 shutdown_connections 2023-03-03T13:18:31.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.035+0000 7f62a3a6d700 1 --2- 172.21.15.7:0/2955540030 >> v2:172.21.15.143:3300/0 conn(0x7f629c073c30 0x7f629c157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.035+0000 7f62a3a6d700 1 --2- 172.21.15.7:0/2955540030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c151d40 0x7f629c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.035+0000 7f62a3a6d700 1 --2- 172.21.15.7:0/2955540030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 0x7f629c157a90 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.035+0000 7f62a3a6d700 1 -- 172.21.15.7:0/2955540030 >> 172.21.15.7:0/2955540030 conn(0x7f629c0a94d0 msgr2=0x7f629c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:18:31.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.035+0000 7f62a3a6d700 1 -- 172.21.15.7:0/2955540030 shutdown_connections 2023-03-03T13:18:31.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.035+0000 7f62a3a6d700 1 -- 172.21.15.7:0/2955540030 wait complete. 2023-03-03T13:18:31.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.036+0000 7f62a3a6d700 1 Processor -- start 2023-03-03T13:18:31.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.036+0000 7f62a3a6d700 1 -- start start 2023-03-03T13:18:31.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a3a6d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c073c30 0x7f629c0acc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:31.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a3a6d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 0x7f629c0be430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a3a6d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f629c151d40 0x7f629c0be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a3a6d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c0aa1d0 con 0x7f629c0c2930 2023-03-03T13:18:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a3a6d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c0aabe0 con 0x7f629c151d40 2023-03-03T13:18:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a3a6d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629c0aaec0 con 0x7f629c073c30 2023-03-03T13:18:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1008700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 0x7f629c0be430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1008700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 0x7f629c0be430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44056/0 (socket says 172.21.15.7:44056) 2023-03-03T13:18:31.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1008700 1 -- 172.21.15.7:0/680343504 learned_addr learned my addr 172.21.15.7:0/680343504 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:18:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1809700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c073c30 0x7f629c0acc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a200a700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f629c151d40 0x7f629c0be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1008700 1 -- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c073c30 msgr2=0x7f629c0acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1008700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c073c30 0x7f629c0acc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1008700 1 -- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f629c151d40 msgr2=0x7f629c0be970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1008700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f629c151d40 0x7f629c0be970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1008700 1 -- 172.21.15.7:0/680343504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6290009550 con 0x7f629c0c2930 2023-03-03T13:18:31.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1809700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c073c30 0x7f629c0acc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:18:31.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a1008700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 0x7f629c0be430 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f6290009fd0 tx=0x7f6290004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:31.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f628e7fc700 1 -- 172.21.15.7:0/680343504 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62900117a0 con 0x7f629c0c2930 2023-03-03T13:18:31.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f628e7fc700 1 -- 172.21.15.7:0/680343504 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629001f070 con 0x7f629c0c2930 2023-03-03T13:18:31.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.037+0000 7f62a3a6d700 1 -- 172.21.15.7:0/680343504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629c06bea0 con 0x7f629c0c2930 2023-03-03T13:18:31.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.038+0000 7f628e7fc700 1 -- 172.21.15.7:0/680343504 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290011b40 con 0x7f629c0c2930 2023-03-03T13:18:31.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.038+0000 7f62a3a6d700 1 -- 172.21.15.7:0/680343504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629c06d0c0 con 0x7f629c0c2930 2023-03-03T13:18:31.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.038+0000 7f62a3a6d700 1 -- 172.21.15.7:0/680343504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629c06c030 con 0x7f629c0c2930 2023-03-03T13:18:31.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.039+0000 7f628e7fc700 1 -- 172.21.15.7:0/680343504 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f629002a020 con 0x7f629c0c2930 2023-03-03T13:18:31.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.039+0000 7f628e7fc700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62800682f0 0x7f628006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:18:31.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.039+0000 7f628e7fc700 1 -- 172.21.15.7:0/680343504 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6290087690 con 0x7f629c0c2930 2023-03-03T13:18:31.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.039+0000 7f62a1809700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62800682f0 0x7f628006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:18:31.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.040+0000 7f62a1809700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62800682f0 0x7f628006a7a0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f629800a9d0 tx=0x7f6298008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:18:31.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.043+0000 7f628e7fc700 1 -- 172.21.15.7:0/680343504 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6290053a30 con 0x7f629c0c2930 2023-03-03T13:18:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.283+0000 7f62a3a6d700 1 -- 172.21.15.7:0/680343504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f629c0ab030 con 0x7f629c0c2930 2023-03-03T13:18:31.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.284+0000 7f628e7fc700 1 -- 172.21.15.7:0/680343504 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6290006d80 con 0x7f629c0c2930 2023-03-03T13:18:31.286 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:18:31.286 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:18:31.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 -- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62800682f0 msgr2=0x7f628006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62800682f0 0x7f628006a7a0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f629800a9d0 tx=0x7f6298008040 comp rx=0 tx=0).stop 2023-03-03T13:18:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 -- 172.21.15.7:0/680343504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 msgr2=0x7f629c0be430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:18:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 0x7f629c0be430 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f6290009fd0 tx=0x7f6290004750 comp rx=0 tx=0).stop 2023-03-03T13:18:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 -- 172.21.15.7:0/680343504 shutdown_connections 2023-03-03T13:18:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f629c151d40 0x7f629c0be970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f629c073c30 0x7f629c0acc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62800682f0 0x7f628006a7a0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 --2- 172.21.15.7:0/680343504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f629c0c2930 0x7f629c0be430 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:18:31.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 -- 172.21.15.7:0/680343504 >> 172.21.15.7:0/680343504 conn(0x7f629c0a94d0 msgr2=0x7f629c071920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:18:31.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 -- 172.21.15.7:0/680343504 shutdown_connections 2023-03-03T13:18:31.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:18:31.286+0000 7f62a3a6d700 1 -- 172.21.15.7:0/680343504 wait complete. 2023-03-03T13:18:31.289 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:18:31.299 INFO:tasks.mds_thrash.fs.[a]:waiting for 119 secs before thrashing 2023-03-03T13:18:40.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:18:40.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:06.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:19:06.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.927+0000 7ff4e462d700 1 Processor -- start 2023-03-03T13:19:06.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.927+0000 7ff4e462d700 1 -- start start 2023-03-03T13:19:06.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.928+0000 7ff4e462d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff4dc13a0f0 0x7ff4dc0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:06.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.928+0000 7ff4e462d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4dc0c1da0 0x7ff4dc06d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:06.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.928+0000 7ff4e462d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4dc06de90 con 0x7ff4dc1586c0 2023-03-03T13:19:06.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.928+0000 7ff4e462d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4dc154840 con 0x7ff4dc13a0f0 2023-03-03T13:19:06.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.928+0000 7ff4e462d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc154560 con 0x7ff4dc0c1da0 2023-03-03T13:19:06.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.928+0000 7ff4e1bc8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff4dc1586c0 0x7ff4dc06d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38084/0 (socket says 172.21.15.7:38084) 2023-03-03T13:19:06.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e1bc8700 1 -- 172.21.15.7:0/268616208 learned_addr learned my addr 172.21.15.7:0/268616208 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:19:06.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e23c9700 1 --2- 172.21.15.7:0/268616208 >> v2:172.21.15.143:3300/0 conn(0x7ff4dc13a0f0 0x7ff4dc0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:06.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e2bca700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4dc0c1da0 0x7ff4dc06d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:06.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691984997 0 0) 0x7ff4d8001110 con 0x7ff4dc1586c0 2023-03-03T13:19:06.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c80035b0 con 0x7ff4dc1586c0 2023-03-03T13:19:06.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2572368596 0 0) 0x7ff4c80035b0 con 0x7ff4dc1586c0 2023-03-03T13:19:06.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4dc0c1da0 msgr2=0x7ff4dc06d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:19:06.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4dc0c1da0 0x7ff4dc06d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:06.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 >> v2:172.21.15.143:3300/0 conn(0x7ff4dc13a0f0 msgr2=0x7ff4dc0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:19:06.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 --2- 172.21.15.7:0/268616208 >> v2:172.21.15.143:3300/0 conn(0x7ff4dc13a0f0 0x7ff4dc0c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc157be0 con 0x7ff4dc1586c0 2023-03-03T13:19:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff4d8001800 con 0x7ff4dc1586c0 2023-03-03T13:19:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 >> v1:172.21.15.7:6789/0 conn(0x7ff4dc1586c0 legacy=0x7ff4dc06d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4c8003a70 0x7ff4c8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4dc0c1da0 0x7ff4c8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c8004d90 0x7ff4c8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:06.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc154840 con 0x7ff4c8003a70 2023-03-03T13:19:06.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc154560 con 0x7ff4c8004d90 2023-03-03T13:19:06.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c800bb40 con 0x7ff4dc0c1da0 2023-03-03T13:19:06.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.929+0000 7ff4e23c9700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4c8003a70 0x7ff4c8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:06.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e1bc8700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4dc0c1da0 0x7ff4c8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:06.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e2bca700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c8004d90 0x7ff4c8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:06.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e23c9700 1 -- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4dc0c1da0 msgr2=0x7ff4c8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:19:06.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e23c9700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4dc0c1da0 0x7ff4c8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:06.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e23c9700 1 -- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c8004d90 msgr2=0x7ff4c8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:19:06.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e23c9700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c8004d90 0x7ff4c8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:06.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e23c9700 1 -- 172.21.15.7:0/268616208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c80035b0 con 0x7ff4c8003a70 2023-03-03T13:19:06.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e2bca700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4c8004d90 0x7ff4c8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:19:06.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e23c9700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4c8003a70 0x7ff4c8003e80 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc009b60 tx=0x7ff4c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaca3f84dcfb2e6b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:19:06.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc00e6c0 con 0x7ff4c8003a70 2023-03-03T13:19:06.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc00e820 con 0x7ff4c8003a70 2023-03-03T13:19:06.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.930+0000 7ff4e13c7700 1 -- 172.21.15.7:0/268616208 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc00eb90 con 0x7ff4c8003a70 2023-03-03T13:19:06.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.931+0000 7ff4e462d700 1 -- 172.21.15.7:0/268616208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4c8003a70 msgr2=0x7ff4c8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:06.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.931+0000 7ff4e462d700 1 --2- 172.21.15.7:0/268616208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4c8003a70 0x7ff4c8003e80 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc009b60 tx=0x7ff4c8005880 comp rx=0 tx=0).stop 2023-03-03T13:19:06.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.931+0000 7ff4e2bca700 1 -- 172.21.15.7:0/268616208 reap_dead start 2023-03-03T13:19:06.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.931+0000 7ff4e462d700 1 -- 172.21.15.7:0/268616208 shutdown_connections 2023-03-03T13:19:06.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.931+0000 7ff4e462d700 1 -- 172.21.15.7:0/268616208 >> 172.21.15.7:0/268616208 conn(0x7ff4dc06c110 msgr2=0x7ff4dc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:19:06.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.931+0000 7ff4e462d700 1 -- 172.21.15.7:0/268616208 shutdown_connections 2023-03-03T13:19:06.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.931+0000 7ff4e462d700 1 -- 172.21.15.7:0/268616208 wait complete. 2023-03-03T13:19:06.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.932+0000 7ff4e462d700 1 Processor -- start 2023-03-03T13:19:06.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.932+0000 7ff4e462d700 1 -- start start 2023-03-03T13:19:06.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e462d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c8003a70 0x7ff4dc0ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:06.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e462d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4dc1344b0 0x7ff4dc0ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:06.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e462d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4dc1568e0 0x7ff4dc156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:06.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e462d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0c8130 con 0x7ff4dc1344b0 2023-03-03T13:19:06.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e462d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0aa8b0 con 0x7ff4dc1568e0 2023-03-03T13:19:06.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e462d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc1572a0 con 0x7ff4c8003a70 2023-03-03T13:19:06.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e1bc8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4dc1344b0 0x7ff4dc0ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:06.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e1bc8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4dc1344b0 0x7ff4dc0ac6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44116/0 (socket says 172.21.15.7:44116) 2023-03-03T13:19:06.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e1bc8700 1 -- 172.21.15.7:0/1319889517 learned_addr learned my addr 172.21.15.7:0/1319889517 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:19:06.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e23c9700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c8003a70 0x7ff4dc0ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:06.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e2bca700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4dc1568e0 0x7ff4dc156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:06.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e1bc8700 1 -- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c8003a70 msgr2=0x7ff4dc0ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:06.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e1bc8700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c8003a70 0x7ff4dc0ac170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:06.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e1bc8700 1 -- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4dc1568e0 msgr2=0x7ff4dc156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:06.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e1bc8700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4dc1568e0 0x7ff4dc156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:06.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e1bc8700 1 -- 172.21.15.7:0/1319889517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc1581a0 con 0x7ff4dc1344b0 2023-03-03T13:19:06.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.933+0000 7ff4e1bc8700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4dc1344b0 0x7ff4dc0ac6b0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8007310 tx=0x7ff4d8007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:19:06.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.934+0000 7ff4d2ffd700 1 -- 172.21.15.7:0/1319889517 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8011580 con 0x7ff4dc1344b0 2023-03-03T13:19:06.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.934+0000 7ff4d2ffd700 1 -- 172.21.15.7:0/1319889517 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d8011740 con 0x7ff4dc1344b0 2023-03-03T13:19:06.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.934+0000 7ff4e462d700 1 -- 172.21.15.7:0/1319889517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4dc157ec0 con 0x7ff4dc1344b0 2023-03-03T13:19:06.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.934+0000 7ff4d2ffd700 1 -- 172.21.15.7:0/1319889517 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8011af0 con 0x7ff4dc1344b0 2023-03-03T13:19:06.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.934+0000 7ff4e462d700 1 -- 172.21.15.7:0/1319889517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4dc06e1a0 con 0x7ff4dc1344b0 2023-03-03T13:19:06.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.935+0000 7ff4e462d700 1 -- 172.21.15.7:0/1319889517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc0aaa20 con 0x7ff4dc1344b0 2023-03-03T13:19:06.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.935+0000 7ff4d2ffd700 1 -- 172.21.15.7:0/1319889517 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff4d8011c50 con 0x7ff4dc1344b0 2023-03-03T13:19:06.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.935+0000 7ff4d2ffd700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c00682f0 0x7ff4c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:06.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.935+0000 7ff4d2ffd700 1 -- 172.21.15.7:0/1319889517 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff4d80860b0 con 0x7ff4dc1344b0 2023-03-03T13:19:06.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.936+0000 7ff4e23c9700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c00682f0 0x7ff4c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:06.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.936+0000 7ff4e23c9700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c00682f0 0x7ff4c006a7a0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0090b0 tx=0x7ff4cc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:19:06.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:06.940+0000 7ff4d2ffd700 1 -- 172.21.15.7:0/1319889517 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff4d8052450 con 0x7ff4dc1344b0 2023-03-03T13:19:07.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.180+0000 7ff4e462d700 1 -- 172.21.15.7:0/1319889517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4dc06de90 con 0x7ff4dc1344b0 2023-03-03T13:19:07.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.180+0000 7ff4d2ffd700 1 -- 172.21.15.7:0/1319889517 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff4dc06de90 con 0x7ff4dc1344b0 2023-03-03T13:19:07.182 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:19:07.182 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:19:07.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 -- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c00682f0 msgr2=0x7ff4c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:07.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c00682f0 0x7ff4c006a7a0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0090b0 tx=0x7ff4cc008030 comp rx=0 tx=0).stop 2023-03-03T13:19:07.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 -- 172.21.15.7:0/1319889517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4dc1344b0 msgr2=0x7ff4dc0ac6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:07.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4dc1344b0 0x7ff4dc0ac6b0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8007310 tx=0x7ff4d8007620 comp rx=0 tx=0).stop 2023-03-03T13:19:07.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 -- 172.21.15.7:0/1319889517 shutdown_connections 2023-03-03T13:19:07.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4dc1568e0 0x7ff4dc156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:07.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4c8003a70 0x7ff4dc0ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:07.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4c00682f0 0x7ff4c006a7a0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:07.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 --2- 172.21.15.7:0/1319889517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4dc1344b0 0x7ff4dc0ac6b0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:07.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 -- 172.21.15.7:0/1319889517 >> 172.21.15.7:0/1319889517 conn(0x7ff4dc06c110 msgr2=0x7ff4dc06d260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:19:07.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 -- 172.21.15.7:0/1319889517 shutdown_connections 2023-03-03T13:19:07.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:07.182+0000 7ff4e462d700 1 -- 172.21.15.7:0/1319889517 wait complete. 2023-03-03T13:19:07.186 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:19:07.196 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T13:19:10.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:10.222 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:25.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:19:25.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.330+0000 7f88727a1700 1 Processor -- start 2023-03-03T13:19:25.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.330+0000 7f88727a1700 1 -- start start 2023-03-03T13:19:25.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.331+0000 7f88727a1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f886c151ad0 0x7f886c0c7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:25.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.331+0000 7f88727a1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886c1583e0 con 0x7f886c154280 2023-03-03T13:19:25.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.331+0000 7f88727a1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886c1577f0 con 0x7f886c0c7540 2023-03-03T13:19:25.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.331+0000 7f88727a1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0a6dd0 con 0x7f886c151ad0 2023-03-03T13:19:25.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.331+0000 7f8871fa0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f886c154280 0x7f886c154700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38114/0 (socket says 172.21.15.7:38114) 2023-03-03T13:19:25.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.331+0000 7f8871fa0700 1 -- 172.21.15.7:0/1857581206 learned_addr learned my addr 172.21.15.7:0/1857581206 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:19:25.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.331+0000 7f887179f700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f886c151ad0 0x7f886c0c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:25.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.331+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675450634 0 0) 0x7f886c1583e0 con 0x7f886c154280 2023-03-03T13:19:25.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88580035b0 con 0x7f886c154280 2023-03-03T13:19:25.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036015968 0 0) 0x7f886c1577f0 con 0x7f886c0c7540 2023-03-03T13:19:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886c1583e0 con 0x7f886c0c7540 2023-03-03T13:19:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 591501430 0 0) 0x7f88580035b0 con 0x7f886c154280 2023-03-03T13:19:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f886c151ad0 msgr2=0x7f886c0c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:19:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f886c151ad0 0x7f886c0c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 >> v1:172.21.15.143:6789/0 conn(0x7f886c0c7540 legacy=0x7f886c0c1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c157ab0 con 0x7f886c154280 2023-03-03T13:19:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f88640019f0 con 0x7f886c154280 2023-03-03T13:19:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 >> v1:172.21.15.7:6789/0 conn(0x7f886c154280 legacy=0x7f886c154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f886c151ad0 0x7f8858003ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8858004250 0x7f8858004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8858004e50 0x7f88580052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0a6dd0 con 0x7f8858004250 2023-03-03T13:19:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885800bd40 con 0x7f8858004e50 2023-03-03T13:19:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885800bf00 con 0x7f886c151ad0 2023-03-03T13:19:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8870f9e700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8858004250 0x7f8858004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8871fa0700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8858004e50 0x7f88580052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f887179f700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f886c151ad0 0x7f8858003ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8870f9e700 1 -- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f886c151ad0 msgr2=0x7f8858003ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:19:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8870f9e700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f886c151ad0 0x7f8858003ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8870f9e700 1 -- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8858004e50 msgr2=0x7f88580052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:19:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8870f9e700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8858004e50 0x7f88580052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8870f9e700 1 -- 172.21.15.7:0/1857581206 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88580035b0 con 0x7f8858004250 2023-03-03T13:19:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f887179f700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f886c151ad0 0x7f8858003ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:19:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.332+0000 7f8870f9e700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8858004250 0x7f8858004850 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f8868007fd0 tx=0x7f8858005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5165182d42999fbe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:19:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.333+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868001430 con 0x7f8858004250 2023-03-03T13:19:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.333+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88680052d0 con 0x7f8858004250 2023-03-03T13:19:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.333+0000 7f8863fff700 1 -- 172.21.15.7:0/1857581206 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868001d40 con 0x7f8858004250 2023-03-03T13:19:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.333+0000 7f88727a1700 1 -- 172.21.15.7:0/1857581206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8858004250 msgr2=0x7f8858004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.333+0000 7f88727a1700 1 --2- 172.21.15.7:0/1857581206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8858004250 0x7f8858004850 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f8868007fd0 tx=0x7f8858005880 comp rx=0 tx=0).stop 2023-03-03T13:19:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.333+0000 7f8871fa0700 1 -- 172.21.15.7:0/1857581206 reap_dead start 2023-03-03T13:19:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.333+0000 7f88727a1700 1 -- 172.21.15.7:0/1857581206 shutdown_connections 2023-03-03T13:19:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.333+0000 7f88727a1700 1 -- 172.21.15.7:0/1857581206 >> 172.21.15.7:0/1857581206 conn(0x7f886c06af40 msgr2=0x7f886c0c68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:19:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.333+0000 7f88727a1700 1 -- 172.21.15.7:0/1857581206 shutdown_connections 2023-03-03T13:19:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.333+0000 7f88727a1700 1 -- 172.21.15.7:0/1857581206 wait complete. 2023-03-03T13:19:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.334+0000 7f88727a1700 1 Processor -- start 2023-03-03T13:19:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.334+0000 7f88727a1700 1 -- start start 2023-03-03T13:19:25.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f88727a1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8858004250 0x7f886c0c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:25.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f88727a1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c0a97c0 0x7f886c14eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:25.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f88727a1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f886c0c2780 0x7f886c0c2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:25.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f88727a1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0ac650 con 0x7f886c0a97c0 2023-03-03T13:19:25.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f88727a1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c1ed1b0 con 0x7f886c0c2780 2023-03-03T13:19:25.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f88727a1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c0ac370 con 0x7f8858004250 2023-03-03T13:19:25.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f8870f9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c0a97c0 0x7f886c14eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:25.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f8870f9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c0a97c0 0x7f886c14eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44144/0 (socket says 172.21.15.7:44144) 2023-03-03T13:19:25.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f8870f9e700 1 -- 172.21.15.7:0/326112025 learned_addr learned my addr 172.21.15.7:0/326112025 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:19:25.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f8871fa0700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f886c0c2780 0x7f886c0c2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:25.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.335+0000 7f887179f700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8858004250 0x7f886c0c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:25.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f8870f9e700 1 -- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8858004250 msgr2=0x7f886c0c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:25.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f8870f9e700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8858004250 0x7f886c0c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:25.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f8870f9e700 1 -- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f886c0c2780 msgr2=0x7f886c0c2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:25.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f8870f9e700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f886c0c2780 0x7f886c0c2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:25.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f8870f9e700 1 -- 172.21.15.7:0/326112025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c158070 con 0x7f886c0a97c0 2023-03-03T13:19:25.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f8871fa0700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f886c0c2780 0x7f886c0c2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:19:25.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f8870f9e700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c0a97c0 0x7f886c14eac0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f8868006b10 tx=0x7f88680012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:19:25.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f8861ffb700 1 -- 172.21.15.7:0/326112025 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868001d40 con 0x7f886c0a97c0 2023-03-03T13:19:25.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f8861ffb700 1 -- 172.21.15.7:0/326112025 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88680013d0 con 0x7f886c0a97c0 2023-03-03T13:19:25.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f88727a1700 1 -- 172.21.15.7:0/326112025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886c157d90 con 0x7f886c0a97c0 2023-03-03T13:19:25.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f8861ffb700 1 -- 172.21.15.7:0/326112025 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868007260 con 0x7f886c0a97c0 2023-03-03T13:19:25.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.336+0000 7f88727a1700 1 -- 172.21.15.7:0/326112025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f886c154e10 con 0x7f886c0a97c0 2023-03-03T13:19:25.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.337+0000 7f88727a1700 1 -- 172.21.15.7:0/326112025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886c0bdbd0 con 0x7f886c0a97c0 2023-03-03T13:19:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.338+0000 7f8861ffb700 1 -- 172.21.15.7:0/326112025 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f886802e070 con 0x7f886c0a97c0 2023-03-03T13:19:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.338+0000 7f8861ffb700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88500682f0 0x7f885006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:19:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.338+0000 7f8861ffb700 1 -- 172.21.15.7:0/326112025 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8868087120 con 0x7f886c0a97c0 2023-03-03T13:19:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.338+0000 7f887179f700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88500682f0 0x7f885006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:19:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.339+0000 7f887179f700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88500682f0 0x7f885006a7a0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f885c009a30 tx=0x7f885c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:19:25.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.343+0000 7f8861ffb700 1 -- 172.21.15.7:0/326112025 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f886c0bdbd0 con 0x7f886c0a97c0 2023-03-03T13:19:25.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.579+0000 7f88727a1700 1 -- 172.21.15.7:0/326112025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f886c0bdbd0 con 0x7f886c0a97c0 2023-03-03T13:19:25.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.580+0000 7f8861ffb700 1 -- 172.21.15.7:0/326112025 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f886c0bdbd0 con 0x7f886c0a97c0 2023-03-03T13:19:25.582 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:19:25.582 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:19:25.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 -- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88500682f0 msgr2=0x7f885006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88500682f0 0x7f885006a7a0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f885c009a30 tx=0x7f885c008040 comp rx=0 tx=0).stop 2023-03-03T13:19:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 -- 172.21.15.7:0/326112025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c0a97c0 msgr2=0x7f886c14eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:19:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c0a97c0 0x7f886c14eac0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f8868006b10 tx=0x7f88680012d0 comp rx=0 tx=0).stop 2023-03-03T13:19:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 -- 172.21.15.7:0/326112025 shutdown_connections 2023-03-03T13:19:25.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f886c0c2780 0x7f886c0c2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:25.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8858004250 0x7f886c0c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:25.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88500682f0 0x7f885006a7a0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:25.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 --2- 172.21.15.7:0/326112025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f886c0a97c0 0x7f886c14eac0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:19:25.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 -- 172.21.15.7:0/326112025 >> 172.21.15.7:0/326112025 conn(0x7f886c06af40 msgr2=0x7f886c06c740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:19:25.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 -- 172.21.15.7:0/326112025 shutdown_connections 2023-03-03T13:19:25.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:19:25.582+0000 7f88727a1700 1 -- 172.21.15.7:0/326112025 wait complete. 2023-03-03T13:19:25.586 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:19:25.595 INFO:tasks.mds_thrash.fs.[b]:waiting for 66 secs before thrashing 2023-03-03T13:19:40.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:19:40.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T13:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T13:20:10.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:10.456 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:30.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:20:30.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.439+0000 7f68239d2700 1 Processor -- start 2023-03-03T13:20:30.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.440+0000 7f68239d2700 1 -- start start 2023-03-03T13:20:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.440+0000 7f68239d2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f681c0719e0 0x7f681c150930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.440+0000 7f68239d2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c0c14f0 0x7f681c0c1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.440+0000 7f68239d2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c0aae60 con 0x7f681c151e20 2023-03-03T13:20:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.440+0000 7f68239d2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c152270 con 0x7f681c0719e0 2023-03-03T13:20:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.440+0000 7f68239d2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0c2760 con 0x7f681c0c14f0 2023-03-03T13:20:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.440+0000 7f682176e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f681c0719e0 0x7f681c150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f682176e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f681c0719e0 0x7f681c150930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44182/0 (socket says 172.21.15.7:44182) 2023-03-03T13:20:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f682176e700 1 -- 172.21.15.7:0/3548999318 learned_addr learned my addr 172.21.15.7:0/3548999318 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:20:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f6821f6f700 1 --2- 172.21.15.7:0/3548999318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c0c14f0 0x7f681c0c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f682176e700 1 -- 172.21.15.7:0/3548999318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c0c14f0 msgr2=0x7f681c0c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:20:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f682176e700 1 --2- 172.21.15.7:0/3548999318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c0c14f0 0x7f681c0c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f682176e700 1 -- 172.21.15.7:0/3548999318 >> v1:172.21.15.143:6789/0 conn(0x7f681c151e20 legacy=0x7f681c0aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f682176e700 1 -- 172.21.15.7:0/3548999318 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c0a93d0 con 0x7f681c0719e0 2023-03-03T13:20:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f682176e700 1 --2- 172.21.15.7:0/3548999318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c0719e0 0x7f681c150930 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f681000bb90 tx=0x7f681000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c104ffcf5d39a0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f680ffff700 1 -- 172.21.15.7:0/3548999318 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681000dd50 con 0x7f681c0719e0 2023-03-03T13:20:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f680ffff700 1 -- 172.21.15.7:0/3548999318 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6810010470 con 0x7f681c0719e0 2023-03-03T13:20:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.441+0000 7f680ffff700 1 -- 172.21.15.7:0/3548999318 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810010800 con 0x7f681c0719e0 2023-03-03T13:20:30.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.442+0000 7f68239d2700 1 -- 172.21.15.7:0/3548999318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c0719e0 msgr2=0x7f681c150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:30.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.442+0000 7f68239d2700 1 --2- 172.21.15.7:0/3548999318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c0719e0 0x7f681c150930 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f681000bb90 tx=0x7f681000bea0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.442+0000 7f68239d2700 1 -- 172.21.15.7:0/3548999318 shutdown_connections 2023-03-03T13:20:30.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.442+0000 7f68239d2700 1 --2- 172.21.15.7:0/3548999318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c0c14f0 0x7f681c0c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.442+0000 7f68239d2700 1 --2- 172.21.15.7:0/3548999318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c0719e0 0x7f681c150930 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.442+0000 7f68239d2700 1 -- 172.21.15.7:0/3548999318 >> 172.21.15.7:0/3548999318 conn(0x7f681c0c62e0 msgr2=0x7f681c0c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:20:30.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.442+0000 7f68239d2700 1 -- 172.21.15.7:0/3548999318 shutdown_connections 2023-03-03T13:20:30.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.442+0000 7f68239d2700 1 -- 172.21.15.7:0/3548999318 wait complete. 2023-03-03T13:20:30.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.443+0000 7f68239d2700 1 Processor -- start 2023-03-03T13:20:30.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.444+0000 7f68239d2700 1 -- start start 2023-03-03T13:20:30.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.444+0000 7f68239d2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c0719e0 0x7f681c14b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:30.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.444+0000 7f68239d2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c0c14f0 0x7f681c1486d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:30.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.444+0000 7f68239d2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c151e20 0x7f681c148c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:30.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.444+0000 7f68239d2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c149150 con 0x7f681c151e20 2023-03-03T13:20:30.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f68239d2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0c60a0 con 0x7f681c0c14f0 2023-03-03T13:20:30.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f68239d2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0c4410 con 0x7f681c0719e0 2023-03-03T13:20:30.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6821f6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c151e20 0x7f681c148c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:30.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f682176e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c0719e0 0x7f681c14b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:30.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6821f6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c151e20 0x7f681c148c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44192/0 (socket says 172.21.15.7:44192) 2023-03-03T13:20:30.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6821f6f700 1 -- 172.21.15.7:0/3990252105 learned_addr learned my addr 172.21.15.7:0/3990252105 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:20:30.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6820f6d700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c0c14f0 0x7f681c1486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:30.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6821f6f700 1 -- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c0719e0 msgr2=0x7f681c14b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:30.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6821f6f700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c0719e0 0x7f681c14b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6821f6f700 1 -- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c0c14f0 msgr2=0x7f681c1486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:30.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6821f6f700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c0c14f0 0x7f681c1486d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6821f6f700 1 -- 172.21.15.7:0/3990252105 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681000b7f0 con 0x7f681c151e20 2023-03-03T13:20:30.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6820f6d700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c0c14f0 0x7f681c1486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:20:30.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f6821f6f700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c151e20 0x7f681c148c10 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f6814007e40 tx=0x7f681400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:30.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f680dffb700 1 -- 172.21.15.7:0/3990252105 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681400ed30 con 0x7f681c151e20 2023-03-03T13:20:30.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f680dffb700 1 -- 172.21.15.7:0/3990252105 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6814014420 con 0x7f681c151e20 2023-03-03T13:20:30.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f68239d2700 1 -- 172.21.15.7:0/3990252105 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c0a9990 con 0x7f681c151e20 2023-03-03T13:20:30.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.445+0000 7f680dffb700 1 -- 172.21.15.7:0/3990252105 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68140147d0 con 0x7f681c151e20 2023-03-03T13:20:30.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.446+0000 7f68239d2700 1 -- 172.21.15.7:0/3990252105 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c0a96b0 con 0x7f681c151e20 2023-03-03T13:20:30.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.447+0000 7f68239d2700 1 -- 172.21.15.7:0/3990252105 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c0a9c70 con 0x7f681c151e20 2023-03-03T13:20:30.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.447+0000 7f680dffb700 1 -- 172.21.15.7:0/3990252105 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6814012050 con 0x7f681c151e20 2023-03-03T13:20:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.447+0000 7f680dffb700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68000682f0 0x7f680006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.447+0000 7f680dffb700 1 -- 172.21.15.7:0/3990252105 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6814086670 con 0x7f681c151e20 2023-03-03T13:20:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.448+0000 7f682176e700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68000682f0 0x7f680006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.448+0000 7f682176e700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68000682f0 0x7f680006a7a0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f681000bee0 tx=0x7f6810013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.452+0000 7f680dffb700 1 -- 172.21.15.7:0/3990252105 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6814052a10 con 0x7f681c151e20 2023-03-03T13:20:30.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.687+0000 7f68239d2700 1 -- 172.21.15.7:0/3990252105 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f681c1e5e90 con 0x7f681c151e20 2023-03-03T13:20:30.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.688+0000 7f680dffb700 1 -- 172.21.15.7:0/3990252105 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6814056030 con 0x7f681c151e20 2023-03-03T13:20:30.689 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:20:30.690 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:20:30.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 -- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68000682f0 msgr2=0x7f680006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:30.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68000682f0 0x7f680006a7a0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f681000bee0 tx=0x7f6810013000 comp rx=0 tx=0).stop 2023-03-03T13:20:30.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 -- 172.21.15.7:0/3990252105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c151e20 msgr2=0x7f681c148c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:30.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c151e20 0x7f681c148c10 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f6814007e40 tx=0x7f681400e3c0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 -- 172.21.15.7:0/3990252105 shutdown_connections 2023-03-03T13:20:30.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68000682f0 0x7f680006a7a0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f681c151e20 0x7f681c148c10 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f681c0719e0 0x7f681c14b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 --2- 172.21.15.7:0/3990252105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f681c0c14f0 0x7f681c1486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:30.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 -- 172.21.15.7:0/3990252105 >> 172.21.15.7:0/3990252105 conn(0x7f681c0c62e0 msgr2=0x7f681c0c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:20:30.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 -- 172.21.15.7:0/3990252105 shutdown_connections 2023-03-03T13:20:30.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:30.690+0000 7f68239d2700 1 -- 172.21.15.7:0/3990252105 wait complete. 2023-03-03T13:20:30.693 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:20:30.704 INFO:tasks.mds_thrash.fs.[a]:waiting for 71 secs before thrashing 2023-03-03T13:20:31.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:20:31.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.733+0000 7f8063973700 1 Processor -- start 2023-03-03T13:20:31.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.733+0000 7f8063973700 1 -- start start 2023-03-03T13:20:31.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8063973700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c155410 0x7f805c0bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:31.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8063973700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c0c08d0 con 0x7f805c0c5b00 2023-03-03T13:20:31.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8063973700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f805c154930 con 0x7f805c151e80 2023-03-03T13:20:31.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8063973700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805c152250 con 0x7f805c155410 2023-03-03T13:20:31.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8061f10700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f805c0c5b00 0x7f805c0bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38190/0 (socket says 172.21.15.7:38190) 2023-03-03T13:20:31.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8061f10700 1 -- 172.21.15.7:0/1791614538 learned_addr learned my addr 172.21.15.7:0/1791614538 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:20:31.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8060f0e700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c155410 0x7f805c0bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:31.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408028250 0 0) 0x7f805c0c08d0 con 0x7f805c0c5b00 2023-03-03T13:20:31.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80480035b0 con 0x7f805c0c5b00 2023-03-03T13:20:31.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047925088 0 0) 0x7f805c154930 con 0x7f805c151e80 2023-03-03T13:20:31.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f805c0c08d0 con 0x7f805c151e80 2023-03-03T13:20:31.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1426816346 0 0) 0x7f80480035b0 con 0x7f805c0c5b00 2023-03-03T13:20:31.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c155410 msgr2=0x7f805c0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:20:31.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8053fff700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c155410 0x7f805c0bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:31.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.734+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 >> v1:172.21.15.143:6789/0 conn(0x7f805c151e80 legacy=0x7f805c0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:31.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c154c10 con 0x7f805c0c5b00 2023-03-03T13:20:31.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f80540019f0 con 0x7f805c0c5b00 2023-03-03T13:20:31.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 >> v1:172.21.15.7:6789/0 conn(0x7f805c0c5b00 legacy=0x7f805c0bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:31.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8048003c30 0x7f8048004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:31.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8048004850 0x7f8048004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:31.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c155410 0x7f8048005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:31.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804800bb10 con 0x7f8048003c30 2023-03-03T13:20:31.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804800bc80 con 0x7f8048004850 2023-03-03T13:20:31.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804800be40 con 0x7f805c155410 2023-03-03T13:20:31.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f806170f700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8048003c30 0x7f8048004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:31.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8060f0e700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8048004850 0x7f8048004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:31.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8061f10700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c155410 0x7f8048005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:31.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f806170f700 1 -- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c155410 msgr2=0x7f8048005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:20:31.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f806170f700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f805c155410 0x7f8048005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:31.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f806170f700 1 -- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8048004850 msgr2=0x7f8048004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:20:31.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f806170f700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8048004850 0x7f8048004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:31.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f806170f700 1 -- 172.21.15.7:0/1791614538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80480035b0 con 0x7f8048003c30 2023-03-03T13:20:31.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f806170f700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8048003c30 0x7f8048004020 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f8058001940 tx=0x7f8048005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddd37385534401e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:31.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058001430 con 0x7f8048003c30 2023-03-03T13:20:31.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8058001d70 con 0x7f8048003c30 2023-03-03T13:20:31.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.735+0000 7f8053fff700 1 -- 172.21.15.7:0/1791614538 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058005510 con 0x7f8048003c30 2023-03-03T13:20:31.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.736+0000 7f8063973700 1 -- 172.21.15.7:0/1791614538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8048003c30 msgr2=0x7f8048004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:31.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.736+0000 7f8063973700 1 --2- 172.21.15.7:0/1791614538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8048003c30 0x7f8048004020 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f8058001940 tx=0x7f8048005880 comp rx=0 tx=0).stop 2023-03-03T13:20:31.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.736+0000 7f8061f10700 1 -- 172.21.15.7:0/1791614538 reap_dead start 2023-03-03T13:20:31.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.736+0000 7f8063973700 1 -- 172.21.15.7:0/1791614538 shutdown_connections 2023-03-03T13:20:31.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.736+0000 7f8063973700 1 -- 172.21.15.7:0/1791614538 >> 172.21.15.7:0/1791614538 conn(0x7f805c071630 msgr2=0x7f805c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:20:31.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.736+0000 7f8063973700 1 -- 172.21.15.7:0/1791614538 shutdown_connections 2023-03-03T13:20:31.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.736+0000 7f8063973700 1 -- 172.21.15.7:0/1791614538 wait complete. 2023-03-03T13:20:31.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.737+0000 7f8063973700 1 Processor -- start 2023-03-03T13:20:31.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.737+0000 7f8063973700 1 -- start start 2023-03-03T13:20:31.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8063973700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8048003c30 0x7f805c149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:31.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8063973700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c14e490 0x7f805c1503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:31.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8063973700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c1eb560 0x7f805c1eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:31.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8063973700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c148d80 con 0x7f805c1eb560 2023-03-03T13:20:31.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8063973700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c14e000 con 0x7f805c14e490 2023-03-03T13:20:31.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8063973700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805c14d6f0 con 0x7f8048003c30 2023-03-03T13:20:31.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8061f10700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c1eb560 0x7f805c1eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:31.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8061f10700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c1eb560 0x7f805c1eb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44222/0 (socket says 172.21.15.7:44222) 2023-03-03T13:20:31.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8061f10700 1 -- 172.21.15.7:0/3176536946 learned_addr learned my addr 172.21.15.7:0/3176536946 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:20:31.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8060f0e700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c14e490 0x7f805c1503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:31.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f806170f700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8048003c30 0x7f805c149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:31.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8061f10700 1 -- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8048003c30 msgr2=0x7f805c149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:31.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8061f10700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8048003c30 0x7f805c149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:31.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8061f10700 1 -- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c14e490 msgr2=0x7f805c1503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:31.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8061f10700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c14e490 0x7f805c1503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:31.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8061f10700 1 -- 172.21.15.7:0/3176536946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c0c5220 con 0x7f805c1eb560 2023-03-03T13:20:31.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.738+0000 7f8060f0e700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c14e490 0x7f805c1503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:20:31.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.739+0000 7f8061f10700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c1eb560 0x7f805c1eb9d0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f805c0b03f0 tx=0x7f805400e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:31.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.739+0000 7f8051ffb700 1 -- 172.21.15.7:0/3176536946 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805400ec70 con 0x7f805c1eb560 2023-03-03T13:20:31.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.739+0000 7f8051ffb700 1 -- 172.21.15.7:0/3176536946 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805400edd0 con 0x7f805c1eb560 2023-03-03T13:20:31.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.739+0000 7f8063973700 1 -- 172.21.15.7:0/3176536946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805c154ef0 con 0x7f805c1eb560 2023-03-03T13:20:31.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.739+0000 7f8051ffb700 1 -- 172.21.15.7:0/3176536946 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054013660 con 0x7f805c1eb560 2023-03-03T13:20:31.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.739+0000 7f8063973700 1 -- 172.21.15.7:0/3176536946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805c0c0530 con 0x7f805c1eb560 2023-03-03T13:20:31.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.740+0000 7f8063973700 1 -- 172.21.15.7:0/3176536946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805c14e1b0 con 0x7f805c1eb560 2023-03-03T13:20:31.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.740+0000 7f8051ffb700 1 -- 172.21.15.7:0/3176536946 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f80540137c0 con 0x7f805c1eb560 2023-03-03T13:20:31.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.741+0000 7f8051ffb700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 0x7f804006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:31.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.741+0000 7f8051ffb700 1 -- 172.21.15.7:0/3176536946 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8054086830 con 0x7f805c1eb560 2023-03-03T13:20:31.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.741+0000 7f806170f700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 0x7f804006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:31.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.742+0000 7f806170f700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 0x7f804006a7d0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f8058015040 tx=0x7f805800a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:31.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.745+0000 7f8051ffb700 1 -- 172.21.15.7:0/3176536946 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f805c14e1b0 con 0x7f805c1eb560 2023-03-03T13:20:31.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.983+0000 7f8063973700 1 -- 172.21.15.7:0/3176536946 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f805c14e170 con 0x7f805c1eb560 2023-03-03T13:20:31.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.983+0000 7f8051ffb700 1 -- 172.21.15.7:0/3176536946 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8054052bd0 con 0x7f805c1eb560 2023-03-03T13:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:20:31.985 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:20:31.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 -- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 msgr2=0x7f804006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:31.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 0x7f804006a7d0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f8058015040 tx=0x7f805800a000 comp rx=0 tx=0).stop 2023-03-03T13:20:31.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 -- 172.21.15.7:0/3176536946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c1eb560 msgr2=0x7f805c1eb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:31.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c1eb560 0x7f805c1eb9d0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f805c0b03f0 tx=0x7f805400e440 comp rx=0 tx=0).stop 2023-03-03T13:20:31.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 -- 172.21.15.7:0/3176536946 shutdown_connections 2023-03-03T13:20:31.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8040068320 0x7f804006a7d0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:31.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f805c1eb560 0x7f805c1eb9d0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:31.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8048003c30 0x7f805c149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:31.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 --2- 172.21.15.7:0/3176536946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f805c14e490 0x7f805c1503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:31.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 -- 172.21.15.7:0/3176536946 >> 172.21.15.7:0/3176536946 conn(0x7f805c071630 msgr2=0x7f805c0c6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:20:31.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 -- 172.21.15.7:0/3176536946 shutdown_connections 2023-03-03T13:20:31.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:31.985+0000 7f8063973700 1 -- 172.21.15.7:0/3176536946 wait complete. 2023-03-03T13:20:31.989 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:20:31.999 INFO:tasks.mds_thrash.fs.[b]:waiting for 17 secs before thrashing 2023-03-03T13:20:40.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:40.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:20:48.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:20:49.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.136+0000 7fab4410b700 1 Processor -- start 2023-03-03T13:20:49.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.136+0000 7fab4410b700 1 -- start start 2023-03-03T13:20:49.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab4410b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab3c0c6010 0x7fab3c0c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:49.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab4410b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3c14ebe0 con 0x7fab3c0c17f0 2023-03-03T13:20:49.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab4410b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3c0c2e70 con 0x7fab3c0c4fc0 2023-03-03T13:20:49.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab4410b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3c152100 con 0x7fab3c0c6010 2023-03-03T13:20:49.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab41ea7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab3c0c6010 0x7fab3c0c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:49.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab416a6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fab3c0c17f0 0x7fab3c06d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38214/0 (socket says 172.21.15.7:38214) 2023-03-03T13:20:49.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab416a6700 1 -- 172.21.15.7:0/3454903611 learned_addr learned my addr 172.21.15.7:0/3454903611 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:20:49.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab41ea7700 1 -- 172.21.15.7:0/3454903611 >> v1:172.21.15.143:6789/0 conn(0x7fab3c0c4fc0 legacy=0x7fab3c06ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:49.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab41ea7700 1 -- 172.21.15.7:0/3454903611 >> v1:172.21.15.7:6789/0 conn(0x7fab3c0c17f0 legacy=0x7fab3c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:49.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab40ea5700 1 -- 172.21.15.7:0/3454903611 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799149443 0 0) 0x7fab3c0c2e70 con 0x7fab3c0c4fc0 2023-03-03T13:20:49.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.137+0000 7fab41ea7700 1 -- 172.21.15.7:0/3454903611 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c0aa680 con 0x7fab3c0c6010 2023-03-03T13:20:49.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab41ea7700 1 --2- 172.21.15.7:0/3454903611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 0x7fab3c0c4a80 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fab2c009840 tx=0x7fab2c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1f77a171519d40 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:49.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab40ea5700 1 -- 172.21.15.7:0/3454903611 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c011830 con 0x7fab3c0c6010 2023-03-03T13:20:49.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab40ea5700 1 -- 172.21.15.7:0/3454903611 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2c011990 con 0x7fab3c0c6010 2023-03-03T13:20:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab40ea5700 1 -- 172.21.15.7:0/3454903611 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c011d20 con 0x7fab3c0c6010 2023-03-03T13:20:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab4410b700 1 -- 172.21.15.7:0/3454903611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 msgr2=0x7fab3c0c4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab4410b700 1 --2- 172.21.15.7:0/3454903611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 0x7fab3c0c4a80 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7fab2c009840 tx=0x7fab2c009b50 comp rx=0 tx=0).stop 2023-03-03T13:20:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab4410b700 1 -- 172.21.15.7:0/3454903611 shutdown_connections 2023-03-03T13:20:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab4410b700 1 --2- 172.21.15.7:0/3454903611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 0x7fab3c0c4a80 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab4410b700 1 -- 172.21.15.7:0/3454903611 >> 172.21.15.7:0/3454903611 conn(0x7fab3c0a9430 msgr2=0x7fab3c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:20:49.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab4410b700 1 -- 172.21.15.7:0/3454903611 shutdown_connections 2023-03-03T13:20:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.138+0000 7fab4410b700 1 -- 172.21.15.7:0/3454903611 wait complete. 2023-03-03T13:20:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.140+0000 7fab4410b700 1 Processor -- start 2023-03-03T13:20:49.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.140+0000 7fab4410b700 1 -- start start 2023-03-03T13:20:49.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.140+0000 7fab4410b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab3c0c17f0 0x7fab3c0c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.140+0000 7fab4410b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c0c4fc0 0x7fab3c154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.140+0000 7fab4410b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 0x7fab3c154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.140+0000 7fab4410b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c1ea930 con 0x7fab3c0c6010 2023-03-03T13:20:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.140+0000 7fab4410b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c0700f0 con 0x7fab3c0c17f0 2023-03-03T13:20:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.140+0000 7fab4410b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c06fba0 con 0x7fab3c0c4fc0 2023-03-03T13:20:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab426a8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 0x7fab3c154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:49.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab426a8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 0x7fab3c154940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44242/0 (socket says 172.21.15.7:44242) 2023-03-03T13:20:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab426a8700 1 -- 172.21.15.7:0/1355552028 learned_addr learned my addr 172.21.15.7:0/1355552028 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:20:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab41ea7700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab3c0c17f0 0x7fab3c0c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab416a6700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c0c4fc0 0x7fab3c154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab426a8700 1 -- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c0c4fc0 msgr2=0x7fab3c154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab426a8700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c0c4fc0 0x7fab3c154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab426a8700 1 -- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab3c0c17f0 msgr2=0x7fab3c0c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:49.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab426a8700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab3c0c17f0 0x7fab3c0c00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:49.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab426a8700 1 -- 172.21.15.7:0/1355552028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c0094a0 con 0x7fab3c0c6010 2023-03-03T13:20:49.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab426a8700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 0x7fab3c154940 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fab34001560 tx=0x7fab3400cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:49.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab327fc700 1 -- 172.21.15.7:0/1355552028 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34003e30 con 0x7fab3c0c6010 2023-03-03T13:20:49.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab327fc700 1 -- 172.21.15.7:0/1355552028 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab34005070 con 0x7fab3c0c6010 2023-03-03T13:20:49.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.141+0000 7fab4410b700 1 -- 172.21.15.7:0/1355552028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3c0c2b00 con 0x7fab3c0c6010 2023-03-03T13:20:49.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.142+0000 7fab327fc700 1 -- 172.21.15.7:0/1355552028 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34005420 con 0x7fab3c0c6010 2023-03-03T13:20:49.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.142+0000 7fab4410b700 1 -- 172.21.15.7:0/1355552028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c0aa960 con 0x7fab3c0c6010 2023-03-03T13:20:49.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.143+0000 7fab4410b700 1 -- 172.21.15.7:0/1355552028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3c0c3300 con 0x7fab3c0c6010 2023-03-03T13:20:49.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.143+0000 7fab327fc700 1 -- 172.21.15.7:0/1355552028 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fab34019030 con 0x7fab3c0c6010 2023-03-03T13:20:49.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.143+0000 7fab327fc700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab200682f0 0x7fab2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:20:49.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.144+0000 7fab327fc700 1 -- 172.21.15.7:0/1355552028 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fab340856f0 con 0x7fab3c0c6010 2023-03-03T13:20:49.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.144+0000 7fab41ea7700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab200682f0 0x7fab2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:20:49.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.144+0000 7fab41ea7700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab200682f0 0x7fab2006a7a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0090b0 tx=0x7fab2c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:20:49.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.148+0000 7fab327fc700 1 -- 172.21.15.7:0/1355552028 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab34051a90 con 0x7fab3c0c6010 2023-03-03T13:20:49.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.386+0000 7fab4410b700 1 -- 172.21.15.7:0/1355552028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab3c074bb0 con 0x7fab3c0c6010 2023-03-03T13:20:49.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.387+0000 7fab327fc700 1 -- 172.21.15.7:0/1355552028 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fab34017090 con 0x7fab3c0c6010 2023-03-03T13:20:49.388 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:20:49.389 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:20:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 -- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab200682f0 msgr2=0x7fab2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab200682f0 0x7fab2006a7a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0090b0 tx=0x7fab2c01c040 comp rx=0 tx=0).stop 2023-03-03T13:20:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 -- 172.21.15.7:0/1355552028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 msgr2=0x7fab3c154940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:20:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 0x7fab3c154940 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fab34001560 tx=0x7fab3400cb50 comp rx=0 tx=0).stop 2023-03-03T13:20:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 -- 172.21.15.7:0/1355552028 shutdown_connections 2023-03-03T13:20:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab200682f0 0x7fab2006a7a0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c6010 0x7fab3c154940 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:49.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab3c0c17f0 0x7fab3c0c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:49.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 --2- 172.21.15.7:0/1355552028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c0c4fc0 0x7fab3c154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:20:49.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 -- 172.21.15.7:0/1355552028 >> 172.21.15.7:0/1355552028 conn(0x7fab3c0a9430 msgr2=0x7fab3c1dcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:20:49.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 -- 172.21.15.7:0/1355552028 shutdown_connections 2023-03-03T13:20:49.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:20:49.389+0000 7fab4410b700 1 -- 172.21.15.7:0/1355552028 wait complete. 2023-03-03T13:20:49.392 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:20:49.402 INFO:tasks.mds_thrash.fs.[b]:waiting for 28 secs before thrashing 2023-03-03T13:21:10.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:10.734 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:17.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:21:17.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.541+0000 7fa41f2da700 1 Processor -- start 2023-03-03T13:21:17.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.541+0000 7fa41f2da700 1 -- start start 2023-03-03T13:21:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.541+0000 7fa41f2da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa418151d40 0x7fa4180736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.541+0000 7fa41f2da700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa4180c2930 0x7fa418157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41f2da700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41806d430 con 0x7fa418073c30 2023-03-03T13:21:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41f2da700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41806c2a0 con 0x7fa4180c2930 2023-03-03T13:21:17.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41f2da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4181586e0 con 0x7fa418151d40 2023-03-03T13:21:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41d076700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa418151d40 0x7fa4180736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41c875700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa4180c2930 0x7fa418157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41d877700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fa418073c30 0x7fa418157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:41770/0 (socket says 172.21.15.7:41770) 2023-03-03T13:21:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41d877700 1 -- 172.21.15.7:0/601586314 learned_addr learned my addr 172.21.15.7:0/601586314 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:21:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa40bfff700 1 -- 172.21.15.7:0/601586314 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882381169 0 0) 0x7fa41806d430 con 0x7fa418073c30 2023-03-03T13:21:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa40bfff700 1 -- 172.21.15.7:0/601586314 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4040035b0 con 0x7fa418073c30 2023-03-03T13:21:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41d076700 1 -- 172.21.15.7:0/601586314 >> v2:172.21.15.143:3300/0 conn(0x7fa4180c2930 msgr2=0x7fa418157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:21:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41d076700 1 --2- 172.21.15.7:0/601586314 >> v2:172.21.15.143:3300/0 conn(0x7fa4180c2930 0x7fa418157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41d076700 1 -- 172.21.15.7:0/601586314 >> v1:172.21.15.143:6789/0 conn(0x7fa418073c30 legacy=0x7fa418157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41d076700 1 -- 172.21.15.7:0/601586314 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41806cb00 con 0x7fa418151d40 2023-03-03T13:21:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.542+0000 7fa41c875700 1 --2- 172.21.15.7:0/601586314 >> v2:172.21.15.143:3300/0 conn(0x7fa4180c2930 0x7fa418157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:21:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.543+0000 7fa41d076700 1 --2- 172.21.15.7:0/601586314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa418151d40 0x7fa4180736f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa414009840 tx=0x7fa414009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1516d22e4626e43a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.543+0000 7fa40bfff700 1 -- 172.21.15.7:0/601586314 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414011830 con 0x7fa418151d40 2023-03-03T13:21:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.543+0000 7fa40bfff700 1 -- 172.21.15.7:0/601586314 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa414011990 con 0x7fa418151d40 2023-03-03T13:21:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.543+0000 7fa40bfff700 1 -- 172.21.15.7:0/601586314 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414011d20 con 0x7fa418151d40 2023-03-03T13:21:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.543+0000 7fa41f2da700 1 -- 172.21.15.7:0/601586314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa418151d40 msgr2=0x7fa4180736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.543+0000 7fa41f2da700 1 --2- 172.21.15.7:0/601586314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa418151d40 0x7fa4180736f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa414009840 tx=0x7fa414009b50 comp rx=0 tx=0).stop 2023-03-03T13:21:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.543+0000 7fa41f2da700 1 -- 172.21.15.7:0/601586314 shutdown_connections 2023-03-03T13:21:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.543+0000 7fa41f2da700 1 --2- 172.21.15.7:0/601586314 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa418151d40 0x7fa4180736f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.543+0000 7fa41f2da700 1 --2- 172.21.15.7:0/601586314 >> v2:172.21.15.143:3300/0 conn(0x7fa4180c2930 0x7fa418157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.543+0000 7fa41f2da700 1 -- 172.21.15.7:0/601586314 >> 172.21.15.7:0/601586314 conn(0x7fa4180a94d0 msgr2=0x7fa4180a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:21:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.544+0000 7fa41f2da700 1 -- 172.21.15.7:0/601586314 shutdown_connections 2023-03-03T13:21:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.544+0000 7fa41f2da700 1 -- 172.21.15.7:0/601586314 wait complete. 2023-03-03T13:21:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.545+0000 7fa41f2da700 1 Processor -- start 2023-03-03T13:21:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.545+0000 7fa41f2da700 1 -- start start 2023-03-03T13:21:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41f2da700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa418073c30 0x7fa41814e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41f2da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4180c2930 0x7fa4180aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41f2da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa418151d40 0x7fa4181eda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41f2da700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4180ab870 con 0x7fa418151d40 2023-03-03T13:21:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41f2da700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41814ed60 con 0x7fa418073c30 2023-03-03T13:21:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41f2da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa418154cf0 con 0x7fa4180c2930 2023-03-03T13:21:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41d877700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa418151d40 0x7fa4181eda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41d877700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa418151d40 0x7fa4181eda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44276/0 (socket says 172.21.15.7:44276) 2023-03-03T13:21:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41d877700 1 -- 172.21.15.7:0/133228194 learned_addr learned my addr 172.21.15.7:0/133228194 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:21:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41c875700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4180c2930 0x7fa4180aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.546+0000 7fa41d076700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa418073c30 0x7fa41814e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa41d877700 1 -- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4180c2930 msgr2=0x7fa4180aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa41d877700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4180c2930 0x7fa4180aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa41d877700 1 -- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa418073c30 msgr2=0x7fa41814e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa41d877700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa418073c30 0x7fa41814e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:17.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa41d877700 1 -- 172.21.15.7:0/133228194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40c0094a0 con 0x7fa418151d40 2023-03-03T13:21:17.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa41d877700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa418151d40 0x7fa4181eda60 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fa41000cad0 tx=0x7fa41000ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:17.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa409ffb700 1 -- 172.21.15.7:0/133228194 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41000aac0 con 0x7fa418151d40 2023-03-03T13:21:17.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa409ffb700 1 -- 172.21.15.7:0/133228194 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41000ac20 con 0x7fa418151d40 2023-03-03T13:21:17.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa41f2da700 1 -- 172.21.15.7:0/133228194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4140094a0 con 0x7fa418151d40 2023-03-03T13:21:17.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa409ffb700 1 -- 172.21.15.7:0/133228194 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa410012400 con 0x7fa418151d40 2023-03-03T13:21:17.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.547+0000 7fa41f2da700 1 -- 172.21.15.7:0/133228194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41806d0c0 con 0x7fa418151d40 2023-03-03T13:21:17.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.548+0000 7fa41f2da700 1 -- 172.21.15.7:0/133228194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41806cde0 con 0x7fa418151d40 2023-03-03T13:21:17.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.548+0000 7fa409ffb700 1 -- 172.21.15.7:0/133228194 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa410019030 con 0x7fa418151d40 2023-03-03T13:21:17.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.549+0000 7fa409ffb700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3fc0682f0 0x7fa3fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:17.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.549+0000 7fa409ffb700 1 -- 172.21.15.7:0/133228194 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa410085d80 con 0x7fa418151d40 2023-03-03T13:21:17.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.549+0000 7fa41d076700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3fc0682f0 0x7fa3fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:17.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.549+0000 7fa41d076700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3fc0682f0 0x7fa3fc06a7a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fa414000c00 tx=0x7fa414008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:17.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.554+0000 7fa409ffb700 1 -- 172.21.15.7:0/133228194 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa410052120 con 0x7fa418151d40 2023-03-03T13:21:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.789+0000 7fa41f2da700 1 -- 172.21.15.7:0/133228194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa41806bea0 con 0x7fa418151d40 2023-03-03T13:21:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.790+0000 7fa409ffb700 1 -- 172.21.15.7:0/133228194 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa410055740 con 0x7fa418151d40 2023-03-03T13:21:17.791 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:21:17.792 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:21:17.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 -- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3fc0682f0 msgr2=0x7fa3fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:17.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3fc0682f0 0x7fa3fc06a7a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fa414000c00 tx=0x7fa414008030 comp rx=0 tx=0).stop 2023-03-03T13:21:17.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 -- 172.21.15.7:0/133228194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa418151d40 msgr2=0x7fa4181eda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:17.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa418151d40 0x7fa4181eda60 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fa41000cad0 tx=0x7fa41000ce90 comp rx=0 tx=0).stop 2023-03-03T13:21:17.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 -- 172.21.15.7:0/133228194 shutdown_connections 2023-03-03T13:21:17.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3fc0682f0 0x7fa3fc06a7a0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa418151d40 0x7fa4181eda60 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa418073c30 0x7fa41814e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 --2- 172.21.15.7:0/133228194 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4180c2930 0x7fa4180aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 -- 172.21.15.7:0/133228194 >> 172.21.15.7:0/133228194 conn(0x7fa4180a94d0 msgr2=0x7fa418157200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:21:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 -- 172.21.15.7:0/133228194 shutdown_connections 2023-03-03T13:21:17.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:17.792+0000 7fa41f2da700 1 -- 172.21.15.7:0/133228194 wait complete. 2023-03-03T13:21:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:21:17.805 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-03-03T13:21:40.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:40.881 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:21:41.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:21:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.842+0000 7f2fb1833700 1 Processor -- start 2023-03-03T13:21:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.842+0000 7f2fb1833700 1 -- start start 2023-03-03T13:21:41.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.843+0000 7f2fb1833700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2fac06e020 0x7f2fac071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:41.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.843+0000 7f2fb1833700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac072520 0x7f2fac151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:41.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.843+0000 7f2fb1833700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac154910 con 0x7f2fac155390 2023-03-03T13:21:41.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.843+0000 7f2fb1833700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac070b90 con 0x7f2fac06e020 2023-03-03T13:21:41.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.843+0000 7f2fb1833700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac151320 con 0x7f2fac072520 2023-03-03T13:21:41.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.843+0000 7f2faa7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2fac06e020 0x7f2fac071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:41.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.843+0000 7f2faaffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f2fac155390 0x7f2fac0c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:41810/0 (socket says 172.21.15.7:41810) 2023-03-03T13:21:41.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.843+0000 7f2faaffd700 1 -- 172.21.15.7:0/2452735306 learned_addr learned my addr 172.21.15.7:0/2452735306 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:21:41.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.843+0000 7f2fab7fe700 1 --2- 172.21.15.7:0/2452735306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac072520 0x7f2fac151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:41.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.844+0000 7f2fab7fe700 1 -- 172.21.15.7:0/2452735306 >> v2:172.21.15.143:3300/0 conn(0x7f2fac06e020 msgr2=0x7f2fac071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:21:41.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.844+0000 7f2fab7fe700 1 --2- 172.21.15.7:0/2452735306 >> v2:172.21.15.143:3300/0 conn(0x7f2fac06e020 0x7f2fac071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:41.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.844+0000 7f2fab7fe700 1 -- 172.21.15.7:0/2452735306 >> v1:172.21.15.143:6789/0 conn(0x7f2fac155390 legacy=0x7f2fac0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:41.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.844+0000 7f2fab7fe700 1 -- 172.21.15.7:0/2452735306 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac0c4540 con 0x7f2fac072520 2023-03-03T13:21:41.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.844+0000 7f2fa9ffb700 1 -- 172.21.15.7:0/2452735306 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737149971 0 0) 0x7f2fac154910 con 0x7f2fac155390 2023-03-03T13:21:41.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.844+0000 7f2fab7fe700 1 --2- 172.21.15.7:0/2452735306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac072520 0x7f2fac151770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4009d90 tx=0x7f2fa400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab58f9e78f9c705e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:41.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.844+0000 7f2fa9ffb700 1 -- 172.21.15.7:0/2452735306 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400e070 con 0x7f2fac072520 2023-03-03T13:21:41.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.844+0000 7f2fa9ffb700 1 -- 172.21.15.7:0/2452735306 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa400bb20 con 0x7f2fac072520 2023-03-03T13:21:41.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.844+0000 7f2fa9ffb700 1 -- 172.21.15.7:0/2452735306 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4012430 con 0x7f2fac072520 2023-03-03T13:21:41.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.845+0000 7f2fb1833700 1 -- 172.21.15.7:0/2452735306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac072520 msgr2=0x7f2fac151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:41.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.845+0000 7f2fb1833700 1 --2- 172.21.15.7:0/2452735306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac072520 0x7f2fac151770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4009d90 tx=0x7f2fa400b330 comp rx=0 tx=0).stop 2023-03-03T13:21:41.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.845+0000 7f2fb1833700 1 -- 172.21.15.7:0/2452735306 shutdown_connections 2023-03-03T13:21:41.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.845+0000 7f2fb1833700 1 --2- 172.21.15.7:0/2452735306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac072520 0x7f2fac151770 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:41.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.845+0000 7f2fb1833700 1 --2- 172.21.15.7:0/2452735306 >> v2:172.21.15.143:3300/0 conn(0x7f2fac06e020 0x7f2fac071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:41.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.845+0000 7f2fb1833700 1 -- 172.21.15.7:0/2452735306 >> 172.21.15.7:0/2452735306 conn(0x7f2fac06c110 msgr2=0x7f2fac06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:21:41.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.845+0000 7f2fb1833700 1 -- 172.21.15.7:0/2452735306 shutdown_connections 2023-03-03T13:21:41.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.845+0000 7f2fb1833700 1 -- 172.21.15.7:0/2452735306 wait complete. 2023-03-03T13:21:41.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.846+0000 7f2fb1833700 1 Processor -- start 2023-03-03T13:21:41.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.846+0000 7f2fb1833700 1 -- start start 2023-03-03T13:21:41.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2fb1833700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fac06e020 0x7f2fac150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:41.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2fb1833700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fac072520 0x7f2fac14c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:41.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2fb1833700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac155390 0x7f2fac147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:41.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2fb1833700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac1ead00 con 0x7f2fac06e020 2023-03-03T13:21:41.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2fb1833700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac1e8500 con 0x7f2fac072520 2023-03-03T13:21:41.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2fb1833700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac1eaf10 con 0x7f2fac155390 2023-03-03T13:21:41.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2faaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fac06e020 0x7f2fac150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:41.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2faaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fac06e020 0x7f2fac150890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44316/0 (socket says 172.21.15.7:44316) 2023-03-03T13:21:41.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2faaffd700 1 -- 172.21.15.7:0/4042525126 learned_addr learned my addr 172.21.15.7:0/4042525126 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:21:41.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2fab7fe700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac155390 0x7f2fac147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:41.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2faa7fc700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fac072520 0x7f2fac14c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:41.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2faaffd700 1 -- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac155390 msgr2=0x7f2fac147f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:41.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2faaffd700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac155390 0x7f2fac147f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:41.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2faaffd700 1 -- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fac072520 msgr2=0x7f2fac14c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:41.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2faaffd700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fac072520 0x7f2fac14c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:41.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.847+0000 7f2faaffd700 1 -- 172.21.15.7:0/4042525126 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa4009a70 con 0x7f2fac06e020 2023-03-03T13:21:41.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.848+0000 7f2faaffd700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fac06e020 0x7f2fac150890 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400c550 tx=0x7f2f9400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.848+0000 7f2fa3fff700 1 -- 172.21.15.7:0/4042525126 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f94004c80 con 0x7f2fac06e020 2023-03-03T13:21:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.848+0000 7f2fa3fff700 1 -- 172.21.15.7:0/4042525126 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f94004e00 con 0x7f2fac06e020 2023-03-03T13:21:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.848+0000 7f2fb1833700 1 -- 172.21.15.7:0/4042525126 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fac070540 con 0x7f2fac06e020 2023-03-03T13:21:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.848+0000 7f2fa3fff700 1 -- 172.21.15.7:0/4042525126 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f94007310 con 0x7f2fac06e020 2023-03-03T13:21:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.848+0000 7f2fb1833700 1 -- 172.21.15.7:0/4042525126 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac0702a0 con 0x7f2fac06e020 2023-03-03T13:21:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.849+0000 7f2fb1833700 1 -- 172.21.15.7:0/4042525126 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fac070820 con 0x7f2fac06e020 2023-03-03T13:21:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.849+0000 7f2fa3fff700 1 -- 172.21.15.7:0/4042525126 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f9401a020 con 0x7f2fac06e020 2023-03-03T13:21:41.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.850+0000 7f2fa3fff700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f8c0682f0 0x7f2f8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:21:41.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.850+0000 7f2fa3fff700 1 -- 172.21.15.7:0/4042525126 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2f94085e00 con 0x7f2fac06e020 2023-03-03T13:21:41.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.850+0000 7f2faa7fc700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f8c0682f0 0x7f2f8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:21:41.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.851+0000 7f2faa7fc700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f8c0682f0 0x7f2f8c06a7a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0c2110 tx=0x7f2f9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:21:41.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:41.854+0000 7f2fa3fff700 1 -- 172.21.15.7:0/4042525126 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f940521f0 con 0x7f2fac06e020 2023-03-03T13:21:42.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.092+0000 7f2fb1833700 1 -- 172.21.15.7:0/4042525126 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fac155760 con 0x7f2fac06e020 2023-03-03T13:21:42.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.093+0000 7f2fa3fff700 1 -- 172.21.15.7:0/4042525126 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2f94055810 con 0x7f2fac06e020 2023-03-03T13:21:42.094 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:21:42.095 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:21:42.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 -- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f8c0682f0 msgr2=0x7f2f8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:42.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f8c0682f0 0x7f2f8c06a7a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0c2110 tx=0x7f2f9c008040 comp rx=0 tx=0).stop 2023-03-03T13:21:42.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 -- 172.21.15.7:0/4042525126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fac06e020 msgr2=0x7f2fac150890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:21:42.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fac06e020 0x7f2fac150890 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400c550 tx=0x7f2f9400c910 comp rx=0 tx=0).stop 2023-03-03T13:21:42.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 -- 172.21.15.7:0/4042525126 shutdown_connections 2023-03-03T13:21:42.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fac155390 0x7f2fac147f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:42.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f8c0682f0 0x7f2f8c06a7a0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:42.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fac06e020 0x7f2fac150890 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:42.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 --2- 172.21.15.7:0/4042525126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fac072520 0x7f2fac14c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:21:42.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 -- 172.21.15.7:0/4042525126 >> 172.21.15.7:0/4042525126 conn(0x7f2fac06c110 msgr2=0x7f2fac06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:21:42.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 -- 172.21.15.7:0/4042525126 shutdown_connections 2023-03-03T13:21:42.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:21:42.095+0000 7f2fb1833700 1 -- 172.21.15.7:0/4042525126 wait complete. 2023-03-03T13:21:42.098 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:21:42.108 INFO:tasks.mds_thrash.fs.[a]:waiting for 100 secs before thrashing 2023-03-03T13:22:10.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:10.994 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:39.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:22:39.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.944+0000 7f9b97d5d700 1 Processor -- start 2023-03-03T13:22:39.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.944+0000 7f9b97d5d700 1 -- start start 2023-03-03T13:22:39.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b97d5d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b900aac00 0x7f9b900c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:39.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b97d5d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9b90155950 0x7f9b90155d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:39.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b97d5d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b900713c0 con 0x7f9b90071c30 2023-03-03T13:22:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b97d5d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b901562d0 con 0x7f9b900aac00 2023-03-03T13:22:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b97d5d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b90152c70 con 0x7f9b90155950 2023-03-03T13:22:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b952f8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9b90071c30 0x7f9b900beb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38336/0 (socket says 172.21.15.7:38336) 2023-03-03T13:22:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b95af9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b900aac00 0x7f9b900c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b952f8700 1 -- 172.21.15.7:0/4059138273 learned_addr learned my addr 172.21.15.7:0/4059138273 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:22:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b962fa700 1 --2- 172.21.15.7:0/4059138273 >> v2:172.21.15.143:3300/0 conn(0x7f9b90155950 0x7f9b90155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:39.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b95af9700 1 -- 172.21.15.7:0/4059138273 >> v2:172.21.15.143:3300/0 conn(0x7f9b90155950 msgr2=0x7f9b90155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:22:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b95af9700 1 --2- 172.21.15.7:0/4059138273 >> v2:172.21.15.143:3300/0 conn(0x7f9b90155950 0x7f9b90155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b95af9700 1 -- 172.21.15.7:0/4059138273 >> v1:172.21.15.7:6789/0 conn(0x7f9b90071c30 legacy=0x7f9b900beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b94af7700 1 -- 172.21.15.7:0/4059138273 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514225231 0 0) 0x7f9b900713c0 con 0x7f9b90071c30 2023-03-03T13:22:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.945+0000 7f9b95af9700 1 -- 172.21.15.7:0/4059138273 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b900c4930 con 0x7f9b900aac00 2023-03-03T13:22:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b95af9700 1 --2- 172.21.15.7:0/4059138273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b900aac00 0x7f9b900c43f0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f9b90071720 tx=0x7f9b80009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94f6495516eb10d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:22:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b94af7700 1 -- 172.21.15.7:0/4059138273 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80011830 con 0x7f9b900aac00 2023-03-03T13:22:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b94af7700 1 -- 172.21.15.7:0/4059138273 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b80011990 con 0x7f9b900aac00 2023-03-03T13:22:39.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b94af7700 1 -- 172.21.15.7:0/4059138273 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80011d20 con 0x7f9b900aac00 2023-03-03T13:22:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b97d5d700 1 -- 172.21.15.7:0/4059138273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b900aac00 msgr2=0x7f9b900c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b97d5d700 1 --2- 172.21.15.7:0/4059138273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b900aac00 0x7f9b900c43f0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f9b90071720 tx=0x7f9b80009b20 comp rx=0 tx=0).stop 2023-03-03T13:22:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b97d5d700 1 -- 172.21.15.7:0/4059138273 shutdown_connections 2023-03-03T13:22:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b97d5d700 1 --2- 172.21.15.7:0/4059138273 >> v2:172.21.15.143:3300/0 conn(0x7f9b90155950 0x7f9b90155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b97d5d700 1 --2- 172.21.15.7:0/4059138273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b900aac00 0x7f9b900c43f0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b97d5d700 1 -- 172.21.15.7:0/4059138273 >> 172.21.15.7:0/4059138273 conn(0x7f9b900736f0 msgr2=0x7f9b90073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:22:39.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b97d5d700 1 -- 172.21.15.7:0/4059138273 shutdown_connections 2023-03-03T13:22:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.946+0000 7f9b97d5d700 1 -- 172.21.15.7:0/4059138273 wait complete. 2023-03-03T13:22:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.948+0000 7f9b97d5d700 1 Processor -- start 2023-03-03T13:22:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.948+0000 7f9b97d5d700 1 -- start start 2023-03-03T13:22:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.948+0000 7f9b97d5d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90071c30 0x7f9b900acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.948+0000 7f9b97d5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b900aac00 0x7f9b901eacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.948+0000 7f9b97d5d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b90155950 0x7f9b90147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:39.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.948+0000 7f9b97d5d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b900ad720 con 0x7f9b90071c30 2023-03-03T13:22:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.948+0000 7f9b97d5d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9014f040 con 0x7f9b90155950 2023-03-03T13:22:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.948+0000 7f9b97d5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9014ed60 con 0x7f9b900aac00 2023-03-03T13:22:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.948+0000 7f9b95af9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90071c30 0x7f9b900acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.948+0000 7f9b95af9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90071c30 0x7f9b900acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44360/0 (socket says 172.21.15.7:44360) 2023-03-03T13:22:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b95af9700 1 -- 172.21.15.7:0/1153396032 learned_addr learned my addr 172.21.15.7:0/1153396032 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:22:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b962fa700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b90155950 0x7f9b90147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b952f8700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b900aac00 0x7f9b901eacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:39.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b95af9700 1 -- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b900aac00 msgr2=0x7f9b901eacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b95af9700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b900aac00 0x7f9b901eacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b95af9700 1 -- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b90155950 msgr2=0x7f9b90147ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b95af9700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b90155950 0x7f9b90147ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b95af9700 1 -- 172.21.15.7:0/1153396032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b800094a0 con 0x7f9b90071c30 2023-03-03T13:22:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b952f8700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b900aac00 0x7f9b901eacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:22:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b95af9700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90071c30 0x7f9b900acb00 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f9b800090b0 tx=0x7f9b80004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:22:39.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b867fc700 1 -- 172.21.15.7:0/1153396032 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b800045b0 con 0x7f9b90071c30 2023-03-03T13:22:39.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b867fc700 1 -- 172.21.15.7:0/1153396032 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8001f070 con 0x7f9b90071c30 2023-03-03T13:22:39.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b97d5d700 1 -- 172.21.15.7:0/1153396032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b900c4b80 con 0x7f9b90071c30 2023-03-03T13:22:39.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b867fc700 1 -- 172.21.15.7:0/1153396032 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b800038c0 con 0x7f9b90071c30 2023-03-03T13:22:39.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.949+0000 7f9b97d5d700 1 -- 172.21.15.7:0/1153396032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b90152360 con 0x7f9b90071c30 2023-03-03T13:22:39.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.950+0000 7f9b97d5d700 1 -- 172.21.15.7:0/1153396032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b90152620 con 0x7f9b90071c30 2023-03-03T13:22:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.951+0000 7f9b867fc700 1 -- 172.21.15.7:0/1153396032 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9b8002a020 con 0x7f9b90071c30 2023-03-03T13:22:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.951+0000 7f9b867fc700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b740682f0 0x7f9b7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:22:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.951+0000 7f9b867fc700 1 -- 172.21.15.7:0/1153396032 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9b80087620 con 0x7f9b90071c30 2023-03-03T13:22:39.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.951+0000 7f9b952f8700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b740682f0 0x7f9b7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:22:39.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.952+0000 7f9b952f8700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b740682f0 0x7f9b7406a7a0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9b90148e30 tx=0x7f9b8c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:22:39.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:39.955+0000 7f9b867fc700 1 -- 172.21.15.7:0/1153396032 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9b800539c0 con 0x7f9b90071c30 2023-03-03T13:22:40.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.192+0000 7f9b97d5d700 1 -- 172.21.15.7:0/1153396032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b901e7720 con 0x7f9b90071c30 2023-03-03T13:22:40.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.193+0000 7f9b867fc700 1 -- 172.21.15.7:0/1153396032 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9b901e7720 con 0x7f9b90071c30 2023-03-03T13:22:40.194 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:22:40.195 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:22:40.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 -- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b740682f0 msgr2=0x7f9b7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b740682f0 0x7f9b7406a7a0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9b90148e30 tx=0x7f9b8c009040 comp rx=0 tx=0).stop 2023-03-03T13:22:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 -- 172.21.15.7:0/1153396032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90071c30 msgr2=0x7f9b900acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:22:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90071c30 0x7f9b900acb00 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f9b800090b0 tx=0x7f9b80004750 comp rx=0 tx=0).stop 2023-03-03T13:22:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 -- 172.21.15.7:0/1153396032 shutdown_connections 2023-03-03T13:22:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b90155950 0x7f9b90147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b740682f0 0x7f9b7406a7a0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:40.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90071c30 0x7f9b900acb00 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 --2- 172.21.15.7:0/1153396032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b900aac00 0x7f9b901eacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:22:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 -- 172.21.15.7:0/1153396032 >> 172.21.15.7:0/1153396032 conn(0x7f9b900736f0 msgr2=0x7f9b90073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:22:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 -- 172.21.15.7:0/1153396032 shutdown_connections 2023-03-03T13:22:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:22:40.195+0000 7f9b97d5d700 1 -- 172.21.15.7:0/1153396032 wait complete. 2023-03-03T13:22:40.198 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:22:40.208 INFO:tasks.mds_thrash.fs.[b]:waiting for 63 secs before thrashing 2023-03-03T13:22:41.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:22:41.115 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:11.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:11.186 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:22.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:23:22.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.245+0000 7fa03cea7700 1 Processor -- start 2023-03-03T13:23:22.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.245+0000 7fa03cea7700 1 -- start start 2023-03-03T13:23:22.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03cea7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa03814f990 0x7fa038152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:22.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03cea7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0381562b0 0x7fa03806b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:22.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03cea7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03814fd60 con 0x7fa03806be60 2023-03-03T13:23:22.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03cea7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0380c29c0 con 0x7fa03814f990 2023-03-03T13:23:22.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03cea7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0380c2400 con 0x7fa0381562b0 2023-03-03T13:23:22.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa036d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa03806be60 0x7fa0380ab5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38404/0 (socket says 172.21.15.7:38404) 2023-03-03T13:23:22.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa03814f990 0x7fa038152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:22.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa036d9d700 1 -- 172.21.15.7:0/1019813648 learned_addr learned my addr 172.21.15.7:0/1019813648 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:23:22.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa035d9b700 1 --2- 172.21.15.7:0/1019813648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0381562b0 0x7fa03806b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:22.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03659c700 1 -- 172.21.15.7:0/1019813648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0381562b0 msgr2=0x7fa03806b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:23:22.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03659c700 1 --2- 172.21.15.7:0/1019813648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0381562b0 0x7fa03806b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:22.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03659c700 1 -- 172.21.15.7:0/1019813648 >> v1:172.21.15.7:6789/0 conn(0x7fa03806be60 legacy=0x7fa0380ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:22.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03559a700 1 -- 172.21.15.7:0/1019813648 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713919556 0 0) 0x7fa03814fd60 con 0x7fa03806be60 2023-03-03T13:23:22.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.246+0000 7fa03659c700 1 -- 172.21.15.7:0/1019813648 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0380c26e0 con 0x7fa03814f990 2023-03-03T13:23:22.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03659c700 1 --2- 172.21.15.7:0/1019813648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa03814f990 0x7fa038152ed0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009840 tx=0x7fa02c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd6ae47873fb49f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:22.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03559a700 1 -- 172.21.15.7:0/1019813648 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c011830 con 0x7fa03814f990 2023-03-03T13:23:22.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03559a700 1 -- 172.21.15.7:0/1019813648 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02c011990 con 0x7fa03814f990 2023-03-03T13:23:22.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03559a700 1 -- 172.21.15.7:0/1019813648 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c011d20 con 0x7fa03814f990 2023-03-03T13:23:22.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03cea7700 1 -- 172.21.15.7:0/1019813648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa03814f990 msgr2=0x7fa038152ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:22.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03cea7700 1 --2- 172.21.15.7:0/1019813648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa03814f990 0x7fa038152ed0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009840 tx=0x7fa02c009b50 comp rx=0 tx=0).stop 2023-03-03T13:23:22.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03cea7700 1 -- 172.21.15.7:0/1019813648 shutdown_connections 2023-03-03T13:23:22.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03cea7700 1 --2- 172.21.15.7:0/1019813648 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa03814f990 0x7fa038152ed0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:22.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03cea7700 1 --2- 172.21.15.7:0/1019813648 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0381562b0 0x7fa03806b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:22.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03cea7700 1 -- 172.21.15.7:0/1019813648 >> 172.21.15.7:0/1019813648 conn(0x7fa0380c62e0 msgr2=0x7fa0380c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:23:22.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03cea7700 1 -- 172.21.15.7:0/1019813648 shutdown_connections 2023-03-03T13:23:22.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.247+0000 7fa03cea7700 1 -- 172.21.15.7:0/1019813648 wait complete. 2023-03-03T13:23:22.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.249+0000 7fa03cea7700 1 Processor -- start 2023-03-03T13:23:22.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.249+0000 7fa03cea7700 1 -- start start 2023-03-03T13:23:22.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.249+0000 7fa03cea7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa03806be60 0x7fa0380c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.249+0000 7fa03cea7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa03814f990 0x7fa038151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.249+0000 7fa03cea7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0381562b0 0x7fa038144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.249+0000 7fa03cea7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381e7fb0 con 0x7fa0381562b0 2023-03-03T13:23:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.249+0000 7fa03cea7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0380c5930 con 0x7fa03814f990 2023-03-03T13:23:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.249+0000 7fa03cea7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381e78f0 con 0x7fa03806be60 2023-03-03T13:23:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa036d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0381562b0 0x7fa038144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:22.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa036d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0381562b0 0x7fa038144080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44428/0 (socket says 172.21.15.7:44428) 2023-03-03T13:23:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa036d9d700 1 -- 172.21.15.7:0/1116431612 learned_addr learned my addr 172.21.15.7:0/1116431612 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:23:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa035d9b700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa03814f990 0x7fa038151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa03659c700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa03806be60 0x7fa0380c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa036d9d700 1 -- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa03806be60 msgr2=0x7fa0380c52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa036d9d700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa03806be60 0x7fa0380c52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa036d9d700 1 -- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa03814f990 msgr2=0x7fa038151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:22.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa036d9d700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa03814f990 0x7fa038151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa036d9d700 1 -- 172.21.15.7:0/1116431612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c0094a0 con 0x7fa0381562b0 2023-03-03T13:23:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa036d9d700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0381562b0 0x7fa038144080 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fa03000cc10 tx=0x7fa0300021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa026ffd700 1 -- 172.21.15.7:0/1116431612 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03000ab80 con 0x7fa0381562b0 2023-03-03T13:23:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa026ffd700 1 -- 172.21.15.7:0/1116431612 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03000ad00 con 0x7fa0381562b0 2023-03-03T13:23:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa03cea7700 1 -- 172.21.15.7:0/1116431612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0381526d0 con 0x7fa0381562b0 2023-03-03T13:23:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa026ffd700 1 -- 172.21.15.7:0/1116431612 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030007310 con 0x7fa0381562b0 2023-03-03T13:23:22.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.250+0000 7fa03cea7700 1 -- 172.21.15.7:0/1116431612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0381523f0 con 0x7fa0381562b0 2023-03-03T13:23:22.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.251+0000 7fa03cea7700 1 -- 172.21.15.7:0/1116431612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0381529b0 con 0x7fa0381562b0 2023-03-03T13:23:22.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.252+0000 7fa026ffd700 1 -- 172.21.15.7:0/1116431612 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa03001a030 con 0x7fa0381562b0 2023-03-03T13:23:22.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.252+0000 7fa026ffd700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0180682f0 0x7fa01806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:22.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.252+0000 7fa026ffd700 1 -- 172.21.15.7:0/1116431612 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa030085e80 con 0x7fa0381562b0 2023-03-03T13:23:22.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.252+0000 7fa03659c700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0180682f0 0x7fa01806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:22.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.253+0000 7fa03659c700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0180682f0 0x7fa01806a7a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0090b0 tx=0x7fa02c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:22.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.257+0000 7fa026ffd700 1 -- 172.21.15.7:0/1116431612 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa030052220 con 0x7fa0381562b0 2023-03-03T13:23:22.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.498+0000 7fa03cea7700 1 -- 172.21.15.7:0/1116431612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0381e80f0 con 0x7fa0381562b0 2023-03-03T13:23:22.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.498+0000 7fa026ffd700 1 -- 172.21.15.7:0/1116431612 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa0381e80f0 con 0x7fa0381562b0 2023-03-03T13:23:22.502 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:23:22.502 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:23:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 -- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0180682f0 msgr2=0x7fa01806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0180682f0 0x7fa01806a7a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0090b0 tx=0x7fa02c01c040 comp rx=0 tx=0).stop 2023-03-03T13:23:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 -- 172.21.15.7:0/1116431612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0381562b0 msgr2=0x7fa038144080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0381562b0 0x7fa038144080 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fa03000cc10 tx=0x7fa0300021c0 comp rx=0 tx=0).stop 2023-03-03T13:23:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 -- 172.21.15.7:0/1116431612 shutdown_connections 2023-03-03T13:23:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0180682f0 0x7fa01806a7a0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:22.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0381562b0 0x7fa038144080 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa03806be60 0x7fa0380c52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 --2- 172.21.15.7:0/1116431612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa03814f990 0x7fa038151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 -- 172.21.15.7:0/1116431612 >> 172.21.15.7:0/1116431612 conn(0x7fa0380c62e0 msgr2=0x7fa03806c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:23:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 -- 172.21.15.7:0/1116431612 shutdown_connections 2023-03-03T13:23:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:22.501+0000 7fa03cea7700 1 -- 172.21.15.7:0/1116431612 wait complete. 2023-03-03T13:23:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:23:22.515 INFO:tasks.mds_thrash.fs.[a]:waiting for 28 secs before thrashing 2023-03-03T13:23:41.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:41.275 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:23:43.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:23:43.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.346+0000 7fb00edf4700 1 Processor -- start 2023-03-03T13:23:43.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.346+0000 7fb00edf4700 1 -- start start 2023-03-03T13:23:43.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.347+0000 7fb00edf4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00806c480 0x7fb0081581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.347+0000 7fb00edf4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00806cb70 con 0x7fb008151b20 2023-03-03T13:23:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.347+0000 7fb00edf4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00806ce50 con 0x7fb0080c2950 2023-03-03T13:23:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.347+0000 7fb00edf4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00806c2f0 con 0x7fb00806c480 2023-03-03T13:23:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.347+0000 7fb00cb90700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb008151b20 0x7fb0080736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38418/0 (socket says 172.21.15.7:38418) 2023-03-03T13:23:43.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.347+0000 7fb00cb90700 1 -- 172.21.15.7:0/2816368785 learned_addr learned my addr 172.21.15.7:0/2816368785 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:23:43.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.347+0000 7fb00d391700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00806c480 0x7fb0081581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:43.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.347+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085367800 0 0) 0x7fb00806cb70 con 0x7fb008151b20 2023-03-03T13:23:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.347+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff80035b0 con 0x7fb008151b20 2023-03-03T13:23:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2930310737 0 0) 0x7faff80035b0 con 0x7fb008151b20 2023-03-03T13:23:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00806c480 msgr2=0x7fb0081581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:23:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00806c480 0x7fb0081581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 >> v1:172.21.15.143:6789/0 conn(0x7fb0080c2950 legacy=0x7fb008157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00806d130 con 0x7fb008151b20 2023-03-03T13:23:43.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7faffc001670 con 0x7fb008151b20 2023-03-03T13:23:43.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 >> v1:172.21.15.7:6789/0 conn(0x7fb008151b20 legacy=0x7fb0080736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:43.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00806c480 0x7faff8003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:43.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff8003fe0 0x7faff8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:43.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff8004d90 0x7faff8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:43.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00806c2f0 con 0x7faff8003fe0 2023-03-03T13:23:43.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff800bba0 con 0x7faff8004d90 2023-03-03T13:23:43.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff800bd30 con 0x7fb00806c480 2023-03-03T13:23:43.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb007fff700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff8003fe0 0x7faff8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:43.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb00cb90700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00806c480 0x7faff8003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:43.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb00d391700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff8004d90 0x7faff8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:43.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb007fff700 1 -- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00806c480 msgr2=0x7faff8003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:23:43.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb007fff700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00806c480 0x7faff8003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:43.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb007fff700 1 -- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff8004d90 msgr2=0x7faff8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:23:43.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb007fff700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faff8004d90 0x7faff8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:43.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb007fff700 1 -- 172.21.15.7:0/2816368785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff80035b0 con 0x7faff8003fe0 2023-03-03T13:23:43.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb007fff700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff8003fe0 0x7faff8004850 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7faff40018b0 tx=0x7faff8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc2a308bfb33754 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:43.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.348+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4001370 con 0x7faff8003fe0 2023-03-03T13:23:43.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.349+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff4012640 con 0x7faff8003fe0 2023-03-03T13:23:43.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.349+0000 7fb0077fe700 1 -- 172.21.15.7:0/2816368785 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40129f0 con 0x7faff8003fe0 2023-03-03T13:23:43.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.349+0000 7fb00edf4700 1 -- 172.21.15.7:0/2816368785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff8003fe0 msgr2=0x7faff8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:43.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.349+0000 7fb00edf4700 1 --2- 172.21.15.7:0/2816368785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faff8003fe0 0x7faff8004850 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7faff40018b0 tx=0x7faff8005880 comp rx=0 tx=0).stop 2023-03-03T13:23:43.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.349+0000 7fb00d391700 1 -- 172.21.15.7:0/2816368785 reap_dead start 2023-03-03T13:23:43.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.349+0000 7fb00edf4700 1 -- 172.21.15.7:0/2816368785 shutdown_connections 2023-03-03T13:23:43.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.349+0000 7fb00edf4700 1 -- 172.21.15.7:0/2816368785 >> 172.21.15.7:0/2816368785 conn(0x7fb0080a94d0 msgr2=0x7fb0080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:23:43.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.349+0000 7fb00edf4700 1 -- 172.21.15.7:0/2816368785 shutdown_connections 2023-03-03T13:23:43.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.349+0000 7fb00edf4700 1 -- 172.21.15.7:0/2816368785 wait complete. 2023-03-03T13:23:43.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.351+0000 7fb00edf4700 1 Processor -- start 2023-03-03T13:23:43.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.351+0000 7fb00edf4700 1 -- start start 2023-03-03T13:23:43.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb00edf4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff8003fe0 0x7fb0080aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:43.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb00edf4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080c6620 0x7fb0081ed420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:43.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb00edf4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0081ed960 0x7fb008156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:43.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb00edf4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0081ecf90 con 0x7fb0080c6620 2023-03-03T13:23:43.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb00edf4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0081eddd0 con 0x7fb0081ed960 2023-03-03T13:23:43.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb00edf4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0080bf450 con 0x7faff8003fe0 2023-03-03T13:23:43.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb007fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080c6620 0x7fb0081ed420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:43.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb007fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080c6620 0x7fb0081ed420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44452/0 (socket says 172.21.15.7:44452) 2023-03-03T13:23:43.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb007fff700 1 -- 172.21.15.7:0/1404252532 learned_addr learned my addr 172.21.15.7:0/1404252532 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:23:43.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb00cb90700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff8003fe0 0x7fb0080aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb00d391700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0081ed960 0x7fb008156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb007fff700 1 -- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff8003fe0 msgr2=0x7fb0080aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb007fff700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff8003fe0 0x7fb0080aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb007fff700 1 -- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0081ed960 msgr2=0x7fb008156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb007fff700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0081ed960 0x7fb008156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.352+0000 7fb007fff700 1 -- 172.21.15.7:0/1404252532 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb008151ef0 con 0x7fb0080c6620 2023-03-03T13:23:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.353+0000 7fb007fff700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080c6620 0x7fb0081ed420 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7faff4000c00 tx=0x7faff4001110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.353+0000 7fb0057fa700 1 -- 172.21.15.7:0/1404252532 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4001370 con 0x7fb0080c6620 2023-03-03T13:23:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.353+0000 7fb0057fa700 1 -- 172.21.15.7:0/1404252532 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff4021070 con 0x7fb0080c6620 2023-03-03T13:23:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.353+0000 7fb00edf4700 1 -- 172.21.15.7:0/1404252532 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00806d410 con 0x7fb0080c6620 2023-03-03T13:23:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.353+0000 7fb0057fa700 1 -- 172.21.15.7:0/1404252532 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40129b0 con 0x7fb0080c6620 2023-03-03T13:23:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.353+0000 7fb00edf4700 1 -- 172.21.15.7:0/1404252532 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb008073e00 con 0x7fb0080c6620 2023-03-03T13:23:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.354+0000 7fb00edf4700 1 -- 172.21.15.7:0/1404252532 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb008074bb0 con 0x7fb0080c6620 2023-03-03T13:23:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.354+0000 7fb0057fa700 1 -- 172.21.15.7:0/1404252532 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faff4010050 con 0x7fb0080c6620 2023-03-03T13:23:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.355+0000 7fb0057fa700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafec0682f0 0x7fafec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.355+0000 7fb0057fa700 1 -- 172.21.15.7:0/1404252532 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faff4087310 con 0x7fb0080c6620 2023-03-03T13:23:43.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.355+0000 7fb00cb90700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafec0682f0 0x7fafec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:43.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.355+0000 7fb00cb90700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafec0682f0 0x7fafec06a7a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7faffc001eb0 tx=0x7faffc000f80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:43.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.359+0000 7fb0057fa700 1 -- 172.21.15.7:0/1404252532 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faff4053650 con 0x7fb0080c6620 2023-03-03T13:23:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.596+0000 7fb00edf4700 1 -- 172.21.15.7:0/1404252532 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0080ab0a0 con 0x7fb0080c6620 2023-03-03T13:23:43.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.597+0000 7fb0057fa700 1 -- 172.21.15.7:0/1404252532 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faff402b090 con 0x7fb0080c6620 2023-03-03T13:23:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:23:43.599 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:23:43.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 -- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafec0682f0 msgr2=0x7fafec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:43.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafec0682f0 0x7fafec06a7a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7faffc001eb0 tx=0x7faffc000f80 comp rx=0 tx=0).stop 2023-03-03T13:23:43.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 -- 172.21.15.7:0/1404252532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080c6620 msgr2=0x7fb0081ed420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:43.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080c6620 0x7fb0081ed420 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7faff4000c00 tx=0x7faff4001110 comp rx=0 tx=0).stop 2023-03-03T13:23:43.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 -- 172.21.15.7:0/1404252532 shutdown_connections 2023-03-03T13:23:43.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0081ed960 0x7fb008156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:43.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faff8003fe0 0x7fb0080aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:43.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafec0682f0 0x7fafec06a7a0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:43.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 --2- 172.21.15.7:0/1404252532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0080c6620 0x7fb0081ed420 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:43.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 -- 172.21.15.7:0/1404252532 >> 172.21.15.7:0/1404252532 conn(0x7fb0080a94d0 msgr2=0x7fb0080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:23:43.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 -- 172.21.15.7:0/1404252532 shutdown_connections 2023-03-03T13:23:43.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:43.599+0000 7fb00edf4700 1 -- 172.21.15.7:0/1404252532 wait complete. 2023-03-03T13:23:43.602 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:23:43.612 INFO:tasks.mds_thrash.fs.[b]:waiting for 14 secs before thrashing 2023-03-03T13:23:50.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:23:50.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.654+0000 7fc09fcf1700 1 Processor -- start 2023-03-03T13:23:50.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.655+0000 7fc09fcf1700 1 -- start start 2023-03-03T13:23:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.655+0000 7fc09fcf1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc09806d240 0x7fc0980c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.655+0000 7fc09fcf1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc0980c3e30 0x7fc0980c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.655+0000 7fc09fcf1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0980ad2b0 con 0x7fc09814fc50 2023-03-03T13:23:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.655+0000 7fc09fcf1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0980c30c0 con 0x7fc0980c3e30 2023-03-03T13:23:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.655+0000 7fc09fcf1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc098156420 con 0x7fc09806d240 2023-03-03T13:23:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09d28c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc09806d240 0x7fc0980c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09d28c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc09806d240 0x7fc0980c27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:52540/0 (socket says 172.21.15.7:52540) 2023-03-03T13:23:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09e28e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc0980c3e30 0x7fc0980c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09d28c700 1 -- 172.21.15.7:0/2694529885 learned_addr learned my addr 172.21.15.7:0/2694529885 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:23:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09da8d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fc09814fc50 0x7fc0980c1f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:41968/0 (socket says 172.21.15.7:41968) 2023-03-03T13:23:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09d28c700 1 -- 172.21.15.7:0/2694529885 >> v2:172.21.15.143:3300/0 conn(0x7fc0980c3e30 msgr2=0x7fc0980c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:23:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09d28c700 1 --2- 172.21.15.7:0/2694529885 >> v2:172.21.15.143:3300/0 conn(0x7fc0980c3e30 0x7fc0980c42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09d28c700 1 -- 172.21.15.7:0/2694529885 >> v1:172.21.15.143:6789/0 conn(0x7fc09814fc50 legacy=0x7fc0980c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09d28c700 1 -- 172.21.15.7:0/2694529885 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0980ad520 con 0x7fc09806d240 2023-03-03T13:23:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09e28e700 1 --2- 172.21.15.7:0/2694529885 >> v2:172.21.15.143:3300/0 conn(0x7fc0980c3e30 0x7fc0980c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:23:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.656+0000 7fc09d28c700 1 --2- 172.21.15.7:0/2694529885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc09806d240 0x7fc0980c27b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0098f0 tx=0x7fc08c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab22f43e5b19524d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09ca8b700 1 -- 172.21.15.7:0/2694529885 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c011830 con 0x7fc09806d240 2023-03-03T13:23:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09ca8b700 1 -- 172.21.15.7:0/2694529885 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08c011990 con 0x7fc09806d240 2023-03-03T13:23:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09ca8b700 1 -- 172.21.15.7:0/2694529885 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c011d20 con 0x7fc09806d240 2023-03-03T13:23:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09fcf1700 1 -- 172.21.15.7:0/2694529885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc09806d240 msgr2=0x7fc0980c27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09fcf1700 1 --2- 172.21.15.7:0/2694529885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc09806d240 0x7fc0980c27b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0098f0 tx=0x7fc08c009c00 comp rx=0 tx=0).stop 2023-03-03T13:23:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09fcf1700 1 -- 172.21.15.7:0/2694529885 shutdown_connections 2023-03-03T13:23:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09fcf1700 1 --2- 172.21.15.7:0/2694529885 >> v2:172.21.15.143:3300/0 conn(0x7fc0980c3e30 0x7fc0980c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09fcf1700 1 --2- 172.21.15.7:0/2694529885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc09806d240 0x7fc0980c27b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09fcf1700 1 -- 172.21.15.7:0/2694529885 >> 172.21.15.7:0/2694529885 conn(0x7fc09806c110 msgr2=0x7fc09806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:23:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09fcf1700 1 -- 172.21.15.7:0/2694529885 shutdown_connections 2023-03-03T13:23:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.657+0000 7fc09fcf1700 1 -- 172.21.15.7:0/2694529885 wait complete. 2023-03-03T13:23:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.658+0000 7fc09fcf1700 1 Processor -- start 2023-03-03T13:23:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09fcf1700 1 -- start start 2023-03-03T13:23:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09fcf1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc09806d240 0x7fc0980714a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09fcf1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0980c3e30 0x7fc09814bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09fcf1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc09814fc50 0x7fc0981548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09fcf1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09814b060 con 0x7fc09806d240 2023-03-03T13:23:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09fcf1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09814aaa0 con 0x7fc09814fc50 2023-03-03T13:23:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09fcf1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc09814ad80 con 0x7fc0980c3e30 2023-03-03T13:23:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09da8d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc09806d240 0x7fc0980714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09da8d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc09806d240 0x7fc0980714a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44474/0 (socket says 172.21.15.7:44474) 2023-03-03T13:23:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09da8d700 1 -- 172.21.15.7:0/768108336 learned_addr learned my addr 172.21.15.7:0/768108336 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:23:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09e28e700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc09814fc50 0x7fc0981548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.659+0000 7fc09d28c700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0980c3e30 0x7fc09814bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc09da8d700 1 -- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0980c3e30 msgr2=0x7fc09814bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc09da8d700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0980c3e30 0x7fc09814bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc09da8d700 1 -- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc09814fc50 msgr2=0x7fc0981548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc09da8d700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc09814fc50 0x7fc0981548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc09da8d700 1 -- 172.21.15.7:0/768108336 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc090009a70 con 0x7fc09806d240 2023-03-03T13:23:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc09da8d700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc09806d240 0x7fc0980714a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fc09400c490 tx=0x7fc09400c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc08a7fc700 1 -- 172.21.15.7:0/768108336 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094005cb0 con 0x7fc09806d240 2023-03-03T13:23:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc08a7fc700 1 -- 172.21.15.7:0/768108336 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09400ce90 con 0x7fc09806d240 2023-03-03T13:23:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc08a7fc700 1 -- 172.21.15.7:0/768108336 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0940072d0 con 0x7fc09806d240 2023-03-03T13:23:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc09fcf1700 1 -- 172.21.15.7:0/768108336 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c009550 con 0x7fc09806d240 2023-03-03T13:23:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.660+0000 7fc09fcf1700 1 -- 172.21.15.7:0/768108336 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc098074220 con 0x7fc09806d240 2023-03-03T13:23:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.661+0000 7fc09fcf1700 1 -- 172.21.15.7:0/768108336 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0980c2cf0 con 0x7fc09806d240 2023-03-03T13:23:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.662+0000 7fc08a7fc700 1 -- 172.21.15.7:0/768108336 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc094019030 con 0x7fc09806d240 2023-03-03T13:23:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.662+0000 7fc08a7fc700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc07c0682f0 0x7fc07c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.662+0000 7fc08a7fc700 1 -- 172.21.15.7:0/768108336 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc094085240 con 0x7fc09806d240 2023-03-03T13:23:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.662+0000 7fc09d28c700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc07c0682f0 0x7fc07c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.663+0000 7fc09d28c700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc07c0682f0 0x7fc07c06a7a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fc08c014040 tx=0x7fc08c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.666+0000 7fc08a7fc700 1 -- 172.21.15.7:0/768108336 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc09404b5e0 con 0x7fc09806d240 2023-03-03T13:23:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.903+0000 7fc09fcf1700 1 -- 172.21.15.7:0/768108336 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc098156690 con 0x7fc09806d240 2023-03-03T13:23:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.904+0000 7fc08a7fc700 1 -- 172.21.15.7:0/768108336 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc094017090 con 0x7fc09806d240 2023-03-03T13:23:50.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:23:50.905 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:23:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 -- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc07c0682f0 msgr2=0x7fc07c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc07c0682f0 0x7fc07c06a7a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fc08c014040 tx=0x7fc08c008030 comp rx=0 tx=0).stop 2023-03-03T13:23:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 -- 172.21.15.7:0/768108336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc09806d240 msgr2=0x7fc0980714a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc09806d240 0x7fc0980714a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fc09400c490 tx=0x7fc09400c7a0 comp rx=0 tx=0).stop 2023-03-03T13:23:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 -- 172.21.15.7:0/768108336 shutdown_connections 2023-03-03T13:23:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc09814fc50 0x7fc0981548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc07c0682f0 0x7fc07c06a7a0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc09806d240 0x7fc0980714a0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:50.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 --2- 172.21.15.7:0/768108336 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0980c3e30 0x7fc09814bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:50.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 -- 172.21.15.7:0/768108336 >> 172.21.15.7:0/768108336 conn(0x7fc09806c110 msgr2=0x7fc0980ac4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:23:50.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 -- 172.21.15.7:0/768108336 shutdown_connections 2023-03-03T13:23:50.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:50.906+0000 7fc09fcf1700 1 -- 172.21.15.7:0/768108336 wait complete. 2023-03-03T13:23:50.909 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:23:50.919 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2023-03-03T13:23:57.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:23:57.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.750+0000 7f2f4184f700 1 Processor -- start 2023-03-03T13:23:57.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.751+0000 7f2f4184f700 1 -- start start 2023-03-03T13:23:57.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.751+0000 7f2f4184f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2f3c0c00e0 0x7f2f3c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:57.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.751+0000 7f2f4184f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c154ff0 0x7f2f3c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:57.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.751+0000 7f2f4184f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3c0c7310 con 0x7f2f3c0caa30 2023-03-03T13:23:57.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.751+0000 7f2f4184f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3c0bfd10 con 0x7f2f3c0c00e0 2023-03-03T13:23:57.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.751+0000 7f2f4184f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0bfa30 con 0x7f2f3c154ff0 2023-03-03T13:23:57.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.752+0000 7f2f3b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c154ff0 0x7f2f3c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:57.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.752+0000 7f2f3b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c154ff0 0x7f2f3c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:52560/0 (socket says 172.21.15.7:52560) 2023-03-03T13:23:57.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.752+0000 7f2f3b7fe700 1 -- 172.21.15.7:0/2459108157 learned_addr learned my addr 172.21.15.7:0/2459108157 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:23:57.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.752+0000 7f2f3a7fc700 1 --2- 172.21.15.7:0/2459108157 >> v2:172.21.15.143:3300/0 conn(0x7f2f3c0c00e0 0x7f2f3c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:57.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.752+0000 7f2f39ffb700 1 -- 172.21.15.7:0/2459108157 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107863019 0 0) 0x7f2f3c0c7310 con 0x7f2f3c0caa30 2023-03-03T13:23:57.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.752+0000 7f2f3b7fe700 1 -- 172.21.15.7:0/2459108157 >> v2:172.21.15.143:3300/0 conn(0x7f2f3c0c00e0 msgr2=0x7f2f3c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:23:57.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.752+0000 7f2f3b7fe700 1 --2- 172.21.15.7:0/2459108157 >> v2:172.21.15.143:3300/0 conn(0x7f2f3c0c00e0 0x7f2f3c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:57.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.752+0000 7f2f3b7fe700 1 -- 172.21.15.7:0/2459108157 >> v1:172.21.15.143:6789/0 conn(0x7f2f3c0caa30 legacy=0x7f2f3c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:57.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.752+0000 7f2f3b7fe700 1 -- 172.21.15.7:0/2459108157 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c154510 con 0x7f2f3c154ff0 2023-03-03T13:23:57.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.752+0000 7f2f3b7fe700 1 --2- 172.21.15.7:0/2459108157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c154ff0 0x7f2f3c0cae00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009e50 tx=0x7f2f3400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=946223669c7e12e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:57.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f39ffb700 1 -- 172.21.15.7:0/2459108157 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3400e070 con 0x7f2f3c154ff0 2023-03-03T13:23:57.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f39ffb700 1 -- 172.21.15.7:0/2459108157 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3400b910 con 0x7f2f3c154ff0 2023-03-03T13:23:57.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f39ffb700 1 -- 172.21.15.7:0/2459108157 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3400bcc0 con 0x7f2f3c154ff0 2023-03-03T13:23:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f4184f700 1 -- 172.21.15.7:0/2459108157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c154ff0 msgr2=0x7f2f3c0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f4184f700 1 --2- 172.21.15.7:0/2459108157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c154ff0 0x7f2f3c0cae00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009e50 tx=0x7f2f3400b330 comp rx=0 tx=0).stop 2023-03-03T13:23:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f4184f700 1 -- 172.21.15.7:0/2459108157 shutdown_connections 2023-03-03T13:23:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f4184f700 1 --2- 172.21.15.7:0/2459108157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c154ff0 0x7f2f3c0cae00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f4184f700 1 --2- 172.21.15.7:0/2459108157 >> v2:172.21.15.143:3300/0 conn(0x7f2f3c0c00e0 0x7f2f3c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f4184f700 1 -- 172.21.15.7:0/2459108157 >> 172.21.15.7:0/2459108157 conn(0x7f2f3c066e40 msgr2=0x7f2f3c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:23:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f4184f700 1 -- 172.21.15.7:0/2459108157 shutdown_connections 2023-03-03T13:23:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.753+0000 7f2f4184f700 1 -- 172.21.15.7:0/2459108157 wait complete. 2023-03-03T13:23:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.754+0000 7f2f4184f700 1 Processor -- start 2023-03-03T13:23:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.755+0000 7f2f4184f700 1 -- start start 2023-03-03T13:23:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.755+0000 7f2f4184f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f3c0c00e0 0x7f2f3c1e97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.755+0000 7f2f4184f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c0caa30 0x7f2f3c1e9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.755+0000 7f2f4184f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f3c154ff0 0x7f2f3c157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.755+0000 7f2f4184f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c0c2b90 con 0x7f2f3c154ff0 2023-03-03T13:23:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.755+0000 7f2f4184f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c1de550 con 0x7f2f3c0c00e0 2023-03-03T13:23:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.755+0000 7f2f4184f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c1deb80 con 0x7f2f3c0caa30 2023-03-03T13:23:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.755+0000 7f2f3b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f3c154ff0 0x7f2f3c157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.755+0000 7f2f3b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f3c154ff0 0x7f2f3c157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44496/0 (socket says 172.21.15.7:44496) 2023-03-03T13:23:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.755+0000 7f2f3b7fe700 1 -- 172.21.15.7:0/2180991676 learned_addr learned my addr 172.21.15.7:0/2180991676 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:23:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f3affd700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f3c0c00e0 0x7f2f3c1e97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f3a7fc700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c0caa30 0x7f2f3c1e9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:57.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f3b7fe700 1 -- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c0caa30 msgr2=0x7f2f3c1e9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:57.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f3b7fe700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c0caa30 0x7f2f3c1e9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:57.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f3b7fe700 1 -- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f3c0c00e0 msgr2=0x7f2f3c1e97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:57.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f3b7fe700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f3c0c00e0 0x7f2f3c1e97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:57.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f3b7fe700 1 -- 172.21.15.7:0/2180991676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f34009b30 con 0x7f2f3c154ff0 2023-03-03T13:23:57.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f3a7fc700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c0caa30 0x7f2f3c1e9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:23:57.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f3b7fe700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f3c154ff0 0x7f2f3c157b00 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0bfb70 tx=0x7f2f34004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f33fff700 1 -- 172.21.15.7:0/2180991676 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34003ea0 con 0x7f2f3c154ff0 2023-03-03T13:23:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f33fff700 1 -- 172.21.15.7:0/2180991676 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f340099b0 con 0x7f2f3c154ff0 2023-03-03T13:23:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f4184f700 1 -- 172.21.15.7:0/2180991676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3c154ad0 con 0x7f2f3c154ff0 2023-03-03T13:23:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f33fff700 1 -- 172.21.15.7:0/2180991676 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3401c680 con 0x7f2f3c154ff0 2023-03-03T13:23:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.756+0000 7f2f4184f700 1 -- 172.21.15.7:0/2180991676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3c1547f0 con 0x7f2f3c154ff0 2023-03-03T13:23:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.757+0000 7f2f4184f700 1 -- 172.21.15.7:0/2180991676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c0bf650 con 0x7f2f3c154ff0 2023-03-03T13:23:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.757+0000 7f2f33fff700 1 -- 172.21.15.7:0/2180991676 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f3402c070 con 0x7f2f3c154ff0 2023-03-03T13:23:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.758+0000 7f2f33fff700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f1c0682f0 0x7f2f1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:23:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.758+0000 7f2f33fff700 1 -- 172.21.15.7:0/2180991676 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2f34087d60 con 0x7f2f3c154ff0 2023-03-03T13:23:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.758+0000 7f2f3affd700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f1c0682f0 0x7f2f1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:23:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.759+0000 7f2f3affd700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f1c0682f0 0x7f2f1c06a7a0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c000f60 tx=0x7f2f2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:23:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:57.762+0000 7f2f33fff700 1 -- 172.21.15.7:0/2180991676 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f34054080 con 0x7f2f3c154ff0 2023-03-03T13:23:58.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.000+0000 7f2f4184f700 1 -- 172.21.15.7:0/2180991676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f3c074bb0 con 0x7f2f3c154ff0 2023-03-03T13:23:58.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.001+0000 7f2f33fff700 1 -- 172.21.15.7:0/2180991676 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2f340576a0 con 0x7f2f3c154ff0 2023-03-03T13:23:58.002 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:23:58.003 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:23:58.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.002+0000 7f2f4184f700 1 -- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f1c0682f0 msgr2=0x7f2f1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:58.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.002+0000 7f2f4184f700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f1c0682f0 0x7f2f1c06a7a0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c000f60 tx=0x7f2f2c009040 comp rx=0 tx=0).stop 2023-03-03T13:23:58.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.002+0000 7f2f4184f700 1 -- 172.21.15.7:0/2180991676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f3c154ff0 msgr2=0x7f2f3c157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:23:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.002+0000 7f2f4184f700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f3c154ff0 0x7f2f3c157b00 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0bfb70 tx=0x7f2f34004ad0 comp rx=0 tx=0).stop 2023-03-03T13:23:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.002+0000 7f2f4184f700 1 -- 172.21.15.7:0/2180991676 shutdown_connections 2023-03-03T13:23:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.002+0000 7f2f4184f700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f1c0682f0 0x7f2f1c06a7a0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.002+0000 7f2f4184f700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f3c154ff0 0x7f2f3c157b00 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.002+0000 7f2f4184f700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f3c0c00e0 0x7f2f3c1e97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.002+0000 7f2f4184f700 1 --2- 172.21.15.7:0/2180991676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f3c0caa30 0x7f2f3c1e9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:23:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.002+0000 7f2f4184f700 1 -- 172.21.15.7:0/2180991676 >> 172.21.15.7:0/2180991676 conn(0x7f2f3c066e40 msgr2=0x7f2f3c06b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:23:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.003+0000 7f2f4184f700 1 -- 172.21.15.7:0/2180991676 shutdown_connections 2023-03-03T13:23:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:23:58.003+0000 7f2f4184f700 1 -- 172.21.15.7:0/2180991676 wait complete. 2023-03-03T13:23:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:23:58.016 INFO:tasks.mds_thrash.fs.[b]:waiting for 98 secs before thrashing 2023-03-03T13:24:11.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:11.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:33.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:24:34.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.057+0000 7fa504612700 1 Processor -- start 2023-03-03T13:24:34.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.057+0000 7fa504612700 1 -- start start 2023-03-03T13:24:34.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa504612700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4fc151b20 0x7fa4fc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:34.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa504612700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa4fc0c2950 0x7fa4fc157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:34.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa504612700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc06cb70 con 0x7fa4fc073c30 2023-03-03T13:24:34.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa504612700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc06c2f0 con 0x7fa4fc151b20 2023-03-03T13:24:34.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa504612700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc06ce50 con 0x7fa4fc0c2950 2023-03-03T13:24:34.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5023ae700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4fc151b20 0x7fa4fc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:34.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5023ae700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4fc151b20 0x7fa4fc0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44514/0 (socket says 172.21.15.7:44514) 2023-03-03T13:24:34.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5023ae700 1 -- 172.21.15.7:0/4214743413 learned_addr learned my addr 172.21.15.7:0/4214743413 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:24:34.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5023ae700 1 -- 172.21.15.7:0/4214743413 >> v2:172.21.15.143:3300/0 conn(0x7fa4fc0c2950 msgr2=0x7fa4fc157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:24:34.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa501bad700 1 --2- 172.21.15.7:0/4214743413 >> v2:172.21.15.143:3300/0 conn(0x7fa4fc0c2950 0x7fa4fc157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:34.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5023ae700 1 --2- 172.21.15.7:0/4214743413 >> v2:172.21.15.143:3300/0 conn(0x7fa4fc0c2950 0x7fa4fc157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:34.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5023ae700 1 -- 172.21.15.7:0/4214743413 >> v1:172.21.15.143:6789/0 conn(0x7fa4fc073c30 legacy=0x7fa4fc158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:24:34.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5023ae700 1 -- 172.21.15.7:0/4214743413 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc06d130 con 0x7fa4fc151b20 2023-03-03T13:24:34.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa501bad700 1 --2- 172.21.15.7:0/4214743413 >> v2:172.21.15.143:3300/0 conn(0x7fa4fc0c2950 0x7fa4fc157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:24:34.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5023ae700 1 --2- 172.21.15.7:0/4214743413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc151b20 0x7fa4fc0736f0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc06b8d0 tx=0x7fa4f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6b21bf67417dd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:24:34.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5013ac700 1 -- 172.21.15.7:0/4214743413 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8004230 con 0x7fa4fc151b20 2023-03-03T13:24:34.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5013ac700 1 -- 172.21.15.7:0/4214743413 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f8004390 con 0x7fa4fc151b20 2023-03-03T13:24:34.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.058+0000 7fa5013ac700 1 -- 172.21.15.7:0/4214743413 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f80036a0 con 0x7fa4fc151b20 2023-03-03T13:24:34.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.059+0000 7fa504612700 1 -- 172.21.15.7:0/4214743413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc151b20 msgr2=0x7fa4fc0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:34.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.059+0000 7fa504612700 1 --2- 172.21.15.7:0/4214743413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc151b20 0x7fa4fc0736f0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc06b8d0 tx=0x7fa4f8009bd0 comp rx=0 tx=0).stop 2023-03-03T13:24:34.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.059+0000 7fa504612700 1 -- 172.21.15.7:0/4214743413 shutdown_connections 2023-03-03T13:24:34.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.059+0000 7fa504612700 1 --2- 172.21.15.7:0/4214743413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc151b20 0x7fa4fc0736f0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:34.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.059+0000 7fa504612700 1 --2- 172.21.15.7:0/4214743413 >> v2:172.21.15.143:3300/0 conn(0x7fa4fc0c2950 0x7fa4fc157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:34.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.059+0000 7fa504612700 1 -- 172.21.15.7:0/4214743413 >> 172.21.15.7:0/4214743413 conn(0x7fa4fc0a94d0 msgr2=0x7fa4fc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:24:34.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.059+0000 7fa504612700 1 -- 172.21.15.7:0/4214743413 shutdown_connections 2023-03-03T13:24:34.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.059+0000 7fa504612700 1 -- 172.21.15.7:0/4214743413 wait complete. 2023-03-03T13:24:34.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.060+0000 7fa504612700 1 Processor -- start 2023-03-03T13:24:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.060+0000 7fa504612700 1 -- start start 2023-03-03T13:24:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa504612700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc073c30 0x7fa4fc0aa360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa504612700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc0c2950 0x7fa4fc14e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa504612700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4fc151b20 0x7fa4fc0c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa504612700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc14bc90 con 0x7fa4fc073c30 2023-03-03T13:24:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa504612700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc0c6db0 con 0x7fa4fc0c2950 2023-03-03T13:24:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa504612700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc14c380 con 0x7fa4fc151b20 2023-03-03T13:24:34.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa5023ae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc073c30 0x7fa4fc0aa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa5023ae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc073c30 0x7fa4fc0aa360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44518/0 (socket says 172.21.15.7:44518) 2023-03-03T13:24:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa5023ae700 1 -- 172.21.15.7:0/2033335620 learned_addr learned my addr 172.21.15.7:0/2033335620 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:24:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa501bad700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc0c2950 0x7fa4fc14e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa502baf700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4fc151b20 0x7fa4fc0c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa5023ae700 1 -- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4fc151b20 msgr2=0x7fa4fc0c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa5023ae700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4fc151b20 0x7fa4fc0c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa5023ae700 1 -- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc0c2950 msgr2=0x7fa4fc14e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:34.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa5023ae700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc0c2950 0x7fa4fc14e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:34.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.061+0000 7fa5023ae700 1 -- 172.21.15.7:0/2033335620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f8009550 con 0x7fa4fc073c30 2023-03-03T13:24:34.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.062+0000 7fa501bad700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc0c2950 0x7fa4fc14e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:24:34.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.062+0000 7fa5023ae700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc073c30 0x7fa4fc0aa360 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc06bb30 tx=0x7fa4f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:24:34.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.062+0000 7fa4eeffd700 1 -- 172.21.15.7:0/2033335620 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8003cd0 con 0x7fa4fc073c30 2023-03-03T13:24:34.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.062+0000 7fa4eeffd700 1 -- 172.21.15.7:0/2033335620 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f801f070 con 0x7fa4fc073c30 2023-03-03T13:24:34.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.062+0000 7fa504612700 1 -- 172.21.15.7:0/2033335620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4fc158730 con 0x7fa4fc073c30 2023-03-03T13:24:34.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.062+0000 7fa4eeffd700 1 -- 172.21.15.7:0/2033335620 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8011870 con 0x7fa4fc073c30 2023-03-03T13:24:34.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.062+0000 7fa504612700 1 -- 172.21.15.7:0/2033335620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4fc06d410 con 0x7fa4fc073c30 2023-03-03T13:24:34.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.063+0000 7fa504612700 1 -- 172.21.15.7:0/2033335620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4fc151ef0 con 0x7fa4fc073c30 2023-03-03T13:24:34.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.063+0000 7fa4eeffd700 1 -- 172.21.15.7:0/2033335620 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa4f8017020 con 0x7fa4fc073c30 2023-03-03T13:24:34.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.064+0000 7fa4eeffd700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4e00682f0 0x7fa4e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:24:34.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.064+0000 7fa4eeffd700 1 -- 172.21.15.7:0/2033335620 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa4f80879c0 con 0x7fa4fc073c30 2023-03-03T13:24:34.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.064+0000 7fa501bad700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4e00682f0 0x7fa4e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:24:34.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.064+0000 7fa501bad700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4e00682f0 0x7fa4e006a7a0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc14efc0 tx=0x7fa4f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:24:34.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.068+0000 7fa4eeffd700 1 -- 172.21.15.7:0/2033335620 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa4f8053c10 con 0x7fa4fc073c30 2023-03-03T13:24:34.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.307+0000 7fa504612700 1 -- 172.21.15.7:0/2033335620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4fc074bb0 con 0x7fa4fc073c30 2023-03-03T13:24:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.308+0000 7fa4eeffd700 1 -- 172.21.15.7:0/2033335620 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa4f8057230 con 0x7fa4fc073c30 2023-03-03T13:24:34.313 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:24:34.313 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:24:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 -- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4e00682f0 msgr2=0x7fa4e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4e00682f0 0x7fa4e006a7a0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc14efc0 tx=0x7fa4f0008040 comp rx=0 tx=0).stop 2023-03-03T13:24:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 -- 172.21.15.7:0/2033335620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc073c30 msgr2=0x7fa4fc0aa360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:24:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc073c30 0x7fa4fc0aa360 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc06bb30 tx=0x7fa4f8004750 comp rx=0 tx=0).stop 2023-03-03T13:24:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 -- 172.21.15.7:0/2033335620 shutdown_connections 2023-03-03T13:24:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa4fc151b20 0x7fa4fc0c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4e00682f0 0x7fa4e006a7a0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4fc073c30 0x7fa4fc0aa360 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 --2- 172.21.15.7:0/2033335620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4fc0c2950 0x7fa4fc14e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:24:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 -- 172.21.15.7:0/2033335620 >> 172.21.15.7:0/2033335620 conn(0x7fa4fc0a94d0 msgr2=0x7fa4fc1ddaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:24:34.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 -- 172.21.15.7:0/2033335620 shutdown_connections 2023-03-03T13:24:34.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:24:34.310+0000 7fa504612700 1 -- 172.21.15.7:0/2033335620 wait complete. 2023-03-03T13:24:34.316 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:24:34.325 INFO:tasks.mds_thrash.fs.[a]:waiting for 81 secs before thrashing 2023-03-03T13:24:41.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:24:41.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:11.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:11.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:36.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:25:36.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.150+0000 7f0060635700 1 Processor -- start 2023-03-03T13:25:36.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.150+0000 7f0060635700 1 -- start start 2023-03-03T13:25:36.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f0060635700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00580717a0 0x7f00580bdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:36.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f0060635700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0058151c30 0x7f0058071e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:36.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f0060635700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00581517a0 con 0x7f005806e7f0 2023-03-03T13:25:36.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f0060635700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00580bd1a0 con 0x7f00580717a0 2023-03-03T13:25:36.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f0060635700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0058151170 con 0x7f0058151c30 2023-03-03T13:25:36.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f005e3d1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00580717a0 0x7f00580bdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:36.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f005dbd0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f005806e7f0 0x7f0058154760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38536/0 (socket says 172.21.15.7:38536) 2023-03-03T13:25:36.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f005e3d1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00580717a0 0x7f00580bdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44556/0 (socket says 172.21.15.7:44556) 2023-03-03T13:25:36.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f005e3d1700 1 -- 172.21.15.7:0/1735485871 learned_addr learned my addr 172.21.15.7:0/1735485871 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:25:36.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f005e3d1700 1 -- 172.21.15.7:0/1735485871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0058151c30 msgr2=0x7f0058071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:25:36.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f005e3d1700 1 --2- 172.21.15.7:0/1735485871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0058151c30 0x7f0058071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:36.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f005e3d1700 1 -- 172.21.15.7:0/1735485871 >> v1:172.21.15.7:6789/0 conn(0x7f005806e7f0 legacy=0x7f0058154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:36.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f005e3d1700 1 -- 172.21.15.7:0/1735485871 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00580bc2f0 con 0x7f00580717a0 2023-03-03T13:25:36.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f005d3cf700 1 -- 172.21.15.7:0/1735485871 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801541264 0 0) 0x7f00581517a0 con 0x7f005806e7f0 2023-03-03T13:25:36.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.151+0000 7f005e3d1700 1 --2- 172.21.15.7:0/1735485871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00580717a0 0x7f00580bdcd0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f00540098f0 tx=0x7f0054009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce95624028788ae4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:25:36.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f005d3cf700 1 -- 172.21.15.7:0/1735485871 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005400b8a0 con 0x7f00580717a0 2023-03-03T13:25:36.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f005d3cf700 1 -- 172.21.15.7:0/1735485871 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005400ba00 con 0x7f00580717a0 2023-03-03T13:25:36.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f005d3cf700 1 -- 172.21.15.7:0/1735485871 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005400bd90 con 0x7f00580717a0 2023-03-03T13:25:36.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f0060635700 1 -- 172.21.15.7:0/1735485871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00580717a0 msgr2=0x7f00580bdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:36.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f0060635700 1 --2- 172.21.15.7:0/1735485871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00580717a0 0x7f00580bdcd0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f00540098f0 tx=0x7f0054009c00 comp rx=0 tx=0).stop 2023-03-03T13:25:36.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f0060635700 1 -- 172.21.15.7:0/1735485871 shutdown_connections 2023-03-03T13:25:36.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f0060635700 1 --2- 172.21.15.7:0/1735485871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0058151c30 0x7f0058071e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:36.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f0060635700 1 --2- 172.21.15.7:0/1735485871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00580717a0 0x7f00580bdcd0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:36.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f0060635700 1 -- 172.21.15.7:0/1735485871 >> 172.21.15.7:0/1735485871 conn(0x7f00580a94c0 msgr2=0x7f00580a98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:25:36.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f0060635700 1 -- 172.21.15.7:0/1735485871 shutdown_connections 2023-03-03T13:25:36.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.152+0000 7f0060635700 1 -- 172.21.15.7:0/1735485871 wait complete. 2023-03-03T13:25:36.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.153+0000 7f0060635700 1 Processor -- start 2023-03-03T13:25:36.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.153+0000 7f0060635700 1 -- start start 2023-03-03T13:25:36.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f0060635700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f005806e7f0 0x7f00580c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f0060635700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00580717a0 0x7f00580c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f0060635700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058151c30 0x7f00580c0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f0060635700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00580c7210 con 0x7f0058151c30 2023-03-03T13:25:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f0060635700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00580c79e0 con 0x7f00580717a0 2023-03-03T13:25:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f0060635700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00580c5d60 con 0x7f005806e7f0 2023-03-03T13:25:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f005ebd2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058151c30 0x7f00580c0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f005ebd2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058151c30 0x7f00580c0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44564/0 (socket says 172.21.15.7:44564) 2023-03-03T13:25:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f005ebd2700 1 -- 172.21.15.7:0/3028433699 learned_addr learned my addr 172.21.15.7:0/3028433699 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:25:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f005dbd0700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00580717a0 0x7f00580c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.154+0000 7f005e3d1700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f005806e7f0 0x7f00580c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f005ebd2700 1 -- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f005806e7f0 msgr2=0x7f00580c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f005ebd2700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f005806e7f0 0x7f00580c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f005ebd2700 1 -- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00580717a0 msgr2=0x7f00580c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f005ebd2700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00580717a0 0x7f00580c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f005ebd2700 1 -- 172.21.15.7:0/3028433699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0054009550 con 0x7f0058151c30 2023-03-03T13:25:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f005ebd2700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058151c30 0x7f00580c0a00 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f005000b9a0 tx=0x7f005000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:25:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f004effd700 1 -- 172.21.15.7:0/3028433699 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005000d850 con 0x7f0058151c30 2023-03-03T13:25:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f004effd700 1 -- 172.21.15.7:0/3028433699 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005000d9d0 con 0x7f0058151c30 2023-03-03T13:25:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f0060635700 1 -- 172.21.15.7:0/3028433699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0058151430 con 0x7f0058151c30 2023-03-03T13:25:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f004effd700 1 -- 172.21.15.7:0/3028433699 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005000dd80 con 0x7f0058151c30 2023-03-03T13:25:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.155+0000 7f0060635700 1 -- 172.21.15.7:0/3028433699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00580bc5d0 con 0x7f0058151c30 2023-03-03T13:25:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.156+0000 7f0060635700 1 -- 172.21.15.7:0/3028433699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00581520b0 con 0x7f0058151c30 2023-03-03T13:25:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.156+0000 7f004effd700 1 -- 172.21.15.7:0/3028433699 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f005000e420 con 0x7f0058151c30 2023-03-03T13:25:36.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.157+0000 7f004effd700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f003c0682f0 0x7f003c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:36.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.157+0000 7f004effd700 1 -- 172.21.15.7:0/3028433699 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0050086350 con 0x7f0058151c30 2023-03-03T13:25:36.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.157+0000 7f005e3d1700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f003c0682f0 0x7f003c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:36.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.158+0000 7f005e3d1700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f003c0682f0 0x7f003c06a7a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0054009c40 tx=0x7f0054008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:25:36.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.161+0000 7f004effd700 1 -- 172.21.15.7:0/3028433699 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0050052690 con 0x7f0058151c30 2023-03-03T13:25:36.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.398+0000 7f0060635700 1 -- 172.21.15.7:0/3028433699 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00580ac4d0 con 0x7f0058151c30 2023-03-03T13:25:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.398+0000 7f004effd700 1 -- 172.21.15.7:0/3028433699 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f005001c090 con 0x7f0058151c30 2023-03-03T13:25:36.401 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:25:36.402 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:25:36.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.400+0000 7f0060635700 1 -- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f003c0682f0 msgr2=0x7f003c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:36.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.400+0000 7f0060635700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f003c0682f0 0x7f003c06a7a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0054009c40 tx=0x7f0054008030 comp rx=0 tx=0).stop 2023-03-03T13:25:36.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.400+0000 7f0060635700 1 -- 172.21.15.7:0/3028433699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058151c30 msgr2=0x7f00580c0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:36.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.400+0000 7f0060635700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058151c30 0x7f00580c0a00 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f005000b9a0 tx=0x7f005000bcb0 comp rx=0 tx=0).stop 2023-03-03T13:25:36.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.401+0000 7f0060635700 1 -- 172.21.15.7:0/3028433699 shutdown_connections 2023-03-03T13:25:36.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.401+0000 7f0060635700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f003c0682f0 0x7f003c06a7a0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:36.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.401+0000 7f0060635700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0058151c30 0x7f00580c0a00 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:36.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.401+0000 7f0060635700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f005806e7f0 0x7f00580c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:36.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.401+0000 7f0060635700 1 --2- 172.21.15.7:0/3028433699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00580717a0 0x7f00580c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:36.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.401+0000 7f0060635700 1 -- 172.21.15.7:0/3028433699 >> 172.21.15.7:0/3028433699 conn(0x7f00580a94c0 msgr2=0x7f00580a9890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:25:36.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.401+0000 7f0060635700 1 -- 172.21.15.7:0/3028433699 shutdown_connections 2023-03-03T13:25:36.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:36.401+0000 7f0060635700 1 -- 172.21.15.7:0/3028433699 wait complete. 2023-03-03T13:25:36.405 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:25:36.415 INFO:tasks.mds_thrash.fs.[b]:waiting for 115 secs before thrashing 2023-03-03T13:25:41.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:41.805 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:25:55.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:25:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.463+0000 7fb0aff3c700 1 Processor -- start 2023-03-03T13:25:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.463+0000 7fb0aff3c700 1 -- start start 2023-03-03T13:25:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.464+0000 7fb0aff3c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a8071e80 0x7fb0a8151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.464+0000 7fb0aff3c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb0a814e900 0x7fb0a815ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.464+0000 7fb0aff3c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a81510d0 con 0x7fb0a8072250 2023-03-03T13:25:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.464+0000 7fb0aff3c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a80c3740 con 0x7fb0a814e900 2023-03-03T13:25:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.464+0000 7fb0aff3c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a8070670 con 0x7fb0a8071e80 2023-03-03T13:25:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0ae4d9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb0a814e900 0x7fb0a815ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0adcd8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a8071e80 0x7fb0a8151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0ad4d7700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb0a8072250 0x7fb0a814e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:42084/0 (socket says 172.21.15.7:42084) 2023-03-03T13:25:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0ad4d7700 1 -- 172.21.15.7:0/4082614197 learned_addr learned my addr 172.21.15.7:0/4082614197 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:25:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0adcd8700 1 -- 172.21.15.7:0/4082614197 >> v2:172.21.15.143:3300/0 conn(0x7fb0a814e900 msgr2=0x7fb0a815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:25:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0adcd8700 1 --2- 172.21.15.7:0/4082614197 >> v2:172.21.15.143:3300/0 conn(0x7fb0a814e900 0x7fb0a815ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0adcd8700 1 -- 172.21.15.7:0/4082614197 >> v1:172.21.15.143:6789/0 conn(0x7fb0a8072250 legacy=0x7fb0a814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0adcd8700 1 -- 172.21.15.7:0/4082614197 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a80c39e0 con 0x7fb0a8071e80 2023-03-03T13:25:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0ae4d9700 1 --2- 172.21.15.7:0/4082614197 >> v2:172.21.15.143:3300/0 conn(0x7fb0a814e900 0x7fb0a815ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:25:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0adcd8700 1 --2- 172.21.15.7:0/4082614197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a8071e80 0x7fb0a8151820 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb098009840 tx=0x7fb098009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cc43a5e3a9799fa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:25:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0accd6700 1 -- 172.21.15.7:0/4082614197 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb098011830 con 0x7fb0a8071e80 2023-03-03T13:25:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0accd6700 1 -- 172.21.15.7:0/4082614197 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb098011990 con 0x7fb0a8071e80 2023-03-03T13:25:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.465+0000 7fb0accd6700 1 -- 172.21.15.7:0/4082614197 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb098011d20 con 0x7fb0a8071e80 2023-03-03T13:25:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.466+0000 7fb0aff3c700 1 -- 172.21.15.7:0/4082614197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a8071e80 msgr2=0x7fb0a8151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.466+0000 7fb0aff3c700 1 --2- 172.21.15.7:0/4082614197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a8071e80 0x7fb0a8151820 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb098009840 tx=0x7fb098009b50 comp rx=0 tx=0).stop 2023-03-03T13:25:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.466+0000 7fb0aff3c700 1 -- 172.21.15.7:0/4082614197 shutdown_connections 2023-03-03T13:25:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.466+0000 7fb0aff3c700 1 --2- 172.21.15.7:0/4082614197 >> v2:172.21.15.143:3300/0 conn(0x7fb0a814e900 0x7fb0a815ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.466+0000 7fb0aff3c700 1 --2- 172.21.15.7:0/4082614197 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a8071e80 0x7fb0a8151820 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.466+0000 7fb0aff3c700 1 -- 172.21.15.7:0/4082614197 >> 172.21.15.7:0/4082614197 conn(0x7fb0a80a94d0 msgr2=0x7fb0a80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:25:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.466+0000 7fb0aff3c700 1 -- 172.21.15.7:0/4082614197 shutdown_connections 2023-03-03T13:25:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.466+0000 7fb0aff3c700 1 -- 172.21.15.7:0/4082614197 wait complete. 2023-03-03T13:25:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.467+0000 7fb0aff3c700 1 Processor -- start 2023-03-03T13:25:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.467+0000 7fb0aff3c700 1 -- start start 2023-03-03T13:25:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0aff3c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0a8071e80 0x7fb0a80aa470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0aff3c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0a8072250 0x7fb0a80abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0aff3c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a814e900 0x7fb0a81dbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0aff3c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a80ba420 con 0x7fb0a8072250 2023-03-03T13:25:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0aff3c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a80ac1d0 con 0x7fb0a8071e80 2023-03-03T13:25:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0aff3c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a8150d30 con 0x7fb0a814e900 2023-03-03T13:25:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0ad4d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0a8072250 0x7fb0a80abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0adcd8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0a8071e80 0x7fb0a80aa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0ae4d9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a814e900 0x7fb0a81dbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0ad4d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0a8072250 0x7fb0a80abc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44594/0 (socket says 172.21.15.7:44594) 2023-03-03T13:25:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0ad4d7700 1 -- 172.21.15.7:0/2980969351 learned_addr learned my addr 172.21.15.7:0/2980969351 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:25:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0ad4d7700 1 -- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a814e900 msgr2=0x7fb0a81dbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0ad4d7700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a814e900 0x7fb0a81dbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0ad4d7700 1 -- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0a8071e80 msgr2=0x7fb0a80aa470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:55.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0ad4d7700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0a8071e80 0x7fb0a80aa470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:55.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.468+0000 7fb0ad4d7700 1 -- 172.21.15.7:0/2980969351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a0009a70 con 0x7fb0a8072250 2023-03-03T13:25:55.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.469+0000 7fb0ad4d7700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0a8072250 0x7fb0a80abc90 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400c530 tx=0x7fb0a400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:25:55.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.469+0000 7fb09e7fc700 1 -- 172.21.15.7:0/2980969351 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4005d70 con 0x7fb0a8072250 2023-03-03T13:25:55.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.469+0000 7fb09e7fc700 1 -- 172.21.15.7:0/2980969351 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a400ce90 con 0x7fb0a8072250 2023-03-03T13:25:55.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.469+0000 7fb0aff3c700 1 -- 172.21.15.7:0/2980969351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0980094a0 con 0x7fb0a8072250 2023-03-03T13:25:55.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.469+0000 7fb09e7fc700 1 -- 172.21.15.7:0/2980969351 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a40072f0 con 0x7fb0a8072250 2023-03-03T13:25:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.469+0000 7fb0aff3c700 1 -- 172.21.15.7:0/2980969351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a806c9a0 con 0x7fb0a8072250 2023-03-03T13:25:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.470+0000 7fb0aff3c700 1 -- 172.21.15.7:0/2980969351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a8151300 con 0x7fb0a8072250 2023-03-03T13:25:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.470+0000 7fb09e7fc700 1 -- 172.21.15.7:0/2980969351 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb0a4019030 con 0x7fb0a8072250 2023-03-03T13:25:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.470+0000 7fb09e7fc700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb08c0682f0 0x7fb08c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:25:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.471+0000 7fb09e7fc700 1 -- 172.21.15.7:0/2980969351 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb0a4085340 con 0x7fb0a8072250 2023-03-03T13:25:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.471+0000 7fb0adcd8700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb08c0682f0 0x7fb08c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:25:55.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.471+0000 7fb0adcd8700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb08c0682f0 0x7fb08c06a7a0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb098000c00 tx=0x7fb098008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:25:55.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.475+0000 7fb09e7fc700 1 -- 172.21.15.7:0/2980969351 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb0a40516e0 con 0x7fb0a8072250 2023-03-03T13:25:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.712+0000 7fb0aff3c700 1 -- 172.21.15.7:0/2980969351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0a806cb30 con 0x7fb0a8072250 2023-03-03T13:25:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.713+0000 7fb09e7fc700 1 -- 172.21.15.7:0/2980969351 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb0a4017090 con 0x7fb0a8072250 2023-03-03T13:25:55.715 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:25:55.715 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:25:55.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 -- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb08c0682f0 msgr2=0x7fb08c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb08c0682f0 0x7fb08c06a7a0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb098000c00 tx=0x7fb098008030 comp rx=0 tx=0).stop 2023-03-03T13:25:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 -- 172.21.15.7:0/2980969351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0a8072250 msgr2=0x7fb0a80abc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:25:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0a8072250 0x7fb0a80abc90 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400c530 tx=0x7fb0a400c840 comp rx=0 tx=0).stop 2023-03-03T13:25:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 -- 172.21.15.7:0/2980969351 shutdown_connections 2023-03-03T13:25:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0a814e900 0x7fb0a81dbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0a8071e80 0x7fb0a80aa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb08c0682f0 0x7fb08c06a7a0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 --2- 172.21.15.7:0/2980969351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0a8072250 0x7fb0a80abc90 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:25:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 -- 172.21.15.7:0/2980969351 >> 172.21.15.7:0/2980969351 conn(0x7fb0a80a94d0 msgr2=0x7fb0a8151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:25:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 -- 172.21.15.7:0/2980969351 shutdown_connections 2023-03-03T13:25:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:25:55.715+0000 7fb0aff3c700 1 -- 172.21.15.7:0/2980969351 wait complete. 2023-03-03T13:25:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:25:55.728 INFO:tasks.mds_thrash.fs.[a]:waiting for 25 secs before thrashing 2023-03-03T13:26:11.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:11.869 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:20.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:26:20.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.866+0000 7f4804218700 1 Processor -- start 2023-03-03T13:26:20.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.867+0000 7f4804218700 1 -- start start 2023-03-03T13:26:20.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.867+0000 7f4804218700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc151d40 0x7f47fc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:20.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.867+0000 7f4804218700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f47fc073c30 0x7f47fc1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:20.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.867+0000 7f4804218700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47fc06d430 con 0x7f47fc0c2930 2023-03-03T13:26:20.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.867+0000 7f4804218700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47fc06c2a0 con 0x7f47fc073c30 2023-03-03T13:26:20.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.867+0000 7f4804218700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc1586e0 con 0x7f47fc151d40 2023-03-03T13:26:20.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f48017b3700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f47fc0c2930 0x7f47fc157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38580/0 (socket says 172.21.15.7:38580) 2023-03-03T13:26:20.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f48017b3700 1 -- 172.21.15.7:0/172033681 learned_addr learned my addr 172.21.15.7:0/172033681 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:26:20.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4801fb4700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc151d40 0x7f47fc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:20.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f48027b5700 1 --2- 172.21.15.7:0/172033681 >> v2:172.21.15.143:3300/0 conn(0x7f47fc073c30 0x7f47fc1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:20.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963434151 0 0) 0x7f47fc06d430 con 0x7f47fc0c2930 2023-03-03T13:26:20.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e80035b0 con 0x7f47fc0c2930 2023-03-03T13:26:20.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3928823703 0 0) 0x7f47e80035b0 con 0x7f47fc0c2930 2023-03-03T13:26:20.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc151d40 msgr2=0x7f47fc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:26:20.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc151d40 0x7f47fc0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:20.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 >> v2:172.21.15.143:3300/0 conn(0x7f47fc073c30 msgr2=0x7f47fc1581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:26:20.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 --2- 172.21.15.7:0/172033681 >> v2:172.21.15.143:3300/0 conn(0x7f47fc073c30 0x7f47fc1581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:20.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc06cb00 con 0x7f47fc0c2930 2023-03-03T13:26:20.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f47ec001670 con 0x7f47fc0c2930 2023-03-03T13:26:20.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 >> v1:172.21.15.7:6789/0 conn(0x7f47fc0c2930 legacy=0x7f47fc157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:20.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47e8003a70 0x7f47e8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:20.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc151d40 0x7f47e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:20.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e8004d90 0x7f47e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:20.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc06c2a0 con 0x7f47e8003a70 2023-03-03T13:26:20.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc1586e0 con 0x7f47e8004d90 2023-03-03T13:26:20.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e800baf0 con 0x7f47fc151d40 2023-03-03T13:26:20.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.868+0000 7f4801fb4700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47e8003a70 0x7f47e8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:20.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f48017b3700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc151d40 0x7f47e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:20.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f48027b5700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e8004d90 0x7f47e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:20.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4801fb4700 1 -- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc151d40 msgr2=0x7f47e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:26:20.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4801fb4700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc151d40 0x7f47e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:20.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4801fb4700 1 -- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e8004d90 msgr2=0x7f47e8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:26:20.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4801fb4700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e8004d90 0x7f47e8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:20.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4801fb4700 1 -- 172.21.15.7:0/172033681 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e80035b0 con 0x7f47e8003a70 2023-03-03T13:26:20.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4801fb4700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47e8003a70 0x7f47e8003e80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009b60 tx=0x7f47e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd63f3c164b01c5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:26:20.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f800e6c0 con 0x7f47e8003a70 2023-03-03T13:26:20.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f800e820 con 0x7f47e8003a70 2023-03-03T13:26:20.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4800fb2700 1 -- 172.21.15.7:0/172033681 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f800eb90 con 0x7f47e8003a70 2023-03-03T13:26:20.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4804218700 1 -- 172.21.15.7:0/172033681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47e8003a70 msgr2=0x7f47e8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:20.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f4804218700 1 --2- 172.21.15.7:0/172033681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47e8003a70 0x7f47e8003e80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009b60 tx=0x7f47e8005880 comp rx=0 tx=0).stop 2023-03-03T13:26:20.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.869+0000 7f48027b5700 1 -- 172.21.15.7:0/172033681 reap_dead start 2023-03-03T13:26:20.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.870+0000 7f4804218700 1 -- 172.21.15.7:0/172033681 shutdown_connections 2023-03-03T13:26:20.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.870+0000 7f4804218700 1 -- 172.21.15.7:0/172033681 >> 172.21.15.7:0/172033681 conn(0x7f47fc0a94d0 msgr2=0x7f47fc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:26:20.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.870+0000 7f4804218700 1 -- 172.21.15.7:0/172033681 shutdown_connections 2023-03-03T13:26:20.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.870+0000 7f4804218700 1 -- 172.21.15.7:0/172033681 wait complete. 2023-03-03T13:26:20.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.871+0000 7f4804218700 1 Processor -- start 2023-03-03T13:26:20.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.871+0000 7f4804218700 1 -- start start 2023-03-03T13:26:20.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f4804218700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e8003a70 0x7f47fc152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:20.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f4804218700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47fc0be430 0x7f47fc1542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:20.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f4804218700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc0bf480 0x7f47fc154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:20.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f4804218700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc1edd20 con 0x7f47fc0be430 2023-03-03T13:26:20.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f4804218700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc1ed110 con 0x7f47e8003a70 2023-03-03T13:26:20.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f4804218700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc1ed3f0 con 0x7f47fc0bf480 2023-03-03T13:26:20.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f48017b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47fc0be430 0x7f47fc1542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:20.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f48017b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47fc0be430 0x7f47fc1542f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44614/0 (socket says 172.21.15.7:44614) 2023-03-03T13:26:20.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f48017b3700 1 -- 172.21.15.7:0/3112115964 learned_addr learned my addr 172.21.15.7:0/3112115964 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:26:20.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f4801fb4700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e8003a70 0x7f47fc152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:20.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f48027b5700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc0bf480 0x7f47fc154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:20.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f48017b3700 1 -- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc0bf480 msgr2=0x7f47fc154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:20.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f48017b3700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc0bf480 0x7f47fc154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:20.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f48017b3700 1 -- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e8003a70 msgr2=0x7f47fc152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:20.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f48017b3700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e8003a70 0x7f47fc152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:20.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f48017b3700 1 -- 172.21.15.7:0/3112115964 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc06d0c0 con 0x7f47fc0be430 2023-03-03T13:26:20.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f48017b3700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47fc0be430 0x7f47fc1542f0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00e780 tx=0x7f47ec00eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:26:20.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f47f27fc700 1 -- 172.21.15.7:0/3112115964 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec012590 con 0x7f47fc0be430 2023-03-03T13:26:20.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f47f27fc700 1 -- 172.21.15.7:0/3112115964 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec0126f0 con 0x7f47fc0be430 2023-03-03T13:26:20.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f4804218700 1 -- 172.21.15.7:0/3112115964 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47fc06cde0 con 0x7f47fc0be430 2023-03-03T13:26:20.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f47f27fc700 1 -- 172.21.15.7:0/3112115964 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec012aa0 con 0x7f47fc0be430 2023-03-03T13:26:20.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.872+0000 7f4804218700 1 -- 172.21.15.7:0/3112115964 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47fc06bea0 con 0x7f47fc0be430 2023-03-03T13:26:20.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.873+0000 7f4804218700 1 -- 172.21.15.7:0/3112115964 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47fc074bb0 con 0x7f47fc0be430 2023-03-03T13:26:20.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.874+0000 7f47f27fc700 1 -- 172.21.15.7:0/3112115964 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f47ec012c00 con 0x7f47fc0be430 2023-03-03T13:26:20.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.874+0000 7f47f27fc700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47e00682f0 0x7f47e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:26:20.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.875+0000 7f47f27fc700 1 -- 172.21.15.7:0/3112115964 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f47ec0860c0 con 0x7f47fc0be430 2023-03-03T13:26:20.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.875+0000 7f4801fb4700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47e00682f0 0x7f47e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:26:20.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.875+0000 7f4801fb4700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47e00682f0 0x7f47e006a7a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009a80 tx=0x7f47f8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:26:20.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:20.879+0000 7f47f27fc700 1 -- 172.21.15.7:0/3112115964 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47ec0523e0 con 0x7f47fc0be430 2023-03-03T13:26:21.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.114+0000 7f4804218700 1 -- 172.21.15.7:0/3112115964 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47fc0c26a0 con 0x7f47fc0be430 2023-03-03T13:26:21.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.115+0000 7f47f27fc700 1 -- 172.21.15.7:0/3112115964 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f47ec055a00 con 0x7f47fc0be430 2023-03-03T13:26:21.117 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:26:21.117 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:26:21.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 -- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47e00682f0 msgr2=0x7f47e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:21.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47e00682f0 0x7f47e006a7a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009a80 tx=0x7f47f8008030 comp rx=0 tx=0).stop 2023-03-03T13:26:21.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 -- 172.21.15.7:0/3112115964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47fc0be430 msgr2=0x7f47fc1542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:26:21.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47fc0be430 0x7f47fc1542f0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f47ec00e780 tx=0x7f47ec00eb40 comp rx=0 tx=0).stop 2023-03-03T13:26:21.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 -- 172.21.15.7:0/3112115964 shutdown_connections 2023-03-03T13:26:21.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47fc0bf480 0x7f47fc154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:21.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47e8003a70 0x7f47fc152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:21.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47e00682f0 0x7f47e006a7a0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:21.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 --2- 172.21.15.7:0/3112115964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47fc0be430 0x7f47fc1542f0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:26:21.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 -- 172.21.15.7:0/3112115964 >> 172.21.15.7:0/3112115964 conn(0x7f47fc0a94d0 msgr2=0x7f47fc0c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:26:21.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 -- 172.21.15.7:0/3112115964 shutdown_connections 2023-03-03T13:26:21.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:26:21.117+0000 7f4804218700 1 -- 172.21.15.7:0/3112115964 wait complete. 2023-03-03T13:26:21.120 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:26:21.131 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2023-03-03T13:26:42.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:26:42.015 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:12.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:12.095 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:31.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:27:31.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.553+0000 7fe47faa1700 1 Processor -- start 2023-03-03T13:27:31.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.554+0000 7fe47faa1700 1 -- start start 2023-03-03T13:27:31.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.554+0000 7fe47faa1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c2ec0 0x7fe47815fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:31.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.554+0000 7fe47faa1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe478157ad0 con 0x7fe4780a60f0 2023-03-03T13:27:31.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.554+0000 7fe47faa1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4780c6340 con 0x7fe47806e0a0 2023-03-03T13:27:31.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.554+0000 7fe47faa1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47806c790 con 0x7fe4780c2ec0 2023-03-03T13:27:31.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47d03c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe4780a60f0 0x7fe478157f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38656/0 (socket says 172.21.15.7:38656) 2023-03-03T13:27:31.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47d03c700 1 -- 172.21.15.7:0/3210197572 learned_addr learned my addr 172.21.15.7:0/3210197572 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:27:31.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47e03e700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c2ec0 0x7fe47815fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:31.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366146959 0 0) 0x7fe478157ad0 con 0x7fe4780a60f0 2023-03-03T13:27:31.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4600035b0 con 0x7fe4780a60f0 2023-03-03T13:27:31.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 301292999 0 0) 0x7fe4600035b0 con 0x7fe4780a60f0 2023-03-03T13:27:31.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c2ec0 msgr2=0x7fe47815fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:27:31.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c2ec0 0x7fe47815fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:31.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 >> v1:172.21.15.143:6789/0 conn(0x7fe47806e0a0 legacy=0x7fe4780a85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:31.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe478155040 con 0x7fe4780a60f0 2023-03-03T13:27:31.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe46c001670 con 0x7fe4780a60f0 2023-03-03T13:27:31.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 >> v1:172.21.15.7:6789/0 conn(0x7fe4780a60f0 legacy=0x7fe478157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:31.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c2ec0 0x7fe460003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:31.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 0x7fe460004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:31.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe460004d90 0x7fe460005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:31.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46000bcc0 con 0x7fe460004020 2023-03-03T13:27:31.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46000be30 con 0x7fe460004d90 2023-03-03T13:27:31.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46000bff0 con 0x7fe4780c2ec0 2023-03-03T13:27:31.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47d03c700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 0x7fe460004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:31.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.555+0000 7fe47d83d700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c2ec0 0x7fe460003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:31.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47e03e700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe460004d90 0x7fe460005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:31.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47d03c700 1 -- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c2ec0 msgr2=0x7fe460003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:27:31.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47d03c700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c2ec0 0x7fe460003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:31.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47d03c700 1 -- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe460004d90 msgr2=0x7fe460005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:27:31.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47d03c700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe460004d90 0x7fe460005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:31.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47d03c700 1 -- 172.21.15.7:0/3210197572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4600035b0 con 0x7fe460004020 2023-03-03T13:27:31.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47d83d700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c2ec0 0x7fe460003aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:27:31.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47d03c700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 0x7fe460004850 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00c210 tx=0x7fe460005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffc5e6ca346de09 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:31.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c007c10 con 0x7fe460004020 2023-03-03T13:27:31.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46c00cdb0 con 0x7fe460004020 2023-03-03T13:27:31.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47c83b700 1 -- 172.21.15.7:0/3210197572 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c005ba0 con 0x7fe460004020 2023-03-03T13:27:31.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.556+0000 7fe47faa1700 1 -- 172.21.15.7:0/3210197572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 msgr2=0x7fe460004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:31.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.557+0000 7fe47faa1700 1 --2- 172.21.15.7:0/3210197572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 0x7fe460004850 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00c210 tx=0x7fe460005880 comp rx=0 tx=0).stop 2023-03-03T13:27:31.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.557+0000 7fe47e03e700 1 -- 172.21.15.7:0/3210197572 reap_dead start 2023-03-03T13:27:31.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.557+0000 7fe47faa1700 1 -- 172.21.15.7:0/3210197572 shutdown_connections 2023-03-03T13:27:31.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.557+0000 7fe47faa1700 1 -- 172.21.15.7:0/3210197572 >> 172.21.15.7:0/3210197572 conn(0x7fe47806af40 msgr2=0x7fe4780c6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:27:31.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.557+0000 7fe47faa1700 1 -- 172.21.15.7:0/3210197572 shutdown_connections 2023-03-03T13:27:31.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.557+0000 7fe47faa1700 1 -- 172.21.15.7:0/3210197572 wait complete. 2023-03-03T13:27:31.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.558+0000 7fe47faa1700 1 Processor -- start 2023-03-03T13:27:31.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.558+0000 7fe47faa1700 1 -- start start 2023-03-03T13:27:31.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47faa1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 0x7fe47806f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:31.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47faa1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe478156c70 0x7fe47814d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:31.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47faa1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c0760 0x7fe4781edb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:31.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47faa1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4781ed3d0 con 0x7fe460004020 2023-03-03T13:27:31.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47faa1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe478157380 con 0x7fe478156c70 2023-03-03T13:27:31.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47faa1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780bd450 con 0x7fe4780c0760 2023-03-03T13:27:31.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d83d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 0x7fe47806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:31.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d83d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 0x7fe47806f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44686/0 (socket says 172.21.15.7:44686) 2023-03-03T13:27:31.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d83d700 1 -- 172.21.15.7:0/3079400095 learned_addr learned my addr 172.21.15.7:0/3079400095 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:27:31.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d03c700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe478156c70 0x7fe47814d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:31.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47e03e700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c0760 0x7fe4781edb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:31.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d83d700 1 -- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c0760 msgr2=0x7fe4781edb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:31.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d83d700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c0760 0x7fe4781edb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:31.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d83d700 1 -- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe478156c70 msgr2=0x7fe47814d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:31.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d83d700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe478156c70 0x7fe47814d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:31.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d83d700 1 -- 172.21.15.7:0/3079400095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4780c6620 con 0x7fe460004020 2023-03-03T13:27:31.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d03c700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe478156c70 0x7fe47814d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:27:31.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47d83d700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 0x7fe47806f840 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fe46800d180 tx=0x7fe46800d540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:31.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe4767fc700 1 -- 172.21.15.7:0/3079400095 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46800ddc0 con 0x7fe460004020 2023-03-03T13:27:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe4767fc700 1 -- 172.21.15.7:0/3079400095 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe468003a50 con 0x7fe460004020 2023-03-03T13:27:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe4767fc700 1 -- 172.21.15.7:0/3079400095 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468003e00 con 0x7fe460004020 2023-03-03T13:27:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47faa1700 1 -- 172.21.15.7:0/3079400095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe478155320 con 0x7fe460004020 2023-03-03T13:27:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.559+0000 7fe47faa1700 1 -- 172.21.15.7:0/3079400095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47806d2c0 con 0x7fe460004020 2023-03-03T13:27:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.560+0000 7fe47faa1700 1 -- 172.21.15.7:0/3079400095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4781ed950 con 0x7fe460004020 2023-03-03T13:27:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.561+0000 7fe4767fc700 1 -- 172.21.15.7:0/3079400095 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe4680050d0 con 0x7fe460004020 2023-03-03T13:27:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.561+0000 7fe4767fc700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 0x7fe46406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:31.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.561+0000 7fe4767fc700 1 -- 172.21.15.7:0/3079400095 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe468086900 con 0x7fe460004020 2023-03-03T13:27:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.562+0000 7fe47d03c700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 0x7fe46406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.562+0000 7fe47d03c700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 0x7fe46406a7a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe46c000c00 tx=0x7fe46c007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:31.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.566+0000 7fe4767fc700 1 -- 172.21.15.7:0/3079400095 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe468052ca0 con 0x7fe460004020 2023-03-03T13:27:31.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.807+0000 7fe47faa1700 1 -- 172.21.15.7:0/3079400095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe47806c2d0 con 0x7fe460004020 2023-03-03T13:27:31.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.808+0000 7fe4767fc700 1 -- 172.21.15.7:0/3079400095 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe4680562c0 con 0x7fe460004020 2023-03-03T13:27:31.810 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:27:31.810 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:27:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 -- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 msgr2=0x7fe46406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 0x7fe46406a7a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe46c000c00 tx=0x7fe46c007000 comp rx=0 tx=0).stop 2023-03-03T13:27:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 -- 172.21.15.7:0/3079400095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 msgr2=0x7fe47806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:31.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 0x7fe47806f840 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fe46800d180 tx=0x7fe46800d540 comp rx=0 tx=0).stop 2023-03-03T13:27:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 -- 172.21.15.7:0/3079400095 shutdown_connections 2023-03-03T13:27:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4780c0760 0x7fe4781edb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4640682f0 0x7fe46406a7a0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe460004020 0x7fe47806f840 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 --2- 172.21.15.7:0/3079400095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe478156c70 0x7fe47814d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 -- 172.21.15.7:0/3079400095 >> 172.21.15.7:0/3079400095 conn(0x7fe47806af40 msgr2=0x7fe478157e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:27:31.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 -- 172.21.15.7:0/3079400095 shutdown_connections 2023-03-03T13:27:31.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:31.810+0000 7fe47faa1700 1 -- 172.21.15.7:0/3079400095 wait complete. 2023-03-03T13:27:31.814 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:27:31.824 INFO:tasks.mds_thrash.fs.[b]:waiting for 10 secs before thrashing 2023-03-03T13:27:40.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:27:40.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.267+0000 7f09a3f5a700 1 Processor -- start 2023-03-03T13:27:40.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.267+0000 7f09a3f5a700 1 -- start start 2023-03-03T13:27:40.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a3f5a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f099c0caa30 0x7f099c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:40.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a3f5a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099c0c7310 con 0x7f099c154ff0 2023-03-03T13:27:40.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a3f5a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099c0bfd10 con 0x7f099c0c00e0 2023-03-03T13:27:40.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a3f5a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f099c155430 con 0x7f099c0caa30 2023-03-03T13:27:40.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a1cf6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f099c0caa30 0x7f099c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a24f7700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f099c154ff0 0x7f099c15ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38684/0 (socket says 172.21.15.7:38684) 2023-03-03T13:27:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a24f7700 1 -- 172.21.15.7:0/1976472426 learned_addr learned my addr 172.21.15.7:0/1976472426 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:27:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a1cf6700 1 -- 172.21.15.7:0/1976472426 >> v1:172.21.15.143:6789/0 conn(0x7f099c0c00e0 legacy=0x7f099c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a1cf6700 1 -- 172.21.15.7:0/1976472426 >> v1:172.21.15.7:6789/0 conn(0x7f099c154ff0 legacy=0x7f099c15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a1cf6700 1 -- 172.21.15.7:0/1976472426 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c0bfa30 con 0x7f099c0caa30 2023-03-03T13:27:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a0cf4700 1 -- 172.21.15.7:0/1976472426 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969457184 0 0) 0x7f099c0c7310 con 0x7f099c154ff0 2023-03-03T13:27:40.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.268+0000 7f09a1cf6700 1 --2- 172.21.15.7:0/1976472426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c0caa30 0x7f099c0c5c60 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f0998009840 tx=0x7f0998009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b639547405e471 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:40.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.269+0000 7f09a0cf4700 1 -- 172.21.15.7:0/1976472426 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0998011830 con 0x7f099c0caa30 2023-03-03T13:27:40.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.269+0000 7f09a0cf4700 1 -- 172.21.15.7:0/1976472426 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0998011990 con 0x7f099c0caa30 2023-03-03T13:27:40.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.269+0000 7f09a0cf4700 1 -- 172.21.15.7:0/1976472426 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0998011d20 con 0x7f099c0caa30 2023-03-03T13:27:40.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.269+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1976472426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c0caa30 msgr2=0x7f099c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:40.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.269+0000 7f09a3f5a700 1 --2- 172.21.15.7:0/1976472426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c0caa30 0x7f099c0c5c60 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f0998009840 tx=0x7f0998009b50 comp rx=0 tx=0).stop 2023-03-03T13:27:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.269+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1976472426 shutdown_connections 2023-03-03T13:27:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.269+0000 7f09a3f5a700 1 --2- 172.21.15.7:0/1976472426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c0caa30 0x7f099c0c5c60 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.269+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1976472426 >> 172.21.15.7:0/1976472426 conn(0x7f099c066e40 msgr2=0x7f099c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:27:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.269+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1976472426 shutdown_connections 2023-03-03T13:27:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.269+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1976472426 wait complete. 2023-03-03T13:27:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.270+0000 7f09a3f5a700 1 Processor -- start 2023-03-03T13:27:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a3f5a700 1 -- start start 2023-03-03T13:27:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a3f5a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f099c0c00e0 0x7f099c14a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a3f5a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f099c0caa30 0x7f099c1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a3f5a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c154ff0 0x7f099c1559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a3f5a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099c1ed330 con 0x7f099c154ff0 2023-03-03T13:27:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a3f5a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099c1571f0 con 0x7f099c0c00e0 2023-03-03T13:27:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a3f5a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099c157b30 con 0x7f099c0caa30 2023-03-03T13:27:40.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a24f7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c154ff0 0x7f099c1559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a24f7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c154ff0 0x7f099c1559f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44710/0 (socket says 172.21.15.7:44710) 2023-03-03T13:27:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a24f7700 1 -- 172.21.15.7:0/1602276909 learned_addr learned my addr 172.21.15.7:0/1602276909 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:27:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a1cf6700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f099c0c00e0 0x7f099c14a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.271+0000 7f09a14f5700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f099c0caa30 0x7f099c1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09a24f7700 1 -- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f099c0caa30 msgr2=0x7f099c1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09a24f7700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f099c0caa30 0x7f099c1575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09a24f7700 1 -- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f099c0c00e0 msgr2=0x7f099c14a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09a24f7700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f099c0c00e0 0x7f099c14a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09a24f7700 1 -- 172.21.15.7:0/1602276909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09980094a0 con 0x7f099c154ff0 2023-03-03T13:27:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09a24f7700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c154ff0 0x7f099c1559f0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f099400cc70 tx=0x7f099400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09927fc700 1 -- 172.21.15.7:0/1602276909 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099400aac0 con 0x7f099c154ff0 2023-03-03T13:27:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09927fc700 1 -- 172.21.15.7:0/1602276909 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099400ac40 con 0x7f099c154ff0 2023-03-03T13:27:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1602276909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099c1547f0 con 0x7f099c154ff0 2023-03-03T13:27:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09927fc700 1 -- 172.21.15.7:0/1602276909 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09940070d0 con 0x7f099c154ff0 2023-03-03T13:27:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.272+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1602276909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099c154510 con 0x7f099c154ff0 2023-03-03T13:27:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.273+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1602276909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099c154ad0 con 0x7f099c154ff0 2023-03-03T13:27:40.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.274+0000 7f09927fc700 1 -- 172.21.15.7:0/1602276909 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f099401a030 con 0x7f099c154ff0 2023-03-03T13:27:40.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.274+0000 7f09927fc700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09800682f0 0x7f098006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:40.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.274+0000 7f09927fc700 1 -- 172.21.15.7:0/1602276909 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0994085e10 con 0x7f099c154ff0 2023-03-03T13:27:40.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.274+0000 7f09a1cf6700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09800682f0 0x7f098006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:40.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.275+0000 7f09a1cf6700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09800682f0 0x7f098006a7a0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f09980090b0 tx=0x7f099801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:40.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.279+0000 7f09927fc700 1 -- 172.21.15.7:0/1602276909 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0994052150 con 0x7f099c154ff0 2023-03-03T13:27:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.519+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1602276909 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f099c1ed4e0 con 0x7f099c154ff0 2023-03-03T13:27:40.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.520+0000 7f09927fc700 1 -- 172.21.15.7:0/1602276909 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0994018090 con 0x7f099c154ff0 2023-03-03T13:27:40.523 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:27:40.523 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:27:40.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09800682f0 msgr2=0x7f098006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:40.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09800682f0 0x7f098006a7a0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f09980090b0 tx=0x7f099801c040 comp rx=0 tx=0).stop 2023-03-03T13:27:40.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1602276909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c154ff0 msgr2=0x7f099c1559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:40.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c154ff0 0x7f099c1559f0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f099400cc70 tx=0x7f099400cf80 comp rx=0 tx=0).stop 2023-03-03T13:27:40.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1602276909 shutdown_connections 2023-03-03T13:27:40.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09800682f0 0x7f098006a7a0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:40.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f099c154ff0 0x7f099c1559f0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:40.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f099c0c00e0 0x7f099c14a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:40.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 --2- 172.21.15.7:0/1602276909 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f099c0caa30 0x7f099c1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:40.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1602276909 >> 172.21.15.7:0/1602276909 conn(0x7f099c066e40 msgr2=0x7f099c06b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:27:40.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1602276909 shutdown_connections 2023-03-03T13:27:40.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:40.522+0000 7f09a3f5a700 1 -- 172.21.15.7:0/1602276909 wait complete. 2023-03-03T13:27:40.527 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:27:40.537 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T13:27:41.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:27:41.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.958+0000 7f5c5c3c7700 1 Processor -- start 2023-03-03T13:27:41.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.958+0000 7f5c5c3c7700 1 -- start start 2023-03-03T13:27:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.959+0000 7f5c5c3c7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c5414b0d0 0x7f5c54071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.959+0000 7f5c5c3c7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5c540aad20 0x7f5c54157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.959+0000 7f5c5c3c7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c540a9c20 con 0x7f5c54071d40 2023-03-03T13:27:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.959+0000 7f5c5c3c7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c54154f10 con 0x7f5c540aad20 2023-03-03T13:27:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.959+0000 7f5c5c3c7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c540ac3c0 con 0x7f5c5414b0d0 2023-03-03T13:27:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.959+0000 7f5c5a163700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c5414b0d0 0x7f5c54071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.959+0000 7f5c59962700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5c540aad20 0x7f5c54157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.959+0000 7f5c5a964700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f5c54071d40 0x7f5c54158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:42218/0 (socket says 172.21.15.7:42218) 2023-03-03T13:27:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.959+0000 7f5c5a964700 1 -- 172.21.15.7:0/564607905 learned_addr learned my addr 172.21.15.7:0/564607905 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:27:41.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c59161700 1 -- 172.21.15.7:0/564607905 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865493483 0 0) 0x7f5c540a9c20 con 0x7f5c54071d40 2023-03-03T13:27:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c59161700 1 -- 172.21.15.7:0/564607905 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c400035b0 con 0x7f5c54071d40 2023-03-03T13:27:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c5a163700 1 -- 172.21.15.7:0/564607905 >> v2:172.21.15.143:3300/0 conn(0x7f5c540aad20 msgr2=0x7f5c54157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:27:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c5a163700 1 --2- 172.21.15.7:0/564607905 >> v2:172.21.15.143:3300/0 conn(0x7f5c540aad20 0x7f5c54157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c5a163700 1 -- 172.21.15.7:0/564607905 >> v1:172.21.15.143:6789/0 conn(0x7f5c54071d40 legacy=0x7f5c54158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c5a163700 1 -- 172.21.15.7:0/564607905 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5406fc70 con 0x7f5c5414b0d0 2023-03-03T13:27:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c59962700 1 --2- 172.21.15.7:0/564607905 >> v2:172.21.15.143:3300/0 conn(0x7f5c540aad20 0x7f5c54157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:27:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c5a163700 1 --2- 172.21.15.7:0/564607905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c5414b0d0 0x7f5c54071800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009840 tx=0x7f5c44009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b380e54c589b32ee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c59161700 1 -- 172.21.15.7:0/564607905 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44011830 con 0x7f5c5414b0d0 2023-03-03T13:27:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c59161700 1 -- 172.21.15.7:0/564607905 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c44011990 con 0x7f5c5414b0d0 2023-03-03T13:27:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.960+0000 7f5c59161700 1 -- 172.21.15.7:0/564607905 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44011d00 con 0x7f5c5414b0d0 2023-03-03T13:27:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.961+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/564607905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c5414b0d0 msgr2=0x7f5c54071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.961+0000 7f5c5c3c7700 1 --2- 172.21.15.7:0/564607905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c5414b0d0 0x7f5c54071800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009840 tx=0x7f5c44009b50 comp rx=0 tx=0).stop 2023-03-03T13:27:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.961+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/564607905 shutdown_connections 2023-03-03T13:27:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.961+0000 7f5c5c3c7700 1 --2- 172.21.15.7:0/564607905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c5414b0d0 0x7f5c54071800 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.961+0000 7f5c5c3c7700 1 --2- 172.21.15.7:0/564607905 >> v2:172.21.15.143:3300/0 conn(0x7f5c540aad20 0x7f5c54157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.961+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/564607905 >> 172.21.15.7:0/564607905 conn(0x7f5c540c8900 msgr2=0x7f5c540c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:27:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.961+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/564607905 shutdown_connections 2023-03-03T13:27:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.961+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/564607905 wait complete. 2023-03-03T13:27:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.962+0000 7f5c5c3c7700 1 Processor -- start 2023-03-03T13:27:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.962+0000 7f5c5c3c7700 1 -- start start 2023-03-03T13:27:41.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5c3c7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c54071d40 0x7f5c540c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:41.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5c3c7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c540aad20 0x7f5c54145680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:41.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5c3c7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c5414b0d0 0x7f5c54145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:41.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5c3c7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c541eb130 con 0x7f5c54071d40 2023-03-03T13:27:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5c3c7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c54146100 con 0x7f5c5414b0d0 2023-03-03T13:27:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5c3c7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c541ddc30 con 0x7f5c540aad20 2023-03-03T13:27:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5a163700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c54071d40 0x7f5c540c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5a163700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c54071d40 0x7f5c540c75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44720/0 (socket says 172.21.15.7:44720) 2023-03-03T13:27:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5a163700 1 -- 172.21.15.7:0/2562723464 learned_addr learned my addr 172.21.15.7:0/2562723464 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:27:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5a964700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c5414b0d0 0x7f5c54145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c59962700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c540aad20 0x7f5c54145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5a163700 1 -- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c540aad20 msgr2=0x7f5c54145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5a163700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c540aad20 0x7f5c54145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5a163700 1 -- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c5414b0d0 msgr2=0x7f5c54145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5a163700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c5414b0d0 0x7f5c54145bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5a163700 1 -- 172.21.15.7:0/2562723464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c500094a0 con 0x7f5c54071d40 2023-03-03T13:27:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c5a163700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c54071d40 0x7f5c540c75d0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f5c44000c00 tx=0x7f5c44004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.963+0000 7f5c4affd700 1 -- 172.21.15.7:0/2562723464 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44004550 con 0x7f5c54071d40 2023-03-03T13:27:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.964+0000 7f5c4affd700 1 -- 172.21.15.7:0/2562723464 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c440036a0 con 0x7f5c54071d40 2023-03-03T13:27:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.964+0000 7f5c4affd700 1 -- 172.21.15.7:0/2562723464 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c44003a10 con 0x7f5c54071d40 2023-03-03T13:27:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.964+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/2562723464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c440094a0 con 0x7f5c54071d40 2023-03-03T13:27:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.964+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/2562723464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5406f6b0 con 0x7f5c54071d40 2023-03-03T13:27:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.964+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/2562723464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c540c8280 con 0x7f5c54071d40 2023-03-03T13:27:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.965+0000 7f5c4affd700 1 -- 172.21.15.7:0/2562723464 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5c4400f050 con 0x7f5c54071d40 2023-03-03T13:27:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.965+0000 7f5c4affd700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c380682f0 0x7f5c3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.965+0000 7f5c4affd700 1 -- 172.21.15.7:0/2562723464 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5c44087390 con 0x7f5c54071d40 2023-03-03T13:27:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.966+0000 7f5c59962700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c380682f0 0x7f5c3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:41.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.966+0000 7f5c59962700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c380682f0 0x7f5c3806a7a0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f5c540c4480 tx=0x7f5c50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:41.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:41.970+0000 7f5c4affd700 1 -- 172.21.15.7:0/2562723464 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5c44053730 con 0x7f5c54071d40 2023-03-03T13:27:42.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:42.208 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:27:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.209+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/2562723464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c5406f990 con 0x7f5c54071d40 2023-03-03T13:27:42.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.209+0000 7f5c4affd700 1 -- 172.21.15.7:0/2562723464 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5c44015090 con 0x7f5c54071d40 2023-03-03T13:27:42.212 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:27:42.212 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:27:42.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c380682f0 msgr2=0x7f5c3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c380682f0 0x7f5c3806a7a0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f5c540c4480 tx=0x7f5c50008040 comp rx=0 tx=0).stop 2023-03-03T13:27:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/2562723464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c54071d40 msgr2=0x7f5c540c75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c54071d40 0x7f5c540c75d0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f5c44000c00 tx=0x7f5c44004750 comp rx=0 tx=0).stop 2023-03-03T13:27:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/2562723464 shutdown_connections 2023-03-03T13:27:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c5414b0d0 0x7f5c54145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c380682f0 0x7f5c3806a7a0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c54071d40 0x7f5c540c75d0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 --2- 172.21.15.7:0/2562723464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c540aad20 0x7f5c54145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/2562723464 >> 172.21.15.7:0/2562723464 conn(0x7f5c540c8900 msgr2=0x7f5c540ad320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:27:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/2562723464 shutdown_connections 2023-03-03T13:27:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:42.212+0000 7f5c5c3c7700 1 -- 172.21.15.7:0/2562723464 wait complete. 2023-03-03T13:27:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:27:42.227 INFO:tasks.mds_thrash.fs.[b]:waiting for 17 secs before thrashing 2023-03-03T13:27:59.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:27:59.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.365+0000 7f8c85868700 1 Processor -- start 2023-03-03T13:27:59.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.365+0000 7f8c85868700 1 -- start start 2023-03-03T13:27:59.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.366+0000 7f8c85868700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c80151cc0 0x7f8c800ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:59.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.366+0000 7f8c85868700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c800c6cf0 con 0x7f8c800b00b0 2023-03-03T13:27:59.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.366+0000 7f8c85868700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c801548f0 con 0x7f8c800cac10 2023-03-03T13:27:59.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.366+0000 7f8c85868700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c80152090 con 0x7f8c80151cc0 2023-03-03T13:27:59.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.366+0000 7f8c7e7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8c800b00b0 0x7f8c800c7050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38716/0 (socket says 172.21.15.7:38716) 2023-03-03T13:27:59.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.366+0000 7f8c7e7fc700 1 -- 172.21.15.7:0/3341773119 learned_addr learned my addr 172.21.15.7:0/3341773119 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:27:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7effd700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c80151cc0 0x7f8c800ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431485914 0 0) 0x7f8c800c6cf0 con 0x7f8c800b00b0 2023-03-03T13:27:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c680035b0 con 0x7f8c800b00b0 2023-03-03T13:27:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1367793144 0 0) 0x7f8c680035b0 con 0x7f8c800b00b0 2023-03-03T13:27:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c80151cc0 msgr2=0x7f8c800ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:27:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c80151cc0 0x7f8c800ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 >> v1:172.21.15.143:6789/0 conn(0x7f8c800cac10 legacy=0x7f8c8015ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c80154bd0 con 0x7f8c800b00b0 2023-03-03T13:27:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8c70001670 con 0x7f8c800b00b0 2023-03-03T13:27:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 >> v1:172.21.15.7:6789/0 conn(0x7f8c800b00b0 legacy=0x7f8c800c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c68003aa0 0x7f8c68003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c68004850 0x7f8c68004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c80151cc0 0x7f8c68005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6800bac0 con 0x7f8c68004850 2023-03-03T13:27:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6800bc30 con 0x7f8c68003aa0 2023-03-03T13:27:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6800bdf0 con 0x7f8c80151cc0 2023-03-03T13:27:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7e7fc700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c68004850 0x7f8c68004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7effd700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c68003aa0 0x7f8c68003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.367+0000 7f8c7f7fe700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c80151cc0 0x7f8c68005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c7e7fc700 1 -- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c80151cc0 msgr2=0x7f8c68005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:27:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c7e7fc700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c80151cc0 0x7f8c68005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c7e7fc700 1 -- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c68003aa0 msgr2=0x7f8c68003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:27:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c7e7fc700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c68003aa0 0x7f8c68003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c7e7fc700 1 -- 172.21.15.7:0/3341773119 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c680035b0 con 0x7f8c68004850 2023-03-03T13:27:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c7e7fc700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c68004850 0x7f8c68004cc0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f8c800b04e0 tx=0x7f8c68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c708c3cd04bddce8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7000ce00 con 0x7f8c68004850 2023-03-03T13:27:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c70005960 con 0x7f8c68004850 2023-03-03T13:27:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c7dffb700 1 -- 172.21.15.7:0/3341773119 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70005d10 con 0x7f8c68004850 2023-03-03T13:27:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c85868700 1 -- 172.21.15.7:0/3341773119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c68004850 msgr2=0x7f8c68004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c85868700 1 --2- 172.21.15.7:0/3341773119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c68004850 0x7f8c68004cc0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f8c800b04e0 tx=0x7f8c68005880 comp rx=0 tx=0).stop 2023-03-03T13:27:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.368+0000 7f8c7f7fe700 1 -- 172.21.15.7:0/3341773119 reap_dead start 2023-03-03T13:27:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.369+0000 7f8c85868700 1 -- 172.21.15.7:0/3341773119 shutdown_connections 2023-03-03T13:27:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.369+0000 7f8c85868700 1 -- 172.21.15.7:0/3341773119 >> 172.21.15.7:0/3341773119 conn(0x7f8c8006b980 msgr2=0x7f8c80071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:27:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.369+0000 7f8c85868700 1 -- 172.21.15.7:0/3341773119 shutdown_connections 2023-03-03T13:27:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.369+0000 7f8c85868700 1 -- 172.21.15.7:0/3341773119 wait complete. 2023-03-03T13:27:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.370+0000 7f8c85868700 1 Processor -- start 2023-03-03T13:27:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.370+0000 7f8c85868700 1 -- start start 2023-03-03T13:27:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c85868700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c68004850 0x7f8c80151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c85868700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c800c3ac0 0x7f8c800c27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c85868700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c2d30 0x7f8c80147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c85868700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c80151750 con 0x7f8c800c2d30 2023-03-03T13:27:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c85868700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8014e640 con 0x7f8c800c3ac0 2023-03-03T13:27:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c85868700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c801501e0 con 0x7f8c68004850 2023-03-03T13:27:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c2d30 0x7f8c80147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c2d30 0x7f8c80147e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44752/0 (socket says 172.21.15.7:44752) 2023-03-03T13:27:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7f7fe700 1 -- 172.21.15.7:0/1074244560 learned_addr learned my addr 172.21.15.7:0/1074244560 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:27:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7e7fc700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c800c3ac0 0x7f8c800c27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7effd700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c68004850 0x7f8c80151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7f7fe700 1 -- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c68004850 msgr2=0x7f8c80151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7f7fe700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c68004850 0x7f8c80151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7f7fe700 1 -- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c800c3ac0 msgr2=0x7f8c800c27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7f7fe700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c800c3ac0 0x7f8c800c27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7f7fe700 1 -- 172.21.15.7:0/1074244560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c80155190 con 0x7f8c800c2d30 2023-03-03T13:27:59.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c7f7fe700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c2d30 0x7f8c80147e90 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f8c800c6020 tx=0x7f8c7800d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:59.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c67fff700 1 -- 172.21.15.7:0/1074244560 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7800dbc0 con 0x7f8c800c2d30 2023-03-03T13:27:59.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c67fff700 1 -- 172.21.15.7:0/1074244560 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7800dd20 con 0x7f8c800c2d30 2023-03-03T13:27:59.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c67fff700 1 -- 172.21.15.7:0/1074244560 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c78003c40 con 0x7f8c800c2d30 2023-03-03T13:27:59.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.371+0000 7f8c85868700 1 -- 172.21.15.7:0/1074244560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c80154eb0 con 0x7f8c800c2d30 2023-03-03T13:27:59.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.372+0000 7f8c85868700 1 -- 172.21.15.7:0/1074244560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c801545a0 con 0x7f8c800c2d30 2023-03-03T13:27:59.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.372+0000 7f8c85868700 1 -- 172.21.15.7:0/1074244560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c800c0440 con 0x7f8c800c2d30 2023-03-03T13:27:59.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.373+0000 7f8c67fff700 1 -- 172.21.15.7:0/1074244560 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c780050d0 con 0x7f8c800c2d30 2023-03-03T13:27:59.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.373+0000 7f8c67fff700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c6c0682f0 0x7f8c6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:27:59.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.373+0000 7f8c67fff700 1 -- 172.21.15.7:0/1074244560 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8c78086650 con 0x7f8c800c2d30 2023-03-03T13:27:59.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.374+0000 7f8c7effd700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c6c0682f0 0x7f8c6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:27:59.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.374+0000 7f8c7effd700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c6c0682f0 0x7f8c6c06a7a0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f8c8014a440 tx=0x7f8c74009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:27:59.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.378+0000 7f8c67fff700 1 -- 172.21.15.7:0/1074244560 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c78052900 con 0x7f8c800c2d30 2023-03-03T13:27:59.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.617+0000 7f8c85868700 1 -- 172.21.15.7:0/1074244560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c80074bb0 con 0x7f8c800c2d30 2023-03-03T13:27:59.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.617+0000 7f8c67fff700 1 -- 172.21.15.7:0/1074244560 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8c78055f20 con 0x7f8c800c2d30 2023-03-03T13:27:59.620 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:27:59.620 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:27:59.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.619+0000 7f8c85868700 1 -- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c6c0682f0 msgr2=0x7f8c6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.619+0000 7f8c85868700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c6c0682f0 0x7f8c6c06a7a0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f8c8014a440 tx=0x7f8c74009210 comp rx=0 tx=0).stop 2023-03-03T13:27:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.619+0000 7f8c85868700 1 -- 172.21.15.7:0/1074244560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c2d30 msgr2=0x7f8c80147e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:27:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.619+0000 7f8c85868700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c2d30 0x7f8c80147e90 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f8c800c6020 tx=0x7f8c7800d660 comp rx=0 tx=0).stop 2023-03-03T13:27:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.619+0000 7f8c85868700 1 -- 172.21.15.7:0/1074244560 shutdown_connections 2023-03-03T13:27:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.619+0000 7f8c85868700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c6c0682f0 0x7f8c6c06a7a0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.619+0000 7f8c85868700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c800c2d30 0x7f8c80147e90 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.619+0000 7f8c85868700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c68004850 0x7f8c80151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.619+0000 7f8c85868700 1 --2- 172.21.15.7:0/1074244560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c800c3ac0 0x7f8c800c27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:27:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.619+0000 7f8c85868700 1 -- 172.21.15.7:0/1074244560 >> 172.21.15.7:0/1074244560 conn(0x7f8c8006b980 msgr2=0x7f8c800c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:27:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.620+0000 7f8c85868700 1 -- 172.21.15.7:0/1074244560 shutdown_connections 2023-03-03T13:27:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:27:59.620+0000 7f8c85868700 1 -- 172.21.15.7:0/1074244560 wait complete. 2023-03-03T13:27:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:27:59.633 INFO:tasks.mds_thrash.fs.[b]:waiting for 20 secs before thrashing 2023-03-03T13:28:12.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:12.355 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:19.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:28:19.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.771+0000 7f8c9a63f700 1 Processor -- start 2023-03-03T13:28:19.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.771+0000 7f8c9a63f700 1 -- start start 2023-03-03T13:28:19.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c9a63f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8c940aac00 0x7f8c940c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:19.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c9a63f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94071c30 0x7f8c940beb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:19.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c9a63f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c940713c0 con 0x7f8c940bf0a0 2023-03-03T13:28:19.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c9a63f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c94152c70 con 0x7f8c940aac00 2023-03-03T13:28:19.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c9a63f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c940c4930 con 0x7f8c94071c30 2023-03-03T13:28:19.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c98bdc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8c940bf0a0 0x7f8c94155950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38754/0 (socket says 172.21.15.7:38754) 2023-03-03T13:28:19.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c98bdc700 1 -- 172.21.15.7:0/2529778686 learned_addr learned my addr 172.21.15.7:0/2529778686 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:28:19.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c937fe700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94071c30 0x7f8c940beb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:19.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c93fff700 1 --2- 172.21.15.7:0/2529778686 >> v2:172.21.15.143:3300/0 conn(0x7f8c940aac00 0x7f8c940c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:19.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078017051 0 0) 0x7f8c940713c0 con 0x7f8c940bf0a0 2023-03-03T13:28:19.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c800035b0 con 0x7f8c940bf0a0 2023-03-03T13:28:19.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2863804647 0 0) 0x7f8c800035b0 con 0x7f8c940bf0a0 2023-03-03T13:28:19.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94071c30 msgr2=0x7f8c940beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:28:19.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c92ffd700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94071c30 0x7f8c940beb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:19.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 >> v2:172.21.15.143:3300/0 conn(0x7f8c940aac00 msgr2=0x7f8c940c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:28:19.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c92ffd700 1 --2- 172.21.15.7:0/2529778686 >> v2:172.21.15.143:3300/0 conn(0x7f8c940aac00 0x7f8c940c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:19.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.772+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c94152360 con 0x7f8c940bf0a0 2023-03-03T13:28:19.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8c8c0019f0 con 0x7f8c940bf0a0 2023-03-03T13:28:19.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 >> v1:172.21.15.7:6789/0 conn(0x7f8c940bf0a0 legacy=0x7f8c94155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:19.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c80003a70 0x7f8c80003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:19.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c80004850 0x7f8c80004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:19.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94071c30 0x7f8c80005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:19.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c94152c70 con 0x7f8c80003a70 2023-03-03T13:28:19.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c940c4930 con 0x7f8c80004850 2023-03-03T13:28:19.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c8000bb10 con 0x7f8c94071c30 2023-03-03T13:28:19.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c93fff700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c80003a70 0x7f8c80003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:19.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c937fe700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c80004850 0x7f8c80004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:19.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c98bdc700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94071c30 0x7f8c80005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:19.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c93fff700 1 -- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94071c30 msgr2=0x7f8c80005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:28:19.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c93fff700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94071c30 0x7f8c80005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:19.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c93fff700 1 -- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c80004850 msgr2=0x7f8c80004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:28:19.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c93fff700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c80004850 0x7f8c80004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:19.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c93fff700 1 -- 172.21.15.7:0/2529778686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c800035b0 con 0x7f8c80003a70 2023-03-03T13:28:19.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c98bdc700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94071c30 0x7f8c80005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:28:19.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c937fe700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c80004850 0x7f8c80004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:28:19.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c93fff700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c80003a70 0x7f8c80003e80 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009b90 tx=0x7f8c80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930bb15afd2537ae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:19.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c00e6c0 con 0x7f8c80003a70 2023-03-03T13:28:19.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7c00e820 con 0x7f8c80003a70 2023-03-03T13:28:19.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.773+0000 7f8c92ffd700 1 -- 172.21.15.7:0/2529778686 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c00eb90 con 0x7f8c80003a70 2023-03-03T13:28:19.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.774+0000 7f8c9a63f700 1 -- 172.21.15.7:0/2529778686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c80003a70 msgr2=0x7f8c80003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:19.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.774+0000 7f8c9a63f700 1 --2- 172.21.15.7:0/2529778686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c80003a70 0x7f8c80003e80 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009b90 tx=0x7f8c80005880 comp rx=0 tx=0).stop 2023-03-03T13:28:19.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.774+0000 7f8c98bdc700 1 -- 172.21.15.7:0/2529778686 reap_dead start 2023-03-03T13:28:19.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.774+0000 7f8c9a63f700 1 -- 172.21.15.7:0/2529778686 shutdown_connections 2023-03-03T13:28:19.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.774+0000 7f8c9a63f700 1 -- 172.21.15.7:0/2529778686 >> 172.21.15.7:0/2529778686 conn(0x7f8c940736f0 msgr2=0x7f8c94073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:28:19.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.774+0000 7f8c9a63f700 1 -- 172.21.15.7:0/2529778686 shutdown_connections 2023-03-03T13:28:19.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.774+0000 7f8c9a63f700 1 -- 172.21.15.7:0/2529778686 wait complete. 2023-03-03T13:28:19.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.775+0000 7f8c9a63f700 1 Processor -- start 2023-03-03T13:28:19.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.775+0000 7f8c9a63f700 1 -- start start 2023-03-03T13:28:19.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c9a63f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c80003a70 0x7f8c940acf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:19.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c9a63f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94155130 0x7f8c94155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:19.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c9a63f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c941eacf0 0x7f8c941eb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:19.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c9a63f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c940c21f0 con 0x7f8c941eacf0 2023-03-03T13:28:19.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c9a63f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c940ab7d0 con 0x7f8c80003a70 2023-03-03T13:28:19.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c9a63f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c94155ac0 con 0x7f8c94155130 2023-03-03T13:28:19.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c98bdc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c941eacf0 0x7f8c941eb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:19.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c98bdc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c941eacf0 0x7f8c941eb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44790/0 (socket says 172.21.15.7:44790) 2023-03-03T13:28:19.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c98bdc700 1 -- 172.21.15.7:0/1204962894 learned_addr learned my addr 172.21.15.7:0/1204962894 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:28:19.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c937fe700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94155130 0x7f8c94155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:19.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c93fff700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c80003a70 0x7f8c940acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:19.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c98bdc700 1 -- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94155130 msgr2=0x7f8c94155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:19.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c98bdc700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94155130 0x7f8c94155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:19.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c98bdc700 1 -- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c80003a70 msgr2=0x7f8c940acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:19.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c98bdc700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c80003a70 0x7f8c940acf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:19.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.776+0000 7f8c98bdc700 1 -- 172.21.15.7:0/1204962894 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c94152620 con 0x7f8c941eacf0 2023-03-03T13:28:19.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.777+0000 7f8c98bdc700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c941eacf0 0x7f8c941eb160 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f8c940bf470 tx=0x7f8c8c007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.777+0000 7f8c90ff9700 1 -- 172.21.15.7:0/1204962894 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c017070 con 0x7f8c941eacf0 2023-03-03T13:28:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.777+0000 7f8c90ff9700 1 -- 172.21.15.7:0/1204962894 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8c00ebb0 con 0x7f8c941eacf0 2023-03-03T13:28:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.777+0000 7f8c9a63f700 1 -- 172.21.15.7:0/1204962894 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c940c4b80 con 0x7f8c941eacf0 2023-03-03T13:28:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.777+0000 7f8c90ff9700 1 -- 172.21.15.7:0/1204962894 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c00ed60 con 0x7f8c941eacf0 2023-03-03T13:28:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.777+0000 7f8c9a63f700 1 -- 172.21.15.7:0/1204962894 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c94152900 con 0x7f8c941eacf0 2023-03-03T13:28:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.778+0000 7f8c9a63f700 1 -- 172.21.15.7:0/1204962894 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c94074bb0 con 0x7f8c941eacf0 2023-03-03T13:28:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.778+0000 7f8c90ff9700 1 -- 172.21.15.7:0/1204962894 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c8c013420 con 0x7f8c941eacf0 2023-03-03T13:28:19.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.779+0000 7f8c90ff9700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c740682f0 0x7f8c7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:19.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.779+0000 7f8c90ff9700 1 -- 172.21.15.7:0/1204962894 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8c8c085ef0 con 0x7f8c941eacf0 2023-03-03T13:28:19.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.779+0000 7f8c93fff700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c740682f0 0x7f8c7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:19.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.780+0000 7f8c93fff700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c740682f0 0x7f8c7406a7a0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0090b0 tx=0x7f8c7c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:19.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:19.783+0000 7f8c90ff9700 1 -- 172.21.15.7:0/1204962894 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c8c052140 con 0x7f8c941eacf0 2023-03-03T13:28:20.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.020+0000 7f8c9a63f700 1 -- 172.21.15.7:0/1204962894 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c940ab910 con 0x7f8c941eacf0 2023-03-03T13:28:20.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.021+0000 7f8c90ff9700 1 -- 172.21.15.7:0/1204962894 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8c8c055760 con 0x7f8c941eacf0 2023-03-03T13:28:20.023 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:28:20.023 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:28:20.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.022+0000 7f8c9a63f700 1 -- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c740682f0 msgr2=0x7f8c7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:20.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.022+0000 7f8c9a63f700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c740682f0 0x7f8c7406a7a0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c0090b0 tx=0x7f8c7c008030 comp rx=0 tx=0).stop 2023-03-03T13:28:20.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.022+0000 7f8c9a63f700 1 -- 172.21.15.7:0/1204962894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c941eacf0 msgr2=0x7f8c941eb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:20.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.022+0000 7f8c9a63f700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c941eacf0 0x7f8c941eb160 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f8c940bf470 tx=0x7f8c8c007c90 comp rx=0 tx=0).stop 2023-03-03T13:28:20.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.022+0000 7f8c9a63f700 1 -- 172.21.15.7:0/1204962894 shutdown_connections 2023-03-03T13:28:20.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.022+0000 7f8c9a63f700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c740682f0 0x7f8c7406a7a0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:20.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.022+0000 7f8c9a63f700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c941eacf0 0x7f8c941eb160 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:20.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.022+0000 7f8c9a63f700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c80003a70 0x7f8c940acf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:20.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.022+0000 7f8c9a63f700 1 --2- 172.21.15.7:0/1204962894 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c94155130 0x7f8c94155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:20.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.022+0000 7f8c9a63f700 1 -- 172.21.15.7:0/1204962894 >> 172.21.15.7:0/1204962894 conn(0x7f8c940736f0 msgr2=0x7f8c940c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:28:20.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.023+0000 7f8c9a63f700 1 -- 172.21.15.7:0/1204962894 shutdown_connections 2023-03-03T13:28:20.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:20.023+0000 7f8c9a63f700 1 -- 172.21.15.7:0/1204962894 wait complete. 2023-03-03T13:28:20.026 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:28:20.036 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before thrashing 2023-03-03T13:28:27.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:28:27.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.171+0000 7f428920e700 1 Processor -- start 2023-03-03T13:28:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.171+0000 7f428920e700 1 -- start start 2023-03-03T13:28:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.171+0000 7f428920e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f428406e020 0x7f4284071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.171+0000 7f428920e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4284072520 0x7f4284151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428920e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4284154910 con 0x7f4284155390 2023-03-03T13:28:27.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428920e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4284070b90 con 0x7f4284072520 2023-03-03T13:28:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428920e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4284151320 con 0x7f428406e020 2023-03-03T13:28:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428359e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4284072520 0x7f4284151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428359e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4284072520 0x7f4284151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44808/0 (socket says 172.21.15.7:44808) 2023-03-03T13:28:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428359e700 1 -- 172.21.15.7:0/2412846304 learned_addr learned my addr 172.21.15.7:0/2412846304 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:28:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428259c700 1 --2- 172.21.15.7:0/2412846304 >> v2:172.21.15.143:3300/0 conn(0x7f428406e020 0x7f4284071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428359e700 1 -- 172.21.15.7:0/2412846304 >> v2:172.21.15.143:3300/0 conn(0x7f428406e020 msgr2=0x7f4284071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:28:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428359e700 1 --2- 172.21.15.7:0/2412846304 >> v2:172.21.15.143:3300/0 conn(0x7f428406e020 0x7f4284071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428359e700 1 -- 172.21.15.7:0/2412846304 >> v1:172.21.15.143:6789/0 conn(0x7f4284155390 legacy=0x7f42840c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428359e700 1 -- 172.21.15.7:0/2412846304 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42840c4540 con 0x7f4284072520 2023-03-03T13:28:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f428359e700 1 --2- 172.21.15.7:0/2412846304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 0x7f4284151770 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f42840ac9e0 tx=0x7f427c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92959f39efbd16ab server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f4281d9b700 1 -- 172.21.15.7:0/2412846304 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c010070 con 0x7f4284072520 2023-03-03T13:28:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f4281d9b700 1 -- 172.21.15.7:0/2412846304 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c00dd90 con 0x7f4284072520 2023-03-03T13:28:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.172+0000 7f4281d9b700 1 -- 172.21.15.7:0/2412846304 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c00b310 con 0x7f4284072520 2023-03-03T13:28:27.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.173+0000 7f428920e700 1 -- 172.21.15.7:0/2412846304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 msgr2=0x7f4284151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:27.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.173+0000 7f428920e700 1 --2- 172.21.15.7:0/2412846304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 0x7f4284151770 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f42840ac9e0 tx=0x7f427c00d430 comp rx=0 tx=0).stop 2023-03-03T13:28:27.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.173+0000 7f428920e700 1 -- 172.21.15.7:0/2412846304 shutdown_connections 2023-03-03T13:28:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.173+0000 7f428920e700 1 --2- 172.21.15.7:0/2412846304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 0x7f4284151770 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.173+0000 7f428920e700 1 --2- 172.21.15.7:0/2412846304 >> v2:172.21.15.143:3300/0 conn(0x7f428406e020 0x7f4284071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.173+0000 7f428920e700 1 -- 172.21.15.7:0/2412846304 >> 172.21.15.7:0/2412846304 conn(0x7f428406c110 msgr2=0x7f428406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:28:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.173+0000 7f428920e700 1 -- 172.21.15.7:0/2412846304 shutdown_connections 2023-03-03T13:28:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.173+0000 7f428920e700 1 -- 172.21.15.7:0/2412846304 wait complete. 2023-03-03T13:28:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.174+0000 7f428920e700 1 Processor -- start 2023-03-03T13:28:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.174+0000 7f428920e700 1 -- start start 2023-03-03T13:28:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f428920e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f428406e020 0x7f42840c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f428920e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 0x7f4284148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f428920e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4284155390 0x7f4284148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f428920e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42841492c0 con 0x7f4284072520 2023-03-03T13:28:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f428920e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42841dbb10 con 0x7f4284155390 2023-03-03T13:28:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f428920e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42840ac7e0 con 0x7f428406e020 2023-03-03T13:28:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f428259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 0x7f4284148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:27.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f428259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 0x7f4284148840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44812/0 (socket says 172.21.15.7:44812) 2023-03-03T13:28:27.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f428259c700 1 -- 172.21.15.7:0/3595315242 learned_addr learned my addr 172.21.15.7:0/3595315242 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:28:27.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f4282d9d700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f428406e020 0x7f42840c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:27.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.175+0000 7f428359e700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4284155390 0x7f4284148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:27.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f428259c700 1 -- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f428406e020 msgr2=0x7f42840c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:27.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f428259c700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f428406e020 0x7f42840c64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:27.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f428259c700 1 -- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4284155390 msgr2=0x7f4284148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:27.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f428259c700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4284155390 0x7f4284148d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:27.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f428259c700 1 -- 172.21.15.7:0/3595315242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c00bd40 con 0x7f4284072520 2023-03-03T13:28:27.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f428259c700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 0x7f4284148840 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f427800e810 tx=0x7f427800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:27.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f426b7fe700 1 -- 172.21.15.7:0/3595315242 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427800c070 con 0x7f4284072520 2023-03-03T13:28:27.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f426b7fe700 1 -- 172.21.15.7:0/3595315242 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4278013610 con 0x7f4284072520 2023-03-03T13:28:27.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f428920e700 1 -- 172.21.15.7:0/3595315242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4284070540 con 0x7f4284072520 2023-03-03T13:28:27.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f426b7fe700 1 -- 172.21.15.7:0/3595315242 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42780139c0 con 0x7f4284072520 2023-03-03T13:28:27.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.176+0000 7f428920e700 1 -- 172.21.15.7:0/3595315242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42840702a0 con 0x7f4284072520 2023-03-03T13:28:27.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.177+0000 7f428920e700 1 -- 172.21.15.7:0/3595315242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4284070820 con 0x7f4284072520 2023-03-03T13:28:27.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.178+0000 7f426b7fe700 1 -- 172.21.15.7:0/3595315242 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4278013b20 con 0x7f4284072520 2023-03-03T13:28:27.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.178+0000 7f426b7fe700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4270068320 0x7f427006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:27.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.178+0000 7f426b7fe700 1 -- 172.21.15.7:0/3595315242 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4278085dc0 con 0x7f4284072520 2023-03-03T13:28:27.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.178+0000 7f4282d9d700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4270068320 0x7f427006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:27.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.179+0000 7f4282d9d700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4270068320 0x7f427006a7d0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f4274007fd0 tx=0x7f4274009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:27.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.182+0000 7f426b7fe700 1 -- 172.21.15.7:0/3595315242 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4278052160 con 0x7f4284072520 2023-03-03T13:28:27.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.419+0000 7f428920e700 1 -- 172.21.15.7:0/3595315242 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42840c25b0 con 0x7f4284072520 2023-03-03T13:28:27.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.420+0000 7f426b7fe700 1 -- 172.21.15.7:0/3595315242 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4278017090 con 0x7f4284072520 2023-03-03T13:28:27.422 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:28:27.422 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:28:27.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 -- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4270068320 msgr2=0x7f427006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:27.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4270068320 0x7f427006a7d0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f4274007fd0 tx=0x7f4274009040 comp rx=0 tx=0).stop 2023-03-03T13:28:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 -- 172.21.15.7:0/3595315242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 msgr2=0x7f4284148840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 0x7f4284148840 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f427800e810 tx=0x7f427800eb20 comp rx=0 tx=0).stop 2023-03-03T13:28:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 -- 172.21.15.7:0/3595315242 shutdown_connections 2023-03-03T13:28:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4284155390 0x7f4284148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f428406e020 0x7f42840c64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4270068320 0x7f427006a7d0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 --2- 172.21.15.7:0/3595315242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284072520 0x7f4284148840 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 -- 172.21.15.7:0/3595315242 >> 172.21.15.7:0/3595315242 conn(0x7f428406c110 msgr2=0x7f4284074100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:28:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 -- 172.21.15.7:0/3595315242 shutdown_connections 2023-03-03T13:28:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:27.422+0000 7f428920e700 1 -- 172.21.15.7:0/3595315242 wait complete. 2023-03-03T13:28:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:28:27.436 INFO:tasks.mds_thrash.fs.[b]:waiting for 26 secs before thrashing 2023-03-03T13:28:42.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:42.419 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:28:46.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:28:46.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.673+0000 7fba2e9c2700 1 Processor -- start 2023-03-03T13:28:46.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.673+0000 7fba2e9c2700 1 -- start start 2023-03-03T13:28:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.674+0000 7fba2e9c2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fba280aa030 0x7fba281583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.674+0000 7fba2e9c2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba281588f0 con 0x7fba2814b020 2023-03-03T13:28:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.674+0000 7fba2e9c2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280728b0 con 0x7fba280aad20 2023-03-03T13:28:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.674+0000 7fba2e9c2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba28155070 con 0x7fba280aa030 2023-03-03T13:28:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.674+0000 7fba27fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fba2814b020 0x7fba280717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38814/0 (socket says 172.21.15.7:38814) 2023-03-03T13:28:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.674+0000 7fba27fff700 1 -- 172.21.15.7:0/4206225280 learned_addr learned my addr 172.21.15.7:0/4206225280 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:28:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba2cf5f700 1 --2- 172.21.15.7:0/4206225280 >> v2:172.21.15.143:3300/0 conn(0x7fba280aa030 0x7fba281583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba26ffd700 1 -- 172.21.15.7:0/4206225280 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670337695 0 0) 0x7fba281588f0 con 0x7fba2814b020 2023-03-03T13:28:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba26ffd700 1 -- 172.21.15.7:0/4206225280 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba100035b0 con 0x7fba2814b020 2023-03-03T13:28:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba2cf5f700 1 -- 172.21.15.7:0/4206225280 >> v1:172.21.15.143:6789/0 conn(0x7fba280aad20 legacy=0x7fba28157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba26ffd700 1 -- 172.21.15.7:0/4206225280 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2836412180 0 0) 0x7fba100035b0 con 0x7fba2814b020 2023-03-03T13:28:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba2cf5f700 1 -- 172.21.15.7:0/4206225280 >> v1:172.21.15.7:6789/0 conn(0x7fba2814b020 legacy=0x7fba280717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba2cf5f700 1 -- 172.21.15.7:0/4206225280 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2806f5b0 con 0x7fba280aa030 2023-03-03T13:28:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba2cf5f700 1 --2- 172.21.15.7:0/4206225280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba280aa030 0x7fba281583b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fba20009d90 tx=0x7fba2000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87a9af788bd9b908 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba26ffd700 1 -- 172.21.15.7:0/4206225280 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2000e070 con 0x7fba280aa030 2023-03-03T13:28:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba26ffd700 1 -- 172.21.15.7:0/4206225280 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba20003a20 con 0x7fba280aa030 2023-03-03T13:28:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.675+0000 7fba26ffd700 1 -- 172.21.15.7:0/4206225280 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18001670 con 0x7fba280aa030 2023-03-03T13:28:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.676+0000 7fba2e9c2700 1 -- 172.21.15.7:0/4206225280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba280aa030 msgr2=0x7fba281583b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.676+0000 7fba2e9c2700 1 --2- 172.21.15.7:0/4206225280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba280aa030 0x7fba281583b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fba20009d90 tx=0x7fba2000b330 comp rx=0 tx=0).stop 2023-03-03T13:28:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.676+0000 7fba2e9c2700 1 -- 172.21.15.7:0/4206225280 shutdown_connections 2023-03-03T13:28:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.676+0000 7fba2e9c2700 1 --2- 172.21.15.7:0/4206225280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba280aa030 0x7fba281583b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.676+0000 7fba2e9c2700 1 -- 172.21.15.7:0/4206225280 >> 172.21.15.7:0/4206225280 conn(0x7fba280c8900 msgr2=0x7fba280c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:28:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.676+0000 7fba2e9c2700 1 -- 172.21.15.7:0/4206225280 shutdown_connections 2023-03-03T13:28:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.676+0000 7fba2e9c2700 1 -- 172.21.15.7:0/4206225280 wait complete. 2023-03-03T13:28:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.677+0000 7fba2e9c2700 1 Processor -- start 2023-03-03T13:28:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.677+0000 7fba2e9c2700 1 -- start start 2023-03-03T13:28:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba2e9c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba280aa030 0x7fba281df060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba2e9c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba280aad20 0x7fba281dfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba2e9c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba2814b020 0x7fba2814d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba2e9c2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba28153f20 con 0x7fba280aad20 2023-03-03T13:28:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba2e9c2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba281df5a0 con 0x7fba2814b020 2023-03-03T13:28:46.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba2e9c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba28154170 con 0x7fba280aa030 2023-03-03T13:28:46.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba277fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba280aad20 0x7fba281dfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:46.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba277fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba280aad20 0x7fba281dfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44842/0 (socket says 172.21.15.7:44842) 2023-03-03T13:28:46.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba277fe700 1 -- 172.21.15.7:0/3548669244 learned_addr learned my addr 172.21.15.7:0/3548669244 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:28:46.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba2cf5f700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba2814b020 0x7fba2814d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:46.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.678+0000 7fba27fff700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba280aa030 0x7fba281df060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:46.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba277fe700 1 -- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba280aa030 msgr2=0x7fba281df060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:46.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba277fe700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba280aa030 0x7fba281df060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:46.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba277fe700 1 -- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba2814b020 msgr2=0x7fba2814d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:46.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba277fe700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba2814b020 0x7fba2814d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:46.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba277fe700 1 -- 172.21.15.7:0/3548669244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20009a70 con 0x7fba280aad20 2023-03-03T13:28:46.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba27fff700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba280aa030 0x7fba281df060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:28:46.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba277fe700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba280aad20 0x7fba281dfd50 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00c500 tx=0x7fba1c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:46.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba24ff9700 1 -- 172.21.15.7:0/3548669244 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c0054e0 con 0x7fba280aad20 2023-03-03T13:28:46.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba24ff9700 1 -- 172.21.15.7:0/3548669244 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1c003680 con 0x7fba280aad20 2023-03-03T13:28:46.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba24ff9700 1 -- 172.21.15.7:0/3548669244 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c003a30 con 0x7fba280aad20 2023-03-03T13:28:46.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba2e9c2700 1 -- 172.21.15.7:0/3548669244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2806fb70 con 0x7fba280aad20 2023-03-03T13:28:46.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.679+0000 7fba2e9c2700 1 -- 172.21.15.7:0/3548669244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2806f890 con 0x7fba280aad20 2023-03-03T13:28:46.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.680+0000 7fba2e9c2700 1 -- 172.21.15.7:0/3548669244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba281deeb0 con 0x7fba280aad20 2023-03-03T13:28:46.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.680+0000 7fba24ff9700 1 -- 172.21.15.7:0/3548669244 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fba1c019030 con 0x7fba280aad20 2023-03-03T13:28:46.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.681+0000 7fba24ff9700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba140682f0 0x7fba1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:46.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.681+0000 7fba24ff9700 1 -- 172.21.15.7:0/3548669244 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fba1c0854f0 con 0x7fba280aad20 2023-03-03T13:28:46.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.681+0000 7fba27fff700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba140682f0 0x7fba1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:46.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.682+0000 7fba27fff700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba140682f0 0x7fba1406a7a0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fba18001eb0 tx=0x7fba18001ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:46.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.685+0000 7fba24ff9700 1 -- 172.21.15.7:0/3548669244 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fba1c051890 con 0x7fba280aad20 2023-03-03T13:28:46.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.927+0000 7fba2e9c2700 1 -- 172.21.15.7:0/3548669244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba28155310 con 0x7fba280aad20 2023-03-03T13:28:46.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.927+0000 7fba24ff9700 1 -- 172.21.15.7:0/3548669244 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fba1c054eb0 con 0x7fba280aad20 2023-03-03T13:28:46.930 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:28:46.930 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:28:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.929+0000 7fba2e9c2700 1 -- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba140682f0 msgr2=0x7fba1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:46.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.929+0000 7fba2e9c2700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba140682f0 0x7fba1406a7a0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fba18001eb0 tx=0x7fba18001ca0 comp rx=0 tx=0).stop 2023-03-03T13:28:46.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.929+0000 7fba2e9c2700 1 -- 172.21.15.7:0/3548669244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba280aad20 msgr2=0x7fba281dfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:46.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.929+0000 7fba2e9c2700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba280aad20 0x7fba281dfd50 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00c500 tx=0x7fba1c00c810 comp rx=0 tx=0).stop 2023-03-03T13:28:46.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.929+0000 7fba2e9c2700 1 -- 172.21.15.7:0/3548669244 shutdown_connections 2023-03-03T13:28:46.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.929+0000 7fba2e9c2700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba2814b020 0x7fba2814d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:46.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.929+0000 7fba2e9c2700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba280aa030 0x7fba281df060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:46.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.929+0000 7fba2e9c2700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba140682f0 0x7fba1406a7a0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.929+0000 7fba2e9c2700 1 --2- 172.21.15.7:0/3548669244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba280aad20 0x7fba281dfd50 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.929+0000 7fba2e9c2700 1 -- 172.21.15.7:0/3548669244 >> 172.21.15.7:0/3548669244 conn(0x7fba280c8900 msgr2=0x7fba280ac940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:28:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.930+0000 7fba2e9c2700 1 -- 172.21.15.7:0/3548669244 shutdown_connections 2023-03-03T13:28:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:46.930+0000 7fba2e9c2700 1 -- 172.21.15.7:0/3548669244 wait complete. 2023-03-03T13:28:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:28:46.944 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-03-03T13:28:53.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:28:53.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.574+0000 7f97cac4a700 1 Processor -- start 2023-03-03T13:28:53.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.574+0000 7f97cac4a700 1 -- start start 2023-03-03T13:28:53.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97cac4a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f97c40ac220 0x7f97c40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97cac4a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97c4073c30 0x7f97c4152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97cac4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ace30 0x7f97c4152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97cac4a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c4071e30 con 0x7f97c4073c30 2023-03-03T13:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97cac4a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c4152ed0 con 0x7f97c40ac220 2023-03-03T13:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97cac4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40ac900 con 0x7f97c40ace30 2023-03-03T13:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97c4073c30 0x7f97c4152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97c4073c30 0x7f97c4152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44850/0 (socket says 172.21.15.7:44850) 2023-03-03T13:28:53.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c3fff700 1 -- 172.21.15.7:0/3584778042 learned_addr learned my addr 172.21.15.7:0/3584778042 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:28:53.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c89e6700 1 --2- 172.21.15.7:0/3584778042 >> v2:172.21.15.143:3300/0 conn(0x7f97c40ac220 0x7f97c40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:53.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c91e7700 1 --2- 172.21.15.7:0/3584778042 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ace30 0x7f97c4152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:53.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c3fff700 1 -- 172.21.15.7:0/3584778042 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ace30 msgr2=0x7f97c4152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:28:53.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c3fff700 1 --2- 172.21.15.7:0/3584778042 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ace30 0x7f97c4152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c3fff700 1 -- 172.21.15.7:0/3584778042 >> v2:172.21.15.143:3300/0 conn(0x7f97c40ac220 msgr2=0x7f97c40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:28:53.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c3fff700 1 --2- 172.21.15.7:0/3584778042 >> v2:172.21.15.143:3300/0 conn(0x7f97c40ac220 0x7f97c40736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c3fff700 1 -- 172.21.15.7:0/3584778042 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c40acb50 con 0x7f97c4073c30 2023-03-03T13:28:53.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c91e7700 1 --2- 172.21.15.7:0/3584778042 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ace30 0x7f97c4152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:28:53.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c89e6700 1 --2- 172.21.15.7:0/3584778042 >> v2:172.21.15.143:3300/0 conn(0x7f97c40ac220 0x7f97c40736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:28:53.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.575+0000 7f97c3fff700 1 --2- 172.21.15.7:0/3584778042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c4073c30 0x7f97c4152450 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f97c406fbf0 tx=0x7f97b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55924d3899cf9a34 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:53.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97c37fe700 1 -- 172.21.15.7:0/3584778042 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b000b870 con 0x7f97c4073c30 2023-03-03T13:28:53.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97c37fe700 1 -- 172.21.15.7:0/3584778042 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b000b9d0 con 0x7f97c4073c30 2023-03-03T13:28:53.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97c37fe700 1 -- 172.21.15.7:0/3584778042 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b000bd60 con 0x7f97c4073c30 2023-03-03T13:28:53.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97cac4a700 1 -- 172.21.15.7:0/3584778042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c4073c30 msgr2=0x7f97c4152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:53.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97cac4a700 1 --2- 172.21.15.7:0/3584778042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c4073c30 0x7f97c4152450 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f97c406fbf0 tx=0x7f97b0009bd0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97cac4a700 1 -- 172.21.15.7:0/3584778042 shutdown_connections 2023-03-03T13:28:53.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97cac4a700 1 --2- 172.21.15.7:0/3584778042 >> v2:172.21.15.143:3300/0 conn(0x7f97c40ac220 0x7f97c40736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97cac4a700 1 --2- 172.21.15.7:0/3584778042 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ace30 0x7f97c4152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97cac4a700 1 --2- 172.21.15.7:0/3584778042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c4073c30 0x7f97c4152450 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97cac4a700 1 -- 172.21.15.7:0/3584778042 >> 172.21.15.7:0/3584778042 conn(0x7f97c40a94d0 msgr2=0x7f97c40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:28:53.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97cac4a700 1 -- 172.21.15.7:0/3584778042 shutdown_connections 2023-03-03T13:28:53.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.576+0000 7f97cac4a700 1 -- 172.21.15.7:0/3584778042 wait complete. 2023-03-03T13:28:53.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.577+0000 7f97cac4a700 1 Processor -- start 2023-03-03T13:28:53.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.578+0000 7f97cac4a700 1 -- start start 2023-03-03T13:28:53.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.578+0000 7f97cac4a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c4073c30 0x7f97c40c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:53.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.578+0000 7f97cac4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ac220 0x7f97c41e8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:53.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.578+0000 7f97cac4a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c40ace30 0x7f97c41e9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:53.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.578+0000 7f97cac4a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c40c08d0 con 0x7f97c40ace30 2023-03-03T13:28:53.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.578+0000 7f97cac4a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c41e99b0 con 0x7f97c4073c30 2023-03-03T13:28:53.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.578+0000 7f97cac4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c41ebf20 con 0x7f97c40ac220 2023-03-03T13:28:53.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.578+0000 7f97c91e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c40ace30 0x7f97c41e9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:53.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c91e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c40ace30 0x7f97c41e9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44858/0 (socket says 172.21.15.7:44858) 2023-03-03T13:28:53.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c91e7700 1 -- 172.21.15.7:0/2483080349 learned_addr learned my addr 172.21.15.7:0/2483080349 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:28:53.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c89e6700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c4073c30 0x7f97c40c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:53.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c3fff700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ac220 0x7f97c41e8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:53.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c91e7700 1 -- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ac220 msgr2=0x7f97c41e8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:53.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c91e7700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ac220 0x7f97c41e8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c91e7700 1 -- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c4073c30 msgr2=0x7f97c40c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:53.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c91e7700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c4073c30 0x7f97c40c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c91e7700 1 -- 172.21.15.7:0/2483080349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b0009550 con 0x7f97c40ace30 2023-03-03T13:28:53.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c91e7700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c40ace30 0x7f97c41e9470 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00cef0 tx=0x7f97bc00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:53.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c17fa700 1 -- 172.21.15.7:0/2483080349 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc00ad00 con 0x7f97c40ace30 2023-03-03T13:28:53.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c17fa700 1 -- 172.21.15.7:0/2483080349 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc00ae60 con 0x7f97c40ace30 2023-03-03T13:28:53.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97cac4a700 1 -- 172.21.15.7:0/2483080349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c4155ca0 con 0x7f97c40ace30 2023-03-03T13:28:53.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97c17fa700 1 -- 172.21.15.7:0/2483080349 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc015680 con 0x7f97c40ace30 2023-03-03T13:28:53.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.579+0000 7f97cac4a700 1 -- 172.21.15.7:0/2483080349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c41559c0 con 0x7f97c40ace30 2023-03-03T13:28:53.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.580+0000 7f97cac4a700 1 -- 172.21.15.7:0/2483080349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c4155f80 con 0x7f97c40ace30 2023-03-03T13:28:53.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.581+0000 7f97c17fa700 1 -- 172.21.15.7:0/2483080349 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f97bc01c030 con 0x7f97c40ace30 2023-03-03T13:28:53.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.581+0000 7f97c17fa700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97a80682f0 0x7f97a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:28:53.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.581+0000 7f97c17fa700 1 -- 172.21.15.7:0/2483080349 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f97bc086420 con 0x7f97c40ace30 2023-03-03T13:28:53.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.581+0000 7f97c89e6700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97a80682f0 0x7f97a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:28:53.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.582+0000 7f97c89e6700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97a80682f0 0x7f97a806a7a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c940 tx=0x7f97b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:28:53.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.586+0000 7f97c17fa700 1 -- 172.21.15.7:0/2483080349 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f97bc0527c0 con 0x7f97c40ace30 2023-03-03T13:28:53.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.823+0000 7f97cac4a700 1 -- 172.21.15.7:0/2483080349 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97c4074bb0 con 0x7f97c40ace30 2023-03-03T13:28:53.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.824+0000 7f97c17fa700 1 -- 172.21.15.7:0/2483080349 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f97bc01a090 con 0x7f97c40ace30 2023-03-03T13:28:53.826 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:28:53.826 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:28:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 -- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97a80682f0 msgr2=0x7f97a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97a80682f0 0x7f97a806a7a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f97b800c940 tx=0x7f97b8008040 comp rx=0 tx=0).stop 2023-03-03T13:28:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 -- 172.21.15.7:0/2483080349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c40ace30 msgr2=0x7f97c41e9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:28:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c40ace30 0x7f97c41e9470 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00cef0 tx=0x7f97bc00cf20 comp rx=0 tx=0).stop 2023-03-03T13:28:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 -- 172.21.15.7:0/2483080349 shutdown_connections 2023-03-03T13:28:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97a80682f0 0x7f97a806a7a0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97c40ace30 0x7f97c41e9470 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97c4073c30 0x7f97c40c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 --2- 172.21.15.7:0/2483080349 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97c40ac220 0x7f97c41e8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:28:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 -- 172.21.15.7:0/2483080349 >> 172.21.15.7:0/2483080349 conn(0x7f97c40a94d0 msgr2=0x7f97c40ab1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:28:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 -- 172.21.15.7:0/2483080349 shutdown_connections 2023-03-03T13:28:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:28:53.826+0000 7f97cac4a700 1 -- 172.21.15.7:0/2483080349 wait complete. 2023-03-03T13:28:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:28:53.840 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2023-03-03T13:29:12.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:12.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:12.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:29:13.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.083+0000 7fbf7b470700 1 Processor -- start 2023-03-03T13:29:13.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.083+0000 7fbf7b470700 1 -- start start 2023-03-03T13:29:13.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf7b470700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf74071e80 0x7fbf74151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:13.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf7b470700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbf7414e900 0x7fbf7415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:13.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf7b470700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf741510d0 con 0x7fbf74072250 2023-03-03T13:29:13.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf7b470700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf740c3740 con 0x7fbf7414e900 2023-03-03T13:29:13.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf7b470700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74070670 con 0x7fbf74071e80 2023-03-03T13:29:13.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf79a0d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbf7414e900 0x7fbf7415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:13.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf78a0b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbf74072250 0x7fbf7414e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38856/0 (socket says 172.21.15.7:38856) 2023-03-03T13:29:13.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf78a0b700 1 -- 172.21.15.7:0/1082573494 learned_addr learned my addr 172.21.15.7:0/1082573494 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:29:13.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf7920c700 1 --2- 172.21.15.7:0/1082573494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf74071e80 0x7fbf74151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:13.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf79a0d700 1 -- 172.21.15.7:0/1082573494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf74071e80 msgr2=0x7fbf74151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:29:13.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf79a0d700 1 --2- 172.21.15.7:0/1082573494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf74071e80 0x7fbf74151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:13.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf79a0d700 1 -- 172.21.15.7:0/1082573494 >> v1:172.21.15.7:6789/0 conn(0x7fbf74072250 legacy=0x7fbf7414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:13.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf6bfff700 1 -- 172.21.15.7:0/1082573494 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826447651 0 0) 0x7fbf741510d0 con 0x7fbf74072250 2023-03-03T13:29:13.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf79a0d700 1 -- 172.21.15.7:0/1082573494 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf740c39e0 con 0x7fbf7414e900 2023-03-03T13:29:13.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.084+0000 7fbf79a0d700 1 --2- 172.21.15.7:0/1082573494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 0x7fbf7415ba50 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009d90 tx=0x7fbf6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e201a330aafeb77b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:29:13.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf6bfff700 1 -- 172.21.15.7:0/1082573494 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c00bd20 con 0x7fbf7414e900 2023-03-03T13:29:13.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf6bfff700 1 -- 172.21.15.7:0/1082573494 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c00be80 con 0x7fbf7414e900 2023-03-03T13:29:13.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf6bfff700 1 -- 172.21.15.7:0/1082573494 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c0116a0 con 0x7fbf7414e900 2023-03-03T13:29:13.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf7b470700 1 -- 172.21.15.7:0/1082573494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 msgr2=0x7fbf7415ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:13.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf7b470700 1 --2- 172.21.15.7:0/1082573494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 0x7fbf7415ba50 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c009d90 tx=0x7fbf6c00b330 comp rx=0 tx=0).stop 2023-03-03T13:29:13.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf7b470700 1 -- 172.21.15.7:0/1082573494 shutdown_connections 2023-03-03T13:29:13.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf7b470700 1 --2- 172.21.15.7:0/1082573494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 0x7fbf7415ba50 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:13.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf7b470700 1 --2- 172.21.15.7:0/1082573494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf74071e80 0x7fbf74151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:13.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf7b470700 1 -- 172.21.15.7:0/1082573494 >> 172.21.15.7:0/1082573494 conn(0x7fbf740a94d0 msgr2=0x7fbf740a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:29:13.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf7b470700 1 -- 172.21.15.7:0/1082573494 shutdown_connections 2023-03-03T13:29:13.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.085+0000 7fbf7b470700 1 -- 172.21.15.7:0/1082573494 wait complete. 2023-03-03T13:29:13.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.087+0000 7fbf7b470700 1 Processor -- start 2023-03-03T13:29:13.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.087+0000 7fbf7b470700 1 -- start start 2023-03-03T13:29:13.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf7b470700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf74071e80 0x7fbf741440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:13.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf7b470700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf74072250 0x7fbf74144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:13.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf7b470700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 0x7fbf7414a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:13.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf7b470700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74150840 con 0x7fbf7414e900 2023-03-03T13:29:13.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf7b470700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf74148df0 con 0x7fbf74072250 2023-03-03T13:29:13.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf7b470700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf741eae30 con 0x7fbf74071e80 2023-03-03T13:29:13.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf79a0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 0x7fbf7414a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:13.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf79a0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 0x7fbf7414a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44884/0 (socket says 172.21.15.7:44884) 2023-03-03T13:29:13.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf79a0d700 1 -- 172.21.15.7:0/333285940 learned_addr learned my addr 172.21.15.7:0/333285940 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:29:13.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf7920c700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf74071e80 0x7fbf741440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:13.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf78a0b700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf74072250 0x7fbf74144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:13.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf79a0d700 1 -- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf74071e80 msgr2=0x7fbf741440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:13.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf79a0d700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf74071e80 0x7fbf741440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:13.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf79a0d700 1 -- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf74072250 msgr2=0x7fbf74144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:13.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf79a0d700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf74072250 0x7fbf74144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:13.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf79a0d700 1 -- 172.21.15.7:0/333285940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c009a70 con 0x7fbf7414e900 2023-03-03T13:29:13.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.088+0000 7fbf79a0d700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 0x7fbf7414a700 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00b680 tx=0x7fbf6c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:29:13.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.089+0000 7fbf69ffb700 1 -- 172.21.15.7:0/333285940 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c0048b0 con 0x7fbf7414e900 2023-03-03T13:29:13.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.089+0000 7fbf69ffb700 1 -- 172.21.15.7:0/333285940 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c011460 con 0x7fbf7414e900 2023-03-03T13:29:13.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.089+0000 7fbf7b470700 1 -- 172.21.15.7:0/333285940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7406c9a0 con 0x7fbf7414e900 2023-03-03T13:29:13.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.089+0000 7fbf69ffb700 1 -- 172.21.15.7:0/333285940 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c011810 con 0x7fbf7414e900 2023-03-03T13:29:13.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.089+0000 7fbf7b470700 1 -- 172.21.15.7:0/333285940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf74151300 con 0x7fbf7414e900 2023-03-03T13:29:13.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.090+0000 7fbf7b470700 1 -- 172.21.15.7:0/333285940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7406cb30 con 0x7fbf7414e900 2023-03-03T13:29:13.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.090+0000 7fbf69ffb700 1 -- 172.21.15.7:0/333285940 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbf6c011970 con 0x7fbf7414e900 2023-03-03T13:29:13.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.091+0000 7fbf69ffb700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf580682f0 0x7fbf5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:13.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.091+0000 7fbf69ffb700 1 -- 172.21.15.7:0/333285940 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbf6c088110 con 0x7fbf7414e900 2023-03-03T13:29:13.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.091+0000 7fbf7920c700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf580682f0 0x7fbf5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:13.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.091+0000 7fbf7920c700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf580682f0 0x7fbf5806a7a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fbf70009be0 tx=0x7fbf70009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:29:13.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.095+0000 7fbf69ffb700 1 -- 172.21.15.7:0/333285940 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbf6c0543c0 con 0x7fbf7414e900 2023-03-03T13:29:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.334+0000 7fbf7b470700 1 -- 172.21.15.7:0/333285940 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf74074bb0 con 0x7fbf7414e900 2023-03-03T13:29:13.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.335+0000 7fbf69ffb700 1 -- 172.21.15.7:0/333285940 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbf6c0579e0 con 0x7fbf7414e900 2023-03-03T13:29:13.337 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:29:13.338 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:29:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 -- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf580682f0 msgr2=0x7fbf5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf580682f0 0x7fbf5806a7a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fbf70009be0 tx=0x7fbf70009210 comp rx=0 tx=0).stop 2023-03-03T13:29:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 -- 172.21.15.7:0/333285940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 msgr2=0x7fbf7414a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 0x7fbf7414a700 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00b680 tx=0x7fbf6c004b40 comp rx=0 tx=0).stop 2023-03-03T13:29:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 -- 172.21.15.7:0/333285940 shutdown_connections 2023-03-03T13:29:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf580682f0 0x7fbf5806a7a0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf7414e900 0x7fbf7414a700 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf74071e80 0x7fbf741440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 --2- 172.21.15.7:0/333285940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf74072250 0x7fbf74144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 -- 172.21.15.7:0/333285940 >> 172.21.15.7:0/333285940 conn(0x7fbf740a94d0 msgr2=0x7fbf740736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:29:13.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 -- 172.21.15.7:0/333285940 shutdown_connections 2023-03-03T13:29:13.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:13.337+0000 7fbf7b470700 1 -- 172.21.15.7:0/333285940 wait complete. 2023-03-03T13:29:13.341 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:29:13.351 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T13:29:20.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:29:20.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.977+0000 7fb6bb2c2700 1 Processor -- start 2023-03-03T13:29:20.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.978+0000 7fb6bb2c2700 1 -- start start 2023-03-03T13:29:20.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.978+0000 7fb6bb2c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b4155410 0x7fb6b40bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:20.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.978+0000 7fb6bb2c2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b40c08d0 con 0x7fb6b4151e80 2023-03-03T13:29:20.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.978+0000 7fb6bb2c2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b4154930 con 0x7fb6b40c5b00 2023-03-03T13:29:20.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.978+0000 7fb6bb2c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b4152250 con 0x7fb6b4155410 2023-03-03T13:29:20.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6b905e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb6b4151e80 0x7fb6b40c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38878/0 (socket says 172.21.15.7:38878) 2023-03-03T13:29:20.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6b905e700 1 -- 172.21.15.7:0/3628766765 learned_addr learned my addr 172.21.15.7:0/3628766765 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:29:20.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6b885d700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b4155410 0x7fb6b40bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:20.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171619493 0 0) 0x7fb6b40c08d0 con 0x7fb6b4151e80 2023-03-03T13:29:20.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a00035b0 con 0x7fb6b4151e80 2023-03-03T13:29:20.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1127821228 0 0) 0x7fb6a00035b0 con 0x7fb6b4151e80 2023-03-03T13:29:20.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b4155410 msgr2=0x7fb6b40bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:29:20.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b4155410 0x7fb6b40bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:20.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 >> v1:172.21.15.143:6789/0 conn(0x7fb6b40c5b00 legacy=0x7fb6b40bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:20.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b4154c10 con 0x7fb6b4151e80 2023-03-03T13:29:20.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb6b0001670 con 0x7fb6b4151e80 2023-03-03T13:29:20.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 >> v1:172.21.15.7:6789/0 conn(0x7fb6b4151e80 legacy=0x7fb6b40c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:20.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b4155410 0x7fb6a0003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:20.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6a0004020 0x7fb6a0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:20.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6a0004d90 0x7fb6a0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:20.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a000bb20 con 0x7fb6a0004d90 2023-03-03T13:29:20.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a000bc90 con 0x7fb6a0004020 2023-03-03T13:29:20.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a000be50 con 0x7fb6b4155410 2023-03-03T13:29:20.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.979+0000 7fb6b985f700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6a0004d90 0x7fb6a0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:20.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6b885d700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6a0004020 0x7fb6a0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:20.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6b905e700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b4155410 0x7fb6a0003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:20.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6b985f700 1 -- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b4155410 msgr2=0x7fb6a0003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:29:20.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6b985f700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b4155410 0x7fb6a0003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:20.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6b985f700 1 -- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6a0004020 msgr2=0x7fb6a0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:29:20.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6b985f700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6a0004020 0x7fb6a0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:20.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6b985f700 1 -- 172.21.15.7:0/3628766765 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a00035b0 con 0x7fb6a0004d90 2023-03-03T13:29:20.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6b985f700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6a0004d90 0x7fb6a0005200 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00c190 tx=0x7fb6a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68fb05124c296121 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:29:20.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac001db0 con 0x7fb6a0004d90 2023-03-03T13:29:20.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ac00ccd0 con 0x7fb6a0004d90 2023-03-03T13:29:20.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.980+0000 7fb6abfff700 1 -- 172.21.15.7:0/3628766765 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac003c40 con 0x7fb6a0004d90 2023-03-03T13:29:20.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.981+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/3628766765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6a0004d90 msgr2=0x7fb6a0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:20.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.981+0000 7fb6bb2c2700 1 --2- 172.21.15.7:0/3628766765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6a0004d90 0x7fb6a0005200 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00c190 tx=0x7fb6a0005880 comp rx=0 tx=0).stop 2023-03-03T13:29:20.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.981+0000 7fb6b985f700 1 -- 172.21.15.7:0/3628766765 reap_dead start 2023-03-03T13:29:20.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.981+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/3628766765 shutdown_connections 2023-03-03T13:29:20.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.981+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/3628766765 >> 172.21.15.7:0/3628766765 conn(0x7fb6b4071630 msgr2=0x7fb6b406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:29:20.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.981+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/3628766765 shutdown_connections 2023-03-03T13:29:20.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.981+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/3628766765 wait complete. 2023-03-03T13:29:20.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.982+0000 7fb6bb2c2700 1 Processor -- start 2023-03-03T13:29:20.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.982+0000 7fb6bb2c2700 1 -- start start 2023-03-03T13:29:20.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6bb2c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6b4158310 0x7fb6b41586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:20.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6bb2c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6b4157bf0 0x7fb6b4149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:20.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6bb2c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b414a0d0 0x7fb6b40c28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:20.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6bb2c2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b414a570 con 0x7fb6b4158310 2023-03-03T13:29:20.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6bb2c2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b41ea3b0 con 0x7fb6b4157bf0 2023-03-03T13:29:20.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6bb2c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b41eb380 con 0x7fb6b414a0d0 2023-03-03T13:29:20.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b905e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6b4158310 0x7fb6b41586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:20.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b905e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6b4158310 0x7fb6b41586e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44910/0 (socket says 172.21.15.7:44910) 2023-03-03T13:29:20.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b905e700 1 -- 172.21.15.7:0/2945569023 learned_addr learned my addr 172.21.15.7:0/2945569023 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:29:20.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b885d700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6b4157bf0 0x7fb6b4149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:20.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b985f700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b414a0d0 0x7fb6b40c28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:20.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b905e700 1 -- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b414a0d0 msgr2=0x7fb6b40c28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:20.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b905e700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b414a0d0 0x7fb6b40c28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:20.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b905e700 1 -- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6b4157bf0 msgr2=0x7fb6b4149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:20.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b905e700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6b4157bf0 0x7fb6b4149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:20.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b905e700 1 -- 172.21.15.7:0/2945569023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b40c5220 con 0x7fb6b4158310 2023-03-03T13:29:20.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b985f700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b414a0d0 0x7fb6b40c28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:29:20.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6b905e700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6b4158310 0x7fb6b41586e0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00048d0 tx=0x7fb6b0007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:29:20.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6a9ffb700 1 -- 172.21.15.7:0/2945569023 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b000f850 con 0x7fb6b4158310 2023-03-03T13:29:20.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6a9ffb700 1 -- 172.21.15.7:0/2945569023 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b000f9d0 con 0x7fb6b4158310 2023-03-03T13:29:20.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/2945569023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b4154ef0 con 0x7fb6b4158310 2023-03-03T13:29:20.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.983+0000 7fb6a9ffb700 1 -- 172.21.15.7:0/2945569023 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b000fd80 con 0x7fb6b4158310 2023-03-03T13:29:20.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.984+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/2945569023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b40c0530 con 0x7fb6b4158310 2023-03-03T13:29:20.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.984+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/2945569023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b4074bb0 con 0x7fb6b4158310 2023-03-03T13:29:20.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.985+0000 7fb6a9ffb700 1 -- 172.21.15.7:0/2945569023 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb6b001d460 con 0x7fb6b4158310 2023-03-03T13:29:20.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.985+0000 7fb6a9ffb700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6980682f0 0x7fb69806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:29:20.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.985+0000 7fb6a9ffb700 1 -- 172.21.15.7:0/2945569023 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb6b00863b0 con 0x7fb6b4158310 2023-03-03T13:29:20.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.986+0000 7fb6b885d700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6980682f0 0x7fb69806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:29:20.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.986+0000 7fb6b885d700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6980682f0 0x7fb69806a7a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4158000 tx=0x7fb6a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:29:20.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:20.990+0000 7fb6a9ffb700 1 -- 172.21.15.7:0/2945569023 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb6b00526f0 con 0x7fb6b4158310 2023-03-03T13:29:21.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.233+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/2945569023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6b40c4880 con 0x7fb6b4158310 2023-03-03T13:29:21.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.234+0000 7fb6a9ffb700 1 -- 172.21.15.7:0/2945569023 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb6b0055d10 con 0x7fb6b4158310 2023-03-03T13:29:21.236 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:29:21.236 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:29:21.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6980682f0 msgr2=0x7fb69806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:21.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6980682f0 0x7fb69806a7a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4158000 tx=0x7fb6a4008040 comp rx=0 tx=0).stop 2023-03-03T13:29:21.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/2945569023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6b4158310 msgr2=0x7fb6b41586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:29:21.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6b4158310 0x7fb6b41586e0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00048d0 tx=0x7fb6b0007620 comp rx=0 tx=0).stop 2023-03-03T13:29:21.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/2945569023 shutdown_connections 2023-03-03T13:29:21.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6b414a0d0 0x7fb6b40c28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:21.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6980682f0 0x7fb69806a7a0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:21.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6b4158310 0x7fb6b41586e0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:21.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 --2- 172.21.15.7:0/2945569023 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6b4157bf0 0x7fb6b4149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:29:21.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/2945569023 >> 172.21.15.7:0/2945569023 conn(0x7fb6b4071630 msgr2=0x7fb6b40caa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:29:21.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/2945569023 shutdown_connections 2023-03-03T13:29:21.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:29:21.236+0000 7fb6bb2c2700 1 -- 172.21.15.7:0/2945569023 wait complete. 2023-03-03T13:29:21.240 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:29:21.249 INFO:tasks.mds_thrash.fs.[b]:waiting for 112 secs before thrashing 2023-03-03T13:29:42.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:29:42.713 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T13:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T13:30:12.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:12.776 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:17.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:30:17.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.485+0000 7f479c8a6700 1 Processor -- start 2023-03-03T13:30:17.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.485+0000 7f479c8a6700 1 -- start start 2023-03-03T13:30:17.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.486+0000 7f479c8a6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47981530d0 0x7f4798156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:17.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.486+0000 7f479c8a6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47980c2f30 con 0x7f479814fda0 2023-03-03T13:30:17.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.486+0000 7f479c8a6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47980c31d0 con 0x7f479814f9d0 2023-03-03T13:30:17.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.486+0000 7f479c8a6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47981524a0 con 0x7f47981530d0 2023-03-03T13:30:17.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.486+0000 7f4795d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f479814fda0 0x7f47981529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38942/0 (socket says 172.21.15.7:38942) 2023-03-03T13:30:17.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.486+0000 7f4795d9b700 1 -- 172.21.15.7:0/3181051554 learned_addr learned my addr 172.21.15.7:0/3181051554 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:30:17.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.486+0000 7f4796d9d700 1 --2- 172.21.15.7:0/3181051554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47981530d0 0x7f4798156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:17.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f479559a700 1 -- 172.21.15.7:0/3181051554 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115720739 0 0) 0x7f47980c2f30 con 0x7f479814fda0 2023-03-03T13:30:17.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f479559a700 1 -- 172.21.15.7:0/3181051554 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47800035b0 con 0x7f479814fda0 2023-03-03T13:30:17.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f479559a700 1 -- 172.21.15.7:0/3181051554 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641647639 0 0) 0x7f47980c31d0 con 0x7f479814f9d0 2023-03-03T13:30:17.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f479559a700 1 -- 172.21.15.7:0/3181051554 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47980c2f30 con 0x7f479814f9d0 2023-03-03T13:30:17.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f479559a700 1 -- 172.21.15.7:0/3181051554 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 722599589 0 0) 0x7f47800035b0 con 0x7f479814fda0 2023-03-03T13:30:17.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f4796d9d700 1 -- 172.21.15.7:0/3181051554 >> v1:172.21.15.143:6789/0 conn(0x7f479814f9d0 legacy=0x7f4798071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:17.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f4796d9d700 1 -- 172.21.15.7:0/3181051554 >> v1:172.21.15.7:6789/0 conn(0x7f479814fda0 legacy=0x7f47981529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:17.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f4796d9d700 1 -- 172.21.15.7:0/3181051554 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47980c34b0 con 0x7f47981530d0 2023-03-03T13:30:17.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f4796d9d700 1 --2- 172.21.15.7:0/3181051554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47981530d0 0x7f4798156430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4790009dc0 tx=0x7f479000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9b91a70280ca5d7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:30:17.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f479559a700 1 -- 172.21.15.7:0/3181051554 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479000e070 con 0x7f47981530d0 2023-03-03T13:30:17.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f479559a700 1 -- 172.21.15.7:0/3181051554 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790003a20 con 0x7f47981530d0 2023-03-03T13:30:17.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.487+0000 7f479559a700 1 -- 172.21.15.7:0/3181051554 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c001670 con 0x7f47981530d0 2023-03-03T13:30:17.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.488+0000 7f479c8a6700 1 -- 172.21.15.7:0/3181051554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47981530d0 msgr2=0x7f4798156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:17.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.488+0000 7f479c8a6700 1 --2- 172.21.15.7:0/3181051554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47981530d0 0x7f4798156430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4790009dc0 tx=0x7f479000b330 comp rx=0 tx=0).stop 2023-03-03T13:30:17.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.488+0000 7f479c8a6700 1 -- 172.21.15.7:0/3181051554 shutdown_connections 2023-03-03T13:30:17.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.488+0000 7f479c8a6700 1 --2- 172.21.15.7:0/3181051554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47981530d0 0x7f4798156430 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:17.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.488+0000 7f479c8a6700 1 -- 172.21.15.7:0/3181051554 >> 172.21.15.7:0/3181051554 conn(0x7f47980c8900 msgr2=0x7f47980c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:30:17.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.488+0000 7f479c8a6700 1 -- 172.21.15.7:0/3181051554 shutdown_connections 2023-03-03T13:30:17.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.488+0000 7f479c8a6700 1 -- 172.21.15.7:0/3181051554 wait complete. 2023-03-03T13:30:17.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.489+0000 7f479c8a6700 1 Processor -- start 2023-03-03T13:30:17.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.489+0000 7f479c8a6700 1 -- start start 2023-03-03T13:30:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479c8a6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f479814f9d0 0x7f47981eafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479c8a6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f479814fda0 0x7f479806e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479c8a6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47981530d0 0x7f479806e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479c8a6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479806ecd0 con 0x7f479814f9d0 2023-03-03T13:30:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479c8a6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47980bd3d0 con 0x7f47981530d0 2023-03-03T13:30:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479c8a6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47980be8d0 con 0x7f479814fda0 2023-03-03T13:30:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f479814f9d0 0x7f47981eafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f479814f9d0 0x7f47981eafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44966/0 (socket says 172.21.15.7:44966) 2023-03-03T13:30:17.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479659c700 1 -- 172.21.15.7:0/3884399991 learned_addr learned my addr 172.21.15.7:0/3884399991 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:30:17.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f4796d9d700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47981530d0 0x7f479806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:17.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479659c700 1 -- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f479814fda0 msgr2=0x7f479806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:17.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f4795d9b700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f479814fda0 0x7f479806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:17.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479659c700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f479814fda0 0x7f479806e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:17.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479659c700 1 -- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47981530d0 msgr2=0x7f479806e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:17.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479659c700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47981530d0 0x7f479806e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:17.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f479659c700 1 -- 172.21.15.7:0/3884399991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4790009a70 con 0x7f479814f9d0 2023-03-03T13:30:17.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.490+0000 7f4796d9d700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47981530d0 0x7f479806e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:30:17.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.491+0000 7f479659c700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f479814f9d0 0x7f47981eafe0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f478800c500 tx=0x7f478800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:30:17.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.491+0000 7f4786ffd700 1 -- 172.21.15.7:0/3884399991 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47880052d0 con 0x7f479814f9d0 2023-03-03T13:30:17.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.491+0000 7f4786ffd700 1 -- 172.21.15.7:0/3884399991 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788005450 con 0x7f479814f9d0 2023-03-03T13:30:17.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.491+0000 7f479c8a6700 1 -- 172.21.15.7:0/3884399991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47980ac350 con 0x7f479814f9d0 2023-03-03T13:30:17.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.491+0000 7f4786ffd700 1 -- 172.21.15.7:0/3884399991 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47880038c0 con 0x7f479814f9d0 2023-03-03T13:30:17.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.491+0000 7f479c8a6700 1 -- 172.21.15.7:0/3884399991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47980ab630 con 0x7f479814f9d0 2023-03-03T13:30:17.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.492+0000 7f479c8a6700 1 -- 172.21.15.7:0/3884399991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4798071fa0 con 0x7f479814f9d0 2023-03-03T13:30:17.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.492+0000 7f4786ffd700 1 -- 172.21.15.7:0/3884399991 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4788019030 con 0x7f479814f9d0 2023-03-03T13:30:17.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.493+0000 7f4786ffd700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47780682f0 0x7f477806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:30:17.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.493+0000 7f4786ffd700 1 -- 172.21.15.7:0/3884399991 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4788085440 con 0x7f479814f9d0 2023-03-03T13:30:17.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.493+0000 7f4795d9b700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47780682f0 0x7f477806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:30:17.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.493+0000 7f4795d9b700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47780682f0 0x7f477806a7a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f47980c25c0 tx=0x7f478c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:30:17.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.497+0000 7f4786ffd700 1 -- 172.21.15.7:0/3884399991 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47880517e0 con 0x7f479814f9d0 2023-03-03T13:30:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.735+0000 7f479c8a6700 1 -- 172.21.15.7:0/3884399991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47980acfa0 con 0x7f479814f9d0 2023-03-03T13:30:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.736+0000 7f4786ffd700 1 -- 172.21.15.7:0/3884399991 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4788054e00 con 0x7f479814f9d0 2023-03-03T13:30:17.738 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:30:17.738 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:30:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 -- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47780682f0 msgr2=0x7f477806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47780682f0 0x7f477806a7a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f47980c25c0 tx=0x7f478c009040 comp rx=0 tx=0).stop 2023-03-03T13:30:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 -- 172.21.15.7:0/3884399991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f479814f9d0 msgr2=0x7f47981eafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:30:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f479814f9d0 0x7f47981eafe0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f478800c500 tx=0x7f478800c8c0 comp rx=0 tx=0).stop 2023-03-03T13:30:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 -- 172.21.15.7:0/3884399991 shutdown_connections 2023-03-03T13:30:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47981530d0 0x7f479806e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47780682f0 0x7f477806a7a0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f479814f9d0 0x7f47981eafe0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 --2- 172.21.15.7:0/3884399991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f479814fda0 0x7f479806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:30:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 -- 172.21.15.7:0/3884399991 >> 172.21.15.7:0/3884399991 conn(0x7f47980c8900 msgr2=0x7f47980ab7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:30:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 -- 172.21.15.7:0/3884399991 shutdown_connections 2023-03-03T13:30:17.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:30:17.738+0000 7f479c8a6700 1 -- 172.21.15.7:0/3884399991 wait complete. 2023-03-03T13:30:17.742 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:30:17.752 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T13:30:42.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:30:42.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:13.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:13.003 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:13.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:31:13.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.386+0000 7fa0e9f0a700 1 Processor -- start 2023-03-03T13:31:13.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.386+0000 7fa0e9f0a700 1 -- start start 2023-03-03T13:31:13.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.387+0000 7fa0e9f0a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0e414f9d0 0x7fa0e4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:13.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.387+0000 7fa0e9f0a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa0e414fda0 0x7fa0e4071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:13.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.387+0000 7fa0e9f0a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40abe20 0x7fa0e41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:13.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.387+0000 7fa0e9f0a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e40c2f30 con 0x7fa0e414f9d0 2023-03-03T13:31:13.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.387+0000 7fa0e9f0a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e40c31d0 con 0x7fa0e414fda0 2023-03-03T13:31:13.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.387+0000 7fa0e9f0a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e41524a0 con 0x7fa0e40abe20 2023-03-03T13:31:13.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.387+0000 7fa0e37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0e414f9d0 0x7fa0e4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:13.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.387+0000 7fa0e37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa0e414f9d0 0x7fa0e4156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45008/0 (socket says 172.21.15.7:45008) 2023-03-03T13:31:13.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.387+0000 7fa0e37fe700 1 -- 172.21.15.7:0/2269260837 learned_addr learned my addr 172.21.15.7:0/2269260837 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:31:13.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e2ffd700 1 --2- 172.21.15.7:0/2269260837 >> v2:172.21.15.143:3300/0 conn(0x7fa0e414fda0 0x7fa0e4071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e3fff700 1 --2- 172.21.15.7:0/2269260837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40abe20 0x7fa0e41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e37fe700 1 -- 172.21.15.7:0/2269260837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40abe20 msgr2=0x7fa0e41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:31:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e37fe700 1 --2- 172.21.15.7:0/2269260837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40abe20 0x7fa0e41529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e37fe700 1 -- 172.21.15.7:0/2269260837 >> v2:172.21.15.143:3300/0 conn(0x7fa0e414fda0 msgr2=0x7fa0e4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:31:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e37fe700 1 --2- 172.21.15.7:0/2269260837 >> v2:172.21.15.143:3300/0 conn(0x7fa0e414fda0 0x7fa0e4071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e37fe700 1 -- 172.21.15.7:0/2269260837 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e40c34b0 con 0x7fa0e414f9d0 2023-03-03T13:31:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e37fe700 1 --2- 172.21.15.7:0/2269260837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 0x7fa0e4156430 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40709b0 tx=0x7fa0cc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a341dd43e6cd47 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:13.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e27fc700 1 -- 172.21.15.7:0/2269260837 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc011830 con 0x7fa0e414f9d0 2023-03-03T13:31:13.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e27fc700 1 -- 172.21.15.7:0/2269260837 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc011990 con 0x7fa0e414f9d0 2023-03-03T13:31:13.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.388+0000 7fa0e27fc700 1 -- 172.21.15.7:0/2269260837 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc011d20 con 0x7fa0e414f9d0 2023-03-03T13:31:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.389+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/2269260837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 msgr2=0x7fa0e4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.389+0000 7fa0e9f0a700 1 --2- 172.21.15.7:0/2269260837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 0x7fa0e4156430 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40709b0 tx=0x7fa0cc009bd0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.389+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/2269260837 shutdown_connections 2023-03-03T13:31:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.389+0000 7fa0e9f0a700 1 --2- 172.21.15.7:0/2269260837 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40abe20 0x7fa0e41529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.389+0000 7fa0e9f0a700 1 --2- 172.21.15.7:0/2269260837 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 0x7fa0e4156430 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.389+0000 7fa0e9f0a700 1 --2- 172.21.15.7:0/2269260837 >> v2:172.21.15.143:3300/0 conn(0x7fa0e414fda0 0x7fa0e4071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.389+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/2269260837 >> 172.21.15.7:0/2269260837 conn(0x7fa0e40c8900 msgr2=0x7fa0e40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:31:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.389+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/2269260837 shutdown_connections 2023-03-03T13:31:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.389+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/2269260837 wait complete. 2023-03-03T13:31:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.390+0000 7fa0e9f0a700 1 Processor -- start 2023-03-03T13:31:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.390+0000 7fa0e9f0a700 1 -- start start 2023-03-03T13:31:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e9f0a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40abe20 0x7fa0e4149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:13.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e9f0a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 0x7fa0e406e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:13.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e9f0a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e414fda0 0x7fa0e406ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:13.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e9f0a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4153cf0 con 0x7fa0e414f9d0 2023-03-03T13:31:13.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e9f0a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e406f140 con 0x7fa0e414fda0 2023-03-03T13:31:13.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e9f0a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e41e6470 con 0x7fa0e40abe20 2023-03-03T13:31:13.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 0x7fa0e406e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:13.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 0x7fa0e406e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45016/0 (socket says 172.21.15.7:45016) 2023-03-03T13:31:13.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e2ffd700 1 -- 172.21.15.7:0/3428926925 learned_addr learned my addr 172.21.15.7:0/3428926925 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:31:13.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e3fff700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e414fda0 0x7fa0e406ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:13.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.391+0000 7fa0e37fe700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40abe20 0x7fa0e4149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:13.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0e2ffd700 1 -- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40abe20 msgr2=0x7fa0e4149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:13.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0e2ffd700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40abe20 0x7fa0e4149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0e2ffd700 1 -- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e414fda0 msgr2=0x7fa0e406ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:13.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0e2ffd700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e414fda0 0x7fa0e406ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0e2ffd700 1 -- 172.21.15.7:0/3428926925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc009550 con 0x7fa0e414f9d0 2023-03-03T13:31:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0e2ffd700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 0x7fa0e406e6c0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400e980 tx=0x7fa0d400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3428926925 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4009710 con 0x7fa0e414f9d0 2023-03-03T13:31:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3428926925 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d4013680 con 0x7fa0e414f9d0 2023-03-03T13:31:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/3428926925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e40ac350 con 0x7fa0e414f9d0 2023-03-03T13:31:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3428926925 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4013a30 con 0x7fa0e414f9d0 2023-03-03T13:31:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.392+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/3428926925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e40ab630 con 0x7fa0e414f9d0 2023-03-03T13:31:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.393+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/3428926925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e40c3e10 con 0x7fa0e414f9d0 2023-03-03T13:31:13.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.393+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3428926925 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa0d4019030 con 0x7fa0e414f9d0 2023-03-03T13:31:13.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.394+0000 7fa0dbfff700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c40682f0 0x7fa0c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:13.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.394+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3428926925 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa0d4085ef0 con 0x7fa0e414f9d0 2023-03-03T13:31:13.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.394+0000 7fa0e37fe700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c40682f0 0x7fa0c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:13.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.395+0000 7fa0e37fe700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c40682f0 0x7fa0c406a7a0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009c10 tx=0x7fa0cc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:13.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.398+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3428926925 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa0d4052290 con 0x7fa0e414f9d0 2023-03-03T13:31:13.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.636+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/3428926925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0e41e8b30 con 0x7fa0e414f9d0 2023-03-03T13:31:13.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.637+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3428926925 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa0d40558b0 con 0x7fa0e414f9d0 2023-03-03T13:31:13.639 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:31:13.639 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:31:13.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c40682f0 msgr2=0x7fa0c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:13.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c40682f0 0x7fa0c406a7a0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009c10 tx=0x7fa0cc008030 comp rx=0 tx=0).stop 2023-03-03T13:31:13.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/3428926925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 msgr2=0x7fa0e406e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:13.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 0x7fa0e406e6c0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400e980 tx=0x7fa0d400ec90 comp rx=0 tx=0).stop 2023-03-03T13:31:13.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/3428926925 shutdown_connections 2023-03-03T13:31:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e414fda0 0x7fa0e406ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40abe20 0x7fa0e4149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c40682f0 0x7fa0c406a7a0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 --2- 172.21.15.7:0/3428926925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414f9d0 0x7fa0e406e6c0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/3428926925 >> 172.21.15.7:0/3428926925 conn(0x7fa0e40c8900 msgr2=0x7fa0e4150ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:31:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/3428926925 shutdown_connections 2023-03-03T13:31:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:13.639+0000 7fa0e9f0a700 1 -- 172.21.15.7:0/3428926925 wait complete. 2023-03-03T13:31:13.643 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:31:13.653 INFO:tasks.mds_thrash.fs.[b]:waiting for 16 secs before thrashing 2023-03-03T13:31:23.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:31:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.889+0000 7f91f5a68700 1 Processor -- start 2023-03-03T13:31:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.889+0000 7f91f5a68700 1 -- start start 2023-03-03T13:31:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.889+0000 7f91f5a68700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91f0155410 0x7f91f00bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.889+0000 7f91f5a68700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f91f00c5b00 0x7f91f00bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.889+0000 7f91f5a68700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f00c08d0 con 0x7f91f0151e80 2023-03-03T13:31:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.889+0000 7f91f5a68700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f0152250 con 0x7f91f0155410 2023-03-03T13:31:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.889+0000 7f91f5a68700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00c0fb0 con 0x7f91f00c5b00 2023-03-03T13:31:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91eeffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91f0155410 0x7f91f00bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91eeffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91f0155410 0x7f91f00bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45028/0 (socket says 172.21.15.7:45028) 2023-03-03T13:31:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91eeffd700 1 -- 172.21.15.7:0/3723409516 learned_addr learned my addr 172.21.15.7:0/3723409516 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:31:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91effff700 1 --2- 172.21.15.7:0/3723409516 >> v2:172.21.15.143:3300/0 conn(0x7f91f00c5b00 0x7f91f00bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91eeffd700 1 -- 172.21.15.7:0/3723409516 >> v2:172.21.15.143:3300/0 conn(0x7f91f00c5b00 msgr2=0x7f91f00bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:31:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91eeffd700 1 --2- 172.21.15.7:0/3723409516 >> v2:172.21.15.143:3300/0 conn(0x7f91f00c5b00 0x7f91f00bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91eeffd700 1 -- 172.21.15.7:0/3723409516 >> v1:172.21.15.143:6789/0 conn(0x7f91f0151e80 legacy=0x7f91f00c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:31:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91eeffd700 1 -- 172.21.15.7:0/3723409516 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0154930 con 0x7f91f0155410 2023-03-03T13:31:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91eeffd700 1 --2- 172.21.15.7:0/3723409516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f0155410 0x7f91f00bf8e0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f91f006bc30 tx=0x7f91e4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e90ea945def239d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91ee7fc700 1 -- 172.21.15.7:0/3723409516 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e400b870 con 0x7f91f0155410 2023-03-03T13:31:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91ee7fc700 1 -- 172.21.15.7:0/3723409516 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e400b9d0 con 0x7f91f0155410 2023-03-03T13:31:23.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.890+0000 7f91ee7fc700 1 -- 172.21.15.7:0/3723409516 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e400bd60 con 0x7f91f0155410 2023-03-03T13:31:23.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.891+0000 7f91f5a68700 1 -- 172.21.15.7:0/3723409516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f0155410 msgr2=0x7f91f00bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.891+0000 7f91f5a68700 1 --2- 172.21.15.7:0/3723409516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f0155410 0x7f91f00bf8e0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f91f006bc30 tx=0x7f91e4009bd0 comp rx=0 tx=0).stop 2023-03-03T13:31:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.891+0000 7f91f5a68700 1 -- 172.21.15.7:0/3723409516 shutdown_connections 2023-03-03T13:31:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.891+0000 7f91f5a68700 1 --2- 172.21.15.7:0/3723409516 >> v2:172.21.15.143:3300/0 conn(0x7f91f00c5b00 0x7f91f00bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.891+0000 7f91f5a68700 1 --2- 172.21.15.7:0/3723409516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f0155410 0x7f91f00bf8e0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.891+0000 7f91f5a68700 1 -- 172.21.15.7:0/3723409516 >> 172.21.15.7:0/3723409516 conn(0x7f91f0071630 msgr2=0x7f91f006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:31:23.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.891+0000 7f91f5a68700 1 -- 172.21.15.7:0/3723409516 shutdown_connections 2023-03-03T13:31:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.891+0000 7f91f5a68700 1 -- 172.21.15.7:0/3723409516 wait complete. 2023-03-03T13:31:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.892+0000 7f91f5a68700 1 Processor -- start 2023-03-03T13:31:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.893+0000 7f91f5a68700 1 -- start start 2023-03-03T13:31:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91f5a68700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f00c5b00 0x7f91f00c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91f5a68700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f0151e80 0x7f91f01e11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91f5a68700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f0155410 0x7f91f01463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91f5a68700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f014eff0 con 0x7f91f00c5b00 2023-03-03T13:31:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91f5a68700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f014f570 con 0x7f91f0155410 2023-03-03T13:31:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91f5a68700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f0147130 con 0x7f91f0151e80 2023-03-03T13:31:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91ef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f00c5b00 0x7f91f00c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91ef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f00c5b00 0x7f91f00c9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45032/0 (socket says 172.21.15.7:45032) 2023-03-03T13:31:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91ef7fe700 1 -- 172.21.15.7:0/1577147995 learned_addr learned my addr 172.21.15.7:0/1577147995 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:31:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91effff700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f0155410 0x7f91f01463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91eeffd700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f0151e80 0x7f91f01e11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:23.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91ef7fe700 1 -- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f0151e80 msgr2=0x7f91f01e11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:23.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91ef7fe700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f0151e80 0x7f91f01e11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:23.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91ef7fe700 1 -- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f0155410 msgr2=0x7f91f01463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:23.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91ef7fe700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f0155410 0x7f91f01463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:23.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91ef7fe700 1 -- 172.21.15.7:0/1577147995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e4009550 con 0x7f91f00c5b00 2023-03-03T13:31:23.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91ef7fe700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f00c5b00 0x7f91f00c9230 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f91e000c4e0 tx=0x7f91e000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:23.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91d7fff700 1 -- 172.21.15.7:0/1577147995 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e00052d0 con 0x7f91f00c5b00 2023-03-03T13:31:23.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91d7fff700 1 -- 172.21.15.7:0/1577147995 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e0005430 con 0x7f91f00c5b00 2023-03-03T13:31:23.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91f5a68700 1 -- 172.21.15.7:0/1577147995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f0154ef0 con 0x7f91f00c5b00 2023-03-03T13:31:23.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.894+0000 7f91d7fff700 1 -- 172.21.15.7:0/1577147995 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e00038c0 con 0x7f91f00c5b00 2023-03-03T13:31:23.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.895+0000 7f91f5a68700 1 -- 172.21.15.7:0/1577147995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f0154c10 con 0x7f91f00c5b00 2023-03-03T13:31:23.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.896+0000 7f91f5a68700 1 -- 172.21.15.7:0/1577147995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f00c5220 con 0x7f91f00c5b00 2023-03-03T13:31:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.896+0000 7f91d7fff700 1 -- 172.21.15.7:0/1577147995 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f91e0019030 con 0x7f91f00c5b00 2023-03-03T13:31:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.896+0000 7f91d7fff700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91dc0682f0 0x7f91dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.896+0000 7f91d7fff700 1 -- 172.21.15.7:0/1577147995 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f91e0085540 con 0x7f91f00c5b00 2023-03-03T13:31:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.896+0000 7f91eeffd700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91dc0682f0 0x7f91dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.897+0000 7f91eeffd700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91dc0682f0 0x7f91dc06a7a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f91f006bc30 tx=0x7f91e4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:23.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:23.901+0000 7f91d7fff700 1 -- 172.21.15.7:0/1577147995 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f91e00518e0 con 0x7f91f00c5b00 2023-03-03T13:31:24.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.135+0000 7f91f5a68700 1 -- 172.21.15.7:0/1577147995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91f0074bb0 con 0x7f91f00c5b00 2023-03-03T13:31:24.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.136+0000 7f91d7fff700 1 -- 172.21.15.7:0/1577147995 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f91e0054f00 con 0x7f91f00c5b00 2023-03-03T13:31:24.138 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:31:24.138 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:31:24.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 -- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91dc0682f0 msgr2=0x7f91dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91dc0682f0 0x7f91dc06a7a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f91f006bc30 tx=0x7f91e4008030 comp rx=0 tx=0).stop 2023-03-03T13:31:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 -- 172.21.15.7:0/1577147995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f00c5b00 msgr2=0x7f91f00c9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f00c5b00 0x7f91f00c9230 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f91e000c4e0 tx=0x7f91e000c8a0 comp rx=0 tx=0).stop 2023-03-03T13:31:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 -- 172.21.15.7:0/1577147995 shutdown_connections 2023-03-03T13:31:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f0155410 0x7f91f01463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91dc0682f0 0x7f91dc06a7a0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:24.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f00c5b00 0x7f91f00c9230 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:24.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 --2- 172.21.15.7:0/1577147995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f0151e80 0x7f91f01e11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:24.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 -- 172.21.15.7:0/1577147995 >> 172.21.15.7:0/1577147995 conn(0x7f91f0071630 msgr2=0x7f91f00bf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:31:24.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 -- 172.21.15.7:0/1577147995 shutdown_connections 2023-03-03T13:31:24.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:24.138+0000 7f91f5a68700 1 -- 172.21.15.7:0/1577147995 wait complete. 2023-03-03T13:31:24.142 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:31:24.152 INFO:tasks.mds_thrash.fs.[a]:waiting for 44 secs before thrashing 2023-03-03T13:31:29.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:31:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.792+0000 7fbb02419700 1 Processor -- start 2023-03-03T13:31:29.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.792+0000 7fbb02419700 1 -- start start 2023-03-03T13:31:29.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbb02419700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbafc071e80 0x7fbafc151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:29.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbb02419700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc1510d0 con 0x7fbafc072250 2023-03-03T13:31:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbb02419700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0c3740 con 0x7fbafc14e900 2023-03-03T13:31:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbb02419700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbafc070670 con 0x7fbafc071e80 2023-03-03T13:31:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbafb7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbafc072250 0x7fbafc14e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39020/0 (socket says 172.21.15.7:39020) 2023-03-03T13:31:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbafbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbafc071e80 0x7fbafc151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbafb7fe700 1 -- 172.21.15.7:0/2343402259 learned_addr learned my addr 172.21.15.7:0/2343402259 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:31:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbafbfff700 1 -- 172.21.15.7:0/2343402259 >> v1:172.21.15.143:6789/0 conn(0x7fbafc14e900 legacy=0x7fbafc15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:29.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbafbfff700 1 -- 172.21.15.7:0/2343402259 >> v1:172.21.15.7:6789/0 conn(0x7fbafc072250 legacy=0x7fbafc14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbafaffd700 1 -- 172.21.15.7:0/2343402259 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593146238 0 0) 0x7fbafc1510d0 con 0x7fbafc072250 2023-03-03T13:31:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.793+0000 7fbafbfff700 1 -- 172.21.15.7:0/2343402259 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0c39e0 con 0x7fbafc071e80 2023-03-03T13:31:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbafbfff700 1 --2- 172.21.15.7:0/2343402259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc071e80 0x7fbafc151820 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009840 tx=0x7fbaec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=764ee9bb146c02a7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbafaffd700 1 -- 172.21.15.7:0/2343402259 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec011830 con 0x7fbafc071e80 2023-03-03T13:31:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbafaffd700 1 -- 172.21.15.7:0/2343402259 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec011990 con 0x7fbafc071e80 2023-03-03T13:31:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbafaffd700 1 -- 172.21.15.7:0/2343402259 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec011d20 con 0x7fbafc071e80 2023-03-03T13:31:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbb02419700 1 -- 172.21.15.7:0/2343402259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc071e80 msgr2=0x7fbafc151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:29.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbb02419700 1 --2- 172.21.15.7:0/2343402259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc071e80 0x7fbafc151820 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009840 tx=0x7fbaec009b50 comp rx=0 tx=0).stop 2023-03-03T13:31:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbb02419700 1 -- 172.21.15.7:0/2343402259 shutdown_connections 2023-03-03T13:31:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbb02419700 1 --2- 172.21.15.7:0/2343402259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc071e80 0x7fbafc151820 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbb02419700 1 -- 172.21.15.7:0/2343402259 >> 172.21.15.7:0/2343402259 conn(0x7fbafc0a94d0 msgr2=0x7fbafc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:31:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbb02419700 1 -- 172.21.15.7:0/2343402259 shutdown_connections 2023-03-03T13:31:29.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.794+0000 7fbb02419700 1 -- 172.21.15.7:0/2343402259 wait complete. 2023-03-03T13:31:29.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.796+0000 7fbb02419700 1 Processor -- start 2023-03-03T13:31:29.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.796+0000 7fbb02419700 1 -- start start 2023-03-03T13:31:29.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb02419700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbafc071e80 0x7fbafc0abec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb02419700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbafc072250 0x7fbafc0ac400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb02419700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc14e900 0x7fbafc1425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb02419700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0bd660 con 0x7fbafc14e900 2023-03-03T13:31:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb02419700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0aba30 con 0x7fbafc072250 2023-03-03T13:31:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb02419700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0bdb90 con 0x7fbafc071e80 2023-03-03T13:31:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb009b6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc14e900 0x7fbafc1425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:29.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb009b6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc14e900 0x7fbafc1425d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45050/0 (socket says 172.21.15.7:45050) 2023-03-03T13:31:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb009b6700 1 -- 172.21.15.7:0/3892157468 learned_addr learned my addr 172.21.15.7:0/3892157468 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:31:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbafbfff700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbafc071e80 0x7fbafc0abec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbafb7fe700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbafc072250 0x7fbafc0ac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb009b6700 1 -- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbafc071e80 msgr2=0x7fbafc0abec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.797+0000 7fbb009b6700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbafc071e80 0x7fbafc0abec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.798+0000 7fbb009b6700 1 -- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbafc072250 msgr2=0x7fbafc0ac400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.798+0000 7fbb009b6700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbafc072250 0x7fbafc0ac400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.798+0000 7fbb009b6700 1 -- 172.21.15.7:0/3892157468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec0094a0 con 0x7fbafc14e900 2023-03-03T13:31:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.798+0000 7fbb009b6700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc14e900 0x7fbafc1425d0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400cbf0 tx=0x7fbaf400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.798+0000 7fbaf8ff9700 1 -- 172.21.15.7:0/3892157468 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4003c00 con 0x7fbafc14e900 2023-03-03T13:31:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.798+0000 7fbaf8ff9700 1 -- 172.21.15.7:0/3892157468 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf4003d60 con 0x7fbafc14e900 2023-03-03T13:31:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.798+0000 7fbaf8ff9700 1 -- 172.21.15.7:0/3892157468 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf400aaa0 con 0x7fbafc14e900 2023-03-03T13:31:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.798+0000 7fbb02419700 1 -- 172.21.15.7:0/3892157468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc06c9a0 con 0x7fbafc14e900 2023-03-03T13:31:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.798+0000 7fbb02419700 1 -- 172.21.15.7:0/3892157468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc151300 con 0x7fbafc14e900 2023-03-03T13:31:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.799+0000 7fbb02419700 1 -- 172.21.15.7:0/3892157468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbafc06cb30 con 0x7fbafc14e900 2023-03-03T13:31:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.799+0000 7fbaf8ff9700 1 -- 172.21.15.7:0/3892157468 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbaf4010050 con 0x7fbafc14e900 2023-03-03T13:31:29.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.800+0000 7fbaf8ff9700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbae80682f0 0x7fbae806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:29.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.800+0000 7fbaf8ff9700 1 -- 172.21.15.7:0/3892157468 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbaf4085f40 con 0x7fbafc14e900 2023-03-03T13:31:29.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.800+0000 7fbafbfff700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbae80682f0 0x7fbae806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:29.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.800+0000 7fbafbfff700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbae80682f0 0x7fbae806a7a0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fbaec0090b0 tx=0x7fbaec01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:29.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:29.804+0000 7fbaf8ff9700 1 -- 172.21.15.7:0/3892157468 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbaf4052280 con 0x7fbafc14e900 2023-03-03T13:31:30.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.042+0000 7fbb02419700 1 -- 172.21.15.7:0/3892157468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbafc074bb0 con 0x7fbafc14e900 2023-03-03T13:31:30.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.042+0000 7fbaf8ff9700 1 -- 172.21.15.7:0/3892157468 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbaf40558a0 con 0x7fbafc14e900 2023-03-03T13:31:30.045 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:31:30.045 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:31:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 -- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbae80682f0 msgr2=0x7fbae806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbae80682f0 0x7fbae806a7a0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fbaec0090b0 tx=0x7fbaec01c040 comp rx=0 tx=0).stop 2023-03-03T13:31:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 -- 172.21.15.7:0/3892157468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc14e900 msgr2=0x7fbafc1425d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc14e900 0x7fbafc1425d0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400cbf0 tx=0x7fbaf400cf00 comp rx=0 tx=0).stop 2023-03-03T13:31:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 -- 172.21.15.7:0/3892157468 shutdown_connections 2023-03-03T13:31:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbae80682f0 0x7fbae806a7a0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbafc14e900 0x7fbafc1425d0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:30.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbafc071e80 0x7fbafc0abec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:30.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 --2- 172.21.15.7:0/3892157468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbafc072250 0x7fbafc0ac400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:30.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 -- 172.21.15.7:0/3892157468 >> 172.21.15.7:0/3892157468 conn(0x7fbafc0a94d0 msgr2=0x7fbafc14c720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:31:30.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 -- 172.21.15.7:0/3892157468 shutdown_connections 2023-03-03T13:31:30.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:30.045+0000 7fbb02419700 1 -- 172.21.15.7:0/3892157468 wait complete. 2023-03-03T13:31:30.048 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:31:30.059 INFO:tasks.mds_thrash.fs.[b]:waiting for 28 secs before thrashing 2023-03-03T13:31:43.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:43.141 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:31:58.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:31:58.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.196+0000 7f1d495c1700 1 Processor -- start 2023-03-03T13:31:58.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.196+0000 7f1d495c1700 1 -- start start 2023-03-03T13:31:58.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d495c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440caa30 0x7f1d440c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:58.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d495c1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1d440c00e0 0x7f1d44154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:58.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d495c1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1d440cae00 0x7f1d440c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:58.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d495c1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d440cb280 con 0x7f1d440cae00 2023-03-03T13:31:58.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d495c1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d440bfd10 con 0x7f1d440c00e0 2023-03-03T13:31:58.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d495c1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440bfa30 con 0x7f1d440caa30 2023-03-03T13:31:58.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d437fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1d440cae00 0x7f1d440c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:58.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d437fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1d440cae00 0x7f1d440c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45074/0 (socket says 172.21.15.7:45074) 2023-03-03T13:31:58.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d437fe700 1 -- 172.21.15.7:0/2547138000 learned_addr learned my addr 172.21.15.7:0/2547138000 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:31:58.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d427fc700 1 --2- 172.21.15.7:0/2547138000 >> v2:172.21.15.143:3300/0 conn(0x7f1d440c00e0 0x7f1d44154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:58.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d42ffd700 1 --2- 172.21.15.7:0/2547138000 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440caa30 0x7f1d440c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:58.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d437fe700 1 -- 172.21.15.7:0/2547138000 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440caa30 msgr2=0x7f1d440c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:31:58.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d437fe700 1 --2- 172.21.15.7:0/2547138000 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440caa30 0x7f1d440c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d437fe700 1 -- 172.21.15.7:0/2547138000 >> v2:172.21.15.143:3300/0 conn(0x7f1d440c00e0 msgr2=0x7f1d44154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:31:58.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d437fe700 1 --2- 172.21.15.7:0/2547138000 >> v2:172.21.15.143:3300/0 conn(0x7f1d440c00e0 0x7f1d44154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d437fe700 1 -- 172.21.15.7:0/2547138000 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44154510 con 0x7f1d440cae00 2023-03-03T13:31:58.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d437fe700 1 --2- 172.21.15.7:0/2547138000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440cae00 0x7f1d440c6c00 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f1d440c81e0 tx=0x7f1d3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576e9757dd6b27b8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:58.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.197+0000 7f1d41ffb700 1 -- 172.21.15.7:0/2547138000 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c00e070 con 0x7f1d440cae00 2023-03-03T13:31:58.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d41ffb700 1 -- 172.21.15.7:0/2547138000 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c00b910 con 0x7f1d440cae00 2023-03-03T13:31:58.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d41ffb700 1 -- 172.21.15.7:0/2547138000 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c00bcc0 con 0x7f1d440cae00 2023-03-03T13:31:58.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d495c1700 1 -- 172.21.15.7:0/2547138000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440cae00 msgr2=0x7f1d440c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:58.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d495c1700 1 --2- 172.21.15.7:0/2547138000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440cae00 0x7f1d440c6c00 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f1d440c81e0 tx=0x7f1d3c00b330 comp rx=0 tx=0).stop 2023-03-03T13:31:58.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d495c1700 1 -- 172.21.15.7:0/2547138000 shutdown_connections 2023-03-03T13:31:58.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d495c1700 1 --2- 172.21.15.7:0/2547138000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440cae00 0x7f1d440c6c00 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d495c1700 1 --2- 172.21.15.7:0/2547138000 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440caa30 0x7f1d440c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d495c1700 1 --2- 172.21.15.7:0/2547138000 >> v2:172.21.15.143:3300/0 conn(0x7f1d440c00e0 0x7f1d44154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d495c1700 1 -- 172.21.15.7:0/2547138000 >> 172.21.15.7:0/2547138000 conn(0x7f1d44066e40 msgr2=0x7f1d44071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:31:58.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d495c1700 1 -- 172.21.15.7:0/2547138000 shutdown_connections 2023-03-03T13:31:58.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.198+0000 7f1d495c1700 1 -- 172.21.15.7:0/2547138000 wait complete. 2023-03-03T13:31:58.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.199+0000 7f1d495c1700 1 Processor -- start 2023-03-03T13:31:58.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.199+0000 7f1d495c1700 1 -- start start 2023-03-03T13:31:58.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d495c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440c00e0 0x7f1d44073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:58.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d495c1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440caa30 0x7f1d441562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:58.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d495c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d440cae00 0x7f1d4414c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:58.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d495c1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d441de030 con 0x7f1d440caa30 2023-03-03T13:31:58.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d495c1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d441dda40 con 0x7f1d440cae00 2023-03-03T13:31:58.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d495c1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440c4380 con 0x7f1d440c00e0 2023-03-03T13:31:58.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d427fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440caa30 0x7f1d441562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:58.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d427fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440caa30 0x7f1d441562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45078/0 (socket says 172.21.15.7:45078) 2023-03-03T13:31:58.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d427fc700 1 -- 172.21.15.7:0/3980476915 learned_addr learned my addr 172.21.15.7:0/3980476915 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:31:58.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d437fe700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d440cae00 0x7f1d4414c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:58.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.200+0000 7f1d42ffd700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440c00e0 0x7f1d44073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:58.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d427fc700 1 -- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440c00e0 msgr2=0x7f1d44073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:58.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d427fc700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440c00e0 0x7f1d44073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d427fc700 1 -- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d440cae00 msgr2=0x7f1d4414c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:58.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d427fc700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d440cae00 0x7f1d4414c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d427fc700 1 -- 172.21.15.7:0/3980476915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c009b30 con 0x7f1d440caa30 2023-03-03T13:31:58.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d42ffd700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440c00e0 0x7f1d44073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:31:58.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d427fc700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440caa30 0x7f1d441562e0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800e900 tx=0x7f1d3800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:58.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d2b7fe700 1 -- 172.21.15.7:0/3980476915 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38009650 con 0x7f1d440caa30 2023-03-03T13:31:58.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d2b7fe700 1 -- 172.21.15.7:0/3980476915 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3800ee50 con 0x7f1d440caa30 2023-03-03T13:31:58.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d495c1700 1 -- 172.21.15.7:0/3980476915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d44154ad0 con 0x7f1d440caa30 2023-03-03T13:31:58.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d2b7fe700 1 -- 172.21.15.7:0/3980476915 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3800ca50 con 0x7f1d440caa30 2023-03-03T13:31:58.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.201+0000 7f1d495c1700 1 -- 172.21.15.7:0/3980476915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d441547f0 con 0x7f1d440caa30 2023-03-03T13:31:58.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.202+0000 7f1d495c1700 1 -- 172.21.15.7:0/3980476915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d440bf650 con 0x7f1d440caa30 2023-03-03T13:31:58.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.202+0000 7f1d2b7fe700 1 -- 172.21.15.7:0/3980476915 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1d38023070 con 0x7f1d440caa30 2023-03-03T13:31:58.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.203+0000 7f1d2b7fe700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d300682f0 0x7f1d3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:31:58.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.203+0000 7f1d2b7fe700 1 -- 172.21.15.7:0/3980476915 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1d38086360 con 0x7f1d440caa30 2023-03-03T13:31:58.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.203+0000 7f1d42ffd700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d300682f0 0x7f1d3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:31:58.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.204+0000 7f1d42ffd700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d300682f0 0x7f1d3006a7a0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009bb0 tx=0x7f1d34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:31:58.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.208+0000 7f1d2b7fe700 1 -- 172.21.15.7:0/3980476915 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1d380526a0 con 0x7f1d440caa30 2023-03-03T13:31:58.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.446+0000 7f1d495c1700 1 -- 172.21.15.7:0/3980476915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d44074bb0 con 0x7f1d440caa30 2023-03-03T13:31:58.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.446+0000 7f1d2b7fe700 1 -- 172.21.15.7:0/3980476915 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1d38055cc0 con 0x7f1d440caa30 2023-03-03T13:31:58.450 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:31:58.450 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:31:58.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 -- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d300682f0 msgr2=0x7f1d3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:58.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d300682f0 0x7f1d3006a7a0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009bb0 tx=0x7f1d34008040 comp rx=0 tx=0).stop 2023-03-03T13:31:58.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 -- 172.21.15.7:0/3980476915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440caa30 msgr2=0x7f1d441562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:31:58.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440caa30 0x7f1d441562e0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800e900 tx=0x7f1d3800ec10 comp rx=0 tx=0).stop 2023-03-03T13:31:58.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 -- 172.21.15.7:0/3980476915 shutdown_connections 2023-03-03T13:31:58.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d440cae00 0x7f1d4414c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d440c00e0 0x7f1d44073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d300682f0 0x7f1d3006a7a0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 --2- 172.21.15.7:0/3980476915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d440caa30 0x7f1d441562e0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:31:58.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 -- 172.21.15.7:0/3980476915 >> 172.21.15.7:0/3980476915 conn(0x7f1d44066e40 msgr2=0x7f1d440c7720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:31:58.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 -- 172.21.15.7:0/3980476915 shutdown_connections 2023-03-03T13:31:58.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:31:58.449+0000 7f1d495c1700 1 -- 172.21.15.7:0/3980476915 wait complete. 2023-03-03T13:31:58.454 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:31:58.464 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T13:32:08.154 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:32:08.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.292+0000 7f5ceaeda700 1 Processor -- start 2023-03-03T13:32:08.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.292+0000 7f5ceaeda700 1 -- start start 2023-03-03T13:32:08.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ceaeda700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5ce414fda0 0x7f5ce4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:08.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ceaeda700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5ce40abe20 0x7f5ce41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ceaeda700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce40c2f30 con 0x7f5ce414f9d0 2023-03-03T13:32:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ceaeda700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce40c31d0 con 0x7f5ce414fda0 2023-03-03T13:32:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ceaeda700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce41524a0 con 0x7f5ce40abe20 2023-03-03T13:32:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5ce414fda0 0x7f5ce4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce8c76700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5ce414f9d0 0x7f5ce4071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39068/0 (socket says 172.21.15.7:39068) 2023-03-03T13:32:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce8c76700 1 -- 172.21.15.7:0/414943485 learned_addr learned my addr 172.21.15.7:0/414943485 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:32:08.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce9477700 1 --2- 172.21.15.7:0/414943485 >> v2:172.21.15.143:3300/0 conn(0x7f5ce40abe20 0x7f5ce41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce3fff700 1 -- 172.21.15.7:0/414943485 >> v2:172.21.15.143:3300/0 conn(0x7f5ce40abe20 msgr2=0x7f5ce41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:32:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce3fff700 1 --2- 172.21.15.7:0/414943485 >> v2:172.21.15.143:3300/0 conn(0x7f5ce40abe20 0x7f5ce41529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce3fff700 1 -- 172.21.15.7:0/414943485 >> v1:172.21.15.7:6789/0 conn(0x7f5ce414f9d0 legacy=0x7f5ce4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce3fff700 1 -- 172.21.15.7:0/414943485 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40c34b0 con 0x7f5ce414fda0 2023-03-03T13:32:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce37fe700 1 -- 172.21.15.7:0/414943485 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752557658 0 0) 0x7f5ce40c2f30 con 0x7f5ce414f9d0 2023-03-03T13:32:08.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce3fff700 1 --2- 172.21.15.7:0/414943485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce414fda0 0x7f5ce4156430 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4071230 tx=0x7f5cd0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dadf5e3691ca9040 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:32:08.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce37fe700 1 -- 172.21.15.7:0/414943485 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0011830 con 0x7f5ce414fda0 2023-03-03T13:32:08.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.293+0000 7f5ce37fe700 1 -- 172.21.15.7:0/414943485 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd0011990 con 0x7f5ce414fda0 2023-03-03T13:32:08.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.294+0000 7f5ce37fe700 1 -- 172.21.15.7:0/414943485 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0011d20 con 0x7f5ce414fda0 2023-03-03T13:32:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.294+0000 7f5ceaeda700 1 -- 172.21.15.7:0/414943485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce414fda0 msgr2=0x7f5ce4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.294+0000 7f5ceaeda700 1 --2- 172.21.15.7:0/414943485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce414fda0 0x7f5ce4156430 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4071230 tx=0x7f5cd0009b20 comp rx=0 tx=0).stop 2023-03-03T13:32:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.294+0000 7f5ceaeda700 1 -- 172.21.15.7:0/414943485 shutdown_connections 2023-03-03T13:32:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.294+0000 7f5ceaeda700 1 --2- 172.21.15.7:0/414943485 >> v2:172.21.15.143:3300/0 conn(0x7f5ce40abe20 0x7f5ce41529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:08.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.294+0000 7f5ceaeda700 1 --2- 172.21.15.7:0/414943485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce414fda0 0x7f5ce4156430 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.294+0000 7f5ceaeda700 1 -- 172.21.15.7:0/414943485 >> 172.21.15.7:0/414943485 conn(0x7f5ce40c8900 msgr2=0x7f5ce40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:32:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.294+0000 7f5ceaeda700 1 -- 172.21.15.7:0/414943485 shutdown_connections 2023-03-03T13:32:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.294+0000 7f5ceaeda700 1 -- 172.21.15.7:0/414943485 wait complete. 2023-03-03T13:32:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.295+0000 7f5ceaeda700 1 Processor -- start 2023-03-03T13:32:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.295+0000 7f5ceaeda700 1 -- start start 2023-03-03T13:32:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.296+0000 7f5ceaeda700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce40abe20 0x7f5ce41ed570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.296+0000 7f5ceaeda700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce414f9d0 0x7f5ce40c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:08.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.296+0000 7f5ceaeda700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce414fda0 0x7f5ce406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.296+0000 7f5ceaeda700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce41e7240 con 0x7f5ce40abe20 2023-03-03T13:32:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.296+0000 7f5ceaeda700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce41edab0 con 0x7f5ce414f9d0 2023-03-03T13:32:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.296+0000 7f5ceaeda700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce406a4e0 con 0x7f5ce414fda0 2023-03-03T13:32:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.296+0000 7f5ce8c76700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce40abe20 0x7f5ce41ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.296+0000 7f5ce8c76700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce40abe20 0x7f5ce41ed570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45094/0 (socket says 172.21.15.7:45094) 2023-03-03T13:32:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.296+0000 7f5ce8c76700 1 -- 172.21.15.7:0/3979900846 learned_addr learned my addr 172.21.15.7:0/3979900846 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:32:08.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce3fff700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce414f9d0 0x7f5ce40c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:08.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce9477700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce414fda0 0x7f5ce406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:08.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce8c76700 1 -- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce414fda0 msgr2=0x7f5ce406f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:08.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce8c76700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce414fda0 0x7f5ce406f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:08.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce8c76700 1 -- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce414f9d0 msgr2=0x7f5ce40c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:08.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce8c76700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce414f9d0 0x7f5ce40c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:08.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce8c76700 1 -- 172.21.15.7:0/3979900846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd00094a0 con 0x7f5ce40abe20 2023-03-03T13:32:08.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce3fff700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce414f9d0 0x7f5ce40c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:32:08.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce8c76700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce40abe20 0x7f5ce41ed570 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80011e0 tx=0x7f5cd800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:32:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce17fa700 1 -- 172.21.15.7:0/3979900846 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8004c80 con 0x7f5ce40abe20 2023-03-03T13:32:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce17fa700 1 -- 172.21.15.7:0/3979900846 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd8004e00 con 0x7f5ce40abe20 2023-03-03T13:32:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ceaeda700 1 -- 172.21.15.7:0/3979900846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce40ac350 con 0x7f5ce40abe20 2023-03-03T13:32:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ce17fa700 1 -- 172.21.15.7:0/3979900846 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8007310 con 0x7f5ce40abe20 2023-03-03T13:32:08.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.297+0000 7f5ceaeda700 1 -- 172.21.15.7:0/3979900846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce40ab630 con 0x7f5ce40abe20 2023-03-03T13:32:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.298+0000 7f5ceaeda700 1 -- 172.21.15.7:0/3979900846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce4071fa0 con 0x7f5ce40abe20 2023-03-03T13:32:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.299+0000 7f5ce17fa700 1 -- 172.21.15.7:0/3979900846 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5cd801a020 con 0x7f5ce40abe20 2023-03-03T13:32:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.299+0000 7f5ce17fa700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cc80682f0 0x7f5cc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.299+0000 7f5ce17fa700 1 -- 172.21.15.7:0/3979900846 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5cd8085e20 con 0x7f5ce40abe20 2023-03-03T13:32:08.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.299+0000 7f5ce3fff700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cc80682f0 0x7f5cc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:08.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.300+0000 7f5ce3fff700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cc80682f0 0x7f5cc806a7a0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5ce41501d0 tx=0x7f5cd0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:32:08.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.303+0000 7f5ce17fa700 1 -- 172.21.15.7:0/3979900846 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5cd80521c0 con 0x7f5ce40abe20 2023-03-03T13:32:08.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.540+0000 7f5ceaeda700 1 -- 172.21.15.7:0/3979900846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ce404e6e0 con 0x7f5ce40abe20 2023-03-03T13:32:08.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.540+0000 7f5ce17fa700 1 -- 172.21.15.7:0/3979900846 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5cd80557e0 con 0x7f5ce40abe20 2023-03-03T13:32:08.543 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:32:08.543 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:32:08.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.542+0000 7f5ceaeda700 1 -- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cc80682f0 msgr2=0x7f5cc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:08.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.542+0000 7f5ceaeda700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cc80682f0 0x7f5cc806a7a0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5ce41501d0 tx=0x7f5cd0008030 comp rx=0 tx=0).stop 2023-03-03T13:32:08.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.542+0000 7f5ceaeda700 1 -- 172.21.15.7:0/3979900846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce40abe20 msgr2=0x7f5ce41ed570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:08.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.542+0000 7f5ceaeda700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce40abe20 0x7f5ce41ed570 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80011e0 tx=0x7f5cd800cae0 comp rx=0 tx=0).stop 2023-03-03T13:32:08.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.543+0000 7f5ceaeda700 1 -- 172.21.15.7:0/3979900846 shutdown_connections 2023-03-03T13:32:08.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.543+0000 7f5ceaeda700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce414fda0 0x7f5ce406f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:08.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.543+0000 7f5ceaeda700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5cc80682f0 0x7f5cc806a7a0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:08.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.543+0000 7f5ceaeda700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce40abe20 0x7f5ce41ed570 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:08.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.543+0000 7f5ceaeda700 1 --2- 172.21.15.7:0/3979900846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ce414f9d0 0x7f5ce40c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:08.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.543+0000 7f5ceaeda700 1 -- 172.21.15.7:0/3979900846 >> 172.21.15.7:0/3979900846 conn(0x7f5ce40c8900 msgr2=0x7f5ce414f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:32:08.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.543+0000 7f5ceaeda700 1 -- 172.21.15.7:0/3979900846 shutdown_connections 2023-03-03T13:32:08.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:08.543+0000 7f5ceaeda700 1 -- 172.21.15.7:0/3979900846 wait complete. 2023-03-03T13:32:08.546 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:32:08.556 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before thrashing 2023-03-03T13:32:13.268 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:13.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:17.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:32:17.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.695+0000 7f8e45c0d700 1 Processor -- start 2023-03-03T13:32:17.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.695+0000 7f8e45c0d700 1 -- start start 2023-03-03T13:32:17.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e45c0d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e40152e30 0x7f8e400c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:17.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e45c0d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e400beab0 0x7f8e400c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:17.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e45c0d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e40155d80 0x7f8e40156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:17.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e45c0d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e401523b0 con 0x7f8e400beab0 2023-03-03T13:32:17.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e45c0d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e40072010 con 0x7f8e40155d80 2023-03-03T13:32:17.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e45c0d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e400aaaf0 con 0x7f8e40152e30 2023-03-03T13:32:17.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e400beab0 0x7f8e400c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:17.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e400beab0 0x7f8e400c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45108/0 (socket says 172.21.15.7:45108) 2023-03-03T13:32:17.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3effd700 1 -- 172.21.15.7:0/3180441292 learned_addr learned my addr 172.21.15.7:0/3180441292 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:32:17.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3f7fe700 1 --2- 172.21.15.7:0/3180441292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e40152e30 0x7f8e400c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:17.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3ffff700 1 --2- 172.21.15.7:0/3180441292 >> v2:172.21.15.143:3300/0 conn(0x7f8e40155d80 0x7f8e40156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:17.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3effd700 1 -- 172.21.15.7:0/3180441292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e40152e30 msgr2=0x7f8e400c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:32:17.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3effd700 1 --2- 172.21.15.7:0/3180441292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e40152e30 0x7f8e400c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3effd700 1 -- 172.21.15.7:0/3180441292 >> v2:172.21.15.143:3300/0 conn(0x7f8e40155d80 msgr2=0x7f8e40156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:32:17.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3effd700 1 --2- 172.21.15.7:0/3180441292 >> v2:172.21.15.143:3300/0 conn(0x7f8e40155d80 0x7f8e40156220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3effd700 1 -- 172.21.15.7:0/3180441292 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e400aadd0 con 0x7f8e400beab0 2023-03-03T13:32:17.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.696+0000 7f8e3effd700 1 --2- 172.21.15.7:0/3180441292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e400beab0 0x7f8e400c4910 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f8e400c5970 tx=0x7f8e34009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad5a7b77698c217 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:32:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.698+0000 7f8e3e7fc700 1 -- 172.21.15.7:0/3180441292 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3400b870 con 0x7f8e400beab0 2023-03-03T13:32:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.698+0000 7f8e3e7fc700 1 -- 172.21.15.7:0/3180441292 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3400b9d0 con 0x7f8e400beab0 2023-03-03T13:32:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.698+0000 7f8e3e7fc700 1 -- 172.21.15.7:0/3180441292 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3400bd60 con 0x7f8e400beab0 2023-03-03T13:32:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.698+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3180441292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e400beab0 msgr2=0x7f8e400c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.698+0000 7f8e45c0d700 1 --2- 172.21.15.7:0/3180441292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e400beab0 0x7f8e400c4910 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f8e400c5970 tx=0x7f8e34009bd0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.699+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3180441292 shutdown_connections 2023-03-03T13:32:17.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.699+0000 7f8e45c0d700 1 --2- 172.21.15.7:0/3180441292 >> v2:172.21.15.143:3300/0 conn(0x7f8e40155d80 0x7f8e40156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.699+0000 7f8e45c0d700 1 --2- 172.21.15.7:0/3180441292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e40152e30 0x7f8e400c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.699+0000 7f8e45c0d700 1 --2- 172.21.15.7:0/3180441292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e400beab0 0x7f8e400c4910 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.699+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3180441292 >> 172.21.15.7:0/3180441292 conn(0x7f8e400736f0 msgr2=0x7f8e40073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:32:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.699+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3180441292 shutdown_connections 2023-03-03T13:32:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.699+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3180441292 wait complete. 2023-03-03T13:32:17.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.700+0000 7f8e45c0d700 1 Processor -- start 2023-03-03T13:32:17.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.700+0000 7f8e45c0d700 1 -- start start 2023-03-03T13:32:17.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e45c0d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e400beab0 0x7f8e400c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:17.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e45c0d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e40152e30 0x7f8e401e82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:17.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e45c0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e40155d80 0x7f8e401e8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:17.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e45c0d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e400ad860 con 0x7f8e40155d80 2023-03-03T13:32:17.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e45c0d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4014bca0 con 0x7f8e40152e30 2023-03-03T13:32:17.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e45c0d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e401e7ec0 con 0x7f8e400beab0 2023-03-03T13:32:17.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e40155d80 0x7f8e401e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:17.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e40155d80 0x7f8e401e8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45116/0 (socket says 172.21.15.7:45116) 2023-03-03T13:32:17.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3ffff700 1 -- 172.21.15.7:0/3549440669 learned_addr learned my addr 172.21.15.7:0/3549440669 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:32:17.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3effd700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e40152e30 0x7f8e401e82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:17.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3f7fe700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e400beab0 0x7f8e400c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3ffff700 1 -- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e400beab0 msgr2=0x7f8e400c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3ffff700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e400beab0 0x7f8e400c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3ffff700 1 -- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e40152e30 msgr2=0x7f8e401e82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3ffff700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e40152e30 0x7f8e401e82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3ffff700 1 -- 172.21.15.7:0/3549440669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e34009550 con 0x7f8e40155d80 2023-03-03T13:32:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.701+0000 7f8e3ffff700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e40155d80 0x7f8e401e8820 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800cd70 tx=0x7f8e3800a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:32:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.702+0000 7f8e27fff700 1 -- 172.21.15.7:0/3549440669 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38003a20 con 0x7f8e40155d80 2023-03-03T13:32:17.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.702+0000 7f8e27fff700 1 -- 172.21.15.7:0/3549440669 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e38003b80 con 0x7f8e40155d80 2023-03-03T13:32:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.702+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3549440669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e40156760 con 0x7f8e40155d80 2023-03-03T13:32:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.702+0000 7f8e27fff700 1 -- 172.21.15.7:0/3549440669 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38004020 con 0x7f8e40155d80 2023-03-03T13:32:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.702+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3549440669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e40071c70 con 0x7f8e40155d80 2023-03-03T13:32:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.703+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3549440669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e401568f0 con 0x7f8e40155d80 2023-03-03T13:32:17.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.703+0000 7f8e27fff700 1 -- 172.21.15.7:0/3549440669 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e3801c020 con 0x7f8e40155d80 2023-03-03T13:32:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.704+0000 7f8e27fff700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e2c068320 0x7f8e2c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:32:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.704+0000 7f8e27fff700 1 -- 172.21.15.7:0/3549440669 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8e38012070 con 0x7f8e40155d80 2023-03-03T13:32:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.704+0000 7f8e3f7fe700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e2c068320 0x7f8e2c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:32:17.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.704+0000 7f8e3f7fe700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e2c068320 0x7f8e2c06a7d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000ab70 tx=0x7f8e30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:32:17.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.708+0000 7f8e27fff700 1 -- 172.21.15.7:0/3549440669 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e38052b40 con 0x7f8e40155d80 2023-03-03T13:32:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.946+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3549440669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e401e8000 con 0x7f8e40155d80 2023-03-03T13:32:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.947+0000 7f8e27fff700 1 -- 172.21.15.7:0/3549440669 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8e401e8000 con 0x7f8e40155d80 2023-03-03T13:32:17.949 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:32:17.949 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:32:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e2c068320 msgr2=0x7f8e2c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e2c068320 0x7f8e2c06a7d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000ab70 tx=0x7f8e30008040 comp rx=0 tx=0).stop 2023-03-03T13:32:17.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3549440669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e40155d80 msgr2=0x7f8e401e8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:32:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e40155d80 0x7f8e401e8820 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800cd70 tx=0x7f8e3800a570 comp rx=0 tx=0).stop 2023-03-03T13:32:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3549440669 shutdown_connections 2023-03-03T13:32:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e2c068320 0x7f8e2c06a7d0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e40155d80 0x7f8e401e8820 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e400beab0 0x7f8e400c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 --2- 172.21.15.7:0/3549440669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e40152e30 0x7f8e401e82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:32:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3549440669 >> 172.21.15.7:0/3549440669 conn(0x7f8e400736f0 msgr2=0x7f8e400a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:32:17.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3549440669 shutdown_connections 2023-03-03T13:32:17.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:32:17.949+0000 7f8e45c0d700 1 -- 172.21.15.7:0/3549440669 wait complete. 2023-03-03T13:32:17.953 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:32:17.963 INFO:tasks.mds_thrash.fs.[a]:waiting for 95 secs before thrashing 2023-03-03T13:32:43.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:32:43.352 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:13.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:13.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:43.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:43.628 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:33:52.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:33:53.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.101+0000 7f8a967a4700 1 Processor -- start 2023-03-03T13:33:53.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.101+0000 7f8a967a4700 1 -- start start 2023-03-03T13:33:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a967a4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a900c25c0 0x7f8a900a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a967a4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a900c1b40 0x7f8a90071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a967a4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a90071620 con 0x7f8a900c1770 2023-03-03T13:33:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a967a4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a900c39e0 con 0x7f8a900c1b40 2023-03-03T13:33:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a967a4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9006c100 con 0x7f8a900c25c0 2023-03-03T13:33:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a94d41700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a900c1b40 0x7f8a90071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a8ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8a900c1770 0x7f8a90154e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39138/0 (socket says 172.21.15.7:39138) 2023-03-03T13:33:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a8ffff700 1 -- 172.21.15.7:0/587868696 learned_addr learned my addr 172.21.15.7:0/587868696 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:33:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a8f7fe700 1 --2- 172.21.15.7:0/587868696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a900c25c0 0x7f8a900a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a94d41700 1 -- 172.21.15.7:0/587868696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a900c25c0 msgr2=0x7f8a900a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:33:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a94d41700 1 --2- 172.21.15.7:0/587868696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a900c25c0 0x7f8a900a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a94d41700 1 -- 172.21.15.7:0/587868696 >> v1:172.21.15.7:6789/0 conn(0x7f8a900c1770 legacy=0x7f8a90154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.102+0000 7f8a94d41700 1 -- 172.21.15.7:0/587868696 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a900c1fc0 con 0x7f8a900c1b40 2023-03-03T13:33:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a94d41700 1 --2- 172.21.15.7:0/587868696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1b40 0x7f8a90071ab0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f8a88009d90 tx=0x7f8a8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c7fb5e47700dd36 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:33:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a8effd700 1 -- 172.21.15.7:0/587868696 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8800e070 con 0x7f8a900c1b40 2023-03-03T13:33:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a8effd700 1 -- 172.21.15.7:0/587868696 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8800bbd0 con 0x7f8a900c1b40 2023-03-03T13:33:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a8effd700 1 -- 172.21.15.7:0/587868696 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88012430 con 0x7f8a900c1b40 2023-03-03T13:33:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a967a4700 1 -- 172.21.15.7:0/587868696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1b40 msgr2=0x7f8a90071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a967a4700 1 --2- 172.21.15.7:0/587868696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1b40 0x7f8a90071ab0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f8a88009d90 tx=0x7f8a8800b330 comp rx=0 tx=0).stop 2023-03-03T13:33:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a967a4700 1 -- 172.21.15.7:0/587868696 shutdown_connections 2023-03-03T13:33:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a967a4700 1 --2- 172.21.15.7:0/587868696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1b40 0x7f8a90071ab0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a967a4700 1 --2- 172.21.15.7:0/587868696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a900c25c0 0x7f8a900a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a967a4700 1 -- 172.21.15.7:0/587868696 >> 172.21.15.7:0/587868696 conn(0x7f8a900c8400 msgr2=0x7f8a900c8800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:33:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a967a4700 1 -- 172.21.15.7:0/587868696 shutdown_connections 2023-03-03T13:33:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.103+0000 7f8a967a4700 1 -- 172.21.15.7:0/587868696 wait complete. 2023-03-03T13:33:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.105+0000 7f8a967a4700 1 Processor -- start 2023-03-03T13:33:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.105+0000 7f8a967a4700 1 -- start start 2023-03-03T13:33:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.105+0000 7f8a967a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1770 0x7f8a9014a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.105+0000 7f8a967a4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a900c1b40 0x7f8a9014bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.105+0000 7f8a967a4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a900c25c0 0x7f8a901ea200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a967a4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9014a790 con 0x7f8a900c1770 2023-03-03T13:33:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a967a4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a901ed500 con 0x7f8a900c1b40 2023-03-03T13:33:53.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a967a4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a901ee360 con 0x7f8a900c25c0 2023-03-03T13:33:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1770 0x7f8a9014a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1770 0x7f8a9014a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45164/0 (socket says 172.21.15.7:45164) 2023-03-03T13:33:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8ffff700 1 -- 172.21.15.7:0/3283944434 learned_addr learned my addr 172.21.15.7:0/3283944434 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:33:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8f7fe700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a900c1b40 0x7f8a9014bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a94d41700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a900c25c0 0x7f8a901ea200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8ffff700 1 -- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a900c25c0 msgr2=0x7f8a901ea200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8ffff700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a900c25c0 0x7f8a901ea200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:53.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8ffff700 1 -- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a900c1b40 msgr2=0x7f8a9014bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8ffff700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a900c1b40 0x7f8a9014bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8ffff700 1 -- 172.21.15.7:0/3283944434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a88009a70 con 0x7f8a900c1770 2023-03-03T13:33:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8ffff700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1770 0x7f8a9014a250 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000c480 tx=0x7f8a8000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:33:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8cff9700 1 -- 172.21.15.7:0/3283944434 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a80005d70 con 0x7f8a900c1770 2023-03-03T13:33:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8cff9700 1 -- 172.21.15.7:0/3283944434 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8000ce90 con 0x7f8a900c1770 2023-03-03T13:33:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a967a4700 1 -- 172.21.15.7:0/3283944434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a900c3190 con 0x7f8a900c1770 2023-03-03T13:33:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a8cff9700 1 -- 172.21.15.7:0/3283944434 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a800072d0 con 0x7f8a900c1770 2023-03-03T13:33:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.106+0000 7f8a967a4700 1 -- 172.21.15.7:0/3283944434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a900c2eb0 con 0x7f8a900c1770 2023-03-03T13:33:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.107+0000 7f8a967a4700 1 -- 172.21.15.7:0/3283944434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a900c3470 con 0x7f8a900c1770 2023-03-03T13:33:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.108+0000 7f8a8cff9700 1 -- 172.21.15.7:0/3283944434 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a80019030 con 0x7f8a900c1770 2023-03-03T13:33:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.108+0000 7f8a8cff9700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a70068320 0x7f8a7006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.108+0000 7f8a8cff9700 1 -- 172.21.15.7:0/3283944434 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8a800851e0 con 0x7f8a900c1770 2023-03-03T13:33:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.108+0000 7f8a8f7fe700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a70068320 0x7f8a7006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.109+0000 7f8a8f7fe700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a70068320 0x7f8a7006a7d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f8a90152000 tx=0x7f8a78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:33:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.113+0000 7f8a8cff9700 1 -- 172.21.15.7:0/3283944434 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8a8004b5e0 con 0x7f8a900c1770 2023-03-03T13:33:53.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.349+0000 7f8a967a4700 1 -- 172.21.15.7:0/3283944434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a900a9570 con 0x7f8a900c1770 2023-03-03T13:33:53.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.349+0000 7f8a8cff9700 1 -- 172.21.15.7:0/3283944434 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8a80051630 con 0x7f8a900c1770 2023-03-03T13:33:53.351 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:33:53.352 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:33:53.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.351+0000 7f8a967a4700 1 -- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a70068320 msgr2=0x7f8a7006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.351+0000 7f8a967a4700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a70068320 0x7f8a7006a7d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f8a90152000 tx=0x7f8a78008040 comp rx=0 tx=0).stop 2023-03-03T13:33:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.351+0000 7f8a967a4700 1 -- 172.21.15.7:0/3283944434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1770 msgr2=0x7f8a9014a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.351+0000 7f8a967a4700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1770 0x7f8a9014a250 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000c480 tx=0x7f8a8000c790 comp rx=0 tx=0).stop 2023-03-03T13:33:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.351+0000 7f8a967a4700 1 -- 172.21.15.7:0/3283944434 shutdown_connections 2023-03-03T13:33:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.351+0000 7f8a967a4700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a900c25c0 0x7f8a901ea200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.351+0000 7f8a967a4700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a70068320 0x7f8a7006a7d0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.351+0000 7f8a967a4700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a900c1770 0x7f8a9014a250 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.351+0000 7f8a967a4700 1 --2- 172.21.15.7:0/3283944434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a900c1b40 0x7f8a9014bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.351+0000 7f8a967a4700 1 -- 172.21.15.7:0/3283944434 >> 172.21.15.7:0/3283944434 conn(0x7f8a900c8400 msgr2=0x7f8a9014cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:33:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.352+0000 7f8a967a4700 1 -- 172.21.15.7:0/3283944434 shutdown_connections 2023-03-03T13:33:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:53.352+0000 7f8a967a4700 1 -- 172.21.15.7:0/3283944434 wait complete. 2023-03-03T13:33:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:33:53.365 INFO:tasks.mds_thrash.fs.[a]:waiting for 45 secs before thrashing 2023-03-03T13:33:54.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:33:54.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.601+0000 7fd160e6e700 1 Processor -- start 2023-03-03T13:33:54.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.601+0000 7fd160e6e700 1 -- start start 2023-03-03T13:33:54.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.602+0000 7fd160e6e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd15c14fc50 0x7fd15c0c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:54.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.602+0000 7fd160e6e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c0ad2b0 con 0x7fd15c06d240 2023-03-03T13:33:54.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.602+0000 7fd160e6e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c156420 con 0x7fd15c0c3e30 2023-03-03T13:33:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.602+0000 7fd160e6e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15c0c30c0 con 0x7fd15c14fc50 2023-03-03T13:33:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.602+0000 7fd159d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd15c06d240 0x7fd15c0c27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39152/0 (socket says 172.21.15.7:39152) 2023-03-03T13:33:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.602+0000 7fd159d9b700 1 -- 172.21.15.7:0/2422608229 learned_addr learned my addr 172.21.15.7:0/2422608229 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:33:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.602+0000 7fd15a59c700 1 --2- 172.21.15.7:0/2422608229 >> v2:172.21.15.143:3300/0 conn(0x7fd15c14fc50 0x7fd15c0c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.602+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504058062 0 0) 0x7fd15c0ad2b0 con 0x7fd15c06d240 2023-03-03T13:33:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.602+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1480035b0 con 0x7fd15c06d240 2023-03-03T13:33:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4079498171 0 0) 0x7fd1480035b0 con 0x7fd15c06d240 2023-03-03T13:33:54.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 >> v2:172.21.15.143:3300/0 conn(0x7fd15c14fc50 msgr2=0x7fd15c0c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:33:54.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 --2- 172.21.15.7:0/2422608229 >> v2:172.21.15.143:3300/0 conn(0x7fd15c14fc50 0x7fd15c0c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 >> v1:172.21.15.143:6789/0 conn(0x7fd15c0c3e30 legacy=0x7fd15c15cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:54.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c0ad520 con 0x7fd15c06d240 2023-03-03T13:33:54.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd14c001670 con 0x7fd15c06d240 2023-03-03T13:33:54.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 >> v1:172.21.15.7:6789/0 conn(0x7fd15c06d240 legacy=0x7fd15c0c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:54.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd148003a70 0x7fd148003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd148004850 0x7fd148004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148005270 0x7fd14800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800c180 con 0x7fd148005270 2023-03-03T13:33:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800c2f0 con 0x7fd148004850 2023-03-03T13:33:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800c4b0 con 0x7fd148003a70 2023-03-03T13:33:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd159d9b700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd148004850 0x7fd148004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15a59c700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd148003a70 0x7fd148003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15ad9d700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148005270 0x7fd14800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:54.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15ad9d700 1 -- 172.21.15.7:0/2422608229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd148003a70 msgr2=0x7fd148003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:33:54.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15ad9d700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd148003a70 0x7fd148003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15ad9d700 1 -- 172.21.15.7:0/2422608229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd148004850 msgr2=0x7fd148004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:33:54.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15ad9d700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd148004850 0x7fd148004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15ad9d700 1 -- 172.21.15.7:0/2422608229 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480035b0 con 0x7fd148005270 2023-03-03T13:33:54.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd159d9b700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd148004850 0x7fd148004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:33:54.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15ad9d700 1 -- 172.21.15.7:0/2422608229 reap_dead start 2023-03-03T13:33:54.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.603+0000 7fd15ad9d700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148005270 0x7fd14800ba10 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd15400c370 tx=0x7fd148005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c445317e2457223c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:33:54.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.604+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154001f20 con 0x7fd148005270 2023-03-03T13:33:54.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.604+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15400ce10 con 0x7fd148005270 2023-03-03T13:33:54.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.604+0000 7fd15959a700 1 -- 172.21.15.7:0/2422608229 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154003c40 con 0x7fd148005270 2023-03-03T13:33:54.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.604+0000 7fd160e6e700 1 -- 172.21.15.7:0/2422608229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148005270 msgr2=0x7fd14800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:54.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.604+0000 7fd160e6e700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148005270 0x7fd14800ba10 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd15400c370 tx=0x7fd148005880 comp rx=0 tx=0).stop 2023-03-03T13:33:54.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.604+0000 7fd160e6e700 1 -- 172.21.15.7:0/2422608229 shutdown_connections 2023-03-03T13:33:54.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.604+0000 7fd160e6e700 1 --2- 172.21.15.7:0/2422608229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd148005270 0x7fd14800ba10 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.604+0000 7fd160e6e700 1 -- 172.21.15.7:0/2422608229 >> 172.21.15.7:0/2422608229 conn(0x7fd15c06c110 msgr2=0x7fd15c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:33:54.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.604+0000 7fd160e6e700 1 -- 172.21.15.7:0/2422608229 shutdown_connections 2023-03-03T13:33:54.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.604+0000 7fd160e6e700 1 -- 172.21.15.7:0/2422608229 wait complete. 2023-03-03T13:33:54.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.606+0000 7fd160e6e700 1 Processor -- start 2023-03-03T13:33:54.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.606+0000 7fd160e6e700 1 -- start start 2023-03-03T13:33:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd160e6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd148005270 0x7fd15c1534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd160e6e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd15c155640 0x7fd15c0c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd160e6e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd15c071f30 0x7fd15c1ea2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd160e6e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c1edf60 con 0x7fd15c071f30 2023-03-03T13:33:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd160e6e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c1e9e80 con 0x7fd148005270 2023-03-03T13:33:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd160e6e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c0711e0 con 0x7fd15c155640 2023-03-03T13:33:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd15ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd15c071f30 0x7fd15c1ea2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd15ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd15c071f30 0x7fd15c1ea2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45188/0 (socket says 172.21.15.7:45188) 2023-03-03T13:33:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd15ad9d700 1 -- 172.21.15.7:0/2509646991 learned_addr learned my addr 172.21.15.7:0/2509646991 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:33:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd15a59c700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd148005270 0x7fd15c1534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd159d9b700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd15c155640 0x7fd15c0c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd15ad9d700 1 -- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd15c155640 msgr2=0x7fd15c0c68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd15ad9d700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd15c155640 0x7fd15c0c68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd15ad9d700 1 -- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd148005270 msgr2=0x7fd15c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd15ad9d700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd148005270 0x7fd15c1534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd15ad9d700 1 -- 172.21.15.7:0/2509646991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c156690 con 0x7fd15c071f30 2023-03-03T13:33:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.607+0000 7fd159d9b700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd15c155640 0x7fd15c0c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:33:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.608+0000 7fd15ad9d700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd15c071f30 0x7fd15c1ea2a0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fd15c070190 tx=0x7fd1540016a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:33:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.608+0000 7fd152ffd700 1 -- 172.21.15.7:0/2509646991 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15401a040 con 0x7fd15c071f30 2023-03-03T13:33:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.608+0000 7fd152ffd700 1 -- 172.21.15.7:0/2509646991 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1540052e0 con 0x7fd15c071f30 2023-03-03T13:33:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.608+0000 7fd160e6e700 1 -- 172.21.15.7:0/2509646991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c074220 con 0x7fd15c071f30 2023-03-03T13:33:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.608+0000 7fd152ffd700 1 -- 172.21.15.7:0/2509646991 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154005690 con 0x7fd15c071f30 2023-03-03T13:33:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.608+0000 7fd160e6e700 1 -- 172.21.15.7:0/2509646991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c0acf60 con 0x7fd15c071f30 2023-03-03T13:33:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.609+0000 7fd160e6e700 1 -- 172.21.15.7:0/2509646991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c074bb0 con 0x7fd15c071f30 2023-03-03T13:33:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.609+0000 7fd152ffd700 1 -- 172.21.15.7:0/2509646991 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd154010050 con 0x7fd15c071f30 2023-03-03T13:33:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.610+0000 7fd152ffd700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c0682f0 0x7fd13c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:33:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.610+0000 7fd152ffd700 1 -- 172.21.15.7:0/2509646991 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd154086740 con 0x7fd15c071f30 2023-03-03T13:33:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.610+0000 7fd15a59c700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c0682f0 0x7fd13c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:33:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.610+0000 7fd15a59c700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c0682f0 0x7fd13c06a7a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fd15c149a90 tx=0x7fd144009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:33:54.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.614+0000 7fd152ffd700 1 -- 172.21.15.7:0/2509646991 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd154052aa0 con 0x7fd15c071f30 2023-03-03T13:33:54.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.855+0000 7fd160e6e700 1 -- 172.21.15.7:0/2509646991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd15c1e9fc0 con 0x7fd15c071f30 2023-03-03T13:33:54.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.855+0000 7fd152ffd700 1 -- 172.21.15.7:0/2509646991 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd15c1e9fc0 con 0x7fd15c071f30 2023-03-03T13:33:54.857 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:33:54.857 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:33:54.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 -- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c0682f0 msgr2=0x7fd13c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:54.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c0682f0 0x7fd13c06a7a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fd15c149a90 tx=0x7fd144009210 comp rx=0 tx=0).stop 2023-03-03T13:33:54.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 -- 172.21.15.7:0/2509646991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd15c071f30 msgr2=0x7fd15c1ea2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:33:54.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd15c071f30 0x7fd15c1ea2a0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fd15c070190 tx=0x7fd1540016a0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 -- 172.21.15.7:0/2509646991 shutdown_connections 2023-03-03T13:33:54.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c0682f0 0x7fd13c06a7a0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd15c071f30 0x7fd15c1ea2a0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd148005270 0x7fd15c1534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 --2- 172.21.15.7:0/2509646991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd15c155640 0x7fd15c0c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:33:54.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 -- 172.21.15.7:0/2509646991 >> 172.21.15.7:0/2509646991 conn(0x7fd15c06c110 msgr2=0x7fd15c14ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:33:54.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 -- 172.21.15.7:0/2509646991 shutdown_connections 2023-03-03T13:33:54.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:33:54.857+0000 7fd160e6e700 1 -- 172.21.15.7:0/2509646991 wait complete. 2023-03-03T13:33:54.861 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:33:54.871 INFO:tasks.mds_thrash.fs.[b]:waiting for 30 secs before thrashing 2023-03-03T13:34:13.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:13.718 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:24.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:34:25.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.008+0000 7f31b63dd700 1 Processor -- start 2023-03-03T13:34:25.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.008+0000 7f31b63dd700 1 -- start start 2023-03-03T13:34:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.009+0000 7f31b63dd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f31b00c00e0 0x7f31b00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.009+0000 7f31b63dd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f31b0154ff0 0x7f31b00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.009+0000 7f31b63dd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b00c7310 con 0x7f31b00caa30 2023-03-03T13:34:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.009+0000 7f31b63dd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b0155430 con 0x7f31b00c00e0 2023-03-03T13:34:25.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.009+0000 7f31b63dd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b00bfd10 con 0x7f31b0154ff0 2023-03-03T13:34:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.009+0000 7f31af7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f31b00c00e0 0x7f31b00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.009+0000 7f31af7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f31b00c00e0 0x7f31b00c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45220/0 (socket says 172.21.15.7:45220) 2023-03-03T13:34:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.009+0000 7f31af7fe700 1 -- 172.21.15.7:0/2451171509 learned_addr learned my addr 172.21.15.7:0/2451171509 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:34:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.010+0000 7f31b497a700 1 --2- 172.21.15.7:0/2451171509 >> v2:172.21.15.143:3300/0 conn(0x7f31b0154ff0 0x7f31b00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.010+0000 7f31af7fe700 1 -- 172.21.15.7:0/2451171509 >> v2:172.21.15.143:3300/0 conn(0x7f31b0154ff0 msgr2=0x7f31b00cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:34:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.010+0000 7f31af7fe700 1 --2- 172.21.15.7:0/2451171509 >> v2:172.21.15.143:3300/0 conn(0x7f31b0154ff0 0x7f31b00cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:25.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.010+0000 7f31af7fe700 1 -- 172.21.15.7:0/2451171509 >> v1:172.21.15.143:6789/0 conn(0x7f31b00caa30 legacy=0x7f31b00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:34:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.010+0000 7f31af7fe700 1 -- 172.21.15.7:0/2451171509 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b00bfa30 con 0x7f31b00c00e0 2023-03-03T13:34:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.010+0000 7f31af7fe700 1 --2- 172.21.15.7:0/2451171509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00c00e0 0x7f31b00c5c60 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f31b0071530 tx=0x7f31a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ff1aa82662673f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:34:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.010+0000 7f31aeffd700 1 -- 172.21.15.7:0/2451171509 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4005e30 con 0x7f31b00c00e0 2023-03-03T13:34:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.010+0000 7f31aeffd700 1 -- 172.21.15.7:0/2451171509 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a4006030 con 0x7f31b00c00e0 2023-03-03T13:34:25.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.010+0000 7f31aeffd700 1 -- 172.21.15.7:0/2451171509 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a40063e0 con 0x7f31b00c00e0 2023-03-03T13:34:25.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.011+0000 7f31b63dd700 1 -- 172.21.15.7:0/2451171509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00c00e0 msgr2=0x7f31b00c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:25.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.011+0000 7f31b63dd700 1 --2- 172.21.15.7:0/2451171509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00c00e0 0x7f31b00c5c60 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f31b0071530 tx=0x7f31a4009bd0 comp rx=0 tx=0).stop 2023-03-03T13:34:25.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.011+0000 7f31b63dd700 1 -- 172.21.15.7:0/2451171509 shutdown_connections 2023-03-03T13:34:25.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.011+0000 7f31b63dd700 1 --2- 172.21.15.7:0/2451171509 >> v2:172.21.15.143:3300/0 conn(0x7f31b0154ff0 0x7f31b00cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:25.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.011+0000 7f31b63dd700 1 --2- 172.21.15.7:0/2451171509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00c00e0 0x7f31b00c5c60 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:25.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.011+0000 7f31b63dd700 1 -- 172.21.15.7:0/2451171509 >> 172.21.15.7:0/2451171509 conn(0x7f31b0066e40 msgr2=0x7f31b0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:34:25.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.011+0000 7f31b63dd700 1 -- 172.21.15.7:0/2451171509 shutdown_connections 2023-03-03T13:34:25.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.011+0000 7f31b63dd700 1 -- 172.21.15.7:0/2451171509 wait complete. 2023-03-03T13:34:25.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.012+0000 7f31b63dd700 1 Processor -- start 2023-03-03T13:34:25.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.012+0000 7f31b63dd700 1 -- start start 2023-03-03T13:34:25.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31b63dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31b00c00e0 0x7f31b00c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:25.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31b63dd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00caa30 0x7f31b01eb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:25.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31b63dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31b0154ff0 0x7f31b01eb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:25.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31b63dd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b00c20a0 con 0x7f31b00caa30 2023-03-03T13:34:25.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31b63dd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b0156430 con 0x7f31b0154ff0 2023-03-03T13:34:25.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31b63dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b01dfcf0 con 0x7f31b00c00e0 2023-03-03T13:34:25.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31af7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00caa30 0x7f31b01eb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:25.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31af7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00caa30 0x7f31b01eb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45228/0 (socket says 172.21.15.7:45228) 2023-03-03T13:34:25.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31af7fe700 1 -- 172.21.15.7:0/2747917626 learned_addr learned my addr 172.21.15.7:0/2747917626 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:34:25.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31b497a700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31b0154ff0 0x7f31b01eb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:25.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31af7fe700 1 -- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31b00c00e0 msgr2=0x7f31b00c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:25.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31affff700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31b00c00e0 0x7f31b00c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:25.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31af7fe700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31b00c00e0 0x7f31b00c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:25.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31af7fe700 1 -- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31b0154ff0 msgr2=0x7f31b01eb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:25.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31af7fe700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31b0154ff0 0x7f31b01eb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:25.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31af7fe700 1 -- 172.21.15.7:0/2747917626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a4009500 con 0x7f31b00caa30 2023-03-03T13:34:25.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31b497a700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31b0154ff0 0x7f31b01eb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:34:25.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31affff700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31b00c00e0 0x7f31b00c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:34:25.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31af7fe700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00caa30 0x7f31b01eb4b0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f31a4006000 tx=0x7f31a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:34:25.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.013+0000 7f31acff9700 1 -- 172.21.15.7:0/2747917626 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4004250 con 0x7f31b00caa30 2023-03-03T13:34:25.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.014+0000 7f31acff9700 1 -- 172.21.15.7:0/2747917626 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a40043b0 con 0x7f31b00caa30 2023-03-03T13:34:25.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.014+0000 7f31b63dd700 1 -- 172.21.15.7:0/2747917626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b01547f0 con 0x7f31b00caa30 2023-03-03T13:34:25.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.014+0000 7f31acff9700 1 -- 172.21.15.7:0/2747917626 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a40036a0 con 0x7f31b00caa30 2023-03-03T13:34:25.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.014+0000 7f31b63dd700 1 -- 172.21.15.7:0/2747917626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b0154510 con 0x7f31b00caa30 2023-03-03T13:34:25.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.015+0000 7f31b63dd700 1 -- 172.21.15.7:0/2747917626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b0154ad0 con 0x7f31b00caa30 2023-03-03T13:34:25.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.015+0000 7f31acff9700 1 -- 172.21.15.7:0/2747917626 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f31a400f050 con 0x7f31b00caa30 2023-03-03T13:34:25.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.016+0000 7f31acff9700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f319c0682f0 0x7f319c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:25.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.016+0000 7f31acff9700 1 -- 172.21.15.7:0/2747917626 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f31a4088020 con 0x7f31b00caa30 2023-03-03T13:34:25.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.016+0000 7f31affff700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f319c0682f0 0x7f319c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:25.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.016+0000 7f31affff700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f319c0682f0 0x7f319c06a7a0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f31b014dfa0 tx=0x7f31a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:34:25.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.020+0000 7f31acff9700 1 -- 172.21.15.7:0/2747917626 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f31a4054460 con 0x7f31b00caa30 2023-03-03T13:34:25.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.255+0000 7f31b63dd700 1 -- 172.21.15.7:0/2747917626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31b01df5d0 con 0x7f31b00caa30 2023-03-03T13:34:25.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.256+0000 7f31acff9700 1 -- 172.21.15.7:0/2747917626 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f31b01df5d0 con 0x7f31b00caa30 2023-03-03T13:34:25.259 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:34:25.259 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:34:25.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 -- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f319c0682f0 msgr2=0x7f319c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:25.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f319c0682f0 0x7f319c06a7a0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f31b014dfa0 tx=0x7f31a0009040 comp rx=0 tx=0).stop 2023-03-03T13:34:25.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 -- 172.21.15.7:0/2747917626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00caa30 msgr2=0x7f31b01eb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:25.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00caa30 0x7f31b01eb4b0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f31a4006000 tx=0x7f31a4004750 comp rx=0 tx=0).stop 2023-03-03T13:34:25.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 -- 172.21.15.7:0/2747917626 shutdown_connections 2023-03-03T13:34:25.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31b0154ff0 0x7f31b01eb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:25.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31b00c00e0 0x7f31b00c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:25.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f319c0682f0 0x7f319c06a7a0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:25.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 --2- 172.21.15.7:0/2747917626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b00caa30 0x7f31b01eb4b0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:25.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 -- 172.21.15.7:0/2747917626 >> 172.21.15.7:0/2747917626 conn(0x7f31b0066e40 msgr2=0x7f31b0153040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:34:25.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 -- 172.21.15.7:0/2747917626 shutdown_connections 2023-03-03T13:34:25.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:25.258+0000 7f31b63dd700 1 -- 172.21.15.7:0/2747917626 wait complete. 2023-03-03T13:34:25.263 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:34:25.273 INFO:tasks.mds_thrash.fs.[b]:waiting for 100 secs before thrashing 2023-03-03T13:34:38.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:34:38.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.503+0000 7fa2910a6700 1 Processor -- start 2023-03-03T13:34:38.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.503+0000 7fa2910a6700 1 -- start start 2023-03-03T13:34:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.504+0000 7fa2910a6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c14e560 0x7fa28c071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.504+0000 7fa2910a6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28c072920 con 0x7fa28c0c4840 2023-03-03T13:34:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.504+0000 7fa2910a6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28c06bef0 con 0x7fa28c0c43d0 2023-03-03T13:34:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.504+0000 7fa2910a6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c06b910 con 0x7fa28c14e560 2023-03-03T13:34:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.504+0000 7fa28b59e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa28c0c4840 0x7fa28c15ef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39220/0 (socket says 172.21.15.7:39220) 2023-03-03T13:34:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.504+0000 7fa28b59e700 1 -- 172.21.15.7:0/1780187250 learned_addr learned my addr 172.21.15.7:0/1780187250 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:34:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.504+0000 7fa28ad9d700 1 --2- 172.21.15.7:0/1780187250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c14e560 0x7fa28c071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.504+0000 7fa289d9b700 1 -- 172.21.15.7:0/1780187250 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981118739 0 0) 0x7fa284001490 con 0x7fa28c0c4840 2023-03-03T13:34:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.504+0000 7fa289d9b700 1 -- 172.21.15.7:0/1780187250 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2740035b0 con 0x7fa28c0c4840 2023-03-03T13:34:38.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.505+0000 7fa289d9b700 1 -- 172.21.15.7:0/1780187250 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672903406 0 0) 0x7fa28c06bef0 con 0x7fa28c0c43d0 2023-03-03T13:34:38.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.505+0000 7fa289d9b700 1 -- 172.21.15.7:0/1780187250 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa284001490 con 0x7fa28c0c43d0 2023-03-03T13:34:38.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.505+0000 7fa28ad9d700 1 -- 172.21.15.7:0/1780187250 >> v1:172.21.15.143:6789/0 conn(0x7fa28c0c43d0 legacy=0x7fa28c154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:38.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.505+0000 7fa28ad9d700 1 -- 172.21.15.7:0/1780187250 >> v1:172.21.15.7:6789/0 conn(0x7fa28c0c4840 legacy=0x7fa28c15ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:38.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.505+0000 7fa289d9b700 1 -- 172.21.15.7:0/1780187250 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 899609377 0 0) 0x7fa2740035b0 con 0x7fa28c0c4840 2023-03-03T13:34:38.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.505+0000 7fa28ad9d700 1 -- 172.21.15.7:0/1780187250 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c0c4ca0 con 0x7fa28c14e560 2023-03-03T13:34:38.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.505+0000 7fa28ad9d700 1 --2- 172.21.15.7:0/1780187250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c14e560 0x7fa28c071c70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa280009870 tx=0x7fa280009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef5aed93b7deeeca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:34:38.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.505+0000 7fa289d9b700 1 -- 172.21.15.7:0/1780187250 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa284001b80 con 0x7fa28c14e560 2023-03-03T13:34:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.505+0000 7fa289d9b700 1 -- 172.21.15.7:0/1780187250 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa280004030 con 0x7fa28c14e560 2023-03-03T13:34:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.505+0000 7fa289d9b700 1 -- 172.21.15.7:0/1780187250 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2800043c0 con 0x7fa28c14e560 2023-03-03T13:34:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.506+0000 7fa2910a6700 1 -- 172.21.15.7:0/1780187250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c14e560 msgr2=0x7fa28c071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.506+0000 7fa2910a6700 1 --2- 172.21.15.7:0/1780187250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c14e560 0x7fa28c071c70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa280009870 tx=0x7fa280009b80 comp rx=0 tx=0).stop 2023-03-03T13:34:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.506+0000 7fa2910a6700 1 -- 172.21.15.7:0/1780187250 shutdown_connections 2023-03-03T13:34:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.506+0000 7fa2910a6700 1 --2- 172.21.15.7:0/1780187250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c14e560 0x7fa28c071c70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.506+0000 7fa2910a6700 1 -- 172.21.15.7:0/1780187250 >> 172.21.15.7:0/1780187250 conn(0x7fa28c0736f0 msgr2=0x7fa28c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:34:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.506+0000 7fa2910a6700 1 -- 172.21.15.7:0/1780187250 shutdown_connections 2023-03-03T13:34:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.506+0000 7fa2910a6700 1 -- 172.21.15.7:0/1780187250 wait complete. 2023-03-03T13:34:38.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.507+0000 7fa2910a6700 1 Processor -- start 2023-03-03T13:34:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.507+0000 7fa2910a6700 1 -- start start 2023-03-03T13:34:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa2910a6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c0c43d0 0x7fa28c14f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa2910a6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa28c0c4840 0x7fa28c0ad630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa2910a6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa28c14e560 0x7fa28c0bf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa2910a6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c14ff90 con 0x7fa28c14e560 2023-03-03T13:34:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa2910a6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c1ea350 con 0x7fa28c0c4840 2023-03-03T13:34:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa2910a6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c1e9a60 con 0x7fa28c0c43d0 2023-03-03T13:34:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa28b59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa28c14e560 0x7fa28c0bf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa28b59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa28c14e560 0x7fa28c0bf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45244/0 (socket says 172.21.15.7:45244) 2023-03-03T13:34:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa28b59e700 1 -- 172.21.15.7:0/2201741232 learned_addr learned my addr 172.21.15.7:0/2201741232 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:34:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa28ad9d700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c0c43d0 0x7fa28c14f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa28a59c700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa28c0c4840 0x7fa28c0ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa28b59e700 1 -- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c0c43d0 msgr2=0x7fa28c14f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa28b59e700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c0c43d0 0x7fa28c14f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa28b59e700 1 -- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa28c0c4840 msgr2=0x7fa28c0ad630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa28b59e700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa28c0c4840 0x7fa28c0ad630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.508+0000 7fa28b59e700 1 -- 172.21.15.7:0/2201741232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2800094a0 con 0x7fa28c14e560 2023-03-03T13:34:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.509+0000 7fa28b59e700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa28c14e560 0x7fa28c0bf600 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fa284001b80 tx=0x7fa28400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:34:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.509+0000 7fa2737fe700 1 -- 172.21.15.7:0/2201741232 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28400ae20 con 0x7fa28c14e560 2023-03-03T13:34:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.509+0000 7fa2737fe700 1 -- 172.21.15.7:0/2201741232 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2840070d0 con 0x7fa28c14e560 2023-03-03T13:34:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.509+0000 7fa2910a6700 1 -- 172.21.15.7:0/2201741232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28c06bb80 con 0x7fa28c14e560 2023-03-03T13:34:38.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.509+0000 7fa2737fe700 1 -- 172.21.15.7:0/2201741232 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa284007480 con 0x7fa28c14e560 2023-03-03T13:34:38.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.509+0000 7fa2910a6700 1 -- 172.21.15.7:0/2201741232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28c0aacc0 con 0x7fa28c14e560 2023-03-03T13:34:38.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.510+0000 7fa2910a6700 1 -- 172.21.15.7:0/2201741232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28c1ea500 con 0x7fa28c14e560 2023-03-03T13:34:38.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.510+0000 7fa2737fe700 1 -- 172.21.15.7:0/2201741232 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa28401a030 con 0x7fa28c14e560 2023-03-03T13:34:38.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.511+0000 7fa2737fe700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2780682f0 0x7fa27806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:34:38.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.511+0000 7fa2737fe700 1 -- 172.21.15.7:0/2201741232 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa2840860b0 con 0x7fa28c14e560 2023-03-03T13:34:38.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.511+0000 7fa28ad9d700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2780682f0 0x7fa27806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:34:38.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.511+0000 7fa28ad9d700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2780682f0 0x7fa27806a7a0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fa2800097c0 tx=0x7fa280013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:34:38.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.515+0000 7fa2737fe700 1 -- 172.21.15.7:0/2201741232 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa28c1ea500 con 0x7fa28c14e560 2023-03-03T13:34:38.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.754+0000 7fa2910a6700 1 -- 172.21.15.7:0/2201741232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa28c1ea500 con 0x7fa28c14e560 2023-03-03T13:34:38.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.754+0000 7fa2737fe700 1 -- 172.21.15.7:0/2201741232 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa284018090 con 0x7fa28c14e560 2023-03-03T13:34:38.756 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:34:38.756 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:34:38.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 -- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2780682f0 msgr2=0x7fa27806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:38.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2780682f0 0x7fa27806a7a0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fa2800097c0 tx=0x7fa280013040 comp rx=0 tx=0).stop 2023-03-03T13:34:38.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 -- 172.21.15.7:0/2201741232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa28c14e560 msgr2=0x7fa28c0bf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:34:38.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa28c14e560 0x7fa28c0bf600 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fa284001b80 tx=0x7fa28400a340 comp rx=0 tx=0).stop 2023-03-03T13:34:38.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 -- 172.21.15.7:0/2201741232 shutdown_connections 2023-03-03T13:34:38.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2780682f0 0x7fa27806a7a0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:38.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa28c14e560 0x7fa28c0bf600 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:38.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa28c0c43d0 0x7fa28c14f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:38.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 --2- 172.21.15.7:0/2201741232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa28c0c4840 0x7fa28c0ad630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:34:38.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 -- 172.21.15.7:0/2201741232 >> 172.21.15.7:0/2201741232 conn(0x7fa28c0736f0 msgr2=0x7fa28c06a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:34:38.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 -- 172.21.15.7:0/2201741232 shutdown_connections 2023-03-03T13:34:38.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:34:38.756+0000 7fa2910a6700 1 -- 172.21.15.7:0/2201741232 wait complete. 2023-03-03T13:34:38.760 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:34:38.770 INFO:tasks.mds_thrash.fs.[a]:waiting for 112 secs before thrashing 2023-03-03T13:34:43.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:34:43.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:13.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:13.855 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:43.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:35:43.926 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:05.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:36:05.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.412+0000 7fba181db700 1 Processor -- start 2023-03-03T13:36:05.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.412+0000 7fba181db700 1 -- start start 2023-03-03T13:36:05.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba181db700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 0x7fba10156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba181db700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fba100abe20 0x7fba101529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba181db700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba100c2f30 con 0x7fba1014f9d0 2023-03-03T13:36:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba181db700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba100c31d0 con 0x7fba100abe20 2023-03-03T13:36:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba181db700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba101524a0 con 0x7fba1014fda0 2023-03-03T13:36:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba15f77700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fba1014f9d0 0x7fba10071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39290/0 (socket says 172.21.15.7:39290) 2023-03-03T13:36:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba15f77700 1 -- 172.21.15.7:0/1413607229 learned_addr learned my addr 172.21.15.7:0/1413607229 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:36:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba15776700 1 --2- 172.21.15.7:0/1413607229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 0x7fba10156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:05.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba16778700 1 --2- 172.21.15.7:0/1413607229 >> v2:172.21.15.143:3300/0 conn(0x7fba100abe20 0x7fba101529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba14f75700 1 -- 172.21.15.7:0/1413607229 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268745996 0 0) 0x7fba100c2f30 con 0x7fba1014f9d0 2023-03-03T13:36:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba14f75700 1 -- 172.21.15.7:0/1413607229 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9fc0035b0 con 0x7fba1014f9d0 2023-03-03T13:36:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba15776700 1 -- 172.21.15.7:0/1413607229 >> v2:172.21.15.143:3300/0 conn(0x7fba100abe20 msgr2=0x7fba101529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:36:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba15776700 1 --2- 172.21.15.7:0/1413607229 >> v2:172.21.15.143:3300/0 conn(0x7fba100abe20 0x7fba101529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba15776700 1 -- 172.21.15.7:0/1413607229 >> v1:172.21.15.7:6789/0 conn(0x7fba1014f9d0 legacy=0x7fba10071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba14f75700 1 -- 172.21.15.7:0/1413607229 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251271655 0 0) 0x7fb9fc0035b0 con 0x7fba1014f9d0 2023-03-03T13:36:05.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.413+0000 7fba15776700 1 -- 172.21.15.7:0/1413607229 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba100c34b0 con 0x7fba1014fda0 2023-03-03T13:36:05.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba15776700 1 --2- 172.21.15.7:0/1413607229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 0x7fba10156430 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009840 tx=0x7fba0c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=418aeb2d88818d7b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:36:05.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba14f75700 1 -- 172.21.15.7:0/1413607229 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00001670 con 0x7fba1014fda0 2023-03-03T13:36:05.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba14f75700 1 -- 172.21.15.7:0/1413607229 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c004030 con 0x7fba1014fda0 2023-03-03T13:36:05.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba14f75700 1 -- 172.21.15.7:0/1413607229 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c0043c0 con 0x7fba1014fda0 2023-03-03T13:36:05.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba181db700 1 -- 172.21.15.7:0/1413607229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 msgr2=0x7fba10156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:05.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba181db700 1 --2- 172.21.15.7:0/1413607229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 0x7fba10156430 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009840 tx=0x7fba0c009b50 comp rx=0 tx=0).stop 2023-03-03T13:36:05.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba181db700 1 -- 172.21.15.7:0/1413607229 shutdown_connections 2023-03-03T13:36:05.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba181db700 1 --2- 172.21.15.7:0/1413607229 >> v2:172.21.15.143:3300/0 conn(0x7fba100abe20 0x7fba101529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:05.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba181db700 1 --2- 172.21.15.7:0/1413607229 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 0x7fba10156430 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:05.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba181db700 1 -- 172.21.15.7:0/1413607229 >> 172.21.15.7:0/1413607229 conn(0x7fba100c8900 msgr2=0x7fba100c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:36:05.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba181db700 1 -- 172.21.15.7:0/1413607229 shutdown_connections 2023-03-03T13:36:05.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.414+0000 7fba181db700 1 -- 172.21.15.7:0/1413607229 wait complete. 2023-03-03T13:36:05.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba181db700 1 Processor -- start 2023-03-03T13:36:05.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba181db700 1 -- start start 2023-03-03T13:36:05.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba181db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba100abe20 0x7fba101e8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:05.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba181db700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1014f9d0 0x7fba101e4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:05.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba181db700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 0x7fba1014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:05.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba181db700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1014a300 con 0x7fba100abe20 2023-03-03T13:36:05.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba181db700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba101e88a0 con 0x7fba1014f9d0 2023-03-03T13:36:05.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba181db700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba10145030 con 0x7fba1014fda0 2023-03-03T13:36:05.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba15f77700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba100abe20 0x7fba101e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:05.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba15f77700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba100abe20 0x7fba101e8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45316/0 (socket says 172.21.15.7:45316) 2023-03-03T13:36:05.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.416+0000 7fba15f77700 1 -- 172.21.15.7:0/2035270132 learned_addr learned my addr 172.21.15.7:0/2035270132 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:36:05.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba16778700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 0x7fba1014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:05.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba15f77700 1 -- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 msgr2=0x7fba1014ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:05.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba15f77700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 0x7fba1014ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:05.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba15f77700 1 -- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1014f9d0 msgr2=0x7fba101e4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:05.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba15776700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1014f9d0 0x7fba101e4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:05.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba15f77700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1014f9d0 0x7fba101e4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:05.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba15f77700 1 -- 172.21.15.7:0/2035270132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c0094a0 con 0x7fba100abe20 2023-03-03T13:36:05.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba15f77700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba100abe20 0x7fba101e8360 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fba0000c780 tx=0x7fba0000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:36:05.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba067fc700 1 -- 172.21.15.7:0/2035270132 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00004c80 con 0x7fba100abe20 2023-03-03T13:36:05.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba067fc700 1 -- 172.21.15.7:0/2035270132 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba00004de0 con 0x7fba100abe20 2023-03-03T13:36:05.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba181db700 1 -- 172.21.15.7:0/2035270132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba100ac350 con 0x7fba100abe20 2023-03-03T13:36:05.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba067fc700 1 -- 172.21.15.7:0/2035270132 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba000072f0 con 0x7fba100abe20 2023-03-03T13:36:05.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.417+0000 7fba181db700 1 -- 172.21.15.7:0/2035270132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba100ab630 con 0x7fba100abe20 2023-03-03T13:36:05.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.418+0000 7fba181db700 1 -- 172.21.15.7:0/2035270132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba10071fa0 con 0x7fba100abe20 2023-03-03T13:36:05.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.419+0000 7fba067fc700 1 -- 172.21.15.7:0/2035270132 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fba0001a030 con 0x7fba100abe20 2023-03-03T13:36:05.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.419+0000 7fba067fc700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9f40682f0 0x7fb9f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:05.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.419+0000 7fba067fc700 1 -- 172.21.15.7:0/2035270132 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fba00085650 con 0x7fba100abe20 2023-03-03T13:36:05.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.419+0000 7fba15776700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9f40682f0 0x7fb9f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:05.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.420+0000 7fba15776700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9f40682f0 0x7fb9f406a7a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fba101501d0 tx=0x7fba0c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:36:05.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.423+0000 7fba067fc700 1 -- 172.21.15.7:0/2035270132 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fba00051990 con 0x7fba100abe20 2023-03-03T13:36:05.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.661+0000 7fba181db700 1 -- 172.21.15.7:0/2035270132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba10074bb0 con 0x7fba100abe20 2023-03-03T13:36:05.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.661+0000 7fba067fc700 1 -- 172.21.15.7:0/2035270132 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fba00054fb0 con 0x7fba100abe20 2023-03-03T13:36:05.663 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:36:05.664 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:36:05.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 -- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9f40682f0 msgr2=0x7fb9f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:05.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9f40682f0 0x7fb9f406a7a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fba101501d0 tx=0x7fba0c008030 comp rx=0 tx=0).stop 2023-03-03T13:36:05.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 -- 172.21.15.7:0/2035270132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba100abe20 msgr2=0x7fba101e8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:05.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba100abe20 0x7fba101e8360 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fba0000c780 tx=0x7fba0000cb40 comp rx=0 tx=0).stop 2023-03-03T13:36:05.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 -- 172.21.15.7:0/2035270132 shutdown_connections 2023-03-03T13:36:05.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1014fda0 0x7fba1014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:05.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9f40682f0 0x7fb9f406a7a0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:05.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba100abe20 0x7fba101e8360 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:05.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 --2- 172.21.15.7:0/2035270132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1014f9d0 0x7fba101e4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:05.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 -- 172.21.15.7:0/2035270132 >> 172.21.15.7:0/2035270132 conn(0x7fba100c8900 msgr2=0x7fba101528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:36:05.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 -- 172.21.15.7:0/2035270132 shutdown_connections 2023-03-03T13:36:05.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:05.663+0000 7fba181db700 1 -- 172.21.15.7:0/2035270132 wait complete. 2023-03-03T13:36:05.667 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:36:05.677 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T13:36:14.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:14.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:30.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:36:30.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.908+0000 7f3a548ea700 1 Processor -- start 2023-03-03T13:36:30.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.908+0000 7f3a548ea700 1 -- start start 2023-03-03T13:36:30.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.909+0000 7f3a548ea700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3a5006f140 0x7f3a50154c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:30.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.909+0000 7f3a548ea700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a50071cd0 con 0x7f3a50151ba0 2023-03-03T13:36:30.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.909+0000 7f3a548ea700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a50072290 con 0x7f3a50071200 2023-03-03T13:36:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.909+0000 7f3a548ea700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a50155170 con 0x7f3a5006f140 2023-03-03T13:36:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.909+0000 7f3a4e59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3a50151ba0 0x7f3a500aa720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39324/0 (socket says 172.21.15.7:39324) 2023-03-03T13:36:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.909+0000 7f3a4e59c700 1 -- 172.21.15.7:0/441049162 learned_addr learned my addr 172.21.15.7:0/441049162 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:36:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4ed9d700 1 --2- 172.21.15.7:0/441049162 >> v2:172.21.15.143:3300/0 conn(0x7f3a5006f140 0x7f3a50154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4d59a700 1 -- 172.21.15.7:0/441049162 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182891428 0 0) 0x7f3a50071cd0 con 0x7f3a50151ba0 2023-03-03T13:36:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4d59a700 1 -- 172.21.15.7:0/441049162 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a3c0035b0 con 0x7f3a50151ba0 2023-03-03T13:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4d59a700 1 -- 172.21.15.7:0/441049162 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3012075386 0 0) 0x7f3a3c0035b0 con 0x7f3a50151ba0 2023-03-03T13:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4ed9d700 1 -- 172.21.15.7:0/441049162 >> v1:172.21.15.143:6789/0 conn(0x7f3a50071200 legacy=0x7f3a50154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4ed9d700 1 -- 172.21.15.7:0/441049162 >> v1:172.21.15.7:6789/0 conn(0x7f3a50151ba0 legacy=0x7f3a500aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4ed9d700 1 -- 172.21.15.7:0/441049162 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a50151f70 con 0x7f3a5006f140 2023-03-03T13:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4ed9d700 1 --2- 172.21.15.7:0/441049162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a5006f140 0x7f3a50154c30 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009d90 tx=0x7f3a4800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bea268d236a4e29 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4d59a700 1 -- 172.21.15.7:0/441049162 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4800e070 con 0x7f3a5006f140 2023-03-03T13:36:30.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4d59a700 1 -- 172.21.15.7:0/441049162 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a48003a20 con 0x7f3a5006f140 2023-03-03T13:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.910+0000 7f3a4d59a700 1 -- 172.21.15.7:0/441049162 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40001670 con 0x7f3a5006f140 2023-03-03T13:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.911+0000 7f3a548ea700 1 -- 172.21.15.7:0/441049162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a5006f140 msgr2=0x7f3a50154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.911+0000 7f3a548ea700 1 --2- 172.21.15.7:0/441049162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a5006f140 0x7f3a50154c30 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009d90 tx=0x7f3a4800b330 comp rx=0 tx=0).stop 2023-03-03T13:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.911+0000 7f3a548ea700 1 -- 172.21.15.7:0/441049162 shutdown_connections 2023-03-03T13:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.911+0000 7f3a548ea700 1 --2- 172.21.15.7:0/441049162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a5006f140 0x7f3a50154c30 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.911+0000 7f3a548ea700 1 -- 172.21.15.7:0/441049162 >> 172.21.15.7:0/441049162 conn(0x7f3a500736f0 msgr2=0x7f3a50073af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:36:30.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.911+0000 7f3a548ea700 1 -- 172.21.15.7:0/441049162 shutdown_connections 2023-03-03T13:36:30.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.911+0000 7f3a548ea700 1 -- 172.21.15.7:0/441049162 wait complete. 2023-03-03T13:36:30.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.912+0000 7f3a548ea700 1 Processor -- start 2023-03-03T13:36:30.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.912+0000 7f3a548ea700 1 -- start start 2023-03-03T13:36:30.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.913+0000 7f3a548ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a5006f140 0x7f3a50148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:30.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.913+0000 7f3a548ea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a50071200 0x7f3a50144880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:30.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.913+0000 7f3a548ea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a50151ba0 0x7f3a501dece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:30.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.913+0000 7f3a548ea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a501eaf20 con 0x7f3a5006f140 2023-03-03T13:36:30.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.913+0000 7f3a548ea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5006ce80 con 0x7f3a50151ba0 2023-03-03T13:36:30.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.913+0000 7f3a548ea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a501eb190 con 0x7f3a50071200 2023-03-03T13:36:30.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.913+0000 7f3a4e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a5006f140 0x7f3a50148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:30.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.913+0000 7f3a4e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a5006f140 0x7f3a50148460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45350/0 (socket says 172.21.15.7:45350) 2023-03-03T13:36:30.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.913+0000 7f3a4e59c700 1 -- 172.21.15.7:0/1386520157 learned_addr learned my addr 172.21.15.7:0/1386520157 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:36:30.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.913+0000 7f3a4dd9b700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a50071200 0x7f3a50144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:30.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a4ed9d700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a50151ba0 0x7f3a501dece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:30.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a4e59c700 1 -- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a50071200 msgr2=0x7f3a50144880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:30.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a4e59c700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a50071200 0x7f3a50144880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:30.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a4e59c700 1 -- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a50151ba0 msgr2=0x7f3a501dece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:30.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a4e59c700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a50151ba0 0x7f3a501dece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:30.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a4e59c700 1 -- 172.21.15.7:0/1386520157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a48009a70 con 0x7f3a5006f140 2023-03-03T13:36:30.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a4e59c700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a5006f140 0x7f3a50148460 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000c520 tx=0x7f3a4000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:36:30.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a46ffd700 1 -- 172.21.15.7:0/1386520157 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40004c80 con 0x7f3a5006f140 2023-03-03T13:36:30.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a46ffd700 1 -- 172.21.15.7:0/1386520157 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a40004e20 con 0x7f3a5006f140 2023-03-03T13:36:30.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a46ffd700 1 -- 172.21.15.7:0/1386520157 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40007310 con 0x7f3a5006f140 2023-03-03T13:36:30.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a548ea700 1 -- 172.21.15.7:0/1386520157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a50071950 con 0x7f3a5006f140 2023-03-03T13:36:30.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.914+0000 7f3a548ea700 1 -- 172.21.15.7:0/1386520157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a50071670 con 0x7f3a5006f140 2023-03-03T13:36:30.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.915+0000 7f3a548ea700 1 -- 172.21.15.7:0/1386520157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a50071f20 con 0x7f3a5006f140 2023-03-03T13:36:30.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.915+0000 7f3a46ffd700 1 -- 172.21.15.7:0/1386520157 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3a4001a020 con 0x7f3a5006f140 2023-03-03T13:36:30.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.916+0000 7f3a46ffd700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a300682f0 0x7f3a3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:36:30.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.916+0000 7f3a46ffd700 1 -- 172.21.15.7:0/1386520157 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3a40085e70 con 0x7f3a5006f140 2023-03-03T13:36:30.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.916+0000 7f3a4dd9b700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a300682f0 0x7f3a3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:36:30.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.917+0000 7f3a4dd9b700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a300682f0 0x7f3a3006a7a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f3a5006ebf0 tx=0x7f3a38009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:36:30.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:30.920+0000 7f3a46ffd700 1 -- 172.21.15.7:0/1386520157 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3a40052210 con 0x7f3a5006f140 2023-03-03T13:36:31.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.158+0000 7f3a548ea700 1 -- 172.21.15.7:0/1386520157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a501482b0 con 0x7f3a5006f140 2023-03-03T13:36:31.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.158+0000 7f3a46ffd700 1 -- 172.21.15.7:0/1386520157 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3a40055830 con 0x7f3a5006f140 2023-03-03T13:36:31.161 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:36:31.161 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:36:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 -- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a300682f0 msgr2=0x7f3a3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a300682f0 0x7f3a3006a7a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f3a5006ebf0 tx=0x7f3a38009040 comp rx=0 tx=0).stop 2023-03-03T13:36:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 -- 172.21.15.7:0/1386520157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a5006f140 msgr2=0x7f3a50148460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:36:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a5006f140 0x7f3a50148460 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000c520 tx=0x7f3a4000c8e0 comp rx=0 tx=0).stop 2023-03-03T13:36:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 -- 172.21.15.7:0/1386520157 shutdown_connections 2023-03-03T13:36:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a50151ba0 0x7f3a501dece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a300682f0 0x7f3a3006a7a0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a5006f140 0x7f3a50148460 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 --2- 172.21.15.7:0/1386520157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a50071200 0x7f3a50144880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:36:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 -- 172.21.15.7:0/1386520157 >> 172.21.15.7:0/1386520157 conn(0x7f3a500736f0 msgr2=0x7f3a5014ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:36:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 -- 172.21.15.7:0/1386520157 shutdown_connections 2023-03-03T13:36:31.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:36:31.160+0000 7f3a548ea700 1 -- 172.21.15.7:0/1386520157 wait complete. 2023-03-03T13:36:31.164 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:36:31.174 INFO:tasks.mds_thrash.fs.[a]:waiting for 41 secs before thrashing 2023-03-03T13:36:44.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:36:44.105 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:09.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:37:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.815+0000 7f344e2e9700 1 Processor -- start 2023-03-03T13:37:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.815+0000 7f344e2e9700 1 -- start start 2023-03-03T13:37:09.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.816+0000 7f344e2e9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f344814b0d0 0x7f3448071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:09.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.816+0000 7f344e2e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3448071d40 0x7f3448158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.816+0000 7f344e2e9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34480a9c20 con 0x7f34480aad20 2023-03-03T13:37:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.816+0000 7f344e2e9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3448154f10 con 0x7f344814b0d0 2023-03-03T13:37:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.816+0000 7f344e2e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34480ac3c0 con 0x7f3448071d40 2023-03-03T13:37:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.816+0000 7f3447fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f344814b0d0 0x7f3448071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.816+0000 7f3447fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f344814b0d0 0x7f3448071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45380/0 (socket says 172.21.15.7:45380) 2023-03-03T13:37:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.816+0000 7f3447fff700 1 -- 172.21.15.7:0/1968589075 learned_addr learned my addr 172.21.15.7:0/1968589075 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:37:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.817+0000 7f344c886700 1 --2- 172.21.15.7:0/1968589075 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3448071d40 0x7f3448158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:09.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.817+0000 7f3447fff700 1 -- 172.21.15.7:0/1968589075 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3448071d40 msgr2=0x7f3448158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:37:09.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.817+0000 7f3447fff700 1 --2- 172.21.15.7:0/1968589075 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3448071d40 0x7f3448158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:09.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.817+0000 7f3447fff700 1 -- 172.21.15.7:0/1968589075 >> v1:172.21.15.143:6789/0 conn(0x7f34480aad20 legacy=0x7f3448157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:37:09.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.817+0000 7f3447fff700 1 -- 172.21.15.7:0/1968589075 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344806fc70 con 0x7f344814b0d0 2023-03-03T13:37:09.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.817+0000 7f3447fff700 1 --2- 172.21.15.7:0/1968589075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f344814b0d0 0x7f3448071800 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f34380098f0 tx=0x7f3438009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d1ff603c6297bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:37:09.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.817+0000 7f3446ffd700 1 -- 172.21.15.7:0/1968589075 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343800b8a0 con 0x7f344814b0d0 2023-03-03T13:37:09.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.817+0000 7f3446ffd700 1 -- 172.21.15.7:0/1968589075 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343800ba00 con 0x7f344814b0d0 2023-03-03T13:37:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.817+0000 7f3446ffd700 1 -- 172.21.15.7:0/1968589075 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343800bd90 con 0x7f344814b0d0 2023-03-03T13:37:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.818+0000 7f344e2e9700 1 -- 172.21.15.7:0/1968589075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f344814b0d0 msgr2=0x7f3448071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.818+0000 7f344e2e9700 1 --2- 172.21.15.7:0/1968589075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f344814b0d0 0x7f3448071800 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f34380098f0 tx=0x7f3438009c00 comp rx=0 tx=0).stop 2023-03-03T13:37:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.818+0000 7f344e2e9700 1 -- 172.21.15.7:0/1968589075 shutdown_connections 2023-03-03T13:37:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.818+0000 7f344e2e9700 1 --2- 172.21.15.7:0/1968589075 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3448071d40 0x7f3448158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.818+0000 7f344e2e9700 1 --2- 172.21.15.7:0/1968589075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f344814b0d0 0x7f3448071800 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.818+0000 7f344e2e9700 1 -- 172.21.15.7:0/1968589075 >> 172.21.15.7:0/1968589075 conn(0x7f34480c8900 msgr2=0x7f34480c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:37:09.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.818+0000 7f344e2e9700 1 -- 172.21.15.7:0/1968589075 shutdown_connections 2023-03-03T13:37:09.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.818+0000 7f344e2e9700 1 -- 172.21.15.7:0/1968589075 wait complete. 2023-03-03T13:37:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.819+0000 7f344e2e9700 1 Processor -- start 2023-03-03T13:37:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.819+0000 7f344e2e9700 1 -- start start 2023-03-03T13:37:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f344e2e9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3448071d40 0x7f3448157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f344e2e9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34480aad20 0x7f34481e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:09.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f344e2e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f344814b0d0 0x7f34480c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:09.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f344e2e9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34481ee050 con 0x7f3448071d40 2023-03-03T13:37:09.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f344e2e9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34481577c0 con 0x7f34480aad20 2023-03-03T13:37:09.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f344e2e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34481520c0 con 0x7f344814b0d0 2023-03-03T13:37:09.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f3447fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3448071d40 0x7f3448157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:09.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f3447fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3448071d40 0x7f3448157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45386/0 (socket says 172.21.15.7:45386) 2023-03-03T13:37:09.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f3447fff700 1 -- 172.21.15.7:0/1613545125 learned_addr learned my addr 172.21.15.7:0/1613545125 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:37:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f344c886700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f344814b0d0 0x7f34480c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f34477fe700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34480aad20 0x7f34481e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f3447fff700 1 -- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f344814b0d0 msgr2=0x7f34480c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f3447fff700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f344814b0d0 0x7f34480c75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f3447fff700 1 -- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34480aad20 msgr2=0x7f34481e9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f3447fff700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34480aad20 0x7f34481e9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f3447fff700 1 -- 172.21.15.7:0/1613545125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3438009550 con 0x7f3448071d40 2023-03-03T13:37:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f3447fff700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3448071d40 0x7f3448157280 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f3438009c40 tx=0x7f3438004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:37:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.820+0000 7f3444ff9700 1 -- 172.21.15.7:0/1613545125 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34380036a0 con 0x7f3448071d40 2023-03-03T13:37:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.821+0000 7f3444ff9700 1 -- 172.21.15.7:0/1613545125 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343801e070 con 0x7f3448071d40 2023-03-03T13:37:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.821+0000 7f344e2e9700 1 -- 172.21.15.7:0/1613545125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344806f6b0 con 0x7f3448071d40 2023-03-03T13:37:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.821+0000 7f3444ff9700 1 -- 172.21.15.7:0/1613545125 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438003a60 con 0x7f3448071d40 2023-03-03T13:37:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.821+0000 7f344e2e9700 1 -- 172.21.15.7:0/1613545125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3448155160 con 0x7f3448071d40 2023-03-03T13:37:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.822+0000 7f344e2e9700 1 -- 172.21.15.7:0/1613545125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344806f990 con 0x7f3448071d40 2023-03-03T13:37:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.822+0000 7f3444ff9700 1 -- 172.21.15.7:0/1613545125 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f343800f050 con 0x7f3448071d40 2023-03-03T13:37:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.823+0000 7f3444ff9700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34340682f0 0x7f343406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.823+0000 7f3444ff9700 1 -- 172.21.15.7:0/1613545125 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3438020070 con 0x7f3448071d40 2023-03-03T13:37:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.823+0000 7f34477fe700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34340682f0 0x7f343406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.823+0000 7f34477fe700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34340682f0 0x7f343406a7a0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f34481448f0 tx=0x7f343c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:37:09.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:09.827+0000 7f3444ff9700 1 -- 172.21.15.7:0/1613545125 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f34380539d0 con 0x7f3448071d40 2023-03-03T13:37:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.067+0000 7f344e2e9700 1 -- 172.21.15.7:0/1613545125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34481ee330 con 0x7f3448071d40 2023-03-03T13:37:10.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.068+0000 7f3444ff9700 1 -- 172.21.15.7:0/1613545125 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3438056ff0 con 0x7f3448071d40 2023-03-03T13:37:10.070 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:37:10.070 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:37:10.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 -- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34340682f0 msgr2=0x7f343406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34340682f0 0x7f343406a7a0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f34481448f0 tx=0x7f343c009040 comp rx=0 tx=0).stop 2023-03-03T13:37:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 -- 172.21.15.7:0/1613545125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3448071d40 msgr2=0x7f3448157280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3448071d40 0x7f3448157280 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f3438009c40 tx=0x7f3438004750 comp rx=0 tx=0).stop 2023-03-03T13:37:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 -- 172.21.15.7:0/1613545125 shutdown_connections 2023-03-03T13:37:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f344814b0d0 0x7f34480c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34340682f0 0x7f343406a7a0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:10.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3448071d40 0x7f3448157280 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 --2- 172.21.15.7:0/1613545125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f34480aad20 0x7f34481e9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 -- 172.21.15.7:0/1613545125 >> 172.21.15.7:0/1613545125 conn(0x7f34480c8900 msgr2=0x7f3448149ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:37:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 -- 172.21.15.7:0/1613545125 shutdown_connections 2023-03-03T13:37:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:10.070+0000 7f344e2e9700 1 -- 172.21.15.7:0/1613545125 wait complete. 2023-03-03T13:37:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:37:10.083 INFO:tasks.mds_thrash.fs.[b]:waiting for 114 secs before thrashing 2023-03-03T13:37:12.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:37:12.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.313+0000 7fd803c97700 1 Processor -- start 2023-03-03T13:37:12.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.313+0000 7fd803c97700 1 -- start start 2023-03-03T13:37:12.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd803c97700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc071e80 0x7fd7fc151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:12.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd803c97700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd7fc14e900 0x7fd7fc15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:12.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd803c97700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7fc1510d0 con 0x7fd7fc072250 2023-03-03T13:37:12.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd803c97700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7fc0c3740 con 0x7fd7fc14e900 2023-03-03T13:37:12.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd803c97700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc070670 con 0x7fd7fc071e80 2023-03-03T13:37:12.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd801232700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd7fc072250 0x7fd7fc14e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39374/0 (socket says 172.21.15.7:39374) 2023-03-03T13:37:12.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd801232700 1 -- 172.21.15.7:0/3880257642 learned_addr learned my addr 172.21.15.7:0/3880257642 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:37:12.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd801a33700 1 --2- 172.21.15.7:0/3880257642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc071e80 0x7fd7fc151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd802234700 1 --2- 172.21.15.7:0/3880257642 >> v2:172.21.15.143:3300/0 conn(0x7fd7fc14e900 0x7fd7fc15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd800a31700 1 -- 172.21.15.7:0/3880257642 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508941086 0 0) 0x7fd7fc1510d0 con 0x7fd7fc072250 2023-03-03T13:37:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd800a31700 1 -- 172.21.15.7:0/3880257642 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7e80035b0 con 0x7fd7fc072250 2023-03-03T13:37:12.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd800a31700 1 -- 172.21.15.7:0/3880257642 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3661201873 0 0) 0x7fd7e80035b0 con 0x7fd7fc072250 2023-03-03T13:37:12.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd802234700 1 -- 172.21.15.7:0/3880257642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc071e80 msgr2=0x7fd7fc151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:37:12.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd802234700 1 --2- 172.21.15.7:0/3880257642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc071e80 0x7fd7fc151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.314+0000 7fd802234700 1 -- 172.21.15.7:0/3880257642 >> v1:172.21.15.7:6789/0 conn(0x7fd7fc072250 legacy=0x7fd7fc14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:12.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd802234700 1 -- 172.21.15.7:0/3880257642 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7fc0c39e0 con 0x7fd7fc14e900 2023-03-03T13:37:12.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd801a33700 1 --2- 172.21.15.7:0/3880257642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc071e80 0x7fd7fc151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:37:12.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd802234700 1 --2- 172.21.15.7:0/3880257642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc14e900 0x7fd7fc15ba50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4009d90 tx=0x7fd7f400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b52592909702bb7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:37:12.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd800a31700 1 -- 172.21.15.7:0/3880257642 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f400e070 con 0x7fd7fc14e900 2023-03-03T13:37:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd800a31700 1 -- 172.21.15.7:0/3880257642 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f4003a20 con 0x7fd7fc14e900 2023-03-03T13:37:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd800a31700 1 -- 172.21.15.7:0/3880257642 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec001670 con 0x7fd7fc14e900 2023-03-03T13:37:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd803c97700 1 -- 172.21.15.7:0/3880257642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc14e900 msgr2=0x7fd7fc15ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd803c97700 1 --2- 172.21.15.7:0/3880257642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc14e900 0x7fd7fc15ba50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4009d90 tx=0x7fd7f400b330 comp rx=0 tx=0).stop 2023-03-03T13:37:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd803c97700 1 -- 172.21.15.7:0/3880257642 shutdown_connections 2023-03-03T13:37:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd803c97700 1 --2- 172.21.15.7:0/3880257642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc14e900 0x7fd7fc15ba50 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd803c97700 1 --2- 172.21.15.7:0/3880257642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc071e80 0x7fd7fc151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.315+0000 7fd803c97700 1 -- 172.21.15.7:0/3880257642 >> 172.21.15.7:0/3880257642 conn(0x7fd7fc0a94d0 msgr2=0x7fd7fc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:37:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.316+0000 7fd803c97700 1 -- 172.21.15.7:0/3880257642 shutdown_connections 2023-03-03T13:37:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.316+0000 7fd803c97700 1 -- 172.21.15.7:0/3880257642 wait complete. 2023-03-03T13:37:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.317+0000 7fd803c97700 1 Processor -- start 2023-03-03T13:37:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.317+0000 7fd803c97700 1 -- start start 2023-03-03T13:37:12.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.317+0000 7fd803c97700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc071e80 0x7fd7fc0bd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:12.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.317+0000 7fd803c97700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc072250 0x7fd7fc149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.317+0000 7fd803c97700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc14e900 0x7fd7fc0bae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.317+0000 7fd803c97700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc148700 con 0x7fd7fc071e80 2023-03-03T13:37:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.317+0000 7fd803c97700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc149240 con 0x7fd7fc072250 2023-03-03T13:37:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.317+0000 7fd803c97700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc148910 con 0x7fd7fc14e900 2023-03-03T13:37:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd801a33700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc071e80 0x7fd7fc0bd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd801a33700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc071e80 0x7fd7fc0bd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45400/0 (socket says 172.21.15.7:45400) 2023-03-03T13:37:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd801a33700 1 -- 172.21.15.7:0/1060669321 learned_addr learned my addr 172.21.15.7:0/1060669321 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:37:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd801232700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc072250 0x7fd7fc149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd802234700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc14e900 0x7fd7fc0bae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd801a33700 1 -- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc14e900 msgr2=0x7fd7fc0bae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd801a33700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc14e900 0x7fd7fc0bae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd801a33700 1 -- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc072250 msgr2=0x7fd7fc149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:12.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd801a33700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc072250 0x7fd7fc149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd801a33700 1 -- 172.21.15.7:0/1060669321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f80094a0 con 0x7fd7fc071e80 2023-03-03T13:37:12.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd802234700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc14e900 0x7fd7fc0bae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:37:12.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd801a33700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc071e80 0x7fd7fc0bd880 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800ebe0 tx=0x7fd7f8009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:37:12.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd7f27fc700 1 -- 172.21.15.7:0/1060669321 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f800c910 con 0x7fd7fc071e80 2023-03-03T13:37:12.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd7f27fc700 1 -- 172.21.15.7:0/1060669321 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f800ca70 con 0x7fd7fc071e80 2023-03-03T13:37:12.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd803c97700 1 -- 172.21.15.7:0/1060669321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f4009a70 con 0x7fd7fc071e80 2023-03-03T13:37:12.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd7f27fc700 1 -- 172.21.15.7:0/1060669321 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f800cde0 con 0x7fd7fc071e80 2023-03-03T13:37:12.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.318+0000 7fd803c97700 1 -- 172.21.15.7:0/1060669321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7fc06c9a0 con 0x7fd7fc071e80 2023-03-03T13:37:12.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.320+0000 7fd7f27fc700 1 -- 172.21.15.7:0/1060669321 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd7f8019030 con 0x7fd7fc071e80 2023-03-03T13:37:12.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.320+0000 7fd803c97700 1 -- 172.21.15.7:0/1060669321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7fc151300 con 0x7fd7fc071e80 2023-03-03T13:37:12.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.320+0000 7fd7f27fc700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7e00682f0 0x7fd7e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:37:12.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.320+0000 7fd7f27fc700 1 -- 172.21.15.7:0/1060669321 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd7f8086300 con 0x7fd7fc071e80 2023-03-03T13:37:12.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.321+0000 7fd801232700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7e00682f0 0x7fd7e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:37:12.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.321+0000 7fd801232700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7e00682f0 0x7fd7e006a7a0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec001eb0 tx=0x7fd7ec001ce0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:37:12.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.325+0000 7fd7f27fc700 1 -- 172.21.15.7:0/1060669321 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7f80525b0 con 0x7fd7fc071e80 2023-03-03T13:37:12.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.561+0000 7fd803c97700 1 -- 172.21.15.7:0/1060669321 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7fc06cb30 con 0x7fd7fc071e80 2023-03-03T13:37:12.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.562+0000 7fd7f27fc700 1 -- 172.21.15.7:0/1060669321 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd7f8055bd0 con 0x7fd7fc071e80 2023-03-03T13:37:12.564 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:37:12.564 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:37:12.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 -- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7e00682f0 msgr2=0x7fd7e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:12.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7e00682f0 0x7fd7e006a7a0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec001eb0 tx=0x7fd7ec001ce0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 -- 172.21.15.7:0/1060669321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc071e80 msgr2=0x7fd7fc0bd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:37:12.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc071e80 0x7fd7fc0bd880 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800ebe0 tx=0x7fd7f8009ec0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 -- 172.21.15.7:0/1060669321 shutdown_connections 2023-03-03T13:37:12.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7fc14e900 0x7fd7fc0bae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7e00682f0 0x7fd7e006a7a0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc071e80 0x7fd7fc0bd880 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 --2- 172.21.15.7:0/1060669321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc072250 0x7fd7fc149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:37:12.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 -- 172.21.15.7:0/1060669321 >> 172.21.15.7:0/1060669321 conn(0x7fd7fc0a94d0 msgr2=0x7fd7fc14dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:37:12.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 -- 172.21.15.7:0/1060669321 shutdown_connections 2023-03-03T13:37:12.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:37:12.564+0000 7fd803c97700 1 -- 172.21.15.7:0/1060669321 wait complete. 2023-03-03T13:37:12.567 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:37:12.577 INFO:tasks.mds_thrash.fs.[a]:waiting for 111 secs before thrashing 2023-03-03T13:37:14.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:14.175 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:44.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:37:44.247 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:14.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:14.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:44.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:38:44.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:03.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:39:03.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.714+0000 7f76daccc700 1 Processor -- start 2023-03-03T13:39:03.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.714+0000 7f76daccc700 1 -- start start 2023-03-03T13:39:03.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.715+0000 7f76daccc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76d4152e30 0x7f76d40c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:03.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.715+0000 7f76daccc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40c4910 0x7f76d4156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:03.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.715+0000 7f76daccc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d41523b0 con 0x7f76d40beab0 2023-03-03T13:39:03.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.715+0000 7f76daccc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d4072010 con 0x7f76d4152e30 2023-03-03T13:39:03.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.715+0000 7f76daccc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40aaaf0 con 0x7f76d40c4910 2023-03-03T13:39:03.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.715+0000 7f76d8a68700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76d4152e30 0x7f76d40c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.715+0000 7f76d8a68700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76d4152e30 0x7f76d40c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45488/0 (socket says 172.21.15.7:45488) 2023-03-03T13:39:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.715+0000 7f76d8a68700 1 -- 172.21.15.7:0/1192301480 learned_addr learned my addr 172.21.15.7:0/1192301480 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:39:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.716+0000 7f76d9269700 1 --2- 172.21.15.7:0/1192301480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40c4910 0x7f76d4156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.716+0000 7f76d8a68700 1 -- 172.21.15.7:0/1192301480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40c4910 msgr2=0x7f76d4156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:39:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.716+0000 7f76d8a68700 1 --2- 172.21.15.7:0/1192301480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40c4910 0x7f76d4156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.716+0000 7f76d8a68700 1 -- 172.21.15.7:0/1192301480 >> v1:172.21.15.143:6789/0 conn(0x7f76d40beab0 legacy=0x7f76d4155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:03.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.716+0000 7f76d8a68700 1 -- 172.21.15.7:0/1192301480 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d40aadd0 con 0x7f76d4152e30 2023-03-03T13:39:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.716+0000 7f76d8a68700 1 --2- 172.21.15.7:0/1192301480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 0x7f76d40c43d0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f76c80098f0 tx=0x7f76c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ffc808174c3e6d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:39:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.716+0000 7f76d37fe700 1 -- 172.21.15.7:0/1192301480 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c800b8a0 con 0x7f76d4152e30 2023-03-03T13:39:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.716+0000 7f76d37fe700 1 -- 172.21.15.7:0/1192301480 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c800ba00 con 0x7f76d4152e30 2023-03-03T13:39:03.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.716+0000 7f76d37fe700 1 -- 172.21.15.7:0/1192301480 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c800bd90 con 0x7f76d4152e30 2023-03-03T13:39:03.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.717+0000 7f76daccc700 1 -- 172.21.15.7:0/1192301480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 msgr2=0x7f76d40c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:03.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.717+0000 7f76daccc700 1 --2- 172.21.15.7:0/1192301480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 0x7f76d40c43d0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f76c80098f0 tx=0x7f76c8009c00 comp rx=0 tx=0).stop 2023-03-03T13:39:03.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.717+0000 7f76daccc700 1 -- 172.21.15.7:0/1192301480 shutdown_connections 2023-03-03T13:39:03.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.717+0000 7f76daccc700 1 --2- 172.21.15.7:0/1192301480 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40c4910 0x7f76d4156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:03.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.717+0000 7f76daccc700 1 --2- 172.21.15.7:0/1192301480 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 0x7f76d40c43d0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.717+0000 7f76daccc700 1 -- 172.21.15.7:0/1192301480 >> 172.21.15.7:0/1192301480 conn(0x7f76d40736f0 msgr2=0x7f76d4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:39:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.717+0000 7f76daccc700 1 -- 172.21.15.7:0/1192301480 shutdown_connections 2023-03-03T13:39:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.717+0000 7f76daccc700 1 -- 172.21.15.7:0/1192301480 wait complete. 2023-03-03T13:39:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.718+0000 7f76daccc700 1 Processor -- start 2023-03-03T13:39:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.718+0000 7f76daccc700 1 -- start start 2023-03-03T13:39:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76daccc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40beab0 0x7f76d40ac600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76daccc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40c4910 0x7f76d41545f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76daccc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 0x7f76d40c6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76daccc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d41555f0 con 0x7f76d4152e30 2023-03-03T13:39:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76daccc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40c8370 con 0x7f76d40c4910 2023-03-03T13:39:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76daccc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d4147fa0 con 0x7f76d40beab0 2023-03-03T13:39:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d9269700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 0x7f76d40c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d9269700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 0x7f76d40c6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45498/0 (socket says 172.21.15.7:45498) 2023-03-03T13:39:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d9269700 1 -- 172.21.15.7:0/3039511920 learned_addr learned my addr 172.21.15.7:0/3039511920 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:39:03.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d3fff700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40c4910 0x7f76d41545f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d8a68700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40beab0 0x7f76d40ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d9269700 1 -- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40beab0 msgr2=0x7f76d40ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d9269700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40beab0 0x7f76d40ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d9269700 1 -- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40c4910 msgr2=0x7f76d41545f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d9269700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40c4910 0x7f76d41545f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d9269700 1 -- 172.21.15.7:0/3039511920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c8009550 con 0x7f76d4152e30 2023-03-03T13:39:03.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d8a68700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40beab0 0x7f76d40ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:39:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d3fff700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40c4910 0x7f76d41545f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:39:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.719+0000 7f76d9269700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 0x7f76d40c6330 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f76cc007e70 tx=0x7f76cc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:39:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.720+0000 7f76d17fa700 1 -- 172.21.15.7:0/3039511920 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc00ee50 con 0x7f76d4152e30 2023-03-03T13:39:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.720+0000 7f76d17fa700 1 -- 172.21.15.7:0/3039511920 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc014420 con 0x7f76d4152e30 2023-03-03T13:39:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.720+0000 7f76daccc700 1 -- 172.21.15.7:0/3039511920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d4153200 con 0x7f76d4152e30 2023-03-03T13:39:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.720+0000 7f76d17fa700 1 -- 172.21.15.7:0/3039511920 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0147d0 con 0x7f76d4152e30 2023-03-03T13:39:03.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.720+0000 7f76daccc700 1 -- 172.21.15.7:0/3039511920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d4071c70 con 0x7f76d4152e30 2023-03-03T13:39:03.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.721+0000 7f76daccc700 1 -- 172.21.15.7:0/3039511920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d4153390 con 0x7f76d4152e30 2023-03-03T13:39:03.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.721+0000 7f76d17fa700 1 -- 172.21.15.7:0/3039511920 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f76cc012050 con 0x7f76d4152e30 2023-03-03T13:39:03.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.721+0000 7f76d17fa700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b80682f0 0x7f76b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:03.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.721+0000 7f76d17fa700 1 -- 172.21.15.7:0/3039511920 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f76cc0867e0 con 0x7f76d4152e30 2023-03-03T13:39:03.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.722+0000 7f76d8a68700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b80682f0 0x7f76b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:03.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.722+0000 7f76d8a68700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b80682f0 0x7f76b806a7a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009c40 tx=0x7f76c8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:39:03.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.726+0000 7f76d17fa700 1 -- 172.21.15.7:0/3039511920 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f76cc052b80 con 0x7f76d4152e30 2023-03-03T13:39:03.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.963+0000 7f76daccc700 1 -- 172.21.15.7:0/3039511920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76d4074bb0 con 0x7f76d4152e30 2023-03-03T13:39:03.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.964+0000 7f76d17fa700 1 -- 172.21.15.7:0/3039511920 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f76cc0561a0 con 0x7f76d4152e30 2023-03-03T13:39:03.965 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:39:03.966 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:39:03.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.965+0000 7f76daccc700 1 -- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b80682f0 msgr2=0x7f76b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:03.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.965+0000 7f76daccc700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b80682f0 0x7f76b806a7a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009c40 tx=0x7f76c8008030 comp rx=0 tx=0).stop 2023-03-03T13:39:03.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.965+0000 7f76daccc700 1 -- 172.21.15.7:0/3039511920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 msgr2=0x7f76d40c6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.965+0000 7f76daccc700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 0x7f76d40c6330 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f76cc007e70 tx=0x7f76cc00e3c0 comp rx=0 tx=0).stop 2023-03-03T13:39:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.965+0000 7f76daccc700 1 -- 172.21.15.7:0/3039511920 shutdown_connections 2023-03-03T13:39:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.965+0000 7f76daccc700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76b80682f0 0x7f76b806a7a0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.965+0000 7f76daccc700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76d4152e30 0x7f76d40c6330 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.965+0000 7f76daccc700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76d40beab0 0x7f76d40ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.965+0000 7f76daccc700 1 --2- 172.21.15.7:0/3039511920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d40c4910 0x7f76d41545f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.966+0000 7f76daccc700 1 -- 172.21.15.7:0/3039511920 >> 172.21.15.7:0/3039511920 conn(0x7f76d40736f0 msgr2=0x7f76d414f260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:39:03.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.966+0000 7f76daccc700 1 -- 172.21.15.7:0/3039511920 shutdown_connections 2023-03-03T13:39:03.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:03.966+0000 7f76daccc700 1 -- 172.21.15.7:0/3039511920 wait complete. 2023-03-03T13:39:03.969 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:39:03.979 INFO:tasks.mds_thrash.fs.[a]:waiting for 109 secs before thrashing 2023-03-03T13:39:04.084 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:39:04.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.207+0000 7f0d178d0700 1 Processor -- start 2023-03-03T13:39:04.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.207+0000 7f0d178d0700 1 -- start start 2023-03-03T13:39:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.207+0000 7f0d178d0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d1006d870 0x7f0d101550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.207+0000 7f0d178d0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0d100c1da0 0x7f0d100c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d178d0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d100c3620 con 0x7f0d100c3e30 2023-03-03T13:39:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d178d0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d100c2af0 con 0x7f0d100c1da0 2023-03-03T13:39:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d178d0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d101545e0 con 0x7f0d1006d870 2023-03-03T13:39:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d14e6b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0d100c1da0 0x7f0d100c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d15e6d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0d100c3e30 0x7f0d10157ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39484/0 (socket says 172.21.15.7:39484) 2023-03-03T13:39:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d15e6d700 1 -- 172.21.15.7:0/443317530 learned_addr learned my addr 172.21.15.7:0/443317530 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:39:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d1566c700 1 --2- 172.21.15.7:0/443317530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d1006d870 0x7f0d101550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d14e6b700 1 -- 172.21.15.7:0/443317530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d1006d870 msgr2=0x7f0d101550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:39:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d14e6b700 1 --2- 172.21.15.7:0/443317530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d1006d870 0x7f0d101550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d14e6b700 1 -- 172.21.15.7:0/443317530 >> v1:172.21.15.7:6789/0 conn(0x7f0d100c3e30 legacy=0x7f0d10157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d03fff700 1 -- 172.21.15.7:0/443317530 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230369383 0 0) 0x7f0d100c3620 con 0x7f0d100c3e30 2023-03-03T13:39:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d14e6b700 1 -- 172.21.15.7:0/443317530 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d101548c0 con 0x7f0d100c1da0 2023-03-03T13:39:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d14e6b700 1 --2- 172.21.15.7:0/443317530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d100c1da0 0x7f0d100c7350 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009840 tx=0x7f0d04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56becd40b66aebf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:39:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d03fff700 1 -- 172.21.15.7:0/443317530 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04011830 con 0x7f0d100c1da0 2023-03-03T13:39:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d03fff700 1 -- 172.21.15.7:0/443317530 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d04011990 con 0x7f0d100c1da0 2023-03-03T13:39:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.208+0000 7f0d03fff700 1 -- 172.21.15.7:0/443317530 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d04011d20 con 0x7f0d100c1da0 2023-03-03T13:39:04.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.209+0000 7f0d178d0700 1 -- 172.21.15.7:0/443317530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d100c1da0 msgr2=0x7f0d100c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:04.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.209+0000 7f0d178d0700 1 --2- 172.21.15.7:0/443317530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d100c1da0 0x7f0d100c7350 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009840 tx=0x7f0d04009b50 comp rx=0 tx=0).stop 2023-03-03T13:39:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.209+0000 7f0d178d0700 1 -- 172.21.15.7:0/443317530 shutdown_connections 2023-03-03T13:39:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.209+0000 7f0d178d0700 1 --2- 172.21.15.7:0/443317530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d1006d870 0x7f0d101550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.209+0000 7f0d178d0700 1 --2- 172.21.15.7:0/443317530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d100c1da0 0x7f0d100c7350 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.209+0000 7f0d178d0700 1 -- 172.21.15.7:0/443317530 >> 172.21.15.7:0/443317530 conn(0x7f0d1006c110 msgr2=0x7f0d1006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:39:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.209+0000 7f0d178d0700 1 -- 172.21.15.7:0/443317530 shutdown_connections 2023-03-03T13:39:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.209+0000 7f0d178d0700 1 -- 172.21.15.7:0/443317530 wait complete. 2023-03-03T13:39:04.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.210+0000 7f0d178d0700 1 Processor -- start 2023-03-03T13:39:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.210+0000 7f0d178d0700 1 -- start start 2023-03-03T13:39:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d178d0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1006d870 0x7f0d101ed670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d178d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d100c1da0 0x7f0d10142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d178d0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d100c3e30 0x7f0d10152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d178d0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d100ad3e0 con 0x7f0d1006d870 2023-03-03T13:39:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d178d0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d101edbb0 con 0x7f0d100c1da0 2023-03-03T13:39:04.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d178d0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d101558a0 con 0x7f0d100c3e30 2023-03-03T13:39:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d1566c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1006d870 0x7f0d101ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d1566c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1006d870 0x7f0d101ed670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45508/0 (socket says 172.21.15.7:45508) 2023-03-03T13:39:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d1566c700 1 -- 172.21.15.7:0/574424221 learned_addr learned my addr 172.21.15.7:0/574424221 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:39:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d15e6d700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d100c3e30 0x7f0d10152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d14e6b700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d100c1da0 0x7f0d10142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d1566c700 1 -- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d100c3e30 msgr2=0x7f0d10152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d1566c700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d100c3e30 0x7f0d10152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:04.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d1566c700 1 -- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d100c1da0 msgr2=0x7f0d10142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d1566c700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d100c1da0 0x7f0d10142800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.211+0000 7f0d1566c700 1 -- 172.21.15.7:0/574424221 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d040094a0 con 0x7f0d1006d870 2023-03-03T13:39:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.212+0000 7f0d1566c700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1006d870 0x7f0d101ed670 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e810 tx=0x7f0d0c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:39:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.212+0000 7f0d01ffb700 1 -- 172.21.15.7:0/574424221 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c00c070 con 0x7f0d1006d870 2023-03-03T13:39:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.212+0000 7f0d01ffb700 1 -- 172.21.15.7:0/574424221 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c013610 con 0x7f0d1006d870 2023-03-03T13:39:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.212+0000 7f0d178d0700 1 -- 172.21.15.7:0/574424221 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1006dc40 con 0x7f0d1006d870 2023-03-03T13:39:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.212+0000 7f0d01ffb700 1 -- 172.21.15.7:0/574424221 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c0139c0 con 0x7f0d1006d870 2023-03-03T13:39:04.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.212+0000 7f0d178d0700 1 -- 172.21.15.7:0/574424221 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d10154ba0 con 0x7f0d1006d870 2023-03-03T13:39:04.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.213+0000 7f0d178d0700 1 -- 172.21.15.7:0/574424221 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d100c38e0 con 0x7f0d1006d870 2023-03-03T13:39:04.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.213+0000 7f0d01ffb700 1 -- 172.21.15.7:0/574424221 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0d0c013b20 con 0x7f0d1006d870 2023-03-03T13:39:04.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.214+0000 7f0d01ffb700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 0x7f0cf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:39:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.214+0000 7f0d01ffb700 1 -- 172.21.15.7:0/574424221 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0d0c085e30 con 0x7f0d1006d870 2023-03-03T13:39:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.214+0000 7f0d14e6b700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 0x7f0cf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:39:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.214+0000 7f0d14e6b700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 0x7f0cf406a7a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f0d040090b0 tx=0x7f0d04008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:39:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.218+0000 7f0d01ffb700 1 -- 172.21.15.7:0/574424221 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0d0c0521d0 con 0x7f0d1006d870 2023-03-03T13:39:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.458+0000 7f0d178d0700 1 -- 172.21.15.7:0/574424221 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d10074bb0 con 0x7f0d1006d870 2023-03-03T13:39:04.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.458+0000 7f0d01ffb700 1 -- 172.21.15.7:0/574424221 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0d0c017090 con 0x7f0d1006d870 2023-03-03T13:39:04.460 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:39:04.460 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:39:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 -- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 msgr2=0x7f0cf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 0x7f0cf406a7a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f0d040090b0 tx=0x7f0d04008030 comp rx=0 tx=0).stop 2023-03-03T13:39:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 -- 172.21.15.7:0/574424221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1006d870 msgr2=0x7f0d101ed670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:39:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1006d870 0x7f0d101ed670 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e810 tx=0x7f0d0c00eb20 comp rx=0 tx=0).stop 2023-03-03T13:39:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 -- 172.21.15.7:0/574424221 shutdown_connections 2023-03-03T13:39:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d100c3e30 0x7f0d10152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0cf40682f0 0x7f0cf406a7a0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d1006d870 0x7f0d101ed670 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 --2- 172.21.15.7:0/574424221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d100c1da0 0x7f0d10142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:39:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 -- 172.21.15.7:0/574424221 >> 172.21.15.7:0/574424221 conn(0x7f0d1006c110 msgr2=0x7f0d1006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:39:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 -- 172.21.15.7:0/574424221 shutdown_connections 2023-03-03T13:39:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:39:04.460+0000 7f0d178d0700 1 -- 172.21.15.7:0/574424221 wait complete. 2023-03-03T13:39:04.464 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:39:04.474 INFO:tasks.mds_thrash.fs.[b]:waiting for 113 secs before thrashing 2023-03-03T13:39:14.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:14.528 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:44.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:39:44.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:00.002 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T13:39:59.998+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T13:40:14.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:14.772 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:44.882 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:44.885 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:40:52.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:40:53.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.118+0000 7f0a8e8b5700 1 Processor -- start 2023-03-03T13:40:53.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.118+0000 7f0a8e8b5700 1 -- start start 2023-03-03T13:40:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.119+0000 7f0a8e8b5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0a88071e80 0x7f0a88151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.119+0000 7f0a8e8b5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a881510d0 con 0x7f0a88072250 2023-03-03T13:40:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.119+0000 7f0a8e8b5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a880c3740 con 0x7f0a8814e900 2023-03-03T13:40:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.119+0000 7f0a8e8b5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a88070670 con 0x7f0a88071e80 2023-03-03T13:40:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.119+0000 7f0a877fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0a88072250 0x7f0a8814e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39590/0 (socket says 172.21.15.7:39590) 2023-03-03T13:40:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.119+0000 7f0a877fe700 1 -- 172.21.15.7:0/1711531299 learned_addr learned my addr 172.21.15.7:0/1711531299 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:40:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.119+0000 7f0a87fff700 1 --2- 172.21.15.7:0/1711531299 >> v2:172.21.15.143:3300/0 conn(0x7f0a88071e80 0x7f0a88151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.119+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934225208 0 0) 0x7f0a881510d0 con 0x7f0a88072250 2023-03-03T13:40:53.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.119+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a740035b0 con 0x7f0a88072250 2023-03-03T13:40:53.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276186155 0 0) 0x7f0a880c3740 con 0x7f0a8814e900 2023-03-03T13:40:53.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a881510d0 con 0x7f0a8814e900 2023-03-03T13:40:53.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2798960099 0 0) 0x7f0a740035b0 con 0x7f0a88072250 2023-03-03T13:40:53.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 >> v2:172.21.15.143:3300/0 conn(0x7f0a88071e80 msgr2=0x7f0a88151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:40:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 --2- 172.21.15.7:0/1711531299 >> v2:172.21.15.143:3300/0 conn(0x7f0a88071e80 0x7f0a88151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 >> v1:172.21.15.143:6789/0 conn(0x7f0a8814e900 legacy=0x7f0a8815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a880c39e0 con 0x7f0a88072250 2023-03-03T13:40:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0a78001670 con 0x7f0a88072250 2023-03-03T13:40:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 >> v1:172.21.15.7:6789/0 conn(0x7f0a88072250 legacy=0x7f0a8814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a74003c00 0x7f0a74003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74004850 0x7f0a74004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74005250 0x7f0a7400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a88070670 con 0x7f0a74005250 2023-03-03T13:40:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7400c160 con 0x7f0a74004850 2023-03-03T13:40:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7400e530 con 0x7f0a74003c00 2023-03-03T13:40:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a8ce52700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74005250 0x7f0a7400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a877fe700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74004850 0x7f0a74004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a87fff700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a74003c00 0x7f0a74003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a8ce52700 1 -- 172.21.15.7:0/1711531299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a74003c00 msgr2=0x7f0a74003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:40:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a8ce52700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a74003c00 0x7f0a74003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a8ce52700 1 -- 172.21.15.7:0/1711531299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74004850 msgr2=0x7f0a74004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:40:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a8ce52700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74004850 0x7f0a74004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a8ce52700 1 -- 172.21.15.7:0/1711531299 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a740035b0 con 0x7f0a74005250 2023-03-03T13:40:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a877fe700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74004850 0x7f0a74004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:40:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a87fff700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a74003c00 0x7f0a74003ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:40:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a8ce52700 1 -- 172.21.15.7:0/1711531299 reap_dead start 2023-03-03T13:40:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.120+0000 7f0a8ce52700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74005250 0x7f0a7400ba10 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0a80002230 tx=0x7f0a74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be46cdd53bf2b91 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.121+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80001f30 con 0x7f0a74005250 2023-03-03T13:40:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.121+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8000ce60 con 0x7f0a74005250 2023-03-03T13:40:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.121+0000 7f0a86ffd700 1 -- 172.21.15.7:0/1711531299 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80003c20 con 0x7f0a74005250 2023-03-03T13:40:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.121+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/1711531299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74005250 msgr2=0x7f0a7400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.121+0000 7f0a8e8b5700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74005250 0x7f0a7400ba10 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0a80002230 tx=0x7f0a74005880 comp rx=0 tx=0).stop 2023-03-03T13:40:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.121+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/1711531299 shutdown_connections 2023-03-03T13:40:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.121+0000 7f0a8e8b5700 1 --2- 172.21.15.7:0/1711531299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74005250 0x7f0a7400ba10 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.121+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/1711531299 >> 172.21.15.7:0/1711531299 conn(0x7f0a880a94d0 msgr2=0x7f0a880a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:40:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.121+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/1711531299 shutdown_connections 2023-03-03T13:40:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.121+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/1711531299 wait complete. 2023-03-03T13:40:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.122+0000 7f0a8e8b5700 1 Processor -- start 2023-03-03T13:40:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.123+0000 7f0a8e8b5700 1 -- start start 2023-03-03T13:40:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.123+0000 7f0a8e8b5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74005250 0x7f0a880c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.123+0000 7f0a8e8b5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880acfc0 0x7f0a8814bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.123+0000 7f0a8e8b5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a880aa1d0 0x7f0a88147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.123+0000 7f0a8e8b5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8814d300 con 0x7f0a880aa1d0 2023-03-03T13:40:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.123+0000 7f0a8e8b5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a881eade0 con 0x7f0a74005250 2023-03-03T13:40:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.123+0000 7f0a8e8b5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a881eb080 con 0x7f0a880acfc0 2023-03-03T13:40:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.123+0000 7f0a8ce52700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a880aa1d0 0x7f0a88147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.123+0000 7f0a8ce52700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a880aa1d0 0x7f0a88147350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45626/0 (socket says 172.21.15.7:45626) 2023-03-03T13:40:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.123+0000 7f0a8ce52700 1 -- 172.21.15.7:0/4020461838 learned_addr learned my addr 172.21.15.7:0/4020461838 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:40:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a87fff700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74005250 0x7f0a880c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a877fe700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880acfc0 0x7f0a8814bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a8ce52700 1 -- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880acfc0 msgr2=0x7f0a8814bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a8ce52700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880acfc0 0x7f0a8814bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a8ce52700 1 -- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74005250 msgr2=0x7f0a880c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a8ce52700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74005250 0x7f0a880c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a8ce52700 1 -- 172.21.15.7:0/4020461838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8806c9a0 con 0x7f0a880aa1d0 2023-03-03T13:40:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a87fff700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74005250 0x7f0a880c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:40:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a877fe700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880acfc0 0x7f0a8814bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:40:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a8ce52700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a880aa1d0 0x7f0a88147350 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f0a88071650 tx=0x7f0a80001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:53.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4020461838 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80021070 con 0x7f0a880aa1d0 2023-03-03T13:40:53.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4020461838 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8000cc60 con 0x7f0a880aa1d0 2023-03-03T13:40:53.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/4020461838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a88151300 con 0x7f0a880aa1d0 2023-03-03T13:40:53.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4020461838 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80005520 con 0x7f0a880aa1d0 2023-03-03T13:40:53.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.124+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/4020461838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8806cb30 con 0x7f0a880aa1d0 2023-03-03T13:40:53.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.125+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/4020461838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a88074bb0 con 0x7f0a880aa1d0 2023-03-03T13:40:53.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.125+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4020461838 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0a80010050 con 0x7f0a880aa1d0 2023-03-03T13:40:53.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.126+0000 7f0a84ff9700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a680682f0 0x7f0a6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.126+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4020461838 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0a800865b0 con 0x7f0a880aa1d0 2023-03-03T13:40:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.126+0000 7f0a87fff700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a680682f0 0x7f0a6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.127+0000 7f0a87fff700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a680682f0 0x7f0a6806a7a0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f0a881e8100 tx=0x7f0a70008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.130+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4020461838 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0a800528f0 con 0x7f0a880aa1d0 2023-03-03T13:40:53.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.379+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/4020461838 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a880a9fb0 con 0x7f0a880aa1d0 2023-03-03T13:40:53.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.380+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4020461838 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0a80055f10 con 0x7f0a880aa1d0 2023-03-03T13:40:53.382 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:40:53.382 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:40:53.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a680682f0 msgr2=0x7f0a6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:53.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a680682f0 0x7f0a6806a7a0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f0a881e8100 tx=0x7f0a70008040 comp rx=0 tx=0).stop 2023-03-03T13:40:53.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/4020461838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a880aa1d0 msgr2=0x7f0a88147350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:53.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a880aa1d0 0x7f0a88147350 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f0a88071650 tx=0x7f0a80001650 comp rx=0 tx=0).stop 2023-03-03T13:40:53.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/4020461838 shutdown_connections 2023-03-03T13:40:53.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a680682f0 0x7f0a6806a7a0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a880aa1d0 0x7f0a88147350 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74005250 0x7f0a880c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 --2- 172.21.15.7:0/4020461838 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880acfc0 0x7f0a8814bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/4020461838 >> 172.21.15.7:0/4020461838 conn(0x7f0a880a94d0 msgr2=0x7f0a880c2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:40:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/4020461838 shutdown_connections 2023-03-03T13:40:53.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:53.382+0000 7f0a8e8b5700 1 -- 172.21.15.7:0/4020461838 wait complete. 2023-03-03T13:40:53.386 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:40:53.395 INFO:tasks.mds_thrash.fs.[a]:waiting for 12 secs before thrashing 2023-03-03T13:40:57.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:40:57.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.612+0000 7f41435d2700 1 Processor -- start 2023-03-03T13:40:57.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.613+0000 7f41435d2700 1 -- start start 2023-03-03T13:40:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.613+0000 7f41435d2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f413c06bea0 0x7f413c06c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.613+0000 7f41435d2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c157dc0 0x7f413c158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:57.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.613+0000 7f41435d2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413c06d780 con 0x7f413c151ab0 2023-03-03T13:40:57.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.613+0000 7f41435d2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413c06c830 con 0x7f413c06bea0 2023-03-03T13:40:57.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.613+0000 7f41435d2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413c06cb70 con 0x7f413c157dc0 2023-03-03T13:40:57.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4141b6f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c157dc0 0x7f413c158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:57.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4140b6d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f413c06bea0 0x7f413c06c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:57.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4141b6f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c157dc0 0x7f413c158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:53706/0 (socket says 172.21.15.7:53706) 2023-03-03T13:40:57.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4141b6f700 1 -- 172.21.15.7:0/503982595 learned_addr learned my addr 172.21.15.7:0/503982595 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:40:57.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4140b6d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f413c06bea0 0x7f413c06c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:41406/0 (socket says 172.21.15.7:41406) 2023-03-03T13:40:57.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f414136e700 1 --1- 172.21.15.7:0/503982595 >> v1:172.21.15.143:6789/0 conn(0x7f413c151ab0 0x7f413c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:43132/0 (socket says 172.21.15.7:43132) 2023-03-03T13:40:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4141b6f700 1 -- 172.21.15.7:0/503982595 >> v2:172.21.15.143:3300/0 conn(0x7f413c06bea0 msgr2=0x7f413c06c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:40:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4141b6f700 1 --2- 172.21.15.7:0/503982595 >> v2:172.21.15.143:3300/0 conn(0x7f413c06bea0 0x7f413c06c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4141b6f700 1 -- 172.21.15.7:0/503982595 >> v1:172.21.15.143:6789/0 conn(0x7f413c151ab0 legacy=0x7f413c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4141b6f700 1 -- 172.21.15.7:0/503982595 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c06ce50 con 0x7f413c157dc0 2023-03-03T13:40:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4140b6d700 1 --2- 172.21.15.7:0/503982595 >> v2:172.21.15.143:3300/0 conn(0x7f413c06bea0 0x7f413c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:40:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4141b6f700 1 --2- 172.21.15.7:0/503982595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c157dc0 0x7f413c158240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4134009e50 tx=0x7f413400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=446031547d0dcca9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4133fff700 1 -- 172.21.15.7:0/503982595 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413400e070 con 0x7f413c157dc0 2023-03-03T13:40:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4133fff700 1 -- 172.21.15.7:0/503982595 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413400b910 con 0x7f413c157dc0 2023-03-03T13:40:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.614+0000 7f4133fff700 1 -- 172.21.15.7:0/503982595 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413400bcc0 con 0x7f413c157dc0 2023-03-03T13:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.615+0000 7f41435d2700 1 -- 172.21.15.7:0/503982595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c157dc0 msgr2=0x7f413c158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.615+0000 7f41435d2700 1 --2- 172.21.15.7:0/503982595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c157dc0 0x7f413c158240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4134009e50 tx=0x7f413400b330 comp rx=0 tx=0).stop 2023-03-03T13:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.615+0000 7f41435d2700 1 -- 172.21.15.7:0/503982595 shutdown_connections 2023-03-03T13:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.615+0000 7f41435d2700 1 --2- 172.21.15.7:0/503982595 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c157dc0 0x7f413c158240 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.615+0000 7f41435d2700 1 --2- 172.21.15.7:0/503982595 >> v2:172.21.15.143:3300/0 conn(0x7f413c06bea0 0x7f413c06c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.615+0000 7f41435d2700 1 -- 172.21.15.7:0/503982595 >> 172.21.15.7:0/503982595 conn(0x7f413c0a94d0 msgr2=0x7f413c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:40:57.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.615+0000 7f41435d2700 1 -- 172.21.15.7:0/503982595 shutdown_connections 2023-03-03T13:40:57.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.615+0000 7f41435d2700 1 -- 172.21.15.7:0/503982595 wait complete. 2023-03-03T13:40:57.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.617+0000 7f41435d2700 1 Processor -- start 2023-03-03T13:40:57.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.617+0000 7f41435d2700 1 -- start start 2023-03-03T13:40:57.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f41435d2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413c06bea0 0x7f413c14e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:57.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f41435d2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c151ab0 0x7f413c1eb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:57.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f41435d2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413c157dc0 0x7f413c1eb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:57.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f41435d2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c1ebad0 con 0x7f413c157dc0 2023-03-03T13:40:57.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f41435d2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c14b860 con 0x7f413c06bea0 2023-03-03T13:40:57.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f41435d2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413c1467b0 con 0x7f413c151ab0 2023-03-03T13:40:57.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4141b6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413c157dc0 0x7f413c1eb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:57.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4141b6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413c157dc0 0x7f413c1eb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45640/0 (socket says 172.21.15.7:45640) 2023-03-03T13:40:57.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4141b6f700 1 -- 172.21.15.7:0/2811483212 learned_addr learned my addr 172.21.15.7:0/2811483212 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:40:57.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4140b6d700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c151ab0 0x7f413c1eb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:57.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f414136e700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413c06bea0 0x7f413c14e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:57.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4141b6f700 1 -- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c151ab0 msgr2=0x7f413c1eb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:57.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4141b6f700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c151ab0 0x7f413c1eb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:57.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4141b6f700 1 -- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413c06bea0 msgr2=0x7f413c14e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:57.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4141b6f700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413c06bea0 0x7f413c14e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:57.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4141b6f700 1 -- 172.21.15.7:0/2811483212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41380094a0 con 0x7f413c157dc0 2023-03-03T13:40:57.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4140b6d700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c151ab0 0x7f413c1eb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:40:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f414136e700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413c06bea0 0x7f413c14e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:40:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4141b6f700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413c157dc0 0x7f413c1eb590 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f413c06ccb0 tx=0x7f4134004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.618+0000 7f4131ffb700 1 -- 172.21.15.7:0/2811483212 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4134004960 con 0x7f413c157dc0 2023-03-03T13:40:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.619+0000 7f4131ffb700 1 -- 172.21.15.7:0/2811483212 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41340092e0 con 0x7f413c157dc0 2023-03-03T13:40:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.619+0000 7f41435d2700 1 -- 172.21.15.7:0/2811483212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4134009b30 con 0x7f413c157dc0 2023-03-03T13:40:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.619+0000 7f4131ffb700 1 -- 172.21.15.7:0/2811483212 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413401c660 con 0x7f413c157dc0 2023-03-03T13:40:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.619+0000 7f41435d2700 1 -- 172.21.15.7:0/2811483212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413c06d410 con 0x7f413c157dc0 2023-03-03T13:40:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.620+0000 7f41435d2700 1 -- 172.21.15.7:0/2811483212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413c06d130 con 0x7f413c157dc0 2023-03-03T13:40:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.620+0000 7f4131ffb700 1 -- 172.21.15.7:0/2811483212 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f413402c070 con 0x7f413c157dc0 2023-03-03T13:40:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.621+0000 7f4131ffb700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4120068320 0x7f412006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:40:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.621+0000 7f4131ffb700 1 -- 172.21.15.7:0/2811483212 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4134087dd0 con 0x7f413c157dc0 2023-03-03T13:40:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.621+0000 7f414136e700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4120068320 0x7f412006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:40:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.621+0000 7f414136e700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4120068320 0x7f412006a7d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f412c000f60 tx=0x7f412c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:40:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.625+0000 7f4131ffb700 1 -- 172.21.15.7:0/2811483212 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4134054000 con 0x7f413c157dc0 2023-03-03T13:40:57.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.863+0000 7f41435d2700 1 -- 172.21.15.7:0/2811483212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f413c073e00 con 0x7f413c157dc0 2023-03-03T13:40:57.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.864+0000 7f4131ffb700 1 -- 172.21.15.7:0/2811483212 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4134057620 con 0x7f413c157dc0 2023-03-03T13:40:57.866 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:40:57.866 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:40:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 -- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4120068320 msgr2=0x7f412006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4120068320 0x7f412006a7d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f412c000f60 tx=0x7f412c009040 comp rx=0 tx=0).stop 2023-03-03T13:40:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 -- 172.21.15.7:0/2811483212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413c157dc0 msgr2=0x7f413c1eb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:40:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413c157dc0 0x7f413c1eb590 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f413c06ccb0 tx=0x7f4134004ad0 comp rx=0 tx=0).stop 2023-03-03T13:40:57.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 -- 172.21.15.7:0/2811483212 shutdown_connections 2023-03-03T13:40:57.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4120068320 0x7f412006a7d0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:57.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413c157dc0 0x7f413c1eb590 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:57.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413c06bea0 0x7f413c14e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:57.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 --2- 172.21.15.7:0/2811483212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f413c151ab0 0x7f413c1eb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:40:57.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 -- 172.21.15.7:0/2811483212 >> 172.21.15.7:0/2811483212 conn(0x7f413c0a94d0 msgr2=0x7f413c155ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:40:57.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 -- 172.21.15.7:0/2811483212 shutdown_connections 2023-03-03T13:40:57.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:40:57.866+0000 7f41435d2700 1 -- 172.21.15.7:0/2811483212 wait complete. 2023-03-03T13:40:57.870 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:40:57.879 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T13:41:05.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:41:05.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.533+0000 7f90b0498700 1 Processor -- start 2023-03-03T13:41:05.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.533+0000 7f90b0498700 1 -- start start 2023-03-03T13:41:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.533+0000 7f90b0498700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f90a8151d40 0x7f90a80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:05.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.533+0000 7f90b0498700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f90a80c2930 0x7f90a8157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:05.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90b0498700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a806d430 con 0x7f90a8073c30 2023-03-03T13:41:05.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90b0498700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a806c2a0 con 0x7f90a8151d40 2023-03-03T13:41:05.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90b0498700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a81586e0 con 0x7f90a80c2930 2023-03-03T13:41:05.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ae234700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f90a8151d40 0x7f90a80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:05.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90aea35700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f90a8073c30 0x7f90a8157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39634/0 (socket says 172.21.15.7:39634) 2023-03-03T13:41:05.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90aea35700 1 -- 172.21.15.7:0/2597117054 learned_addr learned my addr 172.21.15.7:0/2597117054 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:41:05.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ada33700 1 --2- 172.21.15.7:0/2597117054 >> v2:172.21.15.143:3300/0 conn(0x7f90a80c2930 0x7f90a8157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:05.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ae234700 1 -- 172.21.15.7:0/2597117054 >> v2:172.21.15.143:3300/0 conn(0x7f90a80c2930 msgr2=0x7f90a8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:41:05.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ae234700 1 --2- 172.21.15.7:0/2597117054 >> v2:172.21.15.143:3300/0 conn(0x7f90a80c2930 0x7f90a8157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:05.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ae234700 1 -- 172.21.15.7:0/2597117054 >> v1:172.21.15.7:6789/0 conn(0x7f90a8073c30 legacy=0x7f90a8157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:05.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ae234700 1 -- 172.21.15.7:0/2597117054 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a806cb00 con 0x7f90a8151d40 2023-03-03T13:41:05.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ad232700 1 -- 172.21.15.7:0/2597117054 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536300446 0 0) 0x7f90a806d430 con 0x7f90a8073c30 2023-03-03T13:41:05.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ae234700 1 --2- 172.21.15.7:0/2597117054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 0x7f90a80736f0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f90a4009840 tx=0x7f90a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5722104c9e313b96 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:41:05.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ad232700 1 -- 172.21.15.7:0/2597117054 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4011830 con 0x7f90a8151d40 2023-03-03T13:41:05.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ad232700 1 -- 172.21.15.7:0/2597117054 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a4011990 con 0x7f90a8151d40 2023-03-03T13:41:05.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.534+0000 7f90ad232700 1 -- 172.21.15.7:0/2597117054 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4011d20 con 0x7f90a8151d40 2023-03-03T13:41:05.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.535+0000 7f90b0498700 1 -- 172.21.15.7:0/2597117054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 msgr2=0x7f90a80736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:05.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.535+0000 7f90b0498700 1 --2- 172.21.15.7:0/2597117054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 0x7f90a80736f0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f90a4009840 tx=0x7f90a4009b50 comp rx=0 tx=0).stop 2023-03-03T13:41:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.535+0000 7f90b0498700 1 -- 172.21.15.7:0/2597117054 shutdown_connections 2023-03-03T13:41:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.535+0000 7f90b0498700 1 --2- 172.21.15.7:0/2597117054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 0x7f90a80736f0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.535+0000 7f90b0498700 1 --2- 172.21.15.7:0/2597117054 >> v2:172.21.15.143:3300/0 conn(0x7f90a80c2930 0x7f90a8157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.535+0000 7f90b0498700 1 -- 172.21.15.7:0/2597117054 >> 172.21.15.7:0/2597117054 conn(0x7f90a80a94d0 msgr2=0x7f90a80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:41:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.535+0000 7f90b0498700 1 -- 172.21.15.7:0/2597117054 shutdown_connections 2023-03-03T13:41:05.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.535+0000 7f90b0498700 1 -- 172.21.15.7:0/2597117054 wait complete. 2023-03-03T13:41:05.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.536+0000 7f90b0498700 1 Processor -- start 2023-03-03T13:41:05.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.536+0000 7f90b0498700 1 -- start start 2023-03-03T13:41:05.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90b0498700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90a8073c30 0x7f90a8148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:05.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90b0498700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90a80c2930 0x7f90a814e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:05.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90b0498700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 0x7f90a81ea770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:05.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90b0498700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a80be430 con 0x7f90a8151d40 2023-03-03T13:41:05.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90b0498700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a8148cc0 con 0x7f90a8073c30 2023-03-03T13:41:05.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90b0498700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a81534a0 con 0x7f90a80c2930 2023-03-03T13:41:05.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90aea35700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 0x7f90a81ea770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:05.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90aea35700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 0x7f90a81ea770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45660/0 (socket says 172.21.15.7:45660) 2023-03-03T13:41:05.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90aea35700 1 -- 172.21.15.7:0/3796620393 learned_addr learned my addr 172.21.15.7:0/3796620393 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:41:05.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90ae234700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90a8073c30 0x7f90a8148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:05.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90ada33700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90a80c2930 0x7f90a814e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:05.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90aea35700 1 -- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90a80c2930 msgr2=0x7f90a814e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:05.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90aea35700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90a80c2930 0x7f90a814e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:05.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90aea35700 1 -- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90a8073c30 msgr2=0x7f90a8148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:05.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.537+0000 7f90aea35700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90a8073c30 0x7f90a8148780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:05.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.538+0000 7f90aea35700 1 -- 172.21.15.7:0/3796620393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a40094a0 con 0x7f90a8151d40 2023-03-03T13:41:05.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.538+0000 7f90ada33700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90a80c2930 0x7f90a814e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:41:05.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.538+0000 7f90aea35700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 0x7f90a81ea770 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f90a000cc10 tx=0x7f90a000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:41:05.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.538+0000 7f909effd700 1 -- 172.21.15.7:0/3796620393 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a000ab50 con 0x7f90a8151d40 2023-03-03T13:41:05.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.538+0000 7f909effd700 1 -- 172.21.15.7:0/3796620393 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a000acd0 con 0x7f90a8151d40 2023-03-03T13:41:05.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.538+0000 7f90b0498700 1 -- 172.21.15.7:0/3796620393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a806d0c0 con 0x7f90a8151d40 2023-03-03T13:41:05.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.538+0000 7f909effd700 1 -- 172.21.15.7:0/3796620393 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a0007310 con 0x7f90a8151d40 2023-03-03T13:41:05.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.538+0000 7f90b0498700 1 -- 172.21.15.7:0/3796620393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a806cde0 con 0x7f90a8151d40 2023-03-03T13:41:05.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.539+0000 7f90b0498700 1 -- 172.21.15.7:0/3796620393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a806bea0 con 0x7f90a8151d40 2023-03-03T13:41:05.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.539+0000 7f909effd700 1 -- 172.21.15.7:0/3796620393 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f90a001a030 con 0x7f90a8151d40 2023-03-03T13:41:05.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.540+0000 7f909effd700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f908c0682f0 0x7f908c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:05.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.540+0000 7f909effd700 1 -- 172.21.15.7:0/3796620393 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f90a0085d60 con 0x7f90a8151d40 2023-03-03T13:41:05.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.540+0000 7f90ae234700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f908c0682f0 0x7f908c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:05.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.540+0000 7f90ae234700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f908c0682f0 0x7f908c06a7a0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f90a40090b0 tx=0x7f90a4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:41:05.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.544+0000 7f909effd700 1 -- 172.21.15.7:0/3796620393 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f90a0052100 con 0x7f90a8151d40 2023-03-03T13:41:05.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.783+0000 7f90b0498700 1 -- 172.21.15.7:0/3796620393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a81ea3f0 con 0x7f90a8151d40 2023-03-03T13:41:05.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.783+0000 7f909effd700 1 -- 172.21.15.7:0/3796620393 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f90a0018090 con 0x7f90a8151d40 2023-03-03T13:41:05.786 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:41:05.786 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:41:05.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 -- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f908c0682f0 msgr2=0x7f908c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:05.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f908c0682f0 0x7f908c06a7a0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f90a40090b0 tx=0x7f90a4008030 comp rx=0 tx=0).stop 2023-03-03T13:41:05.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 -- 172.21.15.7:0/3796620393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 msgr2=0x7f90a81ea770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:05.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 0x7f90a81ea770 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f90a000cc10 tx=0x7f90a000cf20 comp rx=0 tx=0).stop 2023-03-03T13:41:05.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 -- 172.21.15.7:0/3796620393 shutdown_connections 2023-03-03T13:41:05.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f908c0682f0 0x7f908c06a7a0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:05.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90a8151d40 0x7f90a81ea770 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:05.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90a8073c30 0x7f90a8148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:05.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 --2- 172.21.15.7:0/3796620393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90a80c2930 0x7f90a814e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:05.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 -- 172.21.15.7:0/3796620393 >> 172.21.15.7:0/3796620393 conn(0x7f90a80a94d0 msgr2=0x7f90a8150000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:41:05.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 -- 172.21.15.7:0/3796620393 shutdown_connections 2023-03-03T13:41:05.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:05.785+0000 7f90b0498700 1 -- 172.21.15.7:0/3796620393 wait complete. 2023-03-03T13:41:05.790 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:41:05.800 INFO:tasks.mds_thrash.fs.[a]:waiting for 28 secs before thrashing 2023-03-03T13:41:14.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:14.955 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:33.801 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:41:33.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.938+0000 7f6b9fec1700 1 Processor -- start 2023-03-03T13:41:33.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.939+0000 7f6b9fec1700 1 -- start start 2023-03-03T13:41:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.939+0000 7f6b9fec1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6b9806e020 0x7f6b98071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.939+0000 7f6b9fec1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b98072520 0x7f6b98151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.939+0000 7f6b9fec1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b98154910 con 0x7f6b98155390 2023-03-03T13:41:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.939+0000 7f6b9fec1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b98070b90 con 0x7f6b98072520 2023-03-03T13:41:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.939+0000 7f6b9fec1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b98151320 con 0x7f6b9806e020 2023-03-03T13:41:33.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9e45e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b98072520 0x7f6b98151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9e45e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b98072520 0x7f6b98151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45696/0 (socket says 172.21.15.7:45696) 2023-03-03T13:41:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9e45e700 1 -- 172.21.15.7:0/2960538807 learned_addr learned my addr 172.21.15.7:0/2960538807 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:41:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9d45c700 1 --2- 172.21.15.7:0/2960538807 >> v2:172.21.15.143:3300/0 conn(0x7f6b9806e020 0x7f6b98071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9e45e700 1 -- 172.21.15.7:0/2960538807 >> v2:172.21.15.143:3300/0 conn(0x7f6b9806e020 msgr2=0x7f6b98071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:41:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9e45e700 1 --2- 172.21.15.7:0/2960538807 >> v2:172.21.15.143:3300/0 conn(0x7f6b9806e020 0x7f6b98071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9e45e700 1 -- 172.21.15.7:0/2960538807 >> v1:172.21.15.143:6789/0 conn(0x7f6b98155390 legacy=0x7f6b980c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:33.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9e45e700 1 -- 172.21.15.7:0/2960538807 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b980c4540 con 0x7f6b98072520 2023-03-03T13:41:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9e45e700 1 --2- 172.21.15.7:0/2960538807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98072520 0x7f6b98151770 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f6b980ac9e0 tx=0x7f6b9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d776f664288176b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:41:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9cc5b700 1 -- 172.21.15.7:0/2960538807 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9000e070 con 0x7f6b98072520 2023-03-03T13:41:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9cc5b700 1 -- 172.21.15.7:0/2960538807 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9000b9c0 con 0x7f6b98072520 2023-03-03T13:41:33.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.940+0000 7f6b9cc5b700 1 -- 172.21.15.7:0/2960538807 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9000bd70 con 0x7f6b98072520 2023-03-03T13:41:33.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.941+0000 7f6b9fec1700 1 -- 172.21.15.7:0/2960538807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98072520 msgr2=0x7f6b98151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:33.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.941+0000 7f6b9fec1700 1 --2- 172.21.15.7:0/2960538807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98072520 0x7f6b98151770 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f6b980ac9e0 tx=0x7f6b9000b330 comp rx=0 tx=0).stop 2023-03-03T13:41:33.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.941+0000 7f6b9fec1700 1 -- 172.21.15.7:0/2960538807 shutdown_connections 2023-03-03T13:41:33.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.941+0000 7f6b9fec1700 1 --2- 172.21.15.7:0/2960538807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98072520 0x7f6b98151770 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.941+0000 7f6b9fec1700 1 --2- 172.21.15.7:0/2960538807 >> v2:172.21.15.143:3300/0 conn(0x7f6b9806e020 0x7f6b98071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.941+0000 7f6b9fec1700 1 -- 172.21.15.7:0/2960538807 >> 172.21.15.7:0/2960538807 conn(0x7f6b9806c110 msgr2=0x7f6b9806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:41:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.941+0000 7f6b9fec1700 1 -- 172.21.15.7:0/2960538807 shutdown_connections 2023-03-03T13:41:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.941+0000 7f6b9fec1700 1 -- 172.21.15.7:0/2960538807 wait complete. 2023-03-03T13:41:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.942+0000 7f6b9fec1700 1 Processor -- start 2023-03-03T13:41:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.942+0000 7f6b9fec1700 1 -- start start 2023-03-03T13:41:33.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9fec1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b9806e020 0x7f6b981dbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9fec1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98072520 0x7f6b981da980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9fec1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98155390 0x7f6b98147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9fec1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b98149670 con 0x7f6b98155390 2023-03-03T13:41:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9fec1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b98148e40 con 0x7f6b98072520 2023-03-03T13:41:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9fec1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b98148b60 con 0x7f6b9806e020 2023-03-03T13:41:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9e45e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98155390 0x7f6b98147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9e45e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98155390 0x7f6b98147f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45702/0 (socket says 172.21.15.7:45702) 2023-03-03T13:41:33.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9e45e700 1 -- 172.21.15.7:0/1718103878 learned_addr learned my addr 172.21.15.7:0/1718103878 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:41:33.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9d45c700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98072520 0x7f6b981da980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:33.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9dc5d700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b9806e020 0x7f6b981dbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:33.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9e45e700 1 -- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b9806e020 msgr2=0x7f6b981dbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:33.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9e45e700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b9806e020 0x7f6b981dbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:33.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9e45e700 1 -- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98072520 msgr2=0x7f6b981da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:33.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9e45e700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98072520 0x7f6b981da980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:33.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9e45e700 1 -- 172.21.15.7:0/1718103878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b90009b30 con 0x7f6b98155390 2023-03-03T13:41:33.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9d45c700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98072520 0x7f6b981da980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:41:33.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9e45e700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98155390 0x7f6b98147f40 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f6b980c4a90 tx=0x7f6b90004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:41:33.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b8a7fc700 1 -- 172.21.15.7:0/1718103878 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b900092e0 con 0x7f6b98155390 2023-03-03T13:41:33.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b8a7fc700 1 -- 172.21.15.7:0/1718103878 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9001c440 con 0x7f6b98155390 2023-03-03T13:41:33.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.943+0000 7f6b9fec1700 1 -- 172.21.15.7:0/1718103878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b98070540 con 0x7f6b98155390 2023-03-03T13:41:33.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.944+0000 7f6b8a7fc700 1 -- 172.21.15.7:0/1718103878 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9001c7f0 con 0x7f6b98155390 2023-03-03T13:41:33.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.944+0000 7f6b9fec1700 1 -- 172.21.15.7:0/1718103878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b980702a0 con 0x7f6b98155390 2023-03-03T13:41:33.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.945+0000 7f6b9fec1700 1 -- 172.21.15.7:0/1718103878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b98070820 con 0x7f6b98155390 2023-03-03T13:41:33.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.945+0000 7f6b8a7fc700 1 -- 172.21.15.7:0/1718103878 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6b9002c070 con 0x7f6b98155390 2023-03-03T13:41:33.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.945+0000 7f6b8a7fc700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 0x7f6b7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:41:33.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.945+0000 7f6b8a7fc700 1 -- 172.21.15.7:0/1718103878 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6b90087bc0 con 0x7f6b98155390 2023-03-03T13:41:33.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.946+0000 7f6b9dc5d700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 0x7f6b7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:41:33.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.946+0000 7f6b9dc5d700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 0x7f6b7c06a7a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c000f60 tx=0x7f6b8c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:41:33.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:33.950+0000 7f6b8a7fc700 1 -- 172.21.15.7:0/1718103878 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b90053f60 con 0x7f6b98155390 2023-03-03T13:41:34.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.188+0000 7f6b9fec1700 1 -- 172.21.15.7:0/1718103878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b980c5e90 con 0x7f6b98155390 2023-03-03T13:41:34.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.189+0000 7f6b8a7fc700 1 -- 172.21.15.7:0/1718103878 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6b90057580 con 0x7f6b98155390 2023-03-03T13:41:34.191 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:41:34.191 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:41:34.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 -- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 msgr2=0x7f6b7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:34.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 0x7f6b7c06a7a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c000f60 tx=0x7f6b8c009040 comp rx=0 tx=0).stop 2023-03-03T13:41:34.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 -- 172.21.15.7:0/1718103878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98155390 msgr2=0x7f6b98147f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:41:34.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98155390 0x7f6b98147f40 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f6b980c4a90 tx=0x7f6b90004ad0 comp rx=0 tx=0).stop 2023-03-03T13:41:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 -- 172.21.15.7:0/1718103878 shutdown_connections 2023-03-03T13:41:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b7c0682f0 0x7f6b7c06a7a0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b98155390 0x7f6b98147f40 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b9806e020 0x7f6b981dbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 --2- 172.21.15.7:0/1718103878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b98072520 0x7f6b981da980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:41:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 -- 172.21.15.7:0/1718103878 >> 172.21.15.7:0/1718103878 conn(0x7f6b9806c110 msgr2=0x7f6b98073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:41:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 -- 172.21.15.7:0/1718103878 shutdown_connections 2023-03-03T13:41:34.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:41:34.191+0000 7f6b9fec1700 1 -- 172.21.15.7:0/1718103878 wait complete. 2023-03-03T13:41:34.195 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:41:34.204 INFO:tasks.mds_thrash.fs.[a]:waiting for 109 secs before thrashing 2023-03-03T13:41:45.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:45.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:41:59.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:42:00.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.018+0000 7fba24914700 1 Processor -- start 2023-03-03T13:42:00.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.018+0000 7fba24914700 1 -- start start 2023-03-03T13:42:00.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba24914700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fba200c2950 0x7fba20157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:00.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba24914700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba2006c480 0x7fba20158020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:00.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba24914700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2006cb70 con 0x7fba20151b20 2023-03-03T13:42:00.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba24914700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba2006c2f0 con 0x7fba200c2950 2023-03-03T13:42:00.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba24914700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2006ce50 con 0x7fba2006c480 2023-03-03T13:42:00.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba1ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba2006c480 0x7fba20158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:00.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba1dd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fba200c2950 0x7fba20157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:00.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba1ed9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba2006c480 0x7fba20158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:53794/0 (socket says 172.21.15.7:53794) 2023-03-03T13:42:00.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba1ed9d700 1 -- 172.21.15.7:0/2756761534 learned_addr learned my addr 172.21.15.7:0/2756761534 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:42:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba1ed9d700 1 -- 172.21.15.7:0/2756761534 >> v2:172.21.15.143:3300/0 conn(0x7fba200c2950 msgr2=0x7fba20157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:42:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba1ed9d700 1 --2- 172.21.15.7:0/2756761534 >> v2:172.21.15.143:3300/0 conn(0x7fba200c2950 0x7fba20157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba1ed9d700 1 -- 172.21.15.7:0/2756761534 >> v1:172.21.15.143:6789/0 conn(0x7fba20151b20 legacy=0x7fba200736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba1ed9d700 1 -- 172.21.15.7:0/2756761534 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2006d130 con 0x7fba2006c480 2023-03-03T13:42:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.019+0000 7fba1dd9b700 1 --2- 172.21.15.7:0/2756761534 >> v2:172.21.15.143:3300/0 conn(0x7fba200c2950 0x7fba20157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:42:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.020+0000 7fba1ed9d700 1 --2- 172.21.15.7:0/2756761534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba2006c480 0x7fba20158020 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fba18009e50 tx=0x7fba1800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f69546cc2a64ebaf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:42:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.020+0000 7fba1d59a700 1 -- 172.21.15.7:0/2756761534 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1800e070 con 0x7fba2006c480 2023-03-03T13:42:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.020+0000 7fba1d59a700 1 -- 172.21.15.7:0/2756761534 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1800b910 con 0x7fba2006c480 2023-03-03T13:42:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.020+0000 7fba1d59a700 1 -- 172.21.15.7:0/2756761534 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1800bcc0 con 0x7fba2006c480 2023-03-03T13:42:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.020+0000 7fba24914700 1 -- 172.21.15.7:0/2756761534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba2006c480 msgr2=0x7fba20158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:00.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.020+0000 7fba24914700 1 --2- 172.21.15.7:0/2756761534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba2006c480 0x7fba20158020 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fba18009e50 tx=0x7fba1800b330 comp rx=0 tx=0).stop 2023-03-03T13:42:00.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.020+0000 7fba24914700 1 -- 172.21.15.7:0/2756761534 shutdown_connections 2023-03-03T13:42:00.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.020+0000 7fba24914700 1 --2- 172.21.15.7:0/2756761534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba2006c480 0x7fba20158020 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:00.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.020+0000 7fba24914700 1 --2- 172.21.15.7:0/2756761534 >> v2:172.21.15.143:3300/0 conn(0x7fba200c2950 0x7fba20157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:00.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.020+0000 7fba24914700 1 -- 172.21.15.7:0/2756761534 >> 172.21.15.7:0/2756761534 conn(0x7fba200a94d0 msgr2=0x7fba200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:42:00.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.021+0000 7fba24914700 1 -- 172.21.15.7:0/2756761534 shutdown_connections 2023-03-03T13:42:00.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.021+0000 7fba24914700 1 -- 172.21.15.7:0/2756761534 wait complete. 2023-03-03T13:42:00.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.022+0000 7fba24914700 1 Processor -- start 2023-03-03T13:42:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.022+0000 7fba24914700 1 -- start start 2023-03-03T13:42:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba24914700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba2006c480 0x7fba200bf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba24914700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba200c2950 0x7fba20153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba24914700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba20151b20 0x7fba20148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba24914700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba200bff80 con 0x7fba20151b20 2023-03-03T13:42:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba24914700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba200c14f0 con 0x7fba2006c480 2023-03-03T13:42:00.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba24914700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba20153000 con 0x7fba200c2950 2023-03-03T13:42:00.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba20151b20 0x7fba20148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:00.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba20151b20 0x7fba20148740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45728/0 (socket says 172.21.15.7:45728) 2023-03-03T13:42:00.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1ed9d700 1 -- 172.21.15.7:0/3343397589 learned_addr learned my addr 172.21.15.7:0/3343397589 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:42:00.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1e59c700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba2006c480 0x7fba200bf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:00.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1dd9b700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba200c2950 0x7fba20153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:00.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1ed9d700 1 -- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba200c2950 msgr2=0x7fba20153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:00.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1ed9d700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba200c2950 0x7fba20153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:00.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1ed9d700 1 -- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba2006c480 msgr2=0x7fba200bf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:00.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1ed9d700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba2006c480 0x7fba200bf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:00.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1ed9d700 1 -- 172.21.15.7:0/3343397589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba080094a0 con 0x7fba20151b20 2023-03-03T13:42:00.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba1ed9d700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba20151b20 0x7fba20148740 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fba1800b680 tx=0x7fba18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:42:00.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba16ffd700 1 -- 172.21.15.7:0/3343397589 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba180099b0 con 0x7fba20151b20 2023-03-03T13:42:00.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba16ffd700 1 -- 172.21.15.7:0/3343397589 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba180092e0 con 0x7fba20151b20 2023-03-03T13:42:00.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba24914700 1 -- 172.21.15.7:0/3343397589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba18009b30 con 0x7fba20151b20 2023-03-03T13:42:00.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.023+0000 7fba16ffd700 1 -- 172.21.15.7:0/3343397589 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1801b630 con 0x7fba20151b20 2023-03-03T13:42:00.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.024+0000 7fba24914700 1 -- 172.21.15.7:0/3343397589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba20151ef0 con 0x7fba20151b20 2023-03-03T13:42:00.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.024+0000 7fba24914700 1 -- 172.21.15.7:0/3343397589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2014f880 con 0x7fba20151b20 2023-03-03T13:42:00.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.025+0000 7fba16ffd700 1 -- 172.21.15.7:0/3343397589 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fba1802b070 con 0x7fba20151b20 2023-03-03T13:42:00.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.025+0000 7fba16ffd700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 0x7fba0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:42:00.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.025+0000 7fba16ffd700 1 -- 172.21.15.7:0/3343397589 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fba18087960 con 0x7fba20151b20 2023-03-03T13:42:00.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.026+0000 7fba1e59c700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 0x7fba0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:42:00.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.026+0000 7fba1e59c700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 0x7fba0006a7a0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fba200c0b10 tx=0x7fba10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:42:00.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.030+0000 7fba16ffd700 1 -- 172.21.15.7:0/3343397589 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fba18053d00 con 0x7fba20151b20 2023-03-03T13:42:00.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.268+0000 7fba24914700 1 -- 172.21.15.7:0/3343397589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba20073e00 con 0x7fba20151b20 2023-03-03T13:42:00.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.268+0000 7fba16ffd700 1 -- 172.21.15.7:0/3343397589 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fba18057320 con 0x7fba20151b20 2023-03-03T13:42:00.270 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:42:00.270 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:42:00.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 -- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 msgr2=0x7fba0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:00.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 0x7fba0006a7a0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fba200c0b10 tx=0x7fba10009040 comp rx=0 tx=0).stop 2023-03-03T13:42:00.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 -- 172.21.15.7:0/3343397589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba20151b20 msgr2=0x7fba20148740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:42:00.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba20151b20 0x7fba20148740 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fba1800b680 tx=0x7fba18004ad0 comp rx=0 tx=0).stop 2023-03-03T13:42:00.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 -- 172.21.15.7:0/3343397589 shutdown_connections 2023-03-03T13:42:00.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 0x7fba0006a7a0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:00.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba20151b20 0x7fba20148740 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:00.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba2006c480 0x7fba200bf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:00.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 --2- 172.21.15.7:0/3343397589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba200c2950 0x7fba20153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:42:00.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 -- 172.21.15.7:0/3343397589 >> 172.21.15.7:0/3343397589 conn(0x7fba200a94d0 msgr2=0x7fba20158210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:42:00.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 -- 172.21.15.7:0/3343397589 shutdown_connections 2023-03-03T13:42:00.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:42:00.270+0000 7fba24914700 1 -- 172.21.15.7:0/3343397589 wait complete. 2023-03-03T13:42:00.274 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:42:00.284 INFO:tasks.mds_thrash.fs.[b]:waiting for 110 secs before thrashing 2023-03-03T13:42:15.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:15.165 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:45.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:42:45.320 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:15.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:15.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:23.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:43:23.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.342+0000 7f3553c82700 1 Processor -- start 2023-03-03T13:43:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.342+0000 7f3553c82700 1 -- start start 2023-03-03T13:43:23.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f3553c82700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f354c06e020 0x7f354c071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f3553c82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c072520 0x7f354c151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f3553c82700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354c154910 con 0x7f354c155390 2023-03-03T13:43:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f3553c82700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f354c070b90 con 0x7f354c06e020 2023-03-03T13:43:23.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f3553c82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354c151320 con 0x7f354c072520 2023-03-03T13:43:23.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f355221f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c072520 0x7f354c151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:23.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f355121d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f354c06e020 0x7f354c071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:23.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f3551a1e700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f354c155390 0x7f354c0c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:43318/0 (socket says 172.21.15.7:43318) 2023-03-03T13:43:23.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f355221f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c072520 0x7f354c151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:53894/0 (socket says 172.21.15.7:53894) 2023-03-03T13:43:23.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f3551a1e700 1 -- 172.21.15.7:0/981908445 learned_addr learned my addr 172.21.15.7:0/981908445 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:43:23.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.343+0000 7f355121d700 1 --2- 172.21.15.7:0/981908445 >> v2:172.21.15.143:3300/0 conn(0x7f354c06e020 0x7f354c071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:41594/0 (socket says 172.21.15.7:41594) 2023-03-03T13:43:23.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.344+0000 7f355121d700 1 -- 172.21.15.7:0/981908445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c072520 msgr2=0x7f354c151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:43:23.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.344+0000 7f355121d700 1 --2- 172.21.15.7:0/981908445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c072520 0x7f354c151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:23.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.344+0000 7f355121d700 1 -- 172.21.15.7:0/981908445 >> v1:172.21.15.143:6789/0 conn(0x7f354c155390 legacy=0x7f354c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:23.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.344+0000 7f355121d700 1 -- 172.21.15.7:0/981908445 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c0c4540 con 0x7f354c06e020 2023-03-03T13:43:23.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.344+0000 7f3550a1c700 1 -- 172.21.15.7:0/981908445 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350384638 0 0) 0x7f354c154910 con 0x7f354c155390 2023-03-03T13:43:23.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.344+0000 7f355221f700 1 --2- 172.21.15.7:0/981908445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c072520 0x7f354c151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:43:23.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.344+0000 7f355121d700 1 --2- 172.21.15.7:0/981908445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f354c06e020 0x7f354c071fe0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3548009840 tx=0x7f3548009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cb1f8353d763396 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:23.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.344+0000 7f3550a1c700 1 -- 172.21.15.7:0/981908445 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548011830 con 0x7f354c06e020 2023-03-03T13:43:23.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.344+0000 7f3550a1c700 1 -- 172.21.15.7:0/981908445 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3548011990 con 0x7f354c06e020 2023-03-03T13:43:23.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.344+0000 7f3550a1c700 1 -- 172.21.15.7:0/981908445 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548011d20 con 0x7f354c06e020 2023-03-03T13:43:23.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.345+0000 7f3553c82700 1 -- 172.21.15.7:0/981908445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f354c06e020 msgr2=0x7f354c071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:23.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.345+0000 7f3553c82700 1 --2- 172.21.15.7:0/981908445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f354c06e020 0x7f354c071fe0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3548009840 tx=0x7f3548009b50 comp rx=0 tx=0).stop 2023-03-03T13:43:23.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.345+0000 7f3553c82700 1 -- 172.21.15.7:0/981908445 shutdown_connections 2023-03-03T13:43:23.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.345+0000 7f3553c82700 1 --2- 172.21.15.7:0/981908445 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c072520 0x7f354c151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:23.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.345+0000 7f3553c82700 1 --2- 172.21.15.7:0/981908445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f354c06e020 0x7f354c071fe0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:23.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.345+0000 7f3553c82700 1 -- 172.21.15.7:0/981908445 >> 172.21.15.7:0/981908445 conn(0x7f354c06c110 msgr2=0x7f354c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:43:23.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.345+0000 7f3553c82700 1 -- 172.21.15.7:0/981908445 shutdown_connections 2023-03-03T13:43:23.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.345+0000 7f3553c82700 1 -- 172.21.15.7:0/981908445 wait complete. 2023-03-03T13:43:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.346+0000 7f3553c82700 1 Processor -- start 2023-03-03T13:43:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.346+0000 7f3553c82700 1 -- start start 2023-03-03T13:43:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f3553c82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c06e020 0x7f354c1e8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f3553c82700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f354c072520 0x7f354c0c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:23.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f3553c82700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f354c155390 0x7f354c0aaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f3553c82700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c1e7be0 con 0x7f354c072520 2023-03-03T13:43:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f3553c82700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c0abfa0 con 0x7f354c155390 2023-03-03T13:43:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f3553c82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354c0c71b0 con 0x7f354c06e020 2023-03-03T13:43:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f355121d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f354c072520 0x7f354c0c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f355121d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f354c072520 0x7f354c0c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45826/0 (socket says 172.21.15.7:45826) 2023-03-03T13:43:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f355121d700 1 -- 172.21.15.7:0/417994261 learned_addr learned my addr 172.21.15.7:0/417994261 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:43:23.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f355221f700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f354c155390 0x7f354c0aaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f3551a1e700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c06e020 0x7f354c1e8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f355121d700 1 -- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c06e020 msgr2=0x7f354c1e8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f355121d700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c06e020 0x7f354c1e8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f355121d700 1 -- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f354c155390 msgr2=0x7f354c0aaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f355121d700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f354c155390 0x7f354c0aaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f355121d700 1 -- 172.21.15.7:0/417994261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3544009a70 con 0x7f354c072520 2023-03-03T13:43:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.347+0000 7f355121d700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f354c072520 0x7f354c0c53b0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f3548009fd0 tx=0x7f3548004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:23.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.348+0000 7f35427fc700 1 -- 172.21.15.7:0/417994261 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35480045d0 con 0x7f354c072520 2023-03-03T13:43:23.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.348+0000 7f35427fc700 1 -- 172.21.15.7:0/417994261 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354801f070 con 0x7f354c072520 2023-03-03T13:43:23.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.348+0000 7f3553c82700 1 -- 172.21.15.7:0/417994261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35480094a0 con 0x7f354c072520 2023-03-03T13:43:23.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.348+0000 7f35427fc700 1 -- 172.21.15.7:0/417994261 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35480038a0 con 0x7f354c072520 2023-03-03T13:43:23.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.348+0000 7f3553c82700 1 -- 172.21.15.7:0/417994261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354c070540 con 0x7f354c072520 2023-03-03T13:43:23.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.349+0000 7f3553c82700 1 -- 172.21.15.7:0/417994261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354c1520c0 con 0x7f354c072520 2023-03-03T13:43:23.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.349+0000 7f35427fc700 1 -- 172.21.15.7:0/417994261 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f354800f050 con 0x7f354c072520 2023-03-03T13:43:23.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.349+0000 7f35427fc700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35300682f0 0x7f353006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:23.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.349+0000 7f35427fc700 1 -- 172.21.15.7:0/417994261 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3548021070 con 0x7f354c072520 2023-03-03T13:43:23.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.350+0000 7f3551a1e700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35300682f0 0x7f353006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:23.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.350+0000 7f3551a1e700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35300682f0 0x7f353006a7a0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f353c000f60 tx=0x7f353c007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:23.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.354+0000 7f35427fc700 1 -- 172.21.15.7:0/417994261 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3548053bd0 con 0x7f354c072520 2023-03-03T13:43:23.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.592+0000 7f3553c82700 1 -- 172.21.15.7:0/417994261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f354c070820 con 0x7f354c072520 2023-03-03T13:43:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.592+0000 7f35427fc700 1 -- 172.21.15.7:0/417994261 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f35480571f0 con 0x7f354c072520 2023-03-03T13:43:23.594 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:43:23.594 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:43:23.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 -- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35300682f0 msgr2=0x7f353006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:23.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35300682f0 0x7f353006a7a0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f353c000f60 tx=0x7f353c007e70 comp rx=0 tx=0).stop 2023-03-03T13:43:23.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 -- 172.21.15.7:0/417994261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f354c072520 msgr2=0x7f354c0c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f354c072520 0x7f354c0c53b0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f3548009fd0 tx=0x7f3548004750 comp rx=0 tx=0).stop 2023-03-03T13:43:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 -- 172.21.15.7:0/417994261 shutdown_connections 2023-03-03T13:43:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f354c155390 0x7f354c0aaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f354c06e020 0x7f354c1e8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35300682f0 0x7f353006a7a0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 --2- 172.21.15.7:0/417994261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f354c072520 0x7f354c0c53b0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 -- 172.21.15.7:0/417994261 >> 172.21.15.7:0/417994261 conn(0x7f354c06c110 msgr2=0x7f354c1dcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:43:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 -- 172.21.15.7:0/417994261 shutdown_connections 2023-03-03T13:43:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:23.594+0000 7f3553c82700 1 -- 172.21.15.7:0/417994261 wait complete. 2023-03-03T13:43:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:43:23.608 INFO:tasks.mds_thrash.fs.[a]:waiting for 25 secs before thrashing 2023-03-03T13:43:45.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:45.620 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:43:48.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:43:48.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.747+0000 7f01561e2700 1 Processor -- start 2023-03-03T13:43:48.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.747+0000 7f01561e2700 1 -- start start 2023-03-03T13:43:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.747+0000 7f01561e2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f015014fda0 0x7f0150156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.747+0000 7f01561e2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01500abe20 0x7f01501529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.747+0000 7f01561e2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01500c2f30 con 0x7f015014f9d0 2023-03-03T13:43:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.747+0000 7f01561e2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01500c31d0 con 0x7f01500abe20 2023-03-03T13:43:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.747+0000 7f01561e2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01501524a0 con 0x7f015014fda0 2023-03-03T13:43:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01500abe20 0x7f01501529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01500abe20 0x7f01501529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45868/0 (socket says 172.21.15.7:45868) 2023-03-03T13:43:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014ffff700 1 -- 172.21.15.7:0/977687989 learned_addr learned my addr 172.21.15.7:0/977687989 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:43:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014effd700 1 --2- 172.21.15.7:0/977687989 >> v2:172.21.15.143:3300/0 conn(0x7f015014fda0 0x7f0150156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014ffff700 1 -- 172.21.15.7:0/977687989 >> v2:172.21.15.143:3300/0 conn(0x7f015014fda0 msgr2=0x7f0150156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:43:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014ffff700 1 --2- 172.21.15.7:0/977687989 >> v2:172.21.15.143:3300/0 conn(0x7f015014fda0 0x7f0150156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014ffff700 1 -- 172.21.15.7:0/977687989 >> v1:172.21.15.143:6789/0 conn(0x7f015014f9d0 legacy=0x7f0150071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014ffff700 1 -- 172.21.15.7:0/977687989 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01500c34b0 con 0x7f01500abe20 2023-03-03T13:43:48.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014effd700 1 --2- 172.21.15.7:0/977687989 >> v2:172.21.15.143:3300/0 conn(0x7f015014fda0 0x7f0150156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:43:48.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014ffff700 1 --2- 172.21.15.7:0/977687989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01500abe20 0x7f01501529c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f01500709b0 tx=0x7f014800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8033089c2c2aa023 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:48.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014e7fc700 1 -- 172.21.15.7:0/977687989 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014800e070 con 0x7f01500abe20 2023-03-03T13:43:48.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014e7fc700 1 -- 172.21.15.7:0/977687989 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014800b9c0 con 0x7f01500abe20 2023-03-03T13:43:48.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.748+0000 7f014e7fc700 1 -- 172.21.15.7:0/977687989 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014800bd70 con 0x7f01500abe20 2023-03-03T13:43:48.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.749+0000 7f01561e2700 1 -- 172.21.15.7:0/977687989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01500abe20 msgr2=0x7f01501529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:48.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.749+0000 7f01561e2700 1 --2- 172.21.15.7:0/977687989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01500abe20 0x7f01501529c0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f01500709b0 tx=0x7f014800b330 comp rx=0 tx=0).stop 2023-03-03T13:43:48.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.749+0000 7f01561e2700 1 -- 172.21.15.7:0/977687989 shutdown_connections 2023-03-03T13:43:48.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.749+0000 7f01561e2700 1 --2- 172.21.15.7:0/977687989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01500abe20 0x7f01501529c0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:48.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.749+0000 7f01561e2700 1 --2- 172.21.15.7:0/977687989 >> v2:172.21.15.143:3300/0 conn(0x7f015014fda0 0x7f0150156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:48.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.749+0000 7f01561e2700 1 -- 172.21.15.7:0/977687989 >> 172.21.15.7:0/977687989 conn(0x7f01500c8900 msgr2=0x7f01500c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:43:48.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.749+0000 7f01561e2700 1 -- 172.21.15.7:0/977687989 shutdown_connections 2023-03-03T13:43:48.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.749+0000 7f01561e2700 1 -- 172.21.15.7:0/977687989 wait complete. 2023-03-03T13:43:48.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.750+0000 7f01561e2700 1 Processor -- start 2023-03-03T13:43:48.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.750+0000 7f01561e2700 1 -- start start 2023-03-03T13:43:48.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f01561e2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01500abe20 0x7f015006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:48.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f01561e2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015014f9d0 0x7f01501eb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:48.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f01561e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f015014fda0 0x7f01500a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:48.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f01561e2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01501dee90 con 0x7f015014f9d0 2023-03-03T13:43:48.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f01561e2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01501df170 con 0x7f01500abe20 2023-03-03T13:43:48.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f01561e2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01501ea8b0 con 0x7f015014fda0 2023-03-03T13:43:48.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f014effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015014f9d0 0x7f01501eb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:48.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f014effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015014f9d0 0x7f01501eb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45872/0 (socket says 172.21.15.7:45872) 2023-03-03T13:43:48.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f014effd700 1 -- 172.21.15.7:0/2483634615 learned_addr learned my addr 172.21.15.7:0/2483634615 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:43:48.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f014f7fe700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01500abe20 0x7f015006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:48.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f014ffff700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f015014fda0 0x7f01500a9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:48.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f014effd700 1 -- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f015014fda0 msgr2=0x7f01500a9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:48.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f014effd700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f015014fda0 0x7f01500a9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:48.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f014effd700 1 -- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01500abe20 msgr2=0x7f015006d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:48.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f014effd700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01500abe20 0x7f015006d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:48.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.751+0000 7f014effd700 1 -- 172.21.15.7:0/2483634615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0148009b30 con 0x7f015014f9d0 2023-03-03T13:43:48.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.752+0000 7f014effd700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015014f9d0 0x7f01501eb090 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f014400e7a0 tx=0x7f014400eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:48.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.752+0000 7f0137fff700 1 -- 172.21.15.7:0/2483634615 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144013610 con 0x7f015014f9d0 2023-03-03T13:43:48.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.752+0000 7f0137fff700 1 -- 172.21.15.7:0/2483634615 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0144013770 con 0x7f015014f9d0 2023-03-03T13:43:48.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.752+0000 7f01561e2700 1 -- 172.21.15.7:0/2483634615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01500ac350 con 0x7f015014f9d0 2023-03-03T13:43:48.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.752+0000 7f0137fff700 1 -- 172.21.15.7:0/2483634615 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144013b00 con 0x7f015014f9d0 2023-03-03T13:43:48.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.752+0000 7f01561e2700 1 -- 172.21.15.7:0/2483634615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01500ab630 con 0x7f015014f9d0 2023-03-03T13:43:48.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.753+0000 7f01561e2700 1 -- 172.21.15.7:0/2483634615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0150071fa0 con 0x7f015014f9d0 2023-03-03T13:43:48.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.753+0000 7f0137fff700 1 -- 172.21.15.7:0/2483634615 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0144023070 con 0x7f015014f9d0 2023-03-03T13:43:48.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.754+0000 7f0137fff700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f013c0682f0 0x7f013c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.754+0000 7f0137fff700 1 -- 172.21.15.7:0/2483634615 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0144085c00 con 0x7f015014f9d0 2023-03-03T13:43:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.754+0000 7f014f7fe700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f013c0682f0 0x7f013c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.755+0000 7f014f7fe700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f013c0682f0 0x7f013c06a7a0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0140007fd0 tx=0x7f0140009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:48.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.758+0000 7f0137fff700 1 -- 172.21.15.7:0/2483634615 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f014408b020 con 0x7f015014f9d0 2023-03-03T13:43:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.995+0000 7f01561e2700 1 -- 172.21.15.7:0/2483634615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0150074bb0 con 0x7f015014f9d0 2023-03-03T13:43:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.996+0000 7f0137fff700 1 -- 172.21.15.7:0/2483634615 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0144051fa0 con 0x7f015014f9d0 2023-03-03T13:43:48.998 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:43:48.998 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:43:48.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 -- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f013c0682f0 msgr2=0x7f013c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:49.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f013c0682f0 0x7f013c06a7a0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f0140007fd0 tx=0x7f0140009040 comp rx=0 tx=0).stop 2023-03-03T13:43:49.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 -- 172.21.15.7:0/2483634615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015014f9d0 msgr2=0x7f01501eb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:49.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015014f9d0 0x7f01501eb090 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f014400e7a0 tx=0x7f014400eab0 comp rx=0 tx=0).stop 2023-03-03T13:43:49.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 -- 172.21.15.7:0/2483634615 shutdown_connections 2023-03-03T13:43:49.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f015014fda0 0x7f01500a9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:49.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01500abe20 0x7f015006d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f013c0682f0 0x7f013c06a7a0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 --2- 172.21.15.7:0/2483634615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015014f9d0 0x7f01501eb090 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 -- 172.21.15.7:0/2483634615 >> 172.21.15.7:0/2483634615 conn(0x7f01500c8900 msgr2=0x7f01501556a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:43:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 -- 172.21.15.7:0/2483634615 shutdown_connections 2023-03-03T13:43:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:48.998+0000 7f01561e2700 1 -- 172.21.15.7:0/2483634615 wait complete. 2023-03-03T13:43:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:43:49.011 INFO:tasks.mds_thrash.fs.[a]:waiting for 23 secs before thrashing 2023-03-03T13:43:50.285 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:43:50.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.422+0000 7f8a09720700 1 Processor -- start 2023-03-03T13:43:50.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.422+0000 7f8a09720700 1 -- start start 2023-03-03T13:43:50.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a09720700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8a040caa30 0x7f8a040c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a09720700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a040c00e0 0x7f8a04154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a09720700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a040c7310 con 0x7f8a040cae00 2023-03-03T13:43:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a09720700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a040cb240 con 0x7f8a040c00e0 2023-03-03T13:43:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a09720700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a040bfd10 con 0x7f8a040caa30 2023-03-03T13:43:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a027fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a040c00e0 0x7f8a04154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a027fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a040c00e0 0x7f8a04154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45880/0 (socket says 172.21.15.7:45880) 2023-03-03T13:43:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a027fc700 1 -- 172.21.15.7:0/3693759695 learned_addr learned my addr 172.21.15.7:0/3693759695 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:43:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a02ffd700 1 --2- 172.21.15.7:0/3693759695 >> v2:172.21.15.143:3300/0 conn(0x7f8a040caa30 0x7f8a040c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a027fc700 1 -- 172.21.15.7:0/3693759695 >> v2:172.21.15.143:3300/0 conn(0x7f8a040caa30 msgr2=0x7f8a040c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:43:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a027fc700 1 --2- 172.21.15.7:0/3693759695 >> v2:172.21.15.143:3300/0 conn(0x7f8a040caa30 0x7f8a040c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a027fc700 1 -- 172.21.15.7:0/3693759695 >> v1:172.21.15.143:6789/0 conn(0x7f8a040cae00 legacy=0x7f8a040c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:43:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a027fc700 1 -- 172.21.15.7:0/3693759695 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a040bfa30 con 0x7f8a040c00e0 2023-03-03T13:43:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a02ffd700 1 --2- 172.21.15.7:0/3693759695 >> v2:172.21.15.143:3300/0 conn(0x7f8a040caa30 0x7f8a040c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:43:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.423+0000 7f8a027fc700 1 --2- 172.21.15.7:0/3693759695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040c00e0 0x7f8a04154ff0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f8a040c81b0 tx=0x7f89f8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfeb50dbe468c5a5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a01ffb700 1 -- 172.21.15.7:0/3693759695 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f8005e30 con 0x7f8a040c00e0 2023-03-03T13:43:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a01ffb700 1 -- 172.21.15.7:0/3693759695 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f8006030 con 0x7f8a040c00e0 2023-03-03T13:43:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a01ffb700 1 -- 172.21.15.7:0/3693759695 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f80063e0 con 0x7f8a040c00e0 2023-03-03T13:43:50.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a09720700 1 -- 172.21.15.7:0/3693759695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040c00e0 msgr2=0x7f8a04154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a09720700 1 --2- 172.21.15.7:0/3693759695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040c00e0 0x7f8a04154ff0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f8a040c81b0 tx=0x7f89f8009bd0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a09720700 1 -- 172.21.15.7:0/3693759695 shutdown_connections 2023-03-03T13:43:50.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a09720700 1 --2- 172.21.15.7:0/3693759695 >> v2:172.21.15.143:3300/0 conn(0x7f8a040caa30 0x7f8a040c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a09720700 1 --2- 172.21.15.7:0/3693759695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040c00e0 0x7f8a04154ff0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a09720700 1 -- 172.21.15.7:0/3693759695 >> 172.21.15.7:0/3693759695 conn(0x7f8a04066e40 msgr2=0x7f8a04071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:43:50.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a09720700 1 -- 172.21.15.7:0/3693759695 shutdown_connections 2023-03-03T13:43:50.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.424+0000 7f8a09720700 1 -- 172.21.15.7:0/3693759695 wait complete. 2023-03-03T13:43:50.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.425+0000 7f8a09720700 1 Processor -- start 2023-03-03T13:43:50.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.426+0000 7f8a09720700 1 -- start start 2023-03-03T13:43:50.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.426+0000 7f8a09720700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a040c00e0 0x7f8a041ddcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.426+0000 7f8a09720700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a040caa30 0x7f8a040c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.426+0000 7f8a09720700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040cae00 0x7f8a040c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.426+0000 7f8a09720700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a041ee360 con 0x7f8a040cae00 2023-03-03T13:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.426+0000 7f8a09720700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a040c1db0 con 0x7f8a040c00e0 2023-03-03T13:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.426+0000 7f8a09720700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a041ed490 con 0x7f8a040caa30 2023-03-03T13:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.426+0000 7f8a037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040cae00 0x7f8a040c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.426+0000 7f8a037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040cae00 0x7f8a040c3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45888/0 (socket says 172.21.15.7:45888) 2023-03-03T13:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.426+0000 7f8a037fe700 1 -- 172.21.15.7:0/4001740005 learned_addr learned my addr 172.21.15.7:0/4001740005 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a02ffd700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a040c00e0 0x7f8a041ddcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a027fc700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a040caa30 0x7f8a040c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a037fe700 1 -- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a040caa30 msgr2=0x7f8a040c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a037fe700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a040caa30 0x7f8a040c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a037fe700 1 -- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a040c00e0 msgr2=0x7f8a041ddcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a037fe700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a040c00e0 0x7f8a041ddcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a037fe700 1 -- 172.21.15.7:0/4001740005 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f8009500 con 0x7f8a040cae00 2023-03-03T13:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a02ffd700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a040c00e0 0x7f8a041ddcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a037fe700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040cae00 0x7f8a040c3c80 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00b970 tx=0x7f89fc00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:50.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f89eb7fe700 1 -- 172.21.15.7:0/4001740005 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc00c9b0 con 0x7f8a040cae00 2023-03-03T13:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f89eb7fe700 1 -- 172.21.15.7:0/4001740005 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc00cb10 con 0x7f8a040cae00 2023-03-03T13:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a09720700 1 -- 172.21.15.7:0/4001740005 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a041547f0 con 0x7f8a040cae00 2023-03-03T13:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f89eb7fe700 1 -- 172.21.15.7:0/4001740005 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc014440 con 0x7f8a040cae00 2023-03-03T13:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.427+0000 7f8a09720700 1 -- 172.21.15.7:0/4001740005 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a04154510 con 0x7f8a040cae00 2023-03-03T13:43:50.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.428+0000 7f8a09720700 1 -- 172.21.15.7:0/4001740005 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a04154ad0 con 0x7f8a040cae00 2023-03-03T13:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.428+0000 7f89eb7fe700 1 -- 172.21.15.7:0/4001740005 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f89fc01b030 con 0x7f8a040cae00 2023-03-03T13:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.429+0000 7f89eb7fe700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89f00682f0 0x7f89f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.429+0000 7f89eb7fe700 1 -- 172.21.15.7:0/4001740005 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f89fc085ca0 con 0x7f8a040cae00 2023-03-03T13:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.429+0000 7f8a02ffd700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89f00682f0 0x7f89f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:43:50.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.430+0000 7f8a02ffd700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89f00682f0 0x7f89f006a7a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f89f4006fd0 tx=0x7f89f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:43:50.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.433+0000 7f89eb7fe700 1 -- 172.21.15.7:0/4001740005 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f89fc08a070 con 0x7f8a040cae00 2023-03-03T13:43:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.669+0000 7f8a09720700 1 -- 172.21.15.7:0/4001740005 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a04074bb0 con 0x7f8a040cae00 2023-03-03T13:43:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.670+0000 7f89eb7fe700 1 -- 172.21.15.7:0/4001740005 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f89fc051fe0 con 0x7f8a040cae00 2023-03-03T13:43:50.672 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:43:50.672 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:43:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 -- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89f00682f0 msgr2=0x7f89f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89f00682f0 0x7f89f006a7a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f89f4006fd0 tx=0x7f89f4008040 comp rx=0 tx=0).stop 2023-03-03T13:43:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 -- 172.21.15.7:0/4001740005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040cae00 msgr2=0x7f8a040c3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:43:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040cae00 0x7f8a040c3c80 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00b970 tx=0x7f89fc00bd30 comp rx=0 tx=0).stop 2023-03-03T13:43:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 -- 172.21.15.7:0/4001740005 shutdown_connections 2023-03-03T13:43:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89f00682f0 0x7f89f006a7a0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a040cae00 0x7f8a040c3c80 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a040c00e0 0x7f8a041ddcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 --2- 172.21.15.7:0/4001740005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a040caa30 0x7f8a040c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:43:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 -- 172.21.15.7:0/4001740005 >> 172.21.15.7:0/4001740005 conn(0x7f8a04066e40 msgr2=0x7f8a0406b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:43:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 -- 172.21.15.7:0/4001740005 shutdown_connections 2023-03-03T13:43:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:43:50.672+0000 7f8a09720700 1 -- 172.21.15.7:0/4001740005 wait complete. 2023-03-03T13:43:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:43:50.686 INFO:tasks.mds_thrash.fs.[b]:waiting for 101 secs before thrashing 2023-03-03T13:44:12.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:44:12.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.150+0000 7f281f389700 1 Processor -- start 2023-03-03T13:44:12.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.150+0000 7f281f389700 1 -- start start 2023-03-03T13:44:12.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281f389700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f281814e560 0x7f2818071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:44:12.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281f389700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28180c43d0 0x7f28180c4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:44:12.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281f389700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2818072920 con 0x7f2818154590 2023-03-03T13:44:12.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281f389700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281806bef0 con 0x7f28180c43d0 2023-03-03T13:44:12.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281f389700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281806b910 con 0x7f281814e560 2023-03-03T13:44:12.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281c924700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28180c43d0 0x7f28180c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:44:12.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281c924700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28180c43d0 0x7f28180c4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45920/0 (socket says 172.21.15.7:45920) 2023-03-03T13:44:12.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281c924700 1 -- 172.21.15.7:0/1835962357 learned_addr learned my addr 172.21.15.7:0/1835962357 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:44:12.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281d125700 1 --2- 172.21.15.7:0/1835962357 >> v2:172.21.15.143:3300/0 conn(0x7f281814e560 0x7f2818071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:44:12.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281c924700 1 -- 172.21.15.7:0/1835962357 >> v2:172.21.15.143:3300/0 conn(0x7f281814e560 msgr2=0x7f2818071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:44:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281c924700 1 --2- 172.21.15.7:0/1835962357 >> v2:172.21.15.143:3300/0 conn(0x7f281814e560 0x7f2818071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:44:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281c924700 1 -- 172.21.15.7:0/1835962357 >> v1:172.21.15.143:6789/0 conn(0x7f2818154590 legacy=0x7f28181549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:44:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281c924700 1 -- 172.21.15.7:0/1835962357 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28180aacc0 con 0x7f28180c43d0 2023-03-03T13:44:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281d125700 1 --2- 172.21.15.7:0/1835962357 >> v2:172.21.15.143:3300/0 conn(0x7f281814e560 0x7f2818071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:44:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.151+0000 7f281c924700 1 --2- 172.21.15.7:0/1835962357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 0x7f28180c4850 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f28180c5e80 tx=0x7f2808009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ffbe29c63ff080 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:44:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f280ffff700 1 -- 172.21.15.7:0/1835962357 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808004230 con 0x7f28180c43d0 2023-03-03T13:44:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f280ffff700 1 -- 172.21.15.7:0/1835962357 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2808004390 con 0x7f28180c43d0 2023-03-03T13:44:12.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f280ffff700 1 -- 172.21.15.7:0/1835962357 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28080036a0 con 0x7f28180c43d0 2023-03-03T13:44:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f281f389700 1 -- 172.21.15.7:0/1835962357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 msgr2=0x7f28180c4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:44:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f281f389700 1 --2- 172.21.15.7:0/1835962357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 0x7f28180c4850 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f28180c5e80 tx=0x7f2808009bd0 comp rx=0 tx=0).stop 2023-03-03T13:44:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f281f389700 1 -- 172.21.15.7:0/1835962357 shutdown_connections 2023-03-03T13:44:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f281f389700 1 --2- 172.21.15.7:0/1835962357 >> v2:172.21.15.143:3300/0 conn(0x7f281814e560 0x7f2818071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:44:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f281f389700 1 --2- 172.21.15.7:0/1835962357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 0x7f28180c4850 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:44:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f281f389700 1 -- 172.21.15.7:0/1835962357 >> 172.21.15.7:0/1835962357 conn(0x7f28180736f0 msgr2=0x7f2818073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:44:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f281f389700 1 -- 172.21.15.7:0/1835962357 shutdown_connections 2023-03-03T13:44:12.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.152+0000 7f281f389700 1 -- 172.21.15.7:0/1835962357 wait complete. 2023-03-03T13:44:12.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.153+0000 7f281f389700 1 Processor -- start 2023-03-03T13:44:12.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.153+0000 7f281f389700 1 -- start start 2023-03-03T13:44:12.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281f389700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 0x7f2818146ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:44:12.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281f389700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f281814e560 0x7f281814d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:44:12.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281f389700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2818154590 0x7f2818157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:44:12.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281f389700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281814c7a0 con 0x7f28180c43d0 2023-03-03T13:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281f389700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28180c7370 con 0x7f2818154590 2023-03-03T13:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281f389700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281814ca60 con 0x7f281814e560 2023-03-03T13:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281d125700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 0x7f2818146ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281d125700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 0x7f2818146ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45924/0 (socket says 172.21.15.7:45924) 2023-03-03T13:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281d125700 1 -- 172.21.15.7:0/3725929998 learned_addr learned my addr 172.21.15.7:0/3725929998 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281c924700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f281814e560 0x7f281814d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281d125700 1 -- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f281814e560 msgr2=0x7f281814d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:44:12.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281d125700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f281814e560 0x7f281814d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.154+0000 7f281d926700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2818154590 0x7f2818157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.155+0000 7f281d125700 1 -- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2818154590 msgr2=0x7f2818157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.155+0000 7f281d125700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2818154590 0x7f2818157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.155+0000 7f281d125700 1 -- 172.21.15.7:0/3725929998 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2808009550 con 0x7f28180c43d0 2023-03-03T13:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.155+0000 7f281d926700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2818154590 0x7f2818157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.155+0000 7f281d125700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 0x7f2818146ec0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f281400c750 tx=0x7f281400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:44:12.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.155+0000 7f280dffb700 1 -- 172.21.15.7:0/3725929998 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2814011800 con 0x7f28180c43d0 2023-03-03T13:44:12.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.155+0000 7f280dffb700 1 -- 172.21.15.7:0/3725929998 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2814011960 con 0x7f28180c43d0 2023-03-03T13:44:12.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.155+0000 7f281f389700 1 -- 172.21.15.7:0/3725929998 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28180aa970 con 0x7f28180c43d0 2023-03-03T13:44:12.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.155+0000 7f280dffb700 1 -- 172.21.15.7:0/3725929998 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2814011cd0 con 0x7f28180c43d0 2023-03-03T13:44:12.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.155+0000 7f281f389700 1 -- 172.21.15.7:0/3725929998 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281806bb80 con 0x7f28180c43d0 2023-03-03T13:44:12.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.156+0000 7f281f389700 1 -- 172.21.15.7:0/3725929998 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28180aafa0 con 0x7f28180c43d0 2023-03-03T13:44:12.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.156+0000 7f280dffb700 1 -- 172.21.15.7:0/3725929998 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2814020070 con 0x7f28180c43d0 2023-03-03T13:44:12.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.157+0000 7f280dffb700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27fc0682f0 0x7f27fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:44:12.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.157+0000 7f280dffb700 1 -- 172.21.15.7:0/3725929998 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2814085ff0 con 0x7f28180c43d0 2023-03-03T13:44:12.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.157+0000 7f281c924700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27fc0682f0 0x7f27fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:44:12.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.158+0000 7f281c924700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27fc0682f0 0x7f27fc06a7a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f281806b880 tx=0x7f2808008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:44:12.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.161+0000 7f280dffb700 1 -- 172.21.15.7:0/3725929998 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2814052390 con 0x7f28180c43d0 2023-03-03T13:44:12.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.399+0000 7f281f389700 1 -- 172.21.15.7:0/3725929998 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28180bd6b0 con 0x7f28180c43d0 2023-03-03T13:44:12.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.399+0000 7f280dffb700 1 -- 172.21.15.7:0/3725929998 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f281401e020 con 0x7f28180c43d0 2023-03-03T13:44:12.401 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:44:12.402 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:44:12.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 -- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27fc0682f0 msgr2=0x7f27fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:44:12.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27fc0682f0 0x7f27fc06a7a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f281806b880 tx=0x7f2808008030 comp rx=0 tx=0).stop 2023-03-03T13:44:12.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 -- 172.21.15.7:0/3725929998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 msgr2=0x7f2818146ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:44:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 0x7f2818146ec0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f281400c750 tx=0x7f281400cb10 comp rx=0 tx=0).stop 2023-03-03T13:44:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 -- 172.21.15.7:0/3725929998 shutdown_connections 2023-03-03T13:44:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2818154590 0x7f2818157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:44:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27fc0682f0 0x7f27fc06a7a0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:44:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28180c43d0 0x7f2818146ec0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:44:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 --2- 172.21.15.7:0/3725929998 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f281814e560 0x7f281814d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:44:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 -- 172.21.15.7:0/3725929998 >> 172.21.15.7:0/3725929998 conn(0x7f28180736f0 msgr2=0x7f2818073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:44:12.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.401+0000 7f281f389700 1 -- 172.21.15.7:0/3725929998 shutdown_connections 2023-03-03T13:44:12.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:44:12.402+0000 7f281f389700 1 -- 172.21.15.7:0/3725929998 wait complete. 2023-03-03T13:44:12.405 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:44:12.415 INFO:tasks.mds_thrash.fs.[a]:waiting for 107 secs before thrashing 2023-03-03T13:44:15.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:15.751 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:45.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:44:45.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:15.990 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:15.993 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:31.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:45:31.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.823+0000 7f7ad1f15700 1 Processor -- start 2023-03-03T13:45:31.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.823+0000 7f7ad1f15700 1 -- start start 2023-03-03T13:45:31.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.824+0000 7f7ad1f15700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7acc14e560 0x7f7acc071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:31.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.824+0000 7f7ad1f15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc0c43d0 0x7f7acc0c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:31.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.824+0000 7f7ad1f15700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc072920 con 0x7f7acc154590 2023-03-03T13:45:31.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.824+0000 7f7ad1f15700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7acc06bef0 con 0x7f7acc14e560 2023-03-03T13:45:31.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.824+0000 7f7ad1f15700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc06b910 con 0x7f7acc0c43d0 2023-03-03T13:45:31.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acaffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc0c43d0 0x7f7acc0c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:31.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acb7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7acc14e560 0x7f7acc071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:31.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acaffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc0c43d0 0x7f7acc0c4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:54046/0 (socket says 172.21.15.7:54046) 2023-03-03T13:45:31.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acb7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7acc14e560 0x7f7acc071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:41748/0 (socket says 172.21.15.7:41748) 2023-03-03T13:45:31.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acaffd700 1 -- 172.21.15.7:0/1739864805 learned_addr learned my addr 172.21.15.7:0/1739864805 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:45:31.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acb7fe700 1 -- 172.21.15.7:0/1739864805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc0c43d0 msgr2=0x7f7acc0c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:45:31.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acb7fe700 1 --2- 172.21.15.7:0/1739864805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc0c43d0 0x7f7acc0c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:31.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acb7fe700 1 -- 172.21.15.7:0/1739864805 >> v1:172.21.15.143:6789/0 conn(0x7f7acc154590 legacy=0x7f7acc1549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:31.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acb7fe700 1 -- 172.21.15.7:0/1739864805 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc0aacc0 con 0x7f7acc14e560 2023-03-03T13:45:31.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acaffd700 1 --2- 172.21.15.7:0/1739864805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc0c43d0 0x7f7acc0c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:45:31.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7acb7fe700 1 --2- 172.21.15.7:0/1739864805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7acc14e560 0x7f7acc071c70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009840 tx=0x7f7abc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a90e9a80d74b253e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7aca7fc700 1 -- 172.21.15.7:0/1739864805 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc011830 con 0x7f7acc14e560 2023-03-03T13:45:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7aca7fc700 1 -- 172.21.15.7:0/1739864805 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc011990 con 0x7f7acc14e560 2023-03-03T13:45:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.825+0000 7f7aca7fc700 1 -- 172.21.15.7:0/1739864805 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc011d20 con 0x7f7acc14e560 2023-03-03T13:45:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.826+0000 7f7ad1f15700 1 -- 172.21.15.7:0/1739864805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7acc14e560 msgr2=0x7f7acc071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.826+0000 7f7ad1f15700 1 --2- 172.21.15.7:0/1739864805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7acc14e560 0x7f7acc071c70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009840 tx=0x7f7abc009b50 comp rx=0 tx=0).stop 2023-03-03T13:45:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.826+0000 7f7ad1f15700 1 -- 172.21.15.7:0/1739864805 shutdown_connections 2023-03-03T13:45:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.826+0000 7f7ad1f15700 1 --2- 172.21.15.7:0/1739864805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7acc14e560 0x7f7acc071c70 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:31.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.826+0000 7f7ad1f15700 1 --2- 172.21.15.7:0/1739864805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc0c43d0 0x7f7acc0c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.826+0000 7f7ad1f15700 1 -- 172.21.15.7:0/1739864805 >> 172.21.15.7:0/1739864805 conn(0x7f7acc0736f0 msgr2=0x7f7acc073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:45:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.826+0000 7f7ad1f15700 1 -- 172.21.15.7:0/1739864805 shutdown_connections 2023-03-03T13:45:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.826+0000 7f7ad1f15700 1 -- 172.21.15.7:0/1739864805 wait complete. 2023-03-03T13:45:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.827+0000 7f7ad1f15700 1 Processor -- start 2023-03-03T13:45:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.827+0000 7f7ad1f15700 1 -- start start 2023-03-03T13:45:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7ad1f15700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7acc0c43d0 0x7f7acc0adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7ad1f15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc14e560 0x7f7acc0c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7ad1f15700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7acc154590 0x7f7acc148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:31.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7ad1f15700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0ace00 con 0x7f7acc154590 2023-03-03T13:45:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7ad1f15700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0ab8e0 con 0x7f7acc0c43d0 2023-03-03T13:45:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7ad1f15700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc1584e0 con 0x7f7acc14e560 2023-03-03T13:45:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7acbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7acc154590 0x7f7acc148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7acbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7acc154590 0x7f7acc148670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45982/0 (socket says 172.21.15.7:45982) 2023-03-03T13:45:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7acbfff700 1 -- 172.21.15.7:0/4250619941 learned_addr learned my addr 172.21.15.7:0/4250619941 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:45:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7acb7fe700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7acc0c43d0 0x7f7acc0adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.828+0000 7f7acaffd700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc14e560 0x7f7acc0c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:31.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7acbfff700 1 -- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc14e560 msgr2=0x7f7acc0c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:31.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7acbfff700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc14e560 0x7f7acc0c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:31.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7acbfff700 1 -- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7acc0c43d0 msgr2=0x7f7acc0adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:31.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7acbfff700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7acc0c43d0 0x7f7acc0adee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:31.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7acbfff700 1 -- 172.21.15.7:0/4250619941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac0009550 con 0x7f7acc154590 2023-03-03T13:45:31.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7acbfff700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7acc154590 0x7f7acc148670 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400cc50 tx=0x7f7ac400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:31.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7ab3fff700 1 -- 172.21.15.7:0/4250619941 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4003c00 con 0x7f7acc154590 2023-03-03T13:45:31.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7ab3fff700 1 -- 172.21.15.7:0/4250619941 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac4003d60 con 0x7f7acc154590 2023-03-03T13:45:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7ab3fff700 1 -- 172.21.15.7:0/4250619941 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac400aab0 con 0x7f7acc154590 2023-03-03T13:45:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7ad1f15700 1 -- 172.21.15.7:0/4250619941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7abc0094a0 con 0x7f7acc154590 2023-03-03T13:45:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.829+0000 7f7ad1f15700 1 -- 172.21.15.7:0/4250619941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc155100 con 0x7f7acc154590 2023-03-03T13:45:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.830+0000 7f7ad1f15700 1 -- 172.21.15.7:0/4250619941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc06bb80 con 0x7f7acc154590 2023-03-03T13:45:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.830+0000 7f7ab3fff700 1 -- 172.21.15.7:0/4250619941 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7ac4010050 con 0x7f7acc154590 2023-03-03T13:45:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.831+0000 7f7ab3fff700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ab80682f0 0x7f7ab806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.831+0000 7f7ab3fff700 1 -- 172.21.15.7:0/4250619941 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7ac4085db0 con 0x7f7acc154590 2023-03-03T13:45:31.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.831+0000 7f7acb7fe700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ab80682f0 0x7f7ab806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.832+0000 7f7acb7fe700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ab80682f0 0x7f7ab806a7a0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7abc000c00 tx=0x7f7abc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:31.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:31.835+0000 7f7ab3fff700 1 -- 172.21.15.7:0/4250619941 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7ac4052150 con 0x7f7acc154590 2023-03-03T13:45:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.075+0000 7f7ad1f15700 1 -- 172.21.15.7:0/4250619941 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7acc074bb0 con 0x7f7acc154590 2023-03-03T13:45:32.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.076+0000 7f7ab3fff700 1 -- 172.21.15.7:0/4250619941 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7ac4055770 con 0x7f7acc154590 2023-03-03T13:45:32.079 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:45:32.079 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:45:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 -- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ab80682f0 msgr2=0x7f7ab806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ab80682f0 0x7f7ab806a7a0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7abc000c00 tx=0x7f7abc008030 comp rx=0 tx=0).stop 2023-03-03T13:45:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 -- 172.21.15.7:0/4250619941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7acc154590 msgr2=0x7f7acc148670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7acc154590 0x7f7acc148670 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400cc50 tx=0x7f7ac400cf60 comp rx=0 tx=0).stop 2023-03-03T13:45:32.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 -- 172.21.15.7:0/4250619941 shutdown_connections 2023-03-03T13:45:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ab80682f0 0x7f7ab806a7a0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7acc154590 0x7f7acc148670 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7acc0c43d0 0x7f7acc0adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 --2- 172.21.15.7:0/4250619941 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7acc14e560 0x7f7acc0c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 -- 172.21.15.7:0/4250619941 >> 172.21.15.7:0/4250619941 conn(0x7f7acc0736f0 msgr2=0x7f7acc153e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:45:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 -- 172.21.15.7:0/4250619941 shutdown_connections 2023-03-03T13:45:32.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:32.078+0000 7f7ad1f15700 1 -- 172.21.15.7:0/4250619941 wait complete. 2023-03-03T13:45:32.083 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:45:32.093 INFO:tasks.mds_thrash.fs.[b]:waiting for 112 secs before thrashing 2023-03-03T13:45:46.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:46.131 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:45:59.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:45:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.553+0000 7fc580603700 1 Processor -- start 2023-03-03T13:45:59.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.553+0000 7fc580603700 1 -- start start 2023-03-03T13:45:59.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc580603700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc57814b0d0 0x7fc578071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc580603700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 0x7fc578158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc580603700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5780a9c20 con 0x7fc5780aad20 2023-03-03T13:45:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc580603700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc578154f10 con 0x7fc57814b0d0 2023-03-03T13:45:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc580603700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5780ac3c0 con 0x7fc578071d40 2023-03-03T13:45:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc57db9e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc5780aad20 0x7fc578157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39982/0 (socket says 172.21.15.7:39982) 2023-03-03T13:45:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc57db9e700 1 -- 172.21.15.7:0/3562646655 learned_addr learned my addr 172.21.15.7:0/3562646655 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:45:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc57e39f700 1 --2- 172.21.15.7:0/3562646655 >> v2:172.21.15.143:3300/0 conn(0x7fc57814b0d0 0x7fc578071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc57eba0700 1 --2- 172.21.15.7:0/3562646655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 0x7fc578158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc57d39d700 1 -- 172.21.15.7:0/3562646655 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452895694 0 0) 0x7fc5780a9c20 con 0x7fc5780aad20 2023-03-03T13:45:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc57d39d700 1 -- 172.21.15.7:0/3562646655 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5640035b0 con 0x7fc5780aad20 2023-03-03T13:45:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc57d39d700 1 -- 172.21.15.7:0/3562646655 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 38718527 0 0) 0x7fc5640035b0 con 0x7fc5780aad20 2023-03-03T13:45:59.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc57eba0700 1 -- 172.21.15.7:0/3562646655 >> v2:172.21.15.143:3300/0 conn(0x7fc57814b0d0 msgr2=0x7fc578071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:45:59.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc57eba0700 1 --2- 172.21.15.7:0/3562646655 >> v2:172.21.15.143:3300/0 conn(0x7fc57814b0d0 0x7fc578071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:59.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.554+0000 7fc57eba0700 1 -- 172.21.15.7:0/3562646655 >> v1:172.21.15.7:6789/0 conn(0x7fc5780aad20 legacy=0x7fc578157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:59.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.555+0000 7fc57eba0700 1 -- 172.21.15.7:0/3562646655 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57806fc70 con 0x7fc578071d40 2023-03-03T13:45:59.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.555+0000 7fc57e39f700 1 --2- 172.21.15.7:0/3562646655 >> v2:172.21.15.143:3300/0 conn(0x7fc57814b0d0 0x7fc578071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:45:59.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.555+0000 7fc57eba0700 1 --2- 172.21.15.7:0/3562646655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 0x7fc578158500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc570009d90 tx=0x7fc57000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa0c2a39e1229e62 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:59.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.555+0000 7fc57d39d700 1 -- 172.21.15.7:0/3562646655 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57000e070 con 0x7fc578071d40 2023-03-03T13:45:59.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.555+0000 7fc57d39d700 1 -- 172.21.15.7:0/3562646655 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc570003a20 con 0x7fc578071d40 2023-03-03T13:45:59.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.555+0000 7fc57d39d700 1 -- 172.21.15.7:0/3562646655 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568001670 con 0x7fc578071d40 2023-03-03T13:45:59.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.555+0000 7fc580603700 1 -- 172.21.15.7:0/3562646655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 msgr2=0x7fc578158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:59.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.555+0000 7fc580603700 1 --2- 172.21.15.7:0/3562646655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 0x7fc578158500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc570009d90 tx=0x7fc57000b330 comp rx=0 tx=0).stop 2023-03-03T13:45:59.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.556+0000 7fc580603700 1 -- 172.21.15.7:0/3562646655 shutdown_connections 2023-03-03T13:45:59.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.556+0000 7fc580603700 1 --2- 172.21.15.7:0/3562646655 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 0x7fc578158500 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:59.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.556+0000 7fc580603700 1 --2- 172.21.15.7:0/3562646655 >> v2:172.21.15.143:3300/0 conn(0x7fc57814b0d0 0x7fc578071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:59.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.556+0000 7fc580603700 1 -- 172.21.15.7:0/3562646655 >> 172.21.15.7:0/3562646655 conn(0x7fc5780c8900 msgr2=0x7fc5780c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:45:59.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.556+0000 7fc580603700 1 -- 172.21.15.7:0/3562646655 shutdown_connections 2023-03-03T13:45:59.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.556+0000 7fc580603700 1 -- 172.21.15.7:0/3562646655 wait complete. 2023-03-03T13:45:59.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.557+0000 7fc580603700 1 Processor -- start 2023-03-03T13:45:59.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.557+0000 7fc580603700 1 -- start start 2023-03-03T13:45:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc580603700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 0x7fc5781ea160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc580603700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5780aad20 0x7fc5781ed660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc580603700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc57814b0d0 0x7fc5780c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc580603700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc578157150 con 0x7fc5780aad20 2023-03-03T13:45:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc580603700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc578156940 con 0x7fc57814b0d0 2023-03-03T13:45:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc580603700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5781ea6a0 con 0x7fc578071d40 2023-03-03T13:45:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57db9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5780aad20 0x7fc5781ed660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:59.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57db9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5780aad20 0x7fc5781ed660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46008/0 (socket says 172.21.15.7:46008) 2023-03-03T13:45:59.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57db9e700 1 -- 172.21.15.7:0/2724183874 learned_addr learned my addr 172.21.15.7:0/2724183874 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:45:59.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57e39f700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 0x7fc5781ea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:59.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57eba0700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc57814b0d0 0x7fc5780c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:59.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57db9e700 1 -- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 msgr2=0x7fc5781ea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:59.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57db9e700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 0x7fc5781ea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:59.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57db9e700 1 -- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc57814b0d0 msgr2=0x7fc5780c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:59.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57db9e700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc57814b0d0 0x7fc5780c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57db9e700 1 -- 172.21.15.7:0/2724183874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5740094a0 con 0x7fc5780aad20 2023-03-03T13:45:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc57db9e700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5780aad20 0x7fc5781ed660 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fc56800c780 tx=0x7fc56800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc56effd700 1 -- 172.21.15.7:0/2724183874 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568004c80 con 0x7fc5780aad20 2023-03-03T13:45:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.558+0000 7fc56effd700 1 -- 172.21.15.7:0/2724183874 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc568004de0 con 0x7fc5780aad20 2023-03-03T13:45:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.559+0000 7fc580603700 1 -- 172.21.15.7:0/2724183874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc570009a70 con 0x7fc5780aad20 2023-03-03T13:45:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.559+0000 7fc56effd700 1 -- 172.21.15.7:0/2724183874 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5680072d0 con 0x7fc5780aad20 2023-03-03T13:45:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.559+0000 7fc580603700 1 -- 172.21.15.7:0/2724183874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57806f6b0 con 0x7fc5780aad20 2023-03-03T13:45:59.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.560+0000 7fc580603700 1 -- 172.21.15.7:0/2724183874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc578155160 con 0x7fc5780aad20 2023-03-03T13:45:59.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.560+0000 7fc56effd700 1 -- 172.21.15.7:0/2724183874 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc56801a020 con 0x7fc5780aad20 2023-03-03T13:45:59.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.561+0000 7fc56effd700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc55c0682f0 0x7fc55c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:45:59.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.561+0000 7fc56effd700 1 -- 172.21.15.7:0/2724183874 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc568085d50 con 0x7fc5780aad20 2023-03-03T13:45:59.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.561+0000 7fc57e39f700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc55c0682f0 0x7fc55c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:45:59.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.561+0000 7fc57e39f700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc55c0682f0 0x7fc55c06a7a0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fc574009cd0 tx=0x7fc574009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:45:59.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.565+0000 7fc56effd700 1 -- 172.21.15.7:0/2724183874 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5680521a0 con 0x7fc5780aad20 2023-03-03T13:45:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.806+0000 7fc580603700 1 -- 172.21.15.7:0/2724183874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc57806f990 con 0x7fc5780aad20 2023-03-03T13:45:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.806+0000 7fc56effd700 1 -- 172.21.15.7:0/2724183874 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc5680557c0 con 0x7fc5780aad20 2023-03-03T13:45:59.808 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:45:59.808 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:45:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.808+0000 7fc580603700 1 -- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc55c0682f0 msgr2=0x7fc55c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.808+0000 7fc580603700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc55c0682f0 0x7fc55c06a7a0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fc574009cd0 tx=0x7fc574009210 comp rx=0 tx=0).stop 2023-03-03T13:45:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.808+0000 7fc580603700 1 -- 172.21.15.7:0/2724183874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5780aad20 msgr2=0x7fc5781ed660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:45:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.808+0000 7fc580603700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5780aad20 0x7fc5781ed660 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fc56800c780 tx=0x7fc56800ca90 comp rx=0 tx=0).stop 2023-03-03T13:45:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.808+0000 7fc580603700 1 -- 172.21.15.7:0/2724183874 shutdown_connections 2023-03-03T13:45:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.808+0000 7fc580603700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc57814b0d0 0x7fc5780c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.808+0000 7fc580603700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc578071d40 0x7fc5781ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.808+0000 7fc580603700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc55c0682f0 0x7fc55c06a7a0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.808+0000 7fc580603700 1 --2- 172.21.15.7:0/2724183874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5780aad20 0x7fc5781ed660 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:45:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.808+0000 7fc580603700 1 -- 172.21.15.7:0/2724183874 >> 172.21.15.7:0/2724183874 conn(0x7fc5780c8900 msgr2=0x7fc5780c3730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:45:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.809+0000 7fc580603700 1 -- 172.21.15.7:0/2724183874 shutdown_connections 2023-03-03T13:45:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:45:59.809+0000 7fc580603700 1 -- 172.21.15.7:0/2724183874 wait complete. 2023-03-03T13:45:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:45:59.822 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T13:46:16.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:16.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:41.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:46:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.961+0000 7fdaf9e7a700 1 Processor -- start 2023-03-03T13:46:41.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.961+0000 7fdaf9e7a700 1 -- start start 2023-03-03T13:46:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.962+0000 7fdaf9e7a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdaf41530d0 0x7fdaf4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.962+0000 7fdaf9e7a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf40c2f30 con 0x7fdaf414fda0 2023-03-03T13:46:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.962+0000 7fdaf9e7a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf40c31d0 con 0x7fdaf414f9d0 2023-03-03T13:46:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.962+0000 7fdaf9e7a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf41524a0 con 0x7fdaf41530d0 2023-03-03T13:46:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.962+0000 7fdaf3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdaf41530d0 0x7fdaf4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.962+0000 7fdaeaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdaf414fda0 0x7fdaf41529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40012/0 (socket says 172.21.15.7:40012) 2023-03-03T13:46:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.962+0000 7fdaeaffd700 1 -- 172.21.15.7:0/1093683889 learned_addr learned my addr 172.21.15.7:0/1093683889 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:46:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.963+0000 7fdaf3fff700 1 -- 172.21.15.7:0/1093683889 >> v1:172.21.15.143:6789/0 conn(0x7fdaf414f9d0 legacy=0x7fdaf4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.963+0000 7fdaf3fff700 1 -- 172.21.15.7:0/1093683889 >> v1:172.21.15.7:6789/0 conn(0x7fdaf414fda0 legacy=0x7fdaf41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.963+0000 7fdaf3fff700 1 -- 172.21.15.7:0/1093683889 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf40c34b0 con 0x7fdaf41530d0 2023-03-03T13:46:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.963+0000 7fdaf3fff700 1 --2- 172.21.15.7:0/1093683889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf41530d0 0x7fdaf4156430 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fdaec009d90 tx=0x7fdaec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9595e38470c2f6c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:46:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.965+0000 7fdaf2ffd700 1 -- 172.21.15.7:0/1093683889 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec00e070 con 0x7fdaf41530d0 2023-03-03T13:46:41.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.965+0000 7fdaf2ffd700 1 -- 172.21.15.7:0/1093683889 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec00bbd0 con 0x7fdaf41530d0 2023-03-03T13:46:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.965+0000 7fdaf2ffd700 1 -- 172.21.15.7:0/1093683889 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec012430 con 0x7fdaf41530d0 2023-03-03T13:46:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.965+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1093683889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf41530d0 msgr2=0x7fdaf4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.965+0000 7fdaf9e7a700 1 --2- 172.21.15.7:0/1093683889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf41530d0 0x7fdaf4156430 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fdaec009d90 tx=0x7fdaec00b330 comp rx=0 tx=0).stop 2023-03-03T13:46:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.965+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1093683889 shutdown_connections 2023-03-03T13:46:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.965+0000 7fdaf9e7a700 1 --2- 172.21.15.7:0/1093683889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf41530d0 0x7fdaf4156430 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.965+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1093683889 >> 172.21.15.7:0/1093683889 conn(0x7fdaf40c8900 msgr2=0x7fdaf40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:46:41.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.965+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1093683889 shutdown_connections 2023-03-03T13:46:41.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.965+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1093683889 wait complete. 2023-03-03T13:46:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.967+0000 7fdaf9e7a700 1 Processor -- start 2023-03-03T13:46:41.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.967+0000 7fdaf9e7a700 1 -- start start 2023-03-03T13:46:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.967+0000 7fdaf9e7a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf414f9d0 0x7fdaf41ea030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.967+0000 7fdaf9e7a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf414fda0 0x7fdaf414aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.967+0000 7fdaf9e7a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaf41530d0 0x7fdaf40c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.967+0000 7fdaf9e7a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf414cdd0 con 0x7fdaf414f9d0 2023-03-03T13:46:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.967+0000 7fdaf9e7a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf41552b0 con 0x7fdaf414fda0 2023-03-03T13:46:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.967+0000 7fdaf9e7a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf414c830 con 0x7fdaf41530d0 2023-03-03T13:46:41.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf414f9d0 0x7fdaf41ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf414f9d0 0x7fdaf41ea030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46038/0 (socket says 172.21.15.7:46038) 2023-03-03T13:46:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf37fe700 1 -- 172.21.15.7:0/1179688685 learned_addr learned my addr 172.21.15.7:0/1179688685 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:46:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf3fff700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaf41530d0 0x7fdaf40c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaeaffd700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf414fda0 0x7fdaf414aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf37fe700 1 -- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaf41530d0 msgr2=0x7fdaf40c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf37fe700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaf41530d0 0x7fdaf40c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:41.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf37fe700 1 -- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf414fda0 msgr2=0x7fdaf414aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:41.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf37fe700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf414fda0 0x7fdaf414aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:41.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf37fe700 1 -- 172.21.15.7:0/1179688685 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec009a70 con 0x7fdaf414f9d0 2023-03-03T13:46:41.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf37fe700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf414f9d0 0x7fdaf41ea030 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fdae4007fd0 tx=0x7fdae400c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:46:41.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf0ff9700 1 -- 172.21.15.7:0/1179688685 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae40052d0 con 0x7fdaf414f9d0 2023-03-03T13:46:41.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf0ff9700 1 -- 172.21.15.7:0/1179688685 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae4005430 con 0x7fdaf414f9d0 2023-03-03T13:46:41.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1179688685 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf40ac350 con 0x7fdaf414f9d0 2023-03-03T13:46:41.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf0ff9700 1 -- 172.21.15.7:0/1179688685 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae40038a0 con 0x7fdaf414f9d0 2023-03-03T13:46:41.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.968+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1179688685 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf40ab630 con 0x7fdaf414f9d0 2023-03-03T13:46:41.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.969+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1179688685 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf4071fa0 con 0x7fdaf414f9d0 2023-03-03T13:46:41.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.970+0000 7fdaf0ff9700 1 -- 172.21.15.7:0/1179688685 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdae4019030 con 0x7fdaf414f9d0 2023-03-03T13:46:41.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.970+0000 7fdaf0ff9700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdad40682f0 0x7fdad406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:46:41.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.971+0000 7fdaf0ff9700 1 -- 172.21.15.7:0/1179688685 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdae40854d0 con 0x7fdaf414f9d0 2023-03-03T13:46:41.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.971+0000 7fdaeaffd700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdad40682f0 0x7fdad406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:46:41.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.971+0000 7fdaeaffd700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdad40682f0 0x7fdad406a7a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40ad270 tx=0x7fdadc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:46:41.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:41.975+0000 7fdaf0ff9700 1 -- 172.21.15.7:0/1179688685 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdae4051870 con 0x7fdaf414f9d0 2023-03-03T13:46:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.216+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1179688685 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaf4074bb0 con 0x7fdaf414f9d0 2023-03-03T13:46:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.217+0000 7fdaf0ff9700 1 -- 172.21.15.7:0/1179688685 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdae4054e90 con 0x7fdaf414f9d0 2023-03-03T13:46:42.219 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:46:42.219 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:46:42.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.218+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdad40682f0 msgr2=0x7fdad406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.218+0000 7fdaf9e7a700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdad40682f0 0x7fdad406a7a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fdaf40ad270 tx=0x7fdadc009040 comp rx=0 tx=0).stop 2023-03-03T13:46:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.218+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1179688685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf414f9d0 msgr2=0x7fdaf41ea030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:46:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.218+0000 7fdaf9e7a700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf414f9d0 0x7fdaf41ea030 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fdae4007fd0 tx=0x7fdae400c6e0 comp rx=0 tx=0).stop 2023-03-03T13:46:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.218+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1179688685 shutdown_connections 2023-03-03T13:46:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.219+0000 7fdaf9e7a700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdaf41530d0 0x7fdaf40c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.219+0000 7fdaf9e7a700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdad40682f0 0x7fdad406a7a0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.219+0000 7fdaf9e7a700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdaf414f9d0 0x7fdaf41ea030 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.219+0000 7fdaf9e7a700 1 --2- 172.21.15.7:0/1179688685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf414fda0 0x7fdaf414aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:46:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.219+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1179688685 >> 172.21.15.7:0/1179688685 conn(0x7fdaf40c8900 msgr2=0x7fdaf40c3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:46:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.219+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1179688685 shutdown_connections 2023-03-03T13:46:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:46:42.219+0000 7fdaf9e7a700 1 -- 172.21.15.7:0/1179688685 wait complete. 2023-03-03T13:46:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:46:42.232 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2023-03-03T13:46:46.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:46:46.425 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:16.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:16.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:24.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:47:24.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.231+0000 7fbd5a47d700 1 Processor -- start 2023-03-03T13:47:24.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.231+0000 7fbd5a47d700 1 -- start start 2023-03-03T13:47:24.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.232+0000 7fbd5a47d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd540c1da0 0x7fbd541550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:24.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.232+0000 7fbd5a47d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbd540c3e30 0x7fbd540c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:24.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.232+0000 7fbd5a47d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd540c7a60 con 0x7fbd5406d870 2023-03-03T13:47:24.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.232+0000 7fbd5a47d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd540c2af0 con 0x7fbd540c3e30 2023-03-03T13:47:24.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.232+0000 7fbd5a47d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540c3620 con 0x7fbd540c1da0 2023-03-03T13:47:24.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.232+0000 7fbd58a1a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbd540c3e30 0x7fbd540c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:24.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.232+0000 7fbd53fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbd5406d870 0x7fbd540c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40058/0 (socket says 172.21.15.7:40058) 2023-03-03T13:47:24.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.232+0000 7fbd53fff700 1 -- 172.21.15.7:0/2143923927 learned_addr learned my addr 172.21.15.7:0/2143923927 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:47:24.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.233+0000 7fbd537fe700 1 --2- 172.21.15.7:0/2143923927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd540c1da0 0x7fbd541550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:24.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.233+0000 7fbd58a1a700 1 -- 172.21.15.7:0/2143923927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd540c1da0 msgr2=0x7fbd541550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:47:24.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.233+0000 7fbd58a1a700 1 --2- 172.21.15.7:0/2143923927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd540c1da0 0x7fbd541550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.233+0000 7fbd58a1a700 1 -- 172.21.15.7:0/2143923927 >> v1:172.21.15.7:6789/0 conn(0x7fbd5406d870 legacy=0x7fbd540c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:24.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.233+0000 7fbd58a1a700 1 -- 172.21.15.7:0/2143923927 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd541545e0 con 0x7fbd540c3e30 2023-03-03T13:47:24.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.233+0000 7fbd52ffd700 1 -- 172.21.15.7:0/2143923927 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593364733 0 0) 0x7fbd540c7a60 con 0x7fbd5406d870 2023-03-03T13:47:24.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.233+0000 7fbd58a1a700 1 --2- 172.21.15.7:0/2143923927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 0x7fbd540c4290 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009d90 tx=0x7fbd4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a5e6731ae2c655 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:47:24.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.233+0000 7fbd52ffd700 1 -- 172.21.15.7:0/2143923927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c00e070 con 0x7fbd540c3e30 2023-03-03T13:47:24.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.233+0000 7fbd52ffd700 1 -- 172.21.15.7:0/2143923927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c00bbd0 con 0x7fbd540c3e30 2023-03-03T13:47:24.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.233+0000 7fbd52ffd700 1 -- 172.21.15.7:0/2143923927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c012430 con 0x7fbd540c3e30 2023-03-03T13:47:24.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.234+0000 7fbd5a47d700 1 -- 172.21.15.7:0/2143923927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 msgr2=0x7fbd540c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:24.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.234+0000 7fbd5a47d700 1 --2- 172.21.15.7:0/2143923927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 0x7fbd540c4290 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c009d90 tx=0x7fbd4c00b330 comp rx=0 tx=0).stop 2023-03-03T13:47:24.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.234+0000 7fbd5a47d700 1 -- 172.21.15.7:0/2143923927 shutdown_connections 2023-03-03T13:47:24.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.234+0000 7fbd5a47d700 1 --2- 172.21.15.7:0/2143923927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 0x7fbd540c4290 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.234+0000 7fbd5a47d700 1 --2- 172.21.15.7:0/2143923927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd540c1da0 0x7fbd541550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.234+0000 7fbd5a47d700 1 -- 172.21.15.7:0/2143923927 >> 172.21.15.7:0/2143923927 conn(0x7fbd5406c110 msgr2=0x7fbd5406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:47:24.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.234+0000 7fbd5a47d700 1 -- 172.21.15.7:0/2143923927 shutdown_connections 2023-03-03T13:47:24.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.234+0000 7fbd5a47d700 1 -- 172.21.15.7:0/2143923927 wait complete. 2023-03-03T13:47:24.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.235+0000 7fbd5a47d700 1 Processor -- start 2023-03-03T13:47:24.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.235+0000 7fbd5a47d700 1 -- start start 2023-03-03T13:47:24.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd5a47d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5406d870 0x7fbd541558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:24.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd5a47d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd540c1da0 0x7fbd541ea2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:24.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd5a47d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 0x7fbd540c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:24.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd5a47d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd541ea800 con 0x7fbd540c3e30 2023-03-03T13:47:24.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd5a47d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540ab540 con 0x7fbd540c1da0 2023-03-03T13:47:24.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd5a47d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd540ab820 con 0x7fbd5406d870 2023-03-03T13:47:24.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd58a1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 0x7fbd540c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:24.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd58a1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 0x7fbd540c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46088/0 (socket says 172.21.15.7:46088) 2023-03-03T13:47:24.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd58a1a700 1 -- 172.21.15.7:0/238940318 learned_addr learned my addr 172.21.15.7:0/238940318 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:47:24.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd53fff700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5406d870 0x7fbd541558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:24.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd537fe700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd540c1da0 0x7fbd541ea2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:24.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd58a1a700 1 -- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5406d870 msgr2=0x7fbd541558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:24.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd58a1a700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5406d870 0x7fbd541558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd58a1a700 1 -- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd540c1da0 msgr2=0x7fbd541ea2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:24.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd58a1a700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd540c1da0 0x7fbd541ea2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd58a1a700 1 -- 172.21.15.7:0/238940318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c009a70 con 0x7fbd540c3e30 2023-03-03T13:47:24.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.236+0000 7fbd58a1a700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 0x7fbd540c65f0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fbd540c4a90 tx=0x7fbd4c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:47:24.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.237+0000 7fbd50ff9700 1 -- 172.21.15.7:0/238940318 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c012430 con 0x7fbd540c3e30 2023-03-03T13:47:24.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.237+0000 7fbd50ff9700 1 -- 172.21.15.7:0/238940318 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c020070 con 0x7fbd540c3e30 2023-03-03T13:47:24.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.237+0000 7fbd5a47d700 1 -- 172.21.15.7:0/238940318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd54154ba0 con 0x7fbd540c3e30 2023-03-03T13:47:24.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.237+0000 7fbd50ff9700 1 -- 172.21.15.7:0/238940318 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c004280 con 0x7fbd540c3e30 2023-03-03T13:47:24.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.237+0000 7fbd5a47d700 1 -- 172.21.15.7:0/238940318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd541548c0 con 0x7fbd540c3e30 2023-03-03T13:47:24.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.238+0000 7fbd5a47d700 1 -- 172.21.15.7:0/238940318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5406dc40 con 0x7fbd540c3e30 2023-03-03T13:47:24.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.238+0000 7fbd50ff9700 1 -- 172.21.15.7:0/238940318 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd4c0043e0 con 0x7fbd540c3e30 2023-03-03T13:47:24.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.238+0000 7fbd50ff9700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:24.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.239+0000 7fbd50ff9700 1 -- 172.21.15.7:0/238940318 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbd4c022070 con 0x7fbd540c3e30 2023-03-03T13:47:24.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.239+0000 7fbd53fff700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:24.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.239+0000 7fbd53fff700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fbd44000f60 tx=0x7fbd44007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:47:24.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.243+0000 7fbd50ff9700 1 -- 172.21.15.7:0/238940318 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbd4c054d10 con 0x7fbd540c3e30 2023-03-03T13:47:24.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.479+0000 7fbd5a47d700 1 -- 172.21.15.7:0/238940318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd54069730 con 0x7fbd540c3e30 2023-03-03T13:47:24.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.480+0000 7fbd50ff9700 1 -- 172.21.15.7:0/238940318 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbd4c025380 con 0x7fbd540c3e30 2023-03-03T13:47:24.482 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:47:24.482 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:47:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 -- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 msgr2=0x7fbd4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fbd44000f60 tx=0x7fbd44007be0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 -- 172.21.15.7:0/238940318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 msgr2=0x7fbd540c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 0x7fbd540c65f0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fbd540c4a90 tx=0x7fbd4c004ad0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 -- 172.21.15.7:0/238940318 shutdown_connections 2023-03-03T13:47:24.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd540c3e30 0x7fbd540c65f0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5406d870 0x7fbd541558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 --2- 172.21.15.7:0/238940318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd540c1da0 0x7fbd541ea2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 -- 172.21.15.7:0/238940318 >> 172.21.15.7:0/238940318 conn(0x7fbd5406c110 msgr2=0x7fbd54152340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:47:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 -- 172.21.15.7:0/238940318 shutdown_connections 2023-03-03T13:47:24.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:24.482+0000 7fbd5a47d700 1 -- 172.21.15.7:0/238940318 wait complete. 2023-03-03T13:47:24.486 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:47:24.496 INFO:tasks.mds_thrash.fs.[b]:waiting for 65 secs before thrashing 2023-03-03T13:47:25.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:47:25.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.366+0000 7f59b648c700 1 Processor -- start 2023-03-03T13:47:25.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.366+0000 7f59b648c700 1 -- start start 2023-03-03T13:47:25.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59b648c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0155390 0x7f59b00c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:25.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59b648c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59b006e020 0x7f59b0071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:25.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59b648c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59b0072520 0x7f59b0151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:25.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59b648c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b0070b90 con 0x7f59b006e020 2023-03-03T13:47:25.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59b648c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b0151320 con 0x7f59b0072520 2023-03-03T13:47:25.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59b648c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b00c4780 con 0x7f59b0155390 2023-03-03T13:47:25.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59af7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59b006e020 0x7f59b0071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:25.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59af7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59b006e020 0x7f59b0071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46092/0 (socket says 172.21.15.7:46092) 2023-03-03T13:47:25.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59af7fe700 1 -- 172.21.15.7:0/3746533342 learned_addr learned my addr 172.21.15.7:0/3746533342 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:47:25.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59affff700 1 --2- 172.21.15.7:0/3746533342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0155390 0x7f59b00c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:25.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59b4a29700 1 --2- 172.21.15.7:0/3746533342 >> v2:172.21.15.143:3300/0 conn(0x7f59b0072520 0x7f59b0151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:25.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59af7fe700 1 -- 172.21.15.7:0/3746533342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0155390 msgr2=0x7f59b00c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:47:25.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59af7fe700 1 --2- 172.21.15.7:0/3746533342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0155390 0x7f59b00c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59af7fe700 1 -- 172.21.15.7:0/3746533342 >> v2:172.21.15.143:3300/0 conn(0x7f59b0072520 msgr2=0x7f59b0151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:47:25.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59af7fe700 1 --2- 172.21.15.7:0/3746533342 >> v2:172.21.15.143:3300/0 conn(0x7f59b0072520 0x7f59b0151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59af7fe700 1 -- 172.21.15.7:0/3746533342 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b0154910 con 0x7f59b006e020 2023-03-03T13:47:25.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59af7fe700 1 --2- 172.21.15.7:0/3746533342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b006e020 0x7f59b0071fe0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f59b00c3d80 tx=0x7f59a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7904fcdc8caee2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:47:25.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.367+0000 7f59aeffd700 1 -- 172.21.15.7:0/3746533342 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a400b870 con 0x7f59b006e020 2023-03-03T13:47:25.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59aeffd700 1 -- 172.21.15.7:0/3746533342 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a400b9d0 con 0x7f59b006e020 2023-03-03T13:47:25.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59aeffd700 1 -- 172.21.15.7:0/3746533342 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a400bd60 con 0x7f59b006e020 2023-03-03T13:47:25.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59b648c700 1 -- 172.21.15.7:0/3746533342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b006e020 msgr2=0x7f59b0071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:25.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59b648c700 1 --2- 172.21.15.7:0/3746533342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b006e020 0x7f59b0071fe0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f59b00c3d80 tx=0x7f59a4009bd0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59b648c700 1 -- 172.21.15.7:0/3746533342 shutdown_connections 2023-03-03T13:47:25.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59b648c700 1 --2- 172.21.15.7:0/3746533342 >> v2:172.21.15.143:3300/0 conn(0x7f59b0072520 0x7f59b0151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59b648c700 1 --2- 172.21.15.7:0/3746533342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0155390 0x7f59b00c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59b648c700 1 --2- 172.21.15.7:0/3746533342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b006e020 0x7f59b0071fe0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59b648c700 1 -- 172.21.15.7:0/3746533342 >> 172.21.15.7:0/3746533342 conn(0x7f59b006c110 msgr2=0x7f59b006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:47:25.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59b648c700 1 -- 172.21.15.7:0/3746533342 shutdown_connections 2023-03-03T13:47:25.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.368+0000 7f59b648c700 1 -- 172.21.15.7:0/3746533342 wait complete. 2023-03-03T13:47:25.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.369+0000 7f59b648c700 1 Processor -- start 2023-03-03T13:47:25.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.369+0000 7f59b648c700 1 -- start start 2023-03-03T13:47:25.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59b648c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59b006e020 0x7f59b00abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59b648c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0072520 0x7f59b0146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59b648c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b0155390 0x7f59b01ea0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59b648c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b0151010 con 0x7f59b0155390 2023-03-03T13:47:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59b648c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b0150400 con 0x7f59b006e020 2023-03-03T13:47:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59b648c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b01506e0 con 0x7f59b0072520 2023-03-03T13:47:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59b4a29700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b0155390 0x7f59b01ea0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:25.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59b4a29700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b0155390 0x7f59b01ea0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46102/0 (socket says 172.21.15.7:46102) 2023-03-03T13:47:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59b4a29700 1 -- 172.21.15.7:0/2043335062 learned_addr learned my addr 172.21.15.7:0/2043335062 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:47:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59af7fe700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0072520 0x7f59b0146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.370+0000 7f59affff700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59b006e020 0x7f59b00abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59b4a29700 1 -- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0072520 msgr2=0x7f59b0146e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59b4a29700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0072520 0x7f59b0146e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59b4a29700 1 -- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59b006e020 msgr2=0x7f59b00abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59b4a29700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59b006e020 0x7f59b00abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59b4a29700 1 -- 172.21.15.7:0/2043335062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a4009550 con 0x7f59b0155390 2023-03-03T13:47:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59af7fe700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0072520 0x7f59b0146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:47:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59b4a29700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b0155390 0x7f59b01ea0a0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f59a8007ef0 tx=0x7f59a80105d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:47:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59acff9700 1 -- 172.21.15.7:0/2043335062 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a8013670 con 0x7f59b0155390 2023-03-03T13:47:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59acff9700 1 -- 172.21.15.7:0/2043335062 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a80137f0 con 0x7f59b0155390 2023-03-03T13:47:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59b648c700 1 -- 172.21.15.7:0/2043335062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b0070540 con 0x7f59b0155390 2023-03-03T13:47:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59acff9700 1 -- 172.21.15.7:0/2043335062 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a8013ba0 con 0x7f59b0155390 2023-03-03T13:47:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.371+0000 7f59b648c700 1 -- 172.21.15.7:0/2043335062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b00702a0 con 0x7f59b0155390 2023-03-03T13:47:25.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.372+0000 7f59b648c700 1 -- 172.21.15.7:0/2043335062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b0070820 con 0x7f59b0155390 2023-03-03T13:47:25.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.372+0000 7f59acff9700 1 -- 172.21.15.7:0/2043335062 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f59a8024070 con 0x7f59b0155390 2023-03-03T13:47:25.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.373+0000 7f59acff9700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f599c0682f0 0x7f599c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:47:25.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.373+0000 7f59acff9700 1 -- 172.21.15.7:0/2043335062 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f59a8088f90 con 0x7f59b0155390 2023-03-03T13:47:25.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.373+0000 7f59affff700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f599c0682f0 0x7f599c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:47:25.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.374+0000 7f59affff700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f599c0682f0 0x7f599c06a7a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f59a0006b20 tx=0x7f59a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:47:25.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.377+0000 7f59acff9700 1 -- 172.21.15.7:0/2043335062 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f59a80552b0 con 0x7f59b0155390 2023-03-03T13:47:25.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.614+0000 7f59b648c700 1 -- 172.21.15.7:0/2043335062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59b0074bb0 con 0x7f59b0155390 2023-03-03T13:47:25.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.614+0000 7f59acff9700 1 -- 172.21.15.7:0/2043335062 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f59a80588d0 con 0x7f59b0155390 2023-03-03T13:47:25.616 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:47:25.616 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:47:25.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 -- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f599c0682f0 msgr2=0x7f599c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:25.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f599c0682f0 0x7f599c06a7a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f59a0006b20 tx=0x7f59a0008040 comp rx=0 tx=0).stop 2023-03-03T13:47:25.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 -- 172.21.15.7:0/2043335062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b0155390 msgr2=0x7f59b01ea0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:47:25.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b0155390 0x7f59b01ea0a0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f59a8007ef0 tx=0x7f59a80105d0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 -- 172.21.15.7:0/2043335062 shutdown_connections 2023-03-03T13:47:25.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f599c0682f0 0x7f599c06a7a0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59b0155390 0x7f59b01ea0a0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59b006e020 0x7f59b00abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 --2- 172.21.15.7:0/2043335062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59b0072520 0x7f59b0146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:47:25.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 -- 172.21.15.7:0/2043335062 >> 172.21.15.7:0/2043335062 conn(0x7f59b006c110 msgr2=0x7f59b00c2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:47:25.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 -- 172.21.15.7:0/2043335062 shutdown_connections 2023-03-03T13:47:25.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:47:25.616+0000 7f59b648c700 1 -- 172.21.15.7:0/2043335062 wait complete. 2023-03-03T13:47:25.620 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:47:25.629 INFO:tasks.mds_thrash.fs.[a]:waiting for 50 secs before thrashing 2023-03-03T13:47:46.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:47:46.584 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:15.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:48:15.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.769+0000 7f55a0e29700 1 Processor -- start 2023-03-03T13:48:15.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.769+0000 7f55a0e29700 1 -- start start 2023-03-03T13:48:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f55a0e29700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f559c14f9d0 0x7f559c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f55a0e29700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f559c14fda0 0x7f559c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f55a0e29700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c0c2f30 con 0x7f559c0abe20 2023-03-03T13:48:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f55a0e29700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f559c0c31d0 con 0x7f559c14fda0 2023-03-03T13:48:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f55a0e29700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f559c1524a0 con 0x7f559c14f9d0 2023-03-03T13:48:15.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f5599d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f559c14fda0 0x7f559c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:15.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f5599d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f559c14fda0 0x7f559c071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46132/0 (socket says 172.21.15.7:46132) 2023-03-03T13:48:15.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f5599d9b700 1 -- 172.21.15.7:0/1747760811 learned_addr learned my addr 172.21.15.7:0/1747760811 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:48:15.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f559a59c700 1 --2- 172.21.15.7:0/1747760811 >> v2:172.21.15.143:3300/0 conn(0x7f559c14f9d0 0x7f559c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:15.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f5599d9b700 1 -- 172.21.15.7:0/1747760811 >> v2:172.21.15.143:3300/0 conn(0x7f559c14f9d0 msgr2=0x7f559c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:48:15.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f5599d9b700 1 --2- 172.21.15.7:0/1747760811 >> v2:172.21.15.143:3300/0 conn(0x7f559c14f9d0 0x7f559c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:15.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f5599d9b700 1 -- 172.21.15.7:0/1747760811 >> v1:172.21.15.143:6789/0 conn(0x7f559c0abe20 legacy=0x7f559c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:48:15.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f5599d9b700 1 -- 172.21.15.7:0/1747760811 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c0c34b0 con 0x7f559c14fda0 2023-03-03T13:48:15.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f5599d9b700 1 --2- 172.21.15.7:0/1747760811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c14fda0 0x7f559c071890 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f559c0710a0 tx=0x7f5590009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bac54f0af483eb4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:15.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.770+0000 7f559959a700 1 -- 172.21.15.7:0/1747760811 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5590004230 con 0x7f559c14fda0 2023-03-03T13:48:15.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.771+0000 7f559959a700 1 -- 172.21.15.7:0/1747760811 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5590004390 con 0x7f559c14fda0 2023-03-03T13:48:15.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.771+0000 7f559959a700 1 -- 172.21.15.7:0/1747760811 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55900036a0 con 0x7f559c14fda0 2023-03-03T13:48:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.771+0000 7f55a0e29700 1 -- 172.21.15.7:0/1747760811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c14fda0 msgr2=0x7f559c071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.771+0000 7f55a0e29700 1 --2- 172.21.15.7:0/1747760811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c14fda0 0x7f559c071890 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f559c0710a0 tx=0x7f5590009bd0 comp rx=0 tx=0).stop 2023-03-03T13:48:15.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.771+0000 7f55a0e29700 1 -- 172.21.15.7:0/1747760811 shutdown_connections 2023-03-03T13:48:15.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.771+0000 7f55a0e29700 1 --2- 172.21.15.7:0/1747760811 >> v2:172.21.15.143:3300/0 conn(0x7f559c14f9d0 0x7f559c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:15.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.771+0000 7f55a0e29700 1 --2- 172.21.15.7:0/1747760811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c14fda0 0x7f559c071890 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:15.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.771+0000 7f55a0e29700 1 -- 172.21.15.7:0/1747760811 >> 172.21.15.7:0/1747760811 conn(0x7f559c0c8900 msgr2=0x7f559c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:48:15.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.771+0000 7f55a0e29700 1 -- 172.21.15.7:0/1747760811 shutdown_connections 2023-03-03T13:48:15.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.771+0000 7f55a0e29700 1 -- 172.21.15.7:0/1747760811 wait complete. 2023-03-03T13:48:15.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.772+0000 7f55a0e29700 1 Processor -- start 2023-03-03T13:48:15.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f55a0e29700 1 -- start start 2023-03-03T13:48:15.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f55a0e29700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c0abe20 0x7f559c06a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:15.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f55a0e29700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f559c14f9d0 0x7f559c14dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:15.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f55a0e29700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f559c14fda0 0x7f559c149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:15.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f55a0e29700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c14e800 con 0x7f559c0abe20 2023-03-03T13:48:15.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f55a0e29700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c155630 con 0x7f559c14fda0 2023-03-03T13:48:15.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f55a0e29700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559c155b60 con 0x7f559c14f9d0 2023-03-03T13:48:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f559a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c0abe20 0x7f559c06a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f559a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c0abe20 0x7f559c06a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46136/0 (socket says 172.21.15.7:46136) 2023-03-03T13:48:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f559a59c700 1 -- 172.21.15.7:0/2586207976 learned_addr learned my addr 172.21.15.7:0/2586207976 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:48:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f559ad9d700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f559c14fda0 0x7f559c149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.773+0000 7f5599d9b700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f559c14f9d0 0x7f559c14dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f559a59c700 1 -- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f559c14f9d0 msgr2=0x7f559c14dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f559a59c700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f559c14f9d0 0x7f559c14dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f559a59c700 1 -- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f559c14fda0 msgr2=0x7f559c149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f559a59c700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f559c14fda0 0x7f559c149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f559a59c700 1 -- 172.21.15.7:0/2586207976 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5590009550 con 0x7f559c0abe20 2023-03-03T13:48:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f5599d9b700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f559c14f9d0 0x7f559c14dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:48:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f559a59c700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c0abe20 0x7f559c06a2b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f558c00e810 tx=0x7f558c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f558affd700 1 -- 172.21.15.7:0/2586207976 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c013610 con 0x7f559c0abe20 2023-03-03T13:48:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f558affd700 1 -- 172.21.15.7:0/2586207976 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558c013770 con 0x7f559c0abe20 2023-03-03T13:48:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f55a0e29700 1 -- 172.21.15.7:0/2586207976 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559c0ac350 con 0x7f559c0abe20 2023-03-03T13:48:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f558affd700 1 -- 172.21.15.7:0/2586207976 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c013b20 con 0x7f559c0abe20 2023-03-03T13:48:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.774+0000 7f55a0e29700 1 -- 172.21.15.7:0/2586207976 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559c0ab630 con 0x7f559c0abe20 2023-03-03T13:48:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.775+0000 7f55a0e29700 1 -- 172.21.15.7:0/2586207976 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559c0c3e10 con 0x7f559c0abe20 2023-03-03T13:48:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.776+0000 7f558affd700 1 -- 172.21.15.7:0/2586207976 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f558c023070 con 0x7f559c0abe20 2023-03-03T13:48:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.776+0000 7f558affd700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f557c0682f0 0x7f557c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.776+0000 7f558affd700 1 -- 172.21.15.7:0/2586207976 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f558c085e90 con 0x7f559c0abe20 2023-03-03T13:48:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.776+0000 7f5599d9b700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f557c0682f0 0x7f557c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.777+0000 7f5599d9b700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f557c0682f0 0x7f557c06a7a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f559c1501d0 tx=0x7f5590008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:15.781+0000 7f558affd700 1 -- 172.21.15.7:0/2586207976 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f558c0521b0 con 0x7f559c0abe20 2023-03-03T13:48:16.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.017+0000 7f55a0e29700 1 -- 172.21.15.7:0/2586207976 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f559c0ac4e0 con 0x7f559c0abe20 2023-03-03T13:48:16.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.018+0000 7f558affd700 1 -- 172.21.15.7:0/2586207976 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f558c0557d0 con 0x7f559c0abe20 2023-03-03T13:48:16.020 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:48:16.020 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:48:16.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.019+0000 7f55a0e29700 1 -- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f557c0682f0 msgr2=0x7f557c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:16.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.019+0000 7f55a0e29700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f557c0682f0 0x7f557c06a7a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f559c1501d0 tx=0x7f5590008030 comp rx=0 tx=0).stop 2023-03-03T13:48:16.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.020+0000 7f55a0e29700 1 -- 172.21.15.7:0/2586207976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c0abe20 msgr2=0x7f559c06a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:16.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.020+0000 7f55a0e29700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c0abe20 0x7f559c06a2b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f558c00e810 tx=0x7f558c00eb20 comp rx=0 tx=0).stop 2023-03-03T13:48:16.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.020+0000 7f55a0e29700 1 -- 172.21.15.7:0/2586207976 shutdown_connections 2023-03-03T13:48:16.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.020+0000 7f55a0e29700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f559c14fda0 0x7f559c149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:16.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.020+0000 7f55a0e29700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f557c0682f0 0x7f557c06a7a0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:16.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.020+0000 7f55a0e29700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f559c0abe20 0x7f559c06a2b0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:16.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.020+0000 7f55a0e29700 1 --2- 172.21.15.7:0/2586207976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f559c14f9d0 0x7f559c14dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:16.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.020+0000 7f55a0e29700 1 -- 172.21.15.7:0/2586207976 >> 172.21.15.7:0/2586207976 conn(0x7f559c0c8900 msgr2=0x7f559c0c1870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:48:16.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.020+0000 7f55a0e29700 1 -- 172.21.15.7:0/2586207976 shutdown_connections 2023-03-03T13:48:16.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:16.020+0000 7f55a0e29700 1 -- 172.21.15.7:0/2586207976 wait complete. 2023-03-03T13:48:16.023 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:48:16.033 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2023-03-03T13:48:16.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:16.750 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:29.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:48:29.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.632+0000 7f3cb571c700 1 Processor -- start 2023-03-03T13:48:29.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.632+0000 7f3cb571c700 1 -- start start 2023-03-03T13:48:29.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.633+0000 7f3cb571c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3cb00caa30 0x7f3cb00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:29.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.633+0000 7f3cb571c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3cb0154ff0 0x7f3cb00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:29.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.633+0000 7f3cb571c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb00c7310 con 0x7f3cb00c00e0 2023-03-03T13:48:29.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.633+0000 7f3cb571c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb0155430 con 0x7f3cb0154ff0 2023-03-03T13:48:29.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.633+0000 7f3cb571c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb00bfd10 con 0x7f3cb00caa30 2023-03-03T13:48:29.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.633+0000 7f3caf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3cb0154ff0 0x7f3cb00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:29.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.633+0000 7f3caf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3cb0154ff0 0x7f3cb00cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46150/0 (socket says 172.21.15.7:46150) 2023-03-03T13:48:29.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.633+0000 7f3caf7fe700 1 -- 172.21.15.7:0/3822012105 learned_addr learned my addr 172.21.15.7:0/3822012105 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:48:29.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.634+0000 7f3caeffd700 1 --2- 172.21.15.7:0/3822012105 >> v2:172.21.15.143:3300/0 conn(0x7f3cb00caa30 0x7f3cb00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:29.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.634+0000 7f3caf7fe700 1 -- 172.21.15.7:0/3822012105 >> v2:172.21.15.143:3300/0 conn(0x7f3cb00caa30 msgr2=0x7f3cb00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:48:29.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.634+0000 7f3caf7fe700 1 --2- 172.21.15.7:0/3822012105 >> v2:172.21.15.143:3300/0 conn(0x7f3cb00caa30 0x7f3cb00c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:29.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.634+0000 7f3caf7fe700 1 -- 172.21.15.7:0/3822012105 >> v1:172.21.15.143:6789/0 conn(0x7f3cb00c00e0 legacy=0x7f3cb00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:48:29.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.634+0000 7f3caf7fe700 1 -- 172.21.15.7:0/3822012105 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb00bfa30 con 0x7f3cb0154ff0 2023-03-03T13:48:29.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.634+0000 7f3caf7fe700 1 --2- 172.21.15.7:0/3822012105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb0154ff0 0x7f3cb00cae00 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00c81b0 tx=0x7f3ca800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ec651bee143ee3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.634+0000 7f3cadffb700 1 -- 172.21.15.7:0/3822012105 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca800e070 con 0x7f3cb0154ff0 2023-03-03T13:48:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.634+0000 7f3cadffb700 1 -- 172.21.15.7:0/3822012105 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca8003a20 con 0x7f3cb0154ff0 2023-03-03T13:48:29.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.634+0000 7f3cadffb700 1 -- 172.21.15.7:0/3822012105 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8003dd0 con 0x7f3cb0154ff0 2023-03-03T13:48:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.635+0000 7f3cb571c700 1 -- 172.21.15.7:0/3822012105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb0154ff0 msgr2=0x7f3cb00cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.635+0000 7f3cb571c700 1 --2- 172.21.15.7:0/3822012105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb0154ff0 0x7f3cb00cae00 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00c81b0 tx=0x7f3ca800b330 comp rx=0 tx=0).stop 2023-03-03T13:48:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.635+0000 7f3cb571c700 1 -- 172.21.15.7:0/3822012105 shutdown_connections 2023-03-03T13:48:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.635+0000 7f3cb571c700 1 --2- 172.21.15.7:0/3822012105 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb0154ff0 0x7f3cb00cae00 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.635+0000 7f3cb571c700 1 --2- 172.21.15.7:0/3822012105 >> v2:172.21.15.143:3300/0 conn(0x7f3cb00caa30 0x7f3cb00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:29.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.635+0000 7f3cb571c700 1 -- 172.21.15.7:0/3822012105 >> 172.21.15.7:0/3822012105 conn(0x7f3cb0066e40 msgr2=0x7f3cb0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:48:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.635+0000 7f3cb571c700 1 -- 172.21.15.7:0/3822012105 shutdown_connections 2023-03-03T13:48:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.635+0000 7f3cb571c700 1 -- 172.21.15.7:0/3822012105 wait complete. 2023-03-03T13:48:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.636+0000 7f3cb571c700 1 Processor -- start 2023-03-03T13:48:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.636+0000 7f3cb571c700 1 -- start start 2023-03-03T13:48:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cb571c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb00c00e0 0x7f3cb00c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cb571c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb00caa30 0x7f3cb014d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:29.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cb571c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb0154ff0 0x7f3cb00c24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cb571c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb014d8c0 con 0x7f3cb00caa30 2023-03-03T13:48:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cb571c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb01df080 con 0x7f3cb00c00e0 2023-03-03T13:48:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cb571c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb01ddb70 con 0x7f3cb0154ff0 2023-03-03T13:48:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cae7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb00caa30 0x7f3cb014d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cae7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb00caa30 0x7f3cb014d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46154/0 (socket says 172.21.15.7:46154) 2023-03-03T13:48:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cae7fc700 1 -- 172.21.15.7:0/3645916849 learned_addr learned my addr 172.21.15.7:0/3645916849 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:48:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3caeffd700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb00c00e0 0x7f3cb00c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:29.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3caf7fe700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb0154ff0 0x7f3cb00c24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cae7fc700 1 -- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb0154ff0 msgr2=0x7f3cb00c24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cae7fc700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb0154ff0 0x7f3cb00c24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cae7fc700 1 -- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb00c00e0 msgr2=0x7f3cb00c4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cae7fc700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb00c00e0 0x7f3cb00c4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cae7fc700 1 -- 172.21.15.7:0/3645916849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca8009b30 con 0x7f3cb00caa30 2023-03-03T13:48:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.637+0000 7f3cae7fc700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb00caa30 0x7f3cb014d380 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f3ca000c550 tx=0x7f3ca000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.638+0000 7f3c977fe700 1 -- 172.21.15.7:0/3645916849 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca00054e0 con 0x7f3cb00caa30 2023-03-03T13:48:29.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.638+0000 7f3c977fe700 1 -- 172.21.15.7:0/3645916849 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca00036d0 con 0x7f3cb00caa30 2023-03-03T13:48:29.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.638+0000 7f3cb571c700 1 -- 172.21.15.7:0/3645916849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb01547f0 con 0x7f3cb00caa30 2023-03-03T13:48:29.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.638+0000 7f3c977fe700 1 -- 172.21.15.7:0/3645916849 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0003a80 con 0x7f3cb00caa30 2023-03-03T13:48:29.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.638+0000 7f3cb571c700 1 -- 172.21.15.7:0/3645916849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb0154510 con 0x7f3cb00caa30 2023-03-03T13:48:29.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.639+0000 7f3cb571c700 1 -- 172.21.15.7:0/3645916849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb0154ad0 con 0x7f3cb00caa30 2023-03-03T13:48:29.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.639+0000 7f3c977fe700 1 -- 172.21.15.7:0/3645916849 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3ca0019030 con 0x7f3cb00caa30 2023-03-03T13:48:29.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.640+0000 7f3c977fe700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c9c0682f0 0x7f3c9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:48:29.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.640+0000 7f3c977fe700 1 -- 172.21.15.7:0/3645916849 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3ca00853c0 con 0x7f3cb00caa30 2023-03-03T13:48:29.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.640+0000 7f3caeffd700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c9c0682f0 0x7f3c9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:48:29.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.640+0000 7f3caeffd700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c9c0682f0 0x7f3c9c06a7a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f3cb01edf70 tx=0x7f3ca4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:48:29.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.644+0000 7f3c977fe700 1 -- 172.21.15.7:0/3645916849 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3ca0051760 con 0x7f3cb00caa30 2023-03-03T13:48:29.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.880+0000 7f3cb571c700 1 -- 172.21.15.7:0/3645916849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cb00c4590 con 0x7f3cb00caa30 2023-03-03T13:48:29.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.880+0000 7f3c977fe700 1 -- 172.21.15.7:0/3645916849 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3ca0017090 con 0x7f3cb00caa30 2023-03-03T13:48:29.882 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:48:29.882 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:48:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 -- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c9c0682f0 msgr2=0x7f3c9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c9c0682f0 0x7f3c9c06a7a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f3cb01edf70 tx=0x7f3ca4008040 comp rx=0 tx=0).stop 2023-03-03T13:48:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 -- 172.21.15.7:0/3645916849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb00caa30 msgr2=0x7f3cb014d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:48:29.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb00caa30 0x7f3cb014d380 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f3ca000c550 tx=0x7f3ca000c860 comp rx=0 tx=0).stop 2023-03-03T13:48:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 -- 172.21.15.7:0/3645916849 shutdown_connections 2023-03-03T13:48:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb0154ff0 0x7f3cb00c24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb00c00e0 0x7f3cb00c4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c9c0682f0 0x7f3c9c06a7a0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 --2- 172.21.15.7:0/3645916849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cb00caa30 0x7f3cb014d380 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:48:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 -- 172.21.15.7:0/3645916849 >> 172.21.15.7:0/3645916849 conn(0x7f3cb0066e40 msgr2=0x7f3cb00c62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:48:29.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 -- 172.21.15.7:0/3645916849 shutdown_connections 2023-03-03T13:48:29.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:48:29.882+0000 7f3cb571c700 1 -- 172.21.15.7:0/3645916849 wait complete. 2023-03-03T13:48:29.886 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:48:29.896 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T13:48:46.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:48:46.833 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:17.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:17.065 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:24.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:49:24.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.171+0000 7fe4f4c4a700 1 Processor -- start 2023-03-03T13:49:24.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.171+0000 7fe4f4c4a700 1 -- start start 2023-03-03T13:49:24.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4f4c4a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe4f0151ab0 0x7fe4f00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:24.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4f4c4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 0x7fe4f006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:24.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4f4c4a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f006d780 con 0x7fe4f006c3e0 2023-03-03T13:49:24.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4f4c4a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f006cb70 con 0x7fe4f0151ab0 2023-03-03T13:49:24.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4f4c4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f006ce50 con 0x7fe4f0073c30 2023-03-03T13:49:24.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4eed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe4f006c3e0 0x7fe4f0157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40190/0 (socket says 172.21.15.7:40190) 2023-03-03T13:49:24.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4eed9d700 1 -- 172.21.15.7:0/49792268 learned_addr learned my addr 172.21.15.7:0/49792268 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:49:24.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4edd9b700 1 --2- 172.21.15.7:0/49792268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 0x7fe4f006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:24.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4ee59c700 1 --2- 172.21.15.7:0/49792268 >> v2:172.21.15.143:3300/0 conn(0x7fe4f0151ab0 0x7fe4f00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:24.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4ed59a700 1 -- 172.21.15.7:0/49792268 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647706121 0 0) 0x7fe4f006d780 con 0x7fe4f006c3e0 2023-03-03T13:49:24.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.172+0000 7fe4ed59a700 1 -- 172.21.15.7:0/49792268 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d80035b0 con 0x7fe4f006c3e0 2023-03-03T13:49:24.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.173+0000 7fe4edd9b700 1 -- 172.21.15.7:0/49792268 >> v2:172.21.15.143:3300/0 conn(0x7fe4f0151ab0 msgr2=0x7fe4f00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:49:24.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.173+0000 7fe4edd9b700 1 --2- 172.21.15.7:0/49792268 >> v2:172.21.15.143:3300/0 conn(0x7fe4f0151ab0 0x7fe4f00736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:24.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.173+0000 7fe4edd9b700 1 -- 172.21.15.7:0/49792268 >> v1:172.21.15.7:6789/0 conn(0x7fe4f006c3e0 legacy=0x7fe4f0157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:24.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.173+0000 7fe4edd9b700 1 -- 172.21.15.7:0/49792268 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f006d130 con 0x7fe4f0073c30 2023-03-03T13:49:24.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.173+0000 7fe4ee59c700 1 --2- 172.21.15.7:0/49792268 >> v2:172.21.15.143:3300/0 conn(0x7fe4f0151ab0 0x7fe4f00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:49:24.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.173+0000 7fe4edd9b700 1 --2- 172.21.15.7:0/49792268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 0x7fe4f006bea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009840 tx=0x7fe4e0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f71b468e6309551 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:49:24.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.173+0000 7fe4ed59a700 1 -- 172.21.15.7:0/49792268 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0011830 con 0x7fe4f0073c30 2023-03-03T13:49:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.173+0000 7fe4ed59a700 1 -- 172.21.15.7:0/49792268 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e0011990 con 0x7fe4f0073c30 2023-03-03T13:49:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.173+0000 7fe4ed59a700 1 -- 172.21.15.7:0/49792268 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0011d20 con 0x7fe4f0073c30 2023-03-03T13:49:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.174+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/49792268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 msgr2=0x7fe4f006bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.174+0000 7fe4f4c4a700 1 --2- 172.21.15.7:0/49792268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 0x7fe4f006bea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009840 tx=0x7fe4e0009b50 comp rx=0 tx=0).stop 2023-03-03T13:49:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.174+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/49792268 shutdown_connections 2023-03-03T13:49:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.174+0000 7fe4f4c4a700 1 --2- 172.21.15.7:0/49792268 >> v2:172.21.15.143:3300/0 conn(0x7fe4f0151ab0 0x7fe4f00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:24.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.174+0000 7fe4f4c4a700 1 --2- 172.21.15.7:0/49792268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 0x7fe4f006bea0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.174+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/49792268 >> 172.21.15.7:0/49792268 conn(0x7fe4f00a94d0 msgr2=0x7fe4f00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:49:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.174+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/49792268 shutdown_connections 2023-03-03T13:49:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.174+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/49792268 wait complete. 2023-03-03T13:49:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.175+0000 7fe4f4c4a700 1 Processor -- start 2023-03-03T13:49:24.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.176+0000 7fe4f4c4a700 1 -- start start 2023-03-03T13:49:24.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.176+0000 7fe4f4c4a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f006c3e0 0x7fe4f014e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:24.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.176+0000 7fe4f4c4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 0x7fe4f01eb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:24.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.176+0000 7fe4f4c4a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f0151ab0 0x7fe4f01eb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:24.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.176+0000 7fe4f4c4a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f014b580 con 0x7fe4f0151ab0 2023-03-03T13:49:24.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.176+0000 7fe4f4c4a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f014a970 con 0x7fe4f006c3e0 2023-03-03T13:49:24.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.176+0000 7fe4f4c4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f014ac50 con 0x7fe4f0073c30 2023-03-03T13:49:24.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4eed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f0151ab0 0x7fe4f01eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:24.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4eed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f0151ab0 0x7fe4f01eb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46216/0 (socket says 172.21.15.7:46216) 2023-03-03T13:49:24.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4eed9d700 1 -- 172.21.15.7:0/4186210403 learned_addr learned my addr 172.21.15.7:0/4186210403 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:49:24.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4edd9b700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 0x7fe4f01eb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:24.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4ee59c700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f006c3e0 0x7fe4f014e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4eed9d700 1 -- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 msgr2=0x7fe4f01eb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4eed9d700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 0x7fe4f01eb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4eed9d700 1 -- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f006c3e0 msgr2=0x7fe4f014e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4eed9d700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f006c3e0 0x7fe4f014e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4eed9d700 1 -- 172.21.15.7:0/4186210403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e40094a0 con 0x7fe4f0151ab0 2023-03-03T13:49:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4eed9d700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f0151ab0 0x7fe4f01eb620 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800cb40 tx=0x7fe4e800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:49:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4deffd700 1 -- 172.21.15.7:0/4186210403 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e800aaa0 con 0x7fe4f0151ab0 2023-03-03T13:49:24.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4deffd700 1 -- 172.21.15.7:0/4186210403 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e800ac00 con 0x7fe4f0151ab0 2023-03-03T13:49:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/4186210403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e00094a0 con 0x7fe4f0151ab0 2023-03-03T13:49:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4deffd700 1 -- 172.21.15.7:0/4186210403 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8012400 con 0x7fe4f0151ab0 2023-03-03T13:49:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.177+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/4186210403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f0151e80 con 0x7fe4f0151ab0 2023-03-03T13:49:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.178+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/4186210403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f006d410 con 0x7fe4f0151ab0 2023-03-03T13:49:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.179+0000 7fe4deffd700 1 -- 172.21.15.7:0/4186210403 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe4e8019030 con 0x7fe4f0151ab0 2023-03-03T13:49:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.179+0000 7fe4deffd700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d00682f0 0x7fe4d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.179+0000 7fe4deffd700 1 -- 172.21.15.7:0/4186210403 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe4e8085d70 con 0x7fe4f0151ab0 2023-03-03T13:49:24.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.180+0000 7fe4ee59c700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d00682f0 0x7fe4d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:24.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.180+0000 7fe4ee59c700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d00682f0 0x7fe4d006a7a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4009c60 tx=0x7fe4e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:49:24.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.184+0000 7fe4deffd700 1 -- 172.21.15.7:0/4186210403 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe4e80520b0 con 0x7fe4f0151ab0 2023-03-03T13:49:24.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.422+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/4186210403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4f0152010 con 0x7fe4f0151ab0 2023-03-03T13:49:24.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.422+0000 7fe4deffd700 1 -- 172.21.15.7:0/4186210403 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe4e80556d0 con 0x7fe4f0151ab0 2023-03-03T13:49:24.424 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:49:24.425 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:49:24.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d00682f0 msgr2=0x7fe4d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:24.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d00682f0 0x7fe4d006a7a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4009c60 tx=0x7fe4e4008040 comp rx=0 tx=0).stop 2023-03-03T13:49:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/4186210403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f0151ab0 msgr2=0x7fe4f01eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f0151ab0 0x7fe4f01eb620 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fe4e800cb40 tx=0x7fe4e800ce50 comp rx=0 tx=0).stop 2023-03-03T13:49:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/4186210403 shutdown_connections 2023-03-03T13:49:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d00682f0 0x7fe4d006a7a0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f0151ab0 0x7fe4f01eb620 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f006c3e0 0x7fe4f014e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 --2- 172.21.15.7:0/4186210403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f0073c30 0x7fe4f01eb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/4186210403 >> 172.21.15.7:0/4186210403 conn(0x7fe4f00a94d0 msgr2=0x7fe4f00710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:49:24.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.424+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/4186210403 shutdown_connections 2023-03-03T13:49:24.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:24.425+0000 7fe4f4c4a700 1 -- 172.21.15.7:0/4186210403 wait complete. 2023-03-03T13:49:24.428 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:49:24.438 INFO:tasks.mds_thrash.fs.[a]:waiting for 113 secs before thrashing 2023-03-03T13:49:33.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:49:34.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.036+0000 7fd92f39d700 1 Processor -- start 2023-03-03T13:49:34.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.036+0000 7fd92f39d700 1 -- start start 2023-03-03T13:49:34.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.037+0000 7fd92f39d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9280c20d0 0x7fd9281562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:34.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.037+0000 7fd92f39d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd9280c27c0 0x7fd9280736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:34.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.037+0000 7fd92f39d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd928155e60 con 0x7fd92814f860 2023-03-03T13:49:34.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.037+0000 7fd92f39d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd92806d350 con 0x7fd9280c27c0 2023-03-03T13:49:34.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.037+0000 7fd92f39d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9281558e0 con 0x7fd9280c20d0 2023-03-03T13:49:34.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.037+0000 7fd92d139700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd92814f860 0x7fd928152d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40202/0 (socket says 172.21.15.7:40202) 2023-03-03T13:49:34.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.037+0000 7fd92d139700 1 -- 172.21.15.7:0/4216978185 learned_addr learned my addr 172.21.15.7:0/4216978185 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:49:34.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.037+0000 7fd92d93a700 1 --2- 172.21.15.7:0/4216978185 >> v2:172.21.15.7:3300/0 conn(0x7fd9280c27c0 0x7fd9280736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:34.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd92c938700 1 --2- 172.21.15.7:0/4216978185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9280c20d0 0x7fd9281562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:34.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd91ffff700 1 -- 172.21.15.7:0/4216978185 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636656048 0 0) 0x7fd928155e60 con 0x7fd92814f860 2023-03-03T13:49:34.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd91ffff700 1 -- 172.21.15.7:0/4216978185 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9140035b0 con 0x7fd92814f860 2023-03-03T13:49:34.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd92d93a700 1 -- 172.21.15.7:0/4216978185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9280c20d0 msgr2=0x7fd9281562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:49:34.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd92d93a700 1 --2- 172.21.15.7:0/4216978185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9280c20d0 0x7fd9281562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:34.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd92d93a700 1 -- 172.21.15.7:0/4216978185 >> v1:172.21.15.7:6789/0 conn(0x7fd92814f860 legacy=0x7fd928152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:34.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd92d93a700 1 -- 172.21.15.7:0/4216978185 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92806cd90 con 0x7fd9280c27c0 2023-03-03T13:49:34.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd92c938700 1 --2- 172.21.15.7:0/4216978185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9280c20d0 0x7fd9281562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:49:34.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd92d93a700 1 --2- 172.21.15.7:0/4216978185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c27c0 0x7fd9280736f0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fd920009d90 tx=0x7fd92000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9e11fed2e90d85 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:49:34.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd91ffff700 1 -- 172.21.15.7:0/4216978185 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92000e070 con 0x7fd9280c27c0 2023-03-03T13:49:34.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd91ffff700 1 -- 172.21.15.7:0/4216978185 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92000bb20 con 0x7fd9280c27c0 2023-03-03T13:49:34.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.038+0000 7fd91ffff700 1 -- 172.21.15.7:0/4216978185 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920012430 con 0x7fd9280c27c0 2023-03-03T13:49:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.039+0000 7fd92f39d700 1 -- 172.21.15.7:0/4216978185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c27c0 msgr2=0x7fd9280736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.039+0000 7fd92f39d700 1 --2- 172.21.15.7:0/4216978185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c27c0 0x7fd9280736f0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fd920009d90 tx=0x7fd92000b330 comp rx=0 tx=0).stop 2023-03-03T13:49:34.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.039+0000 7fd92f39d700 1 -- 172.21.15.7:0/4216978185 shutdown_connections 2023-03-03T13:49:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.039+0000 7fd92f39d700 1 --2- 172.21.15.7:0/4216978185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c27c0 0x7fd9280736f0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.039+0000 7fd92f39d700 1 --2- 172.21.15.7:0/4216978185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9280c20d0 0x7fd9281562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.039+0000 7fd92f39d700 1 -- 172.21.15.7:0/4216978185 >> 172.21.15.7:0/4216978185 conn(0x7fd9280a94d0 msgr2=0x7fd9280a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:49:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.039+0000 7fd92f39d700 1 -- 172.21.15.7:0/4216978185 shutdown_connections 2023-03-03T13:49:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.039+0000 7fd92f39d700 1 -- 172.21.15.7:0/4216978185 wait complete. 2023-03-03T13:49:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.040+0000 7fd92f39d700 1 Processor -- start 2023-03-03T13:49:34.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.040+0000 7fd92f39d700 1 -- start start 2023-03-03T13:49:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92f39d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c20d0 0x7fd92814af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92f39d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9280c27c0 0x7fd9281e7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92f39d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd92814f860 0x7fd92814a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92f39d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92814b490 con 0x7fd9280c20d0 2023-03-03T13:49:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92f39d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd928155370 con 0x7fd9280c27c0 2023-03-03T13:49:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92f39d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9280aa6c0 con 0x7fd92814f860 2023-03-03T13:49:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92d139700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c20d0 0x7fd92814af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:34.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92d139700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c20d0 0x7fd92814af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46228/0 (socket says 172.21.15.7:46228) 2023-03-03T13:49:34.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92d139700 1 -- 172.21.15.7:0/3934773304 learned_addr learned my addr 172.21.15.7:0/3934773304 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:49:34.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92d93a700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd92814f860 0x7fd92814a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:34.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92c938700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9280c27c0 0x7fd9281e7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:34.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92d139700 1 -- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd92814f860 msgr2=0x7fd92814a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:34.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92d139700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd92814f860 0x7fd92814a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:34.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92d139700 1 -- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9280c27c0 msgr2=0x7fd9281e7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:34.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92d139700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9280c27c0 0x7fd9281e7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:34.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92d139700 1 -- 172.21.15.7:0/3934773304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd920009a70 con 0x7fd9280c20d0 2023-03-03T13:49:34.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92d139700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c20d0 0x7fd92814af50 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c4d0 tx=0x7fd91800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:49:34.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd91dffb700 1 -- 172.21.15.7:0/3934773304 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91800cea0 con 0x7fd9280c20d0 2023-03-03T13:49:34.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd91dffb700 1 -- 172.21.15.7:0/3934773304 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd918005d70 con 0x7fd9280c20d0 2023-03-03T13:49:34.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd92f39d700 1 -- 172.21.15.7:0/3934773304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd928155af0 con 0x7fd9280c20d0 2023-03-03T13:49:34.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.041+0000 7fd91dffb700 1 -- 172.21.15.7:0/3934773304 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9180072f0 con 0x7fd9280c20d0 2023-03-03T13:49:34.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.042+0000 7fd92f39d700 1 -- 172.21.15.7:0/3934773304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92806d070 con 0x7fd9280c20d0 2023-03-03T13:49:34.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.043+0000 7fd92f39d700 1 -- 172.21.15.7:0/3934773304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92814fc30 con 0x7fd9280c20d0 2023-03-03T13:49:34.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.043+0000 7fd91dffb700 1 -- 172.21.15.7:0/3934773304 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd918019030 con 0x7fd9280c20d0 2023-03-03T13:49:34.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.043+0000 7fd91dffb700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd90c0682f0 0x7fd90c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:49:34.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.043+0000 7fd91dffb700 1 -- 172.21.15.7:0/3934773304 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd9180853f0 con 0x7fd9280c20d0 2023-03-03T13:49:34.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.044+0000 7fd92c938700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd90c0682f0 0x7fd90c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:49:34.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.044+0000 7fd92c938700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd90c0682f0 0x7fd90c06a7a0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fd9281dfa40 tx=0x7fd924008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:49:34.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.048+0000 7fd91dffb700 1 -- 172.21.15.7:0/3934773304 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd918051790 con 0x7fd9280c20d0 2023-03-03T13:49:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.283+0000 7fd92f39d700 1 -- 172.21.15.7:0/3934773304 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd928074bb0 con 0x7fd9280c20d0 2023-03-03T13:49:34.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.284+0000 7fd91dffb700 1 -- 172.21.15.7:0/3934773304 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd918054db0 con 0x7fd9280c20d0 2023-03-03T13:49:34.286 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:49:34.286 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:49:34.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 -- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd90c0682f0 msgr2=0x7fd90c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:34.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd90c0682f0 0x7fd90c06a7a0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fd9281dfa40 tx=0x7fd924008040 comp rx=0 tx=0).stop 2023-03-03T13:49:34.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 -- 172.21.15.7:0/3934773304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c20d0 msgr2=0x7fd92814af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:49:34.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c20d0 0x7fd92814af50 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c4d0 tx=0x7fd91800c7e0 comp rx=0 tx=0).stop 2023-03-03T13:49:34.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 -- 172.21.15.7:0/3934773304 shutdown_connections 2023-03-03T13:49:34.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd92814f860 0x7fd92814a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:34.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd90c0682f0 0x7fd90c06a7a0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:34.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9280c20d0 0x7fd92814af50 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:34.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 --2- 172.21.15.7:0/3934773304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9280c27c0 0x7fd9281e7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:49:34.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 -- 172.21.15.7:0/3934773304 >> 172.21.15.7:0/3934773304 conn(0x7fd9280a94d0 msgr2=0x7fd9280a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:49:34.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 -- 172.21.15.7:0/3934773304 shutdown_connections 2023-03-03T13:49:34.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:49:34.286+0000 7fd92f39d700 1 -- 172.21.15.7:0/3934773304 wait complete. 2023-03-03T13:49:34.289 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:49:34.299 INFO:tasks.mds_thrash.fs.[b]:waiting for 101 secs before thrashing 2023-03-03T13:49:47.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:49:47.128 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T13:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T13:50:17.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:17.225 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:47.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:50:47.288 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:15.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:51:15.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.438+0000 7f92c2891700 1 Processor -- start 2023-03-03T13:51:15.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.438+0000 7f92c2891700 1 -- start start 2023-03-03T13:51:15.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.439+0000 7f92c2891700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc0beab0 0x7f92bc155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:15.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.439+0000 7f92c2891700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f92bc1562c0 0x7f92bc15cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:15.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.439+0000 7f92c2891700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92bc1523b0 con 0x7f92bc152e30 2023-03-03T13:51:15.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.439+0000 7f92c2891700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92bc072010 con 0x7f92bc1562c0 2023-03-03T13:51:15.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.439+0000 7f92c2891700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc0aaaf0 con 0x7f92bc0beab0 2023-03-03T13:51:15.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.439+0000 7f92c0e2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f92bc1562c0 0x7f92bc15cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:15.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.439+0000 7f92bb7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc0beab0 0x7f92bc155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:15.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.439+0000 7f92bbfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f92bc152e30 0x7f92bc0c43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:43796/0 (socket says 172.21.15.7:43796) 2023-03-03T13:51:15.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.439+0000 7f92bb7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc0beab0 0x7f92bc155d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:54370/0 (socket says 172.21.15.7:54370) 2023-03-03T13:51:15.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.439+0000 7f92bbfff700 1 -- 172.21.15.7:0/3649664471 learned_addr learned my addr 172.21.15.7:0/3649664471 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:51:15.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.440+0000 7f92c0e2e700 1 -- 172.21.15.7:0/3649664471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc0beab0 msgr2=0x7f92bc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:51:15.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.440+0000 7f92c0e2e700 1 --2- 172.21.15.7:0/3649664471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc0beab0 0x7f92bc155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:15.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.440+0000 7f92c0e2e700 1 -- 172.21.15.7:0/3649664471 >> v1:172.21.15.143:6789/0 conn(0x7f92bc152e30 legacy=0x7f92bc0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:15.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.440+0000 7f92c0e2e700 1 -- 172.21.15.7:0/3649664471 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc0aadd0 con 0x7f92bc1562c0 2023-03-03T13:51:15.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.440+0000 7f92baffd700 1 -- 172.21.15.7:0/3649664471 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923153557 0 0) 0x7f92bc1523b0 con 0x7f92bc152e30 2023-03-03T13:51:15.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.440+0000 7f92bb7fe700 1 --2- 172.21.15.7:0/3649664471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc0beab0 0x7f92bc155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:51:15.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.440+0000 7f92c0e2e700 1 --2- 172.21.15.7:0/3649664471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc1562c0 0x7f92bc15cd30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009d90 tx=0x7f92b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8396f15ec7277356 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:51:15.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.440+0000 7f92baffd700 1 -- 172.21.15.7:0/3649664471 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b400e070 con 0x7f92bc1562c0 2023-03-03T13:51:15.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.440+0000 7f92baffd700 1 -- 172.21.15.7:0/3649664471 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b400bb20 con 0x7f92bc1562c0 2023-03-03T13:51:15.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.440+0000 7f92baffd700 1 -- 172.21.15.7:0/3649664471 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4012430 con 0x7f92bc1562c0 2023-03-03T13:51:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.441+0000 7f92c2891700 1 -- 172.21.15.7:0/3649664471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc1562c0 msgr2=0x7f92bc15cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.441+0000 7f92c2891700 1 --2- 172.21.15.7:0/3649664471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc1562c0 0x7f92bc15cd30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009d90 tx=0x7f92b400b330 comp rx=0 tx=0).stop 2023-03-03T13:51:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.441+0000 7f92c2891700 1 -- 172.21.15.7:0/3649664471 shutdown_connections 2023-03-03T13:51:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.441+0000 7f92c2891700 1 --2- 172.21.15.7:0/3649664471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc1562c0 0x7f92bc15cd30 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.441+0000 7f92c2891700 1 --2- 172.21.15.7:0/3649664471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc0beab0 0x7f92bc155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.441+0000 7f92c2891700 1 -- 172.21.15.7:0/3649664471 >> 172.21.15.7:0/3649664471 conn(0x7f92bc0736f0 msgr2=0x7f92bc073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:51:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.441+0000 7f92c2891700 1 -- 172.21.15.7:0/3649664471 shutdown_connections 2023-03-03T13:51:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.441+0000 7f92c2891700 1 -- 172.21.15.7:0/3649664471 wait complete. 2023-03-03T13:51:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.442+0000 7f92c2891700 1 Processor -- start 2023-03-03T13:51:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.442+0000 7f92c2891700 1 -- start start 2023-03-03T13:51:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92c2891700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92bc0beab0 0x7f92bc0ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92c2891700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc152e30 0x7f92bc1e85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92c2891700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc1562c0 0x7f92bc0c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92c2891700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc153b10 con 0x7f92bc0beab0 2023-03-03T13:51:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92c2891700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc1e8b20 con 0x7f92bc1562c0 2023-03-03T13:51:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92c2891700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc1eb150 con 0x7f92bc152e30 2023-03-03T13:51:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92bc0beab0 0x7f92bc0ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92bc0beab0 0x7f92bc0ab460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46302/0 (socket says 172.21.15.7:46302) 2023-03-03T13:51:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92bbfff700 1 -- 172.21.15.7:0/2008425482 learned_addr learned my addr 172.21.15.7:0/2008425482 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:51:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92bb7fe700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc152e30 0x7f92bc1e85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.443+0000 7f92c0e2e700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc1562c0 0x7f92bc0c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92bbfff700 1 -- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc152e30 msgr2=0x7f92bc1e85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92bbfff700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc152e30 0x7f92bc1e85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:15.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92bbfff700 1 -- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc1562c0 msgr2=0x7f92bc0c45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:15.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92bbfff700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc1562c0 0x7f92bc0c45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:15.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92bbfff700 1 -- 172.21.15.7:0/2008425482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a40094a0 con 0x7f92bc0beab0 2023-03-03T13:51:15.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92bbfff700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92bc0beab0 0x7f92bc0ab460 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0011e0 tx=0x7f92ac00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:51:15.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92b8ff9700 1 -- 172.21.15.7:0/2008425482 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac004c80 con 0x7f92bc0beab0 2023-03-03T13:51:15.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92b8ff9700 1 -- 172.21.15.7:0/2008425482 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ac004e00 con 0x7f92bc0beab0 2023-03-03T13:51:15.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92c2891700 1 -- 172.21.15.7:0/2008425482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b4009a70 con 0x7f92bc0beab0 2023-03-03T13:51:15.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92b8ff9700 1 -- 172.21.15.7:0/2008425482 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ac007310 con 0x7f92bc0beab0 2023-03-03T13:51:15.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.444+0000 7f92c2891700 1 -- 172.21.15.7:0/2008425482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92bc153200 con 0x7f92bc0beab0 2023-03-03T13:51:15.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.445+0000 7f92c2891700 1 -- 172.21.15.7:0/2008425482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92bc153390 con 0x7f92bc0beab0 2023-03-03T13:51:15.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.446+0000 7f92b8ff9700 1 -- 172.21.15.7:0/2008425482 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f92ac01a020 con 0x7f92bc0beab0 2023-03-03T13:51:15.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.446+0000 7f92b8ff9700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f929c0682f0 0x7f929c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:15.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.446+0000 7f92b8ff9700 1 -- 172.21.15.7:0/2008425482 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f92ac085c40 con 0x7f92bc0beab0 2023-03-03T13:51:15.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.447+0000 7f92bb7fe700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f929c0682f0 0x7f929c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:15.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.447+0000 7f92bb7fe700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f929c0682f0 0x7f929c06a7a0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f92bc1dc770 tx=0x7f92a4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:51:15.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.451+0000 7f92b8ff9700 1 -- 172.21.15.7:0/2008425482 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f92ac051fe0 con 0x7f92bc0beab0 2023-03-03T13:51:15.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.688+0000 7f92c2891700 1 -- 172.21.15.7:0/2008425482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92bc070a70 con 0x7f92bc0beab0 2023-03-03T13:51:15.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.689+0000 7f92b8ff9700 1 -- 172.21.15.7:0/2008425482 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f92ac055600 con 0x7f92bc0beab0 2023-03-03T13:51:15.691 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:51:15.691 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:51:15.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 -- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f929c0682f0 msgr2=0x7f929c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:15.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f929c0682f0 0x7f929c06a7a0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f92bc1dc770 tx=0x7f92a4009210 comp rx=0 tx=0).stop 2023-03-03T13:51:15.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 -- 172.21.15.7:0/2008425482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92bc0beab0 msgr2=0x7f92bc0ab460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:15.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92bc0beab0 0x7f92bc0ab460 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f92ac0011e0 tx=0x7f92ac00ca30 comp rx=0 tx=0).stop 2023-03-03T13:51:15.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 -- 172.21.15.7:0/2008425482 shutdown_connections 2023-03-03T13:51:15.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92bc1562c0 0x7f92bc0c45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:15.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f929c0682f0 0x7f929c06a7a0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:15.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92bc0beab0 0x7f92bc0ab460 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:15.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 --2- 172.21.15.7:0/2008425482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92bc152e30 0x7f92bc1e85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:15.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 -- 172.21.15.7:0/2008425482 >> 172.21.15.7:0/2008425482 conn(0x7f92bc0736f0 msgr2=0x7f92bc0be230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:51:15.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 -- 172.21.15.7:0/2008425482 shutdown_connections 2023-03-03T13:51:15.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:15.691+0000 7f92c2891700 1 -- 172.21.15.7:0/2008425482 wait complete. 2023-03-03T13:51:15.695 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:51:15.705 INFO:tasks.mds_thrash.fs.[b]:waiting for 110 secs before thrashing 2023-03-03T13:51:17.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:17.349 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:17.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:51:17.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.573+0000 7f98041b5700 1 Processor -- start 2023-03-03T13:51:17.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.574+0000 7f98041b5700 1 -- start start 2023-03-03T13:51:17.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.574+0000 7f98041b5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 0x7f97fc157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.574+0000 7f98041b5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97fc06d430 con 0x7f97fc151d40 2023-03-03T13:51:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.574+0000 7f98041b5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97fc1586e0 con 0x7f97fc06c480 2023-03-03T13:51:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.574+0000 7f98041b5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc06c2a0 con 0x7f97fc0c2930 2023-03-03T13:51:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.574+0000 7f9801f51700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f97fc151d40 0x7f97fc0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40290/0 (socket says 172.21.15.7:40290) 2023-03-03T13:51:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.574+0000 7f9801f51700 1 -- 172.21.15.7:0/1650818911 learned_addr learned my addr 172.21.15.7:0/1650818911 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:51:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.575+0000 7f9801750700 1 --2- 172.21.15.7:0/1650818911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 0x7f97fc157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.575+0000 7f9800f4f700 1 -- 172.21.15.7:0/1650818911 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347930907 0 0) 0x7f97fc06d430 con 0x7f97fc151d40 2023-03-03T13:51:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.575+0000 7f9800f4f700 1 -- 172.21.15.7:0/1650818911 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97e80035b0 con 0x7f97fc151d40 2023-03-03T13:51:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.575+0000 7f9800f4f700 1 -- 172.21.15.7:0/1650818911 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2734378080 0 0) 0x7f97e80035b0 con 0x7f97fc151d40 2023-03-03T13:51:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.575+0000 7f9801750700 1 -- 172.21.15.7:0/1650818911 >> v1:172.21.15.143:6789/0 conn(0x7f97fc06c480 legacy=0x7f97fc157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.575+0000 7f9801750700 1 -- 172.21.15.7:0/1650818911 >> v1:172.21.15.7:6789/0 conn(0x7f97fc151d40 legacy=0x7f97fc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.575+0000 7f9801750700 1 -- 172.21.15.7:0/1650818911 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc06cb00 con 0x7f97fc0c2930 2023-03-03T13:51:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.575+0000 7f9801750700 1 --2- 172.21.15.7:0/1650818911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 0x7f97fc157a90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f97f0009870 tx=0x7f97f0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbe7e093b8f618d7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:51:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.575+0000 7f9800f4f700 1 -- 172.21.15.7:0/1650818911 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f8001670 con 0x7f97fc0c2930 2023-03-03T13:51:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.575+0000 7f9800f4f700 1 -- 172.21.15.7:0/1650818911 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f0004030 con 0x7f97fc0c2930 2023-03-03T13:51:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.576+0000 7f9800f4f700 1 -- 172.21.15.7:0/1650818911 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f00043c0 con 0x7f97fc0c2930 2023-03-03T13:51:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.576+0000 7f98041b5700 1 -- 172.21.15.7:0/1650818911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 msgr2=0x7f97fc157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.576+0000 7f98041b5700 1 --2- 172.21.15.7:0/1650818911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 0x7f97fc157a90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f97f0009870 tx=0x7f97f0009b80 comp rx=0 tx=0).stop 2023-03-03T13:51:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.576+0000 7f98041b5700 1 -- 172.21.15.7:0/1650818911 shutdown_connections 2023-03-03T13:51:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.576+0000 7f98041b5700 1 --2- 172.21.15.7:0/1650818911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 0x7f97fc157a90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.576+0000 7f98041b5700 1 -- 172.21.15.7:0/1650818911 >> 172.21.15.7:0/1650818911 conn(0x7f97fc0a94d0 msgr2=0x7f97fc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:51:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.576+0000 7f98041b5700 1 -- 172.21.15.7:0/1650818911 shutdown_connections 2023-03-03T13:51:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.576+0000 7f98041b5700 1 -- 172.21.15.7:0/1650818911 wait complete. 2023-03-03T13:51:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.577+0000 7f98041b5700 1 Processor -- start 2023-03-03T13:51:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.577+0000 7f98041b5700 1 -- start start 2023-03-03T13:51:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f98041b5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc06c480 0x7f97fc14afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f98041b5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 0x7f97fc14b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f98041b5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97fc151d40 0x7f97fc0c0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f98041b5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc0bf040 con 0x7f97fc06c480 2023-03-03T13:51:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f98041b5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc1eaba0 con 0x7f97fc151d40 2023-03-03T13:51:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f98041b5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97fc14bca0 con 0x7f97fc0c2930 2023-03-03T13:51:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f9801f51700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc06c480 0x7f97fc14afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f9801f51700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc06c480 0x7f97fc14afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46316/0 (socket says 172.21.15.7:46316) 2023-03-03T13:51:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f9801f51700 1 -- 172.21.15.7:0/1809408796 learned_addr learned my addr 172.21.15.7:0/1809408796 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:51:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f9801750700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 0x7f97fc14b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f9802752700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97fc151d40 0x7f97fc0c0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.578+0000 7f9801f51700 1 -- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 msgr2=0x7f97fc14b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f9801f51700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 0x7f97fc14b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f9801f51700 1 -- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97fc151d40 msgr2=0x7f97fc0c0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f9801f51700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97fc151d40 0x7f97fc0c0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f9801f51700 1 -- 172.21.15.7:0/1809408796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f00094a0 con 0x7f97fc06c480 2023-03-03T13:51:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f9802752700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97fc151d40 0x7f97fc0c0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:51:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f9801f51700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc06c480 0x7f97fc14afb0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f97f800c780 tx=0x7f97f800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:51:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f97ee7fc700 1 -- 172.21.15.7:0/1809408796 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f8004c80 con 0x7f97fc06c480 2023-03-03T13:51:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f97ee7fc700 1 -- 172.21.15.7:0/1809408796 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97f8004e00 con 0x7f97fc06c480 2023-03-03T13:51:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f98041b5700 1 -- 172.21.15.7:0/1809408796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97fc06d0c0 con 0x7f97fc06c480 2023-03-03T13:51:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f97ee7fc700 1 -- 172.21.15.7:0/1809408796 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f8007310 con 0x7f97fc06c480 2023-03-03T13:51:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.579+0000 7f98041b5700 1 -- 172.21.15.7:0/1809408796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97fc06cde0 con 0x7f97fc06c480 2023-03-03T13:51:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.580+0000 7f98041b5700 1 -- 172.21.15.7:0/1809408796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc073e00 con 0x7f97fc06c480 2023-03-03T13:51:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.580+0000 7f97ee7fc700 1 -- 172.21.15.7:0/1809408796 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f97f801a020 con 0x7f97fc06c480 2023-03-03T13:51:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.581+0000 7f97ee7fc700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97e00682f0 0x7f97e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:51:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.581+0000 7f97ee7fc700 1 -- 172.21.15.7:0/1809408796 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f97f8085e60 con 0x7f97fc06c480 2023-03-03T13:51:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.581+0000 7f9801750700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97e00682f0 0x7f97e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:51:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.581+0000 7f9801750700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97e00682f0 0x7f97e006a7a0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f97f0000c00 tx=0x7f97f0008000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:51:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.585+0000 7f97ee7fc700 1 -- 172.21.15.7:0/1809408796 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f97f8052230 con 0x7f97fc06c480 2023-03-03T13:51:17.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.821+0000 7f98041b5700 1 -- 172.21.15.7:0/1809408796 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97fc074bb0 con 0x7f97fc06c480 2023-03-03T13:51:17.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.822+0000 7f97ee7fc700 1 -- 172.21.15.7:0/1809408796 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f97f8055850 con 0x7f97fc06c480 2023-03-03T13:51:17.824 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:51:17.824 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:51:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 -- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97e00682f0 msgr2=0x7f97e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97e00682f0 0x7f97e006a7a0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f97f0000c00 tx=0x7f97f0008000 comp rx=0 tx=0).stop 2023-03-03T13:51:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 -- 172.21.15.7:0/1809408796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc06c480 msgr2=0x7f97fc14afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:51:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc06c480 0x7f97fc14afb0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f97f800c780 tx=0x7f97f800ca90 comp rx=0 tx=0).stop 2023-03-03T13:51:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 -- 172.21.15.7:0/1809408796 shutdown_connections 2023-03-03T13:51:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97fc151d40 0x7f97fc0c0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97e00682f0 0x7f97e006a7a0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc06c480 0x7f97fc14afb0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 --2- 172.21.15.7:0/1809408796 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97fc0c2930 0x7f97fc14b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:51:17.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 -- 172.21.15.7:0/1809408796 >> 172.21.15.7:0/1809408796 conn(0x7f97fc0a94d0 msgr2=0x7f97fc070aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:51:17.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 -- 172.21.15.7:0/1809408796 shutdown_connections 2023-03-03T13:51:17.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:51:17.824+0000 7f98041b5700 1 -- 172.21.15.7:0/1809408796 wait complete. 2023-03-03T13:51:17.828 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:51:17.838 INFO:tasks.mds_thrash.fs.[a]:waiting for 94 secs before thrashing 2023-03-03T13:51:47.414 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:51:47.417 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:17.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:17.494 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:47.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:47.556 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:52:51.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:52:51.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.975+0000 7fe548b8d700 1 Processor -- start 2023-03-03T13:52:51.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.975+0000 7fe548b8d700 1 -- start start 2023-03-03T13:52:51.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe548b8d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe54406e020 0x7fe544071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:51.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe548b8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544072520 0x7fe544151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:51.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe548b8d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe544154910 con 0x7fe544155390 2023-03-03T13:52:51.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe548b8d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe544070b90 con 0x7fe54406e020 2023-03-03T13:52:51.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe548b8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe544151320 con 0x7fe544072520 2023-03-03T13:52:51.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe541d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe54406e020 0x7fe544071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:51.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe542d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544072520 0x7fe544151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:51.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe541d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe54406e020 0x7fe544071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:42148/0 (socket says 172.21.15.7:42148) 2023-03-03T13:52:51.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe542d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544072520 0x7fe544151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:54448/0 (socket says 172.21.15.7:54448) 2023-03-03T13:52:51.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe541d9b700 1 -- 172.21.15.7:0/450312803 learned_addr learned my addr 172.21.15.7:0/450312803 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:52:51.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe54259c700 1 --1- 172.21.15.7:0/450312803 >> v1:172.21.15.143:6789/0 conn(0x7fe544155390 0x7fe5440c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:43872/0 (socket says 172.21.15.7:43872) 2023-03-03T13:52:51.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe541d9b700 1 -- 172.21.15.7:0/450312803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544072520 msgr2=0x7fe544151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:52:51.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.976+0000 7fe541d9b700 1 --2- 172.21.15.7:0/450312803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544072520 0x7fe544151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:51.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.977+0000 7fe541d9b700 1 -- 172.21.15.7:0/450312803 >> v1:172.21.15.143:6789/0 conn(0x7fe544155390 legacy=0x7fe5440c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:51.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.977+0000 7fe541d9b700 1 -- 172.21.15.7:0/450312803 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5440c4540 con 0x7fe54406e020 2023-03-03T13:52:51.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.977+0000 7fe54159a700 1 -- 172.21.15.7:0/450312803 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794483365 0 0) 0x7fe544154910 con 0x7fe544155390 2023-03-03T13:52:51.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.977+0000 7fe542d9d700 1 --2- 172.21.15.7:0/450312803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544072520 0x7fe544151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:52:51.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.977+0000 7fe541d9b700 1 --2- 172.21.15.7:0/450312803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe54406e020 0x7fe544071fe0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe53800bb90 tx=0x7fe53800bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1eac3b97006f1c0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:52:51.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.977+0000 7fe54159a700 1 -- 172.21.15.7:0/450312803 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53800dd30 con 0x7fe54406e020 2023-03-03T13:52:51.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.977+0000 7fe54159a700 1 -- 172.21.15.7:0/450312803 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53800de90 con 0x7fe54406e020 2023-03-03T13:52:51.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.977+0000 7fe54159a700 1 -- 172.21.15.7:0/450312803 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538010690 con 0x7fe54406e020 2023-03-03T13:52:51.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.977+0000 7fe548b8d700 1 -- 172.21.15.7:0/450312803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe54406e020 msgr2=0x7fe544071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:51.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.977+0000 7fe548b8d700 1 --2- 172.21.15.7:0/450312803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe54406e020 0x7fe544071fe0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe53800bb90 tx=0x7fe53800bea0 comp rx=0 tx=0).stop 2023-03-03T13:52:51.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.978+0000 7fe548b8d700 1 -- 172.21.15.7:0/450312803 shutdown_connections 2023-03-03T13:52:51.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.978+0000 7fe548b8d700 1 --2- 172.21.15.7:0/450312803 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544072520 0x7fe544151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.978+0000 7fe548b8d700 1 --2- 172.21.15.7:0/450312803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe54406e020 0x7fe544071fe0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.978+0000 7fe548b8d700 1 -- 172.21.15.7:0/450312803 >> 172.21.15.7:0/450312803 conn(0x7fe54406c110 msgr2=0x7fe54406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:52:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.978+0000 7fe548b8d700 1 -- 172.21.15.7:0/450312803 shutdown_connections 2023-03-03T13:52:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.978+0000 7fe548b8d700 1 -- 172.21.15.7:0/450312803 wait complete. 2023-03-03T13:52:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.979+0000 7fe548b8d700 1 Processor -- start 2023-03-03T13:52:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.979+0000 7fe548b8d700 1 -- start start 2023-03-03T13:52:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe548b8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe54406e020 0x7fe5441539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe548b8d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe544072520 0x7fe5441496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe548b8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544155390 0x7fe5440ab860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe548b8d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5441e64a0 con 0x7fe544072520 2023-03-03T13:52:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe548b8d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54413fe90 con 0x7fe54406e020 2023-03-03T13:52:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe548b8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5441e55d0 con 0x7fe544155390 2023-03-03T13:52:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe541d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe544072520 0x7fe5441496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe541d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe544072520 0x7fe5441496a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46380/0 (socket says 172.21.15.7:46380) 2023-03-03T13:52:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe541d9b700 1 -- 172.21.15.7:0/3814992545 learned_addr learned my addr 172.21.15.7:0/3814992545 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:52:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe542d9d700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544155390 0x7fe5440ab860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe54259c700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe54406e020 0x7fe5441539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe541d9b700 1 -- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544155390 msgr2=0x7fe5440ab860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe541d9b700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544155390 0x7fe5440ab860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe541d9b700 1 -- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe54406e020 msgr2=0x7fe5441539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe541d9b700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe54406e020 0x7fe5441539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe541d9b700 1 -- 172.21.15.7:0/3814992545 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c009a70 con 0x7fe544072520 2023-03-03T13:52:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.980+0000 7fe541d9b700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe544072520 0x7fe5441496a0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fe538000c00 tx=0x7fe538004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:52:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.981+0000 7fe532ffd700 1 -- 172.21.15.7:0/3814992545 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53801f070 con 0x7fe544072520 2023-03-03T13:52:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.981+0000 7fe532ffd700 1 -- 172.21.15.7:0/3814992545 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5380039f0 con 0x7fe544072520 2023-03-03T13:52:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.981+0000 7fe532ffd700 1 -- 172.21.15.7:0/3814992545 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538003d60 con 0x7fe544072520 2023-03-03T13:52:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.981+0000 7fe548b8d700 1 -- 172.21.15.7:0/3814992545 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53800b7f0 con 0x7fe544072520 2023-03-03T13:52:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.981+0000 7fe548b8d700 1 -- 172.21.15.7:0/3814992545 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe544070540 con 0x7fe544072520 2023-03-03T13:52:51.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.982+0000 7fe548b8d700 1 -- 172.21.15.7:0/3814992545 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5440702a0 con 0x7fe544072520 2023-03-03T13:52:51.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.982+0000 7fe532ffd700 1 -- 172.21.15.7:0/3814992545 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe538019400 con 0x7fe544072520 2023-03-03T13:52:51.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.983+0000 7fe532ffd700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5240682f0 0x7fe52406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:52:51.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.983+0000 7fe532ffd700 1 -- 172.21.15.7:0/3814992545 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe538089780 con 0x7fe544072520 2023-03-03T13:52:51.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.983+0000 7fe54259c700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5240682f0 0x7fe52406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:52:51.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.983+0000 7fe54259c700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5240682f0 0x7fe52406a7a0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fe534000f60 tx=0x7fe534009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:52:51.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:51.987+0000 7fe532ffd700 1 -- 172.21.15.7:0/3814992545 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5380559d0 con 0x7fe544072520 2023-03-03T13:52:52.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.229+0000 7fe548b8d700 1 -- 172.21.15.7:0/3814992545 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe544070820 con 0x7fe544072520 2023-03-03T13:52:52.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.229+0000 7fe532ffd700 1 -- 172.21.15.7:0/3814992545 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe538058ff0 con 0x7fe544072520 2023-03-03T13:52:52.231 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:52:52.231 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:52:52.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 -- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5240682f0 msgr2=0x7fe52406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:52.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5240682f0 0x7fe52406a7a0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fe534000f60 tx=0x7fe534009040 comp rx=0 tx=0).stop 2023-03-03T13:52:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 -- 172.21.15.7:0/3814992545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe544072520 msgr2=0x7fe5441496a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:52:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe544072520 0x7fe5441496a0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fe538000c00 tx=0x7fe538004750 comp rx=0 tx=0).stop 2023-03-03T13:52:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 -- 172.21.15.7:0/3814992545 shutdown_connections 2023-03-03T13:52:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe544155390 0x7fe5440ab860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe54406e020 0x7fe5441539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5240682f0 0x7fe52406a7a0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 --2- 172.21.15.7:0/3814992545 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe544072520 0x7fe5441496a0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:52:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 -- 172.21.15.7:0/3814992545 >> 172.21.15.7:0/3814992545 conn(0x7fe54406c110 msgr2=0x7fe54406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:52:52.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 -- 172.21.15.7:0/3814992545 shutdown_connections 2023-03-03T13:52:52.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:52:52.231+0000 7fe548b8d700 1 -- 172.21.15.7:0/3814992545 wait complete. 2023-03-03T13:52:52.235 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:52:52.245 INFO:tasks.mds_thrash.fs.[a]:waiting for 76 secs before thrashing 2023-03-03T13:53:05.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:53:05.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.843+0000 7f43d897a700 1 Processor -- start 2023-03-03T13:53:05.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.843+0000 7f43d897a700 1 -- start start 2023-03-03T13:53:05.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.844+0000 7f43d897a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f43d414b020 0x7f43d40717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:05.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.844+0000 7f43d897a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d40aad20 0x7f43d4157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:05.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.844+0000 7f43d897a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d41588f0 con 0x7f43d4071d20 2023-03-03T13:53:05.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.844+0000 7f43d897a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d40728b0 con 0x7f43d414b020 2023-03-03T13:53:05.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.844+0000 7f43d897a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4155070 con 0x7f43d40aad20 2023-03-03T13:53:05.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.844+0000 7f43d2d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f43d4071d20 0x7f43d41581e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40370/0 (socket says 172.21.15.7:40370) 2023-03-03T13:53:05.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.844+0000 7f43d2d9d700 1 -- 172.21.15.7:0/3461496548 learned_addr learned my addr 172.21.15.7:0/3461496548 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:53:05.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.844+0000 7f43d1d9b700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d40aad20 0x7f43d4157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:05.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.844+0000 7f43d259c700 1 --2- 172.21.15.7:0/3461496548 >> v2:172.21.15.143:3300/0 conn(0x7f43d414b020 0x7f43d40717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:05.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040345823 0 0) 0x7f43d41588f0 con 0x7f43d4071d20 2023-03-03T13:53:05.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43bc0035b0 con 0x7f43d4071d20 2023-03-03T13:53:05.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2146259182 0 0) 0x7f43bc0035b0 con 0x7f43d4071d20 2023-03-03T13:53:05.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d40aad20 msgr2=0x7f43d4157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:53:05.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d40aad20 0x7f43d4157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:05.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 >> v2:172.21.15.143:3300/0 conn(0x7f43d414b020 msgr2=0x7f43d40717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:53:05.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 --2- 172.21.15.7:0/3461496548 >> v2:172.21.15.143:3300/0 conn(0x7f43d414b020 0x7f43d40717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:05.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d406f5b0 con 0x7f43d4071d20 2023-03-03T13:53:05.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d259c700 1 --2- 172.21.15.7:0/3461496548 >> v2:172.21.15.143:3300/0 conn(0x7f43d414b020 0x7f43d40717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:53:05.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f43cc0019f0 con 0x7f43d4071d20 2023-03-03T13:53:05.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 >> v1:172.21.15.7:6789/0 conn(0x7f43d4071d20 legacy=0x7f43d41581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:05.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43bc003a70 0x7f43bc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:05.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d40aad20 0x7f43bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:05.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 0x7f43bc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:05.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d40728b0 con 0x7f43bc004d90 2023-03-03T13:53:05.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4155070 con 0x7f43bc003a70 2023-03-03T13:53:05.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc00ba90 con 0x7f43d40aad20 2023-03-03T13:53:05.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d2d9d700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 0x7f43bc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:05.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d259c700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43bc003a70 0x7f43bc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:05.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d1d9b700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d40aad20 0x7f43bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:05.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d2d9d700 1 -- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d40aad20 msgr2=0x7f43bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:53:05.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d2d9d700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d40aad20 0x7f43bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:05.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d2d9d700 1 -- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43bc003a70 msgr2=0x7f43bc003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:53:05.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d2d9d700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43bc003a70 0x7f43bc003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:05.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.845+0000 7f43d2d9d700 1 -- 172.21.15.7:0/3461496548 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43bc0035b0 con 0x7f43bc004d90 2023-03-03T13:53:05.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d2d9d700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 0x7f43bc005200 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00a860 tx=0x7f43bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fec1fa3ae9522d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:53:05.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc012600 con 0x7f43bc004d90 2023-03-03T13:53:05.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc012780 con 0x7f43bc004d90 2023-03-03T13:53:05.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d159a700 1 -- 172.21.15.7:0/3461496548 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc012b30 con 0x7f43bc004d90 2023-03-03T13:53:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d897a700 1 -- 172.21.15.7:0/3461496548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 msgr2=0x7f43bc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d897a700 1 --2- 172.21.15.7:0/3461496548 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 0x7f43bc005200 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00a860 tx=0x7f43bc005880 comp rx=0 tx=0).stop 2023-03-03T13:53:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d2d9d700 1 -- 172.21.15.7:0/3461496548 reap_dead start 2023-03-03T13:53:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d897a700 1 -- 172.21.15.7:0/3461496548 shutdown_connections 2023-03-03T13:53:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d897a700 1 -- 172.21.15.7:0/3461496548 >> 172.21.15.7:0/3461496548 conn(0x7f43d40c8900 msgr2=0x7f43d40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:53:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d897a700 1 -- 172.21.15.7:0/3461496548 shutdown_connections 2023-03-03T13:53:05.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.846+0000 7f43d897a700 1 -- 172.21.15.7:0/3461496548 wait complete. 2023-03-03T13:53:05.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.848+0000 7f43d897a700 1 Processor -- start 2023-03-03T13:53:05.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.848+0000 7f43d897a700 1 -- start start 2023-03-03T13:53:05.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.848+0000 7f43d897a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 0x7f43d41e9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:05.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.848+0000 7f43d897a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d414ea60 0x7f43d4156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:05.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.848+0000 7f43d897a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d4153ed0 0x7f43d4152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:05.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.848+0000 7f43d897a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d41570e0 con 0x7f43bc004d90 2023-03-03T13:53:05.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.848+0000 7f43d897a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d41ea530 con 0x7f43d4153ed0 2023-03-03T13:53:05.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.848+0000 7f43d897a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4153090 con 0x7f43d414ea60 2023-03-03T13:53:05.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 0x7f43d41e9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:05.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 0x7f43d41e9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46398/0 (socket says 172.21.15.7:46398) 2023-03-03T13:53:05.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d259c700 1 -- 172.21.15.7:0/4139451498 learned_addr learned my addr 172.21.15.7:0/4139451498 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:53:05.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d2d9d700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d4153ed0 0x7f43d4152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:05.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d1d9b700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d414ea60 0x7f43d4156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:05.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d259c700 1 -- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d414ea60 msgr2=0x7f43d4156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:05.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d259c700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d414ea60 0x7f43d4156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:05.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d259c700 1 -- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d4153ed0 msgr2=0x7f43d4152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:05.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d259c700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d4153ed0 0x7f43d4152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:05.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d259c700 1 -- 172.21.15.7:0/4139451498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d406fb70 con 0x7f43bc004d90 2023-03-03T13:53:05.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d2d9d700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d4153ed0 0x7f43d4152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:53:05.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d259c700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 0x7f43d41e9ff0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f43c8011cd0 tx=0x7f43c800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:53:05.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43c2ffd700 1 -- 172.21.15.7:0/4139451498 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c800fb30 con 0x7f43bc004d90 2023-03-03T13:53:05.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43c2ffd700 1 -- 172.21.15.7:0/4139451498 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c800fc90 con 0x7f43bc004d90 2023-03-03T13:53:05.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d897a700 1 -- 172.21.15.7:0/4139451498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d406f890 con 0x7f43bc004d90 2023-03-03T13:53:05.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43c2ffd700 1 -- 172.21.15.7:0/4139451498 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8004030 con 0x7f43bc004d90 2023-03-03T13:53:05.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.849+0000 7f43d897a700 1 -- 172.21.15.7:0/4139451498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d4155350 con 0x7f43bc004d90 2023-03-03T13:53:05.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.850+0000 7f43d897a700 1 -- 172.21.15.7:0/4139451498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d4074bb0 con 0x7f43bc004d90 2023-03-03T13:53:05.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.851+0000 7f43c2ffd700 1 -- 172.21.15.7:0/4139451498 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f43c8024070 con 0x7f43bc004d90 2023-03-03T13:53:05.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.851+0000 7f43c2ffd700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43b40682f0 0x7f43b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:53:05.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.851+0000 7f43c2ffd700 1 -- 172.21.15.7:0/4139451498 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f43c8086740 con 0x7f43bc004d90 2023-03-03T13:53:05.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.851+0000 7f43d1d9b700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43b40682f0 0x7f43b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:53:05.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.852+0000 7f43d1d9b700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43b40682f0 0x7f43b406a7a0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f43d40ac980 tx=0x7f43c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:53:05.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:05.856+0000 7f43c2ffd700 1 -- 172.21.15.7:0/4139451498 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f43c8052ae0 con 0x7f43bc004d90 2023-03-03T13:53:06.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.094+0000 7f43d897a700 1 -- 172.21.15.7:0/4139451498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43d40a9e10 con 0x7f43bc004d90 2023-03-03T13:53:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.094+0000 7f43c2ffd700 1 -- 172.21.15.7:0/4139451498 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f43c8056100 con 0x7f43bc004d90 2023-03-03T13:53:06.096 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:53:06.096 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:53:06.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 -- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43b40682f0 msgr2=0x7f43b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:06.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43b40682f0 0x7f43b406a7a0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f43d40ac980 tx=0x7f43c4008040 comp rx=0 tx=0).stop 2023-03-03T13:53:06.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 -- 172.21.15.7:0/4139451498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 msgr2=0x7f43d41e9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:53:06.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 0x7f43d41e9ff0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f43c8011cd0 tx=0x7f43c800f520 comp rx=0 tx=0).stop 2023-03-03T13:53:06.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 -- 172.21.15.7:0/4139451498 shutdown_connections 2023-03-03T13:53:06.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43d4153ed0 0x7f43d4152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43b40682f0 0x7f43b406a7a0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43bc004d90 0x7f43d41e9ff0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 --2- 172.21.15.7:0/4139451498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43d414ea60 0x7f43d4156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:53:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 -- 172.21.15.7:0/4139451498 >> 172.21.15.7:0/4139451498 conn(0x7f43d40c8900 msgr2=0x7f43d414a240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:53:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 -- 172.21.15.7:0/4139451498 shutdown_connections 2023-03-03T13:53:06.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:53:06.096+0000 7f43d897a700 1 -- 172.21.15.7:0/4139451498 wait complete. 2023-03-03T13:53:06.100 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:53:06.110 INFO:tasks.mds_thrash.fs.[b]:waiting for 114 secs before thrashing 2023-03-03T13:53:17.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:17.620 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:47.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:53:47.675 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:08.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:54:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.385+0000 7fccca51b700 1 Processor -- start 2023-03-03T13:54:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.385+0000 7fccca51b700 1 -- start start 2023-03-03T13:54:08.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccca51b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccc4151d40 0x7fccc40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccca51b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fccc40c2930 0x7fccc4157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccca51b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc406d430 con 0x7fccc4073c30 2023-03-03T13:54:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccca51b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc406c2a0 con 0x7fccc40c2930 2023-03-03T13:54:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccca51b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc41586e0 con 0x7fccc4151d40 2023-03-03T13:54:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccc37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fccc40c2930 0x7fccc4157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccc8ab8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fccc4073c30 0x7fccc4157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40434/0 (socket says 172.21.15.7:40434) 2023-03-03T13:54:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccc8ab8700 1 -- 172.21.15.7:0/2429778797 learned_addr learned my addr 172.21.15.7:0/2429778797 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:54:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccc3fff700 1 --2- 172.21.15.7:0/2429778797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccc4151d40 0x7fccc40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccc2ffd700 1 -- 172.21.15.7:0/2429778797 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009916025 0 0) 0x7fccc406d430 con 0x7fccc4073c30 2023-03-03T13:54:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccc2ffd700 1 -- 172.21.15.7:0/2429778797 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccb00035b0 con 0x7fccc4073c30 2023-03-03T13:54:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccc37fe700 1 -- 172.21.15.7:0/2429778797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccc4151d40 msgr2=0x7fccc40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:54:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccc37fe700 1 --2- 172.21.15.7:0/2429778797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccc4151d40 0x7fccc40736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccc37fe700 1 -- 172.21.15.7:0/2429778797 >> v1:172.21.15.7:6789/0 conn(0x7fccc4073c30 legacy=0x7fccc4157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.386+0000 7fccc37fe700 1 -- 172.21.15.7:0/2429778797 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc406cb00 con 0x7fccc40c2930 2023-03-03T13:54:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccc37fe700 1 --2- 172.21.15.7:0/2429778797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc40c2930 0x7fccc4157a90 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fccac009840 tx=0x7fccac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e6406ea588df49 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:54:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccc2ffd700 1 -- 172.21.15.7:0/2429778797 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac011830 con 0x7fccc40c2930 2023-03-03T13:54:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccc2ffd700 1 -- 172.21.15.7:0/2429778797 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac011990 con 0x7fccc40c2930 2023-03-03T13:54:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccc2ffd700 1 -- 172.21.15.7:0/2429778797 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac011d20 con 0x7fccc40c2930 2023-03-03T13:54:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccca51b700 1 -- 172.21.15.7:0/2429778797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc40c2930 msgr2=0x7fccc4157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccca51b700 1 --2- 172.21.15.7:0/2429778797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc40c2930 0x7fccc4157a90 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fccac009840 tx=0x7fccac009b50 comp rx=0 tx=0).stop 2023-03-03T13:54:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccca51b700 1 -- 172.21.15.7:0/2429778797 shutdown_connections 2023-03-03T13:54:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccca51b700 1 --2- 172.21.15.7:0/2429778797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccc4151d40 0x7fccc40736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:08.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccca51b700 1 --2- 172.21.15.7:0/2429778797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc40c2930 0x7fccc4157a90 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccca51b700 1 -- 172.21.15.7:0/2429778797 >> 172.21.15.7:0/2429778797 conn(0x7fccc40a94d0 msgr2=0x7fccc40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:54:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccca51b700 1 -- 172.21.15.7:0/2429778797 shutdown_connections 2023-03-03T13:54:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.387+0000 7fccca51b700 1 -- 172.21.15.7:0/2429778797 wait complete. 2023-03-03T13:54:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.389+0000 7fccca51b700 1 Processor -- start 2023-03-03T13:54:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.389+0000 7fccca51b700 1 -- start start 2023-03-03T13:54:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.389+0000 7fccca51b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc4073c30 0x7fccc4156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:08.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.389+0000 7fccca51b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccc40c2930 0x7fccc41ed540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.389+0000 7fccca51b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccc4151d40 0x7fccc41eda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.389+0000 7fccca51b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc41576c0 con 0x7fccc4073c30 2023-03-03T13:54:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.389+0000 7fccca51b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc41eaa50 con 0x7fccc4151d40 2023-03-03T13:54:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.389+0000 7fccca51b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc41ead30 con 0x7fccc40c2930 2023-03-03T13:54:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc4073c30 0x7fccc4156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc4073c30 0x7fccc4156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46456/0 (socket says 172.21.15.7:46456) 2023-03-03T13:54:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc3fff700 1 -- 172.21.15.7:0/1592600937 learned_addr learned my addr 172.21.15.7:0/1592600937 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:54:08.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc37fe700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccc40c2930 0x7fccc41ed540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc8ab8700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccc4151d40 0x7fccc41eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc3fff700 1 -- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccc40c2930 msgr2=0x7fccc41ed540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc3fff700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccc40c2930 0x7fccc41ed540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc3fff700 1 -- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccc4151d40 msgr2=0x7fccc41eda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc3fff700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccc4151d40 0x7fccc41eda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc3fff700 1 -- 172.21.15.7:0/1592600937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac0094a0 con 0x7fccc4073c30 2023-03-03T13:54:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc3fff700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc4073c30 0x7fccc4156ba0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fccb400e9b0 tx=0x7fccb400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:54:08.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc0ff9700 1 -- 172.21.15.7:0/1592600937 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4009710 con 0x7fccc4073c30 2023-03-03T13:54:08.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc0ff9700 1 -- 172.21.15.7:0/1592600937 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb400c850 con 0x7fccc4073c30 2023-03-03T13:54:08.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccca51b700 1 -- 172.21.15.7:0/1592600937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc406d0c0 con 0x7fccc4073c30 2023-03-03T13:54:08.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccc0ff9700 1 -- 172.21.15.7:0/1592600937 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb400cbc0 con 0x7fccc4073c30 2023-03-03T13:54:08.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.390+0000 7fccca51b700 1 -- 172.21.15.7:0/1592600937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc406cde0 con 0x7fccc4073c30 2023-03-03T13:54:08.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.391+0000 7fccca51b700 1 -- 172.21.15.7:0/1592600937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc406bea0 con 0x7fccc4073c30 2023-03-03T13:54:08.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.392+0000 7fccc0ff9700 1 -- 172.21.15.7:0/1592600937 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fccb4019030 con 0x7fccc4073c30 2023-03-03T13:54:08.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.392+0000 7fccc0ff9700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcca40682f0 0x7fcca406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:54:08.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.392+0000 7fccc0ff9700 1 -- 172.21.15.7:0/1592600937 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fccb4086160 con 0x7fccc4073c30 2023-03-03T13:54:08.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.392+0000 7fccc37fe700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcca40682f0 0x7fcca406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:54:08.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.393+0000 7fccc37fe700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcca40682f0 0x7fcca406a7a0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fccac000c00 tx=0x7fccac01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:54:08.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.397+0000 7fccc0ff9700 1 -- 172.21.15.7:0/1592600937 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fccb4052500 con 0x7fccc4073c30 2023-03-03T13:54:08.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.635+0000 7fccca51b700 1 -- 172.21.15.7:0/1592600937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccc4074bb0 con 0x7fccc4073c30 2023-03-03T13:54:08.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.635+0000 7fccc0ff9700 1 -- 172.21.15.7:0/1592600937 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fccb4017090 con 0x7fccc4073c30 2023-03-03T13:54:08.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:54:08.638 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:54:08.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.637+0000 7fccca51b700 1 -- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcca40682f0 msgr2=0x7fcca406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:08.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.637+0000 7fccca51b700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcca40682f0 0x7fcca406a7a0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fccac000c00 tx=0x7fccac01c040 comp rx=0 tx=0).stop 2023-03-03T13:54:08.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.637+0000 7fccca51b700 1 -- 172.21.15.7:0/1592600937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc4073c30 msgr2=0x7fccc4156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:54:08.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.637+0000 7fccca51b700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc4073c30 0x7fccc4156ba0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fccb400e9b0 tx=0x7fccb400ed70 comp rx=0 tx=0).stop 2023-03-03T13:54:08.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.637+0000 7fccca51b700 1 -- 172.21.15.7:0/1592600937 shutdown_connections 2023-03-03T13:54:08.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.637+0000 7fccca51b700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccc4151d40 0x7fccc41eda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:08.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.637+0000 7fccca51b700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcca40682f0 0x7fcca406a7a0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:08.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.637+0000 7fccca51b700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccc4073c30 0x7fccc4156ba0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:08.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.637+0000 7fccca51b700 1 --2- 172.21.15.7:0/1592600937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccc40c2930 0x7fccc41ed540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:54:08.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.637+0000 7fccca51b700 1 -- 172.21.15.7:0/1592600937 >> 172.21.15.7:0/1592600937 conn(0x7fccc40a94d0 msgr2=0x7fccc406af00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:54:08.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.638+0000 7fccca51b700 1 -- 172.21.15.7:0/1592600937 shutdown_connections 2023-03-03T13:54:08.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:54:08.638+0000 7fccca51b700 1 -- 172.21.15.7:0/1592600937 wait complete. 2023-03-03T13:54:08.641 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:54:08.651 INFO:tasks.mds_thrash.fs.[a]:waiting for 103 secs before thrashing 2023-03-03T13:54:17.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:17.763 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:47.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:54:47.826 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:00.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:55:00.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.248+0000 7f3984f4c700 1 Processor -- start 2023-03-03T13:55:00.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.248+0000 7f3984f4c700 1 -- start start 2023-03-03T13:55:00.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.249+0000 7f3984f4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39801586c0 0x7f398006d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:00.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.249+0000 7f3984f4c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f39800c1da0 0x7f398006d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:00.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.249+0000 7f3984f4c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3980154560 con 0x7f398013a0f0 2023-03-03T13:55:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.249+0000 7f3984f4c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3980154840 con 0x7f39800c1da0 2023-03-03T13:55:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.249+0000 7f3984f4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3980157be0 con 0x7f39801586c0 2023-03-03T13:55:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.249+0000 7f397e59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f398013a0f0 0x7f39800c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40482/0 (socket says 172.21.15.7:40482) 2023-03-03T13:55:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.249+0000 7f397e59c700 1 -- 172.21.15.7:0/702232530 learned_addr learned my addr 172.21.15.7:0/702232530 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:55:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397ed9d700 1 --2- 172.21.15.7:0/702232530 >> v2:172.21.15.143:3300/0 conn(0x7f39800c1da0 0x7f398006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397dd9b700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39801586c0 0x7f398006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160077255 0 0) 0x7f3980154560 con 0x7f398013a0f0 2023-03-03T13:55:00.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39680035b0 con 0x7f398013a0f0 2023-03-03T13:55:00.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 642741245 0 0) 0x7f39680035b0 con 0x7f398013a0f0 2023-03-03T13:55:00.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39801586c0 msgr2=0x7f398006d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:55:00.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39801586c0 0x7f398006d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 >> v2:172.21.15.143:3300/0 conn(0x7f39800c1da0 msgr2=0x7f398006d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:55:00.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 --2- 172.21.15.7:0/702232530 >> v2:172.21.15.143:3300/0 conn(0x7f39800c1da0 0x7f398006d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3980157ec0 con 0x7f398013a0f0 2023-03-03T13:55:00.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f3970001670 con 0x7f398013a0f0 2023-03-03T13:55:00.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 >> v1:172.21.15.7:6789/0 conn(0x7f398013a0f0 legacy=0x7f39800c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:00.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39801586c0 0x7f3968003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:00.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3968003ff0 0x7f3968004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:00.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3968004d90 0x7f3968005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:00.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3980154840 con 0x7f3968003ff0 2023-03-03T13:55:00.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3980157be0 con 0x7f3968004d90 2023-03-03T13:55:00.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396800bb70 con 0x7f39801586c0 2023-03-03T13:55:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397dd9b700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3968003ff0 0x7f3968004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397e59c700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39801586c0 0x7f3968003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.250+0000 7f397ed9d700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3968004d90 0x7f3968005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f397dd9b700 1 -- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39801586c0 msgr2=0x7f3968003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:55:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f397dd9b700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39801586c0 0x7f3968003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f397dd9b700 1 -- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3968004d90 msgr2=0x7f3968005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:55:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f397dd9b700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3968004d90 0x7f3968005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f397dd9b700 1 -- 172.21.15.7:0/702232530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39680035b0 con 0x7f3968003ff0 2023-03-03T13:55:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f397dd9b700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3968003ff0 0x7f3968004850 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f3980158ad0 tx=0x7f3968005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e8e813bc165a57 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:55:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397400e6c0 con 0x7f3968003ff0 2023-03-03T13:55:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397400e820 con 0x7f3968003ff0 2023-03-03T13:55:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f397d59a700 1 -- 172.21.15.7:0/702232530 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397400ebb0 con 0x7f3968003ff0 2023-03-03T13:55:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f3984f4c700 1 -- 172.21.15.7:0/702232530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3968003ff0 msgr2=0x7f3968004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f3984f4c700 1 --2- 172.21.15.7:0/702232530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3968003ff0 0x7f3968004850 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f3980158ad0 tx=0x7f3968005880 comp rx=0 tx=0).stop 2023-03-03T13:55:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f397ed9d700 1 -- 172.21.15.7:0/702232530 reap_dead start 2023-03-03T13:55:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f3984f4c700 1 -- 172.21.15.7:0/702232530 shutdown_connections 2023-03-03T13:55:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.251+0000 7f3984f4c700 1 -- 172.21.15.7:0/702232530 >> 172.21.15.7:0/702232530 conn(0x7f398006c110 msgr2=0x7f398006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:55:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.252+0000 7f3984f4c700 1 -- 172.21.15.7:0/702232530 shutdown_connections 2023-03-03T13:55:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.252+0000 7f3984f4c700 1 -- 172.21.15.7:0/702232530 wait complete. 2023-03-03T13:55:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.253+0000 7f3984f4c700 1 Processor -- start 2023-03-03T13:55:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.253+0000 7f3984f4c700 1 -- start start 2023-03-03T13:55:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f3984f4c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3968003ff0 0x7f3980156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f3984f4c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3980156d90 0x7f3980136740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f3984f4c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3980136c80 0x7f39800c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f3984f4c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800c5d30 con 0x7f3980156d90 2023-03-03T13:55:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f3984f4c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398013ab30 con 0x7f3980136c80 2023-03-03T13:55:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f3984f4c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800c5990 con 0x7f3968003ff0 2023-03-03T13:55:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3980156d90 0x7f3980136740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3980156d90 0x7f3980136740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46516/0 (socket says 172.21.15.7:46516) 2023-03-03T13:55:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397dd9b700 1 -- 172.21.15.7:0/3286573659 learned_addr learned my addr 172.21.15.7:0/3286573659 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:55:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397ed9d700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3980136c80 0x7f39800c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397e59c700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3968003ff0 0x7f3980156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397dd9b700 1 -- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3968003ff0 msgr2=0x7f3980156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397dd9b700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3968003ff0 0x7f3980156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397dd9b700 1 -- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3980136c80 msgr2=0x7f39800c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397dd9b700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3980136c80 0x7f39800c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397dd9b700 1 -- 172.21.15.7:0/3286573659 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398006e1a0 con 0x7f3980156d90 2023-03-03T13:55:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397ed9d700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3980136c80 0x7f39800c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:55:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397e59c700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3968003ff0 0x7f3980156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:55:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f397dd9b700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3980156d90 0x7f3980136740 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f396800bac0 tx=0x7f39740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:55:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f396effd700 1 -- 172.21.15.7:0/3286573659 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397400e6c0 con 0x7f3980156d90 2023-03-03T13:55:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f396effd700 1 -- 172.21.15.7:0/3286573659 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3974003ab0 con 0x7f3980156d90 2023-03-03T13:55:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f3984f4c700 1 -- 172.21.15.7:0/3286573659 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39801581a0 con 0x7f3980156d90 2023-03-03T13:55:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f396effd700 1 -- 172.21.15.7:0/3286573659 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974006e60 con 0x7f3980156d90 2023-03-03T13:55:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.254+0000 7f3984f4c700 1 -- 172.21.15.7:0/3286573659 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398006e330 con 0x7f3980156d90 2023-03-03T13:55:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.255+0000 7f3984f4c700 1 -- 172.21.15.7:0/3286573659 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3980074bb0 con 0x7f3980156d90 2023-03-03T13:55:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.256+0000 7f396effd700 1 -- 172.21.15.7:0/3286573659 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3974019030 con 0x7f3980156d90 2023-03-03T13:55:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.256+0000 7f396effd700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 0x7f396006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.256+0000 7f396effd700 1 -- 172.21.15.7:0/3286573659 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f39740877f0 con 0x7f3980156d90 2023-03-03T13:55:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.257+0000 7f397e59c700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 0x7f396006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.257+0000 7f397e59c700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 0x7f396006a7a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f3970000c00 tx=0x7f3970009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:55:00.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.261+0000 7f396effd700 1 -- 172.21.15.7:0/3286573659 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3974053b90 con 0x7f3980156d90 2023-03-03T13:55:00.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.498+0000 7f3984f4c700 1 -- 172.21.15.7:0/3286573659 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39800c2b90 con 0x7f3980156d90 2023-03-03T13:55:00.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.499+0000 7f396effd700 1 -- 172.21.15.7:0/3286573659 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f39740571b0 con 0x7f3980156d90 2023-03-03T13:55:00.500 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:55:00.501 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:55:00.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.500+0000 7f3984f4c700 1 -- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 msgr2=0x7f396006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:00.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.500+0000 7f3984f4c700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 0x7f396006a7a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f3970000c00 tx=0x7f3970009040 comp rx=0 tx=0).stop 2023-03-03T13:55:00.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.501+0000 7f3984f4c700 1 -- 172.21.15.7:0/3286573659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3980156d90 msgr2=0x7f3980136740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:00.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.501+0000 7f3984f4c700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3980156d90 0x7f3980136740 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f396800bac0 tx=0x7f39740047c0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.501+0000 7f3984f4c700 1 -- 172.21.15.7:0/3286573659 shutdown_connections 2023-03-03T13:55:00.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.501+0000 7f3984f4c700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3980136c80 0x7f39800c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.501+0000 7f3984f4c700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3968003ff0 0x7f3980156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.501+0000 7f3984f4c700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 0x7f396006a7a0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.501+0000 7f3984f4c700 1 --2- 172.21.15.7:0/3286573659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3980156d90 0x7f3980136740 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:00.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.501+0000 7f3984f4c700 1 -- 172.21.15.7:0/3286573659 >> 172.21.15.7:0/3286573659 conn(0x7f398006c110 msgr2=0x7f3980153040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:55:00.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.501+0000 7f3984f4c700 1 -- 172.21.15.7:0/3286573659 shutdown_connections 2023-03-03T13:55:00.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:00.501+0000 7f3984f4c700 1 -- 172.21.15.7:0/3286573659 wait complete. 2023-03-03T13:55:00.504 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:55:00.514 INFO:tasks.mds_thrash.fs.[b]:waiting for 92 secs before thrashing 2023-03-03T13:55:17.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:17.906 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:47.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:47.969 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:55:51.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:55:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.788+0000 7fbf7189e700 1 Processor -- start 2023-03-03T13:55:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.789+0000 7fbf7189e700 1 -- start start 2023-03-03T13:55:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.789+0000 7fbf7189e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbf6c0c5c60 0x7fbf6c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.789+0000 7fbf7189e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6c0c7310 con 0x7fbf6c0caa30 2023-03-03T13:55:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.789+0000 7fbf7189e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6c0bfd10 con 0x7fbf6c0c00e0 2023-03-03T13:55:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.789+0000 7fbf7189e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6c0c60a0 con 0x7fbf6c0c5c60 2023-03-03T13:55:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.789+0000 7fbf6affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbf6c0caa30 0x7fbf6c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40556/0 (socket says 172.21.15.7:40556) 2023-03-03T13:55:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.789+0000 7fbf6affd700 1 -- 172.21.15.7:0/3310550994 learned_addr learned my addr 172.21.15.7:0/3310550994 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:55:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146176102 0 0) 0x7fbf6c0c7310 con 0x7fbf6c0caa30 2023-03-03T13:55:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf6b7fe700 1 --2- 172.21.15.7:0/3310550994 >> v2:172.21.15.143:3300/0 conn(0x7fbf6c0c5c60 0x7fbf6c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf580035b0 con 0x7fbf6c0caa30 2023-03-03T13:55:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2383380933 0 0) 0x7fbf580035b0 con 0x7fbf6c0caa30 2023-03-03T13:55:51.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 >> v2:172.21.15.143:3300/0 conn(0x7fbf6c0c5c60 msgr2=0x7fbf6c154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:55:51.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 --2- 172.21.15.7:0/3310550994 >> v2:172.21.15.143:3300/0 conn(0x7fbf6c0c5c60 0x7fbf6c154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:51.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 >> v1:172.21.15.143:6789/0 conn(0x7fbf6c0c00e0 legacy=0x7fbf6c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:51.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c0bfa30 con 0x7fbf6c0caa30 2023-03-03T13:55:51.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fbf5c001670 con 0x7fbf6c0caa30 2023-03-03T13:55:51.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 >> v1:172.21.15.7:6789/0 conn(0x7fbf6c0caa30 legacy=0x7fbf6c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:51.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf58003a70 0x7fbf58003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:51.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf58004850 0x7fbf58004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:51.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58005250 0x7fbf5800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:51.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5800c160 con 0x7fbf58004850 2023-03-03T13:55:51.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5800c2d0 con 0x7fbf58003a70 2023-03-03T13:55:51.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5800c490 con 0x7fbf58005250 2023-03-03T13:55:51.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf6a7fc700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf58004850 0x7fbf58004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:51.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf6affd700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf58003a70 0x7fbf58003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:51.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.790+0000 7fbf6b7fe700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58005250 0x7fbf5800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:51.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf6a7fc700 1 -- 172.21.15.7:0/3310550994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58005250 msgr2=0x7fbf5800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:55:51.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf6a7fc700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58005250 0x7fbf5800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:51.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf6a7fc700 1 -- 172.21.15.7:0/3310550994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf58003a70 msgr2=0x7fbf58003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:55:51.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf6a7fc700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf58003a70 0x7fbf58003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:51.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf6a7fc700 1 -- 172.21.15.7:0/3310550994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf580035b0 con 0x7fbf58004850 2023-03-03T13:55:51.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf6b7fe700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58005250 0x7fbf5800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:55:51.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf6b7fe700 1 -- 172.21.15.7:0/3310550994 reap_dead start 2023-03-03T13:55:51.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf6a7fc700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf58004850 0x7fbf58004cc0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fbf540017f0 tx=0x7fbf58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1b223aa05ee746 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:55:51.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf540012e0 con 0x7fbf58004850 2023-03-03T13:55:51.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf540052f0 con 0x7fbf58004850 2023-03-03T13:55:51.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf69ffb700 1 -- 172.21.15.7:0/3310550994 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf540036d0 con 0x7fbf58004850 2023-03-03T13:55:51.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf7189e700 1 -- 172.21.15.7:0/3310550994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf58004850 msgr2=0x7fbf58004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:51.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.791+0000 7fbf7189e700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf58004850 0x7fbf58004cc0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fbf540017f0 tx=0x7fbf58005880 comp rx=0 tx=0).stop 2023-03-03T13:55:51.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.792+0000 7fbf7189e700 1 -- 172.21.15.7:0/3310550994 shutdown_connections 2023-03-03T13:55:51.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.792+0000 7fbf7189e700 1 --2- 172.21.15.7:0/3310550994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf58004850 0x7fbf58004cc0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:51.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.792+0000 7fbf7189e700 1 -- 172.21.15.7:0/3310550994 >> 172.21.15.7:0/3310550994 conn(0x7fbf6c066e40 msgr2=0x7fbf6c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:55:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.792+0000 7fbf7189e700 1 -- 172.21.15.7:0/3310550994 shutdown_connections 2023-03-03T13:55:51.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.792+0000 7fbf7189e700 1 -- 172.21.15.7:0/3310550994 wait complete. 2023-03-03T13:55:51.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.793+0000 7fbf7189e700 1 Processor -- start 2023-03-03T13:55:51.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.793+0000 7fbf7189e700 1 -- start start 2023-03-03T13:55:51.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf7189e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58004850 0x7fbf6c0a9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:51.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf7189e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c156e20 0x7fbf6c0c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:51.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf7189e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf6c155da0 0x7fbf6c156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:51.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf7189e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c154fe0 con 0x7fbf6c156e20 2023-03-03T13:55:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf7189e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c1552c0 con 0x7fbf6c155da0 2023-03-03T13:55:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf7189e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c1555a0 con 0x7fbf58004850 2023-03-03T13:55:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c156e20 0x7fbf6c0c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c156e20 0x7fbf6c0c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46590/0 (socket says 172.21.15.7:46590) 2023-03-03T13:55:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6a7fc700 1 -- 172.21.15.7:0/2952301162 learned_addr learned my addr 172.21.15.7:0/2952301162 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:55:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6affd700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58004850 0x7fbf6c0a9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6b7fe700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf6c155da0 0x7fbf6c156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:51.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6a7fc700 1 -- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58004850 msgr2=0x7fbf6c0a9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:51.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6a7fc700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58004850 0x7fbf6c0a9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:51.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6a7fc700 1 -- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf6c155da0 msgr2=0x7fbf6c156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:51.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6a7fc700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf6c155da0 0x7fbf6c156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:51.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6a7fc700 1 -- 172.21.15.7:0/2952301162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c1547f0 con 0x7fbf6c156e20 2023-03-03T13:55:51.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6affd700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58004850 0x7fbf6c0a9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:55:51.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6b7fe700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf6c155da0 0x7fbf6c156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:55:51.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf6a7fc700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c156e20 0x7fbf6c0c3e20 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fbf54000c00 tx=0x7fbf54001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:55:51.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.794+0000 7fbf63fff700 1 -- 172.21.15.7:0/2952301162 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54006b40 con 0x7fbf6c156e20 2023-03-03T13:55:51.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.795+0000 7fbf63fff700 1 -- 172.21.15.7:0/2952301162 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf54006ca0 con 0x7fbf6c156e20 2023-03-03T13:55:51.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.795+0000 7fbf7189e700 1 -- 172.21.15.7:0/2952301162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c154510 con 0x7fbf6c156e20 2023-03-03T13:55:51.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.795+0000 7fbf63fff700 1 -- 172.21.15.7:0/2952301162 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54003910 con 0x7fbf6c156e20 2023-03-03T13:55:51.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.795+0000 7fbf7189e700 1 -- 172.21.15.7:0/2952301162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c154ad0 con 0x7fbf6c156e20 2023-03-03T13:55:51.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.796+0000 7fbf7189e700 1 -- 172.21.15.7:0/2952301162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6c155bf0 con 0x7fbf6c156e20 2023-03-03T13:55:51.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.796+0000 7fbf63fff700 1 -- 172.21.15.7:0/2952301162 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbf54010050 con 0x7fbf6c156e20 2023-03-03T13:55:51.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.796+0000 7fbf63fff700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf4c0682f0 0x7fbf4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:55:51.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.796+0000 7fbf63fff700 1 -- 172.21.15.7:0/2952301162 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbf540865e0 con 0x7fbf6c156e20 2023-03-03T13:55:51.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.797+0000 7fbf6affd700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf4c0682f0 0x7fbf4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:55:51.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.797+0000 7fbf6affd700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf4c0682f0 0x7fbf4c06a7a0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c007dd0 tx=0x7fbf5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:55:51.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:51.801+0000 7fbf63fff700 1 -- 172.21.15.7:0/2952301162 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbf54052900 con 0x7fbf6c156e20 2023-03-03T13:55:52.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.040+0000 7fbf7189e700 1 -- 172.21.15.7:0/2952301162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf6c074bb0 con 0x7fbf6c156e20 2023-03-03T13:55:52.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.041+0000 7fbf63fff700 1 -- 172.21.15.7:0/2952301162 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbf54055f20 con 0x7fbf6c156e20 2023-03-03T13:55:52.043 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:55:52.043 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:55:52.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 -- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf4c0682f0 msgr2=0x7fbf4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf4c0682f0 0x7fbf4c06a7a0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c007dd0 tx=0x7fbf5c009040 comp rx=0 tx=0).stop 2023-03-03T13:55:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 -- 172.21.15.7:0/2952301162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c156e20 msgr2=0x7fbf6c0c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:55:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c156e20 0x7fbf6c0c3e20 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fbf54000c00 tx=0x7fbf54001170 comp rx=0 tx=0).stop 2023-03-03T13:55:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 -- 172.21.15.7:0/2952301162 shutdown_connections 2023-03-03T13:55:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbf6c155da0 0x7fbf6c156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:52.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf58004850 0x7fbf6c0a9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbf4c0682f0 0x7fbf4c06a7a0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 --2- 172.21.15.7:0/2952301162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf6c156e20 0x7fbf6c0c3e20 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:55:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 -- 172.21.15.7:0/2952301162 >> 172.21.15.7:0/2952301162 conn(0x7fbf6c066e40 msgr2=0x7fbf6c0c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:55:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 -- 172.21.15.7:0/2952301162 shutdown_connections 2023-03-03T13:55:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:55:52.043+0000 7fbf7189e700 1 -- 172.21.15.7:0/2952301162 wait complete. 2023-03-03T13:55:52.046 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:55:52.056 INFO:tasks.mds_thrash.fs.[a]:waiting for 102 secs before thrashing 2023-03-03T13:56:18.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:18.043 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:32.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:56:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.648+0000 7f9abc8b2700 1 Processor -- start 2023-03-03T13:56:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.648+0000 7f9abc8b2700 1 -- start start 2023-03-03T13:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9abc8b2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9ab8151e80 0x7f9ab80c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9abc8b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8155410 0x7f9ab80bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:32.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9abc8b2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab80c08d0 con 0x7f9ab80c6fe0 2023-03-03T13:56:32.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9abc8b2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab8152250 con 0x7f9ab8151e80 2023-03-03T13:56:32.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9abc8b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80c0fb0 con 0x7f9ab8155410 2023-03-03T13:56:32.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab5d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8155410 0x7f9ab80bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:32.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab659c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9ab8151e80 0x7f9ab80c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:32.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab6d9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f9ab80c6fe0 0x7f9ab80bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:44128/0 (socket says 172.21.15.7:44128) 2023-03-03T13:56:32.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab5d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8155410 0x7f9ab80bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:54698/0 (socket says 172.21.15.7:54698) 2023-03-03T13:56:32.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab6d9d700 1 -- 172.21.15.7:0/3993329198 learned_addr learned my addr 172.21.15.7:0/3993329198 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:56:32.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab5d9b700 1 -- 172.21.15.7:0/3993329198 >> v2:172.21.15.143:3300/0 conn(0x7f9ab8151e80 msgr2=0x7f9ab80c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:56:32.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab5d9b700 1 --2- 172.21.15.7:0/3993329198 >> v2:172.21.15.143:3300/0 conn(0x7f9ab8151e80 0x7f9ab80c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:32.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab5d9b700 1 -- 172.21.15.7:0/3993329198 >> v1:172.21.15.143:6789/0 conn(0x7f9ab80c6fe0 legacy=0x7f9ab80bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:32.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab559a700 1 -- 172.21.15.7:0/3993329198 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950469361 0 0) 0x7f9ab80c08d0 con 0x7f9ab80c6fe0 2023-03-03T13:56:32.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab5d9b700 1 -- 172.21.15.7:0/3993329198 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8154930 con 0x7f9ab8155410 2023-03-03T13:56:32.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.649+0000 7f9ab659c700 1 --2- 172.21.15.7:0/3993329198 >> v2:172.21.15.143:3300/0 conn(0x7f9ab8151e80 0x7f9ab80c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T13:56:32.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9ab5d9b700 1 --2- 172.21.15.7:0/3993329198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8155410 0x7f9ab80bf8e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009840 tx=0x7f9aac009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=135f460cda633c48 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:56:32.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9ab559a700 1 -- 172.21.15.7:0/3993329198 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac011830 con 0x7f9ab8155410 2023-03-03T13:56:32.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9ab559a700 1 -- 172.21.15.7:0/3993329198 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac011990 con 0x7f9ab8155410 2023-03-03T13:56:32.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9ab559a700 1 -- 172.21.15.7:0/3993329198 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac011d20 con 0x7f9ab8155410 2023-03-03T13:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9abc8b2700 1 -- 172.21.15.7:0/3993329198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8155410 msgr2=0x7f9ab80bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9abc8b2700 1 --2- 172.21.15.7:0/3993329198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8155410 0x7f9ab80bf8e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009840 tx=0x7f9aac009b50 comp rx=0 tx=0).stop 2023-03-03T13:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9abc8b2700 1 -- 172.21.15.7:0/3993329198 shutdown_connections 2023-03-03T13:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9abc8b2700 1 --2- 172.21.15.7:0/3993329198 >> v2:172.21.15.143:3300/0 conn(0x7f9ab8151e80 0x7f9ab80c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9abc8b2700 1 --2- 172.21.15.7:0/3993329198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab8155410 0x7f9ab80bf8e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:32.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9abc8b2700 1 -- 172.21.15.7:0/3993329198 >> 172.21.15.7:0/3993329198 conn(0x7f9ab8071630 msgr2=0x7f9ab806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.650+0000 7f9abc8b2700 1 -- 172.21.15.7:0/3993329198 shutdown_connections 2023-03-03T13:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.651+0000 7f9abc8b2700 1 -- 172.21.15.7:0/3993329198 wait complete. 2023-03-03T13:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.652+0000 7f9abc8b2700 1 Processor -- start 2023-03-03T13:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.652+0000 7f9abc8b2700 1 -- start start 2023-03-03T13:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.652+0000 7f9abc8b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab80c6fe0 0x7f9ab814ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.652+0000 7f9abc8b2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8151e80 0x7f9ab81edac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:32.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.652+0000 7f9abc8b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8155410 0x7f9ab81ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.652+0000 7f9abc8b2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab814f9c0 con 0x7f9ab8151e80 2023-03-03T13:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.652+0000 7f9abc8b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80c8990 con 0x7f9ab8155410 2023-03-03T13:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.652+0000 7f9abc8b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80c86f0 con 0x7f9ab80c6fe0 2023-03-03T13:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8151e80 0x7f9ab81edac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8151e80 0x7f9ab81edac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46630/0 (socket says 172.21.15.7:46630) 2023-03-03T13:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab5d9b700 1 -- 172.21.15.7:0/2139723462 learned_addr learned my addr 172.21.15.7:0/2139723462 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:56:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab659c700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab80c6fe0 0x7f9ab814ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab6d9d700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8155410 0x7f9ab81ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab5d9b700 1 -- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab80c6fe0 msgr2=0x7f9ab814ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab5d9b700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab80c6fe0 0x7f9ab814ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab5d9b700 1 -- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8155410 msgr2=0x7f9ab81ee000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab5d9b700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8155410 0x7f9ab81ee000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab5d9b700 1 -- 172.21.15.7:0/2139723462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa80094a0 con 0x7f9ab8151e80 2023-03-03T13:56:32.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9ab5d9b700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8151e80 0x7f9ab81edac0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009fd0 tx=0x7f9aac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:56:32.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9aa6ffd700 1 -- 172.21.15.7:0/2139723462 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac0036a0 con 0x7f9ab8151e80 2023-03-03T13:56:32.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9aa6ffd700 1 -- 172.21.15.7:0/2139723462 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aac01f070 con 0x7f9ab8151e80 2023-03-03T13:56:32.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9abc8b2700 1 -- 172.21.15.7:0/2139723462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aac0094a0 con 0x7f9ab8151e80 2023-03-03T13:56:32.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9aa6ffd700 1 -- 172.21.15.7:0/2139723462 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aac003a40 con 0x7f9ab8151e80 2023-03-03T13:56:32.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.653+0000 7f9abc8b2700 1 -- 172.21.15.7:0/2139723462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab8154ef0 con 0x7f9ab8151e80 2023-03-03T13:56:32.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.654+0000 7f9abc8b2700 1 -- 172.21.15.7:0/2139723462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab8154c10 con 0x7f9ab8151e80 2023-03-03T13:56:32.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.655+0000 7f9aa6ffd700 1 -- 172.21.15.7:0/2139723462 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9aac00f050 con 0x7f9ab8151e80 2023-03-03T13:56:32.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.655+0000 7f9aa6ffd700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a980682f0 0x7f9a9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:56:32.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.655+0000 7f9aa6ffd700 1 -- 172.21.15.7:0/2139723462 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9aac021070 con 0x7f9ab8151e80 2023-03-03T13:56:32.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.655+0000 7f9ab659c700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a980682f0 0x7f9a9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:56:32.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.656+0000 7f9ab659c700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a980682f0 0x7f9a9806a7a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009d00 tx=0x7f9aa8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:56:32.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.660+0000 7f9aa6ffd700 1 -- 172.21.15.7:0/2139723462 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9aac053ea0 con 0x7f9ab8151e80 2023-03-03T13:56:32.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.899+0000 7f9abc8b2700 1 -- 172.21.15.7:0/2139723462 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ab80c5220 con 0x7f9ab8151e80 2023-03-03T13:56:32.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.900+0000 7f9aa6ffd700 1 -- 172.21.15.7:0/2139723462 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9aac015090 con 0x7f9ab8151e80 2023-03-03T13:56:32.903 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:56:32.903 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:56:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 -- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a980682f0 msgr2=0x7f9a9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a980682f0 0x7f9a9806a7a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009d00 tx=0x7f9aa8008040 comp rx=0 tx=0).stop 2023-03-03T13:56:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 -- 172.21.15.7:0/2139723462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8151e80 msgr2=0x7f9ab81edac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:56:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8151e80 0x7f9ab81edac0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009fd0 tx=0x7f9aac004750 comp rx=0 tx=0).stop 2023-03-03T13:56:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 -- 172.21.15.7:0/2139723462 shutdown_connections 2023-03-03T13:56:32.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9ab8155410 0x7f9ab81ee000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ab80c6fe0 0x7f9ab814ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a980682f0 0x7f9a9806a7a0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 --2- 172.21.15.7:0/2139723462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8151e80 0x7f9ab81edac0 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:56:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 -- 172.21.15.7:0/2139723462 >> 172.21.15.7:0/2139723462 conn(0x7f9ab8071630 msgr2=0x7f9ab81528f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:56:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 -- 172.21.15.7:0/2139723462 shutdown_connections 2023-03-03T13:56:32.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:56:32.902+0000 7f9abc8b2700 1 -- 172.21.15.7:0/2139723462 wait complete. 2023-03-03T13:56:32.907 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:56:32.916 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2023-03-03T13:56:48.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:48.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:56:59.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:57:00.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.055+0000 7fae828bd700 1 Processor -- start 2023-03-03T13:57:00.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.055+0000 7fae828bd700 1 -- start start 2023-03-03T13:57:00.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.055+0000 7fae828bd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae7c0c1da0 0x7fae7c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:00.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.055+0000 7fae828bd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae7c0c3e30 0x7fae7c0c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:00.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.055+0000 7fae828bd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c0c7a60 con 0x7fae7c06d870 2023-03-03T13:57:00.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.055+0000 7fae828bd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7c0c2af0 con 0x7fae7c0c1da0 2023-03-03T13:57:00.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.055+0000 7fae828bd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7c0c3620 con 0x7fae7c0c3e30 2023-03-03T13:57:00.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae7c0c1da0 0x7fae7c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:00.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae7c0c1da0 0x7fae7c1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46654/0 (socket says 172.21.15.7:46654) 2023-03-03T13:57:00.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7b7fe700 1 -- 172.21.15.7:0/3467666360 learned_addr learned my addr 172.21.15.7:0/3467666360 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:57:00.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae80e5a700 1 --2- 172.21.15.7:0/3467666360 >> v2:172.21.15.143:3300/0 conn(0x7fae7c0c3e30 0x7fae7c0c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:00.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7b7fe700 1 -- 172.21.15.7:0/3467666360 >> v2:172.21.15.143:3300/0 conn(0x7fae7c0c3e30 msgr2=0x7fae7c0c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:57:00.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7b7fe700 1 --2- 172.21.15.7:0/3467666360 >> v2:172.21.15.143:3300/0 conn(0x7fae7c0c3e30 0x7fae7c0c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7b7fe700 1 -- 172.21.15.7:0/3467666360 >> v1:172.21.15.143:6789/0 conn(0x7fae7c06d870 legacy=0x7fae7c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:00.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7b7fe700 1 -- 172.21.15.7:0/3467666360 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c1545e0 con 0x7fae7c0c1da0 2023-03-03T13:57:00.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7b7fe700 1 --2- 172.21.15.7:0/3467666360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c1da0 0x7fae7c1550c0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fae7c073830 tx=0x7fae70009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bcc420a4ed7372c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:00.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7affd700 1 -- 172.21.15.7:0/3467666360 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae70011830 con 0x7fae7c0c1da0 2023-03-03T13:57:00.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7affd700 1 -- 172.21.15.7:0/3467666360 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae70011990 con 0x7fae7c0c1da0 2023-03-03T13:57:00.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.056+0000 7fae7affd700 1 -- 172.21.15.7:0/3467666360 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae70011d20 con 0x7fae7c0c1da0 2023-03-03T13:57:00.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.057+0000 7fae828bd700 1 -- 172.21.15.7:0/3467666360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c1da0 msgr2=0x7fae7c1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:00.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.057+0000 7fae828bd700 1 --2- 172.21.15.7:0/3467666360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c1da0 0x7fae7c1550c0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fae7c073830 tx=0x7fae70009bd0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.057+0000 7fae828bd700 1 -- 172.21.15.7:0/3467666360 shutdown_connections 2023-03-03T13:57:00.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.057+0000 7fae828bd700 1 --2- 172.21.15.7:0/3467666360 >> v2:172.21.15.143:3300/0 conn(0x7fae7c0c3e30 0x7fae7c0c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.057+0000 7fae828bd700 1 --2- 172.21.15.7:0/3467666360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c1da0 0x7fae7c1550c0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.057+0000 7fae828bd700 1 -- 172.21.15.7:0/3467666360 >> 172.21.15.7:0/3467666360 conn(0x7fae7c06c110 msgr2=0x7fae7c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:57:00.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.057+0000 7fae828bd700 1 -- 172.21.15.7:0/3467666360 shutdown_connections 2023-03-03T13:57:00.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.057+0000 7fae828bd700 1 -- 172.21.15.7:0/3467666360 wait complete. 2023-03-03T13:57:00.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.058+0000 7fae828bd700 1 Processor -- start 2023-03-03T13:57:00.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.058+0000 7fae828bd700 1 -- start start 2023-03-03T13:57:00.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae828bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae7c06d870 0x7fae7c1e9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:00.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae828bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae7c0c1da0 0x7fae7c1e9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:00.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae828bd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c3e30 0x7fae7c142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:00.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae828bd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c0c8180 con 0x7fae7c0c3e30 2023-03-03T13:57:00.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae828bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c1ea480 con 0x7fae7c0c1da0 2023-03-03T13:57:00.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae828bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c14d3e0 con 0x7fae7c06d870 2023-03-03T13:57:00.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae80e5a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c3e30 0x7fae7c142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:00.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae80e5a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c3e30 0x7fae7c142800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46662/0 (socket says 172.21.15.7:46662) 2023-03-03T13:57:00.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae80e5a700 1 -- 172.21.15.7:0/4082408372 learned_addr learned my addr 172.21.15.7:0/4082408372 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:57:00.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae7b7fe700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae7c0c1da0 0x7fae7c1e9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:00.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.059+0000 7fae7bfff700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae7c06d870 0x7fae7c1e9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:00.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae80e5a700 1 -- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae7c06d870 msgr2=0x7fae7c1e9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:00.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae80e5a700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae7c06d870 0x7fae7c1e9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae80e5a700 1 -- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae7c0c1da0 msgr2=0x7fae7c1e9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:00.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae80e5a700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae7c0c1da0 0x7fae7c1e9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae80e5a700 1 -- 172.21.15.7:0/4082408372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae70009550 con 0x7fae7c0c3e30 2023-03-03T13:57:00.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae7b7fe700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae7c0c1da0 0x7fae7c1e9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:57:00.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae80e5a700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c3e30 0x7fae7c142800 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fae7400efc0 tx=0x7fae7400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:00.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae78ff9700 1 -- 172.21.15.7:0/4082408372 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74009ba0 con 0x7fae7c0c3e30 2023-03-03T13:57:00.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae78ff9700 1 -- 172.21.15.7:0/4082408372 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7400f040 con 0x7fae7c0c3e30 2023-03-03T13:57:00.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae78ff9700 1 -- 172.21.15.7:0/4082408372 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74014480 con 0x7fae7c0c3e30 2023-03-03T13:57:00.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae828bd700 1 -- 172.21.15.7:0/4082408372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c154ba0 con 0x7fae7c0c3e30 2023-03-03T13:57:00.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.060+0000 7fae828bd700 1 -- 172.21.15.7:0/4082408372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c1548c0 con 0x7fae7c0c3e30 2023-03-03T13:57:00.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.061+0000 7fae828bd700 1 -- 172.21.15.7:0/4082408372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7c06dc40 con 0x7fae7c0c3e30 2023-03-03T13:57:00.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.061+0000 7fae78ff9700 1 -- 172.21.15.7:0/4082408372 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae740145e0 con 0x7fae7c0c3e30 2023-03-03T13:57:00.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.062+0000 7fae78ff9700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae680682f0 0x7fae6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:00.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.062+0000 7fae78ff9700 1 -- 172.21.15.7:0/4082408372 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fae74086b50 con 0x7fae7c0c3e30 2023-03-03T13:57:00.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.062+0000 7fae7bfff700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae680682f0 0x7fae6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:00.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.062+0000 7fae7bfff700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae680682f0 0x7fae6806a7a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fae6c000f60 tx=0x7fae6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:00.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.066+0000 7fae78ff9700 1 -- 172.21.15.7:0/4082408372 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae74052ef0 con 0x7fae7c0c3e30 2023-03-03T13:57:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.306+0000 7fae828bd700 1 -- 172.21.15.7:0/4082408372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae7c074bb0 con 0x7fae7c0c3e30 2023-03-03T13:57:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.307+0000 7fae78ff9700 1 -- 172.21.15.7:0/4082408372 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fae74056510 con 0x7fae7c0c3e30 2023-03-03T13:57:00.309 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:57:00.309 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:57:00.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 -- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae680682f0 msgr2=0x7fae6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:00.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae680682f0 0x7fae6806a7a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fae6c000f60 tx=0x7fae6c009040 comp rx=0 tx=0).stop 2023-03-03T13:57:00.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 -- 172.21.15.7:0/4082408372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c3e30 msgr2=0x7fae7c142800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:00.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c3e30 0x7fae7c142800 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fae7400efc0 tx=0x7fae7400c3a0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 -- 172.21.15.7:0/4082408372 shutdown_connections 2023-03-03T13:57:00.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae680682f0 0x7fae6806a7a0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae7c0c3e30 0x7fae7c142800 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae7c06d870 0x7fae7c1e9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 --2- 172.21.15.7:0/4082408372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae7c0c1da0 0x7fae7c1e9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:00.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 -- 172.21.15.7:0/4082408372 >> 172.21.15.7:0/4082408372 conn(0x7fae7c06c110 msgr2=0x7fae7c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:57:00.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 -- 172.21.15.7:0/4082408372 shutdown_connections 2023-03-03T13:57:00.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:00.309+0000 7fae828bd700 1 -- 172.21.15.7:0/4082408372 wait complete. 2023-03-03T13:57:00.312 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:57:00.322 INFO:tasks.mds_thrash.fs.[b]:waiting for 20 secs before thrashing 2023-03-03T13:57:18.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:18.179 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:20.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:57:20.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.460+0000 7fde511ab700 1 Processor -- start 2023-03-03T13:57:20.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.460+0000 7fde511ab700 1 -- start start 2023-03-03T13:57:20.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde511ab700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde4c06dd60 0x7fde4c0c1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:20.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde511ab700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde4c154ea0 0x7fde4c06bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:20.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde511ab700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c0c3390 con 0x7fde4c06e130 2023-03-03T13:57:20.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde511ab700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4c0c2dd0 con 0x7fde4c154ea0 2023-03-03T13:57:20.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde511ab700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde4c0c30b0 con 0x7fde4c06dd60 2023-03-03T13:57:20.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde4b59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde4c154ea0 0x7fde4c06bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:20.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde4b59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde4c154ea0 0x7fde4c06bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46684/0 (socket says 172.21.15.7:46684) 2023-03-03T13:57:20.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde4b59e700 1 -- 172.21.15.7:0/2302861232 learned_addr learned my addr 172.21.15.7:0/2302861232 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:57:20.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde4ad9d700 1 --2- 172.21.15.7:0/2302861232 >> v2:172.21.15.143:3300/0 conn(0x7fde4c06dd60 0x7fde4c0c1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:20.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde4b59e700 1 -- 172.21.15.7:0/2302861232 >> v2:172.21.15.143:3300/0 conn(0x7fde4c06dd60 msgr2=0x7fde4c0c1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:57:20.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde4b59e700 1 --2- 172.21.15.7:0/2302861232 >> v2:172.21.15.143:3300/0 conn(0x7fde4c06dd60 0x7fde4c0c1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde4b59e700 1 -- 172.21.15.7:0/2302861232 >> v1:172.21.15.143:6789/0 conn(0x7fde4c06e130 legacy=0x7fde4c154790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde4b59e700 1 -- 172.21.15.7:0/2302861232 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c0c15d0 con 0x7fde4c154ea0 2023-03-03T13:57:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde4ad9d700 1 --2- 172.21.15.7:0/2302861232 >> v2:172.21.15.143:3300/0 conn(0x7fde4c06dd60 0x7fde4c0c1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:57:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.461+0000 7fde4b59e700 1 --2- 172.21.15.7:0/2302861232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c154ea0 0x7fde4c06bae0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fde44009e50 tx=0x7fde4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661b055ed79099cb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde4a59c700 1 -- 172.21.15.7:0/2302861232 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4400e070 con 0x7fde4c154ea0 2023-03-03T13:57:20.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde4a59c700 1 -- 172.21.15.7:0/2302861232 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4400b980 con 0x7fde4c154ea0 2023-03-03T13:57:20.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde4a59c700 1 -- 172.21.15.7:0/2302861232 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4400bd30 con 0x7fde4c154ea0 2023-03-03T13:57:20.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde511ab700 1 -- 172.21.15.7:0/2302861232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c154ea0 msgr2=0x7fde4c06bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:20.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde511ab700 1 --2- 172.21.15.7:0/2302861232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c154ea0 0x7fde4c06bae0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fde44009e50 tx=0x7fde4400b330 comp rx=0 tx=0).stop 2023-03-03T13:57:20.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde511ab700 1 -- 172.21.15.7:0/2302861232 shutdown_connections 2023-03-03T13:57:20.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde511ab700 1 --2- 172.21.15.7:0/2302861232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c154ea0 0x7fde4c06bae0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:20.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde511ab700 1 --2- 172.21.15.7:0/2302861232 >> v2:172.21.15.143:3300/0 conn(0x7fde4c06dd60 0x7fde4c0c1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:20.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde511ab700 1 -- 172.21.15.7:0/2302861232 >> 172.21.15.7:0/2302861232 conn(0x7fde4c06af40 msgr2=0x7fde4c0c6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:57:20.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde511ab700 1 -- 172.21.15.7:0/2302861232 shutdown_connections 2023-03-03T13:57:20.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.462+0000 7fde511ab700 1 -- 172.21.15.7:0/2302861232 wait complete. 2023-03-03T13:57:20.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.464+0000 7fde511ab700 1 Processor -- start 2023-03-03T13:57:20.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.464+0000 7fde511ab700 1 -- start start 2023-03-03T13:57:20.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde511ab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde4c06dd60 0x7fde4c151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:20.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde511ab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c06e130 0x7fde4c1e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde511ab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde4c154ea0 0x7fde4c1e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde511ab700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c0ac8d0 con 0x7fde4c06e130 2023-03-03T13:57:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde511ab700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c0ac030 con 0x7fde4c06dd60 2023-03-03T13:57:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde511ab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4c0ac310 con 0x7fde4c154ea0 2023-03-03T13:57:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde4259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c06e130 0x7fde4c1e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde4259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c06e130 0x7fde4c1e5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46688/0 (socket says 172.21.15.7:46688) 2023-03-03T13:57:20.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde4259c700 1 -- 172.21.15.7:0/4051386616 learned_addr learned my addr 172.21.15.7:0/4051386616 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:57:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde4ad9d700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde4c06dd60 0x7fde4c151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.465+0000 7fde4b59e700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde4c154ea0 0x7fde4c1e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde4259c700 1 -- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde4c154ea0 msgr2=0x7fde4c1e8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde4259c700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde4c154ea0 0x7fde4c1e8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde4259c700 1 -- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde4c06dd60 msgr2=0x7fde4c151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde4259c700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde4c06dd60 0x7fde4c151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:20.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde4259c700 1 -- 172.21.15.7:0/4051386616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde44009b30 con 0x7fde4c06e130 2023-03-03T13:57:20.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde4259c700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c06e130 0x7fde4c1e5db0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fde3400c500 tx=0x7fde3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:20.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde43fff700 1 -- 172.21.15.7:0/4051386616 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde340052d0 con 0x7fde4c06e130 2023-03-03T13:57:20.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde43fff700 1 -- 172.21.15.7:0/4051386616 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde34005430 con 0x7fde4c06e130 2023-03-03T13:57:20.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde511ab700 1 -- 172.21.15.7:0/4051386616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c0ba240 con 0x7fde4c06e130 2023-03-03T13:57:20.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde43fff700 1 -- 172.21.15.7:0/4051386616 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde34006aa0 con 0x7fde4c06e130 2023-03-03T13:57:20.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.466+0000 7fde511ab700 1 -- 172.21.15.7:0/4051386616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4c0c18b0 con 0x7fde4c06e130 2023-03-03T13:57:20.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.467+0000 7fde511ab700 1 -- 172.21.15.7:0/4051386616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4c0ba3d0 con 0x7fde4c06e130 2023-03-03T13:57:20.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.468+0000 7fde43fff700 1 -- 172.21.15.7:0/4051386616 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fde34010050 con 0x7fde4c06e130 2023-03-03T13:57:20.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.468+0000 7fde43fff700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde2c0682f0 0x7fde2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:20.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.468+0000 7fde43fff700 1 -- 172.21.15.7:0/4051386616 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fde340852d0 con 0x7fde4c06e130 2023-03-03T13:57:20.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.468+0000 7fde4ad9d700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde2c0682f0 0x7fde2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:20.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.469+0000 7fde4ad9d700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde2c0682f0 0x7fde2c06a7a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fde3c009930 tx=0x7fde3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:20.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.472+0000 7fde43fff700 1 -- 172.21.15.7:0/4051386616 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fde3404a5e0 con 0x7fde4c06e130 2023-03-03T13:57:20.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.712+0000 7fde511ab700 1 -- 172.21.15.7:0/4051386616 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde4c074bb0 con 0x7fde4c06e130 2023-03-03T13:57:20.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.712+0000 7fde43fff700 1 -- 172.21.15.7:0/4051386616 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fde34051670 con 0x7fde4c06e130 2023-03-03T13:57:20.714 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:57:20.714 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:57:20.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 -- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde2c0682f0 msgr2=0x7fde2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:20.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde2c0682f0 0x7fde2c06a7a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fde3c009930 tx=0x7fde3c008040 comp rx=0 tx=0).stop 2023-03-03T13:57:20.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 -- 172.21.15.7:0/4051386616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c06e130 msgr2=0x7fde4c1e5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:20.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c06e130 0x7fde4c1e5db0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fde3400c500 tx=0x7fde3400c8c0 comp rx=0 tx=0).stop 2023-03-03T13:57:20.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 -- 172.21.15.7:0/4051386616 shutdown_connections 2023-03-03T13:57:20.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde4c154ea0 0x7fde4c1e8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:20.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde4c06dd60 0x7fde4c151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:20.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde2c0682f0 0x7fde2c06a7a0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:20.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 --2- 172.21.15.7:0/4051386616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde4c06e130 0x7fde4c1e5db0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:20.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 -- 172.21.15.7:0/4051386616 >> 172.21.15.7:0/4051386616 conn(0x7fde4c06af40 msgr2=0x7fde4c152b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:57:20.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 -- 172.21.15.7:0/4051386616 shutdown_connections 2023-03-03T13:57:20.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:20.714+0000 7fde511ab700 1 -- 172.21.15.7:0/4051386616 wait complete. 2023-03-03T13:57:20.718 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:57:20.728 INFO:tasks.mds_thrash.fs.[b]:waiting for 104 secs before thrashing 2023-03-03T13:57:34.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:57:34.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.195+0000 7f3dc5339700 1 Processor -- start 2023-03-03T13:57:34.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.195+0000 7f3dc5339700 1 -- start start 2023-03-03T13:57:34.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.196+0000 7f3dc5339700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3dc014b0d0 0x7f3dc0071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.196+0000 7f3dc5339700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3dc0071d40 0x7f3dc0158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.196+0000 7f3dc5339700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc00a9c20 con 0x7f3dc00aad20 2023-03-03T13:57:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.196+0000 7f3dc5339700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc0154f10 con 0x7f3dc0071d40 2023-03-03T13:57:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.196+0000 7f3dc5339700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc00ac3c0 con 0x7f3dc014b0d0 2023-03-03T13:57:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.196+0000 7f3dbf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3dc0071d40 0x7f3dc0158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.196+0000 7f3dbf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3dc0071d40 0x7f3dc0158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46710/0 (socket says 172.21.15.7:46710) 2023-03-03T13:57:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.196+0000 7f3dbf7fe700 1 -- 172.21.15.7:0/2305658985 learned_addr learned my addr 172.21.15.7:0/2305658985 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:57:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.196+0000 7f3dbeffd700 1 --2- 172.21.15.7:0/2305658985 >> v2:172.21.15.143:3300/0 conn(0x7f3dc014b0d0 0x7f3dc0071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dbf7fe700 1 -- 172.21.15.7:0/2305658985 >> v2:172.21.15.143:3300/0 conn(0x7f3dc014b0d0 msgr2=0x7f3dc0071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:57:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dbf7fe700 1 --2- 172.21.15.7:0/2305658985 >> v2:172.21.15.143:3300/0 conn(0x7f3dc014b0d0 0x7f3dc0071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dbf7fe700 1 -- 172.21.15.7:0/2305658985 >> v1:172.21.15.143:6789/0 conn(0x7f3dc00aad20 legacy=0x7f3dc0157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:57:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dbf7fe700 1 -- 172.21.15.7:0/2305658985 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc006fc70 con 0x7f3dc0071d40 2023-03-03T13:57:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dbf7fe700 1 --2- 172.21.15.7:0/2305658985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc0071d40 0x7f3dc0158500 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0070930 tx=0x7f3db800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d645f5d0bd066db2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dbdffb700 1 -- 172.21.15.7:0/2305658985 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db800e070 con 0x7f3dc0071d40 2023-03-03T13:57:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dbdffb700 1 -- 172.21.15.7:0/2305658985 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db8003a20 con 0x7f3dc0071d40 2023-03-03T13:57:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dbdffb700 1 -- 172.21.15.7:0/2305658985 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8003dd0 con 0x7f3dc0071d40 2023-03-03T13:57:34.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dc5339700 1 -- 172.21.15.7:0/2305658985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc0071d40 msgr2=0x7f3dc0158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:34.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dc5339700 1 --2- 172.21.15.7:0/2305658985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc0071d40 0x7f3dc0158500 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0070930 tx=0x7f3db800b330 comp rx=0 tx=0).stop 2023-03-03T13:57:34.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dc5339700 1 -- 172.21.15.7:0/2305658985 shutdown_connections 2023-03-03T13:57:34.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dc5339700 1 --2- 172.21.15.7:0/2305658985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc0071d40 0x7f3dc0158500 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:34.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dc5339700 1 --2- 172.21.15.7:0/2305658985 >> v2:172.21.15.143:3300/0 conn(0x7f3dc014b0d0 0x7f3dc0071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:34.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.197+0000 7f3dc5339700 1 -- 172.21.15.7:0/2305658985 >> 172.21.15.7:0/2305658985 conn(0x7f3dc00c8900 msgr2=0x7f3dc00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:57:34.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.198+0000 7f3dc5339700 1 -- 172.21.15.7:0/2305658985 shutdown_connections 2023-03-03T13:57:34.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.198+0000 7f3dc5339700 1 -- 172.21.15.7:0/2305658985 wait complete. 2023-03-03T13:57:34.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.199+0000 7f3dc5339700 1 Processor -- start 2023-03-03T13:57:34.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.199+0000 7f3dc5339700 1 -- start start 2023-03-03T13:57:34.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dc5339700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dc0071d40 0x7f3dc0156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:34.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dc5339700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc00aad20 0x7f3dc0146580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:34.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dc5339700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc014b0d0 0x7f3dc0146ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:34.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dc5339700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc01431d0 con 0x7f3dc014b0d0 2023-03-03T13:57:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dc5339700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc0151bb0 con 0x7f3dc00aad20 2023-03-03T13:57:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dc5339700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc0156f70 con 0x7f3dc0071d40 2023-03-03T13:57:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc014b0d0 0x7f3dc0146ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc014b0d0 0x7f3dc0146ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46716/0 (socket says 172.21.15.7:46716) 2023-03-03T13:57:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbf7fe700 1 -- 172.21.15.7:0/1850030441 learned_addr learned my addr 172.21.15.7:0/1850030441 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:57:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbe7fc700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc00aad20 0x7f3dc0146580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:34.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbeffd700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dc0071d40 0x7f3dc0156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbf7fe700 1 -- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dc0071d40 msgr2=0x7f3dc0156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbf7fe700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dc0071d40 0x7f3dc0156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbf7fe700 1 -- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc00aad20 msgr2=0x7f3dc0146580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbf7fe700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc00aad20 0x7f3dc0146580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbf7fe700 1 -- 172.21.15.7:0/1850030441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db8009b30 con 0x7f3dc014b0d0 2023-03-03T13:57:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbe7fc700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc00aad20 0x7f3dc0146580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:57:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dbf7fe700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc014b0d0 0x7f3dc0146ac0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f3db800b370 tx=0x7f3db8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:34.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3da77fe700 1 -- 172.21.15.7:0/1850030441 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8004240 con 0x7f3dc014b0d0 2023-03-03T13:57:34.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3da77fe700 1 -- 172.21.15.7:0/1850030441 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db80043c0 con 0x7f3dc014b0d0 2023-03-03T13:57:34.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3dc5339700 1 -- 172.21.15.7:0/1850030441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc006f6b0 con 0x7f3dc014b0d0 2023-03-03T13:57:34.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.200+0000 7f3da77fe700 1 -- 172.21.15.7:0/1850030441 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db8004770 con 0x7f3dc014b0d0 2023-03-03T13:57:34.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.201+0000 7f3dc5339700 1 -- 172.21.15.7:0/1850030441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc0155160 con 0x7f3dc014b0d0 2023-03-03T13:57:34.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.201+0000 7f3dc5339700 1 -- 172.21.15.7:0/1850030441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc006f990 con 0x7f3dc014b0d0 2023-03-03T13:57:34.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.203+0000 7f3da77fe700 1 -- 172.21.15.7:0/1850030441 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3db802e430 con 0x7f3dc014b0d0 2023-03-03T13:57:34.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.203+0000 7f3da77fe700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3dac0682f0 0x7f3dac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:57:34.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.203+0000 7f3da77fe700 1 -- 172.21.15.7:0/1850030441 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3db8087770 con 0x7f3dc014b0d0 2023-03-03T13:57:34.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.203+0000 7f3dbeffd700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3dac0682f0 0x7f3dac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:57:34.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.204+0000 7f3dbeffd700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3dac0682f0 0x7f3dac06a7a0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009950 tx=0x7f3db4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:57:34.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.207+0000 7f3da77fe700 1 -- 172.21.15.7:0/1850030441 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3db8053b50 con 0x7f3dc014b0d0 2023-03-03T13:57:34.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.444+0000 7f3dc5339700 1 -- 172.21.15.7:0/1850030441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3dc0143310 con 0x7f3dc014b0d0 2023-03-03T13:57:34.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.445+0000 7f3da77fe700 1 -- 172.21.15.7:0/1850030441 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3dc0143310 con 0x7f3dc014b0d0 2023-03-03T13:57:34.447 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:57:34.447 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:57:34.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 -- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3dac0682f0 msgr2=0x7f3dac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:34.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3dac0682f0 0x7f3dac06a7a0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009950 tx=0x7f3db4008040 comp rx=0 tx=0).stop 2023-03-03T13:57:34.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 -- 172.21.15.7:0/1850030441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc014b0d0 msgr2=0x7f3dc0146ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:57:34.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc014b0d0 0x7f3dc0146ac0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f3db800b370 tx=0x7f3db8004ad0 comp rx=0 tx=0).stop 2023-03-03T13:57:34.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 -- 172.21.15.7:0/1850030441 shutdown_connections 2023-03-03T13:57:34.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3dac0682f0 0x7f3dac06a7a0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:34.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3dc014b0d0 0x7f3dc0146ac0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:34.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3dc0071d40 0x7f3dc0156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:34.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 --2- 172.21.15.7:0/1850030441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3dc00aad20 0x7f3dc0146580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:57:34.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 -- 172.21.15.7:0/1850030441 >> 172.21.15.7:0/1850030441 conn(0x7f3dc00c8900 msgr2=0x7f3dc00ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:57:34.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 -- 172.21.15.7:0/1850030441 shutdown_connections 2023-03-03T13:57:34.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:57:34.447+0000 7f3dc5339700 1 -- 172.21.15.7:0/1850030441 wait complete. 2023-03-03T13:57:34.450 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:57:34.460 INFO:tasks.mds_thrash.fs.[a]:waiting for 114 secs before thrashing 2023-03-03T13:57:48.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:57:48.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:18.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:18.326 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:48.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:58:48.397 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:04.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:59:04.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.866+0000 7fc1798ad700 1 Processor -- start 2023-03-03T13:59:04.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.866+0000 7fc1798ad700 1 -- start start 2023-03-03T13:59:04.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.867+0000 7fc1798ad700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1740c43d0 0x7fc1740c4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:04.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.867+0000 7fc1798ad700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc174154590 0x7fc1741549b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:04.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.867+0000 7fc1798ad700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc174072920 con 0x7fc17414e560 2023-03-03T13:59:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.867+0000 7fc1798ad700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc174072380 con 0x7fc1740c43d0 2023-03-03T13:59:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.867+0000 7fc1798ad700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17406bef0 con 0x7fc174154590 2023-03-03T13:59:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.867+0000 7fc1727fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1740c43d0 0x7fc1740c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.867+0000 7fc1727fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1740c43d0 0x7fc1740c4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46786/0 (socket says 172.21.15.7:46786) 2023-03-03T13:59:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.867+0000 7fc1727fc700 1 -- 172.21.15.7:0/3861999229 learned_addr learned my addr 172.21.15.7:0/3861999229 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:59:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.868+0000 7fc1737fe700 1 --2- 172.21.15.7:0/3861999229 >> v2:172.21.15.143:3300/0 conn(0x7fc174154590 0x7fc1741549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:04.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.868+0000 7fc1727fc700 1 -- 172.21.15.7:0/3861999229 >> v2:172.21.15.143:3300/0 conn(0x7fc174154590 msgr2=0x7fc1741549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:59:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.868+0000 7fc1727fc700 1 --2- 172.21.15.7:0/3861999229 >> v2:172.21.15.143:3300/0 conn(0x7fc174154590 0x7fc1741549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.868+0000 7fc1727fc700 1 -- 172.21.15.7:0/3861999229 >> v1:172.21.15.143:6789/0 conn(0x7fc17414e560 legacy=0x7fc174071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:59:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.868+0000 7fc1727fc700 1 -- 172.21.15.7:0/3861999229 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17406b910 con 0x7fc1740c43d0 2023-03-03T13:59:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.868+0000 7fc1737fe700 1 --2- 172.21.15.7:0/3861999229 >> v2:172.21.15.143:3300/0 conn(0x7fc174154590 0x7fc1741549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T13:59:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.868+0000 7fc1727fc700 1 --2- 172.21.15.7:0/3861999229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 0x7fc1740c4850 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fc17406f510 tx=0x7fc15c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a16c26850101888 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:59:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.868+0000 7fc171ffb700 1 -- 172.21.15.7:0/3861999229 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c004230 con 0x7fc1740c43d0 2023-03-03T13:59:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.868+0000 7fc171ffb700 1 -- 172.21.15.7:0/3861999229 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c004390 con 0x7fc1740c43d0 2023-03-03T13:59:04.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.868+0000 7fc171ffb700 1 -- 172.21.15.7:0/3861999229 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0036a0 con 0x7fc1740c43d0 2023-03-03T13:59:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.869+0000 7fc1798ad700 1 -- 172.21.15.7:0/3861999229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 msgr2=0x7fc1740c4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.869+0000 7fc1798ad700 1 --2- 172.21.15.7:0/3861999229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 0x7fc1740c4850 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fc17406f510 tx=0x7fc15c009bd0 comp rx=0 tx=0).stop 2023-03-03T13:59:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.869+0000 7fc1798ad700 1 -- 172.21.15.7:0/3861999229 shutdown_connections 2023-03-03T13:59:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.869+0000 7fc1798ad700 1 --2- 172.21.15.7:0/3861999229 >> v2:172.21.15.143:3300/0 conn(0x7fc174154590 0x7fc1741549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.869+0000 7fc1798ad700 1 --2- 172.21.15.7:0/3861999229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 0x7fc1740c4850 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.869+0000 7fc1798ad700 1 -- 172.21.15.7:0/3861999229 >> 172.21.15.7:0/3861999229 conn(0x7fc1740736f0 msgr2=0x7fc174073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:59:04.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.869+0000 7fc1798ad700 1 -- 172.21.15.7:0/3861999229 shutdown_connections 2023-03-03T13:59:04.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.869+0000 7fc1798ad700 1 -- 172.21.15.7:0/3861999229 wait complete. 2023-03-03T13:59:04.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.870+0000 7fc1798ad700 1 Processor -- start 2023-03-03T13:59:04.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.870+0000 7fc1798ad700 1 -- start start 2023-03-03T13:59:04.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc1798ad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 0x7fc1740c6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:04.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc1798ad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc17414e560 0x7fc174149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:04.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc1798ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc174154590 0x7fc1741e9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:04.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc1798ad700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc174147ef0 con 0x7fc1740c43d0 2023-03-03T13:59:04.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc1798ad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17414ecb0 con 0x7fc17414e560 2023-03-03T13:59:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc1798ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1741e97b0 con 0x7fc174154590 2023-03-03T13:59:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc172ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 0x7fc1740c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc172ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 0x7fc1740c6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46790/0 (socket says 172.21.15.7:46790) 2023-03-03T13:59:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc172ffd700 1 -- 172.21.15.7:0/1552831743 learned_addr learned my addr 172.21.15.7:0/1552831743 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:59:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc1727fc700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc17414e560 0x7fc174149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc1737fe700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc174154590 0x7fc1741e9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:04.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc172ffd700 1 -- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc174154590 msgr2=0x7fc1741e9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc172ffd700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc174154590 0x7fc1741e9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc172ffd700 1 -- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc17414e560 msgr2=0x7fc174149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc172ffd700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc17414e560 0x7fc174149650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc172ffd700 1 -- 172.21.15.7:0/1552831743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c009550 con 0x7fc1740c43d0 2023-03-03T13:59:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc172ffd700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 0x7fc1740c6330 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fc16400c4d0 tx=0x7fc16400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:59:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.871+0000 7fc16bfff700 1 -- 172.21.15.7:0/1552831743 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc164011790 con 0x7fc1740c43d0 2023-03-03T13:59:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.872+0000 7fc16bfff700 1 -- 172.21.15.7:0/1552831743 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1640118f0 con 0x7fc1740c43d0 2023-03-03T13:59:04.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.872+0000 7fc1798ad700 1 -- 172.21.15.7:0/1552831743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17406bb80 con 0x7fc1740c43d0 2023-03-03T13:59:04.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.872+0000 7fc16bfff700 1 -- 172.21.15.7:0/1552831743 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc164011ca0 con 0x7fc1740c43d0 2023-03-03T13:59:04.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.872+0000 7fc1798ad700 1 -- 172.21.15.7:0/1552831743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1740aacc0 con 0x7fc1740c43d0 2023-03-03T13:59:04.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.872+0000 7fc1798ad700 1 -- 172.21.15.7:0/1552831743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1741e9de0 con 0x7fc1740c43d0 2023-03-03T13:59:04.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.873+0000 7fc16bfff700 1 -- 172.21.15.7:0/1552831743 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc164010050 con 0x7fc1740c43d0 2023-03-03T13:59:04.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.873+0000 7fc16bfff700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1540682f0 0x7fc15406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:04.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.873+0000 7fc16bfff700 1 -- 172.21.15.7:0/1552831743 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc164085f20 con 0x7fc1740c43d0 2023-03-03T13:59:04.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.874+0000 7fc1727fc700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1540682f0 0x7fc15406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:04.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.874+0000 7fc1727fc700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1540682f0 0x7fc15406a7a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fc15c009130 tx=0x7fc15c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:59:04.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:04.878+0000 7fc16bfff700 1 -- 172.21.15.7:0/1552831743 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc1640521d0 con 0x7fc1740c43d0 2023-03-03T13:59:05.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.115+0000 7fc1798ad700 1 -- 172.21.15.7:0/1552831743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc174074bb0 con 0x7fc1740c43d0 2023-03-03T13:59:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.116+0000 7fc16bfff700 1 -- 172.21.15.7:0/1552831743 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc1640557f0 con 0x7fc1740c43d0 2023-03-03T13:59:05.118 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:59:05.120 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 -- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1540682f0 msgr2=0x7fc15406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1540682f0 0x7fc15406a7a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fc15c009130 tx=0x7fc15c008030 comp rx=0 tx=0).stop 2023-03-03T13:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 -- 172.21.15.7:0/1552831743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 msgr2=0x7fc1740c6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 0x7fc1740c6330 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fc16400c4d0 tx=0x7fc16400c890 comp rx=0 tx=0).stop 2023-03-03T13:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 -- 172.21.15.7:0/1552831743 shutdown_connections 2023-03-03T13:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc174154590 0x7fc1741e9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1540682f0 0x7fc15406a7a0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1740c43d0 0x7fc1740c6330 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 --2- 172.21.15.7:0/1552831743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc17414e560 0x7fc174149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 -- 172.21.15.7:0/1552831743 >> 172.21.15.7:0/1552831743 conn(0x7fc1740736f0 msgr2=0x7fc1740c55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 -- 172.21.15.7:0/1552831743 shutdown_connections 2023-03-03T13:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:05.118+0000 7fc1798ad700 1 -- 172.21.15.7:0/1552831743 wait complete. 2023-03-03T13:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:59:05.131 INFO:tasks.mds_thrash.fs.[b]:waiting for 99 secs before thrashing 2023-03-03T13:59:18.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:18.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:28.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T13:59:28.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.599+0000 7f7e345dd700 1 Processor -- start 2023-03-03T13:59:28.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.599+0000 7f7e345dd700 1 -- start start 2023-03-03T13:59:28.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.600+0000 7f7e345dd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e2c0aad20 0x7f7e2c157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:28.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.600+0000 7f7e345dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aa030 0x7f7e2c1581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:28.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.600+0000 7f7e345dd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0728b0 con 0x7f7e2c14b020 2023-03-03T13:59:28.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.600+0000 7f7e345dd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2c155070 con 0x7f7e2c0aad20 2023-03-03T13:59:28.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.600+0000 7f7e345dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c06f5b0 con 0x7f7e2c0aa030 2023-03-03T13:59:28.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.600+0000 7f7e31b78700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e2c0aad20 0x7f7e2c157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:28.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.600+0000 7f7e32379700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7e2c14b020 0x7f7e2c0717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40784/0 (socket says 172.21.15.7:40784) 2023-03-03T13:59:28.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.600+0000 7f7e32379700 1 -- 172.21.15.7:0/3805406193 learned_addr learned my addr 172.21.15.7:0/3805406193 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T13:59:28.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.601+0000 7f7e32b7a700 1 --2- 172.21.15.7:0/3805406193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aa030 0x7f7e2c1581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:28.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.601+0000 7f7e31b78700 1 -- 172.21.15.7:0/3805406193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aa030 msgr2=0x7f7e2c1581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T13:59:28.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.601+0000 7f7e31b78700 1 --2- 172.21.15.7:0/3805406193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aa030 0x7f7e2c1581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:28.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.601+0000 7f7e31b78700 1 -- 172.21.15.7:0/3805406193 >> v1:172.21.15.7:6789/0 conn(0x7f7e2c14b020 legacy=0x7f7e2c0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:28.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.601+0000 7f7e31b78700 1 -- 172.21.15.7:0/3805406193 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c06f890 con 0x7f7e2c0aad20 2023-03-03T13:59:28.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.601+0000 7f7e31377700 1 -- 172.21.15.7:0/3805406193 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463183246 0 0) 0x7f7e2c0728b0 con 0x7f7e2c14b020 2023-03-03T13:59:28.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.601+0000 7f7e31b78700 1 --2- 172.21.15.7:0/3805406193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c0aad20 0x7f7e2c157ca0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009840 tx=0x7f7e20009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95973c5121974766 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:59:28.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.601+0000 7f7e31377700 1 -- 172.21.15.7:0/3805406193 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20011830 con 0x7f7e2c0aad20 2023-03-03T13:59:28.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.601+0000 7f7e31377700 1 -- 172.21.15.7:0/3805406193 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e20011990 con 0x7f7e2c0aad20 2023-03-03T13:59:28.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.601+0000 7f7e31377700 1 -- 172.21.15.7:0/3805406193 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20011d20 con 0x7f7e2c0aad20 2023-03-03T13:59:28.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.602+0000 7f7e345dd700 1 -- 172.21.15.7:0/3805406193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c0aad20 msgr2=0x7f7e2c157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:28.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.602+0000 7f7e345dd700 1 --2- 172.21.15.7:0/3805406193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c0aad20 0x7f7e2c157ca0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009840 tx=0x7f7e20009b50 comp rx=0 tx=0).stop 2023-03-03T13:59:28.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.602+0000 7f7e345dd700 1 -- 172.21.15.7:0/3805406193 shutdown_connections 2023-03-03T13:59:28.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.602+0000 7f7e345dd700 1 --2- 172.21.15.7:0/3805406193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aa030 0x7f7e2c1581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:28.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.602+0000 7f7e345dd700 1 --2- 172.21.15.7:0/3805406193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c0aad20 0x7f7e2c157ca0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:28.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.602+0000 7f7e345dd700 1 -- 172.21.15.7:0/3805406193 >> 172.21.15.7:0/3805406193 conn(0x7f7e2c0c8900 msgr2=0x7f7e2c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:59:28.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.602+0000 7f7e345dd700 1 -- 172.21.15.7:0/3805406193 shutdown_connections 2023-03-03T13:59:28.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.602+0000 7f7e345dd700 1 -- 172.21.15.7:0/3805406193 wait complete. 2023-03-03T13:59:28.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.603+0000 7f7e345dd700 1 Processor -- start 2023-03-03T13:59:28.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.604+0000 7f7e345dd700 1 -- start start 2023-03-03T13:59:28.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.604+0000 7f7e345dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e2c0aa030 0x7f7e2c0c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:28.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.604+0000 7f7e345dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aad20 0x7f7e2c145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:28.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.604+0000 7f7e345dd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c14b020 0x7f7e2c152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:28.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.604+0000 7f7e345dd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c1eac30 con 0x7f7e2c14b020 2023-03-03T13:59:28.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.604+0000 7f7e345dd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c1ea020 con 0x7f7e2c0aa030 2023-03-03T13:59:28.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.604+0000 7f7e345dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c1ea300 con 0x7f7e2c0aad20 2023-03-03T13:59:28.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e32b7a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c14b020 0x7f7e2c152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:28.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e32b7a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c14b020 0x7f7e2c152070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46814/0 (socket says 172.21.15.7:46814) 2023-03-03T13:59:28.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e32b7a700 1 -- 172.21.15.7:0/1607992530 learned_addr learned my addr 172.21.15.7:0/1607992530 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T13:59:28.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e31b78700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aad20 0x7f7e2c145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:28.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e32379700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e2c0aa030 0x7f7e2c0c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:28.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e32b7a700 1 -- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aad20 msgr2=0x7f7e2c145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:28.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e32b7a700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aad20 0x7f7e2c145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:28.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e32b7a700 1 -- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e2c0aa030 msgr2=0x7f7e2c0c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:28.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e32b7a700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e2c0aa030 0x7f7e2c0c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:28.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e32b7a700 1 -- 172.21.15.7:0/1607992530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e200094a0 con 0x7f7e2c14b020 2023-03-03T13:59:28.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e31b78700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aad20 0x7f7e2c145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T13:59:28.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e32b7a700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c14b020 0x7f7e2c152070 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f7e2400efc0 tx=0x7f7e2400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:59:28.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e1effd700 1 -- 172.21.15.7:0/1607992530 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24009ba0 con 0x7f7e2c14b020 2023-03-03T13:59:28.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e1effd700 1 -- 172.21.15.7:0/1607992530 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2400f040 con 0x7f7e2c14b020 2023-03-03T13:59:28.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e345dd700 1 -- 172.21.15.7:0/1607992530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c155350 con 0x7f7e2c14b020 2023-03-03T13:59:28.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e1effd700 1 -- 172.21.15.7:0/1607992530 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24014480 con 0x7f7e2c14b020 2023-03-03T13:59:28.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.605+0000 7f7e345dd700 1 -- 172.21.15.7:0/1607992530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c06fb70 con 0x7f7e2c14b020 2023-03-03T13:59:28.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.606+0000 7f7e345dd700 1 -- 172.21.15.7:0/1607992530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c071ef0 con 0x7f7e2c14b020 2023-03-03T13:59:28.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.607+0000 7f7e1effd700 1 -- 172.21.15.7:0/1607992530 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7e240145e0 con 0x7f7e2c14b020 2023-03-03T13:59:28.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.607+0000 7f7e1effd700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e100682f0 0x7f7e1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T13:59:28.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.607+0000 7f7e1effd700 1 -- 172.21.15.7:0/1607992530 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7e24086f40 con 0x7f7e2c14b020 2023-03-03T13:59:28.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.607+0000 7f7e32379700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e100682f0 0x7f7e1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T13:59:28.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.608+0000 7f7e32379700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e100682f0 0x7f7e1006a7a0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f7e28000f60 tx=0x7f7e28007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T13:59:28.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.611+0000 7f7e1effd700 1 -- 172.21.15.7:0/1607992530 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7e240531f0 con 0x7f7e2c14b020 2023-03-03T13:59:28.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.852+0000 7f7e345dd700 1 -- 172.21.15.7:0/1607992530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e2c1eade0 con 0x7f7e2c14b020 2023-03-03T13:59:28.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.853+0000 7f7e1effd700 1 -- 172.21.15.7:0/1607992530 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7e2c1eade0 con 0x7f7e2c14b020 2023-03-03T13:59:28.854 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T13:59:28.855 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T13:59:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 -- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e100682f0 msgr2=0x7f7e1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e100682f0 0x7f7e1006a7a0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f7e28000f60 tx=0x7f7e28007c90 comp rx=0 tx=0).stop 2023-03-03T13:59:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 -- 172.21.15.7:0/1607992530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c14b020 msgr2=0x7f7e2c152070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T13:59:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c14b020 0x7f7e2c152070 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f7e2400efc0 tx=0x7f7e2400c3a0 comp rx=0 tx=0).stop 2023-03-03T13:59:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 -- 172.21.15.7:0/1607992530 shutdown_connections 2023-03-03T13:59:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e100682f0 0x7f7e1006a7a0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e2c14b020 0x7f7e2c152070 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e2c0aa030 0x7f7e2c0c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 --2- 172.21.15.7:0/1607992530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e2c0aad20 0x7f7e2c145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T13:59:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 -- 172.21.15.7:0/1607992530 >> 172.21.15.7:0/1607992530 conn(0x7f7e2c0c8900 msgr2=0x7f7e2c158360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T13:59:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 -- 172.21.15.7:0/1607992530 shutdown_connections 2023-03-03T13:59:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T13:59:28.855+0000 7f7e345dd700 1 -- 172.21.15.7:0/1607992530 wait complete. 2023-03-03T13:59:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T13:59:28.868 INFO:tasks.mds_thrash.fs.[a]:waiting for 59 secs before thrashing 2023-03-03T13:59:48.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T13:59:48.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T13:59:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T14:00:18.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:18.704 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:27.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:00:28.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.006+0000 7fe7e7d96700 1 Processor -- start 2023-03-03T14:00:28.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.007+0000 7fe7e7d96700 1 -- start start 2023-03-03T14:00:28.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.007+0000 7fe7e7d96700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7e0151d40 0x7fe7e00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:28.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.007+0000 7fe7e7d96700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e00c2930 0x7fe7e0157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:28.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.007+0000 7fe7e7d96700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e006d430 con 0x7fe7e0073c30 2023-03-03T14:00:28.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.007+0000 7fe7e7d96700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e006c2a0 con 0x7fe7e0151d40 2023-03-03T14:00:28.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.007+0000 7fe7e7d96700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e01586e0 con 0x7fe7e00c2930 2023-03-03T14:00:28.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e5b32700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7e0151d40 0x7fe7e00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:28.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e5b32700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7e0151d40 0x7fe7e00736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46862/0 (socket says 172.21.15.7:46862) 2023-03-03T14:00:28.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e5b32700 1 -- 172.21.15.7:0/3222909951 learned_addr learned my addr 172.21.15.7:0/3222909951 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:00:28.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e5331700 1 --2- 172.21.15.7:0/3222909951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e00c2930 0x7fe7e0157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:28.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e5b32700 1 -- 172.21.15.7:0/3222909951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e00c2930 msgr2=0x7fe7e0157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:00:28.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e5b32700 1 --2- 172.21.15.7:0/3222909951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e00c2930 0x7fe7e0157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:28.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e5b32700 1 -- 172.21.15.7:0/3222909951 >> v1:172.21.15.143:6789/0 conn(0x7fe7e0073c30 legacy=0x7fe7e0157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:28.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e5b32700 1 -- 172.21.15.7:0/3222909951 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e006cb00 con 0x7fe7e0151d40 2023-03-03T14:00:28.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e5b32700 1 --2- 172.21.15.7:0/3222909951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0151d40 0x7fe7e00736f0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40098f0 tx=0x7fe7d4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c7d38fef089c33 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:28.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e4b30700 1 -- 172.21.15.7:0/3222909951 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4004230 con 0x7fe7e0151d40 2023-03-03T14:00:28.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e4b30700 1 -- 172.21.15.7:0/3222909951 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d4004390 con 0x7fe7e0151d40 2023-03-03T14:00:28.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.008+0000 7fe7e4b30700 1 -- 172.21.15.7:0/3222909951 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d40036a0 con 0x7fe7e0151d40 2023-03-03T14:00:28.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.009+0000 7fe7e7d96700 1 -- 172.21.15.7:0/3222909951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0151d40 msgr2=0x7fe7e00736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:28.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.009+0000 7fe7e7d96700 1 --2- 172.21.15.7:0/3222909951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0151d40 0x7fe7e00736f0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40098f0 tx=0x7fe7d4009c00 comp rx=0 tx=0).stop 2023-03-03T14:00:28.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.009+0000 7fe7e7d96700 1 -- 172.21.15.7:0/3222909951 shutdown_connections 2023-03-03T14:00:28.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.009+0000 7fe7e7d96700 1 --2- 172.21.15.7:0/3222909951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0151d40 0x7fe7e00736f0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:28.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.009+0000 7fe7e7d96700 1 --2- 172.21.15.7:0/3222909951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e00c2930 0x7fe7e0157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:28.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.009+0000 7fe7e7d96700 1 -- 172.21.15.7:0/3222909951 >> 172.21.15.7:0/3222909951 conn(0x7fe7e00a94d0 msgr2=0x7fe7e00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:00:28.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.009+0000 7fe7e7d96700 1 -- 172.21.15.7:0/3222909951 shutdown_connections 2023-03-03T14:00:28.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.009+0000 7fe7e7d96700 1 -- 172.21.15.7:0/3222909951 wait complete. 2023-03-03T14:00:28.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.010+0000 7fe7e7d96700 1 Processor -- start 2023-03-03T14:00:28.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.010+0000 7fe7e7d96700 1 -- start start 2023-03-03T14:00:28.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e7d96700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0073c30 0x7fe7e01ed8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:28.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e7d96700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e00c2930 0x7fe7e01dfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:28.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e7d96700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e0151d40 0x7fe7e01e0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:28.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e7d96700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e0153320 con 0x7fe7e0073c30 2023-03-03T14:00:28.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e7d96700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e01e0580 con 0x7fe7e0151d40 2023-03-03T14:00:28.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e7d96700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00be560 con 0x7fe7e00c2930 2023-03-03T14:00:28.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e5b32700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0073c30 0x7fe7e01ed8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:28.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e5b32700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0073c30 0x7fe7e01ed8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46868/0 (socket says 172.21.15.7:46868) 2023-03-03T14:00:28.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e5b32700 1 -- 172.21.15.7:0/1281734043 learned_addr learned my addr 172.21.15.7:0/1281734043 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:00:28.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e6333700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e0151d40 0x7fe7e01e0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:28.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e5331700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e00c2930 0x7fe7e01dfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:28.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e5b32700 1 -- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e00c2930 msgr2=0x7fe7e01dfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:28.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e5b32700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e00c2930 0x7fe7e01dfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:28.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e5b32700 1 -- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e0151d40 msgr2=0x7fe7e01e0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:28.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e5b32700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e0151d40 0x7fe7e01e0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:28.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e5b32700 1 -- 172.21.15.7:0/1281734043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d4009550 con 0x7fe7e0073c30 2023-03-03T14:00:28.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e6333700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e0151d40 0x7fe7e01e0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:00:28.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.011+0000 7fe7e5b32700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0073c30 0x7fe7e01ed8e0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009840 tx=0x7fe7d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:28.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.012+0000 7fe7d27fc700 1 -- 172.21.15.7:0/1281734043 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4003cc0 con 0x7fe7e0073c30 2023-03-03T14:00:28.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.012+0000 7fe7d27fc700 1 -- 172.21.15.7:0/1281734043 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d401f070 con 0x7fe7e0073c30 2023-03-03T14:00:28.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.012+0000 7fe7e7d96700 1 -- 172.21.15.7:0/1281734043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e006d0c0 con 0x7fe7e0073c30 2023-03-03T14:00:28.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.012+0000 7fe7d27fc700 1 -- 172.21.15.7:0/1281734043 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d400b8a0 con 0x7fe7e0073c30 2023-03-03T14:00:28.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.012+0000 7fe7e7d96700 1 -- 172.21.15.7:0/1281734043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e006cde0 con 0x7fe7e0073c30 2023-03-03T14:00:28.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.013+0000 7fe7e7d96700 1 -- 172.21.15.7:0/1281734043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e006bea0 con 0x7fe7e0073c30 2023-03-03T14:00:28.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.013+0000 7fe7d27fc700 1 -- 172.21.15.7:0/1281734043 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe7d402a020 con 0x7fe7e0073c30 2023-03-03T14:00:28.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.013+0000 7fe7d27fc700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7c40682f0 0x7fe7c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:28.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.014+0000 7fe7d27fc700 1 -- 172.21.15.7:0/1281734043 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe7d40878d0 con 0x7fe7e0073c30 2023-03-03T14:00:28.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.014+0000 7fe7e5331700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7c40682f0 0x7fe7c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:28.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.014+0000 7fe7e5331700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7c40682f0 0x7fe7c406a7a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0071cd0 tx=0x7fe7dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:28.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.018+0000 7fe7d27fc700 1 -- 172.21.15.7:0/1281734043 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe7d4053c70 con 0x7fe7e0073c30 2023-03-03T14:00:28.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.254+0000 7fe7e7d96700 1 -- 172.21.15.7:0/1281734043 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7e006bea0 con 0x7fe7e0073c30 2023-03-03T14:00:28.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.254+0000 7fe7d27fc700 1 -- 172.21.15.7:0/1281734043 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe7d4057290 con 0x7fe7e0073c30 2023-03-03T14:00:28.256 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:00:28.256 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:00:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 -- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7c40682f0 msgr2=0x7fe7c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7c40682f0 0x7fe7c406a7a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0071cd0 tx=0x7fe7dc008040 comp rx=0 tx=0).stop 2023-03-03T14:00:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 -- 172.21.15.7:0/1281734043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0073c30 msgr2=0x7fe7e01ed8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:28.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0073c30 0x7fe7e01ed8e0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009840 tx=0x7fe7d4004750 comp rx=0 tx=0).stop 2023-03-03T14:00:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 -- 172.21.15.7:0/1281734043 shutdown_connections 2023-03-03T14:00:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e0151d40 0x7fe7e01e0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7c40682f0 0x7fe7c406a7a0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e0073c30 0x7fe7e01ed8e0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 --2- 172.21.15.7:0/1281734043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e00c2930 0x7fe7e01dfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 -- 172.21.15.7:0/1281734043 >> 172.21.15.7:0/1281734043 conn(0x7fe7e00a94d0 msgr2=0x7fe7e006af00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:00:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 -- 172.21.15.7:0/1281734043 shutdown_connections 2023-03-03T14:00:28.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:28.256+0000 7fe7e7d96700 1 -- 172.21.15.7:0/1281734043 wait complete. 2023-03-03T14:00:28.260 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:00:28.270 INFO:tasks.mds_thrash.fs.[a]:waiting for 114 secs before thrashing 2023-03-03T14:00:44.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:00:44.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.270+0000 7f35237f8700 1 Processor -- start 2023-03-03T14:00:44.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.270+0000 7f35237f8700 1 -- start start 2023-03-03T14:00:44.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.271+0000 7f35237f8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f351c071e80 0x7f351c151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.271+0000 7f35237f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c072250 0x7f351c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.271+0000 7f35237f8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351c1510d0 con 0x7f351c0c1ce0 2023-03-03T14:00:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.271+0000 7f35237f8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351c0c3740 con 0x7f351c071e80 2023-03-03T14:00:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.271+0000 7f35237f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351c070670 con 0x7f351c072250 2023-03-03T14:00:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.271+0000 7f3521d95700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f351c0c1ce0 0x7f351c1546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40864/0 (socket says 172.21.15.7:40864) 2023-03-03T14:00:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.271+0000 7f3521d95700 1 -- 172.21.15.7:0/3658349797 learned_addr learned my addr 172.21.15.7:0/3658349797 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:00:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.271+0000 7f3520d93700 1 --2- 172.21.15.7:0/3658349797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c072250 0x7f351c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:44.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.271+0000 7f3521594700 1 --2- 172.21.15.7:0/3658349797 >> v2:172.21.15.143:3300/0 conn(0x7f351c071e80 0x7f351c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:44.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.271+0000 7f3513fff700 1 -- 172.21.15.7:0/3658349797 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128471790 0 0) 0x7f351c1510d0 con 0x7f351c0c1ce0 2023-03-03T14:00:44.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.272+0000 7f3513fff700 1 -- 172.21.15.7:0/3658349797 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35080035b0 con 0x7f351c0c1ce0 2023-03-03T14:00:44.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.272+0000 7f3513fff700 1 -- 172.21.15.7:0/3658349797 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 274115846 0 0) 0x7f35080035b0 con 0x7f351c0c1ce0 2023-03-03T14:00:44.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.272+0000 7f3520d93700 1 -- 172.21.15.7:0/3658349797 >> v2:172.21.15.143:3300/0 conn(0x7f351c071e80 msgr2=0x7f351c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:00:44.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.272+0000 7f3520d93700 1 --2- 172.21.15.7:0/3658349797 >> v2:172.21.15.143:3300/0 conn(0x7f351c071e80 0x7f351c151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.272+0000 7f3520d93700 1 -- 172.21.15.7:0/3658349797 >> v1:172.21.15.7:6789/0 conn(0x7f351c0c1ce0 legacy=0x7f351c1546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:44.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.272+0000 7f3520d93700 1 -- 172.21.15.7:0/3658349797 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c0c39e0 con 0x7f351c072250 2023-03-03T14:00:44.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.272+0000 7f3520d93700 1 --2- 172.21.15.7:0/3658349797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c072250 0x7f351c14e1f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3518009840 tx=0x7f3518009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c07d9b60ead4a154 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:44.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.272+0000 7f3513fff700 1 -- 172.21.15.7:0/3658349797 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35140019f0 con 0x7f351c072250 2023-03-03T14:00:44.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.272+0000 7f3513fff700 1 -- 172.21.15.7:0/3658349797 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3518004030 con 0x7f351c072250 2023-03-03T14:00:44.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.272+0000 7f3513fff700 1 -- 172.21.15.7:0/3658349797 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35180043c0 con 0x7f351c072250 2023-03-03T14:00:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.273+0000 7f35237f8700 1 -- 172.21.15.7:0/3658349797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c072250 msgr2=0x7f351c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.273+0000 7f35237f8700 1 --2- 172.21.15.7:0/3658349797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c072250 0x7f351c14e1f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3518009840 tx=0x7f3518009b50 comp rx=0 tx=0).stop 2023-03-03T14:00:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.273+0000 7f35237f8700 1 -- 172.21.15.7:0/3658349797 shutdown_connections 2023-03-03T14:00:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.273+0000 7f35237f8700 1 --2- 172.21.15.7:0/3658349797 >> v2:172.21.15.143:3300/0 conn(0x7f351c071e80 0x7f351c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.273+0000 7f35237f8700 1 --2- 172.21.15.7:0/3658349797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c072250 0x7f351c14e1f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.273+0000 7f35237f8700 1 -- 172.21.15.7:0/3658349797 >> 172.21.15.7:0/3658349797 conn(0x7f351c0a94d0 msgr2=0x7f351c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:00:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.273+0000 7f35237f8700 1 -- 172.21.15.7:0/3658349797 shutdown_connections 2023-03-03T14:00:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.273+0000 7f35237f8700 1 -- 172.21.15.7:0/3658349797 wait complete. 2023-03-03T14:00:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.274+0000 7f35237f8700 1 Processor -- start 2023-03-03T14:00:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.275+0000 7f35237f8700 1 -- start start 2023-03-03T14:00:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.275+0000 7f35237f8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c071e80 0x7f351c0c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.275+0000 7f35237f8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f351c072250 0x7f351c1428e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.275+0000 7f35237f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c0c1ce0 0x7f351c142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.275+0000 7f35237f8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c1436e0 con 0x7f351c072250 2023-03-03T14:00:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.275+0000 7f35237f8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0aa2b0 con 0x7f351c071e80 2023-03-03T14:00:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.275+0000 7f35237f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351c1ea6e0 con 0x7f351c0c1ce0 2023-03-03T14:00:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3520d93700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f351c072250 0x7f351c1428e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3520d93700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f351c072250 0x7f351c1428e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46888/0 (socket says 172.21.15.7:46888) 2023-03-03T14:00:44.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3520d93700 1 -- 172.21.15.7:0/79616628 learned_addr learned my addr 172.21.15.7:0/79616628 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:00:44.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3521594700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c071e80 0x7f351c0c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:44.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3521d95700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c0c1ce0 0x7f351c142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:44.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3520d93700 1 -- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c0c1ce0 msgr2=0x7f351c142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:44.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3520d93700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c0c1ce0 0x7f351c142e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3520d93700 1 -- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c071e80 msgr2=0x7f351c0c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:44.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3520d93700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c071e80 0x7f351c0c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3520d93700 1 -- 172.21.15.7:0/79616628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35180094a0 con 0x7f351c072250 2023-03-03T14:00:44.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3521d95700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c0c1ce0 0x7f351c142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:00:44.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3521594700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c071e80 0x7f351c0c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:00:44.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3520d93700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f351c072250 0x7f351c1428e0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f35180090b0 tx=0x7f3518004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:44.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3511ffb700 1 -- 172.21.15.7:0/79616628 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3518003e20 con 0x7f351c072250 2023-03-03T14:00:44.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3511ffb700 1 -- 172.21.15.7:0/79616628 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351801f070 con 0x7f351c072250 2023-03-03T14:00:44.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f35237f8700 1 -- 172.21.15.7:0/79616628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c06c9a0 con 0x7f351c072250 2023-03-03T14:00:44.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.276+0000 7f3511ffb700 1 -- 172.21.15.7:0/79616628 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3518011870 con 0x7f351c072250 2023-03-03T14:00:44.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.277+0000 7f35237f8700 1 -- 172.21.15.7:0/79616628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c151300 con 0x7f351c072250 2023-03-03T14:00:44.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.277+0000 7f35237f8700 1 -- 172.21.15.7:0/79616628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c06cb30 con 0x7f351c072250 2023-03-03T14:00:44.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.278+0000 7f3511ffb700 1 -- 172.21.15.7:0/79616628 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3518017020 con 0x7f351c072250 2023-03-03T14:00:44.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.278+0000 7f3511ffb700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35000682f0 0x7f350006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:00:44.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.279+0000 7f3511ffb700 1 -- 172.21.15.7:0/79616628 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3518021070 con 0x7f351c072250 2023-03-03T14:00:44.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.279+0000 7f3521594700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35000682f0 0x7f350006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:00:44.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.279+0000 7f3521594700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35000682f0 0x7f350006a7a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f351c151010 tx=0x7f350c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:00:44.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.283+0000 7f3511ffb700 1 -- 172.21.15.7:0/79616628 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3518053db0 con 0x7f351c072250 2023-03-03T14:00:44.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.524+0000 7f35237f8700 1 -- 172.21.15.7:0/79616628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f351c074bb0 con 0x7f351c072250 2023-03-03T14:00:44.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.524+0000 7f3511ffb700 1 -- 172.21.15.7:0/79616628 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f35180573d0 con 0x7f351c072250 2023-03-03T14:00:44.526 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:00:44.526 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:00:44.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 -- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35000682f0 msgr2=0x7f350006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:44.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35000682f0 0x7f350006a7a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f351c151010 tx=0x7f350c008040 comp rx=0 tx=0).stop 2023-03-03T14:00:44.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 -- 172.21.15.7:0/79616628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f351c072250 msgr2=0x7f351c1428e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:00:44.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f351c072250 0x7f351c1428e0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f35180090b0 tx=0x7f3518004750 comp rx=0 tx=0).stop 2023-03-03T14:00:44.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 -- 172.21.15.7:0/79616628 shutdown_connections 2023-03-03T14:00:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f351c0c1ce0 0x7f351c142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f351c071e80 0x7f351c0c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35000682f0 0x7f350006a7a0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 --2- 172.21.15.7:0/79616628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f351c072250 0x7f351c1428e0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:00:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 -- 172.21.15.7:0/79616628 >> 172.21.15.7:0/79616628 conn(0x7f351c0a94d0 msgr2=0x7f351c06d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:00:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 -- 172.21.15.7:0/79616628 shutdown_connections 2023-03-03T14:00:44.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:00:44.526+0000 7f35237f8700 1 -- 172.21.15.7:0/79616628 wait complete. 2023-03-03T14:00:44.530 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:00:44.540 INFO:tasks.mds_thrash.fs.[b]:waiting for 55 secs before thrashing 2023-03-03T14:00:48.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:00:48.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:18.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:18.972 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:39.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:01:39.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.680+0000 7ff4d8548700 1 Processor -- start 2023-03-03T14:01:39.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.680+0000 7ff4d8548700 1 -- start start 2023-03-03T14:01:39.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.681+0000 7ff4d8548700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4d00bfc90 0x7ff4d0151610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:39.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.681+0000 7ff4d8548700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d006bbb0 con 0x7ff4d00c0c60 2023-03-03T14:01:39.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.681+0000 7ff4d8548700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d006d270 con 0x7ff4d00bf8c0 2023-03-03T14:01:39.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.681+0000 7ff4d8548700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00a96f0 con 0x7ff4d00bfc90 2023-03-03T14:01:39.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d6ae5700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff4d00c0c60 0x7ff4d006c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40918/0 (socket says 172.21.15.7:40918) 2023-03-03T14:01:39.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d6ae5700 1 -- 172.21.15.7:0/479895035 learned_addr learned my addr 172.21.15.7:0/479895035 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:01:39.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d5ae3700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4d00bfc90 0x7ff4d0151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:39.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416455239 0 0) 0x7ff4d006bbb0 con 0x7ff4d00c0c60 2023-03-03T14:01:39.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4bc0035b0 con 0x7ff4d00c0c60 2023-03-03T14:01:39.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23761382 0 0) 0x7ff4d006d270 con 0x7ff4d00bf8c0 2023-03-03T14:01:39.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d006bbb0 con 0x7ff4d00bf8c0 2023-03-03T14:01:39.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3993719010 0 0) 0x7ff4bc0035b0 con 0x7ff4d00c0c60 2023-03-03T14:01:39.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4d00bfc90 msgr2=0x7ff4d0151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:01:39.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4d00bfc90 0x7ff4d0151610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:39.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 >> v1:172.21.15.143:6789/0 conn(0x7ff4d00bf8c0 legacy=0x7ff4d0154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:39.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d006ccb0 con 0x7ff4d00c0c60 2023-03-03T14:01:39.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff4c80019f0 con 0x7ff4d00c0c60 2023-03-03T14:01:39.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 >> v1:172.21.15.7:6789/0 conn(0x7ff4d00c0c60 legacy=0x7ff4d006c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:39.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4d00bfc90 0x7ff4bc003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:39.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4bc004200 0x7ff4bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:39.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4bc004de0 0x7ff4bc005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:39.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00a96f0 con 0x7ff4bc004200 2023-03-03T14:01:39.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc00bcf0 con 0x7ff4bc004de0 2023-03-03T14:01:39.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc00beb0 con 0x7ff4d00bfc90 2023-03-03T14:01:39.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d5ae3700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4bc004200 0x7ff4bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:39.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.682+0000 7ff4d62e4700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4d00bfc90 0x7ff4bc003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:39.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.683+0000 7ff4d6ae5700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4bc004de0 0x7ff4bc005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:39.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.683+0000 7ff4d5ae3700 1 -- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4d00bfc90 msgr2=0x7ff4bc003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:01:39.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.683+0000 7ff4d5ae3700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4d00bfc90 0x7ff4bc003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:39.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.683+0000 7ff4d5ae3700 1 -- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4bc004de0 msgr2=0x7ff4bc005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:01:39.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.683+0000 7ff4d5ae3700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4bc004de0 0x7ff4bc005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:39.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.683+0000 7ff4d5ae3700 1 -- 172.21.15.7:0/479895035 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc0035b0 con 0x7ff4bc004200 2023-03-03T14:01:39.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.683+0000 7ff4d5ae3700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4bc004200 0x7ff4bc004850 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc009b30 tx=0x7ff4bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a086b2fd54af56b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:01:39.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.683+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc00e6c0 con 0x7ff4bc004200 2023-03-03T14:01:39.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.683+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc00e820 con 0x7ff4bc004200 2023-03-03T14:01:39.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.683+0000 7ff4d52e2700 1 -- 172.21.15.7:0/479895035 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc00ebb0 con 0x7ff4bc004200 2023-03-03T14:01:39.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.684+0000 7ff4d8548700 1 -- 172.21.15.7:0/479895035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4bc004200 msgr2=0x7ff4bc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:39.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.684+0000 7ff4d8548700 1 --2- 172.21.15.7:0/479895035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4bc004200 0x7ff4bc004850 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc009b30 tx=0x7ff4bc005880 comp rx=0 tx=0).stop 2023-03-03T14:01:39.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.684+0000 7ff4d6ae5700 1 -- 172.21.15.7:0/479895035 reap_dead start 2023-03-03T14:01:39.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.684+0000 7ff4d8548700 1 -- 172.21.15.7:0/479895035 shutdown_connections 2023-03-03T14:01:39.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.684+0000 7ff4d8548700 1 -- 172.21.15.7:0/479895035 >> 172.21.15.7:0/479895035 conn(0x7ff4d00c62e0 msgr2=0x7ff4d00c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:01:39.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.684+0000 7ff4d8548700 1 -- 172.21.15.7:0/479895035 shutdown_connections 2023-03-03T14:01:39.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.684+0000 7ff4d8548700 1 -- 172.21.15.7:0/479895035 wait complete. 2023-03-03T14:01:39.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.685+0000 7ff4d8548700 1 Processor -- start 2023-03-03T14:01:39.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.685+0000 7ff4d8548700 1 -- start start 2023-03-03T14:01:39.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.686+0000 7ff4d8548700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4bc004200 0x7ff4d01eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:39.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.686+0000 7ff4d8548700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4d014aec0 0x7ff4d00c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:39.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.686+0000 7ff4d8548700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4d013c230 0x7ff4d013c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:39.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.686+0000 7ff4d8548700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d0071190 con 0x7ff4d014aec0 2023-03-03T14:01:39.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.686+0000 7ff4d8548700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d01eaf80 con 0x7ff4d013c230 2023-03-03T14:01:39.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.686+0000 7ff4d8548700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d014b310 con 0x7ff4bc004200 2023-03-03T14:01:39.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.686+0000 7ff4d5ae3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4d014aec0 0x7ff4d00c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:39.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.686+0000 7ff4d5ae3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4d014aec0 0x7ff4d00c5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46948/0 (socket says 172.21.15.7:46948) 2023-03-03T14:01:39.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.686+0000 7ff4d5ae3700 1 -- 172.21.15.7:0/2163245780 learned_addr learned my addr 172.21.15.7:0/2163245780 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:01:39.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d62e4700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4bc004200 0x7ff4d01eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:39.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d6ae5700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4d013c230 0x7ff4d013c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:39.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d5ae3700 1 -- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4bc004200 msgr2=0x7ff4d01eaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:39.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d5ae3700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4bc004200 0x7ff4d01eaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:39.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d5ae3700 1 -- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4d013c230 msgr2=0x7ff4d013c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:39.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d5ae3700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4d013c230 0x7ff4d013c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:39.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d5ae3700 1 -- 172.21.15.7:0/2163245780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d006b840 con 0x7ff4d014aec0 2023-03-03T14:01:39.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d6ae5700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4d013c230 0x7ff4d013c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:01:39.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d5ae3700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4d014aec0 0x7ff4d00c5170 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00bcc0 tx=0x7ff4cc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:01:39.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4c2ffd700 1 -- 172.21.15.7:0/2163245780 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc00e6c0 con 0x7ff4d014aec0 2023-03-03T14:01:39.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4c2ffd700 1 -- 172.21.15.7:0/2163245780 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc013da0 con 0x7ff4d014aec0 2023-03-03T14:01:39.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d8548700 1 -- 172.21.15.7:0/2163245780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d006cf90 con 0x7ff4d014aec0 2023-03-03T14:01:39.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4c2ffd700 1 -- 172.21.15.7:0/2163245780 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc01c690 con 0x7ff4d014aec0 2023-03-03T14:01:39.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.687+0000 7ff4d8548700 1 -- 172.21.15.7:0/2163245780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d006b340 con 0x7ff4d014aec0 2023-03-03T14:01:39.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.688+0000 7ff4d8548700 1 -- 172.21.15.7:0/2163245780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d0154f70 con 0x7ff4d014aec0 2023-03-03T14:01:39.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.688+0000 7ff4c2ffd700 1 -- 172.21.15.7:0/2163245780 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff4cc019030 con 0x7ff4d014aec0 2023-03-03T14:01:39.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.689+0000 7ff4c2ffd700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4b40682f0 0x7ff4b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:01:39.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.689+0000 7ff4c2ffd700 1 -- 172.21.15.7:0/2163245780 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff4cc087660 con 0x7ff4d014aec0 2023-03-03T14:01:39.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.689+0000 7ff4d62e4700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4b40682f0 0x7ff4b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:01:39.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.690+0000 7ff4d62e4700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4b40682f0 0x7ff4b406a7a0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4007fd0 tx=0x7ff4c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:01:39.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.693+0000 7ff4c2ffd700 1 -- 172.21.15.7:0/2163245780 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff4cc053a80 con 0x7ff4d014aec0 2023-03-03T14:01:39.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.933+0000 7ff4d8548700 1 -- 172.21.15.7:0/2163245780 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4d0071170 con 0x7ff4d014aec0 2023-03-03T14:01:39.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.934+0000 7ff4c2ffd700 1 -- 172.21.15.7:0/2163245780 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff4cc057020 con 0x7ff4d014aec0 2023-03-03T14:01:39.936 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:01:39.936 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:01:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.935+0000 7ff4d8548700 1 -- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4b40682f0 msgr2=0x7ff4b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4b40682f0 0x7ff4b406a7a0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4007fd0 tx=0x7ff4c4009040 comp rx=0 tx=0).stop 2023-03-03T14:01:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 -- 172.21.15.7:0/2163245780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4d014aec0 msgr2=0x7ff4d00c5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:01:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4d014aec0 0x7ff4d00c5170 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00bcc0 tx=0x7ff4cc0047c0 comp rx=0 tx=0).stop 2023-03-03T14:01:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 -- 172.21.15.7:0/2163245780 shutdown_connections 2023-03-03T14:01:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4d013c230 0x7ff4d013c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4bc004200 0x7ff4d01eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff4b40682f0 0x7ff4b406a7a0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 --2- 172.21.15.7:0/2163245780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff4d014aec0 0x7ff4d00c5170 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:01:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 -- 172.21.15.7:0/2163245780 >> 172.21.15.7:0/2163245780 conn(0x7ff4d00c62e0 msgr2=0x7ff4d006c220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:01:39.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 -- 172.21.15.7:0/2163245780 shutdown_connections 2023-03-03T14:01:39.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:01:39.936+0000 7ff4d8548700 1 -- 172.21.15.7:0/2163245780 wait complete. 2023-03-03T14:01:39.939 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:01:39.949 INFO:tasks.mds_thrash.fs.[b]:waiting for 80 secs before thrashing 2023-03-03T14:01:49.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:01:49.043 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:19.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:19.115 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:22.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:02:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.407+0000 7fd00de8a700 1 Processor -- start 2023-03-03T14:02:22.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.407+0000 7fd00de8a700 1 -- start start 2023-03-03T14:02:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd00de8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd00814b0d0 0x7fd008071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd00de8a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd0080aad20 0x7fd008157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd00de8a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0080a9c20 con 0x7fd008071d40 2023-03-03T14:02:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd00de8a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd008154f10 con 0x7fd0080aad20 2023-03-03T14:02:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd00de8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080ac3c0 con 0x7fd00814b0d0 2023-03-03T14:02:22.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd007fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd008071d40 0x7fd008158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40976/0 (socket says 172.21.15.7:40976) 2023-03-03T14:02:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd007fff700 1 -- 172.21.15.7:0/3533442520 learned_addr learned my addr 172.21.15.7:0/3533442520 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:02:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd006ffd700 1 --2- 172.21.15.7:0/3533442520 >> v2:172.21.15.143:3300/0 conn(0x7fd0080aad20 0x7fd008157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd0077fe700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd00814b0d0 0x7fd008071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158266861 0 0) 0x7fd0080a9c20 con 0x7fd008071d40 2023-03-03T14:02:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.408+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff40035b0 con 0x7fd008071d40 2023-03-03T14:02:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912562141 0 0) 0x7fcff40035b0 con 0x7fd008071d40 2023-03-03T14:02:22.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd00814b0d0 msgr2=0x7fd008071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:02:22.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd00814b0d0 0x7fd008071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:22.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 >> v2:172.21.15.143:3300/0 conn(0x7fd0080aad20 msgr2=0x7fd008157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:02:22.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 --2- 172.21.15.7:0/3533442520 >> v2:172.21.15.143:3300/0 conn(0x7fd0080aad20 0x7fd008157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:22.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00806fc70 con 0x7fd008071d40 2023-03-03T14:02:22.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd0000019f0 con 0x7fd008071d40 2023-03-03T14:02:22.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 >> v1:172.21.15.7:6789/0 conn(0x7fd008071d40 legacy=0x7fd008158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:22.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcff4003a70 0x7fcff4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd00814b0d0 0x7fcff4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcff4004d90 0x7fcff4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd008154f10 con 0x7fcff4003a70 2023-03-03T14:02:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080ac3c0 con 0x7fcff4004d90 2023-03-03T14:02:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff400ba60 con 0x7fd00814b0d0 2023-03-03T14:02:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0077fe700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcff4003a70 0x7fcff4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd006ffd700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd00814b0d0 0x7fcff4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd007fff700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcff4004d90 0x7fcff4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0077fe700 1 -- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd00814b0d0 msgr2=0x7fcff4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:02:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0077fe700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd00814b0d0 0x7fcff4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0077fe700 1 -- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcff4004d90 msgr2=0x7fcff4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:02:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0077fe700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcff4004d90 0x7fcff4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0077fe700 1 -- 172.21.15.7:0/3533442520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff40035b0 con 0x7fcff4003a70 2023-03-03T14:02:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd007fff700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcff4004d90 0x7fcff4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:02:22.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.409+0000 7fd0077fe700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcff4003a70 0x7fcff4003e80 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009b90 tx=0x7fcff4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233fd6fedb790885 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:02:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.410+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff800e6c0 con 0x7fcff4003a70 2023-03-03T14:02:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.410+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff800e820 con 0x7fcff4003a70 2023-03-03T14:02:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.410+0000 7fd0067fc700 1 -- 172.21.15.7:0/3533442520 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff800eb90 con 0x7fcff4003a70 2023-03-03T14:02:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.410+0000 7fd00de8a700 1 -- 172.21.15.7:0/3533442520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcff4003a70 msgr2=0x7fcff4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.410+0000 7fd00de8a700 1 --2- 172.21.15.7:0/3533442520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcff4003a70 0x7fcff4003e80 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009b90 tx=0x7fcff4005880 comp rx=0 tx=0).stop 2023-03-03T14:02:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.410+0000 7fd007fff700 1 -- 172.21.15.7:0/3533442520 reap_dead start 2023-03-03T14:02:22.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.410+0000 7fd00de8a700 1 -- 172.21.15.7:0/3533442520 shutdown_connections 2023-03-03T14:02:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.410+0000 7fd00de8a700 1 -- 172.21.15.7:0/3533442520 >> 172.21.15.7:0/3533442520 conn(0x7fd0080c8900 msgr2=0x7fd0080c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:02:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.410+0000 7fd00de8a700 1 -- 172.21.15.7:0/3533442520 shutdown_connections 2023-03-03T14:02:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.410+0000 7fd00de8a700 1 -- 172.21.15.7:0/3533442520 wait complete. 2023-03-03T14:02:22.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.411+0000 7fd00de8a700 1 Processor -- start 2023-03-03T14:02:22.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.412+0000 7fd00de8a700 1 -- start start 2023-03-03T14:02:22.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.412+0000 7fd00de8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcff4003a70 0x7fd0081ed890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:22.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.412+0000 7fd00de8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0081ddaf0 0x7fd008147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:22.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.412+0000 7fd00de8a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008147560 0x7fd0081ea0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:22.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.412+0000 7fd00de8a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0081de470 con 0x7fd008147560 2023-03-03T14:02:22.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.412+0000 7fd00de8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0081de750 con 0x7fcff4003a70 2023-03-03T14:02:22.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.412+0000 7fd00de8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd008147fa0 con 0x7fd0081ddaf0 2023-03-03T14:02:22.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd007fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008147560 0x7fd0081ea0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:22.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd007fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008147560 0x7fd0081ea0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47010/0 (socket says 172.21.15.7:47010) 2023-03-03T14:02:22.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd007fff700 1 -- 172.21.15.7:0/2307801643 learned_addr learned my addr 172.21.15.7:0/2307801643 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:02:22.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd006ffd700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0081ddaf0 0x7fd008147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:22.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd0077fe700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcff4003a70 0x7fd0081ed890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:22.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd007fff700 1 -- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0081ddaf0 msgr2=0x7fd008147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:22.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd007fff700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0081ddaf0 0x7fd008147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:22.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd007fff700 1 -- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcff4003a70 msgr2=0x7fd0081ed890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:22.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd007fff700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcff4003a70 0x7fd0081ed890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:22.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd007fff700 1 -- 172.21.15.7:0/2307801643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00806f6b0 con 0x7fd008147560 2023-03-03T14:02:22.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd006ffd700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0081ddaf0 0x7fd008147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:02:22.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd007fff700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008147560 0x7fd0081ea0b0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fd008070930 tx=0x7fd00000e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:02:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fcffffff700 1 -- 172.21.15.7:0/2307801643 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd000001af0 con 0x7fd008147560 2023-03-03T14:02:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fcffffff700 1 -- 172.21.15.7:0/2307801643 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00000ec70 con 0x7fd008147560 2023-03-03T14:02:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd00de8a700 1 -- 172.21.15.7:0/2307801643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd008155160 con 0x7fd008147560 2023-03-03T14:02:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fcffffff700 1 -- 172.21.15.7:0/2307801643 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd000013420 con 0x7fd008147560 2023-03-03T14:02:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.413+0000 7fd00de8a700 1 -- 172.21.15.7:0/2307801643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00806f990 con 0x7fd008147560 2023-03-03T14:02:22.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.414+0000 7fd00de8a700 1 -- 172.21.15.7:0/2307801643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0080c4cd0 con 0x7fd008147560 2023-03-03T14:02:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.415+0000 7fcffffff700 1 -- 172.21.15.7:0/2307801643 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd0000135a0 con 0x7fd008147560 2023-03-03T14:02:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.415+0000 7fcffffff700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 0x7fcfe806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.415+0000 7fcffffff700 1 -- 172.21.15.7:0/2307801643 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd0000868a0 con 0x7fd008147560 2023-03-03T14:02:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.415+0000 7fd0077fe700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 0x7fcfe806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:22.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.416+0000 7fd0077fe700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 0x7fcfe806a7a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fcff80090b0 tx=0x7fcff8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:02:22.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.420+0000 7fcffffff700 1 -- 172.21.15.7:0/2307801643 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd000052be0 con 0x7fd008147560 2023-03-03T14:02:22.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.658+0000 7fd00de8a700 1 -- 172.21.15.7:0/2307801643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd00804e6e0 con 0x7fd008147560 2023-03-03T14:02:22.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.659+0000 7fcffffff700 1 -- 172.21.15.7:0/2307801643 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd000056200 con 0x7fd008147560 2023-03-03T14:02:22.661 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:02:22.661 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:02:22.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 -- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 msgr2=0x7fcfe806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:22.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 0x7fcfe806a7a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fcff80090b0 tx=0x7fcff8008030 comp rx=0 tx=0).stop 2023-03-03T14:02:22.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 -- 172.21.15.7:0/2307801643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008147560 msgr2=0x7fd0081ea0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:22.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008147560 0x7fd0081ea0b0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fd008070930 tx=0x7fd00000e440 comp rx=0 tx=0).stop 2023-03-03T14:02:22.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 -- 172.21.15.7:0/2307801643 shutdown_connections 2023-03-03T14:02:22.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcfe80682f0 0x7fcfe806a7a0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:22.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008147560 0x7fd0081ea0b0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:22.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcff4003a70 0x7fd0081ed890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:22.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 --2- 172.21.15.7:0/2307801643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0081ddaf0 0x7fd008147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 -- 172.21.15.7:0/2307801643 >> 172.21.15.7:0/2307801643 conn(0x7fd0080c8900 msgr2=0x7fd0080ad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:02:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 -- 172.21.15.7:0/2307801643 shutdown_connections 2023-03-03T14:02:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:22.661+0000 7fd00de8a700 1 -- 172.21.15.7:0/2307801643 wait complete. 2023-03-03T14:02:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:02:22.674 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-03-03T14:02:48.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:02:48.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.809+0000 7ff66ab68700 1 Processor -- start 2023-03-03T14:02:48.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.810+0000 7ff66ab68700 1 -- start start 2023-03-03T14:02:48.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.810+0000 7ff66ab68700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff66414b020 0x7ff6640717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:48.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.810+0000 7ff66ab68700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6640aad20 0x7ff664157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:48.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.810+0000 7ff66ab68700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6641588f0 con 0x7ff664071d20 2023-03-03T14:02:48.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.810+0000 7ff66ab68700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6640728b0 con 0x7ff6640aad20 2023-03-03T14:02:48.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.810+0000 7ff66ab68700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff664155070 con 0x7ff66414b020 2023-03-03T14:02:48.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.810+0000 7ff663fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff6640aad20 0x7ff664157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.810+0000 7ff669105700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff664071d20 0x7ff6641581e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41012/0 (socket says 172.21.15.7:41012) 2023-03-03T14:02:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff669105700 1 -- 172.21.15.7:0/2922805394 learned_addr learned my addr 172.21.15.7:0/2922805394 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:02:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff668904700 1 --2- 172.21.15.7:0/2922805394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff66414b020 0x7ff6640717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff663fff700 1 -- 172.21.15.7:0/2922805394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff66414b020 msgr2=0x7ff6640717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:02:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff663fff700 1 --2- 172.21.15.7:0/2922805394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff66414b020 0x7ff6640717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff663fff700 1 -- 172.21.15.7:0/2922805394 >> v1:172.21.15.7:6789/0 conn(0x7ff664071d20 legacy=0x7ff6641581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:48.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff6637fe700 1 -- 172.21.15.7:0/2922805394 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590753753 0 0) 0x7ff6641588f0 con 0x7ff664071d20 2023-03-03T14:02:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff663fff700 1 -- 172.21.15.7:0/2922805394 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66406f5b0 con 0x7ff6640aad20 2023-03-03T14:02:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff663fff700 1 --2- 172.21.15.7:0/2922805394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 0x7ff664157ca0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7ff650009840 tx=0x7ff650009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c45c84961abed96 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:02:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff6637fe700 1 -- 172.21.15.7:0/2922805394 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650011830 con 0x7ff6640aad20 2023-03-03T14:02:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff6637fe700 1 -- 172.21.15.7:0/2922805394 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff650011990 con 0x7ff6640aad20 2023-03-03T14:02:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.811+0000 7ff6637fe700 1 -- 172.21.15.7:0/2922805394 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650011d20 con 0x7ff6640aad20 2023-03-03T14:02:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.812+0000 7ff66ab68700 1 -- 172.21.15.7:0/2922805394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 msgr2=0x7ff664157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.812+0000 7ff66ab68700 1 --2- 172.21.15.7:0/2922805394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 0x7ff664157ca0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7ff650009840 tx=0x7ff650009b50 comp rx=0 tx=0).stop 2023-03-03T14:02:48.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.812+0000 7ff66ab68700 1 -- 172.21.15.7:0/2922805394 shutdown_connections 2023-03-03T14:02:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.812+0000 7ff66ab68700 1 --2- 172.21.15.7:0/2922805394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff66414b020 0x7ff6640717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.812+0000 7ff66ab68700 1 --2- 172.21.15.7:0/2922805394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 0x7ff664157ca0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.812+0000 7ff66ab68700 1 -- 172.21.15.7:0/2922805394 >> 172.21.15.7:0/2922805394 conn(0x7ff6640c8900 msgr2=0x7ff6640c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:02:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.812+0000 7ff66ab68700 1 -- 172.21.15.7:0/2922805394 shutdown_connections 2023-03-03T14:02:48.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.812+0000 7ff66ab68700 1 -- 172.21.15.7:0/2922805394 wait complete. 2023-03-03T14:02:48.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.813+0000 7ff66ab68700 1 Processor -- start 2023-03-03T14:02:48.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.813+0000 7ff66ab68700 1 -- start start 2023-03-03T14:02:48.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff66ab68700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff664071d20 0x7ff664142a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:48.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff66ab68700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 0x7ff664153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:48.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff66ab68700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66414b020 0x7ff66413f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:48.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff66ab68700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6641486d0 con 0x7ff6640aad20 2023-03-03T14:02:48.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff66ab68700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff664152dc0 con 0x7ff66414b020 2023-03-03T14:02:48.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff66ab68700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff664153380 con 0x7ff664071d20 2023-03-03T14:02:48.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff663fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 0x7ff664153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:48.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff663fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 0x7ff664153ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47036/0 (socket says 172.21.15.7:47036) 2023-03-03T14:02:48.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff663fff700 1 -- 172.21.15.7:0/338258274 learned_addr learned my addr 172.21.15.7:0/338258274 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:02:48.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff668904700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff664071d20 0x7ff664142a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:48.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff669105700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66414b020 0x7ff66413f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:48.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff663fff700 1 -- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff664071d20 msgr2=0x7ff664142a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:48.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff663fff700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff664071d20 0x7ff664142a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:48.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff663fff700 1 -- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66414b020 msgr2=0x7ff66413f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:48.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff663fff700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66414b020 0x7ff66413f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:48.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff663fff700 1 -- 172.21.15.7:0/338258274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6500094a0 con 0x7ff6640aad20 2023-03-03T14:02:48.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.814+0000 7ff669105700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66414b020 0x7ff66413f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:02:48.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.815+0000 7ff663fff700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 0x7ff664153ce0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7ff6500090b0 tx=0x7ff650004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:02:48.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.815+0000 7ff6617fa700 1 -- 172.21.15.7:0/338258274 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6500036a0 con 0x7ff6640aad20 2023-03-03T14:02:48.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.815+0000 7ff6617fa700 1 -- 172.21.15.7:0/338258274 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65001e070 con 0x7ff6640aad20 2023-03-03T14:02:48.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.815+0000 7ff66ab68700 1 -- 172.21.15.7:0/338258274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66406fb70 con 0x7ff6640aad20 2023-03-03T14:02:48.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.815+0000 7ff6617fa700 1 -- 172.21.15.7:0/338258274 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650003a20 con 0x7ff6640aad20 2023-03-03T14:02:48.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.815+0000 7ff66ab68700 1 -- 172.21.15.7:0/338258274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66406f890 con 0x7ff6640aad20 2023-03-03T14:02:48.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.816+0000 7ff66ab68700 1 -- 172.21.15.7:0/338258274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff664155350 con 0x7ff6640aad20 2023-03-03T14:02:48.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.816+0000 7ff6617fa700 1 -- 172.21.15.7:0/338258274 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff65000f050 con 0x7ff6640aad20 2023-03-03T14:02:48.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.817+0000 7ff6617fa700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6480682f0 0x7ff64806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:02:48.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.817+0000 7ff6617fa700 1 -- 172.21.15.7:0/338258274 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff650020070 con 0x7ff6640aad20 2023-03-03T14:02:48.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.817+0000 7ff668904700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6480682f0 0x7ff64806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:02:48.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.818+0000 7ff668904700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6480682f0 0x7ff64806a7a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7ff6640c8450 tx=0x7ff658008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:02:48.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:48.821+0000 7ff6617fa700 1 -- 172.21.15.7:0/338258274 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff650053900 con 0x7ff6640aad20 2023-03-03T14:02:49.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.057+0000 7ff66ab68700 1 -- 172.21.15.7:0/338258274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6641427a0 con 0x7ff6640aad20 2023-03-03T14:02:49.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.058+0000 7ff6617fa700 1 -- 172.21.15.7:0/338258274 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff6641427a0 con 0x7ff6640aad20 2023-03-03T14:02:49.059 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:02:49.060 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:02:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.059+0000 7ff66ab68700 1 -- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6480682f0 msgr2=0x7ff64806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.059+0000 7ff66ab68700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6480682f0 0x7ff64806a7a0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7ff6640c8450 tx=0x7ff658008040 comp rx=0 tx=0).stop 2023-03-03T14:02:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.059+0000 7ff66ab68700 1 -- 172.21.15.7:0/338258274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 msgr2=0x7ff664153ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:02:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.059+0000 7ff66ab68700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 0x7ff664153ce0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7ff6500090b0 tx=0x7ff650004750 comp rx=0 tx=0).stop 2023-03-03T14:02:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.059+0000 7ff66ab68700 1 -- 172.21.15.7:0/338258274 shutdown_connections 2023-03-03T14:02:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.059+0000 7ff66ab68700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66414b020 0x7ff66413f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.059+0000 7ff66ab68700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff664071d20 0x7ff664142a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.059+0000 7ff66ab68700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6480682f0 0x7ff64806a7a0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.059+0000 7ff66ab68700 1 --2- 172.21.15.7:0/338258274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6640aad20 0x7ff664153ce0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:02:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.059+0000 7ff66ab68700 1 -- 172.21.15.7:0/338258274 >> 172.21.15.7:0/338258274 conn(0x7ff6640c8900 msgr2=0x7ff66414a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:02:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.060+0000 7ff66ab68700 1 -- 172.21.15.7:0/338258274 shutdown_connections 2023-03-03T14:02:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:02:49.060+0000 7ff66ab68700 1 -- 172.21.15.7:0/338258274 wait complete. 2023-03-03T14:02:49.063 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:02:49.073 INFO:tasks.mds_thrash.fs.[a]:waiting for 33 secs before thrashing 2023-03-03T14:02:49.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:49.177 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:02:59.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:03:00.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.087+0000 7fabcce9d700 1 Processor -- start 2023-03-03T14:03:00.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.087+0000 7fabcce9d700 1 -- start start 2023-03-03T14:03:00.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.088+0000 7fabcce9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc8155410 0x7fabc80bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:00.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.088+0000 7fabcce9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabc80c5b00 0x7fabc80bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:00.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.088+0000 7fabcce9d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc80c08d0 con 0x7fabc8151e80 2023-03-03T14:03:00.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.088+0000 7fabcce9d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc8152250 con 0x7fabc80c5b00 2023-03-03T14:03:00.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.088+0000 7fabcce9d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80c0fb0 con 0x7fabc8155410 2023-03-03T14:03:00.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.088+0000 7fabc6d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabc80c5b00 0x7fabc80bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:00.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.088+0000 7fabc659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fabc8151e80 0x7fabc80c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41026/0 (socket says 172.21.15.7:41026) 2023-03-03T14:03:00.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.088+0000 7fabc659c700 1 -- 172.21.15.7:0/678848552 learned_addr learned my addr 172.21.15.7:0/678848552 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:03:00.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.088+0000 7fabc5d9b700 1 --2- 172.21.15.7:0/678848552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc8155410 0x7fabc80bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:00.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabc6d9d700 1 -- 172.21.15.7:0/678848552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc8155410 msgr2=0x7fabc80bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:03:00.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabc6d9d700 1 --2- 172.21.15.7:0/678848552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc8155410 0x7fabc80bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:00.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabc6d9d700 1 -- 172.21.15.7:0/678848552 >> v1:172.21.15.7:6789/0 conn(0x7fabc8151e80 legacy=0x7fabc80c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:00.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabc559a700 1 -- 172.21.15.7:0/678848552 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321490964 0 0) 0x7fabc80c08d0 con 0x7fabc8151e80 2023-03-03T14:03:00.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabc6d9d700 1 -- 172.21.15.7:0/678848552 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc8154930 con 0x7fabc80c5b00 2023-03-03T14:03:00.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabc6d9d700 1 --2- 172.21.15.7:0/678848552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc80c5b00 0x7fabc80bfe20 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009dc0 tx=0x7fabc000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93eb9968519d34fb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:03:00.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabc559a700 1 -- 172.21.15.7:0/678848552 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc000e070 con 0x7fabc80c5b00 2023-03-03T14:03:00.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabc559a700 1 -- 172.21.15.7:0/678848552 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc000bb90 con 0x7fabc80c5b00 2023-03-03T14:03:00.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabc559a700 1 -- 172.21.15.7:0/678848552 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0012430 con 0x7fabc80c5b00 2023-03-03T14:03:00.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabcce9d700 1 -- 172.21.15.7:0/678848552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc80c5b00 msgr2=0x7fabc80bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:00.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.089+0000 7fabcce9d700 1 --2- 172.21.15.7:0/678848552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc80c5b00 0x7fabc80bfe20 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009dc0 tx=0x7fabc000b330 comp rx=0 tx=0).stop 2023-03-03T14:03:00.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.090+0000 7fabcce9d700 1 -- 172.21.15.7:0/678848552 shutdown_connections 2023-03-03T14:03:00.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.090+0000 7fabcce9d700 1 --2- 172.21.15.7:0/678848552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc80c5b00 0x7fabc80bfe20 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:00.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.090+0000 7fabcce9d700 1 --2- 172.21.15.7:0/678848552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc8155410 0x7fabc80bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:00.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.090+0000 7fabcce9d700 1 -- 172.21.15.7:0/678848552 >> 172.21.15.7:0/678848552 conn(0x7fabc8071630 msgr2=0x7fabc806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:03:00.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.090+0000 7fabcce9d700 1 -- 172.21.15.7:0/678848552 shutdown_connections 2023-03-03T14:03:00.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.090+0000 7fabcce9d700 1 -- 172.21.15.7:0/678848552 wait complete. 2023-03-03T14:03:00.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.091+0000 7fabcce9d700 1 Processor -- start 2023-03-03T14:03:00.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.091+0000 7fabcce9d700 1 -- start start 2023-03-03T14:03:00.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabcce9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc80c5b00 0x7fabc80c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:00.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabcce9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabc8151e80 0x7fabc8157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:00.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabcce9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc8155410 0x7fabc814ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:00.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabcce9d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80ca980 con 0x7fabc8155410 2023-03-03T14:03:00.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabcce9d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80ca480 con 0x7fabc8151e80 2023-03-03T14:03:00.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabcce9d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc8158b00 con 0x7fabc80c5b00 2023-03-03T14:03:00.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc8155410 0x7fabc814ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:00.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc8155410 0x7fabc814ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47056/0 (socket says 172.21.15.7:47056) 2023-03-03T14:03:00.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc6d9d700 1 -- 172.21.15.7:0/2217754866 learned_addr learned my addr 172.21.15.7:0/2217754866 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:03:00.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc6d9d700 1 -- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc80c5b00 msgr2=0x7fabc80c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:00.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc659c700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc80c5b00 0x7fabc80c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:00.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc5d9b700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabc8151e80 0x7fabc8157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc6d9d700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc80c5b00 0x7fabc80c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc6d9d700 1 -- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabc8151e80 msgr2=0x7fabc8157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc6d9d700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabc8151e80 0x7fabc8157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc6d9d700 1 -- 172.21.15.7:0/2217754866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc0009a70 con 0x7fabc8155410 2023-03-03T14:03:00.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc659c700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc80c5b00 0x7fabc80c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:03:00.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc5d9b700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabc8151e80 0x7fabc8157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:03:00.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.092+0000 7fabc6d9d700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc8155410 0x7fabc814ce60 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fabc0014040 tx=0x7fabc0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:03:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.093+0000 7fabb6ffd700 1 -- 172.21.15.7:0/2217754866 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0012430 con 0x7fabc8155410 2023-03-03T14:03:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.093+0000 7fabb6ffd700 1 -- 172.21.15.7:0/2217754866 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc0020070 con 0x7fabc8155410 2023-03-03T14:03:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.093+0000 7fabcce9d700 1 -- 172.21.15.7:0/2217754866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc8154ef0 con 0x7fabc8155410 2023-03-03T14:03:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.093+0000 7fabb6ffd700 1 -- 172.21.15.7:0/2217754866 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc0004280 con 0x7fabc8155410 2023-03-03T14:03:00.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.093+0000 7fabcce9d700 1 -- 172.21.15.7:0/2217754866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc8154c10 con 0x7fabc8155410 2023-03-03T14:03:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.094+0000 7fabcce9d700 1 -- 172.21.15.7:0/2217754866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc80c5220 con 0x7fabc8155410 2023-03-03T14:03:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.094+0000 7fabb6ffd700 1 -- 172.21.15.7:0/2217754866 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fabc00043e0 con 0x7fabc8155410 2023-03-03T14:03:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.095+0000 7fabb6ffd700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faba8068320 0x7faba806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.095+0000 7fabb6ffd700 1 -- 172.21.15.7:0/2217754866 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fabc0022070 con 0x7fabc8155410 2023-03-03T14:03:00.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.095+0000 7fabc659c700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faba8068320 0x7faba806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:00.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.095+0000 7fabc659c700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faba8068320 0x7faba806a7d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fabbc007fd0 tx=0x7fabbc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:03:00.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.099+0000 7fabb6ffd700 1 -- 172.21.15.7:0/2217754866 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fabc0054b40 con 0x7fabc8155410 2023-03-03T14:03:00.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.336+0000 7fabcce9d700 1 -- 172.21.15.7:0/2217754866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabc8074bb0 con 0x7fabc8155410 2023-03-03T14:03:00.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.336+0000 7fabb6ffd700 1 -- 172.21.15.7:0/2217754866 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fabc0058160 con 0x7fabc8155410 2023-03-03T14:03:00.338 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:03:00.339 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:03:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 -- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faba8068320 msgr2=0x7faba806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faba8068320 0x7faba806a7d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fabbc007fd0 tx=0x7fabbc009040 comp rx=0 tx=0).stop 2023-03-03T14:03:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 -- 172.21.15.7:0/2217754866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc8155410 msgr2=0x7fabc814ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc8155410 0x7fabc814ce60 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fabc0014040 tx=0x7fabc0004ad0 comp rx=0 tx=0).stop 2023-03-03T14:03:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 -- 172.21.15.7:0/2217754866 shutdown_connections 2023-03-03T14:03:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faba8068320 0x7faba806a7d0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabc8155410 0x7fabc814ce60 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabc80c5b00 0x7fabc80c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 --2- 172.21.15.7:0/2217754866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabc8151e80 0x7fabc8157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 -- 172.21.15.7:0/2217754866 >> 172.21.15.7:0/2217754866 conn(0x7fabc8071630 msgr2=0x7fabc81536c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:03:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.338+0000 7fabcce9d700 1 -- 172.21.15.7:0/2217754866 shutdown_connections 2023-03-03T14:03:00.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:00.339+0000 7fabcce9d700 1 -- 172.21.15.7:0/2217754866 wait complete. 2023-03-03T14:03:00.342 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:03:00.352 INFO:tasks.mds_thrash.fs.[b]:waiting for 70 secs before thrashing 2023-03-03T14:03:19.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:19.240 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:22.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:03:22.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.210+0000 7f1711cd3700 1 Processor -- start 2023-03-03T14:03:22.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.210+0000 7f1711cd3700 1 -- start start 2023-03-03T14:03:22.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f1711cd3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 0x7f170c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:22.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f1711cd3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f170c073c30 0x7f170c15cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:22.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f1711cd3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170c0c2730 con 0x7f170c0ac9e0 2023-03-03T14:03:22.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f1711cd3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f170c0c22b0 con 0x7f170c073c30 2023-03-03T14:03:22.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f1711cd3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0c2980 con 0x7f170c14fa20 2023-03-03T14:03:22.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f170bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f170c073c30 0x7f170c15cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:22.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f170bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f170c073c30 0x7f170c15cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47070/0 (socket says 172.21.15.7:47070) 2023-03-03T14:03:22.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f170bfff700 1 -- 172.21.15.7:0/3537532469 learned_addr learned my addr 172.21.15.7:0/3537532469 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:03:22.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f170b7fe700 1 --2- 172.21.15.7:0/3537532469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 0x7f170c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:22.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f170bfff700 1 -- 172.21.15.7:0/3537532469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 msgr2=0x7f170c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:03:22.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f170bfff700 1 --2- 172.21.15.7:0/3537532469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 0x7f170c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:22.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f170bfff700 1 -- 172.21.15.7:0/3537532469 >> v1:172.21.15.143:6789/0 conn(0x7f170c0ac9e0 legacy=0x7f170c1524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:22.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.211+0000 7f170bfff700 1 -- 172.21.15.7:0/3537532469 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170c155a50 con 0x7f170c073c30 2023-03-03T14:03:22.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f170bfff700 1 --2- 172.21.15.7:0/3537532469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 0x7f170c15cd80 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1704009e50 tx=0x7f170400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797f069c1f1bc39c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:03:22.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f170a7fc700 1 -- 172.21.15.7:0/3537532469 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170400e070 con 0x7f170c073c30 2023-03-03T14:03:22.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f170a7fc700 1 -- 172.21.15.7:0/3537532469 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170400b910 con 0x7f170c073c30 2023-03-03T14:03:22.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f170a7fc700 1 -- 172.21.15.7:0/3537532469 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170400bcc0 con 0x7f170c073c30 2023-03-03T14:03:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f1711cd3700 1 -- 172.21.15.7:0/3537532469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 msgr2=0x7f170c15cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f1711cd3700 1 --2- 172.21.15.7:0/3537532469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 0x7f170c15cd80 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1704009e50 tx=0x7f170400b330 comp rx=0 tx=0).stop 2023-03-03T14:03:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f1711cd3700 1 -- 172.21.15.7:0/3537532469 shutdown_connections 2023-03-03T14:03:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f1711cd3700 1 --2- 172.21.15.7:0/3537532469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 0x7f170c15cd80 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f1711cd3700 1 --2- 172.21.15.7:0/3537532469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 0x7f170c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f1711cd3700 1 -- 172.21.15.7:0/3537532469 >> 172.21.15.7:0/3537532469 conn(0x7f170c0a94d0 msgr2=0x7f170c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:03:22.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f1711cd3700 1 -- 172.21.15.7:0/3537532469 shutdown_connections 2023-03-03T14:03:22.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.212+0000 7f1711cd3700 1 -- 172.21.15.7:0/3537532469 wait complete. 2023-03-03T14:03:22.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.214+0000 7f1711cd3700 1 Processor -- start 2023-03-03T14:03:22.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.214+0000 7f1711cd3700 1 -- start start 2023-03-03T14:03:22.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.214+0000 7f1711cd3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 0x7f170c06a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.214+0000 7f1711cd3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f170c0ac9e0 0x7f170c1e86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.214+0000 7f1711cd3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 0x7f170c1e8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.214+0000 7f1711cd3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0c5c50 con 0x7f170c073c30 2023-03-03T14:03:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.214+0000 7f1711cd3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170c0c7120 con 0x7f170c0ac9e0 2023-03-03T14:03:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.214+0000 7f1711cd3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170c1e9160 con 0x7f170c14fa20 2023-03-03T14:03:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 0x7f170c06a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:22.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 0x7f170c06a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47072/0 (socket says 172.21.15.7:47072) 2023-03-03T14:03:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170b7fe700 1 -- 172.21.15.7:0/4180876919 learned_addr learned my addr 172.21.15.7:0/4180876919 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:03:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170bfff700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 0x7f170c1e8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170affd700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f170c0ac9e0 0x7f170c1e86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170b7fe700 1 -- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 msgr2=0x7f170c1e8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170b7fe700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 0x7f170c1e8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170b7fe700 1 -- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f170c0ac9e0 msgr2=0x7f170c1e86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:22.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170b7fe700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f170c0ac9e0 0x7f170c1e86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170b7fe700 1 -- 172.21.15.7:0/4180876919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1704009b30 con 0x7f170c073c30 2023-03-03T14:03:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170bfff700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 0x7f170c1e8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:03:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170affd700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f170c0ac9e0 0x7f170c1e86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:03:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f170b7fe700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 0x7f170c06a950 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00e8a0 tx=0x7f16fc00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:03:22.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f1703fff700 1 -- 172.21.15.7:0/4180876919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc009590 con 0x7f170c073c30 2023-03-03T14:03:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f1703fff700 1 -- 172.21.15.7:0/4180876919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc00eea0 con 0x7f170c073c30 2023-03-03T14:03:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f1711cd3700 1 -- 172.21.15.7:0/4180876919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170c156010 con 0x7f170c073c30 2023-03-03T14:03:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f1703fff700 1 -- 172.21.15.7:0/4180876919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc00ca50 con 0x7f170c073c30 2023-03-03T14:03:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.215+0000 7f1711cd3700 1 -- 172.21.15.7:0/4180876919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170c155d30 con 0x7f170c073c30 2023-03-03T14:03:22.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.216+0000 7f1711cd3700 1 -- 172.21.15.7:0/4180876919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170c1562f0 con 0x7f170c073c30 2023-03-03T14:03:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.217+0000 7f1703fff700 1 -- 172.21.15.7:0/4180876919 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f16fc019030 con 0x7f170c073c30 2023-03-03T14:03:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.217+0000 7f1703fff700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16ec0682f0 0x7f16ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:03:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.217+0000 7f1703fff700 1 -- 172.21.15.7:0/4180876919 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f16fc086140 con 0x7f170c073c30 2023-03-03T14:03:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.217+0000 7f170affd700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16ec0682f0 0x7f16ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:03:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.218+0000 7f170affd700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16ec0682f0 0x7f16ec06a7a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f16f4000f60 tx=0x7f16f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:03:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.222+0000 7f1703fff700 1 -- 172.21.15.7:0/4180876919 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f16fc052370 con 0x7f170c073c30 2023-03-03T14:03:22.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.463+0000 7f1711cd3700 1 -- 172.21.15.7:0/4180876919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f170c074bb0 con 0x7f170c073c30 2023-03-03T14:03:22.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.464+0000 7f1703fff700 1 -- 172.21.15.7:0/4180876919 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f16fc055990 con 0x7f170c073c30 2023-03-03T14:03:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:03:22.467 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:03:22.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 -- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16ec0682f0 msgr2=0x7f16ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:22.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16ec0682f0 0x7f16ec06a7a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f16f4000f60 tx=0x7f16f4009040 comp rx=0 tx=0).stop 2023-03-03T14:03:22.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 -- 172.21.15.7:0/4180876919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 msgr2=0x7f170c06a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:03:22.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 0x7f170c06a950 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00e8a0 tx=0x7f16fc00ec60 comp rx=0 tx=0).stop 2023-03-03T14:03:22.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 -- 172.21.15.7:0/4180876919 shutdown_connections 2023-03-03T14:03:22.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f170c14fa20 0x7f170c1e8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:22.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16ec0682f0 0x7f16ec06a7a0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:22.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f170c073c30 0x7f170c06a950 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:22.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 --2- 172.21.15.7:0/4180876919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f170c0ac9e0 0x7f170c1e86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:03:22.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 -- 172.21.15.7:0/4180876919 >> 172.21.15.7:0/4180876919 conn(0x7f170c0a94d0 msgr2=0x7f170c0ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:03:22.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 -- 172.21.15.7:0/4180876919 shutdown_connections 2023-03-03T14:03:22.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:03:22.466+0000 7f1711cd3700 1 -- 172.21.15.7:0/4180876919 wait complete. 2023-03-03T14:03:22.470 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:03:22.480 INFO:tasks.mds_thrash.fs.[a]:waiting for 89 secs before thrashing 2023-03-03T14:03:49.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:03:49.378 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:10.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:04:10.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.490+0000 7faed04f1700 1 Processor -- start 2023-03-03T14:04:10.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.490+0000 7faed04f1700 1 -- start start 2023-03-03T14:04:10.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faed04f1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faec806a2b0 0x7faec806a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:10.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faed04f1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec80c1c10 con 0x7faec80ab0c0 2023-03-03T14:04:10.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faed04f1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec8154530 con 0x7faec8157a40 2023-03-03T14:04:10.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faed04f1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faec8154810 con 0x7faec806a2b0 2023-03-03T14:04:10.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faecda8c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faec806a2b0 0x7faec806a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:10.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faece28d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faec80ab0c0 0x7faec80c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41092/0 (socket says 172.21.15.7:41092) 2023-03-03T14:04:10.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faece28d700 1 -- 172.21.15.7:0/1834164646 learned_addr learned my addr 172.21.15.7:0/1834164646 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:04:10.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faecda8c700 1 -- 172.21.15.7:0/1834164646 >> v1:172.21.15.143:6789/0 conn(0x7faec8157a40 legacy=0x7faec8157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:04:10.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faecda8c700 1 -- 172.21.15.7:0/1834164646 >> v1:172.21.15.7:6789/0 conn(0x7faec80ab0c0 legacy=0x7faec80c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:10.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faecda8c700 1 -- 172.21.15.7:0/1834164646 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec8154af0 con 0x7faec806a2b0 2023-03-03T14:04:10.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.491+0000 7faecd28b700 1 -- 172.21.15.7:0/1834164646 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328594587 0 0) 0x7faeb8001110 con 0x7faec80ab0c0 2023-03-03T14:04:10.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.492+0000 7faecda8c700 1 --2- 172.21.15.7:0/1834164646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec806a2b0 0x7faec806a750 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7faec4009840 tx=0x7faec4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe931814dc16c92 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:04:10.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.492+0000 7faecd28b700 1 -- 172.21.15.7:0/1834164646 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4004230 con 0x7faec806a2b0 2023-03-03T14:04:10.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.492+0000 7faecd28b700 1 -- 172.21.15.7:0/1834164646 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec4004390 con 0x7faec806a2b0 2023-03-03T14:04:10.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.492+0000 7faecd28b700 1 -- 172.21.15.7:0/1834164646 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec40036a0 con 0x7faec806a2b0 2023-03-03T14:04:10.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.492+0000 7faed04f1700 1 -- 172.21.15.7:0/1834164646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec806a2b0 msgr2=0x7faec806a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:10.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.492+0000 7faed04f1700 1 --2- 172.21.15.7:0/1834164646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec806a2b0 0x7faec806a750 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7faec4009840 tx=0x7faec4009b50 comp rx=0 tx=0).stop 2023-03-03T14:04:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.492+0000 7faed04f1700 1 -- 172.21.15.7:0/1834164646 shutdown_connections 2023-03-03T14:04:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.492+0000 7faed04f1700 1 --2- 172.21.15.7:0/1834164646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec806a2b0 0x7faec806a750 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.492+0000 7faed04f1700 1 -- 172.21.15.7:0/1834164646 >> 172.21.15.7:0/1834164646 conn(0x7faec80a9430 msgr2=0x7faec80c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:04:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.492+0000 7faed04f1700 1 -- 172.21.15.7:0/1834164646 shutdown_connections 2023-03-03T14:04:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.493+0000 7faed04f1700 1 -- 172.21.15.7:0/1834164646 wait complete. 2023-03-03T14:04:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.494+0000 7faed04f1700 1 Processor -- start 2023-03-03T14:04:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.494+0000 7faed04f1700 1 -- start start 2023-03-03T14:04:10.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.494+0000 7faed04f1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faec806a2b0 0x7faec8072060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:10.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.494+0000 7faed04f1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faec80ab0c0 0x7faec80c3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:10.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.494+0000 7faed04f1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec8157a40 0x7faec806fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:10.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.494+0000 7faed04f1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec8071250 con 0x7faec8157a40 2023-03-03T14:04:10.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.494+0000 7faed04f1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec81ea2c0 con 0x7faec806a2b0 2023-03-03T14:04:10.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.494+0000 7faed04f1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec80726d0 con 0x7faec80ab0c0 2023-03-03T14:04:10.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faecea8e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec8157a40 0x7faec806fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:10.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faecea8e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec8157a40 0x7faec806fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47120/0 (socket says 172.21.15.7:47120) 2023-03-03T14:04:10.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faecea8e700 1 -- 172.21.15.7:0/2073372334 learned_addr learned my addr 172.21.15.7:0/2073372334 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:04:10.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faece28d700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faec806a2b0 0x7faec8072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:10.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faecda8c700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faec80ab0c0 0x7faec80c3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:10.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faecea8e700 1 -- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faec80ab0c0 msgr2=0x7faec80c3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:10.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faecea8e700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faec80ab0c0 0x7faec80c3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:10.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faecea8e700 1 -- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faec806a2b0 msgr2=0x7faec8072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:10.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faecea8e700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faec806a2b0 0x7faec8072060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faecea8e700 1 -- 172.21.15.7:0/2073372334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec40094a0 con 0x7faec8157a40 2023-03-03T14:04:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faece28d700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faec806a2b0 0x7faec8072060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:04:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faecea8e700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec8157a40 0x7faec806fa20 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7faec000c7a0 tx=0x7faec000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:04:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faebeffd700 1 -- 172.21.15.7:0/2073372334 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0003e10 con 0x7faec8157a40 2023-03-03T14:04:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faebeffd700 1 -- 172.21.15.7:0/2073372334 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec0005070 con 0x7faec8157a40 2023-03-03T14:04:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faebeffd700 1 -- 172.21.15.7:0/2073372334 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0005420 con 0x7faec8157a40 2023-03-03T14:04:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faed04f1700 1 -- 172.21.15.7:0/2073372334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec81550b0 con 0x7faec8157a40 2023-03-03T14:04:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.495+0000 7faed04f1700 1 -- 172.21.15.7:0/2073372334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec8154dd0 con 0x7faec8157a40 2023-03-03T14:04:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.496+0000 7faed04f1700 1 -- 172.21.15.7:0/2073372334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec81585f0 con 0x7faec8157a40 2023-03-03T14:04:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.497+0000 7faebeffd700 1 -- 172.21.15.7:0/2073372334 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faec0010070 con 0x7faec8157a40 2023-03-03T14:04:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.497+0000 7faebeffd700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeac0682f0 0x7faeac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:10.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.497+0000 7faebeffd700 1 -- 172.21.15.7:0/2073372334 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faec0085380 con 0x7faec8157a40 2023-03-03T14:04:10.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.497+0000 7faece28d700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeac0682f0 0x7faeac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:10.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.498+0000 7faece28d700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeac0682f0 0x7faeac06a7a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7faeb8007fd0 tx=0x7faeb8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:04:10.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.502+0000 7faebeffd700 1 -- 172.21.15.7:0/2073372334 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faec0051720 con 0x7faec8157a40 2023-03-03T14:04:10.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.738+0000 7faed04f1700 1 -- 172.21.15.7:0/2073372334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faec80729b0 con 0x7faec8157a40 2023-03-03T14:04:10.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.739+0000 7faebeffd700 1 -- 172.21.15.7:0/2073372334 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faec0017090 con 0x7faec8157a40 2023-03-03T14:04:10.741 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:04:10.741 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:04:10.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 -- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeac0682f0 msgr2=0x7faeac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:10.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeac0682f0 0x7faeac06a7a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7faeb8007fd0 tx=0x7faeb8009040 comp rx=0 tx=0).stop 2023-03-03T14:04:10.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 -- 172.21.15.7:0/2073372334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec8157a40 msgr2=0x7faec806fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:10.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec8157a40 0x7faec806fa20 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7faec000c7a0 tx=0x7faec000cb60 comp rx=0 tx=0).stop 2023-03-03T14:04:10.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 -- 172.21.15.7:0/2073372334 shutdown_connections 2023-03-03T14:04:10.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeac0682f0 0x7faeac06a7a0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:10.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faec8157a40 0x7faec806fa20 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:10.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faec806a2b0 0x7faec8072060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:10.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 --2- 172.21.15.7:0/2073372334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faec80ab0c0 0x7faec80c3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:10.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 -- 172.21.15.7:0/2073372334 >> 172.21.15.7:0/2073372334 conn(0x7faec80a9430 msgr2=0x7faec8157ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:04:10.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 -- 172.21.15.7:0/2073372334 shutdown_connections 2023-03-03T14:04:10.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:10.741+0000 7faed04f1700 1 -- 172.21.15.7:0/2073372334 wait complete. 2023-03-03T14:04:10.745 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:04:10.754 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-03-03T14:04:19.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:19.449 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:49.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:49.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:04:51.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:04:51.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.619+0000 7fd2aa938700 1 Processor -- start 2023-03-03T14:04:51.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.619+0000 7fd2aa938700 1 -- start start 2023-03-03T14:04:51.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.619+0000 7fd2aa938700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a414f9d0 0x7fd2a4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:51.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2aa938700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd2a414fda0 0x7fd2a4071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:51.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2aa938700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a40c2f30 con 0x7fd2a40abe20 2023-03-03T14:04:51.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2aa938700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a40c31d0 con 0x7fd2a414fda0 2023-03-03T14:04:51.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2aa938700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41524a0 con 0x7fd2a414f9d0 2023-03-03T14:04:51.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a8ed5700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd2a40abe20 0x7fd2a41529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41138/0 (socket says 172.21.15.7:41138) 2023-03-03T14:04:51.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a8ed5700 1 -- 172.21.15.7:0/2764697501 learned_addr learned my addr 172.21.15.7:0/2764697501 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:04:51.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a3fff700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a414f9d0 0x7fd2a4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:51.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a37fe700 1 --2- 172.21.15.7:0/2764697501 >> v2:172.21.15.143:3300/0 conn(0x7fd2a414fda0 0x7fd2a4071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:51.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573569873 0 0) 0x7fd2a40c2f30 con 0x7fd2a40abe20 2023-03-03T14:04:51.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd28c0035b0 con 0x7fd2a40abe20 2023-03-03T14:04:51.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2906151995 0 0) 0x7fd28c0035b0 con 0x7fd2a40abe20 2023-03-03T14:04:51.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a414f9d0 msgr2=0x7fd2a4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:04:51.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a414f9d0 0x7fd2a4156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:51.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 >> v2:172.21.15.143:3300/0 conn(0x7fd2a414fda0 msgr2=0x7fd2a4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:04:51.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 --2- 172.21.15.7:0/2764697501 >> v2:172.21.15.143:3300/0 conn(0x7fd2a414fda0 0x7fd2a4071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:51.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a40c34b0 con 0x7fd2a40abe20 2023-03-03T14:04:51.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd29c0019f0 con 0x7fd2a40abe20 2023-03-03T14:04:51.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 >> v1:172.21.15.7:6789/0 conn(0x7fd2a40abe20 legacy=0x7fd2a41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd28c003a70 0x7fd28c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a414f9d0 0x7fd28c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd28c004d90 0x7fd28c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40c31d0 con 0x7fd28c004d90 2023-03-03T14:04:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41524a0 con 0x7fd28c003a70 2023-03-03T14:04:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.620+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c00baf0 con 0x7fd2a414f9d0 2023-03-03T14:04:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a8ed5700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd28c004d90 0x7fd28c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a3fff700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd28c003a70 0x7fd28c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a37fe700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a414f9d0 0x7fd28c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a8ed5700 1 -- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a414f9d0 msgr2=0x7fd28c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:04:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a8ed5700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a414f9d0 0x7fd28c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a8ed5700 1 -- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd28c003a70 msgr2=0x7fd28c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:04:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a8ed5700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd28c003a70 0x7fd28c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a8ed5700 1 -- 172.21.15.7:0/2764697501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c0035b0 con 0x7fd28c004d90 2023-03-03T14:04:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a8ed5700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd28c004d90 0x7fd28c005200 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00a860 tx=0x7fd28c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c4f475e407268b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:04:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c012600 con 0x7fd28c004d90 2023-03-03T14:04:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c012780 con 0x7fd28c004d90 2023-03-03T14:04:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.621+0000 7fd2a2ffd700 1 -- 172.21.15.7:0/2764697501 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c012b30 con 0x7fd28c004d90 2023-03-03T14:04:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.622+0000 7fd2aa938700 1 -- 172.21.15.7:0/2764697501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd28c004d90 msgr2=0x7fd28c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.622+0000 7fd2aa938700 1 --2- 172.21.15.7:0/2764697501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd28c004d90 0x7fd28c005200 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00a860 tx=0x7fd28c005880 comp rx=0 tx=0).stop 2023-03-03T14:04:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.622+0000 7fd2a8ed5700 1 -- 172.21.15.7:0/2764697501 reap_dead start 2023-03-03T14:04:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.622+0000 7fd2aa938700 1 -- 172.21.15.7:0/2764697501 shutdown_connections 2023-03-03T14:04:51.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.622+0000 7fd2aa938700 1 -- 172.21.15.7:0/2764697501 >> 172.21.15.7:0/2764697501 conn(0x7fd2a40c8900 msgr2=0x7fd2a40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:04:51.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.622+0000 7fd2aa938700 1 -- 172.21.15.7:0/2764697501 shutdown_connections 2023-03-03T14:04:51.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.622+0000 7fd2aa938700 1 -- 172.21.15.7:0/2764697501 wait complete. 2023-03-03T14:04:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.623+0000 7fd2aa938700 1 Processor -- start 2023-03-03T14:04:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.623+0000 7fd2aa938700 1 -- start start 2023-03-03T14:04:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2aa938700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a41dd790 0x7fd2a40bd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2aa938700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a41ddb60 0x7fd2a406f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:51.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2aa938700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2a414f0f0 0x7fd2a406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:51.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2aa938700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40bd3d0 con 0x7fd2a414f0f0 2023-03-03T14:04:51.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2aa938700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41df030 con 0x7fd2a41dd790 2023-03-03T14:04:51.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2aa938700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a406ced0 con 0x7fd2a41ddb60 2023-03-03T14:04:51.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2a8ed5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2a414f0f0 0x7fd2a406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:51.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2a8ed5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2a414f0f0 0x7fd2a406da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47170/0 (socket says 172.21.15.7:47170) 2023-03-03T14:04:51.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2a8ed5700 1 -- 172.21.15.7:0/985383563 learned_addr learned my addr 172.21.15.7:0/985383563 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:04:51.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2a3fff700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a41dd790 0x7fd2a40bd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:51.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2a37fe700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a41ddb60 0x7fd2a406f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:51.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2a8ed5700 1 -- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a41ddb60 msgr2=0x7fd2a406f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:51.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2a8ed5700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a41ddb60 0x7fd2a406f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:51.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2a8ed5700 1 -- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a41dd790 msgr2=0x7fd2a40bd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:51.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2a8ed5700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a41dd790 0x7fd2a40bd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:51.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.624+0000 7fd2a8ed5700 1 -- 172.21.15.7:0/985383563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a40ac350 con 0x7fd2a414f0f0 2023-03-03T14:04:51.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.625+0000 7fd2a8ed5700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2a414f0f0 0x7fd2a406da10 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fd29c015040 tx=0x7fd29c001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:04:51.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.625+0000 7fd2a0ff9700 1 -- 172.21.15.7:0/985383563 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c01d5f0 con 0x7fd2a414f0f0 2023-03-03T14:04:51.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.625+0000 7fd2a0ff9700 1 -- 172.21.15.7:0/985383563 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c001360 con 0x7fd2a414f0f0 2023-03-03T14:04:51.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.625+0000 7fd2aa938700 1 -- 172.21.15.7:0/985383563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a40ab630 con 0x7fd2a414f0f0 2023-03-03T14:04:51.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.625+0000 7fd2a0ff9700 1 -- 172.21.15.7:0/985383563 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c01d9b0 con 0x7fd2a414f0f0 2023-03-03T14:04:51.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.625+0000 7fd2aa938700 1 -- 172.21.15.7:0/985383563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a40c3e10 con 0x7fd2a414f0f0 2023-03-03T14:04:51.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.626+0000 7fd2aa938700 1 -- 172.21.15.7:0/985383563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a4074bb0 con 0x7fd2a414f0f0 2023-03-03T14:04:51.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.626+0000 7fd2a0ff9700 1 -- 172.21.15.7:0/985383563 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd29c02e070 con 0x7fd2a414f0f0 2023-03-03T14:04:51.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.627+0000 7fd2a0ff9700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd290068320 0x7fd29006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:04:51.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.627+0000 7fd2a0ff9700 1 -- 172.21.15.7:0/985383563 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd29c0878e0 con 0x7fd2a414f0f0 2023-03-03T14:04:51.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.627+0000 7fd2a3fff700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd290068320 0x7fd29006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:04:51.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.627+0000 7fd2a3fff700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd290068320 0x7fd29006a7d0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fd294009be0 tx=0x7fd294009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:04:51.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.631+0000 7fd2a0ff9700 1 -- 172.21.15.7:0/985383563 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd29c053cc0 con 0x7fd2a414f0f0 2023-03-03T14:04:51.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.869+0000 7fd2aa938700 1 -- 172.21.15.7:0/985383563 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2a40bd580 con 0x7fd2a414f0f0 2023-03-03T14:04:51.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.870+0000 7fd2a0ff9700 1 -- 172.21.15.7:0/985383563 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd2a40bd580 con 0x7fd2a414f0f0 2023-03-03T14:04:51.872 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:04:51.872 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:04:51.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 -- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd290068320 msgr2=0x7fd29006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:51.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd290068320 0x7fd29006a7d0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fd294009be0 tx=0x7fd294009210 comp rx=0 tx=0).stop 2023-03-03T14:04:51.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 -- 172.21.15.7:0/985383563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2a414f0f0 msgr2=0x7fd2a406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:04:51.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2a414f0f0 0x7fd2a406da10 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fd29c015040 tx=0x7fd29c001d80 comp rx=0 tx=0).stop 2023-03-03T14:04:51.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 -- 172.21.15.7:0/985383563 shutdown_connections 2023-03-03T14:04:51.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd290068320 0x7fd29006a7d0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:51.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2a414f0f0 0x7fd2a406da10 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:51.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a41dd790 0x7fd2a40bd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:51.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 --2- 172.21.15.7:0/985383563 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2a41ddb60 0x7fd2a406f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:04:51.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 -- 172.21.15.7:0/985383563 >> 172.21.15.7:0/985383563 conn(0x7fd2a40c8900 msgr2=0x7fd2a40c6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:04:51.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 -- 172.21.15.7:0/985383563 shutdown_connections 2023-03-03T14:04:51.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:04:51.872+0000 7fd2aa938700 1 -- 172.21.15.7:0/985383563 wait complete. 2023-03-03T14:04:51.875 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:04:51.885 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2023-03-03T14:05:10.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:05:10.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.894+0000 7f300d09e700 1 Processor -- start 2023-03-03T14:05:10.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.894+0000 7f300d09e700 1 -- start start 2023-03-03T14:05:10.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.894+0000 7f300d09e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008151b20 0x7f3008071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:10.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.894+0000 7f300d09e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3008154f10 0x7f3008158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:10.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.894+0000 7f300d09e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3008071d60 0x7f3008158540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:10.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.894+0000 7f300d09e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3008151ef0 con 0x7f3008154f10 2023-03-03T14:05:10.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.894+0000 7f300d09e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3008157b20 con 0x7f3008071d60 2023-03-03T14:05:10.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.894+0000 7f300d09e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3008155410 con 0x7f3008151b20 2023-03-03T14:05:10.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.894+0000 7f300659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3008154f10 0x7f3008158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:10.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f300659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3008154f10 0x7f3008158000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47194/0 (socket says 172.21.15.7:47194) 2023-03-03T14:05:10.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f300659c700 1 -- 172.21.15.7:0/4023942307 learned_addr learned my addr 172.21.15.7:0/4023942307 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:05:10.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f300759e700 1 --2- 172.21.15.7:0/4023942307 >> v2:172.21.15.143:3300/0 conn(0x7f3008071d60 0x7f3008158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:10.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f300659c700 1 -- 172.21.15.7:0/4023942307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008151b20 msgr2=0x7f3008071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:05:10.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f3006d9d700 1 --2- 172.21.15.7:0/4023942307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008151b20 0x7f3008071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:10.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f300659c700 1 --2- 172.21.15.7:0/4023942307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008151b20 0x7f3008071820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:10.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f300659c700 1 -- 172.21.15.7:0/4023942307 >> v2:172.21.15.143:3300/0 conn(0x7f3008071d60 msgr2=0x7f3008158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:05:10.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f300659c700 1 --2- 172.21.15.7:0/4023942307 >> v2:172.21.15.143:3300/0 conn(0x7f3008071d60 0x7f3008158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:10.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f300659c700 1 -- 172.21.15.7:0/4023942307 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300806f640 con 0x7f3008154f10 2023-03-03T14:05:10.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f300759e700 1 --2- 172.21.15.7:0/4023942307 >> v2:172.21.15.143:3300/0 conn(0x7f3008071d60 0x7f3008158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:05:10.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f3006d9d700 1 --2- 172.21.15.7:0/4023942307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008151b20 0x7f3008071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:05:10.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f300659c700 1 --2- 172.21.15.7:0/4023942307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008154f10 0x7f3008158000 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f30080c84a0 tx=0x7f2ff0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f09f9445370df43 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:10.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f3005d9b700 1 -- 172.21.15.7:0/4023942307 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff000b870 con 0x7f3008154f10 2023-03-03T14:05:10.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f3005d9b700 1 -- 172.21.15.7:0/4023942307 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff000b9d0 con 0x7f3008154f10 2023-03-03T14:05:10.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.895+0000 7f3005d9b700 1 -- 172.21.15.7:0/4023942307 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff000bd60 con 0x7f3008154f10 2023-03-03T14:05:10.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.896+0000 7f300d09e700 1 -- 172.21.15.7:0/4023942307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008154f10 msgr2=0x7f3008158000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:10.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.896+0000 7f300d09e700 1 --2- 172.21.15.7:0/4023942307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008154f10 0x7f3008158000 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f30080c84a0 tx=0x7f2ff0009bd0 comp rx=0 tx=0).stop 2023-03-03T14:05:10.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.896+0000 7f300d09e700 1 -- 172.21.15.7:0/4023942307 shutdown_connections 2023-03-03T14:05:10.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.896+0000 7f300d09e700 1 --2- 172.21.15.7:0/4023942307 >> v2:172.21.15.143:3300/0 conn(0x7f3008071d60 0x7f3008158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:10.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.896+0000 7f300d09e700 1 --2- 172.21.15.7:0/4023942307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008151b20 0x7f3008071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:10.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.896+0000 7f300d09e700 1 --2- 172.21.15.7:0/4023942307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008154f10 0x7f3008158000 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:10.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.896+0000 7f300d09e700 1 -- 172.21.15.7:0/4023942307 >> 172.21.15.7:0/4023942307 conn(0x7f30080c8900 msgr2=0x7f30080c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:05:10.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.896+0000 7f300d09e700 1 -- 172.21.15.7:0/4023942307 shutdown_connections 2023-03-03T14:05:10.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.896+0000 7f300d09e700 1 -- 172.21.15.7:0/4023942307 wait complete. 2023-03-03T14:05:10.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.897+0000 7f300d09e700 1 Processor -- start 2023-03-03T14:05:10.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.897+0000 7f300d09e700 1 -- start start 2023-03-03T14:05:10.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f300d09e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008071d60 0x7f3008154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:10.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f300d09e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008151b20 0x7f30081e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:10.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f300d09e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008154f10 0x7f300814fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:10.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f300d09e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30080c8090 con 0x7f3008151b20 2023-03-03T14:05:10.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f300d09e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30081ed8d0 con 0x7f3008071d60 2023-03-03T14:05:10.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f300d09e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30081ea400 con 0x7f3008154f10 2023-03-03T14:05:10.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f300659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008151b20 0x7f30081e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:10.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f300659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008151b20 0x7f30081e9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47202/0 (socket says 172.21.15.7:47202) 2023-03-03T14:05:10.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f300659c700 1 -- 172.21.15.7:0/2808256739 learned_addr learned my addr 172.21.15.7:0/2808256739 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:05:10.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f3006d9d700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008071d60 0x7f3008154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:10.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.898+0000 7f300759e700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008154f10 0x7f300814fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:10.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f300659c700 1 -- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008154f10 msgr2=0x7f300814fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:10.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f300659c700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008154f10 0x7f300814fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:10.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f300659c700 1 -- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008071d60 msgr2=0x7f3008154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:10.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f300659c700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008071d60 0x7f3008154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:10.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f300659c700 1 -- 172.21.15.7:0/2808256739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff0009550 con 0x7f3008151b20 2023-03-03T14:05:10.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f300659c700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008151b20 0x7f30081e9ec0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0014040 tx=0x7f2ff0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:10.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f2fff7fe700 1 -- 172.21.15.7:0/2808256739 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0004250 con 0x7f3008151b20 2023-03-03T14:05:10.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f2fff7fe700 1 -- 172.21.15.7:0/2808256739 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff001e070 con 0x7f3008151b20 2023-03-03T14:05:10.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f2fff7fe700 1 -- 172.21.15.7:0/2808256739 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff00036a0 con 0x7f3008151b20 2023-03-03T14:05:10.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f300d09e700 1 -- 172.21.15.7:0/2808256739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300806fc00 con 0x7f3008151b20 2023-03-03T14:05:10.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.899+0000 7f300d09e700 1 -- 172.21.15.7:0/2808256739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300806f920 con 0x7f3008151b20 2023-03-03T14:05:10.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.900+0000 7f300d09e700 1 -- 172.21.15.7:0/2808256739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30080aae90 con 0x7f3008151b20 2023-03-03T14:05:10.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.901+0000 7f2fff7fe700 1 -- 172.21.15.7:0/2808256739 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ff0029020 con 0x7f3008151b20 2023-03-03T14:05:10.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.901+0000 7f2fff7fe700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fe80682f0 0x7f2fe806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:10.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.901+0000 7f2fff7fe700 1 -- 172.21.15.7:0/2808256739 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2ff00872e0 con 0x7f3008151b20 2023-03-03T14:05:10.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.901+0000 7f3006d9d700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fe80682f0 0x7f2fe806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:10.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.902+0000 7f3006d9d700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fe80682f0 0x7f2fe806a7a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f300814b2d0 tx=0x7f2ff8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:10.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:10.905+0000 7f2fff7fe700 1 -- 172.21.15.7:0/2808256739 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ff0053740 con 0x7f3008151b20 2023-03-03T14:05:11.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.143+0000 7f300d09e700 1 -- 172.21.15.7:0/2808256739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3008074bb0 con 0x7f3008151b20 2023-03-03T14:05:11.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.144+0000 7f2fff7fe700 1 -- 172.21.15.7:0/2808256739 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2ff0056d60 con 0x7f3008151b20 2023-03-03T14:05:11.146 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:05:11.146 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:05:11.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 -- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fe80682f0 msgr2=0x7f2fe806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:11.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fe80682f0 0x7f2fe806a7a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f300814b2d0 tx=0x7f2ff8008040 comp rx=0 tx=0).stop 2023-03-03T14:05:11.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 -- 172.21.15.7:0/2808256739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008151b20 msgr2=0x7f30081e9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:11.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008151b20 0x7f30081e9ec0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0014040 tx=0x7f2ff0004750 comp rx=0 tx=0).stop 2023-03-03T14:05:11.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 -- 172.21.15.7:0/2808256739 shutdown_connections 2023-03-03T14:05:11.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3008154f10 0x7f300814fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:11.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3008071d60 0x7f3008154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:11.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fe80682f0 0x7f2fe806a7a0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:11.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 --2- 172.21.15.7:0/2808256739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3008151b20 0x7f30081e9ec0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:11.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 -- 172.21.15.7:0/2808256739 >> 172.21.15.7:0/2808256739 conn(0x7f30080c8900 msgr2=0x7f30080c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:05:11.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 -- 172.21.15.7:0/2808256739 shutdown_connections 2023-03-03T14:05:11.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:11.146+0000 7f300d09e700 1 -- 172.21.15.7:0/2808256739 wait complete. 2023-03-03T14:05:11.150 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:05:11.160 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T14:05:19.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:19.725 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:29.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:05:29.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.295+0000 7f3c4f325700 1 Processor -- start 2023-03-03T14:05:29.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.296+0000 7f3c4f325700 1 -- start start 2023-03-03T14:05:29.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.296+0000 7f3c4f325700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c48071c30 0x7f3c480c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:29.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.296+0000 7f3c4f325700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3c48155950 0x7f3c48155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:29.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.296+0000 7f3c4f325700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c480713c0 con 0x7f3c480aac00 2023-03-03T14:05:29.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.296+0000 7f3c4f325700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c48156310 con 0x7f3c48155950 2023-03-03T14:05:29.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.296+0000 7f3c4f325700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c48152c70 con 0x7f3c48071c30 2023-03-03T14:05:29.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.297+0000 7f3c4d8c2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3c48155950 0x7f3c48155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:29.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.297+0000 7f3c4c8c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c48071c30 0x7f3c480c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:29.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.297+0000 7f3c4d0c1700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f3c480aac00 0x7f3c480beb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:44716/0 (socket says 172.21.15.7:44716) 2023-03-03T14:05:29.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.297+0000 7f3c4d8c2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3c48155950 0x7f3c48155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:42994/0 (socket says 172.21.15.7:42994) 2023-03-03T14:05:29.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.297+0000 7f3c4d0c1700 1 -- 172.21.15.7:0/346056136 learned_addr learned my addr 172.21.15.7:0/346056136 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:05:29.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.297+0000 7f3c4d8c2700 1 -- 172.21.15.7:0/346056136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c48071c30 msgr2=0x7f3c480c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:05:29.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.297+0000 7f3c4d8c2700 1 --2- 172.21.15.7:0/346056136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c48071c30 0x7f3c480c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:29.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.297+0000 7f3c4d8c2700 1 -- 172.21.15.7:0/346056136 >> v1:172.21.15.143:6789/0 conn(0x7f3c480aac00 legacy=0x7f3c480beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:29.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.297+0000 7f3c4d8c2700 1 -- 172.21.15.7:0/346056136 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c480c4930 con 0x7f3c48155950 2023-03-03T14:05:29.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.297+0000 7f3c4d8c2700 1 --2- 172.21.15.7:0/346056136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c48155950 0x7f3c48155dd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009d90 tx=0x7f3c4000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80c9b494f1b88012 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:29.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c3bfff700 1 -- 172.21.15.7:0/346056136 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4000e070 con 0x7f3c48155950 2023-03-03T14:05:29.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c3bfff700 1 -- 172.21.15.7:0/346056136 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4000bb20 con 0x7f3c48155950 2023-03-03T14:05:29.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c3bfff700 1 -- 172.21.15.7:0/346056136 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40012430 con 0x7f3c48155950 2023-03-03T14:05:29.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c4f325700 1 -- 172.21.15.7:0/346056136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c48155950 msgr2=0x7f3c48155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:29.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c4f325700 1 --2- 172.21.15.7:0/346056136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c48155950 0x7f3c48155dd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009d90 tx=0x7f3c4000b330 comp rx=0 tx=0).stop 2023-03-03T14:05:29.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c4f325700 1 -- 172.21.15.7:0/346056136 shutdown_connections 2023-03-03T14:05:29.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c4f325700 1 --2- 172.21.15.7:0/346056136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c48155950 0x7f3c48155dd0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:29.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c4f325700 1 --2- 172.21.15.7:0/346056136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c48071c30 0x7f3c480c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:29.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c4f325700 1 -- 172.21.15.7:0/346056136 >> 172.21.15.7:0/346056136 conn(0x7f3c480736f0 msgr2=0x7f3c48073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:05:29.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c4f325700 1 -- 172.21.15.7:0/346056136 shutdown_connections 2023-03-03T14:05:29.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.298+0000 7f3c4f325700 1 -- 172.21.15.7:0/346056136 wait complete. 2023-03-03T14:05:29.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.299+0000 7f3c4f325700 1 Processor -- start 2023-03-03T14:05:29.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.299+0000 7f3c4f325700 1 -- start start 2023-03-03T14:05:29.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4f325700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c48071c30 0x7f3c480adac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:29.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4f325700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c480aac00 0x7f3c4806e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:29.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4f325700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c48155950 0x7f3c481eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:29.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4f325700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c480ae5d0 con 0x7f3c48155950 2023-03-03T14:05:29.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4f325700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c481e7510 con 0x7f3c480aac00 2023-03-03T14:05:29.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4f325700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c480ae2f0 con 0x7f3c48071c30 2023-03-03T14:05:29.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4d8c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c48155950 0x7f3c481eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:29.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4c8c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c480aac00 0x7f3c4806e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:29.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4d0c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c48071c30 0x7f3c480adac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:29.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4c8c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c480aac00 0x7f3c4806e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:42998/0 (socket says 172.21.15.7:42998) 2023-03-03T14:05:29.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4c8c0700 1 -- 172.21.15.7:0/3449323261 learned_addr learned my addr 172.21.15.7:0/3449323261 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:05:29.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.300+0000 7f3c4d8c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c48155950 0x7f3c481eaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47226/0 (socket says 172.21.15.7:47226) 2023-03-03T14:05:29.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c4c8c0700 1 -- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c48071c30 msgr2=0x7f3c480adac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:29.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c4c8c0700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c48071c30 0x7f3c480adac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:29.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c4c8c0700 1 -- 172.21.15.7:0/3449323261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c48155950 msgr2=0x7f3c481eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:29.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c4c8c0700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c48155950 0x7f3c481eaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:29.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c4c8c0700 1 -- 172.21.15.7:0/3449323261 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c40009a70 con 0x7f3c480aac00 2023-03-03T14:05:29.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c4c8c0700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c480aac00 0x7f3c4806e560 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400e8a0 tx=0x7f3c4400ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:29.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c39ffb700 1 -- 172.21.15.7:0/3449323261 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44009590 con 0x7f3c480aac00 2023-03-03T14:05:29.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c39ffb700 1 -- 172.21.15.7:0/3449323261 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c44013680 con 0x7f3c480aac00 2023-03-03T14:05:29.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c4f325700 1 -- 172.21.15.7:0/3449323261 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c480c4b80 con 0x7f3c480aac00 2023-03-03T14:05:29.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c39ffb700 1 -- 172.21.15.7:0/3449323261 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c440139f0 con 0x7f3c480aac00 2023-03-03T14:05:29.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.301+0000 7f3c4f325700 1 -- 172.21.15.7:0/3449323261 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c48152360 con 0x7f3c480aac00 2023-03-03T14:05:29.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.302+0000 7f3c4f325700 1 -- 172.21.15.7:0/3449323261 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c48152620 con 0x7f3c480aac00 2023-03-03T14:05:29.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.303+0000 7f3c39ffb700 1 -- 172.21.15.7:0/3449323261 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3c44019030 con 0x7f3c480aac00 2023-03-03T14:05:29.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.303+0000 7f3c39ffb700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c2c0682f0 0x7f3c2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:29.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.303+0000 7f3c39ffb700 1 -- 172.21.15.7:0/3449323261 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3c44085e80 con 0x7f3c480aac00 2023-03-03T14:05:29.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.303+0000 7f3c4d0c1700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c2c0682f0 0x7f3c2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:29.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.304+0000 7f3c4d0c1700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c2c0682f0 0x7f3c2c06a7a0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c000f60 tx=0x7f3c3c007f10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:29.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.307+0000 7f3c39ffb700 1 -- 172.21.15.7:0/3449323261 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c44052220 con 0x7f3c480aac00 2023-03-03T14:05:29.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.549+0000 7f3c4f325700 1 -- 172.21.15.7:0/3449323261 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c48074bb0 con 0x7f3c480aac00 2023-03-03T14:05:29.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.550+0000 7f3c39ffb700 1 -- 172.21.15.7:0/3449323261 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3c44055840 con 0x7f3c480aac00 2023-03-03T14:05:29.552 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:05:29.553 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:05:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 -- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c2c0682f0 msgr2=0x7f3c2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c2c0682f0 0x7f3c2c06a7a0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c000f60 tx=0x7f3c3c007f10 comp rx=0 tx=0).stop 2023-03-03T14:05:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 -- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c480aac00 msgr2=0x7f3c4806e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c480aac00 0x7f3c4806e560 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400e8a0 tx=0x7f3c4400ebb0 comp rx=0 tx=0).stop 2023-03-03T14:05:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 -- 172.21.15.7:0/3449323261 shutdown_connections 2023-03-03T14:05:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c2c0682f0 0x7f3c2c06a7a0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c48155950 0x7f3c481eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c48071c30 0x7f3c480adac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 --2- 172.21.15.7:0/3449323261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c480aac00 0x7f3c4806e560 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 -- 172.21.15.7:0/3449323261 >> 172.21.15.7:0/3449323261 conn(0x7f3c480736f0 msgr2=0x7f3c48151c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:05:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 -- 172.21.15.7:0/3449323261 shutdown_connections 2023-03-03T14:05:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:29.552+0000 7f3c4f325700 1 -- 172.21.15.7:0/3449323261 wait complete. 2023-03-03T14:05:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:05:29.567 INFO:tasks.mds_thrash.fs.[b]:waiting for 14 secs before thrashing 2023-03-03T14:05:34.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:05:35.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.022+0000 7fa43afcb700 1 Processor -- start 2023-03-03T14:05:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.023+0000 7fa43afcb700 1 -- start start 2023-03-03T14:05:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.023+0000 7fa43afcb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414fda0 0x7fa434156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.023+0000 7fa43afcb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4340abe20 0x7fa4341529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.023+0000 7fa43afcb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4340c2f30 con 0x7fa43414f9d0 2023-03-03T14:05:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.023+0000 7fa43afcb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4340c31d0 con 0x7fa4340abe20 2023-03-03T14:05:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.023+0000 7fa43afcb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4341524a0 con 0x7fa43414fda0 2023-03-03T14:05:35.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa439568700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa4340abe20 0x7fa4341529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa438d67700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa43414f9d0 0x7fa434071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41210/0 (socket says 172.21.15.7:41210) 2023-03-03T14:05:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa438d67700 1 -- 172.21.15.7:0/3528547425 learned_addr learned my addr 172.21.15.7:0/3528547425 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:05:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa433fff700 1 --2- 172.21.15.7:0/3528547425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414fda0 0x7fa434156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa439568700 1 -- 172.21.15.7:0/3528547425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414fda0 msgr2=0x7fa434156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:05:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa439568700 1 --2- 172.21.15.7:0/3528547425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414fda0 0x7fa434156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:35.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa439568700 1 -- 172.21.15.7:0/3528547425 >> v1:172.21.15.7:6789/0 conn(0x7fa43414f9d0 legacy=0x7fa434071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa439568700 1 -- 172.21.15.7:0/3528547425 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4340c34b0 con 0x7fa4340abe20 2023-03-03T14:05:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa433fff700 1 --2- 172.21.15.7:0/3528547425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414fda0 0x7fa434156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:05:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa4337fe700 1 -- 172.21.15.7:0/3528547425 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763636060 0 0) 0x7fa4340c2f30 con 0x7fa43414f9d0 2023-03-03T14:05:35.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa439568700 1 --2- 172.21.15.7:0/3528547425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4340abe20 0x7fa4341529c0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009d90 tx=0x7fa42c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719ad1000fb526c6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa4337fe700 1 -- 172.21.15.7:0/3528547425 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c00e070 con 0x7fa4340abe20 2023-03-03T14:05:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa4337fe700 1 -- 172.21.15.7:0/3528547425 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42c00bbd0 con 0x7fa4340abe20 2023-03-03T14:05:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.024+0000 7fa4337fe700 1 -- 172.21.15.7:0/3528547425 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c012430 con 0x7fa4340abe20 2023-03-03T14:05:35.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.025+0000 7fa43afcb700 1 -- 172.21.15.7:0/3528547425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4340abe20 msgr2=0x7fa4341529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:35.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.025+0000 7fa43afcb700 1 --2- 172.21.15.7:0/3528547425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4340abe20 0x7fa4341529c0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009d90 tx=0x7fa42c00b330 comp rx=0 tx=0).stop 2023-03-03T14:05:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.025+0000 7fa43afcb700 1 -- 172.21.15.7:0/3528547425 shutdown_connections 2023-03-03T14:05:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.025+0000 7fa43afcb700 1 --2- 172.21.15.7:0/3528547425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa4340abe20 0x7fa4341529c0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:35.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.025+0000 7fa43afcb700 1 --2- 172.21.15.7:0/3528547425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414fda0 0x7fa434156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.025+0000 7fa43afcb700 1 -- 172.21.15.7:0/3528547425 >> 172.21.15.7:0/3528547425 conn(0x7fa4340c8900 msgr2=0x7fa4340c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:05:35.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.025+0000 7fa43afcb700 1 -- 172.21.15.7:0/3528547425 shutdown_connections 2023-03-03T14:05:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.025+0000 7fa43afcb700 1 -- 172.21.15.7:0/3528547425 wait complete. 2023-03-03T14:05:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.026+0000 7fa43afcb700 1 Processor -- start 2023-03-03T14:05:35.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.026+0000 7fa43afcb700 1 -- start start 2023-03-03T14:05:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa43afcb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4340abe20 0x7fa434154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa43afcb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414f9d0 0x7fa43414abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:35.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa43afcb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa43414fda0 0x7fa43414b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa43afcb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa434155c30 con 0x7fa43414fda0 2023-03-03T14:05:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa43afcb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4341e7fa0 con 0x7fa4340abe20 2023-03-03T14:05:35.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa43afcb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4341e8240 con 0x7fa43414f9d0 2023-03-03T14:05:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa439568700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa43414fda0 0x7fa43414b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:35.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa439568700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa43414fda0 0x7fa43414b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47240/0 (socket says 172.21.15.7:47240) 2023-03-03T14:05:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa439568700 1 -- 172.21.15.7:0/888946377 learned_addr learned my addr 172.21.15.7:0/888946377 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:05:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa433fff700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414f9d0 0x7fa43414abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa438d67700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4340abe20 0x7fa434154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:35.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa439568700 1 -- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414f9d0 msgr2=0x7fa43414abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:35.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa439568700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414f9d0 0x7fa43414abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:35.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa439568700 1 -- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4340abe20 msgr2=0x7fa434154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:35.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa439568700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4340abe20 0x7fa434154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:35.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa439568700 1 -- 172.21.15.7:0/888946377 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c009a70 con 0x7fa43414fda0 2023-03-03T14:05:35.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.027+0000 7fa438d67700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4340abe20 0x7fa434154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:05:35.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.028+0000 7fa439568700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa43414fda0 0x7fa43414b0f0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fa42c014040 tx=0x7fa42c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:35.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.028+0000 7fa4317fa700 1 -- 172.21.15.7:0/888946377 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c012430 con 0x7fa43414fda0 2023-03-03T14:05:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.028+0000 7fa4317fa700 1 -- 172.21.15.7:0/888946377 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42c020070 con 0x7fa43414fda0 2023-03-03T14:05:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.028+0000 7fa4317fa700 1 -- 172.21.15.7:0/888946377 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c004040 con 0x7fa43414fda0 2023-03-03T14:05:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.028+0000 7fa43afcb700 1 -- 172.21.15.7:0/888946377 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4340ac350 con 0x7fa43414fda0 2023-03-03T14:05:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.028+0000 7fa43afcb700 1 -- 172.21.15.7:0/888946377 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4340ab630 con 0x7fa43414fda0 2023-03-03T14:05:35.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.029+0000 7fa43afcb700 1 -- 172.21.15.7:0/888946377 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa434071fa0 con 0x7fa43414fda0 2023-03-03T14:05:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.029+0000 7fa4317fa700 1 -- 172.21.15.7:0/888946377 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa42c0041a0 con 0x7fa43414fda0 2023-03-03T14:05:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.030+0000 7fa4317fa700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4180682f0 0x7fa41806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.030+0000 7fa4317fa700 1 -- 172.21.15.7:0/888946377 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa42c022070 con 0x7fa43414fda0 2023-03-03T14:05:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.030+0000 7fa438d67700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4180682f0 0x7fa41806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.031+0000 7fa438d67700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4180682f0 0x7fa41806a7a0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa428000f60 tx=0x7fa428009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.034+0000 7fa4317fa700 1 -- 172.21.15.7:0/888946377 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa42c054820 con 0x7fa43414fda0 2023-03-03T14:05:35.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.272+0000 7fa43afcb700 1 -- 172.21.15.7:0/888946377 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa43406c880 con 0x7fa43414fda0 2023-03-03T14:05:35.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.272+0000 7fa4317fa700 1 -- 172.21.15.7:0/888946377 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa43406c880 con 0x7fa43414fda0 2023-03-03T14:05:35.275 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:05:35.275 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:05:35.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 -- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4180682f0 msgr2=0x7fa41806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:35.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4180682f0 0x7fa41806a7a0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa428000f60 tx=0x7fa428009040 comp rx=0 tx=0).stop 2023-03-03T14:05:35.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 -- 172.21.15.7:0/888946377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa43414fda0 msgr2=0x7fa43414b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:35.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa43414fda0 0x7fa43414b0f0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fa42c014040 tx=0x7fa42c004ad0 comp rx=0 tx=0).stop 2023-03-03T14:05:35.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 -- 172.21.15.7:0/888946377 shutdown_connections 2023-03-03T14:05:35.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa4180682f0 0x7fa41806a7a0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:35.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa43414fda0 0x7fa43414b0f0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:35.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4340abe20 0x7fa434154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:35.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 --2- 172.21.15.7:0/888946377 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa43414f9d0 0x7fa43414abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:35.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 -- 172.21.15.7:0/888946377 >> 172.21.15.7:0/888946377 conn(0x7fa4340c8900 msgr2=0x7fa4340c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:05:35.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.274+0000 7fa43afcb700 1 -- 172.21.15.7:0/888946377 shutdown_connections 2023-03-03T14:05:35.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:35.275+0000 7fa43afcb700 1 -- 172.21.15.7:0/888946377 wait complete. 2023-03-03T14:05:35.279 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:05:35.289 INFO:tasks.mds_thrash.fs.[a]:waiting for 59 secs before thrashing 2023-03-03T14:05:43.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:05:43.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.706+0000 7f264e243700 1 Processor -- start 2023-03-03T14:05:43.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.706+0000 7f264e243700 1 -- start start 2023-03-03T14:05:43.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.706+0000 7f264e243700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f26480c2e30 0x7f2648158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:43.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.706+0000 7f264e243700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f264806c6c0 0x7f26480736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:43.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.706+0000 7f264e243700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2648157ff0 con 0x7f2648151cd0 2023-03-03T14:05:43.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f264e243700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264806d640 con 0x7f26480c2e30 2023-03-03T14:05:43.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f264e243700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264806cda0 con 0x7f264806c6c0 2023-03-03T14:05:43.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f2646ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f26480c2e30 0x7f2648158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f26477fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2648151cd0 0x7f2648154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41230/0 (socket says 172.21.15.7:41230) 2023-03-03T14:05:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f26477fe700 1 -- 172.21.15.7:0/1217566466 learned_addr learned my addr 172.21.15.7:0/1217566466 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:05:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f2647fff700 1 --2- 172.21.15.7:0/1217566466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f264806c6c0 0x7f26480736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f2646ffd700 1 -- 172.21.15.7:0/1217566466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f264806c6c0 msgr2=0x7f26480736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:05:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f2646ffd700 1 --2- 172.21.15.7:0/1217566466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f264806c6c0 0x7f26480736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f2646ffd700 1 -- 172.21.15.7:0/1217566466 >> v1:172.21.15.7:6789/0 conn(0x7f2648151cd0 legacy=0x7f2648154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f2646ffd700 1 -- 172.21.15.7:0/1217566466 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264806d080 con 0x7f26480c2e30 2023-03-03T14:05:43.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f26467fc700 1 -- 172.21.15.7:0/1217566466 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817907090 0 0) 0x7f2648157ff0 con 0x7f2648151cd0 2023-03-03T14:05:43.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f2646ffd700 1 --2- 172.21.15.7:0/1217566466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26480c2e30 0x7f2648158480 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f263c009840 tx=0x7f263c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9b3131d90cb53 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:43.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f26467fc700 1 -- 172.21.15.7:0/1217566466 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c011830 con 0x7f26480c2e30 2023-03-03T14:05:43.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f26467fc700 1 -- 172.21.15.7:0/1217566466 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c011990 con 0x7f26480c2e30 2023-03-03T14:05:43.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.707+0000 7f26467fc700 1 -- 172.21.15.7:0/1217566466 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c011d20 con 0x7f26480c2e30 2023-03-03T14:05:43.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.708+0000 7f264e243700 1 -- 172.21.15.7:0/1217566466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26480c2e30 msgr2=0x7f2648158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:43.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.708+0000 7f264e243700 1 --2- 172.21.15.7:0/1217566466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26480c2e30 0x7f2648158480 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f263c009840 tx=0x7f263c009b50 comp rx=0 tx=0).stop 2023-03-03T14:05:43.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.708+0000 7f264e243700 1 -- 172.21.15.7:0/1217566466 shutdown_connections 2023-03-03T14:05:43.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.708+0000 7f264e243700 1 --2- 172.21.15.7:0/1217566466 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f264806c6c0 0x7f26480736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.708+0000 7f264e243700 1 --2- 172.21.15.7:0/1217566466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26480c2e30 0x7f2648158480 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.708+0000 7f264e243700 1 -- 172.21.15.7:0/1217566466 >> 172.21.15.7:0/1217566466 conn(0x7f26480a94d0 msgr2=0x7f26480a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.708+0000 7f264e243700 1 -- 172.21.15.7:0/1217566466 shutdown_connections 2023-03-03T14:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.708+0000 7f264e243700 1 -- 172.21.15.7:0/1217566466 wait complete. 2023-03-03T14:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.709+0000 7f264e243700 1 Processor -- start 2023-03-03T14:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.709+0000 7f264e243700 1 -- start start 2023-03-03T14:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f264e243700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264806c6c0 0x7f264814e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f264e243700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26480c2e30 0x7f26480c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:43.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f264e243700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2648151cd0 0x7f2648155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f264e243700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26480aa720 con 0x7f264806c6c0 2023-03-03T14:05:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f264e243700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264814ed60 con 0x7f2648151cd0 2023-03-03T14:05:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f264e243700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264814f0a0 con 0x7f26480c2e30 2023-03-03T14:05:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f26477fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264806c6c0 0x7f264814e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f26477fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264806c6c0 0x7f264814e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47258/0 (socket says 172.21.15.7:47258) 2023-03-03T14:05:43.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f26477fe700 1 -- 172.21.15.7:0/172212403 learned_addr learned my addr 172.21.15.7:0/172212403 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:05:43.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f2646ffd700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26480c2e30 0x7f26480c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:43.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.710+0000 7f2647fff700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2648151cd0 0x7f2648155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:43.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f26477fe700 1 -- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26480c2e30 msgr2=0x7f26480c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:43.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f26477fe700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26480c2e30 0x7f26480c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:43.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f26477fe700 1 -- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2648151cd0 msgr2=0x7f2648155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:43.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f26477fe700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2648151cd0 0x7f2648155900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:43.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f26477fe700 1 -- 172.21.15.7:0/172212403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c0094a0 con 0x7f264806c6c0 2023-03-03T14:05:43.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f26477fe700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264806c6c0 0x7f264814e820 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f263800c580 tx=0x7f263800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:43.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f262ffff700 1 -- 172.21.15.7:0/172212403 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638004c80 con 0x7f264806c6c0 2023-03-03T14:05:43.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f262ffff700 1 -- 172.21.15.7:0/172212403 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2638004e00 con 0x7f264806c6c0 2023-03-03T14:05:43.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f262ffff700 1 -- 172.21.15.7:0/172212403 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638007310 con 0x7f264806c6c0 2023-03-03T14:05:43.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f264e243700 1 -- 172.21.15.7:0/172212403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2648157c80 con 0x7f264806c6c0 2023-03-03T14:05:43.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.711+0000 7f264e243700 1 -- 172.21.15.7:0/172212403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264806d360 con 0x7f264806c6c0 2023-03-03T14:05:43.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.712+0000 7f264e243700 1 -- 172.21.15.7:0/172212403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26481589c0 con 0x7f264806c6c0 2023-03-03T14:05:43.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.712+0000 7f262ffff700 1 -- 172.21.15.7:0/172212403 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f263801a020 con 0x7f264806c6c0 2023-03-03T14:05:43.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.713+0000 7f262ffff700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26340682f0 0x7f263406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:05:43.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.713+0000 7f262ffff700 1 -- 172.21.15.7:0/172212403 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2638085b50 con 0x7f264806c6c0 2023-03-03T14:05:43.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.713+0000 7f2646ffd700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26340682f0 0x7f263406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:05:43.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.714+0000 7f2646ffd700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26340682f0 0x7f263406a7a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f263c0090b0 tx=0x7f263c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:05:43.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.717+0000 7f262ffff700 1 -- 172.21.15.7:0/172212403 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2638051ef0 con 0x7f264806c6c0 2023-03-03T14:05:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.955+0000 7f264e243700 1 -- 172.21.15.7:0/172212403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2648074bb0 con 0x7f264806c6c0 2023-03-03T14:05:43.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.955+0000 7f262ffff700 1 -- 172.21.15.7:0/172212403 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2638055510 con 0x7f264806c6c0 2023-03-03T14:05:43.958 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:05:43.958 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:05:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.957+0000 7f264e243700 1 -- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26340682f0 msgr2=0x7f263406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.957+0000 7f264e243700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26340682f0 0x7f263406a7a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f263c0090b0 tx=0x7f263c01c040 comp rx=0 tx=0).stop 2023-03-03T14:05:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.957+0000 7f264e243700 1 -- 172.21.15.7:0/172212403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264806c6c0 msgr2=0x7f264814e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:05:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.957+0000 7f264e243700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264806c6c0 0x7f264814e820 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f263800c580 tx=0x7f263800c890 comp rx=0 tx=0).stop 2023-03-03T14:05:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.957+0000 7f264e243700 1 -- 172.21.15.7:0/172212403 shutdown_connections 2023-03-03T14:05:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.957+0000 7f264e243700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2648151cd0 0x7f2648155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.957+0000 7f264e243700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26340682f0 0x7f263406a7a0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.957+0000 7f264e243700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264806c6c0 0x7f264814e820 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.957+0000 7f264e243700 1 --2- 172.21.15.7:0/172212403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26480c2e30 0x7f26480c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:05:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.957+0000 7f264e243700 1 -- 172.21.15.7:0/172212403 >> 172.21.15.7:0/172212403 conn(0x7f26480a94d0 msgr2=0x7f2648070f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:05:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.958+0000 7f264e243700 1 -- 172.21.15.7:0/172212403 shutdown_connections 2023-03-03T14:05:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:05:43.958+0000 7f264e243700 1 -- 172.21.15.7:0/172212403 wait complete. 2023-03-03T14:05:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:05:43.971 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-03-03T14:05:49.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:05:49.790 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:19.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:19.857 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:34.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:06:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.427+0000 7f677f521700 1 Processor -- start 2023-03-03T14:06:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.428+0000 7f677f521700 1 -- start start 2023-03-03T14:06:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.428+0000 7f677f521700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f67780c1da0 0x7f67781550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.428+0000 7f677f521700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67780c7a60 con 0x7f677806d870 2023-03-03T14:06:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.428+0000 7f677f521700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67780c3620 con 0x7f67780c3e30 2023-03-03T14:06:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.428+0000 7f677f521700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67780c2af0 con 0x7f67780c1da0 2023-03-03T14:06:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.428+0000 7f677cabc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f67780c1da0 0x7f67781550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.429+0000 7f677d2bd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f677806d870 0x7f67780c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41274/0 (socket says 172.21.15.7:41274) 2023-03-03T14:06:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.429+0000 7f677d2bd700 1 -- 172.21.15.7:0/3294603871 learned_addr learned my addr 172.21.15.7:0/3294603871 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:06:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.429+0000 7f677cabc700 1 -- 172.21.15.7:0/3294603871 >> v1:172.21.15.143:6789/0 conn(0x7f67780c3e30 legacy=0x7f677815ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:06:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.429+0000 7f677cabc700 1 -- 172.21.15.7:0/3294603871 >> v1:172.21.15.7:6789/0 conn(0x7f677806d870 legacy=0x7f67780c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.429+0000 7f676bfff700 1 -- 172.21.15.7:0/3294603871 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378072109 0 0) 0x7f67780c7a60 con 0x7f677806d870 2023-03-03T14:06:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.429+0000 7f677cabc700 1 -- 172.21.15.7:0/3294603871 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67781545e0 con 0x7f67780c1da0 2023-03-03T14:06:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.429+0000 7f677cabc700 1 --2- 172.21.15.7:0/3294603871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 0x7f67781550c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f6774009840 tx=0x7f6774009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378355b540d36335 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:06:34.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.429+0000 7f676bfff700 1 -- 172.21.15.7:0/3294603871 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774011830 con 0x7f67780c1da0 2023-03-03T14:06:34.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.429+0000 7f676bfff700 1 -- 172.21.15.7:0/3294603871 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6774011990 con 0x7f67780c1da0 2023-03-03T14:06:34.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.429+0000 7f676bfff700 1 -- 172.21.15.7:0/3294603871 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774011d20 con 0x7f67780c1da0 2023-03-03T14:06:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.430+0000 7f677f521700 1 -- 172.21.15.7:0/3294603871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 msgr2=0x7f67781550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.430+0000 7f677f521700 1 --2- 172.21.15.7:0/3294603871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 0x7f67781550c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f6774009840 tx=0x7f6774009b50 comp rx=0 tx=0).stop 2023-03-03T14:06:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.430+0000 7f677f521700 1 -- 172.21.15.7:0/3294603871 shutdown_connections 2023-03-03T14:06:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.430+0000 7f677f521700 1 --2- 172.21.15.7:0/3294603871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 0x7f67781550c0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.430+0000 7f677f521700 1 -- 172.21.15.7:0/3294603871 >> 172.21.15.7:0/3294603871 conn(0x7f677806c110 msgr2=0x7f677806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:06:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.430+0000 7f677f521700 1 -- 172.21.15.7:0/3294603871 shutdown_connections 2023-03-03T14:06:34.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.430+0000 7f677f521700 1 -- 172.21.15.7:0/3294603871 wait complete. 2023-03-03T14:06:34.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.431+0000 7f677f521700 1 Processor -- start 2023-03-03T14:06:34.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.431+0000 7f677f521700 1 -- start start 2023-03-03T14:06:34.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677f521700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677806d870 0x7f67780c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:34.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677f521700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 0x7f677813a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:34.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677f521700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67780c3e30 0x7f677813a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:34.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677f521700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677814ee00 con 0x7f67780c1da0 2023-03-03T14:06:34.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677f521700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6778156cb0 con 0x7f67780c3e30 2023-03-03T14:06:34.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677f521700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677814fc00 con 0x7f677806d870 2023-03-03T14:06:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677cabc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 0x7f677813a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677cabc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 0x7f677813a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47302/0 (socket says 172.21.15.7:47302) 2023-03-03T14:06:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677cabc700 1 -- 172.21.15.7:0/1875792992 learned_addr learned my addr 172.21.15.7:0/1875792992 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:06:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677d2bd700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677806d870 0x7f67780c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677dabe700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67780c3e30 0x7f677813a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677cabc700 1 -- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677806d870 msgr2=0x7f67780c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677cabc700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677806d870 0x7f67780c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677cabc700 1 -- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67780c3e30 msgr2=0x7f677813a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:34.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677cabc700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67780c3e30 0x7f677813a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677cabc700 1 -- 172.21.15.7:0/1875792992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67740094a0 con 0x7f67780c1da0 2023-03-03T14:06:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677dabe700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67780c3e30 0x7f677813a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:06:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f677cabc700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 0x7f677813a080 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f67780c4a90 tx=0x7f6774004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:06:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.432+0000 7f6769ffb700 1 -- 172.21.15.7:0/1875792992 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677401a8c0 con 0x7f67780c1da0 2023-03-03T14:06:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.433+0000 7f6769ffb700 1 -- 172.21.15.7:0/1875792992 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677401aa20 con 0x7f67780c1da0 2023-03-03T14:06:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.433+0000 7f677f521700 1 -- 172.21.15.7:0/1875792992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6778154ba0 con 0x7f67780c1da0 2023-03-03T14:06:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.433+0000 7f6769ffb700 1 -- 172.21.15.7:0/1875792992 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677401ad90 con 0x7f67780c1da0 2023-03-03T14:06:34.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.433+0000 7f677f521700 1 -- 172.21.15.7:0/1875792992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67781548c0 con 0x7f67780c1da0 2023-03-03T14:06:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.434+0000 7f677f521700 1 -- 172.21.15.7:0/1875792992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677806dc40 con 0x7f67780c1da0 2023-03-03T14:06:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.434+0000 7f6769ffb700 1 -- 172.21.15.7:0/1875792992 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f677402b030 con 0x7f67780c1da0 2023-03-03T14:06:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.435+0000 7f6769ffb700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f675c068320 0x7f675c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:34.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.435+0000 7f6769ffb700 1 -- 172.21.15.7:0/1875792992 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f67740874c0 con 0x7f67780c1da0 2023-03-03T14:06:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.435+0000 7f677d2bd700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f675c068320 0x7f675c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.435+0000 7f677d2bd700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f675c068320 0x7f675c06a7d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f676c000f60 tx=0x7f676c007d90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:06:34.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.439+0000 7f6769ffb700 1 -- 172.21.15.7:0/1875792992 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6774053910 con 0x7f67780c1da0 2023-03-03T14:06:34.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.679+0000 7f677f521700 1 -- 172.21.15.7:0/1875792992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6778074210 con 0x7f67780c1da0 2023-03-03T14:06:34.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.680+0000 7f6769ffb700 1 -- 172.21.15.7:0/1875792992 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6774056f30 con 0x7f67780c1da0 2023-03-03T14:06:34.683 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:06:34.683 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:06:34.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 -- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f675c068320 msgr2=0x7f675c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:34.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f675c068320 0x7f675c06a7d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f676c000f60 tx=0x7f676c007d90 comp rx=0 tx=0).stop 2023-03-03T14:06:34.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 -- 172.21.15.7:0/1875792992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 msgr2=0x7f677813a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:34.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 0x7f677813a080 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f67780c4a90 tx=0x7f6774004750 comp rx=0 tx=0).stop 2023-03-03T14:06:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 -- 172.21.15.7:0/1875792992 shutdown_connections 2023-03-03T14:06:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67780c3e30 0x7f677813a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f677806d870 0x7f67780c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f675c068320 0x7f675c06a7d0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 --2- 172.21.15.7:0/1875792992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67780c1da0 0x7f677813a080 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 -- 172.21.15.7:0/1875792992 >> 172.21.15.7:0/1875792992 conn(0x7f677806c110 msgr2=0x7f677806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:06:34.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 -- 172.21.15.7:0/1875792992 shutdown_connections 2023-03-03T14:06:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:34.682+0000 7f677f521700 1 -- 172.21.15.7:0/1875792992 wait complete. 2023-03-03T14:06:34.688 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:06:34.697 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before thrashing 2023-03-03T14:06:43.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:06:43.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.836+0000 7ff3a61e3700 1 Processor -- start 2023-03-03T14:06:43.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.836+0000 7ff3a61e3700 1 -- start start 2023-03-03T14:06:43.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.837+0000 7ff3a61e3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3a013a0f0 0x7ff3a00c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:43.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.837+0000 7ff3a61e3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff3a00c1da0 0x7ff3a006d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:43.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.837+0000 7ff3a61e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a006de90 con 0x7ff3a01586c0 2023-03-03T14:06:43.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.837+0000 7ff3a61e3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a0154840 con 0x7ff3a013a0f0 2023-03-03T14:06:43.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.837+0000 7ff3a61e3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a0154560 con 0x7ff3a00c1da0 2023-03-03T14:06:43.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.837+0000 7ff39f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3a013a0f0 0x7ff3a00c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:43.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.837+0000 7ff39f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3a013a0f0 0x7ff3a00c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47314/0 (socket says 172.21.15.7:47314) 2023-03-03T14:06:43.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.837+0000 7ff39f7fe700 1 -- 172.21.15.7:0/3402168150 learned_addr learned my addr 172.21.15.7:0/3402168150 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:06:43.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.838+0000 7ff39ffff700 1 --2- 172.21.15.7:0/3402168150 >> v2:172.21.15.143:3300/0 conn(0x7ff3a00c1da0 0x7ff3a006d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:43.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.838+0000 7ff39f7fe700 1 -- 172.21.15.7:0/3402168150 >> v2:172.21.15.143:3300/0 conn(0x7ff3a00c1da0 msgr2=0x7ff3a006d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:06:43.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.838+0000 7ff39f7fe700 1 --2- 172.21.15.7:0/3402168150 >> v2:172.21.15.143:3300/0 conn(0x7ff3a00c1da0 0x7ff3a006d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:43.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.838+0000 7ff39f7fe700 1 -- 172.21.15.7:0/3402168150 >> v1:172.21.15.143:6789/0 conn(0x7ff3a01586c0 legacy=0x7ff3a006d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:43.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.838+0000 7ff39f7fe700 1 -- 172.21.15.7:0/3402168150 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a0157be0 con 0x7ff3a013a0f0 2023-03-03T14:06:43.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.838+0000 7ff39f7fe700 1 --2- 172.21.15.7:0/3402168150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a013a0f0 0x7ff3a00c07d0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0070000 tx=0x7ff390009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb41eb517c7c1b0a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:06:43.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.838+0000 7ff39e7fc700 1 -- 172.21.15.7:0/3402168150 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39000b870 con 0x7ff3a013a0f0 2023-03-03T14:06:43.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.838+0000 7ff39e7fc700 1 -- 172.21.15.7:0/3402168150 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39000b9d0 con 0x7ff3a013a0f0 2023-03-03T14:06:43.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.838+0000 7ff39e7fc700 1 -- 172.21.15.7:0/3402168150 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39000bd60 con 0x7ff3a013a0f0 2023-03-03T14:06:43.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.839+0000 7ff3a61e3700 1 -- 172.21.15.7:0/3402168150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a013a0f0 msgr2=0x7ff3a00c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:43.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.839+0000 7ff3a61e3700 1 --2- 172.21.15.7:0/3402168150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a013a0f0 0x7ff3a00c07d0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0070000 tx=0x7ff390009bd0 comp rx=0 tx=0).stop 2023-03-03T14:06:43.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.839+0000 7ff3a61e3700 1 -- 172.21.15.7:0/3402168150 shutdown_connections 2023-03-03T14:06:43.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.839+0000 7ff3a61e3700 1 --2- 172.21.15.7:0/3402168150 >> v2:172.21.15.143:3300/0 conn(0x7ff3a00c1da0 0x7ff3a006d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:43.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.839+0000 7ff3a61e3700 1 --2- 172.21.15.7:0/3402168150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a013a0f0 0x7ff3a00c07d0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:43.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.839+0000 7ff3a61e3700 1 -- 172.21.15.7:0/3402168150 >> 172.21.15.7:0/3402168150 conn(0x7ff3a006c110 msgr2=0x7ff3a006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:06:43.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.839+0000 7ff3a61e3700 1 -- 172.21.15.7:0/3402168150 shutdown_connections 2023-03-03T14:06:43.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.839+0000 7ff3a61e3700 1 -- 172.21.15.7:0/3402168150 wait complete. 2023-03-03T14:06:43.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.840+0000 7ff3a61e3700 1 Processor -- start 2023-03-03T14:06:43.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.840+0000 7ff3a61e3700 1 -- start start 2023-03-03T14:06:43.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff3a61e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a00c1da0 0x7ff3a00c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:43.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff3a61e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a013a0f0 0x7ff3a00c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:43.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff3a61e3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a01586c0 0x7ff3a00702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:43.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff3a61e3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a00c5200 con 0x7ff3a01586c0 2023-03-03T14:06:43.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff3a61e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a01eadb0 con 0x7ff3a013a0f0 2023-03-03T14:06:43.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff3a61e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a00c77d0 con 0x7ff3a00c1da0 2023-03-03T14:06:43.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a01586c0 0x7ff3a00702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:43.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a01586c0 0x7ff3a00702b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47324/0 (socket says 172.21.15.7:47324) 2023-03-03T14:06:43.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39ffff700 1 -- 172.21.15.7:0/1735919057 learned_addr learned my addr 172.21.15.7:0/1735919057 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:06:43.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39f7fe700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a00c1da0 0x7ff3a00c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:43.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39effd700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a013a0f0 0x7ff3a00c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:43.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39ffff700 1 -- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a00c1da0 msgr2=0x7ff3a00c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:43.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39ffff700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a00c1da0 0x7ff3a00c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:43.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39ffff700 1 -- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a013a0f0 msgr2=0x7ff3a00c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:43.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39ffff700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a013a0f0 0x7ff3a00c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:43.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39ffff700 1 -- 172.21.15.7:0/1735919057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff390009550 con 0x7ff3a01586c0 2023-03-03T14:06:43.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.841+0000 7ff39ffff700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a01586c0 0x7ff3a00702b0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7ff398007e60 tx=0x7ff39800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:06:43.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.842+0000 7ff397fff700 1 -- 172.21.15.7:0/1735919057 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39800ede0 con 0x7ff3a01586c0 2023-03-03T14:06:43.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.842+0000 7ff397fff700 1 -- 172.21.15.7:0/1735919057 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff398014420 con 0x7ff3a01586c0 2023-03-03T14:06:43.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.842+0000 7ff3a61e3700 1 -- 172.21.15.7:0/1735919057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a01581a0 con 0x7ff3a01586c0 2023-03-03T14:06:43.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.842+0000 7ff397fff700 1 -- 172.21.15.7:0/1735919057 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3980147d0 con 0x7ff3a01586c0 2023-03-03T14:06:43.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.842+0000 7ff3a61e3700 1 -- 172.21.15.7:0/1735919057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a0157ec0 con 0x7ff3a01586c0 2023-03-03T14:06:43.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.843+0000 7ff3a61e3700 1 -- 172.21.15.7:0/1735919057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a006e1a0 con 0x7ff3a01586c0 2023-03-03T14:06:43.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.843+0000 7ff397fff700 1 -- 172.21.15.7:0/1735919057 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff398012050 con 0x7ff3a01586c0 2023-03-03T14:06:43.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.844+0000 7ff397fff700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3800682f0 0x7ff38006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:06:43.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.844+0000 7ff397fff700 1 -- 172.21.15.7:0/1735919057 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff398086860 con 0x7ff3a01586c0 2023-03-03T14:06:43.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.844+0000 7ff39f7fe700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3800682f0 0x7ff38006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:06:43.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.844+0000 7ff39f7fe700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3800682f0 0x7ff38006a7a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7ff390000c00 tx=0x7ff390008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:06:43.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:43.848+0000 7ff397fff700 1 -- 172.21.15.7:0/1735919057 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff398052b10 con 0x7ff3a01586c0 2023-03-03T14:06:44.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.084+0000 7ff3a61e3700 1 -- 172.21.15.7:0/1735919057 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3a0152130 con 0x7ff3a01586c0 2023-03-03T14:06:44.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.085+0000 7ff397fff700 1 -- 172.21.15.7:0/1735919057 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff398056130 con 0x7ff3a01586c0 2023-03-03T14:06:44.087 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:06:44.087 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:06:44.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 -- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3800682f0 msgr2=0x7ff38006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:44.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3800682f0 0x7ff38006a7a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7ff390000c00 tx=0x7ff390008030 comp rx=0 tx=0).stop 2023-03-03T14:06:44.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 -- 172.21.15.7:0/1735919057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a01586c0 msgr2=0x7ff3a00702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:06:44.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a01586c0 0x7ff3a00702b0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7ff398007e60 tx=0x7ff39800e3c0 comp rx=0 tx=0).stop 2023-03-03T14:06:44.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 -- 172.21.15.7:0/1735919057 shutdown_connections 2023-03-03T14:06:44.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3800682f0 0x7ff38006a7a0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:44.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3a01586c0 0x7ff3a00702b0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:44.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3a00c1da0 0x7ff3a00c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:44.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 --2- 172.21.15.7:0/1735919057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3a013a0f0 0x7ff3a00c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:06:44.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 -- 172.21.15.7:0/1735919057 >> 172.21.15.7:0/1735919057 conn(0x7ff3a006c110 msgr2=0x7ff3a006d220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:06:44.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 -- 172.21.15.7:0/1735919057 shutdown_connections 2023-03-03T14:06:44.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:06:44.087+0000 7ff3a61e3700 1 -- 172.21.15.7:0/1735919057 wait complete. 2023-03-03T14:06:44.091 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:06:44.101 INFO:tasks.mds_thrash.fs.[a]:waiting for 115 secs before thrashing 2023-03-03T14:06:49.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:06:49.920 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:05.972 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:07:06.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.111+0000 7fa7e515d700 1 Processor -- start 2023-03-03T14:07:06.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.111+0000 7fa7e515d700 1 -- start start 2023-03-03T14:07:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.111+0000 7fa7e515d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0071990 0x7fa7e0071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7e515d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa7e014b460 0x7fa7e00722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7e515d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7e0151c00 0x7fa7e006aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7e515d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e01521a0 con 0x7fa7e0151c00 2023-03-03T14:07:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7e515d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e0070360 con 0x7fa7e014b460 2023-03-03T14:07:06.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7e515d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e006a5e0 con 0x7fa7e0071990 2023-03-03T14:07:06.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7df59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7e0151c00 0x7fa7e006aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:06.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7df59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7e0151c00 0x7fa7e006aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47352/0 (socket says 172.21.15.7:47352) 2023-03-03T14:07:06.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7df59e700 1 -- 172.21.15.7:0/3270552152 learned_addr learned my addr 172.21.15.7:0/3270552152 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:07:06.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7de59c700 1 --2- 172.21.15.7:0/3270552152 >> v2:172.21.15.143:3300/0 conn(0x7fa7e014b460 0x7fa7e00722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:06.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7ded9d700 1 --2- 172.21.15.7:0/3270552152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0071990 0x7fa7e0071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:06.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7df59e700 1 -- 172.21.15.7:0/3270552152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0071990 msgr2=0x7fa7e0071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:07:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7df59e700 1 --2- 172.21.15.7:0/3270552152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0071990 0x7fa7e0071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7df59e700 1 -- 172.21.15.7:0/3270552152 >> v2:172.21.15.143:3300/0 conn(0x7fa7e014b460 msgr2=0x7fa7e00722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:07:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7df59e700 1 --2- 172.21.15.7:0/3270552152 >> v2:172.21.15.143:3300/0 conn(0x7fa7e014b460 0x7fa7e00722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7df59e700 1 -- 172.21.15.7:0/3270552152 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00c38b0 con 0x7fa7e0151c00 2023-03-03T14:07:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7ded9d700 1 --2- 172.21.15.7:0/3270552152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0071990 0x7fa7e0071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:07:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7df59e700 1 --2- 172.21.15.7:0/3270552152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0151c00 0x7fa7e006aa70 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fa7e006bd20 tx=0x7fa7d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b68de2095fb764f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:07:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7ddd9b700 1 -- 172.21.15.7:0/3270552152 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d800e070 con 0x7fa7e0151c00 2023-03-03T14:07:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7ddd9b700 1 -- 172.21.15.7:0/3270552152 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d800b9c0 con 0x7fa7e0151c00 2023-03-03T14:07:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.112+0000 7fa7ddd9b700 1 -- 172.21.15.7:0/3270552152 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d800bd70 con 0x7fa7e0151c00 2023-03-03T14:07:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.113+0000 7fa7e515d700 1 -- 172.21.15.7:0/3270552152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0151c00 msgr2=0x7fa7e006aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.113+0000 7fa7e515d700 1 --2- 172.21.15.7:0/3270552152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0151c00 0x7fa7e006aa70 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fa7e006bd20 tx=0x7fa7d800b330 comp rx=0 tx=0).stop 2023-03-03T14:07:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.113+0000 7fa7e515d700 1 -- 172.21.15.7:0/3270552152 shutdown_connections 2023-03-03T14:07:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.113+0000 7fa7e515d700 1 --2- 172.21.15.7:0/3270552152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0151c00 0x7fa7e006aa70 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.113+0000 7fa7e515d700 1 --2- 172.21.15.7:0/3270552152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0071990 0x7fa7e0071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.113+0000 7fa7e515d700 1 --2- 172.21.15.7:0/3270552152 >> v2:172.21.15.143:3300/0 conn(0x7fa7e014b460 0x7fa7e00722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.113+0000 7fa7e515d700 1 -- 172.21.15.7:0/3270552152 >> 172.21.15.7:0/3270552152 conn(0x7fa7e00a94d0 msgr2=0x7fa7e00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:07:06.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.113+0000 7fa7e515d700 1 -- 172.21.15.7:0/3270552152 shutdown_connections 2023-03-03T14:07:06.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.113+0000 7fa7e515d700 1 -- 172.21.15.7:0/3270552152 wait complete. 2023-03-03T14:07:06.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.114+0000 7fa7e515d700 1 Processor -- start 2023-03-03T14:07:06.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.115+0000 7fa7e515d700 1 -- start start 2023-03-03T14:07:06.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.115+0000 7fa7e515d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0071990 0x7fa7e00bf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.115+0000 7fa7e515d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e0151c00 0x7fa7e01e74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.115+0000 7fa7e515d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0154050 0x7fa7e006c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.115+0000 7fa7e515d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00c06c0 con 0x7fa7e0071990 2023-03-03T14:07:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.115+0000 7fa7e515d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00bfe00 con 0x7fa7e0151c00 2023-03-03T14:07:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.115+0000 7fa7e515d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e00c00e0 con 0x7fa7e0154050 2023-03-03T14:07:06.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.115+0000 7fa7ded9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0071990 0x7fa7e00bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.115+0000 7fa7ded9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0071990 0x7fa7e00bf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47354/0 (socket says 172.21.15.7:47354) 2023-03-03T14:07:06.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.115+0000 7fa7ded9d700 1 -- 172.21.15.7:0/1074505094 learned_addr learned my addr 172.21.15.7:0/1074505094 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:07:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7de59c700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e0151c00 0x7fa7e01e74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7df59e700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0154050 0x7fa7e006c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7ded9d700 1 -- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0154050 msgr2=0x7fa7e006c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7ded9d700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0154050 0x7fa7e006c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7ded9d700 1 -- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e0151c00 msgr2=0x7fa7e01e74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7ded9d700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e0151c00 0x7fa7e01e74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7ded9d700 1 -- 172.21.15.7:0/1074505094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d8009b30 con 0x7fa7e0071990 2023-03-03T14:07:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7ded9d700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0071990 0x7fa7e00bf770 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400ed40 tx=0x7fa7d4007380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:07:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7c77fe700 1 -- 172.21.15.7:0/1074505094 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4007de0 con 0x7fa7e0071990 2023-03-03T14:07:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7c77fe700 1 -- 172.21.15.7:0/1074505094 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d400ee20 con 0x7fa7e0071990 2023-03-03T14:07:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7c77fe700 1 -- 172.21.15.7:0/1074505094 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4010640 con 0x7fa7e0071990 2023-03-03T14:07:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7e515d700 1 -- 172.21.15.7:0/1074505094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e00c3b50 con 0x7fa7e0071990 2023-03-03T14:07:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.116+0000 7fa7e515d700 1 -- 172.21.15.7:0/1074505094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e006a2b0 con 0x7fa7e0071990 2023-03-03T14:07:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.117+0000 7fa7e515d700 1 -- 172.21.15.7:0/1074505094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e006afb0 con 0x7fa7e0071990 2023-03-03T14:07:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.118+0000 7fa7c77fe700 1 -- 172.21.15.7:0/1074505094 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa7d401e020 con 0x7fa7e0071990 2023-03-03T14:07:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.118+0000 7fa7c77fe700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7cc0682f0 0x7fa7cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:07:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.118+0000 7fa7c77fe700 1 -- 172.21.15.7:0/1074505094 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa7d4014070 con 0x7fa7e0071990 2023-03-03T14:07:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.118+0000 7fa7de59c700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7cc0682f0 0x7fa7cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:07:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.119+0000 7fa7de59c700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7cc0682f0 0x7fa7cc06a7a0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fa7e006cf10 tx=0x7fa7d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:07:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.122+0000 7fa7c77fe700 1 -- 172.21.15.7:0/1074505094 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7d4054cb0 con 0x7fa7e0071990 2023-03-03T14:07:06.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.362+0000 7fa7e515d700 1 -- 172.21.15.7:0/1074505094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7e0074bb0 con 0x7fa7e0071990 2023-03-03T14:07:06.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.362+0000 7fa7c77fe700 1 -- 172.21.15.7:0/1074505094 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa7d40582d0 con 0x7fa7e0071990 2023-03-03T14:07:06.364 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:07:06.364 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:07:06.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 -- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7cc0682f0 msgr2=0x7fa7cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7cc0682f0 0x7fa7cc06a7a0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fa7e006cf10 tx=0x7fa7d0008040 comp rx=0 tx=0).stop 2023-03-03T14:07:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 -- 172.21.15.7:0/1074505094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0071990 msgr2=0x7fa7e00bf770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:07:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0071990 0x7fa7e00bf770 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400ed40 tx=0x7fa7d4007380 comp rx=0 tx=0).stop 2023-03-03T14:07:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 -- 172.21.15.7:0/1074505094 shutdown_connections 2023-03-03T14:07:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7e0154050 0x7fa7e006c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7cc0682f0 0x7fa7cc06a7a0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7e0071990 0x7fa7e00bf770 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 --2- 172.21.15.7:0/1074505094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7e0151c00 0x7fa7e01e74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:07:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 -- 172.21.15.7:0/1074505094 >> 172.21.15.7:0/1074505094 conn(0x7fa7e00a94d0 msgr2=0x7fa7e00c6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:07:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 -- 172.21.15.7:0/1074505094 shutdown_connections 2023-03-03T14:07:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:07:06.364+0000 7fa7e515d700 1 -- 172.21.15.7:0/1074505094 wait complete. 2023-03-03T14:07:06.367 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:07:06.378 INFO:tasks.mds_thrash.fs.[b]:waiting for 110 secs before thrashing 2023-03-03T14:07:19.988 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:19.993 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:50.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:07:50.059 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:20.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:20.127 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:39.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:08:39.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.241+0000 7f248c8c5700 1 Processor -- start 2023-03-03T14:08:39.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.241+0000 7f248c8c5700 1 -- start start 2023-03-03T14:08:39.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f248c8c5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f24880c00e0 0x7f24880c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:39.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f248c8c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2488154ff0 0x7f24880cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f248c8c5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24880c7310 con 0x7f24880caa30 2023-03-03T14:08:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f248c8c5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2488155430 con 0x7f24880c00e0 2023-03-03T14:08:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f248c8c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24880bfd10 con 0x7f2488154ff0 2023-03-03T14:08:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f2485d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f24880c00e0 0x7f24880c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f248659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f24880caa30 0x7f24880c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41390/0 (socket says 172.21.15.7:41390) 2023-03-03T14:08:39.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f248659c700 1 -- 172.21.15.7:0/3462110634 learned_addr learned my addr 172.21.15.7:0/3462110634 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:08:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f2486d9d700 1 --2- 172.21.15.7:0/3462110634 >> v2:172.21.15.143:3300/0 conn(0x7f2488154ff0 0x7f24880cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f2485d9b700 1 -- 172.21.15.7:0/3462110634 >> v2:172.21.15.143:3300/0 conn(0x7f2488154ff0 msgr2=0x7f24880cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:08:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f2485d9b700 1 --2- 172.21.15.7:0/3462110634 >> v2:172.21.15.143:3300/0 conn(0x7f2488154ff0 0x7f24880cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f2485d9b700 1 -- 172.21.15.7:0/3462110634 >> v1:172.21.15.7:6789/0 conn(0x7f24880caa30 legacy=0x7f24880c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f2485d9b700 1 -- 172.21.15.7:0/3462110634 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24880bfa30 con 0x7f24880c00e0 2023-03-03T14:08:39.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.242+0000 7f2486d9d700 1 --2- 172.21.15.7:0/3462110634 >> v2:172.21.15.143:3300/0 conn(0x7f2488154ff0 0x7f24880cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:08:39.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f2485d9b700 1 --2- 172.21.15.7:0/3462110634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880c00e0 0x7f24880c5c60 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f2488071530 tx=0x7f2478009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e253cb8c4d8a63d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:08:39.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248559a700 1 -- 172.21.15.7:0/3462110634 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478005e30 con 0x7f24880c00e0 2023-03-03T14:08:39.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248559a700 1 -- 172.21.15.7:0/3462110634 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2478006030 con 0x7f24880c00e0 2023-03-03T14:08:39.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248559a700 1 -- 172.21.15.7:0/3462110634 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24780063c0 con 0x7f24880c00e0 2023-03-03T14:08:39.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248c8c5700 1 -- 172.21.15.7:0/3462110634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880c00e0 msgr2=0x7f24880c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:39.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248c8c5700 1 --2- 172.21.15.7:0/3462110634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880c00e0 0x7f24880c5c60 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f2488071530 tx=0x7f2478009b00 comp rx=0 tx=0).stop 2023-03-03T14:08:39.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248c8c5700 1 -- 172.21.15.7:0/3462110634 shutdown_connections 2023-03-03T14:08:39.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248c8c5700 1 --2- 172.21.15.7:0/3462110634 >> v2:172.21.15.143:3300/0 conn(0x7f2488154ff0 0x7f24880cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:39.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248c8c5700 1 --2- 172.21.15.7:0/3462110634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880c00e0 0x7f24880c5c60 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:39.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248c8c5700 1 -- 172.21.15.7:0/3462110634 >> 172.21.15.7:0/3462110634 conn(0x7f2488066e40 msgr2=0x7f2488071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:08:39.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248c8c5700 1 -- 172.21.15.7:0/3462110634 shutdown_connections 2023-03-03T14:08:39.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.243+0000 7f248c8c5700 1 -- 172.21.15.7:0/3462110634 wait complete. 2023-03-03T14:08:39.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.245+0000 7f248c8c5700 1 Processor -- start 2023-03-03T14:08:39.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.245+0000 7f248c8c5700 1 -- start start 2023-03-03T14:08:39.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.245+0000 7f248c8c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c00e0 0x7f24880c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:39.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.245+0000 7f248c8c5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880caa30 0x7f248814a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:39.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.245+0000 7f248c8c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2488154ff0 0x7f248814a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:39.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.245+0000 7f248c8c5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2488158450 con 0x7f24880caa30 2023-03-03T14:08:39.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.245+0000 7f248c8c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24881ecf90 con 0x7f24880c00e0 2023-03-03T14:08:39.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.245+0000 7f248c8c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24881ed830 con 0x7f2488154ff0 2023-03-03T14:08:39.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2485d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880caa30 0x7f248814a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:39.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2485d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880caa30 0x7f248814a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47418/0 (socket says 172.21.15.7:47418) 2023-03-03T14:08:39.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2485d9b700 1 -- 172.21.15.7:0/1646609326 learned_addr learned my addr 172.21.15.7:0/1646609326 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:08:39.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f248659c700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c00e0 0x7f24880c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:39.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2486d9d700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2488154ff0 0x7f248814a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:39.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2485d9b700 1 -- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2488154ff0 msgr2=0x7f248814a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:39.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2485d9b700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2488154ff0 0x7f248814a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:39.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2485d9b700 1 -- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c00e0 msgr2=0x7f24880c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:39.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2485d9b700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c00e0 0x7f24880c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:39.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2485d9b700 1 -- 172.21.15.7:0/1646609326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2478009430 con 0x7f24880caa30 2023-03-03T14:08:39.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f248659c700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c00e0 0x7f24880c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:08:39.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2486d9d700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2488154ff0 0x7f248814a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:08:39.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2485d9b700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880caa30 0x7f248814a090 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f2478009b40 tx=0x7f2478004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:08:39.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2476ffd700 1 -- 172.21.15.7:0/1646609326 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2478004240 con 0x7f24880caa30 2023-03-03T14:08:39.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2476ffd700 1 -- 172.21.15.7:0/1646609326 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24780043e0 con 0x7f24880caa30 2023-03-03T14:08:39.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f248c8c5700 1 -- 172.21.15.7:0/1646609326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24881547f0 con 0x7f24880caa30 2023-03-03T14:08:39.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f2476ffd700 1 -- 172.21.15.7:0/1646609326 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24780036a0 con 0x7f24880caa30 2023-03-03T14:08:39.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.246+0000 7f248c8c5700 1 -- 172.21.15.7:0/1646609326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2488154510 con 0x7f24880caa30 2023-03-03T14:08:39.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.247+0000 7f248c8c5700 1 -- 172.21.15.7:0/1646609326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2488154ad0 con 0x7f24880caa30 2023-03-03T14:08:39.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.248+0000 7f2476ffd700 1 -- 172.21.15.7:0/1646609326 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f247800f050 con 0x7f24880caa30 2023-03-03T14:08:39.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.248+0000 7f2476ffd700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24680682f0 0x7f246806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:39.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.249+0000 7f2476ffd700 1 -- 172.21.15.7:0/1646609326 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2478087da0 con 0x7f24880caa30 2023-03-03T14:08:39.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.249+0000 7f248659c700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24680682f0 0x7f246806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:39.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.249+0000 7f248659c700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24680682f0 0x7f246806a7a0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f247c005910 tx=0x7f247c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:08:39.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.253+0000 7f2476ffd700 1 -- 172.21.15.7:0/1646609326 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24780540c0 con 0x7f24880caa30 2023-03-03T14:08:39.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.491+0000 7f248c8c5700 1 -- 172.21.15.7:0/1646609326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2488074bb0 con 0x7f24880caa30 2023-03-03T14:08:39.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.492+0000 7f2476ffd700 1 -- 172.21.15.7:0/1646609326 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f247802a020 con 0x7f24880caa30 2023-03-03T14:08:39.493 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:08:39.494 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:08:39.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 -- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24680682f0 msgr2=0x7f246806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:39.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24680682f0 0x7f246806a7a0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f247c005910 tx=0x7f247c009040 comp rx=0 tx=0).stop 2023-03-03T14:08:39.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 -- 172.21.15.7:0/1646609326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880caa30 msgr2=0x7f248814a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:39.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880caa30 0x7f248814a090 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f2478009b40 tx=0x7f2478004750 comp rx=0 tx=0).stop 2023-03-03T14:08:39.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 -- 172.21.15.7:0/1646609326 shutdown_connections 2023-03-03T14:08:39.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2488154ff0 0x7f248814a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:39.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24880c00e0 0x7f24880c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:39.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24680682f0 0x7f246806a7a0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:39.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 --2- 172.21.15.7:0/1646609326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24880caa30 0x7f248814a090 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:39.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 -- 172.21.15.7:0/1646609326 >> 172.21.15.7:0/1646609326 conn(0x7f2488066e40 msgr2=0x7f24880736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:08:39.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 -- 172.21.15.7:0/1646609326 shutdown_connections 2023-03-03T14:08:39.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:39.494+0000 7f248c8c5700 1 -- 172.21.15.7:0/1646609326 wait complete. 2023-03-03T14:08:39.497 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:08:39.509 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2023-03-03T14:08:50.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:50.193 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:08:56.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:08:56.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.519+0000 7f322859a700 1 Processor -- start 2023-03-03T14:08:56.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.519+0000 7f322859a700 1 -- start start 2023-03-03T14:08:56.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3220151d40 0x7f32200736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:56.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32200c2930 0x7f3220157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:56.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3220073c30 0x7f3220157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:56.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322006c2a0 con 0x7f3220073c30 2023-03-03T14:08:56.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322006d430 con 0x7f3220151d40 2023-03-03T14:08:56.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322006cb00 con 0x7f32200c2930 2023-03-03T14:08:56.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3226b37700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3220073c30 0x7f3220157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:56.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3226b37700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3220073c30 0x7f3220157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47428/0 (socket says 172.21.15.7:47428) 2023-03-03T14:08:56.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3226b37700 1 -- 172.21.15.7:0/1894523118 learned_addr learned my addr 172.21.15.7:0/1894523118 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:08:56.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3225b35700 1 --2- 172.21.15.7:0/1894523118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32200c2930 0x7f3220157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3226336700 1 --2- 172.21.15.7:0/1894523118 >> v2:172.21.15.143:3300/0 conn(0x7f3220151d40 0x7f32200736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3226b37700 1 -- 172.21.15.7:0/1894523118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32200c2930 msgr2=0x7f3220157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:08:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3226b37700 1 --2- 172.21.15.7:0/1894523118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32200c2930 0x7f3220157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3226b37700 1 -- 172.21.15.7:0/1894523118 >> v2:172.21.15.143:3300/0 conn(0x7f3220151d40 msgr2=0x7f32200736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:08:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3226b37700 1 --2- 172.21.15.7:0/1894523118 >> v2:172.21.15.143:3300/0 conn(0x7f3220151d40 0x7f32200736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3226b37700 1 -- 172.21.15.7:0/1894523118 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322006cde0 con 0x7f3220073c30 2023-03-03T14:08:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3226b37700 1 --2- 172.21.15.7:0/1894523118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3220073c30 0x7f3220157fd0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f322006e440 tx=0x7f321800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f3e7b1b213d84f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:08:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3225334700 1 -- 172.21.15.7:0/1894523118 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321800e070 con 0x7f3220073c30 2023-03-03T14:08:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3225334700 1 -- 172.21.15.7:0/1894523118 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321800b910 con 0x7f3220073c30 2023-03-03T14:08:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f3225334700 1 -- 172.21.15.7:0/1894523118 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321800bcc0 con 0x7f3220073c30 2023-03-03T14:08:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 -- 172.21.15.7:0/1894523118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3220073c30 msgr2=0x7f3220157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 --2- 172.21.15.7:0/1894523118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3220073c30 0x7f3220157fd0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f322006e440 tx=0x7f321800b330 comp rx=0 tx=0).stop 2023-03-03T14:08:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 -- 172.21.15.7:0/1894523118 shutdown_connections 2023-03-03T14:08:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 --2- 172.21.15.7:0/1894523118 >> v2:172.21.15.143:3300/0 conn(0x7f3220151d40 0x7f32200736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 --2- 172.21.15.7:0/1894523118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3220073c30 0x7f3220157fd0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 --2- 172.21.15.7:0/1894523118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32200c2930 0x7f3220157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 -- 172.21.15.7:0/1894523118 >> 172.21.15.7:0/1894523118 conn(0x7f32200a94d0 msgr2=0x7f32200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:08:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 -- 172.21.15.7:0/1894523118 shutdown_connections 2023-03-03T14:08:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.521+0000 7f322859a700 1 -- 172.21.15.7:0/1894523118 wait complete. 2023-03-03T14:08:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.522+0000 7f322859a700 1 Processor -- start 2023-03-03T14:08:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f322859a700 1 -- start start 2023-03-03T14:08:56.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f322859a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3220073c30 0x7f32200c00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:56.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f322859a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32200c2930 0x7f322014e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f322859a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220151d40 0x7f322014b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f322859a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32200c0610 con 0x7f32200c2930 2023-03-03T14:08:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f322859a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32200acae0 con 0x7f3220151d40 2023-03-03T14:08:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f322859a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32200acdc0 con 0x7f3220073c30 2023-03-03T14:08:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f3225b35700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32200c2930 0x7f322014e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f3225b35700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32200c2930 0x7f322014e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47432/0 (socket says 172.21.15.7:47432) 2023-03-03T14:08:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f3225b35700 1 -- 172.21.15.7:0/867270814 learned_addr learned my addr 172.21.15.7:0/867270814 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:08:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f3226336700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3220073c30 0x7f32200c00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.523+0000 7f3226b37700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220151d40 0x7f322014b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f3225b35700 1 -- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3220073c30 msgr2=0x7f32200c00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f3225b35700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3220073c30 0x7f32200c00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f3225b35700 1 -- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220151d40 msgr2=0x7f322014b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f3225b35700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220151d40 0x7f322014b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f3225b35700 1 -- 172.21.15.7:0/867270814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3218009b30 con 0x7f32200c2930 2023-03-03T14:08:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f3225b35700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32200c2930 0x7f322014e020 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f321000e9b0 tx=0x7f321000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:08:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f3216ffd700 1 -- 172.21.15.7:0/867270814 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210009710 con 0x7f32200c2930 2023-03-03T14:08:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f3216ffd700 1 -- 172.21.15.7:0/867270814 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321000ee50 con 0x7f32200c2930 2023-03-03T14:08:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f322859a700 1 -- 172.21.15.7:0/867270814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322006bea0 con 0x7f32200c2930 2023-03-03T14:08:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f3216ffd700 1 -- 172.21.15.7:0/867270814 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321000ca50 con 0x7f32200c2930 2023-03-03T14:08:56.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.524+0000 7f322859a700 1 -- 172.21.15.7:0/867270814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322006d0c0 con 0x7f32200c2930 2023-03-03T14:08:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.525+0000 7f322859a700 1 -- 172.21.15.7:0/867270814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322006c030 con 0x7f32200c2930 2023-03-03T14:08:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.526+0000 7f3216ffd700 1 -- 172.21.15.7:0/867270814 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3210023070 con 0x7f32200c2930 2023-03-03T14:08:56.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.526+0000 7f3216ffd700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32040682f0 0x7f320406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:08:56.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.526+0000 7f3216ffd700 1 -- 172.21.15.7:0/867270814 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3210086300 con 0x7f32200c2930 2023-03-03T14:08:56.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.526+0000 7f3226336700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32040682f0 0x7f320406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:08:56.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.527+0000 7f3226336700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32040682f0 0x7f320406a7a0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f322014e8d0 tx=0x7f321c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:08:56.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.531+0000 7f3216ffd700 1 -- 172.21.15.7:0/867270814 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3210052640 con 0x7f32200c2930 2023-03-03T14:08:56.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.768+0000 7f322859a700 1 -- 172.21.15.7:0/867270814 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32200bea20 con 0x7f32200c2930 2023-03-03T14:08:56.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.768+0000 7f3216ffd700 1 -- 172.21.15.7:0/867270814 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f32200bea20 con 0x7f32200c2930 2023-03-03T14:08:56.771 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:08:56.771 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:08:56.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 -- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32040682f0 msgr2=0x7f320406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:56.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32040682f0 0x7f320406a7a0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f322014e8d0 tx=0x7f321c009210 comp rx=0 tx=0).stop 2023-03-03T14:08:56.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 -- 172.21.15.7:0/867270814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32200c2930 msgr2=0x7f322014e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:08:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32200c2930 0x7f322014e020 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f321000e9b0 tx=0x7f321000ecc0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 -- 172.21.15.7:0/867270814 shutdown_connections 2023-03-03T14:08:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3220151d40 0x7f322014b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3220073c30 0x7f32200c00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32040682f0 0x7f320406a7a0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 --2- 172.21.15.7:0/867270814 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32200c2930 0x7f322014e020 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:08:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 -- 172.21.15.7:0/867270814 >> 172.21.15.7:0/867270814 conn(0x7f32200a94d0 msgr2=0x7f32200714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:08:56.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 -- 172.21.15.7:0/867270814 shutdown_connections 2023-03-03T14:08:56.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:08:56.770+0000 7f322859a700 1 -- 172.21.15.7:0/867270814 wait complete. 2023-03-03T14:08:56.780 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:08:56.787 INFO:tasks.mds_thrash.fs.[b]:waiting for 61 secs before thrashing 2023-03-03T14:09:20.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:20.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:50.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:50.354 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:09:57.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:09:57.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.937+0000 7f759039e700 1 Processor -- start 2023-03-03T14:09:57.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.937+0000 7f759039e700 1 -- start start 2023-03-03T14:09:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f759039e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75880ac340 0x7f75881510b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f759039e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f75880ab520 0x7f75880c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f759039e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75881515f0 con 0x7f75880c7820 2023-03-03T14:09:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f759039e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f758806d9c0 con 0x7f75880ab520 2023-03-03T14:09:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f759039e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7588151830 con 0x7f75880ac340 2023-03-03T14:09:57.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f758d939700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f75880ab520 0x7f75880c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f758e93b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f75880c7820 0x7f75880c4360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41474/0 (socket says 172.21.15.7:41474) 2023-03-03T14:09:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f758e93b700 1 -- 172.21.15.7:0/3445967380 learned_addr learned my addr 172.21.15.7:0/3445967380 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:09:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f758e13a700 1 --2- 172.21.15.7:0/3445967380 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75880ac340 0x7f75881510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f758d939700 1 -- 172.21.15.7:0/3445967380 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75880ac340 msgr2=0x7f75881510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:09:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f758d939700 1 --2- 172.21.15.7:0/3445967380 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75880ac340 0x7f75881510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f758d939700 1 -- 172.21.15.7:0/3445967380 >> v1:172.21.15.7:6789/0 conn(0x7f75880c7820 legacy=0x7f75880c4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f758d138700 1 -- 172.21.15.7:0/3445967380 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113923290 0 0) 0x7f7580001490 con 0x7f75880c7820 2023-03-03T14:09:57.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f758d939700 1 -- 172.21.15.7:0/3445967380 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7588151b10 con 0x7f75880ab520 2023-03-03T14:09:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.938+0000 7f758d939700 1 --2- 172.21.15.7:0/3445967380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880ab520 0x7f75880c72e0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f7578009840 tx=0x7f7578009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c50152d49fcd6c2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f758d138700 1 -- 172.21.15.7:0/3445967380 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578011830 con 0x7f75880ab520 2023-03-03T14:09:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f758d138700 1 -- 172.21.15.7:0/3445967380 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7578011990 con 0x7f75880ab520 2023-03-03T14:09:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f758d138700 1 -- 172.21.15.7:0/3445967380 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7578011d20 con 0x7f75880ab520 2023-03-03T14:09:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f759039e700 1 -- 172.21.15.7:0/3445967380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880ab520 msgr2=0x7f75880c72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f759039e700 1 --2- 172.21.15.7:0/3445967380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880ab520 0x7f75880c72e0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f7578009840 tx=0x7f7578009b50 comp rx=0 tx=0).stop 2023-03-03T14:09:57.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f759039e700 1 -- 172.21.15.7:0/3445967380 shutdown_connections 2023-03-03T14:09:57.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f759039e700 1 --2- 172.21.15.7:0/3445967380 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75880ac340 0x7f75881510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:57.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f759039e700 1 --2- 172.21.15.7:0/3445967380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880ab520 0x7f75880c72e0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:57.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f759039e700 1 -- 172.21.15.7:0/3445967380 >> 172.21.15.7:0/3445967380 conn(0x7f758806c060 msgr2=0x7f758806c460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:09:57.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f759039e700 1 -- 172.21.15.7:0/3445967380 shutdown_connections 2023-03-03T14:09:57.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.939+0000 7f759039e700 1 -- 172.21.15.7:0/3445967380 wait complete. 2023-03-03T14:09:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.940+0000 7f759039e700 1 Processor -- start 2023-03-03T14:09:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.941+0000 7f759039e700 1 -- start start 2023-03-03T14:09:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.941+0000 7f759039e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75880ab520 0x7f7588155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.941+0000 7f759039e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75880ac340 0x7f7588149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.941+0000 7f759039e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880c7820 0x7f7588152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.941+0000 7f759039e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75881535b0 con 0x7f75880c7820 2023-03-03T14:09:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.941+0000 7f759039e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758814abe0 con 0x7f75880ab520 2023-03-03T14:09:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.941+0000 7f759039e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7588153280 con 0x7f75880ac340 2023-03-03T14:09:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.941+0000 7f758e93b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880c7820 0x7f7588152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.941+0000 7f758e93b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880c7820 0x7f7588152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47500/0 (socket says 172.21.15.7:47500) 2023-03-03T14:09:57.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.941+0000 7f758e93b700 1 -- 172.21.15.7:0/3276373046 learned_addr learned my addr 172.21.15.7:0/3276373046 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:09:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f758d939700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75880ac340 0x7f7588149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f758e13a700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75880ab520 0x7f7588155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f758e93b700 1 -- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75880ac340 msgr2=0x7f7588149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f758e93b700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75880ac340 0x7f7588149890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f758e93b700 1 -- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75880ab520 msgr2=0x7f7588155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f758e93b700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75880ab520 0x7f7588155030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f758e93b700 1 -- 172.21.15.7:0/3276373046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75780094a0 con 0x7f75880c7820 2023-03-03T14:09:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f758e93b700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880c7820 0x7f7588152260 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f758000cca0 tx=0x7f758000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f757effd700 1 -- 172.21.15.7:0/3276373046 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758000cea0 con 0x7f75880c7820 2023-03-03T14:09:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f757effd700 1 -- 172.21.15.7:0/3276373046 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758000ada0 con 0x7f75880c7820 2023-03-03T14:09:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f757effd700 1 -- 172.21.15.7:0/3276373046 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580007310 con 0x7f75880c7820 2023-03-03T14:09:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f759039e700 1 -- 172.21.15.7:0/3276373046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75880c3e40 con 0x7f75880c7820 2023-03-03T14:09:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.942+0000 7f759039e700 1 -- 172.21.15.7:0/3276373046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7588151df0 con 0x7f75880c7820 2023-03-03T14:09:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.943+0000 7f759039e700 1 -- 172.21.15.7:0/3276373046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75880c3b00 con 0x7f75880c7820 2023-03-03T14:09:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.943+0000 7f757effd700 1 -- 172.21.15.7:0/3276373046 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7580010070 con 0x7f75880c7820 2023-03-03T14:09:57.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.944+0000 7f757effd700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f756c0682f0 0x7f756c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:57.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.944+0000 7f757effd700 1 -- 172.21.15.7:0/3276373046 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f75800859f0 con 0x7f75880c7820 2023-03-03T14:09:57.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.944+0000 7f758e13a700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f756c0682f0 0x7f756c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:57.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.945+0000 7f758e13a700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f756c0682f0 0x7f756c06a7a0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f758400b840 tx=0x7f7584008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:57.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:57.948+0000 7f757effd700 1 -- 172.21.15.7:0/3276373046 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7580087e60 con 0x7f75880c7820 2023-03-03T14:09:58.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.184+0000 7f759039e700 1 -- 172.21.15.7:0/3276373046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75881515f0 con 0x7f75880c7820 2023-03-03T14:09:58.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.185+0000 7f757effd700 1 -- 172.21.15.7:0/3276373046 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f75881515f0 con 0x7f75880c7820 2023-03-03T14:09:58.188 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:09:58.188 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:09:58.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.186+0000 7f759039e700 1 -- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f756c0682f0 msgr2=0x7f756c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:58.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.186+0000 7f759039e700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f756c0682f0 0x7f756c06a7a0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f758400b840 tx=0x7f7584008040 comp rx=0 tx=0).stop 2023-03-03T14:09:58.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.186+0000 7f759039e700 1 -- 172.21.15.7:0/3276373046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880c7820 msgr2=0x7f7588152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:58.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.186+0000 7f759039e700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880c7820 0x7f7588152260 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f758000cca0 tx=0x7f758000a340 comp rx=0 tx=0).stop 2023-03-03T14:09:58.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.186+0000 7f759039e700 1 -- 172.21.15.7:0/3276373046 shutdown_connections 2023-03-03T14:09:58.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.186+0000 7f759039e700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f756c0682f0 0x7f756c06a7a0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.187+0000 7f759039e700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75880c7820 0x7f7588152260 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.187+0000 7f759039e700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75880ab520 0x7f7588155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.187+0000 7f759039e700 1 --2- 172.21.15.7:0/3276373046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75880ac340 0x7f7588149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.187+0000 7f759039e700 1 -- 172.21.15.7:0/3276373046 >> 172.21.15.7:0/3276373046 conn(0x7f758806c060 msgr2=0x7f7588070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:09:58.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.187+0000 7f759039e700 1 -- 172.21.15.7:0/3276373046 shutdown_connections 2023-03-03T14:09:58.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.187+0000 7f759039e700 1 -- 172.21.15.7:0/3276373046 wait complete. 2023-03-03T14:09:58.191 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:09:58.202 INFO:tasks.mds_thrash.fs.[b]:waiting for 87 secs before thrashing 2023-03-03T14:09:58.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:09:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.641+0000 7fe89c3bc700 1 Processor -- start 2023-03-03T14:09:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.641+0000 7fe89c3bc700 1 -- start start 2023-03-03T14:09:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89c3bc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe8940bebd0 0x7fe8940c4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89c3bc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8940aab00 con 0x7fe8940c4530 2023-03-03T14:09:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89c3bc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8940aade0 con 0x7fe894072000 2023-03-03T14:09:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89c3bc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8940ab0c0 con 0x7fe8940bebd0 2023-03-03T14:09:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89a158700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe8940c4530 0x7fe89406e430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41484/0 (socket says 172.21.15.7:41484) 2023-03-03T14:09:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89a158700 1 -- 172.21.15.7:0/222679310 learned_addr learned my addr 172.21.15.7:0/222679310 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:09:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89a959700 1 --2- 172.21.15.7:0/222679310 >> v2:172.21.15.143:3300/0 conn(0x7fe8940bebd0 0x7fe8940c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe899156700 1 -- 172.21.15.7:0/222679310 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860950044 0 0) 0x7fe8940aab00 con 0x7fe8940c4530 2023-03-03T14:09:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe899156700 1 -- 172.21.15.7:0/222679310 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8800035b0 con 0x7fe8940c4530 2023-03-03T14:09:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89a959700 1 -- 172.21.15.7:0/222679310 >> v1:172.21.15.143:6789/0 conn(0x7fe894072000 legacy=0x7fe894152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89a959700 1 -- 172.21.15.7:0/222679310 >> v1:172.21.15.7:6789/0 conn(0x7fe8940c4530 legacy=0x7fe89406e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89a959700 1 -- 172.21.15.7:0/222679310 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894152730 con 0x7fe8940bebd0 2023-03-03T14:09:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.643+0000 7fe89a959700 1 --2- 172.21.15.7:0/222679310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8940bebd0 0x7fe8940c4900 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009d90 tx=0x7fe88c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c148ceada0d1dd67 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.644+0000 7fe899156700 1 -- 172.21.15.7:0/222679310 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c00e070 con 0x7fe8940bebd0 2023-03-03T14:09:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.644+0000 7fe899156700 1 -- 172.21.15.7:0/222679310 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c00bb20 con 0x7fe8940bebd0 2023-03-03T14:09:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.644+0000 7fe899156700 1 -- 172.21.15.7:0/222679310 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c012430 con 0x7fe8940bebd0 2023-03-03T14:09:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.644+0000 7fe89c3bc700 1 -- 172.21.15.7:0/222679310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8940bebd0 msgr2=0x7fe8940c4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.644+0000 7fe89c3bc700 1 --2- 172.21.15.7:0/222679310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8940bebd0 0x7fe8940c4900 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009d90 tx=0x7fe88c00b330 comp rx=0 tx=0).stop 2023-03-03T14:09:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.644+0000 7fe89c3bc700 1 -- 172.21.15.7:0/222679310 shutdown_connections 2023-03-03T14:09:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.644+0000 7fe89c3bc700 1 --2- 172.21.15.7:0/222679310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8940bebd0 0x7fe8940c4900 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.644+0000 7fe89c3bc700 1 -- 172.21.15.7:0/222679310 >> 172.21.15.7:0/222679310 conn(0x7fe8940c8510 msgr2=0x7fe8940c8910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:09:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.644+0000 7fe89c3bc700 1 -- 172.21.15.7:0/222679310 shutdown_connections 2023-03-03T14:09:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.644+0000 7fe89c3bc700 1 -- 172.21.15.7:0/222679310 wait complete. 2023-03-03T14:09:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.646+0000 7fe89c3bc700 1 Processor -- start 2023-03-03T14:09:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.646+0000 7fe89c3bc700 1 -- start start 2023-03-03T14:09:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe89c3bc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894072000 0x7fe89414d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe89c3bc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8940bebd0 0x7fe8940a9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe89c3bc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8940c4530 0x7fe8941eb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe89c3bc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89414cf20 con 0x7fe8940bebd0 2023-03-03T14:09:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe89c3bc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe894145590 con 0x7fe894072000 2023-03-03T14:09:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe89c3bc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8940c8330 con 0x7fe8940c4530 2023-03-03T14:09:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe899957700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8940bebd0 0x7fe8940a9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe899957700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8940bebd0 0x7fe8940a9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47512/0 (socket says 172.21.15.7:47512) 2023-03-03T14:09:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe899957700 1 -- 172.21.15.7:0/1484078 learned_addr learned my addr 172.21.15.7:0/1484078 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:09:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe89a158700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894072000 0x7fe89414d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe89a959700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8940c4530 0x7fe8941eb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe899957700 1 -- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8940c4530 msgr2=0x7fe8941eb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe899957700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8940c4530 0x7fe8941eb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe899957700 1 -- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894072000 msgr2=0x7fe89414d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:58.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe899957700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894072000 0x7fe89414d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe899957700 1 -- 172.21.15.7:0/1484078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c009a70 con 0x7fe8940bebd0 2023-03-03T14:09:58.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe89a158700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894072000 0x7fe89414d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:09:58.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe899957700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8940bebd0 0x7fe8940a9c30 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fe89000c450 tx=0x7fe89000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:58.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe88affd700 1 -- 172.21.15.7:0/1484078 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8900052d0 con 0x7fe8940bebd0 2023-03-03T14:09:58.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe88affd700 1 -- 172.21.15.7:0/1484078 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe890005430 con 0x7fe8940bebd0 2023-03-03T14:09:58.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe88affd700 1 -- 172.21.15.7:0/1484078 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8900038c0 con 0x7fe8940bebd0 2023-03-03T14:09:58.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.647+0000 7fe89c3bc700 1 -- 172.21.15.7:0/1484078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89414fe70 con 0x7fe8940bebd0 2023-03-03T14:09:58.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.648+0000 7fe89c3bc700 1 -- 172.21.15.7:0/1484078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe894152a10 con 0x7fe8940bebd0 2023-03-03T14:09:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.648+0000 7fe89c3bc700 1 -- 172.21.15.7:0/1484078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8940c2e90 con 0x7fe8940bebd0 2023-03-03T14:09:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.649+0000 7fe88affd700 1 -- 172.21.15.7:0/1484078 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe890019030 con 0x7fe8940bebd0 2023-03-03T14:09:58.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.649+0000 7fe88affd700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe878068340 0x7fe87806a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:09:58.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.649+0000 7fe88affd700 1 -- 172.21.15.7:0/1484078 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe8900853b0 con 0x7fe8940bebd0 2023-03-03T14:09:58.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.650+0000 7fe89a158700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe878068340 0x7fe87806a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:09:58.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.650+0000 7fe89a158700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe878068340 0x7fe87806a7f0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fe8840011e0 tx=0x7fe884009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:09:58.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.654+0000 7fe88affd700 1 -- 172.21.15.7:0/1484078 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8900517f0 con 0x7fe8940bebd0 2023-03-03T14:09:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.893+0000 7fe89c3bc700 1 -- 172.21.15.7:0/1484078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe89414d060 con 0x7fe8940bebd0 2023-03-03T14:09:58.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.894+0000 7fe88affd700 1 -- 172.21.15.7:0/1484078 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe89414d060 con 0x7fe8940bebd0 2023-03-03T14:09:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:09:58.897 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:09:58.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 -- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe878068340 msgr2=0x7fe87806a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:58.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe878068340 0x7fe87806a7f0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fe8840011e0 tx=0x7fe884009040 comp rx=0 tx=0).stop 2023-03-03T14:09:58.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 -- 172.21.15.7:0/1484078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8940bebd0 msgr2=0x7fe8940a9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:09:58.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8940bebd0 0x7fe8940a9c30 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fe89000c450 tx=0x7fe89000c810 comp rx=0 tx=0).stop 2023-03-03T14:09:58.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 -- 172.21.15.7:0/1484078 shutdown_connections 2023-03-03T14:09:58.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8940c4530 0x7fe8941eb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe894072000 0x7fe89414d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe878068340 0x7fe87806a7f0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 --2- 172.21.15.7:0/1484078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8940bebd0 0x7fe8940a9c30 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:09:58.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 -- 172.21.15.7:0/1484078 >> 172.21.15.7:0/1484078 conn(0x7fe8940c8510 msgr2=0x7fe89414f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:09:58.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 -- 172.21.15.7:0/1484078 shutdown_connections 2023-03-03T14:09:58.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:09:58.896+0000 7fe89c3bc700 1 -- 172.21.15.7:0/1484078 wait complete. 2023-03-03T14:09:58.903 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:09:58.911 INFO:tasks.mds_thrash.fs.[a]:waiting for 40 secs before thrashing 2023-03-03T14:10:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T14:09:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T14:10:20.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:20.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:38.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:10:39.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.050+0000 7fb83e377700 1 Processor -- start 2023-03-03T14:10:39.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.050+0000 7fb83e377700 1 -- start start 2023-03-03T14:10:39.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.051+0000 7fb83e377700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb838155410 0x7fb8380bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:10:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.051+0000 7fb83e377700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb8380c5b00 0x7fb8380bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:10:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.051+0000 7fb83e377700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8380c08d0 con 0x7fb838151e80 2023-03-03T14:10:39.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.051+0000 7fb83e377700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb838152250 con 0x7fb8380c5b00 2023-03-03T14:10:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.051+0000 7fb83e377700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8380c0fb0 con 0x7fb838155410 2023-03-03T14:10:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.051+0000 7fb83c914700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb8380c5b00 0x7fb8380bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:10:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.051+0000 7fb83c914700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb8380c5b00 0x7fb8380bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47556/0 (socket says 172.21.15.7:47556) 2023-03-03T14:10:39.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.051+0000 7fb837fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb838151e80 0x7fb8380c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41532/0 (socket says 172.21.15.7:41532) 2023-03-03T14:10:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.051+0000 7fb83c914700 1 -- 172.21.15.7:0/1843880025 learned_addr learned my addr 172.21.15.7:0/1843880025 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:10:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.052+0000 7fb8377fe700 1 --2- 172.21.15.7:0/1843880025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb838155410 0x7fb8380bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:10:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.052+0000 7fb83c914700 1 -- 172.21.15.7:0/1843880025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb838155410 msgr2=0x7fb8380bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:10:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.052+0000 7fb83c914700 1 --2- 172.21.15.7:0/1843880025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb838155410 0x7fb8380bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:10:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.052+0000 7fb83c914700 1 -- 172.21.15.7:0/1843880025 >> v1:172.21.15.7:6789/0 conn(0x7fb838151e80 legacy=0x7fb8380c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:10:39.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.052+0000 7fb83c914700 1 -- 172.21.15.7:0/1843880025 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb838154930 con 0x7fb8380c5b00 2023-03-03T14:10:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.052+0000 7fb836ffd700 1 -- 172.21.15.7:0/1843880025 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895163289 0 0) 0x7fb8380c08d0 con 0x7fb838151e80 2023-03-03T14:10:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.052+0000 7fb83c914700 1 --2- 172.21.15.7:0/1843880025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8380c5b00 0x7fb8380bfe20 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fb830009e80 tx=0x7fb83000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447d37b0dfb2b75e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:10:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.052+0000 7fb836ffd700 1 -- 172.21.15.7:0/1843880025 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83000e070 con 0x7fb8380c5b00 2023-03-03T14:10:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.052+0000 7fb836ffd700 1 -- 172.21.15.7:0/1843880025 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83000b980 con 0x7fb8380c5b00 2023-03-03T14:10:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.052+0000 7fb836ffd700 1 -- 172.21.15.7:0/1843880025 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83000bd30 con 0x7fb8380c5b00 2023-03-03T14:10:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.053+0000 7fb83e377700 1 -- 172.21.15.7:0/1843880025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8380c5b00 msgr2=0x7fb8380bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:10:39.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.053+0000 7fb83e377700 1 --2- 172.21.15.7:0/1843880025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8380c5b00 0x7fb8380bfe20 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fb830009e80 tx=0x7fb83000b330 comp rx=0 tx=0).stop 2023-03-03T14:10:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.053+0000 7fb83e377700 1 -- 172.21.15.7:0/1843880025 shutdown_connections 2023-03-03T14:10:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.053+0000 7fb83e377700 1 --2- 172.21.15.7:0/1843880025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb8380c5b00 0x7fb8380bfe20 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:10:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.053+0000 7fb83e377700 1 --2- 172.21.15.7:0/1843880025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb838155410 0x7fb8380bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:10:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.053+0000 7fb83e377700 1 -- 172.21.15.7:0/1843880025 >> 172.21.15.7:0/1843880025 conn(0x7fb838071630 msgr2=0x7fb83806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:10:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.053+0000 7fb83e377700 1 -- 172.21.15.7:0/1843880025 shutdown_connections 2023-03-03T14:10:39.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.053+0000 7fb83e377700 1 -- 172.21.15.7:0/1843880025 wait complete. 2023-03-03T14:10:39.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.054+0000 7fb83e377700 1 Processor -- start 2023-03-03T14:10:39.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.054+0000 7fb83e377700 1 -- start start 2023-03-03T14:10:39.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb83e377700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8380c5b00 0x7fb838153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:10:39.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb83e377700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb838151e80 0x7fb8381edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:10:39.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb83e377700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb838155410 0x7fb8381ee080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:10:39.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb83e377700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83814f280 con 0x7fb838151e80 2023-03-03T14:10:39.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb83e377700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8380b0240 con 0x7fb8380c5b00 2023-03-03T14:10:39.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb83e377700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8380aa2a0 con 0x7fb838155410 2023-03-03T14:10:39.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb8377fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb838151e80 0x7fb8381edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:10:39.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb8377fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb838151e80 0x7fb8381edb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47560/0 (socket says 172.21.15.7:47560) 2023-03-03T14:10:39.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb8377fe700 1 -- 172.21.15.7:0/96703536 learned_addr learned my addr 172.21.15.7:0/96703536 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:10:39.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb83c914700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb838155410 0x7fb8381ee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:10:39.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb837fff700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8380c5b00 0x7fb838153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:10:39.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb8377fe700 1 -- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb838155410 msgr2=0x7fb8381ee080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:10:39.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb8377fe700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb838155410 0x7fb8381ee080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:10:39.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb8377fe700 1 -- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8380c5b00 msgr2=0x7fb838153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:10:39.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb8377fe700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8380c5b00 0x7fb838153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:10:39.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb8377fe700 1 -- 172.21.15.7:0/96703536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb830009b30 con 0x7fb838151e80 2023-03-03T14:10:39.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.055+0000 7fb8377fe700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb838151e80 0x7fb8381edb40 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00e840 tx=0x7fb82c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:10:39.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.056+0000 7fb834ff9700 1 -- 172.21.15.7:0/96703536 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c013610 con 0x7fb838151e80 2023-03-03T14:10:39.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.056+0000 7fb834ff9700 1 -- 172.21.15.7:0/96703536 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c013770 con 0x7fb838151e80 2023-03-03T14:10:39.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.056+0000 7fb83e377700 1 -- 172.21.15.7:0/96703536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb838154ef0 con 0x7fb838151e80 2023-03-03T14:10:39.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.056+0000 7fb834ff9700 1 -- 172.21.15.7:0/96703536 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c013b20 con 0x7fb838151e80 2023-03-03T14:10:39.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.056+0000 7fb83e377700 1 -- 172.21.15.7:0/96703536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb838154c10 con 0x7fb838151e80 2023-03-03T14:10:39.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.057+0000 7fb83e377700 1 -- 172.21.15.7:0/96703536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8380c5220 con 0x7fb838151e80 2023-03-03T14:10:39.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.057+0000 7fb834ff9700 1 -- 172.21.15.7:0/96703536 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb82c023070 con 0x7fb838151e80 2023-03-03T14:10:39.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.057+0000 7fb834ff9700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8240682f0 0x7fb82406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:10:39.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.058+0000 7fb834ff9700 1 -- 172.21.15.7:0/96703536 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb82c085e50 con 0x7fb838151e80 2023-03-03T14:10:39.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.058+0000 7fb837fff700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8240682f0 0x7fb82406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:10:39.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.058+0000 7fb837fff700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8240682f0 0x7fb82406a7a0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb828000f60 tx=0x7fb828007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:10:39.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.062+0000 7fb834ff9700 1 -- 172.21.15.7:0/96703536 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb82c0521f0 con 0x7fb838151e80 2023-03-03T14:10:39.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.307+0000 7fb83e377700 1 -- 172.21.15.7:0/96703536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb838074bb0 con 0x7fb838151e80 2023-03-03T14:10:39.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.308+0000 7fb834ff9700 1 -- 172.21.15.7:0/96703536 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb82c055810 con 0x7fb838151e80 2023-03-03T14:10:39.322 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:10:39.322 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:10:39.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 -- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8240682f0 msgr2=0x7fb82406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:10:39.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8240682f0 0x7fb82406a7a0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb828000f60 tx=0x7fb828007c90 comp rx=0 tx=0).stop 2023-03-03T14:10:39.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 -- 172.21.15.7:0/96703536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb838151e80 msgr2=0x7fb8381edb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:10:39.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb838151e80 0x7fb8381edb40 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00e840 tx=0x7fb82c00ec00 comp rx=0 tx=0).stop 2023-03-03T14:10:39.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 -- 172.21.15.7:0/96703536 shutdown_connections 2023-03-03T14:10:39.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb838155410 0x7fb8381ee080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:10:39.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8380c5b00 0x7fb838153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:10:39.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8240682f0 0x7fb82406a7a0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:10:39.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 --2- 172.21.15.7:0/96703536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb838151e80 0x7fb8381edb40 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:10:39.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 -- 172.21.15.7:0/96703536 >> 172.21.15.7:0/96703536 conn(0x7fb838071630 msgr2=0x7fb8380c3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:10:39.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 -- 172.21.15.7:0/96703536 shutdown_connections 2023-03-03T14:10:39.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:10:39.310+0000 7fb83e377700 1 -- 172.21.15.7:0/96703536 wait complete. 2023-03-03T14:10:39.325 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:10:39.325 INFO:tasks.mds_thrash.fs.[a]:waiting for 35 secs before thrashing 2023-03-03T14:10:50.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:10:50.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:14.329 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:11:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.469+0000 7fc06eff7700 1 Processor -- start 2023-03-03T14:11:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.470+0000 7fc06eff7700 1 -- start start 2023-03-03T14:11:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.470+0000 7fc06eff7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc068151b20 0x7fc0680736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.470+0000 7fc06eff7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc0680c2950 0x7fc068157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.470+0000 7fc06eff7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06806cb70 con 0x7fc068073c30 2023-03-03T14:11:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.470+0000 7fc06eff7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc06806c2f0 con 0x7fc0680c2950 2023-03-03T14:11:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.470+0000 7fc06eff7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06806ce50 con 0x7fc068151b20 2023-03-03T14:11:14.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.470+0000 7fc06d594700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc068073c30 0x7fc068158020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41570/0 (socket says 172.21.15.7:41570) 2023-03-03T14:11:14.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.470+0000 7fc06d594700 1 -- 172.21.15.7:0/3165685753 learned_addr learned my addr 172.21.15.7:0/3165685753 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:11:14.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc067fff700 1 --2- 172.21.15.7:0/3165685753 >> v2:172.21.15.143:3300/0 conn(0x7fc0680c2950 0x7fc068157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:14.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc06cd93700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc068151b20 0x7fc0680736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:14.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627810528 0 0) 0x7fc06806cb70 con 0x7fc068073c30 2023-03-03T14:11:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0580035b0 con 0x7fc068073c30 2023-03-03T14:11:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1608948970 0 0) 0x7fc0580035b0 con 0x7fc068073c30 2023-03-03T14:11:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc068151b20 msgr2=0x7fc0680736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:11:14.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc068151b20 0x7fc0680736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:14.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 >> v2:172.21.15.143:3300/0 conn(0x7fc0680c2950 msgr2=0x7fc068157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:11:14.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 --2- 172.21.15.7:0/3165685753 >> v2:172.21.15.143:3300/0 conn(0x7fc0680c2950 0x7fc068157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:14.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06806d130 con 0x7fc068073c30 2023-03-03T14:11:14.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc0600019f0 con 0x7fc068073c30 2023-03-03T14:11:14.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc067fff700 1 --2- 172.21.15.7:0/3165685753 >> v2:172.21.15.143:3300/0 conn(0x7fc0680c2950 0x7fc068157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:11:14.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 >> v1:172.21.15.7:6789/0 conn(0x7fc068073c30 legacy=0x7fc068158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:14.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc058003a70 0x7fc058003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:14.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc068151b20 0x7fc058004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:14.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc058004d90 0x7fc058005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:14.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06806c2f0 con 0x7fc058004d90 2023-03-03T14:11:14.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06806ce50 con 0x7fc058003a70 2023-03-03T14:11:14.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc05800ba60 con 0x7fc068151b20 2023-03-03T14:11:14.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc06d594700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc058004d90 0x7fc058005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:14.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc06cd93700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc058003a70 0x7fc058003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:14.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.471+0000 7fc067fff700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc068151b20 0x7fc058004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:14.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.472+0000 7fc06d594700 1 -- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc068151b20 msgr2=0x7fc058004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:11:14.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.472+0000 7fc06d594700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc068151b20 0x7fc058004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:14.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.472+0000 7fc06d594700 1 -- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc058003a70 msgr2=0x7fc058003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:11:14.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.472+0000 7fc06d594700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc058003a70 0x7fc058003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:14.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.472+0000 7fc06d594700 1 -- 172.21.15.7:0/3165685753 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0580035b0 con 0x7fc058004d90 2023-03-03T14:11:14.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.472+0000 7fc06d594700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc058004d90 0x7fc058005200 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fc06000a890 tx=0x7fc058005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c47e26b55a6d103 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:11:14.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.472+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060012600 con 0x7fc058004d90 2023-03-03T14:11:14.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.472+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc060012760 con 0x7fc058004d90 2023-03-03T14:11:14.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.472+0000 7fc0677fe700 1 -- 172.21.15.7:0/3165685753 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060012b10 con 0x7fc058004d90 2023-03-03T14:11:14.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.472+0000 7fc06eff7700 1 -- 172.21.15.7:0/3165685753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc058004d90 msgr2=0x7fc058005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:14.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.473+0000 7fc06eff7700 1 --2- 172.21.15.7:0/3165685753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc058004d90 0x7fc058005200 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fc06000a890 tx=0x7fc058005880 comp rx=0 tx=0).stop 2023-03-03T14:11:14.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.473+0000 7fc06d594700 1 -- 172.21.15.7:0/3165685753 reap_dead start 2023-03-03T14:11:14.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.473+0000 7fc06eff7700 1 -- 172.21.15.7:0/3165685753 shutdown_connections 2023-03-03T14:11:14.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.473+0000 7fc06eff7700 1 -- 172.21.15.7:0/3165685753 >> 172.21.15.7:0/3165685753 conn(0x7fc0680a94d0 msgr2=0x7fc0680a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:11:14.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.473+0000 7fc06eff7700 1 -- 172.21.15.7:0/3165685753 shutdown_connections 2023-03-03T14:11:14.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.473+0000 7fc06eff7700 1 -- 172.21.15.7:0/3165685753 wait complete. 2023-03-03T14:11:14.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.474+0000 7fc06eff7700 1 Processor -- start 2023-03-03T14:11:14.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.474+0000 7fc06eff7700 1 -- start start 2023-03-03T14:11:14.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc06eff7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc058004d90 0x7fc0681ed710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:14.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc06eff7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0680be430 0x7fc0680bf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:14.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc06eff7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0680bf5d0 0x7fc068148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:14.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc06eff7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680c0fc0 con 0x7fc0680be430 2023-03-03T14:11:14.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc06eff7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680c1410 con 0x7fc058004d90 2023-03-03T14:11:14.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc06eff7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680be9b0 con 0x7fc0680bf5d0 2023-03-03T14:11:14.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc067fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0680be430 0x7fc0680bf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:14.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc067fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0680be430 0x7fc0680bf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47600/0 (socket says 172.21.15.7:47600) 2023-03-03T14:11:14.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc067fff700 1 -- 172.21.15.7:0/2500567912 learned_addr learned my addr 172.21.15.7:0/2500567912 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:11:14.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc06d594700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0680bf5d0 0x7fc068148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:14.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc06cd93700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc058004d90 0x7fc0681ed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:14.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc067fff700 1 -- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0680bf5d0 msgr2=0x7fc068148510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:14.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc067fff700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0680bf5d0 0x7fc068148510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:14.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc067fff700 1 -- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc058004d90 msgr2=0x7fc0681ed710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:14.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc067fff700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc058004d90 0x7fc0681ed710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:14.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc067fff700 1 -- 172.21.15.7:0/2500567912 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc068158730 con 0x7fc0680be430 2023-03-03T14:11:14.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.475+0000 7fc067fff700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0680be430 0x7fc0680bf090 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fc054011c20 tx=0x7fc05400f240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:11:14.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.476+0000 7fc0657fa700 1 -- 172.21.15.7:0/2500567912 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05400fb80 con 0x7fc0680be430 2023-03-03T14:11:14.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.476+0000 7fc0657fa700 1 -- 172.21.15.7:0/2500567912 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05400fce0 con 0x7fc0680be430 2023-03-03T14:11:14.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.476+0000 7fc06eff7700 1 -- 172.21.15.7:0/2500567912 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06806d410 con 0x7fc0680be430 2023-03-03T14:11:14.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.476+0000 7fc0657fa700 1 -- 172.21.15.7:0/2500567912 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054004240 con 0x7fc0680be430 2023-03-03T14:11:14.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.476+0000 7fc06eff7700 1 -- 172.21.15.7:0/2500567912 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc068151ef0 con 0x7fc0680be430 2023-03-03T14:11:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.477+0000 7fc06eff7700 1 -- 172.21.15.7:0/2500567912 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc068074bb0 con 0x7fc0680be430 2023-03-03T14:11:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.477+0000 7fc0657fa700 1 -- 172.21.15.7:0/2500567912 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc054024070 con 0x7fc0680be430 2023-03-03T14:11:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.478+0000 7fc0657fa700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc04c0682f0 0x7fc04c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.478+0000 7fc0657fa700 1 -- 172.21.15.7:0/2500567912 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc054086750 con 0x7fc0680be430 2023-03-03T14:11:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.478+0000 7fc06cd93700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc04c0682f0 0x7fc04c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:14.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.479+0000 7fc06cd93700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc04c0682f0 0x7fc04c06a7a0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fc05c009b20 tx=0x7fc05c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:11:14.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.482+0000 7fc0657fa700 1 -- 172.21.15.7:0/2500567912 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc054052a60 con 0x7fc0680be430 2023-03-03T14:11:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.718+0000 7fc06eff7700 1 -- 172.21.15.7:0/2500567912 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0680c3d00 con 0x7fc0680be430 2023-03-03T14:11:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.719+0000 7fc0657fa700 1 -- 172.21.15.7:0/2500567912 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc054022070 con 0x7fc0680be430 2023-03-03T14:11:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:11:14.723 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:11:14.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.720+0000 7fc06eff7700 1 -- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc04c0682f0 msgr2=0x7fc04c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:14.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.720+0000 7fc06eff7700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc04c0682f0 0x7fc04c06a7a0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fc05c009b20 tx=0x7fc05c008040 comp rx=0 tx=0).stop 2023-03-03T14:11:14.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.720+0000 7fc06eff7700 1 -- 172.21.15.7:0/2500567912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0680be430 msgr2=0x7fc0680bf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:14.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.720+0000 7fc06eff7700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0680be430 0x7fc0680bf090 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fc054011c20 tx=0x7fc05400f240 comp rx=0 tx=0).stop 2023-03-03T14:11:14.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.720+0000 7fc06eff7700 1 -- 172.21.15.7:0/2500567912 shutdown_connections 2023-03-03T14:11:14.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.720+0000 7fc06eff7700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc0680bf5d0 0x7fc068148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.720+0000 7fc06eff7700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc058004d90 0x7fc0681ed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:14.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.720+0000 7fc06eff7700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc04c0682f0 0x7fc04c06a7a0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:14.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.720+0000 7fc06eff7700 1 --2- 172.21.15.7:0/2500567912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0680be430 0x7fc0680bf090 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.721+0000 7fc06eff7700 1 -- 172.21.15.7:0/2500567912 >> 172.21.15.7:0/2500567912 conn(0x7fc0680a94d0 msgr2=0x7fc068071080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:11:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.721+0000 7fc06eff7700 1 -- 172.21.15.7:0/2500567912 shutdown_connections 2023-03-03T14:11:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:14.721+0000 7fc06eff7700 1 -- 172.21.15.7:0/2500567912 wait complete. 2023-03-03T14:11:14.736 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:11:14.750 INFO:tasks.mds_thrash.fs.[a]:waiting for 72 secs before thrashing 2023-03-03T14:11:20.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:20.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:25.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:11:25.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.343+0000 7f65d49d4700 1 Processor -- start 2023-03-03T14:11:25.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.343+0000 7f65d49d4700 1 -- start start 2023-03-03T14:11:25.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65d49d4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65d0151ab0 0x7f65d00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:25.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65d49d4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f65d006c5b0 0x7f65d0157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65d49d4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d006d780 con 0x7f65d0073c30 2023-03-03T14:11:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65d49d4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d006cb70 con 0x7f65d0151ab0 2023-03-03T14:11:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65d49d4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d006ce50 con 0x7f65d006c5b0 2023-03-03T14:11:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65ce59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65d0151ab0 0x7f65d00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65cdd9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f65d0073c30 0x7f65d006bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41590/0 (socket says 172.21.15.7:41590) 2023-03-03T14:11:25.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65cdd9b700 1 -- 172.21.15.7:0/1852532580 learned_addr learned my addr 172.21.15.7:0/1852532580 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:11:25.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65ced9d700 1 --2- 172.21.15.7:0/1852532580 >> v2:172.21.15.143:3300/0 conn(0x7f65d006c5b0 0x7f65d0157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:25.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65ce59c700 1 -- 172.21.15.7:0/1852532580 >> v2:172.21.15.143:3300/0 conn(0x7f65d006c5b0 msgr2=0x7f65d0157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:11:25.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65ce59c700 1 --2- 172.21.15.7:0/1852532580 >> v2:172.21.15.143:3300/0 conn(0x7f65d006c5b0 0x7f65d0157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:25.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65ce59c700 1 -- 172.21.15.7:0/1852532580 >> v1:172.21.15.7:6789/0 conn(0x7f65d0073c30 legacy=0x7f65d006bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:25.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65cd59a700 1 -- 172.21.15.7:0/1852532580 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523842848 0 0) 0x7f65d006d780 con 0x7f65d0073c30 2023-03-03T14:11:25.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65ce59c700 1 -- 172.21.15.7:0/1852532580 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d006d130 con 0x7f65d0151ab0 2023-03-03T14:11:25.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.344+0000 7f65ce59c700 1 --2- 172.21.15.7:0/1852532580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 0x7f65d00736f0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f65d0074080 tx=0x7f65c0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdbd00292fd7230 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:11:25.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65cd59a700 1 -- 172.21.15.7:0/1852532580 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0011830 con 0x7f65d0151ab0 2023-03-03T14:11:25.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65cd59a700 1 -- 172.21.15.7:0/1852532580 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c0011990 con 0x7f65d0151ab0 2023-03-03T14:11:25.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65cd59a700 1 -- 172.21.15.7:0/1852532580 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0011d20 con 0x7f65d0151ab0 2023-03-03T14:11:25.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65d49d4700 1 -- 172.21.15.7:0/1852532580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 msgr2=0x7f65d00736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:25.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65d49d4700 1 --2- 172.21.15.7:0/1852532580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 0x7f65d00736f0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f65d0074080 tx=0x7f65c0009b20 comp rx=0 tx=0).stop 2023-03-03T14:11:25.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65d49d4700 1 -- 172.21.15.7:0/1852532580 shutdown_connections 2023-03-03T14:11:25.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65d49d4700 1 --2- 172.21.15.7:0/1852532580 >> v2:172.21.15.143:3300/0 conn(0x7f65d006c5b0 0x7f65d0157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:25.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65d49d4700 1 --2- 172.21.15.7:0/1852532580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 0x7f65d00736f0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:25.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65d49d4700 1 -- 172.21.15.7:0/1852532580 >> 172.21.15.7:0/1852532580 conn(0x7f65d00a94d0 msgr2=0x7f65d00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:11:25.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65d49d4700 1 -- 172.21.15.7:0/1852532580 shutdown_connections 2023-03-03T14:11:25.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.345+0000 7f65d49d4700 1 -- 172.21.15.7:0/1852532580 wait complete. 2023-03-03T14:11:25.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.347+0000 7f65d49d4700 1 Processor -- start 2023-03-03T14:11:25.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.347+0000 7f65d49d4700 1 -- start start 2023-03-03T14:11:25.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65d49d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d006c5b0 0x7f65d01ed7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:25.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65d49d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65d0073c30 0x7f65d01df290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:25.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65d49d4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 0x7f65d014f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:25.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65d49d4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d00721d0 con 0x7f65d0151ab0 2023-03-03T14:11:25.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65d49d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d01edd30 con 0x7f65d006c5b0 2023-03-03T14:11:25.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65d49d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d01ecda0 con 0x7f65d0073c30 2023-03-03T14:11:25.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65ced9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 0x7f65d014f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:25.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65ced9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 0x7f65d014f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47616/0 (socket says 172.21.15.7:47616) 2023-03-03T14:11:25.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65ced9d700 1 -- 172.21.15.7:0/1148830737 learned_addr learned my addr 172.21.15.7:0/1148830737 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:11:25.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65ce59c700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d006c5b0 0x7f65d01ed7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:25.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.348+0000 7f65cdd9b700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65d0073c30 0x7f65d01df290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:25.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65ced9d700 1 -- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65d0073c30 msgr2=0x7f65d01df290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:25.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65ced9d700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65d0073c30 0x7f65d01df290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:25.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65ced9d700 1 -- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d006c5b0 msgr2=0x7f65d01ed7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:25.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65ced9d700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d006c5b0 0x7f65d01ed7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:25.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65ced9d700 1 -- 172.21.15.7:0/1148830737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c00094a0 con 0x7f65d0151ab0 2023-03-03T14:11:25.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65ced9d700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 0x7f65d014f4d0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f65c8007e60 tx=0x7f65c800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:11:25.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65beffd700 1 -- 172.21.15.7:0/1148830737 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c800ecd0 con 0x7f65d0151ab0 2023-03-03T14:11:25.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65beffd700 1 -- 172.21.15.7:0/1148830737 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c800ee30 con 0x7f65d0151ab0 2023-03-03T14:11:25.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65d49d4700 1 -- 172.21.15.7:0/1148830737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d0151e80 con 0x7f65d0151ab0 2023-03-03T14:11:25.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65beffd700 1 -- 172.21.15.7:0/1148830737 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8014660 con 0x7f65d0151ab0 2023-03-03T14:11:25.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.349+0000 7f65d49d4700 1 -- 172.21.15.7:0/1148830737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d006d410 con 0x7f65d0151ab0 2023-03-03T14:11:25.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.350+0000 7f65d49d4700 1 -- 172.21.15.7:0/1148830737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d0152010 con 0x7f65d0151ab0 2023-03-03T14:11:25.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.350+0000 7f65beffd700 1 -- 172.21.15.7:0/1148830737 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f65c8012050 con 0x7f65d0151ab0 2023-03-03T14:11:25.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.351+0000 7f65beffd700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65b00682f0 0x7f65b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:11:25.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.351+0000 7f65beffd700 1 -- 172.21.15.7:0/1148830737 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f65c80865c0 con 0x7f65d0151ab0 2023-03-03T14:11:25.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.351+0000 7f65ce59c700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65b00682f0 0x7f65b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:11:25.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.352+0000 7f65ce59c700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65b00682f0 0x7f65b006a7a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f65c00090b0 tx=0x7f65c001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:11:25.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.355+0000 7f65beffd700 1 -- 172.21.15.7:0/1148830737 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f65c8052960 con 0x7f65d0151ab0 2023-03-03T14:11:25.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.591+0000 7f65d49d4700 1 -- 172.21.15.7:0/1148830737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65d00742d0 con 0x7f65d0151ab0 2023-03-03T14:11:25.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.591+0000 7f65beffd700 1 -- 172.21.15.7:0/1148830737 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f65c8055f80 con 0x7f65d0151ab0 2023-03-03T14:11:25.593 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:11:25.594 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:11:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.593+0000 7f65d49d4700 1 -- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65b00682f0 msgr2=0x7f65b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.593+0000 7f65d49d4700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65b00682f0 0x7f65b006a7a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f65c00090b0 tx=0x7f65c001c040 comp rx=0 tx=0).stop 2023-03-03T14:11:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.593+0000 7f65d49d4700 1 -- 172.21.15.7:0/1148830737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 msgr2=0x7f65d014f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:11:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.593+0000 7f65d49d4700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 0x7f65d014f4d0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f65c8007e60 tx=0x7f65c800e3c0 comp rx=0 tx=0).stop 2023-03-03T14:11:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.593+0000 7f65d49d4700 1 -- 172.21.15.7:0/1148830737 shutdown_connections 2023-03-03T14:11:25.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.593+0000 7f65d49d4700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65b00682f0 0x7f65b006a7a0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:25.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.593+0000 7f65d49d4700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65d0151ab0 0x7f65d014f4d0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:25.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.593+0000 7f65d49d4700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65d006c5b0 0x7f65d01ed7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:25.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.593+0000 7f65d49d4700 1 --2- 172.21.15.7:0/1148830737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65d0073c30 0x7f65d01df290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:11:25.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.593+0000 7f65d49d4700 1 -- 172.21.15.7:0/1148830737 >> 172.21.15.7:0/1148830737 conn(0x7f65d00a94d0 msgr2=0x7f65d0157dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:11:25.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.594+0000 7f65d49d4700 1 -- 172.21.15.7:0/1148830737 shutdown_connections 2023-03-03T14:11:25.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:11:25.594+0000 7f65d49d4700 1 -- 172.21.15.7:0/1148830737 wait complete. 2023-03-03T14:11:25.598 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:11:25.610 INFO:tasks.mds_thrash.fs.[b]:waiting for 47 secs before thrashing 2023-03-03T14:11:50.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:11:50.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:12.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:12:12.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.749+0000 7f002c278700 1 Processor -- start 2023-03-03T14:12:12.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.749+0000 7f002c278700 1 -- start start 2023-03-03T14:12:12.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002c278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 0x7f0024156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:12.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002c278700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00240abe20 0x7f00241529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:12.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002c278700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00240c2f30 con 0x7f002414fda0 2023-03-03T14:12:12.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002c278700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00240c31d0 con 0x7f00240abe20 2023-03-03T14:12:12.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002c278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00241524a0 con 0x7f002414f9d0 2023-03-03T14:12:12.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002a815700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00240abe20 0x7f00241529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:12.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002a815700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00240abe20 0x7f00241529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47656/0 (socket says 172.21.15.7:47656) 2023-03-03T14:12:12.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002a815700 1 -- 172.21.15.7:0/1433523330 learned_addr learned my addr 172.21.15.7:0/1433523330 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:12:12.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002a014700 1 --2- 172.21.15.7:0/1433523330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 0x7f0024156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:12.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002a815700 1 -- 172.21.15.7:0/1433523330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 msgr2=0x7f0024156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:12:12.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002a815700 1 --2- 172.21.15.7:0/1433523330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 0x7f0024156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:12.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002a815700 1 -- 172.21.15.7:0/1433523330 >> v1:172.21.15.143:6789/0 conn(0x7f002414fda0 legacy=0x7f0024071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:12.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002a815700 1 -- 172.21.15.7:0/1433523330 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00240c34b0 con 0x7f00240abe20 2023-03-03T14:12:12.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f002a815700 1 --2- 172.21.15.7:0/1433523330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 0x7f00241529c0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f001c009e50 tx=0x7f001c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff61ab5ecd2b34c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:12:12.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f0029012700 1 -- 172.21.15.7:0/1433523330 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c00e070 con 0x7f00240abe20 2023-03-03T14:12:12.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f0029012700 1 -- 172.21.15.7:0/1433523330 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001c00b9c0 con 0x7f00240abe20 2023-03-03T14:12:12.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.750+0000 7f0029012700 1 -- 172.21.15.7:0/1433523330 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c00bd70 con 0x7f00240abe20 2023-03-03T14:12:12.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.751+0000 7f002c278700 1 -- 172.21.15.7:0/1433523330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 msgr2=0x7f00241529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:12.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.751+0000 7f002c278700 1 --2- 172.21.15.7:0/1433523330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 0x7f00241529c0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f001c009e50 tx=0x7f001c00b330 comp rx=0 tx=0).stop 2023-03-03T14:12:12.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.751+0000 7f002c278700 1 -- 172.21.15.7:0/1433523330 shutdown_connections 2023-03-03T14:12:12.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.751+0000 7f002c278700 1 --2- 172.21.15.7:0/1433523330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 0x7f00241529c0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:12.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.751+0000 7f002c278700 1 --2- 172.21.15.7:0/1433523330 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 0x7f0024156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:12.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.751+0000 7f002c278700 1 -- 172.21.15.7:0/1433523330 >> 172.21.15.7:0/1433523330 conn(0x7f00240c8900 msgr2=0x7f00240c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:12:12.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.751+0000 7f002c278700 1 -- 172.21.15.7:0/1433523330 shutdown_connections 2023-03-03T14:12:12.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.751+0000 7f002c278700 1 -- 172.21.15.7:0/1433523330 wait complete. 2023-03-03T14:12:12.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.752+0000 7f002c278700 1 Processor -- start 2023-03-03T14:12:12.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.752+0000 7f002c278700 1 -- start start 2023-03-03T14:12:12.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002c278700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 0x7f00240c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:12.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002c278700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 0x7f00241dc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:12.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002c278700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002414fda0 0x7f00241dcb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:12.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002c278700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002406a820 con 0x7f00240abe20 2023-03-03T14:12:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002c278700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002406b550 con 0x7f002414fda0 2023-03-03T14:12:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002c278700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00241e98a0 con 0x7f002414f9d0 2023-03-03T14:12:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002a014700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 0x7f00240c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002a014700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 0x7f00240c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47658/0 (socket says 172.21.15.7:47658) 2023-03-03T14:12:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002a014700 1 -- 172.21.15.7:0/2373216418 learned_addr learned my addr 172.21.15.7:0/2373216418 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:12:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002a815700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002414fda0 0x7f00241dcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f0029813700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 0x7f00241dc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:12.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002a014700 1 -- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 msgr2=0x7f00241dc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002a014700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 0x7f00241dc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002a014700 1 -- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002414fda0 msgr2=0x7f00241dcb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002a014700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002414fda0 0x7f00241dcb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002a014700 1 -- 172.21.15.7:0/2373216418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c009b30 con 0x7f00240abe20 2023-03-03T14:12:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f002a815700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002414fda0 0x7f00241dcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:12:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.753+0000 7f0029813700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 0x7f00241dc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:12:12.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.754+0000 7f002a014700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 0x7f00240c1a50 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f002000e810 tx=0x7f002000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:12:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.754+0000 7f001affd700 1 -- 172.21.15.7:0/2373216418 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020013610 con 0x7f00240abe20 2023-03-03T14:12:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.754+0000 7f001affd700 1 -- 172.21.15.7:0/2373216418 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0020013770 con 0x7f00240abe20 2023-03-03T14:12:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.754+0000 7f002c278700 1 -- 172.21.15.7:0/2373216418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00240ac350 con 0x7f00240abe20 2023-03-03T14:12:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.754+0000 7f001affd700 1 -- 172.21.15.7:0/2373216418 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0020013b20 con 0x7f00240abe20 2023-03-03T14:12:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.754+0000 7f002c278700 1 -- 172.21.15.7:0/2373216418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00240ab630 con 0x7f00240abe20 2023-03-03T14:12:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.755+0000 7f002c278700 1 -- 172.21.15.7:0/2373216418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0024071fa0 con 0x7f00240abe20 2023-03-03T14:12:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.755+0000 7f001affd700 1 -- 172.21.15.7:0/2373216418 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0020023070 con 0x7f00240abe20 2023-03-03T14:12:12.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.756+0000 7f001affd700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 0x7f000806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:12.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.756+0000 7f001affd700 1 -- 172.21.15.7:0/2373216418 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0020085e60 con 0x7f00240abe20 2023-03-03T14:12:12.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.756+0000 7f0029813700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 0x7f000806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:12.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.756+0000 7f0029813700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 0x7f000806a7a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f00240c37f0 tx=0x7f0014009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:12:12.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.760+0000 7f001affd700 1 -- 172.21.15.7:0/2373216418 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0020052180 con 0x7f00240abe20 2023-03-03T14:12:13.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.999+0000 7f002c278700 1 -- 172.21.15.7:0/2373216418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00240ac4e0 con 0x7f00240abe20 2023-03-03T14:12:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:12.999+0000 7f001affd700 1 -- 172.21.15.7:0/2373216418 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f00200557a0 con 0x7f00240abe20 2023-03-03T14:12:13.002 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:12:13.002 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:12:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 -- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 msgr2=0x7f000806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 0x7f000806a7a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f00240c37f0 tx=0x7f0014009040 comp rx=0 tx=0).stop 2023-03-03T14:12:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 -- 172.21.15.7:0/2373216418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 msgr2=0x7f00240c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 0x7f00240c1a50 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f002000e810 tx=0x7f002000ebd0 comp rx=0 tx=0).stop 2023-03-03T14:12:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 -- 172.21.15.7:0/2373216418 shutdown_connections 2023-03-03T14:12:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f002414fda0 0x7f00241dcb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00080682f0 0x7f000806a7a0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00240abe20 0x7f00240c1a50 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 --2- 172.21.15.7:0/2373216418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f002414f9d0 0x7f00241dc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 -- 172.21.15.7:0/2373216418 >> 172.21.15.7:0/2373216418 conn(0x7f00240c8900 msgr2=0x7f0024151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:12:13.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 -- 172.21.15.7:0/2373216418 shutdown_connections 2023-03-03T14:12:13.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:13.001+0000 7f002c278700 1 -- 172.21.15.7:0/2373216418 wait complete. 2023-03-03T14:12:13.006 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:12:13.026 INFO:tasks.mds_thrash.fs.[b]:waiting for 81 secs before thrashing 2023-03-03T14:12:20.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:20.919 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:26.751 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:12:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.891+0000 7f1639bb7700 1 Processor -- start 2023-03-03T14:12:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.891+0000 7f1639bb7700 1 -- start start 2023-03-03T14:12:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1639bb7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163414f9d0 0x7f1634156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:26.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1639bb7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16340abe20 0x7f16341529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:26.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1639bb7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16340c2f30 con 0x7f163414fda0 2023-03-03T14:12:26.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1639bb7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16340c31d0 con 0x7f16340abe20 2023-03-03T14:12:26.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1639bb7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16341524a0 con 0x7f163414f9d0 2023-03-03T14:12:26.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1633fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16340abe20 0x7f16341529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:26.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1632ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f163414fda0 0x7f1634071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41668/0 (socket says 172.21.15.7:41668) 2023-03-03T14:12:26.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1632ffd700 1 -- 172.21.15.7:0/3429083520 learned_addr learned my addr 172.21.15.7:0/3429083520 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:12:26.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f16337fe700 1 --2- 172.21.15.7:0/3429083520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163414f9d0 0x7f1634156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:26.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1633fff700 1 -- 172.21.15.7:0/3429083520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163414f9d0 msgr2=0x7f1634156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:12:26.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1633fff700 1 --2- 172.21.15.7:0/3429083520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163414f9d0 0x7f1634156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:26.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1633fff700 1 -- 172.21.15.7:0/3429083520 >> v1:172.21.15.7:6789/0 conn(0x7f163414fda0 legacy=0x7f1634071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:26.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.892+0000 7f1633fff700 1 -- 172.21.15.7:0/3429083520 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16340c34b0 con 0x7f16340abe20 2023-03-03T14:12:26.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f1633fff700 1 --2- 172.21.15.7:0/3429083520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16340abe20 0x7f16341529c0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f162c009d90 tx=0x7f162c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef161c1d6ccd90f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:12:26.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f16327fc700 1 -- 172.21.15.7:0/3429083520 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c00e070 con 0x7f16340abe20 2023-03-03T14:12:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f16327fc700 1 -- 172.21.15.7:0/3429083520 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c00bbd0 con 0x7f16340abe20 2023-03-03T14:12:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f16327fc700 1 -- 172.21.15.7:0/3429083520 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c012430 con 0x7f16340abe20 2023-03-03T14:12:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f1639bb7700 1 -- 172.21.15.7:0/3429083520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16340abe20 msgr2=0x7f16341529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f1639bb7700 1 --2- 172.21.15.7:0/3429083520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16340abe20 0x7f16341529c0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f162c009d90 tx=0x7f162c00b330 comp rx=0 tx=0).stop 2023-03-03T14:12:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f1639bb7700 1 -- 172.21.15.7:0/3429083520 shutdown_connections 2023-03-03T14:12:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f1639bb7700 1 --2- 172.21.15.7:0/3429083520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16340abe20 0x7f16341529c0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f1639bb7700 1 --2- 172.21.15.7:0/3429083520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f163414f9d0 0x7f1634156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f1639bb7700 1 -- 172.21.15.7:0/3429083520 >> 172.21.15.7:0/3429083520 conn(0x7f16340c8900 msgr2=0x7f16340c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:12:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f1639bb7700 1 -- 172.21.15.7:0/3429083520 shutdown_connections 2023-03-03T14:12:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.893+0000 7f1639bb7700 1 -- 172.21.15.7:0/3429083520 wait complete. 2023-03-03T14:12:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.895+0000 7f1639bb7700 1 Processor -- start 2023-03-03T14:12:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.895+0000 7f1639bb7700 1 -- start start 2023-03-03T14:12:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1639bb7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16340abe20 0x7f163406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1639bb7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f163414f9d0 0x7f16340aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:26.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1639bb7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f163414fda0 0x7f1634143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:26.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1639bb7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163406b1d0 con 0x7f163414fda0 2023-03-03T14:12:26.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1639bb7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163406c000 con 0x7f163414f9d0 2023-03-03T14:12:26.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1639bb7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163406b480 con 0x7f16340abe20 2023-03-03T14:12:26.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1633fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f163414fda0 0x7f1634143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:26.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1633fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f163414fda0 0x7f1634143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47696/0 (socket says 172.21.15.7:47696) 2023-03-03T14:12:26.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1633fff700 1 -- 172.21.15.7:0/3021993638 learned_addr learned my addr 172.21.15.7:0/3021993638 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:12:26.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f16337fe700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16340abe20 0x7f163406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:26.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1632ffd700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f163414f9d0 0x7f16340aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:26.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1633fff700 1 -- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16340abe20 msgr2=0x7f163406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:26.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1633fff700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16340abe20 0x7f163406ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:26.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1633fff700 1 -- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f163414f9d0 msgr2=0x7f16340aa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:26.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1633fff700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f163414f9d0 0x7f16340aa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:26.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1633fff700 1 -- 172.21.15.7:0/3021993638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c009a70 con 0x7f163414fda0 2023-03-03T14:12:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1633fff700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f163414fda0 0x7f1634143910 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f162c014040 tx=0x7f162c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:12:26.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f162bfff700 1 -- 172.21.15.7:0/3021993638 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c012430 con 0x7f163414fda0 2023-03-03T14:12:26.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f162bfff700 1 -- 172.21.15.7:0/3021993638 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c020070 con 0x7f163414fda0 2023-03-03T14:12:26.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f162bfff700 1 -- 172.21.15.7:0/3021993638 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c004280 con 0x7f163414fda0 2023-03-03T14:12:26.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.896+0000 7f1639bb7700 1 -- 172.21.15.7:0/3021993638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16340ac350 con 0x7f163414fda0 2023-03-03T14:12:26.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.897+0000 7f1639bb7700 1 -- 172.21.15.7:0/3021993638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16340ab630 con 0x7f163414fda0 2023-03-03T14:12:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.897+0000 7f1639bb7700 1 -- 172.21.15.7:0/3021993638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1634071fa0 con 0x7f163414fda0 2023-03-03T14:12:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.898+0000 7f162bfff700 1 -- 172.21.15.7:0/3021993638 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f162c02e070 con 0x7f163414fda0 2023-03-03T14:12:26.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.898+0000 7f162bfff700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16140682f0 0x7f161406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:12:26.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.898+0000 7f162bfff700 1 -- 172.21.15.7:0/3021993638 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f162c022070 con 0x7f163414fda0 2023-03-03T14:12:26.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.898+0000 7f16337fe700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16140682f0 0x7f161406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:12:26.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.899+0000 7f16337fe700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16140682f0 0x7f161406a7a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f16340abd40 tx=0x7f1624008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:12:26.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:26.903+0000 7f162bfff700 1 -- 172.21.15.7:0/3021993638 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f162c054210 con 0x7f163414fda0 2023-03-03T14:12:27.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.143+0000 7f1639bb7700 1 -- 172.21.15.7:0/3021993638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1634074bb0 con 0x7f163414fda0 2023-03-03T14:12:27.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.144+0000 7f162bfff700 1 -- 172.21.15.7:0/3021993638 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f162c057830 con 0x7f163414fda0 2023-03-03T14:12:27.151 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:12:27.151 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:12:27.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 -- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16140682f0 msgr2=0x7f161406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:27.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16140682f0 0x7f161406a7a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f16340abd40 tx=0x7f1624008040 comp rx=0 tx=0).stop 2023-03-03T14:12:27.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 -- 172.21.15.7:0/3021993638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f163414fda0 msgr2=0x7f1634143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:12:27.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f163414fda0 0x7f1634143910 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f162c014040 tx=0x7f162c004ad0 comp rx=0 tx=0).stop 2023-03-03T14:12:27.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 -- 172.21.15.7:0/3021993638 shutdown_connections 2023-03-03T14:12:27.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16140682f0 0x7f161406a7a0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:27.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f163414fda0 0x7f1634143910 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:27.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16340abe20 0x7f163406ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:27.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 --2- 172.21.15.7:0/3021993638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f163414f9d0 0x7f16340aa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:12:27.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 -- 172.21.15.7:0/3021993638 >> 172.21.15.7:0/3021993638 conn(0x7f16340c8900 msgr2=0x7f1634151a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:12:27.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 -- 172.21.15.7:0/3021993638 shutdown_connections 2023-03-03T14:12:27.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:12:27.146+0000 7f1639bb7700 1 -- 172.21.15.7:0/3021993638 wait complete. 2023-03-03T14:12:27.155 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:12:27.161 INFO:tasks.mds_thrash.fs.[a]:waiting for 105 secs before thrashing 2023-03-03T14:12:51.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:12:51.062 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:21.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:21.130 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:34.027 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:13:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.165+0000 7f32dff5d700 1 Processor -- start 2023-03-03T14:13:34.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.165+0000 7f32dff5d700 1 -- start start 2023-03-03T14:13:34.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.166+0000 7f32dff5d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f32d8151e80 0x7f32d80c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:34.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.166+0000 7f32dff5d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d80c08d0 con 0x7f32d80c6fe0 2023-03-03T14:13:34.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.166+0000 7f32dff5d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d8154930 con 0x7f32d8155410 2023-03-03T14:13:34.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.166+0000 7f32dff5d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d8152250 con 0x7f32d8151e80 2023-03-03T14:13:34.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.166+0000 7f32de4fa700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f32d80c6fe0 0x7f32d80bfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41722/0 (socket says 172.21.15.7:41722) 2023-03-03T14:13:34.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.166+0000 7f32de4fa700 1 -- 172.21.15.7:0/344791778 learned_addr learned my addr 172.21.15.7:0/344791778 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:13:34.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.166+0000 7f32ddcf9700 1 --2- 172.21.15.7:0/344791778 >> v2:172.21.15.143:3300/0 conn(0x7f32d8151e80 0x7f32d80c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:34.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606992964 0 0) 0x7f32d80c08d0 con 0x7f32d80c6fe0 2023-03-03T14:13:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c40035b0 con 0x7f32d80c6fe0 2023-03-03T14:13:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442569430 0 0) 0x7f32d8154930 con 0x7f32d8155410 2023-03-03T14:13:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d80c08d0 con 0x7f32d8155410 2023-03-03T14:13:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3655285335 0 0) 0x7f32c40035b0 con 0x7f32d80c6fe0 2023-03-03T14:13:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 >> v2:172.21.15.143:3300/0 conn(0x7f32d8151e80 msgr2=0x7f32d80c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:13:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 --2- 172.21.15.7:0/344791778 >> v2:172.21.15.143:3300/0 conn(0x7f32d8151e80 0x7f32d80c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 >> v1:172.21.15.143:6789/0 conn(0x7f32d8155410 legacy=0x7f32d80bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d8154c10 con 0x7f32d80c6fe0 2023-03-03T14:13:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f32d00019f0 con 0x7f32d80c6fe0 2023-03-03T14:13:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 >> v1:172.21.15.7:6789/0 conn(0x7f32d80c6fe0 legacy=0x7f32d80bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32c4003c50 0x7f32c4004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4004850 0x7f32c4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c4005200 0x7f32c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c400c0e0 con 0x7f32c4005200 2023-03-03T14:13:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c400c250 con 0x7f32c4004850 2023-03-03T14:13:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c400c410 con 0x7f32c4003c50 2023-03-03T14:13:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32ddcf9700 1 --2- 172.21.15.7:0/344791778 >> v2:172.21.15.143:3300/0 conn(0x7f32d8151e80 0x7f32d80c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:13:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32de4fa700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c4005200 0x7f32c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dd4f8700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4004850 0x7f32c4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32ddcf9700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32c4003c50 0x7f32c4004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32de4fa700 1 -- 172.21.15.7:0/344791778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32c4003c50 msgr2=0x7f32c4004040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:13:34.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32de4fa700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32c4003c50 0x7f32c4004040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32de4fa700 1 -- 172.21.15.7:0/344791778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4004850 msgr2=0x7f32c4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:13:34.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32de4fa700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4004850 0x7f32c4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32de4fa700 1 -- 172.21.15.7:0/344791778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c40035b0 con 0x7f32c4005200 2023-03-03T14:13:34.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.167+0000 7f32dd4f8700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4004850 0x7f32c4004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:13:34.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.168+0000 7f32de4fa700 1 -- 172.21.15.7:0/344791778 reap_dead start 2023-03-03T14:13:34.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.168+0000 7f32de4fa700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c4005200 0x7f32c400ba10 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f32d000a710 tx=0x7f32c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24baaeb9255fd573 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:13:34.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.168+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00072d0 con 0x7f32c4005200 2023-03-03T14:13:34.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.168+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d0007430 con 0x7f32c4005200 2023-03-03T14:13:34.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.168+0000 7f32dccf7700 1 -- 172.21.15.7:0/344791778 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00077e0 con 0x7f32c4005200 2023-03-03T14:13:34.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.168+0000 7f32dff5d700 1 -- 172.21.15.7:0/344791778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c4005200 msgr2=0x7f32c400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:34.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.168+0000 7f32dff5d700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c4005200 0x7f32c400ba10 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f32d000a710 tx=0x7f32c4005880 comp rx=0 tx=0).stop 2023-03-03T14:13:34.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.168+0000 7f32dff5d700 1 -- 172.21.15.7:0/344791778 shutdown_connections 2023-03-03T14:13:34.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.168+0000 7f32dff5d700 1 --2- 172.21.15.7:0/344791778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c4005200 0x7f32c400ba10 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.168+0000 7f32dff5d700 1 -- 172.21.15.7:0/344791778 >> 172.21.15.7:0/344791778 conn(0x7f32d8071630 msgr2=0x7f32d806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:13:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.169+0000 7f32dff5d700 1 -- 172.21.15.7:0/344791778 shutdown_connections 2023-03-03T14:13:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.169+0000 7f32dff5d700 1 -- 172.21.15.7:0/344791778 wait complete. 2023-03-03T14:13:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.170+0000 7f32dff5d700 1 Processor -- start 2023-03-03T14:13:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.170+0000 7f32dff5d700 1 -- start start 2023-03-03T14:13:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.170+0000 7f32dff5d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4005200 0x7f32d81501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.170+0000 7f32dff5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d81ee0c0 0x7f32d81485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32dff5d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32d81ddd20 0x7f32d8148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32dff5d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8155d50 con 0x7f32d81ddd20 2023-03-03T14:13:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32dff5d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d81ee580 con 0x7f32c4005200 2023-03-03T14:13:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32dff5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8157050 con 0x7f32d81ee0c0 2023-03-03T14:13:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32de4fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32d81ddd20 0x7f32d8148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32de4fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32d81ddd20 0x7f32d8148ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47754/0 (socket says 172.21.15.7:47754) 2023-03-03T14:13:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32de4fa700 1 -- 172.21.15.7:0/3614256184 learned_addr learned my addr 172.21.15.7:0/3614256184 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:13:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32dd4f8700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d81ee0c0 0x7f32d81485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32ddcf9700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4005200 0x7f32d81501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32de4fa700 1 -- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d81ee0c0 msgr2=0x7f32d81485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32de4fa700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d81ee0c0 0x7f32d81485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32de4fa700 1 -- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4005200 msgr2=0x7f32d81501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32de4fa700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4005200 0x7f32d81501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32de4fa700 1 -- 172.21.15.7:0/3614256184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d80c5220 con 0x7f32d81ddd20 2023-03-03T14:13:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32ddcf9700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4005200 0x7f32d81501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:13:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32de4fa700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32d81ddd20 0x7f32d8148ae0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f32d0002230 tx=0x7f32d0001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:13:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32ce7fc700 1 -- 172.21.15.7:0/3614256184 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00079b0 con 0x7f32d81ddd20 2023-03-03T14:13:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32ce7fc700 1 -- 172.21.15.7:0/3614256184 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d0007b10 con 0x7f32d81ddd20 2023-03-03T14:13:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32dff5d700 1 -- 172.21.15.7:0/3614256184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d8154ef0 con 0x7f32d81ddd20 2023-03-03T14:13:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32ce7fc700 1 -- 172.21.15.7:0/3614256184 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d001f670 con 0x7f32d81ddd20 2023-03-03T14:13:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.171+0000 7f32dff5d700 1 -- 172.21.15.7:0/3614256184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d80c5b00 con 0x7f32d81ddd20 2023-03-03T14:13:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.172+0000 7f32dff5d700 1 -- 172.21.15.7:0/3614256184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d81573a0 con 0x7f32d81ddd20 2023-03-03T14:13:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.173+0000 7f32ce7fc700 1 -- 172.21.15.7:0/3614256184 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f32d0010070 con 0x7f32d81ddd20 2023-03-03T14:13:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.173+0000 7f32ce7fc700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32bc068320 0x7f32bc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.173+0000 7f32ce7fc700 1 -- 172.21.15.7:0/3614256184 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f32d0087a30 con 0x7f32d81ddd20 2023-03-03T14:13:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.173+0000 7f32ddcf9700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32bc068320 0x7f32bc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.174+0000 7f32ddcf9700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32bc068320 0x7f32bc06a7d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009d70 tx=0x7f32c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:13:34.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.178+0000 7f32ce7fc700 1 -- 172.21.15.7:0/3614256184 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f32d0053dd0 con 0x7f32d81ddd20 2023-03-03T14:13:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.419+0000 7f32dff5d700 1 -- 172.21.15.7:0/3614256184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32d8156eb0 con 0x7f32d81ddd20 2023-03-03T14:13:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.419+0000 7f32ce7fc700 1 -- 172.21.15.7:0/3614256184 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f32d8156eb0 con 0x7f32d81ddd20 2023-03-03T14:13:34.421 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:13:34.421 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:13:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.421+0000 7f32dff5d700 1 -- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32bc068320 msgr2=0x7f32bc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.421+0000 7f32dff5d700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32bc068320 0x7f32bc06a7d0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009d70 tx=0x7f32c8008040 comp rx=0 tx=0).stop 2023-03-03T14:13:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.421+0000 7f32dff5d700 1 -- 172.21.15.7:0/3614256184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32d81ddd20 msgr2=0x7f32d8148ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.421+0000 7f32dff5d700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32d81ddd20 0x7f32d8148ae0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f32d0002230 tx=0x7f32d0001cb0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.421+0000 7f32dff5d700 1 -- 172.21.15.7:0/3614256184 shutdown_connections 2023-03-03T14:13:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.421+0000 7f32dff5d700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32bc068320 0x7f32bc06a7d0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.421+0000 7f32dff5d700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32d81ddd20 0x7f32d8148ae0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.421+0000 7f32dff5d700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c4005200 0x7f32d81501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.421+0000 7f32dff5d700 1 --2- 172.21.15.7:0/3614256184 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32d81ee0c0 0x7f32d81485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.421+0000 7f32dff5d700 1 -- 172.21.15.7:0/3614256184 >> 172.21.15.7:0/3614256184 conn(0x7f32d8071630 msgr2=0x7f32d80c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:13:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.422+0000 7f32dff5d700 1 -- 172.21.15.7:0/3614256184 shutdown_connections 2023-03-03T14:13:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:34.422+0000 7f32dff5d700 1 -- 172.21.15.7:0/3614256184 wait complete. 2023-03-03T14:13:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:13:34.435 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before thrashing 2023-03-03T14:13:35.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:13:35.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.574+0000 7febd26a9700 1 Processor -- start 2023-03-03T14:13:35.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.574+0000 7febd26a9700 1 -- start start 2023-03-03T14:13:35.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.574+0000 7febd26a9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7febcc151e80 0x7febcc0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:35.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.574+0000 7febd26a9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7febcc0c6fe0 0x7febcc0bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:35.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.574+0000 7febd26a9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febcc0c08d0 con 0x7febcc155410 2023-03-03T14:13:35.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.574+0000 7febd26a9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febcc152250 con 0x7febcc0c6fe0 2023-03-03T14:13:35.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.574+0000 7febd26a9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febcc0c0fb0 con 0x7febcc151e80 2023-03-03T14:13:35.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febd0c46700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7febcc0c6fe0 0x7febcc0bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:35.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febd0c46700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7febcc0c6fe0 0x7febcc0bfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47762/0 (socket says 172.21.15.7:47762) 2023-03-03T14:13:35.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febd0c46700 1 -- 172.21.15.7:0/96736899 learned_addr learned my addr 172.21.15.7:0/96736899 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:13:35.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febcbfff700 1 --2- 172.21.15.7:0/96736899 >> v2:172.21.15.143:3300/0 conn(0x7febcc151e80 0x7febcc0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:35.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febd0c46700 1 -- 172.21.15.7:0/96736899 >> v2:172.21.15.143:3300/0 conn(0x7febcc151e80 msgr2=0x7febcc0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:13:35.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febd0c46700 1 --2- 172.21.15.7:0/96736899 >> v2:172.21.15.143:3300/0 conn(0x7febcc151e80 0x7febcc0c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febd0c46700 1 -- 172.21.15.7:0/96736899 >> v1:172.21.15.143:6789/0 conn(0x7febcc155410 legacy=0x7febcc0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febd0c46700 1 -- 172.21.15.7:0/96736899 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febcc154930 con 0x7febcc0c6fe0 2023-03-03T14:13:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febd0c46700 1 --2- 172.21.15.7:0/96736899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 0x7febcc0bfff0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7febcc0cb250 tx=0x7febc400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee7bf7be5fc2bf9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:13:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febcaffd700 1 -- 172.21.15.7:0/96736899 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc400e070 con 0x7febcc0c6fe0 2023-03-03T14:13:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febcaffd700 1 -- 172.21.15.7:0/96736899 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc400b910 con 0x7febcc0c6fe0 2023-03-03T14:13:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.575+0000 7febcaffd700 1 -- 172.21.15.7:0/96736899 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc400bcc0 con 0x7febcc0c6fe0 2023-03-03T14:13:35.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.576+0000 7febd26a9700 1 -- 172.21.15.7:0/96736899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 msgr2=0x7febcc0bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:35.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.576+0000 7febd26a9700 1 --2- 172.21.15.7:0/96736899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 0x7febcc0bfff0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7febcc0cb250 tx=0x7febc400b330 comp rx=0 tx=0).stop 2023-03-03T14:13:35.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.576+0000 7febd26a9700 1 -- 172.21.15.7:0/96736899 shutdown_connections 2023-03-03T14:13:35.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.576+0000 7febd26a9700 1 --2- 172.21.15.7:0/96736899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 0x7febcc0bfff0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:35.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.576+0000 7febd26a9700 1 --2- 172.21.15.7:0/96736899 >> v2:172.21.15.143:3300/0 conn(0x7febcc151e80 0x7febcc0c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:35.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.576+0000 7febd26a9700 1 -- 172.21.15.7:0/96736899 >> 172.21.15.7:0/96736899 conn(0x7febcc071630 msgr2=0x7febcc06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:13:35.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.576+0000 7febd26a9700 1 -- 172.21.15.7:0/96736899 shutdown_connections 2023-03-03T14:13:35.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.576+0000 7febd26a9700 1 -- 172.21.15.7:0/96736899 wait complete. 2023-03-03T14:13:35.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.577+0000 7febd26a9700 1 Processor -- start 2023-03-03T14:13:35.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.578+0000 7febd26a9700 1 -- start start 2023-03-03T14:13:35.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febd26a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 0x7febcc0bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:35.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febd26a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febcc151e80 0x7febcc1ea450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:35.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febd26a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febcc155410 0x7febcc1ea990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:35.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febd26a9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febcc158c30 con 0x7febcc0c6fe0 2023-03-03T14:13:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febd26a9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febcc158700 con 0x7febcc155410 2023-03-03T14:13:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febd26a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febcc1528b0 con 0x7febcc151e80 2023-03-03T14:13:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 0x7febcc0bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 0x7febcc0bf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47766/0 (socket says 172.21.15.7:47766) 2023-03-03T14:13:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcbfff700 1 -- 172.21.15.7:0/4255105858 learned_addr learned my addr 172.21.15.7:0/4255105858 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:13:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febd0c46700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febcc155410 0x7febcc1ea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcb7fe700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febcc151e80 0x7febcc1ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:35.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcbfff700 1 -- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febcc151e80 msgr2=0x7febcc1ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcbfff700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febcc151e80 0x7febcc1ea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcbfff700 1 -- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febcc155410 msgr2=0x7febcc1ea990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcbfff700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febcc155410 0x7febcc1ea990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcbfff700 1 -- 172.21.15.7:0/4255105858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc4009b30 con 0x7febcc0c6fe0 2023-03-03T14:13:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcb7fe700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febcc151e80 0x7febcc1ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:13:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febcbfff700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 0x7febcc0bf940 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7febbc00e9e0 tx=0x7febbc00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:13:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febc8ff9700 1 -- 172.21.15.7:0/4255105858 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc009710 con 0x7febcc0c6fe0 2023-03-03T14:13:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febc8ff9700 1 -- 172.21.15.7:0/4255105858 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc00c7f0 con 0x7febcc0c6fe0 2023-03-03T14:13:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febd26a9700 1 -- 172.21.15.7:0/4255105858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febcc154ef0 con 0x7febcc0c6fe0 2023-03-03T14:13:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.579+0000 7febc8ff9700 1 -- 172.21.15.7:0/4255105858 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc00cb60 con 0x7febcc0c6fe0 2023-03-03T14:13:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.580+0000 7febd26a9700 1 -- 172.21.15.7:0/4255105858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febcc154c10 con 0x7febcc0c6fe0 2023-03-03T14:13:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.581+0000 7febd26a9700 1 -- 172.21.15.7:0/4255105858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febcc0c5220 con 0x7febcc0c6fe0 2023-03-03T14:13:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.581+0000 7febc8ff9700 1 -- 172.21.15.7:0/4255105858 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7febbc019030 con 0x7febcc0c6fe0 2023-03-03T14:13:35.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.581+0000 7febc8ff9700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febb8068320 0x7febb806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:13:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.581+0000 7febc8ff9700 1 -- 172.21.15.7:0/4255105858 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7febbc086070 con 0x7febcc0c6fe0 2023-03-03T14:13:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.581+0000 7febcb7fe700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febb8068320 0x7febb806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:13:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.582+0000 7febcb7fe700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febb8068320 0x7febb806a7d0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7febcc14c850 tx=0x7febc0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:13:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.586+0000 7febc8ff9700 1 -- 172.21.15.7:0/4255105858 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7febbc052410 con 0x7febcc0c6fe0 2023-03-03T14:13:35.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.823+0000 7febd26a9700 1 -- 172.21.15.7:0/4255105858 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febcc074bb0 con 0x7febcc0c6fe0 2023-03-03T14:13:35.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.823+0000 7febc8ff9700 1 -- 172.21.15.7:0/4255105858 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7febbc017090 con 0x7febcc0c6fe0 2023-03-03T14:13:35.825 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:13:35.826 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:13:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 -- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febb8068320 msgr2=0x7febb806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febb8068320 0x7febb806a7d0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7febcc14c850 tx=0x7febc0009040 comp rx=0 tx=0).stop 2023-03-03T14:13:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 -- 172.21.15.7:0/4255105858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 msgr2=0x7febcc0bf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:13:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 0x7febcc0bf940 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7febbc00e9e0 tx=0x7febbc00eda0 comp rx=0 tx=0).stop 2023-03-03T14:13:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 -- 172.21.15.7:0/4255105858 shutdown_connections 2023-03-03T14:13:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febcc155410 0x7febcc1ea990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7febb8068320 0x7febb806a7d0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febcc0c6fe0 0x7febcc0bf940 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 --2- 172.21.15.7:0/4255105858 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7febcc151e80 0x7febcc1ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:13:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 -- 172.21.15.7:0/4255105858 >> 172.21.15.7:0/4255105858 conn(0x7febcc071630 msgr2=0x7febcc0c3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:13:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 -- 172.21.15.7:0/4255105858 shutdown_connections 2023-03-03T14:13:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:13:35.826+0000 7febd26a9700 1 -- 172.21.15.7:0/4255105858 wait complete. 2023-03-03T14:13:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:13:35.839 INFO:tasks.mds_thrash.fs.[b]:waiting for 81 secs before thrashing 2023-03-03T14:13:51.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:13:51.216 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:12.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:14:12.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.303+0000 7f190e60c700 1 Processor -- start 2023-03-03T14:14:12.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.303+0000 7f190e60c700 1 -- start start 2023-03-03T14:14:12.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.303+0000 7f190e60c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1908072250 0x7f190814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:12.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.303+0000 7f190e60c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19080c1ce0 0x7f190814e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:12.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.303+0000 7f190e60c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19081510d0 con 0x7f1908071e80 2023-03-03T14:14:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.303+0000 7f190e60c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19080c3740 con 0x7f19080c1ce0 2023-03-03T14:14:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.303+0000 7f190e60c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1908070670 con 0x7f1908072250 2023-03-03T14:14:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f190cba9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19080c1ce0 0x7f190814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f1907fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1908071e80 0x7f1908151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41794/0 (socket says 172.21.15.7:41794) 2023-03-03T14:14:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f1907fff700 1 -- 172.21.15.7:0/3936047222 learned_addr learned my addr 172.21.15.7:0/3936047222 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:14:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f19077fe700 1 --2- 172.21.15.7:0/3936047222 >> v2:172.21.15.143:3300/0 conn(0x7f1908072250 0x7f190814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:12.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f190cba9700 1 -- 172.21.15.7:0/3936047222 >> v2:172.21.15.143:3300/0 conn(0x7f1908072250 msgr2=0x7f190814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:14:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f190cba9700 1 --2- 172.21.15.7:0/3936047222 >> v2:172.21.15.143:3300/0 conn(0x7f1908072250 0x7f190814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f190cba9700 1 -- 172.21.15.7:0/3936047222 >> v1:172.21.15.7:6789/0 conn(0x7f1908071e80 legacy=0x7f1908151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f1906ffd700 1 -- 172.21.15.7:0/3936047222 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699426184 0 0) 0x7f19081510d0 con 0x7f1908071e80 2023-03-03T14:14:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f190cba9700 1 -- 172.21.15.7:0/3936047222 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19080c39e0 con 0x7f19080c1ce0 2023-03-03T14:14:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f19077fe700 1 --2- 172.21.15.7:0/3936047222 >> v2:172.21.15.143:3300/0 conn(0x7f1908072250 0x7f190814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:14:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f190cba9700 1 --2- 172.21.15.7:0/3936047222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19080c1ce0 0x7f190814e730 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f190806b520 tx=0x7f190000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541fcab915eea15d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:14:12.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f1906ffd700 1 -- 172.21.15.7:0/3936047222 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190000e070 con 0x7f19080c1ce0 2023-03-03T14:14:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f1906ffd700 1 -- 172.21.15.7:0/3936047222 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190000bbd0 con 0x7f19080c1ce0 2023-03-03T14:14:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.304+0000 7f1906ffd700 1 -- 172.21.15.7:0/3936047222 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1900012430 con 0x7f19080c1ce0 2023-03-03T14:14:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.305+0000 7f190e60c700 1 -- 172.21.15.7:0/3936047222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19080c1ce0 msgr2=0x7f190814e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.305+0000 7f190e60c700 1 --2- 172.21.15.7:0/3936047222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19080c1ce0 0x7f190814e730 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f190806b520 tx=0x7f190000b330 comp rx=0 tx=0).stop 2023-03-03T14:14:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.305+0000 7f190e60c700 1 -- 172.21.15.7:0/3936047222 shutdown_connections 2023-03-03T14:14:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.305+0000 7f190e60c700 1 --2- 172.21.15.7:0/3936047222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19080c1ce0 0x7f190814e730 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.305+0000 7f190e60c700 1 --2- 172.21.15.7:0/3936047222 >> v2:172.21.15.143:3300/0 conn(0x7f1908072250 0x7f190814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:12.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.305+0000 7f190e60c700 1 -- 172.21.15.7:0/3936047222 >> 172.21.15.7:0/3936047222 conn(0x7f19080a94d0 msgr2=0x7f19080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:14:12.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.305+0000 7f190e60c700 1 -- 172.21.15.7:0/3936047222 shutdown_connections 2023-03-03T14:14:12.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.305+0000 7f190e60c700 1 -- 172.21.15.7:0/3936047222 wait complete. 2023-03-03T14:14:12.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.307+0000 7f190e60c700 1 Processor -- start 2023-03-03T14:14:12.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.307+0000 7f190e60c700 1 -- start start 2023-03-03T14:14:12.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f190e60c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1908071e80 0x7f19081eb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:12.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f190e60c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1908072250 0x7f1908149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:12.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f190e60c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19080c1ce0 0x7f19080c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:12.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f190e60c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19081eab20 con 0x7f1908071e80 2023-03-03T14:14:12.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f190e60c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19080aad10 con 0x7f1908072250 2023-03-03T14:14:12.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f190e60c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19080aaff0 con 0x7f19080c1ce0 2023-03-03T14:14:12.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1907fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1908071e80 0x7f19081eb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:12.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1907fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1908071e80 0x7f19081eb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47820/0 (socket says 172.21.15.7:47820) 2023-03-03T14:14:12.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1907fff700 1 -- 172.21.15.7:0/139453777 learned_addr learned my addr 172.21.15.7:0/139453777 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:14:12.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f19077fe700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1908072250 0x7f1908149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:12.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1907fff700 1 -- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19080c1ce0 msgr2=0x7f19080c56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:12.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f190cba9700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19080c1ce0 0x7f19080c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:12.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1907fff700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19080c1ce0 0x7f19080c56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:12.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1907fff700 1 -- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1908072250 msgr2=0x7f1908149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:12.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1907fff700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1908072250 0x7f1908149bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:12.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1907fff700 1 -- 172.21.15.7:0/139453777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1900009a70 con 0x7f1908071e80 2023-03-03T14:14:12.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1907fff700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1908071e80 0x7f19081eb570 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f18f80011e0 tx=0x7f18f800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:14:12.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1904ff9700 1 -- 172.21.15.7:0/139453777 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8004c80 con 0x7f1908071e80 2023-03-03T14:14:12.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1904ff9700 1 -- 172.21.15.7:0/139453777 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f8004e00 con 0x7f1908071e80 2023-03-03T14:14:12.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f190e60c700 1 -- 172.21.15.7:0/139453777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1908151f30 con 0x7f1908071e80 2023-03-03T14:14:12.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.308+0000 7f1904ff9700 1 -- 172.21.15.7:0/139453777 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8007310 con 0x7f1908071e80 2023-03-03T14:14:12.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.309+0000 7f190e60c700 1 -- 172.21.15.7:0/139453777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1908151300 con 0x7f1908071e80 2023-03-03T14:14:12.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.310+0000 7f190e60c700 1 -- 172.21.15.7:0/139453777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190806c9a0 con 0x7f1908071e80 2023-03-03T14:14:12.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.310+0000 7f1904ff9700 1 -- 172.21.15.7:0/139453777 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f18f801a020 con 0x7f1908071e80 2023-03-03T14:14:12.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.310+0000 7f1904ff9700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18f40682f0 0x7f18f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:12.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.310+0000 7f1904ff9700 1 -- 172.21.15.7:0/139453777 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f18f8085da0 con 0x7f1908071e80 2023-03-03T14:14:12.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.311+0000 7f19077fe700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18f40682f0 0x7f18f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:12.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.311+0000 7f19077fe700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18f40682f0 0x7f18f406a7a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f18fc006fd0 tx=0x7f18fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:14:12.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.315+0000 7f1904ff9700 1 -- 172.21.15.7:0/139453777 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18f8052140 con 0x7f1908071e80 2023-03-03T14:14:12.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.554+0000 7f190e60c700 1 -- 172.21.15.7:0/139453777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1908074bb0 con 0x7f1908071e80 2023-03-03T14:14:12.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.554+0000 7f1904ff9700 1 -- 172.21.15.7:0/139453777 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f18f8055760 con 0x7f1908071e80 2023-03-03T14:14:12.557 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:14:12.557 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:14:12.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 -- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18f40682f0 msgr2=0x7f18f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:12.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18f40682f0 0x7f18f406a7a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f18fc006fd0 tx=0x7f18fc008040 comp rx=0 tx=0).stop 2023-03-03T14:14:12.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 -- 172.21.15.7:0/139453777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1908071e80 msgr2=0x7f19081eb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:12.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1908071e80 0x7f19081eb570 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f18f80011e0 tx=0x7f18f800cae0 comp rx=0 tx=0).stop 2023-03-03T14:14:12.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 -- 172.21.15.7:0/139453777 shutdown_connections 2023-03-03T14:14:12.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19080c1ce0 0x7f19080c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:12.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18f40682f0 0x7f18f406a7a0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:12.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1908071e80 0x7f19081eb570 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:12.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 --2- 172.21.15.7:0/139453777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1908072250 0x7f1908149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:12.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 -- 172.21.15.7:0/139453777 >> 172.21.15.7:0/139453777 conn(0x7f19080a94d0 msgr2=0x7f190814dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:14:12.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 -- 172.21.15.7:0/139453777 shutdown_connections 2023-03-03T14:14:12.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:12.556+0000 7f190e60c700 1 -- 172.21.15.7:0/139453777 wait complete. 2023-03-03T14:14:12.561 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:14:12.570 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2023-03-03T14:14:21.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:21.299 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:51.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:51.362 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:14:55.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:14:55.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.705+0000 7f1ce4f25700 1 Processor -- start 2023-03-03T14:14:55.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.705+0000 7f1ce4f25700 1 -- start start 2023-03-03T14:14:55.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1ce4f25700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce014fda0 0x7f1ce0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:55.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1ce4f25700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce00c2f30 con 0x7f1ce00abe20 2023-03-03T14:14:55.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1ce4f25700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce00c31d0 con 0x7f1ce014f9d0 2023-03-03T14:14:55.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1ce4f25700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce01524a0 con 0x7f1ce014fda0 2023-03-03T14:14:55.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cded9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1ce00abe20 0x7f1ce01529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41840/0 (socket says 172.21.15.7:41840) 2023-03-03T14:14:55.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cded9d700 1 -- 172.21.15.7:0/1406859267 learned_addr learned my addr 172.21.15.7:0/1406859267 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:14:55.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cddd9b700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce014fda0 0x7f1ce0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:55.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515563608 0 0) 0x7f1ce00c2f30 con 0x7f1ce00abe20 2023-03-03T14:14:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc80035b0 con 0x7f1ce00abe20 2023-03-03T14:14:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1754862238 0 0) 0x7f1cc80035b0 con 0x7f1ce00abe20 2023-03-03T14:14:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce014fda0 msgr2=0x7f1ce0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:14:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce014fda0 0x7f1ce0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 >> v1:172.21.15.143:6789/0 conn(0x7f1ce014f9d0 legacy=0x7f1ce0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce00c34b0 con 0x7f1ce00abe20 2023-03-03T14:14:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f1cd80019f0 con 0x7f1ce00abe20 2023-03-03T14:14:55.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 >> v1:172.21.15.7:6789/0 conn(0x7f1ce00abe20 legacy=0x7f1ce01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:55.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cc8003aa0 0x7f1cc8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:55.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.706+0000 7f1cdd59a700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 0x7f1cc8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:55.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cdd59a700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce014fda0 0x7f1cc80052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:55.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce01524a0 con 0x7f1cc8004850 2023-03-03T14:14:55.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc800bba0 con 0x7f1cc8003aa0 2023-03-03T14:14:55.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc8010170 con 0x7f1ce014fda0 2023-03-03T14:14:55.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cddd9b700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 0x7f1cc8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:55.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cded9d700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce014fda0 0x7f1cc80052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:55.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cde59c700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cc8003aa0 0x7f1cc8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:55.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cddd9b700 1 -- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce014fda0 msgr2=0x7f1cc80052b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:14:55.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cddd9b700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce014fda0 0x7f1cc80052b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:55.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cddd9b700 1 -- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cc8003aa0 msgr2=0x7f1cc8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:14:55.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cddd9b700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1cc8003aa0 0x7f1cc8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:55.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cddd9b700 1 -- 172.21.15.7:0/1406859267 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc80035b0 con 0x7f1cc8004850 2023-03-03T14:14:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cded9d700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce014fda0 0x7f1cc80052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:14:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cddd9b700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 0x7f1cc8004cc0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f1ce01501d0 tx=0x7f1cc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a66db442408184 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:14:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd400e6c0 con 0x7f1cc8004850 2023-03-03T14:14:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd400e820 con 0x7f1cc8004850 2023-03-03T14:14:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.707+0000 7f1cdd59a700 1 -- 172.21.15.7:0/1406859267 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd400ebd0 con 0x7f1cc8004850 2023-03-03T14:14:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.708+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1406859267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 msgr2=0x7f1cc8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.708+0000 7f1ce4f25700 1 --2- 172.21.15.7:0/1406859267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 0x7f1cc8004cc0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f1ce01501d0 tx=0x7f1cc8005880 comp rx=0 tx=0).stop 2023-03-03T14:14:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.708+0000 7f1cded9d700 1 -- 172.21.15.7:0/1406859267 reap_dead start 2023-03-03T14:14:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.708+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1406859267 shutdown_connections 2023-03-03T14:14:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.708+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1406859267 >> 172.21.15.7:0/1406859267 conn(0x7f1ce00c8900 msgr2=0x7f1ce00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:14:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.708+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1406859267 shutdown_connections 2023-03-03T14:14:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.708+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1406859267 wait complete. 2023-03-03T14:14:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.709+0000 7f1ce4f25700 1 Processor -- start 2023-03-03T14:14:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.709+0000 7f1ce4f25700 1 -- start start 2023-03-03T14:14:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1ce4f25700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 0x7f1ce006b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1ce4f25700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce01e6ab0 0x7f1ce0154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1ce4f25700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce01550a0 0x7f1ce01e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1ce4f25700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce0147ae0 con 0x7f1cc8004850 2023-03-03T14:14:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1ce4f25700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce01e98f0 con 0x7f1ce01e6ab0 2023-03-03T14:14:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1ce4f25700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce01546d0 con 0x7f1ce01550a0 2023-03-03T14:14:55.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1cde59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 0x7f1ce006b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:55.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1cde59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 0x7f1ce006b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47868/0 (socket says 172.21.15.7:47868) 2023-03-03T14:14:55.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1cde59c700 1 -- 172.21.15.7:0/1984608571 learned_addr learned my addr 172.21.15.7:0/1984608571 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:14:55.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1cded9d700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce01550a0 0x7f1ce01e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:55.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1cddd9b700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce01e6ab0 0x7f1ce0154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:55.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1cde59c700 1 -- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce01550a0 msgr2=0x7f1ce01e8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:55.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1cde59c700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce01550a0 0x7f1ce01e8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:55.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1cde59c700 1 -- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce01e6ab0 msgr2=0x7f1ce0154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:55.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1cde59c700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce01e6ab0 0x7f1ce0154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:55.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.710+0000 7f1cde59c700 1 -- 172.21.15.7:0/1984608571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce00ac350 con 0x7f1cc8004850 2023-03-03T14:14:55.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.711+0000 7f1cde59c700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 0x7f1ce006b3c0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000f890 tx=0x7f1cd000fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:14:55.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.711+0000 7f1cceffd700 1 -- 172.21.15.7:0/1984608571 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00052e0 con 0x7f1cc8004850 2023-03-03T14:14:55.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.711+0000 7f1cceffd700 1 -- 172.21.15.7:0/1984608571 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd0005460 con 0x7f1cc8004850 2023-03-03T14:14:55.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.711+0000 7f1cceffd700 1 -- 172.21.15.7:0/1984608571 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00038c0 con 0x7f1cc8004850 2023-03-03T14:14:55.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.711+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1984608571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce00ab630 con 0x7f1cc8004850 2023-03-03T14:14:55.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.711+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1984608571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce0071fa0 con 0x7f1cc8004850 2023-03-03T14:14:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.712+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1984608571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce0074bb0 con 0x7f1cc8004850 2023-03-03T14:14:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.712+0000 7f1cceffd700 1 -- 172.21.15.7:0/1984608571 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1cd0003a20 con 0x7f1cc8004850 2023-03-03T14:14:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.713+0000 7f1cceffd700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc0068320 0x7f1cc006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.713+0000 7f1cceffd700 1 -- 172.21.15.7:0/1984608571 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1cd0086000 con 0x7f1cc8004850 2023-03-03T14:14:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.713+0000 7f1cddd9b700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc0068320 0x7f1cc006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.713+0000 7f1cddd9b700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc0068320 0x7f1cc006a7d0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f1cd40090b0 tx=0x7f1cd401e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:14:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.717+0000 7f1cceffd700 1 -- 172.21.15.7:0/1984608571 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1cd00523a0 con 0x7f1cc8004850 2023-03-03T14:14:55.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.952+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1984608571 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ce00c5e20 con 0x7f1cc8004850 2023-03-03T14:14:55.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.953+0000 7f1cceffd700 1 -- 172.21.15.7:0/1984608571 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1cd00559c0 con 0x7f1cc8004850 2023-03-03T14:14:55.954 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:14:55.955 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:14:55.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc0068320 msgr2=0x7f1cc006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:55.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc0068320 0x7f1cc006a7d0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f1cd40090b0 tx=0x7f1cd401e040 comp rx=0 tx=0).stop 2023-03-03T14:14:55.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1984608571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 msgr2=0x7f1ce006b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:55.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 0x7f1ce006b3c0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000f890 tx=0x7f1cd000fba0 comp rx=0 tx=0).stop 2023-03-03T14:14:55.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1984608571 shutdown_connections 2023-03-03T14:14:55.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1ce01550a0 0x7f1ce01e8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:55.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1cc0068320 0x7f1cc006a7d0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:55.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1cc8004850 0x7f1ce006b3c0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:55.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 --2- 172.21.15.7:0/1984608571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ce01e6ab0 0x7f1ce0154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:55.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1984608571 >> 172.21.15.7:0/1984608571 conn(0x7f1ce00c8900 msgr2=0x7f1ce01528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:14:55.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1984608571 shutdown_connections 2023-03-03T14:14:55.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:55.955+0000 7f1ce4f25700 1 -- 172.21.15.7:0/1984608571 wait complete. 2023-03-03T14:14:55.958 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:14:55.968 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2023-03-03T14:14:56.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:14:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.978+0000 7f8605c63700 1 Processor -- start 2023-03-03T14:14:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.979+0000 7f8605c63700 1 -- start start 2023-03-03T14:14:56.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.979+0000 7f8605c63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f860014fda0 0x7f8600156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:56.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.979+0000 7f8605c63700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f86000abe20 0x7f86001529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:56.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f8605c63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86000c2f30 con 0x7f860014f9d0 2023-03-03T14:14:56.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f8605c63700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86000c31d0 con 0x7f86000abe20 2023-03-03T14:14:56.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f8605c63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86001524a0 con 0x7f860014fda0 2023-03-03T14:14:56.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f86000abe20 0x7f86001529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:56.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f86000abe20 0x7f86001529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47880/0 (socket says 172.21.15.7:47880) 2023-03-03T14:14:56.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fffff700 1 -- 172.21.15.7:0/3234059419 learned_addr learned my addr 172.21.15.7:0/3234059419 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:14:56.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85feffd700 1 --2- 172.21.15.7:0/3234059419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f860014fda0 0x7f8600156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:56.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fffff700 1 -- 172.21.15.7:0/3234059419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f860014fda0 msgr2=0x7f8600156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:14:56.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fffff700 1 --2- 172.21.15.7:0/3234059419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f860014fda0 0x7f8600156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:56.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fffff700 1 -- 172.21.15.7:0/3234059419 >> v1:172.21.15.143:6789/0 conn(0x7f860014f9d0 legacy=0x7f8600071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:56.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fffff700 1 -- 172.21.15.7:0/3234059419 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86000c34b0 con 0x7f86000abe20 2023-03-03T14:14:56.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fffff700 1 --2- 172.21.15.7:0/3234059419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86000abe20 0x7f86001529c0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009e50 tx=0x7f85f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11eeac26d1160c52 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:14:56.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fe7fc700 1 -- 172.21.15.7:0/3234059419 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f800e070 con 0x7f86000abe20 2023-03-03T14:14:56.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fe7fc700 1 -- 172.21.15.7:0/3234059419 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f800b9c0 con 0x7f86000abe20 2023-03-03T14:14:56.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.980+0000 7f85fe7fc700 1 -- 172.21.15.7:0/3234059419 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f800bd70 con 0x7f86000abe20 2023-03-03T14:14:56.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.981+0000 7f8605c63700 1 -- 172.21.15.7:0/3234059419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86000abe20 msgr2=0x7f86001529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:56.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.981+0000 7f8605c63700 1 --2- 172.21.15.7:0/3234059419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86000abe20 0x7f86001529c0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009e50 tx=0x7f85f800b330 comp rx=0 tx=0).stop 2023-03-03T14:14:56.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.981+0000 7f8605c63700 1 -- 172.21.15.7:0/3234059419 shutdown_connections 2023-03-03T14:14:56.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.981+0000 7f8605c63700 1 --2- 172.21.15.7:0/3234059419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86000abe20 0x7f86001529c0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:56.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.981+0000 7f8605c63700 1 --2- 172.21.15.7:0/3234059419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f860014fda0 0x7f8600156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:56.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.981+0000 7f8605c63700 1 -- 172.21.15.7:0/3234059419 >> 172.21.15.7:0/3234059419 conn(0x7f86000c8900 msgr2=0x7f86000c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:14:56.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.981+0000 7f8605c63700 1 -- 172.21.15.7:0/3234059419 shutdown_connections 2023-03-03T14:14:56.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.981+0000 7f8605c63700 1 -- 172.21.15.7:0/3234059419 wait complete. 2023-03-03T14:14:56.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.983+0000 7f8605c63700 1 Processor -- start 2023-03-03T14:14:56.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.983+0000 7f8605c63700 1 -- start start 2023-03-03T14:14:56.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f8605c63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86000abe20 0x7f86000a9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:56.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f8605c63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860014f9d0 0x7f860006cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:56.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f8605c63700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860014fda0 0x7f860006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:56.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f8605c63700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86000c2530 con 0x7f860014fda0 2023-03-03T14:14:56.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f8605c63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86000c1f00 con 0x7f860014f9d0 2023-03-03T14:14:56.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f8605c63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860014edd0 con 0x7f86000abe20 2023-03-03T14:14:56.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85fffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860014fda0 0x7f860006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:56.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85fffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860014fda0 0x7f860006d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47886/0 (socket says 172.21.15.7:47886) 2023-03-03T14:14:56.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85fffff700 1 -- 172.21.15.7:0/1779606246 learned_addr learned my addr 172.21.15.7:0/1779606246 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:14:56.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85feffd700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860014f9d0 0x7f860006cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:56.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85ff7fe700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86000abe20 0x7f86000a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:56.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85fffff700 1 -- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86000abe20 msgr2=0x7f86000a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:56.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85fffff700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86000abe20 0x7f86000a9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:56.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85fffff700 1 -- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860014f9d0 msgr2=0x7f860006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:56.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85fffff700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860014f9d0 0x7f860006cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:56.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85fffff700 1 -- 172.21.15.7:0/1779606246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f8009b30 con 0x7f860014fda0 2023-03-03T14:14:56.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.984+0000 7f85fffff700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860014fda0 0x7f860006d4f0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f8600072520 tx=0x7f85f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:14:56.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.985+0000 7f85e7fff700 1 -- 172.21.15.7:0/1779606246 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8003ea0 con 0x7f860014fda0 2023-03-03T14:14:56.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.985+0000 7f85e7fff700 1 -- 172.21.15.7:0/1779606246 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f80048e0 con 0x7f860014fda0 2023-03-03T14:14:56.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.985+0000 7f8605c63700 1 -- 172.21.15.7:0/1779606246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86000ac350 con 0x7f860014fda0 2023-03-03T14:14:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.985+0000 7f85e7fff700 1 -- 172.21.15.7:0/1779606246 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f801b670 con 0x7f860014fda0 2023-03-03T14:14:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.985+0000 7f8605c63700 1 -- 172.21.15.7:0/1779606246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86000ab630 con 0x7f860014fda0 2023-03-03T14:14:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.985+0000 7f8605c63700 1 -- 172.21.15.7:0/1779606246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8600071fa0 con 0x7f860014fda0 2023-03-03T14:14:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.986+0000 7f85e7fff700 1 -- 172.21.15.7:0/1779606246 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f85f802b070 con 0x7f860014fda0 2023-03-03T14:14:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.986+0000 7f85e7fff700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85ec068320 0x7f85ec06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:14:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.987+0000 7f85e7fff700 1 -- 172.21.15.7:0/1779606246 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f85f8087740 con 0x7f860014fda0 2023-03-03T14:14:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.987+0000 7f85ff7fe700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85ec068320 0x7f85ec06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:14:56.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.987+0000 7f85ff7fe700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85ec068320 0x7f85ec06a7d0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f85f00018a0 tx=0x7f85f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:14:56.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:56.991+0000 7f85e7fff700 1 -- 172.21.15.7:0/1779606246 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f85f8053ae0 con 0x7f860014fda0 2023-03-03T14:14:57.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.229+0000 7f8605c63700 1 -- 172.21.15.7:0/1779606246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8600074bb0 con 0x7f860014fda0 2023-03-03T14:14:57.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.230+0000 7f85e7fff700 1 -- 172.21.15.7:0/1779606246 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f85f8057100 con 0x7f860014fda0 2023-03-03T14:14:57.231 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:14:57.232 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:14:57.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 -- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85ec068320 msgr2=0x7f85ec06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:57.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85ec068320 0x7f85ec06a7d0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f85f00018a0 tx=0x7f85f0009040 comp rx=0 tx=0).stop 2023-03-03T14:14:57.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 -- 172.21.15.7:0/1779606246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860014fda0 msgr2=0x7f860006d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:14:57.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860014fda0 0x7f860006d4f0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f8600072520 tx=0x7f85f8004ad0 comp rx=0 tx=0).stop 2023-03-03T14:14:57.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 -- 172.21.15.7:0/1779606246 shutdown_connections 2023-03-03T14:14:57.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85ec068320 0x7f85ec06a7d0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:57.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f860014fda0 0x7f860006d4f0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:57.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f86000abe20 0x7f86000a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:57.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 --2- 172.21.15.7:0/1779606246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f860014f9d0 0x7f860006cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:14:57.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 -- 172.21.15.7:0/1779606246 >> 172.21.15.7:0/1779606246 conn(0x7f86000c8900 msgr2=0x7f86000c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:14:57.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 -- 172.21.15.7:0/1779606246 shutdown_connections 2023-03-03T14:14:57.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:14:57.232+0000 7f8605c63700 1 -- 172.21.15.7:0/1779606246 wait complete. 2023-03-03T14:14:57.236 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:14:57.246 INFO:tasks.mds_thrash.fs.[b]:waiting for 49 secs before thrashing 2023-03-03T14:15:21.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:21.425 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:38.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:15:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.108+0000 7f6253c41700 1 Processor -- start 2023-03-03T14:15:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.108+0000 7f6253c41700 1 -- start start 2023-03-03T14:15:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f6253c41700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f624c0c17f0 0x7f624c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f6253c41700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f624c0c5190 0x7f624c06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f6253c41700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f624c14ebe0 con 0x7f624c0c6010 2023-03-03T14:15:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f6253c41700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f624c152100 con 0x7f624c0c17f0 2023-03-03T14:15:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f6253c41700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624c0c2e70 con 0x7f624c0c5190 2023-03-03T14:15:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f62511dc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f624c0c17f0 0x7f624c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f62519dd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f624c0c6010 0x7f624c0c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41890/0 (socket says 172.21.15.7:41890) 2023-03-03T14:15:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f62519dd700 1 -- 172.21.15.7:0/2069926477 learned_addr learned my addr 172.21.15.7:0/2069926477 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:15:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f62521de700 1 --2- 172.21.15.7:0/2069926477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f624c0c5190 0x7f624c06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f62511dc700 1 -- 172.21.15.7:0/2069926477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f624c0c5190 msgr2=0x7f624c06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:15:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f62511dc700 1 --2- 172.21.15.7:0/2069926477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f624c0c5190 0x7f624c06dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f62511dc700 1 -- 172.21.15.7:0/2069926477 >> v1:172.21.15.7:6789/0 conn(0x7f624c0c6010 legacy=0x7f624c0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f62511dc700 1 -- 172.21.15.7:0/2069926477 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c0aa680 con 0x7f624c0c17f0 2023-03-03T14:15:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f62509db700 1 -- 172.21.15.7:0/2069926477 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779212372 0 0) 0x7f624c14ebe0 con 0x7f624c0c6010 2023-03-03T14:15:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.109+0000 7f62511dc700 1 --2- 172.21.15.7:0/2069926477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c17f0 0x7f624c06d6a0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f623c009840 tx=0x7f623c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf46bd627d64dd5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f62509db700 1 -- 172.21.15.7:0/2069926477 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c011830 con 0x7f624c0c17f0 2023-03-03T14:15:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f62509db700 1 -- 172.21.15.7:0/2069926477 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c011990 con 0x7f624c0c17f0 2023-03-03T14:15:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f62509db700 1 -- 172.21.15.7:0/2069926477 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c011d20 con 0x7f624c0c17f0 2023-03-03T14:15:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f6253c41700 1 -- 172.21.15.7:0/2069926477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c17f0 msgr2=0x7f624c06d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f6253c41700 1 --2- 172.21.15.7:0/2069926477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c17f0 0x7f624c06d6a0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f623c009840 tx=0x7f623c009b50 comp rx=0 tx=0).stop 2023-03-03T14:15:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f6253c41700 1 -- 172.21.15.7:0/2069926477 shutdown_connections 2023-03-03T14:15:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f6253c41700 1 --2- 172.21.15.7:0/2069926477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f624c0c5190 0x7f624c06dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f6253c41700 1 --2- 172.21.15.7:0/2069926477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c17f0 0x7f624c06d6a0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f6253c41700 1 -- 172.21.15.7:0/2069926477 >> 172.21.15.7:0/2069926477 conn(0x7f624c0a9430 msgr2=0x7f624c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:15:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f6253c41700 1 -- 172.21.15.7:0/2069926477 shutdown_connections 2023-03-03T14:15:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.110+0000 7f6253c41700 1 -- 172.21.15.7:0/2069926477 wait complete. 2023-03-03T14:15:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.111+0000 7f6253c41700 1 Processor -- start 2023-03-03T14:15:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f6253c41700 1 -- start start 2023-03-03T14:15:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f6253c41700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f624c0c17f0 0x7f624c06fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f6253c41700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c5190 0x7f624c06cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f6253c41700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f624c0c6010 0x7f624c14dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f6253c41700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c070af0 con 0x7f624c0c5190 2023-03-03T14:15:39.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f6253c41700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c1e8450 con 0x7f624c0c6010 2023-03-03T14:15:39.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f6253c41700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624c1534f0 con 0x7f624c0c17f0 2023-03-03T14:15:39.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f62511dc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c5190 0x7f624c06cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:39.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f62511dc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c5190 0x7f624c06cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47918/0 (socket says 172.21.15.7:47918) 2023-03-03T14:15:39.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f62511dc700 1 -- 172.21.15.7:0/2758180284 learned_addr learned my addr 172.21.15.7:0/2758180284 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:15:39.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f62519dd700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f624c0c17f0 0x7f624c06fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:39.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.112+0000 7f62521de700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f624c0c6010 0x7f624c14dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:39.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f62511dc700 1 -- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f624c0c17f0 msgr2=0x7f624c06fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:39.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f62511dc700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f624c0c17f0 0x7f624c06fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:39.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f62511dc700 1 -- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f624c0c6010 msgr2=0x7f624c14dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:39.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f62511dc700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f624c0c6010 0x7f624c14dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:39.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f62511dc700 1 -- 172.21.15.7:0/2758180284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623c0094a0 con 0x7f624c0c5190 2023-03-03T14:15:39.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f62511dc700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c5190 0x7f624c06cb50 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f623c014040 tx=0x7f623c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:39.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f62427fc700 1 -- 172.21.15.7:0/2758180284 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c0036a0 con 0x7f624c0c5190 2023-03-03T14:15:39.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f62427fc700 1 -- 172.21.15.7:0/2758180284 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c01e070 con 0x7f624c0c5190 2023-03-03T14:15:39.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f62427fc700 1 -- 172.21.15.7:0/2758180284 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c003a40 con 0x7f624c0c5190 2023-03-03T14:15:39.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f6253c41700 1 -- 172.21.15.7:0/2758180284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624c0c2b00 con 0x7f624c0c5190 2023-03-03T14:15:39.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.113+0000 7f6253c41700 1 -- 172.21.15.7:0/2758180284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f624c0aa960 con 0x7f624c0c5190 2023-03-03T14:15:39.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.114+0000 7f6253c41700 1 -- 172.21.15.7:0/2758180284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f624c0c32e0 con 0x7f624c0c5190 2023-03-03T14:15:39.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.114+0000 7f62427fc700 1 -- 172.21.15.7:0/2758180284 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f623c00f050 con 0x7f624c0c5190 2023-03-03T14:15:39.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.115+0000 7f62427fc700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62300682f0 0x7f623006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:39.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.115+0000 7f62427fc700 1 -- 172.21.15.7:0/2758180284 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f623c087260 con 0x7f624c0c5190 2023-03-03T14:15:39.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.115+0000 7f62519dd700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62300682f0 0x7f623006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:39.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.116+0000 7f62519dd700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62300682f0 0x7f623006a7a0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f6248007fd0 tx=0x7f6248007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:39.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.120+0000 7f62427fc700 1 -- 172.21.15.7:0/2758180284 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f623c053600 con 0x7f624c0c5190 2023-03-03T14:15:39.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.358+0000 7f6253c41700 1 -- 172.21.15.7:0/2758180284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f624c1e8980 con 0x7f624c0c5190 2023-03-03T14:15:39.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.358+0000 7f62427fc700 1 -- 172.21.15.7:0/2758180284 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f623c015090 con 0x7f624c0c5190 2023-03-03T14:15:39.361 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:15:39.361 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:15:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 -- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62300682f0 msgr2=0x7f623006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62300682f0 0x7f623006a7a0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f6248007fd0 tx=0x7f6248007de0 comp rx=0 tx=0).stop 2023-03-03T14:15:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 -- 172.21.15.7:0/2758180284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c5190 msgr2=0x7f624c06cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c5190 0x7f624c06cb50 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f623c014040 tx=0x7f623c004750 comp rx=0 tx=0).stop 2023-03-03T14:15:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 -- 172.21.15.7:0/2758180284 shutdown_connections 2023-03-03T14:15:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f624c0c6010 0x7f624c14dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f624c0c17f0 0x7f624c06fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62300682f0 0x7f623006a7a0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 --2- 172.21.15.7:0/2758180284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f624c0c5190 0x7f624c06cb50 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 -- 172.21.15.7:0/2758180284 >> 172.21.15.7:0/2758180284 conn(0x7f624c0a9430 msgr2=0x7f624c1512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:15:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 -- 172.21.15.7:0/2758180284 shutdown_connections 2023-03-03T14:15:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:39.360+0000 7f6253c41700 1 -- 172.21.15.7:0/2758180284 wait complete. 2023-03-03T14:15:39.365 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:15:39.374 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2023-03-03T14:15:46.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:15:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.382+0000 7f27036c1700 1 Processor -- start 2023-03-03T14:15:46.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.382+0000 7f27036c1700 1 -- start start 2023-03-03T14:15:46.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f27036c1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f26fc151770 0x7f26fc151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:46.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f27036c1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc154910 con 0x7f26fc155390 2023-03-03T14:15:46.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f27036c1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc0c4540 con 0x7f26fc06e020 2023-03-03T14:15:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f27036c1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc070b90 con 0x7f26fc151770 2023-03-03T14:15:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f2701c5e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f26fc151770 0x7f26fc151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f270145d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f26fc155390 0x7f26fc0c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41904/0 (socket says 172.21.15.7:41904) 2023-03-03T14:15:46.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f270145d700 1 -- 172.21.15.7:0/377502172 learned_addr learned my addr 172.21.15.7:0/377502172 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:15:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f2701c5e700 1 -- 172.21.15.7:0/377502172 >> v1:172.21.15.143:6789/0 conn(0x7f26fc06e020 legacy=0x7f26fc071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:46.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f2701c5e700 1 -- 172.21.15.7:0/377502172 >> v1:172.21.15.7:6789/0 conn(0x7f26fc155390 legacy=0x7f26fc0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f2701c5e700 1 -- 172.21.15.7:0/377502172 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc0702a0 con 0x7f26fc151770 2023-03-03T14:15:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.383+0000 7f26effff700 1 -- 172.21.15.7:0/377502172 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426293005 0 0) 0x7f26fc154910 con 0x7f26fc155390 2023-03-03T14:15:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f2701c5e700 1 --2- 172.21.15.7:0/377502172 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc151770 0x7f26fc151bd0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009d90 tx=0x7f26f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ef9d082ca1d3bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f26effff700 1 -- 172.21.15.7:0/377502172 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f400e070 con 0x7f26fc151770 2023-03-03T14:15:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f26effff700 1 -- 172.21.15.7:0/377502172 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f400bbd0 con 0x7f26fc151770 2023-03-03T14:15:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f26effff700 1 -- 172.21.15.7:0/377502172 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4012430 con 0x7f26fc151770 2023-03-03T14:15:46.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f27036c1700 1 -- 172.21.15.7:0/377502172 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc151770 msgr2=0x7f26fc151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:46.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f27036c1700 1 --2- 172.21.15.7:0/377502172 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc151770 0x7f26fc151bd0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009d90 tx=0x7f26f400b330 comp rx=0 tx=0).stop 2023-03-03T14:15:46.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f27036c1700 1 -- 172.21.15.7:0/377502172 shutdown_connections 2023-03-03T14:15:46.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f27036c1700 1 --2- 172.21.15.7:0/377502172 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc151770 0x7f26fc151bd0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:46.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f27036c1700 1 -- 172.21.15.7:0/377502172 >> 172.21.15.7:0/377502172 conn(0x7f26fc06c110 msgr2=0x7f26fc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:15:46.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f27036c1700 1 -- 172.21.15.7:0/377502172 shutdown_connections 2023-03-03T14:15:46.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.384+0000 7f27036c1700 1 -- 172.21.15.7:0/377502172 wait complete. 2023-03-03T14:15:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f27036c1700 1 Processor -- start 2023-03-03T14:15:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f27036c1700 1 -- start start 2023-03-03T14:15:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f27036c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26fc06e020 0x7f26fc1e5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:46.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f27036c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26fc151770 0x7f26fc153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f27036c1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc155390 0x7f26fc154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f27036c1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc150d20 con 0x7f26fc155390 2023-03-03T14:15:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f27036c1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc1e64f0 con 0x7f26fc151770 2023-03-03T14:15:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f27036c1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc14f190 con 0x7f26fc06e020 2023-03-03T14:15:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f2701c5e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc155390 0x7f26fc154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f2701c5e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc155390 0x7f26fc154320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47934/0 (socket says 172.21.15.7:47934) 2023-03-03T14:15:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f2701c5e700 1 -- 172.21.15.7:0/1176347590 learned_addr learned my addr 172.21.15.7:0/1176347590 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:15:46.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.386+0000 7f2700c5c700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26fc151770 0x7f26fc153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:46.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f270145d700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26fc06e020 0x7f26fc1e5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:46.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f2701c5e700 1 -- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26fc06e020 msgr2=0x7f26fc1e5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:46.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f2701c5e700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26fc06e020 0x7f26fc1e5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:46.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f2701c5e700 1 -- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26fc151770 msgr2=0x7f26fc153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:46.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f2701c5e700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26fc151770 0x7f26fc153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:46.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f2701c5e700 1 -- 172.21.15.7:0/1176347590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f4009a70 con 0x7f26fc155390 2023-03-03T14:15:46.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f2701c5e700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc155390 0x7f26fc154320 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f26f4014040 tx=0x7f26f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:46.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f26edffb700 1 -- 172.21.15.7:0/1176347590 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4012430 con 0x7f26fc155390 2023-03-03T14:15:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f26edffb700 1 -- 172.21.15.7:0/1176347590 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f4020070 con 0x7f26fc155390 2023-03-03T14:15:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f27036c1700 1 -- 172.21.15.7:0/1176347590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc070820 con 0x7f26fc155390 2023-03-03T14:15:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f26edffb700 1 -- 172.21.15.7:0/1176347590 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4004280 con 0x7f26fc155390 2023-03-03T14:15:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.387+0000 7f27036c1700 1 -- 172.21.15.7:0/1176347590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc070540 con 0x7f26fc155390 2023-03-03T14:15:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.388+0000 7f27036c1700 1 -- 172.21.15.7:0/1176347590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc155760 con 0x7f26fc155390 2023-03-03T14:15:46.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.389+0000 7f26edffb700 1 -- 172.21.15.7:0/1176347590 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f26f40043e0 con 0x7f26fc155390 2023-03-03T14:15:46.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.390+0000 7f26edffb700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26e00682f0 0x7f26e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:46.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.390+0000 7f26edffb700 1 -- 172.21.15.7:0/1176347590 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f26f4022070 con 0x7f26fc155390 2023-03-03T14:15:46.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.390+0000 7f270145d700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26e00682f0 0x7f26e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:46.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.391+0000 7f270145d700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26e00682f0 0x7f26e006a7a0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f26f8007c40 tx=0x7f26f8007b90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.394+0000 7f26edffb700 1 -- 172.21.15.7:0/1176347590 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f26f4054c60 con 0x7f26fc155390 2023-03-03T14:15:46.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.632+0000 7f27036c1700 1 -- 172.21.15.7:0/1176347590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26fc074bb0 con 0x7f26fc155390 2023-03-03T14:15:46.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.632+0000 7f26edffb700 1 -- 172.21.15.7:0/1176347590 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f26f40263e0 con 0x7f26fc155390 2023-03-03T14:15:46.634 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:15:46.634 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:15:46.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.634+0000 7f27036c1700 1 -- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26e00682f0 msgr2=0x7f26e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:46.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.634+0000 7f27036c1700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26e00682f0 0x7f26e006a7a0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f26f8007c40 tx=0x7f26f8007b90 comp rx=0 tx=0).stop 2023-03-03T14:15:46.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.634+0000 7f27036c1700 1 -- 172.21.15.7:0/1176347590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc155390 msgr2=0x7f26fc154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:46.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.634+0000 7f27036c1700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc155390 0x7f26fc154320 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f26f4014040 tx=0x7f26f4004ad0 comp rx=0 tx=0).stop 2023-03-03T14:15:46.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.634+0000 7f27036c1700 1 -- 172.21.15.7:0/1176347590 shutdown_connections 2023-03-03T14:15:46.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.634+0000 7f27036c1700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26e00682f0 0x7f26e006a7a0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:46.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.634+0000 7f27036c1700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26fc155390 0x7f26fc154320 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:46.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.634+0000 7f27036c1700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26fc06e020 0x7f26fc1e5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:46.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.634+0000 7f27036c1700 1 --2- 172.21.15.7:0/1176347590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26fc151770 0x7f26fc153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:46.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.634+0000 7f27036c1700 1 -- 172.21.15.7:0/1176347590 >> 172.21.15.7:0/1176347590 conn(0x7f26fc06c110 msgr2=0x7f26fc0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:15:46.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.635+0000 7f27036c1700 1 -- 172.21.15.7:0/1176347590 shutdown_connections 2023-03-03T14:15:46.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:46.635+0000 7f27036c1700 1 -- 172.21.15.7:0/1176347590 wait complete. 2023-03-03T14:15:46.638 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:15:46.648 INFO:tasks.mds_thrash.fs.[b]:waiting for 55 secs before thrashing 2023-03-03T14:15:51.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:51.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:15:54.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:15:54.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.513+0000 7fb5075e7700 1 Processor -- start 2023-03-03T14:15:54.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.513+0000 7fb5075e7700 1 -- start start 2023-03-03T14:15:54.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb5075e7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb50014f9d0 0x7fb500156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:54.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb5075e7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5000abe20 0x7fb5001529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:54.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb5075e7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5000c2f30 con 0x7fb50014fda0 2023-03-03T14:15:54.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb5075e7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5000c31d0 con 0x7fb5000abe20 2023-03-03T14:15:54.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb5075e7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5001524a0 con 0x7fb50014f9d0 2023-03-03T14:15:54.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb504b82700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb50014fda0 0x7fb500071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41930/0 (socket says 172.21.15.7:41930) 2023-03-03T14:15:54.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb504b82700 1 -- 172.21.15.7:0/1573147496 learned_addr learned my addr 172.21.15.7:0/1573147496 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:15:54.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096734254 0 0) 0x7fb5000c2f30 con 0x7fb50014fda0 2023-03-03T14:15:54.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb505383700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb50014f9d0 0x7fb500156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:54.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4ec0035b0 con 0x7fb50014fda0 2023-03-03T14:15:54.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb505b84700 1 --2- 172.21.15.7:0/1573147496 >> v2:172.21.15.143:3300/0 conn(0x7fb5000abe20 0x7fb5001529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:54.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1963513211 0 0) 0x7fb4ec0035b0 con 0x7fb50014fda0 2023-03-03T14:15:54.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb50014f9d0 msgr2=0x7fb500156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:15:54.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb50014f9d0 0x7fb500156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:54.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 >> v2:172.21.15.143:3300/0 conn(0x7fb5000abe20 msgr2=0x7fb5001529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:15:54.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 --2- 172.21.15.7:0/1573147496 >> v2:172.21.15.143:3300/0 conn(0x7fb5000abe20 0x7fb5001529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:54.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5000c34b0 con 0x7fb50014fda0 2023-03-03T14:15:54.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb4f4001670 con 0x7fb50014fda0 2023-03-03T14:15:54.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 >> v1:172.21.15.7:6789/0 conn(0x7fb50014fda0 legacy=0x7fb500071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:54.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb50014f9d0 0x7fb4ec003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec003ff0 0x7fb4ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.514+0000 7fb4f3fff700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec004d90 0x7fb4ec005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5000c31d0 con 0x7fb4ec003ff0 2023-03-03T14:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5001524a0 con 0x7fb4ec004d90 2023-03-03T14:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec00bb70 con 0x7fb50014f9d0 2023-03-03T14:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb504b82700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec003ff0 0x7fb4ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb505383700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb50014f9d0 0x7fb4ec003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb505b84700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec004d90 0x7fb4ec005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb504b82700 1 -- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb50014f9d0 msgr2=0x7fb4ec003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb504b82700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb50014f9d0 0x7fb4ec003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb504b82700 1 -- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec004d90 msgr2=0x7fb4ec005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb504b82700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec004d90 0x7fb4ec005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb504b82700 1 -- 172.21.15.7:0/1573147496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec0035b0 con 0x7fb4ec003ff0 2023-03-03T14:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb504b82700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec003ff0 0x7fb4ec004850 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4001050 tx=0x7fb4ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f16c671a4231b17 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4007c60 con 0x7fb4ec003ff0 2023-03-03T14:15:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f400ce50 con 0x7fb4ec003ff0 2023-03-03T14:15:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.515+0000 7fb4f3fff700 1 -- 172.21.15.7:0/1573147496 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4005ba0 con 0x7fb4ec003ff0 2023-03-03T14:15:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.516+0000 7fb5075e7700 1 -- 172.21.15.7:0/1573147496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec003ff0 msgr2=0x7fb4ec004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.516+0000 7fb5075e7700 1 --2- 172.21.15.7:0/1573147496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec003ff0 0x7fb4ec004850 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4001050 tx=0x7fb4ec005880 comp rx=0 tx=0).stop 2023-03-03T14:15:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.516+0000 7fb505b84700 1 -- 172.21.15.7:0/1573147496 reap_dead start 2023-03-03T14:15:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.516+0000 7fb5075e7700 1 -- 172.21.15.7:0/1573147496 shutdown_connections 2023-03-03T14:15:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.516+0000 7fb5075e7700 1 -- 172.21.15.7:0/1573147496 >> 172.21.15.7:0/1573147496 conn(0x7fb5000c8900 msgr2=0x7fb5000c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:15:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.516+0000 7fb5075e7700 1 -- 172.21.15.7:0/1573147496 shutdown_connections 2023-03-03T14:15:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.516+0000 7fb5075e7700 1 -- 172.21.15.7:0/1573147496 wait complete. 2023-03-03T14:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.517+0000 7fb5075e7700 1 Processor -- start 2023-03-03T14:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.517+0000 7fb5075e7700 1 -- start start 2023-03-03T14:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb5075e7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec003ff0 0x7fb5001dba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb5075e7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5001ea760 0x7fb5000c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb5075e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5001edb20 0x7fb50006f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb5075e7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5001ed700 con 0x7fb5001edb20 2023-03-03T14:15:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb5075e7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5000c2990 con 0x7fb5001ea760 2023-03-03T14:15:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb5075e7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb500155280 con 0x7fb4ec003ff0 2023-03-03T14:15:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb505b84700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5001edb20 0x7fb50006f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb505b84700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5001edb20 0x7fb50006f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47964/0 (socket says 172.21.15.7:47964) 2023-03-03T14:15:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb505b84700 1 -- 172.21.15.7:0/518209091 learned_addr learned my addr 172.21.15.7:0/518209091 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:15:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb505383700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec003ff0 0x7fb5001dba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb504b82700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5001ea760 0x7fb5000c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb505b84700 1 -- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec003ff0 msgr2=0x7fb5001dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb505b84700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec003ff0 0x7fb5001dba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb505b84700 1 -- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5001ea760 msgr2=0x7fb5000c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb505b84700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5001ea760 0x7fb5000c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb505b84700 1 -- 172.21.15.7:0/518209091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5000ac350 con 0x7fb5001edb20 2023-03-03T14:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb505383700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec003ff0 0x7fb5001dba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.518+0000 7fb504b82700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5001ea760 0x7fb5000c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:15:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.519+0000 7fb505b84700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5001edb20 0x7fb50006f130 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fb5000ab150 tx=0x7fb4f8011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.519+0000 7fb4f1ffb700 1 -- 172.21.15.7:0/518209091 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f800e040 con 0x7fb5001edb20 2023-03-03T14:15:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.519+0000 7fb4f1ffb700 1 -- 172.21.15.7:0/518209091 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f800f040 con 0x7fb5001edb20 2023-03-03T14:15:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.519+0000 7fb5075e7700 1 -- 172.21.15.7:0/518209091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5000ab630 con 0x7fb5001edb20 2023-03-03T14:15:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.519+0000 7fb4f1ffb700 1 -- 172.21.15.7:0/518209091 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f800fdb0 con 0x7fb5001edb20 2023-03-03T14:15:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.519+0000 7fb5075e7700 1 -- 172.21.15.7:0/518209091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb500071fa0 con 0x7fb5001edb20 2023-03-03T14:15:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.520+0000 7fb5075e7700 1 -- 172.21.15.7:0/518209091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb500074bb0 con 0x7fb5001edb20 2023-03-03T14:15:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.520+0000 7fb4f1ffb700 1 -- 172.21.15.7:0/518209091 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb4f8015430 con 0x7fb5001edb20 2023-03-03T14:15:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.521+0000 7fb4f1ffb700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4e40682f0 0x7fb4e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:15:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.521+0000 7fb4f1ffb700 1 -- 172.21.15.7:0/518209091 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb4f8086c00 con 0x7fb5001edb20 2023-03-03T14:15:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.521+0000 7fb505383700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4e40682f0 0x7fb4e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:15:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.522+0000 7fb505383700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4e40682f0 0x7fb4e406a7a0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009b80 tx=0x7fb4fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:15:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.525+0000 7fb4f1ffb700 1 -- 172.21.15.7:0/518209091 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb4f808c020 con 0x7fb5001edb20 2023-03-03T14:15:54.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.767+0000 7fb5075e7700 1 -- 172.21.15.7:0/518209091 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5001ed840 con 0x7fb5001edb20 2023-03-03T14:15:54.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.767+0000 7fb4f1ffb700 1 -- 172.21.15.7:0/518209091 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb5001ed840 con 0x7fb5001edb20 2023-03-03T14:15:54.770 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:15:54.770 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:15:54.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 -- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4e40682f0 msgr2=0x7fb4e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:54.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4e40682f0 0x7fb4e406a7a0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009b80 tx=0x7fb4fc008040 comp rx=0 tx=0).stop 2023-03-03T14:15:54.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 -- 172.21.15.7:0/518209091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5001edb20 msgr2=0x7fb50006f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:15:54.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5001edb20 0x7fb50006f130 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fb5000ab150 tx=0x7fb4f8011f80 comp rx=0 tx=0).stop 2023-03-03T14:15:54.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 -- 172.21.15.7:0/518209091 shutdown_connections 2023-03-03T14:15:54.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4e40682f0 0x7fb4e406a7a0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5001edb20 0x7fb50006f130 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec003ff0 0x7fb5001dba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 --2- 172.21.15.7:0/518209091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5001ea760 0x7fb5000c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:15:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 -- 172.21.15.7:0/518209091 >> 172.21.15.7:0/518209091 conn(0x7fb5000c8900 msgr2=0x7fb5000c63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:15:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.769+0000 7fb5075e7700 1 -- 172.21.15.7:0/518209091 shutdown_connections 2023-03-03T14:15:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:15:54.770+0000 7fb5075e7700 1 -- 172.21.15.7:0/518209091 wait complete. 2023-03-03T14:15:54.773 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:15:54.783 INFO:tasks.mds_thrash.fs.[a]:waiting for 105 secs before thrashing 2023-03-03T14:16:21.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:21.556 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:41.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:16:41.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.786+0000 7fb484651700 1 Processor -- start 2023-03-03T14:16:41.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.786+0000 7fb484651700 1 -- start start 2023-03-03T14:16:41.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb484651700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb47c13a0f0 0x7fb47c0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb484651700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb47c1586c0 0x7fb47c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb484651700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb47c0c1da0 0x7fb47c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:41.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb484651700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c154840 con 0x7fb47c0c1da0 2023-03-03T14:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb484651700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c154560 con 0x7fb47c13a0f0 2023-03-03T14:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb484651700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c157be0 con 0x7fb47c1586c0 2023-03-03T14:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb482bee700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb47c0c1da0 0x7fb47c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:41.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb482bee700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb47c0c1da0 0x7fb47c06d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48000/0 (socket says 172.21.15.7:48000) 2023-03-03T14:16:41.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb482bee700 1 -- 172.21.15.7:0/2880541313 learned_addr learned my addr 172.21.15.7:0/2880541313 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:16:41.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb4823ed700 1 --2- 172.21.15.7:0/2880541313 >> v2:172.21.15.143:3300/0 conn(0x7fb47c13a0f0 0x7fb47c0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:41.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.787+0000 7fb481bec700 1 --2- 172.21.15.7:0/2880541313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb47c1586c0 0x7fb47c06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:41.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb482bee700 1 -- 172.21.15.7:0/2880541313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb47c1586c0 msgr2=0x7fb47c06d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:16:41.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb482bee700 1 --2- 172.21.15.7:0/2880541313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb47c1586c0 0x7fb47c06d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:41.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb482bee700 1 -- 172.21.15.7:0/2880541313 >> v2:172.21.15.143:3300/0 conn(0x7fb47c13a0f0 msgr2=0x7fb47c0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:16:41.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb482bee700 1 --2- 172.21.15.7:0/2880541313 >> v2:172.21.15.143:3300/0 conn(0x7fb47c13a0f0 0x7fb47c0c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:41.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb482bee700 1 -- 172.21.15.7:0/2880541313 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c157ec0 con 0x7fb47c0c1da0 2023-03-03T14:16:41.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb481bec700 1 --2- 172.21.15.7:0/2880541313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb47c1586c0 0x7fb47c06d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:16:41.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb4823ed700 1 --2- 172.21.15.7:0/2880541313 >> v2:172.21.15.143:3300/0 conn(0x7fb47c13a0f0 0x7fb47c0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:16:41.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb482bee700 1 --2- 172.21.15.7:0/2880541313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c0c1da0 0x7fb47c06d780 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fb47c06cf70 tx=0x7fb47400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73de1508a2f72d0d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:16:41.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb4813eb700 1 -- 172.21.15.7:0/2880541313 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47400e070 con 0x7fb47c0c1da0 2023-03-03T14:16:41.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb4813eb700 1 -- 172.21.15.7:0/2880541313 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47400b9c0 con 0x7fb47c0c1da0 2023-03-03T14:16:41.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.788+0000 7fb4813eb700 1 -- 172.21.15.7:0/2880541313 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47400bd70 con 0x7fb47c0c1da0 2023-03-03T14:16:41.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.789+0000 7fb484651700 1 -- 172.21.15.7:0/2880541313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c0c1da0 msgr2=0x7fb47c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:41.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.789+0000 7fb484651700 1 --2- 172.21.15.7:0/2880541313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c0c1da0 0x7fb47c06d780 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fb47c06cf70 tx=0x7fb47400b330 comp rx=0 tx=0).stop 2023-03-03T14:16:41.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.789+0000 7fb484651700 1 -- 172.21.15.7:0/2880541313 shutdown_connections 2023-03-03T14:16:41.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.789+0000 7fb484651700 1 --2- 172.21.15.7:0/2880541313 >> v2:172.21.15.143:3300/0 conn(0x7fb47c13a0f0 0x7fb47c0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:41.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.789+0000 7fb484651700 1 --2- 172.21.15.7:0/2880541313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c0c1da0 0x7fb47c06d780 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:41.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.789+0000 7fb484651700 1 --2- 172.21.15.7:0/2880541313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb47c1586c0 0x7fb47c06d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:41.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.789+0000 7fb484651700 1 -- 172.21.15.7:0/2880541313 >> 172.21.15.7:0/2880541313 conn(0x7fb47c06c110 msgr2=0x7fb47c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:16:41.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.789+0000 7fb484651700 1 -- 172.21.15.7:0/2880541313 shutdown_connections 2023-03-03T14:16:41.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.789+0000 7fb484651700 1 -- 172.21.15.7:0/2880541313 wait complete. 2023-03-03T14:16:41.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.790+0000 7fb484651700 1 Processor -- start 2023-03-03T14:16:41.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.790+0000 7fb484651700 1 -- start start 2023-03-03T14:16:41.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb484651700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb47c0c1da0 0x7fb47c0c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:41.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb484651700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c13a0f0 0x7fb47c1ed0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:41.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb484651700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb47c1586c0 0x7fb47c1ed620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:41.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb484651700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c151660 con 0x7fb47c13a0f0 2023-03-03T14:16:41.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb484651700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0c6d90 con 0x7fb47c0c1da0 2023-03-03T14:16:41.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb484651700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c071410 con 0x7fb47c1586c0 2023-03-03T14:16:41.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb481bec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c13a0f0 0x7fb47c1ed0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:41.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb481bec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c13a0f0 0x7fb47c1ed0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48004/0 (socket says 172.21.15.7:48004) 2023-03-03T14:16:41.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb481bec700 1 -- 172.21.15.7:0/2897848437 learned_addr learned my addr 172.21.15.7:0/2897848437 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:16:41.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb481bec700 1 -- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb47c1586c0 msgr2=0x7fb47c1ed620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:41.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb481bec700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb47c1586c0 0x7fb47c1ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:41.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb4823ed700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb47c0c1da0 0x7fb47c0c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:41.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb481bec700 1 -- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb47c0c1da0 msgr2=0x7fb47c0c0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:41.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb481bec700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb47c0c1da0 0x7fb47c0c0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:41.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb481bec700 1 -- 172.21.15.7:0/2897848437 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb474009b30 con 0x7fb47c13a0f0 2023-03-03T14:16:41.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb481bec700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c13a0f0 0x7fb47c1ed0e0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fb47800e840 tx=0x7fb47800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:16:41.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb46effd700 1 -- 172.21.15.7:0/2897848437 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb478013610 con 0x7fb47c13a0f0 2023-03-03T14:16:41.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb46effd700 1 -- 172.21.15.7:0/2897848437 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb478013770 con 0x7fb47c13a0f0 2023-03-03T14:16:41.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb484651700 1 -- 172.21.15.7:0/2897848437 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c06e1a0 con 0x7fb47c13a0f0 2023-03-03T14:16:41.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb46effd700 1 -- 172.21.15.7:0/2897848437 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb478013b00 con 0x7fb47c13a0f0 2023-03-03T14:16:41.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.791+0000 7fb484651700 1 -- 172.21.15.7:0/2897848437 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c1581a0 con 0x7fb47c13a0f0 2023-03-03T14:16:41.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.793+0000 7fb484651700 1 -- 172.21.15.7:0/2897848437 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c06e330 con 0x7fb47c13a0f0 2023-03-03T14:16:41.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.793+0000 7fb46effd700 1 -- 172.21.15.7:0/2897848437 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb478023070 con 0x7fb47c13a0f0 2023-03-03T14:16:41.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.793+0000 7fb46effd700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4600682f0 0x7fb46006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:16:41.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.793+0000 7fb46effd700 1 -- 172.21.15.7:0/2897848437 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb478085d60 con 0x7fb47c13a0f0 2023-03-03T14:16:41.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.794+0000 7fb4823ed700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4600682f0 0x7fb46006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:16:41.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.794+0000 7fb4823ed700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4600682f0 0x7fb46006a7a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fb470009910 tx=0x7fb470008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:16:41.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:41.798+0000 7fb46effd700 1 -- 172.21.15.7:0/2897848437 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb478052100 con 0x7fb47c13a0f0 2023-03-03T14:16:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.035+0000 7fb484651700 1 -- 172.21.15.7:0/2897848437 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb47c136160 con 0x7fb47c13a0f0 2023-03-03T14:16:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.036+0000 7fb46effd700 1 -- 172.21.15.7:0/2897848437 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb478055720 con 0x7fb47c13a0f0 2023-03-03T14:16:42.037 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:16:42.037 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:16:42.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 -- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4600682f0 msgr2=0x7fb46006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:42.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4600682f0 0x7fb46006a7a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fb470009910 tx=0x7fb470008040 comp rx=0 tx=0).stop 2023-03-03T14:16:42.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 -- 172.21.15.7:0/2897848437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c13a0f0 msgr2=0x7fb47c1ed0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:16:42.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c13a0f0 0x7fb47c1ed0e0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fb47800e840 tx=0x7fb47800ec00 comp rx=0 tx=0).stop 2023-03-03T14:16:42.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 -- 172.21.15.7:0/2897848437 shutdown_connections 2023-03-03T14:16:42.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb47c1586c0 0x7fb47c1ed620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:42.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb47c0c1da0 0x7fb47c0c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:42.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4600682f0 0x7fb46006a7a0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:42.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 --2- 172.21.15.7:0/2897848437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb47c13a0f0 0x7fb47c1ed0e0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:16:42.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 -- 172.21.15.7:0/2897848437 >> 172.21.15.7:0/2897848437 conn(0x7fb47c06c110 msgr2=0x7fb47c0c3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:16:42.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 -- 172.21.15.7:0/2897848437 shutdown_connections 2023-03-03T14:16:42.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:16:42.038+0000 7fb484651700 1 -- 172.21.15.7:0/2897848437 wait complete. 2023-03-03T14:16:42.041 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:16:42.051 INFO:tasks.mds_thrash.fs.[b]:waiting for 50 secs before thrashing 2023-03-03T14:16:51.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:16:51.634 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:21.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:21.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:32.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:17:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.188+0000 7fd1956b8700 1 Processor -- start 2023-03-03T14:17:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.188+0000 7fd1956b8700 1 -- start start 2023-03-03T14:17:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.189+0000 7fd1956b8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1900c7e00 0x7fd190154a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.189+0000 7fd1956b8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd190157bb0 con 0x7fd1900c7a30 2023-03-03T14:17:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.189+0000 7fd1956b8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd19006a2f0 con 0x7fd19006b6f0 2023-03-03T14:17:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.189+0000 7fd1956b8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19006ab00 con 0x7fd1900c7e00 2023-03-03T14:17:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.189+0000 7fd18effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd1900c7a30 0x7fd19014e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42014/0 (socket says 172.21.15.7:42014) 2023-03-03T14:17:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.189+0000 7fd18effd700 1 -- 172.21.15.7:0/3526968168 learned_addr learned my addr 172.21.15.7:0/3526968168 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:17:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.190+0000 7fd18e7fc700 1 --2- 172.21.15.7:0/3526968168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1900c7e00 0x7fd190154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.190+0000 7fd18dffb700 1 -- 172.21.15.7:0/3526968168 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989740073 0 0) 0x7fd190157bb0 con 0x7fd1900c7a30 2023-03-03T14:17:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.190+0000 7fd18dffb700 1 -- 172.21.15.7:0/3526968168 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1780035b0 con 0x7fd1900c7a30 2023-03-03T14:17:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.190+0000 7fd18dffb700 1 -- 172.21.15.7:0/3526968168 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964080876 0 0) 0x7fd19006a2f0 con 0x7fd19006b6f0 2023-03-03T14:17:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.190+0000 7fd18e7fc700 1 -- 172.21.15.7:0/3526968168 >> v1:172.21.15.143:6789/0 conn(0x7fd19006b6f0 legacy=0x7fd190154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.190+0000 7fd18e7fc700 1 -- 172.21.15.7:0/3526968168 >> v1:172.21.15.7:6789/0 conn(0x7fd1900c7a30 legacy=0x7fd19014e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.190+0000 7fd18e7fc700 1 -- 172.21.15.7:0/3526968168 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1900bfbb0 con 0x7fd1900c7e00 2023-03-03T14:17:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.190+0000 7fd18e7fc700 1 --2- 172.21.15.7:0/3526968168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1900c7e00 0x7fd190154a10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd184009870 tx=0x7fd184009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcbc37611ec25998 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:17:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.190+0000 7fd18dffb700 1 -- 172.21.15.7:0/3526968168 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1800017f0 con 0x7fd1900c7e00 2023-03-03T14:17:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.191+0000 7fd18dffb700 1 -- 172.21.15.7:0/3526968168 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd184004030 con 0x7fd1900c7e00 2023-03-03T14:17:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.191+0000 7fd18dffb700 1 -- 172.21.15.7:0/3526968168 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1840043a0 con 0x7fd1900c7e00 2023-03-03T14:17:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.191+0000 7fd1956b8700 1 -- 172.21.15.7:0/3526968168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1900c7e00 msgr2=0x7fd190154a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.191+0000 7fd1956b8700 1 --2- 172.21.15.7:0/3526968168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1900c7e00 0x7fd190154a10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd184009870 tx=0x7fd184009b80 comp rx=0 tx=0).stop 2023-03-03T14:17:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.191+0000 7fd1956b8700 1 -- 172.21.15.7:0/3526968168 shutdown_connections 2023-03-03T14:17:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.191+0000 7fd1956b8700 1 --2- 172.21.15.7:0/3526968168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1900c7e00 0x7fd190154a10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.191+0000 7fd1956b8700 1 -- 172.21.15.7:0/3526968168 >> 172.21.15.7:0/3526968168 conn(0x7fd190073d20 msgr2=0x7fd1900716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:17:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.191+0000 7fd1956b8700 1 -- 172.21.15.7:0/3526968168 shutdown_connections 2023-03-03T14:17:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.191+0000 7fd1956b8700 1 -- 172.21.15.7:0/3526968168 wait complete. 2023-03-03T14:17:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.192+0000 7fd1956b8700 1 Processor -- start 2023-03-03T14:17:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.192+0000 7fd1956b8700 1 -- start start 2023-03-03T14:17:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd1956b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1900c7a30 0x7fd19014ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd1956b8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1900c7e00 0x7fd1900af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd1956b8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19014cee0 0x7fd190151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd1956b8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1901529c0 con 0x7fd1900c7a30 2023-03-03T14:17:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd1956b8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1900c46f0 con 0x7fd1900c7e00 2023-03-03T14:17:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd1956b8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1901ecd00 con 0x7fd19014cee0 2023-03-03T14:17:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd18effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1900c7a30 0x7fd19014ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd18effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1900c7a30 0x7fd19014ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48042/0 (socket says 172.21.15.7:48042) 2023-03-03T14:17:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd18effd700 1 -- 172.21.15.7:0/3573474704 learned_addr learned my addr 172.21.15.7:0/3573474704 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:17:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd18e7fc700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1900c7e00 0x7fd1900af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.193+0000 7fd18f7fe700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19014cee0 0x7fd190151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd18effd700 1 -- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19014cee0 msgr2=0x7fd190151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd18effd700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19014cee0 0x7fd190151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd18effd700 1 -- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1900c7e00 msgr2=0x7fd1900af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd18effd700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1900c7e00 0x7fd1900af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd18effd700 1 -- 172.21.15.7:0/3573474704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1840094a0 con 0x7fd1900c7a30 2023-03-03T14:17:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd18effd700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1900c7a30 0x7fd19014ec50 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fd180001920 tx=0x7fd18000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:17:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd1777fe700 1 -- 172.21.15.7:0/3573474704 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1800074e0 con 0x7fd1900c7a30 2023-03-03T14:17:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd1777fe700 1 -- 172.21.15.7:0/3573474704 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd180007660 con 0x7fd1900c7a30 2023-03-03T14:17:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd1956b8700 1 -- 172.21.15.7:0/3573474704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19006a820 con 0x7fd1900c7a30 2023-03-03T14:17:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd1777fe700 1 -- 172.21.15.7:0/3573474704 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180005a70 con 0x7fd1900c7a30 2023-03-03T14:17:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.194+0000 7fd1956b8700 1 -- 172.21.15.7:0/3573474704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19006a540 con 0x7fd1900c7a30 2023-03-03T14:17:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.195+0000 7fd1956b8700 1 -- 172.21.15.7:0/3573474704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19014e870 con 0x7fd1900c7a30 2023-03-03T14:17:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.195+0000 7fd1777fe700 1 -- 172.21.15.7:0/3573474704 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd180019030 con 0x7fd1900c7a30 2023-03-03T14:17:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.196+0000 7fd1777fe700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd17c0682f0 0x7fd17c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.196+0000 7fd1777fe700 1 -- 172.21.15.7:0/3573474704 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd180085620 con 0x7fd1900c7a30 2023-03-03T14:17:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.196+0000 7fd18e7fc700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd17c0682f0 0x7fd17c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.197+0000 7fd18e7fc700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd17c0682f0 0x7fd17c06a7a0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd1840097c0 tx=0x7fd184008000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:17:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.200+0000 7fd1777fe700 1 -- 172.21.15.7:0/3573474704 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd180051940 con 0x7fd1900c7a30 2023-03-03T14:17:32.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.437+0000 7fd1956b8700 1 -- 172.21.15.7:0/3573474704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd190074bb0 con 0x7fd1900c7a30 2023-03-03T14:17:32.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.437+0000 7fd1777fe700 1 -- 172.21.15.7:0/3573474704 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd180054f60 con 0x7fd1900c7a30 2023-03-03T14:17:32.440 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:17:32.440 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:17:32.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 -- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd17c0682f0 msgr2=0x7fd17c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:32.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd17c0682f0 0x7fd17c06a7a0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fd1840097c0 tx=0x7fd184008000 comp rx=0 tx=0).stop 2023-03-03T14:17:32.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 -- 172.21.15.7:0/3573474704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1900c7a30 msgr2=0x7fd19014ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:32.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1900c7a30 0x7fd19014ec50 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fd180001920 tx=0x7fd18000ca30 comp rx=0 tx=0).stop 2023-03-03T14:17:32.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 -- 172.21.15.7:0/3573474704 shutdown_connections 2023-03-03T14:17:32.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd19014cee0 0x7fd190151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd17c0682f0 0x7fd17c06a7a0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1900c7a30 0x7fd19014ec50 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 --2- 172.21.15.7:0/3573474704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1900c7e00 0x7fd1900af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 -- 172.21.15.7:0/3573474704 >> 172.21.15.7:0/3573474704 conn(0x7fd19006b6f0 msgr2=0x7fd190154a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:17:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 -- 172.21.15.7:0/3573474704 shutdown_connections 2023-03-03T14:17:32.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:32.439+0000 7fd1956b8700 1 -- 172.21.15.7:0/3573474704 wait complete. 2023-03-03T14:17:32.444 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:17:32.453 INFO:tasks.mds_thrash.fs.[b]:waiting for 51 secs before thrashing 2023-03-03T14:17:39.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:17:39.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.921+0000 7f7103cfb700 1 Processor -- start 2023-03-03T14:17:39.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.921+0000 7f7103cfb700 1 -- start start 2023-03-03T14:17:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7103cfb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc151b20 0x7f70fc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7103cfb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70fc06cb70 con 0x7f70fc073c30 2023-03-03T14:17:39.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7103cfb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70fc06ce50 con 0x7f70fc0c2950 2023-03-03T14:17:39.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7103cfb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc06c2f0 con 0x7f70fc151b20 2023-03-03T14:17:39.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7102298700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f70fc073c30 0x7f70fc1581f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42032/0 (socket says 172.21.15.7:42032) 2023-03-03T14:17:39.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7102298700 1 -- 172.21.15.7:0/694725881 learned_addr learned my addr 172.21.15.7:0/694725881 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:17:39.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7101a97700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc151b20 0x7f70fc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:39.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748456250 0 0) 0x7f70fc06cb70 con 0x7f70fc073c30 2023-03-03T14:17:39.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70e80035b0 con 0x7f70fc073c30 2023-03-03T14:17:39.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2285626021 0 0) 0x7f70e80035b0 con 0x7f70fc073c30 2023-03-03T14:17:39.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc151b20 msgr2=0x7f70fc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:17:39.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7100a95700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc151b20 0x7f70fc0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:39.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.922+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 >> v1:172.21.15.143:6789/0 conn(0x7f70fc0c2950 legacy=0x7f70fc157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:39.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc06d130 con 0x7f70fc073c30 2023-03-03T14:17:39.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f70f40019f0 con 0x7f70fc073c30 2023-03-03T14:17:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 >> v1:172.21.15.7:6789/0 conn(0x7f70fc073c30 legacy=0x7f70fc1581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 0x7f70e8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e8004850 0x7f70e8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc151b20 0x7f70e80052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc06c2f0 con 0x7f70e8003aa0 2023-03-03T14:17:39.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e800bba0 con 0x7f70e8004850 2023-03-03T14:17:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e800bd60 con 0x7f70fc151b20 2023-03-03T14:17:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7101a97700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 0x7f70e8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7101296700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e8004850 0x7f70e8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7102298700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc151b20 0x7f70e80052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7101a97700 1 -- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc151b20 msgr2=0x7f70e80052b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:17:39.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7101a97700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc151b20 0x7f70e80052b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:39.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7101a97700 1 -- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e8004850 msgr2=0x7f70e8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:17:39.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7101a97700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70e8004850 0x7f70e8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:39.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7101a97700 1 -- 172.21.15.7:0/694725881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e80035b0 con 0x7f70e8003aa0 2023-03-03T14:17:39.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7101a97700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 0x7f70e8003e70 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f70ec009b30 tx=0x7f70e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c8aa7e9341c106 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:17:39.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec00e6c0 con 0x7f70e8003aa0 2023-03-03T14:17:39.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec00e820 con 0x7f70e8003aa0 2023-03-03T14:17:39.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.923+0000 7f7100a95700 1 -- 172.21.15.7:0/694725881 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec00eb90 con 0x7f70e8003aa0 2023-03-03T14:17:39.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.924+0000 7f7103cfb700 1 -- 172.21.15.7:0/694725881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 msgr2=0x7f70e8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:39.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.924+0000 7f7103cfb700 1 --2- 172.21.15.7:0/694725881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 0x7f70e8003e70 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f70ec009b30 tx=0x7f70e8005880 comp rx=0 tx=0).stop 2023-03-03T14:17:39.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.924+0000 7f7102298700 1 -- 172.21.15.7:0/694725881 reap_dead start 2023-03-03T14:17:39.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.924+0000 7f7103cfb700 1 -- 172.21.15.7:0/694725881 shutdown_connections 2023-03-03T14:17:39.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.924+0000 7f7103cfb700 1 -- 172.21.15.7:0/694725881 >> 172.21.15.7:0/694725881 conn(0x7f70fc0a94d0 msgr2=0x7f70fc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:17:39.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.924+0000 7f7103cfb700 1 -- 172.21.15.7:0/694725881 shutdown_connections 2023-03-03T14:17:39.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.924+0000 7f7103cfb700 1 -- 172.21.15.7:0/694725881 wait complete. 2023-03-03T14:17:39.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.925+0000 7f7103cfb700 1 Processor -- start 2023-03-03T14:17:39.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.925+0000 7f7103cfb700 1 -- start start 2023-03-03T14:17:39.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7103cfb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 0x7f70fc0c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:39.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7103cfb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70fc1ed3a0 0x7f70fc1ed810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:39.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7103cfb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc156930 0x7f70fc156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:39.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7103cfb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc0bece0 con 0x7f70e8003aa0 2023-03-03T14:17:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7103cfb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc1edd50 con 0x7f70fc1ed3a0 2023-03-03T14:17:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7103cfb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc151400 con 0x7f70fc156930 2023-03-03T14:17:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7101a97700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 0x7f70fc0c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7101a97700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 0x7f70fc0c6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48060/0 (socket says 172.21.15.7:48060) 2023-03-03T14:17:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7101a97700 1 -- 172.21.15.7:0/1275992038 learned_addr learned my addr 172.21.15.7:0/1275992038 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:17:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7101296700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70fc1ed3a0 0x7f70fc1ed810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:39.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7101a97700 1 -- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc156930 msgr2=0x7f70fc156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.926+0000 7f7102298700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc156930 0x7f70fc156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f7101a97700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc156930 0x7f70fc156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f7101a97700 1 -- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70fc1ed3a0 msgr2=0x7f70fc1ed810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f7101a97700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70fc1ed3a0 0x7f70fc1ed810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f7101a97700 1 -- 172.21.15.7:0/1275992038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc151ef0 con 0x7f70e8003aa0 2023-03-03T14:17:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f7101296700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70fc1ed3a0 0x7f70fc1ed810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:17:39.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f7102298700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc156930 0x7f70fc156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:17:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f7101a97700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 0x7f70fc0c6550 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f70e8005830 tx=0x7f70ec0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:17:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f70f27fc700 1 -- 172.21.15.7:0/1275992038 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec021070 con 0x7f70e8003aa0 2023-03-03T14:17:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f70f27fc700 1 -- 172.21.15.7:0/1275992038 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec0045d0 con 0x7f70e8003aa0 2023-03-03T14:17:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f7103cfb700 1 -- 172.21.15.7:0/1275992038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70fc06d410 con 0x7f70e8003aa0 2023-03-03T14:17:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f70f27fc700 1 -- 172.21.15.7:0/1275992038 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec00e6c0 con 0x7f70e8003aa0 2023-03-03T14:17:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.927+0000 7f7103cfb700 1 -- 172.21.15.7:0/1275992038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70fc06bea0 con 0x7f70e8003aa0 2023-03-03T14:17:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.928+0000 7f7103cfb700 1 -- 172.21.15.7:0/1275992038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70fc04e6e0 con 0x7f70e8003aa0 2023-03-03T14:17:39.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.928+0000 7f70f27fc700 1 -- 172.21.15.7:0/1275992038 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f70ec02e030 con 0x7f70e8003aa0 2023-03-03T14:17:39.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.929+0000 7f70f27fc700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70e00682f0 0x7f70e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:17:39.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.929+0000 7f70f27fc700 1 -- 172.21.15.7:0/1275992038 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f70ec087340 con 0x7f70e8003aa0 2023-03-03T14:17:39.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.929+0000 7f7101296700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70e00682f0 0x7f70e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:17:39.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.930+0000 7f7101296700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70e00682f0 0x7f70e006a7a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f70fc1579d0 tx=0x7f70f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:17:39.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:39.933+0000 7f70f27fc700 1 -- 172.21.15.7:0/1275992038 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70ec0536e0 con 0x7f70e8003aa0 2023-03-03T14:17:40.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.170+0000 7f7103cfb700 1 -- 172.21.15.7:0/1275992038 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70fc074bb0 con 0x7f70e8003aa0 2023-03-03T14:17:40.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.170+0000 7f70f27fc700 1 -- 172.21.15.7:0/1275992038 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f70ec056d00 con 0x7f70e8003aa0 2023-03-03T14:17:40.173 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:17:40.173 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:17:40.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 -- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70e00682f0 msgr2=0x7f70e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:40.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70e00682f0 0x7f70e006a7a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f70fc1579d0 tx=0x7f70f8009040 comp rx=0 tx=0).stop 2023-03-03T14:17:40.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 -- 172.21.15.7:0/1275992038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 msgr2=0x7f70fc0c6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:17:40.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 0x7f70fc0c6550 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f70e8005830 tx=0x7f70ec0047c0 comp rx=0 tx=0).stop 2023-03-03T14:17:40.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 -- 172.21.15.7:0/1275992038 shutdown_connections 2023-03-03T14:17:40.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70fc156930 0x7f70fc156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:40.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70e00682f0 0x7f70e006a7a0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:40.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70e8003aa0 0x7f70fc0c6550 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:40.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 --2- 172.21.15.7:0/1275992038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70fc1ed3a0 0x7f70fc1ed810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:17:40.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 -- 172.21.15.7:0/1275992038 >> 172.21.15.7:0/1275992038 conn(0x7f70fc0a94d0 msgr2=0x7f70fc071200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:17:40.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.172+0000 7f7103cfb700 1 -- 172.21.15.7:0/1275992038 shutdown_connections 2023-03-03T14:17:40.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:17:40.173+0000 7f7103cfb700 1 -- 172.21.15.7:0/1275992038 wait complete. 2023-03-03T14:17:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:17:40.186 INFO:tasks.mds_thrash.fs.[a]:waiting for 33 secs before thrashing 2023-03-03T14:17:51.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:17:51.862 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:13.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:18:13.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.326+0000 7fda2aaea700 1 Processor -- start 2023-03-03T14:18:13.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.326+0000 7fda2aaea700 1 -- start start 2023-03-03T14:18:13.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.326+0000 7fda2aaea700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda240c00e0 0x7fda240c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:13.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.326+0000 7fda2aaea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda24154ff0 0x7fda240cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:13.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.326+0000 7fda2aaea700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda240c7310 con 0x7fda240caa30 2023-03-03T14:18:13.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.326+0000 7fda2aaea700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda24155430 con 0x7fda24154ff0 2023-03-03T14:18:13.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.326+0000 7fda2aaea700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda240bfd10 con 0x7fda240c00e0 2023-03-03T14:18:13.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda29087700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fda24154ff0 0x7fda240cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:13.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda28886700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fda240caa30 0x7fda240c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42070/0 (socket says 172.21.15.7:42070) 2023-03-03T14:18:13.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda28886700 1 -- 172.21.15.7:0/936661356 learned_addr learned my addr 172.21.15.7:0/936661356 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:18:13.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda23fff700 1 --2- 172.21.15.7:0/936661356 >> v2:172.21.15.143:3300/0 conn(0x7fda240c00e0 0x7fda240c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:13.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda29087700 1 -- 172.21.15.7:0/936661356 >> v2:172.21.15.143:3300/0 conn(0x7fda240c00e0 msgr2=0x7fda240c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:18:13.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda29087700 1 --2- 172.21.15.7:0/936661356 >> v2:172.21.15.143:3300/0 conn(0x7fda240c00e0 0x7fda240c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:13.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda29087700 1 -- 172.21.15.7:0/936661356 >> v1:172.21.15.7:6789/0 conn(0x7fda240caa30 legacy=0x7fda240c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:13.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda29087700 1 -- 172.21.15.7:0/936661356 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda240bfa30 con 0x7fda24154ff0 2023-03-03T14:18:13.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda237fe700 1 -- 172.21.15.7:0/936661356 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060276727 0 0) 0x7fda240c7310 con 0x7fda240caa30 2023-03-03T14:18:13.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda29087700 1 --2- 172.21.15.7:0/936661356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24154ff0 0x7fda240cae00 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fda240ca910 tx=0x7fda1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26be748ba8b1bddd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:13.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda237fe700 1 -- 172.21.15.7:0/936661356 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c00e070 con 0x7fda24154ff0 2023-03-03T14:18:13.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda237fe700 1 -- 172.21.15.7:0/936661356 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c00bbd0 con 0x7fda24154ff0 2023-03-03T14:18:13.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.327+0000 7fda237fe700 1 -- 172.21.15.7:0/936661356 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c012430 con 0x7fda24154ff0 2023-03-03T14:18:13.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.328+0000 7fda2aaea700 1 -- 172.21.15.7:0/936661356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24154ff0 msgr2=0x7fda240cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:13.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.328+0000 7fda2aaea700 1 --2- 172.21.15.7:0/936661356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24154ff0 0x7fda240cae00 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fda240ca910 tx=0x7fda1c00b330 comp rx=0 tx=0).stop 2023-03-03T14:18:13.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.328+0000 7fda2aaea700 1 -- 172.21.15.7:0/936661356 shutdown_connections 2023-03-03T14:18:13.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.328+0000 7fda2aaea700 1 --2- 172.21.15.7:0/936661356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24154ff0 0x7fda240cae00 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:13.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.328+0000 7fda2aaea700 1 --2- 172.21.15.7:0/936661356 >> v2:172.21.15.143:3300/0 conn(0x7fda240c00e0 0x7fda240c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:13.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.328+0000 7fda2aaea700 1 -- 172.21.15.7:0/936661356 >> 172.21.15.7:0/936661356 conn(0x7fda24066e40 msgr2=0x7fda24071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:18:13.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.328+0000 7fda2aaea700 1 -- 172.21.15.7:0/936661356 shutdown_connections 2023-03-03T14:18:13.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.328+0000 7fda2aaea700 1 -- 172.21.15.7:0/936661356 wait complete. 2023-03-03T14:18:13.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.329+0000 7fda2aaea700 1 Processor -- start 2023-03-03T14:18:13.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.329+0000 7fda2aaea700 1 -- start start 2023-03-03T14:18:13.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda2aaea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda240c00e0 0x7fda24142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:13.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda2aaea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda240caa30 0x7fda24157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:13.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda2aaea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda24154ff0 0x7fda24156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:13.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda2aaea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda240aa940 con 0x7fda240c00e0 2023-03-03T14:18:13.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda2aaea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda24158420 con 0x7fda240caa30 2023-03-03T14:18:13.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda2aaea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda241eda10 con 0x7fda24154ff0 2023-03-03T14:18:13.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda28886700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda240c00e0 0x7fda24142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:13.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda28886700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda240c00e0 0x7fda24142e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48096/0 (socket says 172.21.15.7:48096) 2023-03-03T14:18:13.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda28886700 1 -- 172.21.15.7:0/331538809 learned_addr learned my addr 172.21.15.7:0/331538809 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:18:13.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda28886700 1 -- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda24154ff0 msgr2=0x7fda24156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda28886700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda24154ff0 0x7fda24156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda28886700 1 -- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda240caa30 msgr2=0x7fda24157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda23fff700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda240caa30 0x7fda24157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda28886700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda240caa30 0x7fda24157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda28886700 1 -- 172.21.15.7:0/331538809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c009a70 con 0x7fda240c00e0 2023-03-03T14:18:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.330+0000 7fda23fff700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda240caa30 0x7fda24157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:18:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.331+0000 7fda28886700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda240c00e0 0x7fda24142e20 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fda1800c750 tx=0x7fda1800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:13.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.331+0000 7fda217fa700 1 -- 172.21.15.7:0/331538809 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18004c80 con 0x7fda240c00e0 2023-03-03T14:18:13.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.331+0000 7fda217fa700 1 -- 172.21.15.7:0/331538809 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda18004e00 con 0x7fda240c00e0 2023-03-03T14:18:13.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.331+0000 7fda2aaea700 1 -- 172.21.15.7:0/331538809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda241547f0 con 0x7fda240c00e0 2023-03-03T14:18:13.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.331+0000 7fda217fa700 1 -- 172.21.15.7:0/331538809 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18007310 con 0x7fda240c00e0 2023-03-03T14:18:13.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.331+0000 7fda2aaea700 1 -- 172.21.15.7:0/331538809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda24154510 con 0x7fda240c00e0 2023-03-03T14:18:13.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.332+0000 7fda2aaea700 1 -- 172.21.15.7:0/331538809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda24154ad0 con 0x7fda240c00e0 2023-03-03T14:18:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.332+0000 7fda217fa700 1 -- 172.21.15.7:0/331538809 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda1801a020 con 0x7fda240c00e0 2023-03-03T14:18:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.333+0000 7fda217fa700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda080682f0 0x7fda0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.333+0000 7fda217fa700 1 -- 172.21.15.7:0/331538809 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fda18085d50 con 0x7fda240c00e0 2023-03-03T14:18:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.333+0000 7fda23fff700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda080682f0 0x7fda0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:13.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.334+0000 7fda23fff700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda080682f0 0x7fda0806a7a0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fda240c00b0 tx=0x7fda10008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:13.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.337+0000 7fda217fa700 1 -- 172.21.15.7:0/331538809 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda180520f0 con 0x7fda240c00e0 2023-03-03T14:18:13.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.574+0000 7fda2aaea700 1 -- 172.21.15.7:0/331538809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda24156110 con 0x7fda240c00e0 2023-03-03T14:18:13.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.575+0000 7fda217fa700 1 -- 172.21.15.7:0/331538809 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fda18055710 con 0x7fda240c00e0 2023-03-03T14:18:13.576 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:18:13.577 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:18:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 -- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda080682f0 msgr2=0x7fda0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda080682f0 0x7fda0806a7a0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fda240c00b0 tx=0x7fda10008040 comp rx=0 tx=0).stop 2023-03-03T14:18:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 -- 172.21.15.7:0/331538809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda240c00e0 msgr2=0x7fda24142e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda240c00e0 0x7fda24142e20 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fda1800c750 tx=0x7fda1800cb10 comp rx=0 tx=0).stop 2023-03-03T14:18:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 -- 172.21.15.7:0/331538809 shutdown_connections 2023-03-03T14:18:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda24154ff0 0x7fda24156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda080682f0 0x7fda0806a7a0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda240c00e0 0x7fda24142e20 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 --2- 172.21.15.7:0/331538809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda240caa30 0x7fda24157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 -- 172.21.15.7:0/331538809 >> 172.21.15.7:0/331538809 conn(0x7fda24066e40 msgr2=0x7fda24153b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:18:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 -- 172.21.15.7:0/331538809 shutdown_connections 2023-03-03T14:18:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:13.577+0000 7fda2aaea700 1 -- 172.21.15.7:0/331538809 wait complete. 2023-03-03T14:18:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:18:13.590 INFO:tasks.mds_thrash.fs.[a]:waiting for 111 secs before thrashing 2023-03-03T14:18:21.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:21.922 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:23.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:18:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.593+0000 7f449f263700 1 Processor -- start 2023-03-03T14:18:23.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.593+0000 7f449f263700 1 -- start start 2023-03-03T14:18:23.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.594+0000 7f449f263700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4498158120 0x7f449806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:23.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.594+0000 7f449f263700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44980bc260 con 0x7f44980abf00 2023-03-03T14:18:23.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.594+0000 7f449f263700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4498155440 con 0x7f4498155070 2023-03-03T14:18:23.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.594+0000 7f449f263700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449806da20 con 0x7f4498158120 2023-03-03T14:18:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.594+0000 7f449d800700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4498158120 0x7f449806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.594+0000 7f449d800700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4498158120 0x7f449806d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48116/0 (socket says 172.21.15.7:48116) 2023-03-03T14:18:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.594+0000 7f449d800700 1 -- 172.21.15.7:0/3313293065 learned_addr learned my addr 172.21.15.7:0/3313293065 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:18:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.594+0000 7f4497fff700 1 --1- 172.21.15.7:0/3313293065 >> v1:172.21.15.7:6789/0 conn(0x7f44980abf00 0x7f449806cdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42092/0 (socket says 172.21.15.7:42092) 2023-03-03T14:18:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.594+0000 7f449d800700 1 -- 172.21.15.7:0/3313293065 >> v1:172.21.15.143:6789/0 conn(0x7f4498155070 legacy=0x7f4498157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.595+0000 7f449d800700 1 -- 172.21.15.7:0/3313293065 >> v1:172.21.15.7:6789/0 conn(0x7f44980abf00 legacy=0x7f449806cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.595+0000 7f449d800700 1 -- 172.21.15.7:0/3313293065 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4498158950 con 0x7f4498158120 2023-03-03T14:18:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.595+0000 7f449d800700 1 --2- 172.21.15.7:0/3313293065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4498158120 0x7f449806d4e0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f4490009e50 tx=0x7f449000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d284a3debdbe9c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.595+0000 7f44977fe700 1 -- 172.21.15.7:0/3313293065 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449000e070 con 0x7f4498158120 2023-03-03T14:18:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.595+0000 7f44977fe700 1 -- 172.21.15.7:0/3313293065 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449000b9c0 con 0x7f4498158120 2023-03-03T14:18:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.595+0000 7f44977fe700 1 -- 172.21.15.7:0/3313293065 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449000bd70 con 0x7f4498158120 2023-03-03T14:18:23.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.595+0000 7f449f263700 1 -- 172.21.15.7:0/3313293065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4498158120 msgr2=0x7f449806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:23.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.595+0000 7f449f263700 1 --2- 172.21.15.7:0/3313293065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4498158120 0x7f449806d4e0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f4490009e50 tx=0x7f449000b330 comp rx=0 tx=0).stop 2023-03-03T14:18:23.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.596+0000 7f449f263700 1 -- 172.21.15.7:0/3313293065 shutdown_connections 2023-03-03T14:18:23.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.596+0000 7f449f263700 1 --2- 172.21.15.7:0/3313293065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4498158120 0x7f449806d4e0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:23.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.596+0000 7f449f263700 1 -- 172.21.15.7:0/3313293065 >> 172.21.15.7:0/3313293065 conn(0x7f44980c6750 msgr2=0x7f44980c6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:18:23.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.596+0000 7f449f263700 1 -- 172.21.15.7:0/3313293065 shutdown_connections 2023-03-03T14:18:23.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.596+0000 7f449f263700 1 -- 172.21.15.7:0/3313293065 wait complete. 2023-03-03T14:18:23.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.597+0000 7f449f263700 1 Processor -- start 2023-03-03T14:18:23.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.597+0000 7f449f263700 1 -- start start 2023-03-03T14:18:23.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449f263700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980abf00 0x7f449814f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:23.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449f263700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4498155070 0x7f44981e9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449f263700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4498158120 0x7f449806fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449f263700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449814f970 con 0x7f44980abf00 2023-03-03T14:18:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449f263700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44980bdc30 con 0x7f4498158120 2023-03-03T14:18:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449f263700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44981edf30 con 0x7f4498155070 2023-03-03T14:18:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449cfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980abf00 0x7f449814f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449cfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980abf00 0x7f449814f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48118/0 (socket says 172.21.15.7:48118) 2023-03-03T14:18:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449cfff700 1 -- 172.21.15.7:0/3519893643 learned_addr learned my addr 172.21.15.7:0/3519893643 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:18:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449d800700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4498158120 0x7f449806fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f4497fff700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4498155070 0x7f44981e9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449cfff700 1 -- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4498155070 msgr2=0x7f44981e9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449cfff700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4498155070 0x7f44981e9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449cfff700 1 -- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4498158120 msgr2=0x7f449806fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449cfff700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4498158120 0x7f449806fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449cfff700 1 -- 172.21.15.7:0/3519893643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4490009b30 con 0x7f44980abf00 2023-03-03T14:18:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449d800700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4498158120 0x7f449806fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:18:23.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.598+0000 7f449cfff700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980abf00 0x7f449814f430 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f448c00c4e0 tx=0x7f448c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:23.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.599+0000 7f44957fa700 1 -- 172.21.15.7:0/3519893643 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c0052d0 con 0x7f44980abf00 2023-03-03T14:18:23.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.599+0000 7f44957fa700 1 -- 172.21.15.7:0/3519893643 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448c005430 con 0x7f44980abf00 2023-03-03T14:18:23.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.599+0000 7f449f263700 1 -- 172.21.15.7:0/3519893643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44980bbef0 con 0x7f44980abf00 2023-03-03T14:18:23.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.599+0000 7f44957fa700 1 -- 172.21.15.7:0/3519893643 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c0038c0 con 0x7f44980abf00 2023-03-03T14:18:23.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.599+0000 7f449f263700 1 -- 172.21.15.7:0/3519893643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4498158670 con 0x7f44980abf00 2023-03-03T14:18:23.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.600+0000 7f449f263700 1 -- 172.21.15.7:0/3519893643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44980c1670 con 0x7f44980abf00 2023-03-03T14:18:23.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.600+0000 7f44957fa700 1 -- 172.21.15.7:0/3519893643 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f448c019030 con 0x7f44980abf00 2023-03-03T14:18:23.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.600+0000 7f44957fa700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f447c0682f0 0x7f447c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:18:23.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.600+0000 7f44957fa700 1 -- 172.21.15.7:0/3519893643 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f448c085540 con 0x7f44980abf00 2023-03-03T14:18:23.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.601+0000 7f4497fff700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f447c0682f0 0x7f447c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:18:23.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.601+0000 7f4497fff700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f447c0682f0 0x7f447c06a7a0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f4484000f60 tx=0x7f4484009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:18:23.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.605+0000 7f44957fa700 1 -- 172.21.15.7:0/3519893643 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f448c051860 con 0x7f44980abf00 2023-03-03T14:18:23.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.844+0000 7f449f263700 1 -- 172.21.15.7:0/3519893643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4498074bb0 con 0x7f44980abf00 2023-03-03T14:18:23.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.844+0000 7f44957fa700 1 -- 172.21.15.7:0/3519893643 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f448c054e80 con 0x7f44980abf00 2023-03-03T14:18:23.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:18:23.846 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:18:23.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 -- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f447c0682f0 msgr2=0x7f447c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:23.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f447c0682f0 0x7f447c06a7a0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f4484000f60 tx=0x7f4484009040 comp rx=0 tx=0).stop 2023-03-03T14:18:23.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 -- 172.21.15.7:0/3519893643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980abf00 msgr2=0x7f449814f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:18:23.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980abf00 0x7f449814f430 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f448c00c4e0 tx=0x7f448c00c8a0 comp rx=0 tx=0).stop 2023-03-03T14:18:23.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 -- 172.21.15.7:0/3519893643 shutdown_connections 2023-03-03T14:18:23.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4498158120 0x7f449806fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:23.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f447c0682f0 0x7f447c06a7a0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:23.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44980abf00 0x7f449814f430 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:23.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 --2- 172.21.15.7:0/3519893643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4498155070 0x7f44981e9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:18:23.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 -- 172.21.15.7:0/3519893643 >> 172.21.15.7:0/3519893643 conn(0x7f44980c6750 msgr2=0x7f449806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:18:23.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 -- 172.21.15.7:0/3519893643 shutdown_connections 2023-03-03T14:18:23.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:18:23.846+0000 7f449f263700 1 -- 172.21.15.7:0/3519893643 wait complete. 2023-03-03T14:18:23.850 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:18:23.860 INFO:tasks.mds_thrash.fs.[b]:waiting for 88 secs before thrashing 2023-03-03T14:18:52.034 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:18:52.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:22.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:22.203 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:51.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:19:52.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:51.999+0000 7f4b9712d700 1 Processor -- start 2023-03-03T14:19:52.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:51.999+0000 7f4b9712d700 1 -- start start 2023-03-03T14:19:52.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b9712d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4b90151770 0x7f4b90151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:52.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b9712d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b90154910 con 0x7f4b90155390 2023-03-03T14:19:52.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b9712d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b900c4540 con 0x7f4b9006e020 2023-03-03T14:19:52.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b9712d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b90070b90 con 0x7f4b90151770 2023-03-03T14:19:52.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b956ca700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4b90151770 0x7f4b90151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:52.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b94ec9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4b90155390 0x7f4b900c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42178/0 (socket says 172.21.15.7:42178) 2023-03-03T14:19:52.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b94ec9700 1 -- 172.21.15.7:0/3948211051 learned_addr learned my addr 172.21.15.7:0/3948211051 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:19:52.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b956ca700 1 -- 172.21.15.7:0/3948211051 >> v1:172.21.15.143:6789/0 conn(0x7f4b9006e020 legacy=0x7f4b90071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:19:52.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b956ca700 1 -- 172.21.15.7:0/3948211051 >> v1:172.21.15.7:6789/0 conn(0x7f4b90155390 legacy=0x7f4b900c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:52.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b956ca700 1 -- 172.21.15.7:0/3948211051 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b900702a0 con 0x7f4b90151770 2023-03-03T14:19:52.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b8f7fe700 1 -- 172.21.15.7:0/3948211051 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231757352 0 0) 0x7f4b90154910 con 0x7f4b90155390 2023-03-03T14:19:52.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.000+0000 7f4b956ca700 1 --2- 172.21.15.7:0/3948211051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90151770 0x7f4b90151bd0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009d90 tx=0x7f4b8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59c5d6963037902 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:19:52.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.001+0000 7f4b8f7fe700 1 -- 172.21.15.7:0/3948211051 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8800e070 con 0x7f4b90151770 2023-03-03T14:19:52.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.001+0000 7f4b8f7fe700 1 -- 172.21.15.7:0/3948211051 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8800bbd0 con 0x7f4b90151770 2023-03-03T14:19:52.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.001+0000 7f4b8f7fe700 1 -- 172.21.15.7:0/3948211051 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88012430 con 0x7f4b90151770 2023-03-03T14:19:52.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.001+0000 7f4b9712d700 1 -- 172.21.15.7:0/3948211051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90151770 msgr2=0x7f4b90151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:52.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.001+0000 7f4b9712d700 1 --2- 172.21.15.7:0/3948211051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90151770 0x7f4b90151bd0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009d90 tx=0x7f4b8800b330 comp rx=0 tx=0).stop 2023-03-03T14:19:52.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.001+0000 7f4b9712d700 1 -- 172.21.15.7:0/3948211051 shutdown_connections 2023-03-03T14:19:52.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.001+0000 7f4b9712d700 1 --2- 172.21.15.7:0/3948211051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90151770 0x7f4b90151bd0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:52.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.001+0000 7f4b9712d700 1 -- 172.21.15.7:0/3948211051 >> 172.21.15.7:0/3948211051 conn(0x7f4b9006c110 msgr2=0x7f4b9006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:19:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.001+0000 7f4b9712d700 1 -- 172.21.15.7:0/3948211051 shutdown_connections 2023-03-03T14:19:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.001+0000 7f4b9712d700 1 -- 172.21.15.7:0/3948211051 wait complete. 2023-03-03T14:19:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.002+0000 7f4b9712d700 1 Processor -- start 2023-03-03T14:19:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.002+0000 7f4b9712d700 1 -- start start 2023-03-03T14:19:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.003+0000 7f4b9712d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9006e020 0x7f4b900c10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.003+0000 7f4b9712d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b90151770 0x7f4b901540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.003+0000 7f4b9712d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90155390 0x7f4b901eb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.003+0000 7f4b9712d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b90152bd0 con 0x7f4b90155390 2023-03-03T14:19:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.003+0000 7f4b9712d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b90152eb0 con 0x7f4b9006e020 2023-03-03T14:19:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.003+0000 7f4b9712d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b90153190 con 0x7f4b90151770 2023-03-03T14:19:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.003+0000 7f4b956ca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90155390 0x7f4b901eb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.003+0000 7f4b956ca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90155390 0x7f4b901eb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48208/0 (socket says 172.21.15.7:48208) 2023-03-03T14:19:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b956ca700 1 -- 172.21.15.7:0/2089903652 learned_addr learned my addr 172.21.15.7:0/2089903652 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:19:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b94ec9700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9006e020 0x7f4b900c10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b8ffff700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b90151770 0x7f4b901540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b956ca700 1 -- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b90151770 msgr2=0x7f4b901540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b956ca700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b90151770 0x7f4b901540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b956ca700 1 -- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9006e020 msgr2=0x7f4b900c10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b956ca700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9006e020 0x7f4b900c10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b956ca700 1 -- 172.21.15.7:0/2089903652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b88009a70 con 0x7f4b90155390 2023-03-03T14:19:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b956ca700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90155390 0x7f4b901eb390 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f4b88014040 tx=0x7f4b88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:19:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b8d7fa700 1 -- 172.21.15.7:0/2089903652 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8801b810 con 0x7f4b90155390 2023-03-03T14:19:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b8d7fa700 1 -- 172.21.15.7:0/2089903652 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b88020070 con 0x7f4b90155390 2023-03-03T14:19:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b9712d700 1 -- 172.21.15.7:0/2089903652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b90070820 con 0x7f4b90155390 2023-03-03T14:19:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b8d7fa700 1 -- 172.21.15.7:0/2089903652 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8801bbd0 con 0x7f4b90155390 2023-03-03T14:19:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.004+0000 7f4b9712d700 1 -- 172.21.15.7:0/2089903652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b90070540 con 0x7f4b90155390 2023-03-03T14:19:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.005+0000 7f4b9712d700 1 -- 172.21.15.7:0/2089903652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b90155760 con 0x7f4b90155390 2023-03-03T14:19:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.006+0000 7f4b8d7fa700 1 -- 172.21.15.7:0/2089903652 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4b8801bd30 con 0x7f4b90155390 2023-03-03T14:19:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.006+0000 7f4b8d7fa700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b740682f0 0x7f4b7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:19:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.006+0000 7f4b8d7fa700 1 -- 172.21.15.7:0/2089903652 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4b88022070 con 0x7f4b90155390 2023-03-03T14:19:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.007+0000 7f4b94ec9700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b740682f0 0x7f4b7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:19:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.007+0000 7f4b94ec9700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b740682f0 0x7f4b7406a7a0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f4b84000f60 tx=0x7f4b84004a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:19:52.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.011+0000 7f4b8d7fa700 1 -- 172.21.15.7:0/2089903652 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4b88054c40 con 0x7f4b90155390 2023-03-03T14:19:52.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.249+0000 7f4b9712d700 1 -- 172.21.15.7:0/2089903652 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b90074bb0 con 0x7f4b90155390 2023-03-03T14:19:52.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.250+0000 7f4b8d7fa700 1 -- 172.21.15.7:0/2089903652 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4b88058260 con 0x7f4b90155390 2023-03-03T14:19:52.253 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:19:52.253 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:19:52.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 -- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b740682f0 msgr2=0x7f4b7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:52.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b740682f0 0x7f4b7406a7a0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f4b84000f60 tx=0x7f4b84004a50 comp rx=0 tx=0).stop 2023-03-03T14:19:52.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 -- 172.21.15.7:0/2089903652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90155390 msgr2=0x7f4b901eb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:19:52.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90155390 0x7f4b901eb390 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f4b88014040 tx=0x7f4b88004ad0 comp rx=0 tx=0).stop 2023-03-03T14:19:52.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 -- 172.21.15.7:0/2089903652 shutdown_connections 2023-03-03T14:19:52.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b740682f0 0x7f4b7406a7a0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:52.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4b90155390 0x7f4b901eb390 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:52.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b9006e020 0x7f4b900c10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:52.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 --2- 172.21.15.7:0/2089903652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4b90151770 0x7f4b901540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:19:52.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 -- 172.21.15.7:0/2089903652 >> 172.21.15.7:0/2089903652 conn(0x7f4b9006c110 msgr2=0x7f4b900c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:19:52.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 -- 172.21.15.7:0/2089903652 shutdown_connections 2023-03-03T14:19:52.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:19:52.252+0000 7f4b9712d700 1 -- 172.21.15.7:0/2089903652 wait complete. 2023-03-03T14:19:52.256 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:19:52.266 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-03-03T14:19:52.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:19:52.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T14:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T14:20:04.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:20:04.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.730+0000 7f72b30ad700 1 Processor -- start 2023-03-03T14:20:04.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.730+0000 7f72b30ad700 1 -- start start 2023-03-03T14:20:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.731+0000 7f72b30ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ac14fc50 0x7f72ac0c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.731+0000 7f72b30ad700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0ad2b0 con 0x7f72ac06d240 2023-03-03T14:20:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.731+0000 7f72b30ad700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac156420 con 0x7f72ac0c3e30 2023-03-03T14:20:04.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.731+0000 7f72b30ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac0c30c0 con 0x7f72ac14fc50 2023-03-03T14:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.731+0000 7f72abfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f72ac06d240 0x7f72ac0c27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42208/0 (socket says 172.21.15.7:42208) 2023-03-03T14:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.731+0000 7f72abfff700 1 -- 172.21.15.7:0/1749899449 learned_addr learned my addr 172.21.15.7:0/1749899449 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.731+0000 7f72b0e49700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ac14fc50 0x7f72ac0c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.731+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768312044 0 0) 0x7f72ac0ad2b0 con 0x7f72ac06d240 2023-03-03T14:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f729c0035b0 con 0x7f72ac06d240 2023-03-03T14:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315661544 0 0) 0x7f72ac156420 con 0x7f72ac0c3e30 2023-03-03T14:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac0ad2b0 con 0x7f72ac0c3e30 2023-03-03T14:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2382712840 0 0) 0x7f729c0035b0 con 0x7f72ac06d240 2023-03-03T14:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ac14fc50 msgr2=0x7f72ac0c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ac14fc50 0x7f72ac0c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 >> v1:172.21.15.143:6789/0 conn(0x7f72ac0c3e30 legacy=0x7f72ac15cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac0ad520 con 0x7f72ac06d240 2023-03-03T14:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7298001670 con 0x7f72ac06d240 2023-03-03T14:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 >> v1:172.21.15.7:6789/0 conn(0x7f72ac06d240 legacy=0x7f72ac0c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ac14fc50 0x7f729c003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f729c0041e0 0x7f729c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f729c004de0 0x7f729c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c00bc90 con 0x7f729c004de0 2023-03-03T14:20:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c00be00 con 0x7f729c0041e0 2023-03-03T14:20:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f729c00bfc0 con 0x7f72ac14fc50 2023-03-03T14:20:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72b164a700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f729c004de0 0x7f729c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72b0e49700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ac14fc50 0x7f729c003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:04.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72abfff700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f729c0041e0 0x7f729c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72b164a700 1 -- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ac14fc50 msgr2=0x7f729c003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72b164a700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ac14fc50 0x7f729c003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72b164a700 1 -- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f729c0041e0 msgr2=0x7f729c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72b164a700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f729c0041e0 0x7f729c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.732+0000 7f72b164a700 1 -- 172.21.15.7:0/1749899449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c0035b0 con 0x7f729c004de0 2023-03-03T14:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72b164a700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f729c004de0 0x7f729c005250 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f72a400c2f0 tx=0x7f729c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa3a72e63de94d4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4001f30 con 0x7f729c004de0 2023-03-03T14:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a400ce90 con 0x7f729c004de0 2023-03-03T14:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72ab7fe700 1 -- 172.21.15.7:0/1749899449 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a4003c40 con 0x7f729c004de0 2023-03-03T14:20:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72b30ad700 1 -- 172.21.15.7:0/1749899449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f729c004de0 msgr2=0x7f729c005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72b30ad700 1 --2- 172.21.15.7:0/1749899449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f729c004de0 0x7f729c005250 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f72a400c2f0 tx=0x7f729c005880 comp rx=0 tx=0).stop 2023-03-03T14:20:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72b164a700 1 -- 172.21.15.7:0/1749899449 reap_dead start 2023-03-03T14:20:04.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72b30ad700 1 -- 172.21.15.7:0/1749899449 shutdown_connections 2023-03-03T14:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72b30ad700 1 -- 172.21.15.7:0/1749899449 >> 172.21.15.7:0/1749899449 conn(0x7f72ac06c110 msgr2=0x7f72ac06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72b30ad700 1 -- 172.21.15.7:0/1749899449 shutdown_connections 2023-03-03T14:20:04.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.733+0000 7f72b30ad700 1 -- 172.21.15.7:0/1749899449 wait complete. 2023-03-03T14:20:04.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.735+0000 7f72b30ad700 1 Processor -- start 2023-03-03T14:20:04.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.735+0000 7f72b30ad700 1 -- start start 2023-03-03T14:20:04.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b30ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f729c004de0 0x7f72ac1548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:04.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b30ad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ac149ef0 0x7f72ac0c0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:04.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b30ad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ac0c0f40 0x7f72ac0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:04.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b30ad700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac1469d0 con 0x7f72ac0c0f40 2023-03-03T14:20:04.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b30ad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac14c7a0 con 0x7f72ac149ef0 2023-03-03T14:20:04.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b30ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac0c81b0 con 0x7f729c004de0 2023-03-03T14:20:04.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b164a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ac0c0f40 0x7f72ac0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b164a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ac0c0f40 0x7f72ac0705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48244/0 (socket says 172.21.15.7:48244) 2023-03-03T14:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b164a700 1 -- 172.21.15.7:0/841988839 learned_addr learned my addr 172.21.15.7:0/841988839 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72abfff700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ac149ef0 0x7f72ac0c0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b0e49700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f729c004de0 0x7f72ac1548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b164a700 1 -- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f729c004de0 msgr2=0x7f72ac1548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b164a700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f729c004de0 0x7f72ac1548f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:04.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b164a700 1 -- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ac149ef0 msgr2=0x7f72ac0c0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b164a700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ac149ef0 0x7f72ac0c0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b164a700 1 -- 172.21.15.7:0/841988839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac156690 con 0x7f72ac0c0f40 2023-03-03T14:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72abfff700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ac149ef0 0x7f72ac0c0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b164a700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ac0c0f40 0x7f72ac0705c0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f729c00bdd0 tx=0x7f72a4001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72a97fa700 1 -- 172.21.15.7:0/841988839 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a40050d0 con 0x7f72ac0c0f40 2023-03-03T14:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72a97fa700 1 -- 172.21.15.7:0/841988839 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a4005230 con 0x7f72ac0c0f40 2023-03-03T14:20:04.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b30ad700 1 -- 172.21.15.7:0/841988839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ac074220 con 0x7f72ac0c0f40 2023-03-03T14:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72a97fa700 1 -- 172.21.15.7:0/841988839 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a40055e0 con 0x7f72ac0c0f40 2023-03-03T14:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.736+0000 7f72b30ad700 1 -- 172.21.15.7:0/841988839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ac0acf60 con 0x7f72ac0c0f40 2023-03-03T14:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.737+0000 7f72b30ad700 1 -- 172.21.15.7:0/841988839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ac074bb0 con 0x7f72ac0c0f40 2023-03-03T14:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.738+0000 7f72a97fa700 1 -- 172.21.15.7:0/841988839 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f72a402f070 con 0x7f72ac0c0f40 2023-03-03T14:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.738+0000 7f72a97fa700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72900682f0 0x7f729006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.738+0000 7f72a97fa700 1 -- 172.21.15.7:0/841988839 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f72a40875f0 con 0x7f72ac0c0f40 2023-03-03T14:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.738+0000 7f72b0e49700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72900682f0 0x7f729006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:04.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.739+0000 7f72b0e49700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72900682f0 0x7f729006a7a0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009aa0 tx=0x7f72a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:04.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.743+0000 7f72a97fa700 1 -- 172.21.15.7:0/841988839 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f72a40538a0 con 0x7f72ac0c0f40 2023-03-03T14:20:04.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.980+0000 7f72b30ad700 1 -- 172.21.15.7:0/841988839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72ac073ed0 con 0x7f72ac0c0f40 2023-03-03T14:20:04.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.981+0000 7f72a97fa700 1 -- 172.21.15.7:0/841988839 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f72a40011c0 con 0x7f72ac0c0f40 2023-03-03T14:20:04.982 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:20:04.983 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:20:04.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 -- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72900682f0 msgr2=0x7f729006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:04.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72900682f0 0x7f729006a7a0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009aa0 tx=0x7f72a0008040 comp rx=0 tx=0).stop 2023-03-03T14:20:04.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 -- 172.21.15.7:0/841988839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ac0c0f40 msgr2=0x7f72ac0705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:04.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ac0c0f40 0x7f72ac0705c0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f729c00bdd0 tx=0x7f72a4001650 comp rx=0 tx=0).stop 2023-03-03T14:20:04.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 -- 172.21.15.7:0/841988839 shutdown_connections 2023-03-03T14:20:04.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72900682f0 0x7f729006a7a0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:04.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ac0c0f40 0x7f72ac0705c0 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:04.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f729c004de0 0x7f72ac1548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:04.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 --2- 172.21.15.7:0/841988839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ac149ef0 0x7f72ac0c0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:04.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 -- 172.21.15.7:0/841988839 >> 172.21.15.7:0/841988839 conn(0x7f72ac06c110 msgr2=0x7f72ac06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:20:04.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 -- 172.21.15.7:0/841988839 shutdown_connections 2023-03-03T14:20:04.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:04.983+0000 7f72b30ad700 1 -- 172.21.15.7:0/841988839 wait complete. 2023-03-03T14:20:04.986 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:20:04.996 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before thrashing 2023-03-03T14:20:06.998 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:20:07.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.136+0000 7ffb09462700 1 Processor -- start 2023-03-03T14:20:07.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.136+0000 7ffb09462700 1 -- start start 2023-03-03T14:20:07.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb09462700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 0x7ffb04158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:07.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb09462700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffb040c47b0 0x7ffb04158540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:07.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb09462700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb04152190 con 0x7ffb04151dc0 2023-03-03T14:20:07.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb09462700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb040c6310 con 0x7ffb040c47b0 2023-03-03T14:20:07.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb09462700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb04155720 con 0x7ffb040c6980 2023-03-03T14:20:07.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb037fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffb040c47b0 0x7ffb04158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:07.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb02ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ffb04151dc0 0x7ffb040ca630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42228/0 (socket says 172.21.15.7:42228) 2023-03-03T14:20:07.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb02ffd700 1 -- 172.21.15.7:0/1363954313 learned_addr learned my addr 172.21.15.7:0/1363954313 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:20:07.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb027fc700 1 --2- 172.21.15.7:0/1363954313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 0x7ffb04158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:07.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb037fe700 1 -- 172.21.15.7:0/1363954313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 msgr2=0x7ffb04158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:20:07.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb037fe700 1 --2- 172.21.15.7:0/1363954313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 0x7ffb04158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:07.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb037fe700 1 -- 172.21.15.7:0/1363954313 >> v1:172.21.15.7:6789/0 conn(0x7ffb04151dc0 legacy=0x7ffb040ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:07.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb037fe700 1 -- 172.21.15.7:0/1363954313 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04155160 con 0x7ffb040c47b0 2023-03-03T14:20:07.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb01ffb700 1 -- 172.21.15.7:0/1363954313 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174507217 0 0) 0x7ffb04152190 con 0x7ffb04151dc0 2023-03-03T14:20:07.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.137+0000 7ffb027fc700 1 --2- 172.21.15.7:0/1363954313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 0x7ffb04158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:20:07.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb037fe700 1 --2- 172.21.15.7:0/1363954313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb040c47b0 0x7ffb04158540 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009d90 tx=0x7ffafc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512232d973e95931 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:07.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb01ffb700 1 -- 172.21.15.7:0/1363954313 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc00e070 con 0x7ffb040c47b0 2023-03-03T14:20:07.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb01ffb700 1 -- 172.21.15.7:0/1363954313 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc00bbd0 con 0x7ffb040c47b0 2023-03-03T14:20:07.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb01ffb700 1 -- 172.21.15.7:0/1363954313 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc012430 con 0x7ffb040c47b0 2023-03-03T14:20:07.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb09462700 1 -- 172.21.15.7:0/1363954313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb040c47b0 msgr2=0x7ffb04158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:07.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb09462700 1 --2- 172.21.15.7:0/1363954313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb040c47b0 0x7ffb04158540 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009d90 tx=0x7ffafc00b330 comp rx=0 tx=0).stop 2023-03-03T14:20:07.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb09462700 1 -- 172.21.15.7:0/1363954313 shutdown_connections 2023-03-03T14:20:07.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb09462700 1 --2- 172.21.15.7:0/1363954313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb040c47b0 0x7ffb04158540 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:07.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb09462700 1 --2- 172.21.15.7:0/1363954313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 0x7ffb04158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:07.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb09462700 1 -- 172.21.15.7:0/1363954313 >> 172.21.15.7:0/1363954313 conn(0x7ffb0406b430 msgr2=0x7ffb04071490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:20:07.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb09462700 1 -- 172.21.15.7:0/1363954313 shutdown_connections 2023-03-03T14:20:07.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.138+0000 7ffb09462700 1 -- 172.21.15.7:0/1363954313 wait complete. 2023-03-03T14:20:07.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.140+0000 7ffb09462700 1 Processor -- start 2023-03-03T14:20:07.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.140+0000 7ffb09462700 1 -- start start 2023-03-03T14:20:07.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.140+0000 7ffb09462700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb040c47b0 0x7ffb0414ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:07.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.140+0000 7ffb09462700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 0x7ffb040bff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:07.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.140+0000 7ffb09462700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb04151dc0 0x7ffb041df280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:07.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.140+0000 7ffb09462700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb041e04c0 con 0x7ffb04151dc0 2023-03-03T14:20:07.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.140+0000 7ffb09462700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb041ea0d0 con 0x7ffb040c47b0 2023-03-03T14:20:07.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.140+0000 7ffb09462700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb041ea390 con 0x7ffb040c6980 2023-03-03T14:20:07.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb04151dc0 0x7ffb041df280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:07.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb04151dc0 0x7ffb041df280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48258/0 (socket says 172.21.15.7:48258) 2023-03-03T14:20:07.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb037fe700 1 -- 172.21.15.7:0/1336721713 learned_addr learned my addr 172.21.15.7:0/1336721713 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:20:07.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb02ffd700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb040c47b0 0x7ffb0414ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:07.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb027fc700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 0x7ffb040bff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:07.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb037fe700 1 -- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 msgr2=0x7ffb040bff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:07.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb037fe700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 0x7ffb040bff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:07.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb037fe700 1 -- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb040c47b0 msgr2=0x7ffb0414ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:07.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb037fe700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb040c47b0 0x7ffb0414ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:07.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb037fe700 1 -- 172.21.15.7:0/1336721713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc009a70 con 0x7ffb04151dc0 2023-03-03T14:20:07.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb037fe700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb04151dc0 0x7ffb041df280 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7ffb040c8220 tx=0x7ffafc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:07.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffaeb7fe700 1 -- 172.21.15.7:0/1336721713 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc012430 con 0x7ffb04151dc0 2023-03-03T14:20:07.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffaeb7fe700 1 -- 172.21.15.7:0/1336721713 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc020070 con 0x7ffb04151dc0 2023-03-03T14:20:07.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb09462700 1 -- 172.21.15.7:0/1336721713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb041553b0 con 0x7ffb04151dc0 2023-03-03T14:20:07.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffaeb7fe700 1 -- 172.21.15.7:0/1336721713 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc003ea0 con 0x7ffb04151dc0 2023-03-03T14:20:07.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.141+0000 7ffb09462700 1 -- 172.21.15.7:0/1336721713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb040b0640 con 0x7ffb04151dc0 2023-03-03T14:20:07.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.142+0000 7ffb09462700 1 -- 172.21.15.7:0/1336721713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb040c8270 con 0x7ffb04151dc0 2023-03-03T14:20:07.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.143+0000 7ffaeb7fe700 1 -- 172.21.15.7:0/1336721713 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffafc004280 con 0x7ffb04151dc0 2023-03-03T14:20:07.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.143+0000 7ffaeb7fe700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffaf00682f0 0x7ffaf006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:07.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.143+0000 7ffaeb7fe700 1 -- 172.21.15.7:0/1336721713 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ffafc022070 con 0x7ffb04151dc0 2023-03-03T14:20:07.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.143+0000 7ffb02ffd700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffaf00682f0 0x7ffaf006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:07.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.144+0000 7ffb02ffd700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffaf00682f0 0x7ffaf006a7a0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8007fd0 tx=0x7ffaf8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:07.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.148+0000 7ffaeb7fe700 1 -- 172.21.15.7:0/1336721713 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffafc054950 con 0x7ffb04151dc0 2023-03-03T14:20:07.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.387+0000 7ffb09462700 1 -- 172.21.15.7:0/1336721713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb04074bb0 con 0x7ffb04151dc0 2023-03-03T14:20:07.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.388+0000 7ffaeb7fe700 1 -- 172.21.15.7:0/1336721713 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ffafc057f70 con 0x7ffb04151dc0 2023-03-03T14:20:07.389 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:20:07.389 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:20:07.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 -- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffaf00682f0 msgr2=0x7ffaf006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:07.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffaf00682f0 0x7ffaf006a7a0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8007fd0 tx=0x7ffaf8009040 comp rx=0 tx=0).stop 2023-03-03T14:20:07.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 -- 172.21.15.7:0/1336721713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb04151dc0 msgr2=0x7ffb041df280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:07.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb04151dc0 0x7ffb041df280 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7ffb040c8220 tx=0x7ffafc004ad0 comp rx=0 tx=0).stop 2023-03-03T14:20:07.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 -- 172.21.15.7:0/1336721713 shutdown_connections 2023-03-03T14:20:07.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffaf00682f0 0x7ffaf006a7a0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:07.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffb04151dc0 0x7ffb041df280 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:07.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb040c47b0 0x7ffb0414ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:07.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 --2- 172.21.15.7:0/1336721713 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffb040c6980 0x7ffb040bff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:07.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 -- 172.21.15.7:0/1336721713 >> 172.21.15.7:0/1336721713 conn(0x7ffb0406b430 msgr2=0x7ffb040ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:20:07.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 -- 172.21.15.7:0/1336721713 shutdown_connections 2023-03-03T14:20:07.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:07.390+0000 7ffb09462700 1 -- 172.21.15.7:0/1336721713 wait complete. 2023-03-03T14:20:07.393 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:20:07.403 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before thrashing 2023-03-03T14:20:15.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:20:15.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.542+0000 7effeee48700 1 Processor -- start 2023-03-03T14:20:15.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.542+0000 7effeee48700 1 -- start start 2023-03-03T14:20:15.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effeee48700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7effe80caa30 0x7effe80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:15.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effeee48700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8154ff0 0x7effe80cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:15.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effeee48700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe80c7310 con 0x7effe80c00e0 2023-03-03T14:20:15.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effeee48700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe80bfd10 con 0x7effe80caa30 2023-03-03T14:20:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effeee48700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe80bfa30 con 0x7effe8154ff0 2023-03-03T14:20:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effecbe4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7effe80caa30 0x7effe80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effecbe4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7effe80caa30 0x7effe80c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48266/0 (socket says 172.21.15.7:48266) 2023-03-03T14:20:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effecbe4700 1 -- 172.21.15.7:0/3824803781 learned_addr learned my addr 172.21.15.7:0/3824803781 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:20:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effe7fff700 1 --1- 172.21.15.7:0/3824803781 >> v1:172.21.15.7:6789/0 conn(0x7effe80c00e0 0x7effe80c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42248/0 (socket says 172.21.15.7:42248) 2023-03-03T14:20:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effed3e5700 1 --2- 172.21.15.7:0/3824803781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8154ff0 0x7effe80cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effecbe4700 1 -- 172.21.15.7:0/3824803781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8154ff0 msgr2=0x7effe80cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:20:15.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effecbe4700 1 --2- 172.21.15.7:0/3824803781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8154ff0 0x7effe80cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effecbe4700 1 -- 172.21.15.7:0/3824803781 >> v1:172.21.15.7:6789/0 conn(0x7effe80c00e0 legacy=0x7effe80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effecbe4700 1 -- 172.21.15.7:0/3824803781 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe8154510 con 0x7effe80caa30 2023-03-03T14:20:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effe77fe700 1 -- 172.21.15.7:0/3824803781 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442437086 0 0) 0x7effe80c7310 con 0x7effe80c00e0 2023-03-03T14:20:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.543+0000 7effed3e5700 1 --2- 172.21.15.7:0/3824803781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8154ff0 0x7effe80cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:20:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effecbe4700 1 --2- 172.21.15.7:0/3824803781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe80caa30 0x7effe80c5c60 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7effdc0098f0 tx=0x7effdc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3accc273ca7e39 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effe77fe700 1 -- 172.21.15.7:0/3824803781 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc00b8a0 con 0x7effe80caa30 2023-03-03T14:20:15.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effe77fe700 1 -- 172.21.15.7:0/3824803781 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effdc00ba00 con 0x7effe80caa30 2023-03-03T14:20:15.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effe77fe700 1 -- 172.21.15.7:0/3824803781 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc00bd90 con 0x7effe80caa30 2023-03-03T14:20:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effeee48700 1 -- 172.21.15.7:0/3824803781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe80caa30 msgr2=0x7effe80c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effeee48700 1 --2- 172.21.15.7:0/3824803781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe80caa30 0x7effe80c5c60 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7effdc0098f0 tx=0x7effdc009c00 comp rx=0 tx=0).stop 2023-03-03T14:20:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effeee48700 1 -- 172.21.15.7:0/3824803781 shutdown_connections 2023-03-03T14:20:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effeee48700 1 --2- 172.21.15.7:0/3824803781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe8154ff0 0x7effe80cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:15.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effeee48700 1 --2- 172.21.15.7:0/3824803781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe80caa30 0x7effe80c5c60 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effeee48700 1 -- 172.21.15.7:0/3824803781 >> 172.21.15.7:0/3824803781 conn(0x7effe8066e40 msgr2=0x7effe8071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:20:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effeee48700 1 -- 172.21.15.7:0/3824803781 shutdown_connections 2023-03-03T14:20:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.544+0000 7effeee48700 1 -- 172.21.15.7:0/3824803781 wait complete. 2023-03-03T14:20:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.546+0000 7effeee48700 1 Processor -- start 2023-03-03T14:20:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.546+0000 7effeee48700 1 -- start start 2023-03-03T14:20:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.546+0000 7effeee48700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe80c00e0 0x7effe80c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:15.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.546+0000 7effeee48700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe80caa30 0x7effe81ed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.546+0000 7effeee48700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8154ff0 0x7effe81edb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.546+0000 7effeee48700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe8156810 con 0x7effe8154ff0 2023-03-03T14:20:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.546+0000 7effeee48700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe81ea820 con 0x7effe80c00e0 2023-03-03T14:20:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.546+0000 7effeee48700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe8156250 con 0x7effe80caa30 2023-03-03T14:20:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effed3e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8154ff0 0x7effe81edb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effed3e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8154ff0 0x7effe81edb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48276/0 (socket says 172.21.15.7:48276) 2023-03-03T14:20:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effed3e5700 1 -- 172.21.15.7:0/2548096467 learned_addr learned my addr 172.21.15.7:0/2548096467 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:20:15.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effecbe4700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe80c00e0 0x7effe80c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:15.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effe7fff700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe80caa30 0x7effe81ed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:15.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effed3e5700 1 -- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe80caa30 msgr2=0x7effe81ed5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:15.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effed3e5700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe80caa30 0x7effe81ed5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:15.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effed3e5700 1 -- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe80c00e0 msgr2=0x7effe80c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:15.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effed3e5700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe80c00e0 0x7effe80c6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:15.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effed3e5700 1 -- 172.21.15.7:0/2548096467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc009550 con 0x7effe8154ff0 2023-03-03T14:20:15.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effed3e5700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8154ff0 0x7effe81edb00 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7effe000cd50 tx=0x7effe000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:15.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effe57fa700 1 -- 172.21.15.7:0/2548096467 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe000f480 con 0x7effe8154ff0 2023-03-03T14:20:15.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effe57fa700 1 -- 172.21.15.7:0/2548096467 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe000f600 con 0x7effe8154ff0 2023-03-03T14:20:15.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effeee48700 1 -- 172.21.15.7:0/2548096467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe8154ad0 con 0x7effe8154ff0 2023-03-03T14:20:15.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effe57fa700 1 -- 172.21.15.7:0/2548096467 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe000f9b0 con 0x7effe8154ff0 2023-03-03T14:20:15.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.547+0000 7effeee48700 1 -- 172.21.15.7:0/2548096467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe81547f0 con 0x7effe8154ff0 2023-03-03T14:20:15.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.548+0000 7effeee48700 1 -- 172.21.15.7:0/2548096467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe80bf650 con 0x7effe8154ff0 2023-03-03T14:20:15.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.549+0000 7effe57fa700 1 -- 172.21.15.7:0/2548096467 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7effe0009030 con 0x7effe8154ff0 2023-03-03T14:20:15.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.549+0000 7effe57fa700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 0x7effcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:15.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.549+0000 7effe57fa700 1 -- 172.21.15.7:0/2548096467 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7effe0086aa0 con 0x7effe8154ff0 2023-03-03T14:20:15.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.549+0000 7effecbe4700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 0x7effcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:15.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.550+0000 7effecbe4700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 0x7effcc06a7a0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7effe8155ab0 tx=0x7effdc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:15.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.553+0000 7effe57fa700 1 -- 172.21.15.7:0/2548096467 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7effe0052e40 con 0x7effe8154ff0 2023-03-03T14:20:15.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.790+0000 7effeee48700 1 -- 172.21.15.7:0/2548096467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effe8074bb0 con 0x7effe8154ff0 2023-03-03T14:20:15.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.791+0000 7effe57fa700 1 -- 172.21.15.7:0/2548096467 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7effe0056460 con 0x7effe8154ff0 2023-03-03T14:20:15.792 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:20:15.793 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:20:15.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 -- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 msgr2=0x7effcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:15.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 0x7effcc06a7a0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7effe8155ab0 tx=0x7effdc008030 comp rx=0 tx=0).stop 2023-03-03T14:20:15.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 -- 172.21.15.7:0/2548096467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8154ff0 msgr2=0x7effe81edb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:15.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8154ff0 0x7effe81edb00 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7effe000cd50 tx=0x7effe000a560 comp rx=0 tx=0).stop 2023-03-03T14:20:15.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 -- 172.21.15.7:0/2548096467 shutdown_connections 2023-03-03T14:20:15.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effcc0682f0 0x7effcc06a7a0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:15.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effe8154ff0 0x7effe81edb00 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:15.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effe80c00e0 0x7effe80c6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:15.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 --2- 172.21.15.7:0/2548096467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effe80caa30 0x7effe81ed5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:15.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 -- 172.21.15.7:0/2548096467 >> 172.21.15.7:0/2548096467 conn(0x7effe8066e40 msgr2=0x7effe8066040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:20:15.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 -- 172.21.15.7:0/2548096467 shutdown_connections 2023-03-03T14:20:15.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:15.793+0000 7effeee48700 1 -- 172.21.15.7:0/2548096467 wait complete. 2023-03-03T14:20:15.796 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:20:15.806 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T14:20:22.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:22.453 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:25.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:20:25.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.945+0000 7f65596d6700 1 Processor -- start 2023-03-03T14:20:25.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.946+0000 7f65596d6700 1 -- start start 2023-03-03T14:20:25.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65596d6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f65540caa30 0x7f65540c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65596d6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6554154ff0 0x7f65540cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65596d6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540c7310 con 0x7f65540c00e0 2023-03-03T14:20:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65596d6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65540bfd10 con 0x7f65540caa30 2023-03-03T14:20:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65596d6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65540bfa30 con 0x7f6554154ff0 2023-03-03T14:20:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65527fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f65540c00e0 0x7f65540c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42266/0 (socket says 172.21.15.7:42266) 2023-03-03T14:20:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65527fc700 1 -- 172.21.15.7:0/2408065303 learned_addr learned my addr 172.21.15.7:0/2408065303 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:20:25.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65537fe700 1 --2- 172.21.15.7:0/2408065303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6554154ff0 0x7f65540cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:25.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f6552ffd700 1 --2- 172.21.15.7:0/2408065303 >> v2:172.21.15.143:3300/0 conn(0x7f65540caa30 0x7f65540c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:25.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f6551ffb700 1 -- 172.21.15.7:0/2408065303 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573819601 0 0) 0x7f65540c7310 con 0x7f65540c00e0 2023-03-03T14:20:25.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f6551ffb700 1 -- 172.21.15.7:0/2408065303 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f653c0035b0 con 0x7f65540c00e0 2023-03-03T14:20:25.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65537fe700 1 -- 172.21.15.7:0/2408065303 >> v2:172.21.15.143:3300/0 conn(0x7f65540caa30 msgr2=0x7f65540c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:20:25.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65537fe700 1 --2- 172.21.15.7:0/2408065303 >> v2:172.21.15.143:3300/0 conn(0x7f65540caa30 0x7f65540c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:25.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65537fe700 1 -- 172.21.15.7:0/2408065303 >> v1:172.21.15.7:6789/0 conn(0x7f65540c00e0 legacy=0x7f65540c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:25.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f6551ffb700 1 -- 172.21.15.7:0/2408065303 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2023180199 0 0) 0x7f653c0035b0 con 0x7f65540c00e0 2023-03-03T14:20:25.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.947+0000 7f65537fe700 1 -- 172.21.15.7:0/2408065303 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6554154510 con 0x7f6554154ff0 2023-03-03T14:20:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f65537fe700 1 --2- 172.21.15.7:0/2408065303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6554154ff0 0x7f65540cae00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f654c009d90 tx=0x7f654c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fd3bee52de61825 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f6551ffb700 1 -- 172.21.15.7:0/2408065303 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c00e070 con 0x7f6554154ff0 2023-03-03T14:20:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f6551ffb700 1 -- 172.21.15.7:0/2408065303 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654c003a20 con 0x7f6554154ff0 2023-03-03T14:20:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f6551ffb700 1 -- 172.21.15.7:0/2408065303 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65440017f0 con 0x7f6554154ff0 2023-03-03T14:20:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f65596d6700 1 -- 172.21.15.7:0/2408065303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6554154ff0 msgr2=0x7f65540cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f65596d6700 1 --2- 172.21.15.7:0/2408065303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6554154ff0 0x7f65540cae00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f654c009d90 tx=0x7f654c00b330 comp rx=0 tx=0).stop 2023-03-03T14:20:25.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f65596d6700 1 -- 172.21.15.7:0/2408065303 shutdown_connections 2023-03-03T14:20:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f65596d6700 1 --2- 172.21.15.7:0/2408065303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6554154ff0 0x7f65540cae00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f65596d6700 1 --2- 172.21.15.7:0/2408065303 >> v2:172.21.15.143:3300/0 conn(0x7f65540caa30 0x7f65540c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f65596d6700 1 -- 172.21.15.7:0/2408065303 >> 172.21.15.7:0/2408065303 conn(0x7f6554066e40 msgr2=0x7f6554071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:20:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.948+0000 7f65596d6700 1 -- 172.21.15.7:0/2408065303 shutdown_connections 2023-03-03T14:20:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.949+0000 7f65596d6700 1 -- 172.21.15.7:0/2408065303 wait complete. 2023-03-03T14:20:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.950+0000 7f65596d6700 1 Processor -- start 2023-03-03T14:20:25.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.950+0000 7f65596d6700 1 -- start start 2023-03-03T14:20:25.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.951+0000 7f65596d6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65540c00e0 0x7f655414a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:25.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.951+0000 7f65596d6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65540caa30 0x7f65541ed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:25.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.951+0000 7f65596d6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6554154ff0 0x7f6554148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:25.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.951+0000 7f65596d6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65541edee0 con 0x7f65540caa30 2023-03-03T14:20:25.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.951+0000 7f65596d6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655414af00 con 0x7f6554154ff0 2023-03-03T14:20:25.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.951+0000 7f65596d6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655414b1e0 con 0x7f65540c00e0 2023-03-03T14:20:25.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.951+0000 7f65527fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65540caa30 0x7f65541ed4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:25.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.951+0000 7f65527fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65540caa30 0x7f65541ed4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48294/0 (socket says 172.21.15.7:48294) 2023-03-03T14:20:25.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.951+0000 7f65527fc700 1 -- 172.21.15.7:0/1302155449 learned_addr learned my addr 172.21.15.7:0/1302155449 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:20:25.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f6552ffd700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65540c00e0 0x7f655414a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:25.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f65537fe700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6554154ff0 0x7f6554148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:25.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f65527fc700 1 -- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65540c00e0 msgr2=0x7f655414a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:25.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f65527fc700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65540c00e0 0x7f655414a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:25.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f65527fc700 1 -- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6554154ff0 msgr2=0x7f6554148d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:25.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f65527fc700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6554154ff0 0x7f6554148d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:25.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f65527fc700 1 -- 172.21.15.7:0/1302155449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654c009a70 con 0x7f65540caa30 2023-03-03T14:20:25.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f65537fe700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6554154ff0 0x7f6554148d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:20:25.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f65527fc700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65540caa30 0x7f65541ed4a0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f654400c750 tx=0x7f654400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:25.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f653b7fe700 1 -- 172.21.15.7:0/1302155449 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65440074e0 con 0x7f65540caa30 2023-03-03T14:20:25.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f653b7fe700 1 -- 172.21.15.7:0/1302155449 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6544007640 con 0x7f65540caa30 2023-03-03T14:20:25.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f653b7fe700 1 -- 172.21.15.7:0/1302155449 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544005a70 con 0x7f65540caa30 2023-03-03T14:20:25.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f65596d6700 1 -- 172.21.15.7:0/1302155449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6554154ad0 con 0x7f65540caa30 2023-03-03T14:20:25.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.952+0000 7f65596d6700 1 -- 172.21.15.7:0/1302155449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65541547f0 con 0x7f65540caa30 2023-03-03T14:20:25.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.953+0000 7f65596d6700 1 -- 172.21.15.7:0/1302155449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65540bf650 con 0x7f65540caa30 2023-03-03T14:20:25.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.954+0000 7f653b7fe700 1 -- 172.21.15.7:0/1302155449 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6544019030 con 0x7f65540caa30 2023-03-03T14:20:25.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.954+0000 7f653b7fe700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65400682c0 0x7f654006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:20:25.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.954+0000 7f653b7fe700 1 -- 172.21.15.7:0/1302155449 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f65440855a0 con 0x7f65540caa30 2023-03-03T14:20:25.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.954+0000 7f6552ffd700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65400682c0 0x7f654006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:20:25.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.955+0000 7f6552ffd700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65400682c0 0x7f654006a770 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6548009be0 tx=0x7f6548009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:20:25.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:25.958+0000 7f653b7fe700 1 -- 172.21.15.7:0/1302155449 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6544051940 con 0x7f65540caa30 2023-03-03T14:20:26.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.198+0000 7f65596d6700 1 -- 172.21.15.7:0/1302155449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f655414b6a0 con 0x7f65540caa30 2023-03-03T14:20:26.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.199+0000 7f653b7fe700 1 -- 172.21.15.7:0/1302155449 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6544017090 con 0x7f65540caa30 2023-03-03T14:20:26.200 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:20:26.201 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:20:26.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 -- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65400682c0 msgr2=0x7f654006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:26.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65400682c0 0x7f654006a770 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f6548009be0 tx=0x7f6548009210 comp rx=0 tx=0).stop 2023-03-03T14:20:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 -- 172.21.15.7:0/1302155449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65540caa30 msgr2=0x7f65541ed4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:20:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65540caa30 0x7f65541ed4a0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f654400c750 tx=0x7f654400ca60 comp rx=0 tx=0).stop 2023-03-03T14:20:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 -- 172.21.15.7:0/1302155449 shutdown_connections 2023-03-03T14:20:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6554154ff0 0x7f6554148d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65540c00e0 0x7f655414a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65400682c0 0x7f654006a770 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:26.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 --2- 172.21.15.7:0/1302155449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65540caa30 0x7f65541ed4a0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:20:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 -- 172.21.15.7:0/1302155449 >> 172.21.15.7:0/1302155449 conn(0x7f6554066e40 msgr2=0x7f6554153bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:20:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 -- 172.21.15.7:0/1302155449 shutdown_connections 2023-03-03T14:20:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:20:26.201+0000 7f65596d6700 1 -- 172.21.15.7:0/1302155449 wait complete. 2023-03-03T14:20:26.204 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:20:26.214 INFO:tasks.mds_thrash.fs.[a]:waiting for 35 secs before thrashing 2023-03-03T14:20:52.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:20:52.533 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:01.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:21:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.356+0000 7f1c60494700 1 Processor -- start 2023-03-03T14:21:01.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.356+0000 7f1c60494700 1 -- start start 2023-03-03T14:21:01.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.357+0000 7f1c60494700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1c5814f9d0 0x7f1c58156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:01.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.357+0000 7f1c60494700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c580abe20 0x7f1c581529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:01.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.357+0000 7f1c60494700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c580c2f30 con 0x7f1c5814fda0 2023-03-03T14:21:01.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.357+0000 7f1c60494700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c580c31d0 con 0x7f1c5814f9d0 2023-03-03T14:21:01.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.357+0000 7f1c60494700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c581524a0 con 0x7f1c580abe20 2023-03-03T14:21:01.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.357+0000 7f1c5e230700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1c5814f9d0 0x7f1c58156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:01.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.357+0000 7f1c5e230700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1c5814f9d0 0x7f1c58156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48332/0 (socket says 172.21.15.7:48332) 2023-03-03T14:21:01.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.357+0000 7f1c5e230700 1 -- 172.21.15.7:0/547654339 learned_addr learned my addr 172.21.15.7:0/547654339 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:21:01.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.358+0000 7f1c5ea31700 1 --2- 172.21.15.7:0/547654339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c580abe20 0x7f1c581529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:01.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.358+0000 7f1c5e230700 1 -- 172.21.15.7:0/547654339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c580abe20 msgr2=0x7f1c581529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:21:01.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.358+0000 7f1c5e230700 1 --2- 172.21.15.7:0/547654339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c580abe20 0x7f1c581529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.358+0000 7f1c5e230700 1 -- 172.21.15.7:0/547654339 >> v1:172.21.15.143:6789/0 conn(0x7f1c5814fda0 legacy=0x7f1c58071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.358+0000 7f1c5e230700 1 -- 172.21.15.7:0/547654339 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c580c34b0 con 0x7f1c5814f9d0 2023-03-03T14:21:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.358+0000 7f1c5e230700 1 --2- 172.21.15.7:0/547654339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814f9d0 0x7f1c58156430 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f1c540098f0 tx=0x7f1c54009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f77002e3da4d30 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.358+0000 7f1c5d22e700 1 -- 172.21.15.7:0/547654339 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5400b8a0 con 0x7f1c5814f9d0 2023-03-03T14:21:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.358+0000 7f1c5d22e700 1 -- 172.21.15.7:0/547654339 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5400ba00 con 0x7f1c5814f9d0 2023-03-03T14:21:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.358+0000 7f1c5d22e700 1 -- 172.21.15.7:0/547654339 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5400bd90 con 0x7f1c5814f9d0 2023-03-03T14:21:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.359+0000 7f1c60494700 1 -- 172.21.15.7:0/547654339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814f9d0 msgr2=0x7f1c58156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:01.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.359+0000 7f1c60494700 1 --2- 172.21.15.7:0/547654339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814f9d0 0x7f1c58156430 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f1c540098f0 tx=0x7f1c54009c00 comp rx=0 tx=0).stop 2023-03-03T14:21:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.359+0000 7f1c60494700 1 -- 172.21.15.7:0/547654339 shutdown_connections 2023-03-03T14:21:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.359+0000 7f1c60494700 1 --2- 172.21.15.7:0/547654339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c580abe20 0x7f1c581529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.359+0000 7f1c60494700 1 --2- 172.21.15.7:0/547654339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814f9d0 0x7f1c58156430 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.359+0000 7f1c60494700 1 -- 172.21.15.7:0/547654339 >> 172.21.15.7:0/547654339 conn(0x7f1c580c8900 msgr2=0x7f1c580c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:21:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.359+0000 7f1c60494700 1 -- 172.21.15.7:0/547654339 shutdown_connections 2023-03-03T14:21:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.359+0000 7f1c60494700 1 -- 172.21.15.7:0/547654339 wait complete. 2023-03-03T14:21:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.360+0000 7f1c60494700 1 Processor -- start 2023-03-03T14:21:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.360+0000 7f1c60494700 1 -- start start 2023-03-03T14:21:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c60494700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c580abe20 0x7f1c580c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c60494700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c5814f9d0 0x7f1c5814b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c60494700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814fda0 0x7f1c5814b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c60494700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c581df600 con 0x7f1c5814fda0 2023-03-03T14:21:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c60494700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c581eaa40 con 0x7f1c580abe20 2023-03-03T14:21:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c60494700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5814c7d0 con 0x7f1c5814f9d0 2023-03-03T14:21:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5ea31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814fda0 0x7f1c5814b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5ea31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814fda0 0x7f1c5814b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48342/0 (socket says 172.21.15.7:48342) 2023-03-03T14:21:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5ea31700 1 -- 172.21.15.7:0/1059947125 learned_addr learned my addr 172.21.15.7:0/1059947125 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:21:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5e230700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c580abe20 0x7f1c580c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5da2f700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c5814f9d0 0x7f1c5814b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5ea31700 1 -- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c5814f9d0 msgr2=0x7f1c5814b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5ea31700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c5814f9d0 0x7f1c5814b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5ea31700 1 -- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c580abe20 msgr2=0x7f1c580c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5ea31700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c580abe20 0x7f1c580c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5ea31700 1 -- 172.21.15.7:0/1059947125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c54009550 con 0x7f1c5814fda0 2023-03-03T14:21:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5e230700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c580abe20 0x7f1c580c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:21:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.361+0000 7f1c5ea31700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814fda0 0x7f1c5814b8d0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f1c50007d80 tx=0x7f1c5000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.362+0000 7f1c4affd700 1 -- 172.21.15.7:0/1059947125 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5000ede0 con 0x7f1c5814fda0 2023-03-03T14:21:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.362+0000 7f1c4affd700 1 -- 172.21.15.7:0/1059947125 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c50014420 con 0x7f1c5814fda0 2023-03-03T14:21:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.362+0000 7f1c60494700 1 -- 172.21.15.7:0/1059947125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c580ac350 con 0x7f1c5814fda0 2023-03-03T14:21:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.362+0000 7f1c4affd700 1 -- 172.21.15.7:0/1059947125 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c500147d0 con 0x7f1c5814fda0 2023-03-03T14:21:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.362+0000 7f1c60494700 1 -- 172.21.15.7:0/1059947125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c580ab630 con 0x7f1c5814fda0 2023-03-03T14:21:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.363+0000 7f1c60494700 1 -- 172.21.15.7:0/1059947125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c58071fa0 con 0x7f1c5814fda0 2023-03-03T14:21:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.363+0000 7f1c4affd700 1 -- 172.21.15.7:0/1059947125 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1c50012050 con 0x7f1c5814fda0 2023-03-03T14:21:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.364+0000 7f1c4affd700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c3c0682f0 0x7f1c3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.364+0000 7f1c4affd700 1 -- 172.21.15.7:0/1059947125 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1c50086850 con 0x7f1c5814fda0 2023-03-03T14:21:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.364+0000 7f1c5e230700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c3c0682f0 0x7f1c3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.364+0000 7f1c5e230700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c3c0682f0 0x7f1c3c06a7a0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f1c58155d90 tx=0x7f1c54008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:01.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.368+0000 7f1c4affd700 1 -- 172.21.15.7:0/1059947125 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1c50052b00 con 0x7f1c5814fda0 2023-03-03T14:21:01.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.607+0000 7f1c60494700 1 -- 172.21.15.7:0/1059947125 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c58074bb0 con 0x7f1c5814fda0 2023-03-03T14:21:01.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.607+0000 7f1c4affd700 1 -- 172.21.15.7:0/1059947125 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1c50056120 con 0x7f1c5814fda0 2023-03-03T14:21:01.610 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:21:01.610 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:21:01.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 -- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c3c0682f0 msgr2=0x7f1c3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:01.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c3c0682f0 0x7f1c3c06a7a0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f1c58155d90 tx=0x7f1c54008030 comp rx=0 tx=0).stop 2023-03-03T14:21:01.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 -- 172.21.15.7:0/1059947125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814fda0 msgr2=0x7f1c5814b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:01.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814fda0 0x7f1c5814b8d0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f1c50007d80 tx=0x7f1c5000e3c0 comp rx=0 tx=0).stop 2023-03-03T14:21:01.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 -- 172.21.15.7:0/1059947125 shutdown_connections 2023-03-03T14:21:01.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1c3c0682f0 0x7f1c3c06a7a0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:01.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1c5814fda0 0x7f1c5814b8d0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:01.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c580abe20 0x7f1c580c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:01.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 --2- 172.21.15.7:0/1059947125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1c5814f9d0 0x7f1c5814b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:01.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 -- 172.21.15.7:0/1059947125 >> 172.21.15.7:0/1059947125 conn(0x7f1c580c8900 msgr2=0x7f1c580c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:21:01.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 -- 172.21.15.7:0/1059947125 shutdown_connections 2023-03-03T14:21:01.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:01.609+0000 7f1c60494700 1 -- 172.21.15.7:0/1059947125 wait complete. 2023-03-03T14:21:01.614 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:21:01.623 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before thrashing 2023-03-03T14:21:06.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:21:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.762+0000 7f8879bf3700 1 Processor -- start 2023-03-03T14:21:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.762+0000 7f8879bf3700 1 -- start start 2023-03-03T14:21:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.763+0000 7f8879bf3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8874072250 0x7f887414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.763+0000 7f8879bf3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88740c1ce0 0x7f887414e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.763+0000 7f8879bf3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88741510d0 con 0x7f8874071e80 2023-03-03T14:21:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.763+0000 7f8879bf3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88740c3740 con 0x7f88740c1ce0 2023-03-03T14:21:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.763+0000 7f8879bf3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8874070670 con 0x7f8874072250 2023-03-03T14:21:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.763+0000 7f8873fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88740c1ce0 0x7f887414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.763+0000 7f88737fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8874071e80 0x7f8874151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42330/0 (socket says 172.21.15.7:42330) 2023-03-03T14:21:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.763+0000 7f88737fe700 1 -- 172.21.15.7:0/3102038735 learned_addr learned my addr 172.21.15.7:0/3102038735 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:21:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.763+0000 7f8872ffd700 1 --2- 172.21.15.7:0/3102038735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8874072250 0x7f887414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f88727fc700 1 -- 172.21.15.7:0/3102038735 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149491338 0 0) 0x7f88741510d0 con 0x7f8874071e80 2023-03-03T14:21:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f88727fc700 1 -- 172.21.15.7:0/3102038735 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88600035b0 con 0x7f8874071e80 2023-03-03T14:21:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f8873fff700 1 -- 172.21.15.7:0/3102038735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8874072250 msgr2=0x7f887414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:21:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f8873fff700 1 --2- 172.21.15.7:0/3102038735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8874072250 0x7f887414e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f8873fff700 1 -- 172.21.15.7:0/3102038735 >> v1:172.21.15.7:6789/0 conn(0x7f8874071e80 legacy=0x7f8874151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f8873fff700 1 -- 172.21.15.7:0/3102038735 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88740c39e0 con 0x7f88740c1ce0 2023-03-03T14:21:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f8873fff700 1 --2- 172.21.15.7:0/3102038735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1ce0 0x7f887414e730 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f886c009d90 tx=0x7f886c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdcf795289121fe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f88727fc700 1 -- 172.21.15.7:0/3102038735 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c00e070 con 0x7f88740c1ce0 2023-03-03T14:21:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f88727fc700 1 -- 172.21.15.7:0/3102038735 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886c00bb20 con 0x7f88740c1ce0 2023-03-03T14:21:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f88727fc700 1 -- 172.21.15.7:0/3102038735 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c012430 con 0x7f88740c1ce0 2023-03-03T14:21:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f8879bf3700 1 -- 172.21.15.7:0/3102038735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1ce0 msgr2=0x7f887414e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:06.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.764+0000 7f8879bf3700 1 --2- 172.21.15.7:0/3102038735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1ce0 0x7f887414e730 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f886c009d90 tx=0x7f886c00b330 comp rx=0 tx=0).stop 2023-03-03T14:21:06.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.765+0000 7f8879bf3700 1 -- 172.21.15.7:0/3102038735 shutdown_connections 2023-03-03T14:21:06.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.765+0000 7f8879bf3700 1 --2- 172.21.15.7:0/3102038735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1ce0 0x7f887414e730 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:06.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.765+0000 7f8879bf3700 1 --2- 172.21.15.7:0/3102038735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8874072250 0x7f887414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:06.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.765+0000 7f8879bf3700 1 -- 172.21.15.7:0/3102038735 >> 172.21.15.7:0/3102038735 conn(0x7f88740a94d0 msgr2=0x7f88740a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:21:06.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.765+0000 7f8879bf3700 1 -- 172.21.15.7:0/3102038735 shutdown_connections 2023-03-03T14:21:06.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.765+0000 7f8879bf3700 1 -- 172.21.15.7:0/3102038735 wait complete. 2023-03-03T14:21:06.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.766+0000 7f8879bf3700 1 Processor -- start 2023-03-03T14:21:06.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.766+0000 7f8879bf3700 1 -- start start 2023-03-03T14:21:06.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f8879bf3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8874071e80 0x7f88741400b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:06.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f8879bf3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8874072250 0x7f8874149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:06.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f8879bf3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88740c1ce0 0x7f88741e6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:06.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f8879bf3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88741441f0 con 0x7f8874071e80 2023-03-03T14:21:06.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f8879bf3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88741ead20 con 0x7f8874072250 2023-03-03T14:21:06.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f8879bf3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88741446e0 con 0x7f88740c1ce0 2023-03-03T14:21:06.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f88737fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8874071e80 0x7f88741400b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f88737fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8874071e80 0x7f88741400b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48356/0 (socket says 172.21.15.7:48356) 2023-03-03T14:21:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f88737fe700 1 -- 172.21.15.7:0/1273862328 learned_addr learned my addr 172.21.15.7:0/1273862328 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:21:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f8873fff700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88740c1ce0 0x7f88741e6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f8872ffd700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8874072250 0x7f8874149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f88737fe700 1 -- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88740c1ce0 msgr2=0x7f88741e6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f88737fe700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88740c1ce0 0x7f88741e6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f88737fe700 1 -- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8874072250 msgr2=0x7f8874149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:06.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f88737fe700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8874072250 0x7f8874149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:06.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f88737fe700 1 -- 172.21.15.7:0/1273862328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c009a70 con 0x7f8874071e80 2023-03-03T14:21:06.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f8872ffd700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8874072250 0x7f8874149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:21:06.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f88737fe700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8874071e80 0x7f88741400b0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f886400c700 tx=0x7f886400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:06.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f886bfff700 1 -- 172.21.15.7:0/1273862328 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8864005d70 con 0x7f8874071e80 2023-03-03T14:21:06.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f886bfff700 1 -- 172.21.15.7:0/1273862328 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88640070d0 con 0x7f8874071e80 2023-03-03T14:21:06.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f8879bf3700 1 -- 172.21.15.7:0/1273862328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8874151f30 con 0x7f8874071e80 2023-03-03T14:21:06.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.767+0000 7f886bfff700 1 -- 172.21.15.7:0/1273862328 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8864007460 con 0x7f8874071e80 2023-03-03T14:21:06.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.768+0000 7f8879bf3700 1 -- 172.21.15.7:0/1273862328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8874151300 con 0x7f8874071e80 2023-03-03T14:21:06.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.768+0000 7f8879bf3700 1 -- 172.21.15.7:0/1273862328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887406c9a0 con 0x7f8874071e80 2023-03-03T14:21:06.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.769+0000 7f886bfff700 1 -- 172.21.15.7:0/1273862328 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8864019030 con 0x7f8874071e80 2023-03-03T14:21:06.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.769+0000 7f886bfff700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88540682f0 0x7f885406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:06.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.769+0000 7f886bfff700 1 -- 172.21.15.7:0/1273862328 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8864085690 con 0x7f8874071e80 2023-03-03T14:21:06.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.770+0000 7f8872ffd700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88540682f0 0x7f885406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:06.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.771+0000 7f8872ffd700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88540682f0 0x7f885406a7a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f88740c3040 tx=0x7f885c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:06.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:06.774+0000 7f886bfff700 1 -- 172.21.15.7:0/1273862328 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f88640519b0 con 0x7f8874071e80 2023-03-03T14:21:07.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.008+0000 7f8879bf3700 1 -- 172.21.15.7:0/1273862328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88741452b0 con 0x7f8874071e80 2023-03-03T14:21:07.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.009+0000 7f886bfff700 1 -- 172.21.15.7:0/1273862328 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f88741452b0 con 0x7f8874071e80 2023-03-03T14:21:07.011 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:21:07.011 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:21:07.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 -- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88540682f0 msgr2=0x7f885406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:07.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88540682f0 0x7f885406a7a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f88740c3040 tx=0x7f885c008040 comp rx=0 tx=0).stop 2023-03-03T14:21:07.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 -- 172.21.15.7:0/1273862328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8874071e80 msgr2=0x7f88741400b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:07.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8874071e80 0x7f88741400b0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f886400c700 tx=0x7f886400cac0 comp rx=0 tx=0).stop 2023-03-03T14:21:07.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 -- 172.21.15.7:0/1273862328 shutdown_connections 2023-03-03T14:21:07.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88740c1ce0 0x7f88741e6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:07.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88540682f0 0x7f885406a7a0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:07.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8874071e80 0x7f88741400b0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:07.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 --2- 172.21.15.7:0/1273862328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8874072250 0x7f8874149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:07.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 -- 172.21.15.7:0/1273862328 >> 172.21.15.7:0/1273862328 conn(0x7f88740a94d0 msgr2=0x7f887414dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:21:07.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 -- 172.21.15.7:0/1273862328 shutdown_connections 2023-03-03T14:21:07.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:07.011+0000 7f8879bf3700 1 -- 172.21.15.7:0/1273862328 wait complete. 2023-03-03T14:21:07.014 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:21:07.024 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2023-03-03T14:21:22.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:21:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.164+0000 7f1b7c33c700 1 Processor -- start 2023-03-03T14:21:22.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.164+0000 7f1b7c33c700 1 -- start start 2023-03-03T14:21:22.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b7c33c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7414fc50 0x7f1b740c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b7c33c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1b7406d240 0x7f1b740c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b7c33c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b740ad2b0 con 0x7f1b740c3e30 2023-03-03T14:21:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b7c33c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b740c30c0 con 0x7f1b7406d240 2023-03-03T14:21:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b7c33c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b74156420 con 0x7f1b7414fc50 2023-03-03T14:21:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b798d7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1b7406d240 0x7f1b740c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b7a8d9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1b740c3e30 0x7f1b740c7390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42364/0 (socket says 172.21.15.7:42364) 2023-03-03T14:21:22.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b7a8d9700 1 -- 172.21.15.7:0/1983497276 learned_addr learned my addr 172.21.15.7:0/1983497276 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:21:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b798d7700 1 -- 172.21.15.7:0/1983497276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7414fc50 msgr2=0x7f1b740c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:21:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b798d7700 1 --2- 172.21.15.7:0/1983497276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7414fc50 0x7f1b740c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b798d7700 1 -- 172.21.15.7:0/1983497276 >> v1:172.21.15.7:6789/0 conn(0x7f1b740c3e30 legacy=0x7f1b740c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b798d7700 1 -- 172.21.15.7:0/1983497276 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b740ad520 con 0x7f1b7406d240 2023-03-03T14:21:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b798d7700 1 --2- 172.21.15.7:0/1983497276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 0x7f1b740c27b0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009840 tx=0x7f1b64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2051e489bf9f10b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b790d6700 1 -- 172.21.15.7:0/1983497276 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64011830 con 0x7f1b7406d240 2023-03-03T14:21:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b790d6700 1 -- 172.21.15.7:0/1983497276 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b64011990 con 0x7f1b7406d240 2023-03-03T14:21:22.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.165+0000 7f1b790d6700 1 -- 172.21.15.7:0/1983497276 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64011d20 con 0x7f1b7406d240 2023-03-03T14:21:22.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.166+0000 7f1b7c33c700 1 -- 172.21.15.7:0/1983497276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 msgr2=0x7f1b740c27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:22.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.166+0000 7f1b7c33c700 1 --2- 172.21.15.7:0/1983497276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 0x7f1b740c27b0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009840 tx=0x7f1b64009b50 comp rx=0 tx=0).stop 2023-03-03T14:21:22.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.166+0000 7f1b7c33c700 1 -- 172.21.15.7:0/1983497276 shutdown_connections 2023-03-03T14:21:22.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.166+0000 7f1b7c33c700 1 --2- 172.21.15.7:0/1983497276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7414fc50 0x7f1b740c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:22.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.166+0000 7f1b7c33c700 1 --2- 172.21.15.7:0/1983497276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 0x7f1b740c27b0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:22.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.166+0000 7f1b7c33c700 1 -- 172.21.15.7:0/1983497276 >> 172.21.15.7:0/1983497276 conn(0x7f1b7406c110 msgr2=0x7f1b7406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:21:22.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.166+0000 7f1b7c33c700 1 -- 172.21.15.7:0/1983497276 shutdown_connections 2023-03-03T14:21:22.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.166+0000 7f1b7c33c700 1 -- 172.21.15.7:0/1983497276 wait complete. 2023-03-03T14:21:22.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.167+0000 7f1b7c33c700 1 Processor -- start 2023-03-03T14:21:22.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.167+0000 7f1b7c33c700 1 -- start start 2023-03-03T14:21:22.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b7c33c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 0x7f1b741de090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:22.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b7c33c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b740c3e30 0x7f1b74146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:22.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b7c33c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7414fc50 0x7f1b74146d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:22.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b7c33c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b741ebe20 con 0x7f1b7406d240 2023-03-03T14:21:22.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b7c33c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b741e7960 con 0x7f1b740c3e30 2023-03-03T14:21:22.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b7c33c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b741e7c20 con 0x7f1b7414fc50 2023-03-03T14:21:22.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b7a0d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 0x7f1b741de090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:22.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b7a0d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 0x7f1b741de090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48386/0 (socket says 172.21.15.7:48386) 2023-03-03T14:21:22.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b7a0d8700 1 -- 172.21.15.7:0/33744468 learned_addr learned my addr 172.21.15.7:0/33744468 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:21:22.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b798d7700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b740c3e30 0x7f1b74146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:22.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.168+0000 7f1b7a8d9700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7414fc50 0x7f1b74146d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:22.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b7a0d8700 1 -- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7414fc50 msgr2=0x7f1b74146d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:22.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b7a0d8700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7414fc50 0x7f1b74146d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:22.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b7a0d8700 1 -- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b740c3e30 msgr2=0x7f1b74146840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:22.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b7a0d8700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b740c3e30 0x7f1b74146840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:22.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b7a0d8700 1 -- 172.21.15.7:0/33744468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b640094a0 con 0x7f1b7406d240 2023-03-03T14:21:22.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b7a0d8700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 0x7f1b741de090 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000b530 tx=0x7f1b7000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:22.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b6affd700 1 -- 172.21.15.7:0/33744468 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70011830 con 0x7f1b7406d240 2023-03-03T14:21:22.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b6affd700 1 -- 172.21.15.7:0/33744468 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b70011990 con 0x7f1b7406d240 2023-03-03T14:21:22.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b7c33c700 1 -- 172.21.15.7:0/33744468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b740c2cf0 con 0x7f1b7406d240 2023-03-03T14:21:22.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b6affd700 1 -- 172.21.15.7:0/33744468 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70011d40 con 0x7f1b7406d240 2023-03-03T14:21:22.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.169+0000 7f1b7c33c700 1 -- 172.21.15.7:0/33744468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b740c2490 con 0x7f1b7406d240 2023-03-03T14:21:22.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.170+0000 7f1b7c33c700 1 -- 172.21.15.7:0/33744468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b74074220 con 0x7f1b7406d240 2023-03-03T14:21:22.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.171+0000 7f1b6affd700 1 -- 172.21.15.7:0/33744468 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1b70021070 con 0x7f1b7406d240 2023-03-03T14:21:22.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.171+0000 7f1b6affd700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b580682f0 0x7f1b5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:22.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.171+0000 7f1b6affd700 1 -- 172.21.15.7:0/33744468 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1b70085ff0 con 0x7f1b7406d240 2023-03-03T14:21:22.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.171+0000 7f1b798d7700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b580682f0 0x7f1b5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:22.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.172+0000 7f1b798d7700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b580682f0 0x7f1b5806a7a0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f1b640090b0 tx=0x7f1b6401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:22.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.175+0000 7f1b6affd700 1 -- 172.21.15.7:0/33744468 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1b700523f0 con 0x7f1b7406d240 2023-03-03T14:21:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.414+0000 7f1b7c33c700 1 -- 172.21.15.7:0/33744468 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b7414ab90 con 0x7f1b7406d240 2023-03-03T14:21:22.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.414+0000 7f1b6affd700 1 -- 172.21.15.7:0/33744468 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1b70015020 con 0x7f1b7406d240 2023-03-03T14:21:22.416 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:21:22.416 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:21:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 -- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b580682f0 msgr2=0x7f1b5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b580682f0 0x7f1b5806a7a0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f1b640090b0 tx=0x7f1b6401c040 comp rx=0 tx=0).stop 2023-03-03T14:21:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 -- 172.21.15.7:0/33744468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 msgr2=0x7f1b741de090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 0x7f1b741de090 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000b530 tx=0x7f1b7000b840 comp rx=0 tx=0).stop 2023-03-03T14:21:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 -- 172.21.15.7:0/33744468 shutdown_connections 2023-03-03T14:21:22.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1b7414fc50 0x7f1b74146d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:22.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1b580682f0 0x7f1b5806a7a0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:22.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b7406d240 0x7f1b741de090 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:22.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 --2- 172.21.15.7:0/33744468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b740c3e30 0x7f1b74146840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:22.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 -- 172.21.15.7:0/33744468 >> 172.21.15.7:0/33744468 conn(0x7f1b7406c110 msgr2=0x7f1b7414e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:21:22.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 -- 172.21.15.7:0/33744468 shutdown_connections 2023-03-03T14:21:22.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:22.416+0000 7f1b7c33c700 1 -- 172.21.15.7:0/33744468 wait complete. 2023-03-03T14:21:22.420 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:21:22.429 INFO:tasks.mds_thrash.fs.[a]:waiting for 51 secs before thrashing 2023-03-03T14:21:22.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:22.603 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:41.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:21:41.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.405+0000 7ff0c2789700 1 Processor -- start 2023-03-03T14:21:41.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.405+0000 7ff0c2789700 1 -- start start 2023-03-03T14:21:41.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.406+0000 7ff0c2789700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0bc0aad20 0x7ff0bc157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:41.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.406+0000 7ff0c2789700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0bc0a9c20 con 0x7ff0bc0aa030 2023-03-03T14:21:41.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.406+0000 7ff0c2789700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0bc154f10 con 0x7ff0bc14b0d0 2023-03-03T14:21:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.406+0000 7ff0c2789700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0bc0ac3c0 con 0x7ff0bc0aad20 2023-03-03T14:21:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.406+0000 7ff0bb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0bc0aad20 0x7ff0bc157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.406+0000 7ff0c0d26700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff0bc0aa030 0x7ff0bc158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42398/0 (socket says 172.21.15.7:42398) 2023-03-03T14:21:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.406+0000 7ff0c0d26700 1 -- 172.21.15.7:0/2520529179 learned_addr learned my addr 172.21.15.7:0/2520529179 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:21:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0bb7fe700 1 -- 172.21.15.7:0/2520529179 >> v1:172.21.15.143:6789/0 conn(0x7ff0bc14b0d0 legacy=0x7ff0bc071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:21:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0bb7fe700 1 -- 172.21.15.7:0/2520529179 >> v1:172.21.15.7:6789/0 conn(0x7ff0bc0aa030 legacy=0x7ff0bc158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0bb7fe700 1 -- 172.21.15.7:0/2520529179 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc06fc70 con 0x7ff0bc0aad20 2023-03-03T14:21:41.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0baffd700 1 -- 172.21.15.7:0/2520529179 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973986105 0 0) 0x7ff0bc0a9c20 con 0x7ff0bc0aa030 2023-03-03T14:21:41.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0bb7fe700 1 --2- 172.21.15.7:0/2520529179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aad20 0x7ff0bc157df0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009840 tx=0x7ff0a4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff5022dfcf183e9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:41.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0baffd700 1 -- 172.21.15.7:0/2520529179 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4004230 con 0x7ff0bc0aad20 2023-03-03T14:21:41.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0baffd700 1 -- 172.21.15.7:0/2520529179 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a4004390 con 0x7ff0bc0aad20 2023-03-03T14:21:41.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0baffd700 1 -- 172.21.15.7:0/2520529179 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a40036a0 con 0x7ff0bc0aad20 2023-03-03T14:21:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0c2789700 1 -- 172.21.15.7:0/2520529179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aad20 msgr2=0x7ff0bc157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0c2789700 1 --2- 172.21.15.7:0/2520529179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aad20 0x7ff0bc157df0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009840 tx=0x7ff0a4009b50 comp rx=0 tx=0).stop 2023-03-03T14:21:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0c2789700 1 -- 172.21.15.7:0/2520529179 shutdown_connections 2023-03-03T14:21:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0c2789700 1 --2- 172.21.15.7:0/2520529179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aad20 0x7ff0bc157df0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.407+0000 7ff0c2789700 1 -- 172.21.15.7:0/2520529179 >> 172.21.15.7:0/2520529179 conn(0x7ff0bc0c8900 msgr2=0x7ff0bc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:21:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.408+0000 7ff0c2789700 1 -- 172.21.15.7:0/2520529179 shutdown_connections 2023-03-03T14:21:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.408+0000 7ff0c2789700 1 -- 172.21.15.7:0/2520529179 wait complete. 2023-03-03T14:21:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.409+0000 7ff0c2789700 1 Processor -- start 2023-03-03T14:21:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.409+0000 7ff0c2789700 1 -- start start 2023-03-03T14:21:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0c2789700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aa030 0x7ff0bc1ea460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0c2789700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0bc0aad20 0x7ff0bc1df430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:41.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0c2789700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc14b0d0 0x7ff0bc1dfef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:41.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0c2789700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc0c7d80 con 0x7ff0bc0aa030 2023-03-03T14:21:41.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0c2789700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc1e99a0 con 0x7ff0bc0aad20 2023-03-03T14:21:41.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0c2789700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc0c6310 con 0x7ff0bc14b0d0 2023-03-03T14:21:41.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aa030 0x7ff0bc1ea460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:41.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aa030 0x7ff0bc1ea460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48420/0 (socket says 172.21.15.7:48420) 2023-03-03T14:21:41.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0bbfff700 1 -- 172.21.15.7:0/678582392 learned_addr learned my addr 172.21.15.7:0/678582392 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:21:41.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0bb7fe700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0bc0aad20 0x7ff0bc1df430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:41.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0c0d26700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc14b0d0 0x7ff0bc1dfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:41.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0bbfff700 1 -- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc14b0d0 msgr2=0x7ff0bc1dfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:41.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0bbfff700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc14b0d0 0x7ff0bc1dfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:41.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0bbfff700 1 -- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0bc0aad20 msgr2=0x7ff0bc1df430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:41.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0bbfff700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0bc0aad20 0x7ff0bc1df430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:41.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0bbfff700 1 -- 172.21.15.7:0/678582392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a40094a0 con 0x7ff0bc0aa030 2023-03-03T14:21:41.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0bbfff700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aa030 0x7ff0bc1ea460 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00c4d0 tx=0x7ff0ac00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:41.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0b8ff9700 1 -- 172.21.15.7:0/678582392 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac011790 con 0x7ff0bc0aa030 2023-03-03T14:21:41.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0b8ff9700 1 -- 172.21.15.7:0/678582392 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac0118f0 con 0x7ff0bc0aa030 2023-03-03T14:21:41.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0c2789700 1 -- 172.21.15.7:0/678582392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0bc06f6b0 con 0x7ff0bc0aa030 2023-03-03T14:21:41.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.410+0000 7ff0b8ff9700 1 -- 172.21.15.7:0/678582392 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac011ca0 con 0x7ff0bc0aa030 2023-03-03T14:21:41.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.411+0000 7ff0c2789700 1 -- 172.21.15.7:0/678582392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0bc155160 con 0x7ff0bc0aa030 2023-03-03T14:21:41.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.411+0000 7ff0c2789700 1 -- 172.21.15.7:0/678582392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0bc06f990 con 0x7ff0bc0aa030 2023-03-03T14:21:41.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.412+0000 7ff0b8ff9700 1 -- 172.21.15.7:0/678582392 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff0ac010050 con 0x7ff0bc0aa030 2023-03-03T14:21:41.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.412+0000 7ff0b8ff9700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff09c0682f0 0x7ff09c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:21:41.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.412+0000 7ff0b8ff9700 1 -- 172.21.15.7:0/678582392 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff0ac085ed0 con 0x7ff0bc0aa030 2023-03-03T14:21:41.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.413+0000 7ff0bb7fe700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff09c0682f0 0x7ff09c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:21:41.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.413+0000 7ff0bb7fe700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff09c0682f0 0x7ff09c06a7a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009840 tx=0x7ff0a401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:21:41.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.417+0000 7ff0b8ff9700 1 -- 172.21.15.7:0/678582392 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff0ac052270 con 0x7ff0bc0aa030 2023-03-03T14:21:41.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.652+0000 7ff0c2789700 1 -- 172.21.15.7:0/678582392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0bc1e9fd0 con 0x7ff0bc0aa030 2023-03-03T14:21:41.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.653+0000 7ff0b8ff9700 1 -- 172.21.15.7:0/678582392 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff0ac055890 con 0x7ff0bc0aa030 2023-03-03T14:21:41.654 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:21:41.655 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:21:41.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 -- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff09c0682f0 msgr2=0x7ff09c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:41.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff09c0682f0 0x7ff09c06a7a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009840 tx=0x7ff0a401c040 comp rx=0 tx=0).stop 2023-03-03T14:21:41.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 -- 172.21.15.7:0/678582392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aa030 msgr2=0x7ff0bc1ea460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:21:41.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aa030 0x7ff0bc1ea460 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00c4d0 tx=0x7ff0ac00c890 comp rx=0 tx=0).stop 2023-03-03T14:21:41.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 -- 172.21.15.7:0/678582392 shutdown_connections 2023-03-03T14:21:41.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc14b0d0 0x7ff0bc1dfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:41.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff09c0682f0 0x7ff09c06a7a0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:41.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc0aa030 0x7ff0bc1ea460 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:41.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 --2- 172.21.15.7:0/678582392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0bc0aad20 0x7ff0bc1df430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:21:41.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 -- 172.21.15.7:0/678582392 >> 172.21.15.7:0/678582392 conn(0x7ff0bc0c8900 msgr2=0x7ff0bc0aca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:21:41.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 -- 172.21.15.7:0/678582392 shutdown_connections 2023-03-03T14:21:41.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:21:41.655+0000 7ff0c2789700 1 -- 172.21.15.7:0/678582392 wait complete. 2023-03-03T14:21:41.658 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:21:41.668 INFO:tasks.mds_thrash.fs.[b]:waiting for 28 secs before thrashing 2023-03-03T14:21:52.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:21:52.667 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:09.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:22:09.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.806+0000 7f31f72be700 1 Processor -- start 2023-03-03T14:22:09.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.806+0000 7f31f72be700 1 -- start start 2023-03-03T14:22:09.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.807+0000 7f31f72be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f31f014b460 0x7f31f006aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.807+0000 7f31f72be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 0x7f31f006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.807+0000 7f31f72be700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f0070360 con 0x7f31f0071990 2023-03-03T14:22:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.807+0000 7f31f72be700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f01521a0 con 0x7f31f014b460 2023-03-03T14:22:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.807+0000 7f31f72be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f006a5e0 con 0x7f31f0072470 2023-03-03T14:22:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.807+0000 7f31f4859700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f31f014b460 0x7f31f006aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.807+0000 7f31f585b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 0x7f31f006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:09.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.807+0000 7f31f505a700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f31f0071990 0x7f31f0071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:45960/0 (socket says 172.21.15.7:45960) 2023-03-03T14:22:09.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.807+0000 7f31f505a700 1 -- 172.21.15.7:0/911862984 learned_addr learned my addr 172.21.15.7:0/911862984 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:22:09.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.807+0000 7f31f4859700 1 --2- 172.21.15.7:0/911862984 >> v2:172.21.15.143:3300/0 conn(0x7f31f014b460 0x7f31f006aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:44236/0 (socket says 172.21.15.7:44236) 2023-03-03T14:22:09.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.808+0000 7f31f4859700 1 -- 172.21.15.7:0/911862984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 msgr2=0x7f31f006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:22:09.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.808+0000 7f31f4859700 1 --2- 172.21.15.7:0/911862984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 0x7f31f006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:09.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.808+0000 7f31f4859700 1 -- 172.21.15.7:0/911862984 >> v1:172.21.15.143:6789/0 conn(0x7f31f0071990 legacy=0x7f31f0071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:09.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.808+0000 7f31f4859700 1 -- 172.21.15.7:0/911862984 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f00c38b0 con 0x7f31f014b460 2023-03-03T14:22:09.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.808+0000 7f31f585b700 1 --2- 172.21.15.7:0/911862984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 0x7f31f006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:22:09.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.808+0000 7f31f4859700 1 --2- 172.21.15.7:0/911862984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f014b460 0x7f31f006aa70 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009840 tx=0x7f31ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=276220fb993ee6e5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:09.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.808+0000 7f31e7fff700 1 -- 172.21.15.7:0/911862984 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec011830 con 0x7f31f014b460 2023-03-03T14:22:09.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.808+0000 7f31e7fff700 1 -- 172.21.15.7:0/911862984 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec011990 con 0x7f31f014b460 2023-03-03T14:22:09.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.808+0000 7f31e7fff700 1 -- 172.21.15.7:0/911862984 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec011d20 con 0x7f31f014b460 2023-03-03T14:22:09.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.809+0000 7f31f72be700 1 -- 172.21.15.7:0/911862984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f014b460 msgr2=0x7f31f006aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:09.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.809+0000 7f31f72be700 1 --2- 172.21.15.7:0/911862984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f014b460 0x7f31f006aa70 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009840 tx=0x7f31ec009b50 comp rx=0 tx=0).stop 2023-03-03T14:22:09.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.809+0000 7f31f72be700 1 -- 172.21.15.7:0/911862984 shutdown_connections 2023-03-03T14:22:09.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.809+0000 7f31f72be700 1 --2- 172.21.15.7:0/911862984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 0x7f31f006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:09.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.809+0000 7f31f72be700 1 --2- 172.21.15.7:0/911862984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f014b460 0x7f31f006aa70 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:09.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.809+0000 7f31f72be700 1 -- 172.21.15.7:0/911862984 >> 172.21.15.7:0/911862984 conn(0x7f31f00a94d0 msgr2=0x7f31f00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:22:09.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.809+0000 7f31f72be700 1 -- 172.21.15.7:0/911862984 shutdown_connections 2023-03-03T14:22:09.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.809+0000 7f31f72be700 1 -- 172.21.15.7:0/911862984 wait complete. 2023-03-03T14:22:09.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.810+0000 7f31f72be700 1 Processor -- start 2023-03-03T14:22:09.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.810+0000 7f31f72be700 1 -- start start 2023-03-03T14:22:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f72be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31f0071990 0x7f31f006c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f72be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 0x7f31f0147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f72be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f006c960 0x7f31f0148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f72be700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00acdb0 con 0x7f31f0071990 2023-03-03T14:22:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f72be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f006d1a0 con 0x7f31f006c960 2023-03-03T14:22:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f72be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f01467d0 con 0x7f31f0072470 2023-03-03T14:22:09.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f505a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31f0071990 0x7f31f006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f505a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31f0071990 0x7f31f006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48466/0 (socket says 172.21.15.7:48466) 2023-03-03T14:22:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f505a700 1 -- 172.21.15.7:0/669657962 learned_addr learned my addr 172.21.15.7:0/669657962 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:22:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f585b700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f006c960 0x7f31f0148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f4859700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 0x7f31f0147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f505a700 1 -- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 msgr2=0x7f31f0147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f505a700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 0x7f31f0147c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:09.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f505a700 1 -- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f006c960 msgr2=0x7f31f0148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:09.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f505a700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f006c960 0x7f31f0148170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:09.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f505a700 1 -- 172.21.15.7:0/669657962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e8009a70 con 0x7f31f0071990 2023-03-03T14:22:09.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f585b700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f006c960 0x7f31f0148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:22:09.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f4859700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 0x7f31f0147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:22:09.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.811+0000 7f31f505a700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31f0071990 0x7f31f006c420 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f31e000e970 tx=0x7f31e000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:09.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.812+0000 7f31e5ffb700 1 -- 172.21.15.7:0/669657962 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0005d70 con 0x7f31f0071990 2023-03-03T14:22:09.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.812+0000 7f31e5ffb700 1 -- 172.21.15.7:0/669657962 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e000ee50 con 0x7f31f0071990 2023-03-03T14:22:09.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.812+0000 7f31f72be700 1 -- 172.21.15.7:0/669657962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ec0094a0 con 0x7f31f0071990 2023-03-03T14:22:09.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.812+0000 7f31e5ffb700 1 -- 172.21.15.7:0/669657962 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0007270 con 0x7f31f0071990 2023-03-03T14:22:09.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.812+0000 7f31f72be700 1 -- 172.21.15.7:0/669657962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f00c3b50 con 0x7f31f0071990 2023-03-03T14:22:09.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.813+0000 7f31f72be700 1 -- 172.21.15.7:0/669657962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f006a2b0 con 0x7f31f0071990 2023-03-03T14:22:09.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.813+0000 7f31e5ffb700 1 -- 172.21.15.7:0/669657962 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f31e0021070 con 0x7f31f0071990 2023-03-03T14:22:09.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.813+0000 7f31e5ffb700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31d40682f0 0x7f31d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:09.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.814+0000 7f31e5ffb700 1 -- 172.21.15.7:0/669657962 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f31e0088130 con 0x7f31f0071990 2023-03-03T14:22:09.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.814+0000 7f31f4859700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31d40682f0 0x7f31d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.814+0000 7f31f4859700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31d40682f0 0x7f31d406a7a0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f31f01eb720 tx=0x7f31ec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:09.818+0000 7f31e5ffb700 1 -- 172.21.15.7:0/669657962 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f31e00544d0 con 0x7f31f0071990 2023-03-03T14:22:10.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.057+0000 7f31f72be700 1 -- 172.21.15.7:0/669657962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31f006b4f0 con 0x7f31f0071990 2023-03-03T14:22:10.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.058+0000 7f31e5ffb700 1 -- 172.21.15.7:0/669657962 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f31e0057af0 con 0x7f31f0071990 2023-03-03T14:22:10.059 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:22:10.060 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:22:10.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 -- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31d40682f0 msgr2=0x7f31d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:10.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31d40682f0 0x7f31d406a7a0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f31f01eb720 tx=0x7f31ec008030 comp rx=0 tx=0).stop 2023-03-03T14:22:10.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 -- 172.21.15.7:0/669657962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31f0071990 msgr2=0x7f31f006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:10.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31f0071990 0x7f31f006c420 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f31e000e970 tx=0x7f31e000ed30 comp rx=0 tx=0).stop 2023-03-03T14:22:10.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 -- 172.21.15.7:0/669657962 shutdown_connections 2023-03-03T14:22:10.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31f006c960 0x7f31f0148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:10.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31d40682f0 0x7f31d406a7a0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:10.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31f0071990 0x7f31f006c420 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:10.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 --2- 172.21.15.7:0/669657962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31f0072470 0x7f31f0147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:10.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 -- 172.21.15.7:0/669657962 >> 172.21.15.7:0/669657962 conn(0x7f31f00a94d0 msgr2=0x7f31f00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:22:10.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 -- 172.21.15.7:0/669657962 shutdown_connections 2023-03-03T14:22:10.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:10.060+0000 7f31f72be700 1 -- 172.21.15.7:0/669657962 wait complete. 2023-03-03T14:22:10.063 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:22:10.073 INFO:tasks.mds_thrash.fs.[b]:waiting for 88 secs before thrashing 2023-03-03T14:22:13.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:22:13.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.568+0000 7f3ce50ca700 1 Processor -- start 2023-03-03T14:22:13.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.568+0000 7f3ce50ca700 1 -- start start 2023-03-03T14:22:13.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3ce50ca700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3ce0155390 0x7f3ce00c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:13.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3ce50ca700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce0154910 con 0x7f3ce0151770 2023-03-03T14:22:13.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3ce50ca700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce01522e0 con 0x7f3ce006e020 2023-03-03T14:22:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3ce50ca700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce0070b90 con 0x7f3ce0155390 2023-03-03T14:22:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3cded9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3ce0155390 0x7f3ce00c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3cdf59e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3ce0151770 0x7f3ce0151bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42454/0 (socket says 172.21.15.7:42454) 2023-03-03T14:22:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3cdf59e700 1 -- 172.21.15.7:0/2346721242 learned_addr learned my addr 172.21.15.7:0/2346721242 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:22:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3cded9d700 1 -- 172.21.15.7:0/2346721242 >> v1:172.21.15.143:6789/0 conn(0x7f3ce006e020 legacy=0x7f3ce0071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:22:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3cded9d700 1 -- 172.21.15.7:0/2346721242 >> v1:172.21.15.7:6789/0 conn(0x7f3ce0151770 legacy=0x7f3ce0151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:13.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3cded9d700 1 -- 172.21.15.7:0/2346721242 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce00702a0 con 0x7f3ce0155390 2023-03-03T14:22:13.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3cded9d700 1 --2- 172.21.15.7:0/2346721242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0155390 0x7f3ce00c3e30 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0009840 tx=0x7f3cd0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27bb4018cdedbca2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:13.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.569+0000 7f3cddd9b700 1 -- 172.21.15.7:0/2346721242 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0004230 con 0x7f3ce0155390 2023-03-03T14:22:13.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.570+0000 7f3cddd9b700 1 -- 172.21.15.7:0/2346721242 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd0004390 con 0x7f3ce0155390 2023-03-03T14:22:13.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.570+0000 7f3cddd9b700 1 -- 172.21.15.7:0/2346721242 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd00036a0 con 0x7f3ce0155390 2023-03-03T14:22:13.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.570+0000 7f3ce50ca700 1 -- 172.21.15.7:0/2346721242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0155390 msgr2=0x7f3ce00c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:13.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.570+0000 7f3ce50ca700 1 --2- 172.21.15.7:0/2346721242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0155390 0x7f3ce00c3e30 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0009840 tx=0x7f3cd0009b50 comp rx=0 tx=0).stop 2023-03-03T14:22:13.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.570+0000 7f3ce50ca700 1 -- 172.21.15.7:0/2346721242 shutdown_connections 2023-03-03T14:22:13.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.570+0000 7f3ce50ca700 1 --2- 172.21.15.7:0/2346721242 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0155390 0x7f3ce00c3e30 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:13.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.570+0000 7f3ce50ca700 1 -- 172.21.15.7:0/2346721242 >> 172.21.15.7:0/2346721242 conn(0x7f3ce006c110 msgr2=0x7f3ce006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:22:13.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.570+0000 7f3ce50ca700 1 -- 172.21.15.7:0/2346721242 shutdown_connections 2023-03-03T14:22:13.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.570+0000 7f3ce50ca700 1 -- 172.21.15.7:0/2346721242 wait complete. 2023-03-03T14:22:13.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.572+0000 7f3ce50ca700 1 Processor -- start 2023-03-03T14:22:13.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.572+0000 7f3ce50ca700 1 -- start start 2023-03-03T14:22:13.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.572+0000 7f3ce50ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ce006e020 0x7f3ce014dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:13.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.572+0000 7f3ce50ca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0151770 0x7f3ce0150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:13.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.572+0000 7f3ce50ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ce0155390 0x7f3ce014b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:13.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.572+0000 7f3ce50ca700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce00abdc0 con 0x7f3ce0151770 2023-03-03T14:22:13.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.572+0000 7f3ce50ca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce00ab930 con 0x7f3ce0155390 2023-03-03T14:22:13.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.572+0000 7f3ce50ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce00ab000 con 0x7f3ce006e020 2023-03-03T14:22:13.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.572+0000 7f3cde59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0151770 0x7f3ce0150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:13.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cde59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0151770 0x7f3ce0150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48480/0 (socket says 172.21.15.7:48480) 2023-03-03T14:22:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cde59c700 1 -- 172.21.15.7:0/1304135294 learned_addr learned my addr 172.21.15.7:0/1304135294 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:22:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cded9d700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ce006e020 0x7f3ce014dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cdf59e700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ce0155390 0x7f3ce014b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cde59c700 1 -- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ce006e020 msgr2=0x7f3ce014dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cde59c700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ce006e020 0x7f3ce014dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cde59c700 1 -- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ce0155390 msgr2=0x7f3ce014b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cde59c700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ce0155390 0x7f3ce014b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:13.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cde59c700 1 -- 172.21.15.7:0/1304135294 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd00094a0 con 0x7f3ce0151770 2023-03-03T14:22:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cdf59e700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ce0155390 0x7f3ce014b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:22:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cde59c700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0151770 0x7f3ce0150200 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c4a0 tx=0x7f3cd400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cc77fe700 1 -- 172.21.15.7:0/1304135294 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40054e0 con 0x7f3ce0151770 2023-03-03T14:22:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cc77fe700 1 -- 172.21.15.7:0/1304135294 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd4003680 con 0x7f3ce0151770 2023-03-03T14:22:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3ce50ca700 1 -- 172.21.15.7:0/1304135294 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce0070820 con 0x7f3ce0151770 2023-03-03T14:22:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3cc77fe700 1 -- 172.21.15.7:0/1304135294 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4003a30 con 0x7f3ce0151770 2023-03-03T14:22:13.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.573+0000 7f3ce50ca700 1 -- 172.21.15.7:0/1304135294 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce0070540 con 0x7f3ce0151770 2023-03-03T14:22:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.574+0000 7f3ce50ca700 1 -- 172.21.15.7:0/1304135294 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce0155760 con 0x7f3ce0151770 2023-03-03T14:22:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.575+0000 7f3cc77fe700 1 -- 172.21.15.7:0/1304135294 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3cd4019030 con 0x7f3ce0151770 2023-03-03T14:22:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.575+0000 7f3cc77fe700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ccc0682f0 0x7f3ccc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:13.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.575+0000 7f3cc77fe700 1 -- 172.21.15.7:0/1304135294 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3cd4085550 con 0x7f3ce0151770 2023-03-03T14:22:13.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.575+0000 7f3cded9d700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ccc0682f0 0x7f3ccc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:13.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.576+0000 7f3cded9d700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ccc0682f0 0x7f3ccc06a7a0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00090b0 tx=0x7f3cd0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:13.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.579+0000 7f3cc77fe700 1 -- 172.21.15.7:0/1304135294 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3cd40518f0 con 0x7f3ce0151770 2023-03-03T14:22:13.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.820+0000 7f3ce50ca700 1 -- 172.21.15.7:0/1304135294 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ce0074bb0 con 0x7f3ce0151770 2023-03-03T14:22:13.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.820+0000 7f3cc77fe700 1 -- 172.21.15.7:0/1304135294 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3cd4054f10 con 0x7f3ce0151770 2023-03-03T14:22:13.822 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:22:13.822 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:22:13.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 -- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ccc0682f0 msgr2=0x7f3ccc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:13.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ccc0682f0 0x7f3ccc06a7a0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00090b0 tx=0x7f3cd0008030 comp rx=0 tx=0).stop 2023-03-03T14:22:13.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 -- 172.21.15.7:0/1304135294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0151770 msgr2=0x7f3ce0150200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:13.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0151770 0x7f3ce0150200 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c4a0 tx=0x7f3cd400c860 comp rx=0 tx=0).stop 2023-03-03T14:22:13.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 -- 172.21.15.7:0/1304135294 shutdown_connections 2023-03-03T14:22:13.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ce0155390 0x7f3ce014b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:13.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ce006e020 0x7f3ce014dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:13.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ccc0682f0 0x7f3ccc06a7a0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:13.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 --2- 172.21.15.7:0/1304135294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ce0151770 0x7f3ce0150200 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:13.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 -- 172.21.15.7:0/1304135294 >> 172.21.15.7:0/1304135294 conn(0x7f3ce006c110 msgr2=0x7f3ce00c6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:22:13.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 -- 172.21.15.7:0/1304135294 shutdown_connections 2023-03-03T14:22:13.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:13.822+0000 7f3ce50ca700 1 -- 172.21.15.7:0/1304135294 wait complete. 2023-03-03T14:22:13.826 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:22:13.835 INFO:tasks.mds_thrash.fs.[a]:waiting for 12 secs before thrashing 2023-03-03T14:22:22.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:22.740 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:25.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:22:25.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.974+0000 7f013139e700 1 Processor -- start 2023-03-03T14:22:25.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.974+0000 7f013139e700 1 -- start start 2023-03-03T14:22:25.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f013139e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f012c0c0ec0 0x7f012c06d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:25.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f013139e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f012c06da90 0x7f012c151660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:25.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f013139e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c06dff0 con 0x7f012c0c1290 2023-03-03T14:22:25.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f013139e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f012c06e2d0 con 0x7f012c06da90 2023-03-03T14:22:25.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f013139e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012c06bc60 con 0x7f012c0c0ec0 2023-03-03T14:22:25.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f0130b9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f012c06da90 0x7f012c151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:25.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f0130b9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f012c06da90 0x7f012c151660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48500/0 (socket says 172.21.15.7:48500) 2023-03-03T14:22:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f0130b9d700 1 -- 172.21.15.7:0/516472855 learned_addr learned my addr 172.21.15.7:0/516472855 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:22:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f0130b9d700 1 -- 172.21.15.7:0/516472855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f012c0c0ec0 msgr2=0x7f012c06d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:22:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f0130b9d700 1 --2- 172.21.15.7:0/516472855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f012c0c0ec0 0x7f012c06d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f0130b9d700 1 -- 172.21.15.7:0/516472855 >> v1:172.21.15.143:6789/0 conn(0x7f012c0c1290 legacy=0x7f012c1577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:22:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f0130b9d700 1 -- 172.21.15.7:0/516472855 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c1543d0 con 0x7f012c06da90 2023-03-03T14:22:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.975+0000 7f0130b9d700 1 --2- 172.21.15.7:0/516472855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c06da90 0x7f012c151660 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f0124009e50 tx=0x7f012400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd35b5cb2df85b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:25.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f012affd700 1 -- 172.21.15.7:0/516472855 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012400e070 con 0x7f012c06da90 2023-03-03T14:22:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f012affd700 1 -- 172.21.15.7:0/516472855 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0124003a20 con 0x7f012c06da90 2023-03-03T14:22:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f012affd700 1 -- 172.21.15.7:0/516472855 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124003dd0 con 0x7f012c06da90 2023-03-03T14:22:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f013139e700 1 -- 172.21.15.7:0/516472855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c06da90 msgr2=0x7f012c151660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f013139e700 1 --2- 172.21.15.7:0/516472855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c06da90 0x7f012c151660 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f0124009e50 tx=0x7f012400b330 comp rx=0 tx=0).stop 2023-03-03T14:22:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f013139e700 1 -- 172.21.15.7:0/516472855 shutdown_connections 2023-03-03T14:22:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f013139e700 1 --2- 172.21.15.7:0/516472855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c06da90 0x7f012c151660 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:25.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f013139e700 1 --2- 172.21.15.7:0/516472855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f012c0c0ec0 0x7f012c06d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f013139e700 1 -- 172.21.15.7:0/516472855 >> 172.21.15.7:0/516472855 conn(0x7f012c0c7dc0 msgr2=0x7f012c0c3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:22:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f013139e700 1 -- 172.21.15.7:0/516472855 shutdown_connections 2023-03-03T14:22:25.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.976+0000 7f013139e700 1 -- 172.21.15.7:0/516472855 wait complete. 2023-03-03T14:22:25.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.978+0000 7f013139e700 1 Processor -- start 2023-03-03T14:22:25.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.978+0000 7f013139e700 1 -- start start 2023-03-03T14:22:25.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f013139e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f012c06da90 0x7f012c0ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:25.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f013139e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c0c0ec0 0x7f012c06aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:25.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f013139e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f012c120d30 0x7f012c1ecda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:25.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f013139e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012c120890 con 0x7f012c0c0ec0 2023-03-03T14:22:25.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f013139e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012c154200 con 0x7f012c120d30 2023-03-03T14:22:25.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f013139e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012c0aaea0 con 0x7f012c06da90 2023-03-03T14:22:25.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f012b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c0c0ec0 0x7f012c06aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:25.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f012b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c0c0ec0 0x7f012c06aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48504/0 (socket says 172.21.15.7:48504) 2023-03-03T14:22:25.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f012b7fe700 1 -- 172.21.15.7:0/3164600076 learned_addr learned my addr 172.21.15.7:0/3164600076 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:22:25.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f0130b9d700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f012c120d30 0x7f012c1ecda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:25.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f012bfff700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f012c06da90 0x7f012c0ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:25.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f012b7fe700 1 -- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f012c06da90 msgr2=0x7f012c0ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:25.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f012b7fe700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f012c06da90 0x7f012c0ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:25.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f012b7fe700 1 -- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f012c120d30 msgr2=0x7f012c1ecda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:25.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f012b7fe700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f012c120d30 0x7f012c1ecda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:25.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.979+0000 7f012b7fe700 1 -- 172.21.15.7:0/3164600076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124009b30 con 0x7f012c0c0ec0 2023-03-03T14:22:25.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.980+0000 7f0130b9d700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f012c120d30 0x7f012c1ecda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:22:25.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.980+0000 7f012bfff700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f012c06da90 0x7f012c0ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:22:25.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.980+0000 7f012b7fe700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c0c0ec0 0x7f012c06aac0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f011400b4a0 tx=0x7f011400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:25.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.980+0000 7f0128ff9700 1 -- 172.21.15.7:0/3164600076 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0114011830 con 0x7f012c0c0ec0 2023-03-03T14:22:25.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.980+0000 7f0128ff9700 1 -- 172.21.15.7:0/3164600076 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01140119b0 con 0x7f012c0c0ec0 2023-03-03T14:22:25.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.980+0000 7f013139e700 1 -- 172.21.15.7:0/3164600076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012c154990 con 0x7f012c0c0ec0 2023-03-03T14:22:25.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.980+0000 7f0128ff9700 1 -- 172.21.15.7:0/3164600076 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0114011d60 con 0x7f012c0c0ec0 2023-03-03T14:22:25.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.980+0000 7f013139e700 1 -- 172.21.15.7:0/3164600076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012c1546b0 con 0x7f012c0c0ec0 2023-03-03T14:22:25.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.981+0000 7f013139e700 1 -- 172.21.15.7:0/3164600076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012c157ed0 con 0x7f012c0c0ec0 2023-03-03T14:22:25.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.981+0000 7f0128ff9700 1 -- 172.21.15.7:0/3164600076 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0114021070 con 0x7f012c0c0ec0 2023-03-03T14:22:25.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.982+0000 7f0128ff9700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f010c0682f0 0x7f010c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:22:25.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.982+0000 7f0128ff9700 1 -- 172.21.15.7:0/3164600076 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0114086070 con 0x7f012c0c0ec0 2023-03-03T14:22:25.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.982+0000 7f012bfff700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f010c0682f0 0x7f010c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:22:25.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.982+0000 7f012bfff700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f010c0682f0 0x7f010c06a7a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f011c0099d0 tx=0x7f011c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:22:25.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:25.986+0000 7f0128ff9700 1 -- 172.21.15.7:0/3164600076 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0114052410 con 0x7f012c0c0ec0 2023-03-03T14:22:26.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.223+0000 7f013139e700 1 -- 172.21.15.7:0/3164600076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f012c120350 con 0x7f012c0c0ec0 2023-03-03T14:22:26.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.224+0000 7f0128ff9700 1 -- 172.21.15.7:0/3164600076 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f012c120350 con 0x7f012c0c0ec0 2023-03-03T14:22:26.225 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:22:26.226 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:22:26.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 -- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f010c0682f0 msgr2=0x7f010c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:26.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f010c0682f0 0x7f010c06a7a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f011c0099d0 tx=0x7f011c008040 comp rx=0 tx=0).stop 2023-03-03T14:22:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 -- 172.21.15.7:0/3164600076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c0c0ec0 msgr2=0x7f012c06aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:22:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c0c0ec0 0x7f012c06aac0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f011400b4a0 tx=0x7f011400b860 comp rx=0 tx=0).stop 2023-03-03T14:22:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 -- 172.21.15.7:0/3164600076 shutdown_connections 2023-03-03T14:22:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f012c120d30 0x7f012c1ecda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f012c06da90 0x7f012c0ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:26.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f010c0682f0 0x7f010c06a7a0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:26.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 --2- 172.21.15.7:0/3164600076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f012c0c0ec0 0x7f012c06aac0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:22:26.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 -- 172.21.15.7:0/3164600076 >> 172.21.15.7:0/3164600076 conn(0x7f012c0c7dc0 msgr2=0x7f012c0c19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:22:26.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 -- 172.21.15.7:0/3164600076 shutdown_connections 2023-03-03T14:22:26.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:22:26.226+0000 7f013139e700 1 -- 172.21.15.7:0/3164600076 wait complete. 2023-03-03T14:22:26.229 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:22:26.239 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T14:22:52.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:22:52.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:08.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:23:08.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.377+0000 7f89dd3ec700 1 Processor -- start 2023-03-03T14:23:08.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.377+0000 7f89dd3ec700 1 -- start start 2023-03-03T14:23:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.378+0000 7f89dd3ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89d80c00e0 0x7f89d80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.378+0000 7f89dd3ec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89d8154ff0 0x7f89d80cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.378+0000 7f89dd3ec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d80c7310 con 0x7f89d80caa30 2023-03-03T14:23:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.378+0000 7f89dd3ec700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89d80bfd10 con 0x7f89d8154ff0 2023-03-03T14:23:08.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.378+0000 7f89dd3ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d80bfa30 con 0x7f89d80c00e0 2023-03-03T14:23:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.378+0000 7f89d77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89d8154ff0 0x7f89d80cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.378+0000 7f89d77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89d8154ff0 0x7f89d80cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48526/0 (socket says 172.21.15.7:48526) 2023-03-03T14:23:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.378+0000 7f89d77fe700 1 -- 172.21.15.7:0/3754458015 learned_addr learned my addr 172.21.15.7:0/3754458015 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:23:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.378+0000 7f89d67fc700 1 --2- 172.21.15.7:0/3754458015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89d80c00e0 0x7f89d80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89d77fe700 1 -- 172.21.15.7:0/3754458015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89d80c00e0 msgr2=0x7f89d80c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:23:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89d77fe700 1 --2- 172.21.15.7:0/3754458015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89d80c00e0 0x7f89d80c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:08.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89d77fe700 1 -- 172.21.15.7:0/3754458015 >> v1:172.21.15.143:6789/0 conn(0x7f89d80caa30 legacy=0x7f89d80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89d77fe700 1 -- 172.21.15.7:0/3754458015 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d8154510 con 0x7f89d8154ff0 2023-03-03T14:23:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89d77fe700 1 --2- 172.21.15.7:0/3754458015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d8154ff0 0x7f89d80cae00 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f89d0009e50 tx=0x7f89d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9030f4df3b7744e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89d5ffb700 1 -- 172.21.15.7:0/3754458015 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d000e070 con 0x7f89d8154ff0 2023-03-03T14:23:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89d5ffb700 1 -- 172.21.15.7:0/3754458015 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d000b9c0 con 0x7f89d8154ff0 2023-03-03T14:23:08.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89d5ffb700 1 -- 172.21.15.7:0/3754458015 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d000bd70 con 0x7f89d8154ff0 2023-03-03T14:23:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89dd3ec700 1 -- 172.21.15.7:0/3754458015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d8154ff0 msgr2=0x7f89d80cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89dd3ec700 1 --2- 172.21.15.7:0/3754458015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d8154ff0 0x7f89d80cae00 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f89d0009e50 tx=0x7f89d000b330 comp rx=0 tx=0).stop 2023-03-03T14:23:08.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89dd3ec700 1 -- 172.21.15.7:0/3754458015 shutdown_connections 2023-03-03T14:23:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89dd3ec700 1 --2- 172.21.15.7:0/3754458015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d8154ff0 0x7f89d80cae00 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89dd3ec700 1 --2- 172.21.15.7:0/3754458015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89d80c00e0 0x7f89d80c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.379+0000 7f89dd3ec700 1 -- 172.21.15.7:0/3754458015 >> 172.21.15.7:0/3754458015 conn(0x7f89d8066e40 msgr2=0x7f89d8071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:23:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.380+0000 7f89dd3ec700 1 -- 172.21.15.7:0/3754458015 shutdown_connections 2023-03-03T14:23:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.380+0000 7f89dd3ec700 1 -- 172.21.15.7:0/3754458015 wait complete. 2023-03-03T14:23:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.381+0000 7f89dd3ec700 1 Processor -- start 2023-03-03T14:23:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.381+0000 7f89dd3ec700 1 -- start start 2023-03-03T14:23:08.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89dd3ec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89d80c00e0 0x7f89d814b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89dd3ec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d80caa30 0x7f89d8157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89dd3ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89d8154ff0 0x7f89d8148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89dd3ec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d814a4d0 con 0x7f89d80caa30 2023-03-03T14:23:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89dd3ec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d814a7b0 con 0x7f89d80c00e0 2023-03-03T14:23:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89dd3ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d8157230 con 0x7f89d8154ff0 2023-03-03T14:23:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d67fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d80caa30 0x7f89d8157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:08.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d67fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d80caa30 0x7f89d8157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48530/0 (socket says 172.21.15.7:48530) 2023-03-03T14:23:08.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d67fc700 1 -- 172.21.15.7:0/100840068 learned_addr learned my addr 172.21.15.7:0/100840068 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:23:08.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d77fe700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89d8154ff0 0x7f89d8148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:08.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d6ffd700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89d80c00e0 0x7f89d814b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:08.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d67fc700 1 -- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89d8154ff0 msgr2=0x7f89d8148930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:08.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d67fc700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89d8154ff0 0x7f89d8148930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:08.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d67fc700 1 -- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89d80c00e0 msgr2=0x7f89d814b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:08.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d67fc700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89d80c00e0 0x7f89d814b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:08.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d67fc700 1 -- 172.21.15.7:0/100840068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d0009b30 con 0x7f89d80caa30 2023-03-03T14:23:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89d67fc700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d80caa30 0x7f89d8157d10 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f89c800d7d0 tx=0x7f89c800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89cf7fe700 1 -- 172.21.15.7:0/100840068 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c8017070 con 0x7f89d80caa30 2023-03-03T14:23:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89cf7fe700 1 -- 172.21.15.7:0/100840068 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c8005a20 con 0x7f89d80caa30 2023-03-03T14:23:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.382+0000 7f89dd3ec700 1 -- 172.21.15.7:0/100840068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d8154ad0 con 0x7f89d80caa30 2023-03-03T14:23:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.383+0000 7f89cf7fe700 1 -- 172.21.15.7:0/100840068 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c8005bd0 con 0x7f89d80caa30 2023-03-03T14:23:08.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.383+0000 7f89dd3ec700 1 -- 172.21.15.7:0/100840068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d81547f0 con 0x7f89d80caa30 2023-03-03T14:23:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.384+0000 7f89dd3ec700 1 -- 172.21.15.7:0/100840068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d80bf650 con 0x7f89d80caa30 2023-03-03T14:23:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.384+0000 7f89cf7fe700 1 -- 172.21.15.7:0/100840068 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f89c8005d30 con 0x7f89d80caa30 2023-03-03T14:23:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.384+0000 7f89cf7fe700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89b80682f0 0x7f89b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:08.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.385+0000 7f89cf7fe700 1 -- 172.21.15.7:0/100840068 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f89c8085c90 con 0x7f89d80caa30 2023-03-03T14:23:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.385+0000 7f89d6ffd700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89b80682f0 0x7f89b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.385+0000 7f89d6ffd700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89b80682f0 0x7f89b806a7a0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f89d80af1a0 tx=0x7f89c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:08.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.389+0000 7f89cf7fe700 1 -- 172.21.15.7:0/100840068 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f89c8052030 con 0x7f89d80caa30 2023-03-03T14:23:08.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.624+0000 7f89dd3ec700 1 -- 172.21.15.7:0/100840068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89d8157b60 con 0x7f89d80caa30 2023-03-03T14:23:08.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.624+0000 7f89cf7fe700 1 -- 172.21.15.7:0/100840068 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f89c8055650 con 0x7f89d80caa30 2023-03-03T14:23:08.627 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:23:08.627 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:23:08.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 -- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89b80682f0 msgr2=0x7f89b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:08.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89b80682f0 0x7f89b806a7a0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f89d80af1a0 tx=0x7f89c0009040 comp rx=0 tx=0).stop 2023-03-03T14:23:08.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 -- 172.21.15.7:0/100840068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d80caa30 msgr2=0x7f89d8157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:08.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d80caa30 0x7f89d8157d10 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f89c800d7d0 tx=0x7f89c800dae0 comp rx=0 tx=0).stop 2023-03-03T14:23:08.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 -- 172.21.15.7:0/100840068 shutdown_connections 2023-03-03T14:23:08.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f89d8154ff0 0x7f89d8148930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:08.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89d80c00e0 0x7f89d814b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:08.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89b80682f0 0x7f89b806a7a0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:08.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 --2- 172.21.15.7:0/100840068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89d80caa30 0x7f89d8157d10 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:08.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 -- 172.21.15.7:0/100840068 >> 172.21.15.7:0/100840068 conn(0x7f89d8066e40 msgr2=0x7f89d80c68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:23:08.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.626+0000 7f89dd3ec700 1 -- 172.21.15.7:0/100840068 shutdown_connections 2023-03-03T14:23:08.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:08.627+0000 7f89dd3ec700 1 -- 172.21.15.7:0/100840068 wait complete. 2023-03-03T14:23:08.631 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:23:08.641 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2023-03-03T14:23:22.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:22.859 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:38.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:23:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.211+0000 7fe7df327700 1 Processor -- start 2023-03-03T14:23:38.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.212+0000 7fe7df327700 1 -- start start 2023-03-03T14:23:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.212+0000 7fe7df327700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe7d8151b20 0x7fe7d80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.212+0000 7fe7df327700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d806cb70 con 0x7fe7d80c2950 2023-03-03T14:23:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.212+0000 7fe7df327700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d806ce50 con 0x7fe7d8073c30 2023-03-03T14:23:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.212+0000 7fe7df327700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d806c2f0 con 0x7fe7d8151b20 2023-03-03T14:23:38.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.212+0000 7fe7dc8c2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe7d80c2950 0x7fe7d8157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42538/0 (socket says 172.21.15.7:42538) 2023-03-03T14:23:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.212+0000 7fe7dc8c2700 1 -- 172.21.15.7:0/3819920366 learned_addr learned my addr 172.21.15.7:0/3819920366 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:23:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7dd0c3700 1 --2- 172.21.15.7:0/3819920366 >> v2:172.21.15.143:3300/0 conn(0x7fe7d8151b20 0x7fe7d80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422342012 0 0) 0x7fe7d806cb70 con 0x7fe7d80c2950 2023-03-03T14:23:38.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c40035b0 con 0x7fe7d80c2950 2023-03-03T14:23:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441841142 0 0) 0x7fe7c40035b0 con 0x7fe7d80c2950 2023-03-03T14:23:38.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 >> v2:172.21.15.143:3300/0 conn(0x7fe7d8151b20 msgr2=0x7fe7d80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:23:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 --2- 172.21.15.7:0/3819920366 >> v2:172.21.15.143:3300/0 conn(0x7fe7d8151b20 0x7fe7d80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 >> v1:172.21.15.143:6789/0 conn(0x7fe7d8073c30 legacy=0x7fe7d81581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d806d130 con 0x7fe7d80c2950 2023-03-03T14:23:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe7cc001670 con 0x7fe7d80c2950 2023-03-03T14:23:38.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 >> v1:172.21.15.7:6789/0 conn(0x7fe7d80c2950 legacy=0x7fe7d8157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:38.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c4003a70 0x7fe7c4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:38.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7c4004850 0x7fe7c4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:38.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7c4005270 0x7fe7c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:38.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d806c2f0 con 0x7fe7c4003a70 2023-03-03T14:23:38.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c400c1b0 con 0x7fe7c4005270 2023-03-03T14:23:38.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c400c370 con 0x7fe7c4004850 2023-03-03T14:23:38.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7dd0c3700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c4003a70 0x7fe7c4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7dc8c2700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7c4004850 0x7fe7c4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.213+0000 7fe7dd8c4700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7c4005270 0x7fe7c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7dd0c3700 1 -- 172.21.15.7:0/3819920366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7c4004850 msgr2=0x7fe7c4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:23:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7dd0c3700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7c4004850 0x7fe7c4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7dd0c3700 1 -- 172.21.15.7:0/3819920366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7c4005270 msgr2=0x7fe7c400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:23:38.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7dd0c3700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7c4005270 0x7fe7c400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:38.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7dd8c4700 1 -- 172.21.15.7:0/3819920366 reap_dead start 2023-03-03T14:23:38.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7dd0c3700 1 -- 172.21.15.7:0/3819920366 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c40035b0 con 0x7fe7c4003a70 2023-03-03T14:23:38.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7dc8c2700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7c4004850 0x7fe7c4004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:23:38.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7dd0c3700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c4003a70 0x7fe7c4003e40 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009b60 tx=0x7fe7c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b1052928d134b0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:38.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d400e6c0 con 0x7fe7c4003a70 2023-03-03T14:23:38.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d400e820 con 0x7fe7c4003a70 2023-03-03T14:23:38.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7cbfff700 1 -- 172.21.15.7:0/3819920366 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d400eb90 con 0x7fe7c4003a70 2023-03-03T14:23:38.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7df327700 1 -- 172.21.15.7:0/3819920366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c4003a70 msgr2=0x7fe7c4003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:38.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7df327700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c4003a70 0x7fe7c4003e40 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009b60 tx=0x7fe7c4005880 comp rx=0 tx=0).stop 2023-03-03T14:23:38.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7df327700 1 -- 172.21.15.7:0/3819920366 shutdown_connections 2023-03-03T14:23:38.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7df327700 1 --2- 172.21.15.7:0/3819920366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c4003a70 0x7fe7c4003e40 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:38.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.214+0000 7fe7df327700 1 -- 172.21.15.7:0/3819920366 >> 172.21.15.7:0/3819920366 conn(0x7fe7d80a94d0 msgr2=0x7fe7d80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:23:38.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.215+0000 7fe7df327700 1 -- 172.21.15.7:0/3819920366 shutdown_connections 2023-03-03T14:23:38.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.215+0000 7fe7df327700 1 -- 172.21.15.7:0/3819920366 wait complete. 2023-03-03T14:23:38.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.216+0000 7fe7df327700 1 Processor -- start 2023-03-03T14:23:38.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.216+0000 7fe7df327700 1 -- start start 2023-03-03T14:23:38.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.216+0000 7fe7df327700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7c4003a70 0x7fe7d80acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:38.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.216+0000 7fe7df327700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7d80aba20 0x7fe7d81ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:38.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.216+0000 7fe7df327700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80aa260 0x7fe7d8152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:38.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.216+0000 7fe7df327700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80aac10 con 0x7fe7d80aba20 2023-03-03T14:23:38.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.216+0000 7fe7df327700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80abf20 con 0x7fe7d80aa260 2023-03-03T14:23:38.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.216+0000 7fe7df327700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80ac200 con 0x7fe7c4003a70 2023-03-03T14:23:38.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dc8c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7d80aba20 0x7fe7d81ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:38.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dc8c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7d80aba20 0x7fe7d81ea4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48570/0 (socket says 172.21.15.7:48570) 2023-03-03T14:23:38.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dc8c2700 1 -- 172.21.15.7:0/3958962094 learned_addr learned my addr 172.21.15.7:0/3958962094 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:23:38.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dd0c3700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7c4003a70 0x7fe7d80acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:38.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dd8c4700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80aa260 0x7fe7d8152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:38.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dc8c2700 1 -- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7c4003a70 msgr2=0x7fe7d80acfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:38.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dc8c2700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7c4003a70 0x7fe7d80acfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:38.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dc8c2700 1 -- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80aa260 msgr2=0x7fe7d8152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:38.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dc8c2700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80aa260 0x7fe7d8152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:38.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dc8c2700 1 -- 172.21.15.7:0/3958962094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d8151ef0 con 0x7fe7d80aba20 2023-03-03T14:23:38.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dd8c4700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80aa260 0x7fe7d8152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:23:38.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7dc8c2700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7d80aba20 0x7fe7d81ea4a0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0010e0 tx=0x7fe7cc00ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:38.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7c9ffb700 1 -- 172.21.15.7:0/3958962094 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc010510 con 0x7fe7d80aba20 2023-03-03T14:23:38.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7c9ffb700 1 -- 172.21.15.7:0/3958962094 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc010670 con 0x7fe7d80aba20 2023-03-03T14:23:38.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7df327700 1 -- 172.21.15.7:0/3958962094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d806d410 con 0x7fe7d80aba20 2023-03-03T14:23:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.217+0000 7fe7c9ffb700 1 -- 172.21.15.7:0/3958962094 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc010a20 con 0x7fe7d80aba20 2023-03-03T14:23:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.218+0000 7fe7df327700 1 -- 172.21.15.7:0/3958962094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d806bea0 con 0x7fe7d80aba20 2023-03-03T14:23:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.219+0000 7fe7df327700 1 -- 172.21.15.7:0/3958962094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d80ace10 con 0x7fe7d80aba20 2023-03-03T14:23:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.219+0000 7fe7c9ffb700 1 -- 172.21.15.7:0/3958962094 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe7cc01b030 con 0x7fe7d80aba20 2023-03-03T14:23:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.219+0000 7fe7c9ffb700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7bc0682f0 0x7fe7bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.219+0000 7fe7c9ffb700 1 -- 172.21.15.7:0/3958962094 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe7cc085760 con 0x7fe7d80aba20 2023-03-03T14:23:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.220+0000 7fe7dd0c3700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7bc0682f0 0x7fe7bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:38.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.220+0000 7fe7dd0c3700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7bc0682f0 0x7fe7bc06a7a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40090b0 tx=0x7fe7d401f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:38.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.224+0000 7fe7c9ffb700 1 -- 172.21.15.7:0/3958962094 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe7cc04d5e0 con 0x7fe7d80aba20 2023-03-03T14:23:38.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.459+0000 7fe7df327700 1 -- 172.21.15.7:0/3958962094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7d8074bb0 con 0x7fe7d80aba20 2023-03-03T14:23:38.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.459+0000 7fe7c9ffb700 1 -- 172.21.15.7:0/3958962094 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe7cc019090 con 0x7fe7d80aba20 2023-03-03T14:23:38.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:23:38.461 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:23:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 -- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7bc0682f0 msgr2=0x7fe7bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7bc0682f0 0x7fe7bc06a7a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40090b0 tx=0x7fe7d401f040 comp rx=0 tx=0).stop 2023-03-03T14:23:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 -- 172.21.15.7:0/3958962094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7d80aba20 msgr2=0x7fe7d81ea4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7d80aba20 0x7fe7d81ea4a0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0010e0 tx=0x7fe7cc00ea60 comp rx=0 tx=0).stop 2023-03-03T14:23:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 -- 172.21.15.7:0/3958962094 shutdown_connections 2023-03-03T14:23:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7d80aa260 0x7fe7d8152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7c4003a70 0x7fe7d80acfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7bc0682f0 0x7fe7bc06a7a0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 --2- 172.21.15.7:0/3958962094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7d80aba20 0x7fe7d81ea4a0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 -- 172.21.15.7:0/3958962094 >> 172.21.15.7:0/3958962094 conn(0x7fe7d80a94d0 msgr2=0x7fe7d8071670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:23:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 -- 172.21.15.7:0/3958962094 shutdown_connections 2023-03-03T14:23:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:38.461+0000 7fe7df327700 1 -- 172.21.15.7:0/3958962094 wait complete. 2023-03-03T14:23:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:23:38.474 INFO:tasks.mds_thrash.fs.[b]:waiting for 92 secs before thrashing 2023-03-03T14:23:51.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:23:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.779+0000 7f61f17bd700 1 Processor -- start 2023-03-03T14:23:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.779+0000 7f61f17bd700 1 -- start start 2023-03-03T14:23:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61f17bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0caa30 0x7f61ec0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61f17bd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61ec0c00e0 0x7f61ec154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61f17bd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f61ec0cae00 0x7f61ec0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61f17bd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61ec0cb280 con 0x7f61ec0cae00 2023-03-03T14:23:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61f17bd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61ec0bfd10 con 0x7f61ec0c00e0 2023-03-03T14:23:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61f17bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec0bfa30 con 0x7f61ec0caa30 2023-03-03T14:23:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61eb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f61ec0cae00 0x7f61ec0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61eb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f61ec0cae00 0x7f61ec0c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48592/0 (socket says 172.21.15.7:48592) 2023-03-03T14:23:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61eb7fe700 1 -- 172.21.15.7:0/3001510734 learned_addr learned my addr 172.21.15.7:0/3001510734 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:23:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61eaffd700 1 --2- 172.21.15.7:0/3001510734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0caa30 0x7f61ec0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61ea7fc700 1 --2- 172.21.15.7:0/3001510734 >> v2:172.21.15.143:3300/0 conn(0x7f61ec0c00e0 0x7f61ec154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61eb7fe700 1 -- 172.21.15.7:0/3001510734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0caa30 msgr2=0x7f61ec0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:23:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61eb7fe700 1 --2- 172.21.15.7:0/3001510734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0caa30 0x7f61ec0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:51.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61eb7fe700 1 -- 172.21.15.7:0/3001510734 >> v2:172.21.15.143:3300/0 conn(0x7f61ec0c00e0 msgr2=0x7f61ec154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:23:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61eb7fe700 1 --2- 172.21.15.7:0/3001510734 >> v2:172.21.15.143:3300/0 conn(0x7f61ec0c00e0 0x7f61ec154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.780+0000 7f61eb7fe700 1 -- 172.21.15.7:0/3001510734 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec154510 con 0x7f61ec0cae00 2023-03-03T14:23:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61eb7fe700 1 --2- 172.21.15.7:0/3001510734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 0x7f61ec0c6c00 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0c81e0 tx=0x7f61e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5deb036be8b0a827 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61e9ffb700 1 -- 172.21.15.7:0/3001510734 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e400e070 con 0x7f61ec0cae00 2023-03-03T14:23:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61e9ffb700 1 -- 172.21.15.7:0/3001510734 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e400b9c0 con 0x7f61ec0cae00 2023-03-03T14:23:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61e9ffb700 1 -- 172.21.15.7:0/3001510734 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e400bd70 con 0x7f61ec0cae00 2023-03-03T14:23:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61f17bd700 1 -- 172.21.15.7:0/3001510734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 msgr2=0x7f61ec0c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61f17bd700 1 --2- 172.21.15.7:0/3001510734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 0x7f61ec0c6c00 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0c81e0 tx=0x7f61e400b330 comp rx=0 tx=0).stop 2023-03-03T14:23:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61f17bd700 1 -- 172.21.15.7:0/3001510734 shutdown_connections 2023-03-03T14:23:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61f17bd700 1 --2- 172.21.15.7:0/3001510734 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 0x7f61ec0c6c00 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61f17bd700 1 --2- 172.21.15.7:0/3001510734 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0caa30 0x7f61ec0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61f17bd700 1 --2- 172.21.15.7:0/3001510734 >> v2:172.21.15.143:3300/0 conn(0x7f61ec0c00e0 0x7f61ec154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61f17bd700 1 -- 172.21.15.7:0/3001510734 >> 172.21.15.7:0/3001510734 conn(0x7f61ec066e40 msgr2=0x7f61ec071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:23:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61f17bd700 1 -- 172.21.15.7:0/3001510734 shutdown_connections 2023-03-03T14:23:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.781+0000 7f61f17bd700 1 -- 172.21.15.7:0/3001510734 wait complete. 2023-03-03T14:23:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.783+0000 7f61f17bd700 1 Processor -- start 2023-03-03T14:23:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.783+0000 7f61f17bd700 1 -- start start 2023-03-03T14:23:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.783+0000 7f61f17bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c00e0 0x7f61ec1e99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.783+0000 7f61f17bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ec0caa30 0x7f61ec0c38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.783+0000 7f61f17bd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 0x7f61ec0c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.783+0000 7f61f17bd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec1de2a0 con 0x7f61ec0cae00 2023-03-03T14:23:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.783+0000 7f61f17bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec0c3440 con 0x7f61ec0caa30 2023-03-03T14:23:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.783+0000 7f61f17bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec1ea230 con 0x7f61ec0c00e0 2023-03-03T14:23:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61eb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 0x7f61ec0c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61eb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 0x7f61ec0c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48598/0 (socket says 172.21.15.7:48598) 2023-03-03T14:23:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61eb7fe700 1 -- 172.21.15.7:0/314038384 learned_addr learned my addr 172.21.15.7:0/314038384 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:23:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61eaffd700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c00e0 0x7f61ec1e99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61ea7fc700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ec0caa30 0x7f61ec0c38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61eb7fe700 1 -- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c00e0 msgr2=0x7f61ec1e99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61eb7fe700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c00e0 0x7f61ec1e99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61eb7fe700 1 -- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ec0caa30 msgr2=0x7f61ec0c38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61eb7fe700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ec0caa30 0x7f61ec0c38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61eb7fe700 1 -- 172.21.15.7:0/314038384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e4009b30 con 0x7f61ec0cae00 2023-03-03T14:23:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61eb7fe700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 0x7f61ec0c3e10 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f61ec06b270 tx=0x7f61e4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61d37fe700 1 -- 172.21.15.7:0/314038384 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4004880 con 0x7f61ec0cae00 2023-03-03T14:23:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61d37fe700 1 -- 172.21.15.7:0/314038384 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e40092e0 con 0x7f61ec0cae00 2023-03-03T14:23:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61d37fe700 1 -- 172.21.15.7:0/314038384 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e401c660 con 0x7f61ec0cae00 2023-03-03T14:23:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61f17bd700 1 -- 172.21.15.7:0/314038384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ec154ad0 con 0x7f61ec0cae00 2023-03-03T14:23:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.784+0000 7f61f17bd700 1 -- 172.21.15.7:0/314038384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ec1547f0 con 0x7f61ec0cae00 2023-03-03T14:23:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.785+0000 7f61f17bd700 1 -- 172.21.15.7:0/314038384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ec0bf650 con 0x7f61ec0cae00 2023-03-03T14:23:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.786+0000 7f61d37fe700 1 -- 172.21.15.7:0/314038384 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f61e4022030 con 0x7f61ec0cae00 2023-03-03T14:23:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.786+0000 7f61d37fe700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61d80682f0 0x7f61d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:23:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.786+0000 7f61d37fe700 1 -- 172.21.15.7:0/314038384 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f61e4087850 con 0x7f61ec0cae00 2023-03-03T14:23:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.786+0000 7f61eaffd700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61d80682f0 0x7f61d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:23:51.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.787+0000 7f61eaffd700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61d80682f0 0x7f61d806a7a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f61e0006b20 tx=0x7f61e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:23:51.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:51.791+0000 7f61d37fe700 1 -- 172.21.15.7:0/314038384 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f61e4053bf0 con 0x7f61ec0cae00 2023-03-03T14:23:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.030+0000 7f61f17bd700 1 -- 172.21.15.7:0/314038384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61ec0c35f0 con 0x7f61ec0cae00 2023-03-03T14:23:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.030+0000 7f61d37fe700 1 -- 172.21.15.7:0/314038384 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f61ec0c35f0 con 0x7f61ec0cae00 2023-03-03T14:23:52.032 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:23:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:23:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 -- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61d80682f0 msgr2=0x7f61d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61d80682f0 0x7f61d806a7a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f61e0006b20 tx=0x7f61e0008040 comp rx=0 tx=0).stop 2023-03-03T14:23:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 -- 172.21.15.7:0/314038384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 msgr2=0x7f61ec0c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:23:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 0x7f61ec0c3e10 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f61ec06b270 tx=0x7f61e4004ad0 comp rx=0 tx=0).stop 2023-03-03T14:23:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 -- 172.21.15.7:0/314038384 shutdown_connections 2023-03-03T14:23:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61d80682f0 0x7f61d806a7a0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ec0cae00 0x7f61ec0c3e10 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ec0c00e0 0x7f61ec1e99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 --2- 172.21.15.7:0/314038384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ec0caa30 0x7f61ec0c38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:23:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 -- 172.21.15.7:0/314038384 >> 172.21.15.7:0/314038384 conn(0x7f61ec066e40 msgr2=0x7f61ec152430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:23:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 -- 172.21.15.7:0/314038384 shutdown_connections 2023-03-03T14:23:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:23:52.033+0000 7f61f17bd700 1 -- 172.21.15.7:0/314038384 wait complete. 2023-03-03T14:23:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:23:52.046 INFO:tasks.mds_thrash.fs.[a]:waiting for 93 secs before thrashing 2023-03-03T14:23:52.919 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:23:52.922 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:22.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:22.995 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:53.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:24:53.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:10.475 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:25:10.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.612+0000 7f4134b1f700 1 Processor -- start 2023-03-03T14:25:10.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.612+0000 7f4134b1f700 1 -- start start 2023-03-03T14:25:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f4134b1f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f413006d870 0x7f41301550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f4134b1f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f41300c1da0 0x7f41300c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f4134b1f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41300c3620 con 0x7f41300c3e30 2023-03-03T14:25:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f4134b1f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41300c2af0 con 0x7f413006d870 2023-03-03T14:25:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f4134b1f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41301545e0 con 0x7f41300c1da0 2023-03-03T14:25:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412e59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f413006d870 0x7f41301550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412ed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f41300c3e30 0x7f4130157ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42654/0 (socket says 172.21.15.7:42654) 2023-03-03T14:25:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412ed9d700 1 -- 172.21.15.7:0/414449562 learned_addr learned my addr 172.21.15.7:0/414449562 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:25:10.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412dd9b700 1 --2- 172.21.15.7:0/414449562 >> v2:172.21.15.143:3300/0 conn(0x7f41300c1da0 0x7f41300c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:10.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412e59c700 1 -- 172.21.15.7:0/414449562 >> v2:172.21.15.143:3300/0 conn(0x7f41300c1da0 msgr2=0x7f41300c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:25:10.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412e59c700 1 --2- 172.21.15.7:0/414449562 >> v2:172.21.15.143:3300/0 conn(0x7f41300c1da0 0x7f41300c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:10.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412d59a700 1 -- 172.21.15.7:0/414449562 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887004272 0 0) 0x7f41300c3620 con 0x7f41300c3e30 2023-03-03T14:25:10.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412e59c700 1 -- 172.21.15.7:0/414449562 >> v1:172.21.15.7:6789/0 conn(0x7f41300c3e30 legacy=0x7f4130157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:10.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412e59c700 1 -- 172.21.15.7:0/414449562 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41301548c0 con 0x7f413006d870 2023-03-03T14:25:10.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412e59c700 1 --2- 172.21.15.7:0/414449562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413006d870 0x7f41301550c0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f41300c33a0 tx=0x7f4118009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a71d7c1c68b685 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:10.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.613+0000 7f412d59a700 1 -- 172.21.15.7:0/414449562 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118011830 con 0x7f413006d870 2023-03-03T14:25:10.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.614+0000 7f412d59a700 1 -- 172.21.15.7:0/414449562 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4118011990 con 0x7f413006d870 2023-03-03T14:25:10.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.614+0000 7f412d59a700 1 -- 172.21.15.7:0/414449562 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118011d20 con 0x7f413006d870 2023-03-03T14:25:10.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.614+0000 7f4134b1f700 1 -- 172.21.15.7:0/414449562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413006d870 msgr2=0x7f41301550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:10.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.614+0000 7f4134b1f700 1 --2- 172.21.15.7:0/414449562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413006d870 0x7f41301550c0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f41300c33a0 tx=0x7f4118009b20 comp rx=0 tx=0).stop 2023-03-03T14:25:10.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.614+0000 7f4134b1f700 1 -- 172.21.15.7:0/414449562 shutdown_connections 2023-03-03T14:25:10.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.614+0000 7f4134b1f700 1 --2- 172.21.15.7:0/414449562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f413006d870 0x7f41301550c0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:10.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.614+0000 7f4134b1f700 1 --2- 172.21.15.7:0/414449562 >> v2:172.21.15.143:3300/0 conn(0x7f41300c1da0 0x7f41300c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:10.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.614+0000 7f4134b1f700 1 -- 172.21.15.7:0/414449562 >> 172.21.15.7:0/414449562 conn(0x7f413006c110 msgr2=0x7f413006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:25:10.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.614+0000 7f4134b1f700 1 -- 172.21.15.7:0/414449562 shutdown_connections 2023-03-03T14:25:10.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.614+0000 7f4134b1f700 1 -- 172.21.15.7:0/414449562 wait complete. 2023-03-03T14:25:10.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.615+0000 7f4134b1f700 1 Processor -- start 2023-03-03T14:25:10.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.616+0000 7f4134b1f700 1 -- start start 2023-03-03T14:25:10.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.616+0000 7f4134b1f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413006d870 0x7f413014e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:10.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.616+0000 7f4134b1f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41300c1da0 0x7f4130155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:10.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.616+0000 7f4134b1f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41300c3e30 0x7f4130156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:10.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.616+0000 7f4134b1f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4130153590 con 0x7f41300c3e30 2023-03-03T14:25:10.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.616+0000 7f4134b1f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41301ea6b0 con 0x7f413006d870 2023-03-03T14:25:10.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.616+0000 7f4134b1f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413014e980 con 0x7f41300c1da0 2023-03-03T14:25:10.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.616+0000 7f412ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41300c3e30 0x7f4130156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:10.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.616+0000 7f412ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41300c3e30 0x7f4130156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48680/0 (socket says 172.21.15.7:48680) 2023-03-03T14:25:10.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.616+0000 7f412ed9d700 1 -- 172.21.15.7:0/2304239385 learned_addr learned my addr 172.21.15.7:0/2304239385 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:25:10.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f412e59c700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413006d870 0x7f413014e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:10.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f412dd9b700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41300c1da0 0x7f4130155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:10.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f412ed9d700 1 -- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41300c1da0 msgr2=0x7f4130155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:10.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f412ed9d700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41300c1da0 0x7f4130155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:10.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f412ed9d700 1 -- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413006d870 msgr2=0x7f413014e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:10.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f412ed9d700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413006d870 0x7f413014e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:10.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f412ed9d700 1 -- 172.21.15.7:0/2304239385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41180094a0 con 0x7f41300c3e30 2023-03-03T14:25:10.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f412e59c700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413006d870 0x7f413014e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:25:10.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f412ed9d700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41300c3e30 0x7f4130156190 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f412800cb90 tx=0x7f412800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:10.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f4126ffd700 1 -- 172.21.15.7:0/2304239385 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412800ab30 con 0x7f41300c3e30 2023-03-03T14:25:10.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f4126ffd700 1 -- 172.21.15.7:0/2304239385 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412800ac90 con 0x7f41300c3e30 2023-03-03T14:25:10.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f4126ffd700 1 -- 172.21.15.7:0/2304239385 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41280070d0 con 0x7f41300c3e30 2023-03-03T14:25:10.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f4134b1f700 1 -- 172.21.15.7:0/2304239385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413006dc40 con 0x7f41300c3e30 2023-03-03T14:25:10.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.617+0000 7f4134b1f700 1 -- 172.21.15.7:0/2304239385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4130154ba0 con 0x7f41300c3e30 2023-03-03T14:25:10.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.618+0000 7f4134b1f700 1 -- 172.21.15.7:0/2304239385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41300c38e0 con 0x7f41300c3e30 2023-03-03T14:25:10.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.619+0000 7f4126ffd700 1 -- 172.21.15.7:0/2304239385 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4128010070 con 0x7f41300c3e30 2023-03-03T14:25:10.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.619+0000 7f4126ffd700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41100682f0 0x7f411006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:10.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.619+0000 7f4126ffd700 1 -- 172.21.15.7:0/2304239385 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4128085b30 con 0x7f41300c3e30 2023-03-03T14:25:10.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.619+0000 7f412e59c700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41100682f0 0x7f411006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:10.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.620+0000 7f412e59c700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41100682f0 0x7f411006a7a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f41180090b0 tx=0x7f4118008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:10.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.623+0000 7f4126ffd700 1 -- 172.21.15.7:0/2304239385 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4128051e70 con 0x7f41300c3e30 2023-03-03T14:25:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.862+0000 7f4134b1f700 1 -- 172.21.15.7:0/2304239385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4130074bb0 con 0x7f41300c3e30 2023-03-03T14:25:10.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.862+0000 7f4126ffd700 1 -- 172.21.15.7:0/2304239385 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4128018090 con 0x7f41300c3e30 2023-03-03T14:25:10.865 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:25:10.865 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:25:10.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 -- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41100682f0 msgr2=0x7f411006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:10.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41100682f0 0x7f411006a7a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f41180090b0 tx=0x7f4118008030 comp rx=0 tx=0).stop 2023-03-03T14:25:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 -- 172.21.15.7:0/2304239385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41300c3e30 msgr2=0x7f4130156190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41300c3e30 0x7f4130156190 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f412800cb90 tx=0x7f412800cf50 comp rx=0 tx=0).stop 2023-03-03T14:25:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 -- 172.21.15.7:0/2304239385 shutdown_connections 2023-03-03T14:25:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41100682f0 0x7f411006a7a0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41300c3e30 0x7f4130156190 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f413006d870 0x7f413014e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 --2- 172.21.15.7:0/2304239385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41300c1da0 0x7f4130155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 -- 172.21.15.7:0/2304239385 >> 172.21.15.7:0/2304239385 conn(0x7f413006c110 msgr2=0x7f4130070100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:25:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 -- 172.21.15.7:0/2304239385 shutdown_connections 2023-03-03T14:25:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:10.864+0000 7f4134b1f700 1 -- 172.21.15.7:0/2304239385 wait complete. 2023-03-03T14:25:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:25:10.878 INFO:tasks.mds_thrash.fs.[b]:waiting for 22 secs before thrashing 2023-03-03T14:25:23.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:23.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:25.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:25:25.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.184+0000 7f52a0779700 1 Processor -- start 2023-03-03T14:25:25.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.184+0000 7f52a0779700 1 -- start start 2023-03-03T14:25:25.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.185+0000 7f52a0779700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5298071e80 0x7f5298151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:25.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.185+0000 7f52a0779700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f529814e900 0x7f529815ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.185+0000 7f52a0779700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52981510d0 con 0x7f5298072250 2023-03-03T14:25:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.185+0000 7f52a0779700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52980c3740 con 0x7f529814e900 2023-03-03T14:25:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.185+0000 7f52a0779700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5298070670 con 0x7f5298071e80 2023-03-03T14:25:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529ed16700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f529814e900 0x7f529815ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529ed16700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f529814e900 0x7f529815ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48692/0 (socket says 172.21.15.7:48692) 2023-03-03T14:25:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529ed16700 1 -- 172.21.15.7:0/2547728059 learned_addr learned my addr 172.21.15.7:0/2547728059 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:25:25.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529e515700 1 --2- 172.21.15.7:0/2547728059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5298071e80 0x7f5298151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:25.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529ed16700 1 -- 172.21.15.7:0/2547728059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5298071e80 msgr2=0x7f5298151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:25:25.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529ed16700 1 --2- 172.21.15.7:0/2547728059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5298071e80 0x7f5298151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:25.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529ed16700 1 -- 172.21.15.7:0/2547728059 >> v1:172.21.15.143:6789/0 conn(0x7f5298072250 legacy=0x7f529814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:25.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529ed16700 1 -- 172.21.15.7:0/2547728059 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52980c39e0 con 0x7f529814e900 2023-03-03T14:25:25.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529e515700 1 --2- 172.21.15.7:0/2547728059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5298071e80 0x7f5298151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:25:25.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529ed16700 1 --2- 172.21.15.7:0/2547728059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f529814e900 0x7f529815ba50 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f529000d150 tx=0x7f529000d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e783e49b6a1f6710 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:25.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529d513700 1 -- 172.21.15.7:0/2547728059 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290010070 con 0x7f529814e900 2023-03-03T14:25:25.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529d513700 1 -- 172.21.15.7:0/2547728059 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529000ddc0 con 0x7f529814e900 2023-03-03T14:25:25.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.186+0000 7f529d513700 1 -- 172.21.15.7:0/2547728059 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529000b310 con 0x7f529814e900 2023-03-03T14:25:25.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.187+0000 7f52a0779700 1 -- 172.21.15.7:0/2547728059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f529814e900 msgr2=0x7f529815ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:25.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.187+0000 7f52a0779700 1 --2- 172.21.15.7:0/2547728059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f529814e900 0x7f529815ba50 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f529000d150 tx=0x7f529000d460 comp rx=0 tx=0).stop 2023-03-03T14:25:25.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.187+0000 7f52a0779700 1 -- 172.21.15.7:0/2547728059 shutdown_connections 2023-03-03T14:25:25.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.187+0000 7f52a0779700 1 --2- 172.21.15.7:0/2547728059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f529814e900 0x7f529815ba50 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:25.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.187+0000 7f52a0779700 1 --2- 172.21.15.7:0/2547728059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5298071e80 0x7f5298151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:25.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.187+0000 7f52a0779700 1 -- 172.21.15.7:0/2547728059 >> 172.21.15.7:0/2547728059 conn(0x7f52980a94d0 msgr2=0x7f52980a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:25:25.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.187+0000 7f52a0779700 1 -- 172.21.15.7:0/2547728059 shutdown_connections 2023-03-03T14:25:25.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.187+0000 7f52a0779700 1 -- 172.21.15.7:0/2547728059 wait complete. 2023-03-03T14:25:25.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.188+0000 7f52a0779700 1 Processor -- start 2023-03-03T14:25:25.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.188+0000 7f52a0779700 1 -- start start 2023-03-03T14:25:25.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f52a0779700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5298071e80 0x7f52980acc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:25.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f52a0779700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5298072250 0x7f52980aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:25.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f52a0779700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f529814e900 0x7f529814c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:25.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f52a0779700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52980abb60 con 0x7f5298072250 2023-03-03T14:25:25.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f52a0779700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52981ea4f0 con 0x7f5298071e80 2023-03-03T14:25:25.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f52a0779700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52981ea7d0 con 0x7f529814e900 2023-03-03T14:25:25.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529dd14700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5298072250 0x7f52980aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:25.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529dd14700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5298072250 0x7f52980aaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48696/0 (socket says 172.21.15.7:48696) 2023-03-03T14:25:25.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529dd14700 1 -- 172.21.15.7:0/3380918978 learned_addr learned my addr 172.21.15.7:0/3380918978 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:25:25.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529ed16700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f529814e900 0x7f529814c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:25.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529e515700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5298071e80 0x7f52980acc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:25.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529dd14700 1 -- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f529814e900 msgr2=0x7f529814c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:25.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529dd14700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f529814e900 0x7f529814c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:25.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529dd14700 1 -- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5298071e80 msgr2=0x7f52980acc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:25.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529dd14700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5298071e80 0x7f52980acc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:25.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529dd14700 1 -- 172.21.15.7:0/3380918978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529000bd40 con 0x7f5298072250 2023-03-03T14:25:25.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.189+0000 7f529dd14700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5298072250 0x7f52980aaee0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f529400c4b0 tx=0x7f529400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:25.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.190+0000 7f528affd700 1 -- 172.21.15.7:0/3380918978 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52940052d0 con 0x7f5298072250 2023-03-03T14:25:25.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.190+0000 7f528affd700 1 -- 172.21.15.7:0/3380918978 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5294005430 con 0x7f5298072250 2023-03-03T14:25:25.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.190+0000 7f52a0779700 1 -- 172.21.15.7:0/3380918978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529806c9a0 con 0x7f5298072250 2023-03-03T14:25:25.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.190+0000 7f528affd700 1 -- 172.21.15.7:0/3380918978 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52940038c0 con 0x7f5298072250 2023-03-03T14:25:25.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.190+0000 7f52a0779700 1 -- 172.21.15.7:0/3380918978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5298151300 con 0x7f5298072250 2023-03-03T14:25:25.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.191+0000 7f52a0779700 1 -- 172.21.15.7:0/3380918978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529806cb30 con 0x7f5298072250 2023-03-03T14:25:25.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.191+0000 7f528affd700 1 -- 172.21.15.7:0/3380918978 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5294019030 con 0x7f5298072250 2023-03-03T14:25:25.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.191+0000 7f528affd700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f527c0682f0 0x7f527c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:25.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.192+0000 7f528affd700 1 -- 172.21.15.7:0/3380918978 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f52940854f0 con 0x7f5298072250 2023-03-03T14:25:25.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.192+0000 7f529e515700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f527c0682f0 0x7f527c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:25.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.192+0000 7f529e515700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f527c0682f0 0x7f527c06a7a0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f528c009930 tx=0x7f528c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:25.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.196+0000 7f528affd700 1 -- 172.21.15.7:0/3380918978 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5294051890 con 0x7f5298072250 2023-03-03T14:25:25.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.432+0000 7f52a0779700 1 -- 172.21.15.7:0/3380918978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f529814d410 con 0x7f5298072250 2023-03-03T14:25:25.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.433+0000 7f528affd700 1 -- 172.21.15.7:0/3380918978 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f529814d410 con 0x7f5298072250 2023-03-03T14:25:25.434 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:25:25.434 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:25:25.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 -- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f527c0682f0 msgr2=0x7f527c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:25.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f527c0682f0 0x7f527c06a7a0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f528c009930 tx=0x7f528c008040 comp rx=0 tx=0).stop 2023-03-03T14:25:25.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 -- 172.21.15.7:0/3380918978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5298072250 msgr2=0x7f52980aaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:25.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5298072250 0x7f52980aaee0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f529400c4b0 tx=0x7f529400c870 comp rx=0 tx=0).stop 2023-03-03T14:25:25.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 -- 172.21.15.7:0/3380918978 shutdown_connections 2023-03-03T14:25:25.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f529814e900 0x7f529814c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:25.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5298071e80 0x7f52980acc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:25.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f527c0682f0 0x7f527c06a7a0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:25.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 --2- 172.21.15.7:0/3380918978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5298072250 0x7f52980aaee0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:25.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 -- 172.21.15.7:0/3380918978 >> 172.21.15.7:0/3380918978 conn(0x7f52980a94d0 msgr2=0x7f52980736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:25:25.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 -- 172.21.15.7:0/3380918978 shutdown_connections 2023-03-03T14:25:25.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:25.435+0000 7f52a0779700 1 -- 172.21.15.7:0/3380918978 wait complete. 2023-03-03T14:25:25.438 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:25:25.448 INFO:tasks.mds_thrash.fs.[a]:waiting for 103 secs before thrashing 2023-03-03T14:25:32.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:25:33.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.016+0000 7f761cd92700 1 Processor -- start 2023-03-03T14:25:33.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.017+0000 7f761cd92700 1 -- start start 2023-03-03T14:25:33.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.017+0000 7f761cd92700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f761814f9d0 0x7f7618156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.017+0000 7f761cd92700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f761814fda0 0x7f7618071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.017+0000 7f761cd92700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76180c2f30 con 0x7f76180abe20 2023-03-03T14:25:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f761cd92700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76180c31d0 con 0x7f761814fda0 2023-03-03T14:25:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f761cd92700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76181524a0 con 0x7f761814f9d0 2023-03-03T14:25:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f7615d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f761814fda0 0x7f7618071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:33.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f7616d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f76180abe20 0x7f76181529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42694/0 (socket says 172.21.15.7:42694) 2023-03-03T14:25:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f7616d9d700 1 -- 172.21.15.7:0/2521436404 learned_addr learned my addr 172.21.15.7:0/2521436404 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:25:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f761659c700 1 --2- 172.21.15.7:0/2521436404 >> v2:172.21.15.143:3300/0 conn(0x7f761814f9d0 0x7f7618156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f7615d9b700 1 -- 172.21.15.7:0/2521436404 >> v2:172.21.15.143:3300/0 conn(0x7f761814f9d0 msgr2=0x7f7618156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:25:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f7615d9b700 1 --2- 172.21.15.7:0/2521436404 >> v2:172.21.15.143:3300/0 conn(0x7f761814f9d0 0x7f7618156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f7615d9b700 1 -- 172.21.15.7:0/2521436404 >> v1:172.21.15.7:6789/0 conn(0x7f76180abe20 legacy=0x7f76181529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f7615d9b700 1 -- 172.21.15.7:0/2521436404 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76180c34b0 con 0x7f761814fda0 2023-03-03T14:25:33.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f761559a700 1 -- 172.21.15.7:0/2521436404 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978174114 0 0) 0x7f76180c2f30 con 0x7f76180abe20 2023-03-03T14:25:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f7615d9b700 1 --2- 172.21.15.7:0/2521436404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f761814fda0 0x7f7618071890 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f76180710a0 tx=0x7f7608009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50489b6fdc889753 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f761559a700 1 -- 172.21.15.7:0/2521436404 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608011830 con 0x7f761814fda0 2023-03-03T14:25:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f761559a700 1 -- 172.21.15.7:0/2521436404 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7608011990 con 0x7f761814fda0 2023-03-03T14:25:33.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.018+0000 7f761559a700 1 -- 172.21.15.7:0/2521436404 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608011d20 con 0x7f761814fda0 2023-03-03T14:25:33.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.019+0000 7f761cd92700 1 -- 172.21.15.7:0/2521436404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f761814fda0 msgr2=0x7f7618071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:33.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.019+0000 7f761cd92700 1 --2- 172.21.15.7:0/2521436404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f761814fda0 0x7f7618071890 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f76180710a0 tx=0x7f7608009b20 comp rx=0 tx=0).stop 2023-03-03T14:25:33.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.019+0000 7f761cd92700 1 -- 172.21.15.7:0/2521436404 shutdown_connections 2023-03-03T14:25:33.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.019+0000 7f761cd92700 1 --2- 172.21.15.7:0/2521436404 >> v2:172.21.15.143:3300/0 conn(0x7f761814f9d0 0x7f7618156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:33.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.019+0000 7f761cd92700 1 --2- 172.21.15.7:0/2521436404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f761814fda0 0x7f7618071890 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:33.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.019+0000 7f761cd92700 1 -- 172.21.15.7:0/2521436404 >> 172.21.15.7:0/2521436404 conn(0x7f76180c8900 msgr2=0x7f76180c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:25:33.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.019+0000 7f761cd92700 1 -- 172.21.15.7:0/2521436404 shutdown_connections 2023-03-03T14:25:33.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.019+0000 7f761cd92700 1 -- 172.21.15.7:0/2521436404 wait complete. 2023-03-03T14:25:33.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.020+0000 7f761cd92700 1 Processor -- start 2023-03-03T14:25:33.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.020+0000 7f761cd92700 1 -- start start 2023-03-03T14:25:33.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.021+0000 7f761cd92700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76180abe20 0x7f761806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:33.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.021+0000 7f761cd92700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f761814f9d0 0x7f76181e9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:33.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.021+0000 7f761cd92700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f761814fda0 0x7f76180c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:33.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.021+0000 7f761cd92700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761806d640 con 0x7f76180abe20 2023-03-03T14:25:33.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.021+0000 7f761cd92700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76181e99a0 con 0x7f761814fda0 2023-03-03T14:25:33.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.021+0000 7f761cd92700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76181e9c40 con 0x7f761814f9d0 2023-03-03T14:25:33.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.021+0000 7f761659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76180abe20 0x7f761806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:33.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.021+0000 7f761659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76180abe20 0x7f761806a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48716/0 (socket says 172.21.15.7:48716) 2023-03-03T14:25:33.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.021+0000 7f761659c700 1 -- 172.21.15.7:0/272832264 learned_addr learned my addr 172.21.15.7:0/272832264 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:25:33.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f7616d9d700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f761814fda0 0x7f76180c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:33.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f7615d9b700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f761814f9d0 0x7f76181e9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:33.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f761659c700 1 -- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f761814f9d0 msgr2=0x7f76181e9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:33.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f761659c700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f761814f9d0 0x7f76181e9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:33.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f761659c700 1 -- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f761814fda0 msgr2=0x7f76180c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:33.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f761659c700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f761814fda0 0x7f76180c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:33.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f761659c700 1 -- 172.21.15.7:0/272832264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76080094a0 con 0x7f76180abe20 2023-03-03T14:25:33.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f761659c700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76180abe20 0x7f761806a890 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f760000e810 tx=0x7f760000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:33.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f760effd700 1 -- 172.21.15.7:0/272832264 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600013610 con 0x7f76180abe20 2023-03-03T14:25:33.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f761cd92700 1 -- 172.21.15.7:0/272832264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76180ac350 con 0x7f76180abe20 2023-03-03T14:25:33.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f760effd700 1 -- 172.21.15.7:0/272832264 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7600013770 con 0x7f76180abe20 2023-03-03T14:25:33.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f760effd700 1 -- 172.21.15.7:0/272832264 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600013b20 con 0x7f76180abe20 2023-03-03T14:25:33.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.022+0000 7f761cd92700 1 -- 172.21.15.7:0/272832264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76180ab630 con 0x7f76180abe20 2023-03-03T14:25:33.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.023+0000 7f761cd92700 1 -- 172.21.15.7:0/272832264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76180c3e10 con 0x7f76180abe20 2023-03-03T14:25:33.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.024+0000 7f760effd700 1 -- 172.21.15.7:0/272832264 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7600023070 con 0x7f76180abe20 2023-03-03T14:25:33.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.024+0000 7f760effd700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75f80682f0 0x7f75f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:25:33.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.024+0000 7f760effd700 1 -- 172.21.15.7:0/272832264 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7600085ef0 con 0x7f76180abe20 2023-03-03T14:25:33.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.024+0000 7f7615d9b700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75f80682f0 0x7f75f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:25:33.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.025+0000 7f7615d9b700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75f80682f0 0x7f75f806a7a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f76181501d0 tx=0x7f7608008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:25:33.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.028+0000 7f760effd700 1 -- 172.21.15.7:0/272832264 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7600052210 con 0x7f76180abe20 2023-03-03T14:25:33.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.264+0000 7f761cd92700 1 -- 172.21.15.7:0/272832264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7618074bb0 con 0x7f76180abe20 2023-03-03T14:25:33.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.265+0000 7f760effd700 1 -- 172.21.15.7:0/272832264 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7600055830 con 0x7f76180abe20 2023-03-03T14:25:33.267 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:25:33.267 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:25:33.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 -- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75f80682f0 msgr2=0x7f75f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:33.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75f80682f0 0x7f75f806a7a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f76181501d0 tx=0x7f7608008030 comp rx=0 tx=0).stop 2023-03-03T14:25:33.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 -- 172.21.15.7:0/272832264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76180abe20 msgr2=0x7f761806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:25:33.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76180abe20 0x7f761806a890 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f760000e810 tx=0x7f760000ebd0 comp rx=0 tx=0).stop 2023-03-03T14:25:33.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 -- 172.21.15.7:0/272832264 shutdown_connections 2023-03-03T14:25:33.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f761814fda0 0x7f76180c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:33.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75f80682f0 0x7f75f806a7a0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:33.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76180abe20 0x7f761806a890 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:33.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 --2- 172.21.15.7:0/272832264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f761814f9d0 0x7f76181e9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:25:33.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 -- 172.21.15.7:0/272832264 >> 172.21.15.7:0/272832264 conn(0x7f76180c8900 msgr2=0x7f76180c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:25:33.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 -- 172.21.15.7:0/272832264 shutdown_connections 2023-03-03T14:25:33.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:25:33.267+0000 7f761cd92700 1 -- 172.21.15.7:0/272832264 wait complete. 2023-03-03T14:25:33.270 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:25:33.280 INFO:tasks.mds_thrash.fs.[b]:waiting for 113 secs before thrashing 2023-03-03T14:25:53.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:25:53.197 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:23.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:23.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:53.332 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:26:53.336 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:08.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:27:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.587+0000 7f8302ff1700 1 Processor -- start 2023-03-03T14:27:08.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.587+0000 7f8302ff1700 1 -- start start 2023-03-03T14:27:08.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.587+0000 7f8302ff1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f82fc0c2950 0x7f82fc157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:08.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.587+0000 7f8302ff1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82fc06c480 0x7f82fc158020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.587+0000 7f8302ff1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82fc06cb70 con 0x7f82fc151b20 2023-03-03T14:27:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.587+0000 7f8302ff1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82fc06c2f0 con 0x7f82fc0c2950 2023-03-03T14:27:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.587+0000 7f8302ff1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82fc06ce50 con 0x7f82fc06c480 2023-03-03T14:27:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f82fbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f82fc0c2950 0x7f82fc157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f8300d8d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f82fc151b20 0x7f82fc0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42772/0 (socket says 172.21.15.7:42772) 2023-03-03T14:27:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f8300d8d700 1 -- 172.21.15.7:0/2495804869 learned_addr learned my addr 172.21.15.7:0/2495804869 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:27:08.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f830158e700 1 --2- 172.21.15.7:0/2495804869 >> v2:172.21.15.143:3300/0 conn(0x7f82fc06c480 0x7f82fc158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f82fbfff700 1 -- 172.21.15.7:0/2495804869 >> v2:172.21.15.143:3300/0 conn(0x7f82fc06c480 msgr2=0x7f82fc158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:27:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f82fbfff700 1 --2- 172.21.15.7:0/2495804869 >> v2:172.21.15.143:3300/0 conn(0x7f82fc06c480 0x7f82fc158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f82fbfff700 1 -- 172.21.15.7:0/2495804869 >> v1:172.21.15.7:6789/0 conn(0x7f82fc151b20 legacy=0x7f82fc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f82fbfff700 1 -- 172.21.15.7:0/2495804869 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc06d130 con 0x7f82fc0c2950 2023-03-03T14:27:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f82fbfff700 1 --2- 172.21.15.7:0/2495804869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 0x7f82fc157ae0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f82fc06e380 tx=0x7f82e8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c4ae17d1f258e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:27:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f82fb7fe700 1 -- 172.21.15.7:0/2495804869 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8011830 con 0x7f82fc0c2950 2023-03-03T14:27:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f82fb7fe700 1 -- 172.21.15.7:0/2495804869 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e8011990 con 0x7f82fc0c2950 2023-03-03T14:27:08.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.588+0000 7f82fb7fe700 1 -- 172.21.15.7:0/2495804869 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8011d20 con 0x7f82fc0c2950 2023-03-03T14:27:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.589+0000 7f8302ff1700 1 -- 172.21.15.7:0/2495804869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 msgr2=0x7f82fc157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.589+0000 7f8302ff1700 1 --2- 172.21.15.7:0/2495804869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 0x7f82fc157ae0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f82fc06e380 tx=0x7f82e8009b20 comp rx=0 tx=0).stop 2023-03-03T14:27:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.589+0000 7f8302ff1700 1 -- 172.21.15.7:0/2495804869 shutdown_connections 2023-03-03T14:27:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.589+0000 7f8302ff1700 1 --2- 172.21.15.7:0/2495804869 >> v2:172.21.15.143:3300/0 conn(0x7f82fc06c480 0x7f82fc158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.589+0000 7f8302ff1700 1 --2- 172.21.15.7:0/2495804869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 0x7f82fc157ae0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.589+0000 7f8302ff1700 1 -- 172.21.15.7:0/2495804869 >> 172.21.15.7:0/2495804869 conn(0x7f82fc0a94d0 msgr2=0x7f82fc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:27:08.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.589+0000 7f8302ff1700 1 -- 172.21.15.7:0/2495804869 shutdown_connections 2023-03-03T14:27:08.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.589+0000 7f8302ff1700 1 -- 172.21.15.7:0/2495804869 wait complete. 2023-03-03T14:27:08.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.590+0000 7f8302ff1700 1 Processor -- start 2023-03-03T14:27:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.590+0000 7f8302ff1700 1 -- start start 2023-03-03T14:27:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f8302ff1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82fc06c480 0x7f82fc0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f8302ff1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 0x7f82fc148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f8302ff1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82fc151b20 0x7f82fc148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f8302ff1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc0be8b0 con 0x7f82fc0c2950 2023-03-03T14:27:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f8302ff1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc14ea90 con 0x7f82fc06c480 2023-03-03T14:27:08.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f8302ff1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc1490e0 con 0x7f82fc151b20 2023-03-03T14:27:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f82fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 0x7f82fc148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f82fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 0x7f82fc148660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48798/0 (socket says 172.21.15.7:48798) 2023-03-03T14:27:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f82fbfff700 1 -- 172.21.15.7:0/3528413915 learned_addr learned my addr 172.21.15.7:0/3528413915 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:27:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f830158e700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82fc151b20 0x7f82fc148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f8300d8d700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82fc06c480 0x7f82fc0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f82fbfff700 1 -- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82fc151b20 msgr2=0x7f82fc148ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:08.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f82fbfff700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82fc151b20 0x7f82fc148ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f82fbfff700 1 -- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82fc06c480 msgr2=0x7f82fc0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f82fbfff700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82fc06c480 0x7f82fc0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f82fbfff700 1 -- 172.21.15.7:0/3528413915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e80094a0 con 0x7f82fc0c2950 2023-03-03T14:27:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f830158e700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82fc151b20 0x7f82fc148ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:27:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.591+0000 7f8300d8d700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82fc06c480 0x7f82fc0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:27:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.592+0000 7f82fbfff700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 0x7f82fc148660 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009b60 tx=0x7f82e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:27:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.592+0000 7f82f97fa700 1 -- 172.21.15.7:0/3528413915 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e80036a0 con 0x7f82fc0c2950 2023-03-03T14:27:08.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.592+0000 7f82f97fa700 1 -- 172.21.15.7:0/3528413915 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e801e070 con 0x7f82fc0c2950 2023-03-03T14:27:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.592+0000 7f8302ff1700 1 -- 172.21.15.7:0/3528413915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82fc151ef0 con 0x7f82fc0c2950 2023-03-03T14:27:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.592+0000 7f82f97fa700 1 -- 172.21.15.7:0/3528413915 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8003a60 con 0x7f82fc0c2950 2023-03-03T14:27:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.592+0000 7f8302ff1700 1 -- 172.21.15.7:0/3528413915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82fc06d410 con 0x7f82fc0c2950 2023-03-03T14:27:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.593+0000 7f8302ff1700 1 -- 172.21.15.7:0/3528413915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82fc073e00 con 0x7f82fc0c2950 2023-03-03T14:27:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.593+0000 7f82f97fa700 1 -- 172.21.15.7:0/3528413915 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f82e800f050 con 0x7f82fc0c2950 2023-03-03T14:27:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.594+0000 7f82f97fa700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82e00682f0 0x7f82e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.594+0000 7f82f97fa700 1 -- 172.21.15.7:0/3528413915 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f82e8087200 con 0x7f82fc0c2950 2023-03-03T14:27:08.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.594+0000 7f8300d8d700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82e00682f0 0x7f82e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.594+0000 7f8300d8d700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82e00682f0 0x7f82e006a7a0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f82f0007fd0 tx=0x7f82f0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:27:08.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.598+0000 7f82f97fa700 1 -- 172.21.15.7:0/3528413915 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f82e80535a0 con 0x7f82fc0c2950 2023-03-03T14:27:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.839+0000 7f8302ff1700 1 -- 172.21.15.7:0/3528413915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82fc0be9f0 con 0x7f82fc0c2950 2023-03-03T14:27:08.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.840+0000 7f82f97fa700 1 -- 172.21.15.7:0/3528413915 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f82fc0be9f0 con 0x7f82fc0c2950 2023-03-03T14:27:08.842 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:27:08.842 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:27:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 -- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82e00682f0 msgr2=0x7f82e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82e00682f0 0x7f82e006a7a0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f82f0007fd0 tx=0x7f82f0009040 comp rx=0 tx=0).stop 2023-03-03T14:27:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 -- 172.21.15.7:0/3528413915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 msgr2=0x7f82fc148660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:08.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 0x7f82fc148660 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009b60 tx=0x7f82e8004750 comp rx=0 tx=0).stop 2023-03-03T14:27:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 -- 172.21.15.7:0/3528413915 shutdown_connections 2023-03-03T14:27:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82fc151b20 0x7f82fc148ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82fc06c480 0x7f82fc0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f82e00682f0 0x7f82e006a7a0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 --2- 172.21.15.7:0/3528413915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82fc0c2950 0x7f82fc148660 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:08.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 -- 172.21.15.7:0/3528413915 >> 172.21.15.7:0/3528413915 conn(0x7f82fc0a94d0 msgr2=0x7f82fc0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:27:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 -- 172.21.15.7:0/3528413915 shutdown_connections 2023-03-03T14:27:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:08.842+0000 7f8302ff1700 1 -- 172.21.15.7:0/3528413915 wait complete. 2023-03-03T14:27:08.845 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:27:08.855 INFO:tasks.mds_thrash.fs.[a]:waiting for 82 secs before thrashing 2023-03-03T14:27:23.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:23.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:26.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:27:26.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.418+0000 7f00b6ef4700 1 Processor -- start 2023-03-03T14:27:26.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.419+0000 7f00b6ef4700 1 -- start start 2023-03-03T14:27:26.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.419+0000 7f00b6ef4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f00b00caa30 0x7f00b00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:26.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.419+0000 7f00b6ef4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00c00e0 0x7f00b0154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:26.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.419+0000 7f00b6ef4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b00cb280 con 0x7f00b00cae00 2023-03-03T14:27:26.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.419+0000 7f00b6ef4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00b00c7310 con 0x7f00b00caa30 2023-03-03T14:27:26.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.419+0000 7f00b6ef4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00bfd10 con 0x7f00b00c00e0 2023-03-03T14:27:26.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00b4c90700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f00b00caa30 0x7f00b00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:26.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00affff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00c00e0 0x7f00b0154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:26.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00b5491700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f00b00cae00 0x7f00b00c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:46320/0 (socket says 172.21.15.7:46320) 2023-03-03T14:27:26.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00affff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00c00e0 0x7f00b0154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:56890/0 (socket says 172.21.15.7:56890) 2023-03-03T14:27:26.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00b4c90700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f00b00caa30 0x7f00b00c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:44590/0 (socket says 172.21.15.7:44590) 2023-03-03T14:27:26.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00b5491700 1 -- 172.21.15.7:0/2352040977 learned_addr learned my addr 172.21.15.7:0/2352040977 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:27:26.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00affff700 1 -- 172.21.15.7:0/2352040977 >> v2:172.21.15.143:3300/0 conn(0x7f00b00caa30 msgr2=0x7f00b00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:27:26.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00affff700 1 --2- 172.21.15.7:0/2352040977 >> v2:172.21.15.143:3300/0 conn(0x7f00b00caa30 0x7f00b00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:26.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00affff700 1 -- 172.21.15.7:0/2352040977 >> v1:172.21.15.143:6789/0 conn(0x7f00b00cae00 legacy=0x7f00b00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:26.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00affff700 1 -- 172.21.15.7:0/2352040977 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b00bfa30 con 0x7f00b00c00e0 2023-03-03T14:27:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00b4c90700 1 --2- 172.21.15.7:0/2352040977 >> v2:172.21.15.143:3300/0 conn(0x7f00b00caa30 0x7f00b00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:27:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.420+0000 7f00affff700 1 --2- 172.21.15.7:0/2352040977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00c00e0 0x7f00b0154ff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f009c009820 tx=0x7f009c009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5871ec34dccec7a4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:27:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00af7fe700 1 -- 172.21.15.7:0/2352040977 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c005e30 con 0x7f00b00c00e0 2023-03-03T14:27:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00af7fe700 1 -- 172.21.15.7:0/2352040977 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c006030 con 0x7f00b00c00e0 2023-03-03T14:27:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00af7fe700 1 -- 172.21.15.7:0/2352040977 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c0063c0 con 0x7f00b00c00e0 2023-03-03T14:27:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00b6ef4700 1 -- 172.21.15.7:0/2352040977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00c00e0 msgr2=0x7f00b0154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00b6ef4700 1 --2- 172.21.15.7:0/2352040977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00c00e0 0x7f00b0154ff0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f009c009820 tx=0x7f009c009b30 comp rx=0 tx=0).stop 2023-03-03T14:27:26.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00b6ef4700 1 -- 172.21.15.7:0/2352040977 shutdown_connections 2023-03-03T14:27:26.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00b6ef4700 1 --2- 172.21.15.7:0/2352040977 >> v2:172.21.15.143:3300/0 conn(0x7f00b00caa30 0x7f00b00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:26.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00b6ef4700 1 --2- 172.21.15.7:0/2352040977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00c00e0 0x7f00b0154ff0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:26.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00b6ef4700 1 -- 172.21.15.7:0/2352040977 >> 172.21.15.7:0/2352040977 conn(0x7f00b0066e40 msgr2=0x7f00b0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:27:26.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00b6ef4700 1 -- 172.21.15.7:0/2352040977 shutdown_connections 2023-03-03T14:27:26.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.421+0000 7f00b6ef4700 1 -- 172.21.15.7:0/2352040977 wait complete. 2023-03-03T14:27:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.422+0000 7f00b6ef4700 1 Processor -- start 2023-03-03T14:27:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.422+0000 7f00b6ef4700 1 -- start start 2023-03-03T14:27:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.423+0000 7f00b6ef4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00b00c00e0 0x7f00b00c2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.423+0000 7f00b6ef4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00caa30 0x7f00b0157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.423+0000 7f00b6ef4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00b00cae00 0x7f00b014cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.423+0000 7f00b6ef4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b014b510 con 0x7f00b00cae00 2023-03-03T14:27:26.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.423+0000 7f00b6ef4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b00c2300 con 0x7f00b00c00e0 2023-03-03T14:27:26.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.423+0000 7f00b6ef4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b0149cf0 con 0x7f00b00caa30 2023-03-03T14:27:26.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.423+0000 7f00b5491700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00b00cae00 0x7f00b014cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:26.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.423+0000 7f00b5491700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00b00cae00 0x7f00b014cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48826/0 (socket says 172.21.15.7:48826) 2023-03-03T14:27:26.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.423+0000 7f00b5491700 1 -- 172.21.15.7:0/1997905396 learned_addr learned my addr 172.21.15.7:0/1997905396 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:27:26.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00b4c90700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00b00c00e0 0x7f00b00c2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:26.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00affff700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00caa30 0x7f00b0157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:26.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00b5491700 1 -- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00caa30 msgr2=0x7f00b0157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:26.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00b5491700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00caa30 0x7f00b0157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:26.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00b5491700 1 -- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00b00c00e0 msgr2=0x7f00b00c2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:26.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00b5491700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00b00c00e0 0x7f00b00c2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:26.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00b5491700 1 -- 172.21.15.7:0/1997905396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40094a0 con 0x7f00b00cae00 2023-03-03T14:27:26.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00affff700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00caa30 0x7f00b0157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:27:26.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00b5491700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00b00cae00 0x7f00b014cbe0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f00a800ca80 tx=0x7f00a800cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:27:26.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00ad7fa700 1 -- 172.21.15.7:0/1997905396 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a800a9e0 con 0x7f00b00cae00 2023-03-03T14:27:26.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00ad7fa700 1 -- 172.21.15.7:0/1997905396 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a800ab40 con 0x7f00b00cae00 2023-03-03T14:27:26.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00b6ef4700 1 -- 172.21.15.7:0/1997905396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009c009430 con 0x7f00b00cae00 2023-03-03T14:27:26.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00ad7fa700 1 -- 172.21.15.7:0/1997905396 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8012400 con 0x7f00b00cae00 2023-03-03T14:27:26.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.424+0000 7f00b6ef4700 1 -- 172.21.15.7:0/1997905396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b01547f0 con 0x7f00b00cae00 2023-03-03T14:27:26.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.425+0000 7f00b6ef4700 1 -- 172.21.15.7:0/1997905396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b0158880 con 0x7f00b00cae00 2023-03-03T14:27:26.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.426+0000 7f00ad7fa700 1 -- 172.21.15.7:0/1997905396 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f00a8019030 con 0x7f00b00cae00 2023-03-03T14:27:26.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.426+0000 7f00ad7fa700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00940682f0 0x7f009406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:27:26.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.426+0000 7f00ad7fa700 1 -- 172.21.15.7:0/1997905396 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f00a8085ac0 con 0x7f00b00cae00 2023-03-03T14:27:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.426+0000 7f00b4c90700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00940682f0 0x7f009406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:27:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.427+0000 7f00b4c90700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00940682f0 0x7f009406a7a0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009d70 tx=0x7f00a4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:27:26.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.431+0000 7f00ad7fa700 1 -- 172.21.15.7:0/1997905396 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f00a804b5e0 con 0x7f00b00cae00 2023-03-03T14:27:26.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.668+0000 7f00b6ef4700 1 -- 172.21.15.7:0/1997905396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00b0154ad0 con 0x7f00b00cae00 2023-03-03T14:27:26.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.669+0000 7f00ad7fa700 1 -- 172.21.15.7:0/1997905396 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f00a8051f10 con 0x7f00b00cae00 2023-03-03T14:27:26.671 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:27:26.672 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:27:26.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 -- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00940682f0 msgr2=0x7f009406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:26.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00940682f0 0x7f009406a7a0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009d70 tx=0x7f00a4009210 comp rx=0 tx=0).stop 2023-03-03T14:27:26.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 -- 172.21.15.7:0/1997905396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00b00cae00 msgr2=0x7f00b014cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:27:26.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00b00cae00 0x7f00b014cbe0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f00a800ca80 tx=0x7f00a800cd90 comp rx=0 tx=0).stop 2023-03-03T14:27:26.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 -- 172.21.15.7:0/1997905396 shutdown_connections 2023-03-03T14:27:26.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00940682f0 0x7f009406a7a0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:26.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00b00cae00 0x7f00b014cbe0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:26.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00b00c00e0 0x7f00b00c2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:26.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 --2- 172.21.15.7:0/1997905396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00b00caa30 0x7f00b0157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:27:26.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 -- 172.21.15.7:0/1997905396 >> 172.21.15.7:0/1997905396 conn(0x7f00b0066e40 msgr2=0x7f00b00c6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:27:26.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 -- 172.21.15.7:0/1997905396 shutdown_connections 2023-03-03T14:27:26.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:27:26.671+0000 7f00b6ef4700 1 -- 172.21.15.7:0/1997905396 wait complete. 2023-03-03T14:27:26.675 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:27:26.685 INFO:tasks.mds_thrash.fs.[b]:waiting for 45 secs before thrashing 2023-03-03T14:27:53.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:27:53.493 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:11.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:28:11.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.824+0000 7f6f9c40f700 1 Processor -- start 2023-03-03T14:28:11.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.824+0000 7f6f9c40f700 1 -- start start 2023-03-03T14:28:11.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.825+0000 7f6f9c40f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6f940aad20 0x7f6f94157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.825+0000 7f6f9c40f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f940a9c20 con 0x7f6f940aa030 2023-03-03T14:28:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.825+0000 7f6f9c40f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f94154f10 con 0x7f6f9414b0d0 2023-03-03T14:28:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.825+0000 7f6f9c40f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f940ac3c0 con 0x7f6f940aad20 2023-03-03T14:28:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.825+0000 7f6f9a9ac700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6f940aa030 0x7f6f94158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42836/0 (socket says 172.21.15.7:42836) 2023-03-03T14:28:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.825+0000 7f6f9a9ac700 1 -- 172.21.15.7:0/122498392 learned_addr learned my addr 172.21.15.7:0/122498392 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:28:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f999aa700 1 --2- 172.21.15.7:0/122498392 >> v2:172.21.15.143:3300/0 conn(0x7f6f940aad20 0x7f6f94157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511244725 0 0) 0x7f6f940a9c20 con 0x7f6f940aa030 2023-03-03T14:28:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f800035b0 con 0x7f6f940aa030 2023-03-03T14:28:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2615884655 0 0) 0x7f6f800035b0 con 0x7f6f940aa030 2023-03-03T14:28:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 >> v2:172.21.15.143:3300/0 conn(0x7f6f940aad20 msgr2=0x7f6f94157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:28:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 --2- 172.21.15.7:0/122498392 >> v2:172.21.15.143:3300/0 conn(0x7f6f940aad20 0x7f6f94157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 >> v1:172.21.15.143:6789/0 conn(0x7f6f9414b0d0 legacy=0x7f6f94071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9406fc70 con 0x7f6f940aa030 2023-03-03T14:28:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f6f8c0019f0 con 0x7f6f940aa030 2023-03-03T14:28:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 >> v1:172.21.15.7:6789/0 conn(0x7f6f940aa030 legacy=0x7f6f94158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f80003a70 0x7f6f80003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f80004850 0x7f6f80004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f80005250 0x7f6f8000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f940ac3c0 con 0x7f6f80003a70 2023-03-03T14:28:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8000c190 con 0x7f6f80004850 2023-03-03T14:28:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8000c350 con 0x7f6f80005250 2023-03-03T14:28:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f9a1ab700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f80003a70 0x7f6f80003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f999aa700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f80004850 0x7f6f80004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f9a9ac700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f80005250 0x7f6f8000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f9a1ab700 1 -- 172.21.15.7:0/122498392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f80005250 msgr2=0x7f6f8000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:28:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.826+0000 7f6f9a1ab700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f80005250 0x7f6f8000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f9a1ab700 1 -- 172.21.15.7:0/122498392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f80004850 msgr2=0x7f6f80004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:28:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f9a1ab700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f80004850 0x7f6f80004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f9a1ab700 1 -- 172.21.15.7:0/122498392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f800035b0 con 0x7f6f80003a70 2023-03-03T14:28:11.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f9a9ac700 1 -- 172.21.15.7:0/122498392 reap_dead start 2023-03-03T14:28:11.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f9a1ab700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f80003a70 0x7f6f80003e80 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f6f90007fd0 tx=0x7f6f80005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41fac5346595097 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:28:11.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f900052d0 con 0x7f6f80003a70 2023-03-03T14:28:11.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f90005470 con 0x7f6f80003a70 2023-03-03T14:28:11.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f991a9700 1 -- 172.21.15.7:0/122498392 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f900038c0 con 0x7f6f80003a70 2023-03-03T14:28:11.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f9c40f700 1 -- 172.21.15.7:0/122498392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f80003a70 msgr2=0x7f6f80003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:11.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f9c40f700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f80003a70 0x7f6f80003e80 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f6f90007fd0 tx=0x7f6f80005880 comp rx=0 tx=0).stop 2023-03-03T14:28:11.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f9c40f700 1 -- 172.21.15.7:0/122498392 shutdown_connections 2023-03-03T14:28:11.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f9c40f700 1 --2- 172.21.15.7:0/122498392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f80003a70 0x7f6f80003e80 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:11.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.827+0000 7f6f9c40f700 1 -- 172.21.15.7:0/122498392 >> 172.21.15.7:0/122498392 conn(0x7f6f940c8900 msgr2=0x7f6f940c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:28:11.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.828+0000 7f6f9c40f700 1 -- 172.21.15.7:0/122498392 shutdown_connections 2023-03-03T14:28:11.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.828+0000 7f6f9c40f700 1 -- 172.21.15.7:0/122498392 wait complete. 2023-03-03T14:28:11.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.829+0000 7f6f9c40f700 1 Processor -- start 2023-03-03T14:28:11.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.829+0000 7f6f9c40f700 1 -- start start 2023-03-03T14:28:11.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.829+0000 7f6f9c40f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f80003a70 0x7f6f941572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:11.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.829+0000 7f6f9c40f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f94155880 0x7f6f941ed6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:11.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.829+0000 7f6f9c40f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f94146440 0x7f6f94143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:11.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.829+0000 7f6f9c40f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f94145cd0 con 0x7f6f94146440 2023-03-03T14:28:11.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.829+0000 7f6f9c40f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f94145fb0 con 0x7f6f94155880 2023-03-03T14:28:11.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.829+0000 7f6f9c40f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f94144460 con 0x7f6f80003a70 2023-03-03T14:28:11.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.829+0000 7f6f9a9ac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f94146440 0x7f6f94143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:11.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9a9ac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f94146440 0x7f6f94143520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48868/0 (socket says 172.21.15.7:48868) 2023-03-03T14:28:11.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9a9ac700 1 -- 172.21.15.7:0/954176039 learned_addr learned my addr 172.21.15.7:0/954176039 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:28:11.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f999aa700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f94155880 0x7f6f941ed6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:11.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9a1ab700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f80003a70 0x7f6f941572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:11.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9a9ac700 1 -- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f80003a70 msgr2=0x7f6f941572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:11.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9a9ac700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f80003a70 0x7f6f941572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:11.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9a9ac700 1 -- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f94155880 msgr2=0x7f6f941ed6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:11.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9a9ac700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f94155880 0x7f6f941ed6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:11.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9a9ac700 1 -- 172.21.15.7:0/954176039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9406f6b0 con 0x7f6f94146440 2023-03-03T14:28:11.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9a9ac700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f94146440 0x7f6f94143520 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f6f940c2900 tx=0x7f6f8c00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:28:11.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f86ffd700 1 -- 172.21.15.7:0/954176039 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c00ec30 con 0x7f6f94146440 2023-03-03T14:28:11.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f86ffd700 1 -- 172.21.15.7:0/954176039 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c00ed90 con 0x7f6f94146440 2023-03-03T14:28:11.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9c40f700 1 -- 172.21.15.7:0/954176039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f94155160 con 0x7f6f94146440 2023-03-03T14:28:11.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f86ffd700 1 -- 172.21.15.7:0/954176039 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c013660 con 0x7f6f94146440 2023-03-03T14:28:11.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.830+0000 7f6f9c40f700 1 -- 172.21.15.7:0/954176039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9406f990 con 0x7f6f94146440 2023-03-03T14:28:11.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.831+0000 7f6f9c40f700 1 -- 172.21.15.7:0/954176039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f94146120 con 0x7f6f94146440 2023-03-03T14:28:11.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.832+0000 7f6f86ffd700 1 -- 172.21.15.7:0/954176039 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6f8c0137c0 con 0x7f6f94146440 2023-03-03T14:28:11.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.832+0000 7f6f86ffd700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f780682f0 0x7f6f7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:11.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.832+0000 7f6f86ffd700 1 -- 172.21.15.7:0/954176039 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6f8c086710 con 0x7f6f94146440 2023-03-03T14:28:11.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.832+0000 7f6f9a1ab700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f780682f0 0x7f6f7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:11.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.833+0000 7f6f9a1ab700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f780682f0 0x7f6f7806a7a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f6f90007fd0 tx=0x7f6f90009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:28:11.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:11.837+0000 7f6f86ffd700 1 -- 172.21.15.7:0/954176039 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6f8c052ab0 con 0x7f6f94146440 2023-03-03T14:28:12.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.075+0000 7f6f9c40f700 1 -- 172.21.15.7:0/954176039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f94074bb0 con 0x7f6f94146440 2023-03-03T14:28:12.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.076+0000 7f6f86ffd700 1 -- 172.21.15.7:0/954176039 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6f8c0560d0 con 0x7f6f94146440 2023-03-03T14:28:12.078 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:28:12.078 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:28:12.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 -- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f780682f0 msgr2=0x7f6f7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:12.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f780682f0 0x7f6f7806a7a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f6f90007fd0 tx=0x7f6f90009030 comp rx=0 tx=0).stop 2023-03-03T14:28:12.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 -- 172.21.15.7:0/954176039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f94146440 msgr2=0x7f6f94143520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:12.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f94146440 0x7f6f94143520 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f6f940c2900 tx=0x7f6f8c00e440 comp rx=0 tx=0).stop 2023-03-03T14:28:12.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 -- 172.21.15.7:0/954176039 shutdown_connections 2023-03-03T14:28:12.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f780682f0 0x7f6f7806a7a0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:12.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f94146440 0x7f6f94143520 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:12.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f80003a70 0x7f6f941572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:12.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 --2- 172.21.15.7:0/954176039 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f94155880 0x7f6f941ed6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:12.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 -- 172.21.15.7:0/954176039 >> 172.21.15.7:0/954176039 conn(0x7f6f940c8900 msgr2=0x7f6f94157cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:28:12.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 -- 172.21.15.7:0/954176039 shutdown_connections 2023-03-03T14:28:12.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:12.078+0000 7f6f9c40f700 1 -- 172.21.15.7:0/954176039 wait complete. 2023-03-03T14:28:12.082 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:28:12.091 INFO:tasks.mds_thrash.fs.[b]:waiting for 89 secs before thrashing 2023-03-03T14:28:23.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:23.567 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:30.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:28:30.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.993+0000 7f30e7883700 1 Processor -- start 2023-03-03T14:28:30.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.993+0000 7f30e7883700 1 -- start start 2023-03-03T14:28:30.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.994+0000 7f30e7883700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c6010 0x7f30e00c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.994+0000 7f30e7883700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f30e00c17f0 0x7f30e00c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.994+0000 7f30e7883700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e00c5500 con 0x7f30e006d6a0 2023-03-03T14:28:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.994+0000 7f30e7883700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e0152100 con 0x7f30e00c17f0 2023-03-03T14:28:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.994+0000 7f30e7883700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e014ebe0 con 0x7f30e00c6010 2023-03-03T14:28:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30e5e20700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f30e006d6a0 0x7f30e006db00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:46392/0 (socket says 172.21.15.7:46392) 2023-03-03T14:28:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30e4e1e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f30e00c17f0 0x7f30e00c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:30.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30e5e20700 1 -- 172.21.15.7:0/2820674818 learned_addr learned my addr 172.21.15.7:0/2820674818 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:28:30.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30e561f700 1 --2- 172.21.15.7:0/2820674818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c6010 0x7f30e00c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:30.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30d7fff700 1 -- 172.21.15.7:0/2820674818 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209499845 0 0) 0x7f30e00c5500 con 0x7f30e006d6a0 2023-03-03T14:28:30.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30d7fff700 1 -- 172.21.15.7:0/2820674818 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30cc0035b0 con 0x7f30e006d6a0 2023-03-03T14:28:30.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30e4e1e700 1 -- 172.21.15.7:0/2820674818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c6010 msgr2=0x7f30e00c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:28:30.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30e4e1e700 1 --2- 172.21.15.7:0/2820674818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c6010 0x7f30e00c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:30.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30e4e1e700 1 -- 172.21.15.7:0/2820674818 >> v1:172.21.15.143:6789/0 conn(0x7f30e006d6a0 legacy=0x7f30e006db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:30.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30e4e1e700 1 -- 172.21.15.7:0/2820674818 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e00c2e70 con 0x7f30e00c17f0 2023-03-03T14:28:31.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30e561f700 1 --2- 172.21.15.7:0/2820674818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c6010 0x7f30e00c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:28:31.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30e4e1e700 1 --2- 172.21.15.7:0/2820674818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30e00c17f0 0x7f30e00c4fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009840 tx=0x7f30dc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e858a55c8641e2b4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:28:31.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30d7fff700 1 -- 172.21.15.7:0/2820674818 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc011830 con 0x7f30e00c17f0 2023-03-03T14:28:31.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30d7fff700 1 -- 172.21.15.7:0/2820674818 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30dc011990 con 0x7f30e00c17f0 2023-03-03T14:28:31.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.995+0000 7f30d7fff700 1 -- 172.21.15.7:0/2820674818 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc011d00 con 0x7f30e00c17f0 2023-03-03T14:28:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.996+0000 7f30e7883700 1 -- 172.21.15.7:0/2820674818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30e00c17f0 msgr2=0x7f30e00c4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.996+0000 7f30e7883700 1 --2- 172.21.15.7:0/2820674818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30e00c17f0 0x7f30e00c4fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009840 tx=0x7f30dc009b50 comp rx=0 tx=0).stop 2023-03-03T14:28:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.996+0000 7f30e7883700 1 -- 172.21.15.7:0/2820674818 shutdown_connections 2023-03-03T14:28:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.996+0000 7f30e7883700 1 --2- 172.21.15.7:0/2820674818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c6010 0x7f30e00c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:31.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.996+0000 7f30e7883700 1 --2- 172.21.15.7:0/2820674818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30e00c17f0 0x7f30e00c4fc0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.996+0000 7f30e7883700 1 -- 172.21.15.7:0/2820674818 >> 172.21.15.7:0/2820674818 conn(0x7f30e00a9430 msgr2=0x7f30e00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:28:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.996+0000 7f30e7883700 1 -- 172.21.15.7:0/2820674818 shutdown_connections 2023-03-03T14:28:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.996+0000 7f30e7883700 1 -- 172.21.15.7:0/2820674818 wait complete. 2023-03-03T14:28:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.997+0000 7f30e7883700 1 Processor -- start 2023-03-03T14:28:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.997+0000 7f30e7883700 1 -- start start 2023-03-03T14:28:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e7883700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30e006d6a0 0x7f30e01e8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:31.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e7883700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c17f0 0x7f30e0154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e7883700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e00c6010 0x7f30e00a7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e7883700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e01e8830 con 0x7f30e00c6010 2023-03-03T14:28:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e7883700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e0152b00 con 0x7f30e006d6a0 2023-03-03T14:28:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e7883700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e0153bf0 con 0x7f30e00c17f0 2023-03-03T14:28:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e5e20700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e00c6010 0x7f30e00a7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e5e20700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e00c6010 0x7f30e00a7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48896/0 (socket says 172.21.15.7:48896) 2023-03-03T14:28:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e5e20700 1 -- 172.21.15.7:0/2471959989 learned_addr learned my addr 172.21.15.7:0/2471959989 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:28:31.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e4e1e700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c17f0 0x7f30e0154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e561f700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30e006d6a0 0x7f30e01e8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e5e20700 1 -- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c17f0 msgr2=0x7f30e0154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e5e20700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c17f0 0x7f30e0154be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e5e20700 1 -- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30e006d6a0 msgr2=0x7f30e01e8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e5e20700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30e006d6a0 0x7f30e01e8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.998+0000 7f30e5e20700 1 -- 172.21.15.7:0/2471959989 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d00094a0 con 0x7f30e00c6010 2023-03-03T14:28:31.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.999+0000 7f30e5e20700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e00c6010 0x7f30e00a7730 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f30d800cb40 tx=0x7f30d800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:28:31.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.999+0000 7f30d5ffb700 1 -- 172.21.15.7:0/2471959989 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d800a9e0 con 0x7f30e00c6010 2023-03-03T14:28:31.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.999+0000 7f30d5ffb700 1 -- 172.21.15.7:0/2471959989 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d800ab40 con 0x7f30e00c6010 2023-03-03T14:28:31.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.999+0000 7f30e7883700 1 -- 172.21.15.7:0/2471959989 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30dc0094a0 con 0x7f30e00c6010 2023-03-03T14:28:31.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.999+0000 7f30d5ffb700 1 -- 172.21.15.7:0/2471959989 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d8012400 con 0x7f30e00c6010 2023-03-03T14:28:31.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:30.999+0000 7f30e7883700 1 -- 172.21.15.7:0/2471959989 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e00aa960 con 0x7f30e00c6010 2023-03-03T14:28:31.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.000+0000 7f30e7883700 1 -- 172.21.15.7:0/2471959989 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e00aa680 con 0x7f30e00c6010 2023-03-03T14:28:31.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.000+0000 7f30d5ffb700 1 -- 172.21.15.7:0/2471959989 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f30d8019030 con 0x7f30e00c6010 2023-03-03T14:28:31.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.000+0000 7f30d5ffb700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30c40682f0 0x7f30c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:28:31.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.001+0000 7f30d5ffb700 1 -- 172.21.15.7:0/2471959989 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f30d8085b40 con 0x7f30e00c6010 2023-03-03T14:28:31.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.001+0000 7f30e561f700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30c40682f0 0x7f30c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:28:31.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.001+0000 7f30e561f700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30c40682f0 0x7f30c406a7a0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009cd0 tx=0x7f30d0009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:28:31.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.005+0000 7f30d5ffb700 1 -- 172.21.15.7:0/2471959989 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30d804b5e0 con 0x7f30e00c6010 2023-03-03T14:28:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.241+0000 7f30e7883700 1 -- 172.21.15.7:0/2471959989 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30e00c2b00 con 0x7f30e00c6010 2023-03-03T14:28:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.241+0000 7f30d5ffb700 1 -- 172.21.15.7:0/2471959989 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f30d8051f90 con 0x7f30e00c6010 2023-03-03T14:28:31.244 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:28:31.244 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:28:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 -- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30c40682f0 msgr2=0x7f30c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30c40682f0 0x7f30c406a7a0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009cd0 tx=0x7f30d0009210 comp rx=0 tx=0).stop 2023-03-03T14:28:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 -- 172.21.15.7:0/2471959989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e00c6010 msgr2=0x7f30e00a7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:28:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e00c6010 0x7f30e00a7730 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f30d800cb40 tx=0x7f30d800ce50 comp rx=0 tx=0).stop 2023-03-03T14:28:31.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 -- 172.21.15.7:0/2471959989 shutdown_connections 2023-03-03T14:28:31.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30c40682f0 0x7f30c406a7a0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:31.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30e00c6010 0x7f30e00a7730 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:31.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30e006d6a0 0x7f30e01e8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:31.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 --2- 172.21.15.7:0/2471959989 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30e00c17f0 0x7f30e0154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:28:31.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 -- 172.21.15.7:0/2471959989 >> 172.21.15.7:0/2471959989 conn(0x7f30e00a9430 msgr2=0x7f30e00a9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:28:31.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 -- 172.21.15.7:0/2471959989 shutdown_connections 2023-03-03T14:28:31.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:28:31.244+0000 7f30e7883700 1 -- 172.21.15.7:0/2471959989 wait complete. 2023-03-03T14:28:31.248 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:28:31.258 INFO:tasks.mds_thrash.fs.[a]:waiting for 61 secs before thrashing 2023-03-03T14:28:53.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:28:53.631 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:23.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:23.707 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:32.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:29:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.407+0000 7fe1e9771700 1 Processor -- start 2023-03-03T14:29:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.407+0000 7fe1e9771700 1 -- start start 2023-03-03T14:29:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.407+0000 7fe1e9771700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e40c00e0 0x7fe1e40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:32.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.407+0000 7fe1e9771700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e40c7310 con 0x7fe1e40caa30 2023-03-03T14:29:32.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.407+0000 7fe1e9771700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e40bfd10 con 0x7fe1e4154ff0 2023-03-03T14:29:32.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.407+0000 7fe1e9771700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40bfa30 con 0x7fe1e40c00e0 2023-03-03T14:29:32.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.408+0000 7fe1e2ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe1e40caa30 0x7fe1e40c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42948/0 (socket says 172.21.15.7:42948) 2023-03-03T14:29:32.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.408+0000 7fe1e2ffd700 1 -- 172.21.15.7:0/298838865 learned_addr learned my addr 172.21.15.7:0/298838865 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:29:32.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.408+0000 7fe1e27fc700 1 --2- 172.21.15.7:0/298838865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e40c00e0 0x7fe1e40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:32.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.408+0000 7fe1e1ffb700 1 -- 172.21.15.7:0/298838865 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904122547 0 0) 0x7fe1e40c7310 con 0x7fe1e40caa30 2023-03-03T14:29:32.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.408+0000 7fe1e1ffb700 1 -- 172.21.15.7:0/298838865 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d00035b0 con 0x7fe1e40caa30 2023-03-03T14:29:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.408+0000 7fe1e1ffb700 1 -- 172.21.15.7:0/298838865 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167306412 0 0) 0x7fe1e40bfd10 con 0x7fe1e4154ff0 2023-03-03T14:29:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.408+0000 7fe1e27fc700 1 -- 172.21.15.7:0/298838865 >> v1:172.21.15.143:6789/0 conn(0x7fe1e4154ff0 legacy=0x7fe1e415ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.408+0000 7fe1e27fc700 1 -- 172.21.15.7:0/298838865 >> v1:172.21.15.7:6789/0 conn(0x7fe1e40caa30 legacy=0x7fe1e40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.408+0000 7fe1e27fc700 1 -- 172.21.15.7:0/298838865 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e4154510 con 0x7fe1e40c00e0 2023-03-03T14:29:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.408+0000 7fe1e27fc700 1 --2- 172.21.15.7:0/298838865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e40c00e0 0x7fe1e40c5c60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009850 tx=0x7fe1cc009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78ecefcc3334bae6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:29:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.409+0000 7fe1e1ffb700 1 -- 172.21.15.7:0/298838865 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4001670 con 0x7fe1e40c00e0 2023-03-03T14:29:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.409+0000 7fe1e1ffb700 1 -- 172.21.15.7:0/298838865 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc005c30 con 0x7fe1e40c00e0 2023-03-03T14:29:32.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.409+0000 7fe1e1ffb700 1 -- 172.21.15.7:0/298838865 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc006030 con 0x7fe1e40c00e0 2023-03-03T14:29:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.409+0000 7fe1e9771700 1 -- 172.21.15.7:0/298838865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e40c00e0 msgr2=0x7fe1e40c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.409+0000 7fe1e9771700 1 --2- 172.21.15.7:0/298838865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e40c00e0 0x7fe1e40c5c60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009850 tx=0x7fe1cc009b60 comp rx=0 tx=0).stop 2023-03-03T14:29:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.409+0000 7fe1e9771700 1 -- 172.21.15.7:0/298838865 shutdown_connections 2023-03-03T14:29:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.409+0000 7fe1e9771700 1 --2- 172.21.15.7:0/298838865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e40c00e0 0x7fe1e40c5c60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.409+0000 7fe1e9771700 1 -- 172.21.15.7:0/298838865 >> 172.21.15.7:0/298838865 conn(0x7fe1e4066e40 msgr2=0x7fe1e4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:29:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.409+0000 7fe1e9771700 1 -- 172.21.15.7:0/298838865 shutdown_connections 2023-03-03T14:29:32.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.409+0000 7fe1e9771700 1 -- 172.21.15.7:0/298838865 wait complete. 2023-03-03T14:29:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.411+0000 7fe1e9771700 1 Processor -- start 2023-03-03T14:29:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.411+0000 7fe1e9771700 1 -- start start 2023-03-03T14:29:32.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e9771700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e40c00e0 0x7fe1e40c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e9771700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1e40caa30 0x7fe1e4157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e9771700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e4154ff0 0x7fe1e4157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e9771700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e41ed0c0 con 0x7fe1e40caa30 2023-03-03T14:29:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e9771700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e41de000 con 0x7fe1e40c00e0 2023-03-03T14:29:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e9771700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e41dda40 con 0x7fe1e4154ff0 2023-03-03T14:29:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1e40caa30 0x7fe1e4157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:32.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1e40caa30 0x7fe1e4157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48976/0 (socket says 172.21.15.7:48976) 2023-03-03T14:29:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e27fc700 1 -- 172.21.15.7:0/2779377559 learned_addr learned my addr 172.21.15.7:0/2779377559 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:29:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e2ffd700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e40c00e0 0x7fe1e40c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e37fe700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e4154ff0 0x7fe1e4157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e27fc700 1 -- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e4154ff0 msgr2=0x7fe1e4157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e27fc700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e4154ff0 0x7fe1e4157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e27fc700 1 -- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e40c00e0 msgr2=0x7fe1e40c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e27fc700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e40c00e0 0x7fe1e40c89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:32.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e27fc700 1 -- 172.21.15.7:0/2779377559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc009430 con 0x7fe1e40caa30 2023-03-03T14:29:32.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e2ffd700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e40c00e0 0x7fe1e40c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:29:32.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.412+0000 7fe1e27fc700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1e40caa30 0x7fe1e4157140 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc011000 tx=0x7fe1cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:29:32.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.413+0000 7fe1db7fe700 1 -- 172.21.15.7:0/2779377559 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc012680 con 0x7fe1e40caa30 2023-03-03T14:29:32.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.413+0000 7fe1db7fe700 1 -- 172.21.15.7:0/2779377559 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc012820 con 0x7fe1e40caa30 2023-03-03T14:29:32.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.413+0000 7fe1e9771700 1 -- 172.21.15.7:0/2779377559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e4154ad0 con 0x7fe1e40caa30 2023-03-03T14:29:32.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.413+0000 7fe1db7fe700 1 -- 172.21.15.7:0/2779377559 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc012bd0 con 0x7fe1e40caa30 2023-03-03T14:29:32.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.413+0000 7fe1e9771700 1 -- 172.21.15.7:0/2779377559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e41547f0 con 0x7fe1e40caa30 2023-03-03T14:29:32.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.414+0000 7fe1e9771700 1 -- 172.21.15.7:0/2779377559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e40bf650 con 0x7fe1e40caa30 2023-03-03T14:29:32.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.415+0000 7fe1db7fe700 1 -- 172.21.15.7:0/2779377559 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe1cc00f050 con 0x7fe1e40caa30 2023-03-03T14:29:32.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.416+0000 7fe1db7fe700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1c40682f0 0x7fe1c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:32.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.416+0000 7fe1db7fe700 1 -- 172.21.15.7:0/2779377559 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe1cc088090 con 0x7fe1e40caa30 2023-03-03T14:29:32.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.416+0000 7fe1e2ffd700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1c40682f0 0x7fe1c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:32.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.417+0000 7fe1e2ffd700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1c40682f0 0x7fe1c406a7a0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40017a0 tx=0x7fe1d40016f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:29:32.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.419+0000 7fe1db7fe700 1 -- 172.21.15.7:0/2779377559 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe1cc0543c0 con 0x7fe1e40caa30 2023-03-03T14:29:32.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.658+0000 7fe1e9771700 1 -- 172.21.15.7:0/2779377559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1e40c5030 con 0x7fe1e40caa30 2023-03-03T14:29:32.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.659+0000 7fe1db7fe700 1 -- 172.21.15.7:0/2779377559 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe1cc017020 con 0x7fe1e40caa30 2023-03-03T14:29:32.716 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:29:32.717 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:29:32.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 -- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1c40682f0 msgr2=0x7fe1c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:32.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1c40682f0 0x7fe1c406a7a0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40017a0 tx=0x7fe1d40016f0 comp rx=0 tx=0).stop 2023-03-03T14:29:32.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 -- 172.21.15.7:0/2779377559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1e40caa30 msgr2=0x7fe1e4157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:32.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1e40caa30 0x7fe1e4157140 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc011000 tx=0x7fe1cc004750 comp rx=0 tx=0).stop 2023-03-03T14:29:32.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 -- 172.21.15.7:0/2779377559 shutdown_connections 2023-03-03T14:29:32.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1e4154ff0 0x7fe1e4157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:32.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1e40c00e0 0x7fe1e40c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:32.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1c40682f0 0x7fe1c406a7a0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:32.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 --2- 172.21.15.7:0/2779377559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1e40caa30 0x7fe1e4157140 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:32.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 -- 172.21.15.7:0/2779377559 >> 172.21.15.7:0/2779377559 conn(0x7fe1e4066e40 msgr2=0x7fe1e4152550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:29:32.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 -- 172.21.15.7:0/2779377559 shutdown_connections 2023-03-03T14:29:32.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:32.661+0000 7fe1e9771700 1 -- 172.21.15.7:0/2779377559 wait complete. 2023-03-03T14:29:32.720 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:29:32.720 INFO:tasks.mds_thrash.fs.[a]:waiting for 78 secs before thrashing 2023-03-03T14:29:41.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:29:41.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.229+0000 7f9175b4e700 1 Processor -- start 2023-03-03T14:29:41.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.229+0000 7f9175b4e700 1 -- start start 2023-03-03T14:29:41.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f9175b4e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f917014fda0 0x7f9170156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:41.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f9175b4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91700abe20 0x7f91701529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:41.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f9175b4e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91700c2f30 con 0x7f917014f9d0 2023-03-03T14:29:41.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f9175b4e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91700c31d0 con 0x7f917014fda0 2023-03-03T14:29:41.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f9175b4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91701524a0 con 0x7f91700abe20 2023-03-03T14:29:41.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f917014fda0 0x7f9170156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f917014fda0 0x7f9170156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48992/0 (socket says 172.21.15.7:48992) 2023-03-03T14:29:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916effd700 1 -- 172.21.15.7:0/3018612476 learned_addr learned my addr 172.21.15.7:0/3018612476 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:29:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916ffff700 1 --2- 172.21.15.7:0/3018612476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91700abe20 0x7f91701529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916effd700 1 -- 172.21.15.7:0/3018612476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91700abe20 msgr2=0x7f91701529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:29:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916effd700 1 --2- 172.21.15.7:0/3018612476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91700abe20 0x7f91701529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916effd700 1 -- 172.21.15.7:0/3018612476 >> v1:172.21.15.143:6789/0 conn(0x7f917014f9d0 legacy=0x7f9170071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:41.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916effd700 1 -- 172.21.15.7:0/3018612476 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91700c34b0 con 0x7f917014fda0 2023-03-03T14:29:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916effd700 1 --2- 172.21.15.7:0/3018612476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f917014fda0 0x7f9170156430 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f91600098f0 tx=0x7f9160009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba4ab3b493901bf4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:29:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916e7fc700 1 -- 172.21.15.7:0/3018612476 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000b8a0 con 0x7f917014fda0 2023-03-03T14:29:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916e7fc700 1 -- 172.21.15.7:0/3018612476 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916000ba00 con 0x7f917014fda0 2023-03-03T14:29:41.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.231+0000 7f916e7fc700 1 -- 172.21.15.7:0/3018612476 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916000bd90 con 0x7f917014fda0 2023-03-03T14:29:41.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.232+0000 7f9175b4e700 1 -- 172.21.15.7:0/3018612476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f917014fda0 msgr2=0x7f9170156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:41.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.232+0000 7f9175b4e700 1 --2- 172.21.15.7:0/3018612476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f917014fda0 0x7f9170156430 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f91600098f0 tx=0x7f9160009c00 comp rx=0 tx=0).stop 2023-03-03T14:29:41.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.232+0000 7f9175b4e700 1 -- 172.21.15.7:0/3018612476 shutdown_connections 2023-03-03T14:29:41.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.232+0000 7f9175b4e700 1 --2- 172.21.15.7:0/3018612476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91700abe20 0x7f91701529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:41.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.232+0000 7f9175b4e700 1 --2- 172.21.15.7:0/3018612476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f917014fda0 0x7f9170156430 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:41.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.232+0000 7f9175b4e700 1 -- 172.21.15.7:0/3018612476 >> 172.21.15.7:0/3018612476 conn(0x7f91700c8900 msgr2=0x7f91700c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:29:41.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.232+0000 7f9175b4e700 1 -- 172.21.15.7:0/3018612476 shutdown_connections 2023-03-03T14:29:41.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.232+0000 7f9175b4e700 1 -- 172.21.15.7:0/3018612476 wait complete. 2023-03-03T14:29:41.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.233+0000 7f9175b4e700 1 Processor -- start 2023-03-03T14:29:41.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f9175b4e700 1 -- start start 2023-03-03T14:29:41.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f9175b4e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91700abe20 0x7f91700bdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f9175b4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f917014f9d0 0x7f91700be320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f9175b4e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f917014fda0 0x7f917014b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f9175b4e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917014c3e0 con 0x7f91700abe20 2023-03-03T14:29:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f9175b4e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91701f2160 con 0x7f917014fda0 2023-03-03T14:29:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f9175b4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91700bf760 con 0x7f917014f9d0 2023-03-03T14:29:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f916f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91700abe20 0x7f91700bdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:41.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f916f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91700abe20 0x7f91700bdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48996/0 (socket says 172.21.15.7:48996) 2023-03-03T14:29:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f916f7fe700 1 -- 172.21.15.7:0/4244061364 learned_addr learned my addr 172.21.15.7:0/4244061364 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:29:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f916effd700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f917014f9d0 0x7f91700be320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.234+0000 7f916ffff700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f917014fda0 0x7f917014b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f916f7fe700 1 -- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f917014f9d0 msgr2=0x7f91700be320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f916f7fe700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f917014f9d0 0x7f91700be320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f916f7fe700 1 -- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f917014fda0 msgr2=0x7f917014b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:41.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f916f7fe700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f917014fda0 0x7f917014b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f916f7fe700 1 -- 172.21.15.7:0/4244061364 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9160009550 con 0x7f91700abe20 2023-03-03T14:29:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f916f7fe700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91700abe20 0x7f91700bdde0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f915800c4e0 tx=0x7f915800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:29:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f9167fff700 1 -- 172.21.15.7:0/4244061364 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91580052d0 con 0x7f91700abe20 2023-03-03T14:29:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f9167fff700 1 -- 172.21.15.7:0/4244061364 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158005430 con 0x7f91700abe20 2023-03-03T14:29:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f9175b4e700 1 -- 172.21.15.7:0/4244061364 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91700ac350 con 0x7f91700abe20 2023-03-03T14:29:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f9167fff700 1 -- 172.21.15.7:0/4244061364 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91580038c0 con 0x7f91700abe20 2023-03-03T14:29:41.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.235+0000 7f9175b4e700 1 -- 172.21.15.7:0/4244061364 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91700ab630 con 0x7f91700abe20 2023-03-03T14:29:41.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.236+0000 7f9175b4e700 1 -- 172.21.15.7:0/4244061364 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9170071fa0 con 0x7f91700abe20 2023-03-03T14:29:41.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.237+0000 7f9167fff700 1 -- 172.21.15.7:0/4244061364 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9158019030 con 0x7f91700abe20 2023-03-03T14:29:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.237+0000 7f9167fff700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91500682f0 0x7f915006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:29:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.237+0000 7f9167fff700 1 -- 172.21.15.7:0/4244061364 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f91580853b0 con 0x7f91700abe20 2023-03-03T14:29:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.238+0000 7f916effd700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91500682f0 0x7f915006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:29:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.238+0000 7f916effd700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91500682f0 0x7f915006a7a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f91701501d0 tx=0x7f9160008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:29:41.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.242+0000 7f9167fff700 1 -- 172.21.15.7:0/4244061364 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9158051750 con 0x7f91700abe20 2023-03-03T14:29:41.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.478+0000 7f9175b4e700 1 -- 172.21.15.7:0/4244061364 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91700bf8a0 con 0x7f91700abe20 2023-03-03T14:29:41.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.479+0000 7f9167fff700 1 -- 172.21.15.7:0/4244061364 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f91700bf8a0 con 0x7f91700abe20 2023-03-03T14:29:41.482 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:29:41.482 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:29:41.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 -- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91500682f0 msgr2=0x7f915006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:41.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91500682f0 0x7f915006a7a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f91701501d0 tx=0x7f9160008030 comp rx=0 tx=0).stop 2023-03-03T14:29:41.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 -- 172.21.15.7:0/4244061364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91700abe20 msgr2=0x7f91700bdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:29:41.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91700abe20 0x7f91700bdde0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f915800c4e0 tx=0x7f915800c7f0 comp rx=0 tx=0).stop 2023-03-03T14:29:41.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 -- 172.21.15.7:0/4244061364 shutdown_connections 2023-03-03T14:29:41.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f917014fda0 0x7f917014b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:41.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91500682f0 0x7f915006a7a0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:41.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91700abe20 0x7f91700bdde0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:41.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 --2- 172.21.15.7:0/4244061364 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f917014f9d0 0x7f91700be320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:29:41.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 -- 172.21.15.7:0/4244061364 >> 172.21.15.7:0/4244061364 conn(0x7f91700c8900 msgr2=0x7f91700c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:29:41.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 -- 172.21.15.7:0/4244061364 shutdown_connections 2023-03-03T14:29:41.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:29:41.481+0000 7f9175b4e700 1 -- 172.21.15.7:0/4244061364 wait complete. 2023-03-03T14:29:41.485 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:29:41.495 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-03-03T14:29:53.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:29:53.774 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T14:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T14:30:23.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:23.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:50.721 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:30:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.859+0000 7f39929f6700 1 Processor -- start 2023-03-03T14:30:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.859+0000 7f39929f6700 1 -- start start 2023-03-03T14:30:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f39929f6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f398c06e020 0x7f398c071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f39929f6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c072520 0x7f398c151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f39929f6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398c154910 con 0x7f398c155390 2023-03-03T14:30:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f39929f6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f398c070b90 con 0x7f398c06e020 2023-03-03T14:30:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f39929f6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398c151320 con 0x7f398c072520 2023-03-03T14:30:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f398c155390 0x7f398c0c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43032/0 (socket says 172.21.15.7:43032) 2023-03-03T14:30:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398bfff700 1 -- 172.21.15.7:0/423605267 learned_addr learned my addr 172.21.15.7:0/423605267 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:30:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959372514 0 0) 0x7f398c154910 con 0x7f398c155390 2023-03-03T14:30:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39740035b0 con 0x7f398c155390 2023-03-03T14:30:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398b7fe700 1 --2- 172.21.15.7:0/423605267 >> v2:172.21.15.143:3300/0 conn(0x7f398c06e020 0x7f398c071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f3990f93700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c072520 0x7f398c151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 33591732 0 0) 0x7f39740035b0 con 0x7f398c155390 2023-03-03T14:30:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c072520 msgr2=0x7f398c151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:30:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c072520 0x7f398c151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 >> v2:172.21.15.143:3300/0 conn(0x7f398c06e020 msgr2=0x7f398c071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:30:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 --2- 172.21.15.7:0/423605267 >> v2:172.21.15.143:3300/0 conn(0x7f398c06e020 0x7f398c071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c0c4540 con 0x7f398c155390 2023-03-03T14:30:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f397c001670 con 0x7f398c155390 2023-03-03T14:30:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398b7fe700 1 --2- 172.21.15.7:0/423605267 >> v2:172.21.15.143:3300/0 conn(0x7f398c06e020 0x7f398c071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:30:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 >> v1:172.21.15.7:6789/0 conn(0x7f398c155390 legacy=0x7f398c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c072520 0x7f3974003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3974003ff0 0x7f3974004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3974004d90 0x7f3974005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c070b90 con 0x7f3974003ff0 2023-03-03T14:30:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c151320 con 0x7f3974004d90 2023-03-03T14:30:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.860+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397400bb70 con 0x7f398c072520 2023-03-03T14:30:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398b7fe700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3974003ff0 0x7f3974004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398bfff700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c072520 0x7f3974003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:50.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f3990f93700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3974004d90 0x7f3974005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:50.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398b7fe700 1 -- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c072520 msgr2=0x7f3974003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:30:50.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398b7fe700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c072520 0x7f3974003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:50.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398b7fe700 1 -- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3974004d90 msgr2=0x7f3974005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:30:50.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398b7fe700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3974004d90 0x7f3974005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:50.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398b7fe700 1 -- 172.21.15.7:0/423605267 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39740035b0 con 0x7f3974003ff0 2023-03-03T14:30:50.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398b7fe700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3974003ff0 0x7f3974004850 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3980009920 tx=0x7f3974005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e108a2b2f7c4180f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:30:50.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398000c070 con 0x7f3974003ff0 2023-03-03T14:30:50.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3980013610 con 0x7f3974003ff0 2023-03-03T14:30:50.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.861+0000 7f398affd700 1 -- 172.21.15.7:0/423605267 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980013980 con 0x7f3974003ff0 2023-03-03T14:30:50.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.862+0000 7f39929f6700 1 -- 172.21.15.7:0/423605267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3974003ff0 msgr2=0x7f3974004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:50.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.862+0000 7f39929f6700 1 --2- 172.21.15.7:0/423605267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3974003ff0 0x7f3974004850 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3980009920 tx=0x7f3974005880 comp rx=0 tx=0).stop 2023-03-03T14:30:50.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.862+0000 7f3990f93700 1 -- 172.21.15.7:0/423605267 reap_dead start 2023-03-03T14:30:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.862+0000 7f39929f6700 1 -- 172.21.15.7:0/423605267 shutdown_connections 2023-03-03T14:30:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.862+0000 7f39929f6700 1 -- 172.21.15.7:0/423605267 >> 172.21.15.7:0/423605267 conn(0x7f398c06c110 msgr2=0x7f398c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:30:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.862+0000 7f39929f6700 1 -- 172.21.15.7:0/423605267 shutdown_connections 2023-03-03T14:30:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.862+0000 7f39929f6700 1 -- 172.21.15.7:0/423605267 wait complete. 2023-03-03T14:30:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.863+0000 7f39929f6700 1 Processor -- start 2023-03-03T14:30:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.864+0000 7f39929f6700 1 -- start start 2023-03-03T14:30:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f39929f6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3974003ff0 0x7f398c1ecf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f39929f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f398c153cc0 0x7f398c1eabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f39929f6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c1ed480 0x7f398c1eb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f39929f6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c152b90 con 0x7f398c153cc0 2023-03-03T14:30:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f39929f6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c153430 con 0x7f3974003ff0 2023-03-03T14:30:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f39929f6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398c1eda50 con 0x7f398c1ed480 2023-03-03T14:30:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f398b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f398c153cc0 0x7f398c1eabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f398b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f398c153cc0 0x7f398c1eabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49066/0 (socket says 172.21.15.7:49066) 2023-03-03T14:30:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f398b7fe700 1 -- 172.21.15.7:0/3600414322 learned_addr learned my addr 172.21.15.7:0/3600414322 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:30:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f3990f93700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c1ed480 0x7f398c1eb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f398bfff700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3974003ff0 0x7f398c1ecf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f398b7fe700 1 -- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c1ed480 msgr2=0x7f398c1eb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f398b7fe700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c1ed480 0x7f398c1eb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f398b7fe700 1 -- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3974003ff0 msgr2=0x7f398c1ecf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f398b7fe700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3974003ff0 0x7f398c1ecf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f398b7fe700 1 -- 172.21.15.7:0/3600414322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c070540 con 0x7f398c153cc0 2023-03-03T14:30:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f3990f93700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c1ed480 0x7f398c1eb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:30:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f398b7fe700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f398c153cc0 0x7f398c1eabd0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f397400bac0 tx=0x7f39800047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:30:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f3988ff9700 1 -- 172.21.15.7:0/3600414322 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980013b90 con 0x7f398c153cc0 2023-03-03T14:30:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f3988ff9700 1 -- 172.21.15.7:0/3600414322 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3980013cf0 con 0x7f398c153cc0 2023-03-03T14:30:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f39929f6700 1 -- 172.21.15.7:0/3600414322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398c0702a0 con 0x7f398c153cc0 2023-03-03T14:30:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f3988ff9700 1 -- 172.21.15.7:0/3600414322 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39800038e0 con 0x7f398c153cc0 2023-03-03T14:30:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.865+0000 7f39929f6700 1 -- 172.21.15.7:0/3600414322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398c070820 con 0x7f398c153cc0 2023-03-03T14:30:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.867+0000 7f39929f6700 1 -- 172.21.15.7:0/3600414322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398c074bb0 con 0x7f398c153cc0 2023-03-03T14:30:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.867+0000 7f3988ff9700 1 -- 172.21.15.7:0/3600414322 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3980003a40 con 0x7f398c153cc0 2023-03-03T14:30:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.867+0000 7f3988ff9700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39780682f0 0x7f397806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.867+0000 7f3988ff9700 1 -- 172.21.15.7:0/3600414322 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f39800873b0 con 0x7f398c153cc0 2023-03-03T14:30:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.868+0000 7f398bfff700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39780682f0 0x7f397806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.868+0000 7f398bfff700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39780682f0 0x7f397806a7a0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f397c000fb0 tx=0x7f397c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:30:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:50.872+0000 7f3988ff9700 1 -- 172.21.15.7:0/3600414322 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3980053750 con 0x7f398c153cc0 2023-03-03T14:30:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.106+0000 7f39929f6700 1 -- 172.21.15.7:0/3600414322 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f398c0c1da0 con 0x7f398c153cc0 2023-03-03T14:30:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.107+0000 7f3988ff9700 1 -- 172.21.15.7:0/3600414322 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3980013e60 con 0x7f398c153cc0 2023-03-03T14:30:51.110 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:30:51.110 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:30:51.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 -- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39780682f0 msgr2=0x7f397806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:51.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39780682f0 0x7f397806a7a0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f397c000fb0 tx=0x7f397c009040 comp rx=0 tx=0).stop 2023-03-03T14:30:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 -- 172.21.15.7:0/3600414322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f398c153cc0 msgr2=0x7f398c1eabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f398c153cc0 0x7f398c1eabd0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f397400bac0 tx=0x7f39800047c0 comp rx=0 tx=0).stop 2023-03-03T14:30:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 -- 172.21.15.7:0/3600414322 shutdown_connections 2023-03-03T14:30:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f398c1ed480 0x7f398c1eb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3974003ff0 0x7f398c1ecf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39780682f0 0x7f397806a7a0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 --2- 172.21.15.7:0/3600414322 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f398c153cc0 0x7f398c1eabd0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 -- 172.21.15.7:0/3600414322 >> 172.21.15.7:0/3600414322 conn(0x7f398c06c110 msgr2=0x7f398c0c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:30:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 -- 172.21.15.7:0/3600414322 shutdown_connections 2023-03-03T14:30:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:51.109+0000 7f39929f6700 1 -- 172.21.15.7:0/3600414322 wait complete. 2023-03-03T14:30:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:30:51.123 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before thrashing 2023-03-03T14:30:53.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:53.912 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:30:57.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:30:57.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.261+0000 7f69dceed700 1 Processor -- start 2023-03-03T14:30:57.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.262+0000 7f69dceed700 1 -- start start 2023-03-03T14:30:57.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.262+0000 7f69dceed700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69d806bea0 0x7f69d806c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:57.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.262+0000 7f69dceed700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f69d8157dc0 0x7f69d8158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:57.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69dceed700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69d806d780 con 0x7f69d8151ab0 2023-03-03T14:30:57.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69dceed700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d806c830 con 0x7f69d806bea0 2023-03-03T14:30:57.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69dceed700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d806cb70 con 0x7f69d8157dc0 2023-03-03T14:30:57.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d5d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69d806bea0 0x7f69d806c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:57.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d5d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69d806bea0 0x7f69d806c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49074/0 (socket says 172.21.15.7:49074) 2023-03-03T14:30:57.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d5d9b700 1 -- 172.21.15.7:0/2524197933 learned_addr learned my addr 172.21.15.7:0/2524197933 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:30:57.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d5d9b700 1 -- 172.21.15.7:0/2524197933 >> v2:172.21.15.143:3300/0 conn(0x7f69d8157dc0 msgr2=0x7f69d8158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:30:57.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d5d9b700 1 --2- 172.21.15.7:0/2524197933 >> v2:172.21.15.143:3300/0 conn(0x7f69d8157dc0 0x7f69d8158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:57.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d5d9b700 1 -- 172.21.15.7:0/2524197933 >> v1:172.21.15.143:6789/0 conn(0x7f69d8151ab0 legacy=0x7f69d80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:30:57.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d5d9b700 1 -- 172.21.15.7:0/2524197933 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d806ce50 con 0x7f69d806bea0 2023-03-03T14:30:57.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d5d9b700 1 --2- 172.21.15.7:0/2524197933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d806bea0 0x7f69d806c2f0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f69c00098f0 tx=0x7f69c0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb578e4fd525d6aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:30:57.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d559a700 1 -- 172.21.15.7:0/2524197933 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c0004230 con 0x7f69d806bea0 2023-03-03T14:30:57.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d559a700 1 -- 172.21.15.7:0/2524197933 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c0004390 con 0x7f69d806bea0 2023-03-03T14:30:57.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.263+0000 7f69d559a700 1 -- 172.21.15.7:0/2524197933 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c00036a0 con 0x7f69d806bea0 2023-03-03T14:30:57.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.264+0000 7f69dceed700 1 -- 172.21.15.7:0/2524197933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d806bea0 msgr2=0x7f69d806c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:57.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.264+0000 7f69dceed700 1 --2- 172.21.15.7:0/2524197933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d806bea0 0x7f69d806c2f0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f69c00098f0 tx=0x7f69c0009c00 comp rx=0 tx=0).stop 2023-03-03T14:30:57.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.264+0000 7f69dceed700 1 -- 172.21.15.7:0/2524197933 shutdown_connections 2023-03-03T14:30:57.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.264+0000 7f69dceed700 1 --2- 172.21.15.7:0/2524197933 >> v2:172.21.15.143:3300/0 conn(0x7f69d8157dc0 0x7f69d8158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:57.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.264+0000 7f69dceed700 1 --2- 172.21.15.7:0/2524197933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d806bea0 0x7f69d806c2f0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:57.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.264+0000 7f69dceed700 1 -- 172.21.15.7:0/2524197933 >> 172.21.15.7:0/2524197933 conn(0x7f69d80a94d0 msgr2=0x7f69d80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:30:57.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.264+0000 7f69dceed700 1 -- 172.21.15.7:0/2524197933 shutdown_connections 2023-03-03T14:30:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.264+0000 7f69dceed700 1 -- 172.21.15.7:0/2524197933 wait complete. 2023-03-03T14:30:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.265+0000 7f69dceed700 1 Processor -- start 2023-03-03T14:30:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.265+0000 7f69dceed700 1 -- start start 2023-03-03T14:30:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69dceed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69d806bea0 0x7f69d8146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69dceed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d8151ab0 0x7f69d80ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69dceed700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69d8157dc0 0x7f69d814be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69dceed700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d81501e0 con 0x7f69d8151ab0 2023-03-03T14:30:57.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69dceed700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d80aadc0 con 0x7f69d806bea0 2023-03-03T14:30:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69dceed700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d80ab080 con 0x7f69d8157dc0 2023-03-03T14:30:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d8151ab0 0x7f69d80ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d8151ab0 0x7f69d80ac680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49080/0 (socket says 172.21.15.7:49080) 2023-03-03T14:30:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d5d9b700 1 -- 172.21.15.7:0/245719508 learned_addr learned my addr 172.21.15.7:0/245719508 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:30:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d659c700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69d806bea0 0x7f69d8146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d6d9d700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69d8157dc0 0x7f69d814be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d5d9b700 1 -- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69d8157dc0 msgr2=0x7f69d814be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d5d9b700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69d8157dc0 0x7f69d814be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:57.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d5d9b700 1 -- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69d806bea0 msgr2=0x7f69d8146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:57.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d5d9b700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69d806bea0 0x7f69d8146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:57.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d5d9b700 1 -- 172.21.15.7:0/245719508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c0009550 con 0x7f69d8151ab0 2023-03-03T14:30:57.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.266+0000 7f69d659c700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69d806bea0 0x7f69d8146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:30:57.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.267+0000 7f69d5d9b700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d8151ab0 0x7f69d80ac680 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f69c0000c00 tx=0x7f69c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:30:57.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.267+0000 7f69ceffd700 1 -- 172.21.15.7:0/245719508 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c0011630 con 0x7f69d8151ab0 2023-03-03T14:30:57.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.267+0000 7f69ceffd700 1 -- 172.21.15.7:0/245719508 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c001f070 con 0x7f69d8151ab0 2023-03-03T14:30:57.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.267+0000 7f69dceed700 1 -- 172.21.15.7:0/245719508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d806d410 con 0x7f69d8151ab0 2023-03-03T14:30:57.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.267+0000 7f69ceffd700 1 -- 172.21.15.7:0/245719508 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c00119d0 con 0x7f69d8151ab0 2023-03-03T14:30:57.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.267+0000 7f69dceed700 1 -- 172.21.15.7:0/245719508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d806d130 con 0x7f69d8151ab0 2023-03-03T14:30:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.268+0000 7f69dceed700 1 -- 172.21.15.7:0/245719508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d8073e00 con 0x7f69d8151ab0 2023-03-03T14:30:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.268+0000 7f69ceffd700 1 -- 172.21.15.7:0/245719508 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f69c002a020 con 0x7f69d8151ab0 2023-03-03T14:30:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.269+0000 7f69ceffd700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69b8068320 0x7f69b806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:30:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.269+0000 7f69ceffd700 1 -- 172.21.15.7:0/245719508 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f69c0087a60 con 0x7f69d8151ab0 2023-03-03T14:30:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.269+0000 7f69d659c700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69b8068320 0x7f69b806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:30:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.270+0000 7f69d659c700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69b8068320 0x7f69b806a7d0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f69c80017d0 tx=0x7f69c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:30:57.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.273+0000 7f69ceffd700 1 -- 172.21.15.7:0/245719508 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f69c0053e00 con 0x7f69d8151ab0 2023-03-03T14:30:57.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.511+0000 7f69dceed700 1 -- 172.21.15.7:0/245719508 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69d8073e00 con 0x7f69d8151ab0 2023-03-03T14:30:57.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.511+0000 7f69ceffd700 1 -- 172.21.15.7:0/245719508 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f69c0010380 con 0x7f69d8151ab0 2023-03-03T14:30:57.513 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:30:57.513 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:30:57.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 -- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69b8068320 msgr2=0x7f69b806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:57.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69b8068320 0x7f69b806a7d0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f69c80017d0 tx=0x7f69c8009040 comp rx=0 tx=0).stop 2023-03-03T14:30:57.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 -- 172.21.15.7:0/245719508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d8151ab0 msgr2=0x7f69d80ac680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:30:57.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d8151ab0 0x7f69d80ac680 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f69c0000c00 tx=0x7f69c0004750 comp rx=0 tx=0).stop 2023-03-03T14:30:57.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 -- 172.21.15.7:0/245719508 shutdown_connections 2023-03-03T14:30:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69d8157dc0 0x7f69d814be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69d806bea0 0x7f69d8146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69b8068320 0x7f69b806a7d0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 --2- 172.21.15.7:0/245719508 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69d8151ab0 0x7f69d80ac680 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:30:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 -- 172.21.15.7:0/245719508 >> 172.21.15.7:0/245719508 conn(0x7f69d80a94d0 msgr2=0x7f69d80736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:30:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.513+0000 7f69dceed700 1 -- 172.21.15.7:0/245719508 shutdown_connections 2023-03-03T14:30:57.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:30:57.514+0000 7f69dceed700 1 -- 172.21.15.7:0/245719508 wait complete. 2023-03-03T14:30:57.517 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:30:57.527 INFO:tasks.mds_thrash.fs.[a]:waiting for 74 secs before thrashing 2023-03-03T14:31:24.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:24.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:30.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:31:30.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.634+0000 7fddeff31700 1 Processor -- start 2023-03-03T14:31:30.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.634+0000 7fddeff31700 1 -- start start 2023-03-03T14:31:30.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.635+0000 7fddeff31700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdde80aac00 0x7fdde80c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:30.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.635+0000 7fddeff31700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdde8155950 0x7fdde8155d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:30.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.635+0000 7fddeff31700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde80713c0 con 0x7fdde8071c30 2023-03-03T14:31:30.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.635+0000 7fddeff31700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde81562d0 con 0x7fdde8155950 2023-03-03T14:31:30.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.635+0000 7fddeff31700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde8152c70 con 0x7fdde80aac00 2023-03-03T14:31:30.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.635+0000 7fddee4ce700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdde8155950 0x7fdde8155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:30.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.635+0000 7fdded4cc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdde8071c30 0x7fdde80beb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43088/0 (socket says 172.21.15.7:43088) 2023-03-03T14:31:30.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.635+0000 7fddee4ce700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdde8155950 0x7fdde8155d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49112/0 (socket says 172.21.15.7:49112) 2023-03-03T14:31:30.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.635+0000 7fdded4cc700 1 -- 172.21.15.7:0/1928030869 learned_addr learned my addr 172.21.15.7:0/1928030869 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:31:30.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.635+0000 7fddedccd700 1 --2- 172.21.15.7:0/1928030869 >> v2:172.21.15.143:3300/0 conn(0x7fdde80aac00 0x7fdde80c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:30.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddee4ce700 1 -- 172.21.15.7:0/1928030869 >> v2:172.21.15.143:3300/0 conn(0x7fdde80aac00 msgr2=0x7fdde80c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:31:30.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddee4ce700 1 --2- 172.21.15.7:0/1928030869 >> v2:172.21.15.143:3300/0 conn(0x7fdde80aac00 0x7fdde80c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:30.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddee4ce700 1 -- 172.21.15.7:0/1928030869 >> v1:172.21.15.7:6789/0 conn(0x7fdde8071c30 legacy=0x7fdde80beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:30.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddecccb700 1 -- 172.21.15.7:0/1928030869 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312479145 0 0) 0x7fdde80713c0 con 0x7fdde8071c30 2023-03-03T14:31:30.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddee4ce700 1 -- 172.21.15.7:0/1928030869 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde80c4930 con 0x7fdde8155950 2023-03-03T14:31:30.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddee4ce700 1 --2- 172.21.15.7:0/1928030869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde8155950 0x7fdde8155d90 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fdde80c9630 tx=0x7fdde000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440296537bad7aff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:31:30.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddecccb700 1 -- 172.21.15.7:0/1928030869 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde000e070 con 0x7fdde8155950 2023-03-03T14:31:30.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddecccb700 1 -- 172.21.15.7:0/1928030869 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde000bb20 con 0x7fdde8155950 2023-03-03T14:31:30.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddecccb700 1 -- 172.21.15.7:0/1928030869 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde0012430 con 0x7fdde8155950 2023-03-03T14:31:30.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddeff31700 1 -- 172.21.15.7:0/1928030869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde8155950 msgr2=0x7fdde8155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:30.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.636+0000 7fddeff31700 1 --2- 172.21.15.7:0/1928030869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde8155950 0x7fdde8155d90 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fdde80c9630 tx=0x7fdde000b330 comp rx=0 tx=0).stop 2023-03-03T14:31:30.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.637+0000 7fddeff31700 1 -- 172.21.15.7:0/1928030869 shutdown_connections 2023-03-03T14:31:30.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.637+0000 7fddeff31700 1 --2- 172.21.15.7:0/1928030869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde8155950 0x7fdde8155d90 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:30.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.637+0000 7fddeff31700 1 --2- 172.21.15.7:0/1928030869 >> v2:172.21.15.143:3300/0 conn(0x7fdde80aac00 0x7fdde80c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.637+0000 7fddeff31700 1 -- 172.21.15.7:0/1928030869 >> 172.21.15.7:0/1928030869 conn(0x7fdde80736f0 msgr2=0x7fdde8073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:31:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.637+0000 7fddeff31700 1 -- 172.21.15.7:0/1928030869 shutdown_connections 2023-03-03T14:31:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.637+0000 7fddeff31700 1 -- 172.21.15.7:0/1928030869 wait complete. 2023-03-03T14:31:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.638+0000 7fddeff31700 1 Processor -- start 2023-03-03T14:31:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.638+0000 7fddeff31700 1 -- start start 2023-03-03T14:31:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.638+0000 7fddeff31700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde8071c30 0x7fdde80c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.638+0000 7fddeff31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde80aac00 0x7fdde8147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.638+0000 7fddeff31700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde8155950 0x7fdde81480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.638+0000 7fddeff31700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde80c1eb0 con 0x7fdde80aac00 2023-03-03T14:31:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.638+0000 7fddeff31700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde80c75e0 con 0x7fdde8071c30 2023-03-03T14:31:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.638+0000 7fddeff31700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde80c29f0 con 0x7fdde8155950 2023-03-03T14:31:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fdded4cc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde80aac00 0x7fdde8147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fdded4cc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde80aac00 0x7fdde8147ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49116/0 (socket says 172.21.15.7:49116) 2023-03-03T14:31:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fdded4cc700 1 -- 172.21.15.7:0/3042257794 learned_addr learned my addr 172.21.15.7:0/3042257794 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:31:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fddedccd700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde8071c30 0x7fdde80c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:30.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fddee4ce700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde8155950 0x7fdde81480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fdded4cc700 1 -- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde8155950 msgr2=0x7fdde81480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fdded4cc700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde8155950 0x7fdde81480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fdded4cc700 1 -- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde8071c30 msgr2=0x7fdde80c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fdded4cc700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde8071c30 0x7fdde80c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fdded4cc700 1 -- 172.21.15.7:0/3042257794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde0009a70 con 0x7fdde80aac00 2023-03-03T14:31:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fddee4ce700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde8155950 0x7fdde81480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:31:30.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fdded4cc700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde80aac00 0x7fdde8147ba0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fdde400c750 tx=0x7fdde400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:31:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fddde7fc700 1 -- 172.21.15.7:0/3042257794 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4004c80 con 0x7fdde80aac00 2023-03-03T14:31:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fddde7fc700 1 -- 172.21.15.7:0/3042257794 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde4004e00 con 0x7fdde80aac00 2023-03-03T14:31:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fddeff31700 1 -- 172.21.15.7:0/3042257794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde80c4b80 con 0x7fdde80aac00 2023-03-03T14:31:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fddde7fc700 1 -- 172.21.15.7:0/3042257794 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4007310 con 0x7fdde80aac00 2023-03-03T14:31:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.639+0000 7fddeff31700 1 -- 172.21.15.7:0/3042257794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde8152360 con 0x7fdde80aac00 2023-03-03T14:31:30.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.640+0000 7fddeff31700 1 -- 172.21.15.7:0/3042257794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde8152620 con 0x7fdde80aac00 2023-03-03T14:31:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.641+0000 7fddde7fc700 1 -- 172.21.15.7:0/3042257794 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdde401a020 con 0x7fdde80aac00 2023-03-03T14:31:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.641+0000 7fddde7fc700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddcc0682f0 0x7fddcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:31:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.641+0000 7fddde7fc700 1 -- 172.21.15.7:0/3042257794 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdde4085cd0 con 0x7fdde80aac00 2023-03-03T14:31:30.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.641+0000 7fddedccd700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddcc0682f0 0x7fddcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:31:30.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.642+0000 7fddedccd700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddcc0682f0 0x7fddcc06a7a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fdde80c95a0 tx=0x7fddd8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:31:30.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.646+0000 7fddde7fc700 1 -- 172.21.15.7:0/3042257794 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdde4052070 con 0x7fdde80aac00 2023-03-03T14:31:30.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.883+0000 7fddeff31700 1 -- 172.21.15.7:0/3042257794 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdde8074bb0 con 0x7fdde80aac00 2023-03-03T14:31:30.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.884+0000 7fddde7fc700 1 -- 172.21.15.7:0/3042257794 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdde4055690 con 0x7fdde80aac00 2023-03-03T14:31:30.885 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:31:30.885 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:31:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 -- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddcc0682f0 msgr2=0x7fddcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddcc0682f0 0x7fddcc06a7a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fdde80c95a0 tx=0x7fddd8009210 comp rx=0 tx=0).stop 2023-03-03T14:31:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 -- 172.21.15.7:0/3042257794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde80aac00 msgr2=0x7fdde8147ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:31:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde80aac00 0x7fdde8147ba0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fdde400c750 tx=0x7fdde400ca60 comp rx=0 tx=0).stop 2023-03-03T14:31:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 -- 172.21.15.7:0/3042257794 shutdown_connections 2023-03-03T14:31:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde8155950 0x7fdde81480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde8071c30 0x7fdde80c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddcc0682f0 0x7fddcc06a7a0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 --2- 172.21.15.7:0/3042257794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde80aac00 0x7fdde8147ba0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:31:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 -- 172.21.15.7:0/3042257794 >> 172.21.15.7:0/3042257794 conn(0x7fdde80736f0 msgr2=0x7fdde8153100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:31:30.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 -- 172.21.15.7:0/3042257794 shutdown_connections 2023-03-03T14:31:30.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:31:30.886+0000 7fddeff31700 1 -- 172.21.15.7:0/3042257794 wait complete. 2023-03-03T14:31:30.889 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:31:30.899 INFO:tasks.mds_thrash.fs.[b]:waiting for 43 secs before thrashing 2023-03-03T14:31:54.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:31:54.155 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:11.528 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:32:11.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.665+0000 7effd3330700 1 Processor -- start 2023-03-03T14:32:11.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.665+0000 7effd3330700 1 -- start start 2023-03-03T14:32:11.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effd3330700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effcc0c00e0 0x7effcc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:11.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effd3330700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc0c7310 con 0x7effcc0caa30 2023-03-03T14:32:11.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effd3330700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc0bfd10 con 0x7effcc154ff0 2023-03-03T14:32:11.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effd3330700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0bfa30 con 0x7effcc0c00e0 2023-03-03T14:32:11.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effd10cc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7effcc0caa30 0x7effcc0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43146/0 (socket says 172.21.15.7:43146) 2023-03-03T14:32:11.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effd10cc700 1 -- 172.21.15.7:0/2755032810 learned_addr learned my addr 172.21.15.7:0/2755032810 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:32:11.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effd08cb700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effcc0c00e0 0x7effcc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:11.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526445633 0 0) 0x7effcc0c7310 con 0x7effcc0caa30 2023-03-03T14:32:11.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb80035b0 con 0x7effcc0caa30 2023-03-03T14:32:11.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1866507065 0 0) 0x7effb80035b0 con 0x7effcc0caa30 2023-03-03T14:32:11.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effcc0c00e0 msgr2=0x7effcc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:32:11.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effc3fff700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effcc0c00e0 0x7effcc0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:11.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.666+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 >> v1:172.21.15.143:6789/0 conn(0x7effcc154ff0 legacy=0x7effcc15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc154510 con 0x7effcc0caa30 2023-03-03T14:32:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7effbc001670 con 0x7effcc0caa30 2023-03-03T14:32:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 >> v1:172.21.15.7:6789/0 conn(0x7effcc0caa30 legacy=0x7effcc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effb8003aa0 0x7effb8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8004850 0x7effb8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effcc0c00e0 0x7effb8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:11.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0bfa30 con 0x7effb8004850 2023-03-03T14:32:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb800baf0 con 0x7effb8003aa0 2023-03-03T14:32:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb800bcb0 con 0x7effcc0c00e0 2023-03-03T14:32:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effd08cb700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8004850 0x7effb8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effd10cc700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effb8003aa0 0x7effb8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effd18cd700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effcc0c00e0 0x7effb8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effd08cb700 1 -- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effcc0c00e0 msgr2=0x7effb8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:32:11.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effd08cb700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effcc0c00e0 0x7effb8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effd08cb700 1 -- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effb8003aa0 msgr2=0x7effb8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:32:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effd08cb700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effb8003aa0 0x7effb8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effd08cb700 1 -- 172.21.15.7:0/2755032810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb80035b0 con 0x7effb8004850 2023-03-03T14:32:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effd08cb700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8004850 0x7effb8004cc0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7effc8009b40 tx=0x7effb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead1d0083e1dfd50 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:32:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8005a20 con 0x7effb8004850 2023-03-03T14:32:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc8005b80 con 0x7effb8004850 2023-03-03T14:32:11.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.667+0000 7effc3fff700 1 -- 172.21.15.7:0/2755032810 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc800e6c0 con 0x7effb8004850 2023-03-03T14:32:11.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.668+0000 7effd3330700 1 -- 172.21.15.7:0/2755032810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8004850 msgr2=0x7effb8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:11.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.668+0000 7effd3330700 1 --2- 172.21.15.7:0/2755032810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb8004850 0x7effb8004cc0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7effc8009b40 tx=0x7effb8005880 comp rx=0 tx=0).stop 2023-03-03T14:32:11.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.668+0000 7effd18cd700 1 -- 172.21.15.7:0/2755032810 reap_dead start 2023-03-03T14:32:11.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.668+0000 7effd3330700 1 -- 172.21.15.7:0/2755032810 shutdown_connections 2023-03-03T14:32:11.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.668+0000 7effd3330700 1 -- 172.21.15.7:0/2755032810 >> 172.21.15.7:0/2755032810 conn(0x7effcc066e40 msgr2=0x7effcc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:32:11.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.668+0000 7effd3330700 1 -- 172.21.15.7:0/2755032810 shutdown_connections 2023-03-03T14:32:11.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.668+0000 7effd3330700 1 -- 172.21.15.7:0/2755032810 wait complete. 2023-03-03T14:32:11.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.669+0000 7effd3330700 1 Processor -- start 2023-03-03T14:32:11.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.669+0000 7effd3330700 1 -- start start 2023-03-03T14:32:11.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd3330700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effb8004850 0x7effcc1501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:11.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd3330700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effcc0c3440 0x7effcc1e6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:11.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd3330700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effcc155410 0x7effcc1e7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:11.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd3330700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc145e50 con 0x7effcc0c3440 2023-03-03T14:32:11.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd3330700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0c3bb0 con 0x7effcc155410 2023-03-03T14:32:11.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd3330700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0c38d0 con 0x7effb8004850 2023-03-03T14:32:11.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd08cb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effcc0c3440 0x7effcc1e6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:11.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd08cb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effcc0c3440 0x7effcc1e6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49176/0 (socket says 172.21.15.7:49176) 2023-03-03T14:32:11.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd08cb700 1 -- 172.21.15.7:0/4226790290 learned_addr learned my addr 172.21.15.7:0/4226790290 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:32:11.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd10cc700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effb8004850 0x7effcc1501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:11.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd18cd700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effcc155410 0x7effcc1e7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:11.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd08cb700 1 -- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effb8004850 msgr2=0x7effcc1501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:11.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd08cb700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effb8004850 0x7effcc1501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:11.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd08cb700 1 -- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effcc155410 msgr2=0x7effcc1e7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:11.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd08cb700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effcc155410 0x7effcc1e7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:11.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.670+0000 7effd08cb700 1 -- 172.21.15.7:0/4226790290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc154ad0 con 0x7effcc0c3440 2023-03-03T14:32:11.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.671+0000 7effd08cb700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effcc0c3440 0x7effcc1e6f20 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7effc8000c00 tx=0x7effc80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:32:11.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.671+0000 7effc1ffb700 1 -- 172.21.15.7:0/4226790290 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8021070 con 0x7effcc0c3440 2023-03-03T14:32:11.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.671+0000 7effc1ffb700 1 -- 172.21.15.7:0/4226790290 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc8004050 con 0x7effcc0c3440 2023-03-03T14:32:11.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.671+0000 7effd3330700 1 -- 172.21.15.7:0/4226790290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effcc1547f0 con 0x7effcc0c3440 2023-03-03T14:32:11.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.671+0000 7effc1ffb700 1 -- 172.21.15.7:0/4226790290 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8004400 con 0x7effcc0c3440 2023-03-03T14:32:11.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.671+0000 7effd3330700 1 -- 172.21.15.7:0/4226790290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effcc0bf650 con 0x7effcc0c3440 2023-03-03T14:32:11.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.672+0000 7effd3330700 1 -- 172.21.15.7:0/4226790290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effcc0c1570 con 0x7effcc0c3440 2023-03-03T14:32:11.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.673+0000 7effc1ffb700 1 -- 172.21.15.7:0/4226790290 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7effc802c030 con 0x7effcc0c3440 2023-03-03T14:32:11.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.673+0000 7effc1ffb700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb0068320 0x7effb006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:11.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.673+0000 7effc1ffb700 1 -- 172.21.15.7:0/4226790290 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7effc8086ff0 con 0x7effcc0c3440 2023-03-03T14:32:11.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.673+0000 7effd10cc700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb0068320 0x7effb006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:11.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.674+0000 7effd10cc700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb0068320 0x7effb006a7d0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7effbc001eb0 tx=0x7effbc000f80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:32:11.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.677+0000 7effc1ffb700 1 -- 172.21.15.7:0/4226790290 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7effc8053390 con 0x7effcc0c3440 2023-03-03T14:32:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.915+0000 7effd3330700 1 -- 172.21.15.7:0/4226790290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effcc074bb0 con 0x7effcc0c3440 2023-03-03T14:32:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.916+0000 7effc1ffb700 1 -- 172.21.15.7:0/4226790290 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7effc80569b0 con 0x7effcc0c3440 2023-03-03T14:32:11.917 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:32:11.918 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:32:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 -- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb0068320 msgr2=0x7effb006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb0068320 0x7effb006a7d0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7effbc001eb0 tx=0x7effbc000f80 comp rx=0 tx=0).stop 2023-03-03T14:32:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 -- 172.21.15.7:0/4226790290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effcc0c3440 msgr2=0x7effcc1e6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effcc0c3440 0x7effcc1e6f20 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7effc8000c00 tx=0x7effc80047c0 comp rx=0 tx=0).stop 2023-03-03T14:32:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 -- 172.21.15.7:0/4226790290 shutdown_connections 2023-03-03T14:32:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effcc155410 0x7effcc1e7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effb8004850 0x7effcc1501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb0068320 0x7effb006a7d0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 --2- 172.21.15.7:0/4226790290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effcc0c3440 0x7effcc1e6f20 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 -- 172.21.15.7:0/4226790290 >> 172.21.15.7:0/4226790290 conn(0x7effcc066e40 msgr2=0x7effcc0c8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:32:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 -- 172.21.15.7:0/4226790290 shutdown_connections 2023-03-03T14:32:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:11.918+0000 7effd3330700 1 -- 172.21.15.7:0/4226790290 wait complete. 2023-03-03T14:32:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:32:11.931 INFO:tasks.mds_thrash.fs.[a]:waiting for 85 secs before thrashing 2023-03-03T14:32:13.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:32:14.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.038+0000 7f924c4af700 1 Processor -- start 2023-03-03T14:32:14.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.038+0000 7f924c4af700 1 -- start start 2023-03-03T14:32:14.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924c4af700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9244155390 0x7f92440c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:14.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924c4af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 0x7f9244071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:14.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924c4af700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9244072520 0x7f9244151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:14.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924c4af700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9244070b90 con 0x7f9244155390 2023-03-03T14:32:14.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924c4af700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9244151320 con 0x7f9244072520 2023-03-03T14:32:14.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924c4af700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92440c4780 con 0x7f924406e020 2023-03-03T14:32:14.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924a24b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9244155390 0x7f92440c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:14.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924a24b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9244155390 0x7f92440c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49190/0 (socket says 172.21.15.7:49190) 2023-03-03T14:32:14.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924a24b700 1 -- 172.21.15.7:0/3341910037 learned_addr learned my addr 172.21.15.7:0/3341910037 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:32:14.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f9249a4a700 1 --2- 172.21.15.7:0/3341910037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 0x7f9244071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:14.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924aa4c700 1 --2- 172.21.15.7:0/3341910037 >> v2:172.21.15.143:3300/0 conn(0x7f9244072520 0x7f9244151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:14.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.039+0000 7f924a24b700 1 -- 172.21.15.7:0/3341910037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 msgr2=0x7f9244071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:32:14.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f924a24b700 1 --2- 172.21.15.7:0/3341910037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 0x7f9244071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f924a24b700 1 -- 172.21.15.7:0/3341910037 >> v2:172.21.15.143:3300/0 conn(0x7f9244072520 msgr2=0x7f9244151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:32:14.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f924a24b700 1 --2- 172.21.15.7:0/3341910037 >> v2:172.21.15.143:3300/0 conn(0x7f9244072520 0x7f9244151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f924a24b700 1 -- 172.21.15.7:0/3341910037 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9244154910 con 0x7f9244155390 2023-03-03T14:32:14.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f9249a4a700 1 --2- 172.21.15.7:0/3341910037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 0x7f9244071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:32:14.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f924aa4c700 1 --2- 172.21.15.7:0/3341910037 >> v2:172.21.15.143:3300/0 conn(0x7f9244072520 0x7f9244151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:32:14.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f924a24b700 1 --2- 172.21.15.7:0/3341910037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 0x7f92440c3e30 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f92440c3d80 tx=0x7f924000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9f6cb498b92f22 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:32:14.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f9249249700 1 -- 172.21.15.7:0/3341910037 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924000dd10 con 0x7f9244155390 2023-03-03T14:32:14.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f9249249700 1 -- 172.21.15.7:0/3341910037 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924000de70 con 0x7f9244155390 2023-03-03T14:32:14.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f9249249700 1 -- 172.21.15.7:0/3341910037 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240010670 con 0x7f9244155390 2023-03-03T14:32:14.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f924c4af700 1 -- 172.21.15.7:0/3341910037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 msgr2=0x7f92440c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:14.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.040+0000 7f924c4af700 1 --2- 172.21.15.7:0/3341910037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 0x7f92440c3e30 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f92440c3d80 tx=0x7f924000bea0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.041+0000 7f924c4af700 1 -- 172.21.15.7:0/3341910037 shutdown_connections 2023-03-03T14:32:14.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.041+0000 7f924c4af700 1 --2- 172.21.15.7:0/3341910037 >> v2:172.21.15.143:3300/0 conn(0x7f9244072520 0x7f9244151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.041+0000 7f924c4af700 1 --2- 172.21.15.7:0/3341910037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 0x7f92440c3e30 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.041+0000 7f924c4af700 1 --2- 172.21.15.7:0/3341910037 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 0x7f9244071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.041+0000 7f924c4af700 1 -- 172.21.15.7:0/3341910037 >> 172.21.15.7:0/3341910037 conn(0x7f924406c110 msgr2=0x7f924406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:32:14.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.041+0000 7f924c4af700 1 -- 172.21.15.7:0/3341910037 shutdown_connections 2023-03-03T14:32:14.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.041+0000 7f924c4af700 1 -- 172.21.15.7:0/3341910037 wait complete. 2023-03-03T14:32:14.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.042+0000 7f924c4af700 1 Processor -- start 2023-03-03T14:32:14.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.042+0000 7f924c4af700 1 -- start start 2023-03-03T14:32:14.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.042+0000 7f924c4af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 0x7f92440c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:14.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.042+0000 7f924c4af700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9244072520 0x7f92440c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:14.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.042+0000 7f924c4af700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 0x7f924414d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:14.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.042+0000 7f924c4af700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92440c6840 con 0x7f9244155390 2023-03-03T14:32:14.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.042+0000 7f924c4af700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92440c6ab0 con 0x7f9244072520 2023-03-03T14:32:14.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924c4af700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92440c6d90 con 0x7f924406e020 2023-03-03T14:32:14.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924aa4c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 0x7f924414d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:14.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924aa4c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 0x7f924414d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49200/0 (socket says 172.21.15.7:49200) 2023-03-03T14:32:14.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924aa4c700 1 -- 172.21.15.7:0/1975943911 learned_addr learned my addr 172.21.15.7:0/1975943911 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:32:14.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924a24b700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 0x7f92440c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f9249a4a700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9244072520 0x7f92440c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924aa4c700 1 -- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 msgr2=0x7f92440c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924aa4c700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 0x7f92440c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924aa4c700 1 -- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9244072520 msgr2=0x7f92440c0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924aa4c700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9244072520 0x7f92440c0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924aa4c700 1 -- 172.21.15.7:0/1975943911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924000b7f0 con 0x7f9244155390 2023-03-03T14:32:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f9249a4a700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9244072520 0x7f92440c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:32:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924aa4c700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 0x7f924414d8e0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f923c00cef0 tx=0x7f923c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:32:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f923affd700 1 -- 172.21.15.7:0/1975943911 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c00ae50 con 0x7f9244155390 2023-03-03T14:32:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f923affd700 1 -- 172.21.15.7:0/1975943911 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923c015440 con 0x7f9244155390 2023-03-03T14:32:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924c4af700 1 -- 172.21.15.7:0/1975943911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9244070540 con 0x7f9244155390 2023-03-03T14:32:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f923affd700 1 -- 172.21.15.7:0/1975943911 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c0157f0 con 0x7f9244155390 2023-03-03T14:32:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.043+0000 7f924c4af700 1 -- 172.21.15.7:0/1975943911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92440702a0 con 0x7f9244155390 2023-03-03T14:32:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.044+0000 7f924c4af700 1 -- 172.21.15.7:0/1975943911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9244070820 con 0x7f9244155390 2023-03-03T14:32:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.045+0000 7f923affd700 1 -- 172.21.15.7:0/1975943911 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f923c01c030 con 0x7f9244155390 2023-03-03T14:32:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.045+0000 7f923affd700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92280682f0 0x7f922806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:32:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.045+0000 7f923affd700 1 -- 172.21.15.7:0/1975943911 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f923c086640 con 0x7f9244155390 2023-03-03T14:32:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.045+0000 7f924a24b700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92280682f0 0x7f922806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:32:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.046+0000 7f924a24b700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92280682f0 0x7f922806a7a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f924000bb10 tx=0x7f9240013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:32:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.050+0000 7f923affd700 1 -- 172.21.15.7:0/1975943911 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f923c052980 con 0x7f9244155390 2023-03-03T14:32:14.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.288+0000 7f924c4af700 1 -- 172.21.15.7:0/1975943911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f924414f0b0 con 0x7f9244155390 2023-03-03T14:32:14.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.288+0000 7f923affd700 1 -- 172.21.15.7:0/1975943911 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f923c055fa0 con 0x7f9244155390 2023-03-03T14:32:14.290 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:32:14.290 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:32:14.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 -- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92280682f0 msgr2=0x7f922806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:14.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92280682f0 0x7f922806a7a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f924000bb10 tx=0x7f9240013000 comp rx=0 tx=0).stop 2023-03-03T14:32:14.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 -- 172.21.15.7:0/1975943911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 msgr2=0x7f924414d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:32:14.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 0x7f924414d8e0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f923c00cef0 tx=0x7f923c00a340 comp rx=0 tx=0).stop 2023-03-03T14:32:14.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 -- 172.21.15.7:0/1975943911 shutdown_connections 2023-03-03T14:32:14.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92280682f0 0x7f922806a7a0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9244155390 0x7f924414d8e0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f924406e020 0x7f92440c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 --2- 172.21.15.7:0/1975943911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9244072520 0x7f92440c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:32:14.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 -- 172.21.15.7:0/1975943911 >> 172.21.15.7:0/1975943911 conn(0x7f924406c110 msgr2=0x7f9244149a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:32:14.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.290+0000 7f924c4af700 1 -- 172.21.15.7:0/1975943911 shutdown_connections 2023-03-03T14:32:14.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:32:14.291+0000 7f924c4af700 1 -- 172.21.15.7:0/1975943911 wait complete. 2023-03-03T14:32:14.294 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:32:14.304 INFO:tasks.mds_thrash.fs.[b]:waiting for 56 secs before thrashing 2023-03-03T14:32:24.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:24.309 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:54.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:32:54.457 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:10.305 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:33:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.442+0000 7f7f616a2700 1 Processor -- start 2023-03-03T14:33:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.442+0000 7f7f616a2700 1 -- start start 2023-03-03T14:33:10.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.443+0000 7f7f616a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c151b20 0x7f7f5c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.443+0000 7f7f616a2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c06cb70 con 0x7f7f5c0c2950 2023-03-03T14:33:10.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.443+0000 7f7f616a2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c06ce50 con 0x7f7f5c073c30 2023-03-03T14:33:10.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.443+0000 7f7f616a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c06c2f0 con 0x7f7f5c151b20 2023-03-03T14:33:10.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.443+0000 7f7f5a7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7f5c0c2950 0x7f7f5c157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43232/0 (socket says 172.21.15.7:43232) 2023-03-03T14:33:10.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.443+0000 7f7f5a7fc700 1 -- 172.21.15.7:0/2082375885 learned_addr learned my addr 172.21.15.7:0/2082375885 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:33:10.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f5affd700 1 --2- 172.21.15.7:0/2082375885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c151b20 0x7f7f5c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f59ffb700 1 -- 172.21.15.7:0/2082375885 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169477751 0 0) 0x7f7f5c06cb70 con 0x7f7f5c0c2950 2023-03-03T14:33:10.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f59ffb700 1 -- 172.21.15.7:0/2082375885 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f480035b0 con 0x7f7f5c0c2950 2023-03-03T14:33:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f59ffb700 1 -- 172.21.15.7:0/2082375885 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581011900 0 0) 0x7f7f5c06ce50 con 0x7f7f5c073c30 2023-03-03T14:33:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f59ffb700 1 -- 172.21.15.7:0/2082375885 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f5c06cb70 con 0x7f7f5c073c30 2023-03-03T14:33:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f5affd700 1 -- 172.21.15.7:0/2082375885 >> v1:172.21.15.143:6789/0 conn(0x7f7f5c073c30 legacy=0x7f7f5c1581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f5affd700 1 -- 172.21.15.7:0/2082375885 >> v1:172.21.15.7:6789/0 conn(0x7f7f5c0c2950 legacy=0x7f7f5c157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f5affd700 1 -- 172.21.15.7:0/2082375885 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c06d130 con 0x7f7f5c151b20 2023-03-03T14:33:10.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f5affd700 1 --2- 172.21.15.7:0/2082375885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c151b20 0x7f7f5c0736f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009870 tx=0x7f7f4c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=407977c64b6839f7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:10.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f59ffb700 1 -- 172.21.15.7:0/2082375885 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c011830 con 0x7f7f5c151b20 2023-03-03T14:33:10.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f59ffb700 1 -- 172.21.15.7:0/2082375885 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c011990 con 0x7f7f5c151b20 2023-03-03T14:33:10.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.444+0000 7f7f59ffb700 1 -- 172.21.15.7:0/2082375885 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c011d00 con 0x7f7f5c151b20 2023-03-03T14:33:10.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.445+0000 7f7f616a2700 1 -- 172.21.15.7:0/2082375885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c151b20 msgr2=0x7f7f5c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.445+0000 7f7f616a2700 1 --2- 172.21.15.7:0/2082375885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c151b20 0x7f7f5c0736f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009870 tx=0x7f7f4c009b80 comp rx=0 tx=0).stop 2023-03-03T14:33:10.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.445+0000 7f7f616a2700 1 -- 172.21.15.7:0/2082375885 shutdown_connections 2023-03-03T14:33:10.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.445+0000 7f7f616a2700 1 --2- 172.21.15.7:0/2082375885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c151b20 0x7f7f5c0736f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.445+0000 7f7f616a2700 1 -- 172.21.15.7:0/2082375885 >> 172.21.15.7:0/2082375885 conn(0x7f7f5c0a94d0 msgr2=0x7f7f5c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:33:10.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.445+0000 7f7f616a2700 1 -- 172.21.15.7:0/2082375885 shutdown_connections 2023-03-03T14:33:10.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.445+0000 7f7f616a2700 1 -- 172.21.15.7:0/2082375885 wait complete. 2023-03-03T14:33:10.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.446+0000 7f7f616a2700 1 Processor -- start 2023-03-03T14:33:10.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.446+0000 7f7f616a2700 1 -- start start 2023-03-03T14:33:10.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f616a2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f5c073c30 0x7f7f5c1e0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f616a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c0c2950 0x7f7f5c14b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f616a2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c14c470 0x7f7f5c14bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f616a2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0721c0 con 0x7f7f5c073c30 2023-03-03T14:33:10.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f616a2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0c5370 con 0x7f7f5c14c470 2023-03-03T14:33:10.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f616a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0c5650 con 0x7f7f5c0c2950 2023-03-03T14:33:10.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f5affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f5c073c30 0x7f7f5c1e0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f5affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f5c073c30 0x7f7f5c1e0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49258/0 (socket says 172.21.15.7:49258) 2023-03-03T14:33:10.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f5affd700 1 -- 172.21.15.7:0/2159041020 learned_addr learned my addr 172.21.15.7:0/2159041020 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:33:10.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f5a7fc700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c0c2950 0x7f7f5c14b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f5b7fe700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c14c470 0x7f7f5c14bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f5affd700 1 -- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c0c2950 msgr2=0x7f7f5c14b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f5affd700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c0c2950 0x7f7f5c14b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f5affd700 1 -- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c14c470 msgr2=0x7f7f5c14bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.447+0000 7f7f5affd700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c14c470 0x7f7f5c14bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.448+0000 7f7f5affd700 1 -- 172.21.15.7:0/2159041020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c0094a0 con 0x7f7f5c073c30 2023-03-03T14:33:10.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.448+0000 7f7f5a7fc700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c0c2950 0x7f7f5c14b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:33:10.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.448+0000 7f7f5b7fe700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c14c470 0x7f7f5c14bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:33:10.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.448+0000 7f7f5affd700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f5c073c30 0x7f7f5c1e0fe0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0090b0 tx=0x7f7f4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:10.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.448+0000 7f7f537fe700 1 -- 172.21.15.7:0/2159041020 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c0045c0 con 0x7f7f5c073c30 2023-03-03T14:33:10.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.448+0000 7f7f537fe700 1 -- 172.21.15.7:0/2159041020 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c01f070 con 0x7f7f5c073c30 2023-03-03T14:33:10.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.448+0000 7f7f616a2700 1 -- 172.21.15.7:0/2159041020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c151ef0 con 0x7f7f5c073c30 2023-03-03T14:33:10.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.448+0000 7f7f537fe700 1 -- 172.21.15.7:0/2159041020 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c003900 con 0x7f7f5c073c30 2023-03-03T14:33:10.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.448+0000 7f7f616a2700 1 -- 172.21.15.7:0/2159041020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c06d410 con 0x7f7f5c073c30 2023-03-03T14:33:10.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.449+0000 7f7f616a2700 1 -- 172.21.15.7:0/2159041020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c06bea0 con 0x7f7f5c073c30 2023-03-03T14:33:10.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.449+0000 7f7f537fe700 1 -- 172.21.15.7:0/2159041020 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7f4c00f050 con 0x7f7f5c073c30 2023-03-03T14:33:10.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.450+0000 7f7f537fe700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f3c0682f0 0x7f7f3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:10.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.450+0000 7f7f537fe700 1 -- 172.21.15.7:0/2159041020 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7f4c021070 con 0x7f7f5c073c30 2023-03-03T14:33:10.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.450+0000 7f7f5a7fc700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f3c0682f0 0x7f7f3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:10.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.451+0000 7f7f5a7fc700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f3c0682f0 0x7f7f3c06a7a0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c14f7b0 tx=0x7f7f44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:10.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.454+0000 7f7f537fe700 1 -- 172.21.15.7:0/2159041020 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7f4c053b60 con 0x7f7f5c073c30 2023-03-03T14:33:10.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.691+0000 7f7f616a2700 1 -- 172.21.15.7:0/2159041020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f5c074bb0 con 0x7f7f5c073c30 2023-03-03T14:33:10.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.692+0000 7f7f537fe700 1 -- 172.21.15.7:0/2159041020 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7f4c006cc0 con 0x7f7f5c073c30 2023-03-03T14:33:10.694 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:33:10.694 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:33:10.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 -- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f3c0682f0 msgr2=0x7f7f3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f3c0682f0 0x7f7f3c06a7a0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c14f7b0 tx=0x7f7f44009040 comp rx=0 tx=0).stop 2023-03-03T14:33:10.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 -- 172.21.15.7:0/2159041020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f5c073c30 msgr2=0x7f7f5c1e0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:10.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f5c073c30 0x7f7f5c1e0fe0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0090b0 tx=0x7f7f4c004750 comp rx=0 tx=0).stop 2023-03-03T14:33:10.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 -- 172.21.15.7:0/2159041020 shutdown_connections 2023-03-03T14:33:10.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f5c14c470 0x7f7f5c14bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7f3c0682f0 0x7f7f3c06a7a0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f5c073c30 0x7f7f5c1e0fe0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 --2- 172.21.15.7:0/2159041020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f5c0c2950 0x7f7f5c14b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:10.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 -- 172.21.15.7:0/2159041020 >> 172.21.15.7:0/2159041020 conn(0x7f7f5c0a94d0 msgr2=0x7f7f5c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:33:10.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 -- 172.21.15.7:0/2159041020 shutdown_connections 2023-03-03T14:33:10.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:10.694+0000 7f7f616a2700 1 -- 172.21.15.7:0/2159041020 wait complete. 2023-03-03T14:33:10.697 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:33:10.707 INFO:tasks.mds_thrash.fs.[b]:waiting for 36 secs before thrashing 2023-03-03T14:33:24.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:24.603 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:36.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:33:37.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.069+0000 7fccb8698700 1 Processor -- start 2023-03-03T14:33:37.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.069+0000 7fccb8698700 1 -- start start 2023-03-03T14:33:37.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb8698700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fccb0155410 0x7fccb00bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb8698700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fccb00c5b00 0x7fccb00bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb8698700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb00c08d0 con 0x7fccb0151e80 2023-03-03T14:33:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb8698700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb0152250 con 0x7fccb00c5b00 2023-03-03T14:33:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb8698700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb00c0fb0 con 0x7fccb0155410 2023-03-03T14:33:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb6c35700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fccb00c5b00 0x7fccb00bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb6434700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fccb0151e80 0x7fccb00c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43274/0 (socket says 172.21.15.7:43274) 2023-03-03T14:33:37.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb6434700 1 -- 172.21.15.7:0/3716923125 learned_addr learned my addr 172.21.15.7:0/3716923125 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:33:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb6c35700 1 -- 172.21.15.7:0/3716923125 >> v2:172.21.15.143:3300/0 conn(0x7fccb0155410 msgr2=0x7fccb00bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:33:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb6c35700 1 --2- 172.21.15.7:0/3716923125 >> v2:172.21.15.143:3300/0 conn(0x7fccb0155410 0x7fccb00bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb6c35700 1 -- 172.21.15.7:0/3716923125 >> v1:172.21.15.7:6789/0 conn(0x7fccb0151e80 legacy=0x7fccb00c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb5432700 1 -- 172.21.15.7:0/3716923125 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731889908 0 0) 0x7fccb00c08d0 con 0x7fccb0151e80 2023-03-03T14:33:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb6c35700 1 -- 172.21.15.7:0/3716923125 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0154930 con 0x7fccb00c5b00 2023-03-03T14:33:37.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb6c35700 1 --2- 172.21.15.7:0/3716923125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb00c5b00 0x7fccb00bfe20 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fccb006c380 tx=0x7fcca800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83e952f2871ecf2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.070+0000 7fccb5432700 1 -- 172.21.15.7:0/3716923125 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca800e070 con 0x7fccb00c5b00 2023-03-03T14:33:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.071+0000 7fccb5432700 1 -- 172.21.15.7:0/3716923125 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca800bbd0 con 0x7fccb00c5b00 2023-03-03T14:33:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.071+0000 7fccb5432700 1 -- 172.21.15.7:0/3716923125 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8012430 con 0x7fccb00c5b00 2023-03-03T14:33:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.071+0000 7fccb8698700 1 -- 172.21.15.7:0/3716923125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb00c5b00 msgr2=0x7fccb00bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.071+0000 7fccb8698700 1 --2- 172.21.15.7:0/3716923125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb00c5b00 0x7fccb00bfe20 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fccb006c380 tx=0x7fcca800b330 comp rx=0 tx=0).stop 2023-03-03T14:33:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.071+0000 7fccb8698700 1 -- 172.21.15.7:0/3716923125 shutdown_connections 2023-03-03T14:33:37.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.071+0000 7fccb8698700 1 --2- 172.21.15.7:0/3716923125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb00c5b00 0x7fccb00bfe20 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.071+0000 7fccb8698700 1 --2- 172.21.15.7:0/3716923125 >> v2:172.21.15.143:3300/0 conn(0x7fccb0155410 0x7fccb00bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.071+0000 7fccb8698700 1 -- 172.21.15.7:0/3716923125 >> 172.21.15.7:0/3716923125 conn(0x7fccb0071630 msgr2=0x7fccb006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:33:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.071+0000 7fccb8698700 1 -- 172.21.15.7:0/3716923125 shutdown_connections 2023-03-03T14:33:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.071+0000 7fccb8698700 1 -- 172.21.15.7:0/3716923125 wait complete. 2023-03-03T14:33:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.072+0000 7fccb8698700 1 Processor -- start 2023-03-03T14:33:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.072+0000 7fccb8698700 1 -- start start 2023-03-03T14:33:37.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb8698700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccb00c5b00 0x7fccb0155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb8698700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb0151e80 0x7fccb00c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb8698700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccb0155410 0x7fccb01edd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb8698700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb01536d0 con 0x7fccb0151e80 2023-03-03T14:33:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb8698700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0148960 con 0x7fccb00c5b00 2023-03-03T14:33:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb8698700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb01ed880 con 0x7fccb0155410 2023-03-03T14:33:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb5c33700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb0151e80 0x7fccb00c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb5c33700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb0151e80 0x7fccb00c6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49302/0 (socket says 172.21.15.7:49302) 2023-03-03T14:33:37.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb5c33700 1 -- 172.21.15.7:0/275170447 learned_addr learned my addr 172.21.15.7:0/275170447 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:33:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb6434700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccb00c5b00 0x7fccb0155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.073+0000 7fccb6c35700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccb0155410 0x7fccb01edd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fccb5c33700 1 -- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccb0155410 msgr2=0x7fccb01edd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fccb5c33700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccb0155410 0x7fccb01edd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fccb5c33700 1 -- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccb00c5b00 msgr2=0x7fccb0155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fccb5c33700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccb00c5b00 0x7fccb0155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fccb5c33700 1 -- 172.21.15.7:0/275170447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca8009a70 con 0x7fccb0151e80 2023-03-03T14:33:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fccb6434700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccb00c5b00 0x7fccb0155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:33:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fccb5c33700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb0151e80 0x7fccb00c6d10 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fcca000b430 tx=0x7fcca000b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fcca6ffd700 1 -- 172.21.15.7:0/275170447 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0011830 con 0x7fccb0151e80 2023-03-03T14:33:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fcca6ffd700 1 -- 172.21.15.7:0/275170447 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca00119b0 con 0x7fccb0151e80 2023-03-03T14:33:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fccb8698700 1 -- 172.21.15.7:0/275170447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb0154ef0 con 0x7fccb0151e80 2023-03-03T14:33:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fcca6ffd700 1 -- 172.21.15.7:0/275170447 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0011d60 con 0x7fccb0151e80 2023-03-03T14:33:37.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.074+0000 7fccb8698700 1 -- 172.21.15.7:0/275170447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb0154c10 con 0x7fccb0151e80 2023-03-03T14:33:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.075+0000 7fccb8698700 1 -- 172.21.15.7:0/275170447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb00c5220 con 0x7fccb0151e80 2023-03-03T14:33:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.075+0000 7fcca6ffd700 1 -- 172.21.15.7:0/275170447 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcca0021070 con 0x7fccb0151e80 2023-03-03T14:33:37.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.076+0000 7fcca6ffd700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc940682f0 0x7fcc9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.076+0000 7fcca6ffd700 1 -- 172.21.15.7:0/275170447 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcca0085fb0 con 0x7fccb0151e80 2023-03-03T14:33:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.076+0000 7fccb6434700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc940682f0 0x7fcc9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.077+0000 7fccb6434700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc940682f0 0x7fcc9406a7a0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fccac000f60 tx=0x7fccac007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:37.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.080+0000 7fcca6ffd700 1 -- 172.21.15.7:0/275170447 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcca0052350 con 0x7fccb0151e80 2023-03-03T14:33:37.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.317+0000 7fccb8698700 1 -- 172.21.15.7:0/275170447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccb0074bb0 con 0x7fccb0151e80 2023-03-03T14:33:37.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.317+0000 7fcca6ffd700 1 -- 172.21.15.7:0/275170447 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcca0015090 con 0x7fccb0151e80 2023-03-03T14:33:37.319 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:33:37.319 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:33:37.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 -- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc940682f0 msgr2=0x7fcc9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:37.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc940682f0 0x7fcc9406a7a0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fccac000f60 tx=0x7fccac007ca0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 -- 172.21.15.7:0/275170447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb0151e80 msgr2=0x7fccb00c6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:37.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb0151e80 0x7fccb00c6d10 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fcca000b430 tx=0x7fcca000b7f0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 -- 172.21.15.7:0/275170447 shutdown_connections 2023-03-03T14:33:37.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fccb0155410 0x7fccb01edd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fccb00c5b00 0x7fccb0155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc940682f0 0x7fcc9406a7a0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 --2- 172.21.15.7:0/275170447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb0151e80 0x7fccb00c6d10 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:37.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 -- 172.21.15.7:0/275170447 >> 172.21.15.7:0/275170447 conn(0x7fccb0071630 msgr2=0x7fccb00c3100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:33:37.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 -- 172.21.15.7:0/275170447 shutdown_connections 2023-03-03T14:33:37.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:37.319+0000 7fccb8698700 1 -- 172.21.15.7:0/275170447 wait complete. 2023-03-03T14:33:37.323 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:33:37.333 INFO:tasks.mds_thrash.fs.[a]:waiting for 54 secs before thrashing 2023-03-03T14:33:46.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:33:46.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.846+0000 7fb51be1b700 1 Processor -- start 2023-03-03T14:33:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.846+0000 7fb51be1b700 1 -- start start 2023-03-03T14:33:46.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb51be1b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb514071990 0x7fb514071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb51be1b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb514070360 con 0x7fb51414b460 2023-03-03T14:33:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb51be1b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51406a5e0 con 0x7fb514151c00 2023-03-03T14:33:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb51be1b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5141521a0 con 0x7fb514071990 2023-03-03T14:33:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb519bb7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb514071990 0x7fb514071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb5193b6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb51414b460 0x7fb51406aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43292/0 (socket says 172.21.15.7:43292) 2023-03-03T14:33:46.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb5193b6700 1 -- 172.21.15.7:0/3997374991 learned_addr learned my addr 172.21.15.7:0/3997374991 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:33:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb519bb7700 1 -- 172.21.15.7:0/3997374991 >> v1:172.21.15.143:6789/0 conn(0x7fb514151c00 legacy=0x7fb51406b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:33:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb519bb7700 1 -- 172.21.15.7:0/3997374991 >> v1:172.21.15.7:6789/0 conn(0x7fb51414b460 legacy=0x7fb51406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb518bb5700 1 -- 172.21.15.7:0/3997374991 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637441304 0 0) 0x7fb514070360 con 0x7fb51414b460 2023-03-03T14:33:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.847+0000 7fb519bb7700 1 -- 172.21.15.7:0/3997374991 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5140c38b0 con 0x7fb514071990 2023-03-03T14:33:46.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb519bb7700 1 --2- 172.21.15.7:0/3997374991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514071990 0x7fb514071d60 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fb504009840 tx=0x7fb504009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44e5b50f76442ad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb518bb5700 1 -- 172.21.15.7:0/3997374991 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb504004230 con 0x7fb514071990 2023-03-03T14:33:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb518bb5700 1 -- 172.21.15.7:0/3997374991 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb504004390 con 0x7fb514071990 2023-03-03T14:33:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb518bb5700 1 -- 172.21.15.7:0/3997374991 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5040036a0 con 0x7fb514071990 2023-03-03T14:33:46.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb51be1b700 1 -- 172.21.15.7:0/3997374991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514071990 msgr2=0x7fb514071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:46.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb51be1b700 1 --2- 172.21.15.7:0/3997374991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514071990 0x7fb514071d60 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fb504009840 tx=0x7fb504009b50 comp rx=0 tx=0).stop 2023-03-03T14:33:46.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb51be1b700 1 -- 172.21.15.7:0/3997374991 shutdown_connections 2023-03-03T14:33:46.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb51be1b700 1 --2- 172.21.15.7:0/3997374991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514071990 0x7fb514071d60 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:46.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb51be1b700 1 -- 172.21.15.7:0/3997374991 >> 172.21.15.7:0/3997374991 conn(0x7fb5140a94d0 msgr2=0x7fb5140a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:33:46.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb51be1b700 1 -- 172.21.15.7:0/3997374991 shutdown_connections 2023-03-03T14:33:46.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.848+0000 7fb51be1b700 1 -- 172.21.15.7:0/3997374991 wait complete. 2023-03-03T14:33:46.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.849+0000 7fb51be1b700 1 Processor -- start 2023-03-03T14:33:46.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.849+0000 7fb51be1b700 1 -- start start 2023-03-03T14:33:46.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb51be1b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514071990 0x7fb5140ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:46.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb51be1b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514151c00 0x7fb5141472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:46.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb51be1b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5141ebe10 0x7fb5141477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:46.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb51be1b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5141eb9f0 con 0x7fb514151c00 2023-03-03T14:33:46.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb51be1b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5141eb0e0 con 0x7fb5141ebe10 2023-03-03T14:33:46.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb51be1b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5141504d0 con 0x7fb514071990 2023-03-03T14:33:46.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb5193b6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514151c00 0x7fb5141472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb5193b6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514151c00 0x7fb5141472b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49318/0 (socket says 172.21.15.7:49318) 2023-03-03T14:33:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb5193b6700 1 -- 172.21.15.7:0/4121531999 learned_addr learned my addr 172.21.15.7:0/4121531999 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:33:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb51a3b8700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5141ebe10 0x7fb5141477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.850+0000 7fb519bb7700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514071990 0x7fb5140ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb5193b6700 1 -- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514071990 msgr2=0x7fb5140ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb5193b6700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514071990 0x7fb5140ac860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb5193b6700 1 -- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5141ebe10 msgr2=0x7fb5141477f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:46.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb5193b6700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5141ebe10 0x7fb5141477f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:46.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb5193b6700 1 -- 172.21.15.7:0/4121531999 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5040094a0 con 0x7fb514151c00 2023-03-03T14:33:46.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb519bb7700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514071990 0x7fb5140ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:33:46.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb51a3b8700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5141ebe10 0x7fb5141477f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:33:46.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb5193b6700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514151c00 0x7fb5141472b0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fb51000c550 tx=0x7fb51000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:46.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb50a7fc700 1 -- 172.21.15.7:0/4121531999 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb510004c80 con 0x7fb514151c00 2023-03-03T14:33:46.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb50a7fc700 1 -- 172.21.15.7:0/4121531999 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb510004e00 con 0x7fb514151c00 2023-03-03T14:33:46.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb51be1b700 1 -- 172.21.15.7:0/4121531999 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5140c3b50 con 0x7fb514151c00 2023-03-03T14:33:46.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb50a7fc700 1 -- 172.21.15.7:0/4121531999 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb510007310 con 0x7fb514151c00 2023-03-03T14:33:46.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.851+0000 7fb51be1b700 1 -- 172.21.15.7:0/4121531999 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51406a2b0 con 0x7fb514151c00 2023-03-03T14:33:46.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.852+0000 7fb51be1b700 1 -- 172.21.15.7:0/4121531999 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5140722a0 con 0x7fb514151c00 2023-03-03T14:33:46.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.852+0000 7fb50a7fc700 1 -- 172.21.15.7:0/4121531999 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb51001a020 con 0x7fb514151c00 2023-03-03T14:33:46.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.853+0000 7fb50a7fc700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4f80682f0 0x7fb4f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:33:46.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.853+0000 7fb50a7fc700 1 -- 172.21.15.7:0/4121531999 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb510085dc0 con 0x7fb514151c00 2023-03-03T14:33:46.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.853+0000 7fb519bb7700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4f80682f0 0x7fb4f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:33:46.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.854+0000 7fb519bb7700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4f80682f0 0x7fb4f806a7a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fb504000c00 tx=0x7fb50401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:33:46.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:46.857+0000 7fb50a7fc700 1 -- 172.21.15.7:0/4121531999 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb510052210 con 0x7fb514151c00 2023-03-03T14:33:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.096+0000 7fb51be1b700 1 -- 172.21.15.7:0/4121531999 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5141ebb30 con 0x7fb514151c00 2023-03-03T14:33:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.097+0000 7fb50a7fc700 1 -- 172.21.15.7:0/4121531999 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb5141ebb30 con 0x7fb514151c00 2023-03-03T14:33:47.099 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:33:47.099 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:33:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 -- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4f80682f0 msgr2=0x7fb4f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4f80682f0 0x7fb4f806a7a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fb504000c00 tx=0x7fb50401c040 comp rx=0 tx=0).stop 2023-03-03T14:33:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 -- 172.21.15.7:0/4121531999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514151c00 msgr2=0x7fb5141472b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:33:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514151c00 0x7fb5141472b0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fb51000c550 tx=0x7fb51000c910 comp rx=0 tx=0).stop 2023-03-03T14:33:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 -- 172.21.15.7:0/4121531999 shutdown_connections 2023-03-03T14:33:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5141ebe10 0x7fb5141477f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb514071990 0x7fb5140ac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4f80682f0 0x7fb4f806a7a0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:47.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 --2- 172.21.15.7:0/4121531999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb514151c00 0x7fb5141472b0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:33:47.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 -- 172.21.15.7:0/4121531999 >> 172.21.15.7:0/4121531999 conn(0x7fb5140a94d0 msgr2=0x7fb5141da7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:33:47.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 -- 172.21.15.7:0/4121531999 shutdown_connections 2023-03-03T14:33:47.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:33:47.099+0000 7fb51be1b700 1 -- 172.21.15.7:0/4121531999 wait complete. 2023-03-03T14:33:47.102 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:33:47.112 INFO:tasks.mds_thrash.fs.[b]:waiting for 69 secs before thrashing 2023-03-03T14:33:54.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:33:54.716 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:24.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:24.872 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:31.334 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:34:31.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.470+0000 7f0a8e2d9700 1 Processor -- start 2023-03-03T14:34:31.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.470+0000 7f0a8e2d9700 1 -- start start 2023-03-03T14:34:31.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a8e2d9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0a8814b020 0x7f0a880717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a8e2d9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0a880aad20 0x7f0a88157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a8e2d9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a881588f0 con 0x7f0a88071d20 2023-03-03T14:34:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a8e2d9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a880728b0 con 0x7f0a8814b020 2023-03-03T14:34:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a8e2d9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a88155070 con 0x7f0a880aad20 2023-03-03T14:34:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a87fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0a8814b020 0x7f0a880717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a87fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0a8814b020 0x7f0a880717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49380/0 (socket says 172.21.15.7:49380) 2023-03-03T14:34:31.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a87fff700 1 -- 172.21.15.7:0/2182215986 learned_addr learned my addr 172.21.15.7:0/2182215986 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:34:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a877fe700 1 --2- 172.21.15.7:0/2182215986 >> v2:172.21.15.143:3300/0 conn(0x7f0a880aad20 0x7f0a88157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a87fff700 1 -- 172.21.15.7:0/2182215986 >> v2:172.21.15.143:3300/0 conn(0x7f0a880aad20 msgr2=0x7f0a88157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:34:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a87fff700 1 --2- 172.21.15.7:0/2182215986 >> v2:172.21.15.143:3300/0 conn(0x7f0a880aad20 0x7f0a88157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a87fff700 1 -- 172.21.15.7:0/2182215986 >> v1:172.21.15.143:6789/0 conn(0x7f0a88071d20 legacy=0x7f0a881581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a87fff700 1 -- 172.21.15.7:0/2182215986 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8806f5b0 con 0x7f0a8814b020 2023-03-03T14:34:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a877fe700 1 --2- 172.21.15.7:0/2182215986 >> v2:172.21.15.143:3300/0 conn(0x7f0a880aad20 0x7f0a88157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:34:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a87fff700 1 --2- 172.21.15.7:0/2182215986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a8814b020 0x7f0a880717e0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f0a880a98d0 tx=0x7f0a78009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190ef449e557a8f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:31.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.471+0000 7f0a86ffd700 1 -- 172.21.15.7:0/2182215986 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7800b870 con 0x7f0a8814b020 2023-03-03T14:34:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.472+0000 7f0a86ffd700 1 -- 172.21.15.7:0/2182215986 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7800b9d0 con 0x7f0a8814b020 2023-03-03T14:34:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.472+0000 7f0a86ffd700 1 -- 172.21.15.7:0/2182215986 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7800bd60 con 0x7f0a8814b020 2023-03-03T14:34:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.472+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/2182215986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a8814b020 msgr2=0x7f0a880717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.472+0000 7f0a8e2d9700 1 --2- 172.21.15.7:0/2182215986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a8814b020 0x7f0a880717e0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f0a880a98d0 tx=0x7f0a78009bd0 comp rx=0 tx=0).stop 2023-03-03T14:34:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.472+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/2182215986 shutdown_connections 2023-03-03T14:34:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.472+0000 7f0a8e2d9700 1 --2- 172.21.15.7:0/2182215986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a8814b020 0x7f0a880717e0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.472+0000 7f0a8e2d9700 1 --2- 172.21.15.7:0/2182215986 >> v2:172.21.15.143:3300/0 conn(0x7f0a880aad20 0x7f0a88157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.472+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/2182215986 >> 172.21.15.7:0/2182215986 conn(0x7f0a880c8900 msgr2=0x7f0a880c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:34:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.472+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/2182215986 shutdown_connections 2023-03-03T14:34:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.472+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/2182215986 wait complete. 2023-03-03T14:34:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.473+0000 7f0a8e2d9700 1 Processor -- start 2023-03-03T14:34:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a8e2d9700 1 -- start start 2023-03-03T14:34:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a8e2d9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a88071d20 0x7f0a881ed5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a8e2d9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880aad20 0x7f0a8806ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a8e2d9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a8814b020 0x7f0a8806efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a8e2d9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a88147460 con 0x7f0a88071d20 2023-03-03T14:34:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a8e2d9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a881edec0 con 0x7f0a8814b020 2023-03-03T14:34:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a8e2d9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a881ee1a0 con 0x7f0a880aad20 2023-03-03T14:34:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a87fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a88071d20 0x7f0a881ed5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a87fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a88071d20 0x7f0a881ed5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49386/0 (socket says 172.21.15.7:49386) 2023-03-03T14:34:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a87fff700 1 -- 172.21.15.7:0/4104796593 learned_addr learned my addr 172.21.15.7:0/4104796593 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:34:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a877fe700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880aad20 0x7f0a8806ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.474+0000 7f0a8c876700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a8814b020 0x7f0a8806efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a87fff700 1 -- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880aad20 msgr2=0x7f0a8806ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a87fff700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880aad20 0x7f0a8806ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a87fff700 1 -- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a8814b020 msgr2=0x7f0a8806efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a87fff700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a8814b020 0x7f0a8806efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a87fff700 1 -- 172.21.15.7:0/4104796593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a78009550 con 0x7f0a88071d20 2023-03-03T14:34:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a877fe700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880aad20 0x7f0a8806ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:34:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a8c876700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a8814b020 0x7f0a8806efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:34:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a87fff700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a88071d20 0x7f0a881ed5f0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f0a78000c00 tx=0x7f0a78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4104796593 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78004520 con 0x7f0a88071d20 2023-03-03T14:34:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4104796593 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7801e070 con 0x7f0a88071d20 2023-03-03T14:34:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/4104796593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8806fb70 con 0x7f0a88071d20 2023-03-03T14:34:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4104796593 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a780038a0 con 0x7f0a88071d20 2023-03-03T14:34:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.475+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/4104796593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8806f890 con 0x7f0a88071d20 2023-03-03T14:34:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.476+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/4104796593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a88155350 con 0x7f0a88071d20 2023-03-03T14:34:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.476+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4104796593 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0a7800f050 con 0x7f0a88071d20 2023-03-03T14:34:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.477+0000 7f0a84ff9700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a740682f0 0x7f0a7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:31.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.477+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4104796593 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0a78020070 con 0x7f0a88071d20 2023-03-03T14:34:31.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.477+0000 7f0a877fe700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a740682f0 0x7f0a7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:31.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.478+0000 7f0a877fe700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a740682f0 0x7f0a7406a7a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f0a88147410 tx=0x7f0a7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:31.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.481+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4104796593 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0a78053ad0 con 0x7f0a88071d20 2023-03-03T14:34:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.720+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/4104796593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a88074bb0 con 0x7f0a88071d20 2023-03-03T14:34:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.720+0000 7f0a84ff9700 1 -- 172.21.15.7:0/4104796593 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0a780570f0 con 0x7f0a88071d20 2023-03-03T14:34:31.722 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:34:31.722 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:34:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a740682f0 msgr2=0x7f0a7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a740682f0 0x7f0a7406a7a0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f0a88147410 tx=0x7f0a7c008040 comp rx=0 tx=0).stop 2023-03-03T14:34:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/4104796593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a88071d20 msgr2=0x7f0a881ed5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a88071d20 0x7f0a881ed5f0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f0a78000c00 tx=0x7f0a78004750 comp rx=0 tx=0).stop 2023-03-03T14:34:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/4104796593 shutdown_connections 2023-03-03T14:34:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a8814b020 0x7f0a8806efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a740682f0 0x7f0a7406a7a0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a88071d20 0x7f0a881ed5f0 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 --2- 172.21.15.7:0/4104796593 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a880aad20 0x7f0a8806ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/4104796593 >> 172.21.15.7:0/4104796593 conn(0x7f0a880c8900 msgr2=0x7f0a880c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:34:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/4104796593 shutdown_connections 2023-03-03T14:34:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:31.722+0000 7f0a8e2d9700 1 -- 172.21.15.7:0/4104796593 wait complete. 2023-03-03T14:34:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:34:31.736 INFO:tasks.mds_thrash.fs.[a]:waiting for 56 secs before thrashing 2023-03-03T14:34:54.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:54.984 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:34:56.114 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:34:56.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.251+0000 7f1eb8059700 1 Processor -- start 2023-03-03T14:34:56.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.251+0000 7f1eb8059700 1 -- start start 2023-03-03T14:34:56.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb8059700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1eb0155410 0x7f1eb00bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:56.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb8059700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1eb00c5b00 0x7f1eb00bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:56.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb8059700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb00c08d0 con 0x7f1eb0151e80 2023-03-03T14:34:56.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb8059700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb0152250 con 0x7f1eb00c5b00 2023-03-03T14:34:56.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb8059700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb00c0fb0 con 0x7f1eb0155410 2023-03-03T14:34:56.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb65f6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1eb00c5b00 0x7f1eb00bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:56.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb65f6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1eb00c5b00 0x7f1eb00bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49412/0 (socket says 172.21.15.7:49412) 2023-03-03T14:34:56.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb65f6700 1 -- 172.21.15.7:0/948848898 learned_addr learned my addr 172.21.15.7:0/948848898 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:34:56.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb55f4700 1 --2- 172.21.15.7:0/948848898 >> v2:172.21.15.143:3300/0 conn(0x7f1eb0155410 0x7f1eb00bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:56.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb65f6700 1 -- 172.21.15.7:0/948848898 >> v2:172.21.15.143:3300/0 conn(0x7f1eb0155410 msgr2=0x7f1eb00bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:34:56.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb65f6700 1 --2- 172.21.15.7:0/948848898 >> v2:172.21.15.143:3300/0 conn(0x7f1eb0155410 0x7f1eb00bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:56.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb65f6700 1 -- 172.21.15.7:0/948848898 >> v1:172.21.15.143:6789/0 conn(0x7f1eb0151e80 legacy=0x7f1eb00c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:56.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb65f6700 1 -- 172.21.15.7:0/948848898 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb0154930 con 0x7f1eb00c5b00 2023-03-03T14:34:56.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.252+0000 7f1eb65f6700 1 --2- 172.21.15.7:0/948848898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb00c5b00 0x7f1eb00bfe20 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f1eb006c380 tx=0x7f1ea800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6eefce78c8249d7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:56.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb4df3700 1 -- 172.21.15.7:0/948848898 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea800e070 con 0x7f1eb00c5b00 2023-03-03T14:34:56.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb4df3700 1 -- 172.21.15.7:0/948848898 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea800b910 con 0x7f1eb00c5b00 2023-03-03T14:34:56.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb4df3700 1 -- 172.21.15.7:0/948848898 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea800bcc0 con 0x7f1eb00c5b00 2023-03-03T14:34:56.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb8059700 1 -- 172.21.15.7:0/948848898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb00c5b00 msgr2=0x7f1eb00bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:56.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb8059700 1 --2- 172.21.15.7:0/948848898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb00c5b00 0x7f1eb00bfe20 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f1eb006c380 tx=0x7f1ea800b330 comp rx=0 tx=0).stop 2023-03-03T14:34:56.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb8059700 1 -- 172.21.15.7:0/948848898 shutdown_connections 2023-03-03T14:34:56.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb8059700 1 --2- 172.21.15.7:0/948848898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb00c5b00 0x7f1eb00bfe20 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:56.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb8059700 1 --2- 172.21.15.7:0/948848898 >> v2:172.21.15.143:3300/0 conn(0x7f1eb0155410 0x7f1eb00bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:56.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb8059700 1 -- 172.21.15.7:0/948848898 >> 172.21.15.7:0/948848898 conn(0x7f1eb0071630 msgr2=0x7f1eb006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:34:56.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb8059700 1 -- 172.21.15.7:0/948848898 shutdown_connections 2023-03-03T14:34:56.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.253+0000 7f1eb8059700 1 -- 172.21.15.7:0/948848898 wait complete. 2023-03-03T14:34:56.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.254+0000 7f1eb8059700 1 Processor -- start 2023-03-03T14:34:56.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb8059700 1 -- start start 2023-03-03T14:34:56.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb8059700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eb00c5b00 0x7f1eb014f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:56.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb8059700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb0151e80 0x7f1eb01573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:56.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb8059700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eb0155410 0x7f1eb01578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:56.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb8059700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb014d990 con 0x7f1eb0151e80 2023-03-03T14:34:56.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb8059700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb014dc70 con 0x7f1eb0155410 2023-03-03T14:34:56.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb8059700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb014df50 con 0x7f1eb00c5b00 2023-03-03T14:34:56.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb55f4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb0151e80 0x7f1eb01573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:56.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb55f4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb0151e80 0x7f1eb01573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49416/0 (socket says 172.21.15.7:49416) 2023-03-03T14:34:56.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb55f4700 1 -- 172.21.15.7:0/1677976643 learned_addr learned my addr 172.21.15.7:0/1677976643 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:34:56.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb65f6700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eb0155410 0x7f1eb01578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:56.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.255+0000 7f1eb5df5700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eb00c5b00 0x7f1eb014f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:56.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1eb55f4700 1 -- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eb00c5b00 msgr2=0x7f1eb014f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:56.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1eb55f4700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eb00c5b00 0x7f1eb014f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:56.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1eb55f4700 1 -- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eb0155410 msgr2=0x7f1eb01578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:56.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1eb55f4700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eb0155410 0x7f1eb01578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:56.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1eb55f4700 1 -- 172.21.15.7:0/1677976643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8009b30 con 0x7f1eb0151e80 2023-03-03T14:34:56.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1eb55f4700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb0151e80 0x7f1eb01573b0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000e870 tx=0x7f1ea000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:56.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1ea67fc700 1 -- 172.21.15.7:0/1677976643 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0013680 con 0x7f1eb0151e80 2023-03-03T14:34:56.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1ea67fc700 1 -- 172.21.15.7:0/1677976643 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea00137e0 con 0x7f1eb0151e80 2023-03-03T14:34:56.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1ea67fc700 1 -- 172.21.15.7:0/1677976643 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0013b70 con 0x7f1eb0151e80 2023-03-03T14:34:56.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1eb8059700 1 -- 172.21.15.7:0/1677976643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb0154ef0 con 0x7f1eb0151e80 2023-03-03T14:34:56.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.256+0000 7f1eb8059700 1 -- 172.21.15.7:0/1677976643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb0154c10 con 0x7f1eb0151e80 2023-03-03T14:34:56.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.257+0000 7f1eb8059700 1 -- 172.21.15.7:0/1677976643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb00c5220 con 0x7f1eb0151e80 2023-03-03T14:34:56.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.257+0000 7f1ea67fc700 1 -- 172.21.15.7:0/1677976643 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1ea0019030 con 0x7f1eb0151e80 2023-03-03T14:34:56.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.258+0000 7f1ea67fc700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e940682f0 0x7f1e9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:34:56.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.258+0000 7f1ea67fc700 1 -- 172.21.15.7:0/1677976643 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1ea0085e30 con 0x7f1eb0151e80 2023-03-03T14:34:56.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.258+0000 7f1eb5df5700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e940682f0 0x7f1e9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:34:56.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.259+0000 7f1eb5df5700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e940682f0 0x7f1e9406a7a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00c3fb0 tx=0x7f1eac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:34:56.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.262+0000 7f1ea67fc700 1 -- 172.21.15.7:0/1677976643 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1ea00521d0 con 0x7f1eb0151e80 2023-03-03T14:34:56.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.498+0000 7f1eb8059700 1 -- 172.21.15.7:0/1677976643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1eb0074bb0 con 0x7f1eb0151e80 2023-03-03T14:34:56.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.499+0000 7f1ea67fc700 1 -- 172.21.15.7:0/1677976643 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1ea00557f0 con 0x7f1eb0151e80 2023-03-03T14:34:56.500 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:34:56.500 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:34:56.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 -- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e940682f0 msgr2=0x7f1e9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:56.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e940682f0 0x7f1e9406a7a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00c3fb0 tx=0x7f1eac009040 comp rx=0 tx=0).stop 2023-03-03T14:34:56.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 -- 172.21.15.7:0/1677976643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb0151e80 msgr2=0x7f1eb01573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:34:56.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb0151e80 0x7f1eb01573b0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000e870 tx=0x7f1ea000eb80 comp rx=0 tx=0).stop 2023-03-03T14:34:56.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 -- 172.21.15.7:0/1677976643 shutdown_connections 2023-03-03T14:34:56.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1eb0155410 0x7f1eb01578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:56.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1eb00c5b00 0x7f1eb014f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:56.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1e940682f0 0x7f1e9406a7a0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:56.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 --2- 172.21.15.7:0/1677976643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1eb0151e80 0x7f1eb01573b0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:34:56.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 -- 172.21.15.7:0/1677976643 >> 172.21.15.7:0/1677976643 conn(0x7f1eb0071630 msgr2=0x7f1eb00bf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:34:56.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 -- 172.21.15.7:0/1677976643 shutdown_connections 2023-03-03T14:34:56.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:34:56.501+0000 7f1eb8059700 1 -- 172.21.15.7:0/1677976643 wait complete. 2023-03-03T14:34:56.504 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:34:56.514 INFO:tasks.mds_thrash.fs.[b]:waiting for 89 secs before thrashing 2023-03-03T14:35:25.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:25.080 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:27.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:35:27.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.874+0000 7fe35a7f3700 1 Processor -- start 2023-03-03T14:35:27.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.875+0000 7fe35a7f3700 1 -- start start 2023-03-03T14:35:27.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.875+0000 7fe35a7f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe354071e80 0x7fe354151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:35:27.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.875+0000 7fe35a7f3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe354072250 0x7fe35414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:35:27.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.875+0000 7fe35a7f3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3541510d0 con 0x7fe3540c1ce0 2023-03-03T14:35:27.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.875+0000 7fe35a7f3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3540c3740 con 0x7fe354072250 2023-03-03T14:35:27.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.875+0000 7fe35a7f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe354070670 con 0x7fe354071e80 2023-03-03T14:35:27.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.875+0000 7fe3537fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe354072250 0x7fe35414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:35:27.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.875+0000 7fe358d90700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe3540c1ce0 0x7fe3541546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43424/0 (socket says 172.21.15.7:43424) 2023-03-03T14:35:27.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.875+0000 7fe358d90700 1 -- 172.21.15.7:0/4017412102 learned_addr learned my addr 172.21.15.7:0/4017412102 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:35:27.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe353fff700 1 --2- 172.21.15.7:0/4017412102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe354071e80 0x7fe354151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:35:27.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe352ffd700 1 -- 172.21.15.7:0/4017412102 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225415443 0 0) 0x7fe3541510d0 con 0x7fe3540c1ce0 2023-03-03T14:35:27.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe352ffd700 1 -- 172.21.15.7:0/4017412102 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe33c0035b0 con 0x7fe3540c1ce0 2023-03-03T14:35:27.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe3537fe700 1 -- 172.21.15.7:0/4017412102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe354071e80 msgr2=0x7fe354151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:35:27.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe3537fe700 1 --2- 172.21.15.7:0/4017412102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe354071e80 0x7fe354151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:35:27.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe3537fe700 1 -- 172.21.15.7:0/4017412102 >> v1:172.21.15.7:6789/0 conn(0x7fe3540c1ce0 legacy=0x7fe3541546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:35:27.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe3537fe700 1 -- 172.21.15.7:0/4017412102 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3540c39e0 con 0x7fe354072250 2023-03-03T14:35:27.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe3537fe700 1 --2- 172.21.15.7:0/4017412102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354072250 0x7fe35414e1f0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fe348009840 tx=0x7fe348009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0f7f8eeac46e24 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:35:27.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe352ffd700 1 -- 172.21.15.7:0/4017412102 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348011830 con 0x7fe354072250 2023-03-03T14:35:27.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe352ffd700 1 -- 172.21.15.7:0/4017412102 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe348011990 con 0x7fe354072250 2023-03-03T14:35:27.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.876+0000 7fe352ffd700 1 -- 172.21.15.7:0/4017412102 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348011d20 con 0x7fe354072250 2023-03-03T14:35:27.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.877+0000 7fe35a7f3700 1 -- 172.21.15.7:0/4017412102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354072250 msgr2=0x7fe35414e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:35:27.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.877+0000 7fe35a7f3700 1 --2- 172.21.15.7:0/4017412102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354072250 0x7fe35414e1f0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fe348009840 tx=0x7fe348009b50 comp rx=0 tx=0).stop 2023-03-03T14:35:27.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.877+0000 7fe35a7f3700 1 -- 172.21.15.7:0/4017412102 shutdown_connections 2023-03-03T14:35:27.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.877+0000 7fe35a7f3700 1 --2- 172.21.15.7:0/4017412102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe354071e80 0x7fe354151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:35:27.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.877+0000 7fe35a7f3700 1 --2- 172.21.15.7:0/4017412102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354072250 0x7fe35414e1f0 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:35:27.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.877+0000 7fe35a7f3700 1 -- 172.21.15.7:0/4017412102 >> 172.21.15.7:0/4017412102 conn(0x7fe3540a94d0 msgr2=0x7fe3540a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:35:27.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.877+0000 7fe35a7f3700 1 -- 172.21.15.7:0/4017412102 shutdown_connections 2023-03-03T14:35:27.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.877+0000 7fe35a7f3700 1 -- 172.21.15.7:0/4017412102 wait complete. 2023-03-03T14:35:27.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.878+0000 7fe35a7f3700 1 Processor -- start 2023-03-03T14:35:27.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.879+0000 7fe35a7f3700 1 -- start start 2023-03-03T14:35:27.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.879+0000 7fe35a7f3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354071e80 0x7fe3540aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:35:27.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.879+0000 7fe35a7f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe354072250 0x7fe3540c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:35:27.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.879+0000 7fe35a7f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3540c1ce0 0x7fe3541eb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:35:27.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.879+0000 7fe35a7f3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe354147d10 con 0x7fe354071e80 2023-03-03T14:35:27.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.879+0000 7fe35a7f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3540a9bf0 con 0x7fe354072250 2023-03-03T14:35:27.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.879+0000 7fe35a7f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3541eae10 con 0x7fe3540c1ce0 2023-03-03T14:35:27.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe353fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354071e80 0x7fe3540aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:35:27.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe353fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354071e80 0x7fe3540aa4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49446/0 (socket says 172.21.15.7:49446) 2023-03-03T14:35:27.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe353fff700 1 -- 172.21.15.7:0/1579579327 learned_addr learned my addr 172.21.15.7:0/1579579327 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:35:27.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe353fff700 1 -- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3540c1ce0 msgr2=0x7fe3541eb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:35:27.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe3537fe700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe354072250 0x7fe3540c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:35:27.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe358d90700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3540c1ce0 0x7fe3541eb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:35:27.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe353fff700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3540c1ce0 0x7fe3541eb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:35:27.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe353fff700 1 -- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe354072250 msgr2=0x7fe3540c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:35:27.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe353fff700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe354072250 0x7fe3540c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:35:27.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe353fff700 1 -- 172.21.15.7:0/1579579327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3480094a0 con 0x7fe354071e80 2023-03-03T14:35:27.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe353fff700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354071e80 0x7fe3540aa4f0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fe34400e810 tx=0x7fe34400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:35:27.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe350ff9700 1 -- 172.21.15.7:0/1579579327 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe344013610 con 0x7fe354071e80 2023-03-03T14:35:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe350ff9700 1 -- 172.21.15.7:0/1579579327 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe344013770 con 0x7fe354071e80 2023-03-03T14:35:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe35a7f3700 1 -- 172.21.15.7:0/1579579327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35406c9a0 con 0x7fe354071e80 2023-03-03T14:35:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe350ff9700 1 -- 172.21.15.7:0/1579579327 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe344013b20 con 0x7fe354071e80 2023-03-03T14:35:27.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.880+0000 7fe35a7f3700 1 -- 172.21.15.7:0/1579579327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe354151300 con 0x7fe354071e80 2023-03-03T14:35:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.881+0000 7fe35a7f3700 1 -- 172.21.15.7:0/1579579327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35406cb30 con 0x7fe354071e80 2023-03-03T14:35:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.882+0000 7fe350ff9700 1 -- 172.21.15.7:0/1579579327 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe344023070 con 0x7fe354071e80 2023-03-03T14:35:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.882+0000 7fe350ff9700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3400682f0 0x7fe34006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:35:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.882+0000 7fe350ff9700 1 -- 172.21.15.7:0/1579579327 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe344085e80 con 0x7fe354071e80 2023-03-03T14:35:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.882+0000 7fe3537fe700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3400682f0 0x7fe34006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:35:27.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.883+0000 7fe3537fe700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3400682f0 0x7fe34006a7a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe348009840 tx=0x7fe348008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:35:27.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:27.887+0000 7fe350ff9700 1 -- 172.21.15.7:0/1579579327 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe3440521a0 con 0x7fe354071e80 2023-03-03T14:35:28.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.127+0000 7fe35a7f3700 1 -- 172.21.15.7:0/1579579327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe354074bb0 con 0x7fe354071e80 2023-03-03T14:35:28.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.127+0000 7fe350ff9700 1 -- 172.21.15.7:0/1579579327 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe3440557c0 con 0x7fe354071e80 2023-03-03T14:35:28.129 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:35:28.129 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:35:28.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 -- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3400682f0 msgr2=0x7fe34006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:35:28.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3400682f0 0x7fe34006a7a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe348009840 tx=0x7fe348008030 comp rx=0 tx=0).stop 2023-03-03T14:35:28.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 -- 172.21.15.7:0/1579579327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354071e80 msgr2=0x7fe3540aa4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:35:28.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354071e80 0x7fe3540aa4f0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fe34400e810 tx=0x7fe34400ebd0 comp rx=0 tx=0).stop 2023-03-03T14:35:28.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 -- 172.21.15.7:0/1579579327 shutdown_connections 2023-03-03T14:35:28.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3540c1ce0 0x7fe3541eb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:35:28.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3400682f0 0x7fe34006a7a0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:35:28.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe354071e80 0x7fe3540aa4f0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:35:28.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 --2- 172.21.15.7:0/1579579327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe354072250 0x7fe3540c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:35:28.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 -- 172.21.15.7:0/1579579327 >> 172.21.15.7:0/1579579327 conn(0x7fe3540a94d0 msgr2=0x7fe35414e150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:35:28.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 -- 172.21.15.7:0/1579579327 shutdown_connections 2023-03-03T14:35:28.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:35:28.129+0000 7fe35a7f3700 1 -- 172.21.15.7:0/1579579327 wait complete. 2023-03-03T14:35:28.132 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:35:28.142 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T14:35:55.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:35:55.136 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:25.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:25.215 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:25.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:36:25.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.652+0000 7fa0eb5ca700 1 Processor -- start 2023-03-03T14:36:25.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.652+0000 7fa0eb5ca700 1 -- start start 2023-03-03T14:36:25.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.653+0000 7fa0eb5ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c00e0 0x7fa0e40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:25.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.653+0000 7fa0eb5ca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa0e4154ff0 0x7fa0e40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:25.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.653+0000 7fa0eb5ca700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e40c7310 con 0x7fa0e40caa30 2023-03-03T14:36:25.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.653+0000 7fa0eb5ca700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e40bfd10 con 0x7fa0e4154ff0 2023-03-03T14:36:25.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.653+0000 7fa0eb5ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40bfa30 con 0x7fa0e40c00e0 2023-03-03T14:36:25.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.653+0000 7fa0e9366700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa0e40caa30 0x7fa0e40c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43476/0 (socket says 172.21.15.7:43476) 2023-03-03T14:36:25.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.653+0000 7fa0e9366700 1 -- 172.21.15.7:0/3444315302 learned_addr learned my addr 172.21.15.7:0/3444315302 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:36:25.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.653+0000 7fa0e9b67700 1 --2- 172.21.15.7:0/3444315302 >> v2:172.21.15.143:3300/0 conn(0x7fa0e4154ff0 0x7fa0e40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:25.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.653+0000 7fa0e8b65700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c00e0 0x7fa0e40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:25.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822385359 0 0) 0x7fa0e40c7310 con 0x7fa0e40caa30 2023-03-03T14:36:25.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d00035b0 con 0x7fa0e40caa30 2023-03-03T14:36:25.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 820186952 0 0) 0x7fa0d00035b0 con 0x7fa0e40caa30 2023-03-03T14:36:25.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c00e0 msgr2=0x7fa0e40c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:36:25.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c00e0 0x7fa0e40c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 >> v2:172.21.15.143:3300/0 conn(0x7fa0e4154ff0 msgr2=0x7fa0e40cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:36:25.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 --2- 172.21.15.7:0/3444315302 >> v2:172.21.15.143:3300/0 conn(0x7fa0e4154ff0 0x7fa0e40cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e4154510 con 0x7fa0e40caa30 2023-03-03T14:36:25.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa0d4001670 con 0x7fa0e40caa30 2023-03-03T14:36:25.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 >> v1:172.21.15.7:6789/0 conn(0x7fa0e40caa30 legacy=0x7fa0e40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:25.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c00e0 0x7fa0d0003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:25.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0003ff0 0x7fa0d0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:25.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0d0004d90 0x7fa0d0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:25.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40bfd10 con 0x7fa0d0003ff0 2023-03-03T14:36:25.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40bfa30 con 0x7fa0d0004d90 2023-03-03T14:36:25.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d000bac0 con 0x7fa0e40c00e0 2023-03-03T14:36:25.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e8b65700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0003ff0 0x7fa0d0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e9366700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c00e0 0x7fa0d0003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e9b67700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0d0004d90 0x7fa0d0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e8b65700 1 -- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c00e0 msgr2=0x7fa0d0003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:36:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e8b65700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c00e0 0x7fa0d0003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e8b65700 1 -- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0d0004d90 msgr2=0x7fa0d0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:36:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e8b65700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0d0004d90 0x7fa0d0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e8b65700 1 -- 172.21.15.7:0/3444315302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d00035b0 con 0x7fa0d0003ff0 2023-03-03T14:36:25.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e9b67700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0d0004d90 0x7fa0d0005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:36:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e9366700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0e40c00e0 0x7fa0d0003a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:36:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.654+0000 7fa0e8b65700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0003ff0 0x7fa0d0004850 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009b10 tx=0x7fa0d0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae417149dbe65d53 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.655+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0005a20 con 0x7fa0d0003ff0 2023-03-03T14:36:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.655+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e0005b80 con 0x7fa0d0003ff0 2023-03-03T14:36:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.655+0000 7fa0dbfff700 1 -- 172.21.15.7:0/3444315302 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000e6c0 con 0x7fa0d0003ff0 2023-03-03T14:36:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.655+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/3444315302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0003ff0 msgr2=0x7fa0d0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.655+0000 7fa0eb5ca700 1 --2- 172.21.15.7:0/3444315302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0003ff0 0x7fa0d0004850 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fa0e0009b10 tx=0x7fa0d0005880 comp rx=0 tx=0).stop 2023-03-03T14:36:25.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.655+0000 7fa0e9b67700 1 -- 172.21.15.7:0/3444315302 reap_dead start 2023-03-03T14:36:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.655+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/3444315302 shutdown_connections 2023-03-03T14:36:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.655+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/3444315302 >> 172.21.15.7:0/3444315302 conn(0x7fa0e4066e40 msgr2=0x7fa0e4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:36:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.655+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/3444315302 shutdown_connections 2023-03-03T14:36:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.655+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/3444315302 wait complete. 2023-03-03T14:36:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0eb5ca700 1 Processor -- start 2023-03-03T14:36:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0eb5ca700 1 -- start start 2023-03-03T14:36:25.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0eb5ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0d0003ff0 0x7fa0e40c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:25.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0eb5ca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414a1b0 0x7fa0e40c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:25.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0eb5ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e4149560 0x7fa0e41499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:25.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0eb5ca700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e414bb00 con 0x7fa0e414a1b0 2023-03-03T14:36:25.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0eb5ca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e414bde0 con 0x7fa0e4149560 2023-03-03T14:36:25.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0eb5ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e414c0c0 con 0x7fa0d0003ff0 2023-03-03T14:36:25.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0e8b65700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414a1b0 0x7fa0e40c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:25.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0e8b65700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414a1b0 0x7fa0e40c1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49510/0 (socket says 172.21.15.7:49510) 2023-03-03T14:36:25.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0e8b65700 1 -- 172.21.15.7:0/2927316636 learned_addr learned my addr 172.21.15.7:0/2927316636 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:36:25.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0e9366700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0d0003ff0 0x7fa0e40c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:25.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.657+0000 7fa0e9b67700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e4149560 0x7fa0e41499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:25.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0e8b65700 1 -- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0d0003ff0 msgr2=0x7fa0e40c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:25.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0e8b65700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0d0003ff0 0x7fa0e40c42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0e8b65700 1 -- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e4149560 msgr2=0x7fa0e41499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:25.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0e8b65700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e4149560 0x7fa0e41499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0e8b65700 1 -- 172.21.15.7:0/2927316636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e4154ad0 con 0x7fa0e414a1b0 2023-03-03T14:36:25.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0e8b65700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414a1b0 0x7fa0e40c1c50 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000ba10 tx=0x7fa0e00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:25.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0d9ffb700 1 -- 172.21.15.7:0/2927316636 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0021070 con 0x7fa0e414a1b0 2023-03-03T14:36:25.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0d9ffb700 1 -- 172.21.15.7:0/2927316636 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e0004030 con 0x7fa0e414a1b0 2023-03-03T14:36:25.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/2927316636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e41547f0 con 0x7fa0e414a1b0 2023-03-03T14:36:25.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0d9ffb700 1 -- 172.21.15.7:0/2927316636 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00041e0 con 0x7fa0e414a1b0 2023-03-03T14:36:25.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.658+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/2927316636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e40bf650 con 0x7fa0e414a1b0 2023-03-03T14:36:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.659+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/2927316636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e4074bb0 con 0x7fa0e414a1b0 2023-03-03T14:36:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.660+0000 7fa0d9ffb700 1 -- 172.21.15.7:0/2927316636 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa0e002c020 con 0x7fa0e414a1b0 2023-03-03T14:36:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.660+0000 7fa0d9ffb700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 0x7fa0c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:25.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.660+0000 7fa0d9ffb700 1 -- 172.21.15.7:0/2927316636 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa0e00870b0 con 0x7fa0e414a1b0 2023-03-03T14:36:25.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.660+0000 7fa0e9366700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 0x7fa0c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:25.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.661+0000 7fa0e9366700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 0x7fa0c806a7a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4001d40 tx=0x7fa0d4007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:25.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.664+0000 7fa0d9ffb700 1 -- 172.21.15.7:0/2927316636 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa0e0053450 con 0x7fa0e414a1b0 2023-03-03T14:36:25.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.904+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/2927316636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0e40c5030 con 0x7fa0e414a1b0 2023-03-03T14:36:25.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.905+0000 7fa0d9ffb700 1 -- 172.21.15.7:0/2927316636 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa0e0056a70 con 0x7fa0e414a1b0 2023-03-03T14:36:25.906 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:36:25.907 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:36:25.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 msgr2=0x7fa0c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:25.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 0x7fa0c806a7a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4001d40 tx=0x7fa0d4007040 comp rx=0 tx=0).stop 2023-03-03T14:36:25.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/2927316636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414a1b0 msgr2=0x7fa0e40c1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:25.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414a1b0 0x7fa0e40c1c50 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000ba10 tx=0x7fa0e00047c0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/2927316636 shutdown_connections 2023-03-03T14:36:25.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0e4149560 0x7fa0e41499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0d0003ff0 0x7fa0e40c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0c80682f0 0x7fa0c806a7a0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 --2- 172.21.15.7:0/2927316636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0e414a1b0 0x7fa0e40c1c50 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:25.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/2927316636 >> 172.21.15.7:0/2927316636 conn(0x7fa0e4066e40 msgr2=0x7fa0e40c9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:36:25.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/2927316636 shutdown_connections 2023-03-03T14:36:25.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:25.907+0000 7fa0eb5ca700 1 -- 172.21.15.7:0/2927316636 wait complete. 2023-03-03T14:36:25.910 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:36:25.920 INFO:tasks.mds_thrash.fs.[b]:waiting for 83 secs before thrashing 2023-03-03T14:36:32.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:36:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.280+0000 7f08183ef700 1 Processor -- start 2023-03-03T14:36:32.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.280+0000 7f08183ef700 1 -- start start 2023-03-03T14:36:32.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.281+0000 7f08183ef700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f08100c4850 0x7f0810154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:32.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.281+0000 7f08183ef700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0810072920 con 0x7f081014e560 2023-03-03T14:36:32.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.281+0000 7f08183ef700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081006bef0 con 0x7f08100c43d0 2023-03-03T14:36:32.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.281+0000 7f08183ef700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0810072380 con 0x7f08100c4850 2023-03-03T14:36:32.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.281+0000 7f081618b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f081014e560 0x7f0810071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43498/0 (socket says 172.21.15.7:43498) 2023-03-03T14:36:32.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.281+0000 7f081618b700 1 -- 172.21.15.7:0/2042089831 learned_addr learned my addr 172.21.15.7:0/2042089831 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:36:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.281+0000 7f081698c700 1 --2- 172.21.15.7:0/2042089831 >> v2:172.21.15.143:3300/0 conn(0x7f08100c4850 0x7f0810154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.281+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167735171 0 0) 0x7f0800001110 con 0x7f081014e560 2023-03-03T14:36:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.281+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc0035b0 con 0x7f081014e560 2023-03-03T14:36:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 136490600 0 0) 0x7f07fc0035b0 con 0x7f081014e560 2023-03-03T14:36:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 >> v2:172.21.15.143:3300/0 conn(0x7f08100c4850 msgr2=0x7f0810154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:36:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 --2- 172.21.15.7:0/2042089831 >> v2:172.21.15.143:3300/0 conn(0x7f08100c4850 0x7f0810154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 >> v1:172.21.15.143:6789/0 conn(0x7f08100c43d0 legacy=0x7f0810154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08101551e0 con 0x7f081014e560 2023-03-03T14:36:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0800001800 con 0x7f081014e560 2023-03-03T14:36:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 >> v1:172.21.15.7:6789/0 conn(0x7f081014e560 legacy=0x7f0810071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07fc003a70 0x7f07fc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07fc004850 0x7f07fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07fc005200 0x7f07fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0810072380 con 0x7f07fc004850 2023-03-03T14:36:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc00c060 con 0x7f07fc003a70 2023-03-03T14:36:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc00c220 con 0x7f07fc005200 2023-03-03T14:36:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f081598a700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07fc004850 0x7f07fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f081618b700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07fc003a70 0x7f07fc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f081698c700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07fc005200 0x7f07fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f081598a700 1 -- 172.21.15.7:0/2042089831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07fc005200 msgr2=0x7f07fc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:36:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f081598a700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07fc005200 0x7f07fc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f081598a700 1 -- 172.21.15.7:0/2042089831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07fc003a70 msgr2=0x7f07fc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:36:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f081598a700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07fc003a70 0x7f07fc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f081598a700 1 -- 172.21.15.7:0/2042089831 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc0035b0 con 0x7f07fc004850 2023-03-03T14:36:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f081698c700 1 -- 172.21.15.7:0/2042089831 reap_dead start 2023-03-03T14:36:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.282+0000 7f081598a700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07fc004850 0x7f07fc004cc0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f080c0018d0 tx=0x7f07fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a1b351a183f29a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.283+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c0052d0 con 0x7f07fc004850 2023-03-03T14:36:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.283+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080c005430 con 0x7f07fc004850 2023-03-03T14:36:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.283+0000 7f0815189700 1 -- 172.21.15.7:0/2042089831 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c0038a0 con 0x7f07fc004850 2023-03-03T14:36:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.283+0000 7f08183ef700 1 -- 172.21.15.7:0/2042089831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07fc004850 msgr2=0x7f07fc004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.283+0000 7f08183ef700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07fc004850 0x7f07fc004cc0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f080c0018d0 tx=0x7f07fc005880 comp rx=0 tx=0).stop 2023-03-03T14:36:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.283+0000 7f08183ef700 1 -- 172.21.15.7:0/2042089831 shutdown_connections 2023-03-03T14:36:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.283+0000 7f08183ef700 1 --2- 172.21.15.7:0/2042089831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07fc004850 0x7f07fc004cc0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.283+0000 7f08183ef700 1 -- 172.21.15.7:0/2042089831 >> 172.21.15.7:0/2042089831 conn(0x7f08100736f0 msgr2=0x7f0810073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:36:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.283+0000 7f08183ef700 1 -- 172.21.15.7:0/2042089831 shutdown_connections 2023-03-03T14:36:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.283+0000 7f08183ef700 1 -- 172.21.15.7:0/2042089831 wait complete. 2023-03-03T14:36:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.284+0000 7f08183ef700 1 Processor -- start 2023-03-03T14:36:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.284+0000 7f08183ef700 1 -- start start 2023-03-03T14:36:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f08183ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07fc004850 0x7f0810153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f08183ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08101491a0 0x7f08100be9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f08183ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08100ab460 0x7f08100beef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f08183ef700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08100ae0a0 con 0x7f08100ab460 2023-03-03T14:36:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f08183ef700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08100bdc80 con 0x7f07fc004850 2023-03-03T14:36:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f08183ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08100adc80 con 0x7f08101491a0 2023-03-03T14:36:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f081698c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08100ab460 0x7f08100beef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f081698c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08100ab460 0x7f08100beef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49534/0 (socket says 172.21.15.7:49534) 2023-03-03T14:36:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f081698c700 1 -- 172.21.15.7:0/3038085277 learned_addr learned my addr 172.21.15.7:0/3038085277 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:36:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f081618b700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07fc004850 0x7f0810153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.285+0000 7f081598a700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08101491a0 0x7f08100be9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f081698c700 1 -- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08101491a0 msgr2=0x7f08100be9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f081698c700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08101491a0 0x7f08100be9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f081698c700 1 -- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07fc004850 msgr2=0x7f0810153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f081698c700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07fc004850 0x7f0810153760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f081698c700 1 -- 172.21.15.7:0/3038085277 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08100c4c70 con 0x7f08100ab460 2023-03-03T14:36:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f081598a700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08101491a0 0x7f08100be9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:36:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f081698c700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08100ab460 0x7f08100beef0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f0810071200 tx=0x7f0808011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f0806ffd700 1 -- 172.21.15.7:0/3038085277 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080800e040 con 0x7f08100ab460 2023-03-03T14:36:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f0806ffd700 1 -- 172.21.15.7:0/3038085277 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080800f040 con 0x7f08100ab460 2023-03-03T14:36:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f08183ef700 1 -- 172.21.15.7:0/3038085277 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081006b910 con 0x7f08100ab460 2023-03-03T14:36:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f0806ffd700 1 -- 172.21.15.7:0/3038085277 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080800fd20 con 0x7f08100ab460 2023-03-03T14:36:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.286+0000 7f08183ef700 1 -- 172.21.15.7:0/3038085277 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08100aacc0 con 0x7f08100ab460 2023-03-03T14:36:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.287+0000 7f08183ef700 1 -- 172.21.15.7:0/3038085277 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0810074bb0 con 0x7f08100ab460 2023-03-03T14:36:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.287+0000 7f0806ffd700 1 -- 172.21.15.7:0/3038085277 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0808015430 con 0x7f08100ab460 2023-03-03T14:36:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.288+0000 7f0806ffd700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07f40682f0 0x7f07f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:32.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.288+0000 7f0806ffd700 1 -- 172.21.15.7:0/3038085277 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f08080867d0 con 0x7f08100ab460 2023-03-03T14:36:32.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.288+0000 7f081618b700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07f40682f0 0x7f07f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:32.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.288+0000 7f081618b700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07f40682f0 0x7f07f406a7a0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f0800001390 tx=0x7f0800007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:32.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.292+0000 7f0806ffd700 1 -- 172.21.15.7:0/3038085277 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0808052b10 con 0x7f08100ab460 2023-03-03T14:36:32.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.529+0000 7f08183ef700 1 -- 172.21.15.7:0/3038085277 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0810071a50 con 0x7f08100ab460 2023-03-03T14:36:32.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.530+0000 7f0806ffd700 1 -- 172.21.15.7:0/3038085277 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0808056130 con 0x7f08100ab460 2023-03-03T14:36:32.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:36:32.531 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:36:32.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.531+0000 7f08183ef700 1 -- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07f40682f0 msgr2=0x7f07f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:32.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.531+0000 7f08183ef700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07f40682f0 0x7f07f406a7a0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f0800001390 tx=0x7f0800007f90 comp rx=0 tx=0).stop 2023-03-03T14:36:32.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.531+0000 7f08183ef700 1 -- 172.21.15.7:0/3038085277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08100ab460 msgr2=0x7f08100beef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:32.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.531+0000 7f08183ef700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08100ab460 0x7f08100beef0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f0810071200 tx=0x7f0808011f80 comp rx=0 tx=0).stop 2023-03-03T14:36:32.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.531+0000 7f08183ef700 1 -- 172.21.15.7:0/3038085277 shutdown_connections 2023-03-03T14:36:32.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.531+0000 7f08183ef700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07f40682f0 0x7f07f406a7a0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:32.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.531+0000 7f08183ef700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08100ab460 0x7f08100beef0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:32.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.531+0000 7f08183ef700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07fc004850 0x7f0810153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:32.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.531+0000 7f08183ef700 1 --2- 172.21.15.7:0/3038085277 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08101491a0 0x7f08100be9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:32.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.531+0000 7f08183ef700 1 -- 172.21.15.7:0/3038085277 >> 172.21.15.7:0/3038085277 conn(0x7f08100736f0 msgr2=0x7f0810073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:36:32.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.532+0000 7f08183ef700 1 -- 172.21.15.7:0/3038085277 shutdown_connections 2023-03-03T14:36:32.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:32.532+0000 7f08183ef700 1 -- 172.21.15.7:0/3038085277 wait complete. 2023-03-03T14:36:32.535 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:36:32.544 INFO:tasks.mds_thrash.fs.[a]:waiting for 23 secs before thrashing 2023-03-03T14:36:55.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:55.278 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:36:55.545 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:36:55.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.683+0000 7f0f9f666700 1 Processor -- start 2023-03-03T14:36:55.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.683+0000 7f0f9f666700 1 -- start start 2023-03-03T14:36:55.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.684+0000 7f0f9f666700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f98151d40 0x7f0f980736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:55.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.684+0000 7f0f9f666700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f98073c30 0x7f0f981581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:55.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.684+0000 7f0f9f666700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9806d430 con 0x7f0f980c2930 2023-03-03T14:36:55.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.684+0000 7f0f9f666700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9806c2a0 con 0x7f0f98151d40 2023-03-03T14:36:55.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.684+0000 7f0f9f666700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f981586e0 con 0x7f0f98073c30 2023-03-03T14:36:55.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.684+0000 7f0f9d402700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f98151d40 0x7f0f980736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:55.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.684+0000 7f0f9cc01700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0f980c2930 0x7f0f98157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43532/0 (socket says 172.21.15.7:43532) 2023-03-03T14:36:55.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.684+0000 7f0f9cc01700 1 -- 172.21.15.7:0/3592218267 learned_addr learned my addr 172.21.15.7:0/3592218267 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:36:55.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.684+0000 7f0f9dc03700 1 --2- 172.21.15.7:0/3592218267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f98073c30 0x7f0f981581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:55.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.685+0000 7f0f9d402700 1 -- 172.21.15.7:0/3592218267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f98073c30 msgr2=0x7f0f981581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:36:55.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.685+0000 7f0f9d402700 1 --2- 172.21.15.7:0/3592218267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f98073c30 0x7f0f981581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:55.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.685+0000 7f0f9d402700 1 -- 172.21.15.7:0/3592218267 >> v1:172.21.15.7:6789/0 conn(0x7f0f980c2930 legacy=0x7f0f98157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:55.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.685+0000 7f0f8ffff700 1 -- 172.21.15.7:0/3592218267 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124753802 0 0) 0x7f0f9806d430 con 0x7f0f980c2930 2023-03-03T14:36:55.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.685+0000 7f0f9d402700 1 -- 172.21.15.7:0/3592218267 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9806cb00 con 0x7f0f98151d40 2023-03-03T14:36:55.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.685+0000 7f0f9d402700 1 --2- 172.21.15.7:0/3592218267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98151d40 0x7f0f980736f0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009840 tx=0x7f0f88009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b456895b69504d1f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:55.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.685+0000 7f0f8ffff700 1 -- 172.21.15.7:0/3592218267 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88011830 con 0x7f0f98151d40 2023-03-03T14:36:55.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.685+0000 7f0f8ffff700 1 -- 172.21.15.7:0/3592218267 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f88011990 con 0x7f0f98151d40 2023-03-03T14:36:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.685+0000 7f0f8ffff700 1 -- 172.21.15.7:0/3592218267 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88011d20 con 0x7f0f98151d40 2023-03-03T14:36:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.686+0000 7f0f9f666700 1 -- 172.21.15.7:0/3592218267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98151d40 msgr2=0x7f0f980736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.686+0000 7f0f9f666700 1 --2- 172.21.15.7:0/3592218267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98151d40 0x7f0f980736f0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009840 tx=0x7f0f88009b50 comp rx=0 tx=0).stop 2023-03-03T14:36:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.686+0000 7f0f9f666700 1 -- 172.21.15.7:0/3592218267 shutdown_connections 2023-03-03T14:36:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.686+0000 7f0f9f666700 1 --2- 172.21.15.7:0/3592218267 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f98073c30 0x7f0f981581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.686+0000 7f0f9f666700 1 --2- 172.21.15.7:0/3592218267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98151d40 0x7f0f980736f0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.686+0000 7f0f9f666700 1 -- 172.21.15.7:0/3592218267 >> 172.21.15.7:0/3592218267 conn(0x7f0f980a94d0 msgr2=0x7f0f980a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:36:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.686+0000 7f0f9f666700 1 -- 172.21.15.7:0/3592218267 shutdown_connections 2023-03-03T14:36:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.686+0000 7f0f9f666700 1 -- 172.21.15.7:0/3592218267 wait complete. 2023-03-03T14:36:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.687+0000 7f0f9f666700 1 Processor -- start 2023-03-03T14:36:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.687+0000 7f0f9f666700 1 -- start start 2023-03-03T14:36:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9f666700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98073c30 0x7f0f980bde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9f666700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f980c2930 0x7f0f981dde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9f666700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f98151d40 0x7f0f98156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9f666700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f981589a0 con 0x7f0f98073c30 2023-03-03T14:36:55.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9f666700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f981ea750 con 0x7f0f98151d40 2023-03-03T14:36:55.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9f666700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f981ea470 con 0x7f0f980c2930 2023-03-03T14:36:55.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9d402700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98073c30 0x7f0f980bde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:55.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9d402700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98073c30 0x7f0f980bde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49558/0 (socket says 172.21.15.7:49558) 2023-03-03T14:36:55.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9d402700 1 -- 172.21.15.7:0/4217990809 learned_addr learned my addr 172.21.15.7:0/4217990809 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:36:55.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9dc03700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f98151d40 0x7f0f98156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:55.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9d402700 1 -- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f980c2930 msgr2=0x7f0f981dde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:55.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9cc01700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f980c2930 0x7f0f981dde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:55.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9d402700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f980c2930 0x7f0f981dde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:55.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9d402700 1 -- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f98151d40 msgr2=0x7f0f98156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:55.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9d402700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f98151d40 0x7f0f98156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:55.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9d402700 1 -- 172.21.15.7:0/4217990809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f880094a0 con 0x7f0f98073c30 2023-03-03T14:36:55.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9dc03700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f98151d40 0x7f0f98156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:36:55.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9cc01700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f980c2930 0x7f0f981dde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:36:55.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.688+0000 7f0f9d402700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98073c30 0x7f0f980bde80 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009fd0 tx=0x7f0f88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:55.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.689+0000 7f0f8dffb700 1 -- 172.21.15.7:0/4217990809 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88004480 con 0x7f0f98073c30 2023-03-03T14:36:55.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.689+0000 7f0f8dffb700 1 -- 172.21.15.7:0/4217990809 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8801f070 con 0x7f0f98073c30 2023-03-03T14:36:55.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.689+0000 7f0f9f666700 1 -- 172.21.15.7:0/4217990809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9806d0c0 con 0x7f0f98073c30 2023-03-03T14:36:55.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.689+0000 7f0f8dffb700 1 -- 172.21.15.7:0/4217990809 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f880038e0 con 0x7f0f98073c30 2023-03-03T14:36:55.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.689+0000 7f0f9f666700 1 -- 172.21.15.7:0/4217990809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9806cde0 con 0x7f0f98073c30 2023-03-03T14:36:55.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.690+0000 7f0f9f666700 1 -- 172.21.15.7:0/4217990809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9806bea0 con 0x7f0f98073c30 2023-03-03T14:36:55.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.690+0000 7f0f8dffb700 1 -- 172.21.15.7:0/4217990809 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0f8802a020 con 0x7f0f98073c30 2023-03-03T14:36:55.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.691+0000 7f0f8dffb700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f7c0682f0 0x7f0f7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:36:55.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.691+0000 7f0f8dffb700 1 -- 172.21.15.7:0/4217990809 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0f88087460 con 0x7f0f98073c30 2023-03-03T14:36:55.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.691+0000 7f0f9cc01700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f7c0682f0 0x7f0f7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:36:55.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.691+0000 7f0f9cc01700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f7c0682f0 0x7f0f7c06a7a0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f0f94007fd0 tx=0x7f0f94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:36:55.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.695+0000 7f0f8dffb700 1 -- 172.21.15.7:0/4217990809 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0f88053800 con 0x7f0f98073c30 2023-03-03T14:36:55.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.934+0000 7f0f9f666700 1 -- 172.21.15.7:0/4217990809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f98074bb0 con 0x7f0f98073c30 2023-03-03T14:36:55.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.935+0000 7f0f8dffb700 1 -- 172.21.15.7:0/4217990809 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0f88056e20 con 0x7f0f98073c30 2023-03-03T14:36:55.936 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:36:55.937 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:36:55.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 -- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f7c0682f0 msgr2=0x7f0f7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:55.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f7c0682f0 0x7f0f7c06a7a0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f0f94007fd0 tx=0x7f0f94009040 comp rx=0 tx=0).stop 2023-03-03T14:36:55.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 -- 172.21.15.7:0/4217990809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98073c30 msgr2=0x7f0f980bde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:36:55.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98073c30 0x7f0f980bde80 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009fd0 tx=0x7f0f88004750 comp rx=0 tx=0).stop 2023-03-03T14:36:55.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 -- 172.21.15.7:0/4217990809 shutdown_connections 2023-03-03T14:36:55.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f98151d40 0x7f0f98156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:55.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0f7c0682f0 0x7f0f7c06a7a0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:55.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f98073c30 0x7f0f980bde80 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:55.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 --2- 172.21.15.7:0/4217990809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f980c2930 0x7f0f981dde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:36:55.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 -- 172.21.15.7:0/4217990809 >> 172.21.15.7:0/4217990809 conn(0x7f0f980a94d0 msgr2=0x7f0f981558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:36:55.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 -- 172.21.15.7:0/4217990809 shutdown_connections 2023-03-03T14:36:55.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:36:55.937+0000 7f0f9f666700 1 -- 172.21.15.7:0/4217990809 wait complete. 2023-03-03T14:36:55.940 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:36:55.951 INFO:tasks.mds_thrash.fs.[a]:waiting for 94 secs before thrashing 2023-03-03T14:37:25.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:25.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:48.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:37:49.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.058+0000 7fcacff0c700 1 Processor -- start 2023-03-03T14:37:49.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.059+0000 7fcacff0c700 1 -- start start 2023-03-03T14:37:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.059+0000 7fcacff0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:37:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.059+0000 7fcacff0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcac8073c30 0x7fcac81581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:37:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcacff0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcac806d430 con 0x7fcac80c2930 2023-03-03T14:37:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcacff0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcac806c2a0 con 0x7fcac8073c30 2023-03-03T14:37:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcacff0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac81586e0 con 0x7fcac8151d40 2023-03-03T14:37:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcacdca8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:37:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcace4a9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcac8073c30 0x7fcac81581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:37:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcacdca8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac80736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:57672/0 (socket says 172.21.15.7:57672) 2023-03-03T14:37:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcace4a9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcac8073c30 0x7fcac81581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:45378/0 (socket says 172.21.15.7:45378) 2023-03-03T14:37:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcacdca8700 1 -- 172.21.15.7:0/983216108 learned_addr learned my addr 172.21.15.7:0/983216108 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:37:49.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcacd4a7700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fcac80c2930 0x7fcac8157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:47102/0 (socket says 172.21.15.7:47102) 2023-03-03T14:37:49.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcace4a9700 1 -- 172.21.15.7:0/983216108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 msgr2=0x7fcac80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:37:49.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcace4a9700 1 --2- 172.21.15.7:0/983216108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:37:49.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcace4a9700 1 -- 172.21.15.7:0/983216108 >> v1:172.21.15.143:6789/0 conn(0x7fcac80c2930 legacy=0x7fcac8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:37:49.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcace4a9700 1 -- 172.21.15.7:0/983216108 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac806cb00 con 0x7fcac8073c30 2023-03-03T14:37:49.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcaccca6700 1 -- 172.21.15.7:0/983216108 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157508945 0 0) 0x7fcac806d430 con 0x7fcac80c2930 2023-03-03T14:37:49.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.060+0000 7fcacdca8700 1 --2- 172.21.15.7:0/983216108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:37:49.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcace4a9700 1 --2- 172.21.15.7:0/983216108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac8073c30 0x7fcac81581a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009d90 tx=0x7fcac000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d4489e1c8604a11 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:37:49.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcaccca6700 1 -- 172.21.15.7:0/983216108 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac000e070 con 0x7fcac8073c30 2023-03-03T14:37:49.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcaccca6700 1 -- 172.21.15.7:0/983216108 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac000bb20 con 0x7fcac8073c30 2023-03-03T14:37:49.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcaccca6700 1 -- 172.21.15.7:0/983216108 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0012430 con 0x7fcac8073c30 2023-03-03T14:37:49.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcacff0c700 1 -- 172.21.15.7:0/983216108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac8073c30 msgr2=0x7fcac81581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:37:49.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcacff0c700 1 --2- 172.21.15.7:0/983216108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac8073c30 0x7fcac81581a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009d90 tx=0x7fcac000b330 comp rx=0 tx=0).stop 2023-03-03T14:37:49.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcacff0c700 1 -- 172.21.15.7:0/983216108 shutdown_connections 2023-03-03T14:37:49.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcacff0c700 1 --2- 172.21.15.7:0/983216108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac8073c30 0x7fcac81581a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:37:49.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcacff0c700 1 --2- 172.21.15.7:0/983216108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:37:49.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcacff0c700 1 -- 172.21.15.7:0/983216108 >> 172.21.15.7:0/983216108 conn(0x7fcac80a94d0 msgr2=0x7fcac80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:37:49.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcacff0c700 1 -- 172.21.15.7:0/983216108 shutdown_connections 2023-03-03T14:37:49.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.061+0000 7fcacff0c700 1 -- 172.21.15.7:0/983216108 wait complete. 2023-03-03T14:37:49.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacff0c700 1 Processor -- start 2023-03-03T14:37:49.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacff0c700 1 -- start start 2023-03-03T14:37:49.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacff0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac8073c30 0x7fcac8158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:37:49.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacff0c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac80c2930 0x7fcac81e0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:37:49.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacff0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac81eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:37:49.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacff0c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac81dfcd0 con 0x7fcac80c2930 2023-03-03T14:37:49.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacff0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac81edd30 con 0x7fcac8073c30 2023-03-03T14:37:49.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacff0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac81eb3d0 con 0x7fcac8151d40 2023-03-03T14:37:49.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacd4a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac80c2930 0x7fcac81e0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:37:49.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacd4a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac80c2930 0x7fcac81e0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49608/0 (socket says 172.21.15.7:49608) 2023-03-03T14:37:49.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.063+0000 7fcacd4a7700 1 -- 172.21.15.7:0/3305094703 learned_addr learned my addr 172.21.15.7:0/3305094703 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:37:49.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcacdca8700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac8073c30 0x7fcac8158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:37:49.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcace4a9700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac81eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:37:49.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcacd4a7700 1 -- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 msgr2=0x7fcac81eacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:37:49.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcacd4a7700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac81eacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:37:49.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcacd4a7700 1 -- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac8073c30 msgr2=0x7fcac8158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:37:49.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcacd4a7700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac8073c30 0x7fcac8158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:37:49.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcacd4a7700 1 -- 172.21.15.7:0/3305094703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac4009550 con 0x7fcac80c2930 2023-03-03T14:37:49.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcacdca8700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac8073c30 0x7fcac8158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:37:49.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcace4a9700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac81eacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:37:49.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcacd4a7700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac80c2930 0x7fcac81e0160 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00c550 tx=0x7fcabc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:37:49.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcaba7fc700 1 -- 172.21.15.7:0/3305094703 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc004c80 con 0x7fcac80c2930 2023-03-03T14:37:49.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcaba7fc700 1 -- 172.21.15.7:0/3305094703 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcabc004e00 con 0x7fcac80c2930 2023-03-03T14:37:49.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcacff0c700 1 -- 172.21.15.7:0/3305094703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac0009a70 con 0x7fcac80c2930 2023-03-03T14:37:49.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcaba7fc700 1 -- 172.21.15.7:0/3305094703 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcabc007310 con 0x7fcac80c2930 2023-03-03T14:37:49.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.064+0000 7fcacff0c700 1 -- 172.21.15.7:0/3305094703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac806d0c0 con 0x7fcac80c2930 2023-03-03T14:37:49.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.065+0000 7fcacff0c700 1 -- 172.21.15.7:0/3305094703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac806cde0 con 0x7fcac80c2930 2023-03-03T14:37:49.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.066+0000 7fcaba7fc700 1 -- 172.21.15.7:0/3305094703 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcabc01a020 con 0x7fcac80c2930 2023-03-03T14:37:49.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.066+0000 7fcaba7fc700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaac0682f0 0x7fcaac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:37:49.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.066+0000 7fcaba7fc700 1 -- 172.21.15.7:0/3305094703 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcabc085e30 con 0x7fcac80c2930 2023-03-03T14:37:49.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.066+0000 7fcacdca8700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaac0682f0 0x7fcaac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:37:49.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.067+0000 7fcacdca8700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaac0682f0 0x7fcaac06a7a0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009db0 tx=0x7fcac4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:37:49.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.071+0000 7fcaba7fc700 1 -- 172.21.15.7:0/3305094703 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcabc0521f0 con 0x7fcac80c2930 2023-03-03T14:37:49.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.308+0000 7fcacff0c700 1 -- 172.21.15.7:0/3305094703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcac806bea0 con 0x7fcac80c2930 2023-03-03T14:37:49.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.308+0000 7fcaba7fc700 1 -- 172.21.15.7:0/3305094703 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcabc055810 con 0x7fcac80c2930 2023-03-03T14:37:49.310 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:37:49.310 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:37:49.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.310+0000 7fcacff0c700 1 -- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaac0682f0 msgr2=0x7fcaac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:37:49.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.310+0000 7fcacff0c700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaac0682f0 0x7fcaac06a7a0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009db0 tx=0x7fcac4008040 comp rx=0 tx=0).stop 2023-03-03T14:37:49.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.310+0000 7fcacff0c700 1 -- 172.21.15.7:0/3305094703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac80c2930 msgr2=0x7fcac81e0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:37:49.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.310+0000 7fcacff0c700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac80c2930 0x7fcac81e0160 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00c550 tx=0x7fcabc00c910 comp rx=0 tx=0).stop 2023-03-03T14:37:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.310+0000 7fcacff0c700 1 -- 172.21.15.7:0/3305094703 shutdown_connections 2023-03-03T14:37:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.310+0000 7fcacff0c700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcac8151d40 0x7fcac81eacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:37:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.311+0000 7fcacff0c700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcac8073c30 0x7fcac8158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:37:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.311+0000 7fcacff0c700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcaac0682f0 0x7fcaac06a7a0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:37:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.311+0000 7fcacff0c700 1 --2- 172.21.15.7:0/3305094703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcac80c2930 0x7fcac81e0160 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:37:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.311+0000 7fcacff0c700 1 -- 172.21.15.7:0/3305094703 >> 172.21.15.7:0/3305094703 conn(0x7fcac80a94d0 msgr2=0x7fcac80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:37:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.311+0000 7fcacff0c700 1 -- 172.21.15.7:0/3305094703 shutdown_connections 2023-03-03T14:37:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:37:49.311+0000 7fcacff0c700 1 -- 172.21.15.7:0/3305094703 wait complete. 2023-03-03T14:37:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:37:49.324 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-03-03T14:37:55.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:37:55.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:25.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:25.483 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:29.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:38:30.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.088+0000 7f5153051700 1 Processor -- start 2023-03-03T14:38:30.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.088+0000 7f5153051700 1 -- start start 2023-03-03T14:38:30.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.089+0000 7f5153051700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f514c06e020 0x7f514c071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:30.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.089+0000 7f5153051700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c072520 0x7f514c151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:30.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.089+0000 7f5153051700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c154910 con 0x7f514c155390 2023-03-03T14:38:30.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.089+0000 7f5153051700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c070b90 con 0x7f514c06e020 2023-03-03T14:38:30.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.089+0000 7f5153051700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c151320 con 0x7f514c072520 2023-03-03T14:38:30.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f51515ee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c072520 0x7f514c151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:30.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f514bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f514c06e020 0x7f514c071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:30.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f5150ded700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f514c155390 0x7f514c0c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:47136/0 (socket says 172.21.15.7:47136) 2023-03-03T14:38:30.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f51515ee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c072520 0x7f514c151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:57712/0 (socket says 172.21.15.7:57712) 2023-03-03T14:38:30.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f5150ded700 1 -- 172.21.15.7:0/3813276196 learned_addr learned my addr 172.21.15.7:0/3813276196 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:38:30.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f514bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f514c06e020 0x7f514c071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:45412/0 (socket says 172.21.15.7:45412) 2023-03-03T14:38:30.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f514bfff700 1 -- 172.21.15.7:0/3813276196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c072520 msgr2=0x7f514c151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:38:30.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f514bfff700 1 --2- 172.21.15.7:0/3813276196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c072520 0x7f514c151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:30.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f514bfff700 1 -- 172.21.15.7:0/3813276196 >> v1:172.21.15.143:6789/0 conn(0x7f514c155390 legacy=0x7f514c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:30.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f514bfff700 1 -- 172.21.15.7:0/3813276196 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c0c4540 con 0x7f514c06e020 2023-03-03T14:38:30.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f514bfff700 1 --2- 172.21.15.7:0/3813276196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c06e020 0x7f514c071fe0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f513800bae0 tx=0x7f513800bdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1132d4a7fb83c3db server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:38:30.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f514b7fe700 1 -- 172.21.15.7:0/3813276196 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513800dc70 con 0x7f514c06e020 2023-03-03T14:38:30.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f514b7fe700 1 -- 172.21.15.7:0/3813276196 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513800ddd0 con 0x7f514c06e020 2023-03-03T14:38:30.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.090+0000 7f514b7fe700 1 -- 172.21.15.7:0/3813276196 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5138010690 con 0x7f514c06e020 2023-03-03T14:38:30.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.091+0000 7f5153051700 1 -- 172.21.15.7:0/3813276196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c06e020 msgr2=0x7f514c071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:30.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.091+0000 7f5153051700 1 --2- 172.21.15.7:0/3813276196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c06e020 0x7f514c071fe0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f513800bae0 tx=0x7f513800bdf0 comp rx=0 tx=0).stop 2023-03-03T14:38:30.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.091+0000 7f5153051700 1 -- 172.21.15.7:0/3813276196 shutdown_connections 2023-03-03T14:38:30.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.091+0000 7f5153051700 1 --2- 172.21.15.7:0/3813276196 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c072520 0x7f514c151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:30.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.091+0000 7f5153051700 1 --2- 172.21.15.7:0/3813276196 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c06e020 0x7f514c071fe0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:30.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.091+0000 7f5153051700 1 -- 172.21.15.7:0/3813276196 >> 172.21.15.7:0/3813276196 conn(0x7f514c06c110 msgr2=0x7f514c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:38:30.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.091+0000 7f5153051700 1 -- 172.21.15.7:0/3813276196 shutdown_connections 2023-03-03T14:38:30.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.091+0000 7f5153051700 1 -- 172.21.15.7:0/3813276196 wait complete. 2023-03-03T14:38:30.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.092+0000 7f5153051700 1 Processor -- start 2023-03-03T14:38:30.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.092+0000 7f5153051700 1 -- start start 2023-03-03T14:38:30.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f5153051700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f514c06e020 0x7f514c1dd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:30.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f5153051700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c072520 0x7f514c1db820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:30.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f5153051700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c155390 0x7f514c1e6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:30.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f5153051700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c148db0 con 0x7f514c06e020 2023-03-03T14:38:30.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f5153051700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0abab0 con 0x7f514c072520 2023-03-03T14:38:30.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f5153051700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0abf40 con 0x7f514c155390 2023-03-03T14:38:30.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f5150ded700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f514c06e020 0x7f514c1dd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:30.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f5150ded700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f514c06e020 0x7f514c1dd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49642/0 (socket says 172.21.15.7:49642) 2023-03-03T14:38:30.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f5150ded700 1 -- 172.21.15.7:0/1400992784 learned_addr learned my addr 172.21.15.7:0/1400992784 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:38:30.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f514bfff700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c072520 0x7f514c1db820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:30.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.093+0000 7f51515ee700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c155390 0x7f514c1e6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:30.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f5150ded700 1 -- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c155390 msgr2=0x7f514c1e6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:30.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f5150ded700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c155390 0x7f514c1e6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:30.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f5150ded700 1 -- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c072520 msgr2=0x7f514c1db820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:30.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f5150ded700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c072520 0x7f514c1db820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:30.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f5150ded700 1 -- 172.21.15.7:0/1400992784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513800b740 con 0x7f514c06e020 2023-03-03T14:38:30.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f5150ded700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f514c06e020 0x7f514c1dd400 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f514000c480 tx=0x7f514000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:38:30.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f51497fa700 1 -- 172.21.15.7:0/1400992784 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5140005d70 con 0x7f514c06e020 2023-03-03T14:38:30.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f51497fa700 1 -- 172.21.15.7:0/1400992784 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514000ce90 con 0x7f514c06e020 2023-03-03T14:38:30.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f5153051700 1 -- 172.21.15.7:0/1400992784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c070540 con 0x7f514c06e020 2023-03-03T14:38:30.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f51497fa700 1 -- 172.21.15.7:0/1400992784 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51400072d0 con 0x7f514c06e020 2023-03-03T14:38:30.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.094+0000 7f5153051700 1 -- 172.21.15.7:0/1400992784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c0702a0 con 0x7f514c06e020 2023-03-03T14:38:30.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.095+0000 7f5153051700 1 -- 172.21.15.7:0/1400992784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c070820 con 0x7f514c06e020 2023-03-03T14:38:30.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.095+0000 7f51497fa700 1 -- 172.21.15.7:0/1400992784 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5140019030 con 0x7f514c06e020 2023-03-03T14:38:30.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.096+0000 7f51497fa700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51300682f0 0x7f513006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:30.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.096+0000 7f51497fa700 1 -- 172.21.15.7:0/1400992784 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f51400851e0 con 0x7f514c06e020 2023-03-03T14:38:30.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.096+0000 7f514bfff700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51300682f0 0x7f513006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:30.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.097+0000 7f514bfff700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51300682f0 0x7f513006a7a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f513800be30 tx=0x7f5138013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:38:30.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.100+0000 7f51497fa700 1 -- 172.21.15.7:0/1400992784 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f514004b5e0 con 0x7f514c06e020 2023-03-03T14:38:30.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.337+0000 7f5153051700 1 -- 172.21.15.7:0/1400992784 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f514c0c4fd0 con 0x7f514c06e020 2023-03-03T14:38:30.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.338+0000 7f51497fa700 1 -- 172.21.15.7:0/1400992784 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5140051630 con 0x7f514c06e020 2023-03-03T14:38:30.341 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:38:30.341 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:38:30.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.339+0000 7f5153051700 1 -- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51300682f0 msgr2=0x7f513006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:30.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.339+0000 7f5153051700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51300682f0 0x7f513006a7a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f513800be30 tx=0x7f5138013000 comp rx=0 tx=0).stop 2023-03-03T14:38:30.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.339+0000 7f5153051700 1 -- 172.21.15.7:0/1400992784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f514c06e020 msgr2=0x7f514c1dd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:30.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.339+0000 7f5153051700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f514c06e020 0x7f514c1dd400 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f514000c480 tx=0x7f514000c790 comp rx=0 tx=0).stop 2023-03-03T14:38:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.340+0000 7f5153051700 1 -- 172.21.15.7:0/1400992784 shutdown_connections 2023-03-03T14:38:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.340+0000 7f5153051700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f514c155390 0x7f514c1e6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.340+0000 7f5153051700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51300682f0 0x7f513006a7a0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.340+0000 7f5153051700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f514c06e020 0x7f514c1dd400 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.340+0000 7f5153051700 1 --2- 172.21.15.7:0/1400992784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f514c072520 0x7f514c1db820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.340+0000 7f5153051700 1 -- 172.21.15.7:0/1400992784 >> 172.21.15.7:0/1400992784 conn(0x7f514c06c110 msgr2=0x7f514c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:38:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.340+0000 7f5153051700 1 -- 172.21.15.7:0/1400992784 shutdown_connections 2023-03-03T14:38:30.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:30.340+0000 7f5153051700 1 -- 172.21.15.7:0/1400992784 wait complete. 2023-03-03T14:38:30.344 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:38:30.354 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T14:38:49.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:38:49.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.463+0000 7fdc18562700 1 Processor -- start 2023-03-03T14:38:49.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.463+0000 7fdc18562700 1 -- start start 2023-03-03T14:38:49.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.464+0000 7fdc18562700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc10071e80 0x7fdc10151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:49.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.464+0000 7fdc18562700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc101510d0 con 0x7fdc1014e900 2023-03-03T14:38:49.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.464+0000 7fdc18562700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc100c3740 con 0x7fdc10072250 2023-03-03T14:38:49.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.464+0000 7fdc18562700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc10070670 con 0x7fdc10071e80 2023-03-03T14:38:49.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.464+0000 7fdc16aff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdc1014e900 0x7fdc1015ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43662/0 (socket says 172.21.15.7:43662) 2023-03-03T14:38:49.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.464+0000 7fdc16aff700 1 -- 172.21.15.7:0/3021907875 learned_addr learned my addr 172.21.15.7:0/3021907875 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:38:49.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.464+0000 7fdc162fe700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc10071e80 0x7fdc10151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:49.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.464+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807664402 0 0) 0x7fdc101510d0 con 0x7fdc1014e900 2023-03-03T14:38:49.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.464+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbfc0035b0 con 0x7fdc1014e900 2023-03-03T14:38:49.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135001573 0 0) 0x7fdc100c3740 con 0x7fdc10072250 2023-03-03T14:38:49.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc101510d0 con 0x7fdc10072250 2023-03-03T14:38:49.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4032495252 0 0) 0x7fdbfc0035b0 con 0x7fdc1014e900 2023-03-03T14:38:49.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc10071e80 msgr2=0x7fdc10151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:38:49.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc10071e80 0x7fdc10151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:49.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 >> v1:172.21.15.143:6789/0 conn(0x7fdc10072250 legacy=0x7fdc1014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:49.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc100c39e0 con 0x7fdc1014e900 2023-03-03T14:38:49.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fdc080019f0 con 0x7fdc1014e900 2023-03-03T14:38:49.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 >> v1:172.21.15.7:6789/0 conn(0x7fdc1014e900 legacy=0x7fdc1015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:49.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc003c30 0x7fdbfc004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:49.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc10071e80 0x7fdbfc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:49.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 0x7fdbfc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:49.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc10070670 con 0x7fdbfc004d90 2023-03-03T14:38:49.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc00bbd0 con 0x7fdbfc003c30 2023-03-03T14:38:49.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc00bd90 con 0x7fdc10071e80 2023-03-03T14:38:49.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc16aff700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 0x7fdbfc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:49.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc15afd700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc10071e80 0x7fdbfc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:49.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc162fe700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc003c30 0x7fdbfc004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:49.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc16aff700 1 -- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc10071e80 msgr2=0x7fdbfc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:38:49.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc16aff700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc10071e80 0x7fdbfc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:49.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc16aff700 1 -- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc003c30 msgr2=0x7fdbfc004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:38:49.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc16aff700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbfc003c30 0x7fdbfc004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:49.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc16aff700 1 -- 172.21.15.7:0/3021907875 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc0035b0 con 0x7fdbfc004d90 2023-03-03T14:38:49.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.465+0000 7fdc16aff700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 0x7fdbfc005200 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800a6e0 tx=0x7fdbfc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4421355eb0a9a5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:38:49.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.466+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc080072d0 con 0x7fdbfc004d90 2023-03-03T14:38:49.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.466+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc08007430 con 0x7fdbfc004d90 2023-03-03T14:38:49.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.466+0000 7fdc152fc700 1 -- 172.21.15.7:0/3021907875 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc080077c0 con 0x7fdbfc004d90 2023-03-03T14:38:49.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.466+0000 7fdc18562700 1 -- 172.21.15.7:0/3021907875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 msgr2=0x7fdbfc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:49.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.466+0000 7fdc18562700 1 --2- 172.21.15.7:0/3021907875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 0x7fdbfc005200 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fdc0800a6e0 tx=0x7fdbfc005880 comp rx=0 tx=0).stop 2023-03-03T14:38:49.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.466+0000 7fdc16aff700 1 -- 172.21.15.7:0/3021907875 reap_dead start 2023-03-03T14:38:49.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.466+0000 7fdc18562700 1 -- 172.21.15.7:0/3021907875 shutdown_connections 2023-03-03T14:38:49.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.466+0000 7fdc18562700 1 -- 172.21.15.7:0/3021907875 >> 172.21.15.7:0/3021907875 conn(0x7fdc100a94d0 msgr2=0x7fdc100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:38:49.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.466+0000 7fdc18562700 1 -- 172.21.15.7:0/3021907875 shutdown_connections 2023-03-03T14:38:49.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.466+0000 7fdc18562700 1 -- 172.21.15.7:0/3021907875 wait complete. 2023-03-03T14:38:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.467+0000 7fdc18562700 1 Processor -- start 2023-03-03T14:38:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.467+0000 7fdc18562700 1 -- start start 2023-03-03T14:38:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc18562700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 0x7fdc10146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:49.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc18562700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1014aea0 0x7fdc10143480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc18562700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc100c6500 0x7fdc101439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc18562700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc10148840 con 0x7fdbfc004d90 2023-03-03T14:38:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc18562700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1014db00 con 0x7fdc1014aea0 2023-03-03T14:38:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc18562700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc100c6ba0 con 0x7fdc100c6500 2023-03-03T14:38:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc162fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 0x7fdc10146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc162fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 0x7fdc10146c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49690/0 (socket says 172.21.15.7:49690) 2023-03-03T14:38:49.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc162fe700 1 -- 172.21.15.7:0/475505819 learned_addr learned my addr 172.21.15.7:0/475505819 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:38:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc16aff700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc100c6500 0x7fdc101439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.468+0000 7fdc15afd700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1014aea0 0x7fdc10143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc162fe700 1 -- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc100c6500 msgr2=0x7fdc101439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc162fe700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc100c6500 0x7fdc101439c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc162fe700 1 -- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1014aea0 msgr2=0x7fdc10143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc162fe700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1014aea0 0x7fdc10143480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc162fe700 1 -- 172.21.15.7:0/475505819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1006c9a0 con 0x7fdbfc004d90 2023-03-03T14:38:49.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc162fe700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 0x7fdc10146c90 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fdc00011aa0 tx=0x7fdc00011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:38:49.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc06ffd700 1 -- 172.21.15.7:0/475505819 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc00011ee0 con 0x7fdbfc004d90 2023-03-03T14:38:49.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc06ffd700 1 -- 172.21.15.7:0/475505819 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0000f940 con 0x7fdbfc004d90 2023-03-03T14:38:49.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc18562700 1 -- 172.21.15.7:0/475505819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc10151300 con 0x7fdbfc004d90 2023-03-03T14:38:49.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc06ffd700 1 -- 172.21.15.7:0/475505819 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0000fcf0 con 0x7fdbfc004d90 2023-03-03T14:38:49.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.469+0000 7fdc18562700 1 -- 172.21.15.7:0/475505819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1006cb30 con 0x7fdbfc004d90 2023-03-03T14:38:49.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.470+0000 7fdc18562700 1 -- 172.21.15.7:0/475505819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc10074bb0 con 0x7fdbfc004d90 2023-03-03T14:38:49.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.470+0000 7fdc06ffd700 1 -- 172.21.15.7:0/475505819 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc00014440 con 0x7fdbfc004d90 2023-03-03T14:38:49.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.471+0000 7fdc06ffd700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbf4068320 0x7fdbf406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:38:49.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.471+0000 7fdc06ffd700 1 -- 172.21.15.7:0/475505819 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdc000868f0 con 0x7fdbfc004d90 2023-03-03T14:38:49.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.471+0000 7fdc15afd700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbf4068320 0x7fdbf406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:38:49.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.472+0000 7fdc15afd700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbf4068320 0x7fdbf406a7d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fdc101de970 tx=0x7fdc0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:38:49.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.475+0000 7fdc06ffd700 1 -- 172.21.15.7:0/475505819 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc00052cf0 con 0x7fdbfc004d90 2023-03-03T14:38:49.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.711+0000 7fdc18562700 1 -- 172.21.15.7:0/475505819 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc10155590 con 0x7fdbfc004d90 2023-03-03T14:38:49.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.712+0000 7fdc06ffd700 1 -- 172.21.15.7:0/475505819 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdc00056310 con 0x7fdbfc004d90 2023-03-03T14:38:49.714 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:38:49.714 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:38:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 -- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbf4068320 msgr2=0x7fdbf406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbf4068320 0x7fdbf406a7d0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fdc101de970 tx=0x7fdc0c009040 comp rx=0 tx=0).stop 2023-03-03T14:38:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 -- 172.21.15.7:0/475505819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 msgr2=0x7fdc10146c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:38:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 0x7fdc10146c90 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fdc00011aa0 tx=0x7fdc00011db0 comp rx=0 tx=0).stop 2023-03-03T14:38:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 -- 172.21.15.7:0/475505819 shutdown_connections 2023-03-03T14:38:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc100c6500 0x7fdc101439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbf4068320 0x7fdbf406a7d0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbfc004d90 0x7fdc10146c90 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 --2- 172.21.15.7:0/475505819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1014aea0 0x7fdc10143480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:38:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 -- 172.21.15.7:0/475505819 >> 172.21.15.7:0/475505819 conn(0x7fdc100a94d0 msgr2=0x7fdc1006d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:38:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 -- 172.21.15.7:0/475505819 shutdown_connections 2023-03-03T14:38:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:38:49.714+0000 7fdc18562700 1 -- 172.21.15.7:0/475505819 wait complete. 2023-03-03T14:38:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:38:49.727 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2023-03-03T14:38:55.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:38:55.538 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:23.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:39:23.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.865+0000 7ff407959700 1 Processor -- start 2023-03-03T14:39:23.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.865+0000 7ff407959700 1 -- start start 2023-03-03T14:39:23.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.866+0000 7ff407959700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 0x7ff4000c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:23.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.866+0000 7ff407959700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4000c7310 con 0x7ff400154ff0 2023-03-03T14:39:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.866+0000 7ff407959700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4000bfd10 con 0x7ff4000c00e0 2023-03-03T14:39:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.866+0000 7ff407959700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4000bfa30 con 0x7ff4000caa30 2023-03-03T14:39:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.866+0000 7ff405ef6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff400154ff0 0x7ff40015ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43690/0 (socket says 172.21.15.7:43690) 2023-03-03T14:39:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.866+0000 7ff405ef6700 1 -- 172.21.15.7:0/1290372319 learned_addr learned my addr 172.21.15.7:0/1290372319 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:39:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff4056f5700 1 --2- 172.21.15.7:0/1290372319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 0x7ff4000c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:23.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff3f3fff700 1 -- 172.21.15.7:0/1290372319 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763861522 0 0) 0x7ff4000c7310 con 0x7ff400154ff0 2023-03-03T14:39:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff3f3fff700 1 -- 172.21.15.7:0/1290372319 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec0035b0 con 0x7ff400154ff0 2023-03-03T14:39:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff3f3fff700 1 -- 172.21.15.7:0/1290372319 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745709179 0 0) 0x7ff4000bfd10 con 0x7ff4000c00e0 2023-03-03T14:39:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff3f3fff700 1 -- 172.21.15.7:0/1290372319 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4000c7310 con 0x7ff4000c00e0 2023-03-03T14:39:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff4056f5700 1 -- 172.21.15.7:0/1290372319 >> v1:172.21.15.143:6789/0 conn(0x7ff4000c00e0 legacy=0x7ff4000c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff4056f5700 1 -- 172.21.15.7:0/1290372319 >> v1:172.21.15.7:6789/0 conn(0x7ff400154ff0 legacy=0x7ff40015ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff4056f5700 1 -- 172.21.15.7:0/1290372319 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff400154510 con 0x7ff4000caa30 2023-03-03T14:39:23.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff4056f5700 1 --2- 172.21.15.7:0/1290372319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 0x7ff4000c5c60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009870 tx=0x7ff3f4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9f9ccd8d60bb1ce server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff3f3fff700 1 -- 172.21.15.7:0/1290372319 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4011830 con 0x7ff4000caa30 2023-03-03T14:39:23.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff3f3fff700 1 -- 172.21.15.7:0/1290372319 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f4011990 con 0x7ff4000caa30 2023-03-03T14:39:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.867+0000 7ff3f3fff700 1 -- 172.21.15.7:0/1290372319 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4011d20 con 0x7ff4000caa30 2023-03-03T14:39:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.868+0000 7ff407959700 1 -- 172.21.15.7:0/1290372319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 msgr2=0x7ff4000c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.868+0000 7ff407959700 1 --2- 172.21.15.7:0/1290372319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 0x7ff4000c5c60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff3f4009870 tx=0x7ff3f4009b80 comp rx=0 tx=0).stop 2023-03-03T14:39:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.868+0000 7ff407959700 1 -- 172.21.15.7:0/1290372319 shutdown_connections 2023-03-03T14:39:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.868+0000 7ff407959700 1 --2- 172.21.15.7:0/1290372319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 0x7ff4000c5c60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.868+0000 7ff407959700 1 -- 172.21.15.7:0/1290372319 >> 172.21.15.7:0/1290372319 conn(0x7ff400066e40 msgr2=0x7ff400071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:39:23.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.868+0000 7ff407959700 1 -- 172.21.15.7:0/1290372319 shutdown_connections 2023-03-03T14:39:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.868+0000 7ff407959700 1 -- 172.21.15.7:0/1290372319 wait complete. 2023-03-03T14:39:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.869+0000 7ff407959700 1 Processor -- start 2023-03-03T14:39:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.869+0000 7ff407959700 1 -- start start 2023-03-03T14:39:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff407959700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4000c00e0 0x7ff4001ed3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff407959700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 0x7ff4001ed8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff407959700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff400154ff0 0x7ff4001577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff407959700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4001de0a0 con 0x7ff400154ff0 2023-03-03T14:39:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff407959700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4000c3eb0 con 0x7ff4000c00e0 2023-03-03T14:39:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff407959700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4001ede20 con 0x7ff4000caa30 2023-03-03T14:39:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff405ef6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff400154ff0 0x7ff4001577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff405ef6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff400154ff0 0x7ff4001577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49716/0 (socket says 172.21.15.7:49716) 2023-03-03T14:39:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff405ef6700 1 -- 172.21.15.7:0/678422874 learned_addr learned my addr 172.21.15.7:0/678422874 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:39:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff404ef4700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 0x7ff4001ed8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff4056f5700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4000c00e0 0x7ff4001ed3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:23.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff405ef6700 1 -- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 msgr2=0x7ff4001ed8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff405ef6700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 0x7ff4001ed8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff405ef6700 1 -- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4000c00e0 msgr2=0x7ff4001ed3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff405ef6700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4000c00e0 0x7ff4001ed3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff405ef6700 1 -- 172.21.15.7:0/678422874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f40094a0 con 0x7ff400154ff0 2023-03-03T14:39:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.870+0000 7ff4056f5700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4000c00e0 0x7ff4001ed3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:39:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.871+0000 7ff405ef6700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff400154ff0 0x7ff4001577f0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800cba0 tx=0x7ff3f800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.871+0000 7ff3f1ffb700 1 -- 172.21.15.7:0/678422874 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800aa70 con 0x7ff400154ff0 2023-03-03T14:39:23.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.871+0000 7ff3f1ffb700 1 -- 172.21.15.7:0/678422874 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f800abd0 con 0x7ff400154ff0 2023-03-03T14:39:23.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.871+0000 7ff3f1ffb700 1 -- 172.21.15.7:0/678422874 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8012400 con 0x7ff400154ff0 2023-03-03T14:39:23.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.871+0000 7ff407959700 1 -- 172.21.15.7:0/678422874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff400154ad0 con 0x7ff400154ff0 2023-03-03T14:39:23.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.871+0000 7ff407959700 1 -- 172.21.15.7:0/678422874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4001547f0 con 0x7ff400154ff0 2023-03-03T14:39:23.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.872+0000 7ff407959700 1 -- 172.21.15.7:0/678422874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4000bf650 con 0x7ff400154ff0 2023-03-03T14:39:23.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.872+0000 7ff3f1ffb700 1 -- 172.21.15.7:0/678422874 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff3f8019030 con 0x7ff400154ff0 2023-03-03T14:39:23.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.873+0000 7ff3f1ffb700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3e4068320 0x7ff3e406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:23.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.873+0000 7ff3f1ffb700 1 -- 172.21.15.7:0/678422874 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff3f8085cc0 con 0x7ff400154ff0 2023-03-03T14:39:23.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.873+0000 7ff4056f5700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3e4068320 0x7ff3e406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:23.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.874+0000 7ff4056f5700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3e4068320 0x7ff3e406a7d0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40097c0 tx=0x7ff3f401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:23.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:23.877+0000 7ff3f1ffb700 1 -- 172.21.15.7:0/678422874 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff3f8052060 con 0x7ff400154ff0 2023-03-03T14:39:24.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.111+0000 7ff407959700 1 -- 172.21.15.7:0/678422874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4000c4020 con 0x7ff400154ff0 2023-03-03T14:39:24.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.112+0000 7ff3f1ffb700 1 -- 172.21.15.7:0/678422874 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff4000c4020 con 0x7ff400154ff0 2023-03-03T14:39:24.113 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:39:24.113 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:39:24.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 -- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3e4068320 msgr2=0x7ff3e406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:24.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3e4068320 0x7ff3e406a7d0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40097c0 tx=0x7ff3f401c040 comp rx=0 tx=0).stop 2023-03-03T14:39:24.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 -- 172.21.15.7:0/678422874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff400154ff0 msgr2=0x7ff4001577f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:24.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff400154ff0 0x7ff4001577f0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800cba0 tx=0x7ff3f800ceb0 comp rx=0 tx=0).stop 2023-03-03T14:39:24.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 -- 172.21.15.7:0/678422874 shutdown_connections 2023-03-03T14:39:24.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3e4068320 0x7ff3e406a7d0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:24.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff400154ff0 0x7ff4001577f0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:24.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4000c00e0 0x7ff4001ed3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:24.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 --2- 172.21.15.7:0/678422874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff4000caa30 0x7ff4001ed8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:24.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 -- 172.21.15.7:0/678422874 >> 172.21.15.7:0/678422874 conn(0x7ff400066e40 msgr2=0x7ff4000c69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:39:24.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 -- 172.21.15.7:0/678422874 shutdown_connections 2023-03-03T14:39:24.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:24.114+0000 7ff407959700 1 -- 172.21.15.7:0/678422874 wait complete. 2023-03-03T14:39:24.117 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:39:24.127 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-03-03T14:39:25.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:25.602 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:35.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:39:35.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.493+0000 7f58e449d700 1 Processor -- start 2023-03-03T14:39:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.493+0000 7f58e449d700 1 -- start start 2023-03-03T14:39:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.494+0000 7f58e449d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc155390 0x7f58dc0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:35.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.494+0000 7f58e449d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f58dc06e020 0x7f58dc071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:35.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.494+0000 7f58e449d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc154910 con 0x7f58dc072520 2023-03-03T14:39:35.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.494+0000 7f58e449d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58dc070b90 con 0x7f58dc06e020 2023-03-03T14:39:35.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.494+0000 7f58e449d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc151320 con 0x7f58dc155390 2023-03-03T14:39:35.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.494+0000 7f58e2239700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc155390 0x7f58dc0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:35.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.494+0000 7f58e1a38700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f58dc06e020 0x7f58dc071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:35.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.494+0000 7f58e2a3a700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f58dc072520 0x7f58dc151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:47232/0 (socket says 172.21.15.7:47232) 2023-03-03T14:39:35.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.494+0000 7f58e2a3a700 1 -- 172.21.15.7:0/1995067027 learned_addr learned my addr 172.21.15.7:0/1995067027 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:39:35.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e1237700 1 -- 172.21.15.7:0/1995067027 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285276326 0 0) 0x7f58dc154910 con 0x7f58dc072520 2023-03-03T14:39:35.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e1237700 1 -- 172.21.15.7:0/1995067027 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c80035b0 con 0x7f58dc072520 2023-03-03T14:39:35.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e1a38700 1 -- 172.21.15.7:0/1995067027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc155390 msgr2=0x7f58dc0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:39:35.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e1a38700 1 --2- 172.21.15.7:0/1995067027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc155390 0x7f58dc0c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:35.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e1a38700 1 -- 172.21.15.7:0/1995067027 >> v1:172.21.15.143:6789/0 conn(0x7f58dc072520 legacy=0x7f58dc151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:35.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e1a38700 1 -- 172.21.15.7:0/1995067027 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc0702a0 con 0x7f58dc06e020 2023-03-03T14:39:35.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e2239700 1 --2- 172.21.15.7:0/1995067027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc155390 0x7f58dc0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:39:35.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e1a38700 1 --2- 172.21.15.7:0/1995067027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58dc06e020 0x7f58dc071fe0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f58d8009840 tx=0x7f58d8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27633badaa39b45a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:35.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e1237700 1 -- 172.21.15.7:0/1995067027 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8011830 con 0x7f58dc06e020 2023-03-03T14:39:35.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e1237700 1 -- 172.21.15.7:0/1995067027 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d8011990 con 0x7f58dc06e020 2023-03-03T14:39:35.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.495+0000 7f58e1237700 1 -- 172.21.15.7:0/1995067027 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8011d20 con 0x7f58dc06e020 2023-03-03T14:39:35.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.496+0000 7f58e449d700 1 -- 172.21.15.7:0/1995067027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58dc06e020 msgr2=0x7f58dc071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:35.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.496+0000 7f58e449d700 1 --2- 172.21.15.7:0/1995067027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58dc06e020 0x7f58dc071fe0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f58d8009840 tx=0x7f58d8009b50 comp rx=0 tx=0).stop 2023-03-03T14:39:35.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.496+0000 7f58e449d700 1 -- 172.21.15.7:0/1995067027 shutdown_connections 2023-03-03T14:39:35.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.496+0000 7f58e449d700 1 --2- 172.21.15.7:0/1995067027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc155390 0x7f58dc0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:35.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.496+0000 7f58e449d700 1 --2- 172.21.15.7:0/1995067027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58dc06e020 0x7f58dc071fe0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:35.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.496+0000 7f58e449d700 1 -- 172.21.15.7:0/1995067027 >> 172.21.15.7:0/1995067027 conn(0x7f58dc06c110 msgr2=0x7f58dc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:39:35.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.496+0000 7f58e449d700 1 -- 172.21.15.7:0/1995067027 shutdown_connections 2023-03-03T14:39:35.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.496+0000 7f58e449d700 1 -- 172.21.15.7:0/1995067027 wait complete. 2023-03-03T14:39:35.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.497+0000 7f58e449d700 1 Processor -- start 2023-03-03T14:39:35.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.497+0000 7f58e449d700 1 -- start start 2023-03-03T14:39:35.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e449d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58dc06e020 0x7f58dc14f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:35.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e449d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc072520 0x7f58dc152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:35.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e449d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58dc155390 0x7f58dc152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:35.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e449d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc154b50 con 0x7f58dc155390 2023-03-03T14:39:35.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e449d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc1ead20 con 0x7f58dc06e020 2023-03-03T14:39:35.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e449d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc1eb000 con 0x7f58dc072520 2023-03-03T14:39:35.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e2a3a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58dc155390 0x7f58dc152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:35.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e2a3a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58dc155390 0x7f58dc152c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49738/0 (socket says 172.21.15.7:49738) 2023-03-03T14:39:35.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e2a3a700 1 -- 172.21.15.7:0/2923509396 learned_addr learned my addr 172.21.15.7:0/2923509396 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:39:35.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e2239700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58dc06e020 0x7f58dc14f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:35.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.498+0000 7f58e1a38700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc072520 0x7f58dc152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:35.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58e2a3a700 1 -- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc072520 msgr2=0x7f58dc152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:35.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58e2a3a700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc072520 0x7f58dc152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:35.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58e2a3a700 1 -- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58dc06e020 msgr2=0x7f58dc14f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:35.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58e2a3a700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58dc06e020 0x7f58dc14f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:35.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58e2a3a700 1 -- 172.21.15.7:0/2923509396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc0094a0 con 0x7f58dc155390 2023-03-03T14:39:35.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58e2a3a700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58dc155390 0x7f58dc152c50 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f58d400cb40 tx=0x7f58d400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:35.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58d2ffd700 1 -- 172.21.15.7:0/2923509396 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d400a9e0 con 0x7f58dc155390 2023-03-03T14:39:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58d2ffd700 1 -- 172.21.15.7:0/2923509396 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d400ab40 con 0x7f58dc155390 2023-03-03T14:39:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58e449d700 1 -- 172.21.15.7:0/2923509396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d80094a0 con 0x7f58dc155390 2023-03-03T14:39:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58d2ffd700 1 -- 172.21.15.7:0/2923509396 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d4012400 con 0x7f58dc155390 2023-03-03T14:39:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.499+0000 7f58e449d700 1 -- 172.21.15.7:0/2923509396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc070820 con 0x7f58dc155390 2023-03-03T14:39:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.500+0000 7f58e449d700 1 -- 172.21.15.7:0/2923509396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc070540 con 0x7f58dc155390 2023-03-03T14:39:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.500+0000 7f58d2ffd700 1 -- 172.21.15.7:0/2923509396 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f58d4019030 con 0x7f58dc155390 2023-03-03T14:39:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.501+0000 7f58d2ffd700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58c00682f0 0x7f58c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:39:35.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.501+0000 7f58d2ffd700 1 -- 172.21.15.7:0/2923509396 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f58d4085c30 con 0x7f58dc155390 2023-03-03T14:39:35.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.501+0000 7f58e2239700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58c00682f0 0x7f58c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:39:35.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.502+0000 7f58e2239700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58c00682f0 0x7f58c006a7a0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009cd0 tx=0x7f58cc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:39:35.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.505+0000 7f58d2ffd700 1 -- 172.21.15.7:0/2923509396 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f58d404b5e0 con 0x7f58dc155390 2023-03-03T14:39:35.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.742+0000 7f58e449d700 1 -- 172.21.15.7:0/2923509396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58dc155760 con 0x7f58dc155390 2023-03-03T14:39:35.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.743+0000 7f58d2ffd700 1 -- 172.21.15.7:0/2923509396 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f58d4051f90 con 0x7f58dc155390 2023-03-03T14:39:35.745 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:39:35.746 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:39:35.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 -- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58c00682f0 msgr2=0x7f58c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:35.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58c00682f0 0x7f58c006a7a0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009cd0 tx=0x7f58cc009210 comp rx=0 tx=0).stop 2023-03-03T14:39:35.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 -- 172.21.15.7:0/2923509396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58dc155390 msgr2=0x7f58dc152c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:39:35.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58dc155390 0x7f58dc152c50 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f58d400cb40 tx=0x7f58d400ce50 comp rx=0 tx=0).stop 2023-03-03T14:39:35.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 -- 172.21.15.7:0/2923509396 shutdown_connections 2023-03-03T14:39:35.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58c00682f0 0x7f58c006a7a0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:35.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58dc155390 0x7f58dc152c50 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:35.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58dc06e020 0x7f58dc14f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:35.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 --2- 172.21.15.7:0/2923509396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f58dc072520 0x7f58dc152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:39:35.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 -- 172.21.15.7:0/2923509396 >> 172.21.15.7:0/2923509396 conn(0x7f58dc06c110 msgr2=0x7f58dc0c1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:39:35.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 -- 172.21.15.7:0/2923509396 shutdown_connections 2023-03-03T14:39:35.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:39:35.745+0000 7f58e449d700 1 -- 172.21.15.7:0/2923509396 wait complete. 2023-03-03T14:39:35.749 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:39:35.759 INFO:tasks.mds_thrash.fs.[a]:waiting for 35 secs before thrashing 2023-03-03T14:39:55.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:39:55.665 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T14:39:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T14:40:10.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:40:10.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.911+0000 7f30ce639700 1 Processor -- start 2023-03-03T14:40:10.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.911+0000 7f30ce639700 1 -- start start 2023-03-03T14:40:10.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.911+0000 7f30ce639700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f30c8150930 0x7f30c8151e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:10.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.911+0000 7f30ce639700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f30c80c14f0 0x7f30c80c18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:10.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.911+0000 7f30ce639700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c80aae60 con 0x7f30c80719e0 2023-03-03T14:40:10.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.911+0000 7f30ce639700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c80c2760 con 0x7f30c80c14f0 2023-03-03T14:40:10.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.911+0000 7f30ce639700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c80a93d0 con 0x7f30c8150930 2023-03-03T14:40:10.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.911+0000 7f30c7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f30c80719e0 0x7f30c80aa750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43764/0 (socket says 172.21.15.7:43764) 2023-03-03T14:40:10.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.911+0000 7f30c7fff700 1 -- 172.21.15.7:0/469526131 learned_addr learned my addr 172.21.15.7:0/469526131 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:40:10.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30ccbd6700 1 --2- 172.21.15.7:0/469526131 >> v2:172.21.15.7:3300/0 conn(0x7f30c80c14f0 0x7f30c80c18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:10.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30c77fe700 1 --2- 172.21.15.7:0/469526131 >> v2:172.21.15.143:3300/0 conn(0x7f30c8150930 0x7f30c8151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:10.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30c6ffd700 1 -- 172.21.15.7:0/469526131 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231444645 0 0) 0x7f30b8001110 con 0x7f30c80719e0 2023-03-03T14:40:10.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30c6ffd700 1 -- 172.21.15.7:0/469526131 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b00035b0 con 0x7f30c80719e0 2023-03-03T14:40:10.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30ccbd6700 1 -- 172.21.15.7:0/469526131 >> v2:172.21.15.143:3300/0 conn(0x7f30c8150930 msgr2=0x7f30c8151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:40:10.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30ccbd6700 1 --2- 172.21.15.7:0/469526131 >> v2:172.21.15.143:3300/0 conn(0x7f30c8150930 0x7f30c8151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:10.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30ccbd6700 1 -- 172.21.15.7:0/469526131 >> v1:172.21.15.7:6789/0 conn(0x7f30c80719e0 legacy=0x7f30c80aa750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:10.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30ccbd6700 1 -- 172.21.15.7:0/469526131 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c80a96b0 con 0x7f30c80c14f0 2023-03-03T14:40:10.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30ccbd6700 1 --2- 172.21.15.7:0/469526131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80c14f0 0x7f30c80c18f0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f30c80c73e0 tx=0x7f30c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9366a1b3206bb97b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:40:10.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30c6ffd700 1 -- 172.21.15.7:0/469526131 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c000bd20 con 0x7f30c80c14f0 2023-03-03T14:40:10.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30c6ffd700 1 -- 172.21.15.7:0/469526131 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c000be80 con 0x7f30c80c14f0 2023-03-03T14:40:10.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.912+0000 7f30c6ffd700 1 -- 172.21.15.7:0/469526131 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c00116a0 con 0x7f30c80c14f0 2023-03-03T14:40:10.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.913+0000 7f30ce639700 1 -- 172.21.15.7:0/469526131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80c14f0 msgr2=0x7f30c80c18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:10.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.913+0000 7f30ce639700 1 --2- 172.21.15.7:0/469526131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80c14f0 0x7f30c80c18f0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f30c80c73e0 tx=0x7f30c000b330 comp rx=0 tx=0).stop 2023-03-03T14:40:10.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.913+0000 7f30ce639700 1 -- 172.21.15.7:0/469526131 shutdown_connections 2023-03-03T14:40:10.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.913+0000 7f30ce639700 1 --2- 172.21.15.7:0/469526131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80c14f0 0x7f30c80c18f0 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:10.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.913+0000 7f30ce639700 1 --2- 172.21.15.7:0/469526131 >> v2:172.21.15.143:3300/0 conn(0x7f30c8150930 0x7f30c8151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:10.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.913+0000 7f30ce639700 1 -- 172.21.15.7:0/469526131 >> 172.21.15.7:0/469526131 conn(0x7f30c80c62e0 msgr2=0x7f30c80c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:40:10.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.913+0000 7f30ce639700 1 -- 172.21.15.7:0/469526131 shutdown_connections 2023-03-03T14:40:10.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.913+0000 7f30ce639700 1 -- 172.21.15.7:0/469526131 wait complete. 2023-03-03T14:40:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.914+0000 7f30ce639700 1 Processor -- start 2023-03-03T14:40:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.915+0000 7f30ce639700 1 -- start start 2023-03-03T14:40:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.915+0000 7f30ce639700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80719e0 0x7f30c80c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.915+0000 7f30ce639700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30c80c14f0 0x7f30c806f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:10.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.915+0000 7f30ce639700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30c8150930 0x7f30c814b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.915+0000 7f30ce639700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c8070bc0 con 0x7f30c80719e0 2023-03-03T14:40:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.915+0000 7f30ce639700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c81db670 con 0x7f30c80c14f0 2023-03-03T14:40:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.915+0000 7f30ce639700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c80708e0 con 0x7f30c8150930 2023-03-03T14:40:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30c7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80719e0 0x7f30c80c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30c7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80719e0 0x7f30c80c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49790/0 (socket says 172.21.15.7:49790) 2023-03-03T14:40:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30c7fff700 1 -- 172.21.15.7:0/2060747753 learned_addr learned my addr 172.21.15.7:0/2060747753 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:40:10.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30ccbd6700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30c8150930 0x7f30c814b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:10.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30c77fe700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30c80c14f0 0x7f30c806f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:10.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30ccbd6700 1 -- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30c80c14f0 msgr2=0x7f30c806f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:10.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30ccbd6700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30c80c14f0 0x7f30c806f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:10.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30ccbd6700 1 -- 172.21.15.7:0/2060747753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80719e0 msgr2=0x7f30c80c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:10.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30ccbd6700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80719e0 0x7f30c80c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:10.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30ccbd6700 1 -- 172.21.15.7:0/2060747753 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c0009a70 con 0x7f30c8150930 2023-03-03T14:40:10.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30c77fe700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30c80c14f0 0x7f30c806f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:40:10.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30c7fff700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80719e0 0x7f30c80c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:40:10.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30ccbd6700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30c8150930 0x7f30c814b420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f30c0018000 tx=0x7f30c0004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:40:10.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30c4ff9700 1 -- 172.21.15.7:0/2060747753 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0020070 con 0x7f30c8150930 2023-03-03T14:40:10.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.916+0000 7f30c4ff9700 1 -- 172.21.15.7:0/2060747753 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c0004630 con 0x7f30c8150930 2023-03-03T14:40:10.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.917+0000 7f30c4ff9700 1 -- 172.21.15.7:0/2060747753 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0011460 con 0x7f30c8150930 2023-03-03T14:40:10.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.917+0000 7f30ce639700 1 -- 172.21.15.7:0/2060747753 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c80a9c70 con 0x7f30c8150930 2023-03-03T14:40:10.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.917+0000 7f30ce639700 1 -- 172.21.15.7:0/2060747753 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c80a9990 con 0x7f30c8150930 2023-03-03T14:40:10.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.917+0000 7f30ce639700 1 -- 172.21.15.7:0/2060747753 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c80a9f50 con 0x7f30c8150930 2023-03-03T14:40:10.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.918+0000 7f30c4ff9700 1 -- 172.21.15.7:0/2060747753 <== mon.2 v2:172.21.15.143:3301/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f30c00115c0 con 0x7f30c8150930 2023-03-03T14:40:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.918+0000 7f30c4ff9700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30b40682f0 0x7f30b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.919+0000 7f30c4ff9700 1 -- 172.21.15.7:0/2060747753 <== mon.2 v2:172.21.15.143:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f30c0089a50 con 0x7f30c8150930 2023-03-03T14:40:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.919+0000 7f30c7fff700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30b40682f0 0x7f30b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.919+0000 7f30c7fff700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30b40682f0 0x7f30b406a7a0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f30b8001370 tx=0x7f30b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:40:10.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:10.923+0000 7f30c4ff9700 1 -- 172.21.15.7:0/2060747753 <== mon.2 v2:172.21.15.143:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30c0055e90 con 0x7f30c8150930 2023-03-03T14:40:11.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.167+0000 7f30ce639700 1 -- 172.21.15.7:0/2060747753 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30c8074bb0 con 0x7f30c8150930 2023-03-03T14:40:11.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.167+0000 7f30c4ff9700 1 -- 172.21.15.7:0/2060747753 <== mon.2 v2:172.21.15.143:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f30c00594b0 con 0x7f30c8150930 2023-03-03T14:40:11.169 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:40:11.169 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:40:11.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 -- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30b40682f0 msgr2=0x7f30b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:11.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30b40682f0 0x7f30b406a7a0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f30b8001370 tx=0x7f30b8009040 comp rx=0 tx=0).stop 2023-03-03T14:40:11.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 -- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30c8150930 msgr2=0x7f30c814b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:11.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30c8150930 0x7f30c814b420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f30c0018000 tx=0x7f30c0004ad0 comp rx=0 tx=0).stop 2023-03-03T14:40:11.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 -- 172.21.15.7:0/2060747753 shutdown_connections 2023-03-03T14:40:11.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30c8150930 0x7f30c814b420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:11.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30b40682f0 0x7f30b406a7a0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:11.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30c80719e0 0x7f30c80c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:11.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 --2- 172.21.15.7:0/2060747753 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30c80c14f0 0x7f30c806f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:11.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 -- 172.21.15.7:0/2060747753 >> 172.21.15.7:0/2060747753 conn(0x7f30c80c62e0 msgr2=0x7f30c8153510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:40:11.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 -- 172.21.15.7:0/2060747753 shutdown_connections 2023-03-03T14:40:11.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:11.170+0000 7f30ce639700 1 -- 172.21.15.7:0/2060747753 wait complete. 2023-03-03T14:40:11.173 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:40:11.183 INFO:tasks.mds_thrash.fs.[a]:waiting for 95 secs before thrashing 2023-03-03T14:40:24.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:40:24.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.266+0000 7f177e312700 1 Processor -- start 2023-03-03T14:40:24.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.266+0000 7f177e312700 1 -- start start 2023-03-03T14:40:24.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f177e312700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1778151e80 0x7f17780c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:24.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f177e312700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1778155410 0x7f17780bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:24.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f177e312700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17780c6fe0 0x7f17780bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:24.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f177e312700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1778152250 con 0x7f1778155410 2023-03-03T14:40:24.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f177e312700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17780c0fb0 con 0x7f1778151e80 2023-03-03T14:40:24.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f177e312700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1778154740 con 0x7f17780c6fe0 2023-03-03T14:40:24.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f17777fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1778155410 0x7f17780bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:24.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f17777fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1778155410 0x7f17780bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49806/0 (socket says 172.21.15.7:49806) 2023-03-03T14:40:24.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f17777fe700 1 -- 172.21.15.7:0/3129934020 learned_addr learned my addr 172.21.15.7:0/3129934020 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:40:24.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f177c8af700 1 --2- 172.21.15.7:0/3129934020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17780c6fe0 0x7f17780bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:24.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f17777fe700 1 -- 172.21.15.7:0/3129934020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17780c6fe0 msgr2=0x7f17780bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:40:24.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f1777fff700 1 --2- 172.21.15.7:0/3129934020 >> v2:172.21.15.143:3300/0 conn(0x7f1778151e80 0x7f17780c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:24.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f17777fe700 1 --2- 172.21.15.7:0/3129934020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17780c6fe0 0x7f17780bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f17777fe700 1 -- 172.21.15.7:0/3129934020 >> v2:172.21.15.143:3300/0 conn(0x7f1778151e80 msgr2=0x7f17780c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:40:24.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f17777fe700 1 --2- 172.21.15.7:0/3129934020 >> v2:172.21.15.143:3300/0 conn(0x7f1778151e80 0x7f17780c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f17777fe700 1 -- 172.21.15.7:0/3129934020 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780c08d0 con 0x7f1778155410 2023-03-03T14:40:24.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.267+0000 7f17777fe700 1 --2- 172.21.15.7:0/3129934020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 0x7f17780bf8e0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f17780cb250 tx=0x7f176c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de57c95e316f0560 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:40:24.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f1776ffd700 1 -- 172.21.15.7:0/3129934020 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c00b870 con 0x7f1778155410 2023-03-03T14:40:24.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f1776ffd700 1 -- 172.21.15.7:0/3129934020 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176c00b9d0 con 0x7f1778155410 2023-03-03T14:40:24.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f1776ffd700 1 -- 172.21.15.7:0/3129934020 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f176c00bd60 con 0x7f1778155410 2023-03-03T14:40:24.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f177e312700 1 -- 172.21.15.7:0/3129934020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 msgr2=0x7f17780bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:24.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f177e312700 1 --2- 172.21.15.7:0/3129934020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 0x7f17780bf8e0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f17780cb250 tx=0x7f176c009bd0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f177e312700 1 -- 172.21.15.7:0/3129934020 shutdown_connections 2023-03-03T14:40:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f177e312700 1 --2- 172.21.15.7:0/3129934020 >> v2:172.21.15.143:3300/0 conn(0x7f1778151e80 0x7f17780c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f177e312700 1 --2- 172.21.15.7:0/3129934020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17780c6fe0 0x7f17780bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f177e312700 1 --2- 172.21.15.7:0/3129934020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 0x7f17780bf8e0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f177e312700 1 -- 172.21.15.7:0/3129934020 >> 172.21.15.7:0/3129934020 conn(0x7f1778071630 msgr2=0x7f177806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:40:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f177e312700 1 -- 172.21.15.7:0/3129934020 shutdown_connections 2023-03-03T14:40:24.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.268+0000 7f177e312700 1 -- 172.21.15.7:0/3129934020 wait complete. 2023-03-03T14:40:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.269+0000 7f177e312700 1 Processor -- start 2023-03-03T14:40:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f177e312700 1 -- start start 2023-03-03T14:40:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f177e312700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17780c6fe0 0x7f1778150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f177e312700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1778151e80 0x7f1778157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f177e312700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 0x7f1778157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f177e312700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177814e040 con 0x7f1778155410 2023-03-03T14:40:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f177e312700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177814e320 con 0x7f17780c6fe0 2023-03-03T14:40:24.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f177e312700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f177814e600 con 0x7f1778151e80 2023-03-03T14:40:24.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f177c8af700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 0x7f1778157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:24.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f177c8af700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 0x7f1778157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49814/0 (socket says 172.21.15.7:49814) 2023-03-03T14:40:24.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f177c8af700 1 -- 172.21.15.7:0/638265955 learned_addr learned my addr 172.21.15.7:0/638265955 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:40:24.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f1777fff700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17780c6fe0 0x7f1778150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:24.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.270+0000 7f17777fe700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1778151e80 0x7f1778157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:24.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f177c8af700 1 -- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1778151e80 msgr2=0x7f1778157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:24.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f177c8af700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1778151e80 0x7f1778157210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f177c8af700 1 -- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17780c6fe0 msgr2=0x7f1778150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:24.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f177c8af700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17780c6fe0 0x7f1778150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f177c8af700 1 -- 172.21.15.7:0/638265955 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c009550 con 0x7f1778155410 2023-03-03T14:40:24.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f177c8af700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 0x7f1778157750 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f177000cc30 tx=0x7f177000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:40:24.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f1774ff9700 1 -- 172.21.15.7:0/638265955 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177000aa50 con 0x7f1778155410 2023-03-03T14:40:24.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f1774ff9700 1 -- 172.21.15.7:0/638265955 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177000abb0 con 0x7f1778155410 2023-03-03T14:40:24.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f177e312700 1 -- 172.21.15.7:0/638265955 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1778154c10 con 0x7f1778155410 2023-03-03T14:40:24.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f1774ff9700 1 -- 172.21.15.7:0/638265955 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1770003a20 con 0x7f1778155410 2023-03-03T14:40:24.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.271+0000 7f177e312700 1 -- 172.21.15.7:0/638265955 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1778154930 con 0x7f1778155410 2023-03-03T14:40:24.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.272+0000 7f177e312700 1 -- 172.21.15.7:0/638265955 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1778154ef0 con 0x7f1778155410 2023-03-03T14:40:24.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.272+0000 7f1774ff9700 1 -- 172.21.15.7:0/638265955 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f177001f070 con 0x7f1778155410 2023-03-03T14:40:24.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.273+0000 7f1774ff9700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17640682f0 0x7f176406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:24.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.273+0000 7f1774ff9700 1 -- 172.21.15.7:0/638265955 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1770086040 con 0x7f1778155410 2023-03-03T14:40:24.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.273+0000 7f1777fff700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17640682f0 0x7f176406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:24.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.274+0000 7f1777fff700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17640682f0 0x7f176406a7a0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f177814a220 tx=0x7f1768008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:40:24.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.277+0000 7f1774ff9700 1 -- 172.21.15.7:0/638265955 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f17700523e0 con 0x7f1778155410 2023-03-03T14:40:24.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.514+0000 7f177e312700 1 -- 172.21.15.7:0/638265955 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1778074bb0 con 0x7f1778155410 2023-03-03T14:40:24.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.515+0000 7f1774ff9700 1 -- 172.21.15.7:0/638265955 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f177001d020 con 0x7f1778155410 2023-03-03T14:40:24.516 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:40:24.516 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:40:24.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 -- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17640682f0 msgr2=0x7f176406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:24.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17640682f0 0x7f176406a7a0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f177814a220 tx=0x7f1768008040 comp rx=0 tx=0).stop 2023-03-03T14:40:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 -- 172.21.15.7:0/638265955 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 msgr2=0x7f1778157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 0x7f1778157750 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f177000cc30 tx=0x7f177000cf40 comp rx=0 tx=0).stop 2023-03-03T14:40:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 -- 172.21.15.7:0/638265955 shutdown_connections 2023-03-03T14:40:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17640682f0 0x7f176406a7a0 secure :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f177814a220 tx=0x7f1768008040 comp rx=0 tx=0).stop 2023-03-03T14:40:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1778155410 0x7f1778157750 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17780c6fe0 0x7f1778150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 --2- 172.21.15.7:0/638265955 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1778151e80 0x7f1778157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 -- 172.21.15.7:0/638265955 >> 172.21.15.7:0/638265955 conn(0x7f1778071630 msgr2=0x7f17780c4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:40:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 -- 172.21.15.7:0/638265955 shutdown_connections 2023-03-03T14:40:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:24.517+0000 7f177e312700 1 -- 172.21.15.7:0/638265955 wait complete. 2023-03-03T14:40:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:40:24.530 INFO:tasks.mds_thrash.fs.[b]:waiting for 12 secs before thrashing 2023-03-03T14:40:25.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:25.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:36.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:40:36.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.669+0000 7f691a22e700 1 Processor -- start 2023-03-03T14:40:36.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.669+0000 7f691a22e700 1 -- start start 2023-03-03T14:40:36.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f691a22e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6914151b20 0x7f6914071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:36.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f691a22e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6914154f10 0x7f6914158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:36.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f691a22e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6914151ef0 con 0x7f6914071d60 2023-03-03T14:40:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f691a22e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6914157b20 con 0x7f6914154f10 2023-03-03T14:40:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f691a22e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6914155410 con 0x7f6914151b20 2023-03-03T14:40:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f6912ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6914154f10 0x7f6914158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f6913fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6914071d60 0x7f69140c17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43804/0 (socket says 172.21.15.7:43804) 2023-03-03T14:40:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f6913fff700 1 -- 172.21.15.7:0/1471628474 learned_addr learned my addr 172.21.15.7:0/1471628474 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:40:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f69137fe700 1 --2- 172.21.15.7:0/1471628474 >> v2:172.21.15.143:3300/0 conn(0x7f6914151b20 0x7f6914071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f6912ffd700 1 -- 172.21.15.7:0/1471628474 >> v2:172.21.15.143:3300/0 conn(0x7f6914151b20 msgr2=0x7f6914071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:40:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f6912ffd700 1 --2- 172.21.15.7:0/1471628474 >> v2:172.21.15.143:3300/0 conn(0x7f6914151b20 0x7f6914071820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f6912ffd700 1 -- 172.21.15.7:0/1471628474 >> v1:172.21.15.7:6789/0 conn(0x7f6914071d60 legacy=0x7f69140c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f6912ffd700 1 -- 172.21.15.7:0/1471628474 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691406f640 con 0x7f6914154f10 2023-03-03T14:40:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f69127fc700 1 -- 172.21.15.7:0/1471628474 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670945403 0 0) 0x7f6914151ef0 con 0x7f6914071d60 2023-03-03T14:40:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.670+0000 7f6912ffd700 1 --2- 172.21.15.7:0/1471628474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 0x7f6914158000 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f69140708e0 tx=0x7f68fc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c7e18fc433eda8e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:40:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f69127fc700 1 -- 172.21.15.7:0/1471628474 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc011830 con 0x7f6914154f10 2023-03-03T14:40:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f69127fc700 1 -- 172.21.15.7:0/1471628474 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc011990 con 0x7f6914154f10 2023-03-03T14:40:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f69127fc700 1 -- 172.21.15.7:0/1471628474 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc011d20 con 0x7f6914154f10 2023-03-03T14:40:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f691a22e700 1 -- 172.21.15.7:0/1471628474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 msgr2=0x7f6914158000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f691a22e700 1 --2- 172.21.15.7:0/1471628474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 0x7f6914158000 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f69140708e0 tx=0x7f68fc009b20 comp rx=0 tx=0).stop 2023-03-03T14:40:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f691a22e700 1 -- 172.21.15.7:0/1471628474 shutdown_connections 2023-03-03T14:40:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f691a22e700 1 --2- 172.21.15.7:0/1471628474 >> v2:172.21.15.143:3300/0 conn(0x7f6914151b20 0x7f6914071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f691a22e700 1 --2- 172.21.15.7:0/1471628474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 0x7f6914158000 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f691a22e700 1 -- 172.21.15.7:0/1471628474 >> 172.21.15.7:0/1471628474 conn(0x7f69140c8900 msgr2=0x7f69140c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:40:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f691a22e700 1 -- 172.21.15.7:0/1471628474 shutdown_connections 2023-03-03T14:40:36.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.671+0000 7f691a22e700 1 -- 172.21.15.7:0/1471628474 wait complete. 2023-03-03T14:40:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.672+0000 7f691a22e700 1 Processor -- start 2023-03-03T14:40:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.673+0000 7f691a22e700 1 -- start start 2023-03-03T14:40:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.673+0000 7f691a22e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6914071d60 0x7f69140c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.673+0000 7f691a22e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6914151b20 0x7f69141ea1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.673+0000 7f691a22e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 0x7f69141442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.673+0000 7f691a22e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69141e97c0 con 0x7f6914154f10 2023-03-03T14:40:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.673+0000 7f691a22e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69141531c0 con 0x7f6914151b20 2023-03-03T14:40:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.673+0000 7f691a22e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69141e9d40 con 0x7f6914071d60 2023-03-03T14:40:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.673+0000 7f6913fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 0x7f69141442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.673+0000 7f6913fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 0x7f69141442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49830/0 (socket says 172.21.15.7:49830) 2023-03-03T14:40:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.673+0000 7f6913fff700 1 -- 172.21.15.7:0/3681910429 learned_addr learned my addr 172.21.15.7:0/3681910429 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:40:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f69137fe700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6914071d60 0x7f69140c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f6912ffd700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6914151b20 0x7f69141ea1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f6913fff700 1 -- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6914071d60 msgr2=0x7f69140c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f6913fff700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6914071d60 0x7f69140c6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f6913fff700 1 -- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6914151b20 msgr2=0x7f69141ea1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f6913fff700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6914151b20 0x7f69141ea1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f6913fff700 1 -- 172.21.15.7:0/3681910429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc0094a0 con 0x7f6914154f10 2023-03-03T14:40:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f6912ffd700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6914151b20 0x7f69141ea1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:40:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f6913fff700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 0x7f69141442b0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f690c00cc10 tx=0x7f690c0021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:40:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f690bfff700 1 -- 172.21.15.7:0/3681910429 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c00ab80 con 0x7f6914154f10 2023-03-03T14:40:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f690bfff700 1 -- 172.21.15.7:0/3681910429 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690c00ad00 con 0x7f6914154f10 2023-03-03T14:40:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f691a22e700 1 -- 172.21.15.7:0/3681910429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691406fc00 con 0x7f6914154f10 2023-03-03T14:40:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f690bfff700 1 -- 172.21.15.7:0/3681910429 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c007310 con 0x7f6914154f10 2023-03-03T14:40:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.674+0000 7f691a22e700 1 -- 172.21.15.7:0/3681910429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691406f920 con 0x7f6914154f10 2023-03-03T14:40:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.675+0000 7f691a22e700 1 -- 172.21.15.7:0/3681910429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69140aae90 con 0x7f6914154f10 2023-03-03T14:40:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.676+0000 7f690bfff700 1 -- 172.21.15.7:0/3681910429 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f690c01a030 con 0x7f6914154f10 2023-03-03T14:40:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.676+0000 7f690bfff700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68f40682f0 0x7f68f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:40:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.676+0000 7f690bfff700 1 -- 172.21.15.7:0/3681910429 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f690c085e80 con 0x7f6914154f10 2023-03-03T14:40:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.676+0000 7f69137fe700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68f40682f0 0x7f68f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:40:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.677+0000 7f69137fe700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68f40682f0 0x7f68f406a7a0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f690400b9b0 tx=0x7f6904008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:40:36.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.681+0000 7f690bfff700 1 -- 172.21.15.7:0/3681910429 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f690c052220 con 0x7f6914154f10 2023-03-03T14:40:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.916+0000 7f691a22e700 1 -- 172.21.15.7:0/3681910429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69140c6850 con 0x7f6914154f10 2023-03-03T14:40:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.916+0000 7f690bfff700 1 -- 172.21.15.7:0/3681910429 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f690c018090 con 0x7f6914154f10 2023-03-03T14:40:36.919 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:40:36.920 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:40:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 -- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68f40682f0 msgr2=0x7f68f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68f40682f0 0x7f68f406a7a0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f690400b9b0 tx=0x7f6904008040 comp rx=0 tx=0).stop 2023-03-03T14:40:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 -- 172.21.15.7:0/3681910429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 msgr2=0x7f69141442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:40:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 0x7f69141442b0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f690c00cc10 tx=0x7f690c0021c0 comp rx=0 tx=0).stop 2023-03-03T14:40:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 -- 172.21.15.7:0/3681910429 shutdown_connections 2023-03-03T14:40:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68f40682f0 0x7f68f406a7a0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6914154f10 0x7f69141442b0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6914071d60 0x7f69140c6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 --2- 172.21.15.7:0/3681910429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6914151b20 0x7f69141ea1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:40:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 -- 172.21.15.7:0/3681910429 >> 172.21.15.7:0/3681910429 conn(0x7f69140c8900 msgr2=0x7f69140c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:40:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 -- 172.21.15.7:0/3681910429 shutdown_connections 2023-03-03T14:40:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:40:36.919+0000 7f691a22e700 1 -- 172.21.15.7:0/3681910429 wait complete. 2023-03-03T14:40:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:40:36.933 INFO:tasks.mds_thrash.fs.[b]:waiting for 26 secs before thrashing 2023-03-03T14:40:55.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:40:55.807 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:02.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:41:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.072+0000 7f3ffa292700 1 Processor -- start 2023-03-03T14:41:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.072+0000 7f3ffa292700 1 -- start start 2023-03-03T14:41:03.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.073+0000 7f3ffa292700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3ff4152e30 0x7f3ff40c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:03.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.073+0000 7f3ffa292700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff41523b0 con 0x7f3ff40beab0 2023-03-03T14:41:03.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.073+0000 7f3ffa292700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff4072010 con 0x7f3ff40c4910 2023-03-03T14:41:03.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.073+0000 7f3ffa292700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ff40aaaf0 con 0x7f3ff4152e30 2023-03-03T14:41:03.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.073+0000 7f3ff37fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3ff40beab0 0x7f3ff4155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43834/0 (socket says 172.21.15.7:43834) 2023-03-03T14:41:03.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.073+0000 7f3ff37fe700 1 -- 172.21.15.7:0/1466128025 learned_addr learned my addr 172.21.15.7:0/1466128025 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:41:03.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.073+0000 7f3ff3fff700 1 --2- 172.21.15.7:0/1466128025 >> v2:172.21.15.143:3300/0 conn(0x7f3ff4152e30 0x7f3ff40c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.073+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368624604 0 0) 0x7f3ff41523b0 con 0x7f3ff40beab0 2023-03-03T14:41:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.073+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe00035b0 con 0x7f3ff40beab0 2023-03-03T14:41:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.073+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2921642901 0 0) 0x7f3fe00035b0 con 0x7f3ff40beab0 2023-03-03T14:41:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 >> v2:172.21.15.143:3300/0 conn(0x7f3ff4152e30 msgr2=0x7f3ff40c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:41:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 --2- 172.21.15.7:0/1466128025 >> v2:172.21.15.143:3300/0 conn(0x7f3ff4152e30 0x7f3ff40c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 >> v1:172.21.15.143:6789/0 conn(0x7f3ff40c4910 legacy=0x7f3ff415cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff40aadd0 con 0x7f3ff40beab0 2023-03-03T14:41:03.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f3fe4001670 con 0x7f3ff40beab0 2023-03-03T14:41:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 >> v1:172.21.15.7:6789/0 conn(0x7f3ff40beab0 legacy=0x7f3ff4155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe0003a70 0x7f3fe0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0004850 0x7f3fe0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe0005250 0x7f3fe000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff40aaaf0 con 0x7f3fe0004850 2023-03-03T14:41:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe000c190 con 0x7f3fe0005250 2023-03-03T14:41:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe000c350 con 0x7f3fe0003a70 2023-03-03T14:41:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff37fe700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0004850 0x7f3fe0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff882f700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe0005250 0x7f3fe000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff3fff700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe0003a70 0x7f3fe0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff37fe700 1 -- 172.21.15.7:0/1466128025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe0003a70 msgr2=0x7f3fe0003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:41:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff37fe700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe0003a70 0x7f3fe0003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff37fe700 1 -- 172.21.15.7:0/1466128025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe0005250 msgr2=0x7f3fe000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:41:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff37fe700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fe0005250 0x7f3fe000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff37fe700 1 -- 172.21.15.7:0/1466128025 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe00035b0 con 0x7f3fe0004850 2023-03-03T14:41:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff882f700 1 -- 172.21.15.7:0/1466128025 reap_dead start 2023-03-03T14:41:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff37fe700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0004850 0x7f3fe0004cc0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f3fe400c270 tx=0x7f3fe0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835127ea7ce4db87 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4007c10 con 0x7f3fe0004850 2023-03-03T14:41:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe400cd90 con 0x7f3fe0004850 2023-03-03T14:41:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.074+0000 7f3ff2ffd700 1 -- 172.21.15.7:0/1466128025 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4005ba0 con 0x7f3fe0004850 2023-03-03T14:41:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.075+0000 7f3ffa292700 1 -- 172.21.15.7:0/1466128025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0004850 msgr2=0x7f3fe0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.075+0000 7f3ffa292700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0004850 0x7f3fe0004cc0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f3fe400c270 tx=0x7f3fe0005880 comp rx=0 tx=0).stop 2023-03-03T14:41:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.075+0000 7f3ffa292700 1 -- 172.21.15.7:0/1466128025 shutdown_connections 2023-03-03T14:41:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.075+0000 7f3ffa292700 1 --2- 172.21.15.7:0/1466128025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fe0004850 0x7f3fe0004cc0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.075+0000 7f3ffa292700 1 -- 172.21.15.7:0/1466128025 >> 172.21.15.7:0/1466128025 conn(0x7f3ff40736f0 msgr2=0x7f3ff4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:41:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.075+0000 7f3ffa292700 1 -- 172.21.15.7:0/1466128025 shutdown_connections 2023-03-03T14:41:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.075+0000 7f3ffa292700 1 -- 172.21.15.7:0/1466128025 wait complete. 2023-03-03T14:41:03.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.076+0000 7f3ffa292700 1 Processor -- start 2023-03-03T14:41:03.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.076+0000 7f3ffa292700 1 -- start start 2023-03-03T14:41:03.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ffa292700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe0004850 0x7f3ff41e8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:03.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ffa292700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ff4154650 0x7f3ff4153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:03.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ffa292700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ff40c75f0 0x7f3ff40ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:03.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ffa292700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff40ad630 con 0x7f3ff4154650 2023-03-03T14:41:03.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ffa292700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff4154420 con 0x7f3ff40c75f0 2023-03-03T14:41:03.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ffa292700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff41e7af0 con 0x7f3fe0004850 2023-03-03T14:41:03.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ff37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ff4154650 0x7f3ff4153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:03.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ff37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ff4154650 0x7f3ff4153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49868/0 (socket says 172.21.15.7:49868) 2023-03-03T14:41:03.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ff37fe700 1 -- 172.21.15.7:0/2524712339 learned_addr learned my addr 172.21.15.7:0/2524712339 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:41:03.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ff3fff700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe0004850 0x7f3ff41e8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:03.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.077+0000 7f3ff882f700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ff40c75f0 0x7f3ff40ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:03.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ff37fe700 1 -- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe0004850 msgr2=0x7f3ff41e8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:03.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ff37fe700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe0004850 0x7f3ff41e8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:03.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ff37fe700 1 -- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ff40c75f0 msgr2=0x7f3ff40ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:03.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ff37fe700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ff40c75f0 0x7f3ff40ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:03.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ff37fe700 1 -- 172.21.15.7:0/2524712339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff4153200 con 0x7f3ff4154650 2023-03-03T14:41:03.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ff37fe700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ff4154650 0x7f3ff4153ee0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000c160 tx=0x7f3ff40c3270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:03.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ff0ff9700 1 -- 172.21.15.7:0/2524712339 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4001640 con 0x7f3ff4154650 2023-03-03T14:41:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ff0ff9700 1 -- 172.21.15.7:0/2524712339 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe40070d0 con 0x7f3ff4154650 2023-03-03T14:41:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ffa292700 1 -- 172.21.15.7:0/2524712339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff4071c70 con 0x7f3ff4154650 2023-03-03T14:41:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ff0ff9700 1 -- 172.21.15.7:0/2524712339 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4007440 con 0x7f3ff4154650 2023-03-03T14:41:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.078+0000 7f3ffa292700 1 -- 172.21.15.7:0/2524712339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff4153390 con 0x7f3ff4154650 2023-03-03T14:41:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.079+0000 7f3ffa292700 1 -- 172.21.15.7:0/2524712339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff4074bb0 con 0x7f3ff4154650 2023-03-03T14:41:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.079+0000 7f3ff0ff9700 1 -- 172.21.15.7:0/2524712339 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3fe4024070 con 0x7f3ff4154650 2023-03-03T14:41:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.080+0000 7f3ff0ff9700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fd40682f0 0x7f3fd406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:03.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.080+0000 7f3ff0ff9700 1 -- 172.21.15.7:0/2524712339 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3fe40871d0 con 0x7f3ff4154650 2023-03-03T14:41:03.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.080+0000 7f3ff3fff700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fd40682f0 0x7f3fd406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:03.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.080+0000 7f3ff3fff700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fd40682f0 0x7f3fd406a7a0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3ff404f440 tx=0x7f3fdc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:03.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.084+0000 7f3ff0ff9700 1 -- 172.21.15.7:0/2524712339 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3fe4053570 con 0x7f3ff4154650 2023-03-03T14:41:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.323+0000 7f3ffa292700 1 -- 172.21.15.7:0/2524712339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ff4071a50 con 0x7f3ff4154650 2023-03-03T14:41:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.323+0000 7f3ff0ff9700 1 -- 172.21.15.7:0/2524712339 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3fe4007d70 con 0x7f3ff4154650 2023-03-03T14:41:03.325 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:41:03.325 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:41:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 -- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fd40682f0 msgr2=0x7f3fd406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fd40682f0 0x7f3fd406a7a0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3ff404f440 tx=0x7f3fdc008040 comp rx=0 tx=0).stop 2023-03-03T14:41:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 -- 172.21.15.7:0/2524712339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ff4154650 msgr2=0x7f3ff4153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ff4154650 0x7f3ff4153ee0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000c160 tx=0x7f3ff40c3270 comp rx=0 tx=0).stop 2023-03-03T14:41:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 -- 172.21.15.7:0/2524712339 shutdown_connections 2023-03-03T14:41:03.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ff40c75f0 0x7f3ff40ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fe0004850 0x7f3ff41e8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3fd40682f0 0x7f3fd406a7a0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 --2- 172.21.15.7:0/2524712339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ff4154650 0x7f3ff4153ee0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 -- 172.21.15.7:0/2524712339 >> 172.21.15.7:0/2524712339 conn(0x7f3ff40736f0 msgr2=0x7f3ff4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:41:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 -- 172.21.15.7:0/2524712339 shutdown_connections 2023-03-03T14:41:03.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:03.325+0000 7f3ffa292700 1 -- 172.21.15.7:0/2524712339 wait complete. 2023-03-03T14:41:03.329 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:41:03.339 INFO:tasks.mds_thrash.fs.[b]:waiting for 91 secs before thrashing 2023-03-03T14:41:25.890 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:25.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:46.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:41:46.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.321+0000 7f85f1ace700 1 Processor -- start 2023-03-03T14:41:46.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.321+0000 7f85f1ace700 1 -- start start 2023-03-03T14:41:46.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85f1ace700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f85ec14e560 0x7f85ec071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:46.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85f1ace700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec0c4850 0x7f85ec154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:46.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85f1ace700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec072920 con 0x7f85ec0c43d0 2023-03-03T14:41:46.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85f1ace700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec06bef0 con 0x7f85ec14e560 2023-03-03T14:41:46.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85f1ace700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec0c4cb0 con 0x7f85ec0c4850 2023-03-03T14:41:46.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85eaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f85ec0c43d0 0x7f85ec154590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43890/0 (socket says 172.21.15.7:43890) 2023-03-03T14:41:46.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85eaffd700 1 -- 172.21.15.7:0/1265949461 learned_addr learned my addr 172.21.15.7:0/1265949461 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:41:46.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504776711 0 0) 0x7f85e0001110 con 0x7f85ec0c43d0 2023-03-03T14:41:46.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85ebfff700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec0c4850 0x7f85ec154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:46.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d40035b0 con 0x7f85ec0c43d0 2023-03-03T14:41:46.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85eb7fe700 1 --2- 172.21.15.7:0/1265949461 >> v2:172.21.15.143:3300/0 conn(0x7f85ec14e560 0x7f85ec071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:46.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3004231746 0 0) 0x7f85d40035b0 con 0x7f85ec0c43d0 2023-03-03T14:41:46.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec0c4850 msgr2=0x7f85ec154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:41:46.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85ea7fc700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec0c4850 0x7f85ec154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 >> v2:172.21.15.143:3300/0 conn(0x7f85ec14e560 msgr2=0x7f85ec071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:41:46.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85ea7fc700 1 --2- 172.21.15.7:0/1265949461 >> v2:172.21.15.143:3300/0 conn(0x7f85ec14e560 0x7f85ec071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec1551e0 con 0x7f85ec0c43d0 2023-03-03T14:41:46.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.322+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f85e0001800 con 0x7f85ec0c43d0 2023-03-03T14:41:46.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 >> v1:172.21.15.7:6789/0 conn(0x7f85ec0c43d0 legacy=0x7f85ec154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:46.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ea7fc700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85d4003a70 0x7f85d4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:46.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ea7fc700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 0x7f85d4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:46.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ea7fc700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec0c4850 0x7f85d4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:46.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec06bef0 con 0x7f85d4004850 2023-03-03T14:41:46.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec0c4cb0 con 0x7f85d4003a70 2023-03-03T14:41:46.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d400ba10 con 0x7f85ec0c4850 2023-03-03T14:41:46.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85eaffd700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 0x7f85d4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:46.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85eb7fe700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85d4003a70 0x7f85d4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:46.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ebfff700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec0c4850 0x7f85d4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:46.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85eaffd700 1 -- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec0c4850 msgr2=0x7f85d4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:41:46.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85eaffd700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec0c4850 0x7f85d4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85eaffd700 1 -- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85d4003a70 msgr2=0x7f85d4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:41:46.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85eaffd700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85d4003a70 0x7f85d4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85eaffd700 1 -- 172.21.15.7:0/1265949461 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d40035b0 con 0x7f85d4004850 2023-03-03T14:41:46.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85eaffd700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 0x7f85d4004cc0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f85e000c240 tx=0x7f85d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f638b3e631d9ad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:46.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e0007db0 con 0x7f85d4004850 2023-03-03T14:41:46.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e000cde0 con 0x7f85d4004850 2023-03-03T14:41:46.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.323+0000 7f85ea7fc700 1 -- 172.21.15.7:0/1265949461 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e0005b80 con 0x7f85d4004850 2023-03-03T14:41:46.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.324+0000 7f85f1ace700 1 -- 172.21.15.7:0/1265949461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 msgr2=0x7f85d4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:46.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.324+0000 7f85f1ace700 1 --2- 172.21.15.7:0/1265949461 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 0x7f85d4004cc0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f85e000c240 tx=0x7f85d4005880 comp rx=0 tx=0).stop 2023-03-03T14:41:46.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.324+0000 7f85ebfff700 1 -- 172.21.15.7:0/1265949461 reap_dead start 2023-03-03T14:41:46.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.324+0000 7f85f1ace700 1 -- 172.21.15.7:0/1265949461 shutdown_connections 2023-03-03T14:41:46.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.324+0000 7f85f1ace700 1 -- 172.21.15.7:0/1265949461 >> 172.21.15.7:0/1265949461 conn(0x7f85ec0736f0 msgr2=0x7f85ec073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:41:46.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.324+0000 7f85f1ace700 1 -- 172.21.15.7:0/1265949461 shutdown_connections 2023-03-03T14:41:46.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.324+0000 7f85f1ace700 1 -- 172.21.15.7:0/1265949461 wait complete. 2023-03-03T14:41:46.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.325+0000 7f85f1ace700 1 Processor -- start 2023-03-03T14:41:46.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.326+0000 7f85f1ace700 1 -- start start 2023-03-03T14:41:46.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.326+0000 7f85f1ace700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 0x7f85ec0bec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:46.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.326+0000 7f85f1ace700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85ec157cb0 0x7f85ec148880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:46.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.326+0000 7f85f1ace700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec148dc0 0x7f85ec153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:46.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.326+0000 7f85f1ace700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec0ab460 con 0x7f85d4004850 2023-03-03T14:41:46.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.326+0000 7f85f1ace700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec0bf260 con 0x7f85ec157cb0 2023-03-03T14:41:46.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.326+0000 7f85f1ace700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec1524a0 con 0x7f85ec148dc0 2023-03-03T14:41:46.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.326+0000 7f85eb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 0x7f85ec0bec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:46.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.326+0000 7f85eb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 0x7f85ec0bec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49922/0 (socket says 172.21.15.7:49922) 2023-03-03T14:41:46.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.326+0000 7f85eb7fe700 1 -- 172.21.15.7:0/1253071836 learned_addr learned my addr 172.21.15.7:0/1253071836 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:41:46.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85eaffd700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85ec157cb0 0x7f85ec148880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:46.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85ebfff700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec148dc0 0x7f85ec153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:46.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85eb7fe700 1 -- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec148dc0 msgr2=0x7f85ec153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:46.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85eb7fe700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec148dc0 0x7f85ec153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85eb7fe700 1 -- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85ec157cb0 msgr2=0x7f85ec148880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:46.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85eb7fe700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85ec157cb0 0x7f85ec148880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85eb7fe700 1 -- 172.21.15.7:0/1253071836 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec0aacc0 con 0x7f85d4004850 2023-03-03T14:41:46.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85eb7fe700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 0x7f85ec0bec80 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f85dc011aa0 tx=0x7f85dc011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:46.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85d3fff700 1 -- 172.21.15.7:0/1253071836 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc011ee0 con 0x7f85d4004850 2023-03-03T14:41:46.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85d3fff700 1 -- 172.21.15.7:0/1253071836 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc00f940 con 0x7f85d4004850 2023-03-03T14:41:46.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85d3fff700 1 -- 172.21.15.7:0/1253071836 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc00fd10 con 0x7f85d4004850 2023-03-03T14:41:46.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85f1ace700 1 -- 172.21.15.7:0/1253071836 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec06b910 con 0x7f85d4004850 2023-03-03T14:41:46.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.327+0000 7f85f1ace700 1 -- 172.21.15.7:0/1253071836 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec06bb80 con 0x7f85d4004850 2023-03-03T14:41:46.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.328+0000 7f85f1ace700 1 -- 172.21.15.7:0/1253071836 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec074bb0 con 0x7f85d4004850 2023-03-03T14:41:46.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.329+0000 7f85d3fff700 1 -- 172.21.15.7:0/1253071836 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f85dc014440 con 0x7f85d4004850 2023-03-03T14:41:46.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.329+0000 7f85d3fff700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85d80682f0 0x7f85d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:41:46.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.329+0000 7f85d3fff700 1 -- 172.21.15.7:0/1253071836 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f85dc0868c0 con 0x7f85d4004850 2023-03-03T14:41:46.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.329+0000 7f85eaffd700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85d80682f0 0x7f85d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:41:46.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.330+0000 7f85eaffd700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85d80682f0 0x7f85d806a7a0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f85e0015040 tx=0x7f85e000a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:41:46.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.334+0000 7f85d3fff700 1 -- 172.21.15.7:0/1253071836 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f85dc052d70 con 0x7f85d4004850 2023-03-03T14:41:46.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.572+0000 7f85f1ace700 1 -- 172.21.15.7:0/1253071836 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85ec071a50 con 0x7f85d4004850 2023-03-03T14:41:46.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.573+0000 7f85d3fff700 1 -- 172.21.15.7:0/1253071836 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f85dc056390 con 0x7f85d4004850 2023-03-03T14:41:46.576 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:41:46.576 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:41:46.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 -- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85d80682f0 msgr2=0x7f85d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85d80682f0 0x7f85d806a7a0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f85e0015040 tx=0x7f85e000a000 comp rx=0 tx=0).stop 2023-03-03T14:41:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 -- 172.21.15.7:0/1253071836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 msgr2=0x7f85ec0bec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:41:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 0x7f85ec0bec80 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f85dc011aa0 tx=0x7f85dc011db0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 -- 172.21.15.7:0/1253071836 shutdown_connections 2023-03-03T14:41:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85ec148dc0 0x7f85ec153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85d80682f0 0x7f85d806a7a0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85d4004850 0x7f85ec0bec80 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 --2- 172.21.15.7:0/1253071836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f85ec157cb0 0x7f85ec148880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:41:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 -- 172.21.15.7:0/1253071836 >> 172.21.15.7:0/1253071836 conn(0x7f85ec0736f0 msgr2=0x7f85ec071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:41:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 -- 172.21.15.7:0/1253071836 shutdown_connections 2023-03-03T14:41:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:41:46.575+0000 7f85f1ace700 1 -- 172.21.15.7:0/1253071836 wait complete. 2023-03-03T14:41:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:41:46.589 INFO:tasks.mds_thrash.fs.[a]:waiting for 91 secs before thrashing 2023-03-03T14:41:55.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:41:55.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:26.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:26.028 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:34.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:42:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.477+0000 7f3a09f82700 1 Processor -- start 2023-03-03T14:42:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.477+0000 7f3a09f82700 1 -- start start 2023-03-03T14:42:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a09f82700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3a0414f9d0 0x7f3a04156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:42:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a09f82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a040abe20 0x7f3a041529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:42:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a09f82700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a040c2f30 con 0x7f3a0414fda0 2023-03-03T14:42:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a09f82700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a040c31d0 con 0x7f3a0414f9d0 2023-03-03T14:42:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a09f82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a041524a0 con 0x7f3a040abe20 2023-03-03T14:42:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a037fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3a0414f9d0 0x7f3a04156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:42:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a02ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3a0414fda0 0x7f3a04071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43936/0 (socket says 172.21.15.7:43936) 2023-03-03T14:42:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a02ffd700 1 -- 172.21.15.7:0/3837922180 learned_addr learned my addr 172.21.15.7:0/3837922180 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:42:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a03fff700 1 --2- 172.21.15.7:0/3837922180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a040abe20 0x7f3a041529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:42:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a037fe700 1 -- 172.21.15.7:0/3837922180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a040abe20 msgr2=0x7f3a041529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:42:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a037fe700 1 --2- 172.21.15.7:0/3837922180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a040abe20 0x7f3a041529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:42:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a037fe700 1 -- 172.21.15.7:0/3837922180 >> v1:172.21.15.7:6789/0 conn(0x7f3a0414fda0 legacy=0x7f3a04071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:42:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a037fe700 1 -- 172.21.15.7:0/3837922180 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a040c34b0 con 0x7f3a0414f9d0 2023-03-03T14:42:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a037fe700 1 --2- 172.21.15.7:0/3837922180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 0x7f3a04156430 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f39f4009840 tx=0x7f39f4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55a60eaabb831ce2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:42:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.478+0000 7f3a027fc700 1 -- 172.21.15.7:0/3837922180 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f4011830 con 0x7f3a0414f9d0 2023-03-03T14:42:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.479+0000 7f3a027fc700 1 -- 172.21.15.7:0/3837922180 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f4011990 con 0x7f3a0414f9d0 2023-03-03T14:42:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.479+0000 7f3a027fc700 1 -- 172.21.15.7:0/3837922180 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f4011d20 con 0x7f3a0414f9d0 2023-03-03T14:42:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.479+0000 7f3a09f82700 1 -- 172.21.15.7:0/3837922180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 msgr2=0x7f3a04156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:42:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.479+0000 7f3a09f82700 1 --2- 172.21.15.7:0/3837922180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 0x7f3a04156430 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f39f4009840 tx=0x7f39f4009b50 comp rx=0 tx=0).stop 2023-03-03T14:42:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.479+0000 7f3a09f82700 1 -- 172.21.15.7:0/3837922180 shutdown_connections 2023-03-03T14:42:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.479+0000 7f3a09f82700 1 --2- 172.21.15.7:0/3837922180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a040abe20 0x7f3a041529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:42:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.479+0000 7f3a09f82700 1 --2- 172.21.15.7:0/3837922180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 0x7f3a04156430 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:42:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.479+0000 7f3a09f82700 1 -- 172.21.15.7:0/3837922180 >> 172.21.15.7:0/3837922180 conn(0x7f3a040c8900 msgr2=0x7f3a040c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:42:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.479+0000 7f3a09f82700 1 -- 172.21.15.7:0/3837922180 shutdown_connections 2023-03-03T14:42:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.479+0000 7f3a09f82700 1 -- 172.21.15.7:0/3837922180 wait complete. 2023-03-03T14:42:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.481+0000 7f3a09f82700 1 Processor -- start 2023-03-03T14:42:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.481+0000 7f3a09f82700 1 -- start start 2023-03-03T14:42:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a09f82700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a040abe20 0x7f3a0414b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:42:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a09f82700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 0x7f3a041dea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:42:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a09f82700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a0414fda0 0x7f3a041defd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:42:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a09f82700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0406e4d0 con 0x7f3a0414f9d0 2023-03-03T14:42:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a09f82700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0406ea50 con 0x7f3a0414fda0 2023-03-03T14:42:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a09f82700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a04152150 con 0x7f3a040abe20 2023-03-03T14:42:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a02ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 0x7f3a041dea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:42:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a02ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 0x7f3a041dea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49964/0 (socket says 172.21.15.7:49964) 2023-03-03T14:42:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a02ffd700 1 -- 172.21.15.7:0/377861735 learned_addr learned my addr 172.21.15.7:0/377861735 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:42:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a037fe700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a040abe20 0x7f3a0414b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:42:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a03fff700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a0414fda0 0x7f3a041defd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:42:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a02ffd700 1 -- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a040abe20 msgr2=0x7f3a0414b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:42:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a02ffd700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a040abe20 0x7f3a0414b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:42:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a02ffd700 1 -- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a0414fda0 msgr2=0x7f3a041defd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:42:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a02ffd700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a0414fda0 0x7f3a041defd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:42:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.482+0000 7f3a02ffd700 1 -- 172.21.15.7:0/377861735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f40094a0 con 0x7f3a0414f9d0 2023-03-03T14:42:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.483+0000 7f3a02ffd700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 0x7f3a041dea90 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f39f800c500 tx=0x7f39f800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:42:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.483+0000 7f39ebfff700 1 -- 172.21.15.7:0/377861735 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f8005d70 con 0x7f3a0414f9d0 2023-03-03T14:42:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.483+0000 7f39ebfff700 1 -- 172.21.15.7:0/377861735 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f80070d0 con 0x7f3a0414f9d0 2023-03-03T14:42:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.483+0000 7f3a09f82700 1 -- 172.21.15.7:0/377861735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a040ac350 con 0x7f3a0414f9d0 2023-03-03T14:42:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.483+0000 7f39ebfff700 1 -- 172.21.15.7:0/377861735 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f8007440 con 0x7f3a0414f9d0 2023-03-03T14:42:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.483+0000 7f3a09f82700 1 -- 172.21.15.7:0/377861735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a040ab630 con 0x7f3a0414f9d0 2023-03-03T14:42:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.484+0000 7f3a09f82700 1 -- 172.21.15.7:0/377861735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a04071fa0 con 0x7f3a0414f9d0 2023-03-03T14:42:34.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.484+0000 7f39ebfff700 1 -- 172.21.15.7:0/377861735 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f39f8019030 con 0x7f3a0414f9d0 2023-03-03T14:42:34.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.485+0000 7f39ebfff700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39f00682f0 0x7f39f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:42:34.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.485+0000 7f39ebfff700 1 -- 172.21.15.7:0/377861735 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f39f80852c0 con 0x7f3a0414f9d0 2023-03-03T14:42:34.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.485+0000 7f3a037fe700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39f00682f0 0x7f39f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:42:34.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.485+0000 7f3a037fe700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39f00682f0 0x7f39f006a7a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f39f4009790 tx=0x7f39f4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:42:34.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.489+0000 7f39ebfff700 1 -- 172.21.15.7:0/377861735 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f39f804b5e0 con 0x7f3a0414f9d0 2023-03-03T14:42:34.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.726+0000 7f3a09f82700 1 -- 172.21.15.7:0/377861735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a04074bb0 con 0x7f3a0414f9d0 2023-03-03T14:42:34.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.727+0000 7f39ebfff700 1 -- 172.21.15.7:0/377861735 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f39f8051710 con 0x7f3a0414f9d0 2023-03-03T14:42:34.729 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:42:34.729 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:42:34.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 -- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39f00682f0 msgr2=0x7f39f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:42:34.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39f00682f0 0x7f39f006a7a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f39f4009790 tx=0x7f39f4008030 comp rx=0 tx=0).stop 2023-03-03T14:42:34.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 -- 172.21.15.7:0/377861735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 msgr2=0x7f3a041dea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:42:34.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 0x7f3a041dea90 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f39f800c500 tx=0x7f39f800c810 comp rx=0 tx=0).stop 2023-03-03T14:42:34.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 -- 172.21.15.7:0/377861735 shutdown_connections 2023-03-03T14:42:34.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a0414fda0 0x7f3a041defd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:42:34.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a040abe20 0x7f3a0414b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:42:34.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39f00682f0 0x7f39f006a7a0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:42:34.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 --2- 172.21.15.7:0/377861735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a0414f9d0 0x7f3a041dea90 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:42:34.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 -- 172.21.15.7:0/377861735 >> 172.21.15.7:0/377861735 conn(0x7f3a040c8900 msgr2=0x7f3a040c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:42:34.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 -- 172.21.15.7:0/377861735 shutdown_connections 2023-03-03T14:42:34.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:42:34.729+0000 7f3a09f82700 1 -- 172.21.15.7:0/377861735 wait complete. 2023-03-03T14:42:34.732 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:42:34.742 INFO:tasks.mds_thrash.fs.[b]:waiting for 69 secs before thrashing 2023-03-03T14:42:56.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:42:56.094 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:17.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:43:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.729+0000 7fdcc4495700 1 Processor -- start 2023-03-03T14:43:17.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.729+0000 7fdcc4495700 1 -- start start 2023-03-03T14:43:17.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc4495700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 0x7fdcbc0c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc4495700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdcbc0c4910 0x7fdcbc156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc4495700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcbc1523b0 con 0x7fdcbc0beab0 2023-03-03T14:43:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc4495700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcbc072010 con 0x7fdcbc0c4910 2023-03-03T14:43:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc4495700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc0aaaf0 con 0x7fdcbc152e30 2023-03-03T14:43:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc2231700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 0x7fdcbc0c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc2a32700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdcbc0c4910 0x7fdcbc156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc2a32700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdcbc0c4910 0x7fdcbc156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:45784/0 (socket says 172.21.15.7:45784) 2023-03-03T14:43:17.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc2231700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 0x7fdcbc0c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:58078/0 (socket says 172.21.15.7:58078) 2023-03-03T14:43:17.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc1a30700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fdcbc0beab0 0x7fdcbc155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:47506/0 (socket says 172.21.15.7:47506) 2023-03-03T14:43:17.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc2a32700 1 -- 172.21.15.7:0/179432482 learned_addr learned my addr 172.21.15.7:0/179432482 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:43:17.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc2a32700 1 -- 172.21.15.7:0/179432482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 msgr2=0x7fdcbc0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:43:17.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc2a32700 1 --2- 172.21.15.7:0/179432482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 0x7fdcbc0c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc2a32700 1 -- 172.21.15.7:0/179432482 >> v1:172.21.15.143:6789/0 conn(0x7fdcbc0beab0 legacy=0x7fdcbc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc2a32700 1 -- 172.21.15.7:0/179432482 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcbc0aadd0 con 0x7fdcbc0c4910 2023-03-03T14:43:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.730+0000 7fdcc2231700 1 --2- 172.21.15.7:0/179432482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 0x7fdcbc0c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:43:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc2a32700 1 --2- 172.21.15.7:0/179432482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcbc0c4910 0x7fdcbc156490 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009e50 tx=0x7fdcb400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d0c09e36e8ce43 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc122f700 1 -- 172.21.15.7:0/179432482 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb400e070 con 0x7fdcbc0c4910 2023-03-03T14:43:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc122f700 1 -- 172.21.15.7:0/179432482 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb400b910 con 0x7fdcbc0c4910 2023-03-03T14:43:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc122f700 1 -- 172.21.15.7:0/179432482 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb400bcc0 con 0x7fdcbc0c4910 2023-03-03T14:43:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc4495700 1 -- 172.21.15.7:0/179432482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcbc0c4910 msgr2=0x7fdcbc156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc4495700 1 --2- 172.21.15.7:0/179432482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcbc0c4910 0x7fdcbc156490 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4009e50 tx=0x7fdcb400b330 comp rx=0 tx=0).stop 2023-03-03T14:43:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc4495700 1 -- 172.21.15.7:0/179432482 shutdown_connections 2023-03-03T14:43:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc4495700 1 --2- 172.21.15.7:0/179432482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcbc0c4910 0x7fdcbc156490 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc4495700 1 --2- 172.21.15.7:0/179432482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 0x7fdcbc0c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc4495700 1 -- 172.21.15.7:0/179432482 >> 172.21.15.7:0/179432482 conn(0x7fdcbc0736f0 msgr2=0x7fdcbc073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:43:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc4495700 1 -- 172.21.15.7:0/179432482 shutdown_connections 2023-03-03T14:43:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.731+0000 7fdcc4495700 1 -- 172.21.15.7:0/179432482 wait complete. 2023-03-03T14:43:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.733+0000 7fdcc4495700 1 Processor -- start 2023-03-03T14:43:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.733+0000 7fdcc4495700 1 -- start start 2023-03-03T14:43:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.733+0000 7fdcc4495700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcbc0beab0 0x7fdcbc1ea030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.733+0000 7fdcc4495700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcbc0c4910 0x7fdcbc0c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.733+0000 7fdcc4495700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 0x7fdcbc0c1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.733+0000 7fdcc4495700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc0ad420 con 0x7fdcbc0beab0 2023-03-03T14:43:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.733+0000 7fdcc4495700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc1ea570 con 0x7fdcbc0c4910 2023-03-03T14:43:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.733+0000 7fdcc4495700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc155200 con 0x7fdcbc152e30 2023-03-03T14:43:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.733+0000 7fdcc2231700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcbc0beab0 0x7fdcbc1ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.733+0000 7fdcc2231700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcbc0beab0 0x7fdcbc1ea030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50012/0 (socket says 172.21.15.7:50012) 2023-03-03T14:43:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc2231700 1 -- 172.21.15.7:0/1708087004 learned_addr learned my addr 172.21.15.7:0/1708087004 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:43:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc1a30700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcbc0c4910 0x7fdcbc0c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc2a32700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 0x7fdcbc0c1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc2231700 1 -- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 msgr2=0x7fdcbc0c1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc2231700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 0x7fdcbc0c1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc2231700 1 -- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcbc0c4910 msgr2=0x7fdcbc0c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc2231700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcbc0c4910 0x7fdcbc0c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc2231700 1 -- 172.21.15.7:0/1708087004 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb80094a0 con 0x7fdcbc0beab0 2023-03-03T14:43:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc2231700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcbc0beab0 0x7fdcbc1ea030 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800ebe0 tx=0x7fdcb8009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcb2ffd700 1 -- 172.21.15.7:0/1708087004 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb800c910 con 0x7fdcbc0beab0 2023-03-03T14:43:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcb2ffd700 1 -- 172.21.15.7:0/1708087004 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb800ca70 con 0x7fdcbc0beab0 2023-03-03T14:43:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc4495700 1 -- 172.21.15.7:0/1708087004 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb4009b30 con 0x7fdcbc0beab0 2023-03-03T14:43:17.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcb2ffd700 1 -- 172.21.15.7:0/1708087004 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb800cde0 con 0x7fdcbc0beab0 2023-03-03T14:43:17.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.734+0000 7fdcc4495700 1 -- 172.21.15.7:0/1708087004 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcbc153200 con 0x7fdcbc0beab0 2023-03-03T14:43:17.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.735+0000 7fdcc4495700 1 -- 172.21.15.7:0/1708087004 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcbc153390 con 0x7fdcbc0beab0 2023-03-03T14:43:17.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.736+0000 7fdcb2ffd700 1 -- 172.21.15.7:0/1708087004 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdcb8019030 con 0x7fdcbc0beab0 2023-03-03T14:43:17.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.736+0000 7fdcb2ffd700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdca00682f0 0x7fdca006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:17.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.736+0000 7fdcb2ffd700 1 -- 172.21.15.7:0/1708087004 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdcb8086260 con 0x7fdcbc0beab0 2023-03-03T14:43:17.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.736+0000 7fdcc1a30700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdca00682f0 0x7fdca006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:17.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.737+0000 7fdcc1a30700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdca00682f0 0x7fdca006a7a0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc156110 tx=0x7fdcac007d70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:17.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.741+0000 7fdcb2ffd700 1 -- 172.21.15.7:0/1708087004 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdcb8052580 con 0x7fdcbc0beab0 2023-03-03T14:43:17.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.981+0000 7fdcc4495700 1 -- 172.21.15.7:0/1708087004 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcbc074bb0 con 0x7fdcbc0beab0 2023-03-03T14:43:17.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.982+0000 7fdcb2ffd700 1 -- 172.21.15.7:0/1708087004 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdcb8055ba0 con 0x7fdcbc0beab0 2023-03-03T14:43:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:43:17.984 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:43:17.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.984+0000 7fdcc4495700 1 -- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdca00682f0 msgr2=0x7fdca006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:17.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.984+0000 7fdcc4495700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdca00682f0 0x7fdca006a7a0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc156110 tx=0x7fdcac007d70 comp rx=0 tx=0).stop 2023-03-03T14:43:17.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.984+0000 7fdcc4495700 1 -- 172.21.15.7:0/1708087004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcbc0beab0 msgr2=0x7fdcbc1ea030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.984+0000 7fdcc4495700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcbc0beab0 0x7fdcbc1ea030 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800ebe0 tx=0x7fdcb8009ec0 comp rx=0 tx=0).stop 2023-03-03T14:43:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.984+0000 7fdcc4495700 1 -- 172.21.15.7:0/1708087004 shutdown_connections 2023-03-03T14:43:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.984+0000 7fdcc4495700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdcbc152e30 0x7fdcbc0c1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.984+0000 7fdcc4495700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdca00682f0 0x7fdca006a7a0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.984+0000 7fdcc4495700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdcbc0beab0 0x7fdcbc1ea030 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.984+0000 7fdcc4495700 1 --2- 172.21.15.7:0/1708087004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcbc0c4910 0x7fdcbc0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.984+0000 7fdcc4495700 1 -- 172.21.15.7:0/1708087004 >> 172.21.15.7:0/1708087004 conn(0x7fdcbc0736f0 msgr2=0x7fdcbc14f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:43:17.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.985+0000 7fdcc4495700 1 -- 172.21.15.7:0/1708087004 shutdown_connections 2023-03-03T14:43:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:17.985+0000 7fdcc4495700 1 -- 172.21.15.7:0/1708087004 wait complete. 2023-03-03T14:43:17.988 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:43:17.998 INFO:tasks.mds_thrash.fs.[a]:waiting for 108 secs before thrashing 2023-03-03T14:43:26.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:26.163 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:43.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:43:43.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.881+0000 7feba679e700 1 Processor -- start 2023-03-03T14:43:43.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.881+0000 7feba679e700 1 -- start start 2023-03-03T14:43:43.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba679e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feba00c20d0 0x7feba01562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:43.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba679e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feba00c27c0 0x7feba00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:43.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba679e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba0155e60 con 0x7feba014f860 2023-03-03T14:43:43.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba679e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba006d350 con 0x7feba00c27c0 2023-03-03T14:43:43.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba679e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba01558e0 con 0x7feba00c20d0 2023-03-03T14:43:43.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba4d3b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feba00c27c0 0x7feba00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:43.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba4d3b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feba00c27c0 0x7feba00736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50044/0 (socket says 172.21.15.7:50044) 2023-03-03T14:43:43.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba4d3b700 1 -- 172.21.15.7:0/2582177966 learned_addr learned my addr 172.21.15.7:0/2582177966 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:43:43.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feb9f7fe700 1 --2- 172.21.15.7:0/2582177966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feba00c20d0 0x7feba01562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:43.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba4d3b700 1 -- 172.21.15.7:0/2582177966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feba00c20d0 msgr2=0x7feba01562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:43:43.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba4d3b700 1 --2- 172.21.15.7:0/2582177966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feba00c20d0 0x7feba01562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:43.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba4d3b700 1 -- 172.21.15.7:0/2582177966 >> v1:172.21.15.143:6789/0 conn(0x7feba014f860 legacy=0x7feba0152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:43.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba4d3b700 1 -- 172.21.15.7:0/2582177966 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba006cd90 con 0x7feba00c27c0 2023-03-03T14:43:43.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.882+0000 7feba4d3b700 1 --2- 172.21.15.7:0/2582177966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 0x7feba00736f0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7feb98009e50 tx=0x7feb9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f635c898b2d336 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:43.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feb9effd700 1 -- 172.21.15.7:0/2582177966 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9800e070 con 0x7feba00c27c0 2023-03-03T14:43:43.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feb9effd700 1 -- 172.21.15.7:0/2582177966 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9800b910 con 0x7feba00c27c0 2023-03-03T14:43:43.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feb9effd700 1 -- 172.21.15.7:0/2582177966 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9800bcc0 con 0x7feba00c27c0 2023-03-03T14:43:43.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feba679e700 1 -- 172.21.15.7:0/2582177966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 msgr2=0x7feba00736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:43.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feba679e700 1 --2- 172.21.15.7:0/2582177966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 0x7feba00736f0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7feb98009e50 tx=0x7feb9800b330 comp rx=0 tx=0).stop 2023-03-03T14:43:43.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feba679e700 1 -- 172.21.15.7:0/2582177966 shutdown_connections 2023-03-03T14:43:43.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feba679e700 1 --2- 172.21.15.7:0/2582177966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 0x7feba00736f0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:43.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feba679e700 1 --2- 172.21.15.7:0/2582177966 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feba00c20d0 0x7feba01562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:43.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feba679e700 1 -- 172.21.15.7:0/2582177966 >> 172.21.15.7:0/2582177966 conn(0x7feba00a94d0 msgr2=0x7feba00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:43:43.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feba679e700 1 -- 172.21.15.7:0/2582177966 shutdown_connections 2023-03-03T14:43:43.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.883+0000 7feba679e700 1 -- 172.21.15.7:0/2582177966 wait complete. 2023-03-03T14:43:43.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.884+0000 7feba679e700 1 Processor -- start 2023-03-03T14:43:43.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.885+0000 7feba679e700 1 -- start start 2023-03-03T14:43:43.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.885+0000 7feba679e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feba00c20d0 0x7feba014cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:43.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.885+0000 7feba679e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 0x7feba0153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:43.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.885+0000 7feba679e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feba014f860 0x7feba00bd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:43.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.885+0000 7feba679e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba01ec770 con 0x7feba00c27c0 2023-03-03T14:43:43.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.885+0000 7feba679e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba00ae650 con 0x7feba014f860 2023-03-03T14:43:43.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.885+0000 7feba679e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba01ebbf0 con 0x7feba00c20d0 2023-03-03T14:43:43.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.885+0000 7feb9f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 0x7feba0153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:43.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 0x7feba0153da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50048/0 (socket says 172.21.15.7:50048) 2023-03-03T14:43:43.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9f7fe700 1 -- 172.21.15.7:0/1497933393 learned_addr learned my addr 172.21.15.7:0/1497933393 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:43:43.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feba4d3b700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feba014f860 0x7feba00bd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:43.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9ffff700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feba00c20d0 0x7feba014cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:43.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9f7fe700 1 -- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feba00c20d0 msgr2=0x7feba014cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:43.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9f7fe700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feba00c20d0 0x7feba014cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:43.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9f7fe700 1 -- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feba014f860 msgr2=0x7feba00bd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:43.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9f7fe700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feba014f860 0x7feba00bd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:43.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9f7fe700 1 -- 172.21.15.7:0/1497933393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98009b30 con 0x7feba00c27c0 2023-03-03T14:43:43.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feba4d3b700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feba014f860 0x7feba00bd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:43:43.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9f7fe700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 0x7feba0153da0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7feb9400e9b0 tx=0x7feb9400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:43.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9cff9700 1 -- 172.21.15.7:0/1497933393 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94009710 con 0x7feba00c27c0 2023-03-03T14:43:43.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9cff9700 1 -- 172.21.15.7:0/1497933393 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9400c850 con 0x7feba00c27c0 2023-03-03T14:43:43.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feba679e700 1 -- 172.21.15.7:0/1497933393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba0155af0 con 0x7feba00c27c0 2023-03-03T14:43:43.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feb9cff9700 1 -- 172.21.15.7:0/1497933393 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9400cbc0 con 0x7feba00c27c0 2023-03-03T14:43:43.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.886+0000 7feba679e700 1 -- 172.21.15.7:0/1497933393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba006d070 con 0x7feba00c27c0 2023-03-03T14:43:43.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.887+0000 7feba679e700 1 -- 172.21.15.7:0/1497933393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba014fc30 con 0x7feba00c27c0 2023-03-03T14:43:43.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.888+0000 7feb9cff9700 1 -- 172.21.15.7:0/1497933393 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feb94019030 con 0x7feba00c27c0 2023-03-03T14:43:43.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.888+0000 7feb9cff9700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb8c0682f0 0x7feb8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:43.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.888+0000 7feb9cff9700 1 -- 172.21.15.7:0/1497933393 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7feb940861e0 con 0x7feba00c27c0 2023-03-03T14:43:43.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.888+0000 7feb9ffff700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb8c0682f0 0x7feb8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:43.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.889+0000 7feb9ffff700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb8c0682f0 0x7feb8c06a7a0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7feb90007fd0 tx=0x7feb90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:43.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:43.892+0000 7feb9cff9700 1 -- 172.21.15.7:0/1497933393 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb94052500 con 0x7feba00c27c0 2023-03-03T14:43:44.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.132+0000 7feba679e700 1 -- 172.21.15.7:0/1497933393 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feba0074bb0 con 0x7feba00c27c0 2023-03-03T14:43:44.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.132+0000 7feb9cff9700 1 -- 172.21.15.7:0/1497933393 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7feb94055b20 con 0x7feba00c27c0 2023-03-03T14:43:44.134 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:43:44.134 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 -- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb8c0682f0 msgr2=0x7feb8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb8c0682f0 0x7feb8c06a7a0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7feb90007fd0 tx=0x7feb90009040 comp rx=0 tx=0).stop 2023-03-03T14:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 -- 172.21.15.7:0/1497933393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 msgr2=0x7feba0153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 0x7feba0153da0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7feb9400e9b0 tx=0x7feb9400ed70 comp rx=0 tx=0).stop 2023-03-03T14:43:44.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 -- 172.21.15.7:0/1497933393 shutdown_connections 2023-03-03T14:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feba014f860 0x7feba00bd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feba00c20d0 0x7feba014cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb8c0682f0 0x7feb8c06a7a0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 --2- 172.21.15.7:0/1497933393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feba00c27c0 0x7feba0153da0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 -- 172.21.15.7:0/1497933393 >> 172.21.15.7:0/1497933393 conn(0x7feba00a94d0 msgr2=0x7feba00c51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 -- 172.21.15.7:0/1497933393 shutdown_connections 2023-03-03T14:43:44.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:44.134+0000 7feba679e700 1 -- 172.21.15.7:0/1497933393 wait complete. 2023-03-03T14:43:44.138 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:43:44.148 INFO:tasks.mds_thrash.fs.[b]:waiting for 12 secs before thrashing 2023-03-03T14:43:56.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:43:56.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:56.226 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:43:56.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.289+0000 7f0d5178b700 1 Processor -- start 2023-03-03T14:43:56.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.289+0000 7f0d5178b700 1 -- start start 2023-03-03T14:43:56.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d5178b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0d4c0c7a30 0x7f0d4c14e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:56.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d5178b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c0c7e00 0x7f0d4c14e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d5178b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d4c157bb0 con 0x7f0d4c06b6f0 2023-03-03T14:43:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d5178b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4c06ab00 con 0x7f0d4c0c7a30 2023-03-03T14:43:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d5178b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c1545c0 con 0x7f0d4c0c7e00 2023-03-03T14:43:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d4b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0d4c06b6f0 0x7f0d4c154a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44042/0 (socket says 172.21.15.7:44042) 2023-03-03T14:43:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d4b7fe700 1 -- 172.21.15.7:0/2362469671 learned_addr learned my addr 172.21.15.7:0/2362469671 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:43:56.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d4a7fc700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c0c7e00 0x7f0d4c14e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:56.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d4affd700 1 --2- 172.21.15.7:0/2362469671 >> v2:172.21.15.143:3300/0 conn(0x7f0d4c0c7a30 0x7f0d4c14e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072892318 0 0) 0x7f0d4c157bb0 con 0x7f0d4c06b6f0 2023-03-03T14:43:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.290+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d380035b0 con 0x7f0d4c06b6f0 2023-03-03T14:43:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3686572646 0 0) 0x7f0d380035b0 con 0x7f0d4c06b6f0 2023-03-03T14:43:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c0c7e00 msgr2=0x7f0d4c14e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:43:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c0c7e00 0x7f0d4c14e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 >> v2:172.21.15.143:3300/0 conn(0x7f0d4c0c7a30 msgr2=0x7f0d4c14e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:43:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 --2- 172.21.15.7:0/2362469671 >> v2:172.21.15.143:3300/0 conn(0x7f0d4c0c7a30 0x7f0d4c14e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c06a2f0 con 0x7f0d4c06b6f0 2023-03-03T14:43:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0d440019f0 con 0x7f0d4c06b6f0 2023-03-03T14:43:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 >> v1:172.21.15.7:6789/0 conn(0x7f0d4c06b6f0 legacy=0x7f0d4c154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c0c7e00 0x7f0d38003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d38003ff0 0x7f0d38004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d38004d90 0x7f0d38005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:56.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c06ab00 con 0x7f0d38003ff0 2023-03-03T14:43:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c1545c0 con 0x7f0d38004d90 2023-03-03T14:43:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3800bac0 con 0x7f0d4c0c7e00 2023-03-03T14:43:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4affd700 1 --2- 172.21.15.7:0/2362469671 >> v2:172.21.15.143:3300/0 conn(0x7f0d4c0c7a30 0x7f0d4c14e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:43:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4a7fc700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d38003ff0 0x7f0d38004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4affd700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c0c7e00 0x7f0d38003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4b7fe700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d38004d90 0x7f0d38005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:56.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4a7fc700 1 -- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c0c7e00 msgr2=0x7f0d38003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:43:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4a7fc700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c0c7e00 0x7f0d38003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4a7fc700 1 -- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d38004d90 msgr2=0x7f0d38005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:43:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4a7fc700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d38004d90 0x7f0d38005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4a7fc700 1 -- 172.21.15.7:0/2362469671 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d380035b0 con 0x7f0d38003ff0 2023-03-03T14:43:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4affd700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c0c7e00 0x7f0d38003a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:43:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d4a7fc700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d38003ff0 0x7f0d38004850 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c009b30 tx=0x7f0d38005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a61a6daf8af510 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.291+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c00e6c0 con 0x7f0d38003ff0 2023-03-03T14:43:56.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.292+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c00e820 con 0x7f0d38003ff0 2023-03-03T14:43:56.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.292+0000 7f0d49ffb700 1 -- 172.21.15.7:0/2362469671 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c00eb90 con 0x7f0d38003ff0 2023-03-03T14:43:56.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.292+0000 7f0d5178b700 1 -- 172.21.15.7:0/2362469671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d38003ff0 msgr2=0x7f0d38004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.292+0000 7f0d5178b700 1 --2- 172.21.15.7:0/2362469671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d38003ff0 0x7f0d38004850 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c009b30 tx=0x7f0d38005880 comp rx=0 tx=0).stop 2023-03-03T14:43:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.292+0000 7f0d4b7fe700 1 -- 172.21.15.7:0/2362469671 reap_dead start 2023-03-03T14:43:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.292+0000 7f0d5178b700 1 -- 172.21.15.7:0/2362469671 shutdown_connections 2023-03-03T14:43:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.292+0000 7f0d5178b700 1 -- 172.21.15.7:0/2362469671 >> 172.21.15.7:0/2362469671 conn(0x7f0d4c073d20 msgr2=0x7f0d4c0716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:43:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.292+0000 7f0d5178b700 1 -- 172.21.15.7:0/2362469671 shutdown_connections 2023-03-03T14:43:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.292+0000 7f0d5178b700 1 -- 172.21.15.7:0/2362469671 wait complete. 2023-03-03T14:43:56.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d5178b700 1 Processor -- start 2023-03-03T14:43:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d5178b700 1 -- start start 2023-03-03T14:43:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d5178b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c14c1a0 0x7f0d4c148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d5178b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d4c14c570 0x7f0d4c148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d5178b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d4c14ec80 0x7f0d4c150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d5178b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c14fa10 con 0x7f0d4c14c570 2023-03-03T14:43:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d5178b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c1502b0 con 0x7f0d4c14ec80 2023-03-03T14:43:56.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d5178b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c0ca4b0 con 0x7f0d4c14c1a0 2023-03-03T14:43:56.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d4a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d4c14c570 0x7f0d4c148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:56.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d4a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d4c14c570 0x7f0d4c148870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50072/0 (socket says 172.21.15.7:50072) 2023-03-03T14:43:56.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.294+0000 7f0d4a7fc700 1 -- 172.21.15.7:0/3002863351 learned_addr learned my addr 172.21.15.7:0/3002863351 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:43:56.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d4affd700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c14c1a0 0x7f0d4c148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:56.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d4b7fe700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d4c14ec80 0x7f0d4c150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:56.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d4a7fc700 1 -- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c14c1a0 msgr2=0x7f0d4c148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:56.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d4a7fc700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c14c1a0 0x7f0d4c148330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d4a7fc700 1 -- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d4c14ec80 msgr2=0x7f0d4c150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:56.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d4a7fc700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d4c14ec80 0x7f0d4c150740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d4a7fc700 1 -- 172.21.15.7:0/3002863351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c06a540 con 0x7f0d4c14c570 2023-03-03T14:43:56.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d4b7fe700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d4c14ec80 0x7f0d4c150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:43:56.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d4a7fc700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d4c14c570 0x7f0d4c148870 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800ba10 tx=0x7f0d3c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:56.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d437fe700 1 -- 172.21.15.7:0/3002863351 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c00e6c0 con 0x7f0d4c14c570 2023-03-03T14:43:56.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d437fe700 1 -- 172.21.15.7:0/3002863351 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c003ab0 con 0x7f0d4c14c570 2023-03-03T14:43:56.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d5178b700 1 -- 172.21.15.7:0/3002863351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c0bfbb0 con 0x7f0d4c14c570 2023-03-03T14:43:56.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d437fe700 1 -- 172.21.15.7:0/3002863351 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c003e60 con 0x7f0d4c14c570 2023-03-03T14:43:56.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.295+0000 7f0d5178b700 1 -- 172.21.15.7:0/3002863351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c06a820 con 0x7f0d4c14c570 2023-03-03T14:43:56.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.296+0000 7f0d5178b700 1 -- 172.21.15.7:0/3002863351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4c150420 con 0x7f0d4c14c570 2023-03-03T14:43:56.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.297+0000 7f0d437fe700 1 -- 172.21.15.7:0/3002863351 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0d3c025070 con 0x7f0d4c14c570 2023-03-03T14:43:56.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.297+0000 7f0d437fe700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d2c0682f0 0x7f0d2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:56.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.297+0000 7f0d437fe700 1 -- 172.21.15.7:0/3002863351 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0d3c087900 con 0x7f0d4c14c570 2023-03-03T14:43:56.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.297+0000 7f0d4affd700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d2c0682f0 0x7f0d2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:56.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.298+0000 7f0d4affd700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d2c0682f0 0x7f0d2c06a7a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009de0 tx=0x7f0d34008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:56.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.302+0000 7f0d437fe700 1 -- 172.21.15.7:0/3002863351 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0d3c053ca0 con 0x7f0d4c14c570 2023-03-03T14:43:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.542+0000 7f0d5178b700 1 -- 172.21.15.7:0/3002863351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d4c074bb0 con 0x7f0d4c14c570 2023-03-03T14:43:56.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.542+0000 7f0d437fe700 1 -- 172.21.15.7:0/3002863351 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0d3c0572c0 con 0x7f0d4c14c570 2023-03-03T14:43:56.545 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:43:56.546 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:43:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 -- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d2c0682f0 msgr2=0x7f0d2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d2c0682f0 0x7f0d2c06a7a0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009de0 tx=0x7f0d34008040 comp rx=0 tx=0).stop 2023-03-03T14:43:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 -- 172.21.15.7:0/3002863351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d4c14c570 msgr2=0x7f0d4c148870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d4c14c570 0x7f0d4c148870 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800ba10 tx=0x7f0d3c0047c0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 -- 172.21.15.7:0/3002863351 shutdown_connections 2023-03-03T14:43:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0d4c14ec80 0x7f0d4c150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0d4c14c1a0 0x7f0d4c148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0d2c0682f0 0x7f0d2c06a7a0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 --2- 172.21.15.7:0/3002863351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d4c14c570 0x7f0d4c148870 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 -- 172.21.15.7:0/3002863351 >> 172.21.15.7:0/3002863351 conn(0x7f0d38003ff0 msgr2=0x7f0d4c152d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:43:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 -- 172.21.15.7:0/3002863351 shutdown_connections 2023-03-03T14:43:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:56.544+0000 7f0d5178b700 1 -- 172.21.15.7:0/3002863351 wait complete. 2023-03-03T14:43:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:43:56.559 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before thrashing 2023-03-03T14:43:58.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:43:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.696+0000 7fa339d73700 1 Processor -- start 2023-03-03T14:43:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.697+0000 7fa339d73700 1 -- start start 2023-03-03T14:43:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.697+0000 7fa339d73700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa334071e80 0x7fa334151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.697+0000 7fa339d73700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa334072250 0x7fa33414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.697+0000 7fa339d73700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa3340c1ce0 0x7fa33414e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.697+0000 7fa339d73700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3341510d0 con 0x7fa3340c1ce0 2023-03-03T14:43:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.697+0000 7fa339d73700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3340c3740 con 0x7fa334071e80 2023-03-03T14:43:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.697+0000 7fa339d73700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa334070670 con 0x7fa334072250 2023-03-03T14:43:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.697+0000 7fa333fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa3340c1ce0 0x7fa33414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa333fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa3340c1ce0 0x7fa33414e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50082/0 (socket says 172.21.15.7:50082) 2023-03-03T14:43:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa333fff700 1 -- 172.21.15.7:0/2267184839 learned_addr learned my addr 172.21.15.7:0/2267184839 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:43:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa3337fe700 1 --2- 172.21.15.7:0/2267184839 >> v2:172.21.15.143:3300/0 conn(0x7fa334071e80 0x7fa334151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa332ffd700 1 --2- 172.21.15.7:0/2267184839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa334072250 0x7fa33414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa333fff700 1 -- 172.21.15.7:0/2267184839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa334072250 msgr2=0x7fa33414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:43:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa333fff700 1 --2- 172.21.15.7:0/2267184839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa334072250 0x7fa33414e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa333fff700 1 -- 172.21.15.7:0/2267184839 >> v2:172.21.15.143:3300/0 conn(0x7fa334071e80 msgr2=0x7fa334151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:43:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa333fff700 1 --2- 172.21.15.7:0/2267184839 >> v2:172.21.15.143:3300/0 conn(0x7fa334071e80 0x7fa334151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa333fff700 1 -- 172.21.15.7:0/2267184839 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3340c39e0 con 0x7fa3340c1ce0 2023-03-03T14:43:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa333fff700 1 --2- 172.21.15.7:0/2267184839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 0x7fa33414e730 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7fa33406bd10 tx=0x7fa32c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe50a3a0c746af9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa3327fc700 1 -- 172.21.15.7:0/2267184839 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c00e070 con 0x7fa3340c1ce0 2023-03-03T14:43:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa3327fc700 1 -- 172.21.15.7:0/2267184839 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c00b9c0 con 0x7fa3340c1ce0 2023-03-03T14:43:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.698+0000 7fa3327fc700 1 -- 172.21.15.7:0/2267184839 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c00bd70 con 0x7fa3340c1ce0 2023-03-03T14:43:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.699+0000 7fa339d73700 1 -- 172.21.15.7:0/2267184839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 msgr2=0x7fa33414e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.699+0000 7fa339d73700 1 --2- 172.21.15.7:0/2267184839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 0x7fa33414e730 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7fa33406bd10 tx=0x7fa32c00b330 comp rx=0 tx=0).stop 2023-03-03T14:43:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.699+0000 7fa339d73700 1 -- 172.21.15.7:0/2267184839 shutdown_connections 2023-03-03T14:43:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.699+0000 7fa339d73700 1 --2- 172.21.15.7:0/2267184839 >> v2:172.21.15.143:3300/0 conn(0x7fa334071e80 0x7fa334151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.699+0000 7fa339d73700 1 --2- 172.21.15.7:0/2267184839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 0x7fa33414e730 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.699+0000 7fa339d73700 1 --2- 172.21.15.7:0/2267184839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa334072250 0x7fa33414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.699+0000 7fa339d73700 1 -- 172.21.15.7:0/2267184839 >> 172.21.15.7:0/2267184839 conn(0x7fa3340a94d0 msgr2=0x7fa3340a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:43:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.699+0000 7fa339d73700 1 -- 172.21.15.7:0/2267184839 shutdown_connections 2023-03-03T14:43:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.699+0000 7fa339d73700 1 -- 172.21.15.7:0/2267184839 wait complete. 2023-03-03T14:43:58.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.700+0000 7fa339d73700 1 Processor -- start 2023-03-03T14:43:58.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.700+0000 7fa339d73700 1 -- start start 2023-03-03T14:43:58.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa339d73700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa334071e80 0x7fa334150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:58.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa339d73700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa334072250 0x7fa334149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa339d73700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 0x7fa334147570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa339d73700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3341ee190 con 0x7fa3340c1ce0 2023-03-03T14:43:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa339d73700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3341ed580 con 0x7fa334071e80 2023-03-03T14:43:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa339d73700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3341ed860 con 0x7fa334072250 2023-03-03T14:43:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa333fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 0x7fa334147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa333fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 0x7fa334147570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50086/0 (socket says 172.21.15.7:50086) 2023-03-03T14:43:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa333fff700 1 -- 172.21.15.7:0/509123051 learned_addr learned my addr 172.21.15.7:0/509123051 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:43:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa332ffd700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa334072250 0x7fa334149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa3337fe700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa334071e80 0x7fa334150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa333fff700 1 -- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa334072250 msgr2=0x7fa334149f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa333fff700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa334072250 0x7fa334149f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa333fff700 1 -- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa334071e80 msgr2=0x7fa334150200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa333fff700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa334071e80 0x7fa334150200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa333fff700 1 -- 172.21.15.7:0/509123051 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c009b30 con 0x7fa3340c1ce0 2023-03-03T14:43:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.701+0000 7fa3337fe700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa334071e80 0x7fa334150200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:43:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.702+0000 7fa333fff700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 0x7fa334147570 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7fa32c012000 tx=0x7fa32c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.702+0000 7fa31bfff700 1 -- 172.21.15.7:0/509123051 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c0099b0 con 0x7fa3340c1ce0 2023-03-03T14:43:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.702+0000 7fa31bfff700 1 -- 172.21.15.7:0/509123051 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c004250 con 0x7fa3340c1ce0 2023-03-03T14:43:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.702+0000 7fa339d73700 1 -- 172.21.15.7:0/509123051 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33406c9a0 con 0x7fa3340c1ce0 2023-03-03T14:43:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.702+0000 7fa31bfff700 1 -- 172.21.15.7:0/509123051 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c004600 con 0x7fa3340c1ce0 2023-03-03T14:43:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.702+0000 7fa339d73700 1 -- 172.21.15.7:0/509123051 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa334151300 con 0x7fa3340c1ce0 2023-03-03T14:43:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.703+0000 7fa339d73700 1 -- 172.21.15.7:0/509123051 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33406cb30 con 0x7fa3340c1ce0 2023-03-03T14:43:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.703+0000 7fa31bfff700 1 -- 172.21.15.7:0/509123051 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa32c004760 con 0x7fa3340c1ce0 2023-03-03T14:43:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.704+0000 7fa31bfff700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa320068320 0x7fa32006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:43:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.704+0000 7fa31bfff700 1 -- 172.21.15.7:0/509123051 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa32c0874f0 con 0x7fa3340c1ce0 2023-03-03T14:43:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.704+0000 7fa3337fe700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa320068320 0x7fa32006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:43:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.705+0000 7fa3337fe700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa320068320 0x7fa32006a7d0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fa324009970 tx=0x7fa324008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:43:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.708+0000 7fa31bfff700 1 -- 172.21.15.7:0/509123051 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa32c0537d0 con 0x7fa3340c1ce0 2023-03-03T14:43:58.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.948+0000 7fa339d73700 1 -- 172.21.15.7:0/509123051 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3341e9eb0 con 0x7fa3340c1ce0 2023-03-03T14:43:58.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.948+0000 7fa31bfff700 1 -- 172.21.15.7:0/509123051 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa3341e9eb0 con 0x7fa3340c1ce0 2023-03-03T14:43:58.951 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:43:58.952 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:43:58.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 -- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa320068320 msgr2=0x7fa32006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa320068320 0x7fa32006a7d0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fa324009970 tx=0x7fa324008040 comp rx=0 tx=0).stop 2023-03-03T14:43:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 -- 172.21.15.7:0/509123051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 msgr2=0x7fa334147570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:43:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 0x7fa334147570 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7fa32c012000 tx=0x7fa32c004ad0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 -- 172.21.15.7:0/509123051 shutdown_connections 2023-03-03T14:43:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa320068320 0x7fa32006a7d0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3340c1ce0 0x7fa334147570 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa334071e80 0x7fa334150200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 --2- 172.21.15.7:0/509123051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa334072250 0x7fa334149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:43:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 -- 172.21.15.7:0/509123051 >> 172.21.15.7:0/509123051 conn(0x7fa3340a94d0 msgr2=0x7fa3340c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:43:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 -- 172.21.15.7:0/509123051 shutdown_connections 2023-03-03T14:43:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:43:58.951+0000 7fa339d73700 1 -- 172.21.15.7:0/509123051 wait complete. 2023-03-03T14:43:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:43:58.965 INFO:tasks.mds_thrash.fs.[b]:waiting for 71 secs before thrashing 2023-03-03T14:44:26.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:26.304 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:56.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:44:56.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:06.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:45:06.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.137+0000 7f5db4906700 1 Processor -- start 2023-03-03T14:45:06.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.137+0000 7f5db4906700 1 -- start start 2023-03-03T14:45:06.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5db4906700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db0071e80 0x7f5db0151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:06.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5db4906700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db01510d0 con 0x7f5db0072250 2023-03-03T14:45:06.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5db4906700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db00c3740 con 0x7f5db014e900 2023-03-03T14:45:06.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5db4906700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db0070670 con 0x7f5db0071e80 2023-03-03T14:45:06.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dadd9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5db0072250 0x7f5db014e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44112/0 (socket says 172.21.15.7:44112) 2023-03-03T14:45:06.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dadd9b700 1 -- 172.21.15.7:0/4210708088 learned_addr learned my addr 172.21.15.7:0/4210708088 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:45:06.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997227690 0 0) 0x7f5db01510d0 con 0x7f5db0072250 2023-03-03T14:45:06.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d9c0035b0 con 0x7f5db0072250 2023-03-03T14:45:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dae59c700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db0071e80 0x7f5db0151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 906073817 0 0) 0x7f5d9c0035b0 con 0x7f5db0072250 2023-03-03T14:45:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db0071e80 msgr2=0x7f5db0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:45:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dad59a700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db0071e80 0x7f5db0151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 >> v1:172.21.15.143:6789/0 conn(0x7f5db014e900 legacy=0x7f5db015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db00c39e0 con 0x7f5db0072250 2023-03-03T14:45:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f5d98001670 con 0x7f5db0072250 2023-03-03T14:45:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.139+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 >> v1:172.21.15.7:6789/0 conn(0x7f5db0072250 legacy=0x7f5db014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dad59a700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d9c003aa0 0x7f5d9c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dad59a700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d9c004850 0x7f5d9c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dad59a700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db0071e80 0x7f5d9c005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db0070670 con 0x7f5d9c003aa0 2023-03-03T14:45:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c00bba0 con 0x7f5d9c004850 2023-03-03T14:45:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9c00bd60 con 0x7f5db0071e80 2023-03-03T14:45:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dae59c700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d9c003aa0 0x7f5d9c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dadd9b700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d9c004850 0x7f5d9c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5daed9d700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db0071e80 0x7f5d9c005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dae59c700 1 -- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db0071e80 msgr2=0x7f5d9c005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:45:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dae59c700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db0071e80 0x7f5d9c005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dae59c700 1 -- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d9c004850 msgr2=0x7f5d9c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:45:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dae59c700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d9c004850 0x7f5d9c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dae59c700 1 -- 172.21.15.7:0/4210708088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d9c0035b0 con 0x7f5d9c003aa0 2023-03-03T14:45:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5daed9d700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db0071e80 0x7f5d9c005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:45:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dae59c700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d9c003aa0 0x7f5d9c003eb0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009b60 tx=0x7f5d9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de37a4eb357da301 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:45:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da000e6c0 con 0x7f5d9c003aa0 2023-03-03T14:45:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da000e820 con 0x7f5d9c003aa0 2023-03-03T14:45:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.140+0000 7f5dad59a700 1 -- 172.21.15.7:0/4210708088 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da000eb90 con 0x7f5d9c003aa0 2023-03-03T14:45:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.141+0000 7f5db4906700 1 -- 172.21.15.7:0/4210708088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d9c003aa0 msgr2=0x7f5d9c003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.141+0000 7f5db4906700 1 --2- 172.21.15.7:0/4210708088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d9c003aa0 0x7f5d9c003eb0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009b60 tx=0x7f5d9c005880 comp rx=0 tx=0).stop 2023-03-03T14:45:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.141+0000 7f5daed9d700 1 -- 172.21.15.7:0/4210708088 reap_dead start 2023-03-03T14:45:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.141+0000 7f5db4906700 1 -- 172.21.15.7:0/4210708088 shutdown_connections 2023-03-03T14:45:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.141+0000 7f5db4906700 1 -- 172.21.15.7:0/4210708088 >> 172.21.15.7:0/4210708088 conn(0x7f5db00a94d0 msgr2=0x7f5db00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:45:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.141+0000 7f5db4906700 1 -- 172.21.15.7:0/4210708088 shutdown_connections 2023-03-03T14:45:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.141+0000 7f5db4906700 1 -- 172.21.15.7:0/4210708088 wait complete. 2023-03-03T14:45:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.142+0000 7f5db4906700 1 Processor -- start 2023-03-03T14:45:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.142+0000 7f5db4906700 1 -- start start 2023-03-03T14:45:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5db4906700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d9c003aa0 0x7f5db01ea080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5db4906700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5db0145ab0 0x7f5db01e6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5db4906700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db01471a0 0x7f5db01dba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5db4906700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db01dac80 con 0x7f5db01471a0 2023-03-03T14:45:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5db4906700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db01e9c60 con 0x7f5db0145ab0 2023-03-03T14:45:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5db4906700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db0144fd0 con 0x7f5d9c003aa0 2023-03-03T14:45:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5daed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db01471a0 0x7f5db01dba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5daed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db01471a0 0x7f5db01dba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50146/0 (socket says 172.21.15.7:50146) 2023-03-03T14:45:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5daed9d700 1 -- 172.21.15.7:0/2653551826 learned_addr learned my addr 172.21.15.7:0/2653551826 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:45:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5dadd9b700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5db0145ab0 0x7f5db01e6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:06.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5dae59c700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d9c003aa0 0x7f5db01ea080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:06.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5daed9d700 1 -- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d9c003aa0 msgr2=0x7f5db01ea080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:06.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5daed9d700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d9c003aa0 0x7f5db01ea080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:06.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5daed9d700 1 -- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5db0145ab0 msgr2=0x7f5db01e6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:06.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5daed9d700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5db0145ab0 0x7f5db01e6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:06.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.143+0000 7f5daed9d700 1 -- 172.21.15.7:0/2653551826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db006c9a0 con 0x7f5db01471a0 2023-03-03T14:45:06.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.144+0000 7f5daed9d700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db01471a0 0x7f5db01dba80 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f5db00badb0 tx=0x7f5da800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:45:06.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.144+0000 7f5da6ffd700 1 -- 172.21.15.7:0/2653551826 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da8003c10 con 0x7f5db01471a0 2023-03-03T14:45:06.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.144+0000 7f5da6ffd700 1 -- 172.21.15.7:0/2653551826 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da8003d90 con 0x7f5db01471a0 2023-03-03T14:45:06.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.144+0000 7f5da6ffd700 1 -- 172.21.15.7:0/2653551826 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da8005310 con 0x7f5db01471a0 2023-03-03T14:45:06.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.144+0000 7f5db4906700 1 -- 172.21.15.7:0/2653551826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db0151300 con 0x7f5db01471a0 2023-03-03T14:45:06.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.144+0000 7f5db4906700 1 -- 172.21.15.7:0/2653551826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db006cb30 con 0x7f5db01471a0 2023-03-03T14:45:06.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.145+0000 7f5db4906700 1 -- 172.21.15.7:0/2653551826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db01e6ff0 con 0x7f5db01471a0 2023-03-03T14:45:06.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.145+0000 7f5da6ffd700 1 -- 172.21.15.7:0/2653551826 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5da8005470 con 0x7f5db01471a0 2023-03-03T14:45:06.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.146+0000 7f5da6ffd700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d900682f0 0x7f5d9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:06.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.146+0000 7f5da6ffd700 1 -- 172.21.15.7:0/2653551826 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5da80861c0 con 0x7f5db01471a0 2023-03-03T14:45:06.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.146+0000 7f5dae59c700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d900682f0 0x7f5d9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:06.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.146+0000 7f5dae59c700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d900682f0 0x7f5d9006a7a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f5da00090b0 tx=0x7f5da0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:45:06.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.150+0000 7f5da6ffd700 1 -- 172.21.15.7:0/2653551826 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5db01e6ff0 con 0x7f5db01471a0 2023-03-03T14:45:06.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.388+0000 7f5db4906700 1 -- 172.21.15.7:0/2653551826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5db01e9da0 con 0x7f5db01471a0 2023-03-03T14:45:06.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.389+0000 7f5da6ffd700 1 -- 172.21.15.7:0/2653551826 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5db01e9da0 con 0x7f5db01471a0 2023-03-03T14:45:06.390 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:45:06.391 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:45:06.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 -- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d900682f0 msgr2=0x7f5d9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:06.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d900682f0 0x7f5d9006a7a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f5da00090b0 tx=0x7f5da0008030 comp rx=0 tx=0).stop 2023-03-03T14:45:06.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 -- 172.21.15.7:0/2653551826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db01471a0 msgr2=0x7f5db01dba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:06.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db01471a0 0x7f5db01dba80 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f5db00badb0 tx=0x7f5da800eda0 comp rx=0 tx=0).stop 2023-03-03T14:45:06.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 -- 172.21.15.7:0/2653551826 shutdown_connections 2023-03-03T14:45:06.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d900682f0 0x7f5d9006a7a0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:06.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db01471a0 0x7f5db01dba80 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:06.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d9c003aa0 0x7f5db01ea080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:06.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 --2- 172.21.15.7:0/2653551826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5db0145ab0 0x7f5db01e6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:06.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 -- 172.21.15.7:0/2653551826 >> 172.21.15.7:0/2653551826 conn(0x7f5db00a94d0 msgr2=0x7f5db006d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:45:06.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 -- 172.21.15.7:0/2653551826 shutdown_connections 2023-03-03T14:45:06.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:06.391+0000 7f5db4906700 1 -- 172.21.15.7:0/2653551826 wait complete. 2023-03-03T14:45:06.394 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:45:06.404 INFO:tasks.mds_thrash.fs.[a]:waiting for 63 secs before thrashing 2023-03-03T14:45:09.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:45:10.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.104+0000 7f39d36b9700 1 Processor -- start 2023-03-03T14:45:10.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.104+0000 7f39d36b9700 1 -- start start 2023-03-03T14:45:10.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39d36b9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f39cc14e900 0x7f39cc15ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:10.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39d36b9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc1510d0 con 0x7f39cc071e80 2023-03-03T14:45:10.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39d36b9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc0c3740 con 0x7f39cc072250 2023-03-03T14:45:10.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39d36b9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39cc070670 con 0x7f39cc14e900 2023-03-03T14:45:10.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39d1455700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f39cc071e80 0x7f39cc151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44130/0 (socket says 172.21.15.7:44130) 2023-03-03T14:45:10.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39d1455700 1 -- 172.21.15.7:0/16570881 learned_addr learned my addr 172.21.15.7:0/16570881 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:45:10.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39d1c56700 1 --2- 172.21.15.7:0/16570881 >> v2:172.21.15.143:3300/0 conn(0x7f39cc14e900 0x7f39cc15ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:10.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716207425 0 0) 0x7f39cc1510d0 con 0x7f39cc071e80 2023-03-03T14:45:10.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b80035b0 con 0x7f39cc071e80 2023-03-03T14:45:10.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1508826294 0 0) 0x7f39b80035b0 con 0x7f39cc071e80 2023-03-03T14:45:10.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 >> v2:172.21.15.143:3300/0 conn(0x7f39cc14e900 msgr2=0x7f39cc15ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:45:10.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39c3fff700 1 --2- 172.21.15.7:0/16570881 >> v2:172.21.15.143:3300/0 conn(0x7f39cc14e900 0x7f39cc15ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 >> v1:172.21.15.143:6789/0 conn(0x7f39cc072250 legacy=0x7f39cc14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:10.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc0c39e0 con 0x7f39cc071e80 2023-03-03T14:45:10.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f39c8001670 con 0x7f39cc071e80 2023-03-03T14:45:10.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.105+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 >> v1:172.21.15.7:6789/0 conn(0x7f39cc071e80 legacy=0x7f39cc151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:10.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39c3fff700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39b8003a70 0x7f39b8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:10.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39c3fff700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b8004850 0x7f39b8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:10.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39c3fff700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:10.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc070670 con 0x7f39b8005250 2023-03-03T14:45:10.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b800c190 con 0x7f39b8004850 2023-03-03T14:45:10.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b800c350 con 0x7f39b8003a70 2023-03-03T14:45:10.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39d1c56700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39d1455700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39b8003a70 0x7f39b8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39d0c54700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b8004850 0x7f39b8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39d1c56700 1 -- 172.21.15.7:0/16570881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39b8003a70 msgr2=0x7f39b8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:45:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39d1c56700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39b8003a70 0x7f39b8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39d1c56700 1 -- 172.21.15.7:0/16570881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b8004850 msgr2=0x7f39b8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:45:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39d1c56700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39b8004850 0x7f39b8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39d1c56700 1 -- 172.21.15.7:0/16570881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80035b0 con 0x7f39b8005250 2023-03-03T14:45:10.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39d1c56700 1 -- 172.21.15.7:0/16570881 reap_dead start 2023-03-03T14:45:10.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39d1c56700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39b800ba10 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0713e0 tx=0x7f39b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75dc8119e80268c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:45:10.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4003c20 con 0x7f39b8005250 2023-03-03T14:45:10.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c400f040 con 0x7f39b8005250 2023-03-03T14:45:10.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.106+0000 7f39c3fff700 1 -- 172.21.15.7:0/16570881 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c400e9d0 con 0x7f39b8005250 2023-03-03T14:45:10.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.107+0000 7f39d36b9700 1 -- 172.21.15.7:0/16570881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 msgr2=0x7f39b800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:10.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.107+0000 7f39d36b9700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39b800ba10 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0713e0 tx=0x7f39b8005880 comp rx=0 tx=0).stop 2023-03-03T14:45:10.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.107+0000 7f39d36b9700 1 -- 172.21.15.7:0/16570881 shutdown_connections 2023-03-03T14:45:10.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.107+0000 7f39d36b9700 1 --2- 172.21.15.7:0/16570881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39b800ba10 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.107+0000 7f39d36b9700 1 -- 172.21.15.7:0/16570881 >> 172.21.15.7:0/16570881 conn(0x7f39cc0a94d0 msgr2=0x7f39cc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:45:10.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.107+0000 7f39d36b9700 1 -- 172.21.15.7:0/16570881 shutdown_connections 2023-03-03T14:45:10.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.107+0000 7f39d36b9700 1 -- 172.21.15.7:0/16570881 wait complete. 2023-03-03T14:45:10.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.108+0000 7f39d36b9700 1 Processor -- start 2023-03-03T14:45:10.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.108+0000 7f39d36b9700 1 -- start start 2023-03-03T14:45:10.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.109+0000 7f39d36b9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39cc1ea040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:10.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.109+0000 7f39d36b9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39cc0c2fa0 0x7f39cc0aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:10.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.109+0000 7f39d36b9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140da0 0x7f39cc141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:10.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.109+0000 7f39d36b9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc142bf0 con 0x7f39b8005250 2023-03-03T14:45:10.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.109+0000 7f39d36b9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc148f30 con 0x7f39cc0c2fa0 2023-03-03T14:45:10.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.109+0000 7f39d36b9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc141d20 con 0x7f39cc140da0 2023-03-03T14:45:10.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d1455700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39cc1ea040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:10.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d1455700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39cc1ea040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50162/0 (socket says 172.21.15.7:50162) 2023-03-03T14:45:10.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d1455700 1 -- 172.21.15.7:0/3063001742 learned_addr learned my addr 172.21.15.7:0/3063001742 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:45:10.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d0c54700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39cc0c2fa0 0x7f39cc0aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:10.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d1c56700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140da0 0x7f39cc141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:10.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d1455700 1 -- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140da0 msgr2=0x7f39cc141220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:10.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d1455700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140da0 0x7f39cc141220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d1455700 1 -- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39cc0c2fa0 msgr2=0x7f39cc0aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:10.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d1455700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39cc0c2fa0 0x7f39cc0aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d1455700 1 -- 172.21.15.7:0/3063001742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc06c9a0 con 0x7f39b8005250 2023-03-03T14:45:10.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.110+0000 7f39d1455700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39cc1ea040 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f39c8007290 tx=0x7f39c80075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:45:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.111+0000 7f39c1ffb700 1 -- 172.21.15.7:0/3063001742 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c8011580 con 0x7f39b8005250 2023-03-03T14:45:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.111+0000 7f39c1ffb700 1 -- 172.21.15.7:0/3063001742 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c8011700 con 0x7f39b8005250 2023-03-03T14:45:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.111+0000 7f39d36b9700 1 -- 172.21.15.7:0/3063001742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39cc151300 con 0x7f39b8005250 2023-03-03T14:45:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.111+0000 7f39c1ffb700 1 -- 172.21.15.7:0/3063001742 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c8011ab0 con 0x7f39b8005250 2023-03-03T14:45:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.111+0000 7f39d36b9700 1 -- 172.21.15.7:0/3063001742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39cc06cb30 con 0x7f39b8005250 2023-03-03T14:45:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.112+0000 7f39d36b9700 1 -- 172.21.15.7:0/3063001742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39cc074bb0 con 0x7f39b8005250 2023-03-03T14:45:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.112+0000 7f39c1ffb700 1 -- 172.21.15.7:0/3063001742 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f39c8011c10 con 0x7f39b8005250 2023-03-03T14:45:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.113+0000 7f39c1ffb700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39b0068320 0x7f39b006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:45:10.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.113+0000 7f39c1ffb700 1 -- 172.21.15.7:0/3063001742 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f39c8086210 con 0x7f39b8005250 2023-03-03T14:45:10.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.113+0000 7f39d0c54700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39b0068320 0x7f39b006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:45:10.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.113+0000 7f39d0c54700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39b0068320 0x7f39b006a7d0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0c27b0 tx=0x7f39bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:45:10.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.117+0000 7f39c1ffb700 1 -- 172.21.15.7:0/3063001742 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f39c80525b0 con 0x7f39b8005250 2023-03-03T14:45:10.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.356+0000 7f39d36b9700 1 -- 172.21.15.7:0/3063001742 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39cc0a9fb0 con 0x7f39b8005250 2023-03-03T14:45:10.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.357+0000 7f39c1ffb700 1 -- 172.21.15.7:0/3063001742 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f39c8055bd0 con 0x7f39b8005250 2023-03-03T14:45:10.358 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:45:10.359 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:45:10.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 -- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39b0068320 msgr2=0x7f39b006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:10.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39b0068320 0x7f39b006a7d0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0c27b0 tx=0x7f39bc009040 comp rx=0 tx=0).stop 2023-03-03T14:45:10.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 -- 172.21.15.7:0/3063001742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 msgr2=0x7f39cc1ea040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:45:10.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39cc1ea040 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f39c8007290 tx=0x7f39c80075a0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 -- 172.21.15.7:0/3063001742 shutdown_connections 2023-03-03T14:45:10.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f39cc140da0 0x7f39cc141220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39b0068320 0x7f39b006a7d0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39b8005250 0x7f39cc1ea040 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 --2- 172.21.15.7:0/3063001742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f39cc0c2fa0 0x7f39cc0aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:45:10.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 -- 172.21.15.7:0/3063001742 >> 172.21.15.7:0/3063001742 conn(0x7f39cc0a94d0 msgr2=0x7f39cc14ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:45:10.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 -- 172.21.15.7:0/3063001742 shutdown_connections 2023-03-03T14:45:10.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:45:10.359+0000 7f39d36b9700 1 -- 172.21.15.7:0/3063001742 wait complete. 2023-03-03T14:45:10.362 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:45:10.372 INFO:tasks.mds_thrash.fs.[b]:waiting for 51 secs before thrashing 2023-03-03T14:45:26.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:26.526 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:56.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:45:56.596 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:01.374 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:46:01.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.510+0000 7fac258b2700 1 Processor -- start 2023-03-03T14:46:01.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.510+0000 7fac258b2700 1 -- start start 2023-03-03T14:46:01.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac258b2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac2014b060 0x7fac2006f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:01.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac258b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 0x7fac200aa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:01.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac258b2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac200c20c0 con 0x7fac200c1850 2023-03-03T14:46:01.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac258b2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac200c4100 con 0x7fac2014b060 2023-03-03T14:46:01.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac258b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2006fd40 con 0x7fac200c1ca0 2023-03-03T14:46:01.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac1effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac2014b060 0x7fac2006f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:01.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac1effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac2014b060 0x7fac2006f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50220/0 (socket says 172.21.15.7:50220) 2023-03-03T14:46:01.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac1effd700 1 -- 172.21.15.7:0/1963718773 learned_addr learned my addr 172.21.15.7:0/1963718773 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:46:01.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac1f7fe700 1 --2- 172.21.15.7:0/1963718773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 0x7fac200aa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:01.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac1effd700 1 -- 172.21.15.7:0/1963718773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 msgr2=0x7fac200aa880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:46:01.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac1effd700 1 --2- 172.21.15.7:0/1963718773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 0x7fac200aa880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:01.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac1effd700 1 -- 172.21.15.7:0/1963718773 >> v1:172.21.15.143:6789/0 conn(0x7fac200c1850 legacy=0x7fac200aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:46:01.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac1effd700 1 -- 172.21.15.7:0/1963718773 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac20154790 con 0x7fac2014b060 2023-03-03T14:46:01.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac1f7fe700 1 --2- 172.21.15.7:0/1963718773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 0x7fac200aa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:46:01.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.511+0000 7fac1effd700 1 --2- 172.21.15.7:0/1963718773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 0x7fac2006f800 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fac100098f0 tx=0x7fac10009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a641593d7fa5c19 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:01.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac1dffb700 1 -- 172.21.15.7:0/1963718773 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1000b8a0 con 0x7fac2014b060 2023-03-03T14:46:01.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac1dffb700 1 -- 172.21.15.7:0/1963718773 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1000ba00 con 0x7fac2014b060 2023-03-03T14:46:01.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac1dffb700 1 -- 172.21.15.7:0/1963718773 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1000bd90 con 0x7fac2014b060 2023-03-03T14:46:01.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac258b2700 1 -- 172.21.15.7:0/1963718773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 msgr2=0x7fac2006f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:01.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac258b2700 1 --2- 172.21.15.7:0/1963718773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 0x7fac2006f800 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fac100098f0 tx=0x7fac10009c00 comp rx=0 tx=0).stop 2023-03-03T14:46:01.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac258b2700 1 -- 172.21.15.7:0/1963718773 shutdown_connections 2023-03-03T14:46:01.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac258b2700 1 --2- 172.21.15.7:0/1963718773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 0x7fac200aa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:01.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac258b2700 1 --2- 172.21.15.7:0/1963718773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 0x7fac2006f800 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:01.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac258b2700 1 -- 172.21.15.7:0/1963718773 >> 172.21.15.7:0/1963718773 conn(0x7fac200c4f20 msgr2=0x7fac200c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:46:01.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac258b2700 1 -- 172.21.15.7:0/1963718773 shutdown_connections 2023-03-03T14:46:01.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.512+0000 7fac258b2700 1 -- 172.21.15.7:0/1963718773 wait complete. 2023-03-03T14:46:01.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.513+0000 7fac258b2700 1 Processor -- start 2023-03-03T14:46:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac258b2700 1 -- start start 2023-03-03T14:46:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac258b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac200c1850 0x7fac201ed1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac258b2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 0x7fac20141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac258b2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 0x7fac20141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac258b2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac201458b0 con 0x7fac2014b060 2023-03-03T14:46:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac258b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac201edc50 con 0x7fac200c1850 2023-03-03T14:46:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac258b2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac20142300 con 0x7fac200c1ca0 2023-03-03T14:46:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac1f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 0x7fac20141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac1f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 0x7fac20141dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50230/0 (socket says 172.21.15.7:50230) 2023-03-03T14:46:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac1f7fe700 1 -- 172.21.15.7:0/3326497735 learned_addr learned my addr 172.21.15.7:0/3326497735 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:46:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac1effd700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac200c1850 0x7fac201ed1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.514+0000 7fac1e7fc700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 0x7fac20141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac1f7fe700 1 -- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 msgr2=0x7fac20141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac1f7fe700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 0x7fac20141880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac1f7fe700 1 -- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac200c1850 msgr2=0x7fac201ed1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac1f7fe700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac200c1850 0x7fac201ed1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac1f7fe700 1 -- 172.21.15.7:0/3326497735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac10009550 con 0x7fac2014b060 2023-03-03T14:46:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac1f7fe700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 0x7fac20141dc0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fac1800cef0 tx=0x7fac1800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac07fff700 1 -- 172.21.15.7:0/3326497735 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1800ad00 con 0x7fac2014b060 2023-03-03T14:46:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac07fff700 1 -- 172.21.15.7:0/3326497735 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1800ae60 con 0x7fac2014b060 2023-03-03T14:46:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac258b2700 1 -- 172.21.15.7:0/3326497735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac20154d50 con 0x7fac2014b060 2023-03-03T14:46:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac07fff700 1 -- 172.21.15.7:0/3326497735 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18015680 con 0x7fac2014b060 2023-03-03T14:46:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.515+0000 7fac258b2700 1 -- 172.21.15.7:0/3326497735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac20154a70 con 0x7fac2014b060 2023-03-03T14:46:01.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.516+0000 7fac258b2700 1 -- 172.21.15.7:0/3326497735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac20155030 con 0x7fac2014b060 2023-03-03T14:46:01.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.517+0000 7fac07fff700 1 -- 172.21.15.7:0/3326497735 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fac1801c030 con 0x7fac2014b060 2023-03-03T14:46:01.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.517+0000 7fac07fff700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 0x7fac0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:01.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.517+0000 7fac07fff700 1 -- 172.21.15.7:0/3326497735 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fac180864a0 con 0x7fac2014b060 2023-03-03T14:46:01.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.517+0000 7fac1effd700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 0x7fac0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:01.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.518+0000 7fac1effd700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 0x7fac0c06a7a0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fac10014040 tx=0x7fac10008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:01.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.522+0000 7fac07fff700 1 -- 172.21.15.7:0/3326497735 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fac18052840 con 0x7fac2014b060 2023-03-03T14:46:01.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.759+0000 7fac258b2700 1 -- 172.21.15.7:0/3326497735 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac20074bb0 con 0x7fac2014b060 2023-03-03T14:46:01.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.760+0000 7fac07fff700 1 -- 172.21.15.7:0/3326497735 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fac18055e60 con 0x7fac2014b060 2023-03-03T14:46:01.763 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:46:01.763 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:46:01.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 -- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 msgr2=0x7fac0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:01.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 0x7fac0c06a7a0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fac10014040 tx=0x7fac10008030 comp rx=0 tx=0).stop 2023-03-03T14:46:01.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 -- 172.21.15.7:0/3326497735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 msgr2=0x7fac20141dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:01.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 0x7fac20141dc0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fac1800cef0 tx=0x7fac1800cf20 comp rx=0 tx=0).stop 2023-03-03T14:46:01.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 -- 172.21.15.7:0/3326497735 shutdown_connections 2023-03-03T14:46:01.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 0x7fac0c06a7a0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:01.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2014b060 0x7fac20141dc0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:01.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac200c1850 0x7fac201ed1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:01.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 --2- 172.21.15.7:0/3326497735 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac200c1ca0 0x7fac20141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:01.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 -- 172.21.15.7:0/3326497735 >> 172.21.15.7:0/3326497735 conn(0x7fac200c4f20 msgr2=0x7fac201491f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:46:01.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 -- 172.21.15.7:0/3326497735 shutdown_connections 2023-03-03T14:46:01.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:01.762+0000 7fac258b2700 1 -- 172.21.15.7:0/3326497735 wait complete. 2023-03-03T14:46:01.766 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:46:01.776 INFO:tasks.mds_thrash.fs.[b]:waiting for 30 secs before thrashing 2023-03-03T14:46:09.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:46:09.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.540+0000 7f312131e700 1 Processor -- start 2023-03-03T14:46:09.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.540+0000 7f312131e700 1 -- start start 2023-03-03T14:46:09.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f312131e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c0caa30 0x7f311c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:09.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f312131e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0c7310 con 0x7f311c0c00e0 2023-03-03T14:46:09.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f312131e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0bfd10 con 0x7f311c154ff0 2023-03-03T14:46:09.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f312131e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0bfa30 con 0x7f311c0caa30 2023-03-03T14:46:09.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f311a7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f311c0c00e0 0x7f311c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44224/0 (socket says 172.21.15.7:44224) 2023-03-03T14:46:09.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f311a7fc700 1 -- 172.21.15.7:0/1981446685 learned_addr learned my addr 172.21.15.7:0/1981446685 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:46:09.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f311affd700 1 --2- 172.21.15.7:0/1981446685 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c0caa30 0x7f311c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:09.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f3119ffb700 1 -- 172.21.15.7:0/1981446685 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941901504 0 0) 0x7f311c0c7310 con 0x7f311c0c00e0 2023-03-03T14:46:09.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f3119ffb700 1 -- 172.21.15.7:0/1981446685 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31040035b0 con 0x7f311c0c00e0 2023-03-03T14:46:09.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f3119ffb700 1 -- 172.21.15.7:0/1981446685 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079657590 0 0) 0x7f311c0bfd10 con 0x7f311c154ff0 2023-03-03T14:46:09.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.541+0000 7f3119ffb700 1 -- 172.21.15.7:0/1981446685 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f311c0c7310 con 0x7f311c154ff0 2023-03-03T14:46:09.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.542+0000 7f3119ffb700 1 -- 172.21.15.7:0/1981446685 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1539268721 0 0) 0x7f31040035b0 con 0x7f311c0c00e0 2023-03-03T14:46:09.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.542+0000 7f311affd700 1 -- 172.21.15.7:0/1981446685 >> v1:172.21.15.143:6789/0 conn(0x7f311c154ff0 legacy=0x7f311c15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:09.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.542+0000 7f311affd700 1 -- 172.21.15.7:0/1981446685 >> v1:172.21.15.7:6789/0 conn(0x7f311c0c00e0 legacy=0x7f311c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:09.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.542+0000 7f311affd700 1 -- 172.21.15.7:0/1981446685 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c154510 con 0x7f311c0caa30 2023-03-03T14:46:09.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.542+0000 7f311affd700 1 --2- 172.21.15.7:0/1981446685 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c0caa30 0x7f311c0c5c60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3110009870 tx=0x7f3110009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efe908cc98e0f8d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:09.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.542+0000 7f3119ffb700 1 -- 172.21.15.7:0/1981446685 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c0017f0 con 0x7f311c0caa30 2023-03-03T14:46:09.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.542+0000 7f3119ffb700 1 -- 172.21.15.7:0/1981446685 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110004030 con 0x7f311c0caa30 2023-03-03T14:46:09.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.542+0000 7f3119ffb700 1 -- 172.21.15.7:0/1981446685 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31100043c0 con 0x7f311c0caa30 2023-03-03T14:46:09.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.543+0000 7f312131e700 1 -- 172.21.15.7:0/1981446685 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c0caa30 msgr2=0x7f311c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:09.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.543+0000 7f312131e700 1 --2- 172.21.15.7:0/1981446685 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c0caa30 0x7f311c0c5c60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3110009870 tx=0x7f3110009b80 comp rx=0 tx=0).stop 2023-03-03T14:46:09.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.543+0000 7f312131e700 1 -- 172.21.15.7:0/1981446685 shutdown_connections 2023-03-03T14:46:09.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.543+0000 7f312131e700 1 --2- 172.21.15.7:0/1981446685 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c0caa30 0x7f311c0c5c60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:09.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.543+0000 7f312131e700 1 -- 172.21.15.7:0/1981446685 >> 172.21.15.7:0/1981446685 conn(0x7f311c066e40 msgr2=0x7f311c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:46:09.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.543+0000 7f312131e700 1 -- 172.21.15.7:0/1981446685 shutdown_connections 2023-03-03T14:46:09.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.543+0000 7f312131e700 1 -- 172.21.15.7:0/1981446685 wait complete. 2023-03-03T14:46:09.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.544+0000 7f312131e700 1 Processor -- start 2023-03-03T14:46:09.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.544+0000 7f312131e700 1 -- start start 2023-03-03T14:46:09.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f312131e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f311c0c00e0 0x7f311c147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:09.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f312131e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f311c0caa30 0x7f311c0c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:09.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f312131e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c154ff0 0x7f311c146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:09.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f312131e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c148450 con 0x7f311c0caa30 2023-03-03T14:46:09.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f312131e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c147850 con 0x7f311c0c00e0 2023-03-03T14:46:09.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f312131e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c1ed740 con 0x7f311c154ff0 2023-03-03T14:46:09.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f311c0caa30 0x7f311c0c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:09.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f311c0caa30 0x7f311c0c9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50252/0 (socket says 172.21.15.7:50252) 2023-03-03T14:46:09.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311a7fc700 1 -- 172.21.15.7:0/3312425359 learned_addr learned my addr 172.21.15.7:0/3312425359 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:46:09.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311b7fe700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c154ff0 0x7f311c146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:09.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311affd700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f311c0c00e0 0x7f311c147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:09.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311a7fc700 1 -- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c154ff0 msgr2=0x7f311c146660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:09.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311a7fc700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c154ff0 0x7f311c146660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:09.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311a7fc700 1 -- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f311c0c00e0 msgr2=0x7f311c147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:09.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311a7fc700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f311c0c00e0 0x7f311c147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:09.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311a7fc700 1 -- 172.21.15.7:0/3312425359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100094a0 con 0x7f311c0caa30 2023-03-03T14:46:09.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311affd700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f311c0c00e0 0x7f311c147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:46:09.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f311a7fc700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f311c0caa30 0x7f311c0c9a80 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f310c001360 tx=0x7f310c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:09.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f31037fe700 1 -- 172.21.15.7:0/3312425359 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c0074e0 con 0x7f311c0caa30 2023-03-03T14:46:09.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.545+0000 7f31037fe700 1 -- 172.21.15.7:0/3312425359 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310c007660 con 0x7f311c0caa30 2023-03-03T14:46:09.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.546+0000 7f312131e700 1 -- 172.21.15.7:0/3312425359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c154ad0 con 0x7f311c0caa30 2023-03-03T14:46:09.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.546+0000 7f31037fe700 1 -- 172.21.15.7:0/3312425359 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310c005a70 con 0x7f311c0caa30 2023-03-03T14:46:09.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.546+0000 7f312131e700 1 -- 172.21.15.7:0/3312425359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c1547f0 con 0x7f311c0caa30 2023-03-03T14:46:09.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.546+0000 7f312131e700 1 -- 172.21.15.7:0/3312425359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c0bf650 con 0x7f311c0caa30 2023-03-03T14:46:09.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.547+0000 7f31037fe700 1 -- 172.21.15.7:0/3312425359 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f310c019030 con 0x7f311c0caa30 2023-03-03T14:46:09.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.547+0000 7f31037fe700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3108068320 0x7f310806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:09.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.547+0000 7f31037fe700 1 -- 172.21.15.7:0/3312425359 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f310c085490 con 0x7f311c0caa30 2023-03-03T14:46:09.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.548+0000 7f311affd700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3108068320 0x7f310806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:09.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.548+0000 7f311affd700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3108068320 0x7f310806a7d0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f31100097c0 tx=0x7f3110008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:09.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.552+0000 7f31037fe700 1 -- 172.21.15.7:0/3312425359 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f310c051830 con 0x7f311c0caa30 2023-03-03T14:46:09.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.790+0000 7f312131e700 1 -- 172.21.15.7:0/3312425359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f311c074bb0 con 0x7f311c0caa30 2023-03-03T14:46:09.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.790+0000 7f31037fe700 1 -- 172.21.15.7:0/3312425359 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f310c054e50 con 0x7f311c0caa30 2023-03-03T14:46:09.793 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:46:09.794 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:46:09.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 -- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3108068320 msgr2=0x7f310806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:09.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3108068320 0x7f310806a7d0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f31100097c0 tx=0x7f3110008030 comp rx=0 tx=0).stop 2023-03-03T14:46:09.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 -- 172.21.15.7:0/3312425359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f311c0caa30 msgr2=0x7f311c0c9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:09.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f311c0caa30 0x7f311c0c9a80 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f310c001360 tx=0x7f310c00c800 comp rx=0 tx=0).stop 2023-03-03T14:46:09.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 -- 172.21.15.7:0/3312425359 shutdown_connections 2023-03-03T14:46:09.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f311c154ff0 0x7f311c146660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:09.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f311c0c00e0 0x7f311c147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:09.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3108068320 0x7f310806a7d0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 --2- 172.21.15.7:0/3312425359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f311c0caa30 0x7f311c0c9a80 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 -- 172.21.15.7:0/3312425359 >> 172.21.15.7:0/3312425359 conn(0x7f311c066e40 msgr2=0x7f311c0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:46:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 -- 172.21.15.7:0/3312425359 shutdown_connections 2023-03-03T14:46:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:09.793+0000 7f312131e700 1 -- 172.21.15.7:0/3312425359 wait complete. 2023-03-03T14:46:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:46:09.807 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before thrashing 2023-03-03T14:46:13.808 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:46:13.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.945+0000 7f91fd417700 1 Processor -- start 2023-03-03T14:46:13.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.946+0000 7f91fd417700 1 -- start start 2023-03-03T14:46:13.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91fd417700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91f80c7a30 0x7f91f814e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:13.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91fd417700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f91f80c7e00 0x7f91f814e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:13.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91fd417700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f806b6f0 0x7f91f8154a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:13.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91fd417700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f806ab00 con 0x7f91f80c7a30 2023-03-03T14:46:13.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91fd417700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f81545c0 con 0x7f91f80c7e00 2023-03-03T14:46:13.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91fd417700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f8157bb0 con 0x7f91f806b6f0 2023-03-03T14:46:13.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91f80c7a30 0x7f91f814e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:13.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f91f80c7a30 0x7f91f814e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50258/0 (socket says 172.21.15.7:50258) 2023-03-03T14:46:13.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f6ffd700 1 -- 172.21.15.7:0/446281652 learned_addr learned my addr 172.21.15.7:0/446281652 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:46:13.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f67fc700 1 --2- 172.21.15.7:0/446281652 >> v2:172.21.15.143:3300/0 conn(0x7f91f80c7e00 0x7f91f814e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:13.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f77fe700 1 --2- 172.21.15.7:0/446281652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f806b6f0 0x7f91f8154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:13.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f6ffd700 1 -- 172.21.15.7:0/446281652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f806b6f0 msgr2=0x7f91f8154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:46:13.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f6ffd700 1 --2- 172.21.15.7:0/446281652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f806b6f0 0x7f91f8154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:13.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f6ffd700 1 -- 172.21.15.7:0/446281652 >> v2:172.21.15.143:3300/0 conn(0x7f91f80c7e00 msgr2=0x7f91f814e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:46:13.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f6ffd700 1 --2- 172.21.15.7:0/446281652 >> v2:172.21.15.143:3300/0 conn(0x7f91f80c7e00 0x7f91f814e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:13.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f6ffd700 1 -- 172.21.15.7:0/446281652 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f806a2f0 con 0x7f91f80c7a30 2023-03-03T14:46:13.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.947+0000 7f91f6ffd700 1 --2- 172.21.15.7:0/446281652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 0x7f91f814e160 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f91f808c770 tx=0x7f91f0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ca8c3b944f50a0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:13.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91f5ffb700 1 -- 172.21.15.7:0/446281652 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0005e30 con 0x7f91f80c7a30 2023-03-03T14:46:13.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91f5ffb700 1 -- 172.21.15.7:0/446281652 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f0004030 con 0x7f91f80c7a30 2023-03-03T14:46:13.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91f5ffb700 1 -- 172.21.15.7:0/446281652 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f00043e0 con 0x7f91f80c7a30 2023-03-03T14:46:13.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91fd417700 1 -- 172.21.15.7:0/446281652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 msgr2=0x7f91f814e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:13.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91fd417700 1 --2- 172.21.15.7:0/446281652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 0x7f91f814e160 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f91f808c770 tx=0x7f91f0009bd0 comp rx=0 tx=0).stop 2023-03-03T14:46:13.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91fd417700 1 -- 172.21.15.7:0/446281652 shutdown_connections 2023-03-03T14:46:13.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91fd417700 1 --2- 172.21.15.7:0/446281652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f806b6f0 0x7f91f8154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:13.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91fd417700 1 --2- 172.21.15.7:0/446281652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 0x7f91f814e160 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:13.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91fd417700 1 --2- 172.21.15.7:0/446281652 >> v2:172.21.15.143:3300/0 conn(0x7f91f80c7e00 0x7f91f814e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:13.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91fd417700 1 -- 172.21.15.7:0/446281652 >> 172.21.15.7:0/446281652 conn(0x7f91f8073d20 msgr2=0x7f91f80716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:46:13.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91fd417700 1 -- 172.21.15.7:0/446281652 shutdown_connections 2023-03-03T14:46:13.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.948+0000 7f91fd417700 1 -- 172.21.15.7:0/446281652 wait complete. 2023-03-03T14:46:13.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.950+0000 7f91fd417700 1 Processor -- start 2023-03-03T14:46:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.950+0000 7f91fd417700 1 -- start start 2023-03-03T14:46:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91fd417700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 0x7f91f814cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91fd417700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f80c7e00 0x7f91f80c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:13.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91fd417700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f80caea0 0x7f91f80c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:13.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91fd417700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f81511c0 con 0x7f91f80c7a30 2023-03-03T14:46:13.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91fd417700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f81edbd0 con 0x7f91f80c7e00 2023-03-03T14:46:13.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91fd417700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f81ee100 con 0x7f91f80caea0 2023-03-03T14:46:13.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91f6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 0x7f91f814cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91f6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 0x7f91f814cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50264/0 (socket says 172.21.15.7:50264) 2023-03-03T14:46:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91f6ffd700 1 -- 172.21.15.7:0/4259189101 learned_addr learned my addr 172.21.15.7:0/4259189101 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:46:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91f67fc700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f80c7e00 0x7f91f80c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91f6ffd700 1 -- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f80caea0 msgr2=0x7f91f80c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:13.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91f6ffd700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f80caea0 0x7f91f80c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:13.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91f6ffd700 1 -- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f80c7e00 msgr2=0x7f91f80c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:13.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91f6ffd700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f80c7e00 0x7f91f80c8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.951+0000 7f91f6ffd700 1 -- 172.21.15.7:0/4259189101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0009500 con 0x7f91f80c7a30 2023-03-03T14:46:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.952+0000 7f91f6ffd700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 0x7f91f814cf30 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f91f0000c00 tx=0x7f91f0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.952+0000 7f91ef7fe700 1 -- 172.21.15.7:0/4259189101 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0003d90 con 0x7f91f80c7a30 2023-03-03T14:46:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.952+0000 7f91ef7fe700 1 -- 172.21.15.7:0/4259189101 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f0011420 con 0x7f91f80c7a30 2023-03-03T14:46:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.952+0000 7f91fd417700 1 -- 172.21.15.7:0/4259189101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f806a540 con 0x7f91f80c7a30 2023-03-03T14:46:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.952+0000 7f91ef7fe700 1 -- 172.21.15.7:0/4259189101 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f00117d0 con 0x7f91f80c7a30 2023-03-03T14:46:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.952+0000 7f91fd417700 1 -- 172.21.15.7:0/4259189101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f80bfbb0 con 0x7f91f80c7a30 2023-03-03T14:46:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.953+0000 7f91fd417700 1 -- 172.21.15.7:0/4259189101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f806a820 con 0x7f91f80c7a30 2023-03-03T14:46:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.953+0000 7f91ef7fe700 1 -- 172.21.15.7:0/4259189101 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f91f002c070 con 0x7f91f80c7a30 2023-03-03T14:46:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.954+0000 7f91ef7fe700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 0x7f91d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.954+0000 7f91ef7fe700 1 -- 172.21.15.7:0/4259189101 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f91f0087250 con 0x7f91f80c7a30 2023-03-03T14:46:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.954+0000 7f91f67fc700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 0x7f91d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.954+0000 7f91f67fc700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 0x7f91d806a7a0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f91f80c4fe0 tx=0x7f91e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:13.958+0000 7f91ef7fe700 1 -- 172.21.15.7:0/4259189101 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f91f00535f0 con 0x7f91f80c7a30 2023-03-03T14:46:14.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.196+0000 7f91fd417700 1 -- 172.21.15.7:0/4259189101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91f80c64a0 con 0x7f91f80c7a30 2023-03-03T14:46:14.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.197+0000 7f91ef7fe700 1 -- 172.21.15.7:0/4259189101 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f91f0056c10 con 0x7f91f80c7a30 2023-03-03T14:46:14.199 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:46:14.199 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:46:14.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 -- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 msgr2=0x7f91d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:14.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 0x7f91d806a7a0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f91f80c4fe0 tx=0x7f91e0008040 comp rx=0 tx=0).stop 2023-03-03T14:46:14.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 -- 172.21.15.7:0/4259189101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 msgr2=0x7f91f814cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:14.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 0x7f91f814cf30 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f91f0000c00 tx=0x7f91f0004750 comp rx=0 tx=0).stop 2023-03-03T14:46:14.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 -- 172.21.15.7:0/4259189101 shutdown_connections 2023-03-03T14:46:14.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f91f80caea0 0x7f91f80c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:14.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91d80682f0 0x7f91d806a7a0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:14.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91f80c7a30 0x7f91f814cf30 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:14.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 --2- 172.21.15.7:0/4259189101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91f80c7e00 0x7f91f80c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:14.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 -- 172.21.15.7:0/4259189101 >> 172.21.15.7:0/4259189101 conn(0x7f91f806b6f0 msgr2=0x7f91f806acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:46:14.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 -- 172.21.15.7:0/4259189101 shutdown_connections 2023-03-03T14:46:14.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:14.199+0000 7f91fd417700 1 -- 172.21.15.7:0/4259189101 wait complete. 2023-03-03T14:46:14.202 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:46:14.212 INFO:tasks.mds_thrash.fs.[a]:waiting for 54 secs before thrashing 2023-03-03T14:46:26.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:26.727 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:31.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:46:31.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.915+0000 7f7383e63700 1 Processor -- start 2023-03-03T14:46:31.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.915+0000 7f7383e63700 1 -- start start 2023-03-03T14:46:31.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.916+0000 7f7383e63700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f737c06c2f0 0x7f737c1584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:31.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.916+0000 7f7383e63700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737c06d780 con 0x7f737c151ab0 2023-03-03T14:46:31.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.916+0000 7f7383e63700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737c06c770 con 0x7f737c06bea0 2023-03-03T14:46:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.916+0000 7f7383e63700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f737c158a10 con 0x7f737c06c2f0 2023-03-03T14:46:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.916+0000 7f7382400700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f737c06c2f0 0x7f737c1584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.916+0000 7f7381bff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f737c151ab0 0x7f737c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44270/0 (socket says 172.21.15.7:44270) 2023-03-03T14:46:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.916+0000 7f7381bff700 1 -- 172.21.15.7:0/3228891036 learned_addr learned my addr 172.21.15.7:0/3228891036 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:46:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.917+0000 7f7382400700 1 -- 172.21.15.7:0/3228891036 >> v1:172.21.15.143:6789/0 conn(0x7f737c06bea0 legacy=0x7f737c157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.917+0000 7f7382400700 1 -- 172.21.15.7:0/3228891036 >> v1:172.21.15.7:6789/0 conn(0x7f737c151ab0 legacy=0x7f737c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:31.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.917+0000 7f7382400700 1 -- 172.21.15.7:0/3228891036 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c06cb70 con 0x7f737c06c2f0 2023-03-03T14:46:31.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.917+0000 7f7380bfd700 1 -- 172.21.15.7:0/3228891036 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700297813 0 0) 0x7f737c06d780 con 0x7f737c151ab0 2023-03-03T14:46:31.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.917+0000 7f7382400700 1 --2- 172.21.15.7:0/3228891036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c06c2f0 0x7f737c1584d0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f7374009dc0 tx=0x7f737400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c40688e802b3ec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:31.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.917+0000 7f7380bfd700 1 -- 172.21.15.7:0/3228891036 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737400e070 con 0x7f737c06c2f0 2023-03-03T14:46:31.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.917+0000 7f7380bfd700 1 -- 172.21.15.7:0/3228891036 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737400bb90 con 0x7f737c06c2f0 2023-03-03T14:46:31.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.917+0000 7f7380bfd700 1 -- 172.21.15.7:0/3228891036 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374012430 con 0x7f737c06c2f0 2023-03-03T14:46:31.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.918+0000 7f7383e63700 1 -- 172.21.15.7:0/3228891036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c06c2f0 msgr2=0x7f737c1584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:31.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.918+0000 7f7383e63700 1 --2- 172.21.15.7:0/3228891036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c06c2f0 0x7f737c1584d0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f7374009dc0 tx=0x7f737400b330 comp rx=0 tx=0).stop 2023-03-03T14:46:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.918+0000 7f7383e63700 1 -- 172.21.15.7:0/3228891036 shutdown_connections 2023-03-03T14:46:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.918+0000 7f7383e63700 1 --2- 172.21.15.7:0/3228891036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c06c2f0 0x7f737c1584d0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.918+0000 7f7383e63700 1 -- 172.21.15.7:0/3228891036 >> 172.21.15.7:0/3228891036 conn(0x7f737c0a94d0 msgr2=0x7f737c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:46:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.918+0000 7f7383e63700 1 -- 172.21.15.7:0/3228891036 shutdown_connections 2023-03-03T14:46:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.918+0000 7f7383e63700 1 -- 172.21.15.7:0/3228891036 wait complete. 2023-03-03T14:46:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.919+0000 7f7383e63700 1 Processor -- start 2023-03-03T14:46:31.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.919+0000 7f7383e63700 1 -- start start 2023-03-03T14:46:31.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7383e63700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f737c06bea0 0x7f737c14f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:31.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7383e63700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f737c06c2f0 0x7f737c1eda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:31.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7383e63700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c151ab0 0x7f737c1edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:31.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7383e63700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c1ed5d0 con 0x7f737c151ab0 2023-03-03T14:46:31.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7383e63700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c0ad2b0 con 0x7f737c06c2f0 2023-03-03T14:46:31.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7383e63700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737c0acd30 con 0x7f737c06bea0 2023-03-03T14:46:31.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7382400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c151ab0 0x7f737c1edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7382400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c151ab0 0x7f737c1edfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50300/0 (socket says 172.21.15.7:50300) 2023-03-03T14:46:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7382400700 1 -- 172.21.15.7:0/1194721007 learned_addr learned my addr 172.21.15.7:0/1194721007 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:46:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7381bff700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f737c06bea0 0x7f737c14f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f73813fe700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f737c06c2f0 0x7f737c1eda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7382400700 1 -- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f737c06bea0 msgr2=0x7f737c14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7382400700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f737c06bea0 0x7f737c14f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:31.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7382400700 1 -- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f737c06c2f0 msgr2=0x7f737c1eda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7382400700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f737c06c2f0 0x7f737c1eda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7382400700 1 -- 172.21.15.7:0/1194721007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7374009a70 con 0x7f737c151ab0 2023-03-03T14:46:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f7381bff700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f737c06bea0 0x7f737c14f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:46:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.920+0000 7f73813fe700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f737c06c2f0 0x7f737c1eda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:46:31.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.921+0000 7f7382400700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c151ab0 0x7f737c1edfa0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f737c06bb50 tx=0x7f7374004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:31.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.921+0000 7f736e7fc700 1 -- 172.21.15.7:0/1194721007 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737401b8e0 con 0x7f737c151ab0 2023-03-03T14:46:31.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.921+0000 7f736e7fc700 1 -- 172.21.15.7:0/1194721007 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7374020070 con 0x7f737c151ab0 2023-03-03T14:46:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.921+0000 7f7383e63700 1 -- 172.21.15.7:0/1194721007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737c06d130 con 0x7f737c151ab0 2023-03-03T14:46:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.921+0000 7f736e7fc700 1 -- 172.21.15.7:0/1194721007 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737401bca0 con 0x7f737c151ab0 2023-03-03T14:46:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.921+0000 7f7383e63700 1 -- 172.21.15.7:0/1194721007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737c06ce50 con 0x7f737c151ab0 2023-03-03T14:46:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.922+0000 7f7383e63700 1 -- 172.21.15.7:0/1194721007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f737c06d410 con 0x7f737c151ab0 2023-03-03T14:46:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.922+0000 7f736e7fc700 1 -- 172.21.15.7:0/1194721007 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7374026400 con 0x7f737c151ab0 2023-03-03T14:46:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.922+0000 7f736e7fc700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73600682f0 0x7f736006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:46:31.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.923+0000 7f736e7fc700 1 -- 172.21.15.7:0/1194721007 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7374088370 con 0x7f737c151ab0 2023-03-03T14:46:31.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.923+0000 7f7381bff700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73600682f0 0x7f736006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:46:31.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.923+0000 7f7381bff700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73600682f0 0x7f736006a7a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f7378000f60 tx=0x7f7378007e50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:46:31.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:31.927+0000 7f736e7fc700 1 -- 172.21.15.7:0/1194721007 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f73740546b0 con 0x7f737c151ab0 2023-03-03T14:46:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.167+0000 7f7383e63700 1 -- 172.21.15.7:0/1194721007 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f737c074bb0 con 0x7f737c151ab0 2023-03-03T14:46:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.167+0000 7f736e7fc700 1 -- 172.21.15.7:0/1194721007 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7374057cd0 con 0x7f737c151ab0 2023-03-03T14:46:32.169 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:46:32.170 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:46:32.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.169+0000 7f7383e63700 1 -- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73600682f0 msgr2=0x7f736006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:32.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.169+0000 7f7383e63700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73600682f0 0x7f736006a7a0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f7378000f60 tx=0x7f7378007e50 comp rx=0 tx=0).stop 2023-03-03T14:46:32.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.169+0000 7f7383e63700 1 -- 172.21.15.7:0/1194721007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c151ab0 msgr2=0x7f737c1edfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:46:32.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.169+0000 7f7383e63700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c151ab0 0x7f737c1edfa0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f737c06bb50 tx=0x7f7374004ad0 comp rx=0 tx=0).stop 2023-03-03T14:46:32.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.170+0000 7f7383e63700 1 -- 172.21.15.7:0/1194721007 shutdown_connections 2023-03-03T14:46:32.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.170+0000 7f7383e63700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73600682f0 0x7f736006a7a0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:32.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.170+0000 7f7383e63700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f737c151ab0 0x7f737c1edfa0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:32.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.170+0000 7f7383e63700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f737c06bea0 0x7f737c14f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:32.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.170+0000 7f7383e63700 1 --2- 172.21.15.7:0/1194721007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f737c06c2f0 0x7f737c1eda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:46:32.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.170+0000 7f7383e63700 1 -- 172.21.15.7:0/1194721007 >> 172.21.15.7:0/1194721007 conn(0x7f737c0a94d0 msgr2=0x7f737c1569f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:46:32.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.170+0000 7f7383e63700 1 -- 172.21.15.7:0/1194721007 shutdown_connections 2023-03-03T14:46:32.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:46:32.170+0000 7f7383e63700 1 -- 172.21.15.7:0/1194721007 wait complete. 2023-03-03T14:46:32.173 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:46:32.183 INFO:tasks.mds_thrash.fs.[b]:waiting for 72 secs before thrashing 2023-03-03T14:46:56.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:46:56.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:08.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:47:08.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.350+0000 7f57cbc67700 1 Processor -- start 2023-03-03T14:47:08.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.350+0000 7f57cbc67700 1 -- start start 2023-03-03T14:47:08.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.351+0000 7f57cbc67700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f57c4073960 0x7f57c41524e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:08.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.351+0000 7f57cbc67700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f57c414ffa0 0x7f57c4152a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:08.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.351+0000 7f57cbc67700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c40acb10 con 0x7f57c4073d30 2023-03-03T14:47:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.351+0000 7f57cbc67700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c40c2b70 con 0x7f57c4073960 2023-03-03T14:47:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.351+0000 7f57cbc67700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c4155bd0 con 0x7f57c414ffa0 2023-03-03T14:47:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.351+0000 7f57c9a03700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f57c4073960 0x7f57c41524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.351+0000 7f57c9202700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f57c4073d30 0x7f57c414f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44300/0 (socket says 172.21.15.7:44300) 2023-03-03T14:47:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.351+0000 7f57c9a03700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f57c4073960 0x7f57c41524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50320/0 (socket says 172.21.15.7:50320) 2023-03-03T14:47:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.351+0000 7f57c9202700 1 -- 172.21.15.7:0/3807647036 learned_addr learned my addr 172.21.15.7:0/3807647036 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:47:08.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.352+0000 7f57ca204700 1 --2- 172.21.15.7:0/3807647036 >> v2:172.21.15.143:3300/0 conn(0x7f57c414ffa0 0x7f57c4152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:08.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.352+0000 7f57c9a03700 1 -- 172.21.15.7:0/3807647036 >> v2:172.21.15.143:3300/0 conn(0x7f57c414ffa0 msgr2=0x7f57c4152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:47:08.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.352+0000 7f57c9a03700 1 --2- 172.21.15.7:0/3807647036 >> v2:172.21.15.143:3300/0 conn(0x7f57c414ffa0 0x7f57c4152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:08.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.352+0000 7f57c9a03700 1 -- 172.21.15.7:0/3807647036 >> v1:172.21.15.7:6789/0 conn(0x7f57c4073d30 legacy=0x7f57c414f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:08.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.352+0000 7f57c9a03700 1 -- 172.21.15.7:0/3807647036 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c4155eb0 con 0x7f57c4073960 2023-03-03T14:47:08.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.352+0000 7f57c8a01700 1 -- 172.21.15.7:0/3807647036 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147795229 0 0) 0x7f57c40acb10 con 0x7f57c4073d30 2023-03-03T14:47:08.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.352+0000 7f57c9a03700 1 --2- 172.21.15.7:0/3807647036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 0x7f57c41524e0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f57c406bce0 tx=0x7f57b4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385e1742711a7dd7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:47:08.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.352+0000 7f57c8a01700 1 -- 172.21.15.7:0/3807647036 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4011830 con 0x7f57c4073960 2023-03-03T14:47:08.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.352+0000 7f57c8a01700 1 -- 172.21.15.7:0/3807647036 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b4011990 con 0x7f57c4073960 2023-03-03T14:47:08.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.352+0000 7f57c8a01700 1 -- 172.21.15.7:0/3807647036 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4011d20 con 0x7f57c4073960 2023-03-03T14:47:08.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.353+0000 7f57cbc67700 1 -- 172.21.15.7:0/3807647036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 msgr2=0x7f57c41524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:08.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.353+0000 7f57cbc67700 1 --2- 172.21.15.7:0/3807647036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 0x7f57c41524e0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f57c406bce0 tx=0x7f57b4009b20 comp rx=0 tx=0).stop 2023-03-03T14:47:08.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.353+0000 7f57cbc67700 1 -- 172.21.15.7:0/3807647036 shutdown_connections 2023-03-03T14:47:08.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.353+0000 7f57cbc67700 1 --2- 172.21.15.7:0/3807647036 >> v2:172.21.15.143:3300/0 conn(0x7f57c414ffa0 0x7f57c4152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:08.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.353+0000 7f57cbc67700 1 --2- 172.21.15.7:0/3807647036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 0x7f57c41524e0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:08.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.353+0000 7f57cbc67700 1 -- 172.21.15.7:0/3807647036 >> 172.21.15.7:0/3807647036 conn(0x7f57c40a94d0 msgr2=0x7f57c40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:47:08.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.353+0000 7f57cbc67700 1 -- 172.21.15.7:0/3807647036 shutdown_connections 2023-03-03T14:47:08.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.353+0000 7f57cbc67700 1 -- 172.21.15.7:0/3807647036 wait complete. 2023-03-03T14:47:08.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.354+0000 7f57cbc67700 1 Processor -- start 2023-03-03T14:47:08.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.354+0000 7f57cbc67700 1 -- start start 2023-03-03T14:47:08.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57cbc67700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 0x7f57c40be6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:08.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57cbc67700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57c4073d30 0x7f57c41484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:08.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57cbc67700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c414ffa0 0x7f57c4148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:08.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57cbc67700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c41ece00 con 0x7f57c4073960 2023-03-03T14:47:08.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57cbc67700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c406c7b0 con 0x7f57c414ffa0 2023-03-03T14:47:08.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57cbc67700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4148f70 con 0x7f57c4073d30 2023-03-03T14:47:08.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57c9a03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 0x7f57c40be6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:08.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57c9a03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 0x7f57c40be6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50324/0 (socket says 172.21.15.7:50324) 2023-03-03T14:47:08.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57c9a03700 1 -- 172.21.15.7:0/852337773 learned_addr learned my addr 172.21.15.7:0/852337773 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:47:08.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57ca204700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c414ffa0 0x7f57c4148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:08.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.355+0000 7f57c9202700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57c4073d30 0x7f57c41484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:08.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57c9a03700 1 -- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57c4073d30 msgr2=0x7f57c41484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:08.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57c9a03700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57c4073d30 0x7f57c41484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:08.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57c9a03700 1 -- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c414ffa0 msgr2=0x7f57c4148a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:08.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57c9a03700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c414ffa0 0x7f57c4148a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:08.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57c9a03700 1 -- 172.21.15.7:0/852337773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40094a0 con 0x7f57c4073960 2023-03-03T14:47:08.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57c9202700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57c4073d30 0x7f57c41484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:47:08.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57c9a03700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 0x7f57c40be6a0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f57c406e440 tx=0x7f57b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:47:08.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57ba7fc700 1 -- 172.21.15.7:0/852337773 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b40036a0 con 0x7f57c4073960 2023-03-03T14:47:08.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57ba7fc700 1 -- 172.21.15.7:0/852337773 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b401f070 con 0x7f57c4073960 2023-03-03T14:47:08.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57cbc67700 1 -- 172.21.15.7:0/852337773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c40c1ce0 con 0x7f57c4073960 2023-03-03T14:47:08.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57ba7fc700 1 -- 172.21.15.7:0/852337773 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4003a20 con 0x7f57c4073960 2023-03-03T14:47:08.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.356+0000 7f57cbc67700 1 -- 172.21.15.7:0/852337773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c4156190 con 0x7f57c4073960 2023-03-03T14:47:08.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.357+0000 7f57cbc67700 1 -- 172.21.15.7:0/852337773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c40c1e70 con 0x7f57c4073960 2023-03-03T14:47:08.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.357+0000 7f57ba7fc700 1 -- 172.21.15.7:0/852337773 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f57b402d070 con 0x7f57c4073960 2023-03-03T14:47:08.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.358+0000 7f57ba7fc700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57a8068320 0x7f57a806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:08.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.358+0000 7f57ba7fc700 1 -- 172.21.15.7:0/852337773 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f57b4021070 con 0x7f57c4073960 2023-03-03T14:47:08.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.358+0000 7f57c9202700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57a8068320 0x7f57a806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:08.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.359+0000 7f57c9202700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57a8068320 0x7f57a806a7d0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f57c4155990 tx=0x7f57c0007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:47:08.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.363+0000 7f57ba7fc700 1 -- 172.21.15.7:0/852337773 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57b40539e0 con 0x7f57c4073960 2023-03-03T14:47:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.603+0000 7f57cbc67700 1 -- 172.21.15.7:0/852337773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57c41ed0a0 con 0x7f57c4073960 2023-03-03T14:47:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.604+0000 7f57ba7fc700 1 -- 172.21.15.7:0/852337773 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f57b4057000 con 0x7f57c4073960 2023-03-03T14:47:08.607 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:47:08.607 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:47:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 -- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57a8068320 msgr2=0x7f57a806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57a8068320 0x7f57a806a7d0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f57c4155990 tx=0x7f57c0007ca0 comp rx=0 tx=0).stop 2023-03-03T14:47:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 -- 172.21.15.7:0/852337773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 msgr2=0x7f57c40be6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:08.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 0x7f57c40be6a0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f57c406e440 tx=0x7f57b4004750 comp rx=0 tx=0).stop 2023-03-03T14:47:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 -- 172.21.15.7:0/852337773 shutdown_connections 2023-03-03T14:47:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c414ffa0 0x7f57c4148a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57a8068320 0x7f57a806a7d0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57c4073960 0x7f57c40be6a0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 --2- 172.21.15.7:0/852337773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57c4073d30 0x7f57c41484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 -- 172.21.15.7:0/852337773 >> 172.21.15.7:0/852337773 conn(0x7f57c40a94d0 msgr2=0x7f57c406a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:47:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 -- 172.21.15.7:0/852337773 shutdown_connections 2023-03-03T14:47:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:08.606+0000 7f57cbc67700 1 -- 172.21.15.7:0/852337773 wait complete. 2023-03-03T14:47:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:47:08.620 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before thrashing 2023-03-03T14:47:13.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:47:13.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.758+0000 7f79eb66a700 1 Processor -- start 2023-03-03T14:47:13.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.758+0000 7f79eb66a700 1 -- start start 2023-03-03T14:47:13.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79eb66a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f79e414f9d0 0x7f79e4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:13.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79eb66a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f79e414fda0 0x7f79e4071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:13.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79eb66a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 0x7f79e41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:13.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79eb66a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e40c2f30 con 0x7f79e414f9d0 2023-03-03T14:47:13.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79eb66a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e40c31d0 con 0x7f79e414fda0 2023-03-03T14:47:13.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79eb66a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e41524a0 con 0x7f79e40abe20 2023-03-03T14:47:13.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e9406700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f79e414f9d0 0x7f79e4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:13.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e9406700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f79e414f9d0 0x7f79e4156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50330/0 (socket says 172.21.15.7:50330) 2023-03-03T14:47:13.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e9406700 1 -- 172.21.15.7:0/1937216831 learned_addr learned my addr 172.21.15.7:0/1937216831 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:47:13.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e9c07700 1 --2- 172.21.15.7:0/1937216831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 0x7f79e41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:13.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e8c05700 1 --2- 172.21.15.7:0/1937216831 >> v2:172.21.15.143:3300/0 conn(0x7f79e414fda0 0x7f79e4071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:13.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e9406700 1 -- 172.21.15.7:0/1937216831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 msgr2=0x7f79e41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:47:13.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e9406700 1 --2- 172.21.15.7:0/1937216831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 0x7f79e41529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:13.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e9406700 1 -- 172.21.15.7:0/1937216831 >> v2:172.21.15.143:3300/0 conn(0x7f79e414fda0 msgr2=0x7f79e4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:47:13.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e9406700 1 --2- 172.21.15.7:0/1937216831 >> v2:172.21.15.143:3300/0 conn(0x7f79e414fda0 0x7f79e4071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:13.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e9406700 1 -- 172.21.15.7:0/1937216831 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e40c34b0 con 0x7f79e414f9d0 2023-03-03T14:47:13.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.759+0000 7f79e9c07700 1 --2- 172.21.15.7:0/1937216831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 0x7f79e41529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:47:13.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79e9406700 1 --2- 172.21.15.7:0/1937216831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 0x7f79e4156430 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f79e40709b0 tx=0x7f79e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c585eea0a1dde6d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:47:13.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79dbfff700 1 -- 172.21.15.7:0/1937216831 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e000b870 con 0x7f79e414f9d0 2023-03-03T14:47:13.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79dbfff700 1 -- 172.21.15.7:0/1937216831 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e000b9d0 con 0x7f79e414f9d0 2023-03-03T14:47:13.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79dbfff700 1 -- 172.21.15.7:0/1937216831 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e000bd60 con 0x7f79e414f9d0 2023-03-03T14:47:13.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79eb66a700 1 -- 172.21.15.7:0/1937216831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 msgr2=0x7f79e4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:13.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79eb66a700 1 --2- 172.21.15.7:0/1937216831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 0x7f79e4156430 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f79e40709b0 tx=0x7f79e0009bd0 comp rx=0 tx=0).stop 2023-03-03T14:47:13.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79eb66a700 1 -- 172.21.15.7:0/1937216831 shutdown_connections 2023-03-03T14:47:13.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79eb66a700 1 --2- 172.21.15.7:0/1937216831 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 0x7f79e41529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:13.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79eb66a700 1 --2- 172.21.15.7:0/1937216831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 0x7f79e4156430 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:13.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79eb66a700 1 --2- 172.21.15.7:0/1937216831 >> v2:172.21.15.143:3300/0 conn(0x7f79e414fda0 0x7f79e4071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:13.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79eb66a700 1 -- 172.21.15.7:0/1937216831 >> 172.21.15.7:0/1937216831 conn(0x7f79e40c8900 msgr2=0x7f79e40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:47:13.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79eb66a700 1 -- 172.21.15.7:0/1937216831 shutdown_connections 2023-03-03T14:47:13.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.760+0000 7f79eb66a700 1 -- 172.21.15.7:0/1937216831 wait complete. 2023-03-03T14:47:13.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79eb66a700 1 Processor -- start 2023-03-03T14:47:13.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79eb66a700 1 -- start start 2023-03-03T14:47:13.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79eb66a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 0x7f79e406ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:13.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79eb66a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 0x7f79e414bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:13.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79eb66a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79e414fda0 0x7f79e414c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:13.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79eb66a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40bf2b0 con 0x7f79e414f9d0 2023-03-03T14:47:13.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79eb66a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e4154f90 con 0x7f79e414fda0 2023-03-03T14:47:13.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79eb66a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e41e7e60 con 0x7f79e40abe20 2023-03-03T14:47:13.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79e8c05700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 0x7f79e414bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:13.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79e8c05700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 0x7f79e414bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50338/0 (socket says 172.21.15.7:50338) 2023-03-03T14:47:13.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.762+0000 7f79e8c05700 1 -- 172.21.15.7:0/2853401332 learned_addr learned my addr 172.21.15.7:0/2853401332 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:47:13.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79e9c07700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79e414fda0 0x7f79e414c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:13.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79e9406700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 0x7f79e406ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:13.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79e8c05700 1 -- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 msgr2=0x7f79e406ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:13.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79e8c05700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 0x7f79e406ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:13.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79e8c05700 1 -- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79e414fda0 msgr2=0x7f79e414c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:13.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79e8c05700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79e414fda0 0x7f79e414c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:13.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79e8c05700 1 -- 172.21.15.7:0/2853401332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e0009550 con 0x7f79e414f9d0 2023-03-03T14:47:13.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79e9406700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 0x7f79e406ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:47:13.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79e9c07700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79e414fda0 0x7f79e414c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:47:13.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79e8c05700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 0x7f79e414bee0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f79d400e840 tx=0x7f79d400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:47:13.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79d9ffb700 1 -- 172.21.15.7:0/2853401332 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d4013610 con 0x7f79e414f9d0 2023-03-03T14:47:13.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79d9ffb700 1 -- 172.21.15.7:0/2853401332 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d4013770 con 0x7f79e414f9d0 2023-03-03T14:47:13.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79d9ffb700 1 -- 172.21.15.7:0/2853401332 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d4013b00 con 0x7f79e414f9d0 2023-03-03T14:47:13.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79eb66a700 1 -- 172.21.15.7:0/2853401332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e40ac350 con 0x7f79e414f9d0 2023-03-03T14:47:13.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.763+0000 7f79eb66a700 1 -- 172.21.15.7:0/2853401332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e40ab630 con 0x7f79e414f9d0 2023-03-03T14:47:13.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.764+0000 7f79eb66a700 1 -- 172.21.15.7:0/2853401332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e40c3e10 con 0x7f79e414f9d0 2023-03-03T14:47:13.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.765+0000 7f79d9ffb700 1 -- 172.21.15.7:0/2853401332 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f79d4023070 con 0x7f79e414f9d0 2023-03-03T14:47:13.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.765+0000 7f79d9ffb700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79c80682f0 0x7f79c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:13.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.765+0000 7f79d9ffb700 1 -- 172.21.15.7:0/2853401332 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f79d4085d20 con 0x7f79e414f9d0 2023-03-03T14:47:13.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.765+0000 7f79e9406700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79c80682f0 0x7f79c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:13.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.766+0000 7f79e9406700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79c80682f0 0x7f79c806a7a0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f79e0000c00 tx=0x7f79e0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:47:13.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:13.770+0000 7f79d9ffb700 1 -- 172.21.15.7:0/2853401332 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f79d40520c0 con 0x7f79e414f9d0 2023-03-03T14:47:14.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.008+0000 7f79eb66a700 1 -- 172.21.15.7:0/2853401332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79e41eaa00 con 0x7f79e414f9d0 2023-03-03T14:47:14.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.009+0000 7f79d9ffb700 1 -- 172.21.15.7:0/2853401332 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f79e41eaa00 con 0x7f79e414f9d0 2023-03-03T14:47:14.012 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:47:14.012 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:47:14.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 -- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79c80682f0 msgr2=0x7f79c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:14.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79c80682f0 0x7f79c806a7a0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f79e0000c00 tx=0x7f79e0008030 comp rx=0 tx=0).stop 2023-03-03T14:47:14.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 -- 172.21.15.7:0/2853401332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 msgr2=0x7f79e414bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:14.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 0x7f79e414bee0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f79d400e840 tx=0x7f79d400ec00 comp rx=0 tx=0).stop 2023-03-03T14:47:14.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 -- 172.21.15.7:0/2853401332 shutdown_connections 2023-03-03T14:47:14.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79e414fda0 0x7f79e414c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:14.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79e40abe20 0x7f79e406ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:14.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79c80682f0 0x7f79c806a7a0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:14.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 --2- 172.21.15.7:0/2853401332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79e414f9d0 0x7f79e414bee0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:14.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 -- 172.21.15.7:0/2853401332 >> 172.21.15.7:0/2853401332 conn(0x7f79e40c8900 msgr2=0x7f79e40c6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:47:14.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 -- 172.21.15.7:0/2853401332 shutdown_connections 2023-03-03T14:47:14.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:14.011+0000 7f79eb66a700 1 -- 172.21.15.7:0/2853401332 wait complete. 2023-03-03T14:47:14.016 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:47:14.026 INFO:tasks.mds_thrash.fs.[a]:waiting for 104 secs before thrashing 2023-03-03T14:47:26.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:26.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:44.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:47:44.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.321+0000 7fec4553f700 1 Processor -- start 2023-03-03T14:47:44.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.322+0000 7fec4553f700 1 -- start start 2023-03-03T14:47:44.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.322+0000 7fec4553f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400caa30 0x7fec400c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:44.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.322+0000 7fec4553f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fec400c00e0 0x7fec40154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:44.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.322+0000 7fec4553f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec400cb280 con 0x7fec400cae00 2023-03-03T14:47:44.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.322+0000 7fec4553f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec400c7310 con 0x7fec400c00e0 2023-03-03T14:47:44.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.322+0000 7fec4553f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec400bfd10 con 0x7fec400caa30 2023-03-03T14:47:44.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3e7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fec400c00e0 0x7fec40154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:44.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400caa30 0x7fec400c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:44.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3f7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fec400cae00 0x7fec400c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:47858/0 (socket says 172.21.15.7:47858) 2023-03-03T14:47:44.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3f7fe700 1 -- 172.21.15.7:0/3223424247 learned_addr learned my addr 172.21.15.7:0/3223424247 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:47:44.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3effd700 1 -- 172.21.15.7:0/3223424247 >> v2:172.21.15.143:3300/0 conn(0x7fec400c00e0 msgr2=0x7fec40154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:47:44.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3effd700 1 --2- 172.21.15.7:0/3223424247 >> v2:172.21.15.143:3300/0 conn(0x7fec400c00e0 0x7fec40154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:44.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3effd700 1 -- 172.21.15.7:0/3223424247 >> v1:172.21.15.143:6789/0 conn(0x7fec400cae00 legacy=0x7fec400c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:44.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3effd700 1 -- 172.21.15.7:0/3223424247 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400bfa30 con 0x7fec400caa30 2023-03-03T14:47:44.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3dffb700 1 -- 172.21.15.7:0/3223424247 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482323401 0 0) 0x7fec400cb280 con 0x7fec400cae00 2023-03-03T14:47:44.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3e7fc700 1 --2- 172.21.15.7:0/3223424247 >> v2:172.21.15.143:3300/0 conn(0x7fec400c00e0 0x7fec40154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:47:44.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.323+0000 7fec3effd700 1 --2- 172.21.15.7:0/3223424247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400caa30 0x7fec400c5c60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fec30009840 tx=0x7fec30009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1f5d29c1bd7686a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:47:44.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec3dffb700 1 -- 172.21.15.7:0/3223424247 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30011830 con 0x7fec400caa30 2023-03-03T14:47:44.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec3dffb700 1 -- 172.21.15.7:0/3223424247 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec30011990 con 0x7fec400caa30 2023-03-03T14:47:44.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec3dffb700 1 -- 172.21.15.7:0/3223424247 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30011d20 con 0x7fec400caa30 2023-03-03T14:47:44.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec4553f700 1 -- 172.21.15.7:0/3223424247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400caa30 msgr2=0x7fec400c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:44.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec4553f700 1 --2- 172.21.15.7:0/3223424247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400caa30 0x7fec400c5c60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fec30009840 tx=0x7fec30009b50 comp rx=0 tx=0).stop 2023-03-03T14:47:44.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec4553f700 1 -- 172.21.15.7:0/3223424247 shutdown_connections 2023-03-03T14:47:44.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec4553f700 1 --2- 172.21.15.7:0/3223424247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400caa30 0x7fec400c5c60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:44.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec4553f700 1 --2- 172.21.15.7:0/3223424247 >> v2:172.21.15.143:3300/0 conn(0x7fec400c00e0 0x7fec40154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:44.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec4553f700 1 -- 172.21.15.7:0/3223424247 >> 172.21.15.7:0/3223424247 conn(0x7fec40066e40 msgr2=0x7fec40071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:47:44.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec4553f700 1 -- 172.21.15.7:0/3223424247 shutdown_connections 2023-03-03T14:47:44.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.324+0000 7fec4553f700 1 -- 172.21.15.7:0/3223424247 wait complete. 2023-03-03T14:47:44.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.325+0000 7fec4553f700 1 Processor -- start 2023-03-03T14:47:44.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec4553f700 1 -- start start 2023-03-03T14:47:44.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec4553f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec400c00e0 0x7fec401ed060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:44.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec4553f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec400caa30 0x7fec40157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec4553f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400cae00 0x7fec40157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec4553f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4014b930 con 0x7fec400caa30 2023-03-03T14:47:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec4553f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec400c4320 con 0x7fec400c00e0 2023-03-03T14:47:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec4553f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec401edd30 con 0x7fec400cae00 2023-03-03T14:47:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec3e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec400caa30 0x7fec40157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec3e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec400caa30 0x7fec40157700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50364/0 (socket says 172.21.15.7:50364) 2023-03-03T14:47:44.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec3e7fc700 1 -- 172.21.15.7:0/2559983978 learned_addr learned my addr 172.21.15.7:0/2559983978 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:47:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.326+0000 7fec3effd700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec400c00e0 0x7fec401ed060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec3f7fe700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400cae00 0x7fec40157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec3e7fc700 1 -- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400cae00 msgr2=0x7fec40157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec3e7fc700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400cae00 0x7fec40157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec3e7fc700 1 -- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec400c00e0 msgr2=0x7fec401ed060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec3e7fc700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec400c00e0 0x7fec401ed060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:44.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec3e7fc700 1 -- 172.21.15.7:0/2559983978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec28009430 con 0x7fec400caa30 2023-03-03T14:47:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec3e7fc700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec400caa30 0x7fec40157700 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fec2800ebe0 tx=0x7fec2800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:47:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec377fe700 1 -- 172.21.15.7:0/2559983978 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec28005a20 con 0x7fec400caa30 2023-03-03T14:47:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec377fe700 1 -- 172.21.15.7:0/2559983978 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec28005b80 con 0x7fec400caa30 2023-03-03T14:47:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec4553f700 1 -- 172.21.15.7:0/2559983978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec300094a0 con 0x7fec400caa30 2023-03-03T14:47:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec377fe700 1 -- 172.21.15.7:0/2559983978 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2800c850 con 0x7fec400caa30 2023-03-03T14:47:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.327+0000 7fec4553f700 1 -- 172.21.15.7:0/2559983978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec401547f0 con 0x7fec400caa30 2023-03-03T14:47:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.328+0000 7fec4553f700 1 -- 172.21.15.7:0/2559983978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec40154510 con 0x7fec400caa30 2023-03-03T14:47:44.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.329+0000 7fec377fe700 1 -- 172.21.15.7:0/2559983978 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fec28022070 con 0x7fec400caa30 2023-03-03T14:47:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.329+0000 7fec377fe700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fec200682f0 0x7fec2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:47:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.329+0000 7fec377fe700 1 -- 172.21.15.7:0/2559983978 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fec280862a0 con 0x7fec400caa30 2023-03-03T14:47:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.329+0000 7fec3effd700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fec200682f0 0x7fec2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:47:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.330+0000 7fec3effd700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fec200682f0 0x7fec2006a7a0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fec30009fd0 tx=0x7fec30008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:47:44.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.333+0000 7fec377fe700 1 -- 172.21.15.7:0/2559983978 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fec280525b0 con 0x7fec400caa30 2023-03-03T14:47:44.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.575+0000 7fec4553f700 1 -- 172.21.15.7:0/2559983978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec40154ad0 con 0x7fec400caa30 2023-03-03T14:47:44.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.576+0000 7fec377fe700 1 -- 172.21.15.7:0/2559983978 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fec28020020 con 0x7fec400caa30 2023-03-03T14:47:44.578 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:47:44.578 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:47:44.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 -- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fec200682f0 msgr2=0x7fec2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:44.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fec200682f0 0x7fec2006a7a0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fec30009fd0 tx=0x7fec30008030 comp rx=0 tx=0).stop 2023-03-03T14:47:44.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 -- 172.21.15.7:0/2559983978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec400caa30 msgr2=0x7fec40157700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:47:44.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec400caa30 0x7fec40157700 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fec2800ebe0 tx=0x7fec2800eef0 comp rx=0 tx=0).stop 2023-03-03T14:47:44.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 -- 172.21.15.7:0/2559983978 shutdown_connections 2023-03-03T14:47:44.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fec400cae00 0x7fec40157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:44.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec400c00e0 0x7fec401ed060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:44.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fec200682f0 0x7fec2006a7a0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:44.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 --2- 172.21.15.7:0/2559983978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec400caa30 0x7fec40157700 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:47:44.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 -- 172.21.15.7:0/2559983978 >> 172.21.15.7:0/2559983978 conn(0x7fec40066e40 msgr2=0x7fec40153d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:47:44.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 -- 172.21.15.7:0/2559983978 shutdown_connections 2023-03-03T14:47:44.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:47:44.578+0000 7fec4553f700 1 -- 172.21.15.7:0/2559983978 wait complete. 2023-03-03T14:47:44.582 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:47:44.591 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T14:47:57.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:47:57.092 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:27.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:27.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:46.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:48:46.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.730+0000 7f0b6b25e700 1 Processor -- start 2023-03-03T14:48:46.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.730+0000 7f0b6b25e700 1 -- start start 2023-03-03T14:48:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b6b25e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b64155410 0x7f0b640bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b6b25e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0b640c5b00 0x7f0b640bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b6b25e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b640c08d0 con 0x7f0b64151e80 2023-03-03T14:48:46.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b6b25e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b64152250 con 0x7f0b640c5b00 2023-03-03T14:48:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b6b25e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b640c0fb0 con 0x7f0b64155410 2023-03-03T14:48:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b68ffa700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0b64151e80 0x7f0b640c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44392/0 (socket says 172.21.15.7:44392) 2023-03-03T14:48:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b68ffa700 1 -- 172.21.15.7:0/3517177661 learned_addr learned my addr 172.21.15.7:0/3517177661 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:48:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815269287 0 0) 0x7f0b640c08d0 con 0x7f0b64151e80 2023-03-03T14:48:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b540035b0 con 0x7f0b64151e80 2023-03-03T14:48:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b63fff700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b64155410 0x7f0b640bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b697fb700 1 --2- 172.21.15.7:0/3517177661 >> v2:172.21.15.143:3300/0 conn(0x7f0b640c5b00 0x7f0b640bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3675664388 0 0) 0x7f0b540035b0 con 0x7f0b64151e80 2023-03-03T14:48:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b64155410 msgr2=0x7f0b640bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:48:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b637fe700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b64155410 0x7f0b640bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 >> v2:172.21.15.143:3300/0 conn(0x7f0b640c5b00 msgr2=0x7f0b640bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:48:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b637fe700 1 --2- 172.21.15.7:0/3517177661 >> v2:172.21.15.143:3300/0 conn(0x7f0b640c5b00 0x7f0b640bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b64154930 con 0x7f0b64151e80 2023-03-03T14:48:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.731+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0b58001670 con 0x7f0b64151e80 2023-03-03T14:48:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 >> v1:172.21.15.7:6789/0 conn(0x7f0b64151e80 legacy=0x7f0b640c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b637fe700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b54003aa0 0x7f0b54003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b637fe700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54004850 0x7f0b54004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b637fe700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b64155410 0x7f0b54005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5400ba10 con 0x7f0b54004850 2023-03-03T14:48:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5400bb50 con 0x7f0b54003aa0 2023-03-03T14:48:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5400bc90 con 0x7f0b64155410 2023-03-03T14:48:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b63fff700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54004850 0x7f0b54004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b68ffa700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b54003aa0 0x7f0b54003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b697fb700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b64155410 0x7f0b54005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b63fff700 1 -- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b64155410 msgr2=0x7f0b54005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:48:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b63fff700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b64155410 0x7f0b54005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b63fff700 1 -- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b54003aa0 msgr2=0x7f0b54003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:48:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b63fff700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b54003aa0 0x7f0b54003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b63fff700 1 -- 172.21.15.7:0/3517177661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b540035b0 con 0x7f0b54004850 2023-03-03T14:48:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b63fff700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54004850 0x7f0b54004cc0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009b90 tx=0x7f0b54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4117aedbef8bcd1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:48:46.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5000e6c0 con 0x7f0b54004850 2023-03-03T14:48:46.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5000e820 con 0x7f0b54004850 2023-03-03T14:48:46.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.732+0000 7f0b637fe700 1 -- 172.21.15.7:0/3517177661 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5000eb90 con 0x7f0b54004850 2023-03-03T14:48:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.733+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3517177661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54004850 msgr2=0x7f0b54004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.733+0000 7f0b6b25e700 1 --2- 172.21.15.7:0/3517177661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b54004850 0x7f0b54004cc0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009b90 tx=0x7f0b54005880 comp rx=0 tx=0).stop 2023-03-03T14:48:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.733+0000 7f0b697fb700 1 -- 172.21.15.7:0/3517177661 reap_dead start 2023-03-03T14:48:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.733+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3517177661 shutdown_connections 2023-03-03T14:48:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.733+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3517177661 >> 172.21.15.7:0/3517177661 conn(0x7f0b64071630 msgr2=0x7f0b6406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:48:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.733+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3517177661 shutdown_connections 2023-03-03T14:48:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.733+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3517177661 wait complete. 2023-03-03T14:48:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.734+0000 7f0b6b25e700 1 Processor -- start 2023-03-03T14:48:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.734+0000 7f0b6b25e700 1 -- start start 2023-03-03T14:48:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b6b25e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b6414a320 0x7f0b641eecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b6b25e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b641584c0 0x7f0b6414a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b6b25e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b641578a0 0x7f0b64157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b6b25e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b641ef290 con 0x7f0b6414a320 2023-03-03T14:48:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b6b25e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6414d5d0 con 0x7f0b641578a0 2023-03-03T14:48:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b6b25e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b64155c50 con 0x7f0b641584c0 2023-03-03T14:48:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b68ffa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b6414a320 0x7f0b641eecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b68ffa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b6414a320 0x7f0b641eecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50424/0 (socket says 172.21.15.7:50424) 2023-03-03T14:48:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b68ffa700 1 -- 172.21.15.7:0/3152164758 learned_addr learned my addr 172.21.15.7:0/3152164758 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:48:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b63fff700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b641584c0 0x7f0b6414a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b697fb700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b641578a0 0x7f0b64157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b68ffa700 1 -- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b641584c0 msgr2=0x7f0b6414a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b68ffa700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b641584c0 0x7f0b6414a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b68ffa700 1 -- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b641578a0 msgr2=0x7f0b64157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b68ffa700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b641578a0 0x7f0b64157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.735+0000 7f0b68ffa700 1 -- 172.21.15.7:0/3152164758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b64154ef0 con 0x7f0b6414a320 2023-03-03T14:48:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.736+0000 7f0b68ffa700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b6414a320 0x7f0b641eecb0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f0b58007090 tx=0x7f0b580073a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:48:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.736+0000 7f0b617fa700 1 -- 172.21.15.7:0/3152164758 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58017070 con 0x7f0b6414a320 2023-03-03T14:48:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.736+0000 7f0b617fa700 1 -- 172.21.15.7:0/3152164758 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b58011580 con 0x7f0b6414a320 2023-03-03T14:48:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.736+0000 7f0b617fa700 1 -- 172.21.15.7:0/3152164758 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58011730 con 0x7f0b6414a320 2023-03-03T14:48:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.736+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3152164758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b64154c10 con 0x7f0b6414a320 2023-03-03T14:48:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.736+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3152164758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b640c5220 con 0x7f0b6414a320 2023-03-03T14:48:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.737+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3152164758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b64156860 con 0x7f0b6414a320 2023-03-03T14:48:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.737+0000 7f0b617fa700 1 -- 172.21.15.7:0/3152164758 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0b58011890 con 0x7f0b6414a320 2023-03-03T14:48:46.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.738+0000 7f0b617fa700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b480682f0 0x7f0b4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:46.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.738+0000 7f0b617fa700 1 -- 172.21.15.7:0/3152164758 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0b58085bf0 con 0x7f0b6414a320 2023-03-03T14:48:46.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.738+0000 7f0b63fff700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b480682f0 0x7f0b4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:46.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.738+0000 7f0b63fff700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b480682f0 0x7f0b4806a7a0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f0b500090b0 tx=0x7f0b50008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:48:46.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.742+0000 7f0b617fa700 1 -- 172.21.15.7:0/3152164758 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0b58051f90 con 0x7f0b6414a320 2023-03-03T14:48:46.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.978+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3152164758 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b64074bb0 con 0x7f0b6414a320 2023-03-03T14:48:46.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.978+0000 7f0b617fa700 1 -- 172.21.15.7:0/3152164758 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0b580555b0 con 0x7f0b6414a320 2023-03-03T14:48:46.980 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:48:46.980 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:48:46.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b480682f0 msgr2=0x7f0b4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:46.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b480682f0 0x7f0b4806a7a0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f0b500090b0 tx=0x7f0b50008030 comp rx=0 tx=0).stop 2023-03-03T14:48:46.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3152164758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b6414a320 msgr2=0x7f0b641eecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:46.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b6414a320 0x7f0b641eecb0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f0b58007090 tx=0x7f0b580073a0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3152164758 shutdown_connections 2023-03-03T14:48:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0b641578a0 0x7f0b64157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0b480682f0 0x7f0b4806a7a0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b6414a320 0x7f0b641eecb0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 --2- 172.21.15.7:0/3152164758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0b641584c0 0x7f0b6414a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3152164758 >> 172.21.15.7:0/3152164758 conn(0x7f0b64071630 msgr2=0x7f0b640c99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:48:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3152164758 shutdown_connections 2023-03-03T14:48:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:46.980+0000 7f0b6b25e700 1 -- 172.21.15.7:0/3152164758 wait complete. 2023-03-03T14:48:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:48:46.995 INFO:tasks.mds_thrash.fs.[b]:waiting for 115 secs before thrashing 2023-03-03T14:48:57.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:57.318 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:48:58.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:48:58.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.163+0000 7f0283891700 1 Processor -- start 2023-03-03T14:48:58.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.163+0000 7f0283891700 1 -- start start 2023-03-03T14:48:58.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.164+0000 7f0283891700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f027c072470 0x7f027c06b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:58.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.164+0000 7f0283891700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027c070360 con 0x7f027c14b460 2023-03-03T14:48:58.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.164+0000 7f0283891700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027c06a5e0 con 0x7f027c071990 2023-03-03T14:48:58.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.164+0000 7f0283891700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027c1521a0 con 0x7f027c072470 2023-03-03T14:48:58.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.164+0000 7f0280e2c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f027c14b460 0x7f027c06aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44416/0 (socket says 172.21.15.7:44416) 2023-03-03T14:48:58.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.164+0000 7f0280e2c700 1 -- 172.21.15.7:0/3143993495 learned_addr learned my addr 172.21.15.7:0/3143993495 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:48:58.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.164+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803542858 0 0) 0x7f027c070360 con 0x7f027c14b460 2023-03-03T14:48:58.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.164+0000 7f0281e2e700 1 --2- 172.21.15.7:0/3143993495 >> v2:172.21.15.143:3300/0 conn(0x7f027c072470 0x7f027c06b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:58.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.164+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02680035b0 con 0x7f027c14b460 2023-03-03T14:48:58.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2674291474 0 0) 0x7f02680035b0 con 0x7f027c14b460 2023-03-03T14:48:58.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 >> v2:172.21.15.143:3300/0 conn(0x7f027c072470 msgr2=0x7f027c06b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:48:58.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 --2- 172.21.15.7:0/3143993495 >> v2:172.21.15.143:3300/0 conn(0x7f027c072470 0x7f027c06b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:58.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 >> v1:172.21.15.143:6789/0 conn(0x7f027c071990 legacy=0x7f027c071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:58.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c0c38b0 con 0x7f027c14b460 2023-03-03T14:48:58.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0278001670 con 0x7f027c14b460 2023-03-03T14:48:58.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 >> v1:172.21.15.7:6789/0 conn(0x7f027c14b460 legacy=0x7f027c06aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:58.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0268003a70 0x7f0268003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:58.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f0268004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:58.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0268005250 0x7f026800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:58.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c1521a0 con 0x7f0268004850 2023-03-03T14:48:58.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026800c190 con 0x7f0268003a70 2023-03-03T14:48:58.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026800c350 con 0x7f0268005250 2023-03-03T14:48:58.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0280e2c700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f0268004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:58.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f028162d700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0268003a70 0x7f0268003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:58.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0281e2e700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0268005250 0x7f026800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:58.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0280e2c700 1 -- 172.21.15.7:0/3143993495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0268005250 msgr2=0x7f026800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:48:58.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0280e2c700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0268005250 0x7f026800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:58.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0280e2c700 1 -- 172.21.15.7:0/3143993495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0268003a70 msgr2=0x7f0268003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:48:58.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0280e2c700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0268003a70 0x7f0268003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:58.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0280e2c700 1 -- 172.21.15.7:0/3143993495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02680035b0 con 0x7f0268004850 2023-03-03T14:48:58.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0281e2e700 1 -- 172.21.15.7:0/3143993495 reap_dead start 2023-03-03T14:48:58.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0280e2c700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f0268004cc0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f0278001eb0 tx=0x7f0268005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e9f2d280e545e6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:48:58.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.165+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278007c10 con 0x7f0268004850 2023-03-03T14:48:58.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.166+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027800cd00 con 0x7f0268004850 2023-03-03T14:48:58.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.166+0000 7f0273fff700 1 -- 172.21.15.7:0/3143993495 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278005ba0 con 0x7f0268004850 2023-03-03T14:48:58.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.166+0000 7f0283891700 1 -- 172.21.15.7:0/3143993495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 msgr2=0x7f0268004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:58.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.166+0000 7f0283891700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f0268004cc0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f0278001eb0 tx=0x7f0268005880 comp rx=0 tx=0).stop 2023-03-03T14:48:58.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.166+0000 7f0283891700 1 -- 172.21.15.7:0/3143993495 shutdown_connections 2023-03-03T14:48:58.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.166+0000 7f0283891700 1 --2- 172.21.15.7:0/3143993495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f0268004cc0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:58.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.166+0000 7f0283891700 1 -- 172.21.15.7:0/3143993495 >> 172.21.15.7:0/3143993495 conn(0x7f027c0a94d0 msgr2=0x7f027c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:48:58.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.166+0000 7f0283891700 1 -- 172.21.15.7:0/3143993495 shutdown_connections 2023-03-03T14:48:58.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.166+0000 7f0283891700 1 -- 172.21.15.7:0/3143993495 wait complete. 2023-03-03T14:48:58.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.167+0000 7f0283891700 1 Processor -- start 2023-03-03T14:48:58.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.167+0000 7f0283891700 1 -- start start 2023-03-03T14:48:58.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f0283891700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f027c143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:58.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f0283891700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c143fd0 0x7f027c13f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:58.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f0283891700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c13e880 0x7f027c13f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:58.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f0283891700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0be760 con 0x7f0268004850 2023-03-03T14:48:58.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f0283891700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0bfb80 con 0x7f027c13e880 2023-03-03T14:48:58.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f0283891700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0acf20 con 0x7f027c143fd0 2023-03-03T14:48:58.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f028162d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f027c143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:58.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f028162d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f027c143a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50446/0 (socket says 172.21.15.7:50446) 2023-03-03T14:48:58.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f028162d700 1 -- 172.21.15.7:0/2700144891 learned_addr learned my addr 172.21.15.7:0/2700144891 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:48:58.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f0281e2e700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c13e880 0x7f027c13f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:58.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.168+0000 7f0280e2c700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c143fd0 0x7f027c13f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:58.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f028162d700 1 -- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c143fd0 msgr2=0x7f027c13f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:58.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f028162d700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c143fd0 0x7f027c13f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:58.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f028162d700 1 -- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c13e880 msgr2=0x7f027c13f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:58.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f028162d700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c13e880 0x7f027c13f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:58.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f028162d700 1 -- 172.21.15.7:0/2700144891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c0c3b50 con 0x7f0268004850 2023-03-03T14:48:58.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f0280e2c700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c143fd0 0x7f027c13f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:48:58.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f028162d700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f027c143a90 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f026c00f850 tx=0x7f026c00fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:48:58.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f0271ffb700 1 -- 172.21.15.7:0/2700144891 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c0052e0 con 0x7f0268004850 2023-03-03T14:48:58.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f0271ffb700 1 -- 172.21.15.7:0/2700144891 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c005440 con 0x7f0268004850 2023-03-03T14:48:58.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f0283891700 1 -- 172.21.15.7:0/2700144891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027c06a2b0 con 0x7f0268004850 2023-03-03T14:48:58.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f0271ffb700 1 -- 172.21.15.7:0/2700144891 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c0038a0 con 0x7f0268004850 2023-03-03T14:48:58.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.169+0000 7f0283891700 1 -- 172.21.15.7:0/2700144891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027c151c00 con 0x7f0268004850 2023-03-03T14:48:58.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.170+0000 7f0283891700 1 -- 172.21.15.7:0/2700144891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c074bb0 con 0x7f0268004850 2023-03-03T14:48:58.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.171+0000 7f0271ffb700 1 -- 172.21.15.7:0/2700144891 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f026c003a00 con 0x7f0268004850 2023-03-03T14:48:58.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.171+0000 7f0271ffb700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02600682f0 0x7f026006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:48:58.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.171+0000 7f0271ffb700 1 -- 172.21.15.7:0/2700144891 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f026c085ee0 con 0x7f0268004850 2023-03-03T14:48:58.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.171+0000 7f0280e2c700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02600682f0 0x7f026006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:48:58.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.172+0000 7f0280e2c700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02600682f0 0x7f026006a7a0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f0278015040 tx=0x7f027800a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:48:58.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.175+0000 7f0271ffb700 1 -- 172.21.15.7:0/2700144891 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f026c052280 con 0x7f0268004850 2023-03-03T14:48:58.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.413+0000 7f0283891700 1 -- 172.21.15.7:0/2700144891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f027c0c16a0 con 0x7f0268004850 2023-03-03T14:48:58.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.414+0000 7f0271ffb700 1 -- 172.21.15.7:0/2700144891 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f026c0558a0 con 0x7f0268004850 2023-03-03T14:48:58.415 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:48:58.416 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:48:58.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 -- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02600682f0 msgr2=0x7f026006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:58.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02600682f0 0x7f026006a7a0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f0278015040 tx=0x7f027800a000 comp rx=0 tx=0).stop 2023-03-03T14:48:58.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 -- 172.21.15.7:0/2700144891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 msgr2=0x7f027c143a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:48:58.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f027c143a90 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f026c00f850 tx=0x7f026c00fb60 comp rx=0 tx=0).stop 2023-03-03T14:48:58.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 -- 172.21.15.7:0/2700144891 shutdown_connections 2023-03-03T14:48:58.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f027c13e880 0x7f027c13f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:58.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02600682f0 0x7f026006a7a0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:58.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0268004850 0x7f027c143a90 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:58.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 --2- 172.21.15.7:0/2700144891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f027c143fd0 0x7f027c13f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:48:58.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 -- 172.21.15.7:0/2700144891 >> 172.21.15.7:0/2700144891 conn(0x7f027c0a94d0 msgr2=0x7f027c0c1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:48:58.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 -- 172.21.15.7:0/2700144891 shutdown_connections 2023-03-03T14:48:58.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:48:58.416+0000 7f0283891700 1 -- 172.21.15.7:0/2700144891 wait complete. 2023-03-03T14:48:58.419 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:48:58.429 INFO:tasks.mds_thrash.fs.[a]:waiting for 31 secs before thrashing 2023-03-03T14:49:27.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:27.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:29.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:49:29.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.567+0000 7f563fbeb700 1 Processor -- start 2023-03-03T14:49:29.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.567+0000 7f563fbeb700 1 -- start start 2023-03-03T14:49:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.568+0000 7f563fbeb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 0x7f56380717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.568+0000 7f563fbeb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f56380aad20 0x7f5638157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.568+0000 7f563fbeb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5638071d20 0x7f56381581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.568+0000 7f563fbeb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56380728b0 con 0x7f56380aad20 2023-03-03T14:49:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.568+0000 7f563fbeb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5638155070 con 0x7f5638071d20 2023-03-03T14:49:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.568+0000 7f563fbeb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563806f5b0 con 0x7f563814b020 2023-03-03T14:49:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.568+0000 7f563d186700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f56380aad20 0x7f5638157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.568+0000 7f563d186700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f56380aad20 0x7f5638157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50470/0 (socket says 172.21.15.7:50470) 2023-03-03T14:49:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.568+0000 7f563d186700 1 -- 172.21.15.7:0/3733569023 learned_addr learned my addr 172.21.15.7:0/3733569023 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:49:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563d987700 1 --2- 172.21.15.7:0/3733569023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 0x7f56380717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563e188700 1 --2- 172.21.15.7:0/3733569023 >> v2:172.21.15.143:3300/0 conn(0x7f5638071d20 0x7f56381581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563d186700 1 -- 172.21.15.7:0/3733569023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 msgr2=0x7f56380717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:49:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563d186700 1 --2- 172.21.15.7:0/3733569023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 0x7f56380717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563d186700 1 -- 172.21.15.7:0/3733569023 >> v2:172.21.15.143:3300/0 conn(0x7f5638071d20 msgr2=0x7f56381581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:49:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563d186700 1 --2- 172.21.15.7:0/3733569023 >> v2:172.21.15.143:3300/0 conn(0x7f5638071d20 0x7f56381581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563d186700 1 -- 172.21.15.7:0/3733569023 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563806f890 con 0x7f56380aad20 2023-03-03T14:49:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563d186700 1 --2- 172.21.15.7:0/3733569023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 0x7f5638157ca0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f56380aa710 tx=0x7f5634009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86576a9410d0e020 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:49:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563c985700 1 -- 172.21.15.7:0/3733569023 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563400b870 con 0x7f56380aad20 2023-03-03T14:49:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563c985700 1 -- 172.21.15.7:0/3733569023 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563400b9d0 con 0x7f56380aad20 2023-03-03T14:49:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.569+0000 7f563c985700 1 -- 172.21.15.7:0/3733569023 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563400bd60 con 0x7f56380aad20 2023-03-03T14:49:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.570+0000 7f563fbeb700 1 -- 172.21.15.7:0/3733569023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 msgr2=0x7f5638157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:29.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.570+0000 7f563fbeb700 1 --2- 172.21.15.7:0/3733569023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 0x7f5638157ca0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f56380aa710 tx=0x7f5634009bd0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.570+0000 7f563fbeb700 1 -- 172.21.15.7:0/3733569023 shutdown_connections 2023-03-03T14:49:29.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.570+0000 7f563fbeb700 1 --2- 172.21.15.7:0/3733569023 >> v2:172.21.15.143:3300/0 conn(0x7f5638071d20 0x7f56381581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.570+0000 7f563fbeb700 1 --2- 172.21.15.7:0/3733569023 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 0x7f56380717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.570+0000 7f563fbeb700 1 --2- 172.21.15.7:0/3733569023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 0x7f5638157ca0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.570+0000 7f563fbeb700 1 -- 172.21.15.7:0/3733569023 >> 172.21.15.7:0/3733569023 conn(0x7f56380c8900 msgr2=0x7f56380c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:49:29.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.570+0000 7f563fbeb700 1 -- 172.21.15.7:0/3733569023 shutdown_connections 2023-03-03T14:49:29.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.570+0000 7f563fbeb700 1 -- 172.21.15.7:0/3733569023 wait complete. 2023-03-03T14:49:29.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.571+0000 7f563fbeb700 1 Processor -- start 2023-03-03T14:49:29.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.571+0000 7f563fbeb700 1 -- start start 2023-03-03T14:49:29.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.572+0000 7f563fbeb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5638071d20 0x7f5638153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:29.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.572+0000 7f563fbeb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 0x7f5638156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:29.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.572+0000 7f563fbeb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 0x7f5638157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:29.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.572+0000 7f563fbeb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56381edb60 con 0x7f56380aad20 2023-03-03T14:49:29.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.572+0000 7f563fbeb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5638145be0 con 0x7f5638071d20 2023-03-03T14:49:29.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.572+0000 7f563fbeb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56381ed880 con 0x7f563814b020 2023-03-03T14:49:29.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.572+0000 7f563d186700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 0x7f5638156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:29.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563d186700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 0x7f5638156fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50476/0 (socket says 172.21.15.7:50476) 2023-03-03T14:49:29.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563d186700 1 -- 172.21.15.7:0/1157255613 learned_addr learned my addr 172.21.15.7:0/1157255613 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:49:29.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563d987700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5638071d20 0x7f5638153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:29.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563e188700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 0x7f5638157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:29.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563d186700 1 -- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 msgr2=0x7f5638157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:29.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563d186700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 0x7f5638157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563d186700 1 -- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5638071d20 msgr2=0x7f5638153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563d186700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5638071d20 0x7f5638153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563d186700 1 -- 172.21.15.7:0/1157255613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5634009550 con 0x7f56380aad20 2023-03-03T14:49:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563e188700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 0x7f5638157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:49:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563d987700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5638071d20 0x7f5638153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:49:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563d186700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 0x7f5638156fe0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f5634014040 tx=0x7f5634004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:49:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f562e7fc700 1 -- 172.21.15.7:0/1157255613 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634004520 con 0x7f56380aad20 2023-03-03T14:49:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f562e7fc700 1 -- 172.21.15.7:0/1157255613 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563401e070 con 0x7f56380aad20 2023-03-03T14:49:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563fbeb700 1 -- 172.21.15.7:0/1157255613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5638155350 con 0x7f56380aad20 2023-03-03T14:49:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f562e7fc700 1 -- 172.21.15.7:0/1157255613 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56340038e0 con 0x7f56380aad20 2023-03-03T14:49:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.573+0000 7f563fbeb700 1 -- 172.21.15.7:0/1157255613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563806fb70 con 0x7f56380aad20 2023-03-03T14:49:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.574+0000 7f563fbeb700 1 -- 172.21.15.7:0/1157255613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5638158720 con 0x7f56380aad20 2023-03-03T14:49:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.575+0000 7f562e7fc700 1 -- 172.21.15.7:0/1157255613 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f563400f050 con 0x7f56380aad20 2023-03-03T14:49:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.575+0000 7f562e7fc700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f561c0682f0 0x7f561c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:49:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.575+0000 7f562e7fc700 1 -- 172.21.15.7:0/1157255613 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5634020070 con 0x7f56380aad20 2023-03-03T14:49:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.576+0000 7f563d987700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f561c0682f0 0x7f561c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:49:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.576+0000 7f563d987700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f561c0682f0 0x7f561c06a7a0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f562800aa50 tx=0x7f5628008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:49:29.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.580+0000 7f562e7fc700 1 -- 172.21.15.7:0/1157255613 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5634053b50 con 0x7f56380aad20 2023-03-03T14:49:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.816+0000 7f563fbeb700 1 -- 172.21.15.7:0/1157255613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f563804e6e0 con 0x7f56380aad20 2023-03-03T14:49:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.817+0000 7f562e7fc700 1 -- 172.21.15.7:0/1157255613 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5634057170 con 0x7f56380aad20 2023-03-03T14:49:29.819 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:49:29.819 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:49:29.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 -- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f561c0682f0 msgr2=0x7f561c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:29.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f561c0682f0 0x7f561c06a7a0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f562800aa50 tx=0x7f5628008040 comp rx=0 tx=0).stop 2023-03-03T14:49:29.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 -- 172.21.15.7:0/1157255613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 msgr2=0x7f5638156fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:49:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 0x7f5638156fe0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f5634014040 tx=0x7f5634004750 comp rx=0 tx=0).stop 2023-03-03T14:49:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 -- 172.21.15.7:0/1157255613 shutdown_connections 2023-03-03T14:49:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f563814b020 0x7f5638157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5638071d20 0x7f5638153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f561c0682f0 0x7f561c06a7a0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 --2- 172.21.15.7:0/1157255613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56380aad20 0x7f5638156fe0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:49:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 -- 172.21.15.7:0/1157255613 >> 172.21.15.7:0/1157255613 conn(0x7f56380c8900 msgr2=0x7f563814a280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:49:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 -- 172.21.15.7:0/1157255613 shutdown_connections 2023-03-03T14:49:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:49:29.819+0000 7f563fbeb700 1 -- 172.21.15.7:0/1157255613 wait complete. 2023-03-03T14:49:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:49:29.833 INFO:tasks.mds_thrash.fs.[a]:waiting for 102 secs before thrashing 2023-03-03T14:49:57.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:49:57.578 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T14:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T14:50:27.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:27.716 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:41.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:50:42.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.133+0000 7f25ffd52700 1 Processor -- start 2023-03-03T14:50:42.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.133+0000 7f25ffd52700 1 -- start start 2023-03-03T14:50:42.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25ffd52700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f8155410 0x7f25f80bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:50:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25ffd52700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f80c08d0 con 0x7f25f8151e80 2023-03-03T14:50:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25ffd52700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f8154930 con 0x7f25f80c5b00 2023-03-03T14:50:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25ffd52700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f8152250 con 0x7f25f8155410 2023-03-03T14:50:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fdaee700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f25f8151e80 0x7f25f80c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44508/0 (socket says 172.21.15.7:44508) 2023-03-03T14:50:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fdaee700 1 -- 172.21.15.7:0/1349750240 learned_addr learned my addr 172.21.15.7:0/1349750240 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:50:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416681978 0 0) 0x7f25f80c08d0 con 0x7f25f8151e80 2023-03-03T14:50:42.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25e40035b0 con 0x7f25f8151e80 2023-03-03T14:50:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fd2ed700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f8155410 0x7f25f80bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:50:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1075807860 0 0) 0x7f25e40035b0 con 0x7f25f8151e80 2023-03-03T14:50:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f8155410 msgr2=0x7f25f80bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:50:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fcaec700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f8155410 0x7f25f80bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:50:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 >> v1:172.21.15.143:6789/0 conn(0x7f25f80c5b00 legacy=0x7f25f80bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:50:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8154c10 con 0x7f25f8151e80 2023-03-03T14:50:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.134+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f25f4001670 con 0x7f25f8151e80 2023-03-03T14:50:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 >> v1:172.21.15.7:6789/0 conn(0x7f25f8151e80 legacy=0x7f25f80c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:50:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fcaec700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25e4003aa0 0x7f25e4003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:50:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fcaec700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e4004850 0x7f25e4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:50:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fcaec700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f8155410 0x7f25e4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:50:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e400baf0 con 0x7f25e4004850 2023-03-03T14:50:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e400bc60 con 0x7f25e4003aa0 2023-03-03T14:50:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e400be20 con 0x7f25f8155410 2023-03-03T14:50:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fd2ed700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e4004850 0x7f25e4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:50:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fdaee700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25e4003aa0 0x7f25e4003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:50:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fe2ef700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f8155410 0x7f25e4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:50:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fd2ed700 1 -- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f8155410 msgr2=0x7f25e4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:50:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fd2ed700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f8155410 0x7f25e4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:50:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fd2ed700 1 -- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25e4003aa0 msgr2=0x7f25e4003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:50:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fd2ed700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25e4003aa0 0x7f25e4003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:50:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fd2ed700 1 -- 172.21.15.7:0/1349750240 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e40035b0 con 0x7f25e4004850 2023-03-03T14:50:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fe2ef700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25f8155410 0x7f25e4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:50:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fd2ed700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e4004850 0x7f25e4004cc0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f25e8009b60 tx=0x7f25e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4733123037132181 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:50:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e800e6c0 con 0x7f25e4004850 2023-03-03T14:50:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e800e820 con 0x7f25e4004850 2023-03-03T14:50:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.135+0000 7f25fcaec700 1 -- 172.21.15.7:0/1349750240 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e800eb90 con 0x7f25e4004850 2023-03-03T14:50:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.136+0000 7f25ffd52700 1 -- 172.21.15.7:0/1349750240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e4004850 msgr2=0x7f25e4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:50:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.136+0000 7f25ffd52700 1 --2- 172.21.15.7:0/1349750240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25e4004850 0x7f25e4004cc0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f25e8009b60 tx=0x7f25e4005880 comp rx=0 tx=0).stop 2023-03-03T14:50:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.136+0000 7f25fe2ef700 1 -- 172.21.15.7:0/1349750240 reap_dead start 2023-03-03T14:50:42.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.136+0000 7f25ffd52700 1 -- 172.21.15.7:0/1349750240 shutdown_connections 2023-03-03T14:50:42.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.136+0000 7f25ffd52700 1 -- 172.21.15.7:0/1349750240 >> 172.21.15.7:0/1349750240 conn(0x7f25f8071630 msgr2=0x7f25f806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:50:42.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.136+0000 7f25ffd52700 1 -- 172.21.15.7:0/1349750240 shutdown_connections 2023-03-03T14:50:42.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.136+0000 7f25ffd52700 1 -- 172.21.15.7:0/1349750240 wait complete. 2023-03-03T14:50:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.137+0000 7f25ffd52700 1 Processor -- start 2023-03-03T14:50:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.137+0000 7f25ffd52700 1 -- start start 2023-03-03T14:50:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25ffd52700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25e4004850 0x7f25f8156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:50:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25ffd52700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f814a6f0 0x7f25f8158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:50:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25ffd52700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f8155ab0 0x7f25f80c3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:50:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25ffd52700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f814ab80 con 0x7f25f814a6f0 2023-03-03T14:50:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25ffd52700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f814a0e0 con 0x7f25f8155ab0 2023-03-03T14:50:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25ffd52700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80c4250 con 0x7f25e4004850 2023-03-03T14:50:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fd2ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f814a6f0 0x7f25f8158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:50:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fd2ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f814a6f0 0x7f25f8158430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50544/0 (socket says 172.21.15.7:50544) 2023-03-03T14:50:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fd2ed700 1 -- 172.21.15.7:0/3101807120 learned_addr learned my addr 172.21.15.7:0/3101807120 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:50:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fdaee700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25e4004850 0x7f25f8156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:50:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fe2ef700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f8155ab0 0x7f25f80c3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:50:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fd2ed700 1 -- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25e4004850 msgr2=0x7f25f8156810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:50:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fd2ed700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25e4004850 0x7f25f8156810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:50:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fd2ed700 1 -- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f8155ab0 msgr2=0x7f25f80c3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:50:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fd2ed700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f8155ab0 0x7f25f80c3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:50:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fd2ed700 1 -- 172.21.15.7:0/3101807120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f80c5220 con 0x7f25f814a6f0 2023-03-03T14:50:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fe2ef700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f8155ab0 0x7f25f80c3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:50:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.138+0000 7f25fd2ed700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f814a6f0 0x7f25f8158430 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f25e400bc30 tx=0x7f25e80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:50:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.139+0000 7f25ee7fc700 1 -- 172.21.15.7:0/3101807120 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8021070 con 0x7f25f814a6f0 2023-03-03T14:50:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.139+0000 7f25ee7fc700 1 -- 172.21.15.7:0/3101807120 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e8004520 con 0x7f25f814a6f0 2023-03-03T14:50:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.139+0000 7f25ffd52700 1 -- 172.21.15.7:0/3101807120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f8154ef0 con 0x7f25f814a6f0 2023-03-03T14:50:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.139+0000 7f25ee7fc700 1 -- 172.21.15.7:0/3101807120 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e800e6c0 con 0x7f25f814a6f0 2023-03-03T14:50:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.139+0000 7f25ffd52700 1 -- 172.21.15.7:0/3101807120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f80c0530 con 0x7f25f814a6f0 2023-03-03T14:50:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.139+0000 7f25ffd52700 1 -- 172.21.15.7:0/3101807120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f8074bb0 con 0x7f25f814a6f0 2023-03-03T14:50:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.140+0000 7f25ee7fc700 1 -- 172.21.15.7:0/3101807120 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f25e802e030 con 0x7f25f814a6f0 2023-03-03T14:50:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.140+0000 7f25ee7fc700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25dc0682f0 0x7f25dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:50:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.140+0000 7f25ee7fc700 1 -- 172.21.15.7:0/3101807120 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f25e8087390 con 0x7f25f814a6f0 2023-03-03T14:50:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.141+0000 7f25fdaee700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25dc0682f0 0x7f25dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:50:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.141+0000 7f25fdaee700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25dc0682f0 0x7f25dc06a7a0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f25f40018e0 tx=0x7f25f4001850 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:50:42.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.145+0000 7f25ee7fc700 1 -- 172.21.15.7:0/3101807120 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f25e8053730 con 0x7f25f814a6f0 2023-03-03T14:50:42.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.385+0000 7f25ffd52700 1 -- 172.21.15.7:0/3101807120 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25f80c6860 con 0x7f25f814a6f0 2023-03-03T14:50:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.385+0000 7f25ee7fc700 1 -- 172.21.15.7:0/3101807120 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f25e8056d50 con 0x7f25f814a6f0 2023-03-03T14:50:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:50:42.388 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:50:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.387+0000 7f25ffd52700 1 -- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25dc0682f0 msgr2=0x7f25dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:50:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.387+0000 7f25ffd52700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25dc0682f0 0x7f25dc06a7a0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f25f40018e0 tx=0x7f25f4001850 comp rx=0 tx=0).stop 2023-03-03T14:50:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.387+0000 7f25ffd52700 1 -- 172.21.15.7:0/3101807120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f814a6f0 msgr2=0x7f25f8158430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:50:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.387+0000 7f25ffd52700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f814a6f0 0x7f25f8158430 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f25e400bc30 tx=0x7f25e80047c0 comp rx=0 tx=0).stop 2023-03-03T14:50:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.387+0000 7f25ffd52700 1 -- 172.21.15.7:0/3101807120 shutdown_connections 2023-03-03T14:50:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.387+0000 7f25ffd52700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25f8155ab0 0x7f25f80c3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:50:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.387+0000 7f25ffd52700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f25e4004850 0x7f25f8156810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:50:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.387+0000 7f25ffd52700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f25dc0682f0 0x7f25dc06a7a0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:50:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.387+0000 7f25ffd52700 1 --2- 172.21.15.7:0/3101807120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25f814a6f0 0x7f25f8158430 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:50:42.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.387+0000 7f25ffd52700 1 -- 172.21.15.7:0/3101807120 >> 172.21.15.7:0/3101807120 conn(0x7f25f8071630 msgr2=0x7f25f8151460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:50:42.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.388+0000 7f25ffd52700 1 -- 172.21.15.7:0/3101807120 shutdown_connections 2023-03-03T14:50:42.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:50:42.388+0000 7f25ffd52700 1 -- 172.21.15.7:0/3101807120 wait complete. 2023-03-03T14:50:42.391 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:50:42.401 INFO:tasks.mds_thrash.fs.[b]:waiting for 22 secs before thrashing 2023-03-03T14:50:57.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:50:57.795 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:04.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:51:04.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.540+0000 7fd871483700 1 Processor -- start 2023-03-03T14:51:04.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.540+0000 7fd871483700 1 -- start start 2023-03-03T14:51:04.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.541+0000 7fd871483700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86c0c00e0 0x7fd86c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.541+0000 7fd871483700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd86c154ff0 0x7fd86c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.541+0000 7fd871483700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86c0c7310 con 0x7fd86c0caa30 2023-03-03T14:51:04.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.541+0000 7fd871483700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c0bfd10 con 0x7fd86c154ff0 2023-03-03T14:51:04.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.541+0000 7fd871483700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0bfa30 con 0x7fd86c0c00e0 2023-03-03T14:51:04.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.541+0000 7fd86b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd86c154ff0 0x7fd86c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.541+0000 7fd86b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd86c154ff0 0x7fd86c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50574/0 (socket says 172.21.15.7:50574) 2023-03-03T14:51:04.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.541+0000 7fd86b7fe700 1 -- 172.21.15.7:0/4126197108 learned_addr learned my addr 172.21.15.7:0/4126197108 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:51:04.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.541+0000 7fd86a7fc700 1 --2- 172.21.15.7:0/4126197108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86c0c00e0 0x7fd86c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.542+0000 7fd86b7fe700 1 -- 172.21.15.7:0/4126197108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86c0c00e0 msgr2=0x7fd86c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:51:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.542+0000 7fd86b7fe700 1 --2- 172.21.15.7:0/4126197108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86c0c00e0 0x7fd86c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.542+0000 7fd86b7fe700 1 -- 172.21.15.7:0/4126197108 >> v1:172.21.15.143:6789/0 conn(0x7fd86c0caa30 legacy=0x7fd86c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.542+0000 7fd86b7fe700 1 -- 172.21.15.7:0/4126197108 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c154510 con 0x7fd86c154ff0 2023-03-03T14:51:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.542+0000 7fd86b7fe700 1 --2- 172.21.15.7:0/4126197108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 0x7fd86c0cae00 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fd864009e50 tx=0x7fd86400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d740b23e557e43 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:51:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.542+0000 7fd869ffb700 1 -- 172.21.15.7:0/4126197108 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86400e070 con 0x7fd86c154ff0 2023-03-03T14:51:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.542+0000 7fd869ffb700 1 -- 172.21.15.7:0/4126197108 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86400b910 con 0x7fd86c154ff0 2023-03-03T14:51:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.542+0000 7fd869ffb700 1 -- 172.21.15.7:0/4126197108 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86400bcc0 con 0x7fd86c154ff0 2023-03-03T14:51:04.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.542+0000 7fd871483700 1 -- 172.21.15.7:0/4126197108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 msgr2=0x7fd86c0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.542+0000 7fd871483700 1 --2- 172.21.15.7:0/4126197108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 0x7fd86c0cae00 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fd864009e50 tx=0x7fd86400b330 comp rx=0 tx=0).stop 2023-03-03T14:51:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.543+0000 7fd871483700 1 -- 172.21.15.7:0/4126197108 shutdown_connections 2023-03-03T14:51:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.543+0000 7fd871483700 1 --2- 172.21.15.7:0/4126197108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 0x7fd86c0cae00 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.543+0000 7fd871483700 1 --2- 172.21.15.7:0/4126197108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86c0c00e0 0x7fd86c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.543+0000 7fd871483700 1 -- 172.21.15.7:0/4126197108 >> 172.21.15.7:0/4126197108 conn(0x7fd86c066e40 msgr2=0x7fd86c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:51:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.543+0000 7fd871483700 1 -- 172.21.15.7:0/4126197108 shutdown_connections 2023-03-03T14:51:04.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.543+0000 7fd871483700 1 -- 172.21.15.7:0/4126197108 wait complete. 2023-03-03T14:51:04.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.544+0000 7fd871483700 1 Processor -- start 2023-03-03T14:51:04.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.544+0000 7fd871483700 1 -- start start 2023-03-03T14:51:04.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd871483700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86c0c00e0 0x7fd86c14b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd871483700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86c0caa30 0x7fd86c157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd871483700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 0x7fd86c148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd871483700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c14a4d0 con 0x7fd86c154ff0 2023-03-03T14:51:04.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd871483700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c14a7b0 con 0x7fd86c0c00e0 2023-03-03T14:51:04.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd871483700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c157230 con 0x7fd86c0caa30 2023-03-03T14:51:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 0x7fd86c148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 0x7fd86c148930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50580/0 (socket says 172.21.15.7:50580) 2023-03-03T14:51:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86b7fe700 1 -- 172.21.15.7:0/4072647950 learned_addr learned my addr 172.21.15.7:0/4072647950 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:51:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86a7fc700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86c0caa30 0x7fd86c157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86affd700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86c0c00e0 0x7fd86c14b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86b7fe700 1 -- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86c0caa30 msgr2=0x7fd86c157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86b7fe700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86c0caa30 0x7fd86c157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86b7fe700 1 -- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86c0c00e0 msgr2=0x7fd86c14b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86b7fe700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86c0c00e0 0x7fd86c14b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86b7fe700 1 -- 172.21.15.7:0/4072647950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd864009b30 con 0x7fd86c154ff0 2023-03-03T14:51:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86affd700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86c0c00e0 0x7fd86c14b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:51:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd86b7fe700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 0x7fd86c148930 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fd86c06b270 tx=0x7fd864004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:51:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.545+0000 7fd8537fe700 1 -- 172.21.15.7:0/4072647950 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864004870 con 0x7fd86c154ff0 2023-03-03T14:51:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.546+0000 7fd8537fe700 1 -- 172.21.15.7:0/4072647950 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8640099b0 con 0x7fd86c154ff0 2023-03-03T14:51:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.546+0000 7fd8537fe700 1 -- 172.21.15.7:0/4072647950 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86401c680 con 0x7fd86c154ff0 2023-03-03T14:51:04.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.546+0000 7fd871483700 1 -- 172.21.15.7:0/4072647950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c154ad0 con 0x7fd86c154ff0 2023-03-03T14:51:04.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.546+0000 7fd871483700 1 -- 172.21.15.7:0/4072647950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c1547f0 con 0x7fd86c154ff0 2023-03-03T14:51:04.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.546+0000 7fd871483700 1 -- 172.21.15.7:0/4072647950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c0bf650 con 0x7fd86c154ff0 2023-03-03T14:51:04.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.547+0000 7fd8537fe700 1 -- 172.21.15.7:0/4072647950 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd86402c070 con 0x7fd86c154ff0 2023-03-03T14:51:04.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.548+0000 7fd8537fe700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8580682f0 0x7fd85806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:04.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.548+0000 7fd8537fe700 1 -- 172.21.15.7:0/4072647950 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd864087e20 con 0x7fd86c154ff0 2023-03-03T14:51:04.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.548+0000 7fd86affd700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8580682f0 0x7fd85806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:04.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.548+0000 7fd86affd700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8580682f0 0x7fd85806a7a0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0af1a0 tx=0x7fd860009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:51:04.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.552+0000 7fd8537fe700 1 -- 172.21.15.7:0/4072647950 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd8640540d0 con 0x7fd86c154ff0 2023-03-03T14:51:04.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.794+0000 7fd871483700 1 -- 172.21.15.7:0/4072647950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd86c074bb0 con 0x7fd86c154ff0 2023-03-03T14:51:04.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.795+0000 7fd8537fe700 1 -- 172.21.15.7:0/4072647950 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd8640576f0 con 0x7fd86c154ff0 2023-03-03T14:51:04.797 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:51:04.797 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:51:04.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 -- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8580682f0 msgr2=0x7fd85806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8580682f0 0x7fd85806a7a0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0af1a0 tx=0x7fd860009040 comp rx=0 tx=0).stop 2023-03-03T14:51:04.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 -- 172.21.15.7:0/4072647950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 msgr2=0x7fd86c148930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:04.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 0x7fd86c148930 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fd86c06b270 tx=0x7fd864004ad0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 -- 172.21.15.7:0/4072647950 shutdown_connections 2023-03-03T14:51:04.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8580682f0 0x7fd85806a7a0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c154ff0 0x7fd86c148930 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86c0c00e0 0x7fd86c14b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 --2- 172.21.15.7:0/4072647950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86c0caa30 0x7fd86c157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:04.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 -- 172.21.15.7:0/4072647950 >> 172.21.15.7:0/4072647950 conn(0x7fd86c066e40 msgr2=0x7fd86c0c68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:51:04.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 -- 172.21.15.7:0/4072647950 shutdown_connections 2023-03-03T14:51:04.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:04.797+0000 7fd871483700 1 -- 172.21.15.7:0/4072647950 wait complete. 2023-03-03T14:51:04.801 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:51:04.811 INFO:tasks.mds_thrash.fs.[b]:waiting for 70 secs before thrashing 2023-03-03T14:51:11.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:51:11.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.971+0000 7fae47b03700 1 Processor -- start 2023-03-03T14:51:11.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.971+0000 7fae47b03700 1 -- start start 2023-03-03T14:51:11.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae47b03700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae401530d0 0x7fae40156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:11.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae47b03700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae400c2f30 con 0x7fae4014fda0 2023-03-03T14:51:11.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae47b03700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae400c31d0 con 0x7fae4014f9d0 2023-03-03T14:51:11.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae47b03700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae401524a0 con 0x7fae401530d0 2023-03-03T14:51:11.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae460a0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae401530d0 0x7fae40156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:11.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae4509e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fae4014fda0 0x7fae401529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44574/0 (socket says 172.21.15.7:44574) 2023-03-03T14:51:11.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae4509e700 1 -- 172.21.15.7:0/4179261037 learned_addr learned my addr 172.21.15.7:0/4179261037 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:51:11.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae460a0700 1 -- 172.21.15.7:0/4179261037 >> v1:172.21.15.143:6789/0 conn(0x7fae4014f9d0 legacy=0x7fae40071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:11.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae460a0700 1 -- 172.21.15.7:0/4179261037 >> v1:172.21.15.7:6789/0 conn(0x7fae4014fda0 legacy=0x7fae401529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:11.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae4489d700 1 -- 172.21.15.7:0/4179261037 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724899299 0 0) 0x7fae400c2f30 con 0x7fae4014fda0 2023-03-03T14:51:11.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.972+0000 7fae460a0700 1 -- 172.21.15.7:0/4179261037 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae400c34b0 con 0x7fae401530d0 2023-03-03T14:51:11.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae460a0700 1 --2- 172.21.15.7:0/4179261037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 0x7fae40156430 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fae38009d90 tx=0x7fae3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f5cf14a408b1e5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:51:11.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae4489d700 1 -- 172.21.15.7:0/4179261037 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3800e070 con 0x7fae401530d0 2023-03-03T14:51:11.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae4489d700 1 -- 172.21.15.7:0/4179261037 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3800bbd0 con 0x7fae401530d0 2023-03-03T14:51:11.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae4489d700 1 -- 172.21.15.7:0/4179261037 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38012430 con 0x7fae401530d0 2023-03-03T14:51:11.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae47b03700 1 -- 172.21.15.7:0/4179261037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 msgr2=0x7fae40156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:11.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae47b03700 1 --2- 172.21.15.7:0/4179261037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 0x7fae40156430 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fae38009d90 tx=0x7fae3800b330 comp rx=0 tx=0).stop 2023-03-03T14:51:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae47b03700 1 -- 172.21.15.7:0/4179261037 shutdown_connections 2023-03-03T14:51:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae47b03700 1 --2- 172.21.15.7:0/4179261037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 0x7fae40156430 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae47b03700 1 -- 172.21.15.7:0/4179261037 >> 172.21.15.7:0/4179261037 conn(0x7fae400c8900 msgr2=0x7fae400c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:51:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae47b03700 1 -- 172.21.15.7:0/4179261037 shutdown_connections 2023-03-03T14:51:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.973+0000 7fae47b03700 1 -- 172.21.15.7:0/4179261037 wait complete. 2023-03-03T14:51:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.975+0000 7fae47b03700 1 Processor -- start 2023-03-03T14:51:11.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.975+0000 7fae47b03700 1 -- start start 2023-03-03T14:51:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.975+0000 7fae47b03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4014f9d0 0x7fae401216f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.975+0000 7fae47b03700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae4014fda0 0x7fae400c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.975+0000 7fae47b03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 0x7fae4006e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.975+0000 7fae47b03700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae401221a0 con 0x7fae401530d0 2023-03-03T14:51:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.975+0000 7fae47b03700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae40145f90 con 0x7fae4014fda0 2023-03-03T14:51:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.975+0000 7fae47b03700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae401e9a70 con 0x7fae4014f9d0 2023-03-03T14:51:11.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae460a0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 0x7fae4006e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:11.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae460a0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 0x7fae4006e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50602/0 (socket says 172.21.15.7:50602) 2023-03-03T14:51:11.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae460a0700 1 -- 172.21.15.7:0/2384343420 learned_addr learned my addr 172.21.15.7:0/2384343420 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:51:11.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae4509e700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae4014fda0 0x7fae400c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:11.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae4589f700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4014f9d0 0x7fae401216f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:11.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae460a0700 1 -- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4014f9d0 msgr2=0x7fae401216f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:11.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae460a0700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4014f9d0 0x7fae401216f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:11.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae460a0700 1 -- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae4014fda0 msgr2=0x7fae400c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:11.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae460a0700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae4014fda0 0x7fae400c7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae460a0700 1 -- 172.21.15.7:0/2384343420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38009a70 con 0x7fae401530d0 2023-03-03T14:51:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae4589f700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4014f9d0 0x7fae401216f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:51:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae460a0700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 0x7fae4006e690 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fae38014040 tx=0x7fae38004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:51:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae367fc700 1 -- 172.21.15.7:0/2384343420 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38012430 con 0x7fae401530d0 2023-03-03T14:51:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae367fc700 1 -- 172.21.15.7:0/2384343420 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae38020070 con 0x7fae401530d0 2023-03-03T14:51:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae47b03700 1 -- 172.21.15.7:0/2384343420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae400ac350 con 0x7fae401530d0 2023-03-03T14:51:11.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae367fc700 1 -- 172.21.15.7:0/2384343420 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3801bc30 con 0x7fae401530d0 2023-03-03T14:51:11.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.976+0000 7fae47b03700 1 -- 172.21.15.7:0/2384343420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae400ab630 con 0x7fae401530d0 2023-03-03T14:51:11.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.977+0000 7fae47b03700 1 -- 172.21.15.7:0/2384343420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae40071fa0 con 0x7fae401530d0 2023-03-03T14:51:11.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.978+0000 7fae367fc700 1 -- 172.21.15.7:0/2384343420 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae38026400 con 0x7fae401530d0 2023-03-03T14:51:11.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.978+0000 7fae367fc700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae240682f0 0x7fae2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:51:11.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.978+0000 7fae367fc700 1 -- 172.21.15.7:0/2384343420 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fae38088260 con 0x7fae401530d0 2023-03-03T14:51:11.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.978+0000 7fae4589f700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae240682f0 0x7fae2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:51:11.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.979+0000 7fae4589f700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae240682f0 0x7fae2406a7a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fae3c000f60 tx=0x7fae3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:51:11.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:11.983+0000 7fae367fc700 1 -- 172.21.15.7:0/2384343420 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae380545a0 con 0x7fae401530d0 2023-03-03T14:51:12.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.222+0000 7fae47b03700 1 -- 172.21.15.7:0/2384343420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae40074bb0 con 0x7fae401530d0 2023-03-03T14:51:12.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.222+0000 7fae367fc700 1 -- 172.21.15.7:0/2384343420 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fae38057bc0 con 0x7fae401530d0 2023-03-03T14:51:12.224 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:51:12.225 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:51:12.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 -- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae240682f0 msgr2=0x7fae2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:12.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae240682f0 0x7fae2406a7a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fae3c000f60 tx=0x7fae3c009040 comp rx=0 tx=0).stop 2023-03-03T14:51:12.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 -- 172.21.15.7:0/2384343420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 msgr2=0x7fae4006e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:51:12.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 0x7fae4006e690 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fae38014040 tx=0x7fae38004ad0 comp rx=0 tx=0).stop 2023-03-03T14:51:12.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 -- 172.21.15.7:0/2384343420 shutdown_connections 2023-03-03T14:51:12.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae240682f0 0x7fae2406a7a0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:12.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae401530d0 0x7fae4006e690 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:12.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae4014f9d0 0x7fae401216f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:12.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 --2- 172.21.15.7:0/2384343420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae4014fda0 0x7fae400c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:51:12.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 -- 172.21.15.7:0/2384343420 >> 172.21.15.7:0/2384343420 conn(0x7fae400c8900 msgr2=0x7fae4014ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:51:12.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 -- 172.21.15.7:0/2384343420 shutdown_connections 2023-03-03T14:51:12.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:51:12.224+0000 7fae47b03700 1 -- 172.21.15.7:0/2384343420 wait complete. 2023-03-03T14:51:12.228 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:51:12.240 INFO:tasks.mds_thrash.fs.[a]:waiting for 83 secs before thrashing 2023-03-03T14:51:27.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:27.869 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:57.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:51:57.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:14.812 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:52:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.950+0000 7f9cef5e7700 1 Processor -- start 2023-03-03T14:52:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.950+0000 7f9cef5e7700 1 -- start start 2023-03-03T14:52:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.951+0000 7f9cef5e7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814fda0 0x7f9ce8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.951+0000 7f9cef5e7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce80c2f30 con 0x7f9ce80abe20 2023-03-03T14:52:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.951+0000 7f9cef5e7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce80c31d0 con 0x7f9ce814f9d0 2023-03-03T14:52:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.951+0000 7f9cef5e7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce81524a0 con 0x7f9ce814fda0 2023-03-03T14:52:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.951+0000 7f9cedb84700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9ce80abe20 0x7f9ce81529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44630/0 (socket says 172.21.15.7:44630) 2023-03-03T14:52:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.951+0000 7f9cedb84700 1 -- 172.21.15.7:0/2613328561 learned_addr learned my addr 172.21.15.7:0/2613328561 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:52:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.951+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574729912 0 0) 0x7f9ce80c2f30 con 0x7f9ce80abe20 2023-03-03T14:52:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.951+0000 7f9cecb82700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814fda0 0x7f9ce8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.951+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd40035b0 con 0x7f9ce80abe20 2023-03-03T14:52:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1390981229 0 0) 0x7f9cd40035b0 con 0x7f9ce80abe20 2023-03-03T14:52:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814fda0 msgr2=0x7f9ce8156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:52:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814fda0 0x7f9ce8156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 >> v1:172.21.15.143:6789/0 conn(0x7f9ce814f9d0 legacy=0x7f9ce8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce80c34b0 con 0x7f9ce80abe20 2023-03-03T14:52:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f9ce00019f0 con 0x7f9ce80abe20 2023-03-03T14:52:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 >> v1:172.21.15.7:6789/0 conn(0x7f9ce80abe20 legacy=0x7f9ce81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814fda0 0x7f9cd4003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cd4003fe0 0x7f9cd4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cd4004d90 0x7f9cd4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce81524a0 con 0x7f9cd4003fe0 2023-03-03T14:52:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd400bc00 con 0x7f9cd4004d90 2023-03-03T14:52:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd400bdc0 con 0x7f9ce814fda0 2023-03-03T14:52:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cecb82700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cd4003fe0 0x7f9cd4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9ced383700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814fda0 0x7f9cd4003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cedb84700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cd4004d90 0x7f9cd4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cecb82700 1 -- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814fda0 msgr2=0x7f9cd4003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:52:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cecb82700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814fda0 0x7f9cd4003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cecb82700 1 -- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cd4004d90 msgr2=0x7f9cd4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:52:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cecb82700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cd4004d90 0x7f9cd4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cecb82700 1 -- 172.21.15.7:0/2613328561 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd40035b0 con 0x7f9cd4003fe0 2023-03-03T14:52:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cedb84700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cd4004d90 0x7f9cd4005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:52:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.952+0000 7f9cecb82700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cd4003fe0 0x7f9cd4004850 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f9ce81501d0 tx=0x7f9cd4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9edb3c291b2b0efd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.953+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce400e6c0 con 0x7f9cd4003fe0 2023-03-03T14:52:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.953+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce400e820 con 0x7f9cd4003fe0 2023-03-03T14:52:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.953+0000 7f9cdffff700 1 -- 172.21.15.7:0/2613328561 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce400ebb0 con 0x7f9cd4003fe0 2023-03-03T14:52:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.953+0000 7f9cef5e7700 1 -- 172.21.15.7:0/2613328561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cd4003fe0 msgr2=0x7f9cd4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.953+0000 7f9cef5e7700 1 --2- 172.21.15.7:0/2613328561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cd4003fe0 0x7f9cd4004850 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f9ce81501d0 tx=0x7f9cd4005880 comp rx=0 tx=0).stop 2023-03-03T14:52:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.953+0000 7f9cedb84700 1 -- 172.21.15.7:0/2613328561 reap_dead start 2023-03-03T14:52:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.953+0000 7f9cef5e7700 1 -- 172.21.15.7:0/2613328561 shutdown_connections 2023-03-03T14:52:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.953+0000 7f9cef5e7700 1 -- 172.21.15.7:0/2613328561 >> 172.21.15.7:0/2613328561 conn(0x7f9ce80c8900 msgr2=0x7f9ce80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:52:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.953+0000 7f9cef5e7700 1 -- 172.21.15.7:0/2613328561 shutdown_connections 2023-03-03T14:52:14.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.953+0000 7f9cef5e7700 1 -- 172.21.15.7:0/2613328561 wait complete. 2023-03-03T14:52:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.954+0000 7f9cef5e7700 1 Processor -- start 2023-03-03T14:52:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.954+0000 7f9cef5e7700 1 -- start start 2023-03-03T14:52:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9cef5e7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cd4003fe0 0x7f9ce8148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9cef5e7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814c6f0 0x7f9ce81dc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9cef5e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce81dcad0 0x7f9ce81ea880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9cef5e7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce81db2e0 con 0x7f9ce81dcad0 2023-03-03T14:52:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9cef5e7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce81db5c0 con 0x7f9cd4003fe0 2023-03-03T14:52:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9cef5e7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce81db8a0 con 0x7f9ce814c6f0 2023-03-03T14:52:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9cedb84700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce81dcad0 0x7f9ce81ea880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9cedb84700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce81dcad0 0x7f9ce81ea880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50662/0 (socket says 172.21.15.7:50662) 2023-03-03T14:52:14.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9cedb84700 1 -- 172.21.15.7:0/100180747 learned_addr learned my addr 172.21.15.7:0/100180747 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:52:14.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9cecb82700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814c6f0 0x7f9ce81dc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:14.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.955+0000 7f9ced383700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cd4003fe0 0x7f9ce8148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:14.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cedb84700 1 -- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814c6f0 msgr2=0x7f9ce81dc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:14.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cedb84700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814c6f0 0x7f9ce81dc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:14.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cedb84700 1 -- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cd4003fe0 msgr2=0x7f9ce8148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:14.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cedb84700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cd4003fe0 0x7f9ce8148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cedb84700 1 -- 172.21.15.7:0/100180747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce80ac350 con 0x7f9ce81dcad0 2023-03-03T14:52:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cedb84700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce81dcad0 0x7f9ce81ea880 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0001f50 tx=0x7f9ce000e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cddffb700 1 -- 172.21.15.7:0/100180747 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce000ec20 con 0x7f9ce81dcad0 2023-03-03T14:52:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cddffb700 1 -- 172.21.15.7:0/100180747 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce000ed80 con 0x7f9ce81dcad0 2023-03-03T14:52:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cef5e7700 1 -- 172.21.15.7:0/100180747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce80ab630 con 0x7f9ce81dcad0 2023-03-03T14:52:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cddffb700 1 -- 172.21.15.7:0/100180747 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0013660 con 0x7f9ce81dcad0 2023-03-03T14:52:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.956+0000 7f9cef5e7700 1 -- 172.21.15.7:0/100180747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce8071fa0 con 0x7f9ce81dcad0 2023-03-03T14:52:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.957+0000 7f9cef5e7700 1 -- 172.21.15.7:0/100180747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce8074bb0 con 0x7f9ce81dcad0 2023-03-03T14:52:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.958+0000 7f9cddffb700 1 -- 172.21.15.7:0/100180747 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ce00137c0 con 0x7f9ce81dcad0 2023-03-03T14:52:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.958+0000 7f9cddffb700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ccc0682f0 0x7f9ccc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.958+0000 7f9cddffb700 1 -- 172.21.15.7:0/100180747 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9ce0086740 con 0x7f9ce81dcad0 2023-03-03T14:52:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.958+0000 7f9ced383700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ccc0682f0 0x7f9ccc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.959+0000 7f9ced383700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ccc0682f0 0x7f9ccc06a7a0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8155a70 tx=0x7f9cd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:14.963+0000 7f9cddffb700 1 -- 172.21.15.7:0/100180747 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ce0052ae0 con 0x7f9ce81dcad0 2023-03-03T14:52:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.205+0000 7f9cef5e7700 1 -- 172.21.15.7:0/100180747 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ce80c81a0 con 0x7f9ce81dcad0 2023-03-03T14:52:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.205+0000 7f9cddffb700 1 -- 172.21.15.7:0/100180747 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9ce0056100 con 0x7f9ce81dcad0 2023-03-03T14:52:15.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:52:15.208 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:52:15.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 -- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ccc0682f0 msgr2=0x7f9ccc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:15.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ccc0682f0 0x7f9ccc06a7a0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8155a70 tx=0x7f9cd8009040 comp rx=0 tx=0).stop 2023-03-03T14:52:15.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 -- 172.21.15.7:0/100180747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce81dcad0 msgr2=0x7f9ce81ea880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:15.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce81dcad0 0x7f9ce81ea880 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0001f50 tx=0x7f9ce000e440 comp rx=0 tx=0).stop 2023-03-03T14:52:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 -- 172.21.15.7:0/100180747 shutdown_connections 2023-03-03T14:52:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ccc0682f0 0x7f9ccc06a7a0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce81dcad0 0x7f9ce81ea880 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cd4003fe0 0x7f9ce8148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 --2- 172.21.15.7:0/100180747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9ce814c6f0 0x7f9ce81dc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 -- 172.21.15.7:0/100180747 >> 172.21.15.7:0/100180747 conn(0x7f9ce80c8900 msgr2=0x7f9ce80c70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:52:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 -- 172.21.15.7:0/100180747 shutdown_connections 2023-03-03T14:52:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:15.207+0000 7f9cef5e7700 1 -- 172.21.15.7:0/100180747 wait complete. 2023-03-03T14:52:15.211 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:52:15.221 INFO:tasks.mds_thrash.fs.[b]:waiting for 80 secs before thrashing 2023-03-03T14:52:27.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:27.997 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:35.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:52:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.378+0000 7fa748f01700 1 Processor -- start 2023-03-03T14:52:35.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.378+0000 7fa748f01700 1 -- start start 2023-03-03T14:52:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.379+0000 7fa748f01700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa744155390 0x7fa7440c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.379+0000 7fa748f01700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa744151770 0x7fa744151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.379+0000 7fa748f01700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa744154910 con 0x7fa74406e020 2023-03-03T14:52:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.379+0000 7fa748f01700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa744070b90 con 0x7fa744151770 2023-03-03T14:52:35.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.379+0000 7fa748f01700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa744151320 con 0x7fa744155390 2023-03-03T14:52:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.379+0000 7fa742d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa744151770 0x7fa744151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.379+0000 7fa742d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa744151770 0x7fa744151bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50684/0 (socket says 172.21.15.7:50684) 2023-03-03T14:52:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.379+0000 7fa742d9d700 1 -- 172.21.15.7:0/2923613609 learned_addr learned my addr 172.21.15.7:0/2923613609 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:52:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa74259c700 1 --2- 172.21.15.7:0/2923613609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa744155390 0x7fa7440c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa742d9d700 1 -- 172.21.15.7:0/2923613609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa744155390 msgr2=0x7fa7440c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:52:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa742d9d700 1 --2- 172.21.15.7:0/2923613609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa744155390 0x7fa7440c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa742d9d700 1 -- 172.21.15.7:0/2923613609 >> v1:172.21.15.143:6789/0 conn(0x7fa74406e020 legacy=0x7fa744071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:52:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa742d9d700 1 -- 172.21.15.7:0/2923613609 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7440702a0 con 0x7fa744151770 2023-03-03T14:52:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa742d9d700 1 --2- 172.21.15.7:0/2923613609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa744151770 0x7fa744151bf0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009e50 tx=0x7fa73c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c036f4edb1fd22 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa74159a700 1 -- 172.21.15.7:0/2923613609 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c00e070 con 0x7fa744151770 2023-03-03T14:52:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa74159a700 1 -- 172.21.15.7:0/2923613609 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c003a20 con 0x7fa744151770 2023-03-03T14:52:35.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa74159a700 1 -- 172.21.15.7:0/2923613609 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c003dd0 con 0x7fa744151770 2023-03-03T14:52:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa748f01700 1 -- 172.21.15.7:0/2923613609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa744151770 msgr2=0x7fa744151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.380+0000 7fa748f01700 1 --2- 172.21.15.7:0/2923613609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa744151770 0x7fa744151bf0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009e50 tx=0x7fa73c00b330 comp rx=0 tx=0).stop 2023-03-03T14:52:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.381+0000 7fa748f01700 1 -- 172.21.15.7:0/2923613609 shutdown_connections 2023-03-03T14:52:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.381+0000 7fa748f01700 1 --2- 172.21.15.7:0/2923613609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa744151770 0x7fa744151bf0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.381+0000 7fa748f01700 1 --2- 172.21.15.7:0/2923613609 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa744155390 0x7fa7440c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.381+0000 7fa748f01700 1 -- 172.21.15.7:0/2923613609 >> 172.21.15.7:0/2923613609 conn(0x7fa74406c110 msgr2=0x7fa74406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:52:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.381+0000 7fa748f01700 1 -- 172.21.15.7:0/2923613609 shutdown_connections 2023-03-03T14:52:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.381+0000 7fa748f01700 1 -- 172.21.15.7:0/2923613609 wait complete. 2023-03-03T14:52:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.382+0000 7fa748f01700 1 Processor -- start 2023-03-03T14:52:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.382+0000 7fa748f01700 1 -- start start 2023-03-03T14:52:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa748f01700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa74406e020 0x7fa744147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa748f01700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa744151770 0x7fa744148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa748f01700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa744155390 0x7fa7441eb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa748f01700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74414e8b0 con 0x7fa74406e020 2023-03-03T14:52:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa748f01700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7441dba10 con 0x7fa744155390 2023-03-03T14:52:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa748f01700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74414ca40 con 0x7fa744151770 2023-03-03T14:52:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa74259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa74406e020 0x7fa744147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa74259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa74406e020 0x7fa744147cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50686/0 (socket says 172.21.15.7:50686) 2023-03-03T14:52:35.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa74259c700 1 -- 172.21.15.7:0/4251604467 learned_addr learned my addr 172.21.15.7:0/4251604467 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:52:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa742d9d700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa744155390 0x7fa7441eb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa741d9b700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa744151770 0x7fa744148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa74259c700 1 -- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa744151770 msgr2=0x7fa744148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa74259c700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa744151770 0x7fa744148230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa74259c700 1 -- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa744155390 msgr2=0x7fa7441eb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa74259c700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa744155390 0x7fa7441eb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa74259c700 1 -- 172.21.15.7:0/4251604467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c009b30 con 0x7fa74406e020 2023-03-03T14:52:35.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.383+0000 7fa74259c700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa74406e020 0x7fa744147cf0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fa73800d830 tx=0x7fa73800dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.384+0000 7fa732ffd700 1 -- 172.21.15.7:0/4251604467 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7380125f0 con 0x7fa74406e020 2023-03-03T14:52:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.384+0000 7fa732ffd700 1 -- 172.21.15.7:0/4251604467 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa738012750 con 0x7fa74406e020 2023-03-03T14:52:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.384+0000 7fa748f01700 1 -- 172.21.15.7:0/4251604467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa744070820 con 0x7fa74406e020 2023-03-03T14:52:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.384+0000 7fa732ffd700 1 -- 172.21.15.7:0/4251604467 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738012b00 con 0x7fa74406e020 2023-03-03T14:52:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.384+0000 7fa748f01700 1 -- 172.21.15.7:0/4251604467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa744070540 con 0x7fa74406e020 2023-03-03T14:52:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.385+0000 7fa748f01700 1 -- 172.21.15.7:0/4251604467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa744155760 con 0x7fa74406e020 2023-03-03T14:52:35.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.385+0000 7fa732ffd700 1 -- 172.21.15.7:0/4251604467 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa738012c60 con 0x7fa74406e020 2023-03-03T14:52:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.386+0000 7fa732ffd700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7240682c0 0x7fa72406a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.386+0000 7fa732ffd700 1 -- 172.21.15.7:0/4251604467 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa738086650 con 0x7fa74406e020 2023-03-03T14:52:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.386+0000 7fa741d9b700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7240682c0 0x7fa72406a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.386+0000 7fa741d9b700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7240682c0 0x7fa72406a770 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fa7440ad0b0 tx=0x7fa734009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:35.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.390+0000 7fa732ffd700 1 -- 172.21.15.7:0/4251604467 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7380529f0 con 0x7fa74406e020 2023-03-03T14:52:35.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.628+0000 7fa748f01700 1 -- 172.21.15.7:0/4251604467 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa744074bb0 con 0x7fa74406e020 2023-03-03T14:52:35.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.628+0000 7fa732ffd700 1 -- 172.21.15.7:0/4251604467 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa738056010 con 0x7fa74406e020 2023-03-03T14:52:35.631 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:52:35.632 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:52:35.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 -- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7240682c0 msgr2=0x7fa72406a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:35.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7240682c0 0x7fa72406a770 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fa7440ad0b0 tx=0x7fa734009040 comp rx=0 tx=0).stop 2023-03-03T14:52:35.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 -- 172.21.15.7:0/4251604467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa74406e020 msgr2=0x7fa744147cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:35.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa74406e020 0x7fa744147cf0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fa73800d830 tx=0x7fa73800dbf0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 -- 172.21.15.7:0/4251604467 shutdown_connections 2023-03-03T14:52:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa744155390 0x7fa7441eb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7240682c0 0x7fa72406a770 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa74406e020 0x7fa744147cf0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 --2- 172.21.15.7:0/4251604467 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa744151770 0x7fa744148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 -- 172.21.15.7:0/4251604467 >> 172.21.15.7:0/4251604467 conn(0x7fa74406c110 msgr2=0x7fa744071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:52:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 -- 172.21.15.7:0/4251604467 shutdown_connections 2023-03-03T14:52:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.630+0000 7fa748f01700 1 -- 172.21.15.7:0/4251604467 wait complete. 2023-03-03T14:52:35.635 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:52:35.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:52:35.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.767+0000 7fa1ce772700 1 Processor -- start 2023-03-03T14:52:35.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.767+0000 7fa1ce772700 1 -- start start 2023-03-03T14:52:35.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ce772700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1c8072250 0x7fa1c814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ce772700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa1c80c1ce0 0x7fa1c814e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ce772700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c81510d0 con 0x7fa1c8071e80 2023-03-03T14:52:35.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ce772700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c80c3740 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ce772700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c8070670 con 0x7fa1c8072250 2023-03-03T14:52:35.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1c7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa1c8071e80 0x7fa1c8151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44674/0 (socket says 172.21.15.7:44674) 2023-03-03T14:52:35.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1c7fff700 1 -- 172.21.15.7:0/3436354265 learned_addr learned my addr 172.21.15.7:0/3436354265 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:52:35.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ccd0f700 1 --2- 172.21.15.7:0/3436354265 >> v2:172.21.15.7:3300/0 conn(0x7fa1c80c1ce0 0x7fa1c814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1c77fe700 1 --2- 172.21.15.7:0/3436354265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1c8072250 0x7fa1c814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1c6ffd700 1 -- 172.21.15.7:0/3436354265 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259794710 0 0) 0x7fa1c81510d0 con 0x7fa1c8071e80 2023-03-03T14:52:35.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1c6ffd700 1 -- 172.21.15.7:0/3436354265 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b00035b0 con 0x7fa1c8071e80 2023-03-03T14:52:35.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ccd0f700 1 -- 172.21.15.7:0/3436354265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1c8072250 msgr2=0x7fa1c814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:52:35.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ccd0f700 1 --2- 172.21.15.7:0/3436354265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1c8072250 0x7fa1c814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ccd0f700 1 -- 172.21.15.7:0/3436354265 >> v1:172.21.15.7:6789/0 conn(0x7fa1c8071e80 legacy=0x7fa1c8151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:35.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ccd0f700 1 -- 172.21.15.7:0/3436354265 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c80c39e0 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1ccd0f700 1 --2- 172.21.15.7:0/3436354265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 0x7fa1c814e730 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0009d90 tx=0x7fa1c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f621199715ad7940 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:35.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1c6ffd700 1 -- 172.21.15.7:0/3436354265 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c000e070 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1c6ffd700 1 -- 172.21.15.7:0/3436354265 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c000bb20 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.768+0000 7fa1c6ffd700 1 -- 172.21.15.7:0/3436354265 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0012430 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.769+0000 7fa1ce772700 1 -- 172.21.15.7:0/3436354265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 msgr2=0x7fa1c814e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:35.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.769+0000 7fa1ce772700 1 --2- 172.21.15.7:0/3436354265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 0x7fa1c814e730 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0009d90 tx=0x7fa1c000b330 comp rx=0 tx=0).stop 2023-03-03T14:52:35.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.769+0000 7fa1ce772700 1 -- 172.21.15.7:0/3436354265 shutdown_connections 2023-03-03T14:52:35.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.769+0000 7fa1ce772700 1 --2- 172.21.15.7:0/3436354265 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 0x7fa1c814e730 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.769+0000 7fa1ce772700 1 --2- 172.21.15.7:0/3436354265 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1c8072250 0x7fa1c814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.769+0000 7fa1ce772700 1 -- 172.21.15.7:0/3436354265 >> 172.21.15.7:0/3436354265 conn(0x7fa1c80a94d0 msgr2=0x7fa1c80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:52:35.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.769+0000 7fa1ce772700 1 -- 172.21.15.7:0/3436354265 shutdown_connections 2023-03-03T14:52:35.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.769+0000 7fa1ce772700 1 -- 172.21.15.7:0/3436354265 wait complete. 2023-03-03T14:52:35.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.770+0000 7fa1ce772700 1 Processor -- start 2023-03-03T14:52:35.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.770+0000 7fa1ce772700 1 -- start start 2023-03-03T14:52:35.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ce772700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1c8071e80 0x7fa1c8149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ce772700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c8072250 0x7fa1c814fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ce772700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 0x7fa1c81502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ce772700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c80c66c0 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ce772700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c814f630 con 0x7fa1c8072250 2023-03-03T14:52:35.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ce772700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c8144ae0 con 0x7fa1c8071e80 2023-03-03T14:52:35.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ccd0f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 0x7fa1c81502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ccd0f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 0x7fa1c81502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50704/0 (socket says 172.21.15.7:50704) 2023-03-03T14:52:35.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ccd0f700 1 -- 172.21.15.7:0/2268702177 learned_addr learned my addr 172.21.15.7:0/2268702177 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:52:35.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1c7fff700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1c8071e80 0x7fa1c8149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1c77fe700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c8072250 0x7fa1c814fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ccd0f700 1 -- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1c8071e80 msgr2=0x7fa1c8149fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:35.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ccd0f700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1c8071e80 0x7fa1c8149fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ccd0f700 1 -- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c8072250 msgr2=0x7fa1c814fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:35.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ccd0f700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c8072250 0x7fa1c814fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:35.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.771+0000 7fa1ccd0f700 1 -- 172.21.15.7:0/2268702177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c0009a70 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.772+0000 7fa1ccd0f700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 0x7fa1c81502e0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0013000 tx=0x7fa1c0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:35.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.772+0000 7fa1c4ff9700 1 -- 172.21.15.7:0/2268702177 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c00129f0 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.772+0000 7fa1c4ff9700 1 -- 172.21.15.7:0/2268702177 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c0020070 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.772+0000 7fa1ce772700 1 -- 172.21.15.7:0/2268702177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c8151f30 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.772+0000 7fa1c4ff9700 1 -- 172.21.15.7:0/2268702177 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0012db0 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.772+0000 7fa1ce772700 1 -- 172.21.15.7:0/2268702177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c8151300 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.773+0000 7fa1ce772700 1 -- 172.21.15.7:0/2268702177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c806c9a0 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.773+0000 7fa1c4ff9700 1 -- 172.21.15.7:0/2268702177 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa1c0004040 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.773+0000 7fa1c4ff9700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1b40682f0 0x7fa1b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:52:35.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.774+0000 7fa1c4ff9700 1 -- 172.21.15.7:0/2268702177 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa1c0022070 con 0x7fa1c80c1ce0 2023-03-03T14:52:35.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.774+0000 7fa1c7fff700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1b40682f0 0x7fa1b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:52:35.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.774+0000 7fa1c7fff700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1b40682f0 0x7fa1b406a7a0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8007fd0 tx=0x7fa1b8007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:52:35.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:35.778+0000 7fa1c4ff9700 1 -- 172.21.15.7:0/2268702177 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa1c0054680 con 0x7fa1c80c1ce0 2023-03-03T14:52:36.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.013+0000 7fa1ce772700 1 -- 172.21.15.7:0/2268702177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1c8074bb0 con 0x7fa1c80c1ce0 2023-03-03T14:52:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.013+0000 7fa1c4ff9700 1 -- 172.21.15.7:0/2268702177 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa1c0057ca0 con 0x7fa1c80c1ce0 2023-03-03T14:52:36.017 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:52:36.018 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:52:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 -- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1b40682f0 msgr2=0x7fa1b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:36.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1b40682f0 0x7fa1b406a7a0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8007fd0 tx=0x7fa1b8007dc0 comp rx=0 tx=0).stop 2023-03-03T14:52:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 -- 172.21.15.7:0/2268702177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 msgr2=0x7fa1c81502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:52:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 0x7fa1c81502e0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0013000 tx=0x7fa1c0004ad0 comp rx=0 tx=0).stop 2023-03-03T14:52:36.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 -- 172.21.15.7:0/2268702177 shutdown_connections 2023-03-03T14:52:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1b40682f0 0x7fa1b406a7a0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1c80c1ce0 0x7fa1c81502e0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1c8071e80 0x7fa1c8149fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 --2- 172.21.15.7:0/2268702177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1c8072250 0x7fa1c814fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:52:36.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 -- 172.21.15.7:0/2268702177 >> 172.21.15.7:0/2268702177 conn(0x7fa1c80a94d0 msgr2=0x7fa1c814dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:52:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 -- 172.21.15.7:0/2268702177 shutdown_connections 2023-03-03T14:52:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:52:36.015+0000 7fa1ce772700 1 -- 172.21.15.7:0/2268702177 wait complete. 2023-03-03T14:52:36.022 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:52:36.030 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-03-03T14:52:58.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:52:58.054 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:02.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:53:02.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.168+0000 7fd868a29700 1 Processor -- start 2023-03-03T14:53:02.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.168+0000 7fd868a29700 1 -- start start 2023-03-03T14:53:02.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd868a29700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86406c480 0x7fd8641581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd868a29700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86406d430 con 0x7fd8640c2930 2023-03-03T14:53:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd868a29700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8641586e0 con 0x7fd864151d40 2023-03-03T14:53:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd868a29700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86406c2a0 con 0x7fd86406c480 2023-03-03T14:53:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd861d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd8640c2930 0x7fd864157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44716/0 (socket says 172.21.15.7:44716) 2023-03-03T14:53:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd861d9b700 1 -- 172.21.15.7:0/1401124118 learned_addr learned my addr 172.21.15.7:0/1401124118 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:53:02.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd862d9d700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86406c480 0x7fd8641581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:02.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189658753 0 0) 0x7fd86406d430 con 0x7fd8640c2930 2023-03-03T14:53:02.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd84c0035b0 con 0x7fd8640c2930 2023-03-03T14:53:02.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 788546799 0 0) 0x7fd84c0035b0 con 0x7fd8640c2930 2023-03-03T14:53:02.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86406c480 msgr2=0x7fd8641581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:53:02.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd86159a700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86406c480 0x7fd8641581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:02.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 >> v1:172.21.15.143:6789/0 conn(0x7fd864151d40 legacy=0x7fd8640736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:02.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86406cb00 con 0x7fd8640c2930 2023-03-03T14:53:02.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.169+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd858001670 con 0x7fd8640c2930 2023-03-03T14:53:02.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 >> v1:172.21.15.7:6789/0 conn(0x7fd8640c2930 legacy=0x7fd864157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:02.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86159a700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd84c003aa0 0x7fd84c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:02.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86159a700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84c004850 0x7fd84c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:02.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86159a700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86406c480 0x7fd84c0052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:02.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86406c2a0 con 0x7fd84c003aa0 2023-03-03T14:53:02.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c00bbd0 con 0x7fd84c004850 2023-03-03T14:53:02.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c00bd90 con 0x7fd86406c480 2023-03-03T14:53:02.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86259c700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd84c003aa0 0x7fd84c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:02.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd861d9b700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84c004850 0x7fd84c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:02.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd862d9d700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86406c480 0x7fd84c0052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:02.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86259c700 1 -- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86406c480 msgr2=0x7fd84c0052b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:53:02.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86259c700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd86406c480 0x7fd84c0052b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:02.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86259c700 1 -- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84c004850 msgr2=0x7fd84c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:53:02.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86259c700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84c004850 0x7fd84c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:02.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86259c700 1 -- 172.21.15.7:0/1401124118 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c0035b0 con 0x7fd84c003aa0 2023-03-03T14:53:02.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86259c700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd84c003aa0 0x7fd84c003e70 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fd854007fd0 tx=0x7fd84c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b45fdde13265693 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:53:02.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85400c660 con 0x7fd84c003aa0 2023-03-03T14:53:02.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85400c7e0 con 0x7fd84c003aa0 2023-03-03T14:53:02.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.170+0000 7fd86159a700 1 -- 172.21.15.7:0/1401124118 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85400cb90 con 0x7fd84c003aa0 2023-03-03T14:53:02.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.171+0000 7fd868a29700 1 -- 172.21.15.7:0/1401124118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd84c003aa0 msgr2=0x7fd84c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:02.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.171+0000 7fd868a29700 1 --2- 172.21.15.7:0/1401124118 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd84c003aa0 0x7fd84c003e70 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fd854007fd0 tx=0x7fd84c005880 comp rx=0 tx=0).stop 2023-03-03T14:53:02.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.171+0000 7fd862d9d700 1 -- 172.21.15.7:0/1401124118 reap_dead start 2023-03-03T14:53:02.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.171+0000 7fd868a29700 1 -- 172.21.15.7:0/1401124118 shutdown_connections 2023-03-03T14:53:02.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.171+0000 7fd868a29700 1 -- 172.21.15.7:0/1401124118 >> 172.21.15.7:0/1401124118 conn(0x7fd8640a94d0 msgr2=0x7fd8640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:53:02.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.171+0000 7fd868a29700 1 -- 172.21.15.7:0/1401124118 shutdown_connections 2023-03-03T14:53:02.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.171+0000 7fd868a29700 1 -- 172.21.15.7:0/1401124118 wait complete. 2023-03-03T14:53:02.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.172+0000 7fd868a29700 1 Processor -- start 2023-03-03T14:53:02.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.172+0000 7fd868a29700 1 -- start start 2023-03-03T14:53:02.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.173+0000 7fd868a29700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 0x7fd8640aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:02.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.173+0000 7fd868a29700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8640bfca0 0x7fd864156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:02.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.173+0000 7fd868a29700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8640acec0 0x7fd8640c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:02.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.173+0000 7fd868a29700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd864157710 con 0x7fd8640acec0 2023-03-03T14:53:02.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.173+0000 7fd868a29700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8640be770 con 0x7fd8640bfca0 2023-03-03T14:53:02.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.173+0000 7fd868a29700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8640be4d0 con 0x7fd84c003aa0 2023-03-03T14:53:02.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.173+0000 7fd862d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8640acec0 0x7fd8640c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:02.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.173+0000 7fd862d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8640acec0 0x7fd8640c0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50750/0 (socket says 172.21.15.7:50750) 2023-03-03T14:53:02.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.173+0000 7fd862d9d700 1 -- 172.21.15.7:0/2802387164 learned_addr learned my addr 172.21.15.7:0/2802387164 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:53:02.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd86259c700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 0x7fd8640aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:02.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd861d9b700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8640bfca0 0x7fd864156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:02.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd862d9d700 1 -- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 msgr2=0x7fd8640aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:02.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd862d9d700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 0x7fd8640aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:02.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd862d9d700 1 -- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8640bfca0 msgr2=0x7fd864156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:02.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd862d9d700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8640bfca0 0x7fd864156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:02.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd862d9d700 1 -- 172.21.15.7:0/2802387164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86406d0c0 con 0x7fd8640acec0 2023-03-03T14:53:02.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd862d9d700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8640acec0 0x7fd8640c0c80 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fd86406bcf0 tx=0x7fd85c011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:53:02.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd852ffd700 1 -- 172.21.15.7:0/2802387164 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00e040 con 0x7fd8640acec0 2023-03-03T14:53:02.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd852ffd700 1 -- 172.21.15.7:0/2802387164 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c00f040 con 0x7fd8640acec0 2023-03-03T14:53:02.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd868a29700 1 -- 172.21.15.7:0/2802387164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86406cde0 con 0x7fd8640acec0 2023-03-03T14:53:02.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd852ffd700 1 -- 172.21.15.7:0/2802387164 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00fe40 con 0x7fd8640acec0 2023-03-03T14:53:02.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.174+0000 7fd868a29700 1 -- 172.21.15.7:0/2802387164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd864073e00 con 0x7fd8640acec0 2023-03-03T14:53:02.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.175+0000 7fd868a29700 1 -- 172.21.15.7:0/2802387164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd864074bb0 con 0x7fd8640acec0 2023-03-03T14:53:02.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.175+0000 7fd852ffd700 1 -- 172.21.15.7:0/2802387164 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd85c022460 con 0x7fd8640acec0 2023-03-03T14:53:02.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.176+0000 7fd852ffd700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 0x7fd84406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:02.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.176+0000 7fd852ffd700 1 -- 172.21.15.7:0/2802387164 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd85c086670 con 0x7fd8640acec0 2023-03-03T14:53:02.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.176+0000 7fd86259c700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 0x7fd84406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:02.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.177+0000 7fd86259c700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 0x7fd84406a7a0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fd854014040 tx=0x7fd854005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:53:02.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.180+0000 7fd852ffd700 1 -- 172.21.15.7:0/2802387164 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd85c0529b0 con 0x7fd8640acec0 2023-03-03T14:53:02.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.420+0000 7fd868a29700 1 -- 172.21.15.7:0/2802387164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8640c4a80 con 0x7fd8640acec0 2023-03-03T14:53:02.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.420+0000 7fd852ffd700 1 -- 172.21.15.7:0/2802387164 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd85c055fd0 con 0x7fd8640acec0 2023-03-03T14:53:02.422 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:53:02.423 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:53:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 -- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 msgr2=0x7fd84406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 0x7fd84406a7a0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fd854014040 tx=0x7fd854005040 comp rx=0 tx=0).stop 2023-03-03T14:53:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 -- 172.21.15.7:0/2802387164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8640acec0 msgr2=0x7fd8640c0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8640acec0 0x7fd8640c0c80 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fd86406bcf0 tx=0x7fd85c011f60 comp rx=0 tx=0).stop 2023-03-03T14:53:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 -- 172.21.15.7:0/2802387164 shutdown_connections 2023-03-03T14:53:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8440682f0 0x7fd84406a7a0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8640acec0 0x7fd8640c0c80 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 0x7fd8640aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 --2- 172.21.15.7:0/2802387164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8640bfca0 0x7fd864156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 -- 172.21.15.7:0/2802387164 >> 172.21.15.7:0/2802387164 conn(0x7fd8640a94d0 msgr2=0x7fd8640736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:53:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 -- 172.21.15.7:0/2802387164 shutdown_connections 2023-03-03T14:53:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:02.422+0000 7fd868a29700 1 -- 172.21.15.7:0/2802387164 wait complete. 2023-03-03T14:53:02.427 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:53:02.436 INFO:tasks.mds_thrash.fs.[a]:waiting for 87 secs before thrashing 2023-03-03T14:53:28.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:28.119 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:35.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:53:35.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.359+0000 7f1ddf3af700 1 Processor -- start 2023-03-03T14:53:35.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.359+0000 7f1ddf3af700 1 -- start start 2023-03-03T14:53:35.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.360+0000 7f1ddf3af700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1dd8151770 0x7f1dd8151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:35.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.360+0000 7f1ddf3af700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd8154910 con 0x7f1dd8155390 2023-03-03T14:53:35.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.360+0000 7f1ddf3af700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd80c4540 con 0x7f1dd806e020 2023-03-03T14:53:35.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.360+0000 7f1ddf3af700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd8070b90 con 0x7f1dd8151770 2023-03-03T14:53:35.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.360+0000 7f1ddd14b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1dd8155390 0x7f1dd80c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44756/0 (socket says 172.21.15.7:44756) 2023-03-03T14:53:35.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.360+0000 7f1ddd14b700 1 -- 172.21.15.7:0/3467880373 learned_addr learned my addr 172.21.15.7:0/3467880373 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:53:35.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.360+0000 7f1ddd94c700 1 --2- 172.21.15.7:0/3467880373 >> v2:172.21.15.143:3300/0 conn(0x7f1dd8151770 0x7f1dd8151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.360+0000 7f1dcffff700 1 -- 172.21.15.7:0/3467880373 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260172127 0 0) 0x7f1dd8154910 con 0x7f1dd8155390 2023-03-03T14:53:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.360+0000 7f1dcffff700 1 -- 172.21.15.7:0/3467880373 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc40035b0 con 0x7f1dd8155390 2023-03-03T14:53:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.361+0000 7f1dcffff700 1 -- 172.21.15.7:0/3467880373 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2215185832 0 0) 0x7f1dc40035b0 con 0x7f1dd8155390 2023-03-03T14:53:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.361+0000 7f1ddd94c700 1 -- 172.21.15.7:0/3467880373 >> v1:172.21.15.143:6789/0 conn(0x7f1dd806e020 legacy=0x7f1dd8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.361+0000 7f1ddd94c700 1 -- 172.21.15.7:0/3467880373 >> v1:172.21.15.7:6789/0 conn(0x7f1dd8155390 legacy=0x7f1dd80c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.361+0000 7f1ddd94c700 1 -- 172.21.15.7:0/3467880373 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd80702a0 con 0x7f1dd8151770 2023-03-03T14:53:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.361+0000 7f1ddd94c700 1 --2- 172.21.15.7:0/3467880373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dd8151770 0x7f1dd8151bd0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009d90 tx=0x7f1dd000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eecbc5414cf994 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:53:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.361+0000 7f1dcffff700 1 -- 172.21.15.7:0/3467880373 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd000e070 con 0x7f1dd8151770 2023-03-03T14:53:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.361+0000 7f1dcffff700 1 -- 172.21.15.7:0/3467880373 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0003a20 con 0x7f1dd8151770 2023-03-03T14:53:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.361+0000 7f1dcffff700 1 -- 172.21.15.7:0/3467880373 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8001670 con 0x7f1dd8151770 2023-03-03T14:53:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.362+0000 7f1ddf3af700 1 -- 172.21.15.7:0/3467880373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dd8151770 msgr2=0x7f1dd8151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.362+0000 7f1ddf3af700 1 --2- 172.21.15.7:0/3467880373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dd8151770 0x7f1dd8151bd0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009d90 tx=0x7f1dd000b330 comp rx=0 tx=0).stop 2023-03-03T14:53:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.362+0000 7f1ddf3af700 1 -- 172.21.15.7:0/3467880373 shutdown_connections 2023-03-03T14:53:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.362+0000 7f1ddf3af700 1 --2- 172.21.15.7:0/3467880373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dd8151770 0x7f1dd8151bd0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.362+0000 7f1ddf3af700 1 -- 172.21.15.7:0/3467880373 >> 172.21.15.7:0/3467880373 conn(0x7f1dd806c110 msgr2=0x7f1dd806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:53:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.362+0000 7f1ddf3af700 1 -- 172.21.15.7:0/3467880373 shutdown_connections 2023-03-03T14:53:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.362+0000 7f1ddf3af700 1 -- 172.21.15.7:0/3467880373 wait complete. 2023-03-03T14:53:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.363+0000 7f1ddf3af700 1 Processor -- start 2023-03-03T14:53:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.363+0000 7f1ddf3af700 1 -- start start 2023-03-03T14:53:35.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddf3af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dd806e020 0x7f1dd8145290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddf3af700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd8151770 0x7f1dd814e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddf3af700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dd8155390 0x7f1dd80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddf3af700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8146030 con 0x7f1dd8151770 2023-03-03T14:53:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddf3af700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd80c3760 con 0x7f1dd8155390 2023-03-03T14:53:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddf3af700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd81457d0 con 0x7f1dd806e020 2023-03-03T14:53:35.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddc94a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd8151770 0x7f1dd814e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddc94a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd8151770 0x7f1dd814e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50784/0 (socket says 172.21.15.7:50784) 2023-03-03T14:53:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddc94a700 1 -- 172.21.15.7:0/564441302 learned_addr learned my addr 172.21.15.7:0/564441302 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:53:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddd94c700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dd8155390 0x7f1dd80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddd14b700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dd806e020 0x7f1dd8145290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddc94a700 1 -- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dd806e020 msgr2=0x7f1dd8145290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddc94a700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dd806e020 0x7f1dd8145290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:35.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddc94a700 1 -- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dd8155390 msgr2=0x7f1dd80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddc94a700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dd8155390 0x7f1dd80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddc94a700 1 -- 172.21.15.7:0/564441302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd0009a70 con 0x7f1dd8151770 2023-03-03T14:53:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddd94c700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dd8155390 0x7f1dd80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:53:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddc94a700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd8151770 0x7f1dd814e1b0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400c510 tx=0x7f1dd400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:53:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1dcdffb700 1 -- 172.21.15.7:0/564441302 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40052d0 con 0x7f1dd8151770 2023-03-03T14:53:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1dcdffb700 1 -- 172.21.15.7:0/564441302 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd4005450 con 0x7f1dd8151770 2023-03-03T14:53:35.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1dcdffb700 1 -- 172.21.15.7:0/564441302 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40038c0 con 0x7f1dd8151770 2023-03-03T14:53:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.364+0000 7f1ddf3af700 1 -- 172.21.15.7:0/564441302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd8070820 con 0x7f1dd8151770 2023-03-03T14:53:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.365+0000 7f1ddf3af700 1 -- 172.21.15.7:0/564441302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd8070540 con 0x7f1dd8151770 2023-03-03T14:53:35.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.366+0000 7f1ddf3af700 1 -- 172.21.15.7:0/564441302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd8155760 con 0x7f1dd8151770 2023-03-03T14:53:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.366+0000 7f1dcdffb700 1 -- 172.21.15.7:0/564441302 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1dd4019030 con 0x7f1dd8151770 2023-03-03T14:53:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.366+0000 7f1dcdffb700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dbc0682f0 0x7f1dbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:53:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.367+0000 7f1dcdffb700 1 -- 172.21.15.7:0/564441302 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1dd40852a0 con 0x7f1dd8151770 2023-03-03T14:53:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.367+0000 7f1ddd14b700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dbc0682f0 0x7f1dbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:53:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.367+0000 7f1ddd14b700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dbc0682f0 0x7f1dbc06a7a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8146e90 tx=0x7f1dc8001bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:53:35.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.371+0000 7f1dcdffb700 1 -- 172.21.15.7:0/564441302 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1dd4051640 con 0x7f1dd8151770 2023-03-03T14:53:35.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.617+0000 7f1ddf3af700 1 -- 172.21.15.7:0/564441302 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dd80c4540 con 0x7f1dd8151770 2023-03-03T14:53:35.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.617+0000 7f1dcdffb700 1 -- 172.21.15.7:0/564441302 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1dd80c4540 con 0x7f1dd8151770 2023-03-03T14:53:35.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:53:35.620 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:53:35.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.619+0000 7f1ddf3af700 1 -- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dbc0682f0 msgr2=0x7f1dbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:35.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.619+0000 7f1ddf3af700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dbc0682f0 0x7f1dbc06a7a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8146e90 tx=0x7f1dc8001bc0 comp rx=0 tx=0).stop 2023-03-03T14:53:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.619+0000 7f1ddf3af700 1 -- 172.21.15.7:0/564441302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd8151770 msgr2=0x7f1dd814e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:53:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.619+0000 7f1ddf3af700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd8151770 0x7f1dd814e1b0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400c510 tx=0x7f1dd400c8d0 comp rx=0 tx=0).stop 2023-03-03T14:53:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.619+0000 7f1ddf3af700 1 -- 172.21.15.7:0/564441302 shutdown_connections 2023-03-03T14:53:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.619+0000 7f1ddf3af700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dd8155390 0x7f1dd80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.619+0000 7f1ddf3af700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dd806e020 0x7f1dd8145290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.619+0000 7f1ddf3af700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dbc0682f0 0x7f1dbc06a7a0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.619+0000 7f1ddf3af700 1 --2- 172.21.15.7:0/564441302 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd8151770 0x7f1dd814e1b0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:53:35.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.620+0000 7f1ddf3af700 1 -- 172.21.15.7:0/564441302 >> 172.21.15.7:0/564441302 conn(0x7f1dd806c110 msgr2=0x7f1dd80c25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:53:35.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.620+0000 7f1ddf3af700 1 -- 172.21.15.7:0/564441302 shutdown_connections 2023-03-03T14:53:35.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:53:35.620+0000 7f1ddf3af700 1 -- 172.21.15.7:0/564441302 wait complete. 2023-03-03T14:53:35.623 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:53:35.633 INFO:tasks.mds_thrash.fs.[b]:waiting for 99 secs before thrashing 2023-03-03T14:53:58.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:53:58.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:28.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:28.276 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:29.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:54:29.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.575+0000 7effb7fd1700 1 Processor -- start 2023-03-03T14:54:29.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.575+0000 7effb7fd1700 1 -- start start 2023-03-03T14:54:29.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.576+0000 7effb7fd1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7effb006d870 0x7effb01550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.576+0000 7effb7fd1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb00c7a60 con 0x7effb00c1da0 2023-03-03T14:54:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.576+0000 7effb7fd1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb00c3620 con 0x7effb00c3e30 2023-03-03T14:54:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.576+0000 7effb7fd1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb00c2af0 con 0x7effb006d870 2023-03-03T14:54:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.576+0000 7effb5d6d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7effb006d870 0x7effb01550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.576+0000 7effb556c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7effb00c1da0 0x7effb00c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44802/0 (socket says 172.21.15.7:44802) 2023-03-03T14:54:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.576+0000 7effb556c700 1 -- 172.21.15.7:0/3495216280 learned_addr learned my addr 172.21.15.7:0/3495216280 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:54:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.577+0000 7effb5d6d700 1 -- 172.21.15.7:0/3495216280 >> v1:172.21.15.143:6789/0 conn(0x7effb00c3e30 legacy=0x7effb015ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:54:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.577+0000 7effb5d6d700 1 -- 172.21.15.7:0/3495216280 >> v1:172.21.15.7:6789/0 conn(0x7effb00c1da0 legacy=0x7effb00c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.577+0000 7effb4d6b700 1 -- 172.21.15.7:0/3495216280 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206295142 0 0) 0x7effb00c7a60 con 0x7effb00c1da0 2023-03-03T14:54:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.577+0000 7effb5d6d700 1 -- 172.21.15.7:0/3495216280 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb01545e0 con 0x7effb006d870 2023-03-03T14:54:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.577+0000 7effb5d6d700 1 --2- 172.21.15.7:0/3495216280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb006d870 0x7effb01550c0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7effa0009840 tx=0x7effa0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bff06c2d868f61 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:29.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.577+0000 7effb4d6b700 1 -- 172.21.15.7:0/3495216280 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa0004230 con 0x7effb006d870 2023-03-03T14:54:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.577+0000 7effb4d6b700 1 -- 172.21.15.7:0/3495216280 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa0004390 con 0x7effb006d870 2023-03-03T14:54:29.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.577+0000 7effb4d6b700 1 -- 172.21.15.7:0/3495216280 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa00036a0 con 0x7effb006d870 2023-03-03T14:54:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.578+0000 7effb7fd1700 1 -- 172.21.15.7:0/3495216280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb006d870 msgr2=0x7effb01550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.578+0000 7effb7fd1700 1 --2- 172.21.15.7:0/3495216280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb006d870 0x7effb01550c0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7effa0009840 tx=0x7effa0009b50 comp rx=0 tx=0).stop 2023-03-03T14:54:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.578+0000 7effb7fd1700 1 -- 172.21.15.7:0/3495216280 shutdown_connections 2023-03-03T14:54:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.578+0000 7effb7fd1700 1 --2- 172.21.15.7:0/3495216280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb006d870 0x7effb01550c0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:29.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.578+0000 7effb7fd1700 1 -- 172.21.15.7:0/3495216280 >> 172.21.15.7:0/3495216280 conn(0x7effb006c110 msgr2=0x7effb006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:54:29.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.578+0000 7effb7fd1700 1 -- 172.21.15.7:0/3495216280 shutdown_connections 2023-03-03T14:54:29.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.578+0000 7effb7fd1700 1 -- 172.21.15.7:0/3495216280 wait complete. 2023-03-03T14:54:29.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.579+0000 7effb7fd1700 1 Processor -- start 2023-03-03T14:54:29.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.579+0000 7effb7fd1700 1 -- start start 2023-03-03T14:54:29.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb7fd1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effb006d870 0x7effb01ed280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:29.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb7fd1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb00c1da0 0x7effb014ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb7fd1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effb00c3e30 0x7effb013b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb7fd1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb013c550 con 0x7effb00c1da0 2023-03-03T14:54:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb7fd1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb0151bf0 con 0x7effb00c3e30 2023-03-03T14:54:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb7fd1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb014eb90 con 0x7effb006d870 2023-03-03T14:54:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb556c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb00c1da0 0x7effb014ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb556c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb00c1da0 0x7effb014ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50830/0 (socket says 172.21.15.7:50830) 2023-03-03T14:54:29.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb556c700 1 -- 172.21.15.7:0/4132334642 learned_addr learned my addr 172.21.15.7:0/4132334642 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:54:29.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb5d6d700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effb006d870 0x7effb01ed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:29.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb656e700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effb00c3e30 0x7effb013b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:29.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb556c700 1 -- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effb006d870 msgr2=0x7effb01ed280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:29.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb556c700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effb006d870 0x7effb01ed280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:29.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb556c700 1 -- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effb00c3e30 msgr2=0x7effb013b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:29.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb556c700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effb00c3e30 0x7effb013b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:29.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb556c700 1 -- 172.21.15.7:0/4132334642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa00094a0 con 0x7effb00c1da0 2023-03-03T14:54:29.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb556c700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb00c1da0 0x7effb014ee50 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7effac00c750 tx=0x7effac00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effa67fc700 1 -- 172.21.15.7:0/4132334642 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac004c80 con 0x7effb00c1da0 2023-03-03T14:54:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effa67fc700 1 -- 172.21.15.7:0/4132334642 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac004e00 con 0x7effb00c1da0 2023-03-03T14:54:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb7fd1700 1 -- 172.21.15.7:0/4132334642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb0154ba0 con 0x7effb00c1da0 2023-03-03T14:54:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effa67fc700 1 -- 172.21.15.7:0/4132334642 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac007310 con 0x7effb00c1da0 2023-03-03T14:54:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.580+0000 7effb7fd1700 1 -- 172.21.15.7:0/4132334642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb01548c0 con 0x7effb00c1da0 2023-03-03T14:54:29.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.581+0000 7effb7fd1700 1 -- 172.21.15.7:0/4132334642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb006dc40 con 0x7effb00c1da0 2023-03-03T14:54:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.582+0000 7effa67fc700 1 -- 172.21.15.7:0/4132334642 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7effac01a020 con 0x7effb00c1da0 2023-03-03T14:54:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.582+0000 7effa67fc700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff940682f0 0x7eff9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:29.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.582+0000 7effa67fc700 1 -- 172.21.15.7:0/4132334642 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7effac085db0 con 0x7effb00c1da0 2023-03-03T14:54:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.582+0000 7effb5d6d700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff940682f0 0x7eff9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.583+0000 7effb5d6d700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff940682f0 0x7eff9406a7a0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7effb013d1f0 tx=0x7effa001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:29.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.587+0000 7effa67fc700 1 -- 172.21.15.7:0/4132334642 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7effac0520f0 con 0x7effb00c1da0 2023-03-03T14:54:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.829+0000 7effb7fd1700 1 -- 172.21.15.7:0/4132334642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effb0074bb0 con 0x7effb00c1da0 2023-03-03T14:54:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.829+0000 7effa67fc700 1 -- 172.21.15.7:0/4132334642 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7effac055710 con 0x7effb00c1da0 2023-03-03T14:54:29.832 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:54:29.832 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:54:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 -- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff940682f0 msgr2=0x7eff9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff940682f0 0x7eff9406a7a0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7effb013d1f0 tx=0x7effa001c040 comp rx=0 tx=0).stop 2023-03-03T14:54:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 -- 172.21.15.7:0/4132334642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb00c1da0 msgr2=0x7effb014ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb00c1da0 0x7effb014ee50 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7effac00c750 tx=0x7effac00cb10 comp rx=0 tx=0).stop 2023-03-03T14:54:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 -- 172.21.15.7:0/4132334642 shutdown_connections 2023-03-03T14:54:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effb00c3e30 0x7effb013b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effb006d870 0x7effb01ed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff940682f0 0x7eff9406a7a0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 --2- 172.21.15.7:0/4132334642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effb00c1da0 0x7effb014ee50 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:29.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 -- 172.21.15.7:0/4132334642 >> 172.21.15.7:0/4132334642 conn(0x7effb006c110 msgr2=0x7effb006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:54:29.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 -- 172.21.15.7:0/4132334642 shutdown_connections 2023-03-03T14:54:29.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:29.831+0000 7effb7fd1700 1 -- 172.21.15.7:0/4132334642 wait complete. 2023-03-03T14:54:29.836 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:54:29.845 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before thrashing 2023-03-03T14:54:32.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:54:32.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.983+0000 7fc83cda2700 1 Processor -- start 2023-03-03T14:54:32.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.983+0000 7fc83cda2700 1 -- start start 2023-03-03T14:54:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.984+0000 7fc83cda2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc83814f9d0 0x7fc838156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.984+0000 7fc83cda2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc8380abe20 0x7fc8381529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.984+0000 7fc83cda2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8380c2f30 con 0x7fc83814fda0 2023-03-03T14:54:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.984+0000 7fc83cda2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8380c31d0 con 0x7fc8380abe20 2023-03-03T14:54:32.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.984+0000 7fc83cda2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8381524a0 con 0x7fc83814f9d0 2023-03-03T14:54:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc836d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc8380abe20 0x7fc8381529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc835d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc83814fda0 0x7fc838071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44818/0 (socket says 172.21.15.7:44818) 2023-03-03T14:54:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc835d9b700 1 -- 172.21.15.7:0/366497976 learned_addr learned my addr 172.21.15.7:0/366497976 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:54:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc83659c700 1 --2- 172.21.15.7:0/366497976 >> v2:172.21.15.143:3300/0 conn(0x7fc83814f9d0 0x7fc838156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:32.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc836d9d700 1 -- 172.21.15.7:0/366497976 >> v2:172.21.15.143:3300/0 conn(0x7fc83814f9d0 msgr2=0x7fc838156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:54:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc836d9d700 1 --2- 172.21.15.7:0/366497976 >> v2:172.21.15.143:3300/0 conn(0x7fc83814f9d0 0x7fc838156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc836d9d700 1 -- 172.21.15.7:0/366497976 >> v1:172.21.15.7:6789/0 conn(0x7fc83814fda0 legacy=0x7fc838071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc836d9d700 1 -- 172.21.15.7:0/366497976 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8380c34b0 con 0x7fc8380abe20 2023-03-03T14:54:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc83659c700 1 --2- 172.21.15.7:0/366497976 >> v2:172.21.15.143:3300/0 conn(0x7fc83814f9d0 0x7fc838156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:54:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc836d9d700 1 --2- 172.21.15.7:0/366497976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8380abe20 0x7fc8381529c0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fc8380710a0 tx=0x7fc83000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eeaab1ec5681e64 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:32.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc83559a700 1 -- 172.21.15.7:0/366497976 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83000e070 con 0x7fc8380abe20 2023-03-03T14:54:32.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc83559a700 1 -- 172.21.15.7:0/366497976 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83000bbd0 con 0x7fc8380abe20 2023-03-03T14:54:32.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.985+0000 7fc83559a700 1 -- 172.21.15.7:0/366497976 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830012430 con 0x7fc8380abe20 2023-03-03T14:54:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.986+0000 7fc83cda2700 1 -- 172.21.15.7:0/366497976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8380abe20 msgr2=0x7fc8381529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.986+0000 7fc83cda2700 1 --2- 172.21.15.7:0/366497976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8380abe20 0x7fc8381529c0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fc8380710a0 tx=0x7fc83000b330 comp rx=0 tx=0).stop 2023-03-03T14:54:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.986+0000 7fc83cda2700 1 -- 172.21.15.7:0/366497976 shutdown_connections 2023-03-03T14:54:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.986+0000 7fc83cda2700 1 --2- 172.21.15.7:0/366497976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8380abe20 0x7fc8381529c0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.986+0000 7fc83cda2700 1 --2- 172.21.15.7:0/366497976 >> v2:172.21.15.143:3300/0 conn(0x7fc83814f9d0 0x7fc838156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:32.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.986+0000 7fc83cda2700 1 -- 172.21.15.7:0/366497976 >> 172.21.15.7:0/366497976 conn(0x7fc8380c8900 msgr2=0x7fc8380c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:54:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.986+0000 7fc83cda2700 1 -- 172.21.15.7:0/366497976 shutdown_connections 2023-03-03T14:54:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.986+0000 7fc83cda2700 1 -- 172.21.15.7:0/366497976 wait complete. 2023-03-03T14:54:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.988+0000 7fc83cda2700 1 Processor -- start 2023-03-03T14:54:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.988+0000 7fc83cda2700 1 -- start start 2023-03-03T14:54:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc83cda2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8380abe20 0x7fc83806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:32.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc83cda2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc83814f9d0 0x7fc838152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:32.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc83cda2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc83814fda0 0x7fc8381456a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:32.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc83cda2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83806f420 con 0x7fc83814fda0 2023-03-03T14:54:32.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc83cda2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8381ea270 con 0x7fc83814f9d0 2023-03-03T14:54:32.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc83cda2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8381ea550 con 0x7fc8380abe20 2023-03-03T14:54:32.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc836d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc83814fda0 0x7fc8381456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:32.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc836d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc83814fda0 0x7fc8381456a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50848/0 (socket says 172.21.15.7:50848) 2023-03-03T14:54:32.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc836d9d700 1 -- 172.21.15.7:0/3089927583 learned_addr learned my addr 172.21.15.7:0/3089927583 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:54:32.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc83659c700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8380abe20 0x7fc83806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:32.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc835d9b700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc83814f9d0 0x7fc838152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:32.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc836d9d700 1 -- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8380abe20 msgr2=0x7fc83806a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:32.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc836d9d700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8380abe20 0x7fc83806a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:32.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc836d9d700 1 -- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc83814f9d0 msgr2=0x7fc838152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:32.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc836d9d700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc83814f9d0 0x7fc838152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:32.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc836d9d700 1 -- 172.21.15.7:0/3089927583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc830009a70 con 0x7fc83814fda0 2023-03-03T14:54:32.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.989+0000 7fc836d9d700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc83814fda0 0x7fc8381456a0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fc830014040 tx=0x7fc830004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.990+0000 7fc826ffd700 1 -- 172.21.15.7:0/3089927583 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830012430 con 0x7fc83814fda0 2023-03-03T14:54:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.990+0000 7fc826ffd700 1 -- 172.21.15.7:0/3089927583 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc830020070 con 0x7fc83814fda0 2023-03-03T14:54:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.990+0000 7fc826ffd700 1 -- 172.21.15.7:0/3089927583 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830004280 con 0x7fc83814fda0 2023-03-03T14:54:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.990+0000 7fc83cda2700 1 -- 172.21.15.7:0/3089927583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8380ac350 con 0x7fc83814fda0 2023-03-03T14:54:32.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.990+0000 7fc83cda2700 1 -- 172.21.15.7:0/3089927583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8380ab630 con 0x7fc83814fda0 2023-03-03T14:54:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.991+0000 7fc83cda2700 1 -- 172.21.15.7:0/3089927583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc838071fa0 con 0x7fc83814fda0 2023-03-03T14:54:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.991+0000 7fc826ffd700 1 -- 172.21.15.7:0/3089927583 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc83002e070 con 0x7fc83814fda0 2023-03-03T14:54:32.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.992+0000 7fc826ffd700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8180682f0 0x7fc81806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:54:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.992+0000 7fc826ffd700 1 -- 172.21.15.7:0/3089927583 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc830022070 con 0x7fc83814fda0 2023-03-03T14:54:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.992+0000 7fc83659c700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8180682f0 0x7fc81806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:54:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.992+0000 7fc83659c700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8180682f0 0x7fc81806a7a0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009990 tx=0x7fc82c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:54:32.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:32.996+0000 7fc826ffd700 1 -- 172.21.15.7:0/3089927583 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc830054330 con 0x7fc83814fda0 2023-03-03T14:54:33.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.234+0000 7fc83cda2700 1 -- 172.21.15.7:0/3089927583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc838074bb0 con 0x7fc83814fda0 2023-03-03T14:54:33.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.234+0000 7fc826ffd700 1 -- 172.21.15.7:0/3089927583 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc830057950 con 0x7fc83814fda0 2023-03-03T14:54:33.236 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:54:33.237 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:54:33.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.236+0000 7fc83cda2700 1 -- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8180682f0 msgr2=0x7fc81806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:33.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.236+0000 7fc83cda2700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8180682f0 0x7fc81806a7a0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fc82c009990 tx=0x7fc82c008040 comp rx=0 tx=0).stop 2023-03-03T14:54:33.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.236+0000 7fc83cda2700 1 -- 172.21.15.7:0/3089927583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc83814fda0 msgr2=0x7fc8381456a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:54:33.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.236+0000 7fc83cda2700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc83814fda0 0x7fc8381456a0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fc830014040 tx=0x7fc830004ad0 comp rx=0 tx=0).stop 2023-03-03T14:54:33.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.236+0000 7fc83cda2700 1 -- 172.21.15.7:0/3089927583 shutdown_connections 2023-03-03T14:54:33.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.236+0000 7fc83cda2700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8180682f0 0x7fc81806a7a0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:33.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.236+0000 7fc83cda2700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc83814fda0 0x7fc8381456a0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:33.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.236+0000 7fc83cda2700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8380abe20 0x7fc83806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:33.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.236+0000 7fc83cda2700 1 --2- 172.21.15.7:0/3089927583 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc83814f9d0 0x7fc838152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:54:33.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.236+0000 7fc83cda2700 1 -- 172.21.15.7:0/3089927583 >> 172.21.15.7:0/3089927583 conn(0x7fc8380c8900 msgr2=0x7fc8380c6550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:54:33.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.237+0000 7fc83cda2700 1 -- 172.21.15.7:0/3089927583 shutdown_connections 2023-03-03T14:54:33.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:54:33.237+0000 7fc83cda2700 1 -- 172.21.15.7:0/3089927583 wait complete. 2023-03-03T14:54:33.240 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:54:33.250 INFO:tasks.mds_thrash.fs.[a]:waiting for 93 secs before thrashing 2023-03-03T14:54:58.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:54:58.351 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:14.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:55:14.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.772+0000 7f7c76cd4700 1 Processor -- start 2023-03-03T14:55:14.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.772+0000 7f7c76cd4700 1 -- start start 2023-03-03T14:55:14.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c76cd4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7c7014b0d0 0x7f7c70071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:14.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c76cd4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c700aad20 0x7f7c70157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c76cd4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c700a9c20 con 0x7f7c70071d40 2023-03-03T14:55:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c76cd4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c70154f10 con 0x7f7c700aad20 2023-03-03T14:55:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c76cd4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c700ac3c0 con 0x7f7c7014b0d0 2023-03-03T14:55:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c6ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c700aad20 0x7f7c70157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c75271700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7c70071d40 0x7f7c70158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44864/0 (socket says 172.21.15.7:44864) 2023-03-03T14:55:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c75271700 1 -- 172.21.15.7:0/3552689968 learned_addr learned my addr 172.21.15.7:0/3552689968 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:55:14.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c6ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c700aad20 0x7f7c70157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50882/0 (socket says 172.21.15.7:50882) 2023-03-03T14:55:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c74a70700 1 --2- 172.21.15.7:0/3552689968 >> v2:172.21.15.143:3300/0 conn(0x7f7c7014b0d0 0x7f7c70071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c6f7fe700 1 -- 172.21.15.7:0/3552689968 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054925665 0 0) 0x7f7c700a9c20 con 0x7f7c70071d40 2023-03-03T14:55:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c6f7fe700 1 -- 172.21.15.7:0/3552689968 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c600035b0 con 0x7f7c70071d40 2023-03-03T14:55:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c6ffff700 1 -- 172.21.15.7:0/3552689968 >> v2:172.21.15.143:3300/0 conn(0x7f7c7014b0d0 msgr2=0x7f7c70071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:55:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c6ffff700 1 --2- 172.21.15.7:0/3552689968 >> v2:172.21.15.143:3300/0 conn(0x7f7c7014b0d0 0x7f7c70071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c6ffff700 1 -- 172.21.15.7:0/3552689968 >> v1:172.21.15.7:6789/0 conn(0x7f7c70071d40 legacy=0x7f7c70158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c6ffff700 1 -- 172.21.15.7:0/3552689968 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7006fc70 con 0x7f7c700aad20 2023-03-03T14:55:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.773+0000 7f7c6ffff700 1 --2- 172.21.15.7:0/3552689968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c700aad20 0x7f7c70157df0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f7c70070930 tx=0x7f7c5c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ff5f31e78a82ef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:55:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c6f7fe700 1 -- 172.21.15.7:0/3552689968 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c011830 con 0x7f7c700aad20 2023-03-03T14:55:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c6f7fe700 1 -- 172.21.15.7:0/3552689968 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c011990 con 0x7f7c700aad20 2023-03-03T14:55:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c6f7fe700 1 -- 172.21.15.7:0/3552689968 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c011d20 con 0x7f7c700aad20 2023-03-03T14:55:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3552689968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c700aad20 msgr2=0x7f7c70157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c76cd4700 1 --2- 172.21.15.7:0/3552689968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c700aad20 0x7f7c70157df0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f7c70070930 tx=0x7f7c5c009b20 comp rx=0 tx=0).stop 2023-03-03T14:55:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3552689968 shutdown_connections 2023-03-03T14:55:14.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c76cd4700 1 --2- 172.21.15.7:0/3552689968 >> v2:172.21.15.143:3300/0 conn(0x7f7c7014b0d0 0x7f7c70071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:14.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c76cd4700 1 --2- 172.21.15.7:0/3552689968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c700aad20 0x7f7c70157df0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:14.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3552689968 >> 172.21.15.7:0/3552689968 conn(0x7f7c700c8900 msgr2=0x7f7c700c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:55:14.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3552689968 shutdown_connections 2023-03-03T14:55:14.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.774+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3552689968 wait complete. 2023-03-03T14:55:14.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.775+0000 7f7c76cd4700 1 Processor -- start 2023-03-03T14:55:14.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.775+0000 7f7c76cd4700 1 -- start start 2023-03-03T14:55:14.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.776+0000 7f7c76cd4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c70071d40 0x7f7c70157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:14.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.776+0000 7f7c76cd4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c700aad20 0x7f7c70143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:14.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.776+0000 7f7c76cd4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c7014b0d0 0x7f7c70143780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:14.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.776+0000 7f7c76cd4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70156500 con 0x7f7c7014b0d0 2023-03-03T14:55:14.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.776+0000 7f7c76cd4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70155880 con 0x7f7c700aad20 2023-03-03T14:55:14.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.776+0000 7f7c76cd4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70155b60 con 0x7f7c70071d40 2023-03-03T14:55:14.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.776+0000 7f7c75271700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c7014b0d0 0x7f7c70143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:14.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.776+0000 7f7c75271700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c7014b0d0 0x7f7c70143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50890/0 (socket says 172.21.15.7:50890) 2023-03-03T14:55:14.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.776+0000 7f7c75271700 1 -- 172.21.15.7:0/3571021180 learned_addr learned my addr 172.21.15.7:0/3571021180 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:55:14.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c74a70700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c70071d40 0x7f7c70157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:14.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c6ffff700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c700aad20 0x7f7c70143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:14.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c75271700 1 -- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c70071d40 msgr2=0x7f7c70157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:14.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c75271700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c70071d40 0x7f7c70157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:14.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c75271700 1 -- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c700aad20 msgr2=0x7f7c70143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:14.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c75271700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c700aad20 0x7f7c70143240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:14.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c75271700 1 -- 172.21.15.7:0/3571021180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c0094a0 con 0x7f7c7014b0d0 2023-03-03T14:55:14.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c74a70700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c70071d40 0x7f7c70157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:55:14.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c6ffff700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c700aad20 0x7f7c70143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:55:14.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c75271700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c7014b0d0 0x7f7c70143780 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800ca70 tx=0x7f7c6800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:55:14.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c6d7fa700 1 -- 172.21.15.7:0/3571021180 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6800aaa0 con 0x7f7c7014b0d0 2023-03-03T14:55:14.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c6d7fa700 1 -- 172.21.15.7:0/3571021180 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6800ac00 con 0x7f7c7014b0d0 2023-03-03T14:55:14.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3571021180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7006f6b0 con 0x7f7c7014b0d0 2023-03-03T14:55:14.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c6d7fa700 1 -- 172.21.15.7:0/3571021180 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c68012400 con 0x7f7c7014b0d0 2023-03-03T14:55:14.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.777+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3571021180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c70155160 con 0x7f7c7014b0d0 2023-03-03T14:55:14.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.778+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3571021180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7006f990 con 0x7f7c7014b0d0 2023-03-03T14:55:14.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.779+0000 7f7c6d7fa700 1 -- 172.21.15.7:0/3571021180 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7c68019030 con 0x7f7c7014b0d0 2023-03-03T14:55:14.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.779+0000 7f7c6d7fa700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 0x7f7c5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:55:14.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.779+0000 7f7c6d7fa700 1 -- 172.21.15.7:0/3571021180 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7c68085c40 con 0x7f7c7014b0d0 2023-03-03T14:55:14.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.779+0000 7f7c74a70700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 0x7f7c5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:55:14.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.780+0000 7f7c74a70700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 0x7f7c5406a7a0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f7c70154430 tx=0x7f7c64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:55:14.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:14.783+0000 7f7c6d7fa700 1 -- 172.21.15.7:0/3571021180 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7c6804b5e0 con 0x7f7c7014b0d0 2023-03-03T14:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.024+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3571021180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c70074bb0 con 0x7f7c7014b0d0 2023-03-03T14:55:15.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.024+0000 7f7c6d7fa700 1 -- 172.21.15.7:0/3571021180 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7c68052030 con 0x7f7c7014b0d0 2023-03-03T14:55:15.027 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:55:15.027 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:55:15.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.026+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 msgr2=0x7f7c5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:15.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.026+0000 7f7c76cd4700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 0x7f7c5406a7a0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f7c70154430 tx=0x7f7c64008040 comp rx=0 tx=0).stop 2023-03-03T14:55:15.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.026+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3571021180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c7014b0d0 msgr2=0x7f7c70143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:55:15.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.026+0000 7f7c76cd4700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c7014b0d0 0x7f7c70143780 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800ca70 tx=0x7f7c6800ce30 comp rx=0 tx=0).stop 2023-03-03T14:55:15.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.027+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3571021180 shutdown_connections 2023-03-03T14:55:15.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.027+0000 7f7c76cd4700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 0x7f7c5406a7a0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:15.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.027+0000 7f7c76cd4700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c7014b0d0 0x7f7c70143780 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:15.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.027+0000 7f7c76cd4700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c70071d40 0x7f7c70157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:15.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.027+0000 7f7c76cd4700 1 --2- 172.21.15.7:0/3571021180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c700aad20 0x7f7c70143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:55:15.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.027+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3571021180 >> 172.21.15.7:0/3571021180 conn(0x7f7c700c8900 msgr2=0x7f7c700acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:55:15.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.027+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3571021180 shutdown_connections 2023-03-03T14:55:15.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:55:15.027+0000 7f7c76cd4700 1 -- 172.21.15.7:0/3571021180 wait complete. 2023-03-03T14:55:15.030 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:55:15.040 INFO:tasks.mds_thrash.fs.[b]:waiting for 63 secs before thrashing 2023-03-03T14:55:28.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:28.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:58.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:55:58.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:06.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:56:06.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.390+0000 7f2d8271d700 1 Processor -- start 2023-03-03T14:56:06.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.390+0000 7f2d8271d700 1 -- start start 2023-03-03T14:56:06.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.391+0000 7f2d8271d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2d7c06a2b0 0x7f2d7c06a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:06.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.391+0000 7f2d8271d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c157a40 0x7f2d7c157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:06.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.391+0000 7f2d8271d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7c0c1c10 con 0x7f2d7c0ab0c0 2023-03-03T14:56:06.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.391+0000 7f2d8271d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7c154530 con 0x7f2d7c06a2b0 2023-03-03T14:56:06.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.391+0000 7f2d8271d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c154810 con 0x7f2d7c157a40 2023-03-03T14:56:06.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.391+0000 7f2d7bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2d7c0ab0c0 0x7f2d7c0c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44898/0 (socket says 172.21.15.7:44898) 2023-03-03T14:56:06.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.391+0000 7f2d7bfff700 1 -- 172.21.15.7:0/2067414772 learned_addr learned my addr 172.21.15.7:0/2067414772 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:56:06.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7b7fe700 1 --2- 172.21.15.7:0/2067414772 >> v2:172.21.15.143:3300/0 conn(0x7f2d7c06a2b0 0x7f2d7c06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:06.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d80cba700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c157a40 0x7f2d7c157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:06.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398347037 0 0) 0x7f2d6c001110 con 0x7f2d7c0ab0c0 2023-03-03T14:56:06.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d680035b0 con 0x7f2d7c0ab0c0 2023-03-03T14:56:06.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1569376422 0 0) 0x7f2d680035b0 con 0x7f2d7c0ab0c0 2023-03-03T14:56:06.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c157a40 msgr2=0x7f2d7c157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:56:06.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c157a40 0x7f2d7c157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 >> v2:172.21.15.143:3300/0 conn(0x7f2d7c06a2b0 msgr2=0x7f2d7c06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:56:06.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 --2- 172.21.15.7:0/2067414772 >> v2:172.21.15.143:3300/0 conn(0x7f2d7c06a2b0 0x7f2d7c06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c154af0 con 0x7f2d7c0ab0c0 2023-03-03T14:56:06.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f2d6c001800 con 0x7f2d7c0ab0c0 2023-03-03T14:56:06.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 >> v1:172.21.15.7:6789/0 conn(0x7f2d7c0ab0c0 legacy=0x7f2d7c0c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:06.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d68003a70 0x7f2d68003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:06.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c157a40 0x7f2d68004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:06.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d68004d90 0x7f2d68005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:06.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c154530 con 0x7f2d68004d90 2023-03-03T14:56:06.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c154810 con 0x7f2d68003a70 2023-03-03T14:56:06.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6800ba10 con 0x7f2d7c157a40 2023-03-03T14:56:06.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d80cba700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d68004d90 0x7f2d68005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:06.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7b7fe700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c157a40 0x7f2d68004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:06.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d7bfff700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d68003a70 0x7f2d68003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:06.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d80cba700 1 -- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c157a40 msgr2=0x7f2d68004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:56:06.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d80cba700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c157a40 0x7f2d68004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d80cba700 1 -- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d68003a70 msgr2=0x7f2d68003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:56:06.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d80cba700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d68003a70 0x7f2d68003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.392+0000 7f2d80cba700 1 -- 172.21.15.7:0/2067414772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680035b0 con 0x7f2d68004d90 2023-03-03T14:56:06.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d80cba700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d68004d90 0x7f2d68005200 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f2d74009fc0 tx=0x7f2d68005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6ba878ec556254 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:06.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7400ebd0 con 0x7f2d68004d90 2023-03-03T14:56:06.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7400f040 con 0x7f2d68004d90 2023-03-03T14:56:06.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d7affd700 1 -- 172.21.15.7:0/2067414772 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74014480 con 0x7f2d68004d90 2023-03-03T14:56:06.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d8271d700 1 -- 172.21.15.7:0/2067414772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d68004d90 msgr2=0x7f2d68005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:06.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d8271d700 1 --2- 172.21.15.7:0/2067414772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d68004d90 0x7f2d68005200 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f2d74009fc0 tx=0x7f2d68005880 comp rx=0 tx=0).stop 2023-03-03T14:56:06.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d80cba700 1 -- 172.21.15.7:0/2067414772 reap_dead start 2023-03-03T14:56:06.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d8271d700 1 -- 172.21.15.7:0/2067414772 shutdown_connections 2023-03-03T14:56:06.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d8271d700 1 -- 172.21.15.7:0/2067414772 >> 172.21.15.7:0/2067414772 conn(0x7f2d7c0a9430 msgr2=0x7f2d7c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:56:06.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d8271d700 1 -- 172.21.15.7:0/2067414772 shutdown_connections 2023-03-03T14:56:06.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.393+0000 7f2d8271d700 1 -- 172.21.15.7:0/2067414772 wait complete. 2023-03-03T14:56:06.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.395+0000 7f2d8271d700 1 Processor -- start 2023-03-03T14:56:06.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.395+0000 7f2d8271d700 1 -- start start 2023-03-03T14:56:06.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.395+0000 7f2d8271d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c1ddc90 0x7f2d7c0c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:06.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d8271d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c1de060 0x7f2d7c0c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:06.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d8271d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0723f0 0x7f2d7c06b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:06.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d8271d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0c58b0 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d8271d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c1de7c0 con 0x7f2d7c1de060 2023-03-03T14:56:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d8271d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c156910 con 0x7f2d7c0723f0 2023-03-03T14:56:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d7bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c1ddc90 0x7f2d7c0c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d7bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c1ddc90 0x7f2d7c0c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50930/0 (socket says 172.21.15.7:50930) 2023-03-03T14:56:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d7bfff700 1 -- 172.21.15.7:0/1280270947 learned_addr learned my addr 172.21.15.7:0/1280270947 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:56:06.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d80cba700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0723f0 0x7f2d7c06b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d7b7fe700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c1de060 0x7f2d7c0c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d7bfff700 1 -- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0723f0 msgr2=0x7f2d7c06b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d7bfff700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0723f0 0x7f2d7c06b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d7bfff700 1 -- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c1de060 msgr2=0x7f2d7c0c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d7bfff700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c1de060 0x7f2d7c0c4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d7bfff700 1 -- 172.21.15.7:0/1280270947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c1550b0 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d7bfff700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c1ddc90 0x7f2d7c0c6be0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c001930 tx=0x7f2d6c00f4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:06.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d78ff9700 1 -- 172.21.15.7:0/1280270947 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c007e00 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d78ff9700 1 -- 172.21.15.7:0/1280270947 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c00fa60 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d8271d700 1 -- 172.21.15.7:0/1280270947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c154dd0 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d78ff9700 1 -- 172.21.15.7:0/1280270947 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c00fe10 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.396+0000 7f2d8271d700 1 -- 172.21.15.7:0/1280270947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c0ab490 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.397+0000 7f2d8271d700 1 -- 172.21.15.7:0/1280270947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c074bb0 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.398+0000 7f2d78ff9700 1 -- 172.21.15.7:0/1280270947 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2d6c01d460 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.398+0000 7f2d78ff9700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d5c0682f0 0x7f2d5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.398+0000 7f2d78ff9700 1 -- 172.21.15.7:0/1280270947 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2d6c086170 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.398+0000 7f2d7b7fe700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d5c0682f0 0x7f2d5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.399+0000 7f2d7b7fe700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d5c0682f0 0x7f2d5c06a7a0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c1579a0 tx=0x7f2d64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:06.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.403+0000 7f2d78ff9700 1 -- 172.21.15.7:0/1280270947 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2d6c0524b0 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.638+0000 7f2d8271d700 1 -- 172.21.15.7:0/1280270947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d7c0c1c10 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.638+0000 7f2d78ff9700 1 -- 172.21.15.7:0/1280270947 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2d7c0c1c10 con 0x7f2d7c1ddc90 2023-03-03T14:56:06.641 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:56:06.641 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:56:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.640+0000 7f2d8271d700 1 -- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d5c0682f0 msgr2=0x7f2d5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.640+0000 7f2d8271d700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d5c0682f0 0x7f2d5c06a7a0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c1579a0 tx=0x7f2d64008040 comp rx=0 tx=0).stop 2023-03-03T14:56:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.640+0000 7f2d8271d700 1 -- 172.21.15.7:0/1280270947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c1ddc90 msgr2=0x7f2d7c0c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.640+0000 7f2d8271d700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c1ddc90 0x7f2d7c0c6be0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c001930 tx=0x7f2d6c00f4b0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.640+0000 7f2d8271d700 1 -- 172.21.15.7:0/1280270947 shutdown_connections 2023-03-03T14:56:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.640+0000 7f2d8271d700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2d7c0723f0 0x7f2d7c06b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.640+0000 7f2d8271d700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2d5c0682f0 0x7f2d5c06a7a0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.640+0000 7f2d8271d700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d7c1ddc90 0x7f2d7c0c6be0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.640+0000 7f2d8271d700 1 --2- 172.21.15.7:0/1280270947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d7c1de060 0x7f2d7c0c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:06.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.640+0000 7f2d8271d700 1 -- 172.21.15.7:0/1280270947 >> 172.21.15.7:0/1280270947 conn(0x7f2d7c0a9430 msgr2=0x7f2d7c0c1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:56:06.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.641+0000 7f2d8271d700 1 -- 172.21.15.7:0/1280270947 shutdown_connections 2023-03-03T14:56:06.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:06.641+0000 7f2d8271d700 1 -- 172.21.15.7:0/1280270947 wait complete. 2023-03-03T14:56:06.646 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:56:06.655 INFO:tasks.mds_thrash.fs.[a]:waiting for 88 secs before thrashing 2023-03-03T14:56:18.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:56:18.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.178+0000 7f2bb9882700 1 Processor -- start 2023-03-03T14:56:18.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.178+0000 7f2bb9882700 1 -- start start 2023-03-03T14:56:18.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.179+0000 7f2bb9882700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2bb4151e80 0x7f2bb40c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:18.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.179+0000 7f2bb9882700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb40c6fe0 0x7f2bb40bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:18.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.179+0000 7f2bb9882700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb40c08d0 con 0x7f2bb4155410 2023-03-03T14:56:18.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.179+0000 7f2bb9882700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb4152250 con 0x7f2bb4151e80 2023-03-03T14:56:18.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.179+0000 7f2bb9882700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb40c0fb0 con 0x7f2bb40c6fe0 2023-03-03T14:56:18.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.179+0000 7f2bb2ffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2bb4151e80 0x7f2bb40c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:18.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.179+0000 7f2bb37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb40c6fe0 0x7f2bb40bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:18.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb40c6fe0 0x7f2bb40bfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59020/0 (socket says 172.21.15.7:59020) 2023-03-03T14:56:18.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb37fe700 1 -- 172.21.15.7:0/4064959017 learned_addr learned my addr 172.21.15.7:0/4064959017 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:56:18.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb27fc700 1 --1- 172.21.15.7:0/4064959017 >> v1:172.21.15.143:6789/0 conn(0x7f2bb4155410 0x7f2bb40bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:48444/0 (socket says 172.21.15.7:48444) 2023-03-03T14:56:18.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb37fe700 1 -- 172.21.15.7:0/4064959017 >> v2:172.21.15.143:3300/0 conn(0x7f2bb4151e80 msgr2=0x7f2bb40c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:56:18.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb37fe700 1 --2- 172.21.15.7:0/4064959017 >> v2:172.21.15.143:3300/0 conn(0x7f2bb4151e80 0x7f2bb40c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb37fe700 1 -- 172.21.15.7:0/4064959017 >> v1:172.21.15.143:6789/0 conn(0x7f2bb4155410 legacy=0x7f2bb40bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb37fe700 1 -- 172.21.15.7:0/4064959017 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb4154930 con 0x7f2bb40c6fe0 2023-03-03T14:56:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb2ffd700 1 --2- 172.21.15.7:0/4064959017 >> v2:172.21.15.143:3300/0 conn(0x7f2bb4151e80 0x7f2bb40c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:56:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb37fe700 1 --2- 172.21.15.7:0/4064959017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb40c6fe0 0x7f2bb40bfff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2bac009e50 tx=0x7f2bac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c8bb1a03c48e604 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb1ffb700 1 -- 172.21.15.7:0/4064959017 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac00e070 con 0x7f2bb40c6fe0 2023-03-03T14:56:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb1ffb700 1 -- 172.21.15.7:0/4064959017 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bac00b910 con 0x7f2bb40c6fe0 2023-03-03T14:56:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.180+0000 7f2bb1ffb700 1 -- 172.21.15.7:0/4064959017 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac00bcc0 con 0x7f2bb40c6fe0 2023-03-03T14:56:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.181+0000 7f2bb9882700 1 -- 172.21.15.7:0/4064959017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb40c6fe0 msgr2=0x7f2bb40bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.181+0000 7f2bb9882700 1 --2- 172.21.15.7:0/4064959017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb40c6fe0 0x7f2bb40bfff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2bac009e50 tx=0x7f2bac00b330 comp rx=0 tx=0).stop 2023-03-03T14:56:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.181+0000 7f2bb9882700 1 -- 172.21.15.7:0/4064959017 shutdown_connections 2023-03-03T14:56:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.181+0000 7f2bb9882700 1 --2- 172.21.15.7:0/4064959017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb40c6fe0 0x7f2bb40bfff0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.181+0000 7f2bb9882700 1 --2- 172.21.15.7:0/4064959017 >> v2:172.21.15.143:3300/0 conn(0x7f2bb4151e80 0x7f2bb40c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.181+0000 7f2bb9882700 1 -- 172.21.15.7:0/4064959017 >> 172.21.15.7:0/4064959017 conn(0x7f2bb4071630 msgr2=0x7f2bb406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:56:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.181+0000 7f2bb9882700 1 -- 172.21.15.7:0/4064959017 shutdown_connections 2023-03-03T14:56:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.181+0000 7f2bb9882700 1 -- 172.21.15.7:0/4064959017 wait complete. 2023-03-03T14:56:18.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.182+0000 7f2bb9882700 1 Processor -- start 2023-03-03T14:56:18.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.182+0000 7f2bb9882700 1 -- start start 2023-03-03T14:56:18.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb9882700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb40c6fe0 0x7f2bb40c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:18.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb9882700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bb4151e80 0x7f2bb414e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:18.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb9882700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb4155410 0x7f2bb40bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:18.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb9882700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb414f5b0 con 0x7f2bb4151e80 2023-03-03T14:56:18.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb9882700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb414f890 con 0x7f2bb40c6fe0 2023-03-03T14:56:18.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb9882700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb414dd70 con 0x7f2bb4155410 2023-03-03T14:56:18.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bb4151e80 0x7f2bb414e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:18.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bb4151e80 0x7f2bb414e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50952/0 (socket says 172.21.15.7:50952) 2023-03-03T14:56:18.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb27fc700 1 -- 172.21.15.7:0/989245430 learned_addr learned my addr 172.21.15.7:0/989245430 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:56:18.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb37fe700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb4155410 0x7f2bb40bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:18.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb2ffd700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb40c6fe0 0x7f2bb40c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:18.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb27fc700 1 -- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb4155410 msgr2=0x7f2bb40bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:18.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb27fc700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb4155410 0x7f2bb40bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:18.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb27fc700 1 -- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb40c6fe0 msgr2=0x7f2bb40c8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:18.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb27fc700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb40c6fe0 0x7f2bb40c8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb27fc700 1 -- 172.21.15.7:0/989245430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c0094a0 con 0x7f2bb4151e80 2023-03-03T14:56:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.183+0000 7f2bb2ffd700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb40c6fe0 0x7f2bb40c8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:56:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.184+0000 7f2bb27fc700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bb4151e80 0x7f2bb414e850 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400c500 tx=0x7f2ba400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.184+0000 7f2babfff700 1 -- 172.21.15.7:0/989245430 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4005cb0 con 0x7f2bb4151e80 2023-03-03T14:56:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.184+0000 7f2babfff700 1 -- 172.21.15.7:0/989245430 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4005e10 con 0x7f2bb4151e80 2023-03-03T14:56:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.184+0000 7f2babfff700 1 -- 172.21.15.7:0/989245430 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba40072f0 con 0x7f2bb4151e80 2023-03-03T14:56:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.184+0000 7f2bb9882700 1 -- 172.21.15.7:0/989245430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bac009b30 con 0x7f2bb4151e80 2023-03-03T14:56:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.184+0000 7f2bb9882700 1 -- 172.21.15.7:0/989245430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb4154ef0 con 0x7f2bb4151e80 2023-03-03T14:56:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.185+0000 7f2bb9882700 1 -- 172.21.15.7:0/989245430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb4154c10 con 0x7f2bb4151e80 2023-03-03T14:56:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.185+0000 7f2babfff700 1 -- 172.21.15.7:0/989245430 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ba4019030 con 0x7f2bb4151e80 2023-03-03T14:56:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.186+0000 7f2babfff700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b940682f0 0x7f2b9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.186+0000 7f2babfff700 1 -- 172.21.15.7:0/989245430 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2ba4085430 con 0x7f2bb4151e80 2023-03-03T14:56:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.186+0000 7f2bb2ffd700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b940682f0 0x7f2b9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.187+0000 7f2bb2ffd700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b940682f0 0x7f2b9406a7a0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009d20 tx=0x7f2b9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:18.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.190+0000 7f2babfff700 1 -- 172.21.15.7:0/989245430 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ba40517d0 con 0x7f2bb4151e80 2023-03-03T14:56:18.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.427+0000 7f2bb9882700 1 -- 172.21.15.7:0/989245430 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bb40c5220 con 0x7f2bb4151e80 2023-03-03T14:56:18.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.428+0000 7f2babfff700 1 -- 172.21.15.7:0/989245430 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2ba4054df0 con 0x7f2bb4151e80 2023-03-03T14:56:18.430 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:56:18.430 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:56:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 -- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b940682f0 msgr2=0x7f2b9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b940682f0 0x7f2b9406a7a0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c009d20 tx=0x7f2b9c008040 comp rx=0 tx=0).stop 2023-03-03T14:56:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 -- 172.21.15.7:0/989245430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bb4151e80 msgr2=0x7f2bb414e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bb4151e80 0x7f2bb414e850 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400c500 tx=0x7f2ba400c810 comp rx=0 tx=0).stop 2023-03-03T14:56:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 -- 172.21.15.7:0/989245430 shutdown_connections 2023-03-03T14:56:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bb4155410 0x7f2bb40bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb40c6fe0 0x7f2bb40c8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b940682f0 0x7f2b9406a7a0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 --2- 172.21.15.7:0/989245430 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bb4151e80 0x7f2bb414e850 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 -- 172.21.15.7:0/989245430 >> 172.21.15.7:0/989245430 conn(0x7f2bb4071630 msgr2=0x7f2bb41ddd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:56:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 -- 172.21.15.7:0/989245430 shutdown_connections 2023-03-03T14:56:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:18.430+0000 7f2bb9882700 1 -- 172.21.15.7:0/989245430 wait complete. 2023-03-03T14:56:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:56:18.444 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before thrashing 2023-03-03T14:56:27.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:56:27.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.582+0000 7f8e9c9ef700 1 Processor -- start 2023-03-03T14:56:27.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.582+0000 7f8e9c9ef700 1 -- start start 2023-03-03T14:56:27.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.583+0000 7f8e9c9ef700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e980c1da0 0x7f8e981550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.583+0000 7f8e9c9ef700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8e980c3e30 0x7f8e980c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.583+0000 7f8e9c9ef700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e980c7a60 con 0x7f8e9806d870 2023-03-03T14:56:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.583+0000 7f8e9c9ef700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e980c2af0 con 0x7f8e980c1da0 2023-03-03T14:56:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.583+0000 7f8e9c9ef700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e980c3620 con 0x7f8e980c3e30 2023-03-03T14:56:27.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.583+0000 7f8e95d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e980c1da0 0x7f8e981550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e95d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e980c1da0 0x7f8e981550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50970/0 (socket says 172.21.15.7:50970) 2023-03-03T14:56:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e95d9b700 1 -- 172.21.15.7:0/4290149441 learned_addr learned my addr 172.21.15.7:0/4290149441 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:56:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e96d9d700 1 --2- 172.21.15.7:0/4290149441 >> v2:172.21.15.143:3300/0 conn(0x7f8e980c3e30 0x7f8e980c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e95d9b700 1 -- 172.21.15.7:0/4290149441 >> v2:172.21.15.143:3300/0 conn(0x7f8e980c3e30 msgr2=0x7f8e980c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:56:27.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e95d9b700 1 --2- 172.21.15.7:0/4290149441 >> v2:172.21.15.143:3300/0 conn(0x7f8e980c3e30 0x7f8e980c4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:27.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e95d9b700 1 -- 172.21.15.7:0/4290149441 >> v1:172.21.15.143:6789/0 conn(0x7f8e9806d870 legacy=0x7f8e980c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:56:27.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e95d9b700 1 -- 172.21.15.7:0/4290149441 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e981545e0 con 0x7f8e980c1da0 2023-03-03T14:56:27.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e95d9b700 1 --2- 172.21.15.7:0/4290149441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c1da0 0x7f8e981550c0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f8e98073830 tx=0x7f8e80009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141d3ec147917eb8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:27.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e9559a700 1 -- 172.21.15.7:0/4290149441 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e80004230 con 0x7f8e980c1da0 2023-03-03T14:56:27.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e9559a700 1 -- 172.21.15.7:0/4290149441 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e80004390 con 0x7f8e980c1da0 2023-03-03T14:56:27.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.584+0000 7f8e9559a700 1 -- 172.21.15.7:0/4290149441 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e800036a0 con 0x7f8e980c1da0 2023-03-03T14:56:27.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.585+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/4290149441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c1da0 msgr2=0x7f8e981550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.585+0000 7f8e9c9ef700 1 --2- 172.21.15.7:0/4290149441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c1da0 0x7f8e981550c0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f8e98073830 tx=0x7f8e80009bd0 comp rx=0 tx=0).stop 2023-03-03T14:56:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.585+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/4290149441 shutdown_connections 2023-03-03T14:56:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.585+0000 7f8e9c9ef700 1 --2- 172.21.15.7:0/4290149441 >> v2:172.21.15.143:3300/0 conn(0x7f8e980c3e30 0x7f8e980c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:27.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.585+0000 7f8e9c9ef700 1 --2- 172.21.15.7:0/4290149441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c1da0 0x7f8e981550c0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.585+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/4290149441 >> 172.21.15.7:0/4290149441 conn(0x7f8e9806c110 msgr2=0x7f8e9806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:56:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.585+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/4290149441 shutdown_connections 2023-03-03T14:56:27.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.585+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/4290149441 wait complete. 2023-03-03T14:56:27.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.586+0000 7f8e9c9ef700 1 Processor -- start 2023-03-03T14:56:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.586+0000 7f8e9c9ef700 1 -- start start 2023-03-03T14:56:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e9c9ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e9806d870 0x7f8e981499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e9c9ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e980c1da0 0x7f8e98147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e9c9ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c3e30 0x7f8e981e9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e9c9ef700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e98157a60 con 0x7f8e980c3e30 2023-03-03T14:56:27.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e9c9ef700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e981e95c0 con 0x7f8e980c1da0 2023-03-03T14:56:27.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e9c9ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e980ac860 con 0x7f8e9806d870 2023-03-03T14:56:27.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e96d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c3e30 0x7f8e981e9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:27.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e96d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c3e30 0x7f8e981e9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50976/0 (socket says 172.21.15.7:50976) 2023-03-03T14:56:27.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e96d9d700 1 -- 172.21.15.7:0/2786503620 learned_addr learned my addr 172.21.15.7:0/2786503620 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:56:27.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e95d9b700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e980c1da0 0x7f8e98147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:27.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e9659c700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e9806d870 0x7f8e981499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:27.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e96d9d700 1 -- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e9806d870 msgr2=0x7f8e981499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:27.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e96d9d700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e9806d870 0x7f8e981499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:27.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e96d9d700 1 -- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e980c1da0 msgr2=0x7f8e98147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:27.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e96d9d700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e980c1da0 0x7f8e98147510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:27.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e96d9d700 1 -- 172.21.15.7:0/2786503620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e80009550 con 0x7f8e980c3e30 2023-03-03T14:56:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.587+0000 7f8e96d9d700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c3e30 0x7f8e981e9cf0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000cc60 tx=0x7f8e9000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.588+0000 7f8e8effd700 1 -- 172.21.15.7:0/2786503620 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90003a20 con 0x7f8e980c3e30 2023-03-03T14:56:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.588+0000 7f8e8effd700 1 -- 172.21.15.7:0/2786503620 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e90003b80 con 0x7f8e980c3e30 2023-03-03T14:56:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.588+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/2786503620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e98154ba0 con 0x7f8e980c3e30 2023-03-03T14:56:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.588+0000 7f8e8effd700 1 -- 172.21.15.7:0/2786503620 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9000aa10 con 0x7f8e980c3e30 2023-03-03T14:56:27.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.588+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/2786503620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e981548c0 con 0x7f8e980c3e30 2023-03-03T14:56:27.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.589+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/2786503620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9806dc40 con 0x7f8e980c3e30 2023-03-03T14:56:27.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.589+0000 7f8e8effd700 1 -- 172.21.15.7:0/2786503620 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e9001f070 con 0x7f8e980c3e30 2023-03-03T14:56:27.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.589+0000 7f8e8effd700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 0x7f8e7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:27.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.590+0000 7f8e8effd700 1 -- 172.21.15.7:0/2786503620 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8e90086050 con 0x7f8e980c3e30 2023-03-03T14:56:27.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.590+0000 7f8e9659c700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 0x7f8e7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:27.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.590+0000 7f8e9659c700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 0x7f8e7806a7a0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f8e88007fd0 tx=0x7f8e88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:27.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.594+0000 7f8e8effd700 1 -- 172.21.15.7:0/2786503620 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e900523f0 con 0x7f8e980c3e30 2023-03-03T14:56:27.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.835+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/2786503620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e98074bb0 con 0x7f8e980c3e30 2023-03-03T14:56:27.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.836+0000 7f8e8effd700 1 -- 172.21.15.7:0/2786503620 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8e9001d070 con 0x7f8e980c3e30 2023-03-03T14:56:27.838 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:56:27.838 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:56:27.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 msgr2=0x7f8e7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:27.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 0x7f8e7806a7a0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f8e88007fd0 tx=0x7f8e88009040 comp rx=0 tx=0).stop 2023-03-03T14:56:27.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/2786503620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c3e30 msgr2=0x7f8e981e9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:27.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c3e30 0x7f8e981e9cf0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f8e9000cc60 tx=0x7f8e9000cf70 comp rx=0 tx=0).stop 2023-03-03T14:56:27.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/2786503620 shutdown_connections 2023-03-03T14:56:27.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e780682f0 0x7f8e7806a7a0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:27.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e980c3e30 0x7f8e981e9cf0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:27.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e9806d870 0x7f8e981499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:27.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 --2- 172.21.15.7:0/2786503620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e980c1da0 0x7f8e98147510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:27.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/2786503620 >> 172.21.15.7:0/2786503620 conn(0x7f8e9806c110 msgr2=0x7f8e9814ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:56:27.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/2786503620 shutdown_connections 2023-03-03T14:56:27.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:27.838+0000 7f8e9c9ef700 1 -- 172.21.15.7:0/2786503620 wait complete. 2023-03-03T14:56:27.842 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:56:27.852 INFO:tasks.mds_thrash.fs.[b]:waiting for 16 secs before thrashing 2023-03-03T14:56:28.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:28.563 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:43.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:56:43.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.990+0000 7f87fd591700 1 Processor -- start 2023-03-03T14:56:43.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.990+0000 7f87fd591700 1 -- start start 2023-03-03T14:56:43.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.991+0000 7f87fd591700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f87f814de50 0x7f87f806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:43.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.991+0000 7f87fd591700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f80a65b0 con 0x7f87f80c65c0 2023-03-03T14:56:43.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.991+0000 7f87fd591700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f81541f0 con 0x7f87f806a7f0 2023-03-03T14:56:43.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.991+0000 7f87fd591700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f8154e00 con 0x7f87f814de50 2023-03-03T14:56:43.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.991+0000 7f87f6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f87f814de50 0x7f87f806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:43.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.991+0000 7f87f67fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f87f80c65c0 0x7f87f80c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44970/0 (socket says 172.21.15.7:44970) 2023-03-03T14:56:43.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.991+0000 7f87f67fc700 1 -- 172.21.15.7:0/2037015780 learned_addr learned my addr 172.21.15.7:0/2037015780 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:56:43.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.992+0000 7f87f6ffd700 1 -- 172.21.15.7:0/2037015780 >> v1:172.21.15.143:6789/0 conn(0x7f87f806a7f0 legacy=0x7f87f80c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:56:43.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.992+0000 7f87f6ffd700 1 -- 172.21.15.7:0/2037015780 >> v1:172.21.15.7:6789/0 conn(0x7f87f80c65c0 legacy=0x7f87f80c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.992+0000 7f87f5ffb700 1 -- 172.21.15.7:0/2037015780 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388018573 0 0) 0x7f87f80a65b0 con 0x7f87f80c65c0 2023-03-03T14:56:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.992+0000 7f87f6ffd700 1 -- 172.21.15.7:0/2037015780 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f81544d0 con 0x7f87f814de50 2023-03-03T14:56:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.992+0000 7f87f6ffd700 1 --2- 172.21.15.7:0/2037015780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f814de50 0x7f87f806a2b0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009840 tx=0x7f87ec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8925142e9e0df929 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.992+0000 7f87f5ffb700 1 -- 172.21.15.7:0/2037015780 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec004230 con 0x7f87f814de50 2023-03-03T14:56:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.992+0000 7f87f5ffb700 1 -- 172.21.15.7:0/2037015780 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec004390 con 0x7f87f814de50 2023-03-03T14:56:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.992+0000 7f87f5ffb700 1 -- 172.21.15.7:0/2037015780 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0036a0 con 0x7f87f814de50 2023-03-03T14:56:43.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.993+0000 7f87fd591700 1 -- 172.21.15.7:0/2037015780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f814de50 msgr2=0x7f87f806a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:43.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.993+0000 7f87fd591700 1 --2- 172.21.15.7:0/2037015780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f814de50 0x7f87f806a2b0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009840 tx=0x7f87ec009b50 comp rx=0 tx=0).stop 2023-03-03T14:56:43.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.993+0000 7f87fd591700 1 -- 172.21.15.7:0/2037015780 shutdown_connections 2023-03-03T14:56:43.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.993+0000 7f87fd591700 1 --2- 172.21.15.7:0/2037015780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f814de50 0x7f87f806a2b0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:43.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.993+0000 7f87fd591700 1 -- 172.21.15.7:0/2037015780 >> 172.21.15.7:0/2037015780 conn(0x7f87f80a9430 msgr2=0x7f87f80c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:56:43.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.993+0000 7f87fd591700 1 -- 172.21.15.7:0/2037015780 shutdown_connections 2023-03-03T14:56:43.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.993+0000 7f87fd591700 1 -- 172.21.15.7:0/2037015780 wait complete. 2023-03-03T14:56:43.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.994+0000 7f87fd591700 1 Processor -- start 2023-03-03T14:56:43.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.994+0000 7f87fd591700 1 -- start start 2023-03-03T14:56:43.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87fd591700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f806a7f0 0x7f87f8071e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:44.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87fd591700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87f80c65c0 0x7f87f8070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:44.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87fd591700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87f814de50 0x7f87f8141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:44.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87fd591700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f80aac30 con 0x7f87f806a7f0 2023-03-03T14:56:44.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87fd591700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f80aa9e0 con 0x7f87f80c65c0 2023-03-03T14:56:44.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87fd591700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f813ce90 con 0x7f87f814de50 2023-03-03T14:56:44.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f806a7f0 0x7f87f8071e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:44.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f806a7f0 0x7f87f8071e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50994/0 (socket says 172.21.15.7:50994) 2023-03-03T14:56:44.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f6ffd700 1 -- 172.21.15.7:0/4179485098 learned_addr learned my addr 172.21.15.7:0/4179485098 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:56:44.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f67fc700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87f80c65c0 0x7f87f8070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:44.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f77fe700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87f814de50 0x7f87f8141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:44.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f6ffd700 1 -- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87f814de50 msgr2=0x7f87f8141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:44.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f6ffd700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87f814de50 0x7f87f8141f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:44.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f6ffd700 1 -- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87f80c65c0 msgr2=0x7f87f8070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:44.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f6ffd700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87f80c65c0 0x7f87f8070ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:44.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f6ffd700 1 -- 172.21.15.7:0/4179485098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec0094a0 con 0x7f87f806a7f0 2023-03-03T14:56:44.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f77fe700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87f814de50 0x7f87f8141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T14:56:44.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.995+0000 7f87f6ffd700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f806a7f0 0x7f87f8071e80 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009b90 tx=0x7f87ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:44.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.996+0000 7f87df7fe700 1 -- 172.21.15.7:0/4179485098 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec011630 con 0x7f87f806a7f0 2023-03-03T14:56:44.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.996+0000 7f87df7fe700 1 -- 172.21.15.7:0/4179485098 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec01e070 con 0x7f87f806a7f0 2023-03-03T14:56:44.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.996+0000 7f87fd591700 1 -- 172.21.15.7:0/4179485098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f8154a90 con 0x7f87f806a7f0 2023-03-03T14:56:44.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.996+0000 7f87df7fe700 1 -- 172.21.15.7:0/4179485098 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0119b0 con 0x7f87f806a7f0 2023-03-03T14:56:44.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.996+0000 7f87fd591700 1 -- 172.21.15.7:0/4179485098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f81547b0 con 0x7f87f806a7f0 2023-03-03T14:56:44.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.997+0000 7f87fd591700 1 -- 172.21.15.7:0/4179485098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f8153e80 con 0x7f87f806a7f0 2023-03-03T14:56:44.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.997+0000 7f87df7fe700 1 -- 172.21.15.7:0/4179485098 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f87ec017020 con 0x7f87f806a7f0 2023-03-03T14:56:44.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.997+0000 7f87df7fe700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87e40682f0 0x7f87e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:56:44.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.998+0000 7f87df7fe700 1 -- 172.21.15.7:0/4179485098 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f87ec0874c0 con 0x7f87f806a7f0 2023-03-03T14:56:44.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.998+0000 7f87f67fc700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87e40682f0 0x7f87e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:56:44.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:43.998+0000 7f87f67fc700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87e40682f0 0x7f87e406a7a0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f87f806feb0 tx=0x7f87e8007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:56:44.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.002+0000 7f87df7fe700 1 -- 172.21.15.7:0/4179485098 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f87ec053860 con 0x7f87f806a7f0 2023-03-03T14:56:44.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.239+0000 7f87fd591700 1 -- 172.21.15.7:0/4179485098 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87f8074bb0 con 0x7f87f806a7f0 2023-03-03T14:56:44.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.240+0000 7f87df7fe700 1 -- 172.21.15.7:0/4179485098 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f87ec056e80 con 0x7f87f806a7f0 2023-03-03T14:56:44.246 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:56:44.246 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:56:44.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 -- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87e40682f0 msgr2=0x7f87e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:44.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87e40682f0 0x7f87e406a7a0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f87f806feb0 tx=0x7f87e8007be0 comp rx=0 tx=0).stop 2023-03-03T14:56:44.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 -- 172.21.15.7:0/4179485098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f806a7f0 msgr2=0x7f87f8071e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:56:44.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f806a7f0 0x7f87f8071e80 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009b90 tx=0x7f87ec004750 comp rx=0 tx=0).stop 2023-03-03T14:56:44.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 -- 172.21.15.7:0/4179485098 shutdown_connections 2023-03-03T14:56:44.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87f814de50 0x7f87f8141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:44.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87e40682f0 0x7f87e406a7a0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:44.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f87f806a7f0 0x7f87f8071e80 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:44.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 --2- 172.21.15.7:0/4179485098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87f80c65c0 0x7f87f8070ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:56:44.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 -- 172.21.15.7:0/4179485098 >> 172.21.15.7:0/4179485098 conn(0x7f87f80a9430 msgr2=0x7f87f8151c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:56:44.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 -- 172.21.15.7:0/4179485098 shutdown_connections 2023-03-03T14:56:44.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:56:44.242+0000 7f87fd591700 1 -- 172.21.15.7:0/4179485098 wait complete. 2023-03-03T14:56:44.249 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:56:44.256 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T14:56:58.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:56:58.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:28.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:28.691 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:34.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:57:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.794+0000 7f38d4c35700 1 Processor -- start 2023-03-03T14:57:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.794+0000 7f38d4c35700 1 -- start start 2023-03-03T14:57:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.795+0000 7f38d4c35700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00c2930 0x7f38d0157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.795+0000 7f38d4c35700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d006d430 con 0x7f38d006c480 2023-03-03T14:57:34.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.795+0000 7f38d4c35700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d01586e0 con 0x7f38d0151d40 2023-03-03T14:57:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.795+0000 7f38d4c35700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d006c2a0 con 0x7f38d00c2930 2023-03-03T14:57:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.795+0000 7f38ced9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f38d006c480 0x7f38d0157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45012/0 (socket says 172.21.15.7:45012) 2023-03-03T14:57:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.795+0000 7f38ced9d700 1 -- 172.21.15.7:0/1224226401 learned_addr learned my addr 172.21.15.7:0/1224226401 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:57:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.795+0000 7f38cdd9b700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00c2930 0x7f38d0157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.795+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708751868 0 0) 0x7f38d006d430 con 0x7f38d006c480 2023-03-03T14:57:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.795+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b80035b0 con 0x7f38d006c480 2023-03-03T14:57:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161692960 0 0) 0x7f38d01586e0 con 0x7f38d0151d40 2023-03-03T14:57:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d006d430 con 0x7f38d0151d40 2023-03-03T14:57:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 640575328 0 0) 0x7f38b80035b0 con 0x7f38d006c480 2023-03-03T14:57:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00c2930 msgr2=0x7f38d0157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:57:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00c2930 0x7f38d0157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 >> v1:172.21.15.143:6789/0 conn(0x7f38d0151d40 legacy=0x7f38d00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d006cb00 con 0x7f38d006c480 2023-03-03T14:57:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f38c80019f0 con 0x7f38d006c480 2023-03-03T14:57:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cdd9b700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00c2930 0x7f38d0157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T14:57:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 >> v1:172.21.15.7:6789/0 conn(0x7f38d006c480 legacy=0x7f38d0157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b8003c30 0x7f38b8004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 0x7f38b8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00c2930 0x7f38b80052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d006c2a0 con 0x7f38b8004850 2023-03-03T14:57:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b800bbd0 con 0x7f38b8003c30 2023-03-03T14:57:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b800bd90 con 0x7f38d00c2930 2023-03-03T14:57:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cdd9b700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 0x7f38b8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38ce59c700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b8003c30 0x7f38b8004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38ced9d700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00c2930 0x7f38b80052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cdd9b700 1 -- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00c2930 msgr2=0x7f38b80052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:57:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cdd9b700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00c2930 0x7f38b80052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cdd9b700 1 -- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b8003c30 msgr2=0x7f38b8004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:57:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cdd9b700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38b8003c30 0x7f38b8004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cdd9b700 1 -- 172.21.15.7:0/1224226401 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c00094a0 con 0x7f38b8004850 2023-03-03T14:57:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.796+0000 7f38cdd9b700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 0x7f38b8004cc0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009d30 tx=0x7f38b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6c4efb61bbe5dc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:57:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.797+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0009e80 con 0x7f38b8004850 2023-03-03T14:57:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.797+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c000e9a0 con 0x7f38b8004850 2023-03-03T14:57:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.797+0000 7f38cd59a700 1 -- 172.21.15.7:0/1224226401 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c000ed10 con 0x7f38b8004850 2023-03-03T14:57:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.797+0000 7f38d4c35700 1 -- 172.21.15.7:0/1224226401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 msgr2=0x7f38b8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.797+0000 7f38d4c35700 1 --2- 172.21.15.7:0/1224226401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 0x7f38b8004cc0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009d30 tx=0x7f38b8005880 comp rx=0 tx=0).stop 2023-03-03T14:57:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.797+0000 7f38ced9d700 1 -- 172.21.15.7:0/1224226401 reap_dead start 2023-03-03T14:57:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.797+0000 7f38d4c35700 1 -- 172.21.15.7:0/1224226401 shutdown_connections 2023-03-03T14:57:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.797+0000 7f38d4c35700 1 -- 172.21.15.7:0/1224226401 >> 172.21.15.7:0/1224226401 conn(0x7f38d00a94d0 msgr2=0x7f38d00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:57:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.797+0000 7f38d4c35700 1 -- 172.21.15.7:0/1224226401 shutdown_connections 2023-03-03T14:57:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.797+0000 7f38d4c35700 1 -- 172.21.15.7:0/1224226401 wait complete. 2023-03-03T14:57:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.799+0000 7f38d4c35700 1 Processor -- start 2023-03-03T14:57:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.799+0000 7f38d4c35700 1 -- start start 2023-03-03T14:57:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.799+0000 7f38d4c35700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 0x7f38d00acc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.799+0000 7f38d4c35700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38d00c08f0 0x7f38d0156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.799+0000 7f38d4c35700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00aa1d0 0x7f38d00aa640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.799+0000 7f38d4c35700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d00c1410 con 0x7f38b8004850 2023-03-03T14:57:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.799+0000 7f38d4c35700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d01ed430 con 0x7f38d00c08f0 2023-03-03T14:57:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.799+0000 7f38d4c35700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d00ab3d0 con 0x7f38d00aa1d0 2023-03-03T14:57:34.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38ce59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 0x7f38d00acc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:34.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38ce59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 0x7f38d00acc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51040/0 (socket says 172.21.15.7:51040) 2023-03-03T14:57:34.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38ce59c700 1 -- 172.21.15.7:0/1408878779 learned_addr learned my addr 172.21.15.7:0/1408878779 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:57:34.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38cdd9b700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38d00c08f0 0x7f38d0156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:34.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38ced9d700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00aa1d0 0x7f38d00aa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38ce59c700 1 -- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00aa1d0 msgr2=0x7f38d00aa640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38ce59c700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00aa1d0 0x7f38d00aa640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38ce59c700 1 -- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38d00c08f0 msgr2=0x7f38d0156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38ce59c700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38d00c08f0 0x7f38d0156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38ce59c700 1 -- 172.21.15.7:0/1408878779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d006d0c0 con 0x7f38b8004850 2023-03-03T14:57:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38ce59c700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 0x7f38d00acc10 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f38c400f7a0 tx=0x7f38c400fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:57:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38beffd700 1 -- 172.21.15.7:0/1408878779 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c40052e0 con 0x7f38b8004850 2023-03-03T14:57:34.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38beffd700 1 -- 172.21.15.7:0/1408878779 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c4005440 con 0x7f38b8004850 2023-03-03T14:57:34.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38d4c35700 1 -- 172.21.15.7:0/1408878779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d006cde0 con 0x7f38b8004850 2023-03-03T14:57:34.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38beffd700 1 -- 172.21.15.7:0/1408878779 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c40038c0 con 0x7f38b8004850 2023-03-03T14:57:34.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.800+0000 7f38d4c35700 1 -- 172.21.15.7:0/1408878779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d0073e00 con 0x7f38b8004850 2023-03-03T14:57:34.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.801+0000 7f38d4c35700 1 -- 172.21.15.7:0/1408878779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d0074bb0 con 0x7f38b8004850 2023-03-03T14:57:34.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.802+0000 7f38beffd700 1 -- 172.21.15.7:0/1408878779 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f38c4003a20 con 0x7f38b8004850 2023-03-03T14:57:34.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.802+0000 7f38beffd700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38b00682f0 0x7f38b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:57:34.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.802+0000 7f38beffd700 1 -- 172.21.15.7:0/1408878779 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f38c4086120 con 0x7f38b8004850 2023-03-03T14:57:34.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.802+0000 7f38cdd9b700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38b00682f0 0x7f38b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:57:34.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.803+0000 7f38cdd9b700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38b00682f0 0x7f38b006a7a0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009c50 tx=0x7f38c0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:57:34.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:34.807+0000 7f38beffd700 1 -- 172.21.15.7:0/1408878779 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f38c40524c0 con 0x7f38b8004850 2023-03-03T14:57:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.045+0000 7f38d4c35700 1 -- 172.21.15.7:0/1408878779 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38d00a9fb0 con 0x7f38b8004850 2023-03-03T14:57:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.045+0000 7f38beffd700 1 -- 172.21.15.7:0/1408878779 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f38c4055ae0 con 0x7f38b8004850 2023-03-03T14:57:35.048 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:57:35.048 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:57:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 -- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38b00682f0 msgr2=0x7f38b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38b00682f0 0x7f38b006a7a0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009c50 tx=0x7f38c0008030 comp rx=0 tx=0).stop 2023-03-03T14:57:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 -- 172.21.15.7:0/1408878779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 msgr2=0x7f38d00acc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:57:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 0x7f38d00acc10 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f38c400f7a0 tx=0x7f38c400fb60 comp rx=0 tx=0).stop 2023-03-03T14:57:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 -- 172.21.15.7:0/1408878779 shutdown_connections 2023-03-03T14:57:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38d00aa1d0 0x7f38d00aa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f38b00682f0 0x7f38b006a7a0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38b8004850 0x7f38d00acc10 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 --2- 172.21.15.7:0/1408878779 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38d00c08f0 0x7f38d0156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:57:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 -- 172.21.15.7:0/1408878779 >> 172.21.15.7:0/1408878779 conn(0x7f38d00a94d0 msgr2=0x7f38d0070aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:57:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 -- 172.21.15.7:0/1408878779 shutdown_connections 2023-03-03T14:57:35.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:57:35.047+0000 7f38d4c35700 1 -- 172.21.15.7:0/1408878779 wait complete. 2023-03-03T14:57:35.054 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:57:35.062 INFO:tasks.mds_thrash.fs.[a]:waiting for 73 secs before thrashing 2023-03-03T14:57:58.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:57:58.761 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:28.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:28.835 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:40.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:58:40.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.396+0000 7faa26899700 1 Processor -- start 2023-03-03T14:58:40.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.396+0000 7faa26899700 1 -- start start 2023-03-03T14:58:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa26899700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faa20154f10 0x7faa20158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa26899700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa200aa010 0x7faa20158540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:40.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa26899700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa20151ef0 con 0x7faa20151b20 2023-03-03T14:58:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa26899700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa20157b20 con 0x7faa200aa010 2023-03-03T14:58:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa26899700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa20155410 con 0x7faa20154f10 2023-03-03T14:58:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa24e36700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa200aa010 0x7faa20158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa24e36700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa200aa010 0x7faa20158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51102/0 (socket says 172.21.15.7:51102) 2023-03-03T14:58:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa24e36700 1 -- 172.21.15.7:0/2752903817 learned_addr learned my addr 172.21.15.7:0/2752903817 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:58:40.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa1f7fe700 1 --2- 172.21.15.7:0/2752903817 >> v2:172.21.15.143:3300/0 conn(0x7faa20154f10 0x7faa20158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa24e36700 1 -- 172.21.15.7:0/2752903817 >> v2:172.21.15.143:3300/0 conn(0x7faa20154f10 msgr2=0x7faa20158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:58:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa24e36700 1 --2- 172.21.15.7:0/2752903817 >> v2:172.21.15.143:3300/0 conn(0x7faa20154f10 0x7faa20158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa24e36700 1 -- 172.21.15.7:0/2752903817 >> v1:172.21.15.143:6789/0 conn(0x7faa20151b20 legacy=0x7faa20071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:58:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.397+0000 7faa24e36700 1 -- 172.21.15.7:0/2752903817 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2006f640 con 0x7faa200aa010 2023-03-03T14:58:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa24e36700 1 --2- 172.21.15.7:0/2752903817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa200aa010 0x7faa20158540 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7faa20070850 tx=0x7faa1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38afb22e4af80a63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:58:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa1effd700 1 -- 172.21.15.7:0/2752903817 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1800e070 con 0x7faa200aa010 2023-03-03T14:58:40.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa1effd700 1 -- 172.21.15.7:0/2752903817 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18003a20 con 0x7faa200aa010 2023-03-03T14:58:40.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa1effd700 1 -- 172.21.15.7:0/2752903817 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18003dd0 con 0x7faa200aa010 2023-03-03T14:58:40.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa26899700 1 -- 172.21.15.7:0/2752903817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa200aa010 msgr2=0x7faa20158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:40.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa26899700 1 --2- 172.21.15.7:0/2752903817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa200aa010 0x7faa20158540 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7faa20070850 tx=0x7faa1800b330 comp rx=0 tx=0).stop 2023-03-03T14:58:40.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa26899700 1 -- 172.21.15.7:0/2752903817 shutdown_connections 2023-03-03T14:58:40.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa26899700 1 --2- 172.21.15.7:0/2752903817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa200aa010 0x7faa20158540 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:40.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa26899700 1 --2- 172.21.15.7:0/2752903817 >> v2:172.21.15.143:3300/0 conn(0x7faa20154f10 0x7faa20158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:40.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa26899700 1 -- 172.21.15.7:0/2752903817 >> 172.21.15.7:0/2752903817 conn(0x7faa200c8900 msgr2=0x7faa200c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:58:40.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa26899700 1 -- 172.21.15.7:0/2752903817 shutdown_connections 2023-03-03T14:58:40.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.398+0000 7faa26899700 1 -- 172.21.15.7:0/2752903817 wait complete. 2023-03-03T14:58:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.400+0000 7faa26899700 1 Processor -- start 2023-03-03T14:58:40.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.400+0000 7faa26899700 1 -- start start 2023-03-03T14:58:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa26899700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa200aa010 0x7faa20150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa26899700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa20151b20 0x7faa201ed160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa26899700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20154f10 0x7faa20153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa26899700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa20154590 con 0x7faa20154f10 2023-03-03T14:58:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa26899700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2006f320 con 0x7faa200aa010 2023-03-03T14:58:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa26899700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa201edaa0 con 0x7faa20151b20 2023-03-03T14:58:40.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa24e36700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20154f10 0x7faa20153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:40.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa24e36700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20154f10 0x7faa20153540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51108/0 (socket says 172.21.15.7:51108) 2023-03-03T14:58:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa24e36700 1 -- 172.21.15.7:0/702193942 learned_addr learned my addr 172.21.15.7:0/702193942 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:58:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa1f7fe700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa20151b20 0x7faa201ed160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa24e36700 1 -- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa20151b20 msgr2=0x7faa201ed160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa24e36700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa20151b20 0x7faa201ed160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:40.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa1ffff700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa200aa010 0x7faa20150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:40.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa24e36700 1 -- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa200aa010 msgr2=0x7faa20150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:40.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa24e36700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa200aa010 0x7faa20150ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:40.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa24e36700 1 -- 172.21.15.7:0/702193942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa18009b30 con 0x7faa20154f10 2023-03-03T14:58:40.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa24e36700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20154f10 0x7faa20153540 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7faa18015040 tx=0x7faa18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:58:40.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa1cff9700 1 -- 172.21.15.7:0/702193942 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18004440 con 0x7faa20154f10 2023-03-03T14:58:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa1cff9700 1 -- 172.21.15.7:0/702193942 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa180045c0 con 0x7faa20154f10 2023-03-03T14:58:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa26899700 1 -- 172.21.15.7:0/702193942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2006fc00 con 0x7faa20154f10 2023-03-03T14:58:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa1cff9700 1 -- 172.21.15.7:0/702193942 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18004970 con 0x7faa20154f10 2023-03-03T14:58:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.401+0000 7faa26899700 1 -- 172.21.15.7:0/702193942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2006f920 con 0x7faa20154f10 2023-03-03T14:58:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.402+0000 7faa26899700 1 -- 172.21.15.7:0/702193942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa20071f30 con 0x7faa20154f10 2023-03-03T14:58:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.403+0000 7faa1cff9700 1 -- 172.21.15.7:0/702193942 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faa18022030 con 0x7faa20154f10 2023-03-03T14:58:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.403+0000 7faa1cff9700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 0x7faa0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:40.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.403+0000 7faa1cff9700 1 -- 172.21.15.7:0/702193942 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faa180874c0 con 0x7faa20154f10 2023-03-03T14:58:40.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.403+0000 7faa1ffff700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 0x7faa0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:40.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.404+0000 7faa1ffff700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 0x7faa0c06a7a0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7faa10003e60 tx=0x7faa10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:58:40.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.408+0000 7faa1cff9700 1 -- 172.21.15.7:0/702193942 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faa18053860 con 0x7faa20154f10 2023-03-03T14:58:40.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.645+0000 7faa26899700 1 -- 172.21.15.7:0/702193942 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa20074bb0 con 0x7faa20154f10 2023-03-03T14:58:40.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.645+0000 7faa1cff9700 1 -- 172.21.15.7:0/702193942 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faa18007310 con 0x7faa20154f10 2023-03-03T14:58:40.648 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:58:40.648 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:58:40.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 -- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 msgr2=0x7faa0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:40.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 0x7faa0c06a7a0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7faa10003e60 tx=0x7faa10009040 comp rx=0 tx=0).stop 2023-03-03T14:58:40.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 -- 172.21.15.7:0/702193942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20154f10 msgr2=0x7faa20153540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:40.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20154f10 0x7faa20153540 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7faa18015040 tx=0x7faa18004ad0 comp rx=0 tx=0).stop 2023-03-03T14:58:40.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 -- 172.21.15.7:0/702193942 shutdown_connections 2023-03-03T14:58:40.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 0x7faa0c06a7a0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:40.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa20154f10 0x7faa20153540 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:40.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa200aa010 0x7faa20150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:40.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 --2- 172.21.15.7:0/702193942 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa20151b20 0x7faa201ed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:40.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 -- 172.21.15.7:0/702193942 >> 172.21.15.7:0/702193942 conn(0x7faa200c8900 msgr2=0x7faa2014b170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:58:40.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 -- 172.21.15.7:0/702193942 shutdown_connections 2023-03-03T14:58:40.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:40.647+0000 7faa26899700 1 -- 172.21.15.7:0/702193942 wait complete. 2023-03-03T14:58:40.653 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:58:40.661 INFO:tasks.mds_thrash.fs.[b]:waiting for 66 secs before thrashing 2023-03-03T14:58:48.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:58:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.203+0000 7f7e10c67700 1 Processor -- start 2023-03-03T14:58:48.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.203+0000 7f7e10c67700 1 -- start start 2023-03-03T14:58:48.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e10c67700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7e0c14f9d0 0x7f7e0c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:48.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e10c67700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e0c14fda0 0x7f7e0c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e10c67700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0c0abe20 0x7f7e0c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e10c67700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0c0c2f30 con 0x7f7e0c14fda0 2023-03-03T14:58:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e10c67700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0c0c31d0 con 0x7f7e0c14f9d0 2023-03-03T14:58:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e10c67700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c1524a0 con 0x7f7e0c0abe20 2023-03-03T14:58:48.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e09d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e0c14fda0 0x7f7e0c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e09d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e0c14fda0 0x7f7e0c071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51114/0 (socket says 172.21.15.7:51114) 2023-03-03T14:58:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e09d9b700 1 -- 172.21.15.7:0/631987 learned_addr learned my addr 172.21.15.7:0/631987 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:58:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e0ad9d700 1 --2- 172.21.15.7:0/631987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0c0abe20 0x7f7e0c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e09d9b700 1 -- 172.21.15.7:0/631987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0c0abe20 msgr2=0x7f7e0c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:58:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e0a59c700 1 --2- 172.21.15.7:0/631987 >> v2:172.21.15.143:3300/0 conn(0x7f7e0c14f9d0 0x7f7e0c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e09d9b700 1 --2- 172.21.15.7:0/631987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0c0abe20 0x7f7e0c1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e09d9b700 1 -- 172.21.15.7:0/631987 >> v2:172.21.15.143:3300/0 conn(0x7f7e0c14f9d0 msgr2=0x7f7e0c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:58:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e09d9b700 1 --2- 172.21.15.7:0/631987 >> v2:172.21.15.143:3300/0 conn(0x7f7e0c14f9d0 0x7f7e0c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e09d9b700 1 -- 172.21.15.7:0/631987 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c0c34b0 con 0x7f7e0c14fda0 2023-03-03T14:58:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e09d9b700 1 --2- 172.21.15.7:0/631987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c14fda0 0x7f7e0c071890 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0710a0 tx=0x7f7e00009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29655918d3e3f857 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:58:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.204+0000 7f7e0959a700 1 -- 172.21.15.7:0/631987 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000b870 con 0x7f7e0c14fda0 2023-03-03T14:58:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e0959a700 1 -- 172.21.15.7:0/631987 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0000b9d0 con 0x7f7e0c14fda0 2023-03-03T14:58:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e0959a700 1 -- 172.21.15.7:0/631987 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000bd60 con 0x7f7e0c14fda0 2023-03-03T14:58:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e10c67700 1 -- 172.21.15.7:0/631987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c14fda0 msgr2=0x7f7e0c071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e10c67700 1 --2- 172.21.15.7:0/631987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c14fda0 0x7f7e0c071890 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0710a0 tx=0x7f7e00009bd0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e10c67700 1 -- 172.21.15.7:0/631987 shutdown_connections 2023-03-03T14:58:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e10c67700 1 --2- 172.21.15.7:0/631987 >> v2:172.21.15.143:3300/0 conn(0x7f7e0c14f9d0 0x7f7e0c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e10c67700 1 --2- 172.21.15.7:0/631987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0c0abe20 0x7f7e0c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e10c67700 1 --2- 172.21.15.7:0/631987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c14fda0 0x7f7e0c071890 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e10c67700 1 -- 172.21.15.7:0/631987 >> 172.21.15.7:0/631987 conn(0x7f7e0c0c8900 msgr2=0x7f7e0c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:58:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e10c67700 1 -- 172.21.15.7:0/631987 shutdown_connections 2023-03-03T14:58:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.205+0000 7f7e10c67700 1 -- 172.21.15.7:0/631987 wait complete. 2023-03-03T14:58:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.206+0000 7f7e10c67700 1 Processor -- start 2023-03-03T14:58:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.207+0000 7f7e10c67700 1 -- start start 2023-03-03T14:58:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.207+0000 7f7e10c67700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c0abe20 0x7f7e0c1436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.207+0000 7f7e10c67700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0c14f9d0 0x7f7e0c06e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:48.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.207+0000 7f7e10c67700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0c14fda0 0x7f7e0c0bf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.207+0000 7f7e10c67700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c0c11f0 con 0x7f7e0c0abe20 2023-03-03T14:58:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.207+0000 7f7e10c67700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c0c28d0 con 0x7f7e0c14f9d0 2023-03-03T14:58:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.207+0000 7f7e10c67700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c0bdf80 con 0x7f7e0c14fda0 2023-03-03T14:58:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.207+0000 7f7e0a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c0abe20 0x7f7e0c1436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.207+0000 7f7e0a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c0abe20 0x7f7e0c1436d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51118/0 (socket says 172.21.15.7:51118) 2023-03-03T14:58:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.207+0000 7f7e0a59c700 1 -- 172.21.15.7:0/2640433002 learned_addr learned my addr 172.21.15.7:0/2640433002 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:58:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e09d9b700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0c14f9d0 0x7f7e0c06e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e0ad9d700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0c14fda0 0x7f7e0c0bf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e0a59c700 1 -- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0c14fda0 msgr2=0x7f7e0c0bf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e0a59c700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0c14fda0 0x7f7e0c0bf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e0a59c700 1 -- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0c14f9d0 msgr2=0x7f7e0c06e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e0a59c700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0c14f9d0 0x7f7e0c06e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e0a59c700 1 -- 172.21.15.7:0/2640433002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00009550 con 0x7f7e0c0abe20 2023-03-03T14:58:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e09d9b700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0c14f9d0 0x7f7e0c06e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T14:58:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e0a59c700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c0abe20 0x7f7e0c1436d0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c7f0 tx=0x7f7dfc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:58:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7dfaffd700 1 -- 172.21.15.7:0/2640433002 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc011800 con 0x7f7e0c0abe20 2023-03-03T14:58:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7dfaffd700 1 -- 172.21.15.7:0/2640433002 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc011960 con 0x7f7e0c0abe20 2023-03-03T14:58:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e10c67700 1 -- 172.21.15.7:0/2640433002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0c0ac350 con 0x7f7e0c0abe20 2023-03-03T14:58:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7dfaffd700 1 -- 172.21.15.7:0/2640433002 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc011cf0 con 0x7f7e0c0abe20 2023-03-03T14:58:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.208+0000 7f7e10c67700 1 -- 172.21.15.7:0/2640433002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0c0ab630 con 0x7f7e0c0abe20 2023-03-03T14:58:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.209+0000 7f7e10c67700 1 -- 172.21.15.7:0/2640433002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0c0c3e10 con 0x7f7e0c0abe20 2023-03-03T14:58:48.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.210+0000 7f7dfaffd700 1 -- 172.21.15.7:0/2640433002 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7dfc021070 con 0x7f7e0c0abe20 2023-03-03T14:58:48.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.210+0000 7f7dfaffd700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dec0682f0 0x7f7dec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:58:48.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.210+0000 7f7dfaffd700 1 -- 172.21.15.7:0/2640433002 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7dfc0864f0 con 0x7f7e0c0abe20 2023-03-03T14:58:48.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.211+0000 7f7e09d9b700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dec0682f0 0x7f7dec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:58:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.211+0000 7f7e09d9b700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dec0682f0 0x7f7dec06a7a0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c1501d0 tx=0x7f7e00008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:58:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.214+0000 7f7dfaffd700 1 -- 172.21.15.7:0/2640433002 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7dfc0527a0 con 0x7f7e0c0abe20 2023-03-03T14:58:48.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.457+0000 7f7e10c67700 1 -- 172.21.15.7:0/2640433002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e0c0bddb0 con 0x7f7e0c0abe20 2023-03-03T14:58:48.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.458+0000 7f7dfaffd700 1 -- 172.21.15.7:0/2640433002 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7dfc055dc0 con 0x7f7e0c0abe20 2023-03-03T14:58:48.460 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:58:48.460 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:58:48.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 -- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dec0682f0 msgr2=0x7f7dec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:48.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dec0682f0 0x7f7dec06a7a0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c1501d0 tx=0x7f7e00008030 comp rx=0 tx=0).stop 2023-03-03T14:58:48.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 -- 172.21.15.7:0/2640433002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c0abe20 msgr2=0x7f7e0c1436d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:58:48.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c0abe20 0x7f7e0c1436d0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c7f0 tx=0x7f7dfc00cbb0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 -- 172.21.15.7:0/2640433002 shutdown_connections 2023-03-03T14:58:48.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e0c14fda0 0x7f7e0c0bf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7dec0682f0 0x7f7dec06a7a0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e0c0abe20 0x7f7e0c1436d0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 --2- 172.21.15.7:0/2640433002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e0c14f9d0 0x7f7e0c06e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:58:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 -- 172.21.15.7:0/2640433002 >> 172.21.15.7:0/2640433002 conn(0x7f7e0c0c8900 msgr2=0x7f7e0c155640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:58:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 -- 172.21.15.7:0/2640433002 shutdown_connections 2023-03-03T14:58:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:58:48.460+0000 7f7e10c67700 1 -- 172.21.15.7:0/2640433002 wait complete. 2023-03-03T14:58:48.467 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:58:48.474 INFO:tasks.mds_thrash.fs.[a]:waiting for 53 secs before thrashing 2023-03-03T14:58:58.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:58:58.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:28.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:28.972 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:41.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:59:41.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.615+0000 7f3258041700 1 Processor -- start 2023-03-03T14:59:41.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.615+0000 7f3258041700 1 -- start start 2023-03-03T14:59:41.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.616+0000 7f3258041700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3250155410 0x7f32500bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:41.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.616+0000 7f3258041700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f32500c5b00 0x7f32500bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:41.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.616+0000 7f3258041700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32500c08d0 con 0x7f3250151e80 2023-03-03T14:59:41.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.616+0000 7f3258041700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3250152250 con 0x7f32500c5b00 2023-03-03T14:59:41.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.616+0000 7f3258041700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32500c0fb0 con 0x7f3250155410 2023-03-03T14:59:41.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.616+0000 7f32565de700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f32500c5b00 0x7f32500bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:41.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.616+0000 7f32565de700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f32500c5b00 0x7f32500bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51164/0 (socket says 172.21.15.7:51164) 2023-03-03T14:59:41.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.616+0000 7f32565de700 1 -- 172.21.15.7:0/558511479 learned_addr learned my addr 172.21.15.7:0/558511479 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:59:41.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.617+0000 7f32555dc700 1 --2- 172.21.15.7:0/558511479 >> v2:172.21.15.143:3300/0 conn(0x7f3250155410 0x7f32500bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:41.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.617+0000 7f32565de700 1 -- 172.21.15.7:0/558511479 >> v2:172.21.15.143:3300/0 conn(0x7f3250155410 msgr2=0x7f32500bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:59:41.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.617+0000 7f32565de700 1 --2- 172.21.15.7:0/558511479 >> v2:172.21.15.143:3300/0 conn(0x7f3250155410 0x7f32500bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:41.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.617+0000 7f32565de700 1 -- 172.21.15.7:0/558511479 >> v1:172.21.15.143:6789/0 conn(0x7f3250151e80 legacy=0x7f32500c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:59:41.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.617+0000 7f32565de700 1 -- 172.21.15.7:0/558511479 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3250154930 con 0x7f32500c5b00 2023-03-03T14:59:41.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.617+0000 7f32565de700 1 --2- 172.21.15.7:0/558511479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 0x7f32500bfe20 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f325006c380 tx=0x7f324800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f512aaef64f3b8b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:59:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.617+0000 7f3254ddb700 1 -- 172.21.15.7:0/558511479 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324800e070 con 0x7f32500c5b00 2023-03-03T14:59:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.617+0000 7f3254ddb700 1 -- 172.21.15.7:0/558511479 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324800b9c0 con 0x7f32500c5b00 2023-03-03T14:59:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.617+0000 7f3254ddb700 1 -- 172.21.15.7:0/558511479 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324800bd70 con 0x7f32500c5b00 2023-03-03T14:59:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.618+0000 7f3258041700 1 -- 172.21.15.7:0/558511479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 msgr2=0x7f32500bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.618+0000 7f3258041700 1 --2- 172.21.15.7:0/558511479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 0x7f32500bfe20 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f325006c380 tx=0x7f324800b330 comp rx=0 tx=0).stop 2023-03-03T14:59:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.618+0000 7f3258041700 1 -- 172.21.15.7:0/558511479 shutdown_connections 2023-03-03T14:59:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.618+0000 7f3258041700 1 --2- 172.21.15.7:0/558511479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 0x7f32500bfe20 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:41.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.618+0000 7f3258041700 1 --2- 172.21.15.7:0/558511479 >> v2:172.21.15.143:3300/0 conn(0x7f3250155410 0x7f32500bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:41.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.618+0000 7f3258041700 1 -- 172.21.15.7:0/558511479 >> 172.21.15.7:0/558511479 conn(0x7f3250071630 msgr2=0x7f325006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:59:41.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.618+0000 7f3258041700 1 -- 172.21.15.7:0/558511479 shutdown_connections 2023-03-03T14:59:41.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.618+0000 7f3258041700 1 -- 172.21.15.7:0/558511479 wait complete. 2023-03-03T14:59:41.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.619+0000 7f3258041700 1 Processor -- start 2023-03-03T14:59:41.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.619+0000 7f3258041700 1 -- start start 2023-03-03T14:59:41.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3258041700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 0x7f325014f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:41.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3258041700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3250151e80 0x7f32501573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:41.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3258041700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3250155410 0x7f32501578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:41.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3258041700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325014d990 con 0x7f32500c5b00 2023-03-03T14:59:41.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3258041700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325014dc70 con 0x7f3250151e80 2023-03-03T14:59:41.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3258041700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325014df50 con 0x7f3250155410 2023-03-03T14:59:41.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3255ddd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 0x7f325014f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:41.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3255ddd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 0x7f325014f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51166/0 (socket says 172.21.15.7:51166) 2023-03-03T14:59:41.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3255ddd700 1 -- 172.21.15.7:0/2612521977 learned_addr learned my addr 172.21.15.7:0/2612521977 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:59:41.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f32555dc700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3250151e80 0x7f32501573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:41.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f32565de700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3250155410 0x7f32501578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:41.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3255ddd700 1 -- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3250155410 msgr2=0x7f32501578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:41.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3255ddd700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3250155410 0x7f32501578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:41.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3255ddd700 1 -- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3250151e80 msgr2=0x7f32501573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:41.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3255ddd700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3250151e80 0x7f32501573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:41.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3255ddd700 1 -- 172.21.15.7:0/2612521977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3248009b30 con 0x7f32500c5b00 2023-03-03T14:59:41.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3255ddd700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 0x7f325014f1e0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f324c00c4e0 tx=0x7f324c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:59:41.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f32467fc700 1 -- 172.21.15.7:0/2612521977 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c0052d0 con 0x7f32500c5b00 2023-03-03T14:59:41.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f32467fc700 1 -- 172.21.15.7:0/2612521977 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c005430 con 0x7f32500c5b00 2023-03-03T14:59:41.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.620+0000 7f3258041700 1 -- 172.21.15.7:0/2612521977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3250154ef0 con 0x7f32500c5b00 2023-03-03T14:59:41.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.621+0000 7f32467fc700 1 -- 172.21.15.7:0/2612521977 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c0038c0 con 0x7f32500c5b00 2023-03-03T14:59:41.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.621+0000 7f3258041700 1 -- 172.21.15.7:0/2612521977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3250154c10 con 0x7f32500c5b00 2023-03-03T14:59:41.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.622+0000 7f3258041700 1 -- 172.21.15.7:0/2612521977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32500c5220 con 0x7f32500c5b00 2023-03-03T14:59:41.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.622+0000 7f32467fc700 1 -- 172.21.15.7:0/2612521977 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f324c019030 con 0x7f32500c5b00 2023-03-03T14:59:41.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.623+0000 7f32467fc700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32340682f0 0x7f323406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:41.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.623+0000 7f32467fc700 1 -- 172.21.15.7:0/2612521977 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f324c085500 con 0x7f32500c5b00 2023-03-03T14:59:41.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.623+0000 7f32555dc700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32340682f0 0x7f323406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:41.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.623+0000 7f32555dc700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32340682f0 0x7f323406a7a0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f32501502b0 tx=0x7f3240008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:59:41.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.627+0000 7f32467fc700 1 -- 172.21.15.7:0/2612521977 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f324c0518a0 con 0x7f32500c5b00 2023-03-03T14:59:41.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.864+0000 7f3258041700 1 -- 172.21.15.7:0/2612521977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3250074bb0 con 0x7f32500c5b00 2023-03-03T14:59:41.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.864+0000 7f32467fc700 1 -- 172.21.15.7:0/2612521977 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f324c054ec0 con 0x7f32500c5b00 2023-03-03T14:59:41.867 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:59:41.867 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:59:41.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 -- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32340682f0 msgr2=0x7f323406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:41.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32340682f0 0x7f323406a7a0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f32501502b0 tx=0x7f3240008040 comp rx=0 tx=0).stop 2023-03-03T14:59:41.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 -- 172.21.15.7:0/2612521977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 msgr2=0x7f325014f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:41.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 0x7f325014f1e0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f324c00c4e0 tx=0x7f324c00c8a0 comp rx=0 tx=0).stop 2023-03-03T14:59:41.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 -- 172.21.15.7:0/2612521977 shutdown_connections 2023-03-03T14:59:41.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3250155410 0x7f32501578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:41.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32340682f0 0x7f323406a7a0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:41.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32500c5b00 0x7f325014f1e0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:41.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 --2- 172.21.15.7:0/2612521977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3250151e80 0x7f32501573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:41.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 -- 172.21.15.7:0/2612521977 >> 172.21.15.7:0/2612521977 conn(0x7f3250071630 msgr2=0x7f32500bf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:59:41.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 -- 172.21.15.7:0/2612521977 shutdown_connections 2023-03-03T14:59:41.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:41.866+0000 7f3258041700 1 -- 172.21.15.7:0/2612521977 wait complete. 2023-03-03T14:59:41.873 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:59:41.880 INFO:tasks.mds_thrash.fs.[a]:waiting for 30 secs before thrashing 2023-03-03T14:59:46.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T14:59:46.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.800+0000 7fc4e54eb700 1 Processor -- start 2023-03-03T14:59:46.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.800+0000 7fc4e54eb700 1 -- start start 2023-03-03T14:59:46.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.801+0000 7fc4e54eb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc4e0151cc0 0x7fc4e00ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:46.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.801+0000 7fc4e54eb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e00c6cf0 con 0x7fc4e00cac10 2023-03-03T14:59:46.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.801+0000 7fc4e54eb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e01548f0 con 0x7fc4e00b00b0 2023-03-03T14:59:46.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.801+0000 7fc4e54eb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e0152090 con 0x7fc4e0151cc0 2023-03-03T14:59:46.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.801+0000 7fc4deffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc4e0151cc0 0x7fc4e00ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:46.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.801+0000 7fc4df7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc4e00cac10 0x7fc4e015ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45156/0 (socket says 172.21.15.7:45156) 2023-03-03T14:59:46.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.801+0000 7fc4df7fe700 1 -- 172.21.15.7:0/640499756 learned_addr learned my addr 172.21.15.7:0/640499756 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T14:59:46.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4deffd700 1 -- 172.21.15.7:0/640499756 >> v1:172.21.15.143:6789/0 conn(0x7fc4e00b00b0 legacy=0x7fc4e00c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T14:59:46.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4deffd700 1 -- 172.21.15.7:0/640499756 >> v1:172.21.15.7:6789/0 conn(0x7fc4e00cac10 legacy=0x7fc4e015ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:46.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4deffd700 1 -- 172.21.15.7:0/640499756 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e0154bd0 con 0x7fc4e0151cc0 2023-03-03T14:59:46.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4ddffb700 1 -- 172.21.15.7:0/640499756 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617047459 0 0) 0x7fc4e00c6cf0 con 0x7fc4e00cac10 2023-03-03T14:59:46.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4deffd700 1 --2- 172.21.15.7:0/640499756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 0x7fc4e00ca6d0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009840 tx=0x7fc4d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d781d88fbd811ab server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:59:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4ddffb700 1 -- 172.21.15.7:0/640499756 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0004230 con 0x7fc4e0151cc0 2023-03-03T14:59:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4ddffb700 1 -- 172.21.15.7:0/640499756 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d0004390 con 0x7fc4e0151cc0 2023-03-03T14:59:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4ddffb700 1 -- 172.21.15.7:0/640499756 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d00036a0 con 0x7fc4e0151cc0 2023-03-03T14:59:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4e54eb700 1 -- 172.21.15.7:0/640499756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 msgr2=0x7fc4e00ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4e54eb700 1 --2- 172.21.15.7:0/640499756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 0x7fc4e00ca6d0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009840 tx=0x7fc4d0009b50 comp rx=0 tx=0).stop 2023-03-03T14:59:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4e54eb700 1 -- 172.21.15.7:0/640499756 shutdown_connections 2023-03-03T14:59:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4e54eb700 1 --2- 172.21.15.7:0/640499756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 0x7fc4e00ca6d0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.802+0000 7fc4e54eb700 1 -- 172.21.15.7:0/640499756 >> 172.21.15.7:0/640499756 conn(0x7fc4e006b980 msgr2=0x7fc4e0071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:59:46.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.803+0000 7fc4e54eb700 1 -- 172.21.15.7:0/640499756 shutdown_connections 2023-03-03T14:59:46.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.803+0000 7fc4e54eb700 1 -- 172.21.15.7:0/640499756 wait complete. 2023-03-03T14:59:46.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.804+0000 7fc4e54eb700 1 Processor -- start 2023-03-03T14:59:46.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.804+0000 7fc4e54eb700 1 -- start start 2023-03-03T14:59:46.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.804+0000 7fc4e54eb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4e00b00b0 0x7fc4e00c3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:46.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4e54eb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e00cac10 0x7fc4e01edf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:46.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4e54eb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 0x7fc4e00c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:46.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4e54eb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e01ee490 con 0x7fc4e0151cc0 2023-03-03T14:59:46.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4e54eb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e00c3c70 con 0x7fc4e00cac10 2023-03-03T14:59:46.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4e54eb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e00ca410 con 0x7fc4e00b00b0 2023-03-03T14:59:46.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4df7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 0x7fc4e00c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:46.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4df7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 0x7fc4e00c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51182/0 (socket says 172.21.15.7:51182) 2023-03-03T14:59:46.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4df7fe700 1 -- 172.21.15.7:0/352488684 learned_addr learned my addr 172.21.15.7:0/352488684 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T14:59:46.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4deffd700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4e00b00b0 0x7fc4e00c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:46.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4de7fc700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e00cac10 0x7fc4e01edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:46.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4df7fe700 1 -- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4e00b00b0 msgr2=0x7fc4e00c3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:46.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4df7fe700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4e00b00b0 0x7fc4e00c3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:46.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4df7fe700 1 -- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e00cac10 msgr2=0x7fc4e01edf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:46.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4df7fe700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e00cac10 0x7fc4e01edf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4df7fe700 1 -- 172.21.15.7:0/352488684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d00094a0 con 0x7fc4e0151cc0 2023-03-03T14:59:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4df7fe700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 0x7fc4e00c9930 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800cca0 tx=0x7fc4d800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:59:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4c77fe700 1 -- 172.21.15.7:0/352488684 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d800cea0 con 0x7fc4e0151cc0 2023-03-03T14:59:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4c77fe700 1 -- 172.21.15.7:0/352488684 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d800ade0 con 0x7fc4e0151cc0 2023-03-03T14:59:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4e54eb700 1 -- 172.21.15.7:0/352488684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e0155190 con 0x7fc4e0151cc0 2023-03-03T14:59:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4c77fe700 1 -- 172.21.15.7:0/352488684 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8007310 con 0x7fc4e0151cc0 2023-03-03T14:59:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.805+0000 7fc4e54eb700 1 -- 172.21.15.7:0/352488684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e0154eb0 con 0x7fc4e0151cc0 2023-03-03T14:59:46.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.806+0000 7fc4e54eb700 1 -- 172.21.15.7:0/352488684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e01545a0 con 0x7fc4e0151cc0 2023-03-03T14:59:46.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.807+0000 7fc4c77fe700 1 -- 172.21.15.7:0/352488684 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc4d801a030 con 0x7fc4e0151cc0 2023-03-03T14:59:46.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.807+0000 7fc4c77fe700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4cc0682f0 0x7fc4cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T14:59:46.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.807+0000 7fc4c77fe700 1 -- 172.21.15.7:0/352488684 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc4d8085f30 con 0x7fc4e0151cc0 2023-03-03T14:59:46.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.807+0000 7fc4deffd700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4cc0682f0 0x7fc4cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T14:59:46.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.808+0000 7fc4deffd700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4cc0682f0 0x7fc4cc06a7a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00090b0 tx=0x7fc4d0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T14:59:46.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:46.811+0000 7fc4c77fe700 1 -- 172.21.15.7:0/352488684 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc4d80522d0 con 0x7fc4e0151cc0 2023-03-03T14:59:47.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.048+0000 7fc4e54eb700 1 -- 172.21.15.7:0/352488684 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4e00c0800 con 0x7fc4e0151cc0 2023-03-03T14:59:47.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.048+0000 7fc4c77fe700 1 -- 172.21.15.7:0/352488684 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc4d8018090 con 0x7fc4e0151cc0 2023-03-03T14:59:47.050 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T14:59:47.050 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T14:59:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 -- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4cc0682f0 msgr2=0x7fc4cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4cc0682f0 0x7fc4cc06a7a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00090b0 tx=0x7fc4d0008030 comp rx=0 tx=0).stop 2023-03-03T14:59:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 -- 172.21.15.7:0/352488684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 msgr2=0x7fc4e00c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T14:59:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 0x7fc4e00c9930 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800cca0 tx=0x7fc4d800a340 comp rx=0 tx=0).stop 2023-03-03T14:59:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 -- 172.21.15.7:0/352488684 shutdown_connections 2023-03-03T14:59:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4cc0682f0 0x7fc4cc06a7a0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4e0151cc0 0x7fc4e00c9930 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4e00b00b0 0x7fc4e00c3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 --2- 172.21.15.7:0/352488684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e00cac10 0x7fc4e01edf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T14:59:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 -- 172.21.15.7:0/352488684 >> 172.21.15.7:0/352488684 conn(0x7fc4e006b980 msgr2=0x7fc4e00c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T14:59:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 -- 172.21.15.7:0/352488684 shutdown_connections 2023-03-03T14:59:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T14:59:47.050+0000 7fc4e54eb700 1 -- 172.21.15.7:0/352488684 wait complete. 2023-03-03T14:59:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T14:59:47.064 INFO:tasks.mds_thrash.fs.[b]:waiting for 49 secs before thrashing 2023-03-03T14:59:59.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T14:59:59.034 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T14:59:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T15:00:11.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:00:12.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.019+0000 7f95650a7700 1 Processor -- start 2023-03-03T15:00:12.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.020+0000 7f95650a7700 1 -- start start 2023-03-03T15:00:12.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.020+0000 7f95650a7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f956014fde0 0x7f95600a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:12.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.020+0000 7f95650a7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f95600abd70 0x7f95600ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:12.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.020+0000 7f95650a7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95600c4fc0 con 0x7f9560155a00 2023-03-03T15:00:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.020+0000 7f95650a7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95600c5730 con 0x7f956014fde0 2023-03-03T15:00:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.020+0000 7f95650a7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956006da50 con 0x7f95600abd70 2023-03-03T15:00:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.020+0000 7f955ed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f956014fde0 0x7f95600a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.020+0000 7f955ed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f956014fde0 0x7f95600a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51206/0 (socket says 172.21.15.7:51206) 2023-03-03T15:00:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.020+0000 7f955ed9d700 1 -- 172.21.15.7:0/731842476 learned_addr learned my addr 172.21.15.7:0/731842476 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:00:12.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.021+0000 7f955e59c700 1 --2- 172.21.15.7:0/731842476 >> v2:172.21.15.143:3300/0 conn(0x7f95600abd70 0x7f95600ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:12.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.021+0000 7f955ed9d700 1 -- 172.21.15.7:0/731842476 >> v2:172.21.15.143:3300/0 conn(0x7f95600abd70 msgr2=0x7f95600ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:00:12.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.021+0000 7f955ed9d700 1 --2- 172.21.15.7:0/731842476 >> v2:172.21.15.143:3300/0 conn(0x7f95600abd70 0x7f95600ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:12.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.021+0000 7f955ed9d700 1 -- 172.21.15.7:0/731842476 >> v1:172.21.15.143:6789/0 conn(0x7f9560155a00 legacy=0x7f9560155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:12.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.021+0000 7f955ed9d700 1 -- 172.21.15.7:0/731842476 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9560156590 con 0x7f956014fde0 2023-03-03T15:00:12.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.021+0000 7f955e59c700 1 --2- 172.21.15.7:0/731842476 >> v2:172.21.15.143:3300/0 conn(0x7f95600abd70 0x7f95600ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:00:12.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.021+0000 7f955ed9d700 1 --2- 172.21.15.7:0/731842476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956014fde0 0x7f95600a5d10 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f95600acb20 tx=0x7f9550009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5162ccf29570afad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:00:12.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.021+0000 7f955dd9b700 1 -- 172.21.15.7:0/731842476 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955000b860 con 0x7f956014fde0 2023-03-03T15:00:12.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.021+0000 7f955dd9b700 1 -- 172.21.15.7:0/731842476 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955000b9c0 con 0x7f956014fde0 2023-03-03T15:00:12.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.021+0000 7f955dd9b700 1 -- 172.21.15.7:0/731842476 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955000bd50 con 0x7f956014fde0 2023-03-03T15:00:12.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.022+0000 7f95650a7700 1 -- 172.21.15.7:0/731842476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956014fde0 msgr2=0x7f95600a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:12.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.022+0000 7f95650a7700 1 --2- 172.21.15.7:0/731842476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956014fde0 0x7f95600a5d10 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f95600acb20 tx=0x7f9550009c00 comp rx=0 tx=0).stop 2023-03-03T15:00:12.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.022+0000 7f95650a7700 1 -- 172.21.15.7:0/731842476 shutdown_connections 2023-03-03T15:00:12.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.022+0000 7f95650a7700 1 --2- 172.21.15.7:0/731842476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f956014fde0 0x7f95600a5d10 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:12.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.022+0000 7f95650a7700 1 --2- 172.21.15.7:0/731842476 >> v2:172.21.15.143:3300/0 conn(0x7f95600abd70 0x7f95600ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:12.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.022+0000 7f95650a7700 1 -- 172.21.15.7:0/731842476 >> 172.21.15.7:0/731842476 conn(0x7f956006a2b0 msgr2=0x7f956006a680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:00:12.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.022+0000 7f95650a7700 1 -- 172.21.15.7:0/731842476 shutdown_connections 2023-03-03T15:00:12.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.022+0000 7f95650a7700 1 -- 172.21.15.7:0/731842476 wait complete. 2023-03-03T15:00:12.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.023+0000 7f95650a7700 1 Processor -- start 2023-03-03T15:00:12.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.023+0000 7f95650a7700 1 -- start start 2023-03-03T15:00:12.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f95650a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95600abd70 0x7f9560154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:12.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f95650a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f956014fde0 0x7f95600a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:12.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f95650a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9560155a00 0x7f95600bef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:12.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f95650a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95600bd480 con 0x7f95600abd70 2023-03-03T15:00:12.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f95650a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95600bd720 con 0x7f956014fde0 2023-03-03T15:00:12.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f95650a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9560150e00 con 0x7f9560155a00 2023-03-03T15:00:12.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95600abd70 0x7f9560154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:12.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95600abd70 0x7f9560154f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51210/0 (socket says 172.21.15.7:51210) 2023-03-03T15:00:12.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955ed9d700 1 -- 172.21.15.7:0/2353565339 learned_addr learned my addr 172.21.15.7:0/2353565339 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:00:12.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955e59c700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f956014fde0 0x7f95600a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:12.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955f59e700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9560155a00 0x7f95600bef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:12.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955ed9d700 1 -- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9560155a00 msgr2=0x7f95600bef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:12.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955ed9d700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9560155a00 0x7f95600bef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:12.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955ed9d700 1 -- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f956014fde0 msgr2=0x7f95600a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:12.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955ed9d700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f956014fde0 0x7f95600a9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:12.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955ed9d700 1 -- 172.21.15.7:0/2353565339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9550009550 con 0x7f95600abd70 2023-03-03T15:00:12.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.024+0000 7f955e59c700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f956014fde0 0x7f95600a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:00:12.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.025+0000 7f955ed9d700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95600abd70 0x7f9560154f20 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f9550009130 tx=0x7f9550004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:00:12.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.025+0000 7f95577fe700 1 -- 172.21.15.7:0/2353565339 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95500045d0 con 0x7f95600abd70 2023-03-03T15:00:12.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.025+0000 7f95577fe700 1 -- 172.21.15.7:0/2353565339 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955001e070 con 0x7f95600abd70 2023-03-03T15:00:12.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.025+0000 7f95650a7700 1 -- 172.21.15.7:0/2353565339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95601501b0 con 0x7f95600abd70 2023-03-03T15:00:12.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.025+0000 7f95577fe700 1 -- 172.21.15.7:0/2353565339 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95500038a0 con 0x7f95600abd70 2023-03-03T15:00:12.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.025+0000 7f95650a7700 1 -- 172.21.15.7:0/2353565339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95600c4a90 con 0x7f95600abd70 2023-03-03T15:00:12.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.026+0000 7f95650a7700 1 -- 172.21.15.7:0/2353565339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95600c5170 con 0x7f95600abd70 2023-03-03T15:00:12.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.026+0000 7f95577fe700 1 -- 172.21.15.7:0/2353565339 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f955000f050 con 0x7f95600abd70 2023-03-03T15:00:12.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.027+0000 7f95577fe700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95400682f0 0x7f954006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:12.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.027+0000 7f95577fe700 1 -- 172.21.15.7:0/2353565339 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9550020070 con 0x7f95600abd70 2023-03-03T15:00:12.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.027+0000 7f955e59c700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95400682f0 0x7f954006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:12.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.027+0000 7f955e59c700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95400682f0 0x7f954006a7a0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f9560150d30 tx=0x7f9548008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:00:12.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.031+0000 7f95577fe700 1 -- 172.21.15.7:0/2353565339 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9550053840 con 0x7f95600abd70 2023-03-03T15:00:12.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.267+0000 7f95650a7700 1 -- 172.21.15.7:0/2353565339 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95600bede0 con 0x7f95600abd70 2023-03-03T15:00:12.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.268+0000 7f95577fe700 1 -- 172.21.15.7:0/2353565339 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f955002a090 con 0x7f95600abd70 2023-03-03T15:00:12.272 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:00:12.272 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:00:12.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 -- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95400682f0 msgr2=0x7f954006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:12.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95400682f0 0x7f954006a7a0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f9560150d30 tx=0x7f9548008040 comp rx=0 tx=0).stop 2023-03-03T15:00:12.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 -- 172.21.15.7:0/2353565339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95600abd70 msgr2=0x7f9560154f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:12.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95600abd70 0x7f9560154f20 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f9550009130 tx=0x7f9550004750 comp rx=0 tx=0).stop 2023-03-03T15:00:12.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 -- 172.21.15.7:0/2353565339 shutdown_connections 2023-03-03T15:00:12.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9560155a00 0x7f95600bef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:12.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95400682f0 0x7f954006a7a0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:12.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95600abd70 0x7f9560154f20 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:12.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 --2- 172.21.15.7:0/2353565339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f956014fde0 0x7f95600a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:12.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 -- 172.21.15.7:0/2353565339 >> 172.21.15.7:0/2353565339 conn(0x7f956006a2b0 msgr2=0x7f95600c29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:00:12.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 -- 172.21.15.7:0/2353565339 shutdown_connections 2023-03-03T15:00:12.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:12.270+0000 7f95650a7700 1 -- 172.21.15.7:0/2353565339 wait complete. 2023-03-03T15:00:12.275 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:00:12.285 INFO:tasks.mds_thrash.fs.[a]:waiting for 57 secs before thrashing 2023-03-03T15:00:29.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:29.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:36.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:00:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.203+0000 7f2242ee2700 1 Processor -- start 2023-03-03T15:00:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.203+0000 7f2242ee2700 1 -- start start 2023-03-03T15:00:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f2242ee2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f223c071e80 0x7f223c151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f2242ee2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c14e900 0x7f223c15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f2242ee2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c1510d0 con 0x7f223c072250 2023-03-03T15:00:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f2242ee2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f223c0c3740 con 0x7f223c071e80 2023-03-03T15:00:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f2242ee2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f223c070670 con 0x7f223c14e900 2023-03-03T15:00:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f223c072250 0x7f223c14e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45220/0 (socket says 172.21.15.7:45220) 2023-03-03T15:00:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223bfff700 1 -- 172.21.15.7:0/2960210173 learned_addr learned my addr 172.21.15.7:0/2960210173 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:00:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325814500 0 0) 0x7f223c1510d0 con 0x7f223c072250 2023-03-03T15:00:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f224147f700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c14e900 0x7f223c15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f2240c7e700 1 --2- 172.21.15.7:0/2960210173 >> v2:172.21.15.143:3300/0 conn(0x7f223c071e80 0x7f223c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c0035b0 con 0x7f223c072250 2023-03-03T15:00:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2169705925 0 0) 0x7f222c0035b0 con 0x7f223c072250 2023-03-03T15:00:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c14e900 msgr2=0x7f223c15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:00:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223b7fe700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c14e900 0x7f223c15ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 >> v2:172.21.15.143:3300/0 conn(0x7f223c071e80 msgr2=0x7f223c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:00:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223b7fe700 1 --2- 172.21.15.7:0/2960210173 >> v2:172.21.15.143:3300/0 conn(0x7f223c071e80 0x7f223c151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c0c39e0 con 0x7f223c072250 2023-03-03T15:00:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.204+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f2228001670 con 0x7f223c072250 2023-03-03T15:00:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 >> v1:172.21.15.7:6789/0 conn(0x7f223c072250 legacy=0x7f223c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223b7fe700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c14e900 0x7f222c003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223b7fe700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f222c003ff0 0x7f222c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223b7fe700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f222c004d90 0x7f222c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c0c3740 con 0x7f222c004d90 2023-03-03T15:00:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c070670 con 0x7f222c003ff0 2023-03-03T15:00:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c00bac0 con 0x7f223c14e900 2023-03-03T15:00:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f224147f700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f222c004d90 0x7f222c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f2240c7e700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c14e900 0x7f222c003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223bfff700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f222c003ff0 0x7f222c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f224147f700 1 -- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c14e900 msgr2=0x7f222c003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:00:36.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f224147f700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c14e900 0x7f222c003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:36.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f224147f700 1 -- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f222c003ff0 msgr2=0x7f222c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:00:36.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f224147f700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f222c003ff0 0x7f222c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:36.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f224147f700 1 -- 172.21.15.7:0/2960210173 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c0035b0 con 0x7f222c004d90 2023-03-03T15:00:36.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f224147f700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f222c004d90 0x7f222c005200 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f2234009f90 tx=0x7f222c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5df8fe7595fad4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:00:36.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223400ec30 con 0x7f222c004d90 2023-03-03T15:00:36.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223400f040 con 0x7f222c004d90 2023-03-03T15:00:36.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.205+0000 7f223b7fe700 1 -- 172.21.15.7:0/2960210173 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2234014480 con 0x7f222c004d90 2023-03-03T15:00:36.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.206+0000 7f2242ee2700 1 -- 172.21.15.7:0/2960210173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f222c004d90 msgr2=0x7f222c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:36.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.206+0000 7f2242ee2700 1 --2- 172.21.15.7:0/2960210173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f222c004d90 0x7f222c005200 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f2234009f90 tx=0x7f222c005880 comp rx=0 tx=0).stop 2023-03-03T15:00:36.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.206+0000 7f224147f700 1 -- 172.21.15.7:0/2960210173 reap_dead start 2023-03-03T15:00:36.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.206+0000 7f2242ee2700 1 -- 172.21.15.7:0/2960210173 shutdown_connections 2023-03-03T15:00:36.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.206+0000 7f2242ee2700 1 -- 172.21.15.7:0/2960210173 >> 172.21.15.7:0/2960210173 conn(0x7f223c0a94d0 msgr2=0x7f223c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:00:36.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.206+0000 7f2242ee2700 1 -- 172.21.15.7:0/2960210173 shutdown_connections 2023-03-03T15:00:36.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.206+0000 7f2242ee2700 1 -- 172.21.15.7:0/2960210173 wait complete. 2023-03-03T15:00:36.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.207+0000 7f2242ee2700 1 Processor -- start 2023-03-03T15:00:36.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.207+0000 7f2242ee2700 1 -- start start 2023-03-03T15:00:36.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f2242ee2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c147a80 0x7f223c0abba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:36.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f2242ee2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f223c1dab90 0x7f223c1eb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:36.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f2242ee2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f223c14a2b0 0x7f223c1ebbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:36.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f2242ee2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c0aa700 con 0x7f223c14a2b0 2023-03-03T15:00:36.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f2242ee2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c0aa1d0 con 0x7f223c1dab90 2023-03-03T15:00:36.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f2242ee2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f223c1490c0 con 0x7f223c147a80 2023-03-03T15:00:36.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f224147f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f223c14a2b0 0x7f223c1ebbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:36.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f224147f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f223c14a2b0 0x7f223c1ebbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51254/0 (socket says 172.21.15.7:51254) 2023-03-03T15:00:36.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f224147f700 1 -- 172.21.15.7:0/3371266283 learned_addr learned my addr 172.21.15.7:0/3371266283 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:00:36.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f2240c7e700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c147a80 0x7f223c0abba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:36.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f223bfff700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f223c1dab90 0x7f223c1eb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:36.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f224147f700 1 -- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c147a80 msgr2=0x7f223c0abba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:36.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f224147f700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c147a80 0x7f223c0abba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:36.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f224147f700 1 -- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f223c1dab90 msgr2=0x7f223c1eb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:36.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f224147f700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f223c1dab90 0x7f223c1eb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:36.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.208+0000 7f224147f700 1 -- 172.21.15.7:0/3371266283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c06c9a0 con 0x7f223c14a2b0 2023-03-03T15:00:36.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.209+0000 7f224147f700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f223c14a2b0 0x7f223c1ebbc0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f222c00ba10 tx=0x7f2234009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:00:36.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.209+0000 7f22397fa700 1 -- 172.21.15.7:0/3371266283 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2234009410 con 0x7f223c14a2b0 2023-03-03T15:00:36.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.209+0000 7f22397fa700 1 -- 172.21.15.7:0/3371266283 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2234015b30 con 0x7f223c14a2b0 2023-03-03T15:00:36.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.209+0000 7f22397fa700 1 -- 172.21.15.7:0/3371266283 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2234029430 con 0x7f223c14a2b0 2023-03-03T15:00:36.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.209+0000 7f2242ee2700 1 -- 172.21.15.7:0/3371266283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223c151300 con 0x7f223c14a2b0 2023-03-03T15:00:36.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.209+0000 7f2242ee2700 1 -- 172.21.15.7:0/3371266283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223c06cb30 con 0x7f223c14a2b0 2023-03-03T15:00:36.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.210+0000 7f2242ee2700 1 -- 172.21.15.7:0/3371266283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223c149c90 con 0x7f223c14a2b0 2023-03-03T15:00:36.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.210+0000 7f22397fa700 1 -- 172.21.15.7:0/3371266283 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2234030070 con 0x7f223c14a2b0 2023-03-03T15:00:36.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.211+0000 7f22397fa700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22200682e0 0x7f222006a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:00:36.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.211+0000 7f22397fa700 1 -- 172.21.15.7:0/3371266283 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2234024030 con 0x7f223c14a2b0 2023-03-03T15:00:36.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.211+0000 7f2240c7e700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22200682e0 0x7f222006a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:00:36.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.212+0000 7f2240c7e700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22200682e0 0x7f222006a790 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f2230009ce0 tx=0x7f2230009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:00:36.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.215+0000 7f22397fa700 1 -- 172.21.15.7:0/3371266283 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2234054040 con 0x7f223c14a2b0 2023-03-03T15:00:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.453+0000 7f2242ee2700 1 -- 172.21.15.7:0/3371266283 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f223c074bb0 con 0x7f223c14a2b0 2023-03-03T15:00:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.454+0000 7f22397fa700 1 -- 172.21.15.7:0/3371266283 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2234057660 con 0x7f223c14a2b0 2023-03-03T15:00:36.457 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:00:36.458 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:00:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.455+0000 7f2242ee2700 1 -- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22200682e0 msgr2=0x7f222006a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.455+0000 7f2242ee2700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22200682e0 0x7f222006a790 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f2230009ce0 tx=0x7f2230009210 comp rx=0 tx=0).stop 2023-03-03T15:00:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.456+0000 7f2242ee2700 1 -- 172.21.15.7:0/3371266283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f223c14a2b0 msgr2=0x7f223c1ebbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:00:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.456+0000 7f2242ee2700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f223c14a2b0 0x7f223c1ebbc0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f222c00ba10 tx=0x7f2234009750 comp rx=0 tx=0).stop 2023-03-03T15:00:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.456+0000 7f2242ee2700 1 -- 172.21.15.7:0/3371266283 shutdown_connections 2023-03-03T15:00:36.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.456+0000 7f2242ee2700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22200682e0 0x7f222006a790 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.456+0000 7f2242ee2700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f223c14a2b0 0x7f223c1ebbc0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.456+0000 7f2242ee2700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f223c147a80 0x7f223c0abba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.456+0000 7f2242ee2700 1 --2- 172.21.15.7:0/3371266283 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f223c1dab90 0x7f223c1eb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:00:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.456+0000 7f2242ee2700 1 -- 172.21.15.7:0/3371266283 >> 172.21.15.7:0/3371266283 conn(0x7f223c0a94d0 msgr2=0x7f223c0c1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:00:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.456+0000 7f2242ee2700 1 -- 172.21.15.7:0/3371266283 shutdown_connections 2023-03-03T15:00:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:00:36.456+0000 7f2242ee2700 1 -- 172.21.15.7:0/3371266283 wait complete. 2023-03-03T15:00:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:00:36.469 INFO:tasks.mds_thrash.fs.[b]:waiting for 58 secs before thrashing 2023-03-03T15:00:59.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:00:59.186 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:09.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:01:09.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.425+0000 7fdca545e700 1 Processor -- start 2023-03-03T15:01:09.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.425+0000 7fdca545e700 1 -- start start 2023-03-03T15:01:09.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.425+0000 7fdca545e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00caa30 0x7fdca00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:09.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdca545e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdca00c00e0 0x7fdca0154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:09.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdca545e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca00cb280 con 0x7fdca00cae00 2023-03-03T15:01:09.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdca545e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca00c7310 con 0x7fdca00c00e0 2023-03-03T15:01:09.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdca545e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca00bfd10 con 0x7fdca00caa30 2023-03-03T15:01:09.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9e7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdca00c00e0 0x7fdca0154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:09.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00caa30 0x7fdca00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:09.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9e7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdca00c00e0 0x7fdca0154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47050/0 (socket says 172.21.15.7:47050) 2023-03-03T15:01:09.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9f7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fdca00cae00 0x7fdca00c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:48778/0 (socket says 172.21.15.7:48778) 2023-03-03T15:01:09.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9effd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00caa30 0x7fdca00c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:59346/0 (socket says 172.21.15.7:59346) 2023-03-03T15:01:09.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9e7fc700 1 -- 172.21.15.7:0/4171284832 learned_addr learned my addr 172.21.15.7:0/4171284832 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:01:09.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9e7fc700 1 -- 172.21.15.7:0/4171284832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00caa30 msgr2=0x7fdca00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:01:09.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9e7fc700 1 --2- 172.21.15.7:0/4171284832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00caa30 0x7fdca00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:09.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9e7fc700 1 -- 172.21.15.7:0/4171284832 >> v1:172.21.15.143:6789/0 conn(0x7fdca00cae00 legacy=0x7fdca00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:09.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9e7fc700 1 -- 172.21.15.7:0/4171284832 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca00bfa30 con 0x7fdca00c00e0 2023-03-03T15:01:09.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.426+0000 7fdc9dffb700 1 -- 172.21.15.7:0/4171284832 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008354437 0 0) 0x7fdca00cb280 con 0x7fdca00cae00 2023-03-03T15:01:09.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdc9e7fc700 1 --2- 172.21.15.7:0/4171284832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdca00c00e0 0x7fdca0154ff0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdc94006fd0 tx=0x7fdc94009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b60d4f8e6f91b29f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:09.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdc9dffb700 1 -- 172.21.15.7:0/4171284832 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94005e30 con 0x7fdca00c00e0 2023-03-03T15:01:09.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdc9dffb700 1 -- 172.21.15.7:0/4171284832 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc94004030 con 0x7fdca00c00e0 2023-03-03T15:01:09.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdc9dffb700 1 -- 172.21.15.7:0/4171284832 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc940043e0 con 0x7fdca00c00e0 2023-03-03T15:01:09.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdca545e700 1 -- 172.21.15.7:0/4171284832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdca00c00e0 msgr2=0x7fdca0154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:09.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdca545e700 1 --2- 172.21.15.7:0/4171284832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdca00c00e0 0x7fdca0154ff0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdc94006fd0 tx=0x7fdc94009bd0 comp rx=0 tx=0).stop 2023-03-03T15:01:09.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdca545e700 1 -- 172.21.15.7:0/4171284832 shutdown_connections 2023-03-03T15:01:09.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdca545e700 1 --2- 172.21.15.7:0/4171284832 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00caa30 0x7fdca00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:09.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdca545e700 1 --2- 172.21.15.7:0/4171284832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdca00c00e0 0x7fdca0154ff0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:09.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdca545e700 1 -- 172.21.15.7:0/4171284832 >> 172.21.15.7:0/4171284832 conn(0x7fdca0066e40 msgr2=0x7fdca0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:01:09.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdca545e700 1 -- 172.21.15.7:0/4171284832 shutdown_connections 2023-03-03T15:01:09.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.427+0000 7fdca545e700 1 -- 172.21.15.7:0/4171284832 wait complete. 2023-03-03T15:01:09.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.429+0000 7fdca545e700 1 Processor -- start 2023-03-03T15:01:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.429+0000 7fdca545e700 1 -- start start 2023-03-03T15:01:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.429+0000 7fdca545e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdca00c00e0 0x7fdca00c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.429+0000 7fdca545e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca00caa30 0x7fdca014ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.429+0000 7fdca545e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00cae00 0x7fdca014d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.429+0000 7fdca545e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca01581d0 con 0x7fdca00caa30 2023-03-03T15:01:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.429+0000 7fdca545e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca014ad80 con 0x7fdca00c00e0 2023-03-03T15:01:09.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.429+0000 7fdca545e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca014b320 con 0x7fdca00cae00 2023-03-03T15:01:09.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.429+0000 7fdc9e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca00caa30 0x7fdca014ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:09.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc9e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca00caa30 0x7fdca014ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51282/0 (socket says 172.21.15.7:51282) 2023-03-03T15:01:09.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc9e7fc700 1 -- 172.21.15.7:0/775091041 learned_addr learned my addr 172.21.15.7:0/775091041 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:01:09.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc9effd700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdca00c00e0 0x7fdca00c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:09.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc9f7fe700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00cae00 0x7fdca014d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:09.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc9e7fc700 1 -- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00cae00 msgr2=0x7fdca014d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc9e7fc700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00cae00 0x7fdca014d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc9e7fc700 1 -- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdca00c00e0 msgr2=0x7fdca00c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc9e7fc700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdca00c00e0 0x7fdca00c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc9e7fc700 1 -- 172.21.15.7:0/775091041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc94009500 con 0x7fdca00caa30 2023-03-03T15:01:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc9e7fc700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca00caa30 0x7fdca014ccb0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009c10 tx=0x7fdc94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc877fe700 1 -- 172.21.15.7:0/775091041 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94003dd0 con 0x7fdca00caa30 2023-03-03T15:01:09.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdca545e700 1 -- 172.21.15.7:0/775091041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca01547f0 con 0x7fdca00caa30 2023-03-03T15:01:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc877fe700 1 -- 172.21.15.7:0/775091041 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9401c420 con 0x7fdca00caa30 2023-03-03T15:01:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdc877fe700 1 -- 172.21.15.7:0/775091041 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9401c7d0 con 0x7fdca00caa30 2023-03-03T15:01:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.430+0000 7fdca545e700 1 -- 172.21.15.7:0/775091041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca0154510 con 0x7fdca00caa30 2023-03-03T15:01:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.431+0000 7fdca545e700 1 -- 172.21.15.7:0/775091041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca0154ad0 con 0x7fdca00caa30 2023-03-03T15:01:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.432+0000 7fdc877fe700 1 -- 172.21.15.7:0/775091041 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc9402c070 con 0x7fdca00caa30 2023-03-03T15:01:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.432+0000 7fdc877fe700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc8c0682f0 0x7fdc8c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.432+0000 7fdc877fe700 1 -- 172.21.15.7:0/775091041 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdc94087280 con 0x7fdca00caa30 2023-03-03T15:01:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.432+0000 7fdc9effd700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc8c0682f0 0x7fdc8c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.433+0000 7fdc9effd700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc8c0682f0 0x7fdc8c06a7a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fdc90009b70 tx=0x7fdc90008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.437+0000 7fdc877fe700 1 -- 172.21.15.7:0/775091041 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc940535a0 con 0x7fdca00caa30 2023-03-03T15:01:09.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.676+0000 7fdca545e700 1 -- 172.21.15.7:0/775091041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdca00c4dd0 con 0x7fdca00caa30 2023-03-03T15:01:09.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.676+0000 7fdc877fe700 1 -- 172.21.15.7:0/775091041 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdc94056bc0 con 0x7fdca00caa30 2023-03-03T15:01:09.678 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:01:09.679 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:01:09.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.678+0000 7fdca545e700 1 -- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc8c0682f0 msgr2=0x7fdc8c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:09.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.678+0000 7fdca545e700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc8c0682f0 0x7fdc8c06a7a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fdc90009b70 tx=0x7fdc90008040 comp rx=0 tx=0).stop 2023-03-03T15:01:09.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.678+0000 7fdca545e700 1 -- 172.21.15.7:0/775091041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca00caa30 msgr2=0x7fdca014ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:09.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.678+0000 7fdca545e700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca00caa30 0x7fdca014ccb0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009c10 tx=0x7fdc94004750 comp rx=0 tx=0).stop 2023-03-03T15:01:09.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.678+0000 7fdca545e700 1 -- 172.21.15.7:0/775091041 shutdown_connections 2023-03-03T15:01:09.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.678+0000 7fdca545e700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdca00cae00 0x7fdca014d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:09.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.678+0000 7fdca545e700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdca00c00e0 0x7fdca00c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:09.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.678+0000 7fdca545e700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc8c0682f0 0x7fdc8c06a7a0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:09.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.678+0000 7fdca545e700 1 --2- 172.21.15.7:0/775091041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdca00caa30 0x7fdca014ccb0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:09.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.678+0000 7fdca545e700 1 -- 172.21.15.7:0/775091041 >> 172.21.15.7:0/775091041 conn(0x7fdca0066e40 msgr2=0x7fdca01523c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:01:09.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.679+0000 7fdca545e700 1 -- 172.21.15.7:0/775091041 shutdown_connections 2023-03-03T15:01:09.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:09.679+0000 7fdca545e700 1 -- 172.21.15.7:0/775091041 wait complete. 2023-03-03T15:01:09.683 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:01:09.692 INFO:tasks.mds_thrash.fs.[a]:waiting for 19 secs before thrashing 2023-03-03T15:01:28.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:01:28.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.830+0000 7f95da979700 1 Processor -- start 2023-03-03T15:01:28.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.830+0000 7f95da979700 1 -- start start 2023-03-03T15:01:28.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.831+0000 7f95da979700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 0x7f95d4158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:28.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.831+0000 7f95da979700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d4151ef0 con 0x7f95d4151b20 2023-03-03T15:01:28.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.831+0000 7f95da979700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d4157b20 con 0x7f95d40aa010 2023-03-03T15:01:28.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.831+0000 7f95da979700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4155410 con 0x7f95d4154f10 2023-03-03T15:01:28.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.831+0000 7f95d3fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f95d4151b20 0x7f95d4071820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45280/0 (socket says 172.21.15.7:45280) 2023-03-03T15:01:28.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.831+0000 7f95d3fff700 1 -- 172.21.15.7:0/1265646765 learned_addr learned my addr 172.21.15.7:0/1265646765 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:01:28.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.831+0000 7f95d37fe700 1 --2- 172.21.15.7:0/1265646765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 0x7f95d4158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:28.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.831+0000 7f95d2ffd700 1 -- 172.21.15.7:0/1265646765 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671953718 0 0) 0x7f95d4151ef0 con 0x7f95d4151b20 2023-03-03T15:01:28.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.831+0000 7f95d2ffd700 1 -- 172.21.15.7:0/1265646765 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c00035b0 con 0x7f95d4151b20 2023-03-03T15:01:28.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.832+0000 7f95d2ffd700 1 -- 172.21.15.7:0/1265646765 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763799598 0 0) 0x7f95d4157b20 con 0x7f95d40aa010 2023-03-03T15:01:28.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.832+0000 7f95d2ffd700 1 -- 172.21.15.7:0/1265646765 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d4151ef0 con 0x7f95d40aa010 2023-03-03T15:01:28.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.832+0000 7f95d37fe700 1 -- 172.21.15.7:0/1265646765 >> v1:172.21.15.143:6789/0 conn(0x7f95d40aa010 legacy=0x7f95d415ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:28.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.832+0000 7f95d37fe700 1 -- 172.21.15.7:0/1265646765 >> v1:172.21.15.7:6789/0 conn(0x7f95d4151b20 legacy=0x7f95d4071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:28.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.832+0000 7f95d37fe700 1 -- 172.21.15.7:0/1265646765 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d406f640 con 0x7f95d4154f10 2023-03-03T15:01:28.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.832+0000 7f95d37fe700 1 --2- 172.21.15.7:0/1265646765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 0x7f95d4158000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009870 tx=0x7f95bc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e454277c108b9a30 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:28.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.832+0000 7f95d2ffd700 1 -- 172.21.15.7:0/1265646765 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc011830 con 0x7f95d4154f10 2023-03-03T15:01:28.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.832+0000 7f95d2ffd700 1 -- 172.21.15.7:0/1265646765 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95bc011990 con 0x7f95d4154f10 2023-03-03T15:01:28.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.832+0000 7f95d2ffd700 1 -- 172.21.15.7:0/1265646765 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc011d20 con 0x7f95d4154f10 2023-03-03T15:01:28.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.833+0000 7f95da979700 1 -- 172.21.15.7:0/1265646765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 msgr2=0x7f95d4158000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:28.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.833+0000 7f95da979700 1 --2- 172.21.15.7:0/1265646765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 0x7f95d4158000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009870 tx=0x7f95bc009b80 comp rx=0 tx=0).stop 2023-03-03T15:01:28.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.833+0000 7f95da979700 1 -- 172.21.15.7:0/1265646765 shutdown_connections 2023-03-03T15:01:28.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.833+0000 7f95da979700 1 --2- 172.21.15.7:0/1265646765 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 0x7f95d4158000 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:28.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.833+0000 7f95da979700 1 -- 172.21.15.7:0/1265646765 >> 172.21.15.7:0/1265646765 conn(0x7f95d40c8900 msgr2=0x7f95d40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:01:28.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.833+0000 7f95da979700 1 -- 172.21.15.7:0/1265646765 shutdown_connections 2023-03-03T15:01:28.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.833+0000 7f95da979700 1 -- 172.21.15.7:0/1265646765 wait complete. 2023-03-03T15:01:28.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.834+0000 7f95da979700 1 Processor -- start 2023-03-03T15:01:28.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.834+0000 7f95da979700 1 -- start start 2023-03-03T15:01:28.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95da979700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95d40aa010 0x7f95d40c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:28.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95da979700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95d4151b20 0x7f95d413fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:28.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95da979700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 0x7f95d4140260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:28.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95da979700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d41407a0 con 0x7f95d40aa010 2023-03-03T15:01:28.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95da979700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d41556e0 con 0x7f95d4151b20 2023-03-03T15:01:28.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95da979700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4155930 con 0x7f95d4154f10 2023-03-03T15:01:28.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95d40aa010 0x7f95d40c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:28.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95d40aa010 0x7f95d40c75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51306/0 (socket says 172.21.15.7:51306) 2023-03-03T15:01:28.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d3fff700 1 -- 172.21.15.7:0/159884450 learned_addr learned my addr 172.21.15.7:0/159884450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:01:28.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d37fe700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95d4151b20 0x7f95d413fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:28.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d8f16700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 0x7f95d4140260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:28.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d3fff700 1 -- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 msgr2=0x7f95d4140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:28.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d3fff700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 0x7f95d4140260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:28.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d3fff700 1 -- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95d4151b20 msgr2=0x7f95d413fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:28.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d3fff700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95d4151b20 0x7f95d413fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:28.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d3fff700 1 -- 172.21.15.7:0/159884450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc0094a0 con 0x7f95d40aa010 2023-03-03T15:01:28.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.835+0000 7f95d3fff700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95d40aa010 0x7f95d40c75d0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f95c400c700 tx=0x7f95c400cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:28.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.836+0000 7f95d0ff9700 1 -- 172.21.15.7:0/159884450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4005d70 con 0x7f95d40aa010 2023-03-03T15:01:28.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.836+0000 7f95d0ff9700 1 -- 172.21.15.7:0/159884450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c40070d0 con 0x7f95d40aa010 2023-03-03T15:01:28.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.836+0000 7f95da979700 1 -- 172.21.15.7:0/159884450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d406fc00 con 0x7f95d40aa010 2023-03-03T15:01:28.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.836+0000 7f95d0ff9700 1 -- 172.21.15.7:0/159884450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4007480 con 0x7f95d40aa010 2023-03-03T15:01:28.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.836+0000 7f95da979700 1 -- 172.21.15.7:0/159884450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d406f920 con 0x7f95d40aa010 2023-03-03T15:01:28.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.837+0000 7f95da979700 1 -- 172.21.15.7:0/159884450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d4071f30 con 0x7f95d40aa010 2023-03-03T15:01:28.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.837+0000 7f95d0ff9700 1 -- 172.21.15.7:0/159884450 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f95c4019030 con 0x7f95d40aa010 2023-03-03T15:01:28.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.837+0000 7f95d0ff9700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95b40682f0 0x7f95b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:28.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.838+0000 7f95d0ff9700 1 -- 172.21.15.7:0/159884450 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f95c4085700 con 0x7f95d40aa010 2023-03-03T15:01:28.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.838+0000 7f95d37fe700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95b40682f0 0x7f95b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:28.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.839+0000 7f95d37fe700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95b40682f0 0x7f95b406a7a0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f95d4155340 tx=0x7f95bc01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:28.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:28.843+0000 7f95d0ff9700 1 -- 172.21.15.7:0/159884450 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f95c40519c0 con 0x7f95d40aa010 2023-03-03T15:01:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.084+0000 7f95da979700 1 -- 172.21.15.7:0/159884450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95d4074bb0 con 0x7f95d40aa010 2023-03-03T15:01:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.085+0000 7f95d0ff9700 1 -- 172.21.15.7:0/159884450 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f95c4054fe0 con 0x7f95d40aa010 2023-03-03T15:01:29.088 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:01:29.088 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:01:29.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.086+0000 7f95da979700 1 -- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95b40682f0 msgr2=0x7f95b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:29.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.086+0000 7f95da979700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95b40682f0 0x7f95b406a7a0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f95d4155340 tx=0x7f95bc01c040 comp rx=0 tx=0).stop 2023-03-03T15:01:29.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.086+0000 7f95da979700 1 -- 172.21.15.7:0/159884450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95d40aa010 msgr2=0x7f95d40c75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:29.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.086+0000 7f95da979700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95d40aa010 0x7f95d40c75d0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f95c400c700 tx=0x7f95c400cac0 comp rx=0 tx=0).stop 2023-03-03T15:01:29.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.087+0000 7f95da979700 1 -- 172.21.15.7:0/159884450 shutdown_connections 2023-03-03T15:01:29.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.087+0000 7f95da979700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f95d4154f10 0x7f95d4140260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:29.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.087+0000 7f95da979700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95b40682f0 0x7f95b406a7a0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:29.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.087+0000 7f95da979700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95d40aa010 0x7f95d40c75d0 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:29.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.087+0000 7f95da979700 1 --2- 172.21.15.7:0/159884450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95d4151b20 0x7f95d413fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:29.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.087+0000 7f95da979700 1 -- 172.21.15.7:0/159884450 >> 172.21.15.7:0/159884450 conn(0x7f95d40c8900 msgr2=0x7f95d4157f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:01:29.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.087+0000 7f95da979700 1 -- 172.21.15.7:0/159884450 shutdown_connections 2023-03-03T15:01:29.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:29.087+0000 7f95da979700 1 -- 172.21.15.7:0/159884450 wait complete. 2023-03-03T15:01:29.093 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:01:29.106 INFO:tasks.mds_thrash.fs.[a]:waiting for 115 secs before thrashing 2023-03-03T15:01:29.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:29.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:34.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:01:34.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.608+0000 7f0a8f3b0700 1 Processor -- start 2023-03-03T15:01:34.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.608+0000 7f0a8f3b0700 1 -- start start 2023-03-03T15:01:34.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a8f3b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a8806c6c0 0x7f0a880736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a8f3b0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a88157ff0 con 0x7f0a880c2e30 2023-03-03T15:01:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a8f3b0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a8806d640 con 0x7f0a88151cd0 2023-03-03T15:01:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a8f3b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8806cda0 con 0x7f0a8806c6c0 2023-03-03T15:01:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a8c94b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0a880c2e30 0x7f0a88158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45300/0 (socket says 172.21.15.7:45300) 2023-03-03T15:01:34.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a8c94b700 1 -- 172.21.15.7:0/888823887 learned_addr learned my addr 172.21.15.7:0/888823887 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:01:34.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736652689 0 0) 0x7f0a88157ff0 con 0x7f0a880c2e30 2023-03-03T15:01:34.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a8d94d700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a8806c6c0 0x7f0a880736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:34.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a740035b0 con 0x7f0a880c2e30 2023-03-03T15:01:34.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 879815867 0 0) 0x7f0a740035b0 con 0x7f0a880c2e30 2023-03-03T15:01:34.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a8806c6c0 msgr2=0x7f0a880736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:01:34.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a8806c6c0 0x7f0a880736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:34.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 >> v1:172.21.15.143:6789/0 conn(0x7f0a88151cd0 legacy=0x7f0a88154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:34.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8806d080 con 0x7f0a880c2e30 2023-03-03T15:01:34.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0a78001670 con 0x7f0a880c2e30 2023-03-03T15:01:34.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 >> v1:172.21.15.7:6789/0 conn(0x7f0a880c2e30 legacy=0x7f0a88158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:34.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74003aa0 0x7f0a74003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:34.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74004850 0x7f0a74004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:34.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a8806c6c0 0x7f0a7400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:34.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8806cda0 con 0x7f0a74004850 2023-03-03T15:01:34.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a74005530 con 0x7f0a74003aa0 2023-03-03T15:01:34.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.609+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740056f0 con 0x7f0a8806c6c0 2023-03-03T15:01:34.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a8c94b700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74004850 0x7f0a74004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:34.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a8d94d700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a8806c6c0 0x7f0a7400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:34.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a8d14c700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74003aa0 0x7f0a74003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:34.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a8c94b700 1 -- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a8806c6c0 msgr2=0x7f0a7400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:01:34.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a8c94b700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a8806c6c0 0x7f0a7400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:34.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a8c94b700 1 -- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74003aa0 msgr2=0x7f0a74003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:01:34.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a8c94b700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a74003aa0 0x7f0a74003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:34.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a8c94b700 1 -- 172.21.15.7:0/888823887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a740035b0 con 0x7f0a74004850 2023-03-03T15:01:34.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a8c94b700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74004850 0x7f0a74004cc0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800c240 tx=0x7f0a74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e2c793dfb1d8aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:34.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78007c10 con 0x7f0a74004850 2023-03-03T15:01:34.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7800cde0 con 0x7f0a74004850 2023-03-03T15:01:34.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.610+0000 7f0a7ffff700 1 -- 172.21.15.7:0/888823887 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78005ba0 con 0x7f0a74004850 2023-03-03T15:01:34.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.611+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/888823887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74004850 msgr2=0x7f0a74004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:34.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.611+0000 7f0a8f3b0700 1 --2- 172.21.15.7:0/888823887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a74004850 0x7f0a74004cc0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800c240 tx=0x7f0a74005880 comp rx=0 tx=0).stop 2023-03-03T15:01:34.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.611+0000 7f0a8d94d700 1 -- 172.21.15.7:0/888823887 reap_dead start 2023-03-03T15:01:34.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.611+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/888823887 shutdown_connections 2023-03-03T15:01:34.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.611+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/888823887 >> 172.21.15.7:0/888823887 conn(0x7f0a880a94d0 msgr2=0x7f0a880a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:01:34.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.611+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/888823887 shutdown_connections 2023-03-03T15:01:34.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.611+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/888823887 wait complete. 2023-03-03T15:01:34.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.612+0000 7f0a8f3b0700 1 Processor -- start 2023-03-03T15:01:34.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.612+0000 7f0a8f3b0700 1 -- start start 2023-03-03T15:01:34.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8f3b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a74004850 0x7f0a88154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:34.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8f3b0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a881578a0 0x7f0a8814d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:34.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8f3b0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a881ea850 0x7f0a8814b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:34.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8f3b0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a88156aa0 con 0x7f0a881578a0 2023-03-03T15:01:34.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8f3b0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a88156740 con 0x7f0a881ea850 2023-03-03T15:01:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8f3b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8814b4b0 con 0x7f0a74004850 2023-03-03T15:01:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8c94b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a881578a0 0x7f0a8814d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8c94b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a881578a0 0x7f0a8814d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51332/0 (socket says 172.21.15.7:51332) 2023-03-03T15:01:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8c94b700 1 -- 172.21.15.7:0/200930189 learned_addr learned my addr 172.21.15.7:0/200930189 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:01:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8d94d700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a881ea850 0x7f0a8814b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8d14c700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a74004850 0x7f0a88154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8c94b700 1 -- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a74004850 msgr2=0x7f0a88154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:34.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8c94b700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a74004850 0x7f0a88154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:34.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8c94b700 1 -- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a881ea850 msgr2=0x7f0a8814b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:34.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8c94b700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a881ea850 0x7f0a8814b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:34.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.613+0000 7f0a8c94b700 1 -- 172.21.15.7:0/200930189 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a88157c80 con 0x7f0a881578a0 2023-03-03T15:01:34.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.614+0000 7f0a8c94b700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a881578a0 0x7f0a8814d940 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800cfd0 tx=0x7f0a78001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:34.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.614+0000 7f0a7dffb700 1 -- 172.21.15.7:0/200930189 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78001430 con 0x7f0a881578a0 2023-03-03T15:01:34.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.614+0000 7f0a7dffb700 1 -- 172.21.15.7:0/200930189 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a780015b0 con 0x7f0a881578a0 2023-03-03T15:01:34.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.614+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/200930189 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8806d360 con 0x7f0a881578a0 2023-03-03T15:01:34.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.614+0000 7f0a7dffb700 1 -- 172.21.15.7:0/200930189 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78007310 con 0x7f0a881578a0 2023-03-03T15:01:34.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.614+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/200930189 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a880736f0 con 0x7f0a881578a0 2023-03-03T15:01:34.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.615+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/200930189 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a88074bb0 con 0x7f0a881578a0 2023-03-03T15:01:34.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.616+0000 7f0a7dffb700 1 -- 172.21.15.7:0/200930189 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0a78010070 con 0x7f0a881578a0 2023-03-03T15:01:34.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.616+0000 7f0a7dffb700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a6c0682f0 0x7f0a6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:34.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.616+0000 7f0a7dffb700 1 -- 172.21.15.7:0/200930189 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0a78087940 con 0x7f0a881578a0 2023-03-03T15:01:34.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.616+0000 7f0a8d14c700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a6c0682f0 0x7f0a6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:34.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.617+0000 7f0a8d14c700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a6c0682f0 0x7f0a6c06a7a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f0a8814c470 tx=0x7f0a84009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:34.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.620+0000 7f0a7dffb700 1 -- 172.21.15.7:0/200930189 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0a78053c60 con 0x7f0a881578a0 2023-03-03T15:01:34.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.854+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/200930189 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a880c36b0 con 0x7f0a881578a0 2023-03-03T15:01:34.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.855+0000 7f0a7dffb700 1 -- 172.21.15.7:0/200930189 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0a78057280 con 0x7f0a881578a0 2023-03-03T15:01:34.858 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:01:34.858 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:01:34.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.856+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a6c0682f0 msgr2=0x7f0a6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:34.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.856+0000 7f0a8f3b0700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a6c0682f0 0x7f0a6c06a7a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f0a8814c470 tx=0x7f0a84009040 comp rx=0 tx=0).stop 2023-03-03T15:01:34.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.856+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/200930189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a881578a0 msgr2=0x7f0a8814d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:34.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.856+0000 7f0a8f3b0700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a881578a0 0x7f0a8814d940 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800cfd0 tx=0x7f0a78001a00 comp rx=0 tx=0).stop 2023-03-03T15:01:34.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.856+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/200930189 shutdown_connections 2023-03-03T15:01:34.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.856+0000 7f0a8f3b0700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a881ea850 0x7f0a8814b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:34.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.856+0000 7f0a8f3b0700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a74004850 0x7f0a88154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:34.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.856+0000 7f0a8f3b0700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a6c0682f0 0x7f0a6c06a7a0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:34.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.856+0000 7f0a8f3b0700 1 --2- 172.21.15.7:0/200930189 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a881578a0 0x7f0a8814d940 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:34.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.857+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/200930189 >> 172.21.15.7:0/200930189 conn(0x7f0a880a94d0 msgr2=0x7f0a88071f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:01:34.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.857+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/200930189 shutdown_connections 2023-03-03T15:01:34.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:34.857+0000 7f0a8f3b0700 1 -- 172.21.15.7:0/200930189 wait complete. 2023-03-03T15:01:34.862 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:01:34.871 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before thrashing 2023-03-03T15:01:38.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:01:39.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.010+0000 7f8b0dfdf700 1 Processor -- start 2023-03-03T15:01:39.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.010+0000 7f8b0dfdf700 1 -- start start 2023-03-03T15:01:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.011+0000 7f8b0dfdf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8b08155410 0x7f8b080bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.011+0000 7f8b0dfdf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b080c5b00 0x7f8b080bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.011+0000 7f8b0dfdf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b080c08d0 con 0x7f8b08151e80 2023-03-03T15:01:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.011+0000 7f8b0dfdf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b08152250 con 0x7f8b080c5b00 2023-03-03T15:01:39.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.011+0000 7f8b0dfdf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b080c0fb0 con 0x7f8b08155410 2023-03-03T15:01:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.011+0000 7f8b07fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b080c5b00 0x7f8b080bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.011+0000 7f8b07fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b080c5b00 0x7f8b080bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51342/0 (socket says 172.21.15.7:51342) 2023-03-03T15:01:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.011+0000 7f8b07fff700 1 -- 172.21.15.7:0/3436998733 learned_addr learned my addr 172.21.15.7:0/3436998733 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:01:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.011+0000 7f8b06ffd700 1 --2- 172.21.15.7:0/3436998733 >> v2:172.21.15.143:3300/0 conn(0x7f8b08155410 0x7f8b080bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.012+0000 7f8b07fff700 1 -- 172.21.15.7:0/3436998733 >> v2:172.21.15.143:3300/0 conn(0x7f8b08155410 msgr2=0x7f8b080bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:01:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.012+0000 7f8b07fff700 1 --2- 172.21.15.7:0/3436998733 >> v2:172.21.15.143:3300/0 conn(0x7f8b08155410 0x7f8b080bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:39.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.012+0000 7f8b07fff700 1 -- 172.21.15.7:0/3436998733 >> v1:172.21.15.143:6789/0 conn(0x7f8b08151e80 legacy=0x7f8b080c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.012+0000 7f8b07fff700 1 -- 172.21.15.7:0/3436998733 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b08154930 con 0x7f8b080c5b00 2023-03-03T15:01:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.012+0000 7f8b07fff700 1 --2- 172.21.15.7:0/3436998733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b080c5b00 0x7f8b080bfe20 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f8b0806c380 tx=0x7f8b0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55291008a11459e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.012+0000 7f8b067fc700 1 -- 172.21.15.7:0/3436998733 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0000e070 con 0x7f8b080c5b00 2023-03-03T15:01:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.012+0000 7f8b067fc700 1 -- 172.21.15.7:0/3436998733 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0000b910 con 0x7f8b080c5b00 2023-03-03T15:01:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.012+0000 7f8b067fc700 1 -- 172.21.15.7:0/3436998733 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0000bcc0 con 0x7f8b080c5b00 2023-03-03T15:01:39.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.012+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3436998733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b080c5b00 msgr2=0x7f8b080bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:39.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.013+0000 7f8b0dfdf700 1 --2- 172.21.15.7:0/3436998733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b080c5b00 0x7f8b080bfe20 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f8b0806c380 tx=0x7f8b0000b330 comp rx=0 tx=0).stop 2023-03-03T15:01:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.013+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3436998733 shutdown_connections 2023-03-03T15:01:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.013+0000 7f8b0dfdf700 1 --2- 172.21.15.7:0/3436998733 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b080c5b00 0x7f8b080bfe20 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.013+0000 7f8b0dfdf700 1 --2- 172.21.15.7:0/3436998733 >> v2:172.21.15.143:3300/0 conn(0x7f8b08155410 0x7f8b080bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.013+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3436998733 >> 172.21.15.7:0/3436998733 conn(0x7f8b08071630 msgr2=0x7f8b0806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:01:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.013+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3436998733 shutdown_connections 2023-03-03T15:01:39.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.013+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3436998733 wait complete. 2023-03-03T15:01:39.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.014+0000 7f8b0dfdf700 1 Processor -- start 2023-03-03T15:01:39.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.014+0000 7f8b0dfdf700 1 -- start start 2023-03-03T15:01:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b0dfdf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b080c5b00 0x7f8b080ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b0dfdf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b08151e80 0x7f8b08150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b0dfdf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b08155410 0x7f8b0814f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b0dfdf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b080c8c10 con 0x7f8b08155410 2023-03-03T15:01:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b0dfdf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b080ca580 con 0x7f8b080c5b00 2023-03-03T15:01:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b0dfdf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0814eca0 con 0x7f8b08151e80 2023-03-03T15:01:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b07fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b08155410 0x7f8b0814f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b07fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b08155410 0x7f8b0814f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51348/0 (socket says 172.21.15.7:51348) 2023-03-03T15:01:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b07fff700 1 -- 172.21.15.7:0/3791532740 learned_addr learned my addr 172.21.15.7:0/3791532740 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:01:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b077fe700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b080c5b00 0x7f8b080ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b06ffd700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b08151e80 0x7f8b08150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b07fff700 1 -- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b08151e80 msgr2=0x7f8b08150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b07fff700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b08151e80 0x7f8b08150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b07fff700 1 -- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b080c5b00 msgr2=0x7f8b080ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b07fff700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b080c5b00 0x7f8b080ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:39.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b07fff700 1 -- 172.21.15.7:0/3791532740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b00009b30 con 0x7f8b08155410 2023-03-03T15:01:39.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b06ffd700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b08151e80 0x7f8b08150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:01:39.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.015+0000 7f8b07fff700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b08155410 0x7f8b0814f540 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f8b080cb4b0 tx=0x7f8b00004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:39.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.016+0000 7f8aeffff700 1 -- 172.21.15.7:0/3791532740 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b00004930 con 0x7f8b08155410 2023-03-03T15:01:39.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.016+0000 7f8aeffff700 1 -- 172.21.15.7:0/3791532740 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b000099b0 con 0x7f8b08155410 2023-03-03T15:01:39.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.016+0000 7f8aeffff700 1 -- 172.21.15.7:0/3791532740 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0001c680 con 0x7f8b08155410 2023-03-03T15:01:39.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.016+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3791532740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b08154ef0 con 0x7f8b08155410 2023-03-03T15:01:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.016+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3791532740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b08154c10 con 0x7f8b08155410 2023-03-03T15:01:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.017+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3791532740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b080c5220 con 0x7f8b08155410 2023-03-03T15:01:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.017+0000 7f8aeffff700 1 -- 172.21.15.7:0/3791532740 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b00022030 con 0x7f8b08155410 2023-03-03T15:01:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.017+0000 7f8aeffff700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 0x7f8af406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:01:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.018+0000 7f8aeffff700 1 -- 172.21.15.7:0/3791532740 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8b00087940 con 0x7f8b08155410 2023-03-03T15:01:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.018+0000 7f8b077fe700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 0x7f8af406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:01:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.018+0000 7f8b077fe700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 0x7f8af406a7a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8b08150e10 tx=0x7f8af8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:01:39.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.022+0000 7f8aeffff700 1 -- 172.21.15.7:0/3791532740 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b00053ce0 con 0x7f8b08155410 2023-03-03T15:01:39.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.258+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3791532740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b080c85a0 con 0x7f8b08155410 2023-03-03T15:01:39.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.258+0000 7f8aeffff700 1 -- 172.21.15.7:0/3791532740 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8b00057300 con 0x7f8b08155410 2023-03-03T15:01:39.261 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:01:39.261 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:01:39.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 msgr2=0x7f8af406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:39.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 0x7f8af406a7a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8b08150e10 tx=0x7f8af8009040 comp rx=0 tx=0).stop 2023-03-03T15:01:39.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3791532740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b08155410 msgr2=0x7f8b0814f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:01:39.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b08155410 0x7f8b0814f540 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f8b080cb4b0 tx=0x7f8b00004ad0 comp rx=0 tx=0).stop 2023-03-03T15:01:39.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3791532740 shutdown_connections 2023-03-03T15:01:39.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 0x7f8af406a7a0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:39.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b08155410 0x7f8b0814f540 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:39.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b080c5b00 0x7f8b080ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:39.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 --2- 172.21.15.7:0/3791532740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b08151e80 0x7f8b08150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:01:39.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3791532740 >> 172.21.15.7:0/3791532740 conn(0x7f8b08071630 msgr2=0x7f8b080bf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:01:39.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3791532740 shutdown_connections 2023-03-03T15:01:39.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:01:39.260+0000 7f8b0dfdf700 1 -- 172.21.15.7:0/3791532740 wait complete. 2023-03-03T15:01:39.267 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:01:39.274 INFO:tasks.mds_thrash.fs.[b]:waiting for 65 secs before thrashing 2023-03-03T15:01:59.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:01:59.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:29.398 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:29.403 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:44.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:02:44.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.414+0000 7f236c502700 1 Processor -- start 2023-03-03T15:02:44.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.414+0000 7f236c502700 1 -- start start 2023-03-03T15:02:44.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236c502700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f236406c480 0x7f23641581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:44.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236c502700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236406d430 con 0x7f2364151d40 2023-03-03T15:02:44.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236c502700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23641586e0 con 0x7f23640c2930 2023-03-03T15:02:44.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236c502700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236406c2a0 con 0x7f236406c480 2023-03-03T15:02:44.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236a29e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2364151d40 0x7f23640736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45364/0 (socket says 172.21.15.7:45364) 2023-03-03T15:02:44.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236a29e700 1 -- 172.21.15.7:0/1177083305 learned_addr learned my addr 172.21.15.7:0/1177083305 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:02:44.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957898199 0 0) 0x7f236406d430 con 0x7f2364151d40 2023-03-03T15:02:44.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236aa9f700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f236406c480 0x7f23641581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:44.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23500035b0 con 0x7f2364151d40 2023-03-03T15:02:44.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 517248179 0 0) 0x7f23500035b0 con 0x7f2364151d40 2023-03-03T15:02:44.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f236406c480 msgr2=0x7f23641581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:02:44.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236929c700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f236406c480 0x7f23641581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:44.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 >> v1:172.21.15.143:6789/0 conn(0x7f23640c2930 legacy=0x7f2364157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:44.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236406cb00 con 0x7f2364151d40 2023-03-03T15:02:44.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.415+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f2360001670 con 0x7f2364151d40 2023-03-03T15:02:44.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 >> v1:172.21.15.7:6789/0 conn(0x7f2364151d40 legacy=0x7f23640736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:44.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236929c700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2350003aa0 0x7f2350003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:44.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236929c700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f236406c480 0x7f2350004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:44.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236929c700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2350004d90 0x7f2350005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:44.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236406c2a0 con 0x7f2350003aa0 2023-03-03T15:02:44.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235000bcc0 con 0x7f2350004d90 2023-03-03T15:02:44.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235000be80 con 0x7f236406c480 2023-03-03T15:02:44.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236a29e700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2350003aa0 0x7f2350003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:44.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f2369a9d700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f236406c480 0x7f2350004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236aa9f700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2350004d90 0x7f2350005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236a29e700 1 -- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f236406c480 msgr2=0x7f2350004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:02:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236a29e700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f236406c480 0x7f2350004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236a29e700 1 -- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2350004d90 msgr2=0x7f2350005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:02:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236a29e700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2350004d90 0x7f2350005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236a29e700 1 -- 172.21.15.7:0/1177083305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23500035b0 con 0x7f2350003aa0 2023-03-03T15:02:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236a29e700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2350003aa0 0x7f2350003eb0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f236000c240 tx=0x7f2350005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232e8954fbbad8f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:02:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360007c10 con 0x7f2350003aa0 2023-03-03T15:02:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236000cde0 con 0x7f2350003aa0 2023-03-03T15:02:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.416+0000 7f236929c700 1 -- 172.21.15.7:0/1177083305 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360005ba0 con 0x7f2350003aa0 2023-03-03T15:02:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.417+0000 7f236c502700 1 -- 172.21.15.7:0/1177083305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2350003aa0 msgr2=0x7f2350003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.417+0000 7f236c502700 1 --2- 172.21.15.7:0/1177083305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2350003aa0 0x7f2350003eb0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f236000c240 tx=0x7f2350005880 comp rx=0 tx=0).stop 2023-03-03T15:02:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.417+0000 7f236aa9f700 1 -- 172.21.15.7:0/1177083305 reap_dead start 2023-03-03T15:02:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.417+0000 7f236c502700 1 -- 172.21.15.7:0/1177083305 shutdown_connections 2023-03-03T15:02:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.417+0000 7f236c502700 1 -- 172.21.15.7:0/1177083305 >> 172.21.15.7:0/1177083305 conn(0x7f23640a94d0 msgr2=0x7f23640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:02:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.417+0000 7f236c502700 1 -- 172.21.15.7:0/1177083305 shutdown_connections 2023-03-03T15:02:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.417+0000 7f236c502700 1 -- 172.21.15.7:0/1177083305 wait complete. 2023-03-03T15:02:44.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.419+0000 7f236c502700 1 Processor -- start 2023-03-03T15:02:44.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.419+0000 7f236c502700 1 -- start start 2023-03-03T15:02:44.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236c502700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2350003aa0 0x7f236414e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236c502700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2364071850 0x7f23641ed0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236c502700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f236414ece0 0x7f23641ed5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236c502700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23640ad0f0 con 0x7f236414ece0 2023-03-03T15:02:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236c502700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23640bf920 con 0x7f2350003aa0 2023-03-03T15:02:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236c502700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23640c0780 con 0x7f2364071850 2023-03-03T15:02:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236aa9f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f236414ece0 0x7f23641ed5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236aa9f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f236414ece0 0x7f23641ed5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51400/0 (socket says 172.21.15.7:51400) 2023-03-03T15:02:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236aa9f700 1 -- 172.21.15.7:0/2987767453 learned_addr learned my addr 172.21.15.7:0/2987767453 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:02:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f2369a9d700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2364071850 0x7f23641ed0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236a29e700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2350003aa0 0x7f236414e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236aa9f700 1 -- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2364071850 msgr2=0x7f23641ed0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236aa9f700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2364071850 0x7f23641ed0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236aa9f700 1 -- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2350003aa0 msgr2=0x7f236414e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236aa9f700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2350003aa0 0x7f236414e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236aa9f700 1 -- 172.21.15.7:0/2987767453 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236406d0c0 con 0x7f236414ece0 2023-03-03T15:02:44.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f2369a9d700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2364071850 0x7f23641ed0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:02:44.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236a29e700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2350003aa0 0x7f236414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:02:44.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236aa9f700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f236414ece0 0x7f23641ed5e0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f236406bcf0 tx=0x7f235c011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:02:44.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f235affd700 1 -- 172.21.15.7:0/2987767453 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c00e040 con 0x7f236414ece0 2023-03-03T15:02:44.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f235affd700 1 -- 172.21.15.7:0/2987767453 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c00f040 con 0x7f236414ece0 2023-03-03T15:02:44.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f236c502700 1 -- 172.21.15.7:0/2987767453 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236406cde0 con 0x7f236414ece0 2023-03-03T15:02:44.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.420+0000 7f235affd700 1 -- 172.21.15.7:0/2987767453 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c00fd50 con 0x7f236414ece0 2023-03-03T15:02:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.421+0000 7f236c502700 1 -- 172.21.15.7:0/2987767453 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2364073e00 con 0x7f236414ece0 2023-03-03T15:02:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.421+0000 7f236c502700 1 -- 172.21.15.7:0/2987767453 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23640c0c10 con 0x7f236414ece0 2023-03-03T15:02:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.422+0000 7f235affd700 1 -- 172.21.15.7:0/2987767453 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f235c015430 con 0x7f236414ece0 2023-03-03T15:02:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.422+0000 7f235affd700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23480682f0 0x7f234806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:02:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.423+0000 7f235affd700 1 -- 172.21.15.7:0/2987767453 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f235c086780 con 0x7f236414ece0 2023-03-03T15:02:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.423+0000 7f236a29e700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23480682f0 0x7f234806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:02:44.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.423+0000 7f236a29e700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23480682f0 0x7f234806a7a0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f2360000c00 tx=0x7f2360007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:02:44.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.427+0000 7f235affd700 1 -- 172.21.15.7:0/2987767453 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f235c052ac0 con 0x7f236414ece0 2023-03-03T15:02:44.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.663+0000 7f236c502700 1 -- 172.21.15.7:0/2987767453 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2364074bb0 con 0x7f236414ece0 2023-03-03T15:02:44.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.663+0000 7f235affd700 1 -- 172.21.15.7:0/2987767453 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f235c0560e0 con 0x7f236414ece0 2023-03-03T15:02:44.666 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:02:44.666 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:02:44.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 -- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23480682f0 msgr2=0x7f234806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:44.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23480682f0 0x7f234806a7a0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f2360000c00 tx=0x7f2360007000 comp rx=0 tx=0).stop 2023-03-03T15:02:44.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 -- 172.21.15.7:0/2987767453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f236414ece0 msgr2=0x7f23641ed5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:02:44.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f236414ece0 0x7f23641ed5e0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f236406bcf0 tx=0x7f235c011f80 comp rx=0 tx=0).stop 2023-03-03T15:02:44.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 -- 172.21.15.7:0/2987767453 shutdown_connections 2023-03-03T15:02:44.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23480682f0 0x7f234806a7a0 secure :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f2360000c00 tx=0x7f2360007000 comp rx=0 tx=0).stop 2023-03-03T15:02:44.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f236414ece0 0x7f23641ed5e0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:44.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2350003aa0 0x7f236414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:44.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 --2- 172.21.15.7:0/2987767453 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2364071850 0x7f23641ed0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:02:44.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 -- 172.21.15.7:0/2987767453 >> 172.21.15.7:0/2987767453 conn(0x7f23640a94d0 msgr2=0x7f23640c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:02:44.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 -- 172.21.15.7:0/2987767453 shutdown_connections 2023-03-03T15:02:44.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:02:44.665+0000 7f236c502700 1 -- 172.21.15.7:0/2987767453 wait complete. 2023-03-03T15:02:44.670 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:02:44.680 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2023-03-03T15:02:59.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:02:59.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:24.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:03:24.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.245+0000 7f925d2c7700 1 Processor -- start 2023-03-03T15:03:24.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.245+0000 7f925d2c7700 1 -- start start 2023-03-03T15:03:24.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f925d2c7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f92580c7e00 0x7f9258154a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:24.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f925d2c7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9258157bb0 con 0x7f925806b6f0 2023-03-03T15:03:24.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f925d2c7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925806a2f0 con 0x7f92580c7a30 2023-03-03T15:03:24.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f925d2c7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925806ab00 con 0x7f92580c7e00 2023-03-03T15:03:24.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f92577fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f925806b6f0 0x7f9258154f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45424/0 (socket says 172.21.15.7:45424) 2023-03-03T15:03:24.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f92577fe700 1 -- 172.21.15.7:0/4159427447 learned_addr learned my addr 172.21.15.7:0/4159427447 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:03:24.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f92567fc700 1 --2- 172.21.15.7:0/4159427447 >> v2:172.21.15.143:3300/0 conn(0x7f92580c7e00 0x7f9258154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:24.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f9255ffb700 1 -- 172.21.15.7:0/4159427447 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601114848 0 0) 0x7f9258157bb0 con 0x7f925806b6f0 2023-03-03T15:03:24.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f9255ffb700 1 -- 172.21.15.7:0/4159427447 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92440035b0 con 0x7f925806b6f0 2023-03-03T15:03:24.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f92567fc700 1 -- 172.21.15.7:0/4159427447 >> v1:172.21.15.143:6789/0 conn(0x7f92580c7a30 legacy=0x7f925814e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:24.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f9255ffb700 1 -- 172.21.15.7:0/4159427447 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315389719 0 0) 0x7f925806a2f0 con 0x7f92580c7a30 2023-03-03T15:03:24.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f92567fc700 1 -- 172.21.15.7:0/4159427447 >> v1:172.21.15.7:6789/0 conn(0x7f925806b6f0 legacy=0x7f9258154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:24.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.246+0000 7f92567fc700 1 -- 172.21.15.7:0/4159427447 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92580bfbb0 con 0x7f92580c7e00 2023-03-03T15:03:24.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f92567fc700 1 --2- 172.21.15.7:0/4159427447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92580c7e00 0x7f9258154a10 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9248009840 tx=0x7f9248009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bcc798a0b50632 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:03:24.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f9255ffb700 1 -- 172.21.15.7:0/4159427447 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92500019f0 con 0x7f92580c7e00 2023-03-03T15:03:24.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f9255ffb700 1 -- 172.21.15.7:0/4159427447 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9248004030 con 0x7f92580c7e00 2023-03-03T15:03:24.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f9255ffb700 1 -- 172.21.15.7:0/4159427447 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92480043c0 con 0x7f92580c7e00 2023-03-03T15:03:24.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f925d2c7700 1 -- 172.21.15.7:0/4159427447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92580c7e00 msgr2=0x7f9258154a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:24.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f925d2c7700 1 --2- 172.21.15.7:0/4159427447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92580c7e00 0x7f9258154a10 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9248009840 tx=0x7f9248009b50 comp rx=0 tx=0).stop 2023-03-03T15:03:24.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f925d2c7700 1 -- 172.21.15.7:0/4159427447 shutdown_connections 2023-03-03T15:03:24.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f925d2c7700 1 --2- 172.21.15.7:0/4159427447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92580c7e00 0x7f9258154a10 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:24.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f925d2c7700 1 -- 172.21.15.7:0/4159427447 >> 172.21.15.7:0/4159427447 conn(0x7f9258073d20 msgr2=0x7f92580716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:03:24.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f925d2c7700 1 -- 172.21.15.7:0/4159427447 shutdown_connections 2023-03-03T15:03:24.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.247+0000 7f925d2c7700 1 -- 172.21.15.7:0/4159427447 wait complete. 2023-03-03T15:03:24.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.249+0000 7f925d2c7700 1 Processor -- start 2023-03-03T15:03:24.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.249+0000 7f925d2c7700 1 -- start start 2023-03-03T15:03:24.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.249+0000 7f925d2c7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92580c7a30 0x7f92581ed550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:24.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.249+0000 7f925d2c7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92580c7e00 0x7f92580c3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:24.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.249+0000 7f925d2c7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9258146d30 0x7f92580c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:24.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.249+0000 7f925d2c7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92580c42c0 con 0x7f9258146d30 2023-03-03T15:03:24.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.249+0000 7f925d2c7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92581eda90 con 0x7f92580c7a30 2023-03-03T15:03:24.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.249+0000 7f925d2c7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9258155680 con 0x7f92580c7e00 2023-03-03T15:03:24.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92577fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9258146d30 0x7f92580c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:24.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92577fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9258146d30 0x7f92580c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51450/0 (socket says 172.21.15.7:51450) 2023-03-03T15:03:24.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92577fe700 1 -- 172.21.15.7:0/1916423018 learned_addr learned my addr 172.21.15.7:0/1916423018 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:03:24.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92567fc700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92580c7e00 0x7f92580c3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:24.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f9256ffd700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92580c7a30 0x7f92581ed550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:24.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92577fe700 1 -- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92580c7e00 msgr2=0x7f92580c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:24.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92577fe700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92580c7e00 0x7f92580c3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:24.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92577fe700 1 -- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92580c7a30 msgr2=0x7f92581ed550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:24.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92577fe700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92580c7a30 0x7f92581ed550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:24.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92577fe700 1 -- 172.21.15.7:0/1916423018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92480094a0 con 0x7f9258146d30 2023-03-03T15:03:24.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92567fc700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92580c7e00 0x7f92580c3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:03:24.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f92577fe700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9258146d30 0x7f92580c93b0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f9250002040 tx=0x7f925000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:03:24.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f924f7fe700 1 -- 172.21.15.7:0/1916423018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925000ada0 con 0x7f9258146d30 2023-03-03T15:03:24.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f924f7fe700 1 -- 172.21.15.7:0/1916423018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92500070d0 con 0x7f9258146d30 2023-03-03T15:03:24.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f925d2c7700 1 -- 172.21.15.7:0/1916423018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925806a820 con 0x7f9258146d30 2023-03-03T15:03:24.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f924f7fe700 1 -- 172.21.15.7:0/1916423018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250007480 con 0x7f9258146d30 2023-03-03T15:03:24.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.250+0000 7f925d2c7700 1 -- 172.21.15.7:0/1916423018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925806a540 con 0x7f9258146d30 2023-03-03T15:03:24.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.251+0000 7f925d2c7700 1 -- 172.21.15.7:0/1916423018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925814e870 con 0x7f9258146d30 2023-03-03T15:03:24.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.252+0000 7f924f7fe700 1 -- 172.21.15.7:0/1916423018 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f925001a030 con 0x7f9258146d30 2023-03-03T15:03:24.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.252+0000 7f924f7fe700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92380682f0 0x7f923806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:03:24.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.252+0000 7f924f7fe700 1 -- 172.21.15.7:0/1916423018 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9250086130 con 0x7f9258146d30 2023-03-03T15:03:24.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.253+0000 7f9256ffd700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92380682f0 0x7f923806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:03:24.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.253+0000 7f9256ffd700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92380682f0 0x7f923806a7a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f9240007fd0 tx=0x7f9240009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:03:24.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.257+0000 7f924f7fe700 1 -- 172.21.15.7:0/1916423018 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f92500524d0 con 0x7f9258146d30 2023-03-03T15:03:24.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.494+0000 7f925d2c7700 1 -- 172.21.15.7:0/1916423018 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f925814b980 con 0x7f9258146d30 2023-03-03T15:03:24.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.495+0000 7f924f7fe700 1 -- 172.21.15.7:0/1916423018 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9250018090 con 0x7f9258146d30 2023-03-03T15:03:24.497 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:03:24.497 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:03:24.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.496+0000 7f925d2c7700 1 -- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92380682f0 msgr2=0x7f923806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:24.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.496+0000 7f925d2c7700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92380682f0 0x7f923806a7a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f9240007fd0 tx=0x7f9240009040 comp rx=0 tx=0).stop 2023-03-03T15:03:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.496+0000 7f925d2c7700 1 -- 172.21.15.7:0/1916423018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9258146d30 msgr2=0x7f92580c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:03:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.496+0000 7f925d2c7700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9258146d30 0x7f92580c93b0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f9250002040 tx=0x7f925000a340 comp rx=0 tx=0).stop 2023-03-03T15:03:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.497+0000 7f925d2c7700 1 -- 172.21.15.7:0/1916423018 shutdown_connections 2023-03-03T15:03:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.497+0000 7f925d2c7700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92380682f0 0x7f923806a7a0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.497+0000 7f925d2c7700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9258146d30 0x7f92580c93b0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.497+0000 7f925d2c7700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92580c7a30 0x7f92581ed550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:24.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.497+0000 7f925d2c7700 1 --2- 172.21.15.7:0/1916423018 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92580c7e00 0x7f92580c3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:03:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.497+0000 7f925d2c7700 1 -- 172.21.15.7:0/1916423018 >> 172.21.15.7:0/1916423018 conn(0x7f925806b6f0 msgr2=0x7f925814cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:03:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.497+0000 7f925d2c7700 1 -- 172.21.15.7:0/1916423018 shutdown_connections 2023-03-03T15:03:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:03:24.497+0000 7f925d2c7700 1 -- 172.21.15.7:0/1916423018 wait complete. 2023-03-03T15:03:24.501 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:03:24.522 INFO:tasks.mds_thrash.fs.[a]:waiting for 118 secs before thrashing 2023-03-03T15:03:29.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:29.538 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:59.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:03:59.602 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:21.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:04:21.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.819+0000 7fbfbe0a7700 1 Processor -- start 2023-03-03T15:04:21.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.820+0000 7fbfbe0a7700 1 -- start start 2023-03-03T15:04:21.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.820+0000 7fbfbe0a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 0x7fbfb80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:04:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.820+0000 7fbfbe0a7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbfb80c2950 0x7fbfb8157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:04:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.820+0000 7fbfbe0a7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb806cb70 con 0x7fbfb8073c30 2023-03-03T15:04:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.820+0000 7fbfbe0a7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb806c2f0 con 0x7fbfb80c2950 2023-03-03T15:04:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.820+0000 7fbfbe0a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb806ce50 con 0x7fbfb8151b20 2023-03-03T15:04:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.820+0000 7fbfb7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbfb8073c30 0x7fbfb8158020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45474/0 (socket says 172.21.15.7:45474) 2023-03-03T15:04:21.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.820+0000 7fbfb6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbfb80c2950 0x7fbfb8157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:04:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.820+0000 7fbfb7fff700 1 -- 172.21.15.7:0/2182356016 learned_addr learned my addr 172.21.15.7:0/2182356016 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:04:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb77fe700 1 --2- 172.21.15.7:0/2182356016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 0x7fbfb80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:04:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb67fc700 1 -- 172.21.15.7:0/2182356016 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526717650 0 0) 0x7fbfb806cb70 con 0x7fbfb8073c30 2023-03-03T15:04:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb67fc700 1 -- 172.21.15.7:0/2182356016 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfa00035b0 con 0x7fbfb8073c30 2023-03-03T15:04:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb6ffd700 1 -- 172.21.15.7:0/2182356016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 msgr2=0x7fbfb80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:04:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb6ffd700 1 --2- 172.21.15.7:0/2182356016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 0x7fbfb80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:04:21.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb6ffd700 1 -- 172.21.15.7:0/2182356016 >> v1:172.21.15.7:6789/0 conn(0x7fbfb8073c30 legacy=0x7fbfb8158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:04:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb6ffd700 1 -- 172.21.15.7:0/2182356016 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb806d130 con 0x7fbfb80c2950 2023-03-03T15:04:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb6ffd700 1 --2- 172.21.15.7:0/2182356016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb80c2950 0x7fbfb8157ae0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009840 tx=0x7fbfa8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e952d4c4d4ef74a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:04:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb67fc700 1 -- 172.21.15.7:0/2182356016 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8011830 con 0x7fbfb80c2950 2023-03-03T15:04:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb67fc700 1 -- 172.21.15.7:0/2182356016 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa8011990 con 0x7fbfb80c2950 2023-03-03T15:04:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.821+0000 7fbfb67fc700 1 -- 172.21.15.7:0/2182356016 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8011d20 con 0x7fbfb80c2950 2023-03-03T15:04:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.822+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2182356016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb80c2950 msgr2=0x7fbfb8157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:04:21.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.822+0000 7fbfbe0a7700 1 --2- 172.21.15.7:0/2182356016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb80c2950 0x7fbfb8157ae0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009840 tx=0x7fbfa8009b50 comp rx=0 tx=0).stop 2023-03-03T15:04:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.822+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2182356016 shutdown_connections 2023-03-03T15:04:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.822+0000 7fbfbe0a7700 1 --2- 172.21.15.7:0/2182356016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 0x7fbfb80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:04:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.822+0000 7fbfbe0a7700 1 --2- 172.21.15.7:0/2182356016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb80c2950 0x7fbfb8157ae0 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:04:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.822+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2182356016 >> 172.21.15.7:0/2182356016 conn(0x7fbfb80a94d0 msgr2=0x7fbfb80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:04:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.822+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2182356016 shutdown_connections 2023-03-03T15:04:21.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.822+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2182356016 wait complete. 2023-03-03T15:04:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.823+0000 7fbfbe0a7700 1 Processor -- start 2023-03-03T15:04:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.823+0000 7fbfbe0a7700 1 -- start start 2023-03-03T15:04:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfbe0a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb8073c30 0x7fbfb80bde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:04:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfbe0a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfb80c2950 0x7fbfb8156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:04:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfbe0a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 0x7fbfb8152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:04:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfbe0a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80c5160 con 0x7fbfb8073c30 2023-03-03T15:04:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfbe0a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80c5370 con 0x7fbfb80c2950 2023-03-03T15:04:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfbe0a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb8156020 con 0x7fbfb8151b20 2023-03-03T15:04:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb8073c30 0x7fbfb80bde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:04:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb8073c30 0x7fbfb80bde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51496/0 (socket says 172.21.15.7:51496) 2023-03-03T15:04:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb77fe700 1 -- 172.21.15.7:0/2892946362 learned_addr learned my addr 172.21.15.7:0/2892946362 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:04:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb77fe700 1 -- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 msgr2=0x7fbfb8152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:04:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb6ffd700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfb80c2950 0x7fbfb8156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:04:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb7fff700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 0x7fbfb8152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:04:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb77fe700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 0x7fbfb8152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:04:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb77fe700 1 -- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfb80c2950 msgr2=0x7fbfb8156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:04:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb77fe700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfb80c2950 0x7fbfb8156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:04:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb77fe700 1 -- 172.21.15.7:0/2892946362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa80094a0 con 0x7fbfb8073c30 2023-03-03T15:04:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb6ffd700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfb80c2950 0x7fbfb8156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:04:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.824+0000 7fbfb7fff700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 0x7fbfb8152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:04:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.825+0000 7fbfb77fe700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb8073c30 0x7fbfb80bde80 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00d800 tx=0x7fbfac00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:04:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.825+0000 7fbf9ffff700 1 -- 172.21.15.7:0/2892946362 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac0125f0 con 0x7fbfb8073c30 2023-03-03T15:04:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.825+0000 7fbf9ffff700 1 -- 172.21.15.7:0/2892946362 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac012750 con 0x7fbfb8073c30 2023-03-03T15:04:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.825+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2892946362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb8158730 con 0x7fbfb8073c30 2023-03-03T15:04:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.825+0000 7fbf9ffff700 1 -- 172.21.15.7:0/2892946362 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac012b00 con 0x7fbfb8073c30 2023-03-03T15:04:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.825+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2892946362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb806d410 con 0x7fbfb8073c30 2023-03-03T15:04:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.826+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2892946362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb8151ef0 con 0x7fbfb8073c30 2023-03-03T15:04:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.826+0000 7fbf9ffff700 1 -- 172.21.15.7:0/2892946362 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbfac012c60 con 0x7fbfb8073c30 2023-03-03T15:04:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.827+0000 7fbf9ffff700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfa40682f0 0x7fbfa406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:04:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.827+0000 7fbf9ffff700 1 -- 172.21.15.7:0/2892946362 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbfac086640 con 0x7fbfb8073c30 2023-03-03T15:04:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.827+0000 7fbfb6ffd700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfa40682f0 0x7fbfa406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:04:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.827+0000 7fbfb6ffd700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfa40682f0 0x7fbfa406a7a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80c3ad0 tx=0x7fbfa8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:04:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:21.831+0000 7fbf9ffff700 1 -- 172.21.15.7:0/2892946362 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbfac0529e0 con 0x7fbfb8073c30 2023-03-03T15:04:22.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.071+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2892946362 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfb8156c30 con 0x7fbfb8073c30 2023-03-03T15:04:22.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.072+0000 7fbf9ffff700 1 -- 172.21.15.7:0/2892946362 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbfac056000 con 0x7fbfb8073c30 2023-03-03T15:04:22.074 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:04:22.074 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:04:22.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfa40682f0 msgr2=0x7fbfa406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:04:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfa40682f0 0x7fbfa406a7a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fbfb80c3ad0 tx=0x7fbfa8008030 comp rx=0 tx=0).stop 2023-03-03T15:04:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2892946362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb8073c30 msgr2=0x7fbfb80bde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:04:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb8073c30 0x7fbfb80bde80 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fbfac00d800 tx=0x7fbfac00dbc0 comp rx=0 tx=0).stop 2023-03-03T15:04:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2892946362 shutdown_connections 2023-03-03T15:04:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfb8151b20 0x7fbfb8152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:04:22.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfa40682f0 0x7fbfa406a7a0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:04:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfb8073c30 0x7fbfb80bde80 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:04:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 --2- 172.21.15.7:0/2892946362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfb80c2950 0x7fbfb8156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:04:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2892946362 >> 172.21.15.7:0/2892946362 conn(0x7fbfb80a94d0 msgr2=0x7fbfb80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:04:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2892946362 shutdown_connections 2023-03-03T15:04:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:04:22.074+0000 7fbfbe0a7700 1 -- 172.21.15.7:0/2892946362 wait complete. 2023-03-03T15:04:22.077 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:04:22.088 INFO:tasks.mds_thrash.fs.[b]:waiting for 56 secs before thrashing 2023-03-03T15:04:29.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:29.697 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:59.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:04:59.762 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:18.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:05:18.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.227+0000 7f5d429d7700 1 Processor -- start 2023-03-03T15:05:18.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.227+0000 7f5d429d7700 1 -- start start 2023-03-03T15:05:18.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.227+0000 7f5d429d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 0x7f5d3c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:18.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.227+0000 7f5d429d7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5d3c14fda0 0x7f5d3c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:18.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.227+0000 7f5d429d7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c0c2f30 con 0x7f5d3c0abe20 2023-03-03T15:05:18.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.227+0000 7f5d429d7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d3c0c31d0 con 0x7f5d3c14fda0 2023-03-03T15:05:18.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.227+0000 7f5d429d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c1524a0 con 0x7f5d3c14f9d0 2023-03-03T15:05:18.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5d3c14fda0 0x7f5d3c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:18.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d40f74700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5d3c0abe20 0x7f5d3c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45516/0 (socket says 172.21.15.7:45516) 2023-03-03T15:05:18.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d40f74700 1 -- 172.21.15.7:0/3730385283 learned_addr learned my addr 172.21.15.7:0/3730385283 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:05:18.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3bfff700 1 --2- 172.21.15.7:0/3730385283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 0x7f5d3c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:18.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3b7fe700 1 -- 172.21.15.7:0/3730385283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 msgr2=0x7f5d3c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:05:18.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3b7fe700 1 --2- 172.21.15.7:0/3730385283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 0x7f5d3c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:18.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3b7fe700 1 -- 172.21.15.7:0/3730385283 >> v1:172.21.15.7:6789/0 conn(0x7f5d3c0abe20 legacy=0x7f5d3c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:18.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3affd700 1 -- 172.21.15.7:0/3730385283 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552957750 0 0) 0x7f5d3c0c2f30 con 0x7f5d3c0abe20 2023-03-03T15:05:18.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3b7fe700 1 -- 172.21.15.7:0/3730385283 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c0c34b0 con 0x7f5d3c14fda0 2023-03-03T15:05:18.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3b7fe700 1 --2- 172.21.15.7:0/3730385283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c14fda0 0x7f5d3c071890 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009840 tx=0x7f5d30009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4492a622dd9c028 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:05:18.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3affd700 1 -- 172.21.15.7:0/3730385283 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d30011830 con 0x7f5d3c14fda0 2023-03-03T15:05:18.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3affd700 1 -- 172.21.15.7:0/3730385283 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d30011990 con 0x7f5d3c14fda0 2023-03-03T15:05:18.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.228+0000 7f5d3affd700 1 -- 172.21.15.7:0/3730385283 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d30011d20 con 0x7f5d3c14fda0 2023-03-03T15:05:18.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.229+0000 7f5d429d7700 1 -- 172.21.15.7:0/3730385283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c14fda0 msgr2=0x7f5d3c071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:18.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.229+0000 7f5d429d7700 1 --2- 172.21.15.7:0/3730385283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c14fda0 0x7f5d3c071890 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009840 tx=0x7f5d30009b50 comp rx=0 tx=0).stop 2023-03-03T15:05:18.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.229+0000 7f5d429d7700 1 -- 172.21.15.7:0/3730385283 shutdown_connections 2023-03-03T15:05:18.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.229+0000 7f5d429d7700 1 --2- 172.21.15.7:0/3730385283 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 0x7f5d3c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:18.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.229+0000 7f5d429d7700 1 --2- 172.21.15.7:0/3730385283 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c14fda0 0x7f5d3c071890 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:18.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.229+0000 7f5d429d7700 1 -- 172.21.15.7:0/3730385283 >> 172.21.15.7:0/3730385283 conn(0x7f5d3c0c8900 msgr2=0x7f5d3c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:05:18.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.229+0000 7f5d429d7700 1 -- 172.21.15.7:0/3730385283 shutdown_connections 2023-03-03T15:05:18.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.229+0000 7f5d429d7700 1 -- 172.21.15.7:0/3730385283 wait complete. 2023-03-03T15:05:18.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.230+0000 7f5d429d7700 1 Processor -- start 2023-03-03T15:05:18.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.230+0000 7f5d429d7700 1 -- start start 2023-03-03T15:05:18.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d429d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c0abe20 0x7f5d3c154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:18.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d429d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 0x7f5d3c1db970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:18.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d429d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d3c14fda0 0x7f5d3c13ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:18.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d429d7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c0c6620 con 0x7f5d3c0abe20 2023-03-03T15:05:18.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d429d7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c154b70 con 0x7f5d3c14fda0 2023-03-03T15:05:18.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d429d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3c153570 con 0x7f5d3c14f9d0 2023-03-03T15:05:18.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d3bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c0abe20 0x7f5d3c154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:18.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d3bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c0abe20 0x7f5d3c154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51538/0 (socket says 172.21.15.7:51538) 2023-03-03T15:05:18.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d3bfff700 1 -- 172.21.15.7:0/1300433090 learned_addr learned my addr 172.21.15.7:0/1300433090 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:05:18.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d3b7fe700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 0x7f5d3c1db970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:18.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d3bfff700 1 -- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 msgr2=0x7f5d3c1db970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:18.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d3bfff700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 0x7f5d3c1db970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:18.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d3bfff700 1 -- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d3c14fda0 msgr2=0x7f5d3c13ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:18.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d3bfff700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d3c14fda0 0x7f5d3c13ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:18.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d3bfff700 1 -- 172.21.15.7:0/1300433090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d300094a0 con 0x7f5d3c0abe20 2023-03-03T15:05:18.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.231+0000 7f5d3b7fe700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 0x7f5d3c1db970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:05:18.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.232+0000 7f5d3bfff700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c0abe20 0x7f5d3c154630 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00e9b0 tx=0x7f5d2c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:05:18.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.232+0000 7f5d38ff9700 1 -- 172.21.15.7:0/1300433090 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c009710 con 0x7f5d3c0abe20 2023-03-03T15:05:18.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.232+0000 7f5d38ff9700 1 -- 172.21.15.7:0/1300433090 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2c00c850 con 0x7f5d3c0abe20 2023-03-03T15:05:18.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.232+0000 7f5d38ff9700 1 -- 172.21.15.7:0/1300433090 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c00cbc0 con 0x7f5d3c0abe20 2023-03-03T15:05:18.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.232+0000 7f5d429d7700 1 -- 172.21.15.7:0/1300433090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c0ac350 con 0x7f5d3c0abe20 2023-03-03T15:05:18.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.232+0000 7f5d429d7700 1 -- 172.21.15.7:0/1300433090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3c0ab630 con 0x7f5d3c0abe20 2023-03-03T15:05:18.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.233+0000 7f5d429d7700 1 -- 172.21.15.7:0/1300433090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3c0c3e10 con 0x7f5d3c0abe20 2023-03-03T15:05:18.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.233+0000 7f5d38ff9700 1 -- 172.21.15.7:0/1300433090 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5d2c019030 con 0x7f5d3c0abe20 2023-03-03T15:05:18.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.234+0000 7f5d38ff9700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d280682f0 0x7f5d2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:18.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.234+0000 7f5d38ff9700 1 -- 172.21.15.7:0/1300433090 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5d2c086190 con 0x7f5d3c0abe20 2023-03-03T15:05:18.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.234+0000 7f5d3b7fe700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d280682f0 0x7f5d2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:18.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.234+0000 7f5d3b7fe700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d280682f0 0x7f5d2806a7a0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c1501d0 tx=0x7f5d30008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:05:18.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.238+0000 7f5d38ff9700 1 -- 172.21.15.7:0/1300433090 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5d2c052530 con 0x7f5d3c0abe20 2023-03-03T15:05:18.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.474+0000 7f5d429d7700 1 -- 172.21.15.7:0/1300433090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d3c074bb0 con 0x7f5d3c0abe20 2023-03-03T15:05:18.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.475+0000 7f5d38ff9700 1 -- 172.21.15.7:0/1300433090 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5d2c055b50 con 0x7f5d3c0abe20 2023-03-03T15:05:18.477 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:05:18.477 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:05:18.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 -- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d280682f0 msgr2=0x7f5d2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:18.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d280682f0 0x7f5d2806a7a0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c1501d0 tx=0x7f5d30008030 comp rx=0 tx=0).stop 2023-03-03T15:05:18.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 -- 172.21.15.7:0/1300433090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c0abe20 msgr2=0x7f5d3c154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:18.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c0abe20 0x7f5d3c154630 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00e9b0 tx=0x7f5d2c00ed70 comp rx=0 tx=0).stop 2023-03-03T15:05:18.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 -- 172.21.15.7:0/1300433090 shutdown_connections 2023-03-03T15:05:18.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d3c14fda0 0x7f5d3c13ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:18.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d280682f0 0x7f5d2806a7a0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:18.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d3c0abe20 0x7f5d3c154630 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:18.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 --2- 172.21.15.7:0/1300433090 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d3c14f9d0 0x7f5d3c1db970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:18.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 -- 172.21.15.7:0/1300433090 >> 172.21.15.7:0/1300433090 conn(0x7f5d3c0c8900 msgr2=0x7f5d3c1df290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:05:18.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 -- 172.21.15.7:0/1300433090 shutdown_connections 2023-03-03T15:05:18.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:18.477+0000 7f5d429d7700 1 -- 172.21.15.7:0/1300433090 wait complete. 2023-03-03T15:05:18.480 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:05:18.490 INFO:tasks.mds_thrash.fs.[b]:waiting for 42 secs before thrashing 2023-03-03T15:05:22.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:05:22.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.660+0000 7f3d260d8700 1 Processor -- start 2023-03-03T15:05:22.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.661+0000 7f3d260d8700 1 -- start start 2023-03-03T15:05:22.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.661+0000 7f3d260d8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d20155410 0x7f3d200bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:22.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.661+0000 7f3d260d8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d200c5b00 0x7f3d200bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:22.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.661+0000 7f3d260d8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d200c08d0 con 0x7f3d20151e80 2023-03-03T15:05:22.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.661+0000 7f3d260d8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d20152250 con 0x7f3d20155410 2023-03-03T15:05:22.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.661+0000 7f3d260d8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d200c0fb0 con 0x7f3d200c5b00 2023-03-03T15:05:22.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.661+0000 7f3d1effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d20155410 0x7f3d200bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:22.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d20155410 0x7f3d200bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51552/0 (socket says 172.21.15.7:51552) 2023-03-03T15:05:22.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1effd700 1 -- 172.21.15.7:0/42647993 learned_addr learned my addr 172.21.15.7:0/42647993 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:05:22.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1ffff700 1 --2- 172.21.15.7:0/42647993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d200c5b00 0x7f3d200bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:22.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1effd700 1 -- 172.21.15.7:0/42647993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d200c5b00 msgr2=0x7f3d200bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:05:22.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1effd700 1 --2- 172.21.15.7:0/42647993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d200c5b00 0x7f3d200bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:22.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1effd700 1 -- 172.21.15.7:0/42647993 >> v1:172.21.15.143:6789/0 conn(0x7f3d20151e80 legacy=0x7f3d200c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:05:22.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1effd700 1 -- 172.21.15.7:0/42647993 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d20154930 con 0x7f3d20155410 2023-03-03T15:05:22.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1ffff700 1 --2- 172.21.15.7:0/42647993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d200c5b00 0x7f3d200bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:05:22.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1effd700 1 --2- 172.21.15.7:0/42647993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 0x7f3d200bf8e0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f3d080098f0 tx=0x7f3d08009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964dbe66baea247e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:05:22.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1e7fc700 1 -- 172.21.15.7:0/42647993 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08004230 con 0x7f3d20155410 2023-03-03T15:05:22.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1e7fc700 1 -- 172.21.15.7:0/42647993 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d08004390 con 0x7f3d20155410 2023-03-03T15:05:22.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.662+0000 7f3d1e7fc700 1 -- 172.21.15.7:0/42647993 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d080036a0 con 0x7f3d20155410 2023-03-03T15:05:22.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.663+0000 7f3d260d8700 1 -- 172.21.15.7:0/42647993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 msgr2=0x7f3d200bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:22.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.663+0000 7f3d260d8700 1 --2- 172.21.15.7:0/42647993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 0x7f3d200bf8e0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f3d080098f0 tx=0x7f3d08009c00 comp rx=0 tx=0).stop 2023-03-03T15:05:22.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.663+0000 7f3d260d8700 1 -- 172.21.15.7:0/42647993 shutdown_connections 2023-03-03T15:05:22.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.663+0000 7f3d260d8700 1 --2- 172.21.15.7:0/42647993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d200c5b00 0x7f3d200bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:22.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.663+0000 7f3d260d8700 1 --2- 172.21.15.7:0/42647993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 0x7f3d200bf8e0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:22.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.663+0000 7f3d260d8700 1 -- 172.21.15.7:0/42647993 >> 172.21.15.7:0/42647993 conn(0x7f3d20071630 msgr2=0x7f3d2006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:05:22.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.663+0000 7f3d260d8700 1 -- 172.21.15.7:0/42647993 shutdown_connections 2023-03-03T15:05:22.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.663+0000 7f3d260d8700 1 -- 172.21.15.7:0/42647993 wait complete. 2023-03-03T15:05:22.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.664+0000 7f3d260d8700 1 Processor -- start 2023-03-03T15:05:22.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.664+0000 7f3d260d8700 1 -- start start 2023-03-03T15:05:22.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d260d8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d200c5b00 0x7f3d200ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:22.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d260d8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d20151e80 0x7f3d2014aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:22.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d260d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 0x7f3d2014b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:22.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d260d8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2014ba70 con 0x7f3d20155410 2023-03-03T15:05:22.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d260d8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d20152ad0 con 0x7f3d200c5b00 2023-03-03T15:05:22.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d260d8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d201eac60 con 0x7f3d20151e80 2023-03-03T15:05:22.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d1ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 0x7f3d2014b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:22.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d1ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 0x7f3d2014b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51560/0 (socket says 172.21.15.7:51560) 2023-03-03T15:05:22.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d1ffff700 1 -- 172.21.15.7:0/1961215236 learned_addr learned my addr 172.21.15.7:0/1961215236 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:05:22.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d1effd700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d20151e80 0x7f3d2014aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:22.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d1f7fe700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d200c5b00 0x7f3d200ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:22.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d1ffff700 1 -- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d20151e80 msgr2=0x7f3d2014aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:22.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d1ffff700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d20151e80 0x7f3d2014aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:22.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d1ffff700 1 -- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d200c5b00 msgr2=0x7f3d200ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:22.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d1ffff700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d200c5b00 0x7f3d200ca990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:22.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.665+0000 7f3d1ffff700 1 -- 172.21.15.7:0/1961215236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d08009550 con 0x7f3d20155410 2023-03-03T15:05:22.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.666+0000 7f3d1ffff700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 0x7f3d2014b530 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000cd50 tx=0x7f3d1000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:05:22.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.666+0000 7f3d17fff700 1 -- 172.21.15.7:0/1961215236 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1000cea0 con 0x7f3d20155410 2023-03-03T15:05:22.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.666+0000 7f3d17fff700 1 -- 172.21.15.7:0/1961215236 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1000f480 con 0x7f3d20155410 2023-03-03T15:05:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.666+0000 7f3d260d8700 1 -- 172.21.15.7:0/1961215236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d20154ef0 con 0x7f3d20155410 2023-03-03T15:05:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.666+0000 7f3d17fff700 1 -- 172.21.15.7:0/1961215236 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1000f830 con 0x7f3d20155410 2023-03-03T15:05:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.666+0000 7f3d260d8700 1 -- 172.21.15.7:0/1961215236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d20154c10 con 0x7f3d20155410 2023-03-03T15:05:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.667+0000 7f3d260d8700 1 -- 172.21.15.7:0/1961215236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d200c5220 con 0x7f3d20155410 2023-03-03T15:05:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.667+0000 7f3d17fff700 1 -- 172.21.15.7:0/1961215236 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d10009030 con 0x7f3d20155410 2023-03-03T15:05:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.668+0000 7f3d17fff700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d000682f0 0x7f3d0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:05:22.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.668+0000 7f3d17fff700 1 -- 172.21.15.7:0/1961215236 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3d10086a40 con 0x7f3d20155410 2023-03-03T15:05:22.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.668+0000 7f3d1f7fe700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d000682f0 0x7f3d0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:05:22.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.669+0000 7f3d1f7fe700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d000682f0 0x7f3d0006a7a0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009960 tx=0x7f3d180090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:05:22.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.672+0000 7f3d17fff700 1 -- 172.21.15.7:0/1961215236 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d10052d60 con 0x7f3d20155410 2023-03-03T15:05:22.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.912+0000 7f3d260d8700 1 -- 172.21.15.7:0/1961215236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d20074bb0 con 0x7f3d20155410 2023-03-03T15:05:22.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.912+0000 7f3d17fff700 1 -- 172.21.15.7:0/1961215236 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3d10056380 con 0x7f3d20155410 2023-03-03T15:05:22.914 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:05:22.914 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:05:22.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 -- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d000682f0 msgr2=0x7f3d0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:22.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d000682f0 0x7f3d0006a7a0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009960 tx=0x7f3d180090e0 comp rx=0 tx=0).stop 2023-03-03T15:05:22.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 -- 172.21.15.7:0/1961215236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 msgr2=0x7f3d2014b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:05:22.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 0x7f3d2014b530 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000cd50 tx=0x7f3d1000a560 comp rx=0 tx=0).stop 2023-03-03T15:05:22.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 -- 172.21.15.7:0/1961215236 shutdown_connections 2023-03-03T15:05:22.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d000682f0 0x7f3d0006a7a0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:22.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d20155410 0x7f3d2014b530 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:22.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d200c5b00 0x7f3d200ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:22.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 --2- 172.21.15.7:0/1961215236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d20151e80 0x7f3d2014aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:05:22.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 -- 172.21.15.7:0/1961215236 >> 172.21.15.7:0/1961215236 conn(0x7f3d20071630 msgr2=0x7f3d20153b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:05:22.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.914+0000 7f3d260d8700 1 -- 172.21.15.7:0/1961215236 shutdown_connections 2023-03-03T15:05:22.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:05:22.915+0000 7f3d260d8700 1 -- 172.21.15.7:0/1961215236 wait complete. 2023-03-03T15:05:22.918 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:05:22.928 INFO:tasks.mds_thrash.fs.[a]:waiting for 51 secs before thrashing 2023-03-03T15:05:29.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:29.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:59.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:05:59.896 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:00.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:06:00.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.628+0000 7fbdc23f1700 1 Processor -- start 2023-03-03T15:06:00.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.628+0000 7fbdc23f1700 1 -- start start 2023-03-03T15:06:00.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.629+0000 7fbdc23f1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdbc152e30 0x7fbdbc0c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:00.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.629+0000 7fbdc23f1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0c4910 0x7fbdbc156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:00.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.629+0000 7fbdc23f1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdbc1523b0 con 0x7fbdbc0beab0 2023-03-03T15:06:00.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.629+0000 7fbdc23f1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdbc072010 con 0x7fbdbc152e30 2023-03-03T15:06:00.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.629+0000 7fbdc23f1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc0aaaf0 con 0x7fbdbc0c4910 2023-03-03T15:06:00.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.629+0000 7fbdc098e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0c4910 0x7fbdbc156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:00.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.629+0000 7fbdbbfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdbc152e30 0x7fbdbc0c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:00.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.629+0000 7fbdbb7fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fbdbc0beab0 0x7fbdbc155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49088/0 (socket says 172.21.15.7:49088) 2023-03-03T15:06:00.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.629+0000 7fbdbb7fe700 1 -- 172.21.15.7:0/3264314551 learned_addr learned my addr 172.21.15.7:0/3264314551 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:06:00.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdbaffd700 1 -- 172.21.15.7:0/3264314551 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286321171 0 0) 0x7fbdbc1523b0 con 0x7fbdbc0beab0 2023-03-03T15:06:00.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdbaffd700 1 -- 172.21.15.7:0/3264314551 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda80035b0 con 0x7fbdbc0beab0 2023-03-03T15:06:00.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdbbfff700 1 -- 172.21.15.7:0/3264314551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0c4910 msgr2=0x7fbdbc156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:06:00.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdbbfff700 1 --2- 172.21.15.7:0/3264314551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0c4910 0x7fbdbc156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:00.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdbbfff700 1 -- 172.21.15.7:0/3264314551 >> v1:172.21.15.143:6789/0 conn(0x7fbdbc0beab0 legacy=0x7fbdbc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:00.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdbbfff700 1 -- 172.21.15.7:0/3264314551 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc0aadd0 con 0x7fbdbc152e30 2023-03-03T15:06:00.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdc098e700 1 --2- 172.21.15.7:0/3264314551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0c4910 0x7fbdbc156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:06:00.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdbbfff700 1 --2- 172.21.15.7:0/3264314551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc152e30 0x7fbdbc0c43d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009840 tx=0x7fbdac009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=443a9e7f380e1a4b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:00.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdbaffd700 1 -- 172.21.15.7:0/3264314551 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac011830 con 0x7fbdbc152e30 2023-03-03T15:06:00.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdbaffd700 1 -- 172.21.15.7:0/3264314551 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdac011990 con 0x7fbdbc152e30 2023-03-03T15:06:00.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.630+0000 7fbdbaffd700 1 -- 172.21.15.7:0/3264314551 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdac011d20 con 0x7fbdbc152e30 2023-03-03T15:06:00.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.631+0000 7fbdc23f1700 1 -- 172.21.15.7:0/3264314551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc152e30 msgr2=0x7fbdbc0c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:00.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.631+0000 7fbdc23f1700 1 --2- 172.21.15.7:0/3264314551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc152e30 0x7fbdbc0c43d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009840 tx=0x7fbdac009b50 comp rx=0 tx=0).stop 2023-03-03T15:06:00.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.631+0000 7fbdc23f1700 1 -- 172.21.15.7:0/3264314551 shutdown_connections 2023-03-03T15:06:00.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.631+0000 7fbdc23f1700 1 --2- 172.21.15.7:0/3264314551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0c4910 0x7fbdbc156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:00.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.631+0000 7fbdc23f1700 1 --2- 172.21.15.7:0/3264314551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc152e30 0x7fbdbc0c43d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:00.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.631+0000 7fbdc23f1700 1 -- 172.21.15.7:0/3264314551 >> 172.21.15.7:0/3264314551 conn(0x7fbdbc0736f0 msgr2=0x7fbdbc073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:06:00.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.631+0000 7fbdc23f1700 1 -- 172.21.15.7:0/3264314551 shutdown_connections 2023-03-03T15:06:00.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.631+0000 7fbdc23f1700 1 -- 172.21.15.7:0/3264314551 wait complete. 2023-03-03T15:06:00.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.632+0000 7fbdc23f1700 1 Processor -- start 2023-03-03T15:06:00.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.632+0000 7fbdc23f1700 1 -- start start 2023-03-03T15:06:00.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdc23f1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0beab0 0x7fbdbc1eb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:00.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdc23f1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc0c4910 0x7fbdbc1dc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:00.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdc23f1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc152e30 0x7fbdbc1e7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:00.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdc23f1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc0ab5d0 con 0x7fbdbc0c4910 2023-03-03T15:06:00.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdc23f1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc14da80 con 0x7fbdbc152e30 2023-03-03T15:06:00.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdc23f1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc1e7460 con 0x7fbdbc0beab0 2023-03-03T15:06:00.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc0c4910 0x7fbdbc1dc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:00.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc0c4910 0x7fbdbc1dc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51594/0 (socket says 172.21.15.7:51594) 2023-03-03T15:06:00.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbb7fe700 1 -- 172.21.15.7:0/1594789402 learned_addr learned my addr 172.21.15.7:0/1594789402 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:06:00.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdc098e700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc152e30 0x7fbdbc1e7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:00.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbbfff700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0beab0 0x7fbdbc1eb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:00.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbb7fe700 1 -- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0beab0 msgr2=0x7fbdbc1eb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:00.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbb7fe700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0beab0 0x7fbdbc1eb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:00.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbb7fe700 1 -- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc152e30 msgr2=0x7fbdbc1e7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:00.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbb7fe700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc152e30 0x7fbdbc1e7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:00.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbb7fe700 1 -- 172.21.15.7:0/1594789402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb4009a70 con 0x7fbdbc0c4910 2023-03-03T15:06:00.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbbfff700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0beab0 0x7fbdbc1eb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:06:00.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdc098e700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc152e30 0x7fbdbc1e7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:06:00.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.633+0000 7fbdbb7fe700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc0c4910 0x7fbdbc1dc220 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fbda4001eb0 tx=0x7fbda400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:00.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.634+0000 7fbdb8ff9700 1 -- 172.21.15.7:0/1594789402 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4005d70 con 0x7fbdbc0c4910 2023-03-03T15:06:00.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.634+0000 7fbdb8ff9700 1 -- 172.21.15.7:0/1594789402 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda40070d0 con 0x7fbdbc0c4910 2023-03-03T15:06:00.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.634+0000 7fbdc23f1700 1 -- 172.21.15.7:0/1594789402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdac0094a0 con 0x7fbdbc0c4910 2023-03-03T15:06:00.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.634+0000 7fbdb8ff9700 1 -- 172.21.15.7:0/1594789402 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4007460 con 0x7fbdbc0c4910 2023-03-03T15:06:00.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.634+0000 7fbdc23f1700 1 -- 172.21.15.7:0/1594789402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdbc153200 con 0x7fbdbc0c4910 2023-03-03T15:06:00.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.635+0000 7fbdc23f1700 1 -- 172.21.15.7:0/1594789402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdbc153390 con 0x7fbdbc0c4910 2023-03-03T15:06:00.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.635+0000 7fbdb8ff9700 1 -- 172.21.15.7:0/1594789402 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbda4019030 con 0x7fbdbc0c4910 2023-03-03T15:06:00.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.635+0000 7fbdb8ff9700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 0x7fbd9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:00.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.636+0000 7fbdb8ff9700 1 -- 172.21.15.7:0/1594789402 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbda4085550 con 0x7fbdbc0c4910 2023-03-03T15:06:00.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.636+0000 7fbdbbfff700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 0x7fbd9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:00.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.636+0000 7fbdbbfff700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 0x7fbd9c06a7a0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fbdac0090b0 tx=0x7fbdac008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:00.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.640+0000 7fbdb8ff9700 1 -- 172.21.15.7:0/1594789402 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbda40518f0 con 0x7fbdbc0c4910 2023-03-03T15:06:00.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.879+0000 7fbdc23f1700 1 -- 172.21.15.7:0/1594789402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdbc1eb8d0 con 0x7fbdbc0c4910 2023-03-03T15:06:00.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.879+0000 7fbdb8ff9700 1 -- 172.21.15.7:0/1594789402 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbdbc1eb8d0 con 0x7fbdbc0c4910 2023-03-03T15:06:00.882 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:06:00.883 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:06:00.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.881+0000 7fbdc23f1700 1 -- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 msgr2=0x7fbd9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:00.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.881+0000 7fbdc23f1700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 0x7fbd9c06a7a0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fbdac0090b0 tx=0x7fbdac008030 comp rx=0 tx=0).stop 2023-03-03T15:06:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.881+0000 7fbdc23f1700 1 -- 172.21.15.7:0/1594789402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc0c4910 msgr2=0x7fbdbc1dc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.881+0000 7fbdc23f1700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc0c4910 0x7fbdbc1dc220 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fbda4001eb0 tx=0x7fbda400c860 comp rx=0 tx=0).stop 2023-03-03T15:06:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.882+0000 7fbdc23f1700 1 -- 172.21.15.7:0/1594789402 shutdown_connections 2023-03-03T15:06:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.882+0000 7fbdc23f1700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc152e30 0x7fbdbc1e7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.882+0000 7fbdc23f1700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc0beab0 0x7fbdbc1eb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.882+0000 7fbdc23f1700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 0x7fbd9c06a7a0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.882+0000 7fbdc23f1700 1 --2- 172.21.15.7:0/1594789402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc0c4910 0x7fbdbc1dc220 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:00.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.882+0000 7fbdc23f1700 1 -- 172.21.15.7:0/1594789402 >> 172.21.15.7:0/1594789402 conn(0x7fbdbc0736f0 msgr2=0x7fbdbc0bd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:06:00.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.882+0000 7fbdc23f1700 1 -- 172.21.15.7:0/1594789402 shutdown_connections 2023-03-03T15:06:00.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:00.882+0000 7fbdc23f1700 1 -- 172.21.15.7:0/1594789402 wait complete. 2023-03-03T15:06:00.886 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:06:00.896 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2023-03-03T15:06:13.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:06:14.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.066+0000 7fcc60a2c700 1 Processor -- start 2023-03-03T15:06:14.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.066+0000 7fcc60a2c700 1 -- start start 2023-03-03T15:06:14.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc60a2c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c06a2b0 0x7fcc5c152280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:14.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc60a2c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c0c2780 con 0x7fcc5c0c5190 2023-03-03T15:06:14.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc60a2c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c0c1b70 con 0x7fcc5c0c1500 2023-03-03T15:06:14.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc60a2c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0c1970 con 0x7fcc5c06a2b0 2023-03-03T15:06:14.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc5ad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcc5c0c5190 0x7fcc5c06d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45600/0 (socket says 172.21.15.7:45600) 2023-03-03T15:06:14.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc5ad9d700 1 -- 172.21.15.7:0/3034919162 learned_addr learned my addr 172.21.15.7:0/3034919162 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:06:14.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc5a59c700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c06a2b0 0x7fcc5c152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:14.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219754468 0 0) 0x7fcc5c0c2780 con 0x7fcc5c0c5190 2023-03-03T15:06:14.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc440035b0 con 0x7fcc5c0c5190 2023-03-03T15:06:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2330992093 0 0) 0x7fcc440035b0 con 0x7fcc5c0c5190 2023-03-03T15:06:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.067+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c06a2b0 msgr2=0x7fcc5c152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:06:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c06a2b0 0x7fcc5c152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 >> v1:172.21.15.143:6789/0 conn(0x7fcc5c0c1500 legacy=0x7fcc5c0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c0c1e50 con 0x7fcc5c0c5190 2023-03-03T15:06:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fcc540019f0 con 0x7fcc5c0c5190 2023-03-03T15:06:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 >> v1:172.21.15.7:6789/0 conn(0x7fcc5c0c5190 legacy=0x7fcc5c06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:14.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc44003aa0 0x7fcc44003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc44004850 0x7fcc44004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c06a2b0 0x7fcc44005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0c1970 con 0x7fcc44004850 2023-03-03T15:06:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4400bac0 con 0x7fcc44003aa0 2023-03-03T15:06:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4400bc30 con 0x7fcc5c06a2b0 2023-03-03T15:06:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc59d9b700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc44004850 0x7fcc44004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:14.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5a59c700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc44003aa0 0x7fcc44003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5ad9d700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c06a2b0 0x7fcc44005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc59d9b700 1 -- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c06a2b0 msgr2=0x7fcc44005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:06:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc59d9b700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c06a2b0 0x7fcc44005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc59d9b700 1 -- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc44003aa0 msgr2=0x7fcc44003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:06:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc59d9b700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc44003aa0 0x7fcc44003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc59d9b700 1 -- 172.21.15.7:0/3034919162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc440035b0 con 0x7fcc44004850 2023-03-03T15:06:14.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc59d9b700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc44004850 0x7fcc44004cc0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fcc500018d0 tx=0x7fcc44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97d0459b7606d89 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000c690 con 0x7fcc44004850 2023-03-03T15:06:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5000c7f0 con 0x7fcc44004850 2023-03-03T15:06:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.068+0000 7fcc5959a700 1 -- 172.21.15.7:0/3034919162 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000cba0 con 0x7fcc44004850 2023-03-03T15:06:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.069+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3034919162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc44004850 msgr2=0x7fcc44004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.069+0000 7fcc60a2c700 1 --2- 172.21.15.7:0/3034919162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc44004850 0x7fcc44004cc0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fcc500018d0 tx=0x7fcc44005880 comp rx=0 tx=0).stop 2023-03-03T15:06:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.069+0000 7fcc5ad9d700 1 -- 172.21.15.7:0/3034919162 reap_dead start 2023-03-03T15:06:14.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.069+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3034919162 shutdown_connections 2023-03-03T15:06:14.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.069+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3034919162 >> 172.21.15.7:0/3034919162 conn(0x7fcc5c0a9430 msgr2=0x7fcc5c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:06:14.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.069+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3034919162 shutdown_connections 2023-03-03T15:06:14.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.069+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3034919162 wait complete. 2023-03-03T15:06:14.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.070+0000 7fcc60a2c700 1 Processor -- start 2023-03-03T15:06:14.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.070+0000 7fcc60a2c700 1 -- start start 2023-03-03T15:06:14.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc60a2c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc44004850 0x7fcc5c1538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:14.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc60a2c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c0a6150 0x7fcc5c145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:14.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc60a2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc5c145f80 0x7fcc5c1ed710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:14.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc60a2c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c156300 con 0x7fcc5c145f80 2023-03-03T15:06:14.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc60a2c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0bfea0 con 0x7fcc44004850 2023-03-03T15:06:14.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc60a2c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0c0180 con 0x7fcc5c0a6150 2023-03-03T15:06:14.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc5ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc5c145f80 0x7fcc5c1ed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:14.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc5ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc5c145f80 0x7fcc5c1ed710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51630/0 (socket says 172.21.15.7:51630) 2023-03-03T15:06:14.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc5ad9d700 1 -- 172.21.15.7:0/3386424333 learned_addr learned my addr 172.21.15.7:0/3386424333 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:06:14.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc5a59c700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc44004850 0x7fcc5c1538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:14.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.071+0000 7fcc59d9b700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c0a6150 0x7fcc5c145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc5ad9d700 1 -- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c0a6150 msgr2=0x7fcc5c145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc5ad9d700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c0a6150 0x7fcc5c145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc5ad9d700 1 -- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc44004850 msgr2=0x7fcc5c1538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc5ad9d700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc44004850 0x7fcc5c1538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc5ad9d700 1 -- 172.21.15.7:0/3386424333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c0c2410 con 0x7fcc5c145f80 2023-03-03T15:06:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc5ad9d700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc5c145f80 0x7fcc5c1ed710 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c0c5560 tx=0x7fcc5400a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:14.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc4affd700 1 -- 172.21.15.7:0/3386424333 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc54001270 con 0x7fcc5c145f80 2023-03-03T15:06:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc4affd700 1 -- 172.21.15.7:0/3386424333 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5400ad50 con 0x7fcc5c145f80 2023-03-03T15:06:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3386424333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c0c2130 con 0x7fcc5c145f80 2023-03-03T15:06:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc4affd700 1 -- 172.21.15.7:0/3386424333 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc54013630 con 0x7fcc5c145f80 2023-03-03T15:06:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.072+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3386424333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c0c33a0 con 0x7fcc5c145f80 2023-03-03T15:06:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.073+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3386424333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c074bb0 con 0x7fcc5c145f80 2023-03-03T15:06:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.073+0000 7fcc4affd700 1 -- 172.21.15.7:0/3386424333 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcc54023070 con 0x7fcc5c145f80 2023-03-03T15:06:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.074+0000 7fcc4affd700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc3c0682f0 0x7fcc3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:14.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.074+0000 7fcc4affd700 1 -- 172.21.15.7:0/3386424333 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcc54086ad0 con 0x7fcc5c145f80 2023-03-03T15:06:14.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.074+0000 7fcc5a59c700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc3c0682f0 0x7fcc3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:14.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.075+0000 7fcc5a59c700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc3c0682f0 0x7fcc3c06a7a0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c072a90 tx=0x7fcc4c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:14.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.078+0000 7fcc4affd700 1 -- 172.21.15.7:0/3386424333 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcc54052e70 con 0x7fcc5c145f80 2023-03-03T15:06:14.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.318+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3386424333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc5c156440 con 0x7fcc5c145f80 2023-03-03T15:06:14.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.319+0000 7fcc4affd700 1 -- 172.21.15.7:0/3386424333 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcc54017090 con 0x7fcc5c145f80 2023-03-03T15:06:14.320 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:06:14.321 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:06:14.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc3c0682f0 msgr2=0x7fcc3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:14.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc3c0682f0 0x7fcc3c06a7a0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c072a90 tx=0x7fcc4c009210 comp rx=0 tx=0).stop 2023-03-03T15:06:14.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3386424333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc5c145f80 msgr2=0x7fcc5c1ed710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:14.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc5c145f80 0x7fcc5c1ed710 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c0c5560 tx=0x7fcc5400a640 comp rx=0 tx=0).stop 2023-03-03T15:06:14.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3386424333 shutdown_connections 2023-03-03T15:06:14.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcc3c0682f0 0x7fcc3c06a7a0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:14.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc5c145f80 0x7fcc5c1ed710 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:14.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc44004850 0x7fcc5c1538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:14.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 --2- 172.21.15.7:0/3386424333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc5c0a6150 0x7fcc5c145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:14.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3386424333 >> 172.21.15.7:0/3386424333 conn(0x7fcc5c0a9430 msgr2=0x7fcc5c152100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:06:14.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3386424333 shutdown_connections 2023-03-03T15:06:14.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:14.321+0000 7fcc60a2c700 1 -- 172.21.15.7:0/3386424333 wait complete. 2023-03-03T15:06:14.324 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:06:14.334 INFO:tasks.mds_thrash.fs.[a]:waiting for 57 secs before thrashing 2023-03-03T15:06:29.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:29.983 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:06:34.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:06:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.077+0000 7fb77a7c0700 1 Processor -- start 2023-03-03T15:06:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.077+0000 7fb77a7c0700 1 -- start start 2023-03-03T15:06:35.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb77a7c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414e900 0x7fb77415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:35.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb77a7c0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7741510d0 con 0x7fb774072250 2023-03-03T15:06:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb77a7c0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7740c3740 con 0x7fb774071e80 2023-03-03T15:06:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb77a7c0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb774070670 con 0x7fb77414e900 2023-03-03T15:06:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb7737fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb774072250 0x7fb77414e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45624/0 (socket says 172.21.15.7:45624) 2023-03-03T15:06:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb7737fe700 1 -- 172.21.15.7:0/2669592114 learned_addr learned my addr 172.21.15.7:0/2669592114 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:06:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb778d5d700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414e900 0x7fb77415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:35.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048065154 0 0) 0x7fb7741510d0 con 0x7fb774072250 2023-03-03T15:06:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7600035b0 con 0x7fb774072250 2023-03-03T15:06:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012434895 0 0) 0x7fb7740c3740 con 0x7fb774071e80 2023-03-03T15:06:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7741510d0 con 0x7fb774071e80 2023-03-03T15:06:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1548634220 0 0) 0x7fb7600035b0 con 0x7fb774072250 2023-03-03T15:06:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414e900 msgr2=0x7fb77415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:06:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.078+0000 7fb772ffd700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414e900 0x7fb77415ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 >> v1:172.21.15.143:6789/0 conn(0x7fb774071e80 legacy=0x7fb774151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7740c39e0 con 0x7fb774072250 2023-03-03T15:06:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb75c001670 con 0x7fb774072250 2023-03-03T15:06:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 >> v1:172.21.15.7:6789/0 conn(0x7fb774072250 legacy=0x7fb77414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 0x7fb760004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414e900 0x7fb760004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb760004d90 0x7fb760005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb774070670 con 0x7fb760003c30 2023-03-03T15:06:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76000bba0 con 0x7fb760004d90 2023-03-03T15:06:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76000bd30 con 0x7fb77414e900 2023-03-03T15:06:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb773fff700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 0x7fb760004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb778d5d700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb760004d90 0x7fb760005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb7737fe700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414e900 0x7fb760004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb773fff700 1 -- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414e900 msgr2=0x7fb760004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:06:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb773fff700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414e900 0x7fb760004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb773fff700 1 -- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb760004d90 msgr2=0x7fb760005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:06:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb773fff700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb760004d90 0x7fb760005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb773fff700 1 -- 172.21.15.7:0/2669592114 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7600035b0 con 0x7fb760003c30 2023-03-03T15:06:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb778d5d700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb760004d90 0x7fb760005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:06:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb7737fe700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414e900 0x7fb760004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:06:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb773fff700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 0x7fb760004020 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fb764001980 tx=0x7fb760005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=affc2a27687927a0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764001470 con 0x7fb760003c30 2023-03-03T15:06:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7640052d0 con 0x7fb760003c30 2023-03-03T15:06:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.079+0000 7fb772ffd700 1 -- 172.21.15.7:0/2669592114 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764003680 con 0x7fb760003c30 2023-03-03T15:06:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.080+0000 7fb77a7c0700 1 -- 172.21.15.7:0/2669592114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 msgr2=0x7fb760004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.080+0000 7fb77a7c0700 1 --2- 172.21.15.7:0/2669592114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 0x7fb760004020 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fb764001980 tx=0x7fb760005880 comp rx=0 tx=0).stop 2023-03-03T15:06:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.080+0000 7fb778d5d700 1 -- 172.21.15.7:0/2669592114 reap_dead start 2023-03-03T15:06:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.080+0000 7fb77a7c0700 1 -- 172.21.15.7:0/2669592114 shutdown_connections 2023-03-03T15:06:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.080+0000 7fb77a7c0700 1 -- 172.21.15.7:0/2669592114 >> 172.21.15.7:0/2669592114 conn(0x7fb7740a94d0 msgr2=0x7fb7740a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:06:35.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.080+0000 7fb77a7c0700 1 -- 172.21.15.7:0/2669592114 shutdown_connections 2023-03-03T15:06:35.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.080+0000 7fb77a7c0700 1 -- 172.21.15.7:0/2669592114 wait complete. 2023-03-03T15:06:35.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.081+0000 7fb77a7c0700 1 Processor -- start 2023-03-03T15:06:35.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.081+0000 7fb77a7c0700 1 -- start start 2023-03-03T15:06:35.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb77a7c0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 0x7fb7740c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:35.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb77a7c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414c3a0 0x7fb77414c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:35.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb77a7c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7741ed1d0 0x7fb7741ed670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:35.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb77a7c0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7741407c0 con 0x7fb760003c30 2023-03-03T15:06:35.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb77a7c0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7741eb140 con 0x7fb7741ed1d0 2023-03-03T15:06:35.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb77a7c0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77414bf10 con 0x7fb77414c3a0 2023-03-03T15:06:35.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb773fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 0x7fb7740c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:35.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb773fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 0x7fb7740c6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51654/0 (socket says 172.21.15.7:51654) 2023-03-03T15:06:35.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb773fff700 1 -- 172.21.15.7:0/4283781078 learned_addr learned my addr 172.21.15.7:0/4283781078 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:06:35.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb7737fe700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414c3a0 0x7fb77414c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:35.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.082+0000 7fb778d5d700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7741ed1d0 0x7fb7741ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:35.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb773fff700 1 -- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414c3a0 msgr2=0x7fb77414c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:35.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb773fff700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414c3a0 0x7fb77414c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:35.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb773fff700 1 -- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7741ed1d0 msgr2=0x7fb7741ed670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:35.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb773fff700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7741ed1d0 0x7fb7741ed670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:35.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb773fff700 1 -- 172.21.15.7:0/4283781078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb774151f30 con 0x7fb760003c30 2023-03-03T15:06:35.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb773fff700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 0x7fb7740c6930 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fb764004000 tx=0x7fb764001310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:35.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb770ff9700 1 -- 172.21.15.7:0/4283781078 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764001110 con 0x7fb760003c30 2023-03-03T15:06:35.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb770ff9700 1 -- 172.21.15.7:0/4283781078 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb764001410 con 0x7fb760003c30 2023-03-03T15:06:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb77a7c0700 1 -- 172.21.15.7:0/4283781078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb774151300 con 0x7fb760003c30 2023-03-03T15:06:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb770ff9700 1 -- 172.21.15.7:0/4283781078 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764007260 con 0x7fb760003c30 2023-03-03T15:06:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.083+0000 7fb77a7c0700 1 -- 172.21.15.7:0/4283781078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77406c9a0 con 0x7fb760003c30 2023-03-03T15:06:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.084+0000 7fb77a7c0700 1 -- 172.21.15.7:0/4283781078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb774074bb0 con 0x7fb760003c30 2023-03-03T15:06:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.084+0000 7fb770ff9700 1 -- 172.21.15.7:0/4283781078 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb76402e070 con 0x7fb760003c30 2023-03-03T15:06:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.085+0000 7fb770ff9700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7540682f0 0x7fb75406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:06:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.085+0000 7fb770ff9700 1 -- 172.21.15.7:0/4283781078 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb764086ee0 con 0x7fb760003c30 2023-03-03T15:06:35.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.085+0000 7fb7737fe700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7540682f0 0x7fb75406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:06:35.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.086+0000 7fb7737fe700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7540682f0 0x7fb75406a7a0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fb7741ed130 tx=0x7fb75c007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:06:35.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.090+0000 7fb770ff9700 1 -- 172.21.15.7:0/4283781078 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb764053330 con 0x7fb760003c30 2023-03-03T15:06:35.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.325+0000 7fb77a7c0700 1 -- 172.21.15.7:0/4283781078 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7740a9fb0 con 0x7fb760003c30 2023-03-03T15:06:35.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.325+0000 7fb770ff9700 1 -- 172.21.15.7:0/4283781078 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb764056950 con 0x7fb760003c30 2023-03-03T15:06:35.327 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:06:35.328 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:06:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 -- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7540682f0 msgr2=0x7fb75406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:35.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7540682f0 0x7fb75406a7a0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fb7741ed130 tx=0x7fb75c007040 comp rx=0 tx=0).stop 2023-03-03T15:06:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 -- 172.21.15.7:0/4283781078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 msgr2=0x7fb7740c6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:06:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 0x7fb7740c6930 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fb764004000 tx=0x7fb764001310 comp rx=0 tx=0).stop 2023-03-03T15:06:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 -- 172.21.15.7:0/4283781078 shutdown_connections 2023-03-03T15:06:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7741ed1d0 0x7fb7741ed670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7540682f0 0x7fb75406a7a0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb760003c30 0x7fb7740c6930 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 --2- 172.21.15.7:0/4283781078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb77414c3a0 0x7fb77414c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:06:35.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 -- 172.21.15.7:0/4283781078 >> 172.21.15.7:0/4283781078 conn(0x7fb7740a94d0 msgr2=0x7fb7740c2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:06:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 -- 172.21.15.7:0/4283781078 shutdown_connections 2023-03-03T15:06:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:06:35.328+0000 7fb77a7c0700 1 -- 172.21.15.7:0/4283781078 wait complete. 2023-03-03T15:06:35.331 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:06:35.341 INFO:tasks.mds_thrash.fs.[b]:waiting for 52 secs before thrashing 2023-03-03T15:07:00.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:00.048 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:11.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:07:11.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.471+0000 7f25a08c6700 1 Processor -- start 2023-03-03T15:07:11.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.471+0000 7f25a08c6700 1 -- start start 2023-03-03T15:07:11.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f25a08c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14f9d0 0x7f259c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:11.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f25a08c6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259c0c2f30 con 0x7f259c14fda0 2023-03-03T15:07:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f25a08c6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259c0c31d0 con 0x7f259c0abe20 2023-03-03T15:07:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f25a08c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c1524a0 con 0x7f259c14f9d0 2023-03-03T15:07:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f2599d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f259c14fda0 0x7f259c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45664/0 (socket says 172.21.15.7:45664) 2023-03-03T15:07:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f2599d9b700 1 -- 172.21.15.7:0/3049920383 learned_addr learned my addr 172.21.15.7:0/3049920383 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:07:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553492154 0 0) 0x7f259c0c2f30 con 0x7f259c14fda0 2023-03-03T15:07:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25840035b0 con 0x7f259c14fda0 2023-03-03T15:07:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259a59c700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14f9d0 0x7f259c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:11.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3412579972 0 0) 0x7f25840035b0 con 0x7f259c14fda0 2023-03-03T15:07:11.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14f9d0 msgr2=0x7f259c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:07:11.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14f9d0 0x7f259c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:11.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 >> v1:172.21.15.143:6789/0 conn(0x7f259c0abe20 legacy=0x7f259c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:11.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c0c34b0 con 0x7f259c14fda0 2023-03-03T15:07:11.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f2590001670 con 0x7f259c14fda0 2023-03-03T15:07:11.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 >> v1:172.21.15.7:6789/0 conn(0x7f259c14fda0 legacy=0x7f259c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:11.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 0x7f2584003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:11.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14f9d0 0x7f2584004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:11.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2584004d90 0x7f2584005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:11.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c1524a0 con 0x7f2584003aa0 2023-03-03T15:07:11.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258400bc90 con 0x7f2584004d90 2023-03-03T15:07:11.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.472+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f258400be50 con 0x7f259c14f9d0 2023-03-03T15:07:11.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259a59c700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 0x7f2584003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f2599d9b700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14f9d0 0x7f2584004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259ad9d700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2584004d90 0x7f2584005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259a59c700 1 -- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14f9d0 msgr2=0x7f2584004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:07:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259a59c700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14f9d0 0x7f2584004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259a59c700 1 -- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2584004d90 msgr2=0x7f2584005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:07:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259a59c700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2584004d90 0x7f2584005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259a59c700 1 -- 172.21.15.7:0/3049920383 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25840035b0 con 0x7f2584003aa0 2023-03-03T15:07:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259a59c700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 0x7f2584003eb0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f258c009b60 tx=0x7f2584005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe6e927e2104688 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:07:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c00e6c0 con 0x7f2584003aa0 2023-03-03T15:07:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c00e820 con 0x7f2584003aa0 2023-03-03T15:07:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.473+0000 7f259959a700 1 -- 172.21.15.7:0/3049920383 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c00eb90 con 0x7f2584003aa0 2023-03-03T15:07:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.474+0000 7f25a08c6700 1 -- 172.21.15.7:0/3049920383 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 msgr2=0x7f2584003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.474+0000 7f25a08c6700 1 --2- 172.21.15.7:0/3049920383 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 0x7f2584003eb0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f258c009b60 tx=0x7f2584005880 comp rx=0 tx=0).stop 2023-03-03T15:07:11.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.474+0000 7f259ad9d700 1 -- 172.21.15.7:0/3049920383 reap_dead start 2023-03-03T15:07:11.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.474+0000 7f25a08c6700 1 -- 172.21.15.7:0/3049920383 shutdown_connections 2023-03-03T15:07:11.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.474+0000 7f25a08c6700 1 -- 172.21.15.7:0/3049920383 >> 172.21.15.7:0/3049920383 conn(0x7f259c0c8900 msgr2=0x7f259c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:07:11.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.474+0000 7f25a08c6700 1 -- 172.21.15.7:0/3049920383 shutdown_connections 2023-03-03T15:07:11.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.474+0000 7f25a08c6700 1 -- 172.21.15.7:0/3049920383 wait complete. 2023-03-03T15:07:11.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.475+0000 7f25a08c6700 1 Processor -- start 2023-03-03T15:07:11.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.475+0000 7f25a08c6700 1 -- start start 2023-03-03T15:07:11.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f25a08c6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 0x7f259c14d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f25a08c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f259c0c26c0 0x7f259c14aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f25a08c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14afd0 0x7f259c153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f25a08c6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c1ead00 con 0x7f2584003aa0 2023-03-03T15:07:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f25a08c6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c1eafe0 con 0x7f259c0c26c0 2023-03-03T15:07:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f25a08c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259c1eb2c0 con 0x7f259c14afd0 2023-03-03T15:07:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f259a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 0x7f259c14d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f259a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 0x7f259c14d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51694/0 (socket says 172.21.15.7:51694) 2023-03-03T15:07:11.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f259a59c700 1 -- 172.21.15.7:0/712608757 learned_addr learned my addr 172.21.15.7:0/712608757 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:07:11.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f259ad9d700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14afd0 0x7f259c153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:11.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.476+0000 7f2599d9b700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f259c0c26c0 0x7f259c14aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:11.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f259a59c700 1 -- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14afd0 msgr2=0x7f259c153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:11.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f259a59c700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14afd0 0x7f259c153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:11.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f259a59c700 1 -- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f259c0c26c0 msgr2=0x7f259c14aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:11.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f259a59c700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f259c0c26c0 0x7f259c14aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:11.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f259a59c700 1 -- 172.21.15.7:0/712608757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c0ac350 con 0x7f2584003aa0 2023-03-03T15:07:11.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f259a59c700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 0x7f259c14d090 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f258400bc60 tx=0x7f258c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:07:11.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f258affd700 1 -- 172.21.15.7:0/712608757 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c00e6c0 con 0x7f2584003aa0 2023-03-03T15:07:11.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f258affd700 1 -- 172.21.15.7:0/712608757 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c0038c0 con 0x7f2584003aa0 2023-03-03T15:07:11.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f258affd700 1 -- 172.21.15.7:0/712608757 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c003c70 con 0x7f2584003aa0 2023-03-03T15:07:11.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f25a08c6700 1 -- 172.21.15.7:0/712608757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c0ab630 con 0x7f2584003aa0 2023-03-03T15:07:11.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.477+0000 7f25a08c6700 1 -- 172.21.15.7:0/712608757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259c071fa0 con 0x7f2584003aa0 2023-03-03T15:07:11.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.478+0000 7f25a08c6700 1 -- 172.21.15.7:0/712608757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259c153c80 con 0x7f2584003aa0 2023-03-03T15:07:11.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.478+0000 7f258affd700 1 -- 172.21.15.7:0/712608757 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f258c025070 con 0x7f2584003aa0 2023-03-03T15:07:11.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.479+0000 7f258affd700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f257c0682f0 0x7f257c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:11.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.479+0000 7f258affd700 1 -- 172.21.15.7:0/712608757 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f258c0877b0 con 0x7f2584003aa0 2023-03-03T15:07:11.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.479+0000 7f2599d9b700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f257c0682f0 0x7f257c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:11.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.479+0000 7f2599d9b700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f257c0682f0 0x7f257c06a7a0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f259c153400 tx=0x7f2590000f80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:07:11.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.483+0000 7f258affd700 1 -- 172.21.15.7:0/712608757 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f258c053b50 con 0x7f2584003aa0 2023-03-03T15:07:11.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.725+0000 7f25a08c6700 1 -- 172.21.15.7:0/712608757 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f259c074bb0 con 0x7f2584003aa0 2023-03-03T15:07:11.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.725+0000 7f258affd700 1 -- 172.21.15.7:0/712608757 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f258c057170 con 0x7f2584003aa0 2023-03-03T15:07:11.728 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:07:11.728 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:07:11.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.727+0000 7f25a08c6700 1 -- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f257c0682f0 msgr2=0x7f257c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:11.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.727+0000 7f25a08c6700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f257c0682f0 0x7f257c06a7a0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f259c153400 tx=0x7f2590000f80 comp rx=0 tx=0).stop 2023-03-03T15:07:11.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.727+0000 7f25a08c6700 1 -- 172.21.15.7:0/712608757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 msgr2=0x7f259c14d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:11.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.727+0000 7f25a08c6700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 0x7f259c14d090 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f258400bc60 tx=0x7f258c0047c0 comp rx=0 tx=0).stop 2023-03-03T15:07:11.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.728+0000 7f25a08c6700 1 -- 172.21.15.7:0/712608757 shutdown_connections 2023-03-03T15:07:11.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.728+0000 7f25a08c6700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f259c14afd0 0x7f259c153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:11.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.728+0000 7f25a08c6700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f257c0682f0 0x7f257c06a7a0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:11.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.728+0000 7f25a08c6700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2584003aa0 0x7f259c14d090 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:11.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.728+0000 7f25a08c6700 1 --2- 172.21.15.7:0/712608757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f259c0c26c0 0x7f259c14aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:11.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.728+0000 7f25a08c6700 1 -- 172.21.15.7:0/712608757 >> 172.21.15.7:0/712608757 conn(0x7f259c0c8900 msgr2=0x7f259c150d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:07:11.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.728+0000 7f25a08c6700 1 -- 172.21.15.7:0/712608757 shutdown_connections 2023-03-03T15:07:11.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:11.728+0000 7f25a08c6700 1 -- 172.21.15.7:0/712608757 wait complete. 2023-03-03T15:07:11.732 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:07:11.742 INFO:tasks.mds_thrash.fs.[a]:waiting for 72 secs before thrashing 2023-03-03T15:07:27.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:07:27.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.481+0000 7fe11971b700 1 Processor -- start 2023-03-03T15:07:27.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.482+0000 7fe11971b700 1 -- start start 2023-03-03T15:07:27.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe11971b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe1140c5c60 0x7fe114154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:27.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe11971b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1140c7310 con 0x7fe1140caa30 2023-03-03T15:07:27.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe11971b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1140bfd10 con 0x7fe1140c00e0 2023-03-03T15:07:27.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe11971b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1140c60a0 con 0x7fe1140c5c60 2023-03-03T15:07:27.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe112ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe1140caa30 0x7fe1140c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45694/0 (socket says 172.21.15.7:45694) 2023-03-03T15:07:27.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe112ffd700 1 -- 172.21.15.7:0/2291301046 learned_addr learned my addr 172.21.15.7:0/2291301046 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:07:27.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe1137fe700 1 --2- 172.21.15.7:0/2291301046 >> v2:172.21.15.143:3300/0 conn(0x7fe1140c5c60 0x7fe114154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:27.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612251188 0 0) 0x7fe1140c7310 con 0x7fe1140caa30 2023-03-03T15:07:27.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1000035b0 con 0x7fe1140caa30 2023-03-03T15:07:27.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2531195966 0 0) 0x7fe1000035b0 con 0x7fe1140caa30 2023-03-03T15:07:27.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 >> v2:172.21.15.143:3300/0 conn(0x7fe1140c5c60 msgr2=0x7fe114154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:07:27.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe111ffb700 1 --2- 172.21.15.7:0/2291301046 >> v2:172.21.15.143:3300/0 conn(0x7fe1140c5c60 0x7fe114154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:27.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 >> v1:172.21.15.143:6789/0 conn(0x7fe1140c00e0 legacy=0x7fe11415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:27.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1140bfa30 con 0x7fe1140caa30 2023-03-03T15:07:27.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.483+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe104001670 con 0x7fe1140caa30 2023-03-03T15:07:27.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 >> v1:172.21.15.7:6789/0 conn(0x7fe1140caa30 legacy=0x7fe1140c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:27.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe111ffb700 1 --2- 172.21.15.7:0/2291301046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe100003a70 0x7fe100003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:27.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe111ffb700 1 --2- 172.21.15.7:0/2291301046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe100004850 0x7fe100004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:27.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe111ffb700 1 --2- 172.21.15.7:0/2291301046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe100005250 0x7fe10000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:27.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10000c160 con 0x7fe100004850 2023-03-03T15:07:27.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10000c2d0 con 0x7fe100005250 2023-03-03T15:07:27.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10000c490 con 0x7fe100003a70 2023-03-03T15:07:27.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe1127fc700 1 --2- 172.21.15.7:0/2291301046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe100004850 0x7fe100004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:27.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe112ffd700 1 --2- 172.21.15.7:0/2291301046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe100003a70 0x7fe100003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:27.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe1127fc700 1 -- 172.21.15.7:0/2291301046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe100003a70 msgr2=0x7fe100003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:07:27.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe1127fc700 1 --2- 172.21.15.7:0/2291301046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe100003a70 0x7fe100003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:27.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe1127fc700 1 -- 172.21.15.7:0/2291301046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe100005250 msgr2=0x7fe10000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:07:27.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe1127fc700 1 --2- 172.21.15.7:0/2291301046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe100005250 0x7fe10000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:27.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe1127fc700 1 -- 172.21.15.7:0/2291301046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1000035b0 con 0x7fe100004850 2023-03-03T15:07:27.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe1137fe700 1 -- 172.21.15.7:0/2291301046 reap_dead start 2023-03-03T15:07:27.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe1127fc700 1 --2- 172.21.15.7:0/2291301046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe100004850 0x7fe100004cc0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0017f0 tx=0x7fe100005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62dd959bf7cfc24 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:07:27.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc0012e0 con 0x7fe100004850 2023-03-03T15:07:27.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc005220 con 0x7fe100004850 2023-03-03T15:07:27.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.484+0000 7fe111ffb700 1 -- 172.21.15.7:0/2291301046 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc0055d0 con 0x7fe100004850 2023-03-03T15:07:27.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.485+0000 7fe11971b700 1 -- 172.21.15.7:0/2291301046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe100004850 msgr2=0x7fe100004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:27.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.485+0000 7fe11971b700 1 --2- 172.21.15.7:0/2291301046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe100004850 0x7fe100004cc0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0017f0 tx=0x7fe100005880 comp rx=0 tx=0).stop 2023-03-03T15:07:27.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.485+0000 7fe11971b700 1 -- 172.21.15.7:0/2291301046 shutdown_connections 2023-03-03T15:07:27.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.485+0000 7fe11971b700 1 --2- 172.21.15.7:0/2291301046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe100004850 0x7fe100004cc0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:27.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.485+0000 7fe11971b700 1 -- 172.21.15.7:0/2291301046 >> 172.21.15.7:0/2291301046 conn(0x7fe114066e40 msgr2=0x7fe114071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:07:27.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.485+0000 7fe11971b700 1 -- 172.21.15.7:0/2291301046 shutdown_connections 2023-03-03T15:07:27.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.485+0000 7fe11971b700 1 -- 172.21.15.7:0/2291301046 wait complete. 2023-03-03T15:07:27.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.486+0000 7fe11971b700 1 Processor -- start 2023-03-03T15:07:27.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.486+0000 7fe11971b700 1 -- start start 2023-03-03T15:07:27.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe11971b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe100004850 0x7fe1141ddcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:27.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe11971b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe11414d720 0x7fe11414cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:27.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe11971b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe11414d040 0x7fe1140c4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:27.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe11971b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1141e1740 con 0x7fe11414d040 2023-03-03T15:07:27.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe11971b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1141eade0 con 0x7fe100004850 2023-03-03T15:07:27.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe11971b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1140c3310 con 0x7fe11414d720 2023-03-03T15:07:27.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe1137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe11414d040 0x7fe1140c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:27.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe1137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe11414d040 0x7fe1140c4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51730/0 (socket says 172.21.15.7:51730) 2023-03-03T15:07:27.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe1137fe700 1 -- 172.21.15.7:0/2332651213 learned_addr learned my addr 172.21.15.7:0/2332651213 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:07:27.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe112ffd700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe100004850 0x7fe1141ddcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe1127fc700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe11414d720 0x7fe11414cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.487+0000 7fe1137fe700 1 -- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe11414d720 msgr2=0x7fe11414cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe1137fe700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe11414d720 0x7fe11414cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe1137fe700 1 -- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe100004850 msgr2=0x7fe1141ddcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe1137fe700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe100004850 0x7fe1141ddcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe1137fe700 1 -- 172.21.15.7:0/2332651213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1141547f0 con 0x7fe11414d040 2023-03-03T15:07:27.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe112ffd700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe100004850 0x7fe1141ddcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:07:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe1137fe700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe11414d040 0x7fe1140c4030 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fe1140c8180 tx=0x7fe10c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:07:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe10b7fe700 1 -- 172.21.15.7:0/2332651213 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c010040 con 0x7fe11414d040 2023-03-03T15:07:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe10b7fe700 1 -- 172.21.15.7:0/2332651213 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c00b980 con 0x7fe11414d040 2023-03-03T15:07:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe11971b700 1 -- 172.21.15.7:0/2332651213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe114154510 con 0x7fe11414d040 2023-03-03T15:07:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe10b7fe700 1 -- 172.21.15.7:0/2332651213 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00bd30 con 0x7fe11414d040 2023-03-03T15:07:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.488+0000 7fe11971b700 1 -- 172.21.15.7:0/2332651213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe114154ad0 con 0x7fe11414d040 2023-03-03T15:07:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.489+0000 7fe11971b700 1 -- 172.21.15.7:0/2332651213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe114074bb0 con 0x7fe11414d040 2023-03-03T15:07:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.489+0000 7fe10b7fe700 1 -- 172.21.15.7:0/2332651213 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe10c01d400 con 0x7fe11414d040 2023-03-03T15:07:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.490+0000 7fe10b7fe700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0f40682f0 0x7fe0f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:07:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.490+0000 7fe10b7fe700 1 -- 172.21.15.7:0/2332651213 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe10c086890 con 0x7fe11414d040 2023-03-03T15:07:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.490+0000 7fe112ffd700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0f40682f0 0x7fe0f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:07:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.491+0000 7fe112ffd700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0f40682f0 0x7fe0f406a7a0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fe104007dd0 tx=0x7fe104009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:07:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.495+0000 7fe10b7fe700 1 -- 172.21.15.7:0/2332651213 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe10c052bd0 con 0x7fe11414d040 2023-03-03T15:07:27.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.730+0000 7fe11971b700 1 -- 172.21.15.7:0/2332651213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1140c6f50 con 0x7fe11414d040 2023-03-03T15:07:27.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.731+0000 7fe10b7fe700 1 -- 172.21.15.7:0/2332651213 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe10c0561f0 con 0x7fe11414d040 2023-03-03T15:07:27.733 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:07:27.733 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:07:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 -- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0f40682f0 msgr2=0x7fe0f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0f40682f0 0x7fe0f406a7a0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fe104007dd0 tx=0x7fe104009040 comp rx=0 tx=0).stop 2023-03-03T15:07:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 -- 172.21.15.7:0/2332651213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe11414d040 msgr2=0x7fe1140c4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:07:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe11414d040 0x7fe1140c4030 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fe1140c8180 tx=0x7fe10c00ee50 comp rx=0 tx=0).stop 2023-03-03T15:07:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 -- 172.21.15.7:0/2332651213 shutdown_connections 2023-03-03T15:07:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0f40682f0 0x7fe0f406a7a0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe11414d040 0x7fe1140c4030 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe100004850 0x7fe1141ddcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:27.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 --2- 172.21.15.7:0/2332651213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe11414d720 0x7fe11414cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:07:27.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 -- 172.21.15.7:0/2332651213 >> 172.21.15.7:0/2332651213 conn(0x7fe114066e40 msgr2=0x7fe114153b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:07:27.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 -- 172.21.15.7:0/2332651213 shutdown_connections 2023-03-03T15:07:27.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:07:27.733+0000 7fe11971b700 1 -- 172.21.15.7:0/2332651213 wait complete. 2023-03-03T15:07:27.736 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:07:27.746 INFO:tasks.mds_thrash.fs.[b]:waiting for 101 secs before thrashing 2023-03-03T15:07:30.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:07:30.128 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:00.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:00.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:23.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:08:23.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.881+0000 7f1634e47700 1 Processor -- start 2023-03-03T15:08:23.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.881+0000 7f1634e47700 1 -- start start 2023-03-03T15:08:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f1634e47700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16300c7760 0x7f163015ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f1634e47700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16300c6cf0 con 0x7f16300b00b0 2023-03-03T15:08:23.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f1634e47700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16301548f0 con 0x7f1630151cc0 2023-03-03T15:08:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f1634e47700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1630152090 con 0x7f16300c7760 2023-03-03T15:08:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f162ed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16300c7760 0x7f163015ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f162dd9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f16300b00b0 0x7f16300c7050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45748/0 (socket says 172.21.15.7:45748) 2023-03-03T15:08:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f162dd9b700 1 -- 172.21.15.7:0/1465849655 learned_addr learned my addr 172.21.15.7:0/1465849655 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:08:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f162ed9d700 1 -- 172.21.15.7:0/1465849655 >> v1:172.21.15.143:6789/0 conn(0x7f1630151cc0 legacy=0x7f16300ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:08:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f162ed9d700 1 -- 172.21.15.7:0/1465849655 >> v1:172.21.15.7:6789/0 conn(0x7f16300b00b0 legacy=0x7f16300c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:23.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f162d59a700 1 -- 172.21.15.7:0/1465849655 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298181479 0 0) 0x7f16300c6cf0 con 0x7f16300b00b0 2023-03-03T15:08:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.882+0000 7f162ed9d700 1 -- 172.21.15.7:0/1465849655 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1630154bd0 con 0x7f16300c7760 2023-03-03T15:08:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f162ed9d700 1 --2- 172.21.15.7:0/1465849655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c7760 0x7f163015ff30 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f1628009d90 tx=0x7f162800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1690a6228b07d35 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:08:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f162d59a700 1 -- 172.21.15.7:0/1465849655 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162800e070 con 0x7f16300c7760 2023-03-03T15:08:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f162d59a700 1 -- 172.21.15.7:0/1465849655 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162800bb90 con 0x7f16300c7760 2023-03-03T15:08:23.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f162d59a700 1 -- 172.21.15.7:0/1465849655 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628003a20 con 0x7f16300c7760 2023-03-03T15:08:23.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f1634e47700 1 -- 172.21.15.7:0/1465849655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c7760 msgr2=0x7f163015ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:23.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f1634e47700 1 --2- 172.21.15.7:0/1465849655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c7760 0x7f163015ff30 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f1628009d90 tx=0x7f162800b330 comp rx=0 tx=0).stop 2023-03-03T15:08:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f1634e47700 1 -- 172.21.15.7:0/1465849655 shutdown_connections 2023-03-03T15:08:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f1634e47700 1 --2- 172.21.15.7:0/1465849655 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300c7760 0x7f163015ff30 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f1634e47700 1 -- 172.21.15.7:0/1465849655 >> 172.21.15.7:0/1465849655 conn(0x7f163006b980 msgr2=0x7f1630071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:08:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f1634e47700 1 -- 172.21.15.7:0/1465849655 shutdown_connections 2023-03-03T15:08:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.883+0000 7f1634e47700 1 -- 172.21.15.7:0/1465849655 wait complete. 2023-03-03T15:08:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.885+0000 7f1634e47700 1 Processor -- start 2023-03-03T15:08:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.885+0000 7f1634e47700 1 -- start start 2023-03-03T15:08:23.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.885+0000 7f1634e47700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300b00b0 0x7f163014d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.885+0000 7f1634e47700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16300c7760 0x7f1630157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.885+0000 7f1634e47700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cc0 0x7f163014bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.885+0000 7f1634e47700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16301edcf0 con 0x7f16300b00b0 2023-03-03T15:08:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.885+0000 7f1634e47700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16301ed400 con 0x7f1630151cc0 2023-03-03T15:08:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.885+0000 7f1634e47700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16301ed6a0 con 0x7f16300c7760 2023-03-03T15:08:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.885+0000 7f162e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300b00b0 0x7f163014d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:23.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300b00b0 0x7f163014d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51772/0 (socket says 172.21.15.7:51772) 2023-03-03T15:08:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162e59c700 1 -- 172.21.15.7:0/775297881 learned_addr learned my addr 172.21.15.7:0/775297881 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:08:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162ed9d700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cc0 0x7f163014bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162e59c700 1 -- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16300c7760 msgr2=0x7f1630157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162dd9b700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16300c7760 0x7f1630157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162e59c700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16300c7760 0x7f1630157760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162e59c700 1 -- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cc0 msgr2=0x7f163014bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162e59c700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cc0 0x7f163014bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:23.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162e59c700 1 -- 172.21.15.7:0/775297881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1628009a70 con 0x7f16300b00b0 2023-03-03T15:08:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162ed9d700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cc0 0x7f163014bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:08:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f162e59c700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300b00b0 0x7f163014d6d0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f162000b4a0 tx=0x7f162000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:08:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f161effd700 1 -- 172.21.15.7:0/775297881 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620011830 con 0x7f16300b00b0 2023-03-03T15:08:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f161effd700 1 -- 172.21.15.7:0/775297881 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1620011990 con 0x7f16300b00b0 2023-03-03T15:08:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f1634e47700 1 -- 172.21.15.7:0/775297881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1630155190 con 0x7f16300b00b0 2023-03-03T15:08:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f161effd700 1 -- 172.21.15.7:0/775297881 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620011d40 con 0x7f16300b00b0 2023-03-03T15:08:23.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.886+0000 7f1634e47700 1 -- 172.21.15.7:0/775297881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1630154eb0 con 0x7f16300b00b0 2023-03-03T15:08:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.887+0000 7f1634e47700 1 -- 172.21.15.7:0/775297881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16301545a0 con 0x7f16300b00b0 2023-03-03T15:08:23.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.888+0000 7f161effd700 1 -- 172.21.15.7:0/775297881 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1620021070 con 0x7f16300b00b0 2023-03-03T15:08:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.888+0000 7f161effd700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 0x7f161006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:08:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.888+0000 7f161effd700 1 -- 172.21.15.7:0/775297881 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1620086010 con 0x7f16300b00b0 2023-03-03T15:08:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.888+0000 7f162dd9b700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 0x7f161006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:08:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.889+0000 7f162dd9b700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 0x7f161006a7a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1630157010 tx=0x7f1624009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:08:23.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:23.892+0000 7f161effd700 1 -- 172.21.15.7:0/775297881 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1620052350 con 0x7f16300b00b0 2023-03-03T15:08:24.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.130+0000 7f1634e47700 1 -- 172.21.15.7:0/775297881 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16301edfd0 con 0x7f16300b00b0 2023-03-03T15:08:24.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.131+0000 7f161effd700 1 -- 172.21.15.7:0/775297881 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1620055970 con 0x7f16300b00b0 2023-03-03T15:08:24.133 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:08:24.133 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:08:24.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 -- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 msgr2=0x7f161006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:24.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 0x7f161006a7a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1630157010 tx=0x7f1624009040 comp rx=0 tx=0).stop 2023-03-03T15:08:24.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 -- 172.21.15.7:0/775297881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300b00b0 msgr2=0x7f163014d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:08:24.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300b00b0 0x7f163014d6d0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f162000b4a0 tx=0x7f162000b860 comp rx=0 tx=0).stop 2023-03-03T15:08:24.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 -- 172.21.15.7:0/775297881 shutdown_connections 2023-03-03T15:08:24.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1630151cc0 0x7f163014bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:24.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16100682f0 0x7f161006a7a0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:24.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16300b00b0 0x7f163014d6d0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:24.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 --2- 172.21.15.7:0/775297881 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16300c7760 0x7f1630157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:08:24.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 -- 172.21.15.7:0/775297881 >> 172.21.15.7:0/775297881 conn(0x7f163006b980 msgr2=0x7f1630065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:08:24.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 -- 172.21.15.7:0/775297881 shutdown_connections 2023-03-03T15:08:24.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:08:24.133+0000 7f1634e47700 1 -- 172.21.15.7:0/775297881 wait complete. 2023-03-03T15:08:24.136 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:08:24.146 INFO:tasks.mds_thrash.fs.[a]:waiting for 90 secs before thrashing 2023-03-03T15:08:30.258 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:08:30.260 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:00.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:00.319 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:08.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:09:08.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.885+0000 7fa2c418a700 1 Processor -- start 2023-03-03T15:09:08.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.885+0000 7fa2c418a700 1 -- start start 2023-03-03T15:09:08.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.885+0000 7fa2c418a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa2bc14f9d0 0x7fa2bc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:08.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c418a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa2bc0abe20 0x7fa2bc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:08.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c418a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2bc0c2f30 con 0x7fa2bc14fda0 2023-03-03T15:09:08.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c418a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2bc0c31d0 con 0x7fa2bc14f9d0 2023-03-03T15:09:08.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c418a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2bc1524a0 con 0x7fa2bc0abe20 2023-03-03T15:09:08.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c1f26700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa2bc14f9d0 0x7fa2bc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:08.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c1725700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa2bc14fda0 0x7fa2bc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45792/0 (socket says 172.21.15.7:45792) 2023-03-03T15:09:08.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c1725700 1 -- 172.21.15.7:0/1988415279 learned_addr learned my addr 172.21.15.7:0/1988415279 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:09:08.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c2727700 1 --2- 172.21.15.7:0/1988415279 >> v2:172.21.15.143:3300/0 conn(0x7fa2bc0abe20 0x7fa2bc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:08.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c1f26700 1 -- 172.21.15.7:0/1988415279 >> v2:172.21.15.143:3300/0 conn(0x7fa2bc0abe20 msgr2=0x7fa2bc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:09:08.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c1f26700 1 --2- 172.21.15.7:0/1988415279 >> v2:172.21.15.143:3300/0 conn(0x7fa2bc0abe20 0x7fa2bc1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:08.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c1f26700 1 -- 172.21.15.7:0/1988415279 >> v1:172.21.15.7:6789/0 conn(0x7fa2bc14fda0 legacy=0x7fa2bc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:08.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c1f26700 1 -- 172.21.15.7:0/1988415279 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc0c34b0 con 0x7fa2bc14f9d0 2023-03-03T15:09:08.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c0f24700 1 -- 172.21.15.7:0/1988415279 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433731483 0 0) 0x7fa2bc0c2f30 con 0x7fa2bc14fda0 2023-03-03T15:09:08.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c1f26700 1 --2- 172.21.15.7:0/1988415279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc14f9d0 0x7fa2bc156430 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc071230 tx=0x7fa2b0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74d2a91391a2fe5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:09:08.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c0f24700 1 -- 172.21.15.7:0/1988415279 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0011830 con 0x7fa2bc14f9d0 2023-03-03T15:09:08.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c0f24700 1 -- 172.21.15.7:0/1988415279 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b0011990 con 0x7fa2bc14f9d0 2023-03-03T15:09:08.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.886+0000 7fa2c0f24700 1 -- 172.21.15.7:0/1988415279 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0011d20 con 0x7fa2bc14f9d0 2023-03-03T15:09:08.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.887+0000 7fa2c418a700 1 -- 172.21.15.7:0/1988415279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc14f9d0 msgr2=0x7fa2bc156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:08.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.887+0000 7fa2c418a700 1 --2- 172.21.15.7:0/1988415279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc14f9d0 0x7fa2bc156430 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc071230 tx=0x7fa2b0009b20 comp rx=0 tx=0).stop 2023-03-03T15:09:08.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.887+0000 7fa2c418a700 1 -- 172.21.15.7:0/1988415279 shutdown_connections 2023-03-03T15:09:08.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.887+0000 7fa2c418a700 1 --2- 172.21.15.7:0/1988415279 >> v2:172.21.15.143:3300/0 conn(0x7fa2bc0abe20 0x7fa2bc1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:08.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.887+0000 7fa2c418a700 1 --2- 172.21.15.7:0/1988415279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc14f9d0 0x7fa2bc156430 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:08.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.887+0000 7fa2c418a700 1 -- 172.21.15.7:0/1988415279 >> 172.21.15.7:0/1988415279 conn(0x7fa2bc0c8900 msgr2=0x7fa2bc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:09:08.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.887+0000 7fa2c418a700 1 -- 172.21.15.7:0/1988415279 shutdown_connections 2023-03-03T15:09:08.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.887+0000 7fa2c418a700 1 -- 172.21.15.7:0/1988415279 wait complete. 2023-03-03T15:09:08.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.888+0000 7fa2c418a700 1 Processor -- start 2023-03-03T15:09:08.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.888+0000 7fa2c418a700 1 -- start start 2023-03-03T15:09:08.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c418a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc0abe20 0x7fa2bc1ea130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:08.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c418a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2bc14f9d0 0x7fa2bc0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:08.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c418a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2bc14fda0 0x7fa2bc1ed200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:08.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c418a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc121df0 con 0x7fa2bc0abe20 2023-03-03T15:09:08.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c418a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1ecdb0 con 0x7fa2bc14f9d0 2023-03-03T15:09:08.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c418a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1ea670 con 0x7fa2bc14fda0 2023-03-03T15:09:08.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c1f26700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc0abe20 0x7fa2bc1ea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:08.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c1f26700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc0abe20 0x7fa2bc1ea130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51818/0 (socket says 172.21.15.7:51818) 2023-03-03T15:09:08.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c1f26700 1 -- 172.21.15.7:0/4225920067 learned_addr learned my addr 172.21.15.7:0/4225920067 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:09:08.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c1725700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2bc14f9d0 0x7fa2bc0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:08.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c2727700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2bc14fda0 0x7fa2bc1ed200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:08.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c1f26700 1 -- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2bc14fda0 msgr2=0x7fa2bc1ed200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:08.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.889+0000 7fa2c1f26700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2bc14fda0 0x7fa2bc1ed200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:08.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.890+0000 7fa2c1f26700 1 -- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2bc14f9d0 msgr2=0x7fa2bc0c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:08.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.890+0000 7fa2c1f26700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2bc14f9d0 0x7fa2bc0c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:08.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.890+0000 7fa2c1f26700 1 -- 172.21.15.7:0/4225920067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b00094a0 con 0x7fa2bc0abe20 2023-03-03T15:09:08.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.890+0000 7fa2c2727700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2bc14fda0 0x7fa2bc1ed200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:09:08.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.890+0000 7fa2c1f26700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc0abe20 0x7fa2bc1ea130 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00090b0 tx=0x7fa2b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:09:08.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.890+0000 7fa2ae7fc700 1 -- 172.21.15.7:0/4225920067 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0004480 con 0x7fa2bc0abe20 2023-03-03T15:09:08.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.890+0000 7fa2ae7fc700 1 -- 172.21.15.7:0/4225920067 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b001f070 con 0x7fa2bc0abe20 2023-03-03T15:09:08.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.890+0000 7fa2c418a700 1 -- 172.21.15.7:0/4225920067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc0ac350 con 0x7fa2bc0abe20 2023-03-03T15:09:08.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.890+0000 7fa2ae7fc700 1 -- 172.21.15.7:0/4225920067 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b00038e0 con 0x7fa2bc0abe20 2023-03-03T15:09:08.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.890+0000 7fa2c418a700 1 -- 172.21.15.7:0/4225920067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc0ab630 con 0x7fa2bc0abe20 2023-03-03T15:09:08.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.891+0000 7fa2c418a700 1 -- 172.21.15.7:0/4225920067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc071fa0 con 0x7fa2bc0abe20 2023-03-03T15:09:08.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.891+0000 7fa2ae7fc700 1 -- 172.21.15.7:0/4225920067 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa2b002a020 con 0x7fa2bc0abe20 2023-03-03T15:09:08.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.892+0000 7fa2ae7fc700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2a00682f0 0x7fa2a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:08.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.892+0000 7fa2ae7fc700 1 -- 172.21.15.7:0/4225920067 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa2b00876e0 con 0x7fa2bc0abe20 2023-03-03T15:09:08.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.892+0000 7fa2c1725700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2a00682f0 0x7fa2a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:08.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.892+0000 7fa2c1725700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2a00682f0 0x7fa2a006a7a0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0c25f0 tx=0x7fa2b8007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:09:08.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:08.896+0000 7fa2ae7fc700 1 -- 172.21.15.7:0/4225920067 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa2b0053990 con 0x7fa2bc0abe20 2023-03-03T15:09:09.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.136+0000 7fa2c418a700 1 -- 172.21.15.7:0/4225920067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2bc06fcf0 con 0x7fa2bc0abe20 2023-03-03T15:09:09.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.136+0000 7fa2ae7fc700 1 -- 172.21.15.7:0/4225920067 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa2b0056fb0 con 0x7fa2bc0abe20 2023-03-03T15:09:09.138 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:09:09.138 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:09:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.138+0000 7fa2c418a700 1 -- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2a00682f0 msgr2=0x7fa2a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.138+0000 7fa2c418a700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2a00682f0 0x7fa2a006a7a0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0c25f0 tx=0x7fa2b8007de0 comp rx=0 tx=0).stop 2023-03-03T15:09:09.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.138+0000 7fa2c418a700 1 -- 172.21.15.7:0/4225920067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc0abe20 msgr2=0x7fa2bc1ea130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.138+0000 7fa2c418a700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc0abe20 0x7fa2bc1ea130 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00090b0 tx=0x7fa2b0004750 comp rx=0 tx=0).stop 2023-03-03T15:09:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.138+0000 7fa2c418a700 1 -- 172.21.15.7:0/4225920067 shutdown_connections 2023-03-03T15:09:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.138+0000 7fa2c418a700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2bc14fda0 0x7fa2bc1ed200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.138+0000 7fa2c418a700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa2a00682f0 0x7fa2a006a7a0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:09.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.138+0000 7fa2c418a700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2bc0abe20 0x7fa2bc1ea130 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:09.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.138+0000 7fa2c418a700 1 --2- 172.21.15.7:0/4225920067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2bc14f9d0 0x7fa2bc0c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:09.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.138+0000 7fa2c418a700 1 -- 172.21.15.7:0/4225920067 >> 172.21.15.7:0/4225920067 conn(0x7fa2bc0c8900 msgr2=0x7fa2bc1518c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:09:09.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.139+0000 7fa2c418a700 1 -- 172.21.15.7:0/4225920067 shutdown_connections 2023-03-03T15:09:09.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:09.139+0000 7fa2c418a700 1 -- 172.21.15.7:0/4225920067 wait complete. 2023-03-03T15:09:09.142 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:09:09.152 INFO:tasks.mds_thrash.fs.[b]:waiting for 104 secs before thrashing 2023-03-03T15:09:30.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:30.424 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:09:54.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:09:54.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.285+0000 7f6c41660700 1 Processor -- start 2023-03-03T15:09:54.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.285+0000 7f6c41660700 1 -- start start 2023-03-03T15:09:54.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.286+0000 7f6c41660700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6c3c1519e0 0x7f6c3c0bf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:54.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.286+0000 7f6c41660700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0c5a70 con 0x7f6c3c0c5e80 2023-03-03T15:09:54.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.286+0000 7f6c41660700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c151db0 con 0x7f6c3c0b0450 2023-03-03T15:09:54.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.286+0000 7f6c41660700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c154d60 con 0x7f6c3c1519e0 2023-03-03T15:09:54.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.286+0000 7f6c3b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6c3c0c5e80 0x7f6c3c0ca610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45834/0 (socket says 172.21.15.7:45834) 2023-03-03T15:09:54.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.286+0000 7f6c3b7fe700 1 -- 172.21.15.7:0/3167919530 learned_addr learned my addr 172.21.15.7:0/3167919530 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:09:54.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.286+0000 7f6c3affd700 1 --2- 172.21.15.7:0/3167919530 >> v2:172.21.15.143:3300/0 conn(0x7f6c3c1519e0 0x7f6c3c0bf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:54.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.286+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774705735 0 0) 0x7f6c3c0c5a70 con 0x7f6c3c0c5e80 2023-03-03T15:09:54.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.286+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c240035b0 con 0x7f6c3c0c5e80 2023-03-03T15:09:54.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2380157729 0 0) 0x7f6c240035b0 con 0x7f6c3c0c5e80 2023-03-03T15:09:54.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 >> v2:172.21.15.143:3300/0 conn(0x7f6c3c1519e0 msgr2=0x7f6c3c0bf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:09:54.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 --2- 172.21.15.7:0/3167919530 >> v2:172.21.15.143:3300/0 conn(0x7f6c3c1519e0 0x7f6c3c0bf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 >> v1:172.21.15.143:6789/0 conn(0x7f6c3c0b0450 legacy=0x7f6c3c0c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:54.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c0bfb90 con 0x7f6c3c0c5e80 2023-03-03T15:09:54.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f6c340019f0 con 0x7f6c3c0c5e80 2023-03-03T15:09:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 >> v1:172.21.15.7:6789/0 conn(0x7f6c3c0c5e80 legacy=0x7f6c3c0ca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c24003a70 0x7f6c24003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c24004850 0x7f6c24004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c24005200 0x7f6c2400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2400c0e0 con 0x7f6c24004850 2023-03-03T15:09:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2400c250 con 0x7f6c24003a70 2023-03-03T15:09:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2400c410 con 0x7f6c24005200 2023-03-03T15:09:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c3a7fc700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c24004850 0x7f6c24004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c3affd700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c24003a70 0x7f6c24003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c3b7fe700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c24005200 0x7f6c2400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c3a7fc700 1 -- 172.21.15.7:0/3167919530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c24005200 msgr2=0x7f6c2400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:09:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c3a7fc700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c24005200 0x7f6c2400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c3a7fc700 1 -- 172.21.15.7:0/3167919530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c24003a70 msgr2=0x7f6c24003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:09:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c3a7fc700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c24003a70 0x7f6c24003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c3a7fc700 1 -- 172.21.15.7:0/3167919530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c240035b0 con 0x7f6c24004850 2023-03-03T15:09:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c3b7fe700 1 -- 172.21.15.7:0/3167919530 reap_dead start 2023-03-03T15:09:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c3a7fc700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c24004850 0x7f6c24004cc0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f6c30007fd0 tx=0x7f6c24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4bcc66eb842847 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:09:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c300052d0 con 0x7f6c24004850 2023-03-03T15:09:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c30005430 con 0x7f6c24004850 2023-03-03T15:09:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.287+0000 7f6c39ffb700 1 -- 172.21.15.7:0/3167919530 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c300038a0 con 0x7f6c24004850 2023-03-03T15:09:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.288+0000 7f6c41660700 1 -- 172.21.15.7:0/3167919530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c24004850 msgr2=0x7f6c24004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.288+0000 7f6c41660700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c24004850 0x7f6c24004cc0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f6c30007fd0 tx=0x7f6c24005880 comp rx=0 tx=0).stop 2023-03-03T15:09:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.288+0000 7f6c41660700 1 -- 172.21.15.7:0/3167919530 shutdown_connections 2023-03-03T15:09:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.288+0000 7f6c41660700 1 --2- 172.21.15.7:0/3167919530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c24004850 0x7f6c24004cc0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.288+0000 7f6c41660700 1 -- 172.21.15.7:0/3167919530 >> 172.21.15.7:0/3167919530 conn(0x7f6c3c066a20 msgr2=0x7f6c3c0714c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:09:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.288+0000 7f6c41660700 1 -- 172.21.15.7:0/3167919530 shutdown_connections 2023-03-03T15:09:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.288+0000 7f6c41660700 1 -- 172.21.15.7:0/3167919530 wait complete. 2023-03-03T15:09:54.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.289+0000 7f6c41660700 1 Processor -- start 2023-03-03T15:09:54.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c41660700 1 -- start start 2023-03-03T15:09:54.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c41660700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c24004850 0x7f6c3c158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:54.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c41660700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3c14b400 0x7f6c3c14e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:54.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c41660700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c14aad0 0x7f6c3c152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:54.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c41660700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0afc00 con 0x7f6c3c14aad0 2023-03-03T15:09:54.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c41660700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c1ea070 con 0x7f6c3c14b400 2023-03-03T15:09:54.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c41660700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c1ea350 con 0x7f6c24004850 2023-03-03T15:09:54.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c3b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c14aad0 0x7f6c3c152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:54.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c3b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c14aad0 0x7f6c3c152c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51866/0 (socket says 172.21.15.7:51866) 2023-03-03T15:09:54.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c3b7fe700 1 -- 172.21.15.7:0/1808030241 learned_addr learned my addr 172.21.15.7:0/1808030241 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:09:54.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c3affd700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c24004850 0x7f6c3c158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:54.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.290+0000 7f6c3a7fc700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3c14b400 0x7f6c3c14e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:54.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c3b7fe700 1 -- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c24004850 msgr2=0x7f6c3c158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:54.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c3b7fe700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c24004850 0x7f6c3c158360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c3b7fe700 1 -- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3c14b400 msgr2=0x7f6c3c14e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:54.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c3b7fe700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3c14b400 0x7f6c3c14e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c3b7fe700 1 -- 172.21.15.7:0/1808030241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c155230 con 0x7f6c3c14aad0 2023-03-03T15:09:54.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c3affd700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c24004850 0x7f6c3c158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:09:54.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c3a7fc700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3c14b400 0x7f6c3c14e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:09:54.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c3b7fe700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c14aad0 0x7f6c3c152c20 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c072030 tx=0x7f6c3400a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:09:54.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c237fe700 1 -- 172.21.15.7:0/1808030241 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3400aca0 con 0x7f6c3c14aad0 2023-03-03T15:09:54.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c237fe700 1 -- 172.21.15.7:0/1808030241 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3400ae40 con 0x7f6c3c14aad0 2023-03-03T15:09:54.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c41660700 1 -- 172.21.15.7:0/1808030241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c154f50 con 0x7f6c3c14aad0 2023-03-03T15:09:54.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c237fe700 1 -- 172.21.15.7:0/1808030241 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c34013630 con 0x7f6c3c14aad0 2023-03-03T15:09:54.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.291+0000 7f6c41660700 1 -- 172.21.15.7:0/1808030241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c0c6f00 con 0x7f6c3c14aad0 2023-03-03T15:09:54.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.292+0000 7f6c41660700 1 -- 172.21.15.7:0/1808030241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c074bb0 con 0x7f6c3c14aad0 2023-03-03T15:09:54.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.292+0000 7f6c237fe700 1 -- 172.21.15.7:0/1808030241 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c34013790 con 0x7f6c3c14aad0 2023-03-03T15:09:54.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.293+0000 7f6c237fe700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c280682f0 0x7f6c2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:09:54.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.293+0000 7f6c237fe700 1 -- 172.21.15.7:0/1808030241 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6c34086a40 con 0x7f6c3c14aad0 2023-03-03T15:09:54.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.293+0000 7f6c3affd700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c280682f0 0x7f6c2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:09:54.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.294+0000 7f6c3affd700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c280682f0 0x7f6c2806a7a0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009c60 tx=0x7f6c2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:09:54.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.297+0000 7f6c237fe700 1 -- 172.21.15.7:0/1808030241 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c34052d80 con 0x7f6c3c14aad0 2023-03-03T15:09:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.533+0000 7f6c41660700 1 -- 172.21.15.7:0/1808030241 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c3c0c5c60 con 0x7f6c3c14aad0 2023-03-03T15:09:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.533+0000 7f6c237fe700 1 -- 172.21.15.7:0/1808030241 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6c340563a0 con 0x7f6c3c14aad0 2023-03-03T15:09:54.535 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:09:54.535 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:09:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 -- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c280682f0 msgr2=0x7f6c2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c280682f0 0x7f6c2806a7a0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009c60 tx=0x7f6c2c008040 comp rx=0 tx=0).stop 2023-03-03T15:09:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 -- 172.21.15.7:0/1808030241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c14aad0 msgr2=0x7f6c3c152c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:09:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c14aad0 0x7f6c3c152c20 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c072030 tx=0x7f6c3400a5b0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 -- 172.21.15.7:0/1808030241 shutdown_connections 2023-03-03T15:09:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c280682f0 0x7f6c2806a7a0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c3c14aad0 0x7f6c3c152c20 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c24004850 0x7f6c3c158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 --2- 172.21.15.7:0/1808030241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3c14b400 0x7f6c3c14e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:09:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 -- 172.21.15.7:0/1808030241 >> 172.21.15.7:0/1808030241 conn(0x7f6c3c066a20 msgr2=0x7f6c3c14c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:09:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 -- 172.21.15.7:0/1808030241 shutdown_connections 2023-03-03T15:09:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:09:54.535+0000 7f6c41660700 1 -- 172.21.15.7:0/1808030241 wait complete. 2023-03-03T15:09:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:09:54.549 INFO:tasks.mds_thrash.fs.[a]:waiting for 38 secs before thrashing 2023-03-03T15:10:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T15:09:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T15:10:00.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:00.486 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:30.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:30.558 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:10:32.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:10:32.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.689+0000 7f94bc773700 1 Processor -- start 2023-03-03T15:10:32.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.689+0000 7f94bc773700 1 -- start start 2023-03-03T15:10:32.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.690+0000 7f94bc773700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f94b40ac220 0x7f94b40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:32.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.690+0000 7f94bc773700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94b4073c30 0x7f94b4152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.690+0000 7f94bc773700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b40ac900 con 0x7f94b40ace30 2023-03-03T15:10:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.690+0000 7f94bc773700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b4071e30 con 0x7f94b4073c30 2023-03-03T15:10:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.690+0000 7f94bc773700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b40acb50 con 0x7f94b40ac220 2023-03-03T15:10:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b9d0e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94b4073c30 0x7f94b4152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b9d0e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94b4073c30 0x7f94b4152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51892/0 (socket says 172.21.15.7:51892) 2023-03-03T15:10:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b9d0e700 1 -- 172.21.15.7:0/3299691698 learned_addr learned my addr 172.21.15.7:0/3299691698 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:10:32.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94ba50f700 1 --2- 172.21.15.7:0/3299691698 >> v2:172.21.15.143:3300/0 conn(0x7f94b40ac220 0x7f94b40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b9d0e700 1 -- 172.21.15.7:0/3299691698 >> v2:172.21.15.143:3300/0 conn(0x7f94b40ac220 msgr2=0x7f94b40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:10:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b9d0e700 1 --2- 172.21.15.7:0/3299691698 >> v2:172.21.15.143:3300/0 conn(0x7f94b40ac220 0x7f94b40736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b9d0e700 1 -- 172.21.15.7:0/3299691698 >> v1:172.21.15.143:6789/0 conn(0x7f94b40ace30 legacy=0x7f94b406a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b9d0e700 1 -- 172.21.15.7:0/3299691698 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b41559c0 con 0x7f94b4073c30 2023-03-03T15:10:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b9d0e700 1 --2- 172.21.15.7:0/3299691698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 0x7f94b4152450 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f94b406fbf0 tx=0x7f94a8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c90adce2ac91db server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:10:32.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b950d700 1 -- 172.21.15.7:0/3299691698 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a800b870 con 0x7f94b4073c30 2023-03-03T15:10:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b950d700 1 -- 172.21.15.7:0/3299691698 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a800b9d0 con 0x7f94b4073c30 2023-03-03T15:10:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.691+0000 7f94b950d700 1 -- 172.21.15.7:0/3299691698 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94a800bd60 con 0x7f94b4073c30 2023-03-03T15:10:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.692+0000 7f94bc773700 1 -- 172.21.15.7:0/3299691698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 msgr2=0x7f94b4152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.692+0000 7f94bc773700 1 --2- 172.21.15.7:0/3299691698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 0x7f94b4152450 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f94b406fbf0 tx=0x7f94a8009bd0 comp rx=0 tx=0).stop 2023-03-03T15:10:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.692+0000 7f94bc773700 1 -- 172.21.15.7:0/3299691698 shutdown_connections 2023-03-03T15:10:32.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.692+0000 7f94bc773700 1 --2- 172.21.15.7:0/3299691698 >> v2:172.21.15.143:3300/0 conn(0x7f94b40ac220 0x7f94b40736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.692+0000 7f94bc773700 1 --2- 172.21.15.7:0/3299691698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 0x7f94b4152450 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.692+0000 7f94bc773700 1 -- 172.21.15.7:0/3299691698 >> 172.21.15.7:0/3299691698 conn(0x7f94b40a94d0 msgr2=0x7f94b40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:10:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.692+0000 7f94bc773700 1 -- 172.21.15.7:0/3299691698 shutdown_connections 2023-03-03T15:10:32.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.692+0000 7f94bc773700 1 -- 172.21.15.7:0/3299691698 wait complete. 2023-03-03T15:10:32.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.693+0000 7f94bc773700 1 Processor -- start 2023-03-03T15:10:32.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.694+0000 7f94bc773700 1 -- start start 2023-03-03T15:10:32.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.694+0000 7f94bc773700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 0x7f94b40be5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:32.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.694+0000 7f94bc773700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94b40ac220 0x7f94b414e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:32.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.694+0000 7f94bc773700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94b40ace30 0x7f94b41e9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:32.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.694+0000 7f94bc773700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b40beb20 con 0x7f94b4073c30 2023-03-03T15:10:32.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.694+0000 7f94bc773700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b4149020 con 0x7f94b40ace30 2023-03-03T15:10:32.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.694+0000 7f94bc773700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b40c15e0 con 0x7f94b40ac220 2023-03-03T15:10:32.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94ba50f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 0x7f94b40be5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:32.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94ba50f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 0x7f94b40be5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51896/0 (socket says 172.21.15.7:51896) 2023-03-03T15:10:32.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94ba50f700 1 -- 172.21.15.7:0/595897602 learned_addr learned my addr 172.21.15.7:0/595897602 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:10:32.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94bad10700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94b40ace30 0x7f94b41e9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:32.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94ba50f700 1 -- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94b40ac220 msgr2=0x7f94b414e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94b9d0e700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94b40ac220 0x7f94b414e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94ba50f700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94b40ac220 0x7f94b414e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94ba50f700 1 -- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94b40ace30 msgr2=0x7f94b41e9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94ba50f700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94b40ace30 0x7f94b41e9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94ba50f700 1 -- 172.21.15.7:0/595897602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a8009550 con 0x7f94b4073c30 2023-03-03T15:10:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94ba50f700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 0x7f94b40be5e0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f94b000e840 tx=0x7f94b000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:10:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94a6ffd700 1 -- 172.21.15.7:0/595897602 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0013610 con 0x7f94b4073c30 2023-03-03T15:10:32.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94a6ffd700 1 -- 172.21.15.7:0/595897602 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0013770 con 0x7f94b4073c30 2023-03-03T15:10:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94bc773700 1 -- 172.21.15.7:0/595897602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b4155f80 con 0x7f94b4073c30 2023-03-03T15:10:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94a6ffd700 1 -- 172.21.15.7:0/595897602 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b0013b20 con 0x7f94b4073c30 2023-03-03T15:10:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.695+0000 7f94bc773700 1 -- 172.21.15.7:0/595897602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b4155ca0 con 0x7f94b4073c30 2023-03-03T15:10:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.696+0000 7f94bc773700 1 -- 172.21.15.7:0/595897602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b4156260 con 0x7f94b4073c30 2023-03-03T15:10:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.697+0000 7f94a6ffd700 1 -- 172.21.15.7:0/595897602 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f94b0023070 con 0x7f94b4073c30 2023-03-03T15:10:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.697+0000 7f94a6ffd700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94980682f0 0x7f949806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:32.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.697+0000 7f94a6ffd700 1 -- 172.21.15.7:0/595897602 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f94b0085e50 con 0x7f94b4073c30 2023-03-03T15:10:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.697+0000 7f94b9d0e700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94980682f0 0x7f949806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.698+0000 7f94b9d0e700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94980682f0 0x7f949806a7a0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f94b41553e0 tx=0x7f94a8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:10:32.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.702+0000 7f94a6ffd700 1 -- 172.21.15.7:0/595897602 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f94b00521f0 con 0x7f94b4073c30 2023-03-03T15:10:32.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.941+0000 7f94bc773700 1 -- 172.21.15.7:0/595897602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94b40c7050 con 0x7f94b4073c30 2023-03-03T15:10:32.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.941+0000 7f94a6ffd700 1 -- 172.21.15.7:0/595897602 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f94b0055810 con 0x7f94b4073c30 2023-03-03T15:10:32.943 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:10:32.943 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:10:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 -- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94980682f0 msgr2=0x7f949806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94980682f0 0x7f949806a7a0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f94b41553e0 tx=0x7f94a8008030 comp rx=0 tx=0).stop 2023-03-03T15:10:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 -- 172.21.15.7:0/595897602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 msgr2=0x7f94b40be5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 0x7f94b40be5e0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f94b000e840 tx=0x7f94b000ec00 comp rx=0 tx=0).stop 2023-03-03T15:10:32.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 -- 172.21.15.7:0/595897602 shutdown_connections 2023-03-03T15:10:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94b40ace30 0x7f94b41e9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94980682f0 0x7f949806a7a0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94b4073c30 0x7f94b40be5e0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 --2- 172.21.15.7:0/595897602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94b40ac220 0x7f94b414e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 -- 172.21.15.7:0/595897602 >> 172.21.15.7:0/595897602 conn(0x7f94b40a94d0 msgr2=0x7f94b4070040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:10:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 -- 172.21.15.7:0/595897602 shutdown_connections 2023-03-03T15:10:32.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:32.943+0000 7f94bc773700 1 -- 172.21.15.7:0/595897602 wait complete. 2023-03-03T15:10:32.947 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:10:32.956 INFO:tasks.mds_thrash.fs.[a]:waiting for 74 secs before thrashing 2023-03-03T15:10:53.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:10:53.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.290+0000 7fb2145e8700 1 Processor -- start 2023-03-03T15:10:53.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.290+0000 7fb2145e8700 1 -- start start 2023-03-03T15:10:53.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.291+0000 7fb2145e8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb20c14f9d0 0x7fb20c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:53.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.291+0000 7fb2145e8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb20c14fda0 0x7fb20c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:53.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.291+0000 7fb2145e8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20c0c2f30 con 0x7fb20c0abe20 2023-03-03T15:10:53.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.291+0000 7fb2145e8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20c0c31d0 con 0x7fb20c14fda0 2023-03-03T15:10:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.291+0000 7fb2145e8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20c1524a0 con 0x7fb20c14f9d0 2023-03-03T15:10:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb211b83700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb20c14fda0 0x7fb20c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb212b85700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb20c0abe20 0x7fb20c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45898/0 (socket says 172.21.15.7:45898) 2023-03-03T15:10:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb212b85700 1 -- 172.21.15.7:0/3682086997 learned_addr learned my addr 172.21.15.7:0/3682086997 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:10:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb212384700 1 --2- 172.21.15.7:0/3682086997 >> v2:172.21.15.143:3300/0 conn(0x7fb20c14f9d0 0x7fb20c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb211b83700 1 -- 172.21.15.7:0/3682086997 >> v2:172.21.15.143:3300/0 conn(0x7fb20c14f9d0 msgr2=0x7fb20c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:10:53.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb211b83700 1 --2- 172.21.15.7:0/3682086997 >> v2:172.21.15.143:3300/0 conn(0x7fb20c14f9d0 0x7fb20c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb211b83700 1 -- 172.21.15.7:0/3682086997 >> v1:172.21.15.7:6789/0 conn(0x7fb20c0abe20 legacy=0x7fb20c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb211b83700 1 -- 172.21.15.7:0/3682086997 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c0c34b0 con 0x7fb20c14fda0 2023-03-03T15:10:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb211382700 1 -- 172.21.15.7:0/3682086997 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091511580 0 0) 0x7fb20c0c2f30 con 0x7fb20c0abe20 2023-03-03T15:10:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb211b83700 1 --2- 172.21.15.7:0/3682086997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 0x7fb20c071890 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0710a0 tx=0x7fb208009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e9fd4e4045db63 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:10:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb211382700 1 -- 172.21.15.7:0/3682086997 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208011830 con 0x7fb20c14fda0 2023-03-03T15:10:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb211382700 1 -- 172.21.15.7:0/3682086997 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb208011990 con 0x7fb20c14fda0 2023-03-03T15:10:53.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.292+0000 7fb211382700 1 -- 172.21.15.7:0/3682086997 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb208011d20 con 0x7fb20c14fda0 2023-03-03T15:10:53.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.293+0000 7fb2145e8700 1 -- 172.21.15.7:0/3682086997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 msgr2=0x7fb20c071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:53.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.293+0000 7fb2145e8700 1 --2- 172.21.15.7:0/3682086997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 0x7fb20c071890 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0710a0 tx=0x7fb208009b20 comp rx=0 tx=0).stop 2023-03-03T15:10:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.293+0000 7fb2145e8700 1 -- 172.21.15.7:0/3682086997 shutdown_connections 2023-03-03T15:10:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.293+0000 7fb2145e8700 1 --2- 172.21.15.7:0/3682086997 >> v2:172.21.15.143:3300/0 conn(0x7fb20c14f9d0 0x7fb20c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.293+0000 7fb2145e8700 1 --2- 172.21.15.7:0/3682086997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 0x7fb20c071890 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.293+0000 7fb2145e8700 1 -- 172.21.15.7:0/3682086997 >> 172.21.15.7:0/3682086997 conn(0x7fb20c0c8900 msgr2=0x7fb20c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:10:53.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.293+0000 7fb2145e8700 1 -- 172.21.15.7:0/3682086997 shutdown_connections 2023-03-03T15:10:53.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.293+0000 7fb2145e8700 1 -- 172.21.15.7:0/3682086997 wait complete. 2023-03-03T15:10:53.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.294+0000 7fb2145e8700 1 Processor -- start 2023-03-03T15:10:53.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.294+0000 7fb2145e8700 1 -- start start 2023-03-03T15:10:53.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb2145e8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c0abe20 0x7fb20c1e7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:53.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb2145e8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb20c14f9d0 0x7fb20c0c1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:53.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb2145e8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 0x7fb20c0bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:53.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb2145e8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0c1060 con 0x7fb20c14fda0 2023-03-03T15:10:53.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb2145e8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c1e81c0 con 0x7fb20c14f9d0 2023-03-03T15:10:53.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb2145e8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0bd7f0 con 0x7fb20c0abe20 2023-03-03T15:10:53.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb212b85700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 0x7fb20c0bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:53.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb212b85700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 0x7fb20c0bfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51924/0 (socket says 172.21.15.7:51924) 2023-03-03T15:10:53.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb212b85700 1 -- 172.21.15.7:0/2972648899 learned_addr learned my addr 172.21.15.7:0/2972648899 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:10:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb211b83700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb20c14f9d0 0x7fb20c0c1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb212384700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c0abe20 0x7fb20c1e7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb212b85700 1 -- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c0abe20 msgr2=0x7fb20c1e7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb212b85700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c0abe20 0x7fb20c1e7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb212b85700 1 -- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb20c14f9d0 msgr2=0x7fb20c0c1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb212b85700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb20c14f9d0 0x7fb20c0c1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.295+0000 7fb212b85700 1 -- 172.21.15.7:0/2972648899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2080094a0 con 0x7fb20c14fda0 2023-03-03T15:10:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.296+0000 7fb212b85700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 0x7fb20c0bfd20 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fb20400cc40 tx=0x7fb20400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:10:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.296+0000 7fb1feffd700 1 -- 172.21.15.7:0/2972648899 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20400aa20 con 0x7fb20c14fda0 2023-03-03T15:10:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.296+0000 7fb1feffd700 1 -- 172.21.15.7:0/2972648899 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20400aba0 con 0x7fb20c14fda0 2023-03-03T15:10:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.296+0000 7fb1feffd700 1 -- 172.21.15.7:0/2972648899 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2040070d0 con 0x7fb20c14fda0 2023-03-03T15:10:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.296+0000 7fb2145e8700 1 -- 172.21.15.7:0/2972648899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c0ac350 con 0x7fb20c14fda0 2023-03-03T15:10:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.296+0000 7fb2145e8700 1 -- 172.21.15.7:0/2972648899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c0ab630 con 0x7fb20c14fda0 2023-03-03T15:10:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.297+0000 7fb2145e8700 1 -- 172.21.15.7:0/2972648899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c0c3e10 con 0x7fb20c14fda0 2023-03-03T15:10:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.297+0000 7fb1feffd700 1 -- 172.21.15.7:0/2972648899 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb204010070 con 0x7fb20c14fda0 2023-03-03T15:10:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.298+0000 7fb1feffd700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1f00682f0 0x7fb1f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:10:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.298+0000 7fb1feffd700 1 -- 172.21.15.7:0/2972648899 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb2040859d0 con 0x7fb20c14fda0 2023-03-03T15:10:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.298+0000 7fb212384700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1f00682f0 0x7fb1f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:10:53.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.299+0000 7fb212384700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1f00682f0 0x7fb1f006a7a0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fb200006b20 tx=0x7fb200008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:10:53.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.302+0000 7fb1feffd700 1 -- 172.21.15.7:0/2972648899 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb204051d70 con 0x7fb20c14fda0 2023-03-03T15:10:53.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.539+0000 7fb2145e8700 1 -- 172.21.15.7:0/2972648899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb20c074bb0 con 0x7fb20c14fda0 2023-03-03T15:10:53.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.540+0000 7fb1feffd700 1 -- 172.21.15.7:0/2972648899 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb204055390 con 0x7fb20c14fda0 2023-03-03T15:10:53.542 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:10:53.542 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:10:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.541+0000 7fb2145e8700 1 -- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1f00682f0 msgr2=0x7fb1f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.541+0000 7fb2145e8700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1f00682f0 0x7fb1f006a7a0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fb200006b20 tx=0x7fb200008040 comp rx=0 tx=0).stop 2023-03-03T15:10:53.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.541+0000 7fb2145e8700 1 -- 172.21.15.7:0/2972648899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 msgr2=0x7fb20c0bfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:10:53.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.541+0000 7fb2145e8700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 0x7fb20c0bfd20 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fb20400cc40 tx=0x7fb20400cf50 comp rx=0 tx=0).stop 2023-03-03T15:10:53.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.542+0000 7fb2145e8700 1 -- 172.21.15.7:0/2972648899 shutdown_connections 2023-03-03T15:10:53.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.542+0000 7fb2145e8700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1f00682f0 0x7fb1f006a7a0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:53.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.542+0000 7fb2145e8700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb20c14fda0 0x7fb20c0bfd20 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:53.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.542+0000 7fb2145e8700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb20c0abe20 0x7fb20c1e7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:53.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.542+0000 7fb2145e8700 1 --2- 172.21.15.7:0/2972648899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb20c14f9d0 0x7fb20c0c1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:10:53.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.542+0000 7fb2145e8700 1 -- 172.21.15.7:0/2972648899 >> 172.21.15.7:0/2972648899 conn(0x7fb20c0c8900 msgr2=0x7fb20c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:10:53.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.542+0000 7fb2145e8700 1 -- 172.21.15.7:0/2972648899 shutdown_connections 2023-03-03T15:10:53.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:10:53.542+0000 7fb2145e8700 1 -- 172.21.15.7:0/2972648899 wait complete. 2023-03-03T15:10:53.545 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:10:53.555 INFO:tasks.mds_thrash.fs.[b]:waiting for 65 secs before thrashing 2023-03-03T15:11:00.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:00.621 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:30.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:30.711 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:11:46.957 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:11:47.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.095+0000 7fa69c462700 1 Processor -- start 2023-03-03T15:11:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.095+0000 7fa69c462700 1 -- start start 2023-03-03T15:11:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa69c462700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa6940c4630 0x7fa69406e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa69c462700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694154420 0x7fa6941548a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa69c462700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69406e730 con 0x7fa694151b00 2023-03-03T15:11:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa69c462700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6941583e0 con 0x7fa6940c4630 2023-03-03T15:11:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa69c462700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa694157ab0 con 0x7fa694154420 2023-03-03T15:11:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa69a9ff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694154420 0x7fa6941548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa6999fd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa6940c4630 0x7fa69406e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa69a1fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fa694151b00 0x7fa6940a94c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49470/0 (socket says 172.21.15.7:49470) 2023-03-03T15:11:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa69a9ff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694154420 0x7fa6941548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:60044/0 (socket says 172.21.15.7:60044) 2023-03-03T15:11:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa6999fd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa6940c4630 0x7fa69406e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:47748/0 (socket says 172.21.15.7:47748) 2023-03-03T15:11:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa69a9ff700 1 -- 172.21.15.7:0/4149212642 learned_addr learned my addr 172.21.15.7:0/4149212642 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:11:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa6999fd700 1 -- 172.21.15.7:0/4149212642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694154420 msgr2=0x7fa6941548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:11:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa6999fd700 1 --2- 172.21.15.7:0/4149212642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694154420 0x7fa6941548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa6999fd700 1 -- 172.21.15.7:0/4149212642 >> v1:172.21.15.143:6789/0 conn(0x7fa694151b00 legacy=0x7fa6940a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.097+0000 7fa6999fd700 1 -- 172.21.15.7:0/4149212642 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa694157d90 con 0x7fa6940c4630 2023-03-03T15:11:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.096+0000 7fa6991fc700 1 -- 172.21.15.7:0/4149212642 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174894499 0 0) 0x7fa69406e730 con 0x7fa694151b00 2023-03-03T15:11:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.097+0000 7fa69a9ff700 1 --2- 172.21.15.7:0/4149212642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694154420 0x7fa6941548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:11:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.097+0000 7fa6999fd700 1 --2- 172.21.15.7:0/4149212642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6940c4630 0x7fa69406e150 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa684009840 tx=0x7fa684009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6fcacaf516ad42f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:11:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.097+0000 7fa6991fc700 1 -- 172.21.15.7:0/4149212642 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684011830 con 0x7fa6940c4630 2023-03-03T15:11:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.097+0000 7fa6991fc700 1 -- 172.21.15.7:0/4149212642 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa684011990 con 0x7fa6940c4630 2023-03-03T15:11:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.097+0000 7fa6991fc700 1 -- 172.21.15.7:0/4149212642 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa684011d20 con 0x7fa6940c4630 2023-03-03T15:11:47.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.097+0000 7fa69c462700 1 -- 172.21.15.7:0/4149212642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6940c4630 msgr2=0x7fa69406e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:47.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.098+0000 7fa69c462700 1 --2- 172.21.15.7:0/4149212642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6940c4630 0x7fa69406e150 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa684009840 tx=0x7fa684009b50 comp rx=0 tx=0).stop 2023-03-03T15:11:47.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.098+0000 7fa69c462700 1 -- 172.21.15.7:0/4149212642 shutdown_connections 2023-03-03T15:11:47.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.098+0000 7fa69c462700 1 --2- 172.21.15.7:0/4149212642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694154420 0x7fa6941548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:47.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.098+0000 7fa69c462700 1 --2- 172.21.15.7:0/4149212642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6940c4630 0x7fa69406e150 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:47.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.098+0000 7fa69c462700 1 -- 172.21.15.7:0/4149212642 >> 172.21.15.7:0/4149212642 conn(0x7fa6940c84d0 msgr2=0x7fa6940c88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:11:47.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.098+0000 7fa69c462700 1 -- 172.21.15.7:0/4149212642 shutdown_connections 2023-03-03T15:11:47.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.098+0000 7fa69c462700 1 -- 172.21.15.7:0/4149212642 wait complete. 2023-03-03T15:11:47.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.099+0000 7fa69c462700 1 Processor -- start 2023-03-03T15:11:47.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.099+0000 7fa69c462700 1 -- start start 2023-03-03T15:11:47.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69c462700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6940c4630 0x7fa694156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:47.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69c462700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694151b00 0x7fa69406f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:47.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69c462700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa694154420 0x7fa6941de820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:47.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69c462700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa694153af0 con 0x7fa6940c4630 2023-03-03T15:11:47.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69c462700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941e07c0 con 0x7fa694154420 2023-03-03T15:11:47.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69c462700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941e0570 con 0x7fa694151b00 2023-03-03T15:11:47.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69a1fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6940c4630 0x7fa694156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:47.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69a1fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6940c4630 0x7fa694156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51976/0 (socket says 172.21.15.7:51976) 2023-03-03T15:11:47.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69a1fe700 1 -- 172.21.15.7:0/315360538 learned_addr learned my addr 172.21.15.7:0/315360538 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:11:47.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa6999fd700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694151b00 0x7fa69406f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:47.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69a9ff700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa694154420 0x7fa6941de820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:47.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69a1fe700 1 -- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694151b00 msgr2=0x7fa69406f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:47.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69a1fe700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694151b00 0x7fa69406f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:47.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69a1fe700 1 -- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa694154420 msgr2=0x7fa6941de820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:47.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69a1fe700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa694154420 0x7fa6941de820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:47.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69a1fe700 1 -- 172.21.15.7:0/315360538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c009b30 con 0x7fa6940c4630 2023-03-03T15:11:47.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69a1fe700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6940c4630 0x7fa694156500 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fa69000c580 tx=0x7fa69000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:11:47.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa68affd700 1 -- 172.21.15.7:0/315360538 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa690004c80 con 0x7fa6940c4630 2023-03-03T15:11:47.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa68affd700 1 -- 172.21.15.7:0/315360538 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa690004e00 con 0x7fa6940c4630 2023-03-03T15:11:47.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.100+0000 7fa69c462700 1 -- 172.21.15.7:0/315360538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6840094a0 con 0x7fa6940c4630 2023-03-03T15:11:47.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.101+0000 7fa68affd700 1 -- 172.21.15.7:0/315360538 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa690007310 con 0x7fa6940c4630 2023-03-03T15:11:47.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.101+0000 7fa69c462700 1 -- 172.21.15.7:0/315360538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa694158870 con 0x7fa6940c4630 2023-03-03T15:11:47.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.102+0000 7fa69c462700 1 -- 172.21.15.7:0/315360538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa694158070 con 0x7fa6940c4630 2023-03-03T15:11:47.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.102+0000 7fa68affd700 1 -- 172.21.15.7:0/315360538 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa69001a020 con 0x7fa6940c4630 2023-03-03T15:11:47.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.102+0000 7fa68affd700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6780682f0 0x7fa67806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:47.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.102+0000 7fa68affd700 1 -- 172.21.15.7:0/315360538 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa690085ba0 con 0x7fa6940c4630 2023-03-03T15:11:47.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.103+0000 7fa6999fd700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6780682f0 0x7fa67806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:47.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.103+0000 7fa6999fd700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6780682f0 0x7fa67806a7a0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fa6840090b0 tx=0x7fa68401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:11:47.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.107+0000 7fa68affd700 1 -- 172.21.15.7:0/315360538 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa690051f40 con 0x7fa6940c4630 2023-03-03T15:11:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.345+0000 7fa69c462700 1 -- 172.21.15.7:0/315360538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6940aace0 con 0x7fa6940c4630 2023-03-03T15:11:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.346+0000 7fa68affd700 1 -- 172.21.15.7:0/315360538 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa690055560 con 0x7fa6940c4630 2023-03-03T15:11:47.348 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:11:47.348 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:11:47.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 -- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6780682f0 msgr2=0x7fa67806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:47.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6780682f0 0x7fa67806a7a0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fa6840090b0 tx=0x7fa68401c040 comp rx=0 tx=0).stop 2023-03-03T15:11:47.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 -- 172.21.15.7:0/315360538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6940c4630 msgr2=0x7fa694156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:47.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6940c4630 0x7fa694156500 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fa69000c580 tx=0x7fa69000c890 comp rx=0 tx=0).stop 2023-03-03T15:11:47.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 -- 172.21.15.7:0/315360538 shutdown_connections 2023-03-03T15:11:47.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa694154420 0x7fa6941de820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:47.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6780682f0 0x7fa67806a7a0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:47.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6940c4630 0x7fa694156500 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:47.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 --2- 172.21.15.7:0/315360538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa694151b00 0x7fa69406f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:47.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 -- 172.21.15.7:0/315360538 >> 172.21.15.7:0/315360538 conn(0x7fa6940c84d0 msgr2=0x7fa6940c4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:11:47.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 -- 172.21.15.7:0/315360538 shutdown_connections 2023-03-03T15:11:47.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:47.348+0000 7fa69c462700 1 -- 172.21.15.7:0/315360538 wait complete. 2023-03-03T15:11:47.352 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:11:47.361 INFO:tasks.mds_thrash.fs.[a]:waiting for 41 secs before thrashing 2023-03-03T15:11:58.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:11:58.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.694+0000 7fdada317700 1 Processor -- start 2023-03-03T15:11:58.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.694+0000 7fdada317700 1 -- start start 2023-03-03T15:11:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.695+0000 7fdada317700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad40ab5e0 0x7fdad40aba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.695+0000 7fdada317700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad414fd60 con 0x7fdad414f990 2023-03-03T15:11:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.695+0000 7fdada317700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad406c030 con 0x7fdad4152ed0 2023-03-03T15:11:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.695+0000 7fdada317700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad4153340 con 0x7fdad40ab5e0 2023-03-03T15:11:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.695+0000 7fdad3fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdad414f990 0x7fdad41562b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45972/0 (socket says 172.21.15.7:45972) 2023-03-03T15:11:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.695+0000 7fdad3fff700 1 -- 172.21.15.7:0/3563076057 learned_addr learned my addr 172.21.15.7:0/3563076057 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:11:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.695+0000 7fdad88b4700 1 --2- 172.21.15.7:0/3563076057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad40ab5e0 0x7fdad40aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad2ffd700 1 -- 172.21.15.7:0/3563076057 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989404843 0 0) 0x7fdad414fd60 con 0x7fdad414f990 2023-03-03T15:11:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad2ffd700 1 -- 172.21.15.7:0/3563076057 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac00035b0 con 0x7fdad414f990 2023-03-03T15:11:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad2ffd700 1 -- 172.21.15.7:0/3563076057 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956504263 0 0) 0x7fdad406c030 con 0x7fdad4152ed0 2023-03-03T15:11:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad2ffd700 1 -- 172.21.15.7:0/3563076057 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad414fd60 con 0x7fdad4152ed0 2023-03-03T15:11:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad88b4700 1 -- 172.21.15.7:0/3563076057 >> v1:172.21.15.143:6789/0 conn(0x7fdad4152ed0 legacy=0x7fdad406b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad88b4700 1 -- 172.21.15.7:0/3563076057 >> v1:172.21.15.7:6789/0 conn(0x7fdad414f990 legacy=0x7fdad41562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad88b4700 1 -- 172.21.15.7:0/3563076057 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad40c29c0 con 0x7fdad40ab5e0 2023-03-03T15:11:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad88b4700 1 --2- 172.21.15.7:0/3563076057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad40ab5e0 0x7fdad40aba60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009dc0 tx=0x7fdacc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cbb40cb604aa583 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:11:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad2ffd700 1 -- 172.21.15.7:0/3563076057 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc00e070 con 0x7fdad40ab5e0 2023-03-03T15:11:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad2ffd700 1 -- 172.21.15.7:0/3563076057 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdacc00bb20 con 0x7fdad40ab5e0 2023-03-03T15:11:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.696+0000 7fdad2ffd700 1 -- 172.21.15.7:0/3563076057 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc012430 con 0x7fdad40ab5e0 2023-03-03T15:11:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.697+0000 7fdada317700 1 -- 172.21.15.7:0/3563076057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad40ab5e0 msgr2=0x7fdad40aba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.697+0000 7fdada317700 1 --2- 172.21.15.7:0/3563076057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad40ab5e0 0x7fdad40aba60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009dc0 tx=0x7fdacc00b330 comp rx=0 tx=0).stop 2023-03-03T15:11:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.697+0000 7fdada317700 1 -- 172.21.15.7:0/3563076057 shutdown_connections 2023-03-03T15:11:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.697+0000 7fdada317700 1 --2- 172.21.15.7:0/3563076057 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad40ab5e0 0x7fdad40aba60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.697+0000 7fdada317700 1 -- 172.21.15.7:0/3563076057 >> 172.21.15.7:0/3563076057 conn(0x7fdad40c62e0 msgr2=0x7fdad40c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:11:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.697+0000 7fdada317700 1 -- 172.21.15.7:0/3563076057 shutdown_connections 2023-03-03T15:11:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.697+0000 7fdada317700 1 -- 172.21.15.7:0/3563076057 wait complete. 2023-03-03T15:11:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.698+0000 7fdada317700 1 Processor -- start 2023-03-03T15:11:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.698+0000 7fdada317700 1 -- start start 2023-03-03T15:11:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdada317700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdad40ab5e0 0x7fdad40c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdada317700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad414f990 0x7fdad40c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdada317700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdad4152ed0 0x7fdad41434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdada317700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40ab170 con 0x7fdad4152ed0 2023-03-03T15:11:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdada317700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40a9be0 con 0x7fdad40ab5e0 2023-03-03T15:11:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdada317700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40aa8d0 con 0x7fdad414f990 2023-03-03T15:11:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad88b4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdad4152ed0 0x7fdad41434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad88b4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdad4152ed0 0x7fdad41434c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52000/0 (socket says 172.21.15.7:52000) 2023-03-03T15:11:58.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad88b4700 1 -- 172.21.15.7:0/69733712 learned_addr learned my addr 172.21.15.7:0/69733712 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:11:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad3fff700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdad40ab5e0 0x7fdad40c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad37fe700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad414f990 0x7fdad40c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad88b4700 1 -- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad414f990 msgr2=0x7fdad40c4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad88b4700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad414f990 0x7fdad40c4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad88b4700 1 -- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdad40ab5e0 msgr2=0x7fdad40c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad88b4700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdad40ab5e0 0x7fdad40c3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:58.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad88b4700 1 -- 172.21.15.7:0/69733712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc009a70 con 0x7fdad4152ed0 2023-03-03T15:11:58.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad37fe700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad414f990 0x7fdad40c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:11:58.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.699+0000 7fdad3fff700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdad40ab5e0 0x7fdad40c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:11:58.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.700+0000 7fdad88b4700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdad4152ed0 0x7fdad41434c0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00b370 tx=0x7fdacc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:11:58.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.700+0000 7fdad0ff9700 1 -- 172.21.15.7:0/69733712 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc01b810 con 0x7fdad4152ed0 2023-03-03T15:11:58.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.700+0000 7fdad0ff9700 1 -- 172.21.15.7:0/69733712 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdacc020070 con 0x7fdad4152ed0 2023-03-03T15:11:58.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.700+0000 7fdada317700 1 -- 172.21.15.7:0/69733712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad40c26e0 con 0x7fdad4152ed0 2023-03-03T15:11:58.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.700+0000 7fdad0ff9700 1 -- 172.21.15.7:0/69733712 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdacc01bbf0 con 0x7fdad4152ed0 2023-03-03T15:11:58.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.700+0000 7fdada317700 1 -- 172.21.15.7:0/69733712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad40c2400 con 0x7fdad4152ed0 2023-03-03T15:11:58.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.701+0000 7fdada317700 1 -- 172.21.15.7:0/69733712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad41523f0 con 0x7fdad4152ed0 2023-03-03T15:11:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.702+0000 7fdad0ff9700 1 -- 172.21.15.7:0/69733712 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdacc01bd50 con 0x7fdad4152ed0 2023-03-03T15:11:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.702+0000 7fdad0ff9700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdab40682f0 0x7fdab406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:11:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.702+0000 7fdad0ff9700 1 -- 172.21.15.7:0/69733712 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdacc022070 con 0x7fdad4152ed0 2023-03-03T15:11:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.702+0000 7fdad3fff700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdab40682f0 0x7fdab406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:11:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.703+0000 7fdad3fff700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdab40682f0 0x7fdab406a7a0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fdac4000f60 tx=0x7fdac4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:11:58.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.706+0000 7fdad0ff9700 1 -- 172.21.15.7:0/69733712 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdacc054e30 con 0x7fdad4152ed0 2023-03-03T15:11:58.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.942+0000 7fdada317700 1 -- 172.21.15.7:0/69733712 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdad40707c0 con 0x7fdad4152ed0 2023-03-03T15:11:58.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.943+0000 7fdad0ff9700 1 -- 172.21.15.7:0/69733712 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdad40707c0 con 0x7fdad4152ed0 2023-03-03T15:11:58.945 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:11:58.945 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:11:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 -- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdab40682f0 msgr2=0x7fdab406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdab40682f0 0x7fdab406a7a0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fdac4000f60 tx=0x7fdac4009040 comp rx=0 tx=0).stop 2023-03-03T15:11:58.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 -- 172.21.15.7:0/69733712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdad4152ed0 msgr2=0x7fdad41434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:11:58.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdad4152ed0 0x7fdad41434c0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00b370 tx=0x7fdacc004ad0 comp rx=0 tx=0).stop 2023-03-03T15:11:58.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 -- 172.21.15.7:0/69733712 shutdown_connections 2023-03-03T15:11:58.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdab40682f0 0x7fdab406a7a0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:58.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdad4152ed0 0x7fdad41434c0 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:58.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdad40ab5e0 0x7fdad40c3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:58.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 --2- 172.21.15.7:0/69733712 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdad414f990 0x7fdad40c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:11:58.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 -- 172.21.15.7:0/69733712 >> 172.21.15.7:0/69733712 conn(0x7fdad40c62e0 msgr2=0x7fdad40c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:11:58.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.945+0000 7fdada317700 1 -- 172.21.15.7:0/69733712 shutdown_connections 2023-03-03T15:11:58.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:11:58.946+0000 7fdada317700 1 -- 172.21.15.7:0/69733712 wait complete. 2023-03-03T15:11:58.949 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:11:58.959 INFO:tasks.mds_thrash.fs.[b]:waiting for 95 secs before thrashing 2023-03-03T15:12:00.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:00.774 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:28.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:12:28.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.499+0000 7fa5fb658700 1 Processor -- start 2023-03-03T15:12:28.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.499+0000 7fa5fb658700 1 -- start start 2023-03-03T15:12:28.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5fb658700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa5f4151e80 0x7fa5f40c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:28.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5fb658700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa5f4155410 0x7fa5f40bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:28.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5fb658700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5f40c6fe0 0x7fa5f40bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:28.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5fb658700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f4152250 con 0x7fa5f4151e80 2023-03-03T15:12:28.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5fb658700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f40c0fb0 con 0x7fa5f4155410 2023-03-03T15:12:28.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5fb658700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4154740 con 0x7fa5f40c6fe0 2023-03-03T15:12:28.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5f93f4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa5f4151e80 0x7fa5f40c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:28.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5f93f4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa5f4151e80 0x7fa5f40c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52018/0 (socket says 172.21.15.7:52018) 2023-03-03T15:12:28.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5f93f4700 1 -- 172.21.15.7:0/597873319 learned_addr learned my addr 172.21.15.7:0/597873319 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:12:28.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5f9bf5700 1 --2- 172.21.15.7:0/597873319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5f40c6fe0 0x7fa5f40bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:28.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5f8bf3700 1 --2- 172.21.15.7:0/597873319 >> v2:172.21.15.143:3300/0 conn(0x7fa5f4155410 0x7fa5f40bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:28.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5f93f4700 1 -- 172.21.15.7:0/597873319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5f40c6fe0 msgr2=0x7fa5f40bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:12:28.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5f93f4700 1 --2- 172.21.15.7:0/597873319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5f40c6fe0 0x7fa5f40bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5f93f4700 1 -- 172.21.15.7:0/597873319 >> v2:172.21.15.143:3300/0 conn(0x7fa5f4155410 msgr2=0x7fa5f40bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:12:28.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5f93f4700 1 --2- 172.21.15.7:0/597873319 >> v2:172.21.15.143:3300/0 conn(0x7fa5f4155410 0x7fa5f40bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.500+0000 7fa5f93f4700 1 -- 172.21.15.7:0/597873319 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f40c08d0 con 0x7fa5f4151e80 2023-03-03T15:12:28.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5f93f4700 1 --2- 172.21.15.7:0/597873319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 0x7fa5f40c6aa0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fa5f406c380 tx=0x7fa5ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e05615a065f6d87f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:12:28.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5ebfff700 1 -- 172.21.15.7:0/597873319 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec00e070 con 0x7fa5f4151e80 2023-03-03T15:12:28.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5ebfff700 1 -- 172.21.15.7:0/597873319 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec00b9c0 con 0x7fa5f4151e80 2023-03-03T15:12:28.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5ebfff700 1 -- 172.21.15.7:0/597873319 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec00bd70 con 0x7fa5f4151e80 2023-03-03T15:12:28.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5fb658700 1 -- 172.21.15.7:0/597873319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 msgr2=0x7fa5f40c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:12:28.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5fb658700 1 --2- 172.21.15.7:0/597873319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 0x7fa5f40c6aa0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fa5f406c380 tx=0x7fa5ec00b330 comp rx=0 tx=0).stop 2023-03-03T15:12:28.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5fb658700 1 -- 172.21.15.7:0/597873319 shutdown_connections 2023-03-03T15:12:28.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5fb658700 1 --2- 172.21.15.7:0/597873319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5f40c6fe0 0x7fa5f40bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5fb658700 1 --2- 172.21.15.7:0/597873319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 0x7fa5f40c6aa0 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5fb658700 1 --2- 172.21.15.7:0/597873319 >> v2:172.21.15.143:3300/0 conn(0x7fa5f4155410 0x7fa5f40bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5fb658700 1 -- 172.21.15.7:0/597873319 >> 172.21.15.7:0/597873319 conn(0x7fa5f4071630 msgr2=0x7fa5f406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:12:28.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.501+0000 7fa5fb658700 1 -- 172.21.15.7:0/597873319 shutdown_connections 2023-03-03T15:12:28.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.502+0000 7fa5fb658700 1 -- 172.21.15.7:0/597873319 wait complete. 2023-03-03T15:12:28.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.503+0000 7fa5fb658700 1 Processor -- start 2023-03-03T15:12:28.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.503+0000 7fa5fb658700 1 -- start start 2023-03-03T15:12:28.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.503+0000 7fa5fb658700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5f40c6fe0 0x7fa5f41ea830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:28.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.503+0000 7fa5fb658700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 0x7fa5f41506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:28.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.503+0000 7fa5fb658700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5f4155410 0x7fa5f41eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:28.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.503+0000 7fa5fb658700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f41ea3a0 con 0x7fa5f4151e80 2023-03-03T15:12:28.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.503+0000 7fa5fb658700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4156430 con 0x7fa5f40c6fe0 2023-03-03T15:12:28.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.503+0000 7fa5fb658700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f41e9d70 con 0x7fa5f4155410 2023-03-03T15:12:28.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.503+0000 7fa5f8bf3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 0x7fa5f41506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:28.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5f8bf3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 0x7fa5f41506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52024/0 (socket says 172.21.15.7:52024) 2023-03-03T15:12:28.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5f8bf3700 1 -- 172.21.15.7:0/2842405579 learned_addr learned my addr 172.21.15.7:0/2842405579 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:12:28.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5f8bf3700 1 -- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5f4155410 msgr2=0x7fa5f41eb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:12:28.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5f8bf3700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5f4155410 0x7fa5f41eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5f8bf3700 1 -- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5f40c6fe0 msgr2=0x7fa5f41ea830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:12:28.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5f93f4700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5f40c6fe0 0x7fa5f41ea830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:28.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5f8bf3700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5f40c6fe0 0x7fa5f41ea830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5f8bf3700 1 -- 172.21.15.7:0/2842405579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec009b30 con 0x7fa5f4151e80 2023-03-03T15:12:28.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5f8bf3700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 0x7fa5f41506a0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400e810 tx=0x7fa5e400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:12:28.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5e9ffb700 1 -- 172.21.15.7:0/2842405579 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4013610 con 0x7fa5f4151e80 2023-03-03T15:12:28.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5e9ffb700 1 -- 172.21.15.7:0/2842405579 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e4013770 con 0x7fa5f4151e80 2023-03-03T15:12:28.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5fb658700 1 -- 172.21.15.7:0/2842405579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f4154c10 con 0x7fa5f4151e80 2023-03-03T15:12:28.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5e9ffb700 1 -- 172.21.15.7:0/2842405579 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4013b20 con 0x7fa5f4151e80 2023-03-03T15:12:28.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.504+0000 7fa5fb658700 1 -- 172.21.15.7:0/2842405579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f4154930 con 0x7fa5f4151e80 2023-03-03T15:12:28.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.505+0000 7fa5fb658700 1 -- 172.21.15.7:0/2842405579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f4154ef0 con 0x7fa5f4151e80 2023-03-03T15:12:28.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.506+0000 7fa5e9ffb700 1 -- 172.21.15.7:0/2842405579 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa5e4023070 con 0x7fa5f4151e80 2023-03-03T15:12:28.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.506+0000 7fa5e9ffb700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5d80682f0 0x7fa5d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:12:28.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.506+0000 7fa5e9ffb700 1 -- 172.21.15.7:0/2842405579 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa5e4085d00 con 0x7fa5f4151e80 2023-03-03T15:12:28.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.507+0000 7fa5f93f4700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5d80682f0 0x7fa5d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:12:28.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.507+0000 7fa5f93f4700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5d80682f0 0x7fa5d806a7a0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec000f80 tx=0x7fa5ec009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:12:28.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.511+0000 7fa5e9ffb700 1 -- 172.21.15.7:0/2842405579 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa5e4089030 con 0x7fa5f4151e80 2023-03-03T15:12:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.751+0000 7fa5fb658700 1 -- 172.21.15.7:0/2842405579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5f4074bb0 con 0x7fa5f4151e80 2023-03-03T15:12:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.751+0000 7fa5e9ffb700 1 -- 172.21.15.7:0/2842405579 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa5e40520a0 con 0x7fa5f4151e80 2023-03-03T15:12:28.754 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:12:28.755 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:12:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 -- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5d80682f0 msgr2=0x7fa5d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:12:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5d80682f0 0x7fa5d806a7a0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec000f80 tx=0x7fa5ec009000 comp rx=0 tx=0).stop 2023-03-03T15:12:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 -- 172.21.15.7:0/2842405579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 msgr2=0x7fa5f41506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:12:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 0x7fa5f41506a0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400e810 tx=0x7fa5e400ebd0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 -- 172.21.15.7:0/2842405579 shutdown_connections 2023-03-03T15:12:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa5f4155410 0x7fa5f41eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa5f40c6fe0 0x7fa5f41ea830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5d80682f0 0x7fa5d806a7a0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 --2- 172.21.15.7:0/2842405579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5f4151e80 0x7fa5f41506a0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:12:28.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 -- 172.21.15.7:0/2842405579 >> 172.21.15.7:0/2842405579 conn(0x7fa5f4071630 msgr2=0x7fa5f40c2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:12:28.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 -- 172.21.15.7:0/2842405579 shutdown_connections 2023-03-03T15:12:28.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:12:28.754+0000 7fa5fb658700 1 -- 172.21.15.7:0/2842405579 wait complete. 2023-03-03T15:12:28.758 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:12:28.768 INFO:tasks.mds_thrash.fs.[a]:waiting for 67 secs before thrashing 2023-03-03T15:12:30.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:12:30.863 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:00.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:00.924 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:30.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:30.998 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:13:33.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:13:34.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.097+0000 7fa10cfa1700 1 Processor -- start 2023-03-03T15:13:34.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.097+0000 7fa10cfa1700 1 -- start start 2023-03-03T15:13:34.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa10cfa1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10814e560 0x7fa108071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa10cfa1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa1080c4840 0x7fa108154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa10cfa1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa108072920 con 0x7fa1080c43d0 2023-03-03T15:13:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa10cfa1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa10806bef0 con 0x7fa1080c4840 2023-03-03T15:13:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa10cfa1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1081551e0 con 0x7fa10814e560 2023-03-03T15:13:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa105d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa1080c43d0 0x7fa108154590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46054/0 (socket says 172.21.15.7:46054) 2023-03-03T15:13:34.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa105d9b700 1 -- 172.21.15.7:0/3971709846 learned_addr learned my addr 172.21.15.7:0/3971709846 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:13:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa106d9d700 1 --2- 172.21.15.7:0/3971709846 >> v2:172.21.15.143:3300/0 conn(0x7fa1080c4840 0x7fa108154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa10659c700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10814e560 0x7fa108071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096152797 0 0) 0x7fa0fc001110 con 0x7fa1080c43d0 2023-03-03T15:13:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f00035b0 con 0x7fa1080c43d0 2023-03-03T15:13:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.098+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1796913811 0 0) 0x7fa0f00035b0 con 0x7fa1080c43d0 2023-03-03T15:13:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10814e560 msgr2=0x7fa108071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:13:34.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10814e560 0x7fa108071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 >> v2:172.21.15.143:3300/0 conn(0x7fa1080c4840 msgr2=0x7fa108154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:13:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 --2- 172.21.15.7:0/3971709846 >> v2:172.21.15.143:3300/0 conn(0x7fa1080c4840 0x7fa108154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10806b910 con 0x7fa1080c43d0 2023-03-03T15:13:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa0fc001800 con 0x7fa1080c43d0 2023-03-03T15:13:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 >> v1:172.21.15.7:6789/0 conn(0x7fa1080c43d0 legacy=0x7fa108154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0f0003a70 0x7fa0f0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:34.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10814e560 0x7fa0f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:34.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0f0004d90 0x7fa0f0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:34.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10806bef0 con 0x7fa0f0004d90 2023-03-03T15:13:34.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1081551e0 con 0x7fa0f0003a70 2023-03-03T15:13:34.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f000ba10 con 0x7fa10814e560 2023-03-03T15:13:34.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa106d9d700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0f0004d90 0x7fa0f0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:34.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa105d9b700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10814e560 0x7fa0f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:34.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10659c700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0f0003a70 0x7fa0f0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:34.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa106d9d700 1 -- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10814e560 msgr2=0x7fa0f0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:13:34.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa106d9d700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10814e560 0x7fa0f0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa106d9d700 1 -- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0f0003a70 msgr2=0x7fa0f0003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:13:34.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa106d9d700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0f0003a70 0x7fa0f0003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa106d9d700 1 -- 172.21.15.7:0/3971709846 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f00035b0 con 0x7fa0f0004d90 2023-03-03T15:13:34.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa106d9d700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0f0004d90 0x7fa0f0005200 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fa10000e170 tx=0x7fa0f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5430dc4f59c4a2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:13:34.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10000ec30 con 0x7fa0f0004d90 2023-03-03T15:13:34.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10000f040 con 0x7fa0f0004d90 2023-03-03T15:13:34.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.099+0000 7fa10559a700 1 -- 172.21.15.7:0/3971709846 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa100014480 con 0x7fa0f0004d90 2023-03-03T15:13:34.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.100+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3971709846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0f0004d90 msgr2=0x7fa0f0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:34.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.100+0000 7fa10cfa1700 1 --2- 172.21.15.7:0/3971709846 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0f0004d90 0x7fa0f0005200 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fa10000e170 tx=0x7fa0f0005880 comp rx=0 tx=0).stop 2023-03-03T15:13:34.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.100+0000 7fa106d9d700 1 -- 172.21.15.7:0/3971709846 reap_dead start 2023-03-03T15:13:34.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.100+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3971709846 shutdown_connections 2023-03-03T15:13:34.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.100+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3971709846 >> 172.21.15.7:0/3971709846 conn(0x7fa1080736f0 msgr2=0x7fa108073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:13:34.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.100+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3971709846 shutdown_connections 2023-03-03T15:13:34.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.100+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3971709846 wait complete. 2023-03-03T15:13:34.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.102+0000 7fa10cfa1700 1 Processor -- start 2023-03-03T15:13:34.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.102+0000 7fa10cfa1700 1 -- start start 2023-03-03T15:13:34.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa10cfa1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0f0004d90 0x7fa1081ed0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:34.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa10cfa1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1081ddbf0 0x7fa1081ed5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:34.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa10cfa1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10814fe70 0x7fa1081502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:34.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa10cfa1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10814ed90 con 0x7fa1081ddbf0 2023-03-03T15:13:34.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa10cfa1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1080be7b0 con 0x7fa10814fe70 2023-03-03T15:13:34.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa10cfa1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1080bec40 con 0x7fa0f0004d90 2023-03-03T15:13:34.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa105d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1081ddbf0 0x7fa1081ed5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:34.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa105d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1081ddbf0 0x7fa1081ed5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52090/0 (socket says 172.21.15.7:52090) 2023-03-03T15:13:34.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa105d9b700 1 -- 172.21.15.7:0/3832725597 learned_addr learned my addr 172.21.15.7:0/3832725597 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:13:34.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa106d9d700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10814fe70 0x7fa1081502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:34.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa10659c700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0f0004d90 0x7fa1081ed0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:34.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa105d9b700 1 -- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0f0004d90 msgr2=0x7fa1081ed0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:34.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa105d9b700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0f0004d90 0x7fa1081ed0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa105d9b700 1 -- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10814fe70 msgr2=0x7fa1081502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:34.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa105d9b700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10814fe70 0x7fa1081502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa105d9b700 1 -- 172.21.15.7:0/3832725597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1080aacc0 con 0x7fa1081ddbf0 2023-03-03T15:13:34.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa105d9b700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1081ddbf0 0x7fa1081ed5e0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc001cf0 tx=0x7fa0fc0075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:13:34.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa0f6ffd700 1 -- 172.21.15.7:0/3832725597 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc00f690 con 0x7fa1081ddbf0 2023-03-03T15:13:34.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.103+0000 7fa0f6ffd700 1 -- 172.21.15.7:0/3832725597 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0fc00f810 con 0x7fa1081ddbf0 2023-03-03T15:13:34.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.104+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3832725597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1080c4ca0 con 0x7fa1081ddbf0 2023-03-03T15:13:34.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.104+0000 7fa0f6ffd700 1 -- 172.21.15.7:0/3832725597 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc00fbc0 con 0x7fa1081ddbf0 2023-03-03T15:13:34.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.104+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3832725597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10806bb80 con 0x7fa1081ddbf0 2023-03-03T15:13:34.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.104+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3832725597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1080be960 con 0x7fa1081ddbf0 2023-03-03T15:13:34.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.105+0000 7fa0f6ffd700 1 -- 172.21.15.7:0/3832725597 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa0fc00fd20 con 0x7fa1081ddbf0 2023-03-03T15:13:34.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.105+0000 7fa0f6ffd700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0e80682f0 0x7fa0e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:34.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.106+0000 7fa0f6ffd700 1 -- 172.21.15.7:0/3832725597 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa0fc086080 con 0x7fa1081ddbf0 2023-03-03T15:13:34.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.106+0000 7fa10659c700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0e80682f0 0x7fa0e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:34.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.106+0000 7fa10659c700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0e80682f0 0x7fa0e806a7a0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa1080c2b10 tx=0x7fa0f8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:13:34.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.110+0000 7fa0f6ffd700 1 -- 172.21.15.7:0/3832725597 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa1080be960 con 0x7fa1081ddbf0 2023-03-03T15:13:34.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.349+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3832725597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1080be960 con 0x7fa1081ddbf0 2023-03-03T15:13:34.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.350+0000 7fa0f6ffd700 1 -- 172.21.15.7:0/3832725597 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa1080be960 con 0x7fa1081ddbf0 2023-03-03T15:13:34.353 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:13:34.353 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:13:34.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0e80682f0 msgr2=0x7fa0e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:34.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0e80682f0 0x7fa0e806a7a0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa1080c2b10 tx=0x7fa0f8009210 comp rx=0 tx=0).stop 2023-03-03T15:13:34.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3832725597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1081ddbf0 msgr2=0x7fa1081ed5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:34.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1081ddbf0 0x7fa1081ed5e0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc001cf0 tx=0x7fa0fc0075f0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3832725597 shutdown_connections 2023-03-03T15:13:34.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa10814fe70 0x7fa1081502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0f0004d90 0x7fa1081ed0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0e80682f0 0x7fa0e806a7a0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 --2- 172.21.15.7:0/3832725597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1081ddbf0 0x7fa1081ed5e0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3832725597 >> 172.21.15.7:0/3832725597 conn(0x7fa1080736f0 msgr2=0x7fa1081549c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:13:34.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3832725597 shutdown_connections 2023-03-03T15:13:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:34.352+0000 7fa10cfa1700 1 -- 172.21.15.7:0/3832725597 wait complete. 2023-03-03T15:13:34.357 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:13:34.366 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T15:13:35.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:13:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.907+0000 7f2adec06700 1 Processor -- start 2023-03-03T15:13:35.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.908+0000 7f2adec06700 1 -- start start 2023-03-03T15:13:35.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.908+0000 7f2adec06700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2ad8072250 0x7f2ad814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.908+0000 7f2adec06700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ad80c1ce0 0x7f2ad814e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.908+0000 7f2adec06700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad81510d0 con 0x7f2ad8071e80 2023-03-03T15:13:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.908+0000 7f2adec06700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ad80c3740 con 0x7f2ad80c1ce0 2023-03-03T15:13:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.908+0000 7f2adec06700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ad8070670 con 0x7f2ad8072250 2023-03-03T15:13:35.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.908+0000 7f2add1a3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ad80c1ce0 0x7f2ad814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2adc9a2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2ad8071e80 0x7f2ad8151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46076/0 (socket says 172.21.15.7:46076) 2023-03-03T15:13:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2adc9a2700 1 -- 172.21.15.7:0/3193168416 learned_addr learned my addr 172.21.15.7:0/3193168416 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:13:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2ad7fff700 1 --2- 172.21.15.7:0/3193168416 >> v2:172.21.15.143:3300/0 conn(0x7f2ad8072250 0x7f2ad814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2add1a3700 1 -- 172.21.15.7:0/3193168416 >> v2:172.21.15.143:3300/0 conn(0x7f2ad8072250 msgr2=0x7f2ad814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:13:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2add1a3700 1 --2- 172.21.15.7:0/3193168416 >> v2:172.21.15.143:3300/0 conn(0x7f2ad8072250 0x7f2ad814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2add1a3700 1 -- 172.21.15.7:0/3193168416 >> v1:172.21.15.7:6789/0 conn(0x7f2ad8071e80 legacy=0x7f2ad8151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:35.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2add1a3700 1 -- 172.21.15.7:0/3193168416 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad80c39e0 con 0x7f2ad80c1ce0 2023-03-03T15:13:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2ad77fe700 1 -- 172.21.15.7:0/3193168416 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979327114 0 0) 0x7f2ad81510d0 con 0x7f2ad8071e80 2023-03-03T15:13:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2add1a3700 1 --2- 172.21.15.7:0/3193168416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad80c1ce0 0x7f2ad814e730 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f2ad806b520 tx=0x7f2ad000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=300f453034d24478 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:13:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2ad77fe700 1 -- 172.21.15.7:0/3193168416 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000e070 con 0x7f2ad80c1ce0 2023-03-03T15:13:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2ad77fe700 1 -- 172.21.15.7:0/3193168416 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad000bbd0 con 0x7f2ad80c1ce0 2023-03-03T15:13:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.909+0000 7f2ad77fe700 1 -- 172.21.15.7:0/3193168416 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0012430 con 0x7f2ad80c1ce0 2023-03-03T15:13:35.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.910+0000 7f2adec06700 1 -- 172.21.15.7:0/3193168416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad80c1ce0 msgr2=0x7f2ad814e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.910+0000 7f2adec06700 1 --2- 172.21.15.7:0/3193168416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad80c1ce0 0x7f2ad814e730 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f2ad806b520 tx=0x7f2ad000b330 comp rx=0 tx=0).stop 2023-03-03T15:13:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.910+0000 7f2adec06700 1 -- 172.21.15.7:0/3193168416 shutdown_connections 2023-03-03T15:13:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.910+0000 7f2adec06700 1 --2- 172.21.15.7:0/3193168416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad80c1ce0 0x7f2ad814e730 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.910+0000 7f2adec06700 1 --2- 172.21.15.7:0/3193168416 >> v2:172.21.15.143:3300/0 conn(0x7f2ad8072250 0x7f2ad814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.910+0000 7f2adec06700 1 -- 172.21.15.7:0/3193168416 >> 172.21.15.7:0/3193168416 conn(0x7f2ad80a94d0 msgr2=0x7f2ad80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:13:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.910+0000 7f2adec06700 1 -- 172.21.15.7:0/3193168416 shutdown_connections 2023-03-03T15:13:35.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.910+0000 7f2adec06700 1 -- 172.21.15.7:0/3193168416 wait complete. 2023-03-03T15:13:35.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.911+0000 7f2adec06700 1 Processor -- start 2023-03-03T15:13:35.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.911+0000 7f2adec06700 1 -- start start 2023-03-03T15:13:35.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2adec06700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ad8071e80 0x7f2ad81464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:35.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2adec06700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad8072250 0x7f2ad8144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:35.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2adec06700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ad80c1ce0 0x7f2ad8144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2adec06700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad81eb9b0 con 0x7f2ad8072250 2023-03-03T15:13:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2adec06700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad81dca10 con 0x7f2ad80c1ce0 2023-03-03T15:13:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2adec06700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad81dcea0 con 0x7f2ad8071e80 2023-03-03T15:13:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2ad7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad8072250 0x7f2ad8144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2ad7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad8072250 0x7f2ad8144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52104/0 (socket says 172.21.15.7:52104) 2023-03-03T15:13:35.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2ad7fff700 1 -- 172.21.15.7:0/1085438394 learned_addr learned my addr 172.21.15.7:0/1085438394 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:13:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2add1a3700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ad80c1ce0 0x7f2ad8144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2adc9a2700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ad8071e80 0x7f2ad81464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2ad7fff700 1 -- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ad8071e80 msgr2=0x7f2ad81464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2ad7fff700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ad8071e80 0x7f2ad81464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2ad7fff700 1 -- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ad80c1ce0 msgr2=0x7f2ad8144560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2ad7fff700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ad80c1ce0 0x7f2ad8144560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:35.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2ad7fff700 1 -- 172.21.15.7:0/1085438394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad0009a70 con 0x7f2ad8072250 2023-03-03T15:13:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.912+0000 7f2add1a3700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ad80c1ce0 0x7f2ad8144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:13:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.913+0000 7f2ad7fff700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad8072250 0x7f2ad8144020 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f2ac400e810 tx=0x7f2ac400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:13:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.913+0000 7f2ad57fa700 1 -- 172.21.15.7:0/1085438394 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4013610 con 0x7f2ad8072250 2023-03-03T15:13:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.913+0000 7f2ad57fa700 1 -- 172.21.15.7:0/1085438394 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac4013770 con 0x7f2ad8072250 2023-03-03T15:13:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.913+0000 7f2adec06700 1 -- 172.21.15.7:0/1085438394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad8151f30 con 0x7f2ad8072250 2023-03-03T15:13:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.913+0000 7f2ad57fa700 1 -- 172.21.15.7:0/1085438394 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4013b00 con 0x7f2ad8072250 2023-03-03T15:13:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.913+0000 7f2adec06700 1 -- 172.21.15.7:0/1085438394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad8151300 con 0x7f2ad8072250 2023-03-03T15:13:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.914+0000 7f2adec06700 1 -- 172.21.15.7:0/1085438394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad806c9a0 con 0x7f2ad8072250 2023-03-03T15:13:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.914+0000 7f2ad57fa700 1 -- 172.21.15.7:0/1085438394 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ac4023070 con 0x7f2ad8072250 2023-03-03T15:13:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.915+0000 7f2ad57fa700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2abc0682f0 0x7f2abc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:13:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.915+0000 7f2ad57fa700 1 -- 172.21.15.7:0/1085438394 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2ac4085e20 con 0x7f2ad8072250 2023-03-03T15:13:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.915+0000 7f2adc9a2700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2abc0682f0 0x7f2abc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:13:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.915+0000 7f2adc9a2700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2abc0682f0 0x7f2abc06a7a0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f2acc000f60 tx=0x7f2acc007c00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:13:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:35.919+0000 7f2ad57fa700 1 -- 172.21.15.7:0/1085438394 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ac40521c0 con 0x7f2ad8072250 2023-03-03T15:13:36.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.156+0000 7f2adec06700 1 -- 172.21.15.7:0/1085438394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ad81dc850 con 0x7f2ad8072250 2023-03-03T15:13:36.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.157+0000 7f2ad57fa700 1 -- 172.21.15.7:0/1085438394 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2ad81dc850 con 0x7f2ad8072250 2023-03-03T15:13:36.160 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:13:36.160 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:13:36.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 -- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2abc0682f0 msgr2=0x7f2abc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2abc0682f0 0x7f2abc06a7a0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f2acc000f60 tx=0x7f2acc007c00 comp rx=0 tx=0).stop 2023-03-03T15:13:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 -- 172.21.15.7:0/1085438394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad8072250 msgr2=0x7f2ad8144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:13:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad8072250 0x7f2ad8144020 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f2ac400e810 tx=0x7f2ac400ebd0 comp rx=0 tx=0).stop 2023-03-03T15:13:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 -- 172.21.15.7:0/1085438394 shutdown_connections 2023-03-03T15:13:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ad80c1ce0 0x7f2ad8144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ad8071e80 0x7f2ad81464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2abc0682f0 0x7f2abc06a7a0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 --2- 172.21.15.7:0/1085438394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ad8072250 0x7f2ad8144020 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:13:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 -- 172.21.15.7:0/1085438394 >> 172.21.15.7:0/1085438394 conn(0x7f2ad80a94d0 msgr2=0x7f2ad80c45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:13:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 -- 172.21.15.7:0/1085438394 shutdown_connections 2023-03-03T15:13:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:13:36.159+0000 7f2adec06700 1 -- 172.21.15.7:0/1085438394 wait complete. 2023-03-03T15:13:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:13:36.173 INFO:tasks.mds_thrash.fs.[a]:waiting for 67 secs before thrashing 2023-03-03T15:14:01.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:01.060 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:31.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:31.133 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:14:38.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:14:38.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.504+0000 7fb15b04b700 1 Processor -- start 2023-03-03T15:14:38.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.504+0000 7fb15b04b700 1 -- start start 2023-03-03T15:14:38.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb15b04b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb15413a0f0 0x7fb1540c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb15b04b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1540c1da0 0x7fb15406d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb15b04b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15406de90 con 0x7fb1541586c0 2023-03-03T15:14:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb15b04b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb154154840 con 0x7fb15413a0f0 2023-03-03T15:14:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb15b04b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb154154560 con 0x7fb1540c1da0 2023-03-03T15:14:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb153fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb1541586c0 0x7fb15406d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46132/0 (socket says 172.21.15.7:46132) 2023-03-03T15:14:38.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb153fff700 1 -- 172.21.15.7:0/1834590701 learned_addr learned my addr 172.21.15.7:0/1834590701 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:14:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb1595e8700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1540c1da0 0x7fb15406d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb158de7700 1 --2- 172.21.15.7:0/1834590701 >> v2:172.21.15.143:3300/0 conn(0x7fb15413a0f0 0x7fb1540c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192675159 0 0) 0x7fb140001110 con 0x7fb1541586c0 2023-03-03T15:14:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1440035b0 con 0x7fb1541586c0 2023-03-03T15:14:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3426589428 0 0) 0x7fb1440035b0 con 0x7fb1541586c0 2023-03-03T15:14:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.505+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1540c1da0 msgr2=0x7fb15406d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:14:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1540c1da0 0x7fb15406d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 >> v2:172.21.15.143:3300/0 conn(0x7fb15413a0f0 msgr2=0x7fb1540c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:14:38.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 --2- 172.21.15.7:0/1834590701 >> v2:172.21.15.143:3300/0 conn(0x7fb15413a0f0 0x7fb1540c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb154157be0 con 0x7fb1541586c0 2023-03-03T15:14:38.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb140001800 con 0x7fb1541586c0 2023-03-03T15:14:38.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 >> v1:172.21.15.7:6789/0 conn(0x7fb1541586c0 legacy=0x7fb15406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:38.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb144003a70 0x7fb144003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:38.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb144004850 0x7fb144004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:38.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144005200 0x7fb14400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb154154840 con 0x7fb144003a70 2023-03-03T15:14:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb154154560 con 0x7fb144005200 2023-03-03T15:14:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14400c050 con 0x7fb144004850 2023-03-03T15:14:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb158de7700 1 --2- 172.21.15.7:0/1834590701 >> v2:172.21.15.143:3300/0 conn(0x7fb15413a0f0 0x7fb1540c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:14:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1595e8700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb1540c1da0 0x7fb15406d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:14:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb158de7700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb144003a70 0x7fb144003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:38.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb153fff700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb144004850 0x7fb144004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1595e8700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144005200 0x7fb14400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb158de7700 1 -- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb144004850 msgr2=0x7fb144004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:14:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb158de7700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb144004850 0x7fb144004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb158de7700 1 -- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144005200 msgr2=0x7fb14400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:14:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb158de7700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144005200 0x7fb14400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb158de7700 1 -- 172.21.15.7:0/1834590701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1440035b0 con 0x7fb144003a70 2023-03-03T15:14:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb158de7700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb144003a70 0x7fb144003e80 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fb148009d10 tx=0x7fb144005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d5e8aed3e8291a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:38.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.506+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb148009da0 con 0x7fb144003a70 2023-03-03T15:14:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.507+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14800e9a0 con 0x7fb144003a70 2023-03-03T15:14:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.507+0000 7fb1537fe700 1 -- 172.21.15.7:0/1834590701 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14800ed10 con 0x7fb144003a70 2023-03-03T15:14:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.507+0000 7fb15b04b700 1 -- 172.21.15.7:0/1834590701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb144003a70 msgr2=0x7fb144003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.507+0000 7fb15b04b700 1 --2- 172.21.15.7:0/1834590701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb144003a70 0x7fb144003e80 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fb148009d10 tx=0x7fb144005880 comp rx=0 tx=0).stop 2023-03-03T15:14:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.507+0000 7fb1595e8700 1 -- 172.21.15.7:0/1834590701 reap_dead start 2023-03-03T15:14:38.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.507+0000 7fb15b04b700 1 -- 172.21.15.7:0/1834590701 shutdown_connections 2023-03-03T15:14:38.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.507+0000 7fb15b04b700 1 -- 172.21.15.7:0/1834590701 >> 172.21.15.7:0/1834590701 conn(0x7fb15406c110 msgr2=0x7fb15406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:14:38.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.507+0000 7fb15b04b700 1 -- 172.21.15.7:0/1834590701 shutdown_connections 2023-03-03T15:14:38.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.507+0000 7fb15b04b700 1 -- 172.21.15.7:0/1834590701 wait complete. 2023-03-03T15:14:38.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.508+0000 7fb15b04b700 1 Processor -- start 2023-03-03T15:14:38.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.508+0000 7fb15b04b700 1 -- start start 2023-03-03T15:14:38.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.509+0000 7fb15b04b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144003a70 0x7fb154134670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:38.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.509+0000 7fb15b04b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1541ee290 0x7fb1540c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:38.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.509+0000 7fb15b04b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb154134c30 0x7fb154155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:38.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.509+0000 7fb15b04b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541ede00 con 0x7fb1541ee290 2023-03-03T15:14:38.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.509+0000 7fb15b04b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb154071eb0 con 0x7fb144003a70 2023-03-03T15:14:38.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.509+0000 7fb15b04b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541ed7b0 con 0x7fb154134c30 2023-03-03T15:14:38.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.509+0000 7fb153fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1541ee290 0x7fb1540c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:38.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.509+0000 7fb153fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1541ee290 0x7fb1540c0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52166/0 (socket says 172.21.15.7:52166) 2023-03-03T15:14:38.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.509+0000 7fb153fff700 1 -- 172.21.15.7:0/4058719982 learned_addr learned my addr 172.21.15.7:0/4058719982 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:14:38.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.509+0000 7fb158de7700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144003a70 0x7fb154134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb1595e8700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb154134c30 0x7fb154155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb153fff700 1 -- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb154134c30 msgr2=0x7fb154155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb153fff700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb154134c30 0x7fb154155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb153fff700 1 -- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144003a70 msgr2=0x7fb154134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb153fff700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144003a70 0x7fb154134670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb153fff700 1 -- 172.21.15.7:0/4058719982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1541581a0 con 0x7fb1541ee290 2023-03-03T15:14:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb153fff700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1541ee290 0x7fb1540c0a40 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7fb140001e50 tx=0x7fb1400075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:38.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb1517fa700 1 -- 172.21.15.7:0/4058719982 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb140011580 con 0x7fb1541ee290 2023-03-03T15:14:38.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb1517fa700 1 -- 172.21.15.7:0/4058719982 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1400116e0 con 0x7fb1541ee290 2023-03-03T15:14:38.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb1517fa700 1 -- 172.21.15.7:0/4058719982 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb140011a50 con 0x7fb1541ee290 2023-03-03T15:14:38.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb15b04b700 1 -- 172.21.15.7:0/4058719982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb154157ec0 con 0x7fb1541ee290 2023-03-03T15:14:38.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.510+0000 7fb15b04b700 1 -- 172.21.15.7:0/4058719982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15406e1a0 con 0x7fb1541ee290 2023-03-03T15:14:38.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.511+0000 7fb15b04b700 1 -- 172.21.15.7:0/4058719982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1541ee0e0 con 0x7fb1541ee290 2023-03-03T15:14:38.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.512+0000 7fb1517fa700 1 -- 172.21.15.7:0/4058719982 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb140011bb0 con 0x7fb1541ee290 2023-03-03T15:14:38.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.512+0000 7fb1517fa700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb138068320 0x7fb13806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:38.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.512+0000 7fb1517fa700 1 -- 172.21.15.7:0/4058719982 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb140086110 con 0x7fb1541ee290 2023-03-03T15:14:38.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.512+0000 7fb158de7700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb138068320 0x7fb13806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:38.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.513+0000 7fb158de7700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb138068320 0x7fb13806a7d0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fb1480090b0 tx=0x7fb148008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:38.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.516+0000 7fb1517fa700 1 -- 172.21.15.7:0/4058719982 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb140052420 con 0x7fb1541ee290 2023-03-03T15:14:38.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.757+0000 7fb15b04b700 1 -- 172.21.15.7:0/4058719982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1541ed610 con 0x7fb1541ee290 2023-03-03T15:14:38.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.757+0000 7fb1517fa700 1 -- 172.21.15.7:0/4058719982 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb1541ed610 con 0x7fb1541ee290 2023-03-03T15:14:38.760 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:14:38.761 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:14:38.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 -- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb138068320 msgr2=0x7fb13806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:38.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb138068320 0x7fb13806a7d0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fb1480090b0 tx=0x7fb148008030 comp rx=0 tx=0).stop 2023-03-03T15:14:38.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 -- 172.21.15.7:0/4058719982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1541ee290 msgr2=0x7fb1540c0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:38.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1541ee290 0x7fb1540c0a40 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7fb140001e50 tx=0x7fb1400075f0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 -- 172.21.15.7:0/4058719982 shutdown_connections 2023-03-03T15:14:38.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb154134c30 0x7fb154155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb144003a70 0x7fb154134670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb138068320 0x7fb13806a7d0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 --2- 172.21.15.7:0/4058719982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1541ee290 0x7fb1540c0a40 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:38.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 -- 172.21.15.7:0/4058719982 >> 172.21.15.7:0/4058719982 conn(0x7fb15406c110 msgr2=0x7fb154139510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:14:38.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 -- 172.21.15.7:0/4058719982 shutdown_connections 2023-03-03T15:14:38.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:38.760+0000 7fb15b04b700 1 -- 172.21.15.7:0/4058719982 wait complete. 2023-03-03T15:14:38.764 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:14:38.774 INFO:tasks.mds_thrash.fs.[b]:waiting for 33 secs before thrashing 2023-03-03T15:14:43.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:14:43.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.311+0000 7f65bf860700 1 Processor -- start 2023-03-03T15:14:43.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.311+0000 7f65bf860700 1 -- start start 2023-03-03T15:14:43.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bf860700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8151e80 0x7f65b80c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:43.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bf860700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65b8155410 0x7f65b80bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:43.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bf860700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f65b80c6fe0 0x7f65b80bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:43.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bf860700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b8152250 con 0x7f65b8155410 2023-03-03T15:14:43.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bf860700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b80c0fb0 con 0x7f65b80c6fe0 2023-03-03T15:14:43.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bf860700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b8154740 con 0x7f65b8151e80 2023-03-03T15:14:43.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bcdfb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65b8155410 0x7f65b80bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:43.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bcdfb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f65b8155410 0x7f65b80bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52178/0 (socket says 172.21.15.7:52178) 2023-03-03T15:14:43.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bcdfb700 1 -- 172.21.15.7:0/73304918 learned_addr learned my addr 172.21.15.7:0/73304918 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:14:43.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bd5fc700 1 --2- 172.21.15.7:0/73304918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8151e80 0x7f65b80c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:43.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bddfd700 1 --2- 172.21.15.7:0/73304918 >> v2:172.21.15.143:3300/0 conn(0x7f65b80c6fe0 0x7f65b80bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:43.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bcdfb700 1 -- 172.21.15.7:0/73304918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8151e80 msgr2=0x7f65b80c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:14:43.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bcdfb700 1 --2- 172.21.15.7:0/73304918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8151e80 0x7f65b80c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bcdfb700 1 -- 172.21.15.7:0/73304918 >> v2:172.21.15.143:3300/0 conn(0x7f65b80c6fe0 msgr2=0x7f65b80bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:14:43.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bcdfb700 1 --2- 172.21.15.7:0/73304918 >> v2:172.21.15.143:3300/0 conn(0x7f65b80c6fe0 0x7f65b80bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bcdfb700 1 -- 172.21.15.7:0/73304918 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b80c08d0 con 0x7f65b8155410 2023-03-03T15:14:43.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.312+0000 7f65bddfd700 1 --2- 172.21.15.7:0/73304918 >> v2:172.21.15.143:3300/0 conn(0x7f65b80c6fe0 0x7f65b80bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:14:43.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65bcdfb700 1 --2- 172.21.15.7:0/73304918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8155410 0x7f65b80bf8e0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f65b808c770 tx=0x7f65b4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad1cc0ffc148d28 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:43.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65affff700 1 -- 172.21.15.7:0/73304918 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4011830 con 0x7f65b8155410 2023-03-03T15:14:43.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65affff700 1 -- 172.21.15.7:0/73304918 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b4011990 con 0x7f65b8155410 2023-03-03T15:14:43.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65affff700 1 -- 172.21.15.7:0/73304918 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4011d20 con 0x7f65b8155410 2023-03-03T15:14:43.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65bf860700 1 -- 172.21.15.7:0/73304918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8155410 msgr2=0x7f65b80bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:43.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65bf860700 1 --2- 172.21.15.7:0/73304918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8155410 0x7f65b80bf8e0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f65b808c770 tx=0x7f65b4009bd0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65bf860700 1 -- 172.21.15.7:0/73304918 shutdown_connections 2023-03-03T15:14:43.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65bf860700 1 --2- 172.21.15.7:0/73304918 >> v2:172.21.15.143:3300/0 conn(0x7f65b80c6fe0 0x7f65b80bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65bf860700 1 --2- 172.21.15.7:0/73304918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8151e80 0x7f65b80c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65bf860700 1 --2- 172.21.15.7:0/73304918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8155410 0x7f65b80bf8e0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65bf860700 1 -- 172.21.15.7:0/73304918 >> 172.21.15.7:0/73304918 conn(0x7f65b8071630 msgr2=0x7f65b806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:14:43.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65bf860700 1 -- 172.21.15.7:0/73304918 shutdown_connections 2023-03-03T15:14:43.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.313+0000 7f65bf860700 1 -- 172.21.15.7:0/73304918 wait complete. 2023-03-03T15:14:43.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.315+0000 7f65bf860700 1 Processor -- start 2023-03-03T15:14:43.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.315+0000 7f65bf860700 1 -- start start 2023-03-03T15:14:43.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.315+0000 7f65bf860700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65b80c6fe0 0x7f65b81ed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:43.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.315+0000 7f65bf860700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8151e80 0x7f65b80bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:43.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.315+0000 7f65bf860700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8155410 0x7f65b81e08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:43.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.315+0000 7f65bf860700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b80c8a60 con 0x7f65b8151e80 2023-03-03T15:14:43.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.315+0000 7f65bf860700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b8144240 con 0x7f65b80c6fe0 2023-03-03T15:14:43.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.315+0000 7f65bf860700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b8143ab0 con 0x7f65b8155410 2023-03-03T15:14:43.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bcdfb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8151e80 0x7f65b80bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:43.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bcdfb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8151e80 0x7f65b80bf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52186/0 (socket says 172.21.15.7:52186) 2023-03-03T15:14:43.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bcdfb700 1 -- 172.21.15.7:0/1444526022 learned_addr learned my addr 172.21.15.7:0/1444526022 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:14:43.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bd5fc700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65b80c6fe0 0x7f65b81ed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:43.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bddfd700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8155410 0x7f65b81e08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:43.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bcdfb700 1 -- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8155410 msgr2=0x7f65b81e08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:43.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bcdfb700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8155410 0x7f65b81e08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bcdfb700 1 -- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65b80c6fe0 msgr2=0x7f65b81ed7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:43.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bcdfb700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65b80c6fe0 0x7f65b81ed7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bcdfb700 1 -- 172.21.15.7:0/1444526022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b4009550 con 0x7f65b8151e80 2023-03-03T15:14:43.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bddfd700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8155410 0x7f65b81e08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:14:43.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bd5fc700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65b80c6fe0 0x7f65b81ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:14:43.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bcdfb700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8151e80 0x7f65b80bf940 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f65b4000c00 tx=0x7f65b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:43.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65adffb700 1 -- 172.21.15.7:0/1444526022 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4004380 con 0x7f65b8151e80 2023-03-03T15:14:43.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65adffb700 1 -- 172.21.15.7:0/1444526022 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b401f070 con 0x7f65b8151e80 2023-03-03T15:14:43.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bf860700 1 -- 172.21.15.7:0/1444526022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b8154c10 con 0x7f65b8151e80 2023-03-03T15:14:43.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65adffb700 1 -- 172.21.15.7:0/1444526022 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40036a0 con 0x7f65b8151e80 2023-03-03T15:14:43.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.316+0000 7f65bf860700 1 -- 172.21.15.7:0/1444526022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b8154930 con 0x7f65b8151e80 2023-03-03T15:14:43.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.317+0000 7f65bf860700 1 -- 172.21.15.7:0/1444526022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b8154ef0 con 0x7f65b8151e80 2023-03-03T15:14:43.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.318+0000 7f65adffb700 1 -- 172.21.15.7:0/1444526022 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f65b402a020 con 0x7f65b8151e80 2023-03-03T15:14:43.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.318+0000 7f65adffb700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f659c0682f0 0x7f659c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:43.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.318+0000 7f65adffb700 1 -- 172.21.15.7:0/1444526022 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f65b4087880 con 0x7f65b8151e80 2023-03-03T15:14:43.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.319+0000 7f65bd5fc700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f659c0682f0 0x7f659c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:43.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.319+0000 7f65bd5fc700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f659c0682f0 0x7f659c06a7a0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009a40 tx=0x7f65a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:43.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.323+0000 7f65adffb700 1 -- 172.21.15.7:0/1444526022 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f65b4053b30 con 0x7f65b8151e80 2023-03-03T15:14:43.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.561+0000 7f65bf860700 1 -- 172.21.15.7:0/1444526022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65b80cb5c0 con 0x7f65b8151e80 2023-03-03T15:14:43.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.562+0000 7f65adffb700 1 -- 172.21.15.7:0/1444526022 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f65b4057150 con 0x7f65b8151e80 2023-03-03T15:14:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:14:43.565 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:14:43.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 -- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f659c0682f0 msgr2=0x7f659c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:43.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f659c0682f0 0x7f659c06a7a0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009a40 tx=0x7f65a8008040 comp rx=0 tx=0).stop 2023-03-03T15:14:43.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 -- 172.21.15.7:0/1444526022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8151e80 msgr2=0x7f65b80bf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:43.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8151e80 0x7f65b80bf940 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f65b4000c00 tx=0x7f65b4004750 comp rx=0 tx=0).stop 2023-03-03T15:14:43.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 -- 172.21.15.7:0/1444526022 shutdown_connections 2023-03-03T15:14:43.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f65b8155410 0x7f65b81e08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f65b80c6fe0 0x7f65b81ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f659c0682f0 0x7f659c06a7a0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 --2- 172.21.15.7:0/1444526022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f65b8151e80 0x7f65b80bf940 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:43.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 -- 172.21.15.7:0/1444526022 >> 172.21.15.7:0/1444526022 conn(0x7f65b8071630 msgr2=0x7f65b80c2770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:14:43.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 -- 172.21.15.7:0/1444526022 shutdown_connections 2023-03-03T15:14:43.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:43.564+0000 7f65bf860700 1 -- 172.21.15.7:0/1444526022 wait complete. 2023-03-03T15:14:43.569 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:14:43.579 INFO:tasks.mds_thrash.fs.[a]:waiting for 16 secs before thrashing 2023-03-03T15:14:59.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:14:59.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.717+0000 7f57ec33d700 1 Processor -- start 2023-03-03T15:14:59.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.717+0000 7f57ec33d700 1 -- start start 2023-03-03T15:14:59.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ec33d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f57e4071990 0x7f57e4071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:59.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ec33d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f57e4151c00 0x7f57e40722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:59.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ec33d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e4070360 con 0x7f57e414b460 2023-03-03T15:14:59.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ec33d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e41521a0 con 0x7f57e4151c00 2023-03-03T15:14:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ec33d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e406a5e0 con 0x7f57e4071990 2023-03-03T15:14:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ea8da700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f57e4151c00 0x7f57e40722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ea8da700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f57e4151c00 0x7f57e40722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52202/0 (socket says 172.21.15.7:52202) 2023-03-03T15:14:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ea8da700 1 -- 172.21.15.7:0/1562430978 learned_addr learned my addr 172.21.15.7:0/1562430978 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:14:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ea0d9700 1 --2- 172.21.15.7:0/1562430978 >> v2:172.21.15.143:3300/0 conn(0x7f57e4071990 0x7f57e4071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ea8da700 1 -- 172.21.15.7:0/1562430978 >> v2:172.21.15.143:3300/0 conn(0x7f57e4071990 msgr2=0x7f57e4071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:14:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ea8da700 1 --2- 172.21.15.7:0/1562430978 >> v2:172.21.15.143:3300/0 conn(0x7f57e4071990 0x7f57e4071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ea8da700 1 -- 172.21.15.7:0/1562430978 >> v1:172.21.15.143:6789/0 conn(0x7f57e414b460 legacy=0x7f57e406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:14:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ea8da700 1 -- 172.21.15.7:0/1562430978 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40c38b0 con 0x7f57e4151c00 2023-03-03T15:14:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.718+0000 7f57ea8da700 1 --2- 172.21.15.7:0/1562430978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4151c00 0x7f57e40722a0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f57e40c1de0 tx=0x7f57dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5006a55e88c3a3f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57e90d7700 1 -- 172.21.15.7:0/1562430978 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00e070 con 0x7f57e4151c00 2023-03-03T15:14:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57e90d7700 1 -- 172.21.15.7:0/1562430978 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc003a20 con 0x7f57e4151c00 2023-03-03T15:14:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57e90d7700 1 -- 172.21.15.7:0/1562430978 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc003dd0 con 0x7f57e4151c00 2023-03-03T15:14:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57ec33d700 1 -- 172.21.15.7:0/1562430978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4151c00 msgr2=0x7f57e40722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57ec33d700 1 --2- 172.21.15.7:0/1562430978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4151c00 0x7f57e40722a0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f57e40c1de0 tx=0x7f57dc00b330 comp rx=0 tx=0).stop 2023-03-03T15:14:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57ec33d700 1 -- 172.21.15.7:0/1562430978 shutdown_connections 2023-03-03T15:14:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57ec33d700 1 --2- 172.21.15.7:0/1562430978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4151c00 0x7f57e40722a0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57ec33d700 1 --2- 172.21.15.7:0/1562430978 >> v2:172.21.15.143:3300/0 conn(0x7f57e4071990 0x7f57e4071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57ec33d700 1 -- 172.21.15.7:0/1562430978 >> 172.21.15.7:0/1562430978 conn(0x7f57e40a94d0 msgr2=0x7f57e40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:14:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57ec33d700 1 -- 172.21.15.7:0/1562430978 shutdown_connections 2023-03-03T15:14:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.719+0000 7f57ec33d700 1 -- 172.21.15.7:0/1562430978 wait complete. 2023-03-03T15:14:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.720+0000 7f57ec33d700 1 Processor -- start 2023-03-03T15:14:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57ec33d700 1 -- start start 2023-03-03T15:14:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57ec33d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4071990 0x7f57e41eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57ec33d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57e4151c00 0x7f57e41e0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:59.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57ec33d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57e40c0e70 0x7f57e41edce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57ec33d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e41ed850 con 0x7f57e4071990 2023-03-03T15:14:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57ec33d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e40ac950 con 0x7f57e4151c00 2023-03-03T15:14:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57ec33d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e41ed550 con 0x7f57e40c0e70 2023-03-03T15:14:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57ea0d9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4071990 0x7f57e41eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57ea0d9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4071990 0x7f57e41eb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52204/0 (socket says 172.21.15.7:52204) 2023-03-03T15:14:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57ea0d9700 1 -- 172.21.15.7:0/60087086 learned_addr learned my addr 172.21.15.7:0/60087086 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:14:59.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.721+0000 7f57e98d8700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57e4151c00 0x7f57e41e0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57ea8da700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57e40c0e70 0x7f57e41edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57ea0d9700 1 -- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57e40c0e70 msgr2=0x7f57e41edce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57ea0d9700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57e40c0e70 0x7f57e41edce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57ea0d9700 1 -- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57e4151c00 msgr2=0x7f57e41e0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57ea0d9700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57e4151c00 0x7f57e41e0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57ea0d9700 1 -- 172.21.15.7:0/60087086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57dc009b30 con 0x7f57e4071990 2023-03-03T15:14:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57ea0d9700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4071990 0x7f57e41eb520 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f57d400e870 tx=0x7f57d400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57daffd700 1 -- 172.21.15.7:0/60087086 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d400c5e0 con 0x7f57e4071990 2023-03-03T15:14:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57daffd700 1 -- 172.21.15.7:0/60087086 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57d400c740 con 0x7f57e4071990 2023-03-03T15:14:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57ec33d700 1 -- 172.21.15.7:0/60087086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e40c3b50 con 0x7f57e4071990 2023-03-03T15:14:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57daffd700 1 -- 172.21.15.7:0/60087086 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d400caf0 con 0x7f57e4071990 2023-03-03T15:14:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.722+0000 7f57ec33d700 1 -- 172.21.15.7:0/60087086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e406a2b0 con 0x7f57e4071990 2023-03-03T15:14:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.723+0000 7f57ec33d700 1 -- 172.21.15.7:0/60087086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e4074bb0 con 0x7f57e4071990 2023-03-03T15:14:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.724+0000 7f57daffd700 1 -- 172.21.15.7:0/60087086 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f57d4023070 con 0x7f57e4071990 2023-03-03T15:14:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.724+0000 7f57daffd700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57c80682f0 0x7f57c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:14:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.724+0000 7f57daffd700 1 -- 172.21.15.7:0/60087086 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f57d4085e20 con 0x7f57e4071990 2023-03-03T15:14:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.724+0000 7f57e98d8700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57c80682f0 0x7f57c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:14:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.725+0000 7f57e98d8700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57c80682f0 0x7f57c806a7a0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f57e414ea70 tx=0x7f57e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:14:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.728+0000 7f57daffd700 1 -- 172.21.15.7:0/60087086 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57d40521c0 con 0x7f57e4071990 2023-03-03T15:14:59.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.967+0000 7f57ec33d700 1 -- 172.21.15.7:0/60087086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57e41ed690 con 0x7f57e4071990 2023-03-03T15:14:59.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.967+0000 7f57daffd700 1 -- 172.21.15.7:0/60087086 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f57e41ed690 con 0x7f57e4071990 2023-03-03T15:14:59.969 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:14:59.970 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:14:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 -- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57c80682f0 msgr2=0x7f57c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57c80682f0 0x7f57c806a7a0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f57e414ea70 tx=0x7f57e0009040 comp rx=0 tx=0).stop 2023-03-03T15:14:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 -- 172.21.15.7:0/60087086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4071990 msgr2=0x7f57e41eb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:14:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4071990 0x7f57e41eb520 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f57d400e870 tx=0x7f57d400ec30 comp rx=0 tx=0).stop 2023-03-03T15:14:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 -- 172.21.15.7:0/60087086 shutdown_connections 2023-03-03T15:14:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57e40c0e70 0x7f57e41edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57c80682f0 0x7f57c806a7a0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57e4071990 0x7f57e41eb520 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 --2- 172.21.15.7:0/60087086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57e4151c00 0x7f57e41e0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:14:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 -- 172.21.15.7:0/60087086 >> 172.21.15.7:0/60087086 conn(0x7f57e40a94d0 msgr2=0x7f57e4071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:14:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 -- 172.21.15.7:0/60087086 shutdown_connections 2023-03-03T15:14:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:14:59.969+0000 7f57ec33d700 1 -- 172.21.15.7:0/60087086 wait complete. 2023-03-03T15:14:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:14:59.983 INFO:tasks.mds_thrash.fs.[a]:waiting for 70 secs before thrashing 2023-03-03T15:15:01.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:01.195 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:11.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:15:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.913+0000 7effd889d700 1 Processor -- start 2023-03-03T15:15:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.913+0000 7effd889d700 1 -- start start 2023-03-03T15:15:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.914+0000 7effd889d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7effd414f860 0x7effd41562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:15:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.914+0000 7effd889d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 0x7effd4152d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:15:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.914+0000 7effd889d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd4155e60 con 0x7effd40c27c0 2023-03-03T15:15:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.914+0000 7effd889d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd406d350 con 0x7effd414f860 2023-03-03T15:15:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.914+0000 7effd889d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd41558e0 con 0x7effd40c20d0 2023-03-03T15:15:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.914+0000 7effd259c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7effd414f860 0x7effd41562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:15:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.914+0000 7effd1d9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 0x7effd4152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:15:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.914+0000 7effd2d9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7effd40c27c0 0x7effd40736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:49720/0 (socket says 172.21.15.7:49720) 2023-03-03T15:15:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.914+0000 7effd2d9d700 1 -- 172.21.15.7:0/3574295530 learned_addr learned my addr 172.21.15.7:0/3574295530 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:15:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd159a700 1 -- 172.21.15.7:0/3574295530 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368617964 0 0) 0x7effd4155e60 con 0x7effd40c27c0 2023-03-03T15:15:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd159a700 1 -- 172.21.15.7:0/3574295530 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effbc0035b0 con 0x7effd40c27c0 2023-03-03T15:15:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd259c700 1 -- 172.21.15.7:0/3574295530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 msgr2=0x7effd4152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:15:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd259c700 1 --2- 172.21.15.7:0/3574295530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 0x7effd4152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:15:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd259c700 1 -- 172.21.15.7:0/3574295530 >> v1:172.21.15.143:6789/0 conn(0x7effd40c27c0 legacy=0x7effd40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:15:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd259c700 1 -- 172.21.15.7:0/3574295530 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd41532b0 con 0x7effd414f860 2023-03-03T15:15:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd1d9b700 1 --2- 172.21.15.7:0/3574295530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 0x7effd4152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:15:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd259c700 1 --2- 172.21.15.7:0/3574295530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effd414f860 0x7effd41562f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7effc8009840 tx=0x7effc8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32d93ecd5df0075c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:15:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd159a700 1 -- 172.21.15.7:0/3574295530 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8011830 con 0x7effd414f860 2023-03-03T15:15:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd159a700 1 -- 172.21.15.7:0/3574295530 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc8011990 con 0x7effd414f860 2023-03-03T15:15:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.915+0000 7effd159a700 1 -- 172.21.15.7:0/3574295530 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8011d00 con 0x7effd414f860 2023-03-03T15:15:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.916+0000 7effd889d700 1 -- 172.21.15.7:0/3574295530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effd414f860 msgr2=0x7effd41562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:15:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.916+0000 7effd889d700 1 --2- 172.21.15.7:0/3574295530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effd414f860 0x7effd41562f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7effc8009840 tx=0x7effc8009b50 comp rx=0 tx=0).stop 2023-03-03T15:15:11.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.916+0000 7effd889d700 1 -- 172.21.15.7:0/3574295530 shutdown_connections 2023-03-03T15:15:11.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.916+0000 7effd889d700 1 --2- 172.21.15.7:0/3574295530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effd414f860 0x7effd41562f0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:15:11.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.916+0000 7effd889d700 1 --2- 172.21.15.7:0/3574295530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 0x7effd4152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:15:11.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.916+0000 7effd889d700 1 -- 172.21.15.7:0/3574295530 >> 172.21.15.7:0/3574295530 conn(0x7effd40a94d0 msgr2=0x7effd40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:15:11.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.916+0000 7effd889d700 1 -- 172.21.15.7:0/3574295530 shutdown_connections 2023-03-03T15:15:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.916+0000 7effd889d700 1 -- 172.21.15.7:0/3574295530 wait complete. 2023-03-03T15:15:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.917+0000 7effd889d700 1 Processor -- start 2023-03-03T15:15:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.917+0000 7effd889d700 1 -- start start 2023-03-03T15:15:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd889d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 0x7effd41ebd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:15:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd889d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effd40c27c0 0x7effd40c2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:15:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd889d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effd414f860 0x7effd41e7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:15:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd889d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd41ec250 con 0x7effd40c27c0 2023-03-03T15:15:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd889d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd414e050 con 0x7effd414f860 2023-03-03T15:15:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd889d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd40c4450 con 0x7effd40c20d0 2023-03-03T15:15:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effd40c27c0 0x7effd40c2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:15:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effd40c27c0 0x7effd40c2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52224/0 (socket says 172.21.15.7:52224) 2023-03-03T15:15:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd1d9b700 1 -- 172.21.15.7:0/4188940264 learned_addr learned my addr 172.21.15.7:0/4188940264 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:15:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd259c700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 0x7effd41ebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:15:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd2d9d700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effd414f860 0x7effd41e7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:15:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd1d9b700 1 -- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 msgr2=0x7effd41ebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:15:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd1d9b700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 0x7effd41ebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:15:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd1d9b700 1 -- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effd414f860 msgr2=0x7effd41e7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:15:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd1d9b700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effd414f860 0x7effd41e7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:15:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.918+0000 7effd1d9b700 1 -- 172.21.15.7:0/4188940264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40094a0 con 0x7effd40c27c0 2023-03-03T15:15:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.919+0000 7effd1d9b700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effd40c27c0 0x7effd40c2e00 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7effc400ebe0 tx=0x7effc400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:15:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.919+0000 7effc2ffd700 1 -- 172.21.15.7:0/4188940264 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc4009760 con 0x7effd40c27c0 2023-03-03T15:15:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.919+0000 7effc2ffd700 1 -- 172.21.15.7:0/4188940264 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc400c850 con 0x7effd40c27c0 2023-03-03T15:15:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.919+0000 7effd889d700 1 -- 172.21.15.7:0/4188940264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc80094a0 con 0x7effd40c27c0 2023-03-03T15:15:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.919+0000 7effc2ffd700 1 -- 172.21.15.7:0/4188940264 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc400cbe0 con 0x7effd40c27c0 2023-03-03T15:15:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.919+0000 7effd889d700 1 -- 172.21.15.7:0/4188940264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd406d070 con 0x7effd40c27c0 2023-03-03T15:15:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.920+0000 7effd889d700 1 -- 172.21.15.7:0/4188940264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd406cd90 con 0x7effd40c27c0 2023-03-03T15:15:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.920+0000 7effc2ffd700 1 -- 172.21.15.7:0/4188940264 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7effc4019030 con 0x7effd40c27c0 2023-03-03T15:15:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.921+0000 7effc2ffd700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb40682f0 0x7effb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:15:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.921+0000 7effc2ffd700 1 -- 172.21.15.7:0/4188940264 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7effc4086010 con 0x7effd40c27c0 2023-03-03T15:15:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.921+0000 7effd259c700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb40682f0 0x7effb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:15:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.921+0000 7effd259c700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb40682f0 0x7effb406a7a0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7effc8009fd0 tx=0x7effc8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:15:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:11.925+0000 7effc2ffd700 1 -- 172.21.15.7:0/4188940264 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7effc40523b0 con 0x7effd40c27c0 2023-03-03T15:15:12.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.166+0000 7effd889d700 1 -- 172.21.15.7:0/4188940264 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effd4155af0 con 0x7effd40c27c0 2023-03-03T15:15:12.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.167+0000 7effc2ffd700 1 -- 172.21.15.7:0/4188940264 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7effc40559d0 con 0x7effd40c27c0 2023-03-03T15:15:12.169 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:15:12.170 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:15:12.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 -- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb40682f0 msgr2=0x7effb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:15:12.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb40682f0 0x7effb406a7a0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7effc8009fd0 tx=0x7effc8008030 comp rx=0 tx=0).stop 2023-03-03T15:15:12.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 -- 172.21.15.7:0/4188940264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effd40c27c0 msgr2=0x7effd40c2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:15:12.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effd40c27c0 0x7effd40c2e00 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7effc400ebe0 tx=0x7effc400eef0 comp rx=0 tx=0).stop 2023-03-03T15:15:12.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 -- 172.21.15.7:0/4188940264 shutdown_connections 2023-03-03T15:15:12.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7effd414f860 0x7effd41e7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:15:12.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7effd40c20d0 0x7effd41ebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:15:12.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7effb40682f0 0x7effb406a7a0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:15:12.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 --2- 172.21.15.7:0/4188940264 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7effd40c27c0 0x7effd40c2e00 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:15:12.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 -- 172.21.15.7:0/4188940264 >> 172.21.15.7:0/4188940264 conn(0x7effd40a94d0 msgr2=0x7effd40c4860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:15:12.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 -- 172.21.15.7:0/4188940264 shutdown_connections 2023-03-03T15:15:12.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:15:12.169+0000 7effd889d700 1 -- 172.21.15.7:0/4188940264 wait complete. 2023-03-03T15:15:12.174 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:15:12.182 INFO:tasks.mds_thrash.fs.[b]:waiting for 90 secs before thrashing 2023-03-03T15:15:31.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:15:31.259 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:01.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:01.323 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:09.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:16:10.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.121+0000 7fe4fca32700 1 Processor -- start 2023-03-03T15:16:10.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.121+0000 7fe4fca32700 1 -- start start 2023-03-03T15:16:10.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4fca32700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80caa30 0x7fe4f80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4fca32700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f80c7310 con 0x7fe4f8154ff0 2023-03-03T15:16:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4fca32700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f80bfd10 con 0x7fe4f80c00e0 2023-03-03T15:16:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4fca32700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f80bfa30 con 0x7fe4f80caa30 2023-03-03T15:16:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f6d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe4f8154ff0 0x7fe4f815ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46258/0 (socket says 172.21.15.7:46258) 2023-03-03T15:16:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f6d9d700 1 -- 172.21.15.7:0/3025535078 learned_addr learned my addr 172.21.15.7:0/3025535078 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:16:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f659c700 1 --2- 172.21.15.7:0/3025535078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80caa30 0x7fe4f80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:10.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f559a700 1 -- 172.21.15.7:0/3025535078 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118589858 0 0) 0x7fe4f80c7310 con 0x7fe4f8154ff0 2023-03-03T15:16:10.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f559a700 1 -- 172.21.15.7:0/3025535078 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e00035b0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f559a700 1 -- 172.21.15.7:0/3025535078 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595113999 0 0) 0x7fe4f80bfd10 con 0x7fe4f80c00e0 2023-03-03T15:16:10.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f559a700 1 -- 172.21.15.7:0/3025535078 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f80c7310 con 0x7fe4f80c00e0 2023-03-03T15:16:10.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f559a700 1 -- 172.21.15.7:0/3025535078 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2145232069 0 0) 0x7fe4e00035b0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f659c700 1 -- 172.21.15.7:0/3025535078 >> v1:172.21.15.143:6789/0 conn(0x7fe4f80c00e0 legacy=0x7fe4f80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:10.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f659c700 1 -- 172.21.15.7:0/3025535078 >> v1:172.21.15.7:6789/0 conn(0x7fe4f8154ff0 legacy=0x7fe4f815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:10.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.122+0000 7fe4f659c700 1 -- 172.21.15.7:0/3025535078 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f8154510 con 0x7fe4f80caa30 2023-03-03T15:16:10.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.123+0000 7fe4f659c700 1 --2- 172.21.15.7:0/3025535078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80caa30 0x7fe4f80c5c60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009870 tx=0x7fe4e8009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d76e0be295e7c470 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:16:10.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.123+0000 7fe4f559a700 1 -- 172.21.15.7:0/3025535078 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00019f0 con 0x7fe4f80caa30 2023-03-03T15:16:10.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.123+0000 7fe4f559a700 1 -- 172.21.15.7:0/3025535078 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e8004030 con 0x7fe4f80caa30 2023-03-03T15:16:10.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.123+0000 7fe4f559a700 1 -- 172.21.15.7:0/3025535078 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e80043a0 con 0x7fe4f80caa30 2023-03-03T15:16:10.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.123+0000 7fe4fca32700 1 -- 172.21.15.7:0/3025535078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80caa30 msgr2=0x7fe4f80c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:10.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.123+0000 7fe4fca32700 1 --2- 172.21.15.7:0/3025535078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80caa30 0x7fe4f80c5c60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009870 tx=0x7fe4e8009b80 comp rx=0 tx=0).stop 2023-03-03T15:16:10.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.123+0000 7fe4fca32700 1 -- 172.21.15.7:0/3025535078 shutdown_connections 2023-03-03T15:16:10.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.123+0000 7fe4fca32700 1 --2- 172.21.15.7:0/3025535078 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80caa30 0x7fe4f80c5c60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:10.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.123+0000 7fe4fca32700 1 -- 172.21.15.7:0/3025535078 >> 172.21.15.7:0/3025535078 conn(0x7fe4f8066e40 msgr2=0x7fe4f8071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:16:10.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.124+0000 7fe4fca32700 1 -- 172.21.15.7:0/3025535078 shutdown_connections 2023-03-03T15:16:10.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.124+0000 7fe4fca32700 1 -- 172.21.15.7:0/3025535078 wait complete. 2023-03-03T15:16:10.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.125+0000 7fe4fca32700 1 Processor -- start 2023-03-03T15:16:10.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.125+0000 7fe4fca32700 1 -- start start 2023-03-03T15:16:10.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4fca32700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80c00e0 0x7fe4f80c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:10.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4fca32700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f80caa30 0x7fe4f81ed030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:10.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4fca32700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f8154ff0 0x7fe4f81ed570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:10.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4fca32700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f80c19e0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4fca32700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f80c3cf0 con 0x7fe4f80caa30 2023-03-03T15:16:10.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4fca32700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f80ca1d0 con 0x7fe4f80c00e0 2023-03-03T15:16:10.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f8154ff0 0x7fe4f81ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:10.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f8154ff0 0x7fe4f81ed570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52284/0 (socket says 172.21.15.7:52284) 2023-03-03T15:16:10.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f6d9d700 1 -- 172.21.15.7:0/1591136343 learned_addr learned my addr 172.21.15.7:0/1591136343 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:16:10.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f659c700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80c00e0 0x7fe4f80c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:10.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f5d9b700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f80caa30 0x7fe4f81ed030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:10.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f6d9d700 1 -- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80c00e0 msgr2=0x7fe4f80c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:10.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f6d9d700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80c00e0 0x7fe4f80c95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:10.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f6d9d700 1 -- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f80caa30 msgr2=0x7fe4f81ed030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:10.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f6d9d700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f80caa30 0x7fe4f81ed030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:10.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f6d9d700 1 -- 172.21.15.7:0/1591136343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e80094a0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4f6d9d700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f8154ff0 0x7fe4f81ed570 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0001ef0 tx=0x7fe4f000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:16:10.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.126+0000 7fe4e6ffd700 1 -- 172.21.15.7:0/1591136343 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f000aad0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.127+0000 7fe4e6ffd700 1 -- 172.21.15.7:0/1591136343 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f000ac30 con 0x7fe4f8154ff0 2023-03-03T15:16:10.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.127+0000 7fe4fca32700 1 -- 172.21.15.7:0/1591136343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f8154ad0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.127+0000 7fe4e6ffd700 1 -- 172.21.15.7:0/1591136343 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00070d0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.127+0000 7fe4fca32700 1 -- 172.21.15.7:0/1591136343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f81547f0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.128+0000 7fe4fca32700 1 -- 172.21.15.7:0/1591136343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f80bf650 con 0x7fe4f8154ff0 2023-03-03T15:16:10.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.128+0000 7fe4e6ffd700 1 -- 172.21.15.7:0/1591136343 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe4f001a030 con 0x7fe4f8154ff0 2023-03-03T15:16:10.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.128+0000 7fe4e6ffd700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d80682f0 0x7fe4d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:10.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.129+0000 7fe4e6ffd700 1 -- 172.21.15.7:0/1591136343 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe4f0085ef0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.129+0000 7fe4f659c700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d80682f0 0x7fe4d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:10.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.129+0000 7fe4f659c700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d80682f0 0x7fe4d806a7a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fe4e80097c0 tx=0x7fe4e8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:16:10.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.133+0000 7fe4e6ffd700 1 -- 172.21.15.7:0/1591136343 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe4f00521a0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.368+0000 7fe4fca32700 1 -- 172.21.15.7:0/1591136343 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4f8074bb0 con 0x7fe4f8154ff0 2023-03-03T15:16:10.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.369+0000 7fe4e6ffd700 1 -- 172.21.15.7:0/1591136343 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe4f0018090 con 0x7fe4f8154ff0 2023-03-03T15:16:10.371 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:16:10.371 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:16:10.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.370+0000 7fe4fca32700 1 -- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d80682f0 msgr2=0x7fe4d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:10.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.370+0000 7fe4fca32700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d80682f0 0x7fe4d806a7a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fe4e80097c0 tx=0x7fe4e8008030 comp rx=0 tx=0).stop 2023-03-03T15:16:10.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.371+0000 7fe4fca32700 1 -- 172.21.15.7:0/1591136343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f8154ff0 msgr2=0x7fe4f81ed570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:10.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.371+0000 7fe4fca32700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f8154ff0 0x7fe4f81ed570 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0001ef0 tx=0x7fe4f000a060 comp rx=0 tx=0).stop 2023-03-03T15:16:10.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.371+0000 7fe4fca32700 1 -- 172.21.15.7:0/1591136343 shutdown_connections 2023-03-03T15:16:10.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.371+0000 7fe4fca32700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4d80682f0 0x7fe4d806a7a0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:10.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.371+0000 7fe4fca32700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4f8154ff0 0x7fe4f81ed570 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:10.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.371+0000 7fe4fca32700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4f80c00e0 0x7fe4f80c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:10.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.371+0000 7fe4fca32700 1 --2- 172.21.15.7:0/1591136343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4f80caa30 0x7fe4f81ed030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:10.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.371+0000 7fe4fca32700 1 -- 172.21.15.7:0/1591136343 >> 172.21.15.7:0/1591136343 conn(0x7fe4f8066e40 msgr2=0x7fe4f8153bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:16:10.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.371+0000 7fe4fca32700 1 -- 172.21.15.7:0/1591136343 shutdown_connections 2023-03-03T15:16:10.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:10.371+0000 7fe4fca32700 1 -- 172.21.15.7:0/1591136343 wait complete. 2023-03-03T15:16:10.374 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:16:10.384 INFO:tasks.mds_thrash.fs.[a]:waiting for 52 secs before thrashing 2023-03-03T15:16:31.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:31.421 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:16:42.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:16:42.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.321+0000 7f9dce2d5700 1 Processor -- start 2023-03-03T15:16:42.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.321+0000 7f9dce2d5700 1 -- start start 2023-03-03T15:16:42.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dce2d5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9dc814f9d0 0x7f9dc8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:42.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dce2d5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc814fda0 0x7f9dc8071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:42.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dce2d5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc80c2f30 con 0x7f9dc80abe20 2023-03-03T15:16:42.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dce2d5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc80c31d0 con 0x7f9dc814f9d0 2023-03-03T15:16:42.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dce2d5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc81524a0 con 0x7f9dc814fda0 2023-03-03T15:16:42.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dc7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9dc814f9d0 0x7f9dc8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:42.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dc7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9dc814f9d0 0x7f9dc8156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52308/0 (socket says 172.21.15.7:52308) 2023-03-03T15:16:42.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dc7fff700 1 -- 172.21.15.7:0/686356577 learned_addr learned my addr 172.21.15.7:0/686356577 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:16:42.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dc77fe700 1 --2- 172.21.15.7:0/686356577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc814fda0 0x7f9dc8071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:42.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dc7fff700 1 -- 172.21.15.7:0/686356577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc814fda0 msgr2=0x7f9dc8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:16:42.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dc7fff700 1 --2- 172.21.15.7:0/686356577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc814fda0 0x7f9dc8071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:42.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dc7fff700 1 -- 172.21.15.7:0/686356577 >> v1:172.21.15.143:6789/0 conn(0x7f9dc80abe20 legacy=0x7f9dc81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:42.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.322+0000 7f9dc7fff700 1 -- 172.21.15.7:0/686356577 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc80c34b0 con 0x7f9dc814f9d0 2023-03-03T15:16:42.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dc7fff700 1 --2- 172.21.15.7:0/686356577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 0x7f9dc8156430 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0098f0 tx=0x7f9dbc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665c6489de291d7c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:16:42.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dc6ffd700 1 -- 172.21.15.7:0/686356577 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc011830 con 0x7f9dc814f9d0 2023-03-03T15:16:42.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dc6ffd700 1 -- 172.21.15.7:0/686356577 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dbc011990 con 0x7f9dc814f9d0 2023-03-03T15:16:42.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dc6ffd700 1 -- 172.21.15.7:0/686356577 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc011d20 con 0x7f9dc814f9d0 2023-03-03T15:16:42.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dce2d5700 1 -- 172.21.15.7:0/686356577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 msgr2=0x7f9dc8156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:42.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dce2d5700 1 --2- 172.21.15.7:0/686356577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 0x7f9dc8156430 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0098f0 tx=0x7f9dbc009c00 comp rx=0 tx=0).stop 2023-03-03T15:16:42.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dce2d5700 1 -- 172.21.15.7:0/686356577 shutdown_connections 2023-03-03T15:16:42.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dce2d5700 1 --2- 172.21.15.7:0/686356577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 0x7f9dc8156430 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:42.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dce2d5700 1 --2- 172.21.15.7:0/686356577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc814fda0 0x7f9dc8071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:42.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dce2d5700 1 -- 172.21.15.7:0/686356577 >> 172.21.15.7:0/686356577 conn(0x7f9dc80c8900 msgr2=0x7f9dc80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:16:42.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dce2d5700 1 -- 172.21.15.7:0/686356577 shutdown_connections 2023-03-03T15:16:42.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.323+0000 7f9dce2d5700 1 -- 172.21.15.7:0/686356577 wait complete. 2023-03-03T15:16:42.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.324+0000 7f9dce2d5700 1 Processor -- start 2023-03-03T15:16:42.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.325+0000 7f9dce2d5700 1 -- start start 2023-03-03T15:16:42.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.325+0000 7f9dce2d5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc80abe20 0x7f9dc814bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:42.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.325+0000 7f9dce2d5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 0x7f9dc8152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:42.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.325+0000 7f9dce2d5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc814fda0 0x7f9dc806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:42.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.325+0000 7f9dce2d5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc806efb0 con 0x7f9dc814f9d0 2023-03-03T15:16:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.325+0000 7f9dce2d5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc814bcc0 con 0x7f9dc80abe20 2023-03-03T15:16:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.325+0000 7f9dce2d5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc806f4e0 con 0x7f9dc814fda0 2023-03-03T15:16:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 0x7f9dc8152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 0x7f9dc8152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52314/0 (socket says 172.21.15.7:52314) 2023-03-03T15:16:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc77fe700 1 -- 172.21.15.7:0/2037343316 learned_addr learned my addr 172.21.15.7:0/2037343316 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:16:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc7fff700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc80abe20 0x7f9dc814bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:42.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dcc872700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc814fda0 0x7f9dc806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:42.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc77fe700 1 -- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc814fda0 msgr2=0x7f9dc806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:42.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc77fe700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc814fda0 0x7f9dc806c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:42.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc77fe700 1 -- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc80abe20 msgr2=0x7f9dc814bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:42.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc77fe700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc80abe20 0x7f9dc814bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:42.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc77fe700 1 -- 172.21.15.7:0/2037343316 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc009550 con 0x7f9dc814f9d0 2023-03-03T15:16:42.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc77fe700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 0x7f9dc8152b30 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f9db800e980 tx=0x7f9db800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:16:42.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc4ff9700 1 -- 172.21.15.7:0/2037343316 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8009710 con 0x7f9dc814f9d0 2023-03-03T15:16:42.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc4ff9700 1 -- 172.21.15.7:0/2037343316 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db800c850 con 0x7f9dc814f9d0 2023-03-03T15:16:42.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dce2d5700 1 -- 172.21.15.7:0/2037343316 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc80ac350 con 0x7f9dc814f9d0 2023-03-03T15:16:42.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dc4ff9700 1 -- 172.21.15.7:0/2037343316 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db800cc00 con 0x7f9dc814f9d0 2023-03-03T15:16:42.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.326+0000 7f9dce2d5700 1 -- 172.21.15.7:0/2037343316 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc80ab630 con 0x7f9dc814f9d0 2023-03-03T15:16:42.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.327+0000 7f9dce2d5700 1 -- 172.21.15.7:0/2037343316 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc80c3e10 con 0x7f9dc814f9d0 2023-03-03T15:16:42.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.328+0000 7f9dc4ff9700 1 -- 172.21.15.7:0/2037343316 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9db8019030 con 0x7f9dc814f9d0 2023-03-03T15:16:42.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.328+0000 7f9dc4ff9700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9db40682f0 0x7f9db406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:16:42.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.328+0000 7f9dc4ff9700 1 -- 172.21.15.7:0/2037343316 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9db8086190 con 0x7f9dc814f9d0 2023-03-03T15:16:42.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.328+0000 7f9dc7fff700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9db40682f0 0x7f9db406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:16:42.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.329+0000 7f9dc7fff700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9db40682f0 0x7f9db406a7a0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009c40 tx=0x7f9dbc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:16:42.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.333+0000 7f9dc4ff9700 1 -- 172.21.15.7:0/2037343316 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9db8052570 con 0x7f9dc814f9d0 2023-03-03T15:16:42.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.569+0000 7f9dce2d5700 1 -- 172.21.15.7:0/2037343316 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dc8074bb0 con 0x7f9dc814f9d0 2023-03-03T15:16:42.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.569+0000 7f9dc4ff9700 1 -- 172.21.15.7:0/2037343316 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9db8055b90 con 0x7f9dc814f9d0 2023-03-03T15:16:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:16:42.572 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:16:42.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.571+0000 7f9dce2d5700 1 -- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9db40682f0 msgr2=0x7f9db406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:42.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.571+0000 7f9dce2d5700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9db40682f0 0x7f9db406a7a0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009c40 tx=0x7f9dbc008030 comp rx=0 tx=0).stop 2023-03-03T15:16:42.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.571+0000 7f9dce2d5700 1 -- 172.21.15.7:0/2037343316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 msgr2=0x7f9dc8152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:16:42.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.571+0000 7f9dce2d5700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 0x7f9dc8152b30 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f9db800e980 tx=0x7f9db800ed40 comp rx=0 tx=0).stop 2023-03-03T15:16:42.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.571+0000 7f9dce2d5700 1 -- 172.21.15.7:0/2037343316 shutdown_connections 2023-03-03T15:16:42.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.571+0000 7f9dce2d5700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9dc814fda0 0x7f9dc806c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:42.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.571+0000 7f9dce2d5700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9dc80abe20 0x7f9dc814bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:42.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.571+0000 7f9dce2d5700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9db40682f0 0x7f9db406a7a0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:42.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.571+0000 7f9dce2d5700 1 --2- 172.21.15.7:0/2037343316 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc814f9d0 0x7f9dc8152b30 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:16:42.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.571+0000 7f9dce2d5700 1 -- 172.21.15.7:0/2037343316 >> 172.21.15.7:0/2037343316 conn(0x7f9dc80c8900 msgr2=0x7f9dc8150c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:16:42.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.572+0000 7f9dce2d5700 1 -- 172.21.15.7:0/2037343316 shutdown_connections 2023-03-03T15:16:42.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:16:42.572+0000 7f9dce2d5700 1 -- 172.21.15.7:0/2037343316 wait complete. 2023-03-03T15:16:42.575 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:16:42.585 INFO:tasks.mds_thrash.fs.[b]:waiting for 118 secs before thrashing 2023-03-03T15:17:01.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:01.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:02.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:17:02.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f854b26c700 1 Processor -- start 2023-03-03T15:17:02.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f854b26c700 1 -- start start 2023-03-03T15:17:02.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f854b26c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f854406c2f0 0x7f85441584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:17:02.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f854b26c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854406d780 con 0x7f8544151ab0 2023-03-03T15:17:02.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f854b26c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854406c770 con 0x7f854406bea0 2023-03-03T15:17:02.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f854b26c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8544158a10 con 0x7f854406c2f0 2023-03-03T15:17:02.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f8549008700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8544151ab0 0x7f85440736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46320/0 (socket says 172.21.15.7:46320) 2023-03-03T15:17:02.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f8549809700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f854406c2f0 0x7f85441584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:17:02.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f8549008700 1 -- 172.21.15.7:0/3869612018 learned_addr learned my addr 172.21.15.7:0/3869612018 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:17:02.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f8549809700 1 -- 172.21.15.7:0/3869612018 >> v1:172.21.15.143:6789/0 conn(0x7f854406bea0 legacy=0x7f8544157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:17:02.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f8549809700 1 -- 172.21.15.7:0/3869612018 >> v1:172.21.15.7:6789/0 conn(0x7f8544151ab0 legacy=0x7f85440736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:17:02.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f853bfff700 1 -- 172.21.15.7:0/3869612018 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783643775 0 0) 0x7f854406d780 con 0x7f8544151ab0 2023-03-03T15:17:02.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.523+0000 7f8549809700 1 -- 172.21.15.7:0/3869612018 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854406cb70 con 0x7f854406c2f0 2023-03-03T15:17:02.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f8549809700 1 --2- 172.21.15.7:0/3869612018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f854406c2f0 0x7f85441584d0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f853c009dc0 tx=0x7f853c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23722172a61bea41 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:17:02.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f853bfff700 1 -- 172.21.15.7:0/3869612018 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c00e070 con 0x7f854406c2f0 2023-03-03T15:17:02.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f853bfff700 1 -- 172.21.15.7:0/3869612018 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c00bb50 con 0x7f854406c2f0 2023-03-03T15:17:02.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f853bfff700 1 -- 172.21.15.7:0/3869612018 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c003a20 con 0x7f854406c2f0 2023-03-03T15:17:02.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f854b26c700 1 -- 172.21.15.7:0/3869612018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f854406c2f0 msgr2=0x7f85441584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:17:02.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f854b26c700 1 --2- 172.21.15.7:0/3869612018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f854406c2f0 0x7f85441584d0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f853c009dc0 tx=0x7f853c00b330 comp rx=0 tx=0).stop 2023-03-03T15:17:02.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f854b26c700 1 -- 172.21.15.7:0/3869612018 shutdown_connections 2023-03-03T15:17:02.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f854b26c700 1 --2- 172.21.15.7:0/3869612018 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f854406c2f0 0x7f85441584d0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:17:02.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f854b26c700 1 -- 172.21.15.7:0/3869612018 >> 172.21.15.7:0/3869612018 conn(0x7f85440a94d0 msgr2=0x7f85440a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:17:02.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f854b26c700 1 -- 172.21.15.7:0/3869612018 shutdown_connections 2023-03-03T15:17:02.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.524+0000 7f854b26c700 1 -- 172.21.15.7:0/3869612018 wait complete. 2023-03-03T15:17:02.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f854b26c700 1 Processor -- start 2023-03-03T15:17:02.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f854b26c700 1 -- start start 2023-03-03T15:17:02.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f854b26c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854406bea0 0x7f85441ee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:17:02.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f854b26c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f854406c2f0 0x7f8544156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:17:02.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f854b26c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544151ab0 0x7f85441eaa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:17:02.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f854b26c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85441ea390 con 0x7f8544151ab0 2023-03-03T15:17:02.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f854b26c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85441e1940 con 0x7f854406bea0 2023-03-03T15:17:02.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f854b26c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85440ac850 con 0x7f854406c2f0 2023-03-03T15:17:02.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f8549809700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544151ab0 0x7f85441eaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:17:02.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f8549809700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544151ab0 0x7f85441eaa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52350/0 (socket says 172.21.15.7:52350) 2023-03-03T15:17:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.526+0000 7f8549809700 1 -- 172.21.15.7:0/1258039178 learned_addr learned my addr 172.21.15.7:0/1258039178 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:17:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8548807700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f854406c2f0 0x7f8544156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:17:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8549008700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854406bea0 0x7f85441ee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:17:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8549809700 1 -- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f854406c2f0 msgr2=0x7f8544156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:17:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8549809700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f854406c2f0 0x7f8544156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:17:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8549809700 1 -- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854406bea0 msgr2=0x7f85441ee2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:17:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8549809700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854406bea0 0x7f85441ee2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:17:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8549809700 1 -- 172.21.15.7:0/1258039178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c009a70 con 0x7f8544151ab0 2023-03-03T15:17:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8549809700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544151ab0 0x7f85441eaa80 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f853c00b370 tx=0x7f853c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:17:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8539ffb700 1 -- 172.21.15.7:0/1258039178 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c012630 con 0x7f8544151ab0 2023-03-03T15:17:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8539ffb700 1 -- 172.21.15.7:0/1258039178 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c0127b0 con 0x7f8544151ab0 2023-03-03T15:17:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f854b26c700 1 -- 172.21.15.7:0/1258039178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854406d130 con 0x7f8544151ab0 2023-03-03T15:17:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f8539ffb700 1 -- 172.21.15.7:0/1258039178 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c012b60 con 0x7f8544151ab0 2023-03-03T15:17:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.527+0000 7f854b26c700 1 -- 172.21.15.7:0/1258039178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854406ce50 con 0x7f8544151ab0 2023-03-03T15:17:02.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.529+0000 7f854b26c700 1 -- 172.21.15.7:0/1258039178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854406d410 con 0x7f8544151ab0 2023-03-03T15:17:02.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.529+0000 7f8539ffb700 1 -- 172.21.15.7:0/1258039178 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f853c012cc0 con 0x7f8544151ab0 2023-03-03T15:17:02.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.529+0000 7f8539ffb700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85280682f0 0x7f852806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:17:02.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.529+0000 7f8539ffb700 1 -- 172.21.15.7:0/1258039178 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f853c088990 con 0x7f8544151ab0 2023-03-03T15:17:02.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.530+0000 7f8549008700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85280682f0 0x7f852806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:17:02.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.530+0000 7f8549008700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85280682f0 0x7f852806a7a0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f8534000f60 tx=0x7f8534007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:17:02.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.534+0000 7f8539ffb700 1 -- 172.21.15.7:0/1258039178 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f853c054d30 con 0x7f8544151ab0 2023-03-03T15:17:02.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.772+0000 7f854b26c700 1 -- 172.21.15.7:0/1258039178 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85441ee800 con 0x7f8544151ab0 2023-03-03T15:17:02.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.772+0000 7f8539ffb700 1 -- 172.21.15.7:0/1258039178 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f853c058350 con 0x7f8544151ab0 2023-03-03T15:17:02.774 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:17:02.774 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:17:02.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 -- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85280682f0 msgr2=0x7f852806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:17:02.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85280682f0 0x7f852806a7a0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f8534000f60 tx=0x7f8534007c90 comp rx=0 tx=0).stop 2023-03-03T15:17:02.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 -- 172.21.15.7:0/1258039178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544151ab0 msgr2=0x7f85441eaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:17:02.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544151ab0 0x7f85441eaa80 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f853c00b370 tx=0x7f853c004ad0 comp rx=0 tx=0).stop 2023-03-03T15:17:02.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 -- 172.21.15.7:0/1258039178 shutdown_connections 2023-03-03T15:17:02.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f85280682f0 0x7f852806a7a0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:17:02.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544151ab0 0x7f85441eaa80 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:17:02.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f854406bea0 0x7f85441ee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:17:02.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 --2- 172.21.15.7:0/1258039178 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f854406c2f0 0x7f8544156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:17:02.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 -- 172.21.15.7:0/1258039178 >> 172.21.15.7:0/1258039178 conn(0x7f85440a94d0 msgr2=0x7f8544073cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:17:02.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 -- 172.21.15.7:0/1258039178 shutdown_connections 2023-03-03T15:17:02.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:17:02.774+0000 7f854b26c700 1 -- 172.21.15.7:0/1258039178 wait complete. 2023-03-03T15:17:02.778 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:17:02.788 INFO:tasks.mds_thrash.fs.[a]:waiting for 103 secs before thrashing 2023-03-03T15:17:31.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:17:31.728 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:01.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:01.876 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:31.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:31.941 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:18:40.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:18:40.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.724+0000 7f4e2e0d0700 1 Processor -- start 2023-03-03T15:18:40.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.724+0000 7f4e2e0d0700 1 -- start start 2023-03-03T15:18:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e2e0d0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e280c2930 0x7f4e28157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e2e0d0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e2806c480 0x7f4e28157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e2e0d0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2806d430 con 0x7f4e28151d40 2023-03-03T15:18:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e2e0d0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2806c2a0 con 0x7f4e280c2930 2023-03-03T15:18:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e2e0d0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2806cb00 con 0x7f4e2806c480 2023-03-03T15:18:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e26ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e280c2930 0x7f4e28157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:40.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e277fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4e28151d40 0x7f4e280736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46398/0 (socket says 172.21.15.7:46398) 2023-03-03T15:18:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e277fe700 1 -- 172.21.15.7:0/538079496 learned_addr learned my addr 172.21.15.7:0/538079496 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:18:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e27fff700 1 --2- 172.21.15.7:0/538079496 >> v2:172.21.15.143:3300/0 conn(0x7f4e2806c480 0x7f4e28157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e26ffd700 1 -- 172.21.15.7:0/538079496 >> v2:172.21.15.143:3300/0 conn(0x7f4e2806c480 msgr2=0x7f4e28157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:18:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e26ffd700 1 --2- 172.21.15.7:0/538079496 >> v2:172.21.15.143:3300/0 conn(0x7f4e2806c480 0x7f4e28157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e26ffd700 1 -- 172.21.15.7:0/538079496 >> v1:172.21.15.7:6789/0 conn(0x7f4e28151d40 legacy=0x7f4e280736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e267fc700 1 -- 172.21.15.7:0/538079496 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511724063 0 0) 0x7f4e2806d430 con 0x7f4e28151d40 2023-03-03T15:18:40.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e26ffd700 1 -- 172.21.15.7:0/538079496 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2806cde0 con 0x7f4e280c2930 2023-03-03T15:18:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e26ffd700 1 --2- 172.21.15.7:0/538079496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e280c2930 0x7f4e28157a90 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009840 tx=0x7f4e1c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd7fb74fc1e4d4e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e267fc700 1 -- 172.21.15.7:0/538079496 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c011830 con 0x7f4e280c2930 2023-03-03T15:18:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.725+0000 7f4e267fc700 1 -- 172.21.15.7:0/538079496 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1c011990 con 0x7f4e280c2930 2023-03-03T15:18:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.726+0000 7f4e267fc700 1 -- 172.21.15.7:0/538079496 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c011d20 con 0x7f4e280c2930 2023-03-03T15:18:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.726+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/538079496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e280c2930 msgr2=0x7f4e28157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.726+0000 7f4e2e0d0700 1 --2- 172.21.15.7:0/538079496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e280c2930 0x7f4e28157a90 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009840 tx=0x7f4e1c009b50 comp rx=0 tx=0).stop 2023-03-03T15:18:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.726+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/538079496 shutdown_connections 2023-03-03T15:18:40.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.726+0000 7f4e2e0d0700 1 --2- 172.21.15.7:0/538079496 >> v2:172.21.15.143:3300/0 conn(0x7f4e2806c480 0x7f4e28157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.726+0000 7f4e2e0d0700 1 --2- 172.21.15.7:0/538079496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e280c2930 0x7f4e28157a90 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.726+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/538079496 >> 172.21.15.7:0/538079496 conn(0x7f4e280a94d0 msgr2=0x7f4e280a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:18:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.726+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/538079496 shutdown_connections 2023-03-03T15:18:40.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.726+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/538079496 wait complete. 2023-03-03T15:18:40.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.727+0000 7f4e2e0d0700 1 Processor -- start 2023-03-03T15:18:40.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e2e0d0700 1 -- start start 2023-03-03T15:18:40.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e2e0d0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2806c480 0x7f4e280bee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:40.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e2e0d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e280c2930 0x7f4e280c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:40.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e2e0d0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e28151d40 0x7f4e280c0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:40.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e2e0d0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e281e9ec0 con 0x7f4e2806c480 2023-03-03T15:18:40.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e2e0d0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280be430 con 0x7f4e280c2930 2023-03-03T15:18:40.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e2e0d0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280be9b0 con 0x7f4e28151d40 2023-03-03T15:18:40.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e277fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2806c480 0x7f4e280bee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:40.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e277fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2806c480 0x7f4e280bee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52424/0 (socket says 172.21.15.7:52424) 2023-03-03T15:18:40.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e277fe700 1 -- 172.21.15.7:0/876150054 learned_addr learned my addr 172.21.15.7:0/876150054 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:18:40.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e27fff700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e28151d40 0x7f4e280c0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:40.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.728+0000 7f4e26ffd700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e280c2930 0x7f4e280c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:40.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e277fe700 1 -- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e28151d40 msgr2=0x7f4e280c0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:40.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e277fe700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e28151d40 0x7f4e280c0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:40.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e277fe700 1 -- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e280c2930 msgr2=0x7f4e280c00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:40.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e277fe700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e280c2930 0x7f4e280c00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:40.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e277fe700 1 -- 172.21.15.7:0/876150054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c0094a0 con 0x7f4e2806c480 2023-03-03T15:18:40.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e277fe700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2806c480 0x7f4e280bee40 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f4e180011e0 tx=0x7f4e1800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:40.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e0ffff700 1 -- 172.21.15.7:0/876150054 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18004c80 con 0x7f4e2806c480 2023-03-03T15:18:40.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e0ffff700 1 -- 172.21.15.7:0/876150054 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e18004e00 con 0x7f4e2806c480 2023-03-03T15:18:40.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/876150054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e28073e00 con 0x7f4e2806c480 2023-03-03T15:18:40.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e0ffff700 1 -- 172.21.15.7:0/876150054 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18007310 con 0x7f4e2806c480 2023-03-03T15:18:40.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.729+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/876150054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2806d0c0 con 0x7f4e2806c480 2023-03-03T15:18:40.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.730+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/876150054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2806bea0 con 0x7f4e2806c480 2023-03-03T15:18:40.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.731+0000 7f4e0ffff700 1 -- 172.21.15.7:0/876150054 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4e1801a020 con 0x7f4e2806c480 2023-03-03T15:18:40.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.731+0000 7f4e0ffff700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e140682f0 0x7f4e1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:40.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.731+0000 7f4e0ffff700 1 -- 172.21.15.7:0/876150054 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4e18085c00 con 0x7f4e2806c480 2023-03-03T15:18:40.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.731+0000 7f4e26ffd700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e140682f0 0x7f4e1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:40.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.732+0000 7f4e26ffd700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e140682f0 0x7f4e1406a7a0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0090b0 tx=0x7f4e1c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:40.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.736+0000 7f4e0ffff700 1 -- 172.21.15.7:0/876150054 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4e1808b020 con 0x7f4e2806c480 2023-03-03T15:18:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.975+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/876150054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e28072600 con 0x7f4e2806c480 2023-03-03T15:18:40.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.976+0000 7f4e0ffff700 1 -- 172.21.15.7:0/876150054 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4e18052050 con 0x7f4e2806c480 2023-03-03T15:18:40.978 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:18:40.978 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:18:40.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e140682f0 msgr2=0x7f4e1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:40.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e140682f0 0x7f4e1406a7a0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0090b0 tx=0x7f4e1c008030 comp rx=0 tx=0).stop 2023-03-03T15:18:40.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/876150054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2806c480 msgr2=0x7f4e280bee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:40.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2806c480 0x7f4e280bee40 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f4e180011e0 tx=0x7f4e1800ca30 comp rx=0 tx=0).stop 2023-03-03T15:18:40.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/876150054 shutdown_connections 2023-03-03T15:18:40.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e28151d40 0x7f4e280c0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:40.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e140682f0 0x7f4e1406a7a0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:40.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e2806c480 0x7f4e280bee40 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:40.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 --2- 172.21.15.7:0/876150054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e280c2930 0x7f4e280c00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:40.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/876150054 >> 172.21.15.7:0/876150054 conn(0x7f4e280a94d0 msgr2=0x7f4e28158310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:18:40.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/876150054 shutdown_connections 2023-03-03T15:18:40.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:40.978+0000 7f4e2e0d0700 1 -- 172.21.15.7:0/876150054 wait complete. 2023-03-03T15:18:40.982 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:18:40.992 INFO:tasks.mds_thrash.fs.[b]:waiting for 96 secs before thrashing 2023-03-03T15:18:45.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:18:45.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.926+0000 7f1bc3bc0700 1 Processor -- start 2023-03-03T15:18:45.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.926+0000 7f1bc3bc0700 1 -- start start 2023-03-03T15:18:45.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.927+0000 7f1bc3bc0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1bbc071e80 0x7f1bbc151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:45.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.927+0000 7f1bc3bc0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bbc1510d0 con 0x7f1bbc14e900 2023-03-03T15:18:45.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.927+0000 7f1bc3bc0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bbc0c3740 con 0x7f1bbc072250 2023-03-03T15:18:45.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.927+0000 7f1bc3bc0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bbc070670 con 0x7f1bbc071e80 2023-03-03T15:18:45.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.927+0000 7f1bc195c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1bbc071e80 0x7f1bbc151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:45.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.927+0000 7f1bc215d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1bbc14e900 0x7f1bbc15ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46420/0 (socket says 172.21.15.7:46420) 2023-03-03T15:18:45.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.927+0000 7f1bc215d700 1 -- 172.21.15.7:0/2368809112 learned_addr learned my addr 172.21.15.7:0/2368809112 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:18:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.928+0000 7f1bc095a700 1 -- 172.21.15.7:0/2368809112 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915203483 0 0) 0x7f1bbc1510d0 con 0x7f1bbc14e900 2023-03-03T15:18:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.928+0000 7f1bc095a700 1 -- 172.21.15.7:0/2368809112 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba80035b0 con 0x7f1bbc14e900 2023-03-03T15:18:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.928+0000 7f1bc195c700 1 -- 172.21.15.7:0/2368809112 >> v1:172.21.15.143:6789/0 conn(0x7f1bbc072250 legacy=0x7f1bbc14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.928+0000 7f1bc195c700 1 -- 172.21.15.7:0/2368809112 >> v1:172.21.15.7:6789/0 conn(0x7f1bbc14e900 legacy=0x7f1bbc15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.928+0000 7f1bc195c700 1 -- 172.21.15.7:0/2368809112 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc0c39e0 con 0x7f1bbc071e80 2023-03-03T15:18:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.928+0000 7f1bc195c700 1 --2- 172.21.15.7:0/2368809112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 0x7f1bbc151820 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009840 tx=0x7f1bac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aee9590bf828752 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:45.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.928+0000 7f1bc095a700 1 -- 172.21.15.7:0/2368809112 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac011830 con 0x7f1bbc071e80 2023-03-03T15:18:45.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.928+0000 7f1bc095a700 1 -- 172.21.15.7:0/2368809112 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac011990 con 0x7f1bbc071e80 2023-03-03T15:18:45.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.928+0000 7f1bc095a700 1 -- 172.21.15.7:0/2368809112 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac011d20 con 0x7f1bbc071e80 2023-03-03T15:18:45.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.929+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/2368809112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 msgr2=0x7f1bbc151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:45.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.929+0000 7f1bc3bc0700 1 --2- 172.21.15.7:0/2368809112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 0x7f1bbc151820 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009840 tx=0x7f1bac009b50 comp rx=0 tx=0).stop 2023-03-03T15:18:45.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.929+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/2368809112 shutdown_connections 2023-03-03T15:18:45.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.929+0000 7f1bc3bc0700 1 --2- 172.21.15.7:0/2368809112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 0x7f1bbc151820 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:45.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.929+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/2368809112 >> 172.21.15.7:0/2368809112 conn(0x7f1bbc0a94d0 msgr2=0x7f1bbc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:18:45.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.929+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/2368809112 shutdown_connections 2023-03-03T15:18:45.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.929+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/2368809112 wait complete. 2023-03-03T15:18:45.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.930+0000 7f1bc3bc0700 1 Processor -- start 2023-03-03T15:18:45.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.930+0000 7f1bc3bc0700 1 -- start start 2023-03-03T15:18:45.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc3bc0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 0x7f1bbc14ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:45.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc3bc0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bbc072250 0x7f1bbc0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:45.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc3bc0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bbc14e900 0x7f1bbc1ea070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:45.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc3bc0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc149560 con 0x7f1bbc071e80 2023-03-03T15:18:45.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc3bc0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc0c7170 con 0x7f1bbc072250 2023-03-03T15:18:45.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc3bc0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc1ed5e0 con 0x7f1bbc14e900 2023-03-03T15:18:45.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc195c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 0x7f1bbc14ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:45.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc195c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 0x7f1bbc14ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52442/0 (socket says 172.21.15.7:52442) 2023-03-03T15:18:45.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc195c700 1 -- 172.21.15.7:0/1530760797 learned_addr learned my addr 172.21.15.7:0/1530760797 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:18:45.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc115b700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bbc072250 0x7f1bbc0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:45.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc215d700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bbc14e900 0x7f1bbc1ea070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:45.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc195c700 1 -- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bbc14e900 msgr2=0x7f1bbc1ea070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:45.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc195c700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bbc14e900 0x7f1bbc1ea070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:45.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc195c700 1 -- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bbc072250 msgr2=0x7f1bbc0c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:45.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc195c700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bbc072250 0x7f1bbc0c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:45.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc195c700 1 -- 172.21.15.7:0/1530760797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac0094a0 con 0x7f1bbc071e80 2023-03-03T15:18:45.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc215d700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bbc14e900 0x7f1bbc1ea070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:18:45.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bc195c700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 0x7f1bbc14ba20 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f1bac000c00 tx=0x7f1bac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:45.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bb27fc700 1 -- 172.21.15.7:0/1530760797 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac0038a0 con 0x7f1bbc071e80 2023-03-03T15:18:45.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.931+0000 7f1bb27fc700 1 -- 172.21.15.7:0/1530760797 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac01f070 con 0x7f1bbc071e80 2023-03-03T15:18:45.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.932+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/1530760797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bbc06c9a0 con 0x7f1bbc071e80 2023-03-03T15:18:45.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.932+0000 7f1bb27fc700 1 -- 172.21.15.7:0/1530760797 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac003c00 con 0x7f1bbc071e80 2023-03-03T15:18:45.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.932+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/1530760797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bbc151300 con 0x7f1bbc071e80 2023-03-03T15:18:45.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.932+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/1530760797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bbc06cb30 con 0x7f1bbc071e80 2023-03-03T15:18:45.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.933+0000 7f1bb27fc700 1 -- 172.21.15.7:0/1530760797 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1bac00f050 con 0x7f1bbc071e80 2023-03-03T15:18:45.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.933+0000 7f1bb27fc700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ba00682f0 0x7f1ba006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:18:45.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.933+0000 7f1bb27fc700 1 -- 172.21.15.7:0/1530760797 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1bac021070 con 0x7f1bbc071e80 2023-03-03T15:18:45.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.934+0000 7f1bc115b700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ba00682f0 0x7f1ba006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:18:45.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.934+0000 7f1bc115b700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ba00682f0 0x7f1ba006a7a0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc14c620 tx=0x7f1bb8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:18:45.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:45.938+0000 7f1bb27fc700 1 -- 172.21.15.7:0/1530760797 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1bac053d40 con 0x7f1bbc071e80 2023-03-03T15:18:46.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.177+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/1530760797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bbc074bb0 con 0x7f1bbc071e80 2023-03-03T15:18:46.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.178+0000 7f1bb27fc700 1 -- 172.21.15.7:0/1530760797 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1bac015090 con 0x7f1bbc071e80 2023-03-03T15:18:46.180 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:18:46.180 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:18:46.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ba00682f0 msgr2=0x7f1ba006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:46.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ba00682f0 0x7f1ba006a7a0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc14c620 tx=0x7f1bb8009040 comp rx=0 tx=0).stop 2023-03-03T15:18:46.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/1530760797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 msgr2=0x7f1bbc14ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:18:46.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 0x7f1bbc14ba20 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f1bac000c00 tx=0x7f1bac004750 comp rx=0 tx=0).stop 2023-03-03T15:18:46.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/1530760797 shutdown_connections 2023-03-03T15:18:46.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1bbc14e900 0x7f1bbc1ea070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:46.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ba00682f0 0x7f1ba006a7a0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:46.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bbc071e80 0x7f1bbc14ba20 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:46.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 --2- 172.21.15.7:0/1530760797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bbc072250 0x7f1bbc0c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:18:46.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/1530760797 >> 172.21.15.7:0/1530760797 conn(0x7f1bbc0a94d0 msgr2=0x7f1bbc06ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:18:46.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/1530760797 shutdown_connections 2023-03-03T15:18:46.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:18:46.180+0000 7f1bc3bc0700 1 -- 172.21.15.7:0/1530760797 wait complete. 2023-03-03T15:18:46.183 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:18:46.193 INFO:tasks.mds_thrash.fs.[a]:waiting for 109 secs before thrashing 2023-03-03T15:19:02.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:02.076 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:32.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:19:32.223 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T15:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T15:20:02.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:02.361 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:16.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:20:17.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.128+0000 7fdd04b0c700 1 Processor -- start 2023-03-03T15:20:17.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.128+0000 7fdd04b0c700 1 -- start start 2023-03-03T15:20:17.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdd04b0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdd00071e80 0x7fdd00151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdd04b0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00072250 0x7fdd0014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdd04b0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd001510d0 con 0x7fdd000c1ce0 2023-03-03T15:20:17.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdd04b0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd000c3740 con 0x7fdd00071e80 2023-03-03T15:20:17.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdd04b0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd00070670 con 0x7fdd00072250 2023-03-03T15:20:17.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdcfe59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdd00071e80 0x7fdd00151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdcfdd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00072250 0x7fdd0014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdcfe59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdd00071e80 0x7fdd00151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:48278/0 (socket says 172.21.15.7:48278) 2023-03-03T15:20:17.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdcfdd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00072250 0x7fdd0014e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:60578/0 (socket says 172.21.15.7:60578) 2023-03-03T15:20:17.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdcfe59c700 1 -- 172.21.15.7:0/3119004829 learned_addr learned my addr 172.21.15.7:0/3119004829 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:20:17.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.129+0000 7fdcfed9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fdd000c1ce0 0x7fdd001546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50008/0 (socket says 172.21.15.7:50008) 2023-03-03T15:20:17.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.130+0000 7fdcfdd9b700 1 -- 172.21.15.7:0/3119004829 >> v2:172.21.15.143:3300/0 conn(0x7fdd00071e80 msgr2=0x7fdd00151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:20:17.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.130+0000 7fdcfdd9b700 1 --2- 172.21.15.7:0/3119004829 >> v2:172.21.15.143:3300/0 conn(0x7fdd00071e80 0x7fdd00151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.130+0000 7fdcfdd9b700 1 -- 172.21.15.7:0/3119004829 >> v1:172.21.15.143:6789/0 conn(0x7fdd000c1ce0 legacy=0x7fdd001546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:17.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.130+0000 7fdcfdd9b700 1 -- 172.21.15.7:0/3119004829 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd000c39e0 con 0x7fdd00072250 2023-03-03T15:20:17.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.130+0000 7fdcfe59c700 1 --2- 172.21.15.7:0/3119004829 >> v2:172.21.15.143:3300/0 conn(0x7fdd00071e80 0x7fdd00151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:20:17.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.130+0000 7fdcfd59a700 1 -- 172.21.15.7:0/3119004829 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455165898 0 0) 0x7fdd001510d0 con 0x7fdd000c1ce0 2023-03-03T15:20:17.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.130+0000 7fdcfdd9b700 1 --2- 172.21.15.7:0/3119004829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00072250 0x7fdd0014e1f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009840 tx=0x7fdcf4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9fcc47c7ba043a3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:17.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.130+0000 7fdcfd59a700 1 -- 172.21.15.7:0/3119004829 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4011830 con 0x7fdd00072250 2023-03-03T15:20:17.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.130+0000 7fdcfd59a700 1 -- 172.21.15.7:0/3119004829 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf4011990 con 0x7fdd00072250 2023-03-03T15:20:17.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.130+0000 7fdcfd59a700 1 -- 172.21.15.7:0/3119004829 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4011d20 con 0x7fdd00072250 2023-03-03T15:20:17.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.131+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3119004829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00072250 msgr2=0x7fdd0014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:17.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.131+0000 7fdd04b0c700 1 --2- 172.21.15.7:0/3119004829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00072250 0x7fdd0014e1f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009840 tx=0x7fdcf4009b50 comp rx=0 tx=0).stop 2023-03-03T15:20:17.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.131+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3119004829 shutdown_connections 2023-03-03T15:20:17.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.131+0000 7fdd04b0c700 1 --2- 172.21.15.7:0/3119004829 >> v2:172.21.15.143:3300/0 conn(0x7fdd00071e80 0x7fdd00151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.131+0000 7fdd04b0c700 1 --2- 172.21.15.7:0/3119004829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00072250 0x7fdd0014e1f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.131+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3119004829 >> 172.21.15.7:0/3119004829 conn(0x7fdd000a94d0 msgr2=0x7fdd000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:17.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.131+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3119004829 shutdown_connections 2023-03-03T15:20:17.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.131+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3119004829 wait complete. 2023-03-03T15:20:17.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.132+0000 7fdd04b0c700 1 Processor -- start 2023-03-03T15:20:17.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.132+0000 7fdd04b0c700 1 -- start start 2023-03-03T15:20:17.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdd04b0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00071e80 0x7fdd001ed420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdd04b0c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd00072250 0x7fdd0014a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdd04b0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd000c1ce0 0x7fdd0014ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdd04b0c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd001ee0f0 con 0x7fdd00072250 2023-03-03T15:20:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdd04b0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd001ede10 con 0x7fdd000c1ce0 2023-03-03T15:20:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdd04b0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd001da9f0 con 0x7fdd00071e80 2023-03-03T15:20:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfdd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd00072250 0x7fdd0014a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfdd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd00072250 0x7fdd0014a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52514/0 (socket says 172.21.15.7:52514) 2023-03-03T15:20:17.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfdd9b700 1 -- 172.21.15.7:0/3697941560 learned_addr learned my addr 172.21.15.7:0/3697941560 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:20:17.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfe59c700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00071e80 0x7fdd001ed420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfed9d700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd000c1ce0 0x7fdd0014ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfdd9b700 1 -- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00071e80 msgr2=0x7fdd001ed420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:17.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfdd9b700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00071e80 0x7fdd001ed420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfdd9b700 1 -- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd000c1ce0 msgr2=0x7fdd0014ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:17.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfdd9b700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd000c1ce0 0x7fdd0014ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfdd9b700 1 -- 172.21.15.7:0/3697941560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf000b7f0 con 0x7fdd00072250 2023-03-03T15:20:17.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.133+0000 7fdcfdd9b700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd00072250 0x7fdd0014a8d0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40090b0 tx=0x7fdcf4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:17.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.134+0000 7fdceeffd700 1 -- 172.21.15.7:0/3697941560 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf40036a0 con 0x7fdd00072250 2023-03-03T15:20:17.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.134+0000 7fdceeffd700 1 -- 172.21.15.7:0/3697941560 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf401f070 con 0x7fdd00072250 2023-03-03T15:20:17.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.134+0000 7fdceeffd700 1 -- 172.21.15.7:0/3697941560 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4003a60 con 0x7fdd00072250 2023-03-03T15:20:17.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.134+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3697941560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf40094a0 con 0x7fdd00072250 2023-03-03T15:20:17.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.134+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3697941560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0006c9a0 con 0x7fdd00072250 2023-03-03T15:20:17.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.135+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3697941560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd00151300 con 0x7fdd00072250 2023-03-03T15:20:17.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.135+0000 7fdceeffd700 1 -- 172.21.15.7:0/3697941560 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdcf400f050 con 0x7fdd00072250 2023-03-03T15:20:17.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.136+0000 7fdceeffd700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdce00682f0 0x7fdce006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:17.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.136+0000 7fdceeffd700 1 -- 172.21.15.7:0/3697941560 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdcf4021070 con 0x7fdd00072250 2023-03-03T15:20:17.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.136+0000 7fdcfe59c700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdce00682f0 0x7fdce006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:17.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.136+0000 7fdcfe59c700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdce00682f0 0x7fdce006a7a0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fdcf000be70 tx=0x7fdcf000b4d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:17.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.140+0000 7fdceeffd700 1 -- 172.21.15.7:0/3697941560 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdcf4053db0 con 0x7fdd00072250 2023-03-03T15:20:17.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.378+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3697941560 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd0006cb30 con 0x7fdd00072250 2023-03-03T15:20:17.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.378+0000 7fdceeffd700 1 -- 172.21.15.7:0/3697941560 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdcf40573d0 con 0x7fdd00072250 2023-03-03T15:20:17.381 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:20:17.381 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:20:17.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdce00682f0 msgr2=0x7fdce006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:17.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdce00682f0 0x7fdce006a7a0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fdcf000be70 tx=0x7fdcf000b4d0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3697941560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd00072250 msgr2=0x7fdd0014a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd00072250 0x7fdd0014a8d0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40090b0 tx=0x7fdcf4004750 comp rx=0 tx=0).stop 2023-03-03T15:20:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3697941560 shutdown_connections 2023-03-03T15:20:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd000c1ce0 0x7fdd0014ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdd00071e80 0x7fdd001ed420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdce00682f0 0x7fdce006a7a0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 --2- 172.21.15.7:0/3697941560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd00072250 0x7fdd0014a8d0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:17.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3697941560 >> 172.21.15.7:0/3697941560 conn(0x7fdd000a94d0 msgr2=0x7fdd0006dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:17.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3697941560 shutdown_connections 2023-03-03T15:20:17.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:17.380+0000 7fdd04b0c700 1 -- 172.21.15.7:0/3697941560 wait complete. 2023-03-03T15:20:17.385 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:20:17.395 INFO:tasks.mds_thrash.fs.[b]:waiting for 32 secs before thrashing 2023-03-03T15:20:32.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:32.427 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:20:35.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:20:35.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.332+0000 7f2cad975700 1 Processor -- start 2023-03-03T15:20:35.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.332+0000 7f2cad975700 1 -- start start 2023-03-03T15:20:35.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.333+0000 7f2cad975700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ca814f9d0 0x7f2ca8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:35.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.333+0000 7f2cad975700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca80abe20 0x7f2ca81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:35.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.333+0000 7f2cad975700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca80c2f30 con 0x7f2ca814fda0 2023-03-03T15:20:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.333+0000 7f2cad975700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca80c31d0 con 0x7f2ca814f9d0 2023-03-03T15:20:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.333+0000 7f2cad975700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca81524a0 con 0x7f2ca80abe20 2023-03-03T15:20:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.333+0000 7f2ca6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ca814f9d0 0x7f2ca8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.333+0000 7f2ca6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ca814f9d0 0x7f2ca8156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52526/0 (socket says 172.21.15.7:52526) 2023-03-03T15:20:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.333+0000 7f2ca6ffd700 1 -- 172.21.15.7:0/1266705125 learned_addr learned my addr 172.21.15.7:0/1266705125 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:20:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.334+0000 7f2ca77fe700 1 --2- 172.21.15.7:0/1266705125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca80abe20 0x7f2ca81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:35.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.334+0000 7f2ca6ffd700 1 -- 172.21.15.7:0/1266705125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca80abe20 msgr2=0x7f2ca81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:20:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.334+0000 7f2ca6ffd700 1 --2- 172.21.15.7:0/1266705125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca80abe20 0x7f2ca81529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.334+0000 7f2ca6ffd700 1 -- 172.21.15.7:0/1266705125 >> v1:172.21.15.143:6789/0 conn(0x7f2ca814fda0 legacy=0x7f2ca8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.334+0000 7f2ca6ffd700 1 -- 172.21.15.7:0/1266705125 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca80c34b0 con 0x7f2ca814f9d0 2023-03-03T15:20:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.334+0000 7f2ca6ffd700 1 --2- 172.21.15.7:0/1266705125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814f9d0 0x7f2ca8156430 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f2c980098f0 tx=0x7f2c98009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8edb02415db27e1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.334+0000 7f2ca5ffb700 1 -- 172.21.15.7:0/1266705125 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98011830 con 0x7f2ca814f9d0 2023-03-03T15:20:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.334+0000 7f2ca5ffb700 1 -- 172.21.15.7:0/1266705125 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c98011990 con 0x7f2ca814f9d0 2023-03-03T15:20:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.334+0000 7f2ca5ffb700 1 -- 172.21.15.7:0/1266705125 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98011d20 con 0x7f2ca814f9d0 2023-03-03T15:20:35.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.335+0000 7f2cad975700 1 -- 172.21.15.7:0/1266705125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814f9d0 msgr2=0x7f2ca8156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.335+0000 7f2cad975700 1 --2- 172.21.15.7:0/1266705125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814f9d0 0x7f2ca8156430 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f2c980098f0 tx=0x7f2c98009c00 comp rx=0 tx=0).stop 2023-03-03T15:20:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.335+0000 7f2cad975700 1 -- 172.21.15.7:0/1266705125 shutdown_connections 2023-03-03T15:20:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.335+0000 7f2cad975700 1 --2- 172.21.15.7:0/1266705125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca80abe20 0x7f2ca81529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.335+0000 7f2cad975700 1 --2- 172.21.15.7:0/1266705125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814f9d0 0x7f2ca8156430 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.335+0000 7f2cad975700 1 -- 172.21.15.7:0/1266705125 >> 172.21.15.7:0/1266705125 conn(0x7f2ca80c8900 msgr2=0x7f2ca80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.335+0000 7f2cad975700 1 -- 172.21.15.7:0/1266705125 shutdown_connections 2023-03-03T15:20:35.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.335+0000 7f2cad975700 1 -- 172.21.15.7:0/1266705125 wait complete. 2023-03-03T15:20:35.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.336+0000 7f2cad975700 1 Processor -- start 2023-03-03T15:20:35.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.336+0000 7f2cad975700 1 -- start start 2023-03-03T15:20:35.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2cad975700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ca80abe20 0x7f2ca814b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:35.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2cad975700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca814f9d0 0x7f2ca81dea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:35.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2cad975700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814fda0 0x7f2ca81defd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:35.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2cad975700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca806e4d0 con 0x7f2ca814fda0 2023-03-03T15:20:35.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2cad975700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca806ea50 con 0x7f2ca80abe20 2023-03-03T15:20:35.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2cad975700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca8152150 con 0x7f2ca814f9d0 2023-03-03T15:20:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814fda0 0x7f2ca81defd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814fda0 0x7f2ca81defd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52536/0 (socket says 172.21.15.7:52536) 2023-03-03T15:20:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca77fe700 1 -- 172.21.15.7:0/646987313 learned_addr learned my addr 172.21.15.7:0/646987313 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:20:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca6ffd700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ca80abe20 0x7f2ca814b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca67fc700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca814f9d0 0x7f2ca81dea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca77fe700 1 -- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca814f9d0 msgr2=0x7f2ca81dea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca77fe700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca814f9d0 0x7f2ca81dea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:35.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca77fe700 1 -- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ca80abe20 msgr2=0x7f2ca814b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca77fe700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ca80abe20 0x7f2ca814b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca77fe700 1 -- 172.21.15.7:0/646987313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c98009550 con 0x7f2ca814fda0 2023-03-03T15:20:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca67fc700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca814f9d0 0x7f2ca81dea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:20:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca6ffd700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ca80abe20 0x7f2ca814b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:20:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.337+0000 7f2ca77fe700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814fda0 0x7f2ca81defd0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000efc0 tx=0x7f2ca000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.338+0000 7f2c9ffff700 1 -- 172.21.15.7:0/646987313 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0009b20 con 0x7f2ca814fda0 2023-03-03T15:20:35.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.338+0000 7f2c9ffff700 1 -- 172.21.15.7:0/646987313 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca000f040 con 0x7f2ca814fda0 2023-03-03T15:20:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.338+0000 7f2cad975700 1 -- 172.21.15.7:0/646987313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca80ac350 con 0x7f2ca814fda0 2023-03-03T15:20:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.338+0000 7f2c9ffff700 1 -- 172.21.15.7:0/646987313 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0009e50 con 0x7f2ca814fda0 2023-03-03T15:20:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.338+0000 7f2cad975700 1 -- 172.21.15.7:0/646987313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca80ab630 con 0x7f2ca814fda0 2023-03-03T15:20:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.339+0000 7f2cad975700 1 -- 172.21.15.7:0/646987313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca8071fa0 con 0x7f2ca814fda0 2023-03-03T15:20:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.339+0000 7f2c9ffff700 1 -- 172.21.15.7:0/646987313 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ca0014480 con 0x7f2ca814fda0 2023-03-03T15:20:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.339+0000 7f2c9ffff700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c880682f0 0x7f2c8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.340+0000 7f2c9ffff700 1 -- 172.21.15.7:0/646987313 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2ca0086e50 con 0x7f2ca814fda0 2023-03-03T15:20:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.340+0000 7f2ca6ffd700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c880682f0 0x7f2c8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:35.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.340+0000 7f2ca6ffd700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c880682f0 0x7f2c8806a7a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009840 tx=0x7f2c98008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:35.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.344+0000 7f2c9ffff700 1 -- 172.21.15.7:0/646987313 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ca0053190 con 0x7f2ca814fda0 2023-03-03T15:20:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.583+0000 7f2cad975700 1 -- 172.21.15.7:0/646987313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ca8074bb0 con 0x7f2ca814fda0 2023-03-03T15:20:35.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.584+0000 7f2c9ffff700 1 -- 172.21.15.7:0/646987313 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2ca0018090 con 0x7f2ca814fda0 2023-03-03T15:20:35.586 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:20:35.586 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:20:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 -- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c880682f0 msgr2=0x7f2c8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:35.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c880682f0 0x7f2c8806a7a0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009840 tx=0x7f2c98008030 comp rx=0 tx=0).stop 2023-03-03T15:20:35.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 -- 172.21.15.7:0/646987313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814fda0 msgr2=0x7f2ca81defd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:35.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814fda0 0x7f2ca81defd0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000efc0 tx=0x7f2ca000c3a0 comp rx=0 tx=0).stop 2023-03-03T15:20:35.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 -- 172.21.15.7:0/646987313 shutdown_connections 2023-03-03T15:20:35.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c880682f0 0x7f2c8806a7a0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:35.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ca814fda0 0x7f2ca81defd0 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:35.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ca80abe20 0x7f2ca814b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:35.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 --2- 172.21.15.7:0/646987313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ca814f9d0 0x7f2ca81dea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:35.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 -- 172.21.15.7:0/646987313 >> 172.21.15.7:0/646987313 conn(0x7f2ca80c8900 msgr2=0x7f2ca80c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 -- 172.21.15.7:0/646987313 shutdown_connections 2023-03-03T15:20:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:35.586+0000 7f2cad975700 1 -- 172.21.15.7:0/646987313 wait complete. 2023-03-03T15:20:35.589 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:20:35.599 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before thrashing 2023-03-03T15:20:40.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:20:40.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.737+0000 7fed6b90f700 1 Processor -- start 2023-03-03T15:20:40.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.738+0000 7fed6b90f700 1 -- start start 2023-03-03T15:20:40.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.738+0000 7fed6b90f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fed640c65c0 0x7fed640c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:40.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.738+0000 7fed6b90f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed640a9af0 0x7fed640c2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:40.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.738+0000 7fed6b90f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed640a65b0 con 0x7fed6414de50 2023-03-03T15:20:40.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed6b90f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed64154e00 con 0x7fed640c65c0 2023-03-03T15:20:40.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed6b90f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed641541f0 con 0x7fed640a9af0 2023-03-03T15:20:40.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed696ab700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fed6414de50 0x7fed6406a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46524/0 (socket says 172.21.15.7:46524) 2023-03-03T15:20:40.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed696ab700 1 -- 172.21.15.7:0/1468274348 learned_addr learned my addr 172.21.15.7:0/1468274348 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:20:40.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed68eaa700 1 --2- 172.21.15.7:0/1468274348 >> v2:172.21.15.143:3300/0 conn(0x7fed640c65c0 0x7fed640c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:40.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed69eac700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed640a9af0 0x7fed640c2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024638723 0 0) 0x7fed640a65b0 con 0x7fed6414de50 2023-03-03T15:20:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed500035b0 con 0x7fed6414de50 2023-03-03T15:20:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 770665879 0 0) 0x7fed500035b0 con 0x7fed6414de50 2023-03-03T15:20:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed640a9af0 msgr2=0x7fed640c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:20:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed640a9af0 0x7fed640c2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 >> v2:172.21.15.143:3300/0 conn(0x7fed640c65c0 msgr2=0x7fed640c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:20:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 --2- 172.21.15.7:0/1468274348 >> v2:172.21.15.143:3300/0 conn(0x7fed640c65c0 0x7fed640c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:40.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed641544d0 con 0x7fed6414de50 2023-03-03T15:20:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fed60001670 con 0x7fed6414de50 2023-03-03T15:20:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 >> v1:172.21.15.7:6789/0 conn(0x7fed6414de50 legacy=0x7fed6406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50003a70 0x7fed50003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 0x7fed50004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed640a9af0 0x7fed50005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed64154e00 con 0x7fed50004850 2023-03-03T15:20:40.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed641541f0 con 0x7fed50003a70 2023-03-03T15:20:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.739+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5000ba10 con 0x7fed640a9af0 2023-03-03T15:20:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed68eaa700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 0x7fed50004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed696ab700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50003a70 0x7fed50003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed69eac700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed640a9af0 0x7fed50005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed68eaa700 1 -- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed640a9af0 msgr2=0x7fed50005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:20:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed68eaa700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed640a9af0 0x7fed50005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed68eaa700 1 -- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50003a70 msgr2=0x7fed50003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:20:40.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed68eaa700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed50003a70 0x7fed50003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed68eaa700 1 -- 172.21.15.7:0/1468274348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed500035b0 con 0x7fed50004850 2023-03-03T15:20:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed69eac700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed640a9af0 0x7fed50005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:20:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed68eaa700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 0x7fed50004cc0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fed58009b30 tx=0x7fed50005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b715bfc10de274 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5800e6c0 con 0x7fed50004850 2023-03-03T15:20:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5800e820 con 0x7fed50004850 2023-03-03T15:20:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.740+0000 7fed57fff700 1 -- 172.21.15.7:0/1468274348 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5800eb90 con 0x7fed50004850 2023-03-03T15:20:40.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.741+0000 7fed6b90f700 1 -- 172.21.15.7:0/1468274348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 msgr2=0x7fed50004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.741+0000 7fed6b90f700 1 --2- 172.21.15.7:0/1468274348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 0x7fed50004cc0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fed58009b30 tx=0x7fed50005880 comp rx=0 tx=0).stop 2023-03-03T15:20:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.741+0000 7fed69eac700 1 -- 172.21.15.7:0/1468274348 reap_dead start 2023-03-03T15:20:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.741+0000 7fed6b90f700 1 -- 172.21.15.7:0/1468274348 shutdown_connections 2023-03-03T15:20:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.741+0000 7fed6b90f700 1 -- 172.21.15.7:0/1468274348 >> 172.21.15.7:0/1468274348 conn(0x7fed640a9430 msgr2=0x7fed640c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.741+0000 7fed6b90f700 1 -- 172.21.15.7:0/1468274348 shutdown_connections 2023-03-03T15:20:40.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.741+0000 7fed6b90f700 1 -- 172.21.15.7:0/1468274348 wait complete. 2023-03-03T15:20:40.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.742+0000 7fed6b90f700 1 Processor -- start 2023-03-03T15:20:40.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.742+0000 7fed6b90f700 1 -- start start 2023-03-03T15:20:40.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed6b90f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 0x7fed640c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:40.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed6b90f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed6414a750 0x7fed6414e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:40.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed6b90f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed64149ba0 0x7fed6414a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:40.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed6b90f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed641e79b0 con 0x7fed50004850 2023-03-03T15:20:40.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed6b90f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed641e8980 con 0x7fed64149ba0 2023-03-03T15:20:40.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed6b90f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed640c41b0 con 0x7fed6414a750 2023-03-03T15:20:40.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed696ab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 0x7fed640c0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:40.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed696ab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 0x7fed640c0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52556/0 (socket says 172.21.15.7:52556) 2023-03-03T15:20:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed696ab700 1 -- 172.21.15.7:0/2748847147 learned_addr learned my addr 172.21.15.7:0/2748847147 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:20:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed69eac700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed64149ba0 0x7fed6414a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed68eaa700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed6414a750 0x7fed6414e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed696ab700 1 -- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed6414a750 msgr2=0x7fed6414e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed696ab700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed6414a750 0x7fed6414e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed696ab700 1 -- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed64149ba0 msgr2=0x7fed6414a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:40.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed696ab700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed64149ba0 0x7fed6414a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.743+0000 7fed696ab700 1 -- 172.21.15.7:0/2748847147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed64154a90 con 0x7fed50004850 2023-03-03T15:20:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.744+0000 7fed696ab700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 0x7fed640c0ad0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fed60007340 tx=0x7fed60007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.744+0000 7fed55ffb700 1 -- 172.21.15.7:0/2748847147 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed60007ca0 con 0x7fed50004850 2023-03-03T15:20:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.744+0000 7fed55ffb700 1 -- 172.21.15.7:0/2748847147 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6000f6c0 con 0x7fed50004850 2023-03-03T15:20:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.744+0000 7fed55ffb700 1 -- 172.21.15.7:0/2748847147 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6000fa50 con 0x7fed50004850 2023-03-03T15:20:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.744+0000 7fed6b90f700 1 -- 172.21.15.7:0/2748847147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed641547b0 con 0x7fed50004850 2023-03-03T15:20:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.744+0000 7fed6b90f700 1 -- 172.21.15.7:0/2748847147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed64153e80 con 0x7fed50004850 2023-03-03T15:20:40.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.745+0000 7fed6b90f700 1 -- 172.21.15.7:0/2748847147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed640c1010 con 0x7fed50004850 2023-03-03T15:20:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.745+0000 7fed55ffb700 1 -- 172.21.15.7:0/2748847147 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed6000fbb0 con 0x7fed50004850 2023-03-03T15:20:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.746+0000 7fed55ffb700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed48068320 0x7fed4806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.746+0000 7fed55ffb700 1 -- 172.21.15.7:0/2748847147 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fed600857e0 con 0x7fed50004850 2023-03-03T15:20:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.746+0000 7fed68eaa700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed48068320 0x7fed4806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.747+0000 7fed68eaa700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed48068320 0x7fed4806a7d0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fed580090b0 tx=0x7fed58008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:40.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.750+0000 7fed55ffb700 1 -- 172.21.15.7:0/2748847147 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed60051c30 con 0x7fed50004850 2023-03-03T15:20:40.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.992+0000 7fed6b90f700 1 -- 172.21.15.7:0/2748847147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed64074bb0 con 0x7fed50004850 2023-03-03T15:20:40.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.993+0000 7fed55ffb700 1 -- 172.21.15.7:0/2748847147 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fed60055250 con 0x7fed50004850 2023-03-03T15:20:40.995 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:20:40.995 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:20:40.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 -- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed48068320 msgr2=0x7fed4806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:40.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed48068320 0x7fed4806a7d0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fed580090b0 tx=0x7fed58008030 comp rx=0 tx=0).stop 2023-03-03T15:20:40.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 -- 172.21.15.7:0/2748847147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 msgr2=0x7fed640c0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:40.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 0x7fed640c0ad0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fed60007340 tx=0x7fed60007650 comp rx=0 tx=0).stop 2023-03-03T15:20:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 -- 172.21.15.7:0/2748847147 shutdown_connections 2023-03-03T15:20:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed64149ba0 0x7fed6414a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fed48068320 0x7fed4806a7d0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed50004850 0x7fed640c0ad0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 --2- 172.21.15.7:0/2748847147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed6414a750 0x7fed6414e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 -- 172.21.15.7:0/2748847147 >> 172.21.15.7:0/2748847147 conn(0x7fed640a9430 msgr2=0x7fed640c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:40.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 -- 172.21.15.7:0/2748847147 shutdown_connections 2023-03-03T15:20:40.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:40.995+0000 7fed6b90f700 1 -- 172.21.15.7:0/2748847147 wait complete. 2023-03-03T15:20:40.999 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:20:41.008 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2023-03-03T15:20:49.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:20:49.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.535+0000 7fb708ef8700 1 Processor -- start 2023-03-03T15:20:49.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.535+0000 7fb708ef8700 1 -- start start 2023-03-03T15:20:49.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb708ef8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb70414b0d0 0x7fb704071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:49.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb708ef8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb7040aad20 0x7fb704157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:49.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb708ef8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7040a9c20 con 0x7fb704071d40 2023-03-03T15:20:49.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb708ef8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb704154f10 con 0x7fb7040aad20 2023-03-03T15:20:49.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb708ef8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040ac3c0 con 0x7fb70414b0d0 2023-03-03T15:20:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb701d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb7040aad20 0x7fb704157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb701d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb7040aad20 0x7fb704157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52568/0 (socket says 172.21.15.7:52568) 2023-03-03T15:20:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb701d9b700 1 -- 172.21.15.7:0/99188711 learned_addr learned my addr 172.21.15.7:0/99188711 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:20:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb70259c700 1 --2- 172.21.15.7:0/99188711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb70414b0d0 0x7fb704071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb701d9b700 1 -- 172.21.15.7:0/99188711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb70414b0d0 msgr2=0x7fb704071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:20:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb701d9b700 1 --2- 172.21.15.7:0/99188711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb70414b0d0 0x7fb704071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb701d9b700 1 -- 172.21.15.7:0/99188711 >> v1:172.21.15.143:6789/0 conn(0x7fb704071d40 legacy=0x7fb704158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb701d9b700 1 -- 172.21.15.7:0/99188711 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70406fc70 con 0x7fb7040aad20 2023-03-03T15:20:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb70259c700 1 --2- 172.21.15.7:0/99188711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb70414b0d0 0x7fb704071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:20:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.536+0000 7fb701d9b700 1 --2- 172.21.15.7:0/99188711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7040aad20 0x7fb704157df0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7fb6f80098f0 tx=0x7fb6f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c095ff366d1c27c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb70159a700 1 -- 172.21.15.7:0/99188711 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f800b8a0 con 0x7fb7040aad20 2023-03-03T15:20:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb70159a700 1 -- 172.21.15.7:0/99188711 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f800ba00 con 0x7fb7040aad20 2023-03-03T15:20:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb70159a700 1 -- 172.21.15.7:0/99188711 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f800bd90 con 0x7fb7040aad20 2023-03-03T15:20:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb708ef8700 1 -- 172.21.15.7:0/99188711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7040aad20 msgr2=0x7fb704157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:49.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb708ef8700 1 --2- 172.21.15.7:0/99188711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7040aad20 0x7fb704157df0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7fb6f80098f0 tx=0x7fb6f8009c00 comp rx=0 tx=0).stop 2023-03-03T15:20:49.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb708ef8700 1 -- 172.21.15.7:0/99188711 shutdown_connections 2023-03-03T15:20:49.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb708ef8700 1 --2- 172.21.15.7:0/99188711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb70414b0d0 0x7fb704071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:49.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb708ef8700 1 --2- 172.21.15.7:0/99188711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7040aad20 0x7fb704157df0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:49.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb708ef8700 1 -- 172.21.15.7:0/99188711 >> 172.21.15.7:0/99188711 conn(0x7fb7040c8900 msgr2=0x7fb7040c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:49.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb708ef8700 1 -- 172.21.15.7:0/99188711 shutdown_connections 2023-03-03T15:20:49.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.537+0000 7fb708ef8700 1 -- 172.21.15.7:0/99188711 wait complete. 2023-03-03T15:20:49.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.539+0000 7fb708ef8700 1 Processor -- start 2023-03-03T15:20:49.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.539+0000 7fb708ef8700 1 -- start start 2023-03-03T15:20:49.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.539+0000 7fb708ef8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb704071d40 0x7fb7041ddd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:49.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.539+0000 7fb708ef8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7040aad20 0x7fb7041e9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:49.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.539+0000 7fb708ef8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb70414b0d0 0x7fb7041ea1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:49.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.539+0000 7fb708ef8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041e98a0 con 0x7fb70414b0d0 2023-03-03T15:20:49.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.539+0000 7fb708ef8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041ea720 con 0x7fb7040aad20 2023-03-03T15:20:49.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.539+0000 7fb708ef8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041ed230 con 0x7fb704071d40 2023-03-03T15:20:49.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb702d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb70414b0d0 0x7fb7041ea1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:49.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb702d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb70414b0d0 0x7fb7041ea1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52574/0 (socket says 172.21.15.7:52574) 2023-03-03T15:20:49.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb702d9d700 1 -- 172.21.15.7:0/906354920 learned_addr learned my addr 172.21.15.7:0/906354920 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:20:49.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb70259c700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb704071d40 0x7fb7041ddd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:49.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb701d9b700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7040aad20 0x7fb7041e9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:49.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb702d9d700 1 -- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb704071d40 msgr2=0x7fb7041ddd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:49.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb702d9d700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb704071d40 0x7fb7041ddd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:49.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb702d9d700 1 -- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7040aad20 msgr2=0x7fb7041e9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:49.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb702d9d700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7040aad20 0x7fb7041e9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:49.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb702d9d700 1 -- 172.21.15.7:0/906354920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f8009550 con 0x7fb70414b0d0 2023-03-03T15:20:49.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb70259c700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb704071d40 0x7fb7041ddd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:20:49.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb701d9b700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7040aad20 0x7fb7041e9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:20:49.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb702d9d700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb70414b0d0 0x7fb7041ea1e0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00cb50 tx=0x7fb6fc00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:49.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb6f2ffd700 1 -- 172.21.15.7:0/906354920 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc003c00 con 0x7fb70414b0d0 2023-03-03T15:20:49.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb6f2ffd700 1 -- 172.21.15.7:0/906354920 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc003d60 con 0x7fb70414b0d0 2023-03-03T15:20:49.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb708ef8700 1 -- 172.21.15.7:0/906354920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70406f6b0 con 0x7fb70414b0d0 2023-03-03T15:20:49.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb6f2ffd700 1 -- 172.21.15.7:0/906354920 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc00ab70 con 0x7fb70414b0d0 2023-03-03T15:20:49.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.540+0000 7fb708ef8700 1 -- 172.21.15.7:0/906354920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb704155160 con 0x7fb70414b0d0 2023-03-03T15:20:49.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.541+0000 7fb708ef8700 1 -- 172.21.15.7:0/906354920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70406f990 con 0x7fb70414b0d0 2023-03-03T15:20:49.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.542+0000 7fb6f2ffd700 1 -- 172.21.15.7:0/906354920 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb6fc010050 con 0x7fb70414b0d0 2023-03-03T15:20:49.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.542+0000 7fb6f2ffd700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6e40682f0 0x7fb6e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:20:49.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.542+0000 7fb6f2ffd700 1 -- 172.21.15.7:0/906354920 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb6fc086030 con 0x7fb70414b0d0 2023-03-03T15:20:49.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.542+0000 7fb70259c700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6e40682f0 0x7fb6e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:20:49.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.543+0000 7fb70259c700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6e40682f0 0x7fb6e406a7a0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009a00 tx=0x7fb6f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:20:49.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.547+0000 7fb6f2ffd700 1 -- 172.21.15.7:0/906354920 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb6fc0523d0 con 0x7fb70414b0d0 2023-03-03T15:20:49.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.785+0000 7fb708ef8700 1 -- 172.21.15.7:0/906354920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb704074bb0 con 0x7fb70414b0d0 2023-03-03T15:20:49.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.785+0000 7fb6f2ffd700 1 -- 172.21.15.7:0/906354920 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb6fc0559f0 con 0x7fb70414b0d0 2023-03-03T15:20:49.787 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:20:49.787 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:20:49.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 -- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6e40682f0 msgr2=0x7fb6e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:49.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6e40682f0 0x7fb6e406a7a0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009a00 tx=0x7fb6f4008040 comp rx=0 tx=0).stop 2023-03-03T15:20:49.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 -- 172.21.15.7:0/906354920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb70414b0d0 msgr2=0x7fb7041ea1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:20:49.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb70414b0d0 0x7fb7041ea1e0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00cb50 tx=0x7fb6fc00cf10 comp rx=0 tx=0).stop 2023-03-03T15:20:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 -- 172.21.15.7:0/906354920 shutdown_connections 2023-03-03T15:20:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb6e40682f0 0x7fb6e406a7a0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb70414b0d0 0x7fb7041ea1e0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb704071d40 0x7fb7041ddd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 --2- 172.21.15.7:0/906354920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7040aad20 0x7fb7041e9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:20:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 -- 172.21.15.7:0/906354920 >> 172.21.15.7:0/906354920 conn(0x7fb7040c8900 msgr2=0x7fb704148880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:20:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 -- 172.21.15.7:0/906354920 shutdown_connections 2023-03-03T15:20:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:20:49.787+0000 7fb708ef8700 1 -- 172.21.15.7:0/906354920 wait complete. 2023-03-03T15:20:49.791 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:20:49.801 INFO:tasks.mds_thrash.fs.[b]:waiting for 81 secs before thrashing 2023-03-03T15:21:02.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:02.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:32.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:21:32.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:00.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:22:00.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.147+0000 7f80ae726700 1 Processor -- start 2023-03-03T15:22:00.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.147+0000 7f80ae726700 1 -- start start 2023-03-03T15:22:00.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.148+0000 7f80ae726700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a8071e80 0x7f80a8151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:00.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.148+0000 7f80ae726700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80a8072250 0x7f80a814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.148+0000 7f80ae726700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a81510d0 con 0x7f80a80c1ce0 2023-03-03T15:22:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.148+0000 7f80ae726700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a80c3740 con 0x7f80a8072250 2023-03-03T15:22:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.148+0000 7f80ae726700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8070670 con 0x7f80a8071e80 2023-03-03T15:22:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.148+0000 7f80a77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80a8072250 0x7f80a814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:00.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.148+0000 7f80accc3700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f80a80c1ce0 0x7f80a81546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46602/0 (socket says 172.21.15.7:46602) 2023-03-03T15:22:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.148+0000 7f80accc3700 1 -- 172.21.15.7:0/1421713185 learned_addr learned my addr 172.21.15.7:0/1421713185 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:22:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.148+0000 7f80a7fff700 1 --2- 172.21.15.7:0/1421713185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a8071e80 0x7f80a8151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80a77fe700 1 -- 172.21.15.7:0/1421713185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a8071e80 msgr2=0x7f80a8151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:22:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80a77fe700 1 --2- 172.21.15.7:0/1421713185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a8071e80 0x7f80a8151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80a77fe700 1 -- 172.21.15.7:0/1421713185 >> v1:172.21.15.7:6789/0 conn(0x7f80a80c1ce0 legacy=0x7f80a81546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80a77fe700 1 -- 172.21.15.7:0/1421713185 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a80c39e0 con 0x7f80a8072250 2023-03-03T15:22:00.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80a7fff700 1 --2- 172.21.15.7:0/1421713185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a8071e80 0x7f80a8151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:22:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80a6ffd700 1 -- 172.21.15.7:0/1421713185 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634642809 0 0) 0x7f80a81510d0 con 0x7f80a80c1ce0 2023-03-03T15:22:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80a77fe700 1 --2- 172.21.15.7:0/1421713185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8072250 0x7f80a814e1f0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f8090009840 tx=0x7f8090009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9c2e689ce89615 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80a6ffd700 1 -- 172.21.15.7:0/1421713185 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090011830 con 0x7f80a8072250 2023-03-03T15:22:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80a6ffd700 1 -- 172.21.15.7:0/1421713185 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8090011990 con 0x7f80a8072250 2023-03-03T15:22:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80a6ffd700 1 -- 172.21.15.7:0/1421713185 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090011d20 con 0x7f80a8072250 2023-03-03T15:22:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80ae726700 1 -- 172.21.15.7:0/1421713185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8072250 msgr2=0x7f80a814e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:00.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80ae726700 1 --2- 172.21.15.7:0/1421713185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8072250 0x7f80a814e1f0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f8090009840 tx=0x7f8090009b50 comp rx=0 tx=0).stop 2023-03-03T15:22:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80ae726700 1 -- 172.21.15.7:0/1421713185 shutdown_connections 2023-03-03T15:22:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80ae726700 1 --2- 172.21.15.7:0/1421713185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a8071e80 0x7f80a8151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80ae726700 1 --2- 172.21.15.7:0/1421713185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8072250 0x7f80a814e1f0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.149+0000 7f80ae726700 1 -- 172.21.15.7:0/1421713185 >> 172.21.15.7:0/1421713185 conn(0x7f80a80a94d0 msgr2=0x7f80a80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:22:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.150+0000 7f80ae726700 1 -- 172.21.15.7:0/1421713185 shutdown_connections 2023-03-03T15:22:00.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.150+0000 7f80ae726700 1 -- 172.21.15.7:0/1421713185 wait complete. 2023-03-03T15:22:00.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.151+0000 7f80ae726700 1 Processor -- start 2023-03-03T15:22:00.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.151+0000 7f80ae726700 1 -- start start 2023-03-03T15:22:00.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80ae726700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8071e80 0x7f80a8142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:00.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80ae726700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a8072250 0x7f80a81e7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:00.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80ae726700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c1ce0 0x7f80a81e7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:00.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80ae726700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a81511a0 con 0x7f80a8071e80 2023-03-03T15:22:00.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80ae726700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8150c70 con 0x7f80a8072250 2023-03-03T15:22:00.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80ae726700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a81eb790 con 0x7f80a80c1ce0 2023-03-03T15:22:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8071e80 0x7f80a8142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8071e80 0x7f80a8142700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52624/0 (socket says 172.21.15.7:52624) 2023-03-03T15:22:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a7fff700 1 -- 172.21.15.7:0/3671426604 learned_addr learned my addr 172.21.15.7:0/3671426604 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:22:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80accc3700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c1ce0 0x7f80a81e7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a77fe700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a8072250 0x7f80a81e7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a7fff700 1 -- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c1ce0 msgr2=0x7f80a81e7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:00.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a7fff700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c1ce0 0x7f80a81e7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a7fff700 1 -- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a8072250 msgr2=0x7f80a81e7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a7fff700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a8072250 0x7f80a81e7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a7fff700 1 -- 172.21.15.7:0/3671426604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80900094a0 con 0x7f80a8071e80 2023-03-03T15:22:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a77fe700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a8072250 0x7f80a81e7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:22:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a7fff700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8071e80 0x7f80a8142700 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f809800e810 tx=0x7f809800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a4ff9700 1 -- 172.21.15.7:0/3671426604 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098004230 con 0x7f80a8071e80 2023-03-03T15:22:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a4ff9700 1 -- 172.21.15.7:0/3671426604 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80980043b0 con 0x7f80a8071e80 2023-03-03T15:22:00.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80ae726700 1 -- 172.21.15.7:0/3671426604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a806c9a0 con 0x7f80a8071e80 2023-03-03T15:22:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.152+0000 7f80a4ff9700 1 -- 172.21.15.7:0/3671426604 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80980038e0 con 0x7f80a8071e80 2023-03-03T15:22:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.153+0000 7f80ae726700 1 -- 172.21.15.7:0/3671426604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a8151300 con 0x7f80a8071e80 2023-03-03T15:22:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.153+0000 7f80ae726700 1 -- 172.21.15.7:0/3671426604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a806cb30 con 0x7f80a8071e80 2023-03-03T15:22:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.154+0000 7f80a4ff9700 1 -- 172.21.15.7:0/3671426604 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8098022070 con 0x7f80a8071e80 2023-03-03T15:22:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.154+0000 7f80a4ff9700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80880682f0 0x7f808806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.154+0000 7f80a4ff9700 1 -- 172.21.15.7:0/3671426604 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8098085be0 con 0x7f80a8071e80 2023-03-03T15:22:00.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.155+0000 7f80a77fe700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80880682f0 0x7f808806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.155+0000 7f80a77fe700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80880682f0 0x7f808806a7a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f8090009840 tx=0x7f8090008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:00.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.159+0000 7f80a4ff9700 1 -- 172.21.15.7:0/3671426604 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8098051f80 con 0x7f80a8071e80 2023-03-03T15:22:00.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.395+0000 7f80ae726700 1 -- 172.21.15.7:0/3671426604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80a8074bb0 con 0x7f80a8071e80 2023-03-03T15:22:00.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.396+0000 7f80a4ff9700 1 -- 172.21.15.7:0/3671426604 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f809800f040 con 0x7f80a8071e80 2023-03-03T15:22:00.398 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:22:00.398 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:22:00.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 -- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80880682f0 msgr2=0x7f808806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:00.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80880682f0 0x7f808806a7a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f8090009840 tx=0x7f8090008030 comp rx=0 tx=0).stop 2023-03-03T15:22:00.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 -- 172.21.15.7:0/3671426604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8071e80 msgr2=0x7f80a8142700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:00.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8071e80 0x7f80a8142700 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f809800e810 tx=0x7f809800eb20 comp rx=0 tx=0).stop 2023-03-03T15:22:00.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 -- 172.21.15.7:0/3671426604 shutdown_connections 2023-03-03T15:22:00.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c1ce0 0x7f80a81e7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:00.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80880682f0 0x7f808806a7a0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:00.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a8071e80 0x7f80a8142700 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:00.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 --2- 172.21.15.7:0/3671426604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a8072250 0x7f80a81e7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:00.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 -- 172.21.15.7:0/3671426604 >> 172.21.15.7:0/3671426604 conn(0x7f80a80a94d0 msgr2=0x7f80a80c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:22:00.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 -- 172.21.15.7:0/3671426604 shutdown_connections 2023-03-03T15:22:00.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:00.398+0000 7f80ae726700 1 -- 172.21.15.7:0/3671426604 wait complete. 2023-03-03T15:22:00.401 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:22:00.411 INFO:tasks.mds_thrash.fs.[a]:waiting for 38 secs before thrashing 2023-03-03T15:22:02.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:02.873 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:10.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:22:10.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.940+0000 7eff18dd4700 1 Processor -- start 2023-03-03T15:22:10.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.941+0000 7eff18dd4700 1 -- start start 2023-03-03T15:22:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.941+0000 7eff18dd4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7eff1414f9d0 0x7eff14156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.941+0000 7eff18dd4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff1414fda0 0x7eff14071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff18dd4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff140c2f30 con 0x7eff140abe20 2023-03-03T15:22:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff18dd4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff140c31d0 con 0x7eff1414fda0 2023-03-03T15:22:10.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff18dd4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff141524a0 con 0x7eff1414f9d0 2023-03-03T15:22:10.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff11d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff1414fda0 0x7eff14071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:10.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff11d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff1414fda0 0x7eff14071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52638/0 (socket says 172.21.15.7:52638) 2023-03-03T15:22:10.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff11d9b700 1 -- 172.21.15.7:0/3719278356 learned_addr learned my addr 172.21.15.7:0/3719278356 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:22:10.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff1259c700 1 --2- 172.21.15.7:0/3719278356 >> v2:172.21.15.143:3300/0 conn(0x7eff1414f9d0 0x7eff14156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:10.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff11d9b700 1 -- 172.21.15.7:0/3719278356 >> v2:172.21.15.143:3300/0 conn(0x7eff1414f9d0 msgr2=0x7eff14156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:22:10.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff11d9b700 1 --2- 172.21.15.7:0/3719278356 >> v2:172.21.15.143:3300/0 conn(0x7eff1414f9d0 0x7eff14156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:10.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff11d9b700 1 -- 172.21.15.7:0/3719278356 >> v1:172.21.15.143:6789/0 conn(0x7eff140abe20 legacy=0x7eff141529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:10.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff11d9b700 1 -- 172.21.15.7:0/3719278356 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff140c34b0 con 0x7eff1414fda0 2023-03-03T15:22:10.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff11d9b700 1 --2- 172.21.15.7:0/3719278356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414fda0 0x7eff14071890 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7eff140710a0 tx=0x7efefc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e520f3e498582ab7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:10.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff1159a700 1 -- 172.21.15.7:0/3719278356 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00b870 con 0x7eff1414fda0 2023-03-03T15:22:10.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.942+0000 7eff1159a700 1 -- 172.21.15.7:0/3719278356 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efefc00b9d0 con 0x7eff1414fda0 2023-03-03T15:22:10.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.943+0000 7eff1159a700 1 -- 172.21.15.7:0/3719278356 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00bd60 con 0x7eff1414fda0 2023-03-03T15:22:10.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.943+0000 7eff18dd4700 1 -- 172.21.15.7:0/3719278356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414fda0 msgr2=0x7eff14071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:10.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.943+0000 7eff18dd4700 1 --2- 172.21.15.7:0/3719278356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414fda0 0x7eff14071890 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7eff140710a0 tx=0x7efefc009bd0 comp rx=0 tx=0).stop 2023-03-03T15:22:10.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.943+0000 7eff18dd4700 1 -- 172.21.15.7:0/3719278356 shutdown_connections 2023-03-03T15:22:10.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.943+0000 7eff18dd4700 1 --2- 172.21.15.7:0/3719278356 >> v2:172.21.15.143:3300/0 conn(0x7eff1414f9d0 0x7eff14156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:10.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.943+0000 7eff18dd4700 1 --2- 172.21.15.7:0/3719278356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414fda0 0x7eff14071890 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:10.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.943+0000 7eff18dd4700 1 -- 172.21.15.7:0/3719278356 >> 172.21.15.7:0/3719278356 conn(0x7eff140c8900 msgr2=0x7eff140c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:22:10.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.943+0000 7eff18dd4700 1 -- 172.21.15.7:0/3719278356 shutdown_connections 2023-03-03T15:22:10.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.943+0000 7eff18dd4700 1 -- 172.21.15.7:0/3719278356 wait complete. 2023-03-03T15:22:10.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.944+0000 7eff18dd4700 1 Processor -- start 2023-03-03T15:22:10.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.945+0000 7eff18dd4700 1 -- start start 2023-03-03T15:22:10.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.945+0000 7eff18dd4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff140abe20 0x7eff140c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:10.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.945+0000 7eff18dd4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414f9d0 0x7eff141427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:10.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.945+0000 7eff18dd4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff1414fda0 0x7eff14142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:10.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.945+0000 7eff18dd4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff14121da0 con 0x7eff1414f9d0 2023-03-03T15:22:10.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.945+0000 7eff18dd4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff14121790 con 0x7eff1414fda0 2023-03-03T15:22:10.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.945+0000 7eff18dd4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff14143220 con 0x7eff140abe20 2023-03-03T15:22:10.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.945+0000 7eff11d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414f9d0 0x7eff141427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:10.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.945+0000 7eff11d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414f9d0 0x7eff141427a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52644/0 (socket says 172.21.15.7:52644) 2023-03-03T15:22:10.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.945+0000 7eff11d9b700 1 -- 172.21.15.7:0/3164426496 learned_addr learned my addr 172.21.15.7:0/3164426496 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:22:10.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff1259c700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff140abe20 0x7eff140c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:10.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff12d9d700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff1414fda0 0x7eff14142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:10.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff11d9b700 1 -- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff140abe20 msgr2=0x7eff140c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:10.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff11d9b700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff140abe20 0x7eff140c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:10.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff11d9b700 1 -- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff1414fda0 msgr2=0x7eff14142ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:10.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff11d9b700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff1414fda0 0x7eff14142ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:10.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff11d9b700 1 -- 172.21.15.7:0/3164426496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc009550 con 0x7eff1414f9d0 2023-03-03T15:22:10.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff12d9d700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff1414fda0 0x7eff14142ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:22:10.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff11d9b700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414f9d0 0x7eff141427a0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7efefc000c00 tx=0x7efefc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:10.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff0affd700 1 -- 172.21.15.7:0/3164426496 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc004540 con 0x7eff1414f9d0 2023-03-03T15:22:10.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff0affd700 1 -- 172.21.15.7:0/3164426496 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efefc0036a0 con 0x7eff1414f9d0 2023-03-03T15:22:10.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff18dd4700 1 -- 172.21.15.7:0/3164426496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff140ac350 con 0x7eff1414f9d0 2023-03-03T15:22:10.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff0affd700 1 -- 172.21.15.7:0/3164426496 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc003a10 con 0x7eff1414f9d0 2023-03-03T15:22:10.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.946+0000 7eff18dd4700 1 -- 172.21.15.7:0/3164426496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff140ab630 con 0x7eff1414f9d0 2023-03-03T15:22:10.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.947+0000 7eff18dd4700 1 -- 172.21.15.7:0/3164426496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff140c3e10 con 0x7eff1414f9d0 2023-03-03T15:22:10.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.948+0000 7eff0affd700 1 -- 172.21.15.7:0/3164426496 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efefc00f050 con 0x7eff1414f9d0 2023-03-03T15:22:10.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.948+0000 7eff0affd700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efef40682f0 0x7efef406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:10.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.948+0000 7eff0affd700 1 -- 172.21.15.7:0/3164426496 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efefc0878d0 con 0x7eff1414f9d0 2023-03-03T15:22:10.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.948+0000 7eff1259c700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efef40682f0 0x7efef406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:10.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.949+0000 7eff1259c700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efef40682f0 0x7efef406a7a0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7eff040099d0 tx=0x7eff04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:10.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:10.953+0000 7eff0affd700 1 -- 172.21.15.7:0/3164426496 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efefc053d20 con 0x7eff1414f9d0 2023-03-03T15:22:11.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.190+0000 7eff18dd4700 1 -- 172.21.15.7:0/3164426496 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff1406ae90 con 0x7eff1414f9d0 2023-03-03T15:22:11.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.190+0000 7eff0affd700 1 -- 172.21.15.7:0/3164426496 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7eff1406ae90 con 0x7eff1414f9d0 2023-03-03T15:22:11.192 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:22:11.193 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:22:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.192+0000 7eff18dd4700 1 -- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efef40682f0 msgr2=0x7efef406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.192+0000 7eff18dd4700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efef40682f0 0x7efef406a7a0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7eff040099d0 tx=0x7eff04008040 comp rx=0 tx=0).stop 2023-03-03T15:22:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.192+0000 7eff18dd4700 1 -- 172.21.15.7:0/3164426496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414f9d0 msgr2=0x7eff141427a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.192+0000 7eff18dd4700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414f9d0 0x7eff141427a0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7efefc000c00 tx=0x7efefc004750 comp rx=0 tx=0).stop 2023-03-03T15:22:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.192+0000 7eff18dd4700 1 -- 172.21.15.7:0/3164426496 shutdown_connections 2023-03-03T15:22:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.192+0000 7eff18dd4700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff1414fda0 0x7eff14142ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.192+0000 7eff18dd4700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff140abe20 0x7eff140c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.192+0000 7eff18dd4700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efef40682f0 0x7efef406a7a0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.192+0000 7eff18dd4700 1 --2- 172.21.15.7:0/3164426496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff1414f9d0 0x7eff141427a0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.192+0000 7eff18dd4700 1 -- 172.21.15.7:0/3164426496 >> 172.21.15.7:0/3164426496 conn(0x7eff140c8900 msgr2=0x7eff1414efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:22:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.193+0000 7eff18dd4700 1 -- 172.21.15.7:0/3164426496 shutdown_connections 2023-03-03T15:22:11.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:11.193+0000 7eff18dd4700 1 -- 172.21.15.7:0/3164426496 wait complete. 2023-03-03T15:22:11.196 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:22:11.206 INFO:tasks.mds_thrash.fs.[b]:waiting for 54 secs before thrashing 2023-03-03T15:22:32.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:32.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:22:38.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:22:38.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.550+0000 7f7e9dc33700 1 Processor -- start 2023-03-03T15:22:38.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.550+0000 7f7e9dc33700 1 -- start start 2023-03-03T15:22:38.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.551+0000 7f7e9dc33700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7e9814fda0 0x7f7e98156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:38.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.551+0000 7f7e9dc33700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e980abe20 0x7f7e981529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:38.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.551+0000 7f7e9dc33700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e980c2f30 con 0x7f7e9814f9d0 2023-03-03T15:22:38.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.551+0000 7f7e9dc33700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e980c31d0 con 0x7f7e9814fda0 2023-03-03T15:22:38.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.551+0000 7f7e9dc33700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e981524a0 con 0x7f7e980abe20 2023-03-03T15:22:38.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.551+0000 7f7e977fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7e9814f9d0 0x7f7e98071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46650/0 (socket says 172.21.15.7:46650) 2023-03-03T15:22:38.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.551+0000 7f7e977fe700 1 -- 172.21.15.7:0/2261962590 learned_addr learned my addr 172.21.15.7:0/2261962590 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:22:38.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e97fff700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e980abe20 0x7f7e981529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:38.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e96ffd700 1 --2- 172.21.15.7:0/2261962590 >> v2:172.21.15.143:3300/0 conn(0x7f7e9814fda0 0x7f7e98156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:38.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844192730 0 0) 0x7f7e980c2f30 con 0x7f7e9814f9d0 2023-03-03T15:22:38.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e840035b0 con 0x7f7e9814f9d0 2023-03-03T15:22:38.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2518689508 0 0) 0x7f7e840035b0 con 0x7f7e9814f9d0 2023-03-03T15:22:38.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e980abe20 msgr2=0x7f7e981529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:22:38.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e980abe20 0x7f7e981529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:38.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 >> v2:172.21.15.143:3300/0 conn(0x7f7e9814fda0 msgr2=0x7f7e98156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:22:38.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 --2- 172.21.15.7:0/2261962590 >> v2:172.21.15.143:3300/0 conn(0x7f7e9814fda0 0x7f7e98156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:38.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e980c34b0 con 0x7f7e9814f9d0 2023-03-03T15:22:38.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7e88001670 con 0x7f7e9814f9d0 2023-03-03T15:22:38.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 >> v1:172.21.15.7:6789/0 conn(0x7f7e9814f9d0 legacy=0x7f7e98071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:38.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84003a70 0x7f7e84003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:38.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e84004850 0x7f7e84004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:38.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e980abe20 0x7f7e84005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:38.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e980c31d0 con 0x7f7e84003a70 2023-03-03T15:22:38.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e981524a0 con 0x7f7e84004850 2023-03-03T15:22:38.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8400bb40 con 0x7f7e980abe20 2023-03-03T15:22:38.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e977fe700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84003a70 0x7f7e84003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:38.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e96ffd700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e84004850 0x7f7e84004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:38.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e97fff700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e980abe20 0x7f7e84005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:38.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e977fe700 1 -- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e980abe20 msgr2=0x7f7e84005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:22:38.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e977fe700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e980abe20 0x7f7e84005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:38.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e977fe700 1 -- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e84004850 msgr2=0x7f7e84004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:22:38.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e977fe700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e84004850 0x7f7e84004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:38.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.552+0000 7f7e977fe700 1 -- 172.21.15.7:0/2261962590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e840035b0 con 0x7f7e84003a70 2023-03-03T15:22:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e977fe700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84003a70 0x7f7e84003e80 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800c320 tx=0x7f7e84005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb500d201bf06f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8800ce60 con 0x7f7e84003a70 2023-03-03T15:22:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e88005960 con 0x7f7e84003a70 2023-03-03T15:22:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e967fc700 1 -- 172.21.15.7:0/2261962590 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e88005cf0 con 0x7f7e84003a70 2023-03-03T15:22:38.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2261962590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84003a70 msgr2=0x7f7e84003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:38.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e9dc33700 1 --2- 172.21.15.7:0/2261962590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e84003a70 0x7f7e84003e80 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800c320 tx=0x7f7e84005880 comp rx=0 tx=0).stop 2023-03-03T15:22:38.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e97fff700 1 -- 172.21.15.7:0/2261962590 reap_dead start 2023-03-03T15:22:38.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2261962590 shutdown_connections 2023-03-03T15:22:38.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2261962590 >> 172.21.15.7:0/2261962590 conn(0x7f7e980c8900 msgr2=0x7f7e980c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:22:38.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2261962590 shutdown_connections 2023-03-03T15:22:38.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.553+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2261962590 wait complete. 2023-03-03T15:22:38.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.555+0000 7f7e9dc33700 1 Processor -- start 2023-03-03T15:22:38.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.555+0000 7f7e9dc33700 1 -- start start 2023-03-03T15:22:38.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.555+0000 7f7e9dc33700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e84003a70 0x7f7e981e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:38.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.555+0000 7f7e9dc33700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e981dc0d0 0x7f7e981548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:38.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.555+0000 7f7e9dc33700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e98154e20 0x7f7e9814b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:38.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.555+0000 7f7e9dc33700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e981dcf90 con 0x7f7e981dc0d0 2023-03-03T15:22:38.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.555+0000 7f7e9dc33700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e981eb960 con 0x7f7e98154e20 2023-03-03T15:22:38.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.555+0000 7f7e9dc33700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e981ebc40 con 0x7f7e84003a70 2023-03-03T15:22:38.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e96ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e981dc0d0 0x7f7e981548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e96ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e981dc0d0 0x7f7e981548e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52686/0 (socket says 172.21.15.7:52686) 2023-03-03T15:22:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e96ffd700 1 -- 172.21.15.7:0/2311615633 learned_addr learned my addr 172.21.15.7:0/2311615633 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:22:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e977fe700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e84003a70 0x7f7e981e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e97fff700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e98154e20 0x7f7e9814b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e96ffd700 1 -- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e84003a70 msgr2=0x7f7e981e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e96ffd700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e84003a70 0x7f7e981e8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e96ffd700 1 -- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e98154e20 msgr2=0x7f7e9814b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:38.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e96ffd700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e98154e20 0x7f7e9814b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e96ffd700 1 -- 172.21.15.7:0/2311615633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e980ac350 con 0x7f7e981dc0d0 2023-03-03T15:22:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e977fe700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e84003a70 0x7f7e981e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:22:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e96ffd700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e981dc0d0 0x7f7e981548e0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f7e800119c0 tx=0x7f7e80011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e8ffff700 1 -- 172.21.15.7:0/2311615633 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e80004240 con 0x7f7e981dc0d0 2023-03-03T15:22:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e8ffff700 1 -- 172.21.15.7:0/2311615633 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e800043a0 con 0x7f7e981dc0d0 2023-03-03T15:22:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2311615633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e980ab630 con 0x7f7e981dc0d0 2023-03-03T15:22:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e8ffff700 1 -- 172.21.15.7:0/2311615633 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8000f730 con 0x7f7e981dc0d0 2023-03-03T15:22:38.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.556+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2311615633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e98071fa0 con 0x7f7e981dc0d0 2023-03-03T15:22:38.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.557+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2311615633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e980c7c70 con 0x7f7e981dc0d0 2023-03-03T15:22:38.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.558+0000 7f7e8ffff700 1 -- 172.21.15.7:0/2311615633 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7e80025070 con 0x7f7e981dc0d0 2023-03-03T15:22:38.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.558+0000 7f7e8ffff700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e780682f0 0x7f7e7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:22:38.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.558+0000 7f7e8ffff700 1 -- 172.21.15.7:0/2311615633 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7e80086540 con 0x7f7e981dc0d0 2023-03-03T15:22:38.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.558+0000 7f7e977fe700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e780682f0 0x7f7e7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:22:38.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.559+0000 7f7e977fe700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e780682f0 0x7f7e7806a7a0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f7e98151c50 tx=0x7f7e8800a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:22:38.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.563+0000 7f7e8ffff700 1 -- 172.21.15.7:0/2311615633 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7e800528e0 con 0x7f7e981dc0d0 2023-03-03T15:22:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.804+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2311615633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e98074bb0 con 0x7f7e981dc0d0 2023-03-03T15:22:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.804+0000 7f7e8ffff700 1 -- 172.21.15.7:0/2311615633 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7e80055f00 con 0x7f7e981dc0d0 2023-03-03T15:22:38.806 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:22:38.806 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:22:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e780682f0 msgr2=0x7f7e7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e780682f0 0x7f7e7806a7a0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f7e98151c50 tx=0x7f7e8800a000 comp rx=0 tx=0).stop 2023-03-03T15:22:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2311615633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e981dc0d0 msgr2=0x7f7e981548e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:22:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e981dc0d0 0x7f7e981548e0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f7e800119c0 tx=0x7f7e80011d80 comp rx=0 tx=0).stop 2023-03-03T15:22:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2311615633 shutdown_connections 2023-03-03T15:22:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7e98154e20 0x7f7e9814b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7e84003a70 0x7f7e981e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7e780682f0 0x7f7e7806a7a0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 --2- 172.21.15.7:0/2311615633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e981dc0d0 0x7f7e981548e0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:22:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2311615633 >> 172.21.15.7:0/2311615633 conn(0x7f7e980c8900 msgr2=0x7f7e98151480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:22:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2311615633 shutdown_connections 2023-03-03T15:22:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:22:38.806+0000 7f7e9dc33700 1 -- 172.21.15.7:0/2311615633 wait complete. 2023-03-03T15:22:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:22:38.820 INFO:tasks.mds_thrash.fs.[a]:waiting for 57 secs before thrashing 2023-03-03T15:23:03.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:03.016 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:05.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:23:05.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.345+0000 7ffa52f0e700 1 Processor -- start 2023-03-03T15:23:05.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.345+0000 7ffa52f0e700 1 -- start start 2023-03-03T15:23:05.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.345+0000 7ffa52f0e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffa4c071e80 0x7ffa4c151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:05.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.345+0000 7ffa52f0e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffa4c072250 0x7ffa4c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:05.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.345+0000 7ffa52f0e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c1510d0 con 0x7ffa4c0c1ce0 2023-03-03T15:23:05.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.345+0000 7ffa52f0e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4c0c3740 con 0x7ffa4c072250 2023-03-03T15:23:05.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.345+0000 7ffa52f0e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4c070670 con 0x7ffa4c071e80 2023-03-03T15:23:05.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffa4c072250 0x7ffa4c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:05.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffa4c072250 0x7ffa4c14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52720/0 (socket says 172.21.15.7:52720) 2023-03-03T15:23:05.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4bfff700 1 -- 172.21.15.7:0/1557434462 learned_addr learned my addr 172.21.15.7:0/1557434462 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:23:05.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa50caa700 1 --2- 172.21.15.7:0/1557434462 >> v2:172.21.15.143:3300/0 conn(0x7ffa4c071e80 0x7ffa4c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:05.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4bfff700 1 -- 172.21.15.7:0/1557434462 >> v2:172.21.15.143:3300/0 conn(0x7ffa4c071e80 msgr2=0x7ffa4c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:23:05.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4bfff700 1 --2- 172.21.15.7:0/1557434462 >> v2:172.21.15.143:3300/0 conn(0x7ffa4c071e80 0x7ffa4c151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:05.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4bfff700 1 -- 172.21.15.7:0/1557434462 >> v1:172.21.15.143:6789/0 conn(0x7ffa4c0c1ce0 legacy=0x7ffa4c1546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:05.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4bfff700 1 -- 172.21.15.7:0/1557434462 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c0c39e0 con 0x7ffa4c072250 2023-03-03T15:23:05.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa50caa700 1 --2- 172.21.15.7:0/1557434462 >> v2:172.21.15.143:3300/0 conn(0x7ffa4c071e80 0x7ffa4c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:23:05.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4bfff700 1 --2- 172.21.15.7:0/1557434462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c072250 0x7ffa4c14e1f0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7ffa380098f0 tx=0x7ffa38009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05088766d1f7bec server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:05.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4b7fe700 1 -- 172.21.15.7:0/1557434462 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3800b860 con 0x7ffa4c072250 2023-03-03T15:23:05.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4b7fe700 1 -- 172.21.15.7:0/1557434462 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3800b9c0 con 0x7ffa4c072250 2023-03-03T15:23:05.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.346+0000 7ffa4b7fe700 1 -- 172.21.15.7:0/1557434462 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3800bd50 con 0x7ffa4c072250 2023-03-03T15:23:05.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.347+0000 7ffa52f0e700 1 -- 172.21.15.7:0/1557434462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c072250 msgr2=0x7ffa4c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:05.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.347+0000 7ffa52f0e700 1 --2- 172.21.15.7:0/1557434462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c072250 0x7ffa4c14e1f0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7ffa380098f0 tx=0x7ffa38009c00 comp rx=0 tx=0).stop 2023-03-03T15:23:05.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.347+0000 7ffa52f0e700 1 -- 172.21.15.7:0/1557434462 shutdown_connections 2023-03-03T15:23:05.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.347+0000 7ffa52f0e700 1 --2- 172.21.15.7:0/1557434462 >> v2:172.21.15.143:3300/0 conn(0x7ffa4c071e80 0x7ffa4c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:05.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.347+0000 7ffa52f0e700 1 --2- 172.21.15.7:0/1557434462 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c072250 0x7ffa4c14e1f0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:05.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.347+0000 7ffa52f0e700 1 -- 172.21.15.7:0/1557434462 >> 172.21.15.7:0/1557434462 conn(0x7ffa4c0a94d0 msgr2=0x7ffa4c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:23:05.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.347+0000 7ffa52f0e700 1 -- 172.21.15.7:0/1557434462 shutdown_connections 2023-03-03T15:23:05.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.347+0000 7ffa52f0e700 1 -- 172.21.15.7:0/1557434462 wait complete. 2023-03-03T15:23:05.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.348+0000 7ffa52f0e700 1 Processor -- start 2023-03-03T15:23:05.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.348+0000 7ffa52f0e700 1 -- start start 2023-03-03T15:23:05.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa52f0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c071e80 0x7ffa4c14a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:05.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa52f0e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa4c072250 0x7ffa4c14aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:05.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa52f0e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa4c0c1ce0 0x7ffa4c1ed620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:05.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa52f0e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c14d770 con 0x7ffa4c071e80 2023-03-03T15:23:05.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa52f0e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c14b030 con 0x7ffa4c0c1ce0 2023-03-03T15:23:05.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa52f0e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c15dc00 con 0x7ffa4c072250 2023-03-03T15:23:05.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa50caa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c071e80 0x7ffa4c14a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:05.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa50caa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c071e80 0x7ffa4c14a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52724/0 (socket says 172.21.15.7:52724) 2023-03-03T15:23:05.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa50caa700 1 -- 172.21.15.7:0/3654057054 learned_addr learned my addr 172.21.15.7:0/3654057054 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:23:05.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa4bfff700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa4c072250 0x7ffa4c14aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:05.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa514ab700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa4c0c1ce0 0x7ffa4c1ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:05.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa50caa700 1 -- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa4c072250 msgr2=0x7ffa4c14aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:05.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa50caa700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa4c072250 0x7ffa4c14aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:05.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa50caa700 1 -- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa4c0c1ce0 msgr2=0x7ffa4c1ed620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:05.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa50caa700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa4c0c1ce0 0x7ffa4c1ed620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:05.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.349+0000 7ffa50caa700 1 -- 172.21.15.7:0/3654057054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa38009550 con 0x7ffa4c071e80 2023-03-03T15:23:05.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.350+0000 7ffa50caa700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c071e80 0x7ffa4c14a5b0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000e840 tx=0x7ffa4000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:05.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.350+0000 7ffa497fa700 1 -- 172.21.15.7:0/3654057054 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa40004230 con 0x7ffa4c071e80 2023-03-03T15:23:05.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.350+0000 7ffa497fa700 1 -- 172.21.15.7:0/3654057054 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa40004390 con 0x7ffa4c071e80 2023-03-03T15:23:05.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.350+0000 7ffa52f0e700 1 -- 172.21.15.7:0/3654057054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c06c9a0 con 0x7ffa4c071e80 2023-03-03T15:23:05.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.350+0000 7ffa497fa700 1 -- 172.21.15.7:0/3654057054 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa400036a0 con 0x7ffa4c071e80 2023-03-03T15:23:05.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.350+0000 7ffa52f0e700 1 -- 172.21.15.7:0/3654057054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c151300 con 0x7ffa4c071e80 2023-03-03T15:23:05.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.351+0000 7ffa52f0e700 1 -- 172.21.15.7:0/3654057054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c06cb30 con 0x7ffa4c071e80 2023-03-03T15:23:05.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.351+0000 7ffa497fa700 1 -- 172.21.15.7:0/3654057054 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffa40022070 con 0x7ffa4c071e80 2023-03-03T15:23:05.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.352+0000 7ffa497fa700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa300682f0 0x7ffa3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:05.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.352+0000 7ffa497fa700 1 -- 172.21.15.7:0/3654057054 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ffa40085c30 con 0x7ffa4c071e80 2023-03-03T15:23:05.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.352+0000 7ffa4bfff700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa300682f0 0x7ffa3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:05.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.352+0000 7ffa4bfff700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa300682f0 0x7ffa3006a7a0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0c1780 tx=0x7ffa38008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:05.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.356+0000 7ffa497fa700 1 -- 172.21.15.7:0/3654057054 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffa40051fd0 con 0x7ffa4c071e80 2023-03-03T15:23:05.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.596+0000 7ffa52f0e700 1 -- 172.21.15.7:0/3654057054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa4c0c2a00 con 0x7ffa4c071e80 2023-03-03T15:23:05.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.596+0000 7ffa497fa700 1 -- 172.21.15.7:0/3654057054 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ffa400094c0 con 0x7ffa4c071e80 2023-03-03T15:23:05.598 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:23:05.598 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:23:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 -- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa300682f0 msgr2=0x7ffa3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa300682f0 0x7ffa3006a7a0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0c1780 tx=0x7ffa38008030 comp rx=0 tx=0).stop 2023-03-03T15:23:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 -- 172.21.15.7:0/3654057054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c071e80 msgr2=0x7ffa4c14a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c071e80 0x7ffa4c14a5b0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7ffa4000e840 tx=0x7ffa4000ec00 comp rx=0 tx=0).stop 2023-03-03T15:23:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 -- 172.21.15.7:0/3654057054 shutdown_connections 2023-03-03T15:23:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffa4c0c1ce0 0x7ffa4c1ed620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffa300682f0 0x7ffa3006a7a0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa4c071e80 0x7ffa4c14a5b0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 --2- 172.21.15.7:0/3654057054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffa4c072250 0x7ffa4c14aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 -- 172.21.15.7:0/3654057054 >> 172.21.15.7:0/3654057054 conn(0x7ffa4c0a94d0 msgr2=0x7ffa4c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:23:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 -- 172.21.15.7:0/3654057054 shutdown_connections 2023-03-03T15:23:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:05.598+0000 7ffa52f0e700 1 -- 172.21.15.7:0/3654057054 wait complete. 2023-03-03T15:23:05.602 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:23:05.612 INFO:tasks.mds_thrash.fs.[b]:waiting for 36 secs before thrashing 2023-03-03T15:23:33.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:33.147 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:23:35.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:23:35.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.958+0000 7f9799bc9700 1 Processor -- start 2023-03-03T15:23:35.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.958+0000 7f9799bc9700 1 -- start start 2023-03-03T15:23:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f9799bc9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f979406d4b0 0x7f979406d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f9799bc9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97940c7b90 con 0x7f97940c1da0 2023-03-03T15:23:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f9799bc9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9794154640 con 0x7f97940c8020 2023-03-03T15:23:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f9799bc9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97940c7330 con 0x7f979406d4b0 2023-03-03T15:23:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f9793fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f97940c1da0 0x7f9794139d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46734/0 (socket says 172.21.15.7:46734) 2023-03-03T15:23:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f9793fff700 1 -- 172.21.15.7:0/2838793268 learned_addr learned my addr 172.21.15.7:0/2838793268 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:23:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f97937fe700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f979406d4b0 0x7f979406d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388078788 0 0) 0x7f97940c7b90 con 0x7f97940c1da0 2023-03-03T15:23:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97800035b0 con 0x7f97940c1da0 2023-03-03T15:23:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250808095 0 0) 0x7f9794154640 con 0x7f97940c8020 2023-03-03T15:23:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.959+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97940c7b90 con 0x7f97940c8020 2023-03-03T15:23:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1227331467 0 0) 0x7f97800035b0 con 0x7f97940c1da0 2023-03-03T15:23:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f979406d4b0 msgr2=0x7f979406d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:23:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f979406d4b0 0x7f979406d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 >> v1:172.21.15.143:6789/0 conn(0x7f97940c8020 legacy=0x7f97940c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97940c7540 con 0x7f97940c1da0 2023-03-03T15:23:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f978c0019f0 con 0x7f97940c1da0 2023-03-03T15:23:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 >> v1:172.21.15.7:6789/0 conn(0x7f97940c1da0 legacy=0x7f9794139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9780003c30 0x7f9780004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9780004850 0x7f9780004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9780005200 0x7f978000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97940c7330 con 0x7f9780005200 2023-03-03T15:23:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978000c0e0 con 0x7f9780003c30 2023-03-03T15:23:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978000c2a0 con 0x7f9780004850 2023-03-03T15:23:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97937fe700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f979406d4b0 0x7f979406d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:23:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f9793fff700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9780005200 0x7f978000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f9792ffd700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9780004850 0x7f9780004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97937fe700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9780003c30 0x7f9780004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f9793fff700 1 -- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9780004850 msgr2=0x7f9780004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:23:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f9793fff700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9780004850 0x7f9780004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f9793fff700 1 -- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9780003c30 msgr2=0x7f9780004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:23:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f9793fff700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9780003c30 0x7f9780004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f9793fff700 1 -- 172.21.15.7:0/2838793268 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97800035b0 con 0x7f9780005200 2023-03-03T15:23:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f9793fff700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9780005200 0x7f978000ba10 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f97940c21e0 tx=0x7f9780005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb59eb172b4ecf41 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c0072d0 con 0x7f9780005200 2023-03-03T15:23:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978c007430 con 0x7f9780005200 2023-03-03T15:23:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.960+0000 7f97927fc700 1 -- 172.21.15.7:0/2838793268 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c0077e0 con 0x7f9780005200 2023-03-03T15:23:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.961+0000 7f9799bc9700 1 -- 172.21.15.7:0/2838793268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9780005200 msgr2=0x7f978000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.961+0000 7f9799bc9700 1 --2- 172.21.15.7:0/2838793268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9780005200 0x7f978000ba10 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f97940c21e0 tx=0x7f9780005880 comp rx=0 tx=0).stop 2023-03-03T15:23:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.961+0000 7f9793fff700 1 -- 172.21.15.7:0/2838793268 reap_dead start 2023-03-03T15:23:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.961+0000 7f9799bc9700 1 -- 172.21.15.7:0/2838793268 shutdown_connections 2023-03-03T15:23:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.961+0000 7f9799bc9700 1 -- 172.21.15.7:0/2838793268 >> 172.21.15.7:0/2838793268 conn(0x7f979406c110 msgr2=0x7f979406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:23:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.961+0000 7f9799bc9700 1 -- 172.21.15.7:0/2838793268 shutdown_connections 2023-03-03T15:23:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.961+0000 7f9799bc9700 1 -- 172.21.15.7:0/2838793268 wait complete. 2023-03-03T15:23:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.962+0000 7f9799bc9700 1 Processor -- start 2023-03-03T15:23:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.962+0000 7f9799bc9700 1 -- start start 2023-03-03T15:23:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f9799bc9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97941371f0 0x7f97941ea9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f9799bc9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97941375c0 0x7f97940708c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f9799bc9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97940aa450 0x7f9794070e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f9799bc9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9794138020 con 0x7f97940aa450 2023-03-03T15:23:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f9799bc9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97941eaef0 con 0x7f97941371f0 2023-03-03T15:23:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f9799bc9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9794070100 con 0x7f97941375c0 2023-03-03T15:23:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f9793fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97940aa450 0x7f9794070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f9793fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97940aa450 0x7f9794070e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52768/0 (socket says 172.21.15.7:52768) 2023-03-03T15:23:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f9793fff700 1 -- 172.21.15.7:0/3533502541 learned_addr learned my addr 172.21.15.7:0/3533502541 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:23:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f97937fe700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97941371f0 0x7f97941ea9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.963+0000 7f9792ffd700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97941375c0 0x7f97940708c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f9793fff700 1 -- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97941375c0 msgr2=0x7f97940708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f9793fff700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97941375c0 0x7f97940708c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f9793fff700 1 -- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97941371f0 msgr2=0x7f97941ea9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f9793fff700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97941371f0 0x7f97941ea9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f9793fff700 1 -- 172.21.15.7:0/3533502541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979406e210 con 0x7f97940aa450 2023-03-03T15:23:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f9793fff700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97940aa450 0x7f9794070e00 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f978c001c30 tx=0x7f978c001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f978bfff700 1 -- 172.21.15.7:0/3533502541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c0079b0 con 0x7f97940aa450 2023-03-03T15:23:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f978bfff700 1 -- 172.21.15.7:0/3533502541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978c001a60 con 0x7f97940aa450 2023-03-03T15:23:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f9799bc9700 1 -- 172.21.15.7:0/3533502541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97940c7820 con 0x7f97940aa450 2023-03-03T15:23:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f978bfff700 1 -- 172.21.15.7:0/3533502541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978c01f670 con 0x7f97940aa450 2023-03-03T15:23:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.964+0000 7f9799bc9700 1 -- 172.21.15.7:0/3533502541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97940c0ee0 con 0x7f97940aa450 2023-03-03T15:23:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.965+0000 7f9799bc9700 1 -- 172.21.15.7:0/3533502541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9794132770 con 0x7f97940aa450 2023-03-03T15:23:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.965+0000 7f978bfff700 1 -- 172.21.15.7:0/3533502541 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f978c010070 con 0x7f97940aa450 2023-03-03T15:23:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.966+0000 7f978bfff700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97740682f0 0x7f977406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.966+0000 7f978bfff700 1 -- 172.21.15.7:0/3533502541 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f978c087780 con 0x7f97940aa450 2023-03-03T15:23:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.966+0000 7f97937fe700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97740682f0 0x7f977406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.966+0000 7f97937fe700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97740682f0 0x7f977406a7a0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f97940aa150 tx=0x7f9784008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:35.970+0000 7f978bfff700 1 -- 172.21.15.7:0/3533502541 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f978c053be0 con 0x7f97940aa450 2023-03-03T15:23:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.209+0000 7f9799bc9700 1 -- 172.21.15.7:0/3533502541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9794152c60 con 0x7f97940aa450 2023-03-03T15:23:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.209+0000 7f978bfff700 1 -- 172.21.15.7:0/3533502541 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f978c057200 con 0x7f97940aa450 2023-03-03T15:23:36.211 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:23:36.211 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:23:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 -- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97740682f0 msgr2=0x7f977406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97740682f0 0x7f977406a7a0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f97940aa150 tx=0x7f9784008040 comp rx=0 tx=0).stop 2023-03-03T15:23:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 -- 172.21.15.7:0/3533502541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97940aa450 msgr2=0x7f9794070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97940aa450 0x7f9794070e00 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f978c001c30 tx=0x7f978c001d00 comp rx=0 tx=0).stop 2023-03-03T15:23:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 -- 172.21.15.7:0/3533502541 shutdown_connections 2023-03-03T15:23:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97740682f0 0x7f977406a7a0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97940aa450 0x7f9794070e00 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97941371f0 0x7f97941ea9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 --2- 172.21.15.7:0/3533502541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97941375c0 0x7f97940708c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 -- 172.21.15.7:0/3533502541 >> 172.21.15.7:0/3533502541 conn(0x7f979406c110 msgr2=0x7f97941549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:23:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.211+0000 7f9799bc9700 1 -- 172.21.15.7:0/3533502541 shutdown_connections 2023-03-03T15:23:36.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:36.212+0000 7f9799bc9700 1 -- 172.21.15.7:0/3533502541 wait complete. 2023-03-03T15:23:36.215 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:23:36.225 INFO:tasks.mds_thrash.fs.[a]:waiting for 98 secs before thrashing 2023-03-03T15:23:41.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:23:41.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.751+0000 7fd2ca285700 1 Processor -- start 2023-03-03T15:23:41.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.751+0000 7fd2ca285700 1 -- start start 2023-03-03T15:23:41.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.751+0000 7fd2ca285700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd2c414f9d0 0x7fd2c4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:41.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.751+0000 7fd2ca285700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd2c414fda0 0x7fd2c4071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:41.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.751+0000 7fd2ca285700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40c2f30 con 0x7fd2c40abe20 2023-03-03T15:23:41.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.751+0000 7fd2ca285700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c40c31d0 con 0x7fd2c414f9d0 2023-03-03T15:23:41.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.751+0000 7fd2ca285700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c41524a0 con 0x7fd2c414fda0 2023-03-03T15:23:41.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd2c414f9d0 0x7fd2c4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:41.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c8822700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd2c40abe20 0x7fd2c41529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46754/0 (socket says 172.21.15.7:46754) 2023-03-03T15:23:41.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c8822700 1 -- 172.21.15.7:0/165471000 learned_addr learned my addr 172.21.15.7:0/165471000 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:23:41.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c37fe700 1 --2- 172.21.15.7:0/165471000 >> v2:172.21.15.143:3300/0 conn(0x7fd2c414fda0 0x7fd2c4071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:41.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c2ffd700 1 -- 172.21.15.7:0/165471000 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166984270 0 0) 0x7fd2c40c2f30 con 0x7fd2c40abe20 2023-03-03T15:23:41.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c2ffd700 1 -- 172.21.15.7:0/165471000 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ac0035b0 con 0x7fd2c40abe20 2023-03-03T15:23:41.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c3fff700 1 -- 172.21.15.7:0/165471000 >> v2:172.21.15.143:3300/0 conn(0x7fd2c414fda0 msgr2=0x7fd2c4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:23:41.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c3fff700 1 --2- 172.21.15.7:0/165471000 >> v2:172.21.15.143:3300/0 conn(0x7fd2c414fda0 0x7fd2c4071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:41.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c3fff700 1 -- 172.21.15.7:0/165471000 >> v1:172.21.15.7:6789/0 conn(0x7fd2c40abe20 legacy=0x7fd2c41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:41.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c3fff700 1 -- 172.21.15.7:0/165471000 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40c34b0 con 0x7fd2c414f9d0 2023-03-03T15:23:41.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c3fff700 1 --2- 172.21.15.7:0/165471000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c414f9d0 0x7fd2c4156430 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40709b0 tx=0x7fd2b4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41f850872333808 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:41.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c2ffd700 1 -- 172.21.15.7:0/165471000 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4011830 con 0x7fd2c414f9d0 2023-03-03T15:23:41.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c2ffd700 1 -- 172.21.15.7:0/165471000 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4011990 con 0x7fd2c414f9d0 2023-03-03T15:23:41.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.752+0000 7fd2c2ffd700 1 -- 172.21.15.7:0/165471000 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4011d20 con 0x7fd2c414f9d0 2023-03-03T15:23:41.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.753+0000 7fd2ca285700 1 -- 172.21.15.7:0/165471000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c414f9d0 msgr2=0x7fd2c4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:41.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.753+0000 7fd2ca285700 1 --2- 172.21.15.7:0/165471000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c414f9d0 0x7fd2c4156430 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40709b0 tx=0x7fd2b4009b20 comp rx=0 tx=0).stop 2023-03-03T15:23:41.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.753+0000 7fd2ca285700 1 -- 172.21.15.7:0/165471000 shutdown_connections 2023-03-03T15:23:41.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.753+0000 7fd2ca285700 1 --2- 172.21.15.7:0/165471000 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c414f9d0 0x7fd2c4156430 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:41.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.753+0000 7fd2ca285700 1 --2- 172.21.15.7:0/165471000 >> v2:172.21.15.143:3300/0 conn(0x7fd2c414fda0 0x7fd2c4071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:41.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.753+0000 7fd2ca285700 1 -- 172.21.15.7:0/165471000 >> 172.21.15.7:0/165471000 conn(0x7fd2c40c8900 msgr2=0x7fd2c40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:23:41.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.753+0000 7fd2ca285700 1 -- 172.21.15.7:0/165471000 shutdown_connections 2023-03-03T15:23:41.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.753+0000 7fd2ca285700 1 -- 172.21.15.7:0/165471000 wait complete. 2023-03-03T15:23:41.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.754+0000 7fd2ca285700 1 Processor -- start 2023-03-03T15:23:41.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.754+0000 7fd2ca285700 1 -- start start 2023-03-03T15:23:41.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2ca285700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c40abe20 0x7fd2c41549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:41.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2ca285700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c414f9d0 0x7fd2c414b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:41.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2ca285700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c414fda0 0x7fd2c414b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:41.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2ca285700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c40bda10 con 0x7fd2c40abe20 2023-03-03T15:23:41.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2ca285700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c40c2420 con 0x7fd2c414fda0 2023-03-03T15:23:41.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2ca285700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c40c1ea0 con 0x7fd2c414f9d0 2023-03-03T15:23:41.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c40abe20 0x7fd2c41549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:41.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c40abe20 0x7fd2c41549c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52776/0 (socket says 172.21.15.7:52776) 2023-03-03T15:23:41.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c3fff700 1 -- 172.21.15.7:0/3249849333 learned_addr learned my addr 172.21.15.7:0/3249849333 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:23:41.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c8822700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c414fda0 0x7fd2c414b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:41.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c3fff700 1 -- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c414f9d0 msgr2=0x7fd2c414b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:41.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c37fe700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c414f9d0 0x7fd2c414b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:41.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c3fff700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c414f9d0 0x7fd2c414b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:41.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c3fff700 1 -- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c414fda0 msgr2=0x7fd2c414b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:41.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c3fff700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c414fda0 0x7fd2c414b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:41.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c3fff700 1 -- 172.21.15.7:0/3249849333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b40094a0 con 0x7fd2c40abe20 2023-03-03T15:23:41.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.755+0000 7fd2c8822700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c414fda0 0x7fd2c414b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:23:41.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.756+0000 7fd2c3fff700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c40abe20 0x7fd2c41549c0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009b60 tx=0x7fd2b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:41.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.756+0000 7fd2c0ff9700 1 -- 172.21.15.7:0/3249849333 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4004570 con 0x7fd2c40abe20 2023-03-03T15:23:41.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.756+0000 7fd2c0ff9700 1 -- 172.21.15.7:0/3249849333 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b401f070 con 0x7fd2c40abe20 2023-03-03T15:23:41.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.756+0000 7fd2ca285700 1 -- 172.21.15.7:0/3249849333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c40ac350 con 0x7fd2c40abe20 2023-03-03T15:23:41.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.756+0000 7fd2c0ff9700 1 -- 172.21.15.7:0/3249849333 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b40038c0 con 0x7fd2c40abe20 2023-03-03T15:23:41.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.756+0000 7fd2ca285700 1 -- 172.21.15.7:0/3249849333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c40ab630 con 0x7fd2c40abe20 2023-03-03T15:23:41.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.757+0000 7fd2ca285700 1 -- 172.21.15.7:0/3249849333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c40c3e10 con 0x7fd2c40abe20 2023-03-03T15:23:41.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.758+0000 7fd2c0ff9700 1 -- 172.21.15.7:0/3249849333 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd2b402a020 con 0x7fd2c40abe20 2023-03-03T15:23:41.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.758+0000 7fd2c0ff9700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2b00682f0 0x7fd2b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:23:41.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.758+0000 7fd2c0ff9700 1 -- 172.21.15.7:0/3249849333 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd2b40878a0 con 0x7fd2c40abe20 2023-03-03T15:23:41.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.758+0000 7fd2c37fe700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2b00682f0 0x7fd2b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:23:41.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.759+0000 7fd2c37fe700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2b00682f0 0x7fd2b006a7a0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd2c41eb7e0 tx=0x7fd2b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:23:41.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:41.762+0000 7fd2c0ff9700 1 -- 172.21.15.7:0/3249849333 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd2b4053ce0 con 0x7fd2c40abe20 2023-03-03T15:23:42.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.000+0000 7fd2ca285700 1 -- 172.21.15.7:0/3249849333 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2c4074bb0 con 0x7fd2c40abe20 2023-03-03T15:23:42.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.000+0000 7fd2c0ff9700 1 -- 172.21.15.7:0/3249849333 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd2b4057300 con 0x7fd2c40abe20 2023-03-03T15:23:42.002 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:23:42.003 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:23:42.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.002+0000 7fd2ca285700 1 -- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2b00682f0 msgr2=0x7fd2b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:42.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.002+0000 7fd2ca285700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2b00682f0 0x7fd2b006a7a0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd2c41eb7e0 tx=0x7fd2b8008040 comp rx=0 tx=0).stop 2023-03-03T15:23:42.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.002+0000 7fd2ca285700 1 -- 172.21.15.7:0/3249849333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c40abe20 msgr2=0x7fd2c41549c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.002+0000 7fd2ca285700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c40abe20 0x7fd2c41549c0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009b60 tx=0x7fd2b4004750 comp rx=0 tx=0).stop 2023-03-03T15:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.002+0000 7fd2ca285700 1 -- 172.21.15.7:0/3249849333 shutdown_connections 2023-03-03T15:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.002+0000 7fd2ca285700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c414fda0 0x7fd2c414b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.002+0000 7fd2ca285700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2b00682f0 0x7fd2b006a7a0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.002+0000 7fd2ca285700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c40abe20 0x7fd2c41549c0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.002+0000 7fd2ca285700 1 --2- 172.21.15.7:0/3249849333 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2c414f9d0 0x7fd2c414b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.002+0000 7fd2ca285700 1 -- 172.21.15.7:0/3249849333 >> 172.21.15.7:0/3249849333 conn(0x7fd2c40c8900 msgr2=0x7fd2c40c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.003+0000 7fd2ca285700 1 -- 172.21.15.7:0/3249849333 shutdown_connections 2023-03-03T15:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:23:42.003+0000 7fd2ca285700 1 -- 172.21.15.7:0/3249849333 wait complete. 2023-03-03T15:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:23:42.016 INFO:tasks.mds_thrash.fs.[b]:waiting for 35 secs before thrashing 2023-03-03T15:24:03.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:03.251 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:17.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:24:17.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.154+0000 7f665ee61700 1 Processor -- start 2023-03-03T15:24:17.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.154+0000 7f665ee61700 1 -- start start 2023-03-03T15:24:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.155+0000 7f665ee61700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6658155390 0x7f66580c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:24:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.155+0000 7f665ee61700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6658154910 con 0x7f6658151770 2023-03-03T15:24:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.155+0000 7f665ee61700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66581522e0 con 0x7f665806e020 2023-03-03T15:24:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.155+0000 7f665ee61700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6658070b90 con 0x7f6658155390 2023-03-03T15:24:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.155+0000 7f665cbfd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6658155390 0x7f66580c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:24:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.155+0000 7f665d3fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6658151770 0x7f6658151bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46780/0 (socket says 172.21.15.7:46780) 2023-03-03T15:24:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.155+0000 7f665d3fe700 1 -- 172.21.15.7:0/3858302885 learned_addr learned my addr 172.21.15.7:0/3858302885 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:24:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.156+0000 7f665cbfd700 1 -- 172.21.15.7:0/3858302885 >> v1:172.21.15.143:6789/0 conn(0x7f665806e020 legacy=0x7f6658071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:24:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.156+0000 7f665cbfd700 1 -- 172.21.15.7:0/3858302885 >> v1:172.21.15.7:6789/0 conn(0x7f6658151770 legacy=0x7f6658151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:24:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.156+0000 7f665cbfd700 1 -- 172.21.15.7:0/3858302885 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66580702a0 con 0x7f6658155390 2023-03-03T15:24:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.156+0000 7f66577fe700 1 -- 172.21.15.7:0/3858302885 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274584472 0 0) 0x7f6658154910 con 0x7f6658151770 2023-03-03T15:24:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.156+0000 7f665cbfd700 1 --2- 172.21.15.7:0/3858302885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 0x7f66580c3e30 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f664c009840 tx=0x7f664c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c345ec193cbec6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:24:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.156+0000 7f66577fe700 1 -- 172.21.15.7:0/3858302885 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c011830 con 0x7f6658155390 2023-03-03T15:24:17.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.156+0000 7f66577fe700 1 -- 172.21.15.7:0/3858302885 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c011990 con 0x7f6658155390 2023-03-03T15:24:17.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.156+0000 7f66577fe700 1 -- 172.21.15.7:0/3858302885 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c011d20 con 0x7f6658155390 2023-03-03T15:24:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.157+0000 7f665ee61700 1 -- 172.21.15.7:0/3858302885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 msgr2=0x7f66580c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:24:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.157+0000 7f665ee61700 1 --2- 172.21.15.7:0/3858302885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 0x7f66580c3e30 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f664c009840 tx=0x7f664c009b50 comp rx=0 tx=0).stop 2023-03-03T15:24:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.157+0000 7f665ee61700 1 -- 172.21.15.7:0/3858302885 shutdown_connections 2023-03-03T15:24:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.157+0000 7f665ee61700 1 --2- 172.21.15.7:0/3858302885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 0x7f66580c3e30 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:24:17.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.157+0000 7f665ee61700 1 -- 172.21.15.7:0/3858302885 >> 172.21.15.7:0/3858302885 conn(0x7f665806c110 msgr2=0x7f665806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:24:17.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.157+0000 7f665ee61700 1 -- 172.21.15.7:0/3858302885 shutdown_connections 2023-03-03T15:24:17.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.157+0000 7f665ee61700 1 -- 172.21.15.7:0/3858302885 wait complete. 2023-03-03T15:24:17.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.158+0000 7f665ee61700 1 Processor -- start 2023-03-03T15:24:17.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.158+0000 7f665ee61700 1 -- start start 2023-03-03T15:24:17.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665ee61700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f665806e020 0x7f66580c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:24:17.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665ee61700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6658151770 0x7f665814ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:24:17.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665ee61700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 0x7f665814a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:24:17.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665ee61700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6658149df0 con 0x7f6658155390 2023-03-03T15:24:17.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665ee61700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665814d660 con 0x7f665806e020 2023-03-03T15:24:17.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665ee61700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665814e330 con 0x7f6658151770 2023-03-03T15:24:17.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665d3fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 0x7f665814a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:24:17.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665d3fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 0x7f665814a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52808/0 (socket says 172.21.15.7:52808) 2023-03-03T15:24:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665d3fe700 1 -- 172.21.15.7:0/3272829385 learned_addr learned my addr 172.21.15.7:0/3272829385 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:24:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f6657fff700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6658151770 0x7f665814ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:24:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665cbfd700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f665806e020 0x7f66580c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:24:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665d3fe700 1 -- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6658151770 msgr2=0x7f665814ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:24:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665d3fe700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6658151770 0x7f665814ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:24:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665d3fe700 1 -- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f665806e020 msgr2=0x7f66580c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:24:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665d3fe700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f665806e020 0x7f66580c2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:24:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.159+0000 7f665d3fe700 1 -- 172.21.15.7:0/3272829385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c0094a0 con 0x7f6658155390 2023-03-03T15:24:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.160+0000 7f665d3fe700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 0x7f665814a210 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f665000cc70 tx=0x7f665000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:24:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.160+0000 7f66557fa700 1 -- 172.21.15.7:0/3272829385 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665000aac0 con 0x7f6658155390 2023-03-03T15:24:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.160+0000 7f66557fa700 1 -- 172.21.15.7:0/3272829385 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665000ac40 con 0x7f6658155390 2023-03-03T15:24:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.160+0000 7f665ee61700 1 -- 172.21.15.7:0/3272829385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6658070820 con 0x7f6658155390 2023-03-03T15:24:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.160+0000 7f66557fa700 1 -- 172.21.15.7:0/3272829385 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66500070d0 con 0x7f6658155390 2023-03-03T15:24:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.160+0000 7f665ee61700 1 -- 172.21.15.7:0/3272829385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6658070540 con 0x7f6658155390 2023-03-03T15:24:17.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.161+0000 7f665ee61700 1 -- 172.21.15.7:0/3272829385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6658155760 con 0x7f6658155390 2023-03-03T15:24:17.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.161+0000 7f66557fa700 1 -- 172.21.15.7:0/3272829385 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f665001a030 con 0x7f6658155390 2023-03-03T15:24:17.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.162+0000 7f66557fa700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f663c0682f0 0x7f663c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:24:17.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.162+0000 7f66557fa700 1 -- 172.21.15.7:0/3272829385 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6650085e10 con 0x7f6658155390 2023-03-03T15:24:17.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.162+0000 7f665cbfd700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f663c0682f0 0x7f663c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:24:17.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.163+0000 7f665cbfd700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f663c0682f0 0x7f663c06a7a0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f664c0090b0 tx=0x7f664c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:24:17.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.166+0000 7f66557fa700 1 -- 172.21.15.7:0/3272829385 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6650052150 con 0x7f6658155390 2023-03-03T15:24:17.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.403+0000 7f665ee61700 1 -- 172.21.15.7:0/3272829385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6658074bb0 con 0x7f6658155390 2023-03-03T15:24:17.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.404+0000 7f66557fa700 1 -- 172.21.15.7:0/3272829385 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6650018090 con 0x7f6658155390 2023-03-03T15:24:17.405 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:24:17.406 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:24:17.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 -- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f663c0682f0 msgr2=0x7f663c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:24:17.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f663c0682f0 0x7f663c06a7a0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f664c0090b0 tx=0x7f664c01c040 comp rx=0 tx=0).stop 2023-03-03T15:24:17.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 -- 172.21.15.7:0/3272829385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 msgr2=0x7f665814a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:24:17.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 0x7f665814a210 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f665000cc70 tx=0x7f665000cf80 comp rx=0 tx=0).stop 2023-03-03T15:24:17.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 -- 172.21.15.7:0/3272829385 shutdown_connections 2023-03-03T15:24:17.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f663c0682f0 0x7f663c06a7a0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:24:17.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6658155390 0x7f665814a210 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:24:17.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f665806e020 0x7f66580c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:24:17.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 --2- 172.21.15.7:0/3272829385 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6658151770 0x7f665814ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:24:17.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 -- 172.21.15.7:0/3272829385 >> 172.21.15.7:0/3272829385 conn(0x7f665806c110 msgr2=0x7f665806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:24:17.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 -- 172.21.15.7:0/3272829385 shutdown_connections 2023-03-03T15:24:17.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:24:17.406+0000 7f665ee61700 1 -- 172.21.15.7:0/3272829385 wait complete. 2023-03-03T15:24:17.409 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:24:17.419 INFO:tasks.mds_thrash.fs.[b]:waiting for 119 secs before thrashing 2023-03-03T15:24:33.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:24:33.414 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:03.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:03.710 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:14.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:25:14.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.363+0000 7fef2c760700 1 Processor -- start 2023-03-03T15:25:14.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.363+0000 7fef2c760700 1 -- start start 2023-03-03T15:25:14.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef2c760700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef240aad20 0x7fef24157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:14.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef2c760700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef240a9c20 con 0x7fef2414b0d0 2023-03-03T15:25:14.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef2c760700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef24154f10 con 0x7fef240aa030 2023-03-03T15:25:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef2c760700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef240ac3c0 con 0x7fef240aad20 2023-03-03T15:25:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef29cfb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef240aad20 0x7fef24157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef2a4fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fef2414b0d0 0x7fef24071800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46844/0 (socket says 172.21.15.7:46844) 2023-03-03T15:25:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef2a4fc700 1 -- 172.21.15.7:0/3320036518 learned_addr learned my addr 172.21.15.7:0/3320036518 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:25:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef29cfb700 1 -- 172.21.15.7:0/3320036518 >> v1:172.21.15.143:6789/0 conn(0x7fef240aa030 legacy=0x7fef24158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:25:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef294fa700 1 -- 172.21.15.7:0/3320036518 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849529880 0 0) 0x7fef240a9c20 con 0x7fef2414b0d0 2023-03-03T15:25:14.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef29cfb700 1 -- 172.21.15.7:0/3320036518 >> v1:172.21.15.7:6789/0 conn(0x7fef2414b0d0 legacy=0x7fef24071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.364+0000 7fef29cfb700 1 -- 172.21.15.7:0/3320036518 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2406fc70 con 0x7fef240aad20 2023-03-03T15:25:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef29cfb700 1 --2- 172.21.15.7:0/3320036518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aad20 0x7fef24157df0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fef14009840 tx=0x7fef14009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88358d3d46c361e4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:25:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef294fa700 1 -- 172.21.15.7:0/3320036518 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14011830 con 0x7fef240aad20 2023-03-03T15:25:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef294fa700 1 -- 172.21.15.7:0/3320036518 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef14011990 con 0x7fef240aad20 2023-03-03T15:25:14.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef294fa700 1 -- 172.21.15.7:0/3320036518 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef14011d20 con 0x7fef240aad20 2023-03-03T15:25:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef2c760700 1 -- 172.21.15.7:0/3320036518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aad20 msgr2=0x7fef24157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef2c760700 1 --2- 172.21.15.7:0/3320036518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aad20 0x7fef24157df0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fef14009840 tx=0x7fef14009b50 comp rx=0 tx=0).stop 2023-03-03T15:25:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef2c760700 1 -- 172.21.15.7:0/3320036518 shutdown_connections 2023-03-03T15:25:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef2c760700 1 --2- 172.21.15.7:0/3320036518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aad20 0x7fef24157df0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef2c760700 1 -- 172.21.15.7:0/3320036518 >> 172.21.15.7:0/3320036518 conn(0x7fef240c8900 msgr2=0x7fef240c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:25:14.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef2c760700 1 -- 172.21.15.7:0/3320036518 shutdown_connections 2023-03-03T15:25:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.365+0000 7fef2c760700 1 -- 172.21.15.7:0/3320036518 wait complete. 2023-03-03T15:25:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.366+0000 7fef2c760700 1 Processor -- start 2023-03-03T15:25:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2c760700 1 -- start start 2023-03-03T15:25:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2c760700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aa030 0x7fef241e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2c760700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef240aad20 0x7fef241ede40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2c760700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef2414b0d0 0x7fef240c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2c760700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef24157830 con 0x7fef240aa030 2023-03-03T15:25:14.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2c760700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef24143710 con 0x7fef240aad20 2023-03-03T15:25:14.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2c760700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef24142f70 con 0x7fef2414b0d0 2023-03-03T15:25:14.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2a4fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aa030 0x7fef241e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:14.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2a4fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aa030 0x7fef241e9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52870/0 (socket says 172.21.15.7:52870) 2023-03-03T15:25:14.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2a4fc700 1 -- 172.21.15.7:0/2945178463 learned_addr learned my addr 172.21.15.7:0/2945178463 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:25:14.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef29cfb700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef240aad20 0x7fef241ede40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:14.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.367+0000 7fef2acfd700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef2414b0d0 0x7fef240c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:14.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef2a4fc700 1 -- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef2414b0d0 msgr2=0x7fef240c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:14.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef2a4fc700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef2414b0d0 0x7fef240c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:14.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef2a4fc700 1 -- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef240aad20 msgr2=0x7fef241ede40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:14.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef2a4fc700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef240aad20 0x7fef241ede40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:14.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef2a4fc700 1 -- 172.21.15.7:0/2945178463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef140094a0 con 0x7fef240aa030 2023-03-03T15:25:14.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef29cfb700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef240aad20 0x7fef241ede40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:25:14.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef2acfd700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef2414b0d0 0x7fef240c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:25:14.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef2a4fc700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aa030 0x7fef241e9ec0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c520 tx=0x7fef2000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:25:14.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef1affd700 1 -- 172.21.15.7:0/2945178463 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef20004c80 con 0x7fef240aa030 2023-03-03T15:25:14.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef1affd700 1 -- 172.21.15.7:0/2945178463 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef20004e00 con 0x7fef240aa030 2023-03-03T15:25:14.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef1affd700 1 -- 172.21.15.7:0/2945178463 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef20007310 con 0x7fef240aa030 2023-03-03T15:25:14.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef2c760700 1 -- 172.21.15.7:0/2945178463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2406f6b0 con 0x7fef240aa030 2023-03-03T15:25:14.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.368+0000 7fef2c760700 1 -- 172.21.15.7:0/2945178463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef24155160 con 0x7fef240aa030 2023-03-03T15:25:14.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.369+0000 7fef2c760700 1 -- 172.21.15.7:0/2945178463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2406f990 con 0x7fef240aa030 2023-03-03T15:25:14.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.369+0000 7fef1affd700 1 -- 172.21.15.7:0/2945178463 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fef2001a020 con 0x7fef240aa030 2023-03-03T15:25:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.370+0000 7fef1affd700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef080682f0 0x7fef0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:25:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.370+0000 7fef1affd700 1 -- 172.21.15.7:0/2945178463 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fef20085db0 con 0x7fef240aa030 2023-03-03T15:25:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.370+0000 7fef29cfb700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef080682f0 0x7fef0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:25:14.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.371+0000 7fef29cfb700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef080682f0 0x7fef0806a7a0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fef14009840 tx=0x7fef14008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:25:14.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.374+0000 7fef1affd700 1 -- 172.21.15.7:0/2945178463 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fef20052150 con 0x7fef240aa030 2023-03-03T15:25:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.615+0000 7fef2c760700 1 -- 172.21.15.7:0/2945178463 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef24074bb0 con 0x7fef240aa030 2023-03-03T15:25:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.616+0000 7fef1affd700 1 -- 172.21.15.7:0/2945178463 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fef20055770 con 0x7fef240aa030 2023-03-03T15:25:14.618 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:25:14.618 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:25:14.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 -- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef080682f0 msgr2=0x7fef0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef080682f0 0x7fef0806a7a0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fef14009840 tx=0x7fef14008030 comp rx=0 tx=0).stop 2023-03-03T15:25:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 -- 172.21.15.7:0/2945178463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aa030 msgr2=0x7fef241e9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:25:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aa030 0x7fef241e9ec0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c520 tx=0x7fef2000c8e0 comp rx=0 tx=0).stop 2023-03-03T15:25:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 -- 172.21.15.7:0/2945178463 shutdown_connections 2023-03-03T15:25:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef2414b0d0 0x7fef240c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef080682f0 0x7fef0806a7a0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef240aa030 0x7fef241e9ec0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 --2- 172.21.15.7:0/2945178463 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef240aad20 0x7fef241ede40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:25:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 -- 172.21.15.7:0/2945178463 >> 172.21.15.7:0/2945178463 conn(0x7fef240c8900 msgr2=0x7fef24157cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:25:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 -- 172.21.15.7:0/2945178463 shutdown_connections 2023-03-03T15:25:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:25:14.618+0000 7fef2c760700 1 -- 172.21.15.7:0/2945178463 wait complete. 2023-03-03T15:25:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:25:14.631 INFO:tasks.mds_thrash.fs.[a]:waiting for 47 secs before thrashing 2023-03-03T15:25:33.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:25:33.774 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:01.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:26:01.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.769+0000 7ff766e6f700 1 Processor -- start 2023-03-03T15:26:01.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.769+0000 7ff766e6f700 1 -- start start 2023-03-03T15:26:01.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.770+0000 7ff766e6f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff760071990 0x7ff760071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:01.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.770+0000 7ff766e6f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff76014b460 0x7ff7600722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:01.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.770+0000 7ff766e6f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff760070360 con 0x7ff760151c00 2023-03-03T15:26:01.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.770+0000 7ff766e6f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7601521a0 con 0x7ff76014b460 2023-03-03T15:26:01.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.770+0000 7ff766e6f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76006a5e0 con 0x7ff760071990 2023-03-03T15:26:01.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.770+0000 7ff75ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff76014b460 0x7ff7600722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:01.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.770+0000 7ff75ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff76014b460 0x7ff7600722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52896/0 (socket says 172.21.15.7:52896) 2023-03-03T15:26:01.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.770+0000 7ff75ffff700 1 -- 172.21.15.7:0/2392816406 learned_addr learned my addr 172.21.15.7:0/2392816406 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:26:01.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.771+0000 7ff764c0b700 1 --2- 172.21.15.7:0/2392816406 >> v2:172.21.15.143:3300/0 conn(0x7ff760071990 0x7ff760071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:01.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.771+0000 7ff75ffff700 1 -- 172.21.15.7:0/2392816406 >> v2:172.21.15.143:3300/0 conn(0x7ff760071990 msgr2=0x7ff760071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:26:01.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.771+0000 7ff75ffff700 1 --2- 172.21.15.7:0/2392816406 >> v2:172.21.15.143:3300/0 conn(0x7ff760071990 0x7ff760071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:01.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.771+0000 7ff75ffff700 1 -- 172.21.15.7:0/2392816406 >> v1:172.21.15.143:6789/0 conn(0x7ff760151c00 legacy=0x7ff76006aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:01.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.771+0000 7ff75ffff700 1 -- 172.21.15.7:0/2392816406 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7600c38b0 con 0x7ff76014b460 2023-03-03T15:26:01.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.771+0000 7ff764c0b700 1 --2- 172.21.15.7:0/2392816406 >> v2:172.21.15.143:3300/0 conn(0x7ff760071990 0x7ff760071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:26:01.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.771+0000 7ff75ffff700 1 --2- 172.21.15.7:0/2392816406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff76014b460 0x7ff7600722a0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7ff7600c1de0 tx=0x7ff74c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e7bb45f3d107a08 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:26:01.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.771+0000 7ff75f7fe700 1 -- 172.21.15.7:0/2392816406 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c00b870 con 0x7ff76014b460 2023-03-03T15:26:01.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.771+0000 7ff75f7fe700 1 -- 172.21.15.7:0/2392816406 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74c00b9d0 con 0x7ff76014b460 2023-03-03T15:26:01.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.771+0000 7ff75f7fe700 1 -- 172.21.15.7:0/2392816406 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff74c00bd60 con 0x7ff76014b460 2023-03-03T15:26:01.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.772+0000 7ff766e6f700 1 -- 172.21.15.7:0/2392816406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff76014b460 msgr2=0x7ff7600722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:01.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.772+0000 7ff766e6f700 1 --2- 172.21.15.7:0/2392816406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff76014b460 0x7ff7600722a0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7ff7600c1de0 tx=0x7ff74c009bd0 comp rx=0 tx=0).stop 2023-03-03T15:26:01.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.772+0000 7ff766e6f700 1 -- 172.21.15.7:0/2392816406 shutdown_connections 2023-03-03T15:26:01.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.772+0000 7ff766e6f700 1 --2- 172.21.15.7:0/2392816406 >> v2:172.21.15.143:3300/0 conn(0x7ff760071990 0x7ff760071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:01.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.772+0000 7ff766e6f700 1 --2- 172.21.15.7:0/2392816406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff76014b460 0x7ff7600722a0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:01.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.772+0000 7ff766e6f700 1 -- 172.21.15.7:0/2392816406 >> 172.21.15.7:0/2392816406 conn(0x7ff7600a94d0 msgr2=0x7ff7600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:26:01.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.772+0000 7ff766e6f700 1 -- 172.21.15.7:0/2392816406 shutdown_connections 2023-03-03T15:26:01.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.772+0000 7ff766e6f700 1 -- 172.21.15.7:0/2392816406 wait complete. 2023-03-03T15:26:01.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.773+0000 7ff766e6f700 1 Processor -- start 2023-03-03T15:26:01.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.773+0000 7ff766e6f700 1 -- start start 2023-03-03T15:26:01.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff766e6f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff760071990 0x7ff7601ddcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:01.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff766e6f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff760151c00 0x7ff760148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:01.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff766e6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7601494c0 0x7ff7601458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:01.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff766e6f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7601e8a80 con 0x7ff7601494c0 2023-03-03T15:26:01.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff766e6f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76014a520 con 0x7ff760071990 2023-03-03T15:26:01.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff766e6f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7601de760 con 0x7ff760151c00 2023-03-03T15:26:01.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff76540c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7601494c0 0x7ff7601458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:01.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff76540c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7601494c0 0x7ff7601458a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52904/0 (socket says 172.21.15.7:52904) 2023-03-03T15:26:01.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff76540c700 1 -- 172.21.15.7:0/2281770249 learned_addr learned my addr 172.21.15.7:0/2281770249 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:26:01.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff764c0b700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff760071990 0x7ff7601ddcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:01.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff75ffff700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff760151c00 0x7ff760148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:01.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff76540c700 1 -- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff760151c00 msgr2=0x7ff760148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:01.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff76540c700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff760151c00 0x7ff760148f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:01.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff76540c700 1 -- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff760071990 msgr2=0x7ff7601ddcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:01.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff76540c700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff760071990 0x7ff7601ddcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:01.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff76540c700 1 -- 172.21.15.7:0/2281770249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74c009550 con 0x7ff7601494c0 2023-03-03T15:26:01.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.774+0000 7ff76540c700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7601494c0 0x7ff7601458a0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7ff75800cb50 tx=0x7ff75800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:26:01.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.775+0000 7ff75d7fa700 1 -- 172.21.15.7:0/2281770249 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff758003c00 con 0x7ff7601494c0 2023-03-03T15:26:01.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.775+0000 7ff75d7fa700 1 -- 172.21.15.7:0/2281770249 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff758003d60 con 0x7ff7601494c0 2023-03-03T15:26:01.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.775+0000 7ff766e6f700 1 -- 172.21.15.7:0/2281770249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7600c3b50 con 0x7ff7601494c0 2023-03-03T15:26:01.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.775+0000 7ff75d7fa700 1 -- 172.21.15.7:0/2281770249 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75800a9c0 con 0x7ff7601494c0 2023-03-03T15:26:01.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.775+0000 7ff766e6f700 1 -- 172.21.15.7:0/2281770249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76006a2b0 con 0x7ff7601494c0 2023-03-03T15:26:01.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.776+0000 7ff766e6f700 1 -- 172.21.15.7:0/2281770249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76006b180 con 0x7ff7601494c0 2023-03-03T15:26:01.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.776+0000 7ff75d7fa700 1 -- 172.21.15.7:0/2281770249 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff758010050 con 0x7ff7601494c0 2023-03-03T15:26:01.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.777+0000 7ff75d7fa700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7440682f0 0x7ff74406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:01.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.777+0000 7ff75d7fa700 1 -- 172.21.15.7:0/2281770249 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff758085f30 con 0x7ff7601494c0 2023-03-03T15:26:01.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.777+0000 7ff764c0b700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7440682f0 0x7ff74406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:01.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.778+0000 7ff764c0b700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7440682f0 0x7ff74406a7a0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7ff7600ad550 tx=0x7ff754008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:26:01.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:01.781+0000 7ff75d7fa700 1 -- 172.21.15.7:0/2281770249 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff7580522d0 con 0x7ff7601494c0 2023-03-03T15:26:02.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.018+0000 7ff766e6f700 1 -- 172.21.15.7:0/2281770249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7601e8bc0 con 0x7ff7601494c0 2023-03-03T15:26:02.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.018+0000 7ff75d7fa700 1 -- 172.21.15.7:0/2281770249 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff7580558f0 con 0x7ff7601494c0 2023-03-03T15:26:02.020 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:26:02.020 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:26:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 -- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7440682f0 msgr2=0x7ff74406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7440682f0 0x7ff74406a7a0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7ff7600ad550 tx=0x7ff754008040 comp rx=0 tx=0).stop 2023-03-03T15:26:02.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 -- 172.21.15.7:0/2281770249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7601494c0 msgr2=0x7ff7601458a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:02.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7601494c0 0x7ff7601458a0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7ff75800cb50 tx=0x7ff75800ce60 comp rx=0 tx=0).stop 2023-03-03T15:26:02.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 -- 172.21.15.7:0/2281770249 shutdown_connections 2023-03-03T15:26:02.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff7440682f0 0x7ff74406a7a0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:02.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7601494c0 0x7ff7601458a0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:02.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff760071990 0x7ff7601ddcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:02.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 --2- 172.21.15.7:0/2281770249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff760151c00 0x7ff760148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:02.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 -- 172.21.15.7:0/2281770249 >> 172.21.15.7:0/2281770249 conn(0x7ff7600a94d0 msgr2=0x7ff7600c5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:26:02.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 -- 172.21.15.7:0/2281770249 shutdown_connections 2023-03-03T15:26:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:02.020+0000 7ff766e6f700 1 -- 172.21.15.7:0/2281770249 wait complete. 2023-03-03T15:26:02.024 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:26:02.034 INFO:tasks.mds_thrash.fs.[a]:waiting for 92 secs before thrashing 2023-03-03T15:26:03.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:03.904 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:16.421 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:26:16.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.560+0000 7f9341967700 1 Processor -- start 2023-03-03T15:26:16.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.560+0000 7f9341967700 1 -- start start 2023-03-03T15:26:16.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f9341967700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f933c151cc0 0x7f933c0ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:16.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f9341967700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f933c0cac10 0x7f933c0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:16.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f9341967700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0c7ca0 con 0x7f933c0b00b0 2023-03-03T15:26:16.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f9341967700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c152090 con 0x7f933c0cac10 2023-03-03T15:26:16.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f9341967700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c0c6cf0 con 0x7f933c151cc0 2023-03-03T15:26:16.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f933b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f933c0cac10 0x7f933c0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:16.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f933a7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f933c0b00b0 0x7f933c0c7050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46896/0 (socket says 172.21.15.7:46896) 2023-03-03T15:26:16.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f933a7fc700 1 -- 172.21.15.7:0/1803904625 learned_addr learned my addr 172.21.15.7:0/1803904625 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:26:16.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f933affd700 1 --2- 172.21.15.7:0/1803904625 >> v2:172.21.15.143:3300/0 conn(0x7f933c151cc0 0x7f933c0ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f9339ffb700 1 -- 172.21.15.7:0/1803904625 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912147866 0 0) 0x7f933c0c7ca0 con 0x7f933c0b00b0 2023-03-03T15:26:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f9339ffb700 1 -- 172.21.15.7:0/1803904625 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93240035b0 con 0x7f933c0b00b0 2023-03-03T15:26:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f933b7fe700 1 -- 172.21.15.7:0/1803904625 >> v2:172.21.15.143:3300/0 conn(0x7f933c151cc0 msgr2=0x7f933c0ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:26:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f933b7fe700 1 --2- 172.21.15.7:0/1803904625 >> v2:172.21.15.143:3300/0 conn(0x7f933c151cc0 0x7f933c0ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f933b7fe700 1 -- 172.21.15.7:0/1803904625 >> v1:172.21.15.7:6789/0 conn(0x7f933c0b00b0 legacy=0x7f933c0c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f933b7fe700 1 -- 172.21.15.7:0/1803904625 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c1548f0 con 0x7f933c0cac10 2023-03-03T15:26:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.561+0000 7f933b7fe700 1 --2- 172.21.15.7:0/1803904625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0cac10 0x7f933c0c7760 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f933c0cb350 tx=0x7f933400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8aa8f2288c104d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:26:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9339ffb700 1 -- 172.21.15.7:0/1803904625 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933400e070 con 0x7f933c0cac10 2023-03-03T15:26:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9339ffb700 1 -- 172.21.15.7:0/1803904625 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933400bb20 con 0x7f933c0cac10 2023-03-03T15:26:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9339ffb700 1 -- 172.21.15.7:0/1803904625 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9334012430 con 0x7f933c0cac10 2023-03-03T15:26:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9341967700 1 -- 172.21.15.7:0/1803904625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0cac10 msgr2=0x7f933c0c7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9341967700 1 --2- 172.21.15.7:0/1803904625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0cac10 0x7f933c0c7760 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f933c0cb350 tx=0x7f933400b330 comp rx=0 tx=0).stop 2023-03-03T15:26:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9341967700 1 -- 172.21.15.7:0/1803904625 shutdown_connections 2023-03-03T15:26:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9341967700 1 --2- 172.21.15.7:0/1803904625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0cac10 0x7f933c0c7760 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9341967700 1 --2- 172.21.15.7:0/1803904625 >> v2:172.21.15.143:3300/0 conn(0x7f933c151cc0 0x7f933c0ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9341967700 1 -- 172.21.15.7:0/1803904625 >> 172.21.15.7:0/1803904625 conn(0x7f933c06b980 msgr2=0x7f933c071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:26:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9341967700 1 -- 172.21.15.7:0/1803904625 shutdown_connections 2023-03-03T15:26:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.562+0000 7f9341967700 1 -- 172.21.15.7:0/1803904625 wait complete. 2023-03-03T15:26:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.564+0000 7f9341967700 1 Processor -- start 2023-03-03T15:26:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.564+0000 7f9341967700 1 -- start start 2023-03-03T15:26:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.564+0000 7f9341967700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0b00b0 0x7f933c0c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f9341967700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0cac10 0x7f933c142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f9341967700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f933c151cc0 0x7f933c143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f9341967700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c143a50 con 0x7f933c0b00b0 2023-03-03T15:26:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f9341967700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c14fc60 con 0x7f933c0cac10 2023-03-03T15:26:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f9341967700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c1edf80 con 0x7f933c151cc0 2023-03-03T15:26:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f933affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0b00b0 0x7f933c0c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f933affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0b00b0 0x7f933c0c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52922/0 (socket says 172.21.15.7:52922) 2023-03-03T15:26:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f933affd700 1 -- 172.21.15.7:0/901259524 learned_addr learned my addr 172.21.15.7:0/901259524 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:26:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f933a7fc700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0cac10 0x7f933c142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f933affd700 1 -- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f933c151cc0 msgr2=0x7f933c143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f933affd700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f933c151cc0 0x7f933c143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f933affd700 1 -- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0cac10 msgr2=0x7f933c142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f933affd700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0cac10 0x7f933c142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:16.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f933affd700 1 -- 172.21.15.7:0/901259524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9334009a70 con 0x7f933c0b00b0 2023-03-03T15:26:16.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f933affd700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0b00b0 0x7f933c0c9930 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f932c00e870 tx=0x7f932c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:26:16.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f9323fff700 1 -- 172.21.15.7:0/901259524 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c009590 con 0x7f933c0b00b0 2023-03-03T15:26:16.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f9323fff700 1 -- 172.21.15.7:0/901259524 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c00ee70 con 0x7f933c0b00b0 2023-03-03T15:26:16.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f9341967700 1 -- 172.21.15.7:0/901259524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c154eb0 con 0x7f933c0b00b0 2023-03-03T15:26:16.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f9323fff700 1 -- 172.21.15.7:0/901259524 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c00ca50 con 0x7f933c0b00b0 2023-03-03T15:26:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.565+0000 7f9341967700 1 -- 172.21.15.7:0/901259524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c154bd0 con 0x7f933c0b00b0 2023-03-03T15:26:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.566+0000 7f9341967700 1 -- 172.21.15.7:0/901259524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c155190 con 0x7f933c0b00b0 2023-03-03T15:26:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.567+0000 7f9323fff700 1 -- 172.21.15.7:0/901259524 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f932c019030 con 0x7f933c0b00b0 2023-03-03T15:26:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.568+0000 7f9323fff700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93280682f0 0x7f932806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:26:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.568+0000 7f9323fff700 1 -- 172.21.15.7:0/901259524 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f932c0866f0 con 0x7f933c0b00b0 2023-03-03T15:26:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.568+0000 7f933a7fc700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93280682f0 0x7f932806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:26:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.569+0000 7f933a7fc700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93280682f0 0x7f932806a7a0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f933c1de5d0 tx=0x7f9330009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:26:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.572+0000 7f9323fff700 1 -- 172.21.15.7:0/901259524 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f932c056560 con 0x7f933c0b00b0 2023-03-03T15:26:16.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.809+0000 7f9341967700 1 -- 172.21.15.7:0/901259524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f933c0c2760 con 0x7f933c0b00b0 2023-03-03T15:26:16.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.809+0000 7f9323fff700 1 -- 172.21.15.7:0/901259524 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f933c0c2760 con 0x7f933c0b00b0 2023-03-03T15:26:16.811 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:26:16.812 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:26:16.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 -- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93280682f0 msgr2=0x7f932806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:16.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93280682f0 0x7f932806a7a0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f933c1de5d0 tx=0x7f9330009040 comp rx=0 tx=0).stop 2023-03-03T15:26:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 -- 172.21.15.7:0/901259524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0b00b0 msgr2=0x7f933c0c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:26:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0b00b0 0x7f933c0c9930 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f932c00e870 tx=0x7f932c00ec30 comp rx=0 tx=0).stop 2023-03-03T15:26:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 -- 172.21.15.7:0/901259524 shutdown_connections 2023-03-03T15:26:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f933c151cc0 0x7f933c143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f93280682f0 0x7f932806a7a0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f933c0b00b0 0x7f933c0c9930 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:16.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 --2- 172.21.15.7:0/901259524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f933c0cac10 0x7f933c142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:26:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 -- 172.21.15.7:0/901259524 >> 172.21.15.7:0/901259524 conn(0x7f933c06b980 msgr2=0x7f933c1df590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:26:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.811+0000 7f9341967700 1 -- 172.21.15.7:0/901259524 shutdown_connections 2023-03-03T15:26:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:26:16.812+0000 7f9341967700 1 -- 172.21.15.7:0/901259524 wait complete. 2023-03-03T15:26:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:26:16.825 INFO:tasks.mds_thrash.fs.[b]:waiting for 112 secs before thrashing 2023-03-03T15:26:34.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:26:34.050 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:04.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:04.164 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:34.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:27:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.172+0000 7f08d779d700 1 Processor -- start 2023-03-03T15:27:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.172+0000 7f08d779d700 1 -- start start 2023-03-03T15:27:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08d779d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f08d0151770 0x7f08d0151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08d779d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d0154910 con 0x7f08d0155390 2023-03-03T15:27:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08d779d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d00c4540 con 0x7f08d006e020 2023-03-03T15:27:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08d779d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d0070b90 con 0x7f08d0151770 2023-03-03T15:27:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08d5539700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f08d0155390 0x7f08d00c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46960/0 (socket says 172.21.15.7:46960) 2023-03-03T15:27:34.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08d5539700 1 -- 172.21.15.7:0/1401884498 learned_addr learned my addr 172.21.15.7:0/1401884498 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:27:34.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835520384 0 0) 0x7f08d0154910 con 0x7f08d0155390 2023-03-03T15:27:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08d5d3a700 1 --2- 172.21.15.7:0/1401884498 >> v2:172.21.15.143:3300/0 conn(0x7f08d0151770 0x7f08d0151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08bc0035b0 con 0x7f08d0155390 2023-03-03T15:27:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3229283336 0 0) 0x7f08bc0035b0 con 0x7f08d0155390 2023-03-03T15:27:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 >> v2:172.21.15.143:3300/0 conn(0x7f08d0151770 msgr2=0x7f08d0151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:27:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08c7fff700 1 --2- 172.21.15.7:0/1401884498 >> v2:172.21.15.143:3300/0 conn(0x7f08d0151770 0x7f08d0151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.173+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 >> v1:172.21.15.143:6789/0 conn(0x7f08d006e020 legacy=0x7f08d0071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d00702a0 con 0x7f08d0155390 2023-03-03T15:27:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f08cc001670 con 0x7f08d0155390 2023-03-03T15:27:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 >> v1:172.21.15.7:6789/0 conn(0x7f08d0155390 legacy=0x7f08d00c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08c7fff700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08bc003a70 0x7f08bc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08c7fff700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc004850 0x7f08bc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08c7fff700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08bc005200 0x7f08bc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0070b90 con 0x7f08bc005200 2023-03-03T15:27:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc00c110 con 0x7f08bc003a70 2023-03-03T15:27:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc00c2d0 con 0x7f08bc004850 2023-03-03T15:27:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d5d3a700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08bc005200 0x7f08bc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d5539700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08bc003a70 0x7f08bc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d4d38700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc004850 0x7f08bc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d5d3a700 1 -- 172.21.15.7:0/1401884498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc004850 msgr2=0x7f08bc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:27:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d5d3a700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc004850 0x7f08bc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d5d3a700 1 -- 172.21.15.7:0/1401884498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08bc003a70 msgr2=0x7f08bc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:27:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d5d3a700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08bc003a70 0x7f08bc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d5d3a700 1 -- 172.21.15.7:0/1401884498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0035b0 con 0x7f08bc005200 2023-03-03T15:27:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d5539700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08bc003a70 0x7f08bc003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:27:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d5d3a700 1 -- 172.21.15.7:0/1401884498 reap_dead start 2023-03-03T15:27:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08d5d3a700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08bc005200 0x7f08bc00ba10 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f08c80112c0 tx=0x7f08bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc0dd0900ddebef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:27:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.174+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c8011de0 con 0x7f08bc005200 2023-03-03T15:27:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.175+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c800e040 con 0x7f08bc005200 2023-03-03T15:27:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.175+0000 7f08c7fff700 1 -- 172.21.15.7:0/1401884498 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c8003c60 con 0x7f08bc005200 2023-03-03T15:27:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.175+0000 7f08d779d700 1 -- 172.21.15.7:0/1401884498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08bc005200 msgr2=0x7f08bc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.175+0000 7f08d779d700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08bc005200 0x7f08bc00ba10 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f08c80112c0 tx=0x7f08bc005880 comp rx=0 tx=0).stop 2023-03-03T15:27:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.175+0000 7f08d779d700 1 -- 172.21.15.7:0/1401884498 shutdown_connections 2023-03-03T15:27:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.175+0000 7f08d779d700 1 --2- 172.21.15.7:0/1401884498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08bc005200 0x7f08bc00ba10 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.175+0000 7f08d779d700 1 -- 172.21.15.7:0/1401884498 >> 172.21.15.7:0/1401884498 conn(0x7f08d006c110 msgr2=0x7f08d006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:27:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.175+0000 7f08d779d700 1 -- 172.21.15.7:0/1401884498 shutdown_connections 2023-03-03T15:27:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.175+0000 7f08d779d700 1 -- 172.21.15.7:0/1401884498 wait complete. 2023-03-03T15:27:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.176+0000 7f08d779d700 1 Processor -- start 2023-03-03T15:27:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.176+0000 7f08d779d700 1 -- start start 2023-03-03T15:27:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d779d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc005200 0x7f08d014d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d779d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d014dab0 0x7f08d00c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d779d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d00c1690 0x7f08d01e8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d779d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d01e6240 con 0x7f08d00c1690 2023-03-03T15:27:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d779d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0147e80 con 0x7f08d014dab0 2023-03-03T15:27:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d779d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0148380 con 0x7f08bc005200 2023-03-03T15:27:34.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d5d3a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d00c1690 0x7f08d01e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:34.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d5539700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc005200 0x7f08d014d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:34.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d5d3a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d00c1690 0x7f08d01e8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52996/0 (socket says 172.21.15.7:52996) 2023-03-03T15:27:34.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d5d3a700 1 -- 172.21.15.7:0/28960547 learned_addr learned my addr 172.21.15.7:0/28960547 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:27:34.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d4d38700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d014dab0 0x7f08d00c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:34.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d5d3a700 1 -- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc005200 msgr2=0x7f08d014d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:34.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.177+0000 7f08d5d3a700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc005200 0x7f08d014d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.178+0000 7f08d5d3a700 1 -- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d014dab0 msgr2=0x7f08d00c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:34.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.178+0000 7f08d5d3a700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d014dab0 0x7f08d00c25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.178+0000 7f08d5d3a700 1 -- 172.21.15.7:0/28960547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d0070820 con 0x7f08d00c1690 2023-03-03T15:27:34.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.178+0000 7f08d5d3a700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d00c1690 0x7f08d01e8960 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f08d00c2580 tx=0x7f08c8004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:27:34.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.178+0000 7f08c5ffb700 1 -- 172.21.15.7:0/28960547 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c8004430 con 0x7f08d00c1690 2023-03-03T15:27:34.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.178+0000 7f08c5ffb700 1 -- 172.21.15.7:0/28960547 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c80045b0 con 0x7f08d00c1690 2023-03-03T15:27:34.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.178+0000 7f08d779d700 1 -- 172.21.15.7:0/28960547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d0070540 con 0x7f08d00c1690 2023-03-03T15:27:34.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.178+0000 7f08c5ffb700 1 -- 172.21.15.7:0/28960547 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c8004960 con 0x7f08d00c1690 2023-03-03T15:27:34.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.178+0000 7f08d779d700 1 -- 172.21.15.7:0/28960547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d0155760 con 0x7f08d00c1690 2023-03-03T15:27:34.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.179+0000 7f08d779d700 1 -- 172.21.15.7:0/28960547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d0074bb0 con 0x7f08d00c1690 2023-03-03T15:27:34.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.180+0000 7f08c5ffb700 1 -- 172.21.15.7:0/28960547 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f08c802f030 con 0x7f08d00c1690 2023-03-03T15:27:34.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.180+0000 7f08c5ffb700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08b40682f0 0x7f08b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:27:34.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.180+0000 7f08c5ffb700 1 -- 172.21.15.7:0/28960547 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f08c8089830 con 0x7f08d00c1690 2023-03-03T15:27:34.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.180+0000 7f08d5539700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08b40682f0 0x7f08b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:27:34.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.181+0000 7f08d5539700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08b40682f0 0x7f08b406a7a0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f08cc001200 tx=0x7f08cc000ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:27:34.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.184+0000 7f08c5ffb700 1 -- 172.21.15.7:0/28960547 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08c8055bd0 con 0x7f08d00c1690 2023-03-03T15:27:34.306 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:34.308 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:27:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.423+0000 7f08d779d700 1 -- 172.21.15.7:0/28960547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08d00c1da0 con 0x7f08d00c1690 2023-03-03T15:27:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.423+0000 7f08c5ffb700 1 -- 172.21.15.7:0/28960547 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f08c80591f0 con 0x7f08d00c1690 2023-03-03T15:27:34.425 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:27:34.425 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:27:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 -- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08b40682f0 msgr2=0x7f08b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08b40682f0 0x7f08b406a7a0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f08cc001200 tx=0x7f08cc000ed0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 -- 172.21.15.7:0/28960547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d00c1690 msgr2=0x7f08d01e8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:27:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d00c1690 0x7f08d01e8960 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f08d00c2580 tx=0x7f08c8004b40 comp rx=0 tx=0).stop 2023-03-03T15:27:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 -- 172.21.15.7:0/28960547 shutdown_connections 2023-03-03T15:27:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08b40682f0 0x7f08b406a7a0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f08d00c1690 0x7f08d01e8960 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08bc005200 0x7f08d014d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 --2- 172.21.15.7:0/28960547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d014dab0 0x7f08d00c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:27:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 -- 172.21.15.7:0/28960547 >> 172.21.15.7:0/28960547 conn(0x7f08d006c110 msgr2=0x7f08d006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:27:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 -- 172.21.15.7:0/28960547 shutdown_connections 2023-03-03T15:27:34.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:27:34.425+0000 7f08d779d700 1 -- 172.21.15.7:0/28960547 wait complete. 2023-03-03T15:27:34.429 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:27:34.439 INFO:tasks.mds_thrash.fs.[a]:waiting for 102 secs before thrashing 2023-03-03T15:28:04.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:04.482 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:08.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:28:08.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.964+0000 7f0c4d95a700 1 Processor -- start 2023-03-03T15:28:08.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.964+0000 7f0c4d95a700 1 -- start start 2023-03-03T15:28:08.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.965+0000 7f0c4d95a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c4806d240 0x7f0c480c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.965+0000 7f0c4d95a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c480aa530 con 0x7f0c480c3e30 2023-03-03T15:28:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.965+0000 7f0c4d95a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c480c2cf0 con 0x7f0c4814fbe0 2023-03-03T15:28:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.965+0000 7f0c4d95a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c480ad2d0 con 0x7f0c4806d240 2023-03-03T15:28:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.965+0000 7f0c467fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c4806d240 0x7f0c480c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.965+0000 7f0c477fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0c480c3e30 0x7f0c4815cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47014/0 (socket says 172.21.15.7:47014) 2023-03-03T15:28:08.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.965+0000 7f0c477fe700 1 -- 172.21.15.7:0/2041427254 learned_addr learned my addr 172.21.15.7:0/2041427254 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:28:08.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.966+0000 7f0c467fc700 1 -- 172.21.15.7:0/2041427254 >> v1:172.21.15.143:6789/0 conn(0x7f0c4814fbe0 legacy=0x7f0c480c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:08.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.966+0000 7f0c467fc700 1 -- 172.21.15.7:0/2041427254 >> v1:172.21.15.7:6789/0 conn(0x7f0c480c3e30 legacy=0x7f0c4815cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:08.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.966+0000 7f0c467fc700 1 -- 172.21.15.7:0/2041427254 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c48074290 con 0x7f0c4806d240 2023-03-03T15:28:08.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.966+0000 7f0c45ffb700 1 -- 172.21.15.7:0/2041427254 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366733118 0 0) 0x7f0c480aa530 con 0x7f0c480c3e30 2023-03-03T15:28:08.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.966+0000 7f0c467fc700 1 --2- 172.21.15.7:0/2041427254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c4806d240 0x7f0c480c27b0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f0c38009840 tx=0x7f0c38009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89adea6e4b3e6690 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:08.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.966+0000 7f0c45ffb700 1 -- 172.21.15.7:0/2041427254 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c38011830 con 0x7f0c4806d240 2023-03-03T15:28:08.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.966+0000 7f0c45ffb700 1 -- 172.21.15.7:0/2041427254 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c38011990 con 0x7f0c4806d240 2023-03-03T15:28:08.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.966+0000 7f0c45ffb700 1 -- 172.21.15.7:0/2041427254 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c38011d20 con 0x7f0c4806d240 2023-03-03T15:28:08.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.967+0000 7f0c4d95a700 1 -- 172.21.15.7:0/2041427254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c4806d240 msgr2=0x7f0c480c27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:08.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.967+0000 7f0c4d95a700 1 --2- 172.21.15.7:0/2041427254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c4806d240 0x7f0c480c27b0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f0c38009840 tx=0x7f0c38009b50 comp rx=0 tx=0).stop 2023-03-03T15:28:08.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.967+0000 7f0c4d95a700 1 -- 172.21.15.7:0/2041427254 shutdown_connections 2023-03-03T15:28:08.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.967+0000 7f0c4d95a700 1 --2- 172.21.15.7:0/2041427254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c4806d240 0x7f0c480c27b0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:08.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.967+0000 7f0c4d95a700 1 -- 172.21.15.7:0/2041427254 >> 172.21.15.7:0/2041427254 conn(0x7f0c4806c110 msgr2=0x7f0c4806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:28:08.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.967+0000 7f0c4d95a700 1 -- 172.21.15.7:0/2041427254 shutdown_connections 2023-03-03T15:28:08.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.967+0000 7f0c4d95a700 1 -- 172.21.15.7:0/2041427254 wait complete. 2023-03-03T15:28:08.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.968+0000 7f0c4d95a700 1 Processor -- start 2023-03-03T15:28:08.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.968+0000 7f0c4d95a700 1 -- start start 2023-03-03T15:28:08.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c4d95a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4806d240 0x7f0c48071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:08.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c4d95a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c480c3e30 0x7f0c480c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:08.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c4d95a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c4814fbe0 0x7f0c481ea880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:08.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c4d95a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c48153300 con 0x7f0c480c3e30 2023-03-03T15:28:08.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c4d95a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c480721e0 con 0x7f0c4814fbe0 2023-03-03T15:28:08.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c4d95a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c481ee970 con 0x7f0c4806d240 2023-03-03T15:28:08.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c467fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c480c3e30 0x7f0c480c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:08.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c467fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c480c3e30 0x7f0c480c0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53038/0 (socket says 172.21.15.7:53038) 2023-03-03T15:28:08.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c467fc700 1 -- 172.21.15.7:0/3784123934 learned_addr learned my addr 172.21.15.7:0/3784123934 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:28:08.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c46ffd700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4806d240 0x7f0c48071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:08.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c477fe700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c4814fbe0 0x7f0c481ea880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:08.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c467fc700 1 -- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4806d240 msgr2=0x7f0c48071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:08.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c467fc700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4806d240 0x7f0c48071ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:08.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c467fc700 1 -- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c4814fbe0 msgr2=0x7f0c481ea880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:08.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c467fc700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c4814fbe0 0x7f0c481ea880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:08.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c467fc700 1 -- 172.21.15.7:0/3784123934 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c380094a0 con 0x7f0c480c3e30 2023-03-03T15:28:08.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c46ffd700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4806d240 0x7f0c48071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:28:08.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c477fe700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c4814fbe0 0x7f0c481ea880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:28:08.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.969+0000 7f0c467fc700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c480c3e30 0x7f0c480c0a40 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f0c380090b0 tx=0x7f0c38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:08.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.970+0000 7f0c2ffff700 1 -- 172.21.15.7:0/3784123934 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3801a900 con 0x7f0c480c3e30 2023-03-03T15:28:08.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.970+0000 7f0c2ffff700 1 -- 172.21.15.7:0/3784123934 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3801aa60 con 0x7f0c480c3e30 2023-03-03T15:28:08.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.970+0000 7f0c4d95a700 1 -- 172.21.15.7:0/3784123934 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c48152f20 con 0x7f0c480c3e30 2023-03-03T15:28:08.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.970+0000 7f0c2ffff700 1 -- 172.21.15.7:0/3784123934 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3801adf0 con 0x7f0c480c3e30 2023-03-03T15:28:08.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.970+0000 7f0c4d95a700 1 -- 172.21.15.7:0/3784123934 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c48074570 con 0x7f0c480c3e30 2023-03-03T15:28:08.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.971+0000 7f0c4d95a700 1 -- 172.21.15.7:0/3784123934 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c480acf60 con 0x7f0c480c3e30 2023-03-03T15:28:08.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.971+0000 7f0c2ffff700 1 -- 172.21.15.7:0/3784123934 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0c3802b030 con 0x7f0c480c3e30 2023-03-03T15:28:08.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.972+0000 7f0c2ffff700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c340682f0 0x7f0c3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:08.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.972+0000 7f0c2ffff700 1 -- 172.21.15.7:0/3784123934 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0c38087440 con 0x7f0c480c3e30 2023-03-03T15:28:08.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.972+0000 7f0c46ffd700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c340682f0 0x7f0c3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:08.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.972+0000 7f0c46ffd700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c340682f0 0x7f0c3406a7a0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c000f60 tx=0x7f0c3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:08.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:08.976+0000 7f0c2ffff700 1 -- 172.21.15.7:0/3784123934 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0c380537e0 con 0x7f0c480c3e30 2023-03-03T15:28:09.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.212+0000 7f0c4d95a700 1 -- 172.21.15.7:0/3784123934 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c48074bb0 con 0x7f0c480c3e30 2023-03-03T15:28:09.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.213+0000 7f0c2ffff700 1 -- 172.21.15.7:0/3784123934 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0c38056e00 con 0x7f0c480c3e30 2023-03-03T15:28:09.215 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:28:09.215 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:28:09.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 -- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c340682f0 msgr2=0x7f0c3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:09.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c340682f0 0x7f0c3406a7a0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c000f60 tx=0x7f0c3c009040 comp rx=0 tx=0).stop 2023-03-03T15:28:09.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 -- 172.21.15.7:0/3784123934 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c480c3e30 msgr2=0x7f0c480c0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:09.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c480c3e30 0x7f0c480c0a40 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f0c380090b0 tx=0x7f0c38004750 comp rx=0 tx=0).stop 2023-03-03T15:28:09.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 -- 172.21.15.7:0/3784123934 shutdown_connections 2023-03-03T15:28:09.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c4814fbe0 0x7f0c481ea880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:09.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4806d240 0x7f0c48071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:09.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c340682f0 0x7f0c3406a7a0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:09.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 --2- 172.21.15.7:0/3784123934 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c480c3e30 0x7f0c480c0a40 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:09.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 -- 172.21.15.7:0/3784123934 >> 172.21.15.7:0/3784123934 conn(0x7f0c4806c110 msgr2=0x7f0c4814a810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:28:09.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 -- 172.21.15.7:0/3784123934 shutdown_connections 2023-03-03T15:28:09.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:09.215+0000 7f0c4d95a700 1 -- 172.21.15.7:0/3784123934 wait complete. 2023-03-03T15:28:09.218 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:28:09.228 INFO:tasks.mds_thrash.fs.[b]:waiting for 15 secs before thrashing 2023-03-03T15:28:24.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:28:24.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.367+0000 7f583be80700 1 Processor -- start 2023-03-03T15:28:24.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.368+0000 7f583be80700 1 -- start start 2023-03-03T15:28:24.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.368+0000 7f583be80700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5834151d40 0x7f58340736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:24.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.368+0000 7f583be80700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f58340c2930 0x7f5834157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:24.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.368+0000 7f583be80700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583406d430 con 0x7f5834073c30 2023-03-03T15:28:24.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.368+0000 7f583be80700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583406c2a0 con 0x7f5834151d40 2023-03-03T15:28:24.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.368+0000 7f583be80700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58341586e0 con 0x7f58340c2930 2023-03-03T15:28:24.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5839c1c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5834151d40 0x7f58340736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:24.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5839c1c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5834151d40 0x7f58340736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53048/0 (socket says 172.21.15.7:53048) 2023-03-03T15:28:24.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5839c1c700 1 -- 172.21.15.7:0/323464374 learned_addr learned my addr 172.21.15.7:0/323464374 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:28:24.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f583941b700 1 --2- 172.21.15.7:0/323464374 >> v2:172.21.15.143:3300/0 conn(0x7f58340c2930 0x7f5834157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:24.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5839c1c700 1 -- 172.21.15.7:0/323464374 >> v2:172.21.15.143:3300/0 conn(0x7f58340c2930 msgr2=0x7f5834157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:28:24.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5839c1c700 1 --2- 172.21.15.7:0/323464374 >> v2:172.21.15.143:3300/0 conn(0x7f58340c2930 0x7f5834157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:24.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5839c1c700 1 -- 172.21.15.7:0/323464374 >> v1:172.21.15.143:6789/0 conn(0x7f5834073c30 legacy=0x7f5834157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:28:24.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5839c1c700 1 -- 172.21.15.7:0/323464374 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583406cb00 con 0x7f5834151d40 2023-03-03T15:28:24.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f583941b700 1 --2- 172.21.15.7:0/323464374 >> v2:172.21.15.143:3300/0 conn(0x7f58340c2930 0x7f5834157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:28:24.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5839c1c700 1 --2- 172.21.15.7:0/323464374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 0x7f58340736f0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f58300098f0 tx=0x7f5830009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bae43f82d642f2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:24.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5838c1a700 1 -- 172.21.15.7:0/323464374 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830004230 con 0x7f5834151d40 2023-03-03T15:28:24.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5838c1a700 1 -- 172.21.15.7:0/323464374 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5830004390 con 0x7f5834151d40 2023-03-03T15:28:24.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.369+0000 7f5838c1a700 1 -- 172.21.15.7:0/323464374 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58300036a0 con 0x7f5834151d40 2023-03-03T15:28:24.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.370+0000 7f583be80700 1 -- 172.21.15.7:0/323464374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 msgr2=0x7f58340736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:24.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.370+0000 7f583be80700 1 --2- 172.21.15.7:0/323464374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 0x7f58340736f0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f58300098f0 tx=0x7f5830009c00 comp rx=0 tx=0).stop 2023-03-03T15:28:24.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.370+0000 7f583be80700 1 -- 172.21.15.7:0/323464374 shutdown_connections 2023-03-03T15:28:24.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.370+0000 7f583be80700 1 --2- 172.21.15.7:0/323464374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 0x7f58340736f0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:24.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.370+0000 7f583be80700 1 --2- 172.21.15.7:0/323464374 >> v2:172.21.15.143:3300/0 conn(0x7f58340c2930 0x7f5834157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:24.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.370+0000 7f583be80700 1 -- 172.21.15.7:0/323464374 >> 172.21.15.7:0/323464374 conn(0x7f58340a94d0 msgr2=0x7f58340a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:28:24.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.370+0000 7f583be80700 1 -- 172.21.15.7:0/323464374 shutdown_connections 2023-03-03T15:28:24.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.370+0000 7f583be80700 1 -- 172.21.15.7:0/323464374 wait complete. 2023-03-03T15:28:24.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.371+0000 7f583be80700 1 Processor -- start 2023-03-03T15:28:24.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.371+0000 7f583be80700 1 -- start start 2023-03-03T15:28:24.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583be80700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5834073c30 0x7f58340bee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:24.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583be80700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58340c2930 0x7f5834153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:24.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583be80700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 0x7f5834153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:24.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583be80700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5834155420 con 0x7f5834151d40 2023-03-03T15:28:24.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583be80700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5834154500 con 0x7f58340c2930 2023-03-03T15:28:24.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583be80700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58341535f0 con 0x7f5834073c30 2023-03-03T15:28:24.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583a41d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 0x7f5834153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:24.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583a41d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 0x7f5834153fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53056/0 (socket says 172.21.15.7:53056) 2023-03-03T15:28:24.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583a41d700 1 -- 172.21.15.7:0/4201358957 learned_addr learned my addr 172.21.15.7:0/4201358957 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:28:24.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f5839c1c700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5834073c30 0x7f58340bee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:24.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583941b700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58340c2930 0x7f5834153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:24.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583a41d700 1 -- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5834073c30 msgr2=0x7f58340bee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583a41d700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5834073c30 0x7f58340bee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583a41d700 1 -- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58340c2930 msgr2=0x7f5834153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583a41d700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58340c2930 0x7f5834153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.372+0000 7f583a41d700 1 -- 172.21.15.7:0/4201358957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5830009550 con 0x7f5834151d40 2023-03-03T15:28:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.373+0000 7f583a41d700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 0x7f5834153fc0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f582c00b9e0 tx=0x7f582c00bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.373+0000 7f58267fc700 1 -- 172.21.15.7:0/4201358957 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c003c00 con 0x7f5834151d40 2023-03-03T15:28:24.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.373+0000 7f58267fc700 1 -- 172.21.15.7:0/4201358957 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c003da0 con 0x7f5834151d40 2023-03-03T15:28:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.373+0000 7f583be80700 1 -- 172.21.15.7:0/4201358957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583406d0c0 con 0x7f5834151d40 2023-03-03T15:28:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.373+0000 7f58267fc700 1 -- 172.21.15.7:0/4201358957 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c00d920 con 0x7f5834151d40 2023-03-03T15:28:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.373+0000 7f583be80700 1 -- 172.21.15.7:0/4201358957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583406cde0 con 0x7f5834151d40 2023-03-03T15:28:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.374+0000 7f583be80700 1 -- 172.21.15.7:0/4201358957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583406bea0 con 0x7f5834151d40 2023-03-03T15:28:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.374+0000 7f58267fc700 1 -- 172.21.15.7:0/4201358957 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f582c01b020 con 0x7f5834151d40 2023-03-03T15:28:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.375+0000 7f58267fc700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5818068320 0x7f581806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:28:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.375+0000 7f58267fc700 1 -- 172.21.15.7:0/4201358957 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f582c011070 con 0x7f5834151d40 2023-03-03T15:28:24.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.375+0000 7f5839c1c700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5818068320 0x7f581806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:28:24.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.376+0000 7f5839c1c700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5818068320 0x7f581806a7d0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f5830009130 tx=0x7f5830008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:28:24.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.379+0000 7f58267fc700 1 -- 172.21.15.7:0/4201358957 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f582c0523a0 con 0x7f5834151d40 2023-03-03T15:28:24.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.619+0000 7f583be80700 1 -- 172.21.15.7:0/4201358957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5834153760 con 0x7f5834151d40 2023-03-03T15:28:24.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.619+0000 7f58267fc700 1 -- 172.21.15.7:0/4201358957 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f582c0559c0 con 0x7f5834151d40 2023-03-03T15:28:24.621 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:28:24.622 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:28:24.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.621+0000 7f583be80700 1 -- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5818068320 msgr2=0x7f581806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:24.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5818068320 0x7f581806a7d0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f5830009130 tx=0x7f5830008030 comp rx=0 tx=0).stop 2023-03-03T15:28:24.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 -- 172.21.15.7:0/4201358957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 msgr2=0x7f5834153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:28:24.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 0x7f5834153fc0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f582c00b9e0 tx=0x7f582c00bcf0 comp rx=0 tx=0).stop 2023-03-03T15:28:24.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 -- 172.21.15.7:0/4201358957 shutdown_connections 2023-03-03T15:28:24.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5818068320 0x7f581806a7d0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:24.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5834151d40 0x7f5834153fc0 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:24.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5834073c30 0x7f58340bee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:24.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 --2- 172.21.15.7:0/4201358957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58340c2930 0x7f5834153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:28:24.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 -- 172.21.15.7:0/4201358957 >> 172.21.15.7:0/4201358957 conn(0x7f58340a94d0 msgr2=0x7f58340c2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:28:24.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 -- 172.21.15.7:0/4201358957 shutdown_connections 2023-03-03T15:28:24.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:28:24.622+0000 7f583be80700 1 -- 172.21.15.7:0/4201358957 wait complete. 2023-03-03T15:28:24.625 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:28:24.635 INFO:tasks.mds_thrash.fs.[b]:waiting for 94 secs before thrashing 2023-03-03T15:28:34.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:28:34.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:04.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:04.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:16.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:29:16.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.576+0000 7f597bbd0700 1 Processor -- start 2023-03-03T15:29:16.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.577+0000 7f597bbd0700 1 -- start start 2023-03-03T15:29:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.577+0000 7f597bbd0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f597406d870 0x7f59741550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.577+0000 7f597bbd0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59740c3e30 0x7f59740c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.577+0000 7f597bbd0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59740c7a60 con 0x7f59740c1da0 2023-03-03T15:29:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.577+0000 7f597bbd0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740c2af0 con 0x7f59740c3e30 2023-03-03T15:29:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.577+0000 7f597bbd0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59740c3620 con 0x7f597406d870 2023-03-03T15:29:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597a16d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59740c3e30 0x7f59740c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597a16d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59740c3e30 0x7f59740c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53080/0 (socket says 172.21.15.7:53080) 2023-03-03T15:29:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597a16d700 1 -- 172.21.15.7:0/3369718716 learned_addr learned my addr 172.21.15.7:0/3369718716 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:29:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597996c700 1 --2- 172.21.15.7:0/3369718716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f597406d870 0x7f59741550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597a16d700 1 -- 172.21.15.7:0/3369718716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f597406d870 msgr2=0x7f59741550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:29:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597a16d700 1 --2- 172.21.15.7:0/3369718716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f597406d870 0x7f59741550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597a16d700 1 -- 172.21.15.7:0/3369718716 >> v1:172.21.15.143:6789/0 conn(0x7f59740c1da0 legacy=0x7f59740c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597a16d700 1 -- 172.21.15.7:0/3369718716 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59741545e0 con 0x7f59740c3e30 2023-03-03T15:29:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597a16d700 1 --2- 172.21.15.7:0/3369718716 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59740c3e30 0x7f59740c4290 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f596c009e50 tx=0x7f596c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f678f91f26787a8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:16.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597896a700 1 -- 172.21.15.7:0/3369718716 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c00e070 con 0x7f59740c3e30 2023-03-03T15:29:16.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597896a700 1 -- 172.21.15.7:0/3369718716 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c00b910 con 0x7f59740c3e30 2023-03-03T15:29:16.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.578+0000 7f597896a700 1 -- 172.21.15.7:0/3369718716 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c00bcc0 con 0x7f59740c3e30 2023-03-03T15:29:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.579+0000 7f597bbd0700 1 -- 172.21.15.7:0/3369718716 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59740c3e30 msgr2=0x7f59740c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.579+0000 7f597bbd0700 1 --2- 172.21.15.7:0/3369718716 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59740c3e30 0x7f59740c4290 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f596c009e50 tx=0x7f596c00b330 comp rx=0 tx=0).stop 2023-03-03T15:29:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.579+0000 7f597bbd0700 1 -- 172.21.15.7:0/3369718716 shutdown_connections 2023-03-03T15:29:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.579+0000 7f597bbd0700 1 --2- 172.21.15.7:0/3369718716 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59740c3e30 0x7f59740c4290 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.579+0000 7f597bbd0700 1 --2- 172.21.15.7:0/3369718716 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f597406d870 0x7f59741550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.579+0000 7f597bbd0700 1 -- 172.21.15.7:0/3369718716 >> 172.21.15.7:0/3369718716 conn(0x7f597406c110 msgr2=0x7f597406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:29:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.579+0000 7f597bbd0700 1 -- 172.21.15.7:0/3369718716 shutdown_connections 2023-03-03T15:29:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.579+0000 7f597bbd0700 1 -- 172.21.15.7:0/3369718716 wait complete. 2023-03-03T15:29:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.580+0000 7f597bbd0700 1 Processor -- start 2023-03-03T15:29:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.580+0000 7f597bbd0700 1 -- start start 2023-03-03T15:29:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597bbd0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f597406d870 0x7f59740aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597bbd0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59740c1da0 0x7f59741e9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597bbd0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59740c3e30 0x7f59740ac920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597bbd0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5974157cc0 con 0x7f597406d870 2023-03-03T15:29:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597bbd0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59740aaa30 con 0x7f59740c3e30 2023-03-03T15:29:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597bbd0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59740aad10 con 0x7f59740c1da0 2023-03-03T15:29:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597996c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f597406d870 0x7f59740aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597996c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f597406d870 0x7f59740aa450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53082/0 (socket says 172.21.15.7:53082) 2023-03-03T15:29:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597996c700 1 -- 172.21.15.7:0/43225654 learned_addr learned my addr 172.21.15.7:0/43225654 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:29:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597a16d700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59740c3e30 0x7f59740ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597916b700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59740c1da0 0x7f59741e9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597996c700 1 -- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59740c1da0 msgr2=0x7f59741e9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597996c700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59740c1da0 0x7f59741e9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597996c700 1 -- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59740c3e30 msgr2=0x7f59740ac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597996c700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59740c3e30 0x7f59740ac920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597996c700 1 -- 172.21.15.7:0/43225654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c009b30 con 0x7f597406d870 2023-03-03T15:29:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.581+0000 7f597916b700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59740c1da0 0x7f59741e9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:29:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.582+0000 7f597996c700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f597406d870 0x7f59740aa450 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f597000e9b0 tx=0x7f597000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.582+0000 7f59667fc700 1 -- 172.21.15.7:0/43225654 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970009710 con 0x7f597406d870 2023-03-03T15:29:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.582+0000 7f59667fc700 1 -- 172.21.15.7:0/43225654 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597000c850 con 0x7f597406d870 2023-03-03T15:29:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.582+0000 7f597bbd0700 1 -- 172.21.15.7:0/43225654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5974154ba0 con 0x7f597406d870 2023-03-03T15:29:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.582+0000 7f59667fc700 1 -- 172.21.15.7:0/43225654 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597000cbc0 con 0x7f597406d870 2023-03-03T15:29:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.582+0000 7f597bbd0700 1 -- 172.21.15.7:0/43225654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59741548c0 con 0x7f597406d870 2023-03-03T15:29:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.583+0000 7f597bbd0700 1 -- 172.21.15.7:0/43225654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597406dc40 con 0x7f597406d870 2023-03-03T15:29:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.583+0000 7f59667fc700 1 -- 172.21.15.7:0/43225654 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5970019030 con 0x7f597406d870 2023-03-03T15:29:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.584+0000 7f59667fc700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59580682f0 0x7f595806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.584+0000 7f59667fc700 1 -- 172.21.15.7:0/43225654 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5970086080 con 0x7f597406d870 2023-03-03T15:29:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.584+0000 7f597916b700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59580682f0 0x7f595806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.584+0000 7f597916b700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59580682f0 0x7f595806a7a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f5974072180 tx=0x7f5968009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.588+0000 7f59667fc700 1 -- 172.21.15.7:0/43225654 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5970052420 con 0x7f597406d870 2023-03-03T15:29:16.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.830+0000 7f597bbd0700 1 -- 172.21.15.7:0/43225654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5974074bb0 con 0x7f597406d870 2023-03-03T15:29:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.830+0000 7f59667fc700 1 -- 172.21.15.7:0/43225654 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5970017090 con 0x7f597406d870 2023-03-03T15:29:16.832 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:29:16.832 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:29:16.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.832+0000 7f597bbd0700 1 -- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59580682f0 msgr2=0x7f595806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.832+0000 7f597bbd0700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59580682f0 0x7f595806a7a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f5974072180 tx=0x7f5968009040 comp rx=0 tx=0).stop 2023-03-03T15:29:16.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.832+0000 7f597bbd0700 1 -- 172.21.15.7:0/43225654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f597406d870 msgr2=0x7f59740aa450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:16.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.832+0000 7f597bbd0700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f597406d870 0x7f59740aa450 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f597000e9b0 tx=0x7f597000ecc0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.833+0000 7f597bbd0700 1 -- 172.21.15.7:0/43225654 shutdown_connections 2023-03-03T15:29:16.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.833+0000 7f597bbd0700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59740c3e30 0x7f59740ac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.833+0000 7f597bbd0700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59580682f0 0x7f595806a7a0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.833+0000 7f597bbd0700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f597406d870 0x7f59740aa450 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.833+0000 7f597bbd0700 1 --2- 172.21.15.7:0/43225654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59740c1da0 0x7f59741e9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:16.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.833+0000 7f597bbd0700 1 -- 172.21.15.7:0/43225654 >> 172.21.15.7:0/43225654 conn(0x7f597406c110 msgr2=0x7f597406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:29:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.833+0000 7f597bbd0700 1 -- 172.21.15.7:0/43225654 shutdown_connections 2023-03-03T15:29:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:16.833+0000 7f597bbd0700 1 -- 172.21.15.7:0/43225654 wait complete. 2023-03-03T15:29:16.836 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:29:16.846 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before thrashing 2023-03-03T15:29:22.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:29:22.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.984+0000 7f8e04234700 1 Processor -- start 2023-03-03T15:29:22.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.984+0000 7f8e04234700 1 -- start start 2023-03-03T15:29:22.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e04234700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8dfc14b0d0 0x7f8dfc071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:22.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e04234700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8dfc071d40 0x7f8dfc158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e04234700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dfc0a9c20 con 0x7f8dfc0aad20 2023-03-03T15:29:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e04234700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dfc154f10 con 0x7f8dfc071d40 2023-03-03T15:29:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e04234700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dfc0ac3c0 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e027d1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8dfc071d40 0x7f8dfc158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e017cf700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8dfc0aad20 0x7f8dfc157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47072/0 (socket says 172.21.15.7:47072) 2023-03-03T15:29:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e017cf700 1 -- 172.21.15.7:0/3390798467 learned_addr learned my addr 172.21.15.7:0/3390798467 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:29:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e01fd0700 1 --2- 172.21.15.7:0/3390798467 >> v2:172.21.15.143:3300/0 conn(0x7f8dfc14b0d0 0x7f8dfc071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e00fce700 1 -- 172.21.15.7:0/3390798467 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114437645 0 0) 0x7f8dfc0a9c20 con 0x7f8dfc0aad20 2023-03-03T15:29:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e00fce700 1 -- 172.21.15.7:0/3390798467 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de80035b0 con 0x7f8dfc0aad20 2023-03-03T15:29:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e027d1700 1 -- 172.21.15.7:0/3390798467 >> v2:172.21.15.143:3300/0 conn(0x7f8dfc14b0d0 msgr2=0x7f8dfc071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:29:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e027d1700 1 --2- 172.21.15.7:0/3390798467 >> v2:172.21.15.143:3300/0 conn(0x7f8dfc14b0d0 0x7f8dfc071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e027d1700 1 -- 172.21.15.7:0/3390798467 >> v1:172.21.15.7:6789/0 conn(0x7f8dfc0aad20 legacy=0x7f8dfc157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.985+0000 7f8e027d1700 1 -- 172.21.15.7:0/3390798467 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dfc06fc70 con 0x7f8dfc071d40 2023-03-03T15:29:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e027d1700 1 --2- 172.21.15.7:0/3390798467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc071d40 0x7f8dfc158500 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc070930 tx=0x7f8df400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80111f25c7e5803f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:22.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e00fce700 1 -- 172.21.15.7:0/3390798467 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df400e070 con 0x7f8dfc071d40 2023-03-03T15:29:22.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e00fce700 1 -- 172.21.15.7:0/3390798467 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df400bb20 con 0x7f8dfc071d40 2023-03-03T15:29:22.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e00fce700 1 -- 172.21.15.7:0/3390798467 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4012430 con 0x7f8dfc071d40 2023-03-03T15:29:22.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e04234700 1 -- 172.21.15.7:0/3390798467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc071d40 msgr2=0x7f8dfc158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:22.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e04234700 1 --2- 172.21.15.7:0/3390798467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc071d40 0x7f8dfc158500 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc070930 tx=0x7f8df400b330 comp rx=0 tx=0).stop 2023-03-03T15:29:22.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e04234700 1 -- 172.21.15.7:0/3390798467 shutdown_connections 2023-03-03T15:29:22.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e04234700 1 --2- 172.21.15.7:0/3390798467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc071d40 0x7f8dfc158500 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:22.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e04234700 1 --2- 172.21.15.7:0/3390798467 >> v2:172.21.15.143:3300/0 conn(0x7f8dfc14b0d0 0x7f8dfc071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:22.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e04234700 1 -- 172.21.15.7:0/3390798467 >> 172.21.15.7:0/3390798467 conn(0x7f8dfc0c8900 msgr2=0x7f8dfc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:29:22.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e04234700 1 -- 172.21.15.7:0/3390798467 shutdown_connections 2023-03-03T15:29:22.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.986+0000 7f8e04234700 1 -- 172.21.15.7:0/3390798467 wait complete. 2023-03-03T15:29:22.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e04234700 1 Processor -- start 2023-03-03T15:29:22.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e04234700 1 -- start start 2023-03-03T15:29:22.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e04234700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dfc071d40 0x7f8dfc149e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:22.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e04234700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dfc0aad20 0x7f8dfc1ea1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:22.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e04234700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc14b0d0 0x7f8dfc1ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:22.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e04234700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc1ee2b0 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e04234700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc152580 con 0x7f8dfc0aad20 2023-03-03T15:29:22.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e04234700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dfc1542e0 con 0x7f8dfc071d40 2023-03-03T15:29:22.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e027d1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc14b0d0 0x7f8dfc1ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:22.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e027d1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc14b0d0 0x7f8dfc1ea6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53100/0 (socket says 172.21.15.7:53100) 2023-03-03T15:29:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.988+0000 7f8e027d1700 1 -- 172.21.15.7:0/3532071793 learned_addr learned my addr 172.21.15.7:0/3532071793 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:29:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e017cf700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dfc0aad20 0x7f8dfc1ea1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e01fd0700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dfc071d40 0x7f8dfc149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e027d1700 1 -- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dfc071d40 msgr2=0x7f8dfc149e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e027d1700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dfc071d40 0x7f8dfc149e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e027d1700 1 -- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dfc0aad20 msgr2=0x7f8dfc1ea1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e027d1700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dfc0aad20 0x7f8dfc1ea1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:22.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e027d1700 1 -- 172.21.15.7:0/3532071793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df4009a70 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e017cf700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dfc0aad20 0x7f8dfc1ea1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:29:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e027d1700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc14b0d0 0x7f8dfc1ea6e0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f8df4013000 tx=0x7f8df4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8df27fc700 1 -- 172.21.15.7:0/3532071793 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4012430 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8df27fc700 1 -- 172.21.15.7:0/3532071793 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df4020070 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e04234700 1 -- 172.21.15.7:0/3532071793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dfc06f6b0 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8df27fc700 1 -- 172.21.15.7:0/3532071793 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df4004280 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.989+0000 7f8e04234700 1 -- 172.21.15.7:0/3532071793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dfc155160 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.990+0000 7f8e04234700 1 -- 172.21.15.7:0/3532071793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dfc06f990 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.991+0000 7f8df27fc700 1 -- 172.21.15.7:0/3532071793 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8df402e070 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.991+0000 7f8df27fc700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8de00682f0 0x7f8de006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:22.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.991+0000 7f8df27fc700 1 -- 172.21.15.7:0/3532071793 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8df4022070 con 0x7f8dfc14b0d0 2023-03-03T15:29:22.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.991+0000 7f8e01fd0700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8de00682f0 0x7f8de006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:22.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.992+0000 7f8e01fd0700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8de00682f0 0x7f8de006a7a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc154e90 tx=0x7f8dec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:22.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:22.996+0000 7f8df27fc700 1 -- 172.21.15.7:0/3532071793 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8df4054250 con 0x7f8dfc14b0d0 2023-03-03T15:29:23.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.232+0000 7f8e04234700 1 -- 172.21.15.7:0/3532071793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dfc074bb0 con 0x7f8dfc14b0d0 2023-03-03T15:29:23.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.233+0000 7f8df27fc700 1 -- 172.21.15.7:0/3532071793 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8df4057870 con 0x7f8dfc14b0d0 2023-03-03T15:29:23.235 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:29:23.235 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:29:23.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 -- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8de00682f0 msgr2=0x7f8de006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8de00682f0 0x7f8de006a7a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc154e90 tx=0x7f8dec008040 comp rx=0 tx=0).stop 2023-03-03T15:29:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 -- 172.21.15.7:0/3532071793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc14b0d0 msgr2=0x7f8dfc1ea6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc14b0d0 0x7f8dfc1ea6e0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f8df4013000 tx=0x7f8df4004ad0 comp rx=0 tx=0).stop 2023-03-03T15:29:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 -- 172.21.15.7:0/3532071793 shutdown_connections 2023-03-03T15:29:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8de00682f0 0x7f8de006a7a0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:23.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dfc14b0d0 0x7f8dfc1ea6e0 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8dfc071d40 0x7f8dfc149e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 --2- 172.21.15.7:0/3532071793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dfc0aad20 0x7f8dfc1ea1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 -- 172.21.15.7:0/3532071793 >> 172.21.15.7:0/3532071793 conn(0x7f8dfc0c8900 msgr2=0x7f8dfc0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:29:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 -- 172.21.15.7:0/3532071793 shutdown_connections 2023-03-03T15:29:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:23.235+0000 7f8e04234700 1 -- 172.21.15.7:0/3532071793 wait complete. 2023-03-03T15:29:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:29:23.248 INFO:tasks.mds_thrash.fs.[a]:waiting for 80 secs before thrashing 2023-03-03T15:29:34.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:34.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:29:58.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:29:58.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.773+0000 7f6c37b83700 1 Processor -- start 2023-03-03T15:29:58.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.773+0000 7f6c37b83700 1 -- start start 2023-03-03T15:29:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c37b83700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c30072250 0x7f6c3014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c37b83700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6c300c1ce0 0x7f6c3014e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c37b83700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c301510d0 con 0x7f6c30071e80 2023-03-03T15:29:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c37b83700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c300c3740 con 0x7f6c300c1ce0 2023-03-03T15:29:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c37b83700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c30070670 con 0x7f6c30072250 2023-03-03T15:29:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3591f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6c30071e80 0x7f6c30151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47092/0 (socket says 172.21.15.7:47092) 2023-03-03T15:29:58.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3591f700 1 -- 172.21.15.7:0/3299253624 learned_addr learned my addr 172.21.15.7:0/3299253624 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:29:58.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c36120700 1 --2- 172.21.15.7:0/3299253624 >> v2:172.21.15.143:3300/0 conn(0x7f6c300c1ce0 0x7f6c3014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:58.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101563298 0 0) 0x7f6c301510d0 con 0x7f6c30071e80 2023-03-03T15:29:58.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3511e700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c30072250 0x7f6c3014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:58.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c1c0035b0 con 0x7f6c30071e80 2023-03-03T15:29:58.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 903154476 0 0) 0x7f6c1c0035b0 con 0x7f6c30071e80 2023-03-03T15:29:58.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c30072250 msgr2=0x7f6c3014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:29:58.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c30072250 0x7f6c3014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:58.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 >> v2:172.21.15.143:3300/0 conn(0x7f6c300c1ce0 msgr2=0x7f6c3014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:29:58.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 --2- 172.21.15.7:0/3299253624 >> v2:172.21.15.143:3300/0 conn(0x7f6c300c1ce0 0x7f6c3014e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:58.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c300c39e0 con 0x7f6c30071e80 2023-03-03T15:29:58.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f6c2c001670 con 0x7f6c30071e80 2023-03-03T15:29:58.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 >> v1:172.21.15.7:6789/0 conn(0x7f6c30071e80 legacy=0x7f6c30151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:58.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c1c003a70 0x7f6c1c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:58.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c30072250 0x7f6c1c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:58.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c1c004d90 0x7f6c1c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:58.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c300c3740 con 0x7f6c1c004d90 2023-03-03T15:29:58.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c30070670 con 0x7f6c1c003a70 2023-03-03T15:29:58.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.774+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c00ba60 con 0x7f6c30072250 2023-03-03T15:29:58.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c36120700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c1c004d90 0x7f6c1c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:58.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c3591f700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c1c003a70 0x7f6c1c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:58.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c3511e700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c30072250 0x7f6c1c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:58.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c36120700 1 -- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c30072250 msgr2=0x7f6c1c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:29:58.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c36120700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c30072250 0x7f6c1c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:58.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c36120700 1 -- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c1c003a70 msgr2=0x7f6c1c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:29:58.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c36120700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c1c003a70 0x7f6c1c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:58.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c36120700 1 -- 172.21.15.7:0/3299253624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c0035b0 con 0x7f6c1c004d90 2023-03-03T15:29:58.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c36120700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c1c004d90 0x7f6c1c005200 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f6c28009fc0 tx=0x7f6c1c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7d249abbebb9e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:58.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2800ec30 con 0x7f6c1c004d90 2023-03-03T15:29:58.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2800f040 con 0x7f6c1c004d90 2023-03-03T15:29:58.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.775+0000 7f6c3491d700 1 -- 172.21.15.7:0/3299253624 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28014480 con 0x7f6c1c004d90 2023-03-03T15:29:58.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.776+0000 7f6c37b83700 1 -- 172.21.15.7:0/3299253624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c1c004d90 msgr2=0x7f6c1c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:58.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.776+0000 7f6c37b83700 1 --2- 172.21.15.7:0/3299253624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c1c004d90 0x7f6c1c005200 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f6c28009fc0 tx=0x7f6c1c005880 comp rx=0 tx=0).stop 2023-03-03T15:29:58.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.776+0000 7f6c36120700 1 -- 172.21.15.7:0/3299253624 reap_dead start 2023-03-03T15:29:58.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.776+0000 7f6c37b83700 1 -- 172.21.15.7:0/3299253624 shutdown_connections 2023-03-03T15:29:58.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.776+0000 7f6c37b83700 1 -- 172.21.15.7:0/3299253624 >> 172.21.15.7:0/3299253624 conn(0x7f6c300a94d0 msgr2=0x7f6c300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:29:58.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.776+0000 7f6c37b83700 1 -- 172.21.15.7:0/3299253624 shutdown_connections 2023-03-03T15:29:58.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.776+0000 7f6c37b83700 1 -- 172.21.15.7:0/3299253624 wait complete. 2023-03-03T15:29:58.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.777+0000 7f6c37b83700 1 Processor -- start 2023-03-03T15:29:58.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.777+0000 7f6c37b83700 1 -- start start 2023-03-03T15:29:58.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c37b83700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c300c6550 0x7f6c30149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:58.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c37b83700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c300c6920 0x7f6c301ea810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:58.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c37b83700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3013ded0 0x7f6c300736f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:58.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c37b83700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c300bda40 con 0x7f6c300c6550 2023-03-03T15:29:58.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c37b83700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c301ea380 con 0x7f6c3013ded0 2023-03-03T15:29:58.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c37b83700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c30150f00 con 0x7f6c300c6920 2023-03-03T15:29:58.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3591f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c300c6550 0x7f6c30149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:58.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3591f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c300c6550 0x7f6c30149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53124/0 (socket says 172.21.15.7:53124) 2023-03-03T15:29:58.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3591f700 1 -- 172.21.15.7:0/497295329 learned_addr learned my addr 172.21.15.7:0/497295329 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:29:58.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c36120700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3013ded0 0x7f6c300736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:58.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3511e700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c300c6920 0x7f6c301ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:58.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3591f700 1 -- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c300c6920 msgr2=0x7f6c301ea810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:58.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3591f700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c300c6920 0x7f6c301ea810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:58.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3591f700 1 -- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3013ded0 msgr2=0x7f6c300736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:58.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3591f700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3013ded0 0x7f6c300736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:58.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3591f700 1 -- 172.21.15.7:0/497295329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c30151f30 con 0x7f6c300c6550 2023-03-03T15:29:58.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c36120700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3013ded0 0x7f6c300736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:29:58.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3511e700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c300c6920 0x7f6c301ea810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:29:58.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c3591f700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c300c6550 0x7f6c30149380 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c000c00 tx=0x7f6c2c0076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:58.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c267fc700 1 -- 172.21.15.7:0/497295329 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c0077c0 con 0x7f6c300c6550 2023-03-03T15:29:58.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.778+0000 7f6c267fc700 1 -- 172.21.15.7:0/497295329 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2c00f8a0 con 0x7f6c300c6550 2023-03-03T15:29:58.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.779+0000 7f6c37b83700 1 -- 172.21.15.7:0/497295329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c30151300 con 0x7f6c300c6550 2023-03-03T15:29:58.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.779+0000 7f6c267fc700 1 -- 172.21.15.7:0/497295329 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c00fc50 con 0x7f6c300c6550 2023-03-03T15:29:58.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.779+0000 7f6c37b83700 1 -- 172.21.15.7:0/497295329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3006c9a0 con 0x7f6c300c6550 2023-03-03T15:29:58.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.779+0000 7f6c37b83700 1 -- 172.21.15.7:0/497295329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c301498c0 con 0x7f6c300c6550 2023-03-03T15:29:58.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.781+0000 7f6c267fc700 1 -- 172.21.15.7:0/497295329 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c2c00fdb0 con 0x7f6c300c6550 2023-03-03T15:29:58.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.781+0000 7f6c267fc700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c140682f0 0x7f6c1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:58.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.781+0000 7f6c267fc700 1 -- 172.21.15.7:0/497295329 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6c2c086de0 con 0x7f6c300c6550 2023-03-03T15:29:58.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.781+0000 7f6c3511e700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c140682f0 0x7f6c1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:58.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.782+0000 7f6c3511e700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c140682f0 0x7f6c1406a7a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f6c301e7d40 tx=0x7f6c20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:58.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:58.785+0000 7f6c267fc700 1 -- 172.21.15.7:0/497295329 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c2c056c50 con 0x7f6c300c6550 2023-03-03T15:29:59.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.022+0000 7f6c37b83700 1 -- 172.21.15.7:0/497295329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c30074bb0 con 0x7f6c300c6550 2023-03-03T15:29:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.022+0000 7f6c267fc700 1 -- 172.21.15.7:0/497295329 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6c2c0567e0 con 0x7f6c300c6550 2023-03-03T15:29:59.025 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:29:59.025 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:29:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.024+0000 7f6c37b83700 1 -- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c140682f0 msgr2=0x7f6c1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.024+0000 7f6c37b83700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c140682f0 0x7f6c1406a7a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f6c301e7d40 tx=0x7f6c20008040 comp rx=0 tx=0).stop 2023-03-03T15:29:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.024+0000 7f6c37b83700 1 -- 172.21.15.7:0/497295329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c300c6550 msgr2=0x7f6c30149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.024+0000 7f6c37b83700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c300c6550 0x7f6c30149380 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c000c00 tx=0x7f6c2c0076a0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.024+0000 7f6c37b83700 1 -- 172.21.15.7:0/497295329 shutdown_connections 2023-03-03T15:29:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.024+0000 7f6c37b83700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c3013ded0 0x7f6c300736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.024+0000 7f6c37b83700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c140682f0 0x7f6c1406a7a0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.024+0000 7f6c37b83700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c300c6550 0x7f6c30149380 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.024+0000 7f6c37b83700 1 --2- 172.21.15.7:0/497295329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c300c6920 0x7f6c301ea810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.025+0000 7f6c37b83700 1 -- 172.21.15.7:0/497295329 >> 172.21.15.7:0/497295329 conn(0x7f6c300a94d0 msgr2=0x7f6c3014dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:29:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.025+0000 7f6c37b83700 1 -- 172.21.15.7:0/497295329 shutdown_connections 2023-03-03T15:29:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.025+0000 7f6c37b83700 1 -- 172.21.15.7:0/497295329 wait complete. 2023-03-03T15:29:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:29:59.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:29:59.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.161+0000 7f2508a4e700 1 Processor -- start 2023-03-03T15:29:59.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.161+0000 7f2508a4e700 1 -- start start 2023-03-03T15:29:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2508a4e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f250414b460 0x7f250406aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2508a4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 0x7f250406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2508a4e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2504070360 con 0x7f2504071990 2023-03-03T15:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2508a4e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25041521a0 con 0x7f250414b460 2023-03-03T15:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2508a4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250406a5e0 con 0x7f2504072470 2023-03-03T15:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f250259c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2504071990 0x7f2504071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47112/0 (socket says 172.21.15.7:47112) 2023-03-03T15:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f250259c700 1 -- 172.21.15.7:0/689511500 learned_addr learned my addr 172.21.15.7:0/689511500 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2502d9d700 1 --2- 172.21.15.7:0/689511500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 0x7f250406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2501d9b700 1 --2- 172.21.15.7:0/689511500 >> v2:172.21.15.143:3300/0 conn(0x7f250414b460 0x7f250406aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f250159a700 1 -- 172.21.15.7:0/689511500 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998025828 0 0) 0x7f2504070360 con 0x7f2504071990 2023-03-03T15:29:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f250159a700 1 -- 172.21.15.7:0/689511500 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ec0035b0 con 0x7f2504071990 2023-03-03T15:29:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f250159a700 1 -- 172.21.15.7:0/689511500 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1577503885 0 0) 0x7f24ec0035b0 con 0x7f2504071990 2023-03-03T15:29:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2502d9d700 1 -- 172.21.15.7:0/689511500 >> v2:172.21.15.143:3300/0 conn(0x7f250414b460 msgr2=0x7f250406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:29:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2502d9d700 1 --2- 172.21.15.7:0/689511500 >> v2:172.21.15.143:3300/0 conn(0x7f250414b460 0x7f250406aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2502d9d700 1 -- 172.21.15.7:0/689511500 >> v1:172.21.15.7:6789/0 conn(0x7f2504071990 legacy=0x7f2504071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.162+0000 7f2502d9d700 1 -- 172.21.15.7:0/689511500 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25040c38b0 con 0x7f2504072470 2023-03-03T15:29:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f2502d9d700 1 --2- 172.21.15.7:0/689511500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 0x7f250406afb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009d90 tx=0x7f24fc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7c75134fe2e519c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f250159a700 1 -- 172.21.15.7:0/689511500 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc00e070 con 0x7f2504072470 2023-03-03T15:29:59.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f250159a700 1 -- 172.21.15.7:0/689511500 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc003a20 con 0x7f2504072470 2023-03-03T15:29:59.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f250159a700 1 -- 172.21.15.7:0/689511500 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f4001670 con 0x7f2504072470 2023-03-03T15:29:59.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f2508a4e700 1 -- 172.21.15.7:0/689511500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 msgr2=0x7f250406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:59.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f2508a4e700 1 --2- 172.21.15.7:0/689511500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 0x7f250406afb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009d90 tx=0x7f24fc00b330 comp rx=0 tx=0).stop 2023-03-03T15:29:59.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f2508a4e700 1 -- 172.21.15.7:0/689511500 shutdown_connections 2023-03-03T15:29:59.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f2508a4e700 1 --2- 172.21.15.7:0/689511500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 0x7f250406afb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f2508a4e700 1 --2- 172.21.15.7:0/689511500 >> v2:172.21.15.143:3300/0 conn(0x7f250414b460 0x7f250406aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f2508a4e700 1 -- 172.21.15.7:0/689511500 >> 172.21.15.7:0/689511500 conn(0x7f25040a94d0 msgr2=0x7f25040a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:29:59.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f2508a4e700 1 -- 172.21.15.7:0/689511500 shutdown_connections 2023-03-03T15:29:59.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.163+0000 7f2508a4e700 1 -- 172.21.15.7:0/689511500 wait complete. 2023-03-03T15:29:59.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.165+0000 7f2508a4e700 1 Processor -- start 2023-03-03T15:29:59.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.165+0000 7f2508a4e700 1 -- start start 2023-03-03T15:29:59.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.165+0000 7f2508a4e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2504071990 0x7f2504143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.165+0000 7f2508a4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 0x7f25041503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.165+0000 7f2508a4e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25040c0d40 0x7f25040c11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.165+0000 7f2508a4e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25040beef0 con 0x7f25040c0d40 2023-03-03T15:29:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.165+0000 7f2508a4e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2504144440 con 0x7f2504071990 2023-03-03T15:29:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2508a4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2504142850 con 0x7f2504072470 2023-03-03T15:29:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2502d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25040c0d40 0x7f25040c11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2502d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25040c0d40 0x7f25040c11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53142/0 (socket says 172.21.15.7:53142) 2023-03-03T15:29:59.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2502d9d700 1 -- 172.21.15.7:0/2628553888 learned_addr learned my addr 172.21.15.7:0/2628553888 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:29:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2501d9b700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 0x7f25041503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f250259c700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2504071990 0x7f2504143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2502d9d700 1 -- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 msgr2=0x7f25041503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2502d9d700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 0x7f25041503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2502d9d700 1 -- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2504071990 msgr2=0x7f2504143f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2502d9d700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2504071990 0x7f2504143f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2502d9d700 1 -- 172.21.15.7:0/2628553888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc009a70 con 0x7f25040c0d40 2023-03-03T15:29:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2501d9b700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 0x7f25041503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:29:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2502d9d700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25040c0d40 0x7f25040c11c0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f250406f280 tx=0x7f24fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f24f2ffd700 1 -- 172.21.15.7:0/2628553888 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc0045f0 con 0x7f25040c0d40 2023-03-03T15:29:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f24f2ffd700 1 -- 172.21.15.7:0/2628553888 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24fc004750 con 0x7f25040c0d40 2023-03-03T15:29:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2508a4e700 1 -- 172.21.15.7:0/2628553888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25040c3b50 con 0x7f25040c0d40 2023-03-03T15:29:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f24f2ffd700 1 -- 172.21.15.7:0/2628553888 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc009410 con 0x7f25040c0d40 2023-03-03T15:29:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.166+0000 7f2508a4e700 1 -- 172.21.15.7:0/2628553888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f250406a2b0 con 0x7f25040c0d40 2023-03-03T15:29:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.167+0000 7f2508a4e700 1 -- 172.21.15.7:0/2628553888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250406b4f0 con 0x7f25040c0d40 2023-03-03T15:29:59.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.168+0000 7f24f2ffd700 1 -- 172.21.15.7:0/2628553888 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f24fc012430 con 0x7f25040c0d40 2023-03-03T15:29:59.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.168+0000 7f24f2ffd700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e40682f0 0x7f24e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:29:59.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.168+0000 7f24f2ffd700 1 -- 172.21.15.7:0/2628553888 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f24fc088030 con 0x7f25040c0d40 2023-03-03T15:29:59.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.168+0000 7f250259c700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e40682f0 0x7f24e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:29:59.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.169+0000 7f250259c700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e40682f0 0x7f24e406a7a0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f24f4001eb0 tx=0x7f24f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:29:59.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.172+0000 7f24f2ffd700 1 -- 172.21.15.7:0/2628553888 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24fc054370 con 0x7f25040c0d40 2023-03-03T15:29:59.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.411+0000 7f2508a4e700 1 -- 172.21.15.7:0/2628553888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2504074bb0 con 0x7f25040c0d40 2023-03-03T15:29:59.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.411+0000 7f24f2ffd700 1 -- 172.21.15.7:0/2628553888 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f24fc057990 con 0x7f25040c0d40 2023-03-03T15:29:59.414 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:29:59.414 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:29:59.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 -- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e40682f0 msgr2=0x7f24e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:59.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e40682f0 0x7f24e406a7a0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f24f4001eb0 tx=0x7f24f4009040 comp rx=0 tx=0).stop 2023-03-03T15:29:59.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 -- 172.21.15.7:0/2628553888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25040c0d40 msgr2=0x7f25040c11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:29:59.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25040c0d40 0x7f25040c11c0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f250406f280 tx=0x7f24fc004ad0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 -- 172.21.15.7:0/2628553888 shutdown_connections 2023-03-03T15:29:59.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e40682f0 0x7f24e406a7a0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25040c0d40 0x7f25040c11c0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2504071990 0x7f2504143f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 --2- 172.21.15.7:0/2628553888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2504072470 0x7f25041503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:29:59.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 -- 172.21.15.7:0/2628553888 >> 172.21.15.7:0/2628553888 conn(0x7f25040a94d0 msgr2=0x7f2504148b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:29:59.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 -- 172.21.15.7:0/2628553888 shutdown_connections 2023-03-03T15:29:59.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:29:59.413+0000 7f2508a4e700 1 -- 172.21.15.7:0/2628553888 wait complete. 2023-03-03T15:29:59.418 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:29:59.428 INFO:tasks.mds_thrash.fs.[b]:waiting for 41 secs before thrashing 2023-03-03T15:30:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T15:29:59.998+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T15:30:05.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:05.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:35.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:35.206 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:30:40.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:30:40.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.561+0000 7eff95524700 1 Processor -- start 2023-03-03T15:30:40.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.562+0000 7eff95524700 1 -- start start 2023-03-03T15:30:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.562+0000 7eff95524700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff900caa30 0x7eff900c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.562+0000 7eff95524700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 0x7eff90154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.562+0000 7eff95524700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff900cb280 con 0x7eff900cae00 2023-03-03T15:30:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.562+0000 7eff95524700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff900c7310 con 0x7eff900caa30 2023-03-03T15:30:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.562+0000 7eff95524700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff900bfd10 con 0x7eff900c00e0 2023-03-03T15:30:40.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.562+0000 7eff8effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff900caa30 0x7eff900c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.562+0000 7eff8f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7eff900cae00 0x7eff900c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47164/0 (socket says 172.21.15.7:47164) 2023-03-03T15:30:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.562+0000 7eff8f7fe700 1 -- 172.21.15.7:0/2837692592 learned_addr learned my addr 172.21.15.7:0/2837692592 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:30:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.563+0000 7eff8e7fc700 1 --2- 172.21.15.7:0/2837692592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 0x7eff90154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.563+0000 7eff8dffb700 1 -- 172.21.15.7:0/2837692592 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133395387 0 0) 0x7eff900cb280 con 0x7eff900cae00 2023-03-03T15:30:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.563+0000 7eff8effd700 1 -- 172.21.15.7:0/2837692592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 msgr2=0x7eff90154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:30:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.563+0000 7eff8effd700 1 --2- 172.21.15.7:0/2837692592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 0x7eff90154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:40.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.563+0000 7eff8effd700 1 -- 172.21.15.7:0/2837692592 >> v1:172.21.15.7:6789/0 conn(0x7eff900cae00 legacy=0x7eff900c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.563+0000 7eff8effd700 1 -- 172.21.15.7:0/2837692592 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff900bfa30 con 0x7eff900caa30 2023-03-03T15:30:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.563+0000 7eff8effd700 1 --2- 172.21.15.7:0/2837692592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 0x7eff900c5c60 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7eff84009840 tx=0x7eff84009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720b221029a7c5d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.563+0000 7eff8dffb700 1 -- 172.21.15.7:0/2837692592 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84011830 con 0x7eff900caa30 2023-03-03T15:30:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.563+0000 7eff8dffb700 1 -- 172.21.15.7:0/2837692592 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff84011990 con 0x7eff900caa30 2023-03-03T15:30:40.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.563+0000 7eff8dffb700 1 -- 172.21.15.7:0/2837692592 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84011d20 con 0x7eff900caa30 2023-03-03T15:30:40.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.564+0000 7eff95524700 1 -- 172.21.15.7:0/2837692592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 msgr2=0x7eff900c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.564+0000 7eff95524700 1 --2- 172.21.15.7:0/2837692592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 0x7eff900c5c60 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7eff84009840 tx=0x7eff84009b50 comp rx=0 tx=0).stop 2023-03-03T15:30:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.564+0000 7eff95524700 1 -- 172.21.15.7:0/2837692592 shutdown_connections 2023-03-03T15:30:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.564+0000 7eff95524700 1 --2- 172.21.15.7:0/2837692592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 0x7eff900c5c60 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.564+0000 7eff95524700 1 --2- 172.21.15.7:0/2837692592 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 0x7eff90154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.564+0000 7eff95524700 1 -- 172.21.15.7:0/2837692592 >> 172.21.15.7:0/2837692592 conn(0x7eff90066e40 msgr2=0x7eff90071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:30:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.564+0000 7eff95524700 1 -- 172.21.15.7:0/2837692592 shutdown_connections 2023-03-03T15:30:40.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.564+0000 7eff95524700 1 -- 172.21.15.7:0/2837692592 wait complete. 2023-03-03T15:30:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.566+0000 7eff95524700 1 Processor -- start 2023-03-03T15:30:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.566+0000 7eff95524700 1 -- start start 2023-03-03T15:30:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.566+0000 7eff95524700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 0x7eff900c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.566+0000 7eff95524700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 0x7eff90157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.566+0000 7eff95524700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff900cae00 0x7eff901ed120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff95524700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff901dda40 con 0x7eff900caa30 2023-03-03T15:30:40.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff95524700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff90158860 con 0x7eff900cae00 2023-03-03T15:30:40.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff95524700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff900c4760 con 0x7eff900c00e0 2023-03-03T15:30:40.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 0x7eff90157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:40.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 0x7eff90157230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53188/0 (socket says 172.21.15.7:53188) 2023-03-03T15:30:40.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8e7fc700 1 -- 172.21.15.7:0/9402193 learned_addr learned my addr 172.21.15.7:0/9402193 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:30:40.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8effd700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 0x7eff900c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:40.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8f7fe700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff900cae00 0x7eff901ed120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:40.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8e7fc700 1 -- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 msgr2=0x7eff900c26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:40.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8e7fc700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 0x7eff900c26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:40.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8e7fc700 1 -- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff900cae00 msgr2=0x7eff901ed120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:40.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8e7fc700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff900cae00 0x7eff901ed120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:40.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8e7fc700 1 -- 172.21.15.7:0/9402193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff840094a0 con 0x7eff900caa30 2023-03-03T15:30:40.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8effd700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 0x7eff900c26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:30:40.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff8e7fc700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 0x7eff90157230 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7eff8000e760 tx=0x7eff8000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:40.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff777fe700 1 -- 172.21.15.7:0/9402193 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80013610 con 0x7eff900caa30 2023-03-03T15:30:40.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff777fe700 1 -- 172.21.15.7:0/9402193 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff80013770 con 0x7eff900caa30 2023-03-03T15:30:40.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff95524700 1 -- 172.21.15.7:0/9402193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff901547f0 con 0x7eff900caa30 2023-03-03T15:30:40.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff777fe700 1 -- 172.21.15.7:0/9402193 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff80013b20 con 0x7eff900caa30 2023-03-03T15:30:40.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.567+0000 7eff95524700 1 -- 172.21.15.7:0/9402193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff90154510 con 0x7eff900caa30 2023-03-03T15:30:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.568+0000 7eff95524700 1 -- 172.21.15.7:0/9402193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff90154ad0 con 0x7eff900caa30 2023-03-03T15:30:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.569+0000 7eff777fe700 1 -- 172.21.15.7:0/9402193 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7eff80019030 con 0x7eff900caa30 2023-03-03T15:30:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.569+0000 7eff777fe700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff7c0682f0 0x7eff7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.569+0000 7eff777fe700 1 -- 172.21.15.7:0/9402193 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7eff80085a90 con 0x7eff900caa30 2023-03-03T15:30:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.569+0000 7eff8effd700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff7c0682f0 0x7eff7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:40.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.570+0000 7eff8effd700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff7c0682f0 0x7eff7c06a7a0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7eff840090b0 tx=0x7eff84008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:40.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.574+0000 7eff777fe700 1 -- 172.21.15.7:0/9402193 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7eff80051e30 con 0x7eff900caa30 2023-03-03T15:30:40.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.811+0000 7eff95524700 1 -- 172.21.15.7:0/9402193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff90074bb0 con 0x7eff900caa30 2023-03-03T15:30:40.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.811+0000 7eff777fe700 1 -- 172.21.15.7:0/9402193 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7eff80055450 con 0x7eff900caa30 2023-03-03T15:30:40.813 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:30:40.813 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:30:40.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 -- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff7c0682f0 msgr2=0x7eff7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:40.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff7c0682f0 0x7eff7c06a7a0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7eff840090b0 tx=0x7eff84008030 comp rx=0 tx=0).stop 2023-03-03T15:30:40.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 -- 172.21.15.7:0/9402193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 msgr2=0x7eff90157230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:40.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 0x7eff90157230 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7eff8000e760 tx=0x7eff8000eb20 comp rx=0 tx=0).stop 2023-03-03T15:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 -- 172.21.15.7:0/9402193 shutdown_connections 2023-03-03T15:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff900cae00 0x7eff901ed120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7eff900c00e0 0x7eff900c26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7eff7c0682f0 0x7eff7c06a7a0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 --2- 172.21.15.7:0/9402193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff900caa30 0x7eff90157230 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 -- 172.21.15.7:0/9402193 >> 172.21.15.7:0/9402193 conn(0x7eff90066e40 msgr2=0x7eff90153c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:30:40.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 -- 172.21.15.7:0/9402193 shutdown_connections 2023-03-03T15:30:40.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:40.813+0000 7eff95524700 1 -- 172.21.15.7:0/9402193 wait complete. 2023-03-03T15:30:40.817 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:30:40.827 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before thrashing 2023-03-03T15:30:43.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:30:43.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.386+0000 7f8f66bf6700 1 Processor -- start 2023-03-03T15:30:43.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.386+0000 7f8f66bf6700 1 -- start start 2023-03-03T15:30:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f66bf6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8f6006d870 0x7f8f601550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f66bf6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f600c3e30 0x7f8f600c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f66bf6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f600c7a60 con 0x7f8f600c1da0 2023-03-03T15:30:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f66bf6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f600c2af0 con 0x7f8f6006d870 2023-03-03T15:30:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f66bf6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f600c3620 con 0x7f8f600c3e30 2023-03-03T15:30:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f64992700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8f6006d870 0x7f8f601550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:43.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f65193700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f600c3e30 0x7f8f600c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f64992700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8f6006d870 0x7f8f601550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:48972/0 (socket says 172.21.15.7:48972) 2023-03-03T15:30:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f65193700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f600c3e30 0x7f8f600c4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:33042/0 (socket says 172.21.15.7:33042) 2023-03-03T15:30:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f5ffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8f600c1da0 0x7f8f600c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:50698/0 (socket says 172.21.15.7:50698) 2023-03-03T15:30:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.387+0000 7f8f64992700 1 -- 172.21.15.7:0/3065789040 learned_addr learned my addr 172.21.15.7:0/3065789040 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:30:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.388+0000 7f8f64992700 1 -- 172.21.15.7:0/3065789040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f600c3e30 msgr2=0x7f8f600c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:30:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.388+0000 7f8f64992700 1 --2- 172.21.15.7:0/3065789040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f600c3e30 0x7f8f600c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:43.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.388+0000 7f8f64992700 1 -- 172.21.15.7:0/3065789040 >> v1:172.21.15.143:6789/0 conn(0x7f8f600c1da0 legacy=0x7f8f600c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:43.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.388+0000 7f8f64992700 1 -- 172.21.15.7:0/3065789040 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f601545e0 con 0x7f8f6006d870 2023-03-03T15:30:43.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.388+0000 7f8f65193700 1 --2- 172.21.15.7:0/3065789040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f600c3e30 0x7f8f600c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:30:43.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.388+0000 7f8f64992700 1 --2- 172.21.15.7:0/3065789040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f6006d870 0x7f8f601550c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8f540098f0 tx=0x7f8f54009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c807840faae71dd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:43.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.388+0000 7f8f5f7fe700 1 -- 172.21.15.7:0/3065789040 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54011830 con 0x7f8f6006d870 2023-03-03T15:30:43.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.388+0000 7f8f5f7fe700 1 -- 172.21.15.7:0/3065789040 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54011990 con 0x7f8f6006d870 2023-03-03T15:30:43.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.388+0000 7f8f5f7fe700 1 -- 172.21.15.7:0/3065789040 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54011d20 con 0x7f8f6006d870 2023-03-03T15:30:43.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.389+0000 7f8f66bf6700 1 -- 172.21.15.7:0/3065789040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f6006d870 msgr2=0x7f8f601550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:43.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.389+0000 7f8f66bf6700 1 --2- 172.21.15.7:0/3065789040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f6006d870 0x7f8f601550c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8f540098f0 tx=0x7f8f54009c00 comp rx=0 tx=0).stop 2023-03-03T15:30:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.389+0000 7f8f66bf6700 1 -- 172.21.15.7:0/3065789040 shutdown_connections 2023-03-03T15:30:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.389+0000 7f8f66bf6700 1 --2- 172.21.15.7:0/3065789040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f600c3e30 0x7f8f600c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.389+0000 7f8f66bf6700 1 --2- 172.21.15.7:0/3065789040 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f6006d870 0x7f8f601550c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.389+0000 7f8f66bf6700 1 -- 172.21.15.7:0/3065789040 >> 172.21.15.7:0/3065789040 conn(0x7f8f6006c110 msgr2=0x7f8f6006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:30:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.389+0000 7f8f66bf6700 1 -- 172.21.15.7:0/3065789040 shutdown_connections 2023-03-03T15:30:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.389+0000 7f8f66bf6700 1 -- 172.21.15.7:0/3065789040 wait complete. 2023-03-03T15:30:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.390+0000 7f8f66bf6700 1 Processor -- start 2023-03-03T15:30:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.390+0000 7f8f66bf6700 1 -- start start 2023-03-03T15:30:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f66bf6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f6006d870 0x7f8f600c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f66bf6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f600c1da0 0x7f8f601505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f66bf6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f600c3e30 0x7f8f60146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f66bf6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f60151980 con 0x7f8f600c1da0 2023-03-03T15:30:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f66bf6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f60070660 con 0x7f8f600c3e30 2023-03-03T15:30:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f66bf6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f601513c0 con 0x7f8f6006d870 2023-03-03T15:30:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f5ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f600c1da0 0x7f8f601505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:43.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f5ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f600c1da0 0x7f8f601505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53206/0 (socket says 172.21.15.7:53206) 2023-03-03T15:30:43.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f5ffff700 1 -- 172.21.15.7:0/657901022 learned_addr learned my addr 172.21.15.7:0/657901022 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:30:43.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f65193700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f600c3e30 0x7f8f60146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:43.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f64992700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f6006d870 0x7f8f600c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:43.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f5ffff700 1 -- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f6006d870 msgr2=0x7f8f600c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:43.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f5ffff700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f6006d870 0x7f8f600c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:43.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f5ffff700 1 -- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f600c3e30 msgr2=0x7f8f60146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:43.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f5ffff700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f600c3e30 0x7f8f60146fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.391+0000 7f8f5ffff700 1 -- 172.21.15.7:0/657901022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f58009a70 con 0x7f8f600c1da0 2023-03-03T15:30:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.392+0000 7f8f5ffff700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f600c1da0 0x7f8f601505f0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c00c520 tx=0x7f8f4c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.392+0000 7f8f5d7fa700 1 -- 172.21.15.7:0/657901022 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c005cb0 con 0x7f8f600c1da0 2023-03-03T15:30:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.392+0000 7f8f5d7fa700 1 -- 172.21.15.7:0/657901022 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4c005e10 con 0x7f8f600c1da0 2023-03-03T15:30:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.392+0000 7f8f66bf6700 1 -- 172.21.15.7:0/657901022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f54009550 con 0x7f8f600c1da0 2023-03-03T15:30:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.392+0000 7f8f5d7fa700 1 -- 172.21.15.7:0/657901022 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c0072d0 con 0x7f8f600c1da0 2023-03-03T15:30:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.392+0000 7f8f66bf6700 1 -- 172.21.15.7:0/657901022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f60154ba0 con 0x7f8f600c1da0 2023-03-03T15:30:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.393+0000 7f8f66bf6700 1 -- 172.21.15.7:0/657901022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f601548c0 con 0x7f8f600c1da0 2023-03-03T15:30:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.393+0000 7f8f5d7fa700 1 -- 172.21.15.7:0/657901022 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8f4c019030 con 0x7f8f600c1da0 2023-03-03T15:30:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.394+0000 7f8f5d7fa700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f440682f0 0x7f8f4406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.394+0000 7f8f5d7fa700 1 -- 172.21.15.7:0/657901022 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8f4c0852c0 con 0x7f8f600c1da0 2023-03-03T15:30:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.394+0000 7f8f64992700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f440682f0 0x7f8f4406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.394+0000 7f8f64992700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f440682f0 0x7f8f4406a7a0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009c40 tx=0x7f8f54008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.398+0000 7f8f5d7fa700 1 -- 172.21.15.7:0/657901022 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8f4c04b5e0 con 0x7f8f600c1da0 2023-03-03T15:30:43.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.636+0000 7f8f66bf6700 1 -- 172.21.15.7:0/657901022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f6006dc40 con 0x7f8f600c1da0 2023-03-03T15:30:43.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.637+0000 7f8f5d7fa700 1 -- 172.21.15.7:0/657901022 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8f4c017090 con 0x7f8f600c1da0 2023-03-03T15:30:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:30:43.639 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:30:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 -- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f440682f0 msgr2=0x7f8f4406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f440682f0 0x7f8f4406a7a0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009c40 tx=0x7f8f54008030 comp rx=0 tx=0).stop 2023-03-03T15:30:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 -- 172.21.15.7:0/657901022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f600c1da0 msgr2=0x7f8f601505f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f600c1da0 0x7f8f601505f0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c00c520 tx=0x7f8f4c00c830 comp rx=0 tx=0).stop 2023-03-03T15:30:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 -- 172.21.15.7:0/657901022 shutdown_connections 2023-03-03T15:30:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f600c3e30 0x7f8f60146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f6006d870 0x7f8f600c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f440682f0 0x7f8f4406a7a0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:43.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 --2- 172.21.15.7:0/657901022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f600c1da0 0x7f8f601505f0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:43.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 -- 172.21.15.7:0/657901022 >> 172.21.15.7:0/657901022 conn(0x7f8f6006c110 msgr2=0x7f8f600c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:30:43.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 -- 172.21.15.7:0/657901022 shutdown_connections 2023-03-03T15:30:43.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:43.639+0000 7f8f66bf6700 1 -- 172.21.15.7:0/657901022 wait complete. 2023-03-03T15:30:43.642 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:30:43.652 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before thrashing 2023-03-03T15:30:46.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:30:46.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.790+0000 7f876f44e700 1 Processor -- start 2023-03-03T15:30:46.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.790+0000 7f876f44e700 1 -- start start 2023-03-03T15:30:46.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876f44e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f876806d870 0x7f87681550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:46.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876f44e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f87680c1da0 0x7f87680c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:46.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876f44e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87680c3620 con 0x7f87680c3e30 2023-03-03T15:30:46.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876f44e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87680c2af0 con 0x7f876806d870 2023-03-03T15:30:46.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876f44e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87681545e0 con 0x7f87680c1da0 2023-03-03T15:30:46.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876d1ea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f876806d870 0x7f87681550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:46.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876d1ea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f876806d870 0x7f87681550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53212/0 (socket says 172.21.15.7:53212) 2023-03-03T15:30:46.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876d1ea700 1 -- 172.21.15.7:0/672004119 learned_addr learned my addr 172.21.15.7:0/672004119 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:30:46.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876d1ea700 1 -- 172.21.15.7:0/672004119 >> v2:172.21.15.143:3300/0 conn(0x7f87680c1da0 msgr2=0x7f87680c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:30:46.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876c9e9700 1 --2- 172.21.15.7:0/672004119 >> v2:172.21.15.143:3300/0 conn(0x7f87680c1da0 0x7f87680c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:46.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876d1ea700 1 --2- 172.21.15.7:0/672004119 >> v2:172.21.15.143:3300/0 conn(0x7f87680c1da0 0x7f87680c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:46.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876d1ea700 1 -- 172.21.15.7:0/672004119 >> v1:172.21.15.143:6789/0 conn(0x7f87680c3e30 legacy=0x7f8768157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:30:46.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876d1ea700 1 -- 172.21.15.7:0/672004119 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87681548c0 con 0x7f876806d870 2023-03-03T15:30:46.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.791+0000 7f876d1ea700 1 --2- 172.21.15.7:0/672004119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 0x7f87681550c0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f87680c33a0 tx=0x7f8764009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f155e2eeb8c055 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:46.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f875ffff700 1 -- 172.21.15.7:0/672004119 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764004230 con 0x7f876806d870 2023-03-03T15:30:46.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f875ffff700 1 -- 172.21.15.7:0/672004119 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8764004390 con 0x7f876806d870 2023-03-03T15:30:46.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f875ffff700 1 -- 172.21.15.7:0/672004119 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640036a0 con 0x7f876806d870 2023-03-03T15:30:46.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f876f44e700 1 -- 172.21.15.7:0/672004119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 msgr2=0x7f87681550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:46.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f876f44e700 1 --2- 172.21.15.7:0/672004119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 0x7f87681550c0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f87680c33a0 tx=0x7f8764009bd0 comp rx=0 tx=0).stop 2023-03-03T15:30:46.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f876f44e700 1 -- 172.21.15.7:0/672004119 shutdown_connections 2023-03-03T15:30:46.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f876f44e700 1 --2- 172.21.15.7:0/672004119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 0x7f87681550c0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:46.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f876f44e700 1 --2- 172.21.15.7:0/672004119 >> v2:172.21.15.143:3300/0 conn(0x7f87680c1da0 0x7f87680c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:46.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f876f44e700 1 -- 172.21.15.7:0/672004119 >> 172.21.15.7:0/672004119 conn(0x7f876806c110 msgr2=0x7f876806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:30:46.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f876f44e700 1 -- 172.21.15.7:0/672004119 shutdown_connections 2023-03-03T15:30:46.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.792+0000 7f876f44e700 1 -- 172.21.15.7:0/672004119 wait complete. 2023-03-03T15:30:46.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.793+0000 7f876f44e700 1 Processor -- start 2023-03-03T15:30:46.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876f44e700 1 -- start start 2023-03-03T15:30:46.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876f44e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 0x7f876814c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:46.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876f44e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87680c1da0 0x7f8768146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:46.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876f44e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87680c3e30 0x7f87681469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:46.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876f44e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87681563a0 con 0x7f876806d870 2023-03-03T15:30:46.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876f44e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8768147530 con 0x7f87680c3e30 2023-03-03T15:30:46.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876f44e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87681558c0 con 0x7f87680c1da0 2023-03-03T15:30:46.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876d1ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 0x7f876814c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:46.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876d1ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 0x7f876814c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53218/0 (socket says 172.21.15.7:53218) 2023-03-03T15:30:46.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876d1ea700 1 -- 172.21.15.7:0/161829729 learned_addr learned my addr 172.21.15.7:0/161829729 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:30:46.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876d9eb700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87680c3e30 0x7f87681469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:46.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.794+0000 7f876c9e9700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87680c1da0 0x7f8768146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:46.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f876d1ea700 1 -- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87680c1da0 msgr2=0x7f8768146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:46.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f876d1ea700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87680c1da0 0x7f8768146470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:46.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f876d1ea700 1 -- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87680c3e30 msgr2=0x7f87681469b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:46.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f876d1ea700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87680c3e30 0x7f87681469b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:46.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f876d1ea700 1 -- 172.21.15.7:0/161829729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8764009550 con 0x7f876806d870 2023-03-03T15:30:46.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f876d1ea700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 0x7f876814c640 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f87680c4a90 tx=0x7f8764004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:46.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f875dffb700 1 -- 172.21.15.7:0/161829729 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764003cd0 con 0x7f876806d870 2023-03-03T15:30:46.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f875dffb700 1 -- 172.21.15.7:0/161829729 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876401f070 con 0x7f876806d870 2023-03-03T15:30:46.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f876f44e700 1 -- 172.21.15.7:0/161829729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f876806dc40 con 0x7f876806d870 2023-03-03T15:30:46.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f875dffb700 1 -- 172.21.15.7:0/161829729 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764011870 con 0x7f876806d870 2023-03-03T15:30:46.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.795+0000 7f876f44e700 1 -- 172.21.15.7:0/161829729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8768154ba0 con 0x7f876806d870 2023-03-03T15:30:46.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.796+0000 7f876f44e700 1 -- 172.21.15.7:0/161829729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87680c38e0 con 0x7f876806d870 2023-03-03T15:30:46.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.797+0000 7f875dffb700 1 -- 172.21.15.7:0/161829729 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8764017020 con 0x7f876806d870 2023-03-03T15:30:46.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.797+0000 7f875dffb700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f874c0682f0 0x7f874c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:46.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.797+0000 7f875dffb700 1 -- 172.21.15.7:0/161829729 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8764087900 con 0x7f876806d870 2023-03-03T15:30:46.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.797+0000 7f876c9e9700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f874c0682f0 0x7f874c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:46.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.798+0000 7f876c9e9700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f874c0682f0 0x7f874c06a7a0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f8768153120 tx=0x7f8758008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:46.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:46.801+0000 7f875dffb700 1 -- 172.21.15.7:0/161829729 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8764053ca0 con 0x7f876806d870 2023-03-03T15:30:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.040+0000 7f876f44e700 1 -- 172.21.15.7:0/161829729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8768146ef0 con 0x7f876806d870 2023-03-03T15:30:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.040+0000 7f875dffb700 1 -- 172.21.15.7:0/161829729 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8768146ef0 con 0x7f876806d870 2023-03-03T15:30:47.042 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:30:47.042 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:30:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 -- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f874c0682f0 msgr2=0x7f874c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f874c0682f0 0x7f874c06a7a0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f8768153120 tx=0x7f8758008040 comp rx=0 tx=0).stop 2023-03-03T15:30:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 -- 172.21.15.7:0/161829729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 msgr2=0x7f876814c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 0x7f876814c640 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f87680c4a90 tx=0x7f8764004750 comp rx=0 tx=0).stop 2023-03-03T15:30:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 -- 172.21.15.7:0/161829729 shutdown_connections 2023-03-03T15:30:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87680c3e30 0x7f87681469b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f874c0682f0 0x7f874c06a7a0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876806d870 0x7f876814c640 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 --2- 172.21.15.7:0/161829729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87680c1da0 0x7f8768146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 -- 172.21.15.7:0/161829729 >> 172.21.15.7:0/161829729 conn(0x7f876806c110 msgr2=0x7f876814adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:30:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 -- 172.21.15.7:0/161829729 shutdown_connections 2023-03-03T15:30:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:47.042+0000 7f876f44e700 1 -- 172.21.15.7:0/161829729 wait complete. 2023-03-03T15:30:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:30:47.056 INFO:tasks.mds_thrash.fs.[a]:waiting for 30 secs before thrashing 2023-03-03T15:30:49.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:30:49.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.964+0000 7f0c1a793700 1 Processor -- start 2023-03-03T15:30:49.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.964+0000 7f0c1a793700 1 -- start start 2023-03-03T15:30:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c1a793700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c1406d870 0x7f0c141550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c1a793700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c140c1da0 0x7f0c140c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c1a793700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0c140c3e30 0x7f0c140c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c1a793700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c140c2af0 con 0x7f0c1406d870 2023-03-03T15:30:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c1a793700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c140c3620 con 0x7f0c140c3e30 2023-03-03T15:30:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c1a793700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c141545e0 con 0x7f0c140c1da0 2023-03-03T15:30:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c13fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c1406d870 0x7f0c141550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c13fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0c1406d870 0x7f0c141550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53224/0 (socket says 172.21.15.7:53224) 2023-03-03T15:30:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c13fff700 1 -- 172.21.15.7:0/113254024 learned_addr learned my addr 172.21.15.7:0/113254024 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:30:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c18d30700 1 --2- 172.21.15.7:0/113254024 >> v2:172.21.15.143:3300/0 conn(0x7f0c140c3e30 0x7f0c140c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c137fe700 1 --2- 172.21.15.7:0/113254024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c140c1da0 0x7f0c140c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c13fff700 1 -- 172.21.15.7:0/113254024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c140c1da0 msgr2=0x7f0c140c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:30:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c13fff700 1 --2- 172.21.15.7:0/113254024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c140c1da0 0x7f0c140c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c13fff700 1 -- 172.21.15.7:0/113254024 >> v2:172.21.15.143:3300/0 conn(0x7f0c140c3e30 msgr2=0x7f0c140c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:30:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c13fff700 1 --2- 172.21.15.7:0/113254024 >> v2:172.21.15.143:3300/0 conn(0x7f0c140c3e30 0x7f0c140c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.965+0000 7f0c13fff700 1 -- 172.21.15.7:0/113254024 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c141548c0 con 0x7f0c1406d870 2023-03-03T15:30:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c13fff700 1 --2- 172.21.15.7:0/113254024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c1406d870 0x7f0c141550c0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f0c1406be80 tx=0x7f0c04009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a61cf6d8d6c92c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c12ffd700 1 -- 172.21.15.7:0/113254024 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04011830 con 0x7f0c1406d870 2023-03-03T15:30:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c12ffd700 1 -- 172.21.15.7:0/113254024 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c04011990 con 0x7f0c1406d870 2023-03-03T15:30:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c12ffd700 1 -- 172.21.15.7:0/113254024 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04011d20 con 0x7f0c1406d870 2023-03-03T15:30:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c1a793700 1 -- 172.21.15.7:0/113254024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c1406d870 msgr2=0x7f0c141550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c1a793700 1 --2- 172.21.15.7:0/113254024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c1406d870 0x7f0c141550c0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f0c1406be80 tx=0x7f0c04009bd0 comp rx=0 tx=0).stop 2023-03-03T15:30:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c1a793700 1 -- 172.21.15.7:0/113254024 shutdown_connections 2023-03-03T15:30:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c1a793700 1 --2- 172.21.15.7:0/113254024 >> v2:172.21.15.143:3300/0 conn(0x7f0c140c3e30 0x7f0c140c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:49.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c1a793700 1 --2- 172.21.15.7:0/113254024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c1406d870 0x7f0c141550c0 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:49.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c1a793700 1 --2- 172.21.15.7:0/113254024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c140c1da0 0x7f0c140c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:49.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c1a793700 1 -- 172.21.15.7:0/113254024 >> 172.21.15.7:0/113254024 conn(0x7f0c1406c110 msgr2=0x7f0c1406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:30:49.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c1a793700 1 -- 172.21.15.7:0/113254024 shutdown_connections 2023-03-03T15:30:49.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.966+0000 7f0c1a793700 1 -- 172.21.15.7:0/113254024 wait complete. 2023-03-03T15:30:49.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.968+0000 7f0c1a793700 1 Processor -- start 2023-03-03T15:30:49.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.968+0000 7f0c1a793700 1 -- start start 2023-03-03T15:30:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.968+0000 7f0c1a793700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c1406d870 0x7f0c140aae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.968+0000 7f0c1a793700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c140c1da0 0x7f0c140716c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.968+0000 7f0c1a793700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c140c3e30 0x7f0c14146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c1a793700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140ab3a0 con 0x7f0c140c1da0 2023-03-03T15:30:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c1a793700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c141558a0 con 0x7f0c140c3e30 2023-03-03T15:30:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c1a793700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140ab5d0 con 0x7f0c1406d870 2023-03-03T15:30:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c140c1da0 0x7f0c140716c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c137fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c140c1da0 0x7f0c140716c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53232/0 (socket says 172.21.15.7:53232) 2023-03-03T15:30:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c137fe700 1 -- 172.21.15.7:0/3733079938 learned_addr learned my addr 172.21.15.7:0/3733079938 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:30:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c18d30700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c140c3e30 0x7f0c14146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c13fff700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c1406d870 0x7f0c140aae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c137fe700 1 -- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c1406d870 msgr2=0x7f0c140aae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c137fe700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c1406d870 0x7f0c140aae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c137fe700 1 -- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c140c3e30 msgr2=0x7f0c14146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c137fe700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c140c3e30 0x7f0c14146a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c137fe700 1 -- 172.21.15.7:0/3733079938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c04009550 con 0x7f0c140c1da0 2023-03-03T15:30:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c137fe700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c140c1da0 0x7f0c140716c0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00e930 tx=0x7f0bfc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c10ff9700 1 -- 172.21.15.7:0/3733079938 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc009690 con 0x7f0c140c1da0 2023-03-03T15:30:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c10ff9700 1 -- 172.21.15.7:0/3733079938 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc00c850 con 0x7f0c140c1da0 2023-03-03T15:30:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c1a793700 1 -- 172.21.15.7:0/3733079938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1406dc40 con 0x7f0c140c1da0 2023-03-03T15:30:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.969+0000 7f0c10ff9700 1 -- 172.21.15.7:0/3733079938 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc00cbe0 con 0x7f0c140c1da0 2023-03-03T15:30:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.970+0000 7f0c1a793700 1 -- 172.21.15.7:0/3733079938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c14154ba0 con 0x7f0c140c1da0 2023-03-03T15:30:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.970+0000 7f0c1a793700 1 -- 172.21.15.7:0/3733079938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c140c38e0 con 0x7f0c140c1da0 2023-03-03T15:30:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.971+0000 7f0c10ff9700 1 -- 172.21.15.7:0/3733079938 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0bfc019030 con 0x7f0c140c1da0 2023-03-03T15:30:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.971+0000 7f0c10ff9700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bf4068320 0x7f0bf406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:30:49.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.971+0000 7f0c10ff9700 1 -- 172.21.15.7:0/3733079938 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0bfc086020 con 0x7f0c140c1da0 2023-03-03T15:30:49.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.972+0000 7f0c13fff700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bf4068320 0x7f0bf406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:30:49.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.972+0000 7f0c13fff700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bf4068320 0x7f0bf406a7d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009c10 tx=0x7f0c04008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:30:49.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:49.976+0000 7f0c10ff9700 1 -- 172.21.15.7:0/3733079938 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0bfc0523c0 con 0x7f0c140c1da0 2023-03-03T15:30:50.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.215+0000 7f0c1a793700 1 -- 172.21.15.7:0/3733079938 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c14069730 con 0x7f0c140c1da0 2023-03-03T15:30:50.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.215+0000 7f0c10ff9700 1 -- 172.21.15.7:0/3733079938 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0bfc0559e0 con 0x7f0c140c1da0 2023-03-03T15:30:50.217 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:30:50.217 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:30:50.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 -- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bf4068320 msgr2=0x7f0bf406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:50.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bf4068320 0x7f0bf406a7d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009c10 tx=0x7f0c04008030 comp rx=0 tx=0).stop 2023-03-03T15:30:50.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 -- 172.21.15.7:0/3733079938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c140c1da0 msgr2=0x7f0c140716c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:30:50.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c140c1da0 0x7f0c140716c0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00e930 tx=0x7f0bfc00ecf0 comp rx=0 tx=0).stop 2023-03-03T15:30:50.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 -- 172.21.15.7:0/3733079938 shutdown_connections 2023-03-03T15:30:50.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c140c3e30 0x7f0c14146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:50.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c1406d870 0x7f0c140aae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:50.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bf4068320 0x7f0bf406a7d0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:50.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 --2- 172.21.15.7:0/3733079938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c140c1da0 0x7f0c140716c0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:30:50.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 -- 172.21.15.7:0/3733079938 >> 172.21.15.7:0/3733079938 conn(0x7f0c1406c110 msgr2=0x7f0c1406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:30:50.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 -- 172.21.15.7:0/3733079938 shutdown_connections 2023-03-03T15:30:50.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:30:50.217+0000 7f0c1a793700 1 -- 172.21.15.7:0/3733079938 wait complete. 2023-03-03T15:30:50.221 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:30:50.231 INFO:tasks.mds_thrash.fs.[b]:waiting for 89 secs before thrashing 2023-03-03T15:31:05.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:05.354 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:17.057 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:31:17.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.194+0000 7ff68e9a8700 1 Processor -- start 2023-03-03T15:31:17.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.194+0000 7ff68e9a8700 1 -- start start 2023-03-03T15:31:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff68e9a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68814b020 0x7ff6880717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff68e9a8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff688071d20 0x7ff6881583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff68e9a8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6881588f0 con 0x7ff6880aad20 2023-03-03T15:31:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff68e9a8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6880728b0 con 0x7ff688071d20 2023-03-03T15:31:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff68e9a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff688155070 con 0x7ff68814b020 2023-03-03T15:31:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff6877fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff6880aad20 0x7ff688157ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47226/0 (socket says 172.21.15.7:47226) 2023-03-03T15:31:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff6877fe700 1 -- 172.21.15.7:0/2506337566 learned_addr learned my addr 172.21.15.7:0/2506337566 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:31:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff687fff700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68814b020 0x7ff6880717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff68cf45700 1 --2- 172.21.15.7:0/2506337566 >> v2:172.21.15.143:3300/0 conn(0x7ff688071d20 0x7ff6881583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487788150 0 0) 0x7ff6881588f0 con 0x7ff6880aad20 2023-03-03T15:31:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.195+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6740035b0 con 0x7ff6880aad20 2023-03-03T15:31:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 561976619 0 0) 0x7ff6740035b0 con 0x7ff6880aad20 2023-03-03T15:31:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68814b020 msgr2=0x7ff6880717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:31:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68814b020 0x7ff6880717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 >> v2:172.21.15.143:3300/0 conn(0x7ff688071d20 msgr2=0x7ff6881583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:31:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 --2- 172.21.15.7:0/2506337566 >> v2:172.21.15.143:3300/0 conn(0x7ff688071d20 0x7ff6881583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68806f5b0 con 0x7ff6880aad20 2023-03-03T15:31:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff670001670 con 0x7ff6880aad20 2023-03-03T15:31:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 >> v1:172.21.15.7:6789/0 conn(0x7ff6880aad20 legacy=0x7ff688157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68814b020 0x7ff674003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff674003ff0 0x7ff674004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff674004d90 0x7ff674005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880728b0 con 0x7ff674004d90 2023-03-03T15:31:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff688155070 con 0x7ff674003ff0 2023-03-03T15:31:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67400bb70 con 0x7ff68814b020 2023-03-03T15:31:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff68cf45700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff674004d90 0x7ff674005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff687fff700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68814b020 0x7ff674003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff6877fe700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff674003ff0 0x7ff674004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff68cf45700 1 -- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68814b020 msgr2=0x7ff674003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:31:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff68cf45700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff68814b020 0x7ff674003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff68cf45700 1 -- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff674003ff0 msgr2=0x7ff674004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:31:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff68cf45700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff674003ff0 0x7ff674004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.196+0000 7ff68cf45700 1 -- 172.21.15.7:0/2506337566 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6740035b0 con 0x7ff674004d90 2023-03-03T15:31:17.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff68cf45700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff674004d90 0x7ff674005200 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7ff680009f90 tx=0x7ff674005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6495caffba2f1bc6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:17.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68000ec30 con 0x7ff674004d90 2023-03-03T15:31:17.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68000f040 con 0x7ff674004d90 2023-03-03T15:31:17.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff686ffd700 1 -- 172.21.15.7:0/2506337566 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff680014480 con 0x7ff674004d90 2023-03-03T15:31:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff68e9a8700 1 -- 172.21.15.7:0/2506337566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff674004d90 msgr2=0x7ff674005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff68e9a8700 1 --2- 172.21.15.7:0/2506337566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff674004d90 0x7ff674005200 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7ff680009f90 tx=0x7ff674005880 comp rx=0 tx=0).stop 2023-03-03T15:31:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff68cf45700 1 -- 172.21.15.7:0/2506337566 reap_dead start 2023-03-03T15:31:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff68e9a8700 1 -- 172.21.15.7:0/2506337566 shutdown_connections 2023-03-03T15:31:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff68e9a8700 1 -- 172.21.15.7:0/2506337566 >> 172.21.15.7:0/2506337566 conn(0x7ff6880c8900 msgr2=0x7ff6880c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:31:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff68e9a8700 1 -- 172.21.15.7:0/2506337566 shutdown_connections 2023-03-03T15:31:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.197+0000 7ff68e9a8700 1 -- 172.21.15.7:0/2506337566 wait complete. 2023-03-03T15:31:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.198+0000 7ff68e9a8700 1 Processor -- start 2023-03-03T15:31:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.199+0000 7ff68e9a8700 1 -- start start 2023-03-03T15:31:17.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.199+0000 7ff68e9a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6880c7b70 0x7ff688146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:17.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.199+0000 7ff68e9a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6880c7f40 0x7ff688147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:17.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.199+0000 7ff68e9a8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff688147790 0x7ff68806de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:17.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.199+0000 7ff68e9a8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6881edcb0 con 0x7ff688147790 2023-03-03T15:31:17.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.199+0000 7ff68e9a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6881ece00 con 0x7ff6880c7f40 2023-03-03T15:31:17.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.199+0000 7ff68e9a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6881ed0a0 con 0x7ff6880c7b70 2023-03-03T15:31:17.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.199+0000 7ff68cf45700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff688147790 0x7ff68806de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:17.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.199+0000 7ff68cf45700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff688147790 0x7ff68806de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53260/0 (socket says 172.21.15.7:53260) 2023-03-03T15:31:17.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.199+0000 7ff68cf45700 1 -- 172.21.15.7:0/4283426441 learned_addr learned my addr 172.21.15.7:0/4283426441 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:31:17.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff6877fe700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6880c7f40 0x7ff688147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:17.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff687fff700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6880c7b70 0x7ff688146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:17.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff68cf45700 1 -- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6880c7b70 msgr2=0x7ff688146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:17.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff68cf45700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6880c7b70 0x7ff688146c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:17.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff68cf45700 1 -- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6880c7f40 msgr2=0x7ff688147140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:17.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff68cf45700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6880c7f40 0x7ff688147140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:17.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff68cf45700 1 -- 172.21.15.7:0/4283426441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68806fb70 con 0x7ff688147790 2023-03-03T15:31:17.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff687fff700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6880c7b70 0x7ff688146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:31:17.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff6877fe700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6880c7f40 0x7ff688147140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:31:17.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff68cf45700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff688147790 0x7ff68806de90 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7ff67400bac0 tx=0x7ff680009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:17.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff684ff9700 1 -- 172.21.15.7:0/4283426441 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff680009410 con 0x7ff688147790 2023-03-03T15:31:17.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff684ff9700 1 -- 172.21.15.7:0/4283426441 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff680014b20 con 0x7ff688147790 2023-03-03T15:31:17.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff68e9a8700 1 -- 172.21.15.7:0/4283426441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68806f890 con 0x7ff688147790 2023-03-03T15:31:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff684ff9700 1 -- 172.21.15.7:0/4283426441 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff680003a20 con 0x7ff688147790 2023-03-03T15:31:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.200+0000 7ff68e9a8700 1 -- 172.21.15.7:0/4283426441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff688155350 con 0x7ff688147790 2023-03-03T15:31:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.201+0000 7ff68e9a8700 1 -- 172.21.15.7:0/4283426441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff688074bb0 con 0x7ff688147790 2023-03-03T15:31:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.202+0000 7ff684ff9700 1 -- 172.21.15.7:0/4283426441 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff68002e070 con 0x7ff688147790 2023-03-03T15:31:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.202+0000 7ff684ff9700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6680682f0 0x7ff66806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:31:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.202+0000 7ff684ff9700 1 -- 172.21.15.7:0/4283426441 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff680087960 con 0x7ff688147790 2023-03-03T15:31:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.202+0000 7ff687fff700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6680682f0 0x7ff66806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:31:17.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.203+0000 7ff687fff700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6680682f0 0x7ff66806a7a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7ff678009c60 tx=0x7ff678008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:31:17.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.206+0000 7ff684ff9700 1 -- 172.21.15.7:0/4283426441 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff680053d00 con 0x7ff688147790 2023-03-03T15:31:17.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.446+0000 7ff68e9a8700 1 -- 172.21.15.7:0/4283426441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6880a9e10 con 0x7ff688147790 2023-03-03T15:31:17.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.447+0000 7ff684ff9700 1 -- 172.21.15.7:0/4283426441 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff680057320 con 0x7ff688147790 2023-03-03T15:31:17.449 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:31:17.449 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:31:17.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 -- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6680682f0 msgr2=0x7ff66806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:17.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6680682f0 0x7ff66806a7a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7ff678009c60 tx=0x7ff678008040 comp rx=0 tx=0).stop 2023-03-03T15:31:17.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 -- 172.21.15.7:0/4283426441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff688147790 msgr2=0x7ff68806de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:31:17.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff688147790 0x7ff68806de90 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7ff67400bac0 tx=0x7ff680009750 comp rx=0 tx=0).stop 2023-03-03T15:31:17.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 -- 172.21.15.7:0/4283426441 shutdown_connections 2023-03-03T15:31:17.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff6680682f0 0x7ff66806a7a0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:17.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff688147790 0x7ff68806de90 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:17.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6880c7b70 0x7ff688146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:17.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 --2- 172.21.15.7:0/4283426441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6880c7f40 0x7ff688147140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:31:17.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 -- 172.21.15.7:0/4283426441 >> 172.21.15.7:0/4283426441 conn(0x7ff6880c8900 msgr2=0x7ff688156a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:31:17.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 -- 172.21.15.7:0/4283426441 shutdown_connections 2023-03-03T15:31:17.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:31:17.449+0000 7ff68e9a8700 1 -- 172.21.15.7:0/4283426441 wait complete. 2023-03-03T15:31:17.453 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:31:17.462 INFO:tasks.mds_thrash.fs.[a]:waiting for 108 secs before thrashing 2023-03-03T15:31:35.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:31:35.474 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:05.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:05.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:19.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:32:19.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.369+0000 7f7b6ae29700 1 Processor -- start 2023-03-03T15:32:19.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.370+0000 7f7b6ae29700 1 -- start start 2023-03-03T15:32:19.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.370+0000 7f7b6ae29700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b640aa030 0x7f7b641583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:32:19.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.370+0000 7f7b6ae29700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b641588f0 con 0x7f7b6414b020 2023-03-03T15:32:19.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.370+0000 7f7b6ae29700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b640728b0 con 0x7f7b640aad20 2023-03-03T15:32:19.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.370+0000 7f7b6ae29700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b64155070 con 0x7f7b640aa030 2023-03-03T15:32:19.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.370+0000 7f7b68bc5700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7b6414b020 0x7f7b640717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47278/0 (socket says 172.21.15.7:47278) 2023-03-03T15:32:19.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b68bc5700 1 -- 172.21.15.7:0/3257163721 learned_addr learned my addr 172.21.15.7:0/3257163721 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:32:19.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b693c6700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b640aa030 0x7f7b641583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:32:19.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187164602 0 0) 0x7f7b641588f0 con 0x7f7b6414b020 2023-03-03T15:32:19.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b540035b0 con 0x7f7b6414b020 2023-03-03T15:32:19.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1528439976 0 0) 0x7f7b540035b0 con 0x7f7b6414b020 2023-03-03T15:32:19.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b640aa030 msgr2=0x7f7b641583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:32:19.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b640aa030 0x7f7b641583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:32:19.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 >> v1:172.21.15.143:6789/0 conn(0x7f7b640aad20 legacy=0x7f7b64157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:32:19.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6406f5b0 con 0x7f7b6414b020 2023-03-03T15:32:19.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7b58001670 con 0x7f7b6414b020 2023-03-03T15:32:19.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 >> v1:172.21.15.7:6789/0 conn(0x7f7b6414b020 legacy=0x7f7b640717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:32:19.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b640aa030 0x7f7b54003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:32:19.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b54003fe0 0x7f7b54004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:32:19.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b54004d90 0x7f7b54005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:32:19.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b64155070 con 0x7f7b54004d90 2023-03-03T15:32:19.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5400bba0 con 0x7f7b54003fe0 2023-03-03T15:32:19.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5400bd30 con 0x7f7b640aa030 2023-03-03T15:32:19.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b693c6700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b54004d90 0x7f7b54005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:32:19.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b68bc5700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b640aa030 0x7f7b54003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:32:19.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.371+0000 7f7b63fff700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b54003fe0 0x7f7b54004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:32:19.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b693c6700 1 -- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b640aa030 msgr2=0x7f7b54003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:32:19.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b693c6700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b640aa030 0x7f7b54003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:32:19.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b693c6700 1 -- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b54003fe0 msgr2=0x7f7b54004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:32:19.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b693c6700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b54003fe0 0x7f7b54004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:32:19.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b693c6700 1 -- 172.21.15.7:0/3257163721 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b540035b0 con 0x7f7b54004d90 2023-03-03T15:32:19.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b693c6700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b54004d90 0x7f7b54005200 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009fc0 tx=0x7f7b54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4b6662f933b30c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:32:19.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c00ec30 con 0x7f7b54004d90 2023-03-03T15:32:19.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5c00f040 con 0x7f7b54004d90 2023-03-03T15:32:19.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b637fe700 1 -- 172.21.15.7:0/3257163721 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c014480 con 0x7f7b54004d90 2023-03-03T15:32:19.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3257163721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b54004d90 msgr2=0x7f7b54005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:32:19.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b6ae29700 1 --2- 172.21.15.7:0/3257163721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b54004d90 0x7f7b54005200 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009fc0 tx=0x7f7b54005880 comp rx=0 tx=0).stop 2023-03-03T15:32:19.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.372+0000 7f7b693c6700 1 -- 172.21.15.7:0/3257163721 reap_dead start 2023-03-03T15:32:19.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.373+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3257163721 shutdown_connections 2023-03-03T15:32:19.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.373+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3257163721 >> 172.21.15.7:0/3257163721 conn(0x7f7b640c8900 msgr2=0x7f7b640c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:32:19.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.373+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3257163721 shutdown_connections 2023-03-03T15:32:19.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.373+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3257163721 wait complete. 2023-03-03T15:32:19.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.374+0000 7f7b6ae29700 1 Processor -- start 2023-03-03T15:32:19.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.374+0000 7f7b6ae29700 1 -- start start 2023-03-03T15:32:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.375+0000 7f7b6ae29700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b54004d90 0x7f7b640c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:32:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.375+0000 7f7b6ae29700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b6406c510 0x7f7b64146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:32:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.375+0000 7f7b6ae29700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6414de00 0x7f7b64146580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:32:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.375+0000 7f7b6ae29700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6414c5c0 con 0x7f7b6414de00 2023-03-03T15:32:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.375+0000 7f7b6ae29700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b64146ac0 con 0x7f7b6406c510 2023-03-03T15:32:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.375+0000 7f7b6ae29700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6414b520 con 0x7f7b54004d90 2023-03-03T15:32:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b693c6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6414de00 0x7f7b64146580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:32:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b693c6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6414de00 0x7f7b64146580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53314/0 (socket says 172.21.15.7:53314) 2023-03-03T15:32:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b693c6700 1 -- 172.21.15.7:0/3384159140 learned_addr learned my addr 172.21.15.7:0/3384159140 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:32:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b68bc5700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b54004d90 0x7f7b640c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:32:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b63fff700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b6406c510 0x7f7b64146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:32:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b693c6700 1 -- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b54004d90 msgr2=0x7f7b640c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:32:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b693c6700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b54004d90 0x7f7b640c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:32:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b693c6700 1 -- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b6406c510 msgr2=0x7f7b64146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:32:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b693c6700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b6406c510 0x7f7b64146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:32:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b693c6700 1 -- 172.21.15.7:0/3384159140 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6406fb70 con 0x7f7b6414de00 2023-03-03T15:32:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b693c6700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6414de00 0x7f7b64146580 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f7b54004790 tx=0x7f7b5c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:32:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b617fa700 1 -- 172.21.15.7:0/3384159140 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c014d20 con 0x7f7b6414de00 2023-03-03T15:32:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b617fa700 1 -- 172.21.15.7:0/3384159140 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b5c014e80 con 0x7f7b6414de00 2023-03-03T15:32:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3384159140 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6406f890 con 0x7f7b6414de00 2023-03-03T15:32:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b617fa700 1 -- 172.21.15.7:0/3384159140 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c003c60 con 0x7f7b6414de00 2023-03-03T15:32:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.376+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3384159140 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b64155350 con 0x7f7b6414de00 2023-03-03T15:32:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.377+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3384159140 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b64074bb0 con 0x7f7b6414de00 2023-03-03T15:32:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.378+0000 7f7b617fa700 1 -- 172.21.15.7:0/3384159140 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7b5c01d420 con 0x7f7b6414de00 2023-03-03T15:32:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.378+0000 7f7b617fa700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b480682f0 0x7f7b4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:32:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.378+0000 7f7b617fa700 1 -- 172.21.15.7:0/3384159140 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7b5c087b50 con 0x7f7b6414de00 2023-03-03T15:32:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.378+0000 7f7b68bc5700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b480682f0 0x7f7b4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:32:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.379+0000 7f7b68bc5700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b480682f0 0x7f7b4806a7a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f7b58000c00 tx=0x7f7b58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:32:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.382+0000 7f7b617fa700 1 -- 172.21.15.7:0/3384159140 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7b5c053e90 con 0x7f7b6414de00 2023-03-03T15:32:19.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.620+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3384159140 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b640a9e10 con 0x7f7b6414de00 2023-03-03T15:32:19.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.620+0000 7f7b617fa700 1 -- 172.21.15.7:0/3384159140 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7b5c0574b0 con 0x7f7b6414de00 2023-03-03T15:32:19.622 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:32:19.622 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:32:19.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.622+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b480682f0 msgr2=0x7f7b4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:32:19.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.622+0000 7f7b6ae29700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b480682f0 0x7f7b4806a7a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f7b58000c00 tx=0x7f7b58009040 comp rx=0 tx=0).stop 2023-03-03T15:32:19.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.622+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3384159140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6414de00 msgr2=0x7f7b64146580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:32:19.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.622+0000 7f7b6ae29700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6414de00 0x7f7b64146580 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f7b54004790 tx=0x7f7b5c009750 comp rx=0 tx=0).stop 2023-03-03T15:32:19.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.623+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3384159140 shutdown_connections 2023-03-03T15:32:19.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.623+0000 7f7b6ae29700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b480682f0 0x7f7b4806a7a0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:32:19.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.623+0000 7f7b6ae29700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b6414de00 0x7f7b64146580 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:32:19.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.623+0000 7f7b6ae29700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b54004d90 0x7f7b640c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:32:19.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.623+0000 7f7b6ae29700 1 --2- 172.21.15.7:0/3384159140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b6406c510 0x7f7b64146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:32:19.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.623+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3384159140 >> 172.21.15.7:0/3384159140 conn(0x7f7b640c8900 msgr2=0x7f7b640acda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:32:19.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.623+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3384159140 shutdown_connections 2023-03-03T15:32:19.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:32:19.623+0000 7f7b6ae29700 1 -- 172.21.15.7:0/3384159140 wait complete. 2023-03-03T15:32:19.626 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:32:19.636 INFO:tasks.mds_thrash.fs.[b]:waiting for 99 secs before thrashing 2023-03-03T15:32:35.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:32:35.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:05.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:33:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.602+0000 7fafad34a700 1 Processor -- start 2023-03-03T15:33:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.602+0000 7fafad34a700 1 -- start start 2023-03-03T15:33:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafad34a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fafa80aad20 0x7fafa806a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafad34a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafa806e2f0 0x7fafa806e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafad34a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa806bee0 con 0x7fafa806b9e0 2023-03-03T15:33:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafad34a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa80720a0 con 0x7fafa80aad20 2023-03-03T15:33:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafad34a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80c3570 con 0x7fafa806e2f0 2023-03-03T15:33:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafa67fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fafa806b9e0 0x7fafa806a880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47336/0 (socket says 172.21.15.7:47336) 2023-03-03T15:33:05.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafa67fc700 1 -- 172.21.15.7:0/3525169290 learned_addr learned my addr 172.21.15.7:0/3525169290 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:33:05.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafa6ffd700 1 --2- 172.21.15.7:0/3525169290 >> v2:172.21.15.143:3300/0 conn(0x7fafa80aad20 0x7fafa806a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:05.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747174019 0 0) 0x7fafa806bee0 con 0x7fafa806b9e0 2023-03-03T15:33:05.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafa77fe700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafa806e2f0 0x7fafa806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:05.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.603+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf900035b0 con 0x7fafa806b9e0 2023-03-03T15:33:05.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3611461544 0 0) 0x7faf900035b0 con 0x7fafa806b9e0 2023-03-03T15:33:05.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafa806e2f0 msgr2=0x7fafa806e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:33:05.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafa806e2f0 0x7fafa806e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 >> v2:172.21.15.143:3300/0 conn(0x7fafa80aad20 msgr2=0x7fafa806a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:33:05.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 --2- 172.21.15.7:0/3525169290 >> v2:172.21.15.143:3300/0 conn(0x7fafa80aad20 0x7fafa806a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa81586b0 con 0x7fafa806b9e0 2023-03-03T15:33:05.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7faf98001670 con 0x7fafa806b9e0 2023-03-03T15:33:05.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 >> v1:172.21.15.7:6789/0 conn(0x7fafa806b9e0 legacy=0x7fafa806a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:05.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf90003a70 0x7faf90003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:05.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf90004850 0x7faf90004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:05.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafa806e2f0 0x7faf90005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:05.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80720a0 con 0x7faf90004850 2023-03-03T15:33:05.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9000bac0 con 0x7faf90003a70 2023-03-03T15:33:05.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9000bc80 con 0x7fafa806e2f0 2023-03-03T15:33:05.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa67fc700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf90004850 0x7faf90004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:05.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa6ffd700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf90003a70 0x7faf90003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:05.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa77fe700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafa806e2f0 0x7faf90005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:05.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa67fc700 1 -- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafa806e2f0 msgr2=0x7faf90005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:33:05.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa67fc700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafa806e2f0 0x7faf90005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa67fc700 1 -- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf90003a70 msgr2=0x7faf90003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:33:05.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa67fc700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf90003a70 0x7faf90003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa67fc700 1 -- 172.21.15.7:0/3525169290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf900035b0 con 0x7faf90004850 2023-03-03T15:33:05.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa6ffd700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf90003a70 0x7faf90003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:33:05.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa77fe700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fafa806e2f0 0x7faf90005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:33:05.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa67fc700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf90004850 0x7faf90004cc0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fafa806be10 tx=0x7faf90005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ea6e61e440c4d4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:05.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98007c60 con 0x7faf90004850 2023-03-03T15:33:05.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9800ce80 con 0x7faf90004850 2023-03-03T15:33:05.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.604+0000 7fafa5ffb700 1 -- 172.21.15.7:0/3525169290 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98005ba0 con 0x7faf90004850 2023-03-03T15:33:05.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.605+0000 7fafad34a700 1 -- 172.21.15.7:0/3525169290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf90004850 msgr2=0x7faf90004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:05.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.605+0000 7fafad34a700 1 --2- 172.21.15.7:0/3525169290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf90004850 0x7faf90004cc0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fafa806be10 tx=0x7faf90005880 comp rx=0 tx=0).stop 2023-03-03T15:33:05.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.605+0000 7fafa77fe700 1 -- 172.21.15.7:0/3525169290 reap_dead start 2023-03-03T15:33:05.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.605+0000 7fafad34a700 1 -- 172.21.15.7:0/3525169290 shutdown_connections 2023-03-03T15:33:05.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.605+0000 7fafad34a700 1 -- 172.21.15.7:0/3525169290 >> 172.21.15.7:0/3525169290 conn(0x7fafa80736f0 msgr2=0x7fafa8073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:33:05.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.605+0000 7fafad34a700 1 -- 172.21.15.7:0/3525169290 shutdown_connections 2023-03-03T15:33:05.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.605+0000 7fafad34a700 1 -- 172.21.15.7:0/3525169290 wait complete. 2023-03-03T15:33:05.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.606+0000 7fafad34a700 1 Processor -- start 2023-03-03T15:33:05.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.606+0000 7fafad34a700 1 -- start start 2023-03-03T15:33:05.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafad34a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf90004850 0x7fafa80ac100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:05.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafad34a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafa80bf600 0x7fafa81ed850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:05.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafad34a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa814fa40 0x7fafa81572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:05.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafad34a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa8156e40 con 0x7fafa80bf600 2023-03-03T15:33:05.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafad34a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80ac640 con 0x7fafa814fa40 2023-03-03T15:33:05.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafad34a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa8157e20 con 0x7faf90004850 2023-03-03T15:33:05.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafa67fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafa80bf600 0x7fafa81ed850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:05.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafa67fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafa80bf600 0x7fafa81ed850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53368/0 (socket says 172.21.15.7:53368) 2023-03-03T15:33:05.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafa67fc700 1 -- 172.21.15.7:0/252638455 learned_addr learned my addr 172.21.15.7:0/252638455 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:33:05.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafa77fe700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa814fa40 0x7fafa81572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:05.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.607+0000 7fafa6ffd700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf90004850 0x7fafa80ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:05.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7fafa67fc700 1 -- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf90004850 msgr2=0x7fafa80ac100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:05.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7fafa67fc700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf90004850 0x7fafa80ac100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7fafa67fc700 1 -- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa814fa40 msgr2=0x7fafa81572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:05.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7fafa67fc700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa814fa40 0x7fafa81572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7fafa67fc700 1 -- 172.21.15.7:0/252638455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa8071cc0 con 0x7fafa80bf600 2023-03-03T15:33:05.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7fafa67fc700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafa80bf600 0x7fafa81ed850 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7faf98004b50 tx=0x7faf98000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:05.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7faf8f7fe700 1 -- 172.21.15.7:0/252638455 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf980070d0 con 0x7fafa80bf600 2023-03-03T15:33:05.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7faf8f7fe700 1 -- 172.21.15.7:0/252638455 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf98007230 con 0x7fafa80bf600 2023-03-03T15:33:05.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7fafad34a700 1 -- 172.21.15.7:0/252638455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa8158990 con 0x7fafa80bf600 2023-03-03T15:33:05.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7faf8f7fe700 1 -- 172.21.15.7:0/252638455 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf980075e0 con 0x7fafa80bf600 2023-03-03T15:33:05.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.608+0000 7fafad34a700 1 -- 172.21.15.7:0/252638455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa806af90 con 0x7fafa80bf600 2023-03-03T15:33:05.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.609+0000 7fafad34a700 1 -- 172.21.15.7:0/252638455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa8156ff0 con 0x7fafa80bf600 2023-03-03T15:33:05.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.609+0000 7faf8f7fe700 1 -- 172.21.15.7:0/252638455 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faf98024030 con 0x7fafa80bf600 2023-03-03T15:33:05.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.610+0000 7faf8f7fe700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf940682f0 0x7faf9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:05.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.610+0000 7faf8f7fe700 1 -- 172.21.15.7:0/252638455 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faf980873e0 con 0x7fafa80bf600 2023-03-03T15:33:05.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.610+0000 7fafa6ffd700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf940682f0 0x7faf9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:05.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.611+0000 7fafa6ffd700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf940682f0 0x7faf9406a7a0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009bc0 tx=0x7faf9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:05.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.614+0000 7faf8f7fe700 1 -- 172.21.15.7:0/252638455 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fafa8156ff0 con 0x7fafa80bf600 2023-03-03T15:33:05.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.849+0000 7fafad34a700 1 -- 172.21.15.7:0/252638455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa80bf3e0 con 0x7fafa80bf600 2023-03-03T15:33:05.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.850+0000 7faf8f7fe700 1 -- 172.21.15.7:0/252638455 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faf98053780 con 0x7fafa80bf600 2023-03-03T15:33:05.852 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:33:05.852 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:33:05.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 -- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf940682f0 msgr2=0x7faf9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:05.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf940682f0 0x7faf9406a7a0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009bc0 tx=0x7faf9c008040 comp rx=0 tx=0).stop 2023-03-03T15:33:05.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 -- 172.21.15.7:0/252638455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafa80bf600 msgr2=0x7fafa81ed850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:05.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafa80bf600 0x7fafa81ed850 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7faf98004b50 tx=0x7faf98000fe0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 -- 172.21.15.7:0/252638455 shutdown_connections 2023-03-03T15:33:05.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fafa814fa40 0x7fafa81572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf90004850 0x7fafa80ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf940682f0 0x7faf9406a7a0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 --2- 172.21.15.7:0/252638455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fafa80bf600 0x7fafa81ed850 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 -- 172.21.15.7:0/252638455 >> 172.21.15.7:0/252638455 conn(0x7fafa80736f0 msgr2=0x7fafa806e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:33:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 -- 172.21.15.7:0/252638455 shutdown_connections 2023-03-03T15:33:05.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:05.852+0000 7fafad34a700 1 -- 172.21.15.7:0/252638455 wait complete. 2023-03-03T15:33:05.856 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:33:05.866 INFO:tasks.mds_thrash.fs.[a]:waiting for 52 secs before thrashing 2023-03-03T15:33:05.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:05.888 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:36.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:36.212 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:33:57.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:33:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.005+0000 7efea0405700 1 Processor -- start 2023-03-03T15:33:58.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.005+0000 7efea0405700 1 -- start start 2023-03-03T15:33:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.005+0000 7efea0405700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efe98151b20 0x7efe98071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.005+0000 7efea0405700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe98158710 0x7efe9815ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.005+0000 7efea0405700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe98151ef0 con 0x7efe98154f10 2023-03-03T15:33:58.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efea0405700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe98157b20 con 0x7efe98151b20 2023-03-03T15:33:58.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efea0405700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe98155410 con 0x7efe98158710 2023-03-03T15:33:58.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9e1a1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efe98151b20 0x7efe98071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9e1a1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efe98151b20 0x7efe98071820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53420/0 (socket says 172.21.15.7:53420) 2023-03-03T15:33:58.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9e1a1700 1 -- 172.21.15.7:0/2599067157 learned_addr learned my addr 172.21.15.7:0/2599067157 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:33:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9e9a2700 1 --2- 172.21.15.7:0/2599067157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe98158710 0x7efe9815ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9e1a1700 1 -- 172.21.15.7:0/2599067157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe98158710 msgr2=0x7efe9815ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:33:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9e1a1700 1 --2- 172.21.15.7:0/2599067157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe98158710 0x7efe9815ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9e1a1700 1 -- 172.21.15.7:0/2599067157 >> v1:172.21.15.143:6789/0 conn(0x7efe98154f10 legacy=0x7efe98158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9e1a1700 1 -- 172.21.15.7:0/2599067157 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9806f640 con 0x7efe98151b20 2023-03-03T15:33:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9e1a1700 1 --2- 172.21.15.7:0/2599067157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 0x7efe98071820 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7efe940098f0 tx=0x7efe94009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b094d55536bdc4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9d19f700 1 -- 172.21.15.7:0/2599067157 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94011830 con 0x7efe98151b20 2023-03-03T15:33:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9d19f700 1 -- 172.21.15.7:0/2599067157 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe94011990 con 0x7efe98151b20 2023-03-03T15:33:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.006+0000 7efe9d19f700 1 -- 172.21.15.7:0/2599067157 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94011d20 con 0x7efe98151b20 2023-03-03T15:33:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.007+0000 7efea0405700 1 -- 172.21.15.7:0/2599067157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 msgr2=0x7efe98071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.007+0000 7efea0405700 1 --2- 172.21.15.7:0/2599067157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 0x7efe98071820 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7efe940098f0 tx=0x7efe94009c00 comp rx=0 tx=0).stop 2023-03-03T15:33:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.007+0000 7efea0405700 1 -- 172.21.15.7:0/2599067157 shutdown_connections 2023-03-03T15:33:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.007+0000 7efea0405700 1 --2- 172.21.15.7:0/2599067157 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe98158710 0x7efe9815ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.007+0000 7efea0405700 1 --2- 172.21.15.7:0/2599067157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 0x7efe98071820 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.007+0000 7efea0405700 1 -- 172.21.15.7:0/2599067157 >> 172.21.15.7:0/2599067157 conn(0x7efe980c8900 msgr2=0x7efe980c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:33:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.007+0000 7efea0405700 1 -- 172.21.15.7:0/2599067157 shutdown_connections 2023-03-03T15:33:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.007+0000 7efea0405700 1 -- 172.21.15.7:0/2599067157 wait complete. 2023-03-03T15:33:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.008+0000 7efea0405700 1 Processor -- start 2023-03-03T15:33:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.008+0000 7efea0405700 1 -- start start 2023-03-03T15:33:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efea0405700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 0x7efe980c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efea0405700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe98154f10 0x7efe9813fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efea0405700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe98158710 0x7efe98140360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efea0405700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe98153470 con 0x7efe98151b20 2023-03-03T15:33:58.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efea0405700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe981408a0 con 0x7efe98154f10 2023-03-03T15:33:58.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efea0405700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe98152c60 con 0x7efe98158710 2023-03-03T15:33:58.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efe9e1a1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 0x7efe980c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efe9e1a1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 0x7efe980c7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53426/0 (socket says 172.21.15.7:53426) 2023-03-03T15:33:58.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efe9e1a1700 1 -- 172.21.15.7:0/654334069 learned_addr learned my addr 172.21.15.7:0/654334069 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:33:58.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efe9d9a0700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe98154f10 0x7efe9813fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efe9e9a2700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe98158710 0x7efe98140360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efe9e1a1700 1 -- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe98158710 msgr2=0x7efe98140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:58.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efe9e1a1700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe98158710 0x7efe98140360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efe9e1a1700 1 -- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe98154f10 msgr2=0x7efe9813fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:58.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efe9e1a1700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe98154f10 0x7efe9813fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.009+0000 7efe9e1a1700 1 -- 172.21.15.7:0/654334069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe94009550 con 0x7efe98151b20 2023-03-03T15:33:58.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.010+0000 7efe9e1a1700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 0x7efe980c7700 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7efe94009c40 tx=0x7efe94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:58.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.010+0000 7efe8effd700 1 -- 172.21.15.7:0/654334069 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe94004510 con 0x7efe98151b20 2023-03-03T15:33:58.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.010+0000 7efe8effd700 1 -- 172.21.15.7:0/654334069 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9401e070 con 0x7efe98151b20 2023-03-03T15:33:58.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.010+0000 7efea0405700 1 -- 172.21.15.7:0/654334069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9806fc00 con 0x7efe98151b20 2023-03-03T15:33:58.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.010+0000 7efe8effd700 1 -- 172.21.15.7:0/654334069 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe940038e0 con 0x7efe98151b20 2023-03-03T15:33:58.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.010+0000 7efea0405700 1 -- 172.21.15.7:0/654334069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9806f920 con 0x7efe98151b20 2023-03-03T15:33:58.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.011+0000 7efea0405700 1 -- 172.21.15.7:0/654334069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe98153d00 con 0x7efe98151b20 2023-03-03T15:33:58.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.011+0000 7efe8effd700 1 -- 172.21.15.7:0/654334069 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efe9400f050 con 0x7efe98151b20 2023-03-03T15:33:58.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.012+0000 7efe8effd700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe7c0682f0 0x7efe7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.012+0000 7efe8effd700 1 -- 172.21.15.7:0/654334069 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efe94086ff0 con 0x7efe98151b20 2023-03-03T15:33:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.012+0000 7efe9d9a0700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe7c0682f0 0x7efe7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.013+0000 7efe9d9a0700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe7c0682f0 0x7efe7c06a7a0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7efe98144210 tx=0x7efe88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.016+0000 7efe8effd700 1 -- 172.21.15.7:0/654334069 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efe94053310 con 0x7efe98151b20 2023-03-03T15:33:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.255+0000 7efea0405700 1 -- 172.21.15.7:0/654334069 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe981537f0 con 0x7efe98151b20 2023-03-03T15:33:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.256+0000 7efe8effd700 1 -- 172.21.15.7:0/654334069 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efe981537f0 con 0x7efe98151b20 2023-03-03T15:33:58.257 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:33:58.258 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:33:58.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 -- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe7c0682f0 msgr2=0x7efe7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe7c0682f0 0x7efe7c06a7a0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7efe98144210 tx=0x7efe88009040 comp rx=0 tx=0).stop 2023-03-03T15:33:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 -- 172.21.15.7:0/654334069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 msgr2=0x7efe980c7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 0x7efe980c7700 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7efe94009c40 tx=0x7efe94004750 comp rx=0 tx=0).stop 2023-03-03T15:33:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 -- 172.21.15.7:0/654334069 shutdown_connections 2023-03-03T15:33:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efe98158710 0x7efe98140360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efe7c0682f0 0x7efe7c06a7a0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe98151b20 0x7efe980c7700 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 --2- 172.21.15.7:0/654334069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe98154f10 0x7efe9813fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 -- 172.21.15.7:0/654334069 >> 172.21.15.7:0/654334069 conn(0x7efe980c8900 msgr2=0x7efe980c1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:33:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 -- 172.21.15.7:0/654334069 shutdown_connections 2023-03-03T15:33:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.258+0000 7efea0405700 1 -- 172.21.15.7:0/654334069 wait complete. 2023-03-03T15:33:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:33:58.271 INFO:tasks.mds_thrash.fs.[a]:waiting for 72 secs before thrashing 2023-03-03T15:33:58.637 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:33:58.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.764+0000 7f2a2cf69700 1 Processor -- start 2023-03-03T15:33:58.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.764+0000 7f2a2cf69700 1 -- start start 2023-03-03T15:33:58.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2cf69700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a2806d4b0 0x7f2a2806d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2cf69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a280c1da0 0x7f2a28139d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2cf69700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a280c7b90 con 0x7f2a280c8020 2023-03-03T15:33:58.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2cf69700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a280c7330 con 0x7f2a2806d4b0 2023-03-03T15:33:58.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2cf69700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a28154420 con 0x7f2a280c1da0 2023-03-03T15:33:58.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a2806d4b0 0x7f2a2806d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a25d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2a280c8020 0x7f2a280c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47414/0 (socket says 172.21.15.7:47414) 2023-03-03T15:33:58.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a25d9b700 1 -- 172.21.15.7:0/3563714156 learned_addr learned my addr 172.21.15.7:0/3563714156 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:33:58.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a26d9d700 1 --2- 172.21.15.7:0/3563714156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a280c1da0 0x7f2a28139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2659c700 1 -- 172.21.15.7:0/3563714156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a280c1da0 msgr2=0x7f2a28139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:33:58.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2659c700 1 --2- 172.21.15.7:0/3563714156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a280c1da0 0x7f2a28139d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2659c700 1 -- 172.21.15.7:0/3563714156 >> v1:172.21.15.7:6789/0 conn(0x7f2a280c8020 legacy=0x7f2a280c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:58.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2659c700 1 -- 172.21.15.7:0/3563714156 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28154640 con 0x7f2a2806d4b0 2023-03-03T15:33:58.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2559a700 1 -- 172.21.15.7:0/3563714156 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990534040 0 0) 0x7f2a280c7b90 con 0x7f2a280c8020 2023-03-03T15:33:58.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2659c700 1 --2- 172.21.15.7:0/3563714156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2806d4b0 0x7f2a2806d880 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009840 tx=0x7f2a18009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5e1df89c8cdccb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:58.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2559a700 1 -- 172.21.15.7:0/3563714156 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18011830 con 0x7f2a2806d4b0 2023-03-03T15:33:58.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2559a700 1 -- 172.21.15.7:0/3563714156 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a18011990 con 0x7f2a2806d4b0 2023-03-03T15:33:58.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.765+0000 7f2a2559a700 1 -- 172.21.15.7:0/3563714156 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18011d20 con 0x7f2a2806d4b0 2023-03-03T15:33:58.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.766+0000 7f2a2cf69700 1 -- 172.21.15.7:0/3563714156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2806d4b0 msgr2=0x7f2a2806d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:58.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.766+0000 7f2a2cf69700 1 --2- 172.21.15.7:0/3563714156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2806d4b0 0x7f2a2806d880 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009840 tx=0x7f2a18009b50 comp rx=0 tx=0).stop 2023-03-03T15:33:58.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.766+0000 7f2a2cf69700 1 -- 172.21.15.7:0/3563714156 shutdown_connections 2023-03-03T15:33:58.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.766+0000 7f2a2cf69700 1 --2- 172.21.15.7:0/3563714156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a280c1da0 0x7f2a28139d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.766+0000 7f2a2cf69700 1 --2- 172.21.15.7:0/3563714156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2806d4b0 0x7f2a2806d880 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.766+0000 7f2a2cf69700 1 -- 172.21.15.7:0/3563714156 >> 172.21.15.7:0/3563714156 conn(0x7f2a2806c110 msgr2=0x7f2a2806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:33:58.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.766+0000 7f2a2cf69700 1 -- 172.21.15.7:0/3563714156 shutdown_connections 2023-03-03T15:33:58.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.766+0000 7f2a2cf69700 1 -- 172.21.15.7:0/3563714156 wait complete. 2023-03-03T15:33:58.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.767+0000 7f2a2cf69700 1 Processor -- start 2023-03-03T15:33:58.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.767+0000 7f2a2cf69700 1 -- start start 2023-03-03T15:33:58.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a2cf69700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a2806d4b0 0x7f2a280aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a2cf69700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a280c1da0 0x7f2a280714f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a2cf69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a280c8020 0x7f2a281db010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a2cf69700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a280abd50 con 0x7f2a280c1da0 2023-03-03T15:33:58.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a2cf69700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a28153a30 con 0x7f2a2806d4b0 2023-03-03T15:33:58.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a2cf69700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a281da7d0 con 0x7f2a280c8020 2023-03-03T15:33:58.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a25d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a280c1da0 0x7f2a280714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a25d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a280c1da0 0x7f2a280714f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53442/0 (socket says 172.21.15.7:53442) 2023-03-03T15:33:58.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a25d9b700 1 -- 172.21.15.7:0/2339446354 learned_addr learned my addr 172.21.15.7:0/2339446354 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:33:58.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a26d9d700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a280c8020 0x7f2a281db010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a2659c700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a2806d4b0 0x7f2a280aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.768+0000 7f2a25d9b700 1 -- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a280c8020 msgr2=0x7f2a281db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:58.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a25d9b700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a280c8020 0x7f2a281db010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a25d9b700 1 -- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a2806d4b0 msgr2=0x7f2a280aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:58.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a25d9b700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a2806d4b0 0x7f2a280aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:58.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a25d9b700 1 -- 172.21.15.7:0/2339446354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180094a0 con 0x7f2a280c1da0 2023-03-03T15:33:58.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a2659c700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a2806d4b0 0x7f2a280aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:33:58.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a25d9b700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a280c1da0 0x7f2a280714f0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00c550 tx=0x7f2a1c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:58.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a16ffd700 1 -- 172.21.15.7:0/2339446354 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c004c80 con 0x7f2a280c1da0 2023-03-03T15:33:58.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a16ffd700 1 -- 172.21.15.7:0/2339446354 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1c004e00 con 0x7f2a280c1da0 2023-03-03T15:33:58.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a2cf69700 1 -- 172.21.15.7:0/2339446354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a280c7820 con 0x7f2a280c1da0 2023-03-03T15:33:58.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a16ffd700 1 -- 172.21.15.7:0/2339446354 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c007310 con 0x7f2a280c1da0 2023-03-03T15:33:58.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.769+0000 7f2a2cf69700 1 -- 172.21.15.7:0/2339446354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a280c7540 con 0x7f2a280c1da0 2023-03-03T15:33:58.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.770+0000 7f2a2cf69700 1 -- 172.21.15.7:0/2339446354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2806ddc0 con 0x7f2a280c1da0 2023-03-03T15:33:58.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.770+0000 7f2a16ffd700 1 -- 172.21.15.7:0/2339446354 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2a1c01a020 con 0x7f2a280c1da0 2023-03-03T15:33:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.771+0000 7f2a16ffd700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a080682f0 0x7f2a0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:33:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.771+0000 7f2a16ffd700 1 -- 172.21.15.7:0/2339446354 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2a1c085d10 con 0x7f2a280c1da0 2023-03-03T15:33:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.771+0000 7f2a2659c700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a080682f0 0x7f2a0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:33:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.771+0000 7f2a2659c700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a080682f0 0x7f2a0806a7a0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f2a180090b0 tx=0x7f2a1801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:33:58.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:58.775+0000 7f2a16ffd700 1 -- 172.21.15.7:0/2339446354 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2a1c052160 con 0x7f2a280c1da0 2023-03-03T15:33:59.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.015+0000 7f2a2cf69700 1 -- 172.21.15.7:0/2339446354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a280c2eb0 con 0x7f2a280c1da0 2023-03-03T15:33:59.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.016+0000 7f2a16ffd700 1 -- 172.21.15.7:0/2339446354 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2a1c055780 con 0x7f2a280c1da0 2023-03-03T15:33:59.017 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:33:59.018 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:33:59.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 -- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a080682f0 msgr2=0x7f2a0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a080682f0 0x7f2a0806a7a0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f2a180090b0 tx=0x7f2a1801c040 comp rx=0 tx=0).stop 2023-03-03T15:33:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 -- 172.21.15.7:0/2339446354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a280c1da0 msgr2=0x7f2a280714f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:33:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a280c1da0 0x7f2a280714f0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00c550 tx=0x7f2a1c00c910 comp rx=0 tx=0).stop 2023-03-03T15:33:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 -- 172.21.15.7:0/2339446354 shutdown_connections 2023-03-03T15:33:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a280c8020 0x7f2a281db010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a2806d4b0 0x7f2a280aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a080682f0 0x7f2a0806a7a0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:59.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 --2- 172.21.15.7:0/2339446354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a280c1da0 0x7f2a280714f0 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:33:59.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 -- 172.21.15.7:0/2339446354 >> 172.21.15.7:0/2339446354 conn(0x7f2a2806c110 msgr2=0x7f2a280c40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:33:59.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 -- 172.21.15.7:0/2339446354 shutdown_connections 2023-03-03T15:33:59.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:33:59.018+0000 7f2a2cf69700 1 -- 172.21.15.7:0/2339446354 wait complete. 2023-03-03T15:33:59.021 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:33:59.031 INFO:tasks.mds_thrash.fs.[b]:waiting for 46 secs before thrashing 2023-03-03T15:34:06.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:06.274 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:36.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:36.356 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:34:45.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:34:45.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.171+0000 7f44f6639700 1 Processor -- start 2023-03-03T15:34:45.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.171+0000 7f44f6639700 1 -- start start 2023-03-03T15:34:45.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.172+0000 7f44f6639700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44f0155390 0x7f44f00c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.172+0000 7f44f6639700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f0154910 con 0x7f44f006e020 2023-03-03T15:34:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.172+0000 7f44f6639700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f01522e0 con 0x7f44f0151770 2023-03-03T15:34:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.172+0000 7f44f6639700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f0070b90 con 0x7f44f0155390 2023-03-03T15:34:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.172+0000 7f44effff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44f0155390 0x7f44f00c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.172+0000 7f44ef7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f44f006e020 0x7f44f0071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47454/0 (socket says 172.21.15.7:47454) 2023-03-03T15:34:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.172+0000 7f44effff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44f0155390 0x7f44f00c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53474/0 (socket says 172.21.15.7:53474) 2023-03-03T15:34:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.172+0000 7f44ef7fe700 1 -- 172.21.15.7:0/3739113323 learned_addr learned my addr 172.21.15.7:0/3739113323 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:34:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44effff700 1 -- 172.21.15.7:0/3739113323 >> v1:172.21.15.143:6789/0 conn(0x7f44f0151770 legacy=0x7f44f0151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:34:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44effff700 1 -- 172.21.15.7:0/3739113323 >> v1:172.21.15.7:6789/0 conn(0x7f44f006e020 legacy=0x7f44f0071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44eeffd700 1 -- 172.21.15.7:0/3739113323 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349381104 0 0) 0x7f44f0154910 con 0x7f44f006e020 2023-03-03T15:34:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44effff700 1 -- 172.21.15.7:0/3739113323 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00702a0 con 0x7f44f0155390 2023-03-03T15:34:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44effff700 1 --2- 172.21.15.7:0/3739113323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0155390 0x7f44f00c3e30 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009840 tx=0x7f44e0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0483790898b035 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:34:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44eeffd700 1 -- 172.21.15.7:0/3739113323 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0004230 con 0x7f44f0155390 2023-03-03T15:34:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44eeffd700 1 -- 172.21.15.7:0/3739113323 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e0004390 con 0x7f44f0155390 2023-03-03T15:34:45.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44eeffd700 1 -- 172.21.15.7:0/3739113323 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e00036a0 con 0x7f44f0155390 2023-03-03T15:34:45.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44f6639700 1 -- 172.21.15.7:0/3739113323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0155390 msgr2=0x7f44f00c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:45.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44f6639700 1 --2- 172.21.15.7:0/3739113323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0155390 0x7f44f00c3e30 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009840 tx=0x7f44e0009b50 comp rx=0 tx=0).stop 2023-03-03T15:34:45.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44f6639700 1 -- 172.21.15.7:0/3739113323 shutdown_connections 2023-03-03T15:34:45.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.173+0000 7f44f6639700 1 --2- 172.21.15.7:0/3739113323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0155390 0x7f44f00c3e30 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.174+0000 7f44f6639700 1 -- 172.21.15.7:0/3739113323 >> 172.21.15.7:0/3739113323 conn(0x7f44f006c110 msgr2=0x7f44f006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:34:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.174+0000 7f44f6639700 1 -- 172.21.15.7:0/3739113323 shutdown_connections 2023-03-03T15:34:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.174+0000 7f44f6639700 1 -- 172.21.15.7:0/3739113323 wait complete. 2023-03-03T15:34:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.175+0000 7f44f6639700 1 Processor -- start 2023-03-03T15:34:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.175+0000 7f44f6639700 1 -- start start 2023-03-03T15:34:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44f6639700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f006e020 0x7f44f0151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44f6639700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0151770 0x7f44f00ac360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44f6639700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f0155390 0x7f44f00721e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44f6639700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00ab6d0 con 0x7f44f0151770 2023-03-03T15:34:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44f6639700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00ab150 con 0x7f44f006e020 2023-03-03T15:34:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44f6639700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00bc3c0 con 0x7f44f0155390 2023-03-03T15:34:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44ef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0151770 0x7f44f00ac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44ef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0151770 0x7f44f00ac360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53480/0 (socket says 172.21.15.7:53480) 2023-03-03T15:34:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44ef7fe700 1 -- 172.21.15.7:0/1750363481 learned_addr learned my addr 172.21.15.7:0/1750363481 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:34:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44effff700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f006e020 0x7f44f0151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44ef7fe700 1 -- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f0155390 msgr2=0x7f44f00721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44f4bd6700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f0155390 0x7f44f00721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44ef7fe700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f0155390 0x7f44f00721e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44ef7fe700 1 -- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f006e020 msgr2=0x7f44f0151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44ef7fe700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f006e020 0x7f44f0151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:45.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44ef7fe700 1 -- 172.21.15.7:0/1750363481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e00094a0 con 0x7f44f0151770 2023-03-03T15:34:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44effff700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f006e020 0x7f44f0151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:34:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44ef7fe700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0151770 0x7f44f00ac360 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f44e40011e0 tx=0x7f44e400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:34:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.176+0000 7f44ecff9700 1 -- 172.21.15.7:0/1750363481 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4004c80 con 0x7f44f0151770 2023-03-03T15:34:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.177+0000 7f44ecff9700 1 -- 172.21.15.7:0/1750363481 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e4004e00 con 0x7f44f0151770 2023-03-03T15:34:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.177+0000 7f44ecff9700 1 -- 172.21.15.7:0/1750363481 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4007310 con 0x7f44f0151770 2023-03-03T15:34:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.177+0000 7f44f6639700 1 -- 172.21.15.7:0/1750363481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f0070820 con 0x7f44f0151770 2023-03-03T15:34:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.177+0000 7f44f6639700 1 -- 172.21.15.7:0/1750363481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f0070540 con 0x7f44f0151770 2023-03-03T15:34:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.178+0000 7f44f6639700 1 -- 172.21.15.7:0/1750363481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f0155760 con 0x7f44f0151770 2023-03-03T15:34:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.178+0000 7f44ecff9700 1 -- 172.21.15.7:0/1750363481 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f44e401a020 con 0x7f44f0151770 2023-03-03T15:34:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.178+0000 7f44ecff9700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44dc0682f0 0x7f44dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:34:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.179+0000 7f44ecff9700 1 -- 172.21.15.7:0/1750363481 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f44e4085d90 con 0x7f44f0151770 2023-03-03T15:34:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.179+0000 7f44effff700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44dc0682f0 0x7f44dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:34:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.179+0000 7f44effff700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44dc0682f0 0x7f44dc06a7a0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f44f00c2310 tx=0x7f44e0013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:34:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.183+0000 7f44ecff9700 1 -- 172.21.15.7:0/1750363481 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f44e40521e0 con 0x7f44f0151770 2023-03-03T15:34:45.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.419+0000 7f44f6639700 1 -- 172.21.15.7:0/1750363481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44f0074bb0 con 0x7f44f0151770 2023-03-03T15:34:45.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.419+0000 7f44ecff9700 1 -- 172.21.15.7:0/1750363481 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f44e4055800 con 0x7f44f0151770 2023-03-03T15:34:45.422 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:34:45.422 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:34:45.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 -- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44dc0682f0 msgr2=0x7f44dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:45.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44dc0682f0 0x7f44dc06a7a0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f44f00c2310 tx=0x7f44e0013040 comp rx=0 tx=0).stop 2023-03-03T15:34:45.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 -- 172.21.15.7:0/1750363481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0151770 msgr2=0x7f44f00ac360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:34:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0151770 0x7f44f00ac360 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f44e40011e0 tx=0x7f44e400cae0 comp rx=0 tx=0).stop 2023-03-03T15:34:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 -- 172.21.15.7:0/1750363481 shutdown_connections 2023-03-03T15:34:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44f0155390 0x7f44f00721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44f006e020 0x7f44f0151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44dc0682f0 0x7f44dc06a7a0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 --2- 172.21.15.7:0/1750363481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44f0151770 0x7f44f00ac360 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:34:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 -- 172.21.15.7:0/1750363481 >> 172.21.15.7:0/1750363481 conn(0x7f44f006c110 msgr2=0x7f44f00736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:34:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.421+0000 7f44f6639700 1 -- 172.21.15.7:0/1750363481 shutdown_connections 2023-03-03T15:34:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:34:45.422+0000 7f44f6639700 1 -- 172.21.15.7:0/1750363481 wait complete. 2023-03-03T15:34:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:34:45.435 INFO:tasks.mds_thrash.fs.[b]:waiting for 92 secs before thrashing 2023-03-03T15:35:06.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:06.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:10.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:35:10.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.409+0000 7f4f5d52d700 1 Processor -- start 2023-03-03T15:35:10.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.409+0000 7f4f5d52d700 1 -- start start 2023-03-03T15:35:10.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f5d52d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4f580c00e0 0x7f4f580c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:10.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f5d52d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f580c7310 con 0x7f4f58154ff0 2023-03-03T15:35:10.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f5d52d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f580bfd10 con 0x7f4f580caa30 2023-03-03T15:35:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f5d52d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f58155430 con 0x7f4f580c00e0 2023-03-03T15:35:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f567fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4f580c00e0 0x7f4f580c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f577fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4f58154ff0 0x7f4f5815ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47490/0 (socket says 172.21.15.7:47490) 2023-03-03T15:35:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f577fe700 1 -- 172.21.15.7:0/990798285 learned_addr learned my addr 172.21.15.7:0/990798285 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:35:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f567fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4f580c00e0 0x7f4f580c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53508/0 (socket says 172.21.15.7:53508) 2023-03-03T15:35:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f567fc700 1 -- 172.21.15.7:0/990798285 >> v1:172.21.15.143:6789/0 conn(0x7f4f580caa30 legacy=0x7f4f580c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:35:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f567fc700 1 -- 172.21.15.7:0/990798285 >> v1:172.21.15.7:6789/0 conn(0x7f4f58154ff0 legacy=0x7f4f5815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f567fc700 1 -- 172.21.15.7:0/990798285 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f580bfa30 con 0x7f4f580c00e0 2023-03-03T15:35:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f55ffb700 1 -- 172.21.15.7:0/990798285 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734668999 0 0) 0x7f4f580c7310 con 0x7f4f58154ff0 2023-03-03T15:35:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.410+0000 7f4f567fc700 1 --2- 172.21.15.7:0/990798285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580c00e0 0x7f4f580c5c60 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009820 tx=0x7f4f48009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4479da2d25df64d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:35:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.411+0000 7f4f55ffb700 1 -- 172.21.15.7:0/990798285 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48005e30 con 0x7f4f580c00e0 2023-03-03T15:35:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.411+0000 7f4f55ffb700 1 -- 172.21.15.7:0/990798285 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f48004030 con 0x7f4f580c00e0 2023-03-03T15:35:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.411+0000 7f4f55ffb700 1 -- 172.21.15.7:0/990798285 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f480043e0 con 0x7f4f580c00e0 2023-03-03T15:35:10.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.411+0000 7f4f5d52d700 1 -- 172.21.15.7:0/990798285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580c00e0 msgr2=0x7f4f580c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:10.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.411+0000 7f4f5d52d700 1 --2- 172.21.15.7:0/990798285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580c00e0 0x7f4f580c5c60 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009820 tx=0x7f4f48009b30 comp rx=0 tx=0).stop 2023-03-03T15:35:10.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.411+0000 7f4f5d52d700 1 -- 172.21.15.7:0/990798285 shutdown_connections 2023-03-03T15:35:10.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.411+0000 7f4f5d52d700 1 --2- 172.21.15.7:0/990798285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580c00e0 0x7f4f580c5c60 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:10.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.411+0000 7f4f5d52d700 1 -- 172.21.15.7:0/990798285 >> 172.21.15.7:0/990798285 conn(0x7f4f58066e40 msgr2=0x7f4f58071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:35:10.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.411+0000 7f4f5d52d700 1 -- 172.21.15.7:0/990798285 shutdown_connections 2023-03-03T15:35:10.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.411+0000 7f4f5d52d700 1 -- 172.21.15.7:0/990798285 wait complete. 2023-03-03T15:35:10.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.412+0000 7f4f5d52d700 1 Processor -- start 2023-03-03T15:35:10.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f5d52d700 1 -- start start 2023-03-03T15:35:10.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f5d52d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f580c00e0 0x7f4f581edc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f5d52d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580caa30 0x7f4f581513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f5d52d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f58154ff0 0x7f4f581573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f5d52d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f581ed7f0 con 0x7f4f580caa30 2023-03-03T15:35:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f5d52d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f58156d50 con 0x7f4f58154ff0 2023-03-03T15:35:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f5d52d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5814bea0 con 0x7f4f580c00e0 2023-03-03T15:35:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f567fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580caa30 0x7f4f581513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f567fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580caa30 0x7f4f581513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53514/0 (socket says 172.21.15.7:53514) 2023-03-03T15:35:10.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f567fc700 1 -- 172.21.15.7:0/3890506233 learned_addr learned my addr 172.21.15.7:0/3890506233 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:35:10.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f56ffd700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f580c00e0 0x7f4f581edc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:10.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.413+0000 7f4f577fe700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f58154ff0 0x7f4f581573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:10.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f567fc700 1 -- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f580c00e0 msgr2=0x7f4f581edc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:10.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f567fc700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f580c00e0 0x7f4f581edc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:10.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f567fc700 1 -- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f58154ff0 msgr2=0x7f4f581573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:10.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f567fc700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f58154ff0 0x7f4f581573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:10.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f567fc700 1 -- 172.21.15.7:0/3890506233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f48009430 con 0x7f4f580caa30 2023-03-03T15:35:10.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f56ffd700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f580c00e0 0x7f4f581edc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:35:10.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f567fc700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580caa30 0x7f4f581513f0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009b70 tx=0x7f4f48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:35:10.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3890506233 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48011620 con 0x7f4f580caa30 2023-03-03T15:35:10.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3890506233 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f48011780 con 0x7f4f580caa30 2023-03-03T15:35:10.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f5d52d700 1 -- 172.21.15.7:0/3890506233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f581547f0 con 0x7f4f580caa30 2023-03-03T15:35:10.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3890506233 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48011b30 con 0x7f4f580caa30 2023-03-03T15:35:10.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.414+0000 7f4f5d52d700 1 -- 172.21.15.7:0/3890506233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f58154510 con 0x7f4f580caa30 2023-03-03T15:35:10.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.415+0000 7f4f5d52d700 1 -- 172.21.15.7:0/3890506233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f58154ad0 con 0x7f4f580caa30 2023-03-03T15:35:10.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.416+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3890506233 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4f4802d070 con 0x7f4f580caa30 2023-03-03T15:35:10.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.416+0000 7f4f4f7fe700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f380682f0 0x7f4f3806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:35:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.416+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3890506233 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4f48087920 con 0x7f4f580caa30 2023-03-03T15:35:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.416+0000 7f4f56ffd700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f380682f0 0x7f4f3806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:35:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.417+0000 7f4f56ffd700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f380682f0 0x7f4f3806a7a0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f4f40003fc0 tx=0x7f4f40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:35:10.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.420+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3890506233 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4f48053cc0 con 0x7f4f580caa30 2023-03-03T15:35:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.657+0000 7f4f5d52d700 1 -- 172.21.15.7:0/3890506233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f580c0b80 con 0x7f4f580caa30 2023-03-03T15:35:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.658+0000 7f4f4f7fe700 1 -- 172.21.15.7:0/3890506233 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4f48016090 con 0x7f4f580caa30 2023-03-03T15:35:10.659 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:35:10.660 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:35:10.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 -- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f380682f0 msgr2=0x7f4f3806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:10.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f380682f0 0x7f4f3806a7a0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f4f40003fc0 tx=0x7f4f40009040 comp rx=0 tx=0).stop 2023-03-03T15:35:10.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 -- 172.21.15.7:0/3890506233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580caa30 msgr2=0x7f4f581513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:35:10.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580caa30 0x7f4f581513f0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009b70 tx=0x7f4f48004750 comp rx=0 tx=0).stop 2023-03-03T15:35:10.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 -- 172.21.15.7:0/3890506233 shutdown_connections 2023-03-03T15:35:10.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f58154ff0 0x7f4f581573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:10.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f580c00e0 0x7f4f581edc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:10.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f380682f0 0x7f4f3806a7a0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:10.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 --2- 172.21.15.7:0/3890506233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f580caa30 0x7f4f581513f0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:35:10.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 -- 172.21.15.7:0/3890506233 >> 172.21.15.7:0/3890506233 conn(0x7f4f58066e40 msgr2=0x7f4f581534c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:35:10.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 -- 172.21.15.7:0/3890506233 shutdown_connections 2023-03-03T15:35:10.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:35:10.660+0000 7f4f5d52d700 1 -- 172.21.15.7:0/3890506233 wait complete. 2023-03-03T15:35:10.663 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:35:10.673 INFO:tasks.mds_thrash.fs.[a]:waiting for 51 secs before thrashing 2023-03-03T15:35:36.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:35:36.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:01.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:36:01.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.811+0000 7f8b3e56c700 1 Processor -- start 2023-03-03T15:36:01.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.811+0000 7f8b3e56c700 1 -- start start 2023-03-03T15:36:01.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b3e56c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b38155390 0x7f8b380c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:01.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b3e56c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8b3806e020 0x7f8b38071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:01.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b3e56c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b38154910 con 0x7f8b38072520 2023-03-03T15:36:01.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b3e56c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b38070b90 con 0x7f8b3806e020 2023-03-03T15:36:01.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b3e56c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b38151320 con 0x7f8b38155390 2023-03-03T15:36:01.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b3cb09700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8b38072520 0x7f8b38151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47536/0 (socket says 172.21.15.7:47536) 2023-03-03T15:36:01.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b3cb09700 1 -- 172.21.15.7:0/1176827614 learned_addr learned my addr 172.21.15.7:0/1176827614 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:36:01.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b377fe700 1 --2- 172.21.15.7:0/1176827614 >> v2:172.21.15.143:3300/0 conn(0x7f8b3806e020 0x7f8b38071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:01.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b37fff700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b38155390 0x7f8b380c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:01.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490115433 0 0) 0x7f8b38154910 con 0x7f8b38072520 2023-03-03T15:36:01.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.812+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b240035b0 con 0x7f8b38072520 2023-03-03T15:36:01.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 817679073 0 0) 0x7f8b240035b0 con 0x7f8b38072520 2023-03-03T15:36:01.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b38155390 msgr2=0x7f8b380c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:36:01.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b38155390 0x7f8b380c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:01.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 >> v2:172.21.15.143:3300/0 conn(0x7f8b3806e020 msgr2=0x7f8b38071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:36:01.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 --2- 172.21.15.7:0/1176827614 >> v2:172.21.15.143:3300/0 conn(0x7f8b3806e020 0x7f8b38071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:01.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b380702a0 con 0x7f8b38072520 2023-03-03T15:36:01.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8b300019f0 con 0x7f8b38072520 2023-03-03T15:36:01.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 >> v1:172.21.15.7:6789/0 conn(0x7f8b38072520 legacy=0x7f8b38151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:01.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b38155390 0x7f8b24003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:01.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b24003ff0 0x7f8b24004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:01.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b24004d90 0x7f8b24005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:01.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b38070b90 con 0x7f8b24004d90 2023-03-03T15:36:01.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b38151320 con 0x7f8b24003ff0 2023-03-03T15:36:01.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2400bac0 con 0x7f8b38155390 2023-03-03T15:36:01.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b377fe700 1 --2- 172.21.15.7:0/1176827614 >> v2:172.21.15.143:3300/0 conn(0x7f8b3806e020 0x7f8b38071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:36:01.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b3cb09700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b24004d90 0x7f8b24005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:01.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b37fff700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b38155390 0x7f8b24003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:01.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b377fe700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b24003ff0 0x7f8b24004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:01.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b3cb09700 1 -- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b38155390 msgr2=0x7f8b24003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:36:01.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b3cb09700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b38155390 0x7f8b24003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:01.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b3cb09700 1 -- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b24003ff0 msgr2=0x7f8b24004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:36:01.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b3cb09700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b24003ff0 0x7f8b24004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:01.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b3cb09700 1 -- 172.21.15.7:0/1176827614 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b240035b0 con 0x7f8b24004d90 2023-03-03T15:36:01.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b377fe700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b24003ff0 0x7f8b24004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.813+0000 7f8b3cb09700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b24004d90 0x7f8b24005200 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000a830 tx=0x7f8b24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4148d2062287114c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.814+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30012600 con 0x7f8b24004d90 2023-03-03T15:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.814+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b30012760 con 0x7f8b24004d90 2023-03-03T15:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.814+0000 7f8b36ffd700 1 -- 172.21.15.7:0/1176827614 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30012b10 con 0x7f8b24004d90 2023-03-03T15:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.814+0000 7f8b3e56c700 1 -- 172.21.15.7:0/1176827614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b24004d90 msgr2=0x7f8b24005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.814+0000 7f8b3e56c700 1 --2- 172.21.15.7:0/1176827614 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b24004d90 0x7f8b24005200 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000a830 tx=0x7f8b24005880 comp rx=0 tx=0).stop 2023-03-03T15:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.814+0000 7f8b3cb09700 1 -- 172.21.15.7:0/1176827614 reap_dead start 2023-03-03T15:36:01.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.814+0000 7f8b3e56c700 1 -- 172.21.15.7:0/1176827614 shutdown_connections 2023-03-03T15:36:01.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.814+0000 7f8b3e56c700 1 -- 172.21.15.7:0/1176827614 >> 172.21.15.7:0/1176827614 conn(0x7f8b3806c110 msgr2=0x7f8b3806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:36:01.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.814+0000 7f8b3e56c700 1 -- 172.21.15.7:0/1176827614 shutdown_connections 2023-03-03T15:36:01.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.814+0000 7f8b3e56c700 1 -- 172.21.15.7:0/1176827614 wait complete. 2023-03-03T15:36:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.815+0000 7f8b3e56c700 1 Processor -- start 2023-03-03T15:36:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.816+0000 7f8b3e56c700 1 -- start start 2023-03-03T15:36:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.816+0000 7f8b3e56c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b38152050 0x7f8b381ed710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.816+0000 7f8b3e56c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b38152420 0x7f8b381edc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:01.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.816+0000 7f8b3e56c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b380c07d0 0x7f8b38147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.816+0000 7f8b3e56c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b380c11a0 con 0x7f8b38152420 2023-03-03T15:36:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.816+0000 7f8b3e56c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b380c5d20 con 0x7f8b38152050 2023-03-03T15:36:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.816+0000 7f8b3e56c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b380c0ec0 con 0x7f8b380c07d0 2023-03-03T15:36:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.816+0000 7f8b377fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b38152420 0x7f8b381edc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b377fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b38152420 0x7f8b381edc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53566/0 (socket says 172.21.15.7:53566) 2023-03-03T15:36:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b377fe700 1 -- 172.21.15.7:0/3530536772 learned_addr learned my addr 172.21.15.7:0/3530536772 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:36:01.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b3cb09700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b380c07d0 0x7f8b38147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:01.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b37fff700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b38152050 0x7f8b381ed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:01.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b377fe700 1 -- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b380c07d0 msgr2=0x7f8b38147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:01.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b377fe700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b380c07d0 0x7f8b38147e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:01.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b377fe700 1 -- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b38152050 msgr2=0x7f8b381ed710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:01.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b377fe700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b38152050 0x7f8b381ed710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:01.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b377fe700 1 -- 172.21.15.7:0/3530536772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b38070820 con 0x7f8b38152420 2023-03-03T15:36:01.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b37fff700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b38152050 0x7f8b381ed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:36:01.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b377fe700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b38152420 0x7f8b381edc50 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f8b20010c30 tx=0x7f8b20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:36:01.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b34ff9700 1 -- 172.21.15.7:0/3530536772 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2000e990 con 0x7f8b38152420 2023-03-03T15:36:01.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b34ff9700 1 -- 172.21.15.7:0/3530536772 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2000eaf0 con 0x7f8b38152420 2023-03-03T15:36:01.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b3e56c700 1 -- 172.21.15.7:0/3530536772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b38070540 con 0x7f8b38152420 2023-03-03T15:36:01.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b34ff9700 1 -- 172.21.15.7:0/3530536772 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2000ee60 con 0x7f8b38152420 2023-03-03T15:36:01.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.817+0000 7f8b3e56c700 1 -- 172.21.15.7:0/3530536772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b38155760 con 0x7f8b38152420 2023-03-03T15:36:01.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.818+0000 7f8b3e56c700 1 -- 172.21.15.7:0/3530536772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b381ea5e0 con 0x7f8b38152420 2023-03-03T15:36:01.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.819+0000 7f8b34ff9700 1 -- 172.21.15.7:0/3530536772 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b2001c030 con 0x7f8b38152420 2023-03-03T15:36:01.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.819+0000 7f8b34ff9700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b180682f0 0x7f8b1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:01.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.819+0000 7f8b34ff9700 1 -- 172.21.15.7:0/3530536772 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8b20086580 con 0x7f8b38152420 2023-03-03T15:36:01.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.819+0000 7f8b37fff700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b180682f0 0x7f8b1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:01.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.820+0000 7f8b37fff700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b180682f0 0x7f8b1806a7a0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f8b3814afa0 tx=0x7f8b28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:36:01.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:01.823+0000 7f8b34ff9700 1 -- 172.21.15.7:0/3530536772 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b381ea5e0 con 0x7f8b38152420 2023-03-03T15:36:02.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.062+0000 7f8b3e56c700 1 -- 172.21.15.7:0/3530536772 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b381ea5e0 con 0x7f8b38152420 2023-03-03T15:36:02.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.063+0000 7f8b34ff9700 1 -- 172.21.15.7:0/3530536772 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8b381ea5e0 con 0x7f8b38152420 2023-03-03T15:36:02.065 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:36:02.065 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:36:02.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 -- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b180682f0 msgr2=0x7f8b1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:02.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b180682f0 0x7f8b1806a7a0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f8b3814afa0 tx=0x7f8b28008040 comp rx=0 tx=0).stop 2023-03-03T15:36:02.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 -- 172.21.15.7:0/3530536772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b38152420 msgr2=0x7f8b381edc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:02.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b38152420 0x7f8b381edc50 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f8b20010c30 tx=0x7f8b20009e30 comp rx=0 tx=0).stop 2023-03-03T15:36:02.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 -- 172.21.15.7:0/3530536772 shutdown_connections 2023-03-03T15:36:02.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b380c07d0 0x7f8b38147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:02.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b38152050 0x7f8b381ed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:02.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b180682f0 0x7f8b1806a7a0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:02.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 --2- 172.21.15.7:0/3530536772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b38152420 0x7f8b381edc50 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:02.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 -- 172.21.15.7:0/3530536772 >> 172.21.15.7:0/3530536772 conn(0x7f8b3806c110 msgr2=0x7f8b381542b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:36:02.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 -- 172.21.15.7:0/3530536772 shutdown_connections 2023-03-03T15:36:02.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:02.065+0000 7f8b3e56c700 1 -- 172.21.15.7:0/3530536772 wait complete. 2023-03-03T15:36:02.068 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:36:02.078 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T15:36:06.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:06.652 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:17.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:36:17.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.574+0000 7f8b5ab85700 1 Processor -- start 2023-03-03T15:36:17.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.574+0000 7f8b5ab85700 1 -- start start 2023-03-03T15:36:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b5ab85700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8b540c4850 0x7f8b54154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b5ab85700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b54072920 con 0x7f8b5414e560 2023-03-03T15:36:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b5ab85700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5406bef0 con 0x7f8b540c43d0 2023-03-03T15:36:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b5ab85700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b54072380 con 0x7f8b540c4850 2023-03-03T15:36:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b58921700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8b5414e560 0x7f8b54071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47562/0 (socket says 172.21.15.7:47562) 2023-03-03T15:36:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b58921700 1 -- 172.21.15.7:0/390326208 learned_addr learned my addr 172.21.15.7:0/390326208 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:36:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b59122700 1 --2- 172.21.15.7:0/390326208 >> v2:172.21.15.143:3300/0 conn(0x7f8b540c4850 0x7f8b54154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269456763 0 0) 0x7f8b48001110 con 0x7f8b5414e560 2023-03-03T15:36:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b440035b0 con 0x7f8b5414e560 2023-03-03T15:36:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1534954180 0 0) 0x7f8b440035b0 con 0x7f8b5414e560 2023-03-03T15:36:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 >> v2:172.21.15.143:3300/0 conn(0x7f8b540c4850 msgr2=0x7f8b54154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:36:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 --2- 172.21.15.7:0/390326208 >> v2:172.21.15.143:3300/0 conn(0x7f8b540c4850 0x7f8b54154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 >> v1:172.21.15.143:6789/0 conn(0x7f8b540c43d0 legacy=0x7f8b54154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b541551e0 con 0x7f8b5414e560 2023-03-03T15:36:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8b48001800 con 0x7f8b5414e560 2023-03-03T15:36:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 >> v1:172.21.15.7:6789/0 conn(0x7f8b5414e560 legacy=0x7f8b54071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b44003a70 0x7f8b44003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b44004850 0x7f8b44004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b44005200 0x7f8b4400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b54072380 con 0x7f8b44003a70 2023-03-03T15:36:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4400c0e0 con 0x7f8b44005200 2023-03-03T15:36:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.575+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4400c2a0 con 0x7f8b44004850 2023-03-03T15:36:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b58921700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b44003a70 0x7f8b44003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b59122700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b44005200 0x7f8b4400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b53fff700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b44004850 0x7f8b44004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b58921700 1 -- 172.21.15.7:0/390326208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b44004850 msgr2=0x7f8b44004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:36:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b58921700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b44004850 0x7f8b44004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b58921700 1 -- 172.21.15.7:0/390326208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b44005200 msgr2=0x7f8b4400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:36:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b58921700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b44005200 0x7f8b4400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b58921700 1 -- 172.21.15.7:0/390326208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b440035b0 con 0x7f8b44003a70 2023-03-03T15:36:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b59122700 1 -- 172.21.15.7:0/390326208 reap_dead start 2023-03-03T15:36:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b53fff700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b44004850 0x7f8b44004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:36:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b58921700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b44003a70 0x7f8b44003e80 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800c210 tx=0x7f8b44005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7caf8da7e33aa8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:36:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48007d90 con 0x7f8b44003a70 2023-03-03T15:36:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4800cd50 con 0x7f8b44003a70 2023-03-03T15:36:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.576+0000 7f8b537fe700 1 -- 172.21.15.7:0/390326208 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48005ba0 con 0x7f8b44003a70 2023-03-03T15:36:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.577+0000 7f8b5ab85700 1 -- 172.21.15.7:0/390326208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b44003a70 msgr2=0x7f8b44003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.577+0000 7f8b5ab85700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b44003a70 0x7f8b44003e80 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800c210 tx=0x7f8b44005880 comp rx=0 tx=0).stop 2023-03-03T15:36:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.577+0000 7f8b5ab85700 1 -- 172.21.15.7:0/390326208 shutdown_connections 2023-03-03T15:36:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.577+0000 7f8b5ab85700 1 --2- 172.21.15.7:0/390326208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b44003a70 0x7f8b44003e80 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.577+0000 7f8b5ab85700 1 -- 172.21.15.7:0/390326208 >> 172.21.15.7:0/390326208 conn(0x7f8b540736f0 msgr2=0x7f8b54073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:36:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.577+0000 7f8b5ab85700 1 -- 172.21.15.7:0/390326208 shutdown_connections 2023-03-03T15:36:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.577+0000 7f8b5ab85700 1 -- 172.21.15.7:0/390326208 wait complete. 2023-03-03T15:36:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.578+0000 7f8b5ab85700 1 Processor -- start 2023-03-03T15:36:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.578+0000 7f8b5ab85700 1 -- start start 2023-03-03T15:36:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b5ab85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b44003a70 0x7f8b54148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b5ab85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b54149290 0x7f8b5414a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b5ab85700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b541ddbf0 0x7f8b5414ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b5ab85700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b541df5f0 con 0x7f8b541ddbf0 2023-03-03T15:36:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b5ab85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540bd3d0 con 0x7f8b44003a70 2023-03-03T15:36:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b5ab85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b541ea450 con 0x7f8b54149290 2023-03-03T15:36:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b59122700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b541ddbf0 0x7f8b5414ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b59122700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b541ddbf0 0x7f8b5414ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53598/0 (socket says 172.21.15.7:53598) 2023-03-03T15:36:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b59122700 1 -- 172.21.15.7:0/3438018971 learned_addr learned my addr 172.21.15.7:0/3438018971 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:36:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b58921700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b44003a70 0x7f8b54148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b53fff700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b54149290 0x7f8b5414a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b59122700 1 -- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b54149290 msgr2=0x7f8b5414a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b59122700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b54149290 0x7f8b5414a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b59122700 1 -- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b44003a70 msgr2=0x7f8b54148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b59122700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b44003a70 0x7f8b54148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.579+0000 7f8b59122700 1 -- 172.21.15.7:0/3438018971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540c4c70 con 0x7f8b541ddbf0 2023-03-03T15:36:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.580+0000 7f8b59122700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b541ddbf0 0x7f8b5414ae20 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f8b54071200 tx=0x7f8b4c011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:36:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.580+0000 7f8b517fa700 1 -- 172.21.15.7:0/3438018971 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c00e040 con 0x7f8b541ddbf0 2023-03-03T15:36:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.580+0000 7f8b517fa700 1 -- 172.21.15.7:0/3438018971 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4c00f040 con 0x7f8b541ddbf0 2023-03-03T15:36:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.580+0000 7f8b5ab85700 1 -- 172.21.15.7:0/3438018971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5406b910 con 0x7f8b541ddbf0 2023-03-03T15:36:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.580+0000 7f8b517fa700 1 -- 172.21.15.7:0/3438018971 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c00fc70 con 0x7f8b541ddbf0 2023-03-03T15:36:17.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.580+0000 7f8b5ab85700 1 -- 172.21.15.7:0/3438018971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b540aacc0 con 0x7f8b541ddbf0 2023-03-03T15:36:17.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.581+0000 7f8b5ab85700 1 -- 172.21.15.7:0/3438018971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b541ed060 con 0x7f8b541ddbf0 2023-03-03T15:36:17.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.581+0000 7f8b517fa700 1 -- 172.21.15.7:0/3438018971 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b4c015430 con 0x7f8b541ddbf0 2023-03-03T15:36:17.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.582+0000 7f8b517fa700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b38068320 0x7f8b3806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:36:17.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.582+0000 7f8b517fa700 1 -- 172.21.15.7:0/3438018971 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8b4c086a30 con 0x7f8b541ddbf0 2023-03-03T15:36:17.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.582+0000 7f8b58921700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b38068320 0x7f8b3806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:36:17.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.582+0000 7f8b58921700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b38068320 0x7f8b3806a7d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f8b48000c00 tx=0x7f8b48007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:36:17.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.586+0000 7f8b517fa700 1 -- 172.21.15.7:0/3438018971 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b4c052e20 con 0x7f8b541ddbf0 2023-03-03T15:36:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.824+0000 7f8b5ab85700 1 -- 172.21.15.7:0/3438018971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b54074bb0 con 0x7f8b541ddbf0 2023-03-03T15:36:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.825+0000 7f8b517fa700 1 -- 172.21.15.7:0/3438018971 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8b4c056440 con 0x7f8b541ddbf0 2023-03-03T15:36:17.826 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:36:17.827 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:36:17.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.826+0000 7f8b5ab85700 1 -- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b38068320 msgr2=0x7f8b3806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:17.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.826+0000 7f8b5ab85700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b38068320 0x7f8b3806a7d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f8b48000c00 tx=0x7f8b48007000 comp rx=0 tx=0).stop 2023-03-03T15:36:17.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.826+0000 7f8b5ab85700 1 -- 172.21.15.7:0/3438018971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b541ddbf0 msgr2=0x7f8b5414ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:36:17.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.826+0000 7f8b5ab85700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b541ddbf0 0x7f8b5414ae20 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f8b54071200 tx=0x7f8b4c011ed0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.827+0000 7f8b5ab85700 1 -- 172.21.15.7:0/3438018971 shutdown_connections 2023-03-03T15:36:17.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.827+0000 7f8b5ab85700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8b38068320 0x7f8b3806a7d0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.827+0000 7f8b5ab85700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b541ddbf0 0x7f8b5414ae20 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.827+0000 7f8b5ab85700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b44003a70 0x7f8b54148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.827+0000 7f8b5ab85700 1 --2- 172.21.15.7:0/3438018971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b54149290 0x7f8b5414a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:36:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.827+0000 7f8b5ab85700 1 -- 172.21.15.7:0/3438018971 >> 172.21.15.7:0/3438018971 conn(0x7f8b540736f0 msgr2=0x7f8b54071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:36:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.827+0000 7f8b5ab85700 1 -- 172.21.15.7:0/3438018971 shutdown_connections 2023-03-03T15:36:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:36:17.827+0000 7f8b5ab85700 1 -- 172.21.15.7:0/3438018971 wait complete. 2023-03-03T15:36:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:36:17.840 INFO:tasks.mds_thrash.fs.[b]:waiting for 118 secs before thrashing 2023-03-03T15:36:36.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:36:36.732 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:06.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:06.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:08.080 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:37:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.218+0000 7f9b0703c700 1 Processor -- start 2023-03-03T15:37:08.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.218+0000 7f9b0703c700 1 -- start start 2023-03-03T15:37:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.218+0000 7f9b0703c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b000c2930 0x7f9b00157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.218+0000 7f9b0703c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9b0006c480 0x7f9b00157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.218+0000 7f9b0703c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0006d430 con 0x7f9b00151d40 2023-03-03T15:37:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.218+0000 7f9b0703c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0006c2a0 con 0x7f9b000c2930 2023-03-03T15:37:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.218+0000 7f9b0703c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0006cb00 con 0x7f9b0006c480 2023-03-03T15:37:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9afffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b000c2930 0x7f9b00157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9afffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9b000c2930 0x7f9b00157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53640/0 (socket says 172.21.15.7:53640) 2023-03-03T15:37:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9afffff700 1 -- 172.21.15.7:0/2946716957 learned_addr learned my addr 172.21.15.7:0/2946716957 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:37:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9afffff700 1 -- 172.21.15.7:0/2946716957 >> v2:172.21.15.143:3300/0 conn(0x7f9b0006c480 msgr2=0x7f9b00157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:37:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9afffff700 1 --2- 172.21.15.7:0/2946716957 >> v2:172.21.15.143:3300/0 conn(0x7f9b0006c480 0x7f9b00157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9afffff700 1 -- 172.21.15.7:0/2946716957 >> v1:172.21.15.143:6789/0 conn(0x7f9b00151d40 legacy=0x7f9b000736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:37:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9afffff700 1 -- 172.21.15.7:0/2946716957 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0006cde0 con 0x7f9b000c2930 2023-03-03T15:37:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9afffff700 1 --2- 172.21.15.7:0/2946716957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b000c2930 0x7f9b00157a90 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0098f0 tx=0x7f9aec009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452040ab7c6bc194 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9aff7fe700 1 -- 172.21.15.7:0/2946716957 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec004230 con 0x7f9b000c2930 2023-03-03T15:37:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9aff7fe700 1 -- 172.21.15.7:0/2946716957 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec004390 con 0x7f9b000c2930 2023-03-03T15:37:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.219+0000 7f9aff7fe700 1 -- 172.21.15.7:0/2946716957 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec0036a0 con 0x7f9b000c2930 2023-03-03T15:37:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.220+0000 7f9b0703c700 1 -- 172.21.15.7:0/2946716957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b000c2930 msgr2=0x7f9b00157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.220+0000 7f9b0703c700 1 --2- 172.21.15.7:0/2946716957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b000c2930 0x7f9b00157a90 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0098f0 tx=0x7f9aec009c00 comp rx=0 tx=0).stop 2023-03-03T15:37:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.220+0000 7f9b0703c700 1 -- 172.21.15.7:0/2946716957 shutdown_connections 2023-03-03T15:37:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.220+0000 7f9b0703c700 1 --2- 172.21.15.7:0/2946716957 >> v2:172.21.15.143:3300/0 conn(0x7f9b0006c480 0x7f9b00157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.220+0000 7f9b0703c700 1 --2- 172.21.15.7:0/2946716957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b000c2930 0x7f9b00157a90 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.220+0000 7f9b0703c700 1 -- 172.21.15.7:0/2946716957 >> 172.21.15.7:0/2946716957 conn(0x7f9b000a94d0 msgr2=0x7f9b000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:37:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.220+0000 7f9b0703c700 1 -- 172.21.15.7:0/2946716957 shutdown_connections 2023-03-03T15:37:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.220+0000 7f9b0703c700 1 -- 172.21.15.7:0/2946716957 wait complete. 2023-03-03T15:37:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.221+0000 7f9b0703c700 1 Processor -- start 2023-03-03T15:37:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.221+0000 7f9b0703c700 1 -- start start 2023-03-03T15:37:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9b0703c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b0006c480 0x7f9b001503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9b0703c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b000c2930 0x7f9b0014ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9b0703c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b00151d40 0x7f9b0014d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9b0703c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000be840 con 0x7f9b0006c480 2023-03-03T15:37:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9b0703c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0014ead0 con 0x7f9b00151d40 2023-03-03T15:37:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9b0703c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b001eac30 con 0x7f9b000c2930 2023-03-03T15:37:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9b04dd8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b0006c480 0x7f9b001503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9b04dd8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b0006c480 0x7f9b001503c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53644/0 (socket says 172.21.15.7:53644) 2023-03-03T15:37:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9b04dd8700 1 -- 172.21.15.7:0/906915886 learned_addr learned my addr 172.21.15.7:0/906915886 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:37:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9afffff700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b000c2930 0x7f9b0014ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.222+0000 7f9b055d9700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b00151d40 0x7f9b0014d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9b04dd8700 1 -- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b000c2930 msgr2=0x7f9b0014ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9b04dd8700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b000c2930 0x7f9b0014ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9b04dd8700 1 -- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b00151d40 msgr2=0x7f9b0014d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9b04dd8700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b00151d40 0x7f9b0014d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9b04dd8700 1 -- 172.21.15.7:0/906915886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec009550 con 0x7f9b0006c480 2023-03-03T15:37:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9b04dd8700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b0006c480 0x7f9b001503c0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f9af400c4a0 tx=0x7f9af400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9afd7fa700 1 -- 172.21.15.7:0/906915886 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af40054e0 con 0x7f9b0006c480 2023-03-03T15:37:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9afd7fa700 1 -- 172.21.15.7:0/906915886 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4003680 con 0x7f9b0006c480 2023-03-03T15:37:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9b0703c700 1 -- 172.21.15.7:0/906915886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b00073e00 con 0x7f9b0006c480 2023-03-03T15:37:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9afd7fa700 1 -- 172.21.15.7:0/906915886 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4003a30 con 0x7f9b0006c480 2023-03-03T15:37:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.223+0000 7f9b0703c700 1 -- 172.21.15.7:0/906915886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0006d0c0 con 0x7f9b0006c480 2023-03-03T15:37:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.224+0000 7f9b0703c700 1 -- 172.21.15.7:0/906915886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0006bea0 con 0x7f9b0006c480 2023-03-03T15:37:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.224+0000 7f9afd7fa700 1 -- 172.21.15.7:0/906915886 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9af4019030 con 0x7f9b0006c480 2023-03-03T15:37:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.225+0000 7f9afd7fa700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ae40682f0 0x7f9ae406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.225+0000 7f9afd7fa700 1 -- 172.21.15.7:0/906915886 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9af40853f0 con 0x7f9b0006c480 2023-03-03T15:37:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.225+0000 7f9afffff700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ae40682f0 0x7f9ae406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.226+0000 7f9afffff700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ae40682f0 0x7f9ae406a7a0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009130 tx=0x7f9aec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:08.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.229+0000 7f9afd7fa700 1 -- 172.21.15.7:0/906915886 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9af4051790 con 0x7f9b0006c480 2023-03-03T15:37:08.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.462+0000 7f9b0703c700 1 -- 172.21.15.7:0/906915886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b00074bb0 con 0x7f9b0006c480 2023-03-03T15:37:08.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.463+0000 7f9afd7fa700 1 -- 172.21.15.7:0/906915886 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9af4017090 con 0x7f9b0006c480 2023-03-03T15:37:08.465 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:37:08.465 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:37:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 -- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ae40682f0 msgr2=0x7f9ae406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ae40682f0 0x7f9ae406a7a0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009130 tx=0x7f9aec008030 comp rx=0 tx=0).stop 2023-03-03T15:37:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 -- 172.21.15.7:0/906915886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b0006c480 msgr2=0x7f9b001503c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b0006c480 0x7f9b001503c0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f9af400c4a0 tx=0x7f9af400c7b0 comp rx=0 tx=0).stop 2023-03-03T15:37:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 -- 172.21.15.7:0/906915886 shutdown_connections 2023-03-03T15:37:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b00151d40 0x7f9b0014d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ae40682f0 0x7f9ae406a7a0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b0006c480 0x7f9b001503c0 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 --2- 172.21.15.7:0/906915886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9b000c2930 0x7f9b0014ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 -- 172.21.15.7:0/906915886 >> 172.21.15.7:0/906915886 conn(0x7f9b000a94d0 msgr2=0x7f9b001558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 -- 172.21.15.7:0/906915886 shutdown_connections 2023-03-03T15:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:08.465+0000 7f9b0703c700 1 -- 172.21.15.7:0/906915886 wait complete. 2023-03-03T15:37:08.468 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:37:08.478 INFO:tasks.mds_thrash.fs.[a]:waiting for 32 secs before thrashing 2023-03-03T15:37:36.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:36.978 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:37:40.479 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:37:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.615+0000 7f3f2cdd8700 1 Processor -- start 2023-03-03T15:37:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.615+0000 7f3f2cdd8700 1 -- start start 2023-03-03T15:37:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.616+0000 7f3f2cdd8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814f9d0 0x7f3f28156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.616+0000 7f3f2cdd8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f2814fda0 0x7f3f28071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:40.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.616+0000 7f3f2cdd8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f280c2f30 con 0x7f3f280abe20 2023-03-03T15:37:40.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.616+0000 7f3f2cdd8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f280c31d0 con 0x7f3f2814fda0 2023-03-03T15:37:40.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.616+0000 7f3f2cdd8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f281524a0 con 0x7f3f2814f9d0 2023-03-03T15:37:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.616+0000 7f3f2659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814f9d0 0x7f3f28156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.616+0000 7f3f25d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f2814fda0 0x7f3f28071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.616+0000 7f3f25d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f2814fda0 0x7f3f28071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:49460/0 (socket says 172.21.15.7:49460) 2023-03-03T15:37:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.616+0000 7f3f26d9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f3f280abe20 0x7f3f281529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:51186/0 (socket says 172.21.15.7:51186) 2023-03-03T15:37:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.616+0000 7f3f2659c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814f9d0 0x7f3f28156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:33526/0 (socket says 172.21.15.7:33526) 2023-03-03T15:37:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.617+0000 7f3f25d9b700 1 -- 172.21.15.7:0/2965026525 learned_addr learned my addr 172.21.15.7:0/2965026525 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:37:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.617+0000 7f3f2659c700 1 -- 172.21.15.7:0/2965026525 >> v2:172.21.15.143:3300/0 conn(0x7f3f2814fda0 msgr2=0x7f3f28071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:37:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.617+0000 7f3f2659c700 1 --2- 172.21.15.7:0/2965026525 >> v2:172.21.15.143:3300/0 conn(0x7f3f2814fda0 0x7f3f28071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.617+0000 7f3f2659c700 1 -- 172.21.15.7:0/2965026525 >> v1:172.21.15.143:6789/0 conn(0x7f3f280abe20 legacy=0x7f3f281529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.617+0000 7f3f2659c700 1 -- 172.21.15.7:0/2965026525 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f280c34b0 con 0x7f3f2814f9d0 2023-03-03T15:37:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.617+0000 7f3f25d9b700 1 --2- 172.21.15.7:0/2965026525 >> v2:172.21.15.143:3300/0 conn(0x7f3f2814fda0 0x7f3f28071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:37:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.617+0000 7f3f2659c700 1 --2- 172.21.15.7:0/2965026525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814f9d0 0x7f3f28156430 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009840 tx=0x7f3f10009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcfa53a4b92c6bbb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.617+0000 7f3f2559a700 1 -- 172.21.15.7:0/2965026525 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10011830 con 0x7f3f2814f9d0 2023-03-03T15:37:40.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.617+0000 7f3f2559a700 1 -- 172.21.15.7:0/2965026525 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f10011990 con 0x7f3f2814f9d0 2023-03-03T15:37:40.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.617+0000 7f3f2559a700 1 -- 172.21.15.7:0/2965026525 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10011d20 con 0x7f3f2814f9d0 2023-03-03T15:37:40.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.618+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/2965026525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814f9d0 msgr2=0x7f3f28156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:40.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.618+0000 7f3f2cdd8700 1 --2- 172.21.15.7:0/2965026525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814f9d0 0x7f3f28156430 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009840 tx=0x7f3f10009b50 comp rx=0 tx=0).stop 2023-03-03T15:37:40.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.618+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/2965026525 shutdown_connections 2023-03-03T15:37:40.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.618+0000 7f3f2cdd8700 1 --2- 172.21.15.7:0/2965026525 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814f9d0 0x7f3f28156430 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:40.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.618+0000 7f3f2cdd8700 1 --2- 172.21.15.7:0/2965026525 >> v2:172.21.15.143:3300/0 conn(0x7f3f2814fda0 0x7f3f28071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:40.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.618+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/2965026525 >> 172.21.15.7:0/2965026525 conn(0x7f3f280c8900 msgr2=0x7f3f280c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:37:40.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.618+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/2965026525 shutdown_connections 2023-03-03T15:37:40.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.618+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/2965026525 wait complete. 2023-03-03T15:37:40.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.619+0000 7f3f2cdd8700 1 Processor -- start 2023-03-03T15:37:40.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.619+0000 7f3f2cdd8700 1 -- start start 2023-03-03T15:37:40.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f2cdd8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f280abe20 0x7f3f281ed650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:40.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f2cdd8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f2814f9d0 0x7f3f2806e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:40.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f2cdd8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814fda0 0x7f3f2814afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:40.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f2cdd8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f28153580 con 0x7f3f2814f9d0 2023-03-03T15:37:40.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f2cdd8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f280c7bc0 con 0x7f3f280abe20 2023-03-03T15:37:40.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f2cdd8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f281edb90 con 0x7f3f2814fda0 2023-03-03T15:37:40.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f25d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f2814f9d0 0x7f3f2806e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:40.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f25d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f2814f9d0 0x7f3f2806e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53690/0 (socket says 172.21.15.7:53690) 2023-03-03T15:37:40.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f25d9b700 1 -- 172.21.15.7:0/3017998174 learned_addr learned my addr 172.21.15.7:0/3017998174 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:37:40.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f2659c700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f280abe20 0x7f3f281ed650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:40.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.620+0000 7f3f26d9d700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814fda0 0x7f3f2814afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:40.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f25d9b700 1 -- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814fda0 msgr2=0x7f3f2814afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:40.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f25d9b700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814fda0 0x7f3f2814afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:40.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f25d9b700 1 -- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f280abe20 msgr2=0x7f3f281ed650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:40.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f25d9b700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f280abe20 0x7f3f281ed650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:40.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f25d9b700 1 -- 172.21.15.7:0/3017998174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f18009550 con 0x7f3f2814f9d0 2023-03-03T15:37:40.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f2659c700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f280abe20 0x7f3f281ed650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:37:40.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f25d9b700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f2814f9d0 0x7f3f2806e900 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800ebe0 tx=0x7f3f1800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:40.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f1effd700 1 -- 172.21.15.7:0/3017998174 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1800c850 con 0x7f3f2814f9d0 2023-03-03T15:37:40.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f1effd700 1 -- 172.21.15.7:0/3017998174 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f1800c9b0 con 0x7f3f2814f9d0 2023-03-03T15:37:40.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/3017998174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f100094a0 con 0x7f3f2814f9d0 2023-03-03T15:37:40.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f1effd700 1 -- 172.21.15.7:0/3017998174 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1800cd40 con 0x7f3f2814f9d0 2023-03-03T15:37:40.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.621+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/3017998174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f280ac350 con 0x7f3f2814f9d0 2023-03-03T15:37:40.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.622+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/3017998174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f280ab630 con 0x7f3f2814f9d0 2023-03-03T15:37:40.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.623+0000 7f3f1effd700 1 -- 172.21.15.7:0/3017998174 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3f18019030 con 0x7f3f2814f9d0 2023-03-03T15:37:40.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.623+0000 7f3f1effd700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f080682f0 0x7f3f0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:40.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.623+0000 7f3f1effd700 1 -- 172.21.15.7:0/3017998174 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3f18086990 con 0x7f3f2814f9d0 2023-03-03T15:37:40.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.624+0000 7f3f2659c700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f080682f0 0x7f3f0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:40.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.624+0000 7f3f2659c700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f080682f0 0x7f3f0806a7a0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f3f10000c00 tx=0x7f3f10008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:40.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.627+0000 7f3f1effd700 1 -- 172.21.15.7:0/3017998174 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3f18052cc0 con 0x7f3f2814f9d0 2023-03-03T15:37:40.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.866+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/3017998174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f280c3e10 con 0x7f3f2814f9d0 2023-03-03T15:37:40.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.867+0000 7f3f1effd700 1 -- 172.21.15.7:0/3017998174 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3f1800cea0 con 0x7f3f2814f9d0 2023-03-03T15:37:40.869 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:37:40.870 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:37:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f080682f0 msgr2=0x7f3f0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:40.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f080682f0 0x7f3f0806a7a0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f3f10000c00 tx=0x7f3f10008030 comp rx=0 tx=0).stop 2023-03-03T15:37:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/3017998174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f2814f9d0 msgr2=0x7f3f2806e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f2814f9d0 0x7f3f2806e900 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800ebe0 tx=0x7f3f1800eef0 comp rx=0 tx=0).stop 2023-03-03T15:37:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/3017998174 shutdown_connections 2023-03-03T15:37:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f2814fda0 0x7f3f2814afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f280abe20 0x7f3f281ed650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f080682f0 0x7f3f0806a7a0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 --2- 172.21.15.7:0/3017998174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f2814f9d0 0x7f3f2806e900 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:40.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/3017998174 >> 172.21.15.7:0/3017998174 conn(0x7f3f280c8900 msgr2=0x7f3f280c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:37:40.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/3017998174 shutdown_connections 2023-03-03T15:37:40.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:40.869+0000 7f3f2cdd8700 1 -- 172.21.15.7:0/3017998174 wait complete. 2023-03-03T15:37:40.873 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:37:40.883 INFO:tasks.mds_thrash.fs.[a]:waiting for 12 secs before thrashing 2023-03-03T15:37:52.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:37:53.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.023+0000 7f7f1af69700 1 Processor -- start 2023-03-03T15:37:53.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.023+0000 7f7f1af69700 1 -- start start 2023-03-03T15:37:53.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f1af69700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7f140c6010 0x7f7f140c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f1af69700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7f140c17f0 0x7f7f140c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f1af69700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140c5500 con 0x7f7f1406d6a0 2023-03-03T15:37:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f1af69700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f14152100 con 0x7f7f140c6010 2023-03-03T15:37:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f1af69700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f1414ebe0 con 0x7f7f140c17f0 2023-03-03T15:37:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f18d05700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7f140c6010 0x7f7f140c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f19506700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7f1406d6a0 0x7f7f1406dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47696/0 (socket says 172.21.15.7:47696) 2023-03-03T15:37:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f19506700 1 -- 172.21.15.7:0/1325325927 learned_addr learned my addr 172.21.15.7:0/1325325927 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:37:53.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f13fff700 1 --2- 172.21.15.7:0/1325325927 >> v2:172.21.15.143:3300/0 conn(0x7f7f140c17f0 0x7f7f140c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f18d05700 1 -- 172.21.15.7:0/1325325927 >> v2:172.21.15.143:3300/0 conn(0x7f7f140c17f0 msgr2=0x7f7f140c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:37:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f18d05700 1 --2- 172.21.15.7:0/1325325927 >> v2:172.21.15.143:3300/0 conn(0x7f7f140c17f0 0x7f7f140c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f18d05700 1 -- 172.21.15.7:0/1325325927 >> v1:172.21.15.7:6789/0 conn(0x7f7f1406d6a0 legacy=0x7f7f1406dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f137fe700 1 -- 172.21.15.7:0/1325325927 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436450763 0 0) 0x7f7f140c5500 con 0x7f7f1406d6a0 2023-03-03T15:37:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f18d05700 1 -- 172.21.15.7:0/1325325927 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f140c2e70 con 0x7f7f140c6010 2023-03-03T15:37:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.024+0000 7f7f13fff700 1 --2- 172.21.15.7:0/1325325927 >> v2:172.21.15.143:3300/0 conn(0x7f7f140c17f0 0x7f7f140c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:37:53.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f18d05700 1 --2- 172.21.15.7:0/1325325927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 0x7f7f140c4a80 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f7f14066df0 tx=0x7f7f08009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ccb37f33061e134 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:53.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f137fe700 1 -- 172.21.15.7:0/1325325927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08011830 con 0x7f7f140c6010 2023-03-03T15:37:53.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f137fe700 1 -- 172.21.15.7:0/1325325927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f08011990 con 0x7f7f140c6010 2023-03-03T15:37:53.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f137fe700 1 -- 172.21.15.7:0/1325325927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08011d20 con 0x7f7f140c6010 2023-03-03T15:37:53.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f1af69700 1 -- 172.21.15.7:0/1325325927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 msgr2=0x7f7f140c4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:53.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f1af69700 1 --2- 172.21.15.7:0/1325325927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 0x7f7f140c4a80 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f7f14066df0 tx=0x7f7f08009b20 comp rx=0 tx=0).stop 2023-03-03T15:37:53.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f1af69700 1 -- 172.21.15.7:0/1325325927 shutdown_connections 2023-03-03T15:37:53.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f1af69700 1 --2- 172.21.15.7:0/1325325927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 0x7f7f140c4a80 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f1af69700 1 --2- 172.21.15.7:0/1325325927 >> v2:172.21.15.143:3300/0 conn(0x7f7f140c17f0 0x7f7f140c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f1af69700 1 -- 172.21.15.7:0/1325325927 >> 172.21.15.7:0/1325325927 conn(0x7f7f140a9430 msgr2=0x7f7f140c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:37:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f1af69700 1 -- 172.21.15.7:0/1325325927 shutdown_connections 2023-03-03T15:37:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.025+0000 7f7f1af69700 1 -- 172.21.15.7:0/1325325927 wait complete. 2023-03-03T15:37:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.027+0000 7f7f1af69700 1 Processor -- start 2023-03-03T15:37:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.027+0000 7f7f1af69700 1 -- start start 2023-03-03T15:37:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.027+0000 7f7f1af69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f1406d6a0 0x7f7f1414c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:53.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.027+0000 7f7f1af69700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f140c17f0 0x7f7f140702d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.027+0000 7f7f1af69700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 0x7f7f1414b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.027+0000 7f7f1af69700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1414beb0 con 0x7f7f140c6010 2023-03-03T15:37:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f1af69700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f14071650 con 0x7f7f140c17f0 2023-03-03T15:37:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f1af69700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f141ede80 con 0x7f7f1406d6a0 2023-03-03T15:37:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f19506700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 0x7f7f1414b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f19506700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 0x7f7f1414b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53722/0 (socket says 172.21.15.7:53722) 2023-03-03T15:37:53.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f19506700 1 -- 172.21.15.7:0/570179988 learned_addr learned my addr 172.21.15.7:0/570179988 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:37:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f13fff700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f140c17f0 0x7f7f140702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f18d05700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f1406d6a0 0x7f7f1414c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f19506700 1 -- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f1406d6a0 msgr2=0x7f7f1414c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f19506700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f1406d6a0 0x7f7f1414c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f19506700 1 -- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f140c17f0 msgr2=0x7f7f140702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f19506700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f140c17f0 0x7f7f140702d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f19506700 1 -- 172.21.15.7:0/570179988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f080094a0 con 0x7f7f140c6010 2023-03-03T15:37:53.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f19506700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 0x7f7f1414b970 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c002230 tx=0x7f7f0c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:53.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f117fa700 1 -- 172.21.15.7:0/570179988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c00aa60 con 0x7f7f140c6010 2023-03-03T15:37:53.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f117fa700 1 -- 172.21.15.7:0/570179988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0c00abe0 con 0x7f7f140c6010 2023-03-03T15:37:53.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f117fa700 1 -- 172.21.15.7:0/570179988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c0070d0 con 0x7f7f140c6010 2023-03-03T15:37:53.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f1af69700 1 -- 172.21.15.7:0/570179988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f140aa960 con 0x7f7f140c6010 2023-03-03T15:37:53.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.028+0000 7f7f1af69700 1 -- 172.21.15.7:0/570179988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f140aa680 con 0x7f7f140c6010 2023-03-03T15:37:53.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.029+0000 7f7f1af69700 1 -- 172.21.15.7:0/570179988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f140c2b00 con 0x7f7f140c6010 2023-03-03T15:37:53.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.030+0000 7f7f117fa700 1 -- 172.21.15.7:0/570179988 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7f0c010070 con 0x7f7f140c6010 2023-03-03T15:37:53.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.030+0000 7f7f117fa700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef80682f0 0x7f7ef806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:37:53.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.031+0000 7f7f117fa700 1 -- 172.21.15.7:0/570179988 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7f0c0859d0 con 0x7f7f140c6010 2023-03-03T15:37:53.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.031+0000 7f7f18d05700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef80682f0 0x7f7ef806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:37:53.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.031+0000 7f7f18d05700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef80682f0 0x7f7ef806a7a0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f7f080090b0 tx=0x7f7f0801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:37:53.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.035+0000 7f7f117fa700 1 -- 172.21.15.7:0/570179988 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7f0c051d10 con 0x7f7f140c6010 2023-03-03T15:37:53.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.274+0000 7f7f1af69700 1 -- 172.21.15.7:0/570179988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f1406fff0 con 0x7f7f140c6010 2023-03-03T15:37:53.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.275+0000 7f7f117fa700 1 -- 172.21.15.7:0/570179988 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7f1406fff0 con 0x7f7f140c6010 2023-03-03T15:37:53.277 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:37:53.277 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:37:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 -- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef80682f0 msgr2=0x7f7ef806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef80682f0 0x7f7ef806a7a0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f7f080090b0 tx=0x7f7f0801c040 comp rx=0 tx=0).stop 2023-03-03T15:37:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 -- 172.21.15.7:0/570179988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 msgr2=0x7f7f1414b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:37:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 0x7f7f1414b970 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c002230 tx=0x7f7f0c00cf50 comp rx=0 tx=0).stop 2023-03-03T15:37:53.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 -- 172.21.15.7:0/570179988 shutdown_connections 2023-03-03T15:37:53.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ef80682f0 0x7f7ef806a7a0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:53.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7f140c6010 0x7f7f1414b970 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:53.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7f1406d6a0 0x7f7f1414c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:53.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 --2- 172.21.15.7:0/570179988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7f140c17f0 0x7f7f140702d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:37:53.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 -- 172.21.15.7:0/570179988 >> 172.21.15.7:0/570179988 conn(0x7f7f140a9430 msgr2=0x7f7f140a9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:37:53.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 -- 172.21.15.7:0/570179988 shutdown_connections 2023-03-03T15:37:53.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:37:53.277+0000 7f7f1af69700 1 -- 172.21.15.7:0/570179988 wait complete. 2023-03-03T15:37:53.281 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:37:53.291 INFO:tasks.mds_thrash.fs.[a]:waiting for 108 secs before thrashing 2023-03-03T15:38:07.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:07.125 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:15.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:38:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.979+0000 7fbdb7f71700 1 Processor -- start 2023-03-03T15:38:15.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.980+0000 7fbdb7f71700 1 -- start start 2023-03-03T15:38:15.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.980+0000 7fbdb7f71700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdb0151cd0 0x7fbdb0154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:15.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.980+0000 7fbdb7f71700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb00c2e30 0x7fbdb0158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:15.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.980+0000 7fbdb7f71700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb0157ff0 con 0x7fbdb006c6c0 2023-03-03T15:38:15.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.980+0000 7fbdb7f71700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb006d640 con 0x7fbdb0151cd0 2023-03-03T15:38:15.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.980+0000 7fbdb7f71700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb006cda0 con 0x7fbdb00c2e30 2023-03-03T15:38:15.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.980+0000 7fbdb650e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbdb006c6c0 0x7fbdb00736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47732/0 (socket says 172.21.15.7:47732) 2023-03-03T15:38:15.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.980+0000 7fbdb650e700 1 -- 172.21.15.7:0/2980736491 learned_addr learned my addr 172.21.15.7:0/2980736491 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:38:15.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929107420 0 0) 0x7fbdb0157ff0 con 0x7fbdb006c6c0 2023-03-03T15:38:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb5d0d700 1 --2- 172.21.15.7:0/2980736491 >> v2:172.21.15.143:3300/0 conn(0x7fbdb0151cd0 0x7fbdb0154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb550c700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb00c2e30 0x7fbdb0158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd9c0035b0 con 0x7fbdb006c6c0 2023-03-03T15:38:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 628193423 0 0) 0x7fbd9c0035b0 con 0x7fbdb006c6c0 2023-03-03T15:38:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb00c2e30 msgr2=0x7fbdb0158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:38:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb00c2e30 0x7fbdb0158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 >> v2:172.21.15.143:3300/0 conn(0x7fbdb0151cd0 msgr2=0x7fbdb0154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:38:15.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 --2- 172.21.15.7:0/2980736491 >> v2:172.21.15.143:3300/0 conn(0x7fbdb0151cd0 0x7fbdb0154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:15.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb006d080 con 0x7fbdb006c6c0 2023-03-03T15:38:15.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fbda80019f0 con 0x7fbdb006c6c0 2023-03-03T15:38:15.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb550c700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb00c2e30 0x7fbdb0158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:38:15.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 >> v1:172.21.15.7:6789/0 conn(0x7fbdb006c6c0 legacy=0x7fbdb00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:15.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 0x7fbd9c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:15.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb00c2e30 0x7fbd9c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:15.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c004d90 0x7fbd9c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:15.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb006d640 con 0x7fbd9c003a70 2023-03-03T15:38:15.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb006cda0 con 0x7fbd9c004d90 2023-03-03T15:38:15.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c00bb70 con 0x7fbdb00c2e30 2023-03-03T15:38:15.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb5d0d700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 0x7fbd9c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:15.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb550c700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb00c2e30 0x7fbd9c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:15.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.981+0000 7fbdb650e700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c004d90 0x7fbd9c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:15.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb5d0d700 1 -- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb00c2e30 msgr2=0x7fbd9c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:38:15.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb5d0d700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb00c2e30 0x7fbd9c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:15.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb5d0d700 1 -- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c004d90 msgr2=0x7fbd9c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:38:15.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb5d0d700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c004d90 0x7fbd9c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:15.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb5d0d700 1 -- 172.21.15.7:0/2980736491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c0035b0 con 0x7fbd9c003a70 2023-03-03T15:38:15.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb550c700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb00c2e30 0x7fbd9c004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:38:15.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb650e700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd9c004d90 0x7fbd9c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:38:15.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb5d0d700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 0x7fbd9c003e80 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7fbda0009b30 tx=0x7fbd9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9ee43bd4ba756b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:15.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda000e6c0 con 0x7fbd9c003a70 2023-03-03T15:38:15.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda000e820 con 0x7fbd9c003a70 2023-03-03T15:38:15.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb4d0b700 1 -- 172.21.15.7:0/2980736491 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda000eb90 con 0x7fbd9c003a70 2023-03-03T15:38:15.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb7f71700 1 -- 172.21.15.7:0/2980736491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 msgr2=0x7fbd9c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:15.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb7f71700 1 --2- 172.21.15.7:0/2980736491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 0x7fbd9c003e80 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7fbda0009b30 tx=0x7fbd9c005880 comp rx=0 tx=0).stop 2023-03-03T15:38:15.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.982+0000 7fbdb650e700 1 -- 172.21.15.7:0/2980736491 reap_dead start 2023-03-03T15:38:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.983+0000 7fbdb7f71700 1 -- 172.21.15.7:0/2980736491 shutdown_connections 2023-03-03T15:38:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.983+0000 7fbdb7f71700 1 -- 172.21.15.7:0/2980736491 >> 172.21.15.7:0/2980736491 conn(0x7fbdb00a94d0 msgr2=0x7fbdb00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:38:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.983+0000 7fbdb7f71700 1 -- 172.21.15.7:0/2980736491 shutdown_connections 2023-03-03T15:38:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.983+0000 7fbdb7f71700 1 -- 172.21.15.7:0/2980736491 wait complete. 2023-03-03T15:38:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.984+0000 7fbdb7f71700 1 Processor -- start 2023-03-03T15:38:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.984+0000 7fbdb7f71700 1 -- start start 2023-03-03T15:38:15.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb7f71700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 0x7fbdb01ed590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb7f71700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb0155c30 0x7fbdb01560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb7f71700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb01565e0 0x7fbdb00707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb7f71700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb014f990 con 0x7fbd9c003a70 2023-03-03T15:38:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb7f71700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb0153dc0 con 0x7fbdb0155c30 2023-03-03T15:38:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb7f71700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb01eddd0 con 0x7fbdb01565e0 2023-03-03T15:38:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb5d0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 0x7fbdb01ed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb5d0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 0x7fbdb01ed590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53762/0 (socket says 172.21.15.7:53762) 2023-03-03T15:38:15.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb5d0d700 1 -- 172.21.15.7:0/355218482 learned_addr learned my addr 172.21.15.7:0/355218482 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:38:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb650e700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb01565e0 0x7fbdb00707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb550c700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb0155c30 0x7fbdb01560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb5d0d700 1 -- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb01565e0 msgr2=0x7fbdb00707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb5d0d700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb01565e0 0x7fbdb00707e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb5d0d700 1 -- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb0155c30 msgr2=0x7fbdb01560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb5d0d700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb0155c30 0x7fbdb01560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:15.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb5d0d700 1 -- 172.21.15.7:0/355218482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0157c80 con 0x7fbd9c003a70 2023-03-03T15:38:15.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb650e700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb01565e0 0x7fbdb00707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:38:15.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb5d0d700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 0x7fbdb01ed590 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00bac0 tx=0x7fbda00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:15.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbda67fc700 1 -- 172.21.15.7:0/355218482 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda0021070 con 0x7fbd9c003a70 2023-03-03T15:38:15.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbda67fc700 1 -- 172.21.15.7:0/355218482 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda0004590 con 0x7fbd9c003a70 2023-03-03T15:38:15.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb7f71700 1 -- 172.21.15.7:0/355218482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb006d360 con 0x7fbd9c003a70 2023-03-03T15:38:15.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbda67fc700 1 -- 172.21.15.7:0/355218482 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda000e6c0 con 0x7fbd9c003a70 2023-03-03T15:38:15.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.985+0000 7fbdb7f71700 1 -- 172.21.15.7:0/355218482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb0154a50 con 0x7fbd9c003a70 2023-03-03T15:38:15.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.986+0000 7fbdb7f71700 1 -- 172.21.15.7:0/355218482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb0074bb0 con 0x7fbd9c003a70 2023-03-03T15:38:15.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.987+0000 7fbda67fc700 1 -- 172.21.15.7:0/355218482 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbda002e030 con 0x7fbd9c003a70 2023-03-03T15:38:15.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.987+0000 7fbda67fc700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd940682f0 0x7fbd9406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:38:15.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.987+0000 7fbda67fc700 1 -- 172.21.15.7:0/355218482 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbda00873b0 con 0x7fbd9c003a70 2023-03-03T15:38:15.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.987+0000 7fbdb550c700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd940682f0 0x7fbd9406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:38:15.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.988+0000 7fbdb550c700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd940682f0 0x7fbd9406a7a0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00721d0 tx=0x7fbdac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:38:15.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:15.992+0000 7fbda67fc700 1 -- 172.21.15.7:0/355218482 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbda0053750 con 0x7fbd9c003a70 2023-03-03T15:38:16.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.230+0000 7fbdb7f71700 1 -- 172.21.15.7:0/355218482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdb00a9fb0 con 0x7fbd9c003a70 2023-03-03T15:38:16.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.231+0000 7fbda67fc700 1 -- 172.21.15.7:0/355218482 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbda0056d70 con 0x7fbd9c003a70 2023-03-03T15:38:16.233 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:38:16.233 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:38:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 -- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd940682f0 msgr2=0x7fbd9406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd940682f0 0x7fbd9406a7a0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00721d0 tx=0x7fbdac008040 comp rx=0 tx=0).stop 2023-03-03T15:38:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 -- 172.21.15.7:0/355218482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 msgr2=0x7fbdb01ed590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:38:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 0x7fbdb01ed590 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00bac0 tx=0x7fbda00047c0 comp rx=0 tx=0).stop 2023-03-03T15:38:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 -- 172.21.15.7:0/355218482 shutdown_connections 2023-03-03T15:38:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb01565e0 0x7fbdb00707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd940682f0 0x7fbd9406a7a0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd9c003a70 0x7fbdb01ed590 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 --2- 172.21.15.7:0/355218482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb0155c30 0x7fbdb01560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:38:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 -- 172.21.15.7:0/355218482 >> 172.21.15.7:0/355218482 conn(0x7fbdb00a94d0 msgr2=0x7fbdb0065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:38:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 -- 172.21.15.7:0/355218482 shutdown_connections 2023-03-03T15:38:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:38:16.233+0000 7fbdb7f71700 1 -- 172.21.15.7:0/355218482 wait complete. 2023-03-03T15:38:16.237 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:38:16.246 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-03-03T15:38:37.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:38:37.206 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:07.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:07.266 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:16.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:39:16.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.384+0000 7ff8e9755700 1 Processor -- start 2023-03-03T15:39:16.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.384+0000 7ff8e9755700 1 -- start start 2023-03-03T15:39:16.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.385+0000 7ff8e9755700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff8e40caa30 0x7ff8e40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:16.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.385+0000 7ff8e9755700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e40c7310 con 0x7ff8e4154ff0 2023-03-03T15:39:16.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.385+0000 7ff8e9755700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e40bfd10 con 0x7ff8e40c00e0 2023-03-03T15:39:16.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.385+0000 7ff8e9755700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e4155430 con 0x7ff8e40caa30 2023-03-03T15:39:16.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e37fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff8e4154ff0 0x7ff8e415ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47808/0 (socket says 172.21.15.7:47808) 2023-03-03T15:39:16.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e37fe700 1 -- 172.21.15.7:0/3304228626 learned_addr learned my addr 172.21.15.7:0/3304228626 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:39:16.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e2ffd700 1 --2- 172.21.15.7:0/3304228626 >> v2:172.21.15.7:3300/0 conn(0x7ff8e40caa30 0x7ff8e40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:16.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e1ffb700 1 -- 172.21.15.7:0/3304228626 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135793230 0 0) 0x7ff8e40c7310 con 0x7ff8e4154ff0 2023-03-03T15:39:16.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e1ffb700 1 -- 172.21.15.7:0/3304228626 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d00035b0 con 0x7ff8e4154ff0 2023-03-03T15:39:16.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e2ffd700 1 -- 172.21.15.7:0/3304228626 >> v1:172.21.15.143:6789/0 conn(0x7ff8e40c00e0 legacy=0x7ff8e40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:16.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e2ffd700 1 -- 172.21.15.7:0/3304228626 >> v1:172.21.15.7:6789/0 conn(0x7ff8e4154ff0 legacy=0x7ff8e415ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:16.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e2ffd700 1 -- 172.21.15.7:0/3304228626 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e40bfa30 con 0x7ff8e40caa30 2023-03-03T15:39:16.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e2ffd700 1 --2- 172.21.15.7:0/3304228626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e40caa30 0x7ff8e40c5c60 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009840 tx=0x7ff8cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1017999d3a6e108a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:16.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e1ffb700 1 -- 172.21.15.7:0/3304228626 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc011830 con 0x7ff8e40caa30 2023-03-03T15:39:16.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e1ffb700 1 -- 172.21.15.7:0/3304228626 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc011990 con 0x7ff8e40caa30 2023-03-03T15:39:16.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.386+0000 7ff8e1ffb700 1 -- 172.21.15.7:0/3304228626 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc011d20 con 0x7ff8e40caa30 2023-03-03T15:39:16.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.387+0000 7ff8e9755700 1 -- 172.21.15.7:0/3304228626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e40caa30 msgr2=0x7ff8e40c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:16.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.387+0000 7ff8e9755700 1 --2- 172.21.15.7:0/3304228626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e40caa30 0x7ff8e40c5c60 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009840 tx=0x7ff8cc009b50 comp rx=0 tx=0).stop 2023-03-03T15:39:16.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.387+0000 7ff8e9755700 1 -- 172.21.15.7:0/3304228626 shutdown_connections 2023-03-03T15:39:16.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.387+0000 7ff8e9755700 1 --2- 172.21.15.7:0/3304228626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e40caa30 0x7ff8e40c5c60 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:16.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.387+0000 7ff8e9755700 1 -- 172.21.15.7:0/3304228626 >> 172.21.15.7:0/3304228626 conn(0x7ff8e4066e40 msgr2=0x7ff8e4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:39:16.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.387+0000 7ff8e9755700 1 -- 172.21.15.7:0/3304228626 shutdown_connections 2023-03-03T15:39:16.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.387+0000 7ff8e9755700 1 -- 172.21.15.7:0/3304228626 wait complete. 2023-03-03T15:39:16.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.389+0000 7ff8e9755700 1 Processor -- start 2023-03-03T15:39:16.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.389+0000 7ff8e9755700 1 -- start start 2023-03-03T15:39:16.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e9755700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8e40c00e0 0x7ff8e41df720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:16.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e9755700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8e40caa30 0x7ff8e40c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:16.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e9755700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e4154ff0 0x7ff8e41564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:16.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e9755700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e41de230 con 0x7ff8e4154ff0 2023-03-03T15:39:16.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e9755700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e40c4790 con 0x7ff8e40c00e0 2023-03-03T15:39:16.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e9755700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e41ddbe0 con 0x7ff8e40caa30 2023-03-03T15:39:16.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e4154ff0 0x7ff8e41564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:16.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e4154ff0 0x7ff8e41564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53834/0 (socket says 172.21.15.7:53834) 2023-03-03T15:39:16.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e37fe700 1 -- 172.21.15.7:0/1143235807 learned_addr learned my addr 172.21.15.7:0/1143235807 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:39:16.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e2ffd700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8e40c00e0 0x7ff8e41df720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:16.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e27fc700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8e40caa30 0x7ff8e40c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:16.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e37fe700 1 -- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8e40caa30 msgr2=0x7ff8e40c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:16.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e37fe700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8e40caa30 0x7ff8e40c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:16.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e37fe700 1 -- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8e40c00e0 msgr2=0x7ff8e41df720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:16.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e37fe700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8e40c00e0 0x7ff8e41df720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:16.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e37fe700 1 -- 172.21.15.7:0/1143235807 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc0094a0 con 0x7ff8e4154ff0 2023-03-03T15:39:16.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e27fc700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8e40caa30 0x7ff8e40c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:39:16.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e2ffd700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8e40c00e0 0x7ff8e41df720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:39:16.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e37fe700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e4154ff0 0x7ff8e41564c0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00cb40 tx=0x7ff8dc00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:16.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8db7fe700 1 -- 172.21.15.7:0/1143235807 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc00ab50 con 0x7ff8e4154ff0 2023-03-03T15:39:16.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8db7fe700 1 -- 172.21.15.7:0/1143235807 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc00acb0 con 0x7ff8e4154ff0 2023-03-03T15:39:16.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8e9755700 1 -- 172.21.15.7:0/1143235807 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e41547f0 con 0x7ff8e4154ff0 2023-03-03T15:39:16.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.390+0000 7ff8db7fe700 1 -- 172.21.15.7:0/1143235807 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc012620 con 0x7ff8e4154ff0 2023-03-03T15:39:16.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.391+0000 7ff8e9755700 1 -- 172.21.15.7:0/1143235807 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e4154510 con 0x7ff8e4154ff0 2023-03-03T15:39:16.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.392+0000 7ff8e9755700 1 -- 172.21.15.7:0/1143235807 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e4154ad0 con 0x7ff8e4154ff0 2023-03-03T15:39:16.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.392+0000 7ff8db7fe700 1 -- 172.21.15.7:0/1143235807 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff8dc019030 con 0x7ff8e4154ff0 2023-03-03T15:39:16.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.392+0000 7ff8db7fe700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8c40682f0 0x7ff8c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:16.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.392+0000 7ff8db7fe700 1 -- 172.21.15.7:0/1143235807 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff8dc085e20 con 0x7ff8e4154ff0 2023-03-03T15:39:16.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.392+0000 7ff8e2ffd700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8c40682f0 0x7ff8c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:16.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.393+0000 7ff8e2ffd700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8c40682f0 0x7ff8c406a7a0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0090b0 tx=0x7ff8cc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:16.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.397+0000 7ff8db7fe700 1 -- 172.21.15.7:0/1143235807 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff8dc0521c0 con 0x7ff8e4154ff0 2023-03-03T15:39:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.636+0000 7ff8e9755700 1 -- 172.21.15.7:0/1143235807 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8e41dda40 con 0x7ff8e4154ff0 2023-03-03T15:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.636+0000 7ff8db7fe700 1 -- 172.21.15.7:0/1143235807 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff8e41dda40 con 0x7ff8e4154ff0 2023-03-03T15:39:16.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:39:16.639 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:39:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 -- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8c40682f0 msgr2=0x7ff8c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8c40682f0 0x7ff8c406a7a0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0090b0 tx=0x7ff8cc008030 comp rx=0 tx=0).stop 2023-03-03T15:39:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 -- 172.21.15.7:0/1143235807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e4154ff0 msgr2=0x7ff8e41564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e4154ff0 0x7ff8e41564c0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00cb40 tx=0x7ff8dc00cf00 comp rx=0 tx=0).stop 2023-03-03T15:39:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 -- 172.21.15.7:0/1143235807 shutdown_connections 2023-03-03T15:39:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8c40682f0 0x7ff8c406a7a0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e4154ff0 0x7ff8e41564c0 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8e40c00e0 0x7ff8e41df720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 --2- 172.21.15.7:0/1143235807 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8e40caa30 0x7ff8e40c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 -- 172.21.15.7:0/1143235807 >> 172.21.15.7:0/1143235807 conn(0x7ff8e4066e40 msgr2=0x7ff8e4153d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:39:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 -- 172.21.15.7:0/1143235807 shutdown_connections 2023-03-03T15:39:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:16.638+0000 7ff8e9755700 1 -- 172.21.15.7:0/1143235807 wait complete. 2023-03-03T15:39:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:39:16.652 INFO:tasks.mds_thrash.fs.[b]:waiting for 21 secs before thrashing 2023-03-03T15:39:37.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:37.346 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:39:37.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:39:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.790+0000 7fde695bb700 1 Processor -- start 2023-03-03T15:39:37.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.790+0000 7fde695bb700 1 -- start start 2023-03-03T15:39:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde695bb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde640caa30 0x7fde640c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde695bb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde640c00e0 0x7fde64154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde695bb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde640c7310 con 0x7fde640cae00 2023-03-03T15:39:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde695bb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde640cb240 con 0x7fde640c00e0 2023-03-03T15:39:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde695bb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde640bfd10 con 0x7fde640caa30 2023-03-03T15:39:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde627fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde640c00e0 0x7fde64154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde627fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fde640c00e0 0x7fde64154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53856/0 (socket says 172.21.15.7:53856) 2023-03-03T15:39:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde627fc700 1 -- 172.21.15.7:0/1758559073 learned_addr learned my addr 172.21.15.7:0/1758559073 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:39:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde62ffd700 1 --2- 172.21.15.7:0/1758559073 >> v2:172.21.15.143:3300/0 conn(0x7fde640caa30 0x7fde640c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde627fc700 1 -- 172.21.15.7:0/1758559073 >> v2:172.21.15.143:3300/0 conn(0x7fde640caa30 msgr2=0x7fde640c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:39:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde627fc700 1 --2- 172.21.15.7:0/1758559073 >> v2:172.21.15.143:3300/0 conn(0x7fde640caa30 0x7fde640c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde627fc700 1 -- 172.21.15.7:0/1758559073 >> v1:172.21.15.143:6789/0 conn(0x7fde640cae00 legacy=0x7fde640c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.791+0000 7fde627fc700 1 -- 172.21.15.7:0/1758559073 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde640bfa30 con 0x7fde640c00e0 2023-03-03T15:39:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde627fc700 1 --2- 172.21.15.7:0/1758559073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640c00e0 0x7fde64154ff0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fde640c81b0 tx=0x7fde54009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a741e366106ba63d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde61ffb700 1 -- 172.21.15.7:0/1758559073 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54005e30 con 0x7fde640c00e0 2023-03-03T15:39:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde61ffb700 1 -- 172.21.15.7:0/1758559073 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde54004030 con 0x7fde640c00e0 2023-03-03T15:39:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde61ffb700 1 -- 172.21.15.7:0/1758559073 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde540043e0 con 0x7fde640c00e0 2023-03-03T15:39:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde695bb700 1 -- 172.21.15.7:0/1758559073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640c00e0 msgr2=0x7fde64154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:37.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde695bb700 1 --2- 172.21.15.7:0/1758559073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640c00e0 0x7fde64154ff0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fde640c81b0 tx=0x7fde54009bd0 comp rx=0 tx=0).stop 2023-03-03T15:39:37.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde695bb700 1 -- 172.21.15.7:0/1758559073 shutdown_connections 2023-03-03T15:39:37.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde695bb700 1 --2- 172.21.15.7:0/1758559073 >> v2:172.21.15.143:3300/0 conn(0x7fde640caa30 0x7fde640c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:37.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde695bb700 1 --2- 172.21.15.7:0/1758559073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640c00e0 0x7fde64154ff0 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:37.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde695bb700 1 -- 172.21.15.7:0/1758559073 >> 172.21.15.7:0/1758559073 conn(0x7fde64066e40 msgr2=0x7fde64071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:39:37.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde695bb700 1 -- 172.21.15.7:0/1758559073 shutdown_connections 2023-03-03T15:39:37.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.792+0000 7fde695bb700 1 -- 172.21.15.7:0/1758559073 wait complete. 2023-03-03T15:39:37.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.794+0000 7fde695bb700 1 Processor -- start 2023-03-03T15:39:37.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.794+0000 7fde695bb700 1 -- start start 2023-03-03T15:39:37.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.794+0000 7fde695bb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde640c00e0 0x7fde641deb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:37.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.794+0000 7fde695bb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640caa30 0x7fde6414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:37.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.794+0000 7fde695bb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde640cae00 0x7fde641ed960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:37.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.794+0000 7fde695bb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde640c2840 con 0x7fde640caa30 2023-03-03T15:39:37.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.794+0000 7fde695bb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde641ed4d0 con 0x7fde640cae00 2023-03-03T15:39:37.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.794+0000 7fde695bb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde641df0c0 con 0x7fde640c00e0 2023-03-03T15:39:37.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde627fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640caa30 0x7fde6414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde627fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640caa30 0x7fde6414cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53862/0 (socket says 172.21.15.7:53862) 2023-03-03T15:39:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde627fc700 1 -- 172.21.15.7:0/3000272465 learned_addr learned my addr 172.21.15.7:0/3000272465 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:39:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde62ffd700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde640c00e0 0x7fde641deb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde637fe700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde640cae00 0x7fde641ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde627fc700 1 -- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde640c00e0 msgr2=0x7fde641deb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde627fc700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde640c00e0 0x7fde641deb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde627fc700 1 -- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde640cae00 msgr2=0x7fde641ed960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:37.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde627fc700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde640cae00 0x7fde641ed960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde627fc700 1 -- 172.21.15.7:0/3000272465 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde54009500 con 0x7fde640caa30 2023-03-03T15:39:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde627fc700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640caa30 0x7fde6414cb60 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7fde6406b270 tx=0x7fde54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde4b7fe700 1 -- 172.21.15.7:0/3000272465 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54003c30 con 0x7fde640caa30 2023-03-03T15:39:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde4b7fe700 1 -- 172.21.15.7:0/3000272465 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde54003d90 con 0x7fde640caa30 2023-03-03T15:39:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde695bb700 1 -- 172.21.15.7:0/3000272465 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde641547f0 con 0x7fde640caa30 2023-03-03T15:39:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde4b7fe700 1 -- 172.21.15.7:0/3000272465 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5401c640 con 0x7fde640caa30 2023-03-03T15:39:37.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.795+0000 7fde695bb700 1 -- 172.21.15.7:0/3000272465 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde64154510 con 0x7fde640caa30 2023-03-03T15:39:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.796+0000 7fde695bb700 1 -- 172.21.15.7:0/3000272465 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde64154ad0 con 0x7fde640caa30 2023-03-03T15:39:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.797+0000 7fde4b7fe700 1 -- 172.21.15.7:0/3000272465 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fde5402c070 con 0x7fde640caa30 2023-03-03T15:39:37.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.797+0000 7fde4b7fe700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde500682f0 0x7fde5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.797+0000 7fde4b7fe700 1 -- 172.21.15.7:0/3000272465 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fde54086fd0 con 0x7fde640caa30 2023-03-03T15:39:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.797+0000 7fde62ffd700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde500682f0 0x7fde5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.798+0000 7fde62ffd700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde500682f0 0x7fde5006a7a0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fde64154450 tx=0x7fde58009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:37.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:37.802+0000 7fde4b7fe700 1 -- 172.21.15.7:0/3000272465 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fde54053370 con 0x7fde640caa30 2023-03-03T15:39:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.040+0000 7fde695bb700 1 -- 172.21.15.7:0/3000272465 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde6404e6e0 con 0x7fde640caa30 2023-03-03T15:39:38.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.040+0000 7fde4b7fe700 1 -- 172.21.15.7:0/3000272465 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fde54056990 con 0x7fde640caa30 2023-03-03T15:39:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:39:38.042 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:39:38.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 -- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde500682f0 msgr2=0x7fde5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:38.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde500682f0 0x7fde5006a7a0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fde64154450 tx=0x7fde58009210 comp rx=0 tx=0).stop 2023-03-03T15:39:38.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 -- 172.21.15.7:0/3000272465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640caa30 msgr2=0x7fde6414cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:38.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640caa30 0x7fde6414cb60 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7fde6406b270 tx=0x7fde54004750 comp rx=0 tx=0).stop 2023-03-03T15:39:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 -- 172.21.15.7:0/3000272465 shutdown_connections 2023-03-03T15:39:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde640cae00 0x7fde641ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde640c00e0 0x7fde641deb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde500682f0 0x7fde5006a7a0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 --2- 172.21.15.7:0/3000272465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde640caa30 0x7fde6414cb60 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 -- 172.21.15.7:0/3000272465 >> 172.21.15.7:0/3000272465 conn(0x7fde64066e40 msgr2=0x7fde640c7720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:39:38.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.042+0000 7fde695bb700 1 -- 172.21.15.7:0/3000272465 shutdown_connections 2023-03-03T15:39:38.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:38.043+0000 7fde695bb700 1 -- 172.21.15.7:0/3000272465 wait complete. 2023-03-03T15:39:38.046 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:39:38.056 INFO:tasks.mds_thrash.fs.[b]:waiting for 91 secs before thrashing 2023-03-03T15:39:41.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:39:41.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.429+0000 7fdc9610a700 1 Processor -- start 2023-03-03T15:39:41.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.429+0000 7fdc9610a700 1 -- start start 2023-03-03T15:39:41.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc9610a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc90151d40 0x7fdc900736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:41.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc9610a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdc90073c30 0x7fdc901581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:41.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc9610a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc9006d430 con 0x7fdc900c2930 2023-03-03T15:39:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc9610a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc9006c2a0 con 0x7fdc90073c30 2023-03-03T15:39:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc9610a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc901586e0 con 0x7fdc90151d40 2023-03-03T15:39:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc8ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdc90073c30 0x7fdc901581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc8effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdc900c2930 0x7fdc90157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47854/0 (socket says 172.21.15.7:47854) 2023-03-03T15:39:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc8effd700 1 -- 172.21.15.7:0/2132620401 learned_addr learned my addr 172.21.15.7:0/2132620401 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:39:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc8f7fe700 1 --2- 172.21.15.7:0/2132620401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc90151d40 0x7fdc900736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:41.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc8ffff700 1 -- 172.21.15.7:0/2132620401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc90151d40 msgr2=0x7fdc900736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:39:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc8ffff700 1 --2- 172.21.15.7:0/2132620401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc90151d40 0x7fdc900736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc8ffff700 1 -- 172.21.15.7:0/2132620401 >> v1:172.21.15.7:6789/0 conn(0x7fdc900c2930 legacy=0x7fdc90157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc8ffff700 1 -- 172.21.15.7:0/2132620401 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9006cb00 con 0x7fdc90073c30 2023-03-03T15:39:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc8e7fc700 1 -- 172.21.15.7:0/2132620401 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329354583 0 0) 0x7fdc9006d430 con 0x7fdc900c2930 2023-03-03T15:39:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.430+0000 7fdc8ffff700 1 --2- 172.21.15.7:0/2132620401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90073c30 0x7fdc901581a0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009d90 tx=0x7fdc8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108d740bab45c074 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc8e7fc700 1 -- 172.21.15.7:0/2132620401 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8800e070 con 0x7fdc90073c30 2023-03-03T15:39:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc8e7fc700 1 -- 172.21.15.7:0/2132620401 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8800bb20 con 0x7fdc90073c30 2023-03-03T15:39:41.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc8e7fc700 1 -- 172.21.15.7:0/2132620401 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88012430 con 0x7fdc90073c30 2023-03-03T15:39:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc9610a700 1 -- 172.21.15.7:0/2132620401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90073c30 msgr2=0x7fdc901581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc9610a700 1 --2- 172.21.15.7:0/2132620401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90073c30 0x7fdc901581a0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009d90 tx=0x7fdc8800b330 comp rx=0 tx=0).stop 2023-03-03T15:39:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc9610a700 1 -- 172.21.15.7:0/2132620401 shutdown_connections 2023-03-03T15:39:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc9610a700 1 --2- 172.21.15.7:0/2132620401 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90073c30 0x7fdc901581a0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc9610a700 1 --2- 172.21.15.7:0/2132620401 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc90151d40 0x7fdc900736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:41.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc9610a700 1 -- 172.21.15.7:0/2132620401 >> 172.21.15.7:0/2132620401 conn(0x7fdc900a94d0 msgr2=0x7fdc900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:39:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc9610a700 1 -- 172.21.15.7:0/2132620401 shutdown_connections 2023-03-03T15:39:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.431+0000 7fdc9610a700 1 -- 172.21.15.7:0/2132620401 wait complete. 2023-03-03T15:39:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.432+0000 7fdc9610a700 1 Processor -- start 2023-03-03T15:39:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.433+0000 7fdc9610a700 1 -- start start 2023-03-03T15:39:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.433+0000 7fdc9610a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc90073c30 0x7fdc90158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.433+0000 7fdc9610a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900c2930 0x7fdc901e0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.433+0000 7fdc9610a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90151d40 0x7fdc901eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:41.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.433+0000 7fdc9610a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc901dfcd0 con 0x7fdc90151d40 2023-03-03T15:39:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.433+0000 7fdc9610a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc901edd30 con 0x7fdc90073c30 2023-03-03T15:39:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.433+0000 7fdc9610a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc901eb3d0 con 0x7fdc900c2930 2023-03-03T15:39:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.433+0000 7fdc8ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90151d40 0x7fdc901eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.433+0000 7fdc8ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90151d40 0x7fdc901eacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53882/0 (socket says 172.21.15.7:53882) 2023-03-03T15:39:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.433+0000 7fdc8ffff700 1 -- 172.21.15.7:0/3519363756 learned_addr learned my addr 172.21.15.7:0/3519363756 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:39:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc8f7fe700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc90073c30 0x7fdc90158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc8ffff700 1 -- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900c2930 msgr2=0x7fdc901e0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:41.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc8effd700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900c2930 0x7fdc901e0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:41.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc8ffff700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900c2930 0x7fdc901e0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:41.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc8ffff700 1 -- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc90073c30 msgr2=0x7fdc90158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:41.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc8ffff700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc90073c30 0x7fdc90158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:41.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc8ffff700 1 -- 172.21.15.7:0/3519363756 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc88009a70 con 0x7fdc90151d40 2023-03-03T15:39:41.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc8f7fe700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc90073c30 0x7fdc90158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:39:41.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc8ffff700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90151d40 0x7fdc901eacf0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fdc9006bb70 tx=0x7fdc88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:41.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc87fff700 1 -- 172.21.15.7:0/3519363756 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc880098f0 con 0x7fdc90151d40 2023-03-03T15:39:41.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc87fff700 1 -- 172.21.15.7:0/3519363756 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88020070 con 0x7fdc90151d40 2023-03-03T15:39:41.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc9610a700 1 -- 172.21.15.7:0/3519363756 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9006d0c0 con 0x7fdc90151d40 2023-03-03T15:39:41.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc87fff700 1 -- 172.21.15.7:0/3519363756 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc880276a0 con 0x7fdc90151d40 2023-03-03T15:39:41.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.434+0000 7fdc9610a700 1 -- 172.21.15.7:0/3519363756 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9006cde0 con 0x7fdc90151d40 2023-03-03T15:39:41.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.435+0000 7fdc9610a700 1 -- 172.21.15.7:0/3519363756 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc9006bea0 con 0x7fdc90151d40 2023-03-03T15:39:41.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.436+0000 7fdc87fff700 1 -- 172.21.15.7:0/3519363756 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc88012430 con 0x7fdc90151d40 2023-03-03T15:39:41.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.436+0000 7fdc87fff700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc700682f0 0x7fdc7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:39:41.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.436+0000 7fdc87fff700 1 -- 172.21.15.7:0/3519363756 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdc88022070 con 0x7fdc90151d40 2023-03-03T15:39:41.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.436+0000 7fdc8f7fe700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc700682f0 0x7fdc7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:39:41.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.437+0000 7fdc8f7fe700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc700682f0 0x7fdc7006a7a0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009b60 tx=0x7fdc80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:39:41.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.440+0000 7fdc87fff700 1 -- 172.21.15.7:0/3519363756 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc88054600 con 0x7fdc90151d40 2023-03-03T15:39:41.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.676+0000 7fdc9610a700 1 -- 172.21.15.7:0/3519363756 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc901dfe80 con 0x7fdc90151d40 2023-03-03T15:39:41.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.676+0000 7fdc87fff700 1 -- 172.21.15.7:0/3519363756 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdc901dfe80 con 0x7fdc90151d40 2023-03-03T15:39:41.678 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:39:41.678 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:39:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 -- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc700682f0 msgr2=0x7fdc7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:41.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc700682f0 0x7fdc7006a7a0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009b60 tx=0x7fdc80008040 comp rx=0 tx=0).stop 2023-03-03T15:39:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 -- 172.21.15.7:0/3519363756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90151d40 msgr2=0x7fdc901eacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:39:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90151d40 0x7fdc901eacf0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fdc9006bb70 tx=0x7fdc88004ad0 comp rx=0 tx=0).stop 2023-03-03T15:39:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 -- 172.21.15.7:0/3519363756 shutdown_connections 2023-03-03T15:39:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc700682f0 0x7fdc7006a7a0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc90151d40 0x7fdc901eacf0 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc90073c30 0x7fdc90158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 --2- 172.21.15.7:0/3519363756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc900c2930 0x7fdc901e0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:39:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 -- 172.21.15.7:0/3519363756 >> 172.21.15.7:0/3519363756 conn(0x7fdc900a94d0 msgr2=0x7fdc900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:39:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 -- 172.21.15.7:0/3519363756 shutdown_connections 2023-03-03T15:39:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:39:41.679+0000 7fdc9610a700 1 -- 172.21.15.7:0/3519363756 wait complete. 2023-03-03T15:39:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:39:41.692 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T15:40:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T15:39:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T15:40:07.414 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:07.417 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:37.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:37.475 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:40:46.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:40:46.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.831+0000 7f322bb5f700 1 Processor -- start 2023-03-03T15:40:46.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.831+0000 7f322bb5f700 1 -- start start 2023-03-03T15:40:46.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f322bb5f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3224151cd0 0x7f3224154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:46.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f322bb5f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f322406c6c0 0x7f32240736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:46.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f322bb5f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3224157ff0 con 0x7f32240c2e30 2023-03-03T15:40:46.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f322bb5f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322406d640 con 0x7f3224151cd0 2023-03-03T15:40:46.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f322bb5f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322406cda0 con 0x7f322406c6c0 2023-03-03T15:40:46.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f32298fb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3224151cd0 0x7f3224154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:46.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f32298fb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3224151cd0 0x7f3224154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53930/0 (socket says 172.21.15.7:53930) 2023-03-03T15:40:46.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f32298fb700 1 -- 172.21.15.7:0/1226022634 learned_addr learned my addr 172.21.15.7:0/1226022634 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:40:46.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f322a0fc700 1 --2- 172.21.15.7:0/1226022634 >> v2:172.21.15.143:3300/0 conn(0x7f322406c6c0 0x7f32240736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:46.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f32298fb700 1 -- 172.21.15.7:0/1226022634 >> v2:172.21.15.143:3300/0 conn(0x7f322406c6c0 msgr2=0x7f32240736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:40:46.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f32298fb700 1 --2- 172.21.15.7:0/1226022634 >> v2:172.21.15.143:3300/0 conn(0x7f322406c6c0 0x7f32240736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:46.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f32298fb700 1 -- 172.21.15.7:0/1226022634 >> v1:172.21.15.143:6789/0 conn(0x7f32240c2e30 legacy=0x7f3224158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:40:46.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f32298fb700 1 -- 172.21.15.7:0/1226022634 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322406d080 con 0x7f3224151cd0 2023-03-03T15:40:46.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.832+0000 7f322a0fc700 1 --2- 172.21.15.7:0/1226022634 >> v2:172.21.15.143:3300/0 conn(0x7f322406c6c0 0x7f32240736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:40:46.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f32298fb700 1 --2- 172.21.15.7:0/1226022634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 0x7f3224154510 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f32240c38a0 tx=0x7f3214009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9812e64ba1c816 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:40:46.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f32288f9700 1 -- 172.21.15.7:0/1226022634 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214004230 con 0x7f3224151cd0 2023-03-03T15:40:46.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f32288f9700 1 -- 172.21.15.7:0/1226022634 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3214004390 con 0x7f3224151cd0 2023-03-03T15:40:46.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f32288f9700 1 -- 172.21.15.7:0/1226022634 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32140036a0 con 0x7f3224151cd0 2023-03-03T15:40:46.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f322bb5f700 1 -- 172.21.15.7:0/1226022634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 msgr2=0x7f3224154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:46.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f322bb5f700 1 --2- 172.21.15.7:0/1226022634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 0x7f3224154510 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f32240c38a0 tx=0x7f3214009bd0 comp rx=0 tx=0).stop 2023-03-03T15:40:46.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f322bb5f700 1 -- 172.21.15.7:0/1226022634 shutdown_connections 2023-03-03T15:40:46.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f322bb5f700 1 --2- 172.21.15.7:0/1226022634 >> v2:172.21.15.143:3300/0 conn(0x7f322406c6c0 0x7f32240736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:46.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f322bb5f700 1 --2- 172.21.15.7:0/1226022634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 0x7f3224154510 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:46.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f322bb5f700 1 -- 172.21.15.7:0/1226022634 >> 172.21.15.7:0/1226022634 conn(0x7f32240a94d0 msgr2=0x7f32240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:40:46.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f322bb5f700 1 -- 172.21.15.7:0/1226022634 shutdown_connections 2023-03-03T15:40:46.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.833+0000 7f322bb5f700 1 -- 172.21.15.7:0/1226022634 wait complete. 2023-03-03T15:40:46.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322bb5f700 1 Processor -- start 2023-03-03T15:40:46.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322bb5f700 1 -- start start 2023-03-03T15:40:46.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322bb5f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322406c6c0 0x7f322414dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:46.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322bb5f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32240c2e30 0x7f32241ea380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:46.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322bb5f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 0x7f32241ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:46.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322bb5f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32240c6680 con 0x7f3224151cd0 2023-03-03T15:40:46.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322bb5f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32240c6be0 con 0x7f322406c6c0 2023-03-03T15:40:46.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322bb5f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32240c7070 con 0x7f32240c2e30 2023-03-03T15:40:46.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322a0fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 0x7f32241ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:46.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322a0fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 0x7f32241ea8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53940/0 (socket says 172.21.15.7:53940) 2023-03-03T15:40:46.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f322a0fc700 1 -- 172.21.15.7:0/247298494 learned_addr learned my addr 172.21.15.7:0/247298494 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:40:46.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f32298fb700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322406c6c0 0x7f322414dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:46.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.835+0000 7f32290fa700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32240c2e30 0x7f32241ea380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:46.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f322a0fc700 1 -- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32240c2e30 msgr2=0x7f32241ea380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:46.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f322a0fc700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32240c2e30 0x7f32241ea380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:46.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f322a0fc700 1 -- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322406c6c0 msgr2=0x7f322414dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f322a0fc700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322406c6c0 0x7f322414dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f322a0fc700 1 -- 172.21.15.7:0/247298494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3214009550 con 0x7f3224151cd0 2023-03-03T15:40:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f322a0fc700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 0x7f32241ea8c0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f321c00cd50 tx=0x7f321c00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:40:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f321a7fc700 1 -- 172.21.15.7:0/247298494 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c00cea0 con 0x7f3224151cd0 2023-03-03T15:40:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f321a7fc700 1 -- 172.21.15.7:0/247298494 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c00f480 con 0x7f3224151cd0 2023-03-03T15:40:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f322bb5f700 1 -- 172.21.15.7:0/247298494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3224157c80 con 0x7f3224151cd0 2023-03-03T15:40:46.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f321a7fc700 1 -- 172.21.15.7:0/247298494 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c00f830 con 0x7f3224151cd0 2023-03-03T15:40:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.836+0000 7f322bb5f700 1 -- 172.21.15.7:0/247298494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322406d360 con 0x7f3224151cd0 2023-03-03T15:40:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.837+0000 7f322bb5f700 1 -- 172.21.15.7:0/247298494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3224154a50 con 0x7f3224151cd0 2023-03-03T15:40:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.838+0000 7f321a7fc700 1 -- 172.21.15.7:0/247298494 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f321c009030 con 0x7f3224151cd0 2023-03-03T15:40:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.838+0000 7f321a7fc700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32080682f0 0x7f320806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:40:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.838+0000 7f321a7fc700 1 -- 172.21.15.7:0/247298494 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f321c0869b0 con 0x7f3224151cd0 2023-03-03T15:40:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.838+0000 7f32298fb700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32080682f0 0x7f320806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:40:46.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.839+0000 7f32298fb700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32080682f0 0x7f320806a7a0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f3214009130 tx=0x7f3214008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:40:46.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:46.842+0000 7f321a7fc700 1 -- 172.21.15.7:0/247298494 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f321c052cf0 con 0x7f3224151cd0 2023-03-03T15:40:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.080+0000 7f322bb5f700 1 -- 172.21.15.7:0/247298494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3224074bb0 con 0x7f3224151cd0 2023-03-03T15:40:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.081+0000 7f321a7fc700 1 -- 172.21.15.7:0/247298494 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f321c056310 con 0x7f3224151cd0 2023-03-03T15:40:47.083 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:40:47.083 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:40:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 -- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32080682f0 msgr2=0x7f320806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32080682f0 0x7f320806a7a0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f3214009130 tx=0x7f3214008030 comp rx=0 tx=0).stop 2023-03-03T15:40:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 -- 172.21.15.7:0/247298494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 msgr2=0x7f32241ea8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:40:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 0x7f32241ea8c0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f321c00cd50 tx=0x7f321c00a560 comp rx=0 tx=0).stop 2023-03-03T15:40:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 -- 172.21.15.7:0/247298494 shutdown_connections 2023-03-03T15:40:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32080682f0 0x7f320806a7a0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:47.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3224151cd0 0x7f32241ea8c0 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f322406c6c0 0x7f322414dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 --2- 172.21.15.7:0/247298494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32240c2e30 0x7f32241ea380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 -- 172.21.15.7:0/247298494 >> 172.21.15.7:0/247298494 conn(0x7f32240a94d0 msgr2=0x7f322406b310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 -- 172.21.15.7:0/247298494 shutdown_connections 2023-03-03T15:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:40:47.083+0000 7f322bb5f700 1 -- 172.21.15.7:0/247298494 wait complete. 2023-03-03T15:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:40:47.096 INFO:tasks.mds_thrash.fs.[a]:waiting for 21 secs before thrashing 2023-03-03T15:41:07.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:07.535 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:08.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:41:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.235+0000 7f1aac747700 1 Processor -- start 2023-03-03T15:41:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.235+0000 7f1aac747700 1 -- start start 2023-03-03T15:41:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aac747700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1aa414f9d0 0x7f1aa4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aac747700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1aa40abe20 0x7f1aa41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aac747700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa40c2f30 con 0x7f1aa414fda0 2023-03-03T15:41:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aac747700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa40c31d0 con 0x7f1aa40abe20 2023-03-03T15:41:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aac747700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa41524a0 con 0x7f1aa414f9d0 2023-03-03T15:41:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aaace4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1aa40abe20 0x7f1aa41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aa9ce2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1aa414fda0 0x7f1aa4071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:47940/0 (socket says 172.21.15.7:47940) 2023-03-03T15:41:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aa9ce2700 1 -- 172.21.15.7:0/64879533 learned_addr learned my addr 172.21.15.7:0/64879533 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:41:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aaa4e3700 1 --2- 172.21.15.7:0/64879533 >> v2:172.21.15.143:3300/0 conn(0x7f1aa414f9d0 0x7f1aa4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aaace4700 1 -- 172.21.15.7:0/64879533 >> v2:172.21.15.143:3300/0 conn(0x7f1aa414f9d0 msgr2=0x7f1aa4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:41:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aaace4700 1 --2- 172.21.15.7:0/64879533 >> v2:172.21.15.143:3300/0 conn(0x7f1aa414f9d0 0x7f1aa4156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aaace4700 1 -- 172.21.15.7:0/64879533 >> v1:172.21.15.7:6789/0 conn(0x7f1aa414fda0 legacy=0x7f1aa4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aaace4700 1 -- 172.21.15.7:0/64879533 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa40c34b0 con 0x7f1aa40abe20 2023-03-03T15:41:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.236+0000 7f1aa94e1700 1 -- 172.21.15.7:0/64879533 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698718927 0 0) 0x7f1aa40c2f30 con 0x7f1aa414fda0 2023-03-03T15:41:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.237+0000 7f1aaace4700 1 --2- 172.21.15.7:0/64879533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa40abe20 0x7f1aa41529c0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40710a0 tx=0x7f1a9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a527e7d92aa190 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:41:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.237+0000 7f1aa94e1700 1 -- 172.21.15.7:0/64879533 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c00e070 con 0x7f1aa40abe20 2023-03-03T15:41:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.237+0000 7f1aa94e1700 1 -- 172.21.15.7:0/64879533 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c00bb20 con 0x7f1aa40abe20 2023-03-03T15:41:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.237+0000 7f1aa94e1700 1 -- 172.21.15.7:0/64879533 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c012430 con 0x7f1aa40abe20 2023-03-03T15:41:08.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.237+0000 7f1aac747700 1 -- 172.21.15.7:0/64879533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa40abe20 msgr2=0x7f1aa41529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.237+0000 7f1aac747700 1 --2- 172.21.15.7:0/64879533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa40abe20 0x7f1aa41529c0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40710a0 tx=0x7f1a9c00b330 comp rx=0 tx=0).stop 2023-03-03T15:41:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.237+0000 7f1aac747700 1 -- 172.21.15.7:0/64879533 shutdown_connections 2023-03-03T15:41:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.237+0000 7f1aac747700 1 --2- 172.21.15.7:0/64879533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa40abe20 0x7f1aa41529c0 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.237+0000 7f1aac747700 1 --2- 172.21.15.7:0/64879533 >> v2:172.21.15.143:3300/0 conn(0x7f1aa414f9d0 0x7f1aa4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.237+0000 7f1aac747700 1 -- 172.21.15.7:0/64879533 >> 172.21.15.7:0/64879533 conn(0x7f1aa40c8900 msgr2=0x7f1aa40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:41:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.238+0000 7f1aac747700 1 -- 172.21.15.7:0/64879533 shutdown_connections 2023-03-03T15:41:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.238+0000 7f1aac747700 1 -- 172.21.15.7:0/64879533 wait complete. 2023-03-03T15:41:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.239+0000 7f1aac747700 1 Processor -- start 2023-03-03T15:41:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.239+0000 7f1aac747700 1 -- start start 2023-03-03T15:41:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.239+0000 7f1aac747700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa40abe20 0x7f1aa41ed600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.239+0000 7f1aac747700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa414f9d0 0x7f1aa4145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.239+0000 7f1aac747700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa414fda0 0x7f1aa40c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.239+0000 7f1aac747700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa406a550 con 0x7f1aa414fda0 2023-03-03T15:41:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.239+0000 7f1aac747700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa406d860 con 0x7f1aa414f9d0 2023-03-03T15:41:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.239+0000 7f1aac747700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa41edb40 con 0x7f1aa40abe20 2023-03-03T15:41:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.239+0000 7f1aaace4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa414fda0 0x7f1aa40c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aaace4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa414fda0 0x7f1aa40c17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53970/0 (socket says 172.21.15.7:53970) 2023-03-03T15:41:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aaace4700 1 -- 172.21.15.7:0/1365542755 learned_addr learned my addr 172.21.15.7:0/1365542755 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:41:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aa9ce2700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa414f9d0 0x7f1aa4145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aaa4e3700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa40abe20 0x7f1aa41ed600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aaace4700 1 -- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa40abe20 msgr2=0x7f1aa41ed600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aaace4700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa40abe20 0x7f1aa41ed600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aaace4700 1 -- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa414f9d0 msgr2=0x7f1aa4145890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aaace4700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa414f9d0 0x7f1aa4145890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aaace4700 1 -- 172.21.15.7:0/1365542755 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c009a70 con 0x7f1aa414fda0 2023-03-03T15:41:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aaace4700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa414fda0 0x7f1aa40c17e0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c013000 tx=0x7f1a9c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:41:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1a9affd700 1 -- 172.21.15.7:0/1365542755 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c012430 con 0x7f1aa414fda0 2023-03-03T15:41:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1a9affd700 1 -- 172.21.15.7:0/1365542755 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c020070 con 0x7f1aa414fda0 2023-03-03T15:41:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1a9affd700 1 -- 172.21.15.7:0/1365542755 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c012e50 con 0x7f1aa414fda0 2023-03-03T15:41:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aac747700 1 -- 172.21.15.7:0/1365542755 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa40ac350 con 0x7f1aa414fda0 2023-03-03T15:41:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.240+0000 7f1aac747700 1 -- 172.21.15.7:0/1365542755 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa40ab630 con 0x7f1aa414fda0 2023-03-03T15:41:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.241+0000 7f1aac747700 1 -- 172.21.15.7:0/1365542755 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa4071fa0 con 0x7f1aa414fda0 2023-03-03T15:41:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.242+0000 7f1a9affd700 1 -- 172.21.15.7:0/1365542755 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1a9c02e070 con 0x7f1aa414fda0 2023-03-03T15:41:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.242+0000 7f1a9affd700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1a880682f0 0x7f1a8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.242+0000 7f1a9affd700 1 -- 172.21.15.7:0/1365542755 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1a9c022070 con 0x7f1aa414fda0 2023-03-03T15:41:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.243+0000 7f1aaa4e3700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1a880682f0 0x7f1a8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.243+0000 7f1aaa4e3700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1a880682f0 0x7f1a8806a7a0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f1aa41e7460 tx=0x7f1a94009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:41:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.247+0000 7f1a9affd700 1 -- 172.21.15.7:0/1365542755 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1a9c053f90 con 0x7f1aa414fda0 2023-03-03T15:41:08.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.485+0000 7f1aac747700 1 -- 172.21.15.7:0/1365542755 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1aa406a2b0 con 0x7f1aa414fda0 2023-03-03T15:41:08.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.486+0000 7f1a9affd700 1 -- 172.21.15.7:0/1365542755 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1aa406a2b0 con 0x7f1aa414fda0 2023-03-03T15:41:08.565 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:41:08.566 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:41:08.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 -- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1a880682f0 msgr2=0x7f1a8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:08.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1a880682f0 0x7f1a8806a7a0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f1aa41e7460 tx=0x7f1a94009210 comp rx=0 tx=0).stop 2023-03-03T15:41:08.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 -- 172.21.15.7:0/1365542755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa414fda0 msgr2=0x7f1aa40c17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:08.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa414fda0 0x7f1aa40c17e0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c013000 tx=0x7f1a9c004ad0 comp rx=0 tx=0).stop 2023-03-03T15:41:08.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 -- 172.21.15.7:0/1365542755 shutdown_connections 2023-03-03T15:41:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1a880682f0 0x7f1a8806a7a0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1aa414fda0 0x7f1aa40c17e0 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1aa40abe20 0x7f1aa41ed600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 --2- 172.21.15.7:0/1365542755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1aa414f9d0 0x7f1aa4145890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 -- 172.21.15.7:0/1365542755 >> 172.21.15.7:0/1365542755 conn(0x7f1aa40c8900 msgr2=0x7f1aa41528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:41:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 -- 172.21.15.7:0/1365542755 shutdown_connections 2023-03-03T15:41:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:08.488+0000 7f1aac747700 1 -- 172.21.15.7:0/1365542755 wait complete. 2023-03-03T15:41:08.568 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:41:08.569 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T15:41:09.056 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:41:09.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.186+0000 7fd325732700 1 Processor -- start 2023-03-03T15:41:09.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.186+0000 7fd325732700 1 -- start start 2023-03-03T15:41:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd325732700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd320151d40 0x7fd3200736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd325732700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 0x7fd320157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd325732700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd320073c30 0x7fd320157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd325732700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32006c2a0 con 0x7fd320073c30 2023-03-03T15:41:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd325732700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32006d430 con 0x7fd320151d40 2023-03-03T15:41:09.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd325732700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32006cb00 con 0x7fd3200c2930 2023-03-03T15:41:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd31f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd320073c30 0x7fd320157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd31f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd320073c30 0x7fd320157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53978/0 (socket says 172.21.15.7:53978) 2023-03-03T15:41:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd31f7fe700 1 -- 172.21.15.7:0/3440274306 learned_addr learned my addr 172.21.15.7:0/3440274306 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:41:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd31effd700 1 --2- 172.21.15.7:0/3440274306 >> v2:172.21.15.143:3300/0 conn(0x7fd320151d40 0x7fd3200736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd31e7fc700 1 --2- 172.21.15.7:0/3440274306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 0x7fd320157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd31f7fe700 1 -- 172.21.15.7:0/3440274306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 msgr2=0x7fd320157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:41:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.187+0000 7fd31f7fe700 1 --2- 172.21.15.7:0/3440274306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 0x7fd320157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd31f7fe700 1 -- 172.21.15.7:0/3440274306 >> v2:172.21.15.143:3300/0 conn(0x7fd320151d40 msgr2=0x7fd3200736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:41:09.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd31f7fe700 1 --2- 172.21.15.7:0/3440274306 >> v2:172.21.15.143:3300/0 conn(0x7fd320151d40 0x7fd3200736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd31f7fe700 1 -- 172.21.15.7:0/3440274306 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32006cde0 con 0x7fd320073c30 2023-03-03T15:41:09.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd31e7fc700 1 --2- 172.21.15.7:0/3440274306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 0x7fd320157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:41:09.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd31f7fe700 1 --2- 172.21.15.7:0/3440274306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320073c30 0x7fd320157fd0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7fd32006e440 tx=0x7fd31800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1aff4fd0771efa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:41:09.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd31dffb700 1 -- 172.21.15.7:0/3440274306 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31800e070 con 0x7fd320073c30 2023-03-03T15:41:09.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd31dffb700 1 -- 172.21.15.7:0/3440274306 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31800b9c0 con 0x7fd320073c30 2023-03-03T15:41:09.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd31dffb700 1 -- 172.21.15.7:0/3440274306 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31800bd70 con 0x7fd320073c30 2023-03-03T15:41:09.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd325732700 1 -- 172.21.15.7:0/3440274306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320073c30 msgr2=0x7fd320157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:09.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd325732700 1 --2- 172.21.15.7:0/3440274306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320073c30 0x7fd320157fd0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7fd32006e440 tx=0x7fd31800b330 comp rx=0 tx=0).stop 2023-03-03T15:41:09.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd325732700 1 -- 172.21.15.7:0/3440274306 shutdown_connections 2023-03-03T15:41:09.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd325732700 1 --2- 172.21.15.7:0/3440274306 >> v2:172.21.15.143:3300/0 conn(0x7fd320151d40 0x7fd3200736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd325732700 1 --2- 172.21.15.7:0/3440274306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320073c30 0x7fd320157fd0 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd325732700 1 --2- 172.21.15.7:0/3440274306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 0x7fd320157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd325732700 1 -- 172.21.15.7:0/3440274306 >> 172.21.15.7:0/3440274306 conn(0x7fd3200a94d0 msgr2=0x7fd3200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:41:09.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.188+0000 7fd325732700 1 -- 172.21.15.7:0/3440274306 shutdown_connections 2023-03-03T15:41:09.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.189+0000 7fd325732700 1 -- 172.21.15.7:0/3440274306 wait complete. 2023-03-03T15:41:09.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.190+0000 7fd325732700 1 Processor -- start 2023-03-03T15:41:09.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.190+0000 7fd325732700 1 -- start start 2023-03-03T15:41:09.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.190+0000 7fd325732700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd320073c30 0x7fd3201ed710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:09.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.190+0000 7fd325732700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 0x7fd32006ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:09.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.190+0000 7fd325732700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320151d40 0x7fd320148820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:09.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.190+0000 7fd325732700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200be8c0 con 0x7fd320151d40 2023-03-03T15:41:09.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.190+0000 7fd325732700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3201e0a60 con 0x7fd320073c30 2023-03-03T15:41:09.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.190+0000 7fd325732700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3200c6910 con 0x7fd3200c2930 2023-03-03T15:41:09.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320151d40 0x7fd320148820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:09.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320151d40 0x7fd320148820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53984/0 (socket says 172.21.15.7:53984) 2023-03-03T15:41:09.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31f7fe700 1 -- 172.21.15.7:0/242162688 learned_addr learned my addr 172.21.15.7:0/242162688 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:41:09.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31e7fc700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 0x7fd32006ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:09.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31effd700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd320073c30 0x7fd3201ed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:09.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31f7fe700 1 -- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 msgr2=0x7fd32006ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:09.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31f7fe700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 0x7fd32006ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31f7fe700 1 -- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd320073c30 msgr2=0x7fd3201ed710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:09.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31f7fe700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd320073c30 0x7fd3201ed710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31f7fe700 1 -- 172.21.15.7:0/242162688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd318009b30 con 0x7fd320151d40 2023-03-03T15:41:09.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd31f7fe700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320151d40 0x7fd320148820 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fd32006bb70 tx=0x7fd318004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:41:09.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd3077fe700 1 -- 172.21.15.7:0/242162688 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318004250 con 0x7fd320151d40 2023-03-03T15:41:09.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd3077fe700 1 -- 172.21.15.7:0/242162688 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3180043b0 con 0x7fd320151d40 2023-03-03T15:41:09.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd325732700 1 -- 172.21.15.7:0/242162688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32006bea0 con 0x7fd320151d40 2023-03-03T15:41:09.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd3077fe700 1 -- 172.21.15.7:0/242162688 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318004760 con 0x7fd320151d40 2023-03-03T15:41:09.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.191+0000 7fd325732700 1 -- 172.21.15.7:0/242162688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32006d0c0 con 0x7fd320151d40 2023-03-03T15:41:09.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.192+0000 7fd325732700 1 -- 172.21.15.7:0/242162688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32006c030 con 0x7fd320151d40 2023-03-03T15:41:09.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.193+0000 7fd3077fe700 1 -- 172.21.15.7:0/242162688 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd31801c440 con 0x7fd320151d40 2023-03-03T15:41:09.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.193+0000 7fd3077fe700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd30c0682f0 0x7fd30c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:41:09.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.193+0000 7fd3077fe700 1 -- 172.21.15.7:0/242162688 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd318087880 con 0x7fd320151d40 2023-03-03T15:41:09.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.193+0000 7fd31effd700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd30c0682f0 0x7fd30c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:41:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.194+0000 7fd31effd700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd30c0682f0 0x7fd30c06a7a0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fd3140096f0 tx=0x7fd314008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:41:09.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.197+0000 7fd3077fe700 1 -- 172.21.15.7:0/242162688 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd318053c50 con 0x7fd320151d40 2023-03-03T15:41:09.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.437+0000 7fd325732700 1 -- 172.21.15.7:0/242162688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd320074bb0 con 0x7fd320151d40 2023-03-03T15:41:09.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.438+0000 7fd3077fe700 1 -- 172.21.15.7:0/242162688 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd318057270 con 0x7fd320151d40 2023-03-03T15:41:09.439 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:41:09.440 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:41:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.439+0000 7fd325732700 1 -- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd30c0682f0 msgr2=0x7fd30c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.439+0000 7fd325732700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd30c0682f0 0x7fd30c06a7a0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fd3140096f0 tx=0x7fd314008040 comp rx=0 tx=0).stop 2023-03-03T15:41:09.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.439+0000 7fd325732700 1 -- 172.21.15.7:0/242162688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320151d40 msgr2=0x7fd320148820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:41:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.439+0000 7fd325732700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320151d40 0x7fd320148820 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fd32006bb70 tx=0x7fd318004ad0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.439+0000 7fd325732700 1 -- 172.21.15.7:0/242162688 shutdown_connections 2023-03-03T15:41:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.440+0000 7fd325732700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd30c0682f0 0x7fd30c06a7a0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.440+0000 7fd325732700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd320151d40 0x7fd320148820 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.440+0000 7fd325732700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd320073c30 0x7fd3201ed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.440+0000 7fd325732700 1 --2- 172.21.15.7:0/242162688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3200c2930 0x7fd32006ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:41:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.440+0000 7fd325732700 1 -- 172.21.15.7:0/242162688 >> 172.21.15.7:0/242162688 conn(0x7fd3200a94d0 msgr2=0x7fd3200c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:41:09.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.440+0000 7fd325732700 1 -- 172.21.15.7:0/242162688 shutdown_connections 2023-03-03T15:41:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:41:09.440+0000 7fd325732700 1 -- 172.21.15.7:0/242162688 wait complete. 2023-03-03T15:41:09.443 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:41:09.453 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2023-03-03T15:41:37.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:41:37.641 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:07.721 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:07.725 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:12.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:42:12.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.707+0000 7f10479b1700 1 Processor -- start 2023-03-03T15:42:12.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.708+0000 7f10479b1700 1 -- start start 2023-03-03T15:42:12.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.708+0000 7f10479b1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f10401530d0 0x7f1040156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:12.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.708+0000 7f10479b1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10400c2f30 con 0x7f104014f9d0 2023-03-03T15:42:12.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.708+0000 7f10479b1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10400c31d0 con 0x7f104014fda0 2023-03-03T15:42:12.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.708+0000 7f10479b1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10401524a0 con 0x7f10401530d0 2023-03-03T15:42:12.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f1045f4e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f10401530d0 0x7f1040156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:12.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f104574d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f104014f9d0 0x7f1040071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48026/0 (socket says 172.21.15.7:48026) 2023-03-03T15:42:12.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f104574d700 1 -- 172.21.15.7:0/1196832759 learned_addr learned my addr 172.21.15.7:0/1196832759 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:42:12.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f1045f4e700 1 -- 172.21.15.7:0/1196832759 >> v1:172.21.15.143:6789/0 conn(0x7f104014fda0 legacy=0x7f10401529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:12.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f1045f4e700 1 -- 172.21.15.7:0/1196832759 >> v1:172.21.15.7:6789/0 conn(0x7f104014f9d0 legacy=0x7f1040071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:12.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f1045f4e700 1 -- 172.21.15.7:0/1196832759 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10400c34b0 con 0x7f10401530d0 2023-03-03T15:42:12.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f1037fff700 1 -- 172.21.15.7:0/1196832759 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726396571 0 0) 0x7f10400c2f30 con 0x7f104014f9d0 2023-03-03T15:42:12.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f1045f4e700 1 --2- 172.21.15.7:0/1196832759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 0x7f1040156430 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f1038009d90 tx=0x7f103800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986c7901914926d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:12.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f1037fff700 1 -- 172.21.15.7:0/1196832759 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103800e070 con 0x7f10401530d0 2023-03-03T15:42:12.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f1037fff700 1 -- 172.21.15.7:0/1196832759 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103800bbd0 con 0x7f10401530d0 2023-03-03T15:42:12.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.709+0000 7f1037fff700 1 -- 172.21.15.7:0/1196832759 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038012430 con 0x7f10401530d0 2023-03-03T15:42:12.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.710+0000 7f10479b1700 1 -- 172.21.15.7:0/1196832759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 msgr2=0x7f1040156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:12.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.710+0000 7f10479b1700 1 --2- 172.21.15.7:0/1196832759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 0x7f1040156430 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f1038009d90 tx=0x7f103800b330 comp rx=0 tx=0).stop 2023-03-03T15:42:12.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.710+0000 7f10479b1700 1 -- 172.21.15.7:0/1196832759 shutdown_connections 2023-03-03T15:42:12.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.710+0000 7f10479b1700 1 --2- 172.21.15.7:0/1196832759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 0x7f1040156430 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:12.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.710+0000 7f10479b1700 1 -- 172.21.15.7:0/1196832759 >> 172.21.15.7:0/1196832759 conn(0x7f10400c8900 msgr2=0x7f10400c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:42:12.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.710+0000 7f10479b1700 1 -- 172.21.15.7:0/1196832759 shutdown_connections 2023-03-03T15:42:12.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.710+0000 7f10479b1700 1 -- 172.21.15.7:0/1196832759 wait complete. 2023-03-03T15:42:12.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.711+0000 7f10479b1700 1 Processor -- start 2023-03-03T15:42:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.711+0000 7f10479b1700 1 -- start start 2023-03-03T15:42:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f10479b1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f104014f9d0 0x7f10400c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f10479b1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f104014fda0 0x7f104006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f10479b1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 0x7f104006e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f10479b1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104006eb00 con 0x7f10401530d0 2023-03-03T15:42:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f10479b1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104006a620 con 0x7f104014f9d0 2023-03-03T15:42:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f10479b1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10401ea080 con 0x7f104014fda0 2023-03-03T15:42:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f1045f4e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 0x7f104006e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f1045f4e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 0x7f104006e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54056/0 (socket says 172.21.15.7:54056) 2023-03-03T15:42:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f1045f4e700 1 -- 172.21.15.7:0/4078696458 learned_addr learned my addr 172.21.15.7:0/4078696458 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:42:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f104574d700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f104014f9d0 0x7f10400c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f1044f4c700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f104014fda0 0x7f104006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f1045f4e700 1 -- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f104014fda0 msgr2=0x7f104006e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f1045f4e700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f104014fda0 0x7f104006e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f1045f4e700 1 -- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f104014f9d0 msgr2=0x7f10400c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f1045f4e700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f104014f9d0 0x7f10400c17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f1045f4e700 1 -- 172.21.15.7:0/4078696458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1038009a70 con 0x7f10401530d0 2023-03-03T15:42:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.712+0000 7f1044f4c700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f104014fda0 0x7f104006e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:42:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.713+0000 7f1045f4e700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 0x7f104006e5c0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f1038014040 tx=0x7f1038004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.713+0000 7f1035ffb700 1 -- 172.21.15.7:0/4078696458 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038012430 con 0x7f10401530d0 2023-03-03T15:42:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.713+0000 7f1035ffb700 1 -- 172.21.15.7:0/4078696458 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1038020070 con 0x7f10401530d0 2023-03-03T15:42:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.713+0000 7f10479b1700 1 -- 172.21.15.7:0/4078696458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10400ac350 con 0x7f10401530d0 2023-03-03T15:42:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.713+0000 7f1035ffb700 1 -- 172.21.15.7:0/4078696458 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10380098f0 con 0x7f10401530d0 2023-03-03T15:42:12.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.713+0000 7f10479b1700 1 -- 172.21.15.7:0/4078696458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10400ab630 con 0x7f10401530d0 2023-03-03T15:42:12.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.714+0000 7f10479b1700 1 -- 172.21.15.7:0/4078696458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1040071fa0 con 0x7f10401530d0 2023-03-03T15:42:12.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.715+0000 7f1035ffb700 1 -- 172.21.15.7:0/4078696458 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1038026400 con 0x7f10401530d0 2023-03-03T15:42:12.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.715+0000 7f1035ffb700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10240682f0 0x7f102406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:12.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.715+0000 7f1035ffb700 1 -- 172.21.15.7:0/4078696458 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f10380882e0 con 0x7f10401530d0 2023-03-03T15:42:12.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.715+0000 7f104574d700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10240682f0 0x7f102406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:12.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.716+0000 7f104574d700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10240682f0 0x7f102406a7a0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f103c000f60 tx=0x7f103c007d90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:12.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.720+0000 7f1035ffb700 1 -- 172.21.15.7:0/4078696458 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f10380546d0 con 0x7f10401530d0 2023-03-03T15:42:12.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.955+0000 7f10479b1700 1 -- 172.21.15.7:0/4078696458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1040154600 con 0x7f10401530d0 2023-03-03T15:42:12.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.956+0000 7f1035ffb700 1 -- 172.21.15.7:0/4078696458 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1038057cf0 con 0x7f10401530d0 2023-03-03T15:42:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:42:12.959 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:42:12.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 -- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10240682f0 msgr2=0x7f102406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:12.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10240682f0 0x7f102406a7a0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f103c000f60 tx=0x7f103c007d90 comp rx=0 tx=0).stop 2023-03-03T15:42:12.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 -- 172.21.15.7:0/4078696458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 msgr2=0x7f104006e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:12.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 0x7f104006e5c0 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f1038014040 tx=0x7f1038004ad0 comp rx=0 tx=0).stop 2023-03-03T15:42:12.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 -- 172.21.15.7:0/4078696458 shutdown_connections 2023-03-03T15:42:12.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10240682f0 0x7f102406a7a0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:12.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10401530d0 0x7f104006e5c0 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:12.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f104014f9d0 0x7f10400c17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:12.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 --2- 172.21.15.7:0/4078696458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f104014fda0 0x7f104006e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:12.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 -- 172.21.15.7:0/4078696458 >> 172.21.15.7:0/4078696458 conn(0x7f10400c8900 msgr2=0x7f104014f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:42:12.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 -- 172.21.15.7:0/4078696458 shutdown_connections 2023-03-03T15:42:12.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:12.958+0000 7f10479b1700 1 -- 172.21.15.7:0/4078696458 wait complete. 2023-03-03T15:42:12.963 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:42:12.972 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T15:42:22.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:42:23.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.111+0000 7f1db40b3700 1 Processor -- start 2023-03-03T15:42:23.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.111+0000 7f1db40b3700 1 -- start start 2023-03-03T15:42:23.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db40b3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1dac06d870 0x7f1dac1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:23.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db40b3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1dac0c3e30 0x7f1dac0c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:23.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db40b3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dac0c7a60 con 0x7f1dac0c1da0 2023-03-03T15:42:23.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db40b3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dac0c2af0 con 0x7f1dac0c3e30 2023-03-03T15:42:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db40b3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dac0c3620 con 0x7f1dac06d870 2023-03-03T15:42:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db2650700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1dac0c3e30 0x7f1dac0c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db2650700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1dac0c3e30 0x7f1dac0c4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54068/0 (socket says 172.21.15.7:54068) 2023-03-03T15:42:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db2650700 1 -- 172.21.15.7:0/3917103931 learned_addr learned my addr 172.21.15.7:0/3917103931 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:42:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db1e4f700 1 --2- 172.21.15.7:0/3917103931 >> v2:172.21.15.143:3300/0 conn(0x7f1dac06d870 0x7f1dac1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db2650700 1 -- 172.21.15.7:0/3917103931 >> v2:172.21.15.143:3300/0 conn(0x7f1dac06d870 msgr2=0x7f1dac1550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:42:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db2650700 1 --2- 172.21.15.7:0/3917103931 >> v2:172.21.15.143:3300/0 conn(0x7f1dac06d870 0x7f1dac1550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:23.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.112+0000 7f1db2650700 1 -- 172.21.15.7:0/3917103931 >> v1:172.21.15.143:6789/0 conn(0x7f1dac0c1da0 legacy=0x7f1dac0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:23.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db2650700 1 -- 172.21.15.7:0/3917103931 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac1545e0 con 0x7f1dac0c3e30 2023-03-03T15:42:23.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db2650700 1 --2- 172.21.15.7:0/3917103931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c3e30 0x7f1dac0c4230 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f1dac073880 tx=0x7f1da400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b8951739a897e7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:23.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db0e4d700 1 -- 172.21.15.7:0/3917103931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da400e070 con 0x7f1dac0c3e30 2023-03-03T15:42:23.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db0e4d700 1 -- 172.21.15.7:0/3917103931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da400b9c0 con 0x7f1dac0c3e30 2023-03-03T15:42:23.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db0e4d700 1 -- 172.21.15.7:0/3917103931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da400bd70 con 0x7f1dac0c3e30 2023-03-03T15:42:23.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db40b3700 1 -- 172.21.15.7:0/3917103931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c3e30 msgr2=0x7f1dac0c4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:23.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db40b3700 1 --2- 172.21.15.7:0/3917103931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c3e30 0x7f1dac0c4230 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f1dac073880 tx=0x7f1da400b330 comp rx=0 tx=0).stop 2023-03-03T15:42:23.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db40b3700 1 -- 172.21.15.7:0/3917103931 shutdown_connections 2023-03-03T15:42:23.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db40b3700 1 --2- 172.21.15.7:0/3917103931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c3e30 0x7f1dac0c4230 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:23.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db40b3700 1 --2- 172.21.15.7:0/3917103931 >> v2:172.21.15.143:3300/0 conn(0x7f1dac06d870 0x7f1dac1550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:23.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.113+0000 7f1db40b3700 1 -- 172.21.15.7:0/3917103931 >> 172.21.15.7:0/3917103931 conn(0x7f1dac06c110 msgr2=0x7f1dac06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:42:23.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.114+0000 7f1db40b3700 1 -- 172.21.15.7:0/3917103931 shutdown_connections 2023-03-03T15:42:23.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.114+0000 7f1db40b3700 1 -- 172.21.15.7:0/3917103931 wait complete. 2023-03-03T15:42:23.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.115+0000 7f1db40b3700 1 Processor -- start 2023-03-03T15:42:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.115+0000 7f1db40b3700 1 -- start start 2023-03-03T15:42:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.115+0000 7f1db40b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dac06d870 0x7f1dac1ea340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.115+0000 7f1db40b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c1da0 0x7f1dac157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.115+0000 7f1db40b3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dac0c3e30 0x7f1dac14eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.115+0000 7f1db40b3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac1569d0 con 0x7f1dac0c1da0 2023-03-03T15:42:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.115+0000 7f1db40b3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac155dc0 con 0x7f1dac0c3e30 2023-03-03T15:42:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.115+0000 7f1db40b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac1560a0 con 0x7f1dac06d870 2023-03-03T15:42:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.115+0000 7f1db164e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c1da0 0x7f1dac157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db164e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c1da0 0x7f1dac157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54072/0 (socket says 172.21.15.7:54072) 2023-03-03T15:42:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db164e700 1 -- 172.21.15.7:0/3409925370 learned_addr learned my addr 172.21.15.7:0/3409925370 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:42:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db1e4f700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dac06d870 0x7f1dac1ea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db164e700 1 -- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dac06d870 msgr2=0x7f1dac1ea340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db2650700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dac0c3e30 0x7f1dac14eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db164e700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dac06d870 0x7f1dac1ea340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db164e700 1 -- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dac0c3e30 msgr2=0x7f1dac14eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:23.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db164e700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dac0c3e30 0x7f1dac14eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:23.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db164e700 1 -- 172.21.15.7:0/3409925370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da4009b30 con 0x7f1dac0c1da0 2023-03-03T15:42:23.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db1e4f700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dac06d870 0x7f1dac1ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:42:23.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db164e700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c1da0 0x7f1dac157f60 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f1da800c4e0 tx=0x7f1da800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:23.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1da27fc700 1 -- 172.21.15.7:0/3409925370 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80052d0 con 0x7f1dac0c1da0 2023-03-03T15:42:23.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1da27fc700 1 -- 172.21.15.7:0/3409925370 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da8005430 con 0x7f1dac0c1da0 2023-03-03T15:42:23.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1da27fc700 1 -- 172.21.15.7:0/3409925370 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80038c0 con 0x7f1dac0c1da0 2023-03-03T15:42:23.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db40b3700 1 -- 172.21.15.7:0/3409925370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac154ba0 con 0x7f1dac0c1da0 2023-03-03T15:42:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.116+0000 7f1db40b3700 1 -- 172.21.15.7:0/3409925370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac1548c0 con 0x7f1dac0c1da0 2023-03-03T15:42:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.117+0000 7f1db40b3700 1 -- 172.21.15.7:0/3409925370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac06dc40 con 0x7f1dac0c1da0 2023-03-03T15:42:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.118+0000 7f1da27fc700 1 -- 172.21.15.7:0/3409925370 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1da8019030 con 0x7f1dac0c1da0 2023-03-03T15:42:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.118+0000 7f1da27fc700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d90068320 0x7f1d9006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.118+0000 7f1da27fc700 1 -- 172.21.15.7:0/3409925370 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1da80854b0 con 0x7f1dac0c1da0 2023-03-03T15:42:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.118+0000 7f1db1e4f700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d90068320 0x7f1d9006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.119+0000 7f1db1e4f700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d90068320 0x7f1d9006a7d0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009a00 tx=0x7f1d9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:23.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.123+0000 7f1da27fc700 1 -- 172.21.15.7:0/3409925370 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1da80517f0 con 0x7f1dac0c1da0 2023-03-03T15:42:23.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.362+0000 7f1db40b3700 1 -- 172.21.15.7:0/3409925370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dac154400 con 0x7f1dac0c1da0 2023-03-03T15:42:23.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.362+0000 7f1da27fc700 1 -- 172.21.15.7:0/3409925370 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1da8054e10 con 0x7f1dac0c1da0 2023-03-03T15:42:23.364 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:42:23.364 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:42:23.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 -- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d90068320 msgr2=0x7f1d9006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:23.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d90068320 0x7f1d9006a7d0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009a00 tx=0x7f1d9c008040 comp rx=0 tx=0).stop 2023-03-03T15:42:23.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 -- 172.21.15.7:0/3409925370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c1da0 msgr2=0x7f1dac157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:23.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c1da0 0x7f1dac157f60 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f1da800c4e0 tx=0x7f1da800c8a0 comp rx=0 tx=0).stop 2023-03-03T15:42:23.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 -- 172.21.15.7:0/3409925370 shutdown_connections 2023-03-03T15:42:23.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1dac0c3e30 0x7f1dac14eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:23.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1dac06d870 0x7f1dac1ea340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:23.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d90068320 0x7f1d9006a7d0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:23.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 --2- 172.21.15.7:0/3409925370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dac0c1da0 0x7f1dac157f60 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:23.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 -- 172.21.15.7:0/3409925370 >> 172.21.15.7:0/3409925370 conn(0x7f1dac06c110 msgr2=0x7f1dac14abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:42:23.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 -- 172.21.15.7:0/3409925370 shutdown_connections 2023-03-03T15:42:23.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:23.364+0000 7f1db40b3700 1 -- 172.21.15.7:0/3409925370 wait complete. 2023-03-03T15:42:23.368 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:42:23.377 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T15:42:35.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:42:35.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.591+0000 7ff3d3096700 1 Processor -- start 2023-03-03T15:42:35.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.592+0000 7ff3d3096700 1 -- start start 2023-03-03T15:42:35.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.592+0000 7ff3d3096700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff3cc152e30 0x7ff3cc0c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:35.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.592+0000 7ff3d3096700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3cc0c4910 0x7ff3cc156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:35.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.592+0000 7ff3d3096700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc1523b0 con 0x7ff3cc0beab0 2023-03-03T15:42:35.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.592+0000 7ff3d3096700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3cc072010 con 0x7ff3cc152e30 2023-03-03T15:42:35.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.592+0000 7ff3d3096700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0aaaf0 con 0x7ff3cc0c4910 2023-03-03T15:42:35.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.592+0000 7ff3cbfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff3cc0beab0 0x7ff3cc155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48070/0 (socket says 172.21.15.7:48070) 2023-03-03T15:42:35.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.592+0000 7ff3cbfff700 1 -- 172.21.15.7:0/392321580 learned_addr learned my addr 172.21.15.7:0/392321580 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:42:35.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3d0e32700 1 --2- 172.21.15.7:0/392321580 >> v2:172.21.15.143:3300/0 conn(0x7ff3cc152e30 0x7ff3cc0c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:35.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3d1633700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3cc0c4910 0x7ff3cc156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:35.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740016430 0 0) 0x7ff3cc1523b0 con 0x7ff3cc0beab0 2023-03-03T15:42:35.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3bc0035b0 con 0x7ff3cc0beab0 2023-03-03T15:42:35.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2162407651 0 0) 0x7ff3bc0035b0 con 0x7ff3cc0beab0 2023-03-03T15:42:35.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3cc0c4910 msgr2=0x7ff3cc156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:42:35.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3cc0c4910 0x7ff3cc156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 >> v2:172.21.15.143:3300/0 conn(0x7ff3cc152e30 msgr2=0x7ff3cc0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:42:35.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 --2- 172.21.15.7:0/392321580 >> v2:172.21.15.143:3300/0 conn(0x7ff3cc152e30 0x7ff3cc0c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0aadd0 con 0x7ff3cc0beab0 2023-03-03T15:42:35.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff3b8001670 con 0x7ff3cc0beab0 2023-03-03T15:42:35.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 >> v1:172.21.15.7:6789/0 conn(0x7ff3cc0beab0 legacy=0x7ff3cc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:35.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3bc003a70 0x7ff3bc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:35.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3bc004850 0x7ff3bc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:35.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3bc005200 0x7ff3bc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:35.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc072010 con 0x7ff3bc003a70 2023-03-03T15:42:35.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0aaaf0 con 0x7ff3bc005200 2023-03-03T15:42:35.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc00bfd0 con 0x7ff3bc004850 2023-03-03T15:42:35.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3d1633700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3cc0c4910 0x7ff3cc156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:42:35.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3d0e32700 1 --2- 172.21.15.7:0/392321580 >> v2:172.21.15.143:3300/0 conn(0x7ff3cc152e30 0x7ff3cc0c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:42:35.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.593+0000 7ff3d0e32700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3bc003a70 0x7ff3bc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:35.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3cbfff700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3bc004850 0x7ff3bc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:35.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3d1633700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3bc005200 0x7ff3bc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:35.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3d0e32700 1 -- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3bc004850 msgr2=0x7ff3bc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:42:35.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3d0e32700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3bc004850 0x7ff3bc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3d0e32700 1 -- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3bc005200 msgr2=0x7ff3bc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:42:35.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3d0e32700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3bc005200 0x7ff3bc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3d0e32700 1 -- 172.21.15.7:0/392321580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc0035b0 con 0x7ff3bc003a70 2023-03-03T15:42:35.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3d0e32700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3bc003a70 0x7ff3bc003e80 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009d10 tx=0x7ff3bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5dbcdf16bf53ce server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:35.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0009e60 con 0x7ff3bc003a70 2023-03-03T15:42:35.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c000e9a0 con 0x7ff3bc003a70 2023-03-03T15:42:35.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.594+0000 7ff3cb7fe700 1 -- 172.21.15.7:0/392321580 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000ed10 con 0x7ff3bc003a70 2023-03-03T15:42:35.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.595+0000 7ff3d3096700 1 -- 172.21.15.7:0/392321580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3bc003a70 msgr2=0x7ff3bc003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:35.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.595+0000 7ff3d3096700 1 --2- 172.21.15.7:0/392321580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3bc003a70 0x7ff3bc003e80 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009d10 tx=0x7ff3bc005880 comp rx=0 tx=0).stop 2023-03-03T15:42:35.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.595+0000 7ff3d1633700 1 -- 172.21.15.7:0/392321580 reap_dead start 2023-03-03T15:42:35.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.595+0000 7ff3d3096700 1 -- 172.21.15.7:0/392321580 shutdown_connections 2023-03-03T15:42:35.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.595+0000 7ff3d3096700 1 -- 172.21.15.7:0/392321580 >> 172.21.15.7:0/392321580 conn(0x7ff3cc0736f0 msgr2=0x7ff3cc073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:42:35.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.595+0000 7ff3d3096700 1 -- 172.21.15.7:0/392321580 shutdown_connections 2023-03-03T15:42:35.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.595+0000 7ff3d3096700 1 -- 172.21.15.7:0/392321580 wait complete. 2023-03-03T15:42:35.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.596+0000 7ff3d3096700 1 Processor -- start 2023-03-03T15:42:35.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.596+0000 7ff3d3096700 1 -- start start 2023-03-03T15:42:35.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3d3096700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3bc003a70 0x7ff3cc0a9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:35.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3d3096700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3cc0aa1d0 0x7ff3cc1ed560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:35.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3d3096700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3cc1536a0 0x7ff3cc14a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:35.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3d3096700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc155320 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3d3096700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc06b4b0 con 0x7ff3cc1536a0 2023-03-03T15:42:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3d3096700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc06aca0 con 0x7ff3bc003a70 2023-03-03T15:42:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3cbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3cc0aa1d0 0x7ff3cc1ed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3cbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3cc0aa1d0 0x7ff3cc1ed560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54104/0 (socket says 172.21.15.7:54104) 2023-03-03T15:42:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3cbfff700 1 -- 172.21.15.7:0/3347652572 learned_addr learned my addr 172.21.15.7:0/3347652572 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:42:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3cbfff700 1 -- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3bc003a70 msgr2=0x7ff3cc0a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3cbfff700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3bc003a70 0x7ff3cc0a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3d1633700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3cc1536a0 0x7ff3cc14a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3cbfff700 1 -- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3cc1536a0 msgr2=0x7ff3cc14a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3cbfff700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3cc1536a0 0x7ff3cc14a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3cbfff700 1 -- 172.21.15.7:0/3347652572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc153200 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.597+0000 7ff3cbfff700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3cc0aa1d0 0x7ff3cc1ed560 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8007310 tx=0x7ff3b8007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.598+0000 7ff3c97fa700 1 -- 172.21.15.7:0/3347652572 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b800f6c0 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.598+0000 7ff3c97fa700 1 -- 172.21.15.7:0/3347652572 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b800f840 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.598+0000 7ff3c97fa700 1 -- 172.21.15.7:0/3347652572 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b800fbf0 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.598+0000 7ff3d3096700 1 -- 172.21.15.7:0/3347652572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc071c70 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.598+0000 7ff3d3096700 1 -- 172.21.15.7:0/3347652572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc153390 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.599+0000 7ff3c97fa700 1 -- 172.21.15.7:0/3347652572 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff3b800fd50 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.599+0000 7ff3d3096700 1 -- 172.21.15.7:0/3347652572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc074bb0 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.599+0000 7ff3c97fa700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3b00682f0 0x7ff3b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:35.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.600+0000 7ff3c97fa700 1 -- 172.21.15.7:0/3347652572 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff3b80861c0 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.600+0000 7ff3d0e32700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3b00682f0 0x7ff3b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:35.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.600+0000 7ff3d0e32700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3b00682f0 0x7ff3b006a7a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0bdee0 tx=0x7ff3c0013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:35.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.604+0000 7ff3c97fa700 1 -- 172.21.15.7:0/3347652572 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff3b8052560 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.843+0000 7ff3d3096700 1 -- 172.21.15.7:0/3347652572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3cc1ecea0 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.843+0000 7ff3c97fa700 1 -- 172.21.15.7:0/3347652572 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff3b8055b80 con 0x7ff3cc0aa1d0 2023-03-03T15:42:35.845 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:42:35.845 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:42:35.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 -- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3b00682f0 msgr2=0x7ff3b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:35.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3b00682f0 0x7ff3b006a7a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0bdee0 tx=0x7ff3c0013000 comp rx=0 tx=0).stop 2023-03-03T15:42:35.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 -- 172.21.15.7:0/3347652572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3cc0aa1d0 msgr2=0x7ff3cc1ed560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:35.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3cc0aa1d0 0x7ff3cc1ed560 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8007310 tx=0x7ff3b8007620 comp rx=0 tx=0).stop 2023-03-03T15:42:35.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 -- 172.21.15.7:0/3347652572 shutdown_connections 2023-03-03T15:42:35.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3cc1536a0 0x7ff3cc14a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3bc003a70 0x7ff3cc0a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3b00682f0 0x7ff3b006a7a0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 --2- 172.21.15.7:0/3347652572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3cc0aa1d0 0x7ff3cc1ed560 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:35.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 -- 172.21.15.7:0/3347652572 >> 172.21.15.7:0/3347652572 conn(0x7ff3cc0736f0 msgr2=0x7ff3cc0be230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:42:35.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 -- 172.21.15.7:0/3347652572 shutdown_connections 2023-03-03T15:42:35.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:35.845+0000 7ff3d3096700 1 -- 172.21.15.7:0/3347652572 wait complete. 2023-03-03T15:42:35.849 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:42:35.859 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before thrashing 2023-03-03T15:42:37.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:37.797 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:42:42.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:42:42.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.997+0000 7fdb05b05700 1 Processor -- start 2023-03-03T15:42:42.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.997+0000 7fdb05b05700 1 -- start start 2023-03-03T15:42:43.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdb05b05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb00151cd0 0x7fdb00154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:43.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdb05b05700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb00157ff0 con 0x7fdb0006c6c0 2023-03-03T15:42:43.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdb05b05700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0006d640 con 0x7fdb000c2e30 2023-03-03T15:42:43.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdb05b05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0006cda0 con 0x7fdb00151cd0 2023-03-03T15:42:43.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdafffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdb0006c6c0 0x7fdb000736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48098/0 (socket says 172.21.15.7:48098) 2023-03-03T15:42:43.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdafffff700 1 -- 172.21.15.7:0/595700111 learned_addr learned my addr 172.21.15.7:0/595700111 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:42:43.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654022009 0 0) 0x7fdb00157ff0 con 0x7fdb0006c6c0 2023-03-03T15:42:43.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae80035b0 con 0x7fdb0006c6c0 2023-03-03T15:42:43.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdaff7fe700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb00151cd0 0x7fdb00154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:43.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4283944754 0 0) 0x7fdae80035b0 con 0x7fdb0006c6c0 2023-03-03T15:42:43.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.998+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb00151cd0 msgr2=0x7fdb00154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:42:43.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb00151cd0 0x7fdb00154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:43.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 >> v1:172.21.15.143:6789/0 conn(0x7fdb000c2e30 legacy=0x7fdb00158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:43.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0006d080 con 0x7fdb0006c6c0 2023-03-03T15:42:43.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fdaf8001670 con 0x7fdb0006c6c0 2023-03-03T15:42:43.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 >> v1:172.21.15.7:6789/0 conn(0x7fdb0006c6c0 legacy=0x7fdb000736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:43.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb00151cd0 0x7fdae8003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:43.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdae8004020 0x7fdae8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:43.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdae8004d90 0x7fdae8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:43.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0006cda0 con 0x7fdae8004d90 2023-03-03T15:42:43.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae800bcf0 con 0x7fdae8004020 2023-03-03T15:42:43.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae800beb0 con 0x7fdb00151cd0 2023-03-03T15:42:43.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafffff700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdae8004d90 0x7fdae8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:43.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdaff7fe700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb00151cd0 0x7fdae8003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:43.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafeffd700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdae8004020 0x7fdae8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:43.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafffff700 1 -- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb00151cd0 msgr2=0x7fdae8003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:42:43.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafffff700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb00151cd0 0x7fdae8003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:43.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafffff700 1 -- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdae8004020 msgr2=0x7fdae8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:42:43.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafffff700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdae8004020 0x7fdae8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:43.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafffff700 1 -- 172.21.15.7:0/595700111 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae80035b0 con 0x7fdae8004d90 2023-03-03T15:42:43.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafffff700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdae8004d90 0x7fdae8005200 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800c210 tx=0x7fdae8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48fd803151e172a1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:43.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8007c10 con 0x7fdae8004d90 2023-03-03T15:42:43.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf800cdd0 con 0x7fdae8004d90 2023-03-03T15:42:43.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:42.999+0000 7fdafe7fc700 1 -- 172.21.15.7:0/595700111 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8005ba0 con 0x7fdae8004d90 2023-03-03T15:42:43.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.000+0000 7fdb05b05700 1 -- 172.21.15.7:0/595700111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdae8004d90 msgr2=0x7fdae8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:43.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.000+0000 7fdb05b05700 1 --2- 172.21.15.7:0/595700111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdae8004d90 0x7fdae8005200 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800c210 tx=0x7fdae8005880 comp rx=0 tx=0).stop 2023-03-03T15:42:43.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.000+0000 7fdafffff700 1 -- 172.21.15.7:0/595700111 reap_dead start 2023-03-03T15:42:43.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.000+0000 7fdb05b05700 1 -- 172.21.15.7:0/595700111 shutdown_connections 2023-03-03T15:42:43.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.000+0000 7fdb05b05700 1 -- 172.21.15.7:0/595700111 >> 172.21.15.7:0/595700111 conn(0x7fdb000a94d0 msgr2=0x7fdb000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:42:43.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.000+0000 7fdb05b05700 1 -- 172.21.15.7:0/595700111 shutdown_connections 2023-03-03T15:42:43.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.000+0000 7fdb05b05700 1 -- 172.21.15.7:0/595700111 wait complete. 2023-03-03T15:42:43.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.001+0000 7fdb05b05700 1 Processor -- start 2023-03-03T15:42:43.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdb05b05700 1 -- start start 2023-03-03T15:42:43.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdb05b05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb0014b8b0 0x7fdb001ea6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:43.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdb05b05700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0014bc80 0x7fdb00152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:43.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdb05b05700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb00152fd0 0x7fdb000acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:43.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdb05b05700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb000aa500 con 0x7fdb00152fd0 2023-03-03T15:42:43.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdb05b05700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb00152380 con 0x7fdb0014bc80 2023-03-03T15:42:43.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdb05b05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb001eac30 con 0x7fdb0014b8b0 2023-03-03T15:42:43.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdafffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb00152fd0 0x7fdb000acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:43.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdafffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb00152fd0 0x7fdb000acad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54128/0 (socket says 172.21.15.7:54128) 2023-03-03T15:42:43.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdafffff700 1 -- 172.21.15.7:0/1050853473 learned_addr learned my addr 172.21.15.7:0/1050853473 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:42:43.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdafeffd700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0014bc80 0x7fdb00152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:43.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.002+0000 7fdaff7fe700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb0014b8b0 0x7fdb001ea6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:43.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdafffff700 1 -- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb0014b8b0 msgr2=0x7fdb001ea6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:43.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdafffff700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb0014b8b0 0x7fdb001ea6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:43.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdafffff700 1 -- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0014bc80 msgr2=0x7fdb00152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:43.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdafffff700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0014bc80 0x7fdb00152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:43.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdafffff700 1 -- 172.21.15.7:0/1050853473 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb00157c80 con 0x7fdb00152fd0 2023-03-03T15:42:43.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdafeffd700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0014bc80 0x7fdb00152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:42:43.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdafffff700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb00152fd0 0x7fdb000acad0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8001930 tx=0x7fdaf8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:43.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdae7fff700 1 -- 172.21.15.7:0/1050853473 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8023070 con 0x7fdb00152fd0 2023-03-03T15:42:43.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdae7fff700 1 -- 172.21.15.7:0/1050853473 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf8001470 con 0x7fdb00152fd0 2023-03-03T15:42:43.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdb05b05700 1 -- 172.21.15.7:0/1050853473 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0006d360 con 0x7fdb00152fd0 2023-03-03T15:42:43.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdae7fff700 1 -- 172.21.15.7:0/1050853473 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8007330 con 0x7fdb00152fd0 2023-03-03T15:42:43.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.003+0000 7fdb05b05700 1 -- 172.21.15.7:0/1050853473 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb00073e00 con 0x7fdb00152fd0 2023-03-03T15:42:43.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.004+0000 7fdb05b05700 1 -- 172.21.15.7:0/1050853473 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb00074bb0 con 0x7fdb00152fd0 2023-03-03T15:42:43.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.005+0000 7fdae7fff700 1 -- 172.21.15.7:0/1050853473 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdaf8010070 con 0x7fdb00152fd0 2023-03-03T15:42:43.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.005+0000 7fdae7fff700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdaec0682f0 0x7fdaec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:42:43.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.005+0000 7fdae7fff700 1 -- 172.21.15.7:0/1050853473 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdaf8085e70 con 0x7fdb00152fd0 2023-03-03T15:42:43.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.005+0000 7fdaff7fe700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdaec0682f0 0x7fdaec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:42:43.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.006+0000 7fdaff7fe700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdaec0682f0 0x7fdaec06a7a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fdb0014c140 tx=0x7fdaf0009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:42:43.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.010+0000 7fdae7fff700 1 -- 172.21.15.7:0/1050853473 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdaf8052250 con 0x7fdb00152fd0 2023-03-03T15:42:43.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.246+0000 7fdb05b05700 1 -- 172.21.15.7:0/1050853473 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb000bac80 con 0x7fdb00152fd0 2023-03-03T15:42:43.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.246+0000 7fdae7fff700 1 -- 172.21.15.7:0/1050853473 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdaf8055870 con 0x7fdb00152fd0 2023-03-03T15:42:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:42:43.248 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:42:43.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.248+0000 7fdb05b05700 1 -- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdaec0682f0 msgr2=0x7fdaec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:43.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.248+0000 7fdb05b05700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdaec0682f0 0x7fdaec06a7a0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fdb0014c140 tx=0x7fdaf0009210 comp rx=0 tx=0).stop 2023-03-03T15:42:43.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.248+0000 7fdb05b05700 1 -- 172.21.15.7:0/1050853473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb00152fd0 msgr2=0x7fdb000acad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:42:43.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.248+0000 7fdb05b05700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb00152fd0 0x7fdb000acad0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8001930 tx=0x7fdaf8001a00 comp rx=0 tx=0).stop 2023-03-03T15:42:43.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.248+0000 7fdb05b05700 1 -- 172.21.15.7:0/1050853473 shutdown_connections 2023-03-03T15:42:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.248+0000 7fdb05b05700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdaec0682f0 0x7fdaec06a7a0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.248+0000 7fdb05b05700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb00152fd0 0x7fdb000acad0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.248+0000 7fdb05b05700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb0014b8b0 0x7fdb001ea6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.248+0000 7fdb05b05700 1 --2- 172.21.15.7:0/1050853473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb0014bc80 0x7fdb00152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:42:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.248+0000 7fdb05b05700 1 -- 172.21.15.7:0/1050853473 >> 172.21.15.7:0/1050853473 conn(0x7fdb000a94d0 msgr2=0x7fdb00151100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:42:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.249+0000 7fdb05b05700 1 -- 172.21.15.7:0/1050853473 shutdown_connections 2023-03-03T15:42:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:42:43.249+0000 7fdb05b05700 1 -- 172.21.15.7:0/1050853473 wait complete. 2023-03-03T15:42:43.252 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:42:43.262 INFO:tasks.mds_thrash.fs.[b]:waiting for 80 secs before thrashing 2023-03-03T15:43:07.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:07.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:12.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:43:12.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.515+0000 7fb6091da700 1 Processor -- start 2023-03-03T15:43:12.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.515+0000 7fb6091da700 1 -- start start 2023-03-03T15:43:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb6091da700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb60414fda0 0x7fb604156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:12.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb6091da700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb6040abe20 0x7fb6041529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb6091da700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6040c2f30 con 0x7fb60414f9d0 2023-03-03T15:43:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb6091da700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6040c31d0 con 0x7fb60414fda0 2023-03-03T15:43:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb6091da700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6041524a0 con 0x7fb6040abe20 2023-03-03T15:43:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb60259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb60414fda0 0x7fb604156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb602d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb60414f9d0 0x7fb604071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48134/0 (socket says 172.21.15.7:48134) 2023-03-03T15:43:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb602d9d700 1 -- 172.21.15.7:0/2468142042 learned_addr learned my addr 172.21.15.7:0/2468142042 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:43:12.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb60359e700 1 --2- 172.21.15.7:0/2468142042 >> v2:172.21.15.143:3300/0 conn(0x7fb6040abe20 0x7fb6041529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:12.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb60259c700 1 -- 172.21.15.7:0/2468142042 >> v2:172.21.15.143:3300/0 conn(0x7fb6040abe20 msgr2=0x7fb6041529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:43:12.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb60259c700 1 --2- 172.21.15.7:0/2468142042 >> v2:172.21.15.143:3300/0 conn(0x7fb6040abe20 0x7fb6041529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:12.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb60259c700 1 -- 172.21.15.7:0/2468142042 >> v1:172.21.15.7:6789/0 conn(0x7fb60414f9d0 legacy=0x7fb604071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:12.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb60259c700 1 -- 172.21.15.7:0/2468142042 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6040c34b0 con 0x7fb60414fda0 2023-03-03T15:43:12.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb601d9b700 1 -- 172.21.15.7:0/2468142042 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918808313 0 0) 0x7fb6040c2f30 con 0x7fb60414f9d0 2023-03-03T15:43:12.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb60359e700 1 --2- 172.21.15.7:0/2468142042 >> v2:172.21.15.143:3300/0 conn(0x7fb6040abe20 0x7fb6041529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:43:12.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb60259c700 1 --2- 172.21.15.7:0/2468142042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb60414fda0 0x7fb604156430 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fb604071230 tx=0x7fb5f8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5af8d5a1a1755b1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:43:12.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.516+0000 7fb601d9b700 1 -- 172.21.15.7:0/2468142042 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8011830 con 0x7fb60414fda0 2023-03-03T15:43:12.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.517+0000 7fb601d9b700 1 -- 172.21.15.7:0/2468142042 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f8011990 con 0x7fb60414fda0 2023-03-03T15:43:12.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.517+0000 7fb601d9b700 1 -- 172.21.15.7:0/2468142042 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8011d20 con 0x7fb60414fda0 2023-03-03T15:43:12.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.517+0000 7fb6091da700 1 -- 172.21.15.7:0/2468142042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb60414fda0 msgr2=0x7fb604156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:12.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.517+0000 7fb6091da700 1 --2- 172.21.15.7:0/2468142042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb60414fda0 0x7fb604156430 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fb604071230 tx=0x7fb5f8009b20 comp rx=0 tx=0).stop 2023-03-03T15:43:12.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.517+0000 7fb6091da700 1 -- 172.21.15.7:0/2468142042 shutdown_connections 2023-03-03T15:43:12.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.517+0000 7fb6091da700 1 --2- 172.21.15.7:0/2468142042 >> v2:172.21.15.143:3300/0 conn(0x7fb6040abe20 0x7fb6041529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:12.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.517+0000 7fb6091da700 1 --2- 172.21.15.7:0/2468142042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb60414fda0 0x7fb604156430 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:12.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.517+0000 7fb6091da700 1 -- 172.21.15.7:0/2468142042 >> 172.21.15.7:0/2468142042 conn(0x7fb6040c8900 msgr2=0x7fb6040c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:43:12.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.517+0000 7fb6091da700 1 -- 172.21.15.7:0/2468142042 shutdown_connections 2023-03-03T15:43:12.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.517+0000 7fb6091da700 1 -- 172.21.15.7:0/2468142042 wait complete. 2023-03-03T15:43:12.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb6091da700 1 Processor -- start 2023-03-03T15:43:12.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb6091da700 1 -- start start 2023-03-03T15:43:12.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb6091da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6040abe20 0x7fb6040be8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:12.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb6091da700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb60414f9d0 0x7fb6040bf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:12.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb6091da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60414fda0 0x7fb6041e7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:12.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb6091da700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb604140cc0 con 0x7fb6040abe20 2023-03-03T15:43:12.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb6091da700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6041e7a80 con 0x7fb60414f9d0 2023-03-03T15:43:12.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb6091da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6040bdbc0 con 0x7fb60414fda0 2023-03-03T15:43:12.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb602d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6040abe20 0x7fb6040be8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:12.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb602d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6040abe20 0x7fb6040be8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54160/0 (socket says 172.21.15.7:54160) 2023-03-03T15:43:12.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.519+0000 7fb602d9d700 1 -- 172.21.15.7:0/719160711 learned_addr learned my addr 172.21.15.7:0/719160711 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:43:12.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb60359e700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60414fda0 0x7fb6041e7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:12.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb60259c700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb60414f9d0 0x7fb6040bf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:12.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb602d9d700 1 -- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60414fda0 msgr2=0x7fb6041e7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:12.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb602d9d700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60414fda0 0x7fb6041e7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:12.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb602d9d700 1 -- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb60414f9d0 msgr2=0x7fb6040bf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:12.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb602d9d700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb60414f9d0 0x7fb6040bf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:12.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb602d9d700 1 -- 172.21.15.7:0/719160711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f80094a0 con 0x7fb6040abe20 2023-03-03T15:43:12.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb60359e700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60414fda0 0x7fb6041e7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:43:12.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb60259c700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb60414f9d0 0x7fb6040bf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:43:12.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb602d9d700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6040abe20 0x7fb6040be8e0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fb5f40011e0 tx=0x7fb5f400cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:43:12.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/719160711 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4004c80 con 0x7fb6040abe20 2023-03-03T15:43:12.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/719160711 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f4004e00 con 0x7fb6040abe20 2023-03-03T15:43:12.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb6091da700 1 -- 172.21.15.7:0/719160711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6040ac350 con 0x7fb6040abe20 2023-03-03T15:43:12.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/719160711 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4007310 con 0x7fb6040abe20 2023-03-03T15:43:12.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.520+0000 7fb6091da700 1 -- 172.21.15.7:0/719160711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6040ab630 con 0x7fb6040abe20 2023-03-03T15:43:12.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.521+0000 7fb6091da700 1 -- 172.21.15.7:0/719160711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb604071fa0 con 0x7fb6040abe20 2023-03-03T15:43:12.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.521+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/719160711 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb5f401a020 con 0x7fb6040abe20 2023-03-03T15:43:12.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.522+0000 7fb5eb7fe700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5f00682f0 0x7fb5f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:43:12.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.522+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/719160711 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb5f4085dd0 con 0x7fb6040abe20 2023-03-03T15:43:12.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.522+0000 7fb60259c700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5f00682f0 0x7fb5f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:43:12.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.523+0000 7fb60259c700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5f00682f0 0x7fb5f006a7a0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fb6041501d0 tx=0x7fb5f8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:43:12.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.526+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/719160711 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb5f4052170 con 0x7fb6040abe20 2023-03-03T15:43:12.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.762+0000 7fb6091da700 1 -- 172.21.15.7:0/719160711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6041e7d60 con 0x7fb6040abe20 2023-03-03T15:43:12.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.762+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/719160711 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb5f4055790 con 0x7fb6040abe20 2023-03-03T15:43:12.765 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:43:12.765 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:43:12.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 -- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5f00682f0 msgr2=0x7fb5f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:12.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5f00682f0 0x7fb5f006a7a0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fb6041501d0 tx=0x7fb5f8008030 comp rx=0 tx=0).stop 2023-03-03T15:43:12.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 -- 172.21.15.7:0/719160711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6040abe20 msgr2=0x7fb6040be8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:43:12.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6040abe20 0x7fb6040be8e0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fb5f40011e0 tx=0x7fb5f400cae0 comp rx=0 tx=0).stop 2023-03-03T15:43:12.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 -- 172.21.15.7:0/719160711 shutdown_connections 2023-03-03T15:43:12.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb60414fda0 0x7fb6041e7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:12.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5f00682f0 0x7fb5f006a7a0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:12.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb6040abe20 0x7fb6040be8e0 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:12.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 --2- 172.21.15.7:0/719160711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb60414f9d0 0x7fb6040bf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:43:12.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 -- 172.21.15.7:0/719160711 >> 172.21.15.7:0/719160711 conn(0x7fb6040c8900 msgr2=0x7fb60414f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:43:12.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 -- 172.21.15.7:0/719160711 shutdown_connections 2023-03-03T15:43:12.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:43:12.764+0000 7fb6091da700 1 -- 172.21.15.7:0/719160711 wait complete. 2023-03-03T15:43:12.769 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:43:12.779 INFO:tasks.mds_thrash.fs.[a]:waiting for 109 secs before thrashing 2023-03-03T15:43:37.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:43:37.928 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:03.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:44:03.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.401+0000 7f5ee59dd700 1 Processor -- start 2023-03-03T15:44:03.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.401+0000 7f5ee59dd700 1 -- start start 2023-03-03T15:44:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.401+0000 7f5ee59dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 0x7f5ee00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:44:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.401+0000 7f5ee59dd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5ee00c00e0 0x7f5ee0154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:44:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.401+0000 7f5ee59dd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee00cb280 con 0x7f5ee00cae00 2023-03-03T15:44:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.401+0000 7f5ee59dd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ee00c7310 con 0x7f5ee00c00e0 2023-03-03T15:44:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.401+0000 7f5ee59dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee00bfd10 con 0x7f5ee00caa30 2023-03-03T15:44:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5edf7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5ee00cae00 0x7f5ee00c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48192/0 (socket says 172.21.15.7:48192) 2023-03-03T15:44:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5edf7fe700 1 -- 172.21.15.7:0/2127467183 learned_addr learned my addr 172.21.15.7:0/2127467183 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:44:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5ede7fc700 1 --2- 172.21.15.7:0/2127467183 >> v2:172.21.15.143:3300/0 conn(0x7f5ee00c00e0 0x7f5ee0154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:44:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5edeffd700 1 --2- 172.21.15.7:0/2127467183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 0x7f5ee00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:44:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5eddffb700 1 -- 172.21.15.7:0/2127467183 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243563456 0 0) 0x7f5ee00cb280 con 0x7f5ee00cae00 2023-03-03T15:44:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5eddffb700 1 -- 172.21.15.7:0/2127467183 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ecc0035b0 con 0x7f5ee00cae00 2023-03-03T15:44:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5ede7fc700 1 -- 172.21.15.7:0/2127467183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 msgr2=0x7f5ee00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:44:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5ede7fc700 1 --2- 172.21.15.7:0/2127467183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 0x7f5ee00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:44:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5eddffb700 1 -- 172.21.15.7:0/2127467183 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 793270240 0 0) 0x7f5ecc0035b0 con 0x7f5ee00cae00 2023-03-03T15:44:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5ede7fc700 1 -- 172.21.15.7:0/2127467183 >> v1:172.21.15.7:6789/0 conn(0x7f5ee00cae00 legacy=0x7f5ee00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:44:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5ede7fc700 1 -- 172.21.15.7:0/2127467183 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee00bfa30 con 0x7f5ee00c00e0 2023-03-03T15:44:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5edeffd700 1 --2- 172.21.15.7:0/2127467183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 0x7f5ee00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:44:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.402+0000 7f5ede7fc700 1 --2- 172.21.15.7:0/2127467183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ee00c00e0 0x7f5ee0154ff0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8009820 tx=0x7f5ec8009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=774b142a67075665 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:44:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5eddffb700 1 -- 172.21.15.7:0/2127467183 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed80019f0 con 0x7f5ee00c00e0 2023-03-03T15:44:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5eddffb700 1 -- 172.21.15.7:0/2127467183 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec8005c30 con 0x7f5ee00c00e0 2023-03-03T15:44:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5eddffb700 1 -- 172.21.15.7:0/2127467183 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8004030 con 0x7f5ee00c00e0 2023-03-03T15:44:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5ee59dd700 1 -- 172.21.15.7:0/2127467183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ee00c00e0 msgr2=0x7f5ee0154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:44:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5ee59dd700 1 --2- 172.21.15.7:0/2127467183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ee00c00e0 0x7f5ee0154ff0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8009820 tx=0x7f5ec8009b30 comp rx=0 tx=0).stop 2023-03-03T15:44:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5ee59dd700 1 -- 172.21.15.7:0/2127467183 shutdown_connections 2023-03-03T15:44:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5ee59dd700 1 --2- 172.21.15.7:0/2127467183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 0x7f5ee00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:44:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5ee59dd700 1 --2- 172.21.15.7:0/2127467183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ee00c00e0 0x7f5ee0154ff0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:44:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5ee59dd700 1 -- 172.21.15.7:0/2127467183 >> 172.21.15.7:0/2127467183 conn(0x7f5ee0066e40 msgr2=0x7f5ee0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:44:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5ee59dd700 1 -- 172.21.15.7:0/2127467183 shutdown_connections 2023-03-03T15:44:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.403+0000 7f5ee59dd700 1 -- 172.21.15.7:0/2127467183 wait complete. 2023-03-03T15:44:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.404+0000 7f5ee59dd700 1 Processor -- start 2023-03-03T15:44:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5ee59dd700 1 -- start start 2023-03-03T15:44:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5ee59dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ee00c00e0 0x7f5ee00c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:44:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5ee59dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 0x7f5ee01ed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:44:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5ee59dd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ee00cae00 0x7f5ee01edef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:44:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5ee59dd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee0149eb0 con 0x7f5ee00cae00 2023-03-03T15:44:03.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5ee59dd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee014b460 con 0x7f5ee00c00e0 2023-03-03T15:44:03.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5ee59dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee014a100 con 0x7f5ee00caa30 2023-03-03T15:44:03.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5edf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ee00cae00 0x7f5ee01edef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:44:03.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5edf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ee00cae00 0x7f5ee01edef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54218/0 (socket says 172.21.15.7:54218) 2023-03-03T15:44:03.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5edf7fe700 1 -- 172.21.15.7:0/1485200133 learned_addr learned my addr 172.21.15.7:0/1485200133 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:44:03.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5ede7fc700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 0x7f5ee01ed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:44:03.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.405+0000 7f5edeffd700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ee00c00e0 0x7f5ee00c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:44:03.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5edf7fe700 1 -- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 msgr2=0x7f5ee01ed9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:44:03.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5edf7fe700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 0x7f5ee01ed9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:44:03.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5edf7fe700 1 -- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ee00c00e0 msgr2=0x7f5ee00c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:44:03.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5edf7fe700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ee00c00e0 0x7f5ee00c4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:44:03.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5edf7fe700 1 -- 172.21.15.7:0/1485200133 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed00094a0 con 0x7f5ee00cae00 2023-03-03T15:44:03.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5edf7fe700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ee00cae00 0x7f5ee01edef0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800cc00 tx=0x7f5ed800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:44:03.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5ed7fff700 1 -- 172.21.15.7:0/1485200133 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed800aa80 con 0x7f5ee00cae00 2023-03-03T15:44:03.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5ed7fff700 1 -- 172.21.15.7:0/1485200133 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed800ac00 con 0x7f5ee00cae00 2023-03-03T15:44:03.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5ee59dd700 1 -- 172.21.15.7:0/1485200133 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec8009430 con 0x7f5ee00cae00 2023-03-03T15:44:03.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5ed7fff700 1 -- 172.21.15.7:0/1485200133 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed80070d0 con 0x7f5ee00cae00 2023-03-03T15:44:03.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.406+0000 7f5ee59dd700 1 -- 172.21.15.7:0/1485200133 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee01547f0 con 0x7f5ee00cae00 2023-03-03T15:44:03.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.407+0000 7f5ee59dd700 1 -- 172.21.15.7:0/1485200133 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee0154510 con 0x7f5ee00cae00 2023-03-03T15:44:03.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.408+0000 7f5ed7fff700 1 -- 172.21.15.7:0/1485200133 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5ed8010070 con 0x7f5ee00cae00 2023-03-03T15:44:03.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.408+0000 7f5ed7fff700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ec00682f0 0x7f5ec006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:44:03.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.408+0000 7f5ed7fff700 1 -- 172.21.15.7:0/1485200133 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5ed8085c00 con 0x7f5ee00cae00 2023-03-03T15:44:03.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.408+0000 7f5edeffd700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ec00682f0 0x7f5ec006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:44:03.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.409+0000 7f5edeffd700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ec00682f0 0x7f5ec006a7a0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0158a40 tx=0x7f5ed0009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:44:03.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.412+0000 7f5ed7fff700 1 -- 172.21.15.7:0/1485200133 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5ed808b020 con 0x7f5ee00cae00 2023-03-03T15:44:03.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.652+0000 7f5ee59dd700 1 -- 172.21.15.7:0/1485200133 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ee01ed800 con 0x7f5ee00cae00 2023-03-03T15:44:03.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.653+0000 7f5ed7fff700 1 -- 172.21.15.7:0/1485200133 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5ed8018090 con 0x7f5ee00cae00 2023-03-03T15:44:03.654 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:44:03.655 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:44:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 -- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ec00682f0 msgr2=0x7f5ec006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:44:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ec00682f0 0x7f5ec006a7a0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0158a40 tx=0x7f5ed0009210 comp rx=0 tx=0).stop 2023-03-03T15:44:03.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 -- 172.21.15.7:0/1485200133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ee00cae00 msgr2=0x7f5ee01edef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:44:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ee00cae00 0x7f5ee01edef0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800cc00 tx=0x7f5ed800cf10 comp rx=0 tx=0).stop 2023-03-03T15:44:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 -- 172.21.15.7:0/1485200133 shutdown_connections 2023-03-03T15:44:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ec00682f0 0x7f5ec006a7a0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:44:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ee00cae00 0x7f5ee01edef0 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:44:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ee00c00e0 0x7f5ee00c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:44:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 --2- 172.21.15.7:0/1485200133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ee00caa30 0x7f5ee01ed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:44:03.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 -- 172.21.15.7:0/1485200133 >> 172.21.15.7:0/1485200133 conn(0x7f5ee0066e40 msgr2=0x7f5ee006b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:44:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 -- 172.21.15.7:0/1485200133 shutdown_connections 2023-03-03T15:44:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:44:03.655+0000 7f5ee59dd700 1 -- 172.21.15.7:0/1485200133 wait complete. 2023-03-03T15:44:03.658 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:44:03.668 INFO:tasks.mds_thrash.fs.[b]:waiting for 78 secs before thrashing 2023-03-03T15:44:07.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:07.995 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:38.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:44:38.063 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:01.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:45:01.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.917+0000 7f66b70f4700 1 Processor -- start 2023-03-03T15:45:01.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.917+0000 7f66b70f4700 1 -- start start 2023-03-03T15:45:01.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66b70f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0151ab0 0x7f66b00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:01.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66b70f4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66b0073c30 0x7f66b006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:01.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66b70f4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f66b006c3e0 0x7f66b0157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:01.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66b70f4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b006d780 con 0x7f66b0073c30 2023-03-03T15:45:01.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66b70f4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b006cb70 con 0x7f66b006c3e0 2023-03-03T15:45:01.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66b70f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b006ce50 con 0x7f66b0151ab0 2023-03-03T15:45:01.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66affff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66b0073c30 0x7f66b006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:01.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66affff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66b0073c30 0x7f66b006bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54274/0 (socket says 172.21.15.7:54274) 2023-03-03T15:45:01.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66affff700 1 -- 172.21.15.7:0/538005046 learned_addr learned my addr 172.21.15.7:0/538005046 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:45:01.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66b4e90700 1 --2- 172.21.15.7:0/538005046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0151ab0 0x7f66b00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:01.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66b5691700 1 --2- 172.21.15.7:0/538005046 >> v2:172.21.15.143:3300/0 conn(0x7f66b006c3e0 0x7f66b0157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:01.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66affff700 1 -- 172.21.15.7:0/538005046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0151ab0 msgr2=0x7f66b00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:45:01.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66affff700 1 --2- 172.21.15.7:0/538005046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0151ab0 0x7f66b00736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66affff700 1 -- 172.21.15.7:0/538005046 >> v2:172.21.15.143:3300/0 conn(0x7f66b006c3e0 msgr2=0x7f66b0157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:45:01.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66affff700 1 --2- 172.21.15.7:0/538005046 >> v2:172.21.15.143:3300/0 conn(0x7f66b006c3e0 0x7f66b0157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66affff700 1 -- 172.21.15.7:0/538005046 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b006d130 con 0x7f66b0073c30 2023-03-03T15:45:01.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66b5691700 1 --2- 172.21.15.7:0/538005046 >> v2:172.21.15.143:3300/0 conn(0x7f66b006c3e0 0x7f66b0157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:45:01.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.918+0000 7f66b4e90700 1 --2- 172.21.15.7:0/538005046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0151ab0 0x7f66b00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:45:01.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66affff700 1 --2- 172.21.15.7:0/538005046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b0073c30 0x7f66b006bea0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f66b00c3db0 tx=0x7f669c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1249db9614b78fc5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:01.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66af7fe700 1 -- 172.21.15.7:0/538005046 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c00b870 con 0x7f66b0073c30 2023-03-03T15:45:01.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66af7fe700 1 -- 172.21.15.7:0/538005046 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c00b9d0 con 0x7f66b0073c30 2023-03-03T15:45:01.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66af7fe700 1 -- 172.21.15.7:0/538005046 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c00bd60 con 0x7f66b0073c30 2023-03-03T15:45:01.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66b70f4700 1 -- 172.21.15.7:0/538005046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b0073c30 msgr2=0x7f66b006bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:01.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66b70f4700 1 --2- 172.21.15.7:0/538005046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b0073c30 0x7f66b006bea0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f66b00c3db0 tx=0x7f669c009bd0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66b70f4700 1 -- 172.21.15.7:0/538005046 shutdown_connections 2023-03-03T15:45:01.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66b70f4700 1 --2- 172.21.15.7:0/538005046 >> v2:172.21.15.143:3300/0 conn(0x7f66b006c3e0 0x7f66b0157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66b70f4700 1 --2- 172.21.15.7:0/538005046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0151ab0 0x7f66b00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66b70f4700 1 --2- 172.21.15.7:0/538005046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b0073c30 0x7f66b006bea0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66b70f4700 1 -- 172.21.15.7:0/538005046 >> 172.21.15.7:0/538005046 conn(0x7f66b00a94d0 msgr2=0x7f66b00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:45:01.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66b70f4700 1 -- 172.21.15.7:0/538005046 shutdown_connections 2023-03-03T15:45:01.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.919+0000 7f66b70f4700 1 -- 172.21.15.7:0/538005046 wait complete. 2023-03-03T15:45:01.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.920+0000 7f66b70f4700 1 Processor -- start 2023-03-03T15:45:01.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.921+0000 7f66b70f4700 1 -- start start 2023-03-03T15:45:01.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.921+0000 7f66b70f4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b006c3e0 0x7f66b014c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:01.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.921+0000 7f66b70f4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0073c30 0x7f66b01467f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:01.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.921+0000 7f66b70f4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b0151ab0 0x7f66b0145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:01.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.921+0000 7f66b70f4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b00c51d0 con 0x7f66b006c3e0 2023-03-03T15:45:01.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.921+0000 7f66b70f4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b01eabd0 con 0x7f66b0151ab0 2023-03-03T15:45:01.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.921+0000 7f66b70f4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b00bde90 con 0x7f66b0073c30 2023-03-03T15:45:01.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.921+0000 7f66b4e90700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b006c3e0 0x7f66b014c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:01.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b4e90700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b006c3e0 0x7f66b014c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54278/0 (socket says 172.21.15.7:54278) 2023-03-03T15:45:01.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b4e90700 1 -- 172.21.15.7:0/1318926544 learned_addr learned my addr 172.21.15.7:0/1318926544 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:45:01.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66affff700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0073c30 0x7f66b01467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:01.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b5691700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b0151ab0 0x7f66b0145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:01.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b4e90700 1 -- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0073c30 msgr2=0x7f66b01467f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:01.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b4e90700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0073c30 0x7f66b01467f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b4e90700 1 -- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b0151ab0 msgr2=0x7f66b0145b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:01.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b4e90700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b0151ab0 0x7f66b0145b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:01.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b4e90700 1 -- 172.21.15.7:0/1318926544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c009550 con 0x7f66b006c3e0 2023-03-03T15:45:01.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b5691700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b0151ab0 0x7f66b0145b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:45:01.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b4e90700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b006c3e0 0x7f66b014c050 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f66a400e760 tx=0x7f66a400ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:01.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66ad7fa700 1 -- 172.21.15.7:0/1318926544 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4013610 con 0x7f66b006c3e0 2023-03-03T15:45:01.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66ad7fa700 1 -- 172.21.15.7:0/1318926544 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a4013770 con 0x7f66b006c3e0 2023-03-03T15:45:01.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b70f4700 1 -- 172.21.15.7:0/1318926544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b0151e80 con 0x7f66b006c3e0 2023-03-03T15:45:01.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66ad7fa700 1 -- 172.21.15.7:0/1318926544 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4013b20 con 0x7f66b006c3e0 2023-03-03T15:45:01.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.922+0000 7f66b70f4700 1 -- 172.21.15.7:0/1318926544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b006d410 con 0x7f66b006c3e0 2023-03-03T15:45:01.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.924+0000 7f66b70f4700 1 -- 172.21.15.7:0/1318926544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b0152010 con 0x7f66b006c3e0 2023-03-03T15:45:01.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.924+0000 7f66ad7fa700 1 -- 172.21.15.7:0/1318926544 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f66a4023070 con 0x7f66b006c3e0 2023-03-03T15:45:01.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.924+0000 7f66ad7fa700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66940682f0 0x7f669406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:01.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.924+0000 7f66ad7fa700 1 -- 172.21.15.7:0/1318926544 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f66a4085bf0 con 0x7f66b006c3e0 2023-03-03T15:45:01.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.924+0000 7f66affff700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66940682f0 0x7f669406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:01.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.925+0000 7f66affff700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66940682f0 0x7f669406a7a0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f669c000c00 tx=0x7f669c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:01.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:01.929+0000 7f66ad7fa700 1 -- 172.21.15.7:0/1318926544 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f66a4051f90 con 0x7f66b006c3e0 2023-03-03T15:45:02.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.168+0000 7f66b70f4700 1 -- 172.21.15.7:0/1318926544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66b0073a20 con 0x7f66b006c3e0 2023-03-03T15:45:02.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.168+0000 7f66ad7fa700 1 -- 172.21.15.7:0/1318926544 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f66a40555b0 con 0x7f66b006c3e0 2023-03-03T15:45:02.170 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:45:02.170 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:45:02.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 -- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66940682f0 msgr2=0x7f669406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:02.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66940682f0 0x7f669406a7a0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f669c000c00 tx=0x7f669c008030 comp rx=0 tx=0).stop 2023-03-03T15:45:02.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 -- 172.21.15.7:0/1318926544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b006c3e0 msgr2=0x7f66b014c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:02.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b006c3e0 0x7f66b014c050 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f66a400e760 tx=0x7f66a400ea70 comp rx=0 tx=0).stop 2023-03-03T15:45:02.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 -- 172.21.15.7:0/1318926544 shutdown_connections 2023-03-03T15:45:02.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b0151ab0 0x7f66b0145b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:02.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66940682f0 0x7f669406a7a0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:02.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66b006c3e0 0x7f66b014c050 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 --2- 172.21.15.7:0/1318926544 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66b0073c30 0x7f66b01467f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 -- 172.21.15.7:0/1318926544 >> 172.21.15.7:0/1318926544 conn(0x7f66b00a94d0 msgr2=0x7f66b0150dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:45:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 -- 172.21.15.7:0/1318926544 shutdown_connections 2023-03-03T15:45:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:02.170+0000 7f66b70f4700 1 -- 172.21.15.7:0/1318926544 wait complete. 2023-03-03T15:45:02.173 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:45:02.183 INFO:tasks.mds_thrash.fs.[a]:waiting for 59 secs before thrashing 2023-03-03T15:45:08.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:08.146 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:21.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:45:21.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.807+0000 7f1f77afd700 1 Processor -- start 2023-03-03T15:45:21.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.807+0000 7f1f77afd700 1 -- start start 2023-03-03T15:45:21.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f77afd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1f701586c0 0x7f1f7006d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f77afd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f700c1da0 0x7f1f7006d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f77afd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f70154560 con 0x7f1f7013a0f0 2023-03-03T15:45:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f77afd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f70154840 con 0x7f1f700c1da0 2023-03-03T15:45:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f77afd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f70157be0 con 0x7f1f701586c0 2023-03-03T15:45:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f7609a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f700c1da0 0x7f1f7006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f7609a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f700c1da0 0x7f1f7006d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54310/0 (socket says 172.21.15.7:54310) 2023-03-03T15:45:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f7609a700 1 -- 172.21.15.7:0/760906666 learned_addr learned my addr 172.21.15.7:0/760906666 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:45:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f75098700 1 --2- 172.21.15.7:0/760906666 >> v2:172.21.15.143:3300/0 conn(0x7f1f701586c0 0x7f1f7006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f7609a700 1 -- 172.21.15.7:0/760906666 >> v2:172.21.15.143:3300/0 conn(0x7f1f701586c0 msgr2=0x7f1f7006d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:45:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f7609a700 1 --2- 172.21.15.7:0/760906666 >> v2:172.21.15.143:3300/0 conn(0x7f1f701586c0 0x7f1f7006d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f7609a700 1 -- 172.21.15.7:0/760906666 >> v1:172.21.15.143:6789/0 conn(0x7f1f7013a0f0 legacy=0x7f1f700c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.808+0000 7f1f7609a700 1 -- 172.21.15.7:0/760906666 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f70157ec0 con 0x7f1f700c1da0 2023-03-03T15:45:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f7609a700 1 --2- 172.21.15.7:0/760906666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f700c1da0 0x7f1f7006d780 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f1f70072a30 tx=0x7f1f6800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79c91af31d2bb5b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:21.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f74897700 1 -- 172.21.15.7:0/760906666 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800e070 con 0x7f1f700c1da0 2023-03-03T15:45:21.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f74897700 1 -- 172.21.15.7:0/760906666 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6800b9c0 con 0x7f1f700c1da0 2023-03-03T15:45:21.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f74897700 1 -- 172.21.15.7:0/760906666 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800bd70 con 0x7f1f700c1da0 2023-03-03T15:45:21.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f77afd700 1 -- 172.21.15.7:0/760906666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f700c1da0 msgr2=0x7f1f7006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:21.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f77afd700 1 --2- 172.21.15.7:0/760906666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f700c1da0 0x7f1f7006d780 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f1f70072a30 tx=0x7f1f6800b330 comp rx=0 tx=0).stop 2023-03-03T15:45:21.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f77afd700 1 -- 172.21.15.7:0/760906666 shutdown_connections 2023-03-03T15:45:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f77afd700 1 --2- 172.21.15.7:0/760906666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f700c1da0 0x7f1f7006d780 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f77afd700 1 --2- 172.21.15.7:0/760906666 >> v2:172.21.15.143:3300/0 conn(0x7f1f701586c0 0x7f1f7006d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f77afd700 1 -- 172.21.15.7:0/760906666 >> 172.21.15.7:0/760906666 conn(0x7f1f7006c110 msgr2=0x7f1f7006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:45:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f77afd700 1 -- 172.21.15.7:0/760906666 shutdown_connections 2023-03-03T15:45:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.809+0000 7f1f77afd700 1 -- 172.21.15.7:0/760906666 wait complete. 2023-03-03T15:45:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.811+0000 7f1f77afd700 1 Processor -- start 2023-03-03T15:45:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.811+0000 7f1f77afd700 1 -- start start 2023-03-03T15:45:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f77afd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f700c1da0 0x7f1f700c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f77afd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f7013a0f0 0x7f1f70070190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f77afd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f701586c0 0x7f1f700706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f77afd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700ab360 con 0x7f1f7013a0f0 2023-03-03T15:45:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f77afd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700c64b0 con 0x7f1f700c1da0 2023-03-03T15:45:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f77afd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700aad30 con 0x7f1f701586c0 2023-03-03T15:45:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f75098700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f7013a0f0 0x7f1f70070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f75098700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f7013a0f0 0x7f1f70070190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54314/0 (socket says 172.21.15.7:54314) 2023-03-03T15:45:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f75098700 1 -- 172.21.15.7:0/2125249907 learned_addr learned my addr 172.21.15.7:0/2125249907 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:45:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f7609a700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f701586c0 0x7f1f700706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f75899700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f700c1da0 0x7f1f700c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f75098700 1 -- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f701586c0 msgr2=0x7f1f700706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f75098700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f701586c0 0x7f1f700706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f75098700 1 -- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f700c1da0 msgr2=0x7f1f700c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f75098700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f700c1da0 0x7f1f700c5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f75098700 1 -- 172.21.15.7:0/2125249907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f68009b30 con 0x7f1f7013a0f0 2023-03-03T15:45:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f75899700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f700c1da0 0x7f1f700c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:45:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.812+0000 7f1f7609a700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f701586c0 0x7f1f700706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:45:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.813+0000 7f1f75098700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f7013a0f0 0x7f1f70070190 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000d830 tx=0x7f1f6000dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.813+0000 7f1f667fc700 1 -- 172.21.15.7:0/2125249907 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f600125f0 con 0x7f1f7013a0f0 2023-03-03T15:45:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.813+0000 7f1f667fc700 1 -- 172.21.15.7:0/2125249907 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f60012750 con 0x7f1f7013a0f0 2023-03-03T15:45:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.813+0000 7f1f667fc700 1 -- 172.21.15.7:0/2125249907 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60012b00 con 0x7f1f7013a0f0 2023-03-03T15:45:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.813+0000 7f1f77afd700 1 -- 172.21.15.7:0/2125249907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7006e1a0 con 0x7f1f7013a0f0 2023-03-03T15:45:21.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.813+0000 7f1f77afd700 1 -- 172.21.15.7:0/2125249907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f701581a0 con 0x7f1f7013a0f0 2023-03-03T15:45:21.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.814+0000 7f1f77afd700 1 -- 172.21.15.7:0/2125249907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7006e330 con 0x7f1f7013a0f0 2023-03-03T15:45:21.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.814+0000 7f1f667fc700 1 -- 172.21.15.7:0/2125249907 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f60012c60 con 0x7f1f7013a0f0 2023-03-03T15:45:21.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.815+0000 7f1f667fc700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f540682f0 0x7f1f5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:45:21.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.815+0000 7f1f667fc700 1 -- 172.21.15.7:0/2125249907 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1f60086640 con 0x7f1f7013a0f0 2023-03-03T15:45:21.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.815+0000 7f1f75899700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f540682f0 0x7f1f5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:45:21.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.815+0000 7f1f75899700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f540682f0 0x7f1f5406a7a0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f1f70135450 tx=0x7f1f6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:45:21.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:21.819+0000 7f1f667fc700 1 -- 172.21.15.7:0/2125249907 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f600529e0 con 0x7f1f7013a0f0 2023-03-03T15:45:22.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.058+0000 7f1f77afd700 1 -- 172.21.15.7:0/2125249907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f70074bb0 con 0x7f1f7013a0f0 2023-03-03T15:45:22.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.059+0000 7f1f667fc700 1 -- 172.21.15.7:0/2125249907 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1f60056000 con 0x7f1f7013a0f0 2023-03-03T15:45:22.061 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:45:22.062 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:45:22.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 -- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f540682f0 msgr2=0x7f1f5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:22.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f540682f0 0x7f1f5406a7a0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f1f70135450 tx=0x7f1f6c009040 comp rx=0 tx=0).stop 2023-03-03T15:45:22.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 -- 172.21.15.7:0/2125249907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f7013a0f0 msgr2=0x7f1f70070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:45:22.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f7013a0f0 0x7f1f70070190 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000d830 tx=0x7f1f6000dbf0 comp rx=0 tx=0).stop 2023-03-03T15:45:22.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 -- 172.21.15.7:0/2125249907 shutdown_connections 2023-03-03T15:45:22.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f701586c0 0x7f1f700706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:22.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f700c1da0 0x7f1f700c5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:22.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f540682f0 0x7f1f5406a7a0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:22.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 --2- 172.21.15.7:0/2125249907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f7013a0f0 0x7f1f70070190 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:45:22.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 -- 172.21.15.7:0/2125249907 >> 172.21.15.7:0/2125249907 conn(0x7f1f7006c110 msgr2=0x7f1f700c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:45:22.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 -- 172.21.15.7:0/2125249907 shutdown_connections 2023-03-03T15:45:22.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:45:22.061+0000 7f1f77afd700 1 -- 172.21.15.7:0/2125249907 wait complete. 2023-03-03T15:45:22.065 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:45:22.075 INFO:tasks.mds_thrash.fs.[b]:waiting for 102 secs before thrashing 2023-03-03T15:45:38.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:45:38.222 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:01.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:46:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.322+0000 7fdceebbb700 1 Processor -- start 2023-03-03T15:46:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.322+0000 7fdceebbb700 1 -- start start 2023-03-03T15:46:01.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.323+0000 7fdceebbb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdce814b0d0 0x7fdce8071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:01.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.323+0000 7fdceebbb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8071d40 0x7fdce8158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.323+0000 7fdceebbb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce80a9c20 con 0x7fdce80aad20 2023-03-03T15:46:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.323+0000 7fdceebbb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdce8154f10 con 0x7fdce814b0d0 2023-03-03T15:46:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.323+0000 7fdceebbb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce80ac3c0 con 0x7fdce8071d40 2023-03-03T15:46:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.323+0000 7fdced158700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8071d40 0x7fdce8158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.323+0000 7fdcec957700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdce814b0d0 0x7fdce8071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.323+0000 7fdce7fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fdce80aad20 0x7fdce8157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:51844/0 (socket says 172.21.15.7:51844) 2023-03-03T15:46:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdce7fff700 1 -- 172.21.15.7:0/3747060628 learned_addr learned my addr 172.21.15.7:0/3747060628 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:46:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdced158700 1 -- 172.21.15.7:0/3747060628 >> v2:172.21.15.143:3300/0 conn(0x7fdce814b0d0 msgr2=0x7fdce8071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:46:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdced158700 1 --2- 172.21.15.7:0/3747060628 >> v2:172.21.15.143:3300/0 conn(0x7fdce814b0d0 0x7fdce8071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdced158700 1 -- 172.21.15.7:0/3747060628 >> v1:172.21.15.143:6789/0 conn(0x7fdce80aad20 legacy=0x7fdce8157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdce77fe700 1 -- 172.21.15.7:0/3747060628 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159253065 0 0) 0x7fdce80a9c20 con 0x7fdce80aad20 2023-03-03T15:46:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdced158700 1 -- 172.21.15.7:0/3747060628 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce806fc70 con 0x7fdce8071d40 2023-03-03T15:46:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdcec957700 1 --2- 172.21.15.7:0/3747060628 >> v2:172.21.15.143:3300/0 conn(0x7fdce814b0d0 0x7fdce8071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:46:01.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdced158700 1 --2- 172.21.15.7:0/3747060628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8071d40 0x7fdce8158500 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009d90 tx=0x7fdce000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a76ff418b047d185 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:46:01.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdce77fe700 1 -- 172.21.15.7:0/3747060628 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce000e070 con 0x7fdce8071d40 2023-03-03T15:46:01.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdce77fe700 1 -- 172.21.15.7:0/3747060628 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce000bb20 con 0x7fdce8071d40 2023-03-03T15:46:01.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.324+0000 7fdce77fe700 1 -- 172.21.15.7:0/3747060628 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0012430 con 0x7fdce8071d40 2023-03-03T15:46:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.325+0000 7fdceebbb700 1 -- 172.21.15.7:0/3747060628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8071d40 msgr2=0x7fdce8158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.325+0000 7fdceebbb700 1 --2- 172.21.15.7:0/3747060628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8071d40 0x7fdce8158500 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdce0009d90 tx=0x7fdce000b330 comp rx=0 tx=0).stop 2023-03-03T15:46:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.325+0000 7fdceebbb700 1 -- 172.21.15.7:0/3747060628 shutdown_connections 2023-03-03T15:46:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.325+0000 7fdceebbb700 1 --2- 172.21.15.7:0/3747060628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce8071d40 0x7fdce8158500 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.325+0000 7fdceebbb700 1 --2- 172.21.15.7:0/3747060628 >> v2:172.21.15.143:3300/0 conn(0x7fdce814b0d0 0x7fdce8071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.325+0000 7fdceebbb700 1 -- 172.21.15.7:0/3747060628 >> 172.21.15.7:0/3747060628 conn(0x7fdce80c8900 msgr2=0x7fdce80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:46:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.325+0000 7fdceebbb700 1 -- 172.21.15.7:0/3747060628 shutdown_connections 2023-03-03T15:46:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.325+0000 7fdceebbb700 1 -- 172.21.15.7:0/3747060628 wait complete. 2023-03-03T15:46:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.326+0000 7fdceebbb700 1 Processor -- start 2023-03-03T15:46:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.326+0000 7fdceebbb700 1 -- start start 2023-03-03T15:46:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdceebbb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8071d40 0x7fdce81ea230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdceebbb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce80aad20 0x7fdce8153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdceebbb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdce814b0d0 0x7fdce8149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdceebbb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce8149f20 con 0x7fdce8071d40 2023-03-03T15:46:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdceebbb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce8153cb0 con 0x7fdce814b0d0 2023-03-03T15:46:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdceebbb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce8154230 con 0x7fdce80aad20 2023-03-03T15:46:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdcec957700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8071d40 0x7fdce81ea230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdcec957700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8071d40 0x7fdce81ea230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54348/0 (socket says 172.21.15.7:54348) 2023-03-03T15:46:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdcec957700 1 -- 172.21.15.7:0/3849589927 learned_addr learned my addr 172.21.15.7:0/3849589927 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:46:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdce7fff700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce80aad20 0x7fdce8153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.327+0000 7fdced158700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdce814b0d0 0x7fdce8149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdcec957700 1 -- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce80aad20 msgr2=0x7fdce8153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdcec957700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce80aad20 0x7fdce8153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdcec957700 1 -- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdce814b0d0 msgr2=0x7fdce8149010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdcec957700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdce814b0d0 0x7fdce8149010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdcec957700 1 -- 172.21.15.7:0/3849589927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc0094a0 con 0x7fdce8071d40 2023-03-03T15:46:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdcec957700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8071d40 0x7fdce81ea230 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc00ebe0 tx=0x7fdcdc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:46:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdce57fa700 1 -- 172.21.15.7:0/3849589927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc00c850 con 0x7fdce8071d40 2023-03-03T15:46:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdce57fa700 1 -- 172.21.15.7:0/3849589927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcdc00c9b0 con 0x7fdce8071d40 2023-03-03T15:46:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdceebbb700 1 -- 172.21.15.7:0/3849589927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce0009a70 con 0x7fdce8071d40 2023-03-03T15:46:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdce57fa700 1 -- 172.21.15.7:0/3849589927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc00cd40 con 0x7fdce8071d40 2023-03-03T15:46:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.328+0000 7fdceebbb700 1 -- 172.21.15.7:0/3849589927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce806f6b0 con 0x7fdce8071d40 2023-03-03T15:46:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.329+0000 7fdceebbb700 1 -- 172.21.15.7:0/3849589927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce8155160 con 0x7fdce8071d40 2023-03-03T15:46:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.330+0000 7fdce57fa700 1 -- 172.21.15.7:0/3849589927 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdcdc019030 con 0x7fdce8071d40 2023-03-03T15:46:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.330+0000 7fdce57fa700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdccc0682f0 0x7fdccc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:46:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.330+0000 7fdce57fa700 1 -- 172.21.15.7:0/3849589927 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdcdc0860b0 con 0x7fdce8071d40 2023-03-03T15:46:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.330+0000 7fdce7fff700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdccc0682f0 0x7fdccc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:46:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.331+0000 7fdce7fff700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdccc0682f0 0x7fdccc06a7a0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fdce814aac0 tx=0x7fdcd4007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:46:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.334+0000 7fdce57fa700 1 -- 172.21.15.7:0/3849589927 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdcdc052450 con 0x7fdce8071d40 2023-03-03T15:46:01.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.571+0000 7fdceebbb700 1 -- 172.21.15.7:0/3849589927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdce806f990 con 0x7fdce8071d40 2023-03-03T15:46:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.572+0000 7fdce57fa700 1 -- 172.21.15.7:0/3849589927 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdcdc017090 con 0x7fdce8071d40 2023-03-03T15:46:01.573 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:46:01.574 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:46:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 -- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdccc0682f0 msgr2=0x7fdccc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdccc0682f0 0x7fdccc06a7a0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fdce814aac0 tx=0x7fdcd4007de0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 -- 172.21.15.7:0/3849589927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8071d40 msgr2=0x7fdce81ea230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8071d40 0x7fdce81ea230 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc00ebe0 tx=0x7fdcdc00eef0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 -- 172.21.15.7:0/3849589927 shutdown_connections 2023-03-03T15:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdce814b0d0 0x7fdce8149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdccc0682f0 0x7fdccc06a7a0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdce8071d40 0x7fdce81ea230 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 --2- 172.21.15.7:0/3849589927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdce80aad20 0x7fdce8153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 -- 172.21.15.7:0/3849589927 >> 172.21.15.7:0/3849589927 conn(0x7fdce80c8900 msgr2=0x7fdce81471b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 -- 172.21.15.7:0/3849589927 shutdown_connections 2023-03-03T15:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:46:01.574+0000 7fdceebbb700 1 -- 172.21.15.7:0/3849589927 wait complete. 2023-03-03T15:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:46:01.587 INFO:tasks.mds_thrash.fs.[a]:waiting for 82 secs before thrashing 2023-03-03T15:46:08.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:08.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:38.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:46:38.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:04.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:47:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.210+0000 7f57d1e37700 1 Processor -- start 2023-03-03T15:47:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.210+0000 7f57d1e37700 1 -- start start 2023-03-03T15:47:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.210+0000 7f57d1e37700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f57cc071c30 0x7f57cc0c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57d1e37700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f57cc155950 0x7f57cc155d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57d1e37700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57cc0713c0 con 0x7f57cc0aac00 2023-03-03T15:47:04.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57d1e37700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57cc1562d0 con 0x7f57cc155950 2023-03-03T15:47:04.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57d1e37700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57cc152c70 con 0x7f57cc071c30 2023-03-03T15:47:04.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57cb7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f57cc0aac00 0x7f57cc0beb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48378/0 (socket says 172.21.15.7:48378) 2023-03-03T15:47:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57cb7fe700 1 -- 172.21.15.7:0/3357950035 learned_addr learned my addr 172.21.15.7:0/3357950035 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:47:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57cbfff700 1 --2- 172.21.15.7:0/3357950035 >> v2:172.21.15.7:3300/0 conn(0x7f57cc155950 0x7f57cc155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57caffd700 1 --2- 172.21.15.7:0/3357950035 >> v2:172.21.15.143:3300/0 conn(0x7f57cc071c30 0x7f57cc0c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57ca7fc700 1 -- 172.21.15.7:0/3357950035 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262732835 0 0) 0x7f57cc0713c0 con 0x7f57cc0aac00 2023-03-03T15:47:04.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57ca7fc700 1 -- 172.21.15.7:0/3357950035 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b40035b0 con 0x7f57cc0aac00 2023-03-03T15:47:04.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57cbfff700 1 -- 172.21.15.7:0/3357950035 >> v2:172.21.15.143:3300/0 conn(0x7f57cc071c30 msgr2=0x7f57cc0c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:04.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57cbfff700 1 --2- 172.21.15.7:0/3357950035 >> v2:172.21.15.143:3300/0 conn(0x7f57cc071c30 0x7f57cc0c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57cbfff700 1 -- 172.21.15.7:0/3357950035 >> v1:172.21.15.7:6789/0 conn(0x7f57cc0aac00 legacy=0x7f57cc0beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57cbfff700 1 -- 172.21.15.7:0/3357950035 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc0c4930 con 0x7f57cc155950 2023-03-03T15:47:04.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57cbfff700 1 --2- 172.21.15.7:0/3357950035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc155950 0x7f57cc155d90 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0c60b0 tx=0x7f57c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85daf251bb74996 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:04.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57ca7fc700 1 -- 172.21.15.7:0/3357950035 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c400e070 con 0x7f57cc155950 2023-03-03T15:47:04.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57ca7fc700 1 -- 172.21.15.7:0/3357950035 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c400bb20 con 0x7f57cc155950 2023-03-03T15:47:04.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.211+0000 7f57ca7fc700 1 -- 172.21.15.7:0/3357950035 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c400be90 con 0x7f57cc155950 2023-03-03T15:47:04.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.212+0000 7f57d1e37700 1 -- 172.21.15.7:0/3357950035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc155950 msgr2=0x7f57cc155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.212+0000 7f57d1e37700 1 --2- 172.21.15.7:0/3357950035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc155950 0x7f57cc155d90 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0c60b0 tx=0x7f57c400b330 comp rx=0 tx=0).stop 2023-03-03T15:47:04.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.212+0000 7f57d1e37700 1 -- 172.21.15.7:0/3357950035 shutdown_connections 2023-03-03T15:47:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.212+0000 7f57d1e37700 1 --2- 172.21.15.7:0/3357950035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc155950 0x7f57cc155d90 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.212+0000 7f57d1e37700 1 --2- 172.21.15.7:0/3357950035 >> v2:172.21.15.143:3300/0 conn(0x7f57cc071c30 0x7f57cc0c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.212+0000 7f57d1e37700 1 -- 172.21.15.7:0/3357950035 >> 172.21.15.7:0/3357950035 conn(0x7f57cc0736f0 msgr2=0x7f57cc073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.212+0000 7f57d1e37700 1 -- 172.21.15.7:0/3357950035 shutdown_connections 2023-03-03T15:47:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.212+0000 7f57d1e37700 1 -- 172.21.15.7:0/3357950035 wait complete. 2023-03-03T15:47:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.213+0000 7f57d1e37700 1 Processor -- start 2023-03-03T15:47:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.213+0000 7f57d1e37700 1 -- start start 2023-03-03T15:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57d1e37700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc071c30 0x7f57cc1ed4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57d1e37700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57cc0aac00 0x7f57cc1e9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57d1e37700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57cc155950 0x7f57cc06b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57d1e37700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc146370 con 0x7f57cc071c30 2023-03-03T15:47:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57d1e37700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc1eda20 con 0x7f57cc0aac00 2023-03-03T15:47:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57d1e37700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57cc1e9560 con 0x7f57cc155950 2023-03-03T15:47:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57cb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc071c30 0x7f57cc1ed4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57cb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc071c30 0x7f57cc1ed4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54404/0 (socket says 172.21.15.7:54404) 2023-03-03T15:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57cb7fe700 1 -- 172.21.15.7:0/683852260 learned_addr learned my addr 172.21.15.7:0/683852260 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57cbfff700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57cc155950 0x7f57cc06b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.214+0000 7f57caffd700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57cc0aac00 0x7f57cc1e9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57cb7fe700 1 -- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57cc155950 msgr2=0x7f57cc06b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57cb7fe700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57cc155950 0x7f57cc06b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57cb7fe700 1 -- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57cc0aac00 msgr2=0x7f57cc1e9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57cb7fe700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57cc0aac00 0x7f57cc1e9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57cb7fe700 1 -- 172.21.15.7:0/683852260 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c4009a70 con 0x7f57cc071c30 2023-03-03T15:47:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57caffd700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57cc0aac00 0x7f57cc1e9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57cb7fe700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc071c30 0x7f57cc1ed4e0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00c4d0 tx=0x7f57bc00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57b3fff700 1 -- 172.21.15.7:0/683852260 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc005d70 con 0x7f57cc071c30 2023-03-03T15:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57b3fff700 1 -- 172.21.15.7:0/683852260 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc0070d0 con 0x7f57cc071c30 2023-03-03T15:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57d1e37700 1 -- 172.21.15.7:0/683852260 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57cc0c4b80 con 0x7f57cc071c30 2023-03-03T15:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57b3fff700 1 -- 172.21.15.7:0/683852260 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc007440 con 0x7f57cc071c30 2023-03-03T15:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.215+0000 7f57d1e37700 1 -- 172.21.15.7:0/683852260 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc152360 con 0x7f57cc071c30 2023-03-03T15:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.216+0000 7f57d1e37700 1 -- 172.21.15.7:0/683852260 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57cc152620 con 0x7f57cc071c30 2023-03-03T15:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.216+0000 7f57b3fff700 1 -- 172.21.15.7:0/683852260 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f57bc019030 con 0x7f57cc071c30 2023-03-03T15:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.217+0000 7f57b3fff700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57b80682f0 0x7f57b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.217+0000 7f57b3fff700 1 -- 172.21.15.7:0/683852260 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f57bc085320 con 0x7f57cc071c30 2023-03-03T15:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.217+0000 7f57caffd700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57b80682f0 0x7f57b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.218+0000 7f57caffd700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57b80682f0 0x7f57b806a7a0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f57cc1522b0 tx=0x7f57c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.221+0000 7f57b3fff700 1 -- 172.21.15.7:0/683852260 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57bc04b5e0 con 0x7f57cc071c30 2023-03-03T15:47:04.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.456+0000 7f57d1e37700 1 -- 172.21.15.7:0/683852260 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57cc074bb0 con 0x7f57cc071c30 2023-03-03T15:47:04.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.457+0000 7f57b3fff700 1 -- 172.21.15.7:0/683852260 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f57bc017090 con 0x7f57cc071c30 2023-03-03T15:47:04.459 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:47:04.460 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:47:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 -- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57b80682f0 msgr2=0x7f57b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57b80682f0 0x7f57b806a7a0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f57cc1522b0 tx=0x7f57c0008040 comp rx=0 tx=0).stop 2023-03-03T15:47:04.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 -- 172.21.15.7:0/683852260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc071c30 msgr2=0x7f57cc1ed4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc071c30 0x7f57cc1ed4e0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00c4d0 tx=0x7f57bc00c7e0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 -- 172.21.15.7:0/683852260 shutdown_connections 2023-03-03T15:47:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57cc155950 0x7f57cc06b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57b80682f0 0x7f57b806a7a0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57cc071c30 0x7f57cc1ed4e0 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 --2- 172.21.15.7:0/683852260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57cc0aac00 0x7f57cc1e9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 -- 172.21.15.7:0/683852260 >> 172.21.15.7:0/683852260 conn(0x7f57cc0736f0 msgr2=0x7f57cc0a9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:04.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 -- 172.21.15.7:0/683852260 shutdown_connections 2023-03-03T15:47:04.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.459+0000 7f57d1e37700 1 -- 172.21.15.7:0/683852260 wait complete. 2023-03-03T15:47:04.464 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:47:04.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:47:04.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.597+0000 7f7c6dd8a700 1 Processor -- start 2023-03-03T15:47:04.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.597+0000 7f7c6dd8a700 1 -- start start 2023-03-03T15:47:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.597+0000 7f7c6dd8a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c680ab5e0 0x7f7c680aba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.597+0000 7f7c6dd8a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6814fd60 con 0x7f7c6814f990 2023-03-03T15:47:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.597+0000 7f7c6dd8a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6806c030 con 0x7f7c68152ed0 2023-03-03T15:47:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.597+0000 7f7c6dd8a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c68153340 con 0x7f7c680ab5e0 2023-03-03T15:47:04.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.597+0000 7f7c67fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c680ab5e0 0x7f7c680aba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.597+0000 7f7c677fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7c6814f990 0x7f7c681562b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48392/0 (socket says 172.21.15.7:48392) 2023-03-03T15:47:04.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.598+0000 7f7c677fe700 1 -- 172.21.15.7:0/591704246 learned_addr learned my addr 172.21.15.7:0/591704246 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:47:04.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.598+0000 7f7c67fff700 1 -- 172.21.15.7:0/591704246 >> v1:172.21.15.143:6789/0 conn(0x7f7c68152ed0 legacy=0x7f7c6806b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:04.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.598+0000 7f7c67fff700 1 -- 172.21.15.7:0/591704246 >> v1:172.21.15.7:6789/0 conn(0x7f7c6814f990 legacy=0x7f7c681562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.598+0000 7f7c67fff700 1 -- 172.21.15.7:0/591704246 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c680c29c0 con 0x7f7c680ab5e0 2023-03-03T15:47:04.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.598+0000 7f7c667fc700 1 -- 172.21.15.7:0/591704246 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012242268 0 0) 0x7f7c6814fd60 con 0x7f7c6814f990 2023-03-03T15:47:04.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.598+0000 7f7c67fff700 1 --2- 172.21.15.7:0/591704246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c680ab5e0 0x7f7c680aba20 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009d90 tx=0x7f7c6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d9d33a94105133 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:04.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.598+0000 7f7c667fc700 1 -- 172.21.15.7:0/591704246 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6000e070 con 0x7f7c680ab5e0 2023-03-03T15:47:04.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.598+0000 7f7c667fc700 1 -- 172.21.15.7:0/591704246 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6000bbd0 con 0x7f7c680ab5e0 2023-03-03T15:47:04.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.598+0000 7f7c667fc700 1 -- 172.21.15.7:0/591704246 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60012430 con 0x7f7c680ab5e0 2023-03-03T15:47:04.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.599+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/591704246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c680ab5e0 msgr2=0x7f7c680aba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.599+0000 7f7c6dd8a700 1 --2- 172.21.15.7:0/591704246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c680ab5e0 0x7f7c680aba20 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009d90 tx=0x7f7c6000b330 comp rx=0 tx=0).stop 2023-03-03T15:47:04.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.599+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/591704246 shutdown_connections 2023-03-03T15:47:04.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.599+0000 7f7c6dd8a700 1 --2- 172.21.15.7:0/591704246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c680ab5e0 0x7f7c680aba20 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.599+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/591704246 >> 172.21.15.7:0/591704246 conn(0x7f7c680c62e0 msgr2=0x7f7c680c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:04.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.599+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/591704246 shutdown_connections 2023-03-03T15:47:04.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.599+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/591704246 wait complete. 2023-03-03T15:47:04.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.600+0000 7f7c6dd8a700 1 Processor -- start 2023-03-03T15:47:04.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.600+0000 7f7c6dd8a700 1 -- start start 2023-03-03T15:47:04.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c6dd8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c680ab5e0 0x7f7c68154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c6dd8a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c6814f990 0x7f7c681e6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c6dd8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c68152ed0 0x7f7c681e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c6dd8a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c681dd290 con 0x7f7c6814f990 2023-03-03T15:47:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c6dd8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c681db940 con 0x7f7c68152ed0 2023-03-03T15:47:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c6dd8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c681e1e20 con 0x7f7c680ab5e0 2023-03-03T15:47:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c66ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c6814f990 0x7f7c681e6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c66ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c6814f990 0x7f7c681e6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54420/0 (socket says 172.21.15.7:54420) 2023-03-03T15:47:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c66ffd700 1 -- 172.21.15.7:0/2001000700 learned_addr learned my addr 172.21.15.7:0/2001000700 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:47:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c677fe700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c680ab5e0 0x7f7c68154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c67fff700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c68152ed0 0x7f7c681e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c66ffd700 1 -- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c680ab5e0 msgr2=0x7f7c68154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c66ffd700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c680ab5e0 0x7f7c68154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c66ffd700 1 -- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c68152ed0 msgr2=0x7f7c681e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c66ffd700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c68152ed0 0x7f7c681e71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c66ffd700 1 -- 172.21.15.7:0/2001000700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c60009a70 con 0x7f7c6814f990 2023-03-03T15:47:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.601+0000 7f7c66ffd700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c6814f990 0x7f7c681e6ca0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00c450 tx=0x7f7c5c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.602+0000 7f7c4ffff700 1 -- 172.21.15.7:0/2001000700 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0052d0 con 0x7f7c6814f990 2023-03-03T15:47:04.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.602+0000 7f7c4ffff700 1 -- 172.21.15.7:0/2001000700 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c005430 con 0x7f7c6814f990 2023-03-03T15:47:04.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.602+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/2001000700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c680c26e0 con 0x7f7c6814f990 2023-03-03T15:47:04.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.602+0000 7f7c4ffff700 1 -- 172.21.15.7:0/2001000700 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0038c0 con 0x7f7c6814f990 2023-03-03T15:47:04.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.602+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/2001000700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c680c2400 con 0x7f7c6814f990 2023-03-03T15:47:04.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.603+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/2001000700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c68074bb0 con 0x7f7c6814f990 2023-03-03T15:47:04.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.603+0000 7f7c4ffff700 1 -- 172.21.15.7:0/2001000700 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7c5c019030 con 0x7f7c6814f990 2023-03-03T15:47:04.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.603+0000 7f7c4ffff700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 0x7f7c5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:04.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.603+0000 7f7c4ffff700 1 -- 172.21.15.7:0/2001000700 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7c5c085440 con 0x7f7c6814f990 2023-03-03T15:47:04.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.604+0000 7f7c677fe700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 0x7f7c5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:04.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.604+0000 7f7c677fe700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 0x7f7c5406a7a0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f7c58007fd0 tx=0x7f7c58009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:04.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.608+0000 7f7c4ffff700 1 -- 172.21.15.7:0/2001000700 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7c5c0517e0 con 0x7f7c6814f990 2023-03-03T15:47:04.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.844+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/2001000700 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c681e77d0 con 0x7f7c6814f990 2023-03-03T15:47:04.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.844+0000 7f7c4ffff700 1 -- 172.21.15.7:0/2001000700 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7c681e77d0 con 0x7f7c6814f990 2023-03-03T15:47:04.846 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:47:04.846 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:47:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 msgr2=0x7f7c5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 0x7f7c5406a7a0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f7c58007fd0 tx=0x7f7c58009040 comp rx=0 tx=0).stop 2023-03-03T15:47:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/2001000700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c6814f990 msgr2=0x7f7c681e6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c6814f990 0x7f7c681e6ca0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00c450 tx=0x7f7c5c00c810 comp rx=0 tx=0).stop 2023-03-03T15:47:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/2001000700 shutdown_connections 2023-03-03T15:47:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c68152ed0 0x7f7c681e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c680ab5e0 0x7f7c68154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c540682f0 0x7f7c5406a7a0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 --2- 172.21.15.7:0/2001000700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c6814f990 0x7f7c681e6ca0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/2001000700 >> 172.21.15.7:0/2001000700 conn(0x7f7c680c62e0 msgr2=0x7f7c6806c680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/2001000700 shutdown_connections 2023-03-03T15:47:04.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:04.847+0000 7f7c6dd8a700 1 -- 172.21.15.7:0/2001000700 wait complete. 2023-03-03T15:47:04.850 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:47:04.860 INFO:tasks.mds_thrash.fs.[b]:waiting for 76 secs before thrashing 2023-03-03T15:47:08.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:08.416 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:23.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:47:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.726+0000 7ff8c8f22700 1 Processor -- start 2023-03-03T15:47:23.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.726+0000 7ff8c8f22700 1 -- start start 2023-03-03T15:47:23.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.727+0000 7ff8c8f22700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8c414fda0 0x7ff8c4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:23.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.727+0000 7ff8c8f22700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40abe20 0x7ff8c41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.727+0000 7ff8c8f22700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c40c2f30 con 0x7ff8c414f9d0 2023-03-03T15:47:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.727+0000 7ff8c8f22700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8c40c31d0 con 0x7ff8c414fda0 2023-03-03T15:47:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.727+0000 7ff8c8f22700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c41524a0 con 0x7ff8c40abe20 2023-03-03T15:47:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.727+0000 7ff8c1d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff8c414fda0 0x7ff8c4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.727+0000 7ff8c2d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40abe20 0x7ff8c41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:23.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.727+0000 7ff8c259c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ff8c414f9d0 0x7ff8c4071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:51932/0 (socket says 172.21.15.7:51932) 2023-03-03T15:47:23.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.727+0000 7ff8c259c700 1 -- 172.21.15.7:0/3689774699 learned_addr learned my addr 172.21.15.7:0/3689774699 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:47:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.728+0000 7ff8c159a700 1 -- 172.21.15.7:0/3689774699 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309187960 0 0) 0x7ff8c40c2f30 con 0x7ff8c414f9d0 2023-03-03T15:47:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.728+0000 7ff8c159a700 1 -- 172.21.15.7:0/3689774699 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ac0035b0 con 0x7ff8c414f9d0 2023-03-03T15:47:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.728+0000 7ff8c2d9d700 1 -- 172.21.15.7:0/3689774699 >> v2:172.21.15.143:3300/0 conn(0x7ff8c414fda0 msgr2=0x7ff8c4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:23.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.728+0000 7ff8c2d9d700 1 --2- 172.21.15.7:0/3689774699 >> v2:172.21.15.143:3300/0 conn(0x7ff8c414fda0 0x7ff8c4156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.728+0000 7ff8c2d9d700 1 -- 172.21.15.7:0/3689774699 >> v1:172.21.15.143:6789/0 conn(0x7ff8c414f9d0 legacy=0x7ff8c4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.728+0000 7ff8c2d9d700 1 -- 172.21.15.7:0/3689774699 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c40c34b0 con 0x7ff8c40abe20 2023-03-03T15:47:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.728+0000 7ff8c2d9d700 1 --2- 172.21.15.7:0/3689774699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40abe20 0x7ff8c41529c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009d90 tx=0x7ff8bc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f89b56e65a62e346 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.728+0000 7ff8c159a700 1 -- 172.21.15.7:0/3689774699 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc00e070 con 0x7ff8c40abe20 2023-03-03T15:47:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.728+0000 7ff8c159a700 1 -- 172.21.15.7:0/3689774699 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8bc00bb20 con 0x7ff8c40abe20 2023-03-03T15:47:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.728+0000 7ff8c159a700 1 -- 172.21.15.7:0/3689774699 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc012430 con 0x7ff8c40abe20 2023-03-03T15:47:23.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.729+0000 7ff8c8f22700 1 -- 172.21.15.7:0/3689774699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40abe20 msgr2=0x7ff8c41529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.729+0000 7ff8c8f22700 1 --2- 172.21.15.7:0/3689774699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40abe20 0x7ff8c41529c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009d90 tx=0x7ff8bc00b330 comp rx=0 tx=0).stop 2023-03-03T15:47:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.729+0000 7ff8c8f22700 1 -- 172.21.15.7:0/3689774699 shutdown_connections 2023-03-03T15:47:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.729+0000 7ff8c8f22700 1 --2- 172.21.15.7:0/3689774699 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c40abe20 0x7ff8c41529c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.729+0000 7ff8c8f22700 1 --2- 172.21.15.7:0/3689774699 >> v2:172.21.15.143:3300/0 conn(0x7ff8c414fda0 0x7ff8c4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.729+0000 7ff8c8f22700 1 -- 172.21.15.7:0/3689774699 >> 172.21.15.7:0/3689774699 conn(0x7ff8c40c8900 msgr2=0x7ff8c40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.729+0000 7ff8c8f22700 1 -- 172.21.15.7:0/3689774699 shutdown_connections 2023-03-03T15:47:23.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.729+0000 7ff8c8f22700 1 -- 172.21.15.7:0/3689774699 wait complete. 2023-03-03T15:47:23.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.730+0000 7ff8c8f22700 1 Processor -- start 2023-03-03T15:47:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.730+0000 7ff8c8f22700 1 -- start start 2023-03-03T15:47:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c8f22700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8c40abe20 0x7ff8c41eda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c8f22700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c414f9d0 0x7ff8c40c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c8f22700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c414fda0 0x7ff8c41dd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c8f22700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c41de9b0 con 0x7ff8c40abe20 2023-03-03T15:47:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c8f22700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c41dbff0 con 0x7ff8c414fda0 2023-03-03T15:47:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c8f22700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c41dba30 con 0x7ff8c414f9d0 2023-03-03T15:47:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8c40abe20 0x7ff8c41eda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8c40abe20 0x7ff8c41eda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54438/0 (socket says 172.21.15.7:54438) 2023-03-03T15:47:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c259c700 1 -- 172.21.15.7:0/1999085910 learned_addr learned my addr 172.21.15.7:0/1999085910 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:47:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c2d9d700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c414fda0 0x7ff8c41dd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c1d9b700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c414f9d0 0x7ff8c40c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c259c700 1 -- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c414f9d0 msgr2=0x7ff8c40c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c259c700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c414f9d0 0x7ff8c40c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c259c700 1 -- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c414fda0 msgr2=0x7ff8c41dd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c259c700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c414fda0 0x7ff8c41dd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.731+0000 7ff8c259c700 1 -- 172.21.15.7:0/1999085910 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc009a70 con 0x7ff8c40abe20 2023-03-03T15:47:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.732+0000 7ff8c259c700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8c40abe20 0x7ff8c41eda70 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7ff8b800c700 tx=0x7ff8b800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.732+0000 7ff8b2ffd700 1 -- 172.21.15.7:0/1999085910 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8005d70 con 0x7ff8c40abe20 2023-03-03T15:47:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.732+0000 7ff8b2ffd700 1 -- 172.21.15.7:0/1999085910 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b80070d0 con 0x7ff8c40abe20 2023-03-03T15:47:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.732+0000 7ff8c8f22700 1 -- 172.21.15.7:0/1999085910 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c40ac350 con 0x7ff8c40abe20 2023-03-03T15:47:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.732+0000 7ff8b2ffd700 1 -- 172.21.15.7:0/1999085910 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8007460 con 0x7ff8c40abe20 2023-03-03T15:47:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.732+0000 7ff8c8f22700 1 -- 172.21.15.7:0/1999085910 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c40ab630 con 0x7ff8c40abe20 2023-03-03T15:47:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.733+0000 7ff8c8f22700 1 -- 172.21.15.7:0/1999085910 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c4071fa0 con 0x7ff8c40abe20 2023-03-03T15:47:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.733+0000 7ff8b2ffd700 1 -- 172.21.15.7:0/1999085910 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff8b8019030 con 0x7ff8c40abe20 2023-03-03T15:47:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.734+0000 7ff8b2ffd700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8a4068320 0x7ff8a406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.734+0000 7ff8b2ffd700 1 -- 172.21.15.7:0/1999085910 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff8b80853e0 con 0x7ff8c40abe20 2023-03-03T15:47:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.734+0000 7ff8c1d9b700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8a4068320 0x7ff8a406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.734+0000 7ff8c1d9b700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8a4068320 0x7ff8a406a7d0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40c2900 tx=0x7ff8b4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.738+0000 7ff8b2ffd700 1 -- 172.21.15.7:0/1999085910 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff8b8051720 con 0x7ff8c40abe20 2023-03-03T15:47:23.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.974+0000 7ff8c8f22700 1 -- 172.21.15.7:0/1999085910 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8c41dde90 con 0x7ff8c40abe20 2023-03-03T15:47:23.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.974+0000 7ff8b2ffd700 1 -- 172.21.15.7:0/1999085910 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff8c41dde90 con 0x7ff8c40abe20 2023-03-03T15:47:23.976 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:47:23.976 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:47:23.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 -- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8a4068320 msgr2=0x7ff8a406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:23.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8a4068320 0x7ff8a406a7d0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40c2900 tx=0x7ff8b4009210 comp rx=0 tx=0).stop 2023-03-03T15:47:23.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 -- 172.21.15.7:0/1999085910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8c40abe20 msgr2=0x7ff8c41eda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:23.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8c40abe20 0x7ff8c41eda70 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7ff8b800c700 tx=0x7ff8b800ca10 comp rx=0 tx=0).stop 2023-03-03T15:47:23.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 -- 172.21.15.7:0/1999085910 shutdown_connections 2023-03-03T15:47:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c414fda0 0x7ff8c41dd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8a4068320 0x7ff8a406a7d0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8c40abe20 0x7ff8c41eda70 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 --2- 172.21.15.7:0/1999085910 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8c414f9d0 0x7ff8c40c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 -- 172.21.15.7:0/1999085910 >> 172.21.15.7:0/1999085910 conn(0x7ff8c40c8900 msgr2=0x7ff8c4151280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 -- 172.21.15.7:0/1999085910 shutdown_connections 2023-03-03T15:47:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:23.976+0000 7ff8c8f22700 1 -- 172.21.15.7:0/1999085910 wait complete. 2023-03-03T15:47:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:47:23.989 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before thrashing 2023-03-03T15:47:28.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:47:29.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.129+0000 7ff0c2d2d700 1 Processor -- start 2023-03-03T15:47:29.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.129+0000 7ff0c2d2d700 1 -- start start 2023-03-03T15:47:29.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c2d2d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff0bc071990 0x7ff0bc071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:29.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c2d2d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc14b460 0x7ff0bc0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:29.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c2d2d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0bc151c00 0x7ff0bc06aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:29.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c2d2d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0bc1521a0 con 0x7ff0bc151c00 2023-03-03T15:47:29.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c2d2d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0bc070360 con 0x7ff0bc071990 2023-03-03T15:47:29.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c2d2d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc06a5e0 con 0x7ff0bc14b460 2023-03-03T15:47:29.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c12ca700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0bc151c00 0x7ff0bc06aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:29.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c12ca700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0bc151c00 0x7ff0bc06aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54452/0 (socket says 172.21.15.7:54452) 2023-03-03T15:47:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c12ca700 1 -- 172.21.15.7:0/1967553519 learned_addr learned my addr 172.21.15.7:0/1967553519 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:47:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0bbfff700 1 --2- 172.21.15.7:0/1967553519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc14b460 0x7ff0bc0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c0ac9700 1 --2- 172.21.15.7:0/1967553519 >> v2:172.21.15.143:3300/0 conn(0x7ff0bc071990 0x7ff0bc071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c12ca700 1 -- 172.21.15.7:0/1967553519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc14b460 msgr2=0x7ff0bc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c12ca700 1 --2- 172.21.15.7:0/1967553519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc14b460 0x7ff0bc0722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c12ca700 1 -- 172.21.15.7:0/1967553519 >> v2:172.21.15.143:3300/0 conn(0x7ff0bc071990 msgr2=0x7ff0bc071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:29.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c12ca700 1 --2- 172.21.15.7:0/1967553519 >> v2:172.21.15.143:3300/0 conn(0x7ff0bc071990 0x7ff0bc071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c12ca700 1 -- 172.21.15.7:0/1967553519 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc0c38b0 con 0x7ff0bc151c00 2023-03-03T15:47:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.130+0000 7ff0c12ca700 1 --2- 172.21.15.7:0/1967553519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 0x7ff0bc06aa70 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0c1de0 tx=0x7ff0b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5265e141f3b93e32 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0bb7fe700 1 -- 172.21.15.7:0/1967553519 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b400e070 con 0x7ff0bc151c00 2023-03-03T15:47:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0bb7fe700 1 -- 172.21.15.7:0/1967553519 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b400b9c0 con 0x7ff0bc151c00 2023-03-03T15:47:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0bb7fe700 1 -- 172.21.15.7:0/1967553519 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b400bd70 con 0x7ff0bc151c00 2023-03-03T15:47:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/1967553519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 msgr2=0x7ff0bc06aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0c2d2d700 1 --2- 172.21.15.7:0/1967553519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 0x7ff0bc06aa70 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0c1de0 tx=0x7ff0b400b330 comp rx=0 tx=0).stop 2023-03-03T15:47:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/1967553519 shutdown_connections 2023-03-03T15:47:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0c2d2d700 1 --2- 172.21.15.7:0/1967553519 >> v2:172.21.15.143:3300/0 conn(0x7ff0bc071990 0x7ff0bc071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0c2d2d700 1 --2- 172.21.15.7:0/1967553519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 0x7ff0bc06aa70 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0c2d2d700 1 --2- 172.21.15.7:0/1967553519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc14b460 0x7ff0bc0722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/1967553519 >> 172.21.15.7:0/1967553519 conn(0x7ff0bc0a94d0 msgr2=0x7ff0bc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/1967553519 shutdown_connections 2023-03-03T15:47:29.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.131+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/1967553519 wait complete. 2023-03-03T15:47:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.132+0000 7ff0c2d2d700 1 Processor -- start 2023-03-03T15:47:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.132+0000 7ff0c2d2d700 1 -- start start 2023-03-03T15:47:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.133+0000 7ff0c2d2d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc071990 0x7ff0bc0aca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.133+0000 7ff0c2d2d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 0x7ff0bc148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.133+0000 7ff0c2d2d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0bc0ba050 0x7ff0bc1ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.133+0000 7ff0c2d2d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc0aa3e0 con 0x7ff0bc151c00 2023-03-03T15:47:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.133+0000 7ff0c2d2d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc1ea470 con 0x7ff0bc0ba050 2023-03-03T15:47:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.133+0000 7ff0c2d2d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc0ba4e0 con 0x7ff0bc071990 2023-03-03T15:47:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.133+0000 7ff0bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 0x7ff0bc148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.133+0000 7ff0bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 0x7ff0bc148d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54456/0 (socket says 172.21.15.7:54456) 2023-03-03T15:47:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.133+0000 7ff0bbfff700 1 -- 172.21.15.7:0/2895469148 learned_addr learned my addr 172.21.15.7:0/2895469148 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:47:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0c0ac9700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc071990 0x7ff0bc0aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0c12ca700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0bc0ba050 0x7ff0bc1ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0bbfff700 1 -- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc071990 msgr2=0x7ff0bc0aca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0bbfff700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc071990 0x7ff0bc0aca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0bbfff700 1 -- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0bc0ba050 msgr2=0x7ff0bc1ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0bbfff700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0bc0ba050 0x7ff0bc1ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0bbfff700 1 -- 172.21.15.7:0/2895469148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b4009b30 con 0x7ff0bc151c00 2023-03-03T15:47:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0c0ac9700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc071990 0x7ff0bc0aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:47:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0bbfff700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 0x7ff0bc148d70 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800e810 tx=0x7ff0a800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0b97fa700 1 -- 172.21.15.7:0/2895469148 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8013610 con 0x7ff0bc151c00 2023-03-03T15:47:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0b97fa700 1 -- 172.21.15.7:0/2895469148 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a8013770 con 0x7ff0bc151c00 2023-03-03T15:47:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/2895469148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0bc0c3b50 con 0x7ff0bc151c00 2023-03-03T15:47:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0b97fa700 1 -- 172.21.15.7:0/2895469148 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8013b20 con 0x7ff0bc151c00 2023-03-03T15:47:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.134+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/2895469148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0bc06a2b0 con 0x7ff0bc151c00 2023-03-03T15:47:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.135+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/2895469148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0bc06afb0 con 0x7ff0bc151c00 2023-03-03T15:47:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.136+0000 7ff0b97fa700 1 -- 172.21.15.7:0/2895469148 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff0a8023070 con 0x7ff0bc151c00 2023-03-03T15:47:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.136+0000 7ff0b97fa700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0a00682f0 0x7ff0a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.136+0000 7ff0b97fa700 1 -- 172.21.15.7:0/2895469148 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff0a8085dd0 con 0x7ff0bc151c00 2023-03-03T15:47:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.136+0000 7ff0c0ac9700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0a00682f0 0x7ff0a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.137+0000 7ff0c0ac9700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0a00682f0 0x7ff0a006a7a0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00099f0 tx=0x7ff0b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.140+0000 7ff0b97fa700 1 -- 172.21.15.7:0/2895469148 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff0a80520f0 con 0x7ff0bc151c00 2023-03-03T15:47:29.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.379+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/2895469148 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0bc1ea750 con 0x7ff0bc151c00 2023-03-03T15:47:29.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.379+0000 7ff0b97fa700 1 -- 172.21.15.7:0/2895469148 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff0a8055710 con 0x7ff0bc151c00 2023-03-03T15:47:29.381 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:47:29.381 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:47:29.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.381+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0a00682f0 msgr2=0x7ff0a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:29.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.381+0000 7ff0c2d2d700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0a00682f0 0x7ff0a006a7a0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00099f0 tx=0x7ff0b0008040 comp rx=0 tx=0).stop 2023-03-03T15:47:29.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.381+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/2895469148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 msgr2=0x7ff0bc148d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:29.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.381+0000 7ff0c2d2d700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 0x7ff0bc148d70 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800e810 tx=0x7ff0a800ebd0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.382+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/2895469148 shutdown_connections 2023-03-03T15:47:29.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.382+0000 7ff0c2d2d700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0bc0ba050 0x7ff0bc1ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.382+0000 7ff0c2d2d700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0bc071990 0x7ff0bc0aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.382+0000 7ff0c2d2d700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0a00682f0 0x7ff0a006a7a0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.382+0000 7ff0c2d2d700 1 --2- 172.21.15.7:0/2895469148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0bc151c00 0x7ff0bc148d70 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:29.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.382+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/2895469148 >> 172.21.15.7:0/2895469148 conn(0x7ff0bc0a94d0 msgr2=0x7ff0bc0c5630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:29.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.382+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/2895469148 shutdown_connections 2023-03-03T15:47:29.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:29.382+0000 7ff0c2d2d700 1 -- 172.21.15.7:0/2895469148 wait complete. 2023-03-03T15:47:29.385 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:47:29.395 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2023-03-03T15:47:38.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:38.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:47:56.396 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:47:56.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.533+0000 7f10a72ad700 1 Processor -- start 2023-03-03T15:47:56.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.534+0000 7f10a72ad700 1 -- start start 2023-03-03T15:47:56.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.534+0000 7f10a72ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00aac00 0x7f10a00c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:56.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.534+0000 7f10a72ad700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f10a0071c30 0x7f10a00beb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:56.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.534+0000 7f10a72ad700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a00713c0 con 0x7f10a00bf0a0 2023-03-03T15:47:56.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.534+0000 7f10a72ad700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a0152c70 con 0x7f10a0071c30 2023-03-03T15:47:56.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.534+0000 7f10a72ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00c4930 con 0x7f10a00aac00 2023-03-03T15:47:56.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f10a584a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f10a00bf0a0 0x7f10a0155950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48468/0 (socket says 172.21.15.7:48468) 2023-03-03T15:47:56.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f10a584a700 1 -- 172.21.15.7:0/2301512697 learned_addr learned my addr 172.21.15.7:0/2301512697 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:47:56.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f10a4848700 1 --2- 172.21.15.7:0/2301512697 >> v2:172.21.15.143:3300/0 conn(0x7f10a0071c30 0x7f10a00beb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:56.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f10a5049700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00aac00 0x7f10a00c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:56.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686856712 0 0) 0x7f10a00713c0 con 0x7f10a00bf0a0 2023-03-03T15:47:56.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f108c0035b0 con 0x7f10a00bf0a0 2023-03-03T15:47:56.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216285130 0 0) 0x7f108c0035b0 con 0x7f10a00bf0a0 2023-03-03T15:47:56.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00aac00 msgr2=0x7f10a00c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:56.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00aac00 0x7f10a00c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 >> v2:172.21.15.143:3300/0 conn(0x7f10a0071c30 msgr2=0x7f10a00beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:56.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 --2- 172.21.15.7:0/2301512697 >> v2:172.21.15.143:3300/0 conn(0x7f10a0071c30 0x7f10a00beb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a0152360 con 0x7f10a00bf0a0 2023-03-03T15:47:56.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f10980019f0 con 0x7f10a00bf0a0 2023-03-03T15:47:56.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 >> v1:172.21.15.7:6789/0 conn(0x7f10a00bf0a0 legacy=0x7f10a0155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00aac00 0x7f108c003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c003ff0 0x7f108c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c004d90 0x7f108c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a0152c70 con 0x7f108c004d90 2023-03-03T15:47:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00c4930 con 0x7f108c003ff0 2023-03-03T15:47:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108c00bac0 con 0x7f10a00aac00 2023-03-03T15:47:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.535+0000 7f10a5049700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00aac00 0x7f108c003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:56.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f10a584a700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c004d90 0x7f108c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f10a4848700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c003ff0 0x7f108c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f10a4848700 1 -- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00aac00 msgr2=0x7f108c003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f10a4848700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00aac00 0x7f108c003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f10a4848700 1 -- 172.21.15.7:0/2301512697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c004d90 msgr2=0x7f108c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:47:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f10a4848700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c004d90 0x7f108c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f10a4848700 1 -- 172.21.15.7:0/2301512697 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c0035b0 con 0x7f108c003ff0 2023-03-03T15:47:56.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f10a584a700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c004d90 0x7f108c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:47:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f10a4848700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c003ff0 0x7f108c004850 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f109c009b30 tx=0x7f108c005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3945253cf00bb8ad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c00e6c0 con 0x7f108c003ff0 2023-03-03T15:47:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109c00e820 con 0x7f108c003ff0 2023-03-03T15:47:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.536+0000 7f1097fff700 1 -- 172.21.15.7:0/2301512697 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c00eb90 con 0x7f108c003ff0 2023-03-03T15:47:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.537+0000 7f10a72ad700 1 -- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c003ff0 msgr2=0x7f108c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.537+0000 7f10a72ad700 1 --2- 172.21.15.7:0/2301512697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c003ff0 0x7f108c004850 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f109c009b30 tx=0x7f108c005880 comp rx=0 tx=0).stop 2023-03-03T15:47:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.537+0000 7f10a584a700 1 -- 172.21.15.7:0/2301512697 reap_dead start 2023-03-03T15:47:56.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.537+0000 7f10a72ad700 1 -- 172.21.15.7:0/2301512697 shutdown_connections 2023-03-03T15:47:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.537+0000 7f10a72ad700 1 -- 172.21.15.7:0/2301512697 >> 172.21.15.7:0/2301512697 conn(0x7f10a00736f0 msgr2=0x7f10a0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.537+0000 7f10a72ad700 1 -- 172.21.15.7:0/2301512697 shutdown_connections 2023-03-03T15:47:56.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.537+0000 7f10a72ad700 1 -- 172.21.15.7:0/2301512697 wait complete. 2023-03-03T15:47:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.538+0000 7f10a72ad700 1 Processor -- start 2023-03-03T15:47:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.538+0000 7f10a72ad700 1 -- start start 2023-03-03T15:47:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a72ad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c003ff0 0x7f10a00c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a72ad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10a006c040 0x7f10a01ea470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:56.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a72ad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00c0740 0x7f10a01ea9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a72ad700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a01ea050 con 0x7f108c003ff0 2023-03-03T15:47:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a72ad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a01ee260 con 0x7f10a006c040 2023-03-03T15:47:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a72ad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a01edce0 con 0x7f10a00c0740 2023-03-03T15:47:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a5049700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c003ff0 0x7f10a00c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a5049700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c003ff0 0x7f10a00c7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54496/0 (socket says 172.21.15.7:54496) 2023-03-03T15:47:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a5049700 1 -- 172.21.15.7:0/2999157763 learned_addr learned my addr 172.21.15.7:0/2999157763 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:47:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a4848700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10a006c040 0x7f10a01ea470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:56.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a584a700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00c0740 0x7f10a01ea9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a5049700 1 -- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00c0740 msgr2=0x7f10a01ea9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a5049700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00c0740 0x7f10a01ea9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a5049700 1 -- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10a006c040 msgr2=0x7f10a01ea470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a5049700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10a006c040 0x7f10a01ea470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a5049700 1 -- 172.21.15.7:0/2999157763 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a0152620 con 0x7f108c003ff0 2023-03-03T15:47:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a584a700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00c0740 0x7f10a01ea9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:47:56.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.539+0000 7f10a5049700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c003ff0 0x7f10a00c7b00 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f1090011ad0 tx=0x7f1090011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.540+0000 7f1095ffb700 1 -- 172.21.15.7:0/2999157763 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109000f940 con 0x7f108c003ff0 2023-03-03T15:47:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.540+0000 7f1095ffb700 1 -- 172.21.15.7:0/2999157763 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109000faa0 con 0x7f108c003ff0 2023-03-03T15:47:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.540+0000 7f10a72ad700 1 -- 172.21.15.7:0/2999157763 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a00c4b80 con 0x7f108c003ff0 2023-03-03T15:47:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.540+0000 7f1095ffb700 1 -- 172.21.15.7:0/2999157763 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109000fe50 con 0x7f108c003ff0 2023-03-03T15:47:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.540+0000 7f10a72ad700 1 -- 172.21.15.7:0/2999157763 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a0152900 con 0x7f108c003ff0 2023-03-03T15:47:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.541+0000 7f10a72ad700 1 -- 172.21.15.7:0/2999157763 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a0074bb0 con 0x7f108c003ff0 2023-03-03T15:47:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.541+0000 7f1095ffb700 1 -- 172.21.15.7:0/2999157763 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f109001d400 con 0x7f108c003ff0 2023-03-03T15:47:56.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.542+0000 7f1095ffb700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10840682f0 0x7f108406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:47:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.542+0000 7f1095ffb700 1 -- 172.21.15.7:0/2999157763 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f10900868b0 con 0x7f108c003ff0 2023-03-03T15:47:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.542+0000 7f10a4848700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10840682f0 0x7f108406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:47:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.543+0000 7f10a4848700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10840682f0 0x7f108406a7a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f109c0090b0 tx=0x7f109c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:47:56.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.546+0000 7f1095ffb700 1 -- 172.21.15.7:0/2999157763 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1090052cb0 con 0x7f108c003ff0 2023-03-03T15:47:56.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.783+0000 7f10a72ad700 1 -- 172.21.15.7:0/2999157763 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10a00c2730 con 0x7f108c003ff0 2023-03-03T15:47:56.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.784+0000 7f1095ffb700 1 -- 172.21.15.7:0/2999157763 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f10900562d0 con 0x7f108c003ff0 2023-03-03T15:47:56.785 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:47:56.786 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:47:56.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 -- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10840682f0 msgr2=0x7f108406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:56.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10840682f0 0x7f108406a7a0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f109c0090b0 tx=0x7f109c008030 comp rx=0 tx=0).stop 2023-03-03T15:47:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 -- 172.21.15.7:0/2999157763 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c003ff0 msgr2=0x7f10a00c7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:47:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c003ff0 0x7f10a00c7b00 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f1090011ad0 tx=0x7f1090011de0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 -- 172.21.15.7:0/2999157763 shutdown_connections 2023-03-03T15:47:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a00c0740 0x7f10a01ea9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10840682f0 0x7f108406a7a0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c003ff0 0x7f10a00c7b00 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 --2- 172.21.15.7:0/2999157763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10a006c040 0x7f10a01ea470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:47:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 -- 172.21.15.7:0/2999157763 >> 172.21.15.7:0/2999157763 conn(0x7f10a00736f0 msgr2=0x7f10a006e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:47:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 -- 172.21.15.7:0/2999157763 shutdown_connections 2023-03-03T15:47:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:47:56.786+0000 7f10a72ad700 1 -- 172.21.15.7:0/2999157763 wait complete. 2023-03-03T15:47:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:47:56.799 INFO:tasks.mds_thrash.fs.[a]:waiting for 46 secs before thrashing 2023-03-03T15:48:08.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:08.593 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:20.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:48:20.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.998+0000 7fab40ba0700 1 Processor -- start 2023-03-03T15:48:20.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.998+0000 7fab40ba0700 1 -- start start 2023-03-03T15:48:21.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab40ba0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c1544e0 0x7fab3c158010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:21.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab40ba0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab3c0c1880 0x7fab3c158550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:21.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab40ba0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3c0c2890 con 0x7fab3c1519b0 2023-03-03T15:48:21.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab40ba0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3c0c6100 con 0x7fab3c0c1880 2023-03-03T15:48:21.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab40ba0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c0c6650 con 0x7fab3c1544e0 2023-03-03T15:48:21.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab3ad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab3c0c1880 0x7fab3c158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:21.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab3ad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fab3c0c1880 0x7fab3c158550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54518/0 (socket says 172.21.15.7:54518) 2023-03-03T15:48:21.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab3ad9d700 1 -- 172.21.15.7:0/3183716233 learned_addr learned my addr 172.21.15.7:0/3183716233 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:48:21.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab39d9b700 1 --2- 172.21.15.7:0/3183716233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c1544e0 0x7fab3c158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:21.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab3ad9d700 1 -- 172.21.15.7:0/3183716233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c1544e0 msgr2=0x7fab3c158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:48:21.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab3ad9d700 1 --2- 172.21.15.7:0/3183716233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c1544e0 0x7fab3c158010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:21.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab3ad9d700 1 -- 172.21.15.7:0/3183716233 >> v1:172.21.15.143:6789/0 conn(0x7fab3c1519b0 legacy=0x7fab3c0c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:21.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab3ad9d700 1 -- 172.21.15.7:0/3183716233 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c0c5370 con 0x7fab3c0c1880 2023-03-03T15:48:21.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab3ad9d700 1 --2- 172.21.15.7:0/3183716233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c1880 0x7fab3c158550 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7fab34009e50 tx=0x7fab3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46cd3e7b7e4e916c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:48:21.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:20.999+0000 7fab3959a700 1 -- 172.21.15.7:0/3183716233 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3400e070 con 0x7fab3c0c1880 2023-03-03T15:48:21.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.000+0000 7fab3959a700 1 -- 172.21.15.7:0/3183716233 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3400b910 con 0x7fab3c0c1880 2023-03-03T15:48:21.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.000+0000 7fab3959a700 1 -- 172.21.15.7:0/3183716233 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3400bcc0 con 0x7fab3c0c1880 2023-03-03T15:48:21.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.000+0000 7fab40ba0700 1 -- 172.21.15.7:0/3183716233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c1880 msgr2=0x7fab3c158550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:21.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.000+0000 7fab40ba0700 1 --2- 172.21.15.7:0/3183716233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c1880 0x7fab3c158550 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7fab34009e50 tx=0x7fab3400b330 comp rx=0 tx=0).stop 2023-03-03T15:48:21.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.000+0000 7fab40ba0700 1 -- 172.21.15.7:0/3183716233 shutdown_connections 2023-03-03T15:48:21.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.000+0000 7fab40ba0700 1 --2- 172.21.15.7:0/3183716233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c0c1880 0x7fab3c158550 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:21.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.000+0000 7fab40ba0700 1 --2- 172.21.15.7:0/3183716233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c1544e0 0x7fab3c158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:21.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.000+0000 7fab40ba0700 1 -- 172.21.15.7:0/3183716233 >> 172.21.15.7:0/3183716233 conn(0x7fab3c06a490 msgr2=0x7fab3c06a860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:48:21.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.000+0000 7fab40ba0700 1 -- 172.21.15.7:0/3183716233 shutdown_connections 2023-03-03T15:48:21.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.000+0000 7fab40ba0700 1 -- 172.21.15.7:0/3183716233 wait complete. 2023-03-03T15:48:21.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.001+0000 7fab40ba0700 1 Processor -- start 2023-03-03T15:48:21.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab40ba0700 1 -- start start 2023-03-03T15:48:21.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab40ba0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab3c0c1880 0x7fab3c1edcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:21.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab40ba0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c1519b0 0x7fab3c155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:21.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab40ba0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c1544e0 0x7fab3c1e9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:21.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab40ba0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c1ee1f0 con 0x7fab3c1544e0 2023-03-03T15:48:21.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab40ba0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c071b50 con 0x7fab3c0c1880 2023-03-03T15:48:21.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab40ba0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c071e30 con 0x7fab3c1519b0 2023-03-03T15:48:21.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab3ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c1544e0 0x7fab3c1e9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:21.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab3ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c1544e0 0x7fab3c1e9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54524/0 (socket says 172.21.15.7:54524) 2023-03-03T15:48:21.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab3ad9d700 1 -- 172.21.15.7:0/4275857254 learned_addr learned my addr 172.21.15.7:0/4275857254 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:48:21.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab3a59c700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab3c0c1880 0x7fab3c1edcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:21.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.002+0000 7fab39d9b700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c1519b0 0x7fab3c155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:21.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab3ad9d700 1 -- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c1519b0 msgr2=0x7fab3c155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:21.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab3ad9d700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c1519b0 0x7fab3c155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:21.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab3ad9d700 1 -- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab3c0c1880 msgr2=0x7fab3c1edcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:21.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab3ad9d700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab3c0c1880 0x7fab3c1edcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:21.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab3ad9d700 1 -- 172.21.15.7:0/4275857254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab34009b30 con 0x7fab3c1544e0 2023-03-03T15:48:21.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab3ad9d700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c1544e0 0x7fab3c1e9da0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0c4390 tx=0x7fab34004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:48:21.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab2affd700 1 -- 172.21.15.7:0/4275857254 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34004840 con 0x7fab3c1544e0 2023-03-03T15:48:21.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab2affd700 1 -- 172.21.15.7:0/4275857254 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab340099b0 con 0x7fab3c1544e0 2023-03-03T15:48:21.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab40ba0700 1 -- 172.21.15.7:0/4275857254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3c157af0 con 0x7fab3c1544e0 2023-03-03T15:48:21.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab2affd700 1 -- 172.21.15.7:0/4275857254 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3401c680 con 0x7fab3c1544e0 2023-03-03T15:48:21.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.003+0000 7fab40ba0700 1 -- 172.21.15.7:0/4275857254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c0c6370 con 0x7fab3c1544e0 2023-03-03T15:48:21.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.004+0000 7fab40ba0700 1 -- 172.21.15.7:0/4275857254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3c0c5650 con 0x7fab3c1544e0 2023-03-03T15:48:21.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.005+0000 7fab2affd700 1 -- 172.21.15.7:0/4275857254 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fab3402c070 con 0x7fab3c1544e0 2023-03-03T15:48:21.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.005+0000 7fab2affd700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab1c0682f0 0x7fab1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:21.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.005+0000 7fab2affd700 1 -- 172.21.15.7:0/4275857254 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fab34087b80 con 0x7fab3c1544e0 2023-03-03T15:48:21.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.005+0000 7fab3a59c700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab1c0682f0 0x7fab1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:21.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.006+0000 7fab3a59c700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab1c0682f0 0x7fab1c06a7a0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fab2c007fd0 tx=0x7fab2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:48:21.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.010+0000 7fab2affd700 1 -- 172.21.15.7:0/4275857254 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fab34053f20 con 0x7fab3c1544e0 2023-03-03T15:48:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.248+0000 7fab40ba0700 1 -- 172.21.15.7:0/4275857254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab3c1ed310 con 0x7fab3c1544e0 2023-03-03T15:48:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.249+0000 7fab2affd700 1 -- 172.21.15.7:0/4275857254 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fab3c1ed310 con 0x7fab3c1544e0 2023-03-03T15:48:21.250 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:48:21.250 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:48:21.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 -- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab1c0682f0 msgr2=0x7fab1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:21.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab1c0682f0 0x7fab1c06a7a0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fab2c007fd0 tx=0x7fab2c009040 comp rx=0 tx=0).stop 2023-03-03T15:48:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 -- 172.21.15.7:0/4275857254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c1544e0 msgr2=0x7fab3c1e9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c1544e0 0x7fab3c1e9da0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0c4390 tx=0x7fab34004ad0 comp rx=0 tx=0).stop 2023-03-03T15:48:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 -- 172.21.15.7:0/4275857254 shutdown_connections 2023-03-03T15:48:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab1c0682f0 0x7fab1c06a7a0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab3c1544e0 0x7fab3c1e9da0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab3c0c1880 0x7fab3c1edcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 --2- 172.21.15.7:0/4275857254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fab3c1519b0 0x7fab3c155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:21.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 -- 172.21.15.7:0/4275857254 >> 172.21.15.7:0/4275857254 conn(0x7fab3c06a490 msgr2=0x7fab3c1572d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:48:21.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 -- 172.21.15.7:0/4275857254 shutdown_connections 2023-03-03T15:48:21.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:21.251+0000 7fab40ba0700 1 -- 172.21.15.7:0/4275857254 wait complete. 2023-03-03T15:48:21.254 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:48:21.264 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2023-03-03T15:48:38.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:38.670 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:48:42.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:48:42.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.937+0000 7ff5ba350700 1 Processor -- start 2023-03-03T15:48:42.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.937+0000 7ff5ba350700 1 -- start start 2023-03-03T15:48:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5ba350700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5b4151e80 0x7ff5b40c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5ba350700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4155410 0x7ff5b40bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5ba350700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff5b40c6fe0 0x7ff5b40bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5ba350700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b4152250 con 0x7ff5b4151e80 2023-03-03T15:48:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5ba350700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b40c0fb0 con 0x7ff5b40c6fe0 2023-03-03T15:48:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5ba350700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b4154740 con 0x7ff5b4155410 2023-03-03T15:48:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5b4151e80 0x7ff5b40c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5b4151e80 0x7ff5b40c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54550/0 (socket says 172.21.15.7:54550) 2023-03-03T15:48:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b3fff700 1 -- 172.21.15.7:0/1351420514 learned_addr learned my addr 172.21.15.7:0/1351420514 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:48:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b37fe700 1 --2- 172.21.15.7:0/1351420514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4155410 0x7ff5b40bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b88ed700 1 --2- 172.21.15.7:0/1351420514 >> v2:172.21.15.143:3300/0 conn(0x7ff5b40c6fe0 0x7ff5b40bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b3fff700 1 -- 172.21.15.7:0/1351420514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4155410 msgr2=0x7ff5b40bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:48:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b3fff700 1 --2- 172.21.15.7:0/1351420514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4155410 0x7ff5b40bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b3fff700 1 -- 172.21.15.7:0/1351420514 >> v2:172.21.15.143:3300/0 conn(0x7ff5b40c6fe0 msgr2=0x7ff5b40bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:48:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b3fff700 1 --2- 172.21.15.7:0/1351420514 >> v2:172.21.15.143:3300/0 conn(0x7ff5b40c6fe0 0x7ff5b40bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b3fff700 1 -- 172.21.15.7:0/1351420514 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b40c08d0 con 0x7ff5b4151e80 2023-03-03T15:48:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b88ed700 1 --2- 172.21.15.7:0/1351420514 >> v2:172.21.15.143:3300/0 conn(0x7ff5b40c6fe0 0x7ff5b40bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:48:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.938+0000 7ff5b37fe700 1 --2- 172.21.15.7:0/1351420514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4155410 0x7ff5b40bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:48:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5b3fff700 1 --2- 172.21.15.7:0/1351420514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4151e80 0x7ff5b40c6aa0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7ff5b408c770 tx=0x7ff5a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b76c333d638b8ea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:48:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5b2ffd700 1 -- 172.21.15.7:0/1351420514 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a400b870 con 0x7ff5b4151e80 2023-03-03T15:48:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5b2ffd700 1 -- 172.21.15.7:0/1351420514 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a400b9d0 con 0x7ff5b4151e80 2023-03-03T15:48:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5b2ffd700 1 -- 172.21.15.7:0/1351420514 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a400bd60 con 0x7ff5b4151e80 2023-03-03T15:48:42.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5ba350700 1 -- 172.21.15.7:0/1351420514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4151e80 msgr2=0x7ff5b40c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:42.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5ba350700 1 --2- 172.21.15.7:0/1351420514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4151e80 0x7ff5b40c6aa0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7ff5b408c770 tx=0x7ff5a4009bd0 comp rx=0 tx=0).stop 2023-03-03T15:48:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5ba350700 1 -- 172.21.15.7:0/1351420514 shutdown_connections 2023-03-03T15:48:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5ba350700 1 --2- 172.21.15.7:0/1351420514 >> v2:172.21.15.143:3300/0 conn(0x7ff5b40c6fe0 0x7ff5b40bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5ba350700 1 --2- 172.21.15.7:0/1351420514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4151e80 0x7ff5b40c6aa0 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5ba350700 1 --2- 172.21.15.7:0/1351420514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4155410 0x7ff5b40bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5ba350700 1 -- 172.21.15.7:0/1351420514 >> 172.21.15.7:0/1351420514 conn(0x7ff5b4071630 msgr2=0x7ff5b406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:48:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5ba350700 1 -- 172.21.15.7:0/1351420514 shutdown_connections 2023-03-03T15:48:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.939+0000 7ff5ba350700 1 -- 172.21.15.7:0/1351420514 wait complete. 2023-03-03T15:48:42.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.940+0000 7ff5ba350700 1 Processor -- start 2023-03-03T15:48:42.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.941+0000 7ff5ba350700 1 -- start start 2023-03-03T15:48:42.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.941+0000 7ff5ba350700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5b40c6fe0 0x7ff5b41e0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:42.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.941+0000 7ff5ba350700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4151e80 0x7ff5b41e07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:42.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.941+0000 7ff5ba350700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4155410 0x7ff5b4146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:42.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.941+0000 7ff5ba350700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b41df8f0 con 0x7ff5b4155410 2023-03-03T15:48:42.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.941+0000 7ff5ba350700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b41deb10 con 0x7ff5b40c6fe0 2023-03-03T15:48:42.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.941+0000 7ff5ba350700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b4150ca0 con 0x7ff5b4151e80 2023-03-03T15:48:42.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b88ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4155410 0x7ff5b4146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b88ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4155410 0x7ff5b4146360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54560/0 (socket says 172.21.15.7:54560) 2023-03-03T15:48:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b88ed700 1 -- 172.21.15.7:0/314267247 learned_addr learned my addr 172.21.15.7:0/314267247 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:48:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b3fff700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5b40c6fe0 0x7ff5b41e0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b37fe700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4151e80 0x7ff5b41e07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b88ed700 1 -- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4151e80 msgr2=0x7ff5b41e07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b88ed700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4151e80 0x7ff5b41e07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:42.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b88ed700 1 -- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5b40c6fe0 msgr2=0x7ff5b41e0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:42.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b88ed700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5b40c6fe0 0x7ff5b41e0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:42.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b88ed700 1 -- 172.21.15.7:0/314267247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a4009550 con 0x7ff5b4155410 2023-03-03T15:48:42.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b88ed700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4155410 0x7ff5b4146360 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00cd50 tx=0x7ff5ac00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:48:42.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b0ff9700 1 -- 172.21.15.7:0/314267247 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac00cea0 con 0x7ff5b4155410 2023-03-03T15:48:42.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b0ff9700 1 -- 172.21.15.7:0/314267247 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ac00f480 con 0x7ff5b4155410 2023-03-03T15:48:42.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5ba350700 1 -- 172.21.15.7:0/314267247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b4154c10 con 0x7ff5b4155410 2023-03-03T15:48:42.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5b0ff9700 1 -- 172.21.15.7:0/314267247 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac00f830 con 0x7ff5b4155410 2023-03-03T15:48:42.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.942+0000 7ff5ba350700 1 -- 172.21.15.7:0/314267247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b4154930 con 0x7ff5b4155410 2023-03-03T15:48:42.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.943+0000 7ff5ba350700 1 -- 172.21.15.7:0/314267247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b4154ef0 con 0x7ff5b4155410 2023-03-03T15:48:42.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.944+0000 7ff5b0ff9700 1 -- 172.21.15.7:0/314267247 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff5ac009030 con 0x7ff5b4155410 2023-03-03T15:48:42.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.944+0000 7ff5b0ff9700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5940682f0 0x7ff59406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:42.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.944+0000 7ff5b0ff9700 1 -- 172.21.15.7:0/314267247 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff5ac086a50 con 0x7ff5b4155410 2023-03-03T15:48:42.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.944+0000 7ff5b3fff700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5940682f0 0x7ff59406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:42.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.945+0000 7ff5b3fff700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5940682f0 0x7ff59406a7a0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4014040 tx=0x7ff5a4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:48:42.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:42.948+0000 7ff5b0ff9700 1 -- 172.21.15.7:0/314267247 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5ac052d10 con 0x7ff5b4155410 2023-03-03T15:48:43.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.181+0000 7ff5ba350700 1 -- 172.21.15.7:0/314267247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5b4074bb0 con 0x7ff5b4155410 2023-03-03T15:48:43.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.182+0000 7ff5b0ff9700 1 -- 172.21.15.7:0/314267247 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff5ac056330 con 0x7ff5b4155410 2023-03-03T15:48:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:48:43.185 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:48:43.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 -- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5940682f0 msgr2=0x7ff59406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:43.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5940682f0 0x7ff59406a7a0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4014040 tx=0x7ff5a4008030 comp rx=0 tx=0).stop 2023-03-03T15:48:43.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 -- 172.21.15.7:0/314267247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4155410 msgr2=0x7ff5b4146360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:43.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4155410 0x7ff5b4146360 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00cd50 tx=0x7ff5ac00a560 comp rx=0 tx=0).stop 2023-03-03T15:48:43.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 -- 172.21.15.7:0/314267247 shutdown_connections 2023-03-03T15:48:43.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5940682f0 0x7ff59406a7a0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:43.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5b4155410 0x7ff5b4146360 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:43.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5b40c6fe0 0x7ff5b41e0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:43.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 --2- 172.21.15.7:0/314267247 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5b4151e80 0x7ff5b41e07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:43.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 -- 172.21.15.7:0/314267247 >> 172.21.15.7:0/314267247 conn(0x7ff5b4071630 msgr2=0x7ff5b40c2210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:48:43.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 -- 172.21.15.7:0/314267247 shutdown_connections 2023-03-03T15:48:43.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:43.184+0000 7ff5ba350700 1 -- 172.21.15.7:0/314267247 wait complete. 2023-03-03T15:48:43.188 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:48:43.198 INFO:tasks.mds_thrash.fs.[a]:waiting for 54 secs before thrashing 2023-03-03T15:48:48.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:48:48.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.403+0000 7f7031be7700 1 Processor -- start 2023-03-03T15:48:48.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.404+0000 7f7031be7700 1 -- start start 2023-03-03T15:48:48.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.404+0000 7f7031be7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aad20 0x7f702c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:48.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.404+0000 7f7031be7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f702c0aa030 0x7f702c158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:48.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.404+0000 7f7031be7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702c0a9c20 con 0x7f702c14b0d0 2023-03-03T15:48:48.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.404+0000 7f7031be7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702c154f10 con 0x7f702c0aa030 2023-03-03T15:48:48.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.404+0000 7f7031be7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702c0ac3c0 con 0x7f702c0aad20 2023-03-03T15:48:48.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.404+0000 7f702bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f702c0aa030 0x7f702c158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:48.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f702c0aa030 0x7f702c158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54568/0 (socket says 172.21.15.7:54568) 2023-03-03T15:48:48.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702bfff700 1 -- 172.21.15.7:0/3438663040 learned_addr learned my addr 172.21.15.7:0/3438663040 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:48:48.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702affd700 1 --2- 172.21.15.7:0/3438663040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aad20 0x7f702c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:48.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702bfff700 1 -- 172.21.15.7:0/3438663040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aad20 msgr2=0x7f702c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:48:48.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702bfff700 1 --2- 172.21.15.7:0/3438663040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aad20 0x7f702c157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:48.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702bfff700 1 -- 172.21.15.7:0/3438663040 >> v1:172.21.15.143:6789/0 conn(0x7f702c14b0d0 legacy=0x7f702c071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:48:48.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702bfff700 1 -- 172.21.15.7:0/3438663040 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c06fc70 con 0x7f702c0aa030 2023-03-03T15:48:48.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702bfff700 1 --2- 172.21.15.7:0/3438663040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aa030 0x7f702c158330 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f7024009e50 tx=0x7f702400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89474c862ac70917 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:48:48.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702a7fc700 1 -- 172.21.15.7:0/3438663040 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702400e070 con 0x7f702c0aa030 2023-03-03T15:48:48.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702a7fc700 1 -- 172.21.15.7:0/3438663040 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7024003a20 con 0x7f702c0aa030 2023-03-03T15:48:48.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.405+0000 7f702a7fc700 1 -- 172.21.15.7:0/3438663040 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7024003dd0 con 0x7f702c0aa030 2023-03-03T15:48:48.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.406+0000 7f7031be7700 1 -- 172.21.15.7:0/3438663040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aa030 msgr2=0x7f702c158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:48.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.406+0000 7f7031be7700 1 --2- 172.21.15.7:0/3438663040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aa030 0x7f702c158330 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7f7024009e50 tx=0x7f702400b330 comp rx=0 tx=0).stop 2023-03-03T15:48:48.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.406+0000 7f7031be7700 1 -- 172.21.15.7:0/3438663040 shutdown_connections 2023-03-03T15:48:48.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.406+0000 7f7031be7700 1 --2- 172.21.15.7:0/3438663040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aa030 0x7f702c158330 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:48.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.406+0000 7f7031be7700 1 --2- 172.21.15.7:0/3438663040 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aad20 0x7f702c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:48.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.406+0000 7f7031be7700 1 -- 172.21.15.7:0/3438663040 >> 172.21.15.7:0/3438663040 conn(0x7f702c0c8900 msgr2=0x7f702c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:48:48.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.406+0000 7f7031be7700 1 -- 172.21.15.7:0/3438663040 shutdown_connections 2023-03-03T15:48:48.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.406+0000 7f7031be7700 1 -- 172.21.15.7:0/3438663040 wait complete. 2023-03-03T15:48:48.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.407+0000 7f7031be7700 1 Processor -- start 2023-03-03T15:48:48.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.407+0000 7f7031be7700 1 -- start start 2023-03-03T15:48:48.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f7031be7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aa030 0x7f702c1435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:48.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f7031be7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aad20 0x7f702c156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:48.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f7031be7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f702c14b0d0 0x7f702c142900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:48.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f7031be7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c144850 con 0x7f702c0aad20 2023-03-03T15:48:48.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f7031be7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c0c7d60 con 0x7f702c14b0d0 2023-03-03T15:48:48.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f7031be7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702c0c6ae0 con 0x7f702c0aa030 2023-03-03T15:48:48.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aad20 0x7f702c156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:48.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aad20 0x7f702c156de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54572/0 (socket says 172.21.15.7:54572) 2023-03-03T15:48:48.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702affd700 1 -- 172.21.15.7:0/2083306488 learned_addr learned my addr 172.21.15.7:0/2083306488 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:48:48.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702bfff700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f702c14b0d0 0x7f702c142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:48.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702b7fe700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aa030 0x7f702c1435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:48.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702affd700 1 -- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aa030 msgr2=0x7f702c1435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:48.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702affd700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aa030 0x7f702c1435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:48.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702affd700 1 -- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f702c14b0d0 msgr2=0x7f702c142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:48.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702affd700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f702c14b0d0 0x7f702c142900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:48.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702affd700 1 -- 172.21.15.7:0/2083306488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7024009b30 con 0x7f702c0aad20 2023-03-03T15:48:48.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702b7fe700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aa030 0x7f702c1435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:48:48.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.408+0000 7f702bfff700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f702c14b0d0 0x7f702c142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:48:48.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.409+0000 7f702affd700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aad20 0x7f702c156de0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f702000d800 tx=0x7f702000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:48:48.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.409+0000 7f7013fff700 1 -- 172.21.15.7:0/2083306488 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70200125f0 con 0x7f702c0aad20 2023-03-03T15:48:48.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.409+0000 7f7013fff700 1 -- 172.21.15.7:0/2083306488 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7020012750 con 0x7f702c0aad20 2023-03-03T15:48:48.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.409+0000 7f7031be7700 1 -- 172.21.15.7:0/2083306488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702c06f6b0 con 0x7f702c0aad20 2023-03-03T15:48:48.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.409+0000 7f7013fff700 1 -- 172.21.15.7:0/2083306488 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020012b00 con 0x7f702c0aad20 2023-03-03T15:48:48.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.409+0000 7f7031be7700 1 -- 172.21.15.7:0/2083306488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702c155160 con 0x7f702c0aad20 2023-03-03T15:48:48.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.410+0000 7f7031be7700 1 -- 172.21.15.7:0/2083306488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702c06f990 con 0x7f702c0aad20 2023-03-03T15:48:48.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.410+0000 7f7013fff700 1 -- 172.21.15.7:0/2083306488 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7020012c60 con 0x7f702c0aad20 2023-03-03T15:48:48.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.411+0000 7f7013fff700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70180682f0 0x7f701806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:48:48.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.411+0000 7f7013fff700 1 -- 172.21.15.7:0/2083306488 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7020086670 con 0x7f702c0aad20 2023-03-03T15:48:48.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.411+0000 7f702b7fe700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70180682f0 0x7f701806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:48:48.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.412+0000 7f702b7fe700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70180682f0 0x7f701806a7a0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f701c003fc0 tx=0x7f701c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:48:48.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.415+0000 7f7013fff700 1 -- 172.21.15.7:0/2083306488 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7020052a10 con 0x7f702c0aad20 2023-03-03T15:48:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.654+0000 7f7031be7700 1 -- 172.21.15.7:0/2083306488 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f702c074bb0 con 0x7f702c0aad20 2023-03-03T15:48:48.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.655+0000 7f7013fff700 1 -- 172.21.15.7:0/2083306488 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7020056030 con 0x7f702c0aad20 2023-03-03T15:48:48.657 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:48:48.658 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:48:48.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 -- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70180682f0 msgr2=0x7f701806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:48.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70180682f0 0x7f701806a7a0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f701c003fc0 tx=0x7f701c009040 comp rx=0 tx=0).stop 2023-03-03T15:48:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 -- 172.21.15.7:0/2083306488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aad20 msgr2=0x7f702c156de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:48:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aad20 0x7f702c156de0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f702000d800 tx=0x7f702000dbc0 comp rx=0 tx=0).stop 2023-03-03T15:48:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 -- 172.21.15.7:0/2083306488 shutdown_connections 2023-03-03T15:48:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f702c14b0d0 0x7f702c142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f702c0aa030 0x7f702c1435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70180682f0 0x7f701806a7a0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 --2- 172.21.15.7:0/2083306488 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c0aad20 0x7f702c156de0 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:48:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 -- 172.21.15.7:0/2083306488 >> 172.21.15.7:0/2083306488 conn(0x7f702c0c8900 msgr2=0x7f702c149670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:48:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 -- 172.21.15.7:0/2083306488 shutdown_connections 2023-03-03T15:48:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:48:48.657+0000 7f7031be7700 1 -- 172.21.15.7:0/2083306488 wait complete. 2023-03-03T15:48:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:48:48.671 INFO:tasks.mds_thrash.fs.[b]:waiting for 84 secs before thrashing 2023-03-03T15:49:08.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:08.733 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:37.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:49:37.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.336+0000 7fb19415a700 1 Processor -- start 2023-03-03T15:49:37.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.336+0000 7fb19415a700 1 -- start start 2023-03-03T15:49:37.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.337+0000 7fb19415a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb18c14fda0 0x7fb18c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:37.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.337+0000 7fb19415a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c0abe20 0x7fb18c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:37.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.337+0000 7fb19415a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18c0c2f30 con 0x7fb18c14f9d0 2023-03-03T15:49:37.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.337+0000 7fb19415a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb18c0c31d0 con 0x7fb18c14fda0 2023-03-03T15:49:37.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.337+0000 7fb19415a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c1524a0 con 0x7fb18c0abe20 2023-03-03T15:49:37.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb1926f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c0abe20 0x7fb18c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:37.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb1916f5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb18c14fda0 0x7fb18c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:37.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb191ef6700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb18c14f9d0 0x7fb18c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:52114/0 (socket says 172.21.15.7:52114) 2023-03-03T15:49:37.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb1926f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c0abe20 0x7fb18c1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:34458/0 (socket says 172.21.15.7:34458) 2023-03-03T15:49:37.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb191ef6700 1 -- 172.21.15.7:0/2568364535 learned_addr learned my addr 172.21.15.7:0/2568364535 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:49:37.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb1926f7700 1 -- 172.21.15.7:0/2568364535 >> v2:172.21.15.143:3300/0 conn(0x7fb18c14fda0 msgr2=0x7fb18c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:49:37.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb1926f7700 1 --2- 172.21.15.7:0/2568364535 >> v2:172.21.15.143:3300/0 conn(0x7fb18c14fda0 0x7fb18c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:37.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb1926f7700 1 -- 172.21.15.7:0/2568364535 >> v1:172.21.15.143:6789/0 conn(0x7fb18c14f9d0 legacy=0x7fb18c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:37.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb1926f7700 1 -- 172.21.15.7:0/2568364535 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c0c34b0 con 0x7fb18c0abe20 2023-03-03T15:49:37.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb1916f5700 1 --2- 172.21.15.7:0/2568364535 >> v2:172.21.15.143:3300/0 conn(0x7fb18c14fda0 0x7fb18c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:49:37.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb1926f7700 1 --2- 172.21.15.7:0/2568364535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c0abe20 0x7fb18c1529c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb184009d90 tx=0x7fb18400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab9d94e63ec179b5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:49:37.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb190ef4700 1 -- 172.21.15.7:0/2568364535 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18400e070 con 0x7fb18c0abe20 2023-03-03T15:49:37.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb190ef4700 1 -- 172.21.15.7:0/2568364535 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18400bb20 con 0x7fb18c0abe20 2023-03-03T15:49:37.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.338+0000 7fb190ef4700 1 -- 172.21.15.7:0/2568364535 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb184012430 con 0x7fb18c0abe20 2023-03-03T15:49:37.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.339+0000 7fb19415a700 1 -- 172.21.15.7:0/2568364535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c0abe20 msgr2=0x7fb18c1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:37.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.339+0000 7fb19415a700 1 --2- 172.21.15.7:0/2568364535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c0abe20 0x7fb18c1529c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb184009d90 tx=0x7fb18400b330 comp rx=0 tx=0).stop 2023-03-03T15:49:37.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.339+0000 7fb19415a700 1 -- 172.21.15.7:0/2568364535 shutdown_connections 2023-03-03T15:49:37.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.339+0000 7fb19415a700 1 --2- 172.21.15.7:0/2568364535 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c0abe20 0x7fb18c1529c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:37.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.339+0000 7fb19415a700 1 --2- 172.21.15.7:0/2568364535 >> v2:172.21.15.143:3300/0 conn(0x7fb18c14fda0 0x7fb18c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:37.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.339+0000 7fb19415a700 1 -- 172.21.15.7:0/2568364535 >> 172.21.15.7:0/2568364535 conn(0x7fb18c0c8900 msgr2=0x7fb18c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:49:37.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.339+0000 7fb19415a700 1 -- 172.21.15.7:0/2568364535 shutdown_connections 2023-03-03T15:49:37.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.339+0000 7fb19415a700 1 -- 172.21.15.7:0/2568364535 wait complete. 2023-03-03T15:49:37.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.340+0000 7fb19415a700 1 Processor -- start 2023-03-03T15:49:37.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.340+0000 7fb19415a700 1 -- start start 2023-03-03T15:49:37.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb19415a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c1de9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:37.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb19415a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c14f9d0 0x7fb18c152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:37.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb19415a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 0x7fb18c1db700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:37.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb19415a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c151950 con 0x7fb18c0abe20 2023-03-03T15:49:37.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb19415a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c06c660 con 0x7fb18c14fda0 2023-03-03T15:49:37.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb19415a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c06c940 con 0x7fb18c14f9d0 2023-03-03T15:49:37.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb191ef6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c1de9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:37.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb191ef6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c1de9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54620/0 (socket says 172.21.15.7:54620) 2023-03-03T15:49:37.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb191ef6700 1 -- 172.21.15.7:0/1393484669 learned_addr learned my addr 172.21.15.7:0/1393484669 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:49:37.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb1916f5700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c14f9d0 0x7fb18c152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:37.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb1926f7700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 0x7fb18c1db700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:37.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb191ef6700 1 -- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c14f9d0 msgr2=0x7fb18c152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:37.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb191ef6700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c14f9d0 0x7fb18c152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:37.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb191ef6700 1 -- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 msgr2=0x7fb18c1db700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:37.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb191ef6700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 0x7fb18c1db700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:37.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.341+0000 7fb191ef6700 1 -- 172.21.15.7:0/1393484669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1800094a0 con 0x7fb18c0abe20 2023-03-03T15:49:37.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.342+0000 7fb191ef6700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c1de9b0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7fb18800c520 tx=0x7fb18800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:49:37.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.342+0000 7fb17e7fc700 1 -- 172.21.15.7:0/1393484669 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188005d70 con 0x7fb18c0abe20 2023-03-03T15:49:37.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.342+0000 7fb17e7fc700 1 -- 172.21.15.7:0/1393484669 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1880070d0 con 0x7fb18c0abe20 2023-03-03T15:49:37.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.342+0000 7fb19415a700 1 -- 172.21.15.7:0/1393484669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb184009a70 con 0x7fb18c0abe20 2023-03-03T15:49:37.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.342+0000 7fb17e7fc700 1 -- 172.21.15.7:0/1393484669 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188007440 con 0x7fb18c0abe20 2023-03-03T15:49:37.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.342+0000 7fb19415a700 1 -- 172.21.15.7:0/1393484669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18c0ac350 con 0x7fb18c0abe20 2023-03-03T15:49:37.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.343+0000 7fb19415a700 1 -- 172.21.15.7:0/1393484669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18c0ab630 con 0x7fb18c0abe20 2023-03-03T15:49:37.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.343+0000 7fb17e7fc700 1 -- 172.21.15.7:0/1393484669 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb188019030 con 0x7fb18c0abe20 2023-03-03T15:49:37.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.344+0000 7fb17e7fc700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1700682f0 0x7fb17006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:37.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.344+0000 7fb17e7fc700 1 -- 172.21.15.7:0/1393484669 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb1880852c0 con 0x7fb18c0abe20 2023-03-03T15:49:37.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.344+0000 7fb1916f5700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1700682f0 0x7fb17006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:37.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.344+0000 7fb1916f5700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1700682f0 0x7fb17006a7a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fb18c14c7e0 tx=0x7fb180009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:49:37.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.348+0000 7fb17e7fc700 1 -- 172.21.15.7:0/1393484669 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb18804b5e0 con 0x7fb18c0abe20 2023-03-03T15:49:37.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.587+0000 7fb19415a700 1 -- 172.21.15.7:0/1393484669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb18c071fa0 con 0x7fb18c0abe20 2023-03-03T15:49:37.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.587+0000 7fb17e7fc700 1 -- 172.21.15.7:0/1393484669 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb188051710 con 0x7fb18c0abe20 2023-03-03T15:49:37.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:49:37.589 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:49:37.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.589+0000 7fb19415a700 1 -- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1700682f0 msgr2=0x7fb17006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:37.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.589+0000 7fb19415a700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1700682f0 0x7fb17006a7a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fb18c14c7e0 tx=0x7fb180009210 comp rx=0 tx=0).stop 2023-03-03T15:49:37.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.589+0000 7fb19415a700 1 -- 172.21.15.7:0/1393484669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 msgr2=0x7fb18c1de9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:37.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.589+0000 7fb19415a700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c1de9b0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7fb18800c520 tx=0x7fb18800c830 comp rx=0 tx=0).stop 2023-03-03T15:49:37.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.590+0000 7fb19415a700 1 -- 172.21.15.7:0/1393484669 shutdown_connections 2023-03-03T15:49:37.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.590+0000 7fb19415a700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 0x7fb18c1db700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:37.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.590+0000 7fb19415a700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1700682f0 0x7fb17006a7a0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:37.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.590+0000 7fb19415a700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c1de9b0 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:37.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.590+0000 7fb19415a700 1 --2- 172.21.15.7:0/1393484669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c14f9d0 0x7fb18c152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:37.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.590+0000 7fb19415a700 1 -- 172.21.15.7:0/1393484669 >> 172.21.15.7:0/1393484669 conn(0x7fb18c0c8900 msgr2=0x7fb18c151b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:49:37.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.590+0000 7fb19415a700 1 -- 172.21.15.7:0/1393484669 shutdown_connections 2023-03-03T15:49:37.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:37.590+0000 7fb19415a700 1 -- 172.21.15.7:0/1393484669 wait complete. 2023-03-03T15:49:37.593 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:49:37.603 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T15:49:38.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:38.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:49:47.604 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:49:47.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.741+0000 7fc635296700 1 Processor -- start 2023-03-03T15:49:47.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.741+0000 7fc635296700 1 -- start start 2023-03-03T15:49:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc635296700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6300c00e0 0x7fc6300c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc635296700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 0x7fc6300cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc635296700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6300c7310 con 0x7fc6300caa30 2023-03-03T15:49:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc635296700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6300bfd10 con 0x7fc6300c00e0 2023-03-03T15:49:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc635296700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300bfa30 con 0x7fc630154ff0 2023-03-03T15:49:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc62e7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6300c00e0 0x7fc6300c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc62effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc6300caa30 0x7fc6300c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48610/0 (socket says 172.21.15.7:48610) 2023-03-03T15:49:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc62effd700 1 -- 172.21.15.7:0/2204698220 learned_addr learned my addr 172.21.15.7:0/2204698220 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:49:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc62f7fe700 1 --2- 172.21.15.7:0/2204698220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 0x7fc6300cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc62e7fc700 1 -- 172.21.15.7:0/2204698220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 msgr2=0x7fc6300cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:49:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc62e7fc700 1 --2- 172.21.15.7:0/2204698220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 0x7fc6300cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc62e7fc700 1 -- 172.21.15.7:0/2204698220 >> v1:172.21.15.7:6789/0 conn(0x7fc6300caa30 legacy=0x7fc6300c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.742+0000 7fc62dffb700 1 -- 172.21.15.7:0/2204698220 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651267836 0 0) 0x7fc6300c7310 con 0x7fc6300caa30 2023-03-03T15:49:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc62e7fc700 1 -- 172.21.15.7:0/2204698220 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc630154510 con 0x7fc6300c00e0 2023-03-03T15:49:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc62f7fe700 1 --2- 172.21.15.7:0/2204698220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 0x7fc6300cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:49:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc62e7fc700 1 --2- 172.21.15.7:0/2204698220 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300c00e0 0x7fc6300c5c60 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fc620009820 tx=0x7fc620009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb075cc8d4d86eb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:49:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc62dffb700 1 -- 172.21.15.7:0/2204698220 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc620005e30 con 0x7fc6300c00e0 2023-03-03T15:49:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc62dffb700 1 -- 172.21.15.7:0/2204698220 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc620006030 con 0x7fc6300c00e0 2023-03-03T15:49:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc62dffb700 1 -- 172.21.15.7:0/2204698220 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6200063e0 con 0x7fc6300c00e0 2023-03-03T15:49:47.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc635296700 1 -- 172.21.15.7:0/2204698220 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300c00e0 msgr2=0x7fc6300c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:47.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc635296700 1 --2- 172.21.15.7:0/2204698220 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300c00e0 0x7fc6300c5c60 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fc620009820 tx=0x7fc620009b30 comp rx=0 tx=0).stop 2023-03-03T15:49:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc635296700 1 -- 172.21.15.7:0/2204698220 shutdown_connections 2023-03-03T15:49:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc635296700 1 --2- 172.21.15.7:0/2204698220 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 0x7fc6300cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc635296700 1 --2- 172.21.15.7:0/2204698220 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300c00e0 0x7fc6300c5c60 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.743+0000 7fc635296700 1 -- 172.21.15.7:0/2204698220 >> 172.21.15.7:0/2204698220 conn(0x7fc630066e40 msgr2=0x7fc630071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:49:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.744+0000 7fc635296700 1 -- 172.21.15.7:0/2204698220 shutdown_connections 2023-03-03T15:49:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.744+0000 7fc635296700 1 -- 172.21.15.7:0/2204698220 wait complete. 2023-03-03T15:49:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.745+0000 7fc635296700 1 Processor -- start 2023-03-03T15:49:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.745+0000 7fc635296700 1 -- start start 2023-03-03T15:49:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.746+0000 7fc635296700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6300c00e0 0x7fc6301dda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.746+0000 7fc635296700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300caa30 0x7fc630152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.746+0000 7fc635296700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 0x7fc630148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.746+0000 7fc635296700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc630158450 con 0x7fc6300caa30 2023-03-03T15:49:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.746+0000 7fc635296700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6301ea190 con 0x7fc6300c00e0 2023-03-03T15:49:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.746+0000 7fc635296700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300c40f0 con 0x7fc630154ff0 2023-03-03T15:49:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.746+0000 7fc62e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300caa30 0x7fc630152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.746+0000 7fc62e7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300caa30 0x7fc630152dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54638/0 (socket says 172.21.15.7:54638) 2023-03-03T15:49:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.746+0000 7fc62e7fc700 1 -- 172.21.15.7:0/713575136 learned_addr learned my addr 172.21.15.7:0/713575136 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:49:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc62effd700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6300c00e0 0x7fc6301dda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc62f7fe700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 0x7fc630148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc62e7fc700 1 -- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 msgr2=0x7fc630148ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc62e7fc700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 0x7fc630148ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc62e7fc700 1 -- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6300c00e0 msgr2=0x7fc6301dda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc62e7fc700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6300c00e0 0x7fc6301dda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc62e7fc700 1 -- 172.21.15.7:0/713575136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc620009430 con 0x7fc6300caa30 2023-03-03T15:49:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc62e7fc700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300caa30 0x7fc630152dc0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7fc620009f30 tx=0x7fc620004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:49:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc6177fe700 1 -- 172.21.15.7:0/713575136 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc620006950 con 0x7fc6300caa30 2023-03-03T15:49:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc6177fe700 1 -- 172.21.15.7:0/713575136 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc620020070 con 0x7fc6300caa30 2023-03-03T15:49:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc6177fe700 1 -- 172.21.15.7:0/713575136 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc620025670 con 0x7fc6300caa30 2023-03-03T15:49:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc635296700 1 -- 172.21.15.7:0/713575136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc630154ad0 con 0x7fc6300caa30 2023-03-03T15:49:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.747+0000 7fc635296700 1 -- 172.21.15.7:0/713575136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6301547f0 con 0x7fc6300caa30 2023-03-03T15:49:47.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.748+0000 7fc635296700 1 -- 172.21.15.7:0/713575136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6300bf650 con 0x7fc6300caa30 2023-03-03T15:49:47.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.749+0000 7fc6177fe700 1 -- 172.21.15.7:0/713575136 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc62002b020 con 0x7fc6300caa30 2023-03-03T15:49:47.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.749+0000 7fc6177fe700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc61c0682f0 0x7fc61c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:49:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.749+0000 7fc6177fe700 1 -- 172.21.15.7:0/713575136 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc620087960 con 0x7fc6300caa30 2023-03-03T15:49:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.749+0000 7fc62effd700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc61c0682f0 0x7fc61c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:49:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.750+0000 7fc62effd700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc61c0682f0 0x7fc61c06a7a0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fc624007fd0 tx=0x7fc624007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:49:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.754+0000 7fc6177fe700 1 -- 172.21.15.7:0/713575136 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc620053d00 con 0x7fc6300caa30 2023-03-03T15:49:47.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.992+0000 7fc635296700 1 -- 172.21.15.7:0/713575136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc630074bb0 con 0x7fc6300caa30 2023-03-03T15:49:47.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.993+0000 7fc6177fe700 1 -- 172.21.15.7:0/713575136 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc620057320 con 0x7fc6300caa30 2023-03-03T15:49:47.995 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:49:47.996 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:49:47.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 -- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc61c0682f0 msgr2=0x7fc61c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:47.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc61c0682f0 0x7fc61c06a7a0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fc624007fd0 tx=0x7fc624007de0 comp rx=0 tx=0).stop 2023-03-03T15:49:47.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 -- 172.21.15.7:0/713575136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300caa30 msgr2=0x7fc630152dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:49:47.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300caa30 0x7fc630152dc0 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7fc620009f30 tx=0x7fc620004750 comp rx=0 tx=0).stop 2023-03-03T15:49:47.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 -- 172.21.15.7:0/713575136 shutdown_connections 2023-03-03T15:49:47.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc630154ff0 0x7fc630148ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:47.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6300c00e0 0x7fc6301dda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:47.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc61c0682f0 0x7fc61c06a7a0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:47.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 --2- 172.21.15.7:0/713575136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6300caa30 0x7fc630152dc0 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:49:47.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 -- 172.21.15.7:0/713575136 >> 172.21.15.7:0/713575136 conn(0x7fc630066e40 msgr2=0x7fc630153890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:49:47.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 -- 172.21.15.7:0/713575136 shutdown_connections 2023-03-03T15:49:47.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:49:47.995+0000 7fc635296700 1 -- 172.21.15.7:0/713575136 wait complete. 2023-03-03T15:49:47.999 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:49:48.009 INFO:tasks.mds_thrash.fs.[a]:waiting for 89 secs before thrashing 2023-03-03T15:50:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T15:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T15:50:08.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:08.878 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:12.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:50:12.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.811+0000 7f231a91c700 1 Processor -- start 2023-03-03T15:50:12.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.811+0000 7f231a91c700 1 -- start start 2023-03-03T15:50:12.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f231a91c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f231414f9d0 0x7f2314156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:12.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f231a91c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23140abe20 0x7f23141529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:12.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f231a91c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23140c2f30 con 0x7f231414fda0 2023-03-03T15:50:12.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f231a91c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23140c31d0 con 0x7f231414f9d0 2023-03-03T15:50:12.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f231a91c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23141524a0 con 0x7f23140abe20 2023-03-03T15:50:12.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f2313fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f231414f9d0 0x7f2314156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:12.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f23137fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f231414fda0 0x7f2314071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48632/0 (socket says 172.21.15.7:48632) 2023-03-03T15:50:12.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f23137fe700 1 -- 172.21.15.7:0/3810680884 learned_addr learned my addr 172.21.15.7:0/3810680884 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:50:12.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f2313fff700 1 -- 172.21.15.7:0/3810680884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23140abe20 msgr2=0x7f23141529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:50:12.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f2313fff700 1 --2- 172.21.15.7:0/3810680884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23140abe20 0x7f23141529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:12.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f2313fff700 1 -- 172.21.15.7:0/3810680884 >> v1:172.21.15.7:6789/0 conn(0x7f231414fda0 legacy=0x7f2314071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:12.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f2313fff700 1 -- 172.21.15.7:0/3810680884 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23140c34b0 con 0x7f231414f9d0 2023-03-03T15:50:12.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f2312ffd700 1 -- 172.21.15.7:0/3810680884 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697183366 0 0) 0x7f23140c2f30 con 0x7f231414fda0 2023-03-03T15:50:12.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.812+0000 7f2313fff700 1 --2- 172.21.15.7:0/3810680884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414f9d0 0x7f2314156430 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009840 tx=0x7f22fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3831f3e24e3e23ea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:50:12.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f2312ffd700 1 -- 172.21.15.7:0/3810680884 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc011830 con 0x7f231414f9d0 2023-03-03T15:50:12.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f2312ffd700 1 -- 172.21.15.7:0/3810680884 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc011990 con 0x7f231414f9d0 2023-03-03T15:50:12.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f2312ffd700 1 -- 172.21.15.7:0/3810680884 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc011d20 con 0x7f231414f9d0 2023-03-03T15:50:12.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f231a91c700 1 -- 172.21.15.7:0/3810680884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414f9d0 msgr2=0x7f2314156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:12.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f231a91c700 1 --2- 172.21.15.7:0/3810680884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414f9d0 0x7f2314156430 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f22fc009840 tx=0x7f22fc009b50 comp rx=0 tx=0).stop 2023-03-03T15:50:12.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f231a91c700 1 -- 172.21.15.7:0/3810680884 shutdown_connections 2023-03-03T15:50:12.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f231a91c700 1 --2- 172.21.15.7:0/3810680884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23140abe20 0x7f23141529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:12.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f231a91c700 1 --2- 172.21.15.7:0/3810680884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414f9d0 0x7f2314156430 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:12.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f231a91c700 1 -- 172.21.15.7:0/3810680884 >> 172.21.15.7:0/3810680884 conn(0x7f23140c8900 msgr2=0x7f23140c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:50:12.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f231a91c700 1 -- 172.21.15.7:0/3810680884 shutdown_connections 2023-03-03T15:50:12.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.813+0000 7f231a91c700 1 -- 172.21.15.7:0/3810680884 wait complete. 2023-03-03T15:50:12.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.814+0000 7f231a91c700 1 Processor -- start 2023-03-03T15:50:12.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.815+0000 7f231a91c700 1 -- start start 2023-03-03T15:50:12.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.815+0000 7f231a91c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23140abe20 0x7f231406f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:12.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.815+0000 7f231a91c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f231414f9d0 0x7f23140c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:12.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.815+0000 7f231a91c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414fda0 0x7f231406b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:12.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.815+0000 7f231a91c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231406fc90 con 0x7f231414fda0 2023-03-03T15:50:12.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.815+0000 7f231a91c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23141e75d0 con 0x7f23140abe20 2023-03-03T15:50:12.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.815+0000 7f231a91c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23140abc80 con 0x7f231414f9d0 2023-03-03T15:50:12.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.815+0000 7f2318eb9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414fda0 0x7f231406b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:12.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2318eb9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414fda0 0x7f231406b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54660/0 (socket says 172.21.15.7:54660) 2023-03-03T15:50:12.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2318eb9700 1 -- 172.21.15.7:0/208249927 learned_addr learned my addr 172.21.15.7:0/208249927 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:50:12.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2313fff700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23140abe20 0x7f231406f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:12.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f23137fe700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f231414f9d0 0x7f23140c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:12.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2318eb9700 1 -- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f231414f9d0 msgr2=0x7f23140c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:12.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2318eb9700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f231414f9d0 0x7f23140c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:12.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2318eb9700 1 -- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23140abe20 msgr2=0x7f231406f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:12.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2318eb9700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23140abe20 0x7f231406f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:12.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2318eb9700 1 -- 172.21.15.7:0/208249927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc0094a0 con 0x7f231414fda0 2023-03-03T15:50:12.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2318eb9700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414fda0 0x7f231406b560 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f230c00b9e0 tx=0x7f230c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:50:12.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2310ff9700 1 -- 172.21.15.7:0/208249927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c00c9b0 con 0x7f231414fda0 2023-03-03T15:50:12.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2310ff9700 1 -- 172.21.15.7:0/208249927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230c00cb30 con 0x7f231414fda0 2023-03-03T15:50:12.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f2310ff9700 1 -- 172.21.15.7:0/208249927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c014440 con 0x7f231414fda0 2023-03-03T15:50:12.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f231a91c700 1 -- 172.21.15.7:0/208249927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23140ac350 con 0x7f231414fda0 2023-03-03T15:50:12.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.816+0000 7f231a91c700 1 -- 172.21.15.7:0/208249927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23140ab630 con 0x7f231414fda0 2023-03-03T15:50:12.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.817+0000 7f231a91c700 1 -- 172.21.15.7:0/208249927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2314071fa0 con 0x7f231414fda0 2023-03-03T15:50:12.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.818+0000 7f2310ff9700 1 -- 172.21.15.7:0/208249927 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f230c01b030 con 0x7f231414fda0 2023-03-03T15:50:12.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.818+0000 7f2310ff9700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22f40682f0 0x7f22f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:12.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.818+0000 7f2310ff9700 1 -- 172.21.15.7:0/208249927 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f230c085d90 con 0x7f231414fda0 2023-03-03T15:50:12.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.818+0000 7f2313fff700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22f40682f0 0x7f22f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:12.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.819+0000 7f2313fff700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22f40682f0 0x7f22f406a7a0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0090b0 tx=0x7f22fc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:50:12.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:12.822+0000 7f2310ff9700 1 -- 172.21.15.7:0/208249927 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f230c052130 con 0x7f231414fda0 2023-03-03T15:50:13.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.059+0000 7f231a91c700 1 -- 172.21.15.7:0/208249927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f231404e6e0 con 0x7f231414fda0 2023-03-03T15:50:13.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.060+0000 7f2310ff9700 1 -- 172.21.15.7:0/208249927 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f230c019090 con 0x7f231414fda0 2023-03-03T15:50:13.061 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:50:13.061 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:50:13.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 -- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22f40682f0 msgr2=0x7f22f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:13.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22f40682f0 0x7f22f406a7a0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f22fc0090b0 tx=0x7f22fc008030 comp rx=0 tx=0).stop 2023-03-03T15:50:13.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 -- 172.21.15.7:0/208249927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414fda0 msgr2=0x7f231406b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:13.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414fda0 0x7f231406b560 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f230c00b9e0 tx=0x7f230c00bda0 comp rx=0 tx=0).stop 2023-03-03T15:50:13.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 -- 172.21.15.7:0/208249927 shutdown_connections 2023-03-03T15:50:13.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22f40682f0 0x7f22f406a7a0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:13.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f231414fda0 0x7f231406b560 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:13.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23140abe20 0x7f231406f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:13.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 --2- 172.21.15.7:0/208249927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f231414f9d0 0x7f23140c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:13.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 -- 172.21.15.7:0/208249927 >> 172.21.15.7:0/208249927 conn(0x7f23140c8900 msgr2=0x7f23141512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:50:13.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 -- 172.21.15.7:0/208249927 shutdown_connections 2023-03-03T15:50:13.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:13.062+0000 7f231a91c700 1 -- 172.21.15.7:0/208249927 wait complete. 2023-03-03T15:50:13.065 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:50:13.075 INFO:tasks.mds_thrash.fs.[b]:waiting for 29 secs before thrashing 2023-03-03T15:50:38.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:38.942 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:50:42.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:50:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.214+0000 7fe831005700 1 Processor -- start 2023-03-03T15:50:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.214+0000 7fe831005700 1 -- start start 2023-03-03T15:50:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe831005700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe82c06d870 0x7fe82c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe831005700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe82c0c3e30 0x7fe82c0c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe831005700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82c0c7a60 con 0x7fe82c0c1da0 2023-03-03T15:50:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe831005700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82c0c2af0 con 0x7fe82c0c3e30 2023-03-03T15:50:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe831005700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82c0c3620 con 0x7fe82c06d870 2023-03-03T15:50:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe82b59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe82c0c3e30 0x7fe82c0c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe82a59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe82c0c1da0 0x7fe82c0c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48652/0 (socket says 172.21.15.7:48652) 2023-03-03T15:50:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe82a59c700 1 -- 172.21.15.7:0/49954806 learned_addr learned my addr 172.21.15.7:0/49954806 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:50:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe82b59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe82c0c3e30 0x7fe82c0c4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54674/0 (socket says 172.21.15.7:54674) 2023-03-03T15:50:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe82ad9d700 1 --2- 172.21.15.7:0/49954806 >> v2:172.21.15.143:3300/0 conn(0x7fe82c06d870 0x7fe82c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe82b59e700 1 -- 172.21.15.7:0/49954806 >> v2:172.21.15.143:3300/0 conn(0x7fe82c06d870 msgr2=0x7fe82c1550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:50:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe82b59e700 1 --2- 172.21.15.7:0/49954806 >> v2:172.21.15.143:3300/0 conn(0x7fe82c06d870 0x7fe82c1550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe82b59e700 1 -- 172.21.15.7:0/49954806 >> v1:172.21.15.7:6789/0 conn(0x7fe82c0c1da0 legacy=0x7fe82c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe82b59e700 1 -- 172.21.15.7:0/49954806 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c1545e0 con 0x7fe82c0c3e30 2023-03-03T15:50:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe82b59e700 1 --2- 172.21.15.7:0/49954806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c3e30 0x7fe82c0c4230 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7fe82c073880 tx=0x7fe82400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11cd06bd0fefa9e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:50:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe829d9b700 1 -- 172.21.15.7:0/49954806 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82400e070 con 0x7fe82c0c3e30 2023-03-03T15:50:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe829d9b700 1 -- 172.21.15.7:0/49954806 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82400bbd0 con 0x7fe82c0c3e30 2023-03-03T15:50:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.215+0000 7fe829d9b700 1 -- 172.21.15.7:0/49954806 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824012430 con 0x7fe82c0c3e30 2023-03-03T15:50:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.216+0000 7fe831005700 1 -- 172.21.15.7:0/49954806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c3e30 msgr2=0x7fe82c0c4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.216+0000 7fe831005700 1 --2- 172.21.15.7:0/49954806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c3e30 0x7fe82c0c4230 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7fe82c073880 tx=0x7fe82400b330 comp rx=0 tx=0).stop 2023-03-03T15:50:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.216+0000 7fe831005700 1 -- 172.21.15.7:0/49954806 shutdown_connections 2023-03-03T15:50:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.216+0000 7fe831005700 1 --2- 172.21.15.7:0/49954806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c3e30 0x7fe82c0c4230 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.216+0000 7fe831005700 1 --2- 172.21.15.7:0/49954806 >> v2:172.21.15.143:3300/0 conn(0x7fe82c06d870 0x7fe82c1550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.216+0000 7fe831005700 1 -- 172.21.15.7:0/49954806 >> 172.21.15.7:0/49954806 conn(0x7fe82c06c110 msgr2=0x7fe82c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:50:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.216+0000 7fe831005700 1 -- 172.21.15.7:0/49954806 shutdown_connections 2023-03-03T15:50:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.216+0000 7fe831005700 1 -- 172.21.15.7:0/49954806 wait complete. 2023-03-03T15:50:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.218+0000 7fe831005700 1 Processor -- start 2023-03-03T15:50:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.218+0000 7fe831005700 1 -- start start 2023-03-03T15:50:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.218+0000 7fe831005700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c06d870 0x7fe82c0ab520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.218+0000 7fe831005700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c1da0 0x7fe82c0aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.218+0000 7fe831005700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c0c3e30 0x7fe82c1ed060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.218+0000 7fe831005700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0acf30 con 0x7fe82c0c1da0 2023-03-03T15:50:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.218+0000 7fe831005700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c14e9c0 con 0x7fe82c06d870 2023-03-03T15:50:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.218+0000 7fe831005700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0ac720 con 0x7fe82c0c3e30 2023-03-03T15:50:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.218+0000 7fe82a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c1da0 0x7fe82c0aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.218+0000 7fe82a59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c1da0 0x7fe82c0aba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54678/0 (socket says 172.21.15.7:54678) 2023-03-03T15:50:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe82a59c700 1 -- 172.21.15.7:0/3279669376 learned_addr learned my addr 172.21.15.7:0/3279669376 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:50:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe82ad9d700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c06d870 0x7fe82c0ab520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe82b59e700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c0c3e30 0x7fe82c1ed060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe82a59c700 1 -- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c0c3e30 msgr2=0x7fe82c1ed060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe82a59c700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c0c3e30 0x7fe82c1ed060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe82a59c700 1 -- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c06d870 msgr2=0x7fe82c0ab520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe82a59c700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c06d870 0x7fe82c0ab520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe82a59c700 1 -- 172.21.15.7:0/3279669376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe824009a70 con 0x7fe82c0c1da0 2023-03-03T15:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe82ad9d700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c06d870 0x7fe82c0ab520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe82a59c700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c1da0 0x7fe82c0aba60 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fe82000c550 tx=0x7fe82000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe8137fe700 1 -- 172.21.15.7:0/3279669376 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820005d70 con 0x7fe82c0c1da0 2023-03-03T15:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe8137fe700 1 -- 172.21.15.7:0/3279669376 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8200070d0 con 0x7fe82c0c1da0 2023-03-03T15:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe831005700 1 -- 172.21.15.7:0/3279669376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82c154ba0 con 0x7fe82c0c1da0 2023-03-03T15:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe8137fe700 1 -- 172.21.15.7:0/3279669376 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820007440 con 0x7fe82c0c1da0 2023-03-03T15:50:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.219+0000 7fe831005700 1 -- 172.21.15.7:0/3279669376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82c1548c0 con 0x7fe82c0c1da0 2023-03-03T15:50:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.220+0000 7fe831005700 1 -- 172.21.15.7:0/3279669376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82c06dc40 con 0x7fe82c0c1da0 2023-03-03T15:50:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.221+0000 7fe8137fe700 1 -- 172.21.15.7:0/3279669376 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe820019030 con 0x7fe82c0c1da0 2023-03-03T15:50:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.221+0000 7fe8137fe700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8180682f0 0x7fe81806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.221+0000 7fe8137fe700 1 -- 172.21.15.7:0/3279669376 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe820085410 con 0x7fe82c0c1da0 2023-03-03T15:50:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.221+0000 7fe82ad9d700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8180682f0 0x7fe81806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.222+0000 7fe82ad9d700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8180682f0 0x7fe81806a7a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0099f0 tx=0x7fe81c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:50:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.226+0000 7fe8137fe700 1 -- 172.21.15.7:0/3279669376 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe82004b5e0 con 0x7fe82c0c1da0 2023-03-03T15:50:42.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.465+0000 7fe831005700 1 -- 172.21.15.7:0/3279669376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe82c0aae60 con 0x7fe82c0c1da0 2023-03-03T15:50:42.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.466+0000 7fe8137fe700 1 -- 172.21.15.7:0/3279669376 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe82c0aae60 con 0x7fe82c0c1da0 2023-03-03T15:50:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:50:42.467 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:50:42.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 -- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8180682f0 msgr2=0x7fe81806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:42.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8180682f0 0x7fe81806a7a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0099f0 tx=0x7fe81c008040 comp rx=0 tx=0).stop 2023-03-03T15:50:42.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 -- 172.21.15.7:0/3279669376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c1da0 msgr2=0x7fe82c0aba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:42.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c1da0 0x7fe82c0aba60 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fe82000c550 tx=0x7fe82000c910 comp rx=0 tx=0).stop 2023-03-03T15:50:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 -- 172.21.15.7:0/3279669376 shutdown_connections 2023-03-03T15:50:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe82c0c3e30 0x7fe82c1ed060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c06d870 0x7fe82c0ab520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8180682f0 0x7fe81806a7a0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 --2- 172.21.15.7:0/3279669376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe82c0c1da0 0x7fe82c0aba60 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 -- 172.21.15.7:0/3279669376 >> 172.21.15.7:0/3279669376 conn(0x7fe82c06c110 msgr2=0x7fe82c14abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:50:42.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 -- 172.21.15.7:0/3279669376 shutdown_connections 2023-03-03T15:50:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:42.468+0000 7fe831005700 1 -- 172.21.15.7:0/3279669376 wait complete. 2023-03-03T15:50:42.471 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:50:42.481 INFO:tasks.mds_thrash.fs.[b]:waiting for 17 secs before thrashing 2023-03-03T15:50:59.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:50:59.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.618+0000 7f5e7ecb7700 1 Processor -- start 2023-03-03T15:50:59.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.618+0000 7f5e7ecb7700 1 -- start start 2023-03-03T15:50:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.618+0000 7f5e7ecb7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e7814b340 0x7f5e78152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.618+0000 7f5e7ecb7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5e781546f0 0x7f5e78154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.618+0000 7f5e7ecb7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7806c920 con 0x7f5e780c2790 2023-03-03T15:50:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.618+0000 7f5e7ecb7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e780abd90 con 0x7f5e781546f0 2023-03-03T15:50:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.618+0000 7f5e7ecb7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e780c1300 con 0x7f5e7814b340 2023-03-03T15:50:59.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.618+0000 7f5e7d254700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5e781546f0 0x7f5e78154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e7d254700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5e781546f0 0x7f5e78154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54702/0 (socket says 172.21.15.7:54702) 2023-03-03T15:50:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e7d254700 1 -- 172.21.15.7:0/3215586889 learned_addr learned my addr 172.21.15.7:0/3215586889 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:50:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e7ca53700 1 --2- 172.21.15.7:0/3215586889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e7814b340 0x7f5e78152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e7d254700 1 -- 172.21.15.7:0/3215586889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e7814b340 msgr2=0x7f5e78152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:50:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e7d254700 1 --2- 172.21.15.7:0/3215586889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e7814b340 0x7f5e78152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e7d254700 1 -- 172.21.15.7:0/3215586889 >> v1:172.21.15.143:6789/0 conn(0x7f5e780c2790 legacy=0x7f5e78152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e7d254700 1 -- 172.21.15.7:0/3215586889 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e78153280 con 0x7f5e781546f0 2023-03-03T15:50:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e7d254700 1 --2- 172.21.15.7:0/3215586889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e781546f0 0x7f5e78154b40 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009e50 tx=0x7f5e7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a957f71476dfa452 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:50:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e777fe700 1 -- 172.21.15.7:0/3215586889 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7000e070 con 0x7f5e781546f0 2023-03-03T15:50:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e777fe700 1 -- 172.21.15.7:0/3215586889 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7000b910 con 0x7f5e781546f0 2023-03-03T15:50:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.619+0000 7f5e777fe700 1 -- 172.21.15.7:0/3215586889 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7000bcc0 con 0x7f5e781546f0 2023-03-03T15:50:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.620+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/3215586889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e781546f0 msgr2=0x7f5e78154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.620+0000 7f5e7ecb7700 1 --2- 172.21.15.7:0/3215586889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e781546f0 0x7f5e78154b40 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009e50 tx=0x7f5e7000b330 comp rx=0 tx=0).stop 2023-03-03T15:50:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.620+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/3215586889 shutdown_connections 2023-03-03T15:50:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.620+0000 7f5e7ecb7700 1 --2- 172.21.15.7:0/3215586889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e781546f0 0x7f5e78154b40 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.620+0000 7f5e7ecb7700 1 --2- 172.21.15.7:0/3215586889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e7814b340 0x7f5e78152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.620+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/3215586889 >> 172.21.15.7:0/3215586889 conn(0x7f5e780c62e0 msgr2=0x7f5e780c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:50:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.620+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/3215586889 shutdown_connections 2023-03-03T15:50:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.620+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/3215586889 wait complete. 2023-03-03T15:50:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.621+0000 7f5e7ecb7700 1 Processor -- start 2023-03-03T15:50:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.621+0000 7f5e7ecb7700 1 -- start start 2023-03-03T15:50:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ecb7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e780c2790 0x7f5e781ed830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ecb7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e7814b340 0x7f5e78147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:59.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ecb7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e781546f0 0x7f5e781ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:59.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ecb7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e780bf980 con 0x7f5e780c2790 2023-03-03T15:50:59.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ecb7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e781ea620 con 0x7f5e781546f0 2023-03-03T15:50:59.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ecb7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e781ee750 con 0x7f5e7814b340 2023-03-03T15:50:59.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ca53700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e780c2790 0x7f5e781ed830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:59.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ca53700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e780c2790 0x7f5e781ed830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54704/0 (socket says 172.21.15.7:54704) 2023-03-03T15:50:59.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ca53700 1 -- 172.21.15.7:0/2618393303 learned_addr learned my addr 172.21.15.7:0/2618393303 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:50:59.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7d254700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e781546f0 0x7f5e781ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:59.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e77fff700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e7814b340 0x7f5e78147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:59.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ca53700 1 -- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e7814b340 msgr2=0x7f5e78147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:59.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ca53700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e7814b340 0x7f5e78147730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:59.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ca53700 1 -- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e781546f0 msgr2=0x7f5e781ee210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:59.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ca53700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e781546f0 0x7f5e781ee210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:59.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ca53700 1 -- 172.21.15.7:0/2618393303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e70009b30 con 0x7f5e780c2790 2023-03-03T15:50:59.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.622+0000 7f5e7ca53700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e780c2790 0x7f5e781ed830 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00e870 tx=0x7f5e6c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:50:59.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.623+0000 7f5e757fa700 1 -- 172.21.15.7:0/2618393303 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c009590 con 0x7f5e780c2790 2023-03-03T15:50:59.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.623+0000 7f5e757fa700 1 -- 172.21.15.7:0/2618393303 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c00ee70 con 0x7f5e780c2790 2023-03-03T15:50:59.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.623+0000 7f5e757fa700 1 -- 172.21.15.7:0/2618393303 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c00c9f0 con 0x7f5e780c2790 2023-03-03T15:50:59.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.623+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/2618393303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7806bff0 con 0x7f5e780c2790 2023-03-03T15:50:59.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.623+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/2618393303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7806bd50 con 0x7f5e780c2790 2023-03-03T15:50:59.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.624+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/2618393303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7806c2d0 con 0x7f5e780c2790 2023-03-03T15:50:59.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.624+0000 7f5e757fa700 1 -- 172.21.15.7:0/2618393303 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5e6c023070 con 0x7f5e780c2790 2023-03-03T15:50:59.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.624+0000 7f5e757fa700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e5c0682f0 0x7f5e5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:50:59.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.625+0000 7f5e757fa700 1 -- 172.21.15.7:0/2618393303 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5e6c0863a0 con 0x7f5e780c2790 2023-03-03T15:50:59.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.625+0000 7f5e77fff700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e5c0682f0 0x7f5e5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:50:59.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.625+0000 7f5e77fff700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e5c0682f0 0x7f5e5c06a7a0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f5e78146940 tx=0x7f5e64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:50:59.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.629+0000 7f5e757fa700 1 -- 172.21.15.7:0/2618393303 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5e6c052650 con 0x7f5e780c2790 2023-03-03T15:50:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.869+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/2618393303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e781eb260 con 0x7f5e780c2790 2023-03-03T15:50:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.869+0000 7f5e757fa700 1 -- 172.21.15.7:0/2618393303 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5e781eb260 con 0x7f5e780c2790 2023-03-03T15:50:59.872 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:50:59.872 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:50:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e5c0682f0 msgr2=0x7f5e5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:59.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e5c0682f0 0x7f5e5c06a7a0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f5e78146940 tx=0x7f5e64009040 comp rx=0 tx=0).stop 2023-03-03T15:50:59.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/2618393303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e780c2790 msgr2=0x7f5e781ed830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:50:59.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e780c2790 0x7f5e781ed830 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00e870 tx=0x7f5e6c00ec30 comp rx=0 tx=0).stop 2023-03-03T15:50:59.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/2618393303 shutdown_connections 2023-03-03T15:50:59.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5e781546f0 0x7f5e781ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:59.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5e5c0682f0 0x7f5e5c06a7a0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:59.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e780c2790 0x7f5e781ed830 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:59.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 --2- 172.21.15.7:0/2618393303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5e7814b340 0x7f5e78147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:50:59.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/2618393303 >> 172.21.15.7:0/2618393303 conn(0x7f5e780c62e0 msgr2=0x7f5e78152530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:50:59.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/2618393303 shutdown_connections 2023-03-03T15:50:59.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:50:59.871+0000 7f5e7ecb7700 1 -- 172.21.15.7:0/2618393303 wait complete. 2023-03-03T15:50:59.875 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:50:59.886 INFO:tasks.mds_thrash.fs.[b]:waiting for 81 secs before thrashing 2023-03-03T15:51:09.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:09.007 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:17.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:51:17.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.148+0000 7f2a73dca700 1 Processor -- start 2023-03-03T15:51:17.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.148+0000 7f2a73dca700 1 -- start start 2023-03-03T15:51:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a73dca700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a6c151ab0 0x7f2a6c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a73dca700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6c06d780 con 0x7f2a6c073c30 2023-03-03T15:51:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a73dca700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6c06cb70 con 0x7f2a6c06c5b0 2023-03-03T15:51:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a73dca700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a6c06ce50 con 0x7f2a6c151ab0 2023-03-03T15:51:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a71b66700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a6c151ab0 0x7f2a6c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a71365700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2a6c073c30 0x7f2a6c06bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48700/0 (socket says 172.21.15.7:48700) 2023-03-03T15:51:17.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a71365700 1 -- 172.21.15.7:0/1198838138 learned_addr learned my addr 172.21.15.7:0/1198838138 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:51:17.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a71b66700 1 -- 172.21.15.7:0/1198838138 >> v1:172.21.15.143:6789/0 conn(0x7f2a6c06c5b0 legacy=0x7f2a6c157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:51:17.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a71b66700 1 -- 172.21.15.7:0/1198838138 >> v1:172.21.15.7:6789/0 conn(0x7f2a6c073c30 legacy=0x7f2a6c06bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:17.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a71b66700 1 -- 172.21.15.7:0/1198838138 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6c06d130 con 0x7f2a6c151ab0 2023-03-03T15:51:17.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a70b64700 1 -- 172.21.15.7:0/1198838138 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604624721 0 0) 0x7f2a6c06d780 con 0x7f2a6c073c30 2023-03-03T15:51:17.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.149+0000 7f2a71b66700 1 --2- 172.21.15.7:0/1198838138 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c151ab0 0x7f2a6c0736f0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009840 tx=0x7f2a68009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f698991562e83 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:51:17.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.150+0000 7f2a70b64700 1 -- 172.21.15.7:0/1198838138 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68004230 con 0x7f2a6c151ab0 2023-03-03T15:51:17.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.150+0000 7f2a70b64700 1 -- 172.21.15.7:0/1198838138 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a68004390 con 0x7f2a6c151ab0 2023-03-03T15:51:17.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.150+0000 7f2a70b64700 1 -- 172.21.15.7:0/1198838138 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a680036a0 con 0x7f2a6c151ab0 2023-03-03T15:51:17.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.150+0000 7f2a73dca700 1 -- 172.21.15.7:0/1198838138 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c151ab0 msgr2=0x7f2a6c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:17.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.150+0000 7f2a73dca700 1 --2- 172.21.15.7:0/1198838138 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c151ab0 0x7f2a6c0736f0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009840 tx=0x7f2a68009b50 comp rx=0 tx=0).stop 2023-03-03T15:51:17.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.150+0000 7f2a73dca700 1 -- 172.21.15.7:0/1198838138 shutdown_connections 2023-03-03T15:51:17.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.150+0000 7f2a73dca700 1 --2- 172.21.15.7:0/1198838138 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c151ab0 0x7f2a6c0736f0 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:17.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.150+0000 7f2a73dca700 1 -- 172.21.15.7:0/1198838138 >> 172.21.15.7:0/1198838138 conn(0x7f2a6c0a94d0 msgr2=0x7f2a6c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:51:17.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.150+0000 7f2a73dca700 1 -- 172.21.15.7:0/1198838138 shutdown_connections 2023-03-03T15:51:17.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.150+0000 7f2a73dca700 1 -- 172.21.15.7:0/1198838138 wait complete. 2023-03-03T15:51:17.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.151+0000 7f2a73dca700 1 Processor -- start 2023-03-03T15:51:17.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.151+0000 7f2a73dca700 1 -- start start 2023-03-03T15:51:17.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a73dca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6c06c5b0 0x7f2a6c1dfff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:17.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a73dca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c073c30 0x7f2a6c1e0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:17.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a73dca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6c151ab0 0x7f2a6c1e11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:17.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a73dca700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c14c9a0 con 0x7f2a6c073c30 2023-03-03T15:51:17.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a73dca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c0c04b0 con 0x7f2a6c151ab0 2023-03-03T15:51:17.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a73dca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c1e0d10 con 0x7f2a6c06c5b0 2023-03-03T15:51:17.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a71365700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c073c30 0x7f2a6c1e0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:17.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a71365700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c073c30 0x7f2a6c1e0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54726/0 (socket says 172.21.15.7:54726) 2023-03-03T15:51:17.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a71365700 1 -- 172.21.15.7:0/2649879833 learned_addr learned my addr 172.21.15.7:0/2649879833 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:51:17.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a71b66700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6c06c5b0 0x7f2a6c1dfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.152+0000 7f2a72367700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6c151ab0 0x7f2a6c1e11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a71365700 1 -- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6c06c5b0 msgr2=0x7f2a6c1dfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a71365700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6c06c5b0 0x7f2a6c1dfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a71365700 1 -- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6c151ab0 msgr2=0x7f2a6c1e11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a71365700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6c151ab0 0x7f2a6c1e11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a71365700 1 -- 172.21.15.7:0/2649879833 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a680094a0 con 0x7f2a6c073c30 2023-03-03T15:51:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a72367700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6c151ab0 0x7f2a6c1e11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:51:17.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a71365700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c073c30 0x7f2a6c1e0530 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c0011e0 tx=0x7f2a5c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:51:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a627fc700 1 -- 172.21.15.7:0/2649879833 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c004c80 con 0x7f2a6c073c30 2023-03-03T15:51:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a627fc700 1 -- 172.21.15.7:0/2649879833 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5c004de0 con 0x7f2a6c073c30 2023-03-03T15:51:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a73dca700 1 -- 172.21.15.7:0/2649879833 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a6c151e80 con 0x7f2a6c073c30 2023-03-03T15:51:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a627fc700 1 -- 172.21.15.7:0/2649879833 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c007310 con 0x7f2a6c073c30 2023-03-03T15:51:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.153+0000 7f2a73dca700 1 -- 172.21.15.7:0/2649879833 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6c06d410 con 0x7f2a6c073c30 2023-03-03T15:51:17.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.154+0000 7f2a73dca700 1 -- 172.21.15.7:0/2649879833 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a6c152010 con 0x7f2a6c073c30 2023-03-03T15:51:17.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.155+0000 7f2a627fc700 1 -- 172.21.15.7:0/2649879833 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2a5c01a020 con 0x7f2a6c073c30 2023-03-03T15:51:17.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.155+0000 7f2a627fc700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a500682f0 0x7f2a5006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:51:17.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.155+0000 7f2a627fc700 1 -- 172.21.15.7:0/2649879833 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2a5c085cd0 con 0x7f2a6c073c30 2023-03-03T15:51:17.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.155+0000 7f2a71b66700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a500682f0 0x7f2a5006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:51:17.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.156+0000 7f2a71b66700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a500682f0 0x7f2a5006a7a0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f2a680090b0 tx=0x7f2a68008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:51:17.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.160+0000 7f2a627fc700 1 -- 172.21.15.7:0/2649879833 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2a5c052070 con 0x7f2a6c073c30 2023-03-03T15:51:17.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.399+0000 7f2a73dca700 1 -- 172.21.15.7:0/2649879833 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a6c1e0ae0 con 0x7f2a6c073c30 2023-03-03T15:51:17.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.400+0000 7f2a627fc700 1 -- 172.21.15.7:0/2649879833 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2a6c1e0ae0 con 0x7f2a6c073c30 2023-03-03T15:51:17.401 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:51:17.401 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:51:17.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 -- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a500682f0 msgr2=0x7f2a5006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:17.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a500682f0 0x7f2a5006a7a0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f2a680090b0 tx=0x7f2a68008030 comp rx=0 tx=0).stop 2023-03-03T15:51:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 -- 172.21.15.7:0/2649879833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c073c30 msgr2=0x7f2a6c1e0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:51:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c073c30 0x7f2a6c1e0530 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c0011e0 tx=0x7f2a5c00ca30 comp rx=0 tx=0).stop 2023-03-03T15:51:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 -- 172.21.15.7:0/2649879833 shutdown_connections 2023-03-03T15:51:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a6c151ab0 0x7f2a6c1e11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a6c06c5b0 0x7f2a6c1dfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a500682f0 0x7f2a5006a7a0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:17.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 --2- 172.21.15.7:0/2649879833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a6c073c30 0x7f2a6c1e0530 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:51:17.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 -- 172.21.15.7:0/2649879833 >> 172.21.15.7:0/2649879833 conn(0x7f2a6c0a94d0 msgr2=0x7f2a6c071760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:51:17.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 -- 172.21.15.7:0/2649879833 shutdown_connections 2023-03-03T15:51:17.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:51:17.402+0000 7f2a73dca700 1 -- 172.21.15.7:0/2649879833 wait complete. 2023-03-03T15:51:17.405 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:51:17.415 INFO:tasks.mds_thrash.fs.[a]:waiting for 53 secs before thrashing 2023-03-03T15:51:39.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:51:39.146 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:09.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:09.276 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:10.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:52:10.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.553+0000 7fea388c2700 1 Processor -- start 2023-03-03T15:52:10.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.553+0000 7fea388c2700 1 -- start start 2023-03-03T15:52:10.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.554+0000 7fea388c2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fea3406d870 0x7fea341550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:10.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.554+0000 7fea388c2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fea340c1da0 0x7fea340c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:10.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.554+0000 7fea388c2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea340c3620 con 0x7fea340c3e30 2023-03-03T15:52:10.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.554+0000 7fea388c2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea340c2af0 con 0x7fea340c1da0 2023-03-03T15:52:10.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.554+0000 7fea388c2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea341545e0 con 0x7fea3406d870 2023-03-03T15:52:10.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.554+0000 7fea31d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fea340c1da0 0x7fea340c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:10.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.554+0000 7fea31d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fea340c1da0 0x7fea340c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54758/0 (socket says 172.21.15.7:54758) 2023-03-03T15:52:10.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.554+0000 7fea31d9b700 1 -- 172.21.15.7:0/3542366422 learned_addr learned my addr 172.21.15.7:0/3542366422 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:52:10.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.554+0000 7fea3259c700 1 --2- 172.21.15.7:0/3542366422 >> v2:172.21.15.143:3300/0 conn(0x7fea3406d870 0x7fea341550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:10.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.555+0000 7fea31d9b700 1 -- 172.21.15.7:0/3542366422 >> v2:172.21.15.143:3300/0 conn(0x7fea3406d870 msgr2=0x7fea341550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:52:10.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.555+0000 7fea31d9b700 1 --2- 172.21.15.7:0/3542366422 >> v2:172.21.15.143:3300/0 conn(0x7fea3406d870 0x7fea341550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.555+0000 7fea31d9b700 1 -- 172.21.15.7:0/3542366422 >> v1:172.21.15.143:6789/0 conn(0x7fea340c3e30 legacy=0x7fea34157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:10.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.555+0000 7fea31d9b700 1 -- 172.21.15.7:0/3542366422 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea341548c0 con 0x7fea340c1da0 2023-03-03T15:52:10.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.555+0000 7fea31d9b700 1 --2- 172.21.15.7:0/3542366422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea340c1da0 0x7fea340c7350 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7fea34073880 tx=0x7fea28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3054d76162c0adc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:10.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.555+0000 7fea3159a700 1 -- 172.21.15.7:0/3542366422 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28011830 con 0x7fea340c1da0 2023-03-03T15:52:10.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.555+0000 7fea3159a700 1 -- 172.21.15.7:0/3542366422 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28011990 con 0x7fea340c1da0 2023-03-03T15:52:10.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.555+0000 7fea3159a700 1 -- 172.21.15.7:0/3542366422 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28011d20 con 0x7fea340c1da0 2023-03-03T15:52:10.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.556+0000 7fea388c2700 1 -- 172.21.15.7:0/3542366422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea340c1da0 msgr2=0x7fea340c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:10.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.556+0000 7fea388c2700 1 --2- 172.21.15.7:0/3542366422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea340c1da0 0x7fea340c7350 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7fea34073880 tx=0x7fea28009bd0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.556+0000 7fea388c2700 1 -- 172.21.15.7:0/3542366422 shutdown_connections 2023-03-03T15:52:10.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.556+0000 7fea388c2700 1 --2- 172.21.15.7:0/3542366422 >> v2:172.21.15.143:3300/0 conn(0x7fea3406d870 0x7fea341550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.556+0000 7fea388c2700 1 --2- 172.21.15.7:0/3542366422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea340c1da0 0x7fea340c7350 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.556+0000 7fea388c2700 1 -- 172.21.15.7:0/3542366422 >> 172.21.15.7:0/3542366422 conn(0x7fea3406c110 msgr2=0x7fea3406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:52:10.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.556+0000 7fea388c2700 1 -- 172.21.15.7:0/3542366422 shutdown_connections 2023-03-03T15:52:10.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.556+0000 7fea388c2700 1 -- 172.21.15.7:0/3542366422 wait complete. 2023-03-03T15:52:10.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.558+0000 7fea388c2700 1 Processor -- start 2023-03-03T15:52:10.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.558+0000 7fea388c2700 1 -- start start 2023-03-03T15:52:10.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.558+0000 7fea388c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea3406d870 0x7fea341ea3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:10.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea388c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea340c1da0 0x7fea340abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:10.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea388c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea340c3e30 0x7fea340c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:10.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea388c2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea34155a50 con 0x7fea3406d870 2023-03-03T15:52:10.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea388c2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea340ac3a0 con 0x7fea340c3e30 2023-03-03T15:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea388c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea341e9f10 con 0x7fea340c1da0 2023-03-03T15:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea3259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea3406d870 0x7fea341ea3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea3259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea3406d870 0x7fea341ea3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54764/0 (socket says 172.21.15.7:54764) 2023-03-03T15:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea3259c700 1 -- 172.21.15.7:0/3400184874 learned_addr learned my addr 172.21.15.7:0/3400184874 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea3259c700 1 -- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea340c1da0 msgr2=0x7fea340abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea3259c700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea340c1da0 0x7fea340abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea3259c700 1 -- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea340c3e30 msgr2=0x7fea340c0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea3259c700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea340c3e30 0x7fea340c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea3259c700 1 -- 172.21.15.7:0/3400184874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea28009550 con 0x7fea3406d870 2023-03-03T15:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea3259c700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea3406d870 0x7fea341ea3a0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7fea2400e9e0 tx=0x7fea2400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea22ffd700 1 -- 172.21.15.7:0/3400184874 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24009710 con 0x7fea3406d870 2023-03-03T15:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea22ffd700 1 -- 172.21.15.7:0/3400184874 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2400c850 con 0x7fea3406d870 2023-03-03T15:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea388c2700 1 -- 172.21.15.7:0/3400184874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3406dc40 con 0x7fea3406d870 2023-03-03T15:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea22ffd700 1 -- 172.21.15.7:0/3400184874 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2400cbc0 con 0x7fea3406d870 2023-03-03T15:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.559+0000 7fea388c2700 1 -- 172.21.15.7:0/3400184874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea34154ba0 con 0x7fea3406d870 2023-03-03T15:52:10.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.560+0000 7fea388c2700 1 -- 172.21.15.7:0/3400184874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea340c38e0 con 0x7fea3406d870 2023-03-03T15:52:10.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.561+0000 7fea22ffd700 1 -- 172.21.15.7:0/3400184874 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fea24019030 con 0x7fea3406d870 2023-03-03T15:52:10.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.561+0000 7fea22ffd700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fea140682f0 0x7fea1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:10.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.561+0000 7fea22ffd700 1 -- 172.21.15.7:0/3400184874 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fea24086160 con 0x7fea3406d870 2023-03-03T15:52:10.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.561+0000 7fea31d9b700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fea140682f0 0x7fea1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:10.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.562+0000 7fea31d9b700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fea140682f0 0x7fea1406a7a0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fea340acd90 tx=0x7fea28010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:10.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.566+0000 7fea22ffd700 1 -- 172.21.15.7:0/3400184874 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fea24052480 con 0x7fea3406d870 2023-03-03T15:52:10.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.801+0000 7fea388c2700 1 -- 172.21.15.7:0/3400184874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea34074bb0 con 0x7fea3406d870 2023-03-03T15:52:10.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.801+0000 7fea22ffd700 1 -- 172.21.15.7:0/3400184874 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fea24017090 con 0x7fea3406d870 2023-03-03T15:52:10.803 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:52:10.803 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:52:10.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.803+0000 7fea388c2700 1 -- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fea140682f0 msgr2=0x7fea1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:10.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.803+0000 7fea388c2700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fea140682f0 0x7fea1406a7a0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fea340acd90 tx=0x7fea28010000 comp rx=0 tx=0).stop 2023-03-03T15:52:10.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.803+0000 7fea388c2700 1 -- 172.21.15.7:0/3400184874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea3406d870 msgr2=0x7fea341ea3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:10.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.803+0000 7fea388c2700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea3406d870 0x7fea341ea3a0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7fea2400e9e0 tx=0x7fea2400eda0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.804+0000 7fea388c2700 1 -- 172.21.15.7:0/3400184874 shutdown_connections 2023-03-03T15:52:10.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.804+0000 7fea388c2700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea340c3e30 0x7fea340c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.804+0000 7fea388c2700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fea140682f0 0x7fea1406a7a0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.804+0000 7fea388c2700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fea3406d870 0x7fea341ea3a0 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.804+0000 7fea388c2700 1 --2- 172.21.15.7:0/3400184874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fea340c1da0 0x7fea340abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:10.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.804+0000 7fea388c2700 1 -- 172.21.15.7:0/3400184874 >> 172.21.15.7:0/3400184874 conn(0x7fea3406c110 msgr2=0x7fea34071b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:52:10.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.804+0000 7fea388c2700 1 -- 172.21.15.7:0/3400184874 shutdown_connections 2023-03-03T15:52:10.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:10.804+0000 7fea388c2700 1 -- 172.21.15.7:0/3400184874 wait complete. 2023-03-03T15:52:10.807 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:52:10.817 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before thrashing 2023-03-03T15:52:15.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:52:15.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.957+0000 7f74f5580700 1 Processor -- start 2023-03-03T15:52:15.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.957+0000 7f74f5580700 1 -- start start 2023-03-03T15:52:15.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74f5580700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f74f00c00e0 0x7f74f00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74f5580700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f0154ff0 0x7f74f00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74f5580700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f00c7310 con 0x7f74f00caa30 2023-03-03T15:52:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74f5580700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f00bfd10 con 0x7f74f00c00e0 2023-03-03T15:52:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74f5580700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f00bfa30 con 0x7f74f0154ff0 2023-03-03T15:52:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74eeffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f74f00caa30 0x7f74f00c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48752/0 (socket says 172.21.15.7:48752) 2023-03-03T15:52:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74eeffd700 1 -- 172.21.15.7:0/3366041200 learned_addr learned my addr 172.21.15.7:0/3366041200 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:52:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74ee7fc700 1 --2- 172.21.15.7:0/3366041200 >> v2:172.21.15.143:3300/0 conn(0x7f74f00c00e0 0x7f74f00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74ef7fe700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f0154ff0 0x7f74f00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347712946 0 0) 0x7f74f00c7310 con 0x7f74f00caa30 2023-03-03T15:52:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d80035b0 con 0x7f74f00caa30 2023-03-03T15:52:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2081021740 0 0) 0x7f74d80035b0 con 0x7f74f00caa30 2023-03-03T15:52:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f0154ff0 msgr2=0x7f74f00cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:52:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74edffb700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f0154ff0 0x7f74f00cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 >> v2:172.21.15.143:3300/0 conn(0x7f74f00c00e0 msgr2=0x7f74f00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:52:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74edffb700 1 --2- 172.21.15.7:0/3366041200 >> v2:172.21.15.143:3300/0 conn(0x7f74f00c00e0 0x7f74f00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f0154510 con 0x7f74f00caa30 2023-03-03T15:52:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f74e4001670 con 0x7f74f00caa30 2023-03-03T15:52:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.958+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 >> v1:172.21.15.7:6789/0 conn(0x7f74f00caa30 legacy=0x7f74f00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74edffb700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8003a70 0x7f74d8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74edffb700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f0154ff0 0x7f74d8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74edffb700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d8004d90 0x7f74d8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f00bfd10 con 0x7f74d8004d90 2023-03-03T15:52:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f00bfa30 con 0x7f74d8003a70 2023-03-03T15:52:15.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d800ba60 con 0x7f74f0154ff0 2023-03-03T15:52:15.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74ef7fe700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d8004d90 0x7f74d8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:15.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74eeffd700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8003a70 0x7f74d8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:15.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74ee7fc700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f0154ff0 0x7f74d8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:15.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74ef7fe700 1 -- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f0154ff0 msgr2=0x7f74d8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:52:15.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74ef7fe700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f0154ff0 0x7f74d8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:15.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74ef7fe700 1 -- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8003a70 msgr2=0x7f74d8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:52:15.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74ef7fe700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8003a70 0x7f74d8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:15.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74ef7fe700 1 -- 172.21.15.7:0/3366041200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d80035b0 con 0x7f74d8004d90 2023-03-03T15:52:15.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74eeffd700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8003a70 0x7f74d8003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:52:15.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74ef7fe700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d8004d90 0x7f74d8005200 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f74e800e170 tx=0x7f74d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34c4d9b5fc7c68e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:15.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e800ec90 con 0x7f74d8004d90 2023-03-03T15:52:15.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e800f040 con 0x7f74d8004d90 2023-03-03T15:52:15.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.959+0000 7f74edffb700 1 -- 172.21.15.7:0/3366041200 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e8014480 con 0x7f74d8004d90 2023-03-03T15:52:15.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.960+0000 7f74f5580700 1 -- 172.21.15.7:0/3366041200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d8004d90 msgr2=0x7f74d8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:15.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.960+0000 7f74f5580700 1 --2- 172.21.15.7:0/3366041200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d8004d90 0x7f74d8005200 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f74e800e170 tx=0x7f74d8005880 comp rx=0 tx=0).stop 2023-03-03T15:52:15.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.960+0000 7f74ef7fe700 1 -- 172.21.15.7:0/3366041200 reap_dead start 2023-03-03T15:52:15.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.960+0000 7f74f5580700 1 -- 172.21.15.7:0/3366041200 shutdown_connections 2023-03-03T15:52:15.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.960+0000 7f74f5580700 1 -- 172.21.15.7:0/3366041200 >> 172.21.15.7:0/3366041200 conn(0x7f74f0066e40 msgr2=0x7f74f0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:52:15.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.960+0000 7f74f5580700 1 -- 172.21.15.7:0/3366041200 shutdown_connections 2023-03-03T15:52:15.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.960+0000 7f74f5580700 1 -- 172.21.15.7:0/3366041200 wait complete. 2023-03-03T15:52:15.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.962+0000 7f74f5580700 1 Processor -- start 2023-03-03T15:52:15.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.962+0000 7f74f5580700 1 -- start start 2023-03-03T15:52:15.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74f5580700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8004d90 0x7f74f00c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:15.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74f5580700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f01edbb0 0x7f74f01e0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:15.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74f5580700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74f014c8d0 0x7f74f01e1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:15.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74f5580700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f0158610 con 0x7f74f014c8d0 2023-03-03T15:52:15.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74f5580700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f01ee0b0 con 0x7f74d8004d90 2023-03-03T15:52:15.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74f5580700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f01567a0 con 0x7f74f01edbb0 2023-03-03T15:52:15.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74ef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74f014c8d0 0x7f74f01e1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:15.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74ef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74f014c8d0 0x7f74f01e1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54788/0 (socket says 172.21.15.7:54788) 2023-03-03T15:52:15.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74ef7fe700 1 -- 172.21.15.7:0/2391601961 learned_addr learned my addr 172.21.15.7:0/2391601961 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:52:15.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74eeffd700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8004d90 0x7f74f00c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:15.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74ee7fc700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f01edbb0 0x7f74f01e0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:15.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74ef7fe700 1 -- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f01edbb0 msgr2=0x7f74f01e0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:15.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74ef7fe700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f01edbb0 0x7f74f01e0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:15.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74ef7fe700 1 -- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8004d90 msgr2=0x7f74f00c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:15.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74ef7fe700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8004d90 0x7f74f00c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:15.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74ef7fe700 1 -- 172.21.15.7:0/2391601961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f0154ad0 con 0x7f74f014c8d0 2023-03-03T15:52:15.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74eeffd700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8004d90 0x7f74f00c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:52:15.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74ef7fe700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74f014c8d0 0x7f74f01e1270 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f74e800e760 tx=0x7f74e8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:15.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.963+0000 7f74d77fe700 1 -- 172.21.15.7:0/2391601961 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e8007000 con 0x7f74f014c8d0 2023-03-03T15:52:15.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.964+0000 7f74d77fe700 1 -- 172.21.15.7:0/2391601961 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e800ee80 con 0x7f74f014c8d0 2023-03-03T15:52:15.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.964+0000 7f74f5580700 1 -- 172.21.15.7:0/2391601961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f01547f0 con 0x7f74f014c8d0 2023-03-03T15:52:15.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.964+0000 7f74d77fe700 1 -- 172.21.15.7:0/2391601961 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e8004430 con 0x7f74f014c8d0 2023-03-03T15:52:15.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.964+0000 7f74f5580700 1 -- 172.21.15.7:0/2391601961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f00bf650 con 0x7f74f014c8d0 2023-03-03T15:52:15.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.965+0000 7f74f5580700 1 -- 172.21.15.7:0/2391601961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f0074bb0 con 0x7f74f014c8d0 2023-03-03T15:52:15.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.965+0000 7f74d77fe700 1 -- 172.21.15.7:0/2391601961 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f74e802e070 con 0x7f74f014c8d0 2023-03-03T15:52:15.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.965+0000 7f74d77fe700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74dc0682f0 0x7f74dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:15.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.965+0000 7f74d77fe700 1 -- 172.21.15.7:0/2391601961 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f74e8087aa0 con 0x7f74f014c8d0 2023-03-03T15:52:15.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.966+0000 7f74eeffd700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74dc0682f0 0x7f74dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:15.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.966+0000 7f74eeffd700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74dc0682f0 0x7f74dc06a7a0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f74e4001770 tx=0x7f74e4000ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:15.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:15.970+0000 7f74d77fe700 1 -- 172.21.15.7:0/2391601961 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f74e8053dc0 con 0x7f74f014c8d0 2023-03-03T15:52:16.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.206+0000 7f74f5580700 1 -- 172.21.15.7:0/2391601961 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74f01568e0 con 0x7f74f014c8d0 2023-03-03T15:52:16.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.206+0000 7f74d77fe700 1 -- 172.21.15.7:0/2391601961 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f74e80573e0 con 0x7f74f014c8d0 2023-03-03T15:52:16.208 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:52:16.208 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:52:16.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 -- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74dc0682f0 msgr2=0x7f74dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:16.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74dc0682f0 0x7f74dc06a7a0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f74e4001770 tx=0x7f74e4000ed0 comp rx=0 tx=0).stop 2023-03-03T15:52:16.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 -- 172.21.15.7:0/2391601961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74f014c8d0 msgr2=0x7f74f01e1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:16.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74f014c8d0 0x7f74f01e1270 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f74e800e760 tx=0x7f74e8009750 comp rx=0 tx=0).stop 2023-03-03T15:52:16.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 -- 172.21.15.7:0/2391601961 shutdown_connections 2023-03-03T15:52:16.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f74dc0682f0 0x7f74dc06a7a0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:16.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74f014c8d0 0x7f74f01e1270 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74d8004d90 0x7f74f00c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 --2- 172.21.15.7:0/2391601961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f74f01edbb0 0x7f74f01e0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 -- 172.21.15.7:0/2391601961 >> 172.21.15.7:0/2391601961 conn(0x7f74f0066e40 msgr2=0x7f74f01514a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:52:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 -- 172.21.15.7:0/2391601961 shutdown_connections 2023-03-03T15:52:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:16.208+0000 7f74f5580700 1 -- 172.21.15.7:0/2391601961 wait complete. 2023-03-03T15:52:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:52:16.221 INFO:tasks.mds_thrash.fs.[a]:waiting for 105 secs before thrashing 2023-03-03T15:52:20.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:52:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.025+0000 7f45d5f99700 1 Processor -- start 2023-03-03T15:52:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.025+0000 7f45d5f99700 1 -- start start 2023-03-03T15:52:21.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45d5f99700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d01530d0 0x7f45d0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:21.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45d5f99700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00c2f30 con 0x7f45d014f9d0 2023-03-03T15:52:21.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45d5f99700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00c31d0 con 0x7f45d014fda0 2023-03-03T15:52:21.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45d5f99700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d01524a0 con 0x7f45d01530d0 2023-03-03T15:52:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45cf7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f45d014f9d0 0x7f45d0071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48774/0 (socket says 172.21.15.7:48774) 2023-03-03T15:52:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45cf7fe700 1 -- 172.21.15.7:0/1427867409 learned_addr learned my addr 172.21.15.7:0/1427867409 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:52:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45cffff700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d01530d0 0x7f45d0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443202134 0 0) 0x7f45d00c2f30 con 0x7f45d014f9d0 2023-03-03T15:52:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45bc0035b0 con 0x7f45d014f9d0 2023-03-03T15:52:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349144342 0 0) 0x7f45d00c31d0 con 0x7f45d014fda0 2023-03-03T15:52:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d00c2f30 con 0x7f45d014fda0 2023-03-03T15:52:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.026+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3431274249 0 0) 0x7f45bc0035b0 con 0x7f45d014f9d0 2023-03-03T15:52:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d01530d0 msgr2=0x7f45d0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:52:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d01530d0 0x7f45d0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 >> v1:172.21.15.143:6789/0 conn(0x7f45d014fda0 legacy=0x7f45d01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d00c34b0 con 0x7f45d014f9d0 2023-03-03T15:52:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f45c0001670 con 0x7f45d014f9d0 2023-03-03T15:52:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 >> v1:172.21.15.7:6789/0 conn(0x7f45d014f9d0 legacy=0x7f45d0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:21.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45bc003c30 0x7f45bc004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45bc004850 0x7f45bc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d01530d0 0x7f45bc005320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d01524a0 con 0x7f45bc003c30 2023-03-03T15:52:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc00bc40 con 0x7f45bc004850 2023-03-03T15:52:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc00be00 con 0x7f45d01530d0 2023-03-03T15:52:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45cf7fe700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45bc003c30 0x7f45bc004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45cffff700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d01530d0 0x7f45bc005320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:21.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ceffd700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45bc004850 0x7f45bc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45cf7fe700 1 -- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d01530d0 msgr2=0x7f45bc005320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:52:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45cf7fe700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d01530d0 0x7f45bc005320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45cf7fe700 1 -- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45bc004850 msgr2=0x7f45bc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:52:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45cf7fe700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45bc004850 0x7f45bc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45cf7fe700 1 -- 172.21.15.7:0/1427867409 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc0035b0 con 0x7f45bc003c30 2023-03-03T15:52:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45cf7fe700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45bc003c30 0x7f45bc004020 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f45c000c210 tx=0x7f45bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcee811f29074098 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:21.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c000cdb0 con 0x7f45bc003c30 2023-03-03T15:52:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c0001af0 con 0x7f45bc003c30 2023-03-03T15:52:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.027+0000 7f45ce7fc700 1 -- 172.21.15.7:0/1427867409 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0005ba0 con 0x7f45bc003c30 2023-03-03T15:52:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.028+0000 7f45d5f99700 1 -- 172.21.15.7:0/1427867409 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45bc003c30 msgr2=0x7f45bc004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.028+0000 7f45d5f99700 1 --2- 172.21.15.7:0/1427867409 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45bc003c30 0x7f45bc004020 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f45c000c210 tx=0x7f45bc005880 comp rx=0 tx=0).stop 2023-03-03T15:52:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.028+0000 7f45cffff700 1 -- 172.21.15.7:0/1427867409 reap_dead start 2023-03-03T15:52:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.028+0000 7f45d5f99700 1 -- 172.21.15.7:0/1427867409 shutdown_connections 2023-03-03T15:52:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.028+0000 7f45d5f99700 1 -- 172.21.15.7:0/1427867409 >> 172.21.15.7:0/1427867409 conn(0x7f45d00c8900 msgr2=0x7f45d00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:52:21.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.028+0000 7f45d5f99700 1 -- 172.21.15.7:0/1427867409 shutdown_connections 2023-03-03T15:52:21.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.028+0000 7f45d5f99700 1 -- 172.21.15.7:0/1427867409 wait complete. 2023-03-03T15:52:21.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.029+0000 7f45d5f99700 1 Processor -- start 2023-03-03T15:52:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45d5f99700 1 -- start start 2023-03-03T15:52:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45d5f99700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45bc003c30 0x7f45d01549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45d5f99700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d00be4b0 0x7f45d006e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45d5f99700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45d014f230 0x7f45d006e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45d5f99700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00bf5c0 con 0x7f45d014f230 2023-03-03T15:52:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45d5f99700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d006c4c0 con 0x7f45bc003c30 2023-03-03T15:52:21.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45d5f99700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d006d300 con 0x7f45d00be4b0 2023-03-03T15:52:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45cffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45d014f230 0x7f45d006e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45cffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45d014f230 0x7f45d006e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54810/0 (socket says 172.21.15.7:54810) 2023-03-03T15:52:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45cffff700 1 -- 172.21.15.7:0/359678740 learned_addr learned my addr 172.21.15.7:0/359678740 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:52:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45cf7fe700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45bc003c30 0x7f45d01549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.030+0000 7f45ceffd700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d00be4b0 0x7f45d006e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45cffff700 1 -- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d00be4b0 msgr2=0x7f45d006e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:21.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45cffff700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d00be4b0 0x7f45d006e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45cffff700 1 -- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45bc003c30 msgr2=0x7f45d01549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45cffff700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45bc003c30 0x7f45d01549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45cffff700 1 -- 172.21.15.7:0/359678740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d00ac350 con 0x7f45d014f230 2023-03-03T15:52:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45ceffd700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d00be4b0 0x7f45d006e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:52:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45cffff700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45d014f230 0x7f45d006e610 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f45d00ab150 tx=0x7f45c8011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45c7fff700 1 -- 172.21.15.7:0/359678740 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c800e040 con 0x7f45d014f230 2023-03-03T15:52:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45c7fff700 1 -- 172.21.15.7:0/359678740 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c800f040 con 0x7f45d014f230 2023-03-03T15:52:21.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45d5f99700 1 -- 172.21.15.7:0/359678740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d00ab630 con 0x7f45d014f230 2023-03-03T15:52:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45c7fff700 1 -- 172.21.15.7:0/359678740 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c800fe90 con 0x7f45d014f230 2023-03-03T15:52:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.031+0000 7f45d5f99700 1 -- 172.21.15.7:0/359678740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d0071fa0 con 0x7f45d014f230 2023-03-03T15:52:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.032+0000 7f45d5f99700 1 -- 172.21.15.7:0/359678740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d0074bb0 con 0x7f45d014f230 2023-03-03T15:52:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.033+0000 7f45c7fff700 1 -- 172.21.15.7:0/359678740 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f45c8015430 con 0x7f45d014f230 2023-03-03T15:52:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.033+0000 7f45c7fff700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45b00682f0 0x7f45b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:52:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.033+0000 7f45c7fff700 1 -- 172.21.15.7:0/359678740 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f45c8086890 con 0x7f45d014f230 2023-03-03T15:52:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.033+0000 7f45cf7fe700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45b00682f0 0x7f45b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:52:21.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.034+0000 7f45cf7fe700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45b00682f0 0x7f45b006a7a0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f45c0000c00 tx=0x7f45c000a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:52:21.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.038+0000 7f45c7fff700 1 -- 172.21.15.7:0/359678740 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f45c8052b40 con 0x7f45d014f230 2023-03-03T15:52:21.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.275+0000 7f45d5f99700 1 -- 172.21.15.7:0/359678740 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45d006d440 con 0x7f45d014f230 2023-03-03T15:52:21.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.276+0000 7f45c7fff700 1 -- 172.21.15.7:0/359678740 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f45d006d440 con 0x7f45d014f230 2023-03-03T15:52:21.277 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:52:21.277 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:52:21.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 -- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45b00682f0 msgr2=0x7f45b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:21.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45b00682f0 0x7f45b006a7a0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f45c0000c00 tx=0x7f45c000a000 comp rx=0 tx=0).stop 2023-03-03T15:52:21.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 -- 172.21.15.7:0/359678740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45d014f230 msgr2=0x7f45d006e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:52:21.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45d014f230 0x7f45d006e610 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f45d00ab150 tx=0x7f45c8011f60 comp rx=0 tx=0).stop 2023-03-03T15:52:21.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 -- 172.21.15.7:0/359678740 shutdown_connections 2023-03-03T15:52:21.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45b00682f0 0x7f45b006a7a0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:21.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45d014f230 0x7f45d006e610 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:21.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f45bc003c30 0x7f45d01549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:21.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 --2- 172.21.15.7:0/359678740 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f45d00be4b0 0x7f45d006e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:52:21.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 -- 172.21.15.7:0/359678740 >> 172.21.15.7:0/359678740 conn(0x7f45d00c8900 msgr2=0x7f45d00c7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:52:21.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 -- 172.21.15.7:0/359678740 shutdown_connections 2023-03-03T15:52:21.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:52:21.278+0000 7f45d5f99700 1 -- 172.21.15.7:0/359678740 wait complete. 2023-03-03T15:52:21.281 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:52:21.291 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T15:52:39.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:52:39.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:09.515 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:09.519 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:23.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:53:23.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.430+0000 7f3a8f63a700 1 Processor -- start 2023-03-03T15:53:23.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.430+0000 7f3a8f63a700 1 -- start start 2023-03-03T15:53:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.431+0000 7f3a8f63a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3a8814b0d0 0x7f3a88071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.431+0000 7f3a8f63a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 0x7f3a88158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.431+0000 7f3a8f63a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a880a9c20 con 0x7f3a880aad20 2023-03-03T15:53:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.431+0000 7f3a8f63a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a88154f10 con 0x7f3a8814b0d0 2023-03-03T15:53:23.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.431+0000 7f3a8f63a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a880ac3c0 con 0x7f3a88071d40 2023-03-03T15:53:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a8dbd7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 0x7f3a88158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a8d3d6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3a8814b0d0 0x7f3a88071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a8cbd5700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f3a880aad20 0x7f3a88157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:52346/0 (socket says 172.21.15.7:52346) 2023-03-03T15:53:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a8cbd5700 1 -- 172.21.15.7:0/2590533915 learned_addr learned my addr 172.21.15.7:0/2590533915 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:53:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a8dbd7700 1 -- 172.21.15.7:0/2590533915 >> v2:172.21.15.143:3300/0 conn(0x7f3a8814b0d0 msgr2=0x7f3a88071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:53:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a7bfff700 1 -- 172.21.15.7:0/2590533915 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908379812 0 0) 0x7f3a880a9c20 con 0x7f3a880aad20 2023-03-03T15:53:23.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a8dbd7700 1 --2- 172.21.15.7:0/2590533915 >> v2:172.21.15.143:3300/0 conn(0x7f3a8814b0d0 0x7f3a88071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a8dbd7700 1 -- 172.21.15.7:0/2590533915 >> v1:172.21.15.143:6789/0 conn(0x7f3a880aad20 legacy=0x7f3a88157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:23.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a8dbd7700 1 -- 172.21.15.7:0/2590533915 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8806fc70 con 0x7f3a88071d40 2023-03-03T15:53:23.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a8dbd7700 1 --2- 172.21.15.7:0/2590533915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 0x7f3a88158500 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009d90 tx=0x7f3a8000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55fa101450848237 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:53:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a7bfff700 1 -- 172.21.15.7:0/2590533915 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8000e070 con 0x7f3a88071d40 2023-03-03T15:53:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a7bfff700 1 -- 172.21.15.7:0/2590533915 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8000bb20 con 0x7f3a88071d40 2023-03-03T15:53:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.432+0000 7f3a7bfff700 1 -- 172.21.15.7:0/2590533915 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a80012430 con 0x7f3a88071d40 2023-03-03T15:53:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.433+0000 7f3a8f63a700 1 -- 172.21.15.7:0/2590533915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 msgr2=0x7f3a88158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.433+0000 7f3a8f63a700 1 --2- 172.21.15.7:0/2590533915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 0x7f3a88158500 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009d90 tx=0x7f3a8000b330 comp rx=0 tx=0).stop 2023-03-03T15:53:23.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.433+0000 7f3a8f63a700 1 -- 172.21.15.7:0/2590533915 shutdown_connections 2023-03-03T15:53:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.433+0000 7f3a8f63a700 1 --2- 172.21.15.7:0/2590533915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 0x7f3a88158500 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.433+0000 7f3a8f63a700 1 --2- 172.21.15.7:0/2590533915 >> v2:172.21.15.143:3300/0 conn(0x7f3a8814b0d0 0x7f3a88071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.433+0000 7f3a8f63a700 1 -- 172.21.15.7:0/2590533915 >> 172.21.15.7:0/2590533915 conn(0x7f3a880c8900 msgr2=0x7f3a880c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:53:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.433+0000 7f3a8f63a700 1 -- 172.21.15.7:0/2590533915 shutdown_connections 2023-03-03T15:53:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.433+0000 7f3a8f63a700 1 -- 172.21.15.7:0/2590533915 wait complete. 2023-03-03T15:53:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.434+0000 7f3a8f63a700 1 Processor -- start 2023-03-03T15:53:23.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.434+0000 7f3a8f63a700 1 -- start start 2023-03-03T15:53:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8f63a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 0x7f3a88156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8f63a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a880aad20 0x7f3a881408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8f63a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a8814b0d0 0x7f3a88140e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8f63a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a88142350 con 0x7f3a8814b0d0 2023-03-03T15:53:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8f63a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a881425f0 con 0x7f3a880aad20 2023-03-03T15:53:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8f63a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a88141660 con 0x7f3a88071d40 2023-03-03T15:53:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8dbd7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a8814b0d0 0x7f3a88140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8dbd7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a8814b0d0 0x7f3a88140e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54854/0 (socket says 172.21.15.7:54854) 2023-03-03T15:53:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8dbd7700 1 -- 172.21.15.7:0/4128024737 learned_addr learned my addr 172.21.15.7:0/4128024737 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:53:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8d3d6700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 0x7f3a88156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8cbd5700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a880aad20 0x7f3a881408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8dbd7700 1 -- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 msgr2=0x7f3a88156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8dbd7700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 0x7f3a88156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8dbd7700 1 -- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a880aad20 msgr2=0x7f3a881408e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:23.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8dbd7700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a880aad20 0x7f3a881408e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8dbd7700 1 -- 172.21.15.7:0/4128024737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a80009a70 con 0x7f3a8814b0d0 2023-03-03T15:53:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.435+0000 7f3a8cbd5700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a880aad20 0x7f3a881408e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:53:23.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.436+0000 7f3a8dbd7700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a8814b0d0 0x7f3a88140e20 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f3a88071250 tx=0x7f3a80004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:53:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.436+0000 7f3a79ffb700 1 -- 172.21.15.7:0/4128024737 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a80012430 con 0x7f3a8814b0d0 2023-03-03T15:53:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.436+0000 7f3a79ffb700 1 -- 172.21.15.7:0/4128024737 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a80020070 con 0x7f3a8814b0d0 2023-03-03T15:53:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.436+0000 7f3a79ffb700 1 -- 172.21.15.7:0/4128024737 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a80004280 con 0x7f3a8814b0d0 2023-03-03T15:53:23.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.436+0000 7f3a8f63a700 1 -- 172.21.15.7:0/4128024737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8806f6b0 con 0x7f3a8814b0d0 2023-03-03T15:53:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.436+0000 7f3a8f63a700 1 -- 172.21.15.7:0/4128024737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a88155160 con 0x7f3a8814b0d0 2023-03-03T15:53:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.437+0000 7f3a8f63a700 1 -- 172.21.15.7:0/4128024737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8806f990 con 0x7f3a8814b0d0 2023-03-03T15:53:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.438+0000 7f3a79ffb700 1 -- 172.21.15.7:0/4128024737 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3a8002e070 con 0x7f3a8814b0d0 2023-03-03T15:53:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.438+0000 7f3a79ffb700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a6c0682f0 0x7f3a6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.438+0000 7f3a79ffb700 1 -- 172.21.15.7:0/4128024737 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3a80022070 con 0x7f3a8814b0d0 2023-03-03T15:53:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.438+0000 7f3a8d3d6700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a6c0682f0 0x7f3a6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.439+0000 7f3a8d3d6700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a6c0682f0 0x7f3a6c06a7a0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009ad0 tx=0x7f3a7c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:53:23.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.443+0000 7f3a79ffb700 1 -- 172.21.15.7:0/4128024737 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3a80054280 con 0x7f3a8814b0d0 2023-03-03T15:53:23.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.677+0000 7f3a8f63a700 1 -- 172.21.15.7:0/4128024737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a88074bb0 con 0x7f3a8814b0d0 2023-03-03T15:53:23.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.678+0000 7f3a79ffb700 1 -- 172.21.15.7:0/4128024737 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3a800578a0 con 0x7f3a8814b0d0 2023-03-03T15:53:23.680 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:53:23.680 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:53:23.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 -- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a6c0682f0 msgr2=0x7f3a6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:23.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a6c0682f0 0x7f3a6c06a7a0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009ad0 tx=0x7f3a7c008040 comp rx=0 tx=0).stop 2023-03-03T15:53:23.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 -- 172.21.15.7:0/4128024737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a8814b0d0 msgr2=0x7f3a88140e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:23.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a8814b0d0 0x7f3a88140e20 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f3a88071250 tx=0x7f3a80004ad0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 -- 172.21.15.7:0/4128024737 shutdown_connections 2023-03-03T15:53:23.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3a6c0682f0 0x7f3a6c06a7a0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a8814b0d0 0x7f3a88140e20 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3a88071d40 0x7f3a88156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 --2- 172.21.15.7:0/4128024737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a880aad20 0x7f3a881408e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 -- 172.21.15.7:0/4128024737 >> 172.21.15.7:0/4128024737 conn(0x7f3a880c8900 msgr2=0x7f3a88146b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:53:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 -- 172.21.15.7:0/4128024737 shutdown_connections 2023-03-03T15:53:23.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.680+0000 7f3a8f63a700 1 -- 172.21.15.7:0/4128024737 wait complete. 2023-03-03T15:53:23.684 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:53:23.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:53:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.817+0000 7f295d11e700 1 Processor -- start 2023-03-03T15:53:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.817+0000 7f295d11e700 1 -- start start 2023-03-03T15:53:23.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.817+0000 7f295d11e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2958151cd0 0x7f2958154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.817+0000 7f295d11e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f29580c2e30 0x7f2958158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.817+0000 7f295d11e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2958157ff0 con 0x7f295806c6c0 2023-03-03T15:53:23.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.817+0000 7f295d11e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295806d640 con 0x7f29580c2e30 2023-03-03T15:53:23.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.817+0000 7f295d11e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295806cda0 con 0x7f2958151cd0 2023-03-03T15:53:23.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f295759e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f295806c6c0 0x7f29580736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48842/0 (socket says 172.21.15.7:48842) 2023-03-03T15:53:23.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f295759e700 1 -- 172.21.15.7:0/1158738553 learned_addr learned my addr 172.21.15.7:0/1158738553 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:53:23.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2956d9d700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2958151cd0 0x7f2958154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f295659c700 1 --2- 172.21.15.7:0/1158738553 >> v2:172.21.15.143:3300/0 conn(0x7f29580c2e30 0x7f2958158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981812409 0 0) 0x7f2958157ff0 con 0x7f295806c6c0 2023-03-03T15:53:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29440035b0 con 0x7f295806c6c0 2023-03-03T15:53:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 628179752 0 0) 0x7f29440035b0 con 0x7f295806c6c0 2023-03-03T15:53:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2958151cd0 msgr2=0x7f2958154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:53:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2958151cd0 0x7f2958154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 >> v2:172.21.15.143:3300/0 conn(0x7f29580c2e30 msgr2=0x7f2958158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:53:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 --2- 172.21.15.7:0/1158738553 >> v2:172.21.15.143:3300/0 conn(0x7f29580c2e30 0x7f2958158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295806d080 con 0x7f295806c6c0 2023-03-03T15:53:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f29500019f0 con 0x7f295806c6c0 2023-03-03T15:53:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 >> v1:172.21.15.7:6789/0 conn(0x7f295806c6c0 legacy=0x7f29580736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2958151cd0 0x7f2944003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2944003ff0 0x7f2944004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2944004d90 0x7f2944005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295806d640 con 0x7f2944003ff0 2023-03-03T15:53:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295806cda0 con 0x7f2944004d90 2023-03-03T15:53:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294400bb70 con 0x7f2958151cd0 2023-03-03T15:53:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f295659c700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2944003ff0 0x7f2944004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.818+0000 7f2956d9d700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2958151cd0 0x7f2944003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f295759e700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2944004d90 0x7f2944005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f295659c700 1 -- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2958151cd0 msgr2=0x7f2944003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:53:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f295659c700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2958151cd0 0x7f2944003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f295659c700 1 -- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2944004d90 msgr2=0x7f2944005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:53:23.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f295659c700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2944004d90 0x7f2944005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f295659c700 1 -- 172.21.15.7:0/1158738553 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29440035b0 con 0x7f2944003ff0 2023-03-03T15:53:23.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f295659c700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2944003ff0 0x7f2944004850 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f2940009b30 tx=0x7f2944005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34871ad2d1c8732c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:53:23.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294000e6c0 con 0x7f2944003ff0 2023-03-03T15:53:23.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294000e820 con 0x7f2944003ff0 2023-03-03T15:53:23.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f2955d9b700 1 -- 172.21.15.7:0/1158738553 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294000eb90 con 0x7f2944003ff0 2023-03-03T15:53:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f295d11e700 1 -- 172.21.15.7:0/1158738553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2944003ff0 msgr2=0x7f2944004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f295d11e700 1 --2- 172.21.15.7:0/1158738553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2944003ff0 0x7f2944004850 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f2940009b30 tx=0x7f2944005880 comp rx=0 tx=0).stop 2023-03-03T15:53:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.819+0000 7f295759e700 1 -- 172.21.15.7:0/1158738553 reap_dead start 2023-03-03T15:53:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.820+0000 7f295d11e700 1 -- 172.21.15.7:0/1158738553 shutdown_connections 2023-03-03T15:53:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.820+0000 7f295d11e700 1 -- 172.21.15.7:0/1158738553 >> 172.21.15.7:0/1158738553 conn(0x7f29580a94d0 msgr2=0x7f29580a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:53:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.820+0000 7f295d11e700 1 -- 172.21.15.7:0/1158738553 shutdown_connections 2023-03-03T15:53:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.820+0000 7f295d11e700 1 -- 172.21.15.7:0/1158738553 wait complete. 2023-03-03T15:53:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.821+0000 7f295d11e700 1 Processor -- start 2023-03-03T15:53:23.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.822+0000 7f295d11e700 1 -- start start 2023-03-03T15:53:23.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.822+0000 7f295d11e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2944003ff0 0x7f29581e9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.822+0000 7f295d11e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29581ecdb0 0x7f29581501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.822+0000 7f295d11e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29580ac4c0 0x7f29580ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.822+0000 7f295d11e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29581569c0 con 0x7f29581ecdb0 2023-03-03T15:53:23.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.822+0000 7f295d11e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29581ed3b0 con 0x7f2944003ff0 2023-03-03T15:53:23.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.822+0000 7f295d11e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29581ed690 con 0x7f29580ac4c0 2023-03-03T15:53:23.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29581ecdb0 0x7f29581501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29581ecdb0 0x7f29581501f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54874/0 (socket says 172.21.15.7:54874) 2023-03-03T15:53:23.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295659c700 1 -- 172.21.15.7:0/264824179 learned_addr learned my addr 172.21.15.7:0/264824179 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:53:23.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295759e700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29580ac4c0 0x7f29580ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f2956d9d700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2944003ff0 0x7f29581e9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295659c700 1 -- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29580ac4c0 msgr2=0x7f29580ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:23.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295659c700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29580ac4c0 0x7f29580ac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295659c700 1 -- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2944003ff0 msgr2=0x7f29581e9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:23.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295659c700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2944003ff0 0x7f29581e9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:23.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295659c700 1 -- 172.21.15.7:0/264824179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2958157c80 con 0x7f29581ecdb0 2023-03-03T15:53:23.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295659c700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29581ecdb0 0x7f29581501f0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f294400bac0 tx=0x7f29400047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:53:23.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f294f7fe700 1 -- 172.21.15.7:0/264824179 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294000e6c0 con 0x7f29581ecdb0 2023-03-03T15:53:23.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f294f7fe700 1 -- 172.21.15.7:0/264824179 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2940003ab0 con 0x7f29581ecdb0 2023-03-03T15:53:23.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295d11e700 1 -- 172.21.15.7:0/264824179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295806d360 con 0x7f29581ecdb0 2023-03-03T15:53:23.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f294f7fe700 1 -- 172.21.15.7:0/264824179 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940003e60 con 0x7f29581ecdb0 2023-03-03T15:53:23.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.823+0000 7f295d11e700 1 -- 172.21.15.7:0/264824179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29581589c0 con 0x7f29581ecdb0 2023-03-03T15:53:23.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.824+0000 7f295d11e700 1 -- 172.21.15.7:0/264824179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2958074bb0 con 0x7f29581ecdb0 2023-03-03T15:53:23.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.825+0000 7f294f7fe700 1 -- 172.21.15.7:0/264824179 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2940025070 con 0x7f29581ecdb0 2023-03-03T15:53:23.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.825+0000 7f294f7fe700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29380682f0 0x7f293806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:53:23.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.826+0000 7f294f7fe700 1 -- 172.21.15.7:0/264824179 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2940087970 con 0x7f29581ecdb0 2023-03-03T15:53:23.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.826+0000 7f2956d9d700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29380682f0 0x7f293806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:53:23.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.826+0000 7f2956d9d700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29380682f0 0x7f293806a7a0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f2958072880 tx=0x7f2948009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:53:23.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:23.830+0000 7f294f7fe700 1 -- 172.21.15.7:0/264824179 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2940053d10 con 0x7f29581ecdb0 2023-03-03T15:53:24.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.066+0000 7f295d11e700 1 -- 172.21.15.7:0/264824179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29580c17b0 con 0x7f29581ecdb0 2023-03-03T15:53:24.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.067+0000 7f294f7fe700 1 -- 172.21.15.7:0/264824179 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2940057330 con 0x7f29581ecdb0 2023-03-03T15:53:24.068 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:53:24.069 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:53:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 -- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29380682f0 msgr2=0x7f293806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29380682f0 0x7f293806a7a0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f2958072880 tx=0x7f2948009210 comp rx=0 tx=0).stop 2023-03-03T15:53:24.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 -- 172.21.15.7:0/264824179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29581ecdb0 msgr2=0x7f29581501f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:53:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29581ecdb0 0x7f29581501f0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f294400bac0 tx=0x7f29400047c0 comp rx=0 tx=0).stop 2023-03-03T15:53:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 -- 172.21.15.7:0/264824179 shutdown_connections 2023-03-03T15:53:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29580ac4c0 0x7f29580ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2944003ff0 0x7f29581e9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29380682f0 0x7f293806a7a0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 --2- 172.21.15.7:0/264824179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29581ecdb0 0x7f29581501f0 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:53:24.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 -- 172.21.15.7:0/264824179 >> 172.21.15.7:0/264824179 conn(0x7f29580a94d0 msgr2=0x7f2958150ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:53:24.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 -- 172.21.15.7:0/264824179 shutdown_connections 2023-03-03T15:53:24.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:53:24.069+0000 7f295d11e700 1 -- 172.21.15.7:0/264824179 wait complete. 2023-03-03T15:53:24.072 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:53:24.082 INFO:tasks.mds_thrash.fs.[b]:waiting for 52 secs before thrashing 2023-03-03T15:53:39.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:53:39.666 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:01.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:54:01.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.361+0000 7f7823731700 1 Processor -- start 2023-03-03T15:54:01.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.361+0000 7f7823731700 1 -- start start 2023-03-03T15:54:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7823731700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 0x7f781c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7823731700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f781c06b920 0x7f781c06a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7823731700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f781c06bfe0 0x7f781c06a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7823731700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781c06f080 con 0x7f781c06bfe0 2023-03-03T15:54:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7823731700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781c1584b0 con 0x7f781c06b920 2023-03-03T15:54:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7823731700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781c0721f0 con 0x7f781c0aad90 2023-03-03T15:54:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7821cce700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f781c06bfe0 0x7f781c06a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7821cce700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f781c06bfe0 0x7f781c06a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54906/0 (socket says 172.21.15.7:54906) 2023-03-03T15:54:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7821cce700 1 -- 172.21.15.7:0/180976683 learned_addr learned my addr 172.21.15.7:0/180976683 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:54:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f78214cd700 1 --2- 172.21.15.7:0/180976683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 0x7f781c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7820ccc700 1 --2- 172.21.15.7:0/180976683 >> v2:172.21.15.143:3300/0 conn(0x7f781c06b920 0x7f781c06a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7821cce700 1 -- 172.21.15.7:0/180976683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 msgr2=0x7f781c06e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:54:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7821cce700 1 --2- 172.21.15.7:0/180976683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 0x7f781c06e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7821cce700 1 -- 172.21.15.7:0/180976683 >> v2:172.21.15.143:3300/0 conn(0x7f781c06b920 msgr2=0x7f781c06a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:54:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7821cce700 1 --2- 172.21.15.7:0/180976683 >> v2:172.21.15.143:3300/0 conn(0x7f781c06b920 0x7f781c06a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7821cce700 1 -- 172.21.15.7:0/180976683 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c06eda0 con 0x7f781c06bfe0 2023-03-03T15:54:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7820ccc700 1 --2- 172.21.15.7:0/180976683 >> v2:172.21.15.143:3300/0 conn(0x7f781c06b920 0x7f781c06a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:54:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f78214cd700 1 --2- 172.21.15.7:0/180976683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 0x7f781c06e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:54:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.362+0000 7f7821cce700 1 --2- 172.21.15.7:0/180976683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 0x7f781c06a7f0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f781c06f380 tx=0x7f781400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184cf31ee6a77e8a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:54:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7813fff700 1 -- 172.21.15.7:0/180976683 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781400e070 con 0x7f781c06bfe0 2023-03-03T15:54:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7813fff700 1 -- 172.21.15.7:0/180976683 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781400b9c0 con 0x7f781c06bfe0 2023-03-03T15:54:01.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7813fff700 1 -- 172.21.15.7:0/180976683 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781400bd70 con 0x7f781c06bfe0 2023-03-03T15:54:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7823731700 1 -- 172.21.15.7:0/180976683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 msgr2=0x7f781c06a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7823731700 1 --2- 172.21.15.7:0/180976683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 0x7f781c06a7f0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f781c06f380 tx=0x7f781400b330 comp rx=0 tx=0).stop 2023-03-03T15:54:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7823731700 1 -- 172.21.15.7:0/180976683 shutdown_connections 2023-03-03T15:54:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7823731700 1 --2- 172.21.15.7:0/180976683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 0x7f781c06a7f0 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7823731700 1 --2- 172.21.15.7:0/180976683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 0x7f781c06e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7823731700 1 --2- 172.21.15.7:0/180976683 >> v2:172.21.15.143:3300/0 conn(0x7f781c06b920 0x7f781c06a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7823731700 1 -- 172.21.15.7:0/180976683 >> 172.21.15.7:0/180976683 conn(0x7f781c0736f0 msgr2=0x7f781c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:54:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7823731700 1 -- 172.21.15.7:0/180976683 shutdown_connections 2023-03-03T15:54:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.363+0000 7f7823731700 1 -- 172.21.15.7:0/180976683 wait complete. 2023-03-03T15:54:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.364+0000 7f7823731700 1 Processor -- start 2023-03-03T15:54:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7823731700 1 -- start start 2023-03-03T15:54:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7823731700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f781c06b920 0x7f781c0c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7823731700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 0x7f781c14cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:01.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7823731700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 0x7f781c0c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:01.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7823731700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781c14d4d0 con 0x7f781c06bfe0 2023-03-03T15:54:01.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7823731700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781c155140 con 0x7f781c06b920 2023-03-03T15:54:01.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7823731700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781c0c7e30 con 0x7f781c0aad90 2023-03-03T15:54:01.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7820ccc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 0x7f781c14cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:01.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7820ccc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 0x7f781c14cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54910/0 (socket says 172.21.15.7:54910) 2023-03-03T15:54:01.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7820ccc700 1 -- 172.21.15.7:0/241373027 learned_addr learned my addr 172.21.15.7:0/241373027 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:54:01.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f78214cd700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f781c06b920 0x7f781c0c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:01.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.365+0000 7f7821cce700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 0x7f781c0c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:01.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7820ccc700 1 -- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 msgr2=0x7f781c0c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:01.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7820ccc700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 0x7f781c0c4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7820ccc700 1 -- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f781c06b920 msgr2=0x7f781c0c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:01.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7820ccc700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f781c06b920 0x7f781c0c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7820ccc700 1 -- 172.21.15.7:0/241373027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7814009b30 con 0x7f781c06bfe0 2023-03-03T15:54:01.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f78214cd700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f781c06b920 0x7f781c0c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:54:01.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7820ccc700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 0x7f781c14cf90 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f780c00d800 tx=0x7f780c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:54:01.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7811ffb700 1 -- 172.21.15.7:0/241373027 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c004030 con 0x7f781c06bfe0 2023-03-03T15:54:01.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7811ffb700 1 -- 172.21.15.7:0/241373027 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780c004190 con 0x7f781c06bfe0 2023-03-03T15:54:01.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7823731700 1 -- 172.21.15.7:0/241373027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781c158980 con 0x7f781c06bfe0 2023-03-03T15:54:01.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7811ffb700 1 -- 172.21.15.7:0/241373027 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c004520 con 0x7f781c06bfe0 2023-03-03T15:54:01.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.366+0000 7f7823731700 1 -- 172.21.15.7:0/241373027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781c1586a0 con 0x7f781c06bfe0 2023-03-03T15:54:01.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.367+0000 7f7823731700 1 -- 172.21.15.7:0/241373027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781c071cd0 con 0x7f781c06bfe0 2023-03-03T15:54:01.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.368+0000 7f7811ffb700 1 -- 172.21.15.7:0/241373027 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f780c01a030 con 0x7f781c06bfe0 2023-03-03T15:54:01.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.368+0000 7f7811ffb700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78000682f0 0x7f780006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:01.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.368+0000 7f7811ffb700 1 -- 172.21.15.7:0/241373027 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f780c085660 con 0x7f781c06bfe0 2023-03-03T15:54:01.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.368+0000 7f78214cd700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78000682f0 0x7f780006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:01.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.369+0000 7f78214cd700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78000682f0 0x7f780006a7a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f781800c960 tx=0x7f7818008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:54:01.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.372+0000 7f7811ffb700 1 -- 172.21.15.7:0/241373027 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f780c04c5e0 con 0x7f781c06bfe0 2023-03-03T15:54:01.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.614+0000 7f7823731700 1 -- 172.21.15.7:0/241373027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f781c074bb0 con 0x7f781c06bfe0 2023-03-03T15:54:01.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.614+0000 7f7811ffb700 1 -- 172.21.15.7:0/241373027 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f780c0554e0 con 0x7f781c06bfe0 2023-03-03T15:54:01.616 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:54:01.616 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:54:01.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.616+0000 7f7823731700 1 -- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78000682f0 msgr2=0x7f780006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:01.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.616+0000 7f7823731700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78000682f0 0x7f780006a7a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f781800c960 tx=0x7f7818008040 comp rx=0 tx=0).stop 2023-03-03T15:54:01.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.616+0000 7f7823731700 1 -- 172.21.15.7:0/241373027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 msgr2=0x7f781c14cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:01.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.616+0000 7f7823731700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 0x7f781c14cf90 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f780c00d800 tx=0x7f780c00db10 comp rx=0 tx=0).stop 2023-03-03T15:54:01.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.617+0000 7f7823731700 1 -- 172.21.15.7:0/241373027 shutdown_connections 2023-03-03T15:54:01.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.617+0000 7f7823731700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f781c0aad90 0x7f781c0c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.617+0000 7f7823731700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f781c06b920 0x7f781c0c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.617+0000 7f7823731700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78000682f0 0x7f780006a7a0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.617+0000 7f7823731700 1 --2- 172.21.15.7:0/241373027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f781c06bfe0 0x7f781c14cf90 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:01.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.617+0000 7f7823731700 1 -- 172.21.15.7:0/241373027 >> 172.21.15.7:0/241373027 conn(0x7f781c0736f0 msgr2=0x7f781c0bd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:54:01.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.617+0000 7f7823731700 1 -- 172.21.15.7:0/241373027 shutdown_connections 2023-03-03T15:54:01.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:01.617+0000 7f7823731700 1 -- 172.21.15.7:0/241373027 wait complete. 2023-03-03T15:54:01.620 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:54:01.630 INFO:tasks.mds_thrash.fs.[a]:waiting for 88 secs before thrashing 2023-03-03T15:54:09.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:09.795 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:16.084 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:54:16.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.220+0000 7fdf0a66a700 1 Processor -- start 2023-03-03T15:54:16.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.220+0000 7fdf0a66a700 1 -- start start 2023-03-03T15:54:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.221+0000 7fdf0a66a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdf0414fda0 0x7fdf04156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.221+0000 7fdf0a66a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040c2f30 con 0x7fdf0414f9d0 2023-03-03T15:54:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.221+0000 7fdf0a66a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040c31d0 con 0x7fdf040abe20 2023-03-03T15:54:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.221+0000 7fdf0a66a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf041524a0 con 0x7fdf0414fda0 2023-03-03T15:54:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.221+0000 7fdf03fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdf0414f9d0 0x7fdf04071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48902/0 (socket says 172.21.15.7:48902) 2023-03-03T15:54:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.221+0000 7fdf03fff700 1 -- 172.21.15.7:0/2192775099 learned_addr learned my addr 172.21.15.7:0/2192775099 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:54:16.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf037fe700 1 --2- 172.21.15.7:0/2192775099 >> v2:172.21.15.143:3300/0 conn(0x7fdf0414fda0 0x7fdf04156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:16.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720222408 0 0) 0x7fdf040c2f30 con 0x7fdf0414f9d0 2023-03-03T15:54:16.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef00035b0 con 0x7fdf0414f9d0 2023-03-03T15:54:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 241598256 0 0) 0x7fdef00035b0 con 0x7fdf0414f9d0 2023-03-03T15:54:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 >> v2:172.21.15.143:3300/0 conn(0x7fdf0414fda0 msgr2=0x7fdf04156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:54:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 --2- 172.21.15.7:0/2192775099 >> v2:172.21.15.143:3300/0 conn(0x7fdf0414fda0 0x7fdf04156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 >> v1:172.21.15.143:6789/0 conn(0x7fdf040abe20 legacy=0x7fdf041529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040c34b0 con 0x7fdf0414f9d0 2023-03-03T15:54:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fdef4001670 con 0x7fdf0414f9d0 2023-03-03T15:54:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 >> v1:172.21.15.7:6789/0 conn(0x7fdf0414f9d0 legacy=0x7fdf04071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdef0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef0004850 0x7fdef0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdef0005320 0x7fdef000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf041524a0 con 0x7fdef0003a70 2023-03-03T15:54:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef000c230 con 0x7fdef0004850 2023-03-03T15:54:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef000c3f0 con 0x7fdef0005320 2023-03-03T15:54:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf03fff700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdef0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf037fe700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef0004850 0x7fdef0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf08c07700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdef0005320 0x7fdef000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.222+0000 7fdf03fff700 1 -- 172.21.15.7:0/2192775099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdef0005320 msgr2=0x7fdef000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:54:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf03fff700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdef0005320 0x7fdef000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf03fff700 1 -- 172.21.15.7:0/2192775099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef0004850 msgr2=0x7fdef0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:54:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf03fff700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef0004850 0x7fdef0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf03fff700 1 -- 172.21.15.7:0/2192775099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef00035b0 con 0x7fdef0003a70 2023-03-03T15:54:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf08c07700 1 -- 172.21.15.7:0/2192775099 reap_dead start 2023-03-03T15:54:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf03fff700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdef0003e80 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7fdef4001050 tx=0x7fdef0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4b4d30ac483711 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:54:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef400cd80 con 0x7fdef0003a70 2023-03-03T15:54:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4005960 con 0x7fdef0003a70 2023-03-03T15:54:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf02ffd700 1 -- 172.21.15.7:0/2192775099 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4005d10 con 0x7fdef0003a70 2023-03-03T15:54:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf0a66a700 1 -- 172.21.15.7:0/2192775099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 msgr2=0x7fdef0003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf0a66a700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdef0003e80 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7fdef4001050 tx=0x7fdef0005880 comp rx=0 tx=0).stop 2023-03-03T15:54:16.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf0a66a700 1 -- 172.21.15.7:0/2192775099 shutdown_connections 2023-03-03T15:54:16.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf0a66a700 1 --2- 172.21.15.7:0/2192775099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdef0003e80 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf0a66a700 1 -- 172.21.15.7:0/2192775099 >> 172.21.15.7:0/2192775099 conn(0x7fdf040c8900 msgr2=0x7fdf040c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:54:16.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf0a66a700 1 -- 172.21.15.7:0/2192775099 shutdown_connections 2023-03-03T15:54:16.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.223+0000 7fdf0a66a700 1 -- 172.21.15.7:0/2192775099 wait complete. 2023-03-03T15:54:16.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.225+0000 7fdf0a66a700 1 Processor -- start 2023-03-03T15:54:16.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.225+0000 7fdf0a66a700 1 -- start start 2023-03-03T15:54:16.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf0a66a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdf0406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:16.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf0a66a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf041df1f0 0x7fdf0406cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf0a66a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf0406d1c0 0x7fdf041eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf0a66a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0406c4c0 con 0x7fdef0003a70 2023-03-03T15:54:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf0a66a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0406e540 con 0x7fdf041df1f0 2023-03-03T15:54:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf0a66a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0414e7d0 con 0x7fdf0406d1c0 2023-03-03T15:54:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf03fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdf0406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf03fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdf0406ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54934/0 (socket says 172.21.15.7:54934) 2023-03-03T15:54:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf03fff700 1 -- 172.21.15.7:0/1342365530 learned_addr learned my addr 172.21.15.7:0/1342365530 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:54:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf08c07700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf0406d1c0 0x7fdf041eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.226+0000 7fdf037fe700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf041df1f0 0x7fdf0406cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf03fff700 1 -- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf0406d1c0 msgr2=0x7fdf041eaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf03fff700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf0406d1c0 0x7fdf041eaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf03fff700 1 -- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf041df1f0 msgr2=0x7fdf0406cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf03fff700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf041df1f0 0x7fdf0406cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf03fff700 1 -- 172.21.15.7:0/1342365530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040ac350 con 0x7fdef0003a70 2023-03-03T15:54:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf03fff700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdf0406ef00 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c200 tx=0x7fdef4000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:54:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf00ff9700 1 -- 172.21.15.7:0/1342365530 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4005ce0 con 0x7fdef0003a70 2023-03-03T15:54:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf00ff9700 1 -- 172.21.15.7:0/1342365530 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4005e40 con 0x7fdef0003a70 2023-03-03T15:54:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf0a66a700 1 -- 172.21.15.7:0/1342365530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf040ab630 con 0x7fdef0003a70 2023-03-03T15:54:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf00ff9700 1 -- 172.21.15.7:0/1342365530 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4007310 con 0x7fdef0003a70 2023-03-03T15:54:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.227+0000 7fdf0a66a700 1 -- 172.21.15.7:0/1342365530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf040c3e10 con 0x7fdef0003a70 2023-03-03T15:54:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.228+0000 7fdf0a66a700 1 -- 172.21.15.7:0/1342365530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0406cad0 con 0x7fdef0003a70 2023-03-03T15:54:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.228+0000 7fdf00ff9700 1 -- 172.21.15.7:0/1342365530 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdef4024030 con 0x7fdef0003a70 2023-03-03T15:54:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.229+0000 7fdf00ff9700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee40682f0 0x7fdee406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:54:16.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.229+0000 7fdf00ff9700 1 -- 172.21.15.7:0/1342365530 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdef4086fc0 con 0x7fdef0003a70 2023-03-03T15:54:16.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.229+0000 7fdf037fe700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee40682f0 0x7fdee406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:54:16.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.230+0000 7fdf037fe700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee40682f0 0x7fdee406a7a0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fdf040c3b60 tx=0x7fdeec009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:54:16.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.234+0000 7fdf00ff9700 1 -- 172.21.15.7:0/1342365530 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdef4053360 con 0x7fdef0003a70 2023-03-03T15:54:16.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.474+0000 7fdf0a66a700 1 -- 172.21.15.7:0/1342365530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf04074bb0 con 0x7fdef0003a70 2023-03-03T15:54:16.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.475+0000 7fdf00ff9700 1 -- 172.21.15.7:0/1342365530 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdef4056980 con 0x7fdef0003a70 2023-03-03T15:54:16.477 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:54:16.477 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:54:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 -- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee40682f0 msgr2=0x7fdee406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee40682f0 0x7fdee406a7a0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fdf040c3b60 tx=0x7fdeec009210 comp rx=0 tx=0).stop 2023-03-03T15:54:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 -- 172.21.15.7:0/1342365530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 msgr2=0x7fdf0406ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:54:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdf0406ef00 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c200 tx=0x7fdef4000fe0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 -- 172.21.15.7:0/1342365530 shutdown_connections 2023-03-03T15:54:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf0406d1c0 0x7fdf041eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee40682f0 0x7fdee406a7a0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdef0003a70 0x7fdf0406ef00 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 --2- 172.21.15.7:0/1342365530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf041df1f0 0x7fdf0406cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:54:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 -- 172.21.15.7:0/1342365530 >> 172.21.15.7:0/1342365530 conn(0x7fdf040c8900 msgr2=0x7fdf040c6830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:54:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 -- 172.21.15.7:0/1342365530 shutdown_connections 2023-03-03T15:54:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:54:16.477+0000 7fdf0a66a700 1 -- 172.21.15.7:0/1342365530 wait complete. 2023-03-03T15:54:16.481 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:54:16.491 INFO:tasks.mds_thrash.fs.[b]:waiting for 57 secs before thrashing 2023-03-03T15:54:39.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:54:39.868 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:09.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:09.980 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:13.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:55:13.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.629+0000 7f26ad670700 1 Processor -- start 2023-03-03T15:55:13.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.629+0000 7f26ad670700 1 -- start start 2023-03-03T15:55:13.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26ad670700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26a814b340 0x7f26a8152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:13.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26ad670700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f26a81546f0 0x7f26a8154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:13.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26ad670700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a806c920 con 0x7f26a80c2790 2023-03-03T15:55:13.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26ad670700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a80abd90 con 0x7f26a81546f0 2023-03-03T15:55:13.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26ad670700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a80c1300 con 0x7f26a814b340 2023-03-03T15:55:13.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26a77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f26a81546f0 0x7f26a8154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:13.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26a67fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f26a80c2790 0x7f26a8152b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48954/0 (socket says 172.21.15.7:48954) 2023-03-03T15:55:13.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26a67fc700 1 -- 172.21.15.7:0/4015434816 learned_addr learned my addr 172.21.15.7:0/4015434816 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:55:13.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26a6ffd700 1 --2- 172.21.15.7:0/4015434816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26a814b340 0x7f26a8152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:13.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26a77fe700 1 -- 172.21.15.7:0/4015434816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26a814b340 msgr2=0x7f26a8152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:55:13.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26a77fe700 1 --2- 172.21.15.7:0/4015434816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26a814b340 0x7f26a8152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:13.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26a77fe700 1 -- 172.21.15.7:0/4015434816 >> v1:172.21.15.7:6789/0 conn(0x7f26a80c2790 legacy=0x7f26a8152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:13.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26a5ffb700 1 -- 172.21.15.7:0/4015434816 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220259446 0 0) 0x7f26a806c920 con 0x7f26a80c2790 2023-03-03T15:55:13.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.630+0000 7f26a77fe700 1 -- 172.21.15.7:0/4015434816 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a8153280 con 0x7f26a81546f0 2023-03-03T15:55:13.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26a77fe700 1 --2- 172.21.15.7:0/4015434816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a81546f0 0x7f26a8154b40 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009d90 tx=0x7f26a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e182427cd7bae8e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:13.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26a5ffb700 1 -- 172.21.15.7:0/4015434816 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a000e070 con 0x7f26a81546f0 2023-03-03T15:55:13.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26a5ffb700 1 -- 172.21.15.7:0/4015434816 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a000bb20 con 0x7f26a81546f0 2023-03-03T15:55:13.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26a5ffb700 1 -- 172.21.15.7:0/4015434816 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0012430 con 0x7f26a81546f0 2023-03-03T15:55:13.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26ad670700 1 -- 172.21.15.7:0/4015434816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a81546f0 msgr2=0x7f26a8154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:13.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26ad670700 1 --2- 172.21.15.7:0/4015434816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a81546f0 0x7f26a8154b40 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009d90 tx=0x7f26a000b330 comp rx=0 tx=0).stop 2023-03-03T15:55:13.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26ad670700 1 -- 172.21.15.7:0/4015434816 shutdown_connections 2023-03-03T15:55:13.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26ad670700 1 --2- 172.21.15.7:0/4015434816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a81546f0 0x7f26a8154b40 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:13.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26ad670700 1 --2- 172.21.15.7:0/4015434816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26a814b340 0x7f26a8152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:13.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26ad670700 1 -- 172.21.15.7:0/4015434816 >> 172.21.15.7:0/4015434816 conn(0x7f26a80c62e0 msgr2=0x7f26a80c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:55:13.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26ad670700 1 -- 172.21.15.7:0/4015434816 shutdown_connections 2023-03-03T15:55:13.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.631+0000 7f26ad670700 1 -- 172.21.15.7:0/4015434816 wait complete. 2023-03-03T15:55:13.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.633+0000 7f26ad670700 1 Processor -- start 2023-03-03T15:55:13.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.633+0000 7f26ad670700 1 -- start start 2023-03-03T15:55:13.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.633+0000 7f26ad670700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26a80c2790 0x7f26a806ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:13.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.633+0000 7f26ad670700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a814b340 0x7f26a8071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:13.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.633+0000 7f26ad670700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26a81546f0 0x7f26a80720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:13.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.633+0000 7f26ad670700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a81e8780 con 0x7f26a814b340 2023-03-03T15:55:13.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.633+0000 7f26ad670700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a81e8cb0 con 0x7f26a80c2790 2023-03-03T15:55:13.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.633+0000 7f26ad670700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a8071720 con 0x7f26a81546f0 2023-03-03T15:55:13.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.633+0000 7f26a67fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a814b340 0x7f26a8071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:13.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26a67fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a814b340 0x7f26a8071bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54980/0 (socket says 172.21.15.7:54980) 2023-03-03T15:55:13.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26a67fc700 1 -- 172.21.15.7:0/3928971874 learned_addr learned my addr 172.21.15.7:0/3928971874 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:55:13.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26a77fe700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26a81546f0 0x7f26a80720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:13.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26a6ffd700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26a80c2790 0x7f26a806ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:13.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26a67fc700 1 -- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26a81546f0 msgr2=0x7f26a80720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:13.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26a67fc700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26a81546f0 0x7f26a80720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:13.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26a67fc700 1 -- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26a80c2790 msgr2=0x7f26a806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:13.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26a67fc700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26a80c2790 0x7f26a806ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:13.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26a67fc700 1 -- 172.21.15.7:0/3928971874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a0009a70 con 0x7f26a814b340 2023-03-03T15:55:13.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26a67fc700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a814b340 0x7f26a8071bb0 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f269800c580 tx=0x7f269800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:13.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f268f7fe700 1 -- 172.21.15.7:0/3928971874 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2698004c80 con 0x7f26a814b340 2023-03-03T15:55:13.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f268f7fe700 1 -- 172.21.15.7:0/3928971874 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2698004e00 con 0x7f26a814b340 2023-03-03T15:55:13.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26ad670700 1 -- 172.21.15.7:0/3928971874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a806bff0 con 0x7f26a814b340 2023-03-03T15:55:13.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f268f7fe700 1 -- 172.21.15.7:0/3928971874 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2698007310 con 0x7f26a814b340 2023-03-03T15:55:13.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.634+0000 7f26ad670700 1 -- 172.21.15.7:0/3928971874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a806bd50 con 0x7f26a814b340 2023-03-03T15:55:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.635+0000 7f26ad670700 1 -- 172.21.15.7:0/3928971874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a806c2d0 con 0x7f26a814b340 2023-03-03T15:55:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.636+0000 7f268f7fe700 1 -- 172.21.15.7:0/3928971874 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f269801a020 con 0x7f26a814b340 2023-03-03T15:55:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.636+0000 7f268f7fe700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26940682f0 0x7f269406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.636+0000 7f268f7fe700 1 -- 172.21.15.7:0/3928971874 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2698085be0 con 0x7f26a814b340 2023-03-03T15:55:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.636+0000 7f26a6ffd700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26940682f0 0x7f269406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:13.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.637+0000 7f26a6ffd700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26940682f0 0x7f269406a7a0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f26a80c03b0 tx=0x7f269c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:13.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.641+0000 7f268f7fe700 1 -- 172.21.15.7:0/3928971874 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2698051fd0 con 0x7f26a814b340 2023-03-03T15:55:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.882+0000 7f26ad670700 1 -- 172.21.15.7:0/3928971874 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26a8074bb0 con 0x7f26a814b340 2023-03-03T15:55:13.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.882+0000 7f268f7fe700 1 -- 172.21.15.7:0/3928971874 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f26980555f0 con 0x7f26a814b340 2023-03-03T15:55:13.885 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:55:13.885 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:55:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 -- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26940682f0 msgr2=0x7f269406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26940682f0 0x7f269406a7a0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f26a80c03b0 tx=0x7f269c009210 comp rx=0 tx=0).stop 2023-03-03T15:55:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 -- 172.21.15.7:0/3928971874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a814b340 msgr2=0x7f26a8071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a814b340 0x7f26a8071bb0 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f269800c580 tx=0x7f269800c890 comp rx=0 tx=0).stop 2023-03-03T15:55:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 -- 172.21.15.7:0/3928971874 shutdown_connections 2023-03-03T15:55:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f26a81546f0 0x7f26a80720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26a80c2790 0x7f26a806ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26940682f0 0x7f269406a7a0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 --2- 172.21.15.7:0/3928971874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26a814b340 0x7f26a8071bb0 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 -- 172.21.15.7:0/3928971874 >> 172.21.15.7:0/3928971874 conn(0x7f26a80c62e0 msgr2=0x7f26a81482d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:55:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 -- 172.21.15.7:0/3928971874 shutdown_connections 2023-03-03T15:55:13.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:13.885+0000 7f26ad670700 1 -- 172.21.15.7:0/3928971874 wait complete. 2023-03-03T15:55:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:55:13.899 INFO:tasks.mds_thrash.fs.[b]:waiting for 15 secs before thrashing 2023-03-03T15:55:28.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:55:29.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.039+0000 7f8b17095700 1 Processor -- start 2023-03-03T15:55:29.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.039+0000 7f8b17095700 1 -- start start 2023-03-03T15:55:29.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.040+0000 7f8b17095700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b100c17f0 0x7f8b1006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.040+0000 7f8b17095700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b100c5190 0x7f8b1006dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.040+0000 7f8b17095700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b1014ebe0 con 0x7f8b100c6010 2023-03-03T15:55:29.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.040+0000 7f8b17095700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b10152100 con 0x7f8b100c17f0 2023-03-03T15:55:29.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.040+0000 7f8b17095700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b100c2e70 con 0x7f8b100c5190 2023-03-03T15:55:29.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.040+0000 7f8b0ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b100c17f0 0x7f8b1006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.040+0000 7f8b0ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b100c17f0 0x7f8b1006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54996/0 (socket says 172.21.15.7:54996) 2023-03-03T15:55:29.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.040+0000 7f8b0ffff700 1 -- 172.21.15.7:0/2789586043 learned_addr learned my addr 172.21.15.7:0/2789586043 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:55:29.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.040+0000 7f8b15632700 1 --2- 172.21.15.7:0/2789586043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b100c5190 0x7f8b1006dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b0ffff700 1 -- 172.21.15.7:0/2789586043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b100c5190 msgr2=0x7f8b1006dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:55:29.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b0ffff700 1 --2- 172.21.15.7:0/2789586043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b100c5190 0x7f8b1006dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b0ffff700 1 -- 172.21.15.7:0/2789586043 >> v1:172.21.15.143:6789/0 conn(0x7f8b100c6010 legacy=0x7f8b100c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:55:29.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b0ffff700 1 -- 172.21.15.7:0/2789586043 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b100aa680 con 0x7f8b100c17f0 2023-03-03T15:55:29.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b0ffff700 1 --2- 172.21.15.7:0/2789586043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c17f0 0x7f8b1006d6a0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00bb90 tx=0x7f8afc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2503aa02130ed3d8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:29.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b0f7fe700 1 -- 172.21.15.7:0/2789586043 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc00dd10 con 0x7f8b100c17f0 2023-03-03T15:55:29.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b0f7fe700 1 -- 172.21.15.7:0/2789586043 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc00de70 con 0x7f8b100c17f0 2023-03-03T15:55:29.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b0f7fe700 1 -- 172.21.15.7:0/2789586043 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc004230 con 0x7f8b100c17f0 2023-03-03T15:55:29.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b17095700 1 -- 172.21.15.7:0/2789586043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c17f0 msgr2=0x7f8b1006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:29.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b17095700 1 --2- 172.21.15.7:0/2789586043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c17f0 0x7f8b1006d6a0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00bb90 tx=0x7f8afc00bea0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.041+0000 7f8b17095700 1 -- 172.21.15.7:0/2789586043 shutdown_connections 2023-03-03T15:55:29.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.042+0000 7f8b17095700 1 --2- 172.21.15.7:0/2789586043 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b100c5190 0x7f8b1006dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.042+0000 7f8b17095700 1 --2- 172.21.15.7:0/2789586043 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c17f0 0x7f8b1006d6a0 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.042+0000 7f8b17095700 1 -- 172.21.15.7:0/2789586043 >> 172.21.15.7:0/2789586043 conn(0x7f8b100a9430 msgr2=0x7f8b100c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:55:29.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.042+0000 7f8b17095700 1 -- 172.21.15.7:0/2789586043 shutdown_connections 2023-03-03T15:55:29.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.042+0000 7f8b17095700 1 -- 172.21.15.7:0/2789586043 wait complete. 2023-03-03T15:55:29.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.043+0000 7f8b17095700 1 Processor -- start 2023-03-03T15:55:29.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.043+0000 7f8b17095700 1 -- start start 2023-03-03T15:55:29.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b17095700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b100c17f0 0x7f8b101e5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b17095700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c5190 0x7f8b100a69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b17095700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b100c6010 0x7f8b1014bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b17095700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b10153dd0 con 0x7f8b100c5190 2023-03-03T15:55:29.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b17095700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b10071770 con 0x7f8b100c17f0 2023-03-03T15:55:29.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b17095700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b101550b0 con 0x7f8b100c6010 2023-03-03T15:55:29.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b0ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c5190 0x7f8b100a69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b0ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c5190 0x7f8b100a69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55002/0 (socket says 172.21.15.7:55002) 2023-03-03T15:55:29.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b0ffff700 1 -- 172.21.15.7:0/1122158044 learned_addr learned my addr 172.21.15.7:0/1122158044 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:55:29.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b15632700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b100c6010 0x7f8b1014bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b14e31700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b100c17f0 0x7f8b101e5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b0ffff700 1 -- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b100c6010 msgr2=0x7f8b1014bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:29.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b0ffff700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b100c6010 0x7f8b1014bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b0ffff700 1 -- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b100c17f0 msgr2=0x7f8b101e5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:29.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b0ffff700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b100c17f0 0x7f8b101e5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b0ffff700 1 -- 172.21.15.7:0/1122158044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc00b7f0 con 0x7f8b100c5190 2023-03-03T15:55:29.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b0ffff700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c5190 0x7f8b100a69d0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f8b100c6900 tx=0x7f8afc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:29.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.044+0000 7f8b0d7fa700 1 -- 172.21.15.7:0/1122158044 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc020070 con 0x7f8b100c5190 2023-03-03T15:55:29.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.045+0000 7f8b0d7fa700 1 -- 172.21.15.7:0/1122158044 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8afc003970 con 0x7f8b100c5190 2023-03-03T15:55:29.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.045+0000 7f8b0d7fa700 1 -- 172.21.15.7:0/1122158044 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc003ce0 con 0x7f8b100c5190 2023-03-03T15:55:29.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.045+0000 7f8b17095700 1 -- 172.21.15.7:0/1122158044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b100c2b00 con 0x7f8b100c5190 2023-03-03T15:55:29.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.045+0000 7f8b17095700 1 -- 172.21.15.7:0/1122158044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b100aa960 con 0x7f8b100c5190 2023-03-03T15:55:29.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.046+0000 7f8b17095700 1 -- 172.21.15.7:0/1122158044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b100c32e0 con 0x7f8b100c5190 2023-03-03T15:55:29.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.046+0000 7f8b0d7fa700 1 -- 172.21.15.7:0/1122158044 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8afc010470 con 0x7f8b100c5190 2023-03-03T15:55:29.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.046+0000 7f8b0d7fa700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 0x7f8af406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.046+0000 7f8b0d7fa700 1 -- 172.21.15.7:0/1122158044 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8afc089570 con 0x7f8b100c5190 2023-03-03T15:55:29.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.047+0000 7f8b14e31700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 0x7f8af406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.047+0000 7f8b14e31700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 0x7f8af406a7a0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f8b04007fd0 tx=0x7f8b04007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:29.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.051+0000 7f8b0d7fa700 1 -- 172.21.15.7:0/1122158044 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8afc055960 con 0x7f8b100c5190 2023-03-03T15:55:29.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.287+0000 7f8b17095700 1 -- 172.21.15.7:0/1122158044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b10074bb0 con 0x7f8b100c5190 2023-03-03T15:55:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.288+0000 7f8b0d7fa700 1 -- 172.21.15.7:0/1122158044 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8afc058f80 con 0x7f8b100c5190 2023-03-03T15:55:29.290 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:55:29.290 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:55:29.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 -- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 msgr2=0x7f8af406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:29.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 0x7f8af406a7a0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f8b04007fd0 tx=0x7f8b04007ec0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 -- 172.21.15.7:0/1122158044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c5190 msgr2=0x7f8b100a69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c5190 0x7f8b100a69d0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f8b100c6900 tx=0x7f8afc004750 comp rx=0 tx=0).stop 2023-03-03T15:55:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 -- 172.21.15.7:0/1122158044 shutdown_connections 2023-03-03T15:55:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b100c6010 0x7f8b1014bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b100c17f0 0x7f8b101e5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8af40682f0 0x7f8af406a7a0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 --2- 172.21.15.7:0/1122158044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b100c5190 0x7f8b100a69d0 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 -- 172.21.15.7:0/1122158044 >> 172.21.15.7:0/1122158044 conn(0x7f8b100a9430 msgr2=0x7f8b1014ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:55:29.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 -- 172.21.15.7:0/1122158044 shutdown_connections 2023-03-03T15:55:29.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.290+0000 7f8b17095700 1 -- 172.21.15.7:0/1122158044 wait complete. 2023-03-03T15:55:29.293 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:55:29.303 INFO:tasks.mds_thrash.fs.[b]:waiting for 67 secs before thrashing 2023-03-03T15:55:29.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:55:29.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.754+0000 7f70f162d700 1 Processor -- start 2023-03-03T15:55:29.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.754+0000 7f70f162d700 1 -- start start 2023-03-03T15:55:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.755+0000 7f70f162d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec151cd0 0x7f70ec154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.755+0000 7f70f162d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f70ec06c6c0 0x7f70ec0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.755+0000 7f70f162d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ec157ff0 con 0x7f70ec0c2e30 2023-03-03T15:55:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.755+0000 7f70f162d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ec06d640 con 0x7f70ec06c6c0 2023-03-03T15:55:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.755+0000 7f70f162d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec06cda0 con 0x7f70ec151cd0 2023-03-03T15:55:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.755+0000 7f70ea7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f70ec0c2e30 0x7f70ec158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48988/0 (socket says 172.21.15.7:48988) 2023-03-03T15:55:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.755+0000 7f70ea7fc700 1 -- 172.21.15.7:0/163660296 learned_addr learned my addr 172.21.15.7:0/163660296 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:55:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.755+0000 7f70eaffd700 1 --2- 172.21.15.7:0/163660296 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec151cd0 0x7f70ec154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.755+0000 7f70eb7fe700 1 --2- 172.21.15.7:0/163660296 >> v2:172.21.15.143:3300/0 conn(0x7f70ec06c6c0 0x7f70ec0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70e9ffb700 1 -- 172.21.15.7:0/163660296 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223038477 0 0) 0x7f70ec157ff0 con 0x7f70ec0c2e30 2023-03-03T15:55:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70e9ffb700 1 -- 172.21.15.7:0/163660296 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d40035b0 con 0x7f70ec0c2e30 2023-03-03T15:55:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70eaffd700 1 -- 172.21.15.7:0/163660296 >> v2:172.21.15.143:3300/0 conn(0x7f70ec06c6c0 msgr2=0x7f70ec0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:55:29.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70eaffd700 1 --2- 172.21.15.7:0/163660296 >> v2:172.21.15.143:3300/0 conn(0x7f70ec06c6c0 0x7f70ec0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70eaffd700 1 -- 172.21.15.7:0/163660296 >> v1:172.21.15.7:6789/0 conn(0x7f70ec0c2e30 legacy=0x7f70ec158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:29.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70e9ffb700 1 -- 172.21.15.7:0/163660296 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826786372 0 0) 0x7f70d40035b0 con 0x7f70ec0c2e30 2023-03-03T15:55:29.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70eaffd700 1 -- 172.21.15.7:0/163660296 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec06d080 con 0x7f70ec151cd0 2023-03-03T15:55:29.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70eaffd700 1 --2- 172.21.15.7:0/163660296 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec151cd0 0x7f70ec154510 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009840 tx=0x7f70e0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c82cabd0b8ea6b6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:29.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70e9ffb700 1 -- 172.21.15.7:0/163660296 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc001670 con 0x7f70ec151cd0 2023-03-03T15:55:29.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70e9ffb700 1 -- 172.21.15.7:0/163660296 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e0004030 con 0x7f70ec151cd0 2023-03-03T15:55:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.756+0000 7f70e9ffb700 1 -- 172.21.15.7:0/163660296 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e00043a0 con 0x7f70ec151cd0 2023-03-03T15:55:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.757+0000 7f70f162d700 1 -- 172.21.15.7:0/163660296 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec151cd0 msgr2=0x7f70ec154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.757+0000 7f70f162d700 1 --2- 172.21.15.7:0/163660296 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec151cd0 0x7f70ec154510 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009840 tx=0x7f70e0009b50 comp rx=0 tx=0).stop 2023-03-03T15:55:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.757+0000 7f70f162d700 1 -- 172.21.15.7:0/163660296 shutdown_connections 2023-03-03T15:55:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.757+0000 7f70f162d700 1 --2- 172.21.15.7:0/163660296 >> v2:172.21.15.143:3300/0 conn(0x7f70ec06c6c0 0x7f70ec0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.757+0000 7f70f162d700 1 --2- 172.21.15.7:0/163660296 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec151cd0 0x7f70ec154510 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.757+0000 7f70f162d700 1 -- 172.21.15.7:0/163660296 >> 172.21.15.7:0/163660296 conn(0x7f70ec0a94d0 msgr2=0x7f70ec0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:55:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.757+0000 7f70f162d700 1 -- 172.21.15.7:0/163660296 shutdown_connections 2023-03-03T15:55:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.757+0000 7f70f162d700 1 -- 172.21.15.7:0/163660296 wait complete. 2023-03-03T15:55:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.758+0000 7f70f162d700 1 Processor -- start 2023-03-03T15:55:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.758+0000 7f70f162d700 1 -- start start 2023-03-03T15:55:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70f162d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70ec06c6c0 0x7f70ec071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70f162d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec0c2e30 0x7f70ec149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70f162d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ec151cd0 0x7f70ec14a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70f162d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec0aad40 con 0x7f70ec06c6c0 2023-03-03T15:55:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70f162d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec0ab990 con 0x7f70ec151cd0 2023-03-03T15:55:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70f162d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ec14a820 con 0x7f70ec0c2e30 2023-03-03T15:55:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70eaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70ec06c6c0 0x7f70ec071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70eaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70ec06c6c0 0x7f70ec071aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55012/0 (socket says 172.21.15.7:55012) 2023-03-03T15:55:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70eaffd700 1 -- 172.21.15.7:0/4056954400 learned_addr learned my addr 172.21.15.7:0/4056954400 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:55:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70eb7fe700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ec151cd0 0x7f70ec14a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70ea7fc700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec0c2e30 0x7f70ec149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70eaffd700 1 -- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec0c2e30 msgr2=0x7f70ec149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70eaffd700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec0c2e30 0x7f70ec149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70eaffd700 1 -- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ec151cd0 msgr2=0x7f70ec14a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70eaffd700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ec151cd0 0x7f70ec14a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70eaffd700 1 -- 172.21.15.7:0/4056954400 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e00094a0 con 0x7f70ec06c6c0 2023-03-03T15:55:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.759+0000 7f70eb7fe700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ec151cd0 0x7f70ec14a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:55:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.760+0000 7f70eaffd700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70ec06c6c0 0x7f70ec071aa0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009b90 tx=0x7f70e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.760+0000 7f70d37fe700 1 -- 172.21.15.7:0/4056954400 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0003df0 con 0x7f70ec06c6c0 2023-03-03T15:55:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.760+0000 7f70d37fe700 1 -- 172.21.15.7:0/4056954400 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e001f070 con 0x7f70ec06c6c0 2023-03-03T15:55:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.760+0000 7f70f162d700 1 -- 172.21.15.7:0/4056954400 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ec157c80 con 0x7f70ec06c6c0 2023-03-03T15:55:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.760+0000 7f70d37fe700 1 -- 172.21.15.7:0/4056954400 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0011870 con 0x7f70ec06c6c0 2023-03-03T15:55:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.760+0000 7f70f162d700 1 -- 172.21.15.7:0/4056954400 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ec06d360 con 0x7f70ec06c6c0 2023-03-03T15:55:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.761+0000 7f70f162d700 1 -- 172.21.15.7:0/4056954400 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ec154a50 con 0x7f70ec06c6c0 2023-03-03T15:55:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.761+0000 7f70d37fe700 1 -- 172.21.15.7:0/4056954400 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f70e002a020 con 0x7f70ec06c6c0 2023-03-03T15:55:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.762+0000 7f70d37fe700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70d80682f0 0x7f70d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.762+0000 7f70d37fe700 1 -- 172.21.15.7:0/4056954400 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f70e00875c0 con 0x7f70ec06c6c0 2023-03-03T15:55:29.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.762+0000 7f70ea7fc700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70d80682f0 0x7f70d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:29.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.762+0000 7f70ea7fc700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70d80682f0 0x7f70d806a7a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f70ec152fc0 tx=0x7f70dc001bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:29.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:29.766+0000 7f70d37fe700 1 -- 172.21.15.7:0/4056954400 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70e0053960 con 0x7f70ec06c6c0 2023-03-03T15:55:30.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.006+0000 7f70f162d700 1 -- 172.21.15.7:0/4056954400 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70ec074bb0 con 0x7f70ec06c6c0 2023-03-03T15:55:30.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.006+0000 7f70d37fe700 1 -- 172.21.15.7:0/4056954400 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f70e0056f80 con 0x7f70ec06c6c0 2023-03-03T15:55:30.007 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:55:30.008 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:55:30.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 -- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70d80682f0 msgr2=0x7f70d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:30.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70d80682f0 0x7f70d806a7a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f70ec152fc0 tx=0x7f70dc001bc0 comp rx=0 tx=0).stop 2023-03-03T15:55:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 -- 172.21.15.7:0/4056954400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70ec06c6c0 msgr2=0x7f70ec071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70ec06c6c0 0x7f70ec071aa0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009b90 tx=0x7f70e0004750 comp rx=0 tx=0).stop 2023-03-03T15:55:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 -- 172.21.15.7:0/4056954400 shutdown_connections 2023-03-03T15:55:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70ec151cd0 0x7f70ec14a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70d80682f0 0x7f70d806a7a0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70ec06c6c0 0x7f70ec071aa0 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:30.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 --2- 172.21.15.7:0/4056954400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70ec0c2e30 0x7f70ec149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 -- 172.21.15.7:0/4056954400 >> 172.21.15.7:0/4056954400 conn(0x7f70ec0a94d0 msgr2=0x7f70ec152380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:55:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 -- 172.21.15.7:0/4056954400 shutdown_connections 2023-03-03T15:55:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:30.008+0000 7f70f162d700 1 -- 172.21.15.7:0/4056954400 wait complete. 2023-03-03T15:55:30.011 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:55:30.021 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before thrashing 2023-03-03T15:55:36.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:55:36.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.159+0000 7f2a854f3700 1 Processor -- start 2023-03-03T15:55:36.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.159+0000 7f2a854f3700 1 -- start start 2023-03-03T15:55:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a854f3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a800b00b0 0x7f2a800c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a854f3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2a800c7590 0x7f2a8015ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a854f3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a800c6cf0 con 0x7f2a80151cc0 2023-03-03T15:55:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a854f3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a80152090 con 0x7f2a800b00b0 2023-03-03T15:55:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a854f3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a801548f0 con 0x7f2a800c7590 2023-03-03T15:55:36.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a7e7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a800b00b0 0x7f2a800c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a7e7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a800b00b0 0x7f2a800c7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55028/0 (socket says 172.21.15.7:55028) 2023-03-03T15:55:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a7e7fc700 1 -- 172.21.15.7:0/2296321277 learned_addr learned my addr 172.21.15.7:0/2296321277 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:55:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a7f7fe700 1 --2- 172.21.15.7:0/2296321277 >> v2:172.21.15.143:3300/0 conn(0x7f2a800c7590 0x7f2a8015ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a7e7fc700 1 -- 172.21.15.7:0/2296321277 >> v2:172.21.15.143:3300/0 conn(0x7f2a800c7590 msgr2=0x7f2a8015ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:55:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a7e7fc700 1 --2- 172.21.15.7:0/2296321277 >> v2:172.21.15.143:3300/0 conn(0x7f2a800c7590 0x7f2a8015ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a7e7fc700 1 -- 172.21.15.7:0/2296321277 >> v1:172.21.15.143:6789/0 conn(0x7f2a80151cc0 legacy=0x7f2a800ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a7e7fc700 1 -- 172.21.15.7:0/2296321277 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a80154bd0 con 0x7f2a800b00b0 2023-03-03T15:55:36.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.160+0000 7f2a7f7fe700 1 --2- 172.21.15.7:0/2296321277 >> v2:172.21.15.143:3300/0 conn(0x7f2a800c7590 0x7f2a8015ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:55:36.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a7e7fc700 1 --2- 172.21.15.7:0/2296321277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 0x7f2a800c7050 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f2a800c1700 tx=0x7f2a70009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd3b4aecaa93dd0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:36.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a7dffb700 1 -- 172.21.15.7:0/2296321277 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7000b870 con 0x7f2a800b00b0 2023-03-03T15:55:36.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a7dffb700 1 -- 172.21.15.7:0/2296321277 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7000b9d0 con 0x7f2a800b00b0 2023-03-03T15:55:36.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a7dffb700 1 -- 172.21.15.7:0/2296321277 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7000bd60 con 0x7f2a800b00b0 2023-03-03T15:55:36.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a854f3700 1 -- 172.21.15.7:0/2296321277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 msgr2=0x7f2a800c7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:36.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a854f3700 1 --2- 172.21.15.7:0/2296321277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 0x7f2a800c7050 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f2a800c1700 tx=0x7f2a70009bd0 comp rx=0 tx=0).stop 2023-03-03T15:55:36.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a854f3700 1 -- 172.21.15.7:0/2296321277 shutdown_connections 2023-03-03T15:55:36.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a854f3700 1 --2- 172.21.15.7:0/2296321277 >> v2:172.21.15.143:3300/0 conn(0x7f2a800c7590 0x7f2a8015ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:36.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a854f3700 1 --2- 172.21.15.7:0/2296321277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 0x7f2a800c7050 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:36.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a854f3700 1 -- 172.21.15.7:0/2296321277 >> 172.21.15.7:0/2296321277 conn(0x7f2a8006b980 msgr2=0x7f2a80071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:55:36.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a854f3700 1 -- 172.21.15.7:0/2296321277 shutdown_connections 2023-03-03T15:55:36.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.161+0000 7f2a854f3700 1 -- 172.21.15.7:0/2296321277 wait complete. 2023-03-03T15:55:36.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.163+0000 7f2a854f3700 1 Processor -- start 2023-03-03T15:55:36.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.163+0000 7f2a854f3700 1 -- start start 2023-03-03T15:55:36.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.163+0000 7f2a854f3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 0x7f2a8014e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:36.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.163+0000 7f2a854f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a800c7590 0x7f2a8014ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:36.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.163+0000 7f2a854f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a80151cc0 0x7f2a801490c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:36.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.163+0000 7f2a854f3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a80157000 con 0x7f2a800b00b0 2023-03-03T15:55:36.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.163+0000 7f2a854f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a800c4b00 con 0x7f2a80151cc0 2023-03-03T15:55:36.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.163+0000 7f2a854f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a80155d30 con 0x7f2a800c7590 2023-03-03T15:55:36.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.163+0000 7f2a7effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 0x7f2a8014e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:36.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a7effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 0x7f2a8014e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55032/0 (socket says 172.21.15.7:55032) 2023-03-03T15:55:36.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a7effd700 1 -- 172.21.15.7:0/2274693290 learned_addr learned my addr 172.21.15.7:0/2274693290 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:55:36.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a7e7fc700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a800c7590 0x7f2a8014ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:36.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a7f7fe700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a80151cc0 0x7f2a801490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:36.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a7effd700 1 -- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a800c7590 msgr2=0x7f2a8014ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:36.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a7effd700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a800c7590 0x7f2a8014ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:36.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a7effd700 1 -- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a80151cc0 msgr2=0x7f2a801490c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:36.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a7effd700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a80151cc0 0x7f2a801490c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:36.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a7effd700 1 -- 172.21.15.7:0/2274693290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a70009550 con 0x7f2a800b00b0 2023-03-03T15:55:36.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a7effd700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 0x7f2a8014e7f0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f2a7400c480 tx=0x7f2a7400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:36.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a677fe700 1 -- 172.21.15.7:0/2274693290 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a740052d0 con 0x7f2a800b00b0 2023-03-03T15:55:36.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a677fe700 1 -- 172.21.15.7:0/2274693290 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a74005430 con 0x7f2a800b00b0 2023-03-03T15:55:36.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a854f3700 1 -- 172.21.15.7:0/2274693290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a80155190 con 0x7f2a800b00b0 2023-03-03T15:55:36.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a677fe700 1 -- 172.21.15.7:0/2274693290 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a740038c0 con 0x7f2a800b00b0 2023-03-03T15:55:36.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.164+0000 7f2a854f3700 1 -- 172.21.15.7:0/2274693290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a80154eb0 con 0x7f2a800b00b0 2023-03-03T15:55:36.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.165+0000 7f2a854f3700 1 -- 172.21.15.7:0/2274693290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a801545a0 con 0x7f2a800b00b0 2023-03-03T15:55:36.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.166+0000 7f2a677fe700 1 -- 172.21.15.7:0/2274693290 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2a74019030 con 0x7f2a800b00b0 2023-03-03T15:55:36.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.166+0000 7f2a677fe700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a6c0682f0 0x7f2a6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:55:36.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.166+0000 7f2a677fe700 1 -- 172.21.15.7:0/2274693290 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2a740852b0 con 0x7f2a800b00b0 2023-03-03T15:55:36.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.166+0000 7f2a7e7fc700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a6c0682f0 0x7f2a6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:55:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.167+0000 7f2a7e7fc700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a6c0682f0 0x7f2a6c06a7a0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f2a800cb170 tx=0x7f2a70008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:55:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.170+0000 7f2a677fe700 1 -- 172.21.15.7:0/2274693290 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2a74051650 con 0x7f2a800b00b0 2023-03-03T15:55:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.408+0000 7f2a854f3700 1 -- 172.21.15.7:0/2274693290 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a80074bb0 con 0x7f2a800b00b0 2023-03-03T15:55:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.408+0000 7f2a677fe700 1 -- 172.21.15.7:0/2274693290 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2a74054c70 con 0x7f2a800b00b0 2023-03-03T15:55:36.410 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:55:36.410 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:55:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 -- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a6c0682f0 msgr2=0x7f2a6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a6c0682f0 0x7f2a6c06a7a0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f2a800cb170 tx=0x7f2a70008030 comp rx=0 tx=0).stop 2023-03-03T15:55:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 -- 172.21.15.7:0/2274693290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 msgr2=0x7f2a8014e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:55:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 0x7f2a8014e7f0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f2a7400c480 tx=0x7f2a7400c790 comp rx=0 tx=0).stop 2023-03-03T15:55:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 -- 172.21.15.7:0/2274693290 shutdown_connections 2023-03-03T15:55:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a80151cc0 0x7f2a801490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a6c0682f0 0x7f2a6c06a7a0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a800b00b0 0x7f2a8014e7f0 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 --2- 172.21.15.7:0/2274693290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a800c7590 0x7f2a8014ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:55:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 -- 172.21.15.7:0/2274693290 >> 172.21.15.7:0/2274693290 conn(0x7f2a8006b980 msgr2=0x7f2a8014a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:55:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 -- 172.21.15.7:0/2274693290 shutdown_connections 2023-03-03T15:55:36.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:55:36.411+0000 7f2a854f3700 1 -- 172.21.15.7:0/2274693290 wait complete. 2023-03-03T15:55:36.414 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:55:36.424 INFO:tasks.mds_thrash.fs.[a]:waiting for 107 secs before thrashing 2023-03-03T15:55:40.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:55:40.111 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:10.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:10.251 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:36.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:56:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.442+0000 7f920eefa700 1 Processor -- start 2023-03-03T15:56:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.442+0000 7f920eefa700 1 -- start start 2023-03-03T15:56:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.443+0000 7f920eefa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f920814f9d0 0x7f9208156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.443+0000 7f920eefa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f920814fda0 0x7f9208071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.443+0000 7f920eefa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92080c2f30 con 0x7f92080abe20 2023-03-03T15:56:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.443+0000 7f920eefa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92080c31d0 con 0x7f920814f9d0 2023-03-03T15:56:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.443+0000 7f920eefa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92081524a0 con 0x7f920814fda0 2023-03-03T15:56:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.443+0000 7f920cc96700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f920814f9d0 0x7f9208156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.443+0000 7f920cc96700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f920814f9d0 0x7f9208156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55070/0 (socket says 172.21.15.7:55070) 2023-03-03T15:56:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.443+0000 7f920cc96700 1 -- 172.21.15.7:0/4132807664 learned_addr learned my addr 172.21.15.7:0/4132807664 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:56:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.443+0000 7f9207fff700 1 --2- 172.21.15.7:0/4132807664 >> v2:172.21.15.143:3300/0 conn(0x7f920814fda0 0x7f9208071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.444+0000 7f920cc96700 1 -- 172.21.15.7:0/4132807664 >> v2:172.21.15.143:3300/0 conn(0x7f920814fda0 msgr2=0x7f9208071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:56:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.444+0000 7f920cc96700 1 --2- 172.21.15.7:0/4132807664 >> v2:172.21.15.143:3300/0 conn(0x7f920814fda0 0x7f9208071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.444+0000 7f920cc96700 1 -- 172.21.15.7:0/4132807664 >> v1:172.21.15.143:6789/0 conn(0x7f92080abe20 legacy=0x7f92081529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.444+0000 7f920cc96700 1 -- 172.21.15.7:0/4132807664 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92080c34b0 con 0x7f920814f9d0 2023-03-03T15:56:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.444+0000 7f920cc96700 1 --2- 172.21.15.7:0/4132807664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f920814f9d0 0x7f9208156430 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f92080709b0 tx=0x7f91fc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=762914402d3e9d98 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:56:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.444+0000 7f92077fe700 1 -- 172.21.15.7:0/4132807664 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc00b870 con 0x7f920814f9d0 2023-03-03T15:56:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.444+0000 7f92077fe700 1 -- 172.21.15.7:0/4132807664 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91fc00b9d0 con 0x7f920814f9d0 2023-03-03T15:56:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.444+0000 7f92077fe700 1 -- 172.21.15.7:0/4132807664 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc00bd60 con 0x7f920814f9d0 2023-03-03T15:56:36.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.445+0000 7f920eefa700 1 -- 172.21.15.7:0/4132807664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f920814f9d0 msgr2=0x7f9208156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:36.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.445+0000 7f920eefa700 1 --2- 172.21.15.7:0/4132807664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f920814f9d0 0x7f9208156430 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f92080709b0 tx=0x7f91fc009bd0 comp rx=0 tx=0).stop 2023-03-03T15:56:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.445+0000 7f920eefa700 1 -- 172.21.15.7:0/4132807664 shutdown_connections 2023-03-03T15:56:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.445+0000 7f920eefa700 1 --2- 172.21.15.7:0/4132807664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f920814f9d0 0x7f9208156430 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.445+0000 7f920eefa700 1 --2- 172.21.15.7:0/4132807664 >> v2:172.21.15.143:3300/0 conn(0x7f920814fda0 0x7f9208071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.445+0000 7f920eefa700 1 -- 172.21.15.7:0/4132807664 >> 172.21.15.7:0/4132807664 conn(0x7f92080c8900 msgr2=0x7f92080c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:56:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.445+0000 7f920eefa700 1 -- 172.21.15.7:0/4132807664 shutdown_connections 2023-03-03T15:56:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.445+0000 7f920eefa700 1 -- 172.21.15.7:0/4132807664 wait complete. 2023-03-03T15:56:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.446+0000 7f920eefa700 1 Processor -- start 2023-03-03T15:56:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.446+0000 7f920eefa700 1 -- start start 2023-03-03T15:56:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920eefa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92080abe20 0x7f920806f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920eefa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f920814f9d0 0x7f9208143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920eefa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f920814fda0 0x7f92081528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920eefa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920806e920 con 0x7f92080abe20 2023-03-03T15:56:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920eefa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920806e2e0 con 0x7f920814fda0 2023-03-03T15:56:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920eefa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920806a350 con 0x7f920814f9d0 2023-03-03T15:56:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920cc96700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92080abe20 0x7f920806f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920cc96700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92080abe20 0x7f920806f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55076/0 (socket says 172.21.15.7:55076) 2023-03-03T15:56:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920cc96700 1 -- 172.21.15.7:0/2561395641 learned_addr learned my addr 172.21.15.7:0/2561395641 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:56:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f9207fff700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f920814f9d0 0x7f9208143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920d497700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f920814fda0 0x7f92081528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920cc96700 1 -- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f920814f9d0 msgr2=0x7f9208143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920cc96700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f920814f9d0 0x7f9208143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920cc96700 1 -- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f920814fda0 msgr2=0x7f92081528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920cc96700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f920814fda0 0x7f92081528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920cc96700 1 -- 172.21.15.7:0/2561395641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91fc009550 con 0x7f92080abe20 2023-03-03T15:56:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f9207fff700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f920814f9d0 0x7f9208143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:56:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f920cc96700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92080abe20 0x7f920806f720 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f91fc000c00 tx=0x7f91fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:56:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.447+0000 7f92057fa700 1 -- 172.21.15.7:0/2561395641 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc004540 con 0x7f92080abe20 2023-03-03T15:56:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.448+0000 7f92057fa700 1 -- 172.21.15.7:0/2561395641 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91fc0036a0 con 0x7f92080abe20 2023-03-03T15:56:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.448+0000 7f920eefa700 1 -- 172.21.15.7:0/2561395641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92080ac350 con 0x7f92080abe20 2023-03-03T15:56:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.448+0000 7f92057fa700 1 -- 172.21.15.7:0/2561395641 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91fc003a30 con 0x7f92080abe20 2023-03-03T15:56:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.448+0000 7f920eefa700 1 -- 172.21.15.7:0/2561395641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92080ab630 con 0x7f92080abe20 2023-03-03T15:56:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.449+0000 7f920eefa700 1 -- 172.21.15.7:0/2561395641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92080c3e10 con 0x7f92080abe20 2023-03-03T15:56:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.449+0000 7f92057fa700 1 -- 172.21.15.7:0/2561395641 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f91fc00f050 con 0x7f92080abe20 2023-03-03T15:56:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.449+0000 7f92057fa700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91ec0682f0 0x7f91ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:56:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.449+0000 7f92057fa700 1 -- 172.21.15.7:0/2561395641 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f91fc087ab0 con 0x7f92080abe20 2023-03-03T15:56:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.450+0000 7f9207fff700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91ec0682f0 0x7f91ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:56:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.450+0000 7f9207fff700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91ec0682f0 0x7f91ec06a7a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f920806c020 tx=0x7f91f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:56:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.454+0000 7f92057fa700 1 -- 172.21.15.7:0/2561395641 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f91fc053d60 con 0x7f92080abe20 2023-03-03T15:56:36.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.693+0000 7f920eefa700 1 -- 172.21.15.7:0/2561395641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9208074bb0 con 0x7f92080abe20 2023-03-03T15:56:36.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.694+0000 7f92057fa700 1 -- 172.21.15.7:0/2561395641 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f91fc057380 con 0x7f92080abe20 2023-03-03T15:56:36.695 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:56:36.695 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:56:36.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 -- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91ec0682f0 msgr2=0x7f91ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:36.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91ec0682f0 0x7f91ec06a7a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f920806c020 tx=0x7f91f4008040 comp rx=0 tx=0).stop 2023-03-03T15:56:36.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 -- 172.21.15.7:0/2561395641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92080abe20 msgr2=0x7f920806f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:56:36.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92080abe20 0x7f920806f720 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f91fc000c00 tx=0x7f91fc004750 comp rx=0 tx=0).stop 2023-03-03T15:56:36.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 -- 172.21.15.7:0/2561395641 shutdown_connections 2023-03-03T15:56:36.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f920814fda0 0x7f92081528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:36.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f91ec0682f0 0x7f91ec06a7a0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:36.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92080abe20 0x7f920806f720 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:36.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 --2- 172.21.15.7:0/2561395641 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f920814f9d0 0x7f9208143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:56:36.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 -- 172.21.15.7:0/2561395641 >> 172.21.15.7:0/2561395641 conn(0x7f92080c8900 msgr2=0x7f9208151af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:56:36.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 -- 172.21.15.7:0/2561395641 shutdown_connections 2023-03-03T15:56:36.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:56:36.696+0000 7f920eefa700 1 -- 172.21.15.7:0/2561395641 wait complete. 2023-03-03T15:56:36.699 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:56:36.712 INFO:tasks.mds_thrash.fs.[b]:waiting for 54 secs before thrashing 2023-03-03T15:56:40.400 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:56:40.425 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:10.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:10.484 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:23.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:57:23.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.563+0000 7fde170f5700 1 Processor -- start 2023-03-03T15:57:23.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.563+0000 7fde170f5700 1 -- start start 2023-03-03T15:57:23.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde170f5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1006c480 0x7fde1006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:23.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde170f5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde10157e50 0x7fde10158270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:23.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde170f5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1006c9b0 con 0x7fde10154fd0 2023-03-03T15:57:23.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde170f5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde10154570 con 0x7fde10157e50 2023-03-03T15:57:23.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde170f5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1006cc20 con 0x7fde1006c480 2023-03-03T15:57:23.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde0ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1006c480 0x7fde1006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:23.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde15692700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde10157e50 0x7fde10158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:23.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde0ffff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1006c480 0x7fde1006bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:34954/0 (socket says 172.21.15.7:34954) 2023-03-03T15:57:23.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde0ffff700 1 -- 172.21.15.7:0/2741043085 learned_addr learned my addr 172.21.15.7:0/2741043085 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:57:23.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde15692700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde10157e50 0x7fde10158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:50890/0 (socket says 172.21.15.7:50890) 2023-03-03T15:57:23.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde14e91700 1 --1- 172.21.15.7:0/2741043085 >> v1:172.21.15.143:6789/0 conn(0x7fde10154fd0 0x7fde100736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:52612/0 (socket says 172.21.15.7:52612) 2023-03-03T15:57:23.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde0ffff700 1 -- 172.21.15.7:0/2741043085 >> v2:172.21.15.143:3300/0 conn(0x7fde10157e50 msgr2=0x7fde10158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:57:23.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde0ffff700 1 --2- 172.21.15.7:0/2741043085 >> v2:172.21.15.143:3300/0 conn(0x7fde10157e50 0x7fde10158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde0ffff700 1 -- 172.21.15.7:0/2741043085 >> v1:172.21.15.143:6789/0 conn(0x7fde10154fd0 legacy=0x7fde100736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:23.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.564+0000 7fde0ffff700 1 -- 172.21.15.7:0/2741043085 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1006cf00 con 0x7fde1006c480 2023-03-03T15:57:23.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde15692700 1 --2- 172.21.15.7:0/2741043085 >> v2:172.21.15.143:3300/0 conn(0x7fde10157e50 0x7fde10158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:57:23.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde0ffff700 1 --2- 172.21.15.7:0/2741043085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1006c480 0x7fde1006bea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0098f0 tx=0x7fddfc009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59741e0fc6f442fb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:23.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde0f7fe700 1 -- 172.21.15.7:0/2741043085 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc011830 con 0x7fde1006c480 2023-03-03T15:57:23.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde0f7fe700 1 -- 172.21.15.7:0/2741043085 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc011990 con 0x7fde1006c480 2023-03-03T15:57:23.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde0f7fe700 1 -- 172.21.15.7:0/2741043085 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc011d20 con 0x7fde1006c480 2023-03-03T15:57:23.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde170f5700 1 -- 172.21.15.7:0/2741043085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1006c480 msgr2=0x7fde1006bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:23.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde170f5700 1 --2- 172.21.15.7:0/2741043085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1006c480 0x7fde1006bea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0098f0 tx=0x7fddfc009c00 comp rx=0 tx=0).stop 2023-03-03T15:57:23.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde170f5700 1 -- 172.21.15.7:0/2741043085 shutdown_connections 2023-03-03T15:57:23.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde170f5700 1 --2- 172.21.15.7:0/2741043085 >> v2:172.21.15.143:3300/0 conn(0x7fde10157e50 0x7fde10158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde170f5700 1 --2- 172.21.15.7:0/2741043085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde1006c480 0x7fde1006bea0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.565+0000 7fde170f5700 1 -- 172.21.15.7:0/2741043085 >> 172.21.15.7:0/2741043085 conn(0x7fde100a94d0 msgr2=0x7fde100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:57:23.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.566+0000 7fde170f5700 1 -- 172.21.15.7:0/2741043085 shutdown_connections 2023-03-03T15:57:23.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.566+0000 7fde170f5700 1 -- 172.21.15.7:0/2741043085 wait complete. 2023-03-03T15:57:23.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.567+0000 7fde170f5700 1 Processor -- start 2023-03-03T15:57:23.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.567+0000 7fde170f5700 1 -- start start 2023-03-03T15:57:23.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde170f5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1006c480 0x7fde100c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:23.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde170f5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde10154fd0 0x7fde101536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:23.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde170f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde10157e50 0x7fde100c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:23.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde170f5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1014ff30 con 0x7fde10157e50 2023-03-03T15:57:23.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde170f5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde101ee450 con 0x7fde1006c480 2023-03-03T15:57:23.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde170f5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde10155b10 con 0x7fde10154fd0 2023-03-03T15:57:23.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde15692700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde10157e50 0x7fde100c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:23.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde15692700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde10157e50 0x7fde100c0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55122/0 (socket says 172.21.15.7:55122) 2023-03-03T15:57:23.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde15692700 1 -- 172.21.15.7:0/1794951932 learned_addr learned my addr 172.21.15.7:0/1794951932 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:57:23.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde14e91700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1006c480 0x7fde100c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:23.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde0ffff700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde10154fd0 0x7fde101536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:23.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde15692700 1 -- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde10154fd0 msgr2=0x7fde101536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:23.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde15692700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde10154fd0 0x7fde101536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde15692700 1 -- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1006c480 msgr2=0x7fde100c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:23.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde15692700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1006c480 0x7fde100c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde15692700 1 -- 172.21.15.7:0/1794951932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde08009a70 con 0x7fde10157e50 2023-03-03T15:57:23.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde0ffff700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde10154fd0 0x7fde101536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:57:23.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde15692700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde10157e50 0x7fde100c0e50 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7fde08009e60 tx=0x7fde0800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:23.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde0d7fa700 1 -- 172.21.15.7:0/1794951932 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08009cd0 con 0x7fde10157e50 2023-03-03T15:57:23.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde0d7fa700 1 -- 172.21.15.7:0/1794951932 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0800f040 con 0x7fde10157e50 2023-03-03T15:57:23.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde170f5700 1 -- 172.21.15.7:0/1794951932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddfc009550 con 0x7fde10157e50 2023-03-03T15:57:23.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.568+0000 7fde0d7fa700 1 -- 172.21.15.7:0/1794951932 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0800ccb0 con 0x7fde10157e50 2023-03-03T15:57:23.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.569+0000 7fde170f5700 1 -- 172.21.15.7:0/1794951932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1006d4c0 con 0x7fde10157e50 2023-03-03T15:57:23.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.569+0000 7fde170f5700 1 -- 172.21.15.7:0/1794951932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1006d1e0 con 0x7fde10157e50 2023-03-03T15:57:23.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.570+0000 7fde0d7fa700 1 -- 172.21.15.7:0/1794951932 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fde08014480 con 0x7fde10157e50 2023-03-03T15:57:23.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.570+0000 7fde0d7fa700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddf40682f0 0x7fddf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:23.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.571+0000 7fde0d7fa700 1 -- 172.21.15.7:0/1794951932 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fde08086a00 con 0x7fde10157e50 2023-03-03T15:57:23.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.571+0000 7fde14e91700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddf40682f0 0x7fddf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:23.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.571+0000 7fde14e91700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddf40682f0 0x7fddf406a7a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fde04000f60 tx=0x7fde04007ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:23.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.575+0000 7fde0d7fa700 1 -- 172.21.15.7:0/1794951932 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fde08052bd0 con 0x7fde10157e50 2023-03-03T15:57:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.814+0000 7fde170f5700 1 -- 172.21.15.7:0/1794951932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde1006d7a0 con 0x7fde10157e50 2023-03-03T15:57:23.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.815+0000 7fde0d7fa700 1 -- 172.21.15.7:0/1794951932 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fde080561f0 con 0x7fde10157e50 2023-03-03T15:57:23.816 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:57:23.817 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:57:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 -- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddf40682f0 msgr2=0x7fddf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddf40682f0 0x7fddf406a7a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fde04000f60 tx=0x7fde04007ed0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 -- 172.21.15.7:0/1794951932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde10157e50 msgr2=0x7fde100c0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde10157e50 0x7fde100c0e50 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7fde08009e60 tx=0x7fde0800c3a0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 -- 172.21.15.7:0/1794951932 shutdown_connections 2023-03-03T15:57:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddf40682f0 0x7fddf406a7a0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde10157e50 0x7fde100c0e50 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde1006c480 0x7fde100c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 --2- 172.21.15.7:0/1794951932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde10154fd0 0x7fde101536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 -- 172.21.15.7:0/1794951932 >> 172.21.15.7:0/1794951932 conn(0x7fde100a94d0 msgr2=0x7fde100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:57:23.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 -- 172.21.15.7:0/1794951932 shutdown_connections 2023-03-03T15:57:23.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:23.817+0000 7fde170f5700 1 -- 172.21.15.7:0/1794951932 wait complete. 2023-03-03T15:57:23.820 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:57:23.830 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2023-03-03T15:57:30.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:57:30.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.845+0000 7fe3d1dd8700 1 Processor -- start 2023-03-03T15:57:30.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.845+0000 7fe3d1dd8700 1 -- start start 2023-03-03T15:57:30.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3d1dd8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe3cc154f10 0x7fe3cc158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:30.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3d1dd8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe3cc0aa010 0x7fe3cc158540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:30.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3d1dd8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3cc151ef0 con 0x7fe3cc151b20 2023-03-03T15:57:30.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3d1dd8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc157b20 con 0x7fe3cc0aa010 2023-03-03T15:57:30.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3d1dd8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc155410 con 0x7fe3cc154f10 2023-03-03T15:57:30.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3cbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe3cc0aa010 0x7fe3cc158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:30.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3cb7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe3cc151b20 0x7fe3cc071820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49108/0 (socket says 172.21.15.7:49108) 2023-03-03T15:57:30.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3cb7fe700 1 -- 172.21.15.7:0/3319484202 learned_addr learned my addr 172.21.15.7:0/3319484202 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:57:30.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3caffd700 1 --2- 172.21.15.7:0/3319484202 >> v2:172.21.15.143:3300/0 conn(0x7fe3cc154f10 0x7fe3cc158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:30.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3cbfff700 1 -- 172.21.15.7:0/3319484202 >> v2:172.21.15.143:3300/0 conn(0x7fe3cc154f10 msgr2=0x7fe3cc158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:57:30.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3cbfff700 1 --2- 172.21.15.7:0/3319484202 >> v2:172.21.15.143:3300/0 conn(0x7fe3cc154f10 0x7fe3cc158000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:30.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3cbfff700 1 -- 172.21.15.7:0/3319484202 >> v1:172.21.15.7:6789/0 conn(0x7fe3cc151b20 legacy=0x7fe3cc071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:30.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3ca7fc700 1 -- 172.21.15.7:0/3319484202 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921671178 0 0) 0x7fe3cc151ef0 con 0x7fe3cc151b20 2023-03-03T15:57:30.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.846+0000 7fe3cbfff700 1 -- 172.21.15.7:0/3319484202 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc06f640 con 0x7fe3cc0aa010 2023-03-03T15:57:30.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3cbfff700 1 --2- 172.21.15.7:0/3319484202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc0aa010 0x7fe3cc158540 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc070850 tx=0x7fe3c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9a6abb1d44a3ff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:30.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3ca7fc700 1 -- 172.21.15.7:0/3319484202 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400e070 con 0x7fe3cc0aa010 2023-03-03T15:57:30.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3ca7fc700 1 -- 172.21.15.7:0/3319484202 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c400bbd0 con 0x7fe3cc0aa010 2023-03-03T15:57:30.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3ca7fc700 1 -- 172.21.15.7:0/3319484202 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4012430 con 0x7fe3cc0aa010 2023-03-03T15:57:30.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3319484202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc0aa010 msgr2=0x7fe3cc158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:30.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3d1dd8700 1 --2- 172.21.15.7:0/3319484202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc0aa010 0x7fe3cc158540 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc070850 tx=0x7fe3c400b330 comp rx=0 tx=0).stop 2023-03-03T15:57:30.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3319484202 shutdown_connections 2023-03-03T15:57:30.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3d1dd8700 1 --2- 172.21.15.7:0/3319484202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc0aa010 0x7fe3cc158540 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:30.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3d1dd8700 1 --2- 172.21.15.7:0/3319484202 >> v2:172.21.15.143:3300/0 conn(0x7fe3cc154f10 0x7fe3cc158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3319484202 >> 172.21.15.7:0/3319484202 conn(0x7fe3cc0c8900 msgr2=0x7fe3cc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:57:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3319484202 shutdown_connections 2023-03-03T15:57:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.847+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3319484202 wait complete. 2023-03-03T15:57:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.848+0000 7fe3d1dd8700 1 Processor -- start 2023-03-03T15:57:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.849+0000 7fe3d1dd8700 1 -- start start 2023-03-03T15:57:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.849+0000 7fe3d1dd8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3cc0aa010 0x7fe3cc14bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.849+0000 7fe3d1dd8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3cc151b20 0x7fe3cc150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.849+0000 7fe3d1dd8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc154f10 0x7fe3cc1ea240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.849+0000 7fe3d1dd8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc14dd90 con 0x7fe3cc154f10 2023-03-03T15:57:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.849+0000 7fe3d1dd8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc14d7d0 con 0x7fe3cc151b20 2023-03-03T15:57:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.849+0000 7fe3d1dd8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc14dab0 con 0x7fe3cc0aa010 2023-03-03T15:57:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc154f10 0x7fe3cc1ea240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc154f10 0x7fe3cc1ea240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55136/0 (socket says 172.21.15.7:55136) 2023-03-03T15:57:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cbfff700 1 -- 172.21.15.7:0/3553044160 learned_addr learned my addr 172.21.15.7:0/3553044160 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:57:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cb7fe700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3cc0aa010 0x7fe3cc14bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3caffd700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3cc151b20 0x7fe3cc150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cbfff700 1 -- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3cc0aa010 msgr2=0x7fe3cc14bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cbfff700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3cc0aa010 0x7fe3cc14bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cbfff700 1 -- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3cc151b20 msgr2=0x7fe3cc150d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cbfff700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3cc151b20 0x7fe3cc150d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cbfff700 1 -- 172.21.15.7:0/3553044160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4009a70 con 0x7fe3cc154f10 2023-03-03T15:57:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3caffd700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3cc151b20 0x7fe3cc150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:57:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cb7fe700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3cc0aa010 0x7fe3cc14bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:57:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3cbfff700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc154f10 0x7fe3cc1ea240 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4014040 tx=0x7fe3c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3c3fff700 1 -- 172.21.15.7:0/3553044160 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4012430 con 0x7fe3cc154f10 2023-03-03T15:57:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3c3fff700 1 -- 172.21.15.7:0/3553044160 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c4020070 con 0x7fe3cc154f10 2023-03-03T15:57:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3c3fff700 1 -- 172.21.15.7:0/3553044160 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4004280 con 0x7fe3cc154f10 2023-03-03T15:57:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3553044160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc06fc00 con 0x7fe3cc154f10 2023-03-03T15:57:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.850+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3553044160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc06f920 con 0x7fe3cc154f10 2023-03-03T15:57:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.851+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3553044160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc071f30 con 0x7fe3cc154f10 2023-03-03T15:57:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.852+0000 7fe3c3fff700 1 -- 172.21.15.7:0/3553044160 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe3c402e070 con 0x7fe3cc154f10 2023-03-03T15:57:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.852+0000 7fe3c3fff700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3ac068320 0x7fe3ac06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.852+0000 7fe3c3fff700 1 -- 172.21.15.7:0/3553044160 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe3c4022070 con 0x7fe3cc154f10 2023-03-03T15:57:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.853+0000 7fe3cb7fe700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3ac068320 0x7fe3ac06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.853+0000 7fe3cb7fe700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3ac068320 0x7fe3ac06a7d0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc000f60 tx=0x7fe3bc007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:30.857+0000 7fe3c3fff700 1 -- 172.21.15.7:0/3553044160 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe3c4054380 con 0x7fe3cc154f10 2023-03-03T15:57:31.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.095+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3553044160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3cc074bb0 con 0x7fe3cc154f10 2023-03-03T15:57:31.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.096+0000 7fe3c3fff700 1 -- 172.21.15.7:0/3553044160 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe3c40579a0 con 0x7fe3cc154f10 2023-03-03T15:57:31.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:57:31.098 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:57:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3ac068320 msgr2=0x7fe3ac06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3ac068320 0x7fe3ac06a7d0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc000f60 tx=0x7fe3bc007ca0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3553044160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc154f10 msgr2=0x7fe3cc1ea240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc154f10 0x7fe3cc1ea240 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4014040 tx=0x7fe3c4004ad0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3553044160 shutdown_connections 2023-03-03T15:57:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3ac068320 0x7fe3ac06a7d0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3cc154f10 0x7fe3cc1ea240 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3cc0aa010 0x7fe3cc14bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 --2- 172.21.15.7:0/3553044160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3cc151b20 0x7fe3cc150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3553044160 >> 172.21.15.7:0/3553044160 conn(0x7fe3cc0c8900 msgr2=0x7fe3cc0c39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:57:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3553044160 shutdown_connections 2023-03-03T15:57:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.098+0000 7fe3d1dd8700 1 -- 172.21.15.7:0/3553044160 wait complete. 2023-03-03T15:57:31.102 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:57:31.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:57:31.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.234+0000 7f8d8253a700 1 Processor -- start 2023-03-03T15:57:31.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.234+0000 7f8d8253a700 1 -- start start 2023-03-03T15:57:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d8253a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c151ab0 0x7f8d7c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d8253a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8d7c073c30 0x7f8d7c06bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d8253a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8d7c06c3e0 0x7f8d7c157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d8253a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7c06d780 con 0x7f8d7c073c30 2023-03-03T15:57:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d8253a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7c06cb70 con 0x7f8d7c06c3e0 2023-03-03T15:57:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d8253a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c06ce50 con 0x7f8d7c151ab0 2023-03-03T15:57:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d7b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8d7c073c30 0x7f8d7c06bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d7b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8d7c073c30 0x7f8d7c06bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55142/0 (socket says 172.21.15.7:55142) 2023-03-03T15:57:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d7b7fe700 1 -- 172.21.15.7:0/3371816885 learned_addr learned my addr 172.21.15.7:0/3371816885 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:57:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d7bfff700 1 --2- 172.21.15.7:0/3371816885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c151ab0 0x7f8d7c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.235+0000 7f8d80ad7700 1 --2- 172.21.15.7:0/3371816885 >> v2:172.21.15.143:3300/0 conn(0x7f8d7c06c3e0 0x7f8d7c157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d7b7fe700 1 -- 172.21.15.7:0/3371816885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c151ab0 msgr2=0x7f8d7c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:57:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d7b7fe700 1 --2- 172.21.15.7:0/3371816885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c151ab0 0x7f8d7c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d7b7fe700 1 -- 172.21.15.7:0/3371816885 >> v2:172.21.15.143:3300/0 conn(0x7f8d7c06c3e0 msgr2=0x7f8d7c157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:57:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d7b7fe700 1 --2- 172.21.15.7:0/3371816885 >> v2:172.21.15.143:3300/0 conn(0x7f8d7c06c3e0 0x7f8d7c157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d7b7fe700 1 -- 172.21.15.7:0/3371816885 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c06d130 con 0x7f8d7c073c30 2023-03-03T15:57:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d80ad7700 1 --2- 172.21.15.7:0/3371816885 >> v2:172.21.15.143:3300/0 conn(0x7f8d7c06c3e0 0x7f8d7c157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T15:57:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d7b7fe700 1 --2- 172.21.15.7:0/3371816885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 0x7f8d7c06bea0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0c3db0 tx=0x7f8d70009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cfe34e596af45f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d7affd700 1 -- 172.21.15.7:0/3371816885 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7000b870 con 0x7f8d7c073c30 2023-03-03T15:57:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d7affd700 1 -- 172.21.15.7:0/3371816885 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7000b9d0 con 0x7f8d7c073c30 2023-03-03T15:57:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d7affd700 1 -- 172.21.15.7:0/3371816885 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7000bd60 con 0x7f8d7c073c30 2023-03-03T15:57:31.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d8253a700 1 -- 172.21.15.7:0/3371816885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 msgr2=0x7f8d7c06bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:31.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d8253a700 1 --2- 172.21.15.7:0/3371816885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 0x7f8d7c06bea0 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0c3db0 tx=0x7f8d70009bd0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d8253a700 1 -- 172.21.15.7:0/3371816885 shutdown_connections 2023-03-03T15:57:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d8253a700 1 --2- 172.21.15.7:0/3371816885 >> v2:172.21.15.143:3300/0 conn(0x7f8d7c06c3e0 0x7f8d7c157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d8253a700 1 --2- 172.21.15.7:0/3371816885 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c151ab0 0x7f8d7c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d8253a700 1 --2- 172.21.15.7:0/3371816885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 0x7f8d7c06bea0 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.236+0000 7f8d8253a700 1 -- 172.21.15.7:0/3371816885 >> 172.21.15.7:0/3371816885 conn(0x7f8d7c0a94d0 msgr2=0x7f8d7c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:57:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.237+0000 7f8d8253a700 1 -- 172.21.15.7:0/3371816885 shutdown_connections 2023-03-03T15:57:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.237+0000 7f8d8253a700 1 -- 172.21.15.7:0/3371816885 wait complete. 2023-03-03T15:57:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.238+0000 7f8d8253a700 1 Processor -- start 2023-03-03T15:57:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.238+0000 7f8d8253a700 1 -- start start 2023-03-03T15:57:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.238+0000 7f8d8253a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7c06c3e0 0x7f8d7c06a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.238+0000 7f8d8253a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 0x7f8d7c0bf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.238+0000 7f8d8253a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c151ab0 0x7f8d7c0bf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.238+0000 7f8d8253a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c1ee190 con 0x7f8d7c073c30 2023-03-03T15:57:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.238+0000 7f8d8253a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c1edc10 con 0x7f8d7c06c3e0 2023-03-03T15:57:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.238+0000 7f8d8253a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c1edeb0 con 0x7f8d7c151ab0 2023-03-03T15:57:31.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d7b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 0x7f8d7c0bf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:31.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d7b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 0x7f8d7c0bf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55150/0 (socket says 172.21.15.7:55150) 2023-03-03T15:57:31.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d7b7fe700 1 -- 172.21.15.7:0/1612023663 learned_addr learned my addr 172.21.15.7:0/1612023663 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:57:31.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d80ad7700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c151ab0 0x7f8d7c0bf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:31.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d7b7fe700 1 -- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c151ab0 msgr2=0x7f8d7c0bf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:31.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d7b7fe700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c151ab0 0x7f8d7c0bf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d7bfff700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7c06c3e0 0x7f8d7c06a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:31.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d7b7fe700 1 -- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7c06c3e0 msgr2=0x7f8d7c06a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d7b7fe700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7c06c3e0 0x7f8d7c06a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d7b7fe700 1 -- 172.21.15.7:0/1612023663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70009550 con 0x7f8d7c073c30 2023-03-03T15:57:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d7b7fe700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 0x7f8d7c0bf220 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f8d70000c00 tx=0x7f8d70004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d78ff9700 1 -- 172.21.15.7:0/1612023663 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d70004540 con 0x7f8d7c073c30 2023-03-03T15:57:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d78ff9700 1 -- 172.21.15.7:0/1612023663 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d700036a0 con 0x7f8d7c073c30 2023-03-03T15:57:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d8253a700 1 -- 172.21.15.7:0/1612023663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7c151e80 con 0x7f8d7c073c30 2023-03-03T15:57:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d78ff9700 1 -- 172.21.15.7:0/1612023663 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d70003a50 con 0x7f8d7c073c30 2023-03-03T15:57:31.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.239+0000 7f8d8253a700 1 -- 172.21.15.7:0/1612023663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7c06d410 con 0x7f8d7c073c30 2023-03-03T15:57:31.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.240+0000 7f8d8253a700 1 -- 172.21.15.7:0/1612023663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7c152010 con 0x7f8d7c073c30 2023-03-03T15:57:31.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.241+0000 7f8d78ff9700 1 -- 172.21.15.7:0/1612023663 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8d7000f050 con 0x7f8d7c073c30 2023-03-03T15:57:31.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.241+0000 7f8d78ff9700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d680682f0 0x7f8d6806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:31.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.241+0000 7f8d78ff9700 1 -- 172.21.15.7:0/1612023663 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8d700879a0 con 0x7f8d7c073c30 2023-03-03T15:57:31.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.241+0000 7f8d7bfff700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d680682f0 0x7f8d6806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:31.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.242+0000 7f8d7bfff700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d680682f0 0x7f8d6806a7a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0c1640 tx=0x7f8d6c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:31.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.245+0000 7f8d78ff9700 1 -- 172.21.15.7:0/1612023663 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8d70053df0 con 0x7f8d7c073c30 2023-03-03T15:57:31.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.482+0000 7f8d8253a700 1 -- 172.21.15.7:0/1612023663 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d7c0716f0 con 0x7f8d7c073c30 2023-03-03T15:57:31.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.482+0000 7f8d78ff9700 1 -- 172.21.15.7:0/1612023663 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8d70057410 con 0x7f8d7c073c30 2023-03-03T15:57:31.484 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:57:31.485 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:57:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.484+0000 7f8d8253a700 1 -- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d680682f0 msgr2=0x7f8d6806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.484+0000 7f8d8253a700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d680682f0 0x7f8d6806a7a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0c1640 tx=0x7f8d6c008040 comp rx=0 tx=0).stop 2023-03-03T15:57:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.484+0000 7f8d8253a700 1 -- 172.21.15.7:0/1612023663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 msgr2=0x7f8d7c0bf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.484+0000 7f8d8253a700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 0x7f8d7c0bf220 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f8d70000c00 tx=0x7f8d70004750 comp rx=0 tx=0).stop 2023-03-03T15:57:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.484+0000 7f8d8253a700 1 -- 172.21.15.7:0/1612023663 shutdown_connections 2023-03-03T15:57:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.484+0000 7f8d8253a700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d7c151ab0 0x7f8d7c0bf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.484+0000 7f8d8253a700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d7c06c3e0 0x7f8d7c06a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.484+0000 7f8d8253a700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d680682f0 0x7f8d6806a7a0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.484+0000 7f8d8253a700 1 --2- 172.21.15.7:0/1612023663 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d7c073c30 0x7f8d7c0bf220 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.484+0000 7f8d8253a700 1 -- 172.21.15.7:0/1612023663 >> 172.21.15.7:0/1612023663 conn(0x7f8d7c0a94d0 msgr2=0x7f8d7c0c4670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:57:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.485+0000 7f8d8253a700 1 -- 172.21.15.7:0/1612023663 shutdown_connections 2023-03-03T15:57:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:31.485+0000 7f8d8253a700 1 -- 172.21.15.7:0/1612023663 wait complete. 2023-03-03T15:57:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:57:31.498 INFO:tasks.mds_thrash.fs.[b]:waiting for 61 secs before thrashing 2023-03-03T15:57:38.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:57:38.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.967+0000 7fd927474700 1 Processor -- start 2023-03-03T15:57:38.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.968+0000 7fd927474700 1 -- start start 2023-03-03T15:57:38.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.968+0000 7fd927474700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd920072250 0x7fd92014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:38.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.968+0000 7fd927474700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd9200c1ce0 0x7fd92014e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:38.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.968+0000 7fd927474700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9201510d0 con 0x7fd920071e80 2023-03-03T15:57:38.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.968+0000 7fd927474700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9200c3740 con 0x7fd9200c1ce0 2023-03-03T15:57:38.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.968+0000 7fd927474700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd920070670 con 0x7fd920072250 2023-03-03T15:57:38.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.968+0000 7fd925210700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd920071e80 0x7fd920151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49152/0 (socket says 172.21.15.7:49152) 2023-03-03T15:57:38.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.968+0000 7fd925210700 1 -- 172.21.15.7:0/3566762305 learned_addr learned my addr 172.21.15.7:0/3566762305 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:57:38.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd924a0f700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd920072250 0x7fd92014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:38.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd925a11700 1 --2- 172.21.15.7:0/3566762305 >> v2:172.21.15.143:3300/0 conn(0x7fd9200c1ce0 0x7fd92014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:38.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430094735 0 0) 0x7fd9201510d0 con 0x7fd920071e80 2023-03-03T15:57:38.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd90c0035b0 con 0x7fd920071e80 2023-03-03T15:57:38.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1344351227 0 0) 0x7fd90c0035b0 con 0x7fd920071e80 2023-03-03T15:57:38.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd920072250 msgr2=0x7fd92014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:57:38.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd920072250 0x7fd92014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:38.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 >> v2:172.21.15.143:3300/0 conn(0x7fd9200c1ce0 msgr2=0x7fd92014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:57:38.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 --2- 172.21.15.7:0/3566762305 >> v2:172.21.15.143:3300/0 conn(0x7fd9200c1ce0 0x7fd92014e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:38.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200c39e0 con 0x7fd920071e80 2023-03-03T15:57:38.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd914001670 con 0x7fd920071e80 2023-03-03T15:57:38.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 >> v1:172.21.15.7:6789/0 conn(0x7fd920071e80 legacy=0x7fd920151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:38.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c003a70 0x7fd90c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:38.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd920072250 0x7fd90c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:38.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c004d90 0x7fd90c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:38.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200c3740 con 0x7fd90c003a70 2023-03-03T15:57:38.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd920070670 con 0x7fd90c004d90 2023-03-03T15:57:38.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c00ba60 con 0x7fd920072250 2023-03-03T15:57:38.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd925210700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c003a70 0x7fd90c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:38.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd924a0f700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd920072250 0x7fd90c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:38.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.969+0000 7fd925a11700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c004d90 0x7fd90c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:38.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd925210700 1 -- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd920072250 msgr2=0x7fd90c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:57:38.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd925210700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd920072250 0x7fd90c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:38.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd925210700 1 -- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c004d90 msgr2=0x7fd90c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:57:38.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd925210700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd90c004d90 0x7fd90c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:38.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd925210700 1 -- 172.21.15.7:0/3566762305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c0035b0 con 0x7fd90c003a70 2023-03-03T15:57:38.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd925210700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c003a70 0x7fd90c003e80 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fd91400c3a0 tx=0x7fd90c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767af5892b321806 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:38.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914007c60 con 0x7fd90c003a70 2023-03-03T15:57:38.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd914005960 con 0x7fd90c003a70 2023-03-03T15:57:38.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd913fff700 1 -- 172.21.15.7:0/3566762305 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91400cea0 con 0x7fd90c003a70 2023-03-03T15:57:38.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd927474700 1 -- 172.21.15.7:0/3566762305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c003a70 msgr2=0x7fd90c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:38.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd927474700 1 --2- 172.21.15.7:0/3566762305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd90c003a70 0x7fd90c003e80 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fd91400c3a0 tx=0x7fd90c005880 comp rx=0 tx=0).stop 2023-03-03T15:57:38.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.970+0000 7fd925a11700 1 -- 172.21.15.7:0/3566762305 reap_dead start 2023-03-03T15:57:38.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.971+0000 7fd927474700 1 -- 172.21.15.7:0/3566762305 shutdown_connections 2023-03-03T15:57:38.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.971+0000 7fd927474700 1 -- 172.21.15.7:0/3566762305 >> 172.21.15.7:0/3566762305 conn(0x7fd9200a94d0 msgr2=0x7fd9200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:57:38.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.971+0000 7fd927474700 1 -- 172.21.15.7:0/3566762305 shutdown_connections 2023-03-03T15:57:38.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.971+0000 7fd927474700 1 -- 172.21.15.7:0/3566762305 wait complete. 2023-03-03T15:57:38.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.972+0000 7fd927474700 1 Processor -- start 2023-03-03T15:57:38.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.972+0000 7fd927474700 1 -- start start 2023-03-03T15:57:38.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd927474700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c003a70 0x7fd9200c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:38.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd927474700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd920149ac0 0x7fd9201473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:38.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd927474700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd920147900 0x7fd9200ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:38.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd927474700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200aa6a0 con 0x7fd920147900 2023-03-03T15:57:38.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd927474700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200abc20 con 0x7fd920149ac0 2023-03-03T15:57:38.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd927474700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200c6690 con 0x7fd90c003a70 2023-03-03T15:57:38.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd925a11700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd920147900 0x7fd9200ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:38.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd925a11700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd920147900 0x7fd9200ac0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55188/0 (socket says 172.21.15.7:55188) 2023-03-03T15:57:38.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd925a11700 1 -- 172.21.15.7:0/277009160 learned_addr learned my addr 172.21.15.7:0/277009160 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:57:38.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd925210700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c003a70 0x7fd9200c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:38.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd924a0f700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd920149ac0 0x7fd9201473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:38.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd925a11700 1 -- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c003a70 msgr2=0x7fd9200c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:38.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd925a11700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c003a70 0x7fd9200c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:38.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd925a11700 1 -- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd920149ac0 msgr2=0x7fd9201473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:38.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd925a11700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd920149ac0 0x7fd9201473c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:38.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd925a11700 1 -- 172.21.15.7:0/277009160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd920151f30 con 0x7fd920147900 2023-03-03T15:57:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd925a11700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd920147900 0x7fd9200ac0b0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fd920072670 tx=0x7fd918011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd911ffb700 1 -- 172.21.15.7:0/277009160 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91800e040 con 0x7fd920147900 2023-03-03T15:57:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd911ffb700 1 -- 172.21.15.7:0/277009160 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91800f040 con 0x7fd920147900 2023-03-03T15:57:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd927474700 1 -- 172.21.15.7:0/277009160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd920151300 con 0x7fd920147900 2023-03-03T15:57:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.973+0000 7fd911ffb700 1 -- 172.21.15.7:0/277009160 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91800fbe0 con 0x7fd920147900 2023-03-03T15:57:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.974+0000 7fd927474700 1 -- 172.21.15.7:0/277009160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92006c9a0 con 0x7fd920147900 2023-03-03T15:57:38.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.975+0000 7fd927474700 1 -- 172.21.15.7:0/277009160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9200abdd0 con 0x7fd920147900 2023-03-03T15:57:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.975+0000 7fd911ffb700 1 -- 172.21.15.7:0/277009160 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd91800fd40 con 0x7fd920147900 2023-03-03T15:57:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.976+0000 7fd911ffb700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd904068320 0x7fd90406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:57:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.976+0000 7fd911ffb700 1 -- 172.21.15.7:0/277009160 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd918086df0 con 0x7fd920147900 2023-03-03T15:57:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.976+0000 7fd925210700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd904068320 0x7fd90406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:57:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.976+0000 7fd925210700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd904068320 0x7fd90406a7d0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fd914001050 tx=0x7fd91400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:57:38.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:38.980+0000 7fd911ffb700 1 -- 172.21.15.7:0/277009160 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd9200abdd0 con 0x7fd920147900 2023-03-03T15:57:39.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.217+0000 7fd927474700 1 -- 172.21.15.7:0/277009160 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9200abd90 con 0x7fd920147900 2023-03-03T15:57:39.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.218+0000 7fd911ffb700 1 -- 172.21.15.7:0/277009160 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd918053040 con 0x7fd920147900 2023-03-03T15:57:39.221 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:57:39.221 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:57:39.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 -- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd904068320 msgr2=0x7fd90406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:39.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd904068320 0x7fd90406a7d0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fd914001050 tx=0x7fd91400a000 comp rx=0 tx=0).stop 2023-03-03T15:57:39.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 -- 172.21.15.7:0/277009160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd920147900 msgr2=0x7fd9200ac0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:57:39.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd920147900 0x7fd9200ac0b0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fd920072670 tx=0x7fd918011ed0 comp rx=0 tx=0).stop 2023-03-03T15:57:39.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 -- 172.21.15.7:0/277009160 shutdown_connections 2023-03-03T15:57:39.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd904068320 0x7fd90406a7d0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:39.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd920147900 0x7fd9200ac0b0 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:39.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd90c003a70 0x7fd9200c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:39.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 --2- 172.21.15.7:0/277009160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd920149ac0 0x7fd9201473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:57:39.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 -- 172.21.15.7:0/277009160 >> 172.21.15.7:0/277009160 conn(0x7fd9200a94d0 msgr2=0x7fd9200c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:57:39.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 -- 172.21.15.7:0/277009160 shutdown_connections 2023-03-03T15:57:39.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:57:39.220+0000 7fd927474700 1 -- 172.21.15.7:0/277009160 wait complete. 2023-03-03T15:57:39.224 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:57:39.234 INFO:tasks.mds_thrash.fs.[a]:waiting for 41 secs before thrashing 2023-03-03T15:57:40.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:57:40.631 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:10.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:10.786 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:20.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:58:20.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.373+0000 7f166c61c700 1 Processor -- start 2023-03-03T15:58:20.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.373+0000 7f166c61c700 1 -- start start 2023-03-03T15:58:20.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.374+0000 7f166c61c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406d240 0x7f16640c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:20.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.374+0000 7f166c61c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f16640c3e30 0x7f16640c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:20.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.374+0000 7f166c61c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640ad2b0 con 0x7f166414fc50 2023-03-03T15:58:20.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.374+0000 7f166c61c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16640c30c0 con 0x7f16640c3e30 2023-03-03T15:58:20.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.374+0000 7f166c61c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1664156420 con 0x7f166406d240 2023-03-03T15:58:20.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.374+0000 7f166a3b8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f166414fc50 0x7f16640c1f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49220/0 (socket says 172.21.15.7:49220) 2023-03-03T15:58:20.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.374+0000 7f166a3b8700 1 -- 172.21.15.7:0/4022059102 learned_addr learned my addr 172.21.15.7:0/4022059102 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:58:20.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.374+0000 7f166abb9700 1 --2- 172.21.15.7:0/4022059102 >> v2:172.21.15.143:3300/0 conn(0x7f16640c3e30 0x7f16640c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:20.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.374+0000 7f1669bb7700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406d240 0x7f16640c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:20.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.374+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188158094 0 0) 0x7f16640ad2b0 con 0x7f166414fc50 2023-03-03T15:58:20.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16500035b0 con 0x7f166414fc50 2023-03-03T15:58:20.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790501862 0 0) 0x7f16500035b0 con 0x7f166414fc50 2023-03-03T15:58:20.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406d240 msgr2=0x7f16640c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:58:20.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406d240 0x7f16640c27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:20.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 >> v2:172.21.15.143:3300/0 conn(0x7f16640c3e30 msgr2=0x7f16640c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:58:20.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 --2- 172.21.15.7:0/4022059102 >> v2:172.21.15.143:3300/0 conn(0x7f16640c3e30 0x7f16640c42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640ad520 con 0x7f166414fc50 2023-03-03T15:58:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f1654001670 con 0x7f166414fc50 2023-03-03T15:58:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f1669bb7700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406d240 0x7f16640c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:58:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f166abb9700 1 --2- 172.21.15.7:0/4022059102 >> v2:172.21.15.143:3300/0 conn(0x7f16640c3e30 0x7f16640c42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:58:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 >> v1:172.21.15.7:6789/0 conn(0x7f166414fc50 legacy=0x7f16640c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 0x7f1650003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406d240 0x7f1650004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1650004d90 0x7f1650005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1664156420 con 0x7f1650003a70 2023-03-03T15:58:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165000ba40 con 0x7f1650004d90 2023-03-03T15:58:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165000bc00 con 0x7f166406d240 2023-03-03T15:58:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f166a3b8700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 0x7f1650003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f1669bb7700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406d240 0x7f1650004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f166abb9700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1650004d90 0x7f1650005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f166a3b8700 1 -- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406d240 msgr2=0x7f1650004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:58:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f166a3b8700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406d240 0x7f1650004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f166a3b8700 1 -- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1650004d90 msgr2=0x7f1650005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:58:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f166a3b8700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1650004d90 0x7f1650005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.375+0000 7f166a3b8700 1 -- 172.21.15.7:0/4022059102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16600094a0 con 0x7f1650003a70 2023-03-03T15:58:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f166a3b8700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 0x7f1650003e80 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f165400c2f0 tx=0x7f1650005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504a91dbde33168d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165400ce30 con 0x7f1650003a70 2023-03-03T15:58:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1654005960 con 0x7f1650003a70 2023-03-03T15:58:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f16693b6700 1 -- 172.21.15.7:0/4022059102 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654005d10 con 0x7f1650003a70 2023-03-03T15:58:20.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f166c61c700 1 -- 172.21.15.7:0/4022059102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 msgr2=0x7f1650003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:20.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f166c61c700 1 --2- 172.21.15.7:0/4022059102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 0x7f1650003e80 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f165400c2f0 tx=0x7f1650005880 comp rx=0 tx=0).stop 2023-03-03T15:58:20.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f166abb9700 1 -- 172.21.15.7:0/4022059102 reap_dead start 2023-03-03T15:58:20.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f166c61c700 1 -- 172.21.15.7:0/4022059102 shutdown_connections 2023-03-03T15:58:20.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f166c61c700 1 -- 172.21.15.7:0/4022059102 >> 172.21.15.7:0/4022059102 conn(0x7f166406c110 msgr2=0x7f166406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:58:20.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f166c61c700 1 -- 172.21.15.7:0/4022059102 shutdown_connections 2023-03-03T15:58:20.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.376+0000 7f166c61c700 1 -- 172.21.15.7:0/4022059102 wait complete. 2023-03-03T15:58:20.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166c61c700 1 Processor -- start 2023-03-03T15:58:20.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166c61c700 1 -- start start 2023-03-03T15:58:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166c61c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 0x7f16640c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166c61c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166414c9b0 0x7f1664145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166c61c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1664146360 0x7f16641ece00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166c61c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640c72f0 con 0x7f1650003a70 2023-03-03T15:58:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166c61c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640720f0 con 0x7f1664146360 2023-03-03T15:58:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166c61c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166414d0c0 con 0x7f166414c9b0 2023-03-03T15:58:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166a3b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 0x7f16640c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166a3b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 0x7f16640c0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55252/0 (socket says 172.21.15.7:55252) 2023-03-03T15:58:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166a3b8700 1 -- 172.21.15.7:0/2931183664 learned_addr learned my addr 172.21.15.7:0/2931183664 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:58:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f1669bb7700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166414c9b0 0x7f1664145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.378+0000 7f166abb9700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1664146360 0x7f16641ece00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f166a3b8700 1 -- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166414c9b0 msgr2=0x7f1664145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f166a3b8700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166414c9b0 0x7f1664145e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f166a3b8700 1 -- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1664146360 msgr2=0x7f16641ece00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f166a3b8700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1664146360 0x7f16641ece00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f166a3b8700 1 -- 172.21.15.7:0/2931183664 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1664156690 con 0x7f1650003a70 2023-03-03T15:58:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f166a3b8700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 0x7f16640c0a40 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f1654008fd0 tx=0x7f1654001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f165affd700 1 -- 172.21.15.7:0/2931183664 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654005ce0 con 0x7f1650003a70 2023-03-03T15:58:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f165affd700 1 -- 172.21.15.7:0/2931183664 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1654005e40 con 0x7f1650003a70 2023-03-03T15:58:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f166c61c700 1 -- 172.21.15.7:0/2931183664 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1664074220 con 0x7f1650003a70 2023-03-03T15:58:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f165affd700 1 -- 172.21.15.7:0/2931183664 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654004ab0 con 0x7f1650003a70 2023-03-03T15:58:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.379+0000 7f166c61c700 1 -- 172.21.15.7:0/2931183664 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16640acf60 con 0x7f1650003a70 2023-03-03T15:58:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.380+0000 7f166c61c700 1 -- 172.21.15.7:0/2931183664 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1664074bb0 con 0x7f1650003a70 2023-03-03T15:58:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.380+0000 7f165affd700 1 -- 172.21.15.7:0/2931183664 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1654024070 con 0x7f1650003a70 2023-03-03T15:58:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.381+0000 7f165affd700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16480682f0 0x7f164806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.381+0000 7f165affd700 1 -- 172.21.15.7:0/2931183664 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1654087240 con 0x7f1650003a70 2023-03-03T15:58:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.381+0000 7f1669bb7700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16480682f0 0x7f164806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.382+0000 7f1669bb7700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16480682f0 0x7f164806a7a0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f166414cda0 tx=0x7f1660009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.385+0000 7f165affd700 1 -- 172.21.15.7:0/2931183664 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1654053640 con 0x7f1650003a70 2023-03-03T15:58:20.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.627+0000 7f166c61c700 1 -- 172.21.15.7:0/2931183664 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1664073ed0 con 0x7f1650003a70 2023-03-03T15:58:20.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.628+0000 7f165affd700 1 -- 172.21.15.7:0/2931183664 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1654007da0 con 0x7f1650003a70 2023-03-03T15:58:20.629 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:58:20.629 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:58:20.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 -- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16480682f0 msgr2=0x7f164806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:20.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16480682f0 0x7f164806a7a0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f166414cda0 tx=0x7f1660009210 comp rx=0 tx=0).stop 2023-03-03T15:58:20.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 -- 172.21.15.7:0/2931183664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 msgr2=0x7f16640c0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:20.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 0x7f16640c0a40 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7f1654008fd0 tx=0x7f1654001a00 comp rx=0 tx=0).stop 2023-03-03T15:58:20.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 -- 172.21.15.7:0/2931183664 shutdown_connections 2023-03-03T15:58:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1664146360 0x7f16641ece00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16480682f0 0x7f164806a7a0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1650003a70 0x7f16640c0a40 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 --2- 172.21.15.7:0/2931183664 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166414c9b0 0x7f1664145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 -- 172.21.15.7:0/2931183664 >> 172.21.15.7:0/2931183664 conn(0x7f166406c110 msgr2=0x7f166414e050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:58:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 -- 172.21.15.7:0/2931183664 shutdown_connections 2023-03-03T15:58:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:20.630+0000 7f166c61c700 1 -- 172.21.15.7:0/2931183664 wait complete. 2023-03-03T15:58:20.633 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:58:20.643 INFO:tasks.mds_thrash.fs.[a]:waiting for 17 secs before thrashing 2023-03-03T15:58:32.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:58:32.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.638+0000 7fdb89467700 1 Processor -- start 2023-03-03T15:58:32.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.638+0000 7fdb89467700 1 -- start start 2023-03-03T15:58:32.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb89467700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdb840caa30 0x7fdb840c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb89467700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840c00e0 0x7fdb84154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb89467700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb840cae00 0x7fdb840c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb89467700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb840cb280 con 0x7fdb840cae00 2023-03-03T15:58:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb89467700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb840bfd10 con 0x7fdb840caa30 2023-03-03T15:58:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb89467700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840bfa30 con 0x7fdb840c00e0 2023-03-03T15:58:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb837fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb840cae00 0x7fdb840c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb837fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb840cae00 0x7fdb840c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55264/0 (socket says 172.21.15.7:55264) 2023-03-03T15:58:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb837fe700 1 -- 172.21.15.7:0/4061772470 learned_addr learned my addr 172.21.15.7:0/4061772470 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:58:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb827fc700 1 --2- 172.21.15.7:0/4061772470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840c00e0 0x7fdb84154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb82ffd700 1 --2- 172.21.15.7:0/4061772470 >> v2:172.21.15.143:3300/0 conn(0x7fdb840caa30 0x7fdb840c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb837fe700 1 -- 172.21.15.7:0/4061772470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840c00e0 msgr2=0x7fdb84154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:58:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb837fe700 1 --2- 172.21.15.7:0/4061772470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840c00e0 0x7fdb84154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb837fe700 1 -- 172.21.15.7:0/4061772470 >> v2:172.21.15.143:3300/0 conn(0x7fdb840caa30 msgr2=0x7fdb840c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:58:32.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb837fe700 1 --2- 172.21.15.7:0/4061772470 >> v2:172.21.15.143:3300/0 conn(0x7fdb840caa30 0x7fdb840c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.639+0000 7fdb837fe700 1 -- 172.21.15.7:0/4061772470 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb84154510 con 0x7fdb840cae00 2023-03-03T15:58:32.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb837fe700 1 --2- 172.21.15.7:0/4061772470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840cae00 0x7fdb840c6c00 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fdb840c81b0 tx=0x7fdb7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56bd0f95d1d8007 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:32.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb81ffb700 1 -- 172.21.15.7:0/4061772470 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c00e070 con 0x7fdb840cae00 2023-03-03T15:58:32.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb81ffb700 1 -- 172.21.15.7:0/4061772470 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7c00b9c0 con 0x7fdb840cae00 2023-03-03T15:58:32.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb81ffb700 1 -- 172.21.15.7:0/4061772470 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c00bd70 con 0x7fdb840cae00 2023-03-03T15:58:32.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb89467700 1 -- 172.21.15.7:0/4061772470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840cae00 msgr2=0x7fdb840c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb89467700 1 --2- 172.21.15.7:0/4061772470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840cae00 0x7fdb840c6c00 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fdb840c81b0 tx=0x7fdb7c00b330 comp rx=0 tx=0).stop 2023-03-03T15:58:32.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb89467700 1 -- 172.21.15.7:0/4061772470 shutdown_connections 2023-03-03T15:58:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb89467700 1 --2- 172.21.15.7:0/4061772470 >> v2:172.21.15.143:3300/0 conn(0x7fdb840caa30 0x7fdb840c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb89467700 1 --2- 172.21.15.7:0/4061772470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840cae00 0x7fdb840c6c00 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb89467700 1 --2- 172.21.15.7:0/4061772470 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840c00e0 0x7fdb84154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb89467700 1 -- 172.21.15.7:0/4061772470 >> 172.21.15.7:0/4061772470 conn(0x7fdb84066e40 msgr2=0x7fdb84071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:58:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb89467700 1 -- 172.21.15.7:0/4061772470 shutdown_connections 2023-03-03T15:58:32.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.640+0000 7fdb89467700 1 -- 172.21.15.7:0/4061772470 wait complete. 2023-03-03T15:58:32.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb89467700 1 Processor -- start 2023-03-03T15:58:32.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb89467700 1 -- start start 2023-03-03T15:58:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb89467700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840c00e0 0x7fdb8414d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb89467700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840caa30 0x7fdb840c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb89467700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb840cae00 0x7fdb841e9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb89467700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840aa930 con 0x7fdb840c00e0 2023-03-03T15:58:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb89467700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb840aa410 con 0x7fdb840cae00 2023-03-03T15:58:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb89467700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8414db50 con 0x7fdb840caa30 2023-03-03T15:58:32.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb82ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840c00e0 0x7fdb8414d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb82ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840c00e0 0x7fdb8414d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55268/0 (socket says 172.21.15.7:55268) 2023-03-03T15:58:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.642+0000 7fdb82ffd700 1 -- 172.21.15.7:0/3458414970 learned_addr learned my addr 172.21.15.7:0/3458414970 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:58:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb827fc700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840caa30 0x7fdb840c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb837fe700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb840cae00 0x7fdb841e9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb82ffd700 1 -- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840caa30 msgr2=0x7fdb840c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb82ffd700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840caa30 0x7fdb840c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb82ffd700 1 -- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb840cae00 msgr2=0x7fdb841e9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb82ffd700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb840cae00 0x7fdb841e9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb82ffd700 1 -- 172.21.15.7:0/3458414970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7c009b30 con 0x7fdb840c00e0 2023-03-03T15:58:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb837fe700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb840cae00 0x7fdb841e9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:58:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb827fc700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840caa30 0x7fdb840c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T15:58:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb82ffd700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840c00e0 0x7fdb8414d610 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800e810 tx=0x7fdb7800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb6b7fe700 1 -- 172.21.15.7:0/3458414970 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78013610 con 0x7fdb840c00e0 2023-03-03T15:58:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb6b7fe700 1 -- 172.21.15.7:0/3458414970 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb78013770 con 0x7fdb840c00e0 2023-03-03T15:58:32.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb89467700 1 -- 172.21.15.7:0/3458414970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb84154ad0 con 0x7fdb840c00e0 2023-03-03T15:58:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb6b7fe700 1 -- 172.21.15.7:0/3458414970 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78013b20 con 0x7fdb840c00e0 2023-03-03T15:58:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.643+0000 7fdb89467700 1 -- 172.21.15.7:0/3458414970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb841547f0 con 0x7fdb840c00e0 2023-03-03T15:58:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.644+0000 7fdb89467700 1 -- 172.21.15.7:0/3458414970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb840bf650 con 0x7fdb840c00e0 2023-03-03T15:58:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.645+0000 7fdb6b7fe700 1 -- 172.21.15.7:0/3458414970 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdb78023070 con 0x7fdb840c00e0 2023-03-03T15:58:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.645+0000 7fdb6b7fe700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb700682f0 0x7fdb7006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.645+0000 7fdb6b7fe700 1 -- 172.21.15.7:0/3458414970 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdb78085e30 con 0x7fdb840c00e0 2023-03-03T15:58:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.645+0000 7fdb827fc700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb700682f0 0x7fdb7006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:32.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.646+0000 7fdb827fc700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb700682f0 0x7fdb7006a7a0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fdb84152570 tx=0x7fdb74008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:32.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.650+0000 7fdb6b7fe700 1 -- 172.21.15.7:0/3458414970 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdb78052150 con 0x7fdb840c00e0 2023-03-03T15:58:32.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.886+0000 7fdb89467700 1 -- 172.21.15.7:0/3458414970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb84074bb0 con 0x7fdb840c00e0 2023-03-03T15:58:32.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.887+0000 7fdb6b7fe700 1 -- 172.21.15.7:0/3458414970 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdb78055770 con 0x7fdb840c00e0 2023-03-03T15:58:32.889 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:58:32.889 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:58:32.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.888+0000 7fdb89467700 1 -- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb700682f0 msgr2=0x7fdb7006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.888+0000 7fdb89467700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb700682f0 0x7fdb7006a7a0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fdb84152570 tx=0x7fdb74008040 comp rx=0 tx=0).stop 2023-03-03T15:58:32.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.888+0000 7fdb89467700 1 -- 172.21.15.7:0/3458414970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840c00e0 msgr2=0x7fdb8414d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:32.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.888+0000 7fdb89467700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840c00e0 0x7fdb8414d610 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800e810 tx=0x7fdb7800ebd0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.888+0000 7fdb89467700 1 -- 172.21.15.7:0/3458414970 shutdown_connections 2023-03-03T15:58:32.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.888+0000 7fdb89467700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb840cae00 0x7fdb841e9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.888+0000 7fdb89467700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb700682f0 0x7fdb7006a7a0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.888+0000 7fdb89467700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb840c00e0 0x7fdb8414d610 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.888+0000 7fdb89467700 1 --2- 172.21.15.7:0/3458414970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb840caa30 0x7fdb840c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:32.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.888+0000 7fdb89467700 1 -- 172.21.15.7:0/3458414970 >> 172.21.15.7:0/3458414970 conn(0x7fdb84066e40 msgr2=0x7fdb8406b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:58:32.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.889+0000 7fdb89467700 1 -- 172.21.15.7:0/3458414970 shutdown_connections 2023-03-03T15:58:32.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:32.889+0000 7fdb89467700 1 -- 172.21.15.7:0/3458414970 wait complete. 2023-03-03T15:58:32.892 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:58:32.902 INFO:tasks.mds_thrash.fs.[b]:waiting for 90 secs before thrashing 2023-03-03T15:58:37.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:58:37.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.781+0000 7f05f5731700 1 Processor -- start 2023-03-03T15:58:37.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.781+0000 7f05f5731700 1 -- start start 2023-03-03T15:58:37.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05f5731700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f05f00c00e0 0x7f05f00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:37.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05f5731700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f0154ff0 0x7f05f00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:37.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05f5731700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f00c7310 con 0x7f05f00caa30 2023-03-03T15:58:37.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05f5731700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f00bfd10 con 0x7f05f00c00e0 2023-03-03T15:58:37.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05f5731700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00bfa30 con 0x7f05f0154ff0 2023-03-03T15:58:37.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05ef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f0154ff0 0x7f05f00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05ee7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f05f00c00e0 0x7f05f00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05eeffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f05f00caa30 0x7f05f00c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:52776/0 (socket says 172.21.15.7:52776) 2023-03-03T15:58:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05ef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f0154ff0 0x7f05f00cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:35118/0 (socket says 172.21.15.7:35118) 2023-03-03T15:58:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05eeffd700 1 -- 172.21.15.7:0/430420610 learned_addr learned my addr 172.21.15.7:0/430420610 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:58:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05ef7fe700 1 -- 172.21.15.7:0/430420610 >> v2:172.21.15.143:3300/0 conn(0x7f05f00c00e0 msgr2=0x7f05f00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T15:58:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05ef7fe700 1 --2- 172.21.15.7:0/430420610 >> v2:172.21.15.143:3300/0 conn(0x7f05f00c00e0 0x7f05f00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05ef7fe700 1 -- 172.21.15.7:0/430420610 >> v1:172.21.15.143:6789/0 conn(0x7f05f00caa30 legacy=0x7f05f00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:37.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05edffb700 1 -- 172.21.15.7:0/430420610 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367467630 0 0) 0x7f05f00c7310 con 0x7f05f00caa30 2023-03-03T15:58:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05ef7fe700 1 -- 172.21.15.7:0/430420610 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0154510 con 0x7f05f0154ff0 2023-03-03T15:58:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.782+0000 7f05ee7fc700 1 --2- 172.21.15.7:0/430420610 >> v2:172.21.15.143:3300/0 conn(0x7f05f00c00e0 0x7f05f00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T15:58:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05ef7fe700 1 --2- 172.21.15.7:0/430420610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f0154ff0 0x7f05f00cae00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009d90 tx=0x7f05e800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0ce4e7f6dd15607 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05edffb700 1 -- 172.21.15.7:0/430420610 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e800e070 con 0x7f05f0154ff0 2023-03-03T15:58:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05edffb700 1 -- 172.21.15.7:0/430420610 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e800bb20 con 0x7f05f0154ff0 2023-03-03T15:58:37.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05edffb700 1 -- 172.21.15.7:0/430420610 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8012430 con 0x7f05f0154ff0 2023-03-03T15:58:37.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05f5731700 1 -- 172.21.15.7:0/430420610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f0154ff0 msgr2=0x7f05f00cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05f5731700 1 --2- 172.21.15.7:0/430420610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f0154ff0 0x7f05f00cae00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009d90 tx=0x7f05e800b330 comp rx=0 tx=0).stop 2023-03-03T15:58:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05f5731700 1 -- 172.21.15.7:0/430420610 shutdown_connections 2023-03-03T15:58:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05f5731700 1 --2- 172.21.15.7:0/430420610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f0154ff0 0x7f05f00cae00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05f5731700 1 --2- 172.21.15.7:0/430420610 >> v2:172.21.15.143:3300/0 conn(0x7f05f00c00e0 0x7f05f00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05f5731700 1 -- 172.21.15.7:0/430420610 >> 172.21.15.7:0/430420610 conn(0x7f05f0066e40 msgr2=0x7f05f0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:58:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05f5731700 1 -- 172.21.15.7:0/430420610 shutdown_connections 2023-03-03T15:58:37.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.783+0000 7f05f5731700 1 -- 172.21.15.7:0/430420610 wait complete. 2023-03-03T15:58:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.785+0000 7f05f5731700 1 Processor -- start 2023-03-03T15:58:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.785+0000 7f05f5731700 1 -- start start 2023-03-03T15:58:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.785+0000 7f05f5731700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f00c00e0 0x7f05f01577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.785+0000 7f05f5731700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f00caa30 0x7f05f014d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.785+0000 7f05f5731700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f0154ff0 0x7f05f014d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.785+0000 7f05f5731700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f01ea700 con 0x7f05f00c00e0 2023-03-03T15:58:37.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.785+0000 7f05f5731700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f0157360 con 0x7f05f0154ff0 2023-03-03T15:58:37.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.785+0000 7f05f5731700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00c2910 con 0x7f05f00caa30 2023-03-03T15:58:37.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05eeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f00c00e0 0x7f05f01577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:37.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05eeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f00c00e0 0x7f05f01577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55282/0 (socket says 172.21.15.7:55282) 2023-03-03T15:58:37.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05eeffd700 1 -- 172.21.15.7:0/1880258412 learned_addr learned my addr 172.21.15.7:0/1880258412 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:58:37.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05ee7fc700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f00caa30 0x7f05f014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:37.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05ef7fe700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f0154ff0 0x7f05f014d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:37.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05eeffd700 1 -- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f00caa30 msgr2=0x7f05f014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:37.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05eeffd700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f00caa30 0x7f05f014d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05eeffd700 1 -- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f0154ff0 msgr2=0x7f05f014d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05eeffd700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f0154ff0 0x7f05f014d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05eeffd700 1 -- 172.21.15.7:0/1880258412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e4009430 con 0x7f05f00c00e0 2023-03-03T15:58:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05eeffd700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f00c00e0 0x7f05f01577f0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f05e00011e0 tx=0x7f05e000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05d77fe700 1 -- 172.21.15.7:0/1880258412 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0004c80 con 0x7f05f00c00e0 2023-03-03T15:58:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05d77fe700 1 -- 172.21.15.7:0/1880258412 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e0004e00 con 0x7f05f00c00e0 2023-03-03T15:58:37.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05f5731700 1 -- 172.21.15.7:0/1880258412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e8009a70 con 0x7f05f00c00e0 2023-03-03T15:58:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05d77fe700 1 -- 172.21.15.7:0/1880258412 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0007310 con 0x7f05f00c00e0 2023-03-03T15:58:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.786+0000 7f05f5731700 1 -- 172.21.15.7:0/1880258412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f0154ad0 con 0x7f05f00c00e0 2023-03-03T15:58:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.787+0000 7f05f5731700 1 -- 172.21.15.7:0/1880258412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f01547f0 con 0x7f05f00c00e0 2023-03-03T15:58:37.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.788+0000 7f05d77fe700 1 -- 172.21.15.7:0/1880258412 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f05e001a020 con 0x7f05f00c00e0 2023-03-03T15:58:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.788+0000 7f05d77fe700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05dc0682f0 0x7f05dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:58:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.788+0000 7f05d77fe700 1 -- 172.21.15.7:0/1880258412 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f05e0085c30 con 0x7f05f00c00e0 2023-03-03T15:58:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.788+0000 7f05ee7fc700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05dc0682f0 0x7f05dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:58:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.789+0000 7f05ee7fc700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05dc0682f0 0x7f05dc06a7a0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f05f00ab400 tx=0x7f05e40091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:58:37.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:37.793+0000 7f05d77fe700 1 -- 172.21.15.7:0/1880258412 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f05e0051fd0 con 0x7f05f00c00e0 2023-03-03T15:58:38.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.029+0000 7f05f5731700 1 -- 172.21.15.7:0/1880258412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05f00bf650 con 0x7f05f00c00e0 2023-03-03T15:58:38.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.030+0000 7f05d77fe700 1 -- 172.21.15.7:0/1880258412 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f05e00555f0 con 0x7f05f00c00e0 2023-03-03T15:58:38.032 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:58:38.032 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:58:38.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 -- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05dc0682f0 msgr2=0x7f05dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:38.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05dc0682f0 0x7f05dc06a7a0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f05f00ab400 tx=0x7f05e40091c0 comp rx=0 tx=0).stop 2023-03-03T15:58:38.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 -- 172.21.15.7:0/1880258412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f00c00e0 msgr2=0x7f05f01577f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:58:38.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f00c00e0 0x7f05f01577f0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f05e00011e0 tx=0x7f05e000ca30 comp rx=0 tx=0).stop 2023-03-03T15:58:38.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 -- 172.21.15.7:0/1880258412 shutdown_connections 2023-03-03T15:58:38.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05f0154ff0 0x7f05f014d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:38.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05dc0682f0 0x7f05dc06a7a0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:38.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05f00c00e0 0x7f05f01577f0 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:38.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 --2- 172.21.15.7:0/1880258412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05f00caa30 0x7f05f014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:58:38.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 -- 172.21.15.7:0/1880258412 >> 172.21.15.7:0/1880258412 conn(0x7f05f0066e40 msgr2=0x7f05f006b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:58:38.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 -- 172.21.15.7:0/1880258412 shutdown_connections 2023-03-03T15:58:38.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:58:38.032+0000 7f05f5731700 1 -- 172.21.15.7:0/1880258412 wait complete. 2023-03-03T15:58:38.036 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:58:38.046 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2023-03-03T15:58:40.879 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:58:40.881 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:11.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:11.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:41.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:41.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T15:59:46.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T15:59:46.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.185+0000 7f0937919700 1 Processor -- start 2023-03-03T15:59:46.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.185+0000 7f0937919700 1 -- start start 2023-03-03T15:59:46.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0937919700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f09300c2950 0x7f0930157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:46.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0937919700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093006cb70 con 0x7f093006c480 2023-03-03T15:59:46.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0937919700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093006ce50 con 0x7f0930151b20 2023-03-03T15:59:46.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0937919700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093006c2f0 con 0x7f09300c2950 2023-03-03T15:59:46.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0935eb6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f093006c480 0x7f0930158020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49308/0 (socket says 172.21.15.7:49308) 2023-03-03T15:59:46.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0934eb4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f09300c2950 0x7f0930157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:46.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0935eb6700 1 -- 172.21.15.7:0/1618741755 learned_addr learned my addr 172.21.15.7:0/1618741755 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T15:59:46.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0934eb4700 1 -- 172.21.15.7:0/1618741755 >> v1:172.21.15.143:6789/0 conn(0x7f0930151b20 legacy=0x7f09300736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:46.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0934eb4700 1 -- 172.21.15.7:0/1618741755 >> v1:172.21.15.7:6789/0 conn(0x7f093006c480 legacy=0x7f0930158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0934eb4700 1 -- 172.21.15.7:0/1618741755 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093006d130 con 0x7f09300c2950 2023-03-03T15:59:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0927fff700 1 -- 172.21.15.7:0/1618741755 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163593784 0 0) 0x7f093006cb70 con 0x7f093006c480 2023-03-03T15:59:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0934eb4700 1 --2- 172.21.15.7:0/1618741755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09300c2950 0x7f0930157ae0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f092c009840 tx=0x7f092c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd1a1a38450160f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:59:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.186+0000 7f0927fff700 1 -- 172.21.15.7:0/1618741755 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c011830 con 0x7f09300c2950 2023-03-03T15:59:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.187+0000 7f0927fff700 1 -- 172.21.15.7:0/1618741755 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092c011990 con 0x7f09300c2950 2023-03-03T15:59:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.187+0000 7f0927fff700 1 -- 172.21.15.7:0/1618741755 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c011d20 con 0x7f09300c2950 2023-03-03T15:59:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.187+0000 7f0937919700 1 -- 172.21.15.7:0/1618741755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09300c2950 msgr2=0x7f0930157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.187+0000 7f0937919700 1 --2- 172.21.15.7:0/1618741755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09300c2950 0x7f0930157ae0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f092c009840 tx=0x7f092c009b50 comp rx=0 tx=0).stop 2023-03-03T15:59:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.187+0000 7f0937919700 1 -- 172.21.15.7:0/1618741755 shutdown_connections 2023-03-03T15:59:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.187+0000 7f0937919700 1 --2- 172.21.15.7:0/1618741755 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09300c2950 0x7f0930157ae0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.187+0000 7f0937919700 1 -- 172.21.15.7:0/1618741755 >> 172.21.15.7:0/1618741755 conn(0x7f09300a94d0 msgr2=0x7f09300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:59:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.187+0000 7f0937919700 1 -- 172.21.15.7:0/1618741755 shutdown_connections 2023-03-03T15:59:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.187+0000 7f0937919700 1 -- 172.21.15.7:0/1618741755 wait complete. 2023-03-03T15:59:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.188+0000 7f0937919700 1 Processor -- start 2023-03-03T15:59:46.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.188+0000 7f0937919700 1 -- start start 2023-03-03T15:59:46.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f0937919700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f093006c480 0x7f0930153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:46.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f0937919700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09300c2950 0x7f09301ed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f0937919700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0930151b20 0x7f09301ed900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f0937919700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09301ede40 con 0x7f093006c480 2023-03-03T15:59:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f0937919700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0930156790 con 0x7f0930151b20 2023-03-03T15:59:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f0937919700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09301522c0 con 0x7f09300c2950 2023-03-03T15:59:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f09356b5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f093006c480 0x7f0930153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f09356b5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f093006c480 0x7f0930153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55330/0 (socket says 172.21.15.7:55330) 2023-03-03T15:59:46.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f09356b5700 1 -- 172.21.15.7:0/55125898 learned_addr learned my addr 172.21.15.7:0/55125898 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T15:59:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f0934eb4700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09300c2950 0x7f09301ed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f0935eb6700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0930151b20 0x7f09301ed900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f09356b5700 1 -- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09300c2950 msgr2=0x7f09301ed3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f09356b5700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09300c2950 0x7f09301ed3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f09356b5700 1 -- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0930151b20 msgr2=0x7f09301ed900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f09356b5700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0930151b20 0x7f09301ed900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:46.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.189+0000 7f09356b5700 1 -- 172.21.15.7:0/55125898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c0094a0 con 0x7f093006c480 2023-03-03T15:59:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.190+0000 7f09356b5700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f093006c480 0x7f0930153660 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f092000c480 tx=0x7f092000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:59:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.190+0000 7f0925ffb700 1 -- 172.21.15.7:0/55125898 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09200052d0 con 0x7f093006c480 2023-03-03T15:59:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.190+0000 7f0925ffb700 1 -- 172.21.15.7:0/55125898 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0920005430 con 0x7f093006c480 2023-03-03T15:59:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.190+0000 7f0937919700 1 -- 172.21.15.7:0/55125898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0930158730 con 0x7f093006c480 2023-03-03T15:59:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.190+0000 7f0925ffb700 1 -- 172.21.15.7:0/55125898 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09200038c0 con 0x7f093006c480 2023-03-03T15:59:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.190+0000 7f0937919700 1 -- 172.21.15.7:0/55125898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093006d410 con 0x7f093006c480 2023-03-03T15:59:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.191+0000 7f0937919700 1 -- 172.21.15.7:0/55125898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0930151ef0 con 0x7f093006c480 2023-03-03T15:59:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.191+0000 7f0925ffb700 1 -- 172.21.15.7:0/55125898 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0920019030 con 0x7f093006c480 2023-03-03T15:59:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.192+0000 7f0925ffb700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09140682f0 0x7f091406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T15:59:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.192+0000 7f0925ffb700 1 -- 172.21.15.7:0/55125898 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f09200852f0 con 0x7f093006c480 2023-03-03T15:59:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.192+0000 7f0934eb4700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09140682f0 0x7f091406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T15:59:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.193+0000 7f0934eb4700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09140682f0 0x7f091406a7a0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f092c0090b0 tx=0x7f092c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T15:59:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.196+0000 7f0925ffb700 1 -- 172.21.15.7:0/55125898 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0920051630 con 0x7f093006c480 2023-03-03T15:59:46.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.434+0000 7f0937919700 1 -- 172.21.15.7:0/55125898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0930074bb0 con 0x7f093006c480 2023-03-03T15:59:46.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.434+0000 7f0925ffb700 1 -- 172.21.15.7:0/55125898 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0920054c50 con 0x7f093006c480 2023-03-03T15:59:46.437 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T15:59:46.437 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T15:59:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 -- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09140682f0 msgr2=0x7f091406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09140682f0 0x7f091406a7a0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f092c0090b0 tx=0x7f092c01c040 comp rx=0 tx=0).stop 2023-03-03T15:59:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 -- 172.21.15.7:0/55125898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f093006c480 msgr2=0x7f0930153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T15:59:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f093006c480 0x7f0930153660 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f092000c480 tx=0x7f092000c790 comp rx=0 tx=0).stop 2023-03-03T15:59:46.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 -- 172.21.15.7:0/55125898 shutdown_connections 2023-03-03T15:59:46.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0930151b20 0x7f09301ed900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:46.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09140682f0 0x7f091406a7a0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:46.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f093006c480 0x7f0930153660 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 --2- 172.21.15.7:0/55125898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09300c2950 0x7f09301ed3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T15:59:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 -- 172.21.15.7:0/55125898 >> 172.21.15.7:0/55125898 conn(0x7f09300a94d0 msgr2=0x7f09300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T15:59:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 -- 172.21.15.7:0/55125898 shutdown_connections 2023-03-03T15:59:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T15:59:46.437+0000 7f0937919700 1 -- 172.21.15.7:0/55125898 wait complete. 2023-03-03T15:59:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T15:59:46.450 INFO:tasks.mds_thrash.fs.[a]:waiting for 97 secs before thrashing 2023-03-03T16:00:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T15:59:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T16:00:02.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:00:03.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.042+0000 7f41d88a0700 1 Processor -- start 2023-03-03T16:00:03.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.042+0000 7f41d88a0700 1 -- start start 2023-03-03T16:00:03.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.043+0000 7f41d88a0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f41d4071e80 0x7f41d4151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:03.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.043+0000 7f41d88a0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f41d414e900 0x7f41d415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:03.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.043+0000 7f41d88a0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d41510d0 con 0x7f41d4072250 2023-03-03T16:00:03.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.043+0000 7f41d88a0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d40c3740 con 0x7f41d4071e80 2023-03-03T16:00:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.043+0000 7f41d88a0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d4070670 con 0x7f41d414e900 2023-03-03T16:00:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.043+0000 7f41d259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f41d4071e80 0x7f41d4151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d1d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f41d4072250 0x7f41d414e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49332/0 (socket says 172.21.15.7:49332) 2023-03-03T16:00:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d1d9b700 1 -- 172.21.15.7:0/697186745 learned_addr learned my addr 172.21.15.7:0/697186745 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:00:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d2d9d700 1 --2- 172.21.15.7:0/697186745 >> v2:172.21.15.143:3300/0 conn(0x7f41d414e900 0x7f41d415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d259c700 1 -- 172.21.15.7:0/697186745 >> v2:172.21.15.143:3300/0 conn(0x7f41d414e900 msgr2=0x7f41d415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:00:03.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d259c700 1 --2- 172.21.15.7:0/697186745 >> v2:172.21.15.143:3300/0 conn(0x7f41d414e900 0x7f41d415ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d259c700 1 -- 172.21.15.7:0/697186745 >> v1:172.21.15.7:6789/0 conn(0x7f41d4072250 legacy=0x7f41d414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d259c700 1 -- 172.21.15.7:0/697186745 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d40c39e0 con 0x7f41d4071e80 2023-03-03T16:00:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d159a700 1 -- 172.21.15.7:0/697186745 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138764486 0 0) 0x7f41d41510d0 con 0x7f41d4072250 2023-03-03T16:00:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d259c700 1 --2- 172.21.15.7:0/697186745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4071e80 0x7f41d4151820 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f41d4070de0 tx=0x7f41c4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3cf943a32de94a3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:00:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d159a700 1 -- 172.21.15.7:0/697186745 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4011830 con 0x7f41d4071e80 2023-03-03T16:00:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d159a700 1 -- 172.21.15.7:0/697186745 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c4011990 con 0x7f41d4071e80 2023-03-03T16:00:03.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.044+0000 7f41d159a700 1 -- 172.21.15.7:0/697186745 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c4011d20 con 0x7f41d4071e80 2023-03-03T16:00:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.045+0000 7f41d88a0700 1 -- 172.21.15.7:0/697186745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4071e80 msgr2=0x7f41d4151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:03.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.045+0000 7f41d88a0700 1 --2- 172.21.15.7:0/697186745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4071e80 0x7f41d4151820 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f41d4070de0 tx=0x7f41c4009b20 comp rx=0 tx=0).stop 2023-03-03T16:00:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.045+0000 7f41d88a0700 1 -- 172.21.15.7:0/697186745 shutdown_connections 2023-03-03T16:00:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.045+0000 7f41d88a0700 1 --2- 172.21.15.7:0/697186745 >> v2:172.21.15.143:3300/0 conn(0x7f41d414e900 0x7f41d415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.045+0000 7f41d88a0700 1 --2- 172.21.15.7:0/697186745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4071e80 0x7f41d4151820 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.045+0000 7f41d88a0700 1 -- 172.21.15.7:0/697186745 >> 172.21.15.7:0/697186745 conn(0x7f41d40a94d0 msgr2=0x7f41d40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:00:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.045+0000 7f41d88a0700 1 -- 172.21.15.7:0/697186745 shutdown_connections 2023-03-03T16:00:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.045+0000 7f41d88a0700 1 -- 172.21.15.7:0/697186745 wait complete. 2023-03-03T16:00:03.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.046+0000 7f41d88a0700 1 Processor -- start 2023-03-03T16:00:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.046+0000 7f41d88a0700 1 -- start start 2023-03-03T16:00:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d88a0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41d4071e80 0x7f41d4142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d88a0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4072250 0x7f41d40c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d88a0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41d414e900 0x7f41d41da730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d88a0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d41e8980 con 0x7f41d4072250 2023-03-03T16:00:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d88a0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d41433b0 con 0x7f41d414e900 2023-03-03T16:00:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d88a0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d41dc420 con 0x7f41d4071e80 2023-03-03T16:00:03.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4072250 0x7f41d40c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4072250 0x7f41d40c53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55358/0 (socket says 172.21.15.7:55358) 2023-03-03T16:00:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d1d9b700 1 -- 172.21.15.7:0/1137690132 learned_addr learned my addr 172.21.15.7:0/1137690132 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:00:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d259c700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41d4071e80 0x7f41d4142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d2d9d700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41d414e900 0x7f41d41da730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d1d9b700 1 -- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41d4071e80 msgr2=0x7f41d4142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d1d9b700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41d4071e80 0x7f41d4142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:03.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d1d9b700 1 -- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41d414e900 msgr2=0x7f41d41da730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d1d9b700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41d414e900 0x7f41d41da730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d1d9b700 1 -- 172.21.15.7:0/1137690132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c40094a0 con 0x7f41d4072250 2023-03-03T16:00:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.047+0000 7f41d2d9d700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41d414e900 0x7f41d41da730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:00:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.048+0000 7f41d1d9b700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4072250 0x7f41d40c53d0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c540 tx=0x7f41c800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:00:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.048+0000 7f41c2ffd700 1 -- 172.21.15.7:0/1137690132 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8004c80 con 0x7f41d4072250 2023-03-03T16:00:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.048+0000 7f41c2ffd700 1 -- 172.21.15.7:0/1137690132 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c8004de0 con 0x7f41d4072250 2023-03-03T16:00:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.048+0000 7f41c2ffd700 1 -- 172.21.15.7:0/1137690132 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8007310 con 0x7f41d4072250 2023-03-03T16:00:03.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.048+0000 7f41d88a0700 1 -- 172.21.15.7:0/1137690132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d406c9a0 con 0x7f41d4072250 2023-03-03T16:00:03.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.048+0000 7f41d88a0700 1 -- 172.21.15.7:0/1137690132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d4151300 con 0x7f41d4072250 2023-03-03T16:00:03.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.049+0000 7f41d88a0700 1 -- 172.21.15.7:0/1137690132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d406cb30 con 0x7f41d4072250 2023-03-03T16:00:03.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.049+0000 7f41c2ffd700 1 -- 172.21.15.7:0/1137690132 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f41c801a020 con 0x7f41d4072250 2023-03-03T16:00:03.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.049+0000 7f41c2ffd700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41b40682f0 0x7f41b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:03.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.050+0000 7f41c2ffd700 1 -- 172.21.15.7:0/1137690132 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f41c8085d70 con 0x7f41d4072250 2023-03-03T16:00:03.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.050+0000 7f41d259c700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41b40682f0 0x7f41b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:03.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.050+0000 7f41d259c700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41b40682f0 0x7f41b406a7a0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f41c4000c00 tx=0x7f41c4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:00:03.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.054+0000 7f41c2ffd700 1 -- 172.21.15.7:0/1137690132 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f41c80521c0 con 0x7f41d4072250 2023-03-03T16:00:03.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.290+0000 7f41d88a0700 1 -- 172.21.15.7:0/1137690132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41d404e6e0 con 0x7f41d4072250 2023-03-03T16:00:03.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.291+0000 7f41c2ffd700 1 -- 172.21.15.7:0/1137690132 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f41c80557e0 con 0x7f41d4072250 2023-03-03T16:00:03.292 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:00:03.292 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:00:03.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 -- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41b40682f0 msgr2=0x7f41b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:03.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41b40682f0 0x7f41b406a7a0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f41c4000c00 tx=0x7f41c4008030 comp rx=0 tx=0).stop 2023-03-03T16:00:03.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 -- 172.21.15.7:0/1137690132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4072250 msgr2=0x7f41d40c53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:03.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4072250 0x7f41d40c53d0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c540 tx=0x7f41c800c850 comp rx=0 tx=0).stop 2023-03-03T16:00:03.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 -- 172.21.15.7:0/1137690132 shutdown_connections 2023-03-03T16:00:03.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f41d414e900 0x7f41d41da730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:03.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f41d4071e80 0x7f41d4142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:03.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41b40682f0 0x7f41b406a7a0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:03.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 --2- 172.21.15.7:0/1137690132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41d4072250 0x7f41d40c53d0 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:03.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 -- 172.21.15.7:0/1137690132 >> 172.21.15.7:0/1137690132 conn(0x7f41d40a94d0 msgr2=0x7f41d40c1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:00:03.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 -- 172.21.15.7:0/1137690132 shutdown_connections 2023-03-03T16:00:03.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:03.293+0000 7f41d88a0700 1 -- 172.21.15.7:0/1137690132 wait complete. 2023-03-03T16:00:03.296 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:00:03.306 INFO:tasks.mds_thrash.fs.[b]:waiting for 43 secs before thrashing 2023-03-03T16:00:11.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:11.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:41.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:41.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:00:46.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:00:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.440+0000 7f8de3b04700 1 Processor -- start 2023-03-03T16:00:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.440+0000 7f8de3b04700 1 -- start start 2023-03-03T16:00:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.441+0000 7f8de3b04700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ddc0c2e30 0x7f8ddc158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.441+0000 7f8de3b04700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc06c6c0 0x7f8ddc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.441+0000 7f8de3b04700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc157ff0 con 0x7f8ddc151cd0 2023-03-03T16:00:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.441+0000 7f8de3b04700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc06d640 con 0x7f8ddc0c2e30 2023-03-03T16:00:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.441+0000 7f8de3b04700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc06cda0 con 0x7f8ddc06c6c0 2023-03-03T16:00:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.441+0000 7f8de18a0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8ddc151cd0 0x7f8ddc154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49374/0 (socket says 172.21.15.7:49374) 2023-03-03T16:00:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.441+0000 7f8de18a0700 1 -- 172.21.15.7:0/1487568128 learned_addr learned my addr 172.21.15.7:0/1487568128 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:00:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de109f700 1 --2- 172.21.15.7:0/1487568128 >> v2:172.21.15.143:3300/0 conn(0x7f8ddc0c2e30 0x7f8ddc158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de20a1700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc06c6c0 0x7f8ddc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724056754 0 0) 0x7f8ddc157ff0 con 0x7f8ddc151cd0 2023-03-03T16:00:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dc80035b0 con 0x7f8ddc151cd0 2023-03-03T16:00:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 799621709 0 0) 0x7f8dc80035b0 con 0x7f8ddc151cd0 2023-03-03T16:00:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc06c6c0 msgr2=0x7f8ddc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:00:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc06c6c0 0x7f8ddc0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 >> v2:172.21.15.143:3300/0 conn(0x7f8ddc0c2e30 msgr2=0x7f8ddc158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:00:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 --2- 172.21.15.7:0/1487568128 >> v2:172.21.15.143:3300/0 conn(0x7f8ddc0c2e30 0x7f8ddc158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc06d080 con 0x7f8ddc151cd0 2023-03-03T16:00:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8dcc001670 con 0x7f8ddc151cd0 2023-03-03T16:00:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de109f700 1 --2- 172.21.15.7:0/1487568128 >> v2:172.21.15.143:3300/0 conn(0x7f8ddc0c2e30 0x7f8ddc158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:00:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de20a1700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc06c6c0 0x7f8ddc0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:00:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 >> v1:172.21.15.7:6789/0 conn(0x7f8ddc151cd0 legacy=0x7f8ddc154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dc8003a70 0x7f8dc8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8004850 0x7f8dc8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc06c6c0 0x7f8dc8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc06d640 con 0x7f8dc8003a70 2023-03-03T16:00:46.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc06cda0 con 0x7f8dc8004850 2023-03-03T16:00:46.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc800ba60 con 0x7f8ddc06c6c0 2023-03-03T16:00:46.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de18a0700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dc8003a70 0x7f8dc8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:46.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de109f700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8004850 0x7f8dc8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:46.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de20a1700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc06c6c0 0x7f8dc8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:46.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de18a0700 1 -- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc06c6c0 msgr2=0x7f8dc8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:00:46.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de18a0700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc06c6c0 0x7f8dc8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de18a0700 1 -- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8004850 msgr2=0x7f8dc8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:00:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de18a0700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8004850 0x7f8dc8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.442+0000 7f8de18a0700 1 -- 172.21.15.7:0/1487568128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd4009a70 con 0x7f8dc8003a70 2023-03-03T16:00:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de109f700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8004850 0x7f8dc8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:00:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de20a1700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc06c6c0 0x7f8dc8005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:00:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de18a0700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dc8003a70 0x7f8dc8003e80 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00c1c0 tx=0x7f8dc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=635a86980b0c5e18 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:00:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc007b70 con 0x7f8dc8003a70 2023-03-03T16:00:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dcc00cd20 con 0x7f8dc8003a70 2023-03-03T16:00:46.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de089e700 1 -- 172.21.15.7:0/1487568128 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc005ba0 con 0x7f8dc8003a70 2023-03-03T16:00:46.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de3b04700 1 -- 172.21.15.7:0/1487568128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dc8003a70 msgr2=0x7f8dc8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:46.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de3b04700 1 --2- 172.21.15.7:0/1487568128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8dc8003a70 0x7f8dc8003e80 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00c1c0 tx=0x7f8dc8005880 comp rx=0 tx=0).stop 2023-03-03T16:00:46.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de20a1700 1 -- 172.21.15.7:0/1487568128 reap_dead start 2023-03-03T16:00:46.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de3b04700 1 -- 172.21.15.7:0/1487568128 shutdown_connections 2023-03-03T16:00:46.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de3b04700 1 -- 172.21.15.7:0/1487568128 >> 172.21.15.7:0/1487568128 conn(0x7f8ddc0a94d0 msgr2=0x7f8ddc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:00:46.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de3b04700 1 -- 172.21.15.7:0/1487568128 shutdown_connections 2023-03-03T16:00:46.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.443+0000 7f8de3b04700 1 -- 172.21.15.7:0/1487568128 wait complete. 2023-03-03T16:00:46.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.445+0000 7f8de3b04700 1 Processor -- start 2023-03-03T16:00:46.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.445+0000 7f8de3b04700 1 -- start start 2023-03-03T16:00:46.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de3b04700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8003a70 0x7f8ddc153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:46.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de3b04700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ddc06b3b0 0x7f8ddc152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:46.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de3b04700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc0bdbe0 0x7f8ddc1531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:46.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de3b04700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc154400 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de3b04700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc0bd880 con 0x7f8dc8003a70 2023-03-03T16:00:46.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de3b04700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc1e1c40 con 0x7f8ddc0bdbe0 2023-03-03T16:00:46.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de109f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ddc06b3b0 0x7f8ddc152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:46.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de109f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ddc06b3b0 0x7f8ddc152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55408/0 (socket says 172.21.15.7:55408) 2023-03-03T16:00:46.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de109f700 1 -- 172.21.15.7:0/811075507 learned_addr learned my addr 172.21.15.7:0/811075507 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:00:46.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de18a0700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8003a70 0x7f8ddc153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:46.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de20a1700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc0bdbe0 0x7f8ddc1531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:46.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de109f700 1 -- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc0bdbe0 msgr2=0x7f8ddc1531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:46.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de109f700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc0bdbe0 0x7f8ddc1531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de109f700 1 -- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8003a70 msgr2=0x7f8ddc153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:46.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de109f700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8003a70 0x7f8ddc153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.446+0000 7f8de109f700 1 -- 172.21.15.7:0/811075507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc157c80 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.447+0000 7f8de18a0700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8003a70 0x7f8ddc153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:00:46.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.447+0000 7f8de109f700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ddc06b3b0 0x7f8ddc152cb0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8010c00 tx=0x7f8dd8009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:00:46.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.447+0000 7f8dd27fc700 1 -- 172.21.15.7:0/811075507 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd800eb10 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.447+0000 7f8dd27fc700 1 -- 172.21.15.7:0/811075507 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd800ec70 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.447+0000 7f8de3b04700 1 -- 172.21.15.7:0/811075507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc06d360 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.447+0000 7f8dd27fc700 1 -- 172.21.15.7:0/811075507 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd801c400 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.447+0000 7f8de3b04700 1 -- 172.21.15.7:0/811075507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc1589c0 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.448+0000 7f8de3b04700 1 -- 172.21.15.7:0/811075507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ddc152b00 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.448+0000 7f8dd27fc700 1 -- 172.21.15.7:0/811075507 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8dd8012040 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.449+0000 7f8dd27fc700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 0x7f8dc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:00:46.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.449+0000 7f8dd27fc700 1 -- 172.21.15.7:0/811075507 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8dd8086110 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.449+0000 7f8de18a0700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 0x7f8dc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:00:46.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.450+0000 7f8de18a0700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 0x7f8dc006a7a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00c160 tx=0x7f8dcc00a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:00:46.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.453+0000 7f8dd27fc700 1 -- 172.21.15.7:0/811075507 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8dd80524b0 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.695+0000 7f8de3b04700 1 -- 172.21.15.7:0/811075507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ddc074bb0 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.695+0000 7f8dd27fc700 1 -- 172.21.15.7:0/811075507 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8dd8055ad0 con 0x7f8ddc06b3b0 2023-03-03T16:00:46.698 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:00:46.698 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:00:46.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 -- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 msgr2=0x7f8dc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:46.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 0x7f8dc006a7a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00c160 tx=0x7f8dcc00a000 comp rx=0 tx=0).stop 2023-03-03T16:00:46.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 -- 172.21.15.7:0/811075507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ddc06b3b0 msgr2=0x7f8ddc152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:00:46.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ddc06b3b0 0x7f8ddc152cb0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8010c00 tx=0x7f8dd8009ec0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 -- 172.21.15.7:0/811075507 shutdown_connections 2023-03-03T16:00:46.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ddc0bdbe0 0x7f8ddc1531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8dc8003a70 0x7f8ddc153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 0x7f8dc006a7a0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 --2- 172.21.15.7:0/811075507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ddc06b3b0 0x7f8ddc152cb0 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:00:46.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 -- 172.21.15.7:0/811075507 >> 172.21.15.7:0/811075507 conn(0x7f8ddc0a94d0 msgr2=0x7f8ddc065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:00:46.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 -- 172.21.15.7:0/811075507 shutdown_connections 2023-03-03T16:00:46.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:00:46.697+0000 7f8de3b04700 1 -- 172.21.15.7:0/811075507 wait complete. 2023-03-03T16:00:46.701 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:00:46.711 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T16:01:11.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:11.482 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:23.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:01:23.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.590+0000 7fd8d6b85700 1 Processor -- start 2023-03-03T16:01:23.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.590+0000 7fd8d6b85700 1 -- start start 2023-03-03T16:01:23.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d6b85700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd8d00caa30 0x7fd8d00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:01:23.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d6b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00c00e0 0x7fd8d0154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:01:23.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d6b85700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d00cb280 con 0x7fd8d00cae00 2023-03-03T16:01:23.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d6b85700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d00c7310 con 0x7fd8d00caa30 2023-03-03T16:01:23.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d6b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d00bfd10 con 0x7fd8d00c00e0 2023-03-03T16:01:23.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d4921700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd8d00caa30 0x7fd8d00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:01:23.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d5122700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd8d00cae00 0x7fd8d00c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49418/0 (socket says 172.21.15.7:49418) 2023-03-03T16:01:23.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d5122700 1 -- 172.21.15.7:0/3826829579 learned_addr learned my addr 172.21.15.7:0/3826829579 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:01:23.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8cffff700 1 --2- 172.21.15.7:0/3826829579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00c00e0 0x7fd8d0154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:01:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d4921700 1 -- 172.21.15.7:0/3826829579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00c00e0 msgr2=0x7fd8d0154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:01:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d4921700 1 --2- 172.21.15.7:0/3826829579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00c00e0 0x7fd8d0154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:01:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d4921700 1 -- 172.21.15.7:0/3826829579 >> v1:172.21.15.7:6789/0 conn(0x7fd8d00cae00 legacy=0x7fd8d00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:01:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8cf7fe700 1 -- 172.21.15.7:0/3826829579 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200430270 0 0) 0x7fd8d00cb280 con 0x7fd8d00cae00 2023-03-03T16:01:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d4921700 1 -- 172.21.15.7:0/3826829579 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d00bfa30 con 0x7fd8d00caa30 2023-03-03T16:01:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.591+0000 7fd8d4921700 1 --2- 172.21.15.7:0/3826829579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00caa30 0x7fd8d00c5c60 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009840 tx=0x7fd8c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7b9b3e262efd2f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:01:23.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8cf7fe700 1 -- 172.21.15.7:0/3826829579 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4011830 con 0x7fd8d00caa30 2023-03-03T16:01:23.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8cf7fe700 1 -- 172.21.15.7:0/3826829579 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c4011990 con 0x7fd8d00caa30 2023-03-03T16:01:23.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8cf7fe700 1 -- 172.21.15.7:0/3826829579 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4011d20 con 0x7fd8d00caa30 2023-03-03T16:01:23.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8d6b85700 1 -- 172.21.15.7:0/3826829579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00caa30 msgr2=0x7fd8d00c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:01:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8d6b85700 1 --2- 172.21.15.7:0/3826829579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00caa30 0x7fd8d00c5c60 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009840 tx=0x7fd8c4009b50 comp rx=0 tx=0).stop 2023-03-03T16:01:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8d6b85700 1 -- 172.21.15.7:0/3826829579 shutdown_connections 2023-03-03T16:01:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8d6b85700 1 --2- 172.21.15.7:0/3826829579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00caa30 0x7fd8d00c5c60 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:01:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8d6b85700 1 --2- 172.21.15.7:0/3826829579 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00c00e0 0x7fd8d0154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:01:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8d6b85700 1 -- 172.21.15.7:0/3826829579 >> 172.21.15.7:0/3826829579 conn(0x7fd8d0066e40 msgr2=0x7fd8d0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:01:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8d6b85700 1 -- 172.21.15.7:0/3826829579 shutdown_connections 2023-03-03T16:01:23.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.592+0000 7fd8d6b85700 1 -- 172.21.15.7:0/3826829579 wait complete. 2023-03-03T16:01:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.594+0000 7fd8d6b85700 1 Processor -- start 2023-03-03T16:01:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.594+0000 7fd8d6b85700 1 -- start start 2023-03-03T16:01:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d6b85700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00c00e0 0x7fd8d00c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:01:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d6b85700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8d00caa30 0x7fd8d0158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:01:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d6b85700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00cae00 0x7fd8d014d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:01:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d6b85700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d014c2d0 con 0x7fd8d00c00e0 2023-03-03T16:01:23.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d6b85700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d01541a0 con 0x7fd8d00caa30 2023-03-03T16:01:23.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d6b85700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d014d110 con 0x7fd8d00cae00 2023-03-03T16:01:23.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d4921700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00c00e0 0x7fd8d00c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:01:23.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d4921700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00c00e0 0x7fd8d00c19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55440/0 (socket says 172.21.15.7:55440) 2023-03-03T16:01:23.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d4921700 1 -- 172.21.15.7:0/1533560642 learned_addr learned my addr 172.21.15.7:0/1533560642 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:01:23.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8cffff700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8d00caa30 0x7fd8d0158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:01:23.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d5122700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00cae00 0x7fd8d014d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:01:23.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d4921700 1 -- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00cae00 msgr2=0x7fd8d014d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:01:23.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d4921700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00cae00 0x7fd8d014d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:01:23.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d4921700 1 -- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8d00caa30 msgr2=0x7fd8d0158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:01:23.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d4921700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8d00caa30 0x7fd8d0158120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:01:23.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d4921700 1 -- 172.21.15.7:0/1533560642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c40094a0 con 0x7fd8d00c00e0 2023-03-03T16:01:23.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.595+0000 7fd8d5122700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00cae00 0x7fd8d014d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:01:23.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.596+0000 7fd8d4921700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00c00e0 0x7fd8d00c19e0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fd8d006b270 tx=0x7fd8c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:01:23.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.596+0000 7fd8cd7fa700 1 -- 172.21.15.7:0/1533560642 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c40045d0 con 0x7fd8d00c00e0 2023-03-03T16:01:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.596+0000 7fd8cd7fa700 1 -- 172.21.15.7:0/1533560642 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c401f070 con 0x7fd8d00c00e0 2023-03-03T16:01:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.596+0000 7fd8d6b85700 1 -- 172.21.15.7:0/1533560642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d01547f0 con 0x7fd8d00c00e0 2023-03-03T16:01:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.597+0000 7fd8cd7fa700 1 -- 172.21.15.7:0/1533560642 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c40038c0 con 0x7fd8d00c00e0 2023-03-03T16:01:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.597+0000 7fd8d6b85700 1 -- 172.21.15.7:0/1533560642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d0154510 con 0x7fd8d00c00e0 2023-03-03T16:01:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.597+0000 7fd8d6b85700 1 -- 172.21.15.7:0/1533560642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d0154ad0 con 0x7fd8d00c00e0 2023-03-03T16:01:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.598+0000 7fd8cd7fa700 1 -- 172.21.15.7:0/1533560642 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd8c400f050 con 0x7fd8d00c00e0 2023-03-03T16:01:23.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.598+0000 7fd8cd7fa700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8b40682f0 0x7fd8b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:01:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.598+0000 7fd8cd7fa700 1 -- 172.21.15.7:0/1533560642 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd8c4021070 con 0x7fd8d00c00e0 2023-03-03T16:01:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.598+0000 7fd8cffff700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8b40682f0 0x7fd8b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:01:23.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.599+0000 7fd8cffff700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8b40682f0 0x7fd8b406a7a0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fd8d014e0d0 tx=0x7fd8bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:01:23.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.603+0000 7fd8cd7fa700 1 -- 172.21.15.7:0/1533560642 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd8c4053e30 con 0x7fd8d00c00e0 2023-03-03T16:01:23.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.841+0000 7fd8d6b85700 1 -- 172.21.15.7:0/1533560642 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8d0074bb0 con 0x7fd8d00c00e0 2023-03-03T16:01:23.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.842+0000 7fd8cd7fa700 1 -- 172.21.15.7:0/1533560642 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd8c4015090 con 0x7fd8d00c00e0 2023-03-03T16:01:23.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:01:23.843 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:01:23.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 -- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8b40682f0 msgr2=0x7fd8b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:01:23.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8b40682f0 0x7fd8b406a7a0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fd8d014e0d0 tx=0x7fd8bc008040 comp rx=0 tx=0).stop 2023-03-03T16:01:23.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 -- 172.21.15.7:0/1533560642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00c00e0 msgr2=0x7fd8d00c19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:01:23.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00c00e0 0x7fd8d00c19e0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fd8d006b270 tx=0x7fd8c4004750 comp rx=0 tx=0).stop 2023-03-03T16:01:23.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 -- 172.21.15.7:0/1533560642 shutdown_connections 2023-03-03T16:01:23.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8d00cae00 0x7fd8d014d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:01:23.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8b40682f0 0x7fd8b406a7a0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:01:23.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8d00c00e0 0x7fd8d00c19e0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:01:23.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 --2- 172.21.15.7:0/1533560642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8d00caa30 0x7fd8d0158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:01:23.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 -- 172.21.15.7:0/1533560642 >> 172.21.15.7:0/1533560642 conn(0x7fd8d0066e40 msgr2=0x7fd8d0065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:01:23.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 -- 172.21.15.7:0/1533560642 shutdown_connections 2023-03-03T16:01:23.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:01:23.844+0000 7fd8d6b85700 1 -- 172.21.15.7:0/1533560642 wait complete. 2023-03-03T16:01:23.847 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:01:23.857 INFO:tasks.mds_thrash.fs.[a]:waiting for 86 secs before thrashing 2023-03-03T16:01:41.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:01:41.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:11.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:11.659 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:41.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:41.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:02:42.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:02:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.850+0000 7fc08f13b700 1 Processor -- start 2023-03-03T16:02:42.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.850+0000 7fc08f13b700 1 -- start start 2023-03-03T16:02:42.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08f13b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc08814ae20 0x7fc088154e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08f13b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc0880c1ac0 0x7fc0880c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08f13b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0881553b0 con 0x7fc0880aabc0 2023-03-03T16:02:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08f13b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0880aa150 con 0x7fc0880c1ac0 2023-03-03T16:02:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08f13b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08806f5b0 con 0x7fc08814ae20 2023-03-03T16:02:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08d6d8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc0880c1ac0 0x7fc0880c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc087fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc0880aabc0 0x7fc088071840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49488/0 (socket says 172.21.15.7:49488) 2023-03-03T16:02:42.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc087fff700 1 -- 172.21.15.7:0/3889903908 learned_addr learned my addr 172.21.15.7:0/3889903908 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:02:42.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08ced7700 1 --2- 172.21.15.7:0/3889903908 >> v2:172.21.15.143:3300/0 conn(0x7fc08814ae20 0x7fc088154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:42.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08d6d8700 1 -- 172.21.15.7:0/3889903908 >> v2:172.21.15.143:3300/0 conn(0x7fc08814ae20 msgr2=0x7fc088154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:02:42.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08d6d8700 1 --2- 172.21.15.7:0/3889903908 >> v2:172.21.15.143:3300/0 conn(0x7fc08814ae20 0x7fc088154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:42.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08d6d8700 1 -- 172.21.15.7:0/3889903908 >> v1:172.21.15.7:6789/0 conn(0x7fc0880aabc0 legacy=0x7fc088071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:42.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08d6d8700 1 -- 172.21.15.7:0/3889903908 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08806f890 con 0x7fc0880c1ac0 2023-03-03T16:02:42.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc0877fe700 1 -- 172.21.15.7:0/3889903908 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823912123 0 0) 0x7fc0881553b0 con 0x7fc0880aabc0 2023-03-03T16:02:42.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08ced7700 1 --2- 172.21.15.7:0/3889903908 >> v2:172.21.15.143:3300/0 conn(0x7fc08814ae20 0x7fc088154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:02:42.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.851+0000 7fc08d6d8700 1 --2- 172.21.15.7:0/3889903908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880c1ac0 0x7fc0880c1f00 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fc088072800 tx=0x7fc08000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb449d04de325370 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:42.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc0877fe700 1 -- 172.21.15.7:0/3889903908 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08000e070 con 0x7fc0880c1ac0 2023-03-03T16:02:42.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc0877fe700 1 -- 172.21.15.7:0/3889903908 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08000bbd0 con 0x7fc0880c1ac0 2023-03-03T16:02:42.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc0877fe700 1 -- 172.21.15.7:0/3889903908 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080012430 con 0x7fc0880c1ac0 2023-03-03T16:02:42.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc08f13b700 1 -- 172.21.15.7:0/3889903908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880c1ac0 msgr2=0x7fc0880c1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:42.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc08f13b700 1 --2- 172.21.15.7:0/3889903908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880c1ac0 0x7fc0880c1f00 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fc088072800 tx=0x7fc08000b330 comp rx=0 tx=0).stop 2023-03-03T16:02:42.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc08f13b700 1 -- 172.21.15.7:0/3889903908 shutdown_connections 2023-03-03T16:02:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc08f13b700 1 --2- 172.21.15.7:0/3889903908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880c1ac0 0x7fc0880c1f00 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc08f13b700 1 --2- 172.21.15.7:0/3889903908 >> v2:172.21.15.143:3300/0 conn(0x7fc08814ae20 0x7fc088154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc08f13b700 1 -- 172.21.15.7:0/3889903908 >> 172.21.15.7:0/3889903908 conn(0x7fc0880c8900 msgr2=0x7fc0880c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:02:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc08f13b700 1 -- 172.21.15.7:0/3889903908 shutdown_connections 2023-03-03T16:02:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.852+0000 7fc08f13b700 1 -- 172.21.15.7:0/3889903908 wait complete. 2023-03-03T16:02:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.853+0000 7fc08f13b700 1 Processor -- start 2023-03-03T16:02:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.854+0000 7fc08f13b700 1 -- start start 2023-03-03T16:02:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.854+0000 7fc08f13b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880aabc0 0x7fc0881ed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.854+0000 7fc08f13b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0880c1ac0 0x7fc0881ed6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.854+0000 7fc08f13b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc08814ae20 0x7fc088145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.854+0000 7fc08f13b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881565f0 con 0x7fc0880aabc0 2023-03-03T16:02:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.854+0000 7fc08f13b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08814e3d0 con 0x7fc0880c1ac0 2023-03-03T16:02:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.854+0000 7fc08f13b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08814e0f0 con 0x7fc08814ae20 2023-03-03T16:02:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.854+0000 7fc08ced7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880aabc0 0x7fc0881ed1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:42.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.854+0000 7fc08ced7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880aabc0 0x7fc0881ed1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55510/0 (socket says 172.21.15.7:55510) 2023-03-03T16:02:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.854+0000 7fc08ced7700 1 -- 172.21.15.7:0/1432836124 learned_addr learned my addr 172.21.15.7:0/1432836124 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:02:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc08d6d8700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc08814ae20 0x7fc088145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc087fff700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0880c1ac0 0x7fc0881ed6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc08ced7700 1 -- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc08814ae20 msgr2=0x7fc088145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc08ced7700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc08814ae20 0x7fc088145d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc08ced7700 1 -- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0880c1ac0 msgr2=0x7fc0881ed6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:42.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc08ced7700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0880c1ac0 0x7fc0881ed6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc08ced7700 1 -- 172.21.15.7:0/1432836124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc080009a70 con 0x7fc0880aabc0 2023-03-03T16:02:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc08d6d8700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc08814ae20 0x7fc088145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:02:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc08ced7700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880aabc0 0x7fc0881ed1a0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00e810 tx=0x7fc07c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc0857fa700 1 -- 172.21.15.7:0/1432836124 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c013610 con 0x7fc0880aabc0 2023-03-03T16:02:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc0857fa700 1 -- 172.21.15.7:0/1432836124 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07c013770 con 0x7fc0880aabc0 2023-03-03T16:02:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc08f13b700 1 -- 172.21.15.7:0/1432836124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0880aa430 con 0x7fc0880aabc0 2023-03-03T16:02:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc0857fa700 1 -- 172.21.15.7:0/1432836124 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c013b20 con 0x7fc0880aabc0 2023-03-03T16:02:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.855+0000 7fc08f13b700 1 -- 172.21.15.7:0/1432836124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08806fb70 con 0x7fc0880aabc0 2023-03-03T16:02:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.856+0000 7fc08f13b700 1 -- 172.21.15.7:0/1432836124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08814b1f0 con 0x7fc0880aabc0 2023-03-03T16:02:42.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.857+0000 7fc0857fa700 1 -- 172.21.15.7:0/1432836124 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc07c023070 con 0x7fc0880aabc0 2023-03-03T16:02:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.857+0000 7fc0857fa700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc06c0682f0 0x7fc06c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.858+0000 7fc0857fa700 1 -- 172.21.15.7:0/1432836124 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc07c085ec0 con 0x7fc0880aabc0 2023-03-03T16:02:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.858+0000 7fc087fff700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc06c0682f0 0x7fc06c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.858+0000 7fc087fff700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc06c0682f0 0x7fc06c06a7a0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fc0881524a0 tx=0x7fc074007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:42.862+0000 7fc0857fa700 1 -- 172.21.15.7:0/1432836124 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc07c052170 con 0x7fc0880aabc0 2023-03-03T16:02:43.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.104+0000 7fc08f13b700 1 -- 172.21.15.7:0/1432836124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc088074bb0 con 0x7fc0880aabc0 2023-03-03T16:02:43.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.105+0000 7fc0857fa700 1 -- 172.21.15.7:0/1432836124 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc07c055790 con 0x7fc0880aabc0 2023-03-03T16:02:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:02:43.106 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:02:43.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 -- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc06c0682f0 msgr2=0x7fc06c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:43.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc06c0682f0 0x7fc06c06a7a0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fc0881524a0 tx=0x7fc074007c90 comp rx=0 tx=0).stop 2023-03-03T16:02:43.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 -- 172.21.15.7:0/1432836124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880aabc0 msgr2=0x7fc0881ed1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:43.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880aabc0 0x7fc0881ed1a0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00e810 tx=0x7fc07c00ebd0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 -- 172.21.15.7:0/1432836124 shutdown_connections 2023-03-03T16:02:43.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc08814ae20 0x7fc088145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc06c0682f0 0x7fc06c06a7a0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0880aabc0 0x7fc0881ed1a0 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 --2- 172.21.15.7:0/1432836124 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0880c1ac0 0x7fc0881ed6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 -- 172.21.15.7:0/1432836124 >> 172.21.15.7:0/1432836124 conn(0x7fc0880c8900 msgr2=0x7fc0881525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:02:43.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 -- 172.21.15.7:0/1432836124 shutdown_connections 2023-03-03T16:02:43.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.107+0000 7fc08f13b700 1 -- 172.21.15.7:0/1432836124 wait complete. 2023-03-03T16:02:43.110 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:02:43.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:02:43.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.243+0000 7f1dee413700 1 Processor -- start 2023-03-03T16:02:43.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.243+0000 7f1dee413700 1 -- start start 2023-03-03T16:02:43.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1dee413700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1de8151d40 0x7f1de80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:43.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1dee413700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de806d430 con 0x7f1de8073c30 2023-03-03T16:02:43.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1dee413700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de806cb00 con 0x7f1de80c2930 2023-03-03T16:02:43.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1dee413700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de806c2a0 con 0x7f1de8151d40 2023-03-03T16:02:43.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1dec9b0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1de8073c30 0x7f1de81581a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49502/0 (socket says 172.21.15.7:49502) 2023-03-03T16:02:43.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1dec9b0700 1 -- 172.21.15.7:0/1512918305 learned_addr learned my addr 172.21.15.7:0/1512918305 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:02:43.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1de7fff700 1 --2- 172.21.15.7:0/1512918305 >> v2:172.21.15.143:3300/0 conn(0x7f1de8151d40 0x7f1de80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:43.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1de6ffd700 1 -- 172.21.15.7:0/1512918305 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675695425 0 0) 0x7f1de806d430 con 0x7f1de8073c30 2023-03-03T16:02:43.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1de6ffd700 1 -- 172.21.15.7:0/1512918305 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dd40035b0 con 0x7f1de8073c30 2023-03-03T16:02:43.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1de7fff700 1 -- 172.21.15.7:0/1512918305 >> v1:172.21.15.143:6789/0 conn(0x7f1de80c2930 legacy=0x7f1de8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:43.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1de7fff700 1 -- 172.21.15.7:0/1512918305 >> v1:172.21.15.7:6789/0 conn(0x7f1de8073c30 legacy=0x7f1de81581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:43.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.244+0000 7f1de7fff700 1 -- 172.21.15.7:0/1512918305 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de806cde0 con 0x7f1de8151d40 2023-03-03T16:02:43.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1de7fff700 1 --2- 172.21.15.7:0/1512918305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de8151d40 0x7f1de80736f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009870 tx=0x7f1dd0009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c6f96463228973c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:43.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1de6ffd700 1 -- 172.21.15.7:0/1512918305 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011830 con 0x7f1de8151d40 2023-03-03T16:02:43.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1de6ffd700 1 -- 172.21.15.7:0/1512918305 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0011990 con 0x7f1de8151d40 2023-03-03T16:02:43.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1de6ffd700 1 -- 172.21.15.7:0/1512918305 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011d20 con 0x7f1de8151d40 2023-03-03T16:02:43.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1dee413700 1 -- 172.21.15.7:0/1512918305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de8151d40 msgr2=0x7f1de80736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:43.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1dee413700 1 --2- 172.21.15.7:0/1512918305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de8151d40 0x7f1de80736f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009870 tx=0x7f1dd0009b80 comp rx=0 tx=0).stop 2023-03-03T16:02:43.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1dee413700 1 -- 172.21.15.7:0/1512918305 shutdown_connections 2023-03-03T16:02:43.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1dee413700 1 --2- 172.21.15.7:0/1512918305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de8151d40 0x7f1de80736f0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1dee413700 1 -- 172.21.15.7:0/1512918305 >> 172.21.15.7:0/1512918305 conn(0x7f1de80a94d0 msgr2=0x7f1de80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:02:43.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1dee413700 1 -- 172.21.15.7:0/1512918305 shutdown_connections 2023-03-03T16:02:43.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.245+0000 7f1dee413700 1 -- 172.21.15.7:0/1512918305 wait complete. 2023-03-03T16:02:43.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.246+0000 7f1dee413700 1 Processor -- start 2023-03-03T16:02:43.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1dee413700 1 -- start start 2023-03-03T16:02:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1dee413700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de8073c30 0x7f1de80c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1dee413700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de80c2930 0x7f1de80c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1dee413700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de8151d40 0x7f1de81ed330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:43.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1dee413700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80beff0 con 0x7f1de8073c30 2023-03-03T16:02:43.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1dee413700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80bfe00 con 0x7f1de8151d40 2023-03-03T16:02:43.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1dee413700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80be430 con 0x7f1de80c2930 2023-03-03T16:02:43.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1de7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de8073c30 0x7f1de80c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:43.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1de7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de8073c30 0x7f1de80c6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55524/0 (socket says 172.21.15.7:55524) 2023-03-03T16:02:43.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1de7fff700 1 -- 172.21.15.7:0/1606823144 learned_addr learned my addr 172.21.15.7:0/1606823144 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:02:43.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1dec9b0700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de8151d40 0x7f1de81ed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:43.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.247+0000 7f1de77fe700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de80c2930 0x7f1de80c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:43.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1de7fff700 1 -- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de80c2930 msgr2=0x7f1de80c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:43.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1de7fff700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de80c2930 0x7f1de80c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1de7fff700 1 -- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de8151d40 msgr2=0x7f1de81ed330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:43.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1de7fff700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de8151d40 0x7f1de81ed330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1de7fff700 1 -- 172.21.15.7:0/1606823144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd00094a0 con 0x7f1de8073c30 2023-03-03T16:02:43.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1de7fff700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de8073c30 0x7f1de80c6a00 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0014040 tx=0x7f1dd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1de4ff9700 1 -- 172.21.15.7:0/1606823144 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd00036a0 con 0x7f1de8073c30 2023-03-03T16:02:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1de4ff9700 1 -- 172.21.15.7:0/1606823144 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd001e070 con 0x7f1de8073c30 2023-03-03T16:02:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1dee413700 1 -- 172.21.15.7:0/1606823144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de806bea0 con 0x7f1de8073c30 2023-03-03T16:02:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1de4ff9700 1 -- 172.21.15.7:0/1606823144 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0003a20 con 0x7f1de8073c30 2023-03-03T16:02:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.248+0000 7f1dee413700 1 -- 172.21.15.7:0/1606823144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de806d0c0 con 0x7f1de8073c30 2023-03-03T16:02:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.249+0000 7f1dee413700 1 -- 172.21.15.7:0/1606823144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de806c030 con 0x7f1de8073c30 2023-03-03T16:02:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.249+0000 7f1de4ff9700 1 -- 172.21.15.7:0/1606823144 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1dd000f050 con 0x7f1de8073c30 2023-03-03T16:02:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.250+0000 7f1de4ff9700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dc80682f0 0x7f1dc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.250+0000 7f1de4ff9700 1 -- 172.21.15.7:0/1606823144 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1dd0087080 con 0x7f1de8073c30 2023-03-03T16:02:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.250+0000 7f1de77fe700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dc80682f0 0x7f1dc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.250+0000 7f1de77fe700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dc80682f0 0x7f1dc806a7a0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8007fd0 tx=0x7f1dd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.254+0000 7f1de4ff9700 1 -- 172.21.15.7:0/1606823144 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1dd0053420 con 0x7f1de8073c30 2023-03-03T16:02:43.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.492+0000 7f1dee413700 1 -- 172.21.15.7:0/1606823144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1de8074550 con 0x7f1de8073c30 2023-03-03T16:02:43.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.492+0000 7f1de4ff9700 1 -- 172.21.15.7:0/1606823144 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1dd0015020 con 0x7f1de8073c30 2023-03-03T16:02:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:02:43.494 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:02:43.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 -- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dc80682f0 msgr2=0x7f1dc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:43.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dc80682f0 0x7f1dc806a7a0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8007fd0 tx=0x7f1dd8009040 comp rx=0 tx=0).stop 2023-03-03T16:02:43.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 -- 172.21.15.7:0/1606823144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de8073c30 msgr2=0x7f1de80c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:43.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de8073c30 0x7f1de80c6a00 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0014040 tx=0x7f1dd0004750 comp rx=0 tx=0).stop 2023-03-03T16:02:43.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 -- 172.21.15.7:0/1606823144 shutdown_connections 2023-03-03T16:02:43.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1de8151d40 0x7f1de81ed330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1dc80682f0 0x7f1dc806a7a0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1de8073c30 0x7f1de80c6a00 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 --2- 172.21.15.7:0/1606823144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1de80c2930 0x7f1de80c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:43.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 -- 172.21.15.7:0/1606823144 >> 172.21.15.7:0/1606823144 conn(0x7f1de80a94d0 msgr2=0x7f1de80c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:02:43.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 -- 172.21.15.7:0/1606823144 shutdown_connections 2023-03-03T16:02:43.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:43.495+0000 7f1dee413700 1 -- 172.21.15.7:0/1606823144 wait complete. 2023-03-03T16:02:43.498 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:02:43.507 INFO:tasks.mds_thrash.fs.[b]:waiting for 87 secs before thrashing 2023-03-03T16:02:49.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:02:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.996+0000 7fb4f212c700 1 Processor -- start 2023-03-03T16:02:49.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.997+0000 7fb4f212c700 1 -- start start 2023-03-03T16:02:49.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.997+0000 7fb4f212c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb4ec06d870 0x7fb4ec1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:49.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.997+0000 7fb4f212c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec0c7a60 con 0x7fb4ec0c1da0 2023-03-03T16:02:49.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.997+0000 7fb4f212c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec0c3620 con 0x7fb4ec0c3e30 2023-03-03T16:02:49.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.997+0000 7fb4f212c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4ec0c2af0 con 0x7fb4ec06d870 2023-03-03T16:02:49.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.997+0000 7fb4eb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb4ec06d870 0x7fb4ec1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4eaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb4ec0c1da0 0x7fb4ec0c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49522/0 (socket says 172.21.15.7:49522) 2023-03-03T16:02:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4eb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb4ec06d870 0x7fb4ec1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55542/0 (socket says 172.21.15.7:55542) 2023-03-03T16:02:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4eaffd700 1 -- 172.21.15.7:0/3860159836 learned_addr learned my addr 172.21.15.7:0/3860159836 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:02:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4eb7fe700 1 -- 172.21.15.7:0/3860159836 >> v1:172.21.15.143:6789/0 conn(0x7fb4ec0c3e30 legacy=0x7fb4ec15ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:02:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4eb7fe700 1 -- 172.21.15.7:0/3860159836 >> v1:172.21.15.7:6789/0 conn(0x7fb4ec0c1da0 legacy=0x7fb4ec0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4eb7fe700 1 -- 172.21.15.7:0/3860159836 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec1545e0 con 0x7fb4ec06d870 2023-03-03T16:02:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4ea7fc700 1 -- 172.21.15.7:0/3860159836 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189334358 0 0) 0x7fb4ec0c7a60 con 0x7fb4ec0c1da0 2023-03-03T16:02:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4eb7fe700 1 --2- 172.21.15.7:0/3860159836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec06d870 0x7fb4ec1550c0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009840 tx=0x7fb4dc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61ad0d9fca348b6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4ea7fc700 1 -- 172.21.15.7:0/3860159836 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc004230 con 0x7fb4ec06d870 2023-03-03T16:02:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4ea7fc700 1 -- 172.21.15.7:0/3860159836 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc004390 con 0x7fb4ec06d870 2023-03-03T16:02:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.998+0000 7fb4ea7fc700 1 -- 172.21.15.7:0/3860159836 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc0036a0 con 0x7fb4ec06d870 2023-03-03T16:02:50.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.999+0000 7fb4f212c700 1 -- 172.21.15.7:0/3860159836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec06d870 msgr2=0x7fb4ec1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:50.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.999+0000 7fb4f212c700 1 --2- 172.21.15.7:0/3860159836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec06d870 0x7fb4ec1550c0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009840 tx=0x7fb4dc009b50 comp rx=0 tx=0).stop 2023-03-03T16:02:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.999+0000 7fb4f212c700 1 -- 172.21.15.7:0/3860159836 shutdown_connections 2023-03-03T16:02:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.999+0000 7fb4f212c700 1 --2- 172.21.15.7:0/3860159836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec06d870 0x7fb4ec1550c0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.999+0000 7fb4f212c700 1 -- 172.21.15.7:0/3860159836 >> 172.21.15.7:0/3860159836 conn(0x7fb4ec06c110 msgr2=0x7fb4ec06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:02:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.999+0000 7fb4f212c700 1 -- 172.21.15.7:0/3860159836 shutdown_connections 2023-03-03T16:02:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:49.999+0000 7fb4f212c700 1 -- 172.21.15.7:0/3860159836 wait complete. 2023-03-03T16:02:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.000+0000 7fb4f212c700 1 Processor -- start 2023-03-03T16:02:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.000+0000 7fb4f212c700 1 -- start start 2023-03-03T16:02:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4f212c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec06d870 0x7fb4ec1ed280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4f212c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c1da0 0x7fb4ec14ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4f212c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0c3e30 0x7fb4ec13b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4f212c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec13c550 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4f212c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec151bf0 con 0x7fb4ec06d870 2023-03-03T16:02:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4f212c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec14eb90 con 0x7fb4ec0c1da0 2023-03-03T16:02:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4ebfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0c3e30 0x7fb4ec13b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4ebfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0c3e30 0x7fb4ec13b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55552/0 (socket says 172.21.15.7:55552) 2023-03-03T16:02:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4ebfff700 1 -- 172.21.15.7:0/3302492177 learned_addr learned my addr 172.21.15.7:0/3302492177 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:02:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4eb7fe700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec06d870 0x7fb4ec1ed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4eaffd700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c1da0 0x7fb4ec14ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4ebfff700 1 -- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c1da0 msgr2=0x7fb4ec14ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4ebfff700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c1da0 0x7fb4ec14ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:50.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4ebfff700 1 -- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec06d870 msgr2=0x7fb4ec1ed280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:50.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4ebfff700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec06d870 0x7fb4ec1ed280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:50.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4ebfff700 1 -- 172.21.15.7:0/3302492177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc0094a0 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4eaffd700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c1da0 0x7fb4ec14ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:02:50.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.001+0000 7fb4eb7fe700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec06d870 0x7fb4ec1ed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:02:50.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.002+0000 7fb4ebfff700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0c3e30 0x7fb4ec13b8a0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400b9a0 tx=0x7fb4e400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:50.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.002+0000 7fb4e3fff700 1 -- 172.21.15.7:0/3302492177 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e400c9b0 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.002+0000 7fb4e3fff700 1 -- 172.21.15.7:0/3302492177 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e400cb10 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.002+0000 7fb4e3fff700 1 -- 172.21.15.7:0/3302492177 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4014440 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.002+0000 7fb4f212c700 1 -- 172.21.15.7:0/3302492177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ec154ba0 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.002+0000 7fb4f212c700 1 -- 172.21.15.7:0/3302492177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ec1548c0 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.003+0000 7fb4f212c700 1 -- 172.21.15.7:0/3302492177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec06dc40 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.003+0000 7fb4e3fff700 1 -- 172.21.15.7:0/3302492177 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb4e401b030 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.004+0000 7fb4e3fff700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 0x7fb4cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:02:50.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.004+0000 7fb4e3fff700 1 -- 172.21.15.7:0/3302492177 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb4e4085d70 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.004+0000 7fb4eb7fe700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 0x7fb4cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:02:50.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.005+0000 7fb4eb7fe700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 0x7fb4cc06a7a0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec13d1f0 tx=0x7fb4dc01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:02:50.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.008+0000 7fb4e3fff700 1 -- 172.21.15.7:0/3302492177 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb4e4052030 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.250+0000 7fb4f212c700 1 -- 172.21.15.7:0/3302492177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4ec074bb0 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.250+0000 7fb4e3fff700 1 -- 172.21.15.7:0/3302492177 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb4e4055650 con 0x7fb4ec0c3e30 2023-03-03T16:02:50.251 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:02:50.251 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:02:50.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.252+0000 7fb4f212c700 1 -- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 msgr2=0x7fb4cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.252+0000 7fb4f212c700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 0x7fb4cc06a7a0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec13d1f0 tx=0x7fb4dc01c040 comp rx=0 tx=0).stop 2023-03-03T16:02:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.252+0000 7fb4f212c700 1 -- 172.21.15.7:0/3302492177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0c3e30 msgr2=0x7fb4ec13b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:02:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.252+0000 7fb4f212c700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0c3e30 0x7fb4ec13b8a0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400b9a0 tx=0x7fb4e400bd60 comp rx=0 tx=0).stop 2023-03-03T16:02:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.252+0000 7fb4f212c700 1 -- 172.21.15.7:0/3302492177 shutdown_connections 2023-03-03T16:02:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.252+0000 7fb4f212c700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 0x7fb4cc06a7a0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.252+0000 7fb4f212c700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0c3e30 0x7fb4ec13b8a0 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.252+0000 7fb4f212c700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec06d870 0x7fb4ec1ed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.252+0000 7fb4f212c700 1 --2- 172.21.15.7:0/3302492177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c1da0 0x7fb4ec14ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:02:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.252+0000 7fb4f212c700 1 -- 172.21.15.7:0/3302492177 >> 172.21.15.7:0/3302492177 conn(0x7fb4ec06c110 msgr2=0x7fb4ec06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:02:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.253+0000 7fb4f212c700 1 -- 172.21.15.7:0/3302492177 shutdown_connections 2023-03-03T16:02:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:02:50.253+0000 7fb4f212c700 1 -- 172.21.15.7:0/3302492177 wait complete. 2023-03-03T16:02:50.255 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:02:50.265 INFO:tasks.mds_thrash.fs.[a]:waiting for 117 secs before thrashing 2023-03-03T16:03:11.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:11.937 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:42.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:03:42.016 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:10.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:04:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.645+0000 7f8fecfee700 1 Processor -- start 2023-03-03T16:04:10.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.645+0000 7f8fecfee700 1 -- start start 2023-03-03T16:04:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.646+0000 7f8fecfee700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8fe814f9d0 0x7f8fe8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:10.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.646+0000 7f8fecfee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe814fda0 0x7f8fe8071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:10.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.646+0000 7f8fecfee700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe80c2f30 con 0x7f8fe80abe20 2023-03-03T16:04:10.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.646+0000 7f8fecfee700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe80c31d0 con 0x7f8fe814f9d0 2023-03-03T16:04:10.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.646+0000 7f8fecfee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe81524a0 con 0x7f8fe814fda0 2023-03-03T16:04:10.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.646+0000 7f8fe659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8fe814f9d0 0x7f8fe8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:10.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.646+0000 7f8fe6d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8fe80abe20 0x7f8fe81529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49578/0 (socket says 172.21.15.7:49578) 2023-03-03T16:04:10.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.646+0000 7f8fe6d9d700 1 -- 172.21.15.7:0/1186336388 learned_addr learned my addr 172.21.15.7:0/1186336388 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:04:10.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.646+0000 7f8fe5d9b700 1 --2- 172.21.15.7:0/1186336388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe814fda0 0x7f8fe8071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:10.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.647+0000 7f8fe659c700 1 -- 172.21.15.7:0/1186336388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe814fda0 msgr2=0x7f8fe8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:04:10.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.647+0000 7f8fe659c700 1 --2- 172.21.15.7:0/1186336388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe814fda0 0x7f8fe8071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:10.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.647+0000 7f8fe659c700 1 -- 172.21.15.7:0/1186336388 >> v1:172.21.15.7:6789/0 conn(0x7f8fe80abe20 legacy=0x7f8fe81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:10.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.647+0000 7f8fe659c700 1 -- 172.21.15.7:0/1186336388 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe80c34b0 con 0x7f8fe814f9d0 2023-03-03T16:04:10.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.647+0000 7f8fe659c700 1 --2- 172.21.15.7:0/1186336388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe814f9d0 0x7f8fe8156430 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8009840 tx=0x7f8fd8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e323803552e1e971 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.647+0000 7f8fe559a700 1 -- 172.21.15.7:0/1186336388 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8011830 con 0x7f8fe814f9d0 2023-03-03T16:04:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.648+0000 7f8fe559a700 1 -- 172.21.15.7:0/1186336388 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd8011990 con 0x7f8fe814f9d0 2023-03-03T16:04:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.648+0000 7f8fe559a700 1 -- 172.21.15.7:0/1186336388 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8011d20 con 0x7f8fe814f9d0 2023-03-03T16:04:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.648+0000 7f8fecfee700 1 -- 172.21.15.7:0/1186336388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe814f9d0 msgr2=0x7f8fe8156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.648+0000 7f8fecfee700 1 --2- 172.21.15.7:0/1186336388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe814f9d0 0x7f8fe8156430 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8009840 tx=0x7f8fd8009b50 comp rx=0 tx=0).stop 2023-03-03T16:04:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.648+0000 7f8fecfee700 1 -- 172.21.15.7:0/1186336388 shutdown_connections 2023-03-03T16:04:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.648+0000 7f8fecfee700 1 --2- 172.21.15.7:0/1186336388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe814f9d0 0x7f8fe8156430 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.648+0000 7f8fecfee700 1 --2- 172.21.15.7:0/1186336388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe814fda0 0x7f8fe8071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.648+0000 7f8fecfee700 1 -- 172.21.15.7:0/1186336388 >> 172.21.15.7:0/1186336388 conn(0x7f8fe80c8900 msgr2=0x7f8fe80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:04:10.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.648+0000 7f8fecfee700 1 -- 172.21.15.7:0/1186336388 shutdown_connections 2023-03-03T16:04:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.648+0000 7f8fecfee700 1 -- 172.21.15.7:0/1186336388 wait complete. 2023-03-03T16:04:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.650+0000 7f8fecfee700 1 Processor -- start 2023-03-03T16:04:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.650+0000 7f8fecfee700 1 -- start start 2023-03-03T16:04:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.650+0000 7f8fecfee700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe80abe20 0x7f8fe806f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.650+0000 7f8fecfee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe814f9d0 0x7f8fe80c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.650+0000 7f8fecfee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe814fda0 0x7f8fe8143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.650+0000 7f8fecfee700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe81ea1e0 con 0x7f8fe80abe20 2023-03-03T16:04:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.650+0000 7f8fecfee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe806f5a0 con 0x7f8fe814fda0 2023-03-03T16:04:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.650+0000 7f8fecfee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe8156760 con 0x7f8fe814f9d0 2023-03-03T16:04:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe80abe20 0x7f8fe806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe80abe20 0x7f8fe806f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55600/0 (socket says 172.21.15.7:55600) 2023-03-03T16:04:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe659c700 1 -- 172.21.15.7:0/2952303423 learned_addr learned my addr 172.21.15.7:0/2952303423 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:04:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe5d9b700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe814f9d0 0x7f8fe80c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe6d9d700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe814fda0 0x7f8fe8143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe659c700 1 -- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe814f9d0 msgr2=0x7f8fe80c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe659c700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe814f9d0 0x7f8fe80c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe659c700 1 -- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe814fda0 msgr2=0x7f8fe8143b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe659c700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe814fda0 0x7f8fe8143b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe659c700 1 -- 172.21.15.7:0/2952303423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd80094a0 con 0x7f8fe80abe20 2023-03-03T16:04:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe6d9d700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe814fda0 0x7f8fe8143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:04:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fe659c700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe80abe20 0x7f8fe806f060 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8009fd0 tx=0x7f8fd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fd6ffd700 1 -- 172.21.15.7:0/2952303423 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd80036a0 con 0x7f8fe80abe20 2023-03-03T16:04:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fd6ffd700 1 -- 172.21.15.7:0/2952303423 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd801f070 con 0x7f8fe80abe20 2023-03-03T16:04:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fecfee700 1 -- 172.21.15.7:0/2952303423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe80ac350 con 0x7f8fe80abe20 2023-03-03T16:04:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fd6ffd700 1 -- 172.21.15.7:0/2952303423 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd8003a60 con 0x7f8fe80abe20 2023-03-03T16:04:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.651+0000 7f8fecfee700 1 -- 172.21.15.7:0/2952303423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe80ab630 con 0x7f8fe80abe20 2023-03-03T16:04:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.652+0000 7f8fecfee700 1 -- 172.21.15.7:0/2952303423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe80c3e10 con 0x7f8fe80abe20 2023-03-03T16:04:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.653+0000 7f8fd6ffd700 1 -- 172.21.15.7:0/2952303423 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8fd800f050 con 0x7f8fe80abe20 2023-03-03T16:04:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.653+0000 7f8fd6ffd700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8fc80682f0 0x7f8fc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.653+0000 7f8fd6ffd700 1 -- 172.21.15.7:0/2952303423 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8fd8021070 con 0x7f8fe80abe20 2023-03-03T16:04:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.653+0000 7f8fe5d9b700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8fc80682f0 0x7f8fc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.654+0000 7f8fe5d9b700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8fc80682f0 0x7f8fc806a7a0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f8fe81501d0 tx=0x7f8fdc009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.657+0000 7f8fd6ffd700 1 -- 172.21.15.7:0/2952303423 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8fd8053dc0 con 0x7f8fe80abe20 2023-03-03T16:04:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.894+0000 7f8fecfee700 1 -- 172.21.15.7:0/2952303423 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fe8074bb0 con 0x7f8fe80abe20 2023-03-03T16:04:10.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.894+0000 7f8fd6ffd700 1 -- 172.21.15.7:0/2952303423 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8fd80573e0 con 0x7f8fe80abe20 2023-03-03T16:04:10.896 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:04:10.897 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:04:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 -- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8fc80682f0 msgr2=0x7f8fc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:10.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8fc80682f0 0x7f8fc806a7a0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f8fe81501d0 tx=0x7f8fdc009210 comp rx=0 tx=0).stop 2023-03-03T16:04:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 -- 172.21.15.7:0/2952303423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe80abe20 msgr2=0x7f8fe806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe80abe20 0x7f8fe806f060 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8009fd0 tx=0x7f8fd8004750 comp rx=0 tx=0).stop 2023-03-03T16:04:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 -- 172.21.15.7:0/2952303423 shutdown_connections 2023-03-03T16:04:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8fe814fda0 0x7f8fe8143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8fc80682f0 0x7f8fc806a7a0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8fe80abe20 0x7f8fe806f060 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:10.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 --2- 172.21.15.7:0/2952303423 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8fe814f9d0 0x7f8fe80c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 -- 172.21.15.7:0/2952303423 >> 172.21.15.7:0/2952303423 conn(0x7f8fe80c8900 msgr2=0x7f8fe80c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:04:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 -- 172.21.15.7:0/2952303423 shutdown_connections 2023-03-03T16:04:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:10.896+0000 7f8fecfee700 1 -- 172.21.15.7:0/2952303423 wait complete. 2023-03-03T16:04:10.900 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:04:10.910 INFO:tasks.mds_thrash.fs.[b]:waiting for 72 secs before thrashing 2023-03-03T16:04:12.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:12.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:42.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:42.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:04:47.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:04:47.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.403+0000 7fe07d9c0700 1 Processor -- start 2023-03-03T16:04:47.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.403+0000 7fe07d9c0700 1 -- start start 2023-03-03T16:04:47.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.404+0000 7fe07d9c0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe0780c1850 0x7fe0780c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:47.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.404+0000 7fe07d9c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 0x7fe0780aa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:47.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.404+0000 7fe07d9c0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe078154790 con 0x7fe07814b060 2023-03-03T16:04:47.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.404+0000 7fe07d9c0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0780c4100 con 0x7fe0780c1850 2023-03-03T16:04:47.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.404+0000 7fe07d9c0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe078154a70 con 0x7fe0780aa170 2023-03-03T16:04:47.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe0767fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe0780c1850 0x7fe0780c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:47.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe076ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fe07814b060 0x7fe07806f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:53136/0 (socket says 172.21.15.7:53136) 2023-03-03T16:04:47.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe076ffd700 1 -- 172.21.15.7:0/1292525409 learned_addr learned my addr 172.21.15.7:0/1292525409 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:04:47.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe0777fe700 1 --2- 172.21.15.7:0/1292525409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 0x7fe0780aa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:47.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe0767fc700 1 -- 172.21.15.7:0/1292525409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 msgr2=0x7fe0780aa5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:04:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe0767fc700 1 --2- 172.21.15.7:0/1292525409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 0x7fe0780aa5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe0767fc700 1 -- 172.21.15.7:0/1292525409 >> v1:172.21.15.143:6789/0 conn(0x7fe07814b060 legacy=0x7fe07806f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe0767fc700 1 -- 172.21.15.7:0/1292525409 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe078154d50 con 0x7fe0780c1850 2023-03-03T16:04:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe075ffb700 1 -- 172.21.15.7:0/1292525409 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794831524 0 0) 0x7fe078154790 con 0x7fe07814b060 2023-03-03T16:04:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe0767fc700 1 --2- 172.21.15.7:0/1292525409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0780c1850 0x7fe0780c1ca0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe060009840 tx=0x7fe060009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af9b7b56d311ed78 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:47.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe075ffb700 1 -- 172.21.15.7:0/1292525409 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060011830 con 0x7fe0780c1850 2023-03-03T16:04:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe075ffb700 1 -- 172.21.15.7:0/1292525409 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe060011990 con 0x7fe0780c1850 2023-03-03T16:04:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.405+0000 7fe075ffb700 1 -- 172.21.15.7:0/1292525409 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060011d20 con 0x7fe0780c1850 2023-03-03T16:04:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.406+0000 7fe07d9c0700 1 -- 172.21.15.7:0/1292525409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0780c1850 msgr2=0x7fe0780c1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.406+0000 7fe07d9c0700 1 --2- 172.21.15.7:0/1292525409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0780c1850 0x7fe0780c1ca0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe060009840 tx=0x7fe060009b50 comp rx=0 tx=0).stop 2023-03-03T16:04:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.406+0000 7fe07d9c0700 1 -- 172.21.15.7:0/1292525409 shutdown_connections 2023-03-03T16:04:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.406+0000 7fe07d9c0700 1 --2- 172.21.15.7:0/1292525409 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 0x7fe0780aa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.406+0000 7fe07d9c0700 1 --2- 172.21.15.7:0/1292525409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0780c1850 0x7fe0780c1ca0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:47.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.406+0000 7fe07d9c0700 1 -- 172.21.15.7:0/1292525409 >> 172.21.15.7:0/1292525409 conn(0x7fe0780c4f20 msgr2=0x7fe0780c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:04:47.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.406+0000 7fe07d9c0700 1 -- 172.21.15.7:0/1292525409 shutdown_connections 2023-03-03T16:04:47.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.406+0000 7fe07d9c0700 1 -- 172.21.15.7:0/1292525409 wait complete. 2023-03-03T16:04:47.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.407+0000 7fe07d9c0700 1 Processor -- start 2023-03-03T16:04:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.407+0000 7fe07d9c0700 1 -- start start 2023-03-03T16:04:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe07d9c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 0x7fe0780ab740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe07d9c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0780c1850 0x7fe0781e6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe07d9c0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe07814b060 0x7fe0781e7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe07d9c0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780c8090 con 0x7fe07814b060 2023-03-03T16:04:47.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe07d9c0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0781ecd20 con 0x7fe0780c1850 2023-03-03T16:04:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe07d9c0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0781edb80 con 0x7fe0780aa170 2023-03-03T16:04:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe0777fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe07814b060 0x7fe0781e7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe0777fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe07814b060 0x7fe0781e7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55646/0 (socket says 172.21.15.7:55646) 2023-03-03T16:04:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe0777fe700 1 -- 172.21.15.7:0/3342899494 learned_addr learned my addr 172.21.15.7:0/3342899494 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:04:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe0767fc700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0780c1850 0x7fe0781e6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.408+0000 7fe076ffd700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 0x7fe0780ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:47.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe0777fe700 1 -- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 msgr2=0x7fe0780ab740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe0777fe700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 0x7fe0780ab740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe0777fe700 1 -- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0780c1850 msgr2=0x7fe0781e6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe0777fe700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0780c1850 0x7fe0781e6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe0777fe700 1 -- 172.21.15.7:0/3342899494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0600094a0 con 0x7fe07814b060 2023-03-03T16:04:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe076ffd700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 0x7fe0780ab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:04:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe0777fe700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe07814b060 0x7fe0781e7070 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7fe070009dd0 tx=0x7fe07000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe06ffff700 1 -- 172.21.15.7:0/3342899494 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070009c40 con 0x7fe07814b060 2023-03-03T16:04:47.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe06ffff700 1 -- 172.21.15.7:0/3342899494 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07000f040 con 0x7fe07814b060 2023-03-03T16:04:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe07d9c0700 1 -- 172.21.15.7:0/3342899494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe078155310 con 0x7fe07814b060 2023-03-03T16:04:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe06ffff700 1 -- 172.21.15.7:0/3342899494 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070014480 con 0x7fe07814b060 2023-03-03T16:04:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.409+0000 7fe07d9c0700 1 -- 172.21.15.7:0/3342899494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe078155030 con 0x7fe07814b060 2023-03-03T16:04:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.410+0000 7fe07d9c0700 1 -- 172.21.15.7:0/3342899494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe078154440 con 0x7fe07814b060 2023-03-03T16:04:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.410+0000 7fe06ffff700 1 -- 172.21.15.7:0/3342899494 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe0700145e0 con 0x7fe07814b060 2023-03-03T16:04:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.411+0000 7fe06ffff700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0580682f0 0x7fe05806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:04:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.411+0000 7fe06ffff700 1 -- 172.21.15.7:0/3342899494 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe070086b50 con 0x7fe07814b060 2023-03-03T16:04:47.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.411+0000 7fe076ffd700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0580682f0 0x7fe05806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:04:47.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.412+0000 7fe076ffd700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0580682f0 0x7fe05806a7a0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fe068000f60 tx=0x7fe068007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:04:47.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.416+0000 7fe06ffff700 1 -- 172.21.15.7:0/3342899494 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe070052ef0 con 0x7fe07814b060 2023-03-03T16:04:47.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.656+0000 7fe07d9c0700 1 -- 172.21.15.7:0/3342899494 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe078074bb0 con 0x7fe07814b060 2023-03-03T16:04:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.656+0000 7fe06ffff700 1 -- 172.21.15.7:0/3342899494 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe070056510 con 0x7fe07814b060 2023-03-03T16:04:47.657 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:04:47.657 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:04:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 -- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0580682f0 msgr2=0x7fe05806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0580682f0 0x7fe05806a7a0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fe068000f60 tx=0x7fe068007d30 comp rx=0 tx=0).stop 2023-03-03T16:04:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 -- 172.21.15.7:0/3342899494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe07814b060 msgr2=0x7fe0781e7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:04:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe07814b060 0x7fe0781e7070 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7fe070009dd0 tx=0x7fe07000c3a0 comp rx=0 tx=0).stop 2023-03-03T16:04:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 -- 172.21.15.7:0/3342899494 shutdown_connections 2023-03-03T16:04:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0580682f0 0x7fe05806a7a0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe07814b060 0x7fe0781e7070 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0780aa170 0x7fe0780ab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 --2- 172.21.15.7:0/3342899494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0780c1850 0x7fe0781e6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:04:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 -- 172.21.15.7:0/3342899494 >> 172.21.15.7:0/3342899494 conn(0x7fe0780c4f20 msgr2=0x7fe078066040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:04:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.658+0000 7fe07d9c0700 1 -- 172.21.15.7:0/3342899494 shutdown_connections 2023-03-03T16:04:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:04:47.659+0000 7fe07d9c0700 1 -- 172.21.15.7:0/3342899494 wait complete. 2023-03-03T16:04:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:04:47.671 INFO:tasks.mds_thrash.fs.[a]:waiting for 96 secs before thrashing 2023-03-03T16:05:12.195 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:12.197 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:22.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:05:23.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.048+0000 7f535fe90700 1 Processor -- start 2023-03-03T16:05:23.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.048+0000 7f535fe90700 1 -- start start 2023-03-03T16:05:23.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535fe90700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5358151d40 0x7f53580736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:23.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535fe90700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53580c2930 0x7f5358157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:23.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535fe90700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535806d430 con 0x7f5358073c30 2023-03-03T16:05:23.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535fe90700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f535806c2a0 con 0x7f5358151d40 2023-03-03T16:05:23.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535fe90700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53581586e0 con 0x7f53580c2930 2023-03-03T16:05:23.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535dc2c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5358151d40 0x7f53580736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:23.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535e42d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5358073c30 0x7f5358157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49668/0 (socket says 172.21.15.7:49668) 2023-03-03T16:05:23.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535e42d700 1 -- 172.21.15.7:0/3048256672 learned_addr learned my addr 172.21.15.7:0/3048256672 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:05:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535d42b700 1 --2- 172.21.15.7:0/3048256672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53580c2930 0x7f5358157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535dc2c700 1 -- 172.21.15.7:0/3048256672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53580c2930 msgr2=0x7f5358157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:05:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535dc2c700 1 --2- 172.21.15.7:0/3048256672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53580c2930 0x7f5358157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535dc2c700 1 -- 172.21.15.7:0/3048256672 >> v1:172.21.15.7:6789/0 conn(0x7f5358073c30 legacy=0x7f5358157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535dc2c700 1 -- 172.21.15.7:0/3048256672 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535806cb00 con 0x7f5358151d40 2023-03-03T16:05:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535cc2a700 1 -- 172.21.15.7:0/3048256672 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818854649 0 0) 0x7f535806d430 con 0x7f5358073c30 2023-03-03T16:05:23.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.049+0000 7f535dc2c700 1 --2- 172.21.15.7:0/3048256672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358151d40 0x7f53580736f0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f5354009840 tx=0x7f5354009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f4ebd7817f4345 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535cc2a700 1 -- 172.21.15.7:0/3048256672 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354011830 con 0x7f5358151d40 2023-03-03T16:05:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535cc2a700 1 -- 172.21.15.7:0/3048256672 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5354011990 con 0x7f5358151d40 2023-03-03T16:05:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535cc2a700 1 -- 172.21.15.7:0/3048256672 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354011d20 con 0x7f5358151d40 2023-03-03T16:05:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535fe90700 1 -- 172.21.15.7:0/3048256672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358151d40 msgr2=0x7f53580736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535fe90700 1 --2- 172.21.15.7:0/3048256672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358151d40 0x7f53580736f0 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f5354009840 tx=0x7f5354009b50 comp rx=0 tx=0).stop 2023-03-03T16:05:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535fe90700 1 -- 172.21.15.7:0/3048256672 shutdown_connections 2023-03-03T16:05:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535fe90700 1 --2- 172.21.15.7:0/3048256672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358151d40 0x7f53580736f0 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535fe90700 1 --2- 172.21.15.7:0/3048256672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53580c2930 0x7f5358157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535fe90700 1 -- 172.21.15.7:0/3048256672 >> 172.21.15.7:0/3048256672 conn(0x7f53580a94d0 msgr2=0x7f53580a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:05:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535fe90700 1 -- 172.21.15.7:0/3048256672 shutdown_connections 2023-03-03T16:05:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.050+0000 7f535fe90700 1 -- 172.21.15.7:0/3048256672 wait complete. 2023-03-03T16:05:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.052+0000 7f535fe90700 1 Processor -- start 2023-03-03T16:05:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.052+0000 7f535fe90700 1 -- start start 2023-03-03T16:05:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535fe90700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358073c30 0x7f53581eb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535fe90700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53580c2930 0x7f53581eb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535fe90700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358151d40 0x7f53580c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535fe90700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53580ab110 con 0x7f5358073c30 2023-03-03T16:05:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535fe90700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53580aabe0 con 0x7f5358151d40 2023-03-03T16:05:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535fe90700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53580aae30 con 0x7f53580c2930 2023-03-03T16:05:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535dc2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358073c30 0x7f53581eb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535dc2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358073c30 0x7f53581eb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55690/0 (socket says 172.21.15.7:55690) 2023-03-03T16:05:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535dc2c700 1 -- 172.21.15.7:0/2579654458 learned_addr learned my addr 172.21.15.7:0/2579654458 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:05:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535d42b700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53580c2930 0x7f53581eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535e42d700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358151d40 0x7f53580c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535dc2c700 1 -- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53580c2930 msgr2=0x7f53581eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535dc2c700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53580c2930 0x7f53581eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535dc2c700 1 -- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358151d40 msgr2=0x7f53580c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535dc2c700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358151d40 0x7f53580c01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:23.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535dc2c700 1 -- 172.21.15.7:0/2579654458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53540094a0 con 0x7f5358073c30 2023-03-03T16:05:23.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535e42d700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358151d40 0x7f53580c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:05:23.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f535dc2c700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358073c30 0x7f53581eb0e0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f53540090b0 tx=0x7f5354004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:23.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.053+0000 7f534e7fc700 1 -- 172.21.15.7:0/2579654458 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53540036a0 con 0x7f5358073c30 2023-03-03T16:05:23.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.054+0000 7f534e7fc700 1 -- 172.21.15.7:0/2579654458 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535401f070 con 0x7f5358073c30 2023-03-03T16:05:23.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.054+0000 7f534e7fc700 1 -- 172.21.15.7:0/2579654458 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354003a20 con 0x7f5358073c30 2023-03-03T16:05:23.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.054+0000 7f535fe90700 1 -- 172.21.15.7:0/2579654458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535806d0c0 con 0x7f5358073c30 2023-03-03T16:05:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.054+0000 7f535fe90700 1 -- 172.21.15.7:0/2579654458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535806cde0 con 0x7f5358073c30 2023-03-03T16:05:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.055+0000 7f535fe90700 1 -- 172.21.15.7:0/2579654458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f535806bea0 con 0x7f5358073c30 2023-03-03T16:05:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.055+0000 7f534e7fc700 1 -- 172.21.15.7:0/2579654458 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f535400f050 con 0x7f5358073c30 2023-03-03T16:05:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.055+0000 7f534e7fc700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 0x7f533c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.056+0000 7f534e7fc700 1 -- 172.21.15.7:0/2579654458 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5354021070 con 0x7f5358073c30 2023-03-03T16:05:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.056+0000 7f535d42b700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 0x7f533c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.056+0000 7f535d42b700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 0x7f533c06a7a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f53581ed0d0 tx=0x7f5348008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.060+0000 7f534e7fc700 1 -- 172.21.15.7:0/2579654458 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5354053d40 con 0x7f5358073c30 2023-03-03T16:05:23.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.302+0000 7f535fe90700 1 -- 172.21.15.7:0/2579654458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53581ed170 con 0x7f5358073c30 2023-03-03T16:05:23.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.302+0000 7f534e7fc700 1 -- 172.21.15.7:0/2579654458 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f53581ed170 con 0x7f5358073c30 2023-03-03T16:05:23.304 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:05:23.305 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:05:23.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.304+0000 7f535fe90700 1 -- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 msgr2=0x7f533c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:23.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.304+0000 7f535fe90700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 0x7f533c06a7a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f53581ed0d0 tx=0x7f5348008040 comp rx=0 tx=0).stop 2023-03-03T16:05:23.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.304+0000 7f535fe90700 1 -- 172.21.15.7:0/2579654458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358073c30 msgr2=0x7f53581eb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:23.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.304+0000 7f535fe90700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358073c30 0x7f53581eb0e0 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f53540090b0 tx=0x7f5354004750 comp rx=0 tx=0).stop 2023-03-03T16:05:23.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.304+0000 7f535fe90700 1 -- 172.21.15.7:0/2579654458 shutdown_connections 2023-03-03T16:05:23.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.304+0000 7f535fe90700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358151d40 0x7f53580c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:23.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.304+0000 7f535fe90700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f533c0682f0 0x7f533c06a7a0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:23.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.304+0000 7f535fe90700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5358073c30 0x7f53581eb0e0 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:23.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.304+0000 7f535fe90700 1 --2- 172.21.15.7:0/2579654458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53580c2930 0x7f53581eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:23.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.304+0000 7f535fe90700 1 -- 172.21.15.7:0/2579654458 >> 172.21.15.7:0/2579654458 conn(0x7f53580a94d0 msgr2=0x7f535806af00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:05:23.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.305+0000 7f535fe90700 1 -- 172.21.15.7:0/2579654458 shutdown_connections 2023-03-03T16:05:23.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:23.305+0000 7f535fe90700 1 -- 172.21.15.7:0/2579654458 wait complete. 2023-03-03T16:05:23.308 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:05:23.318 INFO:tasks.mds_thrash.fs.[b]:waiting for 20 secs before thrashing 2023-03-03T16:05:42.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:42.270 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:05:43.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:05:43.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.458+0000 7f1fc2173700 1 Processor -- start 2023-03-03T16:05:43.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.459+0000 7f1fc2173700 1 -- start start 2023-03-03T16:05:43.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.459+0000 7f1fc2173700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1fbc14de50 0x7f1fbc06a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:43.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.459+0000 7f1fc2173700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1fbc06a7f0 0x7f1fbc0c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:43.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.459+0000 7f1fc2173700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fbc0a65b0 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.459+0000 7f1fc2173700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc154e00 con 0x7f1fbc14de50 2023-03-03T16:05:43.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.459+0000 7f1fc2173700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fbc0c3370 con 0x7f1fbc06a7f0 2023-03-03T16:05:43.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fbb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1fbc14de50 0x7f1fbc06a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:43.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fbaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1fbc0c65c0 0x7f1fbc0c2720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49692/0 (socket says 172.21.15.7:49692) 2023-03-03T16:05:43.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fbaffd700 1 -- 172.21.15.7:0/2455608625 learned_addr learned my addr 172.21.15.7:0/2455608625 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:05:43.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fbbfff700 1 --2- 172.21.15.7:0/2455608625 >> v2:172.21.15.143:3300/0 conn(0x7f1fbc06a7f0 0x7f1fbc0c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:43.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fbb7fe700 1 -- 172.21.15.7:0/2455608625 >> v2:172.21.15.143:3300/0 conn(0x7f1fbc06a7f0 msgr2=0x7f1fbc0c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:05:43.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fbb7fe700 1 --2- 172.21.15.7:0/2455608625 >> v2:172.21.15.143:3300/0 conn(0x7f1fbc06a7f0 0x7f1fbc0c2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:43.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fbb7fe700 1 -- 172.21.15.7:0/2455608625 >> v1:172.21.15.7:6789/0 conn(0x7f1fbc0c65c0 legacy=0x7f1fbc0c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:43.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fba7fc700 1 -- 172.21.15.7:0/2455608625 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255964307 0 0) 0x7f1fbc0a65b0 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fbb7fe700 1 -- 172.21.15.7:0/2455608625 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc1541f0 con 0x7f1fbc14de50 2023-03-03T16:05:43.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fbb7fe700 1 --2- 172.21.15.7:0/2455608625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc14de50 0x7f1fbc06a2b0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0c5f00 tx=0x7f1fac00bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c17eaab9c17af1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:43.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fba7fc700 1 -- 172.21.15.7:0/2455608625 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac00dc30 con 0x7f1fbc14de50 2023-03-03T16:05:43.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fba7fc700 1 -- 172.21.15.7:0/2455608625 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fac00dd90 con 0x7f1fbc14de50 2023-03-03T16:05:43.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.460+0000 7f1fba7fc700 1 -- 172.21.15.7:0/2455608625 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac010690 con 0x7f1fbc14de50 2023-03-03T16:05:43.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.461+0000 7f1fc2173700 1 -- 172.21.15.7:0/2455608625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc14de50 msgr2=0x7f1fbc06a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:43.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.461+0000 7f1fc2173700 1 --2- 172.21.15.7:0/2455608625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc14de50 0x7f1fbc06a2b0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc0c5f00 tx=0x7f1fac00bdf0 comp rx=0 tx=0).stop 2023-03-03T16:05:43.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.461+0000 7f1fc2173700 1 -- 172.21.15.7:0/2455608625 shutdown_connections 2023-03-03T16:05:43.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.461+0000 7f1fc2173700 1 --2- 172.21.15.7:0/2455608625 >> v2:172.21.15.143:3300/0 conn(0x7f1fbc06a7f0 0x7f1fbc0c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:43.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.461+0000 7f1fc2173700 1 --2- 172.21.15.7:0/2455608625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc14de50 0x7f1fbc06a2b0 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:43.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.461+0000 7f1fc2173700 1 -- 172.21.15.7:0/2455608625 >> 172.21.15.7:0/2455608625 conn(0x7f1fbc0a9430 msgr2=0x7f1fbc0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:05:43.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.461+0000 7f1fc2173700 1 -- 172.21.15.7:0/2455608625 shutdown_connections 2023-03-03T16:05:43.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.461+0000 7f1fc2173700 1 -- 172.21.15.7:0/2455608625 wait complete. 2023-03-03T16:05:43.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.462+0000 7f1fc2173700 1 Processor -- start 2023-03-03T16:05:43.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.462+0000 7f1fc2173700 1 -- start start 2023-03-03T16:05:43.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fc2173700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fbc06a7f0 0x7f1fbc1ea250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:43.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fc2173700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc0c65c0 0x7f1fbc0c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:43.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fc2173700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc14de50 0x7f1fbc1ecd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:43.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fc2173700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc1e7420 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fc2173700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc1e9790 con 0x7f1fbc06a7f0 2023-03-03T16:05:43.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fc2173700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc070e20 con 0x7f1fbc14de50 2023-03-03T16:05:43.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc0c65c0 0x7f1fbc0c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:43.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc0c65c0 0x7f1fbc0c00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55718/0 (socket says 172.21.15.7:55718) 2023-03-03T16:05:43.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbaffd700 1 -- 172.21.15.7:0/1550428020 learned_addr learned my addr 172.21.15.7:0/1550428020 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:05:43.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbbfff700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc14de50 0x7f1fbc1ecd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:43.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbb7fe700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fbc06a7f0 0x7f1fbc1ea250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:43.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbaffd700 1 -- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc14de50 msgr2=0x7f1fbc1ecd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:43.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbaffd700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc14de50 0x7f1fbc1ecd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:43.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbaffd700 1 -- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fbc06a7f0 msgr2=0x7f1fbc1ea250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:43.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbaffd700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fbc06a7f0 0x7f1fbc1ea250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:43.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbaffd700 1 -- 172.21.15.7:0/1550428020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac00b740 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.463+0000 7f1fbbfff700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc14de50 0x7f1fbc1ecd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:05:43.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.464+0000 7f1fbaffd700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc0c65c0 0x7f1fbc0c00e0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c540 tx=0x7f1fa400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:43.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.464+0000 7f1fb3fff700 1 -- 172.21.15.7:0/1550428020 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4004c80 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.464+0000 7f1fb3fff700 1 -- 172.21.15.7:0/1550428020 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa4004de0 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.464+0000 7f1fc2173700 1 -- 172.21.15.7:0/1550428020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fbc1547b0 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.464+0000 7f1fb3fff700 1 -- 172.21.15.7:0/1550428020 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4007310 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.464+0000 7f1fc2173700 1 -- 172.21.15.7:0/1550428020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fbc1544d0 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.465+0000 7f1fc2173700 1 -- 172.21.15.7:0/1550428020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fbc154a90 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.465+0000 7f1fb3fff700 1 -- 172.21.15.7:0/1550428020 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1fa401a020 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.466+0000 7f1fb3fff700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f9c0682f0 0x7f1f9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:43.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.466+0000 7f1fb3fff700 1 -- 172.21.15.7:0/1550428020 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1fa4085d10 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.466+0000 7f1fbb7fe700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f9c0682f0 0x7f1f9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:43.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.467+0000 7f1fbb7fe700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f9c0682f0 0x7f1f9c06a7a0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00ba60 tx=0x7f1fac013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:43.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.470+0000 7f1fb3fff700 1 -- 172.21.15.7:0/1550428020 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1fa40520b0 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.710+0000 7f1fc2173700 1 -- 172.21.15.7:0/1550428020 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fbc1ea0a0 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.710+0000 7f1fb3fff700 1 -- 172.21.15.7:0/1550428020 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1fa40556d0 con 0x7f1fbc0c65c0 2023-03-03T16:05:43.711 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:05:43.712 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:05:43.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.712+0000 7f1fc2173700 1 -- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f9c0682f0 msgr2=0x7f1f9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:43.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.712+0000 7f1fc2173700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f9c0682f0 0x7f1f9c06a7a0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f1fac00ba60 tx=0x7f1fac013000 comp rx=0 tx=0).stop 2023-03-03T16:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.712+0000 7f1fc2173700 1 -- 172.21.15.7:0/1550428020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc0c65c0 msgr2=0x7f1fbc0c00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.712+0000 7f1fc2173700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc0c65c0 0x7f1fbc0c00e0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c540 tx=0x7f1fa400c850 comp rx=0 tx=0).stop 2023-03-03T16:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.712+0000 7f1fc2173700 1 -- 172.21.15.7:0/1550428020 shutdown_connections 2023-03-03T16:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.712+0000 7f1fc2173700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fbc14de50 0x7f1fbc1ecd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.712+0000 7f1fc2173700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fbc06a7f0 0x7f1fbc1ea250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:43.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.712+0000 7f1fc2173700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f9c0682f0 0x7f1f9c06a7a0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.713+0000 7f1fc2173700 1 --2- 172.21.15.7:0/1550428020 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fbc0c65c0 0x7f1fbc0c00e0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.713+0000 7f1fc2173700 1 -- 172.21.15.7:0/1550428020 >> 172.21.15.7:0/1550428020 conn(0x7f1fbc0a9430 msgr2=0x7f1fbc0ab6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:05:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.713+0000 7f1fc2173700 1 -- 172.21.15.7:0/1550428020 shutdown_connections 2023-03-03T16:05:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:43.713+0000 7f1fc2173700 1 -- 172.21.15.7:0/1550428020 wait complete. 2023-03-03T16:05:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:05:43.725 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before thrashing 2023-03-03T16:05:46.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:05:46.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.864+0000 7f4972e47700 1 Processor -- start 2023-03-03T16:05:46.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.865+0000 7f4972e47700 1 -- start start 2023-03-03T16:05:46.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.865+0000 7f4972e47700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f496c14f9d0 0x7f496c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:46.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.865+0000 7f4972e47700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f496c0abe20 0x7f496c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:46.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.865+0000 7f4972e47700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c0c2f30 con 0x7f496c14fda0 2023-03-03T16:05:46.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.865+0000 7f4972e47700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f496c0c31d0 con 0x7f496c14f9d0 2023-03-03T16:05:46.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.865+0000 7f4972e47700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496c1524a0 con 0x7f496c0abe20 2023-03-03T16:05:46.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.865+0000 7f4970be3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f496c14f9d0 0x7f496c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:46.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f4970be3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f496c14f9d0 0x7f496c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55724/0 (socket says 172.21.15.7:55724) 2023-03-03T16:05:46.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f4970be3700 1 -- 172.21.15.7:0/1258671760 learned_addr learned my addr 172.21.15.7:0/1258671760 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:05:46.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f49713e4700 1 --2- 172.21.15.7:0/1258671760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f496c0abe20 0x7f496c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:46.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f4970be3700 1 -- 172.21.15.7:0/1258671760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f496c0abe20 msgr2=0x7f496c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:05:46.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f4970be3700 1 --2- 172.21.15.7:0/1258671760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f496c0abe20 0x7f496c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:46.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f4970be3700 1 -- 172.21.15.7:0/1258671760 >> v1:172.21.15.143:6789/0 conn(0x7f496c14fda0 legacy=0x7f496c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:46.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f4970be3700 1 -- 172.21.15.7:0/1258671760 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c0c34b0 con 0x7f496c14f9d0 2023-03-03T16:05:46.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f4970be3700 1 --2- 172.21.15.7:0/1258671760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c14f9d0 0x7f496c156430 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f49600098f0 tx=0x7f4960009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcefa8fac0c31f24 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:46.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f496b7fe700 1 -- 172.21.15.7:0/1258671760 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4960011830 con 0x7f496c14f9d0 2023-03-03T16:05:46.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f496b7fe700 1 -- 172.21.15.7:0/1258671760 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4960011990 con 0x7f496c14f9d0 2023-03-03T16:05:46.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.866+0000 7f496b7fe700 1 -- 172.21.15.7:0/1258671760 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4960011d20 con 0x7f496c14f9d0 2023-03-03T16:05:46.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.867+0000 7f4972e47700 1 -- 172.21.15.7:0/1258671760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c14f9d0 msgr2=0x7f496c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:46.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.867+0000 7f4972e47700 1 --2- 172.21.15.7:0/1258671760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c14f9d0 0x7f496c156430 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f49600098f0 tx=0x7f4960009c00 comp rx=0 tx=0).stop 2023-03-03T16:05:46.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.867+0000 7f4972e47700 1 -- 172.21.15.7:0/1258671760 shutdown_connections 2023-03-03T16:05:46.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.867+0000 7f4972e47700 1 --2- 172.21.15.7:0/1258671760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f496c0abe20 0x7f496c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:46.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.867+0000 7f4972e47700 1 --2- 172.21.15.7:0/1258671760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c14f9d0 0x7f496c156430 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:46.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.867+0000 7f4972e47700 1 -- 172.21.15.7:0/1258671760 >> 172.21.15.7:0/1258671760 conn(0x7f496c0c8900 msgr2=0x7f496c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:05:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.867+0000 7f4972e47700 1 -- 172.21.15.7:0/1258671760 shutdown_connections 2023-03-03T16:05:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.867+0000 7f4972e47700 1 -- 172.21.15.7:0/1258671760 wait complete. 2023-03-03T16:05:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.868+0000 7f4972e47700 1 Processor -- start 2023-03-03T16:05:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.868+0000 7f4972e47700 1 -- start start 2023-03-03T16:05:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4972e47700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c0abe20 0x7f496c1e5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4972e47700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f496c14f9d0 0x7f496c1e8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4972e47700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f496c14fda0 0x7f496c149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4972e47700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c150bf0 con 0x7f496c0abe20 2023-03-03T16:05:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4972e47700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c1e54a0 con 0x7f496c14f9d0 2023-03-03T16:05:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4972e47700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496c06f230 con 0x7f496c14fda0 2023-03-03T16:05:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4970be3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c0abe20 0x7f496c1e5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4970be3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c0abe20 0x7f496c1e5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55730/0 (socket says 172.21.15.7:55730) 2023-03-03T16:05:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4970be3700 1 -- 172.21.15.7:0/3142525094 learned_addr learned my addr 172.21.15.7:0/3142525094 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:05:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f49713e4700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f496c14fda0 0x7f496c149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:46.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f496bfff700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f496c14f9d0 0x7f496c1e8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:46.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4970be3700 1 -- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f496c14fda0 msgr2=0x7f496c149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:46.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4970be3700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f496c14fda0 0x7f496c149ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:46.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4970be3700 1 -- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f496c14f9d0 msgr2=0x7f496c1e8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:46.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4970be3700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f496c14f9d0 0x7f496c1e8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:46.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.869+0000 7f4970be3700 1 -- 172.21.15.7:0/3142525094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4960009550 con 0x7f496c0abe20 2023-03-03T16:05:46.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.870+0000 7f4970be3700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c0abe20 0x7f496c1e5f40 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f4960000c00 tx=0x7f4960004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:46.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.870+0000 7f49697fa700 1 -- 172.21.15.7:0/3142525094 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49600038a0 con 0x7f496c0abe20 2023-03-03T16:05:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.870+0000 7f49697fa700 1 -- 172.21.15.7:0/3142525094 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4960003a00 con 0x7f496c0abe20 2023-03-03T16:05:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.870+0000 7f4972e47700 1 -- 172.21.15.7:0/3142525094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496c0ac350 con 0x7f496c0abe20 2023-03-03T16:05:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.870+0000 7f49697fa700 1 -- 172.21.15.7:0/3142525094 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4960003d90 con 0x7f496c0abe20 2023-03-03T16:05:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.870+0000 7f4972e47700 1 -- 172.21.15.7:0/3142525094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496c0ab630 con 0x7f496c0abe20 2023-03-03T16:05:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.871+0000 7f4972e47700 1 -- 172.21.15.7:0/3142525094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f496c071fa0 con 0x7f496c0abe20 2023-03-03T16:05:46.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.871+0000 7f49697fa700 1 -- 172.21.15.7:0/3142525094 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f496000f050 con 0x7f496c0abe20 2023-03-03T16:05:46.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.872+0000 7f49697fa700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49500682f0 0x7f495006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:05:46.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.872+0000 7f49697fa700 1 -- 172.21.15.7:0/3142525094 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4960087bc0 con 0x7f496c0abe20 2023-03-03T16:05:46.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.872+0000 7f496bfff700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49500682f0 0x7f495006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:05:46.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.873+0000 7f496bfff700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49500682f0 0x7f495006a7a0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f496c155800 tx=0x7f4958009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:05:46.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:46.876+0000 7f49697fa700 1 -- 172.21.15.7:0/3142525094 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4960054000 con 0x7f496c0abe20 2023-03-03T16:05:47.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.112+0000 7f4972e47700 1 -- 172.21.15.7:0/3142525094 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f496c074bb0 con 0x7f496c0abe20 2023-03-03T16:05:47.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.113+0000 7f49697fa700 1 -- 172.21.15.7:0/3142525094 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4960057620 con 0x7f496c0abe20 2023-03-03T16:05:47.114 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:05:47.114 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:05:47.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 -- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49500682f0 msgr2=0x7f495006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:47.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49500682f0 0x7f495006a7a0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f496c155800 tx=0x7f4958009040 comp rx=0 tx=0).stop 2023-03-03T16:05:47.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 -- 172.21.15.7:0/3142525094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c0abe20 msgr2=0x7f496c1e5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:05:47.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c0abe20 0x7f496c1e5f40 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f4960000c00 tx=0x7f4960004750 comp rx=0 tx=0).stop 2023-03-03T16:05:47.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 -- 172.21.15.7:0/3142525094 shutdown_connections 2023-03-03T16:05:47.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f496c14fda0 0x7f496c149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:47.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49500682f0 0x7f495006a7a0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:47.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f496c0abe20 0x7f496c1e5f40 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:47.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 --2- 172.21.15.7:0/3142525094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f496c14f9d0 0x7f496c1e8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:05:47.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 -- 172.21.15.7:0/3142525094 >> 172.21.15.7:0/3142525094 conn(0x7f496c0c8900 msgr2=0x7f496c0c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:05:47.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 -- 172.21.15.7:0/3142525094 shutdown_connections 2023-03-03T16:05:47.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:05:47.115+0000 7f4972e47700 1 -- 172.21.15.7:0/3142525094 wait complete. 2023-03-03T16:05:47.118 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:05:47.127 INFO:tasks.mds_thrash.fs.[b]:waiting for 59 secs before thrashing 2023-03-03T16:06:12.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:12.334 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:23.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:06:23.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.811+0000 7fb9c2a67700 1 Processor -- start 2023-03-03T16:06:23.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.811+0000 7fb9c2a67700 1 -- start start 2023-03-03T16:06:23.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.811+0000 7fb9c2a67700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb9bc1530d0 0x7fb9bc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:23.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9c2a67700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9bc0c2f30 con 0x7fb9bc14f9d0 2023-03-03T16:06:23.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9c2a67700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9bc0c31d0 con 0x7fb9bc14fda0 2023-03-03T16:06:23.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9c2a67700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9bc1524a0 con 0x7fb9bc1530d0 2023-03-03T16:06:23.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9c0803700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb9bc14f9d0 0x7fb9bc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49746/0 (socket says 172.21.15.7:49746) 2023-03-03T16:06:23.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9c0803700 1 -- 172.21.15.7:0/467165438 learned_addr learned my addr 172.21.15.7:0/467165438 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:06:23.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9c1004700 1 --2- 172.21.15.7:0/467165438 >> v2:172.21.15.7:3300/0 conn(0x7fb9bc1530d0 0x7fb9bc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:23.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9bb7fe700 1 -- 172.21.15.7:0/467165438 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901423751 0 0) 0x7fb9bc0c2f30 con 0x7fb9bc14f9d0 2023-03-03T16:06:23.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9bb7fe700 1 -- 172.21.15.7:0/467165438 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ac0035b0 con 0x7fb9bc14f9d0 2023-03-03T16:06:23.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9bb7fe700 1 -- 172.21.15.7:0/467165438 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7189653 0 0) 0x7fb9bc0c31d0 con 0x7fb9bc14fda0 2023-03-03T16:06:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9bb7fe700 1 -- 172.21.15.7:0/467165438 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9bc0c2f30 con 0x7fb9bc14fda0 2023-03-03T16:06:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9c1004700 1 -- 172.21.15.7:0/467165438 >> v1:172.21.15.143:6789/0 conn(0x7fb9bc14fda0 legacy=0x7fb9bc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9c1004700 1 -- 172.21.15.7:0/467165438 >> v1:172.21.15.7:6789/0 conn(0x7fb9bc14f9d0 legacy=0x7fb9bc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9c1004700 1 -- 172.21.15.7:0/467165438 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9bc0c34b0 con 0x7fb9bc1530d0 2023-03-03T16:06:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.812+0000 7fb9c1004700 1 --2- 172.21.15.7:0/467165438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc1530d0 0x7fb9bc156430 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009d90 tx=0x7fb9b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b32bb76d2d018b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:06:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.813+0000 7fb9bb7fe700 1 -- 172.21.15.7:0/467165438 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b400e070 con 0x7fb9bc1530d0 2023-03-03T16:06:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.813+0000 7fb9bb7fe700 1 -- 172.21.15.7:0/467165438 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b400bb20 con 0x7fb9bc1530d0 2023-03-03T16:06:23.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.813+0000 7fb9bb7fe700 1 -- 172.21.15.7:0/467165438 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4012430 con 0x7fb9bc1530d0 2023-03-03T16:06:23.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.813+0000 7fb9c2a67700 1 -- 172.21.15.7:0/467165438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc1530d0 msgr2=0x7fb9bc156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.813+0000 7fb9c2a67700 1 --2- 172.21.15.7:0/467165438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc1530d0 0x7fb9bc156430 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009d90 tx=0x7fb9b400b330 comp rx=0 tx=0).stop 2023-03-03T16:06:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.813+0000 7fb9c2a67700 1 -- 172.21.15.7:0/467165438 shutdown_connections 2023-03-03T16:06:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.813+0000 7fb9c2a67700 1 --2- 172.21.15.7:0/467165438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc1530d0 0x7fb9bc156430 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.813+0000 7fb9c2a67700 1 -- 172.21.15.7:0/467165438 >> 172.21.15.7:0/467165438 conn(0x7fb9bc0c8900 msgr2=0x7fb9bc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:06:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.813+0000 7fb9c2a67700 1 -- 172.21.15.7:0/467165438 shutdown_connections 2023-03-03T16:06:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.813+0000 7fb9c2a67700 1 -- 172.21.15.7:0/467165438 wait complete. 2023-03-03T16:06:23.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.814+0000 7fb9c2a67700 1 Processor -- start 2023-03-03T16:06:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.815+0000 7fb9c2a67700 1 -- start start 2023-03-03T16:06:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.815+0000 7fb9c2a67700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9bc14f9d0 0x7fb9bc1ea310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.815+0000 7fb9c2a67700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc14fda0 0x7fb9bc0c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.815+0000 7fb9c2a67700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9bc1530d0 0x7fb9bc143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.815+0000 7fb9c2a67700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc0c7e90 con 0x7fb9bc14fda0 2023-03-03T16:06:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.815+0000 7fb9c2a67700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc1ea850 con 0x7fb9bc1530d0 2023-03-03T16:06:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.815+0000 7fb9c2a67700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc06c000 con 0x7fb9bc14f9d0 2023-03-03T16:06:23.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc14fda0 0x7fb9bc0c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:23.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc14fda0 0x7fb9bc0c17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55774/0 (socket says 172.21.15.7:55774) 2023-03-03T16:06:23.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9bbfff700 1 -- 172.21.15.7:0/1642270697 learned_addr learned my addr 172.21.15.7:0/1642270697 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:06:23.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9c0803700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9bc14f9d0 0x7fb9bc1ea310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:23.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9c1004700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9bc1530d0 0x7fb9bc143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:23.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9bbfff700 1 -- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9bc14f9d0 msgr2=0x7fb9bc1ea310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:23.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9bbfff700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9bc14f9d0 0x7fb9bc1ea310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:23.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9bbfff700 1 -- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9bc1530d0 msgr2=0x7fb9bc143b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:23.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9bbfff700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9bc1530d0 0x7fb9bc143b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:23.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9bbfff700 1 -- 172.21.15.7:0/1642270697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b4009a70 con 0x7fb9bc14fda0 2023-03-03T16:06:23.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9bbfff700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc14fda0 0x7fb9bc0c17e0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800c4f0 tx=0x7fb9a800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:06:23.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9b97fa700 1 -- 172.21.15.7:0/1642270697 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a80052d0 con 0x7fb9bc14fda0 2023-03-03T16:06:23.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9b97fa700 1 -- 172.21.15.7:0/1642270697 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a8005430 con 0x7fb9bc14fda0 2023-03-03T16:06:23.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9c2a67700 1 -- 172.21.15.7:0/1642270697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9bc0ac350 con 0x7fb9bc14fda0 2023-03-03T16:06:23.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9b97fa700 1 -- 172.21.15.7:0/1642270697 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8003880 con 0x7fb9bc14fda0 2023-03-03T16:06:23.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.816+0000 7fb9c2a67700 1 -- 172.21.15.7:0/1642270697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9bc0ab630 con 0x7fb9bc14fda0 2023-03-03T16:06:23.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.817+0000 7fb9c2a67700 1 -- 172.21.15.7:0/1642270697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9bc071fa0 con 0x7fb9bc14fda0 2023-03-03T16:06:23.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.818+0000 7fb9b97fa700 1 -- 172.21.15.7:0/1642270697 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb9a8019030 con 0x7fb9bc14fda0 2023-03-03T16:06:23.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.818+0000 7fb9b97fa700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9a00682f0 0x7fb9a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:23.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.818+0000 7fb9b97fa700 1 -- 172.21.15.7:0/1642270697 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb9a80852e0 con 0x7fb9bc14fda0 2023-03-03T16:06:23.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.818+0000 7fb9c0803700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9a00682f0 0x7fb9a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:23.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.819+0000 7fb9c0803700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9a00682f0 0x7fb9a006a7a0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0007fd0 tx=0x7fb9b0007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:06:23.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:23.823+0000 7fb9b97fa700 1 -- 172.21.15.7:0/1642270697 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb9a804b5e0 con 0x7fb9bc14fda0 2023-03-03T16:06:24.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.059+0000 7fb9c2a67700 1 -- 172.21.15.7:0/1642270697 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9bc074bb0 con 0x7fb9bc14fda0 2023-03-03T16:06:24.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.060+0000 7fb9b97fa700 1 -- 172.21.15.7:0/1642270697 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb9a8051730 con 0x7fb9bc14fda0 2023-03-03T16:06:24.061 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:06:24.061 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:06:24.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 -- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9a00682f0 msgr2=0x7fb9a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:24.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9a00682f0 0x7fb9a006a7a0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0007fd0 tx=0x7fb9b0007dc0 comp rx=0 tx=0).stop 2023-03-03T16:06:24.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 -- 172.21.15.7:0/1642270697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc14fda0 msgr2=0x7fb9bc0c17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:24.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc14fda0 0x7fb9bc0c17e0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800c4f0 tx=0x7fb9a800c800 comp rx=0 tx=0).stop 2023-03-03T16:06:24.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 -- 172.21.15.7:0/1642270697 shutdown_connections 2023-03-03T16:06:24.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9bc1530d0 0x7fb9bc143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:24.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9bc14f9d0 0x7fb9bc1ea310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:24.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9a00682f0 0x7fb9a006a7a0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:24.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 --2- 172.21.15.7:0/1642270697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9bc14fda0 0x7fb9bc0c17e0 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:24.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 -- 172.21.15.7:0/1642270697 >> 172.21.15.7:0/1642270697 conn(0x7fb9bc0c8900 msgr2=0x7fb9bc0c8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:06:24.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 -- 172.21.15.7:0/1642270697 shutdown_connections 2023-03-03T16:06:24.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:24.062+0000 7fb9c2a67700 1 -- 172.21.15.7:0/1642270697 wait complete. 2023-03-03T16:06:24.064 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:06:24.074 INFO:tasks.mds_thrash.fs.[a]:waiting for 92 secs before thrashing 2023-03-03T16:06:42.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:42.415 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:06:46.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:06:46.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.267+0000 7fe875cec700 1 Processor -- start 2023-03-03T16:06:46.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.267+0000 7fe875cec700 1 -- start start 2023-03-03T16:06:46.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe875cec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe870151cd0 0x7fe870154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:46.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe875cec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe87006c6c0 0x7fe8700736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:46.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe875cec700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe870157ff0 con 0x7fe8700c2e30 2023-03-03T16:06:46.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe875cec700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe87006d640 con 0x7fe870151cd0 2023-03-03T16:06:46.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe875cec700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe87006cda0 con 0x7fe87006c6c0 2023-03-03T16:06:46.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe86f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe870151cd0 0x7fe870154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:46.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe86effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe8700c2e30 0x7fe870158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49784/0 (socket says 172.21.15.7:49784) 2023-03-03T16:06:46.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe86effd700 1 -- 172.21.15.7:0/417976366 learned_addr learned my addr 172.21.15.7:0/417976366 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:06:46.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe86ffff700 1 --2- 172.21.15.7:0/417976366 >> v2:172.21.15.143:3300/0 conn(0x7fe87006c6c0 0x7fe8700736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:46.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe86f7fe700 1 -- 172.21.15.7:0/417976366 >> v2:172.21.15.143:3300/0 conn(0x7fe87006c6c0 msgr2=0x7fe8700736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:06:46.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe86f7fe700 1 --2- 172.21.15.7:0/417976366 >> v2:172.21.15.143:3300/0 conn(0x7fe87006c6c0 0x7fe8700736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:46.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe86f7fe700 1 -- 172.21.15.7:0/417976366 >> v1:172.21.15.7:6789/0 conn(0x7fe8700c2e30 legacy=0x7fe870158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:46.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe86e7fc700 1 -- 172.21.15.7:0/417976366 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839578145 0 0) 0x7fe870157ff0 con 0x7fe8700c2e30 2023-03-03T16:06:46.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe86f7fe700 1 -- 172.21.15.7:0/417976366 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87006d080 con 0x7fe870151cd0 2023-03-03T16:06:46.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.268+0000 7fe86f7fe700 1 --2- 172.21.15.7:0/417976366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe870151cd0 0x7fe870154510 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7fe8700c38a0 tx=0x7fe860009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5f554ae637ce4d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:06:46.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe86e7fc700 1 -- 172.21.15.7:0/417976366 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe860011830 con 0x7fe870151cd0 2023-03-03T16:06:46.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe86e7fc700 1 -- 172.21.15.7:0/417976366 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe860011990 con 0x7fe870151cd0 2023-03-03T16:06:46.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe86e7fc700 1 -- 172.21.15.7:0/417976366 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe860011d20 con 0x7fe870151cd0 2023-03-03T16:06:46.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe875cec700 1 -- 172.21.15.7:0/417976366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe870151cd0 msgr2=0x7fe870154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:46.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe875cec700 1 --2- 172.21.15.7:0/417976366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe870151cd0 0x7fe870154510 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7fe8700c38a0 tx=0x7fe860009b20 comp rx=0 tx=0).stop 2023-03-03T16:06:46.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe875cec700 1 -- 172.21.15.7:0/417976366 shutdown_connections 2023-03-03T16:06:46.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe875cec700 1 --2- 172.21.15.7:0/417976366 >> v2:172.21.15.143:3300/0 conn(0x7fe87006c6c0 0x7fe8700736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:46.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe875cec700 1 --2- 172.21.15.7:0/417976366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe870151cd0 0x7fe870154510 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:46.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe875cec700 1 -- 172.21.15.7:0/417976366 >> 172.21.15.7:0/417976366 conn(0x7fe8700a94d0 msgr2=0x7fe8700a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:06:46.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe875cec700 1 -- 172.21.15.7:0/417976366 shutdown_connections 2023-03-03T16:06:46.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.269+0000 7fe875cec700 1 -- 172.21.15.7:0/417976366 wait complete. 2023-03-03T16:06:46.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.270+0000 7fe875cec700 1 Processor -- start 2023-03-03T16:06:46.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.271+0000 7fe875cec700 1 -- start start 2023-03-03T16:06:46.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.271+0000 7fe875cec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe87006c6c0 0x7fe87014f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:46.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.271+0000 7fe875cec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8700c2e30 0x7fe8701497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:46.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.271+0000 7fe875cec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe870151cd0 0x7fe870149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:46.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.271+0000 7fe875cec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8700c5d10 con 0x7fe87006c6c0 2023-03-03T16:06:46.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.271+0000 7fe875cec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87014d040 con 0x7fe870151cd0 2023-03-03T16:06:46.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.271+0000 7fe875cec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87014a260 con 0x7fe8700c2e30 2023-03-03T16:06:46.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.271+0000 7fe86f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe87006c6c0 0x7fe87014f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:46.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.271+0000 7fe86f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe87006c6c0 0x7fe87014f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55810/0 (socket says 172.21.15.7:55810) 2023-03-03T16:06:46.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.271+0000 7fe86f7fe700 1 -- 172.21.15.7:0/933131405 learned_addr learned my addr 172.21.15.7:0/933131405 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:06:46.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe86effd700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8700c2e30 0x7fe8701497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:46.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe86ffff700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe870151cd0 0x7fe870149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:46.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe86f7fe700 1 -- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8700c2e30 msgr2=0x7fe8701497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:46.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe86f7fe700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8700c2e30 0x7fe8701497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:46.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe86f7fe700 1 -- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe870151cd0 msgr2=0x7fe870149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:46.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe86f7fe700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe870151cd0 0x7fe870149d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:46.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe86f7fe700 1 -- 172.21.15.7:0/933131405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8600094a0 con 0x7fe87006c6c0 2023-03-03T16:06:46.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe86ffff700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe870151cd0 0x7fe870149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:06:46.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe86f7fe700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe87006c6c0 0x7fe87014f3d0 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7fe8600090b0 tx=0x7fe860004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:06:46.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe867fff700 1 -- 172.21.15.7:0/933131405 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8600038a0 con 0x7fe87006c6c0 2023-03-03T16:06:46.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe867fff700 1 -- 172.21.15.7:0/933131405 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86001e070 con 0x7fe87006c6c0 2023-03-03T16:06:46.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe875cec700 1 -- 172.21.15.7:0/933131405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe870157c80 con 0x7fe87006c6c0 2023-03-03T16:06:46.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe867fff700 1 -- 172.21.15.7:0/933131405 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe860003c20 con 0x7fe87006c6c0 2023-03-03T16:06:46.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.272+0000 7fe875cec700 1 -- 172.21.15.7:0/933131405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87006d360 con 0x7fe87006c6c0 2023-03-03T16:06:46.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.273+0000 7fe875cec700 1 -- 172.21.15.7:0/933131405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe870154a50 con 0x7fe87006c6c0 2023-03-03T16:06:46.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.274+0000 7fe867fff700 1 -- 172.21.15.7:0/933131405 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe86000f050 con 0x7fe87006c6c0 2023-03-03T16:06:46.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.274+0000 7fe867fff700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8500682f0 0x7fe85006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:06:46.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.274+0000 7fe867fff700 1 -- 172.21.15.7:0/933131405 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe860020070 con 0x7fe87006c6c0 2023-03-03T16:06:46.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.274+0000 7fe86effd700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8500682f0 0x7fe85006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:06:46.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.275+0000 7fe86effd700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8500682f0 0x7fe85006a7a0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe870157c50 tx=0x7fe858007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:06:46.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.279+0000 7fe867fff700 1 -- 172.21.15.7:0/933131405 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe860053960 con 0x7fe87006c6c0 2023-03-03T16:06:46.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.516+0000 7fe875cec700 1 -- 172.21.15.7:0/933131405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe870074bb0 con 0x7fe87006c6c0 2023-03-03T16:06:46.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.517+0000 7fe867fff700 1 -- 172.21.15.7:0/933131405 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe860056f80 con 0x7fe87006c6c0 2023-03-03T16:06:46.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:06:46.518 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:06:46.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.518+0000 7fe875cec700 1 -- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8500682f0 msgr2=0x7fe85006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:46.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.518+0000 7fe875cec700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8500682f0 0x7fe85006a7a0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe870157c50 tx=0x7fe858007c90 comp rx=0 tx=0).stop 2023-03-03T16:06:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.518+0000 7fe875cec700 1 -- 172.21.15.7:0/933131405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe87006c6c0 msgr2=0x7fe87014f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:06:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.518+0000 7fe875cec700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe87006c6c0 0x7fe87014f3d0 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7fe8600090b0 tx=0x7fe860004750 comp rx=0 tx=0).stop 2023-03-03T16:06:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.518+0000 7fe875cec700 1 -- 172.21.15.7:0/933131405 shutdown_connections 2023-03-03T16:06:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.518+0000 7fe875cec700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe870151cd0 0x7fe870149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:46.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.518+0000 7fe875cec700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8500682f0 0x7fe85006a7a0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:46.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.518+0000 7fe875cec700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe87006c6c0 0x7fe87014f3d0 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:46.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.518+0000 7fe875cec700 1 --2- 172.21.15.7:0/933131405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8700c2e30 0x7fe8701497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:06:46.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.519+0000 7fe875cec700 1 -- 172.21.15.7:0/933131405 >> 172.21.15.7:0/933131405 conn(0x7fe8700a94d0 msgr2=0x7fe870158480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:06:46.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.519+0000 7fe875cec700 1 -- 172.21.15.7:0/933131405 shutdown_connections 2023-03-03T16:06:46.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:06:46.519+0000 7fe875cec700 1 -- 172.21.15.7:0/933131405 wait complete. 2023-03-03T16:06:46.521 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:06:46.531 INFO:tasks.mds_thrash.fs.[b]:waiting for 112 secs before thrashing 2023-03-03T16:07:12.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:12.477 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:42.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:42.550 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:07:56.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:07:56.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.214+0000 7f707d2f5700 1 Processor -- start 2023-03-03T16:07:56.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.214+0000 7f707d2f5700 1 -- start start 2023-03-03T16:07:56.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.215+0000 7f707d2f5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f70780caa30 0x7f70780c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:56.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.215+0000 7f707d2f5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078154ff0 0x7f70780cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:56.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.215+0000 7f707d2f5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70780c7310 con 0x7f70780c00e0 2023-03-03T16:07:56.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.215+0000 7f707d2f5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70780bfd10 con 0x7f70780caa30 2023-03-03T16:07:56.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.215+0000 7f707d2f5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70780bfa30 con 0x7f7078154ff0 2023-03-03T16:07:56.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.215+0000 7f70767fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f70780c00e0 0x7f70780c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49838/0 (socket says 172.21.15.7:49838) 2023-03-03T16:07:56.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.215+0000 7f70767fc700 1 -- 172.21.15.7:0/2615302608 learned_addr learned my addr 172.21.15.7:0/2615302608 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:07:56.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7076ffd700 1 --2- 172.21.15.7:0/2615302608 >> v2:172.21.15.143:3300/0 conn(0x7f70780caa30 0x7f70780c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:56.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f70777fe700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078154ff0 0x7f70780cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:56.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489985161 0 0) 0x7f70780c7310 con 0x7f70780c00e0 2023-03-03T16:07:56.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70640035b0 con 0x7f70780c00e0 2023-03-03T16:07:56.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637254318 0 0) 0x7f70640035b0 con 0x7f70780c00e0 2023-03-03T16:07:56.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078154ff0 msgr2=0x7f70780cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:07:56.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078154ff0 0x7f70780cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 >> v2:172.21.15.143:3300/0 conn(0x7f70780caa30 msgr2=0x7f70780c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:07:56.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 --2- 172.21.15.7:0/2615302608 >> v2:172.21.15.143:3300/0 conn(0x7f70780caa30 0x7f70780c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078154510 con 0x7f70780c00e0 2023-03-03T16:07:56.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f70680017f0 con 0x7f70780c00e0 2023-03-03T16:07:56.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7076ffd700 1 --2- 172.21.15.7:0/2615302608 >> v2:172.21.15.143:3300/0 conn(0x7f70780caa30 0x7f70780c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:07:56.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f70777fe700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078154ff0 0x7f70780cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:07:56.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 >> v1:172.21.15.7:6789/0 conn(0x7f70780c00e0 legacy=0x7f70780c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:56.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7064003a70 0x7f7064003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:56.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078154ff0 0x7f7064004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:56.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 0x7f7064005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:56.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780bfd10 con 0x7f7064004d90 2023-03-03T16:07:56.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780bfa30 con 0x7f7064003a70 2023-03-03T16:07:56.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706400ba10 con 0x7f7078154ff0 2023-03-03T16:07:56.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f70777fe700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 0x7f7064005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:56.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f7076ffd700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7064003a70 0x7f7064003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:56.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.216+0000 7f70767fc700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078154ff0 0x7f7064004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:56.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f70777fe700 1 -- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078154ff0 msgr2=0x7f7064004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:07:56.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f70777fe700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078154ff0 0x7f7064004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f70777fe700 1 -- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7064003a70 msgr2=0x7f7064003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:07:56.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f70777fe700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7064003a70 0x7f7064003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f70777fe700 1 -- 172.21.15.7:0/2615302608 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7070009a70 con 0x7f7064004d90 2023-03-03T16:07:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f70777fe700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 0x7f7064005200 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f707000e350 tx=0x7f7064005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ebe74ecd93e3d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:07:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707000edf0 con 0x7f7064004d90 2023-03-03T16:07:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707000f040 con 0x7f7064004d90 2023-03-03T16:07:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f7075ffb700 1 -- 172.21.15.7:0/2615302608 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70700146a0 con 0x7f7064004d90 2023-03-03T16:07:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f707d2f5700 1 -- 172.21.15.7:0/2615302608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 msgr2=0x7f7064005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f707d2f5700 1 --2- 172.21.15.7:0/2615302608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 0x7f7064005200 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f707000e350 tx=0x7f7064005880 comp rx=0 tx=0).stop 2023-03-03T16:07:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f70777fe700 1 -- 172.21.15.7:0/2615302608 reap_dead start 2023-03-03T16:07:56.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f707d2f5700 1 -- 172.21.15.7:0/2615302608 shutdown_connections 2023-03-03T16:07:56.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.217+0000 7f707d2f5700 1 -- 172.21.15.7:0/2615302608 >> 172.21.15.7:0/2615302608 conn(0x7f7078066e40 msgr2=0x7f7078071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:07:56.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.218+0000 7f707d2f5700 1 -- 172.21.15.7:0/2615302608 shutdown_connections 2023-03-03T16:07:56.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.218+0000 7f707d2f5700 1 -- 172.21.15.7:0/2615302608 wait complete. 2023-03-03T16:07:56.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.219+0000 7f707d2f5700 1 Processor -- start 2023-03-03T16:07:56.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.219+0000 7f707d2f5700 1 -- start start 2023-03-03T16:07:56.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f707d2f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 0x7f7078149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:56.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f707d2f5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078158190 0x7f70780c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:56.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f707d2f5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707814ad00 0x7f707814b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:56.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f707d2f5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707814b6c0 con 0x7f7064004d90 2023-03-03T16:07:56.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f707d2f5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7078155e60 con 0x7f707814ad00 2023-03-03T16:07:56.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f707d2f5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707814a460 con 0x7f7078158190 2023-03-03T16:07:56.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f7076ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 0x7f7078149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:56.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f7076ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 0x7f7078149f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55870/0 (socket says 172.21.15.7:55870) 2023-03-03T16:07:56.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f7076ffd700 1 -- 172.21.15.7:0/3196603970 learned_addr learned my addr 172.21.15.7:0/3196603970 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:07:56.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f70777fe700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707814ad00 0x7f707814b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:56.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f70767fc700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078158190 0x7f70780c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:56.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f7076ffd700 1 -- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078158190 msgr2=0x7f70780c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:56.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f7076ffd700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078158190 0x7f70780c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f7076ffd700 1 -- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707814ad00 msgr2=0x7f707814b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:56.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f7076ffd700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707814ad00 0x7f707814b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f7076ffd700 1 -- 172.21.15.7:0/3196603970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078154ad0 con 0x7f7064004d90 2023-03-03T16:07:56.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f7076ffd700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 0x7f7078149f20 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f7060011cd0 tx=0x7f7060009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:07:56.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f706f7fe700 1 -- 172.21.15.7:0/3196603970 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706000fa00 con 0x7f7064004d90 2023-03-03T16:07:56.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f706f7fe700 1 -- 172.21.15.7:0/3196603970 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706000fb60 con 0x7f7064004d90 2023-03-03T16:07:56.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f707d2f5700 1 -- 172.21.15.7:0/3196603970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70781547f0 con 0x7f7064004d90 2023-03-03T16:07:56.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.220+0000 7f706f7fe700 1 -- 172.21.15.7:0/3196603970 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7060004030 con 0x7f7064004d90 2023-03-03T16:07:56.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.221+0000 7f707d2f5700 1 -- 172.21.15.7:0/3196603970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70780bf650 con 0x7f7064004d90 2023-03-03T16:07:56.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.221+0000 7f707d2f5700 1 -- 172.21.15.7:0/3196603970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7078156e10 con 0x7f7064004d90 2023-03-03T16:07:56.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.222+0000 7f706f7fe700 1 -- 172.21.15.7:0/3196603970 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7060024070 con 0x7f7064004d90 2023-03-03T16:07:56.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.222+0000 7f706f7fe700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70580682f0 0x7f705806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:07:56.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.222+0000 7f706f7fe700 1 -- 172.21.15.7:0/3196603970 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7060086520 con 0x7f7064004d90 2023-03-03T16:07:56.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.222+0000 7f70767fc700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70580682f0 0x7f705806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:07:56.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.223+0000 7f70767fc700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70580682f0 0x7f705806a7a0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f70780c48b0 tx=0x7f7068009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:07:56.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.227+0000 7f706f7fe700 1 -- 172.21.15.7:0/3196603970 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70600528c0 con 0x7f7064004d90 2023-03-03T16:07:56.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.471+0000 7f707d2f5700 1 -- 172.21.15.7:0/3196603970 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7078074bb0 con 0x7f7064004d90 2023-03-03T16:07:56.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.471+0000 7f706f7fe700 1 -- 172.21.15.7:0/3196603970 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7060055ee0 con 0x7f7064004d90 2023-03-03T16:07:56.472 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:07:56.472 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:07:56.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 -- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70580682f0 msgr2=0x7f705806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:56.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70580682f0 0x7f705806a7a0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f70780c48b0 tx=0x7f7068009040 comp rx=0 tx=0).stop 2023-03-03T16:07:56.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 -- 172.21.15.7:0/3196603970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 msgr2=0x7f7078149f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:07:56.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 0x7f7078149f20 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f7060011cd0 tx=0x7f7060009ed0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 -- 172.21.15.7:0/3196603970 shutdown_connections 2023-03-03T16:07:56.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707814ad00 0x7f707814b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70580682f0 0x7f705806a7a0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7064004d90 0x7f7078149f20 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 --2- 172.21.15.7:0/3196603970 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7078158190 0x7f70780c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:07:56.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 -- 172.21.15.7:0/3196603970 >> 172.21.15.7:0/3196603970 conn(0x7f7078066e40 msgr2=0x7f70780c7860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:07:56.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 -- 172.21.15.7:0/3196603970 shutdown_connections 2023-03-03T16:07:56.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:07:56.473+0000 7f707d2f5700 1 -- 172.21.15.7:0/3196603970 wait complete. 2023-03-03T16:07:56.476 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:07:56.486 INFO:tasks.mds_thrash.fs.[a]:waiting for 25 secs before thrashing 2023-03-03T16:08:12.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:12.612 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:21.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:08:21.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.627+0000 7f99ebf66700 1 Processor -- start 2023-03-03T16:08:21.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.627+0000 7f99ebf66700 1 -- start start 2023-03-03T16:08:21.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ebf66700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f99e406e020 0x7f99e4071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:21.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ebf66700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f99e4072520 0x7f99e4151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:21.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ebf66700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e4154910 con 0x7f99e4155390 2023-03-03T16:08:21.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ebf66700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e4070b90 con 0x7f99e4072520 2023-03-03T16:08:21.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ebf66700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e4151320 con 0x7f99e406e020 2023-03-03T16:08:21.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ea503700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f99e4072520 0x7f99e4151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99e9d02700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f99e4155390 0x7f99e40c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49868/0 (socket says 172.21.15.7:49868) 2023-03-03T16:08:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99e9d02700 1 -- 172.21.15.7:0/1091611867 learned_addr learned my addr 172.21.15.7:0/1091611867 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:08:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99e9501700 1 --2- 172.21.15.7:0/1091611867 >> v2:172.21.15.143:3300/0 conn(0x7f99e406e020 0x7f99e4071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ea503700 1 -- 172.21.15.7:0/1091611867 >> v2:172.21.15.143:3300/0 conn(0x7f99e406e020 msgr2=0x7f99e4071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:08:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ea503700 1 --2- 172.21.15.7:0/1091611867 >> v2:172.21.15.143:3300/0 conn(0x7f99e406e020 0x7f99e4071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ea503700 1 -- 172.21.15.7:0/1091611867 >> v1:172.21.15.7:6789/0 conn(0x7f99e4155390 legacy=0x7f99e40c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ea503700 1 -- 172.21.15.7:0/1091611867 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e40c4540 con 0x7f99e4072520 2023-03-03T16:08:21.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99e8d00700 1 -- 172.21.15.7:0/1091611867 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289735987 0 0) 0x7f99e4154910 con 0x7f99e4155390 2023-03-03T16:08:21.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.628+0000 7f99ea503700 1 --2- 172.21.15.7:0/1091611867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 0x7f99e4151770 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f99e40ac9e0 tx=0x7f99dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eac3c054080edad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:21.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99e8d00700 1 -- 172.21.15.7:0/1091611867 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc00e070 con 0x7f99e4072520 2023-03-03T16:08:21.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99e8d00700 1 -- 172.21.15.7:0/1091611867 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99dc00bbd0 con 0x7f99e4072520 2023-03-03T16:08:21.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99e8d00700 1 -- 172.21.15.7:0/1091611867 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc012430 con 0x7f99e4072520 2023-03-03T16:08:21.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99ebf66700 1 -- 172.21.15.7:0/1091611867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 msgr2=0x7f99e4151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:21.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99ebf66700 1 --2- 172.21.15.7:0/1091611867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 0x7f99e4151770 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f99e40ac9e0 tx=0x7f99dc00b330 comp rx=0 tx=0).stop 2023-03-03T16:08:21.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99ebf66700 1 -- 172.21.15.7:0/1091611867 shutdown_connections 2023-03-03T16:08:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99ebf66700 1 --2- 172.21.15.7:0/1091611867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 0x7f99e4151770 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99ebf66700 1 --2- 172.21.15.7:0/1091611867 >> v2:172.21.15.143:3300/0 conn(0x7f99e406e020 0x7f99e4071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99ebf66700 1 -- 172.21.15.7:0/1091611867 >> 172.21.15.7:0/1091611867 conn(0x7f99e406c110 msgr2=0x7f99e406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:08:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99ebf66700 1 -- 172.21.15.7:0/1091611867 shutdown_connections 2023-03-03T16:08:21.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.629+0000 7f99ebf66700 1 -- 172.21.15.7:0/1091611867 wait complete. 2023-03-03T16:08:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.630+0000 7f99ebf66700 1 Processor -- start 2023-03-03T16:08:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.631+0000 7f99ebf66700 1 -- start start 2023-03-03T16:08:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.631+0000 7f99ebf66700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e406e020 0x7f99e40aa970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:21.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.631+0000 7f99ebf66700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 0x7f99e40c1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:21.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.631+0000 7f99ebf66700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99e4155390 0x7f99e414dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:21.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.631+0000 7f99ebf66700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e40ab050 con 0x7f99e4072520 2023-03-03T16:08:21.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.631+0000 7f99ebf66700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e40aaeb0 con 0x7f99e406e020 2023-03-03T16:08:21.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.631+0000 7f99ebf66700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e40aa4e0 con 0x7f99e4155390 2023-03-03T16:08:21.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.631+0000 7f99e9501700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 0x7f99e40c1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:21.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.631+0000 7f99e9501700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 0x7f99e40c1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55896/0 (socket says 172.21.15.7:55896) 2023-03-03T16:08:21.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.631+0000 7f99e9501700 1 -- 172.21.15.7:0/3599201628 learned_addr learned my addr 172.21.15.7:0/3599201628 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:08:21.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99ea503700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99e4155390 0x7f99e414dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:21.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99e9d02700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e406e020 0x7f99e40aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:21.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99e9501700 1 -- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99e4155390 msgr2=0x7f99e414dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:21.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99e9501700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99e4155390 0x7f99e414dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:21.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99e9501700 1 -- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e406e020 msgr2=0x7f99e40aa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:21.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99e9501700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e406e020 0x7f99e40aa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:21.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99e9501700 1 -- 172.21.15.7:0/3599201628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc009a70 con 0x7f99e4072520 2023-03-03T16:08:21.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99ea503700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99e4155390 0x7f99e414dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:08:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99e9d02700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e406e020 0x7f99e40aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:08:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99e9501700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 0x7f99e40c1050 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f99d400e840 tx=0x7f99d400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99da7fc700 1 -- 172.21.15.7:0/3599201628 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4013610 con 0x7f99e4072520 2023-03-03T16:08:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99da7fc700 1 -- 172.21.15.7:0/3599201628 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d4013770 con 0x7f99e4072520 2023-03-03T16:08:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99ebf66700 1 -- 172.21.15.7:0/3599201628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e4070540 con 0x7f99e4072520 2023-03-03T16:08:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99da7fc700 1 -- 172.21.15.7:0/3599201628 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4013b20 con 0x7f99e4072520 2023-03-03T16:08:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.632+0000 7f99ebf66700 1 -- 172.21.15.7:0/3599201628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e40702a0 con 0x7f99e4072520 2023-03-03T16:08:21.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.633+0000 7f99ebf66700 1 -- 172.21.15.7:0/3599201628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e4070820 con 0x7f99e4072520 2023-03-03T16:08:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.634+0000 7f99da7fc700 1 -- 172.21.15.7:0/3599201628 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f99d4023070 con 0x7f99e4072520 2023-03-03T16:08:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.634+0000 7f99da7fc700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99c80682f0 0x7f99c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.634+0000 7f99da7fc700 1 -- 172.21.15.7:0/3599201628 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f99d4085e40 con 0x7f99e4072520 2023-03-03T16:08:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.634+0000 7f99e9d02700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99c80682f0 0x7f99c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.635+0000 7f99e9d02700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99c80682f0 0x7f99c806a7a0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f99e0000f60 tx=0x7f99e0007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:21.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.638+0000 7f99da7fc700 1 -- 172.21.15.7:0/3599201628 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f99d40521e0 con 0x7f99e4072520 2023-03-03T16:08:21.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.879+0000 7f99ebf66700 1 -- 172.21.15.7:0/3599201628 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99e40ad160 con 0x7f99e4072520 2023-03-03T16:08:21.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.880+0000 7f99da7fc700 1 -- 172.21.15.7:0/3599201628 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f99d4085020 con 0x7f99e4072520 2023-03-03T16:08:21.880 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:08:21.881 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:08:21.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.881+0000 7f99ebf66700 1 -- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99c80682f0 msgr2=0x7f99c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:21.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.881+0000 7f99ebf66700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99c80682f0 0x7f99c806a7a0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f99e0000f60 tx=0x7f99e0007d30 comp rx=0 tx=0).stop 2023-03-03T16:08:21.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.882+0000 7f99ebf66700 1 -- 172.21.15.7:0/3599201628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 msgr2=0x7f99e40c1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:21.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.882+0000 7f99ebf66700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 0x7f99e40c1050 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f99d400e840 tx=0x7f99d400ec00 comp rx=0 tx=0).stop 2023-03-03T16:08:21.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.882+0000 7f99ebf66700 1 -- 172.21.15.7:0/3599201628 shutdown_connections 2023-03-03T16:08:21.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.882+0000 7f99ebf66700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99e4155390 0x7f99e414dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:21.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.882+0000 7f99ebf66700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e406e020 0x7f99e40aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:21.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.882+0000 7f99ebf66700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99c80682f0 0x7f99c806a7a0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:21.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.882+0000 7f99ebf66700 1 --2- 172.21.15.7:0/3599201628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99e4072520 0x7f99e40c1050 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:21.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.882+0000 7f99ebf66700 1 -- 172.21.15.7:0/3599201628 >> 172.21.15.7:0/3599201628 conn(0x7f99e406c110 msgr2=0x7f99e414fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:08:21.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.882+0000 7f99ebf66700 1 -- 172.21.15.7:0/3599201628 shutdown_connections 2023-03-03T16:08:21.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:21.882+0000 7f99ebf66700 1 -- 172.21.15.7:0/3599201628 wait complete. 2023-03-03T16:08:21.884 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:08:21.894 INFO:tasks.mds_thrash.fs.[a]:waiting for 63 secs before thrashing 2023-03-03T16:08:38.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:08:38.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.671+0000 7fda415c5700 1 Processor -- start 2023-03-03T16:08:38.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.671+0000 7fda415c5700 1 -- start start 2023-03-03T16:08:38.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.672+0000 7fda415c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fda3c0c00e0 0x7fda3c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:38.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.672+0000 7fda415c5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c0c7310 con 0x7fda3c0caa30 2023-03-03T16:08:38.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.672+0000 7fda415c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c0bfd10 con 0x7fda3c154ff0 2023-03-03T16:08:38.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.672+0000 7fda415c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c155430 con 0x7fda3c0c00e0 2023-03-03T16:08:38.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.672+0000 7fda3affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fda3c0caa30 0x7fda3c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49894/0 (socket says 172.21.15.7:49894) 2023-03-03T16:08:38.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.672+0000 7fda3affd700 1 -- 172.21.15.7:0/3704751254 learned_addr learned my addr 172.21.15.7:0/3704751254 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:08:38.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.672+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673797765 0 0) 0x7fda3c0c7310 con 0x7fda3c0caa30 2023-03-03T16:08:38.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.672+0000 7fda3a7fc700 1 --2- 172.21.15.7:0/3704751254 >> v2:172.21.15.143:3300/0 conn(0x7fda3c0c00e0 0x7fda3c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:38.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.672+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda240035b0 con 0x7fda3c0caa30 2023-03-03T16:08:38.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2015583741 0 0) 0x7fda240035b0 con 0x7fda3c0caa30 2023-03-03T16:08:38.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 >> v2:172.21.15.143:3300/0 conn(0x7fda3c0c00e0 msgr2=0x7fda3c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:08:38.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 --2- 172.21.15.7:0/3704751254 >> v2:172.21.15.143:3300/0 conn(0x7fda3c0c00e0 0x7fda3c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:38.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 >> v1:172.21.15.143:6789/0 conn(0x7fda3c154ff0 legacy=0x7fda3c15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:38.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c0bfa30 con 0x7fda3c0caa30 2023-03-03T16:08:38.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fda30001670 con 0x7fda3c0caa30 2023-03-03T16:08:38.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 >> v1:172.21.15.7:6789/0 conn(0x7fda3c0caa30 legacy=0x7fda3c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:38.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda24003a70 0x7fda24003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:38.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda24004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:38.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda24005250 0x7fda2400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:38.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c155430 con 0x7fda24004850 2023-03-03T16:08:38.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2400c190 con 0x7fda24005250 2023-03-03T16:08:38.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2400c350 con 0x7fda24003a70 2023-03-03T16:08:38.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3a7fc700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda24004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:38.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3b7fe700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda24005250 0x7fda2400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:38.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3affd700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda24003a70 0x7fda24003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:38.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3a7fc700 1 -- 172.21.15.7:0/3704751254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda24003a70 msgr2=0x7fda24003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:08:38.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3a7fc700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda24003a70 0x7fda24003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:38.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3a7fc700 1 -- 172.21.15.7:0/3704751254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda24005250 msgr2=0x7fda2400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:08:38.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3a7fc700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda24005250 0x7fda2400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:38.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3a7fc700 1 -- 172.21.15.7:0/3704751254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda240035b0 con 0x7fda24004850 2023-03-03T16:08:38.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3b7fe700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda24005250 0x7fda2400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:08:38.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3b7fe700 1 -- 172.21.15.7:0/3704751254 reap_dead start 2023-03-03T16:08:38.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda3a7fc700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda24004cc0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009b10 tx=0x7fda24005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883357e9a4d7c73 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:38.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c005a20 con 0x7fda24004850 2023-03-03T16:08:38.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c005b80 con 0x7fda24004850 2023-03-03T16:08:38.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.673+0000 7fda39ffb700 1 -- 172.21.15.7:0/3704751254 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c00e6c0 con 0x7fda24004850 2023-03-03T16:08:38.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.674+0000 7fda415c5700 1 -- 172.21.15.7:0/3704751254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 msgr2=0x7fda24004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:38.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.674+0000 7fda415c5700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda24004cc0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fda2c009b10 tx=0x7fda24005880 comp rx=0 tx=0).stop 2023-03-03T16:08:38.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.674+0000 7fda415c5700 1 -- 172.21.15.7:0/3704751254 shutdown_connections 2023-03-03T16:08:38.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.674+0000 7fda415c5700 1 --2- 172.21.15.7:0/3704751254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda24004cc0 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:38.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.674+0000 7fda415c5700 1 -- 172.21.15.7:0/3704751254 >> 172.21.15.7:0/3704751254 conn(0x7fda3c066e40 msgr2=0x7fda3c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:08:38.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.674+0000 7fda415c5700 1 -- 172.21.15.7:0/3704751254 shutdown_connections 2023-03-03T16:08:38.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.674+0000 7fda415c5700 1 -- 172.21.15.7:0/3704751254 wait complete. 2023-03-03T16:08:38.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.675+0000 7fda415c5700 1 Processor -- start 2023-03-03T16:08:38.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.675+0000 7fda415c5700 1 -- start start 2023-03-03T16:08:38.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda415c5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda3c14aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:38.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda415c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda3c158090 0x7fda3c152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:38.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda415c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda3c1485e0 0x7fda3c1ed000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:38.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda415c5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c149810 con 0x7fda24004850 2023-03-03T16:08:38.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda415c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c158520 con 0x7fda3c1485e0 2023-03-03T16:08:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda415c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c158800 con 0x7fda3c158090 2023-03-03T16:08:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda3affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda3c14aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda3affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda3c14aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55926/0 (socket says 172.21.15.7:55926) 2023-03-03T16:08:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda3affd700 1 -- 172.21.15.7:0/3429624917 learned_addr learned my addr 172.21.15.7:0/3429624917 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:08:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda3a7fc700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda3c158090 0x7fda3c152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.676+0000 7fda3b7fe700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda3c1485e0 0x7fda3c1ed000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda3affd700 1 -- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda3c158090 msgr2=0x7fda3c152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:38.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda3affd700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda3c158090 0x7fda3c152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda3affd700 1 -- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda3c1485e0 msgr2=0x7fda3c1ed000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda3affd700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda3c1485e0 0x7fda3c1ed000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda3affd700 1 -- 172.21.15.7:0/3429624917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c1547f0 con 0x7fda24004850 2023-03-03T16:08:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda3affd700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda3c14aa40 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fda30001200 tx=0x7fda3000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda237fe700 1 -- 172.21.15.7:0/3429624917 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30012590 con 0x7fda24004850 2023-03-03T16:08:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda237fe700 1 -- 172.21.15.7:0/3429624917 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda30012730 con 0x7fda24004850 2023-03-03T16:08:38.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda237fe700 1 -- 172.21.15.7:0/3429624917 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30012ae0 con 0x7fda24004850 2023-03-03T16:08:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda415c5700 1 -- 172.21.15.7:0/3429624917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c154510 con 0x7fda24004850 2023-03-03T16:08:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.677+0000 7fda415c5700 1 -- 172.21.15.7:0/3429624917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c154ad0 con 0x7fda24004850 2023-03-03T16:08:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.678+0000 7fda415c5700 1 -- 172.21.15.7:0/3429624917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c074bb0 con 0x7fda24004850 2023-03-03T16:08:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.678+0000 7fda237fe700 1 -- 172.21.15.7:0/3429624917 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fda30012c40 con 0x7fda24004850 2023-03-03T16:08:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.679+0000 7fda237fe700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda280682f0 0x7fda2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.679+0000 7fda237fe700 1 -- 172.21.15.7:0/3429624917 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fda30086120 con 0x7fda24004850 2023-03-03T16:08:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.679+0000 7fda3a7fc700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda280682f0 0x7fda2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:38.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.679+0000 7fda3a7fc700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda280682f0 0x7fda2806a7a0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fda2c000c00 tx=0x7fda2c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:38.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.683+0000 7fda237fe700 1 -- 172.21.15.7:0/3429624917 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fda300523d0 con 0x7fda24004850 2023-03-03T16:08:38.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.921+0000 7fda415c5700 1 -- 172.21.15.7:0/3429624917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda3c0c5030 con 0x7fda24004850 2023-03-03T16:08:38.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.922+0000 7fda237fe700 1 -- 172.21.15.7:0/3429624917 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fda300559f0 con 0x7fda24004850 2023-03-03T16:08:38.923 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:08:38.923 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:08:38.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 -- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda280682f0 msgr2=0x7fda2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:38.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda280682f0 0x7fda2806a7a0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fda2c000c00 tx=0x7fda2c008030 comp rx=0 tx=0).stop 2023-03-03T16:08:38.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 -- 172.21.15.7:0/3429624917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 msgr2=0x7fda3c14aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:38.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda3c14aa40 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fda30001200 tx=0x7fda3000eb20 comp rx=0 tx=0).stop 2023-03-03T16:08:38.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 -- 172.21.15.7:0/3429624917 shutdown_connections 2023-03-03T16:08:38.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda3c1485e0 0x7fda3c1ed000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:38.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fda280682f0 0x7fda2806a7a0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:38.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24004850 0x7fda3c14aa40 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:38.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 --2- 172.21.15.7:0/3429624917 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fda3c158090 0x7fda3c152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:38.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 -- 172.21.15.7:0/3429624917 >> 172.21.15.7:0/3429624917 conn(0x7fda3c066e40 msgr2=0x7fda3c066040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:08:38.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 -- 172.21.15.7:0/3429624917 shutdown_connections 2023-03-03T16:08:38.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:38.924+0000 7fda415c5700 1 -- 172.21.15.7:0/3429624917 wait complete. 2023-03-03T16:08:38.926 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:08:38.936 INFO:tasks.mds_thrash.fs.[b]:waiting for 19 secs before thrashing 2023-03-03T16:08:42.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:42.683 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:08:57.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:08:58.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.072+0000 7f1032d3e700 1 Processor -- start 2023-03-03T16:08:58.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.072+0000 7f1032d3e700 1 -- start start 2023-03-03T16:08:58.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.073+0000 7f1032d3e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f102c0c2e30 0x7f102c158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:58.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.073+0000 7f1032d3e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c06c6c0 0x7f102c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:58.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.073+0000 7f1032d3e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c157ff0 con 0x7f102c151cd0 2023-03-03T16:08:58.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.073+0000 7f1032d3e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102c06d640 con 0x7f102c0c2e30 2023-03-03T16:08:58.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.073+0000 7f1032d3e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c06cda0 con 0x7f102c06c6c0 2023-03-03T16:08:58.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.073+0000 7f102bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f102c0c2e30 0x7f102c158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:58.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.073+0000 7f102bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f102c0c2e30 0x7f102c158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55962/0 (socket says 172.21.15.7:55962) 2023-03-03T16:08:58.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.073+0000 7f102bfff700 1 -- 172.21.15.7:0/3030506926 learned_addr learned my addr 172.21.15.7:0/3030506926 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:08:58.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.073+0000 7f10312db700 1 --2- 172.21.15.7:0/3030506926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c06c6c0 0x7f102c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:58.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f102bfff700 1 -- 172.21.15.7:0/3030506926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c06c6c0 msgr2=0x7f102c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:08:58.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f102bfff700 1 --2- 172.21.15.7:0/3030506926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c06c6c0 0x7f102c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f102bfff700 1 -- 172.21.15.7:0/3030506926 >> v1:172.21.15.143:6789/0 conn(0x7f102c151cd0 legacy=0x7f102c154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:08:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f102bfff700 1 -- 172.21.15.7:0/3030506926 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c06d080 con 0x7f102c0c2e30 2023-03-03T16:08:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f10312db700 1 --2- 172.21.15.7:0/3030506926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c06c6c0 0x7f102c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:08:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f102bfff700 1 --2- 172.21.15.7:0/3030506926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c0c2e30 0x7f102c158480 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f10180098f0 tx=0x7f1018009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd680cb67e98e652 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f102b7fe700 1 -- 172.21.15.7:0/3030506926 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1018004230 con 0x7f102c0c2e30 2023-03-03T16:08:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f102b7fe700 1 -- 172.21.15.7:0/3030506926 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1018004390 con 0x7f102c0c2e30 2023-03-03T16:08:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f102b7fe700 1 -- 172.21.15.7:0/3030506926 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10180036a0 con 0x7f102c0c2e30 2023-03-03T16:08:58.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f1032d3e700 1 -- 172.21.15.7:0/3030506926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c0c2e30 msgr2=0x7f102c158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f1032d3e700 1 --2- 172.21.15.7:0/3030506926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c0c2e30 0x7f102c158480 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f10180098f0 tx=0x7f1018009c00 comp rx=0 tx=0).stop 2023-03-03T16:08:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f1032d3e700 1 -- 172.21.15.7:0/3030506926 shutdown_connections 2023-03-03T16:08:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f1032d3e700 1 --2- 172.21.15.7:0/3030506926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c06c6c0 0x7f102c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f1032d3e700 1 --2- 172.21.15.7:0/3030506926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c0c2e30 0x7f102c158480 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f1032d3e700 1 -- 172.21.15.7:0/3030506926 >> 172.21.15.7:0/3030506926 conn(0x7f102c0a94d0 msgr2=0x7f102c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:08:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f1032d3e700 1 -- 172.21.15.7:0/3030506926 shutdown_connections 2023-03-03T16:08:58.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.074+0000 7f1032d3e700 1 -- 172.21.15.7:0/3030506926 wait complete. 2023-03-03T16:08:58.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.076+0000 7f1032d3e700 1 Processor -- start 2023-03-03T16:08:58.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.076+0000 7f1032d3e700 1 -- start start 2023-03-03T16:08:58.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.076+0000 7f1032d3e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f102c06c6c0 0x7f102c152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:58.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.076+0000 7f1032d3e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c0c2e30 0x7f102c0c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:58.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.076+0000 7f1032d3e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c151cd0 0x7f102c155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:58.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.076+0000 7f1032d3e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c0c7240 con 0x7f102c151cd0 2023-03-03T16:08:58.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.076+0000 7f1032d3e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c1543b0 con 0x7f102c06c6c0 2023-03-03T16:08:58.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.076+0000 7f1032d3e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c14f0d0 con 0x7f102c0c2e30 2023-03-03T16:08:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10312db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c151cd0 0x7f102c155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10312db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c151cd0 0x7f102c155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55970/0 (socket says 172.21.15.7:55970) 2023-03-03T16:08:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10312db700 1 -- 172.21.15.7:0/2506641386 learned_addr learned my addr 172.21.15.7:0/2506641386 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:08:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f102bfff700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c0c2e30 0x7f102c0c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f1030ada700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f102c06c6c0 0x7f102c152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10312db700 1 -- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c0c2e30 msgr2=0x7f102c0c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:58.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10312db700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c0c2e30 0x7f102c0c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10312db700 1 -- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f102c06c6c0 msgr2=0x7f102c152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10312db700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f102c06c6c0 0x7f102c152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10312db700 1 -- 172.21.15.7:0/2506641386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1018009550 con 0x7f102c151cd0 2023-03-03T16:08:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10312db700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c151cd0 0x7f102c155900 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f102400cd80 tx=0x7f102400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10297fa700 1 -- 172.21.15.7:0/2506641386 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102400f480 con 0x7f102c151cd0 2023-03-03T16:08:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10297fa700 1 -- 172.21.15.7:0/2506641386 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102400f600 con 0x7f102c151cd0 2023-03-03T16:08:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f10297fa700 1 -- 172.21.15.7:0/2506641386 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102400f9b0 con 0x7f102c151cd0 2023-03-03T16:08:58.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f1032d3e700 1 -- 172.21.15.7:0/2506641386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c157c80 con 0x7f102c151cd0 2023-03-03T16:08:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.077+0000 7f1032d3e700 1 -- 172.21.15.7:0/2506641386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102c06d360 con 0x7f102c151cd0 2023-03-03T16:08:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.078+0000 7f1032d3e700 1 -- 172.21.15.7:0/2506641386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102c1589c0 con 0x7f102c151cd0 2023-03-03T16:08:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.079+0000 7f10297fa700 1 -- 172.21.15.7:0/2506641386 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1024009030 con 0x7f102c151cd0 2023-03-03T16:08:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.079+0000 7f10297fa700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10100682f0 0x7f101006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:08:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.079+0000 7f10297fa700 1 -- 172.21.15.7:0/2506641386 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f10240869e0 con 0x7f102c151cd0 2023-03-03T16:08:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.079+0000 7f1030ada700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10100682f0 0x7f101006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:08:58.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.080+0000 7f1030ada700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10100682f0 0x7f101006a7a0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f102c1542d0 tx=0x7f10200090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:08:58.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.084+0000 7f10297fa700 1 -- 172.21.15.7:0/2506641386 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1024052d20 con 0x7f102c151cd0 2023-03-03T16:08:58.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.320+0000 7f1032d3e700 1 -- 172.21.15.7:0/2506641386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f102c071600 con 0x7f102c151cd0 2023-03-03T16:08:58.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.320+0000 7f10297fa700 1 -- 172.21.15.7:0/2506641386 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1024056340 con 0x7f102c151cd0 2023-03-03T16:08:58.322 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:08:58.322 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:08:58.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.322+0000 7f1032d3e700 1 -- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10100682f0 msgr2=0x7f101006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:58.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.322+0000 7f1032d3e700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10100682f0 0x7f101006a7a0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f102c1542d0 tx=0x7f10200090e0 comp rx=0 tx=0).stop 2023-03-03T16:08:58.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.322+0000 7f1032d3e700 1 -- 172.21.15.7:0/2506641386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c151cd0 msgr2=0x7f102c155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:08:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.322+0000 7f1032d3e700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c151cd0 0x7f102c155900 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f102400cd80 tx=0x7f102400a560 comp rx=0 tx=0).stop 2023-03-03T16:08:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.322+0000 7f1032d3e700 1 -- 172.21.15.7:0/2506641386 shutdown_connections 2023-03-03T16:08:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.322+0000 7f1032d3e700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10100682f0 0x7f101006a7a0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.322+0000 7f1032d3e700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f102c151cd0 0x7f102c155900 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.322+0000 7f1032d3e700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f102c06c6c0 0x7f102c152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.322+0000 7f1032d3e700 1 --2- 172.21.15.7:0/2506641386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f102c0c2e30 0x7f102c0c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:08:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.322+0000 7f1032d3e700 1 -- 172.21.15.7:0/2506641386 >> 172.21.15.7:0/2506641386 conn(0x7f102c0a94d0 msgr2=0x7f102c06b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:08:58.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.323+0000 7f1032d3e700 1 -- 172.21.15.7:0/2506641386 shutdown_connections 2023-03-03T16:08:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:08:58.323+0000 7f1032d3e700 1 -- 172.21.15.7:0/2506641386 wait complete. 2023-03-03T16:08:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:08:58.335 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before thrashing 2023-03-03T16:09:03.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:09:03.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.475+0000 7f19e6d39700 1 Processor -- start 2023-03-03T16:09:03.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.475+0000 7f19e6d39700 1 -- start start 2023-03-03T16:09:03.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.476+0000 7f19e6d39700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aad20 0x7f19e0157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:03.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.476+0000 7f19e6d39700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19e00aa030 0x7f19e01581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:03.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.476+0000 7f19e6d39700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e00728b0 con 0x7f19e014b020 2023-03-03T16:09:03.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.476+0000 7f19e6d39700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e0155070 con 0x7f19e00aa030 2023-03-03T16:09:03.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.476+0000 7f19e6d39700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e006f5b0 con 0x7f19e00aad20 2023-03-03T16:09:03.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.476+0000 7f19e52d6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19e00aa030 0x7f19e01581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:03.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.476+0000 7f19e52d6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19e00aa030 0x7f19e01581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55978/0 (socket says 172.21.15.7:55978) 2023-03-03T16:09:03.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.476+0000 7f19e52d6700 1 -- 172.21.15.7:0/1433550830 learned_addr learned my addr 172.21.15.7:0/1433550830 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:09:03.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.476+0000 7f19dffff700 1 --2- 172.21.15.7:0/1433550830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aad20 0x7f19e0157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:03.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19e52d6700 1 -- 172.21.15.7:0/1433550830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aad20 msgr2=0x7f19e0157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:09:03.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19e52d6700 1 --2- 172.21.15.7:0/1433550830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aad20 0x7f19e0157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:03.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19e52d6700 1 -- 172.21.15.7:0/1433550830 >> v1:172.21.15.143:6789/0 conn(0x7f19e014b020 legacy=0x7f19e00717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:03.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19e52d6700 1 -- 172.21.15.7:0/1433550830 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e006f890 con 0x7f19e00aa030 2023-03-03T16:09:03.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19dffff700 1 --2- 172.21.15.7:0/1433550830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aad20 0x7f19e0157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:09:03.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19e52d6700 1 --2- 172.21.15.7:0/1433550830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aa030 0x7f19e01581e0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f19d8009e50 tx=0x7f19d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922906653e23a81 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:09:03.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19df7fe700 1 -- 172.21.15.7:0/1433550830 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d800e070 con 0x7f19e00aa030 2023-03-03T16:09:03.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19df7fe700 1 -- 172.21.15.7:0/1433550830 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d800b9c0 con 0x7f19e00aa030 2023-03-03T16:09:03.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19df7fe700 1 -- 172.21.15.7:0/1433550830 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d800bd70 con 0x7f19e00aa030 2023-03-03T16:09:03.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19e6d39700 1 -- 172.21.15.7:0/1433550830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aa030 msgr2=0x7f19e01581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:03.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.477+0000 7f19e6d39700 1 --2- 172.21.15.7:0/1433550830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aa030 0x7f19e01581e0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f19d8009e50 tx=0x7f19d800b330 comp rx=0 tx=0).stop 2023-03-03T16:09:03.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.478+0000 7f19e6d39700 1 -- 172.21.15.7:0/1433550830 shutdown_connections 2023-03-03T16:09:03.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.478+0000 7f19e6d39700 1 --2- 172.21.15.7:0/1433550830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aa030 0x7f19e01581e0 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:03.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.478+0000 7f19e6d39700 1 --2- 172.21.15.7:0/1433550830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aad20 0x7f19e0157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:03.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.478+0000 7f19e6d39700 1 -- 172.21.15.7:0/1433550830 >> 172.21.15.7:0/1433550830 conn(0x7f19e00c8900 msgr2=0x7f19e00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:09:03.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.478+0000 7f19e6d39700 1 -- 172.21.15.7:0/1433550830 shutdown_connections 2023-03-03T16:09:03.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.478+0000 7f19e6d39700 1 -- 172.21.15.7:0/1433550830 wait complete. 2023-03-03T16:09:03.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.479+0000 7f19e6d39700 1 Processor -- start 2023-03-03T16:09:03.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.479+0000 7f19e6d39700 1 -- start start 2023-03-03T16:09:03.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.480+0000 7f19e6d39700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aa030 0x7f19e0147d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:03.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.480+0000 7f19e6d39700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aad20 0x7f19e014b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:03.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.480+0000 7f19e6d39700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19e014b020 0x7f19e014ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:03.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.480+0000 7f19e6d39700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e01482c0 con 0x7f19e00aad20 2023-03-03T16:09:03.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.480+0000 7f19e6d39700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e014d640 con 0x7f19e014b020 2023-03-03T16:09:03.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.480+0000 7f19e6d39700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e014d920 con 0x7f19e00aa030 2023-03-03T16:09:03.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.480+0000 7f19dffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aad20 0x7f19e014b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:03.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aad20 0x7f19e014b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55982/0 (socket says 172.21.15.7:55982) 2023-03-03T16:09:03.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dffff700 1 -- 172.21.15.7:0/1016723482 learned_addr learned my addr 172.21.15.7:0/1016723482 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:09:03.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19e4ad5700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aa030 0x7f19e0147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:03.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19e52d6700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19e014b020 0x7f19e014ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:03.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dffff700 1 -- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aa030 msgr2=0x7f19e0147d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:03.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dffff700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aa030 0x7f19e0147d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:03.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dffff700 1 -- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19e014b020 msgr2=0x7f19e014ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:03.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dffff700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19e014b020 0x7f19e014ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:03.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dffff700 1 -- 172.21.15.7:0/1016723482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d8009b30 con 0x7f19e00aad20 2023-03-03T16:09:03.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dffff700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aad20 0x7f19e014b660 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00e810 tx=0x7f19cc00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:09:03.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dd7fa700 1 -- 172.21.15.7:0/1016723482 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc00c030 con 0x7f19e00aad20 2023-03-03T16:09:03.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dd7fa700 1 -- 172.21.15.7:0/1016723482 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc013610 con 0x7f19e00aad20 2023-03-03T16:09:03.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19dd7fa700 1 -- 172.21.15.7:0/1016723482 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc0139a0 con 0x7f19e00aad20 2023-03-03T16:09:03.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19e6d39700 1 -- 172.21.15.7:0/1016723482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e0155350 con 0x7f19e00aad20 2023-03-03T16:09:03.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.481+0000 7f19e6d39700 1 -- 172.21.15.7:0/1016723482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e006fb70 con 0x7f19e00aad20 2023-03-03T16:09:03.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.482+0000 7f19e6d39700 1 -- 172.21.15.7:0/1016723482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e0071ef0 con 0x7f19e00aad20 2023-03-03T16:09:03.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.483+0000 7f19dd7fa700 1 -- 172.21.15.7:0/1016723482 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f19cc022070 con 0x7f19e00aad20 2023-03-03T16:09:03.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.483+0000 7f19dd7fa700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19c40682f0 0x7f19c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:03.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.483+0000 7f19dd7fa700 1 -- 172.21.15.7:0/1016723482 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f19cc0859e0 con 0x7f19e00aad20 2023-03-03T16:09:03.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.483+0000 7f19e4ad5700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19c40682f0 0x7f19c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:03.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.484+0000 7f19e4ad5700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19c40682f0 0x7f19c406a7a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f19d4007fd0 tx=0x7f19d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:09:03.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.487+0000 7f19dd7fa700 1 -- 172.21.15.7:0/1016723482 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f19cc051d80 con 0x7f19e00aad20 2023-03-03T16:09:03.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.729+0000 7f19e6d39700 1 -- 172.21.15.7:0/1016723482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19e0074bb0 con 0x7f19e00aad20 2023-03-03T16:09:03.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.729+0000 7f19dd7fa700 1 -- 172.21.15.7:0/1016723482 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f19cc0553a0 con 0x7f19e00aad20 2023-03-03T16:09:03.730 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:09:03.731 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:09:03.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 -- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19c40682f0 msgr2=0x7f19c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:03.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19c40682f0 0x7f19c406a7a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f19d4007fd0 tx=0x7f19d4009040 comp rx=0 tx=0).stop 2023-03-03T16:09:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 -- 172.21.15.7:0/1016723482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aad20 msgr2=0x7f19e014b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aad20 0x7f19e014b660 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00e810 tx=0x7f19cc00eb20 comp rx=0 tx=0).stop 2023-03-03T16:09:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 -- 172.21.15.7:0/1016723482 shutdown_connections 2023-03-03T16:09:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19e014b020 0x7f19e014ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f19e00aa030 0x7f19e0147d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19c40682f0 0x7f19c406a7a0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 --2- 172.21.15.7:0/1016723482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19e00aad20 0x7f19e014b660 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 -- 172.21.15.7:0/1016723482 >> 172.21.15.7:0/1016723482 conn(0x7f19e00c8900 msgr2=0x7f19e014a280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:09:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 -- 172.21.15.7:0/1016723482 shutdown_connections 2023-03-03T16:09:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:03.732+0000 7f19e6d39700 1 -- 172.21.15.7:0/1016723482 wait complete. 2023-03-03T16:09:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:09:03.744 INFO:tasks.mds_thrash.fs.[b]:waiting for 95 secs before thrashing 2023-03-03T16:09:12.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:12.746 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:24.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:09:25.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.033+0000 7fe15602e700 1 Processor -- start 2023-03-03T16:09:25.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.033+0000 7fe15602e700 1 -- start start 2023-03-03T16:09:25.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.034+0000 7fe15602e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe15014fbc0 0x7fe150152e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:25.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.034+0000 7fe15602e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1500a5d20 0x7fe1500a85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:25.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.034+0000 7fe15602e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1500ba3d0 con 0x7fe1500a8ae0 2023-03-03T16:09:25.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.034+0000 7fe15602e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1500c1540 con 0x7fe1500a5d20 2023-03-03T16:09:25.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.034+0000 7fe15602e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1500c1820 con 0x7fe15014fbc0 2023-03-03T16:09:25.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.034+0000 7fe14effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe1500a5d20 0x7fe1500a85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:25.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.034+0000 7fe14ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe1500a8ae0 0x7fe1500c25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49988/0 (socket says 172.21.15.7:49988) 2023-03-03T16:09:25.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.034+0000 7fe14ffff700 1 -- 172.21.15.7:0/4262953691 learned_addr learned my addr 172.21.15.7:0/4262953691 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:09:25.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14f7fe700 1 --2- 172.21.15.7:0/4262953691 >> v2:172.21.15.143:3300/0 conn(0x7fe15014fbc0 0x7fe150152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:25.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14effd700 1 -- 172.21.15.7:0/4262953691 >> v2:172.21.15.143:3300/0 conn(0x7fe15014fbc0 msgr2=0x7fe150152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:09:25.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14effd700 1 --2- 172.21.15.7:0/4262953691 >> v2:172.21.15.143:3300/0 conn(0x7fe15014fbc0 0x7fe150152e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:25.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14effd700 1 -- 172.21.15.7:0/4262953691 >> v1:172.21.15.7:6789/0 conn(0x7fe1500a8ae0 legacy=0x7fe1500c25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:25.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14effd700 1 -- 172.21.15.7:0/4262953691 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1500c1b00 con 0x7fe1500a5d20 2023-03-03T16:09:25.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14e7fc700 1 -- 172.21.15.7:0/4262953691 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050855610 0 0) 0x7fe1500ba3d0 con 0x7fe1500a8ae0 2023-03-03T16:09:25.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14f7fe700 1 --2- 172.21.15.7:0/4262953691 >> v2:172.21.15.143:3300/0 conn(0x7fe15014fbc0 0x7fe150152e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:09:25.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14effd700 1 --2- 172.21.15.7:0/4262953691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 0x7fe1500a85a0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fe15006f740 tx=0x7fe138009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d025a891f028d633 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:09:25.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14e7fc700 1 -- 172.21.15.7:0/4262953691 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138011830 con 0x7fe1500a5d20 2023-03-03T16:09:25.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14e7fc700 1 -- 172.21.15.7:0/4262953691 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe138011990 con 0x7fe1500a5d20 2023-03-03T16:09:25.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe14e7fc700 1 -- 172.21.15.7:0/4262953691 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138011d20 con 0x7fe1500a5d20 2023-03-03T16:09:25.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe15602e700 1 -- 172.21.15.7:0/4262953691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 msgr2=0x7fe1500a85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:25.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.035+0000 7fe15602e700 1 --2- 172.21.15.7:0/4262953691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 0x7fe1500a85a0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fe15006f740 tx=0x7fe138009b20 comp rx=0 tx=0).stop 2023-03-03T16:09:25.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.036+0000 7fe15602e700 1 -- 172.21.15.7:0/4262953691 shutdown_connections 2023-03-03T16:09:25.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.036+0000 7fe15602e700 1 --2- 172.21.15.7:0/4262953691 >> v2:172.21.15.143:3300/0 conn(0x7fe15014fbc0 0x7fe150152e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:25.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.036+0000 7fe15602e700 1 --2- 172.21.15.7:0/4262953691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 0x7fe1500a85a0 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:25.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.036+0000 7fe15602e700 1 -- 172.21.15.7:0/4262953691 >> 172.21.15.7:0/4262953691 conn(0x7fe15006af40 msgr2=0x7fe1500c6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:09:25.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.036+0000 7fe15602e700 1 -- 172.21.15.7:0/4262953691 shutdown_connections 2023-03-03T16:09:25.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.036+0000 7fe15602e700 1 -- 172.21.15.7:0/4262953691 wait complete. 2023-03-03T16:09:25.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.037+0000 7fe15602e700 1 Processor -- start 2023-03-03T16:09:25.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.037+0000 7fe15602e700 1 -- start start 2023-03-03T16:09:25.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe15602e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 0x7fe1500a8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:25.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe15602e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1500a8ae0 0x7fe1500c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:25.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe15602e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15014fbc0 0x7fe1501ed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:25.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe15602e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1500a57c0 con 0x7fe1500a5d20 2023-03-03T16:09:25.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe15602e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1500a6620 con 0x7fe1500a8ae0 2023-03-03T16:09:25.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe15602e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1500a8760 con 0x7fe15014fbc0 2023-03-03T16:09:25.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 0x7fe1500a8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:25.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 0x7fe1500a8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56010/0 (socket says 172.21.15.7:56010) 2023-03-03T16:09:25.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14f7fe700 1 -- 172.21.15.7:0/1935578270 learned_addr learned my addr 172.21.15.7:0/1935578270 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:09:25.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14effd700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1500a8ae0 0x7fe1500c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:25.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14ffff700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15014fbc0 0x7fe1501ed4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:25.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14f7fe700 1 -- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15014fbc0 msgr2=0x7fe1501ed4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:25.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14f7fe700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15014fbc0 0x7fe1501ed4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:25.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14f7fe700 1 -- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1500a8ae0 msgr2=0x7fe1500c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:25.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14f7fe700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1500a8ae0 0x7fe1500c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:25.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14f7fe700 1 -- 172.21.15.7:0/1935578270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1380094a0 con 0x7fe1500a5d20 2023-03-03T16:09:25.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe14f7fe700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 0x7fe1500a8220 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7fe14000c820 tx=0x7fe14000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:09:25.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe147fff700 1 -- 172.21.15.7:0/1935578270 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140011800 con 0x7fe1500a5d20 2023-03-03T16:09:25.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe147fff700 1 -- 172.21.15.7:0/1935578270 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe140011960 con 0x7fe1500a5d20 2023-03-03T16:09:25.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe147fff700 1 -- 172.21.15.7:0/1935578270 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140011d10 con 0x7fe1500a5d20 2023-03-03T16:09:25.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.038+0000 7fe15602e700 1 -- 172.21.15.7:0/1935578270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1500c20c0 con 0x7fe1500a5d20 2023-03-03T16:09:25.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.039+0000 7fe15602e700 1 -- 172.21.15.7:0/1935578270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1500c1de0 con 0x7fe1500a5d20 2023-03-03T16:09:25.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.039+0000 7fe15602e700 1 -- 172.21.15.7:0/1935578270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe150153800 con 0x7fe1500a5d20 2023-03-03T16:09:25.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.040+0000 7fe147fff700 1 -- 172.21.15.7:0/1935578270 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe140021070 con 0x7fe1500a5d20 2023-03-03T16:09:25.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.040+0000 7fe147fff700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1300682f0 0x7fe13006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:09:25.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.040+0000 7fe147fff700 1 -- 172.21.15.7:0/1935578270 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe140086050 con 0x7fe1500a5d20 2023-03-03T16:09:25.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.041+0000 7fe14effd700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1300682f0 0x7fe13006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:09:25.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.041+0000 7fe14effd700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1300682f0 0x7fe13006a7a0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fe1380090b0 tx=0x7fe138008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:09:25.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.045+0000 7fe147fff700 1 -- 172.21.15.7:0/1935578270 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe140052370 con 0x7fe1500a5d20 2023-03-03T16:09:25.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.283+0000 7fe15602e700 1 -- 172.21.15.7:0/1935578270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe150074bb0 con 0x7fe1500a5d20 2023-03-03T16:09:25.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.284+0000 7fe147fff700 1 -- 172.21.15.7:0/1935578270 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe140055990 con 0x7fe1500a5d20 2023-03-03T16:09:25.285 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:09:25.285 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:09:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 -- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1300682f0 msgr2=0x7fe13006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1300682f0 0x7fe13006a7a0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fe1380090b0 tx=0x7fe138008030 comp rx=0 tx=0).stop 2023-03-03T16:09:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 -- 172.21.15.7:0/1935578270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 msgr2=0x7fe1500a8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:09:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 0x7fe1500a8220 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7fe14000c820 tx=0x7fe14000cbe0 comp rx=0 tx=0).stop 2023-03-03T16:09:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 -- 172.21.15.7:0/1935578270 shutdown_connections 2023-03-03T16:09:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15014fbc0 0x7fe1501ed4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:25.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe1300682f0 0x7fe13006a7a0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe1500a5d20 0x7fe1500a8220 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 --2- 172.21.15.7:0/1935578270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1500a8ae0 0x7fe1500c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:09:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 -- 172.21.15.7:0/1935578270 >> 172.21.15.7:0/1935578270 conn(0x7fe15006af40 msgr2=0x7fe1500713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:09:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 -- 172.21.15.7:0/1935578270 shutdown_connections 2023-03-03T16:09:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:09:25.286+0000 7fe15602e700 1 -- 172.21.15.7:0/1935578270 wait complete. 2023-03-03T16:09:25.288 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:09:25.298 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T16:09:42.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:09:42.819 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T16:09:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T16:10:07.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:10:07.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.438+0000 7f280426c700 1 Processor -- start 2023-03-03T16:10:07.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.438+0000 7f280426c700 1 -- start start 2023-03-03T16:10:07.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.439+0000 7f280426c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f27fc14b020 0x7f27fc0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:07.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.439+0000 7f280426c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc071d20 0x7f27fc1583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:07.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.439+0000 7f280426c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc1588f0 con 0x7f27fc0aad20 2023-03-03T16:10:07.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.439+0000 7f280426c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27fc0728b0 con 0x7f27fc14b020 2023-03-03T16:10:07.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.439+0000 7f280426c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc155070 con 0x7f27fc071d20 2023-03-03T16:10:07.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.439+0000 7f2802008700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f27fc14b020 0x7f27fc0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:07.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.439+0000 7f2801807700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f27fc0aad20 0x7f27fc157ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50030/0 (socket says 172.21.15.7:50030) 2023-03-03T16:10:07.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2801807700 1 -- 172.21.15.7:0/160949117 learned_addr learned my addr 172.21.15.7:0/160949117 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:10:07.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2802809700 1 --2- 172.21.15.7:0/160949117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc071d20 0x7f27fc1583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:07.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2802008700 1 -- 172.21.15.7:0/160949117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc071d20 msgr2=0x7f27fc1583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:10:07.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2802008700 1 --2- 172.21.15.7:0/160949117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc071d20 0x7f27fc1583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:07.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2802008700 1 -- 172.21.15.7:0/160949117 >> v1:172.21.15.7:6789/0 conn(0x7f27fc0aad20 legacy=0x7f27fc157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:07.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2802008700 1 -- 172.21.15.7:0/160949117 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc06f5b0 con 0x7f27fc14b020 2023-03-03T16:10:07.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2802809700 1 --2- 172.21.15.7:0/160949117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc071d20 0x7f27fc1583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:10:07.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2801006700 1 -- 172.21.15.7:0/160949117 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506271822 0 0) 0x7f27fc1588f0 con 0x7f27fc0aad20 2023-03-03T16:10:07.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2802008700 1 --2- 172.21.15.7:0/160949117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 0x7f27fc0717e0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009840 tx=0x7f27f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c560d0f99911079 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:07.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2801006700 1 -- 172.21.15.7:0/160949117 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8011830 con 0x7f27fc14b020 2023-03-03T16:10:07.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2801006700 1 -- 172.21.15.7:0/160949117 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f8011990 con 0x7f27fc14b020 2023-03-03T16:10:07.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.440+0000 7f2801006700 1 -- 172.21.15.7:0/160949117 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8011d20 con 0x7f27fc14b020 2023-03-03T16:10:07.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.441+0000 7f280426c700 1 -- 172.21.15.7:0/160949117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 msgr2=0x7f27fc0717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:07.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.441+0000 7f280426c700 1 --2- 172.21.15.7:0/160949117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 0x7f27fc0717e0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009840 tx=0x7f27f8009b50 comp rx=0 tx=0).stop 2023-03-03T16:10:07.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.441+0000 7f280426c700 1 -- 172.21.15.7:0/160949117 shutdown_connections 2023-03-03T16:10:07.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.441+0000 7f280426c700 1 --2- 172.21.15.7:0/160949117 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc071d20 0x7f27fc1583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:07.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.441+0000 7f280426c700 1 --2- 172.21.15.7:0/160949117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 0x7f27fc0717e0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:07.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.441+0000 7f280426c700 1 -- 172.21.15.7:0/160949117 >> 172.21.15.7:0/160949117 conn(0x7f27fc0c8900 msgr2=0x7f27fc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:10:07.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.441+0000 7f280426c700 1 -- 172.21.15.7:0/160949117 shutdown_connections 2023-03-03T16:10:07.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.441+0000 7f280426c700 1 -- 172.21.15.7:0/160949117 wait complete. 2023-03-03T16:10:07.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.442+0000 7f280426c700 1 Processor -- start 2023-03-03T16:10:07.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.442+0000 7f280426c700 1 -- start start 2023-03-03T16:10:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f280426c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27fc071d20 0x7f27fc1536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f280426c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc0aad20 0x7f27fc1e9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f280426c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 0x7f27fc1511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f280426c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc147610 con 0x7f27fc14b020 2023-03-03T16:10:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f280426c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc147370 con 0x7f27fc071d20 2023-03-03T16:10:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f280426c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc1ed600 con 0x7f27fc0aad20 2023-03-03T16:10:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2802809700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 0x7f27fc1511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:07.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2802809700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 0x7f27fc1511e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56056/0 (socket says 172.21.15.7:56056) 2023-03-03T16:10:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2802809700 1 -- 172.21.15.7:0/1304259603 learned_addr learned my addr 172.21.15.7:0/1304259603 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:10:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2801807700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc0aad20 0x7f27fc1e9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2802008700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27fc071d20 0x7f27fc1536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2802809700 1 -- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc0aad20 msgr2=0x7f27fc1e9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2802809700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc0aad20 0x7f27fc1e9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2802809700 1 -- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27fc071d20 msgr2=0x7f27fc1536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:07.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2802809700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27fc071d20 0x7f27fc1536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2802809700 1 -- 172.21.15.7:0/1304259603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f80094a0 con 0x7f27fc14b020 2023-03-03T16:10:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2801807700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc0aad20 0x7f27fc1e9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:10:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.443+0000 7f2802809700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 0x7f27fc1511e0 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f27f4007df0 tx=0x7f27f400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.444+0000 7f27f2ffd700 1 -- 172.21.15.7:0/1304259603 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f400ddd0 con 0x7f27fc14b020 2023-03-03T16:10:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.444+0000 7f27f2ffd700 1 -- 172.21.15.7:0/1304259603 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f400f040 con 0x7f27fc14b020 2023-03-03T16:10:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.444+0000 7f280426c700 1 -- 172.21.15.7:0/1304259603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc06fb70 con 0x7f27fc14b020 2023-03-03T16:10:07.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.444+0000 7f27f2ffd700 1 -- 172.21.15.7:0/1304259603 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f4003c60 con 0x7f27fc14b020 2023-03-03T16:10:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.444+0000 7f280426c700 1 -- 172.21.15.7:0/1304259603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc06f890 con 0x7f27fc14b020 2023-03-03T16:10:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.445+0000 7f280426c700 1 -- 172.21.15.7:0/1304259603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc155350 con 0x7f27fc14b020 2023-03-03T16:10:07.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.445+0000 7f27f2ffd700 1 -- 172.21.15.7:0/1304259603 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f27f401c030 con 0x7f27fc14b020 2023-03-03T16:10:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.445+0000 7f27f2ffd700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27e00682f0 0x7f27e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.445+0000 7f27f2ffd700 1 -- 172.21.15.7:0/1304259603 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f27f40861b0 con 0x7f27fc14b020 2023-03-03T16:10:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.446+0000 7f2802008700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27e00682f0 0x7f27e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.446+0000 7f2802008700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27e00682f0 0x7f27e006a7a0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f27f80090b0 tx=0x7f27f801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:07.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.450+0000 7f27f2ffd700 1 -- 172.21.15.7:0/1304259603 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f27f4052550 con 0x7f27fc14b020 2023-03-03T16:10:07.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.688+0000 7f280426c700 1 -- 172.21.15.7:0/1304259603 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27fc074bb0 con 0x7f27fc14b020 2023-03-03T16:10:07.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.688+0000 7f27f2ffd700 1 -- 172.21.15.7:0/1304259603 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f27f405b090 con 0x7f27fc14b020 2023-03-03T16:10:07.689 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:10:07.689 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:10:07.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 -- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27e00682f0 msgr2=0x7f27e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:07.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27e00682f0 0x7f27e006a7a0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f27f80090b0 tx=0x7f27f801c040 comp rx=0 tx=0).stop 2023-03-03T16:10:07.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 -- 172.21.15.7:0/1304259603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 msgr2=0x7f27fc1511e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:07.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 0x7f27fc1511e0 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f27f4007df0 tx=0x7f27f400d3b0 comp rx=0 tx=0).stop 2023-03-03T16:10:07.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 -- 172.21.15.7:0/1304259603 shutdown_connections 2023-03-03T16:10:07.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f27e00682f0 0x7f27e006a7a0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:07.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27fc14b020 0x7f27fc1511e0 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:07.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27fc071d20 0x7f27fc1536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:07.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 --2- 172.21.15.7:0/1304259603 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f27fc0aad20 0x7f27fc1e9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:07.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 -- 172.21.15.7:0/1304259603 >> 172.21.15.7:0/1304259603 conn(0x7f27fc0c8900 msgr2=0x7f27fc155f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:10:07.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 -- 172.21.15.7:0/1304259603 shutdown_connections 2023-03-03T16:10:07.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:07.690+0000 7f280426c700 1 -- 172.21.15.7:0/1304259603 wait complete. 2023-03-03T16:10:07.693 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:10:07.703 INFO:tasks.mds_thrash.fs.[a]:waiting for 89 secs before thrashing 2023-03-03T16:10:12.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:12.882 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:38.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:10:38.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.883+0000 7f169609e700 1 Processor -- start 2023-03-03T16:10:38.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.883+0000 7f169609e700 1 -- start start 2023-03-03T16:10:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.884+0000 7f169609e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1690151e80 0x7f16900c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.884+0000 7f169609e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1690155410 0x7f16900bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.884+0000 7f169609e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16900c08d0 con 0x7f16900c6fe0 2023-03-03T16:10:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.884+0000 7f169609e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1690152250 con 0x7f1690151e80 2023-03-03T16:10:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.884+0000 7f169609e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16900c0fb0 con 0x7f1690155410 2023-03-03T16:10:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.884+0000 7f168f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1690151e80 0x7f16900c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.884+0000 7f168ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f16900c6fe0 0x7f16900bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50070/0 (socket says 172.21.15.7:50070) 2023-03-03T16:10:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.884+0000 7f168ffff700 1 -- 172.21.15.7:0/496121773 learned_addr learned my addr 172.21.15.7:0/496121773 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:10:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.885+0000 7f168effd700 1 --2- 172.21.15.7:0/496121773 >> v2:172.21.15.143:3300/0 conn(0x7f1690155410 0x7f16900bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.885+0000 7f168f7fe700 1 -- 172.21.15.7:0/496121773 >> v2:172.21.15.143:3300/0 conn(0x7f1690155410 msgr2=0x7f16900bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:10:38.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.885+0000 7f168f7fe700 1 --2- 172.21.15.7:0/496121773 >> v2:172.21.15.143:3300/0 conn(0x7f1690155410 0x7f16900bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:38.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.885+0000 7f168f7fe700 1 -- 172.21.15.7:0/496121773 >> v1:172.21.15.7:6789/0 conn(0x7f16900c6fe0 legacy=0x7f16900bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:38.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.885+0000 7f168e7fc700 1 -- 172.21.15.7:0/496121773 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127682906 0 0) 0x7f16900c08d0 con 0x7f16900c6fe0 2023-03-03T16:10:38.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.885+0000 7f168f7fe700 1 -- 172.21.15.7:0/496121773 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1690154930 con 0x7f1690151e80 2023-03-03T16:10:38.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.885+0000 7f168f7fe700 1 --2- 172.21.15.7:0/496121773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 0x7f16900c6aa0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f169006c380 tx=0x7f1680009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b822bfcb05b83d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:38.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.885+0000 7f168e7fc700 1 -- 172.21.15.7:0/496121773 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680011830 con 0x7f1690151e80 2023-03-03T16:10:38.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.885+0000 7f168e7fc700 1 -- 172.21.15.7:0/496121773 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1680011990 con 0x7f1690151e80 2023-03-03T16:10:38.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.885+0000 7f168e7fc700 1 -- 172.21.15.7:0/496121773 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680011d20 con 0x7f1690151e80 2023-03-03T16:10:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.886+0000 7f169609e700 1 -- 172.21.15.7:0/496121773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 msgr2=0x7f16900c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.886+0000 7f169609e700 1 --2- 172.21.15.7:0/496121773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 0x7f16900c6aa0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f169006c380 tx=0x7f1680009b20 comp rx=0 tx=0).stop 2023-03-03T16:10:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.886+0000 7f169609e700 1 -- 172.21.15.7:0/496121773 shutdown_connections 2023-03-03T16:10:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.886+0000 7f169609e700 1 --2- 172.21.15.7:0/496121773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 0x7f16900c6aa0 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.886+0000 7f169609e700 1 --2- 172.21.15.7:0/496121773 >> v2:172.21.15.143:3300/0 conn(0x7f1690155410 0x7f16900bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.886+0000 7f169609e700 1 -- 172.21.15.7:0/496121773 >> 172.21.15.7:0/496121773 conn(0x7f1690071630 msgr2=0x7f169006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:10:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.886+0000 7f169609e700 1 -- 172.21.15.7:0/496121773 shutdown_connections 2023-03-03T16:10:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.886+0000 7f169609e700 1 -- 172.21.15.7:0/496121773 wait complete. 2023-03-03T16:10:38.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.887+0000 7f169609e700 1 Processor -- start 2023-03-03T16:10:38.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.887+0000 7f169609e700 1 -- start start 2023-03-03T16:10:38.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f169609e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16900c6fe0 0x7f1690146dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:38.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f169609e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 0x7f1690158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:38.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f169609e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1690155410 0x7f16900c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:38.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f169609e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169014acc0 con 0x7f1690151e80 2023-03-03T16:10:38.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f169609e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16901eabc0 con 0x7f1690155410 2023-03-03T16:10:38.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f169609e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16901eb0f0 con 0x7f16900c6fe0 2023-03-03T16:10:38.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 0x7f1690158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:38.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 0x7f1690158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56094/0 (socket says 172.21.15.7:56094) 2023-03-03T16:10:38.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168effd700 1 -- 172.21.15.7:0/4013260762 learned_addr learned my addr 172.21.15.7:0/4013260762 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:10:38.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168f7fe700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16900c6fe0 0x7f1690146dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:38.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168ffff700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1690155410 0x7f16900c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168effd700 1 -- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16900c6fe0 msgr2=0x7f1690146dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168effd700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16900c6fe0 0x7f1690146dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168effd700 1 -- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1690155410 msgr2=0x7f16900c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168effd700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1690155410 0x7f16900c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168effd700 1 -- 172.21.15.7:0/4013260762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16800094a0 con 0x7f1690151e80 2023-03-03T16:10:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168f7fe700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16900c6fe0 0x7f1690146dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:10:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168ffff700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1690155410 0x7f16900c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:10:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.888+0000 7f168effd700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 0x7f1690158370 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f167800e870 tx=0x7f167800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.889+0000 7f1687fff700 1 -- 172.21.15.7:0/4013260762 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167800c5e0 con 0x7f1690151e80 2023-03-03T16:10:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.889+0000 7f1687fff700 1 -- 172.21.15.7:0/4013260762 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167800c740 con 0x7f1690151e80 2023-03-03T16:10:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.889+0000 7f1687fff700 1 -- 172.21.15.7:0/4013260762 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167800cab0 con 0x7f1690151e80 2023-03-03T16:10:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.889+0000 7f169609e700 1 -- 172.21.15.7:0/4013260762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1690154ef0 con 0x7f1690151e80 2023-03-03T16:10:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.889+0000 7f169609e700 1 -- 172.21.15.7:0/4013260762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1690154c10 con 0x7f1690151e80 2023-03-03T16:10:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.890+0000 7f169609e700 1 -- 172.21.15.7:0/4013260762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16900c5220 con 0x7f1690151e80 2023-03-03T16:10:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.890+0000 7f1687fff700 1 -- 172.21.15.7:0/4013260762 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1678023070 con 0x7f1690151e80 2023-03-03T16:10:38.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.890+0000 7f1687fff700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16700682f0 0x7f167006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:10:38.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.891+0000 7f1687fff700 1 -- 172.21.15.7:0/4013260762 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1678086330 con 0x7f1690151e80 2023-03-03T16:10:38.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.891+0000 7f168f7fe700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16700682f0 0x7f167006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:10:38.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.891+0000 7f168f7fe700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16700682f0 0x7f167006a7a0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f16900cb6c0 tx=0x7f1680008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:10:38.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:38.895+0000 7f1687fff700 1 -- 172.21.15.7:0/4013260762 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1678052580 con 0x7f1690151e80 2023-03-03T16:10:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.132+0000 7f169609e700 1 -- 172.21.15.7:0/4013260762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1690074bb0 con 0x7f1690151e80 2023-03-03T16:10:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.132+0000 7f1687fff700 1 -- 172.21.15.7:0/4013260762 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1678055ba0 con 0x7f1690151e80 2023-03-03T16:10:39.133 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:10:39.134 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:10:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 -- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16700682f0 msgr2=0x7f167006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16700682f0 0x7f167006a7a0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f16900cb6c0 tx=0x7f1680008030 comp rx=0 tx=0).stop 2023-03-03T16:10:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 -- 172.21.15.7:0/4013260762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 msgr2=0x7f1690158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:10:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 0x7f1690158370 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f167800e870 tx=0x7f167800ec30 comp rx=0 tx=0).stop 2023-03-03T16:10:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 -- 172.21.15.7:0/4013260762 shutdown_connections 2023-03-03T16:10:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1690155410 0x7f16900c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16900c6fe0 0x7f1690146dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f16700682f0 0x7f167006a7a0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 --2- 172.21.15.7:0/4013260762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1690151e80 0x7f1690158370 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:10:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 -- 172.21.15.7:0/4013260762 >> 172.21.15.7:0/4013260762 conn(0x7f1690071630 msgr2=0x7f16900c2940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:10:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 -- 172.21.15.7:0/4013260762 shutdown_connections 2023-03-03T16:10:39.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:10:39.134+0000 7f169609e700 1 -- 172.21.15.7:0/4013260762 wait complete. 2023-03-03T16:10:39.137 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:10:39.147 INFO:tasks.mds_thrash.fs.[b]:waiting for 99 secs before thrashing 2023-03-03T16:10:42.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:10:42.954 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:13.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:13.018 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:36.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:11:36.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.843+0000 7fe285d8a700 1 Processor -- start 2023-03-03T16:11:36.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.843+0000 7fe285d8a700 1 -- start start 2023-03-03T16:11:36.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe285d8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe280152e30 0x7fe2800c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe285d8a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2800c4910 0x7fe280156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe285d8a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2801523b0 con 0x7fe2800beab0 2023-03-03T16:11:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe285d8a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe280072010 con 0x7fe2800c4910 2023-03-03T16:11:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe285d8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2800aaaf0 con 0x7fe280152e30 2023-03-03T16:11:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe27ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2800c4910 0x7fe280156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:36.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe27ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2800c4910 0x7fe280156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56152/0 (socket says 172.21.15.7:56152) 2023-03-03T16:11:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe27ffff700 1 -- 172.21.15.7:0/2522283823 learned_addr learned my addr 172.21.15.7:0/2522283823 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:11:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe27f7fe700 1 --2- 172.21.15.7:0/2522283823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe280152e30 0x7fe2800c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe27ffff700 1 -- 172.21.15.7:0/2522283823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe280152e30 msgr2=0x7fe2800c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:11:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe27ffff700 1 --2- 172.21.15.7:0/2522283823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe280152e30 0x7fe2800c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe27ffff700 1 -- 172.21.15.7:0/2522283823 >> v1:172.21.15.143:6789/0 conn(0x7fe2800beab0 legacy=0x7fe280155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe27ffff700 1 -- 172.21.15.7:0/2522283823 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800aadd0 con 0x7fe2800c4910 2023-03-03T16:11:36.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.844+0000 7fe27ffff700 1 --2- 172.21.15.7:0/2522283823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800c4910 0x7fe280156490 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fe278009e50 tx=0x7fe27800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22d548933605526 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:11:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe27e7fc700 1 -- 172.21.15.7:0/2522283823 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27800e070 con 0x7fe2800c4910 2023-03-03T16:11:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe27e7fc700 1 -- 172.21.15.7:0/2522283823 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27800b9c0 con 0x7fe2800c4910 2023-03-03T16:11:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe27e7fc700 1 -- 172.21.15.7:0/2522283823 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27800bd70 con 0x7fe2800c4910 2023-03-03T16:11:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe285d8a700 1 -- 172.21.15.7:0/2522283823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800c4910 msgr2=0x7fe280156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe285d8a700 1 --2- 172.21.15.7:0/2522283823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800c4910 0x7fe280156490 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fe278009e50 tx=0x7fe27800b330 comp rx=0 tx=0).stop 2023-03-03T16:11:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe285d8a700 1 -- 172.21.15.7:0/2522283823 shutdown_connections 2023-03-03T16:11:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe285d8a700 1 --2- 172.21.15.7:0/2522283823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800c4910 0x7fe280156490 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:36.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe285d8a700 1 --2- 172.21.15.7:0/2522283823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe280152e30 0x7fe2800c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe285d8a700 1 -- 172.21.15.7:0/2522283823 >> 172.21.15.7:0/2522283823 conn(0x7fe2800736f0 msgr2=0x7fe280073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:11:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe285d8a700 1 -- 172.21.15.7:0/2522283823 shutdown_connections 2023-03-03T16:11:36.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.845+0000 7fe285d8a700 1 -- 172.21.15.7:0/2522283823 wait complete. 2023-03-03T16:11:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.846+0000 7fe285d8a700 1 Processor -- start 2023-03-03T16:11:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.847+0000 7fe285d8a700 1 -- start start 2023-03-03T16:11:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.847+0000 7fe285d8a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800beab0 0x7fe280153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.847+0000 7fe285d8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2800c4910 0x7fe2801ed530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.847+0000 7fe285d8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe280152e30 0x7fe280155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.847+0000 7fe285d8a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe280155010 con 0x7fe2800beab0 2023-03-03T16:11:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.847+0000 7fe285d8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe280153a90 con 0x7fe280152e30 2023-03-03T16:11:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.847+0000 7fe285d8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2801eda70 con 0x7fe2800c4910 2023-03-03T16:11:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.847+0000 7fe27f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800beab0 0x7fe280153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.847+0000 7fe27f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800beab0 0x7fe280153550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56154/0 (socket says 172.21.15.7:56154) 2023-03-03T16:11:36.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.847+0000 7fe27f7fe700 1 -- 172.21.15.7:0/1504081631 learned_addr learned my addr 172.21.15.7:0/1504081631 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:11:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe27f7fe700 1 -- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2800c4910 msgr2=0x7fe2801ed530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe27ffff700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe280152e30 0x7fe280155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe27effd700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2800c4910 0x7fe2801ed530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe27f7fe700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2800c4910 0x7fe2801ed530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe27f7fe700 1 -- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe280152e30 msgr2=0x7fe280155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe27f7fe700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe280152e30 0x7fe280155ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:36.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe27f7fe700 1 -- 172.21.15.7:0/1504081631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe278009b30 con 0x7fe2800beab0 2023-03-03T16:11:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe27f7fe700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800beab0 0x7fe280153550 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fe26800e810 tx=0x7fe26800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:11:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe277fff700 1 -- 172.21.15.7:0/1504081631 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268013610 con 0x7fe2800beab0 2023-03-03T16:11:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe277fff700 1 -- 172.21.15.7:0/1504081631 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe268013770 con 0x7fe2800beab0 2023-03-03T16:11:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe285d8a700 1 -- 172.21.15.7:0/1504081631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe280153200 con 0x7fe2800beab0 2023-03-03T16:11:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe277fff700 1 -- 172.21.15.7:0/1504081631 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268013b20 con 0x7fe2800beab0 2023-03-03T16:11:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.848+0000 7fe285d8a700 1 -- 172.21.15.7:0/1504081631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe280071c70 con 0x7fe2800beab0 2023-03-03T16:11:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.849+0000 7fe285d8a700 1 -- 172.21.15.7:0/1504081631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe280153390 con 0x7fe2800beab0 2023-03-03T16:11:36.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.850+0000 7fe277fff700 1 -- 172.21.15.7:0/1504081631 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe268023070 con 0x7fe2800beab0 2023-03-03T16:11:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.850+0000 7fe277fff700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe260068320 0x7fe26006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:11:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.850+0000 7fe277fff700 1 -- 172.21.15.7:0/1504081631 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe268085ec0 con 0x7fe2800beab0 2023-03-03T16:11:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.850+0000 7fe27effd700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe260068320 0x7fe26006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:11:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.851+0000 7fe27effd700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe260068320 0x7fe26006a7d0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fe2800bd660 tx=0x7fe270009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:11:36.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:36.854+0000 7fe277fff700 1 -- 172.21.15.7:0/1504081631 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2680521e0 con 0x7fe2800beab0 2023-03-03T16:11:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.093+0000 7fe285d8a700 1 -- 172.21.15.7:0/1504081631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe280074bb0 con 0x7fe2800beab0 2023-03-03T16:11:37.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.093+0000 7fe277fff700 1 -- 172.21.15.7:0/1504081631 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe268055800 con 0x7fe2800beab0 2023-03-03T16:11:37.094 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:11:37.094 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:11:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 -- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe260068320 msgr2=0x7fe26006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe260068320 0x7fe26006a7d0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fe2800bd660 tx=0x7fe270009040 comp rx=0 tx=0).stop 2023-03-03T16:11:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 -- 172.21.15.7:0/1504081631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800beab0 msgr2=0x7fe280153550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:11:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800beab0 0x7fe280153550 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fe26800e810 tx=0x7fe26800ebd0 comp rx=0 tx=0).stop 2023-03-03T16:11:37.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 -- 172.21.15.7:0/1504081631 shutdown_connections 2023-03-03T16:11:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe280152e30 0x7fe280155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe260068320 0x7fe26006a7d0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2800beab0 0x7fe280153550 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 --2- 172.21.15.7:0/1504081631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2800c4910 0x7fe2801ed530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:11:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 -- 172.21.15.7:0/1504081631 >> 172.21.15.7:0/1504081631 conn(0x7fe2800736f0 msgr2=0x7fe2800aa020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:11:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 -- 172.21.15.7:0/1504081631 shutdown_connections 2023-03-03T16:11:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:11:37.095+0000 7fe285d8a700 1 -- 172.21.15.7:0/1504081631 wait complete. 2023-03-03T16:11:37.098 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:11:37.108 INFO:tasks.mds_thrash.fs.[a]:waiting for 81 secs before thrashing 2023-03-03T16:11:43.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:11:43.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:13.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:13.168 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:18.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:12:18.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.286+0000 7fa91365c700 1 Processor -- start 2023-03-03T16:12:18.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.286+0000 7fa91365c700 1 -- start start 2023-03-03T16:12:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa90c151e80 0x7fa90c0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90c0c08d0 con 0x7fa90c0c6fe0 2023-03-03T16:12:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90c154930 con 0x7fa90c155410 2023-03-03T16:12:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa90c152250 con 0x7fa90c151e80 2023-03-03T16:12:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa9113f8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa90c151e80 0x7fa90c0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:18.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa911bf9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa90c0c6fe0 0x7fa90c0bfff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50160/0 (socket says 172.21.15.7:50160) 2023-03-03T16:12:18.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa911bf9700 1 -- 172.21.15.7:0/1363588380 learned_addr learned my addr 172.21.15.7:0/1363588380 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:12:18.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa9113f8700 1 -- 172.21.15.7:0/1363588380 >> v1:172.21.15.143:6789/0 conn(0x7fa90c155410 legacy=0x7fa90c0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:12:18.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa9113f8700 1 -- 172.21.15.7:0/1363588380 >> v1:172.21.15.7:6789/0 conn(0x7fa90c0c6fe0 legacy=0x7fa90c0bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:18.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa9113f8700 1 -- 172.21.15.7:0/1363588380 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c154c10 con 0x7fa90c151e80 2023-03-03T16:12:18.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa9113f8700 1 --2- 172.21.15.7:0/1363588380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 0x7fa90c0c6aa0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009840 tx=0x7fa8fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbdca41635ec933 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:12:18.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa903fff700 1 -- 172.21.15.7:0/1363588380 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc004230 con 0x7fa90c151e80 2023-03-03T16:12:18.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa903fff700 1 -- 172.21.15.7:0/1363588380 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8fc004390 con 0x7fa90c151e80 2023-03-03T16:12:18.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa903fff700 1 -- 172.21.15.7:0/1363588380 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc0036a0 con 0x7fa90c151e80 2023-03-03T16:12:18.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 -- 172.21.15.7:0/1363588380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 msgr2=0x7fa90c0c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:18.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 --2- 172.21.15.7:0/1363588380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 0x7fa90c0c6aa0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009840 tx=0x7fa8fc009b50 comp rx=0 tx=0).stop 2023-03-03T16:12:18.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 -- 172.21.15.7:0/1363588380 shutdown_connections 2023-03-03T16:12:18.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 --2- 172.21.15.7:0/1363588380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 0x7fa90c0c6aa0 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:18.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 -- 172.21.15.7:0/1363588380 >> 172.21.15.7:0/1363588380 conn(0x7fa90c071630 msgr2=0x7fa90c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:12:18.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 -- 172.21.15.7:0/1363588380 shutdown_connections 2023-03-03T16:12:18.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.287+0000 7fa91365c700 1 -- 172.21.15.7:0/1363588380 wait complete. 2023-03-03T16:12:18.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.289+0000 7fa91365c700 1 Processor -- start 2023-03-03T16:12:18.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.289+0000 7fa91365c700 1 -- start start 2023-03-03T16:12:18.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.289+0000 7fa91365c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa90c0c6fe0 0x7fa90c1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:18.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.289+0000 7fa91365c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 0x7fa90c0c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:18.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.289+0000 7fa91365c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa90c155410 0x7fa90c0c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:18.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.289+0000 7fa91365c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c1561d0 con 0x7fa90c151e80 2023-03-03T16:12:18.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.289+0000 7fa91365c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c1586a0 con 0x7fa90c0c6fe0 2023-03-03T16:12:18.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.289+0000 7fa91365c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0c8700 con 0x7fa90c155410 2023-03-03T16:12:18.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa910bf7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 0x7fa90c0c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:18.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa910bf7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 0x7fa90c0c9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56186/0 (socket says 172.21.15.7:56186) 2023-03-03T16:12:18.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa910bf7700 1 -- 172.21.15.7:0/1358989702 learned_addr learned my addr 172.21.15.7:0/1358989702 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:12:18.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa9113f8700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa90c0c6fe0 0x7fa90c1580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:18.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa911bf9700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa90c155410 0x7fa90c0c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:18.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa910bf7700 1 -- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa90c155410 msgr2=0x7fa90c0c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:18.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa910bf7700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa90c155410 0x7fa90c0c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:18.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa910bf7700 1 -- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa90c0c6fe0 msgr2=0x7fa90c1580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:18.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa910bf7700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa90c0c6fe0 0x7fa90c1580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa910bf7700 1 -- 172.21.15.7:0/1358989702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc0094a0 con 0x7fa90c151e80 2023-03-03T16:12:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa9113f8700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa90c0c6fe0 0x7fa90c1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:12:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa910bf7700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 0x7fa90c0c9730 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fa90800b480 tx=0x7fa90800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:12:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa901ffb700 1 -- 172.21.15.7:0/1358989702 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908004020 con 0x7fa90c151e80 2023-03-03T16:12:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa901ffb700 1 -- 172.21.15.7:0/1358989702 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa908004180 con 0x7fa90c151e80 2023-03-03T16:12:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa91365c700 1 -- 172.21.15.7:0/1358989702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90c0c5220 con 0x7fa90c151e80 2023-03-03T16:12:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa901ffb700 1 -- 172.21.15.7:0/1358989702 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908004510 con 0x7fa90c151e80 2023-03-03T16:12:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.290+0000 7fa91365c700 1 -- 172.21.15.7:0/1358989702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90c154ef0 con 0x7fa90c151e80 2023-03-03T16:12:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.291+0000 7fa91365c700 1 -- 172.21.15.7:0/1358989702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90c0c8cc0 con 0x7fa90c151e80 2023-03-03T16:12:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.292+0000 7fa901ffb700 1 -- 172.21.15.7:0/1358989702 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa90800f050 con 0x7fa90c151e80 2023-03-03T16:12:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.292+0000 7fa901ffb700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8f00682f0 0x7fa8f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.292+0000 7fa901ffb700 1 -- 172.21.15.7:0/1358989702 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa908085b50 con 0x7fa90c151e80 2023-03-03T16:12:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.292+0000 7fa9113f8700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8f00682f0 0x7fa8f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:18.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.293+0000 7fa9113f8700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8f00682f0 0x7fa8f006a7a0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc000c00 tx=0x7fa8fc01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:12:18.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.296+0000 7fa901ffb700 1 -- 172.21.15.7:0/1358989702 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa908051ef0 con 0x7fa90c151e80 2023-03-03T16:12:18.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.534+0000 7fa91365c700 1 -- 172.21.15.7:0/1358989702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa90c0ca210 con 0x7fa90c151e80 2023-03-03T16:12:18.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.535+0000 7fa901ffb700 1 -- 172.21.15.7:0/1358989702 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa90c0ca210 con 0x7fa90c151e80 2023-03-03T16:12:18.536 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:12:18.536 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:12:18.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 -- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8f00682f0 msgr2=0x7fa8f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8f00682f0 0x7fa8f006a7a0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc000c00 tx=0x7fa8fc01c040 comp rx=0 tx=0).stop 2023-03-03T16:12:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 -- 172.21.15.7:0/1358989702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 msgr2=0x7fa90c0c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 0x7fa90c0c9730 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fa90800b480 tx=0x7fa90800b840 comp rx=0 tx=0).stop 2023-03-03T16:12:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 -- 172.21.15.7:0/1358989702 shutdown_connections 2023-03-03T16:12:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa90c155410 0x7fa90c0c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa90c0c6fe0 0x7fa90c1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:18.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa8f00682f0 0x7fa8f006a7a0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 --2- 172.21.15.7:0/1358989702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa90c151e80 0x7fa90c0c9730 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 -- 172.21.15.7:0/1358989702 >> 172.21.15.7:0/1358989702 conn(0x7fa90c071630 msgr2=0x7fa90c0c2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:12:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 -- 172.21.15.7:0/1358989702 shutdown_connections 2023-03-03T16:12:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:18.537+0000 7fa91365c700 1 -- 172.21.15.7:0/1358989702 wait complete. 2023-03-03T16:12:18.539 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:12:18.549 INFO:tasks.mds_thrash.fs.[b]:waiting for 100 secs before thrashing 2023-03-03T16:12:43.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:43.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:12:58.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:12:58.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.248+0000 7f963711e700 1 Processor -- start 2023-03-03T16:12:58.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.248+0000 7f963711e700 1 -- start start 2023-03-03T16:12:58.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f963711e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9630152e30 0x7f96300c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:58.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f963711e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f96300c4910 0x7f9630156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:58.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f963711e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96301523b0 con 0x7f96300beab0 2023-03-03T16:12:58.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f963711e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9630072010 con 0x7f96300c4910 2023-03-03T16:12:58.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f963711e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96300aaaf0 con 0x7f9630152e30 2023-03-03T16:12:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f96300beab0 0x7f9630155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50206/0 (socket says 172.21.15.7:50206) 2023-03-03T16:12:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962ffff700 1 -- 172.21.15.7:0/2068088288 learned_addr learned my addr 172.21.15.7:0/2068088288 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:12:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f9634eba700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9630152e30 0x7f96300c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f96356bb700 1 --2- 172.21.15.7:0/2068088288 >> v2:172.21.15.143:3300/0 conn(0x7f96300c4910 0x7f9630156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39738291 0 0) 0x7f96301523b0 con 0x7f96300beab0 2023-03-03T16:12:58.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96200035b0 con 0x7f96300beab0 2023-03-03T16:12:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1882057639 0 0) 0x7f96200035b0 con 0x7f96300beab0 2023-03-03T16:12:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9630152e30 msgr2=0x7f96300c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:12:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962f7fe700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9630152e30 0x7f96300c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 >> v2:172.21.15.143:3300/0 conn(0x7f96300c4910 msgr2=0x7f9630156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:12:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962f7fe700 1 --2- 172.21.15.7:0/2068088288 >> v2:172.21.15.143:3300/0 conn(0x7f96300c4910 0x7f9630156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96300aadd0 con 0x7f96300beab0 2023-03-03T16:12:58.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.249+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f961c001670 con 0x7f96300beab0 2023-03-03T16:12:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 >> v1:172.21.15.7:6789/0 conn(0x7f96300beab0 legacy=0x7f9630155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962f7fe700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9620003a70 0x7f9620003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962f7fe700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9630152e30 0x7f9620004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962f7fe700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 0x7f9620005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9630072010 con 0x7f9620004d90 2023-03-03T16:12:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300aaaf0 con 0x7f9620003a70 2023-03-03T16:12:58.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962000ba60 con 0x7f9630152e30 2023-03-03T16:12:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f96356bb700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 0x7f9620005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f9634eba700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9620003a70 0x7f9620003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962ffff700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9630152e30 0x7f9620004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f96356bb700 1 -- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9630152e30 msgr2=0x7f9620004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:12:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f96356bb700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9630152e30 0x7f9620004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f96356bb700 1 -- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9620003a70 msgr2=0x7f9620003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:12:58.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f96356bb700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9620003a70 0x7f9620003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f96356bb700 1 -- 172.21.15.7:0/2068088288 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96200035b0 con 0x7f9620004d90 2023-03-03T16:12:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f96356bb700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 0x7f9620005200 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f962800e170 tx=0x7f9620005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=939af91378435d16 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:12:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962800ec90 con 0x7f9620004d90 2023-03-03T16:12:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962800f040 con 0x7f9620004d90 2023-03-03T16:12:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.250+0000 7f962f7fe700 1 -- 172.21.15.7:0/2068088288 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9628014480 con 0x7f9620004d90 2023-03-03T16:12:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.251+0000 7f963711e700 1 -- 172.21.15.7:0/2068088288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 msgr2=0x7f9620005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:58.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.251+0000 7f963711e700 1 --2- 172.21.15.7:0/2068088288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 0x7f9620005200 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f962800e170 tx=0x7f9620005880 comp rx=0 tx=0).stop 2023-03-03T16:12:58.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.251+0000 7f96356bb700 1 -- 172.21.15.7:0/2068088288 reap_dead start 2023-03-03T16:12:58.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.251+0000 7f963711e700 1 -- 172.21.15.7:0/2068088288 shutdown_connections 2023-03-03T16:12:58.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.251+0000 7f963711e700 1 -- 172.21.15.7:0/2068088288 >> 172.21.15.7:0/2068088288 conn(0x7f96300736f0 msgr2=0x7f9630073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:12:58.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.251+0000 7f963711e700 1 -- 172.21.15.7:0/2068088288 shutdown_connections 2023-03-03T16:12:58.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.251+0000 7f963711e700 1 -- 172.21.15.7:0/2068088288 wait complete. 2023-03-03T16:12:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.252+0000 7f963711e700 1 Processor -- start 2023-03-03T16:12:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.252+0000 7f963711e700 1 -- start start 2023-03-03T16:12:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f963711e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 0x7f9630151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f963711e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9630155750 0x7f963014b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f963711e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96300c2480 0x7f963014b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f963711e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300bf600 con 0x7f9620004d90 2023-03-03T16:12:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f963711e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300c5770 con 0x7f9630155750 2023-03-03T16:12:58.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f963711e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9630153830 con 0x7f96300c2480 2023-03-03T16:12:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f9634eba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 0x7f9630151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f9634eba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 0x7f9630151250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56236/0 (socket says 172.21.15.7:56236) 2023-03-03T16:12:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f9634eba700 1 -- 172.21.15.7:0/995318458 learned_addr learned my addr 172.21.15.7:0/995318458 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:12:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f962ffff700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9630155750 0x7f963014b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f96356bb700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96300c2480 0x7f963014b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f9634eba700 1 -- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96300c2480 msgr2=0x7f963014b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f9634eba700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96300c2480 0x7f963014b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f9634eba700 1 -- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9630155750 msgr2=0x7f963014b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.253+0000 7f9634eba700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9630155750 0x7f963014b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.254+0000 7f9634eba700 1 -- 172.21.15.7:0/995318458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9630153200 con 0x7f9620004d90 2023-03-03T16:12:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.254+0000 7f9634eba700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 0x7f9630151250 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f96240119a0 tx=0x7f9624011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:12:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.254+0000 7f962d7fa700 1 -- 172.21.15.7:0/995318458 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962400f940 con 0x7f9620004d90 2023-03-03T16:12:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.254+0000 7f962d7fa700 1 -- 172.21.15.7:0/995318458 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962400faa0 con 0x7f9620004d90 2023-03-03T16:12:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.254+0000 7f963711e700 1 -- 172.21.15.7:0/995318458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9630071c70 con 0x7f9620004d90 2023-03-03T16:12:58.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.254+0000 7f962d7fa700 1 -- 172.21.15.7:0/995318458 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962400fe30 con 0x7f9620004d90 2023-03-03T16:12:58.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.254+0000 7f963711e700 1 -- 172.21.15.7:0/995318458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9630153390 con 0x7f9620004d90 2023-03-03T16:12:58.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.255+0000 7f963711e700 1 -- 172.21.15.7:0/995318458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9630074bb0 con 0x7f9620004d90 2023-03-03T16:12:58.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.255+0000 7f962d7fa700 1 -- 172.21.15.7:0/995318458 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9624014440 con 0x7f9620004d90 2023-03-03T16:12:58.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.256+0000 7f962d7fa700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9614068320 0x7f961406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:12:58.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.256+0000 7f962d7fa700 1 -- 172.21.15.7:0/995318458 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9624086980 con 0x7f9620004d90 2023-03-03T16:12:58.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.256+0000 7f962ffff700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9614068320 0x7f961406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:12:58.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.256+0000 7f962ffff700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9614068320 0x7f961406a7d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f96300c5310 tx=0x7f961c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:12:58.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.260+0000 7f962d7fa700 1 -- 172.21.15.7:0/995318458 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9624052cc0 con 0x7f9620004d90 2023-03-03T16:12:58.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.496+0000 7f963711e700 1 -- 172.21.15.7:0/995318458 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96300bf740 con 0x7f9620004d90 2023-03-03T16:12:58.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.496+0000 7f962d7fa700 1 -- 172.21.15.7:0/995318458 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f96240562e0 con 0x7f9620004d90 2023-03-03T16:12:58.497 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:12:58.497 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:12:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 -- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9614068320 msgr2=0x7f961406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9614068320 0x7f961406a7d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f96300c5310 tx=0x7f961c009040 comp rx=0 tx=0).stop 2023-03-03T16:12:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 -- 172.21.15.7:0/995318458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 msgr2=0x7f9630151250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:12:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 0x7f9630151250 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f96240119a0 tx=0x7f9624011d60 comp rx=0 tx=0).stop 2023-03-03T16:12:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 -- 172.21.15.7:0/995318458 shutdown_connections 2023-03-03T16:12:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96300c2480 0x7f963014b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9614068320 0x7f961406a7d0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9620004d90 0x7f9630151250 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 --2- 172.21.15.7:0/995318458 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9630155750 0x7f963014b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:12:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 -- 172.21.15.7:0/995318458 >> 172.21.15.7:0/995318458 conn(0x7f96300736f0 msgr2=0x7f963014f340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:12:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 -- 172.21.15.7:0/995318458 shutdown_connections 2023-03-03T16:12:58.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:12:58.498+0000 7f963711e700 1 -- 172.21.15.7:0/995318458 wait complete. 2023-03-03T16:12:58.501 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:12:58.511 INFO:tasks.mds_thrash.fs.[a]:waiting for 77 secs before thrashing 2023-03-03T16:13:13.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:13.304 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:43.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:43.377 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:13:58.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:13:58.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.689+0000 7fc5c1c6a700 1 Processor -- start 2023-03-03T16:13:58.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.689+0000 7fc5c1c6a700 1 -- start start 2023-03-03T16:13:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5c1c6a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5bc14f9d0 0x7fc5bc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5c1c6a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc5bc14fda0 0x7fc5bc071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5c1c6a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc0c2f30 con 0x7fc5bc0abe20 2023-03-03T16:13:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5c1c6a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc0c31d0 con 0x7fc5bc14fda0 2023-03-03T16:13:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5c1c6a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc1524a0 con 0x7fc5bc14f9d0 2023-03-03T16:13:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5baffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc5bc14fda0 0x7fc5bc071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5bb7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5bc14f9d0 0x7fc5bc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5bbfff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fc5bc0abe20 0x7fc5bc1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:53784/0 (socket says 172.21.15.7:53784) 2023-03-03T16:13:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5bbfff700 1 -- 172.21.15.7:0/3769620022 learned_addr learned my addr 172.21.15.7:0/3769620022 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:13:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5baffd700 1 -- 172.21.15.7:0/3769620022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5bc14f9d0 msgr2=0x7fc5bc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:13:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5baffd700 1 --2- 172.21.15.7:0/3769620022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5bc14f9d0 0x7fc5bc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5baffd700 1 -- 172.21.15.7:0/3769620022 >> v1:172.21.15.143:6789/0 conn(0x7fc5bc0abe20 legacy=0x7fc5bc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5ba7fc700 1 -- 172.21.15.7:0/3769620022 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879664969 0 0) 0x7fc5bc0c2f30 con 0x7fc5bc0abe20 2023-03-03T16:13:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.690+0000 7fc5baffd700 1 -- 172.21.15.7:0/3769620022 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc0c34b0 con 0x7fc5bc14fda0 2023-03-03T16:13:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5baffd700 1 --2- 172.21.15.7:0/3769620022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5bc14fda0 0x7fc5bc071890 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009840 tx=0x7fc5b0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be4a34defa86c66 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:13:58.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5ba7fc700 1 -- 172.21.15.7:0/3769620022 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0011830 con 0x7fc5bc14fda0 2023-03-03T16:13:58.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5ba7fc700 1 -- 172.21.15.7:0/3769620022 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b0011990 con 0x7fc5bc14fda0 2023-03-03T16:13:58.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5ba7fc700 1 -- 172.21.15.7:0/3769620022 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0011d20 con 0x7fc5bc14fda0 2023-03-03T16:13:58.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/3769620022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5bc14fda0 msgr2=0x7fc5bc071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:58.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5c1c6a700 1 --2- 172.21.15.7:0/3769620022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5bc14fda0 0x7fc5bc071890 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009840 tx=0x7fc5b0009b50 comp rx=0 tx=0).stop 2023-03-03T16:13:58.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/3769620022 shutdown_connections 2023-03-03T16:13:58.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5c1c6a700 1 --2- 172.21.15.7:0/3769620022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5bc14f9d0 0x7fc5bc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:58.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5c1c6a700 1 --2- 172.21.15.7:0/3769620022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5bc14fda0 0x7fc5bc071890 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:58.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/3769620022 >> 172.21.15.7:0/3769620022 conn(0x7fc5bc0c8900 msgr2=0x7fc5bc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:13:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/3769620022 shutdown_connections 2023-03-03T16:13:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.691+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/3769620022 wait complete. 2023-03-03T16:13:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.693+0000 7fc5c1c6a700 1 Processor -- start 2023-03-03T16:13:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.693+0000 7fc5c1c6a700 1 -- start start 2023-03-03T16:13:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5c1c6a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5bc0abe20 0x7fc5bc1ea2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5c1c6a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5bc14f9d0 0x7fc5bc0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5c1c6a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5bc14fda0 0x7fc5bc143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5c1c6a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc06a3e0 con 0x7fc5bc14fda0 2023-03-03T16:13:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5c1c6a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc1ea7f0 con 0x7fc5bc0abe20 2023-03-03T16:13:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5c1c6a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc06c060 con 0x7fc5bc14f9d0 2023-03-03T16:13:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5bc14fda0 0x7fc5bc143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5bc14fda0 0x7fc5bc143b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56290/0 (socket says 172.21.15.7:56290) 2023-03-03T16:13:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5bbfff700 1 -- 172.21.15.7:0/448597258 learned_addr learned my addr 172.21.15.7:0/448597258 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:13:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5baffd700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5bc14f9d0 0x7fc5bc0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5bb7fe700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5bc0abe20 0x7fc5bc1ea2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5bbfff700 1 -- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5bc14f9d0 msgr2=0x7fc5bc0c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5bbfff700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5bc14f9d0 0x7fc5bc0c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5bbfff700 1 -- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5bc0abe20 msgr2=0x7fc5bc1ea2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5bbfff700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5bc0abe20 0x7fc5bc1ea2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5bbfff700 1 -- 172.21.15.7:0/448597258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b00094a0 con 0x7fc5bc14fda0 2023-03-03T16:13:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5bbfff700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5bc14fda0 0x7fc5bc143b40 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400cb60 tx=0x7fc5b400ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:13:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.694+0000 7fc5a3fff700 1 -- 172.21.15.7:0/448597258 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400a9e0 con 0x7fc5bc14fda0 2023-03-03T16:13:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.695+0000 7fc5a3fff700 1 -- 172.21.15.7:0/448597258 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b400ab40 con 0x7fc5bc14fda0 2023-03-03T16:13:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.695+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/448597258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc0ac350 con 0x7fc5bc14fda0 2023-03-03T16:13:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.695+0000 7fc5a3fff700 1 -- 172.21.15.7:0/448597258 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b40070d0 con 0x7fc5bc14fda0 2023-03-03T16:13:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.695+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/448597258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc0ab630 con 0x7fc5bc14fda0 2023-03-03T16:13:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.696+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/448597258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc0c3e10 con 0x7fc5bc14fda0 2023-03-03T16:13:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.696+0000 7fc5a3fff700 1 -- 172.21.15.7:0/448597258 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc5b401a030 con 0x7fc5bc14fda0 2023-03-03T16:13:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.697+0000 7fc5a3fff700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5a80682f0 0x7fc5a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:13:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.697+0000 7fc5a3fff700 1 -- 172.21.15.7:0/448597258 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc5b4086000 con 0x7fc5bc14fda0 2023-03-03T16:13:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.697+0000 7fc5bb7fe700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5a80682f0 0x7fc5a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:13:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.697+0000 7fc5bb7fe700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5a80682f0 0x7fc5a806a7a0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc06f790 tx=0x7fc5ac009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:13:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.701+0000 7fc5a3fff700 1 -- 172.21.15.7:0/448597258 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc5b4052440 con 0x7fc5bc14fda0 2023-03-03T16:13:58.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.936+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/448597258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5bc074bb0 con 0x7fc5bc14fda0 2023-03-03T16:13:58.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.936+0000 7fc5a3fff700 1 -- 172.21.15.7:0/448597258 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc5b4018020 con 0x7fc5bc14fda0 2023-03-03T16:13:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:13:58.937 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:13:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.938+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5a80682f0 msgr2=0x7fc5a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.938+0000 7fc5c1c6a700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5a80682f0 0x7fc5a806a7a0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc06f790 tx=0x7fc5ac009210 comp rx=0 tx=0).stop 2023-03-03T16:13:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.938+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/448597258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5bc14fda0 msgr2=0x7fc5bc143b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:13:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.938+0000 7fc5c1c6a700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5bc14fda0 0x7fc5bc143b40 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400cb60 tx=0x7fc5b400ce70 comp rx=0 tx=0).stop 2023-03-03T16:13:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.938+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/448597258 shutdown_connections 2023-03-03T16:13:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.938+0000 7fc5c1c6a700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5a80682f0 0x7fc5a806a7a0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.938+0000 7fc5c1c6a700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5bc14fda0 0x7fc5bc143b40 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.938+0000 7fc5c1c6a700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5bc0abe20 0x7fc5bc1ea2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.938+0000 7fc5c1c6a700 1 --2- 172.21.15.7:0/448597258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc5bc14f9d0 0x7fc5bc0c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:13:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.938+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/448597258 >> 172.21.15.7:0/448597258 conn(0x7fc5bc0c8900 msgr2=0x7fc5bc1516e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:13:58.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.939+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/448597258 shutdown_connections 2023-03-03T16:13:58.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:13:58.939+0000 7fc5c1c6a700 1 -- 172.21.15.7:0/448597258 wait complete. 2023-03-03T16:13:58.941 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:13:58.951 INFO:tasks.mds_thrash.fs.[b]:waiting for 48 secs before thrashing 2023-03-03T16:14:13.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:13.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:15.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:14:15.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.652+0000 7f97ac2ec700 1 Processor -- start 2023-03-03T16:14:15.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.652+0000 7f97ac2ec700 1 -- start start 2023-03-03T16:14:15.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.653+0000 7f97ac2ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a4071c30 0x7f97a40c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.653+0000 7f97ac2ec700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a40713c0 con 0x7f97a40aac00 2023-03-03T16:14:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.653+0000 7f97ac2ec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a4152c70 con 0x7f97a4155950 2023-03-03T16:14:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.653+0000 7f97ac2ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a40c4930 con 0x7f97a4071c30 2023-03-03T16:14:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.653+0000 7f97aa088700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f97a40aac00 0x7f97a40beb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50290/0 (socket says 172.21.15.7:50290) 2023-03-03T16:14:15.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.653+0000 7f97aa088700 1 -- 172.21.15.7:0/1052119564 learned_addr learned my addr 172.21.15.7:0/1052119564 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:14:15.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9887700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a4071c30 0x7f97a40c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623699931 0 0) 0x7f97a40713c0 con 0x7f97a40aac00 2023-03-03T16:14:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97900035b0 con 0x7f97a40aac00 2023-03-03T16:14:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 699881417 0 0) 0x7f97900035b0 con 0x7f97a40aac00 2023-03-03T16:14:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a4071c30 msgr2=0x7f97a40c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:14:15.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a4071c30 0x7f97a40c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 >> v1:172.21.15.143:6789/0 conn(0x7f97a4155950 legacy=0x7f97a415cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a4152360 con 0x7f97a40aac00 2023-03-03T16:14:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f9794001670 con 0x7f97a40aac00 2023-03-03T16:14:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 >> v1:172.21.15.7:6789/0 conn(0x7f97a40aac00 legacy=0x7f97a40beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9790003aa0 0x7f9790003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a4071c30 0x7f9790004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:15.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 0x7f9790005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:15.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a40c4930 con 0x7f9790004d90 2023-03-03T16:14:15.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979000bba0 con 0x7f9790003aa0 2023-03-03T16:14:15.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979000bd60 con 0x7f97a4071c30 2023-03-03T16:14:15.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97aa889700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 0x7f9790005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:15.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97a9887700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a4071c30 0x7f9790004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:15.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97aa088700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9790003aa0 0x7f9790003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:15.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97aa889700 1 -- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a4071c30 msgr2=0x7f9790004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:14:15.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97aa889700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a4071c30 0x7f9790004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:15.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97aa889700 1 -- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9790003aa0 msgr2=0x7f9790003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:14:15.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97aa889700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9790003aa0 0x7f9790003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:15.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.654+0000 7f97aa889700 1 -- 172.21.15.7:0/1052119564 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97900035b0 con 0x7f9790004d90 2023-03-03T16:14:15.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97aa889700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 0x7f9790005200 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f979c00c290 tx=0x7f9790005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c251e9061941a4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:14:15.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c001e70 con 0x7f9790004d90 2023-03-03T16:14:15.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979c00ce50 con 0x7f9790004d90 2023-03-03T16:14:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97a9086700 1 -- 172.21.15.7:0/1052119564 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c012660 con 0x7f9790004d90 2023-03-03T16:14:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1052119564 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 msgr2=0x7f9790005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97ac2ec700 1 --2- 172.21.15.7:0/1052119564 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 0x7f9790005200 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f979c00c290 tx=0x7f9790005880 comp rx=0 tx=0).stop 2023-03-03T16:14:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97aa889700 1 -- 172.21.15.7:0/1052119564 reap_dead start 2023-03-03T16:14:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1052119564 shutdown_connections 2023-03-03T16:14:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1052119564 >> 172.21.15.7:0/1052119564 conn(0x7f97a40736f0 msgr2=0x7f97a4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:14:15.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1052119564 shutdown_connections 2023-03-03T16:14:15.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.655+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1052119564 wait complete. 2023-03-03T16:14:15.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.657+0000 7f97ac2ec700 1 Processor -- start 2023-03-03T16:14:15.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.657+0000 7f97ac2ec700 1 -- start start 2023-03-03T16:14:15.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.657+0000 7f97ac2ec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 0x7f97a41dbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:15.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.657+0000 7f97ac2ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a414a680 0x7f97a41dc950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:15.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.657+0000 7f97ac2ec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a413fc20 0x7f97a413e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:15.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.657+0000 7f97ac2ec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a40c8e50 con 0x7f9790004d90 2023-03-03T16:14:15.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.657+0000 7f97ac2ec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a40c90f0 con 0x7f97a413fc20 2023-03-03T16:14:15.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.657+0000 7f97ac2ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a4142670 con 0x7f97a414a680 2023-03-03T16:14:15.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97aa088700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 0x7f97a41dbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:15.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97aa088700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 0x7f97a41dbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56322/0 (socket says 172.21.15.7:56322) 2023-03-03T16:14:15.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97aa088700 1 -- 172.21.15.7:0/1739902653 learned_addr learned my addr 172.21.15.7:0/1739902653 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:14:15.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97a9887700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a414a680 0x7f97a41dc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:15.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97aa889700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a413fc20 0x7f97a413e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:15.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97aa088700 1 -- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a414a680 msgr2=0x7f97a41dc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:15.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97aa088700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a414a680 0x7f97a41dc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:15.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97aa088700 1 -- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a413fc20 msgr2=0x7f97a413e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:15.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97aa088700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a413fc20 0x7f97a413e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:15.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97aa088700 1 -- 172.21.15.7:0/1739902653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a4152620 con 0x7f9790004d90 2023-03-03T16:14:15.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97aa088700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 0x7f97a41dbd80 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f97940017a0 tx=0x7f9794007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:14:15.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f979affd700 1 -- 172.21.15.7:0/1739902653 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979400f6c0 con 0x7f9790004d90 2023-03-03T16:14:15.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f979affd700 1 -- 172.21.15.7:0/1739902653 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979400f860 con 0x7f9790004d90 2023-03-03T16:14:15.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f979affd700 1 -- 172.21.15.7:0/1739902653 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979400fc10 con 0x7f9790004d90 2023-03-03T16:14:15.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1739902653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a40c4b80 con 0x7f9790004d90 2023-03-03T16:14:15.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.658+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1739902653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a4152900 con 0x7f9790004d90 2023-03-03T16:14:15.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.659+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1739902653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a41dbbd0 con 0x7f9790004d90 2023-03-03T16:14:15.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.660+0000 7f979affd700 1 -- 172.21.15.7:0/1739902653 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f979400fd70 con 0x7f9790004d90 2023-03-03T16:14:15.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.660+0000 7f979affd700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97880682f0 0x7f978806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:15.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.660+0000 7f979affd700 1 -- 172.21.15.7:0/1739902653 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9794086260 con 0x7f9790004d90 2023-03-03T16:14:15.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.660+0000 7f97a9887700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97880682f0 0x7f978806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:15.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.661+0000 7f97a9887700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97880682f0 0x7f978806a7a0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f97a40c9500 tx=0x7f97a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:14:15.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.664+0000 7f979affd700 1 -- 172.21.15.7:0/1739902653 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f97940526c0 con 0x7f9790004d90 2023-03-03T16:14:15.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.899+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1739902653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97a4074bb0 con 0x7f9790004d90 2023-03-03T16:14:15.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.900+0000 7f979affd700 1 -- 172.21.15.7:0/1739902653 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9794055ce0 con 0x7f9790004d90 2023-03-03T16:14:15.900 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:14:15.900 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:14:15.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97880682f0 msgr2=0x7f978806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:15.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97880682f0 0x7f978806a7a0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f97a40c9500 tx=0x7f97a0008040 comp rx=0 tx=0).stop 2023-03-03T16:14:15.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1739902653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 msgr2=0x7f97a41dbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:15.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 0x7f97a41dbd80 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f97940017a0 tx=0x7f9794007620 comp rx=0 tx=0).stop 2023-03-03T16:14:15.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1739902653 shutdown_connections 2023-03-03T16:14:15.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f97a413fc20 0x7f97a413e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:15.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f97880682f0 0x7f978806a7a0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:15.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9790004d90 0x7f97a41dbd80 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:15.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 --2- 172.21.15.7:0/1739902653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f97a414a680 0x7f97a41dc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:15.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1739902653 >> 172.21.15.7:0/1739902653 conn(0x7f97a40736f0 msgr2=0x7f97a4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:14:15.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1739902653 shutdown_connections 2023-03-03T16:14:15.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:15.902+0000 7f97ac2ec700 1 -- 172.21.15.7:0/1739902653 wait complete. 2023-03-03T16:14:15.904 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:14:15.914 INFO:tasks.mds_thrash.fs.[a]:waiting for 51 secs before thrashing 2023-03-03T16:14:43.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:43.494 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:14:46.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:14:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.090+0000 7f7bfc56c700 1 Processor -- start 2023-03-03T16:14:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.090+0000 7f7bfc56c700 1 -- start start 2023-03-03T16:14:47.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.091+0000 7f7bfc56c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7bf40beab0 0x7f7bf4155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:47.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.091+0000 7f7bfc56c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf41523b0 con 0x7f7bf4152e30 2023-03-03T16:14:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.091+0000 7f7bfc56c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf4072010 con 0x7f7bf41562c0 2023-03-03T16:14:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bfc56c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf40aaaf0 con 0x7f7bf40beab0 2023-03-03T16:14:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bf9b07700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7bf40beab0 0x7f7bf4155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bfa308700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7bf4152e30 0x7f7bf40c43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50336/0 (socket says 172.21.15.7:50336) 2023-03-03T16:14:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bfa308700 1 -- 172.21.15.7:0/1761568931 learned_addr learned my addr 172.21.15.7:0/1761568931 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:14:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bf9b07700 1 -- 172.21.15.7:0/1761568931 >> v1:172.21.15.143:6789/0 conn(0x7f7bf41562c0 legacy=0x7f7bf415cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:14:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bf9b07700 1 -- 172.21.15.7:0/1761568931 >> v1:172.21.15.7:6789/0 conn(0x7f7bf4152e30 legacy=0x7f7bf40c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bf9b07700 1 -- 172.21.15.7:0/1761568931 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf40aadd0 con 0x7f7bf40beab0 2023-03-03T16:14:47.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bf9306700 1 -- 172.21.15.7:0/1761568931 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392294351 0 0) 0x7f7bf41523b0 con 0x7f7bf4152e30 2023-03-03T16:14:47.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bf9b07700 1 --2- 172.21.15.7:0/1761568931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf40beab0 0x7f7bf4155d80 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009840 tx=0x7f7bf0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba721af8c8cc036a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:14:47.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bf9306700 1 -- 172.21.15.7:0/1761568931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0004230 con 0x7f7bf40beab0 2023-03-03T16:14:47.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bf9306700 1 -- 172.21.15.7:0/1761568931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf0004390 con 0x7f7bf40beab0 2023-03-03T16:14:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.092+0000 7f7bf9306700 1 -- 172.21.15.7:0/1761568931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00036a0 con 0x7f7bf40beab0 2023-03-03T16:14:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.093+0000 7f7bfc56c700 1 -- 172.21.15.7:0/1761568931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf40beab0 msgr2=0x7f7bf4155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.093+0000 7f7bfc56c700 1 --2- 172.21.15.7:0/1761568931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf40beab0 0x7f7bf4155d80 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009840 tx=0x7f7bf0009b50 comp rx=0 tx=0).stop 2023-03-03T16:14:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.093+0000 7f7bfc56c700 1 -- 172.21.15.7:0/1761568931 shutdown_connections 2023-03-03T16:14:47.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.093+0000 7f7bfc56c700 1 --2- 172.21.15.7:0/1761568931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf40beab0 0x7f7bf4155d80 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.093+0000 7f7bfc56c700 1 -- 172.21.15.7:0/1761568931 >> 172.21.15.7:0/1761568931 conn(0x7f7bf40736f0 msgr2=0x7f7bf4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:14:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.093+0000 7f7bfc56c700 1 -- 172.21.15.7:0/1761568931 shutdown_connections 2023-03-03T16:14:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.093+0000 7f7bfc56c700 1 -- 172.21.15.7:0/1761568931 wait complete. 2023-03-03T16:14:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.094+0000 7f7bfc56c700 1 Processor -- start 2023-03-03T16:14:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.094+0000 7f7bfc56c700 1 -- start start 2023-03-03T16:14:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfc56c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf40beab0 0x7f7bf41513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:47.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfc56c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf4152e30 0x7f7bf414d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfc56c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf41562c0 0x7f7bf414d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfc56c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf414de90 con 0x7f7bf41562c0 2023-03-03T16:14:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfc56c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4154ba0 con 0x7f7bf40beab0 2023-03-03T16:14:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfc56c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4154580 con 0x7f7bf4152e30 2023-03-03T16:14:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfab09700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf41562c0 0x7f7bf414d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfab09700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf41562c0 0x7f7bf414d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56366/0 (socket says 172.21.15.7:56366) 2023-03-03T16:14:47.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfab09700 1 -- 172.21.15.7:0/2407247650 learned_addr learned my addr 172.21.15.7:0/2407247650 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:14:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bf9b07700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf4152e30 0x7f7bf414d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfa308700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf40beab0 0x7f7bf41513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfab09700 1 -- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf4152e30 msgr2=0x7f7bf414d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfab09700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf4152e30 0x7f7bf414d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfab09700 1 -- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf40beab0 msgr2=0x7f7bf41513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfab09700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf40beab0 0x7f7bf41513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:47.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.095+0000 7f7bfab09700 1 -- 172.21.15.7:0/2407247650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf00094a0 con 0x7f7bf41562c0 2023-03-03T16:14:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.096+0000 7f7bfab09700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf41562c0 0x7f7bf414d950 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00cc10 tx=0x7f7bec00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:14:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.096+0000 7f7beaffd700 1 -- 172.21.15.7:0/2407247650 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec00abb0 con 0x7f7bf41562c0 2023-03-03T16:14:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.096+0000 7f7beaffd700 1 -- 172.21.15.7:0/2407247650 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec00ad10 con 0x7f7bf41562c0 2023-03-03T16:14:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.096+0000 7f7bfc56c700 1 -- 172.21.15.7:0/2407247650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf4153200 con 0x7f7bf41562c0 2023-03-03T16:14:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.096+0000 7f7beaffd700 1 -- 172.21.15.7:0/2407247650 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec012660 con 0x7f7bf41562c0 2023-03-03T16:14:47.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.096+0000 7f7bfc56c700 1 -- 172.21.15.7:0/2407247650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf4071c70 con 0x7f7bf41562c0 2023-03-03T16:14:47.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.097+0000 7f7bfc56c700 1 -- 172.21.15.7:0/2407247650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf4153390 con 0x7f7bf41562c0 2023-03-03T16:14:47.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.097+0000 7f7beaffd700 1 -- 172.21.15.7:0/2407247650 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7bec022070 con 0x7f7bf41562c0 2023-03-03T16:14:47.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.098+0000 7f7beaffd700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bd80682f0 0x7f7bd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:14:47.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.098+0000 7f7beaffd700 1 -- 172.21.15.7:0/2407247650 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7bec086880 con 0x7f7bf41562c0 2023-03-03T16:14:47.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.098+0000 7f7bfa308700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bd80682f0 0x7f7bd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:14:47.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.098+0000 7f7bfa308700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bd80682f0 0x7f7bd806a7a0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7be4000f60 tx=0x7f7be4007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:14:47.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.102+0000 7f7beaffd700 1 -- 172.21.15.7:0/2407247650 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7bec052b30 con 0x7f7bf41562c0 2023-03-03T16:14:47.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.343+0000 7f7bfc56c700 1 -- 172.21.15.7:0/2407247650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bf4074bb0 con 0x7f7bf41562c0 2023-03-03T16:14:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.343+0000 7f7beaffd700 1 -- 172.21.15.7:0/2407247650 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7bec016090 con 0x7f7bf41562c0 2023-03-03T16:14:47.344 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:14:47.344 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:14:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 -- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bd80682f0 msgr2=0x7f7bd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bd80682f0 0x7f7bd806a7a0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7be4000f60 tx=0x7f7be4007be0 comp rx=0 tx=0).stop 2023-03-03T16:14:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 -- 172.21.15.7:0/2407247650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf41562c0 msgr2=0x7f7bf414d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:14:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf41562c0 0x7f7bf414d950 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00cc10 tx=0x7f7bec00a060 comp rx=0 tx=0).stop 2023-03-03T16:14:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 -- 172.21.15.7:0/2407247650 shutdown_connections 2023-03-03T16:14:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bd80682f0 0x7f7bd806a7a0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7bf41562c0 0x7f7bf414d950 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7bf40beab0 0x7f7bf41513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 --2- 172.21.15.7:0/2407247650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7bf4152e30 0x7f7bf414d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:14:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 -- 172.21.15.7:0/2407247650 >> 172.21.15.7:0/2407247650 conn(0x7f7bf40736f0 msgr2=0x7f7bf40aa190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:14:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 -- 172.21.15.7:0/2407247650 shutdown_connections 2023-03-03T16:14:47.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:14:47.345+0000 7f7bfc56c700 1 -- 172.21.15.7:0/2407247650 wait complete. 2023-03-03T16:14:47.348 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:14:47.358 INFO:tasks.mds_thrash.fs.[b]:waiting for 105 secs before thrashing 2023-03-03T16:15:06.915 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:15:07.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.053+0000 7f7cc3b45700 1 Processor -- start 2023-03-03T16:15:07.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.053+0000 7f7cc3b45700 1 -- start start 2023-03-03T16:15:07.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.054+0000 7f7cc3b45700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7cbc071e80 0x7f7cbc151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:07.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.054+0000 7f7cc3b45700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc1510d0 con 0x7f7cbc072250 2023-03-03T16:15:07.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.054+0000 7f7cc3b45700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc0c3740 con 0x7f7cbc14e900 2023-03-03T16:15:07.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.054+0000 7f7cc3b45700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cbc070670 con 0x7f7cbc071e80 2023-03-03T16:15:07.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.054+0000 7f7cc10e0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7cbc072250 0x7f7cbc14e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50360/0 (socket says 172.21.15.7:50360) 2023-03-03T16:15:07.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.054+0000 7f7cc18e1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7cbc071e80 0x7f7cbc151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:07.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.054+0000 7f7cc10e0700 1 -- 172.21.15.7:0/3563137277 learned_addr learned my addr 172.21.15.7:0/3563137277 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:15:07.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.054+0000 7f7cc18e1700 1 -- 172.21.15.7:0/3563137277 >> v1:172.21.15.143:6789/0 conn(0x7f7cbc14e900 legacy=0x7f7cbc15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:07.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.054+0000 7f7cc18e1700 1 -- 172.21.15.7:0/3563137277 >> v1:172.21.15.7:6789/0 conn(0x7f7cbc072250 legacy=0x7f7cbc14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:07.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.054+0000 7f7cc18e1700 1 -- 172.21.15.7:0/3563137277 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc0c39e0 con 0x7f7cbc071e80 2023-03-03T16:15:07.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc18e1700 1 --2- 172.21.15.7:0/3563137277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 0x7f7cbc151820 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009840 tx=0x7f7cac009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c0ae580cecc9cc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:15:07.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc08df700 1 -- 172.21.15.7:0/3563137277 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac011830 con 0x7f7cbc071e80 2023-03-03T16:15:07.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc08df700 1 -- 172.21.15.7:0/3563137277 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac011990 con 0x7f7cbc071e80 2023-03-03T16:15:07.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc08df700 1 -- 172.21.15.7:0/3563137277 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac011d20 con 0x7f7cbc071e80 2023-03-03T16:15:07.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3563137277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 msgr2=0x7f7cbc151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:07.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc3b45700 1 --2- 172.21.15.7:0/3563137277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 0x7f7cbc151820 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009840 tx=0x7f7cac009b50 comp rx=0 tx=0).stop 2023-03-03T16:15:07.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3563137277 shutdown_connections 2023-03-03T16:15:07.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc3b45700 1 --2- 172.21.15.7:0/3563137277 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 0x7f7cbc151820 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:07.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3563137277 >> 172.21.15.7:0/3563137277 conn(0x7f7cbc0a94d0 msgr2=0x7f7cbc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:15:07.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3563137277 shutdown_connections 2023-03-03T16:15:07.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.055+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3563137277 wait complete. 2023-03-03T16:15:07.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc3b45700 1 Processor -- start 2023-03-03T16:15:07.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc3b45700 1 -- start start 2023-03-03T16:15:07.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc3b45700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 0x7f7cbc0c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc3b45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cbc072250 0x7f7cbc148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc3b45700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cbc14e900 0x7f7cbc149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc3b45700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0be970 con 0x7f7cbc071e80 2023-03-03T16:15:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc3b45700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc1496a0 con 0x7f7cbc072250 2023-03-03T16:15:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc3b45700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc14dd20 con 0x7f7cbc14e900 2023-03-03T16:15:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc18e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 0x7f7cbc0c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:07.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc18e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 0x7f7cbc0c6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56386/0 (socket says 172.21.15.7:56386) 2023-03-03T16:15:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.057+0000 7f7cc18e1700 1 -- 172.21.15.7:0/3990536801 learned_addr learned my addr 172.21.15.7:0/3990536801 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:15:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc10e0700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cbc072250 0x7f7cbc148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc20e2700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cbc14e900 0x7f7cbc149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc18e1700 1 -- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cbc14e900 msgr2=0x7f7cbc149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc18e1700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cbc14e900 0x7f7cbc149160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc18e1700 1 -- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cbc072250 msgr2=0x7f7cbc148c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc18e1700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cbc072250 0x7f7cbc148c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:07.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc18e1700 1 -- 172.21.15.7:0/3990536801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0094a0 con 0x7f7cbc071e80 2023-03-03T16:15:07.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc10e0700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cbc072250 0x7f7cbc148c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:15:07.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc18e1700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 0x7f7cbc0c6550 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f7cac000c00 tx=0x7f7cac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:15:07.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cb27fc700 1 -- 172.21.15.7:0/3990536801 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac0036a0 con 0x7f7cbc071e80 2023-03-03T16:15:07.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cb27fc700 1 -- 172.21.15.7:0/3990536801 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac01f070 con 0x7f7cbc071e80 2023-03-03T16:15:07.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3990536801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cbc06c9a0 con 0x7f7cbc071e80 2023-03-03T16:15:07.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cb27fc700 1 -- 172.21.15.7:0/3990536801 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac003a60 con 0x7f7cbc071e80 2023-03-03T16:15:07.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.058+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3990536801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cbc151300 con 0x7f7cbc071e80 2023-03-03T16:15:07.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.059+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3990536801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cbc06cb30 con 0x7f7cbc071e80 2023-03-03T16:15:07.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.060+0000 7f7cb27fc700 1 -- 172.21.15.7:0/3990536801 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7cac00f050 con 0x7f7cbc071e80 2023-03-03T16:15:07.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.060+0000 7f7cb27fc700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ca00682f0 0x7f7ca006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:15:07.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.060+0000 7f7cb27fc700 1 -- 172.21.15.7:0/3990536801 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7cac021070 con 0x7f7cbc071e80 2023-03-03T16:15:07.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.060+0000 7f7cc10e0700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ca00682f0 0x7f7ca006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:15:07.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.061+0000 7f7cc10e0700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ca00682f0 0x7f7ca006a7a0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8005910 tx=0x7f7cb8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:15:07.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.065+0000 7f7cb27fc700 1 -- 172.21.15.7:0/3990536801 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7cac053e70 con 0x7f7cbc071e80 2023-03-03T16:15:07.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.303+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3990536801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cbc074bb0 con 0x7f7cbc071e80 2023-03-03T16:15:07.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.303+0000 7f7cb27fc700 1 -- 172.21.15.7:0/3990536801 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7cac015090 con 0x7f7cbc071e80 2023-03-03T16:15:07.304 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:15:07.304 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:15:07.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ca00682f0 msgr2=0x7f7ca006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:07.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ca00682f0 0x7f7ca006a7a0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8005910 tx=0x7f7cb8009040 comp rx=0 tx=0).stop 2023-03-03T16:15:07.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3990536801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 msgr2=0x7f7cbc0c6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:15:07.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 0x7f7cbc0c6550 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f7cac000c00 tx=0x7f7cac004750 comp rx=0 tx=0).stop 2023-03-03T16:15:07.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3990536801 shutdown_connections 2023-03-03T16:15:07.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cbc14e900 0x7f7cbc149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:07.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7ca00682f0 0x7f7ca006a7a0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:07.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cbc071e80 0x7f7cbc0c6550 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:07.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 --2- 172.21.15.7:0/3990536801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cbc072250 0x7f7cbc148c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:15:07.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3990536801 >> 172.21.15.7:0/3990536801 conn(0x7f7cbc0a94d0 msgr2=0x7f7cbc06ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:15:07.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3990536801 shutdown_connections 2023-03-03T16:15:07.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:15:07.305+0000 7f7cc3b45700 1 -- 172.21.15.7:0/3990536801 wait complete. 2023-03-03T16:15:07.308 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:15:07.318 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T16:15:13.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:13.558 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:43.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:15:43.629 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:13.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:16:13.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.457+0000 7f77ef805700 1 Processor -- start 2023-03-03T16:16:13.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.457+0000 7f77ef805700 1 -- start start 2023-03-03T16:16:13.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77ef805700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e814e560 0x7f77e8071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:13.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77ef805700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f77e80c43d0 0x7f77e80c4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:13.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77ef805700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e8072920 con 0x7f77e8154590 2023-03-03T16:16:13.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77ef805700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e806bef0 con 0x7f77e80c43d0 2023-03-03T16:16:13.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77ef805700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e806b910 con 0x7f77e814e560 2023-03-03T16:16:13.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77edda2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f77e8154590 0x7f77e81549f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50440/0 (socket says 172.21.15.7:50440) 2023-03-03T16:16:13.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77edda2700 1 -- 172.21.15.7:0/3784197577 learned_addr learned my addr 172.21.15.7:0/3784197577 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:16:13.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77ed5a1700 1 --2- 172.21.15.7:0/3784197577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e814e560 0x7f77e8071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:13.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77ecda0700 1 --2- 172.21.15.7:0/3784197577 >> v2:172.21.15.143:3300/0 conn(0x7f77e80c43d0 0x7f77e80c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:13.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77dffff700 1 -- 172.21.15.7:0/3784197577 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529161251 0 0) 0x7f77e0001490 con 0x7f77e8154590 2023-03-03T16:16:13.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.458+0000 7f77dffff700 1 -- 172.21.15.7:0/3784197577 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d40035b0 con 0x7f77e8154590 2023-03-03T16:16:13.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.459+0000 7f77dffff700 1 -- 172.21.15.7:0/3784197577 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2384013318 0 0) 0x7f77d40035b0 con 0x7f77e8154590 2023-03-03T16:16:13.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.459+0000 7f77ed5a1700 1 -- 172.21.15.7:0/3784197577 >> v2:172.21.15.143:3300/0 conn(0x7f77e80c43d0 msgr2=0x7f77e80c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:16:13.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.459+0000 7f77ed5a1700 1 --2- 172.21.15.7:0/3784197577 >> v2:172.21.15.143:3300/0 conn(0x7f77e80c43d0 0x7f77e80c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:13.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.459+0000 7f77ed5a1700 1 -- 172.21.15.7:0/3784197577 >> v1:172.21.15.7:6789/0 conn(0x7f77e8154590 legacy=0x7f77e81549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:13.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.459+0000 7f77ed5a1700 1 -- 172.21.15.7:0/3784197577 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e80aacc0 con 0x7f77e814e560 2023-03-03T16:16:13.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.459+0000 7f77ecda0700 1 --2- 172.21.15.7:0/3784197577 >> v2:172.21.15.143:3300/0 conn(0x7f77e80c43d0 0x7f77e80c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:16:13.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.459+0000 7f77ed5a1700 1 --2- 172.21.15.7:0/3784197577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e814e560 0x7f77e8071c70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009840 tx=0x7f77d8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adca4c484185a444 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:16:13.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.459+0000 7f77dffff700 1 -- 172.21.15.7:0/3784197577 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0001b80 con 0x7f77e814e560 2023-03-03T16:16:13.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.459+0000 7f77dffff700 1 -- 172.21.15.7:0/3784197577 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d8004030 con 0x7f77e814e560 2023-03-03T16:16:13.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.459+0000 7f77dffff700 1 -- 172.21.15.7:0/3784197577 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d80043a0 con 0x7f77e814e560 2023-03-03T16:16:13.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.460+0000 7f77ef805700 1 -- 172.21.15.7:0/3784197577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e814e560 msgr2=0x7f77e8071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:13.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.460+0000 7f77ef805700 1 --2- 172.21.15.7:0/3784197577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e814e560 0x7f77e8071c70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009840 tx=0x7f77d8009b50 comp rx=0 tx=0).stop 2023-03-03T16:16:13.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.460+0000 7f77ef805700 1 -- 172.21.15.7:0/3784197577 shutdown_connections 2023-03-03T16:16:13.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.460+0000 7f77ef805700 1 --2- 172.21.15.7:0/3784197577 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e814e560 0x7f77e8071c70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:13.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.460+0000 7f77ef805700 1 --2- 172.21.15.7:0/3784197577 >> v2:172.21.15.143:3300/0 conn(0x7f77e80c43d0 0x7f77e80c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:13.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.460+0000 7f77ef805700 1 -- 172.21.15.7:0/3784197577 >> 172.21.15.7:0/3784197577 conn(0x7f77e80736f0 msgr2=0x7f77e8073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:16:13.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.460+0000 7f77ef805700 1 -- 172.21.15.7:0/3784197577 shutdown_connections 2023-03-03T16:16:13.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.460+0000 7f77ef805700 1 -- 172.21.15.7:0/3784197577 wait complete. 2023-03-03T16:16:13.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.461+0000 7f77ef805700 1 Processor -- start 2023-03-03T16:16:13.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.461+0000 7f77ef805700 1 -- start start 2023-03-03T16:16:13.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ef805700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e80c43d0 0x7f77e81e71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:13.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ef805700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77e814e560 0x7f77e81576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:13.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ef805700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77e8154590 0x7f77e8157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:13.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ef805700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e81ea810 con 0x7f77e8154590 2023-03-03T16:16:13.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ef805700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e8157020 con 0x7f77e814e560 2023-03-03T16:16:13.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ef805700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e8158160 con 0x7f77e80c43d0 2023-03-03T16:16:13.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77edda2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77e8154590 0x7f77e8157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:13.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77edda2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77e8154590 0x7f77e8157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56466/0 (socket says 172.21.15.7:56466) 2023-03-03T16:16:13.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77edda2700 1 -- 172.21.15.7:0/2425608144 learned_addr learned my addr 172.21.15.7:0/2425608144 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:16:13.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ed5a1700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e80c43d0 0x7f77e81e71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:13.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ecda0700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77e814e560 0x7f77e81576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:13.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77edda2700 1 -- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e80c43d0 msgr2=0x7f77e81e71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:13.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77edda2700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e80c43d0 0x7f77e81e71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:13.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77edda2700 1 -- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77e814e560 msgr2=0x7f77e81576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:13.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77edda2700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77e814e560 0x7f77e81576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:13.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77edda2700 1 -- 172.21.15.7:0/2425608144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e40094a0 con 0x7f77e8154590 2023-03-03T16:16:13.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77edda2700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77e8154590 0x7f77e8157c20 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f77e000cc10 tx=0x7f77e000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:16:13.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ddffb700 1 -- 172.21.15.7:0/2425608144 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e000aa40 con 0x7f77e8154590 2023-03-03T16:16:13.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ddffb700 1 -- 172.21.15.7:0/2425608144 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e000abc0 con 0x7f77e8154590 2023-03-03T16:16:13.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.462+0000 7f77ef805700 1 -- 172.21.15.7:0/2425608144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d80094a0 con 0x7f77e8154590 2023-03-03T16:16:13.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.463+0000 7f77ddffb700 1 -- 172.21.15.7:0/2425608144 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e00070d0 con 0x7f77e8154590 2023-03-03T16:16:13.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.463+0000 7f77ef805700 1 -- 172.21.15.7:0/2425608144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e8155100 con 0x7f77e8154590 2023-03-03T16:16:13.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.464+0000 7f77ef805700 1 -- 172.21.15.7:0/2425608144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e806bb80 con 0x7f77e8154590 2023-03-03T16:16:13.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.464+0000 7f77ddffb700 1 -- 172.21.15.7:0/2425608144 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f77e001a030 con 0x7f77e8154590 2023-03-03T16:16:13.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.465+0000 7f77ddffb700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77cc0682f0 0x7f77cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:13.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.465+0000 7f77ddffb700 1 -- 172.21.15.7:0/2425608144 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f77e0085c10 con 0x7f77e8154590 2023-03-03T16:16:13.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.465+0000 7f77ed5a1700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77cc0682f0 0x7f77cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:13.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.465+0000 7f77ed5a1700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77cc0682f0 0x7f77cc06a7a0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f77d80090b0 tx=0x7f77d8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:16:13.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.469+0000 7f77ddffb700 1 -- 172.21.15.7:0/2425608144 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f77e0051fb0 con 0x7f77e8154590 2023-03-03T16:16:13.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:13.699 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:13.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.708+0000 7f77ef805700 1 -- 172.21.15.7:0/2425608144 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77e8074bb0 con 0x7f77e8154590 2023-03-03T16:16:13.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.709+0000 7f77ddffb700 1 -- 172.21.15.7:0/2425608144 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f77e00555d0 con 0x7f77e8154590 2023-03-03T16:16:13.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:16:13.710 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:16:13.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 -- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77cc0682f0 msgr2=0x7f77cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:13.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77cc0682f0 0x7f77cc06a7a0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f77d80090b0 tx=0x7f77d8008030 comp rx=0 tx=0).stop 2023-03-03T16:16:13.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 -- 172.21.15.7:0/2425608144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77e8154590 msgr2=0x7f77e8157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:13.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77e8154590 0x7f77e8157c20 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f77e000cc10 tx=0x7f77e000cf20 comp rx=0 tx=0).stop 2023-03-03T16:16:13.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 -- 172.21.15.7:0/2425608144 shutdown_connections 2023-03-03T16:16:13.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77cc0682f0 0x7f77cc06a7a0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:13.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77e8154590 0x7f77e8157c20 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:13.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77e80c43d0 0x7f77e81e71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:13.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 --2- 172.21.15.7:0/2425608144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77e814e560 0x7f77e81576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:13.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 -- 172.21.15.7:0/2425608144 >> 172.21.15.7:0/2425608144 conn(0x7f77e80736f0 msgr2=0x7f77e8073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:16:13.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 -- 172.21.15.7:0/2425608144 shutdown_connections 2023-03-03T16:16:13.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:13.711+0000 7f77ef805700 1 -- 172.21.15.7:0/2425608144 wait complete. 2023-03-03T16:16:13.713 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:16:13.724 INFO:tasks.mds_thrash.fs.[a]:waiting for 12 secs before thrashing 2023-03-03T16:16:25.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:16:25.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.874+0000 7f2c58faa700 1 Processor -- start 2023-03-03T16:16:25.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.874+0000 7f2c58faa700 1 -- start start 2023-03-03T16:16:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.875+0000 7f2c58faa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2c5406e020 0x7f2c54071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.875+0000 7f2c58faa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 0x7f2c54151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.875+0000 7f2c58faa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c54154910 con 0x7f2c54155390 2023-03-03T16:16:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.875+0000 7f2c58faa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c54070b90 con 0x7f2c5406e020 2023-03-03T16:16:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.875+0000 7f2c58faa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c54151320 con 0x7f2c54072520 2023-03-03T16:16:25.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.875+0000 7f2c51d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2c5406e020 0x7f2c54071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:25.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.875+0000 7f2c51d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2c5406e020 0x7f2c54071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:52250/0 (socket says 172.21.15.7:52250) 2023-03-03T16:16:25.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.875+0000 7f2c51d9b700 1 -- 172.21.15.7:0/579613125 learned_addr learned my addr 172.21.15.7:0/579613125 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:16:25.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.875+0000 7f2c52d9d700 1 --2- 172.21.15.7:0/579613125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 0x7f2c54151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:25.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.876+0000 7f2c51d9b700 1 -- 172.21.15.7:0/579613125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 msgr2=0x7f2c54151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:16:25.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.876+0000 7f2c51d9b700 1 --2- 172.21.15.7:0/579613125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 0x7f2c54151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:25.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.876+0000 7f2c51d9b700 1 -- 172.21.15.7:0/579613125 >> v1:172.21.15.143:6789/0 conn(0x7f2c54155390 legacy=0x7f2c540c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:25.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.876+0000 7f2c51d9b700 1 -- 172.21.15.7:0/579613125 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c540c4540 con 0x7f2c5406e020 2023-03-03T16:16:25.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.876+0000 7f2c5159a700 1 -- 172.21.15.7:0/579613125 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284681295 0 0) 0x7f2c54154910 con 0x7f2c54155390 2023-03-03T16:16:25.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.876+0000 7f2c52d9d700 1 --2- 172.21.15.7:0/579613125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 0x7f2c54151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:16:25.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.876+0000 7f2c51d9b700 1 --2- 172.21.15.7:0/579613125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5406e020 0x7f2c54071fe0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2c4400bb90 tx=0x7f2c4400bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8c1e86241fab0be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:16:25.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.876+0000 7f2c5159a700 1 -- 172.21.15.7:0/579613125 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4400dd30 con 0x7f2c5406e020 2023-03-03T16:16:25.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.876+0000 7f2c5159a700 1 -- 172.21.15.7:0/579613125 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4400de90 con 0x7f2c5406e020 2023-03-03T16:16:25.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.876+0000 7f2c5159a700 1 -- 172.21.15.7:0/579613125 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44010690 con 0x7f2c5406e020 2023-03-03T16:16:25.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.877+0000 7f2c58faa700 1 -- 172.21.15.7:0/579613125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5406e020 msgr2=0x7f2c54071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:25.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.877+0000 7f2c58faa700 1 --2- 172.21.15.7:0/579613125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5406e020 0x7f2c54071fe0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2c4400bb90 tx=0x7f2c4400bea0 comp rx=0 tx=0).stop 2023-03-03T16:16:25.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.877+0000 7f2c58faa700 1 -- 172.21.15.7:0/579613125 shutdown_connections 2023-03-03T16:16:25.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.877+0000 7f2c58faa700 1 --2- 172.21.15.7:0/579613125 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 0x7f2c54151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:25.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.877+0000 7f2c58faa700 1 --2- 172.21.15.7:0/579613125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5406e020 0x7f2c54071fe0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:25.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.877+0000 7f2c58faa700 1 -- 172.21.15.7:0/579613125 >> 172.21.15.7:0/579613125 conn(0x7f2c5406c110 msgr2=0x7f2c5406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:16:25.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.877+0000 7f2c58faa700 1 -- 172.21.15.7:0/579613125 shutdown_connections 2023-03-03T16:16:25.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.877+0000 7f2c58faa700 1 -- 172.21.15.7:0/579613125 wait complete. 2023-03-03T16:16:25.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.878+0000 7f2c58faa700 1 Processor -- start 2023-03-03T16:16:25.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.878+0000 7f2c58faa700 1 -- start start 2023-03-03T16:16:25.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c58faa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5406e020 0x7f2c541ea510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:25.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c58faa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 0x7f2c541dd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:25.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c58faa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c54155390 0x7f2c540aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:25.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c58faa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c54147e80 con 0x7f2c54155390 2023-03-03T16:16:25.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c58faa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c54154410 con 0x7f2c5406e020 2023-03-03T16:16:25.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c58faa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c540c1000 con 0x7f2c54072520 2023-03-03T16:16:25.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c52d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c54155390 0x7f2c540aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:25.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c52d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c54155390 0x7f2c540aa450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56484/0 (socket says 172.21.15.7:56484) 2023-03-03T16:16:25.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c52d9d700 1 -- 172.21.15.7:0/2713661633 learned_addr learned my addr 172.21.15.7:0/2713661633 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:16:25.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c5259c700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5406e020 0x7f2c541ea510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:25.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.879+0000 7f2c51d9b700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 0x7f2c541dd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:25.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c52d9d700 1 -- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 msgr2=0x7f2c541dd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:25.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c52d9d700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 0x7f2c541dd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:25.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c52d9d700 1 -- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5406e020 msgr2=0x7f2c541ea510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:25.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c52d9d700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5406e020 0x7f2c541ea510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:25.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c52d9d700 1 -- 172.21.15.7:0/2713661633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c009a70 con 0x7f2c54155390 2023-03-03T16:16:25.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c52d9d700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c54155390 0x7f2c540aa450 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00efc0 tx=0x7f2c4c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:16:25.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c42ffd700 1 -- 172.21.15.7:0/2713661633 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00ca90 con 0x7f2c54155390 2023-03-03T16:16:25.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c58faa700 1 -- 172.21.15.7:0/2713661633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4400b7f0 con 0x7f2c54155390 2023-03-03T16:16:25.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c42ffd700 1 -- 172.21.15.7:0/2713661633 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c00f040 con 0x7f2c54155390 2023-03-03T16:16:25.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c42ffd700 1 -- 172.21.15.7:0/2713661633 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00ce10 con 0x7f2c54155390 2023-03-03T16:16:25.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.880+0000 7f2c58faa700 1 -- 172.21.15.7:0/2713661633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c54070540 con 0x7f2c54155390 2023-03-03T16:16:25.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.881+0000 7f2c58faa700 1 -- 172.21.15.7:0/2713661633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c540702a0 con 0x7f2c54155390 2023-03-03T16:16:25.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.881+0000 7f2c42ffd700 1 -- 172.21.15.7:0/2713661633 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2c4c014480 con 0x7f2c54155390 2023-03-03T16:16:25.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.882+0000 7f2c42ffd700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c340682f0 0x7f2c3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:25.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.882+0000 7f2c42ffd700 1 -- 172.21.15.7:0/2713661633 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2c4c0869b0 con 0x7f2c54155390 2023-03-03T16:16:25.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.882+0000 7f2c5259c700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c340682f0 0x7f2c3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:25.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.883+0000 7f2c5259c700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c340682f0 0x7f2c3406a7a0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f2c48000f60 tx=0x7f2c48009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:16:25.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:25.886+0000 7f2c42ffd700 1 -- 172.21.15.7:0/2713661633 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2c4c052c70 con 0x7f2c54155390 2023-03-03T16:16:26.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.124+0000 7f2c58faa700 1 -- 172.21.15.7:0/2713661633 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c54070820 con 0x7f2c54155390 2023-03-03T16:16:26.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.124+0000 7f2c42ffd700 1 -- 172.21.15.7:0/2713661633 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2c4c009d70 con 0x7f2c54155390 2023-03-03T16:16:26.125 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:16:26.125 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:16:26.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 -- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c340682f0 msgr2=0x7f2c3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:26.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c340682f0 0x7f2c3406a7a0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f2c48000f60 tx=0x7f2c48009040 comp rx=0 tx=0).stop 2023-03-03T16:16:26.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 -- 172.21.15.7:0/2713661633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c54155390 msgr2=0x7f2c540aa450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:26.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c54155390 0x7f2c540aa450 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00efc0 tx=0x7f2c4c00c3a0 comp rx=0 tx=0).stop 2023-03-03T16:16:26.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 -- 172.21.15.7:0/2713661633 shutdown_connections 2023-03-03T16:16:26.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2c340682f0 0x7f2c3406a7a0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:26.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c54155390 0x7f2c540aa450 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:26.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c5406e020 0x7f2c541ea510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:26.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 --2- 172.21.15.7:0/2713661633 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2c54072520 0x7f2c541dd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:26.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 -- 172.21.15.7:0/2713661633 >> 172.21.15.7:0/2713661633 conn(0x7f2c5406c110 msgr2=0x7f2c5406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:16:26.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 -- 172.21.15.7:0/2713661633 shutdown_connections 2023-03-03T16:16:26.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:26.126+0000 7f2c58faa700 1 -- 172.21.15.7:0/2713661633 wait complete. 2023-03-03T16:16:26.129 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:16:26.139 INFO:tasks.mds_thrash.fs.[a]:waiting for 81 secs before thrashing 2023-03-03T16:16:32.359 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:16:32.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.496+0000 7fe5b986f700 1 Processor -- start 2023-03-03T16:16:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.496+0000 7fe5b986f700 1 -- start start 2023-03-03T16:16:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b986f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5b40c7a30 0x7fe5b414e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:32.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b986f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe5b40c7e00 0x7fe5b414e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:32.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b986f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b4157bb0 con 0x7fe5b406b6f0 2023-03-03T16:16:32.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b986f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b406ab00 con 0x7fe5b40c7a30 2023-03-03T16:16:32.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b986f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b41545c0 con 0x7fe5b40c7e00 2023-03-03T16:16:32.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b2ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5b40c7a30 0x7fe5b414e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:32.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b2ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5b40c7a30 0x7fe5b414e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56490/0 (socket says 172.21.15.7:56490) 2023-03-03T16:16:32.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b2ffd700 1 -- 172.21.15.7:0/1904449231 learned_addr learned my addr 172.21.15.7:0/1904449231 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:16:32.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b27fc700 1 --2- 172.21.15.7:0/1904449231 >> v2:172.21.15.143:3300/0 conn(0x7fe5b40c7e00 0x7fe5b414e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:32.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b2ffd700 1 -- 172.21.15.7:0/1904449231 >> v2:172.21.15.143:3300/0 conn(0x7fe5b40c7e00 msgr2=0x7fe5b414e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:16:32.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b2ffd700 1 --2- 172.21.15.7:0/1904449231 >> v2:172.21.15.143:3300/0 conn(0x7fe5b40c7e00 0x7fe5b414e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:32.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b2ffd700 1 -- 172.21.15.7:0/1904449231 >> v1:172.21.15.143:6789/0 conn(0x7fe5b406b6f0 legacy=0x7fe5b4154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:32.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b2ffd700 1 -- 172.21.15.7:0/1904449231 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b406a2f0 con 0x7fe5b40c7a30 2023-03-03T16:16:32.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b2ffd700 1 --2- 172.21.15.7:0/1904449231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 0x7fe5b414e160 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7fe5b408c770 tx=0x7fe5a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aaba3312a1fee3b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:16:32.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b1ffb700 1 -- 172.21.15.7:0/1904449231 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4005e30 con 0x7fe5b40c7a30 2023-03-03T16:16:32.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b1ffb700 1 -- 172.21.15.7:0/1904449231 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a4004030 con 0x7fe5b40c7a30 2023-03-03T16:16:32.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.497+0000 7fe5b1ffb700 1 -- 172.21.15.7:0/1904449231 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40043e0 con 0x7fe5b40c7a30 2023-03-03T16:16:32.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.498+0000 7fe5b986f700 1 -- 172.21.15.7:0/1904449231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 msgr2=0x7fe5b414e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:32.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.498+0000 7fe5b986f700 1 --2- 172.21.15.7:0/1904449231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 0x7fe5b414e160 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7fe5b408c770 tx=0x7fe5a4009bd0 comp rx=0 tx=0).stop 2023-03-03T16:16:32.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.498+0000 7fe5b986f700 1 -- 172.21.15.7:0/1904449231 shutdown_connections 2023-03-03T16:16:32.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.498+0000 7fe5b986f700 1 --2- 172.21.15.7:0/1904449231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 0x7fe5b414e160 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:32.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.498+0000 7fe5b986f700 1 --2- 172.21.15.7:0/1904449231 >> v2:172.21.15.143:3300/0 conn(0x7fe5b40c7e00 0x7fe5b414e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:32.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.498+0000 7fe5b986f700 1 -- 172.21.15.7:0/1904449231 >> 172.21.15.7:0/1904449231 conn(0x7fe5b4073d20 msgr2=0x7fe5b40716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:16:32.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.498+0000 7fe5b986f700 1 -- 172.21.15.7:0/1904449231 shutdown_connections 2023-03-03T16:16:32.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.498+0000 7fe5b986f700 1 -- 172.21.15.7:0/1904449231 wait complete. 2023-03-03T16:16:32.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.499+0000 7fe5b986f700 1 Processor -- start 2023-03-03T16:16:32.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.499+0000 7fe5b986f700 1 -- start start 2023-03-03T16:16:32.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b986f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 0x7fe5b41495f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:32.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b986f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5b40c7e00 0x7fe5b41ed160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:32.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b986f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5b41ed6a0 0x7fe5b413f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:32.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b986f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b40c2090 con 0x7fe5b40c7a30 2023-03-03T16:16:32.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b986f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b414c500 con 0x7fe5b41ed6a0 2023-03-03T16:16:32.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b986f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b4140680 con 0x7fe5b40c7e00 2023-03-03T16:16:32.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 0x7fe5b41495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:32.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 0x7fe5b41495f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56494/0 (socket says 172.21.15.7:56494) 2023-03-03T16:16:32.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b2ffd700 1 -- 172.21.15.7:0/3134807370 learned_addr learned my addr 172.21.15.7:0/3134807370 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:16:32.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b27fc700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5b40c7e00 0x7fe5b41ed160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:32.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.500+0000 7fe5b37fe700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5b41ed6a0 0x7fe5b413f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:32.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe5b2ffd700 1 -- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5b40c7e00 msgr2=0x7fe5b41ed160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:32.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe5b2ffd700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5b40c7e00 0x7fe5b41ed160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:32.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe5b2ffd700 1 -- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5b41ed6a0 msgr2=0x7fe5b413f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:32.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe5b2ffd700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5b41ed6a0 0x7fe5b413f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:32.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe5b2ffd700 1 -- 172.21.15.7:0/3134807370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a4009500 con 0x7fe5b40c7a30 2023-03-03T16:16:32.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe5b37fe700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5b41ed6a0 0x7fe5b413f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:16:32.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe5b27fc700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5b40c7e00 0x7fe5b41ed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:16:32.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe5b2ffd700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 0x7fe5b41495f0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009130 tx=0x7fe5a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:16:32.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe59bfff700 1 -- 172.21.15.7:0/3134807370 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4003dd0 con 0x7fe5b40c7a30 2023-03-03T16:16:32.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe59bfff700 1 -- 172.21.15.7:0/3134807370 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a4011420 con 0x7fe5b40c7a30 2023-03-03T16:16:32.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe5b986f700 1 -- 172.21.15.7:0/3134807370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b406a540 con 0x7fe5b40c7a30 2023-03-03T16:16:32.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe59bfff700 1 -- 172.21.15.7:0/3134807370 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40117d0 con 0x7fe5b40c7a30 2023-03-03T16:16:32.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.501+0000 7fe5b986f700 1 -- 172.21.15.7:0/3134807370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b40bfbb0 con 0x7fe5b40c7a30 2023-03-03T16:16:32.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.502+0000 7fe5b986f700 1 -- 172.21.15.7:0/3134807370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b406a820 con 0x7fe5b40c7a30 2023-03-03T16:16:32.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.502+0000 7fe59bfff700 1 -- 172.21.15.7:0/3134807370 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe5a402c070 con 0x7fe5b40c7a30 2023-03-03T16:16:32.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.503+0000 7fe59bfff700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5a00682f0 0x7fe5a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:16:32.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.503+0000 7fe59bfff700 1 -- 172.21.15.7:0/3134807370 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe5a4087160 con 0x7fe5b40c7a30 2023-03-03T16:16:32.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.503+0000 7fe5b27fc700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5a00682f0 0x7fe5a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:16:32.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.504+0000 7fe5b27fc700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5a00682f0 0x7fe5a006a7a0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40c3650 tx=0x7fe5a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:16:32.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.507+0000 7fe59bfff700 1 -- 172.21.15.7:0/3134807370 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5a4053500 con 0x7fe5b40c7a30 2023-03-03T16:16:32.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.749+0000 7fe5b986f700 1 -- 172.21.15.7:0/3134807370 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5b40c2400 con 0x7fe5b40c7a30 2023-03-03T16:16:32.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.749+0000 7fe59bfff700 1 -- 172.21.15.7:0/3134807370 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe5a4056b20 con 0x7fe5b40c7a30 2023-03-03T16:16:32.751 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:16:32.751 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:16:32.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.751+0000 7fe5b986f700 1 -- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5a00682f0 msgr2=0x7fe5a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:32.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.751+0000 7fe5b986f700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5a00682f0 0x7fe5a006a7a0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40c3650 tx=0x7fe5a8008040 comp rx=0 tx=0).stop 2023-03-03T16:16:32.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.751+0000 7fe5b986f700 1 -- 172.21.15.7:0/3134807370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 msgr2=0x7fe5b41495f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:16:32.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.751+0000 7fe5b986f700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 0x7fe5b41495f0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009130 tx=0x7fe5a4004750 comp rx=0 tx=0).stop 2023-03-03T16:16:32.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.751+0000 7fe5b986f700 1 -- 172.21.15.7:0/3134807370 shutdown_connections 2023-03-03T16:16:32.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.751+0000 7fe5b986f700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5b41ed6a0 0x7fe5b413f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:32.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.751+0000 7fe5b986f700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe5a00682f0 0x7fe5a006a7a0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:32.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.751+0000 7fe5b986f700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5b40c7a30 0x7fe5b41495f0 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:32.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.751+0000 7fe5b986f700 1 --2- 172.21.15.7:0/3134807370 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5b40c7e00 0x7fe5b41ed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:16:32.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.751+0000 7fe5b986f700 1 -- 172.21.15.7:0/3134807370 >> 172.21.15.7:0/3134807370 conn(0x7fe5b406b6f0 msgr2=0x7fe5b406acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:16:32.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.752+0000 7fe5b986f700 1 -- 172.21.15.7:0/3134807370 shutdown_connections 2023-03-03T16:16:32.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:16:32.752+0000 7fe5b986f700 1 -- 172.21.15.7:0/3134807370 wait complete. 2023-03-03T16:16:32.755 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:16:32.765 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2023-03-03T16:16:43.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:16:43.764 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:06.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:17:06.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.905+0000 7fddec3cb700 1 Processor -- start 2023-03-03T16:17:06.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.905+0000 7fddec3cb700 1 -- start start 2023-03-03T16:17:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.906+0000 7fddec3cb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdde40c5b00 0x7fdde40bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.906+0000 7fddec3cb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde40c08d0 con 0x7fdde4151e80 2023-03-03T16:17:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.906+0000 7fddec3cb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde4154930 con 0x7fdde4155410 2023-03-03T16:17:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.906+0000 7fddec3cb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde4152250 con 0x7fdde40c5b00 2023-03-03T16:17:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.906+0000 7fddea968700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdde40c5b00 0x7fdde40bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:06.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.906+0000 7fddea167700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdde4151e80 0x7fdde40c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50506/0 (socket says 172.21.15.7:50506) 2023-03-03T16:17:06.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.906+0000 7fddea167700 1 -- 172.21.15.7:0/3348786670 learned_addr learned my addr 172.21.15.7:0/3348786670 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:17:06.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.907+0000 7fddea968700 1 -- 172.21.15.7:0/3348786670 >> v1:172.21.15.143:6789/0 conn(0x7fdde4155410 legacy=0x7fdde40bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:17:06.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.907+0000 7fddea968700 1 -- 172.21.15.7:0/3348786670 >> v1:172.21.15.7:6789/0 conn(0x7fdde4151e80 legacy=0x7fdde40c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:06.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.907+0000 7fddea968700 1 -- 172.21.15.7:0/3348786670 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4154c10 con 0x7fdde40c5b00 2023-03-03T16:17:06.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.907+0000 7fdde9165700 1 -- 172.21.15.7:0/3348786670 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488856675 0 0) 0x7fdde40c08d0 con 0x7fdde4151e80 2023-03-03T16:17:06.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.907+0000 7fddea968700 1 --2- 172.21.15.7:0/3348786670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde40c5b00 0x7fdde40bfff0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009d90 tx=0x7fdddc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70bbd96b369055cf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:17:06.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.907+0000 7fdde9165700 1 -- 172.21.15.7:0/3348786670 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc00e070 con 0x7fdde40c5b00 2023-03-03T16:17:06.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.907+0000 7fdde9165700 1 -- 172.21.15.7:0/3348786670 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc00bbd0 con 0x7fdde40c5b00 2023-03-03T16:17:06.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.907+0000 7fdde9165700 1 -- 172.21.15.7:0/3348786670 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc012430 con 0x7fdde40c5b00 2023-03-03T16:17:06.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.908+0000 7fddec3cb700 1 -- 172.21.15.7:0/3348786670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde40c5b00 msgr2=0x7fdde40bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:06.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.908+0000 7fddec3cb700 1 --2- 172.21.15.7:0/3348786670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde40c5b00 0x7fdde40bfff0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009d90 tx=0x7fdddc00b330 comp rx=0 tx=0).stop 2023-03-03T16:17:06.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.908+0000 7fddec3cb700 1 -- 172.21.15.7:0/3348786670 shutdown_connections 2023-03-03T16:17:06.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.908+0000 7fddec3cb700 1 --2- 172.21.15.7:0/3348786670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde40c5b00 0x7fdde40bfff0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:06.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.908+0000 7fddec3cb700 1 -- 172.21.15.7:0/3348786670 >> 172.21.15.7:0/3348786670 conn(0x7fdde4071630 msgr2=0x7fdde406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:17:06.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.908+0000 7fddec3cb700 1 -- 172.21.15.7:0/3348786670 shutdown_connections 2023-03-03T16:17:06.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.908+0000 7fddec3cb700 1 -- 172.21.15.7:0/3348786670 wait complete. 2023-03-03T16:17:06.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.909+0000 7fddec3cb700 1 Processor -- start 2023-03-03T16:17:06.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.909+0000 7fddec3cb700 1 -- start start 2023-03-03T16:17:06.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddec3cb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde40c5b00 0x7fdde414bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:06.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddec3cb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde4151e80 0x7fdde41df280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:06.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddec3cb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde4155410 0x7fdde41df7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:06.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddec3cb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde41dfd00 con 0x7fdde4151e80 2023-03-03T16:17:06.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddec3cb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde41e0c90 con 0x7fdde40c5b00 2023-03-03T16:17:06.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddec3cb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde414c530 con 0x7fdde4155410 2023-03-03T16:17:06.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fdde9966700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde4151e80 0x7fdde41df280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:06.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fdde9966700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde4151e80 0x7fdde41df280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56532/0 (socket says 172.21.15.7:56532) 2023-03-03T16:17:06.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fdde9966700 1 -- 172.21.15.7:0/3084052530 learned_addr learned my addr 172.21.15.7:0/3084052530 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:17:06.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddea167700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde40c5b00 0x7fdde414bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:06.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddea968700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde4155410 0x7fdde41df7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:06.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fdde9966700 1 -- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde4155410 msgr2=0x7fdde41df7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:06.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fdde9966700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde4155410 0x7fdde41df7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:06.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fdde9966700 1 -- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde40c5b00 msgr2=0x7fdde414bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:06.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fdde9966700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde40c5b00 0x7fdde414bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:06.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fdde9966700 1 -- 172.21.15.7:0/3084052530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc009a70 con 0x7fdde4151e80 2023-03-03T16:17:06.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddea968700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde4155410 0x7fdde41df7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:17:06.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fdde9966700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde4151e80 0x7fdde41df280 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c480 tx=0x7fddd800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:17:06.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddd6ffd700 1 -- 172.21.15.7:0/3084052530 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd80052d0 con 0x7fdde4151e80 2023-03-03T16:17:06.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddd6ffd700 1 -- 172.21.15.7:0/3084052530 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd8005430 con 0x7fdde4151e80 2023-03-03T16:17:06.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddec3cb700 1 -- 172.21.15.7:0/3084052530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde40c5220 con 0x7fdde4151e80 2023-03-03T16:17:06.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.910+0000 7fddd6ffd700 1 -- 172.21.15.7:0/3084052530 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd80038c0 con 0x7fdde4151e80 2023-03-03T16:17:06.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.911+0000 7fddec3cb700 1 -- 172.21.15.7:0/3084052530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde4154ef0 con 0x7fdde4151e80 2023-03-03T16:17:06.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.912+0000 7fddec3cb700 1 -- 172.21.15.7:0/3084052530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde40c0530 con 0x7fdde4151e80 2023-03-03T16:17:06.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.912+0000 7fddd6ffd700 1 -- 172.21.15.7:0/3084052530 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fddd8019030 con 0x7fdde4151e80 2023-03-03T16:17:06.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.912+0000 7fddd6ffd700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddc80682f0 0x7fddc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:06.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.913+0000 7fddd6ffd700 1 -- 172.21.15.7:0/3084052530 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fddd8085490 con 0x7fdde4151e80 2023-03-03T16:17:06.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.913+0000 7fddea167700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddc80682f0 0x7fddc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:06.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.913+0000 7fddea167700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddc80682f0 0x7fddc806a7a0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fdde0007b10 tx=0x7fdde0007a60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:17:06.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:06.917+0000 7fddd6ffd700 1 -- 172.21.15.7:0/3084052530 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fddd8051800 con 0x7fdde4151e80 2023-03-03T16:17:07.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.152+0000 7fddec3cb700 1 -- 172.21.15.7:0/3084052530 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdde4074bb0 con 0x7fdde4151e80 2023-03-03T16:17:07.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.152+0000 7fddd6ffd700 1 -- 172.21.15.7:0/3084052530 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fddd8054e20 con 0x7fdde4151e80 2023-03-03T16:17:07.153 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:17:07.153 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:17:07.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 -- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddc80682f0 msgr2=0x7fddc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:07.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddc80682f0 0x7fddc806a7a0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fdde0007b10 tx=0x7fdde0007a60 comp rx=0 tx=0).stop 2023-03-03T16:17:07.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 -- 172.21.15.7:0/3084052530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde4151e80 msgr2=0x7fdde41df280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:07.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde4151e80 0x7fdde41df280 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c480 tx=0x7fddd800c790 comp rx=0 tx=0).stop 2023-03-03T16:17:07.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 -- 172.21.15.7:0/3084052530 shutdown_connections 2023-03-03T16:17:07.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdde4155410 0x7fdde41df7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:07.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdde40c5b00 0x7fdde414bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:07.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddc80682f0 0x7fddc806a7a0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:07.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 --2- 172.21.15.7:0/3084052530 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdde4151e80 0x7fdde41df280 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:07.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 -- 172.21.15.7:0/3084052530 >> 172.21.15.7:0/3084052530 conn(0x7fdde4071630 msgr2=0x7fdde4153750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:17:07.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 -- 172.21.15.7:0/3084052530 shutdown_connections 2023-03-03T16:17:07.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:07.154+0000 7fddec3cb700 1 -- 172.21.15.7:0/3084052530 wait complete. 2023-03-03T16:17:07.157 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:17:07.166 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T16:17:13.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:13.835 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:43.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:43.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:17:47.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:17:47.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.277+0000 7f408c8bd700 1 Processor -- start 2023-03-03T16:17:47.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.277+0000 7f408c8bd700 1 -- start start 2023-03-03T16:17:47.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f408c8bd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40880c1850 0x7f40880c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:47.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f408c8bd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40880aa170 0x7f40880aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:47.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f408c8bd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4088154790 con 0x7f408814b060 2023-03-03T16:17:47.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f408c8bd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40880c4100 con 0x7f40880aa170 2023-03-03T16:17:47.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f408c8bd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4088154a70 con 0x7f40880c1850 2023-03-03T16:17:47.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f4086d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40880aa170 0x7f40880aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:47.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f408659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f408814b060 0x7f408806f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50534/0 (socket says 172.21.15.7:50534) 2023-03-03T16:17:47.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f408659c700 1 -- 172.21.15.7:0/1656793520 learned_addr learned my addr 172.21.15.7:0/1656793520 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:17:47.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f4085d9b700 1 --2- 172.21.15.7:0/1656793520 >> v2:172.21.15.143:3300/0 conn(0x7f40880c1850 0x7f40880c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:47.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f408559a700 1 -- 172.21.15.7:0/1656793520 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808721886 0 0) 0x7f4088154790 con 0x7f408814b060 2023-03-03T16:17:47.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f408559a700 1 -- 172.21.15.7:0/1656793520 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40700035b0 con 0x7f408814b060 2023-03-03T16:17:47.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.278+0000 7f4086d9d700 1 -- 172.21.15.7:0/1656793520 >> v2:172.21.15.143:3300/0 conn(0x7f40880c1850 msgr2=0x7f40880c1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:17:47.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.279+0000 7f4086d9d700 1 --2- 172.21.15.7:0/1656793520 >> v2:172.21.15.143:3300/0 conn(0x7f40880c1850 0x7f40880c1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:47.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.279+0000 7f4086d9d700 1 -- 172.21.15.7:0/1656793520 >> v1:172.21.15.7:6789/0 conn(0x7f408814b060 legacy=0x7f408806f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:47.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.279+0000 7f4086d9d700 1 -- 172.21.15.7:0/1656793520 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4088154d50 con 0x7f40880aa170 2023-03-03T16:17:47.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.279+0000 7f4086d9d700 1 --2- 172.21.15.7:0/1656793520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40880aa170 0x7f40880aa570 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f40880724f0 tx=0x7f408000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2cdc0e6f932bc5f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:17:47.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.279+0000 7f408559a700 1 -- 172.21.15.7:0/1656793520 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408000e070 con 0x7f40880aa170 2023-03-03T16:17:47.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.279+0000 7f408559a700 1 -- 172.21.15.7:0/1656793520 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408000bb20 con 0x7f40880aa170 2023-03-03T16:17:47.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.279+0000 7f408559a700 1 -- 172.21.15.7:0/1656793520 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080012430 con 0x7f40880aa170 2023-03-03T16:17:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.279+0000 7f408c8bd700 1 -- 172.21.15.7:0/1656793520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40880aa170 msgr2=0x7f40880aa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.279+0000 7f408c8bd700 1 --2- 172.21.15.7:0/1656793520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40880aa170 0x7f40880aa570 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f40880724f0 tx=0x7f408000b330 comp rx=0 tx=0).stop 2023-03-03T16:17:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.280+0000 7f408c8bd700 1 -- 172.21.15.7:0/1656793520 shutdown_connections 2023-03-03T16:17:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.280+0000 7f408c8bd700 1 --2- 172.21.15.7:0/1656793520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40880aa170 0x7f40880aa570 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.280+0000 7f408c8bd700 1 --2- 172.21.15.7:0/1656793520 >> v2:172.21.15.143:3300/0 conn(0x7f40880c1850 0x7f40880c1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:47.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.280+0000 7f408c8bd700 1 -- 172.21.15.7:0/1656793520 >> 172.21.15.7:0/1656793520 conn(0x7f40880c4f20 msgr2=0x7f40880c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:17:47.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.280+0000 7f408c8bd700 1 -- 172.21.15.7:0/1656793520 shutdown_connections 2023-03-03T16:17:47.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.280+0000 7f408c8bd700 1 -- 172.21.15.7:0/1656793520 wait complete. 2023-03-03T16:17:47.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.281+0000 7f408c8bd700 1 Processor -- start 2023-03-03T16:17:47.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.281+0000 7f408c8bd700 1 -- start start 2023-03-03T16:17:47.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f408c8bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40880aa170 0x7f40881edda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:47.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f408c8bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40880c1850 0x7f4088151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:47.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f408c8bd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408814b060 0x7f40880ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:47.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f408c8bd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40881e9c30 con 0x7f408814b060 2023-03-03T16:17:47.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f408c8bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40881ed220 con 0x7f40880aa170 2023-03-03T16:17:47.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f408c8bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4088155f20 con 0x7f40880c1850 2023-03-03T16:17:47.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4086d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408814b060 0x7f40880ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:47.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4086d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408814b060 0x7f40880ad6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56560/0 (socket says 172.21.15.7:56560) 2023-03-03T16:17:47.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4086d9d700 1 -- 172.21.15.7:0/1958008418 learned_addr learned my addr 172.21.15.7:0/1958008418 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:17:47.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f408659c700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40880aa170 0x7f40881edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:47.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4085d9b700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40880c1850 0x7f4088151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:47.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4086d9d700 1 -- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40880c1850 msgr2=0x7f4088151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:47.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4086d9d700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40880c1850 0x7f4088151310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:47.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4086d9d700 1 -- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40880aa170 msgr2=0x7f40881edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:47.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4086d9d700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40880aa170 0x7f40881edda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:47.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4086d9d700 1 -- 172.21.15.7:0/1958008418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4080009a70 con 0x7f408814b060 2023-03-03T16:17:47.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f408659c700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40880aa170 0x7f40881edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:17:47.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4085d9b700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40880c1850 0x7f4088151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:17:47.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4086d9d700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408814b060 0x7f40880ad6a0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f4080014040 tx=0x7f4080004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:17:47.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4076ffd700 1 -- 172.21.15.7:0/1958008418 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080012430 con 0x7f408814b060 2023-03-03T16:17:47.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f4076ffd700 1 -- 172.21.15.7:0/1958008418 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4080020070 con 0x7f408814b060 2023-03-03T16:17:47.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.282+0000 7f408c8bd700 1 -- 172.21.15.7:0/1958008418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4088155310 con 0x7f408814b060 2023-03-03T16:17:47.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.283+0000 7f4076ffd700 1 -- 172.21.15.7:0/1958008418 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080004280 con 0x7f408814b060 2023-03-03T16:17:47.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.283+0000 7f408c8bd700 1 -- 172.21.15.7:0/1958008418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4088155030 con 0x7f408814b060 2023-03-03T16:17:47.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.283+0000 7f408c8bd700 1 -- 172.21.15.7:0/1958008418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4088154440 con 0x7f408814b060 2023-03-03T16:17:47.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.284+0000 7f4076ffd700 1 -- 172.21.15.7:0/1958008418 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f40800043e0 con 0x7f408814b060 2023-03-03T16:17:47.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.284+0000 7f4076ffd700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40680682f0 0x7f406806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:17:47.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.284+0000 7f4076ffd700 1 -- 172.21.15.7:0/1958008418 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4080022070 con 0x7f408814b060 2023-03-03T16:17:47.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.285+0000 7f408659c700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40680682f0 0x7f406806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:17:47.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.285+0000 7f408659c700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40680682f0 0x7f406806a7a0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f407c0011e0 tx=0x7f407c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:17:47.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.289+0000 7f4076ffd700 1 -- 172.21.15.7:0/1958008418 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4080054760 con 0x7f408814b060 2023-03-03T16:17:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.528+0000 7f408c8bd700 1 -- 172.21.15.7:0/1958008418 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40881e9da0 con 0x7f408814b060 2023-03-03T16:17:47.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.529+0000 7f4076ffd700 1 -- 172.21.15.7:0/1958008418 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f40881e9da0 con 0x7f408814b060 2023-03-03T16:17:47.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:17:47.531 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:17:47.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 -- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40680682f0 msgr2=0x7f406806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:47.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40680682f0 0x7f406806a7a0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f407c0011e0 tx=0x7f407c009040 comp rx=0 tx=0).stop 2023-03-03T16:17:47.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 -- 172.21.15.7:0/1958008418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408814b060 msgr2=0x7f40880ad6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:17:47.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408814b060 0x7f40880ad6a0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f4080014040 tx=0x7f4080004ad0 comp rx=0 tx=0).stop 2023-03-03T16:17:47.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 -- 172.21.15.7:0/1958008418 shutdown_connections 2023-03-03T16:17:47.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40680682f0 0x7f406806a7a0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:47.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408814b060 0x7f40880ad6a0 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:47.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40880aa170 0x7f40881edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:47.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 --2- 172.21.15.7:0/1958008418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40880c1850 0x7f4088151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:17:47.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 -- 172.21.15.7:0/1958008418 >> 172.21.15.7:0/1958008418 conn(0x7f40880c4f20 msgr2=0x7f4088148270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:17:47.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 -- 172.21.15.7:0/1958008418 shutdown_connections 2023-03-03T16:17:47.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:17:47.531+0000 7f408c8bd700 1 -- 172.21.15.7:0/1958008418 wait complete. 2023-03-03T16:17:47.534 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:17:47.544 INFO:tasks.mds_thrash.fs.[a]:waiting for 48 secs before thrashing 2023-03-03T16:18:14.035 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:14.038 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:35.545 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:18:35.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.683+0000 7f71a92de700 1 Processor -- start 2023-03-03T16:18:35.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.683+0000 7f71a92de700 1 -- start start 2023-03-03T16:18:35.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a92de700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f71a40caa30 0x7f71a40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:35.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a92de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71a40c00e0 0x7f71a4154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:35.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a92de700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f71a40cae00 0x7f71a40c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:35.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a92de700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a40cb280 con 0x7f71a40caa30 2023-03-03T16:18:35.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a92de700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a40bfd10 con 0x7f71a40cae00 2023-03-03T16:18:35.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a92de700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40bfa30 con 0x7f71a40c00e0 2023-03-03T16:18:35.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a2ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f71a40caa30 0x7f71a40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:35.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a2ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f71a40caa30 0x7f71a40c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56578/0 (socket says 172.21.15.7:56578) 2023-03-03T16:18:35.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a2ffd700 1 -- 172.21.15.7:0/3477819777 learned_addr learned my addr 172.21.15.7:0/3477819777 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:18:35.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a2ffd700 1 -- 172.21.15.7:0/3477819777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71a40c00e0 msgr2=0x7f71a4154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:18:35.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a2ffd700 1 --2- 172.21.15.7:0/3477819777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71a40c00e0 0x7f71a4154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a2ffd700 1 -- 172.21.15.7:0/3477819777 >> v2:172.21.15.143:3300/0 conn(0x7f71a40cae00 msgr2=0x7f71a40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:18:35.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a2ffd700 1 --2- 172.21.15.7:0/3477819777 >> v2:172.21.15.143:3300/0 conn(0x7f71a40cae00 0x7f71a40c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a2ffd700 1 -- 172.21.15.7:0/3477819777 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a4154510 con 0x7f71a40caa30 2023-03-03T16:18:35.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a2ffd700 1 --2- 172.21.15.7:0/3477819777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 0x7f71a40c5c60 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f71a4158a70 tx=0x7f7198009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773244e82dcca738 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:18:35.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.684+0000 7f71a1ffb700 1 -- 172.21.15.7:0/3477819777 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719800b870 con 0x7f71a40caa30 2023-03-03T16:18:35.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a1ffb700 1 -- 172.21.15.7:0/3477819777 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719800b9d0 con 0x7f71a40caa30 2023-03-03T16:18:35.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a1ffb700 1 -- 172.21.15.7:0/3477819777 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719800bd60 con 0x7f71a40caa30 2023-03-03T16:18:35.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a92de700 1 -- 172.21.15.7:0/3477819777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 msgr2=0x7f71a40c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:35.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a92de700 1 --2- 172.21.15.7:0/3477819777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 0x7f71a40c5c60 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f71a4158a70 tx=0x7f7198009bd0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a92de700 1 -- 172.21.15.7:0/3477819777 shutdown_connections 2023-03-03T16:18:35.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a92de700 1 --2- 172.21.15.7:0/3477819777 >> v2:172.21.15.143:3300/0 conn(0x7f71a40cae00 0x7f71a40c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a92de700 1 --2- 172.21.15.7:0/3477819777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 0x7f71a40c5c60 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a92de700 1 --2- 172.21.15.7:0/3477819777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71a40c00e0 0x7f71a4154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a92de700 1 -- 172.21.15.7:0/3477819777 >> 172.21.15.7:0/3477819777 conn(0x7f71a4066e40 msgr2=0x7f71a4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:18:35.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a92de700 1 -- 172.21.15.7:0/3477819777 shutdown_connections 2023-03-03T16:18:35.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.685+0000 7f71a92de700 1 -- 172.21.15.7:0/3477819777 wait complete. 2023-03-03T16:18:35.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.686+0000 7f71a92de700 1 Processor -- start 2023-03-03T16:18:35.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a92de700 1 -- start start 2023-03-03T16:18:35.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a92de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71a40c00e0 0x7f71a4156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:35.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a92de700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 0x7f71a40c3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:35.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a92de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71a40cae00 0x7f71a4158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:35.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a92de700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a4157d60 con 0x7f71a40caa30 2023-03-03T16:18:35.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a92de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a4156980 con 0x7f71a40cae00 2023-03-03T16:18:35.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a92de700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a41ea930 con 0x7f71a40c00e0 2023-03-03T16:18:35.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 0x7f71a40c3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:35.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 0x7f71a40c3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56586/0 (socket says 172.21.15.7:56586) 2023-03-03T16:18:35.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a27fc700 1 -- 172.21.15.7:0/635135026 learned_addr learned my addr 172.21.15.7:0/635135026 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:18:35.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a2ffd700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71a40c00e0 0x7f71a4156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:35.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.687+0000 7f71a37fe700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71a40cae00 0x7f71a4158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:35.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f71a27fc700 1 -- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71a40c00e0 msgr2=0x7f71a4156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:35.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f71a27fc700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71a40c00e0 0x7f71a4156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f71a27fc700 1 -- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71a40cae00 msgr2=0x7f71a4158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:35.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f71a27fc700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71a40cae00 0x7f71a4158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f71a27fc700 1 -- 172.21.15.7:0/635135026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7198009550 con 0x7f71a40caa30 2023-03-03T16:18:35.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f71a37fe700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71a40cae00 0x7f71a4158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:18:35.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f71a27fc700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 0x7f71a40c3fb0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f719400b480 tx=0x7f719400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:18:35.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f718b7fe700 1 -- 172.21.15.7:0/635135026 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194005a00 con 0x7f71a40caa30 2023-03-03T16:18:35.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f718b7fe700 1 -- 172.21.15.7:0/635135026 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7194005b60 con 0x7f71a40caa30 2023-03-03T16:18:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f71a92de700 1 -- 172.21.15.7:0/635135026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a4154ad0 con 0x7f71a40caa30 2023-03-03T16:18:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f718b7fe700 1 -- 172.21.15.7:0/635135026 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194011630 con 0x7f71a40caa30 2023-03-03T16:18:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.688+0000 7f71a92de700 1 -- 172.21.15.7:0/635135026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a41547f0 con 0x7f71a40caa30 2023-03-03T16:18:35.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.689+0000 7f71a92de700 1 -- 172.21.15.7:0/635135026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a40bf650 con 0x7f71a40caa30 2023-03-03T16:18:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.689+0000 7f718b7fe700 1 -- 172.21.15.7:0/635135026 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f719400f050 con 0x7f71a40caa30 2023-03-03T16:18:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.690+0000 7f718b7fe700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71900682f0 0x7f719006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:18:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.690+0000 7f718b7fe700 1 -- 172.21.15.7:0/635135026 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7194085b10 con 0x7f71a40caa30 2023-03-03T16:18:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.690+0000 7f71a2ffd700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71900682f0 0x7f719006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:18:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.691+0000 7f71a2ffd700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71900682f0 0x7f719006a7a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7198000c00 tx=0x7f7198008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:18:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.695+0000 7f718b7fe700 1 -- 172.21.15.7:0/635135026 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7194051f70 con 0x7f71a40caa30 2023-03-03T16:18:35.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.936+0000 7f71a92de700 1 -- 172.21.15.7:0/635135026 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71a4074bb0 con 0x7f71a40caa30 2023-03-03T16:18:35.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.936+0000 7f718b7fe700 1 -- 172.21.15.7:0/635135026 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7194055590 con 0x7f71a40caa30 2023-03-03T16:18:35.937 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:18:35.937 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:18:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.938+0000 7f71a92de700 1 -- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71900682f0 msgr2=0x7f719006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.938+0000 7f71a92de700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71900682f0 0x7f719006a7a0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7198000c00 tx=0x7f7198008030 comp rx=0 tx=0).stop 2023-03-03T16:18:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.938+0000 7f71a92de700 1 -- 172.21.15.7:0/635135026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 msgr2=0x7f71a40c3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:18:35.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.938+0000 7f71a92de700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 0x7f71a40c3fb0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f719400b480 tx=0x7f719400b790 comp rx=0 tx=0).stop 2023-03-03T16:18:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.939+0000 7f71a92de700 1 -- 172.21.15.7:0/635135026 shutdown_connections 2023-03-03T16:18:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.939+0000 7f71a92de700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71a40cae00 0x7f71a4158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.939+0000 7f71a92de700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f71a40c00e0 0x7f71a4156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.939+0000 7f71a92de700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f71900682f0 0x7f719006a7a0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.939+0000 7f71a92de700 1 --2- 172.21.15.7:0/635135026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71a40caa30 0x7f71a40c3fb0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:18:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.939+0000 7f71a92de700 1 -- 172.21.15.7:0/635135026 >> 172.21.15.7:0/635135026 conn(0x7f71a4066e40 msgr2=0x7f71a4073f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:18:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.939+0000 7f71a92de700 1 -- 172.21.15.7:0/635135026 shutdown_connections 2023-03-03T16:18:35.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:18:35.939+0000 7f71a92de700 1 -- 172.21.15.7:0/635135026 wait complete. 2023-03-03T16:18:35.941 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:18:35.951 INFO:tasks.mds_thrash.fs.[a]:waiting for 91 secs before thrashing 2023-03-03T16:18:44.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:18:44.185 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:03.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:19:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.306+0000 7f12d9588700 1 Processor -- start 2023-03-03T16:19:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.306+0000 7f12d9588700 1 -- start start 2023-03-03T16:19:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.306+0000 7f12d9588700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f12d4151cc0 0x7f12d40ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:19:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.306+0000 7f12d9588700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12d40cac10 0x7f12d40c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:19:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.306+0000 7f12d9588700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d40c7ca0 con 0x7f12d40b00b0 2023-03-03T16:19:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.306+0000 7f12d9588700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d4152090 con 0x7f12d40cac10 2023-03-03T16:19:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.306+0000 7f12d9588700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d40c6cf0 con 0x7f12d4151cc0 2023-03-03T16:19:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12d40cac10 0x7f12d40c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:19:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d27fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f12d40b00b0 0x7f12d40c7050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50596/0 (socket says 172.21.15.7:50596) 2023-03-03T16:19:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d27fc700 1 -- 172.21.15.7:0/2574120458 learned_addr learned my addr 172.21.15.7:0/2574120458 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:19:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d2ffd700 1 --2- 172.21.15.7:0/2574120458 >> v2:172.21.15.143:3300/0 conn(0x7f12d4151cc0 0x7f12d40ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:19:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d37fe700 1 -- 172.21.15.7:0/2574120458 >> v2:172.21.15.143:3300/0 conn(0x7f12d4151cc0 msgr2=0x7f12d40ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:19:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d37fe700 1 --2- 172.21.15.7:0/2574120458 >> v2:172.21.15.143:3300/0 conn(0x7f12d4151cc0 0x7f12d40ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:19:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d37fe700 1 -- 172.21.15.7:0/2574120458 >> v1:172.21.15.7:6789/0 conn(0x7f12d40b00b0 legacy=0x7f12d40c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:19:03.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d37fe700 1 -- 172.21.15.7:0/2574120458 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d41548f0 con 0x7f12d40cac10 2023-03-03T16:19:03.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d1ffb700 1 -- 172.21.15.7:0/2574120458 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107664205 0 0) 0x7f12d40c7ca0 con 0x7f12d40b00b0 2023-03-03T16:19:03.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d37fe700 1 --2- 172.21.15.7:0/2574120458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40cac10 0x7f12d40c7760 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f12d40cb350 tx=0x7f12cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33256d6befd09cfb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:19:03.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d1ffb700 1 -- 172.21.15.7:0/2574120458 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc00e070 con 0x7f12d40cac10 2023-03-03T16:19:03.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d1ffb700 1 -- 172.21.15.7:0/2574120458 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12cc00bbd0 con 0x7f12d40cac10 2023-03-03T16:19:03.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.307+0000 7f12d1ffb700 1 -- 172.21.15.7:0/2574120458 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc012430 con 0x7f12d40cac10 2023-03-03T16:19:03.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.308+0000 7f12d9588700 1 -- 172.21.15.7:0/2574120458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40cac10 msgr2=0x7f12d40c7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:19:03.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.308+0000 7f12d9588700 1 --2- 172.21.15.7:0/2574120458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40cac10 0x7f12d40c7760 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f12d40cb350 tx=0x7f12cc00b330 comp rx=0 tx=0).stop 2023-03-03T16:19:03.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.308+0000 7f12d9588700 1 -- 172.21.15.7:0/2574120458 shutdown_connections 2023-03-03T16:19:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.308+0000 7f12d9588700 1 --2- 172.21.15.7:0/2574120458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40cac10 0x7f12d40c7760 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:19:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.308+0000 7f12d9588700 1 --2- 172.21.15.7:0/2574120458 >> v2:172.21.15.143:3300/0 conn(0x7f12d4151cc0 0x7f12d40ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:19:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.308+0000 7f12d9588700 1 -- 172.21.15.7:0/2574120458 >> 172.21.15.7:0/2574120458 conn(0x7f12d406b980 msgr2=0x7f12d4071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:19:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.308+0000 7f12d9588700 1 -- 172.21.15.7:0/2574120458 shutdown_connections 2023-03-03T16:19:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.308+0000 7f12d9588700 1 -- 172.21.15.7:0/2574120458 wait complete. 2023-03-03T16:19:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.309+0000 7f12d9588700 1 Processor -- start 2023-03-03T16:19:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.309+0000 7f12d9588700 1 -- start start 2023-03-03T16:19:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d9588700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40b00b0 0x7f12d414f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:19:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d9588700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d40cac10 0x7f12d41452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:19:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d9588700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d4151cc0 0x7f12d4146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:19:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d9588700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d4145dc0 con 0x7f12d40b00b0 2023-03-03T16:19:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d9588700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d41ea7b0 con 0x7f12d40cac10 2023-03-03T16:19:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d9588700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d41eaa90 con 0x7f12d4151cc0 2023-03-03T16:19:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40b00b0 0x7f12d414f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:19:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40b00b0 0x7f12d414f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56620/0 (socket says 172.21.15.7:56620) 2023-03-03T16:19:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d2ffd700 1 -- 172.21.15.7:0/3226131536 learned_addr learned my addr 172.21.15.7:0/3226131536 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:19:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d37fe700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d4151cc0 0x7f12d4146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:19:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d27fc700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d40cac10 0x7f12d41452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:19:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d2ffd700 1 -- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d4151cc0 msgr2=0x7f12d4146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:19:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d2ffd700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d4151cc0 0x7f12d4146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:19:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d2ffd700 1 -- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d40cac10 msgr2=0x7f12d41452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:19:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d2ffd700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d40cac10 0x7f12d41452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:19:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d2ffd700 1 -- 172.21.15.7:0/3226131536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12cc009a70 con 0x7f12d40b00b0 2023-03-03T16:19:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.310+0000 7f12d27fc700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d40cac10 0x7f12d41452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:19:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.311+0000 7f12d2ffd700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40b00b0 0x7f12d414f4e0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f12c400c7f0 tx=0x7f12c400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:19:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.311+0000 7f12bb7fe700 1 -- 172.21.15.7:0/3226131536 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4011800 con 0x7f12d40b00b0 2023-03-03T16:19:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.311+0000 7f12bb7fe700 1 -- 172.21.15.7:0/3226131536 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c4011960 con 0x7f12d40b00b0 2023-03-03T16:19:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.311+0000 7f12d9588700 1 -- 172.21.15.7:0/3226131536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d4154eb0 con 0x7f12d40b00b0 2023-03-03T16:19:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.311+0000 7f12bb7fe700 1 -- 172.21.15.7:0/3226131536 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4011cf0 con 0x7f12d40b00b0 2023-03-03T16:19:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.311+0000 7f12d9588700 1 -- 172.21.15.7:0/3226131536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d4154bd0 con 0x7f12d40b00b0 2023-03-03T16:19:03.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.312+0000 7f12d9588700 1 -- 172.21.15.7:0/3226131536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d4155190 con 0x7f12d40b00b0 2023-03-03T16:19:03.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.312+0000 7f12bb7fe700 1 -- 172.21.15.7:0/3226131536 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f12c4021070 con 0x7f12d40b00b0 2023-03-03T16:19:03.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.313+0000 7f12bb7fe700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12c00682f0 0x7f12c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:19:03.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.313+0000 7f12bb7fe700 1 -- 172.21.15.7:0/3226131536 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f12c4085f60 con 0x7f12d40b00b0 2023-03-03T16:19:03.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.313+0000 7f12d27fc700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12c00682f0 0x7f12c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:19:03.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.314+0000 7f12d27fc700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12c00682f0 0x7f12c006a7a0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f12d41510b0 tx=0x7f12c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:19:03.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.317+0000 7f12bb7fe700 1 -- 172.21.15.7:0/3226131536 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f12c4052300 con 0x7f12d40b00b0 2023-03-03T16:19:03.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.555+0000 7f12d9588700 1 -- 172.21.15.7:0/3226131536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12d41eac50 con 0x7f12d40b00b0 2023-03-03T16:19:03.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.555+0000 7f12bb7fe700 1 -- 172.21.15.7:0/3226131536 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f12d41eac50 con 0x7f12d40b00b0 2023-03-03T16:19:03.556 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:19:03.556 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:19:03.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 -- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12c00682f0 msgr2=0x7f12c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:19:03.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12c00682f0 0x7f12c006a7a0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f12d41510b0 tx=0x7f12c8009040 comp rx=0 tx=0).stop 2023-03-03T16:19:03.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 -- 172.21.15.7:0/3226131536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40b00b0 msgr2=0x7f12d414f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:19:03.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40b00b0 0x7f12d414f4e0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f12c400c7f0 tx=0x7f12c400cbb0 comp rx=0 tx=0).stop 2023-03-03T16:19:03.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 -- 172.21.15.7:0/3226131536 shutdown_connections 2023-03-03T16:19:03.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12d4151cc0 0x7f12d4146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:19:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f12c00682f0 0x7f12c006a7a0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:19:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12d40b00b0 0x7f12d414f4e0 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:19:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 --2- 172.21.15.7:0/3226131536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12d40cac10 0x7f12d41452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:19:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 -- 172.21.15.7:0/3226131536 >> 172.21.15.7:0/3226131536 conn(0x7f12d406b980 msgr2=0x7f12d40c6270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:19:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 -- 172.21.15.7:0/3226131536 shutdown_connections 2023-03-03T16:19:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:19:03.557+0000 7f12d9588700 1 -- 172.21.15.7:0/3226131536 wait complete. 2023-03-03T16:19:03.560 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:19:03.570 INFO:tasks.mds_thrash.fs.[b]:waiting for 115 secs before thrashing 2023-03-03T16:19:14.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:14.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:44.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:44.462 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:19:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T16:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T16:20:06.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:20:07.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.091+0000 7fb7712ab700 1 Processor -- start 2023-03-03T16:20:07.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.091+0000 7fb7712ab700 1 -- start start 2023-03-03T16:20:07.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb7712ab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0c00e0 0x7fb76c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:07.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb7712ab700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c0c7310 con 0x7fb76c0caa30 2023-03-03T16:20:07.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb7712ab700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c0bfd10 con 0x7fb76c154ff0 2023-03-03T16:20:07.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb7712ab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c0bfa30 con 0x7fb76c0c00e0 2023-03-03T16:20:07.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb76affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb76c0caa30 0x7fb76c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50668/0 (socket says 172.21.15.7:50668) 2023-03-03T16:20:07.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb76affd700 1 -- 172.21.15.7:0/885944176 learned_addr learned my addr 172.21.15.7:0/885944176 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:20:07.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb76a7fc700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0c00e0 0x7fb76c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:07.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266822602 0 0) 0x7fb76c0c7310 con 0x7fb76c0caa30 2023-03-03T16:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7540035b0 con 0x7fb76c0caa30 2023-03-03T16:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088664013 0 0) 0x7fb76c0bfd10 con 0x7fb76c154ff0 2023-03-03T16:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb76c0c7310 con 0x7fb76c154ff0 2023-03-03T16:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4160229401 0 0) 0x7fb7540035b0 con 0x7fb76c0caa30 2023-03-03T16:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0c00e0 msgr2=0x7fb76c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0c00e0 0x7fb76c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 >> v1:172.21.15.143:6789/0 conn(0x7fb76c154ff0 legacy=0x7fb76c15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:07.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c154510 con 0x7fb76c0caa30 2023-03-03T16:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb75c001670 con 0x7fb76c0caa30 2023-03-03T16:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb76a7fc700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0c00e0 0x7fb76c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.092+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 >> v1:172.21.15.7:6789/0 conn(0x7fb76c0caa30 legacy=0x7fb76c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb769ffb700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0c00e0 0x7fb754003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb769ffb700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb754004170 0x7fb754004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb769ffb700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb754004d90 0x7fb754005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:07.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c0bfa30 con 0x7fb754004d90 2023-03-03T16:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75400bba0 con 0x7fb754004170 2023-03-03T16:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75400bd60 con 0x7fb76c0c00e0 2023-03-03T16:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb76b7fe700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb754004d90 0x7fb754005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb76affd700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0c00e0 0x7fb754003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb76a7fc700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb754004170 0x7fb754004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb76b7fe700 1 -- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0c00e0 msgr2=0x7fb754003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:20:07.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb76b7fe700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c0c00e0 0x7fb754003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb76b7fe700 1 -- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb754004170 msgr2=0x7fb754004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb76b7fe700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb754004170 0x7fb754004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb76b7fe700 1 -- 172.21.15.7:0/885944176 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb760009430 con 0x7fb754004d90 2023-03-03T16:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb76b7fe700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb754004d90 0x7fb754005200 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7fb76400c2f0 tx=0x7fb754005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303f198f3c52451f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764001f30 con 0x7fb754004d90 2023-03-03T16:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76400ce90 con 0x7fb754004d90 2023-03-03T16:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.093+0000 7fb769ffb700 1 -- 172.21.15.7:0/885944176 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764003c40 con 0x7fb754004d90 2023-03-03T16:20:07.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.094+0000 7fb7712ab700 1 -- 172.21.15.7:0/885944176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb754004d90 msgr2=0x7fb754005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.094+0000 7fb7712ab700 1 --2- 172.21.15.7:0/885944176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb754004d90 0x7fb754005200 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7fb76400c2f0 tx=0x7fb754005880 comp rx=0 tx=0).stop 2023-03-03T16:20:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.094+0000 7fb76b7fe700 1 -- 172.21.15.7:0/885944176 reap_dead start 2023-03-03T16:20:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.094+0000 7fb7712ab700 1 -- 172.21.15.7:0/885944176 shutdown_connections 2023-03-03T16:20:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.094+0000 7fb7712ab700 1 -- 172.21.15.7:0/885944176 >> 172.21.15.7:0/885944176 conn(0x7fb76c066e40 msgr2=0x7fb76c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:20:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.094+0000 7fb7712ab700 1 -- 172.21.15.7:0/885944176 shutdown_connections 2023-03-03T16:20:07.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.094+0000 7fb7712ab700 1 -- 172.21.15.7:0/885944176 wait complete. 2023-03-03T16:20:07.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.095+0000 7fb7712ab700 1 Processor -- start 2023-03-03T16:20:07.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.095+0000 7fb7712ab700 1 -- start start 2023-03-03T16:20:07.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb7712ab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c1ddcb0 0x7fb76c1ea9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:07.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb7712ab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb76c1de080 0x7fb76c13ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:07.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb7712ab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb76c13f4b0 0x7fb76c157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:07.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb7712ab700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c1de7d0 con 0x7fb76c13f4b0 2023-03-03T16:20:07.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb7712ab700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c0c31d0 con 0x7fb76c1de080 2023-03-03T16:20:07.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb7712ab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c157900 con 0x7fb76c1ddcb0 2023-03-03T16:20:07.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb76b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb76c13f4b0 0x7fb76c157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb76affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c1ddcb0 0x7fb76c1ea9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb76b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb76c13f4b0 0x7fb76c157d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56704/0 (socket says 172.21.15.7:56704) 2023-03-03T16:20:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb76b7fe700 1 -- 172.21.15.7:0/1713021726 learned_addr learned my addr 172.21.15.7:0/1713021726 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:20:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb76a7fc700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb76c1de080 0x7fb76c13ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb76b7fe700 1 -- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c1ddcb0 msgr2=0x7fb76c1ea9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb76b7fe700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c1ddcb0 0x7fb76c1ea9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb76b7fe700 1 -- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb76c1de080 msgr2=0x7fb76c13ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:07.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb76b7fe700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb76c1de080 0x7fb76c13ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.096+0000 7fb76b7fe700 1 -- 172.21.15.7:0/1713021726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c154ad0 con 0x7fb76c13f4b0 2023-03-03T16:20:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.097+0000 7fb76b7fe700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb76c13f4b0 0x7fb76c157d90 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7fb764015040 tx=0x7fb764001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.097+0000 7fb7537fe700 1 -- 172.21.15.7:0/1713021726 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7640050d0 con 0x7fb76c13f4b0 2023-03-03T16:20:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.097+0000 7fb7537fe700 1 -- 172.21.15.7:0/1713021726 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb764005250 con 0x7fb76c13f4b0 2023-03-03T16:20:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.097+0000 7fb7712ab700 1 -- 172.21.15.7:0/1713021726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c1547f0 con 0x7fb76c13f4b0 2023-03-03T16:20:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.097+0000 7fb7537fe700 1 -- 172.21.15.7:0/1713021726 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764005600 con 0x7fb76c13f4b0 2023-03-03T16:20:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.097+0000 7fb7712ab700 1 -- 172.21.15.7:0/1713021726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76c0bf650 con 0x7fb76c13f4b0 2023-03-03T16:20:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.098+0000 7fb7712ab700 1 -- 172.21.15.7:0/1713021726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76c074bb0 con 0x7fb76c13f4b0 2023-03-03T16:20:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.099+0000 7fb7537fe700 1 -- 172.21.15.7:0/1713021726 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb764010050 con 0x7fb76c13f4b0 2023-03-03T16:20:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.099+0000 7fb7537fe700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7580682f0 0x7fb75806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.099+0000 7fb7537fe700 1 -- 172.21.15.7:0/1713021726 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb7640879e0 con 0x7fb76c13f4b0 2023-03-03T16:20:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.100+0000 7fb76affd700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7580682f0 0x7fb75806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.100+0000 7fb76affd700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7580682f0 0x7fb75806a7a0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fb75c001eb0 tx=0x7fb75c001e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:07.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.103+0000 7fb7537fe700 1 -- 172.21.15.7:0/1713021726 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb764053d10 con 0x7fb76c13f4b0 2023-03-03T16:20:07.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.341+0000 7fb7712ab700 1 -- 172.21.15.7:0/1713021726 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb76c157a40 con 0x7fb76c13f4b0 2023-03-03T16:20:07.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.342+0000 7fb7537fe700 1 -- 172.21.15.7:0/1713021726 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb764002090 con 0x7fb76c13f4b0 2023-03-03T16:20:07.342 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:20:07.342 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:20:07.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 -- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7580682f0 msgr2=0x7fb75806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:07.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7580682f0 0x7fb75806a7a0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fb75c001eb0 tx=0x7fb75c001e00 comp rx=0 tx=0).stop 2023-03-03T16:20:07.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 -- 172.21.15.7:0/1713021726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb76c13f4b0 msgr2=0x7fb76c157d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:07.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb76c13f4b0 0x7fb76c157d90 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7fb764015040 tx=0x7fb764001650 comp rx=0 tx=0).stop 2023-03-03T16:20:07.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 -- 172.21.15.7:0/1713021726 shutdown_connections 2023-03-03T16:20:07.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7580682f0 0x7fb75806a7a0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:07.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb76c13f4b0 0x7fb76c157d90 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:07.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb76c1ddcb0 0x7fb76c1ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:07.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 --2- 172.21.15.7:0/1713021726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb76c1de080 0x7fb76c13ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:07.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 -- 172.21.15.7:0/1713021726 >> 172.21.15.7:0/1713021726 conn(0x7fb76c066e40 msgr2=0x7fb76c065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:20:07.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 -- 172.21.15.7:0/1713021726 shutdown_connections 2023-03-03T16:20:07.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:07.344+0000 7fb7712ab700 1 -- 172.21.15.7:0/1713021726 wait complete. 2023-03-03T16:20:07.346 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:20:07.356 INFO:tasks.mds_thrash.fs.[a]:waiting for 37 secs before thrashing 2023-03-03T16:20:14.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:14.524 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:44.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:20:44.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.496+0000 7f35f337b700 1 Processor -- start 2023-03-03T16:20:44.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.497+0000 7f35f337b700 1 -- start start 2023-03-03T16:20:44.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.497+0000 7f35f337b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f35ec0c2930 0x7f35ec157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:44.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.497+0000 7f35f337b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f35ec06c480 0x7f35ec157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:44.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.497+0000 7f35f337b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec06d430 con 0x7f35ec151d40 2023-03-03T16:20:44.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.497+0000 7f35f337b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec06c2a0 con 0x7f35ec06c480 2023-03-03T16:20:44.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.497+0000 7f35f337b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec06cb00 con 0x7f35ec0c2930 2023-03-03T16:20:44.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35f1918700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f35ec06c480 0x7f35ec157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:44.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35f1117700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f35ec151d40 0x7f35ec0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50706/0 (socket says 172.21.15.7:50706) 2023-03-03T16:20:44.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35f1117700 1 -- 172.21.15.7:0/4097835929 learned_addr learned my addr 172.21.15.7:0/4097835929 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:20:44.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35f0916700 1 --2- 172.21.15.7:0/4097835929 >> v2:172.21.15.143:3300/0 conn(0x7f35ec0c2930 0x7f35ec157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:44.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35f1918700 1 -- 172.21.15.7:0/4097835929 >> v2:172.21.15.143:3300/0 conn(0x7f35ec0c2930 msgr2=0x7f35ec157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:20:44.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35f1918700 1 --2- 172.21.15.7:0/4097835929 >> v2:172.21.15.143:3300/0 conn(0x7f35ec0c2930 0x7f35ec157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35f1918700 1 -- 172.21.15.7:0/4097835929 >> v1:172.21.15.7:6789/0 conn(0x7f35ec151d40 legacy=0x7f35ec0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:44.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35f1918700 1 -- 172.21.15.7:0/4097835929 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec06cde0 con 0x7f35ec06c480 2023-03-03T16:20:44.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35dffff700 1 -- 172.21.15.7:0/4097835929 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397234797 0 0) 0x7f35ec06d430 con 0x7f35ec151d40 2023-03-03T16:20:44.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35f1918700 1 --2- 172.21.15.7:0/4097835929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec06c480 0x7f35ec157fd0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009d90 tx=0x7f35e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecf0ababa59ee08 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:44.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35dffff700 1 -- 172.21.15.7:0/4097835929 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e400e070 con 0x7f35ec06c480 2023-03-03T16:20:44.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35dffff700 1 -- 172.21.15.7:0/4097835929 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e400bae0 con 0x7f35ec06c480 2023-03-03T16:20:44.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.498+0000 7f35dffff700 1 -- 172.21.15.7:0/4097835929 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e400be90 con 0x7f35ec06c480 2023-03-03T16:20:44.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.499+0000 7f35f337b700 1 -- 172.21.15.7:0/4097835929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec06c480 msgr2=0x7f35ec157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:44.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.499+0000 7f35f337b700 1 --2- 172.21.15.7:0/4097835929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec06c480 0x7f35ec157fd0 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009d90 tx=0x7f35e400b330 comp rx=0 tx=0).stop 2023-03-03T16:20:44.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.499+0000 7f35f337b700 1 -- 172.21.15.7:0/4097835929 shutdown_connections 2023-03-03T16:20:44.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.499+0000 7f35f337b700 1 --2- 172.21.15.7:0/4097835929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec06c480 0x7f35ec157fd0 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.499+0000 7f35f337b700 1 --2- 172.21.15.7:0/4097835929 >> v2:172.21.15.143:3300/0 conn(0x7f35ec0c2930 0x7f35ec157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.499+0000 7f35f337b700 1 -- 172.21.15.7:0/4097835929 >> 172.21.15.7:0/4097835929 conn(0x7f35ec0a94d0 msgr2=0x7f35ec0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:20:44.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.499+0000 7f35f337b700 1 -- 172.21.15.7:0/4097835929 shutdown_connections 2023-03-03T16:20:44.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.499+0000 7f35f337b700 1 -- 172.21.15.7:0/4097835929 wait complete. 2023-03-03T16:20:44.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.500+0000 7f35f337b700 1 Processor -- start 2023-03-03T16:20:44.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.500+0000 7f35f337b700 1 -- start start 2023-03-03T16:20:44.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f337b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ec06c480 0x7f35ec0ace50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:44.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f337b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ec0c2930 0x7f35ec0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:44.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f337b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec151d40 0x7f35ec14ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:44.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f337b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec1554f0 con 0x7f35ec151d40 2023-03-03T16:20:44.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f337b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec0aabe0 con 0x7f35ec0c2930 2023-03-03T16:20:44.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f337b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec0aa970 con 0x7f35ec06c480 2023-03-03T16:20:44.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f1918700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec151d40 0x7f35ec14ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:44.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f1918700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec151d40 0x7f35ec14ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56736/0 (socket says 172.21.15.7:56736) 2023-03-03T16:20:44.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f1918700 1 -- 172.21.15.7:0/359944756 learned_addr learned my addr 172.21.15.7:0/359944756 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:20:44.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f0916700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ec0c2930 0x7f35ec0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:44.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f1117700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ec06c480 0x7f35ec0ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:44.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f1918700 1 -- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ec06c480 msgr2=0x7f35ec0ace50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:44.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f1918700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ec06c480 0x7f35ec0ace50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f1918700 1 -- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ec0c2930 msgr2=0x7f35ec0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:44.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f1918700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ec0c2930 0x7f35ec0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.501+0000 7f35f1918700 1 -- 172.21.15.7:0/359944756 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e4009a70 con 0x7f35ec151d40 2023-03-03T16:20:44.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.502+0000 7f35f1918700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec151d40 0x7f35ec14ba70 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f35ec06bb70 tx=0x7f35e4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:44.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.502+0000 7f35ddffb700 1 -- 172.21.15.7:0/359944756 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e40044d0 con 0x7f35ec151d40 2023-03-03T16:20:44.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.502+0000 7f35ddffb700 1 -- 172.21.15.7:0/359944756 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e4020070 con 0x7f35ec151d40 2023-03-03T16:20:44.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.502+0000 7f35f337b700 1 -- 172.21.15.7:0/359944756 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ec073e00 con 0x7f35ec151d40 2023-03-03T16:20:44.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.502+0000 7f35ddffb700 1 -- 172.21.15.7:0/359944756 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4004870 con 0x7f35ec151d40 2023-03-03T16:20:44.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.502+0000 7f35f337b700 1 -- 172.21.15.7:0/359944756 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ec06d0c0 con 0x7f35ec151d40 2023-03-03T16:20:44.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.503+0000 7f35f337b700 1 -- 172.21.15.7:0/359944756 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ec06bea0 con 0x7f35ec151d40 2023-03-03T16:20:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.504+0000 7f35ddffb700 1 -- 172.21.15.7:0/359944756 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f35e4012430 con 0x7f35ec151d40 2023-03-03T16:20:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.504+0000 7f35ddffb700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35d00682f0 0x7f35d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.504+0000 7f35ddffb700 1 -- 172.21.15.7:0/359944756 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f35e4022070 con 0x7f35ec151d40 2023-03-03T16:20:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.504+0000 7f35f1117700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35d00682f0 0x7f35d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:44.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.505+0000 7f35f1117700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35d00682f0 0x7f35d006a7a0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f35e0007fd0 tx=0x7f35e0007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:44.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.508+0000 7f35ddffb700 1 -- 172.21.15.7:0/359944756 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f35e4054740 con 0x7f35ec151d40 2023-03-03T16:20:44.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.746+0000 7f35f337b700 1 -- 172.21.15.7:0/359944756 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35ec0aa5b0 con 0x7f35ec151d40 2023-03-03T16:20:44.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.747+0000 7f35ddffb700 1 -- 172.21.15.7:0/359944756 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f35ec0aa5b0 con 0x7f35ec151d40 2023-03-03T16:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:20:44.747 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:20:44.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 -- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35d00682f0 msgr2=0x7f35d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:44.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35d00682f0 0x7f35d006a7a0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f35e0007fd0 tx=0x7f35e0007de0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 -- 172.21.15.7:0/359944756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec151d40 msgr2=0x7f35ec14ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:44.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec151d40 0x7f35ec14ba70 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f35ec06bb70 tx=0x7f35e4004ad0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 -- 172.21.15.7:0/359944756 shutdown_connections 2023-03-03T16:20:44.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35d00682f0 0x7f35d006a7a0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ec151d40 0x7f35ec14ba70 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ec06c480 0x7f35ec0ace50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 --2- 172.21.15.7:0/359944756 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ec0c2930 0x7f35ec0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:44.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 -- 172.21.15.7:0/359944756 >> 172.21.15.7:0/359944756 conn(0x7f35ec0a94d0 msgr2=0x7f35ec158160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:20:44.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 -- 172.21.15.7:0/359944756 shutdown_connections 2023-03-03T16:20:44.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:44.749+0000 7f35f337b700 1 -- 172.21.15.7:0/359944756 wait complete. 2023-03-03T16:20:44.751 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:20:44.761 INFO:tasks.mds_thrash.fs.[a]:waiting for 63 secs before thrashing 2023-03-03T16:20:44.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:44.838 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:20:58.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:20:58.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.708+0000 7f2653016700 1 Processor -- start 2023-03-03T16:20:58.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.708+0000 7f2653016700 1 -- start start 2023-03-03T16:20:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f2653016700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f264c14fda0 0x7f264c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f2653016700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f264c0abe20 0x7f264c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f2653016700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264c0c2f30 con 0x7f264c14f9d0 2023-03-03T16:20:58.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f2653016700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264c0c31d0 con 0x7f264c14fda0 2023-03-03T16:20:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f2653016700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264c1524a0 con 0x7f264c0abe20 2023-03-03T16:20:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f264bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f264c14fda0 0x7f264c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f2650db2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f264c14f9d0 0x7f264c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50738/0 (socket says 172.21.15.7:50738) 2023-03-03T16:20:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f2650db2700 1 -- 172.21.15.7:0/3922820801 learned_addr learned my addr 172.21.15.7:0/3922820801 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:20:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f26515b3700 1 --2- 172.21.15.7:0/3922820801 >> v2:172.21.15.143:3300/0 conn(0x7f264c0abe20 0x7f264c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f264bfff700 1 -- 172.21.15.7:0/3922820801 >> v2:172.21.15.143:3300/0 conn(0x7f264c0abe20 msgr2=0x7f264c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:20:58.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f264bfff700 1 --2- 172.21.15.7:0/3922820801 >> v2:172.21.15.143:3300/0 conn(0x7f264c0abe20 0x7f264c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.709+0000 7f264bfff700 1 -- 172.21.15.7:0/3922820801 >> v1:172.21.15.7:6789/0 conn(0x7f264c14f9d0 legacy=0x7f264c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f264b7fe700 1 -- 172.21.15.7:0/3922820801 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464964245 0 0) 0x7f264c0c2f30 con 0x7f264c14f9d0 2023-03-03T16:20:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f264bfff700 1 -- 172.21.15.7:0/3922820801 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c0c34b0 con 0x7f264c14fda0 2023-03-03T16:20:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f264bfff700 1 --2- 172.21.15.7:0/3922820801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c14fda0 0x7f264c156430 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f264c071230 tx=0x7f2638009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaac3361f41604e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f264b7fe700 1 -- 172.21.15.7:0/3922820801 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638011830 con 0x7f264c14fda0 2023-03-03T16:20:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f264b7fe700 1 -- 172.21.15.7:0/3922820801 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2638011990 con 0x7f264c14fda0 2023-03-03T16:20:58.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f264b7fe700 1 -- 172.21.15.7:0/3922820801 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2638011d20 con 0x7f264c14fda0 2023-03-03T16:20:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f2653016700 1 -- 172.21.15.7:0/3922820801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c14fda0 msgr2=0x7f264c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f2653016700 1 --2- 172.21.15.7:0/3922820801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c14fda0 0x7f264c156430 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f264c071230 tx=0x7f2638009b20 comp rx=0 tx=0).stop 2023-03-03T16:20:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f2653016700 1 -- 172.21.15.7:0/3922820801 shutdown_connections 2023-03-03T16:20:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f2653016700 1 --2- 172.21.15.7:0/3922820801 >> v2:172.21.15.143:3300/0 conn(0x7f264c0abe20 0x7f264c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f2653016700 1 --2- 172.21.15.7:0/3922820801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c14fda0 0x7f264c156430 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f2653016700 1 -- 172.21.15.7:0/3922820801 >> 172.21.15.7:0/3922820801 conn(0x7f264c0c8900 msgr2=0x7f264c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:20:58.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f2653016700 1 -- 172.21.15.7:0/3922820801 shutdown_connections 2023-03-03T16:20:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.710+0000 7f2653016700 1 -- 172.21.15.7:0/3922820801 wait complete. 2023-03-03T16:20:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.712+0000 7f2653016700 1 Processor -- start 2023-03-03T16:20:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.712+0000 7f2653016700 1 -- start start 2023-03-03T16:20:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2653016700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0abe20 0x7f264c1e7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2653016700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f264c14f9d0 0x7f264c1549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2653016700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f264c14fda0 0x7f264c14acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:58.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2653016700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264c06e880 con 0x7f264c0abe20 2023-03-03T16:20:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2653016700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264c1e8410 con 0x7f264c14f9d0 2023-03-03T16:20:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2653016700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264c0bdb30 con 0x7f264c14fda0 2023-03-03T16:20:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2650db2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0abe20 0x7f264c1e7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2650db2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0abe20 0x7f264c1e7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56764/0 (socket says 172.21.15.7:56764) 2023-03-03T16:20:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2650db2700 1 -- 172.21.15.7:0/3719303680 learned_addr learned my addr 172.21.15.7:0/3719303680 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:20:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f26515b3700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f264c14fda0 0x7f264c14acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:58.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f264bfff700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f264c14f9d0 0x7f264c1549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2650db2700 1 -- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f264c14fda0 msgr2=0x7f264c14acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2650db2700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f264c14fda0 0x7f264c14acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2650db2700 1 -- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f264c14f9d0 msgr2=0x7f264c1549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2650db2700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f264c14f9d0 0x7f264c1549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2650db2700 1 -- 172.21.15.7:0/3719303680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26380094a0 con 0x7f264c0abe20 2023-03-03T16:20:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2650db2700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0abe20 0x7f264c1e7e30 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f26400011e0 tx=0x7f264000cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f26497fa700 1 -- 172.21.15.7:0/3719303680 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2640004c80 con 0x7f264c0abe20 2023-03-03T16:20:58.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f26497fa700 1 -- 172.21.15.7:0/3719303680 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2640015070 con 0x7f264c0abe20 2023-03-03T16:20:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2653016700 1 -- 172.21.15.7:0/3719303680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264c0ac350 con 0x7f264c0abe20 2023-03-03T16:20:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f26497fa700 1 -- 172.21.15.7:0/3719303680 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26400070d0 con 0x7f264c0abe20 2023-03-03T16:20:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.713+0000 7f2653016700 1 -- 172.21.15.7:0/3719303680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264c0ab630 con 0x7f264c0abe20 2023-03-03T16:20:58.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.714+0000 7f2653016700 1 -- 172.21.15.7:0/3719303680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264c071fa0 con 0x7f264c0abe20 2023-03-03T16:20:58.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.715+0000 7f26497fa700 1 -- 172.21.15.7:0/3719303680 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2640021070 con 0x7f264c0abe20 2023-03-03T16:20:58.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.715+0000 7f26497fa700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26300682f0 0x7f263006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:20:58.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.715+0000 7f26497fa700 1 -- 172.21.15.7:0/3719303680 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2640089680 con 0x7f264c0abe20 2023-03-03T16:20:58.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.716+0000 7f264bfff700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26300682f0 0x7f263006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:20:58.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.716+0000 7f264bfff700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26300682f0 0x7f263006a7a0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f264c1501d0 tx=0x7f2638008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:20:58.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.720+0000 7f26497fa700 1 -- 172.21.15.7:0/3719303680 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2640055a20 con 0x7f264c0abe20 2023-03-03T16:20:58.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.956+0000 7f2653016700 1 -- 172.21.15.7:0/3719303680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f264c04e6e0 con 0x7f264c0abe20 2023-03-03T16:20:58.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.957+0000 7f26497fa700 1 -- 172.21.15.7:0/3719303680 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f264001f090 con 0x7f264c0abe20 2023-03-03T16:20:58.959 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:20:58.959 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:20:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 -- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26300682f0 msgr2=0x7f263006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26300682f0 0x7f263006a7a0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f264c1501d0 tx=0x7f2638008030 comp rx=0 tx=0).stop 2023-03-03T16:20:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 -- 172.21.15.7:0/3719303680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0abe20 msgr2=0x7f264c1e7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:20:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0abe20 0x7f264c1e7e30 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f26400011e0 tx=0x7f264000cae0 comp rx=0 tx=0).stop 2023-03-03T16:20:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 -- 172.21.15.7:0/3719303680 shutdown_connections 2023-03-03T16:20:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f264c14fda0 0x7f264c14acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f26300682f0 0x7f263006a7a0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f264c0abe20 0x7f264c1e7e30 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 --2- 172.21.15.7:0/3719303680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f264c14f9d0 0x7f264c1549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:20:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 -- 172.21.15.7:0/3719303680 >> 172.21.15.7:0/3719303680 conn(0x7f264c0c8900 msgr2=0x7f264c14f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:20:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 -- 172.21.15.7:0/3719303680 shutdown_connections 2023-03-03T16:20:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:20:58.959+0000 7f2653016700 1 -- 172.21.15.7:0/3719303680 wait complete. 2023-03-03T16:20:58.963 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:20:58.972 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before thrashing 2023-03-03T16:21:01.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:21:02.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.110+0000 7faa189ce700 1 Processor -- start 2023-03-03T16:21:02.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.110+0000 7faa189ce700 1 -- start start 2023-03-03T16:21:02.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.111+0000 7faa189ce700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa1406e020 0x7faa14071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:02.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.111+0000 7faa189ce700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa14072520 0x7faa14151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:02.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.111+0000 7faa189ce700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa14154910 con 0x7faa14155390 2023-03-03T16:21:02.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.111+0000 7faa189ce700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa14070b90 con 0x7faa1406e020 2023-03-03T16:21:02.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.111+0000 7faa189ce700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa14151320 con 0x7faa14072520 2023-03-03T16:21:02.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.111+0000 7faa11d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa1406e020 0x7faa14071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:02.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.111+0000 7faa11d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa1406e020 0x7faa14071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56774/0 (socket says 172.21.15.7:56774) 2023-03-03T16:21:02.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.111+0000 7faa11d9b700 1 -- 172.21.15.7:0/3907336315 learned_addr learned my addr 172.21.15.7:0/3907336315 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:21:02.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.112+0000 7faa12d9d700 1 --2- 172.21.15.7:0/3907336315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa14072520 0x7faa14151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:02.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.112+0000 7faa11d9b700 1 -- 172.21.15.7:0/3907336315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa14072520 msgr2=0x7faa14151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:21:02.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.112+0000 7faa11d9b700 1 --2- 172.21.15.7:0/3907336315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa14072520 0x7faa14151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:02.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.112+0000 7faa11d9b700 1 -- 172.21.15.7:0/3907336315 >> v1:172.21.15.143:6789/0 conn(0x7faa14155390 legacy=0x7faa140c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:21:02.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.112+0000 7faa11d9b700 1 -- 172.21.15.7:0/3907336315 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa140c4540 con 0x7faa1406e020 2023-03-03T16:21:02.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.112+0000 7faa11d9b700 1 --2- 172.21.15.7:0/3907336315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1406e020 0x7faa14071fe0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7faa0400bb90 tx=0x7faa0400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a3d0a87d2355d1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:02.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.112+0000 7faa1159a700 1 -- 172.21.15.7:0/3907336315 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0400dd10 con 0x7faa1406e020 2023-03-03T16:21:02.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.112+0000 7faa1159a700 1 -- 172.21.15.7:0/3907336315 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0400de70 con 0x7faa1406e020 2023-03-03T16:21:02.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.112+0000 7faa1159a700 1 -- 172.21.15.7:0/3907336315 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04004230 con 0x7faa1406e020 2023-03-03T16:21:02.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.113+0000 7faa189ce700 1 -- 172.21.15.7:0/3907336315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1406e020 msgr2=0x7faa14071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:02.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.113+0000 7faa189ce700 1 --2- 172.21.15.7:0/3907336315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1406e020 0x7faa14071fe0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7faa0400bb90 tx=0x7faa0400bea0 comp rx=0 tx=0).stop 2023-03-03T16:21:02.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.113+0000 7faa189ce700 1 -- 172.21.15.7:0/3907336315 shutdown_connections 2023-03-03T16:21:02.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.113+0000 7faa189ce700 1 --2- 172.21.15.7:0/3907336315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa14072520 0x7faa14151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:02.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.113+0000 7faa189ce700 1 --2- 172.21.15.7:0/3907336315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa1406e020 0x7faa14071fe0 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:02.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.113+0000 7faa189ce700 1 -- 172.21.15.7:0/3907336315 >> 172.21.15.7:0/3907336315 conn(0x7faa1406c110 msgr2=0x7faa1406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:21:02.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.113+0000 7faa189ce700 1 -- 172.21.15.7:0/3907336315 shutdown_connections 2023-03-03T16:21:02.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.113+0000 7faa189ce700 1 -- 172.21.15.7:0/3907336315 wait complete. 2023-03-03T16:21:02.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.114+0000 7faa189ce700 1 Processor -- start 2023-03-03T16:21:02.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.114+0000 7faa189ce700 1 -- start start 2023-03-03T16:21:02.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa189ce700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1406e020 0x7faa1414c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:02.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa189ce700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa14072520 0x7faa14148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:02.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa189ce700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa14155390 0x7faa14149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:02.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa189ce700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa141546b0 con 0x7faa14072520 2023-03-03T16:21:02.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa189ce700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1414e860 con 0x7faa14155390 2023-03-03T16:21:02.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa189ce700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1414e3d0 con 0x7faa1406e020 2023-03-03T16:21:02.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa11d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa14072520 0x7faa14148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:02.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa11d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa14072520 0x7faa14148c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56780/0 (socket says 172.21.15.7:56780) 2023-03-03T16:21:02.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa11d9b700 1 -- 172.21.15.7:0/1414160116 learned_addr learned my addr 172.21.15.7:0/1414160116 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:21:02.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa1259c700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1406e020 0x7faa1414c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:02.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa11d9b700 1 -- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1406e020 msgr2=0x7faa1414c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:02.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa11d9b700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1406e020 0x7faa1414c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:02.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa12d9d700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa14155390 0x7faa14149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:02.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa11d9b700 1 -- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa14155390 msgr2=0x7faa14149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:02.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa11d9b700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa14155390 0x7faa14149160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:02.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa11d9b700 1 -- 172.21.15.7:0/1414160116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0400b7f0 con 0x7faa14072520 2023-03-03T16:21:02.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa11d9b700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa14072520 0x7faa14148c20 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7faa140c4a90 tx=0x7faa04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:02.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa02ffd700 1 -- 172.21.15.7:0/1414160116 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04020070 con 0x7faa14072520 2023-03-03T16:21:02.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa02ffd700 1 -- 172.21.15.7:0/1414160116 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa04003aa0 con 0x7faa14072520 2023-03-03T16:21:02.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.115+0000 7faa189ce700 1 -- 172.21.15.7:0/1414160116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa14070540 con 0x7faa14072520 2023-03-03T16:21:02.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.116+0000 7faa02ffd700 1 -- 172.21.15.7:0/1414160116 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04003e10 con 0x7faa14072520 2023-03-03T16:21:02.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.116+0000 7faa189ce700 1 -- 172.21.15.7:0/1414160116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa140702a0 con 0x7faa14072520 2023-03-03T16:21:02.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.116+0000 7faa189ce700 1 -- 172.21.15.7:0/1414160116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa14070820 con 0x7faa14072520 2023-03-03T16:21:02.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.117+0000 7faa02ffd700 1 -- 172.21.15.7:0/1414160116 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faa040193f0 con 0x7faa14072520 2023-03-03T16:21:02.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.117+0000 7faa02ffd700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9f40682f0 0x7fa9f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:02.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.117+0000 7faa02ffd700 1 -- 172.21.15.7:0/1414160116 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faa040898a0 con 0x7faa14072520 2023-03-03T16:21:02.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.117+0000 7faa1259c700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9f40682f0 0x7fa9f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:02.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.118+0000 7faa1259c700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9f40682f0 0x7fa9f406a7a0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7faa141e6700 tx=0x7faa08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:02.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.122+0000 7faa02ffd700 1 -- 172.21.15.7:0/1414160116 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faa04055a70 con 0x7faa14072520 2023-03-03T16:21:02.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.357+0000 7faa189ce700 1 -- 172.21.15.7:0/1414160116 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa140c4540 con 0x7faa14072520 2023-03-03T16:21:02.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.358+0000 7faa02ffd700 1 -- 172.21.15.7:0/1414160116 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faa140c4540 con 0x7faa14072520 2023-03-03T16:21:02.360 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:21:02.360 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:21:02.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 -- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9f40682f0 msgr2=0x7fa9f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:02.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9f40682f0 0x7fa9f406a7a0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7faa141e6700 tx=0x7faa08008040 comp rx=0 tx=0).stop 2023-03-03T16:21:02.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 -- 172.21.15.7:0/1414160116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa14072520 msgr2=0x7faa14148c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:02.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa14072520 0x7faa14148c20 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7faa140c4a90 tx=0x7faa04004750 comp rx=0 tx=0).stop 2023-03-03T16:21:02.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 -- 172.21.15.7:0/1414160116 shutdown_connections 2023-03-03T16:21:02.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa14155390 0x7faa14149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:02.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa1406e020 0x7faa1414c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:02.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9f40682f0 0x7fa9f406a7a0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:02.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 --2- 172.21.15.7:0/1414160116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa14072520 0x7faa14148c20 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:02.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 -- 172.21.15.7:0/1414160116 >> 172.21.15.7:0/1414160116 conn(0x7faa1406c110 msgr2=0x7faa140c25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:21:02.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 -- 172.21.15.7:0/1414160116 shutdown_connections 2023-03-03T16:21:02.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:02.360+0000 7faa189ce700 1 -- 172.21.15.7:0/1414160116 wait complete. 2023-03-03T16:21:02.363 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:21:02.373 INFO:tasks.mds_thrash.fs.[b]:waiting for 99 secs before thrashing 2023-03-03T16:21:14.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:14.902 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:45.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:45.048 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:21:47.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:21:47.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.901+0000 7fd7737f1700 1 Processor -- start 2023-03-03T16:21:47.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.901+0000 7fd7737f1700 1 -- start start 2023-03-03T16:21:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd7737f1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd76c155390 0x7fd76c0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd7737f1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 0x7fd76c151c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd7737f1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c154910 con 0x7fd76c06e020 2023-03-03T16:21:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd7737f1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd76c070b90 con 0x7fd76c155390 2023-03-03T16:21:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd7737f1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c151320 con 0x7fd76c151770 2023-03-03T16:21:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd77158d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd76c155390 0x7fd76c0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd771d8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 0x7fd76c151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd771d8e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 0x7fd76c151c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:36644/0 (socket says 172.21.15.7:36644) 2023-03-03T16:21:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd770d8c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd76c06e020 0x7fd76c071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:54302/0 (socket says 172.21.15.7:54302) 2023-03-03T16:21:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd77158d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd76c155390 0x7fd76c0c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:52576/0 (socket says 172.21.15.7:52576) 2023-03-03T16:21:47.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd771d8e700 1 -- 172.21.15.7:0/3396402999 learned_addr learned my addr 172.21.15.7:0/3396402999 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:21:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd771d8e700 1 -- 172.21.15.7:0/3396402999 >> v2:172.21.15.143:3300/0 conn(0x7fd76c155390 msgr2=0x7fd76c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:21:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd771d8e700 1 --2- 172.21.15.7:0/3396402999 >> v2:172.21.15.143:3300/0 conn(0x7fd76c155390 0x7fd76c0c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd771d8e700 1 -- 172.21.15.7:0/3396402999 >> v1:172.21.15.143:6789/0 conn(0x7fd76c06e020 legacy=0x7fd76c071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:47.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd771d8e700 1 -- 172.21.15.7:0/3396402999 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c0702a0 con 0x7fd76c151770 2023-03-03T16:21:47.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.902+0000 7fd77158d700 1 --2- 172.21.15.7:0/3396402999 >> v2:172.21.15.143:3300/0 conn(0x7fd76c155390 0x7fd76c0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:21:47.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd771d8e700 1 --2- 172.21.15.7:0/3396402999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 0x7fd76c151c10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd764009e50 tx=0x7fd76400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d837225ab698572 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:47.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd763fff700 1 -- 172.21.15.7:0/3396402999 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76400e070 con 0x7fd76c151770 2023-03-03T16:21:47.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd763fff700 1 -- 172.21.15.7:0/3396402999 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76400b910 con 0x7fd76c151770 2023-03-03T16:21:47.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd763fff700 1 -- 172.21.15.7:0/3396402999 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76400bcc0 con 0x7fd76c151770 2023-03-03T16:21:47.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd7737f1700 1 -- 172.21.15.7:0/3396402999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 msgr2=0x7fd76c151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:47.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd7737f1700 1 --2- 172.21.15.7:0/3396402999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 0x7fd76c151c10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd764009e50 tx=0x7fd76400b330 comp rx=0 tx=0).stop 2023-03-03T16:21:47.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd7737f1700 1 -- 172.21.15.7:0/3396402999 shutdown_connections 2023-03-03T16:21:47.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd7737f1700 1 --2- 172.21.15.7:0/3396402999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 0x7fd76c151c10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:47.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd7737f1700 1 --2- 172.21.15.7:0/3396402999 >> v2:172.21.15.143:3300/0 conn(0x7fd76c155390 0x7fd76c0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:47.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd7737f1700 1 -- 172.21.15.7:0/3396402999 >> 172.21.15.7:0/3396402999 conn(0x7fd76c06c110 msgr2=0x7fd76c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:21:47.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd7737f1700 1 -- 172.21.15.7:0/3396402999 shutdown_connections 2023-03-03T16:21:47.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.903+0000 7fd7737f1700 1 -- 172.21.15.7:0/3396402999 wait complete. 2023-03-03T16:21:47.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd7737f1700 1 Processor -- start 2023-03-03T16:21:47.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd7737f1700 1 -- start start 2023-03-03T16:21:47.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd7737f1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06e020 0x7fd76c14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:47.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd7737f1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 0x7fd76c14d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd7737f1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd76c155390 0x7fd76c0c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd7737f1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c149a00 con 0x7fd76c155390 2023-03-03T16:21:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd7737f1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c1537e0 con 0x7fd76c06e020 2023-03-03T16:21:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd7737f1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c14dce0 con 0x7fd76c151770 2023-03-03T16:21:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd771d8e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd76c155390 0x7fd76c0c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd771d8e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd76c155390 0x7fd76c0c59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56810/0 (socket says 172.21.15.7:56810) 2023-03-03T16:21:47.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.905+0000 7fd771d8e700 1 -- 172.21.15.7:0/63334204 learned_addr learned my addr 172.21.15.7:0/63334204 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:21:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd770d8c700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 0x7fd76c14d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd77158d700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06e020 0x7fd76c14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd771d8e700 1 -- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 msgr2=0x7fd76c14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd771d8e700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 0x7fd76c14d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd771d8e700 1 -- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06e020 msgr2=0x7fd76c14e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd771d8e700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06e020 0x7fd76c14e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd771d8e700 1 -- 172.21.15.7:0/63334204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7680094a0 con 0x7fd76c155390 2023-03-03T16:21:47.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd771d8e700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd76c155390 0x7fd76c0c59b0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fd76c151460 tx=0x7fd764004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd761ffb700 1 -- 172.21.15.7:0/63334204 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764004960 con 0x7fd76c155390 2023-03-03T16:21:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd761ffb700 1 -- 172.21.15.7:0/63334204 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7640099b0 con 0x7fd76c155390 2023-03-03T16:21:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd761ffb700 1 -- 172.21.15.7:0/63334204 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76401c680 con 0x7fd76c155390 2023-03-03T16:21:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd7737f1700 1 -- 172.21.15.7:0/63334204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd764009b30 con 0x7fd76c155390 2023-03-03T16:21:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.906+0000 7fd7737f1700 1 -- 172.21.15.7:0/63334204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76c070820 con 0x7fd76c155390 2023-03-03T16:21:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.907+0000 7fd7737f1700 1 -- 172.21.15.7:0/63334204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76c070540 con 0x7fd76c155390 2023-03-03T16:21:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.908+0000 7fd761ffb700 1 -- 172.21.15.7:0/63334204 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd76402c070 con 0x7fd76c155390 2023-03-03T16:21:47.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.908+0000 7fd761ffb700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7500682f0 0x7fd75006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:21:47.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.908+0000 7fd761ffb700 1 -- 172.21.15.7:0/63334204 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd764087d50 con 0x7fd76c155390 2023-03-03T16:21:47.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.909+0000 7fd77158d700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7500682f0 0x7fd75006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:21:47.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.909+0000 7fd77158d700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7500682f0 0x7fd75006a7a0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0c15f0 tx=0x7fd768008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:21:47.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:47.913+0000 7fd761ffb700 1 -- 172.21.15.7:0/63334204 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7640540f0 con 0x7fd76c155390 2023-03-03T16:21:48.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.149+0000 7fd7737f1700 1 -- 172.21.15.7:0/63334204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd76c152260 con 0x7fd76c155390 2023-03-03T16:21:48.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.149+0000 7fd761ffb700 1 -- 172.21.15.7:0/63334204 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd764057710 con 0x7fd76c155390 2023-03-03T16:21:48.150 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:21:48.150 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:21:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 -- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7500682f0 msgr2=0x7fd75006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7500682f0 0x7fd75006a7a0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0c15f0 tx=0x7fd768008040 comp rx=0 tx=0).stop 2023-03-03T16:21:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 -- 172.21.15.7:0/63334204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd76c155390 msgr2=0x7fd76c0c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:21:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd76c155390 0x7fd76c0c59b0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fd76c151460 tx=0x7fd764004ad0 comp rx=0 tx=0).stop 2023-03-03T16:21:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 -- 172.21.15.7:0/63334204 shutdown_connections 2023-03-03T16:21:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7500682f0 0x7fd75006a7a0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd76c155390 0x7fd76c0c59b0 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd76c06e020 0x7fd76c14e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 --2- 172.21.15.7:0/63334204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd76c151770 0x7fd76c14d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:21:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 -- 172.21.15.7:0/63334204 >> 172.21.15.7:0/63334204 conn(0x7fd76c06c110 msgr2=0x7fd76c0c3280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:21:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 -- 172.21.15.7:0/63334204 shutdown_connections 2023-03-03T16:21:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:21:48.151+0000 7fd7737f1700 1 -- 172.21.15.7:0/63334204 wait complete. 2023-03-03T16:21:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:21:48.164 INFO:tasks.mds_thrash.fs.[a]:waiting for 46 secs before thrashing 2023-03-03T16:22:15.210 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:15.213 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:34.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:22:34.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.303+0000 7fe4aaa92700 1 Processor -- start 2023-03-03T16:22:34.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.303+0000 7fe4aaa92700 1 -- start start 2023-03-03T16:22:34.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4aaa92700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4a4151b20 0x7fe4a4071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:34.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4aaa92700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe4a4154f10 0x7fe4a4158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:34.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4aaa92700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a4151ef0 con 0x7fe4a4071d60 2023-03-03T16:22:34.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4aaa92700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a4157b20 con 0x7fe4a4151b20 2023-03-03T16:22:34.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4aaa92700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a4155410 con 0x7fe4a4154f10 2023-03-03T16:22:34.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4a882e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4a4151b20 0x7fe4a4071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:34.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4a882e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe4a4151b20 0x7fe4a4071820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56826/0 (socket says 172.21.15.7:56826) 2023-03-03T16:22:34.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4a882e700 1 -- 172.21.15.7:0/426577786 learned_addr learned my addr 172.21.15.7:0/426577786 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:22:34.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4a3fff700 1 --2- 172.21.15.7:0/426577786 >> v2:172.21.15.143:3300/0 conn(0x7fe4a4154f10 0x7fe4a4158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:34.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4a882e700 1 -- 172.21.15.7:0/426577786 >> v2:172.21.15.143:3300/0 conn(0x7fe4a4154f10 msgr2=0x7fe4a4158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:22:34.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4a882e700 1 --2- 172.21.15.7:0/426577786 >> v2:172.21.15.143:3300/0 conn(0x7fe4a4154f10 0x7fe4a4158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:34.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4a882e700 1 -- 172.21.15.7:0/426577786 >> v1:172.21.15.143:6789/0 conn(0x7fe4a4071d60 legacy=0x7fe4a40c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:34.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4a882e700 1 -- 172.21.15.7:0/426577786 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a406f640 con 0x7fe4a4151b20 2023-03-03T16:22:34.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.304+0000 7fe4a882e700 1 --2- 172.21.15.7:0/426577786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4151b20 0x7fe4a4071820 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4070850 tx=0x7fe498009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed8a4e6e7029320 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:34.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4a37fe700 1 -- 172.21.15.7:0/426577786 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49800b870 con 0x7fe4a4151b20 2023-03-03T16:22:34.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4a37fe700 1 -- 172.21.15.7:0/426577786 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49800b9d0 con 0x7fe4a4151b20 2023-03-03T16:22:34.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4a37fe700 1 -- 172.21.15.7:0/426577786 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49800bd60 con 0x7fe4a4151b20 2023-03-03T16:22:34.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4aaa92700 1 -- 172.21.15.7:0/426577786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4151b20 msgr2=0x7fe4a4071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:34.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4aaa92700 1 --2- 172.21.15.7:0/426577786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4151b20 0x7fe4a4071820 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4070850 tx=0x7fe498009bd0 comp rx=0 tx=0).stop 2023-03-03T16:22:34.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4aaa92700 1 -- 172.21.15.7:0/426577786 shutdown_connections 2023-03-03T16:22:34.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4aaa92700 1 --2- 172.21.15.7:0/426577786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4151b20 0x7fe4a4071820 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:34.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4aaa92700 1 --2- 172.21.15.7:0/426577786 >> v2:172.21.15.143:3300/0 conn(0x7fe4a4154f10 0x7fe4a4158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4aaa92700 1 -- 172.21.15.7:0/426577786 >> 172.21.15.7:0/426577786 conn(0x7fe4a40c8900 msgr2=0x7fe4a40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:22:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4aaa92700 1 -- 172.21.15.7:0/426577786 shutdown_connections 2023-03-03T16:22:34.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.305+0000 7fe4aaa92700 1 -- 172.21.15.7:0/426577786 wait complete. 2023-03-03T16:22:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.306+0000 7fe4aaa92700 1 Processor -- start 2023-03-03T16:22:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.307+0000 7fe4aaa92700 1 -- start start 2023-03-03T16:22:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.307+0000 7fe4aaa92700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4a4071d60 0x7fe4a41526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.307+0000 7fe4aaa92700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4a4151b20 0x7fe4a4146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.307+0000 7fe4aaa92700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4154f10 0x7fe4a4156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:34.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.307+0000 7fe4aaa92700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a4154740 con 0x7fe4a4154f10 2023-03-03T16:22:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.307+0000 7fe4aaa92700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a41469b0 con 0x7fe4a4071d60 2023-03-03T16:22:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.307+0000 7fe4aaa92700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a41467d0 con 0x7fe4a4151b20 2023-03-03T16:22:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.307+0000 7fe4a902f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4154f10 0x7fe4a4156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a902f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4154f10 0x7fe4a4156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56836/0 (socket says 172.21.15.7:56836) 2023-03-03T16:22:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a902f700 1 -- 172.21.15.7:0/1307612990 learned_addr learned my addr 172.21.15.7:0/1307612990 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:22:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a882e700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4a4071d60 0x7fe4a41526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:34.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a3fff700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4a4151b20 0x7fe4a4146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a902f700 1 -- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4a4151b20 msgr2=0x7fe4a4146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a902f700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4a4151b20 0x7fe4a4146290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a902f700 1 -- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4a4071d60 msgr2=0x7fe4a41526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a902f700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4a4071d60 0x7fe4a41526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a902f700 1 -- 172.21.15.7:0/1307612990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe498009550 con 0x7fe4a4154f10 2023-03-03T16:22:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a3fff700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4a4151b20 0x7fe4a4146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:22:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a902f700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4154f10 0x7fe4a4156b50 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00cbc0 tx=0x7fe49c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:34.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a17fa700 1 -- 172.21.15.7:0/1307612990 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c003c00 con 0x7fe4a4154f10 2023-03-03T16:22:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a17fa700 1 -- 172.21.15.7:0/1307612990 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c003d60 con 0x7fe4a4154f10 2023-03-03T16:22:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4aaa92700 1 -- 172.21.15.7:0/1307612990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a406fc00 con 0x7fe4a4154f10 2023-03-03T16:22:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4a17fa700 1 -- 172.21.15.7:0/1307612990 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c00ab90 con 0x7fe4a4154f10 2023-03-03T16:22:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.308+0000 7fe4aaa92700 1 -- 172.21.15.7:0/1307612990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a406f920 con 0x7fe4a4154f10 2023-03-03T16:22:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.309+0000 7fe4aaa92700 1 -- 172.21.15.7:0/1307612990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a40aae90 con 0x7fe4a4154f10 2023-03-03T16:22:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.310+0000 7fe4a17fa700 1 -- 172.21.15.7:0/1307612990 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe49c010050 con 0x7fe4a4154f10 2023-03-03T16:22:34.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.310+0000 7fe4a17fa700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4880682f0 0x7fe48806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.310+0000 7fe4a17fa700 1 -- 172.21.15.7:0/1307612990 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe49c086000 con 0x7fe4a4154f10 2023-03-03T16:22:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.310+0000 7fe4a882e700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4880682f0 0x7fe48806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.311+0000 7fe4a882e700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4880682f0 0x7fe48806a7a0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fe498000c00 tx=0x7fe498008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:34.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.315+0000 7fe4a17fa700 1 -- 172.21.15.7:0/1307612990 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe49c0523a0 con 0x7fe4a4154f10 2023-03-03T16:22:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.552+0000 7fe4aaa92700 1 -- 172.21.15.7:0/1307612990 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4a4074bb0 con 0x7fe4a4154f10 2023-03-03T16:22:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.552+0000 7fe4a17fa700 1 -- 172.21.15.7:0/1307612990 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe49c0559c0 con 0x7fe4a4154f10 2023-03-03T16:22:34.553 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:22:34.553 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:22:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.554+0000 7fe4aaa92700 1 -- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4880682f0 msgr2=0x7fe48806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.554+0000 7fe4aaa92700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4880682f0 0x7fe48806a7a0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fe498000c00 tx=0x7fe498008030 comp rx=0 tx=0).stop 2023-03-03T16:22:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.554+0000 7fe4aaa92700 1 -- 172.21.15.7:0/1307612990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4154f10 msgr2=0x7fe4a4156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.554+0000 7fe4aaa92700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4154f10 0x7fe4a4156b50 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00cbc0 tx=0x7fe49c00cf80 comp rx=0 tx=0).stop 2023-03-03T16:22:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.555+0000 7fe4aaa92700 1 -- 172.21.15.7:0/1307612990 shutdown_connections 2023-03-03T16:22:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.555+0000 7fe4aaa92700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4880682f0 0x7fe48806a7a0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.555+0000 7fe4aaa92700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4a4154f10 0x7fe4a4156b50 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.555+0000 7fe4aaa92700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4a4071d60 0x7fe4a41526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.555+0000 7fe4aaa92700 1 --2- 172.21.15.7:0/1307612990 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4a4151b20 0x7fe4a4146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.555+0000 7fe4aaa92700 1 -- 172.21.15.7:0/1307612990 >> 172.21.15.7:0/1307612990 conn(0x7fe4a40c8900 msgr2=0x7fe4a40c5740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:22:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.555+0000 7fe4aaa92700 1 -- 172.21.15.7:0/1307612990 shutdown_connections 2023-03-03T16:22:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:34.555+0000 7fe4aaa92700 1 -- 172.21.15.7:0/1307612990 wait complete. 2023-03-03T16:22:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:22:34.567 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T16:22:41.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:22:41.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.513+0000 7f4313728700 1 Processor -- start 2023-03-03T16:22:41.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.513+0000 7f4313728700 1 -- start start 2023-03-03T16:22:41.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.514+0000 7f4313728700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f430c14e900 0x7f430c15ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:41.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.514+0000 7f4313728700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430c1510d0 con 0x7f430c071e80 2023-03-03T16:22:41.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.514+0000 7f4313728700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430c0c3740 con 0x7f430c072250 2023-03-03T16:22:41.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.514+0000 7f4313728700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430c070670 con 0x7f430c14e900 2023-03-03T16:22:41.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.514+0000 7f43114c4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f430c071e80 0x7f430c151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50834/0 (socket says 172.21.15.7:50834) 2023-03-03T16:22:41.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.514+0000 7f43114c4700 1 -- 172.21.15.7:0/3493195649 learned_addr learned my addr 172.21.15.7:0/3493195649 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:22:41.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.514+0000 7f4311cc5700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f430c14e900 0x7f430c15ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:41.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.514+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092972452 0 0) 0x7f430c1510d0 con 0x7f430c071e80 2023-03-03T16:22:41.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42f80035b0 con 0x7f430c071e80 2023-03-03T16:22:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322584281 0 0) 0x7f430c0c3740 con 0x7f430c072250 2023-03-03T16:22:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f430c1510d0 con 0x7f430c072250 2023-03-03T16:22:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1121147953 0 0) 0x7f42f80035b0 con 0x7f430c071e80 2023-03-03T16:22:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f430c14e900 msgr2=0x7f430c15ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:22:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f430c14e900 0x7f430c15ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 >> v1:172.21.15.143:6789/0 conn(0x7f430c072250 legacy=0x7f430c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c0c39e0 con 0x7f430c071e80 2023-03-03T16:22:41.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f42fc001670 con 0x7f430c071e80 2023-03-03T16:22:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 >> v1:172.21.15.7:6789/0 conn(0x7f430c071e80 legacy=0x7f430c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42f8003c30 0x7f42f8004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f430c14e900 0x7f42f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42f8004d90 0x7f42f8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c070670 con 0x7f42f8004d90 2023-03-03T16:22:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f800dde0 con 0x7f42f8003c30 2023-03-03T16:22:41.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f800dfa0 con 0x7f430c14e900 2023-03-03T16:22:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4311cc5700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42f8004d90 0x7f42f8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f43114c4700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42f8003c30 0x7f42f8004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4310cc3700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f430c14e900 0x7f42f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4311cc5700 1 -- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f430c14e900 msgr2=0x7f42f8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:22:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4311cc5700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f430c14e900 0x7f42f8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4311cc5700 1 -- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42f8003c30 msgr2=0x7f42f8004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:22:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4311cc5700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42f8003c30 0x7f42f8004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:41.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.515+0000 7f4311cc5700 1 -- 172.21.15.7:0/3493195649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f80035b0 con 0x7f42f8004d90 2023-03-03T16:22:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4311cc5700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42f8004d90 0x7f42f8005200 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f4304009f90 tx=0x7f42f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739e616a77a36720 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430400ec30 con 0x7f42f8004d90 2023-03-03T16:22:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430400f040 con 0x7f42f8004d90 2023-03-03T16:22:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4303fff700 1 -- 172.21.15.7:0/3493195649 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304014480 con 0x7f42f8004d90 2023-03-03T16:22:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4313728700 1 -- 172.21.15.7:0/3493195649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42f8004d90 msgr2=0x7f42f8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4313728700 1 --2- 172.21.15.7:0/3493195649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f42f8004d90 0x7f42f8005200 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f4304009f90 tx=0x7f42f8005880 comp rx=0 tx=0).stop 2023-03-03T16:22:41.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4311cc5700 1 -- 172.21.15.7:0/3493195649 reap_dead start 2023-03-03T16:22:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4313728700 1 -- 172.21.15.7:0/3493195649 shutdown_connections 2023-03-03T16:22:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4313728700 1 -- 172.21.15.7:0/3493195649 >> 172.21.15.7:0/3493195649 conn(0x7f430c0a94d0 msgr2=0x7f430c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:22:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4313728700 1 -- 172.21.15.7:0/3493195649 shutdown_connections 2023-03-03T16:22:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.516+0000 7f4313728700 1 -- 172.21.15.7:0/3493195649 wait complete. 2023-03-03T16:22:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4313728700 1 Processor -- start 2023-03-03T16:22:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4313728700 1 -- start start 2023-03-03T16:22:41.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4313728700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42f8004d90 0x7f430c1e8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:41.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4313728700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430c14c690 0x7f430c14cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:41.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4313728700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c14d040 0x7f430c153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:41.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4313728700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c153810 con 0x7f430c14c690 2023-03-03T16:22:41.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4313728700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430c154ec0 con 0x7f430c14d040 2023-03-03T16:22:41.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4313728700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430c1e9800 con 0x7f42f8004d90 2023-03-03T16:22:41.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4310cc3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430c14c690 0x7f430c14cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:41.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4310cc3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430c14c690 0x7f430c14cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56868/0 (socket says 172.21.15.7:56868) 2023-03-03T16:22:41.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.518+0000 7f4310cc3700 1 -- 172.21.15.7:0/2434524363 learned_addr learned my addr 172.21.15.7:0/2434524363 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:22:41.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f43114c4700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42f8004d90 0x7f430c1e8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:41.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4311cc5700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c14d040 0x7f430c153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:41.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4310cc3700 1 -- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42f8004d90 msgr2=0x7f430c1e8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:41.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4310cc3700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42f8004d90 0x7f430c1e8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:41.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4310cc3700 1 -- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c14d040 msgr2=0x7f430c153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:41.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4310cc3700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c14d040 0x7f430c153bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:41.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4310cc3700 1 -- 172.21.15.7:0/2434524363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c151f30 con 0x7f430c14c690 2023-03-03T16:22:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4311cc5700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c14d040 0x7f430c153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:22:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4310cc3700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430c14c690 0x7f430c14cb00 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f430800f830 tx=0x7f430800fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4301ffb700 1 -- 172.21.15.7:0/2434524363 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308005320 con 0x7f430c14c690 2023-03-03T16:22:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4301ffb700 1 -- 172.21.15.7:0/2434524363 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4308005480 con 0x7f430c14c690 2023-03-03T16:22:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4313728700 1 -- 172.21.15.7:0/2434524363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430c151300 con 0x7f430c14c690 2023-03-03T16:22:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4301ffb700 1 -- 172.21.15.7:0/2434524363 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308006aa0 con 0x7f430c14c690 2023-03-03T16:22:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.519+0000 7f4313728700 1 -- 172.21.15.7:0/2434524363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430c06c9a0 con 0x7f430c14c690 2023-03-03T16:22:41.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.520+0000 7f4313728700 1 -- 172.21.15.7:0/2434524363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430c155030 con 0x7f430c14c690 2023-03-03T16:22:41.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.520+0000 7f4301ffb700 1 -- 172.21.15.7:0/2434524363 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f430801c030 con 0x7f430c14c690 2023-03-03T16:22:41.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.521+0000 7f4301ffb700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42f00682f0 0x7f42f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:22:41.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.521+0000 7f4301ffb700 1 -- 172.21.15.7:0/2434524363 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4308085a30 con 0x7f430c14c690 2023-03-03T16:22:41.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.521+0000 7f43114c4700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42f00682f0 0x7f42f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:22:41.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.522+0000 7f43114c4700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42f00682f0 0x7f42f006a7a0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f430c147b90 tx=0x7f42fc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:22:41.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.525+0000 7f4301ffb700 1 -- 172.21.15.7:0/2434524363 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4308051dd0 con 0x7f430c14c690 2023-03-03T16:22:41.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.764+0000 7f4313728700 1 -- 172.21.15.7:0/2434524363 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f430c074bb0 con 0x7f430c14c690 2023-03-03T16:22:41.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.765+0000 7f4301ffb700 1 -- 172.21.15.7:0/2434524363 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f43080553f0 con 0x7f430c14c690 2023-03-03T16:22:41.766 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:22:41.766 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:22:41.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 -- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42f00682f0 msgr2=0x7f42f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:41.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42f00682f0 0x7f42f006a7a0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f430c147b90 tx=0x7f42fc009040 comp rx=0 tx=0).stop 2023-03-03T16:22:41.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 -- 172.21.15.7:0/2434524363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430c14c690 msgr2=0x7f430c14cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:22:41.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430c14c690 0x7f430c14cb00 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f430800f830 tx=0x7f430800fbf0 comp rx=0 tx=0).stop 2023-03-03T16:22:41.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 -- 172.21.15.7:0/2434524363 shutdown_connections 2023-03-03T16:22:41.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f430c14d040 0x7f430c153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:41.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42f8004d90 0x7f430c1e8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:41.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42f00682f0 0x7f42f006a7a0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:41.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 --2- 172.21.15.7:0/2434524363 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430c14c690 0x7f430c14cb00 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:22:41.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 -- 172.21.15.7:0/2434524363 >> 172.21.15.7:0/2434524363 conn(0x7f430c0a94d0 msgr2=0x7f430c06d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:22:41.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 -- 172.21.15.7:0/2434524363 shutdown_connections 2023-03-03T16:22:41.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:22:41.767+0000 7f4313728700 1 -- 172.21.15.7:0/2434524363 wait complete. 2023-03-03T16:22:41.769 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:22:41.779 INFO:tasks.mds_thrash.fs.[b]:waiting for 103 secs before thrashing 2023-03-03T16:22:45.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:22:45.274 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:15.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:15.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:23.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:23:23.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.706+0000 7f72f1613700 1 Processor -- start 2023-03-03T16:23:23.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.706+0000 7f72f1613700 1 -- start start 2023-03-03T16:23:23.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.707+0000 7f72f1613700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f72ec06b6f0 0x7f72ec155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:23:23.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.707+0000 7f72f1613700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec157bb0 con 0x7f72ec0c7a30 2023-03-03T16:23:23.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.707+0000 7f72f1613700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec06a2f0 con 0x7f72ec0c7e00 2023-03-03T16:23:23.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.707+0000 7f72f1613700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec06ab00 con 0x7f72ec06b6f0 2023-03-03T16:23:23.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.707+0000 7f72eaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f72ec0c7a30 0x7f72ec14e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50884/0 (socket says 172.21.15.7:50884) 2023-03-03T16:23:23.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.707+0000 7f72eaffd700 1 -- 172.21.15.7:0/4019257585 learned_addr learned my addr 172.21.15.7:0/4019257585 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:23:23.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72eb7fe700 1 --2- 172.21.15.7:0/4019257585 >> v2:172.21.15.143:3300/0 conn(0x7f72ec06b6f0 0x7f72ec155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:23:23.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72e9ffb700 1 -- 172.21.15.7:0/4019257585 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688040642 0 0) 0x7f72ec157bb0 con 0x7f72ec0c7a30 2023-03-03T16:23:23.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72e9ffb700 1 -- 172.21.15.7:0/4019257585 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d40035b0 con 0x7f72ec0c7a30 2023-03-03T16:23:23.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72eb7fe700 1 -- 172.21.15.7:0/4019257585 >> v1:172.21.15.143:6789/0 conn(0x7f72ec0c7e00 legacy=0x7f72ec154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:23:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72eb7fe700 1 -- 172.21.15.7:0/4019257585 >> v1:172.21.15.7:6789/0 conn(0x7f72ec0c7a30 legacy=0x7f72ec14e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:23:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72e9ffb700 1 -- 172.21.15.7:0/4019257585 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3188484845 0 0) 0x7f72d40035b0 con 0x7f72ec0c7a30 2023-03-03T16:23:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72eb7fe700 1 -- 172.21.15.7:0/4019257585 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0bfbb0 con 0x7f72ec06b6f0 2023-03-03T16:23:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72eb7fe700 1 --2- 172.21.15.7:0/4019257585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ec06b6f0 0x7f72ec155120 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f72e4009d90 tx=0x7f72e400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5e02de18a4b138e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:23:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72e9ffb700 1 -- 172.21.15.7:0/4019257585 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e400e070 con 0x7f72ec06b6f0 2023-03-03T16:23:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72e9ffb700 1 -- 172.21.15.7:0/4019257585 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e4003a20 con 0x7f72ec06b6f0 2023-03-03T16:23:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.708+0000 7f72e9ffb700 1 -- 172.21.15.7:0/4019257585 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0017f0 con 0x7f72ec06b6f0 2023-03-03T16:23:23.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.709+0000 7f72f1613700 1 -- 172.21.15.7:0/4019257585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ec06b6f0 msgr2=0x7f72ec155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:23:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.709+0000 7f72f1613700 1 --2- 172.21.15.7:0/4019257585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ec06b6f0 0x7f72ec155120 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f72e4009d90 tx=0x7f72e400b330 comp rx=0 tx=0).stop 2023-03-03T16:23:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.709+0000 7f72f1613700 1 -- 172.21.15.7:0/4019257585 shutdown_connections 2023-03-03T16:23:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.709+0000 7f72f1613700 1 --2- 172.21.15.7:0/4019257585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ec06b6f0 0x7f72ec155120 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:23:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.709+0000 7f72f1613700 1 -- 172.21.15.7:0/4019257585 >> 172.21.15.7:0/4019257585 conn(0x7f72ec073d20 msgr2=0x7f72ec0716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:23:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.709+0000 7f72f1613700 1 -- 172.21.15.7:0/4019257585 shutdown_connections 2023-03-03T16:23:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.709+0000 7f72f1613700 1 -- 172.21.15.7:0/4019257585 wait complete. 2023-03-03T16:23:23.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.710+0000 7f72f1613700 1 Processor -- start 2023-03-03T16:23:23.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.710+0000 7f72f1613700 1 -- start start 2023-03-03T16:23:23.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72f1613700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ec0c7a30 0x7f72ec1decb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:23:23.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72f1613700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ec0c7e00 0x7f72ec156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72f1613700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ec1e0c50 0x7f72ec146f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72f1613700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec1ed810 con 0x7f72ec0c7a30 2023-03-03T16:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72f1613700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0c4910 con 0x7f72ec0c7e00 2023-03-03T16:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72f1613700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec1de000 con 0x7f72ec1e0c50 2023-03-03T16:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72eaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ec0c7a30 0x7f72ec1decb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72eaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ec0c7a30 0x7f72ec1decb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56910/0 (socket says 172.21.15.7:56910) 2023-03-03T16:23:23.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72eaffd700 1 -- 172.21.15.7:0/3122255627 learned_addr learned my addr 172.21.15.7:0/3122255627 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72ea7fc700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ec0c7e00 0x7f72ec156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72eb7fe700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ec1e0c50 0x7f72ec146f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72eaffd700 1 -- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ec1e0c50 msgr2=0x7f72ec146f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72eaffd700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ec1e0c50 0x7f72ec146f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72eaffd700 1 -- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ec0c7e00 msgr2=0x7f72ec156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.711+0000 7f72eaffd700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ec0c7e00 0x7f72ec156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:23:23.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.712+0000 7f72eaffd700 1 -- 172.21.15.7:0/3122255627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e4009a70 con 0x7f72ec0c7a30 2023-03-03T16:23:23.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.712+0000 7f72eaffd700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ec0c7a30 0x7f72ec1decb0 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00c750 tx=0x7f72dc00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:23:23.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.712+0000 7f72d37fe700 1 -- 172.21.15.7:0/3122255627 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0074e0 con 0x7f72ec0c7a30 2023-03-03T16:23:23.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.712+0000 7f72d37fe700 1 -- 172.21.15.7:0/3122255627 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc007640 con 0x7f72ec0c7a30 2023-03-03T16:23:23.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.712+0000 7f72f1613700 1 -- 172.21.15.7:0/3122255627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec06a820 con 0x7f72ec0c7a30 2023-03-03T16:23:23.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.712+0000 7f72d37fe700 1 -- 172.21.15.7:0/3122255627 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc005a70 con 0x7f72ec0c7a30 2023-03-03T16:23:23.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.712+0000 7f72f1613700 1 -- 172.21.15.7:0/3122255627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec06a540 con 0x7f72ec0c7a30 2023-03-03T16:23:23.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.713+0000 7f72f1613700 1 -- 172.21.15.7:0/3122255627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec14e870 con 0x7f72ec0c7a30 2023-03-03T16:23:23.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.713+0000 7f72d37fe700 1 -- 172.21.15.7:0/3122255627 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f72dc019030 con 0x7f72ec0c7a30 2023-03-03T16:23:23.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.714+0000 7f72d37fe700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72d80682f0 0x7f72d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:23:23.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.714+0000 7f72d37fe700 1 -- 172.21.15.7:0/3122255627 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f72dc0854b0 con 0x7f72ec0c7a30 2023-03-03T16:23:23.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.714+0000 7f72ea7fc700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72d80682f0 0x7f72d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:23:23.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.715+0000 7f72ea7fc700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72d80682f0 0x7f72d806a7a0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f72ec14adc0 tx=0x7f72e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:23:23.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.719+0000 7f72d37fe700 1 -- 172.21.15.7:0/3122255627 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f72dc051850 con 0x7f72ec0c7a30 2023-03-03T16:23:23.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.957+0000 7f72f1613700 1 -- 172.21.15.7:0/3122255627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72ec074bb0 con 0x7f72ec0c7a30 2023-03-03T16:23:23.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.958+0000 7f72d37fe700 1 -- 172.21.15.7:0/3122255627 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f72dc054e70 con 0x7f72ec0c7a30 2023-03-03T16:23:23.958 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:23:23.959 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:23:23.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 -- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72d80682f0 msgr2=0x7f72d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:23:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72d80682f0 0x7f72d806a7a0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f72ec14adc0 tx=0x7f72e0009040 comp rx=0 tx=0).stop 2023-03-03T16:23:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 -- 172.21.15.7:0/3122255627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ec0c7a30 msgr2=0x7f72ec1decb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:23:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ec0c7a30 0x7f72ec1decb0 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00c750 tx=0x7f72dc00ca60 comp rx=0 tx=0).stop 2023-03-03T16:23:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 -- 172.21.15.7:0/3122255627 shutdown_connections 2023-03-03T16:23:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72ec1e0c50 0x7f72ec146f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:23:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72d80682f0 0x7f72d806a7a0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:23:23.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72ec0c7a30 0x7f72ec1decb0 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:23:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 --2- 172.21.15.7:0/3122255627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72ec0c7e00 0x7f72ec156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:23:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 -- 172.21.15.7:0/3122255627 >> 172.21.15.7:0/3122255627 conn(0x7f72ec06b6f0 msgr2=0x7f72ec14d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:23:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 -- 172.21.15.7:0/3122255627 shutdown_connections 2023-03-03T16:23:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:23:23.960+0000 7f72f1613700 1 -- 172.21.15.7:0/3122255627 wait complete. 2023-03-03T16:23:23.962 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:23:23.973 INFO:tasks.mds_thrash.fs.[a]:waiting for 97 secs before thrashing 2023-03-03T16:23:45.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:23:45.408 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:15.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:15.474 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:24.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:24:24.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.918+0000 7f6c951c2700 1 Processor -- start 2023-03-03T16:24:24.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.918+0000 7f6c951c2700 1 -- start start 2023-03-03T16:24:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c951c2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6c9014fda0 0x7f6c90156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:24:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c951c2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900c2f30 con 0x7f6c9014f9d0 2023-03-03T16:24:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c951c2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900c31d0 con 0x7f6c900abe20 2023-03-03T16:24:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c951c2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c901524a0 con 0x7f6c9014fda0 2023-03-03T16:24:24.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c8e59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6c9014fda0 0x7f6c90156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:24:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c8ed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6c9014f9d0 0x7f6c90071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50912/0 (socket says 172.21.15.7:50912) 2023-03-03T16:24:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c8ed9d700 1 -- 172.21.15.7:0/2775640346 learned_addr learned my addr 172.21.15.7:0/2775640346 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:24:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c8e59c700 1 -- 172.21.15.7:0/2775640346 >> v1:172.21.15.143:6789/0 conn(0x7f6c900abe20 legacy=0x7f6c901529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:24:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c8e59c700 1 -- 172.21.15.7:0/2775640346 >> v1:172.21.15.7:6789/0 conn(0x7f6c9014f9d0 legacy=0x7f6c90071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c8e59c700 1 -- 172.21.15.7:0/2775640346 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900c34b0 con 0x7f6c9014fda0 2023-03-03T16:24:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c8dd9b700 1 -- 172.21.15.7:0/2775640346 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512459177 0 0) 0x7f6c900c2f30 con 0x7f6c9014f9d0 2023-03-03T16:24:24.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.919+0000 7f6c8e59c700 1 --2- 172.21.15.7:0/2775640346 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 0x7f6c90156430 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009840 tx=0x7f6c84009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10b6a53e4b82904 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:24:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.920+0000 7f6c8dd9b700 1 -- 172.21.15.7:0/2775640346 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84004230 con 0x7f6c9014fda0 2023-03-03T16:24:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.920+0000 7f6c8dd9b700 1 -- 172.21.15.7:0/2775640346 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c84004390 con 0x7f6c9014fda0 2023-03-03T16:24:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.920+0000 7f6c8dd9b700 1 -- 172.21.15.7:0/2775640346 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c840036a0 con 0x7f6c9014fda0 2023-03-03T16:24:24.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.920+0000 7f6c951c2700 1 -- 172.21.15.7:0/2775640346 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 msgr2=0x7f6c90156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:24.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.920+0000 7f6c951c2700 1 --2- 172.21.15.7:0/2775640346 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 0x7f6c90156430 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009840 tx=0x7f6c84009b50 comp rx=0 tx=0).stop 2023-03-03T16:24:24.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.920+0000 7f6c951c2700 1 -- 172.21.15.7:0/2775640346 shutdown_connections 2023-03-03T16:24:24.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.920+0000 7f6c951c2700 1 --2- 172.21.15.7:0/2775640346 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 0x7f6c90156430 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:24.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.920+0000 7f6c951c2700 1 -- 172.21.15.7:0/2775640346 >> 172.21.15.7:0/2775640346 conn(0x7f6c900c8900 msgr2=0x7f6c900c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:24:24.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.920+0000 7f6c951c2700 1 -- 172.21.15.7:0/2775640346 shutdown_connections 2023-03-03T16:24:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.920+0000 7f6c951c2700 1 -- 172.21.15.7:0/2775640346 wait complete. 2023-03-03T16:24:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.922+0000 7f6c951c2700 1 Processor -- start 2023-03-03T16:24:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.922+0000 7f6c951c2700 1 -- start start 2023-03-03T16:24:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.922+0000 7f6c951c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c900abe20 0x7f6c90151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:24:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.922+0000 7f6c951c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9014f9d0 0x7f6c9014b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:24:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.922+0000 7f6c951c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 0x7f6c9006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:24:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.922+0000 7f6c951c2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901dc310 con 0x7f6c9014fda0 2023-03-03T16:24:24.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.922+0000 7f6c951c2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9014a8d0 con 0x7f6c9014f9d0 2023-03-03T16:24:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.922+0000 7f6c951c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9014bb40 con 0x7f6c900abe20 2023-03-03T16:24:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8f59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 0x7f6c9006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:24:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8f59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 0x7f6c9006d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56942/0 (socket says 172.21.15.7:56942) 2023-03-03T16:24:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8f59e700 1 -- 172.21.15.7:0/3538250432 learned_addr learned my addr 172.21.15.7:0/3538250432 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:24:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8e59c700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9014f9d0 0x7f6c9014b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:24:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8f59e700 1 -- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c900abe20 msgr2=0x7f6c90151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8ed9d700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c900abe20 0x7f6c90151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:24:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8f59e700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c900abe20 0x7f6c90151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8f59e700 1 -- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9014f9d0 msgr2=0x7f6c9014b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8f59e700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9014f9d0 0x7f6c9014b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8f59e700 1 -- 172.21.15.7:0/3538250432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c840094a0 con 0x7f6c9014fda0 2023-03-03T16:24:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8e59c700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9014f9d0 0x7f6c9014b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:24:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c8f59e700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 0x7f6c9006d390 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800b9e0 tx=0x7f6c8800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:24:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c777fe700 1 -- 172.21.15.7:0/3538250432 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800c950 con 0x7f6c9014fda0 2023-03-03T16:24:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c777fe700 1 -- 172.21.15.7:0/3538250432 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8800cab0 con 0x7f6c9014fda0 2023-03-03T16:24:24.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c951c2700 1 -- 172.21.15.7:0/3538250432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c900ac350 con 0x7f6c9014fda0 2023-03-03T16:24:24.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c777fe700 1 -- 172.21.15.7:0/3538250432 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800ce60 con 0x7f6c9014fda0 2023-03-03T16:24:24.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.923+0000 7f6c951c2700 1 -- 172.21.15.7:0/3538250432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c900ab630 con 0x7f6c9014fda0 2023-03-03T16:24:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.924+0000 7f6c951c2700 1 -- 172.21.15.7:0/3538250432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c90071fa0 con 0x7f6c9014fda0 2023-03-03T16:24:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.925+0000 7f6c777fe700 1 -- 172.21.15.7:0/3538250432 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c8801b030 con 0x7f6c9014fda0 2023-03-03T16:24:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.925+0000 7f6c777fe700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c7c0682f0 0x7f6c7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:24:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.925+0000 7f6c777fe700 1 -- 172.21.15.7:0/3538250432 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6c88085d70 con 0x7f6c9014fda0 2023-03-03T16:24:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.925+0000 7f6c8ed9d700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c7c0682f0 0x7f6c7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:24:24.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.926+0000 7f6c8ed9d700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c7c0682f0 0x7f6c7c06a7a0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f6c80000f60 tx=0x7f6c80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:24:24.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:24.930+0000 7f6c777fe700 1 -- 172.21.15.7:0/3538250432 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c88052110 con 0x7f6c9014fda0 2023-03-03T16:24:25.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.169+0000 7f6c951c2700 1 -- 172.21.15.7:0/3538250432 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c90074bb0 con 0x7f6c9014fda0 2023-03-03T16:24:25.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.170+0000 7f6c777fe700 1 -- 172.21.15.7:0/3538250432 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6c88019020 con 0x7f6c9014fda0 2023-03-03T16:24:25.172 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:24:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:24:25.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 -- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c7c0682f0 msgr2=0x7f6c7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:25.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c7c0682f0 0x7f6c7c06a7a0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f6c80000f60 tx=0x7f6c80009040 comp rx=0 tx=0).stop 2023-03-03T16:24:25.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 -- 172.21.15.7:0/3538250432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 msgr2=0x7f6c9006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:24:25.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 0x7f6c9006d390 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800b9e0 tx=0x7f6c8800bda0 comp rx=0 tx=0).stop 2023-03-03T16:24:25.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 -- 172.21.15.7:0/3538250432 shutdown_connections 2023-03-03T16:24:25.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c7c0682f0 0x7f6c7c06a7a0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:25.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c9014fda0 0x7f6c9006d390 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:25.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c900abe20 0x7f6c90151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:25.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 --2- 172.21.15.7:0/3538250432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c9014f9d0 0x7f6c9014b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:24:25.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 -- 172.21.15.7:0/3538250432 >> 172.21.15.7:0/3538250432 conn(0x7f6c900c8900 msgr2=0x7f6c900a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:24:25.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 -- 172.21.15.7:0/3538250432 shutdown_connections 2023-03-03T16:24:25.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:24:25.172+0000 7f6c951c2700 1 -- 172.21.15.7:0/3538250432 wait complete. 2023-03-03T16:24:25.175 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:24:25.185 INFO:tasks.mds_thrash.fs.[b]:waiting for 99 secs before thrashing 2023-03-03T16:24:45.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:24:45.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:00.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:25:01.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.112+0000 7f92fef7d700 1 Processor -- start 2023-03-03T16:25:01.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.112+0000 7f92fef7d700 1 -- start start 2023-03-03T16:25:01.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.113+0000 7f92fef7d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f92f806e020 0x7f92f8071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:01.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.113+0000 7f92fef7d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f8072520 0x7f92f8151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:01.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.113+0000 7f92fef7d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f8154910 con 0x7f92f8155390 2023-03-03T16:25:01.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.113+0000 7f92fef7d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f8070b90 con 0x7f92f806e020 2023-03-03T16:25:01.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.113+0000 7f92fef7d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f8151320 con 0x7f92f8072520 2023-03-03T16:25:01.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.113+0000 7f92fcd19700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f92f8155390 0x7f92f80c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50950/0 (socket says 172.21.15.7:50950) 2023-03-03T16:25:01.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.113+0000 7f92fcd19700 1 -- 172.21.15.7:0/710625442 learned_addr learned my addr 172.21.15.7:0/710625442 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:25:01.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f7fff700 1 --2- 172.21.15.7:0/710625442 >> v2:172.21.15.143:3300/0 conn(0x7f92f806e020 0x7f92f8071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:01.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165065321 0 0) 0x7f92f8154910 con 0x7f92f8155390 2023-03-03T16:25:01.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e80035b0 con 0x7f92f8155390 2023-03-03T16:25:01.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92fd51a700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f8072520 0x7f92f8151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:01.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2818526806 0 0) 0x7f92e80035b0 con 0x7f92f8155390 2023-03-03T16:25:01.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f8072520 msgr2=0x7f92f8151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:25:01.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f8072520 0x7f92f8151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 >> v2:172.21.15.143:3300/0 conn(0x7f92f806e020 msgr2=0x7f92f8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:25:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 --2- 172.21.15.7:0/710625442 >> v2:172.21.15.143:3300/0 conn(0x7f92f806e020 0x7f92f8071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f80c4540 con 0x7f92f8155390 2023-03-03T16:25:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f92ec001670 con 0x7f92f8155390 2023-03-03T16:25:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 >> v1:172.21.15.7:6789/0 conn(0x7f92f8155390 legacy=0x7f92f80c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 0x7f92e8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f8072520 0x7f92e8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92e8004d90 0x7f92e8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f8070b90 con 0x7f92e8003a70 2023-03-03T16:25:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f8151320 con 0x7f92e8004d90 2023-03-03T16:25:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e800bac0 con 0x7f92f8072520 2023-03-03T16:25:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92fcd19700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 0x7f92e8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92f7fff700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f8072520 0x7f92e8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92fd51a700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92e8004d90 0x7f92e8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92fcd19700 1 -- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f8072520 msgr2=0x7f92e8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:25:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92fcd19700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f8072520 0x7f92e8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92fcd19700 1 -- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92e8004d90 msgr2=0x7f92e8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:25:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92fcd19700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92e8004d90 0x7f92e8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.114+0000 7f92fcd19700 1 -- 172.21.15.7:0/710625442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e80035b0 con 0x7f92e8003a70 2023-03-03T16:25:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92fcd19700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 0x7f92e8003e80 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f92ec007f10 tx=0x7f92e8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8192a99f84d7de server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:25:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec007b70 con 0x7f92e8003a70 2023-03-03T16:25:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec00cbf0 con 0x7f92e8003a70 2023-03-03T16:25:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92f77fe700 1 -- 172.21.15.7:0/710625442 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec005960 con 0x7f92e8003a70 2023-03-03T16:25:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92fef7d700 1 -- 172.21.15.7:0/710625442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 msgr2=0x7f92e8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92fef7d700 1 --2- 172.21.15.7:0/710625442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 0x7f92e8003e80 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f92ec007f10 tx=0x7f92e8005880 comp rx=0 tx=0).stop 2023-03-03T16:25:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92fd51a700 1 -- 172.21.15.7:0/710625442 reap_dead start 2023-03-03T16:25:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92fef7d700 1 -- 172.21.15.7:0/710625442 shutdown_connections 2023-03-03T16:25:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92fef7d700 1 -- 172.21.15.7:0/710625442 >> 172.21.15.7:0/710625442 conn(0x7f92f806c110 msgr2=0x7f92f806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:25:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92fef7d700 1 -- 172.21.15.7:0/710625442 shutdown_connections 2023-03-03T16:25:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.115+0000 7f92fef7d700 1 -- 172.21.15.7:0/710625442 wait complete. 2023-03-03T16:25:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.117+0000 7f92fef7d700 1 Processor -- start 2023-03-03T16:25:01.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.117+0000 7f92fef7d700 1 -- start start 2023-03-03T16:25:01.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92fef7d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 0x7f92f80abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:01.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92fef7d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f814e610 0x7f92f8144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92fef7d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f81451b0 0x7f92f81e5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92fef7d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f814d7e0 con 0x7f92e8003a70 2023-03-03T16:25:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92fef7d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f814b9e0 con 0x7f92f81451b0 2023-03-03T16:25:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92fef7d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f814bc30 con 0x7f92f814e610 2023-03-03T16:25:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92fcd19700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 0x7f92f80abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92fcd19700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 0x7f92f80abd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56982/0 (socket says 172.21.15.7:56982) 2023-03-03T16:25:01.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92fcd19700 1 -- 172.21.15.7:0/658832048 learned_addr learned my addr 172.21.15.7:0/658832048 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:25:01.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92f7fff700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f814e610 0x7f92f8144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:01.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.118+0000 7f92fd51a700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f81451b0 0x7f92f81e5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:01.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92fcd19700 1 -- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f814e610 msgr2=0x7f92f8144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:01.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92fcd19700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f814e610 0x7f92f8144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:01.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92fcd19700 1 -- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f81451b0 msgr2=0x7f92f81e5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:01.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92fcd19700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f81451b0 0x7f92f81e5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:01.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92fcd19700 1 -- 172.21.15.7:0/658832048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f8070540 con 0x7f92e8003a70 2023-03-03T16:25:01.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92fcd19700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 0x7f92f80abd50 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00c6a0 tx=0x7f92ec001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:25:01.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92f57fa700 1 -- 172.21.15.7:0/658832048 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec004a80 con 0x7f92e8003a70 2023-03-03T16:25:01.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92f57fa700 1 -- 172.21.15.7:0/658832048 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec004be0 con 0x7f92e8003a70 2023-03-03T16:25:01.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92fef7d700 1 -- 172.21.15.7:0/658832048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f80702a0 con 0x7f92e8003a70 2023-03-03T16:25:01.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92f57fa700 1 -- 172.21.15.7:0/658832048 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0070d0 con 0x7f92e8003a70 2023-03-03T16:25:01.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.119+0000 7f92fef7d700 1 -- 172.21.15.7:0/658832048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f8070820 con 0x7f92e8003a70 2023-03-03T16:25:01.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.120+0000 7f92fef7d700 1 -- 172.21.15.7:0/658832048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f8074bb0 con 0x7f92e8003a70 2023-03-03T16:25:01.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.121+0000 7f92f57fa700 1 -- 172.21.15.7:0/658832048 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f92ec02e070 con 0x7f92e8003a70 2023-03-03T16:25:01.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.121+0000 7f92f57fa700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92dc0682f0 0x7f92dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:01.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.121+0000 7f92f57fa700 1 -- 172.21.15.7:0/658832048 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f92ec0872c0 con 0x7f92e8003a70 2023-03-03T16:25:01.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.121+0000 7f92f7fff700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92dc0682f0 0x7f92dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:01.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.122+0000 7f92f7fff700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92dc0682f0 0x7f92dc06a7a0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f92f814e9e0 tx=0x7f92e4013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:25:01.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.126+0000 7f92f57fa700 1 -- 172.21.15.7:0/658832048 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f92ec053720 con 0x7f92e8003a70 2023-03-03T16:25:01.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.364+0000 7f92fef7d700 1 -- 172.21.15.7:0/658832048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92f81e57d0 con 0x7f92e8003a70 2023-03-03T16:25:01.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.365+0000 7f92f57fa700 1 -- 172.21.15.7:0/658832048 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f92ec056d40 con 0x7f92e8003a70 2023-03-03T16:25:01.365 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:25:01.366 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:25:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 -- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92dc0682f0 msgr2=0x7f92dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92dc0682f0 0x7f92dc06a7a0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f92f814e9e0 tx=0x7f92e4013040 comp rx=0 tx=0).stop 2023-03-03T16:25:01.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 -- 172.21.15.7:0/658832048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 msgr2=0x7f92f80abd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 0x7f92f80abd50 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00c6a0 tx=0x7f92ec001930 comp rx=0 tx=0).stop 2023-03-03T16:25:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 -- 172.21.15.7:0/658832048 shutdown_connections 2023-03-03T16:25:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92f81451b0 0x7f92f81e5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92dc0682f0 0x7f92dc06a7a0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92e8003a70 0x7f92f80abd50 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 --2- 172.21.15.7:0/658832048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f92f814e610 0x7f92f8144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:01.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 -- 172.21.15.7:0/658832048 >> 172.21.15.7:0/658832048 conn(0x7f92f806c110 msgr2=0x7f92f814ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:25:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 -- 172.21.15.7:0/658832048 shutdown_connections 2023-03-03T16:25:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:01.367+0000 7f92fef7d700 1 -- 172.21.15.7:0/658832048 wait complete. 2023-03-03T16:25:01.369 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:25:01.379 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before thrashing 2023-03-03T16:25:04.380 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:25:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.519+0000 7f088d2c1700 1 Processor -- start 2023-03-03T16:25:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.519+0000 7f088d2c1700 1 -- start start 2023-03-03T16:25:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.520+0000 7f088d2c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08880c5cb0 0x7f088806a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.520+0000 7f088d2c1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08881558f0 con 0x7f08880a6370 2023-03-03T16:25:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.520+0000 7f088d2c1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08880a6180 con 0x7f088806a2b0 2023-03-03T16:25:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.520+0000 7f088d2c1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088806bab0 con 0x7f08880c5cb0 2023-03-03T16:25:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.520+0000 7f08867fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f08880a6370 0x7f088815abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50972/0 (socket says 172.21.15.7:50972) 2023-03-03T16:25:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.520+0000 7f08867fc700 1 -- 172.21.15.7:0/2993838662 learned_addr learned my addr 172.21.15.7:0/2993838662 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:25:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f08877fe700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08880c5cb0 0x7f088806a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318594237 0 0) 0x7f08881558f0 con 0x7f08880a6370 2023-03-03T16:25:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08700035b0 con 0x7f08880a6370 2023-03-03T16:25:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82528944 0 0) 0x7f08880a6180 con 0x7f088806a2b0 2023-03-03T16:25:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08881558f0 con 0x7f088806a2b0 2023-03-03T16:25:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1773528690 0 0) 0x7f08700035b0 con 0x7f08880a6370 2023-03-03T16:25:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08880c5cb0 msgr2=0x7f088806a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:25:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08880c5cb0 0x7f088806a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 >> v1:172.21.15.143:6789/0 conn(0x7f088806a2b0 legacy=0x7f0888155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08880a5ea0 con 0x7f08880a6370 2023-03-03T16:25:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0878001670 con 0x7f08880a6370 2023-03-03T16:25:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 >> v1:172.21.15.7:6789/0 conn(0x7f08880a6370 legacy=0x7f088815abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0870003c30 0x7f0870004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08880c5cb0 0x7f0870004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0870004d90 0x7f0870005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087000bba0 con 0x7f0870004d90 2023-03-03T16:25:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087000bce0 con 0x7f0870003c30 2023-03-03T16:25:04.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087000be70 con 0x7f08880c5cb0 2023-03-03T16:25:04.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f08877fe700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0870004d90 0x7f0870005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:04.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f0886ffd700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0870003c30 0x7f0870004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:04.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f08867fc700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08880c5cb0 0x7f0870004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:04.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f08877fe700 1 -- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08880c5cb0 msgr2=0x7f0870004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:25:04.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f08877fe700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f08880c5cb0 0x7f0870004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:04.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f08877fe700 1 -- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0870003c30 msgr2=0x7f0870004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:25:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.521+0000 7f08877fe700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0870003c30 0x7f0870004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f08877fe700 1 -- 172.21.15.7:0/2993838662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08700035b0 con 0x7f0870004d90 2023-03-03T16:25:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f08877fe700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0870004d90 0x7f0870005200 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f0880009fc0 tx=0x7f0870005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989f7b35ba2bfd3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:25:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088000ec30 con 0x7f0870004d90 2023-03-03T16:25:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088000f040 con 0x7f0870004d90 2023-03-03T16:25:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f0885ffb700 1 -- 172.21.15.7:0/2993838662 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880014480 con 0x7f0870004d90 2023-03-03T16:25:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f088d2c1700 1 -- 172.21.15.7:0/2993838662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0870004d90 msgr2=0x7f0870005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f088d2c1700 1 --2- 172.21.15.7:0/2993838662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0870004d90 0x7f0870005200 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f0880009fc0 tx=0x7f0870005880 comp rx=0 tx=0).stop 2023-03-03T16:25:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f08877fe700 1 -- 172.21.15.7:0/2993838662 reap_dead start 2023-03-03T16:25:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f088d2c1700 1 -- 172.21.15.7:0/2993838662 shutdown_connections 2023-03-03T16:25:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f088d2c1700 1 -- 172.21.15.7:0/2993838662 >> 172.21.15.7:0/2993838662 conn(0x7f08880a9430 msgr2=0x7f08880c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:25:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f088d2c1700 1 -- 172.21.15.7:0/2993838662 shutdown_connections 2023-03-03T16:25:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.522+0000 7f088d2c1700 1 -- 172.21.15.7:0/2993838662 wait complete. 2023-03-03T16:25:04.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.524+0000 7f088d2c1700 1 Processor -- start 2023-03-03T16:25:04.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.524+0000 7f088d2c1700 1 -- start start 2023-03-03T16:25:04.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f088d2c1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0888151930 0x7f08881e7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:04.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f088d2c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0888151d00 0x7f08881e8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f088d2c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08881e8a50 0x7f0888153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f088d2c1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0888072060 con 0x7f0888151930 2023-03-03T16:25:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f088d2c1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08881eb2d0 con 0x7f08881e8a50 2023-03-03T16:25:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f088d2c1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08881eb520 con 0x7f0888151d00 2023-03-03T16:25:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f0886ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0888151930 0x7f08881e7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f0886ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0888151930 0x7f08881e7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57002/0 (socket says 172.21.15.7:57002) 2023-03-03T16:25:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f0886ffd700 1 -- 172.21.15.7:0/2720486515 learned_addr learned my addr 172.21.15.7:0/2720486515 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:25:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f08877fe700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08881e8a50 0x7f0888153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.525+0000 7f08867fc700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0888151d00 0x7f08881e8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f0886ffd700 1 -- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0888151d00 msgr2=0x7f08881e8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f0886ffd700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0888151d00 0x7f08881e8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f0886ffd700 1 -- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08881e8a50 msgr2=0x7f0888153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f0886ffd700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08881e8a50 0x7f0888153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f0886ffd700 1 -- 172.21.15.7:0/2720486515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0888153160 con 0x7f0888151930 2023-03-03T16:25:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f08877fe700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08881e8a50 0x7f0888153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:25:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f0886ffd700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0888151930 0x7f08881e7fd0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f087c00f840 tx=0x7f087c00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:25:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f086f7fe700 1 -- 172.21.15.7:0/2720486515 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c0052e0 con 0x7f0888151930 2023-03-03T16:25:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f086f7fe700 1 -- 172.21.15.7:0/2720486515 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087c005480 con 0x7f0888151930 2023-03-03T16:25:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f088d2c1700 1 -- 172.21.15.7:0/2720486515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0888152e80 con 0x7f0888151930 2023-03-03T16:25:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f086f7fe700 1 -- 172.21.15.7:0/2720486515 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087c0038c0 con 0x7f0888151930 2023-03-03T16:25:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.526+0000 7f088d2c1700 1 -- 172.21.15.7:0/2720486515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08880c1500 con 0x7f0888151930 2023-03-03T16:25:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.527+0000 7f088d2c1700 1 -- 172.21.15.7:0/2720486515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08881ec1e0 con 0x7f0888151930 2023-03-03T16:25:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.528+0000 7f086f7fe700 1 -- 172.21.15.7:0/2720486515 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f087c003a20 con 0x7f0888151930 2023-03-03T16:25:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.528+0000 7f086f7fe700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 0x7f087406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:25:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.528+0000 7f086f7fe700 1 -- 172.21.15.7:0/2720486515 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f087c086210 con 0x7f0888151930 2023-03-03T16:25:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.528+0000 7f08867fc700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 0x7f087406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:25:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.529+0000 7f08867fc700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 0x7f087406a7a0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f0878000c00 tx=0x7f0878009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:25:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.533+0000 7f086f7fe700 1 -- 172.21.15.7:0/2720486515 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f08881ec1e0 con 0x7f0888151930 2023-03-03T16:25:04.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.772+0000 7f088d2c1700 1 -- 172.21.15.7:0/2720486515 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08881ec1e0 con 0x7f0888151930 2023-03-03T16:25:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.773+0000 7f086f7fe700 1 -- 172.21.15.7:0/2720486515 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f08881ec1e0 con 0x7f0888151930 2023-03-03T16:25:04.773 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:25:04.774 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:25:04.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 -- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 msgr2=0x7f087406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 0x7f087406a7a0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f0878000c00 tx=0x7f0878009040 comp rx=0 tx=0).stop 2023-03-03T16:25:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 -- 172.21.15.7:0/2720486515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0888151930 msgr2=0x7f08881e7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:25:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0888151930 0x7f08881e7fd0 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7f087c00f840 tx=0x7f087c00fc00 comp rx=0 tx=0).stop 2023-03-03T16:25:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 -- 172.21.15.7:0/2720486515 shutdown_connections 2023-03-03T16:25:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08881e8a50 0x7f0888153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f08740682f0 0x7f087406a7a0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0888151930 0x7f08881e7fd0 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 --2- 172.21.15.7:0/2720486515 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0888151d00 0x7f08881e8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:25:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 -- 172.21.15.7:0/2720486515 >> 172.21.15.7:0/2720486515 conn(0x7f08880a9430 msgr2=0x7f088814f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:25:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 -- 172.21.15.7:0/2720486515 shutdown_connections 2023-03-03T16:25:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:25:04.775+0000 7f088d2c1700 1 -- 172.21.15.7:0/2720486515 wait complete. 2023-03-03T16:25:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:25:04.787 INFO:tasks.mds_thrash.fs.[a]:waiting for 107 secs before thrashing 2023-03-03T16:25:15.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:15.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:45.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:25:45.703 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:04.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:26:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.325+0000 7f075f52a700 1 Processor -- start 2023-03-03T16:26:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.325+0000 7f075f52a700 1 -- start start 2023-03-03T16:26:04.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075f52a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07581530d0 0x7f0758156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:04.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075f52a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07580c2f30 con 0x7f075814f9d0 2023-03-03T16:26:04.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075f52a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07580c31d0 con 0x7f075814fda0 2023-03-03T16:26:04.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075f52a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07581524a0 con 0x7f07581530d0 2023-03-03T16:26:04.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075dac7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f07581530d0 0x7f0758156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:04.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075d2c6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f075814f9d0 0x7f0758071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51060/0 (socket says 172.21.15.7:51060) 2023-03-03T16:26:04.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075d2c6700 1 -- 172.21.15.7:0/390030046 learned_addr learned my addr 172.21.15.7:0/390030046 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:26:04.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075dac7700 1 -- 172.21.15.7:0/390030046 >> v1:172.21.15.143:6789/0 conn(0x7f075814fda0 legacy=0x7f07581529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:04.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075dac7700 1 -- 172.21.15.7:0/390030046 >> v1:172.21.15.7:6789/0 conn(0x7f075814f9d0 legacy=0x7f0758071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:04.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075dac7700 1 -- 172.21.15.7:0/390030046 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07580c34b0 con 0x7f07581530d0 2023-03-03T16:26:04.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f074ffff700 1 -- 172.21.15.7:0/390030046 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045821244 0 0) 0x7f07580c2f30 con 0x7f075814f9d0 2023-03-03T16:26:04.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f075dac7700 1 --2- 172.21.15.7:0/390030046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07581530d0 0x7f0758156430 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f0750009d90 tx=0x7f075000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fac0b6fb197071c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:04.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f074ffff700 1 -- 172.21.15.7:0/390030046 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075000e070 con 0x7f07581530d0 2023-03-03T16:26:04.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.326+0000 7f074ffff700 1 -- 172.21.15.7:0/390030046 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075000bbd0 con 0x7f07581530d0 2023-03-03T16:26:04.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.327+0000 7f074ffff700 1 -- 172.21.15.7:0/390030046 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750012430 con 0x7f07581530d0 2023-03-03T16:26:04.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.327+0000 7f075f52a700 1 -- 172.21.15.7:0/390030046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07581530d0 msgr2=0x7f0758156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:04.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.327+0000 7f075f52a700 1 --2- 172.21.15.7:0/390030046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07581530d0 0x7f0758156430 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f0750009d90 tx=0x7f075000b330 comp rx=0 tx=0).stop 2023-03-03T16:26:04.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.327+0000 7f075f52a700 1 -- 172.21.15.7:0/390030046 shutdown_connections 2023-03-03T16:26:04.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.327+0000 7f075f52a700 1 --2- 172.21.15.7:0/390030046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f07581530d0 0x7f0758156430 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:04.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.327+0000 7f075f52a700 1 -- 172.21.15.7:0/390030046 >> 172.21.15.7:0/390030046 conn(0x7f07580c8900 msgr2=0x7f07580c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:26:04.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.327+0000 7f075f52a700 1 -- 172.21.15.7:0/390030046 shutdown_connections 2023-03-03T16:26:04.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.327+0000 7f075f52a700 1 -- 172.21.15.7:0/390030046 wait complete. 2023-03-03T16:26:04.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.328+0000 7f075f52a700 1 Processor -- start 2023-03-03T16:26:04.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.328+0000 7f075f52a700 1 -- start start 2023-03-03T16:26:04.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075f52a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f075814f9d0 0x7f07580c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:04.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075f52a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075814fda0 0x7f07580be110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:04.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075f52a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07581530d0 0x7f07580be650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:04.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075f52a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07580beb90 con 0x7f075814fda0 2023-03-03T16:26:04.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075f52a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07581549f0 con 0x7f075814f9d0 2023-03-03T16:26:04.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075f52a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07581e7440 con 0x7f07581530d0 2023-03-03T16:26:04.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075cac5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075814fda0 0x7f07580be110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:04.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075cac5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075814fda0 0x7f07580be110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57088/0 (socket says 172.21.15.7:57088) 2023-03-03T16:26:04.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075cac5700 1 -- 172.21.15.7:0/3807758527 learned_addr learned my addr 172.21.15.7:0/3807758527 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:26:04.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075d2c6700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f075814f9d0 0x7f07580c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:04.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.329+0000 7f075dac7700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07581530d0 0x7f07580be650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:04.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f075cac5700 1 -- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07581530d0 msgr2=0x7f07580be650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:04.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f075cac5700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07581530d0 0x7f07580be650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:04.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f075cac5700 1 -- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f075814f9d0 msgr2=0x7f07580c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:04.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f075cac5700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f075814f9d0 0x7f07580c6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:04.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f075cac5700 1 -- 172.21.15.7:0/3807758527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0750009a70 con 0x7f075814fda0 2023-03-03T16:26:04.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f075cac5700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075814fda0 0x7f07580be110 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f075400c4b0 tx=0x7f075400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:04.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f074dffb700 1 -- 172.21.15.7:0/3807758527 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07540052d0 con 0x7f075814fda0 2023-03-03T16:26:04.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f074dffb700 1 -- 172.21.15.7:0/3807758527 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0754005430 con 0x7f075814fda0 2023-03-03T16:26:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f075f52a700 1 -- 172.21.15.7:0/3807758527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07580ac350 con 0x7f075814fda0 2023-03-03T16:26:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f074dffb700 1 -- 172.21.15.7:0/3807758527 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07540038c0 con 0x7f075814fda0 2023-03-03T16:26:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.330+0000 7f075f52a700 1 -- 172.21.15.7:0/3807758527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07580ab630 con 0x7f075814fda0 2023-03-03T16:26:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.331+0000 7f075f52a700 1 -- 172.21.15.7:0/3807758527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0758071fa0 con 0x7f075814fda0 2023-03-03T16:26:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.331+0000 7f074dffb700 1 -- 172.21.15.7:0/3807758527 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0754019030 con 0x7f075814fda0 2023-03-03T16:26:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.332+0000 7f074dffb700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f073c0682f0 0x7f073c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:04.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.332+0000 7f074dffb700 1 -- 172.21.15.7:0/3807758527 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0754085350 con 0x7f075814fda0 2023-03-03T16:26:04.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.332+0000 7f075d2c6700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f073c0682f0 0x7f073c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:04.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.333+0000 7f075d2c6700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f073c0682f0 0x7f073c06a7a0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f0748000f60 tx=0x7f0748004a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:04.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.336+0000 7f074dffb700 1 -- 172.21.15.7:0/3807758527 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f07540516f0 con 0x7f075814fda0 2023-03-03T16:26:04.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.571+0000 7f075f52a700 1 -- 172.21.15.7:0/3807758527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07581e7770 con 0x7f075814fda0 2023-03-03T16:26:04.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.571+0000 7f074dffb700 1 -- 172.21.15.7:0/3807758527 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0754054d10 con 0x7f075814fda0 2023-03-03T16:26:04.572 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:26:04.572 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:26:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 -- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f073c0682f0 msgr2=0x7f073c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f073c0682f0 0x7f073c06a7a0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f0748000f60 tx=0x7f0748004a50 comp rx=0 tx=0).stop 2023-03-03T16:26:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 -- 172.21.15.7:0/3807758527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075814fda0 msgr2=0x7f07580be110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075814fda0 0x7f07580be110 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f075400c4b0 tx=0x7f075400c7c0 comp rx=0 tx=0).stop 2023-03-03T16:26:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 -- 172.21.15.7:0/3807758527 shutdown_connections 2023-03-03T16:26:04.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07581530d0 0x7f07580be650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f075814f9d0 0x7f07580c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f073c0682f0 0x7f073c06a7a0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 --2- 172.21.15.7:0/3807758527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075814fda0 0x7f07580be110 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 -- 172.21.15.7:0/3807758527 >> 172.21.15.7:0/3807758527 conn(0x7f07580c8900 msgr2=0x7f07580c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:26:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 -- 172.21.15.7:0/3807758527 shutdown_connections 2023-03-03T16:26:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:04.573+0000 7f075f52a700 1 -- 172.21.15.7:0/3807758527 wait complete. 2023-03-03T16:26:04.576 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:26:04.585 INFO:tasks.mds_thrash.fs.[b]:waiting for 44 secs before thrashing 2023-03-03T16:26:15.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:15.783 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:45.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:45.915 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:26:48.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:26:48.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.736+0000 7f69e4c48700 1 Processor -- start 2023-03-03T16:26:48.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.736+0000 7f69e4c48700 1 -- start start 2023-03-03T16:26:48.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.737+0000 7f69e4c48700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e006d870 0x7f69e01550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:48.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.737+0000 7f69e4c48700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f69e00c1da0 0x7f69e00c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:48.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.737+0000 7f69e4c48700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e00c3620 con 0x7f69e00c3e30 2023-03-03T16:26:48.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.737+0000 7f69e4c48700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e00c2af0 con 0x7f69e00c1da0 2023-03-03T16:26:48.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.737+0000 7f69e4c48700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e01545e0 con 0x7f69e006d870 2023-03-03T16:26:48.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.737+0000 7f69ded9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f69e00c3e30 0x7f69e0157ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51122/0 (socket says 172.21.15.7:51122) 2023-03-03T16:26:48.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.737+0000 7f69ded9d700 1 -- 172.21.15.7:0/3564301355 learned_addr learned my addr 172.21.15.7:0/3564301355 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:26:48.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.737+0000 7f69ddd9b700 1 --2- 172.21.15.7:0/3564301355 >> v2:172.21.15.143:3300/0 conn(0x7f69e00c1da0 0x7f69e00c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:48.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.737+0000 7f69de59c700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e006d870 0x7f69e01550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:48.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533053027 0 0) 0x7f69e00c3620 con 0x7f69e00c3e30 2023-03-03T16:26:48.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69cc0035b0 con 0x7f69e00c3e30 2023-03-03T16:26:48.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 120615628 0 0) 0x7f69cc0035b0 con 0x7f69e00c3e30 2023-03-03T16:26:48.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e006d870 msgr2=0x7f69e01550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:26:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e006d870 0x7f69e01550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 >> v2:172.21.15.143:3300/0 conn(0x7f69e00c1da0 msgr2=0x7f69e00c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:26:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 --2- 172.21.15.7:0/3564301355 >> v2:172.21.15.143:3300/0 conn(0x7f69e00c1da0 0x7f69e00c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e01548c0 con 0x7f69e00c3e30 2023-03-03T16:26:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f69d80019f0 con 0x7f69e00c3e30 2023-03-03T16:26:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 >> v1:172.21.15.7:6789/0 conn(0x7f69e00c3e30 legacy=0x7f69e0157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69cc003a70 0x7f69cc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 0x7f69cc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e006d870 0x7f69cc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e01545e0 con 0x7f69cc004850 2023-03-03T16:26:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc00ba10 con 0x7f69cc003a70 2023-03-03T16:26:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc00bb50 con 0x7f69e006d870 2023-03-03T16:26:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69ddd9b700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 0x7f69cc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69de59c700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69cc003a70 0x7f69cc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69ded9d700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e006d870 0x7f69cc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69ddd9b700 1 -- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e006d870 msgr2=0x7f69cc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:26:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69ddd9b700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e006d870 0x7f69cc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69ddd9b700 1 -- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69cc003a70 msgr2=0x7f69cc003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:26:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69ddd9b700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69cc003a70 0x7f69cc003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.738+0000 7f69ddd9b700 1 -- 172.21.15.7:0/3564301355 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc0035b0 con 0x7f69cc004850 2023-03-03T16:26:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69ddd9b700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 0x7f69cc004cc0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f69d0009b60 tx=0x7f69cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17693f119fb28939 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d000e6c0 con 0x7f69cc004850 2023-03-03T16:26:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d000e820 con 0x7f69cc004850 2023-03-03T16:26:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69dd59a700 1 -- 172.21.15.7:0/3564301355 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d000eb90 con 0x7f69cc004850 2023-03-03T16:26:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69e4c48700 1 -- 172.21.15.7:0/3564301355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 msgr2=0x7f69cc004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69e4c48700 1 --2- 172.21.15.7:0/3564301355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 0x7f69cc004cc0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f69d0009b60 tx=0x7f69cc005880 comp rx=0 tx=0).stop 2023-03-03T16:26:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69ded9d700 1 -- 172.21.15.7:0/3564301355 reap_dead start 2023-03-03T16:26:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69e4c48700 1 -- 172.21.15.7:0/3564301355 shutdown_connections 2023-03-03T16:26:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69e4c48700 1 -- 172.21.15.7:0/3564301355 >> 172.21.15.7:0/3564301355 conn(0x7f69e006c110 msgr2=0x7f69e006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:26:48.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69e4c48700 1 -- 172.21.15.7:0/3564301355 shutdown_connections 2023-03-03T16:26:48.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.739+0000 7f69e4c48700 1 -- 172.21.15.7:0/3564301355 wait complete. 2023-03-03T16:26:48.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.741+0000 7f69e4c48700 1 Processor -- start 2023-03-03T16:26:48.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.741+0000 7f69e4c48700 1 -- start start 2023-03-03T16:26:48.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69e4c48700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 0x7f69e0157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:48.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69e4c48700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e00ad100 0x7f69e0138d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:48.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69e4c48700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69e0139290 0x7f69e01ed2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:48.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69e4c48700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e014f5d0 con 0x7f69cc004850 2023-03-03T16:26:48.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69e4c48700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e014ec30 con 0x7f69e0139290 2023-03-03T16:26:48.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69e4c48700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e014ef10 con 0x7f69e00ad100 2023-03-03T16:26:48.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69de59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 0x7f69e0157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:48.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69de59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 0x7f69e0157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57150/0 (socket says 172.21.15.7:57150) 2023-03-03T16:26:48.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69de59c700 1 -- 172.21.15.7:0/3174308414 learned_addr learned my addr 172.21.15.7:0/3174308414 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:26:48.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69ded9d700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69e0139290 0x7f69e01ed2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:48.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69ddd9b700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e00ad100 0x7f69e0138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:48.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69de59c700 1 -- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e00ad100 msgr2=0x7f69e0138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69de59c700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e00ad100 0x7f69e0138d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69de59c700 1 -- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69e0139290 msgr2=0x7f69e01ed2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69de59c700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69e0139290 0x7f69e01ed2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69de59c700 1 -- 172.21.15.7:0/3174308414 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e006dc40 con 0x7f69cc004850 2023-03-03T16:26:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69ded9d700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69e0139290 0x7f69e01ed2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:26:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.742+0000 7f69ddd9b700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e00ad100 0x7f69e0138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:26:48.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.743+0000 7f69de59c700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 0x7f69e0157100 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f69c8011aa0 tx=0x7f69c8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.743+0000 7f69d6ffd700 1 -- 172.21.15.7:0/3174308414 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c800f980 con 0x7f69cc004850 2023-03-03T16:26:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.743+0000 7f69d6ffd700 1 -- 172.21.15.7:0/3174308414 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c800fae0 con 0x7f69cc004850 2023-03-03T16:26:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.743+0000 7f69e4c48700 1 -- 172.21.15.7:0/3174308414 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e0154ba0 con 0x7f69cc004850 2023-03-03T16:26:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.743+0000 7f69d6ffd700 1 -- 172.21.15.7:0/3174308414 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c800fe90 con 0x7f69cc004850 2023-03-03T16:26:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.743+0000 7f69e4c48700 1 -- 172.21.15.7:0/3174308414 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e00c38e0 con 0x7f69cc004850 2023-03-03T16:26:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.744+0000 7f69e4c48700 1 -- 172.21.15.7:0/3174308414 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e01edfe0 con 0x7f69cc004850 2023-03-03T16:26:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.744+0000 7f69d6ffd700 1 -- 172.21.15.7:0/3174308414 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f69c8014440 con 0x7f69cc004850 2023-03-03T16:26:48.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.745+0000 7f69d6ffd700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69c00682f0 0x7f69c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:48.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.745+0000 7f69d6ffd700 1 -- 172.21.15.7:0/3174308414 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f69c80869a0 con 0x7f69cc004850 2023-03-03T16:26:48.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.745+0000 7f69ddd9b700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69c00682f0 0x7f69c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:48.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.745+0000 7f69ddd9b700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69c00682f0 0x7f69c006a7a0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f69d00090b0 tx=0x7f69d0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:48.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.749+0000 7f69d6ffd700 1 -- 172.21.15.7:0/3174308414 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f69e01edfe0 con 0x7f69cc004850 2023-03-03T16:26:48.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.986+0000 7f69e4c48700 1 -- 172.21.15.7:0/3174308414 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69e01edfe0 con 0x7f69cc004850 2023-03-03T16:26:48.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.986+0000 7f69d6ffd700 1 -- 172.21.15.7:0/3174308414 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f69e01edfe0 con 0x7f69cc004850 2023-03-03T16:26:48.987 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:26:48.987 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:26:48.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 -- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69c00682f0 msgr2=0x7f69c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:48.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69c00682f0 0x7f69c006a7a0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f69d00090b0 tx=0x7f69d0008030 comp rx=0 tx=0).stop 2023-03-03T16:26:48.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 -- 172.21.15.7:0/3174308414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 msgr2=0x7f69e0157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:48.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 0x7f69e0157100 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f69c8011aa0 tx=0x7f69c8011e60 comp rx=0 tx=0).stop 2023-03-03T16:26:48.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 -- 172.21.15.7:0/3174308414 shutdown_connections 2023-03-03T16:26:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69e0139290 0x7f69e01ed2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69c00682f0 0x7f69c006a7a0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69cc004850 0x7f69e0157100 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 --2- 172.21.15.7:0/3174308414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69e00ad100 0x7f69e0138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 -- 172.21.15.7:0/3174308414 >> 172.21.15.7:0/3174308414 conn(0x7f69e006c110 msgr2=0x7f69e0071b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:26:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 -- 172.21.15.7:0/3174308414 shutdown_connections 2023-03-03T16:26:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:48.988+0000 7f69e4c48700 1 -- 172.21.15.7:0/3174308414 wait complete. 2023-03-03T16:26:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:26:49.001 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T16:26:51.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:26:51.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.926+0000 7fc55b8c2700 1 Processor -- start 2023-03-03T16:26:51.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.926+0000 7fc55b8c2700 1 -- start start 2023-03-03T16:26:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.927+0000 7fc55b8c2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc554151e70 0x7fc55406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.927+0000 7fc55b8c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc554157ba0 0x7fc554158020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.927+0000 7fc55b8c2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55406e2d0 con 0x7fc55406a7f0 2023-03-03T16:26:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.927+0000 7fc55b8c2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5540c2720 con 0x7fc554151e70 2023-03-03T16:26:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.927+0000 7fc55b8c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc554066f70 con 0x7fc554157ba0 2023-03-03T16:26:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.927+0000 7fc558e5d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc55406a7f0 0x7fc5540c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51138/0 (socket says 172.21.15.7:51138) 2023-03-03T16:26:51.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.927+0000 7fc558e5d700 1 -- 172.21.15.7:0/2608502051 learned_addr learned my addr 172.21.15.7:0/2608502051 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:26:51.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc559e5f700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc554157ba0 0x7fc554158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:51.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc55965e700 1 --2- 172.21.15.7:0/2608502051 >> v2:172.21.15.143:3300/0 conn(0x7fc554151e70 0x7fc55406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:51.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043459607 0 0) 0x7fc55406e2d0 con 0x7fc55406a7f0 2023-03-03T16:26:51.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5400035b0 con 0x7fc55406a7f0 2023-03-03T16:26:51.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4208304527 0 0) 0x7fc5400035b0 con 0x7fc55406a7f0 2023-03-03T16:26:51.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc554157ba0 msgr2=0x7fc554158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:26:51.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc554157ba0 0x7fc554158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:51.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 >> v2:172.21.15.143:3300/0 conn(0x7fc554151e70 msgr2=0x7fc55406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:26:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 --2- 172.21.15.7:0/2608502051 >> v2:172.21.15.143:3300/0 conn(0x7fc554151e70 0x7fc55406a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5540c53b0 con 0x7fc55406a7f0 2023-03-03T16:26:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc550001670 con 0x7fc55406a7f0 2023-03-03T16:26:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 >> v1:172.21.15.7:6789/0 conn(0x7fc55406a7f0 legacy=0x7fc5540c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc540003a70 0x7fc540003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc540004850 0x7fc540004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc554157ba0 0x7fc540005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc554066f70 con 0x7fc540003a70 2023-03-03T16:26:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54000ba10 con 0x7fc540004850 2023-03-03T16:26:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54000bb80 con 0x7fc554157ba0 2023-03-03T16:26:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc55965e700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc540003a70 0x7fc540003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc558e5d700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc540004850 0x7fc540004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc559e5f700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc554157ba0 0x7fc540005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc55965e700 1 -- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc554157ba0 msgr2=0x7fc540005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:26:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc55965e700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc554157ba0 0x7fc540005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:51.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc55965e700 1 -- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc540004850 msgr2=0x7fc540004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:26:51.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.928+0000 7fc55965e700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc540004850 0x7fc540004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:51.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.929+0000 7fc55965e700 1 -- 172.21.15.7:0/2608502051 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5400035b0 con 0x7fc540003a70 2023-03-03T16:26:51.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.929+0000 7fc559e5f700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc554157ba0 0x7fc540005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:26:51.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.929+0000 7fc55965e700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc540003a70 0x7fc540003e80 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7fc544009b60 tx=0x7fc540005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6e9388f979939e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:51.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.929+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54400e6c0 con 0x7fc540003a70 2023-03-03T16:26:51.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.929+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54400e820 con 0x7fc540003a70 2023-03-03T16:26:51.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.929+0000 7fc54bfff700 1 -- 172.21.15.7:0/2608502051 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54400eb90 con 0x7fc540003a70 2023-03-03T16:26:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.930+0000 7fc55b8c2700 1 -- 172.21.15.7:0/2608502051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc540003a70 msgr2=0x7fc540003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.930+0000 7fc55b8c2700 1 --2- 172.21.15.7:0/2608502051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc540003a70 0x7fc540003e80 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7fc544009b60 tx=0x7fc540005880 comp rx=0 tx=0).stop 2023-03-03T16:26:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.930+0000 7fc559e5f700 1 -- 172.21.15.7:0/2608502051 reap_dead start 2023-03-03T16:26:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.930+0000 7fc55b8c2700 1 -- 172.21.15.7:0/2608502051 shutdown_connections 2023-03-03T16:26:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.930+0000 7fc55b8c2700 1 -- 172.21.15.7:0/2608502051 >> 172.21.15.7:0/2608502051 conn(0x7fc5540a9430 msgr2=0x7fc5540c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:26:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.930+0000 7fc55b8c2700 1 -- 172.21.15.7:0/2608502051 shutdown_connections 2023-03-03T16:26:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.930+0000 7fc55b8c2700 1 -- 172.21.15.7:0/2608502051 wait complete. 2023-03-03T16:26:51.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.931+0000 7fc55b8c2700 1 Processor -- start 2023-03-03T16:26:51.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.931+0000 7fc55b8c2700 1 -- start start 2023-03-03T16:26:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc55b8c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc540003a70 0x7fc5541defa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc55b8c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5540aba90 0x7fc554149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc55b8c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc55414e7b0 0x7fc55414a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc55b8c2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55414dc90 con 0x7fc5540aba90 2023-03-03T16:26:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc55b8c2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5541ea170 con 0x7fc540003a70 2023-03-03T16:26:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc55b8c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55414d210 con 0x7fc55414e7b0 2023-03-03T16:26:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc558e5d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5540aba90 0x7fc554149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:51.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc558e5d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5540aba90 0x7fc554149f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57172/0 (socket says 172.21.15.7:57172) 2023-03-03T16:26:51.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc55965e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc540003a70 0x7fc5541defa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:51.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc558e5d700 1 -- 172.21.15.7:0/388800534 learned_addr learned my addr 172.21.15.7:0/388800534 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:26:51.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.932+0000 7fc559e5f700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc55414e7b0 0x7fc55414a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:51.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc558e5d700 1 -- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc55414e7b0 msgr2=0x7fc55414a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:51.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc558e5d700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc55414e7b0 0x7fc55414a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:51.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc558e5d700 1 -- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc540003a70 msgr2=0x7fc5541defa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc558e5d700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc540003a70 0x7fc5541defa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc558e5d700 1 -- 172.21.15.7:0/388800534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55406dd10 con 0x7fc5540aba90 2023-03-03T16:26:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc558e5d700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5540aba90 0x7fc554149f80 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fc550000fb0 tx=0x7fc5500075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc549ffb700 1 -- 172.21.15.7:0/388800534 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550011580 con 0x7fc5540aba90 2023-03-03T16:26:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc549ffb700 1 -- 172.21.15.7:0/388800534 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc550011700 con 0x7fc5540aba90 2023-03-03T16:26:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc55b8c2700 1 -- 172.21.15.7:0/388800534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55406da30 con 0x7fc5540aba90 2023-03-03T16:26:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc549ffb700 1 -- 172.21.15.7:0/388800534 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550011ab0 con 0x7fc5540aba90 2023-03-03T16:26:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.933+0000 7fc55b8c2700 1 -- 172.21.15.7:0/388800534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55406dff0 con 0x7fc5540aba90 2023-03-03T16:26:51.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.934+0000 7fc55b8c2700 1 -- 172.21.15.7:0/388800534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc554074bb0 con 0x7fc5540aba90 2023-03-03T16:26:51.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.935+0000 7fc549ffb700 1 -- 172.21.15.7:0/388800534 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc550011c10 con 0x7fc5540aba90 2023-03-03T16:26:51.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.935+0000 7fc549ffb700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5380682f0 0x7fc53806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:26:51.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.935+0000 7fc549ffb700 1 -- 172.21.15.7:0/388800534 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc5500861a0 con 0x7fc5540aba90 2023-03-03T16:26:51.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.935+0000 7fc55965e700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5380682f0 0x7fc53806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:26:51.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.936+0000 7fc55965e700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5380682f0 0x7fc53806a7a0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fc5440090b0 tx=0x7fc544008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:26:51.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:51.940+0000 7fc549ffb700 1 -- 172.21.15.7:0/388800534 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc550052540 con 0x7fc5540aba90 2023-03-03T16:26:52.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.176+0000 7fc55b8c2700 1 -- 172.21.15.7:0/388800534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5541ea2b0 con 0x7fc5540aba90 2023-03-03T16:26:52.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.176+0000 7fc549ffb700 1 -- 172.21.15.7:0/388800534 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc5541ea2b0 con 0x7fc5540aba90 2023-03-03T16:26:52.177 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:26:52.177 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:26:52.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 -- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5380682f0 msgr2=0x7fc53806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:52.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5380682f0 0x7fc53806a7a0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fc5440090b0 tx=0x7fc544008030 comp rx=0 tx=0).stop 2023-03-03T16:26:52.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 -- 172.21.15.7:0/388800534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5540aba90 msgr2=0x7fc554149f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:26:52.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5540aba90 0x7fc554149f80 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fc550000fb0 tx=0x7fc5500075f0 comp rx=0 tx=0).stop 2023-03-03T16:26:52.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 -- 172.21.15.7:0/388800534 shutdown_connections 2023-03-03T16:26:52.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc55414e7b0 0x7fc55414a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:52.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc540003a70 0x7fc5541defa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:52.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc5380682f0 0x7fc53806a7a0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:52.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 --2- 172.21.15.7:0/388800534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc5540aba90 0x7fc554149f80 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:26:52.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 -- 172.21.15.7:0/388800534 >> 172.21.15.7:0/388800534 conn(0x7fc5540a9430 msgr2=0x7fc554151180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:26:52.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 -- 172.21.15.7:0/388800534 shutdown_connections 2023-03-03T16:26:52.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:26:52.178+0000 7fc55b8c2700 1 -- 172.21.15.7:0/388800534 wait complete. 2023-03-03T16:26:52.180 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:26:52.190 INFO:tasks.mds_thrash.fs.[a]:waiting for 77 secs before thrashing 2023-03-03T16:27:16.051 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:16.054 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:46.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:46.193 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:27:53.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:27:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.142+0000 7f9a46e1a700 1 Processor -- start 2023-03-03T16:27:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.142+0000 7f9a46e1a700 1 -- start start 2023-03-03T16:27:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a46e1a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a40151ab0 0x7f9a400736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a46e1a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9a4006c5b0 0x7f9a40157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a46e1a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4006d780 con 0x7f9a40073c30 2023-03-03T16:27:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a46e1a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4006cb70 con 0x7f9a40151ab0 2023-03-03T16:27:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a46e1a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a4006ce50 con 0x7f9a4006c5b0 2023-03-03T16:27:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a44bb6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a40151ab0 0x7f9a400736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a44bb6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9a40151ab0 0x7f9a400736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57218/0 (socket says 172.21.15.7:57218) 2023-03-03T16:27:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a44bb6700 1 -- 172.21.15.7:0/3946419745 learned_addr learned my addr 172.21.15.7:0/3946419745 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:27:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a453b7700 1 --2- 172.21.15.7:0/3946419745 >> v2:172.21.15.143:3300/0 conn(0x7f9a4006c5b0 0x7f9a40157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a44bb6700 1 -- 172.21.15.7:0/3946419745 >> v2:172.21.15.143:3300/0 conn(0x7f9a4006c5b0 msgr2=0x7f9a40157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:27:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a44bb6700 1 --2- 172.21.15.7:0/3946419745 >> v2:172.21.15.143:3300/0 conn(0x7f9a4006c5b0 0x7f9a40157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a44bb6700 1 -- 172.21.15.7:0/3946419745 >> v1:172.21.15.143:6789/0 conn(0x7f9a40073c30 legacy=0x7f9a4006bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:27:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.143+0000 7f9a44bb6700 1 -- 172.21.15.7:0/3946419745 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4006d130 con 0x7f9a40151ab0 2023-03-03T16:27:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a44bb6700 1 --2- 172.21.15.7:0/3946419745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 0x7f9a400736f0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f9a40074080 tx=0x7f9a34009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53759c264ee832d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:27:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a3f7fe700 1 -- 172.21.15.7:0/3946419745 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a34004230 con 0x7f9a40151ab0 2023-03-03T16:27:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a3f7fe700 1 -- 172.21.15.7:0/3946419745 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a34004390 con 0x7f9a40151ab0 2023-03-03T16:27:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a3f7fe700 1 -- 172.21.15.7:0/3946419745 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a340036a0 con 0x7f9a40151ab0 2023-03-03T16:27:53.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3946419745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 msgr2=0x7f9a400736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:53.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a46e1a700 1 --2- 172.21.15.7:0/3946419745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 0x7f9a400736f0 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f9a40074080 tx=0x7f9a34009bd0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3946419745 shutdown_connections 2023-03-03T16:27:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a46e1a700 1 --2- 172.21.15.7:0/3946419745 >> v2:172.21.15.143:3300/0 conn(0x7f9a4006c5b0 0x7f9a40157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a46e1a700 1 --2- 172.21.15.7:0/3946419745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 0x7f9a400736f0 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3946419745 >> 172.21.15.7:0/3946419745 conn(0x7f9a400a94d0 msgr2=0x7f9a400a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:27:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3946419745 shutdown_connections 2023-03-03T16:27:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.144+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3946419745 wait complete. 2023-03-03T16:27:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.145+0000 7f9a46e1a700 1 Processor -- start 2023-03-03T16:27:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.146+0000 7f9a46e1a700 1 -- start start 2023-03-03T16:27:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.146+0000 7f9a46e1a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a4006c5b0 0x7f9a40146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.146+0000 7f9a46e1a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a40073c30 0x7f9a401ec8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.146+0000 7f9a46e1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 0x7f9a400c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.146+0000 7f9a46e1a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4014bce0 con 0x7f9a40151ab0 2023-03-03T16:27:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.146+0000 7f9a46e1a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4014b060 con 0x7f9a40073c30 2023-03-03T16:27:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.146+0000 7f9a46e1a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4014b340 con 0x7f9a4006c5b0 2023-03-03T16:27:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a453b7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 0x7f9a400c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:53.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a453b7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 0x7f9a400c61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57228/0 (socket says 172.21.15.7:57228) 2023-03-03T16:27:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a453b7700 1 -- 172.21.15.7:0/3587908108 learned_addr learned my addr 172.21.15.7:0/3587908108 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:27:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a3ffff700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a40073c30 0x7f9a401ec8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a44bb6700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a4006c5b0 0x7f9a40146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a453b7700 1 -- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a4006c5b0 msgr2=0x7f9a40146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a453b7700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a4006c5b0 0x7f9a40146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a453b7700 1 -- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a40073c30 msgr2=0x7f9a401ec8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a453b7700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a40073c30 0x7f9a401ec8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a453b7700 1 -- 172.21.15.7:0/3587908108 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a34009550 con 0x7f9a40151ab0 2023-03-03T16:27:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a453b7700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 0x7f9a400c61b0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f9a38007e30 tx=0x7f9a3800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:27:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a3d7fa700 1 -- 172.21.15.7:0/3587908108 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3800ed30 con 0x7f9a40151ab0 2023-03-03T16:27:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a3d7fa700 1 -- 172.21.15.7:0/3587908108 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3800ee90 con 0x7f9a40151ab0 2023-03-03T16:27:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3587908108 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a40151e80 con 0x7f9a40151ab0 2023-03-03T16:27:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a3d7fa700 1 -- 172.21.15.7:0/3587908108 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38014660 con 0x7f9a40151ab0 2023-03-03T16:27:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.147+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3587908108 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4006d410 con 0x7f9a40151ab0 2023-03-03T16:27:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.148+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3587908108 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a40152010 con 0x7f9a40151ab0 2023-03-03T16:27:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.149+0000 7f9a3d7fa700 1 -- 172.21.15.7:0/3587908108 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9a38012050 con 0x7f9a40151ab0 2023-03-03T16:27:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.149+0000 7f9a3d7fa700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a240682f0 0x7f9a2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:27:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.149+0000 7f9a3d7fa700 1 -- 172.21.15.7:0/3587908108 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9a380866b0 con 0x7f9a40151ab0 2023-03-03T16:27:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.149+0000 7f9a44bb6700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a240682f0 0x7f9a2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:27:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.150+0000 7f9a44bb6700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a240682f0 0x7f9a2406a7a0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009130 tx=0x7f9a34008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:27:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.153+0000 7f9a3d7fa700 1 -- 172.21.15.7:0/3587908108 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9a38052a50 con 0x7f9a40151ab0 2023-03-03T16:27:53.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.388+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3587908108 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a40074bb0 con 0x7f9a40151ab0 2023-03-03T16:27:53.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.389+0000 7f9a3d7fa700 1 -- 172.21.15.7:0/3587908108 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9a38056070 con 0x7f9a40151ab0 2023-03-03T16:27:53.389 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:27:53.390 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:27:53.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a240682f0 msgr2=0x7f9a2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:53.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a240682f0 0x7f9a2406a7a0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009130 tx=0x7f9a34008030 comp rx=0 tx=0).stop 2023-03-03T16:27:53.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3587908108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 msgr2=0x7f9a400c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:27:53.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 0x7f9a400c61b0 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f9a38007e30 tx=0x7f9a3800e3c0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3587908108 shutdown_connections 2023-03-03T16:27:53.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9a240682f0 0x7f9a2406a7a0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a40151ab0 0x7f9a400c61b0 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9a4006c5b0 0x7f9a40146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 --2- 172.21.15.7:0/3587908108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a40073c30 0x7f9a401ec8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:27:53.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3587908108 >> 172.21.15.7:0/3587908108 conn(0x7f9a400a94d0 msgr2=0x7f9a40156b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:27:53.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3587908108 shutdown_connections 2023-03-03T16:27:53.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:27:53.391+0000 7f9a46e1a700 1 -- 172.21.15.7:0/3587908108 wait complete. 2023-03-03T16:27:53.393 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:27:53.403 INFO:tasks.mds_thrash.fs.[b]:waiting for 10 secs before thrashing 2023-03-03T16:28:03.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:28:03.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.544+0000 7f6a1a9af700 1 Processor -- start 2023-03-03T16:28:03.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.544+0000 7f6a1a9af700 1 -- start start 2023-03-03T16:28:03.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.545+0000 7f6a1a9af700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6a14151c60 0x7f6a140c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:03.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.545+0000 7f6a1a9af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14158740 0x7f6a1406a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:03.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.545+0000 7f6a1a9af700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a14152030 con 0x7f6a1406a2b0 2023-03-03T16:28:03.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.545+0000 7f6a1a9af700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a14157b30 con 0x7f6a14151c60 2023-03-03T16:28:03.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.545+0000 7f6a1a9af700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a140c4a80 con 0x7f6a14158740 2023-03-03T16:28:03.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.545+0000 7f6a13fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6a14151c60 0x7f6a140c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.545+0000 7f6a137fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6a1406a2b0 0x7f6a14158030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51218/0 (socket says 172.21.15.7:51218) 2023-03-03T16:28:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.545+0000 7f6a137fe700 1 -- 172.21.15.7:0/1237305747 learned_addr learned my addr 172.21.15.7:0/1237305747 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:28:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.545+0000 7f6a18f4c700 1 --2- 172.21.15.7:0/1237305747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14158740 0x7f6a1406a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.546+0000 7f6a13fff700 1 -- 172.21.15.7:0/1237305747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14158740 msgr2=0x7f6a1406a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:28:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.546+0000 7f6a13fff700 1 --2- 172.21.15.7:0/1237305747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14158740 0x7f6a1406a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:03.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.546+0000 7f6a13fff700 1 -- 172.21.15.7:0/1237305747 >> v1:172.21.15.7:6789/0 conn(0x7f6a1406a2b0 legacy=0x7f6a14158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.546+0000 7f6a13fff700 1 -- 172.21.15.7:0/1237305747 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a140c4d60 con 0x7f6a14151c60 2023-03-03T16:28:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.546+0000 7f6a13fff700 1 --2- 172.21.15.7:0/1237305747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a14151c60 0x7f6a140c1500 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f6a04009840 tx=0x7f6a04009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526c400f3813ce20 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:28:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.546+0000 7f6a12ffd700 1 -- 172.21.15.7:0/1237305747 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a04011830 con 0x7f6a14151c60 2023-03-03T16:28:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.546+0000 7f6a12ffd700 1 -- 172.21.15.7:0/1237305747 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a04011990 con 0x7f6a14151c60 2023-03-03T16:28:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.546+0000 7f6a12ffd700 1 -- 172.21.15.7:0/1237305747 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a04011d20 con 0x7f6a14151c60 2023-03-03T16:28:03.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.547+0000 7f6a1a9af700 1 -- 172.21.15.7:0/1237305747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a14151c60 msgr2=0x7f6a140c1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:03.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.547+0000 7f6a1a9af700 1 --2- 172.21.15.7:0/1237305747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a14151c60 0x7f6a140c1500 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7f6a04009840 tx=0x7f6a04009b50 comp rx=0 tx=0).stop 2023-03-03T16:28:03.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.547+0000 7f6a1a9af700 1 -- 172.21.15.7:0/1237305747 shutdown_connections 2023-03-03T16:28:03.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.547+0000 7f6a1a9af700 1 --2- 172.21.15.7:0/1237305747 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14158740 0x7f6a1406a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:03.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.547+0000 7f6a1a9af700 1 --2- 172.21.15.7:0/1237305747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a14151c60 0x7f6a140c1500 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:03.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.547+0000 7f6a1a9af700 1 -- 172.21.15.7:0/1237305747 >> 172.21.15.7:0/1237305747 conn(0x7f6a140a9430 msgr2=0x7f6a140c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:28:03.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.547+0000 7f6a1a9af700 1 -- 172.21.15.7:0/1237305747 shutdown_connections 2023-03-03T16:28:03.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.547+0000 7f6a1a9af700 1 -- 172.21.15.7:0/1237305747 wait complete. 2023-03-03T16:28:03.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.548+0000 7f6a1a9af700 1 Processor -- start 2023-03-03T16:28:03.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.548+0000 7f6a1a9af700 1 -- start start 2023-03-03T16:28:03.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.549+0000 7f6a1a9af700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a1406a2b0 0x7f6a1406fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:03.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.549+0000 7f6a1a9af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14151c60 0x7f6a1406cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:03.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.549+0000 7f6a1a9af700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a14158740 0x7f6a1406d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:03.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.549+0000 7f6a1a9af700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1406b930 con 0x7f6a1406a2b0 2023-03-03T16:28:03.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.549+0000 7f6a1a9af700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1406bc10 con 0x7f6a14158740 2023-03-03T16:28:03.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.549+0000 7f6a1a9af700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a141edcb0 con 0x7f6a14151c60 2023-03-03T16:28:03.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.549+0000 7f6a13fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a1406a2b0 0x7f6a1406fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:03.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.549+0000 7f6a13fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a1406a2b0 0x7f6a1406fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57240/0 (socket says 172.21.15.7:57240) 2023-03-03T16:28:03.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.549+0000 7f6a13fff700 1 -- 172.21.15.7:0/2896709131 learned_addr learned my addr 172.21.15.7:0/2896709131 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:28:03.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a137fe700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14151c60 0x7f6a1406cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:03.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a18f4c700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a14158740 0x7f6a1406d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:03.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a13fff700 1 -- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14151c60 msgr2=0x7f6a1406cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:03.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a13fff700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14151c60 0x7f6a1406cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:03.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a13fff700 1 -- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a14158740 msgr2=0x7f6a1406d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:03.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a13fff700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a14158740 0x7f6a1406d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:03.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a13fff700 1 -- 172.21.15.7:0/2896709131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a040094a0 con 0x7f6a1406a2b0 2023-03-03T16:28:03.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a137fe700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14151c60 0x7f6a1406cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:28:03.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a13fff700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a1406a2b0 0x7f6a1406fc90 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f6a04009fd0 tx=0x7f6a04004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:28:03.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a10ff9700 1 -- 172.21.15.7:0/2896709131 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a040036a0 con 0x7f6a1406a2b0 2023-03-03T16:28:03.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a10ff9700 1 -- 172.21.15.7:0/2896709131 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0401f070 con 0x7f6a1406a2b0 2023-03-03T16:28:03.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a10ff9700 1 -- 172.21.15.7:0/2896709131 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a04003a60 con 0x7f6a1406a2b0 2023-03-03T16:28:03.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a1a9af700 1 -- 172.21.15.7:0/2896709131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a140c5320 con 0x7f6a1406a2b0 2023-03-03T16:28:03.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.550+0000 7f6a1a9af700 1 -- 172.21.15.7:0/2896709131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a140c5040 con 0x7f6a1406a2b0 2023-03-03T16:28:03.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.551+0000 7f6a1a9af700 1 -- 172.21.15.7:0/2896709131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a140abe90 con 0x7f6a1406a2b0 2023-03-03T16:28:03.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.552+0000 7f6a10ff9700 1 -- 172.21.15.7:0/2896709131 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6a0400f050 con 0x7f6a1406a2b0 2023-03-03T16:28:03.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.552+0000 7f6a10ff9700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a000682f0 0x7f6a0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:03.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.552+0000 7f6a10ff9700 1 -- 172.21.15.7:0/2896709131 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6a04021070 con 0x7f6a1406a2b0 2023-03-03T16:28:03.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.552+0000 7f6a137fe700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a000682f0 0x7f6a0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:03.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.553+0000 7f6a137fe700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a000682f0 0x7f6a0006a7a0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f6a08007fd0 tx=0x7f6a08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:28:03.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.556+0000 7f6a10ff9700 1 -- 172.21.15.7:0/2896709131 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6a04053de0 con 0x7f6a1406a2b0 2023-03-03T16:28:03.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.794+0000 7f6a1a9af700 1 -- 172.21.15.7:0/2896709131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a1406c980 con 0x7f6a1406a2b0 2023-03-03T16:28:03.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.795+0000 7f6a10ff9700 1 -- 172.21.15.7:0/2896709131 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6a04057400 con 0x7f6a1406a2b0 2023-03-03T16:28:03.795 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:28:03.795 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:28:03.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 -- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a000682f0 msgr2=0x7f6a0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:03.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a000682f0 0x7f6a0006a7a0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f6a08007fd0 tx=0x7f6a08009040 comp rx=0 tx=0).stop 2023-03-03T16:28:03.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 -- 172.21.15.7:0/2896709131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a1406a2b0 msgr2=0x7f6a1406fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:03.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a1406a2b0 0x7f6a1406fc90 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f6a04009fd0 tx=0x7f6a04004750 comp rx=0 tx=0).stop 2023-03-03T16:28:03.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 -- 172.21.15.7:0/2896709131 shutdown_connections 2023-03-03T16:28:03.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a14158740 0x7f6a1406d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:03.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6a000682f0 0x7f6a0006a7a0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:03.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6a1406a2b0 0x7f6a1406fc90 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:03.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 --2- 172.21.15.7:0/2896709131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6a14151c60 0x7f6a1406cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:03.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 -- 172.21.15.7:0/2896709131 >> 172.21.15.7:0/2896709131 conn(0x7f6a140a9430 msgr2=0x7f6a140aa100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:28:03.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 -- 172.21.15.7:0/2896709131 shutdown_connections 2023-03-03T16:28:03.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:03.797+0000 7f6a1a9af700 1 -- 172.21.15.7:0/2896709131 wait complete. 2023-03-03T16:28:03.799 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:28:03.809 INFO:tasks.mds_thrash.fs.[b]:waiting for 68 secs before thrashing 2023-03-03T16:28:09.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:28:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.329+0000 7f2e391ec700 1 Processor -- start 2023-03-03T16:28:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.330+0000 7f2e391ec700 1 -- start start 2023-03-03T16:28:09.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.330+0000 7f2e391ec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e340c2e30 0x7f2e34158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:09.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.330+0000 7f2e391ec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2e3406c6c0 0x7f2e340736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:09.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.330+0000 7f2e391ec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e34157ff0 con 0x7f2e34151cd0 2023-03-03T16:28:09.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.330+0000 7f2e391ec700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3406d640 con 0x7f2e340c2e30 2023-03-03T16:28:09.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.330+0000 7f2e391ec700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3406cda0 con 0x7f2e3406c6c0 2023-03-03T16:28:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.330+0000 7f2e3259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e340c2e30 0x7f2e34158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.330+0000 7f2e3259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e340c2e30 0x7f2e34158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57256/0 (socket says 172.21.15.7:57256) 2023-03-03T16:28:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.330+0000 7f2e3259c700 1 -- 172.21.15.7:0/3792345991 learned_addr learned my addr 172.21.15.7:0/3792345991 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:28:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.331+0000 7f2e3259c700 1 -- 172.21.15.7:0/3792345991 >> v2:172.21.15.143:3300/0 conn(0x7f2e3406c6c0 msgr2=0x7f2e340736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:28:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.331+0000 7f2e3359e700 1 --2- 172.21.15.7:0/3792345991 >> v2:172.21.15.143:3300/0 conn(0x7f2e3406c6c0 0x7f2e340736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.331+0000 7f2e3259c700 1 --2- 172.21.15.7:0/3792345991 >> v2:172.21.15.143:3300/0 conn(0x7f2e3406c6c0 0x7f2e340736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.331+0000 7f2e3259c700 1 -- 172.21.15.7:0/3792345991 >> v1:172.21.15.143:6789/0 conn(0x7f2e34151cd0 legacy=0x7f2e34154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:28:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.331+0000 7f2e3259c700 1 -- 172.21.15.7:0/3792345991 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3406d080 con 0x7f2e340c2e30 2023-03-03T16:28:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.331+0000 7f2e3359e700 1 --2- 172.21.15.7:0/3792345991 >> v2:172.21.15.143:3300/0 conn(0x7f2e3406c6c0 0x7f2e340736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:28:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.331+0000 7f2e3259c700 1 --2- 172.21.15.7:0/3792345991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e340c2e30 0x7f2e34158480 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f2e340c38a0 tx=0x7f2e1c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf24279d60e11cc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:28:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.331+0000 7f2e31d9b700 1 -- 172.21.15.7:0/3792345991 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c004230 con 0x7f2e340c2e30 2023-03-03T16:28:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.331+0000 7f2e31d9b700 1 -- 172.21.15.7:0/3792345991 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c004390 con 0x7f2e340c2e30 2023-03-03T16:28:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.331+0000 7f2e31d9b700 1 -- 172.21.15.7:0/3792345991 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0036a0 con 0x7f2e340c2e30 2023-03-03T16:28:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.332+0000 7f2e391ec700 1 -- 172.21.15.7:0/3792345991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e340c2e30 msgr2=0x7f2e34158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:09.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.332+0000 7f2e391ec700 1 --2- 172.21.15.7:0/3792345991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e340c2e30 0x7f2e34158480 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f2e340c38a0 tx=0x7f2e1c009bd0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.332+0000 7f2e391ec700 1 -- 172.21.15.7:0/3792345991 shutdown_connections 2023-03-03T16:28:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.332+0000 7f2e391ec700 1 --2- 172.21.15.7:0/3792345991 >> v2:172.21.15.143:3300/0 conn(0x7f2e3406c6c0 0x7f2e340736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.332+0000 7f2e391ec700 1 --2- 172.21.15.7:0/3792345991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e340c2e30 0x7f2e34158480 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.332+0000 7f2e391ec700 1 -- 172.21.15.7:0/3792345991 >> 172.21.15.7:0/3792345991 conn(0x7f2e340a94d0 msgr2=0x7f2e340a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:28:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.332+0000 7f2e391ec700 1 -- 172.21.15.7:0/3792345991 shutdown_connections 2023-03-03T16:28:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.332+0000 7f2e391ec700 1 -- 172.21.15.7:0/3792345991 wait complete. 2023-03-03T16:28:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.333+0000 7f2e391ec700 1 Processor -- start 2023-03-03T16:28:09.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.333+0000 7f2e391ec700 1 -- start start 2023-03-03T16:28:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e391ec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3406c6c0 0x7f2e340aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e391ec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e340c2e30 0x7f2e341ee650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e391ec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e34151cd0 0x7f2e34156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e391ec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340ab210 con 0x7f2e3406c6c0 2023-03-03T16:28:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e391ec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e341eeb90 con 0x7f2e34151cd0 2023-03-03T16:28:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e391ec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e341541c0 con 0x7f2e340c2e30 2023-03-03T16:28:09.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e32d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3406c6c0 0x7f2e340aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e32d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3406c6c0 0x7f2e340aacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57260/0 (socket says 172.21.15.7:57260) 2023-03-03T16:28:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e32d9d700 1 -- 172.21.15.7:0/2298919357 learned_addr learned my addr 172.21.15.7:0/2298919357 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:28:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e3359e700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e34151cd0 0x7f2e34156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e3259c700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e340c2e30 0x7f2e341ee650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e32d9d700 1 -- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e340c2e30 msgr2=0x7f2e341ee650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e32d9d700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e340c2e30 0x7f2e341ee650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e32d9d700 1 -- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e34151cd0 msgr2=0x7f2e34156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:09.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e32d9d700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e34151cd0 0x7f2e34156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e32d9d700 1 -- 172.21.15.7:0/2298919357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c009550 con 0x7f2e3406c6c0 2023-03-03T16:28:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.334+0000 7f2e3259c700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e340c2e30 0x7f2e341ee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:28:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.335+0000 7f2e32d9d700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3406c6c0 0x7f2e340aacd0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400b4a0 tx=0x7f2e2400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:28:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.335+0000 7f2e2b7fe700 1 -- 172.21.15.7:0/2298919357 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24011830 con 0x7f2e3406c6c0 2023-03-03T16:28:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.335+0000 7f2e2b7fe700 1 -- 172.21.15.7:0/2298919357 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e24011990 con 0x7f2e3406c6c0 2023-03-03T16:28:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.335+0000 7f2e2b7fe700 1 -- 172.21.15.7:0/2298919357 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24011d40 con 0x7f2e3406c6c0 2023-03-03T16:28:09.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.335+0000 7f2e391ec700 1 -- 172.21.15.7:0/2298919357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e34157c80 con 0x7f2e3406c6c0 2023-03-03T16:28:09.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.335+0000 7f2e391ec700 1 -- 172.21.15.7:0/2298919357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3406d360 con 0x7f2e3406c6c0 2023-03-03T16:28:09.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.336+0000 7f2e391ec700 1 -- 172.21.15.7:0/2298919357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e341589c0 con 0x7f2e3406c6c0 2023-03-03T16:28:09.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.336+0000 7f2e2b7fe700 1 -- 172.21.15.7:0/2298919357 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2e2400f050 con 0x7f2e3406c6c0 2023-03-03T16:28:09.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.337+0000 7f2e2b7fe700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e140682f0 0x7f2e1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:09.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.337+0000 7f2e2b7fe700 1 -- 172.21.15.7:0/2298919357 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2e24085c80 con 0x7f2e3406c6c0 2023-03-03T16:28:09.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.337+0000 7f2e3259c700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e140682f0 0x7f2e1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:09.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.338+0000 7f2e3259c700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e140682f0 0x7f2e1406a7a0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009130 tx=0x7f2e1c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:28:09.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.341+0000 7f2e2b7fe700 1 -- 172.21.15.7:0/2298919357 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2e24052020 con 0x7f2e3406c6c0 2023-03-03T16:28:09.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.581+0000 7f2e391ec700 1 -- 172.21.15.7:0/2298919357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e34074bb0 con 0x7f2e3406c6c0 2023-03-03T16:28:09.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.581+0000 7f2e2b7fe700 1 -- 172.21.15.7:0/2298919357 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2e24055640 con 0x7f2e3406c6c0 2023-03-03T16:28:09.582 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:28:09.582 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:28:09.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 -- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e140682f0 msgr2=0x7f2e1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:09.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e140682f0 0x7f2e1406a7a0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009130 tx=0x7f2e1c008030 comp rx=0 tx=0).stop 2023-03-03T16:28:09.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 -- 172.21.15.7:0/2298919357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3406c6c0 msgr2=0x7f2e340aacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:09.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3406c6c0 0x7f2e340aacd0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400b4a0 tx=0x7f2e2400b7b0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 -- 172.21.15.7:0/2298919357 shutdown_connections 2023-03-03T16:28:09.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e34151cd0 0x7f2e34156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e140682f0 0x7f2e1406a7a0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3406c6c0 0x7f2e340aacd0 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 --2- 172.21.15.7:0/2298919357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e340c2e30 0x7f2e341ee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:09.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 -- 172.21.15.7:0/2298919357 >> 172.21.15.7:0/2298919357 conn(0x7f2e340a94d0 msgr2=0x7f2e34071ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:28:09.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 -- 172.21.15.7:0/2298919357 shutdown_connections 2023-03-03T16:28:09.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:09.583+0000 7f2e391ec700 1 -- 172.21.15.7:0/2298919357 wait complete. 2023-03-03T16:28:09.585 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:28:09.595 INFO:tasks.mds_thrash.fs.[a]:waiting for 36 secs before thrashing 2023-03-03T16:28:16.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:16.340 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:45.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:28:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.733+0000 7f757929b700 1 Processor -- start 2023-03-03T16:28:45.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.734+0000 7f757929b700 1 -- start start 2023-03-03T16:28:45.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.734+0000 7f757929b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75740c1850 0x7f75740c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:45.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.734+0000 7f757929b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7574154790 con 0x7f757414b060 2023-03-03T16:28:45.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.734+0000 7f757929b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7574154a70 con 0x7f75740aa170 2023-03-03T16:28:45.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.734+0000 7f757929b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75740c4100 con 0x7f75740c1850 2023-03-03T16:28:45.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.734+0000 7f7572ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f757414b060 0x7f757406f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51272/0 (socket says 172.21.15.7:51272) 2023-03-03T16:28:45.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7572ffd700 1 -- 172.21.15.7:0/685336762 learned_addr learned my addr 172.21.15.7:0/685336762 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:28:45.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f75727fc700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75740c1850 0x7f75740c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:45.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442911656 0 0) 0x7f7574154790 con 0x7f757414b060 2023-03-03T16:28:45.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c0035b0 con 0x7f757414b060 2023-03-03T16:28:45.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1959064466 0 0) 0x7f755c0035b0 con 0x7f757414b060 2023-03-03T16:28:45.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75740c1850 msgr2=0x7f75740c1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:28:45.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75740c1850 0x7f75740c1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:45.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 >> v1:172.21.15.143:6789/0 conn(0x7f75740aa170 legacy=0x7f75740aa5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:45.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7574154d50 con 0x7f757414b060 2023-03-03T16:28:45.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7568001670 con 0x7f757414b060 2023-03-03T16:28:45.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 >> v1:172.21.15.7:6789/0 conn(0x7f757414b060 legacy=0x7f757406f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f755c003aa0 0x7f755c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75740c1850 0x7f755c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 0x7f755c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75740c4100 con 0x7f755c004d90 2023-03-03T16:28:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c00bb10 con 0x7f755c003aa0 2023-03-03T16:28:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c00bcd0 con 0x7f75740c1850 2023-03-03T16:28:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f75737fe700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 0x7f755c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:45.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f7572ffd700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f755c003aa0 0x7f755c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:45.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.735+0000 7f75727fc700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75740c1850 0x7f755c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:45.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.736+0000 7f75737fe700 1 -- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75740c1850 msgr2=0x7f755c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:28:45.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.736+0000 7f75737fe700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75740c1850 0x7f755c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:45.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.736+0000 7f75737fe700 1 -- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f755c003aa0 msgr2=0x7f755c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:28:45.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.736+0000 7f75737fe700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f755c003aa0 0x7f755c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:45.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.736+0000 7f75737fe700 1 -- 172.21.15.7:0/685336762 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c0035b0 con 0x7f755c004d90 2023-03-03T16:28:45.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.736+0000 7f75737fe700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 0x7f755c005200 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f756c00c290 tx=0x7f755c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ee581d15fcfdb1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:28:45.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.736+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c001e70 con 0x7f755c004d90 2023-03-03T16:28:45.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.736+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756c00ce50 con 0x7f755c004d90 2023-03-03T16:28:45.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.736+0000 7f7571ffb700 1 -- 172.21.15.7:0/685336762 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c012660 con 0x7f755c004d90 2023-03-03T16:28:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.737+0000 7f757929b700 1 -- 172.21.15.7:0/685336762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 msgr2=0x7f755c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.737+0000 7f757929b700 1 --2- 172.21.15.7:0/685336762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 0x7f755c005200 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f756c00c290 tx=0x7f755c005880 comp rx=0 tx=0).stop 2023-03-03T16:28:45.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.737+0000 7f75737fe700 1 -- 172.21.15.7:0/685336762 reap_dead start 2023-03-03T16:28:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.737+0000 7f757929b700 1 -- 172.21.15.7:0/685336762 shutdown_connections 2023-03-03T16:28:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.737+0000 7f757929b700 1 -- 172.21.15.7:0/685336762 >> 172.21.15.7:0/685336762 conn(0x7f75740c4f20 msgr2=0x7f75740c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:28:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.737+0000 7f757929b700 1 -- 172.21.15.7:0/685336762 shutdown_connections 2023-03-03T16:28:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.737+0000 7f757929b700 1 -- 172.21.15.7:0/685336762 wait complete. 2023-03-03T16:28:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.738+0000 7f757929b700 1 Processor -- start 2023-03-03T16:28:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.738+0000 7f757929b700 1 -- start start 2023-03-03T16:28:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f757929b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 0x7f75741ecdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:45.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f757929b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75740acfa0 0x7f75741e6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f757929b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7574155730 0x7f75741e7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f757929b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7574144c70 con 0x7f755c004d90 2023-03-03T16:28:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f757929b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75740ad4b0 con 0x7f75740acfa0 2023-03-03T16:28:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f757929b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75740ab9d0 con 0x7f7574155730 2023-03-03T16:28:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f7572ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 0x7f75741ecdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f7572ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 0x7f75741ecdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57304/0 (socket says 172.21.15.7:57304) 2023-03-03T16:28:45.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f7572ffd700 1 -- 172.21.15.7:0/2233901866 learned_addr learned my addr 172.21.15.7:0/2233901866 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:28:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f75727fc700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75740acfa0 0x7f75741e6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.739+0000 7f75737fe700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7574155730 0x7f75741e7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f7572ffd700 1 -- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7574155730 msgr2=0x7f75741e7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f7572ffd700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7574155730 0x7f75741e7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f7572ffd700 1 -- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75740acfa0 msgr2=0x7f75741e6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f7572ffd700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75740acfa0 0x7f75741e6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f7572ffd700 1 -- 172.21.15.7:0/2233901866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7574155310 con 0x7f755c004d90 2023-03-03T16:28:45.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f7572ffd700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 0x7f75741ecdb0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f7568007290 tx=0x7f75680075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:28:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f755b7fe700 1 -- 172.21.15.7:0/2233901866 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7568007e80 con 0x7f755c004d90 2023-03-03T16:28:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f755b7fe700 1 -- 172.21.15.7:0/2233901866 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756800f6c0 con 0x7f755c004d90 2023-03-03T16:28:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f757929b700 1 -- 172.21.15.7:0/2233901866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7574155030 con 0x7f755c004d90 2023-03-03T16:28:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f755b7fe700 1 -- 172.21.15.7:0/2233901866 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756800fa50 con 0x7f755c004d90 2023-03-03T16:28:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.740+0000 7f757929b700 1 -- 172.21.15.7:0/2233901866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7574154440 con 0x7f755c004d90 2023-03-03T16:28:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.741+0000 7f757929b700 1 -- 172.21.15.7:0/2233901866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7574074bb0 con 0x7f755c004d90 2023-03-03T16:28:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.741+0000 7f755b7fe700 1 -- 172.21.15.7:0/2233901866 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f756800fbb0 con 0x7f755c004d90 2023-03-03T16:28:45.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.742+0000 7f755b7fe700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75600682f0 0x7f756006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:28:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.742+0000 7f755b7fe700 1 -- 172.21.15.7:0/2233901866 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7568086120 con 0x7f755c004d90 2023-03-03T16:28:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.742+0000 7f75727fc700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75600682f0 0x7f756006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:28:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.743+0000 7f75727fc700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75600682f0 0x7f756006a7a0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f7574155d30 tx=0x7f7564009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:28:45.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.746+0000 7f755b7fe700 1 -- 172.21.15.7:0/2233901866 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7568052570 con 0x7f755c004d90 2023-03-03T16:28:45.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.986+0000 7f757929b700 1 -- 172.21.15.7:0/2233901866 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75740ade90 con 0x7f755c004d90 2023-03-03T16:28:45.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.987+0000 7f755b7fe700 1 -- 172.21.15.7:0/2233901866 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7568055b90 con 0x7f755c004d90 2023-03-03T16:28:45.988 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:28:45.989 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:28:45.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 -- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75600682f0 msgr2=0x7f756006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:45.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75600682f0 0x7f756006a7a0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f7574155d30 tx=0x7f7564009210 comp rx=0 tx=0).stop 2023-03-03T16:28:45.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 -- 172.21.15.7:0/2233901866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 msgr2=0x7f75741ecdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:28:45.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 0x7f75741ecdb0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f7568007290 tx=0x7f75680075a0 comp rx=0 tx=0).stop 2023-03-03T16:28:45.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 -- 172.21.15.7:0/2233901866 shutdown_connections 2023-03-03T16:28:45.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7574155730 0x7f75741e7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:45.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75600682f0 0x7f756006a7a0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:45.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f755c004d90 0x7f75741ecdb0 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:45.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 --2- 172.21.15.7:0/2233901866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75740acfa0 0x7f75741e6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:28:45.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 -- 172.21.15.7:0/2233901866 >> 172.21.15.7:0/2233901866 conn(0x7f75740c4f20 msgr2=0x7f75740aa540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:28:45.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 -- 172.21.15.7:0/2233901866 shutdown_connections 2023-03-03T16:28:45.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:28:45.989+0000 7f757929b700 1 -- 172.21.15.7:0/2233901866 wait complete. 2023-03-03T16:28:45.992 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:28:46.002 INFO:tasks.mds_thrash.fs.[a]:waiting for 113 secs before thrashing 2023-03-03T16:28:46.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:28:46.402 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:11.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:29:11.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.949+0000 7f421ab50700 1 Processor -- start 2023-03-03T16:29:11.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.949+0000 7f421ab50700 1 -- start start 2023-03-03T16:29:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.950+0000 7f421ab50700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f421406a2b0 0x7f421406a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.950+0000 7f421ab50700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42141558f0 con 0x7f42140a6370 2023-03-03T16:29:11.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.950+0000 7f421ab50700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42140a6180 con 0x7f42140c5cb0 2023-03-03T16:29:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.950+0000 7f421ab50700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f421406bab0 con 0x7f421406a2b0 2023-03-03T16:29:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.950+0000 7f42188ec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f421406a2b0 0x7f421406a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.950+0000 7f4213fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f42140a6370 0x7f4214155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51316/0 (socket says 172.21.15.7:51316) 2023-03-03T16:29:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.950+0000 7f4213fff700 1 -- 172.21.15.7:0/227391046 learned_addr learned my addr 172.21.15.7:0/227391046 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:29:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.951+0000 7f42188ec700 1 -- 172.21.15.7:0/227391046 >> v1:172.21.15.143:6789/0 conn(0x7f42140c5cb0 legacy=0x7f421415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:29:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.951+0000 7f42188ec700 1 -- 172.21.15.7:0/227391046 >> v1:172.21.15.7:6789/0 conn(0x7f42140a6370 legacy=0x7f4214155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.951+0000 7f42188ec700 1 -- 172.21.15.7:0/227391046 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42140a5ea0 con 0x7f421406a2b0 2023-03-03T16:29:11.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.951+0000 7f42137fe700 1 -- 172.21.15.7:0/227391046 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191395372 0 0) 0x7f42141558f0 con 0x7f42140a6370 2023-03-03T16:29:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.951+0000 7f42188ec700 1 --2- 172.21.15.7:0/227391046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 0x7f421406a680 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f4208009840 tx=0x7f4208009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4244e8046aa667ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:29:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.951+0000 7f42137fe700 1 -- 172.21.15.7:0/227391046 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208004230 con 0x7f421406a2b0 2023-03-03T16:29:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.951+0000 7f42137fe700 1 -- 172.21.15.7:0/227391046 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4208004390 con 0x7f421406a2b0 2023-03-03T16:29:11.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.951+0000 7f42137fe700 1 -- 172.21.15.7:0/227391046 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42080036a0 con 0x7f421406a2b0 2023-03-03T16:29:11.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.952+0000 7f421ab50700 1 -- 172.21.15.7:0/227391046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 msgr2=0x7f421406a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:11.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.952+0000 7f421ab50700 1 --2- 172.21.15.7:0/227391046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 0x7f421406a680 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f4208009840 tx=0x7f4208009b50 comp rx=0 tx=0).stop 2023-03-03T16:29:11.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.952+0000 7f421ab50700 1 -- 172.21.15.7:0/227391046 shutdown_connections 2023-03-03T16:29:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.952+0000 7f421ab50700 1 --2- 172.21.15.7:0/227391046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 0x7f421406a680 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.952+0000 7f421ab50700 1 -- 172.21.15.7:0/227391046 >> 172.21.15.7:0/227391046 conn(0x7f42140a9430 msgr2=0x7f42140c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:29:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.952+0000 7f421ab50700 1 -- 172.21.15.7:0/227391046 shutdown_connections 2023-03-03T16:29:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.952+0000 7f421ab50700 1 -- 172.21.15.7:0/227391046 wait complete. 2023-03-03T16:29:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.953+0000 7f421ab50700 1 Processor -- start 2023-03-03T16:29:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.953+0000 7f421ab50700 1 -- start start 2023-03-03T16:29:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f421ab50700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 0x7f42140aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:11.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f421ab50700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42140a6370 0x7f4214072130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f421ab50700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42140c5cb0 0x7f4214151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f421ab50700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42141edee0 con 0x7f421406a2b0 2023-03-03T16:29:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f421ab50700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42140ab310 con 0x7f42140a6370 2023-03-03T16:29:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f421ab50700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42141eda50 con 0x7f42140c5cb0 2023-03-03T16:29:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42188ec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 0x7f42140aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42188ec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 0x7f42140aadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57340/0 (socket says 172.21.15.7:57340) 2023-03-03T16:29:11.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42188ec700 1 -- 172.21.15.7:0/3815321939 learned_addr learned my addr 172.21.15.7:0/3815321939 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:29:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f4213fff700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42140a6370 0x7f4214072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42190ed700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42140c5cb0 0x7f4214151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42188ec700 1 -- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42140c5cb0 msgr2=0x7f4214151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42188ec700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42140c5cb0 0x7f4214151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42188ec700 1 -- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42140a6370 msgr2=0x7f4214072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42188ec700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42140a6370 0x7f4214072130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42188ec700 1 -- 172.21.15.7:0/3815321939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42080094a0 con 0x7f421406a2b0 2023-03-03T16:29:11.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f4213fff700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42140a6370 0x7f4214072130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:29:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42190ed700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42140c5cb0 0x7f4214151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:29:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.954+0000 7f42188ec700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 0x7f42140aadd0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f421406adb0 tx=0x7f4208004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:29:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.955+0000 7f42117fa700 1 -- 172.21.15.7:0/3815321939 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208011630 con 0x7f421406a2b0 2023-03-03T16:29:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.955+0000 7f42117fa700 1 -- 172.21.15.7:0/3815321939 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420801e070 con 0x7f421406a2b0 2023-03-03T16:29:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.955+0000 7f421ab50700 1 -- 172.21.15.7:0/3815321939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4214153160 con 0x7f421406a2b0 2023-03-03T16:29:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.955+0000 7f42117fa700 1 -- 172.21.15.7:0/3815321939 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42080119b0 con 0x7f421406a2b0 2023-03-03T16:29:11.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.955+0000 7f421ab50700 1 -- 172.21.15.7:0/3815321939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4214152e80 con 0x7f421406a2b0 2023-03-03T16:29:11.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.956+0000 7f421ab50700 1 -- 172.21.15.7:0/3815321939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42140c1500 con 0x7f421406a2b0 2023-03-03T16:29:11.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.956+0000 7f42117fa700 1 -- 172.21.15.7:0/3815321939 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4208017020 con 0x7f421406a2b0 2023-03-03T16:29:11.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.956+0000 7f42117fa700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41f80682f0 0x7f41f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:29:11.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.957+0000 7f42117fa700 1 -- 172.21.15.7:0/3815321939 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f42080874a0 con 0x7f421406a2b0 2023-03-03T16:29:11.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.957+0000 7f4213fff700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41f80682f0 0x7f41f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:29:11.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.957+0000 7f4213fff700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41f80682f0 0x7f41f806a7a0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f421406a8d0 tx=0x7f4200009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:29:11.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:11.961+0000 7f42117fa700 1 -- 172.21.15.7:0/3815321939 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4208053840 con 0x7f421406a2b0 2023-03-03T16:29:12.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.197+0000 7f421ab50700 1 -- 172.21.15.7:0/3815321939 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4214074bb0 con 0x7f421406a2b0 2023-03-03T16:29:12.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.198+0000 7f42117fa700 1 -- 172.21.15.7:0/3815321939 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4208056e60 con 0x7f421406a2b0 2023-03-03T16:29:12.199 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:29:12.199 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:29:12.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 -- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41f80682f0 msgr2=0x7f41f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:12.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41f80682f0 0x7f41f806a7a0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f421406a8d0 tx=0x7f4200009040 comp rx=0 tx=0).stop 2023-03-03T16:29:12.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 -- 172.21.15.7:0/3815321939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 msgr2=0x7f42140aadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:29:12.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 0x7f42140aadd0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f421406adb0 tx=0x7f4208004750 comp rx=0 tx=0).stop 2023-03-03T16:29:12.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 -- 172.21.15.7:0/3815321939 shutdown_connections 2023-03-03T16:29:12.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f42140c5cb0 0x7f4214151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:12.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41f80682f0 0x7f41f806a7a0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:12.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f421406a2b0 0x7f42140aadd0 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:12.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 --2- 172.21.15.7:0/3815321939 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42140a6370 0x7f4214072130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:29:12.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 -- 172.21.15.7:0/3815321939 >> 172.21.15.7:0/3815321939 conn(0x7f42140a9430 msgr2=0x7f42140c2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:29:12.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 -- 172.21.15.7:0/3815321939 shutdown_connections 2023-03-03T16:29:12.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:29:12.200+0000 7f421ab50700 1 -- 172.21.15.7:0/3815321939 wait complete. 2023-03-03T16:29:12.202 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:29:12.212 INFO:tasks.mds_thrash.fs.[b]:waiting for 101 secs before thrashing 2023-03-03T16:29:16.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:16.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:46.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:29:46.545 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T16:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T16:30:16.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:16.606 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:39.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:30:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.141+0000 7f9e97b6c700 1 Processor -- start 2023-03-03T16:30:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.142+0000 7f9e97b6c700 1 -- start start 2023-03-03T16:30:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.142+0000 7f9e97b6c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9e900c43d0 0x7f9e900c4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.142+0000 7f9e97b6c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e90072920 con 0x7f9e90154590 2023-03-03T16:30:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.142+0000 7f9e97b6c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9006bef0 con 0x7f9e9014e560 2023-03-03T16:30:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.142+0000 7f9e97b6c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e90072380 con 0x7f9e900c43d0 2023-03-03T16:30:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e95107700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9e900c43d0 0x7f9e900c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e96109700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9e90154590 0x7f9e901549b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51386/0 (socket says 172.21.15.7:51386) 2023-03-03T16:30:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e96109700 1 -- 172.21.15.7:0/3895069520 learned_addr learned my addr 172.21.15.7:0/3895069520 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:30:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e95107700 1 -- 172.21.15.7:0/3895069520 >> v1:172.21.15.143:6789/0 conn(0x7f9e9014e560 legacy=0x7f9e90071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:30:39.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e95107700 1 -- 172.21.15.7:0/3895069520 >> v1:172.21.15.7:6789/0 conn(0x7f9e90154590 legacy=0x7f9e901549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:39.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e94906700 1 -- 172.21.15.7:0/3895069520 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314449925 0 0) 0x7f9e88001490 con 0x7f9e90154590 2023-03-03T16:30:39.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e95107700 1 -- 172.21.15.7:0/3895069520 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9006b910 con 0x7f9e900c43d0 2023-03-03T16:30:39.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e95107700 1 --2- 172.21.15.7:0/3895069520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 0x7f9e900c4850 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009840 tx=0x7f9e84009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e16e79a4b2458dc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:39.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e94906700 1 -- 172.21.15.7:0/3895069520 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84004230 con 0x7f9e900c43d0 2023-03-03T16:30:39.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e94906700 1 -- 172.21.15.7:0/3895069520 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e84004390 con 0x7f9e900c43d0 2023-03-03T16:30:39.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.143+0000 7f9e94906700 1 -- 172.21.15.7:0/3895069520 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840036a0 con 0x7f9e900c43d0 2023-03-03T16:30:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.144+0000 7f9e97b6c700 1 -- 172.21.15.7:0/3895069520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 msgr2=0x7f9e900c4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.144+0000 7f9e97b6c700 1 --2- 172.21.15.7:0/3895069520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 0x7f9e900c4850 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009840 tx=0x7f9e84009b50 comp rx=0 tx=0).stop 2023-03-03T16:30:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.144+0000 7f9e97b6c700 1 -- 172.21.15.7:0/3895069520 shutdown_connections 2023-03-03T16:30:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.144+0000 7f9e97b6c700 1 --2- 172.21.15.7:0/3895069520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 0x7f9e900c4850 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.144+0000 7f9e97b6c700 1 -- 172.21.15.7:0/3895069520 >> 172.21.15.7:0/3895069520 conn(0x7f9e900736f0 msgr2=0x7f9e90073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:30:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.144+0000 7f9e97b6c700 1 -- 172.21.15.7:0/3895069520 shutdown_connections 2023-03-03T16:30:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.144+0000 7f9e97b6c700 1 -- 172.21.15.7:0/3895069520 wait complete. 2023-03-03T16:30:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.145+0000 7f9e97b6c700 1 Processor -- start 2023-03-03T16:30:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.145+0000 7f9e97b6c700 1 -- start start 2023-03-03T16:30:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e97b6c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 0x7f9e900adf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e97b6c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e9014e560 0x7f9e900ac5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e97b6c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e90154590 0x7f9e900be7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e97b6c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e900ab460 con 0x7f9e900c43d0 2023-03-03T16:30:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e97b6c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e900ab990 con 0x7f9e9014e560 2023-03-03T16:30:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e97b6c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e900ab6b0 con 0x7f9e90154590 2023-03-03T16:30:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e95908700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 0x7f9e900adf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e95908700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 0x7f9e900adf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57408/0 (socket says 172.21.15.7:57408) 2023-03-03T16:30:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e95908700 1 -- 172.21.15.7:0/2786518067 learned_addr learned my addr 172.21.15.7:0/2786518067 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:30:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e96109700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e90154590 0x7f9e900be7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e95107700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e9014e560 0x7f9e900ac5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e95908700 1 -- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e90154590 msgr2=0x7f9e900be7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e95908700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e90154590 0x7f9e900be7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e95908700 1 -- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e9014e560 msgr2=0x7f9e900ac5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e95908700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e9014e560 0x7f9e900ac5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.146+0000 7f9e95908700 1 -- 172.21.15.7:0/2786518067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e840094a0 con 0x7f9e900c43d0 2023-03-03T16:30:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.147+0000 7f9e95908700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 0x7f9e900adf60 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00c500 tx=0x7f9e8c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.147+0000 7f9e827fc700 1 -- 172.21.15.7:0/2786518067 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c011790 con 0x7f9e900c43d0 2023-03-03T16:30:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.147+0000 7f9e827fc700 1 -- 172.21.15.7:0/2786518067 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c0118f0 con 0x7f9e900c43d0 2023-03-03T16:30:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.147+0000 7f9e97b6c700 1 -- 172.21.15.7:0/2786518067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9006bb80 con 0x7f9e900c43d0 2023-03-03T16:30:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.147+0000 7f9e827fc700 1 -- 172.21.15.7:0/2786518067 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c011ca0 con 0x7f9e900c43d0 2023-03-03T16:30:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.147+0000 7f9e97b6c700 1 -- 172.21.15.7:0/2786518067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e900aacc0 con 0x7f9e900c43d0 2023-03-03T16:30:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.148+0000 7f9e97b6c700 1 -- 172.21.15.7:0/2786518067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e90074bb0 con 0x7f9e900c43d0 2023-03-03T16:30:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.148+0000 7f9e827fc700 1 -- 172.21.15.7:0/2786518067 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9e8c010050 con 0x7f9e900c43d0 2023-03-03T16:30:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.149+0000 7f9e827fc700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e740682f0 0x7f9e7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:39.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.149+0000 7f9e827fc700 1 -- 172.21.15.7:0/2786518067 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9e8c085d20 con 0x7f9e900c43d0 2023-03-03T16:30:39.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.149+0000 7f9e95107700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e740682f0 0x7f9e7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:39.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.149+0000 7f9e95107700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e740682f0 0x7f9e7406a7a0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f9e840090b0 tx=0x7f9e8401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:39.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.153+0000 7f9e827fc700 1 -- 172.21.15.7:0/2786518067 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9e8c0520c0 con 0x7f9e900c43d0 2023-03-03T16:30:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.391+0000 7f9e97b6c700 1 -- 172.21.15.7:0/2786518067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e900ac1e0 con 0x7f9e900c43d0 2023-03-03T16:30:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.391+0000 7f9e827fc700 1 -- 172.21.15.7:0/2786518067 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9e8c0556e0 con 0x7f9e900c43d0 2023-03-03T16:30:39.392 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:30:39.392 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:30:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 -- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e740682f0 msgr2=0x7f9e7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e740682f0 0x7f9e7406a7a0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f9e840090b0 tx=0x7f9e8401c040 comp rx=0 tx=0).stop 2023-03-03T16:30:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 -- 172.21.15.7:0/2786518067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 msgr2=0x7f9e900adf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 0x7f9e900adf60 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00c500 tx=0x7f9e8c00c810 comp rx=0 tx=0).stop 2023-03-03T16:30:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 -- 172.21.15.7:0/2786518067 shutdown_connections 2023-03-03T16:30:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e90154590 0x7f9e900be7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e740682f0 0x7f9e7406a7a0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e900c43d0 0x7f9e900adf60 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 --2- 172.21.15.7:0/2786518067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e9014e560 0x7f9e900ac5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 -- 172.21.15.7:0/2786518067 >> 172.21.15.7:0/2786518067 conn(0x7f9e900736f0 msgr2=0x7f9e9006ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:30:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 -- 172.21.15.7:0/2786518067 shutdown_connections 2023-03-03T16:30:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:39.393+0000 7f9e97b6c700 1 -- 172.21.15.7:0/2786518067 wait complete. 2023-03-03T16:30:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:30:39.405 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2023-03-03T16:30:46.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:46.686 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:30:53.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:30:53.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.353+0000 7f5afa512700 1 Processor -- start 2023-03-03T16:30:53.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.353+0000 7f5afa512700 1 -- start start 2023-03-03T16:30:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.353+0000 7f5afa512700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5af4072250 0x7f5af414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:53.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.353+0000 7f5afa512700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5af40c1ce0 0x7f5af414e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.353+0000 7f5afa512700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af41510d0 con 0x7f5af4071e80 2023-03-03T16:30:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.353+0000 7f5afa512700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af40c3740 con 0x7f5af4072250 2023-03-03T16:30:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.353+0000 7f5afa512700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af4070670 con 0x7f5af40c1ce0 2023-03-03T16:30:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.353+0000 7f5af37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5af4072250 0x7f5af414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5af4072250 0x7f5af414e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57426/0 (socket says 172.21.15.7:57426) 2023-03-03T16:30:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af37fe700 1 -- 172.21.15.7:0/3937753436 learned_addr learned my addr 172.21.15.7:0/3937753436 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:30:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af8aaf700 1 --2- 172.21.15.7:0/3937753436 >> v2:172.21.15.143:3300/0 conn(0x7f5af40c1ce0 0x7f5af414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af37fe700 1 -- 172.21.15.7:0/3937753436 >> v2:172.21.15.143:3300/0 conn(0x7f5af40c1ce0 msgr2=0x7f5af414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:30:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af37fe700 1 --2- 172.21.15.7:0/3937753436 >> v2:172.21.15.143:3300/0 conn(0x7f5af40c1ce0 0x7f5af414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af37fe700 1 -- 172.21.15.7:0/3937753436 >> v1:172.21.15.143:6789/0 conn(0x7f5af4071e80 legacy=0x7f5af4151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af37fe700 1 -- 172.21.15.7:0/3937753436 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af40c39e0 con 0x7f5af4072250 2023-03-03T16:30:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af8aaf700 1 --2- 172.21.15.7:0/3937753436 >> v2:172.21.15.143:3300/0 conn(0x7f5af40c1ce0 0x7f5af414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:30:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af37fe700 1 --2- 172.21.15.7:0/3937753436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4072250 0x7f5af414e1f0 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f5af4071090 tx=0x7f5ae8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f8e010489b9db3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af2ffd700 1 -- 172.21.15.7:0/3937753436 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae800f8a0 con 0x7f5af4072250 2023-03-03T16:30:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af2ffd700 1 -- 172.21.15.7:0/3937753436 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae800fa00 con 0x7f5af4072250 2023-03-03T16:30:53.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.354+0000 7f5af2ffd700 1 -- 172.21.15.7:0/3937753436 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae800fd90 con 0x7f5af4072250 2023-03-03T16:30:53.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.355+0000 7f5afa512700 1 -- 172.21.15.7:0/3937753436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4072250 msgr2=0x7f5af414e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:53.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.355+0000 7f5afa512700 1 --2- 172.21.15.7:0/3937753436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4072250 0x7f5af414e1f0 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f5af4071090 tx=0x7f5ae8009bd0 comp rx=0 tx=0).stop 2023-03-03T16:30:53.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.355+0000 7f5afa512700 1 -- 172.21.15.7:0/3937753436 shutdown_connections 2023-03-03T16:30:53.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.355+0000 7f5afa512700 1 --2- 172.21.15.7:0/3937753436 >> v2:172.21.15.143:3300/0 conn(0x7f5af40c1ce0 0x7f5af414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:53.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.355+0000 7f5afa512700 1 --2- 172.21.15.7:0/3937753436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4072250 0x7f5af414e1f0 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:53.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.355+0000 7f5afa512700 1 -- 172.21.15.7:0/3937753436 >> 172.21.15.7:0/3937753436 conn(0x7f5af40a94d0 msgr2=0x7f5af40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:30:53.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.355+0000 7f5afa512700 1 -- 172.21.15.7:0/3937753436 shutdown_connections 2023-03-03T16:30:53.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.355+0000 7f5afa512700 1 -- 172.21.15.7:0/3937753436 wait complete. 2023-03-03T16:30:53.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.356+0000 7f5afa512700 1 Processor -- start 2023-03-03T16:30:53.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.356+0000 7f5afa512700 1 -- start start 2023-03-03T16:30:53.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5afa512700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4071e80 0x7f5af4144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:53.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5afa512700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4072250 0x7f5af40ac420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:53.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5afa512700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40c1ce0 0x7f5af41e5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:53.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5afa512700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af41e9ce0 con 0x7f5af4071e80 2023-03-03T16:30:53.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5afa512700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af40c5dd0 con 0x7f5af4072250 2023-03-03T16:30:53.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5afa512700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af414d790 con 0x7f5af40c1ce0 2023-03-03T16:30:53.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4071e80 0x7f5af4144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:53.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4071e80 0x7f5af4144890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57430/0 (socket says 172.21.15.7:57430) 2023-03-03T16:30:53.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af3fff700 1 -- 172.21.15.7:0/857488768 learned_addr learned my addr 172.21.15.7:0/857488768 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:30:53.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af37fe700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4072250 0x7f5af40ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:53.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af8aaf700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40c1ce0 0x7f5af41e5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:53.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af3fff700 1 -- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40c1ce0 msgr2=0x7f5af41e5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:53.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af3fff700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40c1ce0 0x7f5af41e5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:53.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af3fff700 1 -- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4072250 msgr2=0x7f5af40ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:53.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af3fff700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4072250 0x7f5af40ac420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:53.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af3fff700 1 -- 172.21.15.7:0/857488768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae8009550 con 0x7f5af4071e80 2023-03-03T16:30:53.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.357+0000 7f5af3fff700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4071e80 0x7f5af4144890 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400c510 tx=0x7f5ae400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:53.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.358+0000 7f5af0ff9700 1 -- 172.21.15.7:0/857488768 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40052d0 con 0x7f5af4071e80 2023-03-03T16:30:53.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.358+0000 7f5af0ff9700 1 -- 172.21.15.7:0/857488768 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae4005430 con 0x7f5af4071e80 2023-03-03T16:30:53.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.358+0000 7f5af0ff9700 1 -- 172.21.15.7:0/857488768 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40038c0 con 0x7f5af4071e80 2023-03-03T16:30:53.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.358+0000 7f5afa512700 1 -- 172.21.15.7:0/857488768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af4151f30 con 0x7f5af4071e80 2023-03-03T16:30:53.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.358+0000 7f5afa512700 1 -- 172.21.15.7:0/857488768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af4151300 con 0x7f5af4071e80 2023-03-03T16:30:53.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.359+0000 7f5afa512700 1 -- 172.21.15.7:0/857488768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af406c9a0 con 0x7f5af4071e80 2023-03-03T16:30:53.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.359+0000 7f5af0ff9700 1 -- 172.21.15.7:0/857488768 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5ae4019030 con 0x7f5af4071e80 2023-03-03T16:30:53.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.359+0000 7f5af0ff9700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ae00682f0 0x7f5ae006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:30:53.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.360+0000 7f5af0ff9700 1 -- 172.21.15.7:0/857488768 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5ae4085460 con 0x7f5af4071e80 2023-03-03T16:30:53.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.360+0000 7f5af37fe700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ae00682f0 0x7f5ae006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:30:53.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.360+0000 7f5af37fe700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ae00682f0 0x7f5ae006a7a0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f5af40c1780 tx=0x7f5ae8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:30:53.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.364+0000 7f5af0ff9700 1 -- 172.21.15.7:0/857488768 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5ae4051800 con 0x7f5af4071e80 2023-03-03T16:30:53.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.606+0000 7f5afa512700 1 -- 172.21.15.7:0/857488768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5af4074bb0 con 0x7f5af4071e80 2023-03-03T16:30:53.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.607+0000 7f5af0ff9700 1 -- 172.21.15.7:0/857488768 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5ae4054e20 con 0x7f5af4071e80 2023-03-03T16:30:53.607 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:30:53.608 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:30:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 -- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ae00682f0 msgr2=0x7f5ae006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ae00682f0 0x7f5ae006a7a0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f5af40c1780 tx=0x7f5ae8008030 comp rx=0 tx=0).stop 2023-03-03T16:30:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 -- 172.21.15.7:0/857488768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4071e80 msgr2=0x7f5af4144890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:30:53.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4071e80 0x7f5af4144890 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400c510 tx=0x7f5ae400c820 comp rx=0 tx=0).stop 2023-03-03T16:30:53.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 -- 172.21.15.7:0/857488768 shutdown_connections 2023-03-03T16:30:53.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5af40c1ce0 0x7f5af41e5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:53.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ae00682f0 0x7f5ae006a7a0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:53.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5af4071e80 0x7f5af4144890 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:53.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 --2- 172.21.15.7:0/857488768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5af4072250 0x7f5af40ac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:30:53.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 -- 172.21.15.7:0/857488768 >> 172.21.15.7:0/857488768 conn(0x7f5af40a94d0 msgr2=0x7f5af40c4550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:30:53.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 -- 172.21.15.7:0/857488768 shutdown_connections 2023-03-03T16:30:53.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:30:53.609+0000 7f5afa512700 1 -- 172.21.15.7:0/857488768 wait complete. 2023-03-03T16:30:53.611 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:30:53.621 INFO:tasks.mds_thrash.fs.[b]:waiting for 14 secs before thrashing 2023-03-03T16:31:07.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:31:07.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.760+0000 7f53317a2700 1 Processor -- start 2023-03-03T16:31:07.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.760+0000 7f53317a2700 1 -- start start 2023-03-03T16:31:07.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.761+0000 7f53317a2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f532c1519f0 0x7f532c0c3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:07.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.761+0000 7f53317a2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f532c154c90 0x7f532c15ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:07.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.761+0000 7f53317a2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532c1585c0 con 0x7f532c0c3860 2023-03-03T16:31:07.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.761+0000 7f53317a2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532c06d660 con 0x7f532c154c90 2023-03-03T16:31:07.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.761+0000 7f53317a2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532c0a8f70 con 0x7f532c1519f0 2023-03-03T16:31:07.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.761+0000 7f5330fa1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f532c154c90 0x7f532c15ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:07.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.761+0000 7f5330fa1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f532c154c90 0x7f532c15ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57456/0 (socket says 172.21.15.7:57456) 2023-03-03T16:31:07.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.761+0000 7f5330fa1700 1 -- 172.21.15.7:0/2481362881 learned_addr learned my addr 172.21.15.7:0/2481362881 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:31:07.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.761+0000 7f532bfff700 1 --2- 172.21.15.7:0/2481362881 >> v2:172.21.15.143:3300/0 conn(0x7f532c1519f0 0x7f532c0c3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:07.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.761+0000 7f5330fa1700 1 -- 172.21.15.7:0/2481362881 >> v2:172.21.15.143:3300/0 conn(0x7f532c1519f0 msgr2=0x7f532c0c3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:31:07.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f5330fa1700 1 --2- 172.21.15.7:0/2481362881 >> v2:172.21.15.143:3300/0 conn(0x7f532c1519f0 0x7f532c0c3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:07.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f5330fa1700 1 -- 172.21.15.7:0/2481362881 >> v1:172.21.15.143:6789/0 conn(0x7f532c0c3860 legacy=0x7f532c154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:31:07.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f5330fa1700 1 -- 172.21.15.7:0/2481362881 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c0c1430 con 0x7f532c154c90 2023-03-03T16:31:07.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f532bfff700 1 --2- 172.21.15.7:0/2481362881 >> v2:172.21.15.143:3300/0 conn(0x7f532c1519f0 0x7f532c0c3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:31:07.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f5330fa1700 1 --2- 172.21.15.7:0/2481362881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c154c90 0x7f532c15ebf0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f532c06f500 tx=0x7f532400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=363e97443580823e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:31:07.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f532affd700 1 -- 172.21.15.7:0/2481362881 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532400e070 con 0x7f532c154c90 2023-03-03T16:31:07.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f532affd700 1 -- 172.21.15.7:0/2481362881 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5324003a20 con 0x7f532c154c90 2023-03-03T16:31:07.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f532affd700 1 -- 172.21.15.7:0/2481362881 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5324003dd0 con 0x7f532c154c90 2023-03-03T16:31:07.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f53317a2700 1 -- 172.21.15.7:0/2481362881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c154c90 msgr2=0x7f532c15ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:07.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f53317a2700 1 --2- 172.21.15.7:0/2481362881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c154c90 0x7f532c15ebf0 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f532c06f500 tx=0x7f532400b330 comp rx=0 tx=0).stop 2023-03-03T16:31:07.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f53317a2700 1 -- 172.21.15.7:0/2481362881 shutdown_connections 2023-03-03T16:31:07.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f53317a2700 1 --2- 172.21.15.7:0/2481362881 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c154c90 0x7f532c15ebf0 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:07.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f53317a2700 1 --2- 172.21.15.7:0/2481362881 >> v2:172.21.15.143:3300/0 conn(0x7f532c1519f0 0x7f532c0c3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:07.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.762+0000 7f53317a2700 1 -- 172.21.15.7:0/2481362881 >> 172.21.15.7:0/2481362881 conn(0x7f532c06af40 msgr2=0x7f532c0a85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:31:07.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.763+0000 7f53317a2700 1 -- 172.21.15.7:0/2481362881 shutdown_connections 2023-03-03T16:31:07.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.763+0000 7f53317a2700 1 -- 172.21.15.7:0/2481362881 wait complete. 2023-03-03T16:31:07.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.765+0000 7f53317a2700 1 Processor -- start 2023-03-03T16:31:07.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.765+0000 7f53317a2700 1 -- start start 2023-03-03T16:31:07.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.766+0000 7f53317a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f532c0c3860 0x7f532c150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:07.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.766+0000 7f53317a2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c1519f0 0x7f532c152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:07.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.766+0000 7f53317a2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f532c154c90 0x7f532c156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:07.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.766+0000 7f53317a2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c0ac1e0 con 0x7f532c1519f0 2023-03-03T16:31:07.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.766+0000 7f53317a2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1e9320 con 0x7f532c154c90 2023-03-03T16:31:07.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.766+0000 7f53317a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c1ea140 con 0x7f532c0c3860 2023-03-03T16:31:07.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.766+0000 7f532b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c1519f0 0x7f532c152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:07.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.766+0000 7f532b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c1519f0 0x7f532c152d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57460/0 (socket says 172.21.15.7:57460) 2023-03-03T16:31:07.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.766+0000 7f532b7fe700 1 -- 172.21.15.7:0/1818319643 learned_addr learned my addr 172.21.15.7:0/1818319643 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:31:07.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f532bfff700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f532c0c3860 0x7f532c150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:07.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f5330fa1700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f532c154c90 0x7f532c156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:07.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f532b7fe700 1 -- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f532c0c3860 msgr2=0x7f532c150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:07.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f532b7fe700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f532c0c3860 0x7f532c150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:07.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f532b7fe700 1 -- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f532c154c90 msgr2=0x7f532c156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:07.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f532b7fe700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f532c154c90 0x7f532c156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:07.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f532b7fe700 1 -- 172.21.15.7:0/1818319643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5324009b30 con 0x7f532c1519f0 2023-03-03T16:31:07.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f532b7fe700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c1519f0 0x7f532c152d80 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f531c00c4d0 tx=0x7f531c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:31:07.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f5328ff9700 1 -- 172.21.15.7:0/1818319643 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c011790 con 0x7f532c1519f0 2023-03-03T16:31:07.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f5328ff9700 1 -- 172.21.15.7:0/1818319643 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531c0118f0 con 0x7f532c1519f0 2023-03-03T16:31:07.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f5328ff9700 1 -- 172.21.15.7:0/1818319643 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531c011ca0 con 0x7f532c1519f0 2023-03-03T16:31:07.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f53317a2700 1 -- 172.21.15.7:0/1818319643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532c158270 con 0x7f532c1519f0 2023-03-03T16:31:07.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.767+0000 7f53317a2700 1 -- 172.21.15.7:0/1818319643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532c0c1710 con 0x7f532c1519f0 2023-03-03T16:31:07.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.768+0000 7f53317a2700 1 -- 172.21.15.7:0/1818319643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532c0c1a80 con 0x7f532c1519f0 2023-03-03T16:31:07.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.769+0000 7f5328ff9700 1 -- 172.21.15.7:0/1818319643 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f531c010050 con 0x7f532c1519f0 2023-03-03T16:31:07.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.769+0000 7f5328ff9700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f530c0682f0 0x7f530c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:07.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.769+0000 7f5328ff9700 1 -- 172.21.15.7:0/1818319643 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f531c085da0 con 0x7f532c1519f0 2023-03-03T16:31:07.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.769+0000 7f532bfff700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f530c0682f0 0x7f530c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:07.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.770+0000 7f532bfff700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f530c0682f0 0x7f530c06a7a0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f5314006fd0 tx=0x7f5314008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:31:07.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:07.773+0000 7f5328ff9700 1 -- 172.21.15.7:0/1818319643 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f531c0520c0 con 0x7f532c1519f0 2023-03-03T16:31:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.013+0000 7f53317a2700 1 -- 172.21.15.7:0/1818319643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f532c061f40 con 0x7f532c1519f0 2023-03-03T16:31:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.014+0000 7f5328ff9700 1 -- 172.21.15.7:0/1818319643 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f531c0556e0 con 0x7f532c1519f0 2023-03-03T16:31:08.015 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:31:08.015 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:31:08.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 -- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f530c0682f0 msgr2=0x7f530c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:08.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f530c0682f0 0x7f530c06a7a0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f5314006fd0 tx=0x7f5314008040 comp rx=0 tx=0).stop 2023-03-03T16:31:08.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 -- 172.21.15.7:0/1818319643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c1519f0 msgr2=0x7f532c152d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:08.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c1519f0 0x7f532c152d80 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f531c00c4d0 tx=0x7f531c00c7e0 comp rx=0 tx=0).stop 2023-03-03T16:31:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 -- 172.21.15.7:0/1818319643 shutdown_connections 2023-03-03T16:31:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f532c154c90 0x7f532c156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f532c0c3860 0x7f532c150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f530c0682f0 0x7f530c06a7a0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 --2- 172.21.15.7:0/1818319643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f532c1519f0 0x7f532c152d80 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 -- 172.21.15.7:0/1818319643 >> 172.21.15.7:0/1818319643 conn(0x7f532c06af40 msgr2=0x7f532c0a8700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:31:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 -- 172.21.15.7:0/1818319643 shutdown_connections 2023-03-03T16:31:08.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:08.016+0000 7f53317a2700 1 -- 172.21.15.7:0/1818319643 wait complete. 2023-03-03T16:31:08.019 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:31:08.029 INFO:tasks.mds_thrash.fs.[b]:waiting for 58 secs before thrashing 2023-03-03T16:31:16.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:16.741 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:46.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:46.815 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:31:47.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:31:47.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.544+0000 7fb192119700 1 Processor -- start 2023-03-03T16:31:47.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.544+0000 7fb192119700 1 -- start start 2023-03-03T16:31:47.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb192119700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb18c14f9d0 0x7fb18c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:47.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb192119700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18c0c2f30 con 0x7fb18c0abe20 2023-03-03T16:31:47.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb192119700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18c0c31d0 con 0x7fb18c14fda0 2023-03-03T16:31:47.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb192119700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb18c1524a0 con 0x7fb18c14f9d0 2023-03-03T16:31:47.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb18b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb18c14f9d0 0x7fb18c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:47.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb18bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb18c0abe20 0x7fb18c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51464/0 (socket says 172.21.15.7:51464) 2023-03-03T16:31:47.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb18bfff700 1 -- 172.21.15.7:0/3168655075 learned_addr learned my addr 172.21.15.7:0/3168655075 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:31:47.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb18b7fe700 1 -- 172.21.15.7:0/3168655075 >> v1:172.21.15.143:6789/0 conn(0x7fb18c14fda0 legacy=0x7fb18c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:31:47.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb18b7fe700 1 -- 172.21.15.7:0/3168655075 >> v1:172.21.15.7:6789/0 conn(0x7fb18c0abe20 legacy=0x7fb18c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:47.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb18b7fe700 1 -- 172.21.15.7:0/3168655075 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c0c34b0 con 0x7fb18c14f9d0 2023-03-03T16:31:47.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb18a7fc700 1 -- 172.21.15.7:0/3168655075 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726235154 0 0) 0x7fb18c0c2f30 con 0x7fb18c0abe20 2023-03-03T16:31:47.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.545+0000 7fb18b7fe700 1 --2- 172.21.15.7:0/3168655075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c14f9d0 0x7fb18c156430 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009840 tx=0x7fb17c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad5ac46d8ede7c4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:31:47.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.546+0000 7fb18a7fc700 1 -- 172.21.15.7:0/3168655075 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c004230 con 0x7fb18c14f9d0 2023-03-03T16:31:47.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.546+0000 7fb18a7fc700 1 -- 172.21.15.7:0/3168655075 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17c004390 con 0x7fb18c14f9d0 2023-03-03T16:31:47.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.546+0000 7fb18a7fc700 1 -- 172.21.15.7:0/3168655075 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c0036a0 con 0x7fb18c14f9d0 2023-03-03T16:31:47.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.546+0000 7fb192119700 1 -- 172.21.15.7:0/3168655075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c14f9d0 msgr2=0x7fb18c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:47.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.546+0000 7fb192119700 1 --2- 172.21.15.7:0/3168655075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c14f9d0 0x7fb18c156430 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009840 tx=0x7fb17c009b50 comp rx=0 tx=0).stop 2023-03-03T16:31:47.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.546+0000 7fb192119700 1 -- 172.21.15.7:0/3168655075 shutdown_connections 2023-03-03T16:31:47.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.546+0000 7fb192119700 1 --2- 172.21.15.7:0/3168655075 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c14f9d0 0x7fb18c156430 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:47.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.546+0000 7fb192119700 1 -- 172.21.15.7:0/3168655075 >> 172.21.15.7:0/3168655075 conn(0x7fb18c0c8900 msgr2=0x7fb18c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:31:47.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.546+0000 7fb192119700 1 -- 172.21.15.7:0/3168655075 shutdown_connections 2023-03-03T16:31:47.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.546+0000 7fb192119700 1 -- 172.21.15.7:0/3168655075 wait complete. 2023-03-03T16:31:47.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.548+0000 7fb192119700 1 Processor -- start 2023-03-03T16:31:47.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.548+0000 7fb192119700 1 -- start start 2023-03-03T16:31:47.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb192119700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c14abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:47.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb192119700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c14f9d0 0x7fb18c06ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:47.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb192119700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 0x7fb18c06d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:47.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb192119700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c0c24d0 con 0x7fb18c0abe20 2023-03-03T16:31:47.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb192119700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c1e8240 con 0x7fb18c14fda0 2023-03-03T16:31:47.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb192119700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c155bd0 con 0x7fb18c14f9d0 2023-03-03T16:31:47.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c14abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:47.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c14abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57486/0 (socket says 172.21.15.7:57486) 2023-03-03T16:31:47.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18b7fe700 1 -- 172.21.15.7:0/2634088460 learned_addr learned my addr 172.21.15.7:0/2634088460 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:31:47.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18bfff700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 0x7fb18c06d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:47.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18affd700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c14f9d0 0x7fb18c06ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:47.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18b7fe700 1 -- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c14f9d0 msgr2=0x7fb18c06ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:47.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18b7fe700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c14f9d0 0x7fb18c06ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:47.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18b7fe700 1 -- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 msgr2=0x7fb18c06d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:47.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18b7fe700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 0x7fb18c06d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:47.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18b7fe700 1 -- 172.21.15.7:0/2634088460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17c0094a0 con 0x7fb18c0abe20 2023-03-03T16:31:47.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.549+0000 7fb18bfff700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 0x7fb18c06d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:31:47.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.550+0000 7fb18b7fe700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c14abf0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fb18c072520 tx=0x7fb17c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:31:47.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.550+0000 7fb183fff700 1 -- 172.21.15.7:0/2634088460 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c011630 con 0x7fb18c0abe20 2023-03-03T16:31:47.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.550+0000 7fb183fff700 1 -- 172.21.15.7:0/2634088460 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17c01e070 con 0x7fb18c0abe20 2023-03-03T16:31:47.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.550+0000 7fb192119700 1 -- 172.21.15.7:0/2634088460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18c0ac350 con 0x7fb18c0abe20 2023-03-03T16:31:47.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.550+0000 7fb183fff700 1 -- 172.21.15.7:0/2634088460 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c0119b0 con 0x7fb18c0abe20 2023-03-03T16:31:47.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.550+0000 7fb192119700 1 -- 172.21.15.7:0/2634088460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18c0ab630 con 0x7fb18c0abe20 2023-03-03T16:31:47.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.551+0000 7fb192119700 1 -- 172.21.15.7:0/2634088460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18c071fa0 con 0x7fb18c0abe20 2023-03-03T16:31:47.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.551+0000 7fb183fff700 1 -- 172.21.15.7:0/2634088460 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb17c017020 con 0x7fb18c0abe20 2023-03-03T16:31:47.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.552+0000 7fb183fff700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb16c0682f0 0x7fb16c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:31:47.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.552+0000 7fb183fff700 1 -- 172.21.15.7:0/2634088460 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb17c020070 con 0x7fb18c0abe20 2023-03-03T16:31:47.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.552+0000 7fb18affd700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb16c0682f0 0x7fb16c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:31:47.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.553+0000 7fb18affd700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb16c0682f0 0x7fb16c06a7a0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb18c155490 tx=0x7fb174009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:31:47.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.556+0000 7fb183fff700 1 -- 172.21.15.7:0/2634088460 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb17c053c70 con 0x7fb18c0abe20 2023-03-03T16:31:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.799+0000 7fb192119700 1 -- 172.21.15.7:0/2634088460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb18c0c1cb0 con 0x7fb18c0abe20 2023-03-03T16:31:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.799+0000 7fb183fff700 1 -- 172.21.15.7:0/2634088460 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb18c0c1cb0 con 0x7fb18c0abe20 2023-03-03T16:31:47.800 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:31:47.800 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:31:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.801+0000 7fb192119700 1 -- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb16c0682f0 msgr2=0x7fb16c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.801+0000 7fb192119700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb16c0682f0 0x7fb16c06a7a0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fb18c155490 tx=0x7fb174009040 comp rx=0 tx=0).stop 2023-03-03T16:31:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.801+0000 7fb192119700 1 -- 172.21.15.7:0/2634088460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 msgr2=0x7fb18c14abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:31:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.801+0000 7fb192119700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c14abf0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fb18c072520 tx=0x7fb17c004750 comp rx=0 tx=0).stop 2023-03-03T16:31:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.801+0000 7fb192119700 1 -- 172.21.15.7:0/2634088460 shutdown_connections 2023-03-03T16:31:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.801+0000 7fb192119700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb18c14fda0 0x7fb18c06d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.801+0000 7fb192119700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb16c0682f0 0x7fb16c06a7a0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.801+0000 7fb192119700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb18c0abe20 0x7fb18c14abf0 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.801+0000 7fb192119700 1 --2- 172.21.15.7:0/2634088460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb18c14f9d0 0x7fb18c06ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:31:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.802+0000 7fb192119700 1 -- 172.21.15.7:0/2634088460 >> 172.21.15.7:0/2634088460 conn(0x7fb18c0c8900 msgr2=0x7fb18c151160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:31:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.802+0000 7fb192119700 1 -- 172.21.15.7:0/2634088460 shutdown_connections 2023-03-03T16:31:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:31:47.802+0000 7fb192119700 1 -- 172.21.15.7:0/2634088460 wait complete. 2023-03-03T16:31:47.804 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:31:47.814 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T16:32:06.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:32:06.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.180+0000 7f436d01f700 1 Processor -- start 2023-03-03T16:32:06.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.180+0000 7f436d01f700 1 -- start start 2023-03-03T16:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f436d01f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43680c2e30 0x7f4368158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f436d01f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4368157ff0 con 0x7f436806c6c0 2023-03-03T16:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f436d01f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436806d640 con 0x7f4368151cd0 2023-03-03T16:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f436d01f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436806cda0 con 0x7f43680c2e30 2023-03-03T16:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f436759e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f436806c6c0 0x7f43680736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51494/0 (socket says 172.21.15.7:51494) 2023-03-03T16:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f436759e700 1 -- 172.21.15.7:0/2929646272 learned_addr learned my addr 172.21.15.7:0/2929646272 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f436659c700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43680c2e30 0x7f4368158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995148808 0 0) 0x7f4368157ff0 con 0x7f436806c6c0 2023-03-03T16:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43500035b0 con 0x7f436806c6c0 2023-03-03T16:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1101935847 0 0) 0x7f43500035b0 con 0x7f436806c6c0 2023-03-03T16:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43680c2e30 msgr2=0x7f4368158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43680c2e30 0x7f4368158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 >> v1:172.21.15.143:6789/0 conn(0x7f4368151cd0 legacy=0x7f4368154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436806d080 con 0x7f436806c6c0 2023-03-03T16:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f43600019f0 con 0x7f436806c6c0 2023-03-03T16:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 >> v1:172.21.15.7:6789/0 conn(0x7f436806c6c0 legacy=0x7f43680736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4350003aa0 0x7f4350003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4350004850 0x7f4350004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43680c2e30 0x7f43500052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436806cda0 con 0x7f4350003aa0 2023-03-03T16:32:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435000bba0 con 0x7f4350004850 2023-03-03T16:32:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.181+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435000bd60 con 0x7f43680c2e30 2023-03-03T16:32:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f4366d9d700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4350003aa0 0x7f4350003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f436759e700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43680c2e30 0x7f43500052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f436659c700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4350004850 0x7f4350004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:06.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f4366d9d700 1 -- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43680c2e30 msgr2=0x7f43500052b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:32:06.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f4366d9d700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43680c2e30 0x7f43500052b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:06.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f4366d9d700 1 -- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4350004850 msgr2=0x7f4350004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:32:06.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f4366d9d700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4350004850 0x7f4350004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:06.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f4366d9d700 1 -- 172.21.15.7:0/2929646272 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43500035b0 con 0x7f4350003aa0 2023-03-03T16:32:06.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f4366d9d700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4350003aa0 0x7f4350003e70 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f4358001880 tx=0x7f4350005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dea4221258eb2ef server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:06.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4358001370 con 0x7f4350003aa0 2023-03-03T16:32:06.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4358012640 con 0x7f4350003aa0 2023-03-03T16:32:06.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.182+0000 7f4365d9b700 1 -- 172.21.15.7:0/2929646272 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43580129f0 con 0x7f4350003aa0 2023-03-03T16:32:06.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.183+0000 7f436d01f700 1 -- 172.21.15.7:0/2929646272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4350003aa0 msgr2=0x7f4350003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:06.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.183+0000 7f436d01f700 1 --2- 172.21.15.7:0/2929646272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4350003aa0 0x7f4350003e70 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f4358001880 tx=0x7f4350005880 comp rx=0 tx=0).stop 2023-03-03T16:32:06.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.183+0000 7f436759e700 1 -- 172.21.15.7:0/2929646272 reap_dead start 2023-03-03T16:32:06.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.183+0000 7f436d01f700 1 -- 172.21.15.7:0/2929646272 shutdown_connections 2023-03-03T16:32:06.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.183+0000 7f436d01f700 1 -- 172.21.15.7:0/2929646272 >> 172.21.15.7:0/2929646272 conn(0x7f43680a94d0 msgr2=0x7f43680a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:32:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.183+0000 7f436d01f700 1 -- 172.21.15.7:0/2929646272 shutdown_connections 2023-03-03T16:32:06.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.183+0000 7f436d01f700 1 -- 172.21.15.7:0/2929646272 wait complete. 2023-03-03T16:32:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.184+0000 7f436d01f700 1 Processor -- start 2023-03-03T16:32:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.184+0000 7f436d01f700 1 -- start start 2023-03-03T16:32:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f436d01f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4350003aa0 0x7f43680ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f436d01f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436814b9b0 0x7f4368153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:06.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f436d01f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43681ed160 0x7f436814a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f436d01f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43680ab8d0 con 0x7f436814b9b0 2023-03-03T16:32:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f436d01f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43680aa450 con 0x7f4350003aa0 2023-03-03T16:32:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f436d01f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43680aada0 con 0x7f43681ed160 2023-03-03T16:32:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f436659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436814b9b0 0x7f4368153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f436659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436814b9b0 0x7f4368153dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57524/0 (socket says 172.21.15.7:57524) 2023-03-03T16:32:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f436659c700 1 -- 172.21.15.7:0/3685049147 learned_addr learned my addr 172.21.15.7:0/3685049147 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:32:06.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f4366d9d700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4350003aa0 0x7f43680ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.185+0000 7f436759e700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43681ed160 0x7f436814a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f436659c700 1 -- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43681ed160 msgr2=0x7f436814a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f436659c700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43681ed160 0x7f436814a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f436659c700 1 -- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4350003aa0 msgr2=0x7f43680ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f436659c700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4350003aa0 0x7f43680ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f436659c700 1 -- 172.21.15.7:0/3685049147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4368157c80 con 0x7f436814b9b0 2023-03-03T16:32:06.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f436659c700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436814b9b0 0x7f4368153dc0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f435c0119a0 tx=0x7f435c011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f434f7fe700 1 -- 172.21.15.7:0/3685049147 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c00f780 con 0x7f436814b9b0 2023-03-03T16:32:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f434f7fe700 1 -- 172.21.15.7:0/3685049147 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435c00f8e0 con 0x7f436814b9b0 2023-03-03T16:32:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f436d01f700 1 -- 172.21.15.7:0/3685049147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436806d360 con 0x7f436814b9b0 2023-03-03T16:32:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f434f7fe700 1 -- 172.21.15.7:0/3685049147 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c00fc70 con 0x7f436814b9b0 2023-03-03T16:32:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.186+0000 7f436d01f700 1 -- 172.21.15.7:0/3685049147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4368073e00 con 0x7f436814b9b0 2023-03-03T16:32:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.187+0000 7f436d01f700 1 -- 172.21.15.7:0/3685049147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4368074bb0 con 0x7f436814b9b0 2023-03-03T16:32:06.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.187+0000 7f434f7fe700 1 -- 172.21.15.7:0/3685049147 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f435c014440 con 0x7f436814b9b0 2023-03-03T16:32:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.188+0000 7f434f7fe700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43540682f0 0x7f435406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.188+0000 7f434f7fe700 1 -- 172.21.15.7:0/3685049147 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f435c086740 con 0x7f436814b9b0 2023-03-03T16:32:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.188+0000 7f4366d9d700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43540682f0 0x7f435406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.189+0000 7f4366d9d700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43540682f0 0x7f435406a7a0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f4358001880 tx=0x7f4358005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:06.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.192+0000 7f434f7fe700 1 -- 172.21.15.7:0/3685049147 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f435c052a80 con 0x7f436814b9b0 2023-03-03T16:32:06.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.432+0000 7f436d01f700 1 -- 172.21.15.7:0/3685049147 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43680a9fb0 con 0x7f436814b9b0 2023-03-03T16:32:06.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.434+0000 7f434f7fe700 1 -- 172.21.15.7:0/3685049147 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f435c022090 con 0x7f436814b9b0 2023-03-03T16:32:06.435 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:32:06.435 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:32:06.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 -- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43540682f0 msgr2=0x7f435406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:06.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43540682f0 0x7f435406a7a0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f4358001880 tx=0x7f4358005040 comp rx=0 tx=0).stop 2023-03-03T16:32:06.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 -- 172.21.15.7:0/3685049147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436814b9b0 msgr2=0x7f4368153dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:06.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436814b9b0 0x7f4368153dc0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f435c0119a0 tx=0x7f435c011cb0 comp rx=0 tx=0).stop 2023-03-03T16:32:06.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 -- 172.21.15.7:0/3685049147 shutdown_connections 2023-03-03T16:32:06.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43681ed160 0x7f436814a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:06.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4350003aa0 0x7f43680ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:06.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f43540682f0 0x7f435406a7a0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:06.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 --2- 172.21.15.7:0/3685049147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436814b9b0 0x7f4368153dc0 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:06.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 -- 172.21.15.7:0/3685049147 >> 172.21.15.7:0/3685049147 conn(0x7f43680a94d0 msgr2=0x7f43680c2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:32:06.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 -- 172.21.15.7:0/3685049147 shutdown_connections 2023-03-03T16:32:06.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:06.436+0000 7f436d01f700 1 -- 172.21.15.7:0/3685049147 wait complete. 2023-03-03T16:32:06.439 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:32:06.449 INFO:tasks.mds_thrash.fs.[b]:waiting for 30 secs before thrashing 2023-03-03T16:32:16.921 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:16.925 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:36.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:32:36.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.589+0000 7fe161d1a700 1 Processor -- start 2023-03-03T16:32:36.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.589+0000 7fe161d1a700 1 -- start start 2023-03-03T16:32:36.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe161d1a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15c0caa30 0x7fe15c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:36.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe161d1a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe15c0c00e0 0x7fe15c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:36.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe161d1a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe15c0cb280 con 0x7fe15c0cae00 2023-03-03T16:32:36.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe161d1a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c0c7310 con 0x7fe15c0c00e0 2023-03-03T16:32:36.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe161d1a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0bfd10 con 0x7fe15c0caa30 2023-03-03T16:32:36.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe15affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe15c0c00e0 0x7fe15c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:36.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe15bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe15c0cae00 0x7fe15c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51536/0 (socket says 172.21.15.7:51536) 2023-03-03T16:32:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe15bfff700 1 -- 172.21.15.7:0/2644779695 learned_addr learned my addr 172.21.15.7:0/2644779695 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:32:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe15b7fe700 1 --2- 172.21.15.7:0/2644779695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15c0caa30 0x7fe15c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe15affd700 1 -- 172.21.15.7:0/2644779695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15c0caa30 msgr2=0x7fe15c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:32:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe15affd700 1 --2- 172.21.15.7:0/2644779695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15c0caa30 0x7fe15c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe15affd700 1 -- 172.21.15.7:0/2644779695 >> v1:172.21.15.7:6789/0 conn(0x7fe15c0cae00 legacy=0x7fe15c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe15affd700 1 -- 172.21.15.7:0/2644779695 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c0bfa30 con 0x7fe15c0c00e0 2023-03-03T16:32:36.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.590+0000 7fe15affd700 1 --2- 172.21.15.7:0/2644779695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0c00e0 0x7fe15c154ff0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fe144009820 tx=0x7fe144009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5355ec3e0ebb6000 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:36.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe15a7fc700 1 -- 172.21.15.7:0/2644779695 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe144005e30 con 0x7fe15c0c00e0 2023-03-03T16:32:36.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe15a7fc700 1 -- 172.21.15.7:0/2644779695 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe144006030 con 0x7fe15c0c00e0 2023-03-03T16:32:36.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe15a7fc700 1 -- 172.21.15.7:0/2644779695 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1440063e0 con 0x7fe15c0c00e0 2023-03-03T16:32:36.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe161d1a700 1 -- 172.21.15.7:0/2644779695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0c00e0 msgr2=0x7fe15c154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:36.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe161d1a700 1 --2- 172.21.15.7:0/2644779695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0c00e0 0x7fe15c154ff0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fe144009820 tx=0x7fe144009b30 comp rx=0 tx=0).stop 2023-03-03T16:32:36.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe161d1a700 1 -- 172.21.15.7:0/2644779695 shutdown_connections 2023-03-03T16:32:36.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe161d1a700 1 --2- 172.21.15.7:0/2644779695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15c0caa30 0x7fe15c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:36.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe161d1a700 1 --2- 172.21.15.7:0/2644779695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0c00e0 0x7fe15c154ff0 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:36.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe161d1a700 1 -- 172.21.15.7:0/2644779695 >> 172.21.15.7:0/2644779695 conn(0x7fe15c066e40 msgr2=0x7fe15c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:32:36.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe161d1a700 1 -- 172.21.15.7:0/2644779695 shutdown_connections 2023-03-03T16:32:36.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.591+0000 7fe161d1a700 1 -- 172.21.15.7:0/2644779695 wait complete. 2023-03-03T16:32:36.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.592+0000 7fe161d1a700 1 Processor -- start 2023-03-03T16:32:36.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe161d1a700 1 -- start start 2023-03-03T16:32:36.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe161d1a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe15c0c00e0 0x7fe15c1e9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:36.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe161d1a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15c0caa30 0x7fe15c1e1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:36.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe161d1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0cae00 0x7fe15c14c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:36.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe161d1a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0c3e60 con 0x7fe15c0cae00 2023-03-03T16:32:36.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe161d1a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0c1c80 con 0x7fe15c0c00e0 2023-03-03T16:32:36.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe161d1a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c1ea4d0 con 0x7fe15c0caa30 2023-03-03T16:32:36.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe15bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0cae00 0x7fe15c14c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:36.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe15bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0cae00 0x7fe15c14c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57562/0 (socket says 172.21.15.7:57562) 2023-03-03T16:32:36.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe15bfff700 1 -- 172.21.15.7:0/614436857 learned_addr learned my addr 172.21.15.7:0/614436857 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:32:36.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe15b7fe700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe15c0c00e0 0x7fe15c1e9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:36.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.593+0000 7fe15affd700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15c0caa30 0x7fe15c1e1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:36.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe15bfff700 1 -- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15c0caa30 msgr2=0x7fe15c1e1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe15bfff700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15c0caa30 0x7fe15c1e1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe15bfff700 1 -- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe15c0c00e0 msgr2=0x7fe15c1e9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe15bfff700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe15c0c00e0 0x7fe15c1e9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe15bfff700 1 -- 172.21.15.7:0/614436857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe144009430 con 0x7fe15c0cae00 2023-03-03T16:32:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe15b7fe700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe15c0c00e0 0x7fe15c1e9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:32:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe15bfff700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0cae00 0x7fe15c14c040 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fe15400ca80 tx=0x7fe15400cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:36.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe153fff700 1 -- 172.21.15.7:0/614436857 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15400a9e0 con 0x7fe15c0cae00 2023-03-03T16:32:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe153fff700 1 -- 172.21.15.7:0/614436857 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15400ab40 con 0x7fe15c0cae00 2023-03-03T16:32:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe161d1a700 1 -- 172.21.15.7:0/614436857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c1547f0 con 0x7fe15c0cae00 2023-03-03T16:32:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe153fff700 1 -- 172.21.15.7:0/614436857 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154012400 con 0x7fe15c0cae00 2023-03-03T16:32:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.594+0000 7fe161d1a700 1 -- 172.21.15.7:0/614436857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c154510 con 0x7fe15c0cae00 2023-03-03T16:32:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.595+0000 7fe161d1a700 1 -- 172.21.15.7:0/614436857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c154ad0 con 0x7fe15c0cae00 2023-03-03T16:32:36.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.595+0000 7fe153fff700 1 -- 172.21.15.7:0/614436857 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe154019030 con 0x7fe15c0cae00 2023-03-03T16:32:36.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.596+0000 7fe153fff700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe13c0682f0 0x7fe13c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:36.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.596+0000 7fe153fff700 1 -- 172.21.15.7:0/614436857 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe154085b20 con 0x7fe15c0cae00 2023-03-03T16:32:36.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.596+0000 7fe15b7fe700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe13c0682f0 0x7fe13c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.597+0000 7fe15b7fe700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe13c0682f0 0x7fe13c06a7a0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fe14c0099f0 tx=0x7fe14c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:36.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.600+0000 7fe153fff700 1 -- 172.21.15.7:0/614436857 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe15404b5e0 con 0x7fe15c0cae00 2023-03-03T16:32:36.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.837+0000 7fe161d1a700 1 -- 172.21.15.7:0/614436857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe15c074bb0 con 0x7fe15c0cae00 2023-03-03T16:32:36.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.837+0000 7fe153fff700 1 -- 172.21.15.7:0/614436857 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe154051f70 con 0x7fe15c0cae00 2023-03-03T16:32:36.838 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:32:36.838 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:32:36.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 -- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe13c0682f0 msgr2=0x7fe13c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe13c0682f0 0x7fe13c06a7a0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fe14c0099f0 tx=0x7fe14c008040 comp rx=0 tx=0).stop 2023-03-03T16:32:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 -- 172.21.15.7:0/614436857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0cae00 msgr2=0x7fe15c14c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0cae00 0x7fe15c14c040 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fe15400ca80 tx=0x7fe15400cd90 comp rx=0 tx=0).stop 2023-03-03T16:32:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 -- 172.21.15.7:0/614436857 shutdown_connections 2023-03-03T16:32:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe13c0682f0 0x7fe13c06a7a0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe15c0cae00 0x7fe15c14c040 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:36.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe15c0c00e0 0x7fe15c1e9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 --2- 172.21.15.7:0/614436857 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe15c0caa30 0x7fe15c1e1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 -- 172.21.15.7:0/614436857 >> 172.21.15.7:0/614436857 conn(0x7fe15c066e40 msgr2=0x7fe15c06b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:32:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 -- 172.21.15.7:0/614436857 shutdown_connections 2023-03-03T16:32:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:36.839+0000 7fe161d1a700 1 -- 172.21.15.7:0/614436857 wait complete. 2023-03-03T16:32:36.841 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:32:36.851 INFO:tasks.mds_thrash.fs.[b]:waiting for 13 secs before thrashing 2023-03-03T16:32:46.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:46.997 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:32:49.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:32:49.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.991+0000 7f66aaa9d700 1 Processor -- start 2023-03-03T16:32:49.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.991+0000 7f66aaa9d700 1 -- start start 2023-03-03T16:32:49.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.991+0000 7f66aaa9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40ace30 0x7f66a4152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:49.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.991+0000 7f66aaa9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66a40c26e0 0x7f66a4152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:49.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.991+0000 7f66aaa9d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a40ac900 con 0x7f66a40ac220 2023-03-03T16:32:49.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.991+0000 7f66aaa9d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a4071e30 con 0x7f66a40c26e0 2023-03-03T16:32:49.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.991+0000 7f66aaa9d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a4152ed0 con 0x7f66a40ace30 2023-03-03T16:32:49.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a903a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66a40c26e0 0x7f66a4152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:49.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a8839700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f66a40ac220 0x7f66a40736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51552/0 (socket says 172.21.15.7:51552) 2023-03-03T16:32:49.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a8839700 1 -- 172.21.15.7:0/363386931 learned_addr learned my addr 172.21.15.7:0/363386931 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:32:49.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a3fff700 1 --2- 172.21.15.7:0/363386931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40ace30 0x7f66a4152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:49.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a903a700 1 -- 172.21.15.7:0/363386931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40ace30 msgr2=0x7f66a4152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:32:49.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a903a700 1 --2- 172.21.15.7:0/363386931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40ace30 0x7f66a4152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:49.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a903a700 1 -- 172.21.15.7:0/363386931 >> v1:172.21.15.7:6789/0 conn(0x7f66a40ac220 legacy=0x7f66a40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:49.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a903a700 1 -- 172.21.15.7:0/363386931 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a40acb50 con 0x7f66a40c26e0 2023-03-03T16:32:49.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a37fe700 1 -- 172.21.15.7:0/363386931 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888746899 0 0) 0x7f66a40ac900 con 0x7f66a40ac220 2023-03-03T16:32:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a3fff700 1 --2- 172.21.15.7:0/363386931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40ace30 0x7f66a4152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:32:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a903a700 1 --2- 172.21.15.7:0/363386931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40c26e0 0x7f66a4152990 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f669c009d90 tx=0x7f669c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692e1467fbc4d232 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a37fe700 1 -- 172.21.15.7:0/363386931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c00e070 con 0x7f66a40c26e0 2023-03-03T16:32:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a37fe700 1 -- 172.21.15.7:0/363386931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c00bbd0 con 0x7f66a40c26e0 2023-03-03T16:32:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.992+0000 7f66a37fe700 1 -- 172.21.15.7:0/363386931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c012430 con 0x7f66a40c26e0 2023-03-03T16:32:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.993+0000 7f66aaa9d700 1 -- 172.21.15.7:0/363386931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40c26e0 msgr2=0x7f66a4152990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.993+0000 7f66aaa9d700 1 --2- 172.21.15.7:0/363386931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40c26e0 0x7f66a4152990 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f669c009d90 tx=0x7f669c00b330 comp rx=0 tx=0).stop 2023-03-03T16:32:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.993+0000 7f66aaa9d700 1 -- 172.21.15.7:0/363386931 shutdown_connections 2023-03-03T16:32:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.993+0000 7f66aaa9d700 1 --2- 172.21.15.7:0/363386931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40c26e0 0x7f66a4152990 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.993+0000 7f66aaa9d700 1 --2- 172.21.15.7:0/363386931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40ace30 0x7f66a4152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.993+0000 7f66aaa9d700 1 -- 172.21.15.7:0/363386931 >> 172.21.15.7:0/363386931 conn(0x7f66a40a94d0 msgr2=0x7f66a40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:32:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.993+0000 7f66aaa9d700 1 -- 172.21.15.7:0/363386931 shutdown_connections 2023-03-03T16:32:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.993+0000 7f66aaa9d700 1 -- 172.21.15.7:0/363386931 wait complete. 2023-03-03T16:32:49.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.994+0000 7f66aaa9d700 1 Processor -- start 2023-03-03T16:32:49.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.994+0000 7f66aaa9d700 1 -- start start 2023-03-03T16:32:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66aaa9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40ac220 0x7f66a40bc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66aaa9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a40ace30 0x7f66a40be430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66aaa9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40c26e0 0x7f66a40be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66aaa9d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a414ee50 con 0x7f66a40ac220 2023-03-03T16:32:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66aaa9d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40bbfa0 con 0x7f66a40ace30 2023-03-03T16:32:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66aaa9d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40c07e0 con 0x7f66a40c26e0 2023-03-03T16:32:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a8839700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40ac220 0x7f66a40bc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:50.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a8839700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40ac220 0x7f66a40bc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57578/0 (socket says 172.21.15.7:57578) 2023-03-03T16:32:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a8839700 1 -- 172.21.15.7:0/3600729587 learned_addr learned my addr 172.21.15.7:0/3600729587 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:32:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a3fff700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a40ace30 0x7f66a40be430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a903a700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40c26e0 0x7f66a40be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a8839700 1 -- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40c26e0 msgr2=0x7f66a40be970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a8839700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40c26e0 0x7f66a40be970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a8839700 1 -- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a40ace30 msgr2=0x7f66a40be430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:50.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a8839700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a40ace30 0x7f66a40be430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:50.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a8839700 1 -- 172.21.15.7:0/3600729587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c009a70 con 0x7f66a40ac220 2023-03-03T16:32:50.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a3fff700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a40ace30 0x7f66a40be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:32:50.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.995+0000 7f66a8839700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40ac220 0x7f66a40bc3d0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f669800c580 tx=0x7f669800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:50.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.996+0000 7f66a17fa700 1 -- 172.21.15.7:0/3600729587 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6698004c80 con 0x7f66a40ac220 2023-03-03T16:32:50.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.996+0000 7f66a17fa700 1 -- 172.21.15.7:0/3600729587 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6698004e00 con 0x7f66a40ac220 2023-03-03T16:32:50.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.996+0000 7f66aaa9d700 1 -- 172.21.15.7:0/3600729587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a4155ca0 con 0x7f66a40ac220 2023-03-03T16:32:50.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.996+0000 7f66a17fa700 1 -- 172.21.15.7:0/3600729587 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6698007310 con 0x7f66a40ac220 2023-03-03T16:32:50.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.996+0000 7f66aaa9d700 1 -- 172.21.15.7:0/3600729587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a41559c0 con 0x7f66a40ac220 2023-03-03T16:32:50.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.997+0000 7f66aaa9d700 1 -- 172.21.15.7:0/3600729587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a4155f80 con 0x7f66a40ac220 2023-03-03T16:32:50.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.997+0000 7f66a17fa700 1 -- 172.21.15.7:0/3600729587 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f669801a020 con 0x7f66a40ac220 2023-03-03T16:32:50.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.998+0000 7f66a17fa700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66880682f0 0x7f668806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:50.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.998+0000 7f66a17fa700 1 -- 172.21.15.7:0/3600729587 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6698085e30 con 0x7f66a40ac220 2023-03-03T16:32:50.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.998+0000 7f66a3fff700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66880682f0 0x7f668806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:50.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:49.998+0000 7f66a3fff700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66880682f0 0x7f668806a7a0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f66a41df7a0 tx=0x7f6690008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.002+0000 7f66a17fa700 1 -- 172.21.15.7:0/3600729587 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6698052080 con 0x7f66a40ac220 2023-03-03T16:32:50.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.240+0000 7f66aaa9d700 1 -- 172.21.15.7:0/3600729587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66a4074bb0 con 0x7f66a40ac220 2023-03-03T16:32:50.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.240+0000 7f66a17fa700 1 -- 172.21.15.7:0/3600729587 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f66980556a0 con 0x7f66a40ac220 2023-03-03T16:32:50.241 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:32:50.241 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:32:50.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 -- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66880682f0 msgr2=0x7f668806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:50.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66880682f0 0x7f668806a7a0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f66a41df7a0 tx=0x7f6690008040 comp rx=0 tx=0).stop 2023-03-03T16:32:50.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 -- 172.21.15.7:0/3600729587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40ac220 msgr2=0x7f66a40bc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:50.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40ac220 0x7f66a40bc3d0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f669800c580 tx=0x7f669800c940 comp rx=0 tx=0).stop 2023-03-03T16:32:50.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 -- 172.21.15.7:0/3600729587 shutdown_connections 2023-03-03T16:32:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66a40c26e0 0x7f66a40be970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66880682f0 0x7f668806a7a0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66a40ac220 0x7f66a40bc3d0 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 --2- 172.21.15.7:0/3600729587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66a40ace30 0x7f66a40be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 -- 172.21.15.7:0/3600729587 >> 172.21.15.7:0/3600729587 conn(0x7f66a40a94d0 msgr2=0x7f66a40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:32:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 -- 172.21.15.7:0/3600729587 shutdown_connections 2023-03-03T16:32:50.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:50.243+0000 7f66aaa9d700 1 -- 172.21.15.7:0/3600729587 wait complete. 2023-03-03T16:32:50.245 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:32:50.255 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before thrashing 2023-03-03T16:32:53.815 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:32:53.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.953+0000 7f9680aa9700 1 Processor -- start 2023-03-03T16:32:53.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.953+0000 7f9680aa9700 1 -- start start 2023-03-03T16:32:53.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.954+0000 7f9680aa9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f967c0c2930 0x7f967c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:53.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.954+0000 7f9680aa9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c06d430 con 0x7f967c151d40 2023-03-03T16:32:53.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.954+0000 7f9680aa9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967c1586e0 con 0x7f967c06c480 2023-03-03T16:32:53.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.954+0000 7f9680aa9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f967c06c2a0 con 0x7f967c0c2930 2023-03-03T16:32:53.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.954+0000 7f967a59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f967c151d40 0x7f967c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51566/0 (socket says 172.21.15.7:51566) 2023-03-03T16:32:53.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.954+0000 7f967a59c700 1 -- 172.21.15.7:0/2181414815 learned_addr learned my addr 172.21.15.7:0/2181414815 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:32:53.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.954+0000 7f9679d9b700 1 --2- 172.21.15.7:0/2181414815 >> v2:172.21.15.143:3300/0 conn(0x7f967c0c2930 0x7f967c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:53.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.954+0000 7f967959a700 1 -- 172.21.15.7:0/2181414815 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561639753 0 0) 0x7f967c06d430 con 0x7f967c151d40 2023-03-03T16:32:53.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.954+0000 7f967959a700 1 -- 172.21.15.7:0/2181414815 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96640035b0 con 0x7f967c151d40 2023-03-03T16:32:53.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.955+0000 7f9679d9b700 1 -- 172.21.15.7:0/2181414815 >> v1:172.21.15.143:6789/0 conn(0x7f967c06c480 legacy=0x7f967c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:53.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.955+0000 7f9679d9b700 1 -- 172.21.15.7:0/2181414815 >> v1:172.21.15.7:6789/0 conn(0x7f967c151d40 legacy=0x7f967c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:53.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.955+0000 7f9679d9b700 1 -- 172.21.15.7:0/2181414815 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c06cb00 con 0x7f967c0c2930 2023-03-03T16:32:53.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.955+0000 7f9679d9b700 1 --2- 172.21.15.7:0/2181414815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c0c2930 0x7f967c157a90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f966c009870 tx=0x7f966c009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b23a26adc69cfc4e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.955+0000 7f967959a700 1 -- 172.21.15.7:0/2181414815 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c011830 con 0x7f967c0c2930 2023-03-03T16:32:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.955+0000 7f967959a700 1 -- 172.21.15.7:0/2181414815 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f966c011990 con 0x7f967c0c2930 2023-03-03T16:32:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.955+0000 7f967959a700 1 -- 172.21.15.7:0/2181414815 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c011d00 con 0x7f967c0c2930 2023-03-03T16:32:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.956+0000 7f9680aa9700 1 -- 172.21.15.7:0/2181414815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c0c2930 msgr2=0x7f967c157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.956+0000 7f9680aa9700 1 --2- 172.21.15.7:0/2181414815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c0c2930 0x7f967c157a90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f966c009870 tx=0x7f966c009b80 comp rx=0 tx=0).stop 2023-03-03T16:32:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.956+0000 7f9680aa9700 1 -- 172.21.15.7:0/2181414815 shutdown_connections 2023-03-03T16:32:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.956+0000 7f9680aa9700 1 --2- 172.21.15.7:0/2181414815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c0c2930 0x7f967c157a90 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.956+0000 7f9680aa9700 1 -- 172.21.15.7:0/2181414815 >> 172.21.15.7:0/2181414815 conn(0x7f967c0a94d0 msgr2=0x7f967c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:32:53.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.956+0000 7f9680aa9700 1 -- 172.21.15.7:0/2181414815 shutdown_connections 2023-03-03T16:32:53.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.956+0000 7f9680aa9700 1 -- 172.21.15.7:0/2181414815 wait complete. 2023-03-03T16:32:53.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.957+0000 7f9680aa9700 1 Processor -- start 2023-03-03T16:32:53.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.957+0000 7f9680aa9700 1 -- start start 2023-03-03T16:32:53.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f9680aa9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c06c480 0x7f967c0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:53.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f9680aa9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c0c2930 0x7f967c1ed6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:53.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f9680aa9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c151d40 0x7f967c0aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:53.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f9680aa9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c1ea8f0 con 0x7f967c151d40 2023-03-03T16:32:53.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f9680aa9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c0ac3b0 con 0x7f967c06c480 2023-03-03T16:32:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f9680aa9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c1ea2e0 con 0x7f967c0c2930 2023-03-03T16:32:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f967ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c151d40 0x7f967c0aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f967ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c151d40 0x7f967c0aa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57596/0 (socket says 172.21.15.7:57596) 2023-03-03T16:32:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f967ad9d700 1 -- 172.21.15.7:0/3094362392 learned_addr learned my addr 172.21.15.7:0/3094362392 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:32:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f967a59c700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c06c480 0x7f967c0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f9679d9b700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c0c2930 0x7f967c1ed6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f967ad9d700 1 -- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c0c2930 msgr2=0x7f967c1ed6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f967ad9d700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c0c2930 0x7f967c1ed6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f967ad9d700 1 -- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c06c480 msgr2=0x7f967c0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f967ad9d700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c06c480 0x7f967c0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f967ad9d700 1 -- 172.21.15.7:0/3094362392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c0094a0 con 0x7f967c151d40 2023-03-03T16:32:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f9679d9b700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c0c2930 0x7f967c1ed6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:32:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.958+0000 7f967ad9d700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c151d40 0x7f967c0aa1d0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f967400cb50 tx=0x7f967400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.959+0000 7f966affd700 1 -- 172.21.15.7:0/3094362392 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9674003c00 con 0x7f967c151d40 2023-03-03T16:32:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.959+0000 7f966affd700 1 -- 172.21.15.7:0/3094362392 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9674003d60 con 0x7f967c151d40 2023-03-03T16:32:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.959+0000 7f9680aa9700 1 -- 172.21.15.7:0/3094362392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c06d0c0 con 0x7f967c151d40 2023-03-03T16:32:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.959+0000 7f966affd700 1 -- 172.21.15.7:0/3094362392 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967400aac0 con 0x7f967c151d40 2023-03-03T16:32:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.959+0000 7f9680aa9700 1 -- 172.21.15.7:0/3094362392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c06cde0 con 0x7f967c151d40 2023-03-03T16:32:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.960+0000 7f9680aa9700 1 -- 172.21.15.7:0/3094362392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c073e00 con 0x7f967c151d40 2023-03-03T16:32:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.960+0000 7f966affd700 1 -- 172.21.15.7:0/3094362392 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9674010050 con 0x7f967c151d40 2023-03-03T16:32:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.961+0000 7f966affd700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f965c0682f0 0x7f965c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.961+0000 7f966affd700 1 -- 172.21.15.7:0/3094362392 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f96740860e0 con 0x7f967c151d40 2023-03-03T16:32:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.961+0000 7f967a59c700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f965c0682f0 0x7f965c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:53.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.962+0000 7f967a59c700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f965c0682f0 0x7f965c06a7a0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f9670007fd0 tx=0x7f9670007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:53.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:53.965+0000 7f966affd700 1 -- 172.21.15.7:0/3094362392 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9674052310 con 0x7f967c151d40 2023-03-03T16:32:54.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.201+0000 7f9680aa9700 1 -- 172.21.15.7:0/3094362392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f967c1ea140 con 0x7f967c151d40 2023-03-03T16:32:54.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.201+0000 7f966affd700 1 -- 172.21.15.7:0/3094362392 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f967c1ea140 con 0x7f967c151d40 2023-03-03T16:32:54.202 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:32:54.202 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:32:54.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 -- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f965c0682f0 msgr2=0x7f965c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:54.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f965c0682f0 0x7f965c06a7a0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f9670007fd0 tx=0x7f9670007dc0 comp rx=0 tx=0).stop 2023-03-03T16:32:54.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 -- 172.21.15.7:0/3094362392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c151d40 msgr2=0x7f967c0aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:54.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c151d40 0x7f967c0aa1d0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f967400cb50 tx=0x7f967400ce60 comp rx=0 tx=0).stop 2023-03-03T16:32:54.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 -- 172.21.15.7:0/3094362392 shutdown_connections 2023-03-03T16:32:54.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f965c0682f0 0x7f965c06a7a0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:54.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c151d40 0x7f967c0aa1d0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:54.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c06c480 0x7f967c0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 --2- 172.21.15.7:0/3094362392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c0c2930 0x7f967c1ed6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 -- 172.21.15.7:0/3094362392 >> 172.21.15.7:0/3094362392 conn(0x7f967c0a94d0 msgr2=0x7f967c0c3470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:32:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 -- 172.21.15.7:0/3094362392 shutdown_connections 2023-03-03T16:32:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:54.203+0000 7f9680aa9700 1 -- 172.21.15.7:0/3094362392 wait complete. 2023-03-03T16:32:54.205 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:32:54.215 INFO:tasks.mds_thrash.fs.[a]:waiting for 92 secs before thrashing 2023-03-03T16:32:55.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:32:55.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.395+0000 7f6974aad700 1 Processor -- start 2023-03-03T16:32:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.395+0000 7f6974aad700 1 -- start start 2023-03-03T16:32:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.396+0000 7f6974aad700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f697014ec40 0x7f69700c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.396+0000 7f6974aad700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69700ba240 0x7f69701549f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.396+0000 7f6974aad700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700c1c00 con 0x7f69700c6560 2023-03-03T16:32:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.396+0000 7f6974aad700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69700c29a0 con 0x7f69700ba240 2023-03-03T16:32:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.396+0000 7f6974aad700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6970155640 con 0x7f697014ec40 2023-03-03T16:32:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.396+0000 7f696dd9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f69700ba240 0x7f69701549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.396+0000 7f696ed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f69700c6560 0x7f6970154f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51584/0 (socket says 172.21.15.7:51584) 2023-03-03T16:32:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696ed9d700 1 -- 172.21.15.7:0/1805977321 learned_addr learned my addr 172.21.15.7:0/1805977321 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:32:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696e59c700 1 --2- 172.21.15.7:0/1805977321 >> v2:172.21.15.143:3300/0 conn(0x7f697014ec40 0x7f69700c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696dd9b700 1 -- 172.21.15.7:0/1805977321 >> v2:172.21.15.143:3300/0 conn(0x7f697014ec40 msgr2=0x7f69700c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:32:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696dd9b700 1 --2- 172.21.15.7:0/1805977321 >> v2:172.21.15.143:3300/0 conn(0x7f697014ec40 0x7f69700c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696dd9b700 1 -- 172.21.15.7:0/1805977321 >> v1:172.21.15.7:6789/0 conn(0x7f69700c6560 legacy=0x7f6970154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696d59a700 1 -- 172.21.15.7:0/1805977321 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462941764 0 0) 0x7f69700c1c00 con 0x7f69700c6560 2023-03-03T16:32:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696dd9b700 1 -- 172.21.15.7:0/1805977321 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69700c4cf0 con 0x7f69700ba240 2023-03-03T16:32:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696dd9b700 1 --2- 172.21.15.7:0/1805977321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69700ba240 0x7f69701549f0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f69700c13e0 tx=0x7f696000bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93727299afbc0fa8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696d59a700 1 -- 172.21.15.7:0/1805977321 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000dc70 con 0x7f69700ba240 2023-03-03T16:32:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696d59a700 1 -- 172.21.15.7:0/1805977321 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696000ddd0 con 0x7f69700ba240 2023-03-03T16:32:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.397+0000 7f696d59a700 1 -- 172.21.15.7:0/1805977321 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960010690 con 0x7f69700ba240 2023-03-03T16:32:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.398+0000 7f6974aad700 1 -- 172.21.15.7:0/1805977321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69700ba240 msgr2=0x7f69701549f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.398+0000 7f6974aad700 1 --2- 172.21.15.7:0/1805977321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69700ba240 0x7f69701549f0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f69700c13e0 tx=0x7f696000bdc0 comp rx=0 tx=0).stop 2023-03-03T16:32:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.398+0000 7f6974aad700 1 -- 172.21.15.7:0/1805977321 shutdown_connections 2023-03-03T16:32:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.398+0000 7f6974aad700 1 --2- 172.21.15.7:0/1805977321 >> v2:172.21.15.143:3300/0 conn(0x7f697014ec40 0x7f69700c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.398+0000 7f6974aad700 1 --2- 172.21.15.7:0/1805977321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69700ba240 0x7f69701549f0 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.398+0000 7f6974aad700 1 -- 172.21.15.7:0/1805977321 >> 172.21.15.7:0/1805977321 conn(0x7f697006a2b0 msgr2=0x7f697006a680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:32:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.398+0000 7f6974aad700 1 -- 172.21.15.7:0/1805977321 shutdown_connections 2023-03-03T16:32:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.398+0000 7f6974aad700 1 -- 172.21.15.7:0/1805977321 wait complete. 2023-03-03T16:32:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.399+0000 7f6974aad700 1 Processor -- start 2023-03-03T16:32:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.399+0000 7f6974aad700 1 -- start start 2023-03-03T16:32:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f6974aad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69700ba240 0x7f69700c0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f6974aad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69700c6560 0x7f6970154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f6974aad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f697014ec40 0x7f697014b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f6974aad700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69701e5910 con 0x7f697014ec40 2023-03-03T16:32:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f6974aad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69701e68c0 con 0x7f69700c6560 2023-03-03T16:32:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f6974aad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69701e6430 con 0x7f69700ba240 2023-03-03T16:32:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f696ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f697014ec40 0x7f697014b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f696ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f697014ec40 0x7f697014b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57610/0 (socket says 172.21.15.7:57610) 2023-03-03T16:32:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f696ed9d700 1 -- 172.21.15.7:0/2623990286 learned_addr learned my addr 172.21.15.7:0/2623990286 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:32:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f696dd9b700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69700c6560 0x7f6970154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f696e59c700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69700ba240 0x7f69700c0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f696ed9d700 1 -- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69700ba240 msgr2=0x7f69700c0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f696ed9d700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69700ba240 0x7f69700c0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f696ed9d700 1 -- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69700c6560 msgr2=0x7f6970154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f696ed9d700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69700c6560 0x7f6970154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.400+0000 7f696ed9d700 1 -- 172.21.15.7:0/2623990286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696000b740 con 0x7f697014ec40 2023-03-03T16:32:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.401+0000 7f696ed9d700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f697014ec40 0x7f697014b7e0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f696800cc70 tx=0x7f696800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:55.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.401+0000 7f695effd700 1 -- 172.21.15.7:0/2623990286 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696800aa60 con 0x7f697014ec40 2023-03-03T16:32:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.401+0000 7f695effd700 1 -- 172.21.15.7:0/2623990286 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696800abe0 con 0x7f697014ec40 2023-03-03T16:32:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.401+0000 7f6974aad700 1 -- 172.21.15.7:0/2623990286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69700c1890 con 0x7f697014ec40 2023-03-03T16:32:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.401+0000 7f695effd700 1 -- 172.21.15.7:0/2623990286 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69680070d0 con 0x7f697014ec40 2023-03-03T16:32:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.401+0000 7f6974aad700 1 -- 172.21.15.7:0/2623990286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69700c15b0 con 0x7f697014ec40 2023-03-03T16:32:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.402+0000 7f6974aad700 1 -- 172.21.15.7:0/2623990286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69700c4f90 con 0x7f697014ec40 2023-03-03T16:32:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.402+0000 7f695effd700 1 -- 172.21.15.7:0/2623990286 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f696801a030 con 0x7f697014ec40 2023-03-03T16:32:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.403+0000 7f695effd700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69500682f0 0x7f695006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:32:55.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.403+0000 7f695effd700 1 -- 172.21.15.7:0/2623990286 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6968085d30 con 0x7f697014ec40 2023-03-03T16:32:55.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.403+0000 7f696e59c700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69500682f0 0x7f695006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:32:55.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.403+0000 7f696e59c700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69500682f0 0x7f695006a7a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f6964009be0 tx=0x7f6964009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:32:55.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.407+0000 7f695effd700 1 -- 172.21.15.7:0/2623990286 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f69680520d0 con 0x7f697014ec40 2023-03-03T16:32:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.645+0000 7f6974aad700 1 -- 172.21.15.7:0/2623990286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69700c0a80 con 0x7f697014ec40 2023-03-03T16:32:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.645+0000 7f695effd700 1 -- 172.21.15.7:0/2623990286 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6968018090 con 0x7f697014ec40 2023-03-03T16:32:55.646 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:32:55.646 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:32:55.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.647+0000 7f6974aad700 1 -- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69500682f0 msgr2=0x7f695006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:55.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.647+0000 7f6974aad700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69500682f0 0x7f695006a7a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f6964009be0 tx=0x7f6964009210 comp rx=0 tx=0).stop 2023-03-03T16:32:55.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.647+0000 7f6974aad700 1 -- 172.21.15.7:0/2623990286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f697014ec40 msgr2=0x7f697014b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:32:55.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.647+0000 7f6974aad700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f697014ec40 0x7f697014b7e0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f696800cc70 tx=0x7f696800cf80 comp rx=0 tx=0).stop 2023-03-03T16:32:55.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.647+0000 7f6974aad700 1 -- 172.21.15.7:0/2623990286 shutdown_connections 2023-03-03T16:32:55.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.647+0000 7f6974aad700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69500682f0 0x7f695006a7a0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:55.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.648+0000 7f6974aad700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f697014ec40 0x7f697014b7e0 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:55.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.648+0000 7f6974aad700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69700ba240 0x7f69700c0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:55.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.648+0000 7f6974aad700 1 --2- 172.21.15.7:0/2623990286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69700c6560 0x7f6970154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:32:55.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.648+0000 7f6974aad700 1 -- 172.21.15.7:0/2623990286 >> 172.21.15.7:0/2623990286 conn(0x7f697006a2b0 msgr2=0x7f69700a68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:32:55.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.648+0000 7f6974aad700 1 -- 172.21.15.7:0/2623990286 shutdown_connections 2023-03-03T16:32:55.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:32:55.648+0000 7f6974aad700 1 -- 172.21.15.7:0/2623990286 wait complete. 2023-03-03T16:32:55.649 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:32:55.659 INFO:tasks.mds_thrash.fs.[b]:waiting for 83 secs before thrashing 2023-03-03T16:33:17.061 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:17.065 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:47.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:33:47.141 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:17.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:17.226 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:18.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:34:18.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.802+0000 7f2cb9b8d700 1 Processor -- start 2023-03-03T16:34:18.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.802+0000 7f2cb9b8d700 1 -- start start 2023-03-03T16:34:18.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb9b8d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2cb414f9d0 0x7f2cb4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:18.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb9b8d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2cb414fda0 0x7f2cb4071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:18.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb9b8d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40c2f30 con 0x7f2cb40abe20 2023-03-03T16:34:18.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb9b8d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb40c31d0 con 0x7f2cb414fda0 2023-03-03T16:34:18.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb9b8d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb41524a0 con 0x7f2cb414f9d0 2023-03-03T16:34:18.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb2ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2cb414fda0 0x7f2cb4071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:18.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb2ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2cb414fda0 0x7f2cb4071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57674/0 (socket says 172.21.15.7:57674) 2023-03-03T16:34:18.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb2ffd700 1 -- 172.21.15.7:0/156517377 learned_addr learned my addr 172.21.15.7:0/156517377 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:34:18.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb37fe700 1 --2- 172.21.15.7:0/156517377 >> v2:172.21.15.143:3300/0 conn(0x7f2cb414f9d0 0x7f2cb4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:18.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb2ffd700 1 -- 172.21.15.7:0/156517377 >> v2:172.21.15.143:3300/0 conn(0x7f2cb414f9d0 msgr2=0x7f2cb4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:34:18.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb2ffd700 1 --2- 172.21.15.7:0/156517377 >> v2:172.21.15.143:3300/0 conn(0x7f2cb414f9d0 0x7f2cb4156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:18.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb2ffd700 1 -- 172.21.15.7:0/156517377 >> v1:172.21.15.143:6789/0 conn(0x7f2cb40abe20 legacy=0x7f2cb41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:18.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb2ffd700 1 -- 172.21.15.7:0/156517377 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb40c34b0 con 0x7f2cb414fda0 2023-03-03T16:34:18.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.803+0000 7f2cb2ffd700 1 --2- 172.21.15.7:0/156517377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb414fda0 0x7f2cb4071890 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40710a0 tx=0x7f2ca4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b768ed29b5b9fd1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:34:18.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb27fc700 1 -- 172.21.15.7:0/156517377 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca400b870 con 0x7f2cb414fda0 2023-03-03T16:34:18.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb27fc700 1 -- 172.21.15.7:0/156517377 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca400b9d0 con 0x7f2cb414fda0 2023-03-03T16:34:18.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb27fc700 1 -- 172.21.15.7:0/156517377 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca400bd60 con 0x7f2cb414fda0 2023-03-03T16:34:18.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/156517377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb414fda0 msgr2=0x7f2cb4071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:18.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb9b8d700 1 --2- 172.21.15.7:0/156517377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb414fda0 0x7f2cb4071890 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f2cb40710a0 tx=0x7f2ca4009bd0 comp rx=0 tx=0).stop 2023-03-03T16:34:18.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/156517377 shutdown_connections 2023-03-03T16:34:18.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb9b8d700 1 --2- 172.21.15.7:0/156517377 >> v2:172.21.15.143:3300/0 conn(0x7f2cb414f9d0 0x7f2cb4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:18.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb9b8d700 1 --2- 172.21.15.7:0/156517377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb414fda0 0x7f2cb4071890 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:18.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/156517377 >> 172.21.15.7:0/156517377 conn(0x7f2cb40c8900 msgr2=0x7f2cb40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:34:18.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/156517377 shutdown_connections 2023-03-03T16:34:18.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.804+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/156517377 wait complete. 2023-03-03T16:34:18.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.805+0000 7f2cb9b8d700 1 Processor -- start 2023-03-03T16:34:18.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.806+0000 7f2cb9b8d700 1 -- start start 2023-03-03T16:34:18.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.806+0000 7f2cb9b8d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb40abe20 0x7f2cb40c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:18.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.806+0000 7f2cb9b8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cb414f9d0 0x7f2cb41eb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:18.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.806+0000 7f2cb9b8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cb414fda0 0x7f2cb406e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:18.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.806+0000 7f2cb9b8d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb41ebe00 con 0x7f2cb40abe20 2023-03-03T16:34:18.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.806+0000 7f2cb9b8d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb406f710 con 0x7f2cb414f9d0 2023-03-03T16:34:18.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.806+0000 7f2cb9b8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb40bef90 con 0x7f2cb414fda0 2023-03-03T16:34:18.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.806+0000 7f2cb37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb40abe20 0x7f2cb40c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:18.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.806+0000 7f2cb37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb40abe20 0x7f2cb40c20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57678/0 (socket says 172.21.15.7:57678) 2023-03-03T16:34:18.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.806+0000 7f2cb37fe700 1 -- 172.21.15.7:0/1215466813 learned_addr learned my addr 172.21.15.7:0/1215466813 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:34:18.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2cb2ffd700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cb414f9d0 0x7f2cb41eb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:18.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2cb3fff700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cb414fda0 0x7f2cb406e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:18.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2cb37fe700 1 -- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cb414fda0 msgr2=0x7f2cb406e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:18.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2cb37fe700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cb414fda0 0x7f2cb406e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:18.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2cb37fe700 1 -- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cb414f9d0 msgr2=0x7f2cb41eb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:18.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2cb37fe700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cb414f9d0 0x7f2cb41eb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:18.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2cb37fe700 1 -- 172.21.15.7:0/1215466813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca4009550 con 0x7f2cb40abe20 2023-03-03T16:34:18.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2cb37fe700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb40abe20 0x7f2cb40c20f0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800e810 tx=0x7f2ca800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:34:18.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2c9bfff700 1 -- 172.21.15.7:0/1215466813 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8013610 con 0x7f2cb40abe20 2023-03-03T16:34:18.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2c9bfff700 1 -- 172.21.15.7:0/1215466813 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca8013770 con 0x7f2cb40abe20 2023-03-03T16:34:18.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/1215466813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb40ac350 con 0x7f2cb40abe20 2023-03-03T16:34:18.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2c9bfff700 1 -- 172.21.15.7:0/1215466813 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8013b20 con 0x7f2cb40abe20 2023-03-03T16:34:18.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.807+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/1215466813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb40ab630 con 0x7f2cb40abe20 2023-03-03T16:34:18.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.808+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/1215466813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb40c3e10 con 0x7f2cb40abe20 2023-03-03T16:34:18.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.809+0000 7f2c9bfff700 1 -- 172.21.15.7:0/1215466813 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ca8023070 con 0x7f2cb40abe20 2023-03-03T16:34:18.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.809+0000 7f2c9bfff700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 0x7f2ca006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:18.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.809+0000 7f2c9bfff700 1 -- 172.21.15.7:0/1215466813 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2ca8085f20 con 0x7f2cb40abe20 2023-03-03T16:34:18.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.809+0000 7f2cb2ffd700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 0x7f2ca006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:18.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.810+0000 7f2cb2ffd700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 0x7f2ca006a7a0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f2cb41501d0 tx=0x7f2ca4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:34:18.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:18.813+0000 7f2c9bfff700 1 -- 172.21.15.7:0/1215466813 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ca8052240 con 0x7f2cb40abe20 2023-03-03T16:34:19.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.053+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/1215466813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cb4074bb0 con 0x7f2cb40abe20 2023-03-03T16:34:19.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.053+0000 7f2c9bfff700 1 -- 172.21.15.7:0/1215466813 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2ca8059e60 con 0x7f2cb40abe20 2023-03-03T16:34:19.055 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:34:19.055 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:34:19.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 msgr2=0x7f2ca006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:19.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 0x7f2ca006a7a0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f2cb41501d0 tx=0x7f2ca4008030 comp rx=0 tx=0).stop 2023-03-03T16:34:19.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/1215466813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb40abe20 msgr2=0x7f2cb40c20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:19.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb40abe20 0x7f2cb40c20f0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800e810 tx=0x7f2ca800ebd0 comp rx=0 tx=0).stop 2023-03-03T16:34:19.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/1215466813 shutdown_connections 2023-03-03T16:34:19.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cb414fda0 0x7f2cb406e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:19.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2ca00682f0 0x7f2ca006a7a0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:19.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cb40abe20 0x7f2cb40c20f0 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:19.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 --2- 172.21.15.7:0/1215466813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cb414f9d0 0x7f2cb41eb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:19.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/1215466813 >> 172.21.15.7:0/1215466813 conn(0x7f2cb40c8900 msgr2=0x7f2cb414efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:34:19.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/1215466813 shutdown_connections 2023-03-03T16:34:19.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:19.056+0000 7f2cb9b8d700 1 -- 172.21.15.7:0/1215466813 wait complete. 2023-03-03T16:34:19.059 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:34:19.069 INFO:tasks.mds_thrash.fs.[b]:waiting for 68 secs before thrashing 2023-03-03T16:34:26.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:34:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.354+0000 7f183816b700 1 Processor -- start 2023-03-03T16:34:26.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.354+0000 7f183816b700 1 -- start start 2023-03-03T16:34:26.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.355+0000 7f183816b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18300c2e30 0x7f1830158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:26.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.355+0000 7f183816b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f183006c6c0 0x7f18300736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:26.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.355+0000 7f183816b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1830157ff0 con 0x7f1830151cd0 2023-03-03T16:34:26.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.355+0000 7f183816b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183006d640 con 0x7f18300c2e30 2023-03-03T16:34:26.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.355+0000 7f183816b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f183006cda0 con 0x7f183006c6c0 2023-03-03T16:34:26.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.355+0000 7f1835706700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18300c2e30 0x7f1830158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:26.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.355+0000 7f1835706700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18300c2e30 0x7f1830158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57688/0 (socket says 172.21.15.7:57688) 2023-03-03T16:34:26.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.355+0000 7f1835706700 1 -- 172.21.15.7:0/3139626004 learned_addr learned my addr 172.21.15.7:0/3139626004 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:34:26.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f1836708700 1 --2- 172.21.15.7:0/3139626004 >> v2:172.21.15.143:3300/0 conn(0x7f183006c6c0 0x7f18300736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:26.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f1835706700 1 -- 172.21.15.7:0/3139626004 >> v2:172.21.15.143:3300/0 conn(0x7f183006c6c0 msgr2=0x7f18300736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:34:26.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f1835706700 1 --2- 172.21.15.7:0/3139626004 >> v2:172.21.15.143:3300/0 conn(0x7f183006c6c0 0x7f18300736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f1835706700 1 -- 172.21.15.7:0/3139626004 >> v1:172.21.15.143:6789/0 conn(0x7f1830151cd0 legacy=0x7f1830154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:34:26.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f1835706700 1 -- 172.21.15.7:0/3139626004 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f183006d080 con 0x7f18300c2e30 2023-03-03T16:34:26.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f1835706700 1 --2- 172.21.15.7:0/3139626004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 0x7f1830158480 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f18300c38a0 tx=0x7f182c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3fd4451612eddff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:34:26.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f1834f05700 1 -- 172.21.15.7:0/3139626004 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c004230 con 0x7f18300c2e30 2023-03-03T16:34:26.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f1834f05700 1 -- 172.21.15.7:0/3139626004 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c004390 con 0x7f18300c2e30 2023-03-03T16:34:26.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f1834f05700 1 -- 172.21.15.7:0/3139626004 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c0036a0 con 0x7f18300c2e30 2023-03-03T16:34:26.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f183816b700 1 -- 172.21.15.7:0/3139626004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 msgr2=0x7f1830158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:26.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.356+0000 7f183816b700 1 --2- 172.21.15.7:0/3139626004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 0x7f1830158480 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7f18300c38a0 tx=0x7f182c009bd0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.357+0000 7f183816b700 1 -- 172.21.15.7:0/3139626004 shutdown_connections 2023-03-03T16:34:26.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.357+0000 7f183816b700 1 --2- 172.21.15.7:0/3139626004 >> v2:172.21.15.143:3300/0 conn(0x7f183006c6c0 0x7f18300736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.357+0000 7f183816b700 1 --2- 172.21.15.7:0/3139626004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 0x7f1830158480 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.357+0000 7f183816b700 1 -- 172.21.15.7:0/3139626004 >> 172.21.15.7:0/3139626004 conn(0x7f18300a94d0 msgr2=0x7f18300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:34:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.357+0000 7f183816b700 1 -- 172.21.15.7:0/3139626004 shutdown_connections 2023-03-03T16:34:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.357+0000 7f183816b700 1 -- 172.21.15.7:0/3139626004 wait complete. 2023-03-03T16:34:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.358+0000 7f183816b700 1 Processor -- start 2023-03-03T16:34:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.358+0000 7f183816b700 1 -- start start 2023-03-03T16:34:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f183816b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f183006c6c0 0x7f183014ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:26.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f183816b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 0x7f18301e01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f183816b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1830151cd0 0x7f183014e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f183816b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18300aad40 con 0x7f18300c2e30 2023-03-03T16:34:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f183816b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18300c6ab0 con 0x7f183006c6c0 2023-03-03T16:34:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f183816b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18300c6fe0 con 0x7f1830151cd0 2023-03-03T16:34:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f1835706700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 0x7f18301e01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f1835706700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 0x7f18301e01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57694/0 (socket says 172.21.15.7:57694) 2023-03-03T16:34:26.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f1835706700 1 -- 172.21.15.7:0/3749184696 learned_addr learned my addr 172.21.15.7:0/3749184696 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:34:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f1835f07700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f183006c6c0 0x7f183014ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.359+0000 7f1836708700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1830151cd0 0x7f183014e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f1835706700 1 -- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1830151cd0 msgr2=0x7f183014e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f1835706700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1830151cd0 0x7f183014e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f1835706700 1 -- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f183006c6c0 msgr2=0x7f183014ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f1835706700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f183006c6c0 0x7f183014ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f1835706700 1 -- 172.21.15.7:0/3749184696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182c009550 con 0x7f18300c2e30 2023-03-03T16:34:26.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f1835706700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 0x7f18301e01a0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f18300744a0 tx=0x7f182c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:34:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f18267fc700 1 -- 172.21.15.7:0/3749184696 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c003ea0 con 0x7f18300c2e30 2023-03-03T16:34:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f18267fc700 1 -- 172.21.15.7:0/3749184696 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c011630 con 0x7f18300c2e30 2023-03-03T16:34:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f183816b700 1 -- 172.21.15.7:0/3749184696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1830157c80 con 0x7f18300c2e30 2023-03-03T16:34:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f18267fc700 1 -- 172.21.15.7:0/3749184696 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c0119e0 con 0x7f18300c2e30 2023-03-03T16:34:26.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.360+0000 7f183816b700 1 -- 172.21.15.7:0/3749184696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183006d360 con 0x7f18300c2e30 2023-03-03T16:34:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.361+0000 7f183816b700 1 -- 172.21.15.7:0/3749184696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18301589c0 con 0x7f18300c2e30 2023-03-03T16:34:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.362+0000 7f18267fc700 1 -- 172.21.15.7:0/3749184696 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f182c017020 con 0x7f18300c2e30 2023-03-03T16:34:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.362+0000 7f18267fc700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18140682f0 0x7f181406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:34:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.362+0000 7f18267fc700 1 -- 172.21.15.7:0/3749184696 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f182c0874e0 con 0x7f18300c2e30 2023-03-03T16:34:26.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.362+0000 7f1835f07700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18140682f0 0x7f181406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:34:26.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.363+0000 7f1835f07700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18140682f0 0x7f181406a7a0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f1820009960 tx=0x7f18200090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:34:26.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.367+0000 7f18267fc700 1 -- 172.21.15.7:0/3749184696 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f182c053880 con 0x7f18300c2e30 2023-03-03T16:34:26.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.605+0000 7f183816b700 1 -- 172.21.15.7:0/3749184696 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1830074bb0 con 0x7f18300c2e30 2023-03-03T16:34:26.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.606+0000 7f18267fc700 1 -- 172.21.15.7:0/3749184696 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f182c056ea0 con 0x7f18300c2e30 2023-03-03T16:34:26.607 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:34:26.607 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:34:26.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 -- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18140682f0 msgr2=0x7f181406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:26.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18140682f0 0x7f181406a7a0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f1820009960 tx=0x7f18200090e0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 -- 172.21.15.7:0/3749184696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 msgr2=0x7f18301e01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:34:26.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 0x7f18301e01a0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f18300744a0 tx=0x7f182c004750 comp rx=0 tx=0).stop 2023-03-03T16:34:26.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 -- 172.21.15.7:0/3749184696 shutdown_connections 2023-03-03T16:34:26.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1830151cd0 0x7f183014e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f183006c6c0 0x7f183014ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18140682f0 0x7f181406a7a0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 --2- 172.21.15.7:0/3749184696 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18300c2e30 0x7f18301e01a0 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:34:26.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 -- 172.21.15.7:0/3749184696 >> 172.21.15.7:0/3749184696 conn(0x7f18300a94d0 msgr2=0x7f1830071f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:34:26.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 -- 172.21.15.7:0/3749184696 shutdown_connections 2023-03-03T16:34:26.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:34:26.608+0000 7f183816b700 1 -- 172.21.15.7:0/3749184696 wait complete. 2023-03-03T16:34:26.611 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:34:26.621 INFO:tasks.mds_thrash.fs.[a]:waiting for 61 secs before thrashing 2023-03-03T16:34:47.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:34:47.381 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:17.436 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:17.439 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:27.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:35:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.207+0000 7efeb9409700 1 Processor -- start 2023-03-03T16:35:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.207+0000 7efeb9409700 1 -- start start 2023-03-03T16:35:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb9409700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efeb414b020 0x7efeb40717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb9409700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efeb40aad20 0x7efeb4157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb9409700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb41588f0 con 0x7efeb4071d20 2023-03-03T16:35:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb9409700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb40728b0 con 0x7efeb40aad20 2023-03-03T16:35:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb9409700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb4155070 con 0x7efeb414b020 2023-03-03T16:35:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb27fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efeb40aad20 0x7efeb4157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb37fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7efeb4071d20 0x7efeb41581e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51734/0 (socket says 172.21.15.7:51734) 2023-03-03T16:35:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb37fe700 1 -- 172.21.15.7:0/1839316843 learned_addr learned my addr 172.21.15.7:0/1839316843 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:35:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb2ffd700 1 --2- 172.21.15.7:0/1839316843 >> v2:172.21.15.143:3300/0 conn(0x7efeb414b020 0x7efeb40717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb27fc700 1 -- 172.21.15.7:0/1839316843 >> v2:172.21.15.143:3300/0 conn(0x7efeb414b020 msgr2=0x7efeb40717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:35:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb27fc700 1 --2- 172.21.15.7:0/1839316843 >> v2:172.21.15.143:3300/0 conn(0x7efeb414b020 0x7efeb40717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb27fc700 1 -- 172.21.15.7:0/1839316843 >> v1:172.21.15.7:6789/0 conn(0x7efeb4071d20 legacy=0x7efeb41581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb1ffb700 1 -- 172.21.15.7:0/1839316843 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968765044 0 0) 0x7efeb41588f0 con 0x7efeb4071d20 2023-03-03T16:35:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb27fc700 1 -- 172.21.15.7:0/1839316843 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb406f5b0 con 0x7efeb40aad20 2023-03-03T16:35:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb2ffd700 1 --2- 172.21.15.7:0/1839316843 >> v2:172.21.15.143:3300/0 conn(0x7efeb414b020 0x7efeb40717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:35:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.208+0000 7efeb27fc700 1 --2- 172.21.15.7:0/1839316843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 0x7efeb4157ca0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7efeb4070940 tx=0x7efea8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a161ab2da0dad27 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb1ffb700 1 -- 172.21.15.7:0/1839316843 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8011830 con 0x7efeb40aad20 2023-03-03T16:35:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb1ffb700 1 -- 172.21.15.7:0/1839316843 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea8011990 con 0x7efeb40aad20 2023-03-03T16:35:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb1ffb700 1 -- 172.21.15.7:0/1839316843 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8011d20 con 0x7efeb40aad20 2023-03-03T16:35:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb9409700 1 -- 172.21.15.7:0/1839316843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 msgr2=0x7efeb4157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb9409700 1 --2- 172.21.15.7:0/1839316843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 0x7efeb4157ca0 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7efeb4070940 tx=0x7efea8009b20 comp rx=0 tx=0).stop 2023-03-03T16:35:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb9409700 1 -- 172.21.15.7:0/1839316843 shutdown_connections 2023-03-03T16:35:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb9409700 1 --2- 172.21.15.7:0/1839316843 >> v2:172.21.15.143:3300/0 conn(0x7efeb414b020 0x7efeb40717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb9409700 1 --2- 172.21.15.7:0/1839316843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 0x7efeb4157ca0 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb9409700 1 -- 172.21.15.7:0/1839316843 >> 172.21.15.7:0/1839316843 conn(0x7efeb40c8900 msgr2=0x7efeb40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:35:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb9409700 1 -- 172.21.15.7:0/1839316843 shutdown_connections 2023-03-03T16:35:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.209+0000 7efeb9409700 1 -- 172.21.15.7:0/1839316843 wait complete. 2023-03-03T16:35:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.210+0000 7efeb9409700 1 Processor -- start 2023-03-03T16:35:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.210+0000 7efeb9409700 1 -- start start 2023-03-03T16:35:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb9409700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efeb4071d20 0x7efeb41533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb9409700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 0x7efeb4148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb9409700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeb414b020 0x7efeb4149300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb9409700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb4151480 con 0x7efeb40aad20 2023-03-03T16:35:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb9409700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb4149840 con 0x7efeb414b020 2023-03-03T16:35:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb9409700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb41511a0 con 0x7efeb4071d20 2023-03-03T16:35:27.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 0x7efeb4148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb27fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 0x7efeb4148dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57758/0 (socket says 172.21.15.7:57758) 2023-03-03T16:35:27.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb27fc700 1 -- 172.21.15.7:0/2711317737 learned_addr learned my addr 172.21.15.7:0/2711317737 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:35:27.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb2ffd700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efeb4071d20 0x7efeb41533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.211+0000 7efeb37fe700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeb414b020 0x7efeb4149300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efeb27fc700 1 -- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efeb4071d20 msgr2=0x7efeb41533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efeb27fc700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efeb4071d20 0x7efeb41533b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efeb27fc700 1 -- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeb414b020 msgr2=0x7efeb4149300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efeb27fc700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeb414b020 0x7efeb4149300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efeb27fc700 1 -- 172.21.15.7:0/2711317737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea80094a0 con 0x7efeb40aad20 2023-03-03T16:35:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efeb37fe700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeb414b020 0x7efeb4149300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:35:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efeb27fc700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 0x7efeb4148dc0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7efea8000c00 tx=0x7efea8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efe9b7fe700 1 -- 172.21.15.7:0/2711317737 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea80038a0 con 0x7efeb40aad20 2023-03-03T16:35:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efe9b7fe700 1 -- 172.21.15.7:0/2711317737 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea801e070 con 0x7efeb40aad20 2023-03-03T16:35:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efeb9409700 1 -- 172.21.15.7:0/2711317737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb406fb70 con 0x7efeb40aad20 2023-03-03T16:35:27.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efe9b7fe700 1 -- 172.21.15.7:0/2711317737 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8003c40 con 0x7efeb40aad20 2023-03-03T16:35:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.212+0000 7efeb9409700 1 -- 172.21.15.7:0/2711317737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb406f890 con 0x7efeb40aad20 2023-03-03T16:35:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.213+0000 7efeb9409700 1 -- 172.21.15.7:0/2711317737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb4155350 con 0x7efeb40aad20 2023-03-03T16:35:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.214+0000 7efe9b7fe700 1 -- 172.21.15.7:0/2711317737 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efea800f050 con 0x7efeb40aad20 2023-03-03T16:35:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.214+0000 7efe9b7fe700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efea00682f0 0x7efea006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.214+0000 7efe9b7fe700 1 -- 172.21.15.7:0/2711317737 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efea8087280 con 0x7efeb40aad20 2023-03-03T16:35:27.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.214+0000 7efeb2ffd700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efea00682f0 0x7efea006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.215+0000 7efeb2ffd700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efea00682f0 0x7efea006a7a0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7efea4006fd0 tx=0x7efea4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:27.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.218+0000 7efe9b7fe700 1 -- 172.21.15.7:0/2711317737 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efea8053620 con 0x7efeb40aad20 2023-03-03T16:35:27.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.459+0000 7efeb9409700 1 -- 172.21.15.7:0/2711317737 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efeb40a9e10 con 0x7efeb40aad20 2023-03-03T16:35:27.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.459+0000 7efe9b7fe700 1 -- 172.21.15.7:0/2711317737 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efea8020090 con 0x7efeb40aad20 2023-03-03T16:35:27.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:35:27.461 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:35:27.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.461+0000 7efeb9409700 1 -- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efea00682f0 msgr2=0x7efea006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.461+0000 7efeb9409700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efea00682f0 0x7efea006a7a0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7efea4006fd0 tx=0x7efea4008040 comp rx=0 tx=0).stop 2023-03-03T16:35:27.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.461+0000 7efeb9409700 1 -- 172.21.15.7:0/2711317737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 msgr2=0x7efeb4148dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.461+0000 7efeb9409700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 0x7efeb4148dc0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7efea8000c00 tx=0x7efea8004750 comp rx=0 tx=0).stop 2023-03-03T16:35:27.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.462+0000 7efeb9409700 1 -- 172.21.15.7:0/2711317737 shutdown_connections 2023-03-03T16:35:27.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.462+0000 7efeb9409700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efeb414b020 0x7efeb4149300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.462+0000 7efeb9409700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efeb4071d20 0x7efeb41533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.462+0000 7efeb9409700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efea00682f0 0x7efea006a7a0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.462+0000 7efeb9409700 1 --2- 172.21.15.7:0/2711317737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb40aad20 0x7efeb4148dc0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.462+0000 7efeb9409700 1 -- 172.21.15.7:0/2711317737 >> 172.21.15.7:0/2711317737 conn(0x7efeb40c8900 msgr2=0x7efeb4158360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:35:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.462+0000 7efeb9409700 1 -- 172.21.15.7:0/2711317737 shutdown_connections 2023-03-03T16:35:27.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.462+0000 7efeb9409700 1 -- 172.21.15.7:0/2711317737 wait complete. 2023-03-03T16:35:27.465 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:35:27.475 INFO:tasks.mds_thrash.fs.[b]:waiting for 102 secs before thrashing 2023-03-03T16:35:27.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:35:27.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.745+0000 7f3ca3d97700 1 Processor -- start 2023-03-03T16:35:27.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.745+0000 7f3ca3d97700 1 -- start start 2023-03-03T16:35:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.746+0000 7f3ca3d97700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3c9c0aa030 0x7f3c9c158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.746+0000 7f3ca3d97700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9c0a9c20 con 0x7f3c9c14b0d0 2023-03-03T16:35:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.746+0000 7f3ca3d97700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9c154f10 con 0x7f3c9c0aad20 2023-03-03T16:35:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.746+0000 7f3ca3d97700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c9c0ac3c0 con 0x7f3c9c0aa030 2023-03-03T16:35:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.746+0000 7f3ca1b33700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3c9c14b0d0 0x7f3c9c071800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51744/0 (socket says 172.21.15.7:51744) 2023-03-03T16:35:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.746+0000 7f3ca1b33700 1 -- 172.21.15.7:0/4139051337 learned_addr learned my addr 172.21.15.7:0/4139051337 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:35:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.746+0000 7f3ca2334700 1 --2- 172.21.15.7:0/4139051337 >> v2:172.21.15.143:3300/0 conn(0x7f3c9c0aa030 0x7f3c9c158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.746+0000 7f3ca2334700 1 -- 172.21.15.7:0/4139051337 >> v1:172.21.15.143:6789/0 conn(0x7f3c9c0aad20 legacy=0x7f3c9c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.746+0000 7f3ca2334700 1 -- 172.21.15.7:0/4139051337 >> v1:172.21.15.7:6789/0 conn(0x7f3c9c14b0d0 legacy=0x7f3c9c071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.746+0000 7f3ca2334700 1 -- 172.21.15.7:0/4139051337 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c06fc70 con 0x7f3c9c0aa030 2023-03-03T16:35:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.747+0000 7f3ca2334700 1 --2- 172.21.15.7:0/4139051337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c9c0aa030 0x7f3c9c158500 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009d90 tx=0x7f3c9400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2362ddd1d25130e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.751+0000 7f3ca0b31700 1 -- 172.21.15.7:0/4139051337 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400e070 con 0x7f3c9c0aa030 2023-03-03T16:35:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.751+0000 7f3ca0b31700 1 -- 172.21.15.7:0/4139051337 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9400bbd0 con 0x7f3c9c0aa030 2023-03-03T16:35:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.751+0000 7f3ca0b31700 1 -- 172.21.15.7:0/4139051337 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c94012430 con 0x7f3c9c0aa030 2023-03-03T16:35:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.752+0000 7f3ca3d97700 1 -- 172.21.15.7:0/4139051337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c9c0aa030 msgr2=0x7f3c9c158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.752+0000 7f3ca3d97700 1 --2- 172.21.15.7:0/4139051337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c9c0aa030 0x7f3c9c158500 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009d90 tx=0x7f3c9400b330 comp rx=0 tx=0).stop 2023-03-03T16:35:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.752+0000 7f3ca3d97700 1 -- 172.21.15.7:0/4139051337 shutdown_connections 2023-03-03T16:35:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.752+0000 7f3ca3d97700 1 --2- 172.21.15.7:0/4139051337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c9c0aa030 0x7f3c9c158500 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.752+0000 7f3ca3d97700 1 -- 172.21.15.7:0/4139051337 >> 172.21.15.7:0/4139051337 conn(0x7f3c9c0c8900 msgr2=0x7f3c9c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:35:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.752+0000 7f3ca3d97700 1 -- 172.21.15.7:0/4139051337 shutdown_connections 2023-03-03T16:35:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.752+0000 7f3ca3d97700 1 -- 172.21.15.7:0/4139051337 wait complete. 2023-03-03T16:35:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.753+0000 7f3ca3d97700 1 Processor -- start 2023-03-03T16:35:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.753+0000 7f3ca3d97700 1 -- start start 2023-03-03T16:35:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.753+0000 7f3ca3d97700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c0aa030 0x7f3c9c152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca3d97700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c9c0aad20 0x7f3c9c1530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca3d97700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c9c14b0d0 0x7f3c9c0c6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca3d97700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c14a040 con 0x7f3c9c0aa030 2023-03-03T16:35:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca3d97700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c149710 con 0x7f3c9c0aad20 2023-03-03T16:35:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca3d97700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c1499f0 con 0x7f3c9c14b0d0 2023-03-03T16:35:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1b33700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c0aa030 0x7f3c9c152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1b33700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c0aa030 0x7f3c9c152bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57770/0 (socket says 172.21.15.7:57770) 2023-03-03T16:35:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1b33700 1 -- 172.21.15.7:0/1640697930 learned_addr learned my addr 172.21.15.7:0/1640697930 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:35:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca2334700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c9c14b0d0 0x7f3c9c0c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1332700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c9c0aad20 0x7f3c9c1530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1b33700 1 -- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c9c14b0d0 msgr2=0x7f3c9c0c6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1b33700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c9c14b0d0 0x7f3c9c0c6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1b33700 1 -- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c9c0aad20 msgr2=0x7f3c9c1530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1b33700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c9c0aad20 0x7f3c9c1530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1b33700 1 -- 172.21.15.7:0/1640697930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c94009a70 con 0x7f3c9c0aa030 2023-03-03T16:35:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1332700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c9c0aad20 0x7f3c9c1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:35:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca2334700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c9c14b0d0 0x7f3c9c0c6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:35:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca1b33700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c0aa030 0x7f3c9c152bb0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800c520 tx=0x7f3c9800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3c927fc700 1 -- 172.21.15.7:0/1640697930 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98005d70 con 0x7f3c9c0aa030 2023-03-03T16:35:27.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3c927fc700 1 -- 172.21.15.7:0/1640697930 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c980070d0 con 0x7f3c9c0aa030 2023-03-03T16:35:27.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca3d97700 1 -- 172.21.15.7:0/1640697930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9c06f6b0 con 0x7f3c9c0aa030 2023-03-03T16:35:27.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3c927fc700 1 -- 172.21.15.7:0/1640697930 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98007440 con 0x7f3c9c0aa030 2023-03-03T16:35:27.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.754+0000 7f3ca3d97700 1 -- 172.21.15.7:0/1640697930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9c155160 con 0x7f3c9c0aa030 2023-03-03T16:35:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.755+0000 7f3ca3d97700 1 -- 172.21.15.7:0/1640697930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c06f990 con 0x7f3c9c0aa030 2023-03-03T16:35:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.756+0000 7f3c927fc700 1 -- 172.21.15.7:0/1640697930 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3c98019030 con 0x7f3c9c0aa030 2023-03-03T16:35:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.756+0000 7f3c927fc700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c80068320 0x7f3c8006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:35:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.756+0000 7f3c927fc700 1 -- 172.21.15.7:0/1640697930 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3c980853c0 con 0x7f3c9c0aa030 2023-03-03T16:35:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.756+0000 7f3ca1332700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c80068320 0x7f3c8006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:35:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.757+0000 7f3ca1332700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c80068320 0x7f3c8006a7d0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c000f60 tx=0x7f3c8c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:35:27.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.760+0000 7f3c927fc700 1 -- 172.21.15.7:0/1640697930 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3c9804b5e0 con 0x7f3c9c0aa030 2023-03-03T16:35:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.998+0000 7f3ca3d97700 1 -- 172.21.15.7:0/1640697930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c9c074bb0 con 0x7f3c9c0aa030 2023-03-03T16:35:28.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:27.999+0000 7f3c927fc700 1 -- 172.21.15.7:0/1640697930 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3c98051810 con 0x7f3c9c0aa030 2023-03-03T16:35:28.000 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:35:28.001 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:35:28.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 -- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c80068320 msgr2=0x7f3c8006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:28.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c80068320 0x7f3c8006a7d0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c000f60 tx=0x7f3c8c009040 comp rx=0 tx=0).stop 2023-03-03T16:35:28.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 -- 172.21.15.7:0/1640697930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c0aa030 msgr2=0x7f3c9c152bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:35:28.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c0aa030 0x7f3c9c152bb0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800c520 tx=0x7f3c9800c8e0 comp rx=0 tx=0).stop 2023-03-03T16:35:28.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 -- 172.21.15.7:0/1640697930 shutdown_connections 2023-03-03T16:35:28.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3c9c14b0d0 0x7f3c9c0c6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:28.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3c80068320 0x7f3c8006a7d0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:28.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c0aa030 0x7f3c9c152bb0 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:28.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 --2- 172.21.15.7:0/1640697930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3c9c0aad20 0x7f3c9c1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:35:28.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 -- 172.21.15.7:0/1640697930 >> 172.21.15.7:0/1640697930 conn(0x7f3c9c0c8900 msgr2=0x7f3c9c1ddaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:35:28.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 -- 172.21.15.7:0/1640697930 shutdown_connections 2023-03-03T16:35:28.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:35:28.001+0000 7f3ca3d97700 1 -- 172.21.15.7:0/1640697930 wait complete. 2023-03-03T16:35:28.004 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:35:28.014 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T16:35:47.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:35:47.503 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:17.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:17.566 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:32.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:36:32.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.153+0000 7f75a7d95700 1 Processor -- start 2023-03-03T16:36:32.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.153+0000 7f75a7d95700 1 -- start start 2023-03-03T16:36:32.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.154+0000 7f75a7d95700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f75a0151cd0 0x7f75a0154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:36:32.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.154+0000 7f75a7d95700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a0157ff0 con 0x7f75a006c6c0 2023-03-03T16:36:32.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.154+0000 7f75a7d95700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a006d640 con 0x7f75a00c2e30 2023-03-03T16:36:32.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.154+0000 7f75a7d95700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a006cda0 con 0x7f75a0151cd0 2023-03-03T16:36:32.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.154+0000 7f75a6332700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f75a006c6c0 0x7f75a00736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51794/0 (socket says 172.21.15.7:51794) 2023-03-03T16:36:32.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.154+0000 7f75a6332700 1 -- 172.21.15.7:0/4223436673 learned_addr learned my addr 172.21.15.7:0/4223436673 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:36:32.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.154+0000 7f75a5b31700 1 --2- 172.21.15.7:0/4223436673 >> v2:172.21.15.143:3300/0 conn(0x7f75a0151cd0 0x7f75a0154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:36:32.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.154+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651822748 0 0) 0x7f75a0157ff0 con 0x7f75a006c6c0 2023-03-03T16:36:32.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.154+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f758c0035b0 con 0x7f75a006c6c0 2023-03-03T16:36:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.154+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897539970 0 0) 0x7f758c0035b0 con 0x7f75a006c6c0 2023-03-03T16:36:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 >> v2:172.21.15.143:3300/0 conn(0x7f75a0151cd0 msgr2=0x7f75a0154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:36:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 --2- 172.21.15.7:0/4223436673 >> v2:172.21.15.143:3300/0 conn(0x7f75a0151cd0 0x7f75a0154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:36:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 >> v1:172.21.15.143:6789/0 conn(0x7f75a00c2e30 legacy=0x7f75a0158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:36:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a006d080 con 0x7f75a006c6c0 2023-03-03T16:36:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f75980019f0 con 0x7f75a006c6c0 2023-03-03T16:36:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 >> v1:172.21.15.7:6789/0 conn(0x7f75a006c6c0 legacy=0x7f75a00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:36:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f758c003a70 0x7f758c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:36:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f758c004850 0x7f758c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:36:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f758c005320 0x7f758c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:36:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a006cda0 con 0x7f758c003a70 2023-03-03T16:36:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c00c230 con 0x7f758c005320 2023-03-03T16:36:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758c00c3f0 con 0x7f758c004850 2023-03-03T16:36:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a5b31700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f758c003a70 0x7f758c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:36:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a6332700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f758c005320 0x7f758c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:36:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a5330700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f758c004850 0x7f758c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:36:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a5b31700 1 -- 172.21.15.7:0/4223436673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f758c004850 msgr2=0x7f758c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:36:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a5b31700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f758c004850 0x7f758c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:36:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a5b31700 1 -- 172.21.15.7:0/4223436673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f758c005320 msgr2=0x7f758c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:36:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a5b31700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f758c005320 0x7f758c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:36:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a5b31700 1 -- 172.21.15.7:0/4223436673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c0035b0 con 0x7f758c003a70 2023-03-03T16:36:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a6332700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f758c005320 0x7f758c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:36:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a6332700 1 -- 172.21.15.7:0/4223436673 reap_dead start 2023-03-03T16:36:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a5b31700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f758c003a70 0x7f758c003e80 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f7590009b60 tx=0x7f758c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c168a04a726e019 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:36:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759000e6c0 con 0x7f758c003a70 2023-03-03T16:36:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759000e820 con 0x7f758c003a70 2023-03-03T16:36:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.155+0000 7f75a4b2f700 1 -- 172.21.15.7:0/4223436673 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759000eb90 con 0x7f758c003a70 2023-03-03T16:36:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.156+0000 7f75a7d95700 1 -- 172.21.15.7:0/4223436673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f758c003a70 msgr2=0x7f758c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:36:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.156+0000 7f75a7d95700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f758c003a70 0x7f758c003e80 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f7590009b60 tx=0x7f758c005880 comp rx=0 tx=0).stop 2023-03-03T16:36:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.156+0000 7f75a7d95700 1 -- 172.21.15.7:0/4223436673 shutdown_connections 2023-03-03T16:36:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.156+0000 7f75a7d95700 1 --2- 172.21.15.7:0/4223436673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f758c003a70 0x7f758c003e80 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:36:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.156+0000 7f75a7d95700 1 -- 172.21.15.7:0/4223436673 >> 172.21.15.7:0/4223436673 conn(0x7f75a00a94d0 msgr2=0x7f75a00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:36:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.156+0000 7f75a7d95700 1 -- 172.21.15.7:0/4223436673 shutdown_connections 2023-03-03T16:36:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.156+0000 7f75a7d95700 1 -- 172.21.15.7:0/4223436673 wait complete. 2023-03-03T16:36:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.157+0000 7f75a7d95700 1 Processor -- start 2023-03-03T16:36:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.157+0000 7f75a7d95700 1 -- start start 2023-03-03T16:36:32.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a7d95700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f758c003a70 0x7f75a00c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:36:32.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a7d95700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75a014a7f0 0x7f75a006a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:36:32.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a7d95700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75a01577c0 0x7f75a00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:36:32.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a7d95700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a00be240 con 0x7f75a01577c0 2023-03-03T16:36:32.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a7d95700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a014aca0 con 0x7f758c003a70 2023-03-03T16:36:32.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a7d95700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a00c6ef0 con 0x7f75a014a7f0 2023-03-03T16:36:32.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a6332700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75a01577c0 0x7f75a00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:36:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a6332700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75a01577c0 0x7f75a00c51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57826/0 (socket says 172.21.15.7:57826) 2023-03-03T16:36:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a6332700 1 -- 172.21.15.7:0/3877642997 learned_addr learned my addr 172.21.15.7:0/3877642997 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:36:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a5b31700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f758c003a70 0x7f75a00c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:36:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a5330700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75a014a7f0 0x7f75a006a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:36:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a6332700 1 -- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75a014a7f0 msgr2=0x7f75a006a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:36:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a6332700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75a014a7f0 0x7f75a006a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:36:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a6332700 1 -- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f758c003a70 msgr2=0x7f75a00c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:36:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a6332700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f758c003a70 0x7f75a00c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:36:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a6332700 1 -- 172.21.15.7:0/3877642997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a0157c80 con 0x7f75a01577c0 2023-03-03T16:36:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.158+0000 7f75a5330700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75a014a7f0 0x7f75a006a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:36:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.159+0000 7f75a6332700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75a01577c0 0x7f75a00c51f0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f7598000f80 tx=0x7f7598007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:36:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.159+0000 7f75967fc700 1 -- 172.21.15.7:0/3877642997 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759800ec60 con 0x7f75a01577c0 2023-03-03T16:36:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.159+0000 7f75967fc700 1 -- 172.21.15.7:0/3877642997 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759800edc0 con 0x7f75a01577c0 2023-03-03T16:36:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.159+0000 7f75a7d95700 1 -- 172.21.15.7:0/3877642997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a006d360 con 0x7f75a01577c0 2023-03-03T16:36:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.159+0000 7f75967fc700 1 -- 172.21.15.7:0/3877642997 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7598013660 con 0x7f75a01577c0 2023-03-03T16:36:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.159+0000 7f75a7d95700 1 -- 172.21.15.7:0/3877642997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a0158480 con 0x7f75a01577c0 2023-03-03T16:36:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.160+0000 7f75a7d95700 1 -- 172.21.15.7:0/3877642997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a0157490 con 0x7f75a01577c0 2023-03-03T16:36:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.160+0000 7f75967fc700 1 -- 172.21.15.7:0/3877642997 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f75980137c0 con 0x7f75a01577c0 2023-03-03T16:36:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.160+0000 7f75967fc700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75840682f0 0x7f758406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:36:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.161+0000 7f75967fc700 1 -- 172.21.15.7:0/3877642997 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7598086900 con 0x7f75a01577c0 2023-03-03T16:36:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.161+0000 7f75a5b31700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75840682f0 0x7f758406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:36:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.161+0000 7f75a5b31700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75840682f0 0x7f758406a7a0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f75900090b0 tx=0x7f7590008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:36:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.165+0000 7f75967fc700 1 -- 172.21.15.7:0/3877642997 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f75a0157490 con 0x7f75a01577c0 2023-03-03T16:36:32.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.402+0000 7f75a7d95700 1 -- 172.21.15.7:0/3877642997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75a0074bb0 con 0x7f75a01577c0 2023-03-03T16:36:32.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.403+0000 7f75967fc700 1 -- 172.21.15.7:0/3877642997 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7598052ca0 con 0x7f75a01577c0 2023-03-03T16:36:32.404 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:36:32.404 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:36:32.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 -- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75840682f0 msgr2=0x7f758406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:36:32.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75840682f0 0x7f758406a7a0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f75900090b0 tx=0x7f7590008030 comp rx=0 tx=0).stop 2023-03-03T16:36:32.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 -- 172.21.15.7:0/3877642997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75a01577c0 msgr2=0x7f75a00c51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:36:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75a01577c0 0x7f75a00c51f0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f7598000f80 tx=0x7f7598007c90 comp rx=0 tx=0).stop 2023-03-03T16:36:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 -- 172.21.15.7:0/3877642997 shutdown_connections 2023-03-03T16:36:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f75840682f0 0x7f758406a7a0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:36:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f75a01577c0 0x7f75a00c51f0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:36:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f758c003a70 0x7f75a00c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:36:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 --2- 172.21.15.7:0/3877642997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f75a014a7f0 0x7f75a006a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:36:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 -- 172.21.15.7:0/3877642997 >> 172.21.15.7:0/3877642997 conn(0x7f75a00a94d0 msgr2=0x7f75a0071640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:36:32.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 -- 172.21.15.7:0/3877642997 shutdown_connections 2023-03-03T16:36:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:36:32.405+0000 7f75a7d95700 1 -- 172.21.15.7:0/3877642997 wait complete. 2023-03-03T16:36:32.408 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:36:32.418 INFO:tasks.mds_thrash.fs.[a]:waiting for 114 secs before thrashing 2023-03-03T16:36:47.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:36:47.628 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:09.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:37:09.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.626+0000 7f15bb74c700 1 Processor -- start 2023-03-03T16:37:09.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.626+0000 7f15bb74c700 1 -- start start 2023-03-03T16:37:09.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15bb74c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4155390 0x7f15b40c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:09.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15bb74c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f15b4151770 0x7f15b4151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:09.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15bb74c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b4154910 con 0x7f15b406e020 2023-03-03T16:37:09.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15bb74c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b4070b90 con 0x7f15b4151770 2023-03-03T16:37:09.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15bb74c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b4151320 con 0x7f15b4155390 2023-03-03T16:37:09.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15b9ce9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f15b4151770 0x7f15b4151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:09.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15b8ce7700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f15b406e020 0x7f15b4071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51840/0 (socket says 172.21.15.7:51840) 2023-03-03T16:37:09.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15b8ce7700 1 -- 172.21.15.7:0/2916194984 learned_addr learned my addr 172.21.15.7:0/2916194984 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:37:09.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15b94e8700 1 --2- 172.21.15.7:0/2916194984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4155390 0x7f15b40c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:09.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15b9ce9700 1 -- 172.21.15.7:0/2916194984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4155390 msgr2=0x7f15b40c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:37:09.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15b9ce9700 1 --2- 172.21.15.7:0/2916194984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4155390 0x7f15b40c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:09.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.627+0000 7f15b9ce9700 1 -- 172.21.15.7:0/2916194984 >> v1:172.21.15.7:6789/0 conn(0x7f15b406e020 legacy=0x7f15b4071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:09.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15b9ce9700 1 -- 172.21.15.7:0/2916194984 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b40702a0 con 0x7f15b4151770 2023-03-03T16:37:09.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15abfff700 1 -- 172.21.15.7:0/2916194984 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045325614 0 0) 0x7f15b4154910 con 0x7f15b406e020 2023-03-03T16:37:09.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15b94e8700 1 --2- 172.21.15.7:0/2916194984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4155390 0x7f15b40c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:37:09.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15b9ce9700 1 --2- 172.21.15.7:0/2916194984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4151770 0x7f15b4151bf0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009d90 tx=0x7f15ac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eab8b0e25c00566 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:37:09.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15abfff700 1 -- 172.21.15.7:0/2916194984 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac00e070 con 0x7f15b4151770 2023-03-03T16:37:09.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15abfff700 1 -- 172.21.15.7:0/2916194984 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac00bbd0 con 0x7f15b4151770 2023-03-03T16:37:09.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15abfff700 1 -- 172.21.15.7:0/2916194984 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac012430 con 0x7f15b4151770 2023-03-03T16:37:09.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15bb74c700 1 -- 172.21.15.7:0/2916194984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4151770 msgr2=0x7f15b4151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:09.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15bb74c700 1 --2- 172.21.15.7:0/2916194984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4151770 0x7f15b4151bf0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009d90 tx=0x7f15ac00b330 comp rx=0 tx=0).stop 2023-03-03T16:37:09.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15bb74c700 1 -- 172.21.15.7:0/2916194984 shutdown_connections 2023-03-03T16:37:09.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15bb74c700 1 --2- 172.21.15.7:0/2916194984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4151770 0x7f15b4151bf0 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:09.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15bb74c700 1 --2- 172.21.15.7:0/2916194984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4155390 0x7f15b40c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:09.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.628+0000 7f15bb74c700 1 -- 172.21.15.7:0/2916194984 >> 172.21.15.7:0/2916194984 conn(0x7f15b406c110 msgr2=0x7f15b406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:37:09.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.629+0000 7f15bb74c700 1 -- 172.21.15.7:0/2916194984 shutdown_connections 2023-03-03T16:37:09.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.629+0000 7f15bb74c700 1 -- 172.21.15.7:0/2916194984 wait complete. 2023-03-03T16:37:09.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.630+0000 7f15bb74c700 1 Processor -- start 2023-03-03T16:37:09.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.630+0000 7f15bb74c700 1 -- start start 2023-03-03T16:37:09.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15bb74c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b406e020 0x7f15b414de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:09.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15bb74c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4151770 0x7f15b414aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:09.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15bb74c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4155390 0x7f15b414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:09.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15bb74c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b40bfae0 con 0x7f15b4155390 2023-03-03T16:37:09.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15bb74c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b40bea30 con 0x7f15b406e020 2023-03-03T16:37:09.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15bb74c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b414b530 con 0x7f15b4151770 2023-03-03T16:37:09.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15b9ce9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4155390 0x7f15b414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:09.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15b9ce9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4155390 0x7f15b414aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57868/0 (socket says 172.21.15.7:57868) 2023-03-03T16:37:09.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15b9ce9700 1 -- 172.21.15.7:0/2008386493 learned_addr learned my addr 172.21.15.7:0/2008386493 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:37:09.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15b94e8700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b406e020 0x7f15b414de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:09.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.631+0000 7f15b8ce7700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4151770 0x7f15b414aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:09.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15b9ce9700 1 -- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4151770 msgr2=0x7f15b414aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:09.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15b9ce9700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4151770 0x7f15b414aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:09.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15b9ce9700 1 -- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b406e020 msgr2=0x7f15b414de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:09.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15b9ce9700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b406e020 0x7f15b414de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:09.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15b9ce9700 1 -- 172.21.15.7:0/2008386493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac009a70 con 0x7f15b4155390 2023-03-03T16:37:09.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15b9ce9700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4155390 0x7f15b414aff0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f15ac014040 tx=0x7f15ac004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:37:09.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15a9ffb700 1 -- 172.21.15.7:0/2008386493 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac012430 con 0x7f15b4155390 2023-03-03T16:37:09.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15a9ffb700 1 -- 172.21.15.7:0/2008386493 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac020070 con 0x7f15b4155390 2023-03-03T16:37:09.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15bb74c700 1 -- 172.21.15.7:0/2008386493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b4070820 con 0x7f15b4155390 2023-03-03T16:37:09.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15a9ffb700 1 -- 172.21.15.7:0/2008386493 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac004280 con 0x7f15b4155390 2023-03-03T16:37:09.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.632+0000 7f15bb74c700 1 -- 172.21.15.7:0/2008386493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b4070540 con 0x7f15b4155390 2023-03-03T16:37:09.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.633+0000 7f15bb74c700 1 -- 172.21.15.7:0/2008386493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b4155760 con 0x7f15b4155390 2023-03-03T16:37:09.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.634+0000 7f15a9ffb700 1 -- 172.21.15.7:0/2008386493 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f15ac0043e0 con 0x7f15b4155390 2023-03-03T16:37:09.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.634+0000 7f15a9ffb700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15980682f0 0x7f159806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:09.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.634+0000 7f15a9ffb700 1 -- 172.21.15.7:0/2008386493 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f15ac022070 con 0x7f15b4155390 2023-03-03T16:37:09.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.634+0000 7f15b94e8700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15980682f0 0x7f159806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:09.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.635+0000 7f15b94e8700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15980682f0 0x7f159806a7a0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f15a400c940 tx=0x7f15a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:37:09.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.638+0000 7f15a9ffb700 1 -- 172.21.15.7:0/2008386493 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f15ac0547e0 con 0x7f15b4155390 2023-03-03T16:37:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.877+0000 7f15bb74c700 1 -- 172.21.15.7:0/2008386493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15b4074bb0 con 0x7f15b4155390 2023-03-03T16:37:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.878+0000 7f15a9ffb700 1 -- 172.21.15.7:0/2008386493 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f15ac057e00 con 0x7f15b4155390 2023-03-03T16:37:09.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:37:09.878 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:37:09.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 -- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15980682f0 msgr2=0x7f159806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:09.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15980682f0 0x7f159806a7a0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f15a400c940 tx=0x7f15a4008040 comp rx=0 tx=0).stop 2023-03-03T16:37:09.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 -- 172.21.15.7:0/2008386493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4155390 msgr2=0x7f15b414aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:09.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4155390 0x7f15b414aff0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f15ac014040 tx=0x7f15ac004ad0 comp rx=0 tx=0).stop 2023-03-03T16:37:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 -- 172.21.15.7:0/2008386493 shutdown_connections 2023-03-03T16:37:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15980682f0 0x7f159806a7a0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15b4155390 0x7f15b414aff0 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15b406e020 0x7f15b414de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 --2- 172.21.15.7:0/2008386493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15b4151770 0x7f15b414aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 -- 172.21.15.7:0/2008386493 >> 172.21.15.7:0/2008386493 conn(0x7f15b406c110 msgr2=0x7f15b414fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:37:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 -- 172.21.15.7:0/2008386493 shutdown_connections 2023-03-03T16:37:09.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:09.880+0000 7f15bb74c700 1 -- 172.21.15.7:0/2008386493 wait complete. 2023-03-03T16:37:09.882 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:37:09.892 INFO:tasks.mds_thrash.fs.[b]:waiting for 40 secs before thrashing 2023-03-03T16:37:17.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:17.693 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:47.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:47.771 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:37:49.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:37:50.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.031+0000 7fa709a0e700 1 Processor -- start 2023-03-03T16:37:50.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.032+0000 7fa709a0e700 1 -- start start 2023-03-03T16:37:50.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa709a0e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa7040caa30 0x7fa7040c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:50.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa709a0e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7040c7310 con 0x7fa704154ff0 2023-03-03T16:37:50.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa709a0e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7040bfd10 con 0x7fa7040c00e0 2023-03-03T16:37:50.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa709a0e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa704155430 con 0x7fa7040caa30 2023-03-03T16:37:50.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa7037fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa704154ff0 0x7fa70415ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51868/0 (socket says 172.21.15.7:51868) 2023-03-03T16:37:50.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa7037fe700 1 -- 172.21.15.7:0/571740142 learned_addr learned my addr 172.21.15.7:0/571740142 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:37:50.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322639469 0 0) 0x7fa7040c7310 con 0x7fa704154ff0 2023-03-03T16:37:50.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa702ffd700 1 --2- 172.21.15.7:0/571740142 >> v2:172.21.15.143:3300/0 conn(0x7fa7040caa30 0x7fa7040c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:50.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f00035b0 con 0x7fa704154ff0 2023-03-03T16:37:50.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1497456131 0 0) 0x7fa6f00035b0 con 0x7fa704154ff0 2023-03-03T16:37:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 >> v2:172.21.15.143:3300/0 conn(0x7fa7040caa30 msgr2=0x7fa7040c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:37:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa701ffb700 1 --2- 172.21.15.7:0/571740142 >> v2:172.21.15.143:3300/0 conn(0x7fa7040caa30 0x7fa7040c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.033+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 >> v1:172.21.15.143:6789/0 conn(0x7fa7040c00e0 legacy=0x7fa7040c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7040bfa30 con 0x7fa704154ff0 2023-03-03T16:37:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa6fc0019f0 con 0x7fa704154ff0 2023-03-03T16:37:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 >> v1:172.21.15.7:6789/0 conn(0x7fa704154ff0 legacy=0x7fa70415ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f0003a70 0x7fa6f0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f0004850 0x7fa6f0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6f0005250 0x7fa6f000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa704155430 con 0x7fa6f0003a70 2023-03-03T16:37:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f000c190 con 0x7fa6f0004850 2023-03-03T16:37:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f000c350 con 0x7fa6f0005250 2023-03-03T16:37:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa702ffd700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f0003a70 0x7fa6f0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa7027fc700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f0004850 0x7fa6f0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa7037fe700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6f0005250 0x7fa6f000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa702ffd700 1 -- 172.21.15.7:0/571740142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6f0005250 msgr2=0x7fa6f000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:37:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa702ffd700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6f0005250 0x7fa6f000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa702ffd700 1 -- 172.21.15.7:0/571740142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f0004850 msgr2=0x7fa6f0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:37:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa702ffd700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f0004850 0x7fa6f0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa702ffd700 1 -- 172.21.15.7:0/571740142 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f00035b0 con 0x7fa6f0003a70 2023-03-03T16:37:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa7037fe700 1 -- 172.21.15.7:0/571740142 reap_dead start 2023-03-03T16:37:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa702ffd700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f0003a70 0x7fa6f0003e80 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4009b30 tx=0x7fa6f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e9d6f28c979331 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:37:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f400e6c0 con 0x7fa6f0003a70 2023-03-03T16:37:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f400e820 con 0x7fa6f0003a70 2023-03-03T16:37:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.034+0000 7fa701ffb700 1 -- 172.21.15.7:0/571740142 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f400eb90 con 0x7fa6f0003a70 2023-03-03T16:37:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.035+0000 7fa709a0e700 1 -- 172.21.15.7:0/571740142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f0003a70 msgr2=0x7fa6f0003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.035+0000 7fa709a0e700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f0003a70 0x7fa6f0003e80 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4009b30 tx=0x7fa6f0005880 comp rx=0 tx=0).stop 2023-03-03T16:37:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.035+0000 7fa709a0e700 1 -- 172.21.15.7:0/571740142 shutdown_connections 2023-03-03T16:37:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.035+0000 7fa709a0e700 1 --2- 172.21.15.7:0/571740142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6f0003a70 0x7fa6f0003e80 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.035+0000 7fa709a0e700 1 -- 172.21.15.7:0/571740142 >> 172.21.15.7:0/571740142 conn(0x7fa704066e40 msgr2=0x7fa704071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:37:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.035+0000 7fa709a0e700 1 -- 172.21.15.7:0/571740142 shutdown_connections 2023-03-03T16:37:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.035+0000 7fa709a0e700 1 -- 172.21.15.7:0/571740142 wait complete. 2023-03-03T16:37:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.036+0000 7fa709a0e700 1 Processor -- start 2023-03-03T16:37:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.036+0000 7fa709a0e700 1 -- start start 2023-03-03T16:37:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa709a0e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f0003a70 0x7fa70414c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa709a0e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7040c42b0 0x7fa70413dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa709a0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa70413e2a0 0x7fa7040c2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa709a0e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7041ddde0 con 0x7fa70413e2a0 2023-03-03T16:37:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa709a0e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa704157f10 con 0x7fa6f0003a70 2023-03-03T16:37:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa709a0e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70414cb70 con 0x7fa7040c42b0 2023-03-03T16:37:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa7037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa70413e2a0 0x7fa7040c2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa7037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa70413e2a0 0x7fa7040c2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57900/0 (socket says 172.21.15.7:57900) 2023-03-03T16:37:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa7037fe700 1 -- 172.21.15.7:0/2973996123 learned_addr learned my addr 172.21.15.7:0/2973996123 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:37:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa7027fc700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7040c42b0 0x7fa70413dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa702ffd700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f0003a70 0x7fa70414c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa7037fe700 1 -- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7040c42b0 msgr2=0x7fa70413dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa7037fe700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7040c42b0 0x7fa70413dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa7037fe700 1 -- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f0003a70 msgr2=0x7fa70414c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa7037fe700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f0003a70 0x7fa70414c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa7037fe700 1 -- 172.21.15.7:0/2973996123 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7041547f0 con 0x7fa70413e2a0 2023-03-03T16:37:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.037+0000 7fa702ffd700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f0003a70 0x7fa70414c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:37:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.038+0000 7fa7037fe700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa70413e2a0 0x7fa7040c2b40 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7fa7040cb580 tx=0x7fa6fc00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:37:50.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.038+0000 7fa6fbfff700 1 -- 172.21.15.7:0/2973996123 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc017070 con 0x7fa70413e2a0 2023-03-03T16:37:50.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.038+0000 7fa6fbfff700 1 -- 172.21.15.7:0/2973996123 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6fc00ec70 con 0x7fa70413e2a0 2023-03-03T16:37:50.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.038+0000 7fa709a0e700 1 -- 172.21.15.7:0/2973996123 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa704154510 con 0x7fa70413e2a0 2023-03-03T16:37:50.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.038+0000 7fa6fbfff700 1 -- 172.21.15.7:0/2973996123 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc00ee00 con 0x7fa70413e2a0 2023-03-03T16:37:50.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.038+0000 7fa709a0e700 1 -- 172.21.15.7:0/2973996123 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa704154ad0 con 0x7fa70413e2a0 2023-03-03T16:37:50.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.039+0000 7fa709a0e700 1 -- 172.21.15.7:0/2973996123 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7040c3850 con 0x7fa70413e2a0 2023-03-03T16:37:50.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.039+0000 7fa6fbfff700 1 -- 172.21.15.7:0/2973996123 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa6fc013420 con 0x7fa70413e2a0 2023-03-03T16:37:50.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.040+0000 7fa6fbfff700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 0x7fa6e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:37:50.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.040+0000 7fa6fbfff700 1 -- 172.21.15.7:0/2973996123 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa6fc085f10 con 0x7fa70413e2a0 2023-03-03T16:37:50.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.040+0000 7fa702ffd700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 0x7fa6e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:37:50.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.040+0000 7fa702ffd700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 0x7fa6e406a7a0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40090b0 tx=0x7fa6f401e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:37:50.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.044+0000 7fa6fbfff700 1 -- 172.21.15.7:0/2973996123 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa7040c3850 con 0x7fa70413e2a0 2023-03-03T16:37:50.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.283+0000 7fa709a0e700 1 -- 172.21.15.7:0/2973996123 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7040c3850 con 0x7fa70413e2a0 2023-03-03T16:37:50.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.283+0000 7fa6fbfff700 1 -- 172.21.15.7:0/2973996123 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa7040c3850 con 0x7fa70413e2a0 2023-03-03T16:37:50.283 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:37:50.284 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:37:50.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.285+0000 7fa709a0e700 1 -- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 msgr2=0x7fa6e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:50.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.285+0000 7fa709a0e700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 0x7fa6e406a7a0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40090b0 tx=0x7fa6f401e040 comp rx=0 tx=0).stop 2023-03-03T16:37:50.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.285+0000 7fa709a0e700 1 -- 172.21.15.7:0/2973996123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa70413e2a0 msgr2=0x7fa7040c2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:37:50.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.285+0000 7fa709a0e700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa70413e2a0 0x7fa7040c2b40 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7fa7040cb580 tx=0x7fa6fc00e440 comp rx=0 tx=0).stop 2023-03-03T16:37:50.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.285+0000 7fa709a0e700 1 -- 172.21.15.7:0/2973996123 shutdown_connections 2023-03-03T16:37:50.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.285+0000 7fa709a0e700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 0x7fa6e406a7a0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:50.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.285+0000 7fa709a0e700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa70413e2a0 0x7fa7040c2b40 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.285+0000 7fa709a0e700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6f0003a70 0x7fa70414c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.285+0000 7fa709a0e700 1 --2- 172.21.15.7:0/2973996123 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7040c42b0 0x7fa70413dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:37:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.285+0000 7fa709a0e700 1 -- 172.21.15.7:0/2973996123 >> 172.21.15.7:0/2973996123 conn(0x7fa704066e40 msgr2=0x7fa7040c77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:37:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.286+0000 7fa709a0e700 1 -- 172.21.15.7:0/2973996123 shutdown_connections 2023-03-03T16:37:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:37:50.286+0000 7fa709a0e700 1 -- 172.21.15.7:0/2973996123 wait complete. 2023-03-03T16:37:50.287 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:37:50.298 INFO:tasks.mds_thrash.fs.[b]:waiting for 95 secs before thrashing 2023-03-03T16:38:17.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:17.835 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:26.419 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:38:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.557+0000 7f53037d5700 1 Processor -- start 2023-03-03T16:38:26.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.557+0000 7f53037d5700 1 -- start start 2023-03-03T16:38:26.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f53037d5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f52fc071e80 0x7f52fc151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f53037d5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52fc1510d0 con 0x7f52fc14e900 2023-03-03T16:38:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f53037d5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52fc0c3740 con 0x7f52fc072250 2023-03-03T16:38:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f53037d5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52fc070670 con 0x7f52fc071e80 2023-03-03T16:38:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f5301d72700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f52fc14e900 0x7f52fc15ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51922/0 (socket says 172.21.15.7:51922) 2023-03-03T16:38:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f5301d72700 1 -- 172.21.15.7:0/3604347398 learned_addr learned my addr 172.21.15.7:0/3604347398 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:38:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f5301571700 1 --2- 172.21.15.7:0/3604347398 >> v2:172.21.15.7:3300/0 conn(0x7f52fc071e80 0x7f52fc151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:26.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f5301571700 1 -- 172.21.15.7:0/3604347398 >> v1:172.21.15.143:6789/0 conn(0x7f52fc072250 legacy=0x7f52fc14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:38:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f5301571700 1 -- 172.21.15.7:0/3604347398 >> v1:172.21.15.7:6789/0 conn(0x7f52fc14e900 legacy=0x7f52fc15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f5301571700 1 -- 172.21.15.7:0/3604347398 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc0c39e0 con 0x7f52fc071e80 2023-03-03T16:38:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f52f3fff700 1 -- 172.21.15.7:0/3604347398 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262671111 0 0) 0x7f52fc1510d0 con 0x7f52fc14e900 2023-03-03T16:38:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.558+0000 7f5301571700 1 --2- 172.21.15.7:0/3604347398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc071e80 0x7f52fc151820 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009840 tx=0x7f52f8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3ce8c4febb7af5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.559+0000 7f52f3fff700 1 -- 172.21.15.7:0/3604347398 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f8004230 con 0x7f52fc071e80 2023-03-03T16:38:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.559+0000 7f52f3fff700 1 -- 172.21.15.7:0/3604347398 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f8004390 con 0x7f52fc071e80 2023-03-03T16:38:26.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.559+0000 7f52f3fff700 1 -- 172.21.15.7:0/3604347398 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f80036a0 con 0x7f52fc071e80 2023-03-03T16:38:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.559+0000 7f53037d5700 1 -- 172.21.15.7:0/3604347398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc071e80 msgr2=0x7f52fc151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.559+0000 7f53037d5700 1 --2- 172.21.15.7:0/3604347398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc071e80 0x7f52fc151820 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009840 tx=0x7f52f8009b50 comp rx=0 tx=0).stop 2023-03-03T16:38:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.559+0000 7f53037d5700 1 -- 172.21.15.7:0/3604347398 shutdown_connections 2023-03-03T16:38:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.559+0000 7f53037d5700 1 --2- 172.21.15.7:0/3604347398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc071e80 0x7f52fc151820 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:26.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.559+0000 7f53037d5700 1 -- 172.21.15.7:0/3604347398 >> 172.21.15.7:0/3604347398 conn(0x7f52fc0a94d0 msgr2=0x7f52fc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:38:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.559+0000 7f53037d5700 1 -- 172.21.15.7:0/3604347398 shutdown_connections 2023-03-03T16:38:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.559+0000 7f53037d5700 1 -- 172.21.15.7:0/3604347398 wait complete. 2023-03-03T16:38:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.560+0000 7f53037d5700 1 Processor -- start 2023-03-03T16:38:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.560+0000 7f53037d5700 1 -- start start 2023-03-03T16:38:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.561+0000 7f53037d5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52fc071e80 0x7f52fc1dd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.561+0000 7f53037d5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc072250 0x7f52fc1ea460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.561+0000 7f53037d5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc14e900 0x7f52fc1ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.561+0000 7f53037d5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc145940 con 0x7f52fc072250 2023-03-03T16:38:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.561+0000 7f53037d5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc144d30 con 0x7f52fc14e900 2023-03-03T16:38:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.561+0000 7f53037d5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc145010 con 0x7f52fc071e80 2023-03-03T16:38:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.561+0000 7f5300d70700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc072250 0x7f52fc1ea460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.561+0000 7f5300d70700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc072250 0x7f52fc1ea460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57946/0 (socket says 172.21.15.7:57946) 2023-03-03T16:38:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.561+0000 7f5300d70700 1 -- 172.21.15.7:0/1983796187 learned_addr learned my addr 172.21.15.7:0/1983796187 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:38:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f5301571700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52fc071e80 0x7f52fc1dd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f5301d72700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc14e900 0x7f52fc1ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f5300d70700 1 -- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52fc071e80 msgr2=0x7f52fc1dd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f5300d70700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52fc071e80 0x7f52fc1dd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f5300d70700 1 -- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc14e900 msgr2=0x7f52fc1ea9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f5300d70700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc14e900 0x7f52fc1ea9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f5300d70700 1 -- 172.21.15.7:0/1983796187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f80094a0 con 0x7f52fc072250 2023-03-03T16:38:26.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f5301d72700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc14e900 0x7f52fc1ea9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:38:26.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f5300d70700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc072250 0x7f52fc1ea460 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00c4a0 tx=0x7f52ec00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:26.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f52f1ffb700 1 -- 172.21.15.7:0/1983796187 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec011790 con 0x7f52fc072250 2023-03-03T16:38:26.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f52f1ffb700 1 -- 172.21.15.7:0/1983796187 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec0118f0 con 0x7f52fc072250 2023-03-03T16:38:26.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f53037d5700 1 -- 172.21.15.7:0/1983796187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52fc06c9a0 con 0x7f52fc072250 2023-03-03T16:38:26.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f52f1ffb700 1 -- 172.21.15.7:0/1983796187 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec011ca0 con 0x7f52fc072250 2023-03-03T16:38:26.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.562+0000 7f53037d5700 1 -- 172.21.15.7:0/1983796187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52fc151300 con 0x7f52fc072250 2023-03-03T16:38:26.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.563+0000 7f53037d5700 1 -- 172.21.15.7:0/1983796187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52fc06cb30 con 0x7f52fc072250 2023-03-03T16:38:26.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.564+0000 7f52f1ffb700 1 -- 172.21.15.7:0/1983796187 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f52ec010050 con 0x7f52fc072250 2023-03-03T16:38:26.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.564+0000 7f52f1ffb700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52e00682f0 0x7f52e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.564+0000 7f52f1ffb700 1 -- 172.21.15.7:0/1983796187 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f52ec085e50 con 0x7f52fc072250 2023-03-03T16:38:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.564+0000 7f5301571700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52e00682f0 0x7f52e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.565+0000 7f5301571700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52e00682f0 0x7f52e006a7a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f52f8000c00 tx=0x7f52f801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:26.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.568+0000 7f52f1ffb700 1 -- 172.21.15.7:0/1983796187 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f52ec0521f0 con 0x7f52fc072250 2023-03-03T16:38:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.806+0000 7f53037d5700 1 -- 172.21.15.7:0/1983796187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52fc074bb0 con 0x7f52fc072250 2023-03-03T16:38:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.807+0000 7f52f1ffb700 1 -- 172.21.15.7:0/1983796187 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f52ec055810 con 0x7f52fc072250 2023-03-03T16:38:26.807 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:38:26.807 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:38:26.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 -- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52e00682f0 msgr2=0x7f52e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:26.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52e00682f0 0x7f52e006a7a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f52f8000c00 tx=0x7f52f801c040 comp rx=0 tx=0).stop 2023-03-03T16:38:26.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 -- 172.21.15.7:0/1983796187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc072250 msgr2=0x7f52fc1ea460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:26.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc072250 0x7f52fc1ea460 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f52ec00c4a0 tx=0x7f52ec00c860 comp rx=0 tx=0).stop 2023-03-03T16:38:26.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 -- 172.21.15.7:0/1983796187 shutdown_connections 2023-03-03T16:38:26.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52fc14e900 0x7f52fc1ea9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:26.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52fc071e80 0x7f52fc1dd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:26.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52e00682f0 0x7f52e006a7a0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:26.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 --2- 172.21.15.7:0/1983796187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52fc072250 0x7f52fc1ea460 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:26.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 -- 172.21.15.7:0/1983796187 >> 172.21.15.7:0/1983796187 conn(0x7f52fc0a94d0 msgr2=0x7f52fc0c2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:38:26.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 -- 172.21.15.7:0/1983796187 shutdown_connections 2023-03-03T16:38:26.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:26.809+0000 7f53037d5700 1 -- 172.21.15.7:0/1983796187 wait complete. 2023-03-03T16:38:26.811 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:38:26.820 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before thrashing 2023-03-03T16:38:31.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:38:31.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.960+0000 7fe8d59a3700 1 Processor -- start 2023-03-03T16:38:31.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.960+0000 7fe8d59a3700 1 -- start start 2023-03-03T16:38:31.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8d59a3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe8d00b00b0 0x7fe8d00c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:31.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8d59a3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00c6cf0 con 0x7fe8d0151cc0 2023-03-03T16:38:31.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8d59a3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d01548f0 con 0x7fe8d00c7590 2023-03-03T16:38:31.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8d59a3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d0152090 con 0x7fe8d00b00b0 2023-03-03T16:38:31.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8ceffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe8d0151cc0 0x7fe8d00ca6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51940/0 (socket says 172.21.15.7:51940) 2023-03-03T16:38:31.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8ceffd700 1 -- 172.21.15.7:0/574466449 learned_addr learned my addr 172.21.15.7:0/574466449 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:38:31.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811704900 0 0) 0x7fe8d00c6cf0 con 0x7fe8d0151cc0 2023-03-03T16:38:31.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8ce7fc700 1 --2- 172.21.15.7:0/574466449 >> v2:172.21.15.143:3300/0 conn(0x7fe8d00b00b0 0x7fe8d00c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:31.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b80035b0 con 0x7fe8d0151cc0 2023-03-03T16:38:31.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2416760860 0 0) 0x7fe8b80035b0 con 0x7fe8d0151cc0 2023-03-03T16:38:31.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 >> v2:172.21.15.143:3300/0 conn(0x7fe8d00b00b0 msgr2=0x7fe8d00c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:38:31.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 --2- 172.21.15.7:0/574466449 >> v2:172.21.15.143:3300/0 conn(0x7fe8d00b00b0 0x7fe8d00c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:31.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 >> v1:172.21.15.143:6789/0 conn(0x7fe8d00c7590 legacy=0x7fe8d015ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:31.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0154bd0 con 0x7fe8d0151cc0 2023-03-03T16:38:31.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe8c0001670 con 0x7fe8d0151cc0 2023-03-03T16:38:31.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 >> v1:172.21.15.7:6789/0 conn(0x7fe8d0151cc0 legacy=0x7fe8d00ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:31.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8b8003a70 0x7fe8b8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:31.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8b8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:31.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8b8005320 0x7fe8b800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:31.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b800c200 con 0x7fe8b8004850 2023-03-03T16:38:31.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b800c370 con 0x7fe8b8003a70 2023-03-03T16:38:31.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.961+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b800c530 con 0x7fe8b8005320 2023-03-03T16:38:31.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8ce7fc700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8b8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:31.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8ceffd700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8b8003a70 0x7fe8b8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:31.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8cf7fe700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8b8005320 0x7fe8b800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:31.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8ce7fc700 1 -- 172.21.15.7:0/574466449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8b8005320 msgr2=0x7fe8b800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:38:31.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8ce7fc700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8b8005320 0x7fe8b800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:31.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8ce7fc700 1 -- 172.21.15.7:0/574466449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8b8003a70 msgr2=0x7fe8b8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:38:31.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8ce7fc700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8b8003a70 0x7fe8b8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:31.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8ce7fc700 1 -- 172.21.15.7:0/574466449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b80035b0 con 0x7fe8b8004850 2023-03-03T16:38:31.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8cf7fe700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8b8005320 0x7fe8b800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:38:31.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8cf7fe700 1 -- 172.21.15.7:0/574466449 reap_dead start 2023-03-03T16:38:31.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8ce7fc700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8b8004cc0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009b30 tx=0x7fe8b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4c827788e18fd5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:31.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c400e6c0 con 0x7fe8b8004850 2023-03-03T16:38:31.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c400e820 con 0x7fe8b8004850 2023-03-03T16:38:31.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.962+0000 7fe8cdffb700 1 -- 172.21.15.7:0/574466449 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c400eb90 con 0x7fe8b8004850 2023-03-03T16:38:31.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.963+0000 7fe8d59a3700 1 -- 172.21.15.7:0/574466449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 msgr2=0x7fe8b8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:31.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.963+0000 7fe8d59a3700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8b8004cc0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009b30 tx=0x7fe8b8005880 comp rx=0 tx=0).stop 2023-03-03T16:38:31.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.963+0000 7fe8d59a3700 1 -- 172.21.15.7:0/574466449 shutdown_connections 2023-03-03T16:38:31.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.963+0000 7fe8d59a3700 1 --2- 172.21.15.7:0/574466449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8b8004cc0 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:31.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.963+0000 7fe8d59a3700 1 -- 172.21.15.7:0/574466449 >> 172.21.15.7:0/574466449 conn(0x7fe8d006b980 msgr2=0x7fe8d0071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:38:31.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.963+0000 7fe8d59a3700 1 -- 172.21.15.7:0/574466449 shutdown_connections 2023-03-03T16:38:31.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.963+0000 7fe8d59a3700 1 -- 172.21.15.7:0/574466449 wait complete. 2023-03-03T16:38:31.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.964+0000 7fe8d59a3700 1 Processor -- start 2023-03-03T16:38:31.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.965+0000 7fe8d59a3700 1 -- start start 2023-03-03T16:38:31.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.965+0000 7fe8d59a3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8d0146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:31.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.965+0000 7fe8d59a3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d01532d0 0x7fe8d0153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:31.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8d59a3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d01ed540 0x7fe8d01ed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:31.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8d59a3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0150860 con 0x7fe8b8004850 2023-03-03T16:38:31.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8d59a3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0150cf0 con 0x7fe8d01532d0 2023-03-03T16:38:31.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8d59a3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0153d20 con 0x7fe8d01ed540 2023-03-03T16:38:31.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ceffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8d0146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:31.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ceffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8d0146450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57974/0 (socket says 172.21.15.7:57974) 2023-03-03T16:38:31.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ceffd700 1 -- 172.21.15.7:0/3668776353 learned_addr learned my addr 172.21.15.7:0/3668776353 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:38:31.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8cf7fe700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d01ed540 0x7fe8d01ed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:31.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ce7fc700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d01532d0 0x7fe8d0153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:31.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ceffd700 1 -- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d01ed540 msgr2=0x7fe8d01ed9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:31.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ceffd700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d01ed540 0x7fe8d01ed9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:31.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ceffd700 1 -- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d01532d0 msgr2=0x7fe8d0153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:31.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ceffd700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d01532d0 0x7fe8d0153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:31.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ceffd700 1 -- 172.21.15.7:0/3668776353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0155190 con 0x7fe8b8004850 2023-03-03T16:38:31.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8cf7fe700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d01ed540 0x7fe8d01ed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:38:31.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ce7fc700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d01532d0 0x7fe8d0153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:38:31.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8ceffd700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8d0146450 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00072c0 tx=0x7fe8c0007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:31.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8b7fff700 1 -- 172.21.15.7:0/3668776353 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c00077a0 con 0x7fe8b8004850 2023-03-03T16:38:31.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8b7fff700 1 -- 172.21.15.7:0/3668776353 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c000f8d0 con 0x7fe8b8004850 2023-03-03T16:38:31.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8b7fff700 1 -- 172.21.15.7:0/3668776353 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c000fc80 con 0x7fe8b8004850 2023-03-03T16:38:31.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8d59a3700 1 -- 172.21.15.7:0/3668776353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d0154eb0 con 0x7fe8b8004850 2023-03-03T16:38:31.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.966+0000 7fe8d59a3700 1 -- 172.21.15.7:0/3668776353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d01545a0 con 0x7fe8b8004850 2023-03-03T16:38:31.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.967+0000 7fe8d59a3700 1 -- 172.21.15.7:0/3668776353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d0150b40 con 0x7fe8b8004850 2023-03-03T16:38:31.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.968+0000 7fe8b7fff700 1 -- 172.21.15.7:0/3668776353 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe8c001d460 con 0x7fe8b8004850 2023-03-03T16:38:31.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.968+0000 7fe8b7fff700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8bc0682f0 0x7fe8bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:38:31.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.968+0000 7fe8b7fff700 1 -- 172.21.15.7:0/3668776353 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe8c00862b0 con 0x7fe8b8004850 2023-03-03T16:38:31.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.968+0000 7fe8ce7fc700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8bc0682f0 0x7fe8bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:38:31.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.969+0000 7fe8ce7fc700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8bc0682f0 0x7fe8bc06a7a0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fe8d01451f0 tx=0x7fe8c4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:38:31.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:31.973+0000 7fe8b7fff700 1 -- 172.21.15.7:0/3668776353 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8c00525f0 con 0x7fe8b8004850 2023-03-03T16:38:32.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.213+0000 7fe8d59a3700 1 -- 172.21.15.7:0/3668776353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8d0074bb0 con 0x7fe8b8004850 2023-03-03T16:38:32.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.214+0000 7fe8b7fff700 1 -- 172.21.15.7:0/3668776353 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe8c0055c10 con 0x7fe8b8004850 2023-03-03T16:38:32.214 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:38:32.214 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:38:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 -- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8bc0682f0 msgr2=0x7fe8bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8bc0682f0 0x7fe8bc06a7a0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fe8d01451f0 tx=0x7fe8c4008030 comp rx=0 tx=0).stop 2023-03-03T16:38:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 -- 172.21.15.7:0/3668776353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 msgr2=0x7fe8d0146450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:38:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8d0146450 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00072c0 tx=0x7fe8c0007680 comp rx=0 tx=0).stop 2023-03-03T16:38:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 -- 172.21.15.7:0/3668776353 shutdown_connections 2023-03-03T16:38:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d01ed540 0x7fe8d01ed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8bc0682f0 0x7fe8bc06a7a0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8b8004850 0x7fe8d0146450 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 --2- 172.21.15.7:0/3668776353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d01532d0 0x7fe8d0153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:38:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 -- 172.21.15.7:0/3668776353 >> 172.21.15.7:0/3668776353 conn(0x7fe8d006b980 msgr2=0x7fe8d00c6370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:38:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 -- 172.21.15.7:0/3668776353 shutdown_connections 2023-03-03T16:38:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:38:32.216+0000 7fe8d59a3700 1 -- 172.21.15.7:0/3668776353 wait complete. 2023-03-03T16:38:32.218 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:38:32.228 INFO:tasks.mds_thrash.fs.[a]:waiting for 39 secs before thrashing 2023-03-03T16:38:47.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:38:47.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:11.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:39:11.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.369+0000 7f40e3d0c700 1 Processor -- start 2023-03-03T16:39:11.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.369+0000 7f40e3d0c700 1 -- start start 2023-03-03T16:39:11.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e3d0c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f40dc14fb00 0x7f40dc156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e3d0c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40dc155840 con 0x7f40dc0740f0 2023-03-03T16:39:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e3d0c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40dc1567d0 con 0x7f40dc0c1da0 2023-03-03T16:39:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e3d0c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40dc072940 con 0x7f40dc14fb00 2023-03-03T16:39:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e22a9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f40dc0740f0 0x7f40dc15cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51974/0 (socket says 172.21.15.7:51974) 2023-03-03T16:39:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e22a9700 1 -- 172.21.15.7:0/3689529017 learned_addr learned my addr 172.21.15.7:0/3689529017 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:39:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e1aa8700 1 --2- 172.21.15.7:0/3689529017 >> v2:172.21.15.143:3300/0 conn(0x7f40dc14fb00 0x7f40dc156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:11.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724753093 0 0) 0x7f40dc155840 con 0x7f40dc0740f0 2023-03-03T16:39:11.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c80035b0 con 0x7f40dc0740f0 2023-03-03T16:39:11.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 919261460 0 0) 0x7f40c80035b0 con 0x7f40dc0740f0 2023-03-03T16:39:11.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 >> v2:172.21.15.143:3300/0 conn(0x7f40dc14fb00 msgr2=0x7f40dc156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:39:11.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 --2- 172.21.15.7:0/3689529017 >> v2:172.21.15.143:3300/0 conn(0x7f40dc14fb00 0x7f40dc156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:11.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 >> v1:172.21.15.143:6789/0 conn(0x7f40dc0c1da0 legacy=0x7f40dc0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:11.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc0c2330 con 0x7f40dc0740f0 2023-03-03T16:39:11.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f40d40019f0 con 0x7f40dc0740f0 2023-03-03T16:39:11.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 >> v1:172.21.15.7:6789/0 conn(0x7f40dc0740f0 legacy=0x7f40dc15cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:11.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40c8003a70 0x7f40c8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:11.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40c8004850 0x7f40c8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:11.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40c8005200 0x7f40c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc072940 con 0x7f40c8005200 2023-03-03T16:39:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c800c110 con 0x7f40c8004850 2023-03-03T16:39:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.370+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c800c2d0 con 0x7f40c8003a70 2023-03-03T16:39:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e1aa8700 1 --2- 172.21.15.7:0/3689529017 >> v2:172.21.15.143:3300/0 conn(0x7f40dc14fb00 0x7f40dc156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:39:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e22a9700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40c8005200 0x7f40c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e12a7700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40c8004850 0x7f40c8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e1aa8700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40c8003a70 0x7f40c8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e22a9700 1 -- 172.21.15.7:0/3689529017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40c8003a70 msgr2=0x7f40c8003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:39:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e22a9700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40c8003a70 0x7f40c8003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e22a9700 1 -- 172.21.15.7:0/3689529017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40c8004850 msgr2=0x7f40c8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:39:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e22a9700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40c8004850 0x7f40c8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e22a9700 1 -- 172.21.15.7:0/3689529017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c80035b0 con 0x7f40c8005200 2023-03-03T16:39:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e22a9700 1 -- 172.21.15.7:0/3689529017 reap_dead start 2023-03-03T16:39:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e22a9700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40c8005200 0x7f40c800ba10 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f40d4002230 tx=0x7f40c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee3eaaa4efaec60 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d4012600 con 0x7f40c8005200 2023-03-03T16:39:11.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d4012760 con 0x7f40c8005200 2023-03-03T16:39:11.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.371+0000 7f40e0aa6700 1 -- 172.21.15.7:0/3689529017 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d4012b10 con 0x7f40c8005200 2023-03-03T16:39:11.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.372+0000 7f40e3d0c700 1 -- 172.21.15.7:0/3689529017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40c8005200 msgr2=0x7f40c800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:11.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.372+0000 7f40e3d0c700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40c8005200 0x7f40c800ba10 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f40d4002230 tx=0x7f40c8005880 comp rx=0 tx=0).stop 2023-03-03T16:39:11.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.372+0000 7f40e3d0c700 1 -- 172.21.15.7:0/3689529017 shutdown_connections 2023-03-03T16:39:11.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.372+0000 7f40e3d0c700 1 --2- 172.21.15.7:0/3689529017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40c8005200 0x7f40c800ba10 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:11.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.372+0000 7f40e3d0c700 1 -- 172.21.15.7:0/3689529017 >> 172.21.15.7:0/3689529017 conn(0x7f40dc06c110 msgr2=0x7f40dc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:39:11.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.372+0000 7f40e3d0c700 1 -- 172.21.15.7:0/3689529017 shutdown_connections 2023-03-03T16:39:11.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.372+0000 7f40e3d0c700 1 -- 172.21.15.7:0/3689529017 wait complete. 2023-03-03T16:39:11.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.373+0000 7f40e3d0c700 1 Processor -- start 2023-03-03T16:39:11.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.373+0000 7f40e3d0c700 1 -- start start 2023-03-03T16:39:11.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e3d0c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40c8005200 0x7f40dc0c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:11.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e3d0c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40dc1e81e0 0x7f40dc147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:11.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e3d0c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40dc151f40 0x7f40dc144a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:11.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e3d0c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc146d10 con 0x7f40dc151f40 2023-03-03T16:39:11.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e3d0c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc0c7680 con 0x7f40c8005200 2023-03-03T16:39:11.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e3d0c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc1e8670 con 0x7f40dc1e81e0 2023-03-03T16:39:11.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e22a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40dc151f40 0x7f40dc144a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e22a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40dc151f40 0x7f40dc144a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58006/0 (socket says 172.21.15.7:58006) 2023-03-03T16:39:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e22a9700 1 -- 172.21.15.7:0/797560962 learned_addr learned my addr 172.21.15.7:0/797560962 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:39:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e22a9700 1 -- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40dc1e81e0 msgr2=0x7f40dc147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e1aa8700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40c8005200 0x7f40dc0c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e22a9700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40dc1e81e0 0x7f40dc147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e22a9700 1 -- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40c8005200 msgr2=0x7f40dc0c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e22a9700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40c8005200 0x7f40dc0c7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e22a9700 1 -- 172.21.15.7:0/797560962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc153090 con 0x7f40dc151f40 2023-03-03T16:39:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e22a9700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40dc151f40 0x7f40dc144a40 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0ad710 tx=0x7f40d4001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40ce7fc700 1 -- 172.21.15.7:0/797560962 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d401e630 con 0x7f40dc151f40 2023-03-03T16:39:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40ce7fc700 1 -- 172.21.15.7:0/797560962 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d401e7b0 con 0x7f40dc151f40 2023-03-03T16:39:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40e3d0c700 1 -- 172.21.15.7:0/797560962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc152db0 con 0x7f40dc151f40 2023-03-03T16:39:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.374+0000 7f40ce7fc700 1 -- 172.21.15.7:0/797560962 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d401eb60 con 0x7f40dc151f40 2023-03-03T16:39:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.375+0000 7f40e3d0c700 1 -- 172.21.15.7:0/797560962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc155a90 con 0x7f40dc151f40 2023-03-03T16:39:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.375+0000 7f40e3d0c700 1 -- 172.21.15.7:0/797560962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc06e410 con 0x7f40dc151f40 2023-03-03T16:39:11.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.376+0000 7f40ce7fc700 1 -- 172.21.15.7:0/797560962 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f40d4010070 con 0x7f40dc151f40 2023-03-03T16:39:11.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.376+0000 7f40ce7fc700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c00682f0 0x7f40c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:11.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.376+0000 7f40ce7fc700 1 -- 172.21.15.7:0/797560962 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f40d40881d0 con 0x7f40dc151f40 2023-03-03T16:39:11.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.376+0000 7f40e1aa8700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c00682f0 0x7f40c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:11.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.377+0000 7f40e1aa8700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c00682f0 0x7f40c006a7a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f40d8004530 tx=0x7f40d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:11.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.381+0000 7f40ce7fc700 1 -- 172.21.15.7:0/797560962 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f40d40544f0 con 0x7f40dc151f40 2023-03-03T16:39:11.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.618+0000 7f40e3d0c700 1 -- 172.21.15.7:0/797560962 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40dc06e410 con 0x7f40dc151f40 2023-03-03T16:39:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.619+0000 7f40ce7fc700 1 -- 172.21.15.7:0/797560962 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f40d4057b10 con 0x7f40dc151f40 2023-03-03T16:39:11.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:39:11.619 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:39:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 -- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c00682f0 msgr2=0x7f40c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c00682f0 0x7f40c006a7a0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f40d8004530 tx=0x7f40d8008040 comp rx=0 tx=0).stop 2023-03-03T16:39:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 -- 172.21.15.7:0/797560962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40dc151f40 msgr2=0x7f40dc144a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40dc151f40 0x7f40dc144a40 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0ad710 tx=0x7f40d4001d80 comp rx=0 tx=0).stop 2023-03-03T16:39:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 -- 172.21.15.7:0/797560962 shutdown_connections 2023-03-03T16:39:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40c00682f0 0x7f40c006a7a0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40dc151f40 0x7f40dc144a40 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40c8005200 0x7f40dc0c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 --2- 172.21.15.7:0/797560962 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40dc1e81e0 0x7f40dc147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 -- 172.21.15.7:0/797560962 >> 172.21.15.7:0/797560962 conn(0x7f40dc06c110 msgr2=0x7f40dc0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:39:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 -- 172.21.15.7:0/797560962 shutdown_connections 2023-03-03T16:39:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:11.621+0000 7f40e3d0c700 1 -- 172.21.15.7:0/797560962 wait complete. 2023-03-03T16:39:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:39:11.634 INFO:tasks.mds_thrash.fs.[a]:waiting for 11 secs before thrashing 2023-03-03T16:39:17.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:17.962 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:22.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:39:22.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.773+0000 7fc01ee66700 1 Processor -- start 2023-03-03T16:39:22.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.773+0000 7fc01ee66700 1 -- start start 2023-03-03T16:39:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.774+0000 7fc01ee66700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc0180aad20 0x7fc018157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.774+0000 7fc01ee66700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0180a9c20 con 0x7fc01814b0d0 2023-03-03T16:39:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.774+0000 7fc01ee66700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc018154f10 con 0x7fc0180aa030 2023-03-03T16:39:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.774+0000 7fc01ee66700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0180ac3c0 con 0x7fc0180aad20 2023-03-03T16:39:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.774+0000 7fc017fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc0180aad20 0x7fc018157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.774+0000 7fc017fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc0180aad20 0x7fc018157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58020/0 (socket says 172.21.15.7:58020) 2023-03-03T16:39:22.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.774+0000 7fc017fff700 1 -- 172.21.15.7:0/3851568122 learned_addr learned my addr 172.21.15.7:0/3851568122 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:39:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.774+0000 7fc017fff700 1 -- 172.21.15.7:0/3851568122 >> v1:172.21.15.143:6789/0 conn(0x7fc0180aa030 legacy=0x7fc018158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:39:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.774+0000 7fc017fff700 1 -- 172.21.15.7:0/3851568122 >> v1:172.21.15.7:6789/0 conn(0x7fc01814b0d0 legacy=0x7fc018071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.774+0000 7fc017fff700 1 -- 172.21.15.7:0/3851568122 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01806fc70 con 0x7fc0180aad20 2023-03-03T16:39:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.775+0000 7fc017fff700 1 --2- 172.21.15.7:0/3851568122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 0x7fc018157df0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7fc0040098f0 tx=0x7fc004009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a244d7fa2284a10 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.775+0000 7fc0177fe700 1 -- 172.21.15.7:0/3851568122 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004004230 con 0x7fc0180aad20 2023-03-03T16:39:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.775+0000 7fc0177fe700 1 -- 172.21.15.7:0/3851568122 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc004004390 con 0x7fc0180aad20 2023-03-03T16:39:22.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.775+0000 7fc0177fe700 1 -- 172.21.15.7:0/3851568122 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0040036a0 con 0x7fc0180aad20 2023-03-03T16:39:22.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.775+0000 7fc01ee66700 1 -- 172.21.15.7:0/3851568122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 msgr2=0x7fc018157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.775+0000 7fc01ee66700 1 --2- 172.21.15.7:0/3851568122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 0x7fc018157df0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7fc0040098f0 tx=0x7fc004009c00 comp rx=0 tx=0).stop 2023-03-03T16:39:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.775+0000 7fc01ee66700 1 -- 172.21.15.7:0/3851568122 shutdown_connections 2023-03-03T16:39:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.776+0000 7fc01ee66700 1 --2- 172.21.15.7:0/3851568122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 0x7fc018157df0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.776+0000 7fc01ee66700 1 -- 172.21.15.7:0/3851568122 >> 172.21.15.7:0/3851568122 conn(0x7fc0180c8900 msgr2=0x7fc0180c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:39:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.776+0000 7fc01ee66700 1 -- 172.21.15.7:0/3851568122 shutdown_connections 2023-03-03T16:39:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.776+0000 7fc01ee66700 1 -- 172.21.15.7:0/3851568122 wait complete. 2023-03-03T16:39:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.777+0000 7fc01ee66700 1 Processor -- start 2023-03-03T16:39:22.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.777+0000 7fc01ee66700 1 -- start start 2023-03-03T16:39:22.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc01ee66700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180aa030 0x7fc018143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:22.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc01ee66700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 0x7fc0181ed330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:22.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc01ee66700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc01814b0d0 0x7fc0180c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:22.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc01ee66700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0181e0180 con 0x7fc0180aad20 2023-03-03T16:39:22.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc01ee66700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180c75f0 con 0x7fc0180aa030 2023-03-03T16:39:22.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc01ee66700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0181e03d0 con 0x7fc01814b0d0 2023-03-03T16:39:22.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc017fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 0x7fc0181ed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc017fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 0x7fc0181ed330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58026/0 (socket says 172.21.15.7:58026) 2023-03-03T16:39:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc017fff700 1 -- 172.21.15.7:0/1530436818 learned_addr learned my addr 172.21.15.7:0/1530436818 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:39:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc01cc02700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180aa030 0x7fc018143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc017fff700 1 -- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc01814b0d0 msgr2=0x7fc0180c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc017fff700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc01814b0d0 0x7fc0180c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc017fff700 1 -- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180aa030 msgr2=0x7fc018143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc017fff700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180aa030 0x7fc018143bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc017fff700 1 -- 172.21.15.7:0/1530436818 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc004009550 con 0x7fc0180aad20 2023-03-03T16:39:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc01cc02700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180aa030 0x7fc018143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:39:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc017fff700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 0x7fc0181ed330 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7fc004009130 tx=0x7fc004004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.778+0000 7fc0157fa700 1 -- 172.21.15.7:0/1530436818 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004003d70 con 0x7fc0180aad20 2023-03-03T16:39:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.779+0000 7fc0157fa700 1 -- 172.21.15.7:0/1530436818 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00401f070 con 0x7fc0180aad20 2023-03-03T16:39:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.779+0000 7fc0157fa700 1 -- 172.21.15.7:0/1530436818 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00400b8a0 con 0x7fc0180aad20 2023-03-03T16:39:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.779+0000 7fc01ee66700 1 -- 172.21.15.7:0/1530436818 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01806f6b0 con 0x7fc0180aad20 2023-03-03T16:39:22.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.779+0000 7fc01ee66700 1 -- 172.21.15.7:0/1530436818 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc018155160 con 0x7fc0180aad20 2023-03-03T16:39:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.780+0000 7fc01ee66700 1 -- 172.21.15.7:0/1530436818 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01806f990 con 0x7fc0180aad20 2023-03-03T16:39:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.780+0000 7fc0157fa700 1 -- 172.21.15.7:0/1530436818 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc00402a020 con 0x7fc0180aad20 2023-03-03T16:39:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.781+0000 7fc0157fa700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbffc0682f0 0x7fbffc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.781+0000 7fc0157fa700 1 -- 172.21.15.7:0/1530436818 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc0040878d0 con 0x7fc0180aad20 2023-03-03T16:39:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.781+0000 7fc01cc02700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbffc0682f0 0x7fbffc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.782+0000 7fc01cc02700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbffc0682f0 0x7fbffc06a7a0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fc018151a60 tx=0x7fc00c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:22.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:22.785+0000 7fc0157fa700 1 -- 172.21.15.7:0/1530436818 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc004053c70 con 0x7fc0180aad20 2023-03-03T16:39:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.026+0000 7fc01ee66700 1 -- 172.21.15.7:0/1530436818 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc01806f990 con 0x7fc0180aad20 2023-03-03T16:39:23.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.027+0000 7fc0157fa700 1 -- 172.21.15.7:0/1530436818 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc004057290 con 0x7fc0180aad20 2023-03-03T16:39:23.027 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:39:23.027 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:39:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 -- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbffc0682f0 msgr2=0x7fbffc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbffc0682f0 0x7fbffc06a7a0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fc018151a60 tx=0x7fc00c009040 comp rx=0 tx=0).stop 2023-03-03T16:39:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 -- 172.21.15.7:0/1530436818 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 msgr2=0x7fc0181ed330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 0x7fc0181ed330 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7fc004009130 tx=0x7fc004004750 comp rx=0 tx=0).stop 2023-03-03T16:39:23.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 -- 172.21.15.7:0/1530436818 shutdown_connections 2023-03-03T16:39:23.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc01814b0d0 0x7fc0180c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:23.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc0180aa030 0x7fc018143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:23.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbffc0682f0 0x7fbffc06a7a0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:23.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 --2- 172.21.15.7:0/1530436818 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0180aad20 0x7fc0181ed330 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:23.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 -- 172.21.15.7:0/1530436818 >> 172.21.15.7:0/1530436818 conn(0x7fc0180c8900 msgr2=0x7fc0180acbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:39:23.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 -- 172.21.15.7:0/1530436818 shutdown_connections 2023-03-03T16:39:23.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:23.029+0000 7fc01ee66700 1 -- 172.21.15.7:0/1530436818 wait complete. 2023-03-03T16:39:23.031 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:39:23.041 INFO:tasks.mds_thrash.fs.[a]:waiting for 86 secs before thrashing 2023-03-03T16:39:25.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:39:25.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.437+0000 7f34ae859700 1 Processor -- start 2023-03-03T16:39:25.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.437+0000 7f34ae859700 1 -- start start 2023-03-03T16:39:25.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.438+0000 7f34ae859700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f34a8155410 0x7f34a80bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:25.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.438+0000 7f34ae859700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a80c08d0 con 0x7f34a80c5b00 2023-03-03T16:39:25.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.438+0000 7f34ae859700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a8154930 con 0x7f34a8151e80 2023-03-03T16:39:25.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.438+0000 7f34ae859700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a8152250 con 0x7f34a8155410 2023-03-03T16:39:25.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.438+0000 7f34acdf6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f34a80c5b00 0x7f34a80bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52020/0 (socket says 172.21.15.7:52020) 2023-03-03T16:39:25.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.438+0000 7f34acdf6700 1 -- 172.21.15.7:0/1165498258 learned_addr learned my addr 172.21.15.7:0/1165498258 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:39:25.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a77fe700 1 --2- 172.21.15.7:0/1165498258 >> v2:172.21.15.143:3300/0 conn(0x7f34a8155410 0x7f34a80bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:25.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541445220 0 0) 0x7f34a80c08d0 con 0x7f34a80c5b00 2023-03-03T16:39:25.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34900035b0 con 0x7f34a80c5b00 2023-03-03T16:39:25.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3156531222 0 0) 0x7f34900035b0 con 0x7f34a80c5b00 2023-03-03T16:39:25.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 >> v2:172.21.15.143:3300/0 conn(0x7f34a8155410 msgr2=0x7f34a80bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:39:25.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 --2- 172.21.15.7:0/1165498258 >> v2:172.21.15.143:3300/0 conn(0x7f34a8155410 0x7f34a80bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:25.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 >> v1:172.21.15.143:6789/0 conn(0x7f34a8151e80 legacy=0x7f34a80c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:25.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a8154c10 con 0x7f34a80c5b00 2023-03-03T16:39:25.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f34a00019f0 con 0x7f34a80c5b00 2023-03-03T16:39:25.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 >> v1:172.21.15.7:6789/0 conn(0x7f34a80c5b00 legacy=0x7f34a80bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:25.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3490003a70 0x7f3490003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:25.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3490004850 0x7f3490004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:25.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3490005200 0x7f349000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:25.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349000c0e0 con 0x7f3490003a70 2023-03-03T16:39:25.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349000c220 con 0x7f3490005200 2023-03-03T16:39:25.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349000c3e0 con 0x7f3490004850 2023-03-03T16:39:25.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a7fff700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3490003a70 0x7f3490003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:25.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34acdf6700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3490005200 0x7f349000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:25.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a77fe700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3490004850 0x7f3490004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:25.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a7fff700 1 -- 172.21.15.7:0/1165498258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3490004850 msgr2=0x7f3490004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:39:25.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.439+0000 7f34a7fff700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3490004850 0x7f3490004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:25.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34a7fff700 1 -- 172.21.15.7:0/1165498258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3490005200 msgr2=0x7f349000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:39:25.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34a7fff700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3490005200 0x7f349000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:25.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34a7fff700 1 -- 172.21.15.7:0/1165498258 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34900035b0 con 0x7f3490003a70 2023-03-03T16:39:25.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34acdf6700 1 -- 172.21.15.7:0/1165498258 reap_dead start 2023-03-03T16:39:25.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34a7fff700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3490003a70 0x7f3490003e80 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f34980017c0 tx=0x7f3490005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31ab74171824257 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:25.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980052d0 con 0x7f3490003a70 2023-03-03T16:39:25.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3498005430 con 0x7f3490003a70 2023-03-03T16:39:25.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34a6ffd700 1 -- 172.21.15.7:0/1165498258 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980038c0 con 0x7f3490003a70 2023-03-03T16:39:25.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34ae859700 1 -- 172.21.15.7:0/1165498258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3490003a70 msgr2=0x7f3490003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:25.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34ae859700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3490003a70 0x7f3490003e80 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f34980017c0 tx=0x7f3490005880 comp rx=0 tx=0).stop 2023-03-03T16:39:25.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34ae859700 1 -- 172.21.15.7:0/1165498258 shutdown_connections 2023-03-03T16:39:25.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34ae859700 1 --2- 172.21.15.7:0/1165498258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3490003a70 0x7f3490003e80 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:25.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.440+0000 7f34ae859700 1 -- 172.21.15.7:0/1165498258 >> 172.21.15.7:0/1165498258 conn(0x7f34a8071630 msgr2=0x7f34a806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:39:25.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.441+0000 7f34ae859700 1 -- 172.21.15.7:0/1165498258 shutdown_connections 2023-03-03T16:39:25.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.441+0000 7f34ae859700 1 -- 172.21.15.7:0/1165498258 wait complete. 2023-03-03T16:39:25.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.442+0000 7f34ae859700 1 Processor -- start 2023-03-03T16:39:25.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.442+0000 7f34ae859700 1 -- start start 2023-03-03T16:39:25.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34ae859700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3490003a70 0x7f34a80c33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:25.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34ae859700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34a8146460 0x7f34a8145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:25.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34ae859700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a8145c50 0x7f34a81ea0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:25.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34ae859700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a8146b70 con 0x7f34a8145c50 2023-03-03T16:39:25.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34ae859700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a8157400 con 0x7f3490003a70 2023-03-03T16:39:25.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34ae859700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a81576e0 con 0x7f34a8146460 2023-03-03T16:39:25.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34acdf6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a8145c50 0x7f34a81ea0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:25.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34acdf6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a8145c50 0x7f34a81ea0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58052/0 (socket says 172.21.15.7:58052) 2023-03-03T16:39:25.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34acdf6700 1 -- 172.21.15.7:0/3193359730 learned_addr learned my addr 172.21.15.7:0/3193359730 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:39:25.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34a77fe700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34a8146460 0x7f34a8145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:25.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34a7fff700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3490003a70 0x7f34a80c33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:25.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34acdf6700 1 -- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34a8146460 msgr2=0x7f34a8145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:25.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34acdf6700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34a8146460 0x7f34a8145710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:25.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34acdf6700 1 -- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3490003a70 msgr2=0x7f34a80c33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:25.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34acdf6700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3490003a70 0x7f34a80c33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:25.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34acdf6700 1 -- 172.21.15.7:0/3193359730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a80c5220 con 0x7f34a8145c50 2023-03-03T16:39:25.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.443+0000 7f34acdf6700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a8145c50 0x7f34a81ea0f0 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f34a0002230 tx=0x7f34a000e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:25.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.445+0000 7f34a4ff9700 1 -- 172.21.15.7:0/3193359730 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a000ebc0 con 0x7f34a8145c50 2023-03-03T16:39:25.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.445+0000 7f34a4ff9700 1 -- 172.21.15.7:0/3193359730 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a000ed20 con 0x7f34a8145c50 2023-03-03T16:39:25.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.445+0000 7f34a4ff9700 1 -- 172.21.15.7:0/3193359730 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0013660 con 0x7f34a8145c50 2023-03-03T16:39:25.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.445+0000 7f34ae859700 1 -- 172.21.15.7:0/3193359730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a8154ef0 con 0x7f34a8145c50 2023-03-03T16:39:25.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.445+0000 7f34ae859700 1 -- 172.21.15.7:0/3193359730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a80c6aa0 con 0x7f34a8145c50 2023-03-03T16:39:25.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.446+0000 7f34ae859700 1 -- 172.21.15.7:0/3193359730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a8074bb0 con 0x7f34a8145c50 2023-03-03T16:39:25.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.447+0000 7f34a4ff9700 1 -- 172.21.15.7:0/3193359730 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f34a00137c0 con 0x7f34a8145c50 2023-03-03T16:39:25.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.447+0000 7f34a4ff9700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34940682f0 0x7f349406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:25.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.447+0000 7f34a4ff9700 1 -- 172.21.15.7:0/3193359730 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f34a00867c0 con 0x7f34a8145c50 2023-03-03T16:39:25.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.447+0000 7f34a7fff700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34940682f0 0x7f349406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:25.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.448+0000 7f34a7fff700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34940682f0 0x7f349406a7a0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f3498015040 tx=0x7f3498009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:25.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.452+0000 7f34a4ff9700 1 -- 172.21.15.7:0/3193359730 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f34a0052b60 con 0x7f34a8145c50 2023-03-03T16:39:25.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.690+0000 7f34ae859700 1 -- 172.21.15.7:0/3193359730 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34a80c6860 con 0x7f34a8145c50 2023-03-03T16:39:25.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.690+0000 7f34a4ff9700 1 -- 172.21.15.7:0/3193359730 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f34a0056180 con 0x7f34a8145c50 2023-03-03T16:39:25.690 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:39:25.690 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:39:25.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 -- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34940682f0 msgr2=0x7f349406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:25.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34940682f0 0x7f349406a7a0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f3498015040 tx=0x7f3498009030 comp rx=0 tx=0).stop 2023-03-03T16:39:25.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 -- 172.21.15.7:0/3193359730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a8145c50 msgr2=0x7f34a81ea0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:25.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a8145c50 0x7f34a81ea0f0 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f34a0002230 tx=0x7f34a000e440 comp rx=0 tx=0).stop 2023-03-03T16:39:25.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 -- 172.21.15.7:0/3193359730 shutdown_connections 2023-03-03T16:39:25.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34940682f0 0x7f349406a7a0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:25.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34a8145c50 0x7f34a81ea0f0 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:25.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3490003a70 0x7f34a80c33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:25.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 --2- 172.21.15.7:0/3193359730 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34a8146460 0x7f34a8145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:25.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 -- 172.21.15.7:0/3193359730 >> 172.21.15.7:0/3193359730 conn(0x7f34a8071630 msgr2=0x7f34a814fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:39:25.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 -- 172.21.15.7:0/3193359730 shutdown_connections 2023-03-03T16:39:25.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:25.692+0000 7f34ae859700 1 -- 172.21.15.7:0/3193359730 wait complete. 2023-03-03T16:39:25.694 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:39:25.704 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2023-03-03T16:39:48.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:48.026 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:39:59.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:39:59.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.843+0000 7f5d6318a700 1 Processor -- start 2023-03-03T16:39:59.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.844+0000 7f5d6318a700 1 -- start start 2023-03-03T16:39:59.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.844+0000 7f5d6318a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5d5c14fda0 0x7f5d5c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:59.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.844+0000 7f5d6318a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c0c2f30 con 0x7f5d5c14f9d0 2023-03-03T16:39:59.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.844+0000 7f5d6318a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c0c31d0 con 0x7f5d5c0abe20 2023-03-03T16:39:59.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.844+0000 7f5d6318a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5c1524a0 con 0x7f5d5c14fda0 2023-03-03T16:39:59.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.844+0000 7f5d60f26700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5d5c14f9d0 0x7f5d5c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52062/0 (socket says 172.21.15.7:52062) 2023-03-03T16:39:59.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.844+0000 7f5d60f26700 1 -- 172.21.15.7:0/3745848931 learned_addr learned my addr 172.21.15.7:0/3745848931 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:39:59.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.844+0000 7f5d5bfff700 1 --2- 172.21.15.7:0/3745848931 >> v2:172.21.15.7:3300/0 conn(0x7f5d5c14fda0 0x7f5d5c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:59.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.845+0000 7f5d5b7fe700 1 -- 172.21.15.7:0/3745848931 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715698144 0 0) 0x7f5d5c0c2f30 con 0x7f5d5c14f9d0 2023-03-03T16:39:59.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.845+0000 7f5d5b7fe700 1 -- 172.21.15.7:0/3745848931 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d4c0035b0 con 0x7f5d5c14f9d0 2023-03-03T16:39:59.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.845+0000 7f5d5bfff700 1 -- 172.21.15.7:0/3745848931 >> v1:172.21.15.143:6789/0 conn(0x7f5d5c0abe20 legacy=0x7f5d5c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.845+0000 7f5d5bfff700 1 -- 172.21.15.7:0/3745848931 >> v1:172.21.15.7:6789/0 conn(0x7f5d5c14f9d0 legacy=0x7f5d5c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.845+0000 7f5d5bfff700 1 -- 172.21.15.7:0/3745848931 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c0c34b0 con 0x7f5d5c14fda0 2023-03-03T16:39:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.845+0000 7f5d5bfff700 1 --2- 172.21.15.7:0/3745848931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c14fda0 0x7f5d5c156430 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009840 tx=0x7f5d48009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77cc76074181b7cc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.845+0000 7f5d5b7fe700 1 -- 172.21.15.7:0/3745848931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d48011830 con 0x7f5d5c14fda0 2023-03-03T16:39:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.845+0000 7f5d5b7fe700 1 -- 172.21.15.7:0/3745848931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d48011990 con 0x7f5d5c14fda0 2023-03-03T16:39:59.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.845+0000 7f5d5b7fe700 1 -- 172.21.15.7:0/3745848931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d48011d20 con 0x7f5d5c14fda0 2023-03-03T16:39:59.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.846+0000 7f5d6318a700 1 -- 172.21.15.7:0/3745848931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c14fda0 msgr2=0x7f5d5c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:59.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.846+0000 7f5d6318a700 1 --2- 172.21.15.7:0/3745848931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c14fda0 0x7f5d5c156430 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009840 tx=0x7f5d48009b50 comp rx=0 tx=0).stop 2023-03-03T16:39:59.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.846+0000 7f5d6318a700 1 -- 172.21.15.7:0/3745848931 shutdown_connections 2023-03-03T16:39:59.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.846+0000 7f5d6318a700 1 --2- 172.21.15.7:0/3745848931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c14fda0 0x7f5d5c156430 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:59.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.846+0000 7f5d6318a700 1 -- 172.21.15.7:0/3745848931 >> 172.21.15.7:0/3745848931 conn(0x7f5d5c0c8900 msgr2=0x7f5d5c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:39:59.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.846+0000 7f5d6318a700 1 -- 172.21.15.7:0/3745848931 shutdown_connections 2023-03-03T16:39:59.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.846+0000 7f5d6318a700 1 -- 172.21.15.7:0/3745848931 wait complete. 2023-03-03T16:39:59.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.847+0000 7f5d6318a700 1 Processor -- start 2023-03-03T16:39:59.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.847+0000 7f5d6318a700 1 -- start start 2023-03-03T16:39:59.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d6318a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c0abe20 0x7f5d5c1dcca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:59.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d6318a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d5c14f9d0 0x7f5d5c06c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:59.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d6318a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d5c14fda0 0x7f5d5c1ea910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:59.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d6318a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1ddd20 con 0x7f5d5c0abe20 2023-03-03T16:39:59.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d6318a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1dc220 con 0x7f5d5c14fda0 2023-03-03T16:39:59.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d6318a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1dda40 con 0x7f5d5c14f9d0 2023-03-03T16:39:59.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d60f26700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c0abe20 0x7f5d5c1dcca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:59.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d60f26700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c0abe20 0x7f5d5c1dcca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58088/0 (socket says 172.21.15.7:58088) 2023-03-03T16:39:59.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d60f26700 1 -- 172.21.15.7:0/242533657 learned_addr learned my addr 172.21.15.7:0/242533657 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:39:59.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d5bfff700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d5c14f9d0 0x7f5d5c06c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:59.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d61727700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d5c14fda0 0x7f5d5c1ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:59.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d60f26700 1 -- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d5c14f9d0 msgr2=0x7f5d5c06c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:59.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d60f26700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d5c14f9d0 0x7f5d5c06c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:59.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d60f26700 1 -- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d5c14fda0 msgr2=0x7f5d5c1ea910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:39:59.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d60f26700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d5c14fda0 0x7f5d5c1ea910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:39:59.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d60f26700 1 -- 172.21.15.7:0/242533657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d480094a0 con 0x7f5d5c0abe20 2023-03-03T16:39:59.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d60f26700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c0abe20 0x7f5d5c1dcca0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000c700 tx=0x7f5d5000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:59.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.848+0000 7f5d597fa700 1 -- 172.21.15.7:0/242533657 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d50005d70 con 0x7f5d5c0abe20 2023-03-03T16:39:59.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.849+0000 7f5d597fa700 1 -- 172.21.15.7:0/242533657 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d500070d0 con 0x7f5d5c0abe20 2023-03-03T16:39:59.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.849+0000 7f5d597fa700 1 -- 172.21.15.7:0/242533657 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d50007460 con 0x7f5d5c0abe20 2023-03-03T16:39:59.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.849+0000 7f5d6318a700 1 -- 172.21.15.7:0/242533657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5c0ac350 con 0x7f5d5c0abe20 2023-03-03T16:39:59.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.849+0000 7f5d6318a700 1 -- 172.21.15.7:0/242533657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5c0ab630 con 0x7f5d5c0abe20 2023-03-03T16:39:59.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.850+0000 7f5d6318a700 1 -- 172.21.15.7:0/242533657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c071fa0 con 0x7f5d5c0abe20 2023-03-03T16:39:59.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.850+0000 7f5d597fa700 1 -- 172.21.15.7:0/242533657 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5d50019030 con 0x7f5d5c0abe20 2023-03-03T16:39:59.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.850+0000 7f5d597fa700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d400682f0 0x7f5d4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:39:59.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.851+0000 7f5d597fa700 1 -- 172.21.15.7:0/242533657 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5d50085550 con 0x7f5d5c0abe20 2023-03-03T16:39:59.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.851+0000 7f5d5bfff700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d400682f0 0x7f5d4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:39:59.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.851+0000 7f5d5bfff700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d400682f0 0x7f5d4006a7a0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c1501d0 tx=0x7f5d48008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:39:59.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:39:59.855+0000 7f5d597fa700 1 -- 172.21.15.7:0/242533657 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5d500518f0 con 0x7f5d5c0abe20 2023-03-03T16:40:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T16:39:59.998+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T16:40:00.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.091+0000 7f5d6318a700 1 -- 172.21.15.7:0/242533657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d5c074bb0 con 0x7f5d5c0abe20 2023-03-03T16:40:00.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.092+0000 7f5d597fa700 1 -- 172.21.15.7:0/242533657 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5d50054f10 con 0x7f5d5c0abe20 2023-03-03T16:40:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:40:00.093 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:40:00.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.093+0000 7f5d6318a700 1 -- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d400682f0 msgr2=0x7f5d4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:00.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.093+0000 7f5d6318a700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d400682f0 0x7f5d4006a7a0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c1501d0 tx=0x7f5d48008030 comp rx=0 tx=0).stop 2023-03-03T16:40:00.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.094+0000 7f5d6318a700 1 -- 172.21.15.7:0/242533657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c0abe20 msgr2=0x7f5d5c1dcca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:00.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.094+0000 7f5d6318a700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c0abe20 0x7f5d5c1dcca0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000c700 tx=0x7f5d5000ca10 comp rx=0 tx=0).stop 2023-03-03T16:40:00.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.094+0000 7f5d6318a700 1 -- 172.21.15.7:0/242533657 shutdown_connections 2023-03-03T16:40:00.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.094+0000 7f5d6318a700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d5c14fda0 0x7f5d5c1ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:00.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.094+0000 7f5d6318a700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d400682f0 0x7f5d4006a7a0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:00.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.094+0000 7f5d6318a700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d5c0abe20 0x7f5d5c1dcca0 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:00.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.094+0000 7f5d6318a700 1 --2- 172.21.15.7:0/242533657 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d5c14f9d0 0x7f5d5c06c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:00.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.094+0000 7f5d6318a700 1 -- 172.21.15.7:0/242533657 >> 172.21.15.7:0/242533657 conn(0x7f5d5c0c8900 msgr2=0x7f5d5c151140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:00.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.094+0000 7f5d6318a700 1 -- 172.21.15.7:0/242533657 shutdown_connections 2023-03-03T16:40:00.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:00.094+0000 7f5d6318a700 1 -- 172.21.15.7:0/242533657 wait complete. 2023-03-03T16:40:00.096 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:40:00.107 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2023-03-03T16:40:18.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:18.089 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:27.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:40:27.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.247+0000 7f2cfc701700 1 Processor -- start 2023-03-03T16:40:27.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.247+0000 7f2cfc701700 1 -- start start 2023-03-03T16:40:27.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.248+0000 7f2cfc701700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2cf414e560 0x7f2cf4071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:27.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.248+0000 7f2cfc701700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c4850 0x7f2cf4154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:27.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.248+0000 7f2cfc701700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf4072920 con 0x7f2cf40c43d0 2023-03-03T16:40:27.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.248+0000 7f2cfc701700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf406bef0 con 0x7f2cf414e560 2023-03-03T16:40:27.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.248+0000 7f2cfc701700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf40c4cb0 con 0x7f2cf40c4850 2023-03-03T16:40:27.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cfa49d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2cf414e560 0x7f2cf4071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:27.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cfac9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c4850 0x7f2cf4154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:27.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cf9c9c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f2cf40c43d0 0x7f2cf4154590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:55612/0 (socket says 172.21.15.7:55612) 2023-03-03T16:40:27.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cfac9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c4850 0x7f2cf4154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:37956/0 (socket says 172.21.15.7:37956) 2023-03-03T16:40:27.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cfa49d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2cf414e560 0x7f2cf4071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:53886/0 (socket says 172.21.15.7:53886) 2023-03-03T16:40:27.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cf9c9c700 1 -- 172.21.15.7:0/3305392872 learned_addr learned my addr 172.21.15.7:0/3305392872 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:40:27.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cfac9e700 1 -- 172.21.15.7:0/3305392872 >> v2:172.21.15.143:3300/0 conn(0x7f2cf414e560 msgr2=0x7f2cf4071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:40:27.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cfac9e700 1 --2- 172.21.15.7:0/3305392872 >> v2:172.21.15.143:3300/0 conn(0x7f2cf414e560 0x7f2cf4071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:27.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cfac9e700 1 -- 172.21.15.7:0/3305392872 >> v1:172.21.15.143:6789/0 conn(0x7f2cf40c43d0 legacy=0x7f2cf4154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:27.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cfac9e700 1 -- 172.21.15.7:0/3305392872 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf41551e0 con 0x7f2cf40c4850 2023-03-03T16:40:27.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cfa49d700 1 --2- 172.21.15.7:0/3305392872 >> v2:172.21.15.143:3300/0 conn(0x7f2cf414e560 0x7f2cf4071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:40:27.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.249+0000 7f2cfac9e700 1 --2- 172.21.15.7:0/3305392872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c4850 0x7f2cf4154ca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2cec009d90 tx=0x7f2cec00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f594243fcb63088a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:27.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cf949b700 1 -- 172.21.15.7:0/3305392872 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00e070 con 0x7f2cf40c4850 2023-03-03T16:40:27.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cf949b700 1 -- 172.21.15.7:0/3305392872 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec00bb20 con 0x7f2cf40c4850 2023-03-03T16:40:27.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cf949b700 1 -- 172.21.15.7:0/3305392872 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec012430 con 0x7f2cf40c4850 2023-03-03T16:40:27.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cfc701700 1 -- 172.21.15.7:0/3305392872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c4850 msgr2=0x7f2cf4154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:27.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cfc701700 1 --2- 172.21.15.7:0/3305392872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c4850 0x7f2cf4154ca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2cec009d90 tx=0x7f2cec00b330 comp rx=0 tx=0).stop 2023-03-03T16:40:27.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cfc701700 1 -- 172.21.15.7:0/3305392872 shutdown_connections 2023-03-03T16:40:27.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cfc701700 1 --2- 172.21.15.7:0/3305392872 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c4850 0x7f2cf4154ca0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:27.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cfc701700 1 --2- 172.21.15.7:0/3305392872 >> v2:172.21.15.143:3300/0 conn(0x7f2cf414e560 0x7f2cf4071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:27.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cfc701700 1 -- 172.21.15.7:0/3305392872 >> 172.21.15.7:0/3305392872 conn(0x7f2cf40736f0 msgr2=0x7f2cf4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:27.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cfc701700 1 -- 172.21.15.7:0/3305392872 shutdown_connections 2023-03-03T16:40:27.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.250+0000 7f2cfc701700 1 -- 172.21.15.7:0/3305392872 wait complete. 2023-03-03T16:40:27.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.252+0000 7f2cfc701700 1 Processor -- start 2023-03-03T16:40:27.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.252+0000 7f2cfc701700 1 -- start start 2023-03-03T16:40:27.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfc701700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c43d0 0x7f2cf40be420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:27.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfc701700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c4850 0x7f2cf40ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:27.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfc701700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf414e560 0x7f2cf40ab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:27.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfc701700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf41ed050 con 0x7f2cf414e560 2023-03-03T16:40:27.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfc701700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf414c440 con 0x7f2cf40c4850 2023-03-03T16:40:27.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfc701700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf406b1b0 con 0x7f2cf40c43d0 2023-03-03T16:40:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfac9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf414e560 0x7f2cf40ab9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfac9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf414e560 0x7f2cf40ab9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58122/0 (socket says 172.21.15.7:58122) 2023-03-03T16:40:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfac9e700 1 -- 172.21.15.7:0/2161023404 learned_addr learned my addr 172.21.15.7:0/2161023404 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:40:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfa49d700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c43d0 0x7f2cf40be420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cf9c9c700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c4850 0x7f2cf40ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfac9e700 1 -- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c43d0 msgr2=0x7f2cf40be420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfac9e700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c43d0 0x7f2cf40be420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfac9e700 1 -- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c4850 msgr2=0x7f2cf40ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfac9e700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c4850 0x7f2cf40ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfac9e700 1 -- 172.21.15.7:0/2161023404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce40094a0 con 0x7f2cf414e560 2023-03-03T16:40:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cfa49d700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c43d0 0x7f2cf40be420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:40:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.253+0000 7f2cf9c9c700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c4850 0x7f2cf40ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:40:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.254+0000 7f2cfac9e700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf414e560 0x7f2cf40ab9a0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f2cec014040 tx=0x7f2cec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.254+0000 7f2ceaffd700 1 -- 172.21.15.7:0/2161023404 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec012430 con 0x7f2cf414e560 2023-03-03T16:40:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.254+0000 7f2ceaffd700 1 -- 172.21.15.7:0/2161023404 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec020070 con 0x7f2cf414e560 2023-03-03T16:40:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.254+0000 7f2ceaffd700 1 -- 172.21.15.7:0/2161023404 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec003ea0 con 0x7f2cf414e560 2023-03-03T16:40:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.254+0000 7f2cfc701700 1 -- 172.21.15.7:0/2161023404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cec009a70 con 0x7f2cf414e560 2023-03-03T16:40:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.254+0000 7f2cfc701700 1 -- 172.21.15.7:0/2161023404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf40aacc0 con 0x7f2cf414e560 2023-03-03T16:40:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.255+0000 7f2cfc701700 1 -- 172.21.15.7:0/2161023404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf406b910 con 0x7f2cf414e560 2023-03-03T16:40:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.255+0000 7f2ceaffd700 1 -- 172.21.15.7:0/2161023404 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2cec02e070 con 0x7f2cf414e560 2023-03-03T16:40:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.256+0000 7f2ceaffd700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 0x7f2cd806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.256+0000 7f2ceaffd700 1 -- 172.21.15.7:0/2161023404 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2cec022070 con 0x7f2cf414e560 2023-03-03T16:40:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.256+0000 7f2cfa49d700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 0x7f2cd806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.256+0000 7f2cfa49d700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 0x7f2cd806a7a0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009d20 tx=0x7f2ce4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.260+0000 7f2ceaffd700 1 -- 172.21.15.7:0/2161023404 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2cec08b030 con 0x7f2cf414e560 2023-03-03T16:40:27.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.495+0000 7f2cfc701700 1 -- 172.21.15.7:0/2161023404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cf406bb80 con 0x7f2cf414e560 2023-03-03T16:40:27.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.495+0000 7f2ceaffd700 1 -- 172.21.15.7:0/2161023404 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2cec02c090 con 0x7f2cf414e560 2023-03-03T16:40:27.495 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:40:27.496 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:40:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 -- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 msgr2=0x7f2cd806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:27.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 0x7f2cd806a7a0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009d20 tx=0x7f2ce4008040 comp rx=0 tx=0).stop 2023-03-03T16:40:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 -- 172.21.15.7:0/2161023404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf414e560 msgr2=0x7f2cf40ab9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf414e560 0x7f2cf40ab9a0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f2cec014040 tx=0x7f2cec004ad0 comp rx=0 tx=0).stop 2023-03-03T16:40:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 -- 172.21.15.7:0/2161023404 shutdown_connections 2023-03-03T16:40:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cd80682f0 0x7f2cd806a7a0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2cf414e560 0x7f2cf40ab9a0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2cf40c43d0 0x7f2cf40be420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:27.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 --2- 172.21.15.7:0/2161023404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cf40c4850 0x7f2cf40ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 -- 172.21.15.7:0/2161023404 >> 172.21.15.7:0/2161023404 conn(0x7f2cf40736f0 msgr2=0x7f2cf40c8530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 -- 172.21.15.7:0/2161023404 shutdown_connections 2023-03-03T16:40:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:27.497+0000 7f2cfc701700 1 -- 172.21.15.7:0/2161023404 wait complete. 2023-03-03T16:40:27.499 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:40:27.509 INFO:tasks.mds_thrash.fs.[b]:waiting for 20 secs before thrashing 2023-03-03T16:40:47.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:40:47.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.652+0000 7f11c18c5700 1 Processor -- start 2023-03-03T16:40:47.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.652+0000 7f11c18c5700 1 -- start start 2023-03-03T16:40:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11c18c5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f11bc0caa30 0x7f11bc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11c18c5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11bc154ff0 0x7f11bc0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11c18c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc0c7310 con 0x7f11bc0c00e0 2023-03-03T16:40:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11c18c5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11bc155430 con 0x7f11bc154ff0 2023-03-03T16:40:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11c18c5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11bc0bfd10 con 0x7f11bc0caa30 2023-03-03T16:40:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11bb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11bc154ff0 0x7f11bc0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11bb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f11bc154ff0 0x7f11bc0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58132/0 (socket says 172.21.15.7:58132) 2023-03-03T16:40:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11bb7fe700 1 -- 172.21.15.7:0/4082048428 learned_addr learned my addr 172.21.15.7:0/4082048428 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:40:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11baffd700 1 --2- 172.21.15.7:0/4082048428 >> v2:172.21.15.143:3300/0 conn(0x7f11bc0caa30 0x7f11bc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:47.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11bb7fe700 1 -- 172.21.15.7:0/4082048428 >> v2:172.21.15.143:3300/0 conn(0x7f11bc0caa30 msgr2=0x7f11bc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:40:47.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11bb7fe700 1 --2- 172.21.15.7:0/4082048428 >> v2:172.21.15.143:3300/0 conn(0x7f11bc0caa30 0x7f11bc0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:47.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11bb7fe700 1 -- 172.21.15.7:0/4082048428 >> v1:172.21.15.143:6789/0 conn(0x7f11bc0c00e0 legacy=0x7f11bc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:40:47.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11bb7fe700 1 -- 172.21.15.7:0/4082048428 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc0bfa30 con 0x7f11bc154ff0 2023-03-03T16:40:47.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.653+0000 7f11bb7fe700 1 --2- 172.21.15.7:0/4082048428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 0x7f11bc0cae00 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0c81b0 tx=0x7f11b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3449b632558b3b08 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:47.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11b9ffb700 1 -- 172.21.15.7:0/4082048428 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b400e070 con 0x7f11bc154ff0 2023-03-03T16:40:47.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11b9ffb700 1 -- 172.21.15.7:0/4082048428 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b4003a20 con 0x7f11bc154ff0 2023-03-03T16:40:47.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11b9ffb700 1 -- 172.21.15.7:0/4082048428 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4003dd0 con 0x7f11bc154ff0 2023-03-03T16:40:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11c18c5700 1 -- 172.21.15.7:0/4082048428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 msgr2=0x7f11bc0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:47.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11c18c5700 1 --2- 172.21.15.7:0/4082048428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 0x7f11bc0cae00 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0c81b0 tx=0x7f11b400b330 comp rx=0 tx=0).stop 2023-03-03T16:40:47.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11c18c5700 1 -- 172.21.15.7:0/4082048428 shutdown_connections 2023-03-03T16:40:47.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11c18c5700 1 --2- 172.21.15.7:0/4082048428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 0x7f11bc0cae00 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:47.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11c18c5700 1 --2- 172.21.15.7:0/4082048428 >> v2:172.21.15.143:3300/0 conn(0x7f11bc0caa30 0x7f11bc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:47.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11c18c5700 1 -- 172.21.15.7:0/4082048428 >> 172.21.15.7:0/4082048428 conn(0x7f11bc066e40 msgr2=0x7f11bc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:47.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11c18c5700 1 -- 172.21.15.7:0/4082048428 shutdown_connections 2023-03-03T16:40:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.654+0000 7f11c18c5700 1 -- 172.21.15.7:0/4082048428 wait complete. 2023-03-03T16:40:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.655+0000 7f11c18c5700 1 Processor -- start 2023-03-03T16:40:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.656+0000 7f11c18c5700 1 -- start start 2023-03-03T16:40:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.656+0000 7f11c18c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11bc0c00e0 0x7f11bc1583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.656+0000 7f11c18c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11bc0caa30 0x7f11bc1ed080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.656+0000 7f11c18c5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 0x7f11bc0adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.656+0000 7f11c18c5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc0c2cc0 con 0x7f11bc154ff0 2023-03-03T16:40:47.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.656+0000 7f11c18c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc158920 con 0x7f11bc0caa30 2023-03-03T16:40:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.656+0000 7f11c18c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc14cec0 con 0x7f11bc0c00e0 2023-03-03T16:40:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.656+0000 7f11bb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 0x7f11bc0adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.656+0000 7f11bb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 0x7f11bc0adf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58138/0 (socket says 172.21.15.7:58138) 2023-03-03T16:40:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.656+0000 7f11bb7fe700 1 -- 172.21.15.7:0/3408926351 learned_addr learned my addr 172.21.15.7:0/3408926351 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:40:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11baffd700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11bc0c00e0 0x7f11bc1583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11ba7fc700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11bc0caa30 0x7f11bc1ed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11bb7fe700 1 -- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11bc0c00e0 msgr2=0x7f11bc1583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:47.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11bb7fe700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11bc0c00e0 0x7f11bc1583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11bb7fe700 1 -- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11bc0caa30 msgr2=0x7f11bc1ed080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11bb7fe700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11bc0caa30 0x7f11bc1ed080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11bb7fe700 1 -- 172.21.15.7:0/3408926351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b4009b30 con 0x7f11bc154ff0 2023-03-03T16:40:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11ba7fc700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11bc0caa30 0x7f11bc1ed080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:40:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11baffd700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11bc0c00e0 0x7f11bc1583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:40:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11bb7fe700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 0x7f11bc0adf10 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f11b4015040 tx=0x7f11b4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:47.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11a3fff700 1 -- 172.21.15.7:0/3408926351 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4004440 con 0x7f11bc154ff0 2023-03-03T16:40:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11a3fff700 1 -- 172.21.15.7:0/3408926351 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b40045a0 con 0x7f11bc154ff0 2023-03-03T16:40:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11c18c5700 1 -- 172.21.15.7:0/3408926351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc1547f0 con 0x7f11bc154ff0 2023-03-03T16:40:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11a3fff700 1 -- 172.21.15.7:0/3408926351 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b40092e0 con 0x7f11bc154ff0 2023-03-03T16:40:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.657+0000 7f11c18c5700 1 -- 172.21.15.7:0/3408926351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc154510 con 0x7f11bc154ff0 2023-03-03T16:40:47.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.658+0000 7f11c18c5700 1 -- 172.21.15.7:0/3408926351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11bc157870 con 0x7f11bc154ff0 2023-03-03T16:40:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.658+0000 7f11a3fff700 1 -- 172.21.15.7:0/3408926351 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f11b4022030 con 0x7f11bc154ff0 2023-03-03T16:40:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.659+0000 7f11a3fff700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11a80682f0 0x7f11a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.659+0000 7f11a3fff700 1 -- 172.21.15.7:0/3408926351 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f11b4087590 con 0x7f11bc154ff0 2023-03-03T16:40:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.659+0000 7f11baffd700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11a80682f0 0x7f11a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:47.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.660+0000 7f11baffd700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11a80682f0 0x7f11a806a7a0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f11ac006fd0 tx=0x7f11ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:47.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.663+0000 7f11a3fff700 1 -- 172.21.15.7:0/3408926351 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f11b40538b0 con 0x7f11bc154ff0 2023-03-03T16:40:47.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.898+0000 7f11c18c5700 1 -- 172.21.15.7:0/3408926351 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11bc154ad0 con 0x7f11bc154ff0 2023-03-03T16:40:47.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.899+0000 7f11a3fff700 1 -- 172.21.15.7:0/3408926351 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f11b4007310 con 0x7f11bc154ff0 2023-03-03T16:40:47.899 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:40:47.899 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:40:47.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 -- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11a80682f0 msgr2=0x7f11a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:47.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11a80682f0 0x7f11a806a7a0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f11ac006fd0 tx=0x7f11ac008040 comp rx=0 tx=0).stop 2023-03-03T16:40:47.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 -- 172.21.15.7:0/3408926351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 msgr2=0x7f11bc0adf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 0x7f11bc0adf10 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f11b4015040 tx=0x7f11b4004ad0 comp rx=0 tx=0).stop 2023-03-03T16:40:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 -- 172.21.15.7:0/3408926351 shutdown_connections 2023-03-03T16:40:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f11a80682f0 0x7f11a806a7a0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f11bc154ff0 0x7f11bc0adf10 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f11bc0c00e0 0x7f11bc1583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 --2- 172.21.15.7:0/3408926351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f11bc0caa30 0x7f11bc1ed080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 -- 172.21.15.7:0/3408926351 >> 172.21.15.7:0/3408926351 conn(0x7f11bc066e40 msgr2=0x7f11bc151450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:47.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 -- 172.21.15.7:0/3408926351 shutdown_connections 2023-03-03T16:40:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:47.901+0000 7f11c18c5700 1 -- 172.21.15.7:0/3408926351 wait complete. 2023-03-03T16:40:47.903 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:40:47.913 INFO:tasks.mds_thrash.fs.[b]:waiting for 47 secs before thrashing 2023-03-03T16:40:48.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:48.160 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:40:49.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:40:49.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.183+0000 7f6647292700 1 Processor -- start 2023-03-03T16:40:49.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.183+0000 7f6647292700 1 -- start start 2023-03-03T16:40:49.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f6647292700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f664014f9d0 0x7f6640156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:49.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f6647292700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 0x7f6640071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:49.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f6647292700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66400c2f30 con 0x7f66400abe20 2023-03-03T16:40:49.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f6647292700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66400c31d0 con 0x7f664014f9d0 2023-03-03T16:40:49.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f6647292700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66401524a0 con 0x7f664014fda0 2023-03-03T16:40:49.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664502e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f664014f9d0 0x7f6640156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:49.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664482d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 0x7f6640071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:49.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664582f700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f66400abe20 0x7f66401529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:55646/0 (socket says 172.21.15.7:55646) 2023-03-03T16:40:49.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664482d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 0x7f6640071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:37984/0 (socket says 172.21.15.7:37984) 2023-03-03T16:40:49.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664482d700 1 -- 172.21.15.7:0/3129746456 learned_addr learned my addr 172.21.15.7:0/3129746456 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:40:49.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664502e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f664014f9d0 0x7f6640156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:53916/0 (socket says 172.21.15.7:53916) 2023-03-03T16:40:49.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664502e700 1 -- 172.21.15.7:0/3129746456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 msgr2=0x7f6640071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:40:49.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664502e700 1 --2- 172.21.15.7:0/3129746456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 0x7f6640071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:49.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664502e700 1 -- 172.21.15.7:0/3129746456 >> v1:172.21.15.143:6789/0 conn(0x7f66400abe20 legacy=0x7f66401529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:49.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664502e700 1 -- 172.21.15.7:0/3129746456 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66400c34b0 con 0x7f664014f9d0 2023-03-03T16:40:49.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.184+0000 7f664482d700 1 --2- 172.21.15.7:0/3129746456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 0x7f6640071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:40:49.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f664502e700 1 --2- 172.21.15.7:0/3129746456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f664014f9d0 0x7f6640156430 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6630009840 tx=0x7f6630009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=706b119348f34ab2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:49.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6637fff700 1 -- 172.21.15.7:0/3129746456 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630011830 con 0x7f664014f9d0 2023-03-03T16:40:49.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6637fff700 1 -- 172.21.15.7:0/3129746456 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6630011990 con 0x7f664014f9d0 2023-03-03T16:40:49.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6637fff700 1 -- 172.21.15.7:0/3129746456 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630011d20 con 0x7f664014f9d0 2023-03-03T16:40:49.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6647292700 1 -- 172.21.15.7:0/3129746456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f664014f9d0 msgr2=0x7f6640156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:49.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6647292700 1 --2- 172.21.15.7:0/3129746456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f664014f9d0 0x7f6640156430 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6630009840 tx=0x7f6630009b50 comp rx=0 tx=0).stop 2023-03-03T16:40:49.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6647292700 1 -- 172.21.15.7:0/3129746456 shutdown_connections 2023-03-03T16:40:49.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6647292700 1 --2- 172.21.15.7:0/3129746456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f664014f9d0 0x7f6640156430 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:49.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6647292700 1 --2- 172.21.15.7:0/3129746456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 0x7f6640071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:49.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6647292700 1 -- 172.21.15.7:0/3129746456 >> 172.21.15.7:0/3129746456 conn(0x7f66400c8900 msgr2=0x7f66400c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:49.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6647292700 1 -- 172.21.15.7:0/3129746456 shutdown_connections 2023-03-03T16:40:49.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.185+0000 7f6647292700 1 -- 172.21.15.7:0/3129746456 wait complete. 2023-03-03T16:40:49.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.187+0000 7f6647292700 1 Processor -- start 2023-03-03T16:40:49.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.187+0000 7f6647292700 1 -- start start 2023-03-03T16:40:49.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f6647292700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66400abe20 0x7f6640154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:49.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f6647292700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f664014f9d0 0x7f66401eb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:49.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f6647292700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 0x7f66401eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:49.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f6647292700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66401eaf50 con 0x7f66400abe20 2023-03-03T16:40:49.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f6647292700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664006e420 con 0x7f664014f9d0 2023-03-03T16:40:49.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f6647292700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664006e700 con 0x7f664014fda0 2023-03-03T16:40:49.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f664502e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66400abe20 0x7f6640154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:49.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f664502e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66400abe20 0x7f6640154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58148/0 (socket says 172.21.15.7:58148) 2023-03-03T16:40:49.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f664502e700 1 -- 172.21.15.7:0/2164292911 learned_addr learned my addr 172.21.15.7:0/2164292911 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:40:49.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f664482d700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f664014f9d0 0x7f66401eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:49.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f664582f700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 0x7f66401eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:49.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f664502e700 1 -- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 msgr2=0x7f66401eb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:49.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f664502e700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 0x7f66401eb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:49.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f664502e700 1 -- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f664014f9d0 msgr2=0x7f66401eb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:49.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.188+0000 7f664502e700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f664014f9d0 0x7f66401eb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:49.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.189+0000 7f664502e700 1 -- 172.21.15.7:0/2164292911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c009550 con 0x7f66400abe20 2023-03-03T16:40:49.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.189+0000 7f664502e700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66400abe20 0x7f6640154f60 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f6630009fd0 tx=0x7f6630004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:49.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.189+0000 7f6635ffb700 1 -- 172.21.15.7:0/2164292911 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630004530 con 0x7f66400abe20 2023-03-03T16:40:49.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.189+0000 7f6635ffb700 1 -- 172.21.15.7:0/2164292911 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663001f070 con 0x7f66400abe20 2023-03-03T16:40:49.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.189+0000 7f6635ffb700 1 -- 172.21.15.7:0/2164292911 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66300038e0 con 0x7f66400abe20 2023-03-03T16:40:49.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.189+0000 7f6647292700 1 -- 172.21.15.7:0/2164292911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66300094a0 con 0x7f66400abe20 2023-03-03T16:40:49.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.189+0000 7f6647292700 1 -- 172.21.15.7:0/2164292911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66400ac350 con 0x7f66400abe20 2023-03-03T16:40:49.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.190+0000 7f66237fe700 1 -- 172.21.15.7:0/2164292911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c0052e0 con 0x7f66400abe20 2023-03-03T16:40:49.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.191+0000 7f6635ffb700 1 -- 172.21.15.7:0/2164292911 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f663002a020 con 0x7f66400abe20 2023-03-03T16:40:49.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.191+0000 7f6635ffb700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66240682f0 0x7f662406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:40:49.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.191+0000 7f6635ffb700 1 -- 172.21.15.7:0/2164292911 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f66300875b0 con 0x7f66400abe20 2023-03-03T16:40:49.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.191+0000 7f664482d700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66240682f0 0x7f662406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:40:49.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.192+0000 7f664482d700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66240682f0 0x7f662406a7a0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f664006f5c0 tx=0x7f663c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:40:49.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.195+0000 7f6635ffb700 1 -- 172.21.15.7:0/2164292911 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f66300538f0 con 0x7f66400abe20 2023-03-03T16:40:49.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.435+0000 7f66237fe700 1 -- 172.21.15.7:0/2164292911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f661c005b10 con 0x7f66400abe20 2023-03-03T16:40:49.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.435+0000 7f6635ffb700 1 -- 172.21.15.7:0/2164292911 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6630056f10 con 0x7f66400abe20 2023-03-03T16:40:49.436 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:40:49.436 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:40:49.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 -- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66240682f0 msgr2=0x7f662406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:49.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66240682f0 0x7f662406a7a0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f664006f5c0 tx=0x7f663c008040 comp rx=0 tx=0).stop 2023-03-03T16:40:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 -- 172.21.15.7:0/2164292911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66400abe20 msgr2=0x7f6640154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:40:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66400abe20 0x7f6640154f60 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f6630009fd0 tx=0x7f6630004750 comp rx=0 tx=0).stop 2023-03-03T16:40:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 -- 172.21.15.7:0/2164292911 shutdown_connections 2023-03-03T16:40:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f664014fda0 0x7f66401eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66240682f0 0x7f662406a7a0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66400abe20 0x7f6640154f60 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:49.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 --2- 172.21.15.7:0/2164292911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f664014f9d0 0x7f66401eb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:40:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 -- 172.21.15.7:0/2164292911 >> 172.21.15.7:0/2164292911 conn(0x7f66400c8900 msgr2=0x7f66400c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:40:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 -- 172.21.15.7:0/2164292911 shutdown_connections 2023-03-03T16:40:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:40:49.437+0000 7f66237fe700 1 -- 172.21.15.7:0/2164292911 wait complete. 2023-03-03T16:40:49.439 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:40:49.449 INFO:tasks.mds_thrash.fs.[a]:waiting for 85 secs before thrashing 2023-03-03T16:41:18.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:18.233 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:34.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:41:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.051+0000 7fe69b71e700 1 Processor -- start 2023-03-03T16:41:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.052+0000 7fe69b71e700 1 -- start start 2023-03-03T16:41:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.052+0000 7fe69b71e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940c1da0 0x7fe6941550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.052+0000 7fe69b71e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6940c7a60 con 0x7fe6940c3e30 2023-03-03T16:41:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.052+0000 7fe69b71e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6940c3620 con 0x7fe69406d870 2023-03-03T16:41:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.052+0000 7fe69b71e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940c2af0 con 0x7fe6940c1da0 2023-03-03T16:41:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe699cbb700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe6940c3e30 0x7fe69415ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52178/0 (socket says 172.21.15.7:52178) 2023-03-03T16:41:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe699cbb700 1 -- 172.21.15.7:0/3336161328 learned_addr learned my addr 172.21.15.7:0/3336161328 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:41:35.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe698cb9700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940c1da0 0x7fe6941550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:35.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878112169 0 0) 0x7fe6940c7a60 con 0x7fe6940c3e30 2023-03-03T16:41:35.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6800035b0 con 0x7fe6940c3e30 2023-03-03T16:41:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2624539202 0 0) 0x7fe6800035b0 con 0x7fe6940c3e30 2023-03-03T16:41:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940c1da0 msgr2=0x7fe6941550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:41:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940c1da0 0x7fe6941550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 >> v1:172.21.15.143:6789/0 conn(0x7fe69406d870 legacy=0x7fe6940c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:35.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6941545e0 con 0x7fe6940c3e30 2023-03-03T16:41:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe68c0019f0 con 0x7fe6940c3e30 2023-03-03T16:41:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 >> v1:172.21.15.7:6789/0 conn(0x7fe6940c3e30 legacy=0x7fe69415ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940c1da0 0x7fe680003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 0x7fe680004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe680004d90 0x7fe680005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68000bac0 con 0x7fe680003fe0 2023-03-03T16:41:35.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68000bc00 con 0x7fe680004d90 2023-03-03T16:41:35.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68000bdc0 con 0x7fe6940c1da0 2023-03-03T16:41:35.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe698cb9700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 0x7fe680004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:35.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.053+0000 7fe6994ba700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940c1da0 0x7fe680003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:35.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe699cbb700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe680004d90 0x7fe680005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:35.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe698cb9700 1 -- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940c1da0 msgr2=0x7fe680003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:41:35.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe698cb9700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6940c1da0 0x7fe680003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:35.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe698cb9700 1 -- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe680004d90 msgr2=0x7fe680005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:41:35.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe698cb9700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe680004d90 0x7fe680005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:35.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe698cb9700 1 -- 172.21.15.7:0/3336161328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6800035b0 con 0x7fe680003fe0 2023-03-03T16:41:35.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe699cbb700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe680004d90 0x7fe680005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:41:35.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe698cb9700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 0x7fe680004850 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fe688009b30 tx=0x7fe680005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01761d45dd4c6be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:41:35.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68800e6c0 con 0x7fe680003fe0 2023-03-03T16:41:35.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe68800e820 con 0x7fe680003fe0 2023-03-03T16:41:35.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe687fff700 1 -- 172.21.15.7:0/3336161328 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68800eb90 con 0x7fe680003fe0 2023-03-03T16:41:35.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe69b71e700 1 -- 172.21.15.7:0/3336161328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 msgr2=0x7fe680004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe69b71e700 1 --2- 172.21.15.7:0/3336161328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 0x7fe680004850 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7fe688009b30 tx=0x7fe680005880 comp rx=0 tx=0).stop 2023-03-03T16:41:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.054+0000 7fe699cbb700 1 -- 172.21.15.7:0/3336161328 reap_dead start 2023-03-03T16:41:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.055+0000 7fe69b71e700 1 -- 172.21.15.7:0/3336161328 shutdown_connections 2023-03-03T16:41:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.055+0000 7fe69b71e700 1 -- 172.21.15.7:0/3336161328 >> 172.21.15.7:0/3336161328 conn(0x7fe69406c110 msgr2=0x7fe69406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:41:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.055+0000 7fe69b71e700 1 -- 172.21.15.7:0/3336161328 shutdown_connections 2023-03-03T16:41:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.055+0000 7fe69b71e700 1 -- 172.21.15.7:0/3336161328 wait complete. 2023-03-03T16:41:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.056+0000 7fe69b71e700 1 Processor -- start 2023-03-03T16:41:35.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.056+0000 7fe69b71e700 1 -- start start 2023-03-03T16:41:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe69b71e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 0x7fe6941ed180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:35.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe69b71e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6941ed6c0 0x7fe69414ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe69b71e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69414af60 0x7fe69414d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe69b71e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe694139b30 con 0x7fe680003fe0 2023-03-03T16:41:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe69b71e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6941ede50 con 0x7fe69414af60 2023-03-03T16:41:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe69b71e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe694157d50 con 0x7fe6941ed6c0 2023-03-03T16:41:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe6994ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 0x7fe6941ed180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe6994ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 0x7fe6941ed180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58206/0 (socket says 172.21.15.7:58206) 2023-03-03T16:41:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe6994ba700 1 -- 172.21.15.7:0/4018444335 learned_addr learned my addr 172.21.15.7:0/4018444335 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:41:35.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe699cbb700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69414af60 0x7fe69414d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe698cb9700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6941ed6c0 0x7fe69414ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe6994ba700 1 -- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6941ed6c0 msgr2=0x7fe69414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe6994ba700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6941ed6c0 0x7fe69414ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe6994ba700 1 -- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69414af60 msgr2=0x7fe69414d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe6994ba700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69414af60 0x7fe69414d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe6994ba700 1 -- 172.21.15.7:0/4018444335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe694154ba0 con 0x7fe680003fe0 2023-03-03T16:41:35.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe6994ba700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 0x7fe6941ed180 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fe69000e780 tx=0x7fe69000eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:41:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe685ffb700 1 -- 172.21.15.7:0/4018444335 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6900052e0 con 0x7fe680003fe0 2023-03-03T16:41:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe685ffb700 1 -- 172.21.15.7:0/4018444335 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe690005440 con 0x7fe680003fe0 2023-03-03T16:41:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe69b71e700 1 -- 172.21.15.7:0/4018444335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6941548c0 con 0x7fe680003fe0 2023-03-03T16:41:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.057+0000 7fe685ffb700 1 -- 172.21.15.7:0/4018444335 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6900038c0 con 0x7fe680003fe0 2023-03-03T16:41:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.058+0000 7fe69b71e700 1 -- 172.21.15.7:0/4018444335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69406dc40 con 0x7fe680003fe0 2023-03-03T16:41:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.059+0000 7fe69b71e700 1 -- 172.21.15.7:0/4018444335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe694158360 con 0x7fe680003fe0 2023-03-03T16:41:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.059+0000 7fe685ffb700 1 -- 172.21.15.7:0/4018444335 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe690003a20 con 0x7fe680003fe0 2023-03-03T16:41:35.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.059+0000 7fe685ffb700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6780682f0 0x7fe67806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:41:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.060+0000 7fe685ffb700 1 -- 172.21.15.7:0/4018444335 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe690086210 con 0x7fe680003fe0 2023-03-03T16:41:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.060+0000 7fe698cb9700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6780682f0 0x7fe67806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:41:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.060+0000 7fe698cb9700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6780682f0 0x7fe67806a7a0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fe6880090b0 tx=0x7fe688008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:41:35.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.064+0000 7fe685ffb700 1 -- 172.21.15.7:0/4018444335 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe690052440 con 0x7fe680003fe0 2023-03-03T16:41:35.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.303+0000 7fe69b71e700 1 -- 172.21.15.7:0/4018444335 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe69404e6e0 con 0x7fe680003fe0 2023-03-03T16:41:35.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.304+0000 7fe685ffb700 1 -- 172.21.15.7:0/4018444335 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe690055a60 con 0x7fe680003fe0 2023-03-03T16:41:35.304 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:41:35.305 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:41:35.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 -- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6780682f0 msgr2=0x7fe67806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:35.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6780682f0 0x7fe67806a7a0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fe6880090b0 tx=0x7fe688008030 comp rx=0 tx=0).stop 2023-03-03T16:41:35.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 -- 172.21.15.7:0/4018444335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 msgr2=0x7fe6941ed180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:41:35.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 0x7fe6941ed180 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fe69000e780 tx=0x7fe69000eb40 comp rx=0 tx=0).stop 2023-03-03T16:41:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 -- 172.21.15.7:0/4018444335 shutdown_connections 2023-03-03T16:41:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe69414af60 0x7fe69414d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6780682f0 0x7fe67806a7a0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe680003fe0 0x7fe6941ed180 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 --2- 172.21.15.7:0/4018444335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6941ed6c0 0x7fe69414ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:41:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 -- 172.21.15.7:0/4018444335 >> 172.21.15.7:0/4018444335 conn(0x7fe69406c110 msgr2=0x7fe6940718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:41:35.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 -- 172.21.15.7:0/4018444335 shutdown_connections 2023-03-03T16:41:35.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:41:35.306+0000 7fe69b71e700 1 -- 172.21.15.7:0/4018444335 wait complete. 2023-03-03T16:41:35.308 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:41:35.318 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2023-03-03T16:41:48.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:41:48.304 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:09.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:42:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.457+0000 7f79d8d51700 1 Processor -- start 2023-03-03T16:42:09.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.457+0000 7f79d8d51700 1 -- start start 2023-03-03T16:42:09.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d8d51700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79d4151cd0 0x7f79d4154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:09.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d8d51700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f79d40c2e30 0x7f79d4158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:09.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d8d51700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f79d406c6c0 0x7f79d40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:09.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d8d51700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d4157ff0 con 0x7f79d40c2e30 2023-03-03T16:42:09.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d8d51700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79d406d640 con 0x7f79d406c6c0 2023-03-03T16:42:09.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d8d51700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d406cda0 con 0x7f79d4151cd0 2023-03-03T16:42:09.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d1d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f79d40c2e30 0x7f79d4158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:09.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d1d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f79d40c2e30 0x7f79d4158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58236/0 (socket says 172.21.15.7:58236) 2023-03-03T16:42:09.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d1d9b700 1 -- 172.21.15.7:0/549838804 learned_addr learned my addr 172.21.15.7:0/549838804 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:42:09.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d2d9d700 1 --2- 172.21.15.7:0/549838804 >> v2:172.21.15.143:3300/0 conn(0x7f79d406c6c0 0x7f79d40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:09.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d259c700 1 --2- 172.21.15.7:0/549838804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79d4151cd0 0x7f79d4154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:09.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d1d9b700 1 -- 172.21.15.7:0/549838804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79d4151cd0 msgr2=0x7f79d4154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:42:09.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d1d9b700 1 --2- 172.21.15.7:0/549838804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79d4151cd0 0x7f79d4154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d1d9b700 1 -- 172.21.15.7:0/549838804 >> v2:172.21.15.143:3300/0 conn(0x7f79d406c6c0 msgr2=0x7f79d40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:42:09.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d1d9b700 1 --2- 172.21.15.7:0/549838804 >> v2:172.21.15.143:3300/0 conn(0x7f79d406c6c0 0x7f79d40736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d1d9b700 1 -- 172.21.15.7:0/549838804 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d406d080 con 0x7f79d40c2e30 2023-03-03T16:42:09.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.458+0000 7f79d1d9b700 1 --2- 172.21.15.7:0/549838804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d40c2e30 0x7f79d4158480 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f79d406ffb0 tx=0x7f79c8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df09c8d17a9ab654 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:09.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d159a700 1 -- 172.21.15.7:0/549838804 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c800b870 con 0x7f79d40c2e30 2023-03-03T16:42:09.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d159a700 1 -- 172.21.15.7:0/549838804 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c800b9d0 con 0x7f79d40c2e30 2023-03-03T16:42:09.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d159a700 1 -- 172.21.15.7:0/549838804 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79c800bd60 con 0x7f79d40c2e30 2023-03-03T16:42:09.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d8d51700 1 -- 172.21.15.7:0/549838804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d40c2e30 msgr2=0x7f79d4158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:09.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d8d51700 1 --2- 172.21.15.7:0/549838804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d40c2e30 0x7f79d4158480 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f79d406ffb0 tx=0x7f79c8009bd0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d8d51700 1 -- 172.21.15.7:0/549838804 shutdown_connections 2023-03-03T16:42:09.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d8d51700 1 --2- 172.21.15.7:0/549838804 >> v2:172.21.15.143:3300/0 conn(0x7f79d406c6c0 0x7f79d40736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d8d51700 1 --2- 172.21.15.7:0/549838804 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79d4151cd0 0x7f79d4154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d8d51700 1 --2- 172.21.15.7:0/549838804 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d40c2e30 0x7f79d4158480 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d8d51700 1 -- 172.21.15.7:0/549838804 >> 172.21.15.7:0/549838804 conn(0x7f79d40a94d0 msgr2=0x7f79d40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:42:09.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d8d51700 1 -- 172.21.15.7:0/549838804 shutdown_connections 2023-03-03T16:42:09.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.459+0000 7f79d8d51700 1 -- 172.21.15.7:0/549838804 wait complete. 2023-03-03T16:42:09.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.461+0000 7f79d8d51700 1 Processor -- start 2023-03-03T16:42:09.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.461+0000 7f79d8d51700 1 -- start start 2023-03-03T16:42:09.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.461+0000 7f79d8d51700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79d406c6c0 0x7f79d41ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:09.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.461+0000 7f79d8d51700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d40c2e30 0x7f79d414dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:09.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.461+0000 7f79d8d51700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d4151cd0 0x7f79d40709f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:09.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.461+0000 7f79d8d51700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40c68e0 con 0x7f79d4151cd0 2023-03-03T16:42:09.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.461+0000 7f79d8d51700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40c6b30 con 0x7f79d40c2e30 2023-03-03T16:42:09.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.461+0000 7f79d8d51700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d40c6e10 con 0x7f79d406c6c0 2023-03-03T16:42:09.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.461+0000 7f79d2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d4151cd0 0x7f79d40709f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:09.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.461+0000 7f79d2d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d4151cd0 0x7f79d40709f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58246/0 (socket says 172.21.15.7:58246) 2023-03-03T16:42:09.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d2d9d700 1 -- 172.21.15.7:0/2352994620 learned_addr learned my addr 172.21.15.7:0/2352994620 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:42:09.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d259c700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79d406c6c0 0x7f79d41ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:09.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d1d9b700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d40c2e30 0x7f79d414dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:09.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d2d9d700 1 -- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79d406c6c0 msgr2=0x7f79d41ee1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d2d9d700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79d406c6c0 0x7f79d41ee1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d2d9d700 1 -- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d40c2e30 msgr2=0x7f79d414dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d2d9d700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d40c2e30 0x7f79d414dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d2d9d700 1 -- 172.21.15.7:0/2352994620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c8009550 con 0x7f79d4151cd0 2023-03-03T16:42:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d1d9b700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d40c2e30 0x7f79d414dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:42:09.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d2d9d700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d4151cd0 0x7f79d40709f0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f79cc007d20 tx=0x7f79cc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79c2ffd700 1 -- 172.21.15.7:0/2352994620 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00ede0 con 0x7f79d4151cd0 2023-03-03T16:42:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79c2ffd700 1 -- 172.21.15.7:0/2352994620 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc007e70 con 0x7f79d4151cd0 2023-03-03T16:42:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d8d51700 1 -- 172.21.15.7:0/2352994620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d4157c80 con 0x7f79d4151cd0 2023-03-03T16:42:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79c2ffd700 1 -- 172.21.15.7:0/2352994620 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79cc014640 con 0x7f79d4151cd0 2023-03-03T16:42:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.462+0000 7f79d8d51700 1 -- 172.21.15.7:0/2352994620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d406d360 con 0x7f79d4151cd0 2023-03-03T16:42:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.463+0000 7f79d8d51700 1 -- 172.21.15.7:0/2352994620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d4154a50 con 0x7f79d4151cd0 2023-03-03T16:42:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.464+0000 7f79c2ffd700 1 -- 172.21.15.7:0/2352994620 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f79cc012050 con 0x7f79d4151cd0 2023-03-03T16:42:09.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.464+0000 7f79c2ffd700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79b4068320 0x7f79b406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.464+0000 7f79c2ffd700 1 -- 172.21.15.7:0/2352994620 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f79cc0865f0 con 0x7f79d4151cd0 2023-03-03T16:42:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.465+0000 7f79d259c700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79b4068320 0x7f79b406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.465+0000 7f79d259c700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79b4068320 0x7f79b406a7d0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f79c4006b20 tx=0x7f79c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:09.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.469+0000 7f79c2ffd700 1 -- 172.21.15.7:0/2352994620 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f79cc052930 con 0x7f79d4151cd0 2023-03-03T16:42:09.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.711+0000 7f79d8d51700 1 -- 172.21.15.7:0/2352994620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79d4074bb0 con 0x7f79d4151cd0 2023-03-03T16:42:09.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.712+0000 7f79c2ffd700 1 -- 172.21.15.7:0/2352994620 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f79cc055f50 con 0x7f79d4151cd0 2023-03-03T16:42:09.713 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:42:09.713 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:42:09.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 -- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79b4068320 msgr2=0x7f79b406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:09.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79b4068320 0x7f79b406a7d0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f79c4006b20 tx=0x7f79c4008040 comp rx=0 tx=0).stop 2023-03-03T16:42:09.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 -- 172.21.15.7:0/2352994620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d4151cd0 msgr2=0x7f79d40709f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:09.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d4151cd0 0x7f79d40709f0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f79cc007d20 tx=0x7f79cc00e3c0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 -- 172.21.15.7:0/2352994620 shutdown_connections 2023-03-03T16:42:09.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79b4068320 0x7f79b406a7d0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79d4151cd0 0x7f79d40709f0 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79d406c6c0 0x7f79d41ee1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 --2- 172.21.15.7:0/2352994620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79d40c2e30 0x7f79d414dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:09.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 -- 172.21.15.7:0/2352994620 >> 172.21.15.7:0/2352994620 conn(0x7f79d40a94d0 msgr2=0x7f79d4065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:42:09.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 -- 172.21.15.7:0/2352994620 shutdown_connections 2023-03-03T16:42:09.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:09.714+0000 7f79d8d51700 1 -- 172.21.15.7:0/2352994620 wait complete. 2023-03-03T16:42:09.717 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:42:09.727 INFO:tasks.mds_thrash.fs.[b]:waiting for 70 secs before thrashing 2023-03-03T16:42:14.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:42:14.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.599+0000 7fb74bea3700 1 Processor -- start 2023-03-03T16:42:14.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.600+0000 7fb74bea3700 1 -- start start 2023-03-03T16:42:14.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.600+0000 7fb74bea3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb74414f9d0 0x7fb744156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:14.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.600+0000 7fb74bea3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb74414fda0 0x7fb744071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:14.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.600+0000 7fb74bea3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7440c2f30 con 0x7fb7440abe20 2023-03-03T16:42:14.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.600+0000 7fb74bea3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7440c31d0 con 0x7fb74414fda0 2023-03-03T16:42:14.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.600+0000 7fb74bea3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7441524a0 con 0x7fb74414f9d0 2023-03-03T16:42:14.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb74943e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb74414fda0 0x7fb744071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb74943e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb74414fda0 0x7fb744071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58250/0 (socket says 172.21.15.7:58250) 2023-03-03T16:42:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb74943e700 1 -- 172.21.15.7:0/1400020101 learned_addr learned my addr 172.21.15.7:0/1400020101 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:42:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb749c3f700 1 --2- 172.21.15.7:0/1400020101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb74414f9d0 0x7fb744156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb74943e700 1 -- 172.21.15.7:0/1400020101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb74414f9d0 msgr2=0x7fb744156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:42:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb74943e700 1 --2- 172.21.15.7:0/1400020101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb74414f9d0 0x7fb744156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb74943e700 1 -- 172.21.15.7:0/1400020101 >> v1:172.21.15.143:6789/0 conn(0x7fb7440abe20 legacy=0x7fb7441529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:14.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb74943e700 1 -- 172.21.15.7:0/1400020101 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7440c34b0 con 0x7fb74414fda0 2023-03-03T16:42:14.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb74943e700 1 --2- 172.21.15.7:0/1400020101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 0x7fb744071890 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7fb7380098f0 tx=0x7fb738009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b80baac52bd9fd2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:14.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb748c3d700 1 -- 172.21.15.7:0/1400020101 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73800b8a0 con 0x7fb74414fda0 2023-03-03T16:42:14.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb748c3d700 1 -- 172.21.15.7:0/1400020101 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73800ba00 con 0x7fb74414fda0 2023-03-03T16:42:14.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.601+0000 7fb748c3d700 1 -- 172.21.15.7:0/1400020101 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73800bd90 con 0x7fb74414fda0 2023-03-03T16:42:14.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.602+0000 7fb74bea3700 1 -- 172.21.15.7:0/1400020101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 msgr2=0x7fb744071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:14.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.602+0000 7fb74bea3700 1 --2- 172.21.15.7:0/1400020101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 0x7fb744071890 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7fb7380098f0 tx=0x7fb738009c00 comp rx=0 tx=0).stop 2023-03-03T16:42:14.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.602+0000 7fb74bea3700 1 -- 172.21.15.7:0/1400020101 shutdown_connections 2023-03-03T16:42:14.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.602+0000 7fb74bea3700 1 --2- 172.21.15.7:0/1400020101 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb74414f9d0 0x7fb744156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:14.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.602+0000 7fb74bea3700 1 --2- 172.21.15.7:0/1400020101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 0x7fb744071890 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:14.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.602+0000 7fb74bea3700 1 -- 172.21.15.7:0/1400020101 >> 172.21.15.7:0/1400020101 conn(0x7fb7440c8900 msgr2=0x7fb7440c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:42:14.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.602+0000 7fb74bea3700 1 -- 172.21.15.7:0/1400020101 shutdown_connections 2023-03-03T16:42:14.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.602+0000 7fb74bea3700 1 -- 172.21.15.7:0/1400020101 wait complete. 2023-03-03T16:42:14.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.603+0000 7fb74bea3700 1 Processor -- start 2023-03-03T16:42:14.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.603+0000 7fb74bea3700 1 -- start start 2023-03-03T16:42:14.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74bea3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7440abe20 0x7fb7440bd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:14.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74bea3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb74414f9d0 0x7fb7440c2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:14.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74bea3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 0x7fb74406d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:14.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74bea3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7440bf780 con 0x7fb74414fda0 2023-03-03T16:42:14.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74bea3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74414cb30 con 0x7fb74414f9d0 2023-03-03T16:42:14.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74bea3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74406d510 con 0x7fb7440abe20 2023-03-03T16:42:14.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74a440700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 0x7fb74406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:14.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74943e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb74414f9d0 0x7fb7440c2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:14.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74a440700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 0x7fb74406d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58260/0 (socket says 172.21.15.7:58260) 2023-03-03T16:42:14.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74a440700 1 -- 172.21.15.7:0/4165205036 learned_addr learned my addr 172.21.15.7:0/4165205036 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:42:14.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb749c3f700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7440abe20 0x7fb7440bd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:14.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74a440700 1 -- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7440abe20 msgr2=0x7fb7440bd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:14.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74a440700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7440abe20 0x7fb7440bd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:14.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74a440700 1 -- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb74414f9d0 msgr2=0x7fb7440c2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:14.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74a440700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb74414f9d0 0x7fb7440c2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:14.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.604+0000 7fb74a440700 1 -- 172.21.15.7:0/4165205036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb738009550 con 0x7fb74414fda0 2023-03-03T16:42:14.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.605+0000 7fb74a440700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 0x7fb74406d960 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00cb50 tx=0x7fb73c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:14.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.605+0000 7fb7367fc700 1 -- 172.21.15.7:0/4165205036 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c003c00 con 0x7fb74414fda0 2023-03-03T16:42:14.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.605+0000 7fb7367fc700 1 -- 172.21.15.7:0/4165205036 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c003d60 con 0x7fb74414fda0 2023-03-03T16:42:14.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.605+0000 7fb74bea3700 1 -- 172.21.15.7:0/4165205036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7440ac350 con 0x7fb74414fda0 2023-03-03T16:42:14.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.605+0000 7fb7367fc700 1 -- 172.21.15.7:0/4165205036 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00aa20 con 0x7fb74414fda0 2023-03-03T16:42:14.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.605+0000 7fb74bea3700 1 -- 172.21.15.7:0/4165205036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7440ab630 con 0x7fb74414fda0 2023-03-03T16:42:14.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.606+0000 7fb74bea3700 1 -- 172.21.15.7:0/4165205036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7440c3e10 con 0x7fb74414fda0 2023-03-03T16:42:14.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.606+0000 7fb7367fc700 1 -- 172.21.15.7:0/4165205036 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb73c010050 con 0x7fb74414fda0 2023-03-03T16:42:14.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.607+0000 7fb7367fc700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7280682f0 0x7fb72806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:14.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.607+0000 7fb7367fc700 1 -- 172.21.15.7:0/4165205036 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb73c085fe0 con 0x7fb74414fda0 2023-03-03T16:42:14.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.607+0000 7fb749c3f700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7280682f0 0x7fb72806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:14.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.607+0000 7fb749c3f700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7280682f0 0x7fb72806a7a0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fb74406fd30 tx=0x7fb740008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:14.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.611+0000 7fb7367fc700 1 -- 172.21.15.7:0/4165205036 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb73c052380 con 0x7fb74414fda0 2023-03-03T16:42:14.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.848+0000 7fb74bea3700 1 -- 172.21.15.7:0/4165205036 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb74406d680 con 0x7fb74414fda0 2023-03-03T16:42:14.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.849+0000 7fb7367fc700 1 -- 172.21.15.7:0/4165205036 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb74406d680 con 0x7fb74414fda0 2023-03-03T16:42:14.849 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:42:14.849 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:42:14.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 -- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7280682f0 msgr2=0x7fb72806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:14.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7280682f0 0x7fb72806a7a0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fb74406fd30 tx=0x7fb740008040 comp rx=0 tx=0).stop 2023-03-03T16:42:14.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 -- 172.21.15.7:0/4165205036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 msgr2=0x7fb74406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:14.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 0x7fb74406d960 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00cb50 tx=0x7fb73c00ce60 comp rx=0 tx=0).stop 2023-03-03T16:42:14.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 -- 172.21.15.7:0/4165205036 shutdown_connections 2023-03-03T16:42:14.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7280682f0 0x7fb72806a7a0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:14.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb74414fda0 0x7fb74406d960 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:14.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7440abe20 0x7fb7440bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:14.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 --2- 172.21.15.7:0/4165205036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb74414f9d0 0x7fb7440c2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:14.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 -- 172.21.15.7:0/4165205036 >> 172.21.15.7:0/4165205036 conn(0x7fb7440c8900 msgr2=0x7fb744151cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:42:14.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 -- 172.21.15.7:0/4165205036 shutdown_connections 2023-03-03T16:42:14.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:14.851+0000 7fb74bea3700 1 -- 172.21.15.7:0/4165205036 wait complete. 2023-03-03T16:42:14.853 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:42:14.863 INFO:tasks.mds_thrash.fs.[a]:waiting for 38 secs before thrashing 2023-03-03T16:42:18.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:18.368 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:48.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:48.441 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:42:52.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:42:53.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.002+0000 7fccec794700 1 Processor -- start 2023-03-03T16:42:53.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.002+0000 7fccec794700 1 -- start start 2023-03-03T16:42:53.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fccec794700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcce40aae30 0x7fcce40beab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:53.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fccec794700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce406e430 0x7fcce406e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:53.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fccec794700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce40c3020 con 0x7fcce40beff0 2023-03-03T16:42:53.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fccec794700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce41552d0 con 0x7fcce40aae30 2023-03-03T16:42:53.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fccec794700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce41546e0 con 0x7fcce406e430 2023-03-03T16:42:53.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fccead31700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce406e430 0x7fcce406e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:53.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fccea530700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcce40aae30 0x7fcce40beab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:53.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fcce9d2f700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fcce40beff0 0x7fcce40c44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:55784/0 (socket says 172.21.15.7:55784) 2023-03-03T16:42:53.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fccea530700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcce40aae30 0x7fcce40beab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:54056/0 (socket says 172.21.15.7:54056) 2023-03-03T16:42:53.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fcce9d2f700 1 -- 172.21.15.7:0/1448858792 learned_addr learned my addr 172.21.15.7:0/1448858792 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:42:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.003+0000 7fccead31700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce406e430 0x7fcce406e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:38126/0 (socket says 172.21.15.7:38126) 2023-03-03T16:42:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.004+0000 7fccea530700 1 -- 172.21.15.7:0/1448858792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce406e430 msgr2=0x7fcce406e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:42:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.004+0000 7fccea530700 1 --2- 172.21.15.7:0/1448858792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce406e430 0x7fcce406e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.004+0000 7fccea530700 1 -- 172.21.15.7:0/1448858792 >> v1:172.21.15.143:6789/0 conn(0x7fcce40beff0 legacy=0x7fcce40c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.004+0000 7fccea530700 1 -- 172.21.15.7:0/1448858792 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce41549a0 con 0x7fcce40aae30 2023-03-03T16:42:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.004+0000 7fcce952e700 1 -- 172.21.15.7:0/1448858792 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629357912 0 0) 0x7fcce40c3020 con 0x7fcce40beff0 2023-03-03T16:42:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.004+0000 7fccead31700 1 --2- 172.21.15.7:0/1448858792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce406e430 0x7fcce406e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:42:53.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.004+0000 7fccea530700 1 --2- 172.21.15.7:0/1448858792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcce40aae30 0x7fcce40beab0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009840 tx=0x7fcce0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c33fc5aa12b9d04 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:53.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.004+0000 7fcce952e700 1 -- 172.21.15.7:0/1448858792 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0011830 con 0x7fcce40aae30 2023-03-03T16:42:53.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.004+0000 7fcce952e700 1 -- 172.21.15.7:0/1448858792 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce0011990 con 0x7fcce40aae30 2023-03-03T16:42:53.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.004+0000 7fcce952e700 1 -- 172.21.15.7:0/1448858792 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0011d20 con 0x7fcce40aae30 2023-03-03T16:42:53.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.005+0000 7fccec794700 1 -- 172.21.15.7:0/1448858792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcce40aae30 msgr2=0x7fcce40beab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:53.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.005+0000 7fccec794700 1 --2- 172.21.15.7:0/1448858792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcce40aae30 0x7fcce40beab0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009840 tx=0x7fcce0009b50 comp rx=0 tx=0).stop 2023-03-03T16:42:53.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.005+0000 7fccec794700 1 -- 172.21.15.7:0/1448858792 shutdown_connections 2023-03-03T16:42:53.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.005+0000 7fccec794700 1 --2- 172.21.15.7:0/1448858792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce406e430 0x7fcce406e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:53.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.005+0000 7fccec794700 1 --2- 172.21.15.7:0/1448858792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcce40aae30 0x7fcce40beab0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:53.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.005+0000 7fccec794700 1 -- 172.21.15.7:0/1448858792 >> 172.21.15.7:0/1448858792 conn(0x7fcce40c8510 msgr2=0x7fcce40c8910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:42:53.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.005+0000 7fccec794700 1 -- 172.21.15.7:0/1448858792 shutdown_connections 2023-03-03T16:42:53.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.005+0000 7fccec794700 1 -- 172.21.15.7:0/1448858792 wait complete. 2023-03-03T16:42:53.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.006+0000 7fccec794700 1 Processor -- start 2023-03-03T16:42:53.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.006+0000 7fccec794700 1 -- start start 2023-03-03T16:42:53.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.006+0000 7fccec794700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcce406e430 0x7fcce41505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:53.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccec794700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce40aae30 0x7fcce414a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:53.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccec794700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcce40beff0 0x7fcce414aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:53.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccec794700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce40ac7c0 con 0x7fcce40beff0 2023-03-03T16:42:53.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccec794700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce414af80 con 0x7fcce406e430 2023-03-03T16:42:53.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccec794700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce414bc10 con 0x7fcce40aae30 2023-03-03T16:42:53.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccead31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcce40beff0 0x7fcce414aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:53.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccead31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcce40beff0 0x7fcce414aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58292/0 (socket says 172.21.15.7:58292) 2023-03-03T16:42:53.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccead31700 1 -- 172.21.15.7:0/1755716662 learned_addr learned my addr 172.21.15.7:0/1755716662 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:42:53.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccea530700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcce406e430 0x7fcce41505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:53.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fcce9d2f700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce40aae30 0x7fcce414a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:53.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccead31700 1 -- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce40aae30 msgr2=0x7fcce414a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:53.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccead31700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce40aae30 0x7fcce414a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:53.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccead31700 1 -- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcce406e430 msgr2=0x7fcce41505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:53.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccead31700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcce406e430 0x7fcce41505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:53.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccead31700 1 -- 172.21.15.7:0/1755716662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc009a70 con 0x7fcce40beff0 2023-03-03T16:42:53.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccea530700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcce406e430 0x7fcce41505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:42:53.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccead31700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcce40beff0 0x7fcce414aa40 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fccdc009f00 tx=0x7fccdc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:53.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccdaffd700 1 -- 172.21.15.7:0/1755716662 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc009d70 con 0x7fcce40beff0 2023-03-03T16:42:53.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccdaffd700 1 -- 172.21.15.7:0/1755716662 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc00f040 con 0x7fcce40beff0 2023-03-03T16:42:53.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccec794700 1 -- 172.21.15.7:0/1755716662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce00094a0 con 0x7fcce40beff0 2023-03-03T16:42:53.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccdaffd700 1 -- 172.21.15.7:0/1755716662 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc014680 con 0x7fcce40beff0 2023-03-03T16:42:53.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.007+0000 7fccec794700 1 -- 172.21.15.7:0/1755716662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce4154f60 con 0x7fcce40beff0 2023-03-03T16:42:53.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.008+0000 7fccec794700 1 -- 172.21.15.7:0/1755716662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce4154c80 con 0x7fcce40beff0 2023-03-03T16:42:53.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.009+0000 7fccdaffd700 1 -- 172.21.15.7:0/1755716662 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fccdc0147e0 con 0x7fcce40beff0 2023-03-03T16:42:53.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.009+0000 7fccdaffd700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fccc8068320 0x7fccc806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:42:53.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.010+0000 7fccdaffd700 1 -- 172.21.15.7:0/1755716662 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fccdc087090 con 0x7fcce40beff0 2023-03-03T16:42:53.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.010+0000 7fccea530700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fccc8068320 0x7fccc806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:42:53.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.010+0000 7fccea530700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fccc8068320 0x7fccc806a7d0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fcce00090b0 tx=0x7fcce0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:42:53.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.014+0000 7fccdaffd700 1 -- 172.21.15.7:0/1755716662 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fccdc053430 con 0x7fcce40beff0 2023-03-03T16:42:53.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.252+0000 7fccec794700 1 -- 172.21.15.7:0/1755716662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcce4074bb0 con 0x7fcce40beff0 2023-03-03T16:42:53.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.252+0000 7fccdaffd700 1 -- 172.21.15.7:0/1755716662 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fccdc018020 con 0x7fcce40beff0 2023-03-03T16:42:53.253 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:42:53.253 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:42:53.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 -- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fccc8068320 msgr2=0x7fccc806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:53.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fccc8068320 0x7fccc806a7d0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fcce00090b0 tx=0x7fcce0008030 comp rx=0 tx=0).stop 2023-03-03T16:42:53.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 -- 172.21.15.7:0/1755716662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcce40beff0 msgr2=0x7fcce414aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:42:53.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcce40beff0 0x7fcce414aa40 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7fccdc009f00 tx=0x7fccdc00c3a0 comp rx=0 tx=0).stop 2023-03-03T16:42:53.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 -- 172.21.15.7:0/1755716662 shutdown_connections 2023-03-03T16:42:53.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fccc8068320 0x7fccc806a7d0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:53.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcce40beff0 0x7fcce414aa40 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:53.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcce406e430 0x7fcce41505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:53.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 --2- 172.21.15.7:0/1755716662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcce40aae30 0x7fcce414a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:42:53.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 -- 172.21.15.7:0/1755716662 >> 172.21.15.7:0/1755716662 conn(0x7fcce40c8510 msgr2=0x7fcce40be4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:42:53.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 -- 172.21.15.7:0/1755716662 shutdown_connections 2023-03-03T16:42:53.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:42:53.254+0000 7fccec794700 1 -- 172.21.15.7:0/1755716662 wait complete. 2023-03-03T16:42:53.256 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:42:53.266 INFO:tasks.mds_thrash.fs.[a]:waiting for 71 secs before thrashing 2023-03-03T16:43:18.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:18.529 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:19.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:43:19.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.863+0000 7fe2fa4ba700 1 Processor -- start 2023-03-03T16:43:19.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.863+0000 7fe2fa4ba700 1 -- start start 2023-03-03T16:43:19.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.863+0000 7fe2fa4ba700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe2f40c5c60 0x7fe2f4154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:19.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.863+0000 7fe2fa4ba700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f40c7310 con 0x7fe2f40caa30 2023-03-03T16:43:19.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.863+0000 7fe2fa4ba700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f40bfd10 con 0x7fe2f40c00e0 2023-03-03T16:43:19.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.863+0000 7fe2fa4ba700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f40c60a0 con 0x7fe2f40c5c60 2023-03-03T16:43:19.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.863+0000 7fe2f3fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe2f40caa30 0x7fe2f40c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52290/0 (socket says 172.21.15.7:52290) 2023-03-03T16:43:19.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f3fff700 1 -- 172.21.15.7:0/880207380 learned_addr learned my addr 172.21.15.7:0/880207380 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:43:19.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f8a57700 1 --2- 172.21.15.7:0/880207380 >> v2:172.21.15.143:3300/0 conn(0x7fe2f40c5c60 0x7fe2f4154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:19.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053124565 0 0) 0x7fe2f40c7310 con 0x7fe2f40caa30 2023-03-03T16:43:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2dc0035b0 con 0x7fe2f40caa30 2023-03-03T16:43:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2485021693 0 0) 0x7fe2dc0035b0 con 0x7fe2f40caa30 2023-03-03T16:43:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 >> v2:172.21.15.143:3300/0 conn(0x7fe2f40c5c60 msgr2=0x7fe2f4154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:43:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 --2- 172.21.15.7:0/880207380 >> v2:172.21.15.143:3300/0 conn(0x7fe2f40c5c60 0x7fe2f4154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 >> v1:172.21.15.143:6789/0 conn(0x7fe2f40c00e0 legacy=0x7fe2f415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f40bfa30 con 0x7fe2f40caa30 2023-03-03T16:43:19.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe2e4001670 con 0x7fe2f40caa30 2023-03-03T16:43:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 >> v1:172.21.15.7:6789/0 conn(0x7fe2f40caa30 legacy=0x7fe2f40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2dc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2dc004850 0x7fe2dc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2dc005250 0x7fe2dc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc00c160 con 0x7fe2dc003a70 2023-03-03T16:43:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc00c2d0 con 0x7fe2dc004850 2023-03-03T16:43:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc00c490 con 0x7fe2dc005250 2023-03-03T16:43:19.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f3fff700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2dc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f37fe700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2dc004850 0x7fe2dc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.864+0000 7fe2f8a57700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2dc005250 0x7fe2dc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2f3fff700 1 -- 172.21.15.7:0/880207380 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2dc005250 msgr2=0x7fe2dc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:43:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2f3fff700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2dc005250 0x7fe2dc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2f3fff700 1 -- 172.21.15.7:0/880207380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2dc004850 msgr2=0x7fe2dc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:43:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2f3fff700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2dc004850 0x7fe2dc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:19.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2f3fff700 1 -- 172.21.15.7:0/880207380 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc0035b0 con 0x7fe2dc003a70 2023-03-03T16:43:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2f8a57700 1 -- 172.21.15.7:0/880207380 reap_dead start 2023-03-03T16:43:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2f3fff700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2dc003e80 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c270 tx=0x7fe2dc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e7fb96d0cd0be38 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:43:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4007c10 con 0x7fe2dc003a70 2023-03-03T16:43:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e400cd70 con 0x7fe2dc003a70 2023-03-03T16:43:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2f2ffd700 1 -- 172.21.15.7:0/880207380 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4005ba0 con 0x7fe2dc003a70 2023-03-03T16:43:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/880207380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 msgr2=0x7fe2dc003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:19.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2fa4ba700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2dc003e80 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c270 tx=0x7fe2dc005880 comp rx=0 tx=0).stop 2023-03-03T16:43:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/880207380 shutdown_connections 2023-03-03T16:43:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2fa4ba700 1 --2- 172.21.15.7:0/880207380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2dc003e80 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.865+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/880207380 >> 172.21.15.7:0/880207380 conn(0x7fe2f4066e40 msgr2=0x7fe2f4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:43:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.866+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/880207380 shutdown_connections 2023-03-03T16:43:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.866+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/880207380 wait complete. 2023-03-03T16:43:19.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.867+0000 7fe2fa4ba700 1 Processor -- start 2023-03-03T16:43:19.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.867+0000 7fe2fa4ba700 1 -- start start 2023-03-03T16:43:19.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.867+0000 7fe2fa4ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2f4157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:19.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2fa4ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f41ede90 0x7fe2f40c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:19.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2fa4ba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f414a240 0x7fe2f40c4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:19.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2fa4ba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f414daf0 con 0x7fe2dc003a70 2023-03-03T16:43:19.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2fa4ba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f414e020 con 0x7fe2f414a240 2023-03-03T16:43:19.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2fa4ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f414dd40 con 0x7fe2f41ede90 2023-03-03T16:43:19.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2f4157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:19.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2f4157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58322/0 (socket says 172.21.15.7:58322) 2023-03-03T16:43:19.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f3fff700 1 -- 172.21.15.7:0/3643270613 learned_addr learned my addr 172.21.15.7:0/3643270613 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:43:19.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f3fff700 1 -- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f41ede90 msgr2=0x7fe2f40c3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:19.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f3fff700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f41ede90 0x7fe2f40c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:19.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f3fff700 1 -- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f414a240 msgr2=0x7fe2f40c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:19.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f3fff700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f414a240 0x7fe2f40c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:19.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f3fff700 1 -- 172.21.15.7:0/3643270613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f41547f0 con 0x7fe2dc003a70 2023-03-03T16:43:19.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f3fff700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2f4157ae0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c860 tx=0x7fe2f4154400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:43:19.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f0ff9700 1 -- 172.21.15.7:0/3643270613 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4004c80 con 0x7fe2dc003a70 2023-03-03T16:43:19.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f0ff9700 1 -- 172.21.15.7:0/3643270613 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e4004e20 con 0x7fe2dc003a70 2023-03-03T16:43:19.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/3643270613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f4154510 con 0x7fe2dc003a70 2023-03-03T16:43:19.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2f0ff9700 1 -- 172.21.15.7:0/3643270613 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4007310 con 0x7fe2dc003a70 2023-03-03T16:43:19.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.868+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/3643270613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f4154ad0 con 0x7fe2dc003a70 2023-03-03T16:43:19.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.869+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/3643270613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f41edce0 con 0x7fe2dc003a70 2023-03-03T16:43:19.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.870+0000 7fe2f0ff9700 1 -- 172.21.15.7:0/3643270613 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe2e4010070 con 0x7fe2dc003a70 2023-03-03T16:43:19.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.870+0000 7fe2f0ff9700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2e00682f0 0x7fe2e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:19.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.870+0000 7fe2f0ff9700 1 -- 172.21.15.7:0/3643270613 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe2e40869c0 con 0x7fe2dc003a70 2023-03-03T16:43:19.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.870+0000 7fe2f37fe700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2e00682f0 0x7fe2e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:19.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.871+0000 7fe2f37fe700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2e00682f0 0x7fe2e006a7a0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8000c00 tx=0x7fe2e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:43:19.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:19.874+0000 7fe2f0ff9700 1 -- 172.21.15.7:0/3643270613 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2e4052d60 con 0x7fe2dc003a70 2023-03-03T16:43:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.111+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/3643270613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2f4152840 con 0x7fe2dc003a70 2023-03-03T16:43:20.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.112+0000 7fe2f0ff9700 1 -- 172.21.15.7:0/3643270613 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe2e4007d70 con 0x7fe2dc003a70 2023-03-03T16:43:20.112 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:43:20.112 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:43:20.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.113+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2e00682f0 msgr2=0x7fe2e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.113+0000 7fe2fa4ba700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2e00682f0 0x7fe2e006a7a0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8000c00 tx=0x7fe2e8009040 comp rx=0 tx=0).stop 2023-03-03T16:43:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.113+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/3643270613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 msgr2=0x7fe2f4157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.113+0000 7fe2fa4ba700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2f4157ae0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c860 tx=0x7fe2f4154400 comp rx=0 tx=0).stop 2023-03-03T16:43:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.113+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/3643270613 shutdown_connections 2023-03-03T16:43:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.113+0000 7fe2fa4ba700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2f414a240 0x7fe2f40c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:20.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.113+0000 7fe2fa4ba700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2e00682f0 0x7fe2e006a7a0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:20.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.113+0000 7fe2fa4ba700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2dc003a70 0x7fe2f4157ae0 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:20.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.113+0000 7fe2fa4ba700 1 --2- 172.21.15.7:0/3643270613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2f41ede90 0x7fe2f40c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:20.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.113+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/3643270613 >> 172.21.15.7:0/3643270613 conn(0x7fe2f4066e40 msgr2=0x7fe2f40c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:43:20.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.114+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/3643270613 shutdown_connections 2023-03-03T16:43:20.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:20.114+0000 7fe2fa4ba700 1 -- 172.21.15.7:0/3643270613 wait complete. 2023-03-03T16:43:20.115 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:43:20.126 INFO:tasks.mds_thrash.fs.[b]:waiting for 24 secs before thrashing 2023-03-03T16:43:44.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:43:44.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.266+0000 7f5799d9c700 1 Processor -- start 2023-03-03T16:43:44.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.266+0000 7f5799d9c700 1 -- start start 2023-03-03T16:43:44.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f5799d9c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57940beab0 0x7f5794155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:44.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f5799d9c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f57941562c0 0x7f579415cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:44.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f5799d9c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57941523b0 con 0x7f5794152e30 2023-03-03T16:43:44.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f5799d9c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5794072010 con 0x7f57941562c0 2023-03-03T16:43:44.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f5799d9c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57940aaaf0 con 0x7f57940beab0 2023-03-03T16:43:44.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57937fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5794152e30 0x7f57940c43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52326/0 (socket says 172.21.15.7:52326) 2023-03-03T16:43:44.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57937fe700 1 -- 172.21.15.7:0/3921247805 learned_addr learned my addr 172.21.15.7:0/3921247805 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:43:44.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f5793fff700 1 --2- 172.21.15.7:0/3921247805 >> v2:172.21.15.143:3300/0 conn(0x7f57941562c0 0x7f579415cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:44.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f5792ffd700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57940beab0 0x7f5794155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:44.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719773707 0 0) 0x7f57941523b0 con 0x7f5794152e30 2023-03-03T16:43:44.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f577c0035b0 con 0x7f5794152e30 2023-03-03T16:43:44.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2772533296 0 0) 0x7f577c0035b0 con 0x7f5794152e30 2023-03-03T16:43:44.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57940beab0 msgr2=0x7f5794155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:43:44.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57940beab0 0x7f5794155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:44.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 >> v2:172.21.15.143:3300/0 conn(0x7f57941562c0 msgr2=0x7f579415cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:43:44.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 --2- 172.21.15.7:0/3921247805 >> v2:172.21.15.143:3300/0 conn(0x7f57941562c0 0x7f579415cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:44.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57940aadd0 con 0x7f5794152e30 2023-03-03T16:43:44.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f5784001670 con 0x7f5794152e30 2023-03-03T16:43:44.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 >> v1:172.21.15.7:6789/0 conn(0x7f5794152e30 legacy=0x7f57940c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:44.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57940beab0 0x7f577c003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:44.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f577c003ff0 0x7f577c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:44.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f577c004d90 0x7f577c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:44.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5794072010 con 0x7f577c004d90 2023-03-03T16:43:44.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940aaaf0 con 0x7f577c003ff0 2023-03-03T16:43:44.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.267+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577c00bac0 con 0x7f57940beab0 2023-03-03T16:43:44.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f5793fff700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f577c004d90 0x7f577c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:44.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f57937fe700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57940beab0 0x7f577c003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:44.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f5792ffd700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f577c003ff0 0x7f577c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:44.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f5793fff700 1 -- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57940beab0 msgr2=0x7f577c003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:43:44.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f5793fff700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57940beab0 0x7f577c003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:44.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f5793fff700 1 -- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f577c003ff0 msgr2=0x7f577c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:43:44.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f5793fff700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f577c003ff0 0x7f577c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:44.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f5793fff700 1 -- 172.21.15.7:0/3921247805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c0035b0 con 0x7f577c004d90 2023-03-03T16:43:44.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f5793fff700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f577c004d90 0x7f577c005200 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f578c009f90 tx=0x7f577c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f72b0c8479f1bf6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:43:44.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c00ec30 con 0x7f577c004d90 2023-03-03T16:43:44.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c00f040 con 0x7f577c004d90 2023-03-03T16:43:44.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.268+0000 7f57927fc700 1 -- 172.21.15.7:0/3921247805 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c014480 con 0x7f577c004d90 2023-03-03T16:43:44.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.269+0000 7f5799d9c700 1 -- 172.21.15.7:0/3921247805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f577c004d90 msgr2=0x7f577c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.269+0000 7f5799d9c700 1 --2- 172.21.15.7:0/3921247805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f577c004d90 0x7f577c005200 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f578c009f90 tx=0x7f577c005880 comp rx=0 tx=0).stop 2023-03-03T16:43:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.269+0000 7f5793fff700 1 -- 172.21.15.7:0/3921247805 reap_dead start 2023-03-03T16:43:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.269+0000 7f5799d9c700 1 -- 172.21.15.7:0/3921247805 shutdown_connections 2023-03-03T16:43:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.269+0000 7f5799d9c700 1 -- 172.21.15.7:0/3921247805 >> 172.21.15.7:0/3921247805 conn(0x7f57940736f0 msgr2=0x7f5794073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:43:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.269+0000 7f5799d9c700 1 -- 172.21.15.7:0/3921247805 shutdown_connections 2023-03-03T16:43:44.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.269+0000 7f5799d9c700 1 -- 172.21.15.7:0/3921247805 wait complete. 2023-03-03T16:43:44.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.270+0000 7f5799d9c700 1 Processor -- start 2023-03-03T16:43:44.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.270+0000 7f5799d9c700 1 -- start start 2023-03-03T16:43:44.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5799d9c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57941514e0 0x7f57941eba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:44.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5799d9c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57941518b0 0x7f57940ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:44.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5799d9c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57940ab9a0 0x7f579414aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:44.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5799d9c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57941e82b0 con 0x7f57940ab9a0 2023-03-03T16:43:44.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5799d9c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940ad5b0 con 0x7f57941514e0 2023-03-03T16:43:44.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5799d9c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5794154f60 con 0x7f57941518b0 2023-03-03T16:43:44.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5793fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57940ab9a0 0x7f579414aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:44.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5793fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57940ab9a0 0x7f579414aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58362/0 (socket says 172.21.15.7:58362) 2023-03-03T16:43:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5793fff700 1 -- 172.21.15.7:0/316353318 learned_addr learned my addr 172.21.15.7:0/316353318 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:43:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5792ffd700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57941518b0 0x7f57940ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f57937fe700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57941514e0 0x7f57941eba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5793fff700 1 -- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57941518b0 msgr2=0x7f57940ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5793fff700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57941518b0 0x7f57940ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5793fff700 1 -- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57941514e0 msgr2=0x7f57941eba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5793fff700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57941514e0 0x7f57941eba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:44.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5793fff700 1 -- 172.21.15.7:0/316353318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5794153200 con 0x7f57940ab9a0 2023-03-03T16:43:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.271+0000 7f5793fff700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57940ab9a0 0x7f579414aef0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f578c00e700 tx=0x7f578c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:43:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.272+0000 7f577bfff700 1 -- 172.21.15.7:0/316353318 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c009410 con 0x7f57940ab9a0 2023-03-03T16:43:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.272+0000 7f577bfff700 1 -- 172.21.15.7:0/316353318 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c014b50 con 0x7f57940ab9a0 2023-03-03T16:43:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.272+0000 7f5799d9c700 1 -- 172.21.15.7:0/316353318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5794071c70 con 0x7f57940ab9a0 2023-03-03T16:43:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.272+0000 7f577bfff700 1 -- 172.21.15.7:0/316353318 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c003a20 con 0x7f57940ab9a0 2023-03-03T16:43:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.272+0000 7f5799d9c700 1 -- 172.21.15.7:0/316353318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5794153390 con 0x7f57940ab9a0 2023-03-03T16:43:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.273+0000 7f5799d9c700 1 -- 172.21.15.7:0/316353318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5794074bb0 con 0x7f57940ab9a0 2023-03-03T16:43:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.273+0000 7f577bfff700 1 -- 172.21.15.7:0/316353318 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f578c02e070 con 0x7f57940ab9a0 2023-03-03T16:43:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.273+0000 7f577bfff700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57800682f0 0x7f578006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:43:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.274+0000 7f577bfff700 1 -- 172.21.15.7:0/316353318 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f578c087490 con 0x7f57940ab9a0 2023-03-03T16:43:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.274+0000 7f57937fe700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57800682f0 0x7f578006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:43:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.274+0000 7f57937fe700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57800682f0 0x7f578006a7a0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f5784001050 tx=0x7f5784007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:43:44.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.278+0000 7f577bfff700 1 -- 172.21.15.7:0/316353318 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f578c0538f0 con 0x7f57940ab9a0 2023-03-03T16:43:44.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.518+0000 7f5799d9c700 1 -- 172.21.15.7:0/316353318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5794071a50 con 0x7f57940ab9a0 2023-03-03T16:43:44.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.519+0000 7f577bfff700 1 -- 172.21.15.7:0/316353318 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f578c056f10 con 0x7f57940ab9a0 2023-03-03T16:43:44.519 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:43:44.519 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:43:44.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 -- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57800682f0 msgr2=0x7f578006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:44.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57800682f0 0x7f578006a7a0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f5784001050 tx=0x7f5784007980 comp rx=0 tx=0).stop 2023-03-03T16:43:44.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 -- 172.21.15.7:0/316353318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57940ab9a0 msgr2=0x7f579414aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:43:44.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57940ab9a0 0x7f579414aef0 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f578c00e700 tx=0x7f578c009750 comp rx=0 tx=0).stop 2023-03-03T16:43:44.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 -- 172.21.15.7:0/316353318 shutdown_connections 2023-03-03T16:43:44.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57800682f0 0x7f578006a7a0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:44.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57940ab9a0 0x7f579414aef0 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:44.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57941514e0 0x7f57941eba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:44.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 --2- 172.21.15.7:0/316353318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57941518b0 0x7f57940ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:43:44.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 -- 172.21.15.7:0/316353318 >> 172.21.15.7:0/316353318 conn(0x7f57940736f0 msgr2=0x7f57940bd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:43:44.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 -- 172.21.15.7:0/316353318 shutdown_connections 2023-03-03T16:43:44.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:43:44.521+0000 7f5799d9c700 1 -- 172.21.15.7:0/316353318 wait complete. 2023-03-03T16:43:44.523 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:43:44.533 INFO:tasks.mds_thrash.fs.[b]:waiting for 40 secs before thrashing 2023-03-03T16:43:48.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:43:48.600 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:04.267 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:44:04.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.406+0000 7f0f012dd700 1 Processor -- start 2023-03-03T16:44:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.407+0000 7f0f012dd700 1 -- start start 2023-03-03T16:44:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.407+0000 7f0f012dd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0efc0caa30 0x7f0efc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.407+0000 7f0f012dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0efc0c00e0 0x7f0efc154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.407+0000 7f0f012dd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0efc0cae00 0x7f0efc0c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.407+0000 7f0f012dd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0efc0cb280 con 0x7f0efc0cae00 2023-03-03T16:44:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.407+0000 7f0f012dd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0efc0bfd10 con 0x7f0efc0caa30 2023-03-03T16:44:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.407+0000 7f0f012dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0bfa30 con 0x7f0efc0c00e0 2023-03-03T16:44:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.407+0000 7f0efb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0efc0cae00 0x7f0efc0c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efb7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0efc0cae00 0x7f0efc0c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58388/0 (socket says 172.21.15.7:58388) 2023-03-03T16:44:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efb7fe700 1 -- 172.21.15.7:0/3545228527 learned_addr learned my addr 172.21.15.7:0/3545228527 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:44:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efaffd700 1 --2- 172.21.15.7:0/3545228527 >> v2:172.21.15.143:3300/0 conn(0x7f0efc0caa30 0x7f0efc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efa7fc700 1 --2- 172.21.15.7:0/3545228527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0efc0c00e0 0x7f0efc154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efb7fe700 1 -- 172.21.15.7:0/3545228527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0efc0c00e0 msgr2=0x7f0efc154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:44:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efb7fe700 1 --2- 172.21.15.7:0/3545228527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0efc0c00e0 0x7f0efc154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efb7fe700 1 -- 172.21.15.7:0/3545228527 >> v2:172.21.15.143:3300/0 conn(0x7f0efc0caa30 msgr2=0x7f0efc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:44:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efb7fe700 1 --2- 172.21.15.7:0/3545228527 >> v2:172.21.15.143:3300/0 conn(0x7f0efc0caa30 0x7f0efc0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efb7fe700 1 -- 172.21.15.7:0/3545228527 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc154510 con 0x7f0efc0cae00 2023-03-03T16:44:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efaffd700 1 --2- 172.21.15.7:0/3545228527 >> v2:172.21.15.143:3300/0 conn(0x7f0efc0caa30 0x7f0efc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:44:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0efb7fe700 1 --2- 172.21.15.7:0/3545228527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0cae00 0x7f0efc0c6c00 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0c81b0 tx=0x7f0ef400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f3fcc85ee4713c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:04.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0ef9ffb700 1 -- 172.21.15.7:0/3545228527 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef400e070 con 0x7f0efc0cae00 2023-03-03T16:44:04.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0ef9ffb700 1 -- 172.21.15.7:0/3545228527 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef400b9c0 con 0x7f0efc0cae00 2023-03-03T16:44:04.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.408+0000 7f0ef9ffb700 1 -- 172.21.15.7:0/3545228527 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef400bd70 con 0x7f0efc0cae00 2023-03-03T16:44:04.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.409+0000 7f0f012dd700 1 -- 172.21.15.7:0/3545228527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0cae00 msgr2=0x7f0efc0c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:04.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.409+0000 7f0f012dd700 1 --2- 172.21.15.7:0/3545228527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0cae00 0x7f0efc0c6c00 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0c81b0 tx=0x7f0ef400b330 comp rx=0 tx=0).stop 2023-03-03T16:44:04.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.409+0000 7f0f012dd700 1 -- 172.21.15.7:0/3545228527 shutdown_connections 2023-03-03T16:44:04.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.409+0000 7f0f012dd700 1 --2- 172.21.15.7:0/3545228527 >> v2:172.21.15.143:3300/0 conn(0x7f0efc0caa30 0x7f0efc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.409+0000 7f0f012dd700 1 --2- 172.21.15.7:0/3545228527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0cae00 0x7f0efc0c6c00 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.409+0000 7f0f012dd700 1 --2- 172.21.15.7:0/3545228527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0efc0c00e0 0x7f0efc154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.409+0000 7f0f012dd700 1 -- 172.21.15.7:0/3545228527 >> 172.21.15.7:0/3545228527 conn(0x7f0efc066e40 msgr2=0x7f0efc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:44:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.409+0000 7f0f012dd700 1 -- 172.21.15.7:0/3545228527 shutdown_connections 2023-03-03T16:44:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.409+0000 7f0f012dd700 1 -- 172.21.15.7:0/3545228527 wait complete. 2023-03-03T16:44:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.410+0000 7f0f012dd700 1 Processor -- start 2023-03-03T16:44:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.410+0000 7f0f012dd700 1 -- start start 2023-03-03T16:44:04.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0f012dd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0c00e0 0x7f0efc157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0f012dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0efc0caa30 0x7f0efc157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0f012dd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0efc0cae00 0x7f0efc156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0f012dd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc14d980 con 0x7f0efc0c00e0 2023-03-03T16:44:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0f012dd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc1dea20 con 0x7f0efc0caa30 2023-03-03T16:44:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0f012dd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0c2100 con 0x7f0efc0cae00 2023-03-03T16:44:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0efaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0c00e0 0x7f0efc157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0efaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0c00e0 0x7f0efc157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58390/0 (socket says 172.21.15.7:58390) 2023-03-03T16:44:04.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0efaffd700 1 -- 172.21.15.7:0/2431041887 learned_addr learned my addr 172.21.15.7:0/2431041887 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:44:04.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0efa7fc700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0efc0caa30 0x7f0efc157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:04.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0efb7fe700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0efc0cae00 0x7f0efc156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:04.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0efaffd700 1 -- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0efc0cae00 msgr2=0x7f0efc156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:04.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0efaffd700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0efc0cae00 0x7f0efc156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0efaffd700 1 -- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0efc0caa30 msgr2=0x7f0efc157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:04.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0efaffd700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0efc0caa30 0x7f0efc157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.411+0000 7f0efaffd700 1 -- 172.21.15.7:0/2431041887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef4009b30 con 0x7f0efc0c00e0 2023-03-03T16:44:04.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.412+0000 7f0efaffd700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0c00e0 0x7f0efc157780 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00e810 tx=0x7f0eec00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:04.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.412+0000 7f0ee37fe700 1 -- 172.21.15.7:0/2431041887 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec013610 con 0x7f0efc0c00e0 2023-03-03T16:44:04.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.412+0000 7f0ee37fe700 1 -- 172.21.15.7:0/2431041887 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eec013770 con 0x7f0efc0c00e0 2023-03-03T16:44:04.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.412+0000 7f0f012dd700 1 -- 172.21.15.7:0/2431041887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0efc154ad0 con 0x7f0efc0c00e0 2023-03-03T16:44:04.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.412+0000 7f0ee37fe700 1 -- 172.21.15.7:0/2431041887 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec013b20 con 0x7f0efc0c00e0 2023-03-03T16:44:04.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.412+0000 7f0f012dd700 1 -- 172.21.15.7:0/2431041887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0efc1547f0 con 0x7f0efc0c00e0 2023-03-03T16:44:04.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.413+0000 7f0f012dd700 1 -- 172.21.15.7:0/2431041887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0efc0bf650 con 0x7f0efc0c00e0 2023-03-03T16:44:04.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.413+0000 7f0ee37fe700 1 -- 172.21.15.7:0/2431041887 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0eec023070 con 0x7f0efc0c00e0 2023-03-03T16:44:04.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.414+0000 7f0ee37fe700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 0x7f0ee806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:04.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.414+0000 7f0ee37fe700 1 -- 172.21.15.7:0/2431041887 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0eec085c40 con 0x7f0efc0c00e0 2023-03-03T16:44:04.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.414+0000 7f0efa7fc700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 0x7f0ee806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:04.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.415+0000 7f0efa7fc700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 0x7f0ee806a7a0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f0efc1edbf0 tx=0x7f0ef0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:04.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.418+0000 7f0ee37fe700 1 -- 172.21.15.7:0/2431041887 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0eec051fe0 con 0x7f0efc0c00e0 2023-03-03T16:44:04.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.658+0000 7f0f012dd700 1 -- 172.21.15.7:0/2431041887 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0efc074bb0 con 0x7f0efc0c00e0 2023-03-03T16:44:04.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.659+0000 7f0ee37fe700 1 -- 172.21.15.7:0/2431041887 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0eec055600 con 0x7f0efc0c00e0 2023-03-03T16:44:04.659 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:44:04.659 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:44:04.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 -- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 msgr2=0x7f0ee806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:04.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 0x7f0ee806a7a0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f0efc1edbf0 tx=0x7f0ef0008040 comp rx=0 tx=0).stop 2023-03-03T16:44:04.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 -- 172.21.15.7:0/2431041887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0c00e0 msgr2=0x7f0efc157780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:04.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0c00e0 0x7f0efc157780 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00e810 tx=0x7f0eec00eb20 comp rx=0 tx=0).stop 2023-03-03T16:44:04.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 -- 172.21.15.7:0/2431041887 shutdown_connections 2023-03-03T16:44:04.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0efc0cae00 0x7f0efc156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 0x7f0ee806a7a0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0efc0c00e0 0x7f0efc157780 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 --2- 172.21.15.7:0/2431041887 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0efc0caa30 0x7f0efc157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:04.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 -- 172.21.15.7:0/2431041887 >> 172.21.15.7:0/2431041887 conn(0x7f0efc066e40 msgr2=0x7f0efc1512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:44:04.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 -- 172.21.15.7:0/2431041887 shutdown_connections 2023-03-03T16:44:04.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:04.661+0000 7f0f012dd700 1 -- 172.21.15.7:0/2431041887 wait complete. 2023-03-03T16:44:04.663 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:44:04.673 INFO:tasks.mds_thrash.fs.[a]:waiting for 86 secs before thrashing 2023-03-03T16:44:18.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:18.663 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:24.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:44:24.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d9830700 1 Processor -- start 2023-03-03T16:44:24.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d9830700 1 -- start start 2023-03-03T16:44:24.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d9830700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f48d40caa30 0x7f48d40c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:24.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d9830700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f48d4154ff0 0x7f48d40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:24.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d9830700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d40c7310 con 0x7f48d40c00e0 2023-03-03T16:44:24.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d9830700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d4155430 con 0x7f48d4154ff0 2023-03-03T16:44:24.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d9830700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d40bfd10 con 0x7f48d40caa30 2023-03-03T16:44:24.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f48d4154ff0 0x7f48d40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:24.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d27fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f48d40c00e0 0x7f48d40c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52382/0 (socket says 172.21.15.7:52382) 2023-03-03T16:44:24.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d27fc700 1 -- 172.21.15.7:0/111061288 learned_addr learned my addr 172.21.15.7:0/111061288 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:44:24.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d2ffd700 1 --2- 172.21.15.7:0/111061288 >> v2:172.21.15.143:3300/0 conn(0x7f48d40caa30 0x7f48d40c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:24.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d37fe700 1 -- 172.21.15.7:0/111061288 >> v2:172.21.15.143:3300/0 conn(0x7f48d40caa30 msgr2=0x7f48d40c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:44:24.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d37fe700 1 --2- 172.21.15.7:0/111061288 >> v2:172.21.15.143:3300/0 conn(0x7f48d40caa30 0x7f48d40c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:24.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d37fe700 1 -- 172.21.15.7:0/111061288 >> v1:172.21.15.7:6789/0 conn(0x7f48d40c00e0 legacy=0x7f48d40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:24.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d37fe700 1 -- 172.21.15.7:0/111061288 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d40bfa30 con 0x7f48d4154ff0 2023-03-03T16:44:24.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d1ffb700 1 -- 172.21.15.7:0/111061288 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163729835 0 0) 0x7f48d40c7310 con 0x7f48d40c00e0 2023-03-03T16:44:24.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d2ffd700 1 --2- 172.21.15.7:0/111061288 >> v2:172.21.15.143:3300/0 conn(0x7f48d40caa30 0x7f48d40c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:44:24.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.674+0000 7f48d37fe700 1 --2- 172.21.15.7:0/111061288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 0x7f48d40cae00 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f48d40c81b0 tx=0x7f48cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fb0fa58194db78 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:24.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d1ffb700 1 -- 172.21.15.7:0/111061288 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc00e070 con 0x7f48d4154ff0 2023-03-03T16:44:24.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d1ffb700 1 -- 172.21.15.7:0/111061288 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc00bbd0 con 0x7f48d4154ff0 2023-03-03T16:44:24.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d1ffb700 1 -- 172.21.15.7:0/111061288 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc012430 con 0x7f48d4154ff0 2023-03-03T16:44:24.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d9830700 1 -- 172.21.15.7:0/111061288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 msgr2=0x7f48d40cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:24.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d9830700 1 --2- 172.21.15.7:0/111061288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 0x7f48d40cae00 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7f48d40c81b0 tx=0x7f48cc00b330 comp rx=0 tx=0).stop 2023-03-03T16:44:24.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d9830700 1 -- 172.21.15.7:0/111061288 shutdown_connections 2023-03-03T16:44:24.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d9830700 1 --2- 172.21.15.7:0/111061288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 0x7f48d40cae00 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:24.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d9830700 1 --2- 172.21.15.7:0/111061288 >> v2:172.21.15.143:3300/0 conn(0x7f48d40caa30 0x7f48d40c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:24.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d9830700 1 -- 172.21.15.7:0/111061288 >> 172.21.15.7:0/111061288 conn(0x7f48d4066e40 msgr2=0x7f48d4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:44:24.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d9830700 1 -- 172.21.15.7:0/111061288 shutdown_connections 2023-03-03T16:44:24.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.675+0000 7f48d9830700 1 -- 172.21.15.7:0/111061288 wait complete. 2023-03-03T16:44:24.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.676+0000 7f48d9830700 1 Processor -- start 2023-03-03T16:44:24.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.677+0000 7f48d9830700 1 -- start start 2023-03-03T16:44:24.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.677+0000 7f48d9830700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48d40c00e0 0x7f48d4158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:24.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.677+0000 7f48d9830700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48d40caa30 0x7f48d40c2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:24.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.677+0000 7f48d9830700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 0x7f48d414d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:24.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.677+0000 7f48d9830700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d4158850 con 0x7f48d4154ff0 2023-03-03T16:44:24.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.677+0000 7f48d9830700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d4156160 con 0x7f48d40caa30 2023-03-03T16:44:24.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.677+0000 7f48d9830700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d4156440 con 0x7f48d40c00e0 2023-03-03T16:44:24.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.677+0000 7f48d37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 0x7f48d414d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:24.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.677+0000 7f48d37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 0x7f48d414d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58412/0 (socket says 172.21.15.7:58412) 2023-03-03T16:44:24.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.677+0000 7f48d37fe700 1 -- 172.21.15.7:0/1126588749 learned_addr learned my addr 172.21.15.7:0/1126588749 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:44:24.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d27fc700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48d40caa30 0x7f48d40c2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:24.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d2ffd700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48d40c00e0 0x7f48d4158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:24.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d37fe700 1 -- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48d40c00e0 msgr2=0x7f48d4158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:24.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d37fe700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48d40c00e0 0x7f48d4158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:24.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d37fe700 1 -- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48d40caa30 msgr2=0x7f48d40c2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:24.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d37fe700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48d40caa30 0x7f48d40c2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:24.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d37fe700 1 -- 172.21.15.7:0/1126588749 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc009a70 con 0x7f48d4154ff0 2023-03-03T16:44:24.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d2ffd700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48d40c00e0 0x7f48d4158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:44:24.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d37fe700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 0x7f48d414d210 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00b370 tx=0x7f48cc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:24.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48bbfff700 1 -- 172.21.15.7:0/1126588749 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc012430 con 0x7f48d4154ff0 2023-03-03T16:44:24.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48bbfff700 1 -- 172.21.15.7:0/1126588749 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc020070 con 0x7f48d4154ff0 2023-03-03T16:44:24.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d9830700 1 -- 172.21.15.7:0/1126588749 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d41547f0 con 0x7f48d4154ff0 2023-03-03T16:44:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48bbfff700 1 -- 172.21.15.7:0/1126588749 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc004280 con 0x7f48d4154ff0 2023-03-03T16:44:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.678+0000 7f48d9830700 1 -- 172.21.15.7:0/1126588749 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d4154510 con 0x7f48d4154ff0 2023-03-03T16:44:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.679+0000 7f48d9830700 1 -- 172.21.15.7:0/1126588749 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d4154ad0 con 0x7f48d4154ff0 2023-03-03T16:44:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.679+0000 7f48bbfff700 1 -- 172.21.15.7:0/1126588749 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f48cc0043e0 con 0x7f48d4154ff0 2023-03-03T16:44:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.680+0000 7f48bbfff700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48c00682f0 0x7f48c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:24.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.680+0000 7f48bbfff700 1 -- 172.21.15.7:0/1126588749 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f48cc022070 con 0x7f48d4154ff0 2023-03-03T16:44:24.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.680+0000 7f48d2ffd700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48c00682f0 0x7f48c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:24.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.681+0000 7f48d2ffd700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48c00682f0 0x7f48c006a7a0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f48c80089b0 tx=0x7f48c8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:24.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.684+0000 7f48bbfff700 1 -- 172.21.15.7:0/1126588749 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f48cc054800 con 0x7f48d4154ff0 2023-03-03T16:44:24.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.923+0000 7f48d9830700 1 -- 172.21.15.7:0/1126588749 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48d4074bb0 con 0x7f48d4154ff0 2023-03-03T16:44:24.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.924+0000 7f48bbfff700 1 -- 172.21.15.7:0/1126588749 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f48cc057e20 con 0x7f48d4154ff0 2023-03-03T16:44:24.925 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:44:24.925 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:44:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 -- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48c00682f0 msgr2=0x7f48c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48c00682f0 0x7f48c006a7a0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f48c80089b0 tx=0x7f48c8008040 comp rx=0 tx=0).stop 2023-03-03T16:44:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 -- 172.21.15.7:0/1126588749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 msgr2=0x7f48d414d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 0x7f48d414d210 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f48cc00b370 tx=0x7f48cc004ad0 comp rx=0 tx=0).stop 2023-03-03T16:44:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 -- 172.21.15.7:0/1126588749 shutdown_connections 2023-03-03T16:44:24.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f48c00682f0 0x7f48c006a7a0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f48d4154ff0 0x7f48d414d210 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f48d40c00e0 0x7f48d4158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 --2- 172.21.15.7:0/1126588749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48d40caa30 0x7f48d40c2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 -- 172.21.15.7:0/1126588749 >> 172.21.15.7:0/1126588749 conn(0x7f48d4066e40 msgr2=0x7f48d406b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:44:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 -- 172.21.15.7:0/1126588749 shutdown_connections 2023-03-03T16:44:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:24.926+0000 7f48d9830700 1 -- 172.21.15.7:0/1126588749 wait complete. 2023-03-03T16:44:24.929 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:44:24.939 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before thrashing 2023-03-03T16:44:30.939 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:44:31.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.078+0000 7fc42d8ef700 1 Processor -- start 2023-03-03T16:44:31.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.078+0000 7fc42d8ef700 1 -- start start 2023-03-03T16:44:31.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.079+0000 7fc42d8ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 0x7fc428156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:31.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.079+0000 7fc42d8ef700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc4280abe20 0x7fc4281529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:31.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.079+0000 7fc42d8ef700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280c2f30 con 0x7fc42814f9d0 2023-03-03T16:44:31.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.079+0000 7fc42d8ef700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4280c31d0 con 0x7fc4280abe20 2023-03-03T16:44:31.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.079+0000 7fc42d8ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4281524a0 con 0x7fc42814fda0 2023-03-03T16:44:31.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc426ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fc42814f9d0 0x7fc428071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:55922/0 (socket says 172.21.15.7:55922) 2023-03-03T16:44:31.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc426ffd700 1 -- 172.21.15.7:0/279141115 learned_addr learned my addr 172.21.15.7:0/279141115 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:44:31.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc4277fe700 1 --2- 172.21.15.7:0/279141115 >> v2:172.21.15.143:3300/0 conn(0x7fc4280abe20 0x7fc4281529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:31.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc4267fc700 1 --2- 172.21.15.7:0/279141115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 0x7fc428156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:31.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc4277fe700 1 -- 172.21.15.7:0/279141115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 msgr2=0x7fc428156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:44:31.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc4277fe700 1 --2- 172.21.15.7:0/279141115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 0x7fc428156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:31.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc4277fe700 1 -- 172.21.15.7:0/279141115 >> v1:172.21.15.143:6789/0 conn(0x7fc42814f9d0 legacy=0x7fc428071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:31.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc4277fe700 1 -- 172.21.15.7:0/279141115 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4280c34b0 con 0x7fc4280abe20 2023-03-03T16:44:31.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc425ffb700 1 -- 172.21.15.7:0/279141115 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085290324 0 0) 0x7fc4280c2f30 con 0x7fc42814f9d0 2023-03-03T16:44:31.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc4267fc700 1 --2- 172.21.15.7:0/279141115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 0x7fc428156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:44:31.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc4277fe700 1 --2- 172.21.15.7:0/279141115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4280abe20 0x7fc4281529c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fc420009d90 tx=0x7fc42000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55390645d0b30040 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:31.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc425ffb700 1 -- 172.21.15.7:0/279141115 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42000e070 con 0x7fc4280abe20 2023-03-03T16:44:31.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc425ffb700 1 -- 172.21.15.7:0/279141115 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42000bb20 con 0x7fc4280abe20 2023-03-03T16:44:31.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.080+0000 7fc425ffb700 1 -- 172.21.15.7:0/279141115 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420012430 con 0x7fc4280abe20 2023-03-03T16:44:31.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.081+0000 7fc42d8ef700 1 -- 172.21.15.7:0/279141115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4280abe20 msgr2=0x7fc4281529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:31.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.081+0000 7fc42d8ef700 1 --2- 172.21.15.7:0/279141115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4280abe20 0x7fc4281529c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fc420009d90 tx=0x7fc42000b330 comp rx=0 tx=0).stop 2023-03-03T16:44:31.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.081+0000 7fc42d8ef700 1 -- 172.21.15.7:0/279141115 shutdown_connections 2023-03-03T16:44:31.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.081+0000 7fc42d8ef700 1 --2- 172.21.15.7:0/279141115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4280abe20 0x7fc4281529c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:31.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.081+0000 7fc42d8ef700 1 --2- 172.21.15.7:0/279141115 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 0x7fc428156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:31.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.081+0000 7fc42d8ef700 1 -- 172.21.15.7:0/279141115 >> 172.21.15.7:0/279141115 conn(0x7fc4280c8900 msgr2=0x7fc4280c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:44:31.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.081+0000 7fc42d8ef700 1 -- 172.21.15.7:0/279141115 shutdown_connections 2023-03-03T16:44:31.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.081+0000 7fc42d8ef700 1 -- 172.21.15.7:0/279141115 wait complete. 2023-03-03T16:44:31.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.082+0000 7fc42d8ef700 1 Processor -- start 2023-03-03T16:44:31.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.082+0000 7fc42d8ef700 1 -- start start 2023-03-03T16:44:31.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc42d8ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4280abe20 0x7fc4281549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:31.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc42d8ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc42814f9d0 0x7fc42814abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:31.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc42d8ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 0x7fc42814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:31.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc42d8ef700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280bdc40 con 0x7fc42814f9d0 2023-03-03T16:44:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc42d8ef700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42806c8d0 con 0x7fc4280abe20 2023-03-03T16:44:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc42d8ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280bd860 con 0x7fc42814fda0 2023-03-03T16:44:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc4267fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc42814f9d0 0x7fc42814abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc4267fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc42814f9d0 0x7fc42814abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58430/0 (socket says 172.21.15.7:58430) 2023-03-03T16:44:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc4267fc700 1 -- 172.21.15.7:0/771011610 learned_addr learned my addr 172.21.15.7:0/771011610 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:44:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc426ffd700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4280abe20 0x7fc4281549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:31.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.083+0000 7fc4277fe700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 0x7fc42814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc4267fc700 1 -- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 msgr2=0x7fc42814b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc4267fc700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 0x7fc42814b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc4267fc700 1 -- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4280abe20 msgr2=0x7fc4281549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc4267fc700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4280abe20 0x7fc4281549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc4267fc700 1 -- 172.21.15.7:0/771011610 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0094a0 con 0x7fc42814f9d0 2023-03-03T16:44:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc4267fc700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc42814f9d0 0x7fc42814abb0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00ec10 tx=0x7fc41c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc40ffff700 1 -- 172.21.15.7:0/771011610 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c0097a0 con 0x7fc42814f9d0 2023-03-03T16:44:31.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc40ffff700 1 -- 172.21.15.7:0/771011610 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c00c850 con 0x7fc42814f9d0 2023-03-03T16:44:31.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc42d8ef700 1 -- 172.21.15.7:0/771011610 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc420009a70 con 0x7fc42814f9d0 2023-03-03T16:44:31.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc40ffff700 1 -- 172.21.15.7:0/771011610 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c00cbe0 con 0x7fc42814f9d0 2023-03-03T16:44:31.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.084+0000 7fc42d8ef700 1 -- 172.21.15.7:0/771011610 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4280ac350 con 0x7fc42814f9d0 2023-03-03T16:44:31.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.085+0000 7fc42d8ef700 1 -- 172.21.15.7:0/771011610 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4280ab630 con 0x7fc42814f9d0 2023-03-03T16:44:31.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.085+0000 7fc40ffff700 1 -- 172.21.15.7:0/771011610 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc41c019030 con 0x7fc42814f9d0 2023-03-03T16:44:31.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.086+0000 7fc40ffff700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4140682f0 0x7fc41406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:44:31.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.086+0000 7fc40ffff700 1 -- 172.21.15.7:0/771011610 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc41c085d60 con 0x7fc42814f9d0 2023-03-03T16:44:31.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.086+0000 7fc426ffd700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4140682f0 0x7fc41406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:44:31.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.087+0000 7fc426ffd700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4140682f0 0x7fc41406a7a0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fc418007fd0 tx=0x7fc418007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:44:31.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.090+0000 7fc40ffff700 1 -- 172.21.15.7:0/771011610 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc41c0520a0 con 0x7fc42814f9d0 2023-03-03T16:44:31.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.333+0000 7fc42d8ef700 1 -- 172.21.15.7:0/771011610 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc428071fa0 con 0x7fc42814f9d0 2023-03-03T16:44:31.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.333+0000 7fc40ffff700 1 -- 172.21.15.7:0/771011610 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc41c021020 con 0x7fc42814f9d0 2023-03-03T16:44:31.333 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:44:31.334 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:44:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 -- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4140682f0 msgr2=0x7fc41406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4140682f0 0x7fc41406a7a0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fc418007fd0 tx=0x7fc418007de0 comp rx=0 tx=0).stop 2023-03-03T16:44:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 -- 172.21.15.7:0/771011610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc42814f9d0 msgr2=0x7fc42814abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:44:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc42814f9d0 0x7fc42814abb0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fc41c00ec10 tx=0x7fc41c00ef20 comp rx=0 tx=0).stop 2023-03-03T16:44:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 -- 172.21.15.7:0/771011610 shutdown_connections 2023-03-03T16:44:31.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc42814fda0 0x7fc42814b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4280abe20 0x7fc4281549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4140682f0 0x7fc41406a7a0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 --2- 172.21.15.7:0/771011610 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc42814f9d0 0x7fc42814abb0 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:44:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 -- 172.21.15.7:0/771011610 >> 172.21.15.7:0/771011610 conn(0x7fc4280c8900 msgr2=0x7fc4280c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:44:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 -- 172.21.15.7:0/771011610 shutdown_connections 2023-03-03T16:44:31.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:44:31.336+0000 7fc42d8ef700 1 -- 172.21.15.7:0/771011610 wait complete. 2023-03-03T16:44:31.338 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:44:31.348 INFO:tasks.mds_thrash.fs.[b]:waiting for 61 secs before thrashing 2023-03-03T16:44:48.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:44:48.735 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:18.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:18.803 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:30.673 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:45:30.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.810+0000 7fd5fc0a4700 1 Processor -- start 2023-03-03T16:45:30.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.810+0000 7fd5fc0a4700 1 -- start start 2023-03-03T16:45:30.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.811+0000 7fd5fc0a4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd5f414e900 0x7fd5f415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:30.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.811+0000 7fd5fc0a4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f41510d0 con 0x7fd5f4071e80 2023-03-03T16:45:30.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.811+0000 7fd5fc0a4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f40c3740 con 0x7fd5f4072250 2023-03-03T16:45:30.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.811+0000 7fd5fc0a4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f4070670 con 0x7fd5f414e900 2023-03-03T16:45:30.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.811+0000 7fd5fa641700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd5f414e900 0x7fd5f415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:30.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.811+0000 7fd5f9e40700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd5f4071e80 0x7fd5f4151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52448/0 (socket says 172.21.15.7:52448) 2023-03-03T16:45:30.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.811+0000 7fd5f9e40700 1 -- 172.21.15.7:0/2874815704 learned_addr learned my addr 172.21.15.7:0/2874815704 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:45:30.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5fa641700 1 -- 172.21.15.7:0/2874815704 >> v1:172.21.15.143:6789/0 conn(0x7fd5f4072250 legacy=0x7fd5f414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:30.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5fa641700 1 -- 172.21.15.7:0/2874815704 >> v1:172.21.15.7:6789/0 conn(0x7fd5f4071e80 legacy=0x7fd5f4151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:30.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5fa641700 1 -- 172.21.15.7:0/2874815704 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f40c39e0 con 0x7fd5f414e900 2023-03-03T16:45:30.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5f8e3e700 1 -- 172.21.15.7:0/2874815704 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130243370 0 0) 0x7fd5f41510d0 con 0x7fd5f4071e80 2023-03-03T16:45:30.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5fa641700 1 --2- 172.21.15.7:0/2874815704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f414e900 0x7fd5f415ca60 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009d90 tx=0x7fd5ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b966ca33313d22f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:30.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5f8e3e700 1 -- 172.21.15.7:0/2874815704 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec00e070 con 0x7fd5f414e900 2023-03-03T16:45:30.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5f8e3e700 1 -- 172.21.15.7:0/2874815704 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ec00bbd0 con 0x7fd5f414e900 2023-03-03T16:45:30.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5f8e3e700 1 -- 172.21.15.7:0/2874815704 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec012430 con 0x7fd5f414e900 2023-03-03T16:45:30.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/2874815704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f414e900 msgr2=0x7fd5f415ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:30.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5fc0a4700 1 --2- 172.21.15.7:0/2874815704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f414e900 0x7fd5f415ca60 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009d90 tx=0x7fd5ec00b330 comp rx=0 tx=0).stop 2023-03-03T16:45:30.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/2874815704 shutdown_connections 2023-03-03T16:45:30.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.812+0000 7fd5fc0a4700 1 --2- 172.21.15.7:0/2874815704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f414e900 0x7fd5f415ca60 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:30.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.813+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/2874815704 >> 172.21.15.7:0/2874815704 conn(0x7fd5f40a94d0 msgr2=0x7fd5f40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:45:30.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.813+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/2874815704 shutdown_connections 2023-03-03T16:45:30.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.813+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/2874815704 wait complete. 2023-03-03T16:45:30.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.814+0000 7fd5fc0a4700 1 Processor -- start 2023-03-03T16:45:30.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.814+0000 7fd5fc0a4700 1 -- start start 2023-03-03T16:45:30.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.815+0000 7fd5fc0a4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f4071e80 0x7fd5f40c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:30.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.815+0000 7fd5fc0a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4072250 0x7fd5f41e6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:30.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.815+0000 7fd5fc0a4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f414e900 0x7fd5f41e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:30.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.815+0000 7fd5fc0a4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f41e7720 con 0x7fd5f4072250 2023-03-03T16:45:30.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.815+0000 7fd5fc0a4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f41e9a00 con 0x7fd5f4071e80 2023-03-03T16:45:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.815+0000 7fd5fc0a4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f41ed400 con 0x7fd5f414e900 2023-03-03T16:45:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.815+0000 7fd5f963f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4072250 0x7fd5f41e6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.815+0000 7fd5f963f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4072250 0x7fd5f41e6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58476/0 (socket says 172.21.15.7:58476) 2023-03-03T16:45:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.815+0000 7fd5f963f700 1 -- 172.21.15.7:0/3781827428 learned_addr learned my addr 172.21.15.7:0/3781827428 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:45:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5fa641700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f414e900 0x7fd5f41e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5f9e40700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f4071e80 0x7fd5f40c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:30.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5f963f700 1 -- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f414e900 msgr2=0x7fd5f41e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5f963f700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f414e900 0x7fd5f41e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5f963f700 1 -- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f4071e80 msgr2=0x7fd5f40c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5f963f700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f4071e80 0x7fd5f40c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5f963f700 1 -- 172.21.15.7:0/3781827428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec009a70 con 0x7fd5f4072250 2023-03-03T16:45:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5f963f700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4072250 0x7fd5f41e6ca0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400c4b0 tx=0x7fd5e400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5ea7fc700 1 -- 172.21.15.7:0/3781827428 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e40052d0 con 0x7fd5f4072250 2023-03-03T16:45:30.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5ea7fc700 1 -- 172.21.15.7:0/3781827428 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e4005430 con 0x7fd5f4072250 2023-03-03T16:45:30.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/3781827428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f4151f30 con 0x7fd5f4072250 2023-03-03T16:45:30.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5ea7fc700 1 -- 172.21.15.7:0/3781827428 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e40038c0 con 0x7fd5f4072250 2023-03-03T16:45:30.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.816+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/3781827428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f4151300 con 0x7fd5f4072250 2023-03-03T16:45:30.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.817+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/3781827428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f406c9a0 con 0x7fd5f4072250 2023-03-03T16:45:30.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.817+0000 7fd5ea7fc700 1 -- 172.21.15.7:0/3781827428 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd5e4019030 con 0x7fd5f4072250 2023-03-03T16:45:30.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.818+0000 7fd5ea7fc700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 0x7fd5d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:30.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.818+0000 7fd5ea7fc700 1 -- 172.21.15.7:0/3781827428 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd5e40854f0 con 0x7fd5f4072250 2023-03-03T16:45:30.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.818+0000 7fd5f9e40700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 0x7fd5d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:30.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.819+0000 7fd5f9e40700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 0x7fd5d806a7a0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0007fd0 tx=0x7fd5f0007c20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:30.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:30.822+0000 7fd5ea7fc700 1 -- 172.21.15.7:0/3781827428 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd5e4051890 con 0x7fd5f4072250 2023-03-03T16:45:31.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.063+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/3781827428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5f4074bb0 con 0x7fd5f4072250 2023-03-03T16:45:31.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.063+0000 7fd5ea7fc700 1 -- 172.21.15.7:0/3781827428 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd5e4054eb0 con 0x7fd5f4072250 2023-03-03T16:45:31.065 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:45:31.065 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:45:31.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.065+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 msgr2=0x7fd5d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:31.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.065+0000 7fd5fc0a4700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 0x7fd5d806a7a0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0007fd0 tx=0x7fd5f0007c20 comp rx=0 tx=0).stop 2023-03-03T16:45:31.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.065+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/3781827428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4072250 msgr2=0x7fd5f41e6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:31.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.065+0000 7fd5fc0a4700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4072250 0x7fd5f41e6ca0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400c4b0 tx=0x7fd5e400c870 comp rx=0 tx=0).stop 2023-03-03T16:45:31.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.065+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/3781827428 shutdown_connections 2023-03-03T16:45:31.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.065+0000 7fd5fc0a4700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f414e900 0x7fd5f41e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:31.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.065+0000 7fd5fc0a4700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5f4071e80 0x7fd5f40c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:31.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.066+0000 7fd5fc0a4700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 0x7fd5d806a7a0 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:31.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.066+0000 7fd5fc0a4700 1 --2- 172.21.15.7:0/3781827428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f4072250 0x7fd5f41e6ca0 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:31.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.066+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/3781827428 >> 172.21.15.7:0/3781827428 conn(0x7fd5f40a94d0 msgr2=0x7fd5f414d690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:45:31.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.066+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/3781827428 shutdown_connections 2023-03-03T16:45:31.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:31.066+0000 7fd5fc0a4700 1 -- 172.21.15.7:0/3781827428 wait complete. 2023-03-03T16:45:31.068 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:45:31.078 INFO:tasks.mds_thrash.fs.[a]:waiting for 119 secs before thrashing 2023-03-03T16:45:32.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734bc2e700 1 Processor -- start 2023-03-03T16:45:32.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734bc2e700 1 -- start start 2023-03-03T16:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734bc2e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f73440aac00 0x7f73440c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734bc2e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7344155950 0x7f7344155d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734bc2e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73440713c0 con 0x7f7344071c30 2023-03-03T16:45:32.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734bc2e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73441562d0 con 0x7f7344155950 2023-03-03T16:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734bc2e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7344152c70 con 0x7f73440aac00 2023-03-03T16:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734a1cb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7344155950 0x7f7344155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734a1cb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7344155950 0x7f7344155d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58490/0 (socket says 172.21.15.7:58490) 2023-03-03T16:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734a1cb700 1 -- 172.21.15.7:0/2182621391 learned_addr learned my addr 172.21.15.7:0/2182621391 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f73499ca700 1 --2- 172.21.15.7:0/2182621391 >> v2:172.21.15.143:3300/0 conn(0x7f73440aac00 0x7f73440c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734a1cb700 1 -- 172.21.15.7:0/2182621391 >> v2:172.21.15.143:3300/0 conn(0x7f73440aac00 msgr2=0x7f73440c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734a1cb700 1 --2- 172.21.15.7:0/2182621391 >> v2:172.21.15.143:3300/0 conn(0x7f73440aac00 0x7f73440c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:32.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734a1cb700 1 -- 172.21.15.7:0/2182621391 >> v1:172.21.15.143:6789/0 conn(0x7f7344071c30 legacy=0x7f73440beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734a1cb700 1 -- 172.21.15.7:0/2182621391 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73440c4930 con 0x7f7344155950 2023-03-03T16:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.488+0000 7f734a1cb700 1 --2- 172.21.15.7:0/2182621391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344155950 0x7f7344155d90 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f73440c9630 tx=0x7f733c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329c596aa2f40f7f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f73489c8700 1 -- 172.21.15.7:0/2182621391 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c00e070 con 0x7f7344155950 2023-03-03T16:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f73489c8700 1 -- 172.21.15.7:0/2182621391 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c003a20 con 0x7f7344155950 2023-03-03T16:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f73489c8700 1 -- 172.21.15.7:0/2182621391 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733c003dd0 con 0x7f7344155950 2023-03-03T16:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f734bc2e700 1 -- 172.21.15.7:0/2182621391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344155950 msgr2=0x7f7344155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:32.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f734bc2e700 1 --2- 172.21.15.7:0/2182621391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344155950 0x7f7344155d90 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f73440c9630 tx=0x7f733c00b330 comp rx=0 tx=0).stop 2023-03-03T16:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f734bc2e700 1 -- 172.21.15.7:0/2182621391 shutdown_connections 2023-03-03T16:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f734bc2e700 1 --2- 172.21.15.7:0/2182621391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344155950 0x7f7344155d90 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f734bc2e700 1 --2- 172.21.15.7:0/2182621391 >> v2:172.21.15.143:3300/0 conn(0x7f73440aac00 0x7f73440c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f734bc2e700 1 -- 172.21.15.7:0/2182621391 >> 172.21.15.7:0/2182621391 conn(0x7f73440736f0 msgr2=0x7f7344073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f734bc2e700 1 -- 172.21.15.7:0/2182621391 shutdown_connections 2023-03-03T16:45:32.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.489+0000 7f734bc2e700 1 -- 172.21.15.7:0/2182621391 wait complete. 2023-03-03T16:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.490+0000 7f734bc2e700 1 Processor -- start 2023-03-03T16:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.490+0000 7f734bc2e700 1 -- start start 2023-03-03T16:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f734bc2e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344071c30 0x7f73440c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:32.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f734bc2e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73440aac00 0x7f73440c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f734bc2e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7344155950 0x7f7344148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f734bc2e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73440c7be0 con 0x7f7344071c30 2023-03-03T16:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f734bc2e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73440c7620 con 0x7f73440aac00 2023-03-03T16:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f734bc2e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73440c7900 con 0x7f7344155950 2023-03-03T16:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f73499ca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344071c30 0x7f73440c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f73499ca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344071c30 0x7f73440c88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58492/0 (socket says 172.21.15.7:58492) 2023-03-03T16:45:32.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f73499ca700 1 -- 172.21.15.7:0/1809673009 learned_addr learned my addr 172.21.15.7:0/1809673009 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:45:32.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f734a1cb700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7344155950 0x7f7344148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:32.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.491+0000 7f73491c9700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73440aac00 0x7f73440c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:32.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f73499ca700 1 -- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7344155950 msgr2=0x7f7344148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:32.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f73499ca700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7344155950 0x7f7344148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:32.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f73499ca700 1 -- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73440aac00 msgr2=0x7f73440c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:32.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f73499ca700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73440aac00 0x7f73440c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:32.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f73499ca700 1 -- 172.21.15.7:0/1809673009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c009b30 con 0x7f7344071c30 2023-03-03T16:45:32.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f73499ca700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344071c30 0x7f73440c88c0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f734000e810 tx=0x7f734000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f733a7fc700 1 -- 172.21.15.7:0/1809673009 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7340013610 con 0x7f7344071c30 2023-03-03T16:45:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f733a7fc700 1 -- 172.21.15.7:0/1809673009 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7340013770 con 0x7f7344071c30 2023-03-03T16:45:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f734bc2e700 1 -- 172.21.15.7:0/1809673009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73440c4b80 con 0x7f7344071c30 2023-03-03T16:45:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f733a7fc700 1 -- 172.21.15.7:0/1809673009 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7340013b20 con 0x7f7344071c30 2023-03-03T16:45:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.493+0000 7f734bc2e700 1 -- 172.21.15.7:0/1809673009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7344152360 con 0x7f7344071c30 2023-03-03T16:45:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.494+0000 7f734bc2e700 1 -- 172.21.15.7:0/1809673009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7344152620 con 0x7f7344071c30 2023-03-03T16:45:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.494+0000 7f733a7fc700 1 -- 172.21.15.7:0/1809673009 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7340023070 con 0x7f7344071c30 2023-03-03T16:45:32.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.494+0000 7f733a7fc700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73280682f0 0x7f732806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:45:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.494+0000 7f733a7fc700 1 -- 172.21.15.7:0/1809673009 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7340085e80 con 0x7f7344071c30 2023-03-03T16:45:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.494+0000 7f73491c9700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73280682f0 0x7f732806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:45:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.495+0000 7f73491c9700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73280682f0 0x7f732806a7a0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f73441550f0 tx=0x7f7334009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:45:32.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.498+0000 7f733a7fc700 1 -- 172.21.15.7:0/1809673009 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f73400521a0 con 0x7f7344071c30 2023-03-03T16:45:32.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.740+0000 7f734bc2e700 1 -- 172.21.15.7:0/1809673009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73440c6100 con 0x7f7344071c30 2023-03-03T16:45:32.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.741+0000 7f733a7fc700 1 -- 172.21.15.7:0/1809673009 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f73400557c0 con 0x7f7344071c30 2023-03-03T16:45:32.741 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:45:32.742 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:45:32.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 -- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73280682f0 msgr2=0x7f732806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:32.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73280682f0 0x7f732806a7a0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f73441550f0 tx=0x7f7334009040 comp rx=0 tx=0).stop 2023-03-03T16:45:32.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 -- 172.21.15.7:0/1809673009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344071c30 msgr2=0x7f73440c88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:45:32.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344071c30 0x7f73440c88c0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f734000e810 tx=0x7f734000ebd0 comp rx=0 tx=0).stop 2023-03-03T16:45:32.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 -- 172.21.15.7:0/1809673009 shutdown_connections 2023-03-03T16:45:32.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7344155950 0x7f7344148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:32.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73280682f0 0x7f732806a7a0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:32.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7344071c30 0x7f73440c88c0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 --2- 172.21.15.7:0/1809673009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73440aac00 0x7f73440c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:45:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 -- 172.21.15.7:0/1809673009 >> 172.21.15.7:0/1809673009 conn(0x7f73440736f0 msgr2=0x7f73441546b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:45:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 -- 172.21.15.7:0/1809673009 shutdown_connections 2023-03-03T16:45:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:45:32.743+0000 7f734bc2e700 1 -- 172.21.15.7:0/1809673009 wait complete. 2023-03-03T16:45:32.745 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:45:32.755 INFO:tasks.mds_thrash.fs.[b]:waiting for 119 secs before thrashing 2023-03-03T16:45:48.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:45:48.887 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:18.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:18.942 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:49.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:46:49.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:19.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:19.127 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:30.080 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:47:30.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.219+0000 7f7c31bad700 1 Processor -- start 2023-03-03T16:47:30.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.220+0000 7f7c31bad700 1 -- start start 2023-03-03T16:47:30.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.220+0000 7f7c31bad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c13a0f0 0x7f7c2c0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:30.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.220+0000 7f7c31bad700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7c2c0c1da0 0x7f7c2c06d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:30.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.220+0000 7f7c31bad700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2c06de90 con 0x7f7c2c1586c0 2023-03-03T16:47:30.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.220+0000 7f7c31bad700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2c154840 con 0x7f7c2c0c1da0 2023-03-03T16:47:30.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.220+0000 7f7c31bad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c154560 con 0x7f7c2c13a0f0 2023-03-03T16:47:30.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.221+0000 7f7c2bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7c2c0c1da0 0x7f7c2c06d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:30.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.221+0000 7f7c2b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c13a0f0 0x7f7c2c0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:30.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.221+0000 7f7c2bfff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7c2c0c1da0 0x7f7c2c06d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:54342/0 (socket says 172.21.15.7:54342) 2023-03-03T16:47:30.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.221+0000 7f7c2bfff700 1 -- 172.21.15.7:0/440308391 learned_addr learned my addr 172.21.15.7:0/440308391 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:47:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.221+0000 7f7c2bfff700 1 -- 172.21.15.7:0/440308391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c13a0f0 msgr2=0x7f7c2c0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:47:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.221+0000 7f7c2bfff700 1 --2- 172.21.15.7:0/440308391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c13a0f0 0x7f7c2c0c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.221+0000 7f7c2bfff700 1 -- 172.21.15.7:0/440308391 >> v1:172.21.15.143:6789/0 conn(0x7f7c2c1586c0 legacy=0x7f7c2c06d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.221+0000 7f7c2bfff700 1 -- 172.21.15.7:0/440308391 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c157be0 con 0x7f7c2c0c1da0 2023-03-03T16:47:30.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.221+0000 7f7c2b7fe700 1 --2- 172.21.15.7:0/440308391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c13a0f0 0x7f7c2c0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:47:30.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.221+0000 7f7c2bfff700 1 --2- 172.21.15.7:0/440308391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c0c1da0 0x7f7c2c06d950 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009e50 tx=0x7f7c2400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4ee1223201bede5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:30.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c2a7fc700 1 -- 172.21.15.7:0/440308391 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2400e070 con 0x7f7c2c0c1da0 2023-03-03T16:47:30.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c2a7fc700 1 -- 172.21.15.7:0/440308391 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2400b910 con 0x7f7c2c0c1da0 2023-03-03T16:47:30.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c2a7fc700 1 -- 172.21.15.7:0/440308391 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2400bcc0 con 0x7f7c2c0c1da0 2023-03-03T16:47:30.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c31bad700 1 -- 172.21.15.7:0/440308391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c0c1da0 msgr2=0x7f7c2c06d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:30.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c31bad700 1 --2- 172.21.15.7:0/440308391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c0c1da0 0x7f7c2c06d950 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009e50 tx=0x7f7c2400b330 comp rx=0 tx=0).stop 2023-03-03T16:47:30.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c31bad700 1 -- 172.21.15.7:0/440308391 shutdown_connections 2023-03-03T16:47:30.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c31bad700 1 --2- 172.21.15.7:0/440308391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c0c1da0 0x7f7c2c06d950 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:30.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c31bad700 1 --2- 172.21.15.7:0/440308391 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c13a0f0 0x7f7c2c0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:30.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c31bad700 1 -- 172.21.15.7:0/440308391 >> 172.21.15.7:0/440308391 conn(0x7f7c2c06c110 msgr2=0x7f7c2c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:47:30.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c31bad700 1 -- 172.21.15.7:0/440308391 shutdown_connections 2023-03-03T16:47:30.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.222+0000 7f7c31bad700 1 -- 172.21.15.7:0/440308391 wait complete. 2023-03-03T16:47:30.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.224+0000 7f7c31bad700 1 Processor -- start 2023-03-03T16:47:30.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.224+0000 7f7c31bad700 1 -- start start 2023-03-03T16:47:30.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c31bad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c0c1da0 0x7f7c2c1e0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:30.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c31bad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c13a0f0 0x7f7c2c1511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c31bad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2c1586c0 0x7f7c2c151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c31bad700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c156040 con 0x7f7c2c1586c0 2023-03-03T16:47:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c31bad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c155900 con 0x7f7c2c13a0f0 2023-03-03T16:47:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c31bad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c070bc0 con 0x7f7c2c0c1da0 2023-03-03T16:47:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c2bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2c1586c0 0x7f7c2c151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c2bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2c1586c0 0x7f7c2c151700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58574/0 (socket says 172.21.15.7:58574) 2023-03-03T16:47:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c2bfff700 1 -- 172.21.15.7:0/1913679259 learned_addr learned my addr 172.21.15.7:0/1913679259 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:47:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c2b7fe700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c0c1da0 0x7f7c2c1e0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c2affd700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c13a0f0 0x7f7c2c1511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c2bfff700 1 -- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c0c1da0 msgr2=0x7f7c2c1e0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c2bfff700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c0c1da0 0x7f7c2c1e0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c2bfff700 1 -- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c13a0f0 msgr2=0x7f7c2c1511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c2bfff700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c13a0f0 0x7f7c2c1511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.225+0000 7f7c2bfff700 1 -- 172.21.15.7:0/1913679259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c0094a0 con 0x7f7c2c1586c0 2023-03-03T16:47:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.226+0000 7f7c2bfff700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2c1586c0 0x7f7c2c151700 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400b680 tx=0x7f7c24004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.226+0000 7f7c13fff700 1 -- 172.21.15.7:0/1913679259 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24004830 con 0x7f7c2c1586c0 2023-03-03T16:47:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.226+0000 7f7c13fff700 1 -- 172.21.15.7:0/1913679259 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c240099b0 con 0x7f7c2c1586c0 2023-03-03T16:47:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.226+0000 7f7c31bad700 1 -- 172.21.15.7:0/1913679259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c24009b30 con 0x7f7c2c1586c0 2023-03-03T16:47:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.226+0000 7f7c13fff700 1 -- 172.21.15.7:0/1913679259 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2401b670 con 0x7f7c2c1586c0 2023-03-03T16:47:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.226+0000 7f7c31bad700 1 -- 172.21.15.7:0/1913679259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2c1581a0 con 0x7f7c2c1586c0 2023-03-03T16:47:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.227+0000 7f7c31bad700 1 -- 172.21.15.7:0/1913679259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2c157ec0 con 0x7f7c2c1586c0 2023-03-03T16:47:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.227+0000 7f7c13fff700 1 -- 172.21.15.7:0/1913679259 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7c2402b070 con 0x7f7c2c1586c0 2023-03-03T16:47:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.228+0000 7f7c13fff700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c180682f0 0x7f7c1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.228+0000 7f7c13fff700 1 -- 172.21.15.7:0/1913679259 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7c24087790 con 0x7f7c2c1586c0 2023-03-03T16:47:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.228+0000 7f7c2b7fe700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c180682f0 0x7f7c1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:30.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.229+0000 7f7c2b7fe700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c180682f0 0x7f7c1806a7a0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c009cd0 tx=0x7f7c1c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:30.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.232+0000 7f7c13fff700 1 -- 172.21.15.7:0/1913679259 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7c24053b30 con 0x7f7c2c1586c0 2023-03-03T16:47:30.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.472+0000 7f7c31bad700 1 -- 172.21.15.7:0/1913679259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c2c06e1a0 con 0x7f7c2c1586c0 2023-03-03T16:47:30.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.473+0000 7f7c13fff700 1 -- 172.21.15.7:0/1913679259 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7c24057150 con 0x7f7c2c1586c0 2023-03-03T16:47:30.473 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:47:30.473 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:47:30.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 -- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c180682f0 msgr2=0x7f7c1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:30.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c180682f0 0x7f7c1806a7a0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c009cd0 tx=0x7f7c1c009210 comp rx=0 tx=0).stop 2023-03-03T16:47:30.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 -- 172.21.15.7:0/1913679259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2c1586c0 msgr2=0x7f7c2c151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:30.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2c1586c0 0x7f7c2c151700 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400b680 tx=0x7f7c24004ad0 comp rx=0 tx=0).stop 2023-03-03T16:47:30.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 -- 172.21.15.7:0/1913679259 shutdown_connections 2023-03-03T16:47:30.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7c180682f0 0x7f7c1806a7a0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:30.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c2c1586c0 0x7f7c2c151700 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:30.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c2c0c1da0 0x7f7c2c1e0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:30.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 --2- 172.21.15.7:0/1913679259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c2c13a0f0 0x7f7c2c1511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:30.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 -- 172.21.15.7:0/1913679259 >> 172.21.15.7:0/1913679259 conn(0x7f7c2c06c110 msgr2=0x7f7c2c137150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:47:30.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 -- 172.21.15.7:0/1913679259 shutdown_connections 2023-03-03T16:47:30.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:30.475+0000 7f7c31bad700 1 -- 172.21.15.7:0/1913679259 wait complete. 2023-03-03T16:47:30.477 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:47:30.487 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before thrashing 2023-03-03T16:47:31.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:47:31.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.895+0000 7f55f5965700 1 Processor -- start 2023-03-03T16:47:31.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.895+0000 7f55f5965700 1 -- start start 2023-03-03T16:47:31.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.896+0000 7f55f5965700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55f00c6010 0x7f55f00c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.896+0000 7f55f5965700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 0x7f55f006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.896+0000 7f55f5965700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f014ebe0 con 0x7f55f00c17f0 2023-03-03T16:47:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.896+0000 7f55f5965700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f0152100 con 0x7f55f00c6010 2023-03-03T16:47:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.896+0000 7f55f5965700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f00c2e70 con 0x7f55f00c4fc0 2023-03-03T16:47:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.896+0000 7f55eeffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55f00c6010 0x7f55f00c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.896+0000 7f55ef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 0x7f55f006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:31.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55eeffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55f00c6010 0x7f55f00c4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:54352/0 (socket says 172.21.15.7:54352) 2023-03-03T16:47:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55ef7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 0x7f55f006ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:38422/0 (socket says 172.21.15.7:38422) 2023-03-03T16:47:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55eeffd700 1 -- 172.21.15.7:0/3005774982 learned_addr learned my addr 172.21.15.7:0/3005774982 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:47:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55ef7fe700 1 -- 172.21.15.7:0/3005774982 >> v2:172.21.15.143:3300/0 conn(0x7f55f00c6010 msgr2=0x7f55f00c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:47:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55ef7fe700 1 --2- 172.21.15.7:0/3005774982 >> v2:172.21.15.143:3300/0 conn(0x7f55f00c6010 0x7f55f00c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55ef7fe700 1 -- 172.21.15.7:0/3005774982 >> v1:172.21.15.143:6789/0 conn(0x7f55f00c17f0 legacy=0x7f55f006d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55ef7fe700 1 -- 172.21.15.7:0/3005774982 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f00aa680 con 0x7f55f00c4fc0 2023-03-03T16:47:31.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55eeffd700 1 --2- 172.21.15.7:0/3005774982 >> v2:172.21.15.143:3300/0 conn(0x7f55f00c6010 0x7f55f00c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:47:31.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55ef7fe700 1 --2- 172.21.15.7:0/3005774982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 0x7f55f006ddb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009d90 tx=0x7f55e800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18d1e07012848b7c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55edffb700 1 -- 172.21.15.7:0/3005774982 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e800e070 con 0x7f55f00c4fc0 2023-03-03T16:47:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55edffb700 1 -- 172.21.15.7:0/3005774982 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e800bb20 con 0x7f55f00c4fc0 2023-03-03T16:47:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.897+0000 7f55edffb700 1 -- 172.21.15.7:0/3005774982 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8012430 con 0x7f55f00c4fc0 2023-03-03T16:47:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.898+0000 7f55f5965700 1 -- 172.21.15.7:0/3005774982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 msgr2=0x7f55f006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.898+0000 7f55f5965700 1 --2- 172.21.15.7:0/3005774982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 0x7f55f006ddb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009d90 tx=0x7f55e800b330 comp rx=0 tx=0).stop 2023-03-03T16:47:31.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.898+0000 7f55f5965700 1 -- 172.21.15.7:0/3005774982 shutdown_connections 2023-03-03T16:47:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.898+0000 7f55f5965700 1 --2- 172.21.15.7:0/3005774982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 0x7f55f006ddb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.898+0000 7f55f5965700 1 --2- 172.21.15.7:0/3005774982 >> v2:172.21.15.143:3300/0 conn(0x7f55f00c6010 0x7f55f00c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.898+0000 7f55f5965700 1 -- 172.21.15.7:0/3005774982 >> 172.21.15.7:0/3005774982 conn(0x7f55f00a9430 msgr2=0x7f55f00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:47:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.898+0000 7f55f5965700 1 -- 172.21.15.7:0/3005774982 shutdown_connections 2023-03-03T16:47:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.898+0000 7f55f5965700 1 -- 172.21.15.7:0/3005774982 wait complete. 2023-03-03T16:47:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.899+0000 7f55f5965700 1 Processor -- start 2023-03-03T16:47:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.899+0000 7f55f5965700 1 -- start start 2023-03-03T16:47:31.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55f5965700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f00c17f0 0x7f55f0152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:31.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55f5965700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 0x7f55f014d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:31.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55f5965700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55f00c6010 0x7f55f01541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:31.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55f5965700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0155000 con 0x7f55f00c6010 2023-03-03T16:47:31.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55f5965700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0072190 con 0x7f55f00c17f0 2023-03-03T16:47:31.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55f5965700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0141650 con 0x7f55f00c4fc0 2023-03-03T16:47:31.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55ef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55f00c6010 0x7f55f01541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:31.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55ef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55f00c6010 0x7f55f01541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58588/0 (socket says 172.21.15.7:58588) 2023-03-03T16:47:31.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55ef7fe700 1 -- 172.21.15.7:0/3120431680 learned_addr learned my addr 172.21.15.7:0/3120431680 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:47:31.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55ee7fc700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 0x7f55f014d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:31.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55eeffd700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f00c17f0 0x7f55f0152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:31.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55ef7fe700 1 -- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 msgr2=0x7f55f014d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:31.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55ef7fe700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 0x7f55f014d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:31.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55ef7fe700 1 -- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f00c17f0 msgr2=0x7f55f0152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:31.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55ef7fe700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f00c17f0 0x7f55f0152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:31.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.900+0000 7f55ef7fe700 1 -- 172.21.15.7:0/3120431680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e4009550 con 0x7f55f00c6010 2023-03-03T16:47:31.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.901+0000 7f55ef7fe700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55f00c6010 0x7f55f01541f0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f55f00c2fb0 tx=0x7f55e8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:31.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.901+0000 7f55d7fff700 1 -- 172.21.15.7:0/3120431680 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8012430 con 0x7f55f00c6010 2023-03-03T16:47:31.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.901+0000 7f55d7fff700 1 -- 172.21.15.7:0/3120431680 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e8020070 con 0x7f55f00c6010 2023-03-03T16:47:31.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.901+0000 7f55f5965700 1 -- 172.21.15.7:0/3120431680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e8009a70 con 0x7f55f00c6010 2023-03-03T16:47:31.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.901+0000 7f55d7fff700 1 -- 172.21.15.7:0/3120431680 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8003ea0 con 0x7f55f00c6010 2023-03-03T16:47:31.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.901+0000 7f55f5965700 1 -- 172.21.15.7:0/3120431680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f00c2b00 con 0x7f55f00c6010 2023-03-03T16:47:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.902+0000 7f55f5965700 1 -- 172.21.15.7:0/3120431680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f00aa960 con 0x7f55f00c6010 2023-03-03T16:47:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.902+0000 7f55d7fff700 1 -- 172.21.15.7:0/3120431680 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f55e802e070 con 0x7f55f00c6010 2023-03-03T16:47:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.903+0000 7f55d7fff700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55dc0682f0 0x7f55dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:31.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.903+0000 7f55d7fff700 1 -- 172.21.15.7:0/3120431680 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f55e8022070 con 0x7f55f00c6010 2023-03-03T16:47:31.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.903+0000 7f55eeffd700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55dc0682f0 0x7f55dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:31.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.903+0000 7f55eeffd700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55dc0682f0 0x7f55dc06a7a0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f55e4006d30 tx=0x7f55e40092c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:31.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:31.907+0000 7f55d7fff700 1 -- 172.21.15.7:0/3120431680 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f55e8054130 con 0x7f55f00c6010 2023-03-03T16:47:32.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.145+0000 7f55f5965700 1 -- 172.21.15.7:0/3120431680 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55f00c5420 con 0x7f55f00c6010 2023-03-03T16:47:32.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.145+0000 7f55d7fff700 1 -- 172.21.15.7:0/3120431680 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f55e802c090 con 0x7f55f00c6010 2023-03-03T16:47:32.146 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:47:32.146 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:47:32.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 -- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55dc0682f0 msgr2=0x7f55dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:32.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55dc0682f0 0x7f55dc06a7a0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f55e4006d30 tx=0x7f55e40092c0 comp rx=0 tx=0).stop 2023-03-03T16:47:32.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 -- 172.21.15.7:0/3120431680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55f00c6010 msgr2=0x7f55f01541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:32.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55f00c6010 0x7f55f01541f0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f55f00c2fb0 tx=0x7f55e8004ad0 comp rx=0 tx=0).stop 2023-03-03T16:47:32.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 -- 172.21.15.7:0/3120431680 shutdown_connections 2023-03-03T16:47:32.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55dc0682f0 0x7f55dc06a7a0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:32.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55f00c6010 0x7f55f01541f0 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:32.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f00c17f0 0x7f55f0152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:32.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 --2- 172.21.15.7:0/3120431680 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55f00c4fc0 0x7f55f014d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:32.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 -- 172.21.15.7:0/3120431680 >> 172.21.15.7:0/3120431680 conn(0x7f55f00a9430 msgr2=0x7f55f00c1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:47:32.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 -- 172.21.15.7:0/3120431680 shutdown_connections 2023-03-03T16:47:32.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:32.148+0000 7f55f5965700 1 -- 172.21.15.7:0/3120431680 wait complete. 2023-03-03T16:47:32.149 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:47:32.159 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T16:47:39.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:47:39.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.625+0000 7fc2af61e700 1 Processor -- start 2023-03-03T16:47:39.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.625+0000 7fc2af61e700 1 -- start start 2023-03-03T16:47:39.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2af61e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 0x7fc2a81550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2af61e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc2a80c1da0 0x7fc2a80c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2af61e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc2a80c3e30 0x7fc2a80c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2af61e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a80c2af0 con 0x7fc2a80c3e30 2023-03-03T16:47:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2af61e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a80c3620 con 0x7fc2a80c1da0 2023-03-03T16:47:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2af61e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a81545e0 con 0x7fc2a806d870 2023-03-03T16:47:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2adbbb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc2a80c3e30 0x7fc2a80c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:39.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2adbbb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc2a80c3e30 0x7fc2a80c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58598/0 (socket says 172.21.15.7:58598) 2023-03-03T16:47:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2adbbb700 1 -- 172.21.15.7:0/630419650 learned_addr learned my addr 172.21.15.7:0/630419650 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:47:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2ad3ba700 1 --2- 172.21.15.7:0/630419650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 0x7fc2a81550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2acbb9700 1 --2- 172.21.15.7:0/630419650 >> v2:172.21.15.143:3300/0 conn(0x7fc2a80c1da0 0x7fc2a80c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2adbbb700 1 -- 172.21.15.7:0/630419650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 msgr2=0x7fc2a81550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:47:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2adbbb700 1 --2- 172.21.15.7:0/630419650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 0x7fc2a81550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2adbbb700 1 -- 172.21.15.7:0/630419650 >> v2:172.21.15.143:3300/0 conn(0x7fc2a80c1da0 msgr2=0x7fc2a80c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:47:39.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2adbbb700 1 --2- 172.21.15.7:0/630419650 >> v2:172.21.15.143:3300/0 conn(0x7fc2a80c1da0 0x7fc2a80c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2adbbb700 1 -- 172.21.15.7:0/630419650 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a81548c0 con 0x7fc2a80c3e30 2023-03-03T16:47:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.626+0000 7fc2ad3ba700 1 --2- 172.21.15.7:0/630419650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 0x7fc2a81550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:47:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc2adbbb700 1 --2- 172.21.15.7:0/630419650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c3e30 0x7fc2a80c4290 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80aa1d0 tx=0x7fc2a000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ce8c11136189fc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc29ffff700 1 -- 172.21.15.7:0/630419650 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a000e070 con 0x7fc2a80c3e30 2023-03-03T16:47:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc29ffff700 1 -- 172.21.15.7:0/630419650 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a000b9c0 con 0x7fc2a80c3e30 2023-03-03T16:47:39.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc29ffff700 1 -- 172.21.15.7:0/630419650 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a000bd70 con 0x7fc2a80c3e30 2023-03-03T16:47:39.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc2af61e700 1 -- 172.21.15.7:0/630419650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c3e30 msgr2=0x7fc2a80c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:39.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc2af61e700 1 --2- 172.21.15.7:0/630419650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c3e30 0x7fc2a80c4290 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80aa1d0 tx=0x7fc2a000b330 comp rx=0 tx=0).stop 2023-03-03T16:47:39.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc2af61e700 1 -- 172.21.15.7:0/630419650 shutdown_connections 2023-03-03T16:47:39.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc2af61e700 1 --2- 172.21.15.7:0/630419650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c3e30 0x7fc2a80c4290 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc2af61e700 1 --2- 172.21.15.7:0/630419650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 0x7fc2a81550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc2af61e700 1 --2- 172.21.15.7:0/630419650 >> v2:172.21.15.143:3300/0 conn(0x7fc2a80c1da0 0x7fc2a80c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.627+0000 7fc2af61e700 1 -- 172.21.15.7:0/630419650 >> 172.21.15.7:0/630419650 conn(0x7fc2a806c110 msgr2=0x7fc2a806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:47:39.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.628+0000 7fc2af61e700 1 -- 172.21.15.7:0/630419650 shutdown_connections 2023-03-03T16:47:39.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.628+0000 7fc2af61e700 1 -- 172.21.15.7:0/630419650 wait complete. 2023-03-03T16:47:39.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.629+0000 7fc2af61e700 1 Processor -- start 2023-03-03T16:47:39.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.629+0000 7fc2af61e700 1 -- start start 2023-03-03T16:47:39.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.629+0000 7fc2af61e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 0x7fc2a80c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:39.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2af61e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c1da0 0x7fc2a8146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:39.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2af61e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2a80c3e30 0x7fc2a81472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:39.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2af61e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a81edd80 con 0x7fc2a80c1da0 2023-03-03T16:47:39.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2af61e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80ad030 con 0x7fc2a80c3e30 2023-03-03T16:47:39.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2af61e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a8147810 con 0x7fc2a806d870 2023-03-03T16:47:39.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2acbb9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c1da0 0x7fc2a8146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:39.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2acbb9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c1da0 0x7fc2a8146d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58602/0 (socket says 172.21.15.7:58602) 2023-03-03T16:47:39.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2acbb9700 1 -- 172.21.15.7:0/478240629 learned_addr learned my addr 172.21.15.7:0/478240629 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:47:39.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2ad3ba700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 0x7fc2a80c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:39.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2adbbb700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2a80c3e30 0x7fc2a81472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:39.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2acbb9700 1 -- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 msgr2=0x7fc2a80c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:39.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2acbb9700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 0x7fc2a80c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2acbb9700 1 -- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2a80c3e30 msgr2=0x7fc2a81472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:39.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2acbb9700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2a80c3e30 0x7fc2a81472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2acbb9700 1 -- 172.21.15.7:0/478240629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a0009b30 con 0x7fc2a80c1da0 2023-03-03T16:47:39.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2acbb9700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c1da0 0x7fc2a8146d90 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7fc2a400e810 tx=0x7fc2a400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:39.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc29dffb700 1 -- 172.21.15.7:0/478240629 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a400c070 con 0x7fc2a80c1da0 2023-03-03T16:47:39.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc29dffb700 1 -- 172.21.15.7:0/478240629 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a4013610 con 0x7fc2a80c1da0 2023-03-03T16:47:39.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc2af61e700 1 -- 172.21.15.7:0/478240629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a806dc40 con 0x7fc2a80c1da0 2023-03-03T16:47:39.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.630+0000 7fc29dffb700 1 -- 172.21.15.7:0/478240629 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a40139c0 con 0x7fc2a80c1da0 2023-03-03T16:47:39.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.631+0000 7fc2af61e700 1 -- 172.21.15.7:0/478240629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a8154ba0 con 0x7fc2a80c1da0 2023-03-03T16:47:39.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.632+0000 7fc2af61e700 1 -- 172.21.15.7:0/478240629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a80c38e0 con 0x7fc2a80c1da0 2023-03-03T16:47:39.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.632+0000 7fc29dffb700 1 -- 172.21.15.7:0/478240629 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc2a4023070 con 0x7fc2a80c1da0 2023-03-03T16:47:39.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.632+0000 7fc29dffb700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc28c0682f0 0x7fc28c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:47:39.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.632+0000 7fc29dffb700 1 -- 172.21.15.7:0/478240629 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc2a4085f60 con 0x7fc2a80c1da0 2023-03-03T16:47:39.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.633+0000 7fc2ad3ba700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc28c0682f0 0x7fc28c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:47:39.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.633+0000 7fc2ad3ba700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc28c0682f0 0x7fc28c06a7a0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fc2a81e76d0 tx=0x7fc298008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:47:39.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.637+0000 7fc29dffb700 1 -- 172.21.15.7:0/478240629 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc2a4052190 con 0x7fc2a80c1da0 2023-03-03T16:47:39.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.878+0000 7fc2af61e700 1 -- 172.21.15.7:0/478240629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2a81e98e0 con 0x7fc2a80c1da0 2023-03-03T16:47:39.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.879+0000 7fc29dffb700 1 -- 172.21.15.7:0/478240629 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc2a40557b0 con 0x7fc2a80c1da0 2023-03-03T16:47:39.880 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:47:39.880 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:47:39.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 -- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc28c0682f0 msgr2=0x7fc28c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:39.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc28c0682f0 0x7fc28c06a7a0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fc2a81e76d0 tx=0x7fc298008040 comp rx=0 tx=0).stop 2023-03-03T16:47:39.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 -- 172.21.15.7:0/478240629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c1da0 msgr2=0x7fc2a8146d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:47:39.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c1da0 0x7fc2a8146d90 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7fc2a400e810 tx=0x7fc2a400eb20 comp rx=0 tx=0).stop 2023-03-03T16:47:39.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 -- 172.21.15.7:0/478240629 shutdown_connections 2023-03-03T16:47:39.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2a80c3e30 0x7fc2a81472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2a806d870 0x7fc2a80c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc28c0682f0 0x7fc28c06a7a0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 --2- 172.21.15.7:0/478240629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2a80c1da0 0x7fc2a8146d90 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:47:39.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 -- 172.21.15.7:0/478240629 >> 172.21.15.7:0/478240629 conn(0x7fc2a806c110 msgr2=0x7fc2a80706e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:47:39.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 -- 172.21.15.7:0/478240629 shutdown_connections 2023-03-03T16:47:39.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:47:39.881+0000 7fc2af61e700 1 -- 172.21.15.7:0/478240629 wait complete. 2023-03-03T16:47:39.884 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:47:39.894 INFO:tasks.mds_thrash.fs.[a]:waiting for 21 secs before thrashing 2023-03-03T16:47:49.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:47:49.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:00.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:48:01.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.036+0000 7f4bb5185700 1 Processor -- start 2023-03-03T16:48:01.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.036+0000 7f4bb5185700 1 -- start start 2023-03-03T16:48:01.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.036+0000 7f4bb5185700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4bb0071990 0x7f4bb0071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:01.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.036+0000 7f4bb5185700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4bb0151c00 0x7f4bb00722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:01.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.036+0000 7f4bb5185700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb0070360 con 0x7f4bb014b460 2023-03-03T16:48:01.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.036+0000 7f4bb5185700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb01521a0 con 0x7f4bb0151c00 2023-03-03T16:48:01.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.036+0000 7f4bb5185700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb006a5e0 con 0x7f4bb0071990 2023-03-03T16:48:01.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4baf59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4bb0151c00 0x7f4bb00722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:01.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4baf59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4bb0151c00 0x7f4bb00722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58620/0 (socket says 172.21.15.7:58620) 2023-03-03T16:48:01.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4baf59e700 1 -- 172.21.15.7:0/1946073962 learned_addr learned my addr 172.21.15.7:0/1946073962 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:48:01.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4baed9d700 1 --2- 172.21.15.7:0/1946073962 >> v2:172.21.15.143:3300/0 conn(0x7f4bb0071990 0x7f4bb0071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:01.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4baf59e700 1 -- 172.21.15.7:0/1946073962 >> v2:172.21.15.143:3300/0 conn(0x7f4bb0071990 msgr2=0x7f4bb0071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:48:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4baf59e700 1 --2- 172.21.15.7:0/1946073962 >> v2:172.21.15.143:3300/0 conn(0x7f4bb0071990 0x7f4bb0071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4baf59e700 1 -- 172.21.15.7:0/1946073962 >> v1:172.21.15.143:6789/0 conn(0x7f4bb014b460 legacy=0x7f4bb006aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4baf59e700 1 -- 172.21.15.7:0/1946073962 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb00c38b0 con 0x7f4bb0151c00 2023-03-03T16:48:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4baf59e700 1 --2- 172.21.15.7:0/1946073962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0151c00 0x7f4bb00722a0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f4bb00c1de0 tx=0x7f4ba800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1403b4d7ae9ce2f9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4badd9b700 1 -- 172.21.15.7:0/1946073962 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba800e070 con 0x7f4bb0151c00 2023-03-03T16:48:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4badd9b700 1 -- 172.21.15.7:0/1946073962 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba800b910 con 0x7f4bb0151c00 2023-03-03T16:48:01.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.037+0000 7f4badd9b700 1 -- 172.21.15.7:0/1946073962 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba800bcc0 con 0x7f4bb0151c00 2023-03-03T16:48:01.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.038+0000 7f4bb5185700 1 -- 172.21.15.7:0/1946073962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0151c00 msgr2=0x7f4bb00722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.038+0000 7f4bb5185700 1 --2- 172.21.15.7:0/1946073962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0151c00 0x7f4bb00722a0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f4bb00c1de0 tx=0x7f4ba800b330 comp rx=0 tx=0).stop 2023-03-03T16:48:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.038+0000 7f4bb5185700 1 -- 172.21.15.7:0/1946073962 shutdown_connections 2023-03-03T16:48:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.038+0000 7f4bb5185700 1 --2- 172.21.15.7:0/1946073962 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0151c00 0x7f4bb00722a0 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.038+0000 7f4bb5185700 1 --2- 172.21.15.7:0/1946073962 >> v2:172.21.15.143:3300/0 conn(0x7f4bb0071990 0x7f4bb0071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.038+0000 7f4bb5185700 1 -- 172.21.15.7:0/1946073962 >> 172.21.15.7:0/1946073962 conn(0x7f4bb00a94d0 msgr2=0x7f4bb00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:48:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.038+0000 7f4bb5185700 1 -- 172.21.15.7:0/1946073962 shutdown_connections 2023-03-03T16:48:01.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.038+0000 7f4bb5185700 1 -- 172.21.15.7:0/1946073962 wait complete. 2023-03-03T16:48:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.039+0000 7f4bb5185700 1 Processor -- start 2023-03-03T16:48:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.040+0000 7f4bb5185700 1 -- start start 2023-03-03T16:48:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.040+0000 7f4bb5185700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bb0071990 0x7f4bb01df470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.040+0000 7f4bb5185700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bb0151c00 0x7f4bb01dba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.040+0000 7f4bb5185700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0146320 0x7f4bb0146790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.040+0000 7f4bb5185700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb00c0a30 con 0x7f4bb0146320 2023-03-03T16:48:01.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.040+0000 7f4bb5185700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb01ec830 con 0x7f4bb0151c00 2023-03-03T16:48:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.040+0000 7f4bb5185700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb00aa6c0 con 0x7f4bb0071990 2023-03-03T16:48:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.040+0000 7f4baf59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0146320 0x7f4bb0146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.040+0000 7f4baf59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0146320 0x7f4bb0146790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58626/0 (socket says 172.21.15.7:58626) 2023-03-03T16:48:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.040+0000 7f4baf59e700 1 -- 172.21.15.7:0/1681775191 learned_addr learned my addr 172.21.15.7:0/1681775191 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:48:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4baed9d700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bb0071990 0x7f4bb01df470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4bae59c700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bb0151c00 0x7f4bb01dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4baf59e700 1 -- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bb0071990 msgr2=0x7f4bb01df470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:01.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4baf59e700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bb0071990 0x7f4bb01df470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:01.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4baf59e700 1 -- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bb0151c00 msgr2=0x7f4bb01dba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:01.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4baf59e700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bb0151c00 0x7f4bb01dba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:01.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4baf59e700 1 -- 172.21.15.7:0/1681775191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba8009b30 con 0x7f4bb0146320 2023-03-03T16:48:01.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4bae59c700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bb0151c00 0x7f4bb01dba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:48:01.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4baed9d700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bb0071990 0x7f4bb01df470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:48:01.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4baf59e700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0146320 0x7f4bb0146790 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f4bb006bcf0 tx=0x7f4ba8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:01.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4b977fe700 1 -- 172.21.15.7:0/1681775191 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80092e0 con 0x7f4bb0146320 2023-03-03T16:48:01.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4b977fe700 1 -- 172.21.15.7:0/1681775191 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba801b430 con 0x7f4bb0146320 2023-03-03T16:48:01.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4bb5185700 1 -- 172.21.15.7:0/1681775191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb00c3b50 con 0x7f4bb0146320 2023-03-03T16:48:01.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4b977fe700 1 -- 172.21.15.7:0/1681775191 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba801b7e0 con 0x7f4bb0146320 2023-03-03T16:48:01.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.041+0000 7f4bb5185700 1 -- 172.21.15.7:0/1681775191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb006a2b0 con 0x7f4bb0146320 2023-03-03T16:48:01.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.042+0000 7f4bb5185700 1 -- 172.21.15.7:0/1681775191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb006b180 con 0x7f4bb0146320 2023-03-03T16:48:01.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.043+0000 7f4b977fe700 1 -- 172.21.15.7:0/1681775191 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ba8021030 con 0x7f4bb0146320 2023-03-03T16:48:01.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.043+0000 7f4b977fe700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b9c0682f0 0x7f4b9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:01.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.043+0000 7f4b977fe700 1 -- 172.21.15.7:0/1681775191 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4ba8087580 con 0x7f4bb0146320 2023-03-03T16:48:01.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.043+0000 7f4baed9d700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b9c0682f0 0x7f4b9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:01.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.044+0000 7f4baed9d700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b9c0682f0 0x7f4b9c06a7a0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f4bb01557c0 tx=0x7f4ba0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:01.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.048+0000 7f4b977fe700 1 -- 172.21.15.7:0/1681775191 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ba8053920 con 0x7f4bb0146320 2023-03-03T16:48:01.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.288+0000 7f4bb5185700 1 -- 172.21.15.7:0/1681775191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bb00c2840 con 0x7f4bb0146320 2023-03-03T16:48:01.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.289+0000 7f4b977fe700 1 -- 172.21.15.7:0/1681775191 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4ba8056f40 con 0x7f4bb0146320 2023-03-03T16:48:01.289 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:48:01.289 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:48:01.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 -- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b9c0682f0 msgr2=0x7f4b9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:01.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b9c0682f0 0x7f4b9c06a7a0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f4bb01557c0 tx=0x7f4ba0008040 comp rx=0 tx=0).stop 2023-03-03T16:48:01.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 -- 172.21.15.7:0/1681775191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0146320 msgr2=0x7f4bb0146790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:01.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0146320 0x7f4bb0146790 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f4bb006bcf0 tx=0x7f4ba8004ad0 comp rx=0 tx=0).stop 2023-03-03T16:48:01.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 -- 172.21.15.7:0/1681775191 shutdown_connections 2023-03-03T16:48:01.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4b9c0682f0 0x7f4b9c06a7a0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:01.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4bb0146320 0x7f4bb0146790 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:01.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4bb0071990 0x7f4bb01df470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:01.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 --2- 172.21.15.7:0/1681775191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4bb0151c00 0x7f4bb01dba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:01.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 -- 172.21.15.7:0/1681775191 >> 172.21.15.7:0/1681775191 conn(0x7f4bb00a94d0 msgr2=0x7f4bb0065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:48:01.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 -- 172.21.15.7:0/1681775191 shutdown_connections 2023-03-03T16:48:01.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:01.291+0000 7f4bb5185700 1 -- 172.21.15.7:0/1681775191 wait complete. 2023-03-03T16:48:01.293 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:48:01.303 INFO:tasks.mds_thrash.fs.[a]:waiting for 110 secs before thrashing 2023-03-03T16:48:19.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:19.304 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:36.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:48:36.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.298+0000 7f430f5de700 1 Processor -- start 2023-03-03T16:48:36.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.298+0000 7f430f5de700 1 -- start start 2023-03-03T16:48:36.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430f5de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43080c2930 0x7f4308157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430f5de700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f430806c480 0x7f4308157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430f5de700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430806d430 con 0x7f4308151d40 2023-03-03T16:48:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430f5de700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f430806c2a0 con 0x7f430806c480 2023-03-03T16:48:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430f5de700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430806cb00 con 0x7f43080c2930 2023-03-03T16:48:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430db7b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f430806c480 0x7f4308157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430db7b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f430806c480 0x7f4308157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58674/0 (socket says 172.21.15.7:58674) 2023-03-03T16:48:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430db7b700 1 -- 172.21.15.7:0/4115268506 learned_addr learned my addr 172.21.15.7:0/4115268506 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:48:36.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430cb79700 1 --2- 172.21.15.7:0/4115268506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43080c2930 0x7f4308157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430db7b700 1 -- 172.21.15.7:0/4115268506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43080c2930 msgr2=0x7f4308157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:48:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430db7b700 1 --2- 172.21.15.7:0/4115268506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43080c2930 0x7f4308157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430db7b700 1 -- 172.21.15.7:0/4115268506 >> v1:172.21.15.143:6789/0 conn(0x7f4308151d40 legacy=0x7f43080736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:48:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.299+0000 7f430db7b700 1 -- 172.21.15.7:0/4115268506 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430806cde0 con 0x7f430806c480 2023-03-03T16:48:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f430db7b700 1 --2- 172.21.15.7:0/4115268506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 0x7f4308157fd0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f4300009e50 tx=0x7f430000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673072de1eca8043 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f42fffff700 1 -- 172.21.15.7:0/4115268506 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430000e070 con 0x7f430806c480 2023-03-03T16:48:36.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f42fffff700 1 -- 172.21.15.7:0/4115268506 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4300003a20 con 0x7f430806c480 2023-03-03T16:48:36.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f42fffff700 1 -- 172.21.15.7:0/4115268506 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300003dd0 con 0x7f430806c480 2023-03-03T16:48:36.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f430f5de700 1 -- 172.21.15.7:0/4115268506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 msgr2=0x7f4308157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:36.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f430f5de700 1 --2- 172.21.15.7:0/4115268506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 0x7f4308157fd0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f4300009e50 tx=0x7f430000b330 comp rx=0 tx=0).stop 2023-03-03T16:48:36.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f430f5de700 1 -- 172.21.15.7:0/4115268506 shutdown_connections 2023-03-03T16:48:36.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f430f5de700 1 --2- 172.21.15.7:0/4115268506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 0x7f4308157fd0 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:36.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f430f5de700 1 --2- 172.21.15.7:0/4115268506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43080c2930 0x7f4308157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:36.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f430f5de700 1 -- 172.21.15.7:0/4115268506 >> 172.21.15.7:0/4115268506 conn(0x7f43080a94d0 msgr2=0x7f43080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:48:36.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.300+0000 7f430f5de700 1 -- 172.21.15.7:0/4115268506 shutdown_connections 2023-03-03T16:48:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.301+0000 7f430f5de700 1 -- 172.21.15.7:0/4115268506 wait complete. 2023-03-03T16:48:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430f5de700 1 Processor -- start 2023-03-03T16:48:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430f5de700 1 -- start start 2023-03-03T16:48:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430f5de700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 0x7f43080bf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430f5de700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43080c2930 0x7f43080bedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430f5de700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4308151d40 0x7f43081e0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:36.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430f5de700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43080bf330 con 0x7f430806c480 2023-03-03T16:48:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430f5de700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43080aa390 con 0x7f4308151d40 2023-03-03T16:48:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430f5de700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43080aa670 con 0x7f43080c2930 2023-03-03T16:48:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430d37a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 0x7f43080bf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430d37a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 0x7f43080bf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58676/0 (socket says 172.21.15.7:58676) 2023-03-03T16:48:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.302+0000 7f430d37a700 1 -- 172.21.15.7:0/2480385249 learned_addr learned my addr 172.21.15.7:0/2480385249 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:48:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f430cb79700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43080c2930 0x7f43080bedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:36.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f430db7b700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4308151d40 0x7f43081e0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f430d37a700 1 -- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43080c2930 msgr2=0x7f43080bedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f430d37a700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43080c2930 0x7f43080bedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f430d37a700 1 -- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4308151d40 msgr2=0x7f43081e0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f430d37a700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4308151d40 0x7f43081e0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f430d37a700 1 -- 172.21.15.7:0/2480385249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4300009b30 con 0x7f430806c480 2023-03-03T16:48:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f430d37a700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 0x7f43080bf900 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f42f800c4a0 tx=0x7f42f800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f42fdffb700 1 -- 172.21.15.7:0/2480385249 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f80054e0 con 0x7f430806c480 2023-03-03T16:48:36.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f42fdffb700 1 -- 172.21.15.7:0/2480385249 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f8003680 con 0x7f430806c480 2023-03-03T16:48:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f430f5de700 1 -- 172.21.15.7:0/2480385249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4308073e00 con 0x7f430806c480 2023-03-03T16:48:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f42fdffb700 1 -- 172.21.15.7:0/2480385249 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f8003a30 con 0x7f430806c480 2023-03-03T16:48:36.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.303+0000 7f430f5de700 1 -- 172.21.15.7:0/2480385249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430806d0c0 con 0x7f430806c480 2023-03-03T16:48:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.304+0000 7f430f5de700 1 -- 172.21.15.7:0/2480385249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f430806bea0 con 0x7f430806c480 2023-03-03T16:48:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.305+0000 7f42fdffb700 1 -- 172.21.15.7:0/2480385249 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f42f8019030 con 0x7f430806c480 2023-03-03T16:48:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.305+0000 7f42fdffb700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42ec0682f0 0x7f42ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.305+0000 7f42fdffb700 1 -- 172.21.15.7:0/2480385249 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f42f8085550 con 0x7f430806c480 2023-03-03T16:48:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.305+0000 7f430cb79700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42ec0682f0 0x7f42ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:36.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.306+0000 7f430cb79700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42ec0682f0 0x7f42ec06a7a0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f43080715b0 tx=0x7f4304008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:36.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.309+0000 7f42fdffb700 1 -- 172.21.15.7:0/2480385249 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f42f80518f0 con 0x7f430806c480 2023-03-03T16:48:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.545+0000 7f430f5de700 1 -- 172.21.15.7:0/2480385249 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4308074bb0 con 0x7f430806c480 2023-03-03T16:48:36.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.546+0000 7f42fdffb700 1 -- 172.21.15.7:0/2480385249 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f42f8054f10 con 0x7f430806c480 2023-03-03T16:48:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:48:36.546 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:48:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 -- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42ec0682f0 msgr2=0x7f42ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42ec0682f0 0x7f42ec06a7a0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f43080715b0 tx=0x7f4304008040 comp rx=0 tx=0).stop 2023-03-03T16:48:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 -- 172.21.15.7:0/2480385249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 msgr2=0x7f43080bf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 0x7f43080bf900 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f42f800c4a0 tx=0x7f42f800c860 comp rx=0 tx=0).stop 2023-03-03T16:48:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 -- 172.21.15.7:0/2480385249 shutdown_connections 2023-03-03T16:48:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4308151d40 0x7f43081e0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f42ec0682f0 0x7f42ec06a7a0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f430806c480 0x7f43080bf900 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 --2- 172.21.15.7:0/2480385249 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43080c2930 0x7f43080bedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 -- 172.21.15.7:0/2480385249 >> 172.21.15.7:0/2480385249 conn(0x7f43080a94d0 msgr2=0x7f43080c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:48:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 -- 172.21.15.7:0/2480385249 shutdown_connections 2023-03-03T16:48:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:36.548+0000 7f430f5de700 1 -- 172.21.15.7:0/2480385249 wait complete. 2023-03-03T16:48:36.550 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:48:36.559 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before thrashing 2023-03-03T16:48:44.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:48:44.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.698+0000 7f9dac8a2700 1 Processor -- start 2023-03-03T16:48:44.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.698+0000 7f9dac8a2700 1 -- start start 2023-03-03T16:48:44.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.699+0000 7f9dac8a2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9da814fda0 0x7f9da8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:44.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.699+0000 7f9dac8a2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9da80abe20 0x7f9da81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:44.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.699+0000 7f9dac8a2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da80c2f30 con 0x7f9da814f9d0 2023-03-03T16:48:44.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.699+0000 7f9dac8a2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da80c31d0 con 0x7f9da814fda0 2023-03-03T16:48:44.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.699+0000 7f9dac8a2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da81524a0 con 0x7f9da80abe20 2023-03-03T16:48:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.699+0000 7f9da5d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9da814fda0 0x7f9da8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.699+0000 7f9da659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9da814f9d0 0x7f9da8071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52666/0 (socket says 172.21.15.7:52666) 2023-03-03T16:48:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.699+0000 7f9da659c700 1 -- 172.21.15.7:0/158580595 learned_addr learned my addr 172.21.15.7:0/158580595 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:48:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da6d9d700 1 --2- 172.21.15.7:0/158580595 >> v2:172.21.15.143:3300/0 conn(0x7f9da80abe20 0x7f9da81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da559a700 1 -- 172.21.15.7:0/158580595 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208303404 0 0) 0x7f9da80c2f30 con 0x7f9da814f9d0 2023-03-03T16:48:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da559a700 1 -- 172.21.15.7:0/158580595 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d900035b0 con 0x7f9da814f9d0 2023-03-03T16:48:44.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da5d9b700 1 -- 172.21.15.7:0/158580595 >> v2:172.21.15.143:3300/0 conn(0x7f9da80abe20 msgr2=0x7f9da81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:48:44.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da5d9b700 1 --2- 172.21.15.7:0/158580595 >> v2:172.21.15.143:3300/0 conn(0x7f9da80abe20 0x7f9da81529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:44.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da5d9b700 1 -- 172.21.15.7:0/158580595 >> v1:172.21.15.7:6789/0 conn(0x7f9da814f9d0 legacy=0x7f9da8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:44.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da5d9b700 1 -- 172.21.15.7:0/158580595 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da80c34b0 con 0x7f9da814fda0 2023-03-03T16:48:44.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da5d9b700 1 --2- 172.21.15.7:0/158580595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da814fda0 0x7f9da8156430 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f9da8071230 tx=0x7f9d9c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a069448e3cbe9f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:44.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da559a700 1 -- 172.21.15.7:0/158580595 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c011830 con 0x7f9da814fda0 2023-03-03T16:48:44.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da559a700 1 -- 172.21.15.7:0/158580595 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c011990 con 0x7f9da814fda0 2023-03-03T16:48:44.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.700+0000 7f9da559a700 1 -- 172.21.15.7:0/158580595 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c011d20 con 0x7f9da814fda0 2023-03-03T16:48:44.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.701+0000 7f9dac8a2700 1 -- 172.21.15.7:0/158580595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da814fda0 msgr2=0x7f9da8156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.701+0000 7f9dac8a2700 1 --2- 172.21.15.7:0/158580595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da814fda0 0x7f9da8156430 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f9da8071230 tx=0x7f9d9c009b20 comp rx=0 tx=0).stop 2023-03-03T16:48:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.701+0000 7f9dac8a2700 1 -- 172.21.15.7:0/158580595 shutdown_connections 2023-03-03T16:48:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.701+0000 7f9dac8a2700 1 --2- 172.21.15.7:0/158580595 >> v2:172.21.15.143:3300/0 conn(0x7f9da80abe20 0x7f9da81529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.701+0000 7f9dac8a2700 1 --2- 172.21.15.7:0/158580595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da814fda0 0x7f9da8156430 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.701+0000 7f9dac8a2700 1 -- 172.21.15.7:0/158580595 >> 172.21.15.7:0/158580595 conn(0x7f9da80c8900 msgr2=0x7f9da80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:48:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.701+0000 7f9dac8a2700 1 -- 172.21.15.7:0/158580595 shutdown_connections 2023-03-03T16:48:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.701+0000 7f9dac8a2700 1 -- 172.21.15.7:0/158580595 wait complete. 2023-03-03T16:48:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.702+0000 7f9dac8a2700 1 Processor -- start 2023-03-03T16:48:44.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.702+0000 7f9dac8a2700 1 -- start start 2023-03-03T16:48:44.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9dac8a2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da80abe20 0x7f9da81549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:44.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9dac8a2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9da814f9d0 0x7f9da814b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:44.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9dac8a2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da814fda0 0x7f9da814b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:44.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9dac8a2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80bdba0 con 0x7f9da80abe20 2023-03-03T16:48:44.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9dac8a2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80be1c0 con 0x7f9da814fda0 2023-03-03T16:48:44.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9dac8a2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80bd3d0 con 0x7f9da814f9d0 2023-03-03T16:48:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da80abe20 0x7f9da81549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da80abe20 0x7f9da81549c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58692/0 (socket says 172.21.15.7:58692) 2023-03-03T16:48:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da659c700 1 -- 172.21.15.7:0/3391636828 learned_addr learned my addr 172.21.15.7:0/3391636828 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:48:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da6d9d700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da814fda0 0x7f9da814b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da5d9b700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9da814f9d0 0x7f9da814b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da659c700 1 -- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9da814f9d0 msgr2=0x7f9da814b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da659c700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9da814f9d0 0x7f9da814b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:44.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da659c700 1 -- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da814fda0 msgr2=0x7f9da814b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:44.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da659c700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da814fda0 0x7f9da814b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:44.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da659c700 1 -- 172.21.15.7:0/3391636828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c0094a0 con 0x7f9da80abe20 2023-03-03T16:48:44.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.703+0000 7f9da659c700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da80abe20 0x7f9da81549c0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f9d9800c700 tx=0x7f9d9800cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:44.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.704+0000 7f9d96ffd700 1 -- 172.21.15.7:0/3391636828 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d98005d70 con 0x7f9da80abe20 2023-03-03T16:48:44.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.704+0000 7f9d96ffd700 1 -- 172.21.15.7:0/3391636828 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d980070d0 con 0x7f9da80abe20 2023-03-03T16:48:44.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.704+0000 7f9dac8a2700 1 -- 172.21.15.7:0/3391636828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da80ac350 con 0x7f9da80abe20 2023-03-03T16:48:44.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.704+0000 7f9d96ffd700 1 -- 172.21.15.7:0/3391636828 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d98007460 con 0x7f9da80abe20 2023-03-03T16:48:44.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.704+0000 7f9dac8a2700 1 -- 172.21.15.7:0/3391636828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da80ab630 con 0x7f9da80abe20 2023-03-03T16:48:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.705+0000 7f9dac8a2700 1 -- 172.21.15.7:0/3391636828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da8071fa0 con 0x7f9da80abe20 2023-03-03T16:48:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.705+0000 7f9d96ffd700 1 -- 172.21.15.7:0/3391636828 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9d98019030 con 0x7f9da80abe20 2023-03-03T16:48:44.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.706+0000 7f9d96ffd700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d880682f0 0x7f9d8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:48:44.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.706+0000 7f9d96ffd700 1 -- 172.21.15.7:0/3391636828 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9d98085610 con 0x7f9da80abe20 2023-03-03T16:48:44.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.706+0000 7f9da5d9b700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d880682f0 0x7f9d8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:48:44.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.706+0000 7f9da5d9b700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d880682f0 0x7f9d8806a7a0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f9da81501d0 tx=0x7f9d9c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:48:44.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.710+0000 7f9d96ffd700 1 -- 172.21.15.7:0/3391636828 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9d98051930 con 0x7f9da80abe20 2023-03-03T16:48:44.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.948+0000 7f9dac8a2700 1 -- 172.21.15.7:0/3391636828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9da814f7f0 con 0x7f9da80abe20 2023-03-03T16:48:44.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.949+0000 7f9d96ffd700 1 -- 172.21.15.7:0/3391636828 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9d98054f50 con 0x7f9da80abe20 2023-03-03T16:48:44.950 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:48:44.950 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:48:44.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 -- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d880682f0 msgr2=0x7f9d8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:44.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d880682f0 0x7f9d8806a7a0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f9da81501d0 tx=0x7f9d9c008030 comp rx=0 tx=0).stop 2023-03-03T16:48:44.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 -- 172.21.15.7:0/3391636828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da80abe20 msgr2=0x7f9da81549c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:48:44.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da80abe20 0x7f9da81549c0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f9d9800c700 tx=0x7f9d9800cac0 comp rx=0 tx=0).stop 2023-03-03T16:48:44.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 -- 172.21.15.7:0/3391636828 shutdown_connections 2023-03-03T16:48:44.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9da814fda0 0x7f9da814b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:44.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d880682f0 0x7f9d8806a7a0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:44.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9da80abe20 0x7f9da81549c0 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:44.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 --2- 172.21.15.7:0/3391636828 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9da814f9d0 0x7f9da814b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:48:44.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 -- 172.21.15.7:0/3391636828 >> 172.21.15.7:0/3391636828 conn(0x7f9da80c8900 msgr2=0x7f9da8151670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:48:44.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 -- 172.21.15.7:0/3391636828 shutdown_connections 2023-03-03T16:48:44.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:48:44.951+0000 7f9dac8a2700 1 -- 172.21.15.7:0/3391636828 wait complete. 2023-03-03T16:48:44.954 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:48:44.964 INFO:tasks.mds_thrash.fs.[b]:waiting for 81 secs before thrashing 2023-03-03T16:48:49.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:48:49.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:19.444 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:19.447 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:49.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:49.526 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:49:51.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:49:51.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.442+0000 7f2ce80af700 1 Processor -- start 2023-03-03T16:49:51.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.442+0000 7f2ce80af700 1 -- start start 2023-03-03T16:49:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.443+0000 7f2ce80af700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ce014fda0 0x7f2ce0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:49:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.443+0000 7f2ce80af700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce00c2f30 con 0x7f2ce014f9d0 2023-03-03T16:49:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.443+0000 7f2ce80af700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce00c31d0 con 0x7f2ce00abe20 2023-03-03T16:49:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.443+0000 7f2ce80af700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce01524a0 con 0x7f2ce014fda0 2023-03-03T16:49:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.443+0000 7f2ce5e4b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2ce014f9d0 0x7f2ce0071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52718/0 (socket says 172.21.15.7:52718) 2023-03-03T16:49:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.443+0000 7f2ce564a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ce014fda0 0x7f2ce0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:49:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.443+0000 7f2ce5e4b700 1 -- 172.21.15.7:0/2066074469 learned_addr learned my addr 172.21.15.7:0/2066074469 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:49:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.444+0000 7f2ce4e49700 1 -- 172.21.15.7:0/2066074469 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133799831 0 0) 0x7f2ce00c2f30 con 0x7f2ce014f9d0 2023-03-03T16:49:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.444+0000 7f2ce4e49700 1 -- 172.21.15.7:0/2066074469 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ccc0035b0 con 0x7f2ce014f9d0 2023-03-03T16:49:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.444+0000 7f2ce564a700 1 -- 172.21.15.7:0/2066074469 >> v1:172.21.15.143:6789/0 conn(0x7f2ce00abe20 legacy=0x7f2ce01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.444+0000 7f2ce564a700 1 -- 172.21.15.7:0/2066074469 >> v1:172.21.15.7:6789/0 conn(0x7f2ce014f9d0 legacy=0x7f2ce0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.444+0000 7f2ce564a700 1 -- 172.21.15.7:0/2066074469 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce00c34b0 con 0x7f2ce014fda0 2023-03-03T16:49:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.444+0000 7f2ce564a700 1 --2- 172.21.15.7:0/2066074469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 0x7f2ce0156430 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009840 tx=0x7f2cdc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152cd16f8e20f01a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:49:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.444+0000 7f2ce4e49700 1 -- 172.21.15.7:0/2066074469 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc011830 con 0x7f2ce014fda0 2023-03-03T16:49:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.444+0000 7f2ce4e49700 1 -- 172.21.15.7:0/2066074469 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc011990 con 0x7f2ce014fda0 2023-03-03T16:49:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.444+0000 7f2ce4e49700 1 -- 172.21.15.7:0/2066074469 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc011d20 con 0x7f2ce014fda0 2023-03-03T16:49:51.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.445+0000 7f2ce80af700 1 -- 172.21.15.7:0/2066074469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 msgr2=0x7f2ce0156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:51.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.445+0000 7f2ce80af700 1 --2- 172.21.15.7:0/2066074469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 0x7f2ce0156430 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009840 tx=0x7f2cdc009b50 comp rx=0 tx=0).stop 2023-03-03T16:49:51.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.445+0000 7f2ce80af700 1 -- 172.21.15.7:0/2066074469 shutdown_connections 2023-03-03T16:49:51.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.445+0000 7f2ce80af700 1 --2- 172.21.15.7:0/2066074469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 0x7f2ce0156430 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:51.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.445+0000 7f2ce80af700 1 -- 172.21.15.7:0/2066074469 >> 172.21.15.7:0/2066074469 conn(0x7f2ce00c8900 msgr2=0x7f2ce00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:49:51.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.445+0000 7f2ce80af700 1 -- 172.21.15.7:0/2066074469 shutdown_connections 2023-03-03T16:49:51.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.445+0000 7f2ce80af700 1 -- 172.21.15.7:0/2066074469 wait complete. 2023-03-03T16:49:51.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.446+0000 7f2ce80af700 1 Processor -- start 2023-03-03T16:49:51.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.446+0000 7f2ce80af700 1 -- start start 2023-03-03T16:49:51.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce80af700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce00abe20 0x7f2ce006bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:49:51.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce80af700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce014f9d0 0x7f2ce01425a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:49:51.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce80af700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 0x7f2ce0142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:49:51.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce80af700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce006e410 con 0x7f2ce014fda0 2023-03-03T16:49:51.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce80af700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce01edba0 con 0x7f2ce00abe20 2023-03-03T16:49:51.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce80af700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce00c18b0 con 0x7f2ce014f9d0 2023-03-03T16:49:51.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce664c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 0x7f2ce0142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:49:51.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce664c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 0x7f2ce0142ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58748/0 (socket says 172.21.15.7:58748) 2023-03-03T16:49:51.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce664c700 1 -- 172.21.15.7:0/1059992254 learned_addr learned my addr 172.21.15.7:0/1059992254 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:49:51.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce564a700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce014f9d0 0x7f2ce01425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:49:51.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce5e4b700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce00abe20 0x7f2ce006bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:49:51.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce664c700 1 -- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce014f9d0 msgr2=0x7f2ce01425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:51.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce664c700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce014f9d0 0x7f2ce01425a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:51.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce664c700 1 -- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce00abe20 msgr2=0x7f2ce006bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:51.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce664c700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce00abe20 0x7f2ce006bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:51.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce664c700 1 -- 172.21.15.7:0/1059992254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc0094a0 con 0x7f2ce014fda0 2023-03-03T16:49:51.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.447+0000 7f2ce664c700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 0x7f2ce0142ae0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800cbc0 tx=0x7f2cd800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:49:51.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.448+0000 7f2cd67fc700 1 -- 172.21.15.7:0/1059992254 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8003c00 con 0x7f2ce014fda0 2023-03-03T16:49:51.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.448+0000 7f2cd67fc700 1 -- 172.21.15.7:0/1059992254 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8003d60 con 0x7f2ce014fda0 2023-03-03T16:49:51.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.448+0000 7f2ce80af700 1 -- 172.21.15.7:0/1059992254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce00ac350 con 0x7f2ce014fda0 2023-03-03T16:49:51.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.448+0000 7f2cd67fc700 1 -- 172.21.15.7:0/1059992254 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd800aa20 con 0x7f2ce014fda0 2023-03-03T16:49:51.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.448+0000 7f2ce80af700 1 -- 172.21.15.7:0/1059992254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce00ab630 con 0x7f2ce014fda0 2023-03-03T16:49:51.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.449+0000 7f2ce80af700 1 -- 172.21.15.7:0/1059992254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce0071fa0 con 0x7f2ce014fda0 2023-03-03T16:49:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.449+0000 7f2cd67fc700 1 -- 172.21.15.7:0/1059992254 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2cd8010050 con 0x7f2ce014fda0 2023-03-03T16:49:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.450+0000 7f2cd67fc700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc40682f0 0x7f2cc406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:49:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.450+0000 7f2cd67fc700 1 -- 172.21.15.7:0/1059992254 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2cd8085ec0 con 0x7f2ce014fda0 2023-03-03T16:49:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.450+0000 7f2ce5e4b700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc40682f0 0x7f2cc406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:49:51.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.450+0000 7f2ce5e4b700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc40682f0 0x7f2cc406a7a0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0007fd0 tx=0x7f2cd0007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:49:51.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.454+0000 7f2cd67fc700 1 -- 172.21.15.7:0/1059992254 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2cd8052260 con 0x7f2ce014fda0 2023-03-03T16:49:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.692+0000 7f2ce80af700 1 -- 172.21.15.7:0/1059992254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ce00c21e0 con 0x7f2ce014fda0 2023-03-03T16:49:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.692+0000 7f2cd67fc700 1 -- 172.21.15.7:0/1059992254 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2cd8055880 con 0x7f2ce014fda0 2023-03-03T16:49:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:49:51.694 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:49:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 -- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc40682f0 msgr2=0x7f2cc406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc40682f0 0x7f2cc406a7a0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0007fd0 tx=0x7f2cd0007dc0 comp rx=0 tx=0).stop 2023-03-03T16:49:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 -- 172.21.15.7:0/1059992254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 msgr2=0x7f2ce0142ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 0x7f2ce0142ae0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800cbc0 tx=0x7f2cd800ced0 comp rx=0 tx=0).stop 2023-03-03T16:49:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 -- 172.21.15.7:0/1059992254 shutdown_connections 2023-03-03T16:49:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc40682f0 0x7f2cc406a7a0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce014fda0 0x7f2ce0142ae0 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce00abe20 0x7f2ce006bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 --2- 172.21.15.7:0/1059992254 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce014f9d0 0x7f2ce01425a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 -- 172.21.15.7:0/1059992254 >> 172.21.15.7:0/1059992254 conn(0x7f2ce00c8900 msgr2=0x7f2ce014ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:49:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 -- 172.21.15.7:0/1059992254 shutdown_connections 2023-03-03T16:49:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:51.694+0000 7f2ce80af700 1 -- 172.21.15.7:0/1059992254 wait complete. 2023-03-03T16:49:51.697 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:49:51.707 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before thrashing 2023-03-03T16:49:52.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:49:52.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.847+0000 7fe61db01700 1 Processor -- start 2023-03-03T16:49:52.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.848+0000 7fe61db01700 1 -- start start 2023-03-03T16:49:52.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.848+0000 7fe61db01700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe61806a2b0 0x7fe61806a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:49:52.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.848+0000 7fe61db01700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180c1c10 con 0x7fe618157a40 2023-03-03T16:49:52.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.848+0000 7fe61db01700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe618154530 con 0x7fe6180ab0c0 2023-03-03T16:49:52.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.848+0000 7fe61db01700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe618154810 con 0x7fe61806a2b0 2023-03-03T16:49:52.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.848+0000 7fe617fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe618157a40 0x7fe618157ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52738/0 (socket says 172.21.15.7:52738) 2023-03-03T16:49:52.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.848+0000 7fe617fff700 1 -- 172.21.15.7:0/3704697581 learned_addr learned my addr 172.21.15.7:0/3704697581 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:49:52.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.849+0000 7fe616ffd700 1 --2- 172.21.15.7:0/3704697581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe61806a2b0 0x7fe61806a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:49:52.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.849+0000 7fe6167fc700 1 -- 172.21.15.7:0/3704697581 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336447252 0 0) 0x7fe610001490 con 0x7fe618157a40 2023-03-03T16:49:52.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.849+0000 7fe6167fc700 1 -- 172.21.15.7:0/3704697581 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6000035b0 con 0x7fe618157a40 2023-03-03T16:49:52.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.849+0000 7fe6167fc700 1 -- 172.21.15.7:0/3704697581 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084881848 0 0) 0x7fe618154530 con 0x7fe6180ab0c0 2023-03-03T16:49:52.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.849+0000 7fe6167fc700 1 -- 172.21.15.7:0/3704697581 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe610001490 con 0x7fe6180ab0c0 2023-03-03T16:49:52.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.849+0000 7fe6167fc700 1 -- 172.21.15.7:0/3704697581 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2793939993 0 0) 0x7fe6000035b0 con 0x7fe618157a40 2023-03-03T16:49:52.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.849+0000 7fe616ffd700 1 -- 172.21.15.7:0/3704697581 >> v1:172.21.15.143:6789/0 conn(0x7fe6180ab0c0 legacy=0x7fe6180c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:52.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.849+0000 7fe616ffd700 1 -- 172.21.15.7:0/3704697581 >> v1:172.21.15.7:6789/0 conn(0x7fe618157a40 legacy=0x7fe618157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:52.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.849+0000 7fe616ffd700 1 -- 172.21.15.7:0/3704697581 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe618154af0 con 0x7fe61806a2b0 2023-03-03T16:49:52.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.849+0000 7fe616ffd700 1 --2- 172.21.15.7:0/3704697581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe61806a2b0 0x7fe61806a720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009870 tx=0x7fe60c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20d4af68d33dd2ae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:49:52.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.850+0000 7fe6167fc700 1 -- 172.21.15.7:0/3704697581 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe610001b80 con 0x7fe61806a2b0 2023-03-03T16:49:52.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.850+0000 7fe6167fc700 1 -- 172.21.15.7:0/3704697581 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60c004030 con 0x7fe61806a2b0 2023-03-03T16:49:52.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.850+0000 7fe6167fc700 1 -- 172.21.15.7:0/3704697581 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c0043c0 con 0x7fe61806a2b0 2023-03-03T16:49:52.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.850+0000 7fe61db01700 1 -- 172.21.15.7:0/3704697581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe61806a2b0 msgr2=0x7fe61806a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:52.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.850+0000 7fe61db01700 1 --2- 172.21.15.7:0/3704697581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe61806a2b0 0x7fe61806a720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009870 tx=0x7fe60c009b80 comp rx=0 tx=0).stop 2023-03-03T16:49:52.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.850+0000 7fe61db01700 1 -- 172.21.15.7:0/3704697581 shutdown_connections 2023-03-03T16:49:52.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.850+0000 7fe61db01700 1 --2- 172.21.15.7:0/3704697581 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe61806a2b0 0x7fe61806a720 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:52.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.850+0000 7fe61db01700 1 -- 172.21.15.7:0/3704697581 >> 172.21.15.7:0/3704697581 conn(0x7fe6180a9430 msgr2=0x7fe6180c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:49:52.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.850+0000 7fe61db01700 1 -- 172.21.15.7:0/3704697581 shutdown_connections 2023-03-03T16:49:52.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.850+0000 7fe61db01700 1 -- 172.21.15.7:0/3704697581 wait complete. 2023-03-03T16:49:52.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.852+0000 7fe61db01700 1 Processor -- start 2023-03-03T16:49:52.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.852+0000 7fe61db01700 1 -- start start 2023-03-03T16:49:52.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe61db01700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe61806a2b0 0x7fe618156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:49:52.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe61db01700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6180ab0c0 0x7fe618155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:49:52.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe61db01700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe618157a40 0x7fe618152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:49:52.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe61db01700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61806c090 con 0x7fe618157a40 2023-03-03T16:49:52.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe61db01700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61806cd60 con 0x7fe61806a2b0 2023-03-03T16:49:52.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe61db01700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe618150da0 con 0x7fe6180ab0c0 2023-03-03T16:49:52.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe617fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe618157a40 0x7fe618152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:49:52.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe617fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe618157a40 0x7fe618152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58764/0 (socket says 172.21.15.7:58764) 2023-03-03T16:49:52.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe617fff700 1 -- 172.21.15.7:0/3178947079 learned_addr learned my addr 172.21.15.7:0/3178947079 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:49:52.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe6177fe700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe61806a2b0 0x7fe618156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:49:52.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe616ffd700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6180ab0c0 0x7fe618155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:49:52.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe617fff700 1 -- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6180ab0c0 msgr2=0x7fe618155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:52.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe617fff700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6180ab0c0 0x7fe618155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:52.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe617fff700 1 -- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe61806a2b0 msgr2=0x7fe618156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:52.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe617fff700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe61806a2b0 0x7fe618156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:52.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.853+0000 7fe617fff700 1 -- 172.21.15.7:0/3178947079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c0094a0 con 0x7fe618157a40 2023-03-03T16:49:52.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.854+0000 7fe617fff700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe618157a40 0x7fe618152290 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7fe610001bc0 tx=0x7fe61000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:49:52.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.854+0000 7fe5fffff700 1 -- 172.21.15.7:0/3178947079 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000acf0 con 0x7fe618157a40 2023-03-03T16:49:52.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.854+0000 7fe61db01700 1 -- 172.21.15.7:0/3178947079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6181550b0 con 0x7fe618157a40 2023-03-03T16:49:52.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.854+0000 7fe5fffff700 1 -- 172.21.15.7:0/3178947079 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61000ae50 con 0x7fe618157a40 2023-03-03T16:49:52.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.854+0000 7fe5fffff700 1 -- 172.21.15.7:0/3178947079 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6100072d0 con 0x7fe618157a40 2023-03-03T16:49:52.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.854+0000 7fe61db01700 1 -- 172.21.15.7:0/3178947079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe618154dd0 con 0x7fe618157a40 2023-03-03T16:49:52.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.855+0000 7fe61db01700 1 -- 172.21.15.7:0/3178947079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6181585d0 con 0x7fe618157a40 2023-03-03T16:49:52.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.855+0000 7fe5fffff700 1 -- 172.21.15.7:0/3178947079 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe610010070 con 0x7fe618157a40 2023-03-03T16:49:52.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.856+0000 7fe5fffff700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6040682f0 0x7fe60406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:49:52.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.856+0000 7fe5fffff700 1 -- 172.21.15.7:0/3178947079 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe610085cf0 con 0x7fe618157a40 2023-03-03T16:49:52.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.856+0000 7fe6177fe700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6040682f0 0x7fe60406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:49:52.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.856+0000 7fe6177fe700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6040682f0 0x7fe60406a7a0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fe6080011e0 tx=0x7fe608009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:49:52.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:52.860+0000 7fe5fffff700 1 -- 172.21.15.7:0/3178947079 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe610051f20 con 0x7fe618157a40 2023-03-03T16:49:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.099+0000 7fe61db01700 1 -- 172.21.15.7:0/3178947079 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6180c1c10 con 0x7fe618157a40 2023-03-03T16:49:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.099+0000 7fe5fffff700 1 -- 172.21.15.7:0/3178947079 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe6180c1c10 con 0x7fe618157a40 2023-03-03T16:49:53.101 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:49:53.102 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:49:53.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 -- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6040682f0 msgr2=0x7fe60406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6040682f0 0x7fe60406a7a0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fe6080011e0 tx=0x7fe608009040 comp rx=0 tx=0).stop 2023-03-03T16:49:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 -- 172.21.15.7:0/3178947079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe618157a40 msgr2=0x7fe618152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:49:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe618157a40 0x7fe618152290 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7fe610001bc0 tx=0x7fe61000a340 comp rx=0 tx=0).stop 2023-03-03T16:49:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 -- 172.21.15.7:0/3178947079 shutdown_connections 2023-03-03T16:49:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe6040682f0 0x7fe60406a7a0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe618157a40 0x7fe618152290 unknown :-1 s=CLOSED pgs=2713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe61806a2b0 0x7fe618156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 --2- 172.21.15.7:0/3178947079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe6180ab0c0 0x7fe618155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:49:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 -- 172.21.15.7:0/3178947079 >> 172.21.15.7:0/3178947079 conn(0x7fe6180a9430 msgr2=0x7fe6180a7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:49:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 -- 172.21.15.7:0/3178947079 shutdown_connections 2023-03-03T16:49:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:49:53.102+0000 7fe61db01700 1 -- 172.21.15.7:0/3178947079 wait complete. 2023-03-03T16:49:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:49:53.115 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before thrashing 2023-03-03T16:49:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T16:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T16:50:00.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:50:00.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.255+0000 7f923e378700 1 Processor -- start 2023-03-03T16:50:00.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.255+0000 7f923e378700 1 -- start start 2023-03-03T16:50:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f923e378700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814f9d0 0x7f9238156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f923e378700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f923814fda0 0x7f9238071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f923e378700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92380c2f30 con 0x7f92380abe20 2023-03-03T16:50:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f923e378700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92380c31d0 con 0x7f923814fda0 2023-03-03T16:50:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f923e378700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92381524a0 con 0x7f923814f9d0 2023-03-03T16:50:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f92377fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f923814fda0 0x7f9238071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f92377fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f923814fda0 0x7f9238071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58772/0 (socket says 172.21.15.7:58772) 2023-03-03T16:50:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f92377fe700 1 -- 172.21.15.7:0/1887341227 learned_addr learned my addr 172.21.15.7:0/1887341227 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:50:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f9237fff700 1 --2- 172.21.15.7:0/1887341227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814f9d0 0x7f9238156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f92377fe700 1 -- 172.21.15.7:0/1887341227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814f9d0 msgr2=0x7f9238156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:50:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f92377fe700 1 --2- 172.21.15.7:0/1887341227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814f9d0 0x7f9238156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f92377fe700 1 -- 172.21.15.7:0/1887341227 >> v1:172.21.15.143:6789/0 conn(0x7f92380abe20 legacy=0x7f92381529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.256+0000 7f92377fe700 1 -- 172.21.15.7:0/1887341227 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92380c34b0 con 0x7f923814fda0 2023-03-03T16:50:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f92377fe700 1 --2- 172.21.15.7:0/1887341227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814fda0 0x7f9238071890 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f92200098f0 tx=0x7f9220009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=363a9b97aa94140 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f9236ffd700 1 -- 172.21.15.7:0/1887341227 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220011830 con 0x7f923814fda0 2023-03-03T16:50:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f9236ffd700 1 -- 172.21.15.7:0/1887341227 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9220011990 con 0x7f923814fda0 2023-03-03T16:50:00.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f9236ffd700 1 -- 172.21.15.7:0/1887341227 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220011d20 con 0x7f923814fda0 2023-03-03T16:50:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f923e378700 1 -- 172.21.15.7:0/1887341227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814fda0 msgr2=0x7f9238071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f923e378700 1 --2- 172.21.15.7:0/1887341227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814fda0 0x7f9238071890 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f92200098f0 tx=0x7f9220009c00 comp rx=0 tx=0).stop 2023-03-03T16:50:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f923e378700 1 -- 172.21.15.7:0/1887341227 shutdown_connections 2023-03-03T16:50:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f923e378700 1 --2- 172.21.15.7:0/1887341227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814f9d0 0x7f9238156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f923e378700 1 --2- 172.21.15.7:0/1887341227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814fda0 0x7f9238071890 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f923e378700 1 -- 172.21.15.7:0/1887341227 >> 172.21.15.7:0/1887341227 conn(0x7f92380c8900 msgr2=0x7f92380c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:50:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f923e378700 1 -- 172.21.15.7:0/1887341227 shutdown_connections 2023-03-03T16:50:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.257+0000 7f923e378700 1 -- 172.21.15.7:0/1887341227 wait complete. 2023-03-03T16:50:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.259+0000 7f923e378700 1 Processor -- start 2023-03-03T16:50:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.259+0000 7f923e378700 1 -- start start 2023-03-03T16:50:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.259+0000 7f923e378700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92380abe20 0x7f92380bd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.259+0000 7f923e378700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814f9d0 0x7f92380c2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.259+0000 7f923e378700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814fda0 0x7f923806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.259+0000 7f923e378700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92380bf780 con 0x7f923814f9d0 2023-03-03T16:50:00.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.259+0000 7f923e378700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923814cb30 con 0x7f92380abe20 2023-03-03T16:50:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.259+0000 7f923e378700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923806d510 con 0x7f923814fda0 2023-03-03T16:50:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f92377fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814f9d0 0x7f92380c2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f92377fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814f9d0 0x7f92380c2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58780/0 (socket says 172.21.15.7:58780) 2023-03-03T16:50:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f92377fe700 1 -- 172.21.15.7:0/4078173615 learned_addr learned my addr 172.21.15.7:0/4078173615 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:50:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f923c915700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814fda0 0x7f923806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f9237fff700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92380abe20 0x7f92380bd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f92377fe700 1 -- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814fda0 msgr2=0x7f923806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:00.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f92377fe700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814fda0 0x7f923806d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f92377fe700 1 -- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92380abe20 msgr2=0x7f92380bd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f92377fe700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92380abe20 0x7f92380bd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f92377fe700 1 -- 172.21.15.7:0/4078173615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9220009550 con 0x7f923814f9d0 2023-03-03T16:50:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f923c915700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814fda0 0x7f923806d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:50:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f92377fe700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814f9d0 0x7f92380c2040 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f9220000c00 tx=0x7f9220004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f9234ff9700 1 -- 172.21.15.7:0/4078173615 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220004570 con 0x7f923814f9d0 2023-03-03T16:50:00.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f9234ff9700 1 -- 172.21.15.7:0/4078173615 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922001f070 con 0x7f923814f9d0 2023-03-03T16:50:00.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f923e378700 1 -- 172.21.15.7:0/4078173615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92380ac350 con 0x7f923814f9d0 2023-03-03T16:50:00.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f9234ff9700 1 -- 172.21.15.7:0/4078173615 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92200038c0 con 0x7f923814f9d0 2023-03-03T16:50:00.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.260+0000 7f923e378700 1 -- 172.21.15.7:0/4078173615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92380ab630 con 0x7f923814f9d0 2023-03-03T16:50:00.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.261+0000 7f923e378700 1 -- 172.21.15.7:0/4078173615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92380c3e10 con 0x7f923814f9d0 2023-03-03T16:50:00.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.262+0000 7f9234ff9700 1 -- 172.21.15.7:0/4078173615 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f922002a020 con 0x7f923814f9d0 2023-03-03T16:50:00.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.262+0000 7f9234ff9700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92180682f0 0x7f921806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:00.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.262+0000 7f9234ff9700 1 -- 172.21.15.7:0/4078173615 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f92200876c0 con 0x7f923814f9d0 2023-03-03T16:50:00.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.262+0000 7f9237fff700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92180682f0 0x7f921806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:00.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.263+0000 7f9237fff700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92180682f0 0x7f921806a7a0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f923806fd30 tx=0x7f9228008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:00.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.266+0000 7f9234ff9700 1 -- 172.21.15.7:0/4078173615 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9220053a60 con 0x7f923814f9d0 2023-03-03T16:50:00.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.506+0000 7f923e378700 1 -- 172.21.15.7:0/4078173615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f923806d680 con 0x7f923814f9d0 2023-03-03T16:50:00.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.507+0000 7f9234ff9700 1 -- 172.21.15.7:0/4078173615 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9220057080 con 0x7f923814f9d0 2023-03-03T16:50:00.508 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:50:00.508 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:50:00.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 -- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92180682f0 msgr2=0x7f921806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:00.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92180682f0 0x7f921806a7a0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f923806fd30 tx=0x7f9228008040 comp rx=0 tx=0).stop 2023-03-03T16:50:00.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 -- 172.21.15.7:0/4078173615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814f9d0 msgr2=0x7f92380c2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:00.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814f9d0 0x7f92380c2040 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f9220000c00 tx=0x7f9220004750 comp rx=0 tx=0).stop 2023-03-03T16:50:00.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 -- 172.21.15.7:0/4078173615 shutdown_connections 2023-03-03T16:50:00.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f923814fda0 0x7f923806d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:00.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92380abe20 0x7f92380bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:00.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f92180682f0 0x7f921806a7a0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:00.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 --2- 172.21.15.7:0/4078173615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f923814f9d0 0x7f92380c2040 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:00.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 -- 172.21.15.7:0/4078173615 >> 172.21.15.7:0/4078173615 conn(0x7f92380c8900 msgr2=0x7f9238151cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:50:00.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 -- 172.21.15.7:0/4078173615 shutdown_connections 2023-03-03T16:50:00.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:00.509+0000 7f923e378700 1 -- 172.21.15.7:0/4078173615 wait complete. 2023-03-03T16:50:00.512 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:50:00.522 INFO:tasks.mds_thrash.fs.[a]:waiting for 89 secs before thrashing 2023-03-03T16:50:05.965 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:50:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.104+0000 7faeda914700 1 Processor -- start 2023-03-03T16:50:06.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.104+0000 7faeda914700 1 -- start start 2023-03-03T16:50:06.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.105+0000 7faeda914700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 0x7faed4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:06.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.105+0000 7faeda914700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faed414fda0 0x7faed4071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:06.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.105+0000 7faeda914700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faed40abe20 0x7faed41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:06.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.105+0000 7faeda914700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed40c2f30 con 0x7faed40abe20 2023-03-03T16:50:06.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.105+0000 7faeda914700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed40c31d0 con 0x7faed414fda0 2023-03-03T16:50:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.105+0000 7faeda914700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed41524a0 con 0x7faed414f9d0 2023-03-03T16:50:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.105+0000 7faed8eb1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faed40abe20 0x7faed41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.105+0000 7faed8eb1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faed40abe20 0x7faed41529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58798/0 (socket says 172.21.15.7:58798) 2023-03-03T16:50:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.105+0000 7faed8eb1700 1 -- 172.21.15.7:0/1957499769 learned_addr learned my addr 172.21.15.7:0/1957499769 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:50:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed37fe700 1 --2- 172.21.15.7:0/1957499769 >> v2:172.21.15.143:3300/0 conn(0x7faed414fda0 0x7faed4071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed3fff700 1 --2- 172.21.15.7:0/1957499769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 0x7faed4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:06.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed8eb1700 1 -- 172.21.15.7:0/1957499769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 msgr2=0x7faed4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:50:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed8eb1700 1 --2- 172.21.15.7:0/1957499769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 0x7faed4156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed8eb1700 1 -- 172.21.15.7:0/1957499769 >> v2:172.21.15.143:3300/0 conn(0x7faed414fda0 msgr2=0x7faed4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:50:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed8eb1700 1 --2- 172.21.15.7:0/1957499769 >> v2:172.21.15.143:3300/0 conn(0x7faed414fda0 0x7faed4071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed8eb1700 1 -- 172.21.15.7:0/1957499769 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed40c34b0 con 0x7faed40abe20 2023-03-03T16:50:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed37fe700 1 --2- 172.21.15.7:0/1957499769 >> v2:172.21.15.143:3300/0 conn(0x7faed414fda0 0x7faed4071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:50:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed8eb1700 1 --2- 172.21.15.7:0/1957499769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed40abe20 0x7faed41529c0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7faed40ab220 tx=0x7faecc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f6399e22aae810 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed2ffd700 1 -- 172.21.15.7:0/1957499769 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc00e070 con 0x7faed40abe20 2023-03-03T16:50:06.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed2ffd700 1 -- 172.21.15.7:0/1957499769 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc00b9c0 con 0x7faed40abe20 2023-03-03T16:50:06.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.106+0000 7faed2ffd700 1 -- 172.21.15.7:0/1957499769 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc00bd70 con 0x7faed40abe20 2023-03-03T16:50:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.107+0000 7faeda914700 1 -- 172.21.15.7:0/1957499769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed40abe20 msgr2=0x7faed41529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.107+0000 7faeda914700 1 --2- 172.21.15.7:0/1957499769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed40abe20 0x7faed41529c0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7faed40ab220 tx=0x7faecc00b330 comp rx=0 tx=0).stop 2023-03-03T16:50:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.107+0000 7faeda914700 1 -- 172.21.15.7:0/1957499769 shutdown_connections 2023-03-03T16:50:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.107+0000 7faeda914700 1 --2- 172.21.15.7:0/1957499769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed40abe20 0x7faed41529c0 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.107+0000 7faeda914700 1 --2- 172.21.15.7:0/1957499769 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 0x7faed4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.107+0000 7faeda914700 1 --2- 172.21.15.7:0/1957499769 >> v2:172.21.15.143:3300/0 conn(0x7faed414fda0 0x7faed4071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.107+0000 7faeda914700 1 -- 172.21.15.7:0/1957499769 >> 172.21.15.7:0/1957499769 conn(0x7faed40c8900 msgr2=0x7faed40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:50:06.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.107+0000 7faeda914700 1 -- 172.21.15.7:0/1957499769 shutdown_connections 2023-03-03T16:50:06.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.107+0000 7faeda914700 1 -- 172.21.15.7:0/1957499769 wait complete. 2023-03-03T16:50:06.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.108+0000 7faeda914700 1 Processor -- start 2023-03-03T16:50:06.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.109+0000 7faeda914700 1 -- start start 2023-03-03T16:50:06.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.109+0000 7faeda914700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faed40abe20 0x7faed41eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.109+0000 7faeda914700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 0x7faed4155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.109+0000 7faeda914700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed414fda0 0x7faed406cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.109+0000 7faeda914700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed406c4c0 con 0x7faed414fda0 2023-03-03T16:50:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.109+0000 7faeda914700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed406c760 con 0x7faed40abe20 2023-03-03T16:50:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.109+0000 7faeda914700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed41eb990 con 0x7faed414f9d0 2023-03-03T16:50:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed8eb1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed414fda0 0x7faed406cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:06.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed8eb1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed414fda0 0x7faed406cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58804/0 (socket says 172.21.15.7:58804) 2023-03-03T16:50:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed8eb1700 1 -- 172.21.15.7:0/2878249951 learned_addr learned my addr 172.21.15.7:0/2878249951 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:50:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed3fff700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faed40abe20 0x7faed41eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed37fe700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 0x7faed4155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed8eb1700 1 -- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 msgr2=0x7faed4155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed8eb1700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 0x7faed4155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed8eb1700 1 -- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faed40abe20 msgr2=0x7faed41eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed8eb1700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faed40abe20 0x7faed41eb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed8eb1700 1 -- 172.21.15.7:0/2878249951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc009b30 con 0x7faed414fda0 2023-03-03T16:50:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed37fe700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 0x7faed4155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:50:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed8eb1700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed414fda0 0x7faed406cbf0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7faed4072520 tx=0x7faecc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed0ff9700 1 -- 172.21.15.7:0/2878249951 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc004310 con 0x7faed414fda0 2023-03-03T16:50:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed0ff9700 1 -- 172.21.15.7:0/2878249951 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc004470 con 0x7faed414fda0 2023-03-03T16:50:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faeda914700 1 -- 172.21.15.7:0/2878249951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed40ac350 con 0x7faed414fda0 2023-03-03T16:50:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faed0ff9700 1 -- 172.21.15.7:0/2878249951 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc004820 con 0x7faed414fda0 2023-03-03T16:50:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.110+0000 7faeda914700 1 -- 172.21.15.7:0/2878249951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed40ab630 con 0x7faed414fda0 2023-03-03T16:50:06.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.111+0000 7faeda914700 1 -- 172.21.15.7:0/2878249951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed40c3e10 con 0x7faed414fda0 2023-03-03T16:50:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.112+0000 7faed0ff9700 1 -- 172.21.15.7:0/2878249951 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faecc02b070 con 0x7faed414fda0 2023-03-03T16:50:06.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.112+0000 7faed0ff9700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb4068320 0x7faeb406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.112+0000 7faed0ff9700 1 -- 172.21.15.7:0/2878249951 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faecc087190 con 0x7faed414fda0 2023-03-03T16:50:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.113+0000 7faed3fff700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb4068320 0x7faeb406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.113+0000 7faed3fff700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb4068320 0x7faeb406a7d0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7faec400a990 tx=0x7faec4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:06.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.117+0000 7faed0ff9700 1 -- 172.21.15.7:0/2878249951 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faecc053530 con 0x7faed414fda0 2023-03-03T16:50:06.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.354+0000 7faeda914700 1 -- 172.21.15.7:0/2878249951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faed406c910 con 0x7faed414fda0 2023-03-03T16:50:06.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.355+0000 7faed0ff9700 1 -- 172.21.15.7:0/2878249951 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faed406c910 con 0x7faed414fda0 2023-03-03T16:50:06.355 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:50:06.355 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:50:06.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 -- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb4068320 msgr2=0x7faeb406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:06.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb4068320 0x7faeb406a7d0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7faec400a990 tx=0x7faec4008040 comp rx=0 tx=0).stop 2023-03-03T16:50:06.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 -- 172.21.15.7:0/2878249951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed414fda0 msgr2=0x7faed406cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:06.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed414fda0 0x7faed406cbf0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7faed4072520 tx=0x7faecc004ad0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 -- 172.21.15.7:0/2878249951 shutdown_connections 2023-03-03T16:50:06.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb4068320 0x7faeb406a7d0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faed414fda0 0x7faed406cbf0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faed40abe20 0x7faed41eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 --2- 172.21.15.7:0/2878249951 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faed414f9d0 0x7faed4155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:06.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 -- 172.21.15.7:0/2878249951 >> 172.21.15.7:0/2878249951 conn(0x7faed40c8900 msgr2=0x7faed40c7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:50:06.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 -- 172.21.15.7:0/2878249951 shutdown_connections 2023-03-03T16:50:06.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:06.357+0000 7faeda914700 1 -- 172.21.15.7:0/2878249951 wait complete. 2023-03-03T16:50:06.359 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:50:06.368 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T16:50:19.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:19.590 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:24.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:50:24.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.509+0000 7fb0e7b95700 1 Processor -- start 2023-03-03T16:50:24.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.509+0000 7fb0e7b95700 1 -- start start 2023-03-03T16:50:24.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.510+0000 7fb0e7b95700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0e014fcb0 0x7fb0e0070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:24.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.510+0000 7fb0e7b95700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb0e00c2730 0x7fb0e00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:24.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.510+0000 7fb0e7b95700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0e0073c30 0x7fb0e0155b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:24.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.510+0000 7fb0e7b95700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e0071ea0 con 0x7fb0e014fcb0 2023-03-03T16:50:24.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.510+0000 7fb0e7b95700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e00c2eb0 con 0x7fb0e00c2730 2023-03-03T16:50:24.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.510+0000 7fb0e7b95700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e00c3100 con 0x7fb0e0073c30 2023-03-03T16:50:24.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5931700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0e014fcb0 0x7fb0e0070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:24.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5931700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0e014fcb0 0x7fb0e0070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58824/0 (socket says 172.21.15.7:58824) 2023-03-03T16:50:24.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5931700 1 -- 172.21.15.7:0/2501384250 learned_addr learned my addr 172.21.15.7:0/2501384250 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:50:24.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5130700 1 --2- 172.21.15.7:0/2501384250 >> v2:172.21.15.143:3300/0 conn(0x7fb0e00c2730 0x7fb0e00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:24.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5931700 1 -- 172.21.15.7:0/2501384250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0e0073c30 msgr2=0x7fb0e0155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:50:24.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5931700 1 --2- 172.21.15.7:0/2501384250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0e0073c30 0x7fb0e0155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5931700 1 -- 172.21.15.7:0/2501384250 >> v2:172.21.15.143:3300/0 conn(0x7fb0e00c2730 msgr2=0x7fb0e00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:50:24.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5931700 1 --2- 172.21.15.7:0/2501384250 >> v2:172.21.15.143:3300/0 conn(0x7fb0e00c2730 0x7fb0e00736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5931700 1 -- 172.21.15.7:0/2501384250 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e00c33e0 con 0x7fb0e014fcb0 2023-03-03T16:50:24.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5130700 1 --2- 172.21.15.7:0/2501384250 >> v2:172.21.15.143:3300/0 conn(0x7fb0e00c2730 0x7fb0e00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:50:24.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e5931700 1 --2- 172.21.15.7:0/2501384250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 0x7fb0e0070240 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fb0e006e360 tx=0x7fb0d0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf6483fbc2a113f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:24.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e492f700 1 -- 172.21.15.7:0/2501384250 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000b870 con 0x7fb0e014fcb0 2023-03-03T16:50:24.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e492f700 1 -- 172.21.15.7:0/2501384250 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d000b9d0 con 0x7fb0e014fcb0 2023-03-03T16:50:24.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.511+0000 7fb0e492f700 1 -- 172.21.15.7:0/2501384250 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d000bd60 con 0x7fb0e014fcb0 2023-03-03T16:50:24.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.512+0000 7fb0e7b95700 1 -- 172.21.15.7:0/2501384250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 msgr2=0x7fb0e0070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:24.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.512+0000 7fb0e7b95700 1 --2- 172.21.15.7:0/2501384250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 0x7fb0e0070240 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fb0e006e360 tx=0x7fb0d0009bd0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.512+0000 7fb0e7b95700 1 -- 172.21.15.7:0/2501384250 shutdown_connections 2023-03-03T16:50:24.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.512+0000 7fb0e7b95700 1 --2- 172.21.15.7:0/2501384250 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0e0073c30 0x7fb0e0155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.512+0000 7fb0e7b95700 1 --2- 172.21.15.7:0/2501384250 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 0x7fb0e0070240 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.512+0000 7fb0e7b95700 1 --2- 172.21.15.7:0/2501384250 >> v2:172.21.15.143:3300/0 conn(0x7fb0e00c2730 0x7fb0e00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.512+0000 7fb0e7b95700 1 -- 172.21.15.7:0/2501384250 >> 172.21.15.7:0/2501384250 conn(0x7fb0e00a94d0 msgr2=0x7fb0e00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:50:24.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.512+0000 7fb0e7b95700 1 -- 172.21.15.7:0/2501384250 shutdown_connections 2023-03-03T16:50:24.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.512+0000 7fb0e7b95700 1 -- 172.21.15.7:0/2501384250 wait complete. 2023-03-03T16:50:24.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.514+0000 7fb0e7b95700 1 Processor -- start 2023-03-03T16:50:24.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.514+0000 7fb0e7b95700 1 -- start start 2023-03-03T16:50:24.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.514+0000 7fb0e7b95700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e0073c30 0x7fb0e01eb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:24.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e7b95700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0e00c2730 0x7fb0e01eb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:24.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e7b95700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 0x7fb0e00ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:24.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e7b95700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e00be5c0 con 0x7fb0e014fcb0 2023-03-03T16:50:24.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e7b95700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e0153370 con 0x7fb0e0073c30 2023-03-03T16:50:24.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e7b95700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e00aab30 con 0x7fb0e00c2730 2023-03-03T16:50:24.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e6132700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 0x7fb0e00ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:24.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e6132700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 0x7fb0e00ab2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58834/0 (socket says 172.21.15.7:58834) 2023-03-03T16:50:24.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e6132700 1 -- 172.21.15.7:0/680049991 learned_addr learned my addr 172.21.15.7:0/680049991 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:50:24.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e5931700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e0073c30 0x7fb0e01eb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e5130700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0e00c2730 0x7fb0e01eb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e6132700 1 -- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0e00c2730 msgr2=0x7fb0e01eb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e6132700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0e00c2730 0x7fb0e01eb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e6132700 1 -- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e0073c30 msgr2=0x7fb0e01eb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e6132700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e0073c30 0x7fb0e01eb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e6132700 1 -- 172.21.15.7:0/680049991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d0009550 con 0x7fb0e014fcb0 2023-03-03T16:50:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e6132700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 0x7fb0e00ab2f0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800b9a0 tx=0x7fb0d800bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0d67fc700 1 -- 172.21.15.7:0/680049991 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d800c9b0 con 0x7fb0e014fcb0 2023-03-03T16:50:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0d67fc700 1 -- 172.21.15.7:0/680049991 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d800cb30 con 0x7fb0e014fcb0 2023-03-03T16:50:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e7b95700 1 -- 172.21.15.7:0/680049991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e00c39a0 con 0x7fb0e014fcb0 2023-03-03T16:50:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0d67fc700 1 -- 172.21.15.7:0/680049991 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8014440 con 0x7fb0e014fcb0 2023-03-03T16:50:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.515+0000 7fb0e7b95700 1 -- 172.21.15.7:0/680049991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e00c36c0 con 0x7fb0e014fcb0 2023-03-03T16:50:24.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.516+0000 7fb0e7b95700 1 -- 172.21.15.7:0/680049991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e00708d0 con 0x7fb0e014fcb0 2023-03-03T16:50:24.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.517+0000 7fb0d67fc700 1 -- 172.21.15.7:0/680049991 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb0d801b030 con 0x7fb0e014fcb0 2023-03-03T16:50:24.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.517+0000 7fb0d67fc700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0c40682f0 0x7fb0c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:50:24.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.517+0000 7fb0d67fc700 1 -- 172.21.15.7:0/680049991 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb0d8006e40 con 0x7fb0e014fcb0 2023-03-03T16:50:24.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.518+0000 7fb0e5931700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0c40682f0 0x7fb0c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:50:24.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.518+0000 7fb0e5931700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0c40682f0 0x7fb0c406a7a0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0014040 tx=0x7fb0d0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:50:24.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.522+0000 7fb0d67fc700 1 -- 172.21.15.7:0/680049991 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb0d8052160 con 0x7fb0e014fcb0 2023-03-03T16:50:24.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.760+0000 7fb0e7b95700 1 -- 172.21.15.7:0/680049991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0e00aa990 con 0x7fb0e014fcb0 2023-03-03T16:50:24.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.760+0000 7fb0d67fc700 1 -- 172.21.15.7:0/680049991 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb0e00aa990 con 0x7fb0e014fcb0 2023-03-03T16:50:24.760 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:50:24.760 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:50:24.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.762+0000 7fb0e7b95700 1 -- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0c40682f0 msgr2=0x7fb0c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:24.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.762+0000 7fb0e7b95700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0c40682f0 0x7fb0c406a7a0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0014040 tx=0x7fb0d0008030 comp rx=0 tx=0).stop 2023-03-03T16:50:24.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.762+0000 7fb0e7b95700 1 -- 172.21.15.7:0/680049991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 msgr2=0x7fb0e00ab2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:50:24.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.762+0000 7fb0e7b95700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 0x7fb0e00ab2f0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800b9a0 tx=0x7fb0d800bd60 comp rx=0 tx=0).stop 2023-03-03T16:50:24.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.762+0000 7fb0e7b95700 1 -- 172.21.15.7:0/680049991 shutdown_connections 2023-03-03T16:50:24.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.762+0000 7fb0e7b95700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0c40682f0 0x7fb0c406a7a0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.762+0000 7fb0e7b95700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0e014fcb0 0x7fb0e00ab2f0 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.762+0000 7fb0e7b95700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0e0073c30 0x7fb0e01eb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.762+0000 7fb0e7b95700 1 --2- 172.21.15.7:0/680049991 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0e00c2730 0x7fb0e01eb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:50:24.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.762+0000 7fb0e7b95700 1 -- 172.21.15.7:0/680049991 >> 172.21.15.7:0/680049991 conn(0x7fb0e00a94d0 msgr2=0x7fb0e0153bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:50:24.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.763+0000 7fb0e7b95700 1 -- 172.21.15.7:0/680049991 shutdown_connections 2023-03-03T16:50:24.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:50:24.763+0000 7fb0e7b95700 1 -- 172.21.15.7:0/680049991 wait complete. 2023-03-03T16:50:24.764 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:50:24.774 INFO:tasks.mds_thrash.fs.[b]:waiting for 67 secs before thrashing 2023-03-03T16:50:49.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:50:49.662 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:19.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:19.750 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:29.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:51:29.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.662+0000 7fbdbf787700 1 Processor -- start 2023-03-03T16:51:29.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.662+0000 7fbdbf787700 1 -- start start 2023-03-03T16:51:29.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbf787700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb806d9f0 0x7fbdb8151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:29.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbf787700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbdb80c7690 0x7fbdb80c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:29.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbf787700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb81546d0 con 0x7fbdb80aa3e0 2023-03-03T16:51:29.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbf787700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb80abc00 con 0x7fbdb80c7690 2023-03-03T16:51:29.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbf787700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb8154920 con 0x7fbdb806d9f0 2023-03-03T16:51:29.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbdd24700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbdb80c7690 0x7fbdb80c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:29.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbcd22700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbdb80aa3e0 0x7fbdb80ab4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52870/0 (socket says 172.21.15.7:52870) 2023-03-03T16:51:29.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbcd22700 1 -- 172.21.15.7:0/4112004058 learned_addr learned my addr 172.21.15.7:0/4112004058 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:51:29.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbd523700 1 --2- 172.21.15.7:0/4112004058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb806d9f0 0x7fbdb8151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:29.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbdd24700 1 -- 172.21.15.7:0/4112004058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb806d9f0 msgr2=0x7fbdb8151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:51:29.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbdd24700 1 --2- 172.21.15.7:0/4112004058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb806d9f0 0x7fbdb8151cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:29.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbdd24700 1 -- 172.21.15.7:0/4112004058 >> v1:172.21.15.7:6789/0 conn(0x7fbdb80aa3e0 legacy=0x7fbdb80ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:29.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbdd24700 1 -- 172.21.15.7:0/4112004058 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb81511f0 con 0x7fbdb80c7690 2023-03-03T16:51:29.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdaffff700 1 -- 172.21.15.7:0/4112004058 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145346202 0 0) 0x7fbdb81546d0 con 0x7fbdb80aa3e0 2023-03-03T16:51:29.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.663+0000 7fbdbdd24700 1 --2- 172.21.15.7:0/4112004058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb80c7690 0x7fbdb80c7b30 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009d90 tx=0x7fbdb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eef44cfe160b582 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:51:29.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdaffff700 1 -- 172.21.15.7:0/4112004058 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb000e070 con 0x7fbdb80c7690 2023-03-03T16:51:29.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdaffff700 1 -- 172.21.15.7:0/4112004058 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb000bbd0 con 0x7fbdb80c7690 2023-03-03T16:51:29.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdaffff700 1 -- 172.21.15.7:0/4112004058 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0012430 con 0x7fbdb80c7690 2023-03-03T16:51:29.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdbf787700 1 -- 172.21.15.7:0/4112004058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb80c7690 msgr2=0x7fbdb80c7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:29.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdbf787700 1 --2- 172.21.15.7:0/4112004058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb80c7690 0x7fbdb80c7b30 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009d90 tx=0x7fbdb000b330 comp rx=0 tx=0).stop 2023-03-03T16:51:29.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdbf787700 1 -- 172.21.15.7:0/4112004058 shutdown_connections 2023-03-03T16:51:29.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdbf787700 1 --2- 172.21.15.7:0/4112004058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb80c7690 0x7fbdb80c7b30 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:29.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdbf787700 1 --2- 172.21.15.7:0/4112004058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb806d9f0 0x7fbdb8151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:29.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdbf787700 1 -- 172.21.15.7:0/4112004058 >> 172.21.15.7:0/4112004058 conn(0x7fbdb80a94c0 msgr2=0x7fbdb80a9890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:51:29.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdbf787700 1 -- 172.21.15.7:0/4112004058 shutdown_connections 2023-03-03T16:51:29.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.664+0000 7fbdbf787700 1 -- 172.21.15.7:0/4112004058 wait complete. 2023-03-03T16:51:29.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.665+0000 7fbdbf787700 1 Processor -- start 2023-03-03T16:51:29.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.666+0000 7fbdbf787700 1 -- start start 2023-03-03T16:51:29.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.666+0000 7fbdbf787700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb806d9f0 0x7fbdb81eabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:29.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.666+0000 7fbdbf787700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb80aa3e0 0x7fbdb81dedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:29.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.666+0000 7fbdbf787700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb80c7690 0x7fbdb81525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:29.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.666+0000 7fbdbf787700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb8153b10 con 0x7fbdb806d9f0 2023-03-03T16:51:29.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.666+0000 7fbdbf787700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb8151060 con 0x7fbdb80c7690 2023-03-03T16:51:29.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.666+0000 7fbdbf787700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb814ed40 con 0x7fbdb80aa3e0 2023-03-03T16:51:29.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.666+0000 7fbdbd523700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb806d9f0 0x7fbdb81eabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:29.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.666+0000 7fbdbd523700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb806d9f0 0x7fbdb81eabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58896/0 (socket says 172.21.15.7:58896) 2023-03-03T16:51:29.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.666+0000 7fbdbd523700 1 -- 172.21.15.7:0/3818857859 learned_addr learned my addr 172.21.15.7:0/3818857859 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:51:29.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdbdd24700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb80c7690 0x7fbdb81525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:29.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdbcd22700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb80aa3e0 0x7fbdb81dedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:29.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdbd523700 1 -- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb80aa3e0 msgr2=0x7fbdb81dedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:29.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdbd523700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb80aa3e0 0x7fbdb81dedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:29.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdbd523700 1 -- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb80c7690 msgr2=0x7fbdb81525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:29.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdbd523700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb80c7690 0x7fbdb81525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:29.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdbd523700 1 -- 172.21.15.7:0/3818857859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0009a70 con 0x7fbdb806d9f0 2023-03-03T16:51:29.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdbd523700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb806d9f0 0x7fbdb81eabe0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7fbdb400c820 tx=0x7fbdb400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:51:29.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdadffb700 1 -- 172.21.15.7:0/3818857859 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4011800 con 0x7fbdb806d9f0 2023-03-03T16:51:29.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdadffb700 1 -- 172.21.15.7:0/3818857859 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb4011960 con 0x7fbdb806d9f0 2023-03-03T16:51:29.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdbf787700 1 -- 172.21.15.7:0/3818857859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb81517b0 con 0x7fbdb806d9f0 2023-03-03T16:51:29.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdadffb700 1 -- 172.21.15.7:0/3818857859 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb4011cf0 con 0x7fbdb806d9f0 2023-03-03T16:51:29.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.667+0000 7fbdbf787700 1 -- 172.21.15.7:0/3818857859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb81514d0 con 0x7fbdb806d9f0 2023-03-03T16:51:29.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.668+0000 7fbdbf787700 1 -- 172.21.15.7:0/3818857859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb80c3370 con 0x7fbdb806d9f0 2023-03-03T16:51:29.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.669+0000 7fbdadffb700 1 -- 172.21.15.7:0/3818857859 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbdb4021070 con 0x7fbdb806d9f0 2023-03-03T16:51:29.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.669+0000 7fbdadffb700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 0x7fbd9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:29.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.669+0000 7fbdadffb700 1 -- 172.21.15.7:0/3818857859 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbdb4085f40 con 0x7fbdb806d9f0 2023-03-03T16:51:29.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.669+0000 7fbdbcd22700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 0x7fbd9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:29.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.670+0000 7fbdbcd22700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 0x7fbd9c06a7a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fbdb81df5f0 tx=0x7fbda8007c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:51:29.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.674+0000 7fbdadffb700 1 -- 172.21.15.7:0/3818857859 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbdb40522e0 con 0x7fbdb806d9f0 2023-03-03T16:51:29.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.912+0000 7fbdbf787700 1 -- 172.21.15.7:0/3818857859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdb81552f0 con 0x7fbdb806d9f0 2023-03-03T16:51:29.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.913+0000 7fbdadffb700 1 -- 172.21.15.7:0/3818857859 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbdb401f020 con 0x7fbdb806d9f0 2023-03-03T16:51:29.913 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:51:29.913 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:51:29.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 -- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 msgr2=0x7fbd9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:29.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 0x7fbd9c06a7a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fbdb81df5f0 tx=0x7fbda8007c90 comp rx=0 tx=0).stop 2023-03-03T16:51:29.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 -- 172.21.15.7:0/3818857859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb806d9f0 msgr2=0x7fbdb81eabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:29.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb806d9f0 0x7fbdb81eabe0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7fbdb400c820 tx=0x7fbdb400cbe0 comp rx=0 tx=0).stop 2023-03-03T16:51:29.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 -- 172.21.15.7:0/3818857859 shutdown_connections 2023-03-03T16:51:29.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb80c7690 0x7fbdb81525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:29.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd9c0682f0 0x7fbd9c06a7a0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:29.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb806d9f0 0x7fbdb81eabe0 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:29.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 --2- 172.21.15.7:0/3818857859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb80aa3e0 0x7fbdb81dedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:29.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 -- 172.21.15.7:0/3818857859 >> 172.21.15.7:0/3818857859 conn(0x7fbdb80a94c0 msgr2=0x7fbdb8154be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:51:29.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 -- 172.21.15.7:0/3818857859 shutdown_connections 2023-03-03T16:51:29.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:29.915+0000 7fbdbf787700 1 -- 172.21.15.7:0/3818857859 wait complete. 2023-03-03T16:51:29.916 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:51:29.927 INFO:tasks.mds_thrash.fs.[a]:waiting for 93 secs before thrashing 2023-03-03T16:51:31.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:51:31.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.909+0000 7fac262f5700 1 Processor -- start 2023-03-03T16:51:31.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.909+0000 7fac262f5700 1 -- start start 2023-03-03T16:51:31.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac262f5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fac20151ab0 0x7fac200736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:31.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac262f5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac2006c5b0 0x7fac20157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:31.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac262f5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2006d780 con 0x7fac20073c30 2023-03-03T16:51:31.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac262f5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac2006cb70 con 0x7fac2006c5b0 2023-03-03T16:51:31.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac262f5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac2006ce50 con 0x7fac20151ab0 2023-03-03T16:51:31.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac24892700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac2006c5b0 0x7fac20157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:31.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac24892700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac2006c5b0 0x7fac20157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58906/0 (socket says 172.21.15.7:58906) 2023-03-03T16:51:31.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac24892700 1 -- 172.21.15.7:0/3086849985 learned_addr learned my addr 172.21.15.7:0/3086849985 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:51:31.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac1ffff700 1 --2- 172.21.15.7:0/3086849985 >> v2:172.21.15.143:3300/0 conn(0x7fac20151ab0 0x7fac200736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:31.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac24892700 1 -- 172.21.15.7:0/3086849985 >> v2:172.21.15.143:3300/0 conn(0x7fac20151ab0 msgr2=0x7fac200736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:51:31.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac24892700 1 --2- 172.21.15.7:0/3086849985 >> v2:172.21.15.143:3300/0 conn(0x7fac20151ab0 0x7fac200736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:31.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac24892700 1 -- 172.21.15.7:0/3086849985 >> v1:172.21.15.143:6789/0 conn(0x7fac20073c30 legacy=0x7fac2006bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:51:31.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac24892700 1 -- 172.21.15.7:0/3086849985 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2006d130 con 0x7fac2006c5b0 2023-03-03T16:51:31.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.910+0000 7fac24892700 1 --2- 172.21.15.7:0/3086849985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2006c5b0 0x7fac20157dc0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7fac2006bb20 tx=0x7fac1800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3500596e7fde1f1b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:51:31.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac1effd700 1 -- 172.21.15.7:0/3086849985 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1800e070 con 0x7fac2006c5b0 2023-03-03T16:51:31.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac1effd700 1 -- 172.21.15.7:0/3086849985 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac18003a20 con 0x7fac2006c5b0 2023-03-03T16:51:31.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac1effd700 1 -- 172.21.15.7:0/3086849985 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18003dd0 con 0x7fac2006c5b0 2023-03-03T16:51:31.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac262f5700 1 -- 172.21.15.7:0/3086849985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2006c5b0 msgr2=0x7fac20157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:31.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac262f5700 1 --2- 172.21.15.7:0/3086849985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2006c5b0 0x7fac20157dc0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7fac2006bb20 tx=0x7fac1800b330 comp rx=0 tx=0).stop 2023-03-03T16:51:31.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac262f5700 1 -- 172.21.15.7:0/3086849985 shutdown_connections 2023-03-03T16:51:31.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac262f5700 1 --2- 172.21.15.7:0/3086849985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac2006c5b0 0x7fac20157dc0 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:31.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac262f5700 1 --2- 172.21.15.7:0/3086849985 >> v2:172.21.15.143:3300/0 conn(0x7fac20151ab0 0x7fac200736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:31.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac262f5700 1 -- 172.21.15.7:0/3086849985 >> 172.21.15.7:0/3086849985 conn(0x7fac200a94d0 msgr2=0x7fac200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:51:31.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac262f5700 1 -- 172.21.15.7:0/3086849985 shutdown_connections 2023-03-03T16:51:31.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.911+0000 7fac262f5700 1 -- 172.21.15.7:0/3086849985 wait complete. 2023-03-03T16:51:31.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.912+0000 7fac262f5700 1 Processor -- start 2023-03-03T16:51:31.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.913+0000 7fac262f5700 1 -- start start 2023-03-03T16:51:31.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.913+0000 7fac262f5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac2006c5b0 0x7fac2014beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:31.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.913+0000 7fac262f5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac20073c30 0x7fac200c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:31.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.913+0000 7fac262f5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac20151ab0 0x7fac200c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:31.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.913+0000 7fac262f5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac201eaa50 con 0x7fac20151ab0 2023-03-03T16:51:31.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.913+0000 7fac262f5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac200c61f0 con 0x7fac20073c30 2023-03-03T16:51:31.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.913+0000 7fac262f5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac201eaf40 con 0x7fac2006c5b0 2023-03-03T16:51:31.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.913+0000 7fac24892700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac20151ab0 0x7fac200c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:31.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.913+0000 7fac24892700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac20151ab0 0x7fac200c5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58910/0 (socket says 172.21.15.7:58910) 2023-03-03T16:51:31.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.913+0000 7fac24892700 1 -- 172.21.15.7:0/4235799492 learned_addr learned my addr 172.21.15.7:0/4235799492 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:51:31.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac1f7fe700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac20073c30 0x7fac200c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:31.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac1ffff700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac2006c5b0 0x7fac2014beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:31.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac24892700 1 -- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac2006c5b0 msgr2=0x7fac2014beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:31.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac24892700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac2006c5b0 0x7fac2014beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:31.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac24892700 1 -- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac20073c30 msgr2=0x7fac200c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:31.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac24892700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac20073c30 0x7fac200c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:31.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac24892700 1 -- 172.21.15.7:0/4235799492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac18009b30 con 0x7fac20151ab0 2023-03-03T16:51:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac24892700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac20151ab0 0x7fac200c5cb0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7fac1800b370 tx=0x7fac18004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:51:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac1cff9700 1 -- 172.21.15.7:0/4235799492 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18004440 con 0x7fac20151ab0 2023-03-03T16:51:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac1cff9700 1 -- 172.21.15.7:0/4235799492 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac180045a0 con 0x7fac20151ab0 2023-03-03T16:51:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac262f5700 1 -- 172.21.15.7:0/4235799492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac20151e80 con 0x7fac20151ab0 2023-03-03T16:51:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac1cff9700 1 -- 172.21.15.7:0/4235799492 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18004950 con 0x7fac20151ab0 2023-03-03T16:51:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.914+0000 7fac262f5700 1 -- 172.21.15.7:0/4235799492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2006d410 con 0x7fac20151ab0 2023-03-03T16:51:31.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.915+0000 7fac262f5700 1 -- 172.21.15.7:0/4235799492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac20152010 con 0x7fac20151ab0 2023-03-03T16:51:31.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.916+0000 7fac1cff9700 1 -- 172.21.15.7:0/4235799492 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fac18022030 con 0x7fac20151ab0 2023-03-03T16:51:31.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.916+0000 7fac1cff9700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 0x7fac0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:51:31.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.916+0000 7fac1cff9700 1 -- 172.21.15.7:0/4235799492 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fac18087400 con 0x7fac20151ab0 2023-03-03T16:51:31.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.916+0000 7fac1ffff700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 0x7fac0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:51:31.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.917+0000 7fac1ffff700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 0x7fac0c06a7a0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fac2014cce0 tx=0x7fac14008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:51:31.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:31.921+0000 7fac1cff9700 1 -- 172.21.15.7:0/4235799492 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fac180537a0 con 0x7fac20151ab0 2023-03-03T16:51:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.159+0000 7fac262f5700 1 -- 172.21.15.7:0/4235799492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac20074bb0 con 0x7fac20151ab0 2023-03-03T16:51:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.160+0000 7fac1cff9700 1 -- 172.21.15.7:0/4235799492 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fac18056dc0 con 0x7fac20151ab0 2023-03-03T16:51:32.160 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:51:32.160 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:51:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 -- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 msgr2=0x7fac0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 0x7fac0c06a7a0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fac2014cce0 tx=0x7fac14008040 comp rx=0 tx=0).stop 2023-03-03T16:51:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 -- 172.21.15.7:0/4235799492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac20151ab0 msgr2=0x7fac200c5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:51:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac20151ab0 0x7fac200c5cb0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7fac1800b370 tx=0x7fac18004ad0 comp rx=0 tx=0).stop 2023-03-03T16:51:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 -- 172.21.15.7:0/4235799492 shutdown_connections 2023-03-03T16:51:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac0c0682f0 0x7fac0c06a7a0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:32.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac20151ab0 0x7fac200c5cb0 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac2006c5b0 0x7fac2014beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 --2- 172.21.15.7:0/4235799492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac20073c30 0x7fac200c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:51:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 -- 172.21.15.7:0/4235799492 >> 172.21.15.7:0/4235799492 conn(0x7fac200a94d0 msgr2=0x7fac201558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:51:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 -- 172.21.15.7:0/4235799492 shutdown_connections 2023-03-03T16:51:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:51:32.162+0000 7fac262f5700 1 -- 172.21.15.7:0/4235799492 wait complete. 2023-03-03T16:51:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:51:32.173 INFO:tasks.mds_thrash.fs.[b]:waiting for 38 secs before thrashing 2023-03-03T16:51:49.818 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:51:49.821 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:10.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:52:10.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.314+0000 7f66cfe39700 1 Processor -- start 2023-03-03T16:52:10.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.314+0000 7f66cfe39700 1 -- start start 2023-03-03T16:52:10.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cfe39700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66c8155090 0x7f66c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:10.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cfe39700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c806c560 0x7f66c80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:10.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cfe39700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c8157b50 con 0x7f66c8073c30 2023-03-03T16:52:10.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cfe39700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c806c240 con 0x7f66c8155090 2023-03-03T16:52:10.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cfe39700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c80c33b0 con 0x7f66c806c560 2023-03-03T16:52:10.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cdbd5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66c8155090 0x7f66c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:10.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cdbd5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f66c8155090 0x7f66c806d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58936/0 (socket says 172.21.15.7:58936) 2023-03-03T16:52:10.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cdbd5700 1 -- 172.21.15.7:0/2198545179 learned_addr learned my addr 172.21.15.7:0/2198545179 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:52:10.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cd3d4700 1 --2- 172.21.15.7:0/2198545179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c806c560 0x7f66c80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:10.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cdbd5700 1 -- 172.21.15.7:0/2198545179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c806c560 msgr2=0x7f66c80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:52:10.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cdbd5700 1 --2- 172.21.15.7:0/2198545179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c806c560 0x7f66c80736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:10.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cdbd5700 1 -- 172.21.15.7:0/2198545179 >> v1:172.21.15.143:6789/0 conn(0x7f66c8073c30 legacy=0x7f66c8157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:52:10.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cdbd5700 1 -- 172.21.15.7:0/2198545179 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c80c3690 con 0x7f66c8155090 2023-03-03T16:52:10.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.315+0000 7f66cdbd5700 1 --2- 172.21.15.7:0/2198545179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 0x7f66c806d450 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f66c40098f0 tx=0x7f66c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4032ae83fdf504f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:10.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66ccbd3700 1 -- 172.21.15.7:0/2198545179 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4004230 con 0x7f66c8155090 2023-03-03T16:52:10.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66ccbd3700 1 -- 172.21.15.7:0/2198545179 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c4004390 con 0x7f66c8155090 2023-03-03T16:52:10.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66ccbd3700 1 -- 172.21.15.7:0/2198545179 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c40036a0 con 0x7f66c8155090 2023-03-03T16:52:10.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66cfe39700 1 -- 172.21.15.7:0/2198545179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 msgr2=0x7f66c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:10.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66cfe39700 1 --2- 172.21.15.7:0/2198545179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 0x7f66c806d450 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f66c40098f0 tx=0x7f66c4009c00 comp rx=0 tx=0).stop 2023-03-03T16:52:10.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66cfe39700 1 -- 172.21.15.7:0/2198545179 shutdown_connections 2023-03-03T16:52:10.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66cfe39700 1 --2- 172.21.15.7:0/2198545179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 0x7f66c806d450 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:10.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66cfe39700 1 --2- 172.21.15.7:0/2198545179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c806c560 0x7f66c80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:10.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66cfe39700 1 -- 172.21.15.7:0/2198545179 >> 172.21.15.7:0/2198545179 conn(0x7f66c80a94d0 msgr2=0x7f66c80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:52:10.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66cfe39700 1 -- 172.21.15.7:0/2198545179 shutdown_connections 2023-03-03T16:52:10.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.316+0000 7f66cfe39700 1 -- 172.21.15.7:0/2198545179 wait complete. 2023-03-03T16:52:10.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.317+0000 7f66cfe39700 1 Processor -- start 2023-03-03T16:52:10.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.317+0000 7f66cfe39700 1 -- start start 2023-03-03T16:52:10.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66cfe39700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c806c560 0x7f66c80c0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:10.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66cfe39700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c8073c30 0x7f66c8155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:10.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66cfe39700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 0x7f66c8155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:10.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66cfe39700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c80be430 con 0x7f66c8155090 2023-03-03T16:52:10.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66cfe39700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c80be960 con 0x7f66c8073c30 2023-03-03T16:52:10.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66cfe39700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c80c0bf0 con 0x7f66c806c560 2023-03-03T16:52:10.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66ce3d6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 0x7f66c8155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:10.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66ce3d6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 0x7f66c8155e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58946/0 (socket says 172.21.15.7:58946) 2023-03-03T16:52:10.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66ce3d6700 1 -- 172.21.15.7:0/154223770 learned_addr learned my addr 172.21.15.7:0/154223770 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:52:10.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66cd3d4700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c8073c30 0x7f66c8155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:10.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.318+0000 7f66cdbd5700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c806c560 0x7f66c80c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:10.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66ce3d6700 1 -- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c806c560 msgr2=0x7f66c80c0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:10.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66ce3d6700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c806c560 0x7f66c80c0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:10.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66ce3d6700 1 -- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c8073c30 msgr2=0x7f66c8155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:10.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66ce3d6700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c8073c30 0x7f66c8155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:10.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66ce3d6700 1 -- 172.21.15.7:0/154223770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c4009550 con 0x7f66c8155090 2023-03-03T16:52:10.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66ce3d6700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 0x7f66c8155e90 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f66c000cc10 tx=0x7f66c000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:10.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66be7fc700 1 -- 172.21.15.7:0/154223770 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c000aa60 con 0x7f66c8155090 2023-03-03T16:52:10.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66be7fc700 1 -- 172.21.15.7:0/154223770 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c000abc0 con 0x7f66c8155090 2023-03-03T16:52:10.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66cfe39700 1 -- 172.21.15.7:0/154223770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c81545e0 con 0x7f66c8155090 2023-03-03T16:52:10.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66be7fc700 1 -- 172.21.15.7:0/154223770 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0012420 con 0x7f66c8155090 2023-03-03T16:52:10.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.319+0000 7f66cfe39700 1 -- 172.21.15.7:0/154223770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c80c3970 con 0x7f66c8155090 2023-03-03T16:52:10.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.320+0000 7f66cfe39700 1 -- 172.21.15.7:0/154223770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c8155460 con 0x7f66c8155090 2023-03-03T16:52:10.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.321+0000 7f66be7fc700 1 -- 172.21.15.7:0/154223770 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f66c0010050 con 0x7f66c8155090 2023-03-03T16:52:10.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.321+0000 7f66be7fc700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66ac0682f0 0x7f66ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:52:10.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.321+0000 7f66be7fc700 1 -- 172.21.15.7:0/154223770 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f66c0086260 con 0x7f66c8155090 2023-03-03T16:52:10.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.321+0000 7f66cdbd5700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66ac0682f0 0x7f66ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:52:10.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.322+0000 7f66cdbd5700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66ac0682f0 0x7f66ac06a7a0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009130 tx=0x7f66c4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:52:10.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.325+0000 7f66be7fc700 1 -- 172.21.15.7:0/154223770 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f66c00525a0 con 0x7f66c8155090 2023-03-03T16:52:10.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.560+0000 7f66cfe39700 1 -- 172.21.15.7:0/154223770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66c80c0db0 con 0x7f66c8155090 2023-03-03T16:52:10.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.560+0000 7f66be7fc700 1 -- 172.21.15.7:0/154223770 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f66c0055bc0 con 0x7f66c8155090 2023-03-03T16:52:10.561 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:52:10.561 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 -- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66ac0682f0 msgr2=0x7f66ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66ac0682f0 0x7f66ac06a7a0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f66c4009130 tx=0x7f66c4008030 comp rx=0 tx=0).stop 2023-03-03T16:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 -- 172.21.15.7:0/154223770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 msgr2=0x7f66c8155e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 0x7f66c8155e90 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f66c000cc10 tx=0x7f66c000cf20 comp rx=0 tx=0).stop 2023-03-03T16:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 -- 172.21.15.7:0/154223770 shutdown_connections 2023-03-03T16:52:10.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66ac0682f0 0x7f66ac06a7a0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66c8155090 0x7f66c8155e90 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f66c806c560 0x7f66c80c0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 --2- 172.21.15.7:0/154223770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66c8073c30 0x7f66c8155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 -- 172.21.15.7:0/154223770 >> 172.21.15.7:0/154223770 conn(0x7f66c80a94d0 msgr2=0x7f66c8071a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 -- 172.21.15.7:0/154223770 shutdown_connections 2023-03-03T16:52:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:52:10.562+0000 7f66cfe39700 1 -- 172.21.15.7:0/154223770 wait complete. 2023-03-03T16:52:10.565 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:52:10.575 INFO:tasks.mds_thrash.fs.[b]:waiting for 103 secs before thrashing 2023-03-03T16:52:19.957 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:19.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:50.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:52:50.107 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:02.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:53:03.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.068+0000 7f4f7c340700 1 Processor -- start 2023-03-03T16:53:03.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.068+0000 7f4f7c340700 1 -- start start 2023-03-03T16:53:03.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f7c340700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4f7406d880 0x7f4f740c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:03.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f7c340700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f740c7c20 con 0x7f4f7406d4b0 2023-03-03T16:53:03.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f7c340700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f741544c0 con 0x7f4f740c0ee0 2023-03-03T16:53:03.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f7c340700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7413a600 con 0x7f4f7406d880 2023-03-03T16:53:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f7a0dc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4f7406d4b0 0x7f4f740c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52954/0 (socket says 172.21.15.7:52954) 2023-03-03T16:53:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f7a0dc700 1 -- 172.21.15.7:0/2207722422 learned_addr learned my addr 172.21.15.7:0/2207722422 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:53:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633685532 0 0) 0x7f4f740c7c20 con 0x7f4f7406d4b0 2023-03-03T16:53:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f600035b0 con 0x7f4f7406d4b0 2023-03-03T16:53:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f798db700 1 --2- 172.21.15.7:0/2207722422 >> v2:172.21.15.143:3300/0 conn(0x7f4f7406d880 0x7f4f740c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1356053327 0 0) 0x7f4f600035b0 con 0x7f4f7406d4b0 2023-03-03T16:53:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 >> v2:172.21.15.143:3300/0 conn(0x7f4f7406d880 msgr2=0x7f4f740c1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:53:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f790da700 1 --2- 172.21.15.7:0/2207722422 >> v2:172.21.15.143:3300/0 conn(0x7f4f7406d880 0x7f4f740c1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 >> v1:172.21.15.143:6789/0 conn(0x7f4f740c0ee0 legacy=0x7f4f74139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f740c7310 con 0x7f4f7406d4b0 2023-03-03T16:53:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.069+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f4f70001670 con 0x7f4f7406d4b0 2023-03-03T16:53:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 >> v1:172.21.15.7:6789/0 conn(0x7f4f7406d4b0 legacy=0x7f4f740c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f790da700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f60003a70 0x7f4f60003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f790da700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f60004850 0x7f4f60004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f790da700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f60005250 0x7f4f6000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7413a600 con 0x7f4f60005250 2023-03-03T16:53:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6000c190 con 0x7f4f60004850 2023-03-03T16:53:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6000c350 con 0x7f4f60003a70 2023-03-03T16:53:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f7a8dd700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f60005250 0x7f4f6000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f7a0dc700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f60003a70 0x7f4f60003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f798db700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f60004850 0x7f4f60004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f7a8dd700 1 -- 172.21.15.7:0/2207722422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f60003a70 msgr2=0x7f4f60003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:53:03.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f7a8dd700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f60003a70 0x7f4f60003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f7a8dd700 1 -- 172.21.15.7:0/2207722422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f60004850 msgr2=0x7f4f60004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:53:03.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f7a8dd700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f60004850 0x7f4f60004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f7a8dd700 1 -- 172.21.15.7:0/2207722422 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f600035b0 con 0x7f4f60005250 2023-03-03T16:53:03.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f7a8dd700 1 -- 172.21.15.7:0/2207722422 reap_dead start 2023-03-03T16:53:03.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f7a8dd700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f60005250 0x7f4f6000ba10 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f4f740c1340 tx=0x7f4f60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baeb059d4bd204f3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:53:03.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c001df0 con 0x7f4f60005250 2023-03-03T16:53:03.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6c00cd20 con 0x7f4f60005250 2023-03-03T16:53:03.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.070+0000 7f4f790da700 1 -- 172.21.15.7:0/2207722422 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c003c40 con 0x7f4f60005250 2023-03-03T16:53:03.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.071+0000 7f4f7c340700 1 -- 172.21.15.7:0/2207722422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f60005250 msgr2=0x7f4f6000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:03.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.071+0000 7f4f7c340700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f60005250 0x7f4f6000ba10 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f4f740c1340 tx=0x7f4f60005880 comp rx=0 tx=0).stop 2023-03-03T16:53:03.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.071+0000 7f4f7c340700 1 -- 172.21.15.7:0/2207722422 shutdown_connections 2023-03-03T16:53:03.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.071+0000 7f4f7c340700 1 --2- 172.21.15.7:0/2207722422 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f60005250 0x7f4f6000ba10 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.071+0000 7f4f7c340700 1 -- 172.21.15.7:0/2207722422 >> 172.21.15.7:0/2207722422 conn(0x7f4f7406c110 msgr2=0x7f4f7406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:53:03.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.071+0000 7f4f7c340700 1 -- 172.21.15.7:0/2207722422 shutdown_connections 2023-03-03T16:53:03.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.071+0000 7f4f7c340700 1 -- 172.21.15.7:0/2207722422 wait complete. 2023-03-03T16:53:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.072+0000 7f4f7c340700 1 Processor -- start 2023-03-03T16:53:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.072+0000 7f4f7c340700 1 -- start start 2023-03-03T16:53:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f7c340700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f60005250 0x7f4f740c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f7c340700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f74071ac0 0x7f4f74071f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f7c340700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f741ed260 0x7f4f741ed6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f7c340700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f740ab6d0 con 0x7f4f74071ac0 2023-03-03T16:53:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f7c340700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f740aacf0 con 0x7f4f741ed260 2023-03-03T16:53:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f7c340700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f740aa860 con 0x7f4f60005250 2023-03-03T16:53:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f798db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f74071ac0 0x7f4f74071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f798db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f74071ac0 0x7f4f74071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58988/0 (socket says 172.21.15.7:58988) 2023-03-03T16:53:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f798db700 1 -- 172.21.15.7:0/3036616195 learned_addr learned my addr 172.21.15.7:0/3036616195 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:53:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f7a0dc700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f60005250 0x7f4f740c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f7a8dd700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f741ed260 0x7f4f741ed6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f798db700 1 -- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f60005250 msgr2=0x7f4f740c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f798db700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f60005250 0x7f4f740c64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f798db700 1 -- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f741ed260 msgr2=0x7f4f741ed6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f798db700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f741ed260 0x7f4f741ed6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.073+0000 7f4f798db700 1 -- 172.21.15.7:0/3036616195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f740c75d0 con 0x7f4f74071ac0 2023-03-03T16:53:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.074+0000 7f4f798db700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f74071ac0 0x7f4f74071f30 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f4f640119a0 tx=0x7f4f64011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:53:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.074+0000 7f4f6affd700 1 -- 172.21.15.7:0/3036616195 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6400f730 con 0x7f4f74071ac0 2023-03-03T16:53:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.074+0000 7f4f6affd700 1 -- 172.21.15.7:0/3036616195 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6400f890 con 0x7f4f74071ac0 2023-03-03T16:53:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.074+0000 7f4f7c340700 1 -- 172.21.15.7:0/3036616195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f74154740 con 0x7f4f74071ac0 2023-03-03T16:53:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.074+0000 7f4f6affd700 1 -- 172.21.15.7:0/3036616195 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6400fc20 con 0x7f4f74071ac0 2023-03-03T16:53:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.074+0000 7f4f7c340700 1 -- 172.21.15.7:0/3036616195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f740c78b0 con 0x7f4f74071ac0 2023-03-03T16:53:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.075+0000 7f4f7c340700 1 -- 172.21.15.7:0/3036616195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7413dc40 con 0x7f4f74071ac0 2023-03-03T16:53:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.075+0000 7f4f6affd700 1 -- 172.21.15.7:0/3036616195 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4f6400fd80 con 0x7f4f74071ac0 2023-03-03T16:53:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.076+0000 7f4f6affd700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f580682f0 0x7f4f5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.076+0000 7f4f6affd700 1 -- 172.21.15.7:0/3036616195 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4f640868b0 con 0x7f4f74071ac0 2023-03-03T16:53:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.076+0000 7f4f7a0dc700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f580682f0 0x7f4f5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:03.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.077+0000 7f4f7a0dc700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f580682f0 0x7f4f5806a7a0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f4f70007e50 tx=0x7f4f70001510 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:53:03.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.080+0000 7f4f6affd700 1 -- 172.21.15.7:0/3036616195 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4f64052c50 con 0x7f4f74071ac0 2023-03-03T16:53:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.320+0000 7f4f7c340700 1 -- 172.21.15.7:0/3036616195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f740aaa10 con 0x7f4f74071ac0 2023-03-03T16:53:03.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.320+0000 7f4f6affd700 1 -- 172.21.15.7:0/3036616195 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4f740aaa10 con 0x7f4f74071ac0 2023-03-03T16:53:03.320 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:53:03.320 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:53:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.322+0000 7f4f7c340700 1 -- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f580682f0 msgr2=0x7f4f5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:03.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.322+0000 7f4f7c340700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f580682f0 0x7f4f5806a7a0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f4f70007e50 tx=0x7f4f70001510 comp rx=0 tx=0).stop 2023-03-03T16:53:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.322+0000 7f4f7c340700 1 -- 172.21.15.7:0/3036616195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f74071ac0 msgr2=0x7f4f74071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.322+0000 7f4f7c340700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f74071ac0 0x7f4f74071f30 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f4f640119a0 tx=0x7f4f64011cb0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.323+0000 7f4f7c340700 1 -- 172.21.15.7:0/3036616195 shutdown_connections 2023-03-03T16:53:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.323+0000 7f4f7c340700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f741ed260 0x7f4f741ed6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.323+0000 7f4f7c340700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f60005250 0x7f4f740c64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.323+0000 7f4f7c340700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4f580682f0 0x7f4f5806a7a0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.323+0000 7f4f7c340700 1 --2- 172.21.15.7:0/3036616195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f74071ac0 0x7f4f74071f30 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:03.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.323+0000 7f4f7c340700 1 -- 172.21.15.7:0/3036616195 >> 172.21.15.7:0/3036616195 conn(0x7f4f7406c110 msgr2=0x7f4f7406dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:53:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.323+0000 7f4f7c340700 1 -- 172.21.15.7:0/3036616195 shutdown_connections 2023-03-03T16:53:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:03.323+0000 7f4f7c340700 1 -- 172.21.15.7:0/3036616195 wait complete. 2023-03-03T16:53:03.324 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:53:03.334 INFO:tasks.mds_thrash.fs.[a]:waiting for 77 secs before thrashing 2023-03-03T16:53:20.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:20.229 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:50.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:50.366 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:53:53.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:53:53.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.715+0000 7f6fdf522700 1 Processor -- start 2023-03-03T16:53:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.715+0000 7f6fdf522700 1 -- start start 2023-03-03T16:53:53.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdf522700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 0x7f6fd80717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:53.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdf522700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6fd80aad20 0x7f6fd8157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:53.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdf522700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd81588f0 con 0x7f6fd8071d20 2023-03-03T16:53:53.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdf522700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd80728b0 con 0x7f6fd80aad20 2023-03-03T16:53:53.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdf522700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd8155070 con 0x7f6fd814b020 2023-03-03T16:53:53.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdd2be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 0x7f6fd80717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:53.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdcabd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6fd80aad20 0x7f6fd8157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:53.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdd2be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 0x7f6fd80717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:38864/0 (socket says 172.21.15.7:38864) 2023-03-03T16:53:53.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdcabd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6fd80aad20 0x7f6fd8157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:54800/0 (socket says 172.21.15.7:54800) 2023-03-03T16:53:53.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdd2be700 1 -- 172.21.15.7:0/2836655534 learned_addr learned my addr 172.21.15.7:0/2836655534 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:53:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdcabd700 1 -- 172.21.15.7:0/2836655534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 msgr2=0x7f6fd80717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:53:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdcabd700 1 --2- 172.21.15.7:0/2836655534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 0x7f6fd80717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.716+0000 7f6fdcabd700 1 -- 172.21.15.7:0/2836655534 >> v1:172.21.15.143:6789/0 conn(0x7f6fd8071d20 legacy=0x7f6fd81581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fdcabd700 1 -- 172.21.15.7:0/2836655534 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd806f5b0 con 0x7f6fd80aad20 2023-03-03T16:53:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fcbfff700 1 -- 172.21.15.7:0/2836655534 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803707388 0 0) 0x7f6fd81588f0 con 0x7f6fd8071d20 2023-03-03T16:53:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fdd2be700 1 --2- 172.21.15.7:0/2836655534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 0x7f6fd80717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:53:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fdcabd700 1 --2- 172.21.15.7:0/2836655534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd80aad20 0x7f6fd8157ca0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009840 tx=0x7f6fd4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beccc959723dd146 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:53:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fcbfff700 1 -- 172.21.15.7:0/2836655534 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4011830 con 0x7f6fd80aad20 2023-03-03T16:53:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fcbfff700 1 -- 172.21.15.7:0/2836655534 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd4011990 con 0x7f6fd80aad20 2023-03-03T16:53:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fcbfff700 1 -- 172.21.15.7:0/2836655534 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4011d20 con 0x7f6fd80aad20 2023-03-03T16:53:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fdf522700 1 -- 172.21.15.7:0/2836655534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd80aad20 msgr2=0x7f6fd8157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fdf522700 1 --2- 172.21.15.7:0/2836655534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd80aad20 0x7f6fd8157ca0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009840 tx=0x7f6fd4009b50 comp rx=0 tx=0).stop 2023-03-03T16:53:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fdf522700 1 -- 172.21.15.7:0/2836655534 shutdown_connections 2023-03-03T16:53:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fdf522700 1 --2- 172.21.15.7:0/2836655534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 0x7f6fd80717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fdf522700 1 --2- 172.21.15.7:0/2836655534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd80aad20 0x7f6fd8157ca0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.717+0000 7f6fdf522700 1 -- 172.21.15.7:0/2836655534 >> 172.21.15.7:0/2836655534 conn(0x7f6fd80c8900 msgr2=0x7f6fd80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:53:53.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.718+0000 7f6fdf522700 1 -- 172.21.15.7:0/2836655534 shutdown_connections 2023-03-03T16:53:53.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.718+0000 7f6fdf522700 1 -- 172.21.15.7:0/2836655534 wait complete. 2023-03-03T16:53:53.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.719+0000 7f6fdf522700 1 Processor -- start 2023-03-03T16:53:53.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.719+0000 7f6fdf522700 1 -- start start 2023-03-03T16:53:53.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.719+0000 7f6fdf522700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd8071d20 0x7f6fd8153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:53.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.719+0000 7f6fdf522700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fd80aad20 0x7f6fd8148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:53.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.719+0000 7f6fdf522700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 0x7f6fd813f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:53.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.719+0000 7f6fdf522700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd81e9bc0 con 0x7f6fd80aad20 2023-03-03T16:53:53.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.719+0000 7f6fdf522700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd8146760 con 0x7f6fd8071d20 2023-03-03T16:53:53.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.719+0000 7f6fdf522700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd81e98e0 con 0x7f6fd814b020 2023-03-03T16:53:53.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdcabd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fd80aad20 0x7f6fd8148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:53.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdcabd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fd80aad20 0x7f6fd8148860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59032/0 (socket says 172.21.15.7:59032) 2023-03-03T16:53:53.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdcabd700 1 -- 172.21.15.7:0/409658492 learned_addr learned my addr 172.21.15.7:0/409658492 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:53:53.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fddabf700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 0x7f6fd813f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:53.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdd2be700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd8071d20 0x7f6fd8153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:53.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdcabd700 1 -- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 msgr2=0x7f6fd813f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:53.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdcabd700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 0x7f6fd813f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:53.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdcabd700 1 -- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd8071d20 msgr2=0x7f6fd8153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:53.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdcabd700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd8071d20 0x7f6fd8153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:53.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdcabd700 1 -- 172.21.15.7:0/409658492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc009550 con 0x7f6fd80aad20 2023-03-03T16:53:53.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdcabd700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fd80aad20 0x7f6fd8148860 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009fd0 tx=0x7f6fd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:53:53.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fc9ffb700 1 -- 172.21.15.7:0/409658492 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40038a0 con 0x7f6fd80aad20 2023-03-03T16:53:53.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fc9ffb700 1 -- 172.21.15.7:0/409658492 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd401e070 con 0x7f6fd80aad20 2023-03-03T16:53:53.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fdf522700 1 -- 172.21.15.7:0/409658492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd40094a0 con 0x7f6fd80aad20 2023-03-03T16:53:53.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.720+0000 7f6fc9ffb700 1 -- 172.21.15.7:0/409658492 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4003c40 con 0x7f6fd80aad20 2023-03-03T16:53:53.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.721+0000 7f6fdf522700 1 -- 172.21.15.7:0/409658492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd806fb70 con 0x7f6fd80aad20 2023-03-03T16:53:53.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.722+0000 7f6fdf522700 1 -- 172.21.15.7:0/409658492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd806f890 con 0x7f6fd80aad20 2023-03-03T16:53:53.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.722+0000 7f6fc9ffb700 1 -- 172.21.15.7:0/409658492 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6fd400f050 con 0x7f6fd80aad20 2023-03-03T16:53:53.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.722+0000 7f6fc9ffb700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6fbc0682f0 0x7f6fbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:53:53.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.722+0000 7f6fc9ffb700 1 -- 172.21.15.7:0/409658492 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6fd4087120 con 0x7f6fd80aad20 2023-03-03T16:53:53.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.722+0000 7f6fdd2be700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6fbc0682f0 0x7f6fbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:53:53.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.723+0000 7f6fdd2be700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6fbc0682f0 0x7f6fbc06a7a0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f6fd81e73c0 tx=0x7f6fcc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:53:53.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.727+0000 7f6fc9ffb700 1 -- 172.21.15.7:0/409658492 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6fd40534c0 con 0x7f6fd80aad20 2023-03-03T16:53:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.965+0000 7f6fdf522700 1 -- 172.21.15.7:0/409658492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fd8155350 con 0x7f6fd80aad20 2023-03-03T16:53:53.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.965+0000 7f6fc9ffb700 1 -- 172.21.15.7:0/409658492 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6fd4020090 con 0x7f6fd80aad20 2023-03-03T16:53:53.965 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:53:53.965 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:53:53.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 -- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6fbc0682f0 msgr2=0x7f6fbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:53.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6fbc0682f0 0x7f6fbc06a7a0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f6fd81e73c0 tx=0x7f6fcc008040 comp rx=0 tx=0).stop 2023-03-03T16:53:53.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 -- 172.21.15.7:0/409658492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fd80aad20 msgr2=0x7f6fd8148860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:53:53.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fd80aad20 0x7f6fd8148860 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009fd0 tx=0x7f6fd4004750 comp rx=0 tx=0).stop 2023-03-03T16:53:53.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 -- 172.21.15.7:0/409658492 shutdown_connections 2023-03-03T16:53:53.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6fd814b020 0x7f6fd813f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:53.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fd8071d20 0x7f6fd8153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:53.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6fbc0682f0 0x7f6fbc06a7a0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:53.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 --2- 172.21.15.7:0/409658492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6fd80aad20 0x7f6fd8148860 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:53:53.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 -- 172.21.15.7:0/409658492 >> 172.21.15.7:0/409658492 conn(0x7f6fd80c8900 msgr2=0x7f6fd814a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:53:53.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 -- 172.21.15.7:0/409658492 shutdown_connections 2023-03-03T16:53:53.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:53:53.968+0000 7f6fdf522700 1 -- 172.21.15.7:0/409658492 wait complete. 2023-03-03T16:53:53.969 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:53:53.979 INFO:tasks.mds_thrash.fs.[b]:waiting for 59 secs before thrashing 2023-03-03T16:54:20.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:54:20.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:20.427 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:20.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.475+0000 7fe0a5d79700 1 Processor -- start 2023-03-03T16:54:20.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.475+0000 7fe0a5d79700 1 -- start start 2023-03-03T16:54:20.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.475+0000 7fe0a5d79700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe0a00beab0 0x7fe0a0155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:20.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.475+0000 7fe0a5d79700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a01523b0 con 0x7fe0a0152e30 2023-03-03T16:54:20.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.475+0000 7fe0a5d79700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0072010 con 0x7fe0a01562c0 2023-03-03T16:54:20.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.475+0000 7fe0a5d79700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a00aaaf0 con 0x7fe0a00beab0 2023-03-03T16:54:20.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe0a00beab0 0x7fe0a0155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:20.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe0a0152e30 0x7fe0a00c43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53036/0 (socket says 172.21.15.7:53036) 2023-03-03T16:54:20.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09f7fe700 1 -- 172.21.15.7:0/1208667233 learned_addr learned my addr 172.21.15.7:0/1208667233 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:54:20.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09e7fc700 1 -- 172.21.15.7:0/1208667233 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481169104 0 0) 0x7fe0a01523b0 con 0x7fe0a0152e30 2023-03-03T16:54:20.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09e7fc700 1 -- 172.21.15.7:0/1208667233 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0880035b0 con 0x7fe0a0152e30 2023-03-03T16:54:20.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09effd700 1 -- 172.21.15.7:0/1208667233 >> v1:172.21.15.143:6789/0 conn(0x7fe0a01562c0 legacy=0x7fe0a015cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:20.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09effd700 1 -- 172.21.15.7:0/1208667233 >> v1:172.21.15.7:6789/0 conn(0x7fe0a0152e30 legacy=0x7fe0a00c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:20.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09effd700 1 -- 172.21.15.7:0/1208667233 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a00aadd0 con 0x7fe0a00beab0 2023-03-03T16:54:20.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09effd700 1 --2- 172.21.15.7:0/1208667233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a00beab0 0x7fe0a0155d80 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fe094009840 tx=0x7fe094009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661f828024e4e9ff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:20.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09e7fc700 1 -- 172.21.15.7:0/1208667233 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094011830 con 0x7fe0a00beab0 2023-03-03T16:54:20.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09e7fc700 1 -- 172.21.15.7:0/1208667233 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe094011990 con 0x7fe0a00beab0 2023-03-03T16:54:20.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.476+0000 7fe09e7fc700 1 -- 172.21.15.7:0/1208667233 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094011d00 con 0x7fe0a00beab0 2023-03-03T16:54:20.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.477+0000 7fe0a5d79700 1 -- 172.21.15.7:0/1208667233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a00beab0 msgr2=0x7fe0a0155d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:20.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.477+0000 7fe0a5d79700 1 --2- 172.21.15.7:0/1208667233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a00beab0 0x7fe0a0155d80 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fe094009840 tx=0x7fe094009b50 comp rx=0 tx=0).stop 2023-03-03T16:54:20.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.477+0000 7fe0a5d79700 1 -- 172.21.15.7:0/1208667233 shutdown_connections 2023-03-03T16:54:20.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.477+0000 7fe0a5d79700 1 --2- 172.21.15.7:0/1208667233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a00beab0 0x7fe0a0155d80 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:20.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.477+0000 7fe0a5d79700 1 -- 172.21.15.7:0/1208667233 >> 172.21.15.7:0/1208667233 conn(0x7fe0a00736f0 msgr2=0x7fe0a0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:54:20.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.477+0000 7fe0a5d79700 1 -- 172.21.15.7:0/1208667233 shutdown_connections 2023-03-03T16:54:20.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.477+0000 7fe0a5d79700 1 -- 172.21.15.7:0/1208667233 wait complete. 2023-03-03T16:54:20.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.478+0000 7fe0a5d79700 1 Processor -- start 2023-03-03T16:54:20.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.478+0000 7fe0a5d79700 1 -- start start 2023-03-03T16:54:20.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe0a5d79700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a00beab0 0x7fe0a006acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:20.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe0a5d79700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a0152e30 0x7fe0a01514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:20.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe0a5d79700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0a01562c0 0x7fe0a014b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:20.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe0a5d79700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a00ab510 con 0x7fe0a0152e30 2023-03-03T16:54:20.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe0a5d79700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a0154050 con 0x7fe0a00beab0 2023-03-03T16:54:20.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe0a5d79700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a006b200 con 0x7fe0a01562c0 2023-03-03T16:54:20.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe09effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a0152e30 0x7fe0a01514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:20.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe09effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a0152e30 0x7fe0a01514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59064/0 (socket says 172.21.15.7:59064) 2023-03-03T16:54:20.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe09effd700 1 -- 172.21.15.7:0/991044539 learned_addr learned my addr 172.21.15.7:0/991044539 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:54:20.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe09f7fe700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a00beab0 0x7fe0a006acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:20.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe09ffff700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0a01562c0 0x7fe0a014b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:20.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe09effd700 1 -- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0a01562c0 msgr2=0x7fe0a014b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:20.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe09effd700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0a01562c0 0x7fe0a014b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:20.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe09effd700 1 -- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a00beab0 msgr2=0x7fe0a006acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:20.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe09effd700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a00beab0 0x7fe0a006acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:20.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.479+0000 7fe09effd700 1 -- 172.21.15.7:0/991044539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0940094a0 con 0x7fe0a0152e30 2023-03-03T16:54:20.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.480+0000 7fe09effd700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a0152e30 0x7fe0a01514e0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fe094009fd0 tx=0x7fe094004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:20.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.480+0000 7fe087fff700 1 -- 172.21.15.7:0/991044539 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094004550 con 0x7fe0a0152e30 2023-03-03T16:54:20.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.480+0000 7fe087fff700 1 -- 172.21.15.7:0/991044539 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09401e070 con 0x7fe0a0152e30 2023-03-03T16:54:20.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.480+0000 7fe087fff700 1 -- 172.21.15.7:0/991044539 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0940038e0 con 0x7fe0a0152e30 2023-03-03T16:54:20.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.480+0000 7fe0a5d79700 1 -- 172.21.15.7:0/991044539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a0153200 con 0x7fe0a0152e30 2023-03-03T16:54:20.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.480+0000 7fe0a5d79700 1 -- 172.21.15.7:0/991044539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a0071c70 con 0x7fe0a0152e30 2023-03-03T16:54:20.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.481+0000 7fe0a5d79700 1 -- 172.21.15.7:0/991044539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a0153390 con 0x7fe0a0152e30 2023-03-03T16:54:20.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.481+0000 7fe087fff700 1 -- 172.21.15.7:0/991044539 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe09400f050 con 0x7fe0a0152e30 2023-03-03T16:54:20.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.482+0000 7fe087fff700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe08c0682f0 0x7fe08c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:20.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.482+0000 7fe087fff700 1 -- 172.21.15.7:0/991044539 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe094087050 con 0x7fe0a0152e30 2023-03-03T16:54:20.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.482+0000 7fe09f7fe700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe08c0682f0 0x7fe08c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:20.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.482+0000 7fe09f7fe700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe08c0682f0 0x7fe08c06a7a0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fe090007fd0 tx=0x7fe090007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:20.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.486+0000 7fe087fff700 1 -- 172.21.15.7:0/991044539 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe0940533f0 con 0x7fe0a0152e30 2023-03-03T16:54:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.725+0000 7fe0a5d79700 1 -- 172.21.15.7:0/991044539 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0a0074bb0 con 0x7fe0a0152e30 2023-03-03T16:54:20.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.726+0000 7fe087fff700 1 -- 172.21.15.7:0/991044539 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe094015090 con 0x7fe0a0152e30 2023-03-03T16:54:20.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:54:20.727 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:54:20.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 -- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe08c0682f0 msgr2=0x7fe08c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:20.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe08c0682f0 0x7fe08c06a7a0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fe090007fd0 tx=0x7fe090007dc0 comp rx=0 tx=0).stop 2023-03-03T16:54:20.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 -- 172.21.15.7:0/991044539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a0152e30 msgr2=0x7fe0a01514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:20.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a0152e30 0x7fe0a01514e0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7fe094009fd0 tx=0x7fe094004750 comp rx=0 tx=0).stop 2023-03-03T16:54:20.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 -- 172.21.15.7:0/991044539 shutdown_connections 2023-03-03T16:54:20.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0a01562c0 0x7fe0a014b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:20.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a00beab0 0x7fe0a006acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:20.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe08c0682f0 0x7fe08c06a7a0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:20.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 --2- 172.21.15.7:0/991044539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0a0152e30 0x7fe0a01514e0 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:20.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 -- 172.21.15.7:0/991044539 >> 172.21.15.7:0/991044539 conn(0x7fe0a00736f0 msgr2=0x7fe0a00be330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:54:20.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 -- 172.21.15.7:0/991044539 shutdown_connections 2023-03-03T16:54:20.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:20.728+0000 7fe0a5d79700 1 -- 172.21.15.7:0/991044539 wait complete. 2023-03-03T16:54:20.731 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:54:20.740 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T16:54:50.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:50.561 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:54:52.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:54:53.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.120+0000 7f31016fd700 1 Processor -- start 2023-03-03T16:54:53.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.120+0000 7f31016fd700 1 -- start start 2023-03-03T16:54:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f31016fd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 0x7f30fc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f31016fd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f30fc0c00e0 0x7f30fc154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f31016fd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30fc0cb280 con 0x7f30fc0cae00 2023-03-03T16:54:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f31016fd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30fc0c7310 con 0x7f30fc0c00e0 2023-03-03T16:54:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f31016fd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc0bfd10 con 0x7f30fc0caa30 2023-03-03T16:54:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30fb7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f30fc0cae00 0x7f30fc0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53070/0 (socket says 172.21.15.7:53070) 2023-03-03T16:54:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30fb7fe700 1 -- 172.21.15.7:0/3533689367 learned_addr learned my addr 172.21.15.7:0/3533689367 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:54:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30faffd700 1 --2- 172.21.15.7:0/3533689367 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 0x7f30fc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30fa7fc700 1 --2- 172.21.15.7:0/3533689367 >> v2:172.21.15.143:3300/0 conn(0x7f30fc0c00e0 0x7f30fc154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30f9ffb700 1 -- 172.21.15.7:0/3533689367 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682444822 0 0) 0x7f30fc0cb280 con 0x7f30fc0cae00 2023-03-03T16:54:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30f9ffb700 1 -- 172.21.15.7:0/3533689367 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e40035b0 con 0x7f30fc0cae00 2023-03-03T16:54:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30f9ffb700 1 -- 172.21.15.7:0/3533689367 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1668204697 0 0) 0x7f30e40035b0 con 0x7f30fc0cae00 2023-03-03T16:54:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30fa7fc700 1 -- 172.21.15.7:0/3533689367 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 msgr2=0x7f30fc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:54:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30fa7fc700 1 --2- 172.21.15.7:0/3533689367 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 0x7f30fc0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30fa7fc700 1 -- 172.21.15.7:0/3533689367 >> v1:172.21.15.7:6789/0 conn(0x7f30fc0cae00 legacy=0x7f30fc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30fa7fc700 1 -- 172.21.15.7:0/3533689367 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc0bfa30 con 0x7f30fc0c00e0 2023-03-03T16:54:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.121+0000 7f30faffd700 1 --2- 172.21.15.7:0/3533689367 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 0x7f30fc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:54:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f30fa7fc700 1 --2- 172.21.15.7:0/3533689367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30fc0c00e0 0x7f30fc154ff0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009820 tx=0x7f30ec009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87afc7af2c009c4f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:53.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f30f9ffb700 1 -- 172.21.15.7:0/3533689367 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f40019f0 con 0x7f30fc0c00e0 2023-03-03T16:54:53.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f30f9ffb700 1 -- 172.21.15.7:0/3533689367 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec005c30 con 0x7f30fc0c00e0 2023-03-03T16:54:53.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f30f9ffb700 1 -- 172.21.15.7:0/3533689367 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec004030 con 0x7f30fc0c00e0 2023-03-03T16:54:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f31016fd700 1 -- 172.21.15.7:0/3533689367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30fc0c00e0 msgr2=0x7f30fc154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f31016fd700 1 --2- 172.21.15.7:0/3533689367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30fc0c00e0 0x7f30fc154ff0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f30ec009820 tx=0x7f30ec009b30 comp rx=0 tx=0).stop 2023-03-03T16:54:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f31016fd700 1 -- 172.21.15.7:0/3533689367 shutdown_connections 2023-03-03T16:54:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f31016fd700 1 --2- 172.21.15.7:0/3533689367 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 0x7f30fc0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f31016fd700 1 --2- 172.21.15.7:0/3533689367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30fc0c00e0 0x7f30fc154ff0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f31016fd700 1 -- 172.21.15.7:0/3533689367 >> 172.21.15.7:0/3533689367 conn(0x7f30fc066e40 msgr2=0x7f30fc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:54:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f31016fd700 1 -- 172.21.15.7:0/3533689367 shutdown_connections 2023-03-03T16:54:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.122+0000 7f31016fd700 1 -- 172.21.15.7:0/3533689367 wait complete. 2023-03-03T16:54:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.124+0000 7f31016fd700 1 Processor -- start 2023-03-03T16:54:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.124+0000 7f31016fd700 1 -- start start 2023-03-03T16:54:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f31016fd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30fc0c00e0 0x7f30fc1ea5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f31016fd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 0x7f30fc152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f31016fd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30fc0cae00 0x7f30fc14d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f31016fd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc0ab1c0 con 0x7f30fc0cae00 2023-03-03T16:54:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f31016fd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc1eaaf0 con 0x7f30fc0c00e0 2023-03-03T16:54:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f31016fd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30fc0c4a90 con 0x7f30fc0caa30 2023-03-03T16:54:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30fb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30fc0cae00 0x7f30fc14d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30fb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30fc0cae00 0x7f30fc14d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59096/0 (socket says 172.21.15.7:59096) 2023-03-03T16:54:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30fb7fe700 1 -- 172.21.15.7:0/237776920 learned_addr learned my addr 172.21.15.7:0/237776920 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:54:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30fa7fc700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 0x7f30fc152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30faffd700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30fc0c00e0 0x7f30fc1ea5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30fb7fe700 1 -- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 msgr2=0x7f30fc152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30fb7fe700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 0x7f30fc152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30fb7fe700 1 -- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30fc0c00e0 msgr2=0x7f30fc1ea5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:53.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30fb7fe700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30fc0c00e0 0x7f30fc1ea5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:53.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30fb7fe700 1 -- 172.21.15.7:0/237776920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f00094a0 con 0x7f30fc0cae00 2023-03-03T16:54:53.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30fb7fe700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30fc0cae00 0x7f30fc14d110 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f30f400cc10 tx=0x7f30f400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:53.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30e37fe700 1 -- 172.21.15.7:0/237776920 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f400a9c0 con 0x7f30fc0cae00 2023-03-03T16:54:53.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30e37fe700 1 -- 172.21.15.7:0/237776920 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f400ab40 con 0x7f30fc0cae00 2023-03-03T16:54:53.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f31016fd700 1 -- 172.21.15.7:0/237776920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ec009430 con 0x7f30fc0cae00 2023-03-03T16:54:53.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f30e37fe700 1 -- 172.21.15.7:0/237776920 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f40070d0 con 0x7f30fc0cae00 2023-03-03T16:54:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.125+0000 7f31016fd700 1 -- 172.21.15.7:0/237776920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30fc1547f0 con 0x7f30fc0cae00 2023-03-03T16:54:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.127+0000 7f31016fd700 1 -- 172.21.15.7:0/237776920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30fc154510 con 0x7f30fc0cae00 2023-03-03T16:54:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.127+0000 7f30e37fe700 1 -- 172.21.15.7:0/237776920 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f30f4010070 con 0x7f30fc0cae00 2023-03-03T16:54:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.127+0000 7f30e37fe700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30e80682f0 0x7f30e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.127+0000 7f30e37fe700 1 -- 172.21.15.7:0/237776920 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f30f4085a50 con 0x7f30fc0cae00 2023-03-03T16:54:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.128+0000 7f30faffd700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30e80682f0 0x7f30e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.128+0000 7f30faffd700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30e80682f0 0x7f30e806a7a0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f30fc153a60 tx=0x7f30f0009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.132+0000 7f30e37fe700 1 -- 172.21.15.7:0/237776920 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f30f4051d70 con 0x7f30fc0cae00 2023-03-03T16:54:53.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.371+0000 7f31016fd700 1 -- 172.21.15.7:0/237776920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30fc1ed280 con 0x7f30fc0cae00 2023-03-03T16:54:53.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.371+0000 7f30e37fe700 1 -- 172.21.15.7:0/237776920 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f30f4055390 con 0x7f30fc0cae00 2023-03-03T16:54:53.371 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:54:53.372 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:54:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 -- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30e80682f0 msgr2=0x7f30e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30e80682f0 0x7f30e806a7a0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f30fc153a60 tx=0x7f30f0009210 comp rx=0 tx=0).stop 2023-03-03T16:54:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 -- 172.21.15.7:0/237776920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30fc0cae00 msgr2=0x7f30fc14d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30fc0cae00 0x7f30fc14d110 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f30f400cc10 tx=0x7f30f400cf20 comp rx=0 tx=0).stop 2023-03-03T16:54:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 -- 172.21.15.7:0/237776920 shutdown_connections 2023-03-03T16:54:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f30e80682f0 0x7f30e806a7a0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f30fc0cae00 0x7f30fc14d110 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f30fc0c00e0 0x7f30fc1ea5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 --2- 172.21.15.7:0/237776920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f30fc0caa30 0x7f30fc152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 -- 172.21.15.7:0/237776920 >> 172.21.15.7:0/237776920 conn(0x7f30fc066e40 msgr2=0x7f30fc0c6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:54:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 -- 172.21.15.7:0/237776920 shutdown_connections 2023-03-03T16:54:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:53.373+0000 7f31016fd700 1 -- 172.21.15.7:0/237776920 wait complete. 2023-03-03T16:54:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:54:53.385 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before thrashing 2023-03-03T16:54:54.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:54:54.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.525+0000 7fc6a2300700 1 Processor -- start 2023-03-03T16:54:54.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.525+0000 7fc6a2300700 1 -- start start 2023-03-03T16:54:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc6a2300700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc69c14f860 0x7fc69c1562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc6a2300700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc69c0c27c0 0x7fc69c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc6a2300700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c155e60 con 0x7fc69c0c20d0 2023-03-03T16:54:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc6a2300700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc69c06d350 con 0x7fc69c14f860 2023-03-03T16:54:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc6a2300700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c1558e0 con 0x7fc69c0c27c0 2023-03-03T16:54:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc69bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc69c14f860 0x7fc69c1562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc69bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc69c14f860 0x7fc69c1562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59100/0 (socket says 172.21.15.7:59100) 2023-03-03T16:54:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc69bfff700 1 -- 172.21.15.7:0/449623864 learned_addr learned my addr 172.21.15.7:0/449623864 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:54:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc6a089d700 1 --2- 172.21.15.7:0/449623864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc69c0c27c0 0x7fc69c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc69bfff700 1 -- 172.21.15.7:0/449623864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc69c0c27c0 msgr2=0x7fc69c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:54:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc69bfff700 1 --2- 172.21.15.7:0/449623864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc69c0c27c0 0x7fc69c0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc69bfff700 1 -- 172.21.15.7:0/449623864 >> v1:172.21.15.143:6789/0 conn(0x7fc69c0c20d0 legacy=0x7fc69c152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:54:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.526+0000 7fc69bfff700 1 -- 172.21.15.7:0/449623864 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c06cd90 con 0x7fc69c14f860 2023-03-03T16:54:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc69bfff700 1 --2- 172.21.15.7:0/449623864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 0x7fc69c1562f0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0098f0 tx=0x7fc68c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112a017074e14f47 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc69affd700 1 -- 172.21.15.7:0/449623864 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c004230 con 0x7fc69c14f860 2023-03-03T16:54:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc69affd700 1 -- 172.21.15.7:0/449623864 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c004390 con 0x7fc69c14f860 2023-03-03T16:54:54.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc69affd700 1 -- 172.21.15.7:0/449623864 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c0036a0 con 0x7fc69c14f860 2023-03-03T16:54:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc6a2300700 1 -- 172.21.15.7:0/449623864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 msgr2=0x7fc69c1562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc6a2300700 1 --2- 172.21.15.7:0/449623864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 0x7fc69c1562f0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0098f0 tx=0x7fc68c009c00 comp rx=0 tx=0).stop 2023-03-03T16:54:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc6a2300700 1 -- 172.21.15.7:0/449623864 shutdown_connections 2023-03-03T16:54:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc6a2300700 1 --2- 172.21.15.7:0/449623864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc69c0c27c0 0x7fc69c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc6a2300700 1 --2- 172.21.15.7:0/449623864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 0x7fc69c1562f0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc6a2300700 1 -- 172.21.15.7:0/449623864 >> 172.21.15.7:0/449623864 conn(0x7fc69c0a94d0 msgr2=0x7fc69c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:54:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc6a2300700 1 -- 172.21.15.7:0/449623864 shutdown_connections 2023-03-03T16:54:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.527+0000 7fc6a2300700 1 -- 172.21.15.7:0/449623864 wait complete. 2023-03-03T16:54:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.529+0000 7fc6a2300700 1 Processor -- start 2023-03-03T16:54:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.529+0000 7fc6a2300700 1 -- start start 2023-03-03T16:54:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.529+0000 7fc6a2300700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc69c0c20d0 0x7fc69c0c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.529+0000 7fc6a2300700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc69c0c27c0 0x7fc69c1e6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.529+0000 7fc6a2300700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 0x7fc69c1e7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.529+0000 7fc6a2300700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c070650 con 0x7fc69c14f860 2023-03-03T16:54:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.529+0000 7fc6a2300700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c1e7740 con 0x7fc69c0c27c0 2023-03-03T16:54:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.529+0000 7fc6a2300700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c1ed2f0 con 0x7fc69c0c20d0 2023-03-03T16:54:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a089d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 0x7fc69c1e7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a089d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 0x7fc69c1e7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59106/0 (socket says 172.21.15.7:59106) 2023-03-03T16:54:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a089d700 1 -- 172.21.15.7:0/2245884133 learned_addr learned my addr 172.21.15.7:0/2245884133 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:54:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc69bfff700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc69c0c20d0 0x7fc69c0c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc69b7fe700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc69c0c27c0 0x7fc69c1e6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a089d700 1 -- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc69c0c20d0 msgr2=0x7fc69c0c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a089d700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc69c0c20d0 0x7fc69c0c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a089d700 1 -- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc69c0c27c0 msgr2=0x7fc69c1e6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:54.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a089d700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc69c0c27c0 0x7fc69c1e6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a089d700 1 -- 172.21.15.7:0/2245884133 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c009550 con 0x7fc69c14f860 2023-03-03T16:54:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc69b7fe700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc69c0c27c0 0x7fc69c1e6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:54:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a089d700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 0x7fc69c1e7200 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fc69400cda0 tx=0x7fc69400a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc698ff9700 1 -- 172.21.15.7:0/2245884133 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc694003a20 con 0x7fc69c14f860 2023-03-03T16:54:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc698ff9700 1 -- 172.21.15.7:0/2245884133 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc694003b80 con 0x7fc69c14f860 2023-03-03T16:54:54.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a2300700 1 -- 172.21.15.7:0/2245884133 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69c155af0 con 0x7fc69c14f860 2023-03-03T16:54:54.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc698ff9700 1 -- 172.21.15.7:0/2245884133 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc694004020 con 0x7fc69c14f860 2023-03-03T16:54:54.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.530+0000 7fc6a2300700 1 -- 172.21.15.7:0/2245884133 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69c06d070 con 0x7fc69c14f860 2023-03-03T16:54:54.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.531+0000 7fc6a2300700 1 -- 172.21.15.7:0/2245884133 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69c14fc30 con 0x7fc69c14f860 2023-03-03T16:54:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.532+0000 7fc698ff9700 1 -- 172.21.15.7:0/2245884133 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc69401c020 con 0x7fc69c14f860 2023-03-03T16:54:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.532+0000 7fc698ff9700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc67c0682f0 0x7fc67c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:54:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.532+0000 7fc698ff9700 1 -- 172.21.15.7:0/2245884133 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc694012070 con 0x7fc69c14f860 2023-03-03T16:54:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.533+0000 7fc69bfff700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc67c0682f0 0x7fc67c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:54:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.533+0000 7fc69bfff700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc67c0682f0 0x7fc67c06a7a0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009130 tx=0x7fc68c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:54:54.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.536+0000 7fc698ff9700 1 -- 172.21.15.7:0/2245884133 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc694052ca0 con 0x7fc69c14f860 2023-03-03T16:54:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.776+0000 7fc6a2300700 1 -- 172.21.15.7:0/2245884133 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc69c074bb0 con 0x7fc69c14f860 2023-03-03T16:54:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.777+0000 7fc698ff9700 1 -- 172.21.15.7:0/2245884133 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc6940562c0 con 0x7fc69c14f860 2023-03-03T16:54:54.777 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:54:54.777 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:54:54.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 -- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc67c0682f0 msgr2=0x7fc67c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:54.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc67c0682f0 0x7fc67c06a7a0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009130 tx=0x7fc68c008030 comp rx=0 tx=0).stop 2023-03-03T16:54:54.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 -- 172.21.15.7:0/2245884133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 msgr2=0x7fc69c1e7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:54:54.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 0x7fc69c1e7200 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fc69400cda0 tx=0x7fc69400a570 comp rx=0 tx=0).stop 2023-03-03T16:54:54.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 -- 172.21.15.7:0/2245884133 shutdown_connections 2023-03-03T16:54:54.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc67c0682f0 0x7fc67c06a7a0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:54.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc69c14f860 0x7fc69c1e7200 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:54.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc69c0c20d0 0x7fc69c0c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:54.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 --2- 172.21.15.7:0/2245884133 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc69c0c27c0 0x7fc69c1e6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:54:54.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 -- 172.21.15.7:0/2245884133 >> 172.21.15.7:0/2245884133 conn(0x7fc69c0a94d0 msgr2=0x7fc69c14d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:54:54.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 -- 172.21.15.7:0/2245884133 shutdown_connections 2023-03-03T16:54:54.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:54:54.779+0000 7fc6a2300700 1 -- 172.21.15.7:0/2245884133 wait complete. 2023-03-03T16:54:54.781 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:54:54.791 INFO:tasks.mds_thrash.fs.[b]:waiting for 14 secs before thrashing 2023-03-03T16:55:08.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:55:08.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.930+0000 7feb39e78700 1 Processor -- start 2023-03-03T16:55:08.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.930+0000 7feb39e78700 1 -- start start 2023-03-03T16:55:08.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb39e78700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb3414b0d0 0x7feb34071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:08.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb39e78700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7feb340aad20 0x7feb34157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:08.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb39e78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34071d40 0x7feb34158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:08.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb39e78700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb340a9c20 con 0x7feb3414b0d0 2023-03-03T16:55:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb39e78700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb34154f10 con 0x7feb340aad20 2023-03-03T16:55:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb39e78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb340ac3c0 con 0x7feb34071d40 2023-03-03T16:55:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb337fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb3414b0d0 0x7feb34071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb337fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb3414b0d0 0x7feb34071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59120/0 (socket says 172.21.15.7:59120) 2023-03-03T16:55:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb337fe700 1 -- 172.21.15.7:0/3051982294 learned_addr learned my addr 172.21.15.7:0/3051982294 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:55:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb32ffd700 1 --2- 172.21.15.7:0/3051982294 >> v2:172.21.15.143:3300/0 conn(0x7feb340aad20 0x7feb34157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:08.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb33fff700 1 --2- 172.21.15.7:0/3051982294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34071d40 0x7feb34158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb337fe700 1 -- 172.21.15.7:0/3051982294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34071d40 msgr2=0x7feb34158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:55:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb337fe700 1 --2- 172.21.15.7:0/3051982294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34071d40 0x7feb34158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb337fe700 1 -- 172.21.15.7:0/3051982294 >> v2:172.21.15.143:3300/0 conn(0x7feb340aad20 msgr2=0x7feb34157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:55:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb337fe700 1 --2- 172.21.15.7:0/3051982294 >> v2:172.21.15.143:3300/0 conn(0x7feb340aad20 0x7feb34157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb337fe700 1 -- 172.21.15.7:0/3051982294 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3406fc70 con 0x7feb3414b0d0 2023-03-03T16:55:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb32ffd700 1 --2- 172.21.15.7:0/3051982294 >> v2:172.21.15.143:3300/0 conn(0x7feb340aad20 0x7feb34157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:55:08.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb33fff700 1 --2- 172.21.15.7:0/3051982294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34071d40 0x7feb34158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:55:08.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb337fe700 1 --2- 172.21.15.7:0/3051982294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 0x7feb34071800 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7feb340c2860 tx=0x7feb24009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5377443d28664cd8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:08.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.931+0000 7feb327fc700 1 -- 172.21.15.7:0/3051982294 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2400b870 con 0x7feb3414b0d0 2023-03-03T16:55:08.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb327fc700 1 -- 172.21.15.7:0/3051982294 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2400b9d0 con 0x7feb3414b0d0 2023-03-03T16:55:08.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb327fc700 1 -- 172.21.15.7:0/3051982294 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2400bd60 con 0x7feb3414b0d0 2023-03-03T16:55:08.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb39e78700 1 -- 172.21.15.7:0/3051982294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 msgr2=0x7feb34071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:08.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb39e78700 1 --2- 172.21.15.7:0/3051982294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 0x7feb34071800 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7feb340c2860 tx=0x7feb24009bd0 comp rx=0 tx=0).stop 2023-03-03T16:55:08.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb39e78700 1 -- 172.21.15.7:0/3051982294 shutdown_connections 2023-03-03T16:55:08.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb39e78700 1 --2- 172.21.15.7:0/3051982294 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34071d40 0x7feb34158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:08.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb39e78700 1 --2- 172.21.15.7:0/3051982294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 0x7feb34071800 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:08.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb39e78700 1 --2- 172.21.15.7:0/3051982294 >> v2:172.21.15.143:3300/0 conn(0x7feb340aad20 0x7feb34157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:08.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb39e78700 1 -- 172.21.15.7:0/3051982294 >> 172.21.15.7:0/3051982294 conn(0x7feb340c8900 msgr2=0x7feb340c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:08.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb39e78700 1 -- 172.21.15.7:0/3051982294 shutdown_connections 2023-03-03T16:55:08.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.932+0000 7feb39e78700 1 -- 172.21.15.7:0/3051982294 wait complete. 2023-03-03T16:55:08.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.933+0000 7feb39e78700 1 Processor -- start 2023-03-03T16:55:08.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.933+0000 7feb39e78700 1 -- start start 2023-03-03T16:55:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb39e78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb34071d40 0x7feb34153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb39e78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb340aad20 0x7feb341471c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb39e78700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 0x7feb341e0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb39e78700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb34152550 con 0x7feb3414b0d0 2023-03-03T16:55:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb39e78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb34152020 con 0x7feb34071d40 2023-03-03T16:55:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb39e78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb341ede00 con 0x7feb340aad20 2023-03-03T16:55:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb33fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 0x7feb341e0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:08.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb33fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 0x7feb341e0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59130/0 (socket says 172.21.15.7:59130) 2023-03-03T16:55:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb33fff700 1 -- 172.21.15.7:0/27771504 learned_addr learned my addr 172.21.15.7:0/27771504 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:55:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb337fe700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb34071d40 0x7feb34153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.934+0000 7feb32ffd700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb340aad20 0x7feb341471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb33fff700 1 -- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb340aad20 msgr2=0x7feb341471c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb33fff700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb340aad20 0x7feb341471c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb33fff700 1 -- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb34071d40 msgr2=0x7feb34153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:08.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb33fff700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb34071d40 0x7feb34153c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:08.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb33fff700 1 -- 172.21.15.7:0/27771504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb24009550 con 0x7feb3414b0d0 2023-03-03T16:55:08.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb33fff700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 0x7feb341e0970 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00ccc0 tx=0x7feb2c00a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:08.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb1bfff700 1 -- 172.21.15.7:0/27771504 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c00ad20 con 0x7feb3414b0d0 2023-03-03T16:55:08.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb1bfff700 1 -- 172.21.15.7:0/27771504 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c00ae80 con 0x7feb3414b0d0 2023-03-03T16:55:08.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb39e78700 1 -- 172.21.15.7:0/27771504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3406f6b0 con 0x7feb3414b0d0 2023-03-03T16:55:08.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb1bfff700 1 -- 172.21.15.7:0/27771504 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c00f6c0 con 0x7feb3414b0d0 2023-03-03T16:55:08.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.935+0000 7feb39e78700 1 -- 172.21.15.7:0/27771504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb34155160 con 0x7feb3414b0d0 2023-03-03T16:55:08.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.936+0000 7feb39e78700 1 -- 172.21.15.7:0/27771504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3406f990 con 0x7feb3414b0d0 2023-03-03T16:55:08.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.936+0000 7feb1bfff700 1 -- 172.21.15.7:0/27771504 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feb2c009030 con 0x7feb3414b0d0 2023-03-03T16:55:08.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.937+0000 7feb1bfff700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb200682f0 0x7feb2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:08.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.937+0000 7feb1bfff700 1 -- 172.21.15.7:0/27771504 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7feb2c086990 con 0x7feb3414b0d0 2023-03-03T16:55:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.937+0000 7feb337fe700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb200682f0 0x7feb2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.938+0000 7feb337fe700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb200682f0 0x7feb2006a7a0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7feb24000c00 tx=0x7feb24008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:08.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:08.941+0000 7feb1bfff700 1 -- 172.21.15.7:0/27771504 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb2c052cd0 con 0x7feb3414b0d0 2023-03-03T16:55:09.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.177+0000 7feb39e78700 1 -- 172.21.15.7:0/27771504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb34074bb0 con 0x7feb3414b0d0 2023-03-03T16:55:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.177+0000 7feb1bfff700 1 -- 172.21.15.7:0/27771504 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7feb2c0562f0 con 0x7feb3414b0d0 2023-03-03T16:55:09.177 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:55:09.178 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:55:09.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.179+0000 7feb39e78700 1 -- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb200682f0 msgr2=0x7feb2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:09.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.179+0000 7feb39e78700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb200682f0 0x7feb2006a7a0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7feb24000c00 tx=0x7feb24008030 comp rx=0 tx=0).stop 2023-03-03T16:55:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.179+0000 7feb39e78700 1 -- 172.21.15.7:0/27771504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 msgr2=0x7feb341e0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.179+0000 7feb39e78700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 0x7feb341e0970 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00ccc0 tx=0x7feb2c00a280 comp rx=0 tx=0).stop 2023-03-03T16:55:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.179+0000 7feb39e78700 1 -- 172.21.15.7:0/27771504 shutdown_connections 2023-03-03T16:55:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.179+0000 7feb39e78700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb200682f0 0x7feb2006a7a0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.179+0000 7feb39e78700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb3414b0d0 0x7feb341e0970 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.179+0000 7feb39e78700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb34071d40 0x7feb34153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:09.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.179+0000 7feb39e78700 1 --2- 172.21.15.7:0/27771504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb340aad20 0x7feb341471c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.179+0000 7feb39e78700 1 -- 172.21.15.7:0/27771504 >> 172.21.15.7:0/27771504 conn(0x7feb340c8900 msgr2=0x7feb340c5680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.180+0000 7feb39e78700 1 -- 172.21.15.7:0/27771504 shutdown_connections 2023-03-03T16:55:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.180+0000 7feb39e78700 1 -- 172.21.15.7:0/27771504 wait complete. 2023-03-03T16:55:09.181 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:55:09.191 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T16:55:09.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:55:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.871+0000 7fc8f1b6c700 1 Processor -- start 2023-03-03T16:55:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.871+0000 7fc8f1b6c700 1 -- start start 2023-03-03T16:55:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.872+0000 7fc8f1b6c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc8ec071990 0x7fc8ec071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.872+0000 7fc8f1b6c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ec070360 con 0x7fc8ec151c00 2023-03-03T16:55:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.872+0000 7fc8f1b6c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ec06a5e0 con 0x7fc8ec14b460 2023-03-03T16:55:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.872+0000 7fc8f1b6c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec1521a0 con 0x7fc8ec071990 2023-03-03T16:55:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.872+0000 7fc8ebfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc8ec151c00 0x7fc8ec06b180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53122/0 (socket says 172.21.15.7:53122) 2023-03-03T16:55:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.872+0000 7fc8ebfff700 1 -- 172.21.15.7:0/102284376 learned_addr learned my addr 172.21.15.7:0/102284376 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:55:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.872+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963602366 0 0) 0x7fc8ec070360 con 0x7fc8ec151c00 2023-03-03T16:55:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.872+0000 7fc8eb7fe700 1 --2- 172.21.15.7:0/102284376 >> v2:172.21.15.143:3300/0 conn(0x7fc8ec071990 0x7fc8ec071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.872+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d80035b0 con 0x7fc8ec151c00 2023-03-03T16:55:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3286249907 0 0) 0x7fc8d80035b0 con 0x7fc8ec151c00 2023-03-03T16:55:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 >> v2:172.21.15.143:3300/0 conn(0x7fc8ec071990 msgr2=0x7fc8ec071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:55:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 --2- 172.21.15.7:0/102284376 >> v2:172.21.15.143:3300/0 conn(0x7fc8ec071990 0x7fc8ec071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 >> v1:172.21.15.143:6789/0 conn(0x7fc8ec14b460 legacy=0x7fc8ec06aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec0c38b0 con 0x7fc8ec151c00 2023-03-03T16:55:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc8e40019f0 con 0x7fc8ec151c00 2023-03-03T16:55:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 >> v1:172.21.15.7:6789/0 conn(0x7fc8ec151c00 legacy=0x7fc8ec06b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:09.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8d8003a70 0x7fc8d8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8d8004850 0x7fc8d8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8d8005320 0x7fc8d800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec1521a0 con 0x7fc8d8003a70 2023-03-03T16:55:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d800c230 con 0x7fc8d8004850 2023-03-03T16:55:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d800c3f0 con 0x7fc8d8005320 2023-03-03T16:55:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8eb7fe700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8d8003a70 0x7fc8d8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:09.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8eaffd700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8d8004850 0x7fc8d8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ebfff700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8d8005320 0x7fc8d800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8eb7fe700 1 -- 172.21.15.7:0/102284376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8d8005320 msgr2=0x7fc8d800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:55:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8eb7fe700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8d8005320 0x7fc8d800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8eb7fe700 1 -- 172.21.15.7:0/102284376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8d8004850 msgr2=0x7fc8d8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:55:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8eb7fe700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8d8004850 0x7fc8d8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8eb7fe700 1 -- 172.21.15.7:0/102284376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d80035b0 con 0x7fc8d8003a70 2023-03-03T16:55:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ebfff700 1 -- 172.21.15.7:0/102284376 reap_dead start 2023-03-03T16:55:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8eaffd700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8d8004850 0x7fc8d8004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:55:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8eb7fe700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8d8003a70 0x7fc8d8003e80 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc009b60 tx=0x7fc8d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2420e962c8f18654 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc00e6c0 con 0x7fc8d8003a70 2023-03-03T16:55:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8dc00e820 con 0x7fc8d8003a70 2023-03-03T16:55:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.873+0000 7fc8ea7fc700 1 -- 172.21.15.7:0/102284376 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc00eb90 con 0x7fc8d8003a70 2023-03-03T16:55:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.874+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/102284376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8d8003a70 msgr2=0x7fc8d8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:09.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.874+0000 7fc8f1b6c700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8d8003a70 0x7fc8d8003e80 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc009b60 tx=0x7fc8d8005880 comp rx=0 tx=0).stop 2023-03-03T16:55:09.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.874+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/102284376 shutdown_connections 2023-03-03T16:55:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.874+0000 7fc8f1b6c700 1 --2- 172.21.15.7:0/102284376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8d8003a70 0x7fc8d8003e80 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.874+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/102284376 >> 172.21.15.7:0/102284376 conn(0x7fc8ec0a94d0 msgr2=0x7fc8ec0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.874+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/102284376 shutdown_connections 2023-03-03T16:55:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.874+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/102284376 wait complete. 2023-03-03T16:55:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.875+0000 7fc8f1b6c700 1 Processor -- start 2023-03-03T16:55:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.875+0000 7fc8f1b6c700 1 -- start start 2023-03-03T16:55:09.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8f1b6c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8d8003a70 0x7fc8ec148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:09.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8f1b6c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8ec14a390 0x7fc8ec1daac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:09.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8f1b6c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1db000 0x7fc8ec1eb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:09.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8f1b6c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec152930 con 0x7fc8ec1db000 2023-03-03T16:55:09.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8f1b6c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c64a0 con 0x7fc8d8003a70 2023-03-03T16:55:09.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8f1b6c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0c5bb0 con 0x7fc8ec14a390 2023-03-03T16:55:09.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8ebfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1db000 0x7fc8ec1eb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:09.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8ebfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1db000 0x7fc8ec1eb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59154/0 (socket says 172.21.15.7:59154) 2023-03-03T16:55:09.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8ebfff700 1 -- 172.21.15.7:0/2801442708 learned_addr learned my addr 172.21.15.7:0/2801442708 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:55:09.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8eb7fe700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8d8003a70 0x7fc8ec148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:09.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8eaffd700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8ec14a390 0x7fc8ec1daac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:09.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8ebfff700 1 -- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8ec14a390 msgr2=0x7fc8ec1daac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:09.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8ebfff700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8ec14a390 0x7fc8ec1daac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:09.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8ebfff700 1 -- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8d8003a70 msgr2=0x7fc8ec148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:09.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.876+0000 7fc8ebfff700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8d8003a70 0x7fc8ec148d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:09.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.877+0000 7fc8ebfff700 1 -- 172.21.15.7:0/2801442708 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec0c3b50 con 0x7fc8ec1db000 2023-03-03T16:55:09.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.877+0000 7fc8ebfff700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1db000 0x7fc8ec1eb500 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec071680 tx=0x7fc8e4007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:09.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.877+0000 7fc8e3fff700 1 -- 172.21.15.7:0/2801442708 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e400eb10 con 0x7fc8ec1db000 2023-03-03T16:55:09.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.877+0000 7fc8e3fff700 1 -- 172.21.15.7:0/2801442708 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e400ec70 con 0x7fc8ec1db000 2023-03-03T16:55:09.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.877+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/2801442708 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec06a2b0 con 0x7fc8ec1db000 2023-03-03T16:55:09.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.877+0000 7fc8e3fff700 1 -- 172.21.15.7:0/2801442708 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4013420 con 0x7fc8ec1db000 2023-03-03T16:55:09.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.877+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/2801442708 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec071d60 con 0x7fc8ec1db000 2023-03-03T16:55:09.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.878+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/2801442708 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec074bb0 con 0x7fc8ec1db000 2023-03-03T16:55:09.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.878+0000 7fc8e3fff700 1 -- 172.21.15.7:0/2801442708 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc8e4013580 con 0x7fc8ec1db000 2023-03-03T16:55:09.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.879+0000 7fc8e3fff700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8cc0682f0 0x7fc8cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:09.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.879+0000 7fc8e3fff700 1 -- 172.21.15.7:0/2801442708 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc8e40866d0 con 0x7fc8ec1db000 2023-03-03T16:55:09.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.879+0000 7fc8eb7fe700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8cc0682f0 0x7fc8cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:09.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.879+0000 7fc8eb7fe700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8cc0682f0 0x7fc8cc06a7a0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0090b0 tx=0x7fc8dc01e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:09.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:09.883+0000 7fc8e3fff700 1 -- 172.21.15.7:0/2801442708 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc8e4052a70 con 0x7fc8ec1db000 2023-03-03T16:55:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.120+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/2801442708 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8ec0c32b0 con 0x7fc8ec1db000 2023-03-03T16:55:10.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.120+0000 7fc8e3fff700 1 -- 172.21.15.7:0/2801442708 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc8e4056090 con 0x7fc8ec1db000 2023-03-03T16:55:10.120 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:55:10.120 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:55:10.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.122+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8cc0682f0 msgr2=0x7fc8cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:10.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.122+0000 7fc8f1b6c700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8cc0682f0 0x7fc8cc06a7a0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0090b0 tx=0x7fc8dc01e040 comp rx=0 tx=0).stop 2023-03-03T16:55:10.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.122+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/2801442708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1db000 msgr2=0x7fc8ec1eb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:10.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.122+0000 7fc8f1b6c700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1db000 0x7fc8ec1eb500 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec071680 tx=0x7fc8e4007c90 comp rx=0 tx=0).stop 2023-03-03T16:55:10.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.123+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/2801442708 shutdown_connections 2023-03-03T16:55:10.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.123+0000 7fc8f1b6c700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8cc0682f0 0x7fc8cc06a7a0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:10.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.123+0000 7fc8f1b6c700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8ec1db000 0x7fc8ec1eb500 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:10.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.123+0000 7fc8f1b6c700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8d8003a70 0x7fc8ec148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.123+0000 7fc8f1b6c700 1 --2- 172.21.15.7:0/2801442708 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8ec14a390 0x7fc8ec1daac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.123+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/2801442708 >> 172.21.15.7:0/2801442708 conn(0x7fc8ec0a94d0 msgr2=0x7fc8ec0c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.123+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/2801442708 shutdown_connections 2023-03-03T16:55:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:10.123+0000 7fc8f1b6c700 1 -- 172.21.15.7:0/2801442708 wait complete. 2023-03-03T16:55:10.124 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:55:10.134 INFO:tasks.mds_thrash.fs.[a]:waiting for 33 secs before thrashing 2023-03-03T16:55:20.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:55:20.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.331+0000 7f84f11ac700 1 Processor -- start 2023-03-03T16:55:20.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.331+0000 7f84f11ac700 1 -- start start 2023-03-03T16:55:20.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84f11ac700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f84ec071c30 0x7f84ec0c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:20.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84f11ac700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84ec0713c0 con 0x7f84ec155950 2023-03-03T16:55:20.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84f11ac700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84ec152c70 con 0x7f84ec0aac00 2023-03-03T16:55:20.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84f11ac700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84ec0c4930 con 0x7f84ec071c30 2023-03-03T16:55:20.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84eb59e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f84ec155950 0x7f84ec15cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53148/0 (socket says 172.21.15.7:53148) 2023-03-03T16:55:20.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84eb59e700 1 -- 172.21.15.7:0/2461050280 learned_addr learned my addr 172.21.15.7:0/2461050280 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:55:20.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84ea59c700 1 --2- 172.21.15.7:0/2461050280 >> v2:172.21.15.143:3300/0 conn(0x7f84ec071c30 0x7f84ec0c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:20.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84e9d9b700 1 -- 172.21.15.7:0/2461050280 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824370570 0 0) 0x7f84ec0713c0 con 0x7f84ec155950 2023-03-03T16:55:20.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84e9d9b700 1 -- 172.21.15.7:0/2461050280 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d40035b0 con 0x7f84ec155950 2023-03-03T16:55:20.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84ea59c700 1 -- 172.21.15.7:0/2461050280 >> v1:172.21.15.143:6789/0 conn(0x7f84ec0aac00 legacy=0x7f84ec0beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:20.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84e9d9b700 1 -- 172.21.15.7:0/2461050280 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129869482 0 0) 0x7f84ec152c70 con 0x7f84ec0aac00 2023-03-03T16:55:20.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84ea59c700 1 -- 172.21.15.7:0/2461050280 >> v1:172.21.15.7:6789/0 conn(0x7f84ec155950 legacy=0x7f84ec15cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:20.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.332+0000 7f84ea59c700 1 -- 172.21.15.7:0/2461050280 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec152360 con 0x7f84ec071c30 2023-03-03T16:55:20.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84ea59c700 1 --2- 172.21.15.7:0/2461050280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84ec071c30 0x7f84ec0c43f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009840 tx=0x7f84e0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834a87ba1701bbfb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:20.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84e9d9b700 1 -- 172.21.15.7:0/2461050280 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e0011830 con 0x7f84ec071c30 2023-03-03T16:55:20.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84e9d9b700 1 -- 172.21.15.7:0/2461050280 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e0011990 con 0x7f84ec071c30 2023-03-03T16:55:20.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84e9d9b700 1 -- 172.21.15.7:0/2461050280 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e0011d00 con 0x7f84ec071c30 2023-03-03T16:55:20.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84f11ac700 1 -- 172.21.15.7:0/2461050280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84ec071c30 msgr2=0x7f84ec0c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:20.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84f11ac700 1 --2- 172.21.15.7:0/2461050280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84ec071c30 0x7f84ec0c43f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009840 tx=0x7f84e0009b50 comp rx=0 tx=0).stop 2023-03-03T16:55:20.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84f11ac700 1 -- 172.21.15.7:0/2461050280 shutdown_connections 2023-03-03T16:55:20.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84f11ac700 1 --2- 172.21.15.7:0/2461050280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84ec071c30 0x7f84ec0c43f0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:20.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84f11ac700 1 -- 172.21.15.7:0/2461050280 >> 172.21.15.7:0/2461050280 conn(0x7f84ec0736f0 msgr2=0x7f84ec073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:20.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84f11ac700 1 -- 172.21.15.7:0/2461050280 shutdown_connections 2023-03-03T16:55:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.333+0000 7f84f11ac700 1 -- 172.21.15.7:0/2461050280 wait complete. 2023-03-03T16:55:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.335+0000 7f84f11ac700 1 Processor -- start 2023-03-03T16:55:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.335+0000 7f84f11ac700 1 -- start start 2023-03-03T16:55:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84f11ac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec071c30 0x7f84ec0ac6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84f11ac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84ec0aac00 0x7f84ec14a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84f11ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84ec155950 0x7f84ec148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84f11ac700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec0ac0e0 con 0x7f84ec071c30 2023-03-03T16:55:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84f11ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec0be4f0 con 0x7f84ec155950 2023-03-03T16:55:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84f11ac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84ec0be7d0 con 0x7f84ec0aac00 2023-03-03T16:55:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ead9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec071c30 0x7f84ec0ac6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ead9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec071c30 0x7f84ec0ac6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59170/0 (socket says 172.21.15.7:59170) 2023-03-03T16:55:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ead9d700 1 -- 172.21.15.7:0/3056936329 learned_addr learned my addr 172.21.15.7:0/3056936329 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:55:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ea59c700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84ec0aac00 0x7f84ec14a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84eb59e700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84ec155950 0x7f84ec148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ead9d700 1 -- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84ec0aac00 msgr2=0x7f84ec14a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ead9d700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84ec0aac00 0x7f84ec14a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ead9d700 1 -- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84ec155950 msgr2=0x7f84ec148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ead9d700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84ec155950 0x7f84ec148330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ead9d700 1 -- 172.21.15.7:0/3056936329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e00094a0 con 0x7f84ec071c30 2023-03-03T16:55:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ea59c700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84ec0aac00 0x7f84ec14a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:55:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84ead9d700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec071c30 0x7f84ec0ac6c0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f84dc001eb0 tx=0x7f84dc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84d37fe700 1 -- 172.21.15.7:0/3056936329 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0054e0 con 0x7f84ec071c30 2023-03-03T16:55:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84d37fe700 1 -- 172.21.15.7:0/3056936329 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc003680 con 0x7f84ec071c30 2023-03-03T16:55:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84f11ac700 1 -- 172.21.15.7:0/3056936329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ec152620 con 0x7f84ec071c30 2023-03-03T16:55:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.336+0000 7f84d37fe700 1 -- 172.21.15.7:0/3056936329 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0039f0 con 0x7f84ec071c30 2023-03-03T16:55:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.337+0000 7f84f11ac700 1 -- 172.21.15.7:0/3056936329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ec0c4b80 con 0x7f84ec071c30 2023-03-03T16:55:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.337+0000 7f84f11ac700 1 -- 172.21.15.7:0/3056936329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84ec152900 con 0x7f84ec071c30 2023-03-03T16:55:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.338+0000 7f84d37fe700 1 -- 172.21.15.7:0/3056936329 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f84dc019030 con 0x7f84ec071c30 2023-03-03T16:55:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.338+0000 7f84d37fe700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84d80682f0 0x7f84d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.338+0000 7f84d37fe700 1 -- 172.21.15.7:0/3056936329 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f84dc085580 con 0x7f84ec071c30 2023-03-03T16:55:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.339+0000 7f84ea59c700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84d80682f0 0x7f84d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.339+0000 7f84ea59c700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84d80682f0 0x7f84d806a7a0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f84e00090b0 tx=0x7f84e0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.343+0000 7f84d37fe700 1 -- 172.21.15.7:0/3056936329 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f84dc04b5e0 con 0x7f84ec071c30 2023-03-03T16:55:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.581+0000 7f84f11ac700 1 -- 172.21.15.7:0/3056936329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84ec0c3ec0 con 0x7f84ec071c30 2023-03-03T16:55:20.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.581+0000 7f84d37fe700 1 -- 172.21.15.7:0/3056936329 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f84dc017090 con 0x7f84ec071c30 2023-03-03T16:55:20.581 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:55:20.581 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:55:20.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 -- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84d80682f0 msgr2=0x7f84d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84d80682f0 0x7f84d806a7a0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f84e00090b0 tx=0x7f84e0008030 comp rx=0 tx=0).stop 2023-03-03T16:55:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 -- 172.21.15.7:0/3056936329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec071c30 msgr2=0x7f84ec0ac6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec071c30 0x7f84ec0ac6c0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f84dc001eb0 tx=0x7f84dc00c8b0 comp rx=0 tx=0).stop 2023-03-03T16:55:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 -- 172.21.15.7:0/3056936329 shutdown_connections 2023-03-03T16:55:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84ec155950 0x7f84ec148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84d80682f0 0x7f84d806a7a0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84ec071c30 0x7f84ec0ac6c0 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:20.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 --2- 172.21.15.7:0/3056936329 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f84ec0aac00 0x7f84ec14a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:20.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 -- 172.21.15.7:0/3056936329 >> 172.21.15.7:0/3056936329 conn(0x7f84ec0736f0 msgr2=0x7f84ec14f720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:20.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.583+0000 7f84f11ac700 1 -- 172.21.15.7:0/3056936329 shutdown_connections 2023-03-03T16:55:20.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:20.584+0000 7f84f11ac700 1 -- 172.21.15.7:0/3056936329 wait complete. 2023-03-03T16:55:20.585 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:55:20.595 INFO:tasks.mds_thrash.fs.[b]:waiting for 35 secs before thrashing 2023-03-03T16:55:20.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:20.623 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:43.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:55:43.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.274+0000 7fcf03a48700 1 Processor -- start 2023-03-03T16:55:43.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.274+0000 7fcf03a48700 1 -- start start 2023-03-03T16:55:43.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf03a48700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcefc14f9d0 0x7fcefc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:43.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf03a48700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcefc14fda0 0x7fcefc071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf03a48700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcefc0abe20 0x7fcefc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf03a48700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0c2f30 con 0x7fcefc0abe20 2023-03-03T16:55:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf03a48700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc0c31d0 con 0x7fcefc14fda0 2023-03-03T16:55:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf03a48700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc1524a0 con 0x7fcefc14f9d0 2023-03-03T16:55:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf01fe5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcefc0abe20 0x7fcefc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf01fe5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcefc0abe20 0x7fcefc1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59200/0 (socket says 172.21.15.7:59200) 2023-03-03T16:55:43.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf01fe5700 1 -- 172.21.15.7:0/582697686 learned_addr learned my addr 172.21.15.7:0/582697686 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:55:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf017e4700 1 --2- 172.21.15.7:0/582697686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcefc14f9d0 0x7fcefc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.275+0000 7fcf00fe3700 1 --2- 172.21.15.7:0/582697686 >> v2:172.21.15.143:3300/0 conn(0x7fcefc14fda0 0x7fcefc071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcf01fe5700 1 -- 172.21.15.7:0/582697686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcefc14f9d0 msgr2=0x7fcefc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:55:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcf01fe5700 1 --2- 172.21.15.7:0/582697686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcefc14f9d0 0x7fcefc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcf01fe5700 1 -- 172.21.15.7:0/582697686 >> v2:172.21.15.143:3300/0 conn(0x7fcefc14fda0 msgr2=0x7fcefc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:55:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcf01fe5700 1 --2- 172.21.15.7:0/582697686 >> v2:172.21.15.143:3300/0 conn(0x7fcefc14fda0 0x7fcefc071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcf01fe5700 1 -- 172.21.15.7:0/582697686 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0c34b0 con 0x7fcefc0abe20 2023-03-03T16:55:43.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcf01fe5700 1 --2- 172.21.15.7:0/582697686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 0x7fcefc1529c0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0ab220 tx=0x7fcef400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23b426eb82379a2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:43.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcef3fff700 1 -- 172.21.15.7:0/582697686 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef400e070 con 0x7fcefc0abe20 2023-03-03T16:55:43.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcef3fff700 1 -- 172.21.15.7:0/582697686 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef400b910 con 0x7fcefc0abe20 2023-03-03T16:55:43.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcef3fff700 1 -- 172.21.15.7:0/582697686 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef400bcc0 con 0x7fcefc0abe20 2023-03-03T16:55:43.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcf03a48700 1 -- 172.21.15.7:0/582697686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 msgr2=0x7fcefc1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:43.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.276+0000 7fcf03a48700 1 --2- 172.21.15.7:0/582697686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 0x7fcefc1529c0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0ab220 tx=0x7fcef400b330 comp rx=0 tx=0).stop 2023-03-03T16:55:43.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.277+0000 7fcf03a48700 1 -- 172.21.15.7:0/582697686 shutdown_connections 2023-03-03T16:55:43.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.277+0000 7fcf03a48700 1 --2- 172.21.15.7:0/582697686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 0x7fcefc1529c0 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.277+0000 7fcf03a48700 1 --2- 172.21.15.7:0/582697686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcefc14f9d0 0x7fcefc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.277+0000 7fcf03a48700 1 --2- 172.21.15.7:0/582697686 >> v2:172.21.15.143:3300/0 conn(0x7fcefc14fda0 0x7fcefc071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.277+0000 7fcf03a48700 1 -- 172.21.15.7:0/582697686 >> 172.21.15.7:0/582697686 conn(0x7fcefc0c8900 msgr2=0x7fcefc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:43.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.277+0000 7fcf03a48700 1 -- 172.21.15.7:0/582697686 shutdown_connections 2023-03-03T16:55:43.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.277+0000 7fcf03a48700 1 -- 172.21.15.7:0/582697686 wait complete. 2023-03-03T16:55:43.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.278+0000 7fcf03a48700 1 Processor -- start 2023-03-03T16:55:43.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.278+0000 7fcf03a48700 1 -- start start 2023-03-03T16:55:43.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.278+0000 7fcf03a48700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 0x7fcefc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:43.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf03a48700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcefc14f9d0 0x7fcefc06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:43.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf03a48700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcefc14fda0 0x7fcefc1558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:43.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf03a48700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc06ded0 con 0x7fcefc0abe20 2023-03-03T16:55:43.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf03a48700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc06b590 con 0x7fcefc14fda0 2023-03-03T16:55:43.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf03a48700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc1ed650 con 0x7fcefc14f9d0 2023-03-03T16:55:43.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf017e4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 0x7fcefc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:43.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf017e4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 0x7fcefc06ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59202/0 (socket says 172.21.15.7:59202) 2023-03-03T16:55:43.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf017e4700 1 -- 172.21.15.7:0/989285076 learned_addr learned my addr 172.21.15.7:0/989285076 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:55:43.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf01fe5700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcefc14fda0 0x7fcefc1558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:43.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf00fe3700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcefc14f9d0 0x7fcefc06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:43.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf017e4700 1 -- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcefc14f9d0 msgr2=0x7fcefc06d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:43.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf017e4700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcefc14f9d0 0x7fcefc06d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf017e4700 1 -- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcefc14fda0 msgr2=0x7fcefc1558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:43.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf017e4700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcefc14fda0 0x7fcefc1558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf017e4700 1 -- 172.21.15.7:0/989285076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef4009b30 con 0x7fcefc0abe20 2023-03-03T16:55:43.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf017e4700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 0x7fcefc06ece0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7fceec00e9e0 tx=0x7fceec00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:43.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcef1ffb700 1 -- 172.21.15.7:0/989285076 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec009710 con 0x7fcefc0abe20 2023-03-03T16:55:43.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcef1ffb700 1 -- 172.21.15.7:0/989285076 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec00ee80 con 0x7fcefc0abe20 2023-03-03T16:55:43.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcf03a48700 1 -- 172.21.15.7:0/989285076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc0ac350 con 0x7fcefc0abe20 2023-03-03T16:55:43.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.279+0000 7fcef1ffb700 1 -- 172.21.15.7:0/989285076 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec00c9f0 con 0x7fcefc0abe20 2023-03-03T16:55:43.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.280+0000 7fcf03a48700 1 -- 172.21.15.7:0/989285076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc0ab630 con 0x7fcefc0abe20 2023-03-03T16:55:43.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.281+0000 7fcf03a48700 1 -- 172.21.15.7:0/989285076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc0c3e10 con 0x7fcefc0abe20 2023-03-03T16:55:43.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.281+0000 7fcef1ffb700 1 -- 172.21.15.7:0/989285076 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fceec023070 con 0x7fcefc0abe20 2023-03-03T16:55:43.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.281+0000 7fcef1ffb700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcee00682f0 0x7fcee006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:43.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.281+0000 7fcef1ffb700 1 -- 172.21.15.7:0/989285076 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fceec0862b0 con 0x7fcefc0abe20 2023-03-03T16:55:43.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.282+0000 7fcf00fe3700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcee00682f0 0x7fcee006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:43.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.282+0000 7fcf00fe3700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcee00682f0 0x7fcee006a7a0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fcefc1e7310 tx=0x7fcef8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:43.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.286+0000 7fcef1ffb700 1 -- 172.21.15.7:0/989285076 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fceec0525f0 con 0x7fcefc0abe20 2023-03-03T16:55:43.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.523+0000 7fcf03a48700 1 -- 172.21.15.7:0/989285076 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcefc074bb0 con 0x7fcefc0abe20 2023-03-03T16:55:43.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.524+0000 7fcef1ffb700 1 -- 172.21.15.7:0/989285076 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fceec055c10 con 0x7fcefc0abe20 2023-03-03T16:55:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:55:43.524 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:55:43.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 -- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcee00682f0 msgr2=0x7fcee006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:43.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcee00682f0 0x7fcee006a7a0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fcefc1e7310 tx=0x7fcef8009210 comp rx=0 tx=0).stop 2023-03-03T16:55:43.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 -- 172.21.15.7:0/989285076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 msgr2=0x7fcefc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:43.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 0x7fcefc06ece0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7fceec00e9e0 tx=0x7fceec00ecf0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 -- 172.21.15.7:0/989285076 shutdown_connections 2023-03-03T16:55:43.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcefc14fda0 0x7fcefc1558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcee00682f0 0x7fcee006a7a0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcefc0abe20 0x7fcefc06ece0 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 --2- 172.21.15.7:0/989285076 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcefc14f9d0 0x7fcefc06d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:43.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 -- 172.21.15.7:0/989285076 >> 172.21.15.7:0/989285076 conn(0x7fcefc0c8900 msgr2=0x7fcefc0c7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:43.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 -- 172.21.15.7:0/989285076 shutdown_connections 2023-03-03T16:55:43.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:43.526+0000 7fcf03a48700 1 -- 172.21.15.7:0/989285076 wait complete. 2023-03-03T16:55:43.527 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:55:43.537 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before thrashing 2023-03-03T16:55:50.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:55:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.678+0000 7fc39fef4700 1 Processor -- start 2023-03-03T16:55:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.678+0000 7fc39fef4700 1 -- start start 2023-03-03T16:55:50.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39fef4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc3980c0ee0 0x7fc3980c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:50.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39fef4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc39806d240 0x7fc39806d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:50.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39fef4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3980c2ff0 con 0x7fc39813a160 2023-03-03T16:55:50.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39fef4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc398158410 con 0x7fc3980c0ee0 2023-03-03T16:55:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39fef4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc398154450 con 0x7fc39806d240 2023-03-03T16:55:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39d48f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc3980c0ee0 0x7fc3980c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39dc90700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc39813a160 0x7fc3980c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53194/0 (socket says 172.21.15.7:53194) 2023-03-03T16:55:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39dc90700 1 -- 172.21.15.7:0/1441026343 learned_addr learned my addr 172.21.15.7:0/1441026343 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:55:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39e491700 1 --2- 172.21.15.7:0/1441026343 >> v2:172.21.15.143:3300/0 conn(0x7fc39806d240 0x7fc39806d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39d48f700 1 -- 172.21.15.7:0/1441026343 >> v2:172.21.15.143:3300/0 conn(0x7fc39806d240 msgr2=0x7fc39806d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:55:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39d48f700 1 --2- 172.21.15.7:0/1441026343 >> v2:172.21.15.143:3300/0 conn(0x7fc39806d240 0x7fc39806d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39d48f700 1 -- 172.21.15.7:0/1441026343 >> v1:172.21.15.7:6789/0 conn(0x7fc39813a160 legacy=0x7fc3980c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39d48f700 1 -- 172.21.15.7:0/1441026343 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc398157ae0 con 0x7fc3980c0ee0 2023-03-03T16:55:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39cc8e700 1 -- 172.21.15.7:0/1441026343 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47563415 0 0) 0x7fc394001110 con 0x7fc39813a160 2023-03-03T16:55:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39d48f700 1 --2- 172.21.15.7:0/1441026343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 0x7fc3980c1da0 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7fc3980c1380 tx=0x7fc388009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95aba57199494c60 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.679+0000 7fc39cc8e700 1 -- 172.21.15.7:0/1441026343 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388011830 con 0x7fc3980c0ee0 2023-03-03T16:55:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.680+0000 7fc39cc8e700 1 -- 172.21.15.7:0/1441026343 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc388011990 con 0x7fc3980c0ee0 2023-03-03T16:55:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.680+0000 7fc39cc8e700 1 -- 172.21.15.7:0/1441026343 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388011d20 con 0x7fc3980c0ee0 2023-03-03T16:55:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.680+0000 7fc39fef4700 1 -- 172.21.15.7:0/1441026343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 msgr2=0x7fc3980c1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.680+0000 7fc39fef4700 1 --2- 172.21.15.7:0/1441026343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 0x7fc3980c1da0 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7fc3980c1380 tx=0x7fc388009b20 comp rx=0 tx=0).stop 2023-03-03T16:55:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.680+0000 7fc39fef4700 1 -- 172.21.15.7:0/1441026343 shutdown_connections 2023-03-03T16:55:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.680+0000 7fc39fef4700 1 --2- 172.21.15.7:0/1441026343 >> v2:172.21.15.143:3300/0 conn(0x7fc39806d240 0x7fc39806d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.680+0000 7fc39fef4700 1 --2- 172.21.15.7:0/1441026343 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 0x7fc3980c1da0 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.680+0000 7fc39fef4700 1 -- 172.21.15.7:0/1441026343 >> 172.21.15.7:0/1441026343 conn(0x7fc39806c110 msgr2=0x7fc39806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.680+0000 7fc39fef4700 1 -- 172.21.15.7:0/1441026343 shutdown_connections 2023-03-03T16:55:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.680+0000 7fc39fef4700 1 -- 172.21.15.7:0/1441026343 wait complete. 2023-03-03T16:55:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.681+0000 7fc39fef4700 1 Processor -- start 2023-03-03T16:55:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.681+0000 7fc39fef4700 1 -- start start 2023-03-03T16:55:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39fef4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc39806d240 0x7fc398071490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39fef4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 0x7fc398134720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39fef4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc39813a160 0x7fc3981ee120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39fef4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3980c5d10 con 0x7fc3980c0ee0 2023-03-03T16:55:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39fef4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3981ddc80 con 0x7fc39813a160 2023-03-03T16:55:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39fef4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3981521e0 con 0x7fc39806d240 2023-03-03T16:55:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39d48f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 0x7fc398134720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39d48f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 0x7fc398134720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59222/0 (socket says 172.21.15.7:59222) 2023-03-03T16:55:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39d48f700 1 -- 172.21.15.7:0/1573539702 learned_addr learned my addr 172.21.15.7:0/1573539702 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:55:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39e491700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc39813a160 0x7fc3981ee120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.682+0000 7fc39dc90700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc39806d240 0x7fc398071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc39d48f700 1 -- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc39806d240 msgr2=0x7fc398071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc39d48f700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc39806d240 0x7fc398071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc39d48f700 1 -- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc39813a160 msgr2=0x7fc3981ee120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc39d48f700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc39813a160 0x7fc3981ee120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc39d48f700 1 -- 172.21.15.7:0/1573539702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3880094a0 con 0x7fc3980c0ee0 2023-03-03T16:55:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc39dc90700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc39806d240 0x7fc398071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:55:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc39e491700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc39813a160 0x7fc3981ee120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:55:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc39d48f700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 0x7fc398134720 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7fc39806cff0 tx=0x7fc388004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc38e7fc700 1 -- 172.21.15.7:0/1573539702 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3880038a0 con 0x7fc3980c0ee0 2023-03-03T16:55:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc38e7fc700 1 -- 172.21.15.7:0/1573539702 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38801f070 con 0x7fc3980c0ee0 2023-03-03T16:55:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc38e7fc700 1 -- 172.21.15.7:0/1573539702 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388003c20 con 0x7fc3980c0ee0 2023-03-03T16:55:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc39fef4700 1 -- 172.21.15.7:0/1573539702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3981580a0 con 0x7fc3980c0ee0 2023-03-03T16:55:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.683+0000 7fc39fef4700 1 -- 172.21.15.7:0/1573539702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc398157dc0 con 0x7fc3980c0ee0 2023-03-03T16:55:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.684+0000 7fc39fef4700 1 -- 172.21.15.7:0/1573539702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc398158880 con 0x7fc3980c0ee0 2023-03-03T16:55:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.685+0000 7fc38e7fc700 1 -- 172.21.15.7:0/1573539702 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc38800f050 con 0x7fc3980c0ee0 2023-03-03T16:55:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.685+0000 7fc38e7fc700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc37c0682f0 0x7fc37c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.685+0000 7fc38e7fc700 1 -- 172.21.15.7:0/1573539702 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc388021070 con 0x7fc3980c0ee0 2023-03-03T16:55:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.685+0000 7fc39dc90700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc37c0682f0 0x7fc37c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.686+0000 7fc39dc90700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc37c0682f0 0x7fc37c06a7a0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fc394000f60 tx=0x7fc394007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:50.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.689+0000 7fc38e7fc700 1 -- 172.21.15.7:0/1573539702 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc388053e80 con 0x7fc3980c0ee0 2023-03-03T16:55:50.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:50.694 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:55:50.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.930+0000 7fc39fef4700 1 -- 172.21.15.7:0/1573539702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc398074bb0 con 0x7fc3980c0ee0 2023-03-03T16:55:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.930+0000 7fc38e7fc700 1 -- 172.21.15.7:0/1573539702 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc3880574a0 con 0x7fc3980c0ee0 2023-03-03T16:55:50.930 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:55:50.930 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:55:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 -- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc37c0682f0 msgr2=0x7fc37c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc37c0682f0 0x7fc37c06a7a0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fc394000f60 tx=0x7fc394007d30 comp rx=0 tx=0).stop 2023-03-03T16:55:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 -- 172.21.15.7:0/1573539702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 msgr2=0x7fc398134720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 0x7fc398134720 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7fc39806cff0 tx=0x7fc388004750 comp rx=0 tx=0).stop 2023-03-03T16:55:50.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 -- 172.21.15.7:0/1573539702 shutdown_connections 2023-03-03T16:55:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc39813a160 0x7fc3981ee120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc39806d240 0x7fc398071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc37c0682f0 0x7fc37c06a7a0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 --2- 172.21.15.7:0/1573539702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3980c0ee0 0x7fc398134720 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 -- 172.21.15.7:0/1573539702 >> 172.21.15.7:0/1573539702 conn(0x7fc39806c110 msgr2=0x7fc398139490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 -- 172.21.15.7:0/1573539702 shutdown_connections 2023-03-03T16:55:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:50.932+0000 7fc39fef4700 1 -- 172.21.15.7:0/1573539702 wait complete. 2023-03-03T16:55:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:55:50.944 INFO:tasks.mds_thrash.fs.[a]:waiting for 117 secs before thrashing 2023-03-03T16:55:55.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:55:55.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.737+0000 7fe776ee5700 1 Processor -- start 2023-03-03T16:55:55.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.737+0000 7fe776ee5700 1 -- start start 2023-03-03T16:55:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe776ee5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe77006c6c0 0x7fe7700736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe776ee5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe770157ff0 con 0x7fe770151cd0 2023-03-03T16:55:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe776ee5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77006d640 con 0x7fe7700c2e30 2023-03-03T16:55:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe776ee5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe77006cda0 con 0x7fe77006c6c0 2023-03-03T16:55:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe775482700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe77006c6c0 0x7fe7700736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe774c81700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe770151cd0 0x7fe770154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53210/0 (socket says 172.21.15.7:53210) 2023-03-03T16:55:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe774c81700 1 -- 172.21.15.7:0/2821072484 learned_addr learned my addr 172.21.15.7:0/2821072484 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:55:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe775482700 1 -- 172.21.15.7:0/2821072484 >> v1:172.21.15.143:6789/0 conn(0x7fe7700c2e30 legacy=0x7fe770158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe775482700 1 -- 172.21.15.7:0/2821072484 >> v1:172.21.15.7:6789/0 conn(0x7fe770151cd0 legacy=0x7fe770154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe775482700 1 -- 172.21.15.7:0/2821072484 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77006d080 con 0x7fe77006c6c0 2023-03-03T16:55:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe76f7fe700 1 -- 172.21.15.7:0/2821072484 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218803945 0 0) 0x7fe770157ff0 con 0x7fe770151cd0 2023-03-03T16:55:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.738+0000 7fe775482700 1 --2- 172.21.15.7:0/2821072484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe77006c6c0 0x7fe7700736f0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7fe768009d90 tx=0x7fe76800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b69de825978e00f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.739+0000 7fe76f7fe700 1 -- 172.21.15.7:0/2821072484 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76800e070 con 0x7fe77006c6c0 2023-03-03T16:55:55.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.739+0000 7fe76f7fe700 1 -- 172.21.15.7:0/2821072484 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76800bbd0 con 0x7fe77006c6c0 2023-03-03T16:55:55.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.739+0000 7fe76f7fe700 1 -- 172.21.15.7:0/2821072484 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768012430 con 0x7fe77006c6c0 2023-03-03T16:55:55.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.739+0000 7fe776ee5700 1 -- 172.21.15.7:0/2821072484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe77006c6c0 msgr2=0x7fe7700736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:55.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.739+0000 7fe776ee5700 1 --2- 172.21.15.7:0/2821072484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe77006c6c0 0x7fe7700736f0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7fe768009d90 tx=0x7fe76800b330 comp rx=0 tx=0).stop 2023-03-03T16:55:55.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.739+0000 7fe776ee5700 1 -- 172.21.15.7:0/2821072484 shutdown_connections 2023-03-03T16:55:55.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.739+0000 7fe776ee5700 1 --2- 172.21.15.7:0/2821072484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe77006c6c0 0x7fe7700736f0 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:55.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.739+0000 7fe776ee5700 1 -- 172.21.15.7:0/2821072484 >> 172.21.15.7:0/2821072484 conn(0x7fe7700a94d0 msgr2=0x7fe7700a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:55.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.739+0000 7fe776ee5700 1 -- 172.21.15.7:0/2821072484 shutdown_connections 2023-03-03T16:55:55.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.739+0000 7fe776ee5700 1 -- 172.21.15.7:0/2821072484 wait complete. 2023-03-03T16:55:55.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.740+0000 7fe776ee5700 1 Processor -- start 2023-03-03T16:55:55.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.740+0000 7fe776ee5700 1 -- start start 2023-03-03T16:55:55.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.741+0000 7fe776ee5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe77006c6c0 0x7fe7700c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:55.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.741+0000 7fe776ee5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7700c2e30 0x7fe7701566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:55.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.741+0000 7fe776ee5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe770151cd0 0x7fe770156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:55.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.741+0000 7fe776ee5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7701560b0 con 0x7fe770151cd0 2023-03-03T16:55:55.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.741+0000 7fe776ee5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77014dd20 con 0x7fe7700c2e30 2023-03-03T16:55:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.741+0000 7fe776ee5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77014ed60 con 0x7fe77006c6c0 2023-03-03T16:55:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.741+0000 7fe775482700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe770151cd0 0x7fe770156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.741+0000 7fe775482700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe770151cd0 0x7fe770156c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59240/0 (socket says 172.21.15.7:59240) 2023-03-03T16:55:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.741+0000 7fe775482700 1 -- 172.21.15.7:0/2385414729 learned_addr learned my addr 172.21.15.7:0/2385414729 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:55:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe76ffff700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7700c2e30 0x7fe7701566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe774c81700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe77006c6c0 0x7fe7700c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:55.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe775482700 1 -- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe77006c6c0 msgr2=0x7fe7700c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:55.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe775482700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe77006c6c0 0x7fe7700c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:55.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe775482700 1 -- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7700c2e30 msgr2=0x7fe7701566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:55.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe775482700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7700c2e30 0x7fe7701566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:55.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe775482700 1 -- 172.21.15.7:0/2385414729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768009a70 con 0x7fe770151cd0 2023-03-03T16:55:55.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe76ffff700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7700c2e30 0x7fe7701566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:55:55.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe775482700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe770151cd0 0x7fe770156c10 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7fe768014040 tx=0x7fe768004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:55.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe76d7fa700 1 -- 172.21.15.7:0/2385414729 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768012430 con 0x7fe770151cd0 2023-03-03T16:55:55.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe76d7fa700 1 -- 172.21.15.7:0/2385414729 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768020070 con 0x7fe770151cd0 2023-03-03T16:55:55.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe776ee5700 1 -- 172.21.15.7:0/2385414729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe770157c80 con 0x7fe770151cd0 2023-03-03T16:55:55.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe76d7fa700 1 -- 172.21.15.7:0/2385414729 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680098f0 con 0x7fe770151cd0 2023-03-03T16:55:55.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.742+0000 7fe776ee5700 1 -- 172.21.15.7:0/2385414729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77006d360 con 0x7fe770151cd0 2023-03-03T16:55:55.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.743+0000 7fe776ee5700 1 -- 172.21.15.7:0/2385414729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe770073c30 con 0x7fe770151cd0 2023-03-03T16:55:55.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.744+0000 7fe76d7fa700 1 -- 172.21.15.7:0/2385414729 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe768026400 con 0x7fe770151cd0 2023-03-03T16:55:55.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.744+0000 7fe76d7fa700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7540682f0 0x7fe75406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:55:55.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.744+0000 7fe76d7fa700 1 -- 172.21.15.7:0/2385414729 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe7680882f0 con 0x7fe770151cd0 2023-03-03T16:55:55.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.744+0000 7fe774c81700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7540682f0 0x7fe75406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:55:55.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.745+0000 7fe774c81700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7540682f0 0x7fe75406a7a0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fe764000f60 tx=0x7fe764007db0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:55:55.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.748+0000 7fe76d7fa700 1 -- 172.21.15.7:0/2385414729 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe768054630 con 0x7fe770151cd0 2023-03-03T16:55:55.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.987+0000 7fe776ee5700 1 -- 172.21.15.7:0/2385414729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe770074bb0 con 0x7fe770151cd0 2023-03-03T16:55:55.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.988+0000 7fe76d7fa700 1 -- 172.21.15.7:0/2385414729 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe768057c50 con 0x7fe770151cd0 2023-03-03T16:55:55.988 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:55:55.988 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:55:55.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 -- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7540682f0 msgr2=0x7fe75406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:55.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7540682f0 0x7fe75406a7a0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fe764000f60 tx=0x7fe764007db0 comp rx=0 tx=0).stop 2023-03-03T16:55:55.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 -- 172.21.15.7:0/2385414729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe770151cd0 msgr2=0x7fe770156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:55:55.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe770151cd0 0x7fe770156c10 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7fe768014040 tx=0x7fe768004ad0 comp rx=0 tx=0).stop 2023-03-03T16:55:55.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 -- 172.21.15.7:0/2385414729 shutdown_connections 2023-03-03T16:55:55.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7540682f0 0x7fe75406a7a0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:55.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe770151cd0 0x7fe770156c10 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:55.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe77006c6c0 0x7fe7700c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:55.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 --2- 172.21.15.7:0/2385414729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7700c2e30 0x7fe7701566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:55:55.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 -- 172.21.15.7:0/2385414729 >> 172.21.15.7:0/2385414729 conn(0x7fe7700a94d0 msgr2=0x7fe770071ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:55:55.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 -- 172.21.15.7:0/2385414729 shutdown_connections 2023-03-03T16:55:55.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:55:55.990+0000 7fe776ee5700 1 -- 172.21.15.7:0/2385414729 wait complete. 2023-03-03T16:55:55.992 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:55:56.001 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T16:56:20.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:20.768 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:50.852 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:50.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:56:58.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:56:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.142+0000 7feb3ad26700 1 Processor -- start 2023-03-03T16:56:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.142+0000 7feb3ad26700 1 -- start start 2023-03-03T16:56:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.142+0000 7feb3ad26700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb34151960 0x7feb340736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:56:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.142+0000 7feb3ad26700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb340c3ac0 0x7feb341579f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:56:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.142+0000 7feb3ad26700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb34154ee0 con 0x7feb340723f0 2023-03-03T16:56:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.142+0000 7feb3ad26700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb340c3f30 con 0x7feb34151960 2023-03-03T16:56:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.142+0000 7feb3ad26700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb341545f0 con 0x7feb340c3ac0 2023-03-03T16:56:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb38ac2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb34151960 0x7feb340736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:56:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb392c3700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7feb340723f0 0x7feb34157f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53280/0 (socket says 172.21.15.7:53280) 2023-03-03T16:56:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb392c3700 1 -- 172.21.15.7:0/1272301030 learned_addr learned my addr 172.21.15.7:0/1272301030 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:56:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb33fff700 1 --2- 172.21.15.7:0/1272301030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb340c3ac0 0x7feb341579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:56:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb38ac2700 1 -- 172.21.15.7:0/1272301030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb340c3ac0 msgr2=0x7feb341579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:56:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb38ac2700 1 --2- 172.21.15.7:0/1272301030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb340c3ac0 0x7feb341579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:56:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb38ac2700 1 -- 172.21.15.7:0/1272301030 >> v1:172.21.15.7:6789/0 conn(0x7feb340723f0 legacy=0x7feb34157f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:56:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb38ac2700 1 -- 172.21.15.7:0/1272301030 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb34155370 con 0x7feb34151960 2023-03-03T16:56:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb337fe700 1 -- 172.21.15.7:0/1272301030 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618556346 0 0) 0x7feb34154ee0 con 0x7feb340723f0 2023-03-03T16:56:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb38ac2700 1 --2- 172.21.15.7:0/1272301030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb34151960 0x7feb340736f0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7feb28009840 tx=0x7feb28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cafe44b0caa98d3a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:56:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb337fe700 1 -- 172.21.15.7:0/1272301030 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28011830 con 0x7feb34151960 2023-03-03T16:56:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb337fe700 1 -- 172.21.15.7:0/1272301030 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb28011990 con 0x7feb34151960 2023-03-03T16:56:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.143+0000 7feb337fe700 1 -- 172.21.15.7:0/1272301030 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28011d20 con 0x7feb34151960 2023-03-03T16:56:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.144+0000 7feb3ad26700 1 -- 172.21.15.7:0/1272301030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb34151960 msgr2=0x7feb340736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:56:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.144+0000 7feb3ad26700 1 --2- 172.21.15.7:0/1272301030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb34151960 0x7feb340736f0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7feb28009840 tx=0x7feb28009b50 comp rx=0 tx=0).stop 2023-03-03T16:56:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.144+0000 7feb3ad26700 1 -- 172.21.15.7:0/1272301030 shutdown_connections 2023-03-03T16:56:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.144+0000 7feb3ad26700 1 --2- 172.21.15.7:0/1272301030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb34151960 0x7feb340736f0 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:56:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.144+0000 7feb3ad26700 1 --2- 172.21.15.7:0/1272301030 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb340c3ac0 0x7feb341579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:56:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.144+0000 7feb3ad26700 1 -- 172.21.15.7:0/1272301030 >> 172.21.15.7:0/1272301030 conn(0x7feb3406c060 msgr2=0x7feb3406c460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:56:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.144+0000 7feb3ad26700 1 -- 172.21.15.7:0/1272301030 shutdown_connections 2023-03-03T16:56:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.144+0000 7feb3ad26700 1 -- 172.21.15.7:0/1272301030 wait complete. 2023-03-03T16:56:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.145+0000 7feb3ad26700 1 Processor -- start 2023-03-03T16:56:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.145+0000 7feb3ad26700 1 -- start start 2023-03-03T16:56:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb3ad26700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb340723f0 0x7feb34158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:56:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb3ad26700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb340c3ac0 0x7feb34149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:56:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb3ad26700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34151960 0x7feb340c6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:56:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb3ad26700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb340c60b0 con 0x7feb340c3ac0 2023-03-03T16:56:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb3ad26700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb341ea9c0 con 0x7feb340723f0 2023-03-03T16:56:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb3ad26700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb340c5780 con 0x7feb34151960 2023-03-03T16:56:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb33fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb340c3ac0 0x7feb34149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:56:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb33fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb340c3ac0 0x7feb34149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59304/0 (socket says 172.21.15.7:59304) 2023-03-03T16:56:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb33fff700 1 -- 172.21.15.7:0/2816329731 learned_addr learned my addr 172.21.15.7:0/2816329731 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:56:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb392c3700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34151960 0x7feb340c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:56:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb33fff700 1 -- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34151960 msgr2=0x7feb340c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:56:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb38ac2700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb340723f0 0x7feb34158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:56:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb33fff700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34151960 0x7feb340c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:56:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb33fff700 1 -- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb340723f0 msgr2=0x7feb34158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:56:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb33fff700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb340723f0 0x7feb34158280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:56:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb33fff700 1 -- 172.21.15.7:0/2816329731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb280094a0 con 0x7feb340c3ac0 2023-03-03T16:56:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.146+0000 7feb33fff700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb340c3ac0 0x7feb34149a10 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7feb2000e810 tx=0x7feb2000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:56:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.147+0000 7feb317fa700 1 -- 172.21.15.7:0/2816329731 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20013610 con 0x7feb340c3ac0 2023-03-03T16:56:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.147+0000 7feb317fa700 1 -- 172.21.15.7:0/2816329731 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb20013770 con 0x7feb340c3ac0 2023-03-03T16:56:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.147+0000 7feb317fa700 1 -- 172.21.15.7:0/2816329731 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20013b20 con 0x7feb340c3ac0 2023-03-03T16:56:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.147+0000 7feb3ad26700 1 -- 172.21.15.7:0/2816329731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb34154b70 con 0x7feb340c3ac0 2023-03-03T16:56:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.147+0000 7feb3ad26700 1 -- 172.21.15.7:0/2816329731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb34154890 con 0x7feb340c3ac0 2023-03-03T16:56:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.148+0000 7feb3ad26700 1 -- 172.21.15.7:0/2816329731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb34151d30 con 0x7feb340c3ac0 2023-03-03T16:56:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.148+0000 7feb317fa700 1 -- 172.21.15.7:0/2816329731 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feb20023070 con 0x7feb340c3ac0 2023-03-03T16:56:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.148+0000 7feb317fa700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb180682f0 0x7feb1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:56:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.149+0000 7feb317fa700 1 -- 172.21.15.7:0/2816329731 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7feb20085e20 con 0x7feb340c3ac0 2023-03-03T16:56:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.149+0000 7feb38ac2700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb180682f0 0x7feb1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:56:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.149+0000 7feb38ac2700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb180682f0 0x7feb1806a7a0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7feb28000c00 tx=0x7feb28008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:56:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.153+0000 7feb317fa700 1 -- 172.21.15.7:0/2816329731 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feb200521c0 con 0x7feb340c3ac0 2023-03-03T16:56:58.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.388+0000 7feb3ad26700 1 -- 172.21.15.7:0/2816329731 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb34074bb0 con 0x7feb340c3ac0 2023-03-03T16:56:58.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.388+0000 7feb317fa700 1 -- 172.21.15.7:0/2816329731 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7feb200557e0 con 0x7feb340c3ac0 2023-03-03T16:56:58.388 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:56:58.389 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:56:58.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 -- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb180682f0 msgr2=0x7feb1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:56:58.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb180682f0 0x7feb1806a7a0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7feb28000c00 tx=0x7feb28008030 comp rx=0 tx=0).stop 2023-03-03T16:56:58.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 -- 172.21.15.7:0/2816329731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb340c3ac0 msgr2=0x7feb34149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:56:58.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb340c3ac0 0x7feb34149a10 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7feb2000e810 tx=0x7feb2000ebd0 comp rx=0 tx=0).stop 2023-03-03T16:56:58.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 -- 172.21.15.7:0/2816329731 shutdown_connections 2023-03-03T16:56:58.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feb34151960 0x7feb340c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:56:58.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb340723f0 0x7feb34158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:56:58.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feb180682f0 0x7feb1806a7a0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:56:58.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 --2- 172.21.15.7:0/2816329731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb340c3ac0 0x7feb34149a10 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:56:58.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 -- 172.21.15.7:0/2816329731 >> 172.21.15.7:0/2816329731 conn(0x7feb3406c060 msgr2=0x7feb3406d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:56:58.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 -- 172.21.15.7:0/2816329731 shutdown_connections 2023-03-03T16:56:58.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:56:58.390+0000 7feb3ad26700 1 -- 172.21.15.7:0/2816329731 wait complete. 2023-03-03T16:56:58.393 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:56:58.402 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T16:57:20.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:20.978 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:47.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:57:48.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.083+0000 7fd78e566700 1 Processor -- start 2023-03-03T16:57:48.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.083+0000 7fd78e566700 1 -- start start 2023-03-03T16:57:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.085+0000 7fd78e566700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd78814fda0 0x7fd788156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.085+0000 7fd78e566700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7880abe20 0x7fd7881529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.085+0000 7fd78e566700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7880c2f30 con 0x7fd78814f9d0 2023-03-03T16:57:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.085+0000 7fd78e566700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7880c31d0 con 0x7fd78814fda0 2023-03-03T16:57:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.085+0000 7fd78e566700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7881524a0 con 0x7fd7880abe20 2023-03-03T16:57:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd7877fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd78814fda0 0x7fd788156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd78cb03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7880abe20 0x7fd7881529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd7877fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd78814fda0 0x7fd788156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:55098/0 (socket says 172.21.15.7:55098) 2023-03-03T16:57:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd78cb03700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7880abe20 0x7fd7881529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:39166/0 (socket says 172.21.15.7:39166) 2023-03-03T16:57:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd7877fe700 1 -- 172.21.15.7:0/3416068406 learned_addr learned my addr 172.21.15.7:0/3416068406 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:57:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd7877fe700 1 -- 172.21.15.7:0/3416068406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7880abe20 msgr2=0x7fd7881529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:57:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd7877fe700 1 --2- 172.21.15.7:0/3416068406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7880abe20 0x7fd7881529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd7877fe700 1 -- 172.21.15.7:0/3416068406 >> v1:172.21.15.143:6789/0 conn(0x7fd78814f9d0 legacy=0x7fd788071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd7877fe700 1 -- 172.21.15.7:0/3416068406 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7880c34b0 con 0x7fd78814fda0 2023-03-03T16:57:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd78cb03700 1 --2- 172.21.15.7:0/3416068406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7880abe20 0x7fd7881529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:57:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd7877fe700 1 --2- 172.21.15.7:0/3416068406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd78814fda0 0x7fd788156430 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0098f0 tx=0x7fd77c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6a454dd02f0fb21 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:57:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd786ffd700 1 -- 172.21.15.7:0/3416068406 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c011830 con 0x7fd78814fda0 2023-03-03T16:57:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd786ffd700 1 -- 172.21.15.7:0/3416068406 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77c011990 con 0x7fd78814fda0 2023-03-03T16:57:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.086+0000 7fd786ffd700 1 -- 172.21.15.7:0/3416068406 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c011d20 con 0x7fd78814fda0 2023-03-03T16:57:48.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.087+0000 7fd78e566700 1 -- 172.21.15.7:0/3416068406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd78814fda0 msgr2=0x7fd788156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.087+0000 7fd78e566700 1 --2- 172.21.15.7:0/3416068406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd78814fda0 0x7fd788156430 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd77c0098f0 tx=0x7fd77c009c00 comp rx=0 tx=0).stop 2023-03-03T16:57:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.087+0000 7fd78e566700 1 -- 172.21.15.7:0/3416068406 shutdown_connections 2023-03-03T16:57:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.087+0000 7fd78e566700 1 --2- 172.21.15.7:0/3416068406 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7880abe20 0x7fd7881529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.087+0000 7fd78e566700 1 --2- 172.21.15.7:0/3416068406 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd78814fda0 0x7fd788156430 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.087+0000 7fd78e566700 1 -- 172.21.15.7:0/3416068406 >> 172.21.15.7:0/3416068406 conn(0x7fd7880c8900 msgr2=0x7fd7880c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:57:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.087+0000 7fd78e566700 1 -- 172.21.15.7:0/3416068406 shutdown_connections 2023-03-03T16:57:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.087+0000 7fd78e566700 1 -- 172.21.15.7:0/3416068406 wait complete. 2023-03-03T16:57:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.088+0000 7fd78e566700 1 Processor -- start 2023-03-03T16:57:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.088+0000 7fd78e566700 1 -- start start 2023-03-03T16:57:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd78e566700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7880abe20 0x7fd788154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd78e566700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd78814f9d0 0x7fd78806f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd78e566700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd78814fda0 0x7fd7881eb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd78e566700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd788155470 con 0x7fd78814fda0 2023-03-03T16:57:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd78e566700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd788154aa0 con 0x7fd7880abe20 2023-03-03T16:57:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd78e566700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78806d6e0 con 0x7fd78814f9d0 2023-03-03T16:57:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd78cb03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd78814fda0 0x7fd7881eb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd78cb03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd78814fda0 0x7fd7881eb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59332/0 (socket says 172.21.15.7:59332) 2023-03-03T16:57:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd78cb03700 1 -- 172.21.15.7:0/1097495150 learned_addr learned my addr 172.21.15.7:0/1097495150 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:57:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd787fff700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7880abe20 0x7fd788154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.089+0000 7fd7877fe700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd78814f9d0 0x7fd78806f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd78cb03700 1 -- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd78814f9d0 msgr2=0x7fd78806f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd78cb03700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd78814f9d0 0x7fd78806f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd78cb03700 1 -- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7880abe20 msgr2=0x7fd788154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd78cb03700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7880abe20 0x7fd788154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd78cb03700 1 -- 172.21.15.7:0/1097495150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780009a70 con 0x7fd78814fda0 2023-03-03T16:57:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd78cb03700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd78814fda0 0x7fd7881eb190 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7fd780009e10 tx=0x7fd78000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:57:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd784ff9700 1 -- 172.21.15.7:0/1097495150 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd780009cb0 con 0x7fd78814fda0 2023-03-03T16:57:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd78e566700 1 -- 172.21.15.7:0/1097495150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77c009550 con 0x7fd78814fda0 2023-03-03T16:57:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd78e566700 1 -- 172.21.15.7:0/1097495150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7880ac350 con 0x7fd78814fda0 2023-03-03T16:57:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd784ff9700 1 -- 172.21.15.7:0/1097495150 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78000f040 con 0x7fd78814fda0 2023-03-03T16:57:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.090+0000 7fd784ff9700 1 -- 172.21.15.7:0/1097495150 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78000ceb0 con 0x7fd78814fda0 2023-03-03T16:57:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.091+0000 7fd78e566700 1 -- 172.21.15.7:0/1097495150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7880ab630 con 0x7fd78814fda0 2023-03-03T16:57:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.092+0000 7fd784ff9700 1 -- 172.21.15.7:0/1097495150 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd780014480 con 0x7fd78814fda0 2023-03-03T16:57:48.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.092+0000 7fd784ff9700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd774068320 0x7fd77406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:57:48.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.092+0000 7fd784ff9700 1 -- 172.21.15.7:0/1097495150 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd780086e60 con 0x7fd78814fda0 2023-03-03T16:57:48.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.092+0000 7fd787fff700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd774068320 0x7fd77406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:57:48.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.093+0000 7fd787fff700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd774068320 0x7fd77406a7d0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fd7881546c0 tx=0x7fd778009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:57:48.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.097+0000 7fd784ff9700 1 -- 172.21.15.7:0/1097495150 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd780053180 con 0x7fd78814fda0 2023-03-03T16:57:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.337+0000 7fd78e566700 1 -- 172.21.15.7:0/1097495150 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd788071fa0 con 0x7fd78814fda0 2023-03-03T16:57:48.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.337+0000 7fd784ff9700 1 -- 172.21.15.7:0/1097495150 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd780018090 con 0x7fd78814fda0 2023-03-03T16:57:48.337 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:57:48.338 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:57:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 -- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd774068320 msgr2=0x7fd77406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd774068320 0x7fd77406a7d0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fd7881546c0 tx=0x7fd778009040 comp rx=0 tx=0).stop 2023-03-03T16:57:48.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 -- 172.21.15.7:0/1097495150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd78814fda0 msgr2=0x7fd7881eb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:57:48.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd78814fda0 0x7fd7881eb190 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7fd780009e10 tx=0x7fd78000c3a0 comp rx=0 tx=0).stop 2023-03-03T16:57:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 -- 172.21.15.7:0/1097495150 shutdown_connections 2023-03-03T16:57:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd774068320 0x7fd77406a7d0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd78814fda0 0x7fd7881eb190 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7880abe20 0x7fd788154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 --2- 172.21.15.7:0/1097495150 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd78814f9d0 0x7fd78806f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:57:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 -- 172.21.15.7:0/1097495150 >> 172.21.15.7:0/1097495150 conn(0x7fd7880c8900 msgr2=0x7fd788150c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:57:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 -- 172.21.15.7:0/1097495150 shutdown_connections 2023-03-03T16:57:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:57:48.339+0000 7fd78e566700 1 -- 172.21.15.7:0/1097495150 wait complete. 2023-03-03T16:57:48.343 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:57:48.351 INFO:tasks.mds_thrash.fs.[a]:waiting for 114 secs before thrashing 2023-03-03T16:57:51.114 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:57:51.117 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:02.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:58:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.543+0000 7f5763bca700 1 Processor -- start 2023-03-03T16:58:02.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.543+0000 7f5763bca700 1 -- start start 2023-03-03T16:58:02.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5763bca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 0x7f575c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:02.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5763bca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f575c14fda0 0x7f575c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:02.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5763bca700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c0c2f30 con 0x7f575c0abe20 2023-03-03T16:58:02.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5763bca700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c0c31d0 con 0x7f575c14fda0 2023-03-03T16:58:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5763bca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1524a0 con 0x7f575c14f9d0 2023-03-03T16:58:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5762167700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f575c0abe20 0x7f575c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53332/0 (socket says 172.21.15.7:53332) 2023-03-03T16:58:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5762167700 1 -- 172.21.15.7:0/16232365 learned_addr learned my addr 172.21.15.7:0/16232365 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:58:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5761165700 1 --2- 172.21.15.7:0/16232365 >> v2:172.21.15.143:3300/0 conn(0x7f575c14fda0 0x7f575c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5761966700 1 --2- 172.21.15.7:0/16232365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 0x7f575c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5760964700 1 -- 172.21.15.7:0/16232365 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991399879 0 0) 0x7f575c0c2f30 con 0x7f575c0abe20 2023-03-03T16:58:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.544+0000 7f5760964700 1 -- 172.21.15.7:0/16232365 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57480035b0 con 0x7f575c0abe20 2023-03-03T16:58:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.545+0000 7f5761165700 1 -- 172.21.15.7:0/16232365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 msgr2=0x7f575c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:58:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.545+0000 7f5760964700 1 -- 172.21.15.7:0/16232365 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2292499137 0 0) 0x7f57480035b0 con 0x7f575c0abe20 2023-03-03T16:58:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.545+0000 7f5761165700 1 --2- 172.21.15.7:0/16232365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 0x7f575c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.545+0000 7f5761165700 1 -- 172.21.15.7:0/16232365 >> v1:172.21.15.7:6789/0 conn(0x7f575c0abe20 legacy=0x7f575c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.545+0000 7f5761165700 1 -- 172.21.15.7:0/16232365 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0c34b0 con 0x7f575c14fda0 2023-03-03T16:58:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.545+0000 7f5761966700 1 --2- 172.21.15.7:0/16232365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 0x7f575c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T16:58:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.545+0000 7f5761165700 1 --2- 172.21.15.7:0/16232365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c14fda0 0x7f575c071890 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5758009840 tx=0x7f5758009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b7cf22e020c3b34 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:58:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.545+0000 7f5760964700 1 -- 172.21.15.7:0/16232365 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57540019f0 con 0x7f575c14fda0 2023-03-03T16:58:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.545+0000 7f5760964700 1 -- 172.21.15.7:0/16232365 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5758004030 con 0x7f575c14fda0 2023-03-03T16:58:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.545+0000 7f5760964700 1 -- 172.21.15.7:0/16232365 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580043c0 con 0x7f575c14fda0 2023-03-03T16:58:02.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.546+0000 7f5763bca700 1 -- 172.21.15.7:0/16232365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c14fda0 msgr2=0x7f575c071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:02.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.546+0000 7f5763bca700 1 --2- 172.21.15.7:0/16232365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c14fda0 0x7f575c071890 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5758009840 tx=0x7f5758009b50 comp rx=0 tx=0).stop 2023-03-03T16:58:02.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.546+0000 7f5763bca700 1 -- 172.21.15.7:0/16232365 shutdown_connections 2023-03-03T16:58:02.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.546+0000 7f5763bca700 1 --2- 172.21.15.7:0/16232365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 0x7f575c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:02.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.546+0000 7f5763bca700 1 --2- 172.21.15.7:0/16232365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c14fda0 0x7f575c071890 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:02.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.546+0000 7f5763bca700 1 -- 172.21.15.7:0/16232365 >> 172.21.15.7:0/16232365 conn(0x7f575c0c8900 msgr2=0x7f575c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:58:02.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.546+0000 7f5763bca700 1 -- 172.21.15.7:0/16232365 shutdown_connections 2023-03-03T16:58:02.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.546+0000 7f5763bca700 1 -- 172.21.15.7:0/16232365 wait complete. 2023-03-03T16:58:02.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.547+0000 7f5763bca700 1 Processor -- start 2023-03-03T16:58:02.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.547+0000 7f5763bca700 1 -- start start 2023-03-03T16:58:02.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5763bca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c0abe20 0x7f575c14eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:02.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5763bca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 0x7f575c1549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:02.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5763bca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c14fda0 0x7f575c1dbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5763bca700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1db9a0 con 0x7f575c0abe20 2023-03-03T16:58:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5763bca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1dd860 con 0x7f575c14fda0 2023-03-03T16:58:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5763bca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1dd050 con 0x7f575c14f9d0 2023-03-03T16:58:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5761966700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c0abe20 0x7f575c14eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5761966700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c0abe20 0x7f575c14eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59354/0 (socket says 172.21.15.7:59354) 2023-03-03T16:58:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5761966700 1 -- 172.21.15.7:0/3881228524 learned_addr learned my addr 172.21.15.7:0/3881228524 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:58:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5761165700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 0x7f575c1549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5762167700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c14fda0 0x7f575c1dbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5761966700 1 -- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 msgr2=0x7f575c1549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5761966700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 0x7f575c1549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5761966700 1 -- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c14fda0 msgr2=0x7f575c1dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:02.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5761966700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c14fda0 0x7f575c1dbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5761966700 1 -- 172.21.15.7:0/3881228524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c0094a0 con 0x7f575c0abe20 2023-03-03T16:58:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5762167700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c14fda0 0x7f575c1dbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T16:58:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.548+0000 7f5761966700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c0abe20 0x7f575c14eef0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f574c00ec10 tx=0x7f574c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:58:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.549+0000 7f57527fc700 1 -- 172.21.15.7:0/3881228524 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c00c910 con 0x7f575c0abe20 2023-03-03T16:58:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.549+0000 7f57527fc700 1 -- 172.21.15.7:0/3881228524 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c00ca70 con 0x7f575c0abe20 2023-03-03T16:58:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.549+0000 7f5763bca700 1 -- 172.21.15.7:0/3881228524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57580094a0 con 0x7f575c0abe20 2023-03-03T16:58:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.549+0000 7f57527fc700 1 -- 172.21.15.7:0/3881228524 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c00cde0 con 0x7f575c0abe20 2023-03-03T16:58:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.549+0000 7f5763bca700 1 -- 172.21.15.7:0/3881228524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c0ac350 con 0x7f575c0abe20 2023-03-03T16:58:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.550+0000 7f5763bca700 1 -- 172.21.15.7:0/3881228524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c0ab630 con 0x7f575c0abe20 2023-03-03T16:58:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.550+0000 7f57527fc700 1 -- 172.21.15.7:0/3881228524 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f574c019030 con 0x7f575c0abe20 2023-03-03T16:58:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.550+0000 7f57527fc700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57400682f0 0x7f574006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.551+0000 7f57527fc700 1 -- 172.21.15.7:0/3881228524 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f574c086150 con 0x7f575c0abe20 2023-03-03T16:58:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.551+0000 7f5761165700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57400682f0 0x7f574006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.551+0000 7f5761165700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57400682f0 0x7f574006a7a0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f575c1501d0 tx=0x7f5758008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:58:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.555+0000 7f57527fc700 1 -- 172.21.15.7:0/3881228524 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f574c0524f0 con 0x7f575c0abe20 2023-03-03T16:58:02.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.792+0000 7f5763bca700 1 -- 172.21.15.7:0/3881228524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f575c0c3e10 con 0x7f575c0abe20 2023-03-03T16:58:02.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.792+0000 7f57527fc700 1 -- 172.21.15.7:0/3881228524 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f574c017090 con 0x7f575c0abe20 2023-03-03T16:58:02.792 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:58:02.793 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:58:02.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 -- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57400682f0 msgr2=0x7f574006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:02.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57400682f0 0x7f574006a7a0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f575c1501d0 tx=0x7f5758008030 comp rx=0 tx=0).stop 2023-03-03T16:58:02.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 -- 172.21.15.7:0/3881228524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c0abe20 msgr2=0x7f575c14eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c0abe20 0x7f575c14eef0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f574c00ec10 tx=0x7f574c00ef20 comp rx=0 tx=0).stop 2023-03-03T16:58:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 -- 172.21.15.7:0/3881228524 shutdown_connections 2023-03-03T16:58:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c14fda0 0x7f575c1dbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f57400682f0 0x7f574006a7a0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c0abe20 0x7f575c14eef0 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 --2- 172.21.15.7:0/3881228524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f575c14f9d0 0x7f575c1549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 -- 172.21.15.7:0/3881228524 >> 172.21.15.7:0/3881228524 conn(0x7f575c0c8900 msgr2=0x7f575c0c7090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:58:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 -- 172.21.15.7:0/3881228524 shutdown_connections 2023-03-03T16:58:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:02.794+0000 7f5763bca700 1 -- 172.21.15.7:0/3881228524 wait complete. 2023-03-03T16:58:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:58:02.806 INFO:tasks.mds_thrash.fs.[b]:waiting for 45 secs before thrashing 2023-03-03T16:58:21.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:21.390 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:47.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:58:47.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.944+0000 7f549a547700 1 Processor -- start 2023-03-03T16:58:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.944+0000 7f549a547700 1 -- start start 2023-03-03T16:58:47.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.945+0000 7f549a547700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f549414fda0 0x7f5494156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.945+0000 7f549a547700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54940abe20 0x7f54941529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.945+0000 7f549a547700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54940c2f30 con 0x7f549414f9d0 2023-03-03T16:58:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.945+0000 7f549a547700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54940c31d0 con 0x7f54940abe20 2023-03-03T16:58:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.945+0000 7f549a547700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54941524a0 con 0x7f549414fda0 2023-03-03T16:58:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.945+0000 7f5498ae4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54940abe20 0x7f54941529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.945+0000 7f5498ae4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54940abe20 0x7f54941529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59402/0 (socket says 172.21.15.7:59402) 2023-03-03T16:58:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.945+0000 7f5498ae4700 1 -- 172.21.15.7:0/4133445701 learned_addr learned my addr 172.21.15.7:0/4133445701 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:58:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.946+0000 7f5498ae4700 1 -- 172.21.15.7:0/4133445701 >> v2:172.21.15.143:3300/0 conn(0x7f549414fda0 msgr2=0x7f5494156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:58:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.946+0000 7f54937fe700 1 --2- 172.21.15.7:0/4133445701 >> v2:172.21.15.143:3300/0 conn(0x7f549414fda0 0x7f5494156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.946+0000 7f5498ae4700 1 --2- 172.21.15.7:0/4133445701 >> v2:172.21.15.143:3300/0 conn(0x7f549414fda0 0x7f5494156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.946+0000 7f5498ae4700 1 -- 172.21.15.7:0/4133445701 >> v1:172.21.15.143:6789/0 conn(0x7f549414f9d0 legacy=0x7f5494071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:58:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.946+0000 7f5498ae4700 1 -- 172.21.15.7:0/4133445701 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54940c34b0 con 0x7f54940abe20 2023-03-03T16:58:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.946+0000 7f54937fe700 1 --2- 172.21.15.7:0/4133445701 >> v2:172.21.15.143:3300/0 conn(0x7f549414fda0 0x7f5494156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T16:58:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.946+0000 7f5498ae4700 1 --2- 172.21.15.7:0/4133445701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54940abe20 0x7f54941529c0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f54940709b0 tx=0x7f548c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae097cf729ddd5c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:58:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.946+0000 7f5492ffd700 1 -- 172.21.15.7:0/4133445701 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c00e070 con 0x7f54940abe20 2023-03-03T16:58:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.946+0000 7f5492ffd700 1 -- 172.21.15.7:0/4133445701 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c003a20 con 0x7f54940abe20 2023-03-03T16:58:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.946+0000 7f5492ffd700 1 -- 172.21.15.7:0/4133445701 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c003dd0 con 0x7f54940abe20 2023-03-03T16:58:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.947+0000 7f549a547700 1 -- 172.21.15.7:0/4133445701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54940abe20 msgr2=0x7f54941529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.947+0000 7f549a547700 1 --2- 172.21.15.7:0/4133445701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54940abe20 0x7f54941529c0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f54940709b0 tx=0x7f548c00b330 comp rx=0 tx=0).stop 2023-03-03T16:58:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.947+0000 7f549a547700 1 -- 172.21.15.7:0/4133445701 shutdown_connections 2023-03-03T16:58:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.947+0000 7f549a547700 1 --2- 172.21.15.7:0/4133445701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54940abe20 0x7f54941529c0 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.947+0000 7f549a547700 1 --2- 172.21.15.7:0/4133445701 >> v2:172.21.15.143:3300/0 conn(0x7f549414fda0 0x7f5494156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.947+0000 7f549a547700 1 -- 172.21.15.7:0/4133445701 >> 172.21.15.7:0/4133445701 conn(0x7f54940c8900 msgr2=0x7f54940c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:58:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.947+0000 7f549a547700 1 -- 172.21.15.7:0/4133445701 shutdown_connections 2023-03-03T16:58:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.947+0000 7f549a547700 1 -- 172.21.15.7:0/4133445701 wait complete. 2023-03-03T16:58:47.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.948+0000 7f549a547700 1 Processor -- start 2023-03-03T16:58:47.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.948+0000 7f549a547700 1 -- start start 2023-03-03T16:58:47.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f549a547700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54940abe20 0x7f54941ea660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:47.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f549a547700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f549414f9d0 0x7f549414eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:47.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f549a547700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f549414fda0 0x7f54941528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:47.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f549a547700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54941de450 con 0x7f549414fda0 2023-03-03T16:58:47.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f549a547700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54941de120 con 0x7f54940abe20 2023-03-03T16:58:47.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f549a547700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549414f0e0 con 0x7f549414f9d0 2023-03-03T16:58:47.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5498ae4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f549414fda0 0x7f54941528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:47.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5498ae4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f549414fda0 0x7f54941528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59408/0 (socket says 172.21.15.7:59408) 2023-03-03T16:58:47.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5498ae4700 1 -- 172.21.15.7:0/790534763 learned_addr learned my addr 172.21.15.7:0/790534763 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:58:47.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5493fff700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54940abe20 0x7f54941ea660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:47.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f54937fe700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f549414f9d0 0x7f549414eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:47.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5498ae4700 1 -- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f549414f9d0 msgr2=0x7f549414eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:47.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5498ae4700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f549414f9d0 0x7f549414eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:47.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5498ae4700 1 -- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54940abe20 msgr2=0x7f54941ea660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:47.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5498ae4700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54940abe20 0x7f54941ea660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:47.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5498ae4700 1 -- 172.21.15.7:0/790534763 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c009b30 con 0x7f549414fda0 2023-03-03T16:58:47.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5498ae4700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f549414fda0 0x7f54941528c0 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f548c015040 tx=0x7f548c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:58:47.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.949+0000 7f5490ff9700 1 -- 172.21.15.7:0/790534763 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c004440 con 0x7f549414fda0 2023-03-03T16:58:47.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.950+0000 7f5490ff9700 1 -- 172.21.15.7:0/790534763 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c0045a0 con 0x7f549414fda0 2023-03-03T16:58:47.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.950+0000 7f549a547700 1 -- 172.21.15.7:0/790534763 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54940ac350 con 0x7f549414fda0 2023-03-03T16:58:47.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.950+0000 7f5490ff9700 1 -- 172.21.15.7:0/790534763 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c0092e0 con 0x7f549414fda0 2023-03-03T16:58:47.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.950+0000 7f549a547700 1 -- 172.21.15.7:0/790534763 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54940ab630 con 0x7f549414fda0 2023-03-03T16:58:47.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.950+0000 7f549a547700 1 -- 172.21.15.7:0/790534763 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5494071fa0 con 0x7f549414fda0 2023-03-03T16:58:47.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.951+0000 7f5490ff9700 1 -- 172.21.15.7:0/790534763 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f548c022030 con 0x7f549414fda0 2023-03-03T16:58:47.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.951+0000 7f5490ff9700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f54800682f0 0x7f548006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:58:47.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.952+0000 7f5490ff9700 1 -- 172.21.15.7:0/790534763 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f548c087520 con 0x7f549414fda0 2023-03-03T16:58:47.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.952+0000 7f5493fff700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f54800682f0 0x7f548006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:58:47.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.952+0000 7f5493fff700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f54800682f0 0x7f548006a7a0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f5494151ff0 tx=0x7f5484009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:58:47.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:47.956+0000 7f5490ff9700 1 -- 172.21.15.7:0/790534763 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f548c053840 con 0x7f549414fda0 2023-03-03T16:58:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.195+0000 7f549a547700 1 -- 172.21.15.7:0/790534763 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5494152040 con 0x7f549414fda0 2023-03-03T16:58:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.195+0000 7f5490ff9700 1 -- 172.21.15.7:0/790534763 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f548c007310 con 0x7f549414fda0 2023-03-03T16:58:48.197 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:58:48.197 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:58:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.197+0000 7f549a547700 1 -- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f54800682f0 msgr2=0x7f548006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.197+0000 7f549a547700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f54800682f0 0x7f548006a7a0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f5494151ff0 tx=0x7f5484009040 comp rx=0 tx=0).stop 2023-03-03T16:58:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.197+0000 7f549a547700 1 -- 172.21.15.7:0/790534763 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f549414fda0 msgr2=0x7f54941528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:58:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.197+0000 7f549a547700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f549414fda0 0x7f54941528c0 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f548c015040 tx=0x7f548c004ad0 comp rx=0 tx=0).stop 2023-03-03T16:58:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.197+0000 7f549a547700 1 -- 172.21.15.7:0/790534763 shutdown_connections 2023-03-03T16:58:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.197+0000 7f549a547700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f54800682f0 0x7f548006a7a0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.197+0000 7f549a547700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f549414fda0 0x7f54941528c0 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.197+0000 7f549a547700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54940abe20 0x7f54941ea660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.197+0000 7f549a547700 1 --2- 172.21.15.7:0/790534763 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f549414f9d0 0x7f549414eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:58:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.197+0000 7f549a547700 1 -- 172.21.15.7:0/790534763 >> 172.21.15.7:0/790534763 conn(0x7f54940c8900 msgr2=0x7f5494150200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:58:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.198+0000 7f549a547700 1 -- 172.21.15.7:0/790534763 shutdown_connections 2023-03-03T16:58:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:58:48.198+0000 7f549a547700 1 -- 172.21.15.7:0/790534763 wait complete. 2023-03-03T16:58:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:58:48.210 INFO:tasks.mds_thrash.fs.[b]:waiting for 21 secs before thrashing 2023-03-03T16:58:51.471 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:58:51.474 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:09.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:59:09.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.350+0000 7f2bfaa4d700 1 Processor -- start 2023-03-03T16:59:09.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.350+0000 7f2bfaa4d700 1 -- start start 2023-03-03T16:59:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.351+0000 7f2bfaa4d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2bf4152e30 0x7f2bf40c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.351+0000 7f2bfaa4d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf41523b0 con 0x7f2bf40c4910 2023-03-03T16:59:09.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.351+0000 7f2bfaa4d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf4072010 con 0x7f2bf40beab0 2023-03-03T16:59:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.351+0000 7f2bfaa4d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf40aaaf0 con 0x7f2bf4152e30 2023-03-03T16:59:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.351+0000 7f2bf8fea700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2bf40c4910 0x7f2bf415cd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53410/0 (socket says 172.21.15.7:53410) 2023-03-03T16:59:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.351+0000 7f2bf8fea700 1 -- 172.21.15.7:0/2928581354 learned_addr learned my addr 172.21.15.7:0/2928581354 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:59:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf3fff700 1 --2- 172.21.15.7:0/2928581354 >> v2:172.21.15.143:3300/0 conn(0x7f2bf4152e30 0x7f2bf40c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396771515 0 0) 0x7f2bf41523b0 con 0x7f2bf40c4910 2023-03-03T16:59:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be00035b0 con 0x7f2bf40c4910 2023-03-03T16:59:09.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2832017339 0 0) 0x7f2be00035b0 con 0x7f2bf40c4910 2023-03-03T16:59:09.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 >> v2:172.21.15.143:3300/0 conn(0x7f2bf4152e30 msgr2=0x7f2bf40c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:59:09.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 --2- 172.21.15.7:0/2928581354 >> v2:172.21.15.143:3300/0 conn(0x7f2bf4152e30 0x7f2bf40c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:09.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 >> v1:172.21.15.143:6789/0 conn(0x7f2bf40beab0 legacy=0x7f2bf4155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:09.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf40aadd0 con 0x7f2bf40c4910 2023-03-03T16:59:09.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f2bec0019f0 con 0x7f2bf40c4910 2023-03-03T16:59:09.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 >> v1:172.21.15.7:6789/0 conn(0x7f2bf40c4910 legacy=0x7f2bf415cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:09.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2be0003a70 0x7f2be0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:09.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2be0004850 0x7f2be0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2be000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf40aaaf0 con 0x7f2be0005250 2023-03-03T16:59:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be000c190 con 0x7f2be0003a70 2023-03-03T16:59:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be000c350 con 0x7f2be0004850 2023-03-03T16:59:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf8fea700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2be000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf3fff700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2be0003a70 0x7f2be0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:09.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.352+0000 7f2bf37fe700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2be0004850 0x7f2be0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.353+0000 7f2bf8fea700 1 -- 172.21.15.7:0/2928581354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2be0004850 msgr2=0x7f2be0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:59:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.353+0000 7f2bf8fea700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2be0004850 0x7f2be0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.353+0000 7f2bf8fea700 1 -- 172.21.15.7:0/2928581354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2be0003a70 msgr2=0x7f2be0003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:59:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.353+0000 7f2bf8fea700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2be0003a70 0x7f2be0003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.353+0000 7f2bf8fea700 1 -- 172.21.15.7:0/2928581354 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be00035b0 con 0x7f2be0005250 2023-03-03T16:59:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.353+0000 7f2bf8fea700 1 -- 172.21.15.7:0/2928581354 reap_dead start 2023-03-03T16:59:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.353+0000 7f2bf8fea700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2be000ba10 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00a7f0 tx=0x7f2be0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d930c4a8bcdb7bc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:59:09.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.353+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec00a2f0 con 0x7f2be0005250 2023-03-03T16:59:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.353+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bec012600 con 0x7f2be0005250 2023-03-03T16:59:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.353+0000 7f2bf2ffd700 1 -- 172.21.15.7:0/2928581354 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec0129b0 con 0x7f2be0005250 2023-03-03T16:59:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.354+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/2928581354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 msgr2=0x7f2be000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.354+0000 7f2bfaa4d700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2be000ba10 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00a7f0 tx=0x7f2be0005880 comp rx=0 tx=0).stop 2023-03-03T16:59:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.354+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/2928581354 shutdown_connections 2023-03-03T16:59:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.354+0000 7f2bfaa4d700 1 --2- 172.21.15.7:0/2928581354 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2be000ba10 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:09.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.354+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/2928581354 >> 172.21.15.7:0/2928581354 conn(0x7f2bf40736f0 msgr2=0x7f2bf4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:59:09.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.354+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/2928581354 shutdown_connections 2023-03-03T16:59:09.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.354+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/2928581354 wait complete. 2023-03-03T16:59:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.355+0000 7f2bfaa4d700 1 Processor -- start 2023-03-03T16:59:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.355+0000 7f2bfaa4d700 1 -- start start 2023-03-03T16:59:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bfaa4d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2bf4151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bfaa4d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf41517b0 0x7f2bf41ea580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bfaa4d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf41eded0 0x7f2bf41eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:09.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bfaa4d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf40ab6b0 con 0x7f2be0005250 2023-03-03T16:59:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bfaa4d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf41ea0f0 con 0x7f2bf41eded0 2023-03-03T16:59:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bfaa4d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf406bb90 con 0x7f2bf41517b0 2023-03-03T16:59:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2bf4151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2bf4151270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59438/0 (socket says 172.21.15.7:59438) 2023-03-03T16:59:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf3fff700 1 -- 172.21.15.7:0/1952209276 learned_addr learned my addr 172.21.15.7:0/1952209276 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:59:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf8fea700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf41eded0 0x7f2bf41eaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:09.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf37fe700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf41517b0 0x7f2bf41ea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf3fff700 1 -- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf41517b0 msgr2=0x7f2bf41ea580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf3fff700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf41517b0 0x7f2bf41ea580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf3fff700 1 -- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf41eded0 msgr2=0x7f2bf41eaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf3fff700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf41eded0 0x7f2bf41eaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf3fff700 1 -- 172.21.15.7:0/1952209276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf4153200 con 0x7f2be0005250 2023-03-03T16:59:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf3fff700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2bf4151270 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f2be40119a0 tx=0x7f2be4011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:59:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.356+0000 7f2bf0ff9700 1 -- 172.21.15.7:0/1952209276 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be400f940 con 0x7f2be0005250 2023-03-03T16:59:09.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.357+0000 7f2bf0ff9700 1 -- 172.21.15.7:0/1952209276 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be400faa0 con 0x7f2be0005250 2023-03-03T16:59:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.357+0000 7f2bf0ff9700 1 -- 172.21.15.7:0/1952209276 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be400fe30 con 0x7f2be0005250 2023-03-03T16:59:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.357+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/1952209276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf4071c70 con 0x7f2be0005250 2023-03-03T16:59:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.357+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/1952209276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf4153390 con 0x7f2be0005250 2023-03-03T16:59:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.358+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/1952209276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf40c5bf0 con 0x7f2be0005250 2023-03-03T16:59:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.358+0000 7f2bf0ff9700 1 -- 172.21.15.7:0/1952209276 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2be4014440 con 0x7f2be0005250 2023-03-03T16:59:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.358+0000 7f2bf0ff9700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 0x7f2bd406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.359+0000 7f2bf0ff9700 1 -- 172.21.15.7:0/1952209276 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2be4086a50 con 0x7f2be0005250 2023-03-03T16:59:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.359+0000 7f2bf37fe700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 0x7f2bd406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.359+0000 7f2bf37fe700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 0x7f2bd406a7a0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4152270 tx=0x7f2bdc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:59:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.363+0000 7f2bf0ff9700 1 -- 172.21.15.7:0/1952209276 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2be4052ca0 con 0x7f2be0005250 2023-03-03T16:59:09.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.601+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/1952209276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bf4074bb0 con 0x7f2be0005250 2023-03-03T16:59:09.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.602+0000 7f2bf0ff9700 1 -- 172.21.15.7:0/1952209276 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2be40562c0 con 0x7f2be0005250 2023-03-03T16:59:09.602 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:59:09.602 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:59:09.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 msgr2=0x7f2bd406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:09.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 0x7f2bd406a7a0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4152270 tx=0x7f2bdc009040 comp rx=0 tx=0).stop 2023-03-03T16:59:09.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/1952209276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 msgr2=0x7f2bf4151270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:09.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2bf4151270 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f2be40119a0 tx=0x7f2be4011d60 comp rx=0 tx=0).stop 2023-03-03T16:59:09.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/1952209276 shutdown_connections 2023-03-03T16:59:09.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf41eded0 0x7f2bf41eaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:09.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 0x7f2bd406a7a0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:09.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2be0005250 0x7f2bf4151270 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:09.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 --2- 172.21.15.7:0/1952209276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf41517b0 0x7f2bf41ea580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:09.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/1952209276 >> 172.21.15.7:0/1952209276 conn(0x7f2bf40736f0 msgr2=0x7f2bf40be2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:59:09.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/1952209276 shutdown_connections 2023-03-03T16:59:09.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:09.604+0000 7f2bfaa4d700 1 -- 172.21.15.7:0/1952209276 wait complete. 2023-03-03T16:59:09.606 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:59:09.616 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2023-03-03T16:59:21.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:21.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:42.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T16:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.491+0000 7f010149b700 1 Processor -- start 2023-03-03T16:59:42.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.491+0000 7f010149b700 1 -- start start 2023-03-03T16:59:42.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.491+0000 7f010149b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00fc151cc0 0x7f00fc0ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:42.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.491+0000 7f010149b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f00fc0cac10 0x7f00fc0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:42.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.491+0000 7f010149b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc0c7ca0 con 0x7f00fc0b00b0 2023-03-03T16:59:42.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.491+0000 7f010149b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc152090 con 0x7f00fc151cc0 2023-03-03T16:59:42.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.491+0000 7f010149b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc0c6cf0 con 0x7f00fc0cac10 2023-03-03T16:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00faffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00fc151cc0 0x7f00fc0ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00fa7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f00fc0b00b0 0x7f00fc0c7050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53448/0 (socket says 172.21.15.7:53448) 2023-03-03T16:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00fa7fc700 1 -- 172.21.15.7:0/2934201117 learned_addr learned my addr 172.21.15.7:0/2934201117 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T16:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00fb7fe700 1 --2- 172.21.15.7:0/2934201117 >> v2:172.21.15.143:3300/0 conn(0x7f00fc0cac10 0x7f00fc0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00faffd700 1 -- 172.21.15.7:0/2934201117 >> v2:172.21.15.143:3300/0 conn(0x7f00fc0cac10 msgr2=0x7f00fc0c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T16:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00faffd700 1 --2- 172.21.15.7:0/2934201117 >> v2:172.21.15.143:3300/0 conn(0x7f00fc0cac10 0x7f00fc0c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00faffd700 1 -- 172.21.15.7:0/2934201117 >> v1:172.21.15.7:6789/0 conn(0x7f00fc0b00b0 legacy=0x7f00fc0c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00f9ffb700 1 -- 172.21.15.7:0/2934201117 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275646706 0 0) 0x7f00fc0c7ca0 con 0x7f00fc0b00b0 2023-03-03T16:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00faffd700 1 -- 172.21.15.7:0/2934201117 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc1548f0 con 0x7f00fc151cc0 2023-03-03T16:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00faffd700 1 --2- 172.21.15.7:0/2934201117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc151cc0 0x7f00fc0ca6d0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0c1700 tx=0x7f00ec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6880f1e63179b8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00f9ffb700 1 -- 172.21.15.7:0/2934201117 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec011830 con 0x7f00fc151cc0 2023-03-03T16:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00f9ffb700 1 -- 172.21.15.7:0/2934201117 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec011990 con 0x7f00fc151cc0 2023-03-03T16:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.492+0000 7f00f9ffb700 1 -- 172.21.15.7:0/2934201117 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec011d20 con 0x7f00fc151cc0 2023-03-03T16:59:42.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.493+0000 7f010149b700 1 -- 172.21.15.7:0/2934201117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc151cc0 msgr2=0x7f00fc0ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.493+0000 7f010149b700 1 --2- 172.21.15.7:0/2934201117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc151cc0 0x7f00fc0ca6d0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0c1700 tx=0x7f00ec009b20 comp rx=0 tx=0).stop 2023-03-03T16:59:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.493+0000 7f010149b700 1 -- 172.21.15.7:0/2934201117 shutdown_connections 2023-03-03T16:59:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.493+0000 7f010149b700 1 --2- 172.21.15.7:0/2934201117 >> v2:172.21.15.143:3300/0 conn(0x7f00fc0cac10 0x7f00fc0c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.493+0000 7f010149b700 1 --2- 172.21.15.7:0/2934201117 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc151cc0 0x7f00fc0ca6d0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.493+0000 7f010149b700 1 -- 172.21.15.7:0/2934201117 >> 172.21.15.7:0/2934201117 conn(0x7f00fc06b980 msgr2=0x7f00fc071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:59:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.493+0000 7f010149b700 1 -- 172.21.15.7:0/2934201117 shutdown_connections 2023-03-03T16:59:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.493+0000 7f010149b700 1 -- 172.21.15.7:0/2934201117 wait complete. 2023-03-03T16:59:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.494+0000 7f010149b700 1 Processor -- start 2023-03-03T16:59:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.494+0000 7f010149b700 1 -- start start 2023-03-03T16:59:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f010149b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00fc0b00b0 0x7f00fc150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f010149b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc0cac10 0x7f00fc157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f010149b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00fc151cc0 0x7f00fc0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f010149b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0c44b0 con 0x7f00fc0cac10 2023-03-03T16:59:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f010149b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0c8540 con 0x7f00fc0b00b0 2023-03-03T16:59:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f010149b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0c8de0 con 0x7f00fc151cc0 2023-03-03T16:59:42.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f00fa7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc0cac10 0x7f00fc157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:42.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f00fa7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc0cac10 0x7f00fc157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59474/0 (socket says 172.21.15.7:59474) 2023-03-03T16:59:42.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f00fa7fc700 1 -- 172.21.15.7:0/1755217578 learned_addr learned my addr 172.21.15.7:0/1755217578 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T16:59:42.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f00faffd700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00fc0b00b0 0x7f00fc150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:42.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f00fb7fe700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00fc151cc0 0x7f00fc0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:42.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f00fa7fc700 1 -- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00fc151cc0 msgr2=0x7f00fc0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:42.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f00fa7fc700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00fc151cc0 0x7f00fc0c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f00fa7fc700 1 -- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00fc0b00b0 msgr2=0x7f00fc150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f00fa7fc700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00fc0b00b0 0x7f00fc150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.495+0000 7f00fa7fc700 1 -- 172.21.15.7:0/1755217578 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec0094a0 con 0x7f00fc0cac10 2023-03-03T16:59:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.496+0000 7f00fa7fc700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc0cac10 0x7f00fc157690 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f00e400c570 tx=0x7f00e400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:59:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.496+0000 7f00f37fe700 1 -- 172.21.15.7:0/1755217578 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e4004c80 con 0x7f00fc0cac10 2023-03-03T16:59:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.496+0000 7f00f37fe700 1 -- 172.21.15.7:0/1755217578 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e4004de0 con 0x7f00fc0cac10 2023-03-03T16:59:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.496+0000 7f010149b700 1 -- 172.21.15.7:0/1755217578 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc154eb0 con 0x7f00fc0cac10 2023-03-03T16:59:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.496+0000 7f00f37fe700 1 -- 172.21.15.7:0/1755217578 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e4007310 con 0x7f00fc0cac10 2023-03-03T16:59:42.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.496+0000 7f010149b700 1 -- 172.21.15.7:0/1755217578 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc154bd0 con 0x7f00fc0cac10 2023-03-03T16:59:42.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.497+0000 7f010149b700 1 -- 172.21.15.7:0/1755217578 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc155190 con 0x7f00fc0cac10 2023-03-03T16:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.497+0000 7f00f37fe700 1 -- 172.21.15.7:0/1755217578 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f00e401a020 con 0x7f00fc0cac10 2023-03-03T16:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.498+0000 7f00f37fe700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00dc0682f0 0x7f00dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T16:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.498+0000 7f00f37fe700 1 -- 172.21.15.7:0/1755217578 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f00e4085c20 con 0x7f00fc0cac10 2023-03-03T16:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.498+0000 7f00faffd700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00dc0682f0 0x7f00dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T16:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.498+0000 7f00faffd700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00dc0682f0 0x7f00dc06a7a0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0090b0 tx=0x7f00ec01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T16:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.502+0000 7f00f37fe700 1 -- 172.21.15.7:0/1755217578 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f00e4052080 con 0x7f00fc0cac10 2023-03-03T16:59:42.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.739+0000 7f010149b700 1 -- 172.21.15.7:0/1755217578 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00fc074bb0 con 0x7f00fc0cac10 2023-03-03T16:59:42.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.740+0000 7f00f37fe700 1 -- 172.21.15.7:0/1755217578 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f00e40556a0 con 0x7f00fc0cac10 2023-03-03T16:59:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T16:59:42.740 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T16:59:42.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 -- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00dc0682f0 msgr2=0x7f00dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00dc0682f0 0x7f00dc06a7a0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0090b0 tx=0x7f00ec01c040 comp rx=0 tx=0).stop 2023-03-03T16:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 -- 172.21.15.7:0/1755217578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc0cac10 msgr2=0x7f00fc157690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T16:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc0cac10 0x7f00fc157690 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f00e400c570 tx=0x7f00e400c880 comp rx=0 tx=0).stop 2023-03-03T16:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 -- 172.21.15.7:0/1755217578 shutdown_connections 2023-03-03T16:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00fc151cc0 0x7f00fc0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00fc0b00b0 0x7f00fc150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:42.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00dc0682f0 0x7f00dc06a7a0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:42.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 --2- 172.21.15.7:0/1755217578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00fc0cac10 0x7f00fc157690 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T16:59:42.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 -- 172.21.15.7:0/1755217578 >> 172.21.15.7:0/1755217578 conn(0x7f00fc06b980 msgr2=0x7f00fc14a560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T16:59:42.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 -- 172.21.15.7:0/1755217578 shutdown_connections 2023-03-03T16:59:42.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T16:59:42.742+0000 7f010149b700 1 -- 172.21.15.7:0/1755217578 wait complete. 2023-03-03T16:59:42.743 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T16:59:42.753 INFO:tasks.mds_thrash.fs.[a]:waiting for 53 secs before thrashing 2023-03-03T16:59:51.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:51.735 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T16:59:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T16:59:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T17:00:21.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:21.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:35.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:00:35.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:00:35.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.757+0000 7fc1c5e15700 1 Processor -- start 2023-03-03T17:00:35.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.757+0000 7fc1c5e15700 1 -- start start 2023-03-03T17:00:35.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.757+0000 7fc1c5e15700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc1c014f9d0 0x7fc1c0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.757+0000 7fc1c5e15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 0x7fc1c01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.757+0000 7fc1c5e15700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c00c2f30 con 0x7fc1c014fda0 2023-03-03T17:00:35.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.757+0000 7fc1c5e15700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c00c31d0 con 0x7fc1c014f9d0 2023-03-03T17:00:35.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.757+0000 7fc1c5e15700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c01524a0 con 0x7fc1c00abe20 2023-03-03T17:00:35.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1beffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc1c014fda0 0x7fc1c0071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53502/0 (socket says 172.21.15.7:53502) 2023-03-03T17:00:35.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1beffd700 1 -- 172.21.15.7:0/3055270098 learned_addr learned my addr 172.21.15.7:0/3055270098 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:00:35.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1bffff700 1 --2- 172.21.15.7:0/3055270098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 0x7fc1c01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1bf7fe700 1 --2- 172.21.15.7:0/3055270098 >> v2:172.21.15.143:3300/0 conn(0x7fc1c014f9d0 0x7fc1c0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1be7fc700 1 -- 172.21.15.7:0/3055270098 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403558967 0 0) 0x7fc1c00c2f30 con 0x7fc1c014fda0 2023-03-03T17:00:35.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1be7fc700 1 -- 172.21.15.7:0/3055270098 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1ac0035b0 con 0x7fc1c014fda0 2023-03-03T17:00:35.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1bffff700 1 -- 172.21.15.7:0/3055270098 >> v2:172.21.15.143:3300/0 conn(0x7fc1c014f9d0 msgr2=0x7fc1c0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:00:35.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1bffff700 1 --2- 172.21.15.7:0/3055270098 >> v2:172.21.15.143:3300/0 conn(0x7fc1c014f9d0 0x7fc1c0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:35.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1bffff700 1 -- 172.21.15.7:0/3055270098 >> v1:172.21.15.7:6789/0 conn(0x7fc1c014fda0 legacy=0x7fc1c0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:35.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1bffff700 1 -- 172.21.15.7:0/3055270098 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c00c34b0 con 0x7fc1c00abe20 2023-03-03T17:00:35.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1bf7fe700 1 --2- 172.21.15.7:0/3055270098 >> v2:172.21.15.143:3300/0 conn(0x7fc1c014f9d0 0x7fc1c0156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:00:35.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.758+0000 7fc1bffff700 1 --2- 172.21.15.7:0/3055270098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 0x7fc1c01529c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009840 tx=0x7fc1b8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1670e5a34225646f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:00:35.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1be7fc700 1 -- 172.21.15.7:0/3055270098 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8011830 con 0x7fc1c00abe20 2023-03-03T17:00:35.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1be7fc700 1 -- 172.21.15.7:0/3055270098 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b8011990 con 0x7fc1c00abe20 2023-03-03T17:00:35.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1be7fc700 1 -- 172.21.15.7:0/3055270098 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8011d20 con 0x7fc1c00abe20 2023-03-03T17:00:35.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1c5e15700 1 -- 172.21.15.7:0/3055270098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 msgr2=0x7fc1c01529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:35.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1c5e15700 1 --2- 172.21.15.7:0/3055270098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 0x7fc1c01529c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009840 tx=0x7fc1b8009b50 comp rx=0 tx=0).stop 2023-03-03T17:00:35.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1c5e15700 1 -- 172.21.15.7:0/3055270098 shutdown_connections 2023-03-03T17:00:35.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1c5e15700 1 --2- 172.21.15.7:0/3055270098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 0x7fc1c01529c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:35.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1c5e15700 1 --2- 172.21.15.7:0/3055270098 >> v2:172.21.15.143:3300/0 conn(0x7fc1c014f9d0 0x7fc1c0156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:35.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1c5e15700 1 -- 172.21.15.7:0/3055270098 >> 172.21.15.7:0/3055270098 conn(0x7fc1c00c8900 msgr2=0x7fc1c00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:00:35.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1c5e15700 1 -- 172.21.15.7:0/3055270098 shutdown_connections 2023-03-03T17:00:35.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.759+0000 7fc1c5e15700 1 -- 172.21.15.7:0/3055270098 wait complete. 2023-03-03T17:00:35.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.761+0000 7fc1c5e15700 1 Processor -- start 2023-03-03T17:00:35.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.761+0000 7fc1c5e15700 1 -- start start 2023-03-03T17:00:35.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.761+0000 7fc1c5e15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 0x7fc1c01e8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.761+0000 7fc1c5e15700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c014f9d0 0x7fc1c00c08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.761+0000 7fc1c5e15700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c014fda0 0x7fc1c006e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1c5e15700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0140ca0 con 0x7fc1c014fda0 2023-03-03T17:00:35.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1c5e15700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00c7630 con 0x7fc1c014f9d0 2023-03-03T17:00:35.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1c5e15700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c01e8570 con 0x7fc1c00abe20 2023-03-03T17:00:35.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1bffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c014fda0 0x7fc1c006e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1bffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c014fda0 0x7fc1c006e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59530/0 (socket says 172.21.15.7:59530) 2023-03-03T17:00:35.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1bffff700 1 -- 172.21.15.7:0/1031827670 learned_addr learned my addr 172.21.15.7:0/1031827670 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:00:35.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1beffd700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c014f9d0 0x7fc1c00c08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1bf7fe700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 0x7fc1c01e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1bffff700 1 -- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 msgr2=0x7fc1c01e8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:35.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1bffff700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 0x7fc1c01e8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:35.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1bffff700 1 -- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c014f9d0 msgr2=0x7fc1c00c08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:35.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1bffff700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c014f9d0 0x7fc1c00c08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:35.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1bffff700 1 -- 172.21.15.7:0/1031827670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b0009a70 con 0x7fc1c014fda0 2023-03-03T17:00:35.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1bffff700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c014fda0 0x7fc1c006e690 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009b90 tx=0x7fc1b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:00:35.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1b7fff700 1 -- 172.21.15.7:0/1031827670 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80036a0 con 0x7fc1c014fda0 2023-03-03T17:00:35.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1b7fff700 1 -- 172.21.15.7:0/1031827670 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b801e070 con 0x7fc1c014fda0 2023-03-03T17:00:35.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1c5e15700 1 -- 172.21.15.7:0/1031827670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b80094a0 con 0x7fc1c014fda0 2023-03-03T17:00:35.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.762+0000 7fc1b7fff700 1 -- 172.21.15.7:0/1031827670 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8003a60 con 0x7fc1c014fda0 2023-03-03T17:00:35.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.763+0000 7fc1c5e15700 1 -- 172.21.15.7:0/1031827670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c00ac350 con 0x7fc1c014fda0 2023-03-03T17:00:35.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.763+0000 7fc1c5e15700 1 -- 172.21.15.7:0/1031827670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c00ab630 con 0x7fc1c014fda0 2023-03-03T17:00:35.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.764+0000 7fc1b7fff700 1 -- 172.21.15.7:0/1031827670 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc1b800f050 con 0x7fc1c014fda0 2023-03-03T17:00:35.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.764+0000 7fc1b7fff700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 0x7fc1a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.764+0000 7fc1b7fff700 1 -- 172.21.15.7:0/1031827670 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc1b8087000 con 0x7fc1c014fda0 2023-03-03T17:00:35.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.765+0000 7fc1bf7fe700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 0x7fc1a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.765+0000 7fc1bf7fe700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 0x7fc1a006a7a0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0007600 tx=0x7fc1b000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:00:35.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.770+0000 7fc1b7fff700 1 -- 172.21.15.7:0/1031827670 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc1b80533a0 con 0x7fc1c014fda0 2023-03-03T17:00:35.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.883+0000 7f03da087700 1 Processor -- start 2023-03-03T17:00:35.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.883+0000 7f03da087700 1 -- start start 2023-03-03T17:00:35.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.884+0000 7f03da087700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 0x7f03d40c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.884+0000 7f03da087700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f03d4155410 0x7f03d40bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.884+0000 7f03da087700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d40c08d0 con 0x7f03d40c6fe0 2023-03-03T17:00:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.884+0000 7f03da087700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d4152250 con 0x7f03d4155410 2023-03-03T17:00:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.884+0000 7f03da087700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40c0fb0 con 0x7f03d4151e80 2023-03-03T17:00:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.884+0000 7f03d37fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 0x7f03d40c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.884+0000 7f03caffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f03d4155410 0x7f03d40bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03d3fff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f03d40c6fe0 0x7f03d40bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:57038/0 (socket says 172.21.15.7:57038) 2023-03-03T17:00:35.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03caffd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f03d4155410 0x7f03d40bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:55310/0 (socket says 172.21.15.7:55310) 2023-03-03T17:00:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03d3fff700 1 -- 172.21.15.7:0/1932406505 learned_addr learned my addr 172.21.15.7:0/1932406505 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:00:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03caffd700 1 -- 172.21.15.7:0/1932406505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 msgr2=0x7f03d40c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:00:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03caffd700 1 --2- 172.21.15.7:0/1932406505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 0x7f03d40c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03caffd700 1 -- 172.21.15.7:0/1932406505 >> v1:172.21.15.143:6789/0 conn(0x7f03d40c6fe0 legacy=0x7f03d40bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03caffd700 1 -- 172.21.15.7:0/1932406505 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d4154930 con 0x7f03d4155410 2023-03-03T17:00:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03d2ffd700 1 -- 172.21.15.7:0/1932406505 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776799155 0 0) 0x7f03d40c08d0 con 0x7f03d40c6fe0 2023-03-03T17:00:35.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03d37fe700 1 --2- 172.21.15.7:0/1932406505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 0x7f03d40c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:00:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03caffd700 1 --2- 172.21.15.7:0/1932406505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03d4155410 0x7f03d40bf8e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f03bc009870 tx=0x7f03bc009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f522a89e18516541 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:00:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03d2ffd700 1 -- 172.21.15.7:0/1932406505 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc011830 con 0x7f03d4155410 2023-03-03T17:00:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03d2ffd700 1 -- 172.21.15.7:0/1932406505 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc011990 con 0x7f03d4155410 2023-03-03T17:00:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.885+0000 7f03d2ffd700 1 -- 172.21.15.7:0/1932406505 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc011d20 con 0x7f03d4155410 2023-03-03T17:00:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.886+0000 7f03da087700 1 -- 172.21.15.7:0/1932406505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03d4155410 msgr2=0x7f03d40bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.886+0000 7f03da087700 1 --2- 172.21.15.7:0/1932406505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03d4155410 0x7f03d40bf8e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f03bc009870 tx=0x7f03bc009b80 comp rx=0 tx=0).stop 2023-03-03T17:00:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.886+0000 7f03da087700 1 -- 172.21.15.7:0/1932406505 shutdown_connections 2023-03-03T17:00:35.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.886+0000 7f03da087700 1 --2- 172.21.15.7:0/1932406505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 0x7f03d40c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.886+0000 7f03da087700 1 --2- 172.21.15.7:0/1932406505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03d4155410 0x7f03d40bf8e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.886+0000 7f03da087700 1 -- 172.21.15.7:0/1932406505 >> 172.21.15.7:0/1932406505 conn(0x7f03d4071630 msgr2=0x7f03d406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:00:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.886+0000 7f03da087700 1 -- 172.21.15.7:0/1932406505 shutdown_connections 2023-03-03T17:00:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.886+0000 7f03da087700 1 -- 172.21.15.7:0/1932406505 wait complete. 2023-03-03T17:00:35.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.887+0000 7f03da087700 1 Processor -- start 2023-03-03T17:00:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.888+0000 7f03da087700 1 -- start start 2023-03-03T17:00:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.888+0000 7f03da087700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03d40c6fe0 0x7f03d41e04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03da087700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 0x7f03d4157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03da087700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03d4155410 0x7f03d40c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03da087700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d41df0d0 con 0x7f03d4155410 2023-03-03T17:00:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03da087700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d41dec90 con 0x7f03d40c6fe0 2023-03-03T17:00:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03da087700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d41eb3d0 con 0x7f03d4151e80 2023-03-03T17:00:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03d3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03d4155410 0x7f03d40c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03d3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03d4155410 0x7f03d40c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59544/0 (socket says 172.21.15.7:59544) 2023-03-03T17:00:35.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03d3fff700 1 -- 172.21.15.7:0/3195935107 learned_addr learned my addr 172.21.15.7:0/3195935107 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:00:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03caffd700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 0x7f03d4157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03d37fe700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03d40c6fe0 0x7f03d41e04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03d3fff700 1 -- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 msgr2=0x7f03d4157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03d3fff700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 0x7f03d4157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03d3fff700 1 -- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03d40c6fe0 msgr2=0x7f03d41e04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03d3fff700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03d40c6fe0 0x7f03d41e04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03d3fff700 1 -- 172.21.15.7:0/3195935107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c40094a0 con 0x7f03d4155410 2023-03-03T17:00:35.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.889+0000 7f03d3fff700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03d4155410 0x7f03d40c8860 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00cba0 tx=0x7f03cc00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:00:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.890+0000 7f03d0ff9700 1 -- 172.21.15.7:0/3195935107 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc00aaa0 con 0x7f03d4155410 2023-03-03T17:00:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.890+0000 7f03d0ff9700 1 -- 172.21.15.7:0/3195935107 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc00ac20 con 0x7f03d4155410 2023-03-03T17:00:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.890+0000 7f03da087700 1 -- 172.21.15.7:0/3195935107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03bc0094a0 con 0x7f03d4155410 2023-03-03T17:00:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.890+0000 7f03d0ff9700 1 -- 172.21.15.7:0/3195935107 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0070d0 con 0x7f03d4155410 2023-03-03T17:00:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.890+0000 7f03da087700 1 -- 172.21.15.7:0/3195935107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d4154ef0 con 0x7f03d4155410 2023-03-03T17:00:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.890+0000 7f03da087700 1 -- 172.21.15.7:0/3195935107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d4154c10 con 0x7f03d4155410 2023-03-03T17:00:35.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.891+0000 7f03d0ff9700 1 -- 172.21.15.7:0/3195935107 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f03cc01a030 con 0x7f03d4155410 2023-03-03T17:00:35.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.892+0000 7f03d0ff9700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03b40682f0 0x7f03b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:00:35.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.892+0000 7f03d0ff9700 1 -- 172.21.15.7:0/3195935107 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f03cc085d50 con 0x7f03d4155410 2023-03-03T17:00:35.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.892+0000 7f03d37fe700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03b40682f0 0x7f03b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:00:35.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.892+0000 7f03d37fe700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03b40682f0 0x7f03b406a7a0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009cd0 tx=0x7f03c4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:00:35.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:35.896+0000 7f03d0ff9700 1 -- 172.21.15.7:0/3195935107 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f03cc052090 con 0x7f03d4155410 2023-03-03T17:00:36.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.013+0000 7fc1c5e15700 1 -- 172.21.15.7:0/1031827670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1c0071fa0 con 0x7fc1c014fda0 2023-03-03T17:00:36.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.013+0000 7fc1b7fff700 1 -- 172.21.15.7:0/1031827670 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc1b8020090 con 0x7fc1c014fda0 2023-03-03T17:00:36.013 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:00:36.014 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:00:36.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 -- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 msgr2=0x7fc1a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:36.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 0x7fc1a006a7a0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0007600 tx=0x7fc1b000b040 comp rx=0 tx=0).stop 2023-03-03T17:00:36.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 -- 172.21.15.7:0/1031827670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c014fda0 msgr2=0x7fc1c006e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:36.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c014fda0 0x7fc1c006e690 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8009b90 tx=0x7fc1b8004750 comp rx=0 tx=0).stop 2023-03-03T17:00:36.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 -- 172.21.15.7:0/1031827670 shutdown_connections 2023-03-03T17:00:36.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1a00682f0 0x7fc1a006a7a0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:36.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1c014fda0 0x7fc1c006e690 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:36.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1c00abe20 0x7fc1c01e8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 --2- 172.21.15.7:0/1031827670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1c014f9d0 0x7fc1c00c08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 -- 172.21.15.7:0/1031827670 >> 172.21.15.7:0/1031827670 conn(0x7fc1c00c8900 msgr2=0x7fc1c0151830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:00:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 -- 172.21.15.7:0/1031827670 shutdown_connections 2023-03-03T17:00:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.016+0000 7fc1c5e15700 1 -- 172.21.15.7:0/1031827670 wait complete. 2023-03-03T17:00:36.017 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:00:36.032 INFO:tasks.mds_thrash.fs.[b]:waiting for 78 secs before thrashing 2023-03-03T17:00:36.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.139+0000 7f03da087700 1 -- 172.21.15.7:0/3195935107 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03d40c5220 con 0x7f03d4155410 2023-03-03T17:00:36.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.140+0000 7f03d0ff9700 1 -- 172.21.15.7:0/3195935107 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f03cc018090 con 0x7f03d4155410 2023-03-03T17:00:36.140 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:00:36.140 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:00:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 -- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03b40682f0 msgr2=0x7f03b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03b40682f0 0x7f03b406a7a0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009cd0 tx=0x7f03c4009210 comp rx=0 tx=0).stop 2023-03-03T17:00:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 -- 172.21.15.7:0/3195935107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03d4155410 msgr2=0x7f03d40c8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:00:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03d4155410 0x7f03d40c8860 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00cba0 tx=0x7f03cc00ceb0 comp rx=0 tx=0).stop 2023-03-03T17:00:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 -- 172.21.15.7:0/3195935107 shutdown_connections 2023-03-03T17:00:36.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f03b40682f0 0x7f03b406a7a0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f03d4155410 0x7f03d40c8860 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f03d40c6fe0 0x7f03d41e04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 --2- 172.21.15.7:0/3195935107 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03d4151e80 0x7f03d4157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:00:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 -- 172.21.15.7:0/3195935107 >> 172.21.15.7:0/3195935107 conn(0x7f03d4071630 msgr2=0x7f03d41e0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:00:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 -- 172.21.15.7:0/3195935107 shutdown_connections 2023-03-03T17:00:36.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:00:36.142+0000 7f03da087700 1 -- 172.21.15.7:0/3195935107 wait complete. 2023-03-03T17:00:36.144 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:00:36.153 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2023-03-03T17:00:52.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:00:52.005 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:22.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:22.136 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:52.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:52.291 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:01:54.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:01:54.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.173+0000 7f4620855700 1 Processor -- start 2023-03-03T17:01:54.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.173+0000 7f4620855700 1 -- start start 2023-03-03T17:01:54.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4620855700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f461c0caa30 0x7f461c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:54.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4620855700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f461c0c00e0 0x7f461c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:54.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4620855700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f461c0cb280 con 0x7f461c0cae00 2023-03-03T17:01:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4620855700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461c0c7310 con 0x7f461c0c00e0 2023-03-03T17:01:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4620855700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c0bfd10 con 0x7f461c0caa30 2023-03-03T17:01:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4619d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f461c0c00e0 0x7f461c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:54.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4619d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f461c0c00e0 0x7f461c154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59610/0 (socket says 172.21.15.7:59610) 2023-03-03T17:01:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4619d9b700 1 -- 172.21.15.7:0/3461410738 learned_addr learned my addr 172.21.15.7:0/3461410738 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:01:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f461a59c700 1 --2- 172.21.15.7:0/3461410738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f461c0caa30 0x7f461c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4619d9b700 1 -- 172.21.15.7:0/3461410738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f461c0caa30 msgr2=0x7f461c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:01:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4619d9b700 1 --2- 172.21.15.7:0/3461410738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f461c0caa30 0x7f461c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4619d9b700 1 -- 172.21.15.7:0/3461410738 >> v1:172.21.15.143:6789/0 conn(0x7f461c0cae00 legacy=0x7f461c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:54.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.174+0000 7f4619d9b700 1 -- 172.21.15.7:0/3461410738 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c0bfa30 con 0x7f461c0c00e0 2023-03-03T17:01:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.175+0000 7f4619d9b700 1 --2- 172.21.15.7:0/3461410738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0c00e0 0x7f461c154ff0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f460c006fd0 tx=0x7f460c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fb76afc66f7be0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:01:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.175+0000 7f461959a700 1 -- 172.21.15.7:0/3461410738 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c005e30 con 0x7f461c0c00e0 2023-03-03T17:01:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.175+0000 7f461959a700 1 -- 172.21.15.7:0/3461410738 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c004030 con 0x7f461c0c00e0 2023-03-03T17:01:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.175+0000 7f461959a700 1 -- 172.21.15.7:0/3461410738 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c0043e0 con 0x7f461c0c00e0 2023-03-03T17:01:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.175+0000 7f4620855700 1 -- 172.21.15.7:0/3461410738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0c00e0 msgr2=0x7f461c154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.175+0000 7f4620855700 1 --2- 172.21.15.7:0/3461410738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0c00e0 0x7f461c154ff0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f460c006fd0 tx=0x7f460c009bd0 comp rx=0 tx=0).stop 2023-03-03T17:01:54.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.175+0000 7f4620855700 1 -- 172.21.15.7:0/3461410738 shutdown_connections 2023-03-03T17:01:54.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.175+0000 7f4620855700 1 --2- 172.21.15.7:0/3461410738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f461c0caa30 0x7f461c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:54.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.175+0000 7f4620855700 1 --2- 172.21.15.7:0/3461410738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0c00e0 0x7f461c154ff0 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:54.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.176+0000 7f4620855700 1 -- 172.21.15.7:0/3461410738 >> 172.21.15.7:0/3461410738 conn(0x7f461c066e40 msgr2=0x7f461c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:01:54.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.176+0000 7f4620855700 1 -- 172.21.15.7:0/3461410738 shutdown_connections 2023-03-03T17:01:54.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.176+0000 7f4620855700 1 -- 172.21.15.7:0/3461410738 wait complete. 2023-03-03T17:01:54.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.177+0000 7f4620855700 1 Processor -- start 2023-03-03T17:01:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.177+0000 7f4620855700 1 -- start start 2023-03-03T17:01:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4620855700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f461c0c00e0 0x7f461c149680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4620855700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0caa30 0x7f461c1e8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4620855700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f461c0cae00 0x7f461c148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4620855700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c158260 con 0x7f461c0caa30 2023-03-03T17:01:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4620855700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c1494a0 con 0x7f461c0cae00 2023-03-03T17:01:54.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4620855700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c1dfca0 con 0x7f461c0c00e0 2023-03-03T17:01:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4619d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0caa30 0x7f461c1e8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4619d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0caa30 0x7f461c1e8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59616/0 (socket says 172.21.15.7:59616) 2023-03-03T17:01:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4619d9b700 1 -- 172.21.15.7:0/648559232 learned_addr learned my addr 172.21.15.7:0/648559232 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:01:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f461a59c700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f461c0c00e0 0x7f461c149680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f461ad9d700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f461c0cae00 0x7f461c148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4619d9b700 1 -- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f461c0c00e0 msgr2=0x7f461c149680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:54.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4619d9b700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f461c0c00e0 0x7f461c149680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4619d9b700 1 -- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f461c0cae00 msgr2=0x7f461c148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4619d9b700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f461c0cae00 0x7f461c148f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.178+0000 7f4619d9b700 1 -- 172.21.15.7:0/648559232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c009500 con 0x7f461c0caa30 2023-03-03T17:01:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.179+0000 7f4619d9b700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0caa30 0x7f461c1e8150 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f460c009c10 tx=0x7f460c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:01:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.179+0000 7f460affd700 1 -- 172.21.15.7:0/648559232 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c003cb0 con 0x7f461c0caa30 2023-03-03T17:01:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.179+0000 7f460affd700 1 -- 172.21.15.7:0/648559232 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c003e10 con 0x7f461c0caa30 2023-03-03T17:01:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.179+0000 7f4620855700 1 -- 172.21.15.7:0/648559232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461c1547f0 con 0x7f461c0caa30 2023-03-03T17:01:54.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.179+0000 7f460affd700 1 -- 172.21.15.7:0/648559232 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c011660 con 0x7f461c0caa30 2023-03-03T17:01:54.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.179+0000 7f4620855700 1 -- 172.21.15.7:0/648559232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461c154510 con 0x7f461c0caa30 2023-03-03T17:01:54.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.180+0000 7f4620855700 1 -- 172.21.15.7:0/648559232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461c154ad0 con 0x7f461c0caa30 2023-03-03T17:01:54.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.180+0000 7f460affd700 1 -- 172.21.15.7:0/648559232 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f460c02c070 con 0x7f461c0caa30 2023-03-03T17:01:54.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.181+0000 7f460affd700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45fc0682f0 0x7f45fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:54.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.181+0000 7f460affd700 1 -- 172.21.15.7:0/648559232 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f460c0870f0 con 0x7f461c0caa30 2023-03-03T17:01:54.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.181+0000 7f461a59c700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45fc0682f0 0x7f45fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:54.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.182+0000 7f461a59c700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45fc0682f0 0x7f45fc06a7a0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f4610006b20 tx=0x7f4610008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:01:54.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.185+0000 7f460affd700 1 -- 172.21.15.7:0/648559232 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f460c053490 con 0x7f461c0caa30 2023-03-03T17:01:54.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.424+0000 7f4620855700 1 -- 172.21.15.7:0/648559232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f461c074bb0 con 0x7f461c0caa30 2023-03-03T17:01:54.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.425+0000 7f460affd700 1 -- 172.21.15.7:0/648559232 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f460c056ab0 con 0x7f461c0caa30 2023-03-03T17:01:54.426 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:01:54.426 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:01:54.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 -- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45fc0682f0 msgr2=0x7f45fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:54.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45fc0682f0 0x7f45fc06a7a0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f4610006b20 tx=0x7f4610008040 comp rx=0 tx=0).stop 2023-03-03T17:01:54.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 -- 172.21.15.7:0/648559232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0caa30 msgr2=0x7f461c1e8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:54.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0caa30 0x7f461c1e8150 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f460c009c10 tx=0x7f460c004750 comp rx=0 tx=0).stop 2023-03-03T17:01:54.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 -- 172.21.15.7:0/648559232 shutdown_connections 2023-03-03T17:01:54.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f461c0cae00 0x7f461c148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:54.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f461c0c00e0 0x7f461c149680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:54.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45fc0682f0 0x7f45fc06a7a0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:54.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 --2- 172.21.15.7:0/648559232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f461c0caa30 0x7f461c1e8150 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:54.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 -- 172.21.15.7:0/648559232 >> 172.21.15.7:0/648559232 conn(0x7f461c066e40 msgr2=0x7f461c1532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:01:54.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 -- 172.21.15.7:0/648559232 shutdown_connections 2023-03-03T17:01:54.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:54.427+0000 7f4620855700 1 -- 172.21.15.7:0/648559232 wait complete. 2023-03-03T17:01:54.429 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:01:54.440 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-03-03T17:01:55.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:01:55.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.293+0000 7f403cb7b700 1 Processor -- start 2023-03-03T17:01:55.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.293+0000 7f403cb7b700 1 -- start start 2023-03-03T17:01:55.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.294+0000 7f403cb7b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40380c00e0 0x7f40380c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:55.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.294+0000 7f403cb7b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380c7310 con 0x7f40380caa30 2023-03-03T17:01:55.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.294+0000 7f403cb7b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380bfd10 con 0x7f4038154ff0 2023-03-03T17:01:55.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.294+0000 7f403cb7b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40380bfa30 con 0x7f40380c00e0 2023-03-03T17:01:55.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.294+0000 7f403659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f40380caa30 0x7f40380c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53606/0 (socket says 172.21.15.7:53606) 2023-03-03T17:01:55.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.294+0000 7f403659c700 1 -- 172.21.15.7:0/868163482 learned_addr learned my addr 172.21.15.7:0/868163482 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:01:55.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18364200 0 0) 0x7f40380c7310 con 0x7f40380caa30 2023-03-03T17:01:55.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f4035d9b700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40380c00e0 0x7f40380c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:55.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40200035b0 con 0x7f40380caa30 2023-03-03T17:01:55.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3327354897 0 0) 0x7f40200035b0 con 0x7f40380caa30 2023-03-03T17:01:55.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40380c00e0 msgr2=0x7f40380c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:01:55.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40380c00e0 0x7f40380c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:55.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 >> v1:172.21.15.143:6789/0 conn(0x7f4038154ff0 legacy=0x7f403815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:55.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4038154510 con 0x7f40380caa30 2023-03-03T17:01:55.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f4028001670 con 0x7f40380caa30 2023-03-03T17:01:55.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 >> v1:172.21.15.7:6789/0 conn(0x7f40380caa30 legacy=0x7f40380c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:55.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4020003aa0 0x7f4020003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:55.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40380c00e0 0x7f4020004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:55.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4020004d90 0x7f4020005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:55.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40380bfa30 con 0x7f4020004d90 2023-03-03T17:01:55.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402000bb10 con 0x7f4020003aa0 2023-03-03T17:01:55.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402000bcd0 con 0x7f40380c00e0 2023-03-03T17:01:55.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f4036d9d700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4020004d90 0x7f4020005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:55.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f4035d9b700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40380c00e0 0x7f4020004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:55.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f403659c700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4020003aa0 0x7f4020003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:55.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f4036d9d700 1 -- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40380c00e0 msgr2=0x7f4020004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:01:55.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f4036d9d700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f40380c00e0 0x7f4020004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:55.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f4036d9d700 1 -- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4020003aa0 msgr2=0x7f4020003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:01:55.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f4036d9d700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4020003aa0 0x7f4020003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:55.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.295+0000 7f4036d9d700 1 -- 172.21.15.7:0/868163482 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40200035b0 con 0x7f4020004d90 2023-03-03T17:01:55.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f4036d9d700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4020004d90 0x7f4020005200 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f403000c290 tx=0x7f4020005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd96211f406e4a1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:01:55.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030001e70 con 0x7f4020004d90 2023-03-03T17:01:55.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403000ce50 con 0x7f4020004d90 2023-03-03T17:01:55.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f403559a700 1 -- 172.21.15.7:0/868163482 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4030012660 con 0x7f4020004d90 2023-03-03T17:01:55.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f403cb7b700 1 -- 172.21.15.7:0/868163482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4020004d90 msgr2=0x7f4020005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:55.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f403cb7b700 1 --2- 172.21.15.7:0/868163482 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4020004d90 0x7f4020005200 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f403000c290 tx=0x7f4020005880 comp rx=0 tx=0).stop 2023-03-03T17:01:55.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f4036d9d700 1 -- 172.21.15.7:0/868163482 reap_dead start 2023-03-03T17:01:55.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f403cb7b700 1 -- 172.21.15.7:0/868163482 shutdown_connections 2023-03-03T17:01:55.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f403cb7b700 1 -- 172.21.15.7:0/868163482 >> 172.21.15.7:0/868163482 conn(0x7f4038066e40 msgr2=0x7f4038071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:01:55.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f403cb7b700 1 -- 172.21.15.7:0/868163482 shutdown_connections 2023-03-03T17:01:55.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.296+0000 7f403cb7b700 1 -- 172.21.15.7:0/868163482 wait complete. 2023-03-03T17:01:55.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.298+0000 7f403cb7b700 1 Processor -- start 2023-03-03T17:01:55.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.298+0000 7f403cb7b700 1 -- start start 2023-03-03T17:01:55.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.298+0000 7f403cb7b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4020004d90 0x7f40380c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:55.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.298+0000 7f403cb7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f403814d9e0 0x7f403814cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:55.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.298+0000 7f403cb7b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4038156560 0x7f403814d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:55.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.298+0000 7f403cb7b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40381dda40 con 0x7f403814d9e0 2023-03-03T17:01:55.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.298+0000 7f403cb7b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403814d700 con 0x7f4020004d90 2023-03-03T17:01:55.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.298+0000 7f403cb7b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40381edda0 con 0x7f4038156560 2023-03-03T17:01:55.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4035d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f403814d9e0 0x7f403814cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:55.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4035d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f403814d9e0 0x7f403814cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59640/0 (socket says 172.21.15.7:59640) 2023-03-03T17:01:55.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4035d9b700 1 -- 172.21.15.7:0/1528370001 learned_addr learned my addr 172.21.15.7:0/1528370001 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:01:55.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f403659c700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4020004d90 0x7f40380c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:55.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4036d9d700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4038156560 0x7f403814d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:55.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4035d9b700 1 -- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4038156560 msgr2=0x7f403814d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:55.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4035d9b700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4038156560 0x7f403814d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:55.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4035d9b700 1 -- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4020004d90 msgr2=0x7f40380c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:55.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4035d9b700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4020004d90 0x7f40380c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:55.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4035d9b700 1 -- 172.21.15.7:0/1528370001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4038154ad0 con 0x7f403814d9e0 2023-03-03T17:01:55.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4035d9b700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f403814d9e0 0x7f403814cc80 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f402c0119f0 tx=0x7f402c011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:01:55.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4026ffd700 1 -- 172.21.15.7:0/1528370001 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c011e90 con 0x7f403814d9e0 2023-03-03T17:01:55.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4026ffd700 1 -- 172.21.15.7:0/1528370001 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c005c30 con 0x7f403814d9e0 2023-03-03T17:01:55.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f403cb7b700 1 -- 172.21.15.7:0/1528370001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40381547f0 con 0x7f403814d9e0 2023-03-03T17:01:55.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f4026ffd700 1 -- 172.21.15.7:0/1528370001 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c00f730 con 0x7f403814d9e0 2023-03-03T17:01:55.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.299+0000 7f403cb7b700 1 -- 172.21.15.7:0/1528370001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40380bf650 con 0x7f403814d9e0 2023-03-03T17:01:55.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.300+0000 7f403cb7b700 1 -- 172.21.15.7:0/1528370001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4038074bb0 con 0x7f403814d9e0 2023-03-03T17:01:55.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.301+0000 7f4026ffd700 1 -- 172.21.15.7:0/1528370001 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f402c00f890 con 0x7f403814d9e0 2023-03-03T17:01:55.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.301+0000 7f4026ffd700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40180682f0 0x7f401806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:01:55.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.301+0000 7f4026ffd700 1 -- 172.21.15.7:0/1528370001 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f402c086880 con 0x7f403814d9e0 2023-03-03T17:01:55.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.301+0000 7f403659c700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40180682f0 0x7f401806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:01:55.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.302+0000 7f403659c700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40180682f0 0x7f401806a7a0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f4028001cb0 tx=0x7f4028001c40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:01:55.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.305+0000 7f4026ffd700 1 -- 172.21.15.7:0/1528370001 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f402c052ce0 con 0x7f403814d9e0 2023-03-03T17:01:55.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.543+0000 7f403cb7b700 1 -- 172.21.15.7:0/1528370001 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40380c5030 con 0x7f403814d9e0 2023-03-03T17:01:55.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.543+0000 7f4026ffd700 1 -- 172.21.15.7:0/1528370001 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f402c056300 con 0x7f403814d9e0 2023-03-03T17:01:55.545 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:01:55.545 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:01:55.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 -- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40180682f0 msgr2=0x7f401806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:55.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40180682f0 0x7f401806a7a0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f4028001cb0 tx=0x7f4028001c40 comp rx=0 tx=0).stop 2023-03-03T17:01:55.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 -- 172.21.15.7:0/1528370001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f403814d9e0 msgr2=0x7f403814cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:01:55.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f403814d9e0 0x7f403814cc80 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f402c0119f0 tx=0x7f402c011d00 comp rx=0 tx=0).stop 2023-03-03T17:01:55.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 -- 172.21.15.7:0/1528370001 shutdown_connections 2023-03-03T17:01:55.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4038156560 0x7f403814d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:55.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4020004d90 0x7f40380c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:55.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40180682f0 0x7f401806a7a0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:55.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 --2- 172.21.15.7:0/1528370001 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f403814d9e0 0x7f403814cc80 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:01:55.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 -- 172.21.15.7:0/1528370001 >> 172.21.15.7:0/1528370001 conn(0x7f4038066e40 msgr2=0x7f4038065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:01:55.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 -- 172.21.15.7:0/1528370001 shutdown_connections 2023-03-03T17:01:55.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:01:55.545+0000 7f403cb7b700 1 -- 172.21.15.7:0/1528370001 wait complete. 2023-03-03T17:01:55.548 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:01:55.558 INFO:tasks.mds_thrash.fs.[a]:waiting for 84 secs before thrashing 2023-03-03T17:02:22.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:22.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:52.639 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:02:52.642 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:16.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:03:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.581+0000 7f3fa7ce6700 1 Processor -- start 2023-03-03T17:03:16.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.582+0000 7f3fa7ce6700 1 -- start start 2023-03-03T17:03:16.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.582+0000 7f3fa7ce6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3fa0152e30 0x7f3fa00c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:16.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.582+0000 7f3fa7ce6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 0x7f3fa00c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:16.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.582+0000 7f3fa7ce6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa01523b0 con 0x7f3fa0155d80 2023-03-03T17:03:16.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.582+0000 7f3fa7ce6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa0156930 con 0x7f3fa0152e30 2023-03-03T17:03:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.582+0000 7f3fa7ce6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa0072010 con 0x7f3fa00beab0 2023-03-03T17:03:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa5281700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 0x7f3fa00c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa5a82700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3fa0152e30 0x7f3fa00c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa6283700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f3fa0155d80 0x7f3fa0156220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:57204/0 (socket says 172.21.15.7:57204) 2023-03-03T17:03:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa6283700 1 -- 172.21.15.7:0/534251013 learned_addr learned my addr 172.21.15.7:0/534251013 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:03:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa4a80700 1 -- 172.21.15.7:0/534251013 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231118970 0 0) 0x7f3fa01523b0 con 0x7f3fa0155d80 2023-03-03T17:03:16.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa4a80700 1 -- 172.21.15.7:0/534251013 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f8c0035b0 con 0x7f3fa0155d80 2023-03-03T17:03:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa5281700 1 -- 172.21.15.7:0/534251013 >> v2:172.21.15.143:3300/0 conn(0x7f3fa0152e30 msgr2=0x7f3fa00c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:03:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa5281700 1 --2- 172.21.15.7:0/534251013 >> v2:172.21.15.143:3300/0 conn(0x7f3fa0152e30 0x7f3fa00c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa5281700 1 -- 172.21.15.7:0/534251013 >> v1:172.21.15.143:6789/0 conn(0x7f3fa0155d80 legacy=0x7f3fa0156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa5281700 1 -- 172.21.15.7:0/534251013 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa00aaaf0 con 0x7f3fa00beab0 2023-03-03T17:03:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa5a82700 1 --2- 172.21.15.7:0/534251013 >> v2:172.21.15.143:3300/0 conn(0x7f3fa0152e30 0x7f3fa00c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:03:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.583+0000 7f3fa5281700 1 --2- 172.21.15.7:0/534251013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 0x7f3fa00c4910 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009840 tx=0x7f3f94009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd8686919fc37265 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:16.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa4a80700 1 -- 172.21.15.7:0/534251013 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f94011830 con 0x7f3fa00beab0 2023-03-03T17:03:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa4a80700 1 -- 172.21.15.7:0/534251013 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f94011990 con 0x7f3fa00beab0 2023-03-03T17:03:16.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa4a80700 1 -- 172.21.15.7:0/534251013 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f94011d20 con 0x7f3fa00beab0 2023-03-03T17:03:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/534251013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 msgr2=0x7f3fa00c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa7ce6700 1 --2- 172.21.15.7:0/534251013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 0x7f3fa00c4910 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009840 tx=0x7f3f94009b50 comp rx=0 tx=0).stop 2023-03-03T17:03:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/534251013 shutdown_connections 2023-03-03T17:03:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa7ce6700 1 --2- 172.21.15.7:0/534251013 >> v2:172.21.15.143:3300/0 conn(0x7f3fa0152e30 0x7f3fa00c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa7ce6700 1 --2- 172.21.15.7:0/534251013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 0x7f3fa00c4910 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:16.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/534251013 >> 172.21.15.7:0/534251013 conn(0x7f3fa00736f0 msgr2=0x7f3fa0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:03:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/534251013 shutdown_connections 2023-03-03T17:03:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.584+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/534251013 wait complete. 2023-03-03T17:03:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.585+0000 7f3fa7ce6700 1 Processor -- start 2023-03-03T17:03:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.586+0000 7f3fa7ce6700 1 -- start start 2023-03-03T17:03:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.586+0000 7f3fa7ce6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 0x7f3fa014b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:16.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.586+0000 7f3fa7ce6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa0152e30 0x7f3fa0154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.586+0000 7f3fa7ce6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fa0155d80 0x7f3fa0150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.586+0000 7f3fa7ce6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa01e99e0 con 0x7f3fa0155d80 2023-03-03T17:03:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.586+0000 7f3fa7ce6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa014b750 con 0x7f3fa0152e30 2023-03-03T17:03:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.586+0000 7f3fa7ce6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa01525b0 con 0x7f3fa00beab0 2023-03-03T17:03:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.586+0000 7f3fa6283700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fa0155d80 0x7f3fa0150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.586+0000 7f3fa6283700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fa0155d80 0x7f3fa0150c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59710/0 (socket says 172.21.15.7:59710) 2023-03-03T17:03:16.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.586+0000 7f3fa6283700 1 -- 172.21.15.7:0/4269237645 learned_addr learned my addr 172.21.15.7:0/4269237645 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:03:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3fa5281700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa0152e30 0x7f3fa0154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3fa5a82700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 0x7f3fa014b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3fa6283700 1 -- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 msgr2=0x7f3fa014b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3fa6283700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 0x7f3fa014b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3fa6283700 1 -- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa0152e30 msgr2=0x7f3fa0154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3fa6283700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa0152e30 0x7f3fa0154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:16.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3fa6283700 1 -- 172.21.15.7:0/4269237645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c0094a0 con 0x7f3fa0155d80 2023-03-03T17:03:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3fa6283700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fa0155d80 0x7f3fa0150c70 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800cb40 tx=0x7f3f9800ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3f927fc700 1 -- 172.21.15.7:0/4269237645 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9800aaa0 con 0x7f3fa0155d80 2023-03-03T17:03:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3f927fc700 1 -- 172.21.15.7:0/4269237645 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9800ac00 con 0x7f3fa0155d80 2023-03-03T17:03:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/4269237645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f940094a0 con 0x7f3fa0155d80 2023-03-03T17:03:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3f927fc700 1 -- 172.21.15.7:0/4269237645 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98012400 con 0x7f3fa0155d80 2023-03-03T17:03:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.587+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/4269237645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa0071c70 con 0x7f3fa0155d80 2023-03-03T17:03:16.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.588+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/4269237645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa00aadd0 con 0x7f3fa0155d80 2023-03-03T17:03:16.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.589+0000 7f3f927fc700 1 -- 172.21.15.7:0/4269237645 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3f98019030 con 0x7f3fa0155d80 2023-03-03T17:03:16.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.589+0000 7f3f927fc700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f84068320 0x7f3f8406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.589+0000 7f3f927fc700 1 -- 172.21.15.7:0/4269237645 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3f98085d30 con 0x7f3fa0155d80 2023-03-03T17:03:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.589+0000 7f3fa5a82700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f84068320 0x7f3f8406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.590+0000 7f3fa5a82700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f84068320 0x7f3f8406a7d0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009d00 tx=0x7f3f9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:16.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.593+0000 7f3f927fc700 1 -- 172.21.15.7:0/4269237645 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3f980520d0 con 0x7f3fa0155d80 2023-03-03T17:03:16.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.830+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/4269237645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fa0153200 con 0x7f3fa0155d80 2023-03-03T17:03:16.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.830+0000 7f3f927fc700 1 -- 172.21.15.7:0/4269237645 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3f980556f0 con 0x7f3fa0155d80 2023-03-03T17:03:16.830 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:03:16.831 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:03:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f84068320 msgr2=0x7f3f8406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f84068320 0x7f3f8406a7d0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009d00 tx=0x7f3f9c008040 comp rx=0 tx=0).stop 2023-03-03T17:03:16.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/4269237645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fa0155d80 msgr2=0x7f3fa0150c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:16.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fa0155d80 0x7f3fa0150c70 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f3f9800cb40 tx=0x7f3f9800ce50 comp rx=0 tx=0).stop 2023-03-03T17:03:16.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/4269237645 shutdown_connections 2023-03-03T17:03:16.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f84068320 0x7f3f8406a7d0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:16.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fa0155d80 0x7f3fa0150c70 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:16.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3fa00beab0 0x7f3fa014b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:16.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 --2- 172.21.15.7:0/4269237645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fa0152e30 0x7f3fa0154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:16.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/4269237645 >> 172.21.15.7:0/4269237645 conn(0x7f3fa00736f0 msgr2=0x7f3fa00be2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:03:16.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/4269237645 shutdown_connections 2023-03-03T17:03:16.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:16.832+0000 7f3fa7ce6700 1 -- 172.21.15.7:0/4269237645 wait complete. 2023-03-03T17:03:16.834 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:03:16.844 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T17:03:19.560 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:03:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.701+0000 7f5cf0091700 1 Processor -- start 2023-03-03T17:03:19.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.701+0000 7f5cf0091700 1 -- start start 2023-03-03T17:03:19.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cf0091700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce8072470 0x7f5ce806b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:19.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cf0091700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce8070360 con 0x7f5ce8071990 2023-03-03T17:03:19.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cf0091700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce806a5e0 con 0x7f5ce814b460 2023-03-03T17:03:19.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cf0091700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce81521a0 con 0x7f5ce8072470 2023-03-03T17:03:19.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cede2d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5ce8071990 0x7f5ce8071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53700/0 (socket says 172.21.15.7:53700) 2023-03-03T17:03:19.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cede2d700 1 -- 172.21.15.7:0/2380756898 learned_addr learned my addr 172.21.15.7:0/2380756898 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:03:19.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249329500 0 0) 0x7f5ce8070360 con 0x7f5ce8071990 2023-03-03T17:03:19.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd40035b0 con 0x7f5ce8071990 2023-03-03T17:03:19.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cee62e700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce8072470 0x7f5ce806b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:19.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145878617 0 0) 0x7f5cd40035b0 con 0x7f5ce8071990 2023-03-03T17:03:19.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce8072470 msgr2=0x7f5ce806b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:03:19.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cece2b700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce8072470 0x7f5ce806b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:19.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.702+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 >> v1:172.21.15.143:6789/0 conn(0x7f5ce814b460 legacy=0x7f5ce806aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:19.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce80c38b0 con 0x7f5ce8071990 2023-03-03T17:03:19.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f5cd8001670 con 0x7f5ce8071990 2023-03-03T17:03:19.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 >> v1:172.21.15.7:6789/0 conn(0x7f5ce8071990 legacy=0x7f5ce8071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:19.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cd4003aa0 0x7f5cd4003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:19.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce8072470 0x7f5cd4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:19.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cd4004d90 0x7f5cd4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:19.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce81521a0 con 0x7f5cd4003aa0 2023-03-03T17:03:19.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd400bba0 con 0x7f5cd4004d90 2023-03-03T17:03:19.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd400bd60 con 0x7f5ce8072470 2023-03-03T17:03:19.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cede2d700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cd4003aa0 0x7f5cd4003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:19.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5ced62c700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce8072470 0x7f5cd4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:19.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cee62e700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cd4004d90 0x7f5cd4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:19.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cede2d700 1 -- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce8072470 msgr2=0x7f5cd4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:03:19.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cede2d700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce8072470 0x7f5cd4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:19.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cede2d700 1 -- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cd4004d90 msgr2=0x7f5cd4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:03:19.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cede2d700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cd4004d90 0x7f5cd4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:19.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cede2d700 1 -- 172.21.15.7:0/2380756898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd40035b0 con 0x7f5cd4003aa0 2023-03-03T17:03:19.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5ced62c700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce8072470 0x7f5cd4004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:03:19.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cede2d700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cd4003aa0 0x7f5cd4003eb0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f5cd800c240 tx=0x7f5cd4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5edeb24b8a68 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:19.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8007c10 con 0x7f5cd4003aa0 2023-03-03T17:03:19.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd800cde0 con 0x7f5cd4003aa0 2023-03-03T17:03:19.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.703+0000 7f5cece2b700 1 -- 172.21.15.7:0/2380756898 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8005ba0 con 0x7f5cd4003aa0 2023-03-03T17:03:19.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.704+0000 7f5cf0091700 1 -- 172.21.15.7:0/2380756898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cd4003aa0 msgr2=0x7f5cd4003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:19.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.704+0000 7f5cf0091700 1 --2- 172.21.15.7:0/2380756898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5cd4003aa0 0x7f5cd4003eb0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f5cd800c240 tx=0x7f5cd4005880 comp rx=0 tx=0).stop 2023-03-03T17:03:19.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.704+0000 7f5cee62e700 1 -- 172.21.15.7:0/2380756898 reap_dead start 2023-03-03T17:03:19.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.704+0000 7f5cf0091700 1 -- 172.21.15.7:0/2380756898 shutdown_connections 2023-03-03T17:03:19.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.704+0000 7f5cf0091700 1 -- 172.21.15.7:0/2380756898 >> 172.21.15.7:0/2380756898 conn(0x7f5ce80a94d0 msgr2=0x7f5ce80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:03:19.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.704+0000 7f5cf0091700 1 -- 172.21.15.7:0/2380756898 shutdown_connections 2023-03-03T17:03:19.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.704+0000 7f5cf0091700 1 -- 172.21.15.7:0/2380756898 wait complete. 2023-03-03T17:03:19.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.706+0000 7f5cf0091700 1 Processor -- start 2023-03-03T17:03:19.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.706+0000 7f5cf0091700 1 -- start start 2023-03-03T17:03:19.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cf0091700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cd4003aa0 0x7f5ce80abc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:19.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cf0091700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce80c4d00 0x7f5ce80aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:19.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cf0091700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce81e92d0 0x7f5ce8154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:19.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cf0091700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce813f340 con 0x7f5ce81e92d0 2023-03-03T17:03:19.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cf0091700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce81e8890 con 0x7f5cd4003aa0 2023-03-03T17:03:19.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cf0091700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce80aaa60 con 0x7f5ce80c4d00 2023-03-03T17:03:19.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cee62e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce81e92d0 0x7f5ce8154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:19.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cee62e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce81e92d0 0x7f5ce8154d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59736/0 (socket says 172.21.15.7:59736) 2023-03-03T17:03:19.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cee62e700 1 -- 172.21.15.7:0/3585609022 learned_addr learned my addr 172.21.15.7:0/3585609022 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:03:19.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cede2d700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cd4003aa0 0x7f5ce80abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:19.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5ced62c700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce80c4d00 0x7f5ce80aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:19.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cee62e700 1 -- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce80c4d00 msgr2=0x7f5ce80aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:19.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cee62e700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce80c4d00 0x7f5ce80aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:19.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cee62e700 1 -- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cd4003aa0 msgr2=0x7f5ce80abc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:19.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cee62e700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cd4003aa0 0x7f5ce80abc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:19.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cee62e700 1 -- 172.21.15.7:0/3585609022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce80c3b50 con 0x7f5ce81e92d0 2023-03-03T17:03:19.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cede2d700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cd4003aa0 0x7f5ce80abc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:03:19.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cee62e700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce81e92d0 0x7f5ce8154d20 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80be320 tx=0x7f5ce0011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:19.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cde7fc700 1 -- 172.21.15.7:0/3585609022 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce000e040 con 0x7f5ce81e92d0 2023-03-03T17:03:19.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cde7fc700 1 -- 172.21.15.7:0/3585609022 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce000f040 con 0x7f5ce81e92d0 2023-03-03T17:03:19.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cde7fc700 1 -- 172.21.15.7:0/3585609022 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce000f8f0 con 0x7f5ce81e92d0 2023-03-03T17:03:19.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.707+0000 7f5cf0091700 1 -- 172.21.15.7:0/3585609022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce806a2b0 con 0x7f5ce81e92d0 2023-03-03T17:03:19.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.708+0000 7f5cf0091700 1 -- 172.21.15.7:0/3585609022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce8151c00 con 0x7f5ce81e92d0 2023-03-03T17:03:19.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.708+0000 7f5cf0091700 1 -- 172.21.15.7:0/3585609022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce80c5ea0 con 0x7f5ce81e92d0 2023-03-03T17:03:19.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.709+0000 7f5cde7fc700 1 -- 172.21.15.7:0/3585609022 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5ce0025070 con 0x7f5ce81e92d0 2023-03-03T17:03:19.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.709+0000 7f5cde7fc700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ccc0682f0 0x7f5ccc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:19.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.709+0000 7f5cde7fc700 1 -- 172.21.15.7:0/3585609022 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5ce00866c0 con 0x7f5ce81e92d0 2023-03-03T17:03:19.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.709+0000 7f5cede2d700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ccc0682f0 0x7f5ccc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:19.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.710+0000 7f5cede2d700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ccc0682f0 0x7f5ccc06a7a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8001930 tx=0x7f5cd8007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:19.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.714+0000 7f5cde7fc700 1 -- 172.21.15.7:0/3585609022 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5ce0052970 con 0x7f5ce81e92d0 2023-03-03T17:03:19.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.958+0000 7f5cf0091700 1 -- 172.21.15.7:0/3585609022 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ce80aabf0 con 0x7f5ce81e92d0 2023-03-03T17:03:19.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.958+0000 7f5cde7fc700 1 -- 172.21.15.7:0/3585609022 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5ce0055f90 con 0x7f5ce81e92d0 2023-03-03T17:03:19.959 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:03:19.959 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:03:19.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 -- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ccc0682f0 msgr2=0x7f5ccc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:19.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ccc0682f0 0x7f5ccc06a7a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8001930 tx=0x7f5cd8007000 comp rx=0 tx=0).stop 2023-03-03T17:03:19.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 -- 172.21.15.7:0/3585609022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce81e92d0 msgr2=0x7f5ce8154d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:19.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce81e92d0 0x7f5ce8154d20 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80be320 tx=0x7f5ce0011ed0 comp rx=0 tx=0).stop 2023-03-03T17:03:19.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 -- 172.21.15.7:0/3585609022 shutdown_connections 2023-03-03T17:03:19.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5ccc0682f0 0x7f5ccc06a7a0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:19.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ce81e92d0 0x7f5ce8154d20 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:19.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cd4003aa0 0x7f5ce80abc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:19.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 --2- 172.21.15.7:0/3585609022 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ce80c4d00 0x7f5ce80aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:19.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 -- 172.21.15.7:0/3585609022 >> 172.21.15.7:0/3585609022 conn(0x7f5ce80a94d0 msgr2=0x7f5ce813fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:03:19.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 -- 172.21.15.7:0/3585609022 shutdown_connections 2023-03-03T17:03:19.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:19.960+0000 7f5cf0091700 1 -- 172.21.15.7:0/3585609022 wait complete. 2023-03-03T17:03:19.962 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:03:19.972 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T17:03:22.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:22.720 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:29.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:03:30.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.114+0000 7f3f36003700 1 Processor -- start 2023-03-03T17:03:30.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.114+0000 7f3f36003700 1 -- start start 2023-03-03T17:03:30.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f36003700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014b020 0x7f3f300717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:30.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f36003700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f300728b0 con 0x7f3f300aad20 2023-03-03T17:03:30.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f36003700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f30155070 con 0x7f3f30071d20 2023-03-03T17:03:30.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f36003700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3006f5b0 con 0x7f3f3014b020 2023-03-03T17:03:30.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3f300aad20 0x7f3f30157ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53726/0 (socket says 172.21.15.7:53726) 2023-03-03T17:03:30.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2effd700 1 -- 172.21.15.7:0/3607384334 learned_addr learned my addr 172.21.15.7:0/3607384334 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:03:30.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2f7fe700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014b020 0x7f3f300717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:30.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906115628 0 0) 0x7f3f300728b0 con 0x7f3f300aad20 2023-03-03T17:03:30.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f180035b0 con 0x7f3f300aad20 2023-03-03T17:03:30.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220404489 0 0) 0x7f3f180035b0 con 0x7f3f300aad20 2023-03-03T17:03:30.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014b020 msgr2=0x7f3f300717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:03:30.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014b020 0x7f3f300717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:30.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 >> v1:172.21.15.143:6789/0 conn(0x7f3f30071d20 legacy=0x7f3f301583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:30.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3006f890 con 0x7f3f300aad20 2023-03-03T17:03:30.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f3f24001670 con 0x7f3f300aad20 2023-03-03T17:03:30.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 >> v1:172.21.15.7:6789/0 conn(0x7f3f300aad20 legacy=0x7f3f30157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:30.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18003aa0 0x7f3f18003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:30.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014b020 0x7f3f18004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:30.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f18004d90 0x7f3f18005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:30.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3006f5b0 con 0x7f3f18004d90 2023-03-03T17:03:30.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1800bba0 con 0x7f3f18003aa0 2023-03-03T17:03:30.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.115+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1800bd60 con 0x7f3f3014b020 2023-03-03T17:03:30.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2f7fe700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18003aa0 0x7f3f18003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:30.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2ffff700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f18004d90 0x7f3f18005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:30.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2effd700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014b020 0x7f3f18004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:30.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2ffff700 1 -- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014b020 msgr2=0x7f3f18004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:03:30.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2ffff700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014b020 0x7f3f18004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:30.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2ffff700 1 -- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18003aa0 msgr2=0x7f3f18003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:03:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2ffff700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f18003aa0 0x7f3f18003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2ffff700 1 -- 172.21.15.7:0/3607384334 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180035b0 con 0x7f3f18004d90 2023-03-03T17:03:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2ffff700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f18004d90 0x7f3f18005200 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800c290 tx=0x7f3f18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35aab911e037faa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f28001e70 con 0x7f3f18004d90 2023-03-03T17:03:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2800ce50 con 0x7f3f18004d90 2023-03-03T17:03:30.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.116+0000 7f3f2e7fc700 1 -- 172.21.15.7:0/3607384334 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f28012660 con 0x7f3f18004d90 2023-03-03T17:03:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.117+0000 7f3f36003700 1 -- 172.21.15.7:0/3607384334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f18004d90 msgr2=0x7f3f18005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.117+0000 7f3f36003700 1 --2- 172.21.15.7:0/3607384334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f18004d90 0x7f3f18005200 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800c290 tx=0x7f3f18005880 comp rx=0 tx=0).stop 2023-03-03T17:03:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.117+0000 7f3f2ffff700 1 -- 172.21.15.7:0/3607384334 reap_dead start 2023-03-03T17:03:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.117+0000 7f3f36003700 1 -- 172.21.15.7:0/3607384334 shutdown_connections 2023-03-03T17:03:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.117+0000 7f3f36003700 1 -- 172.21.15.7:0/3607384334 >> 172.21.15.7:0/3607384334 conn(0x7f3f300c8900 msgr2=0x7f3f300c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:03:30.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.117+0000 7f3f36003700 1 -- 172.21.15.7:0/3607384334 shutdown_connections 2023-03-03T17:03:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.117+0000 7f3f36003700 1 -- 172.21.15.7:0/3607384334 wait complete. 2023-03-03T17:03:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.118+0000 7f3f36003700 1 Processor -- start 2023-03-03T17:03:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.118+0000 7f3f36003700 1 -- start start 2023-03-03T17:03:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f36003700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014f0d0 0x7f3f30150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f36003700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f3014f4a0 0x7f3f3014c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f36003700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f301ecdc0 0x7f3f301532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:30.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f36003700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3014c5e0 con 0x7f3f3014f4a0 2023-03-03T17:03:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f36003700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30152e60 con 0x7f3f301ecdc0 2023-03-03T17:03:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f36003700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30152250 con 0x7f3f3014f0d0 2023-03-03T17:03:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f2effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f3014f4a0 0x7f3f3014c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f2effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f3014f4a0 0x7f3f3014c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59760/0 (socket says 172.21.15.7:59760) 2023-03-03T17:03:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f2effd700 1 -- 172.21.15.7:0/992512645 learned_addr learned my addr 172.21.15.7:0/992512645 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:03:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f2ffff700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f301ecdc0 0x7f3f301532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f2f7fe700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014f0d0 0x7f3f30150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:30.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f2effd700 1 -- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014f0d0 msgr2=0x7f3f30150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:30.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f2effd700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014f0d0 0x7f3f30150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:30.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f2effd700 1 -- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f301ecdc0 msgr2=0x7f3f301532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:30.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f2effd700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f301ecdc0 0x7f3f301532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:30.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.119+0000 7f3f2effd700 1 -- 172.21.15.7:0/992512645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f30155350 con 0x7f3f3014f4a0 2023-03-03T17:03:30.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.120+0000 7f3f2effd700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f3014f4a0 0x7f3f3014c0a0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f3f240017a0 tx=0x7f3f24007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:30.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.120+0000 7f3f17fff700 1 -- 172.21.15.7:0/992512645 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24000f60 con 0x7f3f3014f4a0 2023-03-03T17:03:30.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.120+0000 7f3f17fff700 1 -- 172.21.15.7:0/992512645 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f24012560 con 0x7f3f3014f4a0 2023-03-03T17:03:30.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.120+0000 7f3f36003700 1 -- 172.21.15.7:0/992512645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3006fb70 con 0x7f3f3014f4a0 2023-03-03T17:03:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.120+0000 7f3f17fff700 1 -- 172.21.15.7:0/992512645 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24012910 con 0x7f3f3014f4a0 2023-03-03T17:03:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.120+0000 7f3f36003700 1 -- 172.21.15.7:0/992512645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f300aa030 con 0x7f3f3014f4a0 2023-03-03T17:03:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.121+0000 7f3f36003700 1 -- 172.21.15.7:0/992512645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f30152fd0 con 0x7f3f3014f4a0 2023-03-03T17:03:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.121+0000 7f3f17fff700 1 -- 172.21.15.7:0/992512645 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3f24012a70 con 0x7f3f3014f4a0 2023-03-03T17:03:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.122+0000 7f3f17fff700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f1c0682f0 0x7f3f1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:03:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.122+0000 7f3f17fff700 1 -- 172.21.15.7:0/992512645 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3f24086200 con 0x7f3f3014f4a0 2023-03-03T17:03:30.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.122+0000 7f3f2f7fe700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f1c0682f0 0x7f3f1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:03:30.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.123+0000 7f3f2f7fe700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f1c0682f0 0x7f3f1c06a7a0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009c60 tx=0x7f3f20009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:03:30.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.126+0000 7f3f17fff700 1 -- 172.21.15.7:0/992512645 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3f24052450 con 0x7f3f3014f4a0 2023-03-03T17:03:30.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.362+0000 7f3f36003700 1 -- 172.21.15.7:0/992512645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f30074bb0 con 0x7f3f3014f4a0 2023-03-03T17:03:30.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.363+0000 7f3f17fff700 1 -- 172.21.15.7:0/992512645 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3f24055a70 con 0x7f3f3014f4a0 2023-03-03T17:03:30.363 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:03:30.363 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:03:30.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 -- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f1c0682f0 msgr2=0x7f3f1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:30.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f1c0682f0 0x7f3f1c06a7a0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009c60 tx=0x7f3f20009210 comp rx=0 tx=0).stop 2023-03-03T17:03:30.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 -- 172.21.15.7:0/992512645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f3014f4a0 msgr2=0x7f3f3014c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:03:30.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f3014f4a0 0x7f3f3014c0a0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f3f240017a0 tx=0x7f3f24007570 comp rx=0 tx=0).stop 2023-03-03T17:03:30.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 -- 172.21.15.7:0/992512645 shutdown_connections 2023-03-03T17:03:30.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f301ecdc0 0x7f3f301532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:30.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f3014f0d0 0x7f3f30150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:30.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f1c0682f0 0x7f3f1c06a7a0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:30.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 --2- 172.21.15.7:0/992512645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f3014f4a0 0x7f3f3014c0a0 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:03:30.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 -- 172.21.15.7:0/992512645 >> 172.21.15.7:0/992512645 conn(0x7f3f300c8900 msgr2=0x7f3f301546a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:03:30.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 -- 172.21.15.7:0/992512645 shutdown_connections 2023-03-03T17:03:30.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:03:30.365+0000 7f3f36003700 1 -- 172.21.15.7:0/992512645 wait complete. 2023-03-03T17:03:30.367 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:03:30.377 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T17:03:52.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:03:52.792 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:22.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:22.856 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:35.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:04:35.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.517+0000 7f8de4809700 1 Processor -- start 2023-03-03T17:04:35.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.518+0000 7f8de4809700 1 -- start start 2023-03-03T17:04:35.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.518+0000 7f8de4809700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8de014b020 0x7f8de00717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:04:35.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.518+0000 7f8de4809700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8de0071d20 0x7f8de01583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:04:35.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.518+0000 7f8de4809700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de01588f0 con 0x7f8de00aad20 2023-03-03T17:04:35.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.518+0000 7f8de4809700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de00728b0 con 0x7f8de014b020 2023-03-03T17:04:35.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.518+0000 7f8de4809700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de0155070 con 0x7f8de0071d20 2023-03-03T17:04:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8dde59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8de014b020 0x7f8de00717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:04:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8dde59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8de014b020 0x7f8de00717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59824/0 (socket says 172.21.15.7:59824) 2023-03-03T17:04:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8dde59c700 1 -- 172.21.15.7:0/1163159817 learned_addr learned my addr 172.21.15.7:0/1163159817 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:04:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8dded9d700 1 --2- 172.21.15.7:0/1163159817 >> v2:172.21.15.143:3300/0 conn(0x7f8de0071d20 0x7f8de01583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:04:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8dde59c700 1 -- 172.21.15.7:0/1163159817 >> v2:172.21.15.143:3300/0 conn(0x7f8de0071d20 msgr2=0x7f8de01583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:04:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8dde59c700 1 --2- 172.21.15.7:0/1163159817 >> v2:172.21.15.143:3300/0 conn(0x7f8de0071d20 0x7f8de01583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:04:35.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8dde59c700 1 -- 172.21.15.7:0/1163159817 >> v1:172.21.15.143:6789/0 conn(0x7f8de00aad20 legacy=0x7f8de0157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:04:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8dde59c700 1 -- 172.21.15.7:0/1163159817 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de006f5b0 con 0x7f8de014b020 2023-03-03T17:04:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8dde59c700 1 --2- 172.21.15.7:0/1163159817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de014b020 0x7f8de00717e0 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f8de00adb00 tx=0x7f8dd0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af7bc93d11c523d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:04:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8ddd59a700 1 -- 172.21.15.7:0/1163159817 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0011830 con 0x7f8de014b020 2023-03-03T17:04:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8ddd59a700 1 -- 172.21.15.7:0/1163159817 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd0011990 con 0x7f8de014b020 2023-03-03T17:04:35.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.519+0000 7f8ddd59a700 1 -- 172.21.15.7:0/1163159817 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0011d20 con 0x7f8de014b020 2023-03-03T17:04:35.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.520+0000 7f8de4809700 1 -- 172.21.15.7:0/1163159817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de014b020 msgr2=0x7f8de00717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:04:35.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.520+0000 7f8de4809700 1 --2- 172.21.15.7:0/1163159817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de014b020 0x7f8de00717e0 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f8de00adb00 tx=0x7f8dd0009bd0 comp rx=0 tx=0).stop 2023-03-03T17:04:35.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.520+0000 7f8de4809700 1 -- 172.21.15.7:0/1163159817 shutdown_connections 2023-03-03T17:04:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.520+0000 7f8de4809700 1 --2- 172.21.15.7:0/1163159817 >> v2:172.21.15.143:3300/0 conn(0x7f8de0071d20 0x7f8de01583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:04:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.520+0000 7f8de4809700 1 --2- 172.21.15.7:0/1163159817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de014b020 0x7f8de00717e0 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:04:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.520+0000 7f8de4809700 1 -- 172.21.15.7:0/1163159817 >> 172.21.15.7:0/1163159817 conn(0x7f8de00c8900 msgr2=0x7f8de00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:04:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.520+0000 7f8de4809700 1 -- 172.21.15.7:0/1163159817 shutdown_connections 2023-03-03T17:04:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.520+0000 7f8de4809700 1 -- 172.21.15.7:0/1163159817 wait complete. 2023-03-03T17:04:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.521+0000 7f8de4809700 1 Processor -- start 2023-03-03T17:04:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.521+0000 7f8de4809700 1 -- start start 2023-03-03T17:04:35.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8de4809700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de0071d20 0x7f8de0146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:04:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8de4809700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8de00aad20 0x7f8de0146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:04:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8de4809700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8de014b020 0x7f8de01ed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:04:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8de4809700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de01471a0 con 0x7f8de0071d20 2023-03-03T17:04:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8de4809700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de0154240 con 0x7f8de014b020 2023-03-03T17:04:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8de4809700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de0154800 con 0x7f8de00aad20 2023-03-03T17:04:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8dde59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de0071d20 0x7f8de0146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:04:35.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8dde59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de0071d20 0x7f8de0146720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59830/0 (socket says 172.21.15.7:59830) 2023-03-03T17:04:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8dde59c700 1 -- 172.21.15.7:0/1048300559 learned_addr learned my addr 172.21.15.7:0/1048300559 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:04:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8dddd9b700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8de00aad20 0x7f8de0146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:04:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.522+0000 7f8dded9d700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8de014b020 0x7f8de01ed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:04:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8dde59c700 1 -- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8de00aad20 msgr2=0x7f8de0146c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:04:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8dde59c700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8de00aad20 0x7f8de0146c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:04:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8dde59c700 1 -- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8de014b020 msgr2=0x7f8de01ed3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:04:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8dde59c700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8de014b020 0x7f8de01ed3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:04:35.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8dde59c700 1 -- 172.21.15.7:0/1048300559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd0009550 con 0x7f8de0071d20 2023-03-03T17:04:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8dde59c700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de0071d20 0x7f8de0146720 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0014040 tx=0x7f8dd0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:04:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8dd6ffd700 1 -- 172.21.15.7:0/1048300559 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd00036a0 con 0x7f8de0071d20 2023-03-03T17:04:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8dd6ffd700 1 -- 172.21.15.7:0/1048300559 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd001e070 con 0x7f8de0071d20 2023-03-03T17:04:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8de4809700 1 -- 172.21.15.7:0/1048300559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de006fb70 con 0x7f8de0071d20 2023-03-03T17:04:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8dd6ffd700 1 -- 172.21.15.7:0/1048300559 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0003a60 con 0x7f8de0071d20 2023-03-03T17:04:35.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.523+0000 7f8de4809700 1 -- 172.21.15.7:0/1048300559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de006f890 con 0x7f8de0071d20 2023-03-03T17:04:35.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.524+0000 7f8de4809700 1 -- 172.21.15.7:0/1048300559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de0155350 con 0x7f8de0071d20 2023-03-03T17:04:35.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.525+0000 7f8dd6ffd700 1 -- 172.21.15.7:0/1048300559 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8dd000f050 con 0x7f8de0071d20 2023-03-03T17:04:35.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.525+0000 7f8dd6ffd700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 0x7f8dc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:04:35.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.525+0000 7f8dd6ffd700 1 -- 172.21.15.7:0/1048300559 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8dd0086f90 con 0x7f8de0071d20 2023-03-03T17:04:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.525+0000 7f8dddd9b700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 0x7f8dc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:04:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.526+0000 7f8dddd9b700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 0x7f8dc006a7a0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f8de00ad250 tx=0x7f8dc8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:04:35.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.529+0000 7f8dd6ffd700 1 -- 172.21.15.7:0/1048300559 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8dd0053330 con 0x7f8de0071d20 2023-03-03T17:04:35.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.767+0000 7f8de4809700 1 -- 172.21.15.7:0/1048300559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8de0074bb0 con 0x7f8de0071d20 2023-03-03T17:04:35.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.768+0000 7f8dd6ffd700 1 -- 172.21.15.7:0/1048300559 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8dd0020090 con 0x7f8de0071d20 2023-03-03T17:04:35.768 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:04:35.768 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:04:35.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 -- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 msgr2=0x7f8dc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:04:35.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 0x7f8dc006a7a0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f8de00ad250 tx=0x7f8dc8009040 comp rx=0 tx=0).stop 2023-03-03T17:04:35.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 -- 172.21.15.7:0/1048300559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de0071d20 msgr2=0x7f8de0146720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:04:35.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de0071d20 0x7f8de0146720 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0014040 tx=0x7f8dd0004750 comp rx=0 tx=0).stop 2023-03-03T17:04:35.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 -- 172.21.15.7:0/1048300559 shutdown_connections 2023-03-03T17:04:35.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8de014b020 0x7f8de01ed3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:04:35.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8dc00682f0 0x7f8dc006a7a0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:04:35.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8de0071d20 0x7f8de0146720 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:04:35.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 --2- 172.21.15.7:0/1048300559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8de00aad20 0x7f8de0146c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:04:35.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 -- 172.21.15.7:0/1048300559 >> 172.21.15.7:0/1048300559 conn(0x7f8de00c8900 msgr2=0x7f8de00ac310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:04:35.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 -- 172.21.15.7:0/1048300559 shutdown_connections 2023-03-03T17:04:35.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:04:35.770+0000 7f8de4809700 1 -- 172.21.15.7:0/1048300559 wait complete. 2023-03-03T17:04:35.772 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:04:35.782 INFO:tasks.mds_thrash.fs.[a]:waiting for 51 secs before thrashing 2023-03-03T17:04:52.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:04:52.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:12.845 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:05:12.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.984+0000 7fbfe0d11700 1 Processor -- start 2023-03-03T17:05:12.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.984+0000 7fbfe0d11700 1 -- start start 2023-03-03T17:05:12.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.985+0000 7fbfe0d11700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbfdc151cd0 0x7fbfdc154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:12.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.985+0000 7fbfe0d11700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfdc06c6c0 0x7fbfdc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:12.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.985+0000 7fbfe0d11700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfdc157ff0 con 0x7fbfdc0c2e30 2023-03-03T17:05:12.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.985+0000 7fbfe0d11700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfdc06d640 con 0x7fbfdc151cd0 2023-03-03T17:05:12.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.985+0000 7fbfe0d11700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc06cda0 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.985+0000 7fbfda59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbfdc151cd0 0x7fbfdc154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:12.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.985+0000 7fbfd9d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbfdc0c2e30 0x7fbfdc158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53824/0 (socket says 172.21.15.7:53824) 2023-03-03T17:05:12.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.985+0000 7fbfd9d9b700 1 -- 172.21.15.7:0/3089383032 learned_addr learned my addr 172.21.15.7:0/3089383032 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:05:12.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.986+0000 7fbfdad9d700 1 --2- 172.21.15.7:0/3089383032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfdc06c6c0 0x7fbfdc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:12.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.986+0000 7fbfda59c700 1 -- 172.21.15.7:0/3089383032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfdc06c6c0 msgr2=0x7fbfdc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:05:12.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.986+0000 7fbfda59c700 1 --2- 172.21.15.7:0/3089383032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfdc06c6c0 0x7fbfdc0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:12.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.986+0000 7fbfda59c700 1 -- 172.21.15.7:0/3089383032 >> v1:172.21.15.7:6789/0 conn(0x7fbfdc0c2e30 legacy=0x7fbfdc158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:12.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.986+0000 7fbfd959a700 1 -- 172.21.15.7:0/3089383032 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121790043 0 0) 0x7fbfdc157ff0 con 0x7fbfdc0c2e30 2023-03-03T17:05:12.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.986+0000 7fbfda59c700 1 -- 172.21.15.7:0/3089383032 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc06d080 con 0x7fbfdc151cd0 2023-03-03T17:05:12.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.986+0000 7fbfda59c700 1 --2- 172.21.15.7:0/3089383032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc151cd0 0x7fbfdc154510 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009840 tx=0x7fbfcc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff39f12eed2e9c85 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:12.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.986+0000 7fbfd959a700 1 -- 172.21.15.7:0/3089383032 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc011830 con 0x7fbfdc151cd0 2023-03-03T17:05:12.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.986+0000 7fbfd959a700 1 -- 172.21.15.7:0/3089383032 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc011990 con 0x7fbfdc151cd0 2023-03-03T17:05:12.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.986+0000 7fbfd959a700 1 -- 172.21.15.7:0/3089383032 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc011d20 con 0x7fbfdc151cd0 2023-03-03T17:05:12.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.987+0000 7fbfe0d11700 1 -- 172.21.15.7:0/3089383032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc151cd0 msgr2=0x7fbfdc154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:12.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.987+0000 7fbfe0d11700 1 --2- 172.21.15.7:0/3089383032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc151cd0 0x7fbfdc154510 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009840 tx=0x7fbfcc009b50 comp rx=0 tx=0).stop 2023-03-03T17:05:12.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.987+0000 7fbfe0d11700 1 -- 172.21.15.7:0/3089383032 shutdown_connections 2023-03-03T17:05:12.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.987+0000 7fbfe0d11700 1 --2- 172.21.15.7:0/3089383032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfdc06c6c0 0x7fbfdc0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:12.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.987+0000 7fbfe0d11700 1 --2- 172.21.15.7:0/3089383032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc151cd0 0x7fbfdc154510 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:12.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.987+0000 7fbfe0d11700 1 -- 172.21.15.7:0/3089383032 >> 172.21.15.7:0/3089383032 conn(0x7fbfdc0a94d0 msgr2=0x7fbfdc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:05:12.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.987+0000 7fbfe0d11700 1 -- 172.21.15.7:0/3089383032 shutdown_connections 2023-03-03T17:05:12.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.987+0000 7fbfe0d11700 1 -- 172.21.15.7:0/3089383032 wait complete. 2023-03-03T17:05:12.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.988+0000 7fbfe0d11700 1 Processor -- start 2023-03-03T17:05:12.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.989+0000 7fbfe0d11700 1 -- start start 2023-03-03T17:05:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.989+0000 7fbfe0d11700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc06c6c0 0x7fbfdc1571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.989+0000 7fbfe0d11700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfdc0c2e30 0x7fbfdc143ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.989+0000 7fbfe0d11700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfdc151cd0 0x7fbfdc144410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfe0d11700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc14a260 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfe0d11700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc0c56a0 con 0x7fbfdc151cd0 2023-03-03T17:05:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfe0d11700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfdc144c20 con 0x7fbfdc0c2e30 2023-03-03T17:05:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfda59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc06c6c0 0x7fbfdc1571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfda59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc06c6c0 0x7fbfdc1571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59848/0 (socket says 172.21.15.7:59848) 2023-03-03T17:05:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfda59c700 1 -- 172.21.15.7:0/2123842854 learned_addr learned my addr 172.21.15.7:0/2123842854 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:05:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfdad9d700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfdc151cd0 0x7fbfdc144410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfd9d9b700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfdc0c2e30 0x7fbfdc143ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfda59c700 1 -- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfdc0c2e30 msgr2=0x7fbfdc143ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfda59c700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfdc0c2e30 0x7fbfdc143ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfda59c700 1 -- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfdc151cd0 msgr2=0x7fbfdc144410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfda59c700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfdc151cd0 0x7fbfdc144410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfda59c700 1 -- 172.21.15.7:0/2123842854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc0094a0 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfdad9d700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfdc151cd0 0x7fbfdc144410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:05:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfda59c700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc06c6c0 0x7fbfdc1571d0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc014040 tx=0x7fbfcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfcaffd700 1 -- 172.21.15.7:0/2123842854 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc0036a0 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfcaffd700 1 -- 172.21.15.7:0/2123842854 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfcc01e070 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfcaffd700 1 -- 172.21.15.7:0/2123842854 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc003a00 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfe0d11700 1 -- 172.21.15.7:0/2123842854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfdc157c80 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.990+0000 7fbfe0d11700 1 -- 172.21.15.7:0/2123842854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfdc06d360 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.991+0000 7fbfe0d11700 1 -- 172.21.15.7:0/2123842854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfdc154a50 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.992+0000 7fbfcaffd700 1 -- 172.21.15.7:0/2123842854 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbfcc00f050 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.992+0000 7fbfcaffd700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfbc0682f0 0x7fbfbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.992+0000 7fbfcaffd700 1 -- 172.21.15.7:0/2123842854 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbfcc0871c0 con 0x7fbfdc06c6c0 2023-03-03T17:05:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.992+0000 7fbfd9d9b700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfbc0682f0 0x7fbfbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:12.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.993+0000 7fbfd9d9b700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfbc0682f0 0x7fbfbc06a7a0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0c5af0 tx=0x7fbfd0007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:12.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:12.997+0000 7fbfcaffd700 1 -- 172.21.15.7:0/2123842854 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbfcc053560 con 0x7fbfdc06c6c0 2023-03-03T17:05:13.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.236+0000 7fbfe0d11700 1 -- 172.21.15.7:0/2123842854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfdc074bb0 con 0x7fbfdc06c6c0 2023-03-03T17:05:13.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.237+0000 7fbfcaffd700 1 -- 172.21.15.7:0/2123842854 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbfcc020090 con 0x7fbfdc06c6c0 2023-03-03T17:05:13.238 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:05:13.238 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:05:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 -- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfbc0682f0 msgr2=0x7fbfbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfbc0682f0 0x7fbfbc06a7a0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0c5af0 tx=0x7fbfd0007e70 comp rx=0 tx=0).stop 2023-03-03T17:05:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 -- 172.21.15.7:0/2123842854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc06c6c0 msgr2=0x7fbfdc1571d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc06c6c0 0x7fbfdc1571d0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc014040 tx=0x7fbfcc004750 comp rx=0 tx=0).stop 2023-03-03T17:05:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 -- 172.21.15.7:0/2123842854 shutdown_connections 2023-03-03T17:05:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfdc151cd0 0x7fbfdc144410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbfbc0682f0 0x7fbfbc06a7a0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfdc06c6c0 0x7fbfdc1571d0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 --2- 172.21.15.7:0/2123842854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbfdc0c2e30 0x7fbfdc143ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 -- 172.21.15.7:0/2123842854 >> 172.21.15.7:0/2123842854 conn(0x7fbfdc0a94d0 msgr2=0x7fbfdc0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:05:13.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 -- 172.21.15.7:0/2123842854 shutdown_connections 2023-03-03T17:05:13.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:13.239+0000 7fbfe0d11700 1 -- 172.21.15.7:0/2123842854 wait complete. 2023-03-03T17:05:13.242 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:05:13.252 INFO:tasks.mds_thrash.fs.[b]:waiting for 37 secs before thrashing 2023-03-03T17:05:22.989 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:22.992 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:26.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:05:26.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.923+0000 7faa2faf9700 1 Processor -- start 2023-03-03T17:05:26.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.923+0000 7faa2faf9700 1 -- start start 2023-03-03T17:05:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2faf9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa2814fda0 0x7faa28156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2faf9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa280c2f30 con 0x7faa2814f9d0 2023-03-03T17:05:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2faf9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa280c31d0 con 0x7faa280abe20 2023-03-03T17:05:26.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2faf9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa281524a0 con 0x7faa2814fda0 2023-03-03T17:05:26.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2d895700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faa2814f9d0 0x7faa28071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53856/0 (socket says 172.21.15.7:53856) 2023-03-03T17:05:26.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2d895700 1 -- 172.21.15.7:0/1819772855 learned_addr learned my addr 172.21.15.7:0/1819772855 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:05:26.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2d094700 1 --2- 172.21.15.7:0/1819772855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa2814fda0 0x7faa28156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:26.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2c893700 1 -- 172.21.15.7:0/1819772855 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128671172 0 0) 0x7faa280c2f30 con 0x7faa2814f9d0 2023-03-03T17:05:26.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2c893700 1 -- 172.21.15.7:0/1819772855 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa140035b0 con 0x7faa2814f9d0 2023-03-03T17:05:26.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2c893700 1 -- 172.21.15.7:0/1819772855 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961565364 0 0) 0x7faa280c31d0 con 0x7faa280abe20 2023-03-03T17:05:26.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.924+0000 7faa2c893700 1 -- 172.21.15.7:0/1819772855 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa280c2f30 con 0x7faa280abe20 2023-03-03T17:05:26.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.925+0000 7faa2d094700 1 -- 172.21.15.7:0/1819772855 >> v1:172.21.15.143:6789/0 conn(0x7faa280abe20 legacy=0x7faa281529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:26.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.925+0000 7faa2d094700 1 -- 172.21.15.7:0/1819772855 >> v1:172.21.15.7:6789/0 conn(0x7faa2814f9d0 legacy=0x7faa28071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:26.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.925+0000 7faa2d094700 1 -- 172.21.15.7:0/1819772855 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa280c34b0 con 0x7faa2814fda0 2023-03-03T17:05:26.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.925+0000 7faa2d094700 1 --2- 172.21.15.7:0/1819772855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa2814fda0 0x7faa28156430 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa18009870 tx=0x7faa18009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb1bbd28a7d42a81 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:26.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.925+0000 7faa2c893700 1 -- 172.21.15.7:0/1819772855 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18011830 con 0x7faa2814fda0 2023-03-03T17:05:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.925+0000 7faa2c893700 1 -- 172.21.15.7:0/1819772855 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18011990 con 0x7faa2814fda0 2023-03-03T17:05:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.925+0000 7faa2c893700 1 -- 172.21.15.7:0/1819772855 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18011d20 con 0x7faa2814fda0 2023-03-03T17:05:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.926+0000 7faa2faf9700 1 -- 172.21.15.7:0/1819772855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa2814fda0 msgr2=0x7faa28156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.926+0000 7faa2faf9700 1 --2- 172.21.15.7:0/1819772855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa2814fda0 0x7faa28156430 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa18009870 tx=0x7faa18009b80 comp rx=0 tx=0).stop 2023-03-03T17:05:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.926+0000 7faa2faf9700 1 -- 172.21.15.7:0/1819772855 shutdown_connections 2023-03-03T17:05:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.926+0000 7faa2faf9700 1 --2- 172.21.15.7:0/1819772855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa2814fda0 0x7faa28156430 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.926+0000 7faa2faf9700 1 -- 172.21.15.7:0/1819772855 >> 172.21.15.7:0/1819772855 conn(0x7faa280c8900 msgr2=0x7faa280c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:05:26.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.926+0000 7faa2faf9700 1 -- 172.21.15.7:0/1819772855 shutdown_connections 2023-03-03T17:05:26.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.926+0000 7faa2faf9700 1 -- 172.21.15.7:0/1819772855 wait complete. 2023-03-03T17:05:26.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.927+0000 7faa2faf9700 1 Processor -- start 2023-03-03T17:05:26.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.927+0000 7faa2faf9700 1 -- start start 2023-03-03T17:05:26.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2faf9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa280abe20 0x7faa2806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:26.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2faf9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2814f9d0 0x7faa28152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:26.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2faf9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa2814fda0 0x7faa2806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:26.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2faf9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2806e0f0 con 0x7faa2814f9d0 2023-03-03T17:05:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2faf9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2806e900 con 0x7faa2814fda0 2023-03-03T17:05:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2faf9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa281dd8b0 con 0x7faa280abe20 2023-03-03T17:05:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2d094700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2814f9d0 0x7faa28152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2d094700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2814f9d0 0x7faa28152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59884/0 (socket says 172.21.15.7:59884) 2023-03-03T17:05:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2d094700 1 -- 172.21.15.7:0/4059642925 learned_addr learned my addr 172.21.15.7:0/4059642925 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:05:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2e096700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa2814fda0 0x7faa2806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2d895700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa280abe20 0x7faa2806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:26.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2d094700 1 -- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa280abe20 msgr2=0x7faa2806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:26.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2d094700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa280abe20 0x7faa2806c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:26.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2d094700 1 -- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa2814fda0 msgr2=0x7faa2806f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:26.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2d094700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa2814fda0 0x7faa2806f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:26.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2d094700 1 -- 172.21.15.7:0/4059642925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa180094a0 con 0x7faa2814f9d0 2023-03-03T17:05:26.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2e096700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa2814fda0 0x7faa2806f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:05:26.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2d094700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2814f9d0 0x7faa28152b30 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7faa180097c0 tx=0x7faa18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:26.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa1e7fc700 1 -- 172.21.15.7:0/4059642925 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1801ba80 con 0x7faa2814f9d0 2023-03-03T17:05:26.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa1e7fc700 1 -- 172.21.15.7:0/4059642925 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1801bbe0 con 0x7faa2814f9d0 2023-03-03T17:05:26.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2faf9700 1 -- 172.21.15.7:0/4059642925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa280ac350 con 0x7faa2814f9d0 2023-03-03T17:05:26.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa1e7fc700 1 -- 172.21.15.7:0/4059642925 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa180253f0 con 0x7faa2814f9d0 2023-03-03T17:05:26.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.928+0000 7faa2faf9700 1 -- 172.21.15.7:0/4059642925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa280ab630 con 0x7faa2814f9d0 2023-03-03T17:05:26.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.929+0000 7faa2faf9700 1 -- 172.21.15.7:0/4059642925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa28071fa0 con 0x7faa2814f9d0 2023-03-03T17:05:26.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.930+0000 7faa1e7fc700 1 -- 172.21.15.7:0/4059642925 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faa1800f050 con 0x7faa2814f9d0 2023-03-03T17:05:26.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.930+0000 7faa1e7fc700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 0x7faa0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:26.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.930+0000 7faa1e7fc700 1 -- 172.21.15.7:0/4059642925 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faa18087330 con 0x7faa2814f9d0 2023-03-03T17:05:26.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.931+0000 7faa2d895700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 0x7faa0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:26.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.931+0000 7faa2d895700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 0x7faa0c06a7a0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7faa24000f60 tx=0x7faa24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:26.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:26.935+0000 7faa1e7fc700 1 -- 172.21.15.7:0/4059642925 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faa180536d0 con 0x7faa2814f9d0 2023-03-03T17:05:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.173+0000 7faa2faf9700 1 -- 172.21.15.7:0/4059642925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa28074bb0 con 0x7faa2814f9d0 2023-03-03T17:05:27.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.173+0000 7faa1e7fc700 1 -- 172.21.15.7:0/4059642925 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faa18015090 con 0x7faa2814f9d0 2023-03-03T17:05:27.174 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:05:27.174 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:05:27.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.175+0000 7faa2faf9700 1 -- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 msgr2=0x7faa0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.175+0000 7faa2faf9700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 0x7faa0c06a7a0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7faa24000f60 tx=0x7faa24009040 comp rx=0 tx=0).stop 2023-03-03T17:05:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.175+0000 7faa2faf9700 1 -- 172.21.15.7:0/4059642925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2814f9d0 msgr2=0x7faa28152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.175+0000 7faa2faf9700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2814f9d0 0x7faa28152b30 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7faa180097c0 tx=0x7faa18004750 comp rx=0 tx=0).stop 2023-03-03T17:05:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.175+0000 7faa2faf9700 1 -- 172.21.15.7:0/4059642925 shutdown_connections 2023-03-03T17:05:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.175+0000 7faa2faf9700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa2814fda0 0x7faa2806f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.175+0000 7faa2faf9700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa280abe20 0x7faa2806c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:27.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.175+0000 7faa2faf9700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa0c0682f0 0x7faa0c06a7a0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:27.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.175+0000 7faa2faf9700 1 --2- 172.21.15.7:0/4059642925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2814f9d0 0x7faa28152b30 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:27.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.176+0000 7faa2faf9700 1 -- 172.21.15.7:0/4059642925 >> 172.21.15.7:0/4059642925 conn(0x7faa280c8900 msgr2=0x7faa280c3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:05:27.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.176+0000 7faa2faf9700 1 -- 172.21.15.7:0/4059642925 shutdown_connections 2023-03-03T17:05:27.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:27.176+0000 7faa2faf9700 1 -- 172.21.15.7:0/4059642925 wait complete. 2023-03-03T17:05:27.177 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:05:27.187 INFO:tasks.mds_thrash.fs.[a]:waiting for 32 secs before thrashing 2023-03-03T17:05:50.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:05:50.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.393+0000 7f17d07d8700 1 Processor -- start 2023-03-03T17:05:50.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.393+0000 7f17d07d8700 1 -- start start 2023-03-03T17:05:50.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17d07d8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f17c8072250 0x7f17c814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:50.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17d07d8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f17c80c1ce0 0x7f17c814e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:50.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17d07d8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c81510d0 con 0x7f17c8071e80 2023-03-03T17:05:50.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17d07d8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c80c3740 con 0x7f17c8072250 2023-03-03T17:05:50.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17d07d8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c8070670 con 0x7f17c80c1ce0 2023-03-03T17:05:50.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17cdd73700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f17c8072250 0x7f17c814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:50.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17ce574700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f17c8071e80 0x7f17c8151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53886/0 (socket says 172.21.15.7:53886) 2023-03-03T17:05:50.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17ce574700 1 -- 172.21.15.7:0/353993645 learned_addr learned my addr 172.21.15.7:0/353993645 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:05:50.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17ced75700 1 --2- 172.21.15.7:0/353993645 >> v2:172.21.15.143:3300/0 conn(0x7f17c80c1ce0 0x7f17c814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:50.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17cdd73700 1 -- 172.21.15.7:0/353993645 >> v2:172.21.15.143:3300/0 conn(0x7f17c80c1ce0 msgr2=0x7f17c814e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:05:50.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17cdd73700 1 --2- 172.21.15.7:0/353993645 >> v2:172.21.15.143:3300/0 conn(0x7f17c80c1ce0 0x7f17c814e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:50.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17cdd73700 1 -- 172.21.15.7:0/353993645 >> v1:172.21.15.7:6789/0 conn(0x7f17c8071e80 legacy=0x7f17c8151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:50.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17cdd73700 1 -- 172.21.15.7:0/353993645 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c80c39e0 con 0x7f17c8072250 2023-03-03T17:05:50.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.394+0000 7f17cd572700 1 -- 172.21.15.7:0/353993645 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957034798 0 0) 0x7f17c81510d0 con 0x7f17c8071e80 2023-03-03T17:05:50.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17cdd73700 1 --2- 172.21.15.7:0/353993645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 0x7f17c814e1f0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f17c8071090 tx=0x7f17b8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f8929eece30d55 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:50.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17cd572700 1 -- 172.21.15.7:0/353993645 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8011830 con 0x7f17c8072250 2023-03-03T17:05:50.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17cd572700 1 -- 172.21.15.7:0/353993645 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b8011990 con 0x7f17c8072250 2023-03-03T17:05:50.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17cd572700 1 -- 172.21.15.7:0/353993645 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8011d20 con 0x7f17c8072250 2023-03-03T17:05:50.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17d07d8700 1 -- 172.21.15.7:0/353993645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 msgr2=0x7f17c814e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:50.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17d07d8700 1 --2- 172.21.15.7:0/353993645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 0x7f17c814e1f0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f17c8071090 tx=0x7f17b8009b20 comp rx=0 tx=0).stop 2023-03-03T17:05:50.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17d07d8700 1 -- 172.21.15.7:0/353993645 shutdown_connections 2023-03-03T17:05:50.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17d07d8700 1 --2- 172.21.15.7:0/353993645 >> v2:172.21.15.143:3300/0 conn(0x7f17c80c1ce0 0x7f17c814e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17d07d8700 1 --2- 172.21.15.7:0/353993645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 0x7f17c814e1f0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17d07d8700 1 -- 172.21.15.7:0/353993645 >> 172.21.15.7:0/353993645 conn(0x7f17c80a94d0 msgr2=0x7f17c80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:05:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17d07d8700 1 -- 172.21.15.7:0/353993645 shutdown_connections 2023-03-03T17:05:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.395+0000 7f17d07d8700 1 -- 172.21.15.7:0/353993645 wait complete. 2023-03-03T17:05:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17d07d8700 1 Processor -- start 2023-03-03T17:05:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17d07d8700 1 -- start start 2023-03-03T17:05:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17d07d8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c8071e80 0x7f17c81502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17d07d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 0x7f17c814bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:50.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17d07d8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17c80c1ce0 0x7f17c81dedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:50.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17d07d8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81ed860 con 0x7f17c8072250 2023-03-03T17:05:50.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17d07d8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81edd90 con 0x7f17c8071e80 2023-03-03T17:05:50.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17d07d8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8150800 con 0x7f17c80c1ce0 2023-03-03T17:05:50.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17cdd73700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 0x7f17c814bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:50.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17cdd73700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 0x7f17c814bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59914/0 (socket says 172.21.15.7:59914) 2023-03-03T17:05:50.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.397+0000 7f17cdd73700 1 -- 172.21.15.7:0/1302974328 learned_addr learned my addr 172.21.15.7:0/1302974328 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:05:50.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17ce574700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c8071e80 0x7f17c81502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:50.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17ced75700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17c80c1ce0 0x7f17c81dedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:50.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17cdd73700 1 -- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17c80c1ce0 msgr2=0x7f17c81dedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:50.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17cdd73700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17c80c1ce0 0x7f17c81dedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:50.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17cdd73700 1 -- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c8071e80 msgr2=0x7f17c81502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:50.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17cdd73700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c8071e80 0x7f17c81502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:50.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17cdd73700 1 -- 172.21.15.7:0/1302974328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b80094a0 con 0x7f17c8072250 2023-03-03T17:05:50.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17ced75700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17c80c1ce0 0x7f17c81dedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:05:50.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17cdd73700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 0x7f17c814bc80 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f17b8013000 tx=0x7f17b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:50.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17beffd700 1 -- 172.21.15.7:0/1302974328 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b80038a0 con 0x7f17c8072250 2023-03-03T17:05:50.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17beffd700 1 -- 172.21.15.7:0/1302974328 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b801f070 con 0x7f17c8072250 2023-03-03T17:05:50.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17beffd700 1 -- 172.21.15.7:0/1302974328 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8003c20 con 0x7f17c8072250 2023-03-03T17:05:50.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17d07d8700 1 -- 172.21.15.7:0/1302974328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c8151f30 con 0x7f17c8072250 2023-03-03T17:05:50.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.398+0000 7f17d07d8700 1 -- 172.21.15.7:0/1302974328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c8151300 con 0x7f17c8072250 2023-03-03T17:05:50.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.399+0000 7f17d07d8700 1 -- 172.21.15.7:0/1302974328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c806c9a0 con 0x7f17c8072250 2023-03-03T17:05:50.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.400+0000 7f17beffd700 1 -- 172.21.15.7:0/1302974328 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f17b800f050 con 0x7f17c8072250 2023-03-03T17:05:50.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.400+0000 7f17beffd700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17ac0682f0 0x7f17ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:50.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.400+0000 7f17beffd700 1 -- 172.21.15.7:0/1302974328 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f17b8021070 con 0x7f17c8072250 2023-03-03T17:05:50.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.400+0000 7f17ce574700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17ac0682f0 0x7f17ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:50.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.401+0000 7f17ce574700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17ac0682f0 0x7f17ac06a7a0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f17c4000f60 tx=0x7f17c4007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:50.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.404+0000 7f17beffd700 1 -- 172.21.15.7:0/1302974328 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f17b8053da0 con 0x7f17c8072250 2023-03-03T17:05:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.641+0000 7f17d07d8700 1 -- 172.21.15.7:0/1302974328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17c8074bb0 con 0x7f17c8072250 2023-03-03T17:05:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.642+0000 7f17beffd700 1 -- 172.21.15.7:0/1302974328 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f17b80573c0 con 0x7f17c8072250 2023-03-03T17:05:50.642 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:05:50.642 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:05:50.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 -- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17ac0682f0 msgr2=0x7f17ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:50.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17ac0682f0 0x7f17ac06a7a0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f17c4000f60 tx=0x7f17c4007be0 comp rx=0 tx=0).stop 2023-03-03T17:05:50.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 -- 172.21.15.7:0/1302974328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 msgr2=0x7f17c814bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:50.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 0x7f17c814bc80 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f17b8013000 tx=0x7f17b8004750 comp rx=0 tx=0).stop 2023-03-03T17:05:50.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 -- 172.21.15.7:0/1302974328 shutdown_connections 2023-03-03T17:05:50.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f17c80c1ce0 0x7f17c81dedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c8071e80 0x7f17c81502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f17ac0682f0 0x7f17ac06a7a0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 --2- 172.21.15.7:0/1302974328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17c8072250 0x7f17c814bc80 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 -- 172.21.15.7:0/1302974328 >> 172.21.15.7:0/1302974328 conn(0x7f17c80a94d0 msgr2=0x7f17c806a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:05:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 -- 172.21.15.7:0/1302974328 shutdown_connections 2023-03-03T17:05:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:50.644+0000 7f17d07d8700 1 -- 172.21.15.7:0/1302974328 wait complete. 2023-03-03T17:05:50.646 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:05:50.656 INFO:tasks.mds_thrash.fs.[b]:waiting for 69 secs before thrashing 2023-03-03T17:05:53.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:53.055 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:05:59.189 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:05:59.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.324+0000 7f9bb4659700 1 Processor -- start 2023-03-03T17:05:59.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.325+0000 7f9bb4659700 1 -- start start 2023-03-03T17:05:59.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb4659700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9bac1529a0 0x7f9bac1592f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:59.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb4659700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac06e7e0 0x7f9bac06ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:59.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb4659700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bac0c6cb0 con 0x7f9bac0c64f0 2023-03-03T17:05:59.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb4659700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bac0c7ec0 con 0x7f9bac1529a0 2023-03-03T17:05:59.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb4659700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac0c7100 con 0x7f9bac06e7e0 2023-03-03T17:05:59.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb23f5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9bac1529a0 0x7f9bac1592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:59.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb23f5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9bac1529a0 0x7f9bac1592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59928/0 (socket says 172.21.15.7:59928) 2023-03-03T17:05:59.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb23f5700 1 -- 172.21.15.7:0/3573312929 learned_addr learned my addr 172.21.15.7:0/3573312929 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:05:59.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb2bf6700 1 --2- 172.21.15.7:0/3573312929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac06e7e0 0x7f9bac06ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:59.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb23f5700 1 -- 172.21.15.7:0/3573312929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac06e7e0 msgr2=0x7f9bac06ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:05:59.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb23f5700 1 --2- 172.21.15.7:0/3573312929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac06e7e0 0x7f9bac06ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:59.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.326+0000 7f9bb23f5700 1 -- 172.21.15.7:0/3573312929 >> v1:172.21.15.143:6789/0 conn(0x7f9bac0c64f0 legacy=0x7f9bac0caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:59.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.327+0000 7f9bb23f5700 1 -- 172.21.15.7:0/3573312929 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac0c73e0 con 0x7f9bac1529a0 2023-03-03T17:05:59.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.327+0000 7f9bb23f5700 1 --2- 172.21.15.7:0/3573312929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac1529a0 0x7f9bac1592f0 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0098f0 tx=0x7f9b9c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f2fe30be53f3fc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:59.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.327+0000 7f9bb13f3700 1 -- 172.21.15.7:0/3573312929 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c011830 con 0x7f9bac1529a0 2023-03-03T17:05:59.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.327+0000 7f9bb13f3700 1 -- 172.21.15.7:0/3573312929 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c011990 con 0x7f9bac1529a0 2023-03-03T17:05:59.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.327+0000 7f9bb13f3700 1 -- 172.21.15.7:0/3573312929 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c011d20 con 0x7f9bac1529a0 2023-03-03T17:05:59.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.327+0000 7f9bb4659700 1 -- 172.21.15.7:0/3573312929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac1529a0 msgr2=0x7f9bac1592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:59.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.327+0000 7f9bb4659700 1 --2- 172.21.15.7:0/3573312929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac1529a0 0x7f9bac1592f0 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0098f0 tx=0x7f9b9c009c00 comp rx=0 tx=0).stop 2023-03-03T17:05:59.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.328+0000 7f9bb4659700 1 -- 172.21.15.7:0/3573312929 shutdown_connections 2023-03-03T17:05:59.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.328+0000 7f9bb4659700 1 --2- 172.21.15.7:0/3573312929 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac06e7e0 0x7f9bac06ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:59.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.328+0000 7f9bb4659700 1 --2- 172.21.15.7:0/3573312929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac1529a0 0x7f9bac1592f0 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:59.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.328+0000 7f9bb4659700 1 -- 172.21.15.7:0/3573312929 >> 172.21.15.7:0/3573312929 conn(0x7f9bac066e40 msgr2=0x7f9bac071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:05:59.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.328+0000 7f9bb4659700 1 -- 172.21.15.7:0/3573312929 shutdown_connections 2023-03-03T17:05:59.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.328+0000 7f9bb4659700 1 -- 172.21.15.7:0/3573312929 wait complete. 2023-03-03T17:05:59.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.329+0000 7f9bb4659700 1 Processor -- start 2023-03-03T17:05:59.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.329+0000 7f9bb4659700 1 -- start start 2023-03-03T17:05:59.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.329+0000 7f9bb4659700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac06e7e0 0x7f9bac1e3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:59.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.329+0000 7f9bb4659700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac0c64f0 0x7f9bac1e3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:59.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.329+0000 7f9bb4659700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac1529a0 0x7f9bac14d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:59.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb4659700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1e5950 con 0x7f9bac0c64f0 2023-03-03T17:05:59.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb4659700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac15be70 con 0x7f9bac1529a0 2023-03-03T17:05:59.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb4659700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac1e2d80 con 0x7f9bac06e7e0 2023-03-03T17:05:59.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb1bf4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac0c64f0 0x7f9bac1e3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:59.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb1bf4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac0c64f0 0x7f9bac1e3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59936/0 (socket says 172.21.15.7:59936) 2023-03-03T17:05:59.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb1bf4700 1 -- 172.21.15.7:0/4195021313 learned_addr learned my addr 172.21.15.7:0/4195021313 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:05:59.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb2bf6700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac1529a0 0x7f9bac14d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:59.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb23f5700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac06e7e0 0x7f9bac1e3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:59.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb1bf4700 1 -- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac06e7e0 msgr2=0x7f9bac1e3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:59.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb1bf4700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac06e7e0 0x7f9bac1e3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:59.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb1bf4700 1 -- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac1529a0 msgr2=0x7f9bac14d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:59.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb1bf4700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac1529a0 0x7f9bac14d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:59.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb1bf4700 1 -- 172.21.15.7:0/4195021313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c009550 con 0x7f9bac0c64f0 2023-03-03T17:05:59.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb2bf6700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac1529a0 0x7f9bac14d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:05:59.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb1bf4700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac0c64f0 0x7f9bac1e3bb0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400cc50 tx=0x7f9ba400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:59.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9ba2ffd700 1 -- 172.21.15.7:0/4195021313 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba400ada0 con 0x7f9bac0c64f0 2023-03-03T17:05:59.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb4659700 1 -- 172.21.15.7:0/4195021313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac0c79a0 con 0x7f9bac0c64f0 2023-03-03T17:05:59.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9ba2ffd700 1 -- 172.21.15.7:0/4195021313 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba400ce50 con 0x7f9bac0c64f0 2023-03-03T17:05:59.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9ba2ffd700 1 -- 172.21.15.7:0/4195021313 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4003c90 con 0x7f9bac0c64f0 2023-03-03T17:05:59.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.330+0000 7f9bb4659700 1 -- 172.21.15.7:0/4195021313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac0c76c0 con 0x7f9bac0c64f0 2023-03-03T17:05:59.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.331+0000 7f9bb4659700 1 -- 172.21.15.7:0/4195021313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac0c6940 con 0x7f9bac0c64f0 2023-03-03T17:05:59.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.332+0000 7f9ba2ffd700 1 -- 172.21.15.7:0/4195021313 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9ba4010050 con 0x7f9bac0c64f0 2023-03-03T17:05:59.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.332+0000 7f9ba2ffd700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 0x7f9b9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:05:59.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.332+0000 7f9ba2ffd700 1 -- 172.21.15.7:0/4195021313 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9ba40861b0 con 0x7f9bac0c64f0 2023-03-03T17:05:59.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.332+0000 7f9bb23f5700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 0x7f9b9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:05:59.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.333+0000 7f9bb23f5700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 0x7f9b9006a7a0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c014040 tx=0x7f9b9c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:05:59.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.336+0000 7f9ba2ffd700 1 -- 172.21.15.7:0/4195021313 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9ba4052470 con 0x7f9bac0c64f0 2023-03-03T17:05:59.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.577+0000 7f9bb4659700 1 -- 172.21.15.7:0/4195021313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bac15bfb0 con 0x7f9bac0c64f0 2023-03-03T17:05:59.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.578+0000 7f9ba2ffd700 1 -- 172.21.15.7:0/4195021313 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9ba4055a90 con 0x7f9bac0c64f0 2023-03-03T17:05:59.578 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:05:59.578 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:05:59.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 -- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 msgr2=0x7f9b9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:59.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 0x7f9b9006a7a0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c014040 tx=0x7f9b9c008030 comp rx=0 tx=0).stop 2023-03-03T17:05:59.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 -- 172.21.15.7:0/4195021313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac0c64f0 msgr2=0x7f9bac1e3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:05:59.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac0c64f0 0x7f9bac1e3bb0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400cc50 tx=0x7f9ba400a340 comp rx=0 tx=0).stop 2023-03-03T17:05:59.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 -- 172.21.15.7:0/4195021313 shutdown_connections 2023-03-03T17:05:59.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9bac1529a0 0x7f9bac14d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:59.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9bac06e7e0 0x7f9bac1e3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:59.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9b900682f0 0x7f9b9006a7a0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:59.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 --2- 172.21.15.7:0/4195021313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9bac0c64f0 0x7f9bac1e3bb0 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:05:59.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 -- 172.21.15.7:0/4195021313 >> 172.21.15.7:0/4195021313 conn(0x7f9bac066e40 msgr2=0x7f9bac0c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:05:59.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 -- 172.21.15.7:0/4195021313 shutdown_connections 2023-03-03T17:05:59.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:05:59.580+0000 7f9bb4659700 1 -- 172.21.15.7:0/4195021313 wait complete. 2023-03-03T17:05:59.582 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:05:59.592 INFO:tasks.mds_thrash.fs.[a]:waiting for 112 secs before thrashing 2023-03-03T17:06:23.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:23.128 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:53.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:53.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:06:59.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:06:59.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.797+0000 7f0783dc4700 1 Processor -- start 2023-03-03T17:06:59.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.797+0000 7f0783dc4700 1 -- start start 2023-03-03T17:06:59.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0783dc4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f077c06d870 0x7f077c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:59.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0783dc4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f077c0c1da0 0x7f077c0c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:59.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0783dc4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077c0c3620 con 0x7f077c0c3e30 2023-03-03T17:06:59.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0783dc4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077c0c2af0 con 0x7f077c06d870 2023-03-03T17:06:59.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0783dc4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077c1545e0 con 0x7f077c0c1da0 2023-03-03T17:06:59.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0781b60700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f077c06d870 0x7f077c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:59.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0781b60700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f077c06d870 0x7f077c1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59990/0 (socket says 172.21.15.7:59990) 2023-03-03T17:06:59.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0781b60700 1 -- 172.21.15.7:0/1041097150 learned_addr learned my addr 172.21.15.7:0/1041097150 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:06:59.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f078135f700 1 --2- 172.21.15.7:0/1041097150 >> v2:172.21.15.143:3300/0 conn(0x7f077c0c1da0 0x7f077c0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:59.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0781b60700 1 -- 172.21.15.7:0/1041097150 >> v2:172.21.15.143:3300/0 conn(0x7f077c0c1da0 msgr2=0x7f077c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:06:59.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0781b60700 1 --2- 172.21.15.7:0/1041097150 >> v2:172.21.15.143:3300/0 conn(0x7f077c0c1da0 0x7f077c0c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:59.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0781b60700 1 -- 172.21.15.7:0/1041097150 >> v1:172.21.15.143:6789/0 conn(0x7f077c0c3e30 legacy=0x7f077c157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:06:59.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0781b60700 1 -- 172.21.15.7:0/1041097150 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077c1548c0 con 0x7f077c06d870 2023-03-03T17:06:59.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.798+0000 7f0781b60700 1 --2- 172.21.15.7:0/1041097150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c06d870 0x7f077c1550c0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f077c0c33a0 tx=0x7f076c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78603ff90c90acb2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:06:59.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0780b5e700 1 -- 172.21.15.7:0/1041097150 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c00b870 con 0x7f077c06d870 2023-03-03T17:06:59.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0780b5e700 1 -- 172.21.15.7:0/1041097150 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c00b9d0 con 0x7f077c06d870 2023-03-03T17:06:59.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0780b5e700 1 -- 172.21.15.7:0/1041097150 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c00bd60 con 0x7f077c06d870 2023-03-03T17:06:59.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0783dc4700 1 -- 172.21.15.7:0/1041097150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c06d870 msgr2=0x7f077c1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:06:59.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0783dc4700 1 --2- 172.21.15.7:0/1041097150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c06d870 0x7f077c1550c0 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f077c0c33a0 tx=0x7f076c009bd0 comp rx=0 tx=0).stop 2023-03-03T17:06:59.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0783dc4700 1 -- 172.21.15.7:0/1041097150 shutdown_connections 2023-03-03T17:06:59.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0783dc4700 1 --2- 172.21.15.7:0/1041097150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c06d870 0x7f077c1550c0 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:59.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0783dc4700 1 --2- 172.21.15.7:0/1041097150 >> v2:172.21.15.143:3300/0 conn(0x7f077c0c1da0 0x7f077c0c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:59.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0783dc4700 1 -- 172.21.15.7:0/1041097150 >> 172.21.15.7:0/1041097150 conn(0x7f077c06c110 msgr2=0x7f077c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:06:59.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0783dc4700 1 -- 172.21.15.7:0/1041097150 shutdown_connections 2023-03-03T17:06:59.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.799+0000 7f0783dc4700 1 -- 172.21.15.7:0/1041097150 wait complete. 2023-03-03T17:06:59.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.800+0000 7f0783dc4700 1 Processor -- start 2023-03-03T17:06:59.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.800+0000 7f0783dc4700 1 -- start start 2023-03-03T17:06:59.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.801+0000 7f0783dc4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f077c06d870 0x7f077c1ed970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.801+0000 7f0783dc4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c0c1da0 0x7f077c158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.801+0000 7f0783dc4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f077c0c3e30 0x7f077c0c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.801+0000 7f0783dc4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c158a40 con 0x7f077c0c1da0 2023-03-03T17:06:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.801+0000 7f0783dc4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c155f90 con 0x7f077c0c3e30 2023-03-03T17:06:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.801+0000 7f0783dc4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f077c147c10 con 0x7f077c06d870 2023-03-03T17:06:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.801+0000 7f078135f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c0c1da0 0x7f077c158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.801+0000 7f078135f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c0c1da0 0x7f077c158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59998/0 (socket says 172.21.15.7:59998) 2023-03-03T17:06:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.801+0000 7f078135f700 1 -- 172.21.15.7:0/1372696572 learned_addr learned my addr 172.21.15.7:0/1372696572 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:06:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.801+0000 7f0781b60700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f077c06d870 0x7f077c1ed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f0782361700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f077c0c3e30 0x7f077c0c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f078135f700 1 -- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f077c06d870 msgr2=0x7f077c1ed970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:06:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f078135f700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f077c06d870 0x7f077c1ed970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f078135f700 1 -- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f077c0c3e30 msgr2=0x7f077c0c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:06:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f078135f700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f077c0c3e30 0x7f077c0c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:06:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f078135f700 1 -- 172.21.15.7:0/1372696572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c009550 con 0x7f077c0c1da0 2023-03-03T17:06:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f078135f700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c0c1da0 0x7f077c158500 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f077800e810 tx=0x7f077800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:06:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f07727fc700 1 -- 172.21.15.7:0/1372696572 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0778013610 con 0x7f077c0c1da0 2023-03-03T17:06:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f07727fc700 1 -- 172.21.15.7:0/1372696572 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0778013770 con 0x7f077c0c1da0 2023-03-03T17:06:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f0783dc4700 1 -- 172.21.15.7:0/1372696572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077c06dc40 con 0x7f077c0c1da0 2023-03-03T17:06:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f07727fc700 1 -- 172.21.15.7:0/1372696572 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0778013b20 con 0x7f077c0c1da0 2023-03-03T17:06:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.802+0000 7f0783dc4700 1 -- 172.21.15.7:0/1372696572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077c154ba0 con 0x7f077c0c1da0 2023-03-03T17:06:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.803+0000 7f0783dc4700 1 -- 172.21.15.7:0/1372696572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077c0c38e0 con 0x7f077c0c1da0 2023-03-03T17:06:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.803+0000 7f07727fc700 1 -- 172.21.15.7:0/1372696572 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0778023070 con 0x7f077c0c1da0 2023-03-03T17:06:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.804+0000 7f07727fc700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07600682f0 0x7f076006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:06:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.804+0000 7f07727fc700 1 -- 172.21.15.7:0/1372696572 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0778085e30 con 0x7f077c0c1da0 2023-03-03T17:06:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.804+0000 7f0781b60700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07600682f0 0x7f076006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:06:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.805+0000 7f0781b60700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07600682f0 0x7f076006a7a0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f076c000c00 tx=0x7f076c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:06:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:06:59.808+0000 7f07727fc700 1 -- 172.21.15.7:0/1372696572 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0778052150 con 0x7f077c0c1da0 2023-03-03T17:07:00.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.046+0000 7f0783dc4700 1 -- 172.21.15.7:0/1372696572 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f077c074bb0 con 0x7f077c0c1da0 2023-03-03T17:07:00.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.046+0000 7f07727fc700 1 -- 172.21.15.7:0/1372696572 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0778055770 con 0x7f077c0c1da0 2023-03-03T17:07:00.046 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:07:00.047 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:07:00.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 -- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07600682f0 msgr2=0x7f076006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:00.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07600682f0 0x7f076006a7a0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f076c000c00 tx=0x7f076c008030 comp rx=0 tx=0).stop 2023-03-03T17:07:00.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 -- 172.21.15.7:0/1372696572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c0c1da0 msgr2=0x7f077c158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:00.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c0c1da0 0x7f077c158500 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f077800e810 tx=0x7f077800ebd0 comp rx=0 tx=0).stop 2023-03-03T17:07:00.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 -- 172.21.15.7:0/1372696572 shutdown_connections 2023-03-03T17:07:00.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f077c0c3e30 0x7f077c0c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:00.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f077c06d870 0x7f077c1ed970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:00.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07600682f0 0x7f076006a7a0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:00.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 --2- 172.21.15.7:0/1372696572 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f077c0c1da0 0x7f077c158500 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:00.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 -- 172.21.15.7:0/1372696572 >> 172.21.15.7:0/1372696572 conn(0x7f077c06c110 msgr2=0x7f077c0c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:07:00.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 -- 172.21.15.7:0/1372696572 shutdown_connections 2023-03-03T17:07:00.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:00.048+0000 7f0783dc4700 1 -- 172.21.15.7:0/1372696572 wait complete. 2023-03-03T17:07:00.050 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:07:00.060 INFO:tasks.mds_thrash.fs.[b]:waiting for 36 secs before thrashing 2023-03-03T17:07:23.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:23.295 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:36.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:07:36.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.201+0000 7f023ba61700 1 Processor -- start 2023-03-03T17:07:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.202+0000 7f023ba61700 1 -- start start 2023-03-03T17:07:36.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.202+0000 7f023ba61700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02340c4850 0x7f0234154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:36.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f023ba61700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0234072920 con 0x7f02340c43d0 2023-03-03T17:07:36.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f023ba61700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023406bef0 con 0x7f023414e560 2023-03-03T17:07:36.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f023ba61700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0234072380 con 0x7f02340c4850 2023-03-03T17:07:36.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0239ffe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f02340c4850 0x7f0234154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0238ffc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f02340c43d0 0x7f0234154590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54006/0 (socket says 172.21.15.7:54006) 2023-03-03T17:07:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0238ffc700 1 -- 172.21.15.7:0/1070968225 learned_addr learned my addr 172.21.15.7:0/1070968225 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:07:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0239ffe700 1 -- 172.21.15.7:0/1070968225 >> v1:172.21.15.143:6789/0 conn(0x7f023414e560 legacy=0x7f0234071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0239ffe700 1 -- 172.21.15.7:0/1070968225 >> v1:172.21.15.7:6789/0 conn(0x7f02340c43d0 legacy=0x7f0234154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0239ffe700 1 -- 172.21.15.7:0/1070968225 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02341551e0 con 0x7f02340c4850 2023-03-03T17:07:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0227fff700 1 -- 172.21.15.7:0/1070968225 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503044886 0 0) 0x7f0230001110 con 0x7f02340c43d0 2023-03-03T17:07:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0239ffe700 1 --2- 172.21.15.7:0/1070968225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02340c4850 0x7f0234154ca0 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f022c009d90 tx=0x7f022c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396c4c9d84837c3a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:36.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0227fff700 1 -- 172.21.15.7:0/1070968225 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c00e070 con 0x7f02340c4850 2023-03-03T17:07:36.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0227fff700 1 -- 172.21.15.7:0/1070968225 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022c00bbd0 con 0x7f02340c4850 2023-03-03T17:07:36.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.203+0000 7f0227fff700 1 -- 172.21.15.7:0/1070968225 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c012430 con 0x7f02340c4850 2023-03-03T17:07:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.204+0000 7f023ba61700 1 -- 172.21.15.7:0/1070968225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02340c4850 msgr2=0x7f0234154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.204+0000 7f023ba61700 1 --2- 172.21.15.7:0/1070968225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02340c4850 0x7f0234154ca0 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f022c009d90 tx=0x7f022c00b330 comp rx=0 tx=0).stop 2023-03-03T17:07:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.204+0000 7f023ba61700 1 -- 172.21.15.7:0/1070968225 shutdown_connections 2023-03-03T17:07:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.204+0000 7f023ba61700 1 --2- 172.21.15.7:0/1070968225 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02340c4850 0x7f0234154ca0 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.204+0000 7f023ba61700 1 -- 172.21.15.7:0/1070968225 >> 172.21.15.7:0/1070968225 conn(0x7f02340736f0 msgr2=0x7f0234073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:07:36.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.204+0000 7f023ba61700 1 -- 172.21.15.7:0/1070968225 shutdown_connections 2023-03-03T17:07:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.204+0000 7f023ba61700 1 -- 172.21.15.7:0/1070968225 wait complete. 2023-03-03T17:07:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.205+0000 7f023ba61700 1 Processor -- start 2023-03-03T17:07:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.205+0000 7f023ba61700 1 -- start start 2023-03-03T17:07:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f023ba61700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02340c43d0 0x7f02340c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f023ba61700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02340c4850 0x7f02340be7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f023ba61700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023414e560 0x7f023414f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:36.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f023ba61700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02341edbc0 con 0x7f023414e560 2023-03-03T17:07:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f023ba61700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02341ee0f0 con 0x7f02340c4850 2023-03-03T17:07:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f023ba61700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02341ede10 con 0x7f02340c43d0 2023-03-03T17:07:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f0239ffe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023414e560 0x7f023414f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f0239ffe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023414e560 0x7f023414f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60032/0 (socket says 172.21.15.7:60032) 2023-03-03T17:07:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f0239ffe700 1 -- 172.21.15.7:0/859228177 learned_addr learned my addr 172.21.15.7:0/859228177 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:07:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f0238ffc700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02340c4850 0x7f02340be7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f02397fd700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02340c43d0 0x7f02340c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:36.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f0239ffe700 1 -- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02340c43d0 msgr2=0x7f02340c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f0239ffe700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02340c43d0 0x7f02340c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f0239ffe700 1 -- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02340c4850 msgr2=0x7f02340be7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.206+0000 7f0239ffe700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02340c4850 0x7f02340be7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.207+0000 7f0239ffe700 1 -- 172.21.15.7:0/859228177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c009a70 con 0x7f023414e560 2023-03-03T17:07:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.207+0000 7f0239ffe700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023414e560 0x7f023414f540 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f022c014040 tx=0x7f022c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.207+0000 7f0225ffb700 1 -- 172.21.15.7:0/859228177 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c01b810 con 0x7f023414e560 2023-03-03T17:07:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.207+0000 7f0225ffb700 1 -- 172.21.15.7:0/859228177 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022c020070 con 0x7f023414e560 2023-03-03T17:07:36.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.207+0000 7f023ba61700 1 -- 172.21.15.7:0/859228177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02340c4c70 con 0x7f023414e560 2023-03-03T17:07:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.207+0000 7f0225ffb700 1 -- 172.21.15.7:0/859228177 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c01bbd0 con 0x7f023414e560 2023-03-03T17:07:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.207+0000 7f023ba61700 1 -- 172.21.15.7:0/859228177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023406b910 con 0x7f023414e560 2023-03-03T17:07:36.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.208+0000 7f023ba61700 1 -- 172.21.15.7:0/859228177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02340aacc0 con 0x7f023414e560 2023-03-03T17:07:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.208+0000 7f0225ffb700 1 -- 172.21.15.7:0/859228177 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f022c02e070 con 0x7f023414e560 2023-03-03T17:07:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.209+0000 7f0225ffb700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02180682f0 0x7f021806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.209+0000 7f0225ffb700 1 -- 172.21.15.7:0/859228177 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f022c022070 con 0x7f023414e560 2023-03-03T17:07:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.209+0000 7f02397fd700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02180682f0 0x7f021806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:36.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.210+0000 7f02397fd700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02180682f0 0x7f021806a7a0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f0228007fd0 tx=0x7f0228009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:36.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.213+0000 7f0225ffb700 1 -- 172.21.15.7:0/859228177 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f022c0545d0 con 0x7f023414e560 2023-03-03T17:07:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.453+0000 7f023ba61700 1 -- 172.21.15.7:0/859228177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02341511e0 con 0x7f023414e560 2023-03-03T17:07:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.453+0000 7f0225ffb700 1 -- 172.21.15.7:0/859228177 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f022c057bf0 con 0x7f023414e560 2023-03-03T17:07:36.454 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:07:36.455 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:07:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 -- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02180682f0 msgr2=0x7f021806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02180682f0 0x7f021806a7a0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f0228007fd0 tx=0x7f0228009040 comp rx=0 tx=0).stop 2023-03-03T17:07:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 -- 172.21.15.7:0/859228177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023414e560 msgr2=0x7f023414f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023414e560 0x7f023414f540 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f022c014040 tx=0x7f022c004ad0 comp rx=0 tx=0).stop 2023-03-03T17:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 -- 172.21.15.7:0/859228177 shutdown_connections 2023-03-03T17:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02180682f0 0x7f021806a7a0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023414e560 0x7f023414f540 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02340c43d0 0x7f02340c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 --2- 172.21.15.7:0/859228177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02340c4850 0x7f02340be7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 -- 172.21.15.7:0/859228177 >> 172.21.15.7:0/859228177 conn(0x7f02340736f0 msgr2=0x7f0234073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 -- 172.21.15.7:0/859228177 shutdown_connections 2023-03-03T17:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:36.456+0000 7f023ba61700 1 -- 172.21.15.7:0/859228177 wait complete. 2023-03-03T17:07:36.458 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:07:36.468 INFO:tasks.mds_thrash.fs.[b]:waiting for 88 secs before thrashing 2023-03-03T17:07:51.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:07:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.733+0000 7f9c222e0700 1 Processor -- start 2023-03-03T17:07:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.734+0000 7f9c222e0700 1 -- start start 2023-03-03T17:07:51.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.734+0000 7f9c222e0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9c1c0c2930 0x7f9c1c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:51.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.734+0000 7f9c222e0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c1c06c480 0x7f9c1c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:51.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.734+0000 7f9c222e0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c06d430 con 0x7f9c1c151d40 2023-03-03T17:07:51.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.734+0000 7f9c222e0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1c06c2a0 con 0x7f9c1c06c480 2023-03-03T17:07:51.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.734+0000 7f9c222e0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1c06cb00 con 0x7f9c1c0c2930 2023-03-03T17:07:51.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.734+0000 7f9c2087d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9c1c06c480 0x7f9c1c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:51.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.734+0000 7f9c1bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9c1c151d40 0x7f9c1c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54034/0 (socket says 172.21.15.7:54034) 2023-03-03T17:07:51.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.734+0000 7f9c1bfff700 1 -- 172.21.15.7:0/3097657452 learned_addr learned my addr 172.21.15.7:0/3097657452 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:07:51.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.735+0000 7f9c1b7fe700 1 --2- 172.21.15.7:0/3097657452 >> v2:172.21.15.143:3300/0 conn(0x7f9c1c0c2930 0x7f9c1c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.735+0000 7f9c2087d700 1 -- 172.21.15.7:0/3097657452 >> v2:172.21.15.143:3300/0 conn(0x7f9c1c0c2930 msgr2=0x7f9c1c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:07:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.735+0000 7f9c2087d700 1 --2- 172.21.15.7:0/3097657452 >> v2:172.21.15.143:3300/0 conn(0x7f9c1c0c2930 0x7f9c1c157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.735+0000 7f9c2087d700 1 -- 172.21.15.7:0/3097657452 >> v1:172.21.15.7:6789/0 conn(0x7f9c1c151d40 legacy=0x7f9c1c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.735+0000 7f9c2087d700 1 -- 172.21.15.7:0/3097657452 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c06cde0 con 0x7f9c1c06c480 2023-03-03T17:07:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.735+0000 7f9c2087d700 1 --2- 172.21.15.7:0/3097657452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c06c480 0x7f9c1c157fd0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f9c14009d90 tx=0x7f9c1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3bbf3d3022d57a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.735+0000 7f9c1affd700 1 -- 172.21.15.7:0/3097657452 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1400e070 con 0x7f9c1c06c480 2023-03-03T17:07:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.735+0000 7f9c1affd700 1 -- 172.21.15.7:0/3097657452 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1400bae0 con 0x7f9c1c06c480 2023-03-03T17:07:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.735+0000 7f9c1affd700 1 -- 172.21.15.7:0/3097657452 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1400be90 con 0x7f9c1c06c480 2023-03-03T17:07:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.736+0000 7f9c222e0700 1 -- 172.21.15.7:0/3097657452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c06c480 msgr2=0x7f9c1c157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.736+0000 7f9c222e0700 1 --2- 172.21.15.7:0/3097657452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c06c480 0x7f9c1c157fd0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f9c14009d90 tx=0x7f9c1400b330 comp rx=0 tx=0).stop 2023-03-03T17:07:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.736+0000 7f9c222e0700 1 -- 172.21.15.7:0/3097657452 shutdown_connections 2023-03-03T17:07:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.736+0000 7f9c222e0700 1 --2- 172.21.15.7:0/3097657452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c06c480 0x7f9c1c157fd0 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.736+0000 7f9c222e0700 1 --2- 172.21.15.7:0/3097657452 >> v2:172.21.15.143:3300/0 conn(0x7f9c1c0c2930 0x7f9c1c157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.736+0000 7f9c222e0700 1 -- 172.21.15.7:0/3097657452 >> 172.21.15.7:0/3097657452 conn(0x7f9c1c0a94d0 msgr2=0x7f9c1c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:07:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.736+0000 7f9c222e0700 1 -- 172.21.15.7:0/3097657452 shutdown_connections 2023-03-03T17:07:51.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.736+0000 7f9c222e0700 1 -- 172.21.15.7:0/3097657452 wait complete. 2023-03-03T17:07:51.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.737+0000 7f9c222e0700 1 Processor -- start 2023-03-03T17:07:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.737+0000 7f9c222e0700 1 -- start start 2023-03-03T17:07:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c222e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c1c06c480 0x7f9c1c0c0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c222e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1c0c2930 0x7f9c1c0be430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c222e0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c151d40 0x7f9c1c0be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c222e0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c1ee140 con 0x7f9c1c151d40 2023-03-03T17:07:51.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c222e0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c1ed380 con 0x7f9c1c06c480 2023-03-03T17:07:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c222e0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c1ed660 con 0x7f9c1c0c2930 2023-03-03T17:07:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c2087d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c151d40 0x7f9c1c0be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c2087d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c151d40 0x7f9c1c0be970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60064/0 (socket says 172.21.15.7:60064) 2023-03-03T17:07:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c2087d700 1 -- 172.21.15.7:0/3343836128 learned_addr learned my addr 172.21.15.7:0/3343836128 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:07:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c1bfff700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c1c06c480 0x7f9c1c0c0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c1b7fe700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1c0c2930 0x7f9c1c0be430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c2087d700 1 -- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1c0c2930 msgr2=0x7f9c1c0be430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:51.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c2087d700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1c0c2930 0x7f9c1c0be430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c2087d700 1 -- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c1c06c480 msgr2=0x7f9c1c0c0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c2087d700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c1c06c480 0x7f9c1c0c0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c2087d700 1 -- 172.21.15.7:0/3343836128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c14009a70 con 0x7f9c1c151d40 2023-03-03T17:07:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.738+0000 7f9c1b7fe700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1c0c2930 0x7f9c1c0be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:07:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.739+0000 7f9c2087d700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c151d40 0x7f9c1c0be970 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f9c14014040 tx=0x7f9c14004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.739+0000 7f9c18ff9700 1 -- 172.21.15.7:0/3343836128 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c14004680 con 0x7f9c1c151d40 2023-03-03T17:07:51.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.739+0000 7f9c18ff9700 1 -- 172.21.15.7:0/3343836128 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c14020070 con 0x7f9c1c151d40 2023-03-03T17:07:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.739+0000 7f9c222e0700 1 -- 172.21.15.7:0/3343836128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1c073e00 con 0x7f9c1c151d40 2023-03-03T17:07:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.739+0000 7f9c18ff9700 1 -- 172.21.15.7:0/3343836128 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c14027480 con 0x7f9c1c151d40 2023-03-03T17:07:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.739+0000 7f9c222e0700 1 -- 172.21.15.7:0/3343836128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1c06d0c0 con 0x7f9c1c151d40 2023-03-03T17:07:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.740+0000 7f9c222e0700 1 -- 172.21.15.7:0/3343836128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1c06bea0 con 0x7f9c1c151d40 2023-03-03T17:07:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.740+0000 7f9c18ff9700 1 -- 172.21.15.7:0/3343836128 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9c14012430 con 0x7f9c1c151d40 2023-03-03T17:07:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.741+0000 7f9c18ff9700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c080682f0 0x7f9c0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:07:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.741+0000 7f9c18ff9700 1 -- 172.21.15.7:0/3343836128 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9c14022070 con 0x7f9c1c151d40 2023-03-03T17:07:51.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.741+0000 7f9c1bfff700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c080682f0 0x7f9c0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:07:51.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.741+0000 7f9c1bfff700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c080682f0 0x7f9c0806a7a0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c005910 tx=0x7f9c0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:07:51.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.745+0000 7f9c18ff9700 1 -- 172.21.15.7:0/3343836128 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9c140b95e0 con 0x7f9c1c151d40 2023-03-03T17:07:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.987+0000 7f9c222e0700 1 -- 172.21.15.7:0/3343836128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c1c06df40 con 0x7f9c1c151d40 2023-03-03T17:07:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.987+0000 7f9c18ff9700 1 -- 172.21.15.7:0/3343836128 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9c14057ef0 con 0x7f9c1c151d40 2023-03-03T17:07:51.988 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:07:51.988 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:07:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 -- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c080682f0 msgr2=0x7f9c0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c080682f0 0x7f9c0806a7a0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c005910 tx=0x7f9c0c009040 comp rx=0 tx=0).stop 2023-03-03T17:07:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 -- 172.21.15.7:0/3343836128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c151d40 msgr2=0x7f9c1c0be970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:07:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c151d40 0x7f9c1c0be970 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f9c14014040 tx=0x7f9c14004ad0 comp rx=0 tx=0).stop 2023-03-03T17:07:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 -- 172.21.15.7:0/3343836128 shutdown_connections 2023-03-03T17:07:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9c080682f0 0x7f9c0806a7a0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c1c151d40 0x7f9c1c0be970 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:51.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c1c06c480 0x7f9c1c0c0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:51.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 --2- 172.21.15.7:0/3343836128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9c1c0c2930 0x7f9c1c0be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:07:51.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 -- 172.21.15.7:0/3343836128 >> 172.21.15.7:0/3343836128 conn(0x7f9c1c0a94d0 msgr2=0x7f9c1c14f430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:07:51.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.989+0000 7f9c222e0700 1 -- 172.21.15.7:0/3343836128 shutdown_connections 2023-03-03T17:07:51.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:07:51.990+0000 7f9c222e0700 1 -- 172.21.15.7:0/3343836128 wait complete. 2023-03-03T17:07:51.994 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:07:52.002 INFO:tasks.mds_thrash.fs.[a]:waiting for 30 secs before thrashing 2023-03-03T17:07:53.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:07:53.367 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:22.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:08:22.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.142+0000 7fae235c8700 1 Processor -- start 2023-03-03T17:08:22.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.142+0000 7fae235c8700 1 -- start start 2023-03-03T17:08:22.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.142+0000 7fae235c8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae1c151d40 0x7fae1c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.142+0000 7fae235c8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae1c0c2930 0x7fae1c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.142+0000 7fae235c8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae1c073c30 0x7fae1c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:22.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.142+0000 7fae235c8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1c06c2a0 con 0x7fae1c0c2930 2023-03-03T17:08:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.142+0000 7fae235c8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1c06d430 con 0x7fae1c073c30 2023-03-03T17:08:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae235c8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c06cb00 con 0x7fae1c151d40 2023-03-03T17:08:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae20b63700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae1c0c2930 0x7fae1c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:22.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae20b63700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae1c0c2930 0x7fae1c157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60074/0 (socket says 172.21.15.7:60074) 2023-03-03T17:08:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae20b63700 1 -- 172.21.15.7:0/263794102 learned_addr learned my addr 172.21.15.7:0/263794102 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:08:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae20b63700 1 -- 172.21.15.7:0/263794102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae1c151d40 msgr2=0x7fae1c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:08:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae20b63700 1 --2- 172.21.15.7:0/263794102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae1c151d40 0x7fae1c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae20b63700 1 -- 172.21.15.7:0/263794102 >> v2:172.21.15.143:3300/0 conn(0x7fae1c073c30 msgr2=0x7fae1c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:08:22.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae20b63700 1 --2- 172.21.15.7:0/263794102 >> v2:172.21.15.143:3300/0 conn(0x7fae1c073c30 0x7fae1c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae20b63700 1 -- 172.21.15.7:0/263794102 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c06cde0 con 0x7fae1c0c2930 2023-03-03T17:08:22.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae20b63700 1 --2- 172.21.15.7:0/263794102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c0c2930 0x7fae1c157a90 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0c3db0 tx=0x7fae10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32a8f4ab1a3e3b7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:08:22.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae0ffff700 1 -- 172.21.15.7:0/263794102 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1000b870 con 0x7fae1c0c2930 2023-03-03T17:08:22.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae0ffff700 1 -- 172.21.15.7:0/263794102 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1000b9d0 con 0x7fae1c0c2930 2023-03-03T17:08:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.143+0000 7fae0ffff700 1 -- 172.21.15.7:0/263794102 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1000bd60 con 0x7fae1c0c2930 2023-03-03T17:08:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.144+0000 7fae235c8700 1 -- 172.21.15.7:0/263794102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c0c2930 msgr2=0x7fae1c157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.144+0000 7fae235c8700 1 --2- 172.21.15.7:0/263794102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c0c2930 0x7fae1c157a90 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0c3db0 tx=0x7fae10009bd0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.144+0000 7fae235c8700 1 -- 172.21.15.7:0/263794102 shutdown_connections 2023-03-03T17:08:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.144+0000 7fae235c8700 1 --2- 172.21.15.7:0/263794102 >> v2:172.21.15.143:3300/0 conn(0x7fae1c073c30 0x7fae1c157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.144+0000 7fae235c8700 1 --2- 172.21.15.7:0/263794102 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae1c151d40 0x7fae1c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.144+0000 7fae235c8700 1 --2- 172.21.15.7:0/263794102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c0c2930 0x7fae1c157a90 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.144+0000 7fae235c8700 1 -- 172.21.15.7:0/263794102 >> 172.21.15.7:0/263794102 conn(0x7fae1c0a94d0 msgr2=0x7fae1c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:08:22.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.144+0000 7fae235c8700 1 -- 172.21.15.7:0/263794102 shutdown_connections 2023-03-03T17:08:22.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.144+0000 7fae235c8700 1 -- 172.21.15.7:0/263794102 wait complete. 2023-03-03T17:08:22.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.145+0000 7fae235c8700 1 Processor -- start 2023-03-03T17:08:22.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.145+0000 7fae235c8700 1 -- start start 2023-03-03T17:08:22.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae235c8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae1c073c30 0x7fae1c145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:22.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae235c8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae1c0c2930 0x7fae1c1458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:22.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae235c8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c151d40 0x7fae1c1e0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:22.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae235c8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c1e04b0 con 0x7fae1c151d40 2023-03-03T17:08:22.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae235c8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c0bea90 con 0x7fae1c073c30 2023-03-03T17:08:22.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae235c8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c1ee4c0 con 0x7fae1c0c2930 2023-03-03T17:08:22.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae21b65700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c151d40 0x7fae1c1e0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:22.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae21b65700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c151d40 0x7fae1c1e0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60084/0 (socket says 172.21.15.7:60084) 2023-03-03T17:08:22.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae21b65700 1 -- 172.21.15.7:0/4034705686 learned_addr learned my addr 172.21.15.7:0/4034705686 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:08:22.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae21364700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae1c073c30 0x7fae1c145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:22.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae20b63700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae1c0c2930 0x7fae1c1458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae21b65700 1 -- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae1c0c2930 msgr2=0x7fae1c1458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae21b65700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae1c0c2930 0x7fae1c1458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae21b65700 1 -- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae1c073c30 msgr2=0x7fae1c145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae21b65700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae1c073c30 0x7fae1c145370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae21b65700 1 -- 172.21.15.7:0/4034705686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae10009550 con 0x7fae1c151d40 2023-03-03T17:08:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.146+0000 7fae21b65700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c151d40 0x7fae1c1e0840 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b970 tx=0x7fae1400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:08:22.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.147+0000 7fae0dffb700 1 -- 172.21.15.7:0/4034705686 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1400d860 con 0x7fae1c151d40 2023-03-03T17:08:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.147+0000 7fae0dffb700 1 -- 172.21.15.7:0/4034705686 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1400d9e0 con 0x7fae1c151d40 2023-03-03T17:08:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.147+0000 7fae235c8700 1 -- 172.21.15.7:0/4034705686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1c06bea0 con 0x7fae1c151d40 2023-03-03T17:08:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.147+0000 7fae0dffb700 1 -- 172.21.15.7:0/4034705686 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1400dd90 con 0x7fae1c151d40 2023-03-03T17:08:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.147+0000 7fae235c8700 1 -- 172.21.15.7:0/4034705686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1c06d0c0 con 0x7fae1c151d40 2023-03-03T17:08:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.148+0000 7fae235c8700 1 -- 172.21.15.7:0/4034705686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1c06c030 con 0x7fae1c151d40 2023-03-03T17:08:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.148+0000 7fae0dffb700 1 -- 172.21.15.7:0/4034705686 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae1400e430 con 0x7fae1c151d40 2023-03-03T17:08:22.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.148+0000 7fae0dffb700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae000682f0 0x7fae0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.149+0000 7fae0dffb700 1 -- 172.21.15.7:0/4034705686 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fae14006e40 con 0x7fae1c151d40 2023-03-03T17:08:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.149+0000 7fae21364700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae000682f0 0x7fae0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.149+0000 7fae21364700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae000682f0 0x7fae0006a7a0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fae1c155630 tx=0x7fae18009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:08:22.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.153+0000 7fae0dffb700 1 -- 172.21.15.7:0/4034705686 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae14052750 con 0x7fae1c151d40 2023-03-03T17:08:22.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.391+0000 7fae235c8700 1 -- 172.21.15.7:0/4034705686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae1c074bb0 con 0x7fae1c151d40 2023-03-03T17:08:22.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.391+0000 7fae0dffb700 1 -- 172.21.15.7:0/4034705686 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fae14055d70 con 0x7fae1c151d40 2023-03-03T17:08:22.391 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:08:22.391 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:08:22.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 -- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae000682f0 msgr2=0x7fae0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:22.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae000682f0 0x7fae0006a7a0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fae1c155630 tx=0x7fae18009110 comp rx=0 tx=0).stop 2023-03-03T17:08:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 -- 172.21.15.7:0/4034705686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c151d40 msgr2=0x7fae1c1e0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c151d40 0x7fae1c1e0840 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b970 tx=0x7fae1400bc80 comp rx=0 tx=0).stop 2023-03-03T17:08:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 -- 172.21.15.7:0/4034705686 shutdown_connections 2023-03-03T17:08:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae000682f0 0x7fae0006a7a0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1c151d40 0x7fae1c1e0840 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae1c073c30 0x7fae1c145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 --2- 172.21.15.7:0/4034705686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae1c0c2930 0x7fae1c1458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 -- 172.21.15.7:0/4034705686 >> 172.21.15.7:0/4034705686 conn(0x7fae1c0a94d0 msgr2=0x7fae1c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:08:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.393+0000 7fae235c8700 1 -- 172.21.15.7:0/4034705686 shutdown_connections 2023-03-03T17:08:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:22.394+0000 7fae235c8700 1 -- 172.21.15.7:0/4034705686 wait complete. 2023-03-03T17:08:22.395 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:08:22.405 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before thrashing 2023-03-03T17:08:23.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:23.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:24.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:08:24.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.544+0000 7f221d13e700 1 Processor -- start 2023-03-03T17:08:24.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.544+0000 7f221d13e700 1 -- start start 2023-03-03T17:08:24.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f221d13e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f22180c2e30 0x7f2218158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:24.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f221d13e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f221806c6c0 0x7f22180736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:24.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f221d13e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2218157ff0 con 0x7f2218151cd0 2023-03-03T17:08:24.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f221d13e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f221806d640 con 0x7f22180c2e30 2023-03-03T17:08:24.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f221d13e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f221806cda0 con 0x7f221806c6c0 2023-03-03T17:08:24.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f2216d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2218151cd0 0x7f2218154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54072/0 (socket says 172.21.15.7:54072) 2023-03-03T17:08:24.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f2216d9d700 1 -- 172.21.15.7:0/3235728676 learned_addr learned my addr 172.21.15.7:0/3235728676 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:08:24.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f221659c700 1 --2- 172.21.15.7:0/3235728676 >> v2:172.21.15.143:3300/0 conn(0x7f22180c2e30 0x7f2218158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:24.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f221759e700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f221806c6c0 0x7f22180736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:24.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582997174 0 0) 0x7f2218157ff0 con 0x7f2218151cd0 2023-03-03T17:08:24.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.545+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22000035b0 con 0x7f2218151cd0 2023-03-03T17:08:24.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 717494178 0 0) 0x7f22000035b0 con 0x7f2218151cd0 2023-03-03T17:08:24.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f221806c6c0 msgr2=0x7f22180736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:08:24.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f221806c6c0 0x7f22180736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 >> v2:172.21.15.143:3300/0 conn(0x7f22180c2e30 msgr2=0x7f2218158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:08:24.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 --2- 172.21.15.7:0/3235728676 >> v2:172.21.15.143:3300/0 conn(0x7f22180c2e30 0x7f2218158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221806d080 con 0x7f2218151cd0 2023-03-03T17:08:24.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f220c001670 con 0x7f2218151cd0 2023-03-03T17:08:24.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 >> v1:172.21.15.7:6789/0 conn(0x7f2218151cd0 legacy=0x7f2218154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:24.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f221806c6c0 0x7f2200003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:24.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2200003ff0 0x7f2200004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:24.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2200004d90 0x7f2200005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:24.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221806d640 con 0x7f2200003ff0 2023-03-03T17:08:24.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221806cda0 con 0x7f2200004d90 2023-03-03T17:08:24.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220000bac0 con 0x7f221806c6c0 2023-03-03T17:08:24.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f221659c700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2200003ff0 0x7f2200004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:24.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f2216d9d700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f221806c6c0 0x7f2200003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:24.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f221759e700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2200004d90 0x7f2200005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:24.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f221659c700 1 -- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f221806c6c0 msgr2=0x7f2200003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:08:24.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f221659c700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f221806c6c0 0x7f2200003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f221659c700 1 -- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2200004d90 msgr2=0x7f2200005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:08:24.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f221659c700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2200004d90 0x7f2200005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f221659c700 1 -- 172.21.15.7:0/3235728676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22000035b0 con 0x7f2200003ff0 2023-03-03T17:08:24.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f221759e700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2200004d90 0x7f2200005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:08:24.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.546+0000 7f221659c700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2200003ff0 0x7f2200004850 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f2208009b30 tx=0x7f2200005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb48f16916a1e45c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:08:24.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.547+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220800e6c0 con 0x7f2200003ff0 2023-03-03T17:08:24.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.547+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220800e820 con 0x7f2200003ff0 2023-03-03T17:08:24.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.547+0000 7f2215d9b700 1 -- 172.21.15.7:0/3235728676 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220800eb90 con 0x7f2200003ff0 2023-03-03T17:08:24.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.547+0000 7f221d13e700 1 -- 172.21.15.7:0/3235728676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2200003ff0 msgr2=0x7f2200004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:24.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.547+0000 7f221d13e700 1 --2- 172.21.15.7:0/3235728676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2200003ff0 0x7f2200004850 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f2208009b30 tx=0x7f2200005880 comp rx=0 tx=0).stop 2023-03-03T17:08:24.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.547+0000 7f221759e700 1 -- 172.21.15.7:0/3235728676 reap_dead start 2023-03-03T17:08:24.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.547+0000 7f221d13e700 1 -- 172.21.15.7:0/3235728676 shutdown_connections 2023-03-03T17:08:24.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.547+0000 7f221d13e700 1 -- 172.21.15.7:0/3235728676 >> 172.21.15.7:0/3235728676 conn(0x7f22180a94d0 msgr2=0x7f22180a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:08:24.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.547+0000 7f221d13e700 1 -- 172.21.15.7:0/3235728676 shutdown_connections 2023-03-03T17:08:24.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.547+0000 7f221d13e700 1 -- 172.21.15.7:0/3235728676 wait complete. 2023-03-03T17:08:24.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.549+0000 7f221d13e700 1 Processor -- start 2023-03-03T17:08:24.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.549+0000 7f221d13e700 1 -- start start 2023-03-03T17:08:24.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.549+0000 7f221d13e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2200003ff0 0x7f221814a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:24.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.549+0000 7f221d13e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2218155900 0x7f22181e1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:24.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.549+0000 7f221d13e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2218156720 0x7f221814cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:24.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.549+0000 7f221d13e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2218157460 con 0x7f2218155900 2023-03-03T17:08:24.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.549+0000 7f221d13e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221814a730 con 0x7f2218156720 2023-03-03T17:08:24.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.549+0000 7f221d13e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f221814c050 con 0x7f2200003ff0 2023-03-03T17:08:24.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2218155900 0x7f22181e1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:24.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2218155900 0x7f22181e1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60104/0 (socket says 172.21.15.7:60104) 2023-03-03T17:08:24.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221659c700 1 -- 172.21.15.7:0/2425699326 learned_addr learned my addr 172.21.15.7:0/2425699326 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:08:24.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221759e700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2218156720 0x7f221814cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:24.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f2216d9d700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2200003ff0 0x7f221814a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:24.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221659c700 1 -- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2200003ff0 msgr2=0x7f221814a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:24.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221659c700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2200003ff0 0x7f221814a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221659c700 1 -- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2218156720 msgr2=0x7f221814cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:24.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221659c700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2218156720 0x7f221814cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221659c700 1 -- 172.21.15.7:0/2425699326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2218157c80 con 0x7f2218155900 2023-03-03T17:08:24.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221759e700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2218156720 0x7f221814cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:08:24.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221659c700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2218155900 0x7f22181e1120 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f220800e4b0 tx=0x7f22080047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:08:24.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f21ff7fe700 1 -- 172.21.15.7:0/2425699326 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220800e6c0 con 0x7f2218155900 2023-03-03T17:08:24.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f21ff7fe700 1 -- 172.21.15.7:0/2425699326 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2208003ab0 con 0x7f2218155900 2023-03-03T17:08:24.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f21ff7fe700 1 -- 172.21.15.7:0/2425699326 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2208003e60 con 0x7f2218155900 2023-03-03T17:08:24.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221d13e700 1 -- 172.21.15.7:0/2425699326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221806d360 con 0x7f2218155900 2023-03-03T17:08:24.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.550+0000 7f221d13e700 1 -- 172.21.15.7:0/2425699326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22181589c0 con 0x7f2218155900 2023-03-03T17:08:24.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.551+0000 7f221d13e700 1 -- 172.21.15.7:0/2425699326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2218157880 con 0x7f2218155900 2023-03-03T17:08:24.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.552+0000 7f21ff7fe700 1 -- 172.21.15.7:0/2425699326 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2208019030 con 0x7f2218155900 2023-03-03T17:08:24.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.552+0000 7f21ff7fe700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22040682f0 0x7f220406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:08:24.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.552+0000 7f21ff7fe700 1 -- 172.21.15.7:0/2425699326 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f22080878c0 con 0x7f2218155900 2023-03-03T17:08:24.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.552+0000 7f2216d9d700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22040682f0 0x7f220406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:08:24.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.553+0000 7f2216d9d700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22040682f0 0x7f220406a7a0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f220c000fb0 tx=0x7f220c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:08:24.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.557+0000 7f21ff7fe700 1 -- 172.21.15.7:0/2425699326 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2208053c60 con 0x7f2218155900 2023-03-03T17:08:24.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.796+0000 7f221d13e700 1 -- 172.21.15.7:0/2425699326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2218074bb0 con 0x7f2218155900 2023-03-03T17:08:24.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.796+0000 7f21ff7fe700 1 -- 172.21.15.7:0/2425699326 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2208057280 con 0x7f2218155900 2023-03-03T17:08:24.796 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:08:24.797 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:08:24.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 -- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22040682f0 msgr2=0x7f220406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:24.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22040682f0 0x7f220406a7a0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f220c000fb0 tx=0x7f220c009040 comp rx=0 tx=0).stop 2023-03-03T17:08:24.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 -- 172.21.15.7:0/2425699326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2218155900 msgr2=0x7f22181e1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:08:24.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2218155900 0x7f22181e1120 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f220800e4b0 tx=0x7f22080047c0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 -- 172.21.15.7:0/2425699326 shutdown_connections 2023-03-03T17:08:24.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2218156720 0x7f221814cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2200003ff0 0x7f221814a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f22040682f0 0x7f220406a7a0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 --2- 172.21.15.7:0/2425699326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2218155900 0x7f22181e1120 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:08:24.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 -- 172.21.15.7:0/2425699326 >> 172.21.15.7:0/2425699326 conn(0x7f22180a94d0 msgr2=0x7f22181503b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:08:24.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 -- 172.21.15.7:0/2425699326 shutdown_connections 2023-03-03T17:08:24.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:08:24.798+0000 7f221d13e700 1 -- 172.21.15.7:0/2425699326 wait complete. 2023-03-03T17:08:24.800 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:08:24.810 INFO:tasks.mds_thrash.fs.[a]:waiting for 50 secs before thrashing 2023-03-03T17:08:53.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:08:53.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:04.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:09:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.607+0000 7f0bf466e700 1 Processor -- start 2023-03-03T17:09:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.607+0000 7f0bf466e700 1 -- start start 2023-03-03T17:09:04.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf466e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0bec0aad20 0x7f0bec157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:04.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf466e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bec0aa030 0x7f0bec1581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:04.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf466e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bec0728b0 con 0x7f0bec14b020 2023-03-03T17:09:04.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf466e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bec155070 con 0x7f0bec0aad20 2023-03-03T17:09:04.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf466e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec06f5b0 con 0x7f0bec0aa030 2023-03-03T17:09:04.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf1c09700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0bec0aad20 0x7f0bec157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:04.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf240a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0bec14b020 0x7f0bec0717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54110/0 (socket says 172.21.15.7:54110) 2023-03-03T17:09:04.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf240a700 1 -- 172.21.15.7:0/1094504417 learned_addr learned my addr 172.21.15.7:0/1094504417 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:09:04.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf2c0b700 1 --2- 172.21.15.7:0/1094504417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bec0aa030 0x7f0bec1581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:04.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf1c09700 1 -- 172.21.15.7:0/1094504417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bec0aa030 msgr2=0x7f0bec1581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:09:04.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf1c09700 1 --2- 172.21.15.7:0/1094504417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bec0aa030 0x7f0bec1581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:04.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf1c09700 1 -- 172.21.15.7:0/1094504417 >> v1:172.21.15.7:6789/0 conn(0x7f0bec14b020 legacy=0x7f0bec0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:04.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf1c09700 1 -- 172.21.15.7:0/1094504417 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bec06f890 con 0x7f0bec0aad20 2023-03-03T17:09:04.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.608+0000 7f0bf1408700 1 -- 172.21.15.7:0/1094504417 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869111977 0 0) 0x7f0bec0728b0 con 0x7f0bec14b020 2023-03-03T17:09:04.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.609+0000 7f0bf1c09700 1 --2- 172.21.15.7:0/1094504417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec0aad20 0x7f0bec157ca0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009840 tx=0x7f0be8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9e49f6239ae6d7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:09:04.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.609+0000 7f0bf1408700 1 -- 172.21.15.7:0/1094504417 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8011830 con 0x7f0bec0aad20 2023-03-03T17:09:04.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.609+0000 7f0bf1408700 1 -- 172.21.15.7:0/1094504417 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be8011990 con 0x7f0bec0aad20 2023-03-03T17:09:04.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.609+0000 7f0bf1408700 1 -- 172.21.15.7:0/1094504417 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8011d20 con 0x7f0bec0aad20 2023-03-03T17:09:04.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.609+0000 7f0bf466e700 1 -- 172.21.15.7:0/1094504417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec0aad20 msgr2=0x7f0bec157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:04.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.609+0000 7f0bf466e700 1 --2- 172.21.15.7:0/1094504417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec0aad20 0x7f0bec157ca0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009840 tx=0x7f0be8009b50 comp rx=0 tx=0).stop 2023-03-03T17:09:04.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.609+0000 7f0bf466e700 1 -- 172.21.15.7:0/1094504417 shutdown_connections 2023-03-03T17:09:04.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.609+0000 7f0bf466e700 1 --2- 172.21.15.7:0/1094504417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bec0aa030 0x7f0bec1581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:04.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.609+0000 7f0bf466e700 1 --2- 172.21.15.7:0/1094504417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec0aad20 0x7f0bec157ca0 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:04.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.609+0000 7f0bf466e700 1 -- 172.21.15.7:0/1094504417 >> 172.21.15.7:0/1094504417 conn(0x7f0bec0c8900 msgr2=0x7f0bec0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:09:04.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.610+0000 7f0bf466e700 1 -- 172.21.15.7:0/1094504417 shutdown_connections 2023-03-03T17:09:04.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.610+0000 7f0bf466e700 1 -- 172.21.15.7:0/1094504417 wait complete. 2023-03-03T17:09:04.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.611+0000 7f0bf466e700 1 Processor -- start 2023-03-03T17:09:04.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.611+0000 7f0bf466e700 1 -- start start 2023-03-03T17:09:04.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf466e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bec0aa030 0x7f0bec0c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:04.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf466e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bec0aad20 0x7f0bec145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:04.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf466e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec14b020 0x7f0bec152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:04.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf466e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec1eac30 con 0x7f0bec14b020 2023-03-03T17:09:04.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf466e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec1ea020 con 0x7f0bec0aa030 2023-03-03T17:09:04.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf466e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bec1ea300 con 0x7f0bec0aad20 2023-03-03T17:09:04.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf2c0b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec14b020 0x7f0bec152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:04.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf2c0b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec14b020 0x7f0bec152070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60140/0 (socket says 172.21.15.7:60140) 2023-03-03T17:09:04.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf2c0b700 1 -- 172.21.15.7:0/2303600332 learned_addr learned my addr 172.21.15.7:0/2303600332 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:09:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf1c09700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bec0aad20 0x7f0bec145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.612+0000 7f0bf240a700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bec0aa030 0x7f0bec0c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0bf2c0b700 1 -- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bec0aad20 msgr2=0x7f0bec145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0bf2c0b700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bec0aad20 0x7f0bec145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:04.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0bf2c0b700 1 -- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bec0aa030 msgr2=0x7f0bec0c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0bf2c0b700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bec0aa030 0x7f0bec0c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0bf2c0b700 1 -- 172.21.15.7:0/2303600332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be80094a0 con 0x7f0bec14b020 2023-03-03T17:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0bf2c0b700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec14b020 0x7f0bec152070 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f0be4007e60 tx=0x7f0be400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0be2ffd700 1 -- 172.21.15.7:0/2303600332 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be400ed30 con 0x7f0bec14b020 2023-03-03T17:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0be2ffd700 1 -- 172.21.15.7:0/2303600332 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be4014420 con 0x7f0bec14b020 2023-03-03T17:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0bf466e700 1 -- 172.21.15.7:0/2303600332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bec155350 con 0x7f0bec14b020 2023-03-03T17:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0be2ffd700 1 -- 172.21.15.7:0/2303600332 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be40147d0 con 0x7f0bec14b020 2023-03-03T17:09:04.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.613+0000 7f0bf466e700 1 -- 172.21.15.7:0/2303600332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bec06fb70 con 0x7f0bec14b020 2023-03-03T17:09:04.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.614+0000 7f0bf466e700 1 -- 172.21.15.7:0/2303600332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bec071ef0 con 0x7f0bec14b020 2023-03-03T17:09:04.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.614+0000 7f0be2ffd700 1 -- 172.21.15.7:0/2303600332 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0be4012050 con 0x7f0bec14b020 2023-03-03T17:09:04.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.615+0000 7f0be2ffd700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bd00682f0 0x7f0bd006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:04.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.615+0000 7f0be2ffd700 1 -- 172.21.15.7:0/2303600332 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0be40865d0 con 0x7f0bec14b020 2023-03-03T17:09:04.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.615+0000 7f0bf240a700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bd00682f0 0x7f0bd006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:04.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.616+0000 7f0bf240a700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bd00682f0 0x7f0bd006a7a0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc000f60 tx=0x7f0bdc007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:09:04.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.619+0000 7f0be2ffd700 1 -- 172.21.15.7:0/2303600332 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0be4052970 con 0x7f0bec14b020 2023-03-03T17:09:04.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.860+0000 7f0bf466e700 1 -- 172.21.15.7:0/2303600332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bec1eade0 con 0x7f0bec14b020 2023-03-03T17:09:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.860+0000 7f0be2ffd700 1 -- 172.21.15.7:0/2303600332 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0bec1eade0 con 0x7f0bec14b020 2023-03-03T17:09:04.860 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:09:04.861 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:09:04.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 -- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bd00682f0 msgr2=0x7f0bd006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:04.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bd00682f0 0x7f0bd006a7a0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc000f60 tx=0x7f0bdc007e70 comp rx=0 tx=0).stop 2023-03-03T17:09:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 -- 172.21.15.7:0/2303600332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec14b020 msgr2=0x7f0bec152070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec14b020 0x7f0bec152070 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7f0be4007e60 tx=0x7f0be400e3c0 comp rx=0 tx=0).stop 2023-03-03T17:09:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 -- 172.21.15.7:0/2303600332 shutdown_connections 2023-03-03T17:09:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0bd00682f0 0x7f0bd006a7a0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0bec14b020 0x7f0bec152070 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bec0aa030 0x7f0bec0c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 --2- 172.21.15.7:0/2303600332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0bec0aad20 0x7f0bec145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 -- 172.21.15.7:0/2303600332 >> 172.21.15.7:0/2303600332 conn(0x7f0bec0c8900 msgr2=0x7f0bec158360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:09:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 -- 172.21.15.7:0/2303600332 shutdown_connections 2023-03-03T17:09:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:04.862+0000 7f0bf466e700 1 -- 172.21.15.7:0/2303600332 wait complete. 2023-03-03T17:09:04.864 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:09:04.874 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-03-03T17:09:14.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:09:14.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.947+0000 7f8d36da2700 1 Processor -- start 2023-03-03T17:09:14.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.947+0000 7f8d36da2700 1 -- start start 2023-03-03T17:09:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.947+0000 7f8d36da2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8d30072250 0x7f8d3014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.947+0000 7f8d36da2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d301510d0 con 0x7f8d300c1ce0 2023-03-03T17:09:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.947+0000 7f8d36da2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d300c3740 con 0x7f8d30071e80 2023-03-03T17:09:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.947+0000 7f8d36da2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d30070670 con 0x7f8d30072250 2023-03-03T17:09:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d2ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8d30072250 0x7f8d3014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d3533f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8d300c1ce0 0x7f8d3015ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54128/0 (socket says 172.21.15.7:54128) 2023-03-03T17:09:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d3533f700 1 -- 172.21.15.7:0/1761188553 learned_addr learned my addr 172.21.15.7:0/1761188553 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:09:14.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d2ffff700 1 -- 172.21.15.7:0/1761188553 >> v1:172.21.15.143:6789/0 conn(0x7f8d30071e80 legacy=0x7f8d30151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d2ffff700 1 -- 172.21.15.7:0/1761188553 >> v1:172.21.15.7:6789/0 conn(0x7f8d300c1ce0 legacy=0x7f8d3015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d2ffff700 1 -- 172.21.15.7:0/1761188553 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d300c39e0 con 0x7f8d30072250 2023-03-03T17:09:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d2f7fe700 1 -- 172.21.15.7:0/1761188553 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180961939 0 0) 0x7f8d301510d0 con 0x7f8d300c1ce0 2023-03-03T17:09:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d2ffff700 1 --2- 172.21.15.7:0/1761188553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30072250 0x7f8d3014e1f0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009870 tx=0x7f8d1c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb70a7b55465286c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:09:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d2f7fe700 1 -- 172.21.15.7:0/1761188553 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c011830 con 0x7f8d30072250 2023-03-03T17:09:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d2f7fe700 1 -- 172.21.15.7:0/1761188553 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c011990 con 0x7f8d30072250 2023-03-03T17:09:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.948+0000 7f8d2f7fe700 1 -- 172.21.15.7:0/1761188553 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c011d20 con 0x7f8d30072250 2023-03-03T17:09:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.949+0000 7f8d36da2700 1 -- 172.21.15.7:0/1761188553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30072250 msgr2=0x7f8d3014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.949+0000 7f8d36da2700 1 --2- 172.21.15.7:0/1761188553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30072250 0x7f8d3014e1f0 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009870 tx=0x7f8d1c009b80 comp rx=0 tx=0).stop 2023-03-03T17:09:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.949+0000 7f8d36da2700 1 -- 172.21.15.7:0/1761188553 shutdown_connections 2023-03-03T17:09:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.949+0000 7f8d36da2700 1 --2- 172.21.15.7:0/1761188553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30072250 0x7f8d3014e1f0 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.949+0000 7f8d36da2700 1 -- 172.21.15.7:0/1761188553 >> 172.21.15.7:0/1761188553 conn(0x7f8d300a94d0 msgr2=0x7f8d300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:09:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.949+0000 7f8d36da2700 1 -- 172.21.15.7:0/1761188553 shutdown_connections 2023-03-03T17:09:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.949+0000 7f8d36da2700 1 -- 172.21.15.7:0/1761188553 wait complete. 2023-03-03T17:09:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.950+0000 7f8d36da2700 1 Processor -- start 2023-03-03T17:09:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.950+0000 7f8d36da2700 1 -- start start 2023-03-03T17:09:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d36da2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30071e80 0x7f8d301edbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d36da2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d30072250 0x7f8d301e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d36da2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d300c1ce0 0x7f8d30073960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d36da2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301ee120 con 0x7f8d30071e80 2023-03-03T17:09:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d36da2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d301ea1a0 con 0x7f8d300c1ce0 2023-03-03T17:09:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d36da2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d300c5160 con 0x7f8d30072250 2023-03-03T17:09:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d34b3e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30071e80 0x7f8d301edbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d34b3e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30071e80 0x7f8d301edbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60150/0 (socket says 172.21.15.7:60150) 2023-03-03T17:09:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d34b3e700 1 -- 172.21.15.7:0/1651314825 learned_addr learned my addr 172.21.15.7:0/1651314825 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:09:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d3533f700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d300c1ce0 0x7f8d30073960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d2ffff700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d30072250 0x7f8d301e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d34b3e700 1 -- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d30072250 msgr2=0x7f8d301e9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d34b3e700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d30072250 0x7f8d301e9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d34b3e700 1 -- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d300c1ce0 msgr2=0x7f8d30073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d34b3e700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d300c1ce0 0x7f8d30073960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d34b3e700 1 -- 172.21.15.7:0/1651314825 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c0094a0 con 0x7f8d30071e80 2023-03-03T17:09:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.951+0000 7f8d3533f700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d300c1ce0 0x7f8d30073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:09:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.952+0000 7f8d34b3e700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30071e80 0x7f8d301edbe0 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400c510 tx=0x7f8d2400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:09:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.952+0000 7f8d2d7fa700 1 -- 172.21.15.7:0/1651314825 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d240052d0 con 0x7f8d30071e80 2023-03-03T17:09:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.952+0000 7f8d2d7fa700 1 -- 172.21.15.7:0/1651314825 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d24005430 con 0x7f8d30071e80 2023-03-03T17:09:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.952+0000 7f8d36da2700 1 -- 172.21.15.7:0/1651314825 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d30151f30 con 0x7f8d30071e80 2023-03-03T17:09:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.952+0000 7f8d2d7fa700 1 -- 172.21.15.7:0/1651314825 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d240038c0 con 0x7f8d30071e80 2023-03-03T17:09:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.952+0000 7f8d36da2700 1 -- 172.21.15.7:0/1651314825 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d30151300 con 0x7f8d30071e80 2023-03-03T17:09:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.953+0000 7f8d36da2700 1 -- 172.21.15.7:0/1651314825 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3006c9a0 con 0x7f8d30071e80 2023-03-03T17:09:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.953+0000 7f8d2d7fa700 1 -- 172.21.15.7:0/1651314825 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8d24019030 con 0x7f8d30071e80 2023-03-03T17:09:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.954+0000 7f8d2d7fa700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d140682f0 0x7f8d1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:09:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.954+0000 7f8d2d7fa700 1 -- 172.21.15.7:0/1651314825 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8d24085500 con 0x7f8d30071e80 2023-03-03T17:09:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.954+0000 7f8d2ffff700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d140682f0 0x7f8d1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:09:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.954+0000 7f8d2ffff700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d140682f0 0x7f8d1406a7a0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f8d300c1780 tx=0x7f8d1c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:09:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:14.958+0000 7f8d2d7fa700 1 -- 172.21.15.7:0/1651314825 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8d240518a0 con 0x7f8d30071e80 2023-03-03T17:09:15.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.195+0000 7f8d36da2700 1 -- 172.21.15.7:0/1651314825 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d30074bb0 con 0x7f8d30071e80 2023-03-03T17:09:15.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.196+0000 7f8d2d7fa700 1 -- 172.21.15.7:0/1651314825 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8d24054ec0 con 0x7f8d30071e80 2023-03-03T17:09:15.196 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:09:15.196 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:09:15.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 -- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d140682f0 msgr2=0x7f8d1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:15.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d140682f0 0x7f8d1406a7a0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f8d300c1780 tx=0x7f8d1c008030 comp rx=0 tx=0).stop 2023-03-03T17:09:15.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 -- 172.21.15.7:0/1651314825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30071e80 msgr2=0x7f8d301edbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:09:15.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30071e80 0x7f8d301edbe0 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400c510 tx=0x7f8d2400c8d0 comp rx=0 tx=0).stop 2023-03-03T17:09:15.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 -- 172.21.15.7:0/1651314825 shutdown_connections 2023-03-03T17:09:15.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d300c1ce0 0x7f8d30073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:15.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d140682f0 0x7f8d1406a7a0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:15.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d30071e80 0x7f8d301edbe0 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:15.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 --2- 172.21.15.7:0/1651314825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d30072250 0x7f8d301e9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:09:15.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 -- 172.21.15.7:0/1651314825 >> 172.21.15.7:0/1651314825 conn(0x7f8d300a94d0 msgr2=0x7f8d3014dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:09:15.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 -- 172.21.15.7:0/1651314825 shutdown_connections 2023-03-03T17:09:15.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:09:15.198+0000 7f8d36da2700 1 -- 172.21.15.7:0/1651314825 wait complete. 2023-03-03T17:09:15.200 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:09:15.210 INFO:tasks.mds_thrash.fs.[a]:waiting for 85 secs before thrashing 2023-03-03T17:09:23.595 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:23.598 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:53.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:53.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:09:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T17:09:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T17:10:23.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:23.726 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:40.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:10:40.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.350+0000 7f1988366700 1 Processor -- start 2023-03-03T17:10:40.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.350+0000 7f1988366700 1 -- start start 2023-03-03T17:10:40.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.351+0000 7f1988366700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19800aad20 0x7f1980157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:40.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.351+0000 7f1988366700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19800a9c20 con 0x7f19800aa030 2023-03-03T17:10:40.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.351+0000 7f1988366700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1980154f10 con 0x7f198014b0d0 2023-03-03T17:10:40.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.351+0000 7f1988366700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19800ac3c0 con 0x7f19800aad20 2023-03-03T17:10:40.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.351+0000 7f1985901700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19800aad20 0x7f1980157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:40.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.351+0000 7f1985901700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f19800aad20 0x7f1980157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60210/0 (socket says 172.21.15.7:60210) 2023-03-03T17:10:40.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.351+0000 7f1986903700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f19800aa030 0x7f1980158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54192/0 (socket says 172.21.15.7:54192) 2023-03-03T17:10:40.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.351+0000 7f1985901700 1 -- 172.21.15.7:0/3921383414 learned_addr learned my addr 172.21.15.7:0/3921383414 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:10:40.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.352+0000 7f1985901700 1 -- 172.21.15.7:0/3921383414 >> v1:172.21.15.143:6789/0 conn(0x7f198014b0d0 legacy=0x7f1980071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:40.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.352+0000 7f1985901700 1 -- 172.21.15.7:0/3921383414 >> v1:172.21.15.7:6789/0 conn(0x7f19800aa030 legacy=0x7f1980158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:40.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.352+0000 7f1985901700 1 -- 172.21.15.7:0/3921383414 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198006fc70 con 0x7f19800aad20 2023-03-03T17:10:40.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.352+0000 7f1985901700 1 --2- 172.21.15.7:0/3921383414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 0x7f1980157df0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f19740098f0 tx=0x7f1974009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869217aa28592ab8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:40.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.352+0000 7f1985100700 1 -- 172.21.15.7:0/3921383414 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197400b8a0 con 0x7f19800aad20 2023-03-03T17:10:40.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.352+0000 7f1985100700 1 -- 172.21.15.7:0/3921383414 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197400ba00 con 0x7f19800aad20 2023-03-03T17:10:40.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.352+0000 7f1985100700 1 -- 172.21.15.7:0/3921383414 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197400bd90 con 0x7f19800aad20 2023-03-03T17:10:40.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.353+0000 7f1988366700 1 -- 172.21.15.7:0/3921383414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 msgr2=0x7f1980157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:40.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.353+0000 7f1988366700 1 --2- 172.21.15.7:0/3921383414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 0x7f1980157df0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f19740098f0 tx=0x7f1974009c00 comp rx=0 tx=0).stop 2023-03-03T17:10:40.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.353+0000 7f1988366700 1 -- 172.21.15.7:0/3921383414 shutdown_connections 2023-03-03T17:10:40.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.353+0000 7f1988366700 1 --2- 172.21.15.7:0/3921383414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 0x7f1980157df0 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:40.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.353+0000 7f1988366700 1 -- 172.21.15.7:0/3921383414 >> 172.21.15.7:0/3921383414 conn(0x7f19800c8900 msgr2=0x7f19800c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:10:40.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.353+0000 7f1988366700 1 -- 172.21.15.7:0/3921383414 shutdown_connections 2023-03-03T17:10:40.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.353+0000 7f1988366700 1 -- 172.21.15.7:0/3921383414 wait complete. 2023-03-03T17:10:40.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.354+0000 7f1988366700 1 Processor -- start 2023-03-03T17:10:40.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.354+0000 7f1988366700 1 -- start start 2023-03-03T17:10:40.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1988366700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19800aa030 0x7f1980147360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:40.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1988366700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 0x7f1980145230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:40.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1988366700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f198014b0d0 0x7f1980145770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:40.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1988366700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1980145d30 con 0x7f19800aad20 2023-03-03T17:10:40.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1988366700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1980156930 con 0x7f19800aa030 2023-03-03T17:10:40.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1988366700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1980145f80 con 0x7f198014b0d0 2023-03-03T17:10:40.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1985901700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 0x7f1980145230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1985901700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 0x7f1980145230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60216/0 (socket says 172.21.15.7:60216) 2023-03-03T17:10:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1985901700 1 -- 172.21.15.7:0/1543854785 learned_addr learned my addr 172.21.15.7:0/1543854785 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:10:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1986102700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19800aa030 0x7f1980147360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1986903700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f198014b0d0 0x7f1980145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1985901700 1 -- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f198014b0d0 msgr2=0x7f1980145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1985901700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f198014b0d0 0x7f1980145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:40.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1985901700 1 -- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19800aa030 msgr2=0x7f1980147360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1985901700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19800aa030 0x7f1980147360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1985901700 1 -- 172.21.15.7:0/1543854785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1974009550 con 0x7f19800aad20 2023-03-03T17:10:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1985901700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 0x7f1980145230 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f1974000c00 tx=0x7f1974004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1972ffd700 1 -- 172.21.15.7:0/1543854785 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974004570 con 0x7f19800aad20 2023-03-03T17:10:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1972ffd700 1 -- 172.21.15.7:0/1543854785 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197401f070 con 0x7f19800aad20 2023-03-03T17:10:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1988366700 1 -- 172.21.15.7:0/1543854785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f198006f6b0 con 0x7f19800aad20 2023-03-03T17:10:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1972ffd700 1 -- 172.21.15.7:0/1543854785 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19740038e0 con 0x7f19800aad20 2023-03-03T17:10:40.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.355+0000 7f1988366700 1 -- 172.21.15.7:0/1543854785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1980155160 con 0x7f19800aad20 2023-03-03T17:10:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.356+0000 7f1988366700 1 -- 172.21.15.7:0/1543854785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f198006f990 con 0x7f19800aad20 2023-03-03T17:10:40.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.357+0000 7f1972ffd700 1 -- 172.21.15.7:0/1543854785 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f197402a020 con 0x7f19800aad20 2023-03-03T17:10:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.357+0000 7f1972ffd700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19640682f0 0x7f196406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.357+0000 7f1972ffd700 1 -- 172.21.15.7:0/1543854785 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1974087710 con 0x7f19800aad20 2023-03-03T17:10:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.357+0000 7f1986102700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19640682f0 0x7f196406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.358+0000 7f1986102700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19640682f0 0x7f196406a7a0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f19800c3dc0 tx=0x7f197c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.362+0000 7f1972ffd700 1 -- 172.21.15.7:0/1543854785 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1974053ab0 con 0x7f19800aad20 2023-03-03T17:10:40.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.597+0000 7f1988366700 1 -- 172.21.15.7:0/1543854785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1980074bb0 con 0x7f19800aad20 2023-03-03T17:10:40.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.597+0000 7f1972ffd700 1 -- 172.21.15.7:0/1543854785 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f19740570d0 con 0x7f19800aad20 2023-03-03T17:10:40.597 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:10:40.598 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:10:40.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 -- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19640682f0 msgr2=0x7f196406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:40.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19640682f0 0x7f196406a7a0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f19800c3dc0 tx=0x7f197c009040 comp rx=0 tx=0).stop 2023-03-03T17:10:40.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 -- 172.21.15.7:0/1543854785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 msgr2=0x7f1980145230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:40.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 0x7f1980145230 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f1974000c00 tx=0x7f1974004750 comp rx=0 tx=0).stop 2023-03-03T17:10:40.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 -- 172.21.15.7:0/1543854785 shutdown_connections 2023-03-03T17:10:40.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f198014b0d0 0x7f1980145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:40.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19800aa030 0x7f1980147360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:40.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19640682f0 0x7f196406a7a0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:40.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 --2- 172.21.15.7:0/1543854785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19800aad20 0x7f1980145230 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:40.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 -- 172.21.15.7:0/1543854785 >> 172.21.15.7:0/1543854785 conn(0x7f19800c8900 msgr2=0x7f19800c4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:10:40.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 -- 172.21.15.7:0/1543854785 shutdown_connections 2023-03-03T17:10:40.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:40.599+0000 7f1988366700 1 -- 172.21.15.7:0/1543854785 wait complete. 2023-03-03T17:10:40.601 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:10:40.611 INFO:tasks.mds_thrash.fs.[a]:waiting for 77 secs before thrashing 2023-03-03T17:10:53.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:53.801 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:10:53.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:10:54.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.010+0000 7fad596dc700 1 Processor -- start 2023-03-03T17:10:54.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.010+0000 7fad596dc700 1 -- start start 2023-03-03T17:10:54.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.011+0000 7fad596dc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fad540c5c60 0x7fad54154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:54.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.011+0000 7fad596dc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad540c7310 con 0x7fad540caa30 2023-03-03T17:10:54.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.011+0000 7fad596dc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad540bfd10 con 0x7fad540c00e0 2023-03-03T17:10:54.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.011+0000 7fad596dc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad540c60a0 con 0x7fad540c5c60 2023-03-03T17:10:54.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.011+0000 7fad537fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fad540c5c60 0x7fad54154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:54.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.011+0000 7fad52ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fad540caa30 0x7fad540c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54220/0 (socket says 172.21.15.7:54220) 2023-03-03T17:10:54.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.011+0000 7fad52ffd700 1 -- 172.21.15.7:0/2889707702 learned_addr learned my addr 172.21.15.7:0/2889707702 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:10:54.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.011+0000 7fad537fe700 1 -- 172.21.15.7:0/2889707702 >> v1:172.21.15.143:6789/0 conn(0x7fad540c00e0 legacy=0x7fad5415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:54.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.011+0000 7fad537fe700 1 -- 172.21.15.7:0/2889707702 >> v1:172.21.15.7:6789/0 conn(0x7fad540caa30 legacy=0x7fad540c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:54.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.011+0000 7fad537fe700 1 -- 172.21.15.7:0/2889707702 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad540bfa30 con 0x7fad540c5c60 2023-03-03T17:10:54.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad537fe700 1 --2- 172.21.15.7:0/2889707702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 0x7fad54154ff0 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7fad4c009d90 tx=0x7fad4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4810c154e7c7b217 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:54.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad51ffb700 1 -- 172.21.15.7:0/2889707702 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c00e070 con 0x7fad540c5c60 2023-03-03T17:10:54.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad51ffb700 1 -- 172.21.15.7:0/2889707702 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4c00bbd0 con 0x7fad540c5c60 2023-03-03T17:10:54.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad51ffb700 1 -- 172.21.15.7:0/2889707702 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4c012430 con 0x7fad540c5c60 2023-03-03T17:10:54.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad596dc700 1 -- 172.21.15.7:0/2889707702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 msgr2=0x7fad54154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:54.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad596dc700 1 --2- 172.21.15.7:0/2889707702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 0x7fad54154ff0 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7fad4c009d90 tx=0x7fad4c00b330 comp rx=0 tx=0).stop 2023-03-03T17:10:54.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad596dc700 1 -- 172.21.15.7:0/2889707702 shutdown_connections 2023-03-03T17:10:54.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad596dc700 1 --2- 172.21.15.7:0/2889707702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 0x7fad54154ff0 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:54.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad596dc700 1 -- 172.21.15.7:0/2889707702 >> 172.21.15.7:0/2889707702 conn(0x7fad54066e40 msgr2=0x7fad54071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:10:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad596dc700 1 -- 172.21.15.7:0/2889707702 shutdown_connections 2023-03-03T17:10:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.012+0000 7fad596dc700 1 -- 172.21.15.7:0/2889707702 wait complete. 2023-03-03T17:10:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.014+0000 7fad596dc700 1 Processor -- start 2023-03-03T17:10:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.014+0000 7fad596dc700 1 -- start start 2023-03-03T17:10:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.014+0000 7fad596dc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540c00e0 0x7fad541ed2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:54.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.014+0000 7fad596dc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 0x7fad541eb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.014+0000 7fad596dc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa30 0x7fad541eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.014+0000 7fad596dc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5414a210 con 0x7fad540c5c60 2023-03-03T17:10:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.014+0000 7fad596dc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad541edd90 con 0x7fad540c00e0 2023-03-03T17:10:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.014+0000 7fad596dc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5414a4d0 con 0x7fad540caa30 2023-03-03T17:10:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad527fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 0x7fad541eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad527fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 0x7fad541eb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60248/0 (socket says 172.21.15.7:60248) 2023-03-03T17:10:54.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad527fc700 1 -- 172.21.15.7:0/1424051914 learned_addr learned my addr 172.21.15.7:0/1424051914 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:10:54.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad52ffd700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540c00e0 0x7fad541ed2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:54.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad537fe700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa30 0x7fad541eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad527fc700 1 -- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa30 msgr2=0x7fad541eb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad527fc700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa30 0x7fad541eb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad527fc700 1 -- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540c00e0 msgr2=0x7fad541ed2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad527fc700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540c00e0 0x7fad541ed2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:54.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad527fc700 1 -- 172.21.15.7:0/1424051914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c009a70 con 0x7fad540c5c60 2023-03-03T17:10:54.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad52ffd700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540c00e0 0x7fad541ed2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:10:54.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad527fc700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 0x7fad541eb280 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7fad4400c4d0 tx=0x7fad4400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:54.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad4b7fe700 1 -- 172.21.15.7:0/1424051914 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad440052d0 con 0x7fad540c5c60 2023-03-03T17:10:54.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad4b7fe700 1 -- 172.21.15.7:0/1424051914 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad44005430 con 0x7fad540c5c60 2023-03-03T17:10:54.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad596dc700 1 -- 172.21.15.7:0/1424051914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad541547f0 con 0x7fad540c5c60 2023-03-03T17:10:54.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad4b7fe700 1 -- 172.21.15.7:0/1424051914 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad440038f0 con 0x7fad540c5c60 2023-03-03T17:10:54.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.015+0000 7fad596dc700 1 -- 172.21.15.7:0/1424051914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad54154510 con 0x7fad540c5c60 2023-03-03T17:10:54.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.016+0000 7fad596dc700 1 -- 172.21.15.7:0/1424051914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad54154ad0 con 0x7fad540c5c60 2023-03-03T17:10:54.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.017+0000 7fad4b7fe700 1 -- 172.21.15.7:0/1424051914 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fad44019030 con 0x7fad540c5c60 2023-03-03T17:10:54.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.017+0000 7fad4b7fe700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad340682f0 0x7fad3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:54.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.017+0000 7fad4b7fe700 1 -- 172.21.15.7:0/1424051914 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fad440855b0 con 0x7fad540c5c60 2023-03-03T17:10:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.017+0000 7fad52ffd700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad340682f0 0x7fad3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.018+0000 7fad52ffd700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad340682f0 0x7fad3406a7a0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fad3c007fd0 tx=0x7fad3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:54.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.022+0000 7fad4b7fe700 1 -- 172.21.15.7:0/1424051914 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fad440518d0 con 0x7fad540c5c60 2023-03-03T17:10:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.260+0000 7fad596dc700 1 -- 172.21.15.7:0/1424051914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad54074bb0 con 0x7fad540c5c60 2023-03-03T17:10:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.261+0000 7fad4b7fe700 1 -- 172.21.15.7:0/1424051914 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fad44054ef0 con 0x7fad540c5c60 2023-03-03T17:10:54.262 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:10:54.262 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:10:54.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 -- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad340682f0 msgr2=0x7fad3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:54.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad340682f0 0x7fad3406a7a0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fad3c007fd0 tx=0x7fad3c009040 comp rx=0 tx=0).stop 2023-03-03T17:10:54.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 -- 172.21.15.7:0/1424051914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 msgr2=0x7fad541eb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:54.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 0x7fad541eb280 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7fad4400c4d0 tx=0x7fad4400c890 comp rx=0 tx=0).stop 2023-03-03T17:10:54.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 -- 172.21.15.7:0/1424051914 shutdown_connections 2023-03-03T17:10:54.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad540caa30 0x7fad541eb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:54.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad540c00e0 0x7fad541ed2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:54.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad340682f0 0x7fad3406a7a0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:54.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 --2- 172.21.15.7:0/1424051914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad540c5c60 0x7fad541eb280 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:54.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 -- 172.21.15.7:0/1424051914 >> 172.21.15.7:0/1424051914 conn(0x7fad54066e40 msgr2=0x7fad54065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:10:54.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 -- 172.21.15.7:0/1424051914 shutdown_connections 2023-03-03T17:10:54.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:54.263+0000 7fad596dc700 1 -- 172.21.15.7:0/1424051914 wait complete. 2023-03-03T17:10:54.266 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:10:54.276 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before thrashing 2023-03-03T17:10:56.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:10:56.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.416+0000 7efd7d611700 1 Processor -- start 2023-03-03T17:10:56.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.416+0000 7efd7d611700 1 -- start start 2023-03-03T17:10:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.416+0000 7efd7d611700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efd780caa30 0x7efd780c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.416+0000 7efd7d611700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd780c00e0 0x7efd78154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd7d611700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efd780cae00 0x7efd780c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd7d611700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd780cb280 con 0x7efd780caa30 2023-03-03T17:10:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd7d611700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd780bfd10 con 0x7efd780cae00 2023-03-03T17:10:56.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd7d611700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd780bfa30 con 0x7efd780c00e0 2023-03-03T17:10:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd76ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efd780caa30 0x7efd780c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd76ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efd780caa30 0x7efd780c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60264/0 (socket says 172.21.15.7:60264) 2023-03-03T17:10:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd76ffd700 1 -- 172.21.15.7:0/3136034977 learned_addr learned my addr 172.21.15.7:0/3136034977 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:10:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd777fe700 1 --2- 172.21.15.7:0/3136034977 >> v2:172.21.15.143:3300/0 conn(0x7efd780cae00 0x7efd780c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd767fc700 1 --2- 172.21.15.7:0/3136034977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd780c00e0 0x7efd78154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd76ffd700 1 -- 172.21.15.7:0/3136034977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd780c00e0 msgr2=0x7efd78154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:10:56.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd76ffd700 1 --2- 172.21.15.7:0/3136034977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd780c00e0 0x7efd78154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd76ffd700 1 -- 172.21.15.7:0/3136034977 >> v2:172.21.15.143:3300/0 conn(0x7efd780cae00 msgr2=0x7efd780c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:10:56.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd76ffd700 1 --2- 172.21.15.7:0/3136034977 >> v2:172.21.15.143:3300/0 conn(0x7efd780cae00 0x7efd780c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd76ffd700 1 -- 172.21.15.7:0/3136034977 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd78154510 con 0x7efd780caa30 2023-03-03T17:10:56.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd76ffd700 1 --2- 172.21.15.7:0/3136034977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780caa30 0x7efd780c5c60 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7efd78158a70 tx=0x7efd68009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44139402b2675ca6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:56.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd75ffb700 1 -- 172.21.15.7:0/3136034977 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6800b870 con 0x7efd780caa30 2023-03-03T17:10:56.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd75ffb700 1 -- 172.21.15.7:0/3136034977 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6800b9d0 con 0x7efd780caa30 2023-03-03T17:10:56.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.417+0000 7efd75ffb700 1 -- 172.21.15.7:0/3136034977 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6800bd60 con 0x7efd780caa30 2023-03-03T17:10:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.418+0000 7efd7d611700 1 -- 172.21.15.7:0/3136034977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780caa30 msgr2=0x7efd780c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.418+0000 7efd7d611700 1 --2- 172.21.15.7:0/3136034977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780caa30 0x7efd780c5c60 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7efd78158a70 tx=0x7efd68009bd0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.418+0000 7efd7d611700 1 -- 172.21.15.7:0/3136034977 shutdown_connections 2023-03-03T17:10:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.418+0000 7efd7d611700 1 --2- 172.21.15.7:0/3136034977 >> v2:172.21.15.143:3300/0 conn(0x7efd780cae00 0x7efd780c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.418+0000 7efd7d611700 1 --2- 172.21.15.7:0/3136034977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780caa30 0x7efd780c5c60 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.418+0000 7efd7d611700 1 --2- 172.21.15.7:0/3136034977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd780c00e0 0x7efd78154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.418+0000 7efd7d611700 1 -- 172.21.15.7:0/3136034977 >> 172.21.15.7:0/3136034977 conn(0x7efd78066e40 msgr2=0x7efd78071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:10:56.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.418+0000 7efd7d611700 1 -- 172.21.15.7:0/3136034977 shutdown_connections 2023-03-03T17:10:56.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.418+0000 7efd7d611700 1 -- 172.21.15.7:0/3136034977 wait complete. 2023-03-03T17:10:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.419+0000 7efd7d611700 1 Processor -- start 2023-03-03T17:10:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.419+0000 7efd7d611700 1 -- start start 2023-03-03T17:10:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd7d611700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd780c00e0 0x7efd78155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd7d611700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd780caa30 0x7efd780c3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd7d611700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780cae00 0x7efd780c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd7d611700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd781ed7d0 con 0x7efd780cae00 2023-03-03T17:10:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd7d611700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd781562c0 con 0x7efd780c00e0 2023-03-03T17:10:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd7d611700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd780c2bd0 con 0x7efd780caa30 2023-03-03T17:10:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd777fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780cae00 0x7efd780c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd777fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780cae00 0x7efd780c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60272/0 (socket says 172.21.15.7:60272) 2023-03-03T17:10:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd777fe700 1 -- 172.21.15.7:0/404923778 learned_addr learned my addr 172.21.15.7:0/404923778 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:10:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd76ffd700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd780c00e0 0x7efd78155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.420+0000 7efd767fc700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd780caa30 0x7efd780c3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd777fe700 1 -- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd780caa30 msgr2=0x7efd780c3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd777fe700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd780caa30 0x7efd780c3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd777fe700 1 -- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd780c00e0 msgr2=0x7efd78155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd777fe700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd780c00e0 0x7efd78155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd777fe700 1 -- 172.21.15.7:0/404923778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68009550 con 0x7efd780cae00 2023-03-03T17:10:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd777fe700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780cae00 0x7efd780c3ed0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7efd70007e40 tx=0x7efd7000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd5f7fe700 1 -- 172.21.15.7:0/404923778 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7000ed30 con 0x7efd780cae00 2023-03-03T17:10:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd5f7fe700 1 -- 172.21.15.7:0/404923778 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd70014420 con 0x7efd780cae00 2023-03-03T17:10:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd7d611700 1 -- 172.21.15.7:0/404923778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd78154ad0 con 0x7efd780cae00 2023-03-03T17:10:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd5f7fe700 1 -- 172.21.15.7:0/404923778 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd700147d0 con 0x7efd780cae00 2023-03-03T17:10:56.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.421+0000 7efd7d611700 1 -- 172.21.15.7:0/404923778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd781547f0 con 0x7efd780cae00 2023-03-03T17:10:56.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.422+0000 7efd7d611700 1 -- 172.21.15.7:0/404923778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd780bf650 con 0x7efd780cae00 2023-03-03T17:10:56.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.422+0000 7efd5f7fe700 1 -- 172.21.15.7:0/404923778 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efd70012050 con 0x7efd780cae00 2023-03-03T17:10:56.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.423+0000 7efd5f7fe700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd640682f0 0x7efd6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:10:56.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.423+0000 7efd5f7fe700 1 -- 172.21.15.7:0/404923778 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efd70086670 con 0x7efd780cae00 2023-03-03T17:10:56.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.423+0000 7efd76ffd700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd640682f0 0x7efd6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:10:56.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.423+0000 7efd76ffd700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd640682f0 0x7efd6406a7a0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7efd68000c00 tx=0x7efd68008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:10:56.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.427+0000 7efd5f7fe700 1 -- 172.21.15.7:0/404923778 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efd70052a10 con 0x7efd780cae00 2023-03-03T17:10:56.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.664+0000 7efd7d611700 1 -- 172.21.15.7:0/404923778 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd780c7c50 con 0x7efd780cae00 2023-03-03T17:10:56.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.665+0000 7efd5f7fe700 1 -- 172.21.15.7:0/404923778 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efd70056030 con 0x7efd780cae00 2023-03-03T17:10:56.666 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:10:56.666 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:10:56.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 -- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd640682f0 msgr2=0x7efd6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:56.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd640682f0 0x7efd6406a7a0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7efd68000c00 tx=0x7efd68008030 comp rx=0 tx=0).stop 2023-03-03T17:10:56.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 -- 172.21.15.7:0/404923778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780cae00 msgr2=0x7efd780c3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:10:56.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780cae00 0x7efd780c3ed0 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7efd70007e40 tx=0x7efd7000e3c0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 -- 172.21.15.7:0/404923778 shutdown_connections 2023-03-03T17:10:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd640682f0 0x7efd6406a7a0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd780cae00 0x7efd780c3ed0 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd780c00e0 0x7efd78155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 --2- 172.21.15.7:0/404923778 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efd780caa30 0x7efd780c3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:10:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 -- 172.21.15.7:0/404923778 >> 172.21.15.7:0/404923778 conn(0x7efd78066e40 msgr2=0x7efd78153cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:10:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 -- 172.21.15.7:0/404923778 shutdown_connections 2023-03-03T17:10:56.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:10:56.667+0000 7efd7d611700 1 -- 172.21.15.7:0/404923778 wait complete. 2023-03-03T17:10:56.670 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:10:56.680 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T17:11:23.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:23.870 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:53.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:53.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:11:57.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:11:57.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.750+0000 7f8225473700 1 Processor -- start 2023-03-03T17:11:57.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.750+0000 7f8225473700 1 -- start start 2023-03-03T17:11:57.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.751+0000 7f8225473700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f822014b0d0 0x7f8220071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:11:57.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.751+0000 7f8225473700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f82200aad20 0x7f8220157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:11:57.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.751+0000 7f8225473700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82200a9c20 con 0x7f8220071d40 2023-03-03T17:11:57.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.751+0000 7f8225473700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8220154f10 con 0x7f82200aad20 2023-03-03T17:11:57.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.751+0000 7f8225473700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82200ac3c0 con 0x7f822014b0d0 2023-03-03T17:11:57.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.751+0000 7f821f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8220071d40 0x7f8220158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54302/0 (socket says 172.21.15.7:54302) 2023-03-03T17:11:57.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.751+0000 7f821f7fe700 1 -- 172.21.15.7:0/253986905 learned_addr learned my addr 172.21.15.7:0/253986905 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:11:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821e7fc700 1 --2- 172.21.15.7:0/253986905 >> v2:172.21.15.143:3300/0 conn(0x7f82200aad20 0x7f8220157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:11:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821effd700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f822014b0d0 0x7f8220071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:11:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775936080 0 0) 0x7f82200a9c20 con 0x7f8220071d40 2023-03-03T17:11:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82080035b0 con 0x7f8220071d40 2023-03-03T17:11:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 715568037 0 0) 0x7f82080035b0 con 0x7f8220071d40 2023-03-03T17:11:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f822014b0d0 msgr2=0x7f8220071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:11:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f822014b0d0 0x7f8220071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:11:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 >> v2:172.21.15.143:3300/0 conn(0x7f82200aad20 msgr2=0x7f8220157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:11:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 --2- 172.21.15.7:0/253986905 >> v2:172.21.15.143:3300/0 conn(0x7f82200aad20 0x7f8220157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:11:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822006fc70 con 0x7f8220071d40 2023-03-03T17:11:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f82180019f0 con 0x7f8220071d40 2023-03-03T17:11:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821e7fc700 1 --2- 172.21.15.7:0/253986905 >> v2:172.21.15.143:3300/0 conn(0x7f82200aad20 0x7f8220157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:11:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 >> v1:172.21.15.7:6789/0 conn(0x7f8220071d40 legacy=0x7f8220158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:11:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8208003a70 0x7f8208003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:11:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f822014b0d0 0x7f8208004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:11:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8208004d90 0x7f8208005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:11:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8220154f10 con 0x7f8208004d90 2023-03-03T17:11:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82200ac3c0 con 0x7f8208003a70 2023-03-03T17:11:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820800ba60 con 0x7f822014b0d0 2023-03-03T17:11:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821f7fe700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8208004d90 0x7f8208005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:11:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821effd700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8208003a70 0x7f8208003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:11:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821e7fc700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f822014b0d0 0x7f8208004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:11:57.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821f7fe700 1 -- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f822014b0d0 msgr2=0x7f8208004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:11:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821f7fe700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f822014b0d0 0x7f8208004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:11:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821f7fe700 1 -- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8208003a70 msgr2=0x7f8208003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:11:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.752+0000 7f821f7fe700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8208003a70 0x7f8208003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:11:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f821f7fe700 1 -- 172.21.15.7:0/253986905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82080035b0 con 0x7f8208004d90 2023-03-03T17:11:57.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f821f7fe700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8208004d90 0x7f8208005200 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f821800a900 tx=0x7f8208005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9cbdf4b6835f047 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:11:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821800a340 con 0x7f8208004d90 2023-03-03T17:11:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8218012620 con 0x7f8208004d90 2023-03-03T17:11:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f821dffb700 1 -- 172.21.15.7:0/253986905 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82180129d0 con 0x7f8208004d90 2023-03-03T17:11:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f8225473700 1 -- 172.21.15.7:0/253986905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8208004d90 msgr2=0x7f8208005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:11:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f8225473700 1 --2- 172.21.15.7:0/253986905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8208004d90 0x7f8208005200 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7f821800a900 tx=0x7f8208005880 comp rx=0 tx=0).stop 2023-03-03T17:11:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f821f7fe700 1 -- 172.21.15.7:0/253986905 reap_dead start 2023-03-03T17:11:57.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f8225473700 1 -- 172.21.15.7:0/253986905 shutdown_connections 2023-03-03T17:11:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f8225473700 1 -- 172.21.15.7:0/253986905 >> 172.21.15.7:0/253986905 conn(0x7f82200c8900 msgr2=0x7f82200c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:11:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f8225473700 1 -- 172.21.15.7:0/253986905 shutdown_connections 2023-03-03T17:11:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.753+0000 7f8225473700 1 -- 172.21.15.7:0/253986905 wait complete. 2023-03-03T17:11:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.755+0000 7f8225473700 1 Processor -- start 2023-03-03T17:11:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.755+0000 7f8225473700 1 -- start start 2023-03-03T17:11:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.755+0000 7f8225473700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8208004d90 0x7f82201de170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:11:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.755+0000 7f8225473700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82201459c0 0x7f82201eda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:11:57.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f8225473700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82201e9ec0 0x7f82201edfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:11:57.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f8225473700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82201e1680 con 0x7f82201e9ec0 2023-03-03T17:11:57.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f8225473700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82201ed5f0 con 0x7f8208004d90 2023-03-03T17:11:57.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f8225473700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8220154410 con 0x7f82201459c0 2023-03-03T17:11:57.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82201e9ec0 0x7f82201edfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:11:57.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82201e9ec0 0x7f82201edfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60334/0 (socket says 172.21.15.7:60334) 2023-03-03T17:11:57.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821f7fe700 1 -- 172.21.15.7:0/3548099032 learned_addr learned my addr 172.21.15.7:0/3548099032 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:11:57.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821effd700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8208004d90 0x7f82201de170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:11:57.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821e7fc700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82201459c0 0x7f82201eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:11:57.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821f7fe700 1 -- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82201459c0 msgr2=0x7f82201eda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:11:57.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821f7fe700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82201459c0 0x7f82201eda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:11:57.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821f7fe700 1 -- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8208004d90 msgr2=0x7f82201de170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:11:57.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821f7fe700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8208004d90 0x7f82201de170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:11:57.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821f7fe700 1 -- 172.21.15.7:0/3548099032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822006f6b0 con 0x7f82201e9ec0 2023-03-03T17:11:57.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f821f7fe700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82201e9ec0 0x7f82201edfc0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f8208004790 tx=0x7f8218001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:11:57.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f82077fe700 1 -- 172.21.15.7:0/3548099032 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821800a340 con 0x7f82201e9ec0 2023-03-03T17:11:57.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f82077fe700 1 -- 172.21.15.7:0/3548099032 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82180019b0 con 0x7f82201e9ec0 2023-03-03T17:11:57.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f8225473700 1 -- 172.21.15.7:0/3548099032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8220155160 con 0x7f82201e9ec0 2023-03-03T17:11:57.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f82077fe700 1 -- 172.21.15.7:0/3548099032 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821801e620 con 0x7f82201e9ec0 2023-03-03T17:11:57.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.756+0000 7f8225473700 1 -- 172.21.15.7:0/3548099032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822006f990 con 0x7f82201e9ec0 2023-03-03T17:11:57.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.757+0000 7f8225473700 1 -- 172.21.15.7:0/3548099032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8220074bb0 con 0x7f82201e9ec0 2023-03-03T17:11:57.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.758+0000 7f82077fe700 1 -- 172.21.15.7:0/3548099032 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f821802e070 con 0x7f82201e9ec0 2023-03-03T17:11:57.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.758+0000 7f82077fe700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f820c0682f0 0x7f820c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:11:57.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.758+0000 7f82077fe700 1 -- 172.21.15.7:0/3548099032 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8218088000 con 0x7f82201e9ec0 2023-03-03T17:11:57.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.758+0000 7f821effd700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f820c0682f0 0x7f820c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:11:57.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.759+0000 7f821effd700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f820c0682f0 0x7f820c06a7a0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f82201536a0 tx=0x7f8214009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:11:57.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:57.763+0000 7f82077fe700 1 -- 172.21.15.7:0/3548099032 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f82180543a0 con 0x7f82201e9ec0 2023-03-03T17:11:58.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.001+0000 7f8225473700 1 -- 172.21.15.7:0/3548099032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82200aa540 con 0x7f82201e9ec0 2023-03-03T17:11:58.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.001+0000 7f82077fe700 1 -- 172.21.15.7:0/3548099032 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f821802c090 con 0x7f82201e9ec0 2023-03-03T17:11:58.002 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:11:58.002 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:11:58.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.003+0000 7f8225473700 1 -- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f820c0682f0 msgr2=0x7f820c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:11:58.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.003+0000 7f8225473700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f820c0682f0 0x7f820c06a7a0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f82201536a0 tx=0x7f8214009210 comp rx=0 tx=0).stop 2023-03-03T17:11:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.003+0000 7f8225473700 1 -- 172.21.15.7:0/3548099032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82201e9ec0 msgr2=0x7f82201edfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:11:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.003+0000 7f8225473700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82201e9ec0 0x7f82201edfc0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7f8208004790 tx=0x7f8218001360 comp rx=0 tx=0).stop 2023-03-03T17:11:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.003+0000 7f8225473700 1 -- 172.21.15.7:0/3548099032 shutdown_connections 2023-03-03T17:11:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.003+0000 7f8225473700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f820c0682f0 0x7f820c06a7a0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:11:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.003+0000 7f8225473700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f82201e9ec0 0x7f82201edfc0 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:11:58.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.003+0000 7f8225473700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8208004d90 0x7f82201de170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:11:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.003+0000 7f8225473700 1 --2- 172.21.15.7:0/3548099032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f82201459c0 0x7f82201eda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:11:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.003+0000 7f8225473700 1 -- 172.21.15.7:0/3548099032 >> 172.21.15.7:0/3548099032 conn(0x7f82200c8900 msgr2=0x7f82200ac560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:11:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.004+0000 7f8225473700 1 -- 172.21.15.7:0/3548099032 shutdown_connections 2023-03-03T17:11:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:11:58.004+0000 7f8225473700 1 -- 172.21.15.7:0/3548099032 wait complete. 2023-03-03T17:11:58.006 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:11:58.021 INFO:tasks.mds_thrash.fs.[a]:waiting for 76 secs before thrashing 2023-03-03T17:12:24.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:24.018 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:52.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:12:52.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.821+0000 7f7666742700 1 Processor -- start 2023-03-03T17:12:52.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.821+0000 7f7666742700 1 -- start start 2023-03-03T17:12:52.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f7666742700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76600aad20 0x7f7660157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:52.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f7666742700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aa030 0x7f7660158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:52.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f7666742700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76600a9c20 con 0x7f766014b0d0 2023-03-03T17:12:52.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f7666742700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7660154f10 con 0x7f76600aad20 2023-03-03T17:12:52.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f7666742700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76600ac3c0 con 0x7f76600aa030 2023-03-03T17:12:52.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f765f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76600aad20 0x7f7660157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:52.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f765f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f76600aad20 0x7f7660157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60368/0 (socket says 172.21.15.7:60368) 2023-03-03T17:12:52.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f765f7fe700 1 -- 172.21.15.7:0/1791547072 learned_addr learned my addr 172.21.15.7:0/1791547072 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:12:52.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f765f7fe700 1 -- 172.21.15.7:0/1791547072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aa030 msgr2=0x7f7660158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:12:52.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f7664cdf700 1 --2- 172.21.15.7:0/1791547072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aa030 0x7f7660158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:52.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f765f7fe700 1 --2- 172.21.15.7:0/1791547072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aa030 0x7f7660158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:52.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f765f7fe700 1 -- 172.21.15.7:0/1791547072 >> v1:172.21.15.143:6789/0 conn(0x7f766014b0d0 legacy=0x7f7660071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:52.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f765f7fe700 1 -- 172.21.15.7:0/1791547072 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766006fc70 con 0x7f76600aad20 2023-03-03T17:12:52.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.822+0000 7f7664cdf700 1 --2- 172.21.15.7:0/1791547072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aa030 0x7f7660158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:12:52.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f765f7fe700 1 --2- 172.21.15.7:0/1791547072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76600aad20 0x7f7660157df0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f76540098f0 tx=0x7f7654009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be40e14bb20dd450 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:12:52.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f765effd700 1 -- 172.21.15.7:0/1791547072 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765400b8a0 con 0x7f76600aad20 2023-03-03T17:12:52.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f765effd700 1 -- 172.21.15.7:0/1791547072 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765400ba00 con 0x7f76600aad20 2023-03-03T17:12:52.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f765effd700 1 -- 172.21.15.7:0/1791547072 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765400bd90 con 0x7f76600aad20 2023-03-03T17:12:52.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f7666742700 1 -- 172.21.15.7:0/1791547072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76600aad20 msgr2=0x7f7660157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:52.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f7666742700 1 --2- 172.21.15.7:0/1791547072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76600aad20 0x7f7660157df0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f76540098f0 tx=0x7f7654009c00 comp rx=0 tx=0).stop 2023-03-03T17:12:52.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f7666742700 1 -- 172.21.15.7:0/1791547072 shutdown_connections 2023-03-03T17:12:52.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f7666742700 1 --2- 172.21.15.7:0/1791547072 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aa030 0x7f7660158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:52.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f7666742700 1 --2- 172.21.15.7:0/1791547072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76600aad20 0x7f7660157df0 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:52.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f7666742700 1 -- 172.21.15.7:0/1791547072 >> 172.21.15.7:0/1791547072 conn(0x7f76600c8900 msgr2=0x7f76600c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:12:52.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.823+0000 7f7666742700 1 -- 172.21.15.7:0/1791547072 shutdown_connections 2023-03-03T17:12:52.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.824+0000 7f7666742700 1 -- 172.21.15.7:0/1791547072 wait complete. 2023-03-03T17:12:52.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.825+0000 7f7666742700 1 Processor -- start 2023-03-03T17:12:52.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.825+0000 7f7666742700 1 -- start start 2023-03-03T17:12:52.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.825+0000 7f7666742700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76600aa030 0x7f7660146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:52.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.825+0000 7f7666742700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aad20 0x7f7660155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:52.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.825+0000 7f7666742700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766014b0d0 0x7f7660155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:52.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.825+0000 7f7666742700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7660152900 con 0x7f766014b0d0 2023-03-03T17:12:52.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.825+0000 7f7666742700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7660072030 con 0x7f76600aa030 2023-03-03T17:12:52.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.825+0000 7f7666742700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7660154240 con 0x7f76600aad20 2023-03-03T17:12:52.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7664cdf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766014b0d0 0x7f7660155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:52.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7664cdf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766014b0d0 0x7f7660155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60376/0 (socket says 172.21.15.7:60376) 2023-03-03T17:12:52.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7664cdf700 1 -- 172.21.15.7:0/2217269274 learned_addr learned my addr 172.21.15.7:0/2217269274 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:12:52.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f765f7fe700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aad20 0x7f7660155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:52.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f765ffff700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76600aa030 0x7f7660146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:52.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7664cdf700 1 -- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aad20 msgr2=0x7f7660155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:52.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7664cdf700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aad20 0x7f7660155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:52.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7664cdf700 1 -- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76600aa030 msgr2=0x7f7660146e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:52.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7664cdf700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76600aa030 0x7f7660146e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:52.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7664cdf700 1 -- 172.21.15.7:0/2217269274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654009550 con 0x7f766014b0d0 2023-03-03T17:12:52.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f765ffff700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76600aa030 0x7f7660146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:12:52.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7664cdf700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766014b0d0 0x7f7660155dc0 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f765800cef0 tx=0x7f765800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:12:52.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f765cff9700 1 -- 172.21.15.7:0/2217269274 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765800adb0 con 0x7f766014b0d0 2023-03-03T17:12:52.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f765cff9700 1 -- 172.21.15.7:0/2217269274 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7658015440 con 0x7f766014b0d0 2023-03-03T17:12:52.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7666742700 1 -- 172.21.15.7:0/2217269274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766006f6b0 con 0x7f766014b0d0 2023-03-03T17:12:52.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f765cff9700 1 -- 172.21.15.7:0/2217269274 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76580157f0 con 0x7f766014b0d0 2023-03-03T17:12:52.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.826+0000 7f7666742700 1 -- 172.21.15.7:0/2217269274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7660155160 con 0x7f766014b0d0 2023-03-03T17:12:52.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.827+0000 7f7666742700 1 -- 172.21.15.7:0/2217269274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766006f990 con 0x7f766014b0d0 2023-03-03T17:12:52.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.828+0000 7f765cff9700 1 -- 172.21.15.7:0/2217269274 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f765801c030 con 0x7f766014b0d0 2023-03-03T17:12:52.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.828+0000 7f765cff9700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f764c0682f0 0x7f764c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:12:52.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.828+0000 7f765cff9700 1 -- 172.21.15.7:0/2217269274 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7658086570 con 0x7f766014b0d0 2023-03-03T17:12:52.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.828+0000 7f765ffff700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f764c0682f0 0x7f764c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:12:52.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.829+0000 7f765ffff700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f764c0682f0 0x7f764c06a7a0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f7650000f60 tx=0x7f7650009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:12:52.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:52.833+0000 7f765cff9700 1 -- 172.21.15.7:0/2217269274 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7658052910 con 0x7f766014b0d0 2023-03-03T17:12:53.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.070+0000 7f7666742700 1 -- 172.21.15.7:0/2217269274 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7660074bb0 con 0x7f766014b0d0 2023-03-03T17:12:53.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.071+0000 7f765cff9700 1 -- 172.21.15.7:0/2217269274 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7658055f30 con 0x7f766014b0d0 2023-03-03T17:12:53.071 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:12:53.072 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:12:53.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 -- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f764c0682f0 msgr2=0x7f764c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:53.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f764c0682f0 0x7f764c06a7a0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f7650000f60 tx=0x7f7650009040 comp rx=0 tx=0).stop 2023-03-03T17:12:53.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 -- 172.21.15.7:0/2217269274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766014b0d0 msgr2=0x7f7660155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:12:53.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766014b0d0 0x7f7660155dc0 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7f765800cef0 tx=0x7f765800cf20 comp rx=0 tx=0).stop 2023-03-03T17:12:53.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 -- 172.21.15.7:0/2217269274 shutdown_connections 2023-03-03T17:12:53.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f764c0682f0 0x7f764c06a7a0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:53.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766014b0d0 0x7f7660155dc0 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:53.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76600aa030 0x7f7660146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:53.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 --2- 172.21.15.7:0/2217269274 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f76600aad20 0x7f7660155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:12:53.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 -- 172.21.15.7:0/2217269274 >> 172.21.15.7:0/2217269274 conn(0x7f76600c8900 msgr2=0x7f7660157cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:12:53.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 -- 172.21.15.7:0/2217269274 shutdown_connections 2023-03-03T17:12:53.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:12:53.073+0000 7f7666742700 1 -- 172.21.15.7:0/2217269274 wait complete. 2023-03-03T17:12:53.075 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:12:53.085 INFO:tasks.mds_thrash.fs.[b]:waiting for 41 secs before thrashing 2023-03-03T17:12:54.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:12:54.090 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:14.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:13:14.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.161+0000 7fed11c7b700 1 Processor -- start 2023-03-03T17:13:14.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.161+0000 7fed11c7b700 1 -- start start 2023-03-03T17:13:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.161+0000 7fed11c7b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fed0c0c2950 0x7fed0c157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.161+0000 7fed11c7b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c06cb70 con 0x7fed0c151b20 2023-03-03T17:13:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.161+0000 7fed11c7b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c06ce50 con 0x7fed0c06c480 2023-03-03T17:13:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.161+0000 7fed11c7b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c06c2f0 con 0x7fed0c0c2950 2023-03-03T17:13:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.162+0000 7fed0affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fed0c0c2950 0x7fed0c157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.162+0000 7fed0b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fed0c151b20 0x7fed0c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54376/0 (socket says 172.21.15.7:54376) 2023-03-03T17:13:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.162+0000 7fed0b7fe700 1 -- 172.21.15.7:0/1445580660 learned_addr learned my addr 172.21.15.7:0/1445580660 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:13:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.162+0000 7fed0affd700 1 -- 172.21.15.7:0/1445580660 >> v1:172.21.15.143:6789/0 conn(0x7fed0c06c480 legacy=0x7fed0c158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:13:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.162+0000 7fed0affd700 1 -- 172.21.15.7:0/1445580660 >> v1:172.21.15.7:6789/0 conn(0x7fed0c151b20 legacy=0x7fed0c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.162+0000 7fed0affd700 1 -- 172.21.15.7:0/1445580660 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c06d130 con 0x7fed0c0c2950 2023-03-03T17:13:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.162+0000 7fed0affd700 1 --2- 172.21.15.7:0/1445580660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c0c2950 0x7fed0c157ae0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fed00009840 tx=0x7fed00009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2dbac92bd09d7f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.162+0000 7fed0a7fc700 1 -- 172.21.15.7:0/1445580660 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00004230 con 0x7fed0c0c2950 2023-03-03T17:13:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.162+0000 7fed0a7fc700 1 -- 172.21.15.7:0/1445580660 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed00004390 con 0x7fed0c0c2950 2023-03-03T17:13:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.162+0000 7fed0a7fc700 1 -- 172.21.15.7:0/1445580660 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed000036a0 con 0x7fed0c0c2950 2023-03-03T17:13:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.163+0000 7fed11c7b700 1 -- 172.21.15.7:0/1445580660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c0c2950 msgr2=0x7fed0c157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.163+0000 7fed11c7b700 1 --2- 172.21.15.7:0/1445580660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c0c2950 0x7fed0c157ae0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7fed00009840 tx=0x7fed00009b50 comp rx=0 tx=0).stop 2023-03-03T17:13:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.163+0000 7fed11c7b700 1 -- 172.21.15.7:0/1445580660 shutdown_connections 2023-03-03T17:13:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.163+0000 7fed11c7b700 1 --2- 172.21.15.7:0/1445580660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c0c2950 0x7fed0c157ae0 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.163+0000 7fed11c7b700 1 -- 172.21.15.7:0/1445580660 >> 172.21.15.7:0/1445580660 conn(0x7fed0c0a94d0 msgr2=0x7fed0c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:13:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.163+0000 7fed11c7b700 1 -- 172.21.15.7:0/1445580660 shutdown_connections 2023-03-03T17:13:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.163+0000 7fed11c7b700 1 -- 172.21.15.7:0/1445580660 wait complete. 2023-03-03T17:13:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.164+0000 7fed11c7b700 1 Processor -- start 2023-03-03T17:13:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.164+0000 7fed11c7b700 1 -- start start 2023-03-03T17:13:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed11c7b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed0c06c480 0x7fed0c1ee200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed11c7b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed0c0c2950 0x7fed0c14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed11c7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c151b20 0x7fed0c1ea110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed11c7b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0c6e40 con 0x7fed0c151b20 2023-03-03T17:13:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed11c7b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0c5160 con 0x7fed0c0c2950 2023-03-03T17:13:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed11c7b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c1edd70 con 0x7fed0c06c480 2023-03-03T17:13:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c151b20 0x7fed0c1ea110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c151b20 0x7fed0c1ea110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60406/0 (socket says 172.21.15.7:60406) 2023-03-03T17:13:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0bfff700 1 -- 172.21.15.7:0/1818914650 learned_addr learned my addr 172.21.15.7:0/1818914650 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:13:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0affd700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed0c0c2950 0x7fed0c14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0b7fe700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed0c06c480 0x7fed0c1ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0bfff700 1 -- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed0c06c480 msgr2=0x7fed0c1ee200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0bfff700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed0c06c480 0x7fed0c1ee200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0bfff700 1 -- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed0c0c2950 msgr2=0x7fed0c14e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0bfff700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed0c0c2950 0x7fed0c14e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0bfff700 1 -- 172.21.15.7:0/1818914650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed000094a0 con 0x7fed0c151b20 2023-03-03T17:13:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.165+0000 7fed0b7fe700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed0c06c480 0x7fed0c1ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:13:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.166+0000 7fed0bfff700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c151b20 0x7fed0c1ea110 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7fed0400b9a0 tx=0x7fed0400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.166+0000 7fecf3fff700 1 -- 172.21.15.7:0/1818914650 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0400c930 con 0x7fed0c151b20 2023-03-03T17:13:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.166+0000 7fecf3fff700 1 -- 172.21.15.7:0/1818914650 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0400ca90 con 0x7fed0c151b20 2023-03-03T17:13:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.166+0000 7fecf3fff700 1 -- 172.21.15.7:0/1818914650 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0400ce40 con 0x7fed0c151b20 2023-03-03T17:13:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.166+0000 7fed11c7b700 1 -- 172.21.15.7:0/1818914650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c158730 con 0x7fed0c151b20 2023-03-03T17:13:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.166+0000 7fed11c7b700 1 -- 172.21.15.7:0/1818914650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c06d410 con 0x7fed0c151b20 2023-03-03T17:13:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.167+0000 7fed11c7b700 1 -- 172.21.15.7:0/1818914650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c151ef0 con 0x7fed0c151b20 2023-03-03T17:13:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.167+0000 7fecf3fff700 1 -- 172.21.15.7:0/1818914650 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fed0401b030 con 0x7fed0c151b20 2023-03-03T17:13:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.168+0000 7fecf3fff700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fecf80682f0 0x7fecf806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.168+0000 7fecf3fff700 1 -- 172.21.15.7:0/1818914650 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fed04085d00 con 0x7fed0c151b20 2023-03-03T17:13:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.168+0000 7fed0b7fe700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fecf80682f0 0x7fecf806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.169+0000 7fed0b7fe700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fecf80682f0 0x7fecf806a7a0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fed0c14ee00 tx=0x7fecfc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.172+0000 7fecf3fff700 1 -- 172.21.15.7:0/1818914650 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fed04051f50 con 0x7fed0c151b20 2023-03-03T17:13:14.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.410+0000 7fed11c7b700 1 -- 172.21.15.7:0/1818914650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed0c074bb0 con 0x7fed0c151b20 2023-03-03T17:13:14.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.411+0000 7fecf3fff700 1 -- 172.21.15.7:0/1818914650 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fed04055570 con 0x7fed0c151b20 2023-03-03T17:13:14.412 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:13:14.412 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:13:14.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 -- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fecf80682f0 msgr2=0x7fecf806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:14.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fecf80682f0 0x7fecf806a7a0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fed0c14ee00 tx=0x7fecfc009040 comp rx=0 tx=0).stop 2023-03-03T17:13:14.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 -- 172.21.15.7:0/1818914650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c151b20 msgr2=0x7fed0c1ea110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:14.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c151b20 0x7fed0c1ea110 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7fed0400b9a0 tx=0x7fed0400bcb0 comp rx=0 tx=0).stop 2023-03-03T17:13:14.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 -- 172.21.15.7:0/1818914650 shutdown_connections 2023-03-03T17:13:14.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fecf80682f0 0x7fecf806a7a0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:14.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed0c151b20 0x7fed0c1ea110 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:14.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fed0c06c480 0x7fed0c1ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:14.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 --2- 172.21.15.7:0/1818914650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed0c0c2950 0x7fed0c14e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:14.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 -- 172.21.15.7:0/1818914650 >> 172.21.15.7:0/1818914650 conn(0x7fed0c0a94d0 msgr2=0x7fed0c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:13:14.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 -- 172.21.15.7:0/1818914650 shutdown_connections 2023-03-03T17:13:14.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:14.413+0000 7fed11c7b700 1 -- 172.21.15.7:0/1818914650 wait complete. 2023-03-03T17:13:14.416 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:13:14.426 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T17:13:24.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:24.161 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:34.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:13:34.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.226+0000 7fc2d9fbf700 1 Processor -- start 2023-03-03T17:13:34.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.226+0000 7fc2d9fbf700 1 -- start start 2023-03-03T17:13:34.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d9fbf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d40c8020 0x7fc2d40c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:34.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d9fbf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d40c7b90 con 0x7fc2d40c1da0 2023-03-03T17:13:34.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d9fbf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d4154640 con 0x7fc2d406d4b0 2023-03-03T17:13:34.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d9fbf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40c7330 con 0x7fc2d40c8020 2023-03-03T17:13:34.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d3fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc2d40c1da0 0x7fc2d4139d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54422/0 (socket says 172.21.15.7:54422) 2023-03-03T17:13:34.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d3fff700 1 -- 172.21.15.7:0/1180463768 learned_addr learned my addr 172.21.15.7:0/1180463768 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:13:34.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d2ffd700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d40c8020 0x7fc2d40c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:34.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781891784 0 0) 0x7fc2d40c7b90 con 0x7fc2d40c1da0 2023-03-03T17:13:34.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c00035b0 con 0x7fc2d40c1da0 2023-03-03T17:13:34.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143981012 0 0) 0x7fc2d4154640 con 0x7fc2d406d4b0 2023-03-03T17:13:34.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.227+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d40c7b90 con 0x7fc2d406d4b0 2023-03-03T17:13:34.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2242215275 0 0) 0x7fc2c00035b0 con 0x7fc2d40c1da0 2023-03-03T17:13:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d40c8020 msgr2=0x7fc2d40c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:13:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d40c8020 0x7fc2d40c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 >> v1:172.21.15.143:6789/0 conn(0x7fc2d406d4b0 legacy=0x7fc2d406d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d40c7540 con 0x7fc2d40c1da0 2023-03-03T17:13:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc2cc0019f0 con 0x7fc2d40c1da0 2023-03-03T17:13:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 >> v1:172.21.15.7:6789/0 conn(0x7fc2d40c1da0 legacy=0x7fc2d4139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2c0003c30 0x7fc2c0004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 0x7fc2c0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d40c8020 0x7fc2c00052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40c7330 con 0x7fc2c0004850 2023-03-03T17:13:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c000bbd0 con 0x7fc2c0003c30 2023-03-03T17:13:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c000bd90 con 0x7fc2d40c8020 2023-03-03T17:13:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d2ffd700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 0x7fc2c0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d37fe700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2c0003c30 0x7fc2c0004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:34.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d3fff700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d40c8020 0x7fc2c00052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:34.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d2ffd700 1 -- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d40c8020 msgr2=0x7fc2c00052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:13:34.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d2ffd700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d40c8020 0x7fc2c00052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:34.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d2ffd700 1 -- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2c0003c30 msgr2=0x7fc2c0004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:13:34.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d2ffd700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2c0003c30 0x7fc2c0004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:34.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d2ffd700 1 -- 172.21.15.7:0/1180463768 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c00035b0 con 0x7fc2c0004850 2023-03-03T17:13:34.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d2ffd700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 0x7fc2c0004cc0 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009b30 tx=0x7fc2c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1970470f4ed1d4ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:34.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc00e6c0 con 0x7fc2c0004850 2023-03-03T17:13:34.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc00e820 con 0x7fc2c0004850 2023-03-03T17:13:34.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.228+0000 7fc2d27fc700 1 -- 172.21.15.7:0/1180463768 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc00eb90 con 0x7fc2c0004850 2023-03-03T17:13:34.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.229+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/1180463768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 msgr2=0x7fc2c0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:34.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.229+0000 7fc2d9fbf700 1 --2- 172.21.15.7:0/1180463768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 0x7fc2c0004cc0 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009b30 tx=0x7fc2c0005880 comp rx=0 tx=0).stop 2023-03-03T17:13:34.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.229+0000 7fc2d3fff700 1 -- 172.21.15.7:0/1180463768 reap_dead start 2023-03-03T17:13:34.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.229+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/1180463768 shutdown_connections 2023-03-03T17:13:34.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.229+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/1180463768 >> 172.21.15.7:0/1180463768 conn(0x7fc2d406c110 msgr2=0x7fc2d406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:13:34.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.229+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/1180463768 shutdown_connections 2023-03-03T17:13:34.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.229+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/1180463768 wait complete. 2023-03-03T17:13:34.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.230+0000 7fc2d9fbf700 1 Processor -- start 2023-03-03T17:13:34.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.230+0000 7fc2d9fbf700 1 -- start start 2023-03-03T17:13:34.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d9fbf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 0x7fc2d40c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:34.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d9fbf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2d413d280 0x7fc2d413d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:34.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d9fbf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d4136b00 0x7fc2d4136f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:34.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d9fbf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40aca50 con 0x7fc2c0004850 2023-03-03T17:13:34.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d9fbf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40aab60 con 0x7fc2d413d280 2023-03-03T17:13:34.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d9fbf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d40701a0 con 0x7fc2d4136b00 2023-03-03T17:13:34.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 0x7fc2d40c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:34.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 0x7fc2d40c0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60450/0 (socket says 172.21.15.7:60450) 2023-03-03T17:13:34.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d37fe700 1 -- 172.21.15.7:0/7725865 learned_addr learned my addr 172.21.15.7:0/7725865 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:13:34.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d2ffd700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2d413d280 0x7fc2d413d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:34.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.231+0000 7fc2d3fff700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d4136b00 0x7fc2d4136f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:34.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2d37fe700 1 -- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d4136b00 msgr2=0x7fc2d4136f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:34.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2d37fe700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d4136b00 0x7fc2d4136f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:34.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2d37fe700 1 -- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2d413d280 msgr2=0x7fc2d413d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:34.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2d37fe700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2d413d280 0x7fc2d413d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:34.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2d37fe700 1 -- 172.21.15.7:0/7725865 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d406e210 con 0x7fc2c0004850 2023-03-03T17:13:34.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2d37fe700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 0x7fc2d40c0a40 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400f7e0 tx=0x7fc2c400faf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:34.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2cbfff700 1 -- 172.21.15.7:0/7725865 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4005320 con 0x7fc2c0004850 2023-03-03T17:13:34.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2cbfff700 1 -- 172.21.15.7:0/7725865 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c40054a0 con 0x7fc2c0004850 2023-03-03T17:13:34.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/7725865 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d40c7820 con 0x7fc2c0004850 2023-03-03T17:13:34.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2cbfff700 1 -- 172.21.15.7:0/7725865 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c4006aa0 con 0x7fc2c0004850 2023-03-03T17:13:34.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.232+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/7725865 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d413a490 con 0x7fc2c0004850 2023-03-03T17:13:34.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.233+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/7725865 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d40704d0 con 0x7fc2c0004850 2023-03-03T17:13:34.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.233+0000 7fc2cbfff700 1 -- 172.21.15.7:0/7725865 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc2c401c030 con 0x7fc2c0004850 2023-03-03T17:13:34.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.234+0000 7fc2cbfff700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2b40682f0 0x7fc2b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:34.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.234+0000 7fc2cbfff700 1 -- 172.21.15.7:0/7725865 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc2c40858e0 con 0x7fc2c0004850 2023-03-03T17:13:34.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.234+0000 7fc2d2ffd700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2b40682f0 0x7fc2b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:34.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.235+0000 7fc2d2ffd700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2b40682f0 0x7fc2b406a7a0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0090b0 tx=0x7fc2bc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:34.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.238+0000 7fc2cbfff700 1 -- 172.21.15.7:0/7725865 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc2c4051c00 con 0x7fc2c0004850 2023-03-03T17:13:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.476+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/7725865 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2d4071ea0 con 0x7fc2c0004850 2023-03-03T17:13:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.477+0000 7fc2cbfff700 1 -- 172.21.15.7:0/7725865 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc2c4055220 con 0x7fc2c0004850 2023-03-03T17:13:34.477 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:13:34.477 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:13:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2b40682f0 msgr2=0x7fc2b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2b40682f0 0x7fc2b406a7a0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0090b0 tx=0x7fc2bc008030 comp rx=0 tx=0).stop 2023-03-03T17:13:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/7725865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 msgr2=0x7fc2d40c0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 0x7fc2d40c0a40 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7fc2c400f7e0 tx=0x7fc2c400faf0 comp rx=0 tx=0).stop 2023-03-03T17:13:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/7725865 shutdown_connections 2023-03-03T17:13:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc2d4136b00 0x7fc2d4136f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc2b40682f0 0x7fc2b406a7a0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc2c0004850 0x7fc2d40c0a40 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 --2- 172.21.15.7:0/7725865 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc2d413d280 0x7fc2d413d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/7725865 >> 172.21.15.7:0/7725865 conn(0x7fc2d406c110 msgr2=0x7fc2d406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:13:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/7725865 shutdown_connections 2023-03-03T17:13:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:34.479+0000 7fc2d9fbf700 1 -- 172.21.15.7:0/7725865 wait complete. 2023-03-03T17:13:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:13:34.491 INFO:tasks.mds_thrash.fs.[b]:waiting for 13 secs before thrashing 2023-03-03T17:13:47.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:13:47.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.635+0000 7f57c1aba700 1 Processor -- start 2023-03-03T17:13:47.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.635+0000 7f57c1aba700 1 -- start start 2023-03-03T17:13:47.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57c1aba700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f57bc14f9d0 0x7f57bc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57c1aba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57c1aba700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57bc0c2f30 con 0x7f57bc0abe20 2023-03-03T17:13:47.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57c1aba700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57bc0c31d0 con 0x7f57bc14f9d0 2023-03-03T17:13:47.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57c1aba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc1524a0 con 0x7f57bc14fda0 2023-03-03T17:13:47.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57baffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57bb7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f57bc14f9d0 0x7f57bc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57baffd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:40306/0 (socket says 172.21.15.7:40306) 2023-03-03T17:13:47.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57baffd700 1 -- 172.21.15.7:0/1572970825 learned_addr learned my addr 172.21.15.7:0/1572970825 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:13:47.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57bb7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f57bc14f9d0 0x7f57bc156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:56240/0 (socket says 172.21.15.7:56240) 2023-03-03T17:13:47.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.636+0000 7f57bbfff700 1 --1- 172.21.15.7:0/1572970825 >> v1:172.21.15.143:6789/0 conn(0x7f57bc0abe20 0x7f57bc1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:57970/0 (socket says 172.21.15.7:57970) 2023-03-03T17:13:47.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.637+0000 7f57baffd700 1 -- 172.21.15.7:0/1572970825 >> v2:172.21.15.143:3300/0 conn(0x7f57bc14f9d0 msgr2=0x7f57bc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:13:47.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.637+0000 7f57baffd700 1 --2- 172.21.15.7:0/1572970825 >> v2:172.21.15.143:3300/0 conn(0x7f57bc14f9d0 0x7f57bc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.637+0000 7f57baffd700 1 -- 172.21.15.7:0/1572970825 >> v1:172.21.15.143:6789/0 conn(0x7f57bc0abe20 legacy=0x7f57bc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.637+0000 7f57baffd700 1 -- 172.21.15.7:0/1572970825 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc0c34b0 con 0x7f57bc14fda0 2023-03-03T17:13:47.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.637+0000 7f57bb7fe700 1 --2- 172.21.15.7:0/1572970825 >> v2:172.21.15.143:3300/0 conn(0x7f57bc14f9d0 0x7f57bc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:13:47.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.637+0000 7f57baffd700 1 --2- 172.21.15.7:0/1572970825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc071890 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f57a40098f0 tx=0x7f57a4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bc09798533f9c2c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:47.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.637+0000 7f57ba7fc700 1 -- 172.21.15.7:0/1572970825 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4011830 con 0x7f57bc14fda0 2023-03-03T17:13:47.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.637+0000 7f57ba7fc700 1 -- 172.21.15.7:0/1572970825 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a4011990 con 0x7f57bc14fda0 2023-03-03T17:13:47.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.637+0000 7f57ba7fc700 1 -- 172.21.15.7:0/1572970825 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4011d20 con 0x7f57bc14fda0 2023-03-03T17:13:47.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.638+0000 7f57c1aba700 1 -- 172.21.15.7:0/1572970825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 msgr2=0x7f57bc071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.638+0000 7f57c1aba700 1 --2- 172.21.15.7:0/1572970825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc071890 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f57a40098f0 tx=0x7f57a4009c00 comp rx=0 tx=0).stop 2023-03-03T17:13:47.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.638+0000 7f57c1aba700 1 -- 172.21.15.7:0/1572970825 shutdown_connections 2023-03-03T17:13:47.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.638+0000 7f57c1aba700 1 --2- 172.21.15.7:0/1572970825 >> v2:172.21.15.143:3300/0 conn(0x7f57bc14f9d0 0x7f57bc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.638+0000 7f57c1aba700 1 --2- 172.21.15.7:0/1572970825 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc071890 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.638+0000 7f57c1aba700 1 -- 172.21.15.7:0/1572970825 >> 172.21.15.7:0/1572970825 conn(0x7f57bc0c8900 msgr2=0x7f57bc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:13:47.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.638+0000 7f57c1aba700 1 -- 172.21.15.7:0/1572970825 shutdown_connections 2023-03-03T17:13:47.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.638+0000 7f57c1aba700 1 -- 172.21.15.7:0/1572970825 wait complete. 2023-03-03T17:13:47.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.639+0000 7f57c1aba700 1 Processor -- start 2023-03-03T17:13:47.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.639+0000 7f57c1aba700 1 -- start start 2023-03-03T17:13:47.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57c1aba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57bc0abe20 0x7f57bc0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57c1aba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57bc14f9d0 0x7f57bc1dd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57c1aba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc1dda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57c1aba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc06a980 con 0x7f57bc14f9d0 2023-03-03T17:13:47.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57c1aba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc1e9690 con 0x7f57bc0abe20 2023-03-03T17:13:47.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57c1aba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc1db7a0 con 0x7f57bc14fda0 2023-03-03T17:13:47.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57baffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57bc14f9d0 0x7f57bc1dd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57baffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57bc14f9d0 0x7f57bc1dd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60474/0 (socket says 172.21.15.7:60474) 2023-03-03T17:13:47.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57baffd700 1 -- 172.21.15.7:0/1537284086 learned_addr learned my addr 172.21.15.7:0/1537284086 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:13:47.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57bbfff700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc1dda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57baffd700 1 -- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 msgr2=0x7f57bc1dda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57baffd700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc1dda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57bb7fe700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57bc0abe20 0x7f57bc0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57baffd700 1 -- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57bc0abe20 msgr2=0x7f57bc0c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57baffd700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57bc0abe20 0x7f57bc0c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57baffd700 1 -- 172.21.15.7:0/1537284086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac0094a0 con 0x7f57bc14f9d0 2023-03-03T17:13:47.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57bbfff700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc1dda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:13:47.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57bb7fe700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57bc0abe20 0x7f57bc0c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:13:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.640+0000 7f57baffd700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57bc14f9d0 0x7f57bc1dd510 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f57a4009c40 tx=0x7f57a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.641+0000 7f57b3fff700 1 -- 172.21.15.7:0/1537284086 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a40036a0 con 0x7f57bc14f9d0 2023-03-03T17:13:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.641+0000 7f57b3fff700 1 -- 172.21.15.7:0/1537284086 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a4003800 con 0x7f57bc14f9d0 2023-03-03T17:13:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.641+0000 7f57c1aba700 1 -- 172.21.15.7:0/1537284086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a4009550 con 0x7f57bc14f9d0 2023-03-03T17:13:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.641+0000 7f57b3fff700 1 -- 172.21.15.7:0/1537284086 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4003bb0 con 0x7f57bc14f9d0 2023-03-03T17:13:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.641+0000 7f57c1aba700 1 -- 172.21.15.7:0/1537284086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57bc0ac350 con 0x7f57bc14f9d0 2023-03-03T17:13:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.642+0000 7f57c1aba700 1 -- 172.21.15.7:0/1537284086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57bc0ab630 con 0x7f57bc14f9d0 2023-03-03T17:13:47.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.642+0000 7f57b3fff700 1 -- 172.21.15.7:0/1537284086 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f57a402a020 con 0x7f57bc14f9d0 2023-03-03T17:13:47.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.642+0000 7f57b3fff700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f579c0682f0 0x7f579c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:13:47.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.643+0000 7f57b3fff700 1 -- 172.21.15.7:0/1537284086 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f57a4087560 con 0x7f57bc14f9d0 2023-03-03T17:13:47.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.643+0000 7f57bb7fe700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f579c0682f0 0x7f579c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:13:47.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.643+0000 7f57bb7fe700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f579c0682f0 0x7f579c06a7a0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009cf0 tx=0x7f57ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:13:47.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.647+0000 7f57b3fff700 1 -- 172.21.15.7:0/1537284086 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57a4053900 con 0x7f57bc14f9d0 2023-03-03T17:13:47.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.884+0000 7f57c1aba700 1 -- 172.21.15.7:0/1537284086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57bc1dc3b0 con 0x7f57bc14f9d0 2023-03-03T17:13:47.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.885+0000 7f57b3fff700 1 -- 172.21.15.7:0/1537284086 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f57a4056f20 con 0x7f57bc14f9d0 2023-03-03T17:13:47.886 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:13:47.886 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:13:47.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 -- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f579c0682f0 msgr2=0x7f579c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f579c0682f0 0x7f579c06a7a0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009cf0 tx=0x7f57ac008040 comp rx=0 tx=0).stop 2023-03-03T17:13:47.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 -- 172.21.15.7:0/1537284086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57bc14f9d0 msgr2=0x7f57bc1dd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:13:47.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57bc14f9d0 0x7f57bc1dd510 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f57a4009c40 tx=0x7f57a4004750 comp rx=0 tx=0).stop 2023-03-03T17:13:47.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 -- 172.21.15.7:0/1537284086 shutdown_connections 2023-03-03T17:13:47.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f57bc14fda0 0x7f57bc1dda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57bc0abe20 0x7f57bc0c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f579c0682f0 0x7f579c06a7a0 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 --2- 172.21.15.7:0/1537284086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57bc14f9d0 0x7f57bc1dd510 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:13:47.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 -- 172.21.15.7:0/1537284086 >> 172.21.15.7:0/1537284086 conn(0x7f57bc0c8900 msgr2=0x7f57bc0c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:13:47.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 -- 172.21.15.7:0/1537284086 shutdown_connections 2023-03-03T17:13:47.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:13:47.887+0000 7f57c1aba700 1 -- 172.21.15.7:0/1537284086 wait complete. 2023-03-03T17:13:47.890 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:13:47.899 INFO:tasks.mds_thrash.fs.[b]:waiting for 104 secs before thrashing 2023-03-03T17:13:54.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:13:54.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:18.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:14:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.566+0000 7fa0f2f12700 1 Processor -- start 2023-03-03T17:14:18.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.566+0000 7fa0f2f12700 1 -- start start 2023-03-03T17:14:18.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.567+0000 7fa0f2f12700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0c00e0 0x7fa0ec0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.567+0000 7fa0f2f12700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa0ec154ff0 0x7fa0ec0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.567+0000 7fa0f2f12700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec0c7310 con 0x7fa0ec0caa30 2023-03-03T17:14:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.567+0000 7fa0f2f12700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0ec0bfd10 con 0x7fa0ec154ff0 2023-03-03T17:14:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.567+0000 7fa0f2f12700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec0bfa30 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.567+0000 7fa0ebfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0c00e0 0x7fa0ec0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.567+0000 7fa0f14af700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa0ec154ff0 0x7fa0ec0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:18.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.567+0000 7fa0f0cae700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fa0ec0caa30 0x7fa0ec0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:58000/0 (socket says 172.21.15.7:58000) 2023-03-03T17:14:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.567+0000 7fa0ebfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0c00e0 0x7fa0ec0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:40342/0 (socket says 172.21.15.7:40342) 2023-03-03T17:14:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.567+0000 7fa0f0cae700 1 -- 172.21.15.7:0/2788395518 learned_addr learned my addr 172.21.15.7:0/2788395518 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:14:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0f14af700 1 -- 172.21.15.7:0/2788395518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0c00e0 msgr2=0x7fa0ec0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:14:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0f14af700 1 --2- 172.21.15.7:0/2788395518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0c00e0 0x7fa0ec0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0f14af700 1 -- 172.21.15.7:0/2788395518 >> v1:172.21.15.143:6789/0 conn(0x7fa0ec0caa30 legacy=0x7fa0ec0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0f14af700 1 -- 172.21.15.7:0/2788395518 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec154510 con 0x7fa0ec154ff0 2023-03-03T17:14:18.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0eb7fe700 1 -- 172.21.15.7:0/2788395518 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635308129 0 0) 0x7fa0ec0c7310 con 0x7fa0ec0caa30 2023-03-03T17:14:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0ebfff700 1 --2- 172.21.15.7:0/2788395518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0c00e0 0x7fa0ec0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:14:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0f14af700 1 --2- 172.21.15.7:0/2788395518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0ec154ff0 0x7fa0ec0cae00 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009d90 tx=0x7fa0e400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ceabfd01c0324b6a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:14:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0eb7fe700 1 -- 172.21.15.7:0/2788395518 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e400e070 con 0x7fa0ec154ff0 2023-03-03T17:14:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0eb7fe700 1 -- 172.21.15.7:0/2788395518 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e400bb20 con 0x7fa0ec154ff0 2023-03-03T17:14:18.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0eb7fe700 1 -- 172.21.15.7:0/2788395518 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4012430 con 0x7fa0ec154ff0 2023-03-03T17:14:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0f2f12700 1 -- 172.21.15.7:0/2788395518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0ec154ff0 msgr2=0x7fa0ec0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:18.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.568+0000 7fa0f2f12700 1 --2- 172.21.15.7:0/2788395518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0ec154ff0 0x7fa0ec0cae00 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009d90 tx=0x7fa0e400b330 comp rx=0 tx=0).stop 2023-03-03T17:14:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.569+0000 7fa0f2f12700 1 -- 172.21.15.7:0/2788395518 shutdown_connections 2023-03-03T17:14:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.569+0000 7fa0f2f12700 1 --2- 172.21.15.7:0/2788395518 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0ec154ff0 0x7fa0ec0cae00 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.569+0000 7fa0f2f12700 1 --2- 172.21.15.7:0/2788395518 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0c00e0 0x7fa0ec0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:18.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.569+0000 7fa0f2f12700 1 -- 172.21.15.7:0/2788395518 >> 172.21.15.7:0/2788395518 conn(0x7fa0ec066e40 msgr2=0x7fa0ec071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:14:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.569+0000 7fa0f2f12700 1 -- 172.21.15.7:0/2788395518 shutdown_connections 2023-03-03T17:14:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.569+0000 7fa0f2f12700 1 -- 172.21.15.7:0/2788395518 wait complete. 2023-03-03T17:14:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.570+0000 7fa0f2f12700 1 Processor -- start 2023-03-03T17:14:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.570+0000 7fa0f2f12700 1 -- start start 2023-03-03T17:14:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f2f12700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0ec0c00e0 0x7fa0ec152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f2f12700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0caa30 0x7fa0ec158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:18.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f2f12700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0ec154ff0 0x7fa0ec0c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f2f12700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec1e1370 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f2f12700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec0c23f0 con 0x7fa0ec154ff0 2023-03-03T17:14:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f2f12700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec1ddbc0 con 0x7fa0ec0caa30 2023-03-03T17:14:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f0cae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0ec0c00e0 0x7fa0ec152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f0cae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0ec0c00e0 0x7fa0ec152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60506/0 (socket says 172.21.15.7:60506) 2023-03-03T17:14:18.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f0cae700 1 -- 172.21.15.7:0/3117065514 learned_addr learned my addr 172.21.15.7:0/3117065514 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:14:18.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0ebfff700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0caa30 0x7fa0ec158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f14af700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0ec154ff0 0x7fa0ec0c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f0cae700 1 -- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0caa30 msgr2=0x7fa0ec158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f0cae700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0caa30 0x7fa0ec158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f0cae700 1 -- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0ec154ff0 msgr2=0x7fa0ec0c33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f0cae700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0ec154ff0 0x7fa0ec0c33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:18.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f0cae700 1 -- 172.21.15.7:0/3117065514 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d8009430 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0f0cae700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0ec0c00e0 0x7fa0ec152da0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c750 tx=0x7fa0e000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:14:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.571+0000 7fa0e97fa700 1 -- 172.21.15.7:0/3117065514 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0004c80 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.572+0000 7fa0e97fa700 1 -- 172.21.15.7:0/3117065514 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e0004e00 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.572+0000 7fa0f2f12700 1 -- 172.21.15.7:0/3117065514 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e4009a70 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.572+0000 7fa0e97fa700 1 -- 172.21.15.7:0/3117065514 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0007310 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.572+0000 7fa0f2f12700 1 -- 172.21.15.7:0/3117065514 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ec154ad0 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.573+0000 7fa0f2f12700 1 -- 172.21.15.7:0/3117065514 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ec1547f0 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.573+0000 7fa0e97fa700 1 -- 172.21.15.7:0/3117065514 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa0e001a020 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.573+0000 7fa0e97fa700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0d0068320 0x7fa0d006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:14:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.573+0000 7fa0e97fa700 1 -- 172.21.15.7:0/3117065514 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa0e0085d90 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.574+0000 7fa0ebfff700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0d0068320 0x7fa0d006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:14:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.574+0000 7fa0ebfff700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0d0068320 0x7fa0d006a7d0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec153950 tx=0x7fa0d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:14:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.578+0000 7fa0e97fa700 1 -- 172.21.15.7:0/3117065514 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa0e00520d0 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.813+0000 7fa0f2f12700 1 -- 172.21.15.7:0/3117065514 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0ec0bf650 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.813+0000 7fa0e97fa700 1 -- 172.21.15.7:0/3117065514 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa0e00556f0 con 0x7fa0ec0c00e0 2023-03-03T17:14:18.815 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:14:18.815 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:14:18.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.815+0000 7fa0f2f12700 1 -- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0d0068320 msgr2=0x7fa0d006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:18.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.815+0000 7fa0f2f12700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0d0068320 0x7fa0d006a7d0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec153950 tx=0x7fa0d8008040 comp rx=0 tx=0).stop 2023-03-03T17:14:18.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.815+0000 7fa0f2f12700 1 -- 172.21.15.7:0/3117065514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0ec0c00e0 msgr2=0x7fa0ec152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:14:18.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.815+0000 7fa0f2f12700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0ec0c00e0 0x7fa0ec152da0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c750 tx=0x7fa0e000cb10 comp rx=0 tx=0).stop 2023-03-03T17:14:18.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.816+0000 7fa0f2f12700 1 -- 172.21.15.7:0/3117065514 shutdown_connections 2023-03-03T17:14:18.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.816+0000 7fa0f2f12700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0ec154ff0 0x7fa0ec0c33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:18.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.816+0000 7fa0f2f12700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0d0068320 0x7fa0d006a7d0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:18.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.816+0000 7fa0f2f12700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0ec0c00e0 0x7fa0ec152da0 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:18.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.816+0000 7fa0f2f12700 1 --2- 172.21.15.7:0/3117065514 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa0ec0caa30 0x7fa0ec158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:14:18.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.816+0000 7fa0f2f12700 1 -- 172.21.15.7:0/3117065514 >> 172.21.15.7:0/3117065514 conn(0x7fa0ec066e40 msgr2=0x7fa0ec065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:14:18.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.816+0000 7fa0f2f12700 1 -- 172.21.15.7:0/3117065514 shutdown_connections 2023-03-03T17:14:18.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:14:18.816+0000 7fa0f2f12700 1 -- 172.21.15.7:0/3117065514 wait complete. 2023-03-03T17:14:18.819 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:14:18.830 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T17:14:24.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:24.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:54.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:14:54.380 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:23.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:15:23.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.970+0000 7f0538855700 1 Processor -- start 2023-03-03T17:15:23.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.970+0000 7f0538855700 1 -- start start 2023-03-03T17:15:23.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.971+0000 7f0538855700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f053414f9d0 0x7f0534156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:23.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.971+0000 7f0538855700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05340abe20 0x7f05341529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:23.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.971+0000 7f0538855700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05340c2f30 con 0x7f053414fda0 2023-03-03T17:15:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.971+0000 7f0538855700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05340c31d0 con 0x7f053414f9d0 2023-03-03T17:15:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.971+0000 7f0538855700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05341524a0 con 0x7f05340abe20 2023-03-03T17:15:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.971+0000 7f053259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f053414f9d0 0x7f0534156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.971+0000 7f0531d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f053414fda0 0x7f0534071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54516/0 (socket says 172.21.15.7:54516) 2023-03-03T17:15:23.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.971+0000 7f0531d9b700 1 -- 172.21.15.7:0/2455357946 learned_addr learned my addr 172.21.15.7:0/2455357946 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:15:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.972+0000 7f0532d9d700 1 --2- 172.21.15.7:0/2455357946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05340abe20 0x7f05341529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.972+0000 7f053259c700 1 -- 172.21.15.7:0/2455357946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05340abe20 msgr2=0x7f05341529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:15:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.972+0000 7f053259c700 1 --2- 172.21.15.7:0/2455357946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05340abe20 0x7f05341529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.972+0000 7f053259c700 1 -- 172.21.15.7:0/2455357946 >> v1:172.21.15.7:6789/0 conn(0x7f053414fda0 legacy=0x7f0534071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.972+0000 7f053259c700 1 -- 172.21.15.7:0/2455357946 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05340c34b0 con 0x7f053414f9d0 2023-03-03T17:15:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.972+0000 7f053159a700 1 -- 172.21.15.7:0/2455357946 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817652360 0 0) 0x7f05340c2f30 con 0x7f053414fda0 2023-03-03T17:15:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.972+0000 7f053259c700 1 --2- 172.21.15.7:0/2455357946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 0x7f0534156430 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f052c009840 tx=0x7f052c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa735d3a4e32367 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:23.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.972+0000 7f053159a700 1 -- 172.21.15.7:0/2455357946 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c011830 con 0x7f053414f9d0 2023-03-03T17:15:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.972+0000 7f053159a700 1 -- 172.21.15.7:0/2455357946 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c011990 con 0x7f053414f9d0 2023-03-03T17:15:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.972+0000 7f053159a700 1 -- 172.21.15.7:0/2455357946 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c011d20 con 0x7f053414f9d0 2023-03-03T17:15:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.973+0000 7f0538855700 1 -- 172.21.15.7:0/2455357946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 msgr2=0x7f0534156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.973+0000 7f0538855700 1 --2- 172.21.15.7:0/2455357946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 0x7f0534156430 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f052c009840 tx=0x7f052c009b50 comp rx=0 tx=0).stop 2023-03-03T17:15:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.973+0000 7f0538855700 1 -- 172.21.15.7:0/2455357946 shutdown_connections 2023-03-03T17:15:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.973+0000 7f0538855700 1 --2- 172.21.15.7:0/2455357946 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05340abe20 0x7f05341529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:23.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.973+0000 7f0538855700 1 --2- 172.21.15.7:0/2455357946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 0x7f0534156430 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.973+0000 7f0538855700 1 -- 172.21.15.7:0/2455357946 >> 172.21.15.7:0/2455357946 conn(0x7f05340c8900 msgr2=0x7f05340c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:15:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.973+0000 7f0538855700 1 -- 172.21.15.7:0/2455357946 shutdown_connections 2023-03-03T17:15:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.973+0000 7f0538855700 1 -- 172.21.15.7:0/2455357946 wait complete. 2023-03-03T17:15:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.975+0000 7f0538855700 1 Processor -- start 2023-03-03T17:15:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.975+0000 7f0538855700 1 -- start start 2023-03-03T17:15:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.975+0000 7f0538855700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05340abe20 0x7f053414b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:23.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0538855700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 0x7f05340c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0538855700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f053414fda0 0x7f053413ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0538855700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05341dea60 con 0x7f053414f9d0 2023-03-03T17:15:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0538855700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0534155990 con 0x7f053414fda0 2023-03-03T17:15:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0538855700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053414bac0 con 0x7f05340abe20 2023-03-03T17:15:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0531d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 0x7f05340c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0531d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 0x7f05340c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60544/0 (socket says 172.21.15.7:60544) 2023-03-03T17:15:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0531d9b700 1 -- 172.21.15.7:0/3051071326 learned_addr learned my addr 172.21.15.7:0/3051071326 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:15:23.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f053259c700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05340abe20 0x7f053414b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0532d9d700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f053414fda0 0x7f053413ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0531d9b700 1 -- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05340abe20 msgr2=0x7f053414b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0531d9b700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05340abe20 0x7f053414b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0531d9b700 1 -- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f053414fda0 msgr2=0x7f053413ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0531d9b700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f053414fda0 0x7f053413ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0531d9b700 1 -- 172.21.15.7:0/3051071326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c0094a0 con 0x7f053414f9d0 2023-03-03T17:15:23.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0532d9d700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f053414fda0 0x7f053413ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:15:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0531d9b700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 0x7f05340c1a50 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f052800c570 tx=0x7f052800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0522ffd700 1 -- 172.21.15.7:0/3051071326 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528004c80 con 0x7f053414f9d0 2023-03-03T17:15:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0522ffd700 1 -- 172.21.15.7:0/3051071326 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0528004de0 con 0x7f053414f9d0 2023-03-03T17:15:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0538855700 1 -- 172.21.15.7:0/3051071326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05340ac350 con 0x7f053414f9d0 2023-03-03T17:15:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0522ffd700 1 -- 172.21.15.7:0/3051071326 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05280072f0 con 0x7f053414f9d0 2023-03-03T17:15:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.976+0000 7f0538855700 1 -- 172.21.15.7:0/3051071326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05340ab630 con 0x7f053414f9d0 2023-03-03T17:15:23.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.977+0000 7f0538855700 1 -- 172.21.15.7:0/3051071326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0534071fa0 con 0x7f053414f9d0 2023-03-03T17:15:23.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.978+0000 7f0522ffd700 1 -- 172.21.15.7:0/3051071326 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f052801a020 con 0x7f053414f9d0 2023-03-03T17:15:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.978+0000 7f0522ffd700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05140682f0 0x7f051406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.978+0000 7f0522ffd700 1 -- 172.21.15.7:0/3051071326 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0528085da0 con 0x7f053414f9d0 2023-03-03T17:15:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.978+0000 7f053259c700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05140682f0 0x7f051406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.979+0000 7f053259c700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05140682f0 0x7f051406a7a0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f052c0090b0 tx=0x7f052c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:23.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:23.983+0000 7f0522ffd700 1 -- 172.21.15.7:0/3051071326 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0528052140 con 0x7f053414f9d0 2023-03-03T17:15:24.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.218+0000 7f0538855700 1 -- 172.21.15.7:0/3051071326 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05341edf40 con 0x7f053414f9d0 2023-03-03T17:15:24.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.218+0000 7f0522ffd700 1 -- 172.21.15.7:0/3051071326 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0528055760 con 0x7f053414f9d0 2023-03-03T17:15:24.220 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:15:24.220 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:15:24.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 -- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05140682f0 msgr2=0x7f051406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:24.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05140682f0 0x7f051406a7a0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f052c0090b0 tx=0x7f052c008030 comp rx=0 tx=0).stop 2023-03-03T17:15:24.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 -- 172.21.15.7:0/3051071326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 msgr2=0x7f05340c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:24.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 0x7f05340c1a50 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f052800c570 tx=0x7f052800c880 comp rx=0 tx=0).stop 2023-03-03T17:15:24.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 -- 172.21.15.7:0/3051071326 shutdown_connections 2023-03-03T17:15:24.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f053414fda0 0x7f053413ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:24.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05340abe20 0x7f053414b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:24.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05140682f0 0x7f051406a7a0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:24.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 --2- 172.21.15.7:0/3051071326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f053414f9d0 0x7f05340c1a50 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:24.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 -- 172.21.15.7:0/3051071326 >> 172.21.15.7:0/3051071326 conn(0x7f05340c8900 msgr2=0x7f05340c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:15:24.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 -- 172.21.15.7:0/3051071326 shutdown_connections 2023-03-03T17:15:24.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:24.220+0000 7f0538855700 1 -- 172.21.15.7:0/3051071326 wait complete. 2023-03-03T17:15:24.226 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:15:24.233 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2023-03-03T17:15:24.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:24.442 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:31.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:15:32.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.040+0000 7f389f323700 1 Processor -- start 2023-03-03T17:15:32.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.040+0000 7f389f323700 1 -- start start 2023-03-03T17:15:32.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389f323700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3898151e80 0x7f38980c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:32.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389f323700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3898155410 0x7f38980bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:32.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389f323700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38980c08d0 con 0x7f38980c6fe0 2023-03-03T17:15:32.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389f323700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3898152250 con 0x7f3898151e80 2023-03-03T17:15:32.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389f323700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38980c0fb0 con 0x7f3898155410 2023-03-03T17:15:32.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389d8c0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f38980c6fe0 0x7f38980bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54536/0 (socket says 172.21.15.7:54536) 2023-03-03T17:15:32.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389d8c0700 1 -- 172.21.15.7:0/3435599974 learned_addr learned my addr 172.21.15.7:0/3435599974 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:15:32.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389d0bf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3898151e80 0x7f38980c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:32.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389c8be700 1 --2- 172.21.15.7:0/3435599974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3898155410 0x7f38980bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:32.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f388ffff700 1 -- 172.21.15.7:0/3435599974 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762506121 0 0) 0x7f38980c08d0 con 0x7f38980c6fe0 2023-03-03T17:15:32.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f388ffff700 1 -- 172.21.15.7:0/3435599974 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38840035b0 con 0x7f38980c6fe0 2023-03-03T17:15:32.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389d0bf700 1 -- 172.21.15.7:0/3435599974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3898155410 msgr2=0x7f38980bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:15:32.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389d0bf700 1 --2- 172.21.15.7:0/3435599974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3898155410 0x7f38980bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:32.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389d0bf700 1 -- 172.21.15.7:0/3435599974 >> v1:172.21.15.7:6789/0 conn(0x7f38980c6fe0 legacy=0x7f38980bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:32.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389d0bf700 1 -- 172.21.15.7:0/3435599974 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3898154930 con 0x7f3898151e80 2023-03-03T17:15:32.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f389d0bf700 1 --2- 172.21.15.7:0/3435599974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898151e80 0x7f38980c6aa0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f3894009870 tx=0x7f3894009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f7adabff7a03aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:32.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f388ffff700 1 -- 172.21.15.7:0/3435599974 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894011830 con 0x7f3898151e80 2023-03-03T17:15:32.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.041+0000 7f388ffff700 1 -- 172.21.15.7:0/3435599974 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3894011990 con 0x7f3898151e80 2023-03-03T17:15:32.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.042+0000 7f388ffff700 1 -- 172.21.15.7:0/3435599974 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894011d20 con 0x7f3898151e80 2023-03-03T17:15:32.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.042+0000 7f389f323700 1 -- 172.21.15.7:0/3435599974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898151e80 msgr2=0x7f38980c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:32.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.042+0000 7f389f323700 1 --2- 172.21.15.7:0/3435599974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898151e80 0x7f38980c6aa0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f3894009870 tx=0x7f3894009b80 comp rx=0 tx=0).stop 2023-03-03T17:15:32.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.042+0000 7f389f323700 1 -- 172.21.15.7:0/3435599974 shutdown_connections 2023-03-03T17:15:32.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.042+0000 7f389f323700 1 --2- 172.21.15.7:0/3435599974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898151e80 0x7f38980c6aa0 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:32.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.042+0000 7f389f323700 1 --2- 172.21.15.7:0/3435599974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3898155410 0x7f38980bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:32.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.042+0000 7f389f323700 1 -- 172.21.15.7:0/3435599974 >> 172.21.15.7:0/3435599974 conn(0x7f3898071630 msgr2=0x7f389806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:15:32.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.042+0000 7f389f323700 1 -- 172.21.15.7:0/3435599974 shutdown_connections 2023-03-03T17:15:32.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.042+0000 7f389f323700 1 -- 172.21.15.7:0/3435599974 wait complete. 2023-03-03T17:15:32.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.043+0000 7f389f323700 1 Processor -- start 2023-03-03T17:15:32.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.043+0000 7f389f323700 1 -- start start 2023-03-03T17:15:32.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389f323700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38980c6fe0 0x7f38980c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:32.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389f323700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3898151e80 0x7f38981e02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:32.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389f323700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898155410 0x7f38981e0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:32.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389f323700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38980ca040 con 0x7f3898155410 2023-03-03T17:15:32.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389f323700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38981560a0 con 0x7f3898151e80 2023-03-03T17:15:32.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389f323700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38980ca290 con 0x7f38980c6fe0 2023-03-03T17:15:32.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389d8c0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898155410 0x7f38981e0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:32.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389d8c0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898155410 0x7f38981e0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60562/0 (socket says 172.21.15.7:60562) 2023-03-03T17:15:32.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389d8c0700 1 -- 172.21.15.7:0/2095641761 learned_addr learned my addr 172.21.15.7:0/2095641761 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:15:32.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389d0bf700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38980c6fe0 0x7f38980c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:32.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389c8be700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3898151e80 0x7f38981e02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:32.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389d8c0700 1 -- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38980c6fe0 msgr2=0x7f38980c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:32.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.044+0000 7f389d8c0700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38980c6fe0 0x7f38980c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:32.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.045+0000 7f389d8c0700 1 -- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3898151e80 msgr2=0x7f38981e02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:32.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.045+0000 7f389d8c0700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3898151e80 0x7f38981e02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:32.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.045+0000 7f389d8c0700 1 -- 172.21.15.7:0/2095641761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38940094a0 con 0x7f3898155410 2023-03-03T17:15:32.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.045+0000 7f389c8be700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3898151e80 0x7f38981e02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:15:32.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.045+0000 7f389d8c0700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898155410 0x7f38981e0800 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f389000caa0 tx=0x7f389000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:32.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.045+0000 7f388dffb700 1 -- 172.21.15.7:0/2095641761 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389000a9e0 con 0x7f3898155410 2023-03-03T17:15:32.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.045+0000 7f388dffb700 1 -- 172.21.15.7:0/2095641761 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389000ab40 con 0x7f3898155410 2023-03-03T17:15:32.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.045+0000 7f388dffb700 1 -- 172.21.15.7:0/2095641761 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3890012400 con 0x7f3898155410 2023-03-03T17:15:32.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.045+0000 7f389f323700 1 -- 172.21.15.7:0/2095641761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3898154ef0 con 0x7f3898155410 2023-03-03T17:15:32.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.045+0000 7f389f323700 1 -- 172.21.15.7:0/2095641761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3898154c10 con 0x7f3898155410 2023-03-03T17:15:32.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.046+0000 7f389f323700 1 -- 172.21.15.7:0/2095641761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38980c5220 con 0x7f3898155410 2023-03-03T17:15:32.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.046+0000 7f388dffb700 1 -- 172.21.15.7:0/2095641761 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3890019030 con 0x7f3898155410 2023-03-03T17:15:32.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.047+0000 7f388dffb700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f387c0682f0 0x7f387c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:32.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.047+0000 7f388dffb700 1 -- 172.21.15.7:0/2095641761 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3890085d80 con 0x7f3898155410 2023-03-03T17:15:32.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.047+0000 7f389d0bf700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f387c0682f0 0x7f387c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.047+0000 7f389d0bf700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f387c0682f0 0x7f387c06a7a0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f38940090b0 tx=0x7f389401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:32.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.051+0000 7f388dffb700 1 -- 172.21.15.7:0/2095641761 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3890052120 con 0x7f3898155410 2023-03-03T17:15:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.289+0000 7f389f323700 1 -- 172.21.15.7:0/2095641761 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3898074bb0 con 0x7f3898155410 2023-03-03T17:15:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.289+0000 7f388dffb700 1 -- 172.21.15.7:0/2095641761 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3890055740 con 0x7f3898155410 2023-03-03T17:15:32.289 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:15:32.289 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:15:32.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 -- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f387c0682f0 msgr2=0x7f387c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:32.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f387c0682f0 0x7f387c06a7a0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f38940090b0 tx=0x7f389401c040 comp rx=0 tx=0).stop 2023-03-03T17:15:32.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 -- 172.21.15.7:0/2095641761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898155410 msgr2=0x7f38981e0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:32.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898155410 0x7f38981e0800 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f389000caa0 tx=0x7f389000cdb0 comp rx=0 tx=0).stop 2023-03-03T17:15:32.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 -- 172.21.15.7:0/2095641761 shutdown_connections 2023-03-03T17:15:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f387c0682f0 0x7f387c06a7a0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3898155410 0x7f38981e0800 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38980c6fe0 0x7f38980c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 --2- 172.21.15.7:0/2095641761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3898151e80 0x7f38981e02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 -- 172.21.15.7:0/2095641761 >> 172.21.15.7:0/2095641761 conn(0x7f3898071630 msgr2=0x7f38980c3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:15:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 -- 172.21.15.7:0/2095641761 shutdown_connections 2023-03-03T17:15:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:32.291+0000 7f389f323700 1 -- 172.21.15.7:0/2095641761 wait complete. 2023-03-03T17:15:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:15:32.303 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T17:15:43.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:15:43.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.445+0000 7f3449ec1700 1 Processor -- start 2023-03-03T17:15:43.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.445+0000 7f3449ec1700 1 -- start start 2023-03-03T17:15:43.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3449ec1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f344413a0f0 0x7f34440c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:43.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3449ec1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f34441586c0 0x7f344406d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:43.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3449ec1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f34440c1da0 0x7f344406d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:43.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3449ec1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3444154840 con 0x7f34440c1da0 2023-03-03T17:15:43.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3449ec1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3444154560 con 0x7f34441586c0 2023-03-03T17:15:43.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3449ec1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3444157be0 con 0x7f344413a0f0 2023-03-03T17:15:43.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3443fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f34440c1da0 0x7f344406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:43.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3443fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f34440c1da0 0x7f344406d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60578/0 (socket says 172.21.15.7:60578) 2023-03-03T17:15:43.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3443fff700 1 -- 172.21.15.7:0/3062656098 learned_addr learned my addr 172.21.15.7:0/3062656098 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:15:43.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f34437fe700 1 --2- 172.21.15.7:0/3062656098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f344413a0f0 0x7f34440c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:43.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3443fff700 1 -- 172.21.15.7:0/3062656098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f344413a0f0 msgr2=0x7f34440c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:15:43.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3443fff700 1 --2- 172.21.15.7:0/3062656098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f344413a0f0 0x7f34440c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3443fff700 1 -- 172.21.15.7:0/3062656098 >> v2:172.21.15.143:3300/0 conn(0x7f34441586c0 msgr2=0x7f344406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:15:43.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3442ffd700 1 --2- 172.21.15.7:0/3062656098 >> v2:172.21.15.143:3300/0 conn(0x7f34441586c0 0x7f344406d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:43.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3443fff700 1 --2- 172.21.15.7:0/3062656098 >> v2:172.21.15.143:3300/0 conn(0x7f34441586c0 0x7f344406d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3443fff700 1 -- 172.21.15.7:0/3062656098 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444157ec0 con 0x7f34440c1da0 2023-03-03T17:15:43.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3442ffd700 1 --2- 172.21.15.7:0/3062656098 >> v2:172.21.15.143:3300/0 conn(0x7f34441586c0 0x7f344406d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:15:43.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.446+0000 7f3443fff700 1 --2- 172.21.15.7:0/3062656098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34440c1da0 0x7f344406d780 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f34440aa1d0 tx=0x7f343c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91cf63c5c6038a58 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:43.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f34427fc700 1 -- 172.21.15.7:0/3062656098 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c00e070 con 0x7f34440c1da0 2023-03-03T17:15:43.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f34427fc700 1 -- 172.21.15.7:0/3062656098 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343c00b9c0 con 0x7f34440c1da0 2023-03-03T17:15:43.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f34427fc700 1 -- 172.21.15.7:0/3062656098 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c00bd70 con 0x7f34440c1da0 2023-03-03T17:15:43.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f3449ec1700 1 -- 172.21.15.7:0/3062656098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34440c1da0 msgr2=0x7f344406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:43.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f3449ec1700 1 --2- 172.21.15.7:0/3062656098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34440c1da0 0x7f344406d780 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f34440aa1d0 tx=0x7f343c00b330 comp rx=0 tx=0).stop 2023-03-03T17:15:43.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f3449ec1700 1 -- 172.21.15.7:0/3062656098 shutdown_connections 2023-03-03T17:15:43.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f3449ec1700 1 --2- 172.21.15.7:0/3062656098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34440c1da0 0x7f344406d780 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f3449ec1700 1 --2- 172.21.15.7:0/3062656098 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f344413a0f0 0x7f34440c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f3449ec1700 1 --2- 172.21.15.7:0/3062656098 >> v2:172.21.15.143:3300/0 conn(0x7f34441586c0 0x7f344406d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f3449ec1700 1 -- 172.21.15.7:0/3062656098 >> 172.21.15.7:0/3062656098 conn(0x7f344406c110 msgr2=0x7f344406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:15:43.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f3449ec1700 1 -- 172.21.15.7:0/3062656098 shutdown_connections 2023-03-03T17:15:43.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.447+0000 7f3449ec1700 1 -- 172.21.15.7:0/3062656098 wait complete. 2023-03-03T17:15:43.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.448+0000 7f3449ec1700 1 Processor -- start 2023-03-03T17:15:43.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3449ec1700 1 -- start start 2023-03-03T17:15:43.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3449ec1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34440c1da0 0x7f3444071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:43.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3449ec1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f344413a0f0 0x7f3444151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:43.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3449ec1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34441586c0 0x7f3444151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:43.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3449ec1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34440acab0 con 0x7f34441586c0 2023-03-03T17:15:43.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3449ec1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34440aa450 con 0x7f344413a0f0 2023-03-03T17:15:43.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3449ec1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34441ed940 con 0x7f34440c1da0 2023-03-03T17:15:43.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3443fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34441586c0 0x7f3444151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:43.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3443fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34441586c0 0x7f3444151890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60584/0 (socket says 172.21.15.7:60584) 2023-03-03T17:15:43.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3443fff700 1 -- 172.21.15.7:0/2985463376 learned_addr learned my addr 172.21.15.7:0/2985463376 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:15:43.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.449+0000 7f3442ffd700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f344413a0f0 0x7f3444151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:43.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f34437fe700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34440c1da0 0x7f3444071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:43.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f3443fff700 1 -- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34440c1da0 msgr2=0x7f3444071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:43.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f3443fff700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34440c1da0 0x7f3444071960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f3443fff700 1 -- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f344413a0f0 msgr2=0x7f3444151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:43.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f3443fff700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f344413a0f0 0x7f3444151350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f3443fff700 1 -- 172.21.15.7:0/2985463376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c009b30 con 0x7f34441586c0 2023-03-03T17:15:43.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f3443fff700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34441586c0 0x7f3444151890 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7f343c00b370 tx=0x7f343c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:43.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f343bfff700 1 -- 172.21.15.7:0/2985463376 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c004250 con 0x7f34441586c0 2023-03-03T17:15:43.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f343bfff700 1 -- 172.21.15.7:0/2985463376 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343c0043f0 con 0x7f34441586c0 2023-03-03T17:15:43.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f3449ec1700 1 -- 172.21.15.7:0/2985463376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344406e1a0 con 0x7f34441586c0 2023-03-03T17:15:43.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f343bfff700 1 -- 172.21.15.7:0/2985463376 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c0047a0 con 0x7f34441586c0 2023-03-03T17:15:43.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.450+0000 7f3449ec1700 1 -- 172.21.15.7:0/2985463376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34441581a0 con 0x7f34441586c0 2023-03-03T17:15:43.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.451+0000 7f3449ec1700 1 -- 172.21.15.7:0/2985463376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344406e330 con 0x7f34441586c0 2023-03-03T17:15:43.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.452+0000 7f343bfff700 1 -- 172.21.15.7:0/2985463376 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f343c02c070 con 0x7f34441586c0 2023-03-03T17:15:43.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.452+0000 7f343bfff700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34240682f0 0x7f342406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:43.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.452+0000 7f343bfff700 1 -- 172.21.15.7:0/2985463376 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f343c087430 con 0x7f34441586c0 2023-03-03T17:15:43.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.452+0000 7f34437fe700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34240682f0 0x7f342406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:43.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.453+0000 7f34437fe700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34240682f0 0x7f342406a7a0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f344413d840 tx=0x7f3434008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:43.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.456+0000 7f343bfff700 1 -- 172.21.15.7:0/2985463376 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f343c0537d0 con 0x7f34441586c0 2023-03-03T17:15:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.693+0000 7f3449ec1700 1 -- 172.21.15.7:0/2985463376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34440c1540 con 0x7f34441586c0 2023-03-03T17:15:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.694+0000 7f343bfff700 1 -- 172.21.15.7:0/2985463376 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f343c056df0 con 0x7f34441586c0 2023-03-03T17:15:43.694 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:15:43.694 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:15:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 -- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34240682f0 msgr2=0x7f342406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34240682f0 0x7f342406a7a0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f344413d840 tx=0x7f3434008040 comp rx=0 tx=0).stop 2023-03-03T17:15:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 -- 172.21.15.7:0/2985463376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34441586c0 msgr2=0x7f3444151890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:43.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34441586c0 0x7f3444151890 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7f343c00b370 tx=0x7f343c004ad0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 -- 172.21.15.7:0/2985463376 shutdown_connections 2023-03-03T17:15:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f34240682f0 0x7f342406a7a0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34441586c0 0x7f3444151890 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f34440c1da0 0x7f3444071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 --2- 172.21.15.7:0/2985463376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f344413a0f0 0x7f3444151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 -- 172.21.15.7:0/2985463376 >> 172.21.15.7:0/2985463376 conn(0x7f344406c110 msgr2=0x7f3444137150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:15:43.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 -- 172.21.15.7:0/2985463376 shutdown_connections 2023-03-03T17:15:43.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:43.696+0000 7f3449ec1700 1 -- 172.21.15.7:0/2985463376 wait complete. 2023-03-03T17:15:43.698 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:15:43.707 INFO:tasks.mds_thrash.fs.[b]:waiting for 105 secs before thrashing 2023-03-03T17:15:51.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:15:51.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.377+0000 7f990e703700 1 Processor -- start 2023-03-03T17:15:51.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.377+0000 7f990e703700 1 -- start start 2023-03-03T17:15:51.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.377+0000 7f990e703700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080aac00 0x7f99080c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.377+0000 7f990e703700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9908071c30 0x7f99080beb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.377+0000 7f990e703700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99080713c0 con 0x7f99080bf0a0 2023-03-03T17:15:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.377+0000 7f990e703700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9908152c70 con 0x7f9908071c30 2023-03-03T17:15:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f990e703700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99080c4930 con 0x7f99080aac00 2023-03-03T17:15:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f99077fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9908071c30 0x7f99080beb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f990cca0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f99080bf0a0 0x7f9908155950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54572/0 (socket says 172.21.15.7:54572) 2023-03-03T17:15:51.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f990cca0700 1 -- 172.21.15.7:0/439023368 learned_addr learned my addr 172.21.15.7:0/439023368 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:15:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f9907fff700 1 --2- 172.21.15.7:0/439023368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080aac00 0x7f99080c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f99077fe700 1 -- 172.21.15.7:0/439023368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080aac00 msgr2=0x7f99080c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:15:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f99077fe700 1 --2- 172.21.15.7:0/439023368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080aac00 0x7f99080c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f99077fe700 1 -- 172.21.15.7:0/439023368 >> v1:172.21.15.7:6789/0 conn(0x7f99080bf0a0 legacy=0x7f9908155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f99077fe700 1 -- 172.21.15.7:0/439023368 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9908152360 con 0x7f9908071c30 2023-03-03T17:15:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f9906ffd700 1 -- 172.21.15.7:0/439023368 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241291946 0 0) 0x7f99080713c0 con 0x7f99080bf0a0 2023-03-03T17:15:51.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f9907fff700 1 --2- 172.21.15.7:0/439023368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080aac00 0x7f99080c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:15:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f99077fe700 1 --2- 172.21.15.7:0/439023368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9908071c30 0x7f99080beb60 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009840 tx=0x7f98fc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc7b4715912fab4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f9906ffd700 1 -- 172.21.15.7:0/439023368 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc011830 con 0x7f9908071c30 2023-03-03T17:15:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f9906ffd700 1 -- 172.21.15.7:0/439023368 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc011990 con 0x7f9908071c30 2023-03-03T17:15:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.378+0000 7f9906ffd700 1 -- 172.21.15.7:0/439023368 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc011d20 con 0x7f9908071c30 2023-03-03T17:15:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.379+0000 7f990e703700 1 -- 172.21.15.7:0/439023368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9908071c30 msgr2=0x7f99080beb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.379+0000 7f990e703700 1 --2- 172.21.15.7:0/439023368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9908071c30 0x7f99080beb60 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009840 tx=0x7f98fc009b50 comp rx=0 tx=0).stop 2023-03-03T17:15:51.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.379+0000 7f990e703700 1 -- 172.21.15.7:0/439023368 shutdown_connections 2023-03-03T17:15:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.379+0000 7f990e703700 1 --2- 172.21.15.7:0/439023368 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080aac00 0x7f99080c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.379+0000 7f990e703700 1 --2- 172.21.15.7:0/439023368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9908071c30 0x7f99080beb60 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.379+0000 7f990e703700 1 -- 172.21.15.7:0/439023368 >> 172.21.15.7:0/439023368 conn(0x7f99080736f0 msgr2=0x7f9908073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:15:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.379+0000 7f990e703700 1 -- 172.21.15.7:0/439023368 shutdown_connections 2023-03-03T17:15:51.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.379+0000 7f990e703700 1 -- 172.21.15.7:0/439023368 wait complete. 2023-03-03T17:15:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.380+0000 7f990e703700 1 Processor -- start 2023-03-03T17:15:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.380+0000 7f990e703700 1 -- start start 2023-03-03T17:15:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f990e703700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9908071c30 0x7f99081ed130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f990e703700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99080aac00 0x7f99081ed670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f990e703700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080bf0a0 0x7f9908155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:51.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f990e703700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99081453f0 con 0x7f99080aac00 2023-03-03T17:15:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f990e703700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99081edbb0 con 0x7f9908071c30 2023-03-03T17:15:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f990e703700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9908144e30 con 0x7f99080bf0a0 2023-03-03T17:15:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f99077fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99080aac00 0x7f99081ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f99077fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99080aac00 0x7f99081ed670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60598/0 (socket says 172.21.15.7:60598) 2023-03-03T17:15:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f99077fe700 1 -- 172.21.15.7:0/1922334396 learned_addr learned my addr 172.21.15.7:0/1922334396 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:15:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f9907fff700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9908071c30 0x7f99081ed130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:51.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.381+0000 7f990cca0700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080bf0a0 0x7f9908155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f99077fe700 1 -- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080bf0a0 msgr2=0x7f9908155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f99077fe700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080bf0a0 0x7f9908155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f99077fe700 1 -- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9908071c30 msgr2=0x7f99081ed130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f99077fe700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9908071c30 0x7f99081ed130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f99077fe700 1 -- 172.21.15.7:0/1922334396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0094a0 con 0x7f99080aac00 2023-03-03T17:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f99077fe700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99080aac00 0x7f99081ed670 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f98fc014040 tx=0x7f98fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f9904ff9700 1 -- 172.21.15.7:0/1922334396 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0036a0 con 0x7f99080aac00 2023-03-03T17:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f9904ff9700 1 -- 172.21.15.7:0/1922334396 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc01e070 con 0x7f99080aac00 2023-03-03T17:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f990e703700 1 -- 172.21.15.7:0/1922334396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9908152620 con 0x7f99080aac00 2023-03-03T17:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f9904ff9700 1 -- 172.21.15.7:0/1922334396 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc003a20 con 0x7f99080aac00 2023-03-03T17:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.382+0000 7f990e703700 1 -- 172.21.15.7:0/1922334396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99080c4b80 con 0x7f99080aac00 2023-03-03T17:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.383+0000 7f990e703700 1 -- 172.21.15.7:0/1922334396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9908152900 con 0x7f99080aac00 2023-03-03T17:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.383+0000 7f9904ff9700 1 -- 172.21.15.7:0/1922334396 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f98fc00f050 con 0x7f99080aac00 2023-03-03T17:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.384+0000 7f9904ff9700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98f40682f0 0x7f98f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.384+0000 7f9904ff9700 1 -- 172.21.15.7:0/1922334396 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f98fc087190 con 0x7f99080aac00 2023-03-03T17:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.384+0000 7f9907fff700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98f40682f0 0x7f98f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:15:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.385+0000 7f9907fff700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98f40682f0 0x7f98f406a7a0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f98f8006fd0 tx=0x7f98f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:15:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.388+0000 7f9904ff9700 1 -- 172.21.15.7:0/1922334396 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f98fc053530 con 0x7f99080aac00 2023-03-03T17:15:51.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.626+0000 7f990e703700 1 -- 172.21.15.7:0/1922334396 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9908074bb0 con 0x7f99080aac00 2023-03-03T17:15:51.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.626+0000 7f9904ff9700 1 -- 172.21.15.7:0/1922334396 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f98fc020090 con 0x7f99080aac00 2023-03-03T17:15:51.626 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:15:51.626 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:15:51.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 -- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98f40682f0 msgr2=0x7f98f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98f40682f0 0x7f98f406a7a0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f98f8006fd0 tx=0x7f98f8008040 comp rx=0 tx=0).stop 2023-03-03T17:15:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 -- 172.21.15.7:0/1922334396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99080aac00 msgr2=0x7f99081ed670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:15:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99080aac00 0x7f99081ed670 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f98fc014040 tx=0x7f98fc004750 comp rx=0 tx=0).stop 2023-03-03T17:15:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 -- 172.21.15.7:0/1922334396 shutdown_connections 2023-03-03T17:15:51.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99080bf0a0 0x7f9908155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9908071c30 0x7f99081ed130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98f40682f0 0x7f98f406a7a0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 --2- 172.21.15.7:0/1922334396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99080aac00 0x7f99081ed670 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:15:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 -- 172.21.15.7:0/1922334396 >> 172.21.15.7:0/1922334396 conn(0x7f99080736f0 msgr2=0x7f990814bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:15:51.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 -- 172.21.15.7:0/1922334396 shutdown_connections 2023-03-03T17:15:51.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:15:51.628+0000 7f990e703700 1 -- 172.21.15.7:0/1922334396 wait complete. 2023-03-03T17:15:51.630 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:15:51.640 INFO:tasks.mds_thrash.fs.[a]:waiting for 114 secs before thrashing 2023-03-03T17:15:54.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:15:54.524 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:24.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:24.585 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:54.652 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:16:54.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:24.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:24.723 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:28.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:17:28.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.848+0000 7f3440c7d700 1 Processor -- start 2023-03-03T17:17:28.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.848+0000 7f3440c7d700 1 -- start start 2023-03-03T17:17:28.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.849+0000 7f3440c7d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f343c071990 0x7f343c071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:28.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.849+0000 7f3440c7d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c14b460 0x7f343c0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.849+0000 7f3440c7d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f343c151c00 0x7f343c06aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.849+0000 7f3440c7d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c1521a0 con 0x7f343c071990 2023-03-03T17:17:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.849+0000 7f3440c7d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343c070360 con 0x7f343c151c00 2023-03-03T17:17:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.849+0000 7f3440c7d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c06a5e0 con 0x7f343c14b460 2023-03-03T17:17:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.849+0000 7f343a59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f343c071990 0x7f343c071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.849+0000 7f343a59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f343c071990 0x7f343c071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60656/0 (socket says 172.21.15.7:60656) 2023-03-03T17:17:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.849+0000 7f343a59c700 1 -- 172.21.15.7:0/2705673051 learned_addr learned my addr 172.21.15.7:0/2705673051 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:17:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f343ad9d700 1 --2- 172.21.15.7:0/2705673051 >> v2:172.21.15.143:3300/0 conn(0x7f343c151c00 0x7f343c06aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f3439d9b700 1 --2- 172.21.15.7:0/2705673051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c14b460 0x7f343c0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f343a59c700 1 -- 172.21.15.7:0/2705673051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c14b460 msgr2=0x7f343c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:17:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f343a59c700 1 --2- 172.21.15.7:0/2705673051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c14b460 0x7f343c0722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f343a59c700 1 -- 172.21.15.7:0/2705673051 >> v2:172.21.15.143:3300/0 conn(0x7f343c151c00 msgr2=0x7f343c06aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:17:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f343a59c700 1 --2- 172.21.15.7:0/2705673051 >> v2:172.21.15.143:3300/0 conn(0x7f343c151c00 0x7f343c06aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f343a59c700 1 -- 172.21.15.7:0/2705673051 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c0c38b0 con 0x7f343c071990 2023-03-03T17:17:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f343a59c700 1 --2- 172.21.15.7:0/2705673051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c071990 0x7f343c071d60 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7f343c0c1ed0 tx=0x7f342c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333d604438019929 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f343959a700 1 -- 172.21.15.7:0/2705673051 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c00b870 con 0x7f343c071990 2023-03-03T17:17:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f343959a700 1 -- 172.21.15.7:0/2705673051 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c00b9d0 con 0x7f343c071990 2023-03-03T17:17:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.850+0000 7f343959a700 1 -- 172.21.15.7:0/2705673051 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c00bd60 con 0x7f343c071990 2023-03-03T17:17:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.851+0000 7f3440c7d700 1 -- 172.21.15.7:0/2705673051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c071990 msgr2=0x7f343c071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.851+0000 7f3440c7d700 1 --2- 172.21.15.7:0/2705673051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c071990 0x7f343c071d60 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7f343c0c1ed0 tx=0x7f342c009bd0 comp rx=0 tx=0).stop 2023-03-03T17:17:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.851+0000 7f3440c7d700 1 -- 172.21.15.7:0/2705673051 shutdown_connections 2023-03-03T17:17:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.851+0000 7f3440c7d700 1 --2- 172.21.15.7:0/2705673051 >> v2:172.21.15.143:3300/0 conn(0x7f343c151c00 0x7f343c06aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.851+0000 7f3440c7d700 1 --2- 172.21.15.7:0/2705673051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c071990 0x7f343c071d60 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.851+0000 7f3440c7d700 1 --2- 172.21.15.7:0/2705673051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c14b460 0x7f343c0722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.851+0000 7f3440c7d700 1 -- 172.21.15.7:0/2705673051 >> 172.21.15.7:0/2705673051 conn(0x7f343c0a94d0 msgr2=0x7f343c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:17:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.851+0000 7f3440c7d700 1 -- 172.21.15.7:0/2705673051 shutdown_connections 2023-03-03T17:17:28.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.851+0000 7f3440c7d700 1 -- 172.21.15.7:0/2705673051 wait complete. 2023-03-03T17:17:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.852+0000 7f3440c7d700 1 Processor -- start 2023-03-03T17:17:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.852+0000 7f3440c7d700 1 -- start start 2023-03-03T17:17:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3440c7d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c071990 0x7f343c1496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3440c7d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c151c00 0x7f343c13f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3440c7d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c1ea3e0 0x7f343c13fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3440c7d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c152e40 con 0x7f343c151c00 2023-03-03T17:17:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3440c7d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343c1e7430 con 0x7f343c1ea3e0 2023-03-03T17:17:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3440c7d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343c1e7710 con 0x7f343c071990 2023-03-03T17:17:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3439d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c151c00 0x7f343c13f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3439d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c151c00 0x7f343c13f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60664/0 (socket says 172.21.15.7:60664) 2023-03-03T17:17:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3439d9b700 1 -- 172.21.15.7:0/260261568 learned_addr learned my addr 172.21.15.7:0/260261568 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:17:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f343ad9d700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c1ea3e0 0x7f343c13fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f343a59c700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c071990 0x7f343c1496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3439d9b700 1 -- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c071990 msgr2=0x7f343c1496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3439d9b700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c071990 0x7f343c1496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3439d9b700 1 -- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c1ea3e0 msgr2=0x7f343c13fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3439d9b700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c1ea3e0 0x7f343c13fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3439d9b700 1 -- 172.21.15.7:0/260261568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c009550 con 0x7f343c151c00 2023-03-03T17:17:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.853+0000 7f3439d9b700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c151c00 0x7f343c13f5f0 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f342400e810 tx=0x7f342400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.854+0000 7f3432ffd700 1 -- 172.21.15.7:0/260261568 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424013610 con 0x7f343c151c00 2023-03-03T17:17:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.854+0000 7f3432ffd700 1 -- 172.21.15.7:0/260261568 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3424013770 con 0x7f343c151c00 2023-03-03T17:17:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.854+0000 7f3440c7d700 1 -- 172.21.15.7:0/260261568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343c0c3b50 con 0x7f343c151c00 2023-03-03T17:17:28.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.854+0000 7f3432ffd700 1 -- 172.21.15.7:0/260261568 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3424013b20 con 0x7f343c151c00 2023-03-03T17:17:28.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.854+0000 7f3440c7d700 1 -- 172.21.15.7:0/260261568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343c06a2b0 con 0x7f343c151c00 2023-03-03T17:17:28.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.855+0000 7f3440c7d700 1 -- 172.21.15.7:0/260261568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343c06afb0 con 0x7f343c151c00 2023-03-03T17:17:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.855+0000 7f3432ffd700 1 -- 172.21.15.7:0/260261568 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3424023070 con 0x7f343c151c00 2023-03-03T17:17:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.856+0000 7f3432ffd700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 0x7f341c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.856+0000 7f3432ffd700 1 -- 172.21.15.7:0/260261568 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3424085e30 con 0x7f343c151c00 2023-03-03T17:17:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.856+0000 7f343a59c700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 0x7f341c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.857+0000 7f343a59c700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 0x7f341c06a7a0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f342c000c00 tx=0x7f342c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:28.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:28.860+0000 7f3432ffd700 1 -- 172.21.15.7:0/260261568 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3424052150 con 0x7f343c151c00 2023-03-03T17:17:29.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.096+0000 7f3440c7d700 1 -- 172.21.15.7:0/260261568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f343c074bb0 con 0x7f343c151c00 2023-03-03T17:17:29.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.096+0000 7f3432ffd700 1 -- 172.21.15.7:0/260261568 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3424055770 con 0x7f343c151c00 2023-03-03T17:17:29.097 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:17:29.097 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:17:29.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.098+0000 7f3440c7d700 1 -- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 msgr2=0x7f341c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:29.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.098+0000 7f3440c7d700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 0x7f341c06a7a0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f342c000c00 tx=0x7f342c008030 comp rx=0 tx=0).stop 2023-03-03T17:17:29.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.098+0000 7f3440c7d700 1 -- 172.21.15.7:0/260261568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c151c00 msgr2=0x7f343c13f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:29.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.098+0000 7f3440c7d700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c151c00 0x7f343c13f5f0 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f342400e810 tx=0x7f342400ebd0 comp rx=0 tx=0).stop 2023-03-03T17:17:29.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.098+0000 7f3440c7d700 1 -- 172.21.15.7:0/260261568 shutdown_connections 2023-03-03T17:17:29.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.098+0000 7f3440c7d700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f343c1ea3e0 0x7f343c13fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:29.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.098+0000 7f3440c7d700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f343c071990 0x7f343c1496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:29.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.098+0000 7f3440c7d700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f341c0682f0 0x7f341c06a7a0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:29.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.098+0000 7f3440c7d700 1 --2- 172.21.15.7:0/260261568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f343c151c00 0x7f343c13f5f0 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:29.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.098+0000 7f3440c7d700 1 -- 172.21.15.7:0/260261568 >> 172.21.15.7:0/260261568 conn(0x7f343c0a94d0 msgr2=0x7f343c070610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:17:29.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.099+0000 7f3440c7d700 1 -- 172.21.15.7:0/260261568 shutdown_connections 2023-03-03T17:17:29.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:29.099+0000 7f3440c7d700 1 -- 172.21.15.7:0/260261568 wait complete. 2023-03-03T17:17:29.100 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:17:29.110 INFO:tasks.mds_thrash.fs.[b]:waiting for 14 secs before thrashing 2023-03-03T17:17:43.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:17:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.250+0000 7f51aa24d700 1 Processor -- start 2023-03-03T17:17:43.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.250+0000 7f51aa24d700 1 -- start start 2023-03-03T17:17:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.251+0000 7f51aa24d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4155390 0x7f51a40c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.251+0000 7f51aa24d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f51a4151770 0x7f51a4151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.251+0000 7f51aa24d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a4154910 con 0x7f51a406e020 2023-03-03T17:17:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.251+0000 7f51aa24d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a4070b90 con 0x7f51a4151770 2023-03-03T17:17:43.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.251+0000 7f51aa24d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a4151320 con 0x7f51a4155390 2023-03-03T17:17:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.251+0000 7f51a2ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f51a406e020 0x7f51a4071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54660/0 (socket says 172.21.15.7:54660) 2023-03-03T17:17:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.251+0000 7f51a2ffd700 1 -- 172.21.15.7:0/228594498 learned_addr learned my addr 172.21.15.7:0/228594498 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:17:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a37fe700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4155390 0x7f51a40c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a3fff700 1 --2- 172.21.15.7:0/228594498 >> v2:172.21.15.143:3300/0 conn(0x7f51a4151770 0x7f51a4151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035972640 0 0) 0x7f51a4154910 con 0x7f51a406e020 2023-03-03T17:17:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c0035b0 con 0x7f51a406e020 2023-03-03T17:17:43.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1831858815 0 0) 0x7f518c0035b0 con 0x7f51a406e020 2023-03-03T17:17:43.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4155390 msgr2=0x7f51a40c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:17:43.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4155390 0x7f51a40c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:43.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 >> v2:172.21.15.143:3300/0 conn(0x7f51a4151770 msgr2=0x7f51a4151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:17:43.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 --2- 172.21.15.7:0/228594498 >> v2:172.21.15.143:3300/0 conn(0x7f51a4151770 0x7f51a4151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:43.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a40702a0 con 0x7f51a406e020 2023-03-03T17:17:43.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f5198001670 con 0x7f51a406e020 2023-03-03T17:17:43.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 >> v1:172.21.15.7:6789/0 conn(0x7f51a406e020 legacy=0x7f51a4071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:43.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4155390 0x7f518c003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:43.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 0x7f518c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:43.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f518c004d90 0x7f518c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:43.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a4070b90 con 0x7f518c003ff0 2023-03-03T17:17:43.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a4151320 con 0x7f518c004d90 2023-03-03T17:17:43.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518c00bac0 con 0x7f51a4155390 2023-03-03T17:17:43.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a2ffd700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 0x7f518c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:43.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a37fe700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4155390 0x7f518c003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:43.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a3fff700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f518c004d90 0x7f518c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:43.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a2ffd700 1 -- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4155390 msgr2=0x7f518c003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:17:43.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.252+0000 7f51a2ffd700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4155390 0x7f518c003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:43.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51a2ffd700 1 -- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f518c004d90 msgr2=0x7f518c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:17:43.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51a2ffd700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f518c004d90 0x7f518c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:43.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51a2ffd700 1 -- 172.21.15.7:0/228594498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c0035b0 con 0x7f518c003ff0 2023-03-03T17:17:43.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51a2ffd700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 0x7f518c004850 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f5198001050 tx=0x7f518c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42b6c5f8d0767d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:43.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5198007c60 con 0x7f518c003ff0 2023-03-03T17:17:43.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519800ce80 con 0x7f518c003ff0 2023-03-03T17:17:43.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51a27fc700 1 -- 172.21.15.7:0/228594498 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5198005ba0 con 0x7f518c003ff0 2023-03-03T17:17:43.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51aa24d700 1 -- 172.21.15.7:0/228594498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 msgr2=0x7f518c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:43.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51aa24d700 1 --2- 172.21.15.7:0/228594498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 0x7f518c004850 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f5198001050 tx=0x7f518c005880 comp rx=0 tx=0).stop 2023-03-03T17:17:43.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51a3fff700 1 -- 172.21.15.7:0/228594498 reap_dead start 2023-03-03T17:17:43.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51aa24d700 1 -- 172.21.15.7:0/228594498 shutdown_connections 2023-03-03T17:17:43.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.253+0000 7f51aa24d700 1 -- 172.21.15.7:0/228594498 >> 172.21.15.7:0/228594498 conn(0x7f51a406c110 msgr2=0x7f51a406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:17:43.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.254+0000 7f51aa24d700 1 -- 172.21.15.7:0/228594498 shutdown_connections 2023-03-03T17:17:43.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.254+0000 7f51aa24d700 1 -- 172.21.15.7:0/228594498 wait complete. 2023-03-03T17:17:43.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.255+0000 7f51aa24d700 1 Processor -- start 2023-03-03T17:17:43.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.255+0000 7f51aa24d700 1 -- start start 2023-03-03T17:17:43.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51aa24d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 0x7f51a4152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:43.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51aa24d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51a414b300 0x7f51a4148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:43.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51aa24d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4141cb0 0x7f51a4142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:43.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51aa24d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a41e96e0 con 0x7f518c003ff0 2023-03-03T17:17:43.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51aa24d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a4152c50 con 0x7f51a414b300 2023-03-03T17:17:43.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51aa24d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a4142660 con 0x7f51a4141cb0 2023-03-03T17:17:43.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 0x7f51a4152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:43.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 0x7f51a4152710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60690/0 (socket says 172.21.15.7:60690) 2023-03-03T17:17:43.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a37fe700 1 -- 172.21.15.7:0/3869961471 learned_addr learned my addr 172.21.15.7:0/3869961471 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:17:43.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a2ffd700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51a414b300 0x7f51a4148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:43.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a37fe700 1 -- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4141cb0 msgr2=0x7f51a4142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:43.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a3fff700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4141cb0 0x7f51a4142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:43.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a37fe700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4141cb0 0x7f51a4142120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:43.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a37fe700 1 -- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51a414b300 msgr2=0x7f51a4148660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:43.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a37fe700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51a414b300 0x7f51a4148660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:43.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a37fe700 1 -- 172.21.15.7:0/3869961471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a4070820 con 0x7f518c003ff0 2023-03-03T17:17:43.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51a37fe700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 0x7f51a4152710 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f5194011ad0 tx=0x7f5194011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:43.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f518bfff700 1 -- 172.21.15.7:0/3869961471 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519400f9a0 con 0x7f518c003ff0 2023-03-03T17:17:43.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f518bfff700 1 -- 172.21.15.7:0/3869961471 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519400fb00 con 0x7f518c003ff0 2023-03-03T17:17:43.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f518bfff700 1 -- 172.21.15.7:0/3869961471 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519400feb0 con 0x7f518c003ff0 2023-03-03T17:17:43.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51aa24d700 1 -- 172.21.15.7:0/3869961471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a4070540 con 0x7f518c003ff0 2023-03-03T17:17:43.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.256+0000 7f51aa24d700 1 -- 172.21.15.7:0/3869961471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a4155760 con 0x7f518c003ff0 2023-03-03T17:17:43.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.257+0000 7f51aa24d700 1 -- 172.21.15.7:0/3869961471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a4074bb0 con 0x7f518c003ff0 2023-03-03T17:17:43.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.258+0000 7f518bfff700 1 -- 172.21.15.7:0/3869961471 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5194014440 con 0x7f518c003ff0 2023-03-03T17:17:43.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.258+0000 7f518bfff700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5190068320 0x7f519006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:43.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.258+0000 7f518bfff700 1 -- 172.21.15.7:0/3869961471 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f51940868e0 con 0x7f518c003ff0 2023-03-03T17:17:43.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.259+0000 7f51a2ffd700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5190068320 0x7f519006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:43.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.259+0000 7f51a2ffd700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5190068320 0x7f519006a7d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f5198015040 tx=0x7f519800a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:43.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.263+0000 7f518bfff700 1 -- 172.21.15.7:0/3869961471 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5194052ce0 con 0x7f518c003ff0 2023-03-03T17:17:43.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.502+0000 7f51aa24d700 1 -- 172.21.15.7:0/3869961471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51a40c2f20 con 0x7f518c003ff0 2023-03-03T17:17:43.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.503+0000 7f518bfff700 1 -- 172.21.15.7:0/3869961471 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5194056300 con 0x7f518c003ff0 2023-03-03T17:17:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:17:43.504 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:17:43.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 -- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5190068320 msgr2=0x7f519006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:43.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5190068320 0x7f519006a7d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f5198015040 tx=0x7f519800a000 comp rx=0 tx=0).stop 2023-03-03T17:17:43.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 -- 172.21.15.7:0/3869961471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 msgr2=0x7f51a4152710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:43.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 0x7f51a4152710 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f5194011ad0 tx=0x7f5194011e90 comp rx=0 tx=0).stop 2023-03-03T17:17:43.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 -- 172.21.15.7:0/3869961471 shutdown_connections 2023-03-03T17:17:43.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51a4141cb0 0x7f51a4142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:43.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5190068320 0x7f519006a7d0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:43.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f518c003ff0 0x7f51a4152710 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:43.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 --2- 172.21.15.7:0/3869961471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51a414b300 0x7f51a4148660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:43.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 -- 172.21.15.7:0/3869961471 >> 172.21.15.7:0/3869961471 conn(0x7f51a406c110 msgr2=0x7f51a4151bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:17:43.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 -- 172.21.15.7:0/3869961471 shutdown_connections 2023-03-03T17:17:43.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:43.505+0000 7f51aa24d700 1 -- 172.21.15.7:0/3869961471 wait complete. 2023-03-03T17:17:43.507 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:17:43.517 INFO:tasks.mds_thrash.fs.[b]:waiting for 13 secs before thrashing 2023-03-03T17:17:45.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:17:45.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.780+0000 7f70a6ac4700 1 Processor -- start 2023-03-03T17:17:45.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.780+0000 7f70a6ac4700 1 -- start start 2023-03-03T17:17:45.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.781+0000 7f70a6ac4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70a0155390 0x7f70a00c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:45.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.781+0000 7f70a6ac4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f70a006e020 0x7f70a0071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.781+0000 7f70a6ac4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a0154910 con 0x7f70a0072520 2023-03-03T17:17:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.781+0000 7f70a6ac4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a0070b90 con 0x7f70a0155390 2023-03-03T17:17:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.781+0000 7f70a6ac4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a0151320 con 0x7f70a006e020 2023-03-03T17:17:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.781+0000 7f70a4860700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f70a0155390 0x7f70a00c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.781+0000 7f70a5061700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f70a0072520 0x7f70a0151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54682/0 (socket says 172.21.15.7:54682) 2023-03-03T17:17:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.781+0000 7f70a5061700 1 -- 172.21.15.7:0/310781067 learned_addr learned my addr 172.21.15.7:0/310781067 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:17:45.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f709ffff700 1 --2- 172.21.15.7:0/310781067 >> v2:172.21.15.143:3300/0 conn(0x7f70a006e020 0x7f70a0071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:45.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f70a4860700 1 -- 172.21.15.7:0/310781067 >> v2:172.21.15.143:3300/0 conn(0x7f70a006e020 msgr2=0x7f70a0071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:17:45.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f70a4860700 1 --2- 172.21.15.7:0/310781067 >> v2:172.21.15.143:3300/0 conn(0x7f70a006e020 0x7f70a0071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:45.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f70a4860700 1 -- 172.21.15.7:0/310781067 >> v1:172.21.15.7:6789/0 conn(0x7f70a0072520 legacy=0x7f70a0151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:45.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f70a4860700 1 -- 172.21.15.7:0/310781067 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a00702a0 con 0x7f70a0155390 2023-03-03T17:17:45.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f709ffff700 1 --2- 172.21.15.7:0/310781067 >> v2:172.21.15.143:3300/0 conn(0x7f70a006e020 0x7f70a0071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:17:45.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f709f7fe700 1 -- 172.21.15.7:0/310781067 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280285284 0 0) 0x7f70a0154910 con 0x7f70a0072520 2023-03-03T17:17:45.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f70a4860700 1 --2- 172.21.15.7:0/310781067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 0x7f70a00c3e30 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f70a00ac9e0 tx=0x7f7094009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca4642fe6ccf8b0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:45.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f709f7fe700 1 -- 172.21.15.7:0/310781067 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7094011830 con 0x7f70a0155390 2023-03-03T17:17:45.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f709f7fe700 1 -- 172.21.15.7:0/310781067 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7094011990 con 0x7f70a0155390 2023-03-03T17:17:45.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.782+0000 7f709f7fe700 1 -- 172.21.15.7:0/310781067 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7094011d20 con 0x7f70a0155390 2023-03-03T17:17:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.783+0000 7f70a6ac4700 1 -- 172.21.15.7:0/310781067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 msgr2=0x7f70a00c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.783+0000 7f70a6ac4700 1 --2- 172.21.15.7:0/310781067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 0x7f70a00c3e30 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f70a00ac9e0 tx=0x7f7094009b20 comp rx=0 tx=0).stop 2023-03-03T17:17:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.783+0000 7f70a6ac4700 1 -- 172.21.15.7:0/310781067 shutdown_connections 2023-03-03T17:17:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.783+0000 7f70a6ac4700 1 --2- 172.21.15.7:0/310781067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 0x7f70a00c3e30 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.783+0000 7f70a6ac4700 1 --2- 172.21.15.7:0/310781067 >> v2:172.21.15.143:3300/0 conn(0x7f70a006e020 0x7f70a0071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:45.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.783+0000 7f70a6ac4700 1 -- 172.21.15.7:0/310781067 >> 172.21.15.7:0/310781067 conn(0x7f70a006c110 msgr2=0x7f70a006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:17:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.783+0000 7f70a6ac4700 1 -- 172.21.15.7:0/310781067 shutdown_connections 2023-03-03T17:17:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.783+0000 7f70a6ac4700 1 -- 172.21.15.7:0/310781067 wait complete. 2023-03-03T17:17:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.784+0000 7f70a6ac4700 1 Processor -- start 2023-03-03T17:17:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.784+0000 7f70a6ac4700 1 -- start start 2023-03-03T17:17:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a6ac4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a006e020 0x7f70a0150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a6ac4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70a0072520 0x7f70a00aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a6ac4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 0x7f70a0146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a6ac4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a01ed810 con 0x7f70a0155390 2023-03-03T17:17:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a6ac4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a0153740 con 0x7f70a006e020 2023-03-03T17:17:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a6ac4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a0154350 con 0x7f70a0072520 2023-03-03T17:17:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a5061700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 0x7f70a0146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a5061700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 0x7f70a0146840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60710/0 (socket says 172.21.15.7:60710) 2023-03-03T17:17:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a5061700 1 -- 172.21.15.7:0/4189724849 learned_addr learned my addr 172.21.15.7:0/4189724849 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:17:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f709ffff700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70a0072520 0x7f70a00aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:45.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a4860700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a006e020 0x7f70a0150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a5061700 1 -- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70a0072520 msgr2=0x7f70a00aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a5061700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70a0072520 0x7f70a00aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a5061700 1 -- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a006e020 msgr2=0x7f70a0150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a5061700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a006e020 0x7f70a0150540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a5061700 1 -- 172.21.15.7:0/4189724849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70940094a0 con 0x7f70a0155390 2023-03-03T17:17:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.785+0000 7f70a5061700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 0x7f70a0146840 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f7098002230 tx=0x7f709800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:45.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.786+0000 7f709d7fa700 1 -- 172.21.15.7:0/4189724849 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709800ac20 con 0x7f70a0155390 2023-03-03T17:17:45.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.786+0000 7f709d7fa700 1 -- 172.21.15.7:0/4189724849 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709800ada0 con 0x7f70a0155390 2023-03-03T17:17:45.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.786+0000 7f70a6ac4700 1 -- 172.21.15.7:0/4189724849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a0070820 con 0x7f70a0155390 2023-03-03T17:17:45.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.786+0000 7f709d7fa700 1 -- 172.21.15.7:0/4189724849 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098007310 con 0x7f70a0155390 2023-03-03T17:17:45.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.786+0000 7f70a6ac4700 1 -- 172.21.15.7:0/4189724849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a0070540 con 0x7f70a0155390 2023-03-03T17:17:45.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.787+0000 7f70a6ac4700 1 -- 172.21.15.7:0/4189724849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a0155760 con 0x7f70a0155390 2023-03-03T17:17:45.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.787+0000 7f709d7fa700 1 -- 172.21.15.7:0/4189724849 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f709801a030 con 0x7f70a0155390 2023-03-03T17:17:45.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.788+0000 7f709d7fa700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70840682f0 0x7f708406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:45.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.788+0000 7f709d7fa700 1 -- 172.21.15.7:0/4189724849 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7098085f10 con 0x7f70a0155390 2023-03-03T17:17:45.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.788+0000 7f70a4860700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70840682f0 0x7f708406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:45.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.789+0000 7f70a4860700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70840682f0 0x7f708406a7a0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f70940090b0 tx=0x7f709401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:45.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:45.792+0000 7f709d7fa700 1 -- 172.21.15.7:0/4189724849 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70980522b0 con 0x7f70a0155390 2023-03-03T17:17:46.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.027+0000 7f70a6ac4700 1 -- 172.21.15.7:0/4189724849 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70a0154740 con 0x7f70a0155390 2023-03-03T17:17:46.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.028+0000 7f709d7fa700 1 -- 172.21.15.7:0/4189724849 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7098018090 con 0x7f70a0155390 2023-03-03T17:17:46.029 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:17:46.029 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:17:46.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 -- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70840682f0 msgr2=0x7f708406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:46.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70840682f0 0x7f708406a7a0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f70940090b0 tx=0x7f709401c040 comp rx=0 tx=0).stop 2023-03-03T17:17:46.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 -- 172.21.15.7:0/4189724849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 msgr2=0x7f70a0146840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:46.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 0x7f70a0146840 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f7098002230 tx=0x7f709800a060 comp rx=0 tx=0).stop 2023-03-03T17:17:46.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 -- 172.21.15.7:0/4189724849 shutdown_connections 2023-03-03T17:17:46.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70840682f0 0x7f708406a7a0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:46.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70a0155390 0x7f70a0146840 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:46.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70a006e020 0x7f70a0150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:46.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 --2- 172.21.15.7:0/4189724849 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f70a0072520 0x7f70a00aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:46.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 -- 172.21.15.7:0/4189724849 >> 172.21.15.7:0/4189724849 conn(0x7f70a006c110 msgr2=0x7f70a00c2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:17:46.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 -- 172.21.15.7:0/4189724849 shutdown_connections 2023-03-03T17:17:46.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:46.030+0000 7f70a6ac4700 1 -- 172.21.15.7:0/4189724849 wait complete. 2023-03-03T17:17:46.033 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:17:46.043 INFO:tasks.mds_thrash.fs.[a]:waiting for 107 secs before thrashing 2023-03-03T17:17:54.796 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:54.799 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:17:56.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:17:56.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.667+0000 7fc5025b9700 1 Processor -- start 2023-03-03T17:17:56.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.667+0000 7fc5025b9700 1 -- start start 2023-03-03T17:17:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc5025b9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc4fc14fda0 0x7fc4fc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc5025b9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc4fc0abe20 0x7fc4fc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc5025b9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc0c2f30 con 0x7fc4fc14f9d0 2023-03-03T17:17:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc5025b9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4fc0c31d0 con 0x7fc4fc0abe20 2023-03-03T17:17:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc5025b9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4fc1524a0 con 0x7fc4fc14fda0 2023-03-03T17:17:56.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc500b56700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc4fc0abe20 0x7fc4fc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:56.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc4fbfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc4fc14f9d0 0x7fc4fc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54704/0 (socket says 172.21.15.7:54704) 2023-03-03T17:17:56.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc4fbfff700 1 -- 172.21.15.7:0/1624387921 learned_addr learned my addr 172.21.15.7:0/1624387921 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:17:56.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc4fb7fe700 1 --2- 172.21.15.7:0/1624387921 >> v2:172.21.15.143:3300/0 conn(0x7fc4fc14fda0 0x7fc4fc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:56.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc500b56700 1 -- 172.21.15.7:0/1624387921 >> v2:172.21.15.143:3300/0 conn(0x7fc4fc14fda0 msgr2=0x7fc4fc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:17:56.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc500b56700 1 --2- 172.21.15.7:0/1624387921 >> v2:172.21.15.143:3300/0 conn(0x7fc4fc14fda0 0x7fc4fc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc500b56700 1 -- 172.21.15.7:0/1624387921 >> v1:172.21.15.7:6789/0 conn(0x7fc4fc14f9d0 legacy=0x7fc4fc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:56.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc500b56700 1 -- 172.21.15.7:0/1624387921 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc0c34b0 con 0x7fc4fc0abe20 2023-03-03T17:17:56.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc4faffd700 1 -- 172.21.15.7:0/1624387921 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069646843 0 0) 0x7fc4fc0c2f30 con 0x7fc4fc14f9d0 2023-03-03T17:17:56.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc500b56700 1 --2- 172.21.15.7:0/1624387921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc0abe20 0x7fc4fc1529c0 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0709b0 tx=0x7fc4f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ea09a1a2962f67 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:56.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.668+0000 7fc4faffd700 1 -- 172.21.15.7:0/1624387921 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f400e070 con 0x7fc4fc0abe20 2023-03-03T17:17:56.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.669+0000 7fc4faffd700 1 -- 172.21.15.7:0/1624387921 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f400bbd0 con 0x7fc4fc0abe20 2023-03-03T17:17:56.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.669+0000 7fc4faffd700 1 -- 172.21.15.7:0/1624387921 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4012430 con 0x7fc4fc0abe20 2023-03-03T17:17:56.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.669+0000 7fc5025b9700 1 -- 172.21.15.7:0/1624387921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc0abe20 msgr2=0x7fc4fc1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:56.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.669+0000 7fc5025b9700 1 --2- 172.21.15.7:0/1624387921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc0abe20 0x7fc4fc1529c0 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0709b0 tx=0x7fc4f400b330 comp rx=0 tx=0).stop 2023-03-03T17:17:56.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.669+0000 7fc5025b9700 1 -- 172.21.15.7:0/1624387921 shutdown_connections 2023-03-03T17:17:56.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.669+0000 7fc5025b9700 1 --2- 172.21.15.7:0/1624387921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc0abe20 0x7fc4fc1529c0 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.669+0000 7fc5025b9700 1 --2- 172.21.15.7:0/1624387921 >> v2:172.21.15.143:3300/0 conn(0x7fc4fc14fda0 0x7fc4fc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.669+0000 7fc5025b9700 1 -- 172.21.15.7:0/1624387921 >> 172.21.15.7:0/1624387921 conn(0x7fc4fc0c8900 msgr2=0x7fc4fc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:17:56.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.669+0000 7fc5025b9700 1 -- 172.21.15.7:0/1624387921 shutdown_connections 2023-03-03T17:17:56.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.669+0000 7fc5025b9700 1 -- 172.21.15.7:0/1624387921 wait complete. 2023-03-03T17:17:56.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.670+0000 7fc5025b9700 1 Processor -- start 2023-03-03T17:17:56.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc5025b9700 1 -- start start 2023-03-03T17:17:56.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc5025b9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc0abe20 0x7fc4fc0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:56.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc5025b9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc14f9d0 0x7fc4fc06e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:56.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc5025b9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc14fda0 0x7fc4fc06e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:56.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc5025b9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc06f330 con 0x7fc4fc14fda0 2023-03-03T17:17:56.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc5025b9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc1ed6a0 con 0x7fc4fc0abe20 2023-03-03T17:17:56.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc5025b9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4fc1ed980 con 0x7fc4fc14f9d0 2023-03-03T17:17:56.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc500b56700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc14fda0 0x7fc4fc06e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:56.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc500b56700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc14fda0 0x7fc4fc06e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60734/0 (socket says 172.21.15.7:60734) 2023-03-03T17:17:56.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc500b56700 1 -- 172.21.15.7:0/1848708459 learned_addr learned my addr 172.21.15.7:0/1848708459 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:17:56.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc4fb7fe700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc14f9d0 0x7fc4fc06e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:56.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.671+0000 7fc4fbfff700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc0abe20 0x7fc4fc0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:56.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc500b56700 1 -- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc14f9d0 msgr2=0x7fc4fc06e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:56.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc500b56700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc14f9d0 0x7fc4fc06e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc500b56700 1 -- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc0abe20 msgr2=0x7fc4fc0c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:56.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc500b56700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc0abe20 0x7fc4fc0c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc500b56700 1 -- 172.21.15.7:0/1848708459 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f4009a70 con 0x7fc4fc14fda0 2023-03-03T17:17:56.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc500b56700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc14fda0 0x7fc4fc06e7f0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4014040 tx=0x7fc4f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:56.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc4f8ff9700 1 -- 172.21.15.7:0/1848708459 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4012430 con 0x7fc4fc14fda0 2023-03-03T17:17:56.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc4f8ff9700 1 -- 172.21.15.7:0/1848708459 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f4020070 con 0x7fc4fc14fda0 2023-03-03T17:17:56.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc5025b9700 1 -- 172.21.15.7:0/1848708459 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4fc0ac350 con 0x7fc4fc14fda0 2023-03-03T17:17:56.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc4f8ff9700 1 -- 172.21.15.7:0/1848708459 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4012e70 con 0x7fc4fc14fda0 2023-03-03T17:17:56.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.672+0000 7fc5025b9700 1 -- 172.21.15.7:0/1848708459 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4fc0ab630 con 0x7fc4fc14fda0 2023-03-03T17:17:56.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.673+0000 7fc5025b9700 1 -- 172.21.15.7:0/1848708459 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4fc071fa0 con 0x7fc4fc14fda0 2023-03-03T17:17:56.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.674+0000 7fc4f8ff9700 1 -- 172.21.15.7:0/1848708459 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc4f4004040 con 0x7fc4fc14fda0 2023-03-03T17:17:56.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.674+0000 7fc4f8ff9700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4dc0682f0 0x7fc4dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:17:56.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.674+0000 7fc4f8ff9700 1 -- 172.21.15.7:0/1848708459 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc4f4022070 con 0x7fc4fc14fda0 2023-03-03T17:17:56.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.674+0000 7fc4fbfff700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4dc0682f0 0x7fc4dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:17:56.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.675+0000 7fc4fbfff700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4dc0682f0 0x7fc4dc06a7a0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec007fd0 tx=0x7fc4ec007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:17:56.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.678+0000 7fc4f8ff9700 1 -- 172.21.15.7:0/1848708459 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc4f4054a80 con 0x7fc4fc14fda0 2023-03-03T17:17:56.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.916+0000 7fc5025b9700 1 -- 172.21.15.7:0/1848708459 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4fc074bb0 con 0x7fc4fc14fda0 2023-03-03T17:17:56.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.917+0000 7fc4f8ff9700 1 -- 172.21.15.7:0/1848708459 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc4f40580a0 con 0x7fc4fc14fda0 2023-03-03T17:17:56.917 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:17:56.917 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:17:56.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 -- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4dc0682f0 msgr2=0x7fc4dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:56.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4dc0682f0 0x7fc4dc06a7a0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec007fd0 tx=0x7fc4ec007de0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 -- 172.21.15.7:0/1848708459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc14fda0 msgr2=0x7fc4fc06e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:17:56.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc14fda0 0x7fc4fc06e7f0 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4014040 tx=0x7fc4f4004ad0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 -- 172.21.15.7:0/1848708459 shutdown_connections 2023-03-03T17:17:56.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4dc0682f0 0x7fc4dc06a7a0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4fc14fda0 0x7fc4fc06e7f0 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4fc0abe20 0x7fc4fc0c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 --2- 172.21.15.7:0/1848708459 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4fc14f9d0 0x7fc4fc06e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:17:56.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 -- 172.21.15.7:0/1848708459 >> 172.21.15.7:0/1848708459 conn(0x7fc4fc0c8900 msgr2=0x7fc4fc1515c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:17:56.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 -- 172.21.15.7:0/1848708459 shutdown_connections 2023-03-03T17:17:56.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:17:56.919+0000 7fc5025b9700 1 -- 172.21.15.7:0/1848708459 wait complete. 2023-03-03T17:17:56.920 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:17:56.930 INFO:tasks.mds_thrash.fs.[b]:waiting for 12 secs before thrashing 2023-03-03T17:18:08.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:18:09.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.070+0000 7ff384c94700 1 Processor -- start 2023-03-03T17:18:09.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.070+0000 7ff384c94700 1 -- start start 2023-03-03T17:18:09.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.071+0000 7ff384c94700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff3800c2950 0x7ff380157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:09.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.071+0000 7ff384c94700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38006cb70 con 0x7ff38006c480 2023-03-03T17:18:09.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.071+0000 7ff384c94700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38006ce50 con 0x7ff380151b20 2023-03-03T17:18:09.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.071+0000 7ff384c94700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38006c2f0 con 0x7ff3800c2950 2023-03-03T17:18:09.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.071+0000 7ff37ed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff38006c480 0x7ff380158020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54726/0 (socket says 172.21.15.7:54726) 2023-03-03T17:18:09.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.071+0000 7ff37ed9d700 1 -- 172.21.15.7:0/2488754908 learned_addr learned my addr 172.21.15.7:0/2488754908 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:18:09.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.071+0000 7ff37dd9b700 1 --2- 172.21.15.7:0/2488754908 >> v2:172.21.15.143:3300/0 conn(0x7ff3800c2950 0x7ff380157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:09.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.071+0000 7ff37d59a700 1 -- 172.21.15.7:0/2488754908 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222980373 0 0) 0x7ff38006cb70 con 0x7ff38006c480 2023-03-03T17:18:09.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.072+0000 7ff37d59a700 1 -- 172.21.15.7:0/2488754908 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3680035b0 con 0x7ff38006c480 2023-03-03T17:18:09.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.072+0000 7ff37dd9b700 1 -- 172.21.15.7:0/2488754908 >> v1:172.21.15.143:6789/0 conn(0x7ff380151b20 legacy=0x7ff3800736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:09.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.072+0000 7ff37d59a700 1 -- 172.21.15.7:0/2488754908 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922345241 0 0) 0x7ff38006ce50 con 0x7ff380151b20 2023-03-03T17:18:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.072+0000 7ff37dd9b700 1 -- 172.21.15.7:0/2488754908 >> v1:172.21.15.7:6789/0 conn(0x7ff38006c480 legacy=0x7ff380158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.072+0000 7ff37dd9b700 1 -- 172.21.15.7:0/2488754908 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38006d130 con 0x7ff3800c2950 2023-03-03T17:18:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.072+0000 7ff37dd9b700 1 --2- 172.21.15.7:0/2488754908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3800c2950 0x7ff380157ae0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff370009840 tx=0x7ff370009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cd356c53a7ee748 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:18:09.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.072+0000 7ff37d59a700 1 -- 172.21.15.7:0/2488754908 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370011830 con 0x7ff3800c2950 2023-03-03T17:18:09.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.072+0000 7ff37d59a700 1 -- 172.21.15.7:0/2488754908 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff370011990 con 0x7ff3800c2950 2023-03-03T17:18:09.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.072+0000 7ff37d59a700 1 -- 172.21.15.7:0/2488754908 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370011d20 con 0x7ff3800c2950 2023-03-03T17:18:09.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.073+0000 7ff384c94700 1 -- 172.21.15.7:0/2488754908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3800c2950 msgr2=0x7ff380157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:09.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.073+0000 7ff384c94700 1 --2- 172.21.15.7:0/2488754908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3800c2950 0x7ff380157ae0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff370009840 tx=0x7ff370009b50 comp rx=0 tx=0).stop 2023-03-03T17:18:09.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.073+0000 7ff384c94700 1 -- 172.21.15.7:0/2488754908 shutdown_connections 2023-03-03T17:18:09.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.073+0000 7ff384c94700 1 --2- 172.21.15.7:0/2488754908 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3800c2950 0x7ff380157ae0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:09.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.073+0000 7ff384c94700 1 -- 172.21.15.7:0/2488754908 >> 172.21.15.7:0/2488754908 conn(0x7ff3800a94d0 msgr2=0x7ff3800a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:18:09.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.073+0000 7ff384c94700 1 -- 172.21.15.7:0/2488754908 shutdown_connections 2023-03-03T17:18:09.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.073+0000 7ff384c94700 1 -- 172.21.15.7:0/2488754908 wait complete. 2023-03-03T17:18:09.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.074+0000 7ff384c94700 1 Processor -- start 2023-03-03T17:18:09.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.074+0000 7ff384c94700 1 -- start start 2023-03-03T17:18:09.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff384c94700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38006c480 0x7ff380071290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:09.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff384c94700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3800c2950 0x7ff3801ede80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff384c94700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff380151b20 0x7ff380157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff384c94700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800aa6b0 con 0x7ff38006c480 2023-03-03T17:18:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff384c94700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3801ed9f0 con 0x7ff3800c2950 2023-03-03T17:18:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff384c94700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff380153960 con 0x7ff380151b20 2023-03-03T17:18:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38006c480 0x7ff380071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38006c480 0x7ff380071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60748/0 (socket says 172.21.15.7:60748) 2023-03-03T17:18:09.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37e59c700 1 -- 172.21.15.7:0/1315547707 learned_addr learned my addr 172.21.15.7:0/1315547707 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:18:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37dd9b700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3800c2950 0x7ff3801ede80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37ed9d700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff380151b20 0x7ff380157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37e59c700 1 -- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff380151b20 msgr2=0x7ff380157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37e59c700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff380151b20 0x7ff380157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37e59c700 1 -- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3800c2950 msgr2=0x7ff3801ede80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37e59c700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3800c2950 0x7ff3801ede80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37e59c700 1 -- 172.21.15.7:0/1315547707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3700094a0 con 0x7ff38006c480 2023-03-03T17:18:09.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.075+0000 7ff37dd9b700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3800c2950 0x7ff3801ede80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:18:09.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.076+0000 7ff37e59c700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38006c480 0x7ff380071290 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7ff37400c550 tx=0x7ff37400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:18:09.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.076+0000 7ff36effd700 1 -- 172.21.15.7:0/1315547707 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3740054e0 con 0x7ff38006c480 2023-03-03T17:18:09.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.076+0000 7ff36effd700 1 -- 172.21.15.7:0/1315547707 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff374003680 con 0x7ff38006c480 2023-03-03T17:18:09.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.076+0000 7ff384c94700 1 -- 172.21.15.7:0/1315547707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff380158730 con 0x7ff38006c480 2023-03-03T17:18:09.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.076+0000 7ff36effd700 1 -- 172.21.15.7:0/1315547707 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374003a30 con 0x7ff38006c480 2023-03-03T17:18:09.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.076+0000 7ff384c94700 1 -- 172.21.15.7:0/1315547707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38006d410 con 0x7ff38006c480 2023-03-03T17:18:09.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.077+0000 7ff384c94700 1 -- 172.21.15.7:0/1315547707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff380151ef0 con 0x7ff38006c480 2023-03-03T17:18:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.077+0000 7ff36effd700 1 -- 172.21.15.7:0/1315547707 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff374019030 con 0x7ff38006c480 2023-03-03T17:18:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.078+0000 7ff36effd700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3600682f0 0x7ff36006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:18:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.078+0000 7ff36effd700 1 -- 172.21.15.7:0/1315547707 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff3740854d0 con 0x7ff38006c480 2023-03-03T17:18:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.078+0000 7ff37dd9b700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3600682f0 0x7ff36006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:18:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.078+0000 7ff37dd9b700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3600682f0 0x7ff36006a7a0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7ff380071c30 tx=0x7ff37001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:18:09.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.082+0000 7ff36effd700 1 -- 172.21.15.7:0/1315547707 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff374051870 con 0x7ff38006c480 2023-03-03T17:18:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.326+0000 7ff384c94700 1 -- 172.21.15.7:0/1315547707 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff380074bb0 con 0x7ff38006c480 2023-03-03T17:18:09.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.326+0000 7ff36effd700 1 -- 172.21.15.7:0/1315547707 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff374017090 con 0x7ff38006c480 2023-03-03T17:18:09.326 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:18:09.326 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:18:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.328+0000 7ff384c94700 1 -- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3600682f0 msgr2=0x7ff36006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.328+0000 7ff384c94700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3600682f0 0x7ff36006a7a0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7ff380071c30 tx=0x7ff37001c040 comp rx=0 tx=0).stop 2023-03-03T17:18:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.328+0000 7ff384c94700 1 -- 172.21.15.7:0/1315547707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38006c480 msgr2=0x7ff380071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:18:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.328+0000 7ff384c94700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38006c480 0x7ff380071290 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7ff37400c550 tx=0x7ff37400c910 comp rx=0 tx=0).stop 2023-03-03T17:18:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.329+0000 7ff384c94700 1 -- 172.21.15.7:0/1315547707 shutdown_connections 2023-03-03T17:18:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.329+0000 7ff384c94700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff380151b20 0x7ff380157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.329+0000 7ff384c94700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3600682f0 0x7ff36006a7a0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.329+0000 7ff384c94700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff38006c480 0x7ff380071290 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.329+0000 7ff384c94700 1 --2- 172.21.15.7:0/1315547707 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3800c2950 0x7ff3801ede80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:18:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.329+0000 7ff384c94700 1 -- 172.21.15.7:0/1315547707 >> 172.21.15.7:0/1315547707 conn(0x7ff3800a94d0 msgr2=0x7ff380154e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:18:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.329+0000 7ff384c94700 1 -- 172.21.15.7:0/1315547707 shutdown_connections 2023-03-03T17:18:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:18:09.329+0000 7ff384c94700 1 -- 172.21.15.7:0/1315547707 wait complete. 2023-03-03T17:18:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:18:09.340 INFO:tasks.mds_thrash.fs.[b]:waiting for 115 secs before thrashing 2023-03-03T17:18:24.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:24.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:54.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:18:54.951 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:25.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:25.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:33.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:19:33.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.184+0000 7f72de39c700 1 Processor -- start 2023-03-03T17:19:33.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.184+0000 7f72de39c700 1 -- start start 2023-03-03T17:19:33.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.184+0000 7f72de39c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f72d8072250 0x7f72d814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:33.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.184+0000 7f72de39c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d81510d0 con 0x7f72d80c1ce0 2023-03-03T17:19:33.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.184+0000 7f72de39c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d80c3740 con 0x7f72d8071e80 2023-03-03T17:19:33.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.184+0000 7f72de39c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72d8070670 con 0x7f72d8072250 2023-03-03T17:19:33.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72dc939700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f72d80c1ce0 0x7f72d815ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54762/0 (socket says 172.21.15.7:54762) 2023-03-03T17:19:33.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72dc939700 1 -- 172.21.15.7:0/2940703016 learned_addr learned my addr 172.21.15.7:0/2940703016 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:19:33.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d77fe700 1 --2- 172.21.15.7:0/2940703016 >> v2:172.21.15.143:3300/0 conn(0x7f72d8072250 0x7f72d814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:33.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826605978 0 0) 0x7f72d81510d0 con 0x7f72d80c1ce0 2023-03-03T17:19:33.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c00035b0 con 0x7f72d80c1ce0 2023-03-03T17:19:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2419508567 0 0) 0x7f72c00035b0 con 0x7f72d80c1ce0 2023-03-03T17:19:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 >> v2:172.21.15.143:3300/0 conn(0x7f72d8072250 msgr2=0x7f72d814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:19:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 --2- 172.21.15.7:0/2940703016 >> v2:172.21.15.143:3300/0 conn(0x7f72d8072250 0x7f72d814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 >> v1:172.21.15.143:6789/0 conn(0x7f72d8071e80 legacy=0x7f72d8151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d80c39e0 con 0x7f72d80c1ce0 2023-03-03T17:19:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f72d00019f0 con 0x7f72d80c1ce0 2023-03-03T17:19:33.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 >> v1:172.21.15.7:6789/0 conn(0x7f72d80c1ce0 legacy=0x7f72d815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c0003a70 0x7f72c0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c0004850 0x7f72c0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c0005200 0x7f72c000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d8070670 con 0x7f72c0005200 2023-03-03T17:19:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c000c110 con 0x7f72c0003a70 2023-03-03T17:19:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c000c2d0 con 0x7f72c0004850 2023-03-03T17:19:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72dc939700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c0005200 0x7f72c000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:33.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d7fff700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c0003a70 0x7f72c0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.185+0000 7f72d77fe700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c0004850 0x7f72c0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72dc939700 1 -- 172.21.15.7:0/2940703016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c0004850 msgr2=0x7f72c0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:19:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72dc939700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72c0004850 0x7f72c0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72dc939700 1 -- 172.21.15.7:0/2940703016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c0003a70 msgr2=0x7f72c0003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:19:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72dc939700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c0003a70 0x7f72c0003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72dc939700 1 -- 172.21.15.7:0/2940703016 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c00035b0 con 0x7f72c0005200 2023-03-03T17:19:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72d7fff700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c0003a70 0x7f72c0003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:19:33.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72dc939700 1 -- 172.21.15.7:0/2940703016 reap_dead start 2023-03-03T17:19:33.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72dc939700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c0005200 0x7f72c000ba10 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f72d0002230 tx=0x7f72c0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d0cba96e945048 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:19:33.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0012600 con 0x7f72c0005200 2023-03-03T17:19:33.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d0012760 con 0x7f72c0005200 2023-03-03T17:19:33.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72d6ffd700 1 -- 172.21.15.7:0/2940703016 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0012b10 con 0x7f72c0005200 2023-03-03T17:19:33.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.186+0000 7f72de39c700 1 -- 172.21.15.7:0/2940703016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c0005200 msgr2=0x7f72c000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:33.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.187+0000 7f72de39c700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c0005200 0x7f72c000ba10 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f72d0002230 tx=0x7f72c0005880 comp rx=0 tx=0).stop 2023-03-03T17:19:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.187+0000 7f72de39c700 1 -- 172.21.15.7:0/2940703016 shutdown_connections 2023-03-03T17:19:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.187+0000 7f72de39c700 1 --2- 172.21.15.7:0/2940703016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c0005200 0x7f72c000ba10 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.187+0000 7f72de39c700 1 -- 172.21.15.7:0/2940703016 >> 172.21.15.7:0/2940703016 conn(0x7f72d80a94d0 msgr2=0x7f72d80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:19:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.187+0000 7f72de39c700 1 -- 172.21.15.7:0/2940703016 shutdown_connections 2023-03-03T17:19:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.187+0000 7f72de39c700 1 -- 172.21.15.7:0/2940703016 wait complete. 2023-03-03T17:19:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.188+0000 7f72de39c700 1 Processor -- start 2023-03-03T17:19:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.188+0000 7f72de39c700 1 -- start start 2023-03-03T17:19:33.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72de39c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c0005200 0x7f72d81ed1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72de39c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8150e20 0x7f72d80c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72de39c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80ba410 0x7f72d81ec670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72de39c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d81e9de0 con 0x7f72d8150e20 2023-03-03T17:19:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72de39c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d81ed790 con 0x7f72c0005200 2023-03-03T17:19:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72de39c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d81eda70 con 0x7f72d80ba410 2023-03-03T17:19:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8150e20 0x7f72d80c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:33.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8150e20 0x7f72d80c6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60792/0 (socket says 172.21.15.7:60792) 2023-03-03T17:19:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d77fe700 1 -- 172.21.15.7:0/1757542687 learned_addr learned my addr 172.21.15.7:0/1757542687 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:19:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d7fff700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c0005200 0x7f72d81ed1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d77fe700 1 -- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80ba410 msgr2=0x7f72d81ec670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d77fe700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80ba410 0x7f72d81ec670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d77fe700 1 -- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c0005200 msgr2=0x7f72d81ed1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d77fe700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c0005200 0x7f72d81ed1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d77fe700 1 -- 172.21.15.7:0/1757542687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d806c9a0 con 0x7f72d8150e20 2023-03-03T17:19:33.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d77fe700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8150e20 0x7f72d80c6550 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f72cc011940 tx=0x7f72cc011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:19:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d4ff9700 1 -- 172.21.15.7:0/1757542687 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc004240 con 0x7f72d8150e20 2023-03-03T17:19:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d4ff9700 1 -- 172.21.15.7:0/1757542687 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72cc0043a0 con 0x7f72d8150e20 2023-03-03T17:19:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.189+0000 7f72d4ff9700 1 -- 172.21.15.7:0/1757542687 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc00f950 con 0x7f72d8150e20 2023-03-03T17:19:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.190+0000 7f72de39c700 1 -- 172.21.15.7:0/1757542687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d8151300 con 0x7f72d8150e20 2023-03-03T17:19:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.190+0000 7f72de39c700 1 -- 172.21.15.7:0/1757542687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d806cb30 con 0x7f72d8150e20 2023-03-03T17:19:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.190+0000 7f72de39c700 1 -- 172.21.15.7:0/1757542687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d8074bb0 con 0x7f72d8150e20 2023-03-03T17:19:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.191+0000 7f72d4ff9700 1 -- 172.21.15.7:0/1757542687 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f72cc025070 con 0x7f72d8150e20 2023-03-03T17:19:33.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.191+0000 7f72d4ff9700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72c40682f0 0x7f72c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:19:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.192+0000 7f72d4ff9700 1 -- 172.21.15.7:0/1757542687 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f72cc086580 con 0x7f72d8150e20 2023-03-03T17:19:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.192+0000 7f72d7fff700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72c40682f0 0x7f72c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:19:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.192+0000 7f72d7fff700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72c40682f0 0x7f72c406a7a0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f72c8007fd0 tx=0x7f72c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:19:33.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.196+0000 7f72d4ff9700 1 -- 172.21.15.7:0/1757542687 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f72cc052920 con 0x7f72d8150e20 2023-03-03T17:19:33.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.430+0000 7f72de39c700 1 -- 172.21.15.7:0/1757542687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72d80a9fb0 con 0x7f72d8150e20 2023-03-03T17:19:33.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.430+0000 7f72d4ff9700 1 -- 172.21.15.7:0/1757542687 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f72cc055f40 con 0x7f72d8150e20 2023-03-03T17:19:33.430 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:19:33.430 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:19:33.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.432+0000 7f72de39c700 1 -- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72c40682f0 msgr2=0x7f72c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:33.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.432+0000 7f72de39c700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72c40682f0 0x7f72c406a7a0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f72c8007fd0 tx=0x7f72c8009040 comp rx=0 tx=0).stop 2023-03-03T17:19:33.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.432+0000 7f72de39c700 1 -- 172.21.15.7:0/1757542687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8150e20 msgr2=0x7f72d80c6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:19:33.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.432+0000 7f72de39c700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8150e20 0x7f72d80c6550 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f72cc011940 tx=0x7f72cc011d00 comp rx=0 tx=0).stop 2023-03-03T17:19:33.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.432+0000 7f72de39c700 1 -- 172.21.15.7:0/1757542687 shutdown_connections 2023-03-03T17:19:33.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.432+0000 7f72de39c700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72d80ba410 0x7f72d81ec670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:33.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.432+0000 7f72de39c700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72c0005200 0x7f72d81ed1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:33.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.432+0000 7f72de39c700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72c40682f0 0x7f72c406a7a0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:33.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.432+0000 7f72de39c700 1 --2- 172.21.15.7:0/1757542687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8150e20 0x7f72d80c6550 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:19:33.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.432+0000 7f72de39c700 1 -- 172.21.15.7:0/1757542687 >> 172.21.15.7:0/1757542687 conn(0x7f72d80a94d0 msgr2=0x7f72d806da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:19:33.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.433+0000 7f72de39c700 1 -- 172.21.15.7:0/1757542687 shutdown_connections 2023-03-03T17:19:33.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:19:33.433+0000 7f72de39c700 1 -- 172.21.15.7:0/1757542687 wait complete. 2023-03-03T17:19:33.434 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:19:33.444 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T17:19:55.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:55.094 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:19:59.998 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T17:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T17:20:04.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:20:04.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.481+0000 7fd16b3ea700 1 Processor -- start 2023-03-03T17:20:04.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.481+0000 7fd16b3ea700 1 -- start start 2023-03-03T17:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd16b3ea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd16414b340 0x7fd164152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd16b3ea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 0x7fd164154b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd16b3ea700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16406c920 con 0x7fd1640c2790 2023-03-03T17:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd16b3ea700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1640abd90 con 0x7fd16414b340 2023-03-03T17:20:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd16b3ea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640c1300 con 0x7fd1641546f0 2023-03-03T17:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd169186700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd16414b340 0x7fd164152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd168985700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd1640c2790 0x7fd164152b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54782/0 (socket says 172.21.15.7:54782) 2023-03-03T17:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd168985700 1 -- 172.21.15.7:0/2201350780 learned_addr learned my addr 172.21.15.7:0/2201350780 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd169987700 1 --2- 172.21.15.7:0/2201350780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 0x7fd164154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd169186700 1 -- 172.21.15.7:0/2201350780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 msgr2=0x7fd164154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd169186700 1 --2- 172.21.15.7:0/2201350780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 0x7fd164154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd169186700 1 -- 172.21.15.7:0/2201350780 >> v1:172.21.15.7:6789/0 conn(0x7fd1640c2790 legacy=0x7fd164152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd15bfff700 1 -- 172.21.15.7:0/2201350780 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121005203 0 0) 0x7fd16406c920 con 0x7fd1640c2790 2023-03-03T17:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd169186700 1 -- 172.21.15.7:0/2201350780 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164153280 con 0x7fd16414b340 2023-03-03T17:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd169987700 1 --2- 172.21.15.7:0/2201350780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 0x7fd164154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.482+0000 7fd169186700 1 --2- 172.21.15.7:0/2201350780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd16414b340 0x7fd164152630 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7fd160009840 tx=0x7fd160009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51265e989780bb6b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd15bfff700 1 -- 172.21.15.7:0/2201350780 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160011830 con 0x7fd16414b340 2023-03-03T17:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd15bfff700 1 -- 172.21.15.7:0/2201350780 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd160011990 con 0x7fd16414b340 2023-03-03T17:20:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd15bfff700 1 -- 172.21.15.7:0/2201350780 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160011d20 con 0x7fd16414b340 2023-03-03T17:20:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2201350780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd16414b340 msgr2=0x7fd164152630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd16b3ea700 1 --2- 172.21.15.7:0/2201350780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd16414b340 0x7fd164152630 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7fd160009840 tx=0x7fd160009b50 comp rx=0 tx=0).stop 2023-03-03T17:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2201350780 shutdown_connections 2023-03-03T17:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd16b3ea700 1 --2- 172.21.15.7:0/2201350780 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 0x7fd164154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd16b3ea700 1 --2- 172.21.15.7:0/2201350780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd16414b340 0x7fd164152630 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2201350780 >> 172.21.15.7:0/2201350780 conn(0x7fd1640c62e0 msgr2=0x7fd1640c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2201350780 shutdown_connections 2023-03-03T17:20:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.483+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2201350780 wait complete. 2023-03-03T17:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.484+0000 7fd16b3ea700 1 Processor -- start 2023-03-03T17:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.484+0000 7fd16b3ea700 1 -- start start 2023-03-03T17:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd16b3ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1640c2790 0x7fd164070950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd16b3ea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16414b340 0x7fd1641e5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd16b3ea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 0x7fd164141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd16b3ea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641e63b0 con 0x7fd1640c2790 2023-03-03T17:20:04.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd16b3ea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640aa570 con 0x7fd16414b340 2023-03-03T17:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd16b3ea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1640a93d0 con 0x7fd1641546f0 2023-03-03T17:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd169186700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1640c2790 0x7fd164070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd169186700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1640c2790 0x7fd164070950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60808/0 (socket says 172.21.15.7:60808) 2023-03-03T17:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd169186700 1 -- 172.21.15.7:0/2259384460 learned_addr learned my addr 172.21.15.7:0/2259384460 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd168985700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16414b340 0x7fd1641e5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.485+0000 7fd169987700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 0x7fd164141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd169186700 1 -- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 msgr2=0x7fd164141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:04.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd169186700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 0x7fd164141630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd169186700 1 -- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16414b340 msgr2=0x7fd1641e5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd169186700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16414b340 0x7fd1641e5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd169186700 1 -- 172.21.15.7:0/2259384460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1600094a0 con 0x7fd1640c2790 2023-03-03T17:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd169987700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 0x7fd164141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd169186700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1640c2790 0x7fd164070950 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7fd1600090b0 tx=0x7fd160004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd159ffb700 1 -- 172.21.15.7:0/2259384460 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1600036a0 con 0x7fd1640c2790 2023-03-03T17:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd159ffb700 1 -- 172.21.15.7:0/2259384460 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16001e070 con 0x7fd1640c2790 2023-03-03T17:20:04.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2259384460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16406bff0 con 0x7fd1640c2790 2023-03-03T17:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd159ffb700 1 -- 172.21.15.7:0/2259384460 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160003a00 con 0x7fd1640c2790 2023-03-03T17:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.486+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2259384460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16406bd50 con 0x7fd1640c2790 2023-03-03T17:20:04.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.487+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2259384460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16406c2d0 con 0x7fd1640c2790 2023-03-03T17:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.488+0000 7fd159ffb700 1 -- 172.21.15.7:0/2259384460 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd16000f050 con 0x7fd1640c2790 2023-03-03T17:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.489+0000 7fd159ffb700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1480682f0 0x7fd14806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.489+0000 7fd159ffb700 1 -- 172.21.15.7:0/2259384460 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd160087660 con 0x7fd1640c2790 2023-03-03T17:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.489+0000 7fd168985700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1480682f0 0x7fd14806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.490+0000 7fd168985700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1480682f0 0x7fd14806a7a0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd164142590 tx=0x7fd154007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:20:04.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.493+0000 7fd159ffb700 1 -- 172.21.15.7:0/2259384460 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd160053aa0 con 0x7fd1640c2790 2023-03-03T17:20:04.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.736+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2259384460 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd164074bb0 con 0x7fd1640c2790 2023-03-03T17:20:04.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.737+0000 7fd159ffb700 1 -- 172.21.15.7:0/2259384460 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd1600570c0 con 0x7fd1640c2790 2023-03-03T17:20:04.736 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:20:04.737 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1480682f0 msgr2=0x7fd14806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:04.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1480682f0 0x7fd14806a7a0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd164142590 tx=0x7fd154007e70 comp rx=0 tx=0).stop 2023-03-03T17:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2259384460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1640c2790 msgr2=0x7fd164070950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1640c2790 0x7fd164070950 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7fd1600090b0 tx=0x7fd160004750 comp rx=0 tx=0).stop 2023-03-03T17:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2259384460 shutdown_connections 2023-03-03T17:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1641546f0 0x7fd164141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1480682f0 0x7fd14806a7a0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1640c2790 0x7fd164070950 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:04.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 --2- 172.21.15.7:0/2259384460 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd16414b340 0x7fd1641e5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2259384460 >> 172.21.15.7:0/2259384460 conn(0x7fd1640c62e0 msgr2=0x7fd164142ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2259384460 shutdown_connections 2023-03-03T17:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:04.739+0000 7fd16b3ea700 1 -- 172.21.15.7:0/2259384460 wait complete. 2023-03-03T17:20:04.740 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:20:04.750 INFO:tasks.mds_thrash.fs.[b]:waiting for 93 secs before thrashing 2023-03-03T17:20:22.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:20:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.582+0000 7f3be60a0700 1 Processor -- start 2023-03-03T17:20:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.582+0000 7f3be60a0700 1 -- start start 2023-03-03T17:20:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.583+0000 7f3be60a0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3be0151b20 0x7f3be00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.583+0000 7f3be60a0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3be00c2950 0x7f3be0157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.583+0000 7f3be60a0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be006cb70 con 0x7f3be0073c30 2023-03-03T17:20:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.583+0000 7f3be60a0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be006c2f0 con 0x7f3be0151b20 2023-03-03T17:20:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.583+0000 7f3be60a0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be006ce50 con 0x7f3be00c2950 2023-03-03T17:20:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.583+0000 7f3bdf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3be0151b20 0x7f3be00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.583+0000 7f3bdf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3be0151b20 0x7f3be00736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60824/0 (socket says 172.21.15.7:60824) 2023-03-03T17:20:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.583+0000 7f3bdf7fe700 1 -- 172.21.15.7:0/3045065678 learned_addr learned my addr 172.21.15.7:0/3045065678 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:20:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.583+0000 7f3bdeffd700 1 --2- 172.21.15.7:0/3045065678 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3be00c2950 0x7f3be0157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.583+0000 7f3bdf7fe700 1 -- 172.21.15.7:0/3045065678 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3be00c2950 msgr2=0x7f3be0157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:20:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3bdf7fe700 1 --2- 172.21.15.7:0/3045065678 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3be00c2950 0x7f3be0157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3bdf7fe700 1 -- 172.21.15.7:0/3045065678 >> v1:172.21.15.143:6789/0 conn(0x7f3be0073c30 legacy=0x7f3be0158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3bdf7fe700 1 -- 172.21.15.7:0/3045065678 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be006d130 con 0x7f3be0151b20 2023-03-03T17:20:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3bdf7fe700 1 --2- 172.21.15.7:0/3045065678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be0151b20 0x7f3be00736f0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f3bd00098f0 tx=0x7f3bd0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f9265722a4f9f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:20:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3bde7fc700 1 -- 172.21.15.7:0/3045065678 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd000b8a0 con 0x7f3be0151b20 2023-03-03T17:20:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3bde7fc700 1 -- 172.21.15.7:0/3045065678 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd000ba00 con 0x7f3be0151b20 2023-03-03T17:20:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3bde7fc700 1 -- 172.21.15.7:0/3045065678 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd000bd90 con 0x7f3be0151b20 2023-03-03T17:20:22.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3be60a0700 1 -- 172.21.15.7:0/3045065678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be0151b20 msgr2=0x7f3be00736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:22.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3be60a0700 1 --2- 172.21.15.7:0/3045065678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be0151b20 0x7f3be00736f0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f3bd00098f0 tx=0x7f3bd0009c00 comp rx=0 tx=0).stop 2023-03-03T17:20:22.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3be60a0700 1 -- 172.21.15.7:0/3045065678 shutdown_connections 2023-03-03T17:20:22.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3be60a0700 1 --2- 172.21.15.7:0/3045065678 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be0151b20 0x7f3be00736f0 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:22.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3be60a0700 1 --2- 172.21.15.7:0/3045065678 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3be00c2950 0x7f3be0157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:22.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.584+0000 7f3be60a0700 1 -- 172.21.15.7:0/3045065678 >> 172.21.15.7:0/3045065678 conn(0x7f3be00a94d0 msgr2=0x7f3be00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:20:22.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.585+0000 7f3be60a0700 1 -- 172.21.15.7:0/3045065678 shutdown_connections 2023-03-03T17:20:22.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.585+0000 7f3be60a0700 1 -- 172.21.15.7:0/3045065678 wait complete. 2023-03-03T17:20:22.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.586+0000 7f3be60a0700 1 Processor -- start 2023-03-03T17:20:22.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.586+0000 7f3be60a0700 1 -- start start 2023-03-03T17:20:22.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3be60a0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3be0073c30 0x7f3be0148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:22.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3be60a0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be00c2950 0x7f3be0156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:22.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3be60a0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3be0151b20 0x7f3be01eda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:22.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3be60a0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be0148e00 con 0x7f3be00c2950 2023-03-03T17:20:22.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3be60a0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be0157880 con 0x7f3be0151b20 2023-03-03T17:20:22.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3be60a0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be00c6f70 con 0x7f3be0073c30 2023-03-03T17:20:22.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be00c2950 0x7f3be0156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:22.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdeffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be00c2950 0x7f3be0156930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60832/0 (socket says 172.21.15.7:60832) 2023-03-03T17:20:22.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdeffd700 1 -- 172.21.15.7:0/3142651919 learned_addr learned my addr 172.21.15.7:0/3142651919 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:20:22.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdf7fe700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3be0073c30 0x7f3be0148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:22.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdffff700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3be0151b20 0x7f3be01eda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:22.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdeffd700 1 -- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3be0073c30 msgr2=0x7f3be0148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdeffd700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3be0073c30 0x7f3be0148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdeffd700 1 -- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3be0151b20 msgr2=0x7f3be01eda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdeffd700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3be0151b20 0x7f3be01eda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdeffd700 1 -- 172.21.15.7:0/3142651919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd0009550 con 0x7f3be00c2950 2023-03-03T17:20:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdffff700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3be0151b20 0x7f3be01eda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:20:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bdeffd700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be00c2950 0x7f3be0156930 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400d830 tx=0x7f3bd400dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:20:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bc7fff700 1 -- 172.21.15.7:0/3142651919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd40125f0 con 0x7f3be00c2950 2023-03-03T17:20:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bc7fff700 1 -- 172.21.15.7:0/3142651919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd4012750 con 0x7f3be00c2950 2023-03-03T17:20:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3be60a0700 1 -- 172.21.15.7:0/3142651919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be0158730 con 0x7f3be00c2950 2023-03-03T17:20:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3bc7fff700 1 -- 172.21.15.7:0/3142651919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4012b00 con 0x7f3be00c2950 2023-03-03T17:20:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.587+0000 7f3be60a0700 1 -- 172.21.15.7:0/3142651919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be006d410 con 0x7f3be00c2950 2023-03-03T17:20:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.588+0000 7f3be60a0700 1 -- 172.21.15.7:0/3142651919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be0151ef0 con 0x7f3be00c2950 2023-03-03T17:20:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.589+0000 7f3bc7fff700 1 -- 172.21.15.7:0/3142651919 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3bd4012c60 con 0x7f3be00c2950 2023-03-03T17:20:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.589+0000 7f3bc7fff700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc068320 0x7f3bcc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:20:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.590+0000 7f3bc7fff700 1 -- 172.21.15.7:0/3142651919 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3bd4086610 con 0x7f3be00c2950 2023-03-03T17:20:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.590+0000 7f3bdf7fe700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc068320 0x7f3bcc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:20:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.590+0000 7f3bdf7fe700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc068320 0x7f3bcc06a7d0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0014040 tx=0x7f3bd0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:20:22.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.594+0000 7f3bc7fff700 1 -- 172.21.15.7:0/3142651919 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3bd40529b0 con 0x7f3be00c2950 2023-03-03T17:20:22.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.831+0000 7f3be60a0700 1 -- 172.21.15.7:0/3142651919 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3be0074bb0 con 0x7f3be00c2950 2023-03-03T17:20:22.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.831+0000 7f3bc7fff700 1 -- 172.21.15.7:0/3142651919 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3bd4055fd0 con 0x7f3be00c2950 2023-03-03T17:20:22.831 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:20:22.831 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:20:22.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.833+0000 7f3be60a0700 1 -- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc068320 msgr2=0x7f3bcc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:22.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.833+0000 7f3be60a0700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc068320 0x7f3bcc06a7d0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0014040 tx=0x7f3bd0008030 comp rx=0 tx=0).stop 2023-03-03T17:20:22.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.833+0000 7f3be60a0700 1 -- 172.21.15.7:0/3142651919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be00c2950 msgr2=0x7f3be0156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:20:22.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.833+0000 7f3be60a0700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be00c2950 0x7f3be0156930 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400d830 tx=0x7f3bd400dbf0 comp rx=0 tx=0).stop 2023-03-03T17:20:22.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.833+0000 7f3be60a0700 1 -- 172.21.15.7:0/3142651919 shutdown_connections 2023-03-03T17:20:22.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.833+0000 7f3be60a0700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3be0151b20 0x7f3be01eda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:22.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.833+0000 7f3be60a0700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3be0073c30 0x7f3be0148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:22.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.833+0000 7f3be60a0700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bcc068320 0x7f3bcc06a7d0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:22.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.833+0000 7f3be60a0700 1 --2- 172.21.15.7:0/3142651919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be00c2950 0x7f3be0156930 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:20:22.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.833+0000 7f3be60a0700 1 -- 172.21.15.7:0/3142651919 >> 172.21.15.7:0/3142651919 conn(0x7f3be00a94d0 msgr2=0x7f3be00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:20:22.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.834+0000 7f3be60a0700 1 -- 172.21.15.7:0/3142651919 shutdown_connections 2023-03-03T17:20:22.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:20:22.834+0000 7f3be60a0700 1 -- 172.21.15.7:0/3142651919 wait complete. 2023-03-03T17:20:22.835 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:20:22.845 INFO:tasks.mds_thrash.fs.[a]:waiting for 39 secs before thrashing 2023-03-03T17:20:25.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:25.235 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:55.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:20:55.365 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:01.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:21:01.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.985+0000 7fbf0ea5b700 1 Processor -- start 2023-03-03T17:21:01.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.985+0000 7fbf0ea5b700 1 -- start start 2023-03-03T17:21:01.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.986+0000 7fbf0ea5b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbf08155410 0x7fbf080bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:01.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.986+0000 7fbf0ea5b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf080c08d0 con 0x7fbf080c5b00 2023-03-03T17:21:01.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.986+0000 7fbf0ea5b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf08154930 con 0x7fbf08151e80 2023-03-03T17:21:01.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.986+0000 7fbf0ea5b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf08152250 con 0x7fbf08155410 2023-03-03T17:21:01.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.986+0000 7fbf0cff8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbf080c5b00 0x7fbf080bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54836/0 (socket says 172.21.15.7:54836) 2023-03-03T17:21:01.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.986+0000 7fbf0cff8700 1 -- 172.21.15.7:0/1667837744 learned_addr learned my addr 172.21.15.7:0/1667837744 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:21:01.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.986+0000 7fbf077fe700 1 --2- 172.21.15.7:0/1667837744 >> v2:172.21.15.143:3300/0 conn(0x7fbf08155410 0x7fbf080bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:01.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.986+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726298204 0 0) 0x7fbf080c08d0 con 0x7fbf080c5b00 2023-03-03T17:21:01.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.986+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef00035b0 con 0x7fbf080c5b00 2023-03-03T17:21:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727335077 0 0) 0x7fbef00035b0 con 0x7fbf080c5b00 2023-03-03T17:21:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 >> v2:172.21.15.143:3300/0 conn(0x7fbf08155410 msgr2=0x7fbf080bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:21:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 --2- 172.21.15.7:0/1667837744 >> v2:172.21.15.143:3300/0 conn(0x7fbf08155410 0x7fbf080bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 >> v1:172.21.15.143:6789/0 conn(0x7fbf08151e80 legacy=0x7fbf080c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf08154c10 con 0x7fbf080c5b00 2023-03-03T17:21:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fbf000019f0 con 0x7fbf080c5b00 2023-03-03T17:21:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 >> v1:172.21.15.7:6789/0 conn(0x7fbf080c5b00 legacy=0x7fbf080bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:01.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbef0003a70 0x7fbef0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:01.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbef0004850 0x7fbef0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:01.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbef0005200 0x7fbef000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:01.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef000c0e0 con 0x7fbef0003a70 2023-03-03T17:21:01.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef000c220 con 0x7fbef0004850 2023-03-03T17:21:01.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef000c3e0 con 0x7fbef0005200 2023-03-03T17:21:01.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf07fff700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbef0003a70 0x7fbef0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:01.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf077fe700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbef0004850 0x7fbef0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf0cff8700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbef0005200 0x7fbef000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf07fff700 1 -- 172.21.15.7:0/1667837744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbef0005200 msgr2=0x7fbef000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:21:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf07fff700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbef0005200 0x7fbef000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf07fff700 1 -- 172.21.15.7:0/1667837744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbef0004850 msgr2=0x7fbef0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:21:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf07fff700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbef0004850 0x7fbef0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:01.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf07fff700 1 -- 172.21.15.7:0/1667837744 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef00035b0 con 0x7fbef0003a70 2023-03-03T17:21:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf0cff8700 1 -- 172.21.15.7:0/1667837744 reap_dead start 2023-03-03T17:21:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf077fe700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbef0004850 0x7fbef0004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:21:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf07fff700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbef0003a70 0x7fbef0003e80 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7fbef80017c0 tx=0x7fbef0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70eb4b4d0b8de754 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef80052d0 con 0x7fbef0003a70 2023-03-03T17:21:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef8005450 con 0x7fbef0003a70 2023-03-03T17:21:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.987+0000 7fbf06ffd700 1 -- 172.21.15.7:0/1667837744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef80038c0 con 0x7fbef0003a70 2023-03-03T17:21:01.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.988+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/1667837744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbef0003a70 msgr2=0x7fbef0003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:01.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.988+0000 7fbf0ea5b700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbef0003a70 0x7fbef0003e80 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7fbef80017c0 tx=0x7fbef0005880 comp rx=0 tx=0).stop 2023-03-03T17:21:01.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.988+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/1667837744 shutdown_connections 2023-03-03T17:21:01.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.988+0000 7fbf0ea5b700 1 --2- 172.21.15.7:0/1667837744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbef0003a70 0x7fbef0003e80 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:01.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.988+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/1667837744 >> 172.21.15.7:0/1667837744 conn(0x7fbf08071630 msgr2=0x7fbf0806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:21:01.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.988+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/1667837744 shutdown_connections 2023-03-03T17:21:01.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.988+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/1667837744 wait complete. 2023-03-03T17:21:01.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.990+0000 7fbf0ea5b700 1 Processor -- start 2023-03-03T17:21:01.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.990+0000 7fbf0ea5b700 1 -- start start 2023-03-03T17:21:01.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0ea5b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbef0003a70 0x7fbf08146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:01.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0ea5b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf08155d50 0x7fbf080bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:01.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0ea5b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf0814dbb0 0x7fbf0814e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:01.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0ea5b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf08146750 con 0x7fbf0814dbb0 2023-03-03T17:21:01.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0ea5b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf08157110 con 0x7fbef0003a70 2023-03-03T17:21:01.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0ea5b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf081eb340 con 0x7fbf08155d50 2023-03-03T17:21:02.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0cff8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf0814dbb0 0x7fbf0814e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:02.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0cff8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf0814dbb0 0x7fbf0814e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60866/0 (socket says 172.21.15.7:60866) 2023-03-03T17:21:02.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0cff8700 1 -- 172.21.15.7:0/711066519 learned_addr learned my addr 172.21.15.7:0/711066519 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:21:02.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf07fff700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbef0003a70 0x7fbf08146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:02.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf077fe700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf08155d50 0x7fbf080bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:02.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0cff8700 1 -- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf08155d50 msgr2=0x7fbf080bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:02.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0cff8700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf08155d50 0x7fbf080bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:02.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0cff8700 1 -- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbef0003a70 msgr2=0x7fbf08146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:02.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0cff8700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbef0003a70 0x7fbf08146210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:02.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.991+0000 7fbf0cff8700 1 -- 172.21.15.7:0/711066519 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf080c5220 con 0x7fbf0814dbb0 2023-03-03T17:21:02.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.992+0000 7fbf0cff8700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf0814dbb0 0x7fbf0814e050 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7fbf00001ac0 tx=0x7fbf00007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:02.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.992+0000 7fbf04ff9700 1 -- 172.21.15.7:0/711066519 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00017070 con 0x7fbf0814dbb0 2023-03-03T17:21:02.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.992+0000 7fbf04ff9700 1 -- 172.21.15.7:0/711066519 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0000ecc0 con 0x7fbf0814dbb0 2023-03-03T17:21:02.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.992+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/711066519 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf08154ef0 con 0x7fbf0814dbb0 2023-03-03T17:21:02.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.992+0000 7fbf04ff9700 1 -- 172.21.15.7:0/711066519 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0000ee70 con 0x7fbf0814dbb0 2023-03-03T17:21:02.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.992+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/711066519 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf080c6aa0 con 0x7fbf0814dbb0 2023-03-03T17:21:02.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.993+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/711066519 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0814fa40 con 0x7fbf0814dbb0 2023-03-03T17:21:02.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.993+0000 7fbf04ff9700 1 -- 172.21.15.7:0/711066519 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbf00013420 con 0x7fbf0814dbb0 2023-03-03T17:21:02.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.994+0000 7fbf04ff9700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbef40682f0 0x7fbef406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:02.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.994+0000 7fbf04ff9700 1 -- 172.21.15.7:0/711066519 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbf00085f60 con 0x7fbf0814dbb0 2023-03-03T17:21:02.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.994+0000 7fbf07fff700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbef40682f0 0x7fbef406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:02.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.994+0000 7fbf07fff700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbef40682f0 0x7fbef406a7a0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fbef8000c00 tx=0x7fbef8009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:02.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:01.998+0000 7fbf04ff9700 1 -- 172.21.15.7:0/711066519 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbf000522a0 con 0x7fbf0814dbb0 2023-03-03T17:21:02.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.236+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/711066519 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf08074bb0 con 0x7fbf0814dbb0 2023-03-03T17:21:02.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.237+0000 7fbf04ff9700 1 -- 172.21.15.7:0/711066519 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbf000558c0 con 0x7fbf0814dbb0 2023-03-03T17:21:02.237 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:21:02.238 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:21:02.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbef40682f0 msgr2=0x7fbef406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:02.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbef40682f0 0x7fbef406a7a0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fbef8000c00 tx=0x7fbef8009030 comp rx=0 tx=0).stop 2023-03-03T17:21:02.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/711066519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf0814dbb0 msgr2=0x7fbf0814e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:02.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf0814dbb0 0x7fbf0814e050 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7fbf00001ac0 tx=0x7fbf00007c90 comp rx=0 tx=0).stop 2023-03-03T17:21:02.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/711066519 shutdown_connections 2023-03-03T17:21:02.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbef40682f0 0x7fbef406a7a0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:02.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf0814dbb0 0x7fbf0814e050 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:02.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbef0003a70 0x7fbf08146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:02.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 --2- 172.21.15.7:0/711066519 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbf08155d50 0x7fbf080bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:02.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/711066519 >> 172.21.15.7:0/711066519 conn(0x7fbf08071630 msgr2=0x7fbf081501b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:21:02.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/711066519 shutdown_connections 2023-03-03T17:21:02.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:02.239+0000 7fbf0ea5b700 1 -- 172.21.15.7:0/711066519 wait complete. 2023-03-03T17:21:02.241 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:21:02.251 INFO:tasks.mds_thrash.fs.[a]:waiting for 34 secs before thrashing 2023-03-03T17:21:25.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:25.536 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:36.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:21:36.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.391+0000 7f29100f3700 1 Processor -- start 2023-03-03T17:21:36.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.391+0000 7f29100f3700 1 -- start start 2023-03-03T17:21:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f29100f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29080c2930 0x7f2908157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f29100f3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f290806c480 0x7f2908157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f29100f3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f290806d430 con 0x7f2908151d40 2023-03-03T17:21:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f29100f3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290806c2a0 con 0x7f290806c480 2023-03-03T17:21:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f29100f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290806cb00 con 0x7f29080c2930 2023-03-03T17:21:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f290e690700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f290806c480 0x7f2908157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f290de8f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2908151d40 0x7f29080736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54876/0 (socket says 172.21.15.7:54876) 2023-03-03T17:21:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f290de8f700 1 -- 172.21.15.7:0/2956610152 learned_addr learned my addr 172.21.15.7:0/2956610152 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:21:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f290d68e700 1 --2- 172.21.15.7:0/2956610152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29080c2930 0x7f2908157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f290e690700 1 -- 172.21.15.7:0/2956610152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29080c2930 msgr2=0x7f2908157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:21:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.392+0000 7f290e690700 1 --2- 172.21.15.7:0/2956610152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29080c2930 0x7f2908157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f290e690700 1 -- 172.21.15.7:0/2956610152 >> v1:172.21.15.7:6789/0 conn(0x7f2908151d40 legacy=0x7f29080736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f290ce8d700 1 -- 172.21.15.7:0/2956610152 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894550528 0 0) 0x7f290806d430 con 0x7f2908151d40 2023-03-03T17:21:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f290e690700 1 -- 172.21.15.7:0/2956610152 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290806cde0 con 0x7f290806c480 2023-03-03T17:21:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f290e690700 1 --2- 172.21.15.7:0/2956610152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f290806c480 0x7f2908157fd0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f2900009d90 tx=0x7f290000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33bf5c1be1ab93e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f290ce8d700 1 -- 172.21.15.7:0/2956610152 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290000e070 con 0x7f290806c480 2023-03-03T17:21:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f290ce8d700 1 -- 172.21.15.7:0/2956610152 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290000bbd0 con 0x7f290806c480 2023-03-03T17:21:36.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f290ce8d700 1 -- 172.21.15.7:0/2956610152 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2900012430 con 0x7f290806c480 2023-03-03T17:21:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f29100f3700 1 -- 172.21.15.7:0/2956610152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f290806c480 msgr2=0x7f2908157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f29100f3700 1 --2- 172.21.15.7:0/2956610152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f290806c480 0x7f2908157fd0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f2900009d90 tx=0x7f290000b330 comp rx=0 tx=0).stop 2023-03-03T17:21:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f29100f3700 1 -- 172.21.15.7:0/2956610152 shutdown_connections 2023-03-03T17:21:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f29100f3700 1 --2- 172.21.15.7:0/2956610152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f290806c480 0x7f2908157fd0 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f29100f3700 1 --2- 172.21.15.7:0/2956610152 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29080c2930 0x7f2908157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.393+0000 7f29100f3700 1 -- 172.21.15.7:0/2956610152 >> 172.21.15.7:0/2956610152 conn(0x7f29080a94d0 msgr2=0x7f29080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:21:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.394+0000 7f29100f3700 1 -- 172.21.15.7:0/2956610152 shutdown_connections 2023-03-03T17:21:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.394+0000 7f29100f3700 1 -- 172.21.15.7:0/2956610152 wait complete. 2023-03-03T17:21:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.395+0000 7f29100f3700 1 Processor -- start 2023-03-03T17:21:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.395+0000 7f29100f3700 1 -- start start 2023-03-03T17:21:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.395+0000 7f29100f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290806c480 0x7f29080c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f29100f3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29080c2930 0x7f290814bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f29100f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2908151d40 0x7f290814c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:36.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f29100f3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29080bf420 con 0x7f29080c2930 2023-03-03T17:21:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f29100f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29080c1330 con 0x7f2908151d40 2023-03-03T17:21:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f29100f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29080ab120 con 0x7f290806c480 2023-03-03T17:21:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290d68e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29080c2930 0x7f290814bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290d68e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29080c2930 0x7f290814bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60904/0 (socket says 172.21.15.7:60904) 2023-03-03T17:21:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290d68e700 1 -- 172.21.15.7:0/2518968977 learned_addr learned my addr 172.21.15.7:0/2518968977 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:21:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290e690700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2908151d40 0x7f290814c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290de8f700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290806c480 0x7f29080c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290d68e700 1 -- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290806c480 msgr2=0x7f29080c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:36.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290d68e700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290806c480 0x7f29080c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:36.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290d68e700 1 -- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2908151d40 msgr2=0x7f290814c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:36.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290d68e700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2908151d40 0x7f290814c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:36.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290d68e700 1 -- 172.21.15.7:0/2518968977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2900009a70 con 0x7f29080c2930 2023-03-03T17:21:36.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290de8f700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290806c480 0x7f29080c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:21:36.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f290d68e700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29080c2930 0x7f290814bc00 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f28f800e810 tx=0x7f28f800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:36.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f28fe7fc700 1 -- 172.21.15.7:0/2518968977 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f8013610 con 0x7f29080c2930 2023-03-03T17:21:36.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f28fe7fc700 1 -- 172.21.15.7:0/2518968977 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f8013770 con 0x7f29080c2930 2023-03-03T17:21:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f29100f3700 1 -- 172.21.15.7:0/2518968977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2908073e00 con 0x7f29080c2930 2023-03-03T17:21:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.396+0000 7f28fe7fc700 1 -- 172.21.15.7:0/2518968977 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f8013b20 con 0x7f29080c2930 2023-03-03T17:21:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.397+0000 7f29100f3700 1 -- 172.21.15.7:0/2518968977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290806d0c0 con 0x7f29080c2930 2023-03-03T17:21:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.397+0000 7f29100f3700 1 -- 172.21.15.7:0/2518968977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f290806bea0 con 0x7f29080c2930 2023-03-03T17:21:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.398+0000 7f28fe7fc700 1 -- 172.21.15.7:0/2518968977 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f28f8023070 con 0x7f29080c2930 2023-03-03T17:21:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.398+0000 7f28fe7fc700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28ec0682f0 0x7f28ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.398+0000 7f28fe7fc700 1 -- 172.21.15.7:0/2518968977 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f28f8085db0 con 0x7f29080c2930 2023-03-03T17:21:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.399+0000 7f290de8f700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28ec0682f0 0x7f28ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.399+0000 7f290de8f700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28ec0682f0 0x7f28ec06a7a0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f2904000f60 tx=0x7f2904007cc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:36.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.403+0000 7f28fe7fc700 1 -- 172.21.15.7:0/2518968977 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f28f80520d0 con 0x7f29080c2930 2023-03-03T17:21:36.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.642+0000 7f29100f3700 1 -- 172.21.15.7:0/2518968977 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2908074bb0 con 0x7f29080c2930 2023-03-03T17:21:36.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.643+0000 7f28fe7fc700 1 -- 172.21.15.7:0/2518968977 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f28f80556f0 con 0x7f29080c2930 2023-03-03T17:21:36.642 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:21:36.643 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:21:36.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 -- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28ec0682f0 msgr2=0x7f28ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:36.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28ec0682f0 0x7f28ec06a7a0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f2904000f60 tx=0x7f2904007cc0 comp rx=0 tx=0).stop 2023-03-03T17:21:36.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 -- 172.21.15.7:0/2518968977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29080c2930 msgr2=0x7f290814bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:36.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29080c2930 0x7f290814bc00 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f28f800e810 tx=0x7f28f800eb20 comp rx=0 tx=0).stop 2023-03-03T17:21:36.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 -- 172.21.15.7:0/2518968977 shutdown_connections 2023-03-03T17:21:36.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2908151d40 0x7f290814c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:36.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f290806c480 0x7f29080c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:36.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28ec0682f0 0x7f28ec06a7a0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:36.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 --2- 172.21.15.7:0/2518968977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29080c2930 0x7f290814bc00 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:36.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 -- 172.21.15.7:0/2518968977 >> 172.21.15.7:0/2518968977 conn(0x7f29080a94d0 msgr2=0x7f29080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:21:36.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 -- 172.21.15.7:0/2518968977 shutdown_connections 2023-03-03T17:21:36.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:36.645+0000 7f29100f3700 1 -- 172.21.15.7:0/2518968977 wait complete. 2023-03-03T17:21:36.646 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:21:36.656 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before thrashing 2023-03-03T17:21:37.752 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:21:37.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.887+0000 7f985cdfa700 1 Processor -- start 2023-03-03T17:21:37.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.887+0000 7f985cdfa700 1 -- start start 2023-03-03T17:21:37.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.888+0000 7f985cdfa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f98580aa030 0x7f9858158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:37.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.888+0000 7f985cdfa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98580a9c20 con 0x7f98580aad20 2023-03-03T17:21:37.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.888+0000 7f985cdfa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9858154f10 con 0x7f985814b0d0 2023-03-03T17:21:37.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.888+0000 7f985cdfa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98580ac3c0 con 0x7f98580aa030 2023-03-03T17:21:37.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.888+0000 7f9856d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f98580aa030 0x7f9858158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:37.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.888+0000 7f9855d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f98580aad20 0x7f9858157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54894/0 (socket says 172.21.15.7:54894) 2023-03-03T17:21:37.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.888+0000 7f9855d9b700 1 -- 172.21.15.7:0/1167996206 learned_addr learned my addr 172.21.15.7:0/1167996206 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:21:37.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.888+0000 7f9856d9d700 1 -- 172.21.15.7:0/1167996206 >> v1:172.21.15.143:6789/0 conn(0x7f985814b0d0 legacy=0x7f9858071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:37.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f9856d9d700 1 -- 172.21.15.7:0/1167996206 >> v1:172.21.15.7:6789/0 conn(0x7f98580aad20 legacy=0x7f9858157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:37.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.888+0000 7f985559a700 1 -- 172.21.15.7:0/1167996206 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898128867 0 0) 0x7f98580a9c20 con 0x7f98580aad20 2023-03-03T17:21:37.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f9856d9d700 1 -- 172.21.15.7:0/1167996206 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985806fc70 con 0x7f98580aa030 2023-03-03T17:21:37.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f9856d9d700 1 --2- 172.21.15.7:0/1167996206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 0x7f9858158500 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f9850009d90 tx=0x7f985000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb5c7598590262d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:37.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f985559a700 1 -- 172.21.15.7:0/1167996206 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985000e070 con 0x7f98580aa030 2023-03-03T17:21:37.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f985559a700 1 -- 172.21.15.7:0/1167996206 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985000bbd0 con 0x7f98580aa030 2023-03-03T17:21:37.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f985559a700 1 -- 172.21.15.7:0/1167996206 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9850012430 con 0x7f98580aa030 2023-03-03T17:21:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f985cdfa700 1 -- 172.21.15.7:0/1167996206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 msgr2=0x7f9858158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f985cdfa700 1 --2- 172.21.15.7:0/1167996206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 0x7f9858158500 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7f9850009d90 tx=0x7f985000b330 comp rx=0 tx=0).stop 2023-03-03T17:21:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f985cdfa700 1 -- 172.21.15.7:0/1167996206 shutdown_connections 2023-03-03T17:21:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f985cdfa700 1 --2- 172.21.15.7:0/1167996206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 0x7f9858158500 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f985cdfa700 1 -- 172.21.15.7:0/1167996206 >> 172.21.15.7:0/1167996206 conn(0x7f98580c8900 msgr2=0x7f98580c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:21:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f985cdfa700 1 -- 172.21.15.7:0/1167996206 shutdown_connections 2023-03-03T17:21:37.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.889+0000 7f985cdfa700 1 -- 172.21.15.7:0/1167996206 wait complete. 2023-03-03T17:21:37.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.891+0000 7f985cdfa700 1 Processor -- start 2023-03-03T17:21:37.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.891+0000 7f985cdfa700 1 -- start start 2023-03-03T17:21:37.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f985cdfa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 0x7f9858153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:37.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f985cdfa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98580aad20 0x7f9858155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:37.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f985cdfa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f985814b0d0 0x7f98581ed5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:37.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f985cdfa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9858152d90 con 0x7f98580aa030 2023-03-03T17:21:37.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f985cdfa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98581ee020 con 0x7f98580aad20 2023-03-03T17:21:37.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f985cdfa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9858153dc0 con 0x7f985814b0d0 2023-03-03T17:21:37.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f985659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 0x7f9858153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:37.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f985659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 0x7f9858153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60920/0 (socket says 172.21.15.7:60920) 2023-03-03T17:21:37.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f985659c700 1 -- 172.21.15.7:0/2119580003 learned_addr learned my addr 172.21.15.7:0/2119580003 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:21:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f9855d9b700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98580aad20 0x7f9858155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.892+0000 7f9856d9d700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f985814b0d0 0x7f98581ed5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f985659c700 1 -- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f985814b0d0 msgr2=0x7f98581ed5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f985659c700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f985814b0d0 0x7f98581ed5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f985659c700 1 -- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98580aad20 msgr2=0x7f9858155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f985659c700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98580aad20 0x7f9858155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f985659c700 1 -- 172.21.15.7:0/2119580003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9850009a70 con 0x7f98580aa030 2023-03-03T17:21:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f9856d9d700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f985814b0d0 0x7f98581ed5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:21:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f985659c700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 0x7f9858153220 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f984c00c4e0 tx=0x7f984c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f9846ffd700 1 -- 172.21.15.7:0/2119580003 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c0052d0 con 0x7f98580aa030 2023-03-03T17:21:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f9846ffd700 1 -- 172.21.15.7:0/2119580003 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984c005430 con 0x7f98580aa030 2023-03-03T17:21:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f985cdfa700 1 -- 172.21.15.7:0/2119580003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985806f6b0 con 0x7f98580aa030 2023-03-03T17:21:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f9846ffd700 1 -- 172.21.15.7:0/2119580003 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c0038c0 con 0x7f98580aa030 2023-03-03T17:21:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.893+0000 7f985cdfa700 1 -- 172.21.15.7:0/2119580003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9858155160 con 0x7f98580aa030 2023-03-03T17:21:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.894+0000 7f985cdfa700 1 -- 172.21.15.7:0/2119580003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985806f990 con 0x7f98580aa030 2023-03-03T17:21:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.894+0000 7f9846ffd700 1 -- 172.21.15.7:0/2119580003 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f984c019030 con 0x7f98580aa030 2023-03-03T17:21:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.895+0000 7f9846ffd700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98380682f0 0x7f983806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.895+0000 7f9846ffd700 1 -- 172.21.15.7:0/2119580003 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f984c085480 con 0x7f98580aa030 2023-03-03T17:21:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.895+0000 7f9855d9b700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98380682f0 0x7f983806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.896+0000 7f9855d9b700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98380682f0 0x7f983806a7a0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f9848007b10 tx=0x7f9848007a60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:37.899+0000 7f9846ffd700 1 -- 172.21.15.7:0/2119580003 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f984c051820 con 0x7f98580aa030 2023-03-03T17:21:38.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.138+0000 7f985cdfa700 1 -- 172.21.15.7:0/2119580003 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9858152ed0 con 0x7f98580aa030 2023-03-03T17:21:38.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.139+0000 7f9846ffd700 1 -- 172.21.15.7:0/2119580003 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9858152ed0 con 0x7f98580aa030 2023-03-03T17:21:38.138 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:21:38.139 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:21:38.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 -- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98380682f0 msgr2=0x7f983806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:38.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98380682f0 0x7f983806a7a0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f9848007b10 tx=0x7f9848007a60 comp rx=0 tx=0).stop 2023-03-03T17:21:38.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 -- 172.21.15.7:0/2119580003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 msgr2=0x7f9858153220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:38.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 0x7f9858153220 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f984c00c4e0 tx=0x7f984c00c7f0 comp rx=0 tx=0).stop 2023-03-03T17:21:38.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 -- 172.21.15.7:0/2119580003 shutdown_connections 2023-03-03T17:21:38.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f985814b0d0 0x7f98581ed5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:38.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98380682f0 0x7f983806a7a0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:38.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98580aa030 0x7f9858153220 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:38.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 --2- 172.21.15.7:0/2119580003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98580aad20 0x7f9858155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:38.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 -- 172.21.15.7:0/2119580003 >> 172.21.15.7:0/2119580003 conn(0x7f98580c8900 msgr2=0x7f98580c3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:21:38.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 -- 172.21.15.7:0/2119580003 shutdown_connections 2023-03-03T17:21:38.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:38.141+0000 7f985cdfa700 1 -- 172.21.15.7:0/2119580003 wait complete. 2023-03-03T17:21:38.142 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:21:38.152 INFO:tasks.mds_thrash.fs.[b]:waiting for 28 secs before thrashing 2023-03-03T17:21:39.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:21:39.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.796+0000 7faa8f842700 1 Processor -- start 2023-03-03T17:21:39.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.797+0000 7faa8f842700 1 -- start start 2023-03-03T17:21:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.797+0000 7faa8f842700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faa880c00e0 0x7faa880c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.797+0000 7faa8f842700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa880c7310 con 0x7faa88154ff0 2023-03-03T17:21:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.797+0000 7faa8f842700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa880bfd10 con 0x7faa880caa30 2023-03-03T17:21:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.797+0000 7faa8f842700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa88155430 con 0x7faa880c00e0 2023-03-03T17:21:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.797+0000 7faa8dddf700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faa88154ff0 0x7faa8815ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54912/0 (socket says 172.21.15.7:54912) 2023-03-03T17:21:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa8dddf700 1 -- 172.21.15.7:0/4182703575 learned_addr learned my addr 172.21.15.7:0/4182703575 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:21:39.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa8cddd700 1 --2- 172.21.15.7:0/4182703575 >> v2:172.21.15.143:3300/0 conn(0x7faa880c00e0 0x7faa880c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:39.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823512268 0 0) 0x7faa880c7310 con 0x7faa88154ff0 2023-03-03T17:21:39.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa740035b0 con 0x7faa88154ff0 2023-03-03T17:21:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2569186656 0 0) 0x7faa740035b0 con 0x7faa88154ff0 2023-03-03T17:21:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 >> v2:172.21.15.143:3300/0 conn(0x7faa880c00e0 msgr2=0x7faa880c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:21:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 --2- 172.21.15.7:0/4182703575 >> v2:172.21.15.143:3300/0 conn(0x7faa880c00e0 0x7faa880c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 >> v1:172.21.15.143:6789/0 conn(0x7faa880caa30 legacy=0x7faa880c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa880bfa30 con 0x7faa88154ff0 2023-03-03T17:21:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7faa800019f0 con 0x7faa88154ff0 2023-03-03T17:21:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 >> v1:172.21.15.7:6789/0 conn(0x7faa88154ff0 legacy=0x7faa8815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa74003a70 0x7faa74003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa74004850 0x7faa74004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa74005250 0x7faa7400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa88155430 con 0x7faa74003a70 2023-03-03T17:21:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7400c190 con 0x7faa74005250 2023-03-03T17:21:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7400c350 con 0x7faa74004850 2023-03-03T17:21:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa8d5de700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa74003a70 0x7faa74003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa8dddf700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa74005250 0x7faa7400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa8cddd700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa74004850 0x7faa74004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa8d5de700 1 -- 172.21.15.7:0/4182703575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa74004850 msgr2=0x7faa74004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:21:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa8d5de700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa74004850 0x7faa74004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa8d5de700 1 -- 172.21.15.7:0/4182703575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa74005250 msgr2=0x7faa7400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:21:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa8d5de700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa74005250 0x7faa7400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.798+0000 7faa8d5de700 1 -- 172.21.15.7:0/4182703575 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa740035b0 con 0x7faa74003a70 2023-03-03T17:21:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.799+0000 7faa8dddf700 1 -- 172.21.15.7:0/4182703575 reap_dead start 2023-03-03T17:21:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.799+0000 7faa8d5de700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa74003a70 0x7faa74003e80 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7faa84007fd0 tx=0x7faa74005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5cf1e5a0a7c55 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.799+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840012f0 con 0x7faa74003a70 2023-03-03T17:21:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.799+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa84005310 con 0x7faa74003a70 2023-03-03T17:21:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.799+0000 7faa7bfff700 1 -- 172.21.15.7:0/4182703575 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84006860 con 0x7faa74003a70 2023-03-03T17:21:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.799+0000 7faa8f842700 1 -- 172.21.15.7:0/4182703575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa74003a70 msgr2=0x7faa74003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.799+0000 7faa8f842700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa74003a70 0x7faa74003e80 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7faa84007fd0 tx=0x7faa74005880 comp rx=0 tx=0).stop 2023-03-03T17:21:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.799+0000 7faa8f842700 1 -- 172.21.15.7:0/4182703575 shutdown_connections 2023-03-03T17:21:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.799+0000 7faa8f842700 1 --2- 172.21.15.7:0/4182703575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa74003a70 0x7faa74003e80 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:39.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.799+0000 7faa8f842700 1 -- 172.21.15.7:0/4182703575 >> 172.21.15.7:0/4182703575 conn(0x7faa88066e40 msgr2=0x7faa88071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:21:39.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.800+0000 7faa8f842700 1 -- 172.21.15.7:0/4182703575 shutdown_connections 2023-03-03T17:21:39.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.800+0000 7faa8f842700 1 -- 172.21.15.7:0/4182703575 wait complete. 2023-03-03T17:21:39.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.801+0000 7faa8f842700 1 Processor -- start 2023-03-03T17:21:39.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.801+0000 7faa8f842700 1 -- start start 2023-03-03T17:21:39.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.801+0000 7faa8f842700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa74003a70 0x7faa881eaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:39.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8f842700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa881eb4e0 0x7faa881e01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:39.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8f842700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa881dec80 0x7faa881df0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8f842700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa881eb030 con 0x7faa881eb4e0 2023-03-03T17:21:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8f842700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa881de7f0 con 0x7faa74003a70 2023-03-03T17:21:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8f842700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa881ddbe0 con 0x7faa881dec80 2023-03-03T17:21:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8cddd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa881eb4e0 0x7faa881e01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8cddd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa881eb4e0 0x7faa881e01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60942/0 (socket says 172.21.15.7:60942) 2023-03-03T17:21:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8cddd700 1 -- 172.21.15.7:0/311288658 learned_addr learned my addr 172.21.15.7:0/311288658 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:21:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8d5de700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa74003a70 0x7faa881eaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8dddf700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa881dec80 0x7faa881df0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8cddd700 1 -- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa881dec80 msgr2=0x7faa881df0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8cddd700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa881dec80 0x7faa881df0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8cddd700 1 -- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa74003a70 msgr2=0x7faa881eaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8cddd700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa74003a70 0x7faa881eaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8cddd700 1 -- 172.21.15.7:0/311288658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa881547f0 con 0x7faa881eb4e0 2023-03-03T17:21:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8cddd700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa881eb4e0 0x7faa881e01f0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0116d0 tx=0x7faa7c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa79ffb700 1 -- 172.21.15.7:0/311288658 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c015870 con 0x7faa881eb4e0 2023-03-03T17:21:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa79ffb700 1 -- 172.21.15.7:0/311288658 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c0159d0 con 0x7faa881eb4e0 2023-03-03T17:21:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa8f842700 1 -- 172.21.15.7:0/311288658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa88154510 con 0x7faa881eb4e0 2023-03-03T17:21:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.802+0000 7faa79ffb700 1 -- 172.21.15.7:0/311288658 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c015d40 con 0x7faa881eb4e0 2023-03-03T17:21:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.803+0000 7faa8f842700 1 -- 172.21.15.7:0/311288658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa88154ad0 con 0x7faa881eb4e0 2023-03-03T17:21:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.804+0000 7faa8f842700 1 -- 172.21.15.7:0/311288658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa881dead0 con 0x7faa881eb4e0 2023-03-03T17:21:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.804+0000 7faa79ffb700 1 -- 172.21.15.7:0/311288658 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faa7c012040 con 0x7faa881eb4e0 2023-03-03T17:21:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.804+0000 7faa79ffb700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa6c0682f0 0x7faa6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:21:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.805+0000 7faa79ffb700 1 -- 172.21.15.7:0/311288658 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faa7c0866e0 con 0x7faa881eb4e0 2023-03-03T17:21:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.805+0000 7faa8d5de700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa6c0682f0 0x7faa6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:21:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.805+0000 7faa8d5de700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa6c0682f0 0x7faa6c06a7a0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7faa84007fd0 tx=0x7faa84011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:21:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:39.809+0000 7faa79ffb700 1 -- 172.21.15.7:0/311288658 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faa7c052b30 con 0x7faa881eb4e0 2023-03-03T17:21:40.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.046+0000 7faa8f842700 1 -- 172.21.15.7:0/311288658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa880c7f90 con 0x7faa881eb4e0 2023-03-03T17:21:40.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.047+0000 7faa79ffb700 1 -- 172.21.15.7:0/311288658 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faa7c056150 con 0x7faa881eb4e0 2023-03-03T17:21:40.046 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:21:40.046 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:21:40.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 -- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa6c0682f0 msgr2=0x7faa6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:40.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa6c0682f0 0x7faa6c06a7a0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7faa84007fd0 tx=0x7faa84011000 comp rx=0 tx=0).stop 2023-03-03T17:21:40.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 -- 172.21.15.7:0/311288658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa881eb4e0 msgr2=0x7faa881e01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:21:40.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa881eb4e0 0x7faa881e01f0 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0116d0 tx=0x7faa7c011a90 comp rx=0 tx=0).stop 2023-03-03T17:21:40.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 -- 172.21.15.7:0/311288658 shutdown_connections 2023-03-03T17:21:40.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa881dec80 0x7faa881df0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:40.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa74003a70 0x7faa881eaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:40.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa6c0682f0 0x7faa6c06a7a0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:40.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 --2- 172.21.15.7:0/311288658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa881eb4e0 0x7faa881e01f0 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:21:40.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 -- 172.21.15.7:0/311288658 >> 172.21.15.7:0/311288658 conn(0x7faa88066e40 msgr2=0x7faa880c6880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:21:40.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 -- 172.21.15.7:0/311288658 shutdown_connections 2023-03-03T17:21:40.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:21:40.049+0000 7faa8f842700 1 -- 172.21.15.7:0/311288658 wait complete. 2023-03-03T17:21:40.050 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:21:40.060 INFO:tasks.mds_thrash.fs.[a]:waiting for 86 secs before thrashing 2023-03-03T17:21:55.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:21:55.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:06.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:22:06.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.291+0000 7f3136585700 1 Processor -- start 2023-03-03T17:22:06.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.291+0000 7f3136585700 1 -- start start 2023-03-03T17:22:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.292+0000 7f3136585700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f313014f9d0 0x7f3130156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.292+0000 7f3136585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31300abe20 0x7f31301529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.292+0000 7f3136585700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31300c2f30 con 0x7f313014fda0 2023-03-03T17:22:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.292+0000 7f3136585700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31300c31d0 con 0x7f313014f9d0 2023-03-03T17:22:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.292+0000 7f3136585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31301524a0 con 0x7f31300abe20 2023-03-03T17:22:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.292+0000 7f312ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f313014f9d0 0x7f3130156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.292+0000 7f312ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f313014f9d0 0x7f3130156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60982/0 (socket says 172.21.15.7:60982) 2023-03-03T17:22:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.292+0000 7f312ffff700 1 -- 172.21.15.7:0/3292998337 learned_addr learned my addr 172.21.15.7:0/3292998337 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:22:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.293+0000 7f3134b22700 1 --2- 172.21.15.7:0/3292998337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31300abe20 0x7f31301529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.293+0000 7f312ffff700 1 -- 172.21.15.7:0/3292998337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31300abe20 msgr2=0x7f31301529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:22:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.293+0000 7f312ffff700 1 --2- 172.21.15.7:0/3292998337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31300abe20 0x7f31301529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.293+0000 7f312ffff700 1 -- 172.21.15.7:0/3292998337 >> v1:172.21.15.143:6789/0 conn(0x7f313014fda0 legacy=0x7f3130071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.293+0000 7f312ffff700 1 -- 172.21.15.7:0/3292998337 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31300c34b0 con 0x7f313014f9d0 2023-03-03T17:22:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.293+0000 7f3134b22700 1 --2- 172.21.15.7:0/3292998337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31300abe20 0x7f31301529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:22:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.293+0000 7f312ffff700 1 --2- 172.21.15.7:0/3292998337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014f9d0 0x7f3130156430 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7f31200098f0 tx=0x7f3120009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7997a9e9c1cba09c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.293+0000 7f312effd700 1 -- 172.21.15.7:0/3292998337 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312000b8a0 con 0x7f313014f9d0 2023-03-03T17:22:06.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.293+0000 7f312effd700 1 -- 172.21.15.7:0/3292998337 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312000ba00 con 0x7f313014f9d0 2023-03-03T17:22:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.293+0000 7f312effd700 1 -- 172.21.15.7:0/3292998337 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312000bd90 con 0x7f313014f9d0 2023-03-03T17:22:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.294+0000 7f3136585700 1 -- 172.21.15.7:0/3292998337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014f9d0 msgr2=0x7f3130156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.294+0000 7f3136585700 1 --2- 172.21.15.7:0/3292998337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014f9d0 0x7f3130156430 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7f31200098f0 tx=0x7f3120009c00 comp rx=0 tx=0).stop 2023-03-03T17:22:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.294+0000 7f3136585700 1 -- 172.21.15.7:0/3292998337 shutdown_connections 2023-03-03T17:22:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.294+0000 7f3136585700 1 --2- 172.21.15.7:0/3292998337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31300abe20 0x7f31301529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.294+0000 7f3136585700 1 --2- 172.21.15.7:0/3292998337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014f9d0 0x7f3130156430 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.294+0000 7f3136585700 1 -- 172.21.15.7:0/3292998337 >> 172.21.15.7:0/3292998337 conn(0x7f31300c8900 msgr2=0x7f31300c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:22:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.294+0000 7f3136585700 1 -- 172.21.15.7:0/3292998337 shutdown_connections 2023-03-03T17:22:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.294+0000 7f3136585700 1 -- 172.21.15.7:0/3292998337 wait complete. 2023-03-03T17:22:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.295+0000 7f3136585700 1 Processor -- start 2023-03-03T17:22:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.295+0000 7f3136585700 1 -- start start 2023-03-03T17:22:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3136585700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31300abe20 0x7f31301549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3136585700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313014f9d0 0x7f313014dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3136585700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014fda0 0x7f313014e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3136585700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313014eb50 con 0x7f313014fda0 2023-03-03T17:22:06.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3136585700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313006f730 con 0x7f31300abe20 2023-03-03T17:22:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3136585700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313014e960 con 0x7f313014f9d0 2023-03-03T17:22:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3134b22700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014fda0 0x7f313014e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3134b22700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014fda0 0x7f313014e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60992/0 (socket says 172.21.15.7:60992) 2023-03-03T17:22:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3134b22700 1 -- 172.21.15.7:0/2469128687 learned_addr learned my addr 172.21.15.7:0/2469128687 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:22:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f312ffff700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31300abe20 0x7f31301549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f312f7fe700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313014f9d0 0x7f313014dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:06.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3134b22700 1 -- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313014f9d0 msgr2=0x7f313014dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3134b22700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313014f9d0 0x7f313014dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3134b22700 1 -- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31300abe20 msgr2=0x7f31301549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3134b22700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31300abe20 0x7f31301549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3134b22700 1 -- 172.21.15.7:0/2469128687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3120009550 con 0x7f313014fda0 2023-03-03T17:22:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.296+0000 7f3134b22700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014fda0 0x7f313014e420 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f312800cdb0 tx=0x7f312800a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.297+0000 7f312cff9700 1 -- 172.21.15.7:0/2469128687 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312800f480 con 0x7f313014fda0 2023-03-03T17:22:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.297+0000 7f312cff9700 1 -- 172.21.15.7:0/2469128687 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312800f600 con 0x7f313014fda0 2023-03-03T17:22:06.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.297+0000 7f3136585700 1 -- 172.21.15.7:0/2469128687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31300ac350 con 0x7f313014fda0 2023-03-03T17:22:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.297+0000 7f312cff9700 1 -- 172.21.15.7:0/2469128687 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312800f9b0 con 0x7f313014fda0 2023-03-03T17:22:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.297+0000 7f3136585700 1 -- 172.21.15.7:0/2469128687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31300ab630 con 0x7f313014fda0 2023-03-03T17:22:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.298+0000 7f3136585700 1 -- 172.21.15.7:0/2469128687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3130071fa0 con 0x7f313014fda0 2023-03-03T17:22:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.298+0000 7f312cff9700 1 -- 172.21.15.7:0/2469128687 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3128009030 con 0x7f313014fda0 2023-03-03T17:22:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.299+0000 7f312cff9700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f311c0682f0 0x7f311c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.299+0000 7f312cff9700 1 -- 172.21.15.7:0/2469128687 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3128086a70 con 0x7f313014fda0 2023-03-03T17:22:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.299+0000 7f312ffff700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f311c0682f0 0x7f311c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:06.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.300+0000 7f312ffff700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f311c0682f0 0x7f311c06a7a0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f3120009c40 tx=0x7f3120008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:06.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.303+0000 7f312cff9700 1 -- 172.21.15.7:0/2469128687 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3128052e10 con 0x7f313014fda0 2023-03-03T17:22:06.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.541+0000 7f3136585700 1 -- 172.21.15.7:0/2469128687 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3130074bb0 con 0x7f313014fda0 2023-03-03T17:22:06.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.541+0000 7f312cff9700 1 -- 172.21.15.7:0/2469128687 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3128057e60 con 0x7f313014fda0 2023-03-03T17:22:06.542 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:22:06.542 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:22:06.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 -- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f311c0682f0 msgr2=0x7f311c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:06.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f311c0682f0 0x7f311c06a7a0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f3120009c40 tx=0x7f3120008030 comp rx=0 tx=0).stop 2023-03-03T17:22:06.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 -- 172.21.15.7:0/2469128687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014fda0 msgr2=0x7f313014e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:06.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014fda0 0x7f313014e420 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f312800cdb0 tx=0x7f312800a560 comp rx=0 tx=0).stop 2023-03-03T17:22:06.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 -- 172.21.15.7:0/2469128687 shutdown_connections 2023-03-03T17:22:06.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f311c0682f0 0x7f311c06a7a0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:06.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f313014fda0 0x7f313014e420 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:06.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31300abe20 0x7f31301549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:06.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 --2- 172.21.15.7:0/2469128687 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f313014f9d0 0x7f313014dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:06.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 -- 172.21.15.7:0/2469128687 >> 172.21.15.7:0/2469128687 conn(0x7f31300c8900 msgr2=0x7f31300c6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:22:06.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 -- 172.21.15.7:0/2469128687 shutdown_connections 2023-03-03T17:22:06.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:06.543+0000 7f3136585700 1 -- 172.21.15.7:0/2469128687 wait complete. 2023-03-03T17:22:06.546 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:22:06.556 INFO:tasks.mds_thrash.fs.[b]:waiting for 14 secs before thrashing 2023-03-03T17:22:20.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:22:20.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.696+0000 7fa1c2e57700 1 Processor -- start 2023-03-03T17:22:20.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.696+0000 7fa1c2e57700 1 -- start start 2023-03-03T17:22:20.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.697+0000 7fa1c2e57700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa1bc151e80 0x7fa1bc0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:20.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.697+0000 7fa1c2e57700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1bc0c6fe0 0x7fa1bc0bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:20.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.697+0000 7fa1c2e57700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc0c08d0 con 0x7fa1bc155410 2023-03-03T17:22:20.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.697+0000 7fa1c2e57700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1bc152250 con 0x7fa1bc151e80 2023-03-03T17:22:20.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.697+0000 7fa1c2e57700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc0c0fb0 con 0x7fa1bc0c6fe0 2023-03-03T17:22:20.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.697+0000 7fa1c0bf3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa1bc151e80 0x7fa1bc0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:20.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.697+0000 7fa1c0bf3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa1bc151e80 0x7fa1bc0c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32768/0 (socket says 172.21.15.7:32768) 2023-03-03T17:22:20.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.697+0000 7fa1c0bf3700 1 -- 172.21.15.7:0/3900245924 learned_addr learned my addr 172.21.15.7:0/3900245924 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:22:20.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.697+0000 7fa1c13f4700 1 --2- 172.21.15.7:0/3900245924 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1bc0c6fe0 0x7fa1bc0bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:20.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.698+0000 7fa1c0bf3700 1 -- 172.21.15.7:0/3900245924 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1bc0c6fe0 msgr2=0x7fa1bc0bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:22:20.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.698+0000 7fa1c0bf3700 1 --2- 172.21.15.7:0/3900245924 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1bc0c6fe0 0x7fa1bc0bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:20.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.698+0000 7fa1c0bf3700 1 -- 172.21.15.7:0/3900245924 >> v1:172.21.15.143:6789/0 conn(0x7fa1bc155410 legacy=0x7fa1bc0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:20.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.698+0000 7fa1c0bf3700 1 -- 172.21.15.7:0/3900245924 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc154930 con 0x7fa1bc151e80 2023-03-03T17:22:20.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.698+0000 7fa1c0bf3700 1 --2- 172.21.15.7:0/3900245924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 0x7fa1bc0c6aa0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00098f0 tx=0x7fa1b0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d090107702150dd2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:20.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.698+0000 7fa1bb7fe700 1 -- 172.21.15.7:0/3900245924 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0011830 con 0x7fa1bc151e80 2023-03-03T17:22:20.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.698+0000 7fa1bb7fe700 1 -- 172.21.15.7:0/3900245924 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b0011990 con 0x7fa1bc151e80 2023-03-03T17:22:20.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.698+0000 7fa1bb7fe700 1 -- 172.21.15.7:0/3900245924 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0011d20 con 0x7fa1bc151e80 2023-03-03T17:22:20.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.699+0000 7fa1c2e57700 1 -- 172.21.15.7:0/3900245924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 msgr2=0x7fa1bc0c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:20.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.699+0000 7fa1c2e57700 1 --2- 172.21.15.7:0/3900245924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 0x7fa1bc0c6aa0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00098f0 tx=0x7fa1b0009c00 comp rx=0 tx=0).stop 2023-03-03T17:22:20.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.699+0000 7fa1c2e57700 1 -- 172.21.15.7:0/3900245924 shutdown_connections 2023-03-03T17:22:20.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.699+0000 7fa1c2e57700 1 --2- 172.21.15.7:0/3900245924 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1bc0c6fe0 0x7fa1bc0bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:20.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.699+0000 7fa1c2e57700 1 --2- 172.21.15.7:0/3900245924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 0x7fa1bc0c6aa0 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:20.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.699+0000 7fa1c2e57700 1 -- 172.21.15.7:0/3900245924 >> 172.21.15.7:0/3900245924 conn(0x7fa1bc071630 msgr2=0x7fa1bc06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:22:20.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.699+0000 7fa1c2e57700 1 -- 172.21.15.7:0/3900245924 shutdown_connections 2023-03-03T17:22:20.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.699+0000 7fa1c2e57700 1 -- 172.21.15.7:0/3900245924 wait complete. 2023-03-03T17:22:20.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.700+0000 7fa1c2e57700 1 Processor -- start 2023-03-03T17:22:20.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.700+0000 7fa1c2e57700 1 -- start start 2023-03-03T17:22:20.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1c2e57700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1bc0c6fe0 0x7fa1bc1e1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:20.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1c2e57700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 0x7fa1bc1ea330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:20.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1c2e57700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1bc155410 0x7fa1bc1ea870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:20.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1c2e57700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc156bd0 con 0x7fa1bc151e80 2023-03-03T17:22:20.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1c2e57700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc157ce0 con 0x7fa1bc0c6fe0 2023-03-03T17:22:20.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1c2e57700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc157440 con 0x7fa1bc155410 2023-03-03T17:22:20.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 0x7fa1bc1ea330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:20.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 0x7fa1bc1ea330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32776/0 (socket says 172.21.15.7:32776) 2023-03-03T17:22:20.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1bbfff700 1 -- 172.21.15.7:0/999461590 learned_addr learned my addr 172.21.15.7:0/999461590 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:22:20.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1c0bf3700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1bc0c6fe0 0x7fa1bc1e1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:20.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1c13f4700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1bc155410 0x7fa1bc1ea870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:20.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1bbfff700 1 -- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1bc155410 msgr2=0x7fa1bc1ea870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:20.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1bbfff700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1bc155410 0x7fa1bc1ea870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:20.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1bbfff700 1 -- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1bc0c6fe0 msgr2=0x7fa1bc1e1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:20.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1bbfff700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1bc0c6fe0 0x7fa1bc1e1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:20.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1bbfff700 1 -- 172.21.15.7:0/999461590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b0009550 con 0x7fa1bc151e80 2023-03-03T17:22:20.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1bbfff700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 0x7fa1bc1ea330 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800c480 tx=0x7fa1a800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:20.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.701+0000 7fa1b97fa700 1 -- 172.21.15.7:0/999461590 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a80052d0 con 0x7fa1bc151e80 2023-03-03T17:22:20.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.702+0000 7fa1b97fa700 1 -- 172.21.15.7:0/999461590 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a8005430 con 0x7fa1bc151e80 2023-03-03T17:22:20.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.702+0000 7fa1c2e57700 1 -- 172.21.15.7:0/999461590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1bc154ef0 con 0x7fa1bc151e80 2023-03-03T17:22:20.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.702+0000 7fa1b97fa700 1 -- 172.21.15.7:0/999461590 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a80038c0 con 0x7fa1bc151e80 2023-03-03T17:22:20.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.702+0000 7fa1c2e57700 1 -- 172.21.15.7:0/999461590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1bc154c10 con 0x7fa1bc151e80 2023-03-03T17:22:20.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.703+0000 7fa1c2e57700 1 -- 172.21.15.7:0/999461590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1bc0c5220 con 0x7fa1bc151e80 2023-03-03T17:22:20.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.703+0000 7fa1b97fa700 1 -- 172.21.15.7:0/999461590 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa1a8019030 con 0x7fa1bc151e80 2023-03-03T17:22:20.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.703+0000 7fa1b97fa700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1a00682f0 0x7fa1a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:20.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.704+0000 7fa1b97fa700 1 -- 172.21.15.7:0/999461590 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa1a80855c0 con 0x7fa1bc151e80 2023-03-03T17:22:20.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.704+0000 7fa1c0bf3700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1a00682f0 0x7fa1a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:20.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.704+0000 7fa1c0bf3700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1a00682f0 0x7fa1a006a7a0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0014040 tx=0x7fa1b0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:20.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.708+0000 7fa1b97fa700 1 -- 172.21.15.7:0/999461590 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa1a80518e0 con 0x7fa1bc151e80 2023-03-03T17:22:20.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.947+0000 7fa1c2e57700 1 -- 172.21.15.7:0/999461590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1bc074bb0 con 0x7fa1bc151e80 2023-03-03T17:22:20.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.947+0000 7fa1b97fa700 1 -- 172.21.15.7:0/999461590 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa1a8054f00 con 0x7fa1bc151e80 2023-03-03T17:22:20.947 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:22:20.947 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:22:20.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 -- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1a00682f0 msgr2=0x7fa1a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:20.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1a00682f0 0x7fa1a006a7a0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0014040 tx=0x7fa1b0008030 comp rx=0 tx=0).stop 2023-03-03T17:22:20.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 -- 172.21.15.7:0/999461590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 msgr2=0x7fa1bc1ea330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:20.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 0x7fa1bc1ea330 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800c480 tx=0x7fa1a800c840 comp rx=0 tx=0).stop 2023-03-03T17:22:20.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 -- 172.21.15.7:0/999461590 shutdown_connections 2023-03-03T17:22:20.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1bc155410 0x7fa1bc1ea870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:20.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1bc0c6fe0 0x7fa1bc1e1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:20.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1a00682f0 0x7fa1a006a7a0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:20.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 --2- 172.21.15.7:0/999461590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1bc151e80 0x7fa1bc1ea330 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:20.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 -- 172.21.15.7:0/999461590 >> 172.21.15.7:0/999461590 conn(0x7fa1bc071630 msgr2=0x7fa1bc0c27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:22:20.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.949+0000 7fa1c2e57700 1 -- 172.21.15.7:0/999461590 shutdown_connections 2023-03-03T17:22:20.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:20.950+0000 7fa1c2e57700 1 -- 172.21.15.7:0/999461590 wait complete. 2023-03-03T17:22:20.951 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:22:20.961 INFO:tasks.mds_thrash.fs.[b]:waiting for 10 secs before thrashing 2023-03-03T17:22:25.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:25.737 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:30.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:22:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.101+0000 7f879ca3d700 1 Processor -- start 2023-03-03T17:22:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.101+0000 7f879ca3d700 1 -- start start 2023-03-03T17:22:31.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.102+0000 7f879ca3d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8798151ab0 0x7f87980736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:31.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.102+0000 7f879ca3d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879806d780 con 0x7f879806c5b0 2023-03-03T17:22:31.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.102+0000 7f879ca3d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879806cb70 con 0x7f8798073c30 2023-03-03T17:22:31.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.102+0000 7f879ca3d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f879806ce50 con 0x7f8798151ab0 2023-03-03T17:22:31.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.102+0000 7f879659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8798151ab0 0x7f87980736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:31.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.102+0000 7f8796d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f879806c5b0 0x7f8798157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55004/0 (socket says 172.21.15.7:55004) 2023-03-03T17:22:31.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.102+0000 7f8796d9d700 1 -- 172.21.15.7:0/2337526380 learned_addr learned my addr 172.21.15.7:0/2337526380 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:22:31.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.103+0000 7f879659c700 1 -- 172.21.15.7:0/2337526380 >> v1:172.21.15.143:6789/0 conn(0x7f8798073c30 legacy=0x7f879806bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:22:31.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.103+0000 7f879659c700 1 -- 172.21.15.7:0/2337526380 >> v1:172.21.15.7:6789/0 conn(0x7f879806c5b0 legacy=0x7f8798157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:31.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.103+0000 7f879659c700 1 -- 172.21.15.7:0/2337526380 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879806d130 con 0x7f8798151ab0 2023-03-03T17:22:31.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.103+0000 7f879659c700 1 --2- 172.21.15.7:0/2337526380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798151ab0 0x7f87980736f0 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f8788009840 tx=0x7f8788009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a985b97b6e948b47 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:31.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.103+0000 7f879559a700 1 -- 172.21.15.7:0/2337526380 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788004230 con 0x7f8798151ab0 2023-03-03T17:22:31.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.103+0000 7f879559a700 1 -- 172.21.15.7:0/2337526380 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8788004390 con 0x7f8798151ab0 2023-03-03T17:22:31.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.103+0000 7f879559a700 1 -- 172.21.15.7:0/2337526380 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87880036a0 con 0x7f8798151ab0 2023-03-03T17:22:31.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.104+0000 7f879ca3d700 1 -- 172.21.15.7:0/2337526380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798151ab0 msgr2=0x7f87980736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:31.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.104+0000 7f879ca3d700 1 --2- 172.21.15.7:0/2337526380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798151ab0 0x7f87980736f0 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f8788009840 tx=0x7f8788009b50 comp rx=0 tx=0).stop 2023-03-03T17:22:31.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.104+0000 7f879ca3d700 1 -- 172.21.15.7:0/2337526380 shutdown_connections 2023-03-03T17:22:31.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.104+0000 7f879ca3d700 1 --2- 172.21.15.7:0/2337526380 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798151ab0 0x7f87980736f0 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:31.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.104+0000 7f879ca3d700 1 -- 172.21.15.7:0/2337526380 >> 172.21.15.7:0/2337526380 conn(0x7f87980a94d0 msgr2=0x7f87980a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:22:31.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.104+0000 7f879ca3d700 1 -- 172.21.15.7:0/2337526380 shutdown_connections 2023-03-03T17:22:31.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.104+0000 7f879ca3d700 1 -- 172.21.15.7:0/2337526380 wait complete. 2023-03-03T17:22:31.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.105+0000 7f879ca3d700 1 Processor -- start 2023-03-03T17:22:31.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.105+0000 7f879ca3d700 1 -- start start 2023-03-03T17:22:31.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f879ca3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f879806c5b0 0x7f87981ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:31.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f879ca3d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798073c30 0x7f879814e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:31.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f879ca3d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798151ab0 0x7f879814adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:31.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f879ca3d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87981eadf0 con 0x7f8798073c30 2023-03-03T17:22:31.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f879ca3d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87981ea420 con 0x7f8798151ab0 2023-03-03T17:22:31.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f879ca3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8798155400 con 0x7f879806c5b0 2023-03-03T17:22:31.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f8795d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798073c30 0x7f879814e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:31.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f8795d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798073c30 0x7f879814e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32796/0 (socket says 172.21.15.7:32796) 2023-03-03T17:22:31.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f8795d9b700 1 -- 172.21.15.7:0/2646717142 learned_addr learned my addr 172.21.15.7:0/2646717142 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:22:31.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f879659c700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f879806c5b0 0x7f87981ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:31.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f8796d9d700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798151ab0 0x7f879814adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:31.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f8795d9b700 1 -- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f879806c5b0 msgr2=0x7f87981ea8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:31.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f8795d9b700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f879806c5b0 0x7f87981ea8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:31.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f8795d9b700 1 -- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798151ab0 msgr2=0x7f879814adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:31.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f8795d9b700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798151ab0 0x7f879814adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:31.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.106+0000 7f8795d9b700 1 -- 172.21.15.7:0/2646717142 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87880094a0 con 0x7f8798073c30 2023-03-03T17:22:31.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.107+0000 7f8795d9b700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798073c30 0x7f879814e890 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f878000c4d0 tx=0x7f878000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:31.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.107+0000 7f878effd700 1 -- 172.21.15.7:0/2646717142 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780011790 con 0x7f8798073c30 2023-03-03T17:22:31.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.107+0000 7f878effd700 1 -- 172.21.15.7:0/2646717142 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87800118f0 con 0x7f8798073c30 2023-03-03T17:22:31.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.107+0000 7f879ca3d700 1 -- 172.21.15.7:0/2646717142 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8798151e80 con 0x7f8798073c30 2023-03-03T17:22:31.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.107+0000 7f878effd700 1 -- 172.21.15.7:0/2646717142 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780011ca0 con 0x7f8798073c30 2023-03-03T17:22:31.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.107+0000 7f879ca3d700 1 -- 172.21.15.7:0/2646717142 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879806d410 con 0x7f8798073c30 2023-03-03T17:22:31.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.108+0000 7f879ca3d700 1 -- 172.21.15.7:0/2646717142 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8798152010 con 0x7f8798073c30 2023-03-03T17:22:31.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.108+0000 7f878effd700 1 -- 172.21.15.7:0/2646717142 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8780010050 con 0x7f8798073c30 2023-03-03T17:22:31.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.109+0000 7f878effd700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87780682f0 0x7f877806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:22:31.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.109+0000 7f878effd700 1 -- 172.21.15.7:0/2646717142 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8780085d70 con 0x7f8798073c30 2023-03-03T17:22:31.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.109+0000 7f879659c700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87780682f0 0x7f877806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:22:31.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.109+0000 7f879659c700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87780682f0 0x7f877806a7a0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f87880090b0 tx=0x7f8788008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:22:31.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.113+0000 7f878effd700 1 -- 172.21.15.7:0/2646717142 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8780052110 con 0x7f8798073c30 2023-03-03T17:22:31.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.350+0000 7f879ca3d700 1 -- 172.21.15.7:0/2646717142 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8798074bb0 con 0x7f8798073c30 2023-03-03T17:22:31.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.351+0000 7f878effd700 1 -- 172.21.15.7:0/2646717142 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8780055730 con 0x7f8798073c30 2023-03-03T17:22:31.350 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:22:31.351 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:22:31.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 -- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87780682f0 msgr2=0x7f877806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:31.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87780682f0 0x7f877806a7a0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f87880090b0 tx=0x7f8788008030 comp rx=0 tx=0).stop 2023-03-03T17:22:31.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 -- 172.21.15.7:0/2646717142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798073c30 msgr2=0x7f879814e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:22:31.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798073c30 0x7f879814e890 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f878000c4d0 tx=0x7f878000c7e0 comp rx=0 tx=0).stop 2023-03-03T17:22:31.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 -- 172.21.15.7:0/2646717142 shutdown_connections 2023-03-03T17:22:31.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8798151ab0 0x7f879814adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:31.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f879806c5b0 0x7f87981ea8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:31.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87780682f0 0x7f877806a7a0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:31.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 --2- 172.21.15.7:0/2646717142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8798073c30 0x7f879814e890 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:22:31.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 -- 172.21.15.7:0/2646717142 >> 172.21.15.7:0/2646717142 conn(0x7f87980a94d0 msgr2=0x7f8798156b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:22:31.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 -- 172.21.15.7:0/2646717142 shutdown_connections 2023-03-03T17:22:31.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:22:31.353+0000 7f879ca3d700 1 -- 172.21.15.7:0/2646717142 wait complete. 2023-03-03T17:22:31.355 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:22:31.365 INFO:tasks.mds_thrash.fs.[b]:waiting for 58 secs before thrashing 2023-03-03T17:22:55.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:22:55.809 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:06.061 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:23:06.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.210+0000 7f1941a30700 1 Processor -- start 2023-03-03T17:23:06.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.210+0000 7f1941a30700 1 -- start start 2023-03-03T17:23:06.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f1941a30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c151cc0 0x7f193c0ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:06.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f1941a30700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f193c0b00b0 0x7f193c0c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:06.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f1941a30700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193c0c7ca0 con 0x7f193c0cac10 2023-03-03T17:23:06.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f1941a30700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193c152090 con 0x7f193c0b00b0 2023-03-03T17:23:06.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f1941a30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c0c6cf0 con 0x7f193c151cc0 2023-03-03T17:23:06.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f193a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f193c0b00b0 0x7f193c0c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:06.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f193a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f193c0b00b0 0x7f193c0c7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32828/0 (socket says 172.21.15.7:32828) 2023-03-03T17:23:06.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f193a7fc700 1 -- 172.21.15.7:0/2306039386 learned_addr learned my addr 172.21.15.7:0/2306039386 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:23:06.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f193affd700 1 --2- 172.21.15.7:0/2306039386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c151cc0 0x7f193c0ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:06.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f193a7fc700 1 -- 172.21.15.7:0/2306039386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c151cc0 msgr2=0x7f193c0ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:23:06.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f193a7fc700 1 --2- 172.21.15.7:0/2306039386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c151cc0 0x7f193c0ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:06.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f193a7fc700 1 -- 172.21.15.7:0/2306039386 >> v1:172.21.15.143:6789/0 conn(0x7f193c0cac10 legacy=0x7f193c0c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:06.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f193a7fc700 1 -- 172.21.15.7:0/2306039386 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c1548f0 con 0x7f193c0b00b0 2023-03-03T17:23:06.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.211+0000 7f193a7fc700 1 --2- 172.21.15.7:0/2306039386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0b00b0 0x7f193c0c7050 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f19300098f0 tx=0x7f1930009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a586a82a40fcd11e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:06.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1939ffb700 1 -- 172.21.15.7:0/2306039386 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930011830 con 0x7f193c0b00b0 2023-03-03T17:23:06.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1939ffb700 1 -- 172.21.15.7:0/2306039386 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1930011990 con 0x7f193c0b00b0 2023-03-03T17:23:06.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1939ffb700 1 -- 172.21.15.7:0/2306039386 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930011d20 con 0x7f193c0b00b0 2023-03-03T17:23:06.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1941a30700 1 -- 172.21.15.7:0/2306039386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0b00b0 msgr2=0x7f193c0c7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:06.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1941a30700 1 --2- 172.21.15.7:0/2306039386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0b00b0 0x7f193c0c7050 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f19300098f0 tx=0x7f1930009c00 comp rx=0 tx=0).stop 2023-03-03T17:23:06.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1941a30700 1 -- 172.21.15.7:0/2306039386 shutdown_connections 2023-03-03T17:23:06.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1941a30700 1 --2- 172.21.15.7:0/2306039386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c151cc0 0x7f193c0ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:06.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1941a30700 1 --2- 172.21.15.7:0/2306039386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0b00b0 0x7f193c0c7050 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:06.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1941a30700 1 -- 172.21.15.7:0/2306039386 >> 172.21.15.7:0/2306039386 conn(0x7f193c06b980 msgr2=0x7f193c071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:23:06.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1941a30700 1 -- 172.21.15.7:0/2306039386 shutdown_connections 2023-03-03T17:23:06.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.212+0000 7f1941a30700 1 -- 172.21.15.7:0/2306039386 wait complete. 2023-03-03T17:23:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.214+0000 7f1941a30700 1 Processor -- start 2023-03-03T17:23:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.214+0000 7f1941a30700 1 -- start start 2023-03-03T17:23:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f1941a30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c0b00b0 0x7f193c0c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f1941a30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0cac10 0x7f193c0c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f1941a30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c151cc0 0x7f193c146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f1941a30700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c14dc70 con 0x7f193c0cac10 2023-03-03T17:23:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f1941a30700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1eed30 con 0x7f193c0b00b0 2023-03-03T17:23:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f1941a30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c1ee120 con 0x7f193c151cc0 2023-03-03T17:23:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0cac10 0x7f193c0c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0cac10 0x7f193c0c3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32834/0 (socket says 172.21.15.7:32834) 2023-03-03T17:23:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193a7fc700 1 -- 172.21.15.7:0/1544759427 learned_addr learned my addr 172.21.15.7:0/1544759427 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:23:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193affd700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c0b00b0 0x7f193c0c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193b7fe700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c151cc0 0x7f193c146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:06.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193a7fc700 1 -- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c151cc0 msgr2=0x7f193c146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:06.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193a7fc700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c151cc0 0x7f193c146f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:06.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193a7fc700 1 -- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c0b00b0 msgr2=0x7f193c0c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:06.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193a7fc700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c0b00b0 0x7f193c0c94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:06.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193a7fc700 1 -- 172.21.15.7:0/1544759427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1930009550 con 0x7f193c0cac10 2023-03-03T17:23:06.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.215+0000 7f193a7fc700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0cac10 0x7f193c0c3df0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f1930000c00 tx=0x7f1930004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:06.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.216+0000 7f1923fff700 1 -- 172.21.15.7:0/1544759427 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930004400 con 0x7f193c0cac10 2023-03-03T17:23:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.216+0000 7f1923fff700 1 -- 172.21.15.7:0/1544759427 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193001f070 con 0x7f193c0cac10 2023-03-03T17:23:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.216+0000 7f1941a30700 1 -- 172.21.15.7:0/1544759427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c154eb0 con 0x7f193c0cac10 2023-03-03T17:23:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.216+0000 7f1923fff700 1 -- 172.21.15.7:0/1544759427 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19300038e0 con 0x7f193c0cac10 2023-03-03T17:23:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.216+0000 7f1941a30700 1 -- 172.21.15.7:0/1544759427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c154bd0 con 0x7f193c0cac10 2023-03-03T17:23:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.217+0000 7f1941a30700 1 -- 172.21.15.7:0/1544759427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c155190 con 0x7f193c0cac10 2023-03-03T17:23:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.217+0000 7f1923fff700 1 -- 172.21.15.7:0/1544759427 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f193002a020 con 0x7f193c0cac10 2023-03-03T17:23:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.217+0000 7f1923fff700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19280682f0 0x7f192806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.218+0000 7f1923fff700 1 -- 172.21.15.7:0/1544759427 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1930087720 con 0x7f193c0cac10 2023-03-03T17:23:06.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.218+0000 7f193affd700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19280682f0 0x7f192806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:06.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.218+0000 7f193affd700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19280682f0 0x7f192806a7a0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f192c009b40 tx=0x7f192c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:06.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.222+0000 7f1923fff700 1 -- 172.21.15.7:0/1544759427 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1930053a40 con 0x7f193c0cac10 2023-03-03T17:23:06.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.459+0000 7f1941a30700 1 -- 172.21.15.7:0/1544759427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f193c1eeea0 con 0x7f193c0cac10 2023-03-03T17:23:06.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.460+0000 7f1923fff700 1 -- 172.21.15.7:0/1544759427 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1930057060 con 0x7f193c0cac10 2023-03-03T17:23:06.460 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:23:06.461 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:23:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 -- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19280682f0 msgr2=0x7f192806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19280682f0 0x7f192806a7a0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f192c009b40 tx=0x7f192c009210 comp rx=0 tx=0).stop 2023-03-03T17:23:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 -- 172.21.15.7:0/1544759427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0cac10 msgr2=0x7f193c0c3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:06.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0cac10 0x7f193c0c3df0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f1930000c00 tx=0x7f1930004750 comp rx=0 tx=0).stop 2023-03-03T17:23:06.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 -- 172.21.15.7:0/1544759427 shutdown_connections 2023-03-03T17:23:06.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f193c151cc0 0x7f193c146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:06.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f193c0b00b0 0x7f193c0c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:06.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19280682f0 0x7f192806a7a0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:06.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 --2- 172.21.15.7:0/1544759427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193c0cac10 0x7f193c0c3df0 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:06.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 -- 172.21.15.7:0/1544759427 >> 172.21.15.7:0/1544759427 conn(0x7f193c06b980 msgr2=0x7f193c14e760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:23:06.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 -- 172.21.15.7:0/1544759427 shutdown_connections 2023-03-03T17:23:06.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:06.462+0000 7f1941a30700 1 -- 172.21.15.7:0/1544759427 wait complete. 2023-03-03T17:23:06.464 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:23:06.474 INFO:tasks.mds_thrash.fs.[a]:waiting for 23 secs before thrashing 2023-03-03T17:23:25.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:25.872 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:29.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:23:29.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:23:29.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.508+0000 7f73e2f4a700 1 Processor -- start 2023-03-03T17:23:29.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.508+0000 7f73e2f4a700 1 -- start start 2023-03-03T17:23:29.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.508+0000 7f73e2f4a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f73dc071990 0x7f73dc071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.508+0000 7f73e2f4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73dc14b460 0x7f73dc0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73e2f4a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73dc070360 con 0x7f73dc151c00 2023-03-03T17:23:29.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73e2f4a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73dc1521a0 con 0x7f73dc071990 2023-03-03T17:23:29.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73e2f4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc06a5e0 con 0x7f73dc14b460 2023-03-03T17:23:29.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73e14e7700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f73dc151c00 0x7f73dc06aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55076/0 (socket says 172.21.15.7:55076) 2023-03-03T17:23:29.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73e14e7700 1 -- 172.21.15.7:0/1587046547 learned_addr learned my addr 172.21.15.7:0/1587046547 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:23:29.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73e0ce6700 1 --2- 172.21.15.7:0/1587046547 >> v2:172.21.15.143:3300/0 conn(0x7f73dc071990 0x7f73dc071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73db7fe700 1 -- 172.21.15.7:0/1587046547 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291469940 0 0) 0x7f73dc070360 con 0x7f73dc151c00 2023-03-03T17:23:29.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73db7fe700 1 -- 172.21.15.7:0/1587046547 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73cc0035b0 con 0x7f73dc151c00 2023-03-03T17:23:29.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73dbfff700 1 --2- 172.21.15.7:0/1587046547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73dc14b460 0x7f73dc0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73e0ce6700 1 -- 172.21.15.7:0/1587046547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73dc14b460 msgr2=0x7f73dc0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:23:29.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73e0ce6700 1 --2- 172.21.15.7:0/1587046547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73dc14b460 0x7f73dc0722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73db7fe700 1 -- 172.21.15.7:0/1587046547 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106553912 0 0) 0x7f73cc0035b0 con 0x7f73dc151c00 2023-03-03T17:23:29.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73e0ce6700 1 -- 172.21.15.7:0/1587046547 >> v1:172.21.15.7:6789/0 conn(0x7f73dc151c00 legacy=0x7f73dc06aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.509+0000 7f73e0ce6700 1 -- 172.21.15.7:0/1587046547 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc0c38b0 con 0x7f73dc071990 2023-03-03T17:23:29.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.510+0000 7f73e0ce6700 1 --2- 172.21.15.7:0/1587046547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73dc071990 0x7f73dc071d60 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009840 tx=0x7f73d0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f8f306d452e66f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:29.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.510+0000 7f73db7fe700 1 -- 172.21.15.7:0/1587046547 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d40019f0 con 0x7f73dc071990 2023-03-03T17:23:29.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.510+0000 7f73db7fe700 1 -- 172.21.15.7:0/1587046547 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d0004030 con 0x7f73dc071990 2023-03-03T17:23:29.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.510+0000 7f73db7fe700 1 -- 172.21.15.7:0/1587046547 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d00043c0 con 0x7f73dc071990 2023-03-03T17:23:29.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.511+0000 7f73e2f4a700 1 -- 172.21.15.7:0/1587046547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73dc071990 msgr2=0x7f73dc071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.511+0000 7f73e2f4a700 1 --2- 172.21.15.7:0/1587046547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73dc071990 0x7f73dc071d60 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009840 tx=0x7f73d0009b50 comp rx=0 tx=0).stop 2023-03-03T17:23:29.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.511+0000 7f73e2f4a700 1 -- 172.21.15.7:0/1587046547 shutdown_connections 2023-03-03T17:23:29.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.511+0000 7f73e2f4a700 1 --2- 172.21.15.7:0/1587046547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73dc071990 0x7f73dc071d60 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.511+0000 7f73e2f4a700 1 --2- 172.21.15.7:0/1587046547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73dc14b460 0x7f73dc0722a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.511+0000 7f73e2f4a700 1 -- 172.21.15.7:0/1587046547 >> 172.21.15.7:0/1587046547 conn(0x7f73dc0a94d0 msgr2=0x7f73dc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:23:29.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.511+0000 7f73e2f4a700 1 -- 172.21.15.7:0/1587046547 shutdown_connections 2023-03-03T17:23:29.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.511+0000 7f73e2f4a700 1 -- 172.21.15.7:0/1587046547 wait complete. 2023-03-03T17:23:29.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.513+0000 7f73e2f4a700 1 Processor -- start 2023-03-03T17:23:29.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.513+0000 7f73e2f4a700 1 -- start start 2023-03-03T17:23:29.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.514+0000 7f73e2f4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73dc071990 0x7f73dc142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.514+0000 7f73e2f4a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73dc151c00 0x7f73dc1e8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.514+0000 7f73e2f4a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73dc150200 0x7f73dc1e8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.514+0000 7f73e2f4a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0aac40 con 0x7f73dc151c00 2023-03-03T17:23:29.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.514+0000 7f73e2f4a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc0ad420 con 0x7f73dc150200 2023-03-03T17:23:29.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.514+0000 7f73e2f4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc1ebda0 con 0x7f73dc071990 2023-03-03T17:23:29.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.514+0000 7f73dbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73dc151c00 0x7f73dc1e8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.514+0000 7f73dbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73dc151c00 0x7f73dc1e8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32868/0 (socket says 172.21.15.7:32868) 2023-03-03T17:23:29.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.514+0000 7f73dbfff700 1 -- 172.21.15.7:0/4237499130 learned_addr learned my addr 172.21.15.7:0/4237499130 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:23:29.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73e0ce6700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73dc071990 0x7f73dc142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73e14e7700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73dc150200 0x7f73dc1e8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73dbfff700 1 -- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73dc071990 msgr2=0x7f73dc142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73dbfff700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73dc071990 0x7f73dc142880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73dbfff700 1 -- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73dc150200 msgr2=0x7f73dc1e8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73dbfff700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73dc150200 0x7f73dc1e8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73dbfff700 1 -- 172.21.15.7:0/4237499130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d00094a0 con 0x7f73dc151c00 2023-03-03T17:23:29.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73dbfff700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73dc151c00 0x7f73dc1e8040 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f73c800e9b0 tx=0x7f73c800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:29.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73d97fa700 1 -- 172.21.15.7:0/4237499130 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8009710 con 0x7f73dc151c00 2023-03-03T17:23:29.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73d97fa700 1 -- 172.21.15.7:0/4237499130 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800ee50 con 0x7f73dc151c00 2023-03-03T17:23:29.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73e2f4a700 1 -- 172.21.15.7:0/4237499130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73dc0c3b50 con 0x7f73dc151c00 2023-03-03T17:23:29.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73d97fa700 1 -- 172.21.15.7:0/4237499130 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c800ca50 con 0x7f73dc151c00 2023-03-03T17:23:29.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.515+0000 7f73e2f4a700 1 -- 172.21.15.7:0/4237499130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73dc06a2b0 con 0x7f73dc151c00 2023-03-03T17:23:29.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.516+0000 7f73e2f4a700 1 -- 172.21.15.7:0/4237499130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73dc06b180 con 0x7f73dc151c00 2023-03-03T17:23:29.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.517+0000 7f73d97fa700 1 -- 172.21.15.7:0/4237499130 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f73c8019030 con 0x7f73dc151c00 2023-03-03T17:23:29.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.517+0000 7f73d97fa700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73c00682f0 0x7f73c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.517+0000 7f73d97fa700 1 -- 172.21.15.7:0/4237499130 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f73c8086050 con 0x7f73dc151c00 2023-03-03T17:23:29.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.518+0000 7f73e0ce6700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73c00682f0 0x7f73c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.518+0000 7f73e0ce6700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73c00682f0 0x7f73c006a7a0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f73d00090b0 tx=0x7f73d0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:29.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.522+0000 7f73d97fa700 1 -- 172.21.15.7:0/4237499130 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f73c8052390 con 0x7f73dc151c00 2023-03-03T17:23:29.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.608+0000 7f299bfc4700 1 Processor -- start 2023-03-03T17:23:29.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.608+0000 7f299bfc4700 1 -- start start 2023-03-03T17:23:29.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.609+0000 7f299bfc4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f299414f9d0 0x7f2994156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.609+0000 7f299bfc4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299414fda0 0x7f2994071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.609+0000 7f299bfc4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29940c2f30 con 0x7f29940abe20 2023-03-03T17:23:29.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.609+0000 7f299bfc4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29940c31d0 con 0x7f299414f9d0 2023-03-03T17:23:29.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.609+0000 7f299bfc4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29941524a0 con 0x7f299414fda0 2023-03-03T17:23:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.609+0000 7f2999d60700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f299414f9d0 0x7f2994156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.609+0000 7f2999d60700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f299414f9d0 0x7f2994156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32874/0 (socket says 172.21.15.7:32874) 2023-03-03T17:23:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.609+0000 7f2999d60700 1 -- 172.21.15.7:0/3339981920 learned_addr learned my addr 172.21.15.7:0/3339981920 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:23:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.609+0000 7f299955f700 1 --2- 172.21.15.7:0/3339981920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299414fda0 0x7f2994071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f2999d60700 1 -- 172.21.15.7:0/3339981920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299414fda0 msgr2=0x7f2994071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:23:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f2999d60700 1 --2- 172.21.15.7:0/3339981920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299414fda0 0x7f2994071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f2999d60700 1 -- 172.21.15.7:0/3339981920 >> v1:172.21.15.143:6789/0 conn(0x7f29940abe20 legacy=0x7f29941529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f2999d60700 1 -- 172.21.15.7:0/3339981920 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29940c34b0 con 0x7f299414f9d0 2023-03-03T17:23:29.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f2999d60700 1 --2- 172.21.15.7:0/3339981920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f299414f9d0 0x7f2994156430 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f29900098f0 tx=0x7f2990009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510cb0316bd59c8b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:29.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f2998d5e700 1 -- 172.21.15.7:0/3339981920 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299000b8a0 con 0x7f299414f9d0 2023-03-03T17:23:29.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f2998d5e700 1 -- 172.21.15.7:0/3339981920 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299000ba00 con 0x7f299414f9d0 2023-03-03T17:23:29.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f2998d5e700 1 -- 172.21.15.7:0/3339981920 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299000bd90 con 0x7f299414f9d0 2023-03-03T17:23:29.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f299bfc4700 1 -- 172.21.15.7:0/3339981920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f299414f9d0 msgr2=0x7f2994156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f299bfc4700 1 --2- 172.21.15.7:0/3339981920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f299414f9d0 0x7f2994156430 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f29900098f0 tx=0x7f2990009c00 comp rx=0 tx=0).stop 2023-03-03T17:23:29.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f299bfc4700 1 -- 172.21.15.7:0/3339981920 shutdown_connections 2023-03-03T17:23:29.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f299bfc4700 1 --2- 172.21.15.7:0/3339981920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f299414f9d0 0x7f2994156430 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f299bfc4700 1 --2- 172.21.15.7:0/3339981920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299414fda0 0x7f2994071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.610+0000 7f299bfc4700 1 -- 172.21.15.7:0/3339981920 >> 172.21.15.7:0/3339981920 conn(0x7f29940c8900 msgr2=0x7f29940c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:23:29.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.611+0000 7f299bfc4700 1 -- 172.21.15.7:0/3339981920 shutdown_connections 2023-03-03T17:23:29.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.611+0000 7f299bfc4700 1 -- 172.21.15.7:0/3339981920 wait complete. 2023-03-03T17:23:29.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f299bfc4700 1 Processor -- start 2023-03-03T17:23:29.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f299bfc4700 1 -- start start 2023-03-03T17:23:29.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f299bfc4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29940abe20 0x7f299414c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f299bfc4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299414f9d0 0x7f29940c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f299bfc4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f299414fda0 0x7f29941e7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f299bfc4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29940beff0 con 0x7f29940abe20 2023-03-03T17:23:29.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f299bfc4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29941eb840 con 0x7f299414fda0 2023-03-03T17:23:29.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f299bfc4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299414ca50 con 0x7f299414f9d0 2023-03-03T17:23:29.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f2999d60700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29940abe20 0x7f299414c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f2999d60700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29940abe20 0x7f299414c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32880/0 (socket says 172.21.15.7:32880) 2023-03-03T17:23:29.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.612+0000 7f2999d60700 1 -- 172.21.15.7:0/3168670434 learned_addr learned my addr 172.21.15.7:0/3168670434 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:23:29.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f299955f700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299414f9d0 0x7f29940c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f299a561700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f299414fda0 0x7f29941e7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f2999d60700 1 -- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299414f9d0 msgr2=0x7f29940c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f2999d60700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299414f9d0 0x7f29940c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f2999d60700 1 -- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f299414fda0 msgr2=0x7f29941e7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f2999d60700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f299414fda0 0x7f29941e7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f2999d60700 1 -- 172.21.15.7:0/3168670434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2990009550 con 0x7f29940abe20 2023-03-03T17:23:29.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f2999d60700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29940abe20 0x7f299414c510 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f2990000c00 tx=0x7f2990004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:29.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f298a7fc700 1 -- 172.21.15.7:0/3168670434 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2990004510 con 0x7f29940abe20 2023-03-03T17:23:29.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f298a7fc700 1 -- 172.21.15.7:0/3168670434 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29900036a0 con 0x7f29940abe20 2023-03-03T17:23:29.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f299bfc4700 1 -- 172.21.15.7:0/3168670434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29940ac350 con 0x7f29940abe20 2023-03-03T17:23:29.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f298a7fc700 1 -- 172.21.15.7:0/3168670434 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2990003a30 con 0x7f29940abe20 2023-03-03T17:23:29.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.613+0000 7f299bfc4700 1 -- 172.21.15.7:0/3168670434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29940ab630 con 0x7f29940abe20 2023-03-03T17:23:29.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.614+0000 7f299bfc4700 1 -- 172.21.15.7:0/3168670434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29940c3e10 con 0x7f29940abe20 2023-03-03T17:23:29.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.614+0000 7f298a7fc700 1 -- 172.21.15.7:0/3168670434 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f299000f050 con 0x7f29940abe20 2023-03-03T17:23:29.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.615+0000 7f298a7fc700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29780682f0 0x7f297806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:23:29.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.615+0000 7f298a7fc700 1 -- 172.21.15.7:0/3168670434 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2990087730 con 0x7f29940abe20 2023-03-03T17:23:29.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.615+0000 7f299955f700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29780682f0 0x7f297806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:23:29.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.615+0000 7f299955f700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29780682f0 0x7f297806a7a0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f29941e5460 tx=0x7f2984008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:23:29.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.619+0000 7f298a7fc700 1 -- 172.21.15.7:0/3168670434 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2990053a50 con 0x7f29940abe20 2023-03-03T17:23:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.763+0000 7f73e2f4a700 1 -- 172.21.15.7:0/4237499130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73dc1ebf10 con 0x7f73dc151c00 2023-03-03T17:23:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.764+0000 7f73d97fa700 1 -- 172.21.15.7:0/4237499130 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f73c8017090 con 0x7f73dc151c00 2023-03-03T17:23:29.763 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:23:29.764 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:23:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 -- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73c00682f0 msgr2=0x7f73c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73c00682f0 0x7f73c006a7a0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f73d00090b0 tx=0x7f73d0008030 comp rx=0 tx=0).stop 2023-03-03T17:23:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 -- 172.21.15.7:0/4237499130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73dc151c00 msgr2=0x7f73dc1e8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73dc151c00 0x7f73dc1e8040 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f73c800e9b0 tx=0x7f73c800ecc0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 -- 172.21.15.7:0/4237499130 shutdown_connections 2023-03-03T17:23:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73dc150200 0x7f73dc1e8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f73dc071990 0x7f73dc142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f73c00682f0 0x7f73c006a7a0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 --2- 172.21.15.7:0/4237499130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f73dc151c00 0x7f73dc1e8040 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 -- 172.21.15.7:0/4237499130 >> 172.21.15.7:0/4237499130 conn(0x7f73dc0a94d0 msgr2=0x7f73dc06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:23:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 -- 172.21.15.7:0/4237499130 shutdown_connections 2023-03-03T17:23:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.766+0000 7f73e2f4a700 1 -- 172.21.15.7:0/4237499130 wait complete. 2023-03-03T17:23:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:23:29.777 INFO:tasks.mds_thrash.fs.[b]:waiting for 69 secs before thrashing 2023-03-03T17:23:29.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.857+0000 7f299bfc4700 1 -- 172.21.15.7:0/3168670434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2994074bb0 con 0x7f29940abe20 2023-03-03T17:23:29.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.858+0000 7f298a7fc700 1 -- 172.21.15.7:0/3168670434 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2990057070 con 0x7f29940abe20 2023-03-03T17:23:29.858 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:23:29.858 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:23:29.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 -- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29780682f0 msgr2=0x7f297806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29780682f0 0x7f297806a7a0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f29941e5460 tx=0x7f2984008040 comp rx=0 tx=0).stop 2023-03-03T17:23:29.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 -- 172.21.15.7:0/3168670434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29940abe20 msgr2=0x7f299414c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:23:29.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29940abe20 0x7f299414c510 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7f2990000c00 tx=0x7f2990004750 comp rx=0 tx=0).stop 2023-03-03T17:23:29.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 -- 172.21.15.7:0/3168670434 shutdown_connections 2023-03-03T17:23:29.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f299414fda0 0x7f29941e7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f29780682f0 0x7f297806a7a0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29940abe20 0x7f299414c510 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 --2- 172.21.15.7:0/3168670434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f299414f9d0 0x7f29940c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:23:29.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 -- 172.21.15.7:0/3168670434 >> 172.21.15.7:0/3168670434 conn(0x7f29940c8900 msgr2=0x7f29940c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:23:29.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 -- 172.21.15.7:0/3168670434 shutdown_connections 2023-03-03T17:23:29.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:23:29.860+0000 7f299bfc4700 1 -- 172.21.15.7:0/3168670434 wait complete. 2023-03-03T17:23:29.861 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:23:29.871 INFO:tasks.mds_thrash.fs.[a]:waiting for 96 secs before thrashing 2023-03-03T17:23:55.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:23:55.944 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:26.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:26.016 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:38.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:24:38.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.918+0000 7f68f4ad0700 1 Processor -- start 2023-03-03T17:24:38.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.918+0000 7f68f4ad0700 1 -- start start 2023-03-03T17:24:38.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68f4ad0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68f014fc50 0x7f68f00c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:38.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68f4ad0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f68f006d240 0x7f68f00c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:38.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68f4ad0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f00ad2b0 con 0x7f68f00c3e30 2023-03-03T17:24:38.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68f4ad0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f00c30c0 con 0x7f68f014fc50 2023-03-03T17:24:38.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68f4ad0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f0156420 con 0x7f68f006d240 2023-03-03T17:24:38.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68ee59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68f014fc50 0x7f68f00c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:38.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68ee59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68f014fc50 0x7f68f00c1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32916/0 (socket says 172.21.15.7:32916) 2023-03-03T17:24:38.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68ee59c700 1 -- 172.21.15.7:0/3743421049 learned_addr learned my addr 172.21.15.7:0/3743421049 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:24:38.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68edd9b700 1 --2- 172.21.15.7:0/3743421049 >> v2:172.21.15.143:3300/0 conn(0x7f68f006d240 0x7f68f00c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:38.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68ee59c700 1 -- 172.21.15.7:0/3743421049 >> v2:172.21.15.143:3300/0 conn(0x7f68f006d240 msgr2=0x7f68f00c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:24:38.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68ee59c700 1 --2- 172.21.15.7:0/3743421049 >> v2:172.21.15.143:3300/0 conn(0x7f68f006d240 0x7f68f00c27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:38.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68ee59c700 1 -- 172.21.15.7:0/3743421049 >> v1:172.21.15.143:6789/0 conn(0x7f68f00c3e30 legacy=0x7f68f00c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:38.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68ee59c700 1 -- 172.21.15.7:0/3743421049 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f00ad520 con 0x7f68f014fc50 2023-03-03T17:24:38.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.919+0000 7f68ee59c700 1 --2- 172.21.15.7:0/3743421049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 0x7f68f00c1f50 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f68f00acf10 tx=0x7f68e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbd7686026da386 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:24:38.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68ed59a700 1 -- 172.21.15.7:0/3743421049 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e000b870 con 0x7f68f014fc50 2023-03-03T17:24:38.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68ed59a700 1 -- 172.21.15.7:0/3743421049 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e000b9d0 con 0x7f68f014fc50 2023-03-03T17:24:38.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68ed59a700 1 -- 172.21.15.7:0/3743421049 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e000bd60 con 0x7f68f014fc50 2023-03-03T17:24:38.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68f4ad0700 1 -- 172.21.15.7:0/3743421049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 msgr2=0x7f68f00c1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:38.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68f4ad0700 1 --2- 172.21.15.7:0/3743421049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 0x7f68f00c1f50 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f68f00acf10 tx=0x7f68e0009bd0 comp rx=0 tx=0).stop 2023-03-03T17:24:38.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68f4ad0700 1 -- 172.21.15.7:0/3743421049 shutdown_connections 2023-03-03T17:24:38.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68f4ad0700 1 --2- 172.21.15.7:0/3743421049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 0x7f68f00c1f50 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:38.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68f4ad0700 1 --2- 172.21.15.7:0/3743421049 >> v2:172.21.15.143:3300/0 conn(0x7f68f006d240 0x7f68f00c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:38.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68f4ad0700 1 -- 172.21.15.7:0/3743421049 >> 172.21.15.7:0/3743421049 conn(0x7f68f006c110 msgr2=0x7f68f006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:24:38.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68f4ad0700 1 -- 172.21.15.7:0/3743421049 shutdown_connections 2023-03-03T17:24:38.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.920+0000 7f68f4ad0700 1 -- 172.21.15.7:0/3743421049 wait complete. 2023-03-03T17:24:38.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.922+0000 7f68f4ad0700 1 Processor -- start 2023-03-03T17:24:38.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.922+0000 7f68f4ad0700 1 -- start start 2023-03-03T17:24:38.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68f4ad0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f006d240 0x7f68f00c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:38.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68f4ad0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f00c3e30 0x7f68f0146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:38.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68f4ad0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 0x7f68f00aaec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:38.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68f4ad0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f01de190 con 0x7f68f014fc50 2023-03-03T17:24:38.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68f4ad0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f00c81a0 con 0x7f68f006d240 2023-03-03T17:24:38.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68f4ad0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f01dd650 con 0x7f68f00c3e30 2023-03-03T17:24:38.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68eed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 0x7f68f00aaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:38.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68eed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 0x7f68f00aaec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32926/0 (socket says 172.21.15.7:32926) 2023-03-03T17:24:38.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68eed9d700 1 -- 172.21.15.7:0/1758559287 learned_addr learned my addr 172.21.15.7:0/1758559287 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:24:38.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68ee59c700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f006d240 0x7f68f00c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:38.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68edd9b700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f00c3e30 0x7f68f0146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:38.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.923+0000 7f68eed9d700 1 -- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f00c3e30 msgr2=0x7f68f0146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:38.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.924+0000 7f68eed9d700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f00c3e30 0x7f68f0146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.924+0000 7f68eed9d700 1 -- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f006d240 msgr2=0x7f68f00c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.924+0000 7f68eed9d700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f006d240 0x7f68f00c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.924+0000 7f68eed9d700 1 -- 172.21.15.7:0/1758559287 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e0009550 con 0x7f68f014fc50 2023-03-03T17:24:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.924+0000 7f68eed9d700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 0x7f68f00aaec0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f68e800cb50 tx=0x7f68e800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:24:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.924+0000 7f68deffd700 1 -- 172.21.15.7:0/1758559287 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e8003c00 con 0x7f68f014fc50 2023-03-03T17:24:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.924+0000 7f68deffd700 1 -- 172.21.15.7:0/1758559287 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e8003d60 con 0x7f68f014fc50 2023-03-03T17:24:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.924+0000 7f68f4ad0700 1 -- 172.21.15.7:0/1758559287 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f00c2cf0 con 0x7f68f014fc50 2023-03-03T17:24:38.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.924+0000 7f68deffd700 1 -- 172.21.15.7:0/1758559287 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e800aa20 con 0x7f68f014fc50 2023-03-03T17:24:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.924+0000 7f68f4ad0700 1 -- 172.21.15.7:0/1758559287 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f00c2490 con 0x7f68f014fc50 2023-03-03T17:24:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.925+0000 7f68f4ad0700 1 -- 172.21.15.7:0/1758559287 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f0074220 con 0x7f68f014fc50 2023-03-03T17:24:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.925+0000 7f68deffd700 1 -- 172.21.15.7:0/1758559287 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f68e8010050 con 0x7f68f014fc50 2023-03-03T17:24:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.926+0000 7f68deffd700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68d00682f0 0x7f68d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:24:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.926+0000 7f68deffd700 1 -- 172.21.15.7:0/1758559287 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f68e8085fe0 con 0x7f68f014fc50 2023-03-03T17:24:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.926+0000 7f68ee59c700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68d00682f0 0x7f68d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:24:38.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.927+0000 7f68ee59c700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68d00682f0 0x7f68d006a7a0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f68e0000c00 tx=0x7f68e0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:24:38.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:38.930+0000 7f68deffd700 1 -- 172.21.15.7:0/1758559287 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68e8052380 con 0x7f68f014fc50 2023-03-03T17:24:39.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.167+0000 7f68f4ad0700 1 -- 172.21.15.7:0/1758559287 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68f0074bb0 con 0x7f68f014fc50 2023-03-03T17:24:39.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.167+0000 7f68deffd700 1 -- 172.21.15.7:0/1758559287 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f68e80559a0 con 0x7f68f014fc50 2023-03-03T17:24:39.167 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:24:39.167 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:24:39.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 -- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68d00682f0 msgr2=0x7f68d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:39.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68d00682f0 0x7f68d006a7a0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f68e0000c00 tx=0x7f68e0008030 comp rx=0 tx=0).stop 2023-03-03T17:24:39.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 -- 172.21.15.7:0/1758559287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 msgr2=0x7f68f00aaec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:24:39.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 0x7f68f00aaec0 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f68e800cb50 tx=0x7f68e800ce60 comp rx=0 tx=0).stop 2023-03-03T17:24:39.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 -- 172.21.15.7:0/1758559287 shutdown_connections 2023-03-03T17:24:39.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68d00682f0 0x7f68d006a7a0 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:39.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68f014fc50 0x7f68f00aaec0 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:39.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68f006d240 0x7f68f00c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:39.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 --2- 172.21.15.7:0/1758559287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68f00c3e30 0x7f68f0146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:24:39.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 -- 172.21.15.7:0/1758559287 >> 172.21.15.7:0/1758559287 conn(0x7f68f006c110 msgr2=0x7f68f006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:24:39.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 -- 172.21.15.7:0/1758559287 shutdown_connections 2023-03-03T17:24:39.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:24:39.169+0000 7f68f4ad0700 1 -- 172.21.15.7:0/1758559287 wait complete. 2023-03-03T17:24:39.171 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:24:39.181 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-03-03T17:24:56.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:24:56.088 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:05.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:25:06.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.012+0000 7f3010eab700 1 Processor -- start 2023-03-03T17:25:06.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.012+0000 7f3010eab700 1 -- start start 2023-03-03T17:25:06.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.013+0000 7f3010eab700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f300c151d40 0x7f300c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.013+0000 7f3010eab700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f300c0c2930 0x7f300c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.013+0000 7f3010eab700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c06d430 con 0x7f300c073c30 2023-03-03T17:25:06.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.013+0000 7f3010eab700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300c06c2a0 con 0x7f300c151d40 2023-03-03T17:25:06.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.013+0000 7f3010eab700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300c1586e0 con 0x7f300c0c2930 2023-03-03T17:25:06.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.013+0000 7f300a59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f300c151d40 0x7f300c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300a59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f300c151d40 0x7f300c0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32954/0 (socket says 172.21.15.7:32954) 2023-03-03T17:25:06.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300ad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f300c073c30 0x7f300c157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55166/0 (socket says 172.21.15.7:55166) 2023-03-03T17:25:06.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300a59c700 1 -- 172.21.15.7:0/3134728541 learned_addr learned my addr 172.21.15.7:0/3134728541 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:25:06.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f3009d9b700 1 --2- 172.21.15.7:0/3134728541 >> v2:172.21.15.143:3300/0 conn(0x7f300c0c2930 0x7f300c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300a59c700 1 -- 172.21.15.7:0/3134728541 >> v2:172.21.15.143:3300/0 conn(0x7f300c0c2930 msgr2=0x7f300c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:25:06.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300a59c700 1 --2- 172.21.15.7:0/3134728541 >> v2:172.21.15.143:3300/0 conn(0x7f300c0c2930 0x7f300c157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300a59c700 1 -- 172.21.15.7:0/3134728541 >> v1:172.21.15.7:6789/0 conn(0x7f300c073c30 legacy=0x7f300c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300959a700 1 -- 172.21.15.7:0/3134728541 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100310213 0 0) 0x7f300c06d430 con 0x7f300c073c30 2023-03-03T17:25:06.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300a59c700 1 -- 172.21.15.7:0/3134728541 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c06cb00 con 0x7f300c151d40 2023-03-03T17:25:06.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300a59c700 1 --2- 172.21.15.7:0/3134728541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 0x7f300c0736f0 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0098f0 tx=0x7f2ffc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d522600a345f01 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:25:06.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300959a700 1 -- 172.21.15.7:0/3134728541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc00b860 con 0x7f300c151d40 2023-03-03T17:25:06.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300959a700 1 -- 172.21.15.7:0/3134728541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ffc00b9c0 con 0x7f300c151d40 2023-03-03T17:25:06.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.014+0000 7f300959a700 1 -- 172.21.15.7:0/3134728541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc00bd50 con 0x7f300c151d40 2023-03-03T17:25:06.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.015+0000 7f3010eab700 1 -- 172.21.15.7:0/3134728541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 msgr2=0x7f300c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.015+0000 7f3010eab700 1 --2- 172.21.15.7:0/3134728541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 0x7f300c0736f0 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0098f0 tx=0x7f2ffc009c00 comp rx=0 tx=0).stop 2023-03-03T17:25:06.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.015+0000 7f3010eab700 1 -- 172.21.15.7:0/3134728541 shutdown_connections 2023-03-03T17:25:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.015+0000 7f3010eab700 1 --2- 172.21.15.7:0/3134728541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 0x7f300c0736f0 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.015+0000 7f3010eab700 1 --2- 172.21.15.7:0/3134728541 >> v2:172.21.15.143:3300/0 conn(0x7f300c0c2930 0x7f300c157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.015+0000 7f3010eab700 1 -- 172.21.15.7:0/3134728541 >> 172.21.15.7:0/3134728541 conn(0x7f300c0a94d0 msgr2=0x7f300c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:25:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.015+0000 7f3010eab700 1 -- 172.21.15.7:0/3134728541 shutdown_connections 2023-03-03T17:25:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.015+0000 7f3010eab700 1 -- 172.21.15.7:0/3134728541 wait complete. 2023-03-03T17:25:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.016+0000 7f3010eab700 1 Processor -- start 2023-03-03T17:25:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.016+0000 7f3010eab700 1 -- start start 2023-03-03T17:25:06.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f3010eab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f300c073c30 0x7f300c148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f3010eab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300c0c2930 0x7f300c14e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f3010eab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 0x7f300c1ea770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f3010eab700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c0be430 con 0x7f300c151d40 2023-03-03T17:25:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f3010eab700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c148cc0 con 0x7f300c073c30 2023-03-03T17:25:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f3010eab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300c1534a0 con 0x7f300c0c2930 2023-03-03T17:25:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f300ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 0x7f300c1ea770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f300ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 0x7f300c1ea770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:32962/0 (socket says 172.21.15.7:32962) 2023-03-03T17:25:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f300ad9d700 1 -- 172.21.15.7:0/2354241606 learned_addr learned my addr 172.21.15.7:0/2354241606 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:25:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f300a59c700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f300c073c30 0x7f300c148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.017+0000 7f3009d9b700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300c0c2930 0x7f300c14e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f300ad9d700 1 -- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300c0c2930 msgr2=0x7f300c14e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f300ad9d700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300c0c2930 0x7f300c14e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f300ad9d700 1 -- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f300c073c30 msgr2=0x7f300c148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f300ad9d700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f300c073c30 0x7f300c148780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f300ad9d700 1 -- 172.21.15.7:0/2354241606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc009550 con 0x7f300c151d40 2023-03-03T17:25:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f300ad9d700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 0x7f300c1ea770 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f300400cc10 tx=0x7f300400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:25:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f3002ffd700 1 -- 172.21.15.7:0/2354241606 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300400ab50 con 0x7f300c151d40 2023-03-03T17:25:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f3002ffd700 1 -- 172.21.15.7:0/2354241606 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300400acd0 con 0x7f300c151d40 2023-03-03T17:25:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f3010eab700 1 -- 172.21.15.7:0/2354241606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c06d0c0 con 0x7f300c151d40 2023-03-03T17:25:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f3002ffd700 1 -- 172.21.15.7:0/2354241606 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3004007310 con 0x7f300c151d40 2023-03-03T17:25:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.018+0000 7f3010eab700 1 -- 172.21.15.7:0/2354241606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c06cde0 con 0x7f300c151d40 2023-03-03T17:25:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.019+0000 7f3010eab700 1 -- 172.21.15.7:0/2354241606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300c06bea0 con 0x7f300c151d40 2023-03-03T17:25:06.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.020+0000 7f3002ffd700 1 -- 172.21.15.7:0/2354241606 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f300401a030 con 0x7f300c151d40 2023-03-03T17:25:06.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.020+0000 7f3002ffd700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fec0682f0 0x7f2fec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:25:06.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.020+0000 7f3002ffd700 1 -- 172.21.15.7:0/2354241606 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3004085d60 con 0x7f300c151d40 2023-03-03T17:25:06.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.020+0000 7f300a59c700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fec0682f0 0x7f2fec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:25:06.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.021+0000 7f300a59c700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fec0682f0 0x7f2fec06a7a0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc009c40 tx=0x7f2ffc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:25:06.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.024+0000 7f3002ffd700 1 -- 172.21.15.7:0/2354241606 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3004052100 con 0x7f300c151d40 2023-03-03T17:25:06.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.267+0000 7f3010eab700 1 -- 172.21.15.7:0/2354241606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f300c1ea3f0 con 0x7f300c151d40 2023-03-03T17:25:06.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.267+0000 7f3002ffd700 1 -- 172.21.15.7:0/2354241606 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3004018090 con 0x7f300c151d40 2023-03-03T17:25:06.267 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:25:06.267 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:25:06.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 -- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fec0682f0 msgr2=0x7f2fec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fec0682f0 0x7f2fec06a7a0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc009c40 tx=0x7f2ffc008030 comp rx=0 tx=0).stop 2023-03-03T17:25:06.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 -- 172.21.15.7:0/2354241606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 msgr2=0x7f300c1ea770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:25:06.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 0x7f300c1ea770 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7f300400cc10 tx=0x7f300400cf20 comp rx=0 tx=0).stop 2023-03-03T17:25:06.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 -- 172.21.15.7:0/2354241606 shutdown_connections 2023-03-03T17:25:06.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fec0682f0 0x7f2fec06a7a0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f300c151d40 0x7f300c1ea770 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f300c073c30 0x7f300c148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 --2- 172.21.15.7:0/2354241606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f300c0c2930 0x7f300c14e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:25:06.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 -- 172.21.15.7:0/2354241606 >> 172.21.15.7:0/2354241606 conn(0x7f300c0a94d0 msgr2=0x7f300c150000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:25:06.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 -- 172.21.15.7:0/2354241606 shutdown_connections 2023-03-03T17:25:06.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:25:06.269+0000 7f3010eab700 1 -- 172.21.15.7:0/2354241606 wait complete. 2023-03-03T17:25:06.271 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:25:06.281 INFO:tasks.mds_thrash.fs.[a]:waiting for 81 secs before thrashing 2023-03-03T17:25:26.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:26.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:56.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:25:56.299 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:01.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:26:01.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.323+0000 7fd3ad069700 1 Processor -- start 2023-03-03T17:26:01.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.323+0000 7fd3ad069700 1 -- start start 2023-03-03T17:26:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3ad069700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8071990 0x7fd3a8071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3ad069700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd3a814b460 0x7fd3a80722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3ad069700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a8070360 con 0x7fd3a8151c00 2023-03-03T17:26:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3ad069700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a81521a0 con 0x7fd3a814b460 2023-03-03T17:26:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3ad069700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a806a5e0 con 0x7fd3a8071990 2023-03-03T17:26:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3a759e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd3a8151c00 0x7fd3a806aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55208/0 (socket says 172.21.15.7:55208) 2023-03-03T17:26:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3a759e700 1 -- 172.21.15.7:0/974788481 learned_addr learned my addr 172.21.15.7:0/974788481 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:26:01.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616604340 0 0) 0x7fd3a8070360 con 0x7fd3a8151c00 2023-03-03T17:26:01.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3a6d9d700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8071990 0x7fd3a8071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3900035b0 con 0x7fd3a8151c00 2023-03-03T17:26:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.324+0000 7fd3a659c700 1 --2- 172.21.15.7:0/974788481 >> v2:172.21.15.143:3300/0 conn(0x7fd3a814b460 0x7fd3a80722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2300345987 0 0) 0x7fd3900035b0 con 0x7fd3a8151c00 2023-03-03T17:26:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8071990 msgr2=0x7fd3a8071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:26:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8071990 0x7fd3a8071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 >> v2:172.21.15.143:3300/0 conn(0x7fd3a814b460 msgr2=0x7fd3a80722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:26:01.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 --2- 172.21.15.7:0/974788481 >> v2:172.21.15.143:3300/0 conn(0x7fd3a814b460 0x7fd3a80722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80c38b0 con 0x7fd3a8151c00 2023-03-03T17:26:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd3a00019f0 con 0x7fd3a8151c00 2023-03-03T17:26:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 >> v1:172.21.15.7:6789/0 conn(0x7fd3a8151c00 legacy=0x7fd3a806aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8071990 0x7fd390003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd390003ff0 0x7fd390004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd390004d90 0x7fd390005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:01.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a81521a0 con 0x7fd390004d90 2023-03-03T17:26:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a806a5e0 con 0x7fd390003ff0 2023-03-03T17:26:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39000bb70 con 0x7fd3a8071990 2023-03-03T17:26:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a759e700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd390004d90 0x7fd390005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a6d9d700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8071990 0x7fd390003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a659c700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd390003ff0 0x7fd390004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a759e700 1 -- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8071990 msgr2=0x7fd390003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:26:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a759e700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8071990 0x7fd390003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a759e700 1 -- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd390003ff0 msgr2=0x7fd390004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:26:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a759e700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd390003ff0 0x7fd390004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a759e700 1 -- 172.21.15.7:0/974788481 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3900035b0 con 0x7fd390004d90 2023-03-03T17:26:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a759e700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd390004d90 0x7fd390005200 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00013d0 tx=0x7fd390005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c231045a3768a7b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.325+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a000a340 con 0x7fd390004d90 2023-03-03T17:26:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.326+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a0012600 con 0x7fd390004d90 2023-03-03T17:26:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.326+0000 7fd3a5d9b700 1 -- 172.21.15.7:0/974788481 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00129b0 con 0x7fd390004d90 2023-03-03T17:26:01.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.326+0000 7fd3ad069700 1 -- 172.21.15.7:0/974788481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd390004d90 msgr2=0x7fd390005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.326+0000 7fd3ad069700 1 --2- 172.21.15.7:0/974788481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd390004d90 0x7fd390005200 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00013d0 tx=0x7fd390005880 comp rx=0 tx=0).stop 2023-03-03T17:26:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.326+0000 7fd3a759e700 1 -- 172.21.15.7:0/974788481 reap_dead start 2023-03-03T17:26:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.326+0000 7fd3ad069700 1 -- 172.21.15.7:0/974788481 shutdown_connections 2023-03-03T17:26:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.326+0000 7fd3ad069700 1 -- 172.21.15.7:0/974788481 >> 172.21.15.7:0/974788481 conn(0x7fd3a80a94d0 msgr2=0x7fd3a80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:26:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.326+0000 7fd3ad069700 1 -- 172.21.15.7:0/974788481 shutdown_connections 2023-03-03T17:26:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.326+0000 7fd3ad069700 1 -- 172.21.15.7:0/974788481 wait complete. 2023-03-03T17:26:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3ad069700 1 Processor -- start 2023-03-03T17:26:01.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3ad069700 1 -- start start 2023-03-03T17:26:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3ad069700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a81df3b0 0x7fd3a80c0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3ad069700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a81df780 0x7fd3a8147600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3ad069700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8147b40 0x7fd3a80aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3ad069700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80ac740 con 0x7fd3a81df780 2023-03-03T17:26:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3ad069700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80aa480 con 0x7fd3a81df3b0 2023-03-03T17:26:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3ad069700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80aa760 con 0x7fd3a8147b40 2023-03-03T17:26:01.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3a659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a81df780 0x7fd3a8147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3a659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a81df780 0x7fd3a8147600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33004/0 (socket says 172.21.15.7:33004) 2023-03-03T17:26:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.328+0000 7fd3a659c700 1 -- 172.21.15.7:0/1984129285 learned_addr learned my addr 172.21.15.7:0/1984129285 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:26:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd3a6d9d700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a81df3b0 0x7fd3a80c0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd3a759e700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8147b40 0x7fd3a80aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd3a659c700 1 -- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8147b40 msgr2=0x7fd3a80aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd3a659c700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8147b40 0x7fd3a80aabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd3a659c700 1 -- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a81df3b0 msgr2=0x7fd3a80c0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd3a659c700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a81df3b0 0x7fd3a80c0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd3a659c700 1 -- 172.21.15.7:0/1984129285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80c3b50 con 0x7fd3a81df780 2023-03-03T17:26:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd3a659c700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a81df780 0x7fd3a8147600 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7fd398011aa0 tx=0x7fd398011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd38f7fe700 1 -- 172.21.15.7:0/1984129285 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398011ed0 con 0x7fd3a81df780 2023-03-03T17:26:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd38f7fe700 1 -- 172.21.15.7:0/1984129285 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39800f940 con 0x7fd3a81df780 2023-03-03T17:26:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd3ad069700 1 -- 172.21.15.7:0/1984129285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a806a2b0 con 0x7fd3a81df780 2023-03-03T17:26:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd38f7fe700 1 -- 172.21.15.7:0/1984129285 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39800fd10 con 0x7fd3a81df780 2023-03-03T17:26:01.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.329+0000 7fd3ad069700 1 -- 172.21.15.7:0/1984129285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a8071d60 con 0x7fd3a81df780 2023-03-03T17:26:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.330+0000 7fd3ad069700 1 -- 172.21.15.7:0/1984129285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a8074bb0 con 0x7fd3a81df780 2023-03-03T17:26:01.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.331+0000 7fd38f7fe700 1 -- 172.21.15.7:0/1984129285 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd398014440 con 0x7fd3a81df780 2023-03-03T17:26:01.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.331+0000 7fd38f7fe700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3940682f0 0x7fd39406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:01.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.331+0000 7fd38f7fe700 1 -- 172.21.15.7:0/1984129285 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd3980868c0 con 0x7fd3a81df780 2023-03-03T17:26:01.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.331+0000 7fd3a6d9d700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3940682f0 0x7fd39406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:01.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.332+0000 7fd3a6d9d700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3940682f0 0x7fd39406a7a0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009c60 tx=0x7fd39c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:01.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.335+0000 7fd38f7fe700 1 -- 172.21.15.7:0/1984129285 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd398052d70 con 0x7fd3a81df780 2023-03-03T17:26:01.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.571+0000 7fd3ad069700 1 -- 172.21.15.7:0/1984129285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3a80c32b0 con 0x7fd3a81df780 2023-03-03T17:26:01.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.572+0000 7fd38f7fe700 1 -- 172.21.15.7:0/1984129285 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd398056390 con 0x7fd3a81df780 2023-03-03T17:26:01.572 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:26:01.572 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:26:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 -- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3940682f0 msgr2=0x7fd39406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:01.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3940682f0 0x7fd39406a7a0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009c60 tx=0x7fd39c009210 comp rx=0 tx=0).stop 2023-03-03T17:26:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 -- 172.21.15.7:0/1984129285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a81df780 msgr2=0x7fd3a8147600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a81df780 0x7fd3a8147600 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7fd398011aa0 tx=0x7fd398011db0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 -- 172.21.15.7:0/1984129285 shutdown_connections 2023-03-03T17:26:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3a8147b40 0x7fd3a80aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3a81df3b0 0x7fd3a80c0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3940682f0 0x7fd39406a7a0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 --2- 172.21.15.7:0/1984129285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3a81df780 0x7fd3a8147600 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:01.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 -- 172.21.15.7:0/1984129285 >> 172.21.15.7:0/1984129285 conn(0x7fd3a80a94d0 msgr2=0x7fd3a8151300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:26:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 -- 172.21.15.7:0/1984129285 shutdown_connections 2023-03-03T17:26:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:01.574+0000 7fd3ad069700 1 -- 172.21.15.7:0/1984129285 wait complete. 2023-03-03T17:26:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:26:01.585 INFO:tasks.mds_thrash.fs.[b]:waiting for 55 secs before thrashing 2023-03-03T17:26:26.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:26.355 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:27.282 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:26:27.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.420+0000 7f3b9aa93700 1 Processor -- start 2023-03-03T17:26:27.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.420+0000 7f3b9aa93700 1 -- start start 2023-03-03T17:26:27.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b9aa93700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b94072250 0x7f3b9414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:27.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b9aa93700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3b940c1ce0 0x7f3b9414e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:27.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b9aa93700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b941510d0 con 0x7f3b94071e80 2023-03-03T17:26:27.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b9aa93700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b940c3740 con 0x7f3b940c1ce0 2023-03-03T17:26:27.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b9aa93700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b94070670 con 0x7f3b94072250 2023-03-03T17:26:27.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b99030700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3b940c1ce0 0x7f3b9414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:27.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b99030700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3b940c1ce0 0x7f3b9414e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33020/0 (socket says 172.21.15.7:33020) 2023-03-03T17:26:27.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b99030700 1 -- 172.21.15.7:0/1377593317 learned_addr learned my addr 172.21.15.7:0/1377593317 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:26:27.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b93fff700 1 --2- 172.21.15.7:0/1377593317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b94072250 0x7f3b9414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:27.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b99030700 1 -- 172.21.15.7:0/1377593317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b94072250 msgr2=0x7f3b9414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:26:27.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.421+0000 7f3b99030700 1 --2- 172.21.15.7:0/1377593317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b94072250 0x7f3b9414e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:27.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.422+0000 7f3b99030700 1 -- 172.21.15.7:0/1377593317 >> v1:172.21.15.143:6789/0 conn(0x7f3b94071e80 legacy=0x7f3b94151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:26:27.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.422+0000 7f3b99030700 1 -- 172.21.15.7:0/1377593317 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b940c39e0 con 0x7f3b940c1ce0 2023-03-03T17:26:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.422+0000 7f3b99030700 1 --2- 172.21.15.7:0/1377593317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b940c1ce0 0x7f3b9414e730 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009e50 tx=0x7f3b8c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a492f81547712b12 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.422+0000 7f3b937fe700 1 -- 172.21.15.7:0/1377593317 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c00e070 con 0x7f3b940c1ce0 2023-03-03T17:26:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.422+0000 7f3b937fe700 1 -- 172.21.15.7:0/1377593317 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8c00b9c0 con 0x7f3b940c1ce0 2023-03-03T17:26:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.422+0000 7f3b937fe700 1 -- 172.21.15.7:0/1377593317 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c00bd70 con 0x7f3b940c1ce0 2023-03-03T17:26:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.422+0000 7f3b9aa93700 1 -- 172.21.15.7:0/1377593317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b940c1ce0 msgr2=0x7f3b9414e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.422+0000 7f3b9aa93700 1 --2- 172.21.15.7:0/1377593317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b940c1ce0 0x7f3b9414e730 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009e50 tx=0x7f3b8c00b330 comp rx=0 tx=0).stop 2023-03-03T17:26:27.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.422+0000 7f3b9aa93700 1 -- 172.21.15.7:0/1377593317 shutdown_connections 2023-03-03T17:26:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.422+0000 7f3b9aa93700 1 --2- 172.21.15.7:0/1377593317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b940c1ce0 0x7f3b9414e730 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.423+0000 7f3b9aa93700 1 --2- 172.21.15.7:0/1377593317 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b94072250 0x7f3b9414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.423+0000 7f3b9aa93700 1 -- 172.21.15.7:0/1377593317 >> 172.21.15.7:0/1377593317 conn(0x7f3b940a94d0 msgr2=0x7f3b940a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:26:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.423+0000 7f3b9aa93700 1 -- 172.21.15.7:0/1377593317 shutdown_connections 2023-03-03T17:26:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.423+0000 7f3b9aa93700 1 -- 172.21.15.7:0/1377593317 wait complete. 2023-03-03T17:26:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.424+0000 7f3b9aa93700 1 Processor -- start 2023-03-03T17:26:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.424+0000 7f3b9aa93700 1 -- start start 2023-03-03T17:26:27.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.424+0000 7f3b9aa93700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b94071e80 0x7f3b941e9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b9aa93700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b94072250 0x7f3b940c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b9aa93700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b940c1ce0 0x7f3b941e5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b9aa93700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941e97e0 con 0x7f3b94072250 2023-03-03T17:26:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b9aa93700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941e9180 con 0x7f3b940c1ce0 2023-03-03T17:26:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b9aa93700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940c5620 con 0x7f3b94071e80 2023-03-03T17:26:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b93fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b94072250 0x7f3b940c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:27.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b93fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b94072250 0x7f3b940c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33024/0 (socket says 172.21.15.7:33024) 2023-03-03T17:26:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b93fff700 1 -- 172.21.15.7:0/3461480826 learned_addr learned my addr 172.21.15.7:0/3461480826 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:26:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b99030700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b940c1ce0 0x7f3b941e5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b9882f700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b94071e80 0x7f3b941e9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b93fff700 1 -- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b94071e80 msgr2=0x7f3b941e9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b93fff700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b94071e80 0x7f3b941e9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b93fff700 1 -- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b940c1ce0 msgr2=0x7f3b941e5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b93fff700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b940c1ce0 0x7f3b941e5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b93fff700 1 -- 172.21.15.7:0/3461480826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c009b30 con 0x7f3b94072250 2023-03-03T17:26:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b99030700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b940c1ce0 0x7f3b941e5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:26:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b93fff700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b94072250 0x7f3b940c5f70 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000e7e0 tx=0x7f3b8000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b917fa700 1 -- 172.21.15.7:0/3461480826 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80013610 con 0x7f3b94072250 2023-03-03T17:26:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b917fa700 1 -- 172.21.15.7:0/3461480826 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b80013770 con 0x7f3b94072250 2023-03-03T17:26:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b9aa93700 1 -- 172.21.15.7:0/3461480826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b94151f30 con 0x7f3b94072250 2023-03-03T17:26:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b917fa700 1 -- 172.21.15.7:0/3461480826 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80013ae0 con 0x7f3b94072250 2023-03-03T17:26:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.425+0000 7f3b9aa93700 1 -- 172.21.15.7:0/3461480826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b94151300 con 0x7f3b94072250 2023-03-03T17:26:27.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.426+0000 7f3b9aa93700 1 -- 172.21.15.7:0/3461480826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b94074bb0 con 0x7f3b94072250 2023-03-03T17:26:27.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.427+0000 7f3b917fa700 1 -- 172.21.15.7:0/3461480826 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3b80023070 con 0x7f3b94072250 2023-03-03T17:26:27.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.427+0000 7f3b917fa700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3b780682f0 0x7f3b7806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:27.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.427+0000 7f3b917fa700 1 -- 172.21.15.7:0/3461480826 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3b80085d80 con 0x7f3b94072250 2023-03-03T17:26:27.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.428+0000 7f3b9882f700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3b780682f0 0x7f3b7806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:27.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.428+0000 7f3b9882f700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3b780682f0 0x7f3b7806a7a0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f3b94146440 tx=0x7f3b88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.432+0000 7f3b917fa700 1 -- 172.21.15.7:0/3461480826 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3b80052120 con 0x7f3b94072250 2023-03-03T17:26:27.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.670+0000 7f3b9aa93700 1 -- 172.21.15.7:0/3461480826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b940be280 con 0x7f3b94072250 2023-03-03T17:26:27.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.671+0000 7f3b917fa700 1 -- 172.21.15.7:0/3461480826 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3b80059680 con 0x7f3b94072250 2023-03-03T17:26:27.672 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:26:27.672 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:26:27.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 -- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3b780682f0 msgr2=0x7f3b7806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:27.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3b780682f0 0x7f3b7806a7a0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f3b94146440 tx=0x7f3b88009040 comp rx=0 tx=0).stop 2023-03-03T17:26:27.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 -- 172.21.15.7:0/3461480826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b94072250 msgr2=0x7f3b940c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:27.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b94072250 0x7f3b940c5f70 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000e7e0 tx=0x7f3b8000eba0 comp rx=0 tx=0).stop 2023-03-03T17:26:27.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 -- 172.21.15.7:0/3461480826 shutdown_connections 2023-03-03T17:26:27.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3b940c1ce0 0x7f3b941e5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:27.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3b94071e80 0x7f3b941e9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:27.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3b780682f0 0x7f3b7806a7a0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:27.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 --2- 172.21.15.7:0/3461480826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b94072250 0x7f3b940c5f70 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:27.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 -- 172.21.15.7:0/3461480826 >> 172.21.15.7:0/3461480826 conn(0x7f3b940a94d0 msgr2=0x7f3b9406a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:26:27.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 -- 172.21.15.7:0/3461480826 shutdown_connections 2023-03-03T17:26:27.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:27.673+0000 7f3b9aa93700 1 -- 172.21.15.7:0/3461480826 wait complete. 2023-03-03T17:26:27.676 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:26:27.685 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before thrashing 2023-03-03T17:26:34.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:26:34.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.826+0000 7fae16e68700 1 Processor -- start 2023-03-03T17:26:34.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.826+0000 7fae16e68700 1 -- start start 2023-03-03T17:26:34.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.827+0000 7fae16e68700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae101530d0 0x7fae10156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:34.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.827+0000 7fae16e68700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae100c2f30 con 0x7fae1014fda0 2023-03-03T17:26:34.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.827+0000 7fae16e68700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae100c31d0 con 0x7fae1014f9d0 2023-03-03T17:26:34.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.827+0000 7fae16e68700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae101524a0 con 0x7fae101530d0 2023-03-03T17:26:34.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.827+0000 7fae0ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fae1014fda0 0x7fae101529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55242/0 (socket says 172.21.15.7:55242) 2023-03-03T17:26:34.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.827+0000 7fae0ffff700 1 -- 172.21.15.7:0/2582420413 learned_addr learned my addr 172.21.15.7:0/2582420413 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:26:34.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.827+0000 7fae15405700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae101530d0 0x7fae10156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:34.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.827+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173194152 0 0) 0x7fae100c2f30 con 0x7fae1014fda0 2023-03-03T17:26:34.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.827+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae000035b0 con 0x7fae1014fda0 2023-03-03T17:26:34.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3717213710 0 0) 0x7fae000035b0 con 0x7fae1014fda0 2023-03-03T17:26:34.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae101530d0 msgr2=0x7fae10156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:26:34.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae101530d0 0x7fae10156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:34.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 >> v1:172.21.15.143:6789/0 conn(0x7fae1014f9d0 legacy=0x7fae10071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:34.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae100c34b0 con 0x7fae1014fda0 2023-03-03T17:26:34.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fadfc001670 con 0x7fae1014fda0 2023-03-03T17:26:34.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 >> v1:172.21.15.7:6789/0 conn(0x7fae1014fda0 legacy=0x7fae101529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:34.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae00003aa0 0x7fae00003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:34.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae101530d0 0x7fae00004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:34.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae00004d90 0x7fae00005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:34.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae101524a0 con 0x7fae00003aa0 2023-03-03T17:26:34.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0000bbd0 con 0x7fae00004d90 2023-03-03T17:26:34.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0000bd90 con 0x7fae101530d0 2023-03-03T17:26:34.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae14c04700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae00003aa0 0x7fae00003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:34.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0ffff700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae101530d0 0x7fae00004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:34.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae15405700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae00004d90 0x7fae00005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:34.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae14c04700 1 -- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae101530d0 msgr2=0x7fae00004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:26:34.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae14c04700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae101530d0 0x7fae00004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:34.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae14c04700 1 -- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae00004d90 msgr2=0x7fae00005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:26:34.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae14c04700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae00004d90 0x7fae00005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:34.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae14c04700 1 -- 172.21.15.7:0/2582420413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae000035b0 con 0x7fae00003aa0 2023-03-03T17:26:34.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0ffff700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae101530d0 0x7fae00004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:26:34.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae14c04700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae00003aa0 0x7fae00003e70 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7fae040018b0 tx=0x7fae00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b848ff6a9fd0e38 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:34.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae04001370 con 0x7fae00003aa0 2023-03-03T17:26:34.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0400c6b0 con 0x7fae00003aa0 2023-03-03T17:26:34.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.828+0000 7fae0f7fe700 1 -- 172.21.15.7:0/2582420413 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0400ca60 con 0x7fae00003aa0 2023-03-03T17:26:34.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.829+0000 7fae16e68700 1 -- 172.21.15.7:0/2582420413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae00003aa0 msgr2=0x7fae00003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:34.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.829+0000 7fae16e68700 1 --2- 172.21.15.7:0/2582420413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae00003aa0 0x7fae00003e70 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7fae040018b0 tx=0x7fae00005880 comp rx=0 tx=0).stop 2023-03-03T17:26:34.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.829+0000 7fae15405700 1 -- 172.21.15.7:0/2582420413 reap_dead start 2023-03-03T17:26:34.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.829+0000 7fae16e68700 1 -- 172.21.15.7:0/2582420413 shutdown_connections 2023-03-03T17:26:34.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.829+0000 7fae16e68700 1 -- 172.21.15.7:0/2582420413 >> 172.21.15.7:0/2582420413 conn(0x7fae100c8900 msgr2=0x7fae100c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:26:34.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.829+0000 7fae16e68700 1 -- 172.21.15.7:0/2582420413 shutdown_connections 2023-03-03T17:26:34.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.829+0000 7fae16e68700 1 -- 172.21.15.7:0/2582420413 wait complete. 2023-03-03T17:26:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.830+0000 7fae16e68700 1 Processor -- start 2023-03-03T17:26:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.830+0000 7fae16e68700 1 -- start start 2023-03-03T17:26:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.831+0000 7fae16e68700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae00003aa0 0x7fae1014e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:34.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.831+0000 7fae16e68700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1006f370 0x7fae101eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:34.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.831+0000 7fae16e68700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae101eb380 0x7fae1014c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:34.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.831+0000 7fae16e68700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1006e280 con 0x7fae1006f370 2023-03-03T17:26:34.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.831+0000 7fae16e68700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae100c2360 con 0x7fae101eb380 2023-03-03T17:26:34.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.831+0000 7fae16e68700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae100c2920 con 0x7fae00003aa0 2023-03-03T17:26:34.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.831+0000 7fae0ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1006f370 0x7fae101eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:34.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.831+0000 7fae0ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1006f370 0x7fae101eae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33044/0 (socket says 172.21.15.7:33044) 2023-03-03T17:26:34.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.831+0000 7fae0ffff700 1 -- 172.21.15.7:0/4120287806 learned_addr learned my addr 172.21.15.7:0/4120287806 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:26:34.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.831+0000 7fae14c04700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae00003aa0 0x7fae1014e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:34.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae15405700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae101eb380 0x7fae1014c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:34.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae0ffff700 1 -- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae00003aa0 msgr2=0x7fae1014e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:34.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae0ffff700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae00003aa0 0x7fae1014e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:34.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae0ffff700 1 -- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae101eb380 msgr2=0x7fae1014c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:34.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae0ffff700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae101eb380 0x7fae1014c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:34.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae0ffff700 1 -- 172.21.15.7:0/4120287806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae100ac350 con 0x7fae1006f370 2023-03-03T17:26:34.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae14c04700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae00003aa0 0x7fae1014e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:26:34.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae0ffff700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1006f370 0x7fae101eae40 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00e7b0 tx=0x7fadfc00eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:34.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae0d7fa700 1 -- 172.21.15.7:0/4120287806 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc010510 con 0x7fae1006f370 2023-03-03T17:26:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae0d7fa700 1 -- 172.21.15.7:0/4120287806 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadfc010670 con 0x7fae1006f370 2023-03-03T17:26:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae16e68700 1 -- 172.21.15.7:0/4120287806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae100ab630 con 0x7fae1006f370 2023-03-03T17:26:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae0d7fa700 1 -- 172.21.15.7:0/4120287806 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc0109e0 con 0x7fae1006f370 2023-03-03T17:26:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.832+0000 7fae16e68700 1 -- 172.21.15.7:0/4120287806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae10071fa0 con 0x7fae1006f370 2023-03-03T17:26:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.833+0000 7fae16e68700 1 -- 172.21.15.7:0/4120287806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae10074bb0 con 0x7fae1006f370 2023-03-03T17:26:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.833+0000 7fae0d7fa700 1 -- 172.21.15.7:0/4120287806 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fadfc01b030 con 0x7fae1006f370 2023-03-03T17:26:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.834+0000 7fae0d7fa700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadf40682f0 0x7fadf406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.834+0000 7fae0d7fa700 1 -- 172.21.15.7:0/4120287806 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fadfc085720 con 0x7fae1006f370 2023-03-03T17:26:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.834+0000 7fae14c04700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadf40682f0 0x7fadf406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.835+0000 7fae14c04700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadf40682f0 0x7fadf406a7a0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fae040018b0 tx=0x7fae04005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:34.838+0000 7fae0d7fa700 1 -- 172.21.15.7:0/4120287806 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fadfc04d5e0 con 0x7fae1006f370 2023-03-03T17:26:35.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.072+0000 7fae16e68700 1 -- 172.21.15.7:0/4120287806 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae100c2a60 con 0x7fae1006f370 2023-03-03T17:26:35.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.073+0000 7fae0d7fa700 1 -- 172.21.15.7:0/4120287806 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fae100c2a60 con 0x7fae1006f370 2023-03-03T17:26:35.074 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:26:35.074 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:26:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 -- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadf40682f0 msgr2=0x7fadf406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadf40682f0 0x7fadf406a7a0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fae040018b0 tx=0x7fae04005040 comp rx=0 tx=0).stop 2023-03-03T17:26:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 -- 172.21.15.7:0/4120287806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1006f370 msgr2=0x7fae101eae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1006f370 0x7fae101eae40 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7fadfc00e7b0 tx=0x7fadfc00eb70 comp rx=0 tx=0).stop 2023-03-03T17:26:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 -- 172.21.15.7:0/4120287806 shutdown_connections 2023-03-03T17:26:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae101eb380 0x7fae1014c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae00003aa0 0x7fae1014e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fadf40682f0 0x7fadf406a7a0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 --2- 172.21.15.7:0/4120287806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae1006f370 0x7fae101eae40 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 -- 172.21.15.7:0/4120287806 >> 172.21.15.7:0/4120287806 conn(0x7fae100c8900 msgr2=0x7fae100716c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:26:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 -- 172.21.15.7:0/4120287806 shutdown_connections 2023-03-03T17:26:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:35.075+0000 7fae16e68700 1 -- 172.21.15.7:0/4120287806 wait complete. 2023-03-03T17:26:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:26:35.088 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T17:26:56.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:56.501 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:26:56.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:26:56.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.722+0000 7f708d5a5700 1 Processor -- start 2023-03-03T17:26:56.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.722+0000 7f708d5a5700 1 -- start start 2023-03-03T17:26:56.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.723+0000 7f708d5a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 0x7f70880736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:56.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.723+0000 7f708d5a5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708806d780 con 0x7f708806c5b0 2023-03-03T17:26:56.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.723+0000 7f708d5a5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708806cb70 con 0x7f7088073c30 2023-03-03T17:26:56.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.723+0000 7f708d5a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708806ce50 con 0x7f7088151ab0 2023-03-03T17:26:56.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.723+0000 7f70877fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f708806c5b0 0x7f7088157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55280/0 (socket says 172.21.15.7:55280) 2023-03-03T17:26:56.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.723+0000 7f70877fe700 1 -- 172.21.15.7:0/2119077449 learned_addr learned my addr 172.21.15.7:0/2119077449 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:26:56.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.723+0000 7f7086ffd700 1 --2- 172.21.15.7:0/2119077449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 0x7f70880736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:56.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7085ffb700 1 -- 172.21.15.7:0/2119077449 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916571103 0 0) 0x7f708806d780 con 0x7f708806c5b0 2023-03-03T17:26:56.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7085ffb700 1 -- 172.21.15.7:0/2119077449 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70700035b0 con 0x7f708806c5b0 2023-03-03T17:26:56.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7085ffb700 1 -- 172.21.15.7:0/2119077449 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839303256 0 0) 0x7f708806cb70 con 0x7f7088073c30 2023-03-03T17:26:56.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7085ffb700 1 -- 172.21.15.7:0/2119077449 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f708806d780 con 0x7f7088073c30 2023-03-03T17:26:56.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7085ffb700 1 -- 172.21.15.7:0/2119077449 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 203008834 0 0) 0x7f70700035b0 con 0x7f708806c5b0 2023-03-03T17:26:56.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7086ffd700 1 -- 172.21.15.7:0/2119077449 >> v1:172.21.15.143:6789/0 conn(0x7f7088073c30 legacy=0x7f708806bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:56.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7086ffd700 1 -- 172.21.15.7:0/2119077449 >> v1:172.21.15.7:6789/0 conn(0x7f708806c5b0 legacy=0x7f7088157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:56.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7086ffd700 1 -- 172.21.15.7:0/2119077449 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708806d130 con 0x7f7088151ab0 2023-03-03T17:26:56.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7086ffd700 1 --2- 172.21.15.7:0/2119077449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 0x7f70880736f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f707c009870 tx=0x7f707c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d37577643c251f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:56.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7085ffb700 1 -- 172.21.15.7:0/2119077449 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70800019f0 con 0x7f7088151ab0 2023-03-03T17:26:56.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7085ffb700 1 -- 172.21.15.7:0/2119077449 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707c004030 con 0x7f7088151ab0 2023-03-03T17:26:56.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.724+0000 7f7085ffb700 1 -- 172.21.15.7:0/2119077449 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c0043a0 con 0x7f7088151ab0 2023-03-03T17:26:56.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.725+0000 7f708d5a5700 1 -- 172.21.15.7:0/2119077449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 msgr2=0x7f70880736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:56.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.725+0000 7f708d5a5700 1 --2- 172.21.15.7:0/2119077449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 0x7f70880736f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f707c009870 tx=0x7f707c009b80 comp rx=0 tx=0).stop 2023-03-03T17:26:56.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.725+0000 7f708d5a5700 1 -- 172.21.15.7:0/2119077449 shutdown_connections 2023-03-03T17:26:56.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.725+0000 7f708d5a5700 1 --2- 172.21.15.7:0/2119077449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 0x7f70880736f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:56.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.725+0000 7f708d5a5700 1 -- 172.21.15.7:0/2119077449 >> 172.21.15.7:0/2119077449 conn(0x7f70880a94d0 msgr2=0x7f70880a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:26:56.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.725+0000 7f708d5a5700 1 -- 172.21.15.7:0/2119077449 shutdown_connections 2023-03-03T17:26:56.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.725+0000 7f708d5a5700 1 -- 172.21.15.7:0/2119077449 wait complete. 2023-03-03T17:26:56.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.726+0000 7f708d5a5700 1 Processor -- start 2023-03-03T17:26:56.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.726+0000 7f708d5a5700 1 -- start start 2023-03-03T17:26:56.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f708d5a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f708806c5b0 0x7f70880c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:56.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f708d5a5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7088073c30 0x7f708814c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:56.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f708d5a5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 0x7f70880aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:56.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f708d5a5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70880c5820 con 0x7f7088073c30 2023-03-03T17:26:56.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f708d5a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70880c6d20 con 0x7f708806c5b0 2023-03-03T17:26:56.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f708d5a5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70880c7000 con 0x7f7088151ab0 2023-03-03T17:26:56.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f70867fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7088073c30 0x7f708814c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:56.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f70867fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7088073c30 0x7f708814c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33072/0 (socket says 172.21.15.7:33072) 2023-03-03T17:26:56.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f70867fc700 1 -- 172.21.15.7:0/2831751048 learned_addr learned my addr 172.21.15.7:0/2831751048 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:26:56.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f7086ffd700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f708806c5b0 0x7f70880c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:56.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f70877fe700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 0x7f70880aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:56.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f70867fc700 1 -- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 msgr2=0x7f70880aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:56.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f70867fc700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 0x7f70880aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:56.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f70867fc700 1 -- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f708806c5b0 msgr2=0x7f70880c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:56.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f70867fc700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f708806c5b0 0x7f70880c5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:56.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f70867fc700 1 -- 172.21.15.7:0/2831751048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c0094a0 con 0x7f7088073c30 2023-03-03T17:26:56.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.727+0000 7f7086ffd700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f708806c5b0 0x7f70880c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:26:56.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.728+0000 7f70867fc700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7088073c30 0x7f708814c050 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f707800c520 tx=0x7f707800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:56.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.728+0000 7f706f7fe700 1 -- 172.21.15.7:0/2831751048 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70780052d0 con 0x7f7088073c30 2023-03-03T17:26:56.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.728+0000 7f706f7fe700 1 -- 172.21.15.7:0/2831751048 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7078005430 con 0x7f7088073c30 2023-03-03T17:26:56.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.728+0000 7f708d5a5700 1 -- 172.21.15.7:0/2831751048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7088151e80 con 0x7f7088073c30 2023-03-03T17:26:56.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.728+0000 7f706f7fe700 1 -- 172.21.15.7:0/2831751048 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70780038c0 con 0x7f7088073c30 2023-03-03T17:26:56.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.728+0000 7f708d5a5700 1 -- 172.21.15.7:0/2831751048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708806d410 con 0x7f7088073c30 2023-03-03T17:26:56.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.729+0000 7f708d5a5700 1 -- 172.21.15.7:0/2831751048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7088152010 con 0x7f7088073c30 2023-03-03T17:26:56.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.729+0000 7f706f7fe700 1 -- 172.21.15.7:0/2831751048 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7078019030 con 0x7f7088073c30 2023-03-03T17:26:56.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.730+0000 7f706f7fe700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70740682f0 0x7f707406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:26:56.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.730+0000 7f706f7fe700 1 -- 172.21.15.7:0/2831751048 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7078085660 con 0x7f7088073c30 2023-03-03T17:26:56.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.730+0000 7f7086ffd700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70740682f0 0x7f707406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:26:56.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.730+0000 7f7086ffd700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70740682f0 0x7f707406a7a0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f707c0090b0 tx=0x7f707c013040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:26:56.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.734+0000 7f706f7fe700 1 -- 172.21.15.7:0/2831751048 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7078051910 con 0x7f7088073c30 2023-03-03T17:26:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.976+0000 7f708d5a5700 1 -- 172.21.15.7:0/2831751048 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7088074bb0 con 0x7f7088073c30 2023-03-03T17:26:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.977+0000 7f706f7fe700 1 -- 172.21.15.7:0/2831751048 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7078054f30 con 0x7f7088073c30 2023-03-03T17:26:56.977 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:26:56.977 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:26:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 -- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70740682f0 msgr2=0x7f707406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70740682f0 0x7f707406a7a0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f707c0090b0 tx=0x7f707c013040 comp rx=0 tx=0).stop 2023-03-03T17:26:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 -- 172.21.15.7:0/2831751048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7088073c30 msgr2=0x7f708814c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:26:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7088073c30 0x7f708814c050 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f707800c520 tx=0x7f707800c8e0 comp rx=0 tx=0).stop 2023-03-03T17:26:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 -- 172.21.15.7:0/2831751048 shutdown_connections 2023-03-03T17:26:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7088151ab0 0x7f70880aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f708806c5b0 0x7f70880c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70740682f0 0x7f707406a7a0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 --2- 172.21.15.7:0/2831751048 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7088073c30 0x7f708814c050 unknown :-1 s=CLOSED pgs=2906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:26:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 -- 172.21.15.7:0/2831751048 >> 172.21.15.7:0/2831751048 conn(0x7f70880a94d0 msgr2=0x7f7088155990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:26:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 -- 172.21.15.7:0/2831751048 shutdown_connections 2023-03-03T17:26:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:26:56.979+0000 7f708d5a5700 1 -- 172.21.15.7:0/2831751048 wait complete. 2023-03-03T17:26:56.981 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:26:56.992 INFO:tasks.mds_thrash.fs.[b]:waiting for 105 secs before thrashing 2023-03-03T17:27:17.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:27:17.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.228+0000 7ff97e636700 1 Processor -- start 2023-03-03T17:27:17.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.228+0000 7ff97e636700 1 -- start start 2023-03-03T17:27:17.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.229+0000 7ff97e636700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff97806d4b0 0x7ff97806d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.229+0000 7ff97e636700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff9780c8020 0x7ff9780c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.229+0000 7ff97e636700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9780c1da0 0x7ff9780c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.229+0000 7ff97e636700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9780c7330 con 0x7ff97806d4b0 2023-03-03T17:27:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.229+0000 7ff97e636700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff978154420 con 0x7ff9780c8020 2023-03-03T17:27:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.229+0000 7ff97e636700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9780c7b90 con 0x7ff9780c1da0 2023-03-03T17:27:17.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.229+0000 7ff977fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff97806d4b0 0x7ff97806d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.229+0000 7ff977fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff97806d4b0 0x7ff97806d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33084/0 (socket says 172.21.15.7:33084) 2023-03-03T17:27:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.229+0000 7ff977fff700 1 -- 172.21.15.7:0/806611484 learned_addr learned my addr 172.21.15.7:0/806611484 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:27:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff9777fe700 1 --2- 172.21.15.7:0/806611484 >> v2:172.21.15.143:3300/0 conn(0x7ff9780c8020 0x7ff9780c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff97cbd3700 1 --2- 172.21.15.7:0/806611484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9780c1da0 0x7ff9780c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff977fff700 1 -- 172.21.15.7:0/806611484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9780c1da0 msgr2=0x7ff9780c0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:27:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff977fff700 1 --2- 172.21.15.7:0/806611484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9780c1da0 0x7ff9780c0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff977fff700 1 -- 172.21.15.7:0/806611484 >> v2:172.21.15.143:3300/0 conn(0x7ff9780c8020 msgr2=0x7ff9780c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:27:17.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff977fff700 1 --2- 172.21.15.7:0/806611484 >> v2:172.21.15.143:3300/0 conn(0x7ff9780c8020 0x7ff9780c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff977fff700 1 -- 172.21.15.7:0/806611484 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff978154640 con 0x7ff97806d4b0 2023-03-03T17:27:17.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff97cbd3700 1 --2- 172.21.15.7:0/806611484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9780c1da0 0x7ff9780c0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:27:17.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff977fff700 1 --2- 172.21.15.7:0/806611484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff97806d4b0 0x7ff97806d880 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7ff97806fcb0 tx=0x7ff960009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d6f0bfeb559bb2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:27:17.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff976ffd700 1 -- 172.21.15.7:0/806611484 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96000b870 con 0x7ff97806d4b0 2023-03-03T17:27:17.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff976ffd700 1 -- 172.21.15.7:0/806611484 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96000b9d0 con 0x7ff97806d4b0 2023-03-03T17:27:17.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.230+0000 7ff976ffd700 1 -- 172.21.15.7:0/806611484 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96000bd60 con 0x7ff97806d4b0 2023-03-03T17:27:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.231+0000 7ff97e636700 1 -- 172.21.15.7:0/806611484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff97806d4b0 msgr2=0x7ff97806d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.231+0000 7ff97e636700 1 --2- 172.21.15.7:0/806611484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff97806d4b0 0x7ff97806d880 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7ff97806fcb0 tx=0x7ff960009bd0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.231+0000 7ff97e636700 1 -- 172.21.15.7:0/806611484 shutdown_connections 2023-03-03T17:27:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.231+0000 7ff97e636700 1 --2- 172.21.15.7:0/806611484 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9780c1da0 0x7ff9780c0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.231+0000 7ff97e636700 1 --2- 172.21.15.7:0/806611484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff97806d4b0 0x7ff97806d880 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.231+0000 7ff97e636700 1 --2- 172.21.15.7:0/806611484 >> v2:172.21.15.143:3300/0 conn(0x7ff9780c8020 0x7ff9780c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.231+0000 7ff97e636700 1 -- 172.21.15.7:0/806611484 >> 172.21.15.7:0/806611484 conn(0x7ff97806c110 msgr2=0x7ff97806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:27:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.231+0000 7ff97e636700 1 -- 172.21.15.7:0/806611484 shutdown_connections 2023-03-03T17:27:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.231+0000 7ff97e636700 1 -- 172.21.15.7:0/806611484 wait complete. 2023-03-03T17:27:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.232+0000 7ff97e636700 1 Processor -- start 2023-03-03T17:27:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.233+0000 7ff97e636700 1 -- start start 2023-03-03T17:27:17.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.233+0000 7ff97e636700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff97806d4b0 0x7ff978072410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:17.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.233+0000 7ff97e636700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9780c1da0 0x7ff978137070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:17.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.233+0000 7ff97e636700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9780c8020 0x7ff9780aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:17.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.233+0000 7ff97e636700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9780acd70 con 0x7ff9780c1da0 2023-03-03T17:27:17.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.233+0000 7ff97e636700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff978071560 con 0x7ff9780c8020 2023-03-03T17:27:17.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.233+0000 7ff97e636700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9781352e0 con 0x7ff97806d4b0 2023-03-03T17:27:17.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff9777fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9780c1da0 0x7ff978137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:17.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff9777fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9780c1da0 0x7ff978137070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33090/0 (socket says 172.21.15.7:33090) 2023-03-03T17:27:17.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff9777fe700 1 -- 172.21.15.7:0/4267517994 learned_addr learned my addr 172.21.15.7:0/4267517994 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:27:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff977fff700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff97806d4b0 0x7ff978072410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff97cbd3700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9780c8020 0x7ff9780aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff9777fe700 1 -- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff97806d4b0 msgr2=0x7ff978072410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff9777fe700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff97806d4b0 0x7ff978072410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff9777fe700 1 -- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9780c8020 msgr2=0x7ff9780aa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff9777fe700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9780c8020 0x7ff9780aa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff9777fe700 1 -- 172.21.15.7:0/4267517994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff960009550 con 0x7ff9780c1da0 2023-03-03T17:27:17.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff9777fe700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9780c1da0 0x7ff978137070 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7ff96800d800 tx=0x7ff96800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:27:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff974ff9700 1 -- 172.21.15.7:0/4267517994 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9680125f0 con 0x7ff9780c1da0 2023-03-03T17:27:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff974ff9700 1 -- 172.21.15.7:0/4267517994 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff968012750 con 0x7ff9780c1da0 2023-03-03T17:27:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff974ff9700 1 -- 172.21.15.7:0/4267517994 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff968012b00 con 0x7ff9780c1da0 2023-03-03T17:27:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff97e636700 1 -- 172.21.15.7:0/4267517994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9780c7820 con 0x7ff9780c1da0 2023-03-03T17:27:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.234+0000 7ff97e636700 1 -- 172.21.15.7:0/4267517994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9780c7540 con 0x7ff9780c1da0 2023-03-03T17:27:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.235+0000 7ff97e636700 1 -- 172.21.15.7:0/4267517994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff97806ddc0 con 0x7ff9780c1da0 2023-03-03T17:27:17.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.236+0000 7ff974ff9700 1 -- 172.21.15.7:0/4267517994 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff968012c60 con 0x7ff9780c1da0 2023-03-03T17:27:17.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.236+0000 7ff974ff9700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9580682f0 0x7ff95806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:17.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.236+0000 7ff974ff9700 1 -- 172.21.15.7:0/4267517994 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff9680861e0 con 0x7ff9780c1da0 2023-03-03T17:27:17.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.237+0000 7ff977fff700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9580682f0 0x7ff95806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:17.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.237+0000 7ff977fff700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9580682f0 0x7ff95806a7a0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7ff960014040 tx=0x7ff960008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:27:17.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.241+0000 7ff974ff9700 1 -- 172.21.15.7:0/4267517994 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff968052580 con 0x7ff9780c1da0 2023-03-03T17:27:17.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.481+0000 7ff97e636700 1 -- 172.21.15.7:0/4267517994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff978074bb0 con 0x7ff9780c1da0 2023-03-03T17:27:17.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.481+0000 7ff974ff9700 1 -- 172.21.15.7:0/4267517994 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff968055ba0 con 0x7ff9780c1da0 2023-03-03T17:27:17.481 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:27:17.481 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:27:17.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 -- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9580682f0 msgr2=0x7ff95806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:17.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9580682f0 0x7ff95806a7a0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7ff960014040 tx=0x7ff960008030 comp rx=0 tx=0).stop 2023-03-03T17:27:17.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 -- 172.21.15.7:0/4267517994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9780c1da0 msgr2=0x7ff978137070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:17.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9780c1da0 0x7ff978137070 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7ff96800d800 tx=0x7ff96800db10 comp rx=0 tx=0).stop 2023-03-03T17:27:17.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 -- 172.21.15.7:0/4267517994 shutdown_connections 2023-03-03T17:27:17.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9780c8020 0x7ff9780aa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff97806d4b0 0x7ff978072410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9580682f0 0x7ff95806a7a0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 --2- 172.21.15.7:0/4267517994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9780c1da0 0x7ff978137070 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:17.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 -- 172.21.15.7:0/4267517994 >> 172.21.15.7:0/4267517994 conn(0x7ff97806c110 msgr2=0x7ff9780c6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:27:17.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 -- 172.21.15.7:0/4267517994 shutdown_connections 2023-03-03T17:27:17.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:17.483+0000 7ff97e636700 1 -- 172.21.15.7:0/4267517994 wait complete. 2023-03-03T17:27:17.485 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:27:17.495 INFO:tasks.mds_thrash.fs.[a]:waiting for 17 secs before thrashing 2023-03-03T17:27:26.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:26.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:34.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:27:34.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.634+0000 7f7673a5d700 1 Processor -- start 2023-03-03T17:27:34.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.634+0000 7f7673a5d700 1 -- start start 2023-03-03T17:27:34.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.635+0000 7f7673a5d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f766c151e80 0x7f766c0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:34.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.635+0000 7f7673a5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c0c6fe0 0x7f766c0bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:34.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.635+0000 7f7673a5d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766c0c08d0 con 0x7f766c155410 2023-03-03T17:27:34.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.635+0000 7f7673a5d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766c152250 con 0x7f766c151e80 2023-03-03T17:27:34.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.635+0000 7f7673a5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c0fb0 con 0x7f766c0c6fe0 2023-03-03T17:27:34.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.635+0000 7f7670ff8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f766c155410 0x7f766c0bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55326/0 (socket says 172.21.15.7:55326) 2023-03-03T17:27:34.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.635+0000 7f7670ff8700 1 -- 172.21.15.7:0/3216977270 learned_addr learned my addr 172.21.15.7:0/3216977270 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:27:34.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f76717f9700 1 --2- 172.21.15.7:0/3216977270 >> v2:172.21.15.143:3300/0 conn(0x7f766c151e80 0x7f766c0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:34.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f7671ffa700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c0c6fe0 0x7f766c0bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:34.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356698277 0 0) 0x7f766c0c08d0 con 0x7f766c155410 2023-03-03T17:27:34.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76580035b0 con 0x7f766c155410 2023-03-03T17:27:34.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3749837685 0 0) 0x7f76580035b0 con 0x7f766c155410 2023-03-03T17:27:34.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c0c6fe0 msgr2=0x7f766c0bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:27:34.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c0c6fe0 0x7f766c0bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:34.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 >> v2:172.21.15.143:3300/0 conn(0x7f766c151e80 msgr2=0x7f766c0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:27:34.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 --2- 172.21.15.7:0/3216977270 >> v2:172.21.15.143:3300/0 conn(0x7f766c151e80 0x7f766c0c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:34.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c154930 con 0x7f766c155410 2023-03-03T17:27:34.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7660001670 con 0x7f766c155410 2023-03-03T17:27:34.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 >> v1:172.21.15.7:6789/0 conn(0x7f766c155410 legacy=0x7f766c0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:34.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7658003a70 0x7f7658003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:34.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7658004850 0x7f7658004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:34.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c0c6fe0 0x7f7658005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:34.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765800ba10 con 0x7f7658004850 2023-03-03T17:27:34.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765800bb50 con 0x7f7658003a70 2023-03-03T17:27:34.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765800bc90 con 0x7f766c0c6fe0 2023-03-03T17:27:34.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f7670ff8700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7658004850 0x7f7658004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:34.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f76717f9700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7658003a70 0x7f7658003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:34.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.636+0000 7f7671ffa700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c0c6fe0 0x7f7658005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:34.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7670ff8700 1 -- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c0c6fe0 msgr2=0x7f7658005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:27:34.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7670ff8700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c0c6fe0 0x7f7658005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:34.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7670ff8700 1 -- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7658003a70 msgr2=0x7f7658003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:27:34.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7670ff8700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7658003a70 0x7f7658003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:34.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7670ff8700 1 -- 172.21.15.7:0/3216977270 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76580035b0 con 0x7f7658004850 2023-03-03T17:27:34.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7670ff8700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7658004850 0x7f7658004cc0 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f766c155870 tx=0x7f7658005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62e098a78340db8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:27:34.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660007c60 con 0x7f7658004850 2023-03-03T17:27:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766000ce50 con 0x7f7658004850 2023-03-03T17:27:34.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f765ffff700 1 -- 172.21.15.7:0/3216977270 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7660005ba0 con 0x7f7658004850 2023-03-03T17:27:34.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7673a5d700 1 -- 172.21.15.7:0/3216977270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7658004850 msgr2=0x7f7658004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:34.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7673a5d700 1 --2- 172.21.15.7:0/3216977270 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7658004850 0x7f7658004cc0 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f766c155870 tx=0x7f7658005880 comp rx=0 tx=0).stop 2023-03-03T17:27:34.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7671ffa700 1 -- 172.21.15.7:0/3216977270 reap_dead start 2023-03-03T17:27:34.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7673a5d700 1 -- 172.21.15.7:0/3216977270 shutdown_connections 2023-03-03T17:27:34.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.637+0000 7f7673a5d700 1 -- 172.21.15.7:0/3216977270 >> 172.21.15.7:0/3216977270 conn(0x7f766c071630 msgr2=0x7f766c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:27:34.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.638+0000 7f7673a5d700 1 -- 172.21.15.7:0/3216977270 shutdown_connections 2023-03-03T17:27:34.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.638+0000 7f7673a5d700 1 -- 172.21.15.7:0/3216977270 wait complete. 2023-03-03T17:27:34.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.639+0000 7f7673a5d700 1 Processor -- start 2023-03-03T17:27:34.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.639+0000 7f7673a5d700 1 -- start start 2023-03-03T17:27:34.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.639+0000 7f7673a5d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7658004850 0x7f766c1ea700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:34.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.639+0000 7f7673a5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c1df280 0x7f766c1e1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:34.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.639+0000 7f7673a5d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c1e1aa0 0x7f766c0c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:34.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.639+0000 7f7673a5d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c2970 con 0x7f766c1e1aa0 2023-03-03T17:27:34.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.639+0000 7f7673a5d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c1eac40 con 0x7f7658004850 2023-03-03T17:27:34.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.639+0000 7f7673a5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766c1ea030 con 0x7f766c1df280 2023-03-03T17:27:34.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7671ffa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c1e1aa0 0x7f766c0c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:34.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7671ffa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c1e1aa0 0x7f766c0c2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33130/0 (socket says 172.21.15.7:33130) 2023-03-03T17:27:34.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7671ffa700 1 -- 172.21.15.7:0/1175974549 learned_addr learned my addr 172.21.15.7:0/1175974549 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:27:34.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f76717f9700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7658004850 0x7f766c1ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:34.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7670ff8700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c1df280 0x7f766c1e1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:34.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7671ffa700 1 -- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c1df280 msgr2=0x7f766c1e1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:34.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7671ffa700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c1df280 0x7f766c1e1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:34.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7671ffa700 1 -- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7658004850 msgr2=0x7f766c1ea700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:34.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7671ffa700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7658004850 0x7f766c1ea700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:34.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7671ffa700 1 -- 172.21.15.7:0/1175974549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c154ef0 con 0x7f766c1e1aa0 2023-03-03T17:27:34.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f76717f9700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7658004850 0x7f766c1ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:27:34.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7670ff8700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c1df280 0x7f766c1e1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:27:34.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7671ffa700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c1e1aa0 0x7f766c0c2d90 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f766c06c350 tx=0x7f7664011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:27:34.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f765dffb700 1 -- 172.21.15.7:0/1175974549 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766400e040 con 0x7f766c1e1aa0 2023-03-03T17:27:34.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f765dffb700 1 -- 172.21.15.7:0/1175974549 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766400f040 con 0x7f766c1e1aa0 2023-03-03T17:27:34.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7673a5d700 1 -- 172.21.15.7:0/1175974549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766c154c10 con 0x7f766c1e1aa0 2023-03-03T17:27:34.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f765dffb700 1 -- 172.21.15.7:0/1175974549 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766400fdb0 con 0x7f766c1e1aa0 2023-03-03T17:27:34.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.640+0000 7f7673a5d700 1 -- 172.21.15.7:0/1175974549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766c0c5220 con 0x7f766c1e1aa0 2023-03-03T17:27:34.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.641+0000 7f7673a5d700 1 -- 172.21.15.7:0/1175974549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766c074bb0 con 0x7f766c1e1aa0 2023-03-03T17:27:34.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.643+0000 7f765dffb700 1 -- 172.21.15.7:0/1175974549 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7664015430 con 0x7f766c1e1aa0 2023-03-03T17:27:34.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.643+0000 7f765dffb700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76500682f0 0x7f765006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:27:34.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.643+0000 7f765dffb700 1 -- 172.21.15.7:0/1175974549 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7664087180 con 0x7f766c1e1aa0 2023-03-03T17:27:34.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.643+0000 7f76717f9700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76500682f0 0x7f765006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:27:34.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.644+0000 7f76717f9700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76500682f0 0x7f765006a7a0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f7668009b50 tx=0x7f7668008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:27:34.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.646+0000 7f765dffb700 1 -- 172.21.15.7:0/1175974549 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7664053560 con 0x7f766c1e1aa0 2023-03-03T17:27:34.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.885+0000 7f7673a5d700 1 -- 172.21.15.7:0/1175974549 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f766c0c2ab0 con 0x7f766c1e1aa0 2023-03-03T17:27:34.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.886+0000 7f765dffb700 1 -- 172.21.15.7:0/1175974549 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f766c0c2ab0 con 0x7f766c1e1aa0 2023-03-03T17:27:34.887 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:27:34.887 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:27:34.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 -- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76500682f0 msgr2=0x7f765006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:34.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76500682f0 0x7f765006a7a0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f7668009b50 tx=0x7f7668008040 comp rx=0 tx=0).stop 2023-03-03T17:27:34.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 -- 172.21.15.7:0/1175974549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c1e1aa0 msgr2=0x7f766c0c2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:27:34.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c1e1aa0 0x7f766c0c2d90 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f766c06c350 tx=0x7f7664011f80 comp rx=0 tx=0).stop 2023-03-03T17:27:34.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 -- 172.21.15.7:0/1175974549 shutdown_connections 2023-03-03T17:27:34.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f76500682f0 0x7f765006a7a0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:34.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f766c1e1aa0 0x7f766c0c2d90 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:34.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7658004850 0x7f766c1ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:34.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 --2- 172.21.15.7:0/1175974549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f766c1df280 0x7f766c1e1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:27:34.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 -- 172.21.15.7:0/1175974549 >> 172.21.15.7:0/1175974549 conn(0x7f766c071630 msgr2=0x7f766c0c9870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:27:34.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 -- 172.21.15.7:0/1175974549 shutdown_connections 2023-03-03T17:27:34.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:27:34.888+0000 7f7673a5d700 1 -- 172.21.15.7:0/1175974549 wait complete. 2023-03-03T17:27:34.890 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:27:34.900 INFO:tasks.mds_thrash.fs.[a]:waiting for 48 secs before thrashing 2023-03-03T17:27:56.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:27:56.662 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:22.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:28:23.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.051+0000 7f6b1bd30700 1 Processor -- start 2023-03-03T17:28:23.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.051+0000 7f6b1bd30700 1 -- start start 2023-03-03T17:28:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1bd30700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6b1406e020 0x7f6b14071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1bd30700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b14072520 0x7f6b14151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1bd30700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b14154910 con 0x7f6b14155390 2023-03-03T17:28:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1bd30700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b14070b90 con 0x7f6b14072520 2023-03-03T17:28:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1bd30700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b14151320 con 0x7f6b1406e020 2023-03-03T17:28:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1a2cd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b14072520 0x7f6b14151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1a2cd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b14072520 0x7f6b14151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33168/0 (socket says 172.21.15.7:33168) 2023-03-03T17:28:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1a2cd700 1 -- 172.21.15.7:0/2663011221 learned_addr learned my addr 172.21.15.7:0/2663011221 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:28:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b192cb700 1 --2- 172.21.15.7:0/2663011221 >> v2:172.21.15.143:3300/0 conn(0x7f6b1406e020 0x7f6b14071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1a2cd700 1 -- 172.21.15.7:0/2663011221 >> v2:172.21.15.143:3300/0 conn(0x7f6b1406e020 msgr2=0x7f6b14071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:28:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1a2cd700 1 --2- 172.21.15.7:0/2663011221 >> v2:172.21.15.143:3300/0 conn(0x7f6b1406e020 0x7f6b14071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1a2cd700 1 -- 172.21.15.7:0/2663011221 >> v1:172.21.15.143:6789/0 conn(0x7f6b14155390 legacy=0x7f6b140c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:28:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1a2cd700 1 -- 172.21.15.7:0/2663011221 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b140c4540 con 0x7f6b14072520 2023-03-03T17:28:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.052+0000 7f6b1a2cd700 1 --2- 172.21.15.7:0/2663011221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b14072520 0x7f6b14151770 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f6b140ac9e0 tx=0x7f6b0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b7b2c531973425 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b18aca700 1 -- 172.21.15.7:0/2663011221 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c00e070 con 0x7f6b14072520 2023-03-03T17:28:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b18aca700 1 -- 172.21.15.7:0/2663011221 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c003a20 con 0x7f6b14072520 2023-03-03T17:28:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b18aca700 1 -- 172.21.15.7:0/2663011221 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c003dd0 con 0x7f6b14072520 2023-03-03T17:28:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b1bd30700 1 -- 172.21.15.7:0/2663011221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b14072520 msgr2=0x7f6b14151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b1bd30700 1 --2- 172.21.15.7:0/2663011221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b14072520 0x7f6b14151770 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f6b140ac9e0 tx=0x7f6b0c00b330 comp rx=0 tx=0).stop 2023-03-03T17:28:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b1bd30700 1 -- 172.21.15.7:0/2663011221 shutdown_connections 2023-03-03T17:28:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b1bd30700 1 --2- 172.21.15.7:0/2663011221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b14072520 0x7f6b14151770 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b1bd30700 1 --2- 172.21.15.7:0/2663011221 >> v2:172.21.15.143:3300/0 conn(0x7f6b1406e020 0x7f6b14071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b1bd30700 1 -- 172.21.15.7:0/2663011221 >> 172.21.15.7:0/2663011221 conn(0x7f6b1406c110 msgr2=0x7f6b1406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:28:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b1bd30700 1 -- 172.21.15.7:0/2663011221 shutdown_connections 2023-03-03T17:28:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.053+0000 7f6b1bd30700 1 -- 172.21.15.7:0/2663011221 wait complete. 2023-03-03T17:28:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.055+0000 7f6b1bd30700 1 Processor -- start 2023-03-03T17:28:23.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.055+0000 7f6b1bd30700 1 -- start start 2023-03-03T17:28:23.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b1bd30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b1406e020 0x7f6b14148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:23.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b1bd30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b14072520 0x7f6b141eb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b1bd30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b14155390 0x7f6b1414c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b1bd30700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1414e1d0 con 0x7f6b1406e020 2023-03-03T17:28:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b1bd30700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1414b9e0 con 0x7f6b14155390 2023-03-03T17:28:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b1bd30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b141ebf50 con 0x7f6b14072520 2023-03-03T17:28:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b19acc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b1406e020 0x7f6b14148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b19acc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b1406e020 0x7f6b14148150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33170/0 (socket says 172.21.15.7:33170) 2023-03-03T17:28:23.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b19acc700 1 -- 172.21.15.7:0/303832842 learned_addr learned my addr 172.21.15.7:0/303832842 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:28:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b1a2cd700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b14155390 0x7f6b1414c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b192cb700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b14072520 0x7f6b141eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b19acc700 1 -- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b14072520 msgr2=0x7f6b141eb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b19acc700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b14072520 0x7f6b141eb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b19acc700 1 -- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b14155390 msgr2=0x7f6b1414c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:23.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b19acc700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b14155390 0x7f6b1414c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.056+0000 7f6b19acc700 1 -- 172.21.15.7:0/303832842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c009b30 con 0x7f6b1406e020 2023-03-03T17:28:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.057+0000 7f6b19acc700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b1406e020 0x7f6b14148150 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f6b0400b4a0 tx=0x7f6b0400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:23.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.057+0000 7f6b0a7fc700 1 -- 172.21.15.7:0/303832842 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04011830 con 0x7f6b1406e020 2023-03-03T17:28:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.057+0000 7f6b0a7fc700 1 -- 172.21.15.7:0/303832842 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b04011990 con 0x7f6b1406e020 2023-03-03T17:28:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.057+0000 7f6b1bd30700 1 -- 172.21.15.7:0/303832842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b14070540 con 0x7f6b1406e020 2023-03-03T17:28:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.057+0000 7f6b0a7fc700 1 -- 172.21.15.7:0/303832842 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04011d40 con 0x7f6b1406e020 2023-03-03T17:28:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.057+0000 7f6b1bd30700 1 -- 172.21.15.7:0/303832842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b140702a0 con 0x7f6b1406e020 2023-03-03T17:28:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.058+0000 7f6b1bd30700 1 -- 172.21.15.7:0/303832842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b14070820 con 0x7f6b1406e020 2023-03-03T17:28:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.058+0000 7f6b0a7fc700 1 -- 172.21.15.7:0/303832842 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6b0400f050 con 0x7f6b1406e020 2023-03-03T17:28:23.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.058+0000 7f6b0a7fc700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 0x7f6af806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.059+0000 7f6b0a7fc700 1 -- 172.21.15.7:0/303832842 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6b04085c30 con 0x7f6b1406e020 2023-03-03T17:28:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.059+0000 7f6b192cb700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 0x7f6af806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.059+0000 7f6b192cb700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 0x7f6af806a7a0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f6b14148930 tx=0x7f6b10008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:23.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.063+0000 7f6b0a7fc700 1 -- 172.21.15.7:0/303832842 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b04051fd0 con 0x7f6b1406e020 2023-03-03T17:28:23.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.300+0000 7f6b1bd30700 1 -- 172.21.15.7:0/303832842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b141503a0 con 0x7f6b1406e020 2023-03-03T17:28:23.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.301+0000 7f6b0a7fc700 1 -- 172.21.15.7:0/303832842 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6b040555f0 con 0x7f6b1406e020 2023-03-03T17:28:23.301 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:28:23.301 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:28:23.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 -- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 msgr2=0x7f6af806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:23.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 0x7f6af806a7a0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f6b14148930 tx=0x7f6b10008040 comp rx=0 tx=0).stop 2023-03-03T17:28:23.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 -- 172.21.15.7:0/303832842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b1406e020 msgr2=0x7f6b14148150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:23.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b1406e020 0x7f6b14148150 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f6b0400b4a0 tx=0x7f6b0400b7b0 comp rx=0 tx=0).stop 2023-03-03T17:28:23.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 -- 172.21.15.7:0/303832842 shutdown_connections 2023-03-03T17:28:23.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b14155390 0x7f6b1414c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:23.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6af80682f0 0x7f6af806a7a0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:23.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b1406e020 0x7f6b14148150 unknown :-1 s=CLOSED pgs=2913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:23.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 --2- 172.21.15.7:0/303832842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b14072520 0x7f6b141eb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:23.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 -- 172.21.15.7:0/303832842 >> 172.21.15.7:0/303832842 conn(0x7f6b1406c110 msgr2=0x7f6b140736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:28:23.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 -- 172.21.15.7:0/303832842 shutdown_connections 2023-03-03T17:28:23.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:23.303+0000 7f6b1bd30700 1 -- 172.21.15.7:0/303832842 wait complete. 2023-03-03T17:28:23.305 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:28:23.315 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T17:28:26.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:26.723 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:33.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:28:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.457+0000 7fa39298f700 1 Processor -- start 2023-03-03T17:28:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.457+0000 7fa39298f700 1 -- start start 2023-03-03T17:28:33.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.458+0000 7fa39298f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa38c072250 0x7fa38c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.458+0000 7fa39298f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c1510d0 con 0x7fa38c071e80 2023-03-03T17:28:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.458+0000 7fa39298f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c0c3740 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.458+0000 7fa39298f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38c070670 con 0x7fa38c072250 2023-03-03T17:28:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.458+0000 7fa38b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa38c072250 0x7fa38c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.458+0000 7fa38bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa38c071e80 0x7fa38c151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55398/0 (socket says 172.21.15.7:55398) 2023-03-03T17:28:33.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.458+0000 7fa38b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa38c072250 0x7fa38c14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33188/0 (socket says 172.21.15.7:33188) 2023-03-03T17:28:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.458+0000 7fa38bfff700 1 -- 172.21.15.7:0/2369997160 learned_addr learned my addr 172.21.15.7:0/2369997160 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:28:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.459+0000 7fa38b7fe700 1 -- 172.21.15.7:0/2369997160 >> v1:172.21.15.143:6789/0 conn(0x7fa38c0c1ce0 legacy=0x7fa38c15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:28:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.459+0000 7fa38b7fe700 1 -- 172.21.15.7:0/2369997160 >> v1:172.21.15.7:6789/0 conn(0x7fa38c071e80 legacy=0x7fa38c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.459+0000 7fa38b7fe700 1 -- 172.21.15.7:0/2369997160 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c0c39e0 con 0x7fa38c072250 2023-03-03T17:28:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.459+0000 7fa38b7fe700 1 --2- 172.21.15.7:0/2369997160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c072250 0x7fa38c14e1f0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7fa380009870 tx=0x7fa380009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5e8723867d1bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:33.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.459+0000 7fa38affd700 1 -- 172.21.15.7:0/2369997160 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa380004230 con 0x7fa38c072250 2023-03-03T17:28:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.459+0000 7fa38affd700 1 -- 172.21.15.7:0/2369997160 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa380004390 con 0x7fa38c072250 2023-03-03T17:28:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.459+0000 7fa38affd700 1 -- 172.21.15.7:0/2369997160 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3800036a0 con 0x7fa38c072250 2023-03-03T17:28:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.459+0000 7fa39298f700 1 -- 172.21.15.7:0/2369997160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c072250 msgr2=0x7fa38c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.459+0000 7fa39298f700 1 --2- 172.21.15.7:0/2369997160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c072250 0x7fa38c14e1f0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7fa380009870 tx=0x7fa380009b80 comp rx=0 tx=0).stop 2023-03-03T17:28:33.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.460+0000 7fa39298f700 1 -- 172.21.15.7:0/2369997160 shutdown_connections 2023-03-03T17:28:33.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.460+0000 7fa39298f700 1 --2- 172.21.15.7:0/2369997160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c072250 0x7fa38c14e1f0 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:33.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.460+0000 7fa39298f700 1 -- 172.21.15.7:0/2369997160 >> 172.21.15.7:0/2369997160 conn(0x7fa38c0a94d0 msgr2=0x7fa38c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:28:33.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.460+0000 7fa39298f700 1 -- 172.21.15.7:0/2369997160 shutdown_connections 2023-03-03T17:28:33.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.460+0000 7fa39298f700 1 -- 172.21.15.7:0/2369997160 wait complete. 2023-03-03T17:28:33.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.461+0000 7fa39298f700 1 Processor -- start 2023-03-03T17:28:33.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.461+0000 7fa39298f700 1 -- start start 2023-03-03T17:28:33.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa39298f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa38c071e80 0x7fa38c0c6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:33.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa39298f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa38c072250 0x7fa38c1435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:33.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa39298f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c0c1ce0 0x7fa38c143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:33.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa39298f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c144020 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa39298f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c145f70 con 0x7fa38c071e80 2023-03-03T17:28:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa39298f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c146250 con 0x7fa38c072250 2023-03-03T17:28:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa390f2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c0c1ce0 0x7fa38c143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:33.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa390f2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c0c1ce0 0x7fa38c143ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33196/0 (socket says 172.21.15.7:33196) 2023-03-03T17:28:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa390f2c700 1 -- 172.21.15.7:0/2420343988 learned_addr learned my addr 172.21.15.7:0/2420343988 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:28:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa38bfff700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa38c071e80 0x7fa38c0c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa38b7fe700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa38c072250 0x7fa38c1435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:33.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa390f2c700 1 -- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa38c072250 msgr2=0x7fa38c1435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa390f2c700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa38c072250 0x7fa38c1435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa390f2c700 1 -- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa38c071e80 msgr2=0x7fa38c0c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:33.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa390f2c700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa38c071e80 0x7fa38c0c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:33.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa390f2c700 1 -- 172.21.15.7:0/2420343988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3800094a0 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa38b7fe700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa38c072250 0x7fa38c1435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:28:33.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.462+0000 7fa390f2c700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c0c1ce0 0x7fa38c143ae0 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7fa38400b9e0 tx=0x7fa38400bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.463+0000 7fa388ff9700 1 -- 172.21.15.7:0/2420343988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38400c930 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.463+0000 7fa388ff9700 1 -- 172.21.15.7:0/2420343988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38400cad0 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.463+0000 7fa39298f700 1 -- 172.21.15.7:0/2420343988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c151f30 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.463+0000 7fa388ff9700 1 -- 172.21.15.7:0/2420343988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38400ce80 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.463+0000 7fa39298f700 1 -- 172.21.15.7:0/2420343988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c151300 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.464+0000 7fa39298f700 1 -- 172.21.15.7:0/2420343988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c06c9a0 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.464+0000 7fa388ff9700 1 -- 172.21.15.7:0/2420343988 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa38401b030 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.465+0000 7fa388ff9700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3780682f0 0x7fa37806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.465+0000 7fa388ff9700 1 -- 172.21.15.7:0/2420343988 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa384085bd0 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.465+0000 7fa38bfff700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3780682f0 0x7fa37806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.465+0000 7fa38bfff700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3780682f0 0x7fa37806a7a0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fa37c005910 tx=0x7fa37c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:33.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.469+0000 7fa388ff9700 1 -- 172.21.15.7:0/2420343988 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa38404d5e0 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.709+0000 7fa39298f700 1 -- 172.21.15.7:0/2420343988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa38c074bb0 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.709+0000 7fa388ff9700 1 -- 172.21.15.7:0/2420343988 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa384051f40 con 0x7fa38c0c1ce0 2023-03-03T17:28:33.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:28:33.709 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:28:33.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 -- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3780682f0 msgr2=0x7fa37806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:33.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3780682f0 0x7fa37806a7a0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fa37c005910 tx=0x7fa37c009040 comp rx=0 tx=0).stop 2023-03-03T17:28:33.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 -- 172.21.15.7:0/2420343988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c0c1ce0 msgr2=0x7fa38c143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:33.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c0c1ce0 0x7fa38c143ae0 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7fa38400b9e0 tx=0x7fa38400bcf0 comp rx=0 tx=0).stop 2023-03-03T17:28:33.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 -- 172.21.15.7:0/2420343988 shutdown_connections 2023-03-03T17:28:33.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3780682f0 0x7fa37806a7a0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:33.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa38c0c1ce0 0x7fa38c143ae0 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:33.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa38c071e80 0x7fa38c0c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:33.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 --2- 172.21.15.7:0/2420343988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa38c072250 0x7fa38c1435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:33.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 -- 172.21.15.7:0/2420343988 >> 172.21.15.7:0/2420343988 conn(0x7fa38c0a94d0 msgr2=0x7fa38c146760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:28:33.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 -- 172.21.15.7:0/2420343988 shutdown_connections 2023-03-03T17:28:33.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:33.711+0000 7fa39298f700 1 -- 172.21.15.7:0/2420343988 wait complete. 2023-03-03T17:28:33.713 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:28:33.723 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before thrashing 2023-03-03T17:28:35.724 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:28:35.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.865+0000 7f2861c10700 1 Processor -- start 2023-03-03T17:28:35.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.865+0000 7f2861c10700 1 -- start start 2023-03-03T17:28:35.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.865+0000 7f2861c10700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f285c151b20 0x7f285c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:35.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.865+0000 7f2861c10700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c0c2950 0x7f285c157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:35.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.865+0000 7f2861c10700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285c06cb70 con 0x7f285c073c30 2023-03-03T17:28:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.865+0000 7f2861c10700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f285c06c2f0 con 0x7f285c151b20 2023-03-03T17:28:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.865+0000 7f2861c10700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c06ce50 con 0x7f285c0c2950 2023-03-03T17:28:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f285c073c30 0x7f285c158020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55420/0 (socket says 172.21.15.7:55420) 2023-03-03T17:28:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285bfff700 1 -- 172.21.15.7:0/3065309670 learned_addr learned my addr 172.21.15.7:0/3065309670 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:28:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550452751 0 0) 0x7f285c06cb70 con 0x7f285c073c30 2023-03-03T17:28:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285b7fe700 1 --2- 172.21.15.7:0/3065309670 >> v2:172.21.15.143:3300/0 conn(0x7f285c151b20 0x7f285c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285affd700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c0c2950 0x7f285c157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:35.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28440035b0 con 0x7f285c073c30 2023-03-03T17:28:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3398958308 0 0) 0x7f28440035b0 con 0x7f285c073c30 2023-03-03T17:28:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c0c2950 msgr2=0x7f285c157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:28:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c0c2950 0x7f285c157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 >> v2:172.21.15.143:3300/0 conn(0x7f285c151b20 msgr2=0x7f285c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:28:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 --2- 172.21.15.7:0/3065309670 >> v2:172.21.15.143:3300/0 conn(0x7f285c151b20 0x7f285c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c06d130 con 0x7f285c073c30 2023-03-03T17:28:35.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f28540019f0 con 0x7f285c073c30 2023-03-03T17:28:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 >> v1:172.21.15.7:6789/0 conn(0x7f285c073c30 legacy=0x7f285c158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2844003a70 0x7f2844003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 0x7f2844004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c0c2950 0x7f2844005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c06c2f0 con 0x7f2844004850 2023-03-03T17:28:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c06ce50 con 0x7f2844003a70 2023-03-03T17:28:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284400ba40 con 0x7f285c0c2950 2023-03-03T17:28:35.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285affd700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 0x7f2844004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.866+0000 7f285b7fe700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2844003a70 0x7f2844003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285bfff700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c0c2950 0x7f2844005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285affd700 1 -- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c0c2950 msgr2=0x7f2844005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:28:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285affd700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c0c2950 0x7f2844005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285affd700 1 -- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2844003a70 msgr2=0x7f2844003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:28:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285affd700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2844003a70 0x7f2844003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:35.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285affd700 1 -- 172.21.15.7:0/3065309670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28440035b0 con 0x7f2844004850 2023-03-03T17:28:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285b7fe700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2844003a70 0x7f2844003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:28:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285affd700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 0x7f2844004cc0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f2850009b60 tx=0x7f2844005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6428aa7b42ffe789 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285000e6c0 con 0x7f2844004850 2023-03-03T17:28:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285000e820 con 0x7f2844004850 2023-03-03T17:28:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.867+0000 7f285a7fc700 1 -- 172.21.15.7:0/3065309670 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285000eb90 con 0x7f2844004850 2023-03-03T17:28:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.868+0000 7f2861c10700 1 -- 172.21.15.7:0/3065309670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 msgr2=0x7f2844004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.868+0000 7f2861c10700 1 --2- 172.21.15.7:0/3065309670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 0x7f2844004cc0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f2850009b60 tx=0x7f2844005880 comp rx=0 tx=0).stop 2023-03-03T17:28:35.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.868+0000 7f285bfff700 1 -- 172.21.15.7:0/3065309670 reap_dead start 2023-03-03T17:28:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.868+0000 7f2861c10700 1 -- 172.21.15.7:0/3065309670 shutdown_connections 2023-03-03T17:28:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.868+0000 7f2861c10700 1 -- 172.21.15.7:0/3065309670 >> 172.21.15.7:0/3065309670 conn(0x7f285c0a94d0 msgr2=0x7f285c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:28:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.868+0000 7f2861c10700 1 -- 172.21.15.7:0/3065309670 shutdown_connections 2023-03-03T17:28:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.868+0000 7f2861c10700 1 -- 172.21.15.7:0/3065309670 wait complete. 2023-03-03T17:28:35.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.869+0000 7f2861c10700 1 Processor -- start 2023-03-03T17:28:35.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.869+0000 7f2861c10700 1 -- start start 2023-03-03T17:28:35.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f2861c10700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 0x7f285c0bb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:35.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f2861c10700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285c1522e0 0x7f285c153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f2861c10700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c1539a0 0x7f285c14b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f2861c10700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0bc2c0 con 0x7f2844004850 2023-03-03T17:28:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f2861c10700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0ad180 con 0x7f285c1522e0 2023-03-03T17:28:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f2861c10700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285c0baf30 con 0x7f285c1539a0 2023-03-03T17:28:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 0x7f285c0bb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 0x7f285c0bb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33220/0 (socket says 172.21.15.7:33220) 2023-03-03T17:28:35.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285b7fe700 1 -- 172.21.15.7:0/2662808360 learned_addr learned my addr 172.21.15.7:0/2662808360 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:28:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285affd700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285c1522e0 0x7f285c153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285bfff700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c1539a0 0x7f285c14b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285b7fe700 1 -- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c1539a0 msgr2=0x7f285c14b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285b7fe700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c1539a0 0x7f285c14b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285b7fe700 1 -- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285c1522e0 msgr2=0x7f285c153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285b7fe700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285c1522e0 0x7f285c153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:35.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285b7fe700 1 -- 172.21.15.7:0/2662808360 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285c158730 con 0x7f2844004850 2023-03-03T17:28:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285affd700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285c1522e0 0x7f285c153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:28:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285bfff700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c1539a0 0x7f285c14b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:28:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.870+0000 7f285b7fe700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 0x7f285c0bb770 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f284c011cd0 tx=0x7f284c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.871+0000 7f2843fff700 1 -- 172.21.15.7:0/2662808360 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c011e70 con 0x7f2844004850 2023-03-03T17:28:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.871+0000 7f2843fff700 1 -- 172.21.15.7:0/2662808360 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f284c00fd40 con 0x7f2844004850 2023-03-03T17:28:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.871+0000 7f2861c10700 1 -- 172.21.15.7:0/2662808360 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c06d410 con 0x7f2844004850 2023-03-03T17:28:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.871+0000 7f2843fff700 1 -- 172.21.15.7:0/2662808360 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284c004240 con 0x7f2844004850 2023-03-03T17:28:35.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.871+0000 7f2861c10700 1 -- 172.21.15.7:0/2662808360 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285c151ef0 con 0x7f2844004850 2023-03-03T17:28:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.872+0000 7f2861c10700 1 -- 172.21.15.7:0/2662808360 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285c074bb0 con 0x7f2844004850 2023-03-03T17:28:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.872+0000 7f2843fff700 1 -- 172.21.15.7:0/2662808360 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f284c024070 con 0x7f2844004850 2023-03-03T17:28:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.873+0000 7f2843fff700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28480682f0 0x7f284806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.873+0000 7f2843fff700 1 -- 172.21.15.7:0/2662808360 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f284c086630 con 0x7f2844004850 2023-03-03T17:28:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.873+0000 7f285affd700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28480682f0 0x7f284806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:35.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.873+0000 7f285affd700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28480682f0 0x7f284806a7a0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f285c154200 tx=0x7f2850008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:35.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:35.877+0000 7f2843fff700 1 -- 172.21.15.7:0/2662808360 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f284c0529d0 con 0x7f2844004850 2023-03-03T17:28:36.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.119+0000 7f2861c10700 1 -- 172.21.15.7:0/2662808360 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f285c074220 con 0x7f2844004850 2023-03-03T17:28:36.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.120+0000 7f2843fff700 1 -- 172.21.15.7:0/2662808360 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f284c022090 con 0x7f2844004850 2023-03-03T17:28:36.119 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:28:36.119 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:28:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 -- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28480682f0 msgr2=0x7f284806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28480682f0 0x7f284806a7a0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f285c154200 tx=0x7f2850008030 comp rx=0 tx=0).stop 2023-03-03T17:28:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 -- 172.21.15.7:0/2662808360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 msgr2=0x7f285c0bb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 0x7f285c0bb770 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f284c011cd0 tx=0x7f284c00f520 comp rx=0 tx=0).stop 2023-03-03T17:28:36.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 -- 172.21.15.7:0/2662808360 shutdown_connections 2023-03-03T17:28:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f285c1539a0 0x7f285c14b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28480682f0 0x7f284806a7a0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2844004850 0x7f285c0bb770 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 --2- 172.21.15.7:0/2662808360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f285c1522e0 0x7f285c153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 -- 172.21.15.7:0/2662808360 >> 172.21.15.7:0/2662808360 conn(0x7f285c0a94d0 msgr2=0x7f285c0713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:28:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 -- 172.21.15.7:0/2662808360 shutdown_connections 2023-03-03T17:28:36.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:36.122+0000 7f2861c10700 1 -- 172.21.15.7:0/2662808360 wait complete. 2023-03-03T17:28:36.124 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:28:36.133 INFO:tasks.mds_thrash.fs.[a]:waiting for 106 secs before thrashing 2023-03-03T17:28:41.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:28:42.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.132+0000 7f887a0bf700 1 Processor -- start 2023-03-03T17:28:42.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.133+0000 7f887a0bf700 1 -- start start 2023-03-03T17:28:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.133+0000 7f887a0bf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88740c1da0 0x7f88741550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.133+0000 7f887a0bf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88740c3e30 0x7f88740c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.133+0000 7f887a0bf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88740c7a60 con 0x7f887406d870 2023-03-03T17:28:42.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.133+0000 7f887a0bf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88740c2af0 con 0x7f88740c1da0 2023-03-03T17:28:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.133+0000 7f887a0bf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88740c3620 con 0x7f88740c3e30 2023-03-03T17:28:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.133+0000 7f8872ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88740c1da0 0x7f88741550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f8872ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88740c1da0 0x7f88741550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33234/0 (socket says 172.21.15.7:33234) 2023-03-03T17:28:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f8872ffd700 1 -- 172.21.15.7:0/1598675619 learned_addr learned my addr 172.21.15.7:0/1598675619 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:28:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f8873fff700 1 --2- 172.21.15.7:0/1598675619 >> v2:172.21.15.143:3300/0 conn(0x7f88740c3e30 0x7f88740c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f8872ffd700 1 -- 172.21.15.7:0/1598675619 >> v2:172.21.15.143:3300/0 conn(0x7f88740c3e30 msgr2=0x7f88740c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:28:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f8872ffd700 1 --2- 172.21.15.7:0/1598675619 >> v2:172.21.15.143:3300/0 conn(0x7f88740c3e30 0x7f88740c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:42.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f8872ffd700 1 -- 172.21.15.7:0/1598675619 >> v1:172.21.15.143:6789/0 conn(0x7f887406d870 legacy=0x7f88740c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f8872ffd700 1 -- 172.21.15.7:0/1598675619 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88741545e0 con 0x7f88740c1da0 2023-03-03T17:28:42.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f8873fff700 1 --2- 172.21.15.7:0/1598675619 >> v2:172.21.15.143:3300/0 conn(0x7f88740c3e30 0x7f88740c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:28:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f8872ffd700 1 --2- 172.21.15.7:0/1598675619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 0x7f88741550c0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f8874073830 tx=0x7f8868009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1a8af592cb8273 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f88727fc700 1 -- 172.21.15.7:0/1598675619 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886800b870 con 0x7f88740c1da0 2023-03-03T17:28:42.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f88727fc700 1 -- 172.21.15.7:0/1598675619 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886800b9d0 con 0x7f88740c1da0 2023-03-03T17:28:42.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.134+0000 7f88727fc700 1 -- 172.21.15.7:0/1598675619 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886800bd60 con 0x7f88740c1da0 2023-03-03T17:28:42.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.135+0000 7f887a0bf700 1 -- 172.21.15.7:0/1598675619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 msgr2=0x7f88741550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:42.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.135+0000 7f887a0bf700 1 --2- 172.21.15.7:0/1598675619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 0x7f88741550c0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f8874073830 tx=0x7f8868009bd0 comp rx=0 tx=0).stop 2023-03-03T17:28:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.135+0000 7f887a0bf700 1 -- 172.21.15.7:0/1598675619 shutdown_connections 2023-03-03T17:28:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.135+0000 7f887a0bf700 1 --2- 172.21.15.7:0/1598675619 >> v2:172.21.15.143:3300/0 conn(0x7f88740c3e30 0x7f88740c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.135+0000 7f887a0bf700 1 --2- 172.21.15.7:0/1598675619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 0x7f88741550c0 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:42.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.135+0000 7f887a0bf700 1 -- 172.21.15.7:0/1598675619 >> 172.21.15.7:0/1598675619 conn(0x7f887406c110 msgr2=0x7f887406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:28:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.135+0000 7f887a0bf700 1 -- 172.21.15.7:0/1598675619 shutdown_connections 2023-03-03T17:28:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.135+0000 7f887a0bf700 1 -- 172.21.15.7:0/1598675619 wait complete. 2023-03-03T17:28:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.136+0000 7f887a0bf700 1 Processor -- start 2023-03-03T17:28:42.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.136+0000 7f887a0bf700 1 -- start start 2023-03-03T17:28:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f887a0bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f887406d870 0x7f8874155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f887a0bf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 0x7f88740aa970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f887a0bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88740c3e30 0x7f88740aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:42.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f887a0bf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8874158490 con 0x7f88740c1da0 2023-03-03T17:28:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f887a0bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88740aa4e0 con 0x7f88740c3e30 2023-03-03T17:28:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f887a0bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88740ad480 con 0x7f887406d870 2023-03-03T17:28:42.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8872ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 0x7f88740aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8872ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 0x7f88740aa970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33240/0 (socket says 172.21.15.7:33240) 2023-03-03T17:28:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8872ffd700 1 -- 172.21.15.7:0/3438904449 learned_addr learned my addr 172.21.15.7:0/3438904449 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:28:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f88737fe700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f887406d870 0x7f8874155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:42.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8873fff700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88740c3e30 0x7f88740aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8872ffd700 1 -- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f887406d870 msgr2=0x7f8874155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8872ffd700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f887406d870 0x7f8874155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:42.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8872ffd700 1 -- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88740c3e30 msgr2=0x7f88740aaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:42.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8872ffd700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88740c3e30 0x7f88740aaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:42.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8872ffd700 1 -- 172.21.15.7:0/3438904449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8868009550 con 0x7f88740c1da0 2023-03-03T17:28:42.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8873fff700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88740c3e30 0x7f88740aaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:28:42.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.137+0000 7f8872ffd700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 0x7f88740aa970 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f8868009fd0 tx=0x7f8868004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:42.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.138+0000 7f885bfff700 1 -- 172.21.15.7:0/3438904449 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868004540 con 0x7f88740c1da0 2023-03-03T17:28:42.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.138+0000 7f885bfff700 1 -- 172.21.15.7:0/3438904449 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88680036a0 con 0x7f88740c1da0 2023-03-03T17:28:42.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.138+0000 7f887a0bf700 1 -- 172.21.15.7:0/3438904449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8874154ba0 con 0x7f88740c1da0 2023-03-03T17:28:42.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.138+0000 7f885bfff700 1 -- 172.21.15.7:0/3438904449 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868003a10 con 0x7f88740c1da0 2023-03-03T17:28:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.138+0000 7f887a0bf700 1 -- 172.21.15.7:0/3438904449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88741548c0 con 0x7f88740c1da0 2023-03-03T17:28:42.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.139+0000 7f887a0bf700 1 -- 172.21.15.7:0/3438904449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887406dc40 con 0x7f88740c1da0 2023-03-03T17:28:42.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.139+0000 7f885bfff700 1 -- 172.21.15.7:0/3438904449 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f886800f050 con 0x7f88740c1da0 2023-03-03T17:28:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.140+0000 7f885bfff700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88600682f0 0x7f886006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.140+0000 7f885bfff700 1 -- 172.21.15.7:0/3438904449 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f88680879e0 con 0x7f88740c1da0 2023-03-03T17:28:42.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.140+0000 7f88737fe700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88600682f0 0x7f886006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:42.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.140+0000 7f88737fe700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88600682f0 0x7f886006a7a0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f8874072400 tx=0x7f8864009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:42.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.144+0000 7f885bfff700 1 -- 172.21.15.7:0/3438904449 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8868053d80 con 0x7f88740c1da0 2023-03-03T17:28:42.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.384+0000 7f887a0bf700 1 -- 172.21.15.7:0/3438904449 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88740ad290 con 0x7f88740c1da0 2023-03-03T17:28:42.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.384+0000 7f885bfff700 1 -- 172.21.15.7:0/3438904449 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f88680573a0 con 0x7f88740c1da0 2023-03-03T17:28:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:28:42.385 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:28:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 -- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88600682f0 msgr2=0x7f886006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88600682f0 0x7f886006a7a0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f8874072400 tx=0x7f8864009040 comp rx=0 tx=0).stop 2023-03-03T17:28:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 -- 172.21.15.7:0/3438904449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 msgr2=0x7f88740aa970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 0x7f88740aa970 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f8868009fd0 tx=0x7f8868004750 comp rx=0 tx=0).stop 2023-03-03T17:28:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 -- 172.21.15.7:0/3438904449 shutdown_connections 2023-03-03T17:28:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88740c3e30 0x7f88740aaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f887406d870 0x7f8874155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f88600682f0 0x7f886006a7a0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 --2- 172.21.15.7:0/3438904449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88740c1da0 0x7f88740aa970 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 -- 172.21.15.7:0/3438904449 >> 172.21.15.7:0/3438904449 conn(0x7f887406c110 msgr2=0x7f8874071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:28:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 -- 172.21.15.7:0/3438904449 shutdown_connections 2023-03-03T17:28:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:42.386+0000 7f887a0bf700 1 -- 172.21.15.7:0/3438904449 wait complete. 2023-03-03T17:28:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:28:42.402 INFO:tasks.mds_thrash.fs.[b]:waiting for 10 secs before thrashing 2023-03-03T17:28:52.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:28:52.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.543+0000 7f18b6240700 1 Processor -- start 2023-03-03T17:28:52.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.543+0000 7f18b6240700 1 -- start start 2023-03-03T17:28:52.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18b6240700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18b0154fd0 0x7f18b006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:52.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18b6240700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b006c480 0x7f18b00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:52.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18b6240700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b006c9b0 con 0x7f18b0073c30 2023-03-03T17:28:52.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18b6240700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b0154570 con 0x7f18b0154fd0 2023-03-03T17:28:52.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18b6240700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b006cc20 con 0x7f18b006c480 2023-03-03T17:28:52.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18af7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f18b0154fd0 0x7f18b006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:52.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18affff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f18b0073c30 0x7f18b0157e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55478/0 (socket says 172.21.15.7:55478) 2023-03-03T17:28:52.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18affff700 1 -- 172.21.15.7:0/3842364926 learned_addr learned my addr 172.21.15.7:0/3842364926 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:28:52.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18aeffd700 1 --2- 172.21.15.7:0/3842364926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b006c480 0x7f18b00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:52.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18af7fe700 1 -- 172.21.15.7:0/3842364926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b006c480 msgr2=0x7f18b00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:28:52.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18af7fe700 1 --2- 172.21.15.7:0/3842364926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b006c480 0x7f18b00736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:52.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18af7fe700 1 -- 172.21.15.7:0/3842364926 >> v1:172.21.15.7:6789/0 conn(0x7f18b0073c30 legacy=0x7f18b0157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:52.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18af7fe700 1 -- 172.21.15.7:0/3842364926 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b006cf00 con 0x7f18b0154fd0 2023-03-03T17:28:52.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.544+0000 7f18ae7fc700 1 -- 172.21.15.7:0/3842364926 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830114388 0 0) 0x7f18b006c9b0 con 0x7f18b0073c30 2023-03-03T17:28:52.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.545+0000 7f18aeffd700 1 --2- 172.21.15.7:0/3842364926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b006c480 0x7f18b00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:28:52.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.545+0000 7f18af7fe700 1 --2- 172.21.15.7:0/3842364926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b0154fd0 0x7f18b006bea0 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009840 tx=0x7f18a0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2322a826e81400 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:52.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.545+0000 7f18ae7fc700 1 -- 172.21.15.7:0/3842364926 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0011830 con 0x7f18b0154fd0 2023-03-03T17:28:52.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.545+0000 7f18ae7fc700 1 -- 172.21.15.7:0/3842364926 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a0011990 con 0x7f18b0154fd0 2023-03-03T17:28:52.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.545+0000 7f18ae7fc700 1 -- 172.21.15.7:0/3842364926 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0011d20 con 0x7f18b0154fd0 2023-03-03T17:28:52.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.545+0000 7f18b6240700 1 -- 172.21.15.7:0/3842364926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b0154fd0 msgr2=0x7f18b006bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:52.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.545+0000 7f18b6240700 1 --2- 172.21.15.7:0/3842364926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b0154fd0 0x7f18b006bea0 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f18a0009840 tx=0x7f18a0009b50 comp rx=0 tx=0).stop 2023-03-03T17:28:52.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.545+0000 7f18b6240700 1 -- 172.21.15.7:0/3842364926 shutdown_connections 2023-03-03T17:28:52.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.545+0000 7f18b6240700 1 --2- 172.21.15.7:0/3842364926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b0154fd0 0x7f18b006bea0 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:52.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.546+0000 7f18b6240700 1 --2- 172.21.15.7:0/3842364926 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b006c480 0x7f18b00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:52.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.546+0000 7f18b6240700 1 -- 172.21.15.7:0/3842364926 >> 172.21.15.7:0/3842364926 conn(0x7f18b00a94d0 msgr2=0x7f18b00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:28:52.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.546+0000 7f18b6240700 1 -- 172.21.15.7:0/3842364926 shutdown_connections 2023-03-03T17:28:52.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.546+0000 7f18b6240700 1 -- 172.21.15.7:0/3842364926 wait complete. 2023-03-03T17:28:52.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.547+0000 7f18b6240700 1 Processor -- start 2023-03-03T17:28:52.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.547+0000 7f18b6240700 1 -- start start 2023-03-03T17:28:52.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18b6240700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b006c480 0x7f18b01539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:52.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18b6240700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b0073c30 0x7f18b014c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:52.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18b6240700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b0154fd0 0x7f18b014cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:52.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18b6240700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b00aa480 con 0x7f18b006c480 2023-03-03T17:28:52.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18b6240700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b01ee3e0 con 0x7f18b0154fd0 2023-03-03T17:28:52.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18b6240700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b014d6e0 con 0x7f18b0073c30 2023-03-03T17:28:52.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18af7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b006c480 0x7f18b01539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:52.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18af7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b006c480 0x7f18b01539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33268/0 (socket says 172.21.15.7:33268) 2023-03-03T17:28:52.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18af7fe700 1 -- 172.21.15.7:0/1742553815 learned_addr learned my addr 172.21.15.7:0/1742553815 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:28:52.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18affff700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b0154fd0 0x7f18b014cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:52.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18aeffd700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b0073c30 0x7f18b014c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:52.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18af7fe700 1 -- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b0073c30 msgr2=0x7f18b014c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:52.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18af7fe700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b0073c30 0x7f18b014c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:52.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18af7fe700 1 -- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b0154fd0 msgr2=0x7f18b014cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:52.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18af7fe700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b0154fd0 0x7f18b014cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:52.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18af7fe700 1 -- 172.21.15.7:0/1742553815 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a00094a0 con 0x7f18b006c480 2023-03-03T17:28:52.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18affff700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b0154fd0 0x7f18b014cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:28:52.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18aeffd700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b0073c30 0x7f18b014c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:28:52.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18af7fe700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b006c480 0x7f18b01539b0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f18a00090b0 tx=0x7f18a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:52.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18a7fff700 1 -- 172.21.15.7:0/1742553815 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a00038a0 con 0x7f18b006c480 2023-03-03T17:28:52.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18a7fff700 1 -- 172.21.15.7:0/1742553815 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a001e070 con 0x7f18b006c480 2023-03-03T17:28:52.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.548+0000 7f18b6240700 1 -- 172.21.15.7:0/1742553815 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b006d4c0 con 0x7f18b006c480 2023-03-03T17:28:52.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.549+0000 7f18a7fff700 1 -- 172.21.15.7:0/1742553815 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a0003c20 con 0x7f18b006c480 2023-03-03T17:28:52.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.549+0000 7f18b6240700 1 -- 172.21.15.7:0/1742553815 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b006d1e0 con 0x7f18b006c480 2023-03-03T17:28:52.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.549+0000 7f18b6240700 1 -- 172.21.15.7:0/1742553815 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b006d7a0 con 0x7f18b006c480 2023-03-03T17:28:52.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.550+0000 7f18a7fff700 1 -- 172.21.15.7:0/1742553815 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f18a000f050 con 0x7f18b006c480 2023-03-03T17:28:52.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.550+0000 7f18a7fff700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18900682f0 0x7f189006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:28:52.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.550+0000 7f18a7fff700 1 -- 172.21.15.7:0/1742553815 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f18a0020070 con 0x7f18b006c480 2023-03-03T17:28:52.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.551+0000 7f18aeffd700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18900682f0 0x7f189006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:28:52.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.551+0000 7f18aeffd700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18900682f0 0x7f189006a7a0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f18b00ad3c0 tx=0x7f1898008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:28:52.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.555+0000 7f18a7fff700 1 -- 172.21.15.7:0/1742553815 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f18a00538a0 con 0x7f18b006c480 2023-03-03T17:28:52.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.793+0000 7f18b6240700 1 -- 172.21.15.7:0/1742553815 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18b0074bb0 con 0x7f18b006c480 2023-03-03T17:28:52.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.794+0000 7f18a7fff700 1 -- 172.21.15.7:0/1742553815 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f18a0056ec0 con 0x7f18b006c480 2023-03-03T17:28:52.793 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:28:52.793 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:28:52.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.795+0000 7f18b6240700 1 -- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18900682f0 msgr2=0x7f189006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:52.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.795+0000 7f18b6240700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18900682f0 0x7f189006a7a0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f18b00ad3c0 tx=0x7f1898008040 comp rx=0 tx=0).stop 2023-03-03T17:28:52.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.795+0000 7f18b6240700 1 -- 172.21.15.7:0/1742553815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b006c480 msgr2=0x7f18b01539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:28:52.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.795+0000 7f18b6240700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b006c480 0x7f18b01539b0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7f18a00090b0 tx=0x7f18a0004750 comp rx=0 tx=0).stop 2023-03-03T17:28:52.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.795+0000 7f18b6240700 1 -- 172.21.15.7:0/1742553815 shutdown_connections 2023-03-03T17:28:52.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.795+0000 7f18b6240700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f18b0154fd0 0x7f18b014cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:52.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.795+0000 7f18b6240700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18900682f0 0x7f189006a7a0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:52.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.795+0000 7f18b6240700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18b006c480 0x7f18b01539b0 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:52.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.795+0000 7f18b6240700 1 --2- 172.21.15.7:0/1742553815 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f18b0073c30 0x7f18b014c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:28:52.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.795+0000 7f18b6240700 1 -- 172.21.15.7:0/1742553815 >> 172.21.15.7:0/1742553815 conn(0x7f18b00a94d0 msgr2=0x7f18b0152590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:28:52.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.796+0000 7f18b6240700 1 -- 172.21.15.7:0/1742553815 shutdown_connections 2023-03-03T17:28:52.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:28:52.796+0000 7f18b6240700 1 -- 172.21.15.7:0/1742553815 wait complete. 2023-03-03T17:28:52.797 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:28:52.807 INFO:tasks.mds_thrash.fs.[b]:waiting for 87 secs before thrashing 2023-03-03T17:28:56.794 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:28:56.798 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:26.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:26.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:56.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:56.966 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:29:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T17:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T17:30:19.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:30:19.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.950+0000 7f896bb64700 1 Processor -- start 2023-03-03T17:30:19.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.951+0000 7f896bb64700 1 -- start start 2023-03-03T17:30:19.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.951+0000 7f896bb64700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f896414f9d0 0x7f8964156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:19.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.951+0000 7f896bb64700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f896414fda0 0x7f8964071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:19.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.951+0000 7f896bb64700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89640abe20 0x7f89641529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:19.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.951+0000 7f896bb64700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89640c2f30 con 0x7f89640abe20 2023-03-03T17:30:19.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.951+0000 7f896bb64700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89640c31d0 con 0x7f896414f9d0 2023-03-03T17:30:19.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.951+0000 7f896bb64700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89641524a0 con 0x7f896414fda0 2023-03-03T17:30:19.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f896a101700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89640abe20 0x7f89641529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:19.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f896a101700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f89640abe20 0x7f89641529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33350/0 (socket says 172.21.15.7:33350) 2023-03-03T17:30:19.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f896a101700 1 -- 172.21.15.7:0/3202813307 learned_addr learned my addr 172.21.15.7:0/3202813307 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:30:19.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f8969900700 1 --2- 172.21.15.7:0/3202813307 >> v2:172.21.15.143:3300/0 conn(0x7f896414f9d0 0x7f8964156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:19.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f896a101700 1 -- 172.21.15.7:0/3202813307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f896414fda0 msgr2=0x7f8964071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:30:19.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f896a101700 1 --2- 172.21.15.7:0/3202813307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f896414fda0 0x7f8964071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:19.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f896a101700 1 -- 172.21.15.7:0/3202813307 >> v2:172.21.15.143:3300/0 conn(0x7f896414f9d0 msgr2=0x7f8964156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:30:19.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f896a101700 1 --2- 172.21.15.7:0/3202813307 >> v2:172.21.15.143:3300/0 conn(0x7f896414f9d0 0x7f8964156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:19.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f896a101700 1 -- 172.21.15.7:0/3202813307 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89640c34b0 con 0x7f89640abe20 2023-03-03T17:30:19.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f8969900700 1 --2- 172.21.15.7:0/3202813307 >> v2:172.21.15.143:3300/0 conn(0x7f896414f9d0 0x7f8964156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:30:19.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f896a101700 1 --2- 172.21.15.7:0/3202813307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89640abe20 0x7f89641529c0 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f89640710a0 tx=0x7f895400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27b44892bb6aede server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:30:19.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f89688fe700 1 -- 172.21.15.7:0/3202813307 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895400e070 con 0x7f89640abe20 2023-03-03T17:30:19.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f89688fe700 1 -- 172.21.15.7:0/3202813307 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895400b9c0 con 0x7f89640abe20 2023-03-03T17:30:19.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.952+0000 7f89688fe700 1 -- 172.21.15.7:0/3202813307 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895400bd70 con 0x7f89640abe20 2023-03-03T17:30:19.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.953+0000 7f896bb64700 1 -- 172.21.15.7:0/3202813307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89640abe20 msgr2=0x7f89641529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:19.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.953+0000 7f896bb64700 1 --2- 172.21.15.7:0/3202813307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89640abe20 0x7f89641529c0 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f89640710a0 tx=0x7f895400b330 comp rx=0 tx=0).stop 2023-03-03T17:30:19.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.953+0000 7f896bb64700 1 -- 172.21.15.7:0/3202813307 shutdown_connections 2023-03-03T17:30:19.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.953+0000 7f896bb64700 1 --2- 172.21.15.7:0/3202813307 >> v2:172.21.15.143:3300/0 conn(0x7f896414f9d0 0x7f8964156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:19.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.953+0000 7f896bb64700 1 --2- 172.21.15.7:0/3202813307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f89640abe20 0x7f89641529c0 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:19.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.953+0000 7f896bb64700 1 --2- 172.21.15.7:0/3202813307 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f896414fda0 0x7f8964071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:19.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.953+0000 7f896bb64700 1 -- 172.21.15.7:0/3202813307 >> 172.21.15.7:0/3202813307 conn(0x7f89640c8900 msgr2=0x7f89640c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:30:19.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.953+0000 7f896bb64700 1 -- 172.21.15.7:0/3202813307 shutdown_connections 2023-03-03T17:30:19.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.953+0000 7f896bb64700 1 -- 172.21.15.7:0/3202813307 wait complete. 2023-03-03T17:30:19.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.954+0000 7f896bb64700 1 Processor -- start 2023-03-03T17:30:19.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.954+0000 7f896bb64700 1 -- start start 2023-03-03T17:30:19.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f896bb64700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89640abe20 0x7f896406f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:19.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f896bb64700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f896414f9d0 0x7f896406e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:19.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f896bb64700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f896414fda0 0x7f89641eb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:19.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f896bb64700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89641eabe0 con 0x7f896414f9d0 2023-03-03T17:30:19.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f896bb64700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89641e8560 con 0x7f89640abe20 2023-03-03T17:30:19.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f896bb64700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89640be4b0 con 0x7f896414fda0 2023-03-03T17:30:19.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f89690ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f896414f9d0 0x7f896406e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:19.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f89690ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f896414f9d0 0x7f896406e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33354/0 (socket says 172.21.15.7:33354) 2023-03-03T17:30:19.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f89690ff700 1 -- 172.21.15.7:0/3075615428 learned_addr learned my addr 172.21.15.7:0/3075615428 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:30:19.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f896a101700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f896414fda0 0x7f89641eb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:19.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f8969900700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89640abe20 0x7f896406f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:19.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f89690ff700 1 -- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f896414fda0 msgr2=0x7f89641eb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:19.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f89690ff700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f896414fda0 0x7f89641eb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:19.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f89690ff700 1 -- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89640abe20 msgr2=0x7f896406f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:19.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f89690ff700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89640abe20 0x7f896406f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:19.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f89690ff700 1 -- 172.21.15.7:0/3075615428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8954009b30 con 0x7f896414f9d0 2023-03-03T17:30:19.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.955+0000 7f89690ff700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f896414f9d0 0x7f896406e6d0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f895800b500 tx=0x7f895800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:30:19.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.956+0000 7f89627fc700 1 -- 172.21.15.7:0/3075615428 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958011830 con 0x7f896414f9d0 2023-03-03T17:30:19.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.956+0000 7f89627fc700 1 -- 172.21.15.7:0/3075615428 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8958011990 con 0x7f896414f9d0 2023-03-03T17:30:19.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.956+0000 7f896bb64700 1 -- 172.21.15.7:0/3075615428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89640ac350 con 0x7f896414f9d0 2023-03-03T17:30:19.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.956+0000 7f89627fc700 1 -- 172.21.15.7:0/3075615428 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958011d40 con 0x7f896414f9d0 2023-03-03T17:30:19.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.956+0000 7f896bb64700 1 -- 172.21.15.7:0/3075615428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89640ab630 con 0x7f896414f9d0 2023-03-03T17:30:19.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.957+0000 7f896bb64700 1 -- 172.21.15.7:0/3075615428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89640c3e10 con 0x7f896414f9d0 2023-03-03T17:30:19.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.957+0000 7f89627fc700 1 -- 172.21.15.7:0/3075615428 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8958021070 con 0x7f896414f9d0 2023-03-03T17:30:19.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.958+0000 7f89627fc700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89500682f0 0x7f895006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:19.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.958+0000 7f89627fc700 1 -- 172.21.15.7:0/3075615428 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8958086060 con 0x7f896414f9d0 2023-03-03T17:30:19.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.958+0000 7f8969900700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89500682f0 0x7f895006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:19.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.958+0000 7f8969900700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89500682f0 0x7f895006a7a0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f895c009990 tx=0x7f895c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:30:19.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:19.962+0000 7f89627fc700 1 -- 172.21.15.7:0/3075615428 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8958052400 con 0x7f896414f9d0 2023-03-03T17:30:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.200+0000 7f896bb64700 1 -- 172.21.15.7:0/3075615428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89641ead20 con 0x7f896414f9d0 2023-03-03T17:30:20.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.201+0000 7f89627fc700 1 -- 172.21.15.7:0/3075615428 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f89641ead20 con 0x7f896414f9d0 2023-03-03T17:30:20.201 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:30:20.201 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:30:20.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 -- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89500682f0 msgr2=0x7f895006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:20.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89500682f0 0x7f895006a7a0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f895c009990 tx=0x7f895c008040 comp rx=0 tx=0).stop 2023-03-03T17:30:20.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 -- 172.21.15.7:0/3075615428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f896414f9d0 msgr2=0x7f896406e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:20.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f896414f9d0 0x7f896406e6d0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f895800b500 tx=0x7f895800b8c0 comp rx=0 tx=0).stop 2023-03-03T17:30:20.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 -- 172.21.15.7:0/3075615428 shutdown_connections 2023-03-03T17:30:20.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f896414fda0 0x7f89641eb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:20.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f89640abe20 0x7f896406f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:20.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89500682f0 0x7f895006a7a0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:20.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 --2- 172.21.15.7:0/3075615428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f896414f9d0 0x7f896406e6d0 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:20.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 -- 172.21.15.7:0/3075615428 >> 172.21.15.7:0/3075615428 conn(0x7f89640c8900 msgr2=0x7f8964150510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:30:20.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 -- 172.21.15.7:0/3075615428 shutdown_connections 2023-03-03T17:30:20.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:20.203+0000 7f896bb64700 1 -- 172.21.15.7:0/3075615428 wait complete. 2023-03-03T17:30:20.207 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:30:20.216 INFO:tasks.mds_thrash.fs.[b]:waiting for 104 secs before thrashing 2023-03-03T17:30:22.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:30:22.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.274+0000 7f3807e70700 1 Processor -- start 2023-03-03T17:30:22.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.274+0000 7f3807e70700 1 -- start start 2023-03-03T17:30:22.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.275+0000 7f3807e70700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3800155390 0x7f38000c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:22.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.275+0000 7f3807e70700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3800154910 con 0x7f3800151770 2023-03-03T17:30:22.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.275+0000 7f3807e70700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3800152300 con 0x7f380006e020 2023-03-03T17:30:22.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.275+0000 7f3807e70700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3800070b90 con 0x7f3800155390 2023-03-03T17:30:22.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.275+0000 7f380640d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3800151770 0x7f3800151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55576/0 (socket says 172.21.15.7:55576) 2023-03-03T17:30:22.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.275+0000 7f380640d700 1 -- 172.21.15.7:0/3677197211 learned_addr learned my addr 172.21.15.7:0/3677197211 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:30:22.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.275+0000 7f3805c0c700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3800155390 0x7f38000c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:22.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.275+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730969324 0 0) 0x7f3800154910 con 0x7f3800151770 2023-03-03T17:30:22.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.275+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ec0035b0 con 0x7f3800151770 2023-03-03T17:30:22.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680040142 0 0) 0x7f3800152300 con 0x7f380006e020 2023-03-03T17:30:22.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3800154910 con 0x7f380006e020 2023-03-03T17:30:22.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3047921367 0 0) 0x7f37ec0035b0 con 0x7f3800151770 2023-03-03T17:30:22.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3800155390 msgr2=0x7f38000c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:30:22.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3800155390 0x7f38000c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:22.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 >> v1:172.21.15.143:6789/0 conn(0x7f380006e020 legacy=0x7f3800071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:22.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38000702a0 con 0x7f3800151770 2023-03-03T17:30:22.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f37f80019f0 con 0x7f3800151770 2023-03-03T17:30:22.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 >> v1:172.21.15.7:6789/0 conn(0x7f3800151770 legacy=0x7f3800151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:22.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3800155390 0x7f37ec003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:22.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37ec004190 0x7f37ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:22.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ec004d90 0x7f37ec005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:22.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3800070b90 con 0x7f37ec004d90 2023-03-03T17:30:22.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec00bbd0 con 0x7f37ec004190 2023-03-03T17:30:22.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec00bd90 con 0x7f3800155390 2023-03-03T17:30:22.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f380640d700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ec004d90 0x7f37ec005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:22.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f3805c0c700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3800155390 0x7f37ec003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:22.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f380540b700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37ec004190 0x7f37ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:22.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f380640d700 1 -- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3800155390 msgr2=0x7f37ec003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:30:22.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f380640d700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3800155390 0x7f37ec003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:22.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f380640d700 1 -- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37ec004190 msgr2=0x7f37ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:30:22.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f380640d700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37ec004190 0x7f37ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:22.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f380640d700 1 -- 172.21.15.7:0/3677197211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec0035b0 con 0x7f37ec004d90 2023-03-03T17:30:22.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.276+0000 7f380640d700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ec004d90 0x7f37ec005200 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7f37f800a7b0 tx=0x7f37ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96add098fffefd54 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:30:22.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.277+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f8012600 con 0x7f37ec004d90 2023-03-03T17:30:22.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.277+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f8012780 con 0x7f37ec004d90 2023-03-03T17:30:22.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.277+0000 7f3804c0a700 1 -- 172.21.15.7:0/3677197211 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f8012b30 con 0x7f37ec004d90 2023-03-03T17:30:22.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.277+0000 7f3807e70700 1 -- 172.21.15.7:0/3677197211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ec004d90 msgr2=0x7f37ec005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:22.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.277+0000 7f3807e70700 1 --2- 172.21.15.7:0/3677197211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ec004d90 0x7f37ec005200 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7f37f800a7b0 tx=0x7f37ec005880 comp rx=0 tx=0).stop 2023-03-03T17:30:22.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.277+0000 7f380640d700 1 -- 172.21.15.7:0/3677197211 reap_dead start 2023-03-03T17:30:22.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.277+0000 7f3807e70700 1 -- 172.21.15.7:0/3677197211 shutdown_connections 2023-03-03T17:30:22.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.277+0000 7f3807e70700 1 -- 172.21.15.7:0/3677197211 >> 172.21.15.7:0/3677197211 conn(0x7f380006c110 msgr2=0x7f380006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:30:22.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.277+0000 7f3807e70700 1 -- 172.21.15.7:0/3677197211 shutdown_connections 2023-03-03T17:30:22.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.277+0000 7f3807e70700 1 -- 172.21.15.7:0/3677197211 wait complete. 2023-03-03T17:30:22.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.278+0000 7f3807e70700 1 Processor -- start 2023-03-03T17:30:22.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f3807e70700 1 -- start start 2023-03-03T17:30:22.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f3807e70700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37ec004d90 0x7f38001e89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:22.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f3807e70700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38001e7fb0 0x7f38000ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:22.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f3807e70700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38001eb2b0 0x7f3800147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:22.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f3807e70700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38001eaaf0 con 0x7f38001eb2b0 2023-03-03T17:30:22.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f3807e70700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3800150cc0 con 0x7f37ec004d90 2023-03-03T17:30:22.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f3807e70700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3800150fa0 con 0x7f38001e7fb0 2023-03-03T17:30:22.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f380640d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38001eb2b0 0x7f3800147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:22.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f380640d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38001eb2b0 0x7f3800147ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33376/0 (socket says 172.21.15.7:33376) 2023-03-03T17:30:22.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f380640d700 1 -- 172.21.15.7:0/1747746911 learned_addr learned my addr 172.21.15.7:0/1747746911 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:30:22.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f3805c0c700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37ec004d90 0x7f38001e89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:22.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.279+0000 7f380540b700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38001e7fb0 0x7f38000ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:22.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f380640d700 1 -- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38001e7fb0 msgr2=0x7f38000ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:22.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f380640d700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38001e7fb0 0x7f38000ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:22.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f380640d700 1 -- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37ec004d90 msgr2=0x7f38001e89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:22.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f380640d700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37ec004d90 0x7f38001e89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f380640d700 1 -- 172.21.15.7:0/1747746911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3800070820 con 0x7f38001eb2b0 2023-03-03T17:30:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f380640d700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38001eb2b0 0x7f3800147ca0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7f37f8012000 tx=0x7f380006d120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:30:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f37f67fc700 1 -- 172.21.15.7:0/1747746911 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f801e600 con 0x7f38001eb2b0 2023-03-03T17:30:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f37f67fc700 1 -- 172.21.15.7:0/1747746911 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f801e760 con 0x7f38001eb2b0 2023-03-03T17:30:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f3807e70700 1 -- 172.21.15.7:0/1747746911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3800070540 con 0x7f38001eb2b0 2023-03-03T17:30:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f37f67fc700 1 -- 172.21.15.7:0/1747746911 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f801eb10 con 0x7f38001eb2b0 2023-03-03T17:30:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.280+0000 7f3807e70700 1 -- 172.21.15.7:0/1747746911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3800155760 con 0x7f38001eb2b0 2023-03-03T17:30:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.281+0000 7f3807e70700 1 -- 172.21.15.7:0/1747746911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3800074bb0 con 0x7f38001eb2b0 2023-03-03T17:30:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.281+0000 7f37f67fc700 1 -- 172.21.15.7:0/1747746911 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f37f8024030 con 0x7f38001eb2b0 2023-03-03T17:30:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.282+0000 7f37f67fc700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37e4068320 0x7f37e406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.282+0000 7f37f67fc700 1 -- 172.21.15.7:0/1747746911 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f37f80881e0 con 0x7f38001eb2b0 2023-03-03T17:30:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.282+0000 7f3805c0c700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37e4068320 0x7f37e406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.283+0000 7f3805c0c700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37e4068320 0x7f37e406a7d0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009c50 tx=0x7f37f0009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:30:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.286+0000 7f37f67fc700 1 -- 172.21.15.7:0/1747746911 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f37f8054630 con 0x7f38001eb2b0 2023-03-03T17:30:22.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.524+0000 7f3807e70700 1 -- 172.21.15.7:0/1747746911 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38001dd7d0 con 0x7f38001eb2b0 2023-03-03T17:30:22.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.524+0000 7f37f67fc700 1 -- 172.21.15.7:0/1747746911 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f37f802c090 con 0x7f38001eb2b0 2023-03-03T17:30:22.524 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:30:22.524 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:30:22.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 -- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37e4068320 msgr2=0x7f37e406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:22.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37e4068320 0x7f37e406a7d0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009c50 tx=0x7f37f0009210 comp rx=0 tx=0).stop 2023-03-03T17:30:22.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 -- 172.21.15.7:0/1747746911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38001eb2b0 msgr2=0x7f3800147ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:22.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38001eb2b0 0x7f3800147ca0 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7f37f8012000 tx=0x7f380006d120 comp rx=0 tx=0).stop 2023-03-03T17:30:22.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 -- 172.21.15.7:0/1747746911 shutdown_connections 2023-03-03T17:30:22.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37e4068320 0x7f37e406a7d0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:22.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f38001eb2b0 0x7f3800147ca0 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:22.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37ec004d90 0x7f38001e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:22.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 --2- 172.21.15.7:0/1747746911 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f38001e7fb0 0x7f38000ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:22.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 -- 172.21.15.7:0/1747746911 >> 172.21.15.7:0/1747746911 conn(0x7f380006c110 msgr2=0x7f380006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:30:22.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 -- 172.21.15.7:0/1747746911 shutdown_connections 2023-03-03T17:30:22.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:22.526+0000 7f3807e70700 1 -- 172.21.15.7:0/1747746911 wait complete. 2023-03-03T17:30:22.528 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:30:22.538 INFO:tasks.mds_thrash.fs.[a]:waiting for 29 secs before thrashing 2023-03-03T17:30:27.041 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:27.045 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:51.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:30:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.679+0000 7f10ae94d700 1 Processor -- start 2023-03-03T17:30:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.679+0000 7f10ae94d700 1 -- start start 2023-03-03T17:30:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10ae94d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f10a80c1da0 0x7f10a81550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10ae94d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f10a80c3e30 0x7f10a80c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10ae94d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a80c7a60 con 0x7f10a806d870 2023-03-03T17:30:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10ae94d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a80c2af0 con 0x7f10a80c1da0 2023-03-03T17:30:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10ae94d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a80c3620 con 0x7f10a80c3e30 2023-03-03T17:30:51.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10a77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f10a80c1da0 0x7f10a81550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10a77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f10a80c1da0 0x7f10a81550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33408/0 (socket says 172.21.15.7:33408) 2023-03-03T17:30:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10a77fe700 1 -- 172.21.15.7:0/4075569460 learned_addr learned my addr 172.21.15.7:0/4075569460 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:30:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10aceea700 1 --2- 172.21.15.7:0/4075569460 >> v2:172.21.15.143:3300/0 conn(0x7f10a80c3e30 0x7f10a80c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10a77fe700 1 -- 172.21.15.7:0/4075569460 >> v2:172.21.15.143:3300/0 conn(0x7f10a80c3e30 msgr2=0x7f10a80c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:30:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10a77fe700 1 --2- 172.21.15.7:0/4075569460 >> v2:172.21.15.143:3300/0 conn(0x7f10a80c3e30 0x7f10a80c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10a77fe700 1 -- 172.21.15.7:0/4075569460 >> v1:172.21.15.143:6789/0 conn(0x7f10a806d870 legacy=0x7f10a80c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.680+0000 7f10a77fe700 1 -- 172.21.15.7:0/4075569460 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a81545e0 con 0x7f10a80c1da0 2023-03-03T17:30:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.681+0000 7f10a77fe700 1 --2- 172.21.15.7:0/4075569460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a80c1da0 0x7f10a81550c0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f10a8073830 tx=0x7f1098009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43aaba80eccaf6a6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:30:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.681+0000 7f10a6ffd700 1 -- 172.21.15.7:0/4075569460 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109800b870 con 0x7f10a80c1da0 2023-03-03T17:30:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.681+0000 7f10a6ffd700 1 -- 172.21.15.7:0/4075569460 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109800b9d0 con 0x7f10a80c1da0 2023-03-03T17:30:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.681+0000 7f10a6ffd700 1 -- 172.21.15.7:0/4075569460 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109800bd60 con 0x7f10a80c1da0 2023-03-03T17:30:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.681+0000 7f10ae94d700 1 -- 172.21.15.7:0/4075569460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a80c1da0 msgr2=0x7f10a81550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.681+0000 7f10ae94d700 1 --2- 172.21.15.7:0/4075569460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a80c1da0 0x7f10a81550c0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f10a8073830 tx=0x7f1098009bd0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.681+0000 7f10ae94d700 1 -- 172.21.15.7:0/4075569460 shutdown_connections 2023-03-03T17:30:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.681+0000 7f10ae94d700 1 --2- 172.21.15.7:0/4075569460 >> v2:172.21.15.143:3300/0 conn(0x7f10a80c3e30 0x7f10a80c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.681+0000 7f10ae94d700 1 --2- 172.21.15.7:0/4075569460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a80c1da0 0x7f10a81550c0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.681+0000 7f10ae94d700 1 -- 172.21.15.7:0/4075569460 >> 172.21.15.7:0/4075569460 conn(0x7f10a806c110 msgr2=0x7f10a806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:30:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.682+0000 7f10ae94d700 1 -- 172.21.15.7:0/4075569460 shutdown_connections 2023-03-03T17:30:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.682+0000 7f10ae94d700 1 -- 172.21.15.7:0/4075569460 wait complete. 2023-03-03T17:30:51.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.683+0000 7f10ae94d700 1 Processor -- start 2023-03-03T17:30:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.683+0000 7f10ae94d700 1 -- start start 2023-03-03T17:30:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.684+0000 7f10ae94d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a806d870 0x7f10a81e9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.684+0000 7f10ae94d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a80c1da0 0x7f10a81e9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.684+0000 7f10ae94d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10a80c3e30 0x7f10a8142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.684+0000 7f10ae94d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a80c8180 con 0x7f10a806d870 2023-03-03T17:30:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.684+0000 7f10ae94d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a81ea480 con 0x7f10a80c3e30 2023-03-03T17:30:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.684+0000 7f10ae94d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a814d3e0 con 0x7f10a80c1da0 2023-03-03T17:30:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.684+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a806d870 0x7f10a81e9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.684+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a806d870 0x7f10a81e9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33412/0 (socket says 172.21.15.7:33412) 2023-03-03T17:30:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.684+0000 7f10a7fff700 1 -- 172.21.15.7:0/483541808 learned_addr learned my addr 172.21.15.7:0/483541808 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:30:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10aceea700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10a80c3e30 0x7f10a8142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a77fe700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a80c1da0 0x7f10a81e9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a7fff700 1 -- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a80c1da0 msgr2=0x7f10a81e9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a7fff700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a80c1da0 0x7f10a81e9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a7fff700 1 -- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10a80c3e30 msgr2=0x7f10a8142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a7fff700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10a80c3e30 0x7f10a8142800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a7fff700 1 -- 172.21.15.7:0/483541808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1098009550 con 0x7f10a806d870 2023-03-03T17:30:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a77fe700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a80c1da0 0x7f10a81e9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:30:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a7fff700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a806d870 0x7f10a81e9a00 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f109000c4e0 tx=0x7f109000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:30:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a4ff9700 1 -- 172.21.15.7:0/483541808 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10900052d0 con 0x7f10a806d870 2023-03-03T17:30:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a4ff9700 1 -- 172.21.15.7:0/483541808 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1090005430 con 0x7f10a806d870 2023-03-03T17:30:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10ae94d700 1 -- 172.21.15.7:0/483541808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a8154ba0 con 0x7f10a806d870 2023-03-03T17:30:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10a4ff9700 1 -- 172.21.15.7:0/483541808 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10900038c0 con 0x7f10a806d870 2023-03-03T17:30:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.685+0000 7f10ae94d700 1 -- 172.21.15.7:0/483541808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a81548c0 con 0x7f10a806d870 2023-03-03T17:30:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.686+0000 7f10ae94d700 1 -- 172.21.15.7:0/483541808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a806dc40 con 0x7f10a806d870 2023-03-03T17:30:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.686+0000 7f10a4ff9700 1 -- 172.21.15.7:0/483541808 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1090019030 con 0x7f10a806d870 2023-03-03T17:30:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.687+0000 7f10a4ff9700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10880682f0 0x7f108806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:30:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.687+0000 7f10a4ff9700 1 -- 172.21.15.7:0/483541808 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1090085480 con 0x7f10a806d870 2023-03-03T17:30:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.687+0000 7f10a77fe700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10880682f0 0x7f108806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:30:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.688+0000 7f10a77fe700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10880682f0 0x7f108806a7a0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f1098000c00 tx=0x7f1098008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:30:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.691+0000 7f10a4ff9700 1 -- 172.21.15.7:0/483541808 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1090051820 con 0x7f10a806d870 2023-03-03T17:30:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.928+0000 7f10ae94d700 1 -- 172.21.15.7:0/483541808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10a8074bb0 con 0x7f10a806d870 2023-03-03T17:30:51.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.929+0000 7f10a4ff9700 1 -- 172.21.15.7:0/483541808 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1090054e40 con 0x7f10a806d870 2023-03-03T17:30:51.929 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:30:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:30:51.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 -- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10880682f0 msgr2=0x7f108806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:51.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10880682f0 0x7f108806a7a0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f1098000c00 tx=0x7f1098008030 comp rx=0 tx=0).stop 2023-03-03T17:30:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 -- 172.21.15.7:0/483541808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a806d870 msgr2=0x7f10a81e9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:30:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a806d870 0x7f10a81e9a00 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f109000c4e0 tx=0x7f109000c7f0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 -- 172.21.15.7:0/483541808 shutdown_connections 2023-03-03T17:30:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10a80c3e30 0x7f10a8142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f10880682f0 0x7f108806a7a0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10a806d870 0x7f10a81e9a00 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 --2- 172.21.15.7:0/483541808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10a80c1da0 0x7f10a81e9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:30:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 -- 172.21.15.7:0/483541808 >> 172.21.15.7:0/483541808 conn(0x7f10a806c110 msgr2=0x7f10a806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:30:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 -- 172.21.15.7:0/483541808 shutdown_connections 2023-03-03T17:30:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:30:51.931+0000 7f10ae94d700 1 -- 172.21.15.7:0/483541808 wait complete. 2023-03-03T17:30:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:30:51.943 INFO:tasks.mds_thrash.fs.[a]:waiting for 116 secs before thrashing 2023-03-03T17:30:57.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:30:57.109 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:27.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:27.175 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:57.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:31:57.253 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:04.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:32:04.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.357+0000 7fe2896c4700 1 Processor -- start 2023-03-03T17:32:04.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.357+0000 7fe2896c4700 1 -- start start 2023-03-03T17:32:04.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2896c4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe284151de0 0x7fe2840c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:04.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2896c4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2840c5140 0x7fe28406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:04.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2896c4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28406dad0 con 0x7fe2840c1a40 2023-03-03T17:32:04.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2896c4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe28406e370 con 0x7fe2840c5140 2023-03-03T17:32:04.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2896c4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28406ddb0 con 0x7fe284151de0 2023-03-03T17:32:04.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2837fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2840c5140 0x7fe28406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:04.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2827fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe2840c1a40 0x7fe284157ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55664/0 (socket says 172.21.15.7:55664) 2023-03-03T17:32:04.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2827fc700 1 -- 172.21.15.7:0/1364827688 learned_addr learned my addr 172.21.15.7:0/1364827688 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:32:04.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe282ffd700 1 --2- 172.21.15.7:0/1364827688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe284151de0 0x7fe2840c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:04.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2837fe700 1 -- 172.21.15.7:0/1364827688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe284151de0 msgr2=0x7fe2840c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:32:04.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2837fe700 1 --2- 172.21.15.7:0/1364827688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe284151de0 0x7fe2840c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:04.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2837fe700 1 -- 172.21.15.7:0/1364827688 >> v1:172.21.15.7:6789/0 conn(0x7fe2840c1a40 legacy=0x7fe284157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe281ffb700 1 -- 172.21.15.7:0/1364827688 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630490472 0 0) 0x7fe28406dad0 con 0x7fe2840c1a40 2023-03-03T17:32:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.358+0000 7fe2837fe700 1 -- 172.21.15.7:0/1364827688 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28406e090 con 0x7fe2840c5140 2023-03-03T17:32:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe2837fe700 1 --2- 172.21.15.7:0/1364827688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c5140 0x7fe28406a2b0 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009d90 tx=0x7fe27c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1530b9860f064f2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:32:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe281ffb700 1 -- 172.21.15.7:0/1364827688 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c00e070 con 0x7fe2840c5140 2023-03-03T17:32:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe281ffb700 1 -- 172.21.15.7:0/1364827688 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27c00bb20 con 0x7fe2840c5140 2023-03-03T17:32:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe281ffb700 1 -- 172.21.15.7:0/1364827688 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c012430 con 0x7fe2840c5140 2023-03-03T17:32:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe2896c4700 1 -- 172.21.15.7:0/1364827688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c5140 msgr2=0x7fe28406a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:04.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe2896c4700 1 --2- 172.21.15.7:0/1364827688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c5140 0x7fe28406a2b0 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009d90 tx=0x7fe27c00b330 comp rx=0 tx=0).stop 2023-03-03T17:32:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe2896c4700 1 -- 172.21.15.7:0/1364827688 shutdown_connections 2023-03-03T17:32:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe2896c4700 1 --2- 172.21.15.7:0/1364827688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c5140 0x7fe28406a2b0 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe2896c4700 1 --2- 172.21.15.7:0/1364827688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe284151de0 0x7fe2840c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:04.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe2896c4700 1 -- 172.21.15.7:0/1364827688 >> 172.21.15.7:0/1364827688 conn(0x7fe2840a9430 msgr2=0x7fe2840c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:32:04.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe2896c4700 1 -- 172.21.15.7:0/1364827688 shutdown_connections 2023-03-03T17:32:04.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.359+0000 7fe2896c4700 1 -- 172.21.15.7:0/1364827688 wait complete. 2023-03-03T17:32:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.361+0000 7fe2896c4700 1 Processor -- start 2023-03-03T17:32:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.361+0000 7fe2896c4700 1 -- start start 2023-03-03T17:32:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe2896c4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c1a40 0x7fe284150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe2896c4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2840c5140 0x7fe28414c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe2896c4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe284151de0 0x7fe28414ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:04.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe2896c4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28406d3c0 con 0x7fe2840c1a40 2023-03-03T17:32:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe2896c4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2841ed910 con 0x7fe284151de0 2023-03-03T17:32:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe2896c4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2841ed670 con 0x7fe2840c5140 2023-03-03T17:32:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe282ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c1a40 0x7fe284150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe282ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c1a40 0x7fe284150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33458/0 (socket says 172.21.15.7:33458) 2023-03-03T17:32:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe282ffd700 1 -- 172.21.15.7:0/3050997978 learned_addr learned my addr 172.21.15.7:0/3050997978 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:32:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe2827fc700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2840c5140 0x7fe28414c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:04.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.362+0000 7fe2837fe700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe284151de0 0x7fe28414ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe282ffd700 1 -- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2840c5140 msgr2=0x7fe28414c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe282ffd700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2840c5140 0x7fe28414c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe282ffd700 1 -- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe284151de0 msgr2=0x7fe28414ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe282ffd700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe284151de0 0x7fe28414ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe282ffd700 1 -- 172.21.15.7:0/3050997978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c009a70 con 0x7fe2840c1a40 2023-03-03T17:32:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe282ffd700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c1a40 0x7fe284150a70 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7fe27800e9e0 tx=0x7fe27800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:32:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe26b7fe700 1 -- 172.21.15.7:0/3050997978 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe278009710 con 0x7fe2840c1a40 2023-03-03T17:32:04.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe26b7fe700 1 -- 172.21.15.7:0/3050997978 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27800c850 con 0x7fe2840c1a40 2023-03-03T17:32:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe2896c4700 1 -- 172.21.15.7:0/3050997978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe284066de0 con 0x7fe2840c1a40 2023-03-03T17:32:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe26b7fe700 1 -- 172.21.15.7:0/3050997978 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27800cbc0 con 0x7fe2840c1a40 2023-03-03T17:32:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.363+0000 7fe2896c4700 1 -- 172.21.15.7:0/3050997978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2840c4c20 con 0x7fe2840c1a40 2023-03-03T17:32:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.364+0000 7fe2896c4700 1 -- 172.21.15.7:0/3050997978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2841582b0 con 0x7fe2840c1a40 2023-03-03T17:32:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.364+0000 7fe26b7fe700 1 -- 172.21.15.7:0/3050997978 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe278019030 con 0x7fe2840c1a40 2023-03-03T17:32:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.365+0000 7fe26b7fe700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2700682f0 0x7fe27006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:04.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.365+0000 7fe26b7fe700 1 -- 172.21.15.7:0/3050997978 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe278086110 con 0x7fe2840c1a40 2023-03-03T17:32:04.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.365+0000 7fe2827fc700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2700682f0 0x7fe27006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:04.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.365+0000 7fe2827fc700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2700682f0 0x7fe27006a7a0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fe28406b850 tx=0x7fe274007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:32:04.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.369+0000 7fe26b7fe700 1 -- 172.21.15.7:0/3050997978 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe278052420 con 0x7fe2840c1a40 2023-03-03T17:32:04.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.607+0000 7fe2896c4700 1 -- 172.21.15.7:0/3050997978 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe28414c650 con 0x7fe2840c1a40 2023-03-03T17:32:04.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.608+0000 7fe26b7fe700 1 -- 172.21.15.7:0/3050997978 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe278017090 con 0x7fe2840c1a40 2023-03-03T17:32:04.608 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:32:04.609 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:32:04.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 -- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2700682f0 msgr2=0x7fe27006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:04.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2700682f0 0x7fe27006a7a0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fe28406b850 tx=0x7fe274007e70 comp rx=0 tx=0).stop 2023-03-03T17:32:04.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 -- 172.21.15.7:0/3050997978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c1a40 msgr2=0x7fe284150a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:04.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c1a40 0x7fe284150a70 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7fe27800e9e0 tx=0x7fe27800ecf0 comp rx=0 tx=0).stop 2023-03-03T17:32:04.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 -- 172.21.15.7:0/3050997978 shutdown_connections 2023-03-03T17:32:04.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe284151de0 0x7fe28414ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:04.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2700682f0 0x7fe27006a7a0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:04.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2840c1a40 0x7fe284150a70 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:04.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 --2- 172.21.15.7:0/3050997978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2840c5140 0x7fe28414c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:04.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 -- 172.21.15.7:0/3050997978 >> 172.21.15.7:0/3050997978 conn(0x7fe2840a9430 msgr2=0x7fe28414ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:32:04.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 -- 172.21.15.7:0/3050997978 shutdown_connections 2023-03-03T17:32:04.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:04.610+0000 7fe2896c4700 1 -- 172.21.15.7:0/3050997978 wait complete. 2023-03-03T17:32:04.612 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:32:04.623 INFO:tasks.mds_thrash.fs.[b]:waiting for 36 secs before thrashing 2023-03-03T17:32:27.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:27.340 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:40.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:32:40.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.764+0000 7f407d70b700 1 Processor -- start 2023-03-03T17:32:40.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.764+0000 7f407d70b700 1 -- start start 2023-03-03T17:32:40.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.764+0000 7f407d70b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40780c1850 0x7f40780c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:40.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.764+0000 7f407d70b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4078154790 con 0x7f40780aa170 2023-03-03T17:32:40.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.764+0000 7f407d70b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4078154a70 con 0x7f407814b060 2023-03-03T17:32:40.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.764+0000 7f407d70b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40780c4100 con 0x7f40780c1850 2023-03-03T17:32:40.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f40767fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f40780c1850 0x7f40780c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f40777fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f40780aa170 0x7f40780aa570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55728/0 (socket says 172.21.15.7:55728) 2023-03-03T17:32:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f40777fe700 1 -- 172.21.15.7:0/2713838049 learned_addr learned my addr 172.21.15.7:0/2713838049 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:32:40.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f40767fc700 1 -- 172.21.15.7:0/2713838049 >> v1:172.21.15.143:6789/0 conn(0x7f407814b060 legacy=0x7f407806f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:40.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f40767fc700 1 -- 172.21.15.7:0/2713838049 >> v1:172.21.15.7:6789/0 conn(0x7f40780aa170 legacy=0x7f40780aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:40.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f40767fc700 1 -- 172.21.15.7:0/2713838049 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4078154d50 con 0x7f40780c1850 2023-03-03T17:32:40.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f4075ffb700 1 -- 172.21.15.7:0/2713838049 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211471035 0 0) 0x7f4078154790 con 0x7f40780aa170 2023-03-03T17:32:40.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f40767fc700 1 --2- 172.21.15.7:0/2713838049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780c1850 0x7f40780c1ca0 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f406c009840 tx=0x7f406c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f32af36f4502c8d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:32:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f4075ffb700 1 -- 172.21.15.7:0/2713838049 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c011830 con 0x7f40780c1850 2023-03-03T17:32:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f4075ffb700 1 -- 172.21.15.7:0/2713838049 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406c011990 con 0x7f40780c1850 2023-03-03T17:32:40.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.765+0000 7f4075ffb700 1 -- 172.21.15.7:0/2713838049 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c011d20 con 0x7f40780c1850 2023-03-03T17:32:40.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.766+0000 7f407d70b700 1 -- 172.21.15.7:0/2713838049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780c1850 msgr2=0x7f40780c1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:40.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.766+0000 7f407d70b700 1 --2- 172.21.15.7:0/2713838049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780c1850 0x7f40780c1ca0 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f406c009840 tx=0x7f406c009b50 comp rx=0 tx=0).stop 2023-03-03T17:32:40.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.766+0000 7f407d70b700 1 -- 172.21.15.7:0/2713838049 shutdown_connections 2023-03-03T17:32:40.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.766+0000 7f407d70b700 1 --2- 172.21.15.7:0/2713838049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780c1850 0x7f40780c1ca0 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:40.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.766+0000 7f407d70b700 1 -- 172.21.15.7:0/2713838049 >> 172.21.15.7:0/2713838049 conn(0x7f40780c4f20 msgr2=0x7f40780c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:32:40.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.766+0000 7f407d70b700 1 -- 172.21.15.7:0/2713838049 shutdown_connections 2023-03-03T17:32:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.766+0000 7f407d70b700 1 -- 172.21.15.7:0/2713838049 wait complete. 2023-03-03T17:32:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.767+0000 7f407d70b700 1 Processor -- start 2023-03-03T17:32:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.767+0000 7f407d70b700 1 -- start start 2023-03-03T17:32:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f407d70b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780aa170 0x7f4078151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f407d70b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40780c1850 0x7f407806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f407d70b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f407814b060 0x7f4078144f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f407d70b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407806a800 con 0x7f40780aa170 2023-03-03T17:32:40.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f407d70b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407814fb40 con 0x7f40780c1850 2023-03-03T17:32:40.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f407d70b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407806aa50 con 0x7f407814b060 2023-03-03T17:32:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f4076ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780aa170 0x7f4078151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f4076ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780aa170 0x7f4078151520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33518/0 (socket says 172.21.15.7:33518) 2023-03-03T17:32:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f4076ffd700 1 -- 172.21.15.7:0/2288585230 learned_addr learned my addr 172.21.15.7:0/2288585230 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:32:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f40777fe700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f407814b060 0x7f4078144f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:40.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f40767fc700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40780c1850 0x7f407806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f4076ffd700 1 -- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f407814b060 msgr2=0x7f4078144f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f4076ffd700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f407814b060 0x7f4078144f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f4076ffd700 1 -- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40780c1850 msgr2=0x7f407806ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f4076ffd700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40780c1850 0x7f407806ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.768+0000 7f4076ffd700 1 -- 172.21.15.7:0/2288585230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c0094a0 con 0x7f40780aa170 2023-03-03T17:32:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.769+0000 7f4076ffd700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780aa170 0x7f4078151520 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f406800c480 tx=0x7f406800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:32:40.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.769+0000 7f405f7fe700 1 -- 172.21.15.7:0/2288585230 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40680052d0 con 0x7f40780aa170 2023-03-03T17:32:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.769+0000 7f405f7fe700 1 -- 172.21.15.7:0/2288585230 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4068005430 con 0x7f40780aa170 2023-03-03T17:32:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.769+0000 7f407d70b700 1 -- 172.21.15.7:0/2288585230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4078155310 con 0x7f40780aa170 2023-03-03T17:32:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.769+0000 7f405f7fe700 1 -- 172.21.15.7:0/2288585230 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40680038c0 con 0x7f40780aa170 2023-03-03T17:32:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.769+0000 7f407d70b700 1 -- 172.21.15.7:0/2288585230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4078155030 con 0x7f40780aa170 2023-03-03T17:32:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.770+0000 7f407d70b700 1 -- 172.21.15.7:0/2288585230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4078074bb0 con 0x7f40780aa170 2023-03-03T17:32:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.770+0000 7f405f7fe700 1 -- 172.21.15.7:0/2288585230 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4068019030 con 0x7f40780aa170 2023-03-03T17:32:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.771+0000 7f405f7fe700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40640682f0 0x7f406406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:40.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.771+0000 7f405f7fe700 1 -- 172.21.15.7:0/2288585230 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4068085320 con 0x7f40780aa170 2023-03-03T17:32:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.771+0000 7f40767fc700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40640682f0 0x7f406406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.771+0000 7f40767fc700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40640682f0 0x7f406406a7a0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f406c0090b0 tx=0x7f406c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:32:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:40.775+0000 7f405f7fe700 1 -- 172.21.15.7:0/2288585230 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f40680516c0 con 0x7f40780aa170 2023-03-03T17:32:41.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.015+0000 7f407d70b700 1 -- 172.21.15.7:0/2288585230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40780a94c0 con 0x7f40780aa170 2023-03-03T17:32:41.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.016+0000 7f405f7fe700 1 -- 172.21.15.7:0/2288585230 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4068054ce0 con 0x7f40780aa170 2023-03-03T17:32:41.015 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:32:41.016 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:32:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 -- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40640682f0 msgr2=0x7f406406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:41.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40640682f0 0x7f406406a7a0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f406c0090b0 tx=0x7f406c008030 comp rx=0 tx=0).stop 2023-03-03T17:32:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 -- 172.21.15.7:0/2288585230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780aa170 msgr2=0x7f4078151520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780aa170 0x7f4078151520 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f406800c480 tx=0x7f406800c790 comp rx=0 tx=0).stop 2023-03-03T17:32:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 -- 172.21.15.7:0/2288585230 shutdown_connections 2023-03-03T17:32:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f407814b060 0x7f4078144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40640682f0 0x7f406406a7a0 secure :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f406c0090b0 tx=0x7f406c008030 comp rx=0 tx=0).stop 2023-03-03T17:32:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f40780aa170 0x7f4078151520 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:41.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 --2- 172.21.15.7:0/2288585230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f40780c1850 0x7f407806ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 -- 172.21.15.7:0/2288585230 >> 172.21.15.7:0/2288585230 conn(0x7f40780c4f20 msgr2=0x7f40780c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:32:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 -- 172.21.15.7:0/2288585230 shutdown_connections 2023-03-03T17:32:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:41.018+0000 7f407d70b700 1 -- 172.21.15.7:0/2288585230 wait complete. 2023-03-03T17:32:41.019 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:32:41.030 INFO:tasks.mds_thrash.fs.[b]:waiting for 22 secs before thrashing 2023-03-03T17:32:47.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:32:48.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.085+0000 7f00cb509700 1 Processor -- start 2023-03-03T17:32:48.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.085+0000 7f00cb509700 1 -- start start 2023-03-03T17:32:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.086+0000 7f00cb509700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00c4151e80 0x7f00c40c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.086+0000 7f00cb509700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00c4155410 0x7f00c40bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.086+0000 7f00cb509700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c40c08d0 con 0x7f00c40c6fe0 2023-03-03T17:32:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.086+0000 7f00cb509700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c4152250 con 0x7f00c4151e80 2023-03-03T17:32:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.086+0000 7f00cb509700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c40c0fb0 con 0x7f00c4155410 2023-03-03T17:32:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.086+0000 7f00c92a5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00c4151e80 0x7f00c40c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.086+0000 7f00c92a5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f00c4151e80 0x7f00c40c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33532/0 (socket says 172.21.15.7:33532) 2023-03-03T17:32:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.086+0000 7f00c92a5700 1 -- 172.21.15.7:0/1958361698 learned_addr learned my addr 172.21.15.7:0/1958361698 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:32:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.087+0000 7f00c8aa4700 1 --2- 172.21.15.7:0/1958361698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00c4155410 0x7f00c40bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.087+0000 7f00c92a5700 1 -- 172.21.15.7:0/1958361698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00c4155410 msgr2=0x7f00c40bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:32:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.087+0000 7f00c92a5700 1 --2- 172.21.15.7:0/1958361698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00c4155410 0x7f00c40bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.087+0000 7f00c92a5700 1 -- 172.21.15.7:0/1958361698 >> v1:172.21.15.143:6789/0 conn(0x7f00c40c6fe0 legacy=0x7f00c40bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.087+0000 7f00c92a5700 1 -- 172.21.15.7:0/1958361698 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c4154930 con 0x7f00c4151e80 2023-03-03T17:32:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.087+0000 7f00c92a5700 1 --2- 172.21.15.7:0/1958361698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4151e80 0x7f00c40c6aa0 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009920 tx=0x7f00b4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee416d3859d1425c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:32:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.087+0000 7f00bbfff700 1 -- 172.21.15.7:0/1958361698 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b400ba20 con 0x7f00c4151e80 2023-03-03T17:32:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.087+0000 7f00bbfff700 1 -- 172.21.15.7:0/1958361698 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b400bb80 con 0x7f00c4151e80 2023-03-03T17:32:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.087+0000 7f00bbfff700 1 -- 172.21.15.7:0/1958361698 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4009e70 con 0x7f00c4151e80 2023-03-03T17:32:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.088+0000 7f00cb509700 1 -- 172.21.15.7:0/1958361698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4151e80 msgr2=0x7f00c40c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.088+0000 7f00cb509700 1 --2- 172.21.15.7:0/1958361698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4151e80 0x7f00c40c6aa0 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009920 tx=0x7f00b4009c30 comp rx=0 tx=0).stop 2023-03-03T17:32:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.088+0000 7f00cb509700 1 -- 172.21.15.7:0/1958361698 shutdown_connections 2023-03-03T17:32:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.088+0000 7f00cb509700 1 --2- 172.21.15.7:0/1958361698 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4151e80 0x7f00c40c6aa0 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.088+0000 7f00cb509700 1 --2- 172.21.15.7:0/1958361698 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00c4155410 0x7f00c40bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.088+0000 7f00cb509700 1 -- 172.21.15.7:0/1958361698 >> 172.21.15.7:0/1958361698 conn(0x7f00c4071630 msgr2=0x7f00c406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:32:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.088+0000 7f00cb509700 1 -- 172.21.15.7:0/1958361698 shutdown_connections 2023-03-03T17:32:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.088+0000 7f00cb509700 1 -- 172.21.15.7:0/1958361698 wait complete. 2023-03-03T17:32:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.089+0000 7f00cb509700 1 Processor -- start 2023-03-03T17:32:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.089+0000 7f00cb509700 1 -- start start 2023-03-03T17:32:48.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00cb509700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00c40c6fe0 0x7f00c414b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00cb509700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00c4151e80 0x7f00c40bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00cb509700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4155410 0x7f00c40c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00cb509700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c414beb0 con 0x7f00c4155410 2023-03-03T17:32:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00cb509700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c40c9b10 con 0x7f00c4151e80 2023-03-03T17:32:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00cb509700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c40c9df0 con 0x7f00c40c6fe0 2023-03-03T17:32:48.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c9aa6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4155410 0x7f00c40c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c9aa6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4155410 0x7f00c40c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33540/0 (socket says 172.21.15.7:33540) 2023-03-03T17:32:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c9aa6700 1 -- 172.21.15.7:0/2749835017 learned_addr learned my addr 172.21.15.7:0/2749835017 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:32:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c8aa4700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00c4151e80 0x7f00c40bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c92a5700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00c40c6fe0 0x7f00c414b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c9aa6700 1 -- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00c40c6fe0 msgr2=0x7f00c414b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c9aa6700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00c40c6fe0 0x7f00c414b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:48.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c9aa6700 1 -- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00c4151e80 msgr2=0x7f00c40bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c9aa6700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00c4151e80 0x7f00c40bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c9aa6700 1 -- 172.21.15.7:0/2749835017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b4009550 con 0x7f00c4155410 2023-03-03T17:32:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.090+0000 7f00c9aa6700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4155410 0x7f00c40c8d20 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00cb50 tx=0x7f00bc00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:32:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.091+0000 7f00b9ffb700 1 -- 172.21.15.7:0/2749835017 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc003c00 con 0x7f00c4155410 2023-03-03T17:32:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.091+0000 7f00b9ffb700 1 -- 172.21.15.7:0/2749835017 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc003d60 con 0x7f00c4155410 2023-03-03T17:32:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.091+0000 7f00cb509700 1 -- 172.21.15.7:0/2749835017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c4154ef0 con 0x7f00c4155410 2023-03-03T17:32:48.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.091+0000 7f00b9ffb700 1 -- 172.21.15.7:0/2749835017 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc00aa20 con 0x7f00c4155410 2023-03-03T17:32:48.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.091+0000 7f00cb509700 1 -- 172.21.15.7:0/2749835017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c4154c10 con 0x7f00c4155410 2023-03-03T17:32:48.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.092+0000 7f00cb509700 1 -- 172.21.15.7:0/2749835017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c40c5220 con 0x7f00c4155410 2023-03-03T17:32:48.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.092+0000 7f00b9ffb700 1 -- 172.21.15.7:0/2749835017 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f00bc010050 con 0x7f00c4155410 2023-03-03T17:32:48.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.093+0000 7f00b9ffb700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00a80682f0 0x7f00a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:32:48.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.093+0000 7f00b9ffb700 1 -- 172.21.15.7:0/2749835017 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f00bc085ff0 con 0x7f00c4155410 2023-03-03T17:32:48.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.093+0000 7f00c92a5700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00a80682f0 0x7f00a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:32:48.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.093+0000 7f00c92a5700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00a80682f0 0x7f00a806a7a0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f00b4014040 tx=0x7f00b4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:32:48.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.097+0000 7f00b9ffb700 1 -- 172.21.15.7:0/2749835017 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f00bc052390 con 0x7f00c4155410 2023-03-03T17:32:48.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.337+0000 7f00cb509700 1 -- 172.21.15.7:0/2749835017 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00c4074bb0 con 0x7f00c4155410 2023-03-03T17:32:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.338+0000 7f00b9ffb700 1 -- 172.21.15.7:0/2749835017 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f00bc0559b0 con 0x7f00c4155410 2023-03-03T17:32:48.339 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:32:48.340 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:32:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 -- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00a80682f0 msgr2=0x7f00a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00a80682f0 0x7f00a806a7a0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f00b4014040 tx=0x7f00b4008030 comp rx=0 tx=0).stop 2023-03-03T17:32:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 -- 172.21.15.7:0/2749835017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4155410 msgr2=0x7f00c40c8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:32:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4155410 0x7f00c40c8d20 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00cb50 tx=0x7f00bc00ce60 comp rx=0 tx=0).stop 2023-03-03T17:32:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 -- 172.21.15.7:0/2749835017 shutdown_connections 2023-03-03T17:32:48.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f00a80682f0 0x7f00a806a7a0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00c4155410 0x7f00c40c8d20 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f00c40c6fe0 0x7f00c414b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 --2- 172.21.15.7:0/2749835017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00c4151e80 0x7f00c40bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:32:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 -- 172.21.15.7:0/2749835017 >> 172.21.15.7:0/2749835017 conn(0x7f00c4071630 msgr2=0x7f00c414ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:32:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 -- 172.21.15.7:0/2749835017 shutdown_connections 2023-03-03T17:32:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:32:48.340+0000 7f00cb509700 1 -- 172.21.15.7:0/2749835017 wait complete. 2023-03-03T17:32:48.342 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:32:48.352 INFO:tasks.mds_thrash.fs.[a]:waiting for 46 secs before thrashing 2023-03-03T17:32:57.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:32:57.454 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:03.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:33:03.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.170+0000 7f9634764700 1 Processor -- start 2023-03-03T17:33:03.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.170+0000 7f9634764700 1 -- start start 2023-03-03T17:33:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.171+0000 7f9634764700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f962c151cd0 0x7f962c154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.171+0000 7f9634764700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f962c06c6c0 0x7f962c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:03.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.171+0000 7f9634764700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c157ff0 con 0x7f962c0c2e30 2023-03-03T17:33:03.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.171+0000 7f9634764700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962c06d640 con 0x7f962c06c6c0 2023-03-03T17:33:03.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.171+0000 7f9634764700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962c06cda0 con 0x7f962c151cd0 2023-03-03T17:33:03.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.171+0000 7f9632d01700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f962c06c6c0 0x7f962c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:03.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.171+0000 7f9632d01700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f962c06c6c0 0x7f962c0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33556/0 (socket says 172.21.15.7:33556) 2023-03-03T17:33:03.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.171+0000 7f9632d01700 1 -- 172.21.15.7:0/6696005 learned_addr learned my addr 172.21.15.7:0/6696005 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:33:03.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.172+0000 7f9632500700 1 --2- 172.21.15.7:0/6696005 >> v2:172.21.15.143:3300/0 conn(0x7f962c151cd0 0x7f962c154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.172+0000 7f9632d01700 1 -- 172.21.15.7:0/6696005 >> v2:172.21.15.143:3300/0 conn(0x7f962c151cd0 msgr2=0x7f962c154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:33:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.172+0000 7f9632d01700 1 --2- 172.21.15.7:0/6696005 >> v2:172.21.15.143:3300/0 conn(0x7f962c151cd0 0x7f962c154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.172+0000 7f9632d01700 1 -- 172.21.15.7:0/6696005 >> v1:172.21.15.143:6789/0 conn(0x7f962c0c2e30 legacy=0x7f962c158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:33:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.172+0000 7f9632d01700 1 -- 172.21.15.7:0/6696005 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c06d080 con 0x7f962c06c6c0 2023-03-03T17:33:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.172+0000 7f9632500700 1 --2- 172.21.15.7:0/6696005 >> v2:172.21.15.143:3300/0 conn(0x7f962c151cd0 0x7f962c154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:33:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.172+0000 7f9632d01700 1 --2- 172.21.15.7:0/6696005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c06c6c0 0x7f962c0736f0 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f962c06f340 tx=0x7f962400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd911894abb8f36 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:33:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.172+0000 7f96314fe700 1 -- 172.21.15.7:0/6696005 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962400e070 con 0x7f962c06c6c0 2023-03-03T17:33:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.172+0000 7f96314fe700 1 -- 172.21.15.7:0/6696005 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9624003a20 con 0x7f962c06c6c0 2023-03-03T17:33:03.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.172+0000 7f96314fe700 1 -- 172.21.15.7:0/6696005 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624003dd0 con 0x7f962c06c6c0 2023-03-03T17:33:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.173+0000 7f9634764700 1 -- 172.21.15.7:0/6696005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c06c6c0 msgr2=0x7f962c0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.173+0000 7f9634764700 1 --2- 172.21.15.7:0/6696005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c06c6c0 0x7f962c0736f0 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f962c06f340 tx=0x7f962400b330 comp rx=0 tx=0).stop 2023-03-03T17:33:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.173+0000 7f9634764700 1 -- 172.21.15.7:0/6696005 shutdown_connections 2023-03-03T17:33:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.173+0000 7f9634764700 1 --2- 172.21.15.7:0/6696005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c06c6c0 0x7f962c0736f0 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:03.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.173+0000 7f9634764700 1 --2- 172.21.15.7:0/6696005 >> v2:172.21.15.143:3300/0 conn(0x7f962c151cd0 0x7f962c154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.173+0000 7f9634764700 1 -- 172.21.15.7:0/6696005 >> 172.21.15.7:0/6696005 conn(0x7f962c0a94d0 msgr2=0x7f962c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:33:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.173+0000 7f9634764700 1 -- 172.21.15.7:0/6696005 shutdown_connections 2023-03-03T17:33:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.173+0000 7f9634764700 1 -- 172.21.15.7:0/6696005 wait complete. 2023-03-03T17:33:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.174+0000 7f9634764700 1 Processor -- start 2023-03-03T17:33:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.174+0000 7f9634764700 1 -- start start 2023-03-03T17:33:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9634764700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f962c06c6c0 0x7f962c14ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:03.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9634764700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f962c0c2e30 0x7f962c149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9634764700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c151cd0 0x7f962c149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9634764700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c149470 con 0x7f962c151cd0 2023-03-03T17:33:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9634764700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c0c6b10 con 0x7f962c06c6c0 2023-03-03T17:33:03.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9634764700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962c0c6df0 con 0x7f962c0c2e30 2023-03-03T17:33:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9632d01700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c151cd0 0x7f962c149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9632d01700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c151cd0 0x7f962c149dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33562/0 (socket says 172.21.15.7:33562) 2023-03-03T17:33:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9632d01700 1 -- 172.21.15.7:0/3803990638 learned_addr learned my addr 172.21.15.7:0/3803990638 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:33:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9632500700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f962c06c6c0 0x7f962c14ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9631cff700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f962c0c2e30 0x7f962c149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9632d01700 1 -- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f962c0c2e30 msgr2=0x7f962c149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:03.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9632d01700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f962c0c2e30 0x7f962c149890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:03.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9632d01700 1 -- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f962c06c6c0 msgr2=0x7f962c14ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:03.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9632d01700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f962c06c6c0 0x7f962c14ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:03.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9632d01700 1 -- 172.21.15.7:0/3803990638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9624009b30 con 0x7f962c151cd0 2023-03-03T17:33:03.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9631cff700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f962c0c2e30 0x7f962c149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:33:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.175+0000 7f9632d01700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c151cd0 0x7f962c149dd0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f9624015040 tx=0x7f9624004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:33:03.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.176+0000 7f9622ffd700 1 -- 172.21.15.7:0/3803990638 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624004440 con 0x7f962c151cd0 2023-03-03T17:33:03.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.176+0000 7f9622ffd700 1 -- 172.21.15.7:0/3803990638 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96240045a0 con 0x7f962c151cd0 2023-03-03T17:33:03.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.176+0000 7f9634764700 1 -- 172.21.15.7:0/3803990638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962c157c80 con 0x7f962c151cd0 2023-03-03T17:33:03.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.176+0000 7f9622ffd700 1 -- 172.21.15.7:0/3803990638 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624004950 con 0x7f962c151cd0 2023-03-03T17:33:03.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.176+0000 7f9634764700 1 -- 172.21.15.7:0/3803990638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962c06d360 con 0x7f962c151cd0 2023-03-03T17:33:03.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.177+0000 7f9634764700 1 -- 172.21.15.7:0/3803990638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962c154a50 con 0x7f962c151cd0 2023-03-03T17:33:03.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.177+0000 7f9622ffd700 1 -- 172.21.15.7:0/3803990638 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9624022030 con 0x7f962c151cd0 2023-03-03T17:33:03.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.178+0000 7f9622ffd700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96100682f0 0x7f961006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:03.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.178+0000 7f9622ffd700 1 -- 172.21.15.7:0/3803990638 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9624087660 con 0x7f962c151cd0 2023-03-03T17:33:03.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.178+0000 7f9632500700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96100682f0 0x7f961006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:03.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.178+0000 7f9632500700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96100682f0 0x7f961006a7a0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f961c006fd0 tx=0x7f961c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:33:03.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.182+0000 7f9622ffd700 1 -- 172.21.15.7:0/3803990638 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9624053890 con 0x7f962c151cd0 2023-03-03T17:33:03.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.418+0000 7f9634764700 1 -- 172.21.15.7:0/3803990638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f962c1495b0 con 0x7f962c151cd0 2023-03-03T17:33:03.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.419+0000 7f9622ffd700 1 -- 172.21.15.7:0/3803990638 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f962c1495b0 con 0x7f962c151cd0 2023-03-03T17:33:03.420 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:33:03.420 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:33:03.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 -- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96100682f0 msgr2=0x7f961006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:03.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96100682f0 0x7f961006a7a0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f961c006fd0 tx=0x7f961c008040 comp rx=0 tx=0).stop 2023-03-03T17:33:03.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 -- 172.21.15.7:0/3803990638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c151cd0 msgr2=0x7f962c149dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:03.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c151cd0 0x7f962c149dd0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f9624015040 tx=0x7f9624004ad0 comp rx=0 tx=0).stop 2023-03-03T17:33:03.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 -- 172.21.15.7:0/3803990638 shutdown_connections 2023-03-03T17:33:03.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96100682f0 0x7f961006a7a0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:03.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f962c151cd0 0x7f962c149dd0 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:03.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f962c06c6c0 0x7f962c14ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:03.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 --2- 172.21.15.7:0/3803990638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f962c0c2e30 0x7f962c149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:03.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 -- 172.21.15.7:0/3803990638 >> 172.21.15.7:0/3803990638 conn(0x7f962c0a94d0 msgr2=0x7f962c14d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:33:03.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 -- 172.21.15.7:0/3803990638 shutdown_connections 2023-03-03T17:33:03.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:03.421+0000 7f9634764700 1 -- 172.21.15.7:0/3803990638 wait complete. 2023-03-03T17:33:03.424 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:33:03.433 INFO:tasks.mds_thrash.fs.[b]:waiting for 90 secs before thrashing 2023-03-03T17:33:27.614 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:27.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:34.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:33:34.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.493+0000 7f1090f9a700 1 Processor -- start 2023-03-03T17:33:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.493+0000 7f1090f9a700 1 -- start start 2023-03-03T17:33:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.494+0000 7f1090f9a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 0x7f108c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.494+0000 7f1090f9a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f108c0abe20 0x7f108c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.494+0000 7f1090f9a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f108c0c2f30 con 0x7f108c14f9d0 2023-03-03T17:33:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.494+0000 7f1090f9a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f108c0c31d0 con 0x7f108c0abe20 2023-03-03T17:33:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.494+0000 7f1090f9a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108c1524a0 con 0x7f108c14fda0 2023-03-03T17:33:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.494+0000 7f108ad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f108c0abe20 0x7f108c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.494+0000 7f108a59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f108c14f9d0 0x7f108c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55812/0 (socket says 172.21.15.7:55812) 2023-03-03T17:33:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.494+0000 7f108a59c700 1 -- 172.21.15.7:0/265716253 learned_addr learned my addr 172.21.15.7:0/265716253 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:33:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f1089d9b700 1 --2- 172.21.15.7:0/265716253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 0x7f108c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f108ad9d700 1 -- 172.21.15.7:0/265716253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 msgr2=0x7f108c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:33:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f108ad9d700 1 --2- 172.21.15.7:0/265716253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 0x7f108c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f108ad9d700 1 -- 172.21.15.7:0/265716253 >> v1:172.21.15.7:6789/0 conn(0x7f108c14f9d0 legacy=0x7f108c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f108ad9d700 1 -- 172.21.15.7:0/265716253 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c0c34b0 con 0x7f108c0abe20 2023-03-03T17:33:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f108959a700 1 -- 172.21.15.7:0/265716253 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457515282 0 0) 0x7f108c0c2f30 con 0x7f108c14f9d0 2023-03-03T17:33:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f1089d9b700 1 --2- 172.21.15.7:0/265716253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 0x7f108c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:33:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f108ad9d700 1 --2- 172.21.15.7:0/265716253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c0abe20 0x7f108c1529c0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f1084009d90 tx=0x7f108400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f01976546f01be4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:33:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f108959a700 1 -- 172.21.15.7:0/265716253 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108400e070 con 0x7f108c0abe20 2023-03-03T17:33:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f108959a700 1 -- 172.21.15.7:0/265716253 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108400bbd0 con 0x7f108c0abe20 2023-03-03T17:33:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.495+0000 7f108959a700 1 -- 172.21.15.7:0/265716253 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084012430 con 0x7f108c0abe20 2023-03-03T17:33:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.496+0000 7f1090f9a700 1 -- 172.21.15.7:0/265716253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c0abe20 msgr2=0x7f108c1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.496+0000 7f1090f9a700 1 --2- 172.21.15.7:0/265716253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c0abe20 0x7f108c1529c0 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f1084009d90 tx=0x7f108400b330 comp rx=0 tx=0).stop 2023-03-03T17:33:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.496+0000 7f1090f9a700 1 -- 172.21.15.7:0/265716253 shutdown_connections 2023-03-03T17:33:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.496+0000 7f1090f9a700 1 --2- 172.21.15.7:0/265716253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c0abe20 0x7f108c1529c0 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.496+0000 7f1090f9a700 1 --2- 172.21.15.7:0/265716253 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 0x7f108c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.496+0000 7f1090f9a700 1 -- 172.21.15.7:0/265716253 >> 172.21.15.7:0/265716253 conn(0x7f108c0c8900 msgr2=0x7f108c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:33:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.496+0000 7f1090f9a700 1 -- 172.21.15.7:0/265716253 shutdown_connections 2023-03-03T17:33:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.496+0000 7f1090f9a700 1 -- 172.21.15.7:0/265716253 wait complete. 2023-03-03T17:33:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.497+0000 7f1090f9a700 1 Processor -- start 2023-03-03T17:33:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.497+0000 7f1090f9a700 1 -- start start 2023-03-03T17:33:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1090f9a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c0abe20 0x7f108c06e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1090f9a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c14f9d0 0x7f108c0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1090f9a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 0x7f108c1ed350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1090f9a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c1ecec0 con 0x7f108c14f9d0 2023-03-03T17:33:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1090f9a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c1ecbe0 con 0x7f108c0abe20 2023-03-03T17:33:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1090f9a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108c06e160 con 0x7f108c14fda0 2023-03-03T17:33:34.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1089d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c14f9d0 0x7f108c0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:34.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1089d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c14f9d0 0x7f108c0c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33606/0 (socket says 172.21.15.7:33606) 2023-03-03T17:33:34.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1089d9b700 1 -- 172.21.15.7:0/916739376 learned_addr learned my addr 172.21.15.7:0/916739376 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:33:34.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f108ad9d700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 0x7f108c1ed350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:34.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f108a59c700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c0abe20 0x7f108c06e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:34.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1089d9b700 1 -- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 msgr2=0x7f108c1ed350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:34.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1089d9b700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 0x7f108c1ed350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1089d9b700 1 -- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c0abe20 msgr2=0x7f108c06e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:34.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1089d9b700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c0abe20 0x7f108c06e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f1089d9b700 1 -- 172.21.15.7:0/916739376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1084009a70 con 0x7f108c14f9d0 2023-03-03T17:33:34.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f108ad9d700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 0x7f108c1ed350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:33:34.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.498+0000 7f108a59c700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c0abe20 0x7f108c06e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:33:34.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.499+0000 7f1089d9b700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c14f9d0 0x7f108c0c1a50 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f107c00c7f0 tx=0x7f107c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:33:34.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.499+0000 7f107affd700 1 -- 172.21.15.7:0/916739376 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107c011800 con 0x7f108c14f9d0 2023-03-03T17:33:34.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.499+0000 7f107affd700 1 -- 172.21.15.7:0/916739376 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f107c011960 con 0x7f108c14f9d0 2023-03-03T17:33:34.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.499+0000 7f1090f9a700 1 -- 172.21.15.7:0/916739376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108c0ac350 con 0x7f108c14f9d0 2023-03-03T17:33:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.499+0000 7f107affd700 1 -- 172.21.15.7:0/916739376 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107c011cf0 con 0x7f108c14f9d0 2023-03-03T17:33:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.499+0000 7f1090f9a700 1 -- 172.21.15.7:0/916739376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108c0ab630 con 0x7f108c14f9d0 2023-03-03T17:33:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.500+0000 7f1090f9a700 1 -- 172.21.15.7:0/916739376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f108c071fa0 con 0x7f108c14f9d0 2023-03-03T17:33:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.500+0000 7f107affd700 1 -- 172.21.15.7:0/916739376 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f107c021070 con 0x7f108c14f9d0 2023-03-03T17:33:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.501+0000 7f107affd700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f106c0682f0 0x7f106c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:33:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.501+0000 7f107affd700 1 -- 172.21.15.7:0/916739376 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f107c086090 con 0x7f108c14f9d0 2023-03-03T17:33:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.501+0000 7f108a59c700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f106c0682f0 0x7f106c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:33:34.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.501+0000 7f108a59c700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f106c0682f0 0x7f106c06a7a0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f1080000f60 tx=0x7f1080007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:33:34.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.505+0000 7f107affd700 1 -- 172.21.15.7:0/916739376 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f107c0523d0 con 0x7f108c14f9d0 2023-03-03T17:33:34.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.743+0000 7f1090f9a700 1 -- 172.21.15.7:0/916739376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f108c1ed070 con 0x7f108c14f9d0 2023-03-03T17:33:34.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.744+0000 7f107affd700 1 -- 172.21.15.7:0/916739376 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f108c1ed070 con 0x7f108c14f9d0 2023-03-03T17:33:34.744 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:33:34.744 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:33:34.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 -- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f106c0682f0 msgr2=0x7f106c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:34.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f106c0682f0 0x7f106c06a7a0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f1080000f60 tx=0x7f1080007ca0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 -- 172.21.15.7:0/916739376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c14f9d0 msgr2=0x7f108c0c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:33:34.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c14f9d0 0x7f108c0c1a50 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f107c00c7f0 tx=0x7f107c00cbb0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 -- 172.21.15.7:0/916739376 shutdown_connections 2023-03-03T17:33:34.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f108c14fda0 0x7f108c1ed350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f108c0abe20 0x7f108c06e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f106c0682f0 0x7f106c06a7a0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 --2- 172.21.15.7:0/916739376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f108c14f9d0 0x7f108c0c1a50 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:33:34.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 -- 172.21.15.7:0/916739376 >> 172.21.15.7:0/916739376 conn(0x7f108c0c8900 msgr2=0x7f108c0c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:33:34.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 -- 172.21.15.7:0/916739376 shutdown_connections 2023-03-03T17:33:34.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:33:34.746+0000 7f1090f9a700 1 -- 172.21.15.7:0/916739376 wait complete. 2023-03-03T17:33:34.748 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:33:34.757 INFO:tasks.mds_thrash.fs.[a]:waiting for 78 secs before thrashing 2023-03-03T17:33:57.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:33:57.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:27.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:27.852 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:33.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:34:33.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.575+0000 7f418c130700 1 Processor -- start 2023-03-03T17:34:33.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.575+0000 7f418c130700 1 -- start start 2023-03-03T17:34:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f418c130700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4184071e80 0x7f4184151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f418c130700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4184072250 0x7f418414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f418c130700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41841510d0 con 0x7f41840c1ce0 2023-03-03T17:34:33.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f418c130700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41840c3740 con 0x7f4184072250 2023-03-03T17:34:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f418c130700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4184070670 con 0x7f4184071e80 2023-03-03T17:34:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f41896cb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4184072250 0x7f418414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:33.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f418a6cd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f41840c1ce0 0x7f41841546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55850/0 (socket says 172.21.15.7:55850) 2023-03-03T17:34:33.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f418a6cd700 1 -- 172.21.15.7:0/279092246 learned_addr learned my addr 172.21.15.7:0/279092246 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:34:33.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f4189ecc700 1 --2- 172.21.15.7:0/279092246 >> v2:172.21.15.143:3300/0 conn(0x7f4184071e80 0x7f4184151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:33.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f41896cb700 1 -- 172.21.15.7:0/279092246 >> v2:172.21.15.143:3300/0 conn(0x7f4184071e80 msgr2=0x7f4184151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:34:33.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f41896cb700 1 --2- 172.21.15.7:0/279092246 >> v2:172.21.15.143:3300/0 conn(0x7f4184071e80 0x7f4184151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:33.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f41896cb700 1 -- 172.21.15.7:0/279092246 >> v1:172.21.15.7:6789/0 conn(0x7f41840c1ce0 legacy=0x7f41841546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:33.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f4188eca700 1 -- 172.21.15.7:0/279092246 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389117802 0 0) 0x7f41841510d0 con 0x7f41840c1ce0 2023-03-03T17:34:33.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f41896cb700 1 -- 172.21.15.7:0/279092246 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41840c39e0 con 0x7f4184072250 2023-03-03T17:34:33.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.576+0000 7f41896cb700 1 --2- 172.21.15.7:0/279092246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4184072250 0x7f418414e1f0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f4180009840 tx=0x7f4180009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d38c608dfc23601 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:34:33.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f4188eca700 1 -- 172.21.15.7:0/279092246 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4180011830 con 0x7f4184072250 2023-03-03T17:34:33.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f4188eca700 1 -- 172.21.15.7:0/279092246 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4180011990 con 0x7f4184072250 2023-03-03T17:34:33.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f4188eca700 1 -- 172.21.15.7:0/279092246 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4180011d20 con 0x7f4184072250 2023-03-03T17:34:33.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f418c130700 1 -- 172.21.15.7:0/279092246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4184072250 msgr2=0x7f418414e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:33.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f418c130700 1 --2- 172.21.15.7:0/279092246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4184072250 0x7f418414e1f0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f4180009840 tx=0x7f4180009b50 comp rx=0 tx=0).stop 2023-03-03T17:34:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f418c130700 1 -- 172.21.15.7:0/279092246 shutdown_connections 2023-03-03T17:34:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f418c130700 1 --2- 172.21.15.7:0/279092246 >> v2:172.21.15.143:3300/0 conn(0x7f4184071e80 0x7f4184151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f418c130700 1 --2- 172.21.15.7:0/279092246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4184072250 0x7f418414e1f0 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f418c130700 1 -- 172.21.15.7:0/279092246 >> 172.21.15.7:0/279092246 conn(0x7f41840a94d0 msgr2=0x7f41840a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:34:33.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f418c130700 1 -- 172.21.15.7:0/279092246 shutdown_connections 2023-03-03T17:34:33.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.577+0000 7f418c130700 1 -- 172.21.15.7:0/279092246 wait complete. 2023-03-03T17:34:33.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.578+0000 7f418c130700 1 Processor -- start 2023-03-03T17:34:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.578+0000 7f418c130700 1 -- start start 2023-03-03T17:34:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f418c130700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4184071e80 0x7f41841dd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f418c130700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4184072250 0x7f41841e66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f418c130700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41840c1ce0 0x7f41841da9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f418c130700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41841dc290 con 0x7f41840c1ce0 2023-03-03T17:34:33.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f418c130700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41841e6c30 con 0x7f4184072250 2023-03-03T17:34:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f418c130700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41840c59a0 con 0x7f4184071e80 2023-03-03T17:34:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f418a6cd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41840c1ce0 0x7f41841da9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f418a6cd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41840c1ce0 0x7f41841da9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33644/0 (socket says 172.21.15.7:33644) 2023-03-03T17:34:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f418a6cd700 1 -- 172.21.15.7:0/1943284272 learned_addr learned my addr 172.21.15.7:0/1943284272 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:34:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f41896cb700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4184072250 0x7f41841e66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.579+0000 7f4189ecc700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4184071e80 0x7f41841dd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:33.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f418a6cd700 1 -- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4184071e80 msgr2=0x7f41841dd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f418a6cd700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4184071e80 0x7f41841dd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f418a6cd700 1 -- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4184072250 msgr2=0x7f41841e66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f418a6cd700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4184072250 0x7f41841e66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f418a6cd700 1 -- 172.21.15.7:0/1943284272 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41800094a0 con 0x7f41840c1ce0 2023-03-03T17:34:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f418a6cd700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41840c1ce0 0x7f41841da9a0 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f417c00cc10 tx=0x7f417c00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:34:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f417a7fc700 1 -- 172.21.15.7:0/1943284272 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c00aaa0 con 0x7f41840c1ce0 2023-03-03T17:34:33.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f417a7fc700 1 -- 172.21.15.7:0/1943284272 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c00ac20 con 0x7f41840c1ce0 2023-03-03T17:34:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f418c130700 1 -- 172.21.15.7:0/1943284272 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418406c9a0 con 0x7f41840c1ce0 2023-03-03T17:34:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f417a7fc700 1 -- 172.21.15.7:0/1943284272 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c0070d0 con 0x7f41840c1ce0 2023-03-03T17:34:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.580+0000 7f418c130700 1 -- 172.21.15.7:0/1943284272 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4184151300 con 0x7f41840c1ce0 2023-03-03T17:34:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.581+0000 7f418c130700 1 -- 172.21.15.7:0/1943284272 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418406cb30 con 0x7f41840c1ce0 2023-03-03T17:34:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.581+0000 7f417a7fc700 1 -- 172.21.15.7:0/1943284272 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f417c01a030 con 0x7f41840c1ce0 2023-03-03T17:34:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.582+0000 7f417a7fc700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 0x7f416806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:33.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.582+0000 7f417a7fc700 1 -- 172.21.15.7:0/1943284272 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f417c085c80 con 0x7f41840c1ce0 2023-03-03T17:34:33.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.582+0000 7f4189ecc700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 0x7f416806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:33.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.582+0000 7f4189ecc700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 0x7f416806a7a0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f4174006b20 tx=0x7f4174008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:34:33.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.586+0000 7f417a7fc700 1 -- 172.21.15.7:0/1943284272 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f417c052020 con 0x7f41840c1ce0 2023-03-03T17:34:33.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.823+0000 7f418c130700 1 -- 172.21.15.7:0/1943284272 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4184074bb0 con 0x7f41840c1ce0 2023-03-03T17:34:33.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.824+0000 7f417a7fc700 1 -- 172.21.15.7:0/1943284272 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f417c055640 con 0x7f41840c1ce0 2023-03-03T17:34:33.823 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:34:33.824 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:34:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 -- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 msgr2=0x7f416806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 0x7f416806a7a0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f4174006b20 tx=0x7f4174008040 comp rx=0 tx=0).stop 2023-03-03T17:34:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 -- 172.21.15.7:0/1943284272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41840c1ce0 msgr2=0x7f41841da9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41840c1ce0 0x7f41841da9a0 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f417c00cc10 tx=0x7f417c00cf20 comp rx=0 tx=0).stop 2023-03-03T17:34:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 -- 172.21.15.7:0/1943284272 shutdown_connections 2023-03-03T17:34:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41680682f0 0x7f416806a7a0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41840c1ce0 0x7f41841da9a0 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4184071e80 0x7f41841dd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 --2- 172.21.15.7:0/1943284272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4184072250 0x7f41841e66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 -- 172.21.15.7:0/1943284272 >> 172.21.15.7:0/1943284272 conn(0x7f41840a94d0 msgr2=0x7f418414e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:34:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 -- 172.21.15.7:0/1943284272 shutdown_connections 2023-03-03T17:34:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:33.826+0000 7f418c130700 1 -- 172.21.15.7:0/1943284272 wait complete. 2023-03-03T17:34:33.828 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:34:33.837 INFO:tasks.mds_thrash.fs.[b]:waiting for 63 secs before thrashing 2023-03-03T17:34:52.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:34:52.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.898+0000 7f83b0bab700 1 Processor -- start 2023-03-03T17:34:52.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.898+0000 7f83b0bab700 1 -- start start 2023-03-03T17:34:52.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.899+0000 7f83b0bab700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f83ac151ab0 0x7f83ac0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:52.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.899+0000 7f83b0bab700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f83ac073c30 0x7f83ac06bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:52.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.899+0000 7f83b0bab700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac06d780 con 0x7f83ac06c3e0 2023-03-03T17:34:52.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.899+0000 7f83b0bab700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ac06cb70 con 0x7f83ac151ab0 2023-03-03T17:34:52.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.899+0000 7f83b0bab700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ac06ce50 con 0x7f83ac073c30 2023-03-03T17:34:52.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.899+0000 7f83aa59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f83ac151ab0 0x7f83ac0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:52.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.899+0000 7f83aad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f83ac06c3e0 0x7f83ac157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55870/0 (socket says 172.21.15.7:55870) 2023-03-03T17:34:52.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83aad9d700 1 -- 172.21.15.7:0/2601501702 learned_addr learned my addr 172.21.15.7:0/2601501702 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:34:52.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83a9d9b700 1 --2- 172.21.15.7:0/2601501702 >> v2:172.21.15.143:3300/0 conn(0x7f83ac073c30 0x7f83ac06bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:52.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83aa59c700 1 -- 172.21.15.7:0/2601501702 >> v2:172.21.15.143:3300/0 conn(0x7f83ac073c30 msgr2=0x7f83ac06bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:34:52.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83aa59c700 1 --2- 172.21.15.7:0/2601501702 >> v2:172.21.15.143:3300/0 conn(0x7f83ac073c30 0x7f83ac06bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:52.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83aa59c700 1 -- 172.21.15.7:0/2601501702 >> v1:172.21.15.7:6789/0 conn(0x7f83ac06c3e0 legacy=0x7f83ac157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:52.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83aa59c700 1 -- 172.21.15.7:0/2601501702 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac06d130 con 0x7f83ac151ab0 2023-03-03T17:34:52.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83a959a700 1 -- 172.21.15.7:0/2601501702 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705244365 0 0) 0x7f83ac06d780 con 0x7f83ac06c3e0 2023-03-03T17:34:52.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83a9d9b700 1 --2- 172.21.15.7:0/2601501702 >> v2:172.21.15.143:3300/0 conn(0x7f83ac073c30 0x7f83ac06bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:34:52.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83aa59c700 1 --2- 172.21.15.7:0/2601501702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 0x7f83ac0736f0 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f839c009840 tx=0x7f839c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59441123a29bc3c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:34:52.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83a959a700 1 -- 172.21.15.7:0/2601501702 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c011830 con 0x7f83ac151ab0 2023-03-03T17:34:52.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83a959a700 1 -- 172.21.15.7:0/2601501702 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c011990 con 0x7f83ac151ab0 2023-03-03T17:34:52.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.900+0000 7f83a959a700 1 -- 172.21.15.7:0/2601501702 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839c011d20 con 0x7f83ac151ab0 2023-03-03T17:34:52.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.901+0000 7f83b0bab700 1 -- 172.21.15.7:0/2601501702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 msgr2=0x7f83ac0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:52.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.901+0000 7f83b0bab700 1 --2- 172.21.15.7:0/2601501702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 0x7f83ac0736f0 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f839c009840 tx=0x7f839c009b50 comp rx=0 tx=0).stop 2023-03-03T17:34:52.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.901+0000 7f83b0bab700 1 -- 172.21.15.7:0/2601501702 shutdown_connections 2023-03-03T17:34:52.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.901+0000 7f83b0bab700 1 --2- 172.21.15.7:0/2601501702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 0x7f83ac0736f0 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:52.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.901+0000 7f83b0bab700 1 --2- 172.21.15.7:0/2601501702 >> v2:172.21.15.143:3300/0 conn(0x7f83ac073c30 0x7f83ac06bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:52.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.901+0000 7f83b0bab700 1 -- 172.21.15.7:0/2601501702 >> 172.21.15.7:0/2601501702 conn(0x7f83ac0a94d0 msgr2=0x7f83ac0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:34:52.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.901+0000 7f83b0bab700 1 -- 172.21.15.7:0/2601501702 shutdown_connections 2023-03-03T17:34:52.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.901+0000 7f83b0bab700 1 -- 172.21.15.7:0/2601501702 wait complete. 2023-03-03T17:34:52.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.903+0000 7f83b0bab700 1 Processor -- start 2023-03-03T17:34:52.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.903+0000 7f83b0bab700 1 -- start start 2023-03-03T17:34:52.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83b0bab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83ac06c3e0 0x7f83ac0aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:52.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83b0bab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83ac073c30 0x7f83ac0c0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:52.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83b0bab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 0x7f83ac1460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:52.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83b0bab700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0bff80 con 0x7f83ac151ab0 2023-03-03T17:34:52.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83b0bab700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac150390 con 0x7f83ac073c30 2023-03-03T17:34:52.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83b0bab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0bf690 con 0x7f83ac06c3e0 2023-03-03T17:34:52.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 0x7f83ac1460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:52.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 0x7f83ac1460a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33664/0 (socket says 172.21.15.7:33664) 2023-03-03T17:34:52.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aad9d700 1 -- 172.21.15.7:0/1424583645 learned_addr learned my addr 172.21.15.7:0/1424583645 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:34:52.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83a9d9b700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83ac073c30 0x7f83ac0c0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:52.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aa59c700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83ac06c3e0 0x7f83ac0aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:52.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aad9d700 1 -- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83ac06c3e0 msgr2=0x7f83ac0aacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:52.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aad9d700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83ac06c3e0 0x7f83ac0aacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:52.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aad9d700 1 -- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83ac073c30 msgr2=0x7f83ac0c0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:52.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aad9d700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83ac073c30 0x7f83ac0c0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:52.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aad9d700 1 -- 172.21.15.7:0/1424583645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c0094a0 con 0x7f83ac151ab0 2023-03-03T17:34:52.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aa59c700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83ac06c3e0 0x7f83ac0aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:34:52.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83aad9d700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 0x7f83ac1460a0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f83a4002230 tx=0x7f83a400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:34:52.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83a2ffd700 1 -- 172.21.15.7:0/1424583645 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a400ab70 con 0x7f83ac151ab0 2023-03-03T17:34:52.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83a2ffd700 1 -- 172.21.15.7:0/1424583645 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a400acf0 con 0x7f83ac151ab0 2023-03-03T17:34:52.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83b0bab700 1 -- 172.21.15.7:0/1424583645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac151e80 con 0x7f83ac151ab0 2023-03-03T17:34:52.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.904+0000 7f83a2ffd700 1 -- 172.21.15.7:0/1424583645 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a4007310 con 0x7f83ac151ab0 2023-03-03T17:34:52.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.905+0000 7f83b0bab700 1 -- 172.21.15.7:0/1424583645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac06d410 con 0x7f83ac151ab0 2023-03-03T17:34:52.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.905+0000 7f83b0bab700 1 -- 172.21.15.7:0/1424583645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac152010 con 0x7f83ac151ab0 2023-03-03T17:34:52.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.906+0000 7f83a2ffd700 1 -- 172.21.15.7:0/1424583645 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f83a4010070 con 0x7f83ac151ab0 2023-03-03T17:34:52.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.906+0000 7f83a2ffd700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f838c0682f0 0x7f838c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:34:52.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.906+0000 7f83a2ffd700 1 -- 172.21.15.7:0/1424583645 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f83a4085b50 con 0x7f83ac151ab0 2023-03-03T17:34:52.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.907+0000 7f83aa59c700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f838c0682f0 0x7f838c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:34:52.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.907+0000 7f83aa59c700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f838c0682f0 0x7f838c06a7a0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f839c0090b0 tx=0x7f839c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:34:52.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:52.911+0000 7f83a2ffd700 1 -- 172.21.15.7:0/1424583645 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f83a4051e70 con 0x7f83ac151ab0 2023-03-03T17:34:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.147+0000 7f83b0bab700 1 -- 172.21.15.7:0/1424583645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83ac074bb0 con 0x7f83ac151ab0 2023-03-03T17:34:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.148+0000 7f83a2ffd700 1 -- 172.21.15.7:0/1424583645 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f83a400ae60 con 0x7f83ac151ab0 2023-03-03T17:34:53.147 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:34:53.148 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:34:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.149+0000 7f83b0bab700 1 -- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f838c0682f0 msgr2=0x7f838c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.149+0000 7f83b0bab700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f838c0682f0 0x7f838c06a7a0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f839c0090b0 tx=0x7f839c01c040 comp rx=0 tx=0).stop 2023-03-03T17:34:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.149+0000 7f83b0bab700 1 -- 172.21.15.7:0/1424583645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 msgr2=0x7f83ac1460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:34:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.149+0000 7f83b0bab700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 0x7f83ac1460a0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f83a4002230 tx=0x7f83a400cf50 comp rx=0 tx=0).stop 2023-03-03T17:34:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.150+0000 7f83b0bab700 1 -- 172.21.15.7:0/1424583645 shutdown_connections 2023-03-03T17:34:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.150+0000 7f83b0bab700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f838c0682f0 0x7f838c06a7a0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.150+0000 7f83b0bab700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83ac151ab0 0x7f83ac1460a0 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.150+0000 7f83b0bab700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83ac06c3e0 0x7f83ac0aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.150+0000 7f83b0bab700 1 --2- 172.21.15.7:0/1424583645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83ac073c30 0x7f83ac0c0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:34:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.150+0000 7f83b0bab700 1 -- 172.21.15.7:0/1424583645 >> 172.21.15.7:0/1424583645 conn(0x7f83ac0a94d0 msgr2=0x7f83ac0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:34:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.150+0000 7f83b0bab700 1 -- 172.21.15.7:0/1424583645 shutdown_connections 2023-03-03T17:34:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:34:53.150+0000 7f83b0bab700 1 -- 172.21.15.7:0/1424583645 wait complete. 2023-03-03T17:34:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:34:53.161 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T17:34:57.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:34:57.999 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:03.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:35:03.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.303+0000 7f2f2e288700 1 Processor -- start 2023-03-03T17:35:03.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.303+0000 7f2f2e288700 1 -- start start 2023-03-03T17:35:03.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f2e288700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2f2814f9d0 0x7f2f28156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:03.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f2e288700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2f2814fda0 0x7f2f28071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:03.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f2e288700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f280c2f30 con 0x7f2f280abe20 2023-03-03T17:35:03.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f2e288700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f280c31d0 con 0x7f2f2814fda0 2023-03-03T17:35:03.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f2e288700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f281524a0 con 0x7f2f2814f9d0 2023-03-03T17:35:03.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f277fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2f2814fda0 0x7f2f28071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:03.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f2c825700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2f280abe20 0x7f2f281529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55892/0 (socket says 172.21.15.7:55892) 2023-03-03T17:35:03.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f2c825700 1 -- 172.21.15.7:0/2612136050 learned_addr learned my addr 172.21.15.7:0/2612136050 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:35:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f27fff700 1 --2- 172.21.15.7:0/2612136050 >> v2:172.21.15.143:3300/0 conn(0x7f2f2814f9d0 0x7f2f28156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f277fe700 1 -- 172.21.15.7:0/2612136050 >> v2:172.21.15.143:3300/0 conn(0x7f2f2814f9d0 msgr2=0x7f2f28156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:35:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f277fe700 1 --2- 172.21.15.7:0/2612136050 >> v2:172.21.15.143:3300/0 conn(0x7f2f2814f9d0 0x7f2f28156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f277fe700 1 -- 172.21.15.7:0/2612136050 >> v1:172.21.15.7:6789/0 conn(0x7f2f280abe20 legacy=0x7f2f281529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f277fe700 1 -- 172.21.15.7:0/2612136050 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f280c34b0 con 0x7f2f2814fda0 2023-03-03T17:35:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.304+0000 7f2f277fe700 1 --2- 172.21.15.7:0/2612136050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 0x7f2f28071890 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f2f280710a0 tx=0x7f2f1c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c0f14ce3428f7d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f26ffd700 1 -- 172.21.15.7:0/2612136050 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c011830 con 0x7f2f2814fda0 2023-03-03T17:35:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f26ffd700 1 -- 172.21.15.7:0/2612136050 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1c011990 con 0x7f2f2814fda0 2023-03-03T17:35:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f26ffd700 1 -- 172.21.15.7:0/2612136050 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c011d20 con 0x7f2f2814fda0 2023-03-03T17:35:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f2e288700 1 -- 172.21.15.7:0/2612136050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 msgr2=0x7f2f28071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:03.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f2e288700 1 --2- 172.21.15.7:0/2612136050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 0x7f2f28071890 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f2f280710a0 tx=0x7f2f1c009b20 comp rx=0 tx=0).stop 2023-03-03T17:35:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f2e288700 1 -- 172.21.15.7:0/2612136050 shutdown_connections 2023-03-03T17:35:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f2e288700 1 --2- 172.21.15.7:0/2612136050 >> v2:172.21.15.143:3300/0 conn(0x7f2f2814f9d0 0x7f2f28156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f2e288700 1 --2- 172.21.15.7:0/2612136050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 0x7f2f28071890 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f2e288700 1 -- 172.21.15.7:0/2612136050 >> 172.21.15.7:0/2612136050 conn(0x7f2f280c8900 msgr2=0x7f2f280c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:35:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f2e288700 1 -- 172.21.15.7:0/2612136050 shutdown_connections 2023-03-03T17:35:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.305+0000 7f2f2e288700 1 -- 172.21.15.7:0/2612136050 wait complete. 2023-03-03T17:35:03.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.306+0000 7f2f2e288700 1 Processor -- start 2023-03-03T17:35:03.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.307+0000 7f2f2e288700 1 -- start start 2023-03-03T17:35:03.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.307+0000 7f2f2e288700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f280abe20 0x7f2f281ed7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:03.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.307+0000 7f2f2e288700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f2814f9d0 0x7f2f280c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:03.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.307+0000 7f2f2e288700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 0x7f2f2814eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.307+0000 7f2f2e288700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f281dcd60 con 0x7f2f2814fda0 2023-03-03T17:35:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.307+0000 7f2f2e288700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f28155210 con 0x7f2f2814f9d0 2023-03-03T17:35:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.307+0000 7f2f2e288700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2806a2b0 con 0x7f2f280abe20 2023-03-03T17:35:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.307+0000 7f2f2c825700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 0x7f2f2814eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f2c825700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 0x7f2f2814eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33686/0 (socket says 172.21.15.7:33686) 2023-03-03T17:35:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f2c825700 1 -- 172.21.15.7:0/212680166 learned_addr learned my addr 172.21.15.7:0/212680166 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:35:03.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f277fe700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f2814f9d0 0x7f2f280c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f27fff700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f280abe20 0x7f2f281ed7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f2c825700 1 -- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f280abe20 msgr2=0x7f2f281ed7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f2c825700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f280abe20 0x7f2f281ed7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f2c825700 1 -- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f2814f9d0 msgr2=0x7f2f280c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f2c825700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f2814f9d0 0x7f2f280c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f2c825700 1 -- 172.21.15.7:0/212680166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c0094a0 con 0x7f2f2814fda0 2023-03-03T17:35:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f2c825700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 0x7f2f2814eda0 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000caa0 tx=0x7f2f2000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:03.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f24ff9700 1 -- 172.21.15.7:0/212680166 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2000a9e0 con 0x7f2f2814fda0 2023-03-03T17:35:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f24ff9700 1 -- 172.21.15.7:0/212680166 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2000ab40 con 0x7f2f2814fda0 2023-03-03T17:35:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f2e288700 1 -- 172.21.15.7:0/212680166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f280ac350 con 0x7f2f2814fda0 2023-03-03T17:35:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f24ff9700 1 -- 172.21.15.7:0/212680166 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20012400 con 0x7f2f2814fda0 2023-03-03T17:35:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.308+0000 7f2f2e288700 1 -- 172.21.15.7:0/212680166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f280ab630 con 0x7f2f2814fda0 2023-03-03T17:35:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.309+0000 7f2f2e288700 1 -- 172.21.15.7:0/212680166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f280c3e10 con 0x7f2f2814fda0 2023-03-03T17:35:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.310+0000 7f2f24ff9700 1 -- 172.21.15.7:0/212680166 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f20019030 con 0x7f2f2814fda0 2023-03-03T17:35:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.310+0000 7f2f24ff9700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f140682f0 0x7f2f1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.310+0000 7f2f24ff9700 1 -- 172.21.15.7:0/212680166 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2f200859d0 con 0x7f2f2814fda0 2023-03-03T17:35:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.310+0000 7f2f27fff700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f140682f0 0x7f2f1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.311+0000 7f2f27fff700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f140682f0 0x7f2f1406a7a0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f2f18006b20 tx=0x7f2f18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.315+0000 7f2f24ff9700 1 -- 172.21.15.7:0/212680166 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f2004b5e0 con 0x7f2f2814fda0 2023-03-03T17:35:03.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.556+0000 7f2f2e288700 1 -- 172.21.15.7:0/212680166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f28074bb0 con 0x7f2f2814fda0 2023-03-03T17:35:03.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.556+0000 7f2f24ff9700 1 -- 172.21.15.7:0/212680166 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2f20051ee0 con 0x7f2f2814fda0 2023-03-03T17:35:03.556 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:35:03.556 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:35:03.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 -- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f140682f0 msgr2=0x7f2f1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:03.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f140682f0 0x7f2f1406a7a0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f2f18006b20 tx=0x7f2f18008040 comp rx=0 tx=0).stop 2023-03-03T17:35:03.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 -- 172.21.15.7:0/212680166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 msgr2=0x7f2f2814eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:03.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 0x7f2f2814eda0 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000caa0 tx=0x7f2f2000cdb0 comp rx=0 tx=0).stop 2023-03-03T17:35:03.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 -- 172.21.15.7:0/212680166 shutdown_connections 2023-03-03T17:35:03.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f140682f0 0x7f2f1406a7a0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f2814fda0 0x7f2f2814eda0 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f280abe20 0x7f2f281ed7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 --2- 172.21.15.7:0/212680166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f2814f9d0 0x7f2f280c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 -- 172.21.15.7:0/212680166 >> 172.21.15.7:0/212680166 conn(0x7f2f280c8900 msgr2=0x7f2f28150510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:35:03.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 -- 172.21.15.7:0/212680166 shutdown_connections 2023-03-03T17:35:03.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:03.558+0000 7f2f2e288700 1 -- 172.21.15.7:0/212680166 wait complete. 2023-03-03T17:35:03.560 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:35:03.569 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T17:35:28.059 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:28.063 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:36.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:35:36.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.979+0000 7fa10a77a700 1 Processor -- start 2023-03-03T17:35:36.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.980+0000 7fa10a77a700 1 -- start start 2023-03-03T17:35:36.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa10a77a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10414fda0 0x7fa104156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:36.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa10a77a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa1040abe20 0x7fa1041529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:36.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa10a77a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1040c2f30 con 0x7fa10414f9d0 2023-03-03T17:35:36.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa10a77a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1040c31d0 con 0x7fa1040abe20 2023-03-03T17:35:36.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa10a77a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041524a0 con 0x7fa10414fda0 2023-03-03T17:35:36.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa103fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa10414f9d0 0x7fa104071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55930/0 (socket says 172.21.15.7:55930) 2023-03-03T17:35:36.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa103fff700 1 -- 172.21.15.7:0/1675843841 learned_addr learned my addr 172.21.15.7:0/1675843841 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:35:36.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa108d17700 1 --2- 172.21.15.7:0/1675843841 >> v2:172.21.15.7:3300/0 conn(0x7fa1040abe20 0x7fa1041529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:36.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa1037fe700 1 --2- 172.21.15.7:0/1675843841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10414fda0 0x7fa104156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:36.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa102ffd700 1 -- 172.21.15.7:0/1675843841 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658315825 0 0) 0x7fa1040c2f30 con 0x7fa10414f9d0 2023-03-03T17:35:36.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa102ffd700 1 -- 172.21.15.7:0/1675843841 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ec0035b0 con 0x7fa10414f9d0 2023-03-03T17:35:36.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa108d17700 1 -- 172.21.15.7:0/1675843841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10414fda0 msgr2=0x7fa104156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:35:36.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa108d17700 1 --2- 172.21.15.7:0/1675843841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10414fda0 0x7fa104156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:36.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa108d17700 1 -- 172.21.15.7:0/1675843841 >> v1:172.21.15.7:6789/0 conn(0x7fa10414f9d0 legacy=0x7fa104071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:36.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa108d17700 1 -- 172.21.15.7:0/1675843841 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1040c34b0 con 0x7fa1040abe20 2023-03-03T17:35:36.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa1037fe700 1 --2- 172.21.15.7:0/1675843841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10414fda0 0x7fa104156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:35:36.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.981+0000 7fa108d17700 1 --2- 172.21.15.7:0/1675843841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1040abe20 0x7fa1041529c0 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009d90 tx=0x7fa0fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7898545ccf5cfa1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:36.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa102ffd700 1 -- 172.21.15.7:0/1675843841 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc00e070 con 0x7fa1040abe20 2023-03-03T17:35:36.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa102ffd700 1 -- 172.21.15.7:0/1675843841 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0fc00bb20 con 0x7fa1040abe20 2023-03-03T17:35:36.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa102ffd700 1 -- 172.21.15.7:0/1675843841 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc00be90 con 0x7fa1040abe20 2023-03-03T17:35:36.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa10a77a700 1 -- 172.21.15.7:0/1675843841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1040abe20 msgr2=0x7fa1041529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:36.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa10a77a700 1 --2- 172.21.15.7:0/1675843841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1040abe20 0x7fa1041529c0 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009d90 tx=0x7fa0fc00b330 comp rx=0 tx=0).stop 2023-03-03T17:35:36.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa10a77a700 1 -- 172.21.15.7:0/1675843841 shutdown_connections 2023-03-03T17:35:36.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa10a77a700 1 --2- 172.21.15.7:0/1675843841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1040abe20 0x7fa1041529c0 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:36.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa10a77a700 1 --2- 172.21.15.7:0/1675843841 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10414fda0 0x7fa104156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:36.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa10a77a700 1 -- 172.21.15.7:0/1675843841 >> 172.21.15.7:0/1675843841 conn(0x7fa1040c8900 msgr2=0x7fa1040c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:35:36.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa10a77a700 1 -- 172.21.15.7:0/1675843841 shutdown_connections 2023-03-03T17:35:36.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.982+0000 7fa10a77a700 1 -- 172.21.15.7:0/1675843841 wait complete. 2023-03-03T17:35:36.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.983+0000 7fa10a77a700 1 Processor -- start 2023-03-03T17:35:36.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.984+0000 7fa10a77a700 1 -- start start 2023-03-03T17:35:36.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.984+0000 7fa10a77a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1040abe20 0x7fa10414d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:36.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.984+0000 7fa10a77a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10414f9d0 0x7fa1040c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:36.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.984+0000 7fa10a77a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa10414fda0 0x7fa10414c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:36.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.984+0000 7fa10a77a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041548d0 con 0x7fa10414fda0 2023-03-03T17:35:36.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.984+0000 7fa10a77a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa104153830 con 0x7fa1040abe20 2023-03-03T17:35:36.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.984+0000 7fa10a77a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa104153b10 con 0x7fa10414f9d0 2023-03-03T17:35:36.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa108d17700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa10414fda0 0x7fa10414c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:36.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa108d17700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa10414fda0 0x7fa10414c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33728/0 (socket says 172.21.15.7:33728) 2023-03-03T17:35:36.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa108d17700 1 -- 172.21.15.7:0/1822982827 learned_addr learned my addr 172.21.15.7:0/1822982827 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:35:36.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa103fff700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1040abe20 0x7fa10414d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:36.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa108d17700 1 -- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10414f9d0 msgr2=0x7fa1040c6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:36.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa108d17700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10414f9d0 0x7fa1040c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:36.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa108d17700 1 -- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1040abe20 msgr2=0x7fa10414d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:36.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa108d17700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1040abe20 0x7fa10414d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:36.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa108d17700 1 -- 172.21.15.7:0/1822982827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc009a70 con 0x7fa10414fda0 2023-03-03T17:35:36.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa108d17700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa10414fda0 0x7fa10414c410 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc014040 tx=0x7fa0fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:36.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa100ff9700 1 -- 172.21.15.7:0/1822982827 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc004750 con 0x7fa10414fda0 2023-03-03T17:35:36.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa100ff9700 1 -- 172.21.15.7:0/1822982827 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0fc020070 con 0x7fa10414fda0 2023-03-03T17:35:36.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa10a77a700 1 -- 172.21.15.7:0/1822982827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1040ac350 con 0x7fa10414fda0 2023-03-03T17:35:36.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa100ff9700 1 -- 172.21.15.7:0/1822982827 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc027480 con 0x7fa10414fda0 2023-03-03T17:35:36.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.985+0000 7fa10a77a700 1 -- 172.21.15.7:0/1822982827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1040ab630 con 0x7fa10414fda0 2023-03-03T17:35:36.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.986+0000 7fa10a77a700 1 -- 172.21.15.7:0/1822982827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa104071fa0 con 0x7fa10414fda0 2023-03-03T17:35:36.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.987+0000 7fa100ff9700 1 -- 172.21.15.7:0/1822982827 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa0fc012430 con 0x7fa10414fda0 2023-03-03T17:35:36.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.987+0000 7fa100ff9700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0f00682f0 0x7fa0f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:36.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.987+0000 7fa100ff9700 1 -- 172.21.15.7:0/1822982827 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa0fc022070 con 0x7fa10414fda0 2023-03-03T17:35:36.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.987+0000 7fa103fff700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0f00682f0 0x7fa0f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:36.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.988+0000 7fa103fff700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0f00682f0 0x7fa0f006a7a0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4000f60 tx=0x7fa0f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:36.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:36.992+0000 7fa100ff9700 1 -- 172.21.15.7:0/1822982827 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa0fc054630 con 0x7fa10414fda0 2023-03-03T17:35:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.234+0000 7fa10a77a700 1 -- 172.21.15.7:0/1822982827 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa104074bb0 con 0x7fa10414fda0 2023-03-03T17:35:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.234+0000 7fa100ff9700 1 -- 172.21.15.7:0/1822982827 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa0fc057c50 con 0x7fa10414fda0 2023-03-03T17:35:37.234 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:35:37.234 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:35:37.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 -- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0f00682f0 msgr2=0x7fa0f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:37.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0f00682f0 0x7fa0f006a7a0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4000f60 tx=0x7fa0f4009040 comp rx=0 tx=0).stop 2023-03-03T17:35:37.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 -- 172.21.15.7:0/1822982827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa10414fda0 msgr2=0x7fa10414c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:37.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa10414fda0 0x7fa10414c410 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc014040 tx=0x7fa0fc004ad0 comp rx=0 tx=0).stop 2023-03-03T17:35:37.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 -- 172.21.15.7:0/1822982827 shutdown_connections 2023-03-03T17:35:37.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa0f00682f0 0x7fa0f006a7a0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:37.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa10414fda0 0x7fa10414c410 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:37.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1040abe20 0x7fa10414d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:37.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 --2- 172.21.15.7:0/1822982827 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa10414f9d0 0x7fa1040c6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:37.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 -- 172.21.15.7:0/1822982827 >> 172.21.15.7:0/1822982827 conn(0x7fa1040c8900 msgr2=0x7fa104150930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:35:37.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 -- 172.21.15.7:0/1822982827 shutdown_connections 2023-03-03T17:35:37.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:37.237+0000 7fa10a77a700 1 -- 172.21.15.7:0/1822982827 wait complete. 2023-03-03T17:35:37.238 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:35:37.248 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T17:35:52.571 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:35:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.710+0000 7fbd63bbc700 1 Processor -- start 2023-03-03T17:35:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.710+0000 7fbd63bbc700 1 -- start start 2023-03-03T17:35:52.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.711+0000 7fbd63bbc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbd5c0c2930 0x7fbd5c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:52.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.711+0000 7fbd63bbc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5c06d430 con 0x7fbd5c151d40 2023-03-03T17:35:52.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.711+0000 7fbd63bbc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5c1586e0 con 0x7fbd5c06c480 2023-03-03T17:35:52.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.711+0000 7fbd63bbc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5c06c2a0 con 0x7fbd5c0c2930 2023-03-03T17:35:52.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.711+0000 7fbd61958700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbd5c151d40 0x7fbd5c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55964/0 (socket says 172.21.15.7:55964) 2023-03-03T17:35:52.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.711+0000 7fbd61958700 1 -- 172.21.15.7:0/4281707404 learned_addr learned my addr 172.21.15.7:0/4281707404 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:35:52.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.711+0000 7fbd61157700 1 --2- 172.21.15.7:0/4281707404 >> v2:172.21.15.143:3300/0 conn(0x7fbd5c0c2930 0x7fbd5c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:52.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.711+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847230922 0 0) 0x7fbd5c06d430 con 0x7fbd5c151d40 2023-03-03T17:35:52.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd480035b0 con 0x7fbd5c151d40 2023-03-03T17:35:52.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3712841891 0 0) 0x7fbd480035b0 con 0x7fbd5c151d40 2023-03-03T17:35:52.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 >> v2:172.21.15.143:3300/0 conn(0x7fbd5c0c2930 msgr2=0x7fbd5c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:35:52.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 --2- 172.21.15.7:0/4281707404 >> v2:172.21.15.143:3300/0 conn(0x7fbd5c0c2930 0x7fbd5c157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:52.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 >> v1:172.21.15.143:6789/0 conn(0x7fbd5c06c480 legacy=0x7fbd5c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:52.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c06cb00 con 0x7fbd5c151d40 2023-03-03T17:35:52.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fbd4c001670 con 0x7fbd5c151d40 2023-03-03T17:35:52.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 >> v1:172.21.15.7:6789/0 conn(0x7fbd5c151d40 legacy=0x7fbd5c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:52.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd48003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:52.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd48004850 0x7fbd48004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:52.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd48005320 0x7fbd4800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:52.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c06c2a0 con 0x7fbd48003a70 2023-03-03T17:35:52.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4800c250 con 0x7fbd48004850 2023-03-03T17:35:52.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4800c410 con 0x7fbd48005320 2023-03-03T17:35:52.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd61958700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd48003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:52.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd61157700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd48004850 0x7fbd48004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:52.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd62159700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd48005320 0x7fbd4800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:52.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd61958700 1 -- 172.21.15.7:0/4281707404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd48005320 msgr2=0x7fbd4800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:35:52.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd61958700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd48005320 0x7fbd4800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:52.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd61958700 1 -- 172.21.15.7:0/4281707404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd48004850 msgr2=0x7fbd48004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:35:52.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd61958700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd48004850 0x7fbd48004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:52.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd61958700 1 -- 172.21.15.7:0/4281707404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd480035b0 con 0x7fbd48003a70 2023-03-03T17:35:52.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd62159700 1 -- 172.21.15.7:0/4281707404 reap_dead start 2023-03-03T17:35:52.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd61157700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd48004850 0x7fbd48004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:35:52.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.712+0000 7fbd61958700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd48003e80 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00c240 tx=0x7fbd48005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d143e8e63dd2ad46 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:52.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.713+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c007c10 con 0x7fbd48003a70 2023-03-03T17:35:52.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.713+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c00ce00 con 0x7fbd48003a70 2023-03-03T17:35:52.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.713+0000 7fbd60956700 1 -- 172.21.15.7:0/4281707404 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c005ba0 con 0x7fbd48003a70 2023-03-03T17:35:52.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.713+0000 7fbd63bbc700 1 -- 172.21.15.7:0/4281707404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 msgr2=0x7fbd48003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:52.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.713+0000 7fbd63bbc700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd48003e80 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00c240 tx=0x7fbd48005880 comp rx=0 tx=0).stop 2023-03-03T17:35:52.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.713+0000 7fbd63bbc700 1 -- 172.21.15.7:0/4281707404 shutdown_connections 2023-03-03T17:35:52.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.713+0000 7fbd63bbc700 1 --2- 172.21.15.7:0/4281707404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd48003e80 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:52.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.713+0000 7fbd63bbc700 1 -- 172.21.15.7:0/4281707404 >> 172.21.15.7:0/4281707404 conn(0x7fbd5c0a94d0 msgr2=0x7fbd5c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:35:52.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.713+0000 7fbd63bbc700 1 -- 172.21.15.7:0/4281707404 shutdown_connections 2023-03-03T17:35:52.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.713+0000 7fbd63bbc700 1 -- 172.21.15.7:0/4281707404 wait complete. 2023-03-03T17:35:52.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd63bbc700 1 Processor -- start 2023-03-03T17:35:52.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd63bbc700 1 -- start start 2023-03-03T17:35:52.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd63bbc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd5c0c0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:52.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd63bbc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd5c153600 0x7fbd5c0c08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:52.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd63bbc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5c14d960 0x7fbd5c1ed780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:52.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd63bbc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c0bff20 con 0x7fbd48003a70 2023-03-03T17:35:52.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd63bbc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c1ee3b0 con 0x7fbd5c153600 2023-03-03T17:35:52.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd63bbc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c14df40 con 0x7fbd5c14d960 2023-03-03T17:35:52.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd61958700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd5c0c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:52.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd61958700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd5c0c0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33764/0 (socket says 172.21.15.7:33764) 2023-03-03T17:35:52.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.715+0000 7fbd61958700 1 -- 172.21.15.7:0/3088100907 learned_addr learned my addr 172.21.15.7:0/3088100907 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:35:52.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd61157700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd5c153600 0x7fbd5c0c08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:52.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd62159700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5c14d960 0x7fbd5c1ed780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:52.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd61958700 1 -- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5c14d960 msgr2=0x7fbd5c1ed780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:52.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd61958700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5c14d960 0x7fbd5c1ed780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:52.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd61958700 1 -- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd5c153600 msgr2=0x7fbd5c0c08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:52.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd61958700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd5c153600 0x7fbd5c0c08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:52.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd61958700 1 -- 172.21.15.7:0/3088100907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c06d0c0 con 0x7fbd48003a70 2023-03-03T17:35:52.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd61958700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd5c0c0370 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800c220 tx=0x7fbd4c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:52.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd527fc700 1 -- 172.21.15.7:0/3088100907 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c001670 con 0x7fbd48003a70 2023-03-03T17:35:52.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd527fc700 1 -- 172.21.15.7:0/3088100907 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c0070d0 con 0x7fbd48003a70 2023-03-03T17:35:52.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd63bbc700 1 -- 172.21.15.7:0/3088100907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c06cde0 con 0x7fbd48003a70 2023-03-03T17:35:52.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd527fc700 1 -- 172.21.15.7:0/3088100907 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c007480 con 0x7fbd48003a70 2023-03-03T17:35:52.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.716+0000 7fbd63bbc700 1 -- 172.21.15.7:0/3088100907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c06bea0 con 0x7fbd48003a70 2023-03-03T17:35:52.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.717+0000 7fbd63bbc700 1 -- 172.21.15.7:0/3088100907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c074bb0 con 0x7fbd48003a70 2023-03-03T17:35:52.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.718+0000 7fbd527fc700 1 -- 172.21.15.7:0/3088100907 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd4c02e070 con 0x7fbd48003a70 2023-03-03T17:35:52.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.718+0000 7fbd527fc700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:52.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.718+0000 7fbd527fc700 1 -- 172.21.15.7:0/3088100907 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbd4c087870 con 0x7fbd48003a70 2023-03-03T17:35:52.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.718+0000 7fbd61157700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:52.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.719+0000 7fbd61157700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0c0f30 tx=0x7fbd58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:52.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.722+0000 7fbd527fc700 1 -- 172.21.15.7:0/3088100907 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbd4c053b90 con 0x7fbd48003a70 2023-03-03T17:35:52.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.963+0000 7fbd63bbc700 1 -- 172.21.15.7:0/3088100907 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd5c0a9fb0 con 0x7fbd48003a70 2023-03-03T17:35:52.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.964+0000 7fbd527fc700 1 -- 172.21.15.7:0/3088100907 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbd4c0571b0 con 0x7fbd48003a70 2023-03-03T17:35:52.967 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:35:52.968 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:35:52.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 -- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 msgr2=0x7fbd4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:52.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0c0f30 tx=0x7fbd58008040 comp rx=0 tx=0).stop 2023-03-03T17:35:52.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 -- 172.21.15.7:0/3088100907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 msgr2=0x7fbd5c0c0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:52.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd5c0c0370 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800c220 tx=0x7fbd4c001a00 comp rx=0 tx=0).stop 2023-03-03T17:35:52.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 -- 172.21.15.7:0/3088100907 shutdown_connections 2023-03-03T17:35:52.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbd5c14d960 0x7fbd5c1ed780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd400682f0 0x7fbd4006a7a0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbd48003a70 0x7fbd5c0c0370 unknown :-1 s=CLOSED pgs=2959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 --2- 172.21.15.7:0/3088100907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbd5c153600 0x7fbd5c0c08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 -- 172.21.15.7:0/3088100907 >> 172.21.15.7:0/3088100907 conn(0x7fbd5c0a94d0 msgr2=0x7fbd5c14f760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:35:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 -- 172.21.15.7:0/3088100907 shutdown_connections 2023-03-03T17:35:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:52.966+0000 7fbd63bbc700 1 -- 172.21.15.7:0/3088100907 wait complete. 2023-03-03T17:35:52.975 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:35:52.980 INFO:tasks.mds_thrash.fs.[a]:waiting for 106 secs before thrashing 2023-03-03T17:35:55.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:35:55.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.389+0000 7f7051c54700 1 Processor -- start 2023-03-03T17:35:55.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.389+0000 7f7051c54700 1 -- start start 2023-03-03T17:35:55.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.390+0000 7f7051c54700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c06e020 0x7f704c071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:55.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.390+0000 7f7051c54700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704c154910 con 0x7f704c072520 2023-03-03T17:35:55.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.390+0000 7f7051c54700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704c0c4540 con 0x7f704c155390 2023-03-03T17:35:55.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.390+0000 7f7051c54700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704c070b90 con 0x7f704c06e020 2023-03-03T17:35:55.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.390+0000 7f704bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f704c072520 0x7f704c151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55988/0 (socket says 172.21.15.7:55988) 2023-03-03T17:35:55.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.390+0000 7f704bfff700 1 -- 172.21.15.7:0/1284939147 learned_addr learned my addr 172.21.15.7:0/1284939147 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:35:55.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.390+0000 7f704affd700 1 --2- 172.21.15.7:0/1284939147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c06e020 0x7f704c071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:55.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.390+0000 7f704a7fc700 1 -- 172.21.15.7:0/1284939147 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171947618 0 0) 0x7f704c154910 con 0x7f704c072520 2023-03-03T17:35:55.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.390+0000 7f704a7fc700 1 -- 172.21.15.7:0/1284939147 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70380035b0 con 0x7f704c072520 2023-03-03T17:35:55.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.391+0000 7f704a7fc700 1 -- 172.21.15.7:0/1284939147 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398912469 0 0) 0x7f703c001110 con 0x7f704c155390 2023-03-03T17:35:55.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.391+0000 7f704a7fc700 1 -- 172.21.15.7:0/1284939147 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f704c154910 con 0x7f704c155390 2023-03-03T17:35:55.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.391+0000 7f704affd700 1 -- 172.21.15.7:0/1284939147 >> v1:172.21.15.143:6789/0 conn(0x7f704c155390 legacy=0x7f704c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:55.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.391+0000 7f704affd700 1 -- 172.21.15.7:0/1284939147 >> v1:172.21.15.7:6789/0 conn(0x7f704c072520 legacy=0x7f704c151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:55.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.391+0000 7f704a7fc700 1 -- 172.21.15.7:0/1284939147 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1504802354 0 0) 0x7f70380035b0 con 0x7f704c072520 2023-03-03T17:35:55.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.391+0000 7f704affd700 1 -- 172.21.15.7:0/1284939147 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c0702a0 con 0x7f704c06e020 2023-03-03T17:35:55.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.391+0000 7f704affd700 1 --2- 172.21.15.7:0/1284939147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c06e020 0x7f704c071fe0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7034009870 tx=0x7f7034009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f83347e67ca18cdb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:55.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.391+0000 7f704a7fc700 1 -- 172.21.15.7:0/1284939147 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70440019f0 con 0x7f704c06e020 2023-03-03T17:35:55.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.391+0000 7f704a7fc700 1 -- 172.21.15.7:0/1284939147 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7034004030 con 0x7f704c06e020 2023-03-03T17:35:55.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.391+0000 7f704a7fc700 1 -- 172.21.15.7:0/1284939147 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70340043c0 con 0x7f704c06e020 2023-03-03T17:35:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.392+0000 7f7051c54700 1 -- 172.21.15.7:0/1284939147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c06e020 msgr2=0x7f704c071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.392+0000 7f7051c54700 1 --2- 172.21.15.7:0/1284939147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c06e020 0x7f704c071fe0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7034009870 tx=0x7f7034009b80 comp rx=0 tx=0).stop 2023-03-03T17:35:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.392+0000 7f7051c54700 1 -- 172.21.15.7:0/1284939147 shutdown_connections 2023-03-03T17:35:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.392+0000 7f7051c54700 1 --2- 172.21.15.7:0/1284939147 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c06e020 0x7f704c071fe0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.392+0000 7f7051c54700 1 -- 172.21.15.7:0/1284939147 >> 172.21.15.7:0/1284939147 conn(0x7f704c06c110 msgr2=0x7f704c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:35:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.392+0000 7f7051c54700 1 -- 172.21.15.7:0/1284939147 shutdown_connections 2023-03-03T17:35:55.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.392+0000 7f7051c54700 1 -- 172.21.15.7:0/1284939147 wait complete. 2023-03-03T17:35:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.393+0000 7f7051c54700 1 Processor -- start 2023-03-03T17:35:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.393+0000 7f7051c54700 1 -- start start 2023-03-03T17:35:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f7051c54700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f704c06e020 0x7f704c0c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f7051c54700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f704c072520 0x7f704c1dc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f7051c54700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c155390 0x7f704c1dcef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f7051c54700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c0ac090 con 0x7f704c06e020 2023-03-03T17:35:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f7051c54700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c1475a0 con 0x7f704c072520 2023-03-03T17:35:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f7051c54700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704c0abad0 con 0x7f704c155390 2023-03-03T17:35:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f704c06e020 0x7f704c0c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f704c06e020 0x7f704c0c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33778/0 (socket says 172.21.15.7:33778) 2023-03-03T17:35:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704b7fe700 1 -- 172.21.15.7:0/1795009284 learned_addr learned my addr 172.21.15.7:0/1795009284 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:35:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704affd700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f704c072520 0x7f704c1dc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704bfff700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c155390 0x7f704c1dcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704b7fe700 1 -- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c155390 msgr2=0x7f704c1dcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704b7fe700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c155390 0x7f704c1dcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704b7fe700 1 -- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f704c072520 msgr2=0x7f704c1dc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704b7fe700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f704c072520 0x7f704c1dc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704b7fe700 1 -- 172.21.15.7:0/1795009284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70340094a0 con 0x7f704c06e020 2023-03-03T17:35:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704affd700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f704c072520 0x7f704c1dc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:35:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.394+0000 7f704b7fe700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f704c06e020 0x7f704c0c65f0 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7f703c00c700 tx=0x7f703c00cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.395+0000 7f7043fff700 1 -- 172.21.15.7:0/1795009284 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c0052d0 con 0x7f704c06e020 2023-03-03T17:35:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.395+0000 7f7043fff700 1 -- 172.21.15.7:0/1795009284 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703c005430 con 0x7f704c06e020 2023-03-03T17:35:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.395+0000 7f7051c54700 1 -- 172.21.15.7:0/1795009284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704c070820 con 0x7f704c06e020 2023-03-03T17:35:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.395+0000 7f7043fff700 1 -- 172.21.15.7:0/1795009284 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c0038a0 con 0x7f704c06e020 2023-03-03T17:35:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.395+0000 7f7051c54700 1 -- 172.21.15.7:0/1795009284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704c070540 con 0x7f704c06e020 2023-03-03T17:35:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.396+0000 7f7051c54700 1 -- 172.21.15.7:0/1795009284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704c155760 con 0x7f704c06e020 2023-03-03T17:35:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.396+0000 7f7043fff700 1 -- 172.21.15.7:0/1795009284 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f703c019030 con 0x7f704c06e020 2023-03-03T17:35:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.396+0000 7f7043fff700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f702c0682f0 0x7f702c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:35:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.397+0000 7f7043fff700 1 -- 172.21.15.7:0/1795009284 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f703c0853d0 con 0x7f704c06e020 2023-03-03T17:35:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.397+0000 7f704affd700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f702c0682f0 0x7f702c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:35:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.397+0000 7f704affd700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f702c0682f0 0x7f702c06a7a0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f70340097c0 tx=0x7f7034008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:35:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.401+0000 7f7043fff700 1 -- 172.21.15.7:0/1795009284 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f703c0516f0 con 0x7f704c06e020 2023-03-03T17:35:55.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.643+0000 7f7051c54700 1 -- 172.21.15.7:0/1795009284 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f704c06ca60 con 0x7f704c06e020 2023-03-03T17:35:55.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.643+0000 7f7043fff700 1 -- 172.21.15.7:0/1795009284 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f703c054d10 con 0x7f704c06e020 2023-03-03T17:35:55.643 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:35:55.643 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:35:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 -- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f702c0682f0 msgr2=0x7f702c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f702c0682f0 0x7f702c06a7a0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f70340097c0 tx=0x7f7034008030 comp rx=0 tx=0).stop 2023-03-03T17:35:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 -- 172.21.15.7:0/1795009284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f704c06e020 msgr2=0x7f704c0c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:35:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f704c06e020 0x7f704c0c65f0 secure :-1 s=READY pgs=2961 cs=0 l=1 rev1=1 crypto rx=0x7f703c00c700 tx=0x7f703c00cac0 comp rx=0 tx=0).stop 2023-03-03T17:35:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 -- 172.21.15.7:0/1795009284 shutdown_connections 2023-03-03T17:35:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f704c155390 0x7f704c1dcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f702c0682f0 0x7f702c06a7a0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f704c06e020 0x7f704c0c65f0 unknown :-1 s=CLOSED pgs=2961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 --2- 172.21.15.7:0/1795009284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f704c072520 0x7f704c1dc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:35:55.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 -- 172.21.15.7:0/1795009284 >> 172.21.15.7:0/1795009284 conn(0x7f704c06c110 msgr2=0x7f704c0c3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:35:55.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 -- 172.21.15.7:0/1795009284 shutdown_connections 2023-03-03T17:35:55.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:35:55.645+0000 7f7051c54700 1 -- 172.21.15.7:0/1795009284 wait complete. 2023-03-03T17:35:55.646 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:35:55.656 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T17:35:58.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:35:58.227 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:13.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:36:13.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.800+0000 7f1528f3c700 1 Processor -- start 2023-03-03T17:36:13.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.800+0000 7f1528f3c700 1 -- start start 2023-03-03T17:36:13.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f1528f3c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f152406d4b0 0x7f152406d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:36:13.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f1528f3c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f15240c1da0 0x7f1524139d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:36:13.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f1528f3c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15240c7b90 con 0x7f15240c8020 2023-03-03T17:36:13.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f1528f3c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15240c7330 con 0x7f152406d4b0 2023-03-03T17:36:13.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f1528f3c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1524154420 con 0x7f15240c1da0 2023-03-03T17:36:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f152259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f152406d4b0 0x7f152406d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:36:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f1521d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f15240c8020 0x7f15240c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56010/0 (socket says 172.21.15.7:56010) 2023-03-03T17:36:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f1521d9b700 1 -- 172.21.15.7:0/3617682899 learned_addr learned my addr 172.21.15.7:0/3617682899 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:36:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f1522d9d700 1 --2- 172.21.15.7:0/3617682899 >> v2:172.21.15.143:3300/0 conn(0x7f15240c1da0 0x7f1524139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:36:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f152259c700 1 -- 172.21.15.7:0/3617682899 >> v2:172.21.15.143:3300/0 conn(0x7f15240c1da0 msgr2=0x7f1524139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:36:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.801+0000 7f152259c700 1 --2- 172.21.15.7:0/3617682899 >> v2:172.21.15.143:3300/0 conn(0x7f15240c1da0 0x7f1524139d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:36:13.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f152259c700 1 -- 172.21.15.7:0/3617682899 >> v1:172.21.15.7:6789/0 conn(0x7f15240c8020 legacy=0x7f15240c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:36:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f152259c700 1 -- 172.21.15.7:0/3617682899 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1524154640 con 0x7f152406d4b0 2023-03-03T17:36:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f152159a700 1 -- 172.21.15.7:0/3617682899 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452522814 0 0) 0x7f15240c7b90 con 0x7f15240c8020 2023-03-03T17:36:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f152259c700 1 --2- 172.21.15.7:0/3617682899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 0x7f152406d880 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f15240c18b0 tx=0x7f1518009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90f48c950acf1c3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:36:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f152159a700 1 -- 172.21.15.7:0/3617682899 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518011830 con 0x7f152406d4b0 2023-03-03T17:36:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f152159a700 1 -- 172.21.15.7:0/3617682899 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1518011990 con 0x7f152406d4b0 2023-03-03T17:36:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f152159a700 1 -- 172.21.15.7:0/3617682899 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518011d20 con 0x7f152406d4b0 2023-03-03T17:36:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f1528f3c700 1 -- 172.21.15.7:0/3617682899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 msgr2=0x7f152406d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:36:13.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f1528f3c700 1 --2- 172.21.15.7:0/3617682899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 0x7f152406d880 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7f15240c18b0 tx=0x7f1518009b20 comp rx=0 tx=0).stop 2023-03-03T17:36:13.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f1528f3c700 1 -- 172.21.15.7:0/3617682899 shutdown_connections 2023-03-03T17:36:13.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f1528f3c700 1 --2- 172.21.15.7:0/3617682899 >> v2:172.21.15.143:3300/0 conn(0x7f15240c1da0 0x7f1524139d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:36:13.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f1528f3c700 1 --2- 172.21.15.7:0/3617682899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 0x7f152406d880 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:36:13.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.802+0000 7f1528f3c700 1 -- 172.21.15.7:0/3617682899 >> 172.21.15.7:0/3617682899 conn(0x7f152406c110 msgr2=0x7f152406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:36:13.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.803+0000 7f1528f3c700 1 -- 172.21.15.7:0/3617682899 shutdown_connections 2023-03-03T17:36:13.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.803+0000 7f1528f3c700 1 -- 172.21.15.7:0/3617682899 wait complete. 2023-03-03T17:36:13.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.804+0000 7f1528f3c700 1 Processor -- start 2023-03-03T17:36:13.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.804+0000 7f1528f3c700 1 -- start start 2023-03-03T17:36:13.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1528f3c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 0x7f152413ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:36:13.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1528f3c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15240c1da0 0x7f1524137280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:36:13.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1528f3c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15240c8020 0x7f15240c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:36:13.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1528f3c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152413a9c0 con 0x7f152406d4b0 2023-03-03T17:36:13.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1528f3c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15241377c0 con 0x7f15240c8020 2023-03-03T17:36:13.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1528f3c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15240ad390 con 0x7f15240c1da0 2023-03-03T17:36:13.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f152259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 0x7f152413ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:36:13.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f152259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 0x7f152413ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33804/0 (socket says 172.21.15.7:33804) 2023-03-03T17:36:13.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f152259c700 1 -- 172.21.15.7:0/449080920 learned_addr learned my addr 172.21.15.7:0/449080920 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:36:13.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1521d9b700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15240c1da0 0x7f1524137280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:36:13.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1522d9d700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15240c8020 0x7f15240c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:36:13.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f152259c700 1 -- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15240c1da0 msgr2=0x7f1524137280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:36:13.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f152259c700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15240c1da0 0x7f1524137280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:36:13.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f152259c700 1 -- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15240c8020 msgr2=0x7f15240c37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:36:13.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f152259c700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15240c8020 0x7f15240c37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:36:13.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f152259c700 1 -- 172.21.15.7:0/449080920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15180094a0 con 0x7f152406d4b0 2023-03-03T17:36:13.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f152259c700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 0x7f152413ae50 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f1518009b60 tx=0x7f1518004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:36:13.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1512ffd700 1 -- 172.21.15.7:0/449080920 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518004550 con 0x7f152406d4b0 2023-03-03T17:36:13.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1512ffd700 1 -- 172.21.15.7:0/449080920 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151801e070 con 0x7f152406d4b0 2023-03-03T17:36:13.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1528f3c700 1 -- 172.21.15.7:0/449080920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15240c7820 con 0x7f152406d4b0 2023-03-03T17:36:13.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1512ffd700 1 -- 172.21.15.7:0/449080920 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15180038e0 con 0x7f152406d4b0 2023-03-03T17:36:13.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.805+0000 7f1528f3c700 1 -- 172.21.15.7:0/449080920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15240c7540 con 0x7f152406d4b0 2023-03-03T17:36:13.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.807+0000 7f1528f3c700 1 -- 172.21.15.7:0/449080920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152406ddc0 con 0x7f152406d4b0 2023-03-03T17:36:13.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.807+0000 7f1512ffd700 1 -- 172.21.15.7:0/449080920 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f151800f050 con 0x7f152406d4b0 2023-03-03T17:36:13.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.807+0000 7f1512ffd700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15040682f0 0x7f150406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:36:13.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.807+0000 7f1512ffd700 1 -- 172.21.15.7:0/449080920 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1518087090 con 0x7f152406d4b0 2023-03-03T17:36:13.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.808+0000 7f1521d9b700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15040682f0 0x7f150406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:36:13.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.808+0000 7f1521d9b700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15040682f0 0x7f150406a7a0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f15240ac830 tx=0x7f1514007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:36:13.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:13.812+0000 7f1512ffd700 1 -- 172.21.15.7:0/449080920 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1518053430 con 0x7f152406d4b0 2023-03-03T17:36:14.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.047+0000 7f1528f3c700 1 -- 172.21.15.7:0/449080920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1524069730 con 0x7f152406d4b0 2023-03-03T17:36:14.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.047+0000 7f1512ffd700 1 -- 172.21.15.7:0/449080920 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1518015020 con 0x7f152406d4b0 2023-03-03T17:36:14.047 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:36:14.047 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:36:14.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 -- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15040682f0 msgr2=0x7f150406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:36:14.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15040682f0 0x7f150406a7a0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f15240ac830 tx=0x7f1514007de0 comp rx=0 tx=0).stop 2023-03-03T17:36:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 -- 172.21.15.7:0/449080920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 msgr2=0x7f152413ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:36:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 0x7f152413ae50 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 crypto rx=0x7f1518009b60 tx=0x7f1518004750 comp rx=0 tx=0).stop 2023-03-03T17:36:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 -- 172.21.15.7:0/449080920 shutdown_connections 2023-03-03T17:36:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15240c8020 0x7f15240c37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:36:14.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f15040682f0 0x7f150406a7a0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:36:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f152406d4b0 0x7f152413ae50 unknown :-1 s=CLOSED pgs=2964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:36:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 --2- 172.21.15.7:0/449080920 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15240c1da0 0x7f1524137280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:36:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 -- 172.21.15.7:0/449080920 >> 172.21.15.7:0/449080920 conn(0x7f152406c110 msgr2=0x7f152406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:36:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 -- 172.21.15.7:0/449080920 shutdown_connections 2023-03-03T17:36:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:36:14.049+0000 7f1528f3c700 1 -- 172.21.15.7:0/449080920 wait complete. 2023-03-03T17:36:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:36:14.060 INFO:tasks.mds_thrash.fs.[b]:waiting for 87 secs before thrashing 2023-03-03T17:36:28.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:28.369 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:58.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:36:58.504 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:28.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:28.575 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:38.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:37:39.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.119+0000 7f9cc5b6d700 1 Processor -- start 2023-03-03T17:37:39.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.119+0000 7f9cc5b6d700 1 -- start start 2023-03-03T17:37:39.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.120+0000 7f9cc5b6d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9cc006d870 0x7f9cc01550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.120+0000 7f9cc5b6d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc00c1da0 0x7f9cc00c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:39.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.120+0000 7f9cc5b6d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9cc00c3e30 0x7f9cc00c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.120+0000 7f9cc5b6d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc00c2af0 con 0x7f9cc006d870 2023-03-03T17:37:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.120+0000 7f9cc5b6d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc00c3620 con 0x7f9cc00c3e30 2023-03-03T17:37:39.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.120+0000 7f9cc5b6d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc01545e0 con 0x7f9cc00c1da0 2023-03-03T17:37:39.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.120+0000 7f9cbf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9cc006d870 0x7f9cc01550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:39.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.120+0000 7f9cbf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9cc006d870 0x7f9cc01550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33874/0 (socket says 172.21.15.7:33874) 2023-03-03T17:37:39.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.120+0000 7f9cbf7fe700 1 -- 172.21.15.7:0/2551933198 learned_addr learned my addr 172.21.15.7:0/2551933198 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:37:39.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbf7fe700 1 -- 172.21.15.7:0/2551933198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc00c1da0 msgr2=0x7f9cc00c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:37:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbf7fe700 1 --2- 172.21.15.7:0/2551933198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc00c1da0 0x7f9cc00c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbf7fe700 1 -- 172.21.15.7:0/2551933198 >> v2:172.21.15.143:3300/0 conn(0x7f9cc00c3e30 msgr2=0x7f9cc00c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:37:39.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbffff700 1 --2- 172.21.15.7:0/2551933198 >> v2:172.21.15.143:3300/0 conn(0x7f9cc00c3e30 0x7f9cc00c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:39.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbf7fe700 1 --2- 172.21.15.7:0/2551933198 >> v2:172.21.15.143:3300/0 conn(0x7f9cc00c3e30 0x7f9cc00c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbf7fe700 1 -- 172.21.15.7:0/2551933198 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc01548c0 con 0x7f9cc006d870 2023-03-03T17:37:39.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbffff700 1 --2- 172.21.15.7:0/2551933198 >> v2:172.21.15.143:3300/0 conn(0x7f9cc00c3e30 0x7f9cc00c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:37:39.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbf7fe700 1 --2- 172.21.15.7:0/2551933198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 0x7f9cc01550c0 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7f9cc006be80 tx=0x7f9cb0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e051543c78775 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:37:39.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbe7fc700 1 -- 172.21.15.7:0/2551933198 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb000b870 con 0x7f9cc006d870 2023-03-03T17:37:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbe7fc700 1 -- 172.21.15.7:0/2551933198 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb000b9d0 con 0x7f9cc006d870 2023-03-03T17:37:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cbe7fc700 1 -- 172.21.15.7:0/2551933198 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb000bd60 con 0x7f9cc006d870 2023-03-03T17:37:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/2551933198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 msgr2=0x7f9cc01550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.121+0000 7f9cc5b6d700 1 --2- 172.21.15.7:0/2551933198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 0x7f9cc01550c0 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7f9cc006be80 tx=0x7f9cb0009bd0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.122+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/2551933198 shutdown_connections 2023-03-03T17:37:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.122+0000 7f9cc5b6d700 1 --2- 172.21.15.7:0/2551933198 >> v2:172.21.15.143:3300/0 conn(0x7f9cc00c3e30 0x7f9cc00c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.122+0000 7f9cc5b6d700 1 --2- 172.21.15.7:0/2551933198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 0x7f9cc01550c0 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.122+0000 7f9cc5b6d700 1 --2- 172.21.15.7:0/2551933198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc00c1da0 0x7f9cc00c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.122+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/2551933198 >> 172.21.15.7:0/2551933198 conn(0x7f9cc006c110 msgr2=0x7f9cc006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:37:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.122+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/2551933198 shutdown_connections 2023-03-03T17:37:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.122+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/2551933198 wait complete. 2023-03-03T17:37:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.123+0000 7f9cc5b6d700 1 Processor -- start 2023-03-03T17:37:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.123+0000 7f9cc5b6d700 1 -- start start 2023-03-03T17:37:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cc5b6d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 0x7f9cc0146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cc5b6d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc00c1da0 0x7f9cc0157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cc5b6d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cc00c3e30 0x7f9cc01562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cc5b6d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc00acf00 con 0x7f9cc006d870 2023-03-03T17:37:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cc5b6d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc01558a0 con 0x7f9cc00c3e30 2023-03-03T17:37:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cc5b6d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc01eab30 con 0x7f9cc00c1da0 2023-03-03T17:37:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 0x7f9cc0146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 0x7f9cc0146840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33880/0 (socket says 172.21.15.7:33880) 2023-03-03T17:37:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbf7fe700 1 -- 172.21.15.7:0/1683156622 learned_addr learned my addr 172.21.15.7:0/1683156622 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:37:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbffff700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cc00c3e30 0x7f9cc01562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbeffd700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc00c1da0 0x7f9cc0157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbf7fe700 1 -- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc00c1da0 msgr2=0x7f9cc0157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbf7fe700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc00c1da0 0x7f9cc0157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbf7fe700 1 -- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cc00c3e30 msgr2=0x7f9cc01562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbf7fe700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cc00c3e30 0x7f9cc01562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbf7fe700 1 -- 172.21.15.7:0/1683156622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb0009550 con 0x7f9cc006d870 2023-03-03T17:37:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbeffd700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc00c1da0 0x7f9cc0157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:37:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cbf7fe700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 0x7f9cc0146840 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00c4a90 tx=0x7f9cb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:37:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.124+0000 7f9cb7fff700 1 -- 172.21.15.7:0/1683156622 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0004310 con 0x7f9cc006d870 2023-03-03T17:37:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.125+0000 7f9cb7fff700 1 -- 172.21.15.7:0/1683156622 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb001f070 con 0x7f9cc006d870 2023-03-03T17:37:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.125+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/1683156622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc006dc40 con 0x7f9cc006d870 2023-03-03T17:37:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.125+0000 7f9cb7fff700 1 -- 172.21.15.7:0/1683156622 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb00036a0 con 0x7f9cc006d870 2023-03-03T17:37:39.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.125+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/1683156622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc0154ba0 con 0x7f9cc006d870 2023-03-03T17:37:39.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.126+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/1683156622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc00c38e0 con 0x7f9cc006d870 2023-03-03T17:37:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.126+0000 7f9cb7fff700 1 -- 172.21.15.7:0/1683156622 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9cb002a020 con 0x7f9cc006d870 2023-03-03T17:37:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.127+0000 7f9cb7fff700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ca00682f0 0x7f9ca006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.127+0000 7f9cb7fff700 1 -- 172.21.15.7:0/1683156622 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9cb0087790 con 0x7f9cc006d870 2023-03-03T17:37:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.127+0000 7f9cbeffd700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ca00682f0 0x7f9ca006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.128+0000 7f9cbeffd700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ca00682f0 0x7f9ca006a7a0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0158830 tx=0x7f9ca8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:37:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.131+0000 7f9cb7fff700 1 -- 172.21.15.7:0/1683156622 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9cb0053b30 con 0x7f9cc006d870 2023-03-03T17:37:39.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.369+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/1683156622 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cc0074bb0 con 0x7f9cc006d870 2023-03-03T17:37:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.370+0000 7f9cb7fff700 1 -- 172.21.15.7:0/1683156622 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9cb0057150 con 0x7f9cc006d870 2023-03-03T17:37:39.371 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:37:39.371 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:37:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ca00682f0 msgr2=0x7f9ca006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ca00682f0 0x7f9ca006a7a0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0158830 tx=0x7f9ca8008040 comp rx=0 tx=0).stop 2023-03-03T17:37:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/1683156622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 msgr2=0x7f9cc0146840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 0x7f9cc0146840 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00c4a90 tx=0x7f9cb0004750 comp rx=0 tx=0).stop 2023-03-03T17:37:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/1683156622 shutdown_connections 2023-03-03T17:37:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9cc00c3e30 0x7f9cc01562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9ca00682f0 0x7f9ca006a7a0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cc006d870 0x7f9cc0146840 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 --2- 172.21.15.7:0/1683156622 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9cc00c1da0 0x7f9cc0157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/1683156622 >> 172.21.15.7:0/1683156622 conn(0x7f9cc006c110 msgr2=0x7f9cc014ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:37:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/1683156622 shutdown_connections 2023-03-03T17:37:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:39.372+0000 7f9cc5b6d700 1 -- 172.21.15.7:0/1683156622 wait complete. 2023-03-03T17:37:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:37:39.386 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T17:37:41.061 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:37:41.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.201+0000 7f726d8ff700 1 Processor -- start 2023-03-03T17:37:41.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.201+0000 7f726d8ff700 1 -- start start 2023-03-03T17:37:41.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.202+0000 7f726d8ff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72680c1da0 0x7f72681550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:41.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.202+0000 7f726d8ff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f72680c3e30 0x7f72680c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:41.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.202+0000 7f726d8ff700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72680c7a60 con 0x7f726806d870 2023-03-03T17:37:41.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.202+0000 7f726d8ff700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72680c2af0 con 0x7f72680c3e30 2023-03-03T17:37:41.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.202+0000 7f726d8ff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72680c3620 con 0x7f72680c1da0 2023-03-03T17:37:41.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.202+0000 7f72677fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f72680c3e30 0x7f72680c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:41.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.202+0000 7f7266ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f726806d870 0x7f72680c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56100/0 (socket says 172.21.15.7:56100) 2023-03-03T17:37:41.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.202+0000 7f7266ffd700 1 -- 172.21.15.7:0/1436107426 learned_addr learned my addr 172.21.15.7:0/1436107426 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:37:41.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.203+0000 7f72667fc700 1 --2- 172.21.15.7:0/1436107426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72680c1da0 0x7f72681550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:41.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.203+0000 7f72677fe700 1 -- 172.21.15.7:0/1436107426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72680c1da0 msgr2=0x7f72681550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:37:41.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.203+0000 7f72677fe700 1 --2- 172.21.15.7:0/1436107426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72680c1da0 0x7f72681550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:41.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.203+0000 7f72677fe700 1 -- 172.21.15.7:0/1436107426 >> v1:172.21.15.7:6789/0 conn(0x7f726806d870 legacy=0x7f72680c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:41.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.203+0000 7f72677fe700 1 -- 172.21.15.7:0/1436107426 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72681545e0 con 0x7f72680c3e30 2023-03-03T17:37:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.203+0000 7f7265ffb700 1 -- 172.21.15.7:0/1436107426 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365985578 0 0) 0x7f72680c7a60 con 0x7f726806d870 2023-03-03T17:37:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.203+0000 7f72677fe700 1 --2- 172.21.15.7:0/1436107426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c3e30 0x7f72680c4290 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7f7260009d90 tx=0x7f726000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9339fa30b9cbc7ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:37:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.203+0000 7f7265ffb700 1 -- 172.21.15.7:0/1436107426 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726000e070 con 0x7f72680c3e30 2023-03-03T17:37:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.204+0000 7f7265ffb700 1 -- 172.21.15.7:0/1436107426 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726000bbd0 con 0x7f72680c3e30 2023-03-03T17:37:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.204+0000 7f7265ffb700 1 -- 172.21.15.7:0/1436107426 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260012430 con 0x7f72680c3e30 2023-03-03T17:37:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.204+0000 7f726d8ff700 1 -- 172.21.15.7:0/1436107426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c3e30 msgr2=0x7f72680c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.204+0000 7f726d8ff700 1 --2- 172.21.15.7:0/1436107426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c3e30 0x7f72680c4290 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7f7260009d90 tx=0x7f726000b330 comp rx=0 tx=0).stop 2023-03-03T17:37:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.204+0000 7f726d8ff700 1 -- 172.21.15.7:0/1436107426 shutdown_connections 2023-03-03T17:37:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.204+0000 7f726d8ff700 1 --2- 172.21.15.7:0/1436107426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c3e30 0x7f72680c4290 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.204+0000 7f726d8ff700 1 --2- 172.21.15.7:0/1436107426 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f72680c1da0 0x7f72681550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.204+0000 7f726d8ff700 1 -- 172.21.15.7:0/1436107426 >> 172.21.15.7:0/1436107426 conn(0x7f726806c110 msgr2=0x7f726806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:37:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.204+0000 7f726d8ff700 1 -- 172.21.15.7:0/1436107426 shutdown_connections 2023-03-03T17:37:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.204+0000 7f726d8ff700 1 -- 172.21.15.7:0/1436107426 wait complete. 2023-03-03T17:37:41.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.206+0000 7f726d8ff700 1 Processor -- start 2023-03-03T17:37:41.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.206+0000 7f726d8ff700 1 -- start start 2023-03-03T17:37:41.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f726d8ff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f726806d870 0x7f7268149190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:41.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f726d8ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c1da0 0x7f726814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:41.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f726d8ff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72680c3e30 0x7f7268150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:41.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f726d8ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7268152d50 con 0x7f72680c1da0 2023-03-03T17:37:41.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f726d8ff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7268149aa0 con 0x7f72680c3e30 2023-03-03T17:37:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f726d8ff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72681454b0 con 0x7f726806d870 2023-03-03T17:37:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f72667fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c1da0 0x7f726814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f72667fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c1da0 0x7f726814fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33896/0 (socket says 172.21.15.7:33896) 2023-03-03T17:37:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f72667fc700 1 -- 172.21.15.7:0/3760104986 learned_addr learned my addr 172.21.15.7:0/3760104986 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:37:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f7266ffd700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f726806d870 0x7f7268149190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.207+0000 7f72677fe700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72680c3e30 0x7f7268150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:41.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f72667fc700 1 -- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f726806d870 msgr2=0x7f7268149190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:41.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f72667fc700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f726806d870 0x7f7268149190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:41.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f72667fc700 1 -- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72680c3e30 msgr2=0x7f7268150120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:41.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f72667fc700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72680c3e30 0x7f7268150120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:41.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f72667fc700 1 -- 172.21.15.7:0/3760104986 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7260009a70 con 0x7f72680c1da0 2023-03-03T17:37:41.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f72667fc700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c1da0 0x7f726814fbe0 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f725000e810 tx=0x7f725000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:37:41.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f725ffff700 1 -- 172.21.15.7:0/3760104986 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725000c070 con 0x7f72680c1da0 2023-03-03T17:37:41.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f725ffff700 1 -- 172.21.15.7:0/3760104986 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7250013610 con 0x7f72680c1da0 2023-03-03T17:37:41.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f726d8ff700 1 -- 172.21.15.7:0/3760104986 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7268154ba0 con 0x7f72680c1da0 2023-03-03T17:37:41.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f725ffff700 1 -- 172.21.15.7:0/3760104986 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72500139c0 con 0x7f72680c1da0 2023-03-03T17:37:41.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.208+0000 7f726d8ff700 1 -- 172.21.15.7:0/3760104986 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72681548c0 con 0x7f72680c1da0 2023-03-03T17:37:41.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.209+0000 7f726d8ff700 1 -- 172.21.15.7:0/3760104986 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7268145790 con 0x7f72680c1da0 2023-03-03T17:37:41.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.209+0000 7f725ffff700 1 -- 172.21.15.7:0/3760104986 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7250023070 con 0x7f72680c1da0 2023-03-03T17:37:41.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.210+0000 7f725ffff700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72480682f0 0x7f724806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:37:41.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.210+0000 7f725ffff700 1 -- 172.21.15.7:0/3760104986 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7250085f20 con 0x7f72680c1da0 2023-03-03T17:37:41.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.210+0000 7f7266ffd700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72480682f0 0x7f724806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:37:41.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.211+0000 7f7266ffd700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72480682f0 0x7f724806a7a0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f7258007fd0 tx=0x7f7258007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:37:41.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.214+0000 7f725ffff700 1 -- 172.21.15.7:0/3760104986 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7250052240 con 0x7f72680c1da0 2023-03-03T17:37:41.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.452+0000 7f726d8ff700 1 -- 172.21.15.7:0/3760104986 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f726806dc40 con 0x7f72680c1da0 2023-03-03T17:37:41.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.453+0000 7f725ffff700 1 -- 172.21.15.7:0/3760104986 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7250059e60 con 0x7f72680c1da0 2023-03-03T17:37:41.453 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:37:41.453 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:37:41.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 -- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72480682f0 msgr2=0x7f724806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:41.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72480682f0 0x7f724806a7a0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f7258007fd0 tx=0x7f7258007de0 comp rx=0 tx=0).stop 2023-03-03T17:37:41.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 -- 172.21.15.7:0/3760104986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c1da0 msgr2=0x7f726814fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:37:41.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c1da0 0x7f726814fbe0 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f725000e810 tx=0x7f725000eb20 comp rx=0 tx=0).stop 2023-03-03T17:37:41.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 -- 172.21.15.7:0/3760104986 shutdown_connections 2023-03-03T17:37:41.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72680c3e30 0x7f7268150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:41.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f726806d870 0x7f7268149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:41.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f72480682f0 0x7f724806a7a0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:41.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 --2- 172.21.15.7:0/3760104986 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72680c1da0 0x7f726814fbe0 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:37:41.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 -- 172.21.15.7:0/3760104986 >> 172.21.15.7:0/3760104986 conn(0x7f726806c110 msgr2=0x7f7268153ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:37:41.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 -- 172.21.15.7:0/3760104986 shutdown_connections 2023-03-03T17:37:41.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:37:41.455+0000 7f726d8ff700 1 -- 172.21.15.7:0/3760104986 wait complete. 2023-03-03T17:37:41.456 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:37:41.466 INFO:tasks.mds_thrash.fs.[b]:waiting for 50 secs before thrashing 2023-03-03T17:37:58.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:37:58.647 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:28.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:28.792 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:31.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:38:31.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.607+0000 7f3983166700 1 Processor -- start 2023-03-03T17:38:31.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.607+0000 7f3983166700 1 -- start start 2023-03-03T17:38:31.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f3983166700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f397c0c5cb0 0x7f397c06a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f3983166700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c1558f0 con 0x7f397c06a2b0 2023-03-03T17:38:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f3983166700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c0a6180 con 0x7f397c0a6370 2023-03-03T17:38:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f3983166700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397c06bab0 con 0x7f397c0c5cb0 2023-03-03T17:38:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f3980f02700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f397c06a2b0 0x7f397c155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56138/0 (socket says 172.21.15.7:56138) 2023-03-03T17:38:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f3980f02700 1 -- 172.21.15.7:0/195813550 learned_addr learned my addr 172.21.15.7:0/195813550 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:38:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f3981703700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f397c0c5cb0 0x7f397c06a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153093272 0 0) 0x7f397c1558f0 con 0x7f397c06a2b0 2023-03-03T17:38:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f396c0035b0 con 0x7f397c06a2b0 2023-03-03T17:38:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2873767558 0 0) 0x7f396c0035b0 con 0x7f397c06a2b0 2023-03-03T17:38:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f397c0c5cb0 msgr2=0x7f397c06a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:38:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f397b7fe700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f397c0c5cb0 0x7f397c06a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.608+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 >> v1:172.21.15.143:6789/0 conn(0x7f397c0a6370 legacy=0x7f397c15abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c0a5ea0 con 0x7f397c06a2b0 2023-03-03T17:38:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f3970001670 con 0x7f397c06a2b0 2023-03-03T17:38:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 >> v1:172.21.15.7:6789/0 conn(0x7f397c06a2b0 legacy=0x7f397c155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f397c0c5cb0 0x7f396c003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f396c003fe0 0x7f396c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f396c004d90 0x7f396c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c00bba0 con 0x7f396c003fe0 2023-03-03T17:38:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c00bce0 con 0x7f396c004d90 2023-03-03T17:38:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c00be70 con 0x7f397c0c5cb0 2023-03-03T17:38:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397bfff700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f396c003fe0 0x7f396c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f3981703700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f396c004d90 0x7f396c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f3980f02700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f397c0c5cb0 0x7f396c003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397bfff700 1 -- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f397c0c5cb0 msgr2=0x7f396c003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:38:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397bfff700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f397c0c5cb0 0x7f396c003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397bfff700 1 -- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f396c004d90 msgr2=0x7f396c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:38:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397bfff700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f396c004d90 0x7f396c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397bfff700 1 -- 172.21.15.7:0/195813550 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c0035b0 con 0x7f396c003fe0 2023-03-03T17:38:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397bfff700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f396c003fe0 0x7f396c004850 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7f39680018d0 tx=0x7f396c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34233aa1195a1cb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:38:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396800c650 con 0x7f396c003fe0 2023-03-03T17:38:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396800c7b0 con 0x7f396c003fe0 2023-03-03T17:38:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.609+0000 7f397b7fe700 1 -- 172.21.15.7:0/195813550 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396800cb60 con 0x7f396c003fe0 2023-03-03T17:38:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.610+0000 7f3983166700 1 -- 172.21.15.7:0/195813550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f396c003fe0 msgr2=0x7f396c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.610+0000 7f3983166700 1 --2- 172.21.15.7:0/195813550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f396c003fe0 0x7f396c004850 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7f39680018d0 tx=0x7f396c005880 comp rx=0 tx=0).stop 2023-03-03T17:38:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.610+0000 7f3981703700 1 -- 172.21.15.7:0/195813550 reap_dead start 2023-03-03T17:38:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.610+0000 7f3983166700 1 -- 172.21.15.7:0/195813550 shutdown_connections 2023-03-03T17:38:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.610+0000 7f3983166700 1 -- 172.21.15.7:0/195813550 >> 172.21.15.7:0/195813550 conn(0x7f397c0a9430 msgr2=0x7f397c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:38:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.610+0000 7f3983166700 1 -- 172.21.15.7:0/195813550 shutdown_connections 2023-03-03T17:38:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.610+0000 7f3983166700 1 -- 172.21.15.7:0/195813550 wait complete. 2023-03-03T17:38:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.611+0000 7f3983166700 1 Processor -- start 2023-03-03T17:38:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.611+0000 7f3983166700 1 -- start start 2023-03-03T17:38:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3983166700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f396c003fe0 0x7f397c0ab550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3983166700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f397c1549b0 0x7f397c0aa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3983166700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397c155230 0x7f397c14b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3983166700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c1ed910 con 0x7f397c155230 2023-03-03T17:38:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3983166700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0aba90 con 0x7f397c1549b0 2023-03-03T17:38:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3983166700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397c1ecd20 con 0x7f396c003fe0 2023-03-03T17:38:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3981703700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397c155230 0x7f397c14b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3981703700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397c155230 0x7f397c14b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33942/0 (socket says 172.21.15.7:33942) 2023-03-03T17:38:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3981703700 1 -- 172.21.15.7:0/9172585 learned_addr learned my addr 172.21.15.7:0/9172585 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:38:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3980f02700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f396c003fe0 0x7f397c0ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f397bfff700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f397c1549b0 0x7f397c0aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:31.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3981703700 1 -- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f396c003fe0 msgr2=0x7f397c0ab550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:31.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3981703700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f396c003fe0 0x7f397c0ab550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:31.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3981703700 1 -- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f397c1549b0 msgr2=0x7f397c0aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:31.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3981703700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f397c1549b0 0x7f397c0aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:31.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3981703700 1 -- 172.21.15.7:0/9172585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c153160 con 0x7f397c155230 2023-03-03T17:38:31.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f3980f02700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f396c003fe0 0x7f397c0ab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:38:31.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.612+0000 7f397bfff700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f397c1549b0 0x7f397c0aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:38:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.613+0000 7f3981703700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397c155230 0x7f397c14b650 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f397c0c7530 tx=0x7f3974011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:38:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.613+0000 7f39797fa700 1 -- 172.21.15.7:0/9172585 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397400e040 con 0x7f397c155230 2023-03-03T17:38:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.613+0000 7f39797fa700 1 -- 172.21.15.7:0/9172585 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397400f040 con 0x7f397c155230 2023-03-03T17:38:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.613+0000 7f3983166700 1 -- 172.21.15.7:0/9172585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c152e80 con 0x7f397c155230 2023-03-03T17:38:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.613+0000 7f39797fa700 1 -- 172.21.15.7:0/9172585 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397400fdb0 con 0x7f397c155230 2023-03-03T17:38:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.613+0000 7f3983166700 1 -- 172.21.15.7:0/9172585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c0c1500 con 0x7f397c155230 2023-03-03T17:38:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.614+0000 7f3983166700 1 -- 172.21.15.7:0/9172585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c074bb0 con 0x7f397c155230 2023-03-03T17:38:31.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.614+0000 7f39797fa700 1 -- 172.21.15.7:0/9172585 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3974015430 con 0x7f397c155230 2023-03-03T17:38:31.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.615+0000 7f39797fa700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 0x7f396006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:31.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.615+0000 7f39797fa700 1 -- 172.21.15.7:0/9172585 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3974086bb0 con 0x7f397c155230 2023-03-03T17:38:31.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.615+0000 7f3980f02700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 0x7f396006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:31.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.615+0000 7f3980f02700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 0x7f396006a7a0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f39700011e0 tx=0x7f3970009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:38:31.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.619+0000 7f39797fa700 1 -- 172.21.15.7:0/9172585 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f397408d080 con 0x7f397c155230 2023-03-03T17:38:31.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.857+0000 7f3983166700 1 -- 172.21.15.7:0/9172585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f397c1eda50 con 0x7f397c155230 2023-03-03T17:38:31.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.858+0000 7f39797fa700 1 -- 172.21.15.7:0/9172585 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f397c1eda50 con 0x7f397c155230 2023-03-03T17:38:31.858 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:38:31.858 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:38:31.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 -- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 msgr2=0x7f396006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:31.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 0x7f396006a7a0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f39700011e0 tx=0x7f3970009040 comp rx=0 tx=0).stop 2023-03-03T17:38:31.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 -- 172.21.15.7:0/9172585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397c155230 msgr2=0x7f397c14b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:31.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397c155230 0x7f397c14b650 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f397c0c7530 tx=0x7f3974011f80 comp rx=0 tx=0).stop 2023-03-03T17:38:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 -- 172.21.15.7:0/9172585 shutdown_connections 2023-03-03T17:38:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f39600682f0 0x7f396006a7a0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397c155230 0x7f397c14b650 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f396c003fe0 0x7f397c0ab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 --2- 172.21.15.7:0/9172585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f397c1549b0 0x7f397c0aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 -- 172.21.15.7:0/9172585 >> 172.21.15.7:0/9172585 conn(0x7f397c0a9430 msgr2=0x7f397c06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:38:31.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 -- 172.21.15.7:0/9172585 shutdown_connections 2023-03-03T17:38:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:31.860+0000 7f3983166700 1 -- 172.21.15.7:0/9172585 wait complete. 2023-03-03T17:38:31.861 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:38:31.871 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T17:38:43.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:38:43.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.527+0000 7fc7d56b1700 1 Processor -- start 2023-03-03T17:38:43.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.527+0000 7fc7d56b1700 1 -- start start 2023-03-03T17:38:43.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.527+0000 7fc7d56b1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7d00736f0 0x7fc7d0073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:43.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.527+0000 7fc7d56b1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d006c3e0 con 0x7fc7d006c910 2023-03-03T17:38:43.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.527+0000 7fc7d56b1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d00c32e0 con 0x7fc7d006bea0 2023-03-03T17:38:43.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.527+0000 7fc7d56b1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d006c630 con 0x7fc7d00736f0 2023-03-03T17:38:43.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7c67fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc7d006c910 0x7fc7d0157d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56160/0 (socket says 172.21.15.7:56160) 2023-03-03T17:38:43.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7ceffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7d00736f0 0x7fc7d0073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:43.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7c67fc700 1 -- 172.21.15.7:0/2278555412 learned_addr learned my addr 172.21.15.7:0/2278555412 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:38:43.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7ceffd700 1 -- 172.21.15.7:0/2278555412 >> v1:172.21.15.143:6789/0 conn(0x7fc7d006bea0 legacy=0x7fc7d015eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:38:43.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7ceffd700 1 -- 172.21.15.7:0/2278555412 >> v1:172.21.15.7:6789/0 conn(0x7fc7d006c910 legacy=0x7fc7d0157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:43.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7ceffd700 1 -- 172.21.15.7:0/2278555412 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d00c3880 con 0x7fc7d00736f0 2023-03-03T17:38:43.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7ce7fc700 1 -- 172.21.15.7:0/2278555412 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767136182 0 0) 0x7fc7d006c3e0 con 0x7fc7d006c910 2023-03-03T17:38:43.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7ceffd700 1 --2- 172.21.15.7:0/2278555412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 0x7fc7d0073ac0 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009840 tx=0x7fc7c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c85cfadb7f577cd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:38:43.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7ce7fc700 1 -- 172.21.15.7:0/2278555412 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0004230 con 0x7fc7d00736f0 2023-03-03T17:38:43.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7ce7fc700 1 -- 172.21.15.7:0/2278555412 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c0004390 con 0x7fc7d00736f0 2023-03-03T17:38:43.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.528+0000 7fc7ce7fc700 1 -- 172.21.15.7:0/2278555412 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00036a0 con 0x7fc7d00736f0 2023-03-03T17:38:43.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.529+0000 7fc7d56b1700 1 -- 172.21.15.7:0/2278555412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 msgr2=0x7fc7d0073ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:43.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.529+0000 7fc7d56b1700 1 --2- 172.21.15.7:0/2278555412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 0x7fc7d0073ac0 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009840 tx=0x7fc7c0009b50 comp rx=0 tx=0).stop 2023-03-03T17:38:43.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.529+0000 7fc7d56b1700 1 -- 172.21.15.7:0/2278555412 shutdown_connections 2023-03-03T17:38:43.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.529+0000 7fc7d56b1700 1 --2- 172.21.15.7:0/2278555412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 0x7fc7d0073ac0 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:43.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.529+0000 7fc7d56b1700 1 -- 172.21.15.7:0/2278555412 >> 172.21.15.7:0/2278555412 conn(0x7fc7d00a94d0 msgr2=0x7fc7d00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:38:43.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.529+0000 7fc7d56b1700 1 -- 172.21.15.7:0/2278555412 shutdown_connections 2023-03-03T17:38:43.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.529+0000 7fc7d56b1700 1 -- 172.21.15.7:0/2278555412 wait complete. 2023-03-03T17:38:43.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.530+0000 7fc7d56b1700 1 Processor -- start 2023-03-03T17:38:43.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.530+0000 7fc7d56b1700 1 -- start start 2023-03-03T17:38:43.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7d56b1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d006bea0 0x7fc7d01eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:43.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7d56b1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d006c910 0x7fc7d0156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:43.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7d56b1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 0x7fc7d0153b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:43.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7d56b1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d01ea010 con 0x7fc7d00736f0 2023-03-03T17:38:43.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7d56b1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d01ee030 con 0x7fc7d006bea0 2023-03-03T17:38:43.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7d56b1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d01ee310 con 0x7fc7d006c910 2023-03-03T17:38:43.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 0x7fc7d0153b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:43.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 0x7fc7d0153b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33956/0 (socket says 172.21.15.7:33956) 2023-03-03T17:38:43.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7cf7fe700 1 -- 172.21.15.7:0/3332181665 learned_addr learned my addr 172.21.15.7:0/3332181665 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:38:43.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7ceffd700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d006bea0 0x7fc7d01eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:43.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7c67fc700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d006c910 0x7fc7d0156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:43.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7cf7fe700 1 -- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d006c910 msgr2=0x7fc7d0156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:43.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d006c910 0x7fc7d0156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:43.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7cf7fe700 1 -- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d006bea0 msgr2=0x7fc7d01eaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:43.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d006bea0 0x7fc7d01eaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:43.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.531+0000 7fc7cf7fe700 1 -- 172.21.15.7:0/3332181665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c00094a0 con 0x7fc7d00736f0 2023-03-03T17:38:43.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.532+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 0x7fc7d0153b10 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800cc10 tx=0x7fc7c800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:38:43.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.532+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3332181665 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c800aa60 con 0x7fc7d00736f0 2023-03-03T17:38:43.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.532+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3332181665 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c800abc0 con 0x7fc7d00736f0 2023-03-03T17:38:43.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.532+0000 7fc7d56b1700 1 -- 172.21.15.7:0/3332181665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d00c3580 con 0x7fc7d00736f0 2023-03-03T17:38:43.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.532+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3332181665 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8012420 con 0x7fc7d00736f0 2023-03-03T17:38:43.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.532+0000 7fc7d56b1700 1 -- 172.21.15.7:0/3332181665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d00c3b60 con 0x7fc7d00736f0 2023-03-03T17:38:43.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.533+0000 7fc7d56b1700 1 -- 172.21.15.7:0/3332181665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d0151e40 con 0x7fc7d00736f0 2023-03-03T17:38:43.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.533+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3332181665 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc7c8010050 con 0x7fc7d00736f0 2023-03-03T17:38:43.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.534+0000 7fc7c7fff700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:38:43.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.534+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3332181665 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc7c80862f0 con 0x7fc7d00736f0 2023-03-03T17:38:43.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.534+0000 7fc7ceffd700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:38:43.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.535+0000 7fc7ceffd700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00090b0 tx=0x7fc7c0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:38:43.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.538+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3332181665 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc7c80525a0 con 0x7fc7d00736f0 2023-03-03T17:38:43.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.776+0000 7fc7d56b1700 1 -- 172.21.15.7:0/3332181665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7d01ee4d0 con 0x7fc7d00736f0 2023-03-03T17:38:43.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.776+0000 7fc7c7fff700 1 -- 172.21.15.7:0/3332181665 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc7d01ee4d0 con 0x7fc7d00736f0 2023-03-03T17:38:43.777 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:38:43.777 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:38:43.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 -- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 msgr2=0x7fc7b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:43.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00090b0 tx=0x7fc7c0008030 comp rx=0 tx=0).stop 2023-03-03T17:38:43.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 -- 172.21.15.7:0/3332181665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 msgr2=0x7fc7d0153b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:38:43.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 0x7fc7d0153b10 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800cc10 tx=0x7fc7c800cf20 comp rx=0 tx=0).stop 2023-03-03T17:38:43.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 -- 172.21.15.7:0/3332181665 shutdown_connections 2023-03-03T17:38:43.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:43.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d00736f0 0x7fc7d0153b10 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:43.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d006bea0 0x7fc7d01eaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:43.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 --2- 172.21.15.7:0/3332181665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d006c910 0x7fc7d0156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:38:43.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 -- 172.21.15.7:0/3332181665 >> 172.21.15.7:0/3332181665 conn(0x7fc7d00a94d0 msgr2=0x7fc7d014ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:38:43.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 -- 172.21.15.7:0/3332181665 shutdown_connections 2023-03-03T17:38:43.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:38:43.778+0000 7fc7d56b1700 1 -- 172.21.15.7:0/3332181665 wait complete. 2023-03-03T17:38:43.780 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:38:43.790 INFO:tasks.mds_thrash.fs.[a]:waiting for 87 secs before thrashing 2023-03-03T17:38:58.885 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:38:58.889 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:29.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:29.063 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:33.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:39:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.012+0000 7f591e6ba700 1 Processor -- start 2023-03-03T17:39:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.012+0000 7f591e6ba700 1 -- start start 2023-03-03T17:39:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f591e6ba700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59180aac00 0x7f59180c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:39:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f591e6ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5918071c30 0x7f59180beb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:39:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f591e6ba700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59180713c0 con 0x7f59180bf0a0 2023-03-03T17:39:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f591e6ba700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5918152c70 con 0x7f59180aac00 2023-03-03T17:39:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f591e6ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59180c4930 con 0x7f5918071c30 2023-03-03T17:39:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f5917fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59180aac00 0x7f59180c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:39:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f5917fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f59180aac00 0x7f59180c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34004/0 (socket says 172.21.15.7:34004) 2023-03-03T17:39:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f5917fff700 1 -- 172.21.15.7:0/285627571 learned_addr learned my addr 172.21.15.7:0/285627571 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:39:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f59177fe700 1 --2- 172.21.15.7:0/285627571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5918071c30 0x7f59180beb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:39:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f5917fff700 1 -- 172.21.15.7:0/285627571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5918071c30 msgr2=0x7f59180beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:39:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f5917fff700 1 --2- 172.21.15.7:0/285627571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5918071c30 0x7f59180beb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:39:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f5917fff700 1 -- 172.21.15.7:0/285627571 >> v1:172.21.15.143:6789/0 conn(0x7f59180bf0a0 legacy=0x7f5918155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:39:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.013+0000 7f5917fff700 1 -- 172.21.15.7:0/285627571 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5918152360 con 0x7f59180aac00 2023-03-03T17:39:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.014+0000 7f5917fff700 1 --2- 172.21.15.7:0/285627571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 0x7f59180c43f0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f59080098f0 tx=0x7f5908009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780a869815ef458 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:39:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.014+0000 7f5916ffd700 1 -- 172.21.15.7:0/285627571 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908004230 con 0x7f59180aac00 2023-03-03T17:39:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.014+0000 7f5916ffd700 1 -- 172.21.15.7:0/285627571 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5908004390 con 0x7f59180aac00 2023-03-03T17:39:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.014+0000 7f5916ffd700 1 -- 172.21.15.7:0/285627571 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59080036a0 con 0x7f59180aac00 2023-03-03T17:39:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.014+0000 7f591e6ba700 1 -- 172.21.15.7:0/285627571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 msgr2=0x7f59180c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:39:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.014+0000 7f591e6ba700 1 --2- 172.21.15.7:0/285627571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 0x7f59180c43f0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f59080098f0 tx=0x7f5908009c00 comp rx=0 tx=0).stop 2023-03-03T17:39:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.014+0000 7f591e6ba700 1 -- 172.21.15.7:0/285627571 shutdown_connections 2023-03-03T17:39:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.014+0000 7f591e6ba700 1 --2- 172.21.15.7:0/285627571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 0x7f59180c43f0 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:39:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.014+0000 7f591e6ba700 1 --2- 172.21.15.7:0/285627571 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5918071c30 0x7f59180beb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:39:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.014+0000 7f591e6ba700 1 -- 172.21.15.7:0/285627571 >> 172.21.15.7:0/285627571 conn(0x7f59180736f0 msgr2=0x7f5918073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:39:34.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.015+0000 7f591e6ba700 1 -- 172.21.15.7:0/285627571 shutdown_connections 2023-03-03T17:39:34.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.015+0000 7f591e6ba700 1 -- 172.21.15.7:0/285627571 wait complete. 2023-03-03T17:39:34.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.016+0000 7f591e6ba700 1 Processor -- start 2023-03-03T17:39:34.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.016+0000 7f591e6ba700 1 -- start start 2023-03-03T17:39:34.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.016+0000 7f591e6ba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5918071c30 0x7f59181eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:39:34.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f591e6ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 0x7f59181547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:39:34.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f591e6ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59180bf0a0 0x7f59180c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:39:34.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f591e6ba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59181555f0 con 0x7f59180aac00 2023-03-03T17:39:34.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f591e6ba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59181e7620 con 0x7f5918071c30 2023-03-03T17:39:34.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f591e6ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59181eba50 con 0x7f59180bf0a0 2023-03-03T17:39:34.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f59177fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 0x7f59181547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:39:34.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f59177fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 0x7f59181547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34012/0 (socket says 172.21.15.7:34012) 2023-03-03T17:39:34.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f59177fe700 1 -- 172.21.15.7:0/4274288190 learned_addr learned my addr 172.21.15.7:0/4274288190 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:39:34.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f591cc57700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59180bf0a0 0x7f59180c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:39:34.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f5917fff700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5918071c30 0x7f59181eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:39:34.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f59177fe700 1 -- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59180bf0a0 msgr2=0x7f59180c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:39:34.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f59177fe700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59180bf0a0 0x7f59180c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:39:34.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f59177fe700 1 -- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5918071c30 msgr2=0x7f59181eb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:39:34.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f59177fe700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5918071c30 0x7f59181eb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:39:34.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f59177fe700 1 -- 172.21.15.7:0/4274288190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5908009550 con 0x7f59180aac00 2023-03-03T17:39:34.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f59177fe700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 0x7f59181547a0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f590000d800 tx=0x7f590000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:39:34.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f5914ff9700 1 -- 172.21.15.7:0/4274288190 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59000125f0 con 0x7f59180aac00 2023-03-03T17:39:34.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f5914ff9700 1 -- 172.21.15.7:0/4274288190 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5900012750 con 0x7f59180aac00 2023-03-03T17:39:34.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f591e6ba700 1 -- 172.21.15.7:0/4274288190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5918152620 con 0x7f59180aac00 2023-03-03T17:39:34.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f5914ff9700 1 -- 172.21.15.7:0/4274288190 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900012b00 con 0x7f59180aac00 2023-03-03T17:39:34.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.017+0000 7f591e6ba700 1 -- 172.21.15.7:0/4274288190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59180c4b80 con 0x7f59180aac00 2023-03-03T17:39:34.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.018+0000 7f591e6ba700 1 -- 172.21.15.7:0/4274288190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5918152900 con 0x7f59180aac00 2023-03-03T17:39:34.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.019+0000 7f5914ff9700 1 -- 172.21.15.7:0/4274288190 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5900012c60 con 0x7f59180aac00 2023-03-03T17:39:34.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.019+0000 7f5914ff9700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f80682f0 0x7f58f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:39:34.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.019+0000 7f5914ff9700 1 -- 172.21.15.7:0/4274288190 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5900086680 con 0x7f59180aac00 2023-03-03T17:39:34.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.020+0000 7f5917fff700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f80682f0 0x7f58f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:39:34.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.020+0000 7f5917fff700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f80682f0 0x7f58f806a7a0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f5908009130 tx=0x7f5908008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:39:34.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.024+0000 7f5914ff9700 1 -- 172.21.15.7:0/4274288190 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5900052a20 con 0x7f59180aac00 2023-03-03T17:39:34.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.259+0000 7f591e6ba700 1 -- 172.21.15.7:0/4274288190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5918074bb0 con 0x7f59180aac00 2023-03-03T17:39:34.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.259+0000 7f5914ff9700 1 -- 172.21.15.7:0/4274288190 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5900056040 con 0x7f59180aac00 2023-03-03T17:39:34.259 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:39:34.259 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:39:34.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.261+0000 7f591e6ba700 1 -- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f80682f0 msgr2=0x7f58f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:39:34.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.261+0000 7f591e6ba700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f80682f0 0x7f58f806a7a0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f5908009130 tx=0x7f5908008030 comp rx=0 tx=0).stop 2023-03-03T17:39:34.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.261+0000 7f591e6ba700 1 -- 172.21.15.7:0/4274288190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 msgr2=0x7f59181547a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:39:34.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.261+0000 7f591e6ba700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 0x7f59181547a0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7f590000d800 tx=0x7f590000dbc0 comp rx=0 tx=0).stop 2023-03-03T17:39:34.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.261+0000 7f591e6ba700 1 -- 172.21.15.7:0/4274288190 shutdown_connections 2023-03-03T17:39:34.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.261+0000 7f591e6ba700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59180bf0a0 0x7f59180c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:39:34.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.261+0000 7f591e6ba700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5918071c30 0x7f59181eb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:39:34.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.261+0000 7f591e6ba700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f58f80682f0 0x7f58f806a7a0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:39:34.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.261+0000 7f591e6ba700 1 --2- 172.21.15.7:0/4274288190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59180aac00 0x7f59181547a0 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:39:34.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.261+0000 7f591e6ba700 1 -- 172.21.15.7:0/4274288190 >> 172.21.15.7:0/4274288190 conn(0x7f59180736f0 msgr2=0x7f591806eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:39:34.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.262+0000 7f591e6ba700 1 -- 172.21.15.7:0/4274288190 shutdown_connections 2023-03-03T17:39:34.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:39:34.262+0000 7f591e6ba700 1 -- 172.21.15.7:0/4274288190 wait complete. 2023-03-03T17:39:34.263 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:39:34.273 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-03-03T17:39:59.213 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:39:59.217 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T17:39:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T17:40:10.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:40:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.932+0000 7ff56051c700 1 Processor -- start 2023-03-03T17:40:10.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.932+0000 7ff56051c700 1 -- start start 2023-03-03T17:40:10.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff56051c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff55814b0d0 0x7ff558071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff56051c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff558071d40 0x7ff558158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff56051c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5580a9c20 con 0x7ff5580aad20 2023-03-03T17:40:10.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff56051c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff558154f10 con 0x7ff55814b0d0 2023-03-03T17:40:10.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff56051c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5580ac3c0 con 0x7ff558071d40 2023-03-03T17:40:10.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff55e2b8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff55814b0d0 0x7ff558071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:10.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff55dab7700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff5580aad20 0x7ff558157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56268/0 (socket says 172.21.15.7:56268) 2023-03-03T17:40:10.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff55dab7700 1 -- 172.21.15.7:0/3923283120 learned_addr learned my addr 172.21.15.7:0/3923283120 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:40:10.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff55eab9700 1 --2- 172.21.15.7:0/3923283120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff558071d40 0x7ff558158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:10.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff55e2b8700 1 -- 172.21.15.7:0/3923283120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff558071d40 msgr2=0x7ff558158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:40:10.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff55e2b8700 1 --2- 172.21.15.7:0/3923283120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff558071d40 0x7ff558158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff55e2b8700 1 -- 172.21.15.7:0/3923283120 >> v1:172.21.15.7:6789/0 conn(0x7ff5580aad20 legacy=0x7ff558157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff55e2b8700 1 -- 172.21.15.7:0/3923283120 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55806fc70 con 0x7ff55814b0d0 2023-03-03T17:40:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.933+0000 7ff55e2b8700 1 --2- 172.21.15.7:0/3923283120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff55814b0d0 0x7ff558071800 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7ff554009840 tx=0x7ff554009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d88472cc5d0f9f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff55d2b6700 1 -- 172.21.15.7:0/3923283120 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff554011830 con 0x7ff55814b0d0 2023-03-03T17:40:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff55d2b6700 1 -- 172.21.15.7:0/3923283120 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff554011990 con 0x7ff55814b0d0 2023-03-03T17:40:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff55d2b6700 1 -- 172.21.15.7:0/3923283120 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff554011d20 con 0x7ff55814b0d0 2023-03-03T17:40:10.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff56051c700 1 -- 172.21.15.7:0/3923283120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff55814b0d0 msgr2=0x7ff558071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff56051c700 1 --2- 172.21.15.7:0/3923283120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff55814b0d0 0x7ff558071800 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7ff554009840 tx=0x7ff554009b50 comp rx=0 tx=0).stop 2023-03-03T17:40:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff56051c700 1 -- 172.21.15.7:0/3923283120 shutdown_connections 2023-03-03T17:40:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff56051c700 1 --2- 172.21.15.7:0/3923283120 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff558071d40 0x7ff558158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff56051c700 1 --2- 172.21.15.7:0/3923283120 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff55814b0d0 0x7ff558071800 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff56051c700 1 -- 172.21.15.7:0/3923283120 >> 172.21.15.7:0/3923283120 conn(0x7ff5580c8900 msgr2=0x7ff5580c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:40:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff56051c700 1 -- 172.21.15.7:0/3923283120 shutdown_connections 2023-03-03T17:40:10.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.934+0000 7ff56051c700 1 -- 172.21.15.7:0/3923283120 wait complete. 2023-03-03T17:40:10.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff56051c700 1 Processor -- start 2023-03-03T17:40:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff56051c700 1 -- start start 2023-03-03T17:40:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff56051c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff558071d40 0x7ff558157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff56051c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580aad20 0x7ff5581e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff56051c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff55814b0d0 0x7ff5580c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff56051c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581ee050 con 0x7ff558071d40 2023-03-03T17:40:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff56051c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581577c0 con 0x7ff55814b0d0 2023-03-03T17:40:10.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff56051c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5581520c0 con 0x7ff5580aad20 2023-03-03T17:40:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff55e2b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff558071d40 0x7ff558157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff55e2b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff558071d40 0x7ff558157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34058/0 (socket says 172.21.15.7:34058) 2023-03-03T17:40:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.936+0000 7ff55e2b8700 1 -- 172.21.15.7:0/2498985793 learned_addr learned my addr 172.21.15.7:0/2498985793 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:40:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff55eab9700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff55814b0d0 0x7ff5580c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff55dab7700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580aad20 0x7ff5581e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff55e2b8700 1 -- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580aad20 msgr2=0x7ff5581e9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:10.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff55e2b8700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580aad20 0x7ff5581e9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:10.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff55e2b8700 1 -- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff55814b0d0 msgr2=0x7ff5580c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:10.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff55e2b8700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff55814b0d0 0x7ff5580c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:10.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff55e2b8700 1 -- 172.21.15.7:0/2498985793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5540094a0 con 0x7ff558071d40 2023-03-03T17:40:10.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff55e2b8700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff558071d40 0x7ff558157280 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7ff554000c00 tx=0x7ff554004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:10.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff54effd700 1 -- 172.21.15.7:0/2498985793 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5540036a0 con 0x7ff558071d40 2023-03-03T17:40:10.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff54effd700 1 -- 172.21.15.7:0/2498985793 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff55401e070 con 0x7ff558071d40 2023-03-03T17:40:10.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff54effd700 1 -- 172.21.15.7:0/2498985793 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff554003a60 con 0x7ff558071d40 2023-03-03T17:40:10.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff56051c700 1 -- 172.21.15.7:0/2498985793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55806f6b0 con 0x7ff558071d40 2023-03-03T17:40:10.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.937+0000 7ff56051c700 1 -- 172.21.15.7:0/2498985793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff558155160 con 0x7ff558071d40 2023-03-03T17:40:10.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.938+0000 7ff56051c700 1 -- 172.21.15.7:0/2498985793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55806f990 con 0x7ff558071d40 2023-03-03T17:40:10.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.938+0000 7ff54effd700 1 -- 172.21.15.7:0/2498985793 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff55400f050 con 0x7ff558071d40 2023-03-03T17:40:10.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.939+0000 7ff54effd700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff53c0682f0 0x7ff53c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:10.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.939+0000 7ff54effd700 1 -- 172.21.15.7:0/2498985793 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff554087320 con 0x7ff558071d40 2023-03-03T17:40:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.939+0000 7ff55dab7700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff53c0682f0 0x7ff53c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:10.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.940+0000 7ff55dab7700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff53c0682f0 0x7ff53c06a7a0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7ff5581448f0 tx=0x7ff548009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:10.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:10.943+0000 7ff54effd700 1 -- 172.21.15.7:0/2498985793 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5540535d0 con 0x7ff558071d40 2023-03-03T17:40:11.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.179+0000 7ff56051c700 1 -- 172.21.15.7:0/2498985793 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff558074bb0 con 0x7ff558071d40 2023-03-03T17:40:11.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.179+0000 7ff54effd700 1 -- 172.21.15.7:0/2498985793 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff554020090 con 0x7ff558071d40 2023-03-03T17:40:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:40:11.179 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:40:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.181+0000 7ff56051c700 1 -- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff53c0682f0 msgr2=0x7ff53c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.181+0000 7ff56051c700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff53c0682f0 0x7ff53c06a7a0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7ff5581448f0 tx=0x7ff548009040 comp rx=0 tx=0).stop 2023-03-03T17:40:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.181+0000 7ff56051c700 1 -- 172.21.15.7:0/2498985793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff558071d40 msgr2=0x7ff558157280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:11.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.181+0000 7ff56051c700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff558071d40 0x7ff558157280 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7ff554000c00 tx=0x7ff554004750 comp rx=0 tx=0).stop 2023-03-03T17:40:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.181+0000 7ff56051c700 1 -- 172.21.15.7:0/2498985793 shutdown_connections 2023-03-03T17:40:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.181+0000 7ff56051c700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff55814b0d0 0x7ff5580c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.181+0000 7ff56051c700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff53c0682f0 0x7ff53c06a7a0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.181+0000 7ff56051c700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff558071d40 0x7ff558157280 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.181+0000 7ff56051c700 1 --2- 172.21.15.7:0/2498985793 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5580aad20 0x7ff5581e9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.181+0000 7ff56051c700 1 -- 172.21.15.7:0/2498985793 >> 172.21.15.7:0/2498985793 conn(0x7ff5580c8900 msgr2=0x7ff558149ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:40:11.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.182+0000 7ff56051c700 1 -- 172.21.15.7:0/2498985793 shutdown_connections 2023-03-03T17:40:11.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:11.182+0000 7ff56051c700 1 -- 172.21.15.7:0/2498985793 wait complete. 2023-03-03T17:40:11.183 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:40:11.193 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before thrashing 2023-03-03T17:40:20.194 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:40:20.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.335+0000 7fd5fc8d4700 1 Processor -- start 2023-03-03T17:40:20.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.335+0000 7fd5fc8d4700 1 -- start start 2023-03-03T17:40:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5fc8d4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd5f8151d40 0x7fd5f80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5fc8d4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f806d430 con 0x7fd5f80c2930 2023-03-03T17:40:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5fc8d4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f806cb00 con 0x7fd5f8073c30 2023-03-03T17:40:20.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5fc8d4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f806c2a0 con 0x7fd5f8151d40 2023-03-03T17:40:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f5d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd5f80c2930 0x7fd5f8157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56288/0 (socket says 172.21.15.7:56288) 2023-03-03T17:40:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f5d9b700 1 -- 172.21.15.7:0/2487995224 learned_addr learned my addr 172.21.15.7:0/2487995224 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:40:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f659c700 1 --2- 172.21.15.7:0/2487995224 >> v2:172.21.15.143:3300/0 conn(0x7fd5f8151d40 0x7fd5f80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850111023 0 0) 0x7fd5f806d430 con 0x7fd5f80c2930 2023-03-03T17:40:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e00035b0 con 0x7fd5f80c2930 2023-03-03T17:40:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210432001 0 0) 0x7fd5f806cb00 con 0x7fd5f8073c30 2023-03-03T17:40:20.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f806d430 con 0x7fd5f8073c30 2023-03-03T17:40:20.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3379717317 0 0) 0x7fd5e00035b0 con 0x7fd5f80c2930 2023-03-03T17:40:20.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 >> v2:172.21.15.143:3300/0 conn(0x7fd5f8151d40 msgr2=0x7fd5f80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:40:20.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 --2- 172.21.15.7:0/2487995224 >> v2:172.21.15.143:3300/0 conn(0x7fd5f8151d40 0x7fd5f80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:20.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 >> v1:172.21.15.143:6789/0 conn(0x7fd5f8073c30 legacy=0x7fd5f81581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:20.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f806cde0 con 0x7fd5f80c2930 2023-03-03T17:40:20.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd5ec001670 con 0x7fd5f80c2930 2023-03-03T17:40:20.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 >> v1:172.21.15.7:6789/0 conn(0x7fd5f80c2930 legacy=0x7fd5f8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:20.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e0003c00 0x7fd5e0003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e0004850 0x7fd5e0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e0005340 0x7fd5e000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.336+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f806c2a0 con 0x7fd5e0003c00 2023-03-03T17:40:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e000c230 con 0x7fd5e0004850 2023-03-03T17:40:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e000c3f0 con 0x7fd5e0005340 2023-03-03T17:40:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f659c700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e0003c00 0x7fd5e0003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:20.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f5d9b700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e0004850 0x7fd5e0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f6d9d700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e0005340 0x7fd5e000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f659c700 1 -- 172.21.15.7:0/2487995224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e0005340 msgr2=0x7fd5e000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:40:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f659c700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e0005340 0x7fd5e000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f659c700 1 -- 172.21.15.7:0/2487995224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e0004850 msgr2=0x7fd5e0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:40:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f659c700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e0004850 0x7fd5e0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f659c700 1 -- 172.21.15.7:0/2487995224 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e00035b0 con 0x7fd5e0003c00 2023-03-03T17:40:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f6d9d700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5e0005340 0x7fd5e000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:40:20.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f6d9d700 1 -- 172.21.15.7:0/2487995224 reap_dead start 2023-03-03T17:40:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f659c700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e0003c00 0x7fd5e0003ff0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009b60 tx=0x7fd5e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b6746f4c54195a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e800e6c0 con 0x7fd5e0003c00 2023-03-03T17:40:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e800e820 con 0x7fd5e0003c00 2023-03-03T17:40:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.337+0000 7fd5f559a700 1 -- 172.21.15.7:0/2487995224 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e800eb90 con 0x7fd5e0003c00 2023-03-03T17:40:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.338+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2487995224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e0003c00 msgr2=0x7fd5e0003ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.338+0000 7fd5fc8d4700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e0003c00 0x7fd5e0003ff0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009b60 tx=0x7fd5e0005880 comp rx=0 tx=0).stop 2023-03-03T17:40:20.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.338+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2487995224 shutdown_connections 2023-03-03T17:40:20.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.338+0000 7fd5fc8d4700 1 --2- 172.21.15.7:0/2487995224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5e0003c00 0x7fd5e0003ff0 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:20.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.338+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2487995224 >> 172.21.15.7:0/2487995224 conn(0x7fd5f80a94d0 msgr2=0x7fd5f80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:40:20.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.338+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2487995224 shutdown_connections 2023-03-03T17:40:20.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.338+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2487995224 wait complete. 2023-03-03T17:40:20.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.339+0000 7fd5fc8d4700 1 Processor -- start 2023-03-03T17:40:20.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.339+0000 7fd5fc8d4700 1 -- start start 2023-03-03T17:40:20.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5fc8d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e0003c00 0x7fd5f814ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:20.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5fc8d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f80ac2e0 0x7fd5f81ed5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:20.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5fc8d4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f81edb20 0x7fd5f81523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:20.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5fc8d4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f81ed150 con 0x7fd5f81edb20 2023-03-03T17:40:20.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5fc8d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f80ab470 con 0x7fd5e0003c00 2023-03-03T17:40:20.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5fc8d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f80aa1d0 con 0x7fd5f80ac2e0 2023-03-03T17:40:20.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f81edb20 0x7fd5f81523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:20.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f81edb20 0x7fd5f81523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34090/0 (socket says 172.21.15.7:34090) 2023-03-03T17:40:20.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f6d9d700 1 -- 172.21.15.7:0/2731060950 learned_addr learned my addr 172.21.15.7:0/2731060950 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:40:20.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f5d9b700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f80ac2e0 0x7fd5f81ed5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:20.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f659c700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e0003c00 0x7fd5f814ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:20.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f6d9d700 1 -- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f80ac2e0 msgr2=0x7fd5f81ed5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:20.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f6d9d700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f80ac2e0 0x7fd5f81ed5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:20.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f6d9d700 1 -- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e0003c00 msgr2=0x7fd5f814ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:20.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f6d9d700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e0003c00 0x7fd5f814ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:20.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f6d9d700 1 -- 172.21.15.7:0/2731060950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f806bea0 con 0x7fd5f81edb20 2023-03-03T17:40:20.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.340+0000 7fd5f659c700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e0003c00 0x7fd5f814ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:40:20.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.341+0000 7fd5f6d9d700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f81edb20 0x7fd5f81523e0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7fd5f806bcf0 tx=0x7fd5f000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:20.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.341+0000 7fd5e6ffd700 1 -- 172.21.15.7:0/2731060950 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f0003c10 con 0x7fd5f81edb20 2023-03-03T17:40:20.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.341+0000 7fd5e6ffd700 1 -- 172.21.15.7:0/2731060950 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f0003d70 con 0x7fd5f81edb20 2023-03-03T17:40:20.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.341+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2731060950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f806d0c0 con 0x7fd5f81edb20 2023-03-03T17:40:20.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.341+0000 7fd5e6ffd700 1 -- 172.21.15.7:0/2731060950 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f000c910 con 0x7fd5f81edb20 2023-03-03T17:40:20.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.341+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2731060950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f806c030 con 0x7fd5f81edb20 2023-03-03T17:40:20.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.342+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2731060950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f8074bb0 con 0x7fd5f81edb20 2023-03-03T17:40:20.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.342+0000 7fd5e6ffd700 1 -- 172.21.15.7:0/2731060950 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd5f000ca70 con 0x7fd5f81edb20 2023-03-03T17:40:20.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.342+0000 7fd5e6ffd700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 0x7fd5d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:20.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.343+0000 7fd5e6ffd700 1 -- 172.21.15.7:0/2731060950 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd5f0086480 con 0x7fd5f81edb20 2023-03-03T17:40:20.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.343+0000 7fd5f659c700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 0x7fd5d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:20.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.343+0000 7fd5f659c700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 0x7fd5d806a7a0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8000c00 tx=0x7fd5e8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:20.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.347+0000 7fd5e6ffd700 1 -- 172.21.15.7:0/2731060950 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd5f0052730 con 0x7fd5f81edb20 2023-03-03T17:40:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.584+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2731060950 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5f81ed300 con 0x7fd5f81edb20 2023-03-03T17:40:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.585+0000 7fd5e6ffd700 1 -- 172.21.15.7:0/2731060950 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd5f81ed300 con 0x7fd5f81edb20 2023-03-03T17:40:20.585 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:40:20.585 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:40:20.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 msgr2=0x7fd5d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:20.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 0x7fd5d806a7a0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8000c00 tx=0x7fd5e8008030 comp rx=0 tx=0).stop 2023-03-03T17:40:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2731060950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f81edb20 msgr2=0x7fd5f81523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f81edb20 0x7fd5f81523e0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7fd5f806bcf0 tx=0x7fd5f000ef40 comp rx=0 tx=0).stop 2023-03-03T17:40:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2731060950 shutdown_connections 2023-03-03T17:40:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5d80682f0 0x7fd5d806a7a0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd5f81edb20 0x7fd5f81523e0 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5e0003c00 0x7fd5f814ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 --2- 172.21.15.7:0/2731060950 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd5f80ac2e0 0x7fd5f81ed5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2731060950 >> 172.21.15.7:0/2731060950 conn(0x7fd5f80a94d0 msgr2=0x7fd5f8158280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:40:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2731060950 shutdown_connections 2023-03-03T17:40:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:20.587+0000 7fd5fc8d4700 1 -- 172.21.15.7:0/2731060950 wait complete. 2023-03-03T17:40:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:40:20.598 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2023-03-03T17:40:29.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:29.338 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:34.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:40:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.415+0000 7f6c31e66700 1 Processor -- start 2023-03-03T17:40:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.415+0000 7f6c31e66700 1 -- start start 2023-03-03T17:40:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.416+0000 7f6c31e66700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6c2c0c5140 0x7f6c2c157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.416+0000 7f6c31e66700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c2c06dad0 con 0x7f6c2c151de0 2023-03-03T17:40:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.416+0000 7f6c31e66700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c2c06ddb0 con 0x7f6c2c1580e0 2023-03-03T17:40:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.416+0000 7f6c31e66700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c2c06e370 con 0x7f6c2c0c5140 2023-03-03T17:40:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.416+0000 7f6c2b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6c2c151de0 0x7f6c2c0c1500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56312/0 (socket says 172.21.15.7:56312) 2023-03-03T17:40:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.416+0000 7f6c2b7fe700 1 -- 172.21.15.7:0/2772893471 learned_addr learned my addr 172.21.15.7:0/2772893471 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:40:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.416+0000 7f6c2affd700 1 --2- 172.21.15.7:0/2772893471 >> v2:172.21.15.143:3300/0 conn(0x7f6c2c0c5140 0x7f6c2c157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.416+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744112469 0 0) 0x7f6c2c06dad0 con 0x7f6c2c151de0 2023-03-03T17:40:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.416+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c180035b0 con 0x7f6c2c151de0 2023-03-03T17:40:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928580117 0 0) 0x7f6c180035b0 con 0x7f6c2c151de0 2023-03-03T17:40:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 >> v2:172.21.15.143:3300/0 conn(0x7f6c2c0c5140 msgr2=0x7f6c2c157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:40:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 --2- 172.21.15.7:0/2772893471 >> v2:172.21.15.143:3300/0 conn(0x7f6c2c0c5140 0x7f6c2c157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 >> v1:172.21.15.143:6789/0 conn(0x7f6c2c1580e0 legacy=0x7f6c2c06a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c06e090 con 0x7f6c2c151de0 2023-03-03T17:40:34.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f6c14001670 con 0x7f6c2c151de0 2023-03-03T17:40:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 >> v1:172.21.15.7:6789/0 conn(0x7f6c2c151de0 legacy=0x7f6c2c0c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c18003a70 0x7f6c18003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c18004850 0x7f6c18004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c1800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c06e370 con 0x7f6c18005200 2023-03-03T17:40:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1800c0e0 con 0x7f6c18004850 2023-03-03T17:40:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1800c2a0 con 0x7f6c18003a70 2023-03-03T17:40:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2bfff700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c1800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2affd700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c18004850 0x7f6c18004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2b7fe700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c18003a70 0x7f6c18003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2bfff700 1 -- 172.21.15.7:0/2772893471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c18003a70 msgr2=0x7f6c18003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:40:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2bfff700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c18003a70 0x7f6c18003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2bfff700 1 -- 172.21.15.7:0/2772893471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c18004850 msgr2=0x7f6c18004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:40:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2bfff700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c18004850 0x7f6c18004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2bfff700 1 -- 172.21.15.7:0/2772893471 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c180035b0 con 0x7f6c18005200 2023-03-03T17:40:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2bfff700 1 -- 172.21.15.7:0/2772893471 reap_dead start 2023-03-03T17:40:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.417+0000 7f6c2bfff700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c1800ba10 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400c190 tx=0x7f6c18005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e075b0a51f23d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.418+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24001db0 con 0x7f6c18005200 2023-03-03T17:40:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.418+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c2400ccd0 con 0x7f6c18005200 2023-03-03T17:40:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.418+0000 7f6c2a7fc700 1 -- 172.21.15.7:0/2772893471 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24003c40 con 0x7f6c18005200 2023-03-03T17:40:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.418+0000 7f6c31e66700 1 -- 172.21.15.7:0/2772893471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 msgr2=0x7f6c1800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.418+0000 7f6c31e66700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c1800ba10 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400c190 tx=0x7f6c18005880 comp rx=0 tx=0).stop 2023-03-03T17:40:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.418+0000 7f6c31e66700 1 -- 172.21.15.7:0/2772893471 shutdown_connections 2023-03-03T17:40:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.418+0000 7f6c31e66700 1 --2- 172.21.15.7:0/2772893471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c1800ba10 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.418+0000 7f6c31e66700 1 -- 172.21.15.7:0/2772893471 >> 172.21.15.7:0/2772893471 conn(0x7f6c2c0a9430 msgr2=0x7f6c2c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:40:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.418+0000 7f6c31e66700 1 -- 172.21.15.7:0/2772893471 shutdown_connections 2023-03-03T17:40:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.418+0000 7f6c31e66700 1 -- 172.21.15.7:0/2772893471 wait complete. 2023-03-03T17:40:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.419+0000 7f6c31e66700 1 Processor -- start 2023-03-03T17:40:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.420+0000 7f6c31e66700 1 -- start start 2023-03-03T17:40:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.420+0000 7f6c31e66700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c2c146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.420+0000 7f6c31e66700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c2c149110 0x7f6c2c1ed470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.420+0000 7f6c31e66700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c2c1ed9b0 0x7f6c2c147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.420+0000 7f6c31e66700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c14d6d0 con 0x7f6c18005200 2023-03-03T17:40:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.420+0000 7f6c31e66700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c14d970 con 0x7f6c2c149110 2023-03-03T17:40:34.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.420+0000 7f6c31e66700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c14dc50 con 0x7f6c2c1ed9b0 2023-03-03T17:40:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.420+0000 7f6c2b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c2c146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.420+0000 7f6c2b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c2c146d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34112/0 (socket says 172.21.15.7:34112) 2023-03-03T17:40:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.420+0000 7f6c2b7fe700 1 -- 172.21.15.7:0/1452564717 learned_addr learned my addr 172.21.15.7:0/1452564717 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:40:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c2affd700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c2c149110 0x7f6c2c1ed470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c2bfff700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c2c1ed9b0 0x7f6c2c147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c2b7fe700 1 -- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c2c1ed9b0 msgr2=0x7f6c2c147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c2b7fe700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c2c1ed9b0 0x7f6c2c147c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c2b7fe700 1 -- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c2c149110 msgr2=0x7f6c2c1ed470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c2b7fe700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c2c149110 0x7f6c2c1ed470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c2b7fe700 1 -- 172.21.15.7:0/1452564717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c066de0 con 0x7f6c18005200 2023-03-03T17:40:34.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c2bfff700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c2c1ed9b0 0x7f6c2c147c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:40:34.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c2b7fe700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c2c146d00 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f6c140072c0 tx=0x7f6c14007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:34.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c23fff700 1 -- 172.21.15.7:0/1452564717 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c140077a0 con 0x7f6c18005200 2023-03-03T17:40:34.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c23fff700 1 -- 172.21.15.7:0/1452564717 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1400f850 con 0x7f6c18005200 2023-03-03T17:40:34.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c31e66700 1 -- 172.21.15.7:0/1452564717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2c0c4c20 con 0x7f6c18005200 2023-03-03T17:40:34.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c23fff700 1 -- 172.21.15.7:0/1452564717 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1400fbe0 con 0x7f6c18005200 2023-03-03T17:40:34.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.421+0000 7f6c31e66700 1 -- 172.21.15.7:0/1452564717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2c157ba0 con 0x7f6c18005200 2023-03-03T17:40:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.422+0000 7f6c31e66700 1 -- 172.21.15.7:0/1452564717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2c1ed2c0 con 0x7f6c18005200 2023-03-03T17:40:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.423+0000 7f6c23fff700 1 -- 172.21.15.7:0/1452564717 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6c1400fd40 con 0x7f6c18005200 2023-03-03T17:40:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.423+0000 7f6c23fff700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c0c0682f0 0x7f6c0c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.423+0000 7f6c23fff700 1 -- 172.21.15.7:0/1452564717 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6c140861b0 con 0x7f6c18005200 2023-03-03T17:40:34.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.423+0000 7f6c2affd700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c0c0682f0 0x7f6c0c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:34.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.424+0000 7f6c2affd700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c0c0682f0 0x7f6c0c06a7a0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c157b70 tx=0x7f6c1c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:34.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.428+0000 7f6c23fff700 1 -- 172.21.15.7:0/1452564717 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6c140524f0 con 0x7f6c18005200 2023-03-03T17:40:34.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.666+0000 7f6c31e66700 1 -- 172.21.15.7:0/1452564717 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c2c074bb0 con 0x7f6c18005200 2023-03-03T17:40:34.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.667+0000 7f6c23fff700 1 -- 172.21.15.7:0/1452564717 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6c1405b070 con 0x7f6c18005200 2023-03-03T17:40:34.671 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:40:34.672 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:40:34.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 -- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c0c0682f0 msgr2=0x7f6c0c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:34.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c0c0682f0 0x7f6c0c06a7a0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c157b70 tx=0x7f6c1c009210 comp rx=0 tx=0).stop 2023-03-03T17:40:34.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 -- 172.21.15.7:0/1452564717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 msgr2=0x7f6c2c146d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:34.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c2c146d00 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7f6c140072c0 tx=0x7f6c14007680 comp rx=0 tx=0).stop 2023-03-03T17:40:34.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 -- 172.21.15.7:0/1452564717 shutdown_connections 2023-03-03T17:40:34.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6c2c1ed9b0 0x7f6c2c147c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:34.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6c0c0682f0 0x7f6c0c06a7a0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:34.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c18005200 0x7f6c2c146d00 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:34.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 --2- 172.21.15.7:0/1452564717 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c2c149110 0x7f6c2c1ed470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:34.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 -- 172.21.15.7:0/1452564717 >> 172.21.15.7:0/1452564717 conn(0x7f6c2c0a9430 msgr2=0x7f6c2c14a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:40:34.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 -- 172.21.15.7:0/1452564717 shutdown_connections 2023-03-03T17:40:34.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:34.669+0000 7f6c31e66700 1 -- 172.21.15.7:0/1452564717 wait complete. 2023-03-03T17:40:34.674 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:40:34.680 INFO:tasks.mds_thrash.fs.[b]:waiting for 119 secs before thrashing 2023-03-03T17:40:47.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:40:47.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.738+0000 7ff329258700 1 Processor -- start 2023-03-03T17:40:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.738+0000 7ff329258700 1 -- start start 2023-03-03T17:40:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.738+0000 7ff329258700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff324071e80 0x7ff324151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.738+0000 7ff329258700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff324072250 0x7ff32414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.738+0000 7ff329258700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3240c1ce0 0x7ff32414e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:47.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.738+0000 7ff329258700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3241510d0 con 0x7ff324071e80 2023-03-03T17:40:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.738+0000 7ff329258700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3240c3740 con 0x7ff324072250 2023-03-03T17:40:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff329258700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff324070670 con 0x7ff3240c1ce0 2023-03-03T17:40:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff322d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff324071e80 0x7ff324151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff322d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff324071e80 0x7ff324151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34120/0 (socket says 172.21.15.7:34120) 2023-03-03T17:40:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff322d9d700 1 -- 172.21.15.7:0/3824232108 learned_addr learned my addr 172.21.15.7:0/3824232108 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:40:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff32359e700 1 --2- 172.21.15.7:0/3824232108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3240c1ce0 0x7ff32414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:47.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff32259c700 1 --2- 172.21.15.7:0/3824232108 >> v2:172.21.15.143:3300/0 conn(0x7ff324072250 0x7ff32414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff322d9d700 1 -- 172.21.15.7:0/3824232108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3240c1ce0 msgr2=0x7ff32414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:40:47.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff322d9d700 1 --2- 172.21.15.7:0/3824232108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3240c1ce0 0x7ff32414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff322d9d700 1 -- 172.21.15.7:0/3824232108 >> v2:172.21.15.143:3300/0 conn(0x7ff324072250 msgr2=0x7ff32414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:40:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff322d9d700 1 --2- 172.21.15.7:0/3824232108 >> v2:172.21.15.143:3300/0 conn(0x7ff324072250 0x7ff32414e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff322d9d700 1 -- 172.21.15.7:0/3824232108 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240c39e0 con 0x7ff324071e80 2023-03-03T17:40:47.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff322d9d700 1 --2- 172.21.15.7:0/3824232108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff324071e80 0x7ff324151820 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7ff32406b520 tx=0x7ff314009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768a59378fa6f261 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:47.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff321d9b700 1 -- 172.21.15.7:0/3824232108 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31400b870 con 0x7ff324071e80 2023-03-03T17:40:47.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff321d9b700 1 -- 172.21.15.7:0/3824232108 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31400b9d0 con 0x7ff324071e80 2023-03-03T17:40:47.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.739+0000 7ff321d9b700 1 -- 172.21.15.7:0/3824232108 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31400bd60 con 0x7ff324071e80 2023-03-03T17:40:47.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.740+0000 7ff329258700 1 -- 172.21.15.7:0/3824232108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff324071e80 msgr2=0x7ff324151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:47.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.740+0000 7ff329258700 1 --2- 172.21.15.7:0/3824232108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff324071e80 0x7ff324151820 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7ff32406b520 tx=0x7ff314009bd0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.740+0000 7ff329258700 1 -- 172.21.15.7:0/3824232108 shutdown_connections 2023-03-03T17:40:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.740+0000 7ff329258700 1 --2- 172.21.15.7:0/3824232108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff3240c1ce0 0x7ff32414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.740+0000 7ff329258700 1 --2- 172.21.15.7:0/3824232108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff324071e80 0x7ff324151820 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.740+0000 7ff329258700 1 --2- 172.21.15.7:0/3824232108 >> v2:172.21.15.143:3300/0 conn(0x7ff324072250 0x7ff32414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.740+0000 7ff329258700 1 -- 172.21.15.7:0/3824232108 >> 172.21.15.7:0/3824232108 conn(0x7ff3240a94d0 msgr2=0x7ff3240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:40:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.740+0000 7ff329258700 1 -- 172.21.15.7:0/3824232108 shutdown_connections 2023-03-03T17:40:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.740+0000 7ff329258700 1 -- 172.21.15.7:0/3824232108 wait complete. 2023-03-03T17:40:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.741+0000 7ff329258700 1 Processor -- start 2023-03-03T17:40:47.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.742+0000 7ff329258700 1 -- start start 2023-03-03T17:40:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.742+0000 7ff329258700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff324071e80 0x7ff324148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.742+0000 7ff329258700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff324072250 0x7ff3241475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.742+0000 7ff329258700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3240c1ce0 0x7ff3241e91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.742+0000 7ff329258700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff324147b30 con 0x7ff3240c1ce0 2023-03-03T17:40:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.742+0000 7ff329258700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff324147350 con 0x7ff324072250 2023-03-03T17:40:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.742+0000 7ff329258700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3241e9aa0 con 0x7ff324071e80 2023-03-03T17:40:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.742+0000 7ff32359e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3240c1ce0 0x7ff3241e91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.742+0000 7ff32359e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3240c1ce0 0x7ff3241e91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34128/0 (socket says 172.21.15.7:34128) 2023-03-03T17:40:47.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.742+0000 7ff32359e700 1 -- 172.21.15.7:0/3345517295 learned_addr learned my addr 172.21.15.7:0/3345517295 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:40:47.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff322d9d700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff324071e80 0x7ff324148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:47.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff32259c700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff324072250 0x7ff3241475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff32359e700 1 -- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff324071e80 msgr2=0x7ff324148ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff32359e700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff324071e80 0x7ff324148ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff32359e700 1 -- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff324072250 msgr2=0x7ff3241475f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff32359e700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff324072250 0x7ff3241475f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff32359e700 1 -- 172.21.15.7:0/3345517295 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff314009550 con 0x7ff3240c1ce0 2023-03-03T17:40:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff32359e700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3240c1ce0 0x7ff3241e91d0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7ff31c007d40 tx=0x7ff31c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff30b7fe700 1 -- 172.21.15.7:0/3345517295 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c00ede0 con 0x7ff3240c1ce0 2023-03-03T17:40:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff30b7fe700 1 -- 172.21.15.7:0/3345517295 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c014420 con 0x7ff3240c1ce0 2023-03-03T17:40:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff329258700 1 -- 172.21.15.7:0/3345517295 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32406c9a0 con 0x7ff3240c1ce0 2023-03-03T17:40:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff30b7fe700 1 -- 172.21.15.7:0/3345517295 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0147d0 con 0x7ff3240c1ce0 2023-03-03T17:40:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.743+0000 7ff329258700 1 -- 172.21.15.7:0/3345517295 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff324151300 con 0x7ff3240c1ce0 2023-03-03T17:40:47.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.744+0000 7ff329258700 1 -- 172.21.15.7:0/3345517295 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32406cb30 con 0x7ff3240c1ce0 2023-03-03T17:40:47.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.745+0000 7ff30b7fe700 1 -- 172.21.15.7:0/3345517295 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff31c012050 con 0x7ff3240c1ce0 2023-03-03T17:40:47.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.745+0000 7ff30b7fe700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3100682f0 0x7ff31006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:40:47.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.745+0000 7ff30b7fe700 1 -- 172.21.15.7:0/3345517295 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff31c086790 con 0x7ff3240c1ce0 2023-03-03T17:40:47.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.745+0000 7ff322d9d700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3100682f0 0x7ff31006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:40:47.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.746+0000 7ff322d9d700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3100682f0 0x7ff31006a7a0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7ff314000c00 tx=0x7ff314008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:40:47.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.749+0000 7ff30b7fe700 1 -- 172.21.15.7:0/3345517295 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff31c052ad0 con 0x7ff3240c1ce0 2023-03-03T17:40:47.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.987+0000 7ff329258700 1 -- 172.21.15.7:0/3345517295 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff324071410 con 0x7ff3240c1ce0 2023-03-03T17:40:47.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.988+0000 7ff30b7fe700 1 -- 172.21.15.7:0/3345517295 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff31c0560f0 con 0x7ff3240c1ce0 2023-03-03T17:40:47.988 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:40:47.988 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:40:47.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 -- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3100682f0 msgr2=0x7ff31006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3100682f0 0x7ff31006a7a0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7ff314000c00 tx=0x7ff314008030 comp rx=0 tx=0).stop 2023-03-03T17:40:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 -- 172.21.15.7:0/3345517295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3240c1ce0 msgr2=0x7ff3241e91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:40:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3240c1ce0 0x7ff3241e91d0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7ff31c007d40 tx=0x7ff31c00e3c0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 -- 172.21.15.7:0/3345517295 shutdown_connections 2023-03-03T17:40:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff3100682f0 0x7ff31006a7a0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3240c1ce0 0x7ff3241e91d0 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff324071e80 0x7ff324148ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 --2- 172.21.15.7:0/3345517295 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff324072250 0x7ff3241475f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:40:47.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 -- 172.21.15.7:0/3345517295 >> 172.21.15.7:0/3345517295 conn(0x7ff3240a94d0 msgr2=0x7ff3240c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:40:47.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 -- 172.21.15.7:0/3345517295 shutdown_connections 2023-03-03T17:40:47.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:40:47.990+0000 7ff329258700 1 -- 172.21.15.7:0/3345517295 wait complete. 2023-03-03T17:40:47.992 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:40:48.002 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T17:40:59.457 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:40:59.460 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:29.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:29.582 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:52.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:41:52.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.142+0000 7fa20d868700 1 Processor -- start 2023-03-03T17:41:52.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.142+0000 7fa20d868700 1 -- start start 2023-03-03T17:41:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.143+0000 7fa20d868700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2080c7a30 0x7fa20814e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:41:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.143+0000 7fa20d868700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa2080c7e00 0x7fa20814e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:41:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.143+0000 7fa20d868700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa20806b6f0 0x7fa208154a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:41:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.143+0000 7fa20d868700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa20806ab00 con 0x7fa20806b6f0 2023-03-03T17:41:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.143+0000 7fa20d868700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2081545c0 con 0x7fa2080c7e00 2023-03-03T17:41:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.143+0000 7fa20d868700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa208157bb0 con 0x7fa2080c7a30 2023-03-03T17:41:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.143+0000 7fa2077fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa20806b6f0 0x7fa208154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:41:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.143+0000 7fa2077fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa20806b6f0 0x7fa208154a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34180/0 (socket says 172.21.15.7:34180) 2023-03-03T17:41:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa2077fe700 1 -- 172.21.15.7:0/4138623036 learned_addr learned my addr 172.21.15.7:0/4138623036 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:41:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa2067fc700 1 --2- 172.21.15.7:0/4138623036 >> v2:172.21.15.143:3300/0 conn(0x7fa2080c7e00 0x7fa20814e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:41:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa206ffd700 1 --2- 172.21.15.7:0/4138623036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2080c7a30 0x7fa20814e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:41:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa2077fe700 1 -- 172.21.15.7:0/4138623036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2080c7a30 msgr2=0x7fa20814e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:41:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa2077fe700 1 --2- 172.21.15.7:0/4138623036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2080c7a30 0x7fa20814e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa2077fe700 1 -- 172.21.15.7:0/4138623036 >> v2:172.21.15.143:3300/0 conn(0x7fa2080c7e00 msgr2=0x7fa20814e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:41:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa2077fe700 1 --2- 172.21.15.7:0/4138623036 >> v2:172.21.15.143:3300/0 conn(0x7fa2080c7e00 0x7fa20814e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa2077fe700 1 -- 172.21.15.7:0/4138623036 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20806a2f0 con 0x7fa20806b6f0 2023-03-03T17:41:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa2077fe700 1 --2- 172.21.15.7:0/4138623036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa20806b6f0 0x7fa208154a10 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fa2080cb890 tx=0x7fa20000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b61167ff89554eb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:41:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa205ffb700 1 -- 172.21.15.7:0/4138623036 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20000e070 con 0x7fa20806b6f0 2023-03-03T17:41:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa205ffb700 1 -- 172.21.15.7:0/4138623036 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20000b9c0 con 0x7fa20806b6f0 2023-03-03T17:41:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.144+0000 7fa205ffb700 1 -- 172.21.15.7:0/4138623036 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20000bd70 con 0x7fa20806b6f0 2023-03-03T17:41:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.145+0000 7fa20d868700 1 -- 172.21.15.7:0/4138623036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa20806b6f0 msgr2=0x7fa208154a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:41:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.145+0000 7fa20d868700 1 --2- 172.21.15.7:0/4138623036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa20806b6f0 0x7fa208154a10 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fa2080cb890 tx=0x7fa20000b330 comp rx=0 tx=0).stop 2023-03-03T17:41:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.145+0000 7fa20d868700 1 -- 172.21.15.7:0/4138623036 shutdown_connections 2023-03-03T17:41:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.145+0000 7fa20d868700 1 --2- 172.21.15.7:0/4138623036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa20806b6f0 0x7fa208154a10 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.145+0000 7fa20d868700 1 --2- 172.21.15.7:0/4138623036 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2080c7a30 0x7fa20814e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.145+0000 7fa20d868700 1 --2- 172.21.15.7:0/4138623036 >> v2:172.21.15.143:3300/0 conn(0x7fa2080c7e00 0x7fa20814e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.145+0000 7fa20d868700 1 -- 172.21.15.7:0/4138623036 >> 172.21.15.7:0/4138623036 conn(0x7fa208073d20 msgr2=0x7fa2080716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:41:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.145+0000 7fa20d868700 1 -- 172.21.15.7:0/4138623036 shutdown_connections 2023-03-03T17:41:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.145+0000 7fa20d868700 1 -- 172.21.15.7:0/4138623036 wait complete. 2023-03-03T17:41:52.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.146+0000 7fa20d868700 1 Processor -- start 2023-03-03T17:41:52.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.147+0000 7fa20d868700 1 -- start start 2023-03-03T17:41:52.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.147+0000 7fa20d868700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2080c7a30 0x7fa2081ed970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:41:52.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.147+0000 7fa20d868700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2080c7e00 0x7fa2081ea400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:41:52.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.147+0000 7fa20d868700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2080c2040 0x7fa208156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:41:52.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.147+0000 7fa20d868700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2081e99f0 con 0x7fa2080c7a30 2023-03-03T17:41:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.147+0000 7fa20d868700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2081edeb0 con 0x7fa2080c2040 2023-03-03T17:41:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.147+0000 7fa20d868700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2081ea940 con 0x7fa2080c7e00 2023-03-03T17:41:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.147+0000 7fa206ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2080c7a30 0x7fa2081ed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:41:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.147+0000 7fa206ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2080c7a30 0x7fa2081ed970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34182/0 (socket says 172.21.15.7:34182) 2023-03-03T17:41:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.147+0000 7fa206ffd700 1 -- 172.21.15.7:0/3411440993 learned_addr learned my addr 172.21.15.7:0/3411440993 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:41:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa2067fc700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2080c7e00 0x7fa2081ea400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:41:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa2077fe700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2080c2040 0x7fa208156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:41:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa206ffd700 1 -- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2080c7e00 msgr2=0x7fa2081ea400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:41:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa206ffd700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2080c7e00 0x7fa2081ea400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa206ffd700 1 -- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2080c2040 msgr2=0x7fa208156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:41:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa206ffd700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2080c2040 0x7fa208156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa206ffd700 1 -- 172.21.15.7:0/3411440993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa200009b30 con 0x7fa2080c7a30 2023-03-03T17:41:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa206ffd700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2080c7a30 0x7fa2081ed970 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800e7e0 tx=0x7fa1f800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:41:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa1effff700 1 -- 172.21.15.7:0/3411440993 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f800c070 con 0x7fa2080c7a30 2023-03-03T17:41:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa1effff700 1 -- 172.21.15.7:0/3411440993 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f8013610 con 0x7fa2080c7a30 2023-03-03T17:41:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa20d868700 1 -- 172.21.15.7:0/3411440993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20806a540 con 0x7fa2080c7a30 2023-03-03T17:41:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa1effff700 1 -- 172.21.15.7:0/3411440993 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80139a0 con 0x7fa2080c7a30 2023-03-03T17:41:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.148+0000 7fa20d868700 1 -- 172.21.15.7:0/3411440993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2080bfbb0 con 0x7fa2080c7a30 2023-03-03T17:41:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.149+0000 7fa20d868700 1 -- 172.21.15.7:0/3411440993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa20806a820 con 0x7fa2080c7a30 2023-03-03T17:41:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.150+0000 7fa1effff700 1 -- 172.21.15.7:0/3411440993 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa1f8023070 con 0x7fa2080c7a30 2023-03-03T17:41:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.150+0000 7fa1effff700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 0x7fa1f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:41:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.150+0000 7fa1effff700 1 -- 172.21.15.7:0/3411440993 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa1f8085f30 con 0x7fa2080c7a30 2023-03-03T17:41:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.150+0000 7fa2067fc700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 0x7fa1f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:41:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.151+0000 7fa2067fc700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 0x7fa1f406a7a0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fa208157660 tx=0x7fa1fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:41:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.154+0000 7fa1effff700 1 -- 172.21.15.7:0/3411440993 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa1f8052250 con 0x7fa2080c7a30 2023-03-03T17:41:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.394+0000 7fa20d868700 1 -- 172.21.15.7:0/3411440993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa20808c270 con 0x7fa2080c7a30 2023-03-03T17:41:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.394+0000 7fa1effff700 1 -- 172.21.15.7:0/3411440993 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa1f8055870 con 0x7fa2080c7a30 2023-03-03T17:41:52.394 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:41:52.395 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:41:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 -- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 msgr2=0x7fa1f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:41:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 0x7fa1f406a7a0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fa208157660 tx=0x7fa1fc008040 comp rx=0 tx=0).stop 2023-03-03T17:41:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 -- 172.21.15.7:0/3411440993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2080c7a30 msgr2=0x7fa2081ed970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:41:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2080c7a30 0x7fa2081ed970 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800e7e0 tx=0x7fa1f800eaf0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 -- 172.21.15.7:0/3411440993 shutdown_connections 2023-03-03T17:41:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2080c2040 0x7fa208156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 0x7fa1f406a7a0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2080c7a30 0x7fa2081ed970 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 --2- 172.21.15.7:0/3411440993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2080c7e00 0x7fa2081ea400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:41:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 -- 172.21.15.7:0/3411440993 >> 172.21.15.7:0/3411440993 conn(0x7fa20806b6f0 msgr2=0x7fa2080718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:41:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 -- 172.21.15.7:0/3411440993 shutdown_connections 2023-03-03T17:41:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:41:52.396+0000 7fa20d868700 1 -- 172.21.15.7:0/3411440993 wait complete. 2023-03-03T17:41:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:41:52.408 INFO:tasks.mds_thrash.fs.[a]:waiting for 107 secs before thrashing 2023-03-03T17:41:59.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:41:59.646 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:29.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:29.792 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:33.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:42:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.822+0000 7fdb2d906700 1 Processor -- start 2023-03-03T17:42:33.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.822+0000 7fdb2d906700 1 -- start start 2023-03-03T17:42:33.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.822+0000 7fdb2d906700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb280caa30 0x7fdb280c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:42:33.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.822+0000 7fdb2d906700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdb28154ff0 0x7fdb280cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:42:33.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.822+0000 7fdb2d906700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb280c7310 con 0x7fdb280c00e0 2023-03-03T17:42:33.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.822+0000 7fdb2d906700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb28155430 con 0x7fdb280caa30 2023-03-03T17:42:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.822+0000 7fdb2d906700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb280bfd10 con 0x7fdb28154ff0 2023-03-03T17:42:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb26ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb280caa30 0x7fdb280c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:42:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb267fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdb280c00e0 0x7fdb280c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56432/0 (socket says 172.21.15.7:56432) 2023-03-03T17:42:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb267fc700 1 -- 172.21.15.7:0/3820982284 learned_addr learned my addr 172.21.15.7:0/3820982284 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:42:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb277fe700 1 --2- 172.21.15.7:0/3820982284 >> v2:172.21.15.143:3300/0 conn(0x7fdb28154ff0 0x7fdb280cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:42:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb26ffd700 1 -- 172.21.15.7:0/3820982284 >> v2:172.21.15.143:3300/0 conn(0x7fdb28154ff0 msgr2=0x7fdb280cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:42:33.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb26ffd700 1 --2- 172.21.15.7:0/3820982284 >> v2:172.21.15.143:3300/0 conn(0x7fdb28154ff0 0x7fdb280cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:42:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb26ffd700 1 -- 172.21.15.7:0/3820982284 >> v1:172.21.15.7:6789/0 conn(0x7fdb280c00e0 legacy=0x7fdb280c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:42:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb26ffd700 1 -- 172.21.15.7:0/3820982284 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb280bfa30 con 0x7fdb280caa30 2023-03-03T17:42:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb25ffb700 1 -- 172.21.15.7:0/3820982284 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257274191 0 0) 0x7fdb280c7310 con 0x7fdb280c00e0 2023-03-03T17:42:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb26ffd700 1 --2- 172.21.15.7:0/3820982284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 0x7fdb280c5c60 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7fdb28071530 tx=0x7fdb1c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d739ada2ba8a0760 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:42:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb25ffb700 1 -- 172.21.15.7:0/3820982284 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c011830 con 0x7fdb280caa30 2023-03-03T17:42:33.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb25ffb700 1 -- 172.21.15.7:0/3820982284 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c011990 con 0x7fdb280caa30 2023-03-03T17:42:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.823+0000 7fdb25ffb700 1 -- 172.21.15.7:0/3820982284 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c011d20 con 0x7fdb280caa30 2023-03-03T17:42:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.824+0000 7fdb2d906700 1 -- 172.21.15.7:0/3820982284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 msgr2=0x7fdb280c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:42:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.824+0000 7fdb2d906700 1 --2- 172.21.15.7:0/3820982284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 0x7fdb280c5c60 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7fdb28071530 tx=0x7fdb1c009b20 comp rx=0 tx=0).stop 2023-03-03T17:42:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.824+0000 7fdb2d906700 1 -- 172.21.15.7:0/3820982284 shutdown_connections 2023-03-03T17:42:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.824+0000 7fdb2d906700 1 --2- 172.21.15.7:0/3820982284 >> v2:172.21.15.143:3300/0 conn(0x7fdb28154ff0 0x7fdb280cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:42:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.824+0000 7fdb2d906700 1 --2- 172.21.15.7:0/3820982284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 0x7fdb280c5c60 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:42:33.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.824+0000 7fdb2d906700 1 -- 172.21.15.7:0/3820982284 >> 172.21.15.7:0/3820982284 conn(0x7fdb28066e40 msgr2=0x7fdb28071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:42:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.824+0000 7fdb2d906700 1 -- 172.21.15.7:0/3820982284 shutdown_connections 2023-03-03T17:42:33.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.824+0000 7fdb2d906700 1 -- 172.21.15.7:0/3820982284 wait complete. 2023-03-03T17:42:33.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.825+0000 7fdb2d906700 1 Processor -- start 2023-03-03T17:42:33.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.825+0000 7fdb2d906700 1 -- start start 2023-03-03T17:42:33.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb2d906700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb280c00e0 0x7fdb28157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:42:33.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb2d906700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 0x7fdb28157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:42:33.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb2d906700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb28154ff0 0x7fdb2813ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:42:33.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb2d906700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281dec00 con 0x7fdb280caa30 2023-03-03T17:42:33.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb2d906700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb280c29d0 con 0x7fdb280c00e0 2023-03-03T17:42:33.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb2d906700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281dda40 con 0x7fdb28154ff0 2023-03-03T17:42:33.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb267fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 0x7fdb28157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:42:33.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb267fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 0x7fdb28157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34226/0 (socket says 172.21.15.7:34226) 2023-03-03T17:42:33.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb267fc700 1 -- 172.21.15.7:0/3728182137 learned_addr learned my addr 172.21.15.7:0/3728182137 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:42:33.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb277fe700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb28154ff0 0x7fdb2813ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:42:33.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb26ffd700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb280c00e0 0x7fdb28157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:42:33.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb267fc700 1 -- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb28154ff0 msgr2=0x7fdb2813ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:42:33.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb267fc700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb28154ff0 0x7fdb2813ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:42:33.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb267fc700 1 -- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb280c00e0 msgr2=0x7fdb28157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:42:33.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb267fc700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb280c00e0 0x7fdb28157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:42:33.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.826+0000 7fdb267fc700 1 -- 172.21.15.7:0/3728182137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c0094a0 con 0x7fdb280caa30 2023-03-03T17:42:33.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.827+0000 7fdb267fc700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 0x7fdb28157770 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fdb180010e0 tx=0x7fdb1800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:42:33.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.827+0000 7fdb0ffff700 1 -- 172.21.15.7:0/3728182137 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb180074e0 con 0x7fdb280caa30 2023-03-03T17:42:33.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.827+0000 7fdb0ffff700 1 -- 172.21.15.7:0/3728182137 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb18007640 con 0x7fdb280caa30 2023-03-03T17:42:33.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.827+0000 7fdb2d906700 1 -- 172.21.15.7:0/3728182137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb281547f0 con 0x7fdb280caa30 2023-03-03T17:42:33.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.827+0000 7fdb0ffff700 1 -- 172.21.15.7:0/3728182137 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18005a70 con 0x7fdb280caa30 2023-03-03T17:42:33.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.827+0000 7fdb2d906700 1 -- 172.21.15.7:0/3728182137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb28154510 con 0x7fdb280caa30 2023-03-03T17:42:33.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.828+0000 7fdb2d906700 1 -- 172.21.15.7:0/3728182137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb28154ad0 con 0x7fdb280caa30 2023-03-03T17:42:33.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.828+0000 7fdb0ffff700 1 -- 172.21.15.7:0/3728182137 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdb18019030 con 0x7fdb280caa30 2023-03-03T17:42:33.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.829+0000 7fdb0ffff700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb140682f0 0x7fdb1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:42:33.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.829+0000 7fdb0ffff700 1 -- 172.21.15.7:0/3728182137 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdb18085490 con 0x7fdb280caa30 2023-03-03T17:42:33.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.829+0000 7fdb26ffd700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb140682f0 0x7fdb1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:42:33.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.830+0000 7fdb26ffd700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb140682f0 0x7fdb1406a7a0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c0090b0 tx=0x7fdb1c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:42:33.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:33.833+0000 7fdb0ffff700 1 -- 172.21.15.7:0/3728182137 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdb18051830 con 0x7fdb280caa30 2023-03-03T17:42:34.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.078+0000 7fdb2d906700 1 -- 172.21.15.7:0/3728182137 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb28074bb0 con 0x7fdb280caa30 2023-03-03T17:42:34.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.079+0000 7fdb0ffff700 1 -- 172.21.15.7:0/3728182137 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdb18007e40 con 0x7fdb280caa30 2023-03-03T17:42:34.078 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:42:34.078 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:42:34.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 -- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb140682f0 msgr2=0x7fdb1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:42:34.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb140682f0 0x7fdb1406a7a0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c0090b0 tx=0x7fdb1c008030 comp rx=0 tx=0).stop 2023-03-03T17:42:34.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 -- 172.21.15.7:0/3728182137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 msgr2=0x7fdb28157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:42:34.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 0x7fdb28157770 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7fdb180010e0 tx=0x7fdb1800ca30 comp rx=0 tx=0).stop 2023-03-03T17:42:34.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 -- 172.21.15.7:0/3728182137 shutdown_connections 2023-03-03T17:42:34.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb28154ff0 0x7fdb2813ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:42:34.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb280c00e0 0x7fdb28157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:42:34.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb140682f0 0x7fdb1406a7a0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:42:34.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 --2- 172.21.15.7:0/3728182137 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb280caa30 0x7fdb28157770 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:42:34.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 -- 172.21.15.7:0/3728182137 >> 172.21.15.7:0/3728182137 conn(0x7fdb28066e40 msgr2=0x7fdb28151210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:42:34.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 -- 172.21.15.7:0/3728182137 shutdown_connections 2023-03-03T17:42:34.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:42:34.081+0000 7fdb2d906700 1 -- 172.21.15.7:0/3728182137 wait complete. 2023-03-03T17:42:34.082 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:42:34.092 INFO:tasks.mds_thrash.fs.[b]:waiting for 95 secs before thrashing 2023-03-03T17:42:59.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:42:59.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:30.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:30.085 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:43:39.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:43:39.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.549+0000 7fc18c13a700 1 Processor -- start 2023-03-03T17:43:39.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.549+0000 7fc18c13a700 1 -- start start 2023-03-03T17:43:39.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.550+0000 7fc18c13a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc184151d40 0x7fc1840736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:43:39.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.550+0000 7fc18c13a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18406d430 con 0x7fc1840c2930 2023-03-03T17:43:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.550+0000 7fc18c13a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18406cb00 con 0x7fc184073c30 2023-03-03T17:43:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.550+0000 7fc18c13a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc18406c2a0 con 0x7fc184151d40 2023-03-03T17:43:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.550+0000 7fc1896d5700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc1840c2930 0x7fc184157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56494/0 (socket says 172.21.15.7:56494) 2023-03-03T17:43:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.550+0000 7fc1896d5700 1 -- 172.21.15.7:0/1782845413 learned_addr learned my addr 172.21.15.7:0/1782845413 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:43:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.550+0000 7fc189ed6700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc184151d40 0x7fc1840736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:43:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640102929 0 0) 0x7fc18406d430 con 0x7fc1840c2930 2023-03-03T17:43:39.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1700035b0 con 0x7fc1840c2930 2023-03-03T17:43:39.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214771521 0 0) 0x7fc18406cb00 con 0x7fc184073c30 2023-03-03T17:43:39.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc18406d430 con 0x7fc184073c30 2023-03-03T17:43:39.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3488087332 0 0) 0x7fc1700035b0 con 0x7fc1840c2930 2023-03-03T17:43:39.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc184151d40 msgr2=0x7fc1840736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:43:39.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc184151d40 0x7fc1840736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:43:39.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 >> v1:172.21.15.143:6789/0 conn(0x7fc184073c30 legacy=0x7fc1841581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:43:39.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18406cde0 con 0x7fc1840c2930 2023-03-03T17:43:39.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc180001670 con 0x7fc1840c2930 2023-03-03T17:43:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 >> v1:172.21.15.7:6789/0 conn(0x7fc1840c2930 legacy=0x7fc184157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:43:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc170003c30 0x7fc170004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:43:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc170004850 0x7fc170004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:43:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc170005270 0x7fc17000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:43:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18406c2a0 con 0x7fc170005270 2023-03-03T17:43:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17000c190 con 0x7fc170004850 2023-03-03T17:43:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17000c350 con 0x7fc170003c30 2023-03-03T17:43:39.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc189ed6700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc184151d40 0x7fc1840736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:43:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc18a6d7700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc170005270 0x7fc17000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:43:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc1896d5700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc170004850 0x7fc170004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:43:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc189ed6700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc170003c30 0x7fc170004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:43:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc18a6d7700 1 -- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc170003c30 msgr2=0x7fc170004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:43:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc18a6d7700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc170003c30 0x7fc170004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:43:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc18a6d7700 1 -- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc170004850 msgr2=0x7fc170004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:43:39.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc18a6d7700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc170004850 0x7fc170004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:43:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc18a6d7700 1 -- 172.21.15.7:0/1782845413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1700035b0 con 0x7fc170005270 2023-03-03T17:43:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.551+0000 7fc189ed6700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc170003c30 0x7fc170004020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:43:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc18a6d7700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc170005270 0x7fc17000ba10 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00c320 tx=0x7fc170005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ceba7b59308626 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:43:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c001f20 con 0x7fc170005270 2023-03-03T17:43:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c003a00 con 0x7fc170005270 2023-03-03T17:43:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc188ed4700 1 -- 172.21.15.7:0/1782845413 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c00cea0 con 0x7fc170005270 2023-03-03T17:43:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc18c13a700 1 -- 172.21.15.7:0/1782845413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc170005270 msgr2=0x7fc17000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:43:39.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc18c13a700 1 --2- 172.21.15.7:0/1782845413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc170005270 0x7fc17000ba10 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00c320 tx=0x7fc170005880 comp rx=0 tx=0).stop 2023-03-03T17:43:39.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc18a6d7700 1 -- 172.21.15.7:0/1782845413 reap_dead start 2023-03-03T17:43:39.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc18c13a700 1 -- 172.21.15.7:0/1782845413 shutdown_connections 2023-03-03T17:43:39.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc18c13a700 1 -- 172.21.15.7:0/1782845413 >> 172.21.15.7:0/1782845413 conn(0x7fc1840a94d0 msgr2=0x7fc1840a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:43:39.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc18c13a700 1 -- 172.21.15.7:0/1782845413 shutdown_connections 2023-03-03T17:43:39.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.552+0000 7fc18c13a700 1 -- 172.21.15.7:0/1782845413 wait complete. 2023-03-03T17:43:39.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc18c13a700 1 Processor -- start 2023-03-03T17:43:39.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc18c13a700 1 -- start start 2023-03-03T17:43:39.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc18c13a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc18414e100 0x7fc184154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:43:39.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc18c13a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc18414e4d0 0x7fc1841549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:43:39.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc18c13a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc184154f10 0x7fc1840c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:43:39.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc18c13a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841edd40 con 0x7fc18414e4d0 2023-03-03T17:43:39.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc18c13a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841ee270 con 0x7fc18414e100 2023-03-03T17:43:39.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc18c13a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc18414ebf0 con 0x7fc184154f10 2023-03-03T17:43:39.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc1896d5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc18414e4d0 0x7fc1841549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:43:39.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc1896d5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc18414e4d0 0x7fc1841549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34296/0 (socket says 172.21.15.7:34296) 2023-03-03T17:43:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.554+0000 7fc1896d5700 1 -- 172.21.15.7:0/1618979014 learned_addr learned my addr 172.21.15.7:0/1618979014 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:43:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc18a6d7700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc184154f10 0x7fc1840c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:43:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc189ed6700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc18414e100 0x7fc184154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:43:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc1896d5700 1 -- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc184154f10 msgr2=0x7fc1840c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:43:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc1896d5700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc184154f10 0x7fc1840c61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:43:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc1896d5700 1 -- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc18414e100 msgr2=0x7fc184154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:43:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc1896d5700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc18414e100 0x7fc184154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:43:39.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc1896d5700 1 -- 172.21.15.7:0/1618979014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18406bea0 con 0x7fc18414e4d0 2023-03-03T17:43:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc1896d5700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc18414e4d0 0x7fc1841549d0 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7fc180007fd0 tx=0x7fc1800073c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:43:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc17a7fc700 1 -- 172.21.15.7:0/1618979014 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180012560 con 0x7fc18414e4d0 2023-03-03T17:43:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc17a7fc700 1 -- 172.21.15.7:0/1618979014 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc180012700 con 0x7fc18414e4d0 2023-03-03T17:43:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc18c13a700 1 -- 172.21.15.7:0/1618979014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18406d0c0 con 0x7fc18414e4d0 2023-03-03T17:43:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc17a7fc700 1 -- 172.21.15.7:0/1618979014 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180012ab0 con 0x7fc18414e4d0 2023-03-03T17:43:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.555+0000 7fc18c13a700 1 -- 172.21.15.7:0/1618979014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18406c030 con 0x7fc18414e4d0 2023-03-03T17:43:39.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.556+0000 7fc18c13a700 1 -- 172.21.15.7:0/1618979014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc184074bb0 con 0x7fc18414e4d0 2023-03-03T17:43:39.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.557+0000 7fc17a7fc700 1 -- 172.21.15.7:0/1618979014 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc180012c10 con 0x7fc18414e4d0 2023-03-03T17:43:39.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.557+0000 7fc17a7fc700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1680682f0 0x7fc16806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:43:39.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.557+0000 7fc17a7fc700 1 -- 172.21.15.7:0/1618979014 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc180086180 con 0x7fc18414e4d0 2023-03-03T17:43:39.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.557+0000 7fc189ed6700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1680682f0 0x7fc16806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:43:39.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.558+0000 7fc189ed6700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1680682f0 0x7fc16806a7a0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fc1841e0830 tx=0x7fc174008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:43:39.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.561+0000 7fc17a7fc700 1 -- 172.21.15.7:0/1618979014 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc1800524a0 con 0x7fc18414e4d0 2023-03-03T17:43:39.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.799+0000 7fc18c13a700 1 -- 172.21.15.7:0/1618979014 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1840a9fb0 con 0x7fc18414e4d0 2023-03-03T17:43:39.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.800+0000 7fc17a7fc700 1 -- 172.21.15.7:0/1618979014 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc180055ac0 con 0x7fc18414e4d0 2023-03-03T17:43:39.799 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:43:39.800 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:43:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 -- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1680682f0 msgr2=0x7fc16806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:43:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1680682f0 0x7fc16806a7a0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fc1841e0830 tx=0x7fc174008040 comp rx=0 tx=0).stop 2023-03-03T17:43:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 -- 172.21.15.7:0/1618979014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc18414e4d0 msgr2=0x7fc1841549d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:43:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc18414e4d0 0x7fc1841549d0 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7fc180007fd0 tx=0x7fc1800073c0 comp rx=0 tx=0).stop 2023-03-03T17:43:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 -- 172.21.15.7:0/1618979014 shutdown_connections 2023-03-03T17:43:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc184154f10 0x7fc1840c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:43:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc18414e100 0x7fc184154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:43:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1680682f0 0x7fc16806a7a0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:43:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 --2- 172.21.15.7:0/1618979014 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc18414e4d0 0x7fc1841549d0 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:43:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 -- 172.21.15.7:0/1618979014 >> 172.21.15.7:0/1618979014 conn(0x7fc1840a94d0 msgr2=0x7fc1840714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:43:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 -- 172.21.15.7:0/1618979014 shutdown_connections 2023-03-03T17:43:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:43:39.802+0000 7fc18c13a700 1 -- 172.21.15.7:0/1618979014 wait complete. 2023-03-03T17:43:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:43:39.813 INFO:tasks.mds_thrash.fs.[a]:waiting for 25 secs before thrashing 2023-03-03T17:44:00.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:00.223 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:04.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:44:04.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.952+0000 7f3bd5462700 1 Processor -- start 2023-03-03T17:44:04.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.952+0000 7f3bd5462700 1 -- start start 2023-03-03T17:44:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.953+0000 7f3bd5462700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3bd00caa30 0x7f3bd00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.953+0000 7f3bd5462700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00c00e0 0x7f3bd0154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.953+0000 7f3bd5462700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3bd00cae00 0x7f3bd00c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.953+0000 7f3bd5462700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd00cb280 con 0x7f3bd00cae00 2023-03-03T17:44:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.953+0000 7f3bd5462700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd00bfd10 con 0x7f3bd00caa30 2023-03-03T17:44:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.953+0000 7f3bd5462700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd00bfa30 con 0x7f3bd00c00e0 2023-03-03T17:44:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3bd00cae00 0x7f3bd00c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3bd00cae00 0x7f3bd00c6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34320/0 (socket says 172.21.15.7:34320) 2023-03-03T17:44:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcf7fe700 1 -- 172.21.15.7:0/2278753091 learned_addr learned my addr 172.21.15.7:0/2278753091 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:44:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bceffd700 1 --2- 172.21.15.7:0/2278753091 >> v2:172.21.15.143:3300/0 conn(0x7f3bd00caa30 0x7f3bd00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bce7fc700 1 --2- 172.21.15.7:0/2278753091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00c00e0 0x7f3bd0154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcf7fe700 1 -- 172.21.15.7:0/2278753091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00c00e0 msgr2=0x7f3bd0154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:44:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcf7fe700 1 --2- 172.21.15.7:0/2278753091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00c00e0 0x7f3bd0154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcf7fe700 1 -- 172.21.15.7:0/2278753091 >> v2:172.21.15.143:3300/0 conn(0x7f3bd00caa30 msgr2=0x7f3bd00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:44:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcf7fe700 1 --2- 172.21.15.7:0/2278753091 >> v2:172.21.15.143:3300/0 conn(0x7f3bd00caa30 0x7f3bd00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcf7fe700 1 -- 172.21.15.7:0/2278753091 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd0154510 con 0x7f3bd00cae00 2023-03-03T17:44:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bceffd700 1 --2- 172.21.15.7:0/2278753091 >> v2:172.21.15.143:3300/0 conn(0x7f3bd00caa30 0x7f3bd00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:44:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcf7fe700 1 --2- 172.21.15.7:0/2278753091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00cae00 0x7f3bd00c6c00 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f3bd00c81b0 tx=0x7f3bc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9299d6511d11be server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:44:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcdffb700 1 -- 172.21.15.7:0/2278753091 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc800e070 con 0x7f3bd00cae00 2023-03-03T17:44:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcdffb700 1 -- 172.21.15.7:0/2278753091 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc800b910 con 0x7f3bd00cae00 2023-03-03T17:44:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.954+0000 7f3bcdffb700 1 -- 172.21.15.7:0/2278753091 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc800bcc0 con 0x7f3bd00cae00 2023-03-03T17:44:04.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.955+0000 7f3bd5462700 1 -- 172.21.15.7:0/2278753091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00cae00 msgr2=0x7f3bd00c6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:04.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.955+0000 7f3bd5462700 1 --2- 172.21.15.7:0/2278753091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00cae00 0x7f3bd00c6c00 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 crypto rx=0x7f3bd00c81b0 tx=0x7f3bc800b330 comp rx=0 tx=0).stop 2023-03-03T17:44:04.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.955+0000 7f3bd5462700 1 -- 172.21.15.7:0/2278753091 shutdown_connections 2023-03-03T17:44:04.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.955+0000 7f3bd5462700 1 --2- 172.21.15.7:0/2278753091 >> v2:172.21.15.143:3300/0 conn(0x7f3bd00caa30 0x7f3bd00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.955+0000 7f3bd5462700 1 --2- 172.21.15.7:0/2278753091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00cae00 0x7f3bd00c6c00 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.955+0000 7f3bd5462700 1 --2- 172.21.15.7:0/2278753091 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00c00e0 0x7f3bd0154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.955+0000 7f3bd5462700 1 -- 172.21.15.7:0/2278753091 >> 172.21.15.7:0/2278753091 conn(0x7f3bd0066e40 msgr2=0x7f3bd0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:44:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.955+0000 7f3bd5462700 1 -- 172.21.15.7:0/2278753091 shutdown_connections 2023-03-03T17:44:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.955+0000 7f3bd5462700 1 -- 172.21.15.7:0/2278753091 wait complete. 2023-03-03T17:44:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.956+0000 7f3bd5462700 1 Processor -- start 2023-03-03T17:44:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.956+0000 7f3bd5462700 1 -- start start 2023-03-03T17:44:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bd5462700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00c00e0 0x7f3bd01561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bd5462700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00caa30 0x7f3bd014d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bd5462700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bd00cae00 0x7f3bd014d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bd5462700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd00ab3e0 con 0x7f3bd00caa30 2023-03-03T17:44:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bd5462700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01edbd0 con 0x7f3bd00cae00 2023-03-03T17:44:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bd5462700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd01570f0 con 0x7f3bd00c00e0 2023-03-03T17:44:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bce7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00caa30 0x7f3bd014d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bce7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00caa30 0x7f3bd014d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34324/0 (socket says 172.21.15.7:34324) 2023-03-03T17:44:04.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bce7fc700 1 -- 172.21.15.7:0/416403618 learned_addr learned my addr 172.21.15.7:0/416403618 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:44:04.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bceffd700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00c00e0 0x7f3bd01561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:04.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bcf7fe700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bd00cae00 0x7f3bd014d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:04.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bce7fc700 1 -- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00c00e0 msgr2=0x7f3bd01561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:04.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bce7fc700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00c00e0 0x7f3bd01561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:04.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bce7fc700 1 -- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bd00cae00 msgr2=0x7f3bd014d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:04.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bce7fc700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bd00cae00 0x7f3bd014d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.957+0000 7f3bce7fc700 1 -- 172.21.15.7:0/416403618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc8009b30 con 0x7f3bd00caa30 2023-03-03T17:44:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.958+0000 7f3bce7fc700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00caa30 0x7f3bd014d2d0 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000e870 tx=0x7f3bc000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:44:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.958+0000 7f3bb77fe700 1 -- 172.21.15.7:0/416403618 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0005a20 con 0x7f3bd00caa30 2023-03-03T17:44:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.958+0000 7f3bb77fe700 1 -- 172.21.15.7:0/416403618 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc0005b80 con 0x7f3bd00caa30 2023-03-03T17:44:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.958+0000 7f3bd5462700 1 -- 172.21.15.7:0/416403618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd0154ad0 con 0x7f3bd00caa30 2023-03-03T17:44:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.958+0000 7f3bb77fe700 1 -- 172.21.15.7:0/416403618 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0013680 con 0x7f3bd00caa30 2023-03-03T17:44:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.958+0000 7f3bd5462700 1 -- 172.21.15.7:0/416403618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd01547f0 con 0x7f3bd00caa30 2023-03-03T17:44:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.959+0000 7f3bd5462700 1 -- 172.21.15.7:0/416403618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd00bf650 con 0x7f3bd00caa30 2023-03-03T17:44:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.959+0000 7f3bb77fe700 1 -- 172.21.15.7:0/416403618 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3bc0022070 con 0x7f3bd00caa30 2023-03-03T17:44:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.960+0000 7f3bb77fe700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bbc068320 0x7f3bbc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.960+0000 7f3bb77fe700 1 -- 172.21.15.7:0/416403618 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3bc0086050 con 0x7f3bd00caa30 2023-03-03T17:44:04.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.960+0000 7f3bceffd700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bbc068320 0x7f3bbc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:04.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.961+0000 7f3bceffd700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bbc068320 0x7f3bbc06a7d0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4006f10 tx=0x7f3bc4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:44:04.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:04.964+0000 7f3bb77fe700 1 -- 172.21.15.7:0/416403618 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3bc0052370 con 0x7f3bd00caa30 2023-03-03T17:44:05.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.205+0000 7f3bd5462700 1 -- 172.21.15.7:0/416403618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bd0074bb0 con 0x7f3bd00caa30 2023-03-03T17:44:05.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.205+0000 7f3bb77fe700 1 -- 172.21.15.7:0/416403618 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3bc0020090 con 0x7f3bd00caa30 2023-03-03T17:44:05.206 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:44:05.206 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:44:05.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.207+0000 7f3bd5462700 1 -- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bbc068320 msgr2=0x7f3bbc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:05.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.207+0000 7f3bd5462700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bbc068320 0x7f3bbc06a7d0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4006f10 tx=0x7f3bc4008040 comp rx=0 tx=0).stop 2023-03-03T17:44:05.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.207+0000 7f3bd5462700 1 -- 172.21.15.7:0/416403618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00caa30 msgr2=0x7f3bd014d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:05.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.207+0000 7f3bd5462700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00caa30 0x7f3bd014d2d0 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000e870 tx=0x7f3bc000eb80 comp rx=0 tx=0).stop 2023-03-03T17:44:05.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.208+0000 7f3bd5462700 1 -- 172.21.15.7:0/416403618 shutdown_connections 2023-03-03T17:44:05.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.208+0000 7f3bd5462700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3bd00cae00 0x7f3bd014d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:05.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.208+0000 7f3bd5462700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3bd00c00e0 0x7f3bd01561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:05.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.208+0000 7f3bd5462700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3bbc068320 0x7f3bbc06a7d0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:05.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.208+0000 7f3bd5462700 1 --2- 172.21.15.7:0/416403618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3bd00caa30 0x7f3bd014d2d0 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:05.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.208+0000 7f3bd5462700 1 -- 172.21.15.7:0/416403618 >> 172.21.15.7:0/416403618 conn(0x7f3bd0066e40 msgr2=0x7f3bd01512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:44:05.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.208+0000 7f3bd5462700 1 -- 172.21.15.7:0/416403618 shutdown_connections 2023-03-03T17:44:05.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:05.208+0000 7f3bd5462700 1 -- 172.21.15.7:0/416403618 wait complete. 2023-03-03T17:44:05.210 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:44:05.220 INFO:tasks.mds_thrash.fs.[a]:waiting for 95 secs before thrashing 2023-03-03T17:44:09.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:44:09.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.233+0000 7f0c4774b700 1 Processor -- start 2023-03-03T17:44:09.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.233+0000 7f0c4774b700 1 -- start start 2023-03-03T17:44:09.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c4774b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0c40071e80 0x7f0c40151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:09.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c4774b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c401510d0 con 0x7f0c40072250 2023-03-03T17:44:09.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c4774b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c400c3740 con 0x7f0c4014e900 2023-03-03T17:44:09.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c4774b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c40070670 con 0x7f0c40071e80 2023-03-03T17:44:09.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c44ce6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0c40072250 0x7f0c4014e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56546/0 (socket says 172.21.15.7:56546) 2023-03-03T17:44:09.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c44ce6700 1 -- 172.21.15.7:0/1582821325 learned_addr learned my addr 172.21.15.7:0/1582821325 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:44:09.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c454e7700 1 --2- 172.21.15.7:0/1582821325 >> v2:172.21.15.143:3300/0 conn(0x7f0c40071e80 0x7f0c40151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:09.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067485522 0 0) 0x7f0c401510d0 con 0x7f0c40072250 2023-03-03T17:44:09.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c2c0035b0 con 0x7f0c40072250 2023-03-03T17:44:09.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889009146 0 0) 0x7f0c400c3740 con 0x7f0c4014e900 2023-03-03T17:44:09.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c401510d0 con 0x7f0c4014e900 2023-03-03T17:44:09.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58484147 0 0) 0x7f0c2c0035b0 con 0x7f0c40072250 2023-03-03T17:44:09.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 >> v2:172.21.15.143:3300/0 conn(0x7f0c40071e80 msgr2=0x7f0c40151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:44:09.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 --2- 172.21.15.7:0/1582821325 >> v2:172.21.15.143:3300/0 conn(0x7f0c40071e80 0x7f0c40151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:09.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 >> v1:172.21.15.143:6789/0 conn(0x7f0c4014e900 legacy=0x7f0c4015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:09.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c400c39e0 con 0x7f0c40072250 2023-03-03T17:44:09.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0c30001670 con 0x7f0c40072250 2023-03-03T17:44:09.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.234+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 >> v1:172.21.15.7:6789/0 conn(0x7f0c40072250 legacy=0x7f0c4014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:09.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c37fff700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c2c003c50 0x7f0c2c004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:09.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c37fff700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2c004850 0x7f0c2c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:09.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c37fff700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c2c0052c0 0x7f0c2c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:09.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c40070670 con 0x7f0c2c003c50 2023-03-03T17:44:09.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c00c1e0 con 0x7f0c2c004850 2023-03-03T17:44:09.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2c00c3a0 con 0x7f0c2c0052c0 2023-03-03T17:44:09.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c454e7700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c2c003c50 0x7f0c2c004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:09.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c44ce6700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2c004850 0x7f0c2c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:09.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c45ce8700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c2c0052c0 0x7f0c2c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:09.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c454e7700 1 -- 172.21.15.7:0/1582821325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c2c0052c0 msgr2=0x7f0c2c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:44:09.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c454e7700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c2c0052c0 0x7f0c2c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:09.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c454e7700 1 -- 172.21.15.7:0/1582821325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2c004850 msgr2=0x7f0c2c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:44:09.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c454e7700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2c004850 0x7f0c2c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:09.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c45ce8700 1 -- 172.21.15.7:0/1582821325 reap_dead start 2023-03-03T17:44:09.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c454e7700 1 -- 172.21.15.7:0/1582821325 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c0035b0 con 0x7f0c2c003c50 2023-03-03T17:44:09.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c454e7700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c2c003c50 0x7f0c2c004040 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c009b60 tx=0x7f0c2c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10864ef1818ddd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:44:09.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c00e6c0 con 0x7f0c2c003c50 2023-03-03T17:44:09.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3c00e820 con 0x7f0c2c003c50 2023-03-03T17:44:09.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.235+0000 7f0c37fff700 1 -- 172.21.15.7:0/1582821325 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c00eb90 con 0x7f0c2c003c50 2023-03-03T17:44:09.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.236+0000 7f0c4774b700 1 -- 172.21.15.7:0/1582821325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c2c003c50 msgr2=0x7f0c2c004040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:09.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.236+0000 7f0c4774b700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c2c003c50 0x7f0c2c004040 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c009b60 tx=0x7f0c2c005880 comp rx=0 tx=0).stop 2023-03-03T17:44:09.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.236+0000 7f0c4774b700 1 -- 172.21.15.7:0/1582821325 shutdown_connections 2023-03-03T17:44:09.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.236+0000 7f0c4774b700 1 --2- 172.21.15.7:0/1582821325 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c2c003c50 0x7f0c2c004040 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:09.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.236+0000 7f0c4774b700 1 -- 172.21.15.7:0/1582821325 >> 172.21.15.7:0/1582821325 conn(0x7f0c400a94d0 msgr2=0x7f0c400a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:44:09.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.236+0000 7f0c4774b700 1 -- 172.21.15.7:0/1582821325 shutdown_connections 2023-03-03T17:44:09.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.236+0000 7f0c4774b700 1 -- 172.21.15.7:0/1582821325 wait complete. 2023-03-03T17:44:09.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.237+0000 7f0c4774b700 1 Processor -- start 2023-03-03T17:44:09.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.237+0000 7f0c4774b700 1 -- start start 2023-03-03T17:44:09.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c4774b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2c003c50 0x7f0c401435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:09.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c4774b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c400ba5d0 0x7f0c401dc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:09.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c4774b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4014d5d0 0x7f0c401dca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:09.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c4774b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c400ba3b0 con 0x7f0c400ba5d0 2023-03-03T17:44:09.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c4774b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c401dcf90 con 0x7f0c2c003c50 2023-03-03T17:44:09.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c4774b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c401ee440 con 0x7f0c4014d5d0 2023-03-03T17:44:09.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c44ce6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c400ba5d0 0x7f0c401dc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:09.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c44ce6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c400ba5d0 0x7f0c401dc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34346/0 (socket says 172.21.15.7:34346) 2023-03-03T17:44:09.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c44ce6700 1 -- 172.21.15.7:0/1021703559 learned_addr learned my addr 172.21.15.7:0/1021703559 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:44:09.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c454e7700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2c003c50 0x7f0c401435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:09.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c45ce8700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4014d5d0 0x7f0c401dca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:09.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c44ce6700 1 -- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4014d5d0 msgr2=0x7f0c401dca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:09.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.238+0000 7f0c44ce6700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4014d5d0 0x7f0c401dca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:09.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.239+0000 7f0c44ce6700 1 -- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2c003c50 msgr2=0x7f0c401435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:09.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.239+0000 7f0c44ce6700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2c003c50 0x7f0c401435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:09.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.239+0000 7f0c44ce6700 1 -- 172.21.15.7:0/1021703559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4006c9a0 con 0x7f0c400ba5d0 2023-03-03T17:44:09.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.239+0000 7f0c45ce8700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4014d5d0 0x7f0c401dca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:44:09.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.239+0000 7f0c44ce6700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c400ba5d0 0x7f0c401dc510 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7f0c30004f50 tx=0x7f0c30007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:44:09.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.239+0000 7f0c35ffb700 1 -- 172.21.15.7:0/1021703559 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c30012560 con 0x7f0c400ba5d0 2023-03-03T17:44:09.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.239+0000 7f0c35ffb700 1 -- 172.21.15.7:0/1021703559 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c300126e0 con 0x7f0c400ba5d0 2023-03-03T17:44:09.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.239+0000 7f0c4774b700 1 -- 172.21.15.7:0/1021703559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c40151300 con 0x7f0c400ba5d0 2023-03-03T17:44:09.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.239+0000 7f0c35ffb700 1 -- 172.21.15.7:0/1021703559 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c30012a90 con 0x7f0c400ba5d0 2023-03-03T17:44:09.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.239+0000 7f0c4774b700 1 -- 172.21.15.7:0/1021703559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4006cb30 con 0x7f0c400ba5d0 2023-03-03T17:44:09.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.240+0000 7f0c4774b700 1 -- 172.21.15.7:0/1021703559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c401ee290 con 0x7f0c400ba5d0 2023-03-03T17:44:09.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.241+0000 7f0c35ffb700 1 -- 172.21.15.7:0/1021703559 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0c30012bf0 con 0x7f0c400ba5d0 2023-03-03T17:44:09.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.241+0000 7f0c35ffb700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c240682f0 0x7f0c2406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:44:09.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.242+0000 7f0c35ffb700 1 -- 172.21.15.7:0/1021703559 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0c30086b10 con 0x7f0c400ba5d0 2023-03-03T17:44:09.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.242+0000 7f0c454e7700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c240682f0 0x7f0c2406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:44:09.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.242+0000 7f0c454e7700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c240682f0 0x7f0c2406a7a0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0090b0 tx=0x7f0c3c01e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:44:09.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.245+0000 7f0c35ffb700 1 -- 172.21.15.7:0/1021703559 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0c30052f50 con 0x7f0c400ba5d0 2023-03-03T17:44:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.481+0000 7f0c4774b700 1 -- 172.21.15.7:0/1021703559 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c40074bb0 con 0x7f0c400ba5d0 2023-03-03T17:44:09.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.481+0000 7f0c35ffb700 1 -- 172.21.15.7:0/1021703559 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0c30056570 con 0x7f0c400ba5d0 2023-03-03T17:44:09.482 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:44:09.482 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:44:09.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 -- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c240682f0 msgr2=0x7f0c2406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c240682f0 0x7f0c2406a7a0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0090b0 tx=0x7f0c3c01e040 comp rx=0 tx=0).stop 2023-03-03T17:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 -- 172.21.15.7:0/1021703559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c400ba5d0 msgr2=0x7f0c401dc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c400ba5d0 0x7f0c401dc510 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7f0c30004f50 tx=0x7f0c30007340 comp rx=0 tx=0).stop 2023-03-03T17:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 -- 172.21.15.7:0/1021703559 shutdown_connections 2023-03-03T17:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0c4014d5d0 0x7f0c401dca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c2c003c50 0x7f0c401435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:09.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0c240682f0 0x7f0c2406a7a0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 --2- 172.21.15.7:0/1021703559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c400ba5d0 0x7f0c401dc510 unknown :-1 s=CLOSED pgs=3003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 -- 172.21.15.7:0/1021703559 >> 172.21.15.7:0/1021703559 conn(0x7f0c400a94d0 msgr2=0x7f0c4006a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 -- 172.21.15.7:0/1021703559 shutdown_connections 2023-03-03T17:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:44:09.483+0000 7f0c4774b700 1 -- 172.21.15.7:0/1021703559 wait complete. 2023-03-03T17:44:09.485 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:44:09.495 INFO:tasks.mds_thrash.fs.[b]:waiting for 71 secs before thrashing 2023-03-03T17:44:30.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:44:30.369 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:00.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:00.525 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:20.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:45:20.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.637+0000 7feffa0c8700 1 Processor -- start 2023-03-03T17:45:20.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.637+0000 7feffa0c8700 1 -- start start 2023-03-03T17:45:20.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.637+0000 7feffa0c8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feff406c480 0x7feff41581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:20.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.637+0000 7feffa0c8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff406cb70 con 0x7feff40c2950 2023-03-03T17:45:20.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feffa0c8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff406ce50 con 0x7feff4151b20 2023-03-03T17:45:20.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feffa0c8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff406c2f0 con 0x7feff406c480 2023-03-03T17:45:20.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feff406c480 0x7feff41581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:20.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff2ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7feff40c2950 0x7feff4157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56608/0 (socket says 172.21.15.7:56608) 2023-03-03T17:45:20.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff2ffd700 1 -- 172.21.15.7:0/2426944428 learned_addr learned my addr 172.21.15.7:0/2426944428 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:45:20.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff3fff700 1 -- 172.21.15.7:0/2426944428 >> v1:172.21.15.143:6789/0 conn(0x7feff4151b20 legacy=0x7feff40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:20.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff3fff700 1 -- 172.21.15.7:0/2426944428 >> v1:172.21.15.7:6789/0 conn(0x7feff40c2950 legacy=0x7feff4157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:20.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff27fc700 1 -- 172.21.15.7:0/2426944428 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148175001 0 0) 0x7feff406cb70 con 0x7feff40c2950 2023-03-03T17:45:20.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff3fff700 1 -- 172.21.15.7:0/2426944428 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff406d130 con 0x7feff406c480 2023-03-03T17:45:20.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff3fff700 1 --2- 172.21.15.7:0/2426944428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff406c480 0x7feff41581f0 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7fefec009d90 tx=0x7fefec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c89cf33c4f6bfe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:45:20.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff27fc700 1 -- 172.21.15.7:0/2426944428 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec00e070 con 0x7feff406c480 2023-03-03T17:45:20.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff27fc700 1 -- 172.21.15.7:0/2426944428 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec00bbd0 con 0x7feff406c480 2023-03-03T17:45:20.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.638+0000 7feff27fc700 1 -- 172.21.15.7:0/2426944428 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec012430 con 0x7feff406c480 2023-03-03T17:45:20.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.639+0000 7feffa0c8700 1 -- 172.21.15.7:0/2426944428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff406c480 msgr2=0x7feff41581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:20.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.639+0000 7feffa0c8700 1 --2- 172.21.15.7:0/2426944428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff406c480 0x7feff41581f0 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 crypto rx=0x7fefec009d90 tx=0x7fefec00b330 comp rx=0 tx=0).stop 2023-03-03T17:45:20.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.639+0000 7feffa0c8700 1 -- 172.21.15.7:0/2426944428 shutdown_connections 2023-03-03T17:45:20.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.639+0000 7feffa0c8700 1 --2- 172.21.15.7:0/2426944428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff406c480 0x7feff41581f0 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:20.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.639+0000 7feffa0c8700 1 -- 172.21.15.7:0/2426944428 >> 172.21.15.7:0/2426944428 conn(0x7feff40a94d0 msgr2=0x7feff40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:45:20.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.639+0000 7feffa0c8700 1 -- 172.21.15.7:0/2426944428 shutdown_connections 2023-03-03T17:45:20.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.639+0000 7feffa0c8700 1 -- 172.21.15.7:0/2426944428 wait complete. 2023-03-03T17:45:20.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.640+0000 7feffa0c8700 1 Processor -- start 2023-03-03T17:45:20.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.640+0000 7feffa0c8700 1 -- start start 2023-03-03T17:45:20.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feffa0c8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feff406c480 0x7feff40c0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:20.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feffa0c8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feff40c2950 0x7feff406a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:20.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feffa0c8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff4151b20 0x7feff414c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:20.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feffa0c8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff40bfe60 con 0x7feff4151b20 2023-03-03T17:45:20.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feffa0c8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff4157a00 con 0x7feff406c480 2023-03-03T17:45:20.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feffa0c8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff4157ce0 con 0x7feff40c2950 2023-03-03T17:45:20.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feff3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff4151b20 0x7feff414c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:20.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feff3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff4151b20 0x7feff414c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34406/0 (socket says 172.21.15.7:34406) 2023-03-03T17:45:20.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feff3fff700 1 -- 172.21.15.7:0/2580340455 learned_addr learned my addr 172.21.15.7:0/2580340455 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:45:20.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feff2ffd700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feff40c2950 0x7feff406a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:20.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feff37fe700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feff406c480 0x7feff40c0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:20.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feff3fff700 1 -- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feff40c2950 msgr2=0x7feff406a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:20.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feff3fff700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feff40c2950 0x7feff406a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:20.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feff3fff700 1 -- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feff406c480 msgr2=0x7feff40c0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:20.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.641+0000 7feff3fff700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feff406c480 0x7feff40c0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:20.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.642+0000 7feff3fff700 1 -- 172.21.15.7:0/2580340455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec009a70 con 0x7feff4151b20 2023-03-03T17:45:20.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.642+0000 7feff3fff700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff4151b20 0x7feff414c310 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7fefec014040 tx=0x7fefec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:45:20.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.642+0000 7fefdbfff700 1 -- 172.21.15.7:0/2580340455 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec01b810 con 0x7feff4151b20 2023-03-03T17:45:20.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.642+0000 7fefdbfff700 1 -- 172.21.15.7:0/2580340455 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec020070 con 0x7feff4151b20 2023-03-03T17:45:20.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.642+0000 7feffa0c8700 1 -- 172.21.15.7:0/2580340455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff4158730 con 0x7feff4151b20 2023-03-03T17:45:20.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.642+0000 7fefdbfff700 1 -- 172.21.15.7:0/2580340455 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec01bbb0 con 0x7feff4151b20 2023-03-03T17:45:20.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.642+0000 7feffa0c8700 1 -- 172.21.15.7:0/2580340455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff406d410 con 0x7feff4151b20 2023-03-03T17:45:20.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.643+0000 7feffa0c8700 1 -- 172.21.15.7:0/2580340455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff4151ef0 con 0x7feff4151b20 2023-03-03T17:45:20.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.643+0000 7fefdbfff700 1 -- 172.21.15.7:0/2580340455 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fefec02e070 con 0x7feff4151b20 2023-03-03T17:45:20.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.644+0000 7fefdbfff700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefe00682f0 0x7fefe006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:20.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.644+0000 7fefdbfff700 1 -- 172.21.15.7:0/2580340455 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fefec022070 con 0x7feff4151b20 2023-03-03T17:45:20.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.644+0000 7feff37fe700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefe00682f0 0x7fefe006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:20.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.645+0000 7feff37fe700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefe00682f0 0x7fefe006a7a0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fefe4007fd0 tx=0x7fefe4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:45:20.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.648+0000 7fefdbfff700 1 -- 172.21.15.7:0/2580340455 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fefec0545c0 con 0x7feff4151b20 2023-03-03T17:45:20.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.887+0000 7feffa0c8700 1 -- 172.21.15.7:0/2580340455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feff40be8e0 con 0x7feff4151b20 2023-03-03T17:45:20.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.887+0000 7fefdbfff700 1 -- 172.21.15.7:0/2580340455 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7feff40be8e0 con 0x7feff4151b20 2023-03-03T17:45:20.887 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:45:20.887 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:45:20.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 -- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefe00682f0 msgr2=0x7fefe006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:20.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefe00682f0 0x7fefe006a7a0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fefe4007fd0 tx=0x7fefe4009040 comp rx=0 tx=0).stop 2023-03-03T17:45:20.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 -- 172.21.15.7:0/2580340455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff4151b20 msgr2=0x7feff414c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff4151b20 0x7feff414c310 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7fefec014040 tx=0x7fefec004ad0 comp rx=0 tx=0).stop 2023-03-03T17:45:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 -- 172.21.15.7:0/2580340455 shutdown_connections 2023-03-03T17:45:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fefe00682f0 0x7fefe006a7a0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feff4151b20 0x7feff414c310 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feff406c480 0x7feff40c0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 --2- 172.21.15.7:0/2580340455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feff40c2950 0x7feff406a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:20.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 -- 172.21.15.7:0/2580340455 >> 172.21.15.7:0/2580340455 conn(0x7feff40a94d0 msgr2=0x7feff414d420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:45:20.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 -- 172.21.15.7:0/2580340455 shutdown_connections 2023-03-03T17:45:20.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:20.890+0000 7feffa0c8700 1 -- 172.21.15.7:0/2580340455 wait complete. 2023-03-03T17:45:20.891 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:45:20.901 INFO:tasks.mds_thrash.fs.[b]:waiting for 45 secs before thrashing 2023-03-03T17:45:30.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:30.654 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:45:40.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:45:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.363+0000 7f670759f700 1 Processor -- start 2023-03-03T17:45:40.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.363+0000 7f670759f700 1 -- start start 2023-03-03T17:45:40.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f670759f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670014e900 0x7f670015ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:40.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f670759f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67001510d0 con 0x7f6700072250 2023-03-03T17:45:40.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f670759f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67000c3740 con 0x7f6700071e80 2023-03-03T17:45:40.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f670759f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6700070670 con 0x7f670014e900 2023-03-03T17:45:40.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f6704b3a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6700072250 0x7f670014e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56632/0 (socket says 172.21.15.7:56632) 2023-03-03T17:45:40.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f6704b3a700 1 -- 172.21.15.7:0/4257409132 learned_addr learned my addr 172.21.15.7:0/4257409132 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:45:40.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f6705b3c700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670014e900 0x7f670015ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:40.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363525177 0 0) 0x7f67001510d0 con 0x7f6700072250 2023-03-03T17:45:40.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66ec0035b0 con 0x7f6700072250 2023-03-03T17:45:40.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1112947657 0 0) 0x7f66ec0035b0 con 0x7f6700072250 2023-03-03T17:45:40.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670014e900 msgr2=0x7f670015ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:45:40.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f66f7fff700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670014e900 0x7f670015ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:40.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 >> v1:172.21.15.143:6789/0 conn(0x7f6700071e80 legacy=0x7f6700151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:40.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67000c39e0 con 0x7f6700072250 2023-03-03T17:45:40.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f66f0001670 con 0x7f6700072250 2023-03-03T17:45:40.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 >> v1:172.21.15.7:6789/0 conn(0x7f6700072250 legacy=0x7f670014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:40.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.364+0000 7f66f7fff700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 0x7f66ec003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:40.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f66f7fff700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670014e900 0x7f66ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:40.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f66f7fff700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66ec004d90 0x7f66ec005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:40.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6700070670 con 0x7f66ec003aa0 2023-03-03T17:45:40.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec00bb10 con 0x7f66ec004d90 2023-03-03T17:45:40.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec00dee0 con 0x7f670014e900 2023-03-03T17:45:40.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f670533b700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 0x7f66ec003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:40.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f6705b3c700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66ec004d90 0x7f66ec005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:40.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f6704b3a700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670014e900 0x7f66ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:40.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f670533b700 1 -- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670014e900 msgr2=0x7f66ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:45:40.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f670533b700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670014e900 0x7f66ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:40.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f670533b700 1 -- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66ec004d90 msgr2=0x7f66ec005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:45:40.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f670533b700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66ec004d90 0x7f66ec005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:40.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f670533b700 1 -- 172.21.15.7:0/4257409132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec0035b0 con 0x7f66ec003aa0 2023-03-03T17:45:40.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f6705b3c700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66ec004d90 0x7f66ec005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:45:40.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f6704b3a700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670014e900 0x7f66ec004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:45:40.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f670533b700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 0x7f66ec003e70 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0018b0 tx=0x7f66ec005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7a0a768ea2fecc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:45:40.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc001370 con 0x7f66ec003aa0 2023-03-03T17:45:40.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66fc00c6b0 con 0x7f66ec003aa0 2023-03-03T17:45:40.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.365+0000 7f66f7fff700 1 -- 172.21.15.7:0/4257409132 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc00ca60 con 0x7f66ec003aa0 2023-03-03T17:45:40.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.366+0000 7f670759f700 1 -- 172.21.15.7:0/4257409132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 msgr2=0x7f66ec003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:40.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.366+0000 7f670759f700 1 --2- 172.21.15.7:0/4257409132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 0x7f66ec003e70 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0018b0 tx=0x7f66ec005880 comp rx=0 tx=0).stop 2023-03-03T17:45:40.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.366+0000 7f6705b3c700 1 -- 172.21.15.7:0/4257409132 reap_dead start 2023-03-03T17:45:40.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.366+0000 7f670759f700 1 -- 172.21.15.7:0/4257409132 shutdown_connections 2023-03-03T17:45:40.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.366+0000 7f670759f700 1 -- 172.21.15.7:0/4257409132 >> 172.21.15.7:0/4257409132 conn(0x7f67000a94d0 msgr2=0x7f67000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:45:40.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.366+0000 7f670759f700 1 -- 172.21.15.7:0/4257409132 shutdown_connections 2023-03-03T17:45:40.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.366+0000 7f670759f700 1 -- 172.21.15.7:0/4257409132 wait complete. 2023-03-03T17:45:40.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.367+0000 7f670759f700 1 Processor -- start 2023-03-03T17:45:40.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.367+0000 7f670759f700 1 -- start start 2023-03-03T17:45:40.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670759f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 0x7f67001e8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:40.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670759f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6700150f30 0x7f6700153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:40.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670759f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67001da730 0x7f6700153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:40.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670759f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6700153620 con 0x7f66ec003aa0 2023-03-03T17:45:40.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670759f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67001e9720 con 0x7f67001da730 2023-03-03T17:45:40.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670759f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67001474e0 con 0x7f6700150f30 2023-03-03T17:45:40.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670533b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 0x7f67001e8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:40.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670533b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 0x7f67001e8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34432/0 (socket says 172.21.15.7:34432) 2023-03-03T17:45:40.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670533b700 1 -- 172.21.15.7:0/3412948958 learned_addr learned my addr 172.21.15.7:0/3412948958 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:45:40.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f6704b3a700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6700150f30 0x7f6700153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:40.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f6705b3c700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67001da730 0x7f6700153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:40.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670533b700 1 -- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6700150f30 msgr2=0x7f6700153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:40.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670533b700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6700150f30 0x7f6700153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:40.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670533b700 1 -- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67001da730 msgr2=0x7f6700153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:40.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670533b700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67001da730 0x7f6700153f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:40.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.368+0000 7f670533b700 1 -- 172.21.15.7:0/3412948958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6700151f30 con 0x7f66ec003aa0 2023-03-03T17:45:40.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.369+0000 7f670533b700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 0x7f67001e8e00 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0011a0 tx=0x7f66fc001110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:45:40.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.369+0000 7f66f5ffb700 1 -- 172.21.15.7:0/3412948958 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc001370 con 0x7f66ec003aa0 2023-03-03T17:45:40.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.369+0000 7f66f5ffb700 1 -- 172.21.15.7:0/3412948958 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66fc00ca20 con 0x7f66ec003aa0 2023-03-03T17:45:40.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.369+0000 7f670759f700 1 -- 172.21.15.7:0/3412948958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6700151300 con 0x7f66ec003aa0 2023-03-03T17:45:40.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.369+0000 7f66f5ffb700 1 -- 172.21.15.7:0/3412948958 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc00cdb0 con 0x7f66ec003aa0 2023-03-03T17:45:40.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.369+0000 7f670759f700 1 -- 172.21.15.7:0/3412948958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670006c9a0 con 0x7f66ec003aa0 2023-03-03T17:45:40.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.370+0000 7f670759f700 1 -- 172.21.15.7:0/3412948958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6700147a00 con 0x7f66ec003aa0 2023-03-03T17:45:40.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.370+0000 7f66f5ffb700 1 -- 172.21.15.7:0/3412948958 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f66fc010050 con 0x7f66ec003aa0 2023-03-03T17:45:40.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.371+0000 7f66f5ffb700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66e40682f0 0x7f66e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:45:40.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.371+0000 7f66f5ffb700 1 -- 172.21.15.7:0/3412948958 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f66fc087740 con 0x7f66ec003aa0 2023-03-03T17:45:40.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.371+0000 7f6704b3a700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66e40682f0 0x7f66e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:45:40.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.372+0000 7f6704b3a700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66e40682f0 0x7f66e406a7a0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f67000c2c40 tx=0x7f66f0007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:45:40.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.376+0000 7f66f5ffb700 1 -- 172.21.15.7:0/3412948958 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f66fc053ae0 con 0x7f66ec003aa0 2023-03-03T17:45:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.617+0000 7f670759f700 1 -- 172.21.15.7:0/3412948958 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6700153760 con 0x7f66ec003aa0 2023-03-03T17:45:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.619+0000 7f66f5ffb700 1 -- 172.21.15.7:0/3412948958 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f66fc057100 con 0x7f66ec003aa0 2023-03-03T17:45:40.618 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:45:40.618 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:45:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 -- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66e40682f0 msgr2=0x7f66e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66e40682f0 0x7f66e406a7a0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f67000c2c40 tx=0x7f66f0007040 comp rx=0 tx=0).stop 2023-03-03T17:45:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 -- 172.21.15.7:0/3412948958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 msgr2=0x7f67001e8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:45:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 0x7f67001e8e00 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7f66fc0011a0 tx=0x7f66fc001110 comp rx=0 tx=0).stop 2023-03-03T17:45:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 -- 172.21.15.7:0/3412948958 shutdown_connections 2023-03-03T17:45:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67001da730 0x7f6700153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66e40682f0 0x7f66e406a7a0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f66ec003aa0 0x7f67001e8e00 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 --2- 172.21.15.7:0/3412948958 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6700150f30 0x7f6700153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:45:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 -- 172.21.15.7:0/3412948958 >> 172.21.15.7:0/3412948958 conn(0x7f67000a94d0 msgr2=0x7f670006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:45:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 -- 172.21.15.7:0/3412948958 shutdown_connections 2023-03-03T17:45:40.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:45:40.621+0000 7f670759f700 1 -- 172.21.15.7:0/3412948958 wait complete. 2023-03-03T17:45:40.622 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:45:40.632 INFO:tasks.mds_thrash.fs.[a]:waiting for 29 secs before thrashing 2023-03-03T17:46:00.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:00.734 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:05.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:46:06.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.040+0000 7fa9e3c3d700 1 Processor -- start 2023-03-03T17:46:06.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.040+0000 7fa9e3c3d700 1 -- start start 2023-03-03T17:46:06.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.041+0000 7fa9e3c3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9dc0ab0c0 0x7fa9dc0c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:06.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.041+0000 7fa9e3c3d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc154530 con 0x7fa9dc06a2b0 2023-03-03T17:46:06.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.041+0000 7fa9e3c3d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc154810 con 0x7fa9dc06a720 2023-03-03T17:46:06.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.041+0000 7fa9e3c3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc154af0 con 0x7fa9dc0ab0c0 2023-03-03T17:46:06.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.041+0000 7fa9e11d8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa9dc06a2b0 0x7fa9dc157a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56666/0 (socket says 172.21.15.7:56666) 2023-03-03T17:46:06.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.041+0000 7fa9e11d8700 1 -- 172.21.15.7:0/3930362487 learned_addr learned my addr 172.21.15.7:0/3930362487 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:46:06.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215932270 0 0) 0x7fa9dc154530 con 0x7fa9dc06a2b0 2023-03-03T17:46:06.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c80035b0 con 0x7fa9dc06a2b0 2023-03-03T17:46:06.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e19d9700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9dc0ab0c0 0x7fa9dc0c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:06.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 848679995 0 0) 0x7fa9c80035b0 con 0x7fa9dc06a2b0 2023-03-03T17:46:06.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9dc0ab0c0 msgr2=0x7fa9dc0c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:46:06.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9dc0ab0c0 0x7fa9dc0c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:06.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 >> v1:172.21.15.143:6789/0 conn(0x7fa9dc06a720 legacy=0x7fa9dc158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:06.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc154dd0 con 0x7fa9dc06a2b0 2023-03-03T17:46:06.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa9d0001670 con 0x7fa9dc06a2b0 2023-03-03T17:46:06.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 >> v1:172.21.15.7:6789/0 conn(0x7fa9dc06a2b0 legacy=0x7fa9dc157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:06.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c8003aa0 0x7fa9c8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:06.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9c8004850 0x7fa9c8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:06.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9dc0ab0c0 0x7fa9c8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:06.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc154af0 con 0x7fa9c8004850 2023-03-03T17:46:06.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c800baf0 con 0x7fa9c8003aa0 2023-03-03T17:46:06.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c800bcb0 con 0x7fa9dc0ab0c0 2023-03-03T17:46:06.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e11d8700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9c8004850 0x7fa9c8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:06.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e19d9700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c8003aa0 0x7fa9c8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:06.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.042+0000 7fa9e21da700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9dc0ab0c0 0x7fa9c8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:06.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e11d8700 1 -- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9dc0ab0c0 msgr2=0x7fa9c8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:46:06.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e11d8700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9dc0ab0c0 0x7fa9c8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:06.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e11d8700 1 -- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c8003aa0 msgr2=0x7fa9c8003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:46:06.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e11d8700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9c8003aa0 0x7fa9c8003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:06.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e11d8700 1 -- 172.21.15.7:0/3930362487 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c80035b0 con 0x7fa9c8004850 2023-03-03T17:46:06.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e11d8700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9c8004850 0x7fa9c8004cc0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0014a50 tx=0x7fa9c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35738efd95cbf0f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:46:06.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0007c10 con 0x7fa9c8004850 2023-03-03T17:46:06.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d0005b80 con 0x7fa9c8004850 2023-03-03T17:46:06.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e09d7700 1 -- 172.21.15.7:0/3930362487 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0014e80 con 0x7fa9c8004850 2023-03-03T17:46:06.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/3930362487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9c8004850 msgr2=0x7fa9c8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:06.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e3c3d700 1 --2- 172.21.15.7:0/3930362487 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9c8004850 0x7fa9c8004cc0 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0014a50 tx=0x7fa9c8005880 comp rx=0 tx=0).stop 2023-03-03T17:46:06.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e21da700 1 -- 172.21.15.7:0/3930362487 reap_dead start 2023-03-03T17:46:06.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/3930362487 shutdown_connections 2023-03-03T17:46:06.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.043+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/3930362487 >> 172.21.15.7:0/3930362487 conn(0x7fa9dc0a9430 msgr2=0x7fa9dc0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:46:06.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.044+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/3930362487 shutdown_connections 2023-03-03T17:46:06.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.044+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/3930362487 wait complete. 2023-03-03T17:46:06.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.045+0000 7fa9e3c3d700 1 Processor -- start 2023-03-03T17:46:06.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.045+0000 7fa9e3c3d700 1 -- start start 2023-03-03T17:46:06.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.045+0000 7fa9e3c3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9c8004850 0x7fa9dc14c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:06.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.045+0000 7fa9e3c3d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9dc06c720 0x7fa9dc149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:06.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.045+0000 7fa9e3c3d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9dc14db00 0x7fa9dc14a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:06.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e3c3d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc071b00 con 0x7fa9dc14db00 2023-03-03T17:46:06.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e3c3d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc06d1e0 con 0x7fa9dc06c720 2023-03-03T17:46:06.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e3c3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1e9800 con 0x7fa9c8004850 2023-03-03T17:46:06.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e21da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9dc14db00 0x7fa9dc14a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:06.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e21da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9dc14db00 0x7fa9dc14a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34470/0 (socket says 172.21.15.7:34470) 2023-03-03T17:46:06.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e21da700 1 -- 172.21.15.7:0/45630918 learned_addr learned my addr 172.21.15.7:0/45630918 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:46:06.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e19d9700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9c8004850 0x7fa9dc14c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:06.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e11d8700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9dc06c720 0x7fa9dc149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:06.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e21da700 1 -- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9c8004850 msgr2=0x7fa9dc14c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:06.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e21da700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9c8004850 0x7fa9dc14c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:06.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e21da700 1 -- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9dc06c720 msgr2=0x7fa9dc149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:06.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e21da700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9dc06c720 0x7fa9dc149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:06.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e21da700 1 -- 172.21.15.7:0/45630918 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc0ab490 con 0x7fa9dc14db00 2023-03-03T17:46:06.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e21da700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9dc14db00 0x7fa9dc14a200 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0c3560 tx=0x7fa9d400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:46:06.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9ce7fc700 1 -- 172.21.15.7:0/45630918 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4003c10 con 0x7fa9dc14db00 2023-03-03T17:46:06.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9ce7fc700 1 -- 172.21.15.7:0/45630918 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d4003d90 con 0x7fa9dc14db00 2023-03-03T17:46:06.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/45630918 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc1550b0 con 0x7fa9dc14db00 2023-03-03T17:46:06.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9ce7fc700 1 -- 172.21.15.7:0/45630918 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d400dbd0 con 0x7fa9dc14db00 2023-03-03T17:46:06.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.046+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/45630918 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc0c31c0 con 0x7fa9dc14db00 2023-03-03T17:46:06.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.047+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/45630918 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc074bb0 con 0x7fa9dc14db00 2023-03-03T17:46:06.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.048+0000 7fa9ce7fc700 1 -- 172.21.15.7:0/45630918 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa9d400dd30 con 0x7fa9dc14db00 2023-03-03T17:46:06.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.048+0000 7fa9ce7fc700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9c00682f0 0x7fa9c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:06.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.048+0000 7fa9ce7fc700 1 -- 172.21.15.7:0/45630918 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa9d4086660 con 0x7fa9dc14db00 2023-03-03T17:46:06.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.048+0000 7fa9e19d9700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9c00682f0 0x7fa9c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:06.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.049+0000 7fa9e19d9700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9c00682f0 0x7fa9c006a7a0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0a8050 tx=0x7fa9d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:46:06.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.053+0000 7fa9ce7fc700 1 -- 172.21.15.7:0/45630918 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa9d4052910 con 0x7fa9dc14db00 2023-03-03T17:46:06.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.294+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/45630918 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9dc066eb0 con 0x7fa9dc14db00 2023-03-03T17:46:06.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.295+0000 7fa9ce7fc700 1 -- 172.21.15.7:0/45630918 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa9d4055f30 con 0x7fa9dc14db00 2023-03-03T17:46:06.295 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:46:06.296 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:46:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9c00682f0 msgr2=0x7fa9c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9c00682f0 0x7fa9c006a7a0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0a8050 tx=0x7fa9d8008040 comp rx=0 tx=0).stop 2023-03-03T17:46:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/45630918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9dc14db00 msgr2=0x7fa9dc14a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:06.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9dc14db00 0x7fa9dc14a200 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc0c3560 tx=0x7fa9d400d430 comp rx=0 tx=0).stop 2023-03-03T17:46:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/45630918 shutdown_connections 2023-03-03T17:46:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa9c00682f0 0x7fa9c006a7a0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa9dc14db00 0x7fa9dc14a200 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa9c8004850 0x7fa9dc14c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 --2- 172.21.15.7:0/45630918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9dc06c720 0x7fa9dc149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/45630918 >> 172.21.15.7:0/45630918 conn(0x7fa9dc0a9430 msgr2=0x7fa9dc1539f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:46:06.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/45630918 shutdown_connections 2023-03-03T17:46:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:06.297+0000 7fa9e3c3d700 1 -- 172.21.15.7:0/45630918 wait complete. 2023-03-03T17:46:06.299 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:46:06.309 INFO:tasks.mds_thrash.fs.[b]:waiting for 117 secs before thrashing 2023-03-03T17:46:09.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:46:09.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.772+0000 7f2cecb7b700 1 Processor -- start 2023-03-03T17:46:09.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.772+0000 7f2cecb7b700 1 -- start start 2023-03-03T17:46:09.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2cecb7b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce814f9d0 0x7f2ce8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:09.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2cecb7b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2ce80abe20 0x7f2ce81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:09.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2cecb7b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce80c2f30 con 0x7f2ce814fda0 2023-03-03T17:46:09.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2cecb7b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce80c31d0 con 0x7f2ce80abe20 2023-03-03T17:46:09.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2cecb7b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce81524a0 con 0x7f2ce814f9d0 2023-03-03T17:46:09.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce5d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2ce814fda0 0x7f2ce8071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56688/0 (socket says 172.21.15.7:56688) 2023-03-03T17:46:09.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce5d9b700 1 -- 172.21.15.7:0/4055034301 learned_addr learned my addr 172.21.15.7:0/4055034301 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:46:09.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce6d9d700 1 --2- 172.21.15.7:0/4055034301 >> v2:172.21.15.143:3300/0 conn(0x7f2ce80abe20 0x7f2ce81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:09.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce659c700 1 --2- 172.21.15.7:0/4055034301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce814f9d0 0x7f2ce8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:09.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce559a700 1 -- 172.21.15.7:0/4055034301 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194553034 0 0) 0x7f2ce80c2f30 con 0x7f2ce814fda0 2023-03-03T17:46:09.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce559a700 1 -- 172.21.15.7:0/4055034301 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd00035b0 con 0x7f2ce814fda0 2023-03-03T17:46:09.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce6d9d700 1 -- 172.21.15.7:0/4055034301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce814f9d0 msgr2=0x7f2ce8156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:46:09.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce6d9d700 1 --2- 172.21.15.7:0/4055034301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce814f9d0 0x7f2ce8156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:09.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce6d9d700 1 -- 172.21.15.7:0/4055034301 >> v1:172.21.15.7:6789/0 conn(0x7f2ce814fda0 legacy=0x7f2ce8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:09.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce6d9d700 1 -- 172.21.15.7:0/4055034301 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce80c34b0 con 0x7f2ce80abe20 2023-03-03T17:46:09.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.773+0000 7f2ce659c700 1 --2- 172.21.15.7:0/4055034301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce814f9d0 0x7f2ce8156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:46:09.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.774+0000 7f2ce6d9d700 1 --2- 172.21.15.7:0/4055034301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80abe20 0x7f2ce81529c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009d90 tx=0x7f2ce000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa0d71ad89116046 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:46:09.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.774+0000 7f2ce559a700 1 -- 172.21.15.7:0/4055034301 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce000e070 con 0x7f2ce80abe20 2023-03-03T17:46:09.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.774+0000 7f2ce559a700 1 -- 172.21.15.7:0/4055034301 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce000bb20 con 0x7f2ce80abe20 2023-03-03T17:46:09.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.774+0000 7f2ce559a700 1 -- 172.21.15.7:0/4055034301 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0012430 con 0x7f2ce80abe20 2023-03-03T17:46:09.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.774+0000 7f2cecb7b700 1 -- 172.21.15.7:0/4055034301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80abe20 msgr2=0x7f2ce81529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:09.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.774+0000 7f2cecb7b700 1 --2- 172.21.15.7:0/4055034301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80abe20 0x7f2ce81529c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009d90 tx=0x7f2ce000b330 comp rx=0 tx=0).stop 2023-03-03T17:46:09.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.774+0000 7f2cecb7b700 1 -- 172.21.15.7:0/4055034301 shutdown_connections 2023-03-03T17:46:09.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.774+0000 7f2cecb7b700 1 --2- 172.21.15.7:0/4055034301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce80abe20 0x7f2ce81529c0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:09.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.774+0000 7f2cecb7b700 1 --2- 172.21.15.7:0/4055034301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce814f9d0 0x7f2ce8156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:09.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.774+0000 7f2cecb7b700 1 -- 172.21.15.7:0/4055034301 >> 172.21.15.7:0/4055034301 conn(0x7f2ce80c8900 msgr2=0x7f2ce80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:46:09.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.775+0000 7f2cecb7b700 1 -- 172.21.15.7:0/4055034301 shutdown_connections 2023-03-03T17:46:09.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.775+0000 7f2cecb7b700 1 -- 172.21.15.7:0/4055034301 wait complete. 2023-03-03T17:46:09.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.776+0000 7f2cecb7b700 1 Processor -- start 2023-03-03T17:46:09.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.776+0000 7f2cecb7b700 1 -- start start 2023-03-03T17:46:09.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2cecb7b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce80abe20 0x7f2ce81549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:09.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2cecb7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce814f9d0 0x7f2ce814ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:09.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2cecb7b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce814fda0 0x7f2ce814b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:09.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2cecb7b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80be850 con 0x7f2ce814f9d0 2023-03-03T17:46:09.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2cecb7b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce814b8c0 con 0x7f2ce814fda0 2023-03-03T17:46:09.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2cecb7b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80bd3d0 con 0x7f2ce80abe20 2023-03-03T17:46:09.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce814f9d0 0x7f2ce814ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:09.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce814f9d0 0x7f2ce814ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34484/0 (socket says 172.21.15.7:34484) 2023-03-03T17:46:09.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce5d9b700 1 -- 172.21.15.7:0/3213913419 learned_addr learned my addr 172.21.15.7:0/3213913419 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:46:09.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce659c700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce80abe20 0x7f2ce81549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:09.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce6d9d700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce814fda0 0x7f2ce814b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:09.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce5d9b700 1 -- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce80abe20 msgr2=0x7f2ce81549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:09.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce5d9b700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce80abe20 0x7f2ce81549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:09.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce5d9b700 1 -- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce814fda0 msgr2=0x7f2ce814b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:09.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce5d9b700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce814fda0 0x7f2ce814b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:09.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce5d9b700 1 -- 172.21.15.7:0/3213913419 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd80094a0 con 0x7f2ce814f9d0 2023-03-03T17:46:09.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2ce5d9b700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce814f9d0 0x7f2ce814ae40 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc005910 tx=0x7f2cdc00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:46:09.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.777+0000 7f2cd6ffd700 1 -- 172.21.15.7:0/3213913419 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc005d70 con 0x7f2ce814f9d0 2023-03-03T17:46:09.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.778+0000 7f2cd6ffd700 1 -- 172.21.15.7:0/3213913419 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc0070d0 con 0x7f2ce814f9d0 2023-03-03T17:46:09.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.778+0000 7f2cecb7b700 1 -- 172.21.15.7:0/3213913419 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce0009a70 con 0x7f2ce814f9d0 2023-03-03T17:46:09.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.778+0000 7f2cd6ffd700 1 -- 172.21.15.7:0/3213913419 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc007460 con 0x7f2ce814f9d0 2023-03-03T17:46:09.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.778+0000 7f2cecb7b700 1 -- 172.21.15.7:0/3213913419 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce80ac350 con 0x7f2ce814f9d0 2023-03-03T17:46:09.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.779+0000 7f2cecb7b700 1 -- 172.21.15.7:0/3213913419 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce80ab630 con 0x7f2ce814f9d0 2023-03-03T17:46:09.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.779+0000 7f2cd6ffd700 1 -- 172.21.15.7:0/3213913419 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2cdc019030 con 0x7f2ce814f9d0 2023-03-03T17:46:09.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.779+0000 7f2cd6ffd700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc80682f0 0x7f2cc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:09.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.779+0000 7f2cd6ffd700 1 -- 172.21.15.7:0/3213913419 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2cdc085450 con 0x7f2ce814f9d0 2023-03-03T17:46:09.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.780+0000 7f2ce659c700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc80682f0 0x7f2cc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:09.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.780+0000 7f2ce659c700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc80682f0 0x7f2cc806a7a0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f2ce81ebb50 tx=0x7f2cd8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:46:09.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:09.784+0000 7f2cd6ffd700 1 -- 172.21.15.7:0/3213913419 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2cdc0517f0 con 0x7f2ce814f9d0 2023-03-03T17:46:10.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.023+0000 7f2cecb7b700 1 -- 172.21.15.7:0/3213913419 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ce8071fa0 con 0x7f2ce814f9d0 2023-03-03T17:46:10.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.024+0000 7f2cd6ffd700 1 -- 172.21.15.7:0/3213913419 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2cdc054e10 con 0x7f2ce814f9d0 2023-03-03T17:46:10.024 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:46:10.024 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:46:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 -- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc80682f0 msgr2=0x7f2cc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc80682f0 0x7f2cc806a7a0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f2ce81ebb50 tx=0x7f2cd8009210 comp rx=0 tx=0).stop 2023-03-03T17:46:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 -- 172.21.15.7:0/3213913419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce814f9d0 msgr2=0x7f2ce814ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce814f9d0 0x7f2ce814ae40 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc005910 tx=0x7f2cdc00c9e0 comp rx=0 tx=0).stop 2023-03-03T17:46:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 -- 172.21.15.7:0/3213913419 shutdown_connections 2023-03-03T17:46:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2ce814fda0 0x7f2ce814b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2ce80abe20 0x7f2ce81549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2cc80682f0 0x7f2cc806a7a0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 --2- 172.21.15.7:0/3213913419 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ce814f9d0 0x7f2ce814ae40 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 -- 172.21.15.7:0/3213913419 >> 172.21.15.7:0/3213913419 conn(0x7f2ce80c8900 msgr2=0x7f2ce814efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:46:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 -- 172.21.15.7:0/3213913419 shutdown_connections 2023-03-03T17:46:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:10.026+0000 7f2cecb7b700 1 -- 172.21.15.7:0/3213913419 wait complete. 2023-03-03T17:46:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:46:10.038 INFO:tasks.mds_thrash.fs.[a]:waiting for 41 secs before thrashing 2023-03-03T17:46:30.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:30.881 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:46:51.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:46:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.178+0000 7f599c80a700 1 Processor -- start 2023-03-03T17:46:51.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.178+0000 7f599c80a700 1 -- start start 2023-03-03T17:46:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.179+0000 7f599c80a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f599814f9d0 0x7f5998156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.179+0000 7f599c80a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f599814fda0 0x7f5998071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.179+0000 7f599c80a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59980c2f30 con 0x7f59980abe20 2023-03-03T17:46:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.179+0000 7f599c80a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59980c31d0 con 0x7f599814f9d0 2023-03-03T17:46:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.179+0000 7f599c80a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59981524a0 con 0x7f599814fda0 2023-03-03T17:46:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.179+0000 7f5996d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f59980abe20 0x7f59981529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56726/0 (socket says 172.21.15.7:56726) 2023-03-03T17:46:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.179+0000 7f5996d9d700 1 -- 172.21.15.7:0/4157750701 learned_addr learned my addr 172.21.15.7:0/4157750701 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:46:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045165049 0 0) 0x7f59980c2f30 con 0x7f59980abe20 2023-03-03T17:46:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59800035b0 con 0x7f59980abe20 2023-03-03T17:46:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f5995d9b700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f599814fda0 0x7f5998071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599659c700 1 --2- 172.21.15.7:0/4157750701 >> v2:172.21.15.143:3300/0 conn(0x7f599814f9d0 0x7f5998156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3254826352 0 0) 0x7f59800035b0 con 0x7f59980abe20 2023-03-03T17:46:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f599814fda0 msgr2=0x7f5998071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:46:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f599814fda0 0x7f5998071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 >> v2:172.21.15.143:3300/0 conn(0x7f599814f9d0 msgr2=0x7f5998156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:46:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 --2- 172.21.15.7:0/4157750701 >> v2:172.21.15.143:3300/0 conn(0x7f599814f9d0 0x7f5998156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59980c34b0 con 0x7f59980abe20 2023-03-03T17:46:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f59900019f0 con 0x7f59980abe20 2023-03-03T17:46:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 >> v1:172.21.15.7:6789/0 conn(0x7f59980abe20 legacy=0x7f59981529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5980003a70 0x7f5980003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5980004850 0x7f5980004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 0x7f598000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59980c31d0 con 0x7f5980005250 2023-03-03T17:46:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59981524a0 con 0x7f5980003a70 2023-03-03T17:46:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598000c1c0 con 0x7f5980004850 2023-03-03T17:46:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599659c700 1 --2- 172.21.15.7:0/4157750701 >> v2:172.21.15.143:3300/0 conn(0x7f599814f9d0 0x7f5998156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:46:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f5995d9b700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f599814fda0 0x7f5998071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:46:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f5996d9d700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 0x7f598000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f599659c700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5980003a70 0x7f5980003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f5995d9b700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5980004850 0x7f5980004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f5996d9d700 1 -- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5980004850 msgr2=0x7f5980004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:46:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f5996d9d700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5980004850 0x7f5980004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f5996d9d700 1 -- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5980003a70 msgr2=0x7f5980003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:46:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f5996d9d700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5980003a70 0x7f5980003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f5996d9d700 1 -- 172.21.15.7:0/4157750701 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59800035b0 con 0x7f5980005250 2023-03-03T17:46:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.180+0000 7f5995d9b700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5980004850 0x7f5980004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:46:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f5996d9d700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 0x7f598000ba10 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7f599000a860 tx=0x7f5980005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83754e2586e819ba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:46:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990012600 con 0x7f5980005250 2023-03-03T17:46:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5990012780 con 0x7f5980005250 2023-03-03T17:46:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f599559a700 1 -- 172.21.15.7:0/4157750701 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990012b30 con 0x7f5980005250 2023-03-03T17:46:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f599c80a700 1 -- 172.21.15.7:0/4157750701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 msgr2=0x7f598000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f599c80a700 1 --2- 172.21.15.7:0/4157750701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 0x7f598000ba10 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7f599000a860 tx=0x7f5980005880 comp rx=0 tx=0).stop 2023-03-03T17:46:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f5996d9d700 1 -- 172.21.15.7:0/4157750701 reap_dead start 2023-03-03T17:46:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f599c80a700 1 -- 172.21.15.7:0/4157750701 shutdown_connections 2023-03-03T17:46:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f599c80a700 1 -- 172.21.15.7:0/4157750701 >> 172.21.15.7:0/4157750701 conn(0x7f59980c8900 msgr2=0x7f59980c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:46:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f599c80a700 1 -- 172.21.15.7:0/4157750701 shutdown_connections 2023-03-03T17:46:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.181+0000 7f599c80a700 1 -- 172.21.15.7:0/4157750701 wait complete. 2023-03-03T17:46:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.183+0000 7f599c80a700 1 Processor -- start 2023-03-03T17:46:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.183+0000 7f599c80a700 1 -- start start 2023-03-03T17:46:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.184+0000 7f599c80a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 0x7f59981441b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.184+0000 7f599c80a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59980c2960 0x7f59981ea910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.184+0000 7f599c80a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59981446f0 0x7f59981eae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.184+0000 7f599c80a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59980c16c0 con 0x7f5980005250 2023-03-03T17:46:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.184+0000 7f599c80a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59980c1ae0 con 0x7f59981446f0 2023-03-03T17:46:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.184+0000 7f599c80a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59981eb390 con 0x7f59980c2960 2023-03-03T17:46:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.184+0000 7f599659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 0x7f59981441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.184+0000 7f599659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 0x7f59981441b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34524/0 (socket says 172.21.15.7:34524) 2023-03-03T17:46:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.184+0000 7f599659c700 1 -- 172.21.15.7:0/1554932177 learned_addr learned my addr 172.21.15.7:0/1554932177 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:46:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f5995d9b700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59980c2960 0x7f59981ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f5996d9d700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59981446f0 0x7f59981eae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f599659c700 1 -- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59980c2960 msgr2=0x7f59981ea910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f599659c700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59980c2960 0x7f59981ea910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f599659c700 1 -- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59981446f0 msgr2=0x7f59981eae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f599659c700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59981446f0 0x7f59981eae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f599659c700 1 -- 172.21.15.7:0/1554932177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59980ac350 con 0x7f5980005250 2023-03-03T17:46:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f5996d9d700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59981446f0 0x7f59981eae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:46:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f599659c700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 0x7f59981441b0 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f5988010840 tx=0x7f5988010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:46:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f5986ffd700 1 -- 172.21.15.7:0/1554932177 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988015880 con 0x7f5980005250 2023-03-03T17:46:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f5986ffd700 1 -- 172.21.15.7:0/1554932177 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598800c050 con 0x7f5980005250 2023-03-03T17:46:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f599c80a700 1 -- 172.21.15.7:0/1554932177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59980ab630 con 0x7f5980005250 2023-03-03T17:46:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f5986ffd700 1 -- 172.21.15.7:0/1554932177 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988015c20 con 0x7f5980005250 2023-03-03T17:46:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.185+0000 7f599c80a700 1 -- 172.21.15.7:0/1554932177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59980c3e10 con 0x7f5980005250 2023-03-03T17:46:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.186+0000 7f599c80a700 1 -- 172.21.15.7:0/1554932177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5998074bb0 con 0x7f5980005250 2023-03-03T17:46:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.186+0000 7f5986ffd700 1 -- 172.21.15.7:0/1554932177 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5988026070 con 0x7f5980005250 2023-03-03T17:46:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.187+0000 7f5986ffd700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59780682f0 0x7f597806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:46:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.187+0000 7f5986ffd700 1 -- 172.21.15.7:0/1554932177 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5988086990 con 0x7f5980005250 2023-03-03T17:46:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.187+0000 7f5995d9b700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59780682f0 0x7f597806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:46:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.188+0000 7f5995d9b700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59780682f0 0x7f597806a7a0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f599806fdd0 tx=0x7f598c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:46:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.191+0000 7f5986ffd700 1 -- 172.21.15.7:0/1554932177 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5988052d30 con 0x7f5980005250 2023-03-03T17:46:51.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.428+0000 7f599c80a700 1 -- 172.21.15.7:0/1554932177 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59980c1930 con 0x7f5980005250 2023-03-03T17:46:51.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.428+0000 7f5986ffd700 1 -- 172.21.15.7:0/1554932177 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5988027350 con 0x7f5980005250 2023-03-03T17:46:51.428 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:46:51.428 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:46:51.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 -- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59780682f0 msgr2=0x7f597806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59780682f0 0x7f597806a7a0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f599806fdd0 tx=0x7f598c008040 comp rx=0 tx=0).stop 2023-03-03T17:46:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 -- 172.21.15.7:0/1554932177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 msgr2=0x7f59981441b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:46:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 0x7f59981441b0 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f5988010840 tx=0x7f5988010c00 comp rx=0 tx=0).stop 2023-03-03T17:46:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 -- 172.21.15.7:0/1554932177 shutdown_connections 2023-03-03T17:46:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59981446f0 0x7f59981eae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59780682f0 0x7f597806a7a0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:51.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5980005250 0x7f59981441b0 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:51.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 --2- 172.21.15.7:0/1554932177 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59980c2960 0x7f59981ea910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:46:51.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 -- 172.21.15.7:0/1554932177 >> 172.21.15.7:0/1554932177 conn(0x7f59980c8900 msgr2=0x7f59980c8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:46:51.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 -- 172.21.15.7:0/1554932177 shutdown_connections 2023-03-03T17:46:51.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:46:51.430+0000 7f599c80a700 1 -- 172.21.15.7:0/1554932177 wait complete. 2023-03-03T17:46:51.431 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:46:51.441 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2023-03-03T17:47:00.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:00.994 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:18.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:47:18.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.581+0000 7f4523499700 1 Processor -- start 2023-03-03T17:47:18.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.581+0000 7f4523499700 1 -- start start 2023-03-03T17:47:18.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4523499700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f451c14f9d0 0x7f451c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:18.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4523499700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c14fda0 0x7f451c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:18.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4523499700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f451c0abe20 0x7f451c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:18.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4523499700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f451c0c2f30 con 0x7f451c0abe20 2023-03-03T17:47:18.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4523499700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f451c0c31d0 con 0x7f451c14f9d0 2023-03-03T17:47:18.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4523499700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451c1524a0 con 0x7f451c14fda0 2023-03-03T17:47:18.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4521a36700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f451c0abe20 0x7f451c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:18.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4521a36700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f451c0abe20 0x7f451c1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34556/0 (socket says 172.21.15.7:34556) 2023-03-03T17:47:18.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4521a36700 1 -- 172.21.15.7:0/3383431845 learned_addr learned my addr 172.21.15.7:0/3383431845 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:47:18.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4520a34700 1 --2- 172.21.15.7:0/3383431845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c14fda0 0x7f451c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:18.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.582+0000 7f4521235700 1 --2- 172.21.15.7:0/3383431845 >> v2:172.21.15.143:3300/0 conn(0x7f451c14f9d0 0x7f451c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:18.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4521a36700 1 -- 172.21.15.7:0/3383431845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c14fda0 msgr2=0x7f451c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:47:18.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4521a36700 1 --2- 172.21.15.7:0/3383431845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c14fda0 0x7f451c071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4521a36700 1 -- 172.21.15.7:0/3383431845 >> v2:172.21.15.143:3300/0 conn(0x7f451c14f9d0 msgr2=0x7f451c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:47:18.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4521a36700 1 --2- 172.21.15.7:0/3383431845 >> v2:172.21.15.143:3300/0 conn(0x7f451c14f9d0 0x7f451c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4521a36700 1 -- 172.21.15.7:0/3383431845 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c0c34b0 con 0x7f451c0abe20 2023-03-03T17:47:18.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4520a34700 1 --2- 172.21.15.7:0/3383431845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c14fda0 0x7f451c071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:47:18.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4521235700 1 --2- 172.21.15.7:0/3383431845 >> v2:172.21.15.143:3300/0 conn(0x7f451c14f9d0 0x7f451c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:47:18.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4521a36700 1 --2- 172.21.15.7:0/3383431845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c0abe20 0x7f451c1529c0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f451c0710a0 tx=0x7f451400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169b01bf5ab3d8b2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:47:18.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4513fff700 1 -- 172.21.15.7:0/3383431845 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451400e070 con 0x7f451c0abe20 2023-03-03T17:47:18.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4513fff700 1 -- 172.21.15.7:0/3383431845 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451400b9c0 con 0x7f451c0abe20 2023-03-03T17:47:18.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.583+0000 7f4513fff700 1 -- 172.21.15.7:0/3383431845 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451400bd70 con 0x7f451c0abe20 2023-03-03T17:47:18.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.584+0000 7f4523499700 1 -- 172.21.15.7:0/3383431845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c0abe20 msgr2=0x7f451c1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:18.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.584+0000 7f4523499700 1 --2- 172.21.15.7:0/3383431845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c0abe20 0x7f451c1529c0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f451c0710a0 tx=0x7f451400b330 comp rx=0 tx=0).stop 2023-03-03T17:47:18.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.584+0000 7f4523499700 1 -- 172.21.15.7:0/3383431845 shutdown_connections 2023-03-03T17:47:18.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.584+0000 7f4523499700 1 --2- 172.21.15.7:0/3383431845 >> v2:172.21.15.143:3300/0 conn(0x7f451c14f9d0 0x7f451c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.584+0000 7f4523499700 1 --2- 172.21.15.7:0/3383431845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c0abe20 0x7f451c1529c0 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.584+0000 7f4523499700 1 --2- 172.21.15.7:0/3383431845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c14fda0 0x7f451c071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.584+0000 7f4523499700 1 -- 172.21.15.7:0/3383431845 >> 172.21.15.7:0/3383431845 conn(0x7f451c0c8900 msgr2=0x7f451c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:47:18.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.584+0000 7f4523499700 1 -- 172.21.15.7:0/3383431845 shutdown_connections 2023-03-03T17:47:18.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.584+0000 7f4523499700 1 -- 172.21.15.7:0/3383431845 wait complete. 2023-03-03T17:47:18.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.585+0000 7f4523499700 1 Processor -- start 2023-03-03T17:47:18.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.585+0000 7f4523499700 1 -- start start 2023-03-03T17:47:18.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.586+0000 7f4523499700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c0abe20 0x7f451c06df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:18.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.586+0000 7f4523499700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c14f9d0 0x7f451c154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:18.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.586+0000 7f4523499700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f451c14fda0 0x7f451c1e7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:18.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.586+0000 7f4523499700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c151d50 con 0x7f451c14f9d0 2023-03-03T17:47:18.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.586+0000 7f4523499700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c1eb370 con 0x7f451c14fda0 2023-03-03T17:47:18.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.586+0000 7f4523499700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451c1e7880 con 0x7f451c0abe20 2023-03-03T17:47:18.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.586+0000 7f4520a34700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c14f9d0 0x7f451c154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:18.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.586+0000 7f4520a34700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c14f9d0 0x7f451c154d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34560/0 (socket says 172.21.15.7:34560) 2023-03-03T17:47:18.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.586+0000 7f4520a34700 1 -- 172.21.15.7:0/165191892 learned_addr learned my addr 172.21.15.7:0/165191892 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:47:18.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4521235700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c0abe20 0x7f451c06df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:18.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4521a36700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f451c14fda0 0x7f451c1e7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:18.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4520a34700 1 -- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c0abe20 msgr2=0x7f451c06df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:18.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4520a34700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c0abe20 0x7f451c06df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4520a34700 1 -- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f451c14fda0 msgr2=0x7f451c1e7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:18.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4520a34700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f451c14fda0 0x7f451c1e7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4520a34700 1 -- 172.21.15.7:0/165191892 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4514009b30 con 0x7f451c14f9d0 2023-03-03T17:47:18.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4521a36700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f451c14fda0 0x7f451c1e7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:47:18.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4521235700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c0abe20 0x7f451c06df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:47:18.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4520a34700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c14f9d0 0x7f451c154d10 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7f451800e7d0 tx=0x7f451800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:47:18.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4511ffb700 1 -- 172.21.15.7:0/165191892 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518013610 con 0x7f451c14f9d0 2023-03-03T17:47:18.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4511ffb700 1 -- 172.21.15.7:0/165191892 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4518013770 con 0x7f451c14f9d0 2023-03-03T17:47:18.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4511ffb700 1 -- 172.21.15.7:0/165191892 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518013b00 con 0x7f451c14f9d0 2023-03-03T17:47:18.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4523499700 1 -- 172.21.15.7:0/165191892 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451c0ac350 con 0x7f451c14f9d0 2023-03-03T17:47:18.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.587+0000 7f4523499700 1 -- 172.21.15.7:0/165191892 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451c0ab630 con 0x7f451c14f9d0 2023-03-03T17:47:18.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.588+0000 7f4523499700 1 -- 172.21.15.7:0/165191892 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451c0c3e10 con 0x7f451c14f9d0 2023-03-03T17:47:18.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.589+0000 7f4511ffb700 1 -- 172.21.15.7:0/165191892 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4518023070 con 0x7f451c14f9d0 2023-03-03T17:47:18.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.589+0000 7f4511ffb700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45000682f0 0x7f450006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:18.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.589+0000 7f4511ffb700 1 -- 172.21.15.7:0/165191892 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4518085cf0 con 0x7f451c14f9d0 2023-03-03T17:47:18.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.589+0000 7f4521235700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45000682f0 0x7f450006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:18.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.590+0000 7f4521235700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45000682f0 0x7f450006a7a0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f450c009a50 tx=0x7f450c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:47:18.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.594+0000 7f4511ffb700 1 -- 172.21.15.7:0/165191892 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4518052090 con 0x7f451c14f9d0 2023-03-03T17:47:18.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.829+0000 7f4523499700 1 -- 172.21.15.7:0/165191892 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f451c074bb0 con 0x7f451c14f9d0 2023-03-03T17:47:18.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.829+0000 7f4511ffb700 1 -- 172.21.15.7:0/165191892 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f45180556b0 con 0x7f451c14f9d0 2023-03-03T17:47:18.830 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:47:18.830 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:47:18.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.831+0000 7f4523499700 1 -- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45000682f0 msgr2=0x7f450006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:18.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.831+0000 7f4523499700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45000682f0 0x7f450006a7a0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f450c009a50 tx=0x7f450c008040 comp rx=0 tx=0).stop 2023-03-03T17:47:18.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.831+0000 7f4523499700 1 -- 172.21.15.7:0/165191892 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c14f9d0 msgr2=0x7f451c154d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:18.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.831+0000 7f4523499700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c14f9d0 0x7f451c154d10 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7f451800e7d0 tx=0x7f451800eb90 comp rx=0 tx=0).stop 2023-03-03T17:47:18.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.831+0000 7f4523499700 1 -- 172.21.15.7:0/165191892 shutdown_connections 2023-03-03T17:47:18.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.831+0000 7f4523499700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f451c14fda0 0x7f451c1e7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.831+0000 7f4523499700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f451c0abe20 0x7f451c06df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.831+0000 7f4523499700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45000682f0 0x7f450006a7a0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.831+0000 7f4523499700 1 --2- 172.21.15.7:0/165191892 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f451c14f9d0 0x7f451c154d10 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:18.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.831+0000 7f4523499700 1 -- 172.21.15.7:0/165191892 >> 172.21.15.7:0/165191892 conn(0x7f451c0c8900 msgr2=0x7f451c1528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:47:18.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.832+0000 7f4523499700 1 -- 172.21.15.7:0/165191892 shutdown_connections 2023-03-03T17:47:18.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:18.832+0000 7f4523499700 1 -- 172.21.15.7:0/165191892 wait complete. 2023-03-03T17:47:18.834 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:47:18.843 INFO:tasks.mds_thrash.fs.[a]:waiting for 33 secs before thrashing 2023-03-03T17:47:31.103 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:31.107 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:47:51.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:47:51.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.983+0000 7f04878f2700 1 Processor -- start 2023-03-03T17:47:51.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.983+0000 7f04878f2700 1 -- start start 2023-03-03T17:47:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.984+0000 7f04878f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048006c480 0x7f04801581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.984+0000 7f04878f2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048006cb70 con 0x7f0480151b20 2023-03-03T17:47:51.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.984+0000 7f04878f2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048006ce50 con 0x7f04800c2950 2023-03-03T17:47:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.984+0000 7f04878f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048006c2f0 con 0x7f048006c480 2023-03-03T17:47:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.984+0000 7f048568e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0480151b20 0x7f04800736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56798/0 (socket says 172.21.15.7:56798) 2023-03-03T17:47:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.984+0000 7f048568e700 1 -- 172.21.15.7:0/4026265533 learned_addr learned my addr 172.21.15.7:0/4026265533 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:47:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.984+0000 7f0485e8f700 1 --2- 172.21.15.7:0/4026265533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048006c480 0x7f04801581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0477fff700 1 -- 172.21.15.7:0/4026265533 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087763706 0 0) 0x7f048006cb70 con 0x7f0480151b20 2023-03-03T17:47:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0477fff700 1 -- 172.21.15.7:0/4026265533 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046c0035b0 con 0x7f0480151b20 2023-03-03T17:47:51.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0477fff700 1 -- 172.21.15.7:0/4026265533 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903380110 0 0) 0x7f048006ce50 con 0x7f04800c2950 2023-03-03T17:47:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0477fff700 1 -- 172.21.15.7:0/4026265533 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048006cb70 con 0x7f04800c2950 2023-03-03T17:47:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0477fff700 1 -- 172.21.15.7:0/4026265533 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1264838049 0 0) 0x7f046c0035b0 con 0x7f0480151b20 2023-03-03T17:47:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0485e8f700 1 -- 172.21.15.7:0/4026265533 >> v1:172.21.15.143:6789/0 conn(0x7f04800c2950 legacy=0x7f0480157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0485e8f700 1 -- 172.21.15.7:0/4026265533 >> v1:172.21.15.7:6789/0 conn(0x7f0480151b20 legacy=0x7f04800736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0485e8f700 1 -- 172.21.15.7:0/4026265533 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048006d130 con 0x7f048006c480 2023-03-03T17:47:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0485e8f700 1 --2- 172.21.15.7:0/4026265533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048006c480 0x7f04801581f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0478009dc0 tx=0x7f047800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e11f6d52296a7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:47:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0477fff700 1 -- 172.21.15.7:0/4026265533 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047800e070 con 0x7f048006c480 2023-03-03T17:47:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0477fff700 1 -- 172.21.15.7:0/4026265533 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0478003a20 con 0x7f048006c480 2023-03-03T17:47:51.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.985+0000 7f0477fff700 1 -- 172.21.15.7:0/4026265533 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c001670 con 0x7f048006c480 2023-03-03T17:47:51.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.986+0000 7f04878f2700 1 -- 172.21.15.7:0/4026265533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048006c480 msgr2=0x7f04801581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:51.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.986+0000 7f04878f2700 1 --2- 172.21.15.7:0/4026265533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048006c480 0x7f04801581f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0478009dc0 tx=0x7f047800b330 comp rx=0 tx=0).stop 2023-03-03T17:47:51.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.986+0000 7f04878f2700 1 -- 172.21.15.7:0/4026265533 shutdown_connections 2023-03-03T17:47:51.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.986+0000 7f04878f2700 1 --2- 172.21.15.7:0/4026265533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f048006c480 0x7f04801581f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:51.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.986+0000 7f04878f2700 1 -- 172.21.15.7:0/4026265533 >> 172.21.15.7:0/4026265533 conn(0x7f04800a94d0 msgr2=0x7f04800a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:47:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.986+0000 7f04878f2700 1 -- 172.21.15.7:0/4026265533 shutdown_connections 2023-03-03T17:47:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.986+0000 7f04878f2700 1 -- 172.21.15.7:0/4026265533 wait complete. 2023-03-03T17:47:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.987+0000 7f04878f2700 1 Processor -- start 2023-03-03T17:47:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.987+0000 7f04878f2700 1 -- start start 2023-03-03T17:47:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f04878f2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f048006c480 0x7f0480154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f04878f2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04800c2950 0x7f0480149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f04878f2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0480151b20 0x7f048014a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:51.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f04878f2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0480152ec0 con 0x7f0480151b20 2023-03-03T17:47:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f04878f2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0480152160 con 0x7f048006c480 2023-03-03T17:47:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f04878f2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0480153e00 con 0x7f04800c2950 2023-03-03T17:47:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0485e8f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0480151b20 0x7f048014a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0485e8f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0480151b20 0x7f048014a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34596/0 (socket says 172.21.15.7:34596) 2023-03-03T17:47:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0485e8f700 1 -- 172.21.15.7:0/4281145337 learned_addr learned my addr 172.21.15.7:0/4281145337 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:47:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f048568e700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f048006c480 0x7f0480154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0484e8d700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04800c2950 0x7f0480149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:51.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0485e8f700 1 -- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04800c2950 msgr2=0x7f0480149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:51.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0485e8f700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04800c2950 0x7f0480149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:51.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0485e8f700 1 -- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f048006c480 msgr2=0x7f0480154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:51.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0485e8f700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f048006c480 0x7f0480154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:51.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0485e8f700 1 -- 172.21.15.7:0/4281145337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0478009a70 con 0x7f0480151b20 2023-03-03T17:47:51.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0485e8f700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0480151b20 0x7f048014a020 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f0478015040 tx=0x7f0478004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:47:51.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.988+0000 7f0475ffb700 1 -- 172.21.15.7:0/4281145337 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478004470 con 0x7f0480151b20 2023-03-03T17:47:51.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.989+0000 7f0475ffb700 1 -- 172.21.15.7:0/4281145337 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04780045d0 con 0x7f0480151b20 2023-03-03T17:47:51.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.989+0000 7f04878f2700 1 -- 172.21.15.7:0/4281145337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0480158730 con 0x7f0480151b20 2023-03-03T17:47:51.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.989+0000 7f0475ffb700 1 -- 172.21.15.7:0/4281145337 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047801c460 con 0x7f0480151b20 2023-03-03T17:47:51.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.989+0000 7f04878f2700 1 -- 172.21.15.7:0/4281145337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048006d410 con 0x7f0480151b20 2023-03-03T17:47:51.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.990+0000 7f04878f2700 1 -- 172.21.15.7:0/4281145337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0480151ef0 con 0x7f0480151b20 2023-03-03T17:47:51.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.990+0000 7f0475ffb700 1 -- 172.21.15.7:0/4281145337 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0478018040 con 0x7f0480151b20 2023-03-03T17:47:51.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.991+0000 7f0475ffb700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04640682f0 0x7f046406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:47:51.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.991+0000 7f0475ffb700 1 -- 172.21.15.7:0/4281145337 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0478087410 con 0x7f0480151b20 2023-03-03T17:47:51.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.991+0000 7f048568e700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04640682f0 0x7f046406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:47:51.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.991+0000 7f048568e700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04640682f0 0x7f046406a7a0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f047c000f60 tx=0x7f047c0014b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:47:51.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:51.995+0000 7f0475ffb700 1 -- 172.21.15.7:0/4281145337 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f04780537b0 con 0x7f0480151b20 2023-03-03T17:47:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.230+0000 7f04878f2700 1 -- 172.21.15.7:0/4281145337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04800be200 con 0x7f0480151b20 2023-03-03T17:47:52.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.231+0000 7f0475ffb700 1 -- 172.21.15.7:0/4281145337 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0478056dd0 con 0x7f0480151b20 2023-03-03T17:47:52.230 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:47:52.230 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:47:52.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 -- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04640682f0 msgr2=0x7f046406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:52.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04640682f0 0x7f046406a7a0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f047c000f60 tx=0x7f047c0014b0 comp rx=0 tx=0).stop 2023-03-03T17:47:52.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 -- 172.21.15.7:0/4281145337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0480151b20 msgr2=0x7f048014a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:47:52.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0480151b20 0x7f048014a020 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f0478015040 tx=0x7f0478004ad0 comp rx=0 tx=0).stop 2023-03-03T17:47:52.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 -- 172.21.15.7:0/4281145337 shutdown_connections 2023-03-03T17:47:52.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04640682f0 0x7f046406a7a0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:52.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0480151b20 0x7f048014a020 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:52.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f048006c480 0x7f0480154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:52.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 --2- 172.21.15.7:0/4281145337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04800c2950 0x7f0480149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:47:52.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 -- 172.21.15.7:0/4281145337 >> 172.21.15.7:0/4281145337 conn(0x7f04800a94d0 msgr2=0x7f04800c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:47:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 -- 172.21.15.7:0/4281145337 shutdown_connections 2023-03-03T17:47:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:47:52.233+0000 7f04878f2700 1 -- 172.21.15.7:0/4281145337 wait complete. 2023-03-03T17:47:52.234 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:47:52.244 INFO:tasks.mds_thrash.fs.[a]:waiting for 110 secs before thrashing 2023-03-03T17:48:01.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:01.180 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:03.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:48:03.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.449+0000 7f5b29259700 1 Processor -- start 2023-03-03T17:48:03.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.449+0000 7f5b29259700 1 -- start start 2023-03-03T17:48:03.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b29259700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5b2414f9d0 0x7f5b24156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:03.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b29259700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5b2414fda0 0x7f5b24071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:03.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b29259700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240c2f30 con 0x7f5b240abe20 2023-03-03T17:48:03.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b29259700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240c31d0 con 0x7f5b2414f9d0 2023-03-03T17:48:03.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b29259700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b241524a0 con 0x7f5b2414fda0 2023-03-03T17:48:03.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b22d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5b2414f9d0 0x7f5b24156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:03.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b22d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5b2414f9d0 0x7f5b24156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34604/0 (socket says 172.21.15.7:34604) 2023-03-03T17:48:03.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b22d9d700 1 -- 172.21.15.7:0/79403438 learned_addr learned my addr 172.21.15.7:0/79403438 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:48:03.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b2259c700 1 --2- 172.21.15.7:0/79403438 >> v2:172.21.15.143:3300/0 conn(0x7f5b2414fda0 0x7f5b24071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:03.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b22d9d700 1 -- 172.21.15.7:0/79403438 >> v2:172.21.15.143:3300/0 conn(0x7f5b2414fda0 msgr2=0x7f5b24071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:48:03.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b22d9d700 1 --2- 172.21.15.7:0/79403438 >> v2:172.21.15.143:3300/0 conn(0x7f5b2414fda0 0x7f5b24071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:03.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b22d9d700 1 -- 172.21.15.7:0/79403438 >> v1:172.21.15.143:6789/0 conn(0x7f5b240abe20 legacy=0x7f5b241529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:48:03.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b22d9d700 1 -- 172.21.15.7:0/79403438 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240c34b0 con 0x7f5b2414f9d0 2023-03-03T17:48:03.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b2259c700 1 --2- 172.21.15.7:0/79403438 >> v2:172.21.15.143:3300/0 conn(0x7f5b2414fda0 0x7f5b24071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:48:03.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.450+0000 7f5b22d9d700 1 --2- 172.21.15.7:0/79403438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414f9d0 0x7f5b24156430 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f5b240709b0 tx=0x7f5b18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e40c101017c0008 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:48:03.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b21d9b700 1 -- 172.21.15.7:0/79403438 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18004230 con 0x7f5b2414f9d0 2023-03-03T17:48:03.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b21d9b700 1 -- 172.21.15.7:0/79403438 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18004390 con 0x7f5b2414f9d0 2023-03-03T17:48:03.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b21d9b700 1 -- 172.21.15.7:0/79403438 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180036a0 con 0x7f5b2414f9d0 2023-03-03T17:48:03.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b29259700 1 -- 172.21.15.7:0/79403438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414f9d0 msgr2=0x7f5b24156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:03.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b29259700 1 --2- 172.21.15.7:0/79403438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414f9d0 0x7f5b24156430 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 crypto rx=0x7f5b240709b0 tx=0x7f5b18009bd0 comp rx=0 tx=0).stop 2023-03-03T17:48:03.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b29259700 1 -- 172.21.15.7:0/79403438 shutdown_connections 2023-03-03T17:48:03.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b29259700 1 --2- 172.21.15.7:0/79403438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b2414f9d0 0x7f5b24156430 unknown :-1 s=CLOSED pgs=3023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:03.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b29259700 1 --2- 172.21.15.7:0/79403438 >> v2:172.21.15.143:3300/0 conn(0x7f5b2414fda0 0x7f5b24071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:03.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b29259700 1 -- 172.21.15.7:0/79403438 >> 172.21.15.7:0/79403438 conn(0x7f5b240c8900 msgr2=0x7f5b240c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:48:03.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b29259700 1 -- 172.21.15.7:0/79403438 shutdown_connections 2023-03-03T17:48:03.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.451+0000 7f5b29259700 1 -- 172.21.15.7:0/79403438 wait complete. 2023-03-03T17:48:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.453+0000 7f5b29259700 1 Processor -- start 2023-03-03T17:48:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.453+0000 7f5b29259700 1 -- start start 2023-03-03T17:48:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.453+0000 7f5b29259700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b240abe20 0x7f5b24152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.453+0000 7f5b29259700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b2414f9d0 0x7f5b2406db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.453+0000 7f5b29259700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2414fda0 0x7f5b2406e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.453+0000 7f5b29259700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2406eef0 con 0x7f5b240abe20 2023-03-03T17:48:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b29259700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2414bc30 con 0x7f5b2414fda0 2023-03-03T17:48:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b29259700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2414c490 con 0x7f5b2414f9d0 2023-03-03T17:48:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b22d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b240abe20 0x7f5b24152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b22d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b240abe20 0x7f5b24152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34610/0 (socket says 172.21.15.7:34610) 2023-03-03T17:48:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b22d9d700 1 -- 172.21.15.7:0/1235569191 learned_addr learned my addr 172.21.15.7:0/1235569191 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:48:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b2259c700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b2414f9d0 0x7f5b2406db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b2359e700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2414fda0 0x7f5b2406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b22d9d700 1 -- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b2414f9d0 msgr2=0x7f5b2406db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b22d9d700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b2414f9d0 0x7f5b2406db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b22d9d700 1 -- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2414fda0 msgr2=0x7f5b2406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b22d9d700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2414fda0 0x7f5b2406e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b22d9d700 1 -- 172.21.15.7:0/1235569191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b18009550 con 0x7f5b240abe20 2023-03-03T17:48:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b22d9d700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b240abe20 0x7f5b24152b30 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f5b24072520 tx=0x7f5b18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:48:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b0b7fe700 1 -- 172.21.15.7:0/1235569191 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18003e90 con 0x7f5b240abe20 2023-03-03T17:48:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b0b7fe700 1 -- 172.21.15.7:0/1235569191 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18011630 con 0x7f5b240abe20 2023-03-03T17:48:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b29259700 1 -- 172.21.15.7:0/1235569191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b240ac350 con 0x7f5b240abe20 2023-03-03T17:48:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b0b7fe700 1 -- 172.21.15.7:0/1235569191 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180119a0 con 0x7f5b240abe20 2023-03-03T17:48:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.454+0000 7f5b29259700 1 -- 172.21.15.7:0/1235569191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b240ab630 con 0x7f5b240abe20 2023-03-03T17:48:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.455+0000 7f5b29259700 1 -- 172.21.15.7:0/1235569191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b240c3e10 con 0x7f5b240abe20 2023-03-03T17:48:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.456+0000 7f5b0b7fe700 1 -- 172.21.15.7:0/1235569191 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5b18017020 con 0x7f5b240abe20 2023-03-03T17:48:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.456+0000 7f5b0b7fe700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b100682f0 0x7f5b1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:03.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.457+0000 7f5b0b7fe700 1 -- 172.21.15.7:0/1235569191 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5b180878c0 con 0x7f5b240abe20 2023-03-03T17:48:03.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.457+0000 7f5b2259c700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b100682f0 0x7f5b1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:03.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.457+0000 7f5b2259c700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b100682f0 0x7f5b1006a7a0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f5b241e5610 tx=0x7f5b14008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:48:03.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.461+0000 7f5b0b7fe700 1 -- 172.21.15.7:0/1235569191 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5b18053c60 con 0x7f5b240abe20 2023-03-03T17:48:03.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.697+0000 7f5b29259700 1 -- 172.21.15.7:0/1235569191 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b240abbc0 con 0x7f5b240abe20 2023-03-03T17:48:03.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.698+0000 7f5b0b7fe700 1 -- 172.21.15.7:0/1235569191 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5b18057280 con 0x7f5b240abe20 2023-03-03T17:48:03.697 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:48:03.698 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:48:03.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 -- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b100682f0 msgr2=0x7f5b1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:03.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b100682f0 0x7f5b1006a7a0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f5b241e5610 tx=0x7f5b14008040 comp rx=0 tx=0).stop 2023-03-03T17:48:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 -- 172.21.15.7:0/1235569191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b240abe20 msgr2=0x7f5b24152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b240abe20 0x7f5b24152b30 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7f5b24072520 tx=0x7f5b18004750 comp rx=0 tx=0).stop 2023-03-03T17:48:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 -- 172.21.15.7:0/1235569191 shutdown_connections 2023-03-03T17:48:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b2414fda0 0x7f5b2406e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b100682f0 0x7f5b1006a7a0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b240abe20 0x7f5b24152b30 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 --2- 172.21.15.7:0/1235569191 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b2414f9d0 0x7f5b2406db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:03.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 -- 172.21.15.7:0/1235569191 >> 172.21.15.7:0/1235569191 conn(0x7f5b240c8900 msgr2=0x7f5b240c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:48:03.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 -- 172.21.15.7:0/1235569191 shutdown_connections 2023-03-03T17:48:03.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:03.700+0000 7f5b29259700 1 -- 172.21.15.7:0/1235569191 wait complete. 2023-03-03T17:48:03.701 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:48:03.711 INFO:tasks.mds_thrash.fs.[b]:waiting for 15 secs before thrashing 2023-03-03T17:48:18.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:48:18.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.850+0000 7fb52a911700 1 Processor -- start 2023-03-03T17:48:18.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.850+0000 7fb52a911700 1 -- start start 2023-03-03T17:48:18.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.851+0000 7fb52a911700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5240aad90 0x7fb52406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:18.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.851+0000 7fb52a911700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb52406bfe0 0x7fb52406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:18.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.851+0000 7fb52a911700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52406eda0 con 0x7fb52406b920 2023-03-03T17:48:18.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.851+0000 7fb52a911700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb52406f080 con 0x7fb52406bfe0 2023-03-03T17:48:18.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.851+0000 7fb52a911700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5241584b0 con 0x7fb5240aad90 2023-03-03T17:48:18.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.852+0000 7fb528eae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb52406bfe0 0x7fb52406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:18.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.852+0000 7fb523fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5240aad90 0x7fb52406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:18.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.852+0000 7fb5237fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb52406b920 0x7fb52406a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:60358/0 (socket says 172.21.15.7:60358) 2023-03-03T17:48:18.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.852+0000 7fb5237fe700 1 -- 172.21.15.7:0/3502971345 learned_addr learned my addr 172.21.15.7:0/3502971345 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:48:18.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.852+0000 7fb523fff700 1 -- 172.21.15.7:0/3502971345 >> v2:172.21.15.143:3300/0 conn(0x7fb52406bfe0 msgr2=0x7fb52406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:48:18.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.852+0000 7fb523fff700 1 --2- 172.21.15.7:0/3502971345 >> v2:172.21.15.143:3300/0 conn(0x7fb52406bfe0 0x7fb52406a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:18.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.852+0000 7fb523fff700 1 -- 172.21.15.7:0/3502971345 >> v1:172.21.15.143:6789/0 conn(0x7fb52406b920 legacy=0x7fb52406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:18.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.852+0000 7fb523fff700 1 -- 172.21.15.7:0/3502971345 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5241586a0 con 0x7fb5240aad90 2023-03-03T17:48:18.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.852+0000 7fb522ffd700 1 -- 172.21.15.7:0/3502971345 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137056717 0 0) 0x7fb52406eda0 con 0x7fb52406b920 2023-03-03T17:48:18.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.852+0000 7fb528eae700 1 --2- 172.21.15.7:0/3502971345 >> v2:172.21.15.143:3300/0 conn(0x7fb52406bfe0 0x7fb52406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T17:48:18.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb523fff700 1 --2- 172.21.15.7:0/3502971345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5240aad90 0x7fb52406e2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb514009840 tx=0x7fb514009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2d7944323327cb7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:48:18.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb522ffd700 1 -- 172.21.15.7:0/3502971345 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb514011830 con 0x7fb5240aad90 2023-03-03T17:48:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb522ffd700 1 -- 172.21.15.7:0/3502971345 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb514011990 con 0x7fb5240aad90 2023-03-03T17:48:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb522ffd700 1 -- 172.21.15.7:0/3502971345 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb514011d20 con 0x7fb5240aad90 2023-03-03T17:48:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb52a911700 1 -- 172.21.15.7:0/3502971345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5240aad90 msgr2=0x7fb52406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb52a911700 1 --2- 172.21.15.7:0/3502971345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5240aad90 0x7fb52406e2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb514009840 tx=0x7fb514009b50 comp rx=0 tx=0).stop 2023-03-03T17:48:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb52a911700 1 -- 172.21.15.7:0/3502971345 shutdown_connections 2023-03-03T17:48:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb52a911700 1 --2- 172.21.15.7:0/3502971345 >> v2:172.21.15.143:3300/0 conn(0x7fb52406bfe0 0x7fb52406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:18.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb52a911700 1 --2- 172.21.15.7:0/3502971345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5240aad90 0x7fb52406e2f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb52a911700 1 -- 172.21.15.7:0/3502971345 >> 172.21.15.7:0/3502971345 conn(0x7fb5240736f0 msgr2=0x7fb524073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:48:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb52a911700 1 -- 172.21.15.7:0/3502971345 shutdown_connections 2023-03-03T17:48:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.853+0000 7fb52a911700 1 -- 172.21.15.7:0/3502971345 wait complete. 2023-03-03T17:48:18.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.854+0000 7fb52a911700 1 Processor -- start 2023-03-03T17:48:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.855+0000 7fb52a911700 1 -- start start 2023-03-03T17:48:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.855+0000 7fb52a911700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52406b920 0x7fb5240c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.855+0000 7fb52a911700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb52406bfe0 0x7fb524157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.855+0000 7fb52a911700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5240aad90 0x7fb5241ed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:18.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.855+0000 7fb52a911700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52414e4a0 con 0x7fb5240aad90 2023-03-03T17:48:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.855+0000 7fb52a911700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5240c0e30 con 0x7fb52406bfe0 2023-03-03T17:48:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.855+0000 7fb52a911700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5240c24f0 con 0x7fb52406b920 2023-03-03T17:48:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb528eae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5240aad90 0x7fb5241ed4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb528eae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5240aad90 0x7fb5241ed4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34634/0 (socket says 172.21.15.7:34634) 2023-03-03T17:48:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb528eae700 1 -- 172.21.15.7:0/1051100145 learned_addr learned my addr 172.21.15.7:0/1051100145 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:48:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb523fff700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52406b920 0x7fb5240c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb5237fe700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb52406bfe0 0x7fb524157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:18.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb528eae700 1 -- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52406b920 msgr2=0x7fb5240c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:18.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb528eae700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52406b920 0x7fb5240c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:18.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb528eae700 1 -- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb52406bfe0 msgr2=0x7fb524157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:18.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb528eae700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb52406bfe0 0x7fb524157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:18.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb528eae700 1 -- 172.21.15.7:0/1051100145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c009a70 con 0x7fb5240aad90 2023-03-03T17:48:18.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb528eae700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5240aad90 0x7fb5241ed4a0 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fb51c00ef90 tx=0x7fb51c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:48:18.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb520ff9700 1 -- 172.21.15.7:0/1051100145 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51c00cc30 con 0x7fb5240aad90 2023-03-03T17:48:18.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb520ff9700 1 -- 172.21.15.7:0/1051100145 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51c00f040 con 0x7fb5240aad90 2023-03-03T17:48:18.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb52a911700 1 -- 172.21.15.7:0/1051100145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5140094a0 con 0x7fb5240aad90 2023-03-03T17:48:18.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb520ff9700 1 -- 172.21.15.7:0/1051100145 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51c014480 con 0x7fb5240aad90 2023-03-03T17:48:18.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.856+0000 7fb52a911700 1 -- 172.21.15.7:0/1051100145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb524071cd0 con 0x7fb5240aad90 2023-03-03T17:48:18.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.857+0000 7fb52a911700 1 -- 172.21.15.7:0/1051100145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb524158980 con 0x7fb5240aad90 2023-03-03T17:48:18.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.858+0000 7fb520ff9700 1 -- 172.21.15.7:0/1051100145 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb51c014600 con 0x7fb5240aad90 2023-03-03T17:48:18.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.858+0000 7fb520ff9700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5040682f0 0x7fb50406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:48:18.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.858+0000 7fb520ff9700 1 -- 172.21.15.7:0/1051100145 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb51c086b30 con 0x7fb5240aad90 2023-03-03T17:48:18.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.858+0000 7fb523fff700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5040682f0 0x7fb50406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:48:18.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.859+0000 7fb523fff700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5040682f0 0x7fb50406a7a0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fb5140090b0 tx=0x7fb514008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:48:18.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:18.863+0000 7fb520ff9700 1 -- 172.21.15.7:0/1051100145 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb51c052ed0 con 0x7fb5240aad90 2023-03-03T17:48:19.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.104+0000 7fb52a911700 1 -- 172.21.15.7:0/1051100145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb524071270 con 0x7fb5240aad90 2023-03-03T17:48:19.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.104+0000 7fb520ff9700 1 -- 172.21.15.7:0/1051100145 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb51c084e60 con 0x7fb5240aad90 2023-03-03T17:48:19.105 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:48:19.105 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:48:19.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 -- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5040682f0 msgr2=0x7fb50406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:19.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5040682f0 0x7fb50406a7a0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fb5140090b0 tx=0x7fb514008030 comp rx=0 tx=0).stop 2023-03-03T17:48:19.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 -- 172.21.15.7:0/1051100145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5240aad90 msgr2=0x7fb5241ed4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:48:19.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5240aad90 0x7fb5241ed4a0 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fb51c00ef90 tx=0x7fb51c00c3a0 comp rx=0 tx=0).stop 2023-03-03T17:48:19.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 -- 172.21.15.7:0/1051100145 shutdown_connections 2023-03-03T17:48:19.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5040682f0 0x7fb50406a7a0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:19.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5240aad90 0x7fb5241ed4a0 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:19.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb52406b920 0x7fb5240c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 --2- 172.21.15.7:0/1051100145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb52406bfe0 0x7fb524157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:48:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 -- 172.21.15.7:0/1051100145 >> 172.21.15.7:0/1051100145 conn(0x7fb5240736f0 msgr2=0x7fb524151630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:48:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.106+0000 7fb52a911700 1 -- 172.21.15.7:0/1051100145 shutdown_connections 2023-03-03T17:48:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:48:19.107+0000 7fb52a911700 1 -- 172.21.15.7:0/1051100145 wait complete. 2023-03-03T17:48:19.108 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:48:19.118 INFO:tasks.mds_thrash.fs.[b]:waiting for 68 secs before thrashing 2023-03-03T17:48:31.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:48:31.242 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:01.321 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:01.324 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:27.119 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:49:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.258+0000 7f444f2bf700 1 Processor -- start 2023-03-03T17:49:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.258+0000 7f444f2bf700 1 -- start start 2023-03-03T17:49:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444f2bf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44480aad90 0x7f444806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444f2bf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f444806b920 0x7f444806a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444f2bf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444806eda0 con 0x7f444806bfe0 2023-03-03T17:49:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444f2bf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f444806f080 con 0x7f44480aad90 2023-03-03T17:49:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444f2bf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44481584b0 con 0x7f444806b920 2023-03-03T17:49:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444d05b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44480aad90 0x7f444806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444d05b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44480aad90 0x7f444806e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34694/0 (socket says 172.21.15.7:34694) 2023-03-03T17:49:27.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444d05b700 1 -- 172.21.15.7:0/892801241 learned_addr learned my addr 172.21.15.7:0/892801241 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:49:27.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444c85a700 1 --2- 172.21.15.7:0/892801241 >> v2:172.21.15.143:3300/0 conn(0x7f444806b920 0x7f444806a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:27.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444d05b700 1 -- 172.21.15.7:0/892801241 >> v2:172.21.15.143:3300/0 conn(0x7f444806b920 msgr2=0x7f444806a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:49:27.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444d05b700 1 --2- 172.21.15.7:0/892801241 >> v2:172.21.15.143:3300/0 conn(0x7f444806b920 0x7f444806a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444d05b700 1 -- 172.21.15.7:0/892801241 >> v1:172.21.15.143:6789/0 conn(0x7f444806bfe0 legacy=0x7f444806a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:49:27.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444d05b700 1 -- 172.21.15.7:0/892801241 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44481586a0 con 0x7f44480aad90 2023-03-03T17:49:27.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444c85a700 1 --2- 172.21.15.7:0/892801241 >> v2:172.21.15.143:3300/0 conn(0x7f444806b920 0x7f444806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:49:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.259+0000 7f444d05b700 1 --2- 172.21.15.7:0/892801241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44480aad90 0x7f444806e2f0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f44480c5d60 tx=0x7f4444009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6875106c993ff841 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:49:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f443ffff700 1 -- 172.21.15.7:0/892801241 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4444004230 con 0x7f44480aad90 2023-03-03T17:49:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f443ffff700 1 -- 172.21.15.7:0/892801241 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4444004390 con 0x7f44480aad90 2023-03-03T17:49:27.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f443ffff700 1 -- 172.21.15.7:0/892801241 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44440036a0 con 0x7f44480aad90 2023-03-03T17:49:27.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f444f2bf700 1 -- 172.21.15.7:0/892801241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44480aad90 msgr2=0x7f444806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:27.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f444f2bf700 1 --2- 172.21.15.7:0/892801241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44480aad90 0x7f444806e2f0 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7f44480c5d60 tx=0x7f4444009bd0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f444f2bf700 1 -- 172.21.15.7:0/892801241 shutdown_connections 2023-03-03T17:49:27.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f444f2bf700 1 --2- 172.21.15.7:0/892801241 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44480aad90 0x7f444806e2f0 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f444f2bf700 1 --2- 172.21.15.7:0/892801241 >> v2:172.21.15.143:3300/0 conn(0x7f444806b920 0x7f444806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f444f2bf700 1 -- 172.21.15.7:0/892801241 >> 172.21.15.7:0/892801241 conn(0x7f44480736f0 msgr2=0x7f4448073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:49:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f444f2bf700 1 -- 172.21.15.7:0/892801241 shutdown_connections 2023-03-03T17:49:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.260+0000 7f444f2bf700 1 -- 172.21.15.7:0/892801241 wait complete. 2023-03-03T17:49:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.262+0000 7f444f2bf700 1 Processor -- start 2023-03-03T17:49:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.262+0000 7f444f2bf700 1 -- start start 2023-03-03T17:49:27.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444f2bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f444806b920 0x7f44481e9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:27.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444f2bf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f444806bfe0 0x7f44480ab950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:27.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444f2bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44480aad90 0x7f44480adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:27.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444f2bf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444814e380 con 0x7f444806bfe0 2023-03-03T17:49:27.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444f2bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4448152270 con 0x7f444806b920 2023-03-03T17:49:27.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444f2bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444814d770 con 0x7f44480aad90 2023-03-03T17:49:27.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444c85a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f444806bfe0 0x7f44480ab950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:27.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444c85a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f444806bfe0 0x7f44480ab950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34702/0 (socket says 172.21.15.7:34702) 2023-03-03T17:49:27.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444c85a700 1 -- 172.21.15.7:0/573345891 learned_addr learned my addr 172.21.15.7:0/573345891 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:49:27.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444d05b700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f444806b920 0x7f44481e9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:27.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444d85c700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44480aad90 0x7f44480adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:27.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444c85a700 1 -- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44480aad90 msgr2=0x7f44480adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:27.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444c85a700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44480aad90 0x7f44480adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444c85a700 1 -- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f444806b920 msgr2=0x7f44481e9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:27.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444c85a700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f444806b920 0x7f44481e9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444c85a700 1 -- 172.21.15.7:0/573345891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4444009550 con 0x7f444806bfe0 2023-03-03T17:49:27.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.263+0000 7f444c85a700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f444806bfe0 0x7f44480ab950 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f443800c7f0 tx=0x7f443800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:49:27.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.264+0000 7f443dffb700 1 -- 172.21.15.7:0/573345891 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4438011800 con 0x7f444806bfe0 2023-03-03T17:49:27.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.264+0000 7f443dffb700 1 -- 172.21.15.7:0/573345891 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4438011960 con 0x7f444806bfe0 2023-03-03T17:49:27.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.264+0000 7f443dffb700 1 -- 172.21.15.7:0/573345891 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4438011d10 con 0x7f444806bfe0 2023-03-03T17:49:27.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.264+0000 7f444f2bf700 1 -- 172.21.15.7:0/573345891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4448071cd0 con 0x7f444806bfe0 2023-03-03T17:49:27.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.264+0000 7f444f2bf700 1 -- 172.21.15.7:0/573345891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4448158980 con 0x7f444806bfe0 2023-03-03T17:49:27.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.265+0000 7f444f2bf700 1 -- 172.21.15.7:0/573345891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4448071270 con 0x7f444806bfe0 2023-03-03T17:49:27.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.265+0000 7f443dffb700 1 -- 172.21.15.7:0/573345891 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4438021070 con 0x7f444806bfe0 2023-03-03T17:49:27.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.265+0000 7f443dffb700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f442c0682f0 0x7f442c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:27.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.266+0000 7f443dffb700 1 -- 172.21.15.7:0/573345891 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4438085fd0 con 0x7f444806bfe0 2023-03-03T17:49:27.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.266+0000 7f444d05b700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f442c0682f0 0x7f442c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:27.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.266+0000 7f444d05b700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f442c0682f0 0x7f442c06a7a0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f4444009130 tx=0x7f4444008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:49:27.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.270+0000 7f443dffb700 1 -- 172.21.15.7:0/573345891 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4438052370 con 0x7f444806bfe0 2023-03-03T17:49:27.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.508+0000 7f444f2bf700 1 -- 172.21.15.7:0/573345891 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4448074bb0 con 0x7f444806bfe0 2023-03-03T17:49:27.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.509+0000 7f443dffb700 1 -- 172.21.15.7:0/573345891 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4438055990 con 0x7f444806bfe0 2023-03-03T17:49:27.509 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:49:27.509 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:49:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.510+0000 7f444f2bf700 1 -- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f442c0682f0 msgr2=0x7f442c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:27.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f442c0682f0 0x7f442c06a7a0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f4444009130 tx=0x7f4444008030 comp rx=0 tx=0).stop 2023-03-03T17:49:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 -- 172.21.15.7:0/573345891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f444806bfe0 msgr2=0x7f44480ab950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f444806bfe0 0x7f44480ab950 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f443800c7f0 tx=0x7f443800cbb0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 -- 172.21.15.7:0/573345891 shutdown_connections 2023-03-03T17:49:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44480aad90 0x7f44480adb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f444806b920 0x7f44481e9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f442c0682f0 0x7f442c06a7a0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 --2- 172.21.15.7:0/573345891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f444806bfe0 0x7f44480ab950 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 -- 172.21.15.7:0/573345891 >> 172.21.15.7:0/573345891 conn(0x7f44480736f0 msgr2=0x7f44480c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:49:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 -- 172.21.15.7:0/573345891 shutdown_connections 2023-03-03T17:49:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:27.511+0000 7f444f2bf700 1 -- 172.21.15.7:0/573345891 wait complete. 2023-03-03T17:49:27.513 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:49:27.523 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2023-03-03T17:49:31.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:31.396 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:49:42.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:49:42.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.385+0000 7f4a218cf700 1 Processor -- start 2023-03-03T17:49:42.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.386+0000 7f4a218cf700 1 -- start start 2023-03-03T17:49:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.386+0000 7f4a218cf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4a1c0c00e0 0x7f4a1c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.386+0000 7f4a218cf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c154ff0 0x7f4a1c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.386+0000 7f4a218cf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c0c7310 con 0x7f4a1c0caa30 2023-03-03T17:49:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.386+0000 7f4a218cf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1c0bfd10 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.386+0000 7f4a218cf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c0bfa30 con 0x7f4a1c154ff0 2023-03-03T17:49:42.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.386+0000 7f4a1a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4a1c0c00e0 0x7f4a1c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.386+0000 7f4a1affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4a1c0caa30 0x7f4a1c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56932/0 (socket says 172.21.15.7:56932) 2023-03-03T17:49:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.386+0000 7f4a1affd700 1 -- 172.21.15.7:0/651237026 learned_addr learned my addr 172.21.15.7:0/651237026 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:49:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a1b7fe700 1 --2- 172.21.15.7:0/651237026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c154ff0 0x7f4a1c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a1a7fc700 1 -- 172.21.15.7:0/651237026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c154ff0 msgr2=0x7f4a1c0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:49:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a1a7fc700 1 --2- 172.21.15.7:0/651237026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c154ff0 0x7f4a1c0cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a1a7fc700 1 -- 172.21.15.7:0/651237026 >> v1:172.21.15.7:6789/0 conn(0x7f4a1c0caa30 legacy=0x7f4a1c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a1a7fc700 1 -- 172.21.15.7:0/651237026 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c154510 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a19ffb700 1 -- 172.21.15.7:0/651237026 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277270447 0 0) 0x7f4a1c0c7310 con 0x7f4a1c0caa30 2023-03-03T17:49:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a1b7fe700 1 --2- 172.21.15.7:0/651237026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c154ff0 0x7f4a1c0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:49:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a1a7fc700 1 --2- 172.21.15.7:0/651237026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 0x7f4a1c0c5c60 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009820 tx=0x7f4a04009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7530fff07323cd79 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:49:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a19ffb700 1 -- 172.21.15.7:0/651237026 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04005e30 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a19ffb700 1 -- 172.21.15.7:0/651237026 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a04006030 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.387+0000 7f4a19ffb700 1 -- 172.21.15.7:0/651237026 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a040063e0 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.388+0000 7f4a218cf700 1 -- 172.21.15.7:0/651237026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 msgr2=0x7f4a1c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:42.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.388+0000 7f4a218cf700 1 --2- 172.21.15.7:0/651237026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 0x7f4a1c0c5c60 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009820 tx=0x7f4a04009b30 comp rx=0 tx=0).stop 2023-03-03T17:49:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.388+0000 7f4a218cf700 1 -- 172.21.15.7:0/651237026 shutdown_connections 2023-03-03T17:49:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.388+0000 7f4a218cf700 1 --2- 172.21.15.7:0/651237026 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c154ff0 0x7f4a1c0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.388+0000 7f4a218cf700 1 --2- 172.21.15.7:0/651237026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 0x7f4a1c0c5c60 unknown :-1 s=CLOSED pgs=3029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.388+0000 7f4a218cf700 1 -- 172.21.15.7:0/651237026 >> 172.21.15.7:0/651237026 conn(0x7f4a1c066e40 msgr2=0x7f4a1c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:49:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.388+0000 7f4a218cf700 1 -- 172.21.15.7:0/651237026 shutdown_connections 2023-03-03T17:49:42.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.388+0000 7f4a218cf700 1 -- 172.21.15.7:0/651237026 wait complete. 2023-03-03T17:49:42.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.389+0000 7f4a218cf700 1 Processor -- start 2023-03-03T17:49:42.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.389+0000 7f4a218cf700 1 -- start start 2023-03-03T17:49:42.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a218cf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 0x7f4a1c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:42.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a218cf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c0caa30 0x7f4a1c1edad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:42.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a218cf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a1c154ff0 0x7f4a1c0c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:42.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a218cf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c1de1c0 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a218cf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c1ee010 con 0x7f4a1c154ff0 2023-03-03T17:49:42.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a218cf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c14d9a0 con 0x7f4a1c0caa30 2023-03-03T17:49:42.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a1affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 0x7f4a1c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:42.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a1affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 0x7f4a1c1564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34726/0 (socket says 172.21.15.7:34726) 2023-03-03T17:49:42.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a1affd700 1 -- 172.21.15.7:0/596121534 learned_addr learned my addr 172.21.15.7:0/596121534 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:49:42.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a1a7fc700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c0caa30 0x7f4a1c1edad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:42.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a1b7fe700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a1c154ff0 0x7f4a1c0c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:42.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a1affd700 1 -- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c0caa30 msgr2=0x7f4a1c1edad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:42.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a1affd700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c0caa30 0x7f4a1c1edad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:42.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a1affd700 1 -- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a1c154ff0 msgr2=0x7f4a1c0c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:42.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a1affd700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a1c154ff0 0x7f4a1c0c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:42.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.390+0000 7f4a1affd700 1 -- 172.21.15.7:0/596121534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a04009430 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.391+0000 7f4a1affd700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 0x7f4a1c1564b0 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00c750 tx=0x7f4a0c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:49:42.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.391+0000 7f4a13fff700 1 -- 172.21.15.7:0/596121534 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c004c80 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.391+0000 7f4a13fff700 1 -- 172.21.15.7:0/596121534 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c004e00 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.391+0000 7f4a218cf700 1 -- 172.21.15.7:0/596121534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1c154ad0 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.391+0000 7f4a13fff700 1 -- 172.21.15.7:0/596121534 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c007310 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.391+0000 7f4a218cf700 1 -- 172.21.15.7:0/596121534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1c1547f0 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.392+0000 7f4a218cf700 1 -- 172.21.15.7:0/596121534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1c0bf650 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.392+0000 7f4a13fff700 1 -- 172.21.15.7:0/596121534 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4a0c01a020 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.393+0000 7f4a13fff700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49fc0682f0 0x7f49fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:49:42.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.393+0000 7f4a13fff700 1 -- 172.21.15.7:0/596121534 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4a0c085d80 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.393+0000 7f4a1a7fc700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49fc0682f0 0x7f49fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:49:42.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.394+0000 7f4a1a7fc700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49fc0682f0 0x7f49fc06a7a0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f4a04013040 tx=0x7f4a04008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:49:42.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.397+0000 7f4a13fff700 1 -- 172.21.15.7:0/596121534 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4a0c052120 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.634+0000 7f4a218cf700 1 -- 172.21.15.7:0/596121534 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a1c074bb0 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.635+0000 7f4a13fff700 1 -- 172.21.15.7:0/596121534 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4a0c059680 con 0x7f4a1c0c00e0 2023-03-03T17:49:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:49:42.634 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:49:42.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 -- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49fc0682f0 msgr2=0x7f49fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:42.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49fc0682f0 0x7f49fc06a7a0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f4a04013040 tx=0x7f4a04008070 comp rx=0 tx=0).stop 2023-03-03T17:49:42.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 -- 172.21.15.7:0/596121534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 msgr2=0x7f4a1c1564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:49:42.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 0x7f4a1c1564b0 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c00c750 tx=0x7f4a0c00cb10 comp rx=0 tx=0).stop 2023-03-03T17:49:42.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 -- 172.21.15.7:0/596121534 shutdown_connections 2023-03-03T17:49:42.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a1c154ff0 0x7f4a1c0c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:42.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f49fc0682f0 0x7f49fc06a7a0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:42.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a1c0c00e0 0x7f4a1c1564b0 unknown :-1 s=CLOSED pgs=3030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:42.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 --2- 172.21.15.7:0/596121534 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a1c0caa30 0x7f4a1c1edad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:49:42.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 -- 172.21.15.7:0/596121534 >> 172.21.15.7:0/596121534 conn(0x7f4a1c066e40 msgr2=0x7f4a1c06b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:49:42.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 -- 172.21.15.7:0/596121534 shutdown_connections 2023-03-03T17:49:42.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:49:42.637+0000 7f4a218cf700 1 -- 172.21.15.7:0/596121534 wait complete. 2023-03-03T17:49:42.638 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:49:42.648 INFO:tasks.mds_thrash.fs.[a]:waiting for 82 secs before thrashing 2023-03-03T17:49:59.998 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T17:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T17:50:01.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:50:01.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:50:31.520 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:50:31.523 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:01.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:01.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:04.524 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:51:04.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:51:04.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.664+0000 7f04e648c700 1 Processor -- start 2023-03-03T17:51:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.664+0000 7f04e648c700 1 -- start start 2023-03-03T17:51:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.665+0000 7f04e648c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04e00caa30 0x7f04e00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.665+0000 7f04e648c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e00c7310 con 0x7f04e0154ff0 2023-03-03T17:51:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.665+0000 7f04e648c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e00bfd10 con 0x7f04e00c00e0 2023-03-03T17:51:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.665+0000 7f04e648c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e0155430 con 0x7f04e00caa30 2023-03-03T17:51:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.665+0000 7f04dffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04e00caa30 0x7f04e00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.665+0000 7f04e4a29700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f04e0154ff0 0x7f04e015ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57020/0 (socket says 172.21.15.7:57020) 2023-03-03T17:51:04.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.665+0000 7f04e4a29700 1 -- 172.21.15.7:0/3409324758 learned_addr learned my addr 172.21.15.7:0/3409324758 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:51:04.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.666+0000 7f04dffff700 1 -- 172.21.15.7:0/3409324758 >> v1:172.21.15.143:6789/0 conn(0x7f04e00c00e0 legacy=0x7f04e00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.666+0000 7f04dffff700 1 -- 172.21.15.7:0/3409324758 >> v1:172.21.15.7:6789/0 conn(0x7f04e0154ff0 legacy=0x7f04e015ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.666+0000 7f04dffff700 1 -- 172.21.15.7:0/3409324758 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e00bfa30 con 0x7f04e00caa30 2023-03-03T17:51:04.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.666+0000 7f04dffff700 1 --2- 172.21.15.7:0/3409324758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e00caa30 0x7f04e00c5c60 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009840 tx=0x7f04d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95618d5799890690 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:51:04.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.667+0000 7f04deffd700 1 -- 172.21.15.7:0/3409324758 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0011830 con 0x7f04e00caa30 2023-03-03T17:51:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.667+0000 7f04deffd700 1 -- 172.21.15.7:0/3409324758 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d0011990 con 0x7f04e00caa30 2023-03-03T17:51:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.667+0000 7f04deffd700 1 -- 172.21.15.7:0/3409324758 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0011d20 con 0x7f04e00caa30 2023-03-03T17:51:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.667+0000 7f04e648c700 1 -- 172.21.15.7:0/3409324758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e00caa30 msgr2=0x7f04e00c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.667+0000 7f04e648c700 1 --2- 172.21.15.7:0/3409324758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e00caa30 0x7f04e00c5c60 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009840 tx=0x7f04d0009b50 comp rx=0 tx=0).stop 2023-03-03T17:51:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.667+0000 7f04e648c700 1 -- 172.21.15.7:0/3409324758 shutdown_connections 2023-03-03T17:51:04.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.667+0000 7f04e648c700 1 --2- 172.21.15.7:0/3409324758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e00caa30 0x7f04e00c5c60 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.667+0000 7f04e648c700 1 -- 172.21.15.7:0/3409324758 >> 172.21.15.7:0/3409324758 conn(0x7f04e0066e40 msgr2=0x7f04e0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:51:04.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.667+0000 7f04e648c700 1 -- 172.21.15.7:0/3409324758 shutdown_connections 2023-03-03T17:51:04.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.667+0000 7f04e648c700 1 -- 172.21.15.7:0/3409324758 wait complete. 2023-03-03T17:51:04.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.668+0000 7f04e648c700 1 Processor -- start 2023-03-03T17:51:04.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.669+0000 7f04e648c700 1 -- start start 2023-03-03T17:51:04.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.669+0000 7f04e648c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04e00c00e0 0x7f04e014b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.669+0000 7f04e648c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04e00caa30 0x7f04e014e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.669+0000 7f04e648c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e0154ff0 0x7f04e00c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.669+0000 7f04e648c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e00c20a0 con 0x7f04e0154ff0 2023-03-03T17:51:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.669+0000 7f04e648c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e00c1d50 con 0x7f04e00c00e0 2023-03-03T17:51:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.669+0000 7f04e648c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e00c4c40 con 0x7f04e00caa30 2023-03-03T17:51:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e4a29700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e0154ff0 0x7f04e00c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e4a29700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e0154ff0 0x7f04e00c37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34814/0 (socket says 172.21.15.7:34814) 2023-03-03T17:51:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e4a29700 1 -- 172.21.15.7:0/2033358704 learned_addr learned my addr 172.21.15.7:0/2033358704 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:51:04.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04dffff700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04e00c00e0 0x7f04e014b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04df7fe700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04e00caa30 0x7f04e014e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e4a29700 1 -- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04e00caa30 msgr2=0x7f04e014e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e4a29700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04e00caa30 0x7f04e014e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e4a29700 1 -- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04e00c00e0 msgr2=0x7f04e014b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e4a29700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04e00c00e0 0x7f04e014b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e4a29700 1 -- 172.21.15.7:0/2033358704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d00094a0 con 0x7f04e0154ff0 2023-03-03T17:51:04.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e4a29700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e0154ff0 0x7f04e00c37d0 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7f04d800cbf0 tx=0x7f04d80021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:51:04.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04dcff9700 1 -- 172.21.15.7:0/2033358704 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d800ab40 con 0x7f04e0154ff0 2023-03-03T17:51:04.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04dcff9700 1 -- 172.21.15.7:0/2033358704 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d800aca0 con 0x7f04e0154ff0 2023-03-03T17:51:04.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e648c700 1 -- 172.21.15.7:0/2033358704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e01547f0 con 0x7f04e0154ff0 2023-03-03T17:51:04.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04dcff9700 1 -- 172.21.15.7:0/2033358704 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d8012600 con 0x7f04e0154ff0 2023-03-03T17:51:04.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.670+0000 7f04e648c700 1 -- 172.21.15.7:0/2033358704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e0154510 con 0x7f04e0154ff0 2023-03-03T17:51:04.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.671+0000 7f04e648c700 1 -- 172.21.15.7:0/2033358704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e0154ad0 con 0x7f04e0154ff0 2023-03-03T17:51:04.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.672+0000 7f04dcff9700 1 -- 172.21.15.7:0/2033358704 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f04d8019030 con 0x7f04e0154ff0 2023-03-03T17:51:04.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.672+0000 7f04dcff9700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04cc0682f0 0x7f04cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.673+0000 7f04dcff9700 1 -- 172.21.15.7:0/2033358704 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f04d8085c00 con 0x7f04e0154ff0 2023-03-03T17:51:04.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.673+0000 7f04dffff700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04cc0682f0 0x7f04cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.673+0000 7f04dffff700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04cc0682f0 0x7f04cc06a7a0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f04d00090b0 tx=0x7f04d0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:51:04.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.677+0000 7f04dcff9700 1 -- 172.21.15.7:0/2033358704 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f04d804b5e0 con 0x7f04e0154ff0 2023-03-03T17:51:04.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.778+0000 7ff66fcd4700 1 Processor -- start 2023-03-03T17:51:04.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.778+0000 7ff66fcd4700 1 -- start start 2023-03-03T17:51:04.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66fcd4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff66814fda0 0x7ff668156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66fcd4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 0x7ff6681529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66fcd4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6680c2f30 con 0x7ff66814f9d0 2023-03-03T17:51:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66fcd4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6680c31d0 con 0x7ff66814fda0 2023-03-03T17:51:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66fcd4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6681524a0 con 0x7ff6680abe20 2023-03-03T17:51:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66d26f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff66814fda0 0x7ff668156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66d26f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff66814fda0 0x7ff668156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34820/0 (socket says 172.21.15.7:34820) 2023-03-03T17:51:04.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66d26f700 1 -- 172.21.15.7:0/3163394704 learned_addr learned my addr 172.21.15.7:0/3163394704 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:51:04.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66e271700 1 --2- 172.21.15.7:0/3163394704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 0x7ff6681529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66d26f700 1 -- 172.21.15.7:0/3163394704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 msgr2=0x7ff6681529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:51:04.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66d26f700 1 --2- 172.21.15.7:0/3163394704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 0x7ff6681529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.779+0000 7ff66d26f700 1 -- 172.21.15.7:0/3163394704 >> v1:172.21.15.143:6789/0 conn(0x7ff66814f9d0 legacy=0x7ff668071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66d26f700 1 -- 172.21.15.7:0/3163394704 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6680c34b0 con 0x7ff66814fda0 2023-03-03T17:51:04.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66e271700 1 --2- 172.21.15.7:0/3163394704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 0x7ff6681529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:51:04.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66d26f700 1 --2- 172.21.15.7:0/3163394704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 0x7ff668156430 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7ff6580098f0 tx=0x7ff658009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5e9ad18893585a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:51:04.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66ca6e700 1 -- 172.21.15.7:0/3163394704 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65800b8a0 con 0x7ff66814fda0 2023-03-03T17:51:04.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66ca6e700 1 -- 172.21.15.7:0/3163394704 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65800ba00 con 0x7ff66814fda0 2023-03-03T17:51:04.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66ca6e700 1 -- 172.21.15.7:0/3163394704 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65800bd90 con 0x7ff66814fda0 2023-03-03T17:51:04.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3163394704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 msgr2=0x7ff668156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66fcd4700 1 --2- 172.21.15.7:0/3163394704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 0x7ff668156430 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7ff6580098f0 tx=0x7ff658009c00 comp rx=0 tx=0).stop 2023-03-03T17:51:04.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3163394704 shutdown_connections 2023-03-03T17:51:04.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66fcd4700 1 --2- 172.21.15.7:0/3163394704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 0x7ff6681529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66fcd4700 1 --2- 172.21.15.7:0/3163394704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 0x7ff668156430 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3163394704 >> 172.21.15.7:0/3163394704 conn(0x7ff6680c8900 msgr2=0x7ff6680c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:51:04.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3163394704 shutdown_connections 2023-03-03T17:51:04.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.780+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3163394704 wait complete. 2023-03-03T17:51:04.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66fcd4700 1 Processor -- start 2023-03-03T17:51:04.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66fcd4700 1 -- start start 2023-03-03T17:51:04.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66fcd4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 0x7ff6681ddb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66fcd4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66814f9d0 0x7ff6681dd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66fcd4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 0x7ff668148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66fcd4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66814ba10 con 0x7ff66814fda0 2023-03-03T17:51:04.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66fcd4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6681de060 con 0x7ff66814f9d0 2023-03-03T17:51:04.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66fcd4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6681559f0 con 0x7ff6680abe20 2023-03-03T17:51:04.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66e271700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 0x7ff668148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66e271700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 0x7ff668148ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34828/0 (socket says 172.21.15.7:34828) 2023-03-03T17:51:04.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.782+0000 7ff66e271700 1 -- 172.21.15.7:0/3642271999 learned_addr learned my addr 172.21.15.7:0/3642271999 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:51:04.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff66d26f700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66814f9d0 0x7ff6681dd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff66da70700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 0x7ff6681ddb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff66e271700 1 -- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 msgr2=0x7ff6681ddb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff66e271700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 0x7ff6681ddb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff66e271700 1 -- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66814f9d0 msgr2=0x7ff6681dd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff66e271700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66814f9d0 0x7ff6681dd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff66e271700 1 -- 172.21.15.7:0/3642271999 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff658009550 con 0x7ff66814fda0 2023-03-03T17:51:04.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff66e271700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 0x7ff668148ae0 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7ff660007e40 tx=0x7ff66000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:51:04.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff65e7fc700 1 -- 172.21.15.7:0/3642271999 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66000de90 con 0x7ff66814fda0 2023-03-03T17:51:04.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff65e7fc700 1 -- 172.21.15.7:0/3642271999 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66000f040 con 0x7ff66814fda0 2023-03-03T17:51:04.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3642271999 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6680ac350 con 0x7ff66814fda0 2023-03-03T17:51:04.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff65e7fc700 1 -- 172.21.15.7:0/3642271999 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff660003c20 con 0x7ff66814fda0 2023-03-03T17:51:04.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.783+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3642271999 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6680ab630 con 0x7ff66814fda0 2023-03-03T17:51:04.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.784+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3642271999 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff668071fa0 con 0x7ff66814fda0 2023-03-03T17:51:04.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.785+0000 7ff65e7fc700 1 -- 172.21.15.7:0/3642271999 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff66001c030 con 0x7ff66814fda0 2023-03-03T17:51:04.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.785+0000 7ff65e7fc700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff64c068320 0x7ff64c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:04.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.785+0000 7ff65e7fc700 1 -- 172.21.15.7:0/3642271999 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff6600860e0 con 0x7ff66814fda0 2023-03-03T17:51:04.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.785+0000 7ff66da70700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff64c068320 0x7ff64c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:04.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.786+0000 7ff66da70700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff64c068320 0x7ff64c06a7d0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7ff664000f60 tx=0x7ff664009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:51:04.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.789+0000 7ff65e7fc700 1 -- 172.21.15.7:0/3642271999 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff660052420 con 0x7ff66814fda0 2023-03-03T17:51:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.920+0000 7f04e648c700 1 -- 172.21.15.7:0/2033358704 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04e0074bb0 con 0x7f04e0154ff0 2023-03-03T17:51:04.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.920+0000 7f04dcff9700 1 -- 172.21.15.7:0/2033358704 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f04d808b020 con 0x7f04e0154ff0 2023-03-03T17:51:04.920 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:51:04.921 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:51:04.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.922+0000 7f04e648c700 1 -- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04cc0682f0 msgr2=0x7f04cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.922+0000 7f04e648c700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04cc0682f0 0x7f04cc06a7a0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f04d00090b0 tx=0x7f04d0008030 comp rx=0 tx=0).stop 2023-03-03T17:51:04.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.922+0000 7f04e648c700 1 -- 172.21.15.7:0/2033358704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e0154ff0 msgr2=0x7f04e00c37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:04.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.922+0000 7f04e648c700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e0154ff0 0x7f04e00c37d0 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7f04d800cbf0 tx=0x7f04d80021c0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.922+0000 7f04e648c700 1 -- 172.21.15.7:0/2033358704 shutdown_connections 2023-03-03T17:51:04.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.922+0000 7f04e648c700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04cc0682f0 0x7f04cc06a7a0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.923+0000 7f04e648c700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e0154ff0 0x7f04e00c37d0 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.923+0000 7f04e648c700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04e00c00e0 0x7f04e014b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.923+0000 7f04e648c700 1 --2- 172.21.15.7:0/2033358704 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04e00caa30 0x7f04e014e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:04.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.923+0000 7f04e648c700 1 -- 172.21.15.7:0/2033358704 >> 172.21.15.7:0/2033358704 conn(0x7f04e0066e40 msgr2=0x7f04e00c6790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:51:04.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.923+0000 7f04e648c700 1 -- 172.21.15.7:0/2033358704 shutdown_connections 2023-03-03T17:51:04.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:04.923+0000 7f04e648c700 1 -- 172.21.15.7:0/2033358704 wait complete. 2023-03-03T17:51:04.924 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:51:04.934 INFO:tasks.mds_thrash.fs.[b]:waiting for 70 secs before thrashing 2023-03-03T17:51:05.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.032+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3642271999 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff668074bb0 con 0x7ff66814fda0 2023-03-03T17:51:05.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.032+0000 7ff65e7fc700 1 -- 172.21.15.7:0/3642271999 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff660055a40 con 0x7ff66814fda0 2023-03-03T17:51:05.032 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:51:05.032 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:51:05.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff64c068320 msgr2=0x7ff64c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:05.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff64c068320 0x7ff64c06a7d0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7ff664000f60 tx=0x7ff664009040 comp rx=0 tx=0).stop 2023-03-03T17:51:05.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3642271999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 msgr2=0x7ff668148ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:05.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 0x7ff668148ae0 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7ff660007e40 tx=0x7ff66000d3b0 comp rx=0 tx=0).stop 2023-03-03T17:51:05.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3642271999 shutdown_connections 2023-03-03T17:51:05.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff64c068320 0x7ff64c06a7d0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:05.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff66814fda0 0x7ff668148ae0 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:05.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6680abe20 0x7ff6681ddb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:05.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 --2- 172.21.15.7:0/3642271999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff66814f9d0 0x7ff6681dd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:05.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3642271999 >> 172.21.15.7:0/3642271999 conn(0x7ff6680c8900 msgr2=0x7ff668150c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:51:05.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3642271999 shutdown_connections 2023-03-03T17:51:05.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:05.034+0000 7ff66fcd4700 1 -- 172.21.15.7:0/3642271999 wait complete. 2023-03-03T17:51:05.036 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:51:05.046 INFO:tasks.mds_thrash.fs.[a]:waiting for 13 secs before thrashing 2023-03-03T17:51:18.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:51:18.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.182+0000 7f366d97a700 1 Processor -- start 2023-03-03T17:51:18.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.183+0000 7f366d97a700 1 -- start start 2023-03-03T17:51:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.183+0000 7f366d97a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3668151cc0 0x7f36680ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.183+0000 7f366d97a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36680c6cf0 con 0x7f36680b00b0 2023-03-03T17:51:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.183+0000 7f366d97a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36681548f0 con 0x7f36680cac10 2023-03-03T17:51:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.183+0000 7f366d97a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3668152090 con 0x7f3668151cc0 2023-03-03T17:51:18.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.183+0000 7f36667fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f36680b00b0 0x7f36680c7050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57066/0 (socket says 172.21.15.7:57066) 2023-03-03T17:51:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f36667fc700 1 -- 172.21.15.7:0/2883798799 learned_addr learned my addr 172.21.15.7:0/2883798799 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:51:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3666ffd700 1 --2- 172.21.15.7:0/2883798799 >> v2:172.21.15.143:3300/0 conn(0x7f3668151cc0 0x7f36680ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793407089 0 0) 0x7f36680c6cf0 con 0x7f36680b00b0 2023-03-03T17:51:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36500035b0 con 0x7f36680b00b0 2023-03-03T17:51:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 332132441 0 0) 0x7f36500035b0 con 0x7f36680b00b0 2023-03-03T17:51:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 >> v2:172.21.15.143:3300/0 conn(0x7f3668151cc0 msgr2=0x7f36680ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:51:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 --2- 172.21.15.7:0/2883798799 >> v2:172.21.15.143:3300/0 conn(0x7f3668151cc0 0x7f36680ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 >> v1:172.21.15.143:6789/0 conn(0x7f36680cac10 legacy=0x7f366815ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3668154bd0 con 0x7f36680b00b0 2023-03-03T17:51:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f365c001670 con 0x7f36680b00b0 2023-03-03T17:51:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 >> v1:172.21.15.7:6789/0 conn(0x7f36680b00b0 legacy=0x7f36680c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f3650003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3650004850 0x7f3650004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3650005250 0x7f365000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365000c160 con 0x7f3650003a70 2023-03-03T17:51:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365000c2d0 con 0x7f3650005250 2023-03-03T17:51:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365000c490 con 0x7f3650004850 2023-03-03T17:51:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f3666ffd700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f3650003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f36667fc700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3650004850 0x7f3650004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.184+0000 7f36677fe700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3650005250 0x7f365000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f3666ffd700 1 -- 172.21.15.7:0/2883798799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3650004850 msgr2=0x7f3650004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:51:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f3666ffd700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3650004850 0x7f3650004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f3666ffd700 1 -- 172.21.15.7:0/2883798799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3650005250 msgr2=0x7f365000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:51:18.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f3666ffd700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3650005250 0x7f365000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f3666ffd700 1 -- 172.21.15.7:0/2883798799 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36500035b0 con 0x7f3650003a70 2023-03-03T17:51:18.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f36667fc700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3650004850 0x7f3650004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:51:18.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f36677fe700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3650005250 0x7f365000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:51:18.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f36677fe700 1 -- 172.21.15.7:0/2883798799 reap_dead start 2023-03-03T17:51:18.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f3666ffd700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f3650003e80 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f3658009b30 tx=0x7f3650005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e5dbd93bf45330 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:51:18.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365800e6c0 con 0x7f3650003a70 2023-03-03T17:51:18.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365800e820 con 0x7f3650003a70 2023-03-03T17:51:18.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f3665ffb700 1 -- 172.21.15.7:0/2883798799 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365800eb90 con 0x7f3650003a70 2023-03-03T17:51:18.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f366d97a700 1 -- 172.21.15.7:0/2883798799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 msgr2=0x7f3650003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:18.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f366d97a700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f3650003e80 secure :-1 s=READY pgs=3037 cs=0 l=1 rev1=1 crypto rx=0x7f3658009b30 tx=0x7f3650005880 comp rx=0 tx=0).stop 2023-03-03T17:51:18.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f366d97a700 1 -- 172.21.15.7:0/2883798799 shutdown_connections 2023-03-03T17:51:18.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f366d97a700 1 --2- 172.21.15.7:0/2883798799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f3650003e80 unknown :-1 s=CLOSED pgs=3037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.185+0000 7f366d97a700 1 -- 172.21.15.7:0/2883798799 >> 172.21.15.7:0/2883798799 conn(0x7f366806b980 msgr2=0x7f3668071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:51:18.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.186+0000 7f366d97a700 1 -- 172.21.15.7:0/2883798799 shutdown_connections 2023-03-03T17:51:18.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.186+0000 7f366d97a700 1 -- 172.21.15.7:0/2883798799 wait complete. 2023-03-03T17:51:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.187+0000 7f366d97a700 1 Processor -- start 2023-03-03T17:51:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.187+0000 7f366d97a700 1 -- start start 2023-03-03T17:51:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f366d97a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f366814f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:18.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f366d97a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36680c8540 0x7f3668141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f366d97a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3668146790 0x7f3668150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f366d97a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36681ee160 con 0x7f3650003a70 2023-03-03T17:51:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f366d97a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36680c48f0 con 0x7f3668146790 2023-03-03T17:51:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f366d97a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3668141640 con 0x7f36680c8540 2023-03-03T17:51:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f3666ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f366814f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f3666ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f366814f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34864/0 (socket says 172.21.15.7:34864) 2023-03-03T17:51:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f3666ffd700 1 -- 172.21.15.7:0/183760937 learned_addr learned my addr 172.21.15.7:0/183760937 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:51:18.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f36677fe700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3668146790 0x7f3668150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f3666ffd700 1 -- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36680c8540 msgr2=0x7f3668141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f36667fc700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36680c8540 0x7f3668141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f3666ffd700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36680c8540 0x7f3668141100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f3666ffd700 1 -- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3668146790 msgr2=0x7f3668150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f3666ffd700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3668146790 0x7f3668150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f3666ffd700 1 -- 172.21.15.7:0/183760937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3668155190 con 0x7f3650003a70 2023-03-03T17:51:18.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f36677fe700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3668146790 0x7f3668150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:51:18.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f3666ffd700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f366814f170 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f365800efd0 tx=0x7f36580047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:51:18.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f364ffff700 1 -- 172.21.15.7:0/183760937 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365800e6c0 con 0x7f3650003a70 2023-03-03T17:51:18.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f364ffff700 1 -- 172.21.15.7:0/183760937 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3658003ab0 con 0x7f3650003a70 2023-03-03T17:51:18.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f366d97a700 1 -- 172.21.15.7:0/183760937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3668154eb0 con 0x7f3650003a70 2023-03-03T17:51:18.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f364ffff700 1 -- 172.21.15.7:0/183760937 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3658003e60 con 0x7f3650003a70 2023-03-03T17:51:18.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.188+0000 7f366d97a700 1 -- 172.21.15.7:0/183760937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36681545a0 con 0x7f3650003a70 2023-03-03T17:51:18.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.189+0000 7f366d97a700 1 -- 172.21.15.7:0/183760937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36681edf30 con 0x7f3650003a70 2023-03-03T17:51:18.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.190+0000 7f364ffff700 1 -- 172.21.15.7:0/183760937 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3658019030 con 0x7f3650003a70 2023-03-03T17:51:18.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.190+0000 7f364ffff700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36540682f0 0x7f365406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:51:18.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.190+0000 7f364ffff700 1 -- 172.21.15.7:0/183760937 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3658087880 con 0x7f3650003a70 2023-03-03T17:51:18.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.190+0000 7f36667fc700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36540682f0 0x7f365406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:51:18.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.191+0000 7f36667fc700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36540682f0 0x7f365406a7a0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f365c000c00 tx=0x7f365c001470 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:51:18.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.195+0000 7f364ffff700 1 -- 172.21.15.7:0/183760937 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f36681edf30 con 0x7f3650003a70 2023-03-03T17:51:18.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.432+0000 7f366d97a700 1 -- 172.21.15.7:0/183760937 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3668074bb0 con 0x7f3650003a70 2023-03-03T17:51:18.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.432+0000 7f364ffff700 1 -- 172.21.15.7:0/183760937 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3658053c20 con 0x7f3650003a70 2023-03-03T17:51:18.432 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:51:18.432 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:51:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.434+0000 7f366d97a700 1 -- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36540682f0 msgr2=0x7f365406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.434+0000 7f366d97a700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36540682f0 0x7f365406a7a0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f365c000c00 tx=0x7f365c001470 comp rx=0 tx=0).stop 2023-03-03T17:51:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.434+0000 7f366d97a700 1 -- 172.21.15.7:0/183760937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 msgr2=0x7f366814f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:51:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.434+0000 7f366d97a700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f366814f170 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f365800efd0 tx=0x7f36580047c0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.434+0000 7f366d97a700 1 -- 172.21.15.7:0/183760937 shutdown_connections 2023-03-03T17:51:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.434+0000 7f366d97a700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3668146790 0x7f3668150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.434+0000 7f366d97a700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f36540682f0 0x7f365406a7a0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.434+0000 7f366d97a700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3650003a70 0x7f366814f170 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.434+0000 7f366d97a700 1 --2- 172.21.15.7:0/183760937 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f36680c8540 0x7f3668141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:51:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.434+0000 7f366d97a700 1 -- 172.21.15.7:0/183760937 >> 172.21.15.7:0/183760937 conn(0x7f366806b980 msgr2=0x7f366814a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:51:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.435+0000 7f366d97a700 1 -- 172.21.15.7:0/183760937 shutdown_connections 2023-03-03T17:51:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:51:18.435+0000 7f366d97a700 1 -- 172.21.15.7:0/183760937 wait complete. 2023-03-03T17:51:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:51:18.446 INFO:tasks.mds_thrash.fs.[a]:waiting for 98 secs before thrashing 2023-03-03T17:51:31.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:51:31.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:01.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:01.853 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:14.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:52:15.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.074+0000 7f5a4b762700 1 Processor -- start 2023-03-03T17:52:15.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.075+0000 7f5a4b762700 1 -- start start 2023-03-03T17:52:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.075+0000 7f5a4b762700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5a4414f9d0 0x7f5a44156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.075+0000 7f5a4b762700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a4414fda0 0x7f5a44071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.075+0000 7f5a4b762700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a440c2f30 con 0x7f5a440abe20 2023-03-03T17:52:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.075+0000 7f5a4b762700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a440c31d0 con 0x7f5a4414fda0 2023-03-03T17:52:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.075+0000 7f5a4b762700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a441524a0 con 0x7f5a4414f9d0 2023-03-03T17:52:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.075+0000 7f5a48cfd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5a4414fda0 0x7f5a44071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a49cff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5a440abe20 0x7f5a441529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57122/0 (socket says 172.21.15.7:57122) 2023-03-03T17:52:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a49cff700 1 -- 172.21.15.7:0/3856834735 learned_addr learned my addr 172.21.15.7:0/3856834735 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:52:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a494fe700 1 --2- 172.21.15.7:0/3856834735 >> v2:172.21.15.143:3300/0 conn(0x7f5a4414f9d0 0x7f5a44156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a48cfd700 1 -- 172.21.15.7:0/3856834735 >> v2:172.21.15.143:3300/0 conn(0x7f5a4414f9d0 msgr2=0x7f5a44156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:52:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a48cfd700 1 --2- 172.21.15.7:0/3856834735 >> v2:172.21.15.143:3300/0 conn(0x7f5a4414f9d0 0x7f5a44156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a48cfd700 1 -- 172.21.15.7:0/3856834735 >> v1:172.21.15.7:6789/0 conn(0x7f5a440abe20 legacy=0x7f5a441529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a48cfd700 1 -- 172.21.15.7:0/3856834735 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a440c34b0 con 0x7f5a4414fda0 2023-03-03T17:52:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a37fff700 1 -- 172.21.15.7:0/3856834735 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367548074 0 0) 0x7f5a440c2f30 con 0x7f5a440abe20 2023-03-03T17:52:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a494fe700 1 --2- 172.21.15.7:0/3856834735 >> v2:172.21.15.143:3300/0 conn(0x7f5a4414f9d0 0x7f5a44156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:52:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a48cfd700 1 --2- 172.21.15.7:0/3856834735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414fda0 0x7f5a44071890 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f5a440710a0 tx=0x7f5a38009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e63805df374ff0f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:52:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a37fff700 1 -- 172.21.15.7:0/3856834735 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38011830 con 0x7f5a4414fda0 2023-03-03T17:52:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a37fff700 1 -- 172.21.15.7:0/3856834735 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a38011990 con 0x7f5a4414fda0 2023-03-03T17:52:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.076+0000 7f5a37fff700 1 -- 172.21.15.7:0/3856834735 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38011d20 con 0x7f5a4414fda0 2023-03-03T17:52:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.077+0000 7f5a4b762700 1 -- 172.21.15.7:0/3856834735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414fda0 msgr2=0x7f5a44071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.077+0000 7f5a4b762700 1 --2- 172.21.15.7:0/3856834735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414fda0 0x7f5a44071890 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f5a440710a0 tx=0x7f5a38009b20 comp rx=0 tx=0).stop 2023-03-03T17:52:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.077+0000 7f5a4b762700 1 -- 172.21.15.7:0/3856834735 shutdown_connections 2023-03-03T17:52:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.077+0000 7f5a4b762700 1 --2- 172.21.15.7:0/3856834735 >> v2:172.21.15.143:3300/0 conn(0x7f5a4414f9d0 0x7f5a44156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:15.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.077+0000 7f5a4b762700 1 --2- 172.21.15.7:0/3856834735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414fda0 0x7f5a44071890 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:15.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.077+0000 7f5a4b762700 1 -- 172.21.15.7:0/3856834735 >> 172.21.15.7:0/3856834735 conn(0x7f5a440c8900 msgr2=0x7f5a440c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:52:15.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.077+0000 7f5a4b762700 1 -- 172.21.15.7:0/3856834735 shutdown_connections 2023-03-03T17:52:15.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.077+0000 7f5a4b762700 1 -- 172.21.15.7:0/3856834735 wait complete. 2023-03-03T17:52:15.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.078+0000 7f5a4b762700 1 Processor -- start 2023-03-03T17:52:15.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.078+0000 7f5a4b762700 1 -- start start 2023-03-03T17:52:15.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a4b762700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a440abe20 0x7f5a441450d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:15.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a4b762700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414f9d0 0x7f5a4406e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:15.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a4b762700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4414fda0 0x7f5a4406ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:15.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a4b762700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4406f110 con 0x7f5a4414f9d0 2023-03-03T17:52:15.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a4b762700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a441ed590 con 0x7f5a4414fda0 2023-03-03T17:52:15.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a4b762700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4406f960 con 0x7f5a440abe20 2023-03-03T17:52:15.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a48cfd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414f9d0 0x7f5a4406e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:15.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a48cfd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414f9d0 0x7f5a4406e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34914/0 (socket says 172.21.15.7:34914) 2023-03-03T17:52:15.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a48cfd700 1 -- 172.21.15.7:0/2702803095 learned_addr learned my addr 172.21.15.7:0/2702803095 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:52:15.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a49cff700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4414fda0 0x7f5a4406ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:15.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a494fe700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a440abe20 0x7f5a441450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:15.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a48cfd700 1 -- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a440abe20 msgr2=0x7f5a441450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:15.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a48cfd700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a440abe20 0x7f5a441450d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:15.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a48cfd700 1 -- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4414fda0 msgr2=0x7f5a4406ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:15.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a48cfd700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4414fda0 0x7f5a4406ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:15.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a48cfd700 1 -- 172.21.15.7:0/2702803095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a380094a0 con 0x7f5a4414f9d0 2023-03-03T17:52:15.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a48cfd700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414f9d0 0x7f5a4406e690 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f5a38000c00 tx=0x7f5a38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:52:15.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.079+0000 7f5a35ffb700 1 -- 172.21.15.7:0/2702803095 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a380038a0 con 0x7f5a4414f9d0 2023-03-03T17:52:15.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.080+0000 7f5a35ffb700 1 -- 172.21.15.7:0/2702803095 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3801f070 con 0x7f5a4414f9d0 2023-03-03T17:52:15.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.080+0000 7f5a4b762700 1 -- 172.21.15.7:0/2702803095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a440ac350 con 0x7f5a4414f9d0 2023-03-03T17:52:15.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.080+0000 7f5a35ffb700 1 -- 172.21.15.7:0/2702803095 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38003c60 con 0x7f5a4414f9d0 2023-03-03T17:52:15.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.080+0000 7f5a4b762700 1 -- 172.21.15.7:0/2702803095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a440ab630 con 0x7f5a4414f9d0 2023-03-03T17:52:15.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.081+0000 7f5a4b762700 1 -- 172.21.15.7:0/2702803095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a440c3e10 con 0x7f5a4414f9d0 2023-03-03T17:52:15.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.081+0000 7f5a35ffb700 1 -- 172.21.15.7:0/2702803095 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5a3800f050 con 0x7f5a4414f9d0 2023-03-03T17:52:15.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.082+0000 7f5a35ffb700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a280682f0 0x7f5a2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:15.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.082+0000 7f5a35ffb700 1 -- 172.21.15.7:0/2702803095 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5a38021070 con 0x7f5a4414f9d0 2023-03-03T17:52:15.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.082+0000 7f5a494fe700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a280682f0 0x7f5a2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:15.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.083+0000 7f5a494fe700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a280682f0 0x7f5a2806a7a0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f5a40006fd0 tx=0x7f5a40008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:52:15.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.086+0000 7f5a35ffb700 1 -- 172.21.15.7:0/2702803095 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a38053e60 con 0x7f5a4414f9d0 2023-03-03T17:52:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.322+0000 7f5a4b762700 1 -- 172.21.15.7:0/2702803095 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a44074bb0 con 0x7f5a4414f9d0 2023-03-03T17:52:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.322+0000 7f5a35ffb700 1 -- 172.21.15.7:0/2702803095 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5a38057480 con 0x7f5a4414f9d0 2023-03-03T17:52:15.322 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:52:15.322 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:52:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 -- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a280682f0 msgr2=0x7f5a2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a280682f0 0x7f5a2806a7a0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f5a40006fd0 tx=0x7f5a40008040 comp rx=0 tx=0).stop 2023-03-03T17:52:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 -- 172.21.15.7:0/2702803095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414f9d0 msgr2=0x7f5a4406e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414f9d0 0x7f5a4406e690 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f5a38000c00 tx=0x7f5a38004750 comp rx=0 tx=0).stop 2023-03-03T17:52:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 -- 172.21.15.7:0/2702803095 shutdown_connections 2023-03-03T17:52:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a4414fda0 0x7f5a4406ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a440abe20 0x7f5a441450d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a280682f0 0x7f5a2806a7a0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 --2- 172.21.15.7:0/2702803095 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a4414f9d0 0x7f5a4406e690 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 -- 172.21.15.7:0/2702803095 >> 172.21.15.7:0/2702803095 conn(0x7f5a440c8900 msgr2=0x7f5a44155490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:52:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 -- 172.21.15.7:0/2702803095 shutdown_connections 2023-03-03T17:52:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:15.325+0000 7f5a4b762700 1 -- 172.21.15.7:0/2702803095 wait complete. 2023-03-03T17:52:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:52:15.336 INFO:tasks.mds_thrash.fs.[b]:waiting for 59 secs before thrashing 2023-03-03T17:52:31.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:31.983 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:52:56.447 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:52:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.587+0000 7f6737732700 1 Processor -- start 2023-03-03T17:52:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.588+0000 7f6737732700 1 -- start start 2023-03-03T17:52:56.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.588+0000 7f6737732700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6730155410 0x7f67300bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:56.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.588+0000 7f6737732700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67300c08d0 con 0x7f6730151e80 2023-03-03T17:52:56.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.588+0000 7f6737732700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6730154930 con 0x7f67300c5b00 2023-03-03T17:52:56.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.588+0000 7f6737732700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6730152250 con 0x7f6730155410 2023-03-03T17:52:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f6734ccd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6730155410 0x7f67300bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f67354ce700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6730151e80 0x7f67300c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57170/0 (socket says 172.21.15.7:57170) 2023-03-03T17:52:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f67354ce700 1 -- 172.21.15.7:0/1842670527 learned_addr learned my addr 172.21.15.7:0/1842670527 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:52:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f6734ccd700 1 -- 172.21.15.7:0/1842670527 >> v1:172.21.15.143:6789/0 conn(0x7f67300c5b00 legacy=0x7f67300bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f6734ccd700 1 -- 172.21.15.7:0/1842670527 >> v1:172.21.15.7:6789/0 conn(0x7f6730151e80 legacy=0x7f67300c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f6734ccd700 1 -- 172.21.15.7:0/1842670527 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6730154c10 con 0x7f6730155410 2023-03-03T17:52:56.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f6723fff700 1 -- 172.21.15.7:0/1842670527 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037260639 0 0) 0x7f67300c08d0 con 0x7f6730151e80 2023-03-03T17:52:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f6734ccd700 1 --2- 172.21.15.7:0/1842670527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730155410 0x7f67300bf8e0 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7f6724009840 tx=0x7f6724009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422dca200fac17bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:52:56.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f6723fff700 1 -- 172.21.15.7:0/1842670527 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724011830 con 0x7f6730155410 2023-03-03T17:52:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f6723fff700 1 -- 172.21.15.7:0/1842670527 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6724011990 con 0x7f6730155410 2023-03-03T17:52:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.589+0000 7f6723fff700 1 -- 172.21.15.7:0/1842670527 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724011d20 con 0x7f6730155410 2023-03-03T17:52:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.590+0000 7f6737732700 1 -- 172.21.15.7:0/1842670527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730155410 msgr2=0x7f67300bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.590+0000 7f6737732700 1 --2- 172.21.15.7:0/1842670527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730155410 0x7f67300bf8e0 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7f6724009840 tx=0x7f6724009b50 comp rx=0 tx=0).stop 2023-03-03T17:52:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.590+0000 7f6737732700 1 -- 172.21.15.7:0/1842670527 shutdown_connections 2023-03-03T17:52:56.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.590+0000 7f6737732700 1 --2- 172.21.15.7:0/1842670527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730155410 0x7f67300bf8e0 unknown :-1 s=CLOSED pgs=3043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.590+0000 7f6737732700 1 -- 172.21.15.7:0/1842670527 >> 172.21.15.7:0/1842670527 conn(0x7f6730071630 msgr2=0x7f673006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:52:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.590+0000 7f6737732700 1 -- 172.21.15.7:0/1842670527 shutdown_connections 2023-03-03T17:52:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.590+0000 7f6737732700 1 -- 172.21.15.7:0/1842670527 wait complete. 2023-03-03T17:52:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.591+0000 7f6737732700 1 Processor -- start 2023-03-03T17:52:56.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.591+0000 7f6737732700 1 -- start start 2023-03-03T17:52:56.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6737732700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67300c5b00 0x7f67301565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:56.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6737732700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730151e80 0x7f67300ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:56.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6737732700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6730155410 0x7f67300c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:56.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6737732700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67300ca6d0 con 0x7f6730151e80 2023-03-03T17:52:56.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6737732700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6730156110 con 0x7f67300c5b00 2023-03-03T17:52:56.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6737732700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6730155b50 con 0x7f6730155410 2023-03-03T17:52:56.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6734ccd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730151e80 0x7f67300ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:56.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6734ccd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730151e80 0x7f67300ca190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34966/0 (socket says 172.21.15.7:34966) 2023-03-03T17:52:56.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6734ccd700 1 -- 172.21.15.7:0/2118086128 learned_addr learned my addr 172.21.15.7:0/2118086128 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:52:56.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f67354ce700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67300c5b00 0x7f67301565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:56.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6735ccf700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6730155410 0x7f67300c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:56.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6734ccd700 1 -- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6730155410 msgr2=0x7f67300c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:56.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6734ccd700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6730155410 0x7f67300c9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:56.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6734ccd700 1 -- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67300c5b00 msgr2=0x7f67301565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:56.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6734ccd700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67300c5b00 0x7f67301565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:56.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6734ccd700 1 -- 172.21.15.7:0/2118086128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67240094a0 con 0x7f6730151e80 2023-03-03T17:52:56.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.592+0000 7f6735ccf700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6730155410 0x7f67300c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:52:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.593+0000 7f6734ccd700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730151e80 0x7f67300ca190 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f67240090b0 tx=0x7f6724004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:52:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.593+0000 7f6721ffb700 1 -- 172.21.15.7:0/2118086128 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672401a850 con 0x7f6730151e80 2023-03-03T17:52:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.593+0000 7f6721ffb700 1 -- 172.21.15.7:0/2118086128 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672401a9b0 con 0x7f6730151e80 2023-03-03T17:52:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.593+0000 7f6737732700 1 -- 172.21.15.7:0/2118086128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67300c5220 con 0x7f6730151e80 2023-03-03T17:52:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.593+0000 7f6721ffb700 1 -- 172.21.15.7:0/2118086128 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672401ad20 con 0x7f6730151e80 2023-03-03T17:52:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.593+0000 7f6737732700 1 -- 172.21.15.7:0/2118086128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6730154ef0 con 0x7f6730151e80 2023-03-03T17:52:56.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.594+0000 7f6737732700 1 -- 172.21.15.7:0/2118086128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67300c0530 con 0x7f6730151e80 2023-03-03T17:52:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.595+0000 7f6721ffb700 1 -- 172.21.15.7:0/2118086128 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f672402b030 con 0x7f6730151e80 2023-03-03T17:52:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.595+0000 7f6721ffb700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67140682f0 0x7f671406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:52:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.595+0000 7f6721ffb700 1 -- 172.21.15.7:0/2118086128 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6724087520 con 0x7f6730151e80 2023-03-03T17:52:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.595+0000 7f67354ce700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67140682f0 0x7f671406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:52:56.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.596+0000 7f67354ce700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67140682f0 0x7f671406a7a0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f672c007b10 tx=0x7f672c007a60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:52:56.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.600+0000 7f6721ffb700 1 -- 172.21.15.7:0/2118086128 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f67240538c0 con 0x7f6730151e80 2023-03-03T17:52:56.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.835+0000 7f6737732700 1 -- 172.21.15.7:0/2118086128 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6730074bb0 con 0x7f6730151e80 2023-03-03T17:52:56.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.836+0000 7f6721ffb700 1 -- 172.21.15.7:0/2118086128 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6724056ee0 con 0x7f6730151e80 2023-03-03T17:52:56.835 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:52:56.836 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:52:56.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.837+0000 7f6737732700 1 -- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67140682f0 msgr2=0x7f671406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:56.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.837+0000 7f6737732700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67140682f0 0x7f671406a7a0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f672c007b10 tx=0x7f672c007a60 comp rx=0 tx=0).stop 2023-03-03T17:52:56.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.837+0000 7f6737732700 1 -- 172.21.15.7:0/2118086128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730151e80 msgr2=0x7f67300ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:52:56.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.837+0000 7f6737732700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730151e80 0x7f67300ca190 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f67240090b0 tx=0x7f6724004750 comp rx=0 tx=0).stop 2023-03-03T17:52:56.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.838+0000 7f6737732700 1 -- 172.21.15.7:0/2118086128 shutdown_connections 2023-03-03T17:52:56.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.838+0000 7f6737732700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6730155410 0x7f67300c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:56.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.838+0000 7f6737732700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67300c5b00 0x7f67301565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:56.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.838+0000 7f6737732700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67140682f0 0x7f671406a7a0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:56.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.838+0000 7f6737732700 1 --2- 172.21.15.7:0/2118086128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6730151e80 0x7f67300ca190 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:52:56.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.838+0000 7f6737732700 1 -- 172.21.15.7:0/2118086128 >> 172.21.15.7:0/2118086128 conn(0x7f6730071630 msgr2=0x7f67300bf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:52:56.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.838+0000 7f6737732700 1 -- 172.21.15.7:0/2118086128 shutdown_connections 2023-03-03T17:52:56.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:52:56.838+0000 7f6737732700 1 -- 172.21.15.7:0/2118086128 wait complete. 2023-03-03T17:52:56.840 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:52:56.849 INFO:tasks.mds_thrash.fs.[a]:waiting for 60 secs before thrashing 2023-03-03T17:53:02.135 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:02.138 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:14.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:53:14.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.477+0000 7fc8e1347700 1 Processor -- start 2023-03-03T17:53:14.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.477+0000 7fc8e1347700 1 -- start start 2023-03-03T17:53:14.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.478+0000 7fc8e1347700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc8dc0aa030 0x7fc8dc158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:14.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.478+0000 7fc8e1347700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc0a9c20 con 0x7fc8dc0aad20 2023-03-03T17:53:14.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.478+0000 7fc8e1347700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8dc154f10 con 0x7fc8dc14b0d0 2023-03-03T17:53:14.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.478+0000 7fc8e1347700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc0ac3c0 con 0x7fc8dc0aa030 2023-03-03T17:53:14.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.478+0000 7fc8da7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc8dc0aad20 0x7fc8dc157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57196/0 (socket says 172.21.15.7:57196) 2023-03-03T17:53:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.478+0000 7fc8da7fc700 1 -- 172.21.15.7:0/2410601629 learned_addr learned my addr 172.21.15.7:0/2410601629 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:53:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.478+0000 7fc8db7fe700 1 --2- 172.21.15.7:0/2410601629 >> v2:172.21.15.143:3300/0 conn(0x7fc8dc0aa030 0x7fc8dc158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.478+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433763106 0 0) 0x7fc8dc0a9c20 con 0x7fc8dc0aad20 2023-03-03T17:53:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8c80035b0 con 0x7fc8dc0aad20 2023-03-03T17:53:14.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 409789012 0 0) 0x7fc8c80035b0 con 0x7fc8dc0aad20 2023-03-03T17:53:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 >> v2:172.21.15.143:3300/0 conn(0x7fc8dc0aa030 msgr2=0x7fc8dc158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:53:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 --2- 172.21.15.7:0/2410601629 >> v2:172.21.15.143:3300/0 conn(0x7fc8dc0aa030 0x7fc8dc158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 >> v1:172.21.15.143:6789/0 conn(0x7fc8dc14b0d0 legacy=0x7fc8dc071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc06fc70 con 0x7fc8dc0aad20 2023-03-03T17:53:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc8c4001670 con 0x7fc8dc0aad20 2023-03-03T17:53:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 >> v1:172.21.15.7:6789/0 conn(0x7fc8dc0aad20 legacy=0x7fc8dc157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:14.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8003a70 0x7fc8c8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c8004850 0x7fc8c8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8c8005200 0x7fc8c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc0ac3c0 con 0x7fc8c8004850 2023-03-03T17:53:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c800c110 con 0x7fc8c8005200 2023-03-03T17:53:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c800c2d0 con 0x7fc8c8003a70 2023-03-03T17:53:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8da7fc700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c8004850 0x7fc8c8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:14.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8db7fe700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8c8005200 0x7fc8c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8daffd700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8003a70 0x7fc8c8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8da7fc700 1 -- 172.21.15.7:0/2410601629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8003a70 msgr2=0x7fc8c8003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:53:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8da7fc700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8003a70 0x7fc8c8003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8da7fc700 1 -- 172.21.15.7:0/2410601629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8c8005200 msgr2=0x7fc8c800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:53:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8da7fc700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8c8005200 0x7fc8c800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8db7fe700 1 -- 172.21.15.7:0/2410601629 reap_dead start 2023-03-03T17:53:14.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8da7fc700 1 -- 172.21.15.7:0/2410601629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c80035b0 con 0x7fc8c8004850 2023-03-03T17:53:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.479+0000 7fc8daffd700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8003a70 0x7fc8c8003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:53:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8da7fc700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c8004850 0x7fc8c8004cc0 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0ab1a0 tx=0x7fc8c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112b535fa2304b5e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:53:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c400cd00 con 0x7fc8c8004850 2023-03-03T17:53:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c400ce80 con 0x7fc8c8004850 2023-03-03T17:53:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8d9ffb700 1 -- 172.21.15.7:0/2410601629 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4005ba0 con 0x7fc8c8004850 2023-03-03T17:53:14.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8e1347700 1 -- 172.21.15.7:0/2410601629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c8004850 msgr2=0x7fc8c8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8e1347700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c8004850 0x7fc8c8004cc0 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0ab1a0 tx=0x7fc8c8005880 comp rx=0 tx=0).stop 2023-03-03T17:53:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8e1347700 1 -- 172.21.15.7:0/2410601629 shutdown_connections 2023-03-03T17:53:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8e1347700 1 --2- 172.21.15.7:0/2410601629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8c8004850 0x7fc8c8004cc0 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8e1347700 1 -- 172.21.15.7:0/2410601629 >> 172.21.15.7:0/2410601629 conn(0x7fc8dc0c8900 msgr2=0x7fc8dc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:53:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8e1347700 1 -- 172.21.15.7:0/2410601629 shutdown_connections 2023-03-03T17:53:14.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.480+0000 7fc8e1347700 1 -- 172.21.15.7:0/2410601629 wait complete. 2023-03-03T17:53:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.482+0000 7fc8e1347700 1 Processor -- start 2023-03-03T17:53:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.482+0000 7fc8e1347700 1 -- start start 2023-03-03T17:53:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8e1347700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8004850 0x7fc8dc14eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8e1347700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc154340 0x7fc8dc14e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8e1347700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8dc14d100 0x7fc8dc14d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:14.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8e1347700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc151770 con 0x7fc8dc14d100 2023-03-03T17:53:14.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8e1347700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc14f000 con 0x7fc8dc154340 2023-03-03T17:53:14.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8e1347700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc152b80 con 0x7fc8c8004850 2023-03-03T17:53:14.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8db7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8dc14d100 0x7fc8dc14d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:14.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8db7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8dc14d100 0x7fc8dc14d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34998/0 (socket says 172.21.15.7:34998) 2023-03-03T17:53:14.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8db7fe700 1 -- 172.21.15.7:0/2261743332 learned_addr learned my addr 172.21.15.7:0/2261743332 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:53:14.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8da7fc700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc154340 0x7fc8dc14e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:14.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8daffd700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8004850 0x7fc8dc14eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:14.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8db7fe700 1 -- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8004850 msgr2=0x7fc8dc14eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:14.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8db7fe700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8004850 0x7fc8dc14eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:14.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8db7fe700 1 -- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc154340 msgr2=0x7fc8dc14e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:14.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.483+0000 7fc8db7fe700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc154340 0x7fc8dc14e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:14.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.484+0000 7fc8db7fe700 1 -- 172.21.15.7:0/2261743332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc06f6b0 con 0x7fc8dc14d100 2023-03-03T17:53:14.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.484+0000 7fc8da7fc700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc154340 0x7fc8dc14e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:53:14.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.484+0000 7fc8daffd700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8004850 0x7fc8dc14eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:53:14.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.484+0000 7fc8db7fe700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8dc14d100 0x7fc8dc14d530 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc072a60 tx=0x7fc8d4011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:53:14.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.484+0000 7fc8d37fe700 1 -- 172.21.15.7:0/2261743332 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d400e040 con 0x7fc8dc14d100 2023-03-03T17:53:14.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.484+0000 7fc8d37fe700 1 -- 172.21.15.7:0/2261743332 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d400f040 con 0x7fc8dc14d100 2023-03-03T17:53:14.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.484+0000 7fc8e1347700 1 -- 172.21.15.7:0/2261743332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc155160 con 0x7fc8dc14d100 2023-03-03T17:53:14.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.484+0000 7fc8d37fe700 1 -- 172.21.15.7:0/2261743332 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d400fd80 con 0x7fc8dc14d100 2023-03-03T17:53:14.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.484+0000 7fc8e1347700 1 -- 172.21.15.7:0/2261743332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc06f990 con 0x7fc8dc14d100 2023-03-03T17:53:14.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.485+0000 7fc8e1347700 1 -- 172.21.15.7:0/2261743332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc074bb0 con 0x7fc8dc14d100 2023-03-03T17:53:14.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.485+0000 7fc8d37fe700 1 -- 172.21.15.7:0/2261743332 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc8d4022460 con 0x7fc8dc14d100 2023-03-03T17:53:14.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.486+0000 7fc8d37fe700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8bc068320 0x7fc8bc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:14.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.486+0000 7fc8d37fe700 1 -- 172.21.15.7:0/2261743332 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc8d40868d0 con 0x7fc8dc14d100 2023-03-03T17:53:14.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.486+0000 7fc8daffd700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8bc068320 0x7fc8bc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:14.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.486+0000 7fc8daffd700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8bc068320 0x7fc8bc06a7d0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc04f440 tx=0x7fc8cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:53:14.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.490+0000 7fc8d37fe700 1 -- 172.21.15.7:0/2261743332 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc8d4052c10 con 0x7fc8dc14d100 2023-03-03T17:53:14.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.727+0000 7fc8e1347700 1 -- 172.21.15.7:0/2261743332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8dc0aa540 con 0x7fc8dc14d100 2023-03-03T17:53:14.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.727+0000 7fc8d37fe700 1 -- 172.21.15.7:0/2261743332 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc8d4056230 con 0x7fc8dc14d100 2023-03-03T17:53:14.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:53:14.727 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:53:14.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 -- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8bc068320 msgr2=0x7fc8bc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:14.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8bc068320 0x7fc8bc06a7d0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc04f440 tx=0x7fc8cc009040 comp rx=0 tx=0).stop 2023-03-03T17:53:14.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 -- 172.21.15.7:0/2261743332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8dc14d100 msgr2=0x7fc8dc14d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:14.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8dc14d100 0x7fc8dc14d530 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc072a60 tx=0x7fc8d4011f80 comp rx=0 tx=0).stop 2023-03-03T17:53:14.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 -- 172.21.15.7:0/2261743332 shutdown_connections 2023-03-03T17:53:14.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8bc068320 0x7fc8bc06a7d0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:14.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8dc14d100 0x7fc8dc14d530 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:14.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8c8004850 0x7fc8dc14eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:14.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 --2- 172.21.15.7:0/2261743332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc154340 0x7fc8dc14e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:14.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 -- 172.21.15.7:0/2261743332 >> 172.21.15.7:0/2261743332 conn(0x7fc8dc0c8900 msgr2=0x7fc8dc148140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:53:14.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 -- 172.21.15.7:0/2261743332 shutdown_connections 2023-03-03T17:53:14.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:14.729+0000 7fc8e1347700 1 -- 172.21.15.7:0/2261743332 wait complete. 2023-03-03T17:53:14.731 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:53:14.741 INFO:tasks.mds_thrash.fs.[b]:waiting for 46 secs before thrashing 2023-03-03T17:53:32.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:32.234 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:53:56.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:53:56.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.990+0000 7fb32786d700 1 Processor -- start 2023-03-03T17:53:56.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.990+0000 7fb32786d700 1 -- start start 2023-03-03T17:53:56.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb32786d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb32006d870 0x7fb3201550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb32786d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb3200c1da0 0x7fb3200c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb32786d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3200c3620 con 0x7fb3200c3e30 2023-03-03T17:53:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb32786d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3200c2af0 con 0x7fb3200c1da0 2023-03-03T17:53:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb32786d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3201545e0 con 0x7fb32006d870 2023-03-03T17:53:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb324e08700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb3200c1da0 0x7fb3200c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb325e0a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb3200c3e30 0x7fb320157ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57242/0 (socket says 172.21.15.7:57242) 2023-03-03T17:53:56.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb325e0a700 1 -- 172.21.15.7:0/1094990051 learned_addr learned my addr 172.21.15.7:0/1094990051 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:53:56.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb325609700 1 --2- 172.21.15.7:0/1094990051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb32006d870 0x7fb3201550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:56.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb324e08700 1 -- 172.21.15.7:0/1094990051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb32006d870 msgr2=0x7fb3201550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:53:56.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb324e08700 1 --2- 172.21.15.7:0/1094990051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb32006d870 0x7fb3201550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:56.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb313fff700 1 -- 172.21.15.7:0/1094990051 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040878485 0 0) 0x7fb3200c3620 con 0x7fb3200c3e30 2023-03-03T17:53:56.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb324e08700 1 -- 172.21.15.7:0/1094990051 >> v1:172.21.15.7:6789/0 conn(0x7fb3200c3e30 legacy=0x7fb320157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:56.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb324e08700 1 -- 172.21.15.7:0/1094990051 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3201548c0 con 0x7fb3200c1da0 2023-03-03T17:53:56.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb324e08700 1 --2- 172.21.15.7:0/1094990051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c1da0 0x7fb3200c7350 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7fb314009840 tx=0x7fb314009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e2fe1b99a64419 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:53:56.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb313fff700 1 -- 172.21.15.7:0/1094990051 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314011830 con 0x7fb3200c1da0 2023-03-03T17:53:56.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.991+0000 7fb313fff700 1 -- 172.21.15.7:0/1094990051 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb314011990 con 0x7fb3200c1da0 2023-03-03T17:53:56.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.992+0000 7fb313fff700 1 -- 172.21.15.7:0/1094990051 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314011d20 con 0x7fb3200c1da0 2023-03-03T17:53:56.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.992+0000 7fb32786d700 1 -- 172.21.15.7:0/1094990051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c1da0 msgr2=0x7fb3200c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:56.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.992+0000 7fb32786d700 1 --2- 172.21.15.7:0/1094990051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c1da0 0x7fb3200c7350 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7fb314009840 tx=0x7fb314009b50 comp rx=0 tx=0).stop 2023-03-03T17:53:56.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.992+0000 7fb32786d700 1 -- 172.21.15.7:0/1094990051 shutdown_connections 2023-03-03T17:53:56.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.992+0000 7fb32786d700 1 --2- 172.21.15.7:0/1094990051 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb32006d870 0x7fb3201550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:56.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.992+0000 7fb32786d700 1 --2- 172.21.15.7:0/1094990051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c1da0 0x7fb3200c7350 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.992+0000 7fb32786d700 1 -- 172.21.15.7:0/1094990051 >> 172.21.15.7:0/1094990051 conn(0x7fb32006c110 msgr2=0x7fb32006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:53:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.992+0000 7fb32786d700 1 -- 172.21.15.7:0/1094990051 shutdown_connections 2023-03-03T17:53:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.992+0000 7fb32786d700 1 -- 172.21.15.7:0/1094990051 wait complete. 2023-03-03T17:53:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.994+0000 7fb32786d700 1 Processor -- start 2023-03-03T17:53:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.994+0000 7fb32786d700 1 -- start start 2023-03-03T17:53:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb32786d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32006d870 0x7fb32014eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb32786d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3200c1da0 0x7fb320147250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:56.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb32786d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c3e30 0x7fb3201ecd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:56.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb32786d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb320150880 con 0x7fb3200c3e30 2023-03-03T17:53:56.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb32786d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3201581f0 con 0x7fb32006d870 2023-03-03T17:53:56.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb32786d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3200718e0 con 0x7fb3200c1da0 2023-03-03T17:53:56.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325e0a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c3e30 0x7fb3201ecd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:56.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325e0a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c3e30 0x7fb3201ecd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35036/0 (socket says 172.21.15.7:35036) 2023-03-03T17:53:56.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325e0a700 1 -- 172.21.15.7:0/3286950381 learned_addr learned my addr 172.21.15.7:0/3286950381 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:53:56.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325609700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32006d870 0x7fb32014eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:56.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb324e08700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3200c1da0 0x7fb320147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:56.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325e0a700 1 -- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3200c1da0 msgr2=0x7fb320147250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:56.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325e0a700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3200c1da0 0x7fb320147250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:56.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325e0a700 1 -- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32006d870 msgr2=0x7fb32014eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:56.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325e0a700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32006d870 0x7fb32014eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:56.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325e0a700 1 -- 172.21.15.7:0/3286950381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3140094a0 con 0x7fb3200c3e30 2023-03-03T17:53:56.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325609700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32006d870 0x7fb32014eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:53:56.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb324e08700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3200c1da0 0x7fb320147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:53:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.995+0000 7fb325e0a700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c3e30 0x7fb3201ecd20 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7fb31800cba0 tx=0x7fb31800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:53:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.996+0000 7fb311ffb700 1 -- 172.21.15.7:0/3286950381 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31800ab80 con 0x7fb3200c3e30 2023-03-03T17:53:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.996+0000 7fb311ffb700 1 -- 172.21.15.7:0/3286950381 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31800ad00 con 0x7fb3200c3e30 2023-03-03T17:53:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.996+0000 7fb32786d700 1 -- 172.21.15.7:0/3286950381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32006dc40 con 0x7fb3200c3e30 2023-03-03T17:53:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.996+0000 7fb311ffb700 1 -- 172.21.15.7:0/3286950381 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb318007310 con 0x7fb3200c3e30 2023-03-03T17:53:56.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.996+0000 7fb32786d700 1 -- 172.21.15.7:0/3286950381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb320154ba0 con 0x7fb3200c3e30 2023-03-03T17:53:57.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.997+0000 7fb32786d700 1 -- 172.21.15.7:0/3286950381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3200c38e0 con 0x7fb3200c3e30 2023-03-03T17:53:57.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.997+0000 7fb311ffb700 1 -- 172.21.15.7:0/3286950381 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb318010070 con 0x7fb3200c3e30 2023-03-03T17:53:57.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.998+0000 7fb311ffb700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3040682f0 0x7fb30406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:53:57.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.998+0000 7fb311ffb700 1 -- 172.21.15.7:0/3286950381 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb318085b20 con 0x7fb3200c3e30 2023-03-03T17:53:57.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.998+0000 7fb325609700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3040682f0 0x7fb30406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:53:57.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:56.998+0000 7fb325609700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3040682f0 0x7fb30406a7a0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fb320072400 tx=0x7fb31c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:53:57.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.002+0000 7fb311ffb700 1 -- 172.21.15.7:0/3286950381 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb318051e60 con 0x7fb3200c3e30 2023-03-03T17:53:57.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.237+0000 7fb32786d700 1 -- 172.21.15.7:0/3286950381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb320146f70 con 0x7fb3200c3e30 2023-03-03T17:53:57.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.238+0000 7fb311ffb700 1 -- 172.21.15.7:0/3286950381 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb320146f70 con 0x7fb3200c3e30 2023-03-03T17:53:57.237 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:53:57.238 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:53:57.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 -- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3040682f0 msgr2=0x7fb30406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:57.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3040682f0 0x7fb30406a7a0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fb320072400 tx=0x7fb31c008040 comp rx=0 tx=0).stop 2023-03-03T17:53:57.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 -- 172.21.15.7:0/3286950381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c3e30 msgr2=0x7fb3201ecd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:53:57.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c3e30 0x7fb3201ecd20 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7fb31800cba0 tx=0x7fb31800cf60 comp rx=0 tx=0).stop 2023-03-03T17:53:57.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 -- 172.21.15.7:0/3286950381 shutdown_connections 2023-03-03T17:53:57.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb3040682f0 0x7fb30406a7a0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:57.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb3200c3e30 0x7fb3201ecd20 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:57.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32006d870 0x7fb32014eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:57.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 --2- 172.21.15.7:0/3286950381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb3200c1da0 0x7fb320147250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:53:57.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 -- 172.21.15.7:0/3286950381 >> 172.21.15.7:0/3286950381 conn(0x7fb32006c110 msgr2=0x7fb320153d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:53:57.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 -- 172.21.15.7:0/3286950381 shutdown_connections 2023-03-03T17:53:57.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:53:57.240+0000 7fb32786d700 1 -- 172.21.15.7:0/3286950381 wait complete. 2023-03-03T17:53:57.241 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:53:57.251 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T17:54:00.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:54:00.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.883+0000 7f8509b54700 1 Processor -- start 2023-03-03T17:54:00.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.883+0000 7f8509b54700 1 -- start start 2023-03-03T17:54:00.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.883+0000 7f8509b54700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85040c1da0 0x7f85041550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:00.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.883+0000 7f8509b54700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c3e30 0x7f85040c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.883+0000 7f8509b54700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85040c7a60 con 0x7f850406d870 2023-03-03T17:54:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.883+0000 7f8509b54700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85040c2af0 con 0x7f85040c1da0 2023-03-03T17:54:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.883+0000 7f8509b54700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85040c3620 con 0x7f85040c3e30 2023-03-03T17:54:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f8502ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85040c1da0 0x7f85041550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f8502ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85040c1da0 0x7f85041550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35046/0 (socket says 172.21.15.7:35046) 2023-03-03T17:54:00.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f8502ffd700 1 -- 172.21.15.7:0/3328467573 learned_addr learned my addr 172.21.15.7:0/3328467573 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:54:00.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f8503fff700 1 --2- 172.21.15.7:0/3328467573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c3e30 0x7f85040c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:00.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f8502ffd700 1 -- 172.21.15.7:0/3328467573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c3e30 msgr2=0x7f85040c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:54:00.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f8502ffd700 1 --2- 172.21.15.7:0/3328467573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c3e30 0x7f85040c4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:00.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f8502ffd700 1 -- 172.21.15.7:0/3328467573 >> v1:172.21.15.143:6789/0 conn(0x7f850406d870 legacy=0x7f85040c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:54:00.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f8502ffd700 1 -- 172.21.15.7:0/3328467573 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85041545e0 con 0x7f85040c1da0 2023-03-03T17:54:00.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f8503fff700 1 --2- 172.21.15.7:0/3328467573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c3e30 0x7f85040c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T17:54:00.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f8502ffd700 1 --2- 172.21.15.7:0/3328467573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c1da0 0x7f85041550c0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f84f80098f0 tx=0x7f84f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4696e8fda0da60c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:54:00.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f85027fc700 1 -- 172.21.15.7:0/3328467573 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f8004230 con 0x7f85040c1da0 2023-03-03T17:54:00.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f85027fc700 1 -- 172.21.15.7:0/3328467573 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f8004390 con 0x7f85040c1da0 2023-03-03T17:54:00.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.884+0000 7f85027fc700 1 -- 172.21.15.7:0/3328467573 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f80036a0 con 0x7f85040c1da0 2023-03-03T17:54:00.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.885+0000 7f8509b54700 1 -- 172.21.15.7:0/3328467573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c1da0 msgr2=0x7f85041550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:54:00.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.885+0000 7f8509b54700 1 --2- 172.21.15.7:0/3328467573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c1da0 0x7f85041550c0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f84f80098f0 tx=0x7f84f8009c00 comp rx=0 tx=0).stop 2023-03-03T17:54:00.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.885+0000 7f8509b54700 1 -- 172.21.15.7:0/3328467573 shutdown_connections 2023-03-03T17:54:00.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.885+0000 7f8509b54700 1 --2- 172.21.15.7:0/3328467573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c3e30 0x7f85040c4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:00.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.885+0000 7f8509b54700 1 --2- 172.21.15.7:0/3328467573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c1da0 0x7f85041550c0 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:00.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.885+0000 7f8509b54700 1 -- 172.21.15.7:0/3328467573 >> 172.21.15.7:0/3328467573 conn(0x7f850406c110 msgr2=0x7f850406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:54:00.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.885+0000 7f8509b54700 1 -- 172.21.15.7:0/3328467573 shutdown_connections 2023-03-03T17:54:00.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.885+0000 7f8509b54700 1 -- 172.21.15.7:0/3328467573 wait complete. 2023-03-03T17:54:00.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.886+0000 7f8509b54700 1 Processor -- start 2023-03-03T17:54:00.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.887+0000 7f8509b54700 1 -- start start 2023-03-03T17:54:00.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.887+0000 7f8509b54700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850406d870 0x7f85040c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:00.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.887+0000 7f8509b54700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c1da0 0x7f8504156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:00.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.887+0000 7f8509b54700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c3e30 0x7f85041569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:00.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.887+0000 7f8509b54700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85040ab910 con 0x7f85040c3e30 2023-03-03T17:54:00.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.887+0000 7f8509b54700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85040abbf0 con 0x7f850406d870 2023-03-03T17:54:00.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.887+0000 7f8509b54700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85040abed0 con 0x7f85040c1da0 2023-03-03T17:54:00.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8503fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c3e30 0x7f85041569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:00.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8503fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c3e30 0x7f85041569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35054/0 (socket says 172.21.15.7:35054) 2023-03-03T17:54:00.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8503fff700 1 -- 172.21.15.7:0/765217266 learned_addr learned my addr 172.21.15.7:0/765217266 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:54:00.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8502ffd700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c1da0 0x7f8504156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:00.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f85037fe700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850406d870 0x7f85040c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:00.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8503fff700 1 -- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c1da0 msgr2=0x7f8504156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:54:00.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8503fff700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c1da0 0x7f8504156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:00.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8503fff700 1 -- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850406d870 msgr2=0x7f85040c6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:54:00.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8503fff700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850406d870 0x7f85040c6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:00.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8503fff700 1 -- 172.21.15.7:0/765217266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f8009550 con 0x7f85040c3e30 2023-03-03T17:54:00.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8503fff700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c3e30 0x7f85041569d0 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00cd50 tx=0x7f84fc00a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:54:00.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f84ebfff700 1 -- 172.21.15.7:0/765217266 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc00cea0 con 0x7f85040c3e30 2023-03-03T17:54:00.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f84ebfff700 1 -- 172.21.15.7:0/765217266 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc00f480 con 0x7f85040c3e30 2023-03-03T17:54:00.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8509b54700 1 -- 172.21.15.7:0/765217266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8504154ba0 con 0x7f85040c3e30 2023-03-03T17:54:00.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f84ebfff700 1 -- 172.21.15.7:0/765217266 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc00f830 con 0x7f85040c3e30 2023-03-03T17:54:00.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.888+0000 7f8509b54700 1 -- 172.21.15.7:0/765217266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85041548c0 con 0x7f85040c3e30 2023-03-03T17:54:00.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.889+0000 7f8509b54700 1 -- 172.21.15.7:0/765217266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850406dc40 con 0x7f85040c3e30 2023-03-03T17:54:00.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.890+0000 7f84ebfff700 1 -- 172.21.15.7:0/765217266 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f84fc009030 con 0x7f85040c3e30 2023-03-03T17:54:00.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.890+0000 7f84ebfff700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84f00682f0 0x7f84f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:00.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.890+0000 7f84ebfff700 1 -- 172.21.15.7:0/765217266 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f84fc0869a0 con 0x7f85040c3e30 2023-03-03T17:54:00.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.890+0000 7f85037fe700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84f00682f0 0x7f84f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:00.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.891+0000 7f85037fe700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84f00682f0 0x7f84f006a7a0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f8504151ed0 tx=0x7f84f4007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:54:00.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:00.895+0000 7f84ebfff700 1 -- 172.21.15.7:0/765217266 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f84fc052d40 con 0x7f85040c3e30 2023-03-03T17:54:01.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.133+0000 7f8509b54700 1 -- 172.21.15.7:0/765217266 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f850406ca10 con 0x7f85040c3e30 2023-03-03T17:54:01.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.134+0000 7f84ebfff700 1 -- 172.21.15.7:0/765217266 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f84fc056360 con 0x7f85040c3e30 2023-03-03T17:54:01.133 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:54:01.133 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:54:01.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 -- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84f00682f0 msgr2=0x7f84f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:54:01.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84f00682f0 0x7f84f006a7a0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f8504151ed0 tx=0x7f84f4007ec0 comp rx=0 tx=0).stop 2023-03-03T17:54:01.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 -- 172.21.15.7:0/765217266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c3e30 msgr2=0x7f85041569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:54:01.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c3e30 0x7f85041569d0 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00cd50 tx=0x7f84fc00a560 comp rx=0 tx=0).stop 2023-03-03T17:54:01.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 -- 172.21.15.7:0/765217266 shutdown_connections 2023-03-03T17:54:01.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84f00682f0 0x7f84f006a7a0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:01.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85040c3e30 0x7f85041569d0 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:01.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850406d870 0x7f85040c6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:01.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 --2- 172.21.15.7:0/765217266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85040c1da0 0x7f8504156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:01.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 -- 172.21.15.7:0/765217266 >> 172.21.15.7:0/765217266 conn(0x7f850406c110 msgr2=0x7f85040c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:54:01.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 -- 172.21.15.7:0/765217266 shutdown_connections 2023-03-03T17:54:01.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:01.136+0000 7f8509b54700 1 -- 172.21.15.7:0/765217266 wait complete. 2023-03-03T17:54:01.137 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:54:01.147 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-03-03T17:54:02.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:02.364 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:32.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:32.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:54:39.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:54:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.387+0000 7fb748b5d700 1 Processor -- start 2023-03-03T17:54:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.387+0000 7fb748b5d700 1 -- start start 2023-03-03T17:54:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb748b5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb744155390 0x7fb7440c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb748b5d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb74406e020 0x7fb744071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb748b5d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb744072520 0x7fb744151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb748b5d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb744070b90 con 0x7fb744072520 2023-03-03T17:54:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb748b5d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb744151320 con 0x7fb74406e020 2023-03-03T17:54:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb748b5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7440c4780 con 0x7fb744155390 2023-03-03T17:54:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb742d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb744072520 0x7fb744151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:39.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb742d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb744072520 0x7fb744151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35084/0 (socket says 172.21.15.7:35084) 2023-03-03T17:54:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb742d9d700 1 -- 172.21.15.7:0/1866950500 learned_addr learned my addr 172.21.15.7:0/1866950500 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:54:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb74259c700 1 --2- 172.21.15.7:0/1866950500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb744155390 0x7fb7440c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb741d9b700 1 --2- 172.21.15.7:0/1866950500 >> v2:172.21.15.143:3300/0 conn(0x7fb74406e020 0x7fb744071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb742d9d700 1 -- 172.21.15.7:0/1866950500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb744155390 msgr2=0x7fb7440c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:54:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb742d9d700 1 --2- 172.21.15.7:0/1866950500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb744155390 0x7fb7440c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb742d9d700 1 -- 172.21.15.7:0/1866950500 >> v2:172.21.15.143:3300/0 conn(0x7fb74406e020 msgr2=0x7fb744071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:54:39.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb742d9d700 1 --2- 172.21.15.7:0/1866950500 >> v2:172.21.15.143:3300/0 conn(0x7fb74406e020 0x7fb744071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb742d9d700 1 -- 172.21.15.7:0/1866950500 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744154910 con 0x7fb744072520 2023-03-03T17:54:39.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb741d9b700 1 --2- 172.21.15.7:0/1866950500 >> v2:172.21.15.143:3300/0 conn(0x7fb74406e020 0x7fb744071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:54:39.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb742d9d700 1 --2- 172.21.15.7:0/1866950500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744072520 0x7fb744151770 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7fb74406f090 tx=0x7fb73c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a035ba6b4def5574 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:54:39.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.388+0000 7fb74159a700 1 -- 172.21.15.7:0/1866950500 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c010070 con 0x7fb744072520 2023-03-03T17:54:39.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb74159a700 1 -- 172.21.15.7:0/1866950500 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c00dce0 con 0x7fb744072520 2023-03-03T17:54:39.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb74159a700 1 -- 172.21.15.7:0/1866950500 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00b310 con 0x7fb744072520 2023-03-03T17:54:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb748b5d700 1 -- 172.21.15.7:0/1866950500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744072520 msgr2=0x7fb744151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:54:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb748b5d700 1 --2- 172.21.15.7:0/1866950500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744072520 0x7fb744151770 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7fb74406f090 tx=0x7fb73c00d430 comp rx=0 tx=0).stop 2023-03-03T17:54:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb748b5d700 1 -- 172.21.15.7:0/1866950500 shutdown_connections 2023-03-03T17:54:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb748b5d700 1 --2- 172.21.15.7:0/1866950500 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744072520 0x7fb744151770 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb748b5d700 1 --2- 172.21.15.7:0/1866950500 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb744155390 0x7fb7440c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb748b5d700 1 --2- 172.21.15.7:0/1866950500 >> v2:172.21.15.143:3300/0 conn(0x7fb74406e020 0x7fb744071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb748b5d700 1 -- 172.21.15.7:0/1866950500 >> 172.21.15.7:0/1866950500 conn(0x7fb74406c110 msgr2=0x7fb74406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:54:39.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb748b5d700 1 -- 172.21.15.7:0/1866950500 shutdown_connections 2023-03-03T17:54:39.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.389+0000 7fb748b5d700 1 -- 172.21.15.7:0/1866950500 wait complete. 2023-03-03T17:54:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.390+0000 7fb748b5d700 1 Processor -- start 2023-03-03T17:54:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.391+0000 7fb748b5d700 1 -- start start 2023-03-03T17:54:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.391+0000 7fb748b5d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb74406e020 0x7fb744143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.391+0000 7fb748b5d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb744072520 0x7fb74414a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:39.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.391+0000 7fb748b5d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744155390 0x7fb7440c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:39.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.391+0000 7fb748b5d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7440ac4c0 con 0x7fb744155390 2023-03-03T17:54:39.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.391+0000 7fb748b5d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744143fc0 con 0x7fb744072520 2023-03-03T17:54:39.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.391+0000 7fb748b5d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7440c1900 con 0x7fb74406e020 2023-03-03T17:54:39.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.391+0000 7fb742d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744155390 0x7fb7440c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:39.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.391+0000 7fb742d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744155390 0x7fb7440c6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35090/0 (socket says 172.21.15.7:35090) 2023-03-03T17:54:39.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.391+0000 7fb742d9d700 1 -- 172.21.15.7:0/2024208773 learned_addr learned my addr 172.21.15.7:0/2024208773 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:54:39.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb74259c700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb74406e020 0x7fb744143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb741d9b700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb744072520 0x7fb74414a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb742d9d700 1 -- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb74406e020 msgr2=0x7fb744143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:54:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb742d9d700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb74406e020 0x7fb744143a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb742d9d700 1 -- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb744072520 msgr2=0x7fb74414a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:54:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb742d9d700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb744072520 0x7fb74414a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb742d9d700 1 -- 172.21.15.7:0/2024208773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c00bd40 con 0x7fb744155390 2023-03-03T17:54:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb741d9b700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb744072520 0x7fb74414a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:54:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb742d9d700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744155390 0x7fb7440c6380 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7fb7440c4a90 tx=0x7fb73c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:54:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb732ffd700 1 -- 172.21.15.7:0/2024208773 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c009a10 con 0x7fb744155390 2023-03-03T17:54:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb732ffd700 1 -- 172.21.15.7:0/2024208773 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c009b70 con 0x7fb744155390 2023-03-03T17:54:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb748b5d700 1 -- 172.21.15.7:0/2024208773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb744070540 con 0x7fb744155390 2023-03-03T17:54:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb732ffd700 1 -- 172.21.15.7:0/2024208773 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c004040 con 0x7fb744155390 2023-03-03T17:54:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.392+0000 7fb748b5d700 1 -- 172.21.15.7:0/2024208773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7440702a0 con 0x7fb744155390 2023-03-03T17:54:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.393+0000 7fb748b5d700 1 -- 172.21.15.7:0/2024208773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb744070820 con 0x7fb744155390 2023-03-03T17:54:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.394+0000 7fb732ffd700 1 -- 172.21.15.7:0/2024208773 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb73c02e070 con 0x7fb744155390 2023-03-03T17:54:39.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.394+0000 7fb732ffd700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7240682f0 0x7fb72406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:54:39.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.394+0000 7fb732ffd700 1 -- 172.21.15.7:0/2024208773 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb73c089f10 con 0x7fb744155390 2023-03-03T17:54:39.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.394+0000 7fb74259c700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7240682f0 0x7fb72406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:54:39.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.395+0000 7fb74259c700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7240682f0 0x7fb72406a7a0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fb734009ab0 tx=0x7fb734008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:54:39.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.399+0000 7fb732ffd700 1 -- 172.21.15.7:0/2024208773 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb73c056250 con 0x7fb744155390 2023-03-03T17:54:39.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.637+0000 7fb748b5d700 1 -- 172.21.15.7:0/2024208773 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb744074bb0 con 0x7fb744155390 2023-03-03T17:54:39.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.638+0000 7fb732ffd700 1 -- 172.21.15.7:0/2024208773 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb73c059870 con 0x7fb744155390 2023-03-03T17:54:39.638 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:54:39.638 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:54:39.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 -- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7240682f0 msgr2=0x7fb72406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:54:39.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7240682f0 0x7fb72406a7a0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fb734009ab0 tx=0x7fb734008040 comp rx=0 tx=0).stop 2023-03-03T17:54:39.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 -- 172.21.15.7:0/2024208773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744155390 msgr2=0x7fb7440c6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:54:39.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744155390 0x7fb7440c6380 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7fb7440c4a90 tx=0x7fb73c004ad0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 -- 172.21.15.7:0/2024208773 shutdown_connections 2023-03-03T17:54:39.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7240682f0 0x7fb72406a7a0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb744155390 0x7fb7440c6380 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb74406e020 0x7fb744143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 --2- 172.21.15.7:0/2024208773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb744072520 0x7fb74414a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:54:39.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 -- 172.21.15.7:0/2024208773 >> 172.21.15.7:0/2024208773 conn(0x7fb74406c110 msgr2=0x7fb7440c2340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:54:39.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 -- 172.21.15.7:0/2024208773 shutdown_connections 2023-03-03T17:54:39.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:54:39.640+0000 7fb748b5d700 1 -- 172.21.15.7:0/2024208773 wait complete. 2023-03-03T17:54:39.642 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:54:39.652 INFO:tasks.mds_thrash.fs.[a]:waiting for 22 secs before thrashing 2023-03-03T17:55:01.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:55:01.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.794+0000 7fe2d1e37700 1 Processor -- start 2023-03-03T17:55:01.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.794+0000 7fe2d1e37700 1 -- start start 2023-03-03T17:55:01.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2d1e37700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc151d40 0x7fe2cc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:01.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2d1e37700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2cc073c30 0x7fe2cc1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:01.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2d1e37700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2cc06d430 con 0x7fe2cc0c2930 2023-03-03T17:55:01.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2d1e37700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2cc06c2a0 con 0x7fe2cc073c30 2023-03-03T17:55:01.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2d1e37700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc1586e0 con 0x7fe2cc151d40 2023-03-03T17:55:01.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2cbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe2cc073c30 0x7fe2cc1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:01.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2caffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe2cc0c2930 0x7fe2cc157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57324/0 (socket says 172.21.15.7:57324) 2023-03-03T17:55:01.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2caffd700 1 -- 172.21.15.7:0/426654222 learned_addr learned my addr 172.21.15.7:0/426654222 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:55:01.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2cbfff700 1 --2- 172.21.15.7:0/426654222 >> v2:172.21.15.7:3300/0 conn(0x7fe2cc073c30 0x7fe2cc1581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35114/0 (socket says 172.21.15.7:35114) 2023-03-03T17:55:01.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2cb7fe700 1 --2- 172.21.15.7:0/426654222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc151d40 0x7fe2cc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:01.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2cbfff700 1 -- 172.21.15.7:0/426654222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc151d40 msgr2=0x7fe2cc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:55:01.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2cbfff700 1 --2- 172.21.15.7:0/426654222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc151d40 0x7fe2cc0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:01.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2cbfff700 1 -- 172.21.15.7:0/426654222 >> v1:172.21.15.7:6789/0 conn(0x7fe2cc0c2930 legacy=0x7fe2cc157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.795+0000 7fe2cbfff700 1 -- 172.21.15.7:0/426654222 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc06cb00 con 0x7fe2cc073c30 2023-03-03T17:55:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2cbfff700 1 --2- 172.21.15.7:0/426654222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc073c30 0x7fe2cc1581a0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009d90 tx=0x7fe2c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc697b1b30801b07 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:55:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2ca7fc700 1 -- 172.21.15.7:0/426654222 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c400e070 con 0x7fe2cc073c30 2023-03-03T17:55:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2ca7fc700 1 -- 172.21.15.7:0/426654222 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c400bbd0 con 0x7fe2cc073c30 2023-03-03T17:55:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2ca7fc700 1 -- 172.21.15.7:0/426654222 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4012430 con 0x7fe2cc073c30 2023-03-03T17:55:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2d1e37700 1 -- 172.21.15.7:0/426654222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc073c30 msgr2=0x7fe2cc1581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2d1e37700 1 --2- 172.21.15.7:0/426654222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc073c30 0x7fe2cc1581a0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4009d90 tx=0x7fe2c400b330 comp rx=0 tx=0).stop 2023-03-03T17:55:01.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2d1e37700 1 -- 172.21.15.7:0/426654222 shutdown_connections 2023-03-03T17:55:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2d1e37700 1 --2- 172.21.15.7:0/426654222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc073c30 0x7fe2cc1581a0 unknown :-1 s=CLOSED pgs=3056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2d1e37700 1 --2- 172.21.15.7:0/426654222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc151d40 0x7fe2cc0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2d1e37700 1 -- 172.21.15.7:0/426654222 >> 172.21.15.7:0/426654222 conn(0x7fe2cc0a94d0 msgr2=0x7fe2cc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:55:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2d1e37700 1 -- 172.21.15.7:0/426654222 shutdown_connections 2023-03-03T17:55:01.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.796+0000 7fe2d1e37700 1 -- 172.21.15.7:0/426654222 wait complete. 2023-03-03T17:55:01.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.798+0000 7fe2d1e37700 1 Processor -- start 2023-03-03T17:55:01.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.798+0000 7fe2d1e37700 1 -- start start 2023-03-03T17:55:01.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.798+0000 7fe2d1e37700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc073c30 0x7fe2cc14e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:01.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.798+0000 7fe2d1e37700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc0c2930 0x7fe2cc14e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:01.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.798+0000 7fe2d1e37700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc151d40 0x7fe2cc0be430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:01.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.798+0000 7fe2d1e37700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0bfc40 con 0x7fe2cc0c2930 2023-03-03T17:55:01.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.798+0000 7fe2d1e37700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0ad090 con 0x7fe2cc073c30 2023-03-03T17:55:01.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.798+0000 7fe2d1e37700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc0c0d90 con 0x7fe2cc151d40 2023-03-03T17:55:01.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2caffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc0c2930 0x7fe2cc14e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:01.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2caffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc0c2930 0x7fe2cc14e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35118/0 (socket says 172.21.15.7:35118) 2023-03-03T17:55:01.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2caffd700 1 -- 172.21.15.7:0/123076686 learned_addr learned my addr 172.21.15.7:0/123076686 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:55:01.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2cbfff700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc151d40 0x7fe2cc0be430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:01.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2caffd700 1 -- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc151d40 msgr2=0x7fe2cc0be430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:01.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2caffd700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc151d40 0x7fe2cc0be430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:01.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2cb7fe700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc073c30 0x7fe2cc14e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:01.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2caffd700 1 -- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc073c30 msgr2=0x7fe2cc14e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:01.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2caffd700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc073c30 0x7fe2cc14e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:01.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2caffd700 1 -- 172.21.15.7:0/123076686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c4009a70 con 0x7fe2cc0c2930 2023-03-03T17:55:01.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2caffd700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc0c2930 0x7fe2cc14e540 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c4b0 tx=0x7fe2c000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:55:01.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2b3fff700 1 -- 172.21.15.7:0/123076686 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0005d70 con 0x7fe2cc0c2930 2023-03-03T17:55:01.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2b3fff700 1 -- 172.21.15.7:0/123076686 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c00070d0 con 0x7fe2cc0c2930 2023-03-03T17:55:01.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2b3fff700 1 -- 172.21.15.7:0/123076686 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0007440 con 0x7fe2cc0c2930 2023-03-03T17:55:01.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2d1e37700 1 -- 172.21.15.7:0/123076686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2cc06d0c0 con 0x7fe2cc0c2930 2023-03-03T17:55:01.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.799+0000 7fe2d1e37700 1 -- 172.21.15.7:0/123076686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2cc06cde0 con 0x7fe2cc0c2930 2023-03-03T17:55:01.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.800+0000 7fe2d1e37700 1 -- 172.21.15.7:0/123076686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2cc06bea0 con 0x7fe2cc0c2930 2023-03-03T17:55:01.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.801+0000 7fe2b3fff700 1 -- 172.21.15.7:0/123076686 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe2c0019030 con 0x7fe2cc0c2930 2023-03-03T17:55:01.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.801+0000 7fe2b3fff700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2b80682f0 0x7fe2b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:01.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.801+0000 7fe2b3fff700 1 -- 172.21.15.7:0/123076686 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe2c0085380 con 0x7fe2cc0c2930 2023-03-03T17:55:01.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.801+0000 7fe2cb7fe700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2b80682f0 0x7fe2b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:01.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.802+0000 7fe2cb7fe700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2b80682f0 0x7fe2b806a7a0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009950 tx=0x7fe2bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:55:01.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:01.805+0000 7fe2b3fff700 1 -- 172.21.15.7:0/123076686 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2c004b5e0 con 0x7fe2cc0c2930 2023-03-03T17:55:02.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.043+0000 7fe2d1e37700 1 -- 172.21.15.7:0/123076686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2cc074bb0 con 0x7fe2cc0c2930 2023-03-03T17:55:02.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.044+0000 7fe2b3fff700 1 -- 172.21.15.7:0/123076686 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe2c00517d0 con 0x7fe2cc0c2930 2023-03-03T17:55:02.044 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:55:02.044 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:55:02.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 -- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2b80682f0 msgr2=0x7fe2b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:02.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2b80682f0 0x7fe2b806a7a0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009950 tx=0x7fe2bc008040 comp rx=0 tx=0).stop 2023-03-03T17:55:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 -- 172.21.15.7:0/123076686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc0c2930 msgr2=0x7fe2cc14e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc0c2930 0x7fe2cc14e540 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000c4b0 tx=0x7fe2c000c870 comp rx=0 tx=0).stop 2023-03-03T17:55:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 -- 172.21.15.7:0/123076686 shutdown_connections 2023-03-03T17:55:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2cc151d40 0x7fe2cc0be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2cc073c30 0x7fe2cc14e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe2b80682f0 0x7fe2b806a7a0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 --2- 172.21.15.7:0/123076686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe2cc0c2930 0x7fe2cc14e540 unknown :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:02.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 -- 172.21.15.7:0/123076686 >> 172.21.15.7:0/123076686 conn(0x7fe2cc0a94d0 msgr2=0x7fe2cc0c45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:55:02.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 -- 172.21.15.7:0/123076686 shutdown_connections 2023-03-03T17:55:02.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:02.046+0000 7fe2d1e37700 1 -- 172.21.15.7:0/123076686 wait complete. 2023-03-03T17:55:02.047 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:55:02.057 INFO:tasks.mds_thrash.fs.[a]:waiting for 70 secs before thrashing 2023-03-03T17:55:02.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:02.593 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:32.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:32.655 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:55:57.149 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:55:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.289+0000 7fc8a40ca700 1 Processor -- start 2023-03-03T17:55:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.290+0000 7fc8a40ca700 1 -- start start 2023-03-03T17:55:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.290+0000 7fc8a40ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c071e80 0x7fc89c151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.290+0000 7fc8a40ca700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc89c072250 0x7fc89c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.290+0000 7fc8a40ca700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89c1510d0 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.290+0000 7fc8a40ca700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc89c0c3740 con 0x7fc89c072250 2023-03-03T17:55:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.290+0000 7fc8a40ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c070670 con 0x7fc89c071e80 2023-03-03T17:55:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.290+0000 7fc8a1665700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc89c072250 0x7fc89c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a2667700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc89c0c1ce0 0x7fc89c1546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57382/0 (socket says 172.21.15.7:57382) 2023-03-03T17:55:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a2667700 1 -- 172.21.15.7:0/504236865 learned_addr learned my addr 172.21.15.7:0/504236865 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:55:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a1e66700 1 --2- 172.21.15.7:0/504236865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c071e80 0x7fc89c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a1665700 1 -- 172.21.15.7:0/504236865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c071e80 msgr2=0x7fc89c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:55:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a1665700 1 --2- 172.21.15.7:0/504236865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c071e80 0x7fc89c151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a1665700 1 -- 172.21.15.7:0/504236865 >> v1:172.21.15.7:6789/0 conn(0x7fc89c0c1ce0 legacy=0x7fc89c1546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a1665700 1 -- 172.21.15.7:0/504236865 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c0c39e0 con 0x7fc89c072250 2023-03-03T17:55:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a0e64700 1 -- 172.21.15.7:0/504236865 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855567506 0 0) 0x7fc89c1510d0 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a1665700 1 --2- 172.21.15.7:0/504236865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c072250 0x7fc89c14e1f0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7fc89000bae0 tx=0x7fc89000bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ddff63d9df1395 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:55:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a0e64700 1 -- 172.21.15.7:0/504236865 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89000dd30 con 0x7fc89c072250 2023-03-03T17:55:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a0e64700 1 -- 172.21.15.7:0/504236865 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89000de90 con 0x7fc89c072250 2023-03-03T17:55:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.291+0000 7fc8a0e64700 1 -- 172.21.15.7:0/504236865 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc890010690 con 0x7fc89c072250 2023-03-03T17:55:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.292+0000 7fc8a40ca700 1 -- 172.21.15.7:0/504236865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c072250 msgr2=0x7fc89c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.292+0000 7fc8a40ca700 1 --2- 172.21.15.7:0/504236865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c072250 0x7fc89c14e1f0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7fc89000bae0 tx=0x7fc89000bdf0 comp rx=0 tx=0).stop 2023-03-03T17:55:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.292+0000 7fc8a40ca700 1 -- 172.21.15.7:0/504236865 shutdown_connections 2023-03-03T17:55:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.292+0000 7fc8a40ca700 1 --2- 172.21.15.7:0/504236865 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c071e80 0x7fc89c151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.292+0000 7fc8a40ca700 1 --2- 172.21.15.7:0/504236865 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c072250 0x7fc89c14e1f0 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.292+0000 7fc8a40ca700 1 -- 172.21.15.7:0/504236865 >> 172.21.15.7:0/504236865 conn(0x7fc89c0a94d0 msgr2=0x7fc89c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:55:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.292+0000 7fc8a40ca700 1 -- 172.21.15.7:0/504236865 shutdown_connections 2023-03-03T17:55:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.292+0000 7fc8a40ca700 1 -- 172.21.15.7:0/504236865 wait complete. 2023-03-03T17:55:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.293+0000 7fc8a40ca700 1 Processor -- start 2023-03-03T17:55:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.293+0000 7fc8a40ca700 1 -- start start 2023-03-03T17:55:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a40ca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c071e80 0x7fc89c142520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a40ca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c072250 0x7fc89c142a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a40ca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c0c1ce0 0x7fc89c149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a40ca700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c142fa0 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a40ca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c14aa30 con 0x7fc89c072250 2023-03-03T17:55:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a40ca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c14b7b0 con 0x7fc89c071e80 2023-03-03T17:55:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a2667700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c0c1ce0 0x7fc89c149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a2667700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c0c1ce0 0x7fc89c149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35176/0 (socket says 172.21.15.7:35176) 2023-03-03T17:55:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a2667700 1 -- 172.21.15.7:0/514860384 learned_addr learned my addr 172.21.15.7:0/514860384 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:55:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a1e66700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c071e80 0x7fc89c142520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a1665700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c072250 0x7fc89c142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a2667700 1 -- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c071e80 msgr2=0x7fc89c142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a2667700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c071e80 0x7fc89c142520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a2667700 1 -- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c072250 msgr2=0x7fc89c142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a2667700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c072250 0x7fc89c142a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a2667700 1 -- 172.21.15.7:0/514860384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89000b740 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc8a2667700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c0c1ce0 0x7fc89c149250 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fc89400cc70 tx=0x7fc89400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:55:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.294+0000 7fc88e7fc700 1 -- 172.21.15.7:0/514860384 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89400aa20 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.295+0000 7fc88e7fc700 1 -- 172.21.15.7:0/514860384 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89400aba0 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.295+0000 7fc8a40ca700 1 -- 172.21.15.7:0/514860384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89c06c9a0 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.295+0000 7fc88e7fc700 1 -- 172.21.15.7:0/514860384 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8940070d0 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.295+0000 7fc8a40ca700 1 -- 172.21.15.7:0/514860384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89c151300 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.296+0000 7fc8a40ca700 1 -- 172.21.15.7:0/514860384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89c06cb30 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.296+0000 7fc88e7fc700 1 -- 172.21.15.7:0/514860384 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc89401a030 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.296+0000 7fc88e7fc700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8800682f0 0x7fc88006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:55:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.296+0000 7fc88e7fc700 1 -- 172.21.15.7:0/514860384 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc894085cd0 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.297+0000 7fc8a1e66700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8800682f0 0x7fc88006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:55:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.297+0000 7fc8a1e66700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8800682f0 0x7fc88006a7a0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fc898006b20 tx=0x7fc898008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:55:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.301+0000 7fc88e7fc700 1 -- 172.21.15.7:0/514860384 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc894052070 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.538+0000 7fc8a40ca700 1 -- 172.21.15.7:0/514860384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc89c1ea9f0 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.539+0000 7fc88e7fc700 1 -- 172.21.15.7:0/514860384 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc89c1ea9f0 con 0x7fc89c0c1ce0 2023-03-03T17:55:57.539 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:55:57.539 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:55:57.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 -- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8800682f0 msgr2=0x7fc88006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8800682f0 0x7fc88006a7a0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fc898006b20 tx=0x7fc898008040 comp rx=0 tx=0).stop 2023-03-03T17:55:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 -- 172.21.15.7:0/514860384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c0c1ce0 msgr2=0x7fc89c149250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:55:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c0c1ce0 0x7fc89c149250 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7fc89400cc70 tx=0x7fc89400cf80 comp rx=0 tx=0).stop 2023-03-03T17:55:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 -- 172.21.15.7:0/514860384 shutdown_connections 2023-03-03T17:55:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc8800682f0 0x7fc88006a7a0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c0c1ce0 0x7fc89c149250 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c071e80 0x7fc89c142520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 --2- 172.21.15.7:0/514860384 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c072250 0x7fc89c142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:55:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 -- 172.21.15.7:0/514860384 >> 172.21.15.7:0/514860384 conn(0x7fc89c0a94d0 msgr2=0x7fc89c0c29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:55:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 -- 172.21.15.7:0/514860384 shutdown_connections 2023-03-03T17:55:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:55:57.541+0000 7fc8a40ca700 1 -- 172.21.15.7:0/514860384 wait complete. 2023-03-03T17:55:57.542 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:55:57.552 INFO:tasks.mds_thrash.fs.[b]:waiting for 81 secs before thrashing 2023-03-03T17:56:02.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:56:02.720 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:56:12.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:56:12.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.198+0000 7f0119180700 1 Processor -- start 2023-03-03T17:56:12.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.198+0000 7f0119180700 1 -- start start 2023-03-03T17:56:12.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.198+0000 7f0119180700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f011413a0f0 0x7f01140c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:56:12.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0119180700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0114154560 con 0x7f01140c1da0 2023-03-03T17:56:12.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0119180700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0114157be0 con 0x7f01141586c0 2023-03-03T17:56:12.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0119180700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0114154840 con 0x7f011413a0f0 2023-03-03T17:56:12.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f011359e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f01140c1da0 0x7f011415edd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57406/0 (socket says 172.21.15.7:57406) 2023-03-03T17:56:12.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f011359e700 1 -- 172.21.15.7:0/3892975479 learned_addr learned my addr 172.21.15.7:0/3892975479 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:56:12.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0112d9d700 1 --2- 172.21.15.7:0/3892975479 >> v2:172.21.15.143:3300/0 conn(0x7f011413a0f0 0x7f01140c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:56:12.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0111d9b700 1 -- 172.21.15.7:0/3892975479 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670197826 0 0) 0x7f0114154560 con 0x7f01140c1da0 2023-03-03T17:56:12.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0111d9b700 1 -- 172.21.15.7:0/3892975479 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00fc0035b0 con 0x7f01140c1da0 2023-03-03T17:56:12.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0112d9d700 1 -- 172.21.15.7:0/3892975479 >> v1:172.21.15.143:6789/0 conn(0x7f01141586c0 legacy=0x7f011406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:56:12.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0112d9d700 1 -- 172.21.15.7:0/3892975479 >> v1:172.21.15.7:6789/0 conn(0x7f01140c1da0 legacy=0x7f011415edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:56:12.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0111d9b700 1 -- 172.21.15.7:0/3892975479 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1247039842 0 0) 0x7f00fc0035b0 con 0x7f01140c1da0 2023-03-03T17:56:12.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0112d9d700 1 -- 172.21.15.7:0/3892975479 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0114157ec0 con 0x7f011413a0f0 2023-03-03T17:56:12.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.199+0000 7f0112d9d700 1 --2- 172.21.15.7:0/3892975479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f011413a0f0 0x7f01140c07d0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f0104009840 tx=0x7f0104009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77376cbdd2ac6b74 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:56:12.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.200+0000 7f0111d9b700 1 -- 172.21.15.7:0/3892975479 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c0019f0 con 0x7f011413a0f0 2023-03-03T17:56:12.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.200+0000 7f0111d9b700 1 -- 172.21.15.7:0/3892975479 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0104004030 con 0x7f011413a0f0 2023-03-03T17:56:12.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.200+0000 7f0111d9b700 1 -- 172.21.15.7:0/3892975479 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01040043c0 con 0x7f011413a0f0 2023-03-03T17:56:12.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.200+0000 7f0119180700 1 -- 172.21.15.7:0/3892975479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f011413a0f0 msgr2=0x7f01140c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:56:12.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.200+0000 7f0119180700 1 --2- 172.21.15.7:0/3892975479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f011413a0f0 0x7f01140c07d0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f0104009840 tx=0x7f0104009b50 comp rx=0 tx=0).stop 2023-03-03T17:56:12.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.200+0000 7f0119180700 1 -- 172.21.15.7:0/3892975479 shutdown_connections 2023-03-03T17:56:12.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.200+0000 7f0119180700 1 --2- 172.21.15.7:0/3892975479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f011413a0f0 0x7f01140c07d0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:56:12.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.200+0000 7f0119180700 1 -- 172.21.15.7:0/3892975479 >> 172.21.15.7:0/3892975479 conn(0x7f011406c110 msgr2=0x7f011406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:56:12.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.200+0000 7f0119180700 1 -- 172.21.15.7:0/3892975479 shutdown_connections 2023-03-03T17:56:12.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.200+0000 7f0119180700 1 -- 172.21.15.7:0/3892975479 wait complete. 2023-03-03T17:56:12.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.202+0000 7f0119180700 1 Processor -- start 2023-03-03T17:56:12.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.202+0000 7f0119180700 1 -- start start 2023-03-03T17:56:12.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.202+0000 7f0119180700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01140c1da0 0x7f01141384f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:56:12.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.202+0000 7f0119180700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f011413a0f0 0x7f01141ea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:56:12.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.202+0000 7f0119180700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01141586c0 0x7f01141ea6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:56:12.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.202+0000 7f0119180700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011413b6c0 con 0x7f01141586c0 2023-03-03T17:56:12.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.202+0000 7f0119180700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c5430 con 0x7f011413a0f0 2023-03-03T17:56:12.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.202+0000 7f0119180700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c6510 con 0x7f01140c1da0 2023-03-03T17:56:12.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.202+0000 7f011359e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01141586c0 0x7f01141ea6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:56:12.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f011359e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01141586c0 0x7f01141ea6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35200/0 (socket says 172.21.15.7:35200) 2023-03-03T17:56:12.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f011359e700 1 -- 172.21.15.7:0/2185008044 learned_addr learned my addr 172.21.15.7:0/2185008044 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:56:12.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f0112d9d700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01140c1da0 0x7f01141384f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:56:12.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f011259c700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f011413a0f0 0x7f01141ea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:56:12.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f011359e700 1 -- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01140c1da0 msgr2=0x7f01141384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:56:12.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f011359e700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01140c1da0 0x7f01141384f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:56:12.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f011359e700 1 -- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f011413a0f0 msgr2=0x7f01141ea1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:56:12.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f011359e700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f011413a0f0 0x7f01141ea1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:56:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f011359e700 1 -- 172.21.15.7:0/2185008044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01040094a0 con 0x7f01141586c0 2023-03-03T17:56:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f011359e700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01141586c0 0x7f01141ea6f0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f010c001ba0 tx=0x7f010c00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:56:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f00fb7fe700 1 -- 172.21.15.7:0/2185008044 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c00aac0 con 0x7f01141586c0 2023-03-03T17:56:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f00fb7fe700 1 -- 172.21.15.7:0/2185008044 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c00ac20 con 0x7f01141586c0 2023-03-03T17:56:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f0119180700 1 -- 172.21.15.7:0/2185008044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011406e1a0 con 0x7f01141586c0 2023-03-03T17:56:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f00fb7fe700 1 -- 172.21.15.7:0/2185008044 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c0070d0 con 0x7f01141586c0 2023-03-03T17:56:12.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.203+0000 7f0119180700 1 -- 172.21.15.7:0/2185008044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01141581a0 con 0x7f01141586c0 2023-03-03T17:56:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.204+0000 7f0119180700 1 -- 172.21.15.7:0/2185008044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011406e330 con 0x7f01141586c0 2023-03-03T17:56:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.205+0000 7f00fb7fe700 1 -- 172.21.15.7:0/2185008044 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f010c01a030 con 0x7f01141586c0 2023-03-03T17:56:12.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.205+0000 7f00fb7fe700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01000682f0 0x7f010006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:56:12.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.205+0000 7f00fb7fe700 1 -- 172.21.15.7:0/2185008044 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f010c085e20 con 0x7f01141586c0 2023-03-03T17:56:12.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.205+0000 7f0112d9d700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01000682f0 0x7f010006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:56:12.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.206+0000 7f0112d9d700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01000682f0 0x7f010006a7a0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f01040090b0 tx=0x7f0104008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:56:12.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.210+0000 7f00fb7fe700 1 -- 172.21.15.7:0/2185008044 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f010c0521c0 con 0x7f01141586c0 2023-03-03T17:56:12.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.449+0000 7f0119180700 1 -- 172.21.15.7:0/2185008044 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0114074bb0 con 0x7f01141586c0 2023-03-03T17:56:12.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.449+0000 7f00fb7fe700 1 -- 172.21.15.7:0/2185008044 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f010c018090 con 0x7f01141586c0 2023-03-03T17:56:12.449 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:56:12.449 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:56:12.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 -- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01000682f0 msgr2=0x7f010006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:56:12.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01000682f0 0x7f010006a7a0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f01040090b0 tx=0x7f0104008030 comp rx=0 tx=0).stop 2023-03-03T17:56:12.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 -- 172.21.15.7:0/2185008044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01141586c0 msgr2=0x7f01141ea6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:56:12.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01141586c0 0x7f01141ea6f0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f010c001ba0 tx=0x7f010c00cf50 comp rx=0 tx=0).stop 2023-03-03T17:56:12.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 -- 172.21.15.7:0/2185008044 shutdown_connections 2023-03-03T17:56:12.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01000682f0 0x7f010006a7a0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:56:12.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01141586c0 0x7f01141ea6f0 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:56:12.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01140c1da0 0x7f01141384f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:56:12.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 --2- 172.21.15.7:0/2185008044 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f011413a0f0 0x7f01141ea1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:56:12.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 -- 172.21.15.7:0/2185008044 >> 172.21.15.7:0/2185008044 conn(0x7f011406c110 msgr2=0x7f0114072110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:56:12.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 -- 172.21.15.7:0/2185008044 shutdown_connections 2023-03-03T17:56:12.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:56:12.451+0000 7f0119180700 1 -- 172.21.15.7:0/2185008044 wait complete. 2023-03-03T17:56:12.452 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:56:12.463 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T17:56:32.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:56:32.790 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:02.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:02.863 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:17.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:57:17.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.728+0000 7fc7807b5700 1 Processor -- start 2023-03-03T17:57:17.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.728+0000 7fc7807b5700 1 -- start start 2023-03-03T17:57:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.729+0000 7fc7807b5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc7780c8020 0x7fc7780c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.729+0000 7fc7807b5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7780c1da0 0x7fc7780c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.729+0000 7fc7807b5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7780c7b90 con 0x7fc77806d4b0 2023-03-03T17:57:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.729+0000 7fc7807b5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7780c7330 con 0x7fc7780c1da0 2023-03-03T17:57:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.729+0000 7fc7807b5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc778154420 con 0x7fc7780c8020 2023-03-03T17:57:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.729+0000 7fc77ed52700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7780c1da0 0x7fc7780c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.729+0000 7fc77ed52700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7780c1da0 0x7fc7780c0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35248/0 (socket says 172.21.15.7:35248) 2023-03-03T17:57:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.729+0000 7fc77ed52700 1 -- 172.21.15.7:0/3238628777 learned_addr learned my addr 172.21.15.7:0/3238628777 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:57:17.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.730+0000 7fc77dd50700 1 --2- 172.21.15.7:0/3238628777 >> v2:172.21.15.143:3300/0 conn(0x7fc7780c8020 0x7fc7780c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:17.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.730+0000 7fc77ed52700 1 -- 172.21.15.7:0/3238628777 >> v2:172.21.15.143:3300/0 conn(0x7fc7780c8020 msgr2=0x7fc7780c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:57:17.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.730+0000 7fc77ed52700 1 --2- 172.21.15.7:0/3238628777 >> v2:172.21.15.143:3300/0 conn(0x7fc7780c8020 0x7fc7780c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:17.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.730+0000 7fc77ed52700 1 -- 172.21.15.7:0/3238628777 >> v1:172.21.15.143:6789/0 conn(0x7fc77806d4b0 legacy=0x7fc77806d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:17.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.730+0000 7fc77ed52700 1 -- 172.21.15.7:0/3238628777 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc778154640 con 0x7fc7780c1da0 2023-03-03T17:57:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.730+0000 7fc77dd50700 1 --2- 172.21.15.7:0/3238628777 >> v2:172.21.15.143:3300/0 conn(0x7fc7780c8020 0x7fc7780c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:57:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.730+0000 7fc77ed52700 1 --2- 172.21.15.7:0/3238628777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7780c1da0 0x7fc7780c0d10 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7fc77806fcb0 tx=0x7fc77000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff1fe1fb1015bdd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:57:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.730+0000 7fc77d54f700 1 -- 172.21.15.7:0/3238628777 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77000e070 con 0x7fc7780c1da0 2023-03-03T17:57:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.730+0000 7fc77d54f700 1 -- 172.21.15.7:0/3238628777 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77000b9c0 con 0x7fc7780c1da0 2023-03-03T17:57:17.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.730+0000 7fc77d54f700 1 -- 172.21.15.7:0/3238628777 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77000bd70 con 0x7fc7780c1da0 2023-03-03T17:57:17.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.731+0000 7fc7807b5700 1 -- 172.21.15.7:0/3238628777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7780c1da0 msgr2=0x7fc7780c0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.731+0000 7fc7807b5700 1 --2- 172.21.15.7:0/3238628777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7780c1da0 0x7fc7780c0d10 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7fc77806fcb0 tx=0x7fc77000b330 comp rx=0 tx=0).stop 2023-03-03T17:57:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.731+0000 7fc7807b5700 1 -- 172.21.15.7:0/3238628777 shutdown_connections 2023-03-03T17:57:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.731+0000 7fc7807b5700 1 --2- 172.21.15.7:0/3238628777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7780c1da0 0x7fc7780c0d10 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.731+0000 7fc7807b5700 1 --2- 172.21.15.7:0/3238628777 >> v2:172.21.15.143:3300/0 conn(0x7fc7780c8020 0x7fc7780c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.731+0000 7fc7807b5700 1 -- 172.21.15.7:0/3238628777 >> 172.21.15.7:0/3238628777 conn(0x7fc77806c110 msgr2=0x7fc77806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:57:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.731+0000 7fc7807b5700 1 -- 172.21.15.7:0/3238628777 shutdown_connections 2023-03-03T17:57:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.731+0000 7fc7807b5700 1 -- 172.21.15.7:0/3238628777 wait complete. 2023-03-03T17:57:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.732+0000 7fc7807b5700 1 Processor -- start 2023-03-03T17:57:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.732+0000 7fc7807b5700 1 -- start start 2023-03-03T17:57:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc7807b5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc77806d4b0 0x7fc778134850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc7807b5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7780c1da0 0x7fc7780713c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc7807b5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7780c8020 0x7fc778150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc7807b5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780ac750 con 0x7fc77806d4b0 2023-03-03T17:57:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc7807b5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc778134ea0 con 0x7fc7780c1da0 2023-03-03T17:57:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc7807b5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7780ac9a0 con 0x7fc7780c8020 2023-03-03T17:57:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc77e551700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc77806d4b0 0x7fc778134850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc77e551700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc77806d4b0 0x7fc778134850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35250/0 (socket says 172.21.15.7:35250) 2023-03-03T17:57:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc77e551700 1 -- 172.21.15.7:0/533382230 learned_addr learned my addr 172.21.15.7:0/533382230 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:57:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc77e551700 1 -- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7780c8020 msgr2=0x7fc778150ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc77e551700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7780c8020 0x7fc778150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc77e551700 1 -- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7780c1da0 msgr2=0x7fc7780713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc77e551700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7780c1da0 0x7fc7780713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc77e551700 1 -- 172.21.15.7:0/533382230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc770009b30 con 0x7fc77806d4b0 2023-03-03T17:57:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc77e551700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc77806d4b0 0x7fc778134850 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00c450 tx=0x7fc76c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:57:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc76affd700 1 -- 172.21.15.7:0/533382230 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c0052d0 con 0x7fc77806d4b0 2023-03-03T17:57:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc76affd700 1 -- 172.21.15.7:0/533382230 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76c005430 con 0x7fc77806d4b0 2023-03-03T17:57:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc7807b5700 1 -- 172.21.15.7:0/533382230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7780c7820 con 0x7fc77806d4b0 2023-03-03T17:57:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.733+0000 7fc76affd700 1 -- 172.21.15.7:0/533382230 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c0038c0 con 0x7fc77806d4b0 2023-03-03T17:57:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.734+0000 7fc7807b5700 1 -- 172.21.15.7:0/533382230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7780c7540 con 0x7fc77806d4b0 2023-03-03T17:57:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.734+0000 7fc7807b5700 1 -- 172.21.15.7:0/533382230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc77806e210 con 0x7fc77806d4b0 2023-03-03T17:57:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.735+0000 7fc76affd700 1 -- 172.21.15.7:0/533382230 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc76c019030 con 0x7fc77806d4b0 2023-03-03T17:57:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.735+0000 7fc76affd700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc75c0682f0 0x7fc75c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.735+0000 7fc76affd700 1 -- 172.21.15.7:0/533382230 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc76c085570 con 0x7fc77806d4b0 2023-03-03T17:57:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.736+0000 7fc77dd50700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc75c0682f0 0x7fc75c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.736+0000 7fc77dd50700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc75c0682f0 0x7fc75c06a7a0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fc7780c6250 tx=0x7fc774008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:57:17.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.740+0000 7fc76affd700 1 -- 172.21.15.7:0/533382230 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc76c051820 con 0x7fc77806d4b0 2023-03-03T17:57:17.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.978+0000 7fc7807b5700 1 -- 172.21.15.7:0/533382230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc778074bb0 con 0x7fc77806d4b0 2023-03-03T17:57:17.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.979+0000 7fc76affd700 1 -- 172.21.15.7:0/533382230 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc76c054e40 con 0x7fc77806d4b0 2023-03-03T17:57:17.979 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:57:17.979 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:57:17.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 -- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc75c0682f0 msgr2=0x7fc75c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:17.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc75c0682f0 0x7fc75c06a7a0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fc7780c6250 tx=0x7fc774008040 comp rx=0 tx=0).stop 2023-03-03T17:57:17.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 -- 172.21.15.7:0/533382230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc77806d4b0 msgr2=0x7fc778134850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:17.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc77806d4b0 0x7fc778134850 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00c450 tx=0x7fc76c00c810 comp rx=0 tx=0).stop 2023-03-03T17:57:17.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 -- 172.21.15.7:0/533382230 shutdown_connections 2023-03-03T17:57:17.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7780c8020 0x7fc778150ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:17.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc75c0682f0 0x7fc75c06a7a0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:17.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc77806d4b0 0x7fc778134850 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:17.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 --2- 172.21.15.7:0/533382230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7780c1da0 0x7fc7780713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:17.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 -- 172.21.15.7:0/533382230 >> 172.21.15.7:0/533382230 conn(0x7fc77806c110 msgr2=0x7fc7780c6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:57:17.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 -- 172.21.15.7:0/533382230 shutdown_connections 2023-03-03T17:57:17.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:17.981+0000 7fc7807b5700 1 -- 172.21.15.7:0/533382230 wait complete. 2023-03-03T17:57:17.983 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:57:17.992 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before thrashing 2023-03-03T17:57:18.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:57:18.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.688+0000 7ff9b6170700 1 Processor -- start 2023-03-03T17:57:18.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.688+0000 7ff9b6170700 1 -- start start 2023-03-03T17:57:18.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.689+0000 7ff9b6170700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b014fc50 0x7ff9b00c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:18.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.689+0000 7ff9b6170700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff9b00c3e30 0x7ff9b00c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:18.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.689+0000 7ff9b6170700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b00ad2b0 con 0x7ff9b006d240 2023-03-03T17:57:18.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.689+0000 7ff9b6170700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b00c30c0 con 0x7ff9b00c3e30 2023-03-03T17:57:18.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.689+0000 7ff9b6170700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b0156420 con 0x7ff9b014fc50 2023-03-03T17:57:18.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.689+0000 7ff9aeffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff9b006d240 0x7ff9b00c27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57470/0 (socket says 172.21.15.7:57470) 2023-03-03T17:57:18.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.689+0000 7ff9aeffd700 1 -- 172.21.15.7:0/450561556 learned_addr learned my addr 172.21.15.7:0/450561556 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:57:18.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9affff700 1 --2- 172.21.15.7:0/450561556 >> v2:172.21.15.143:3300/0 conn(0x7ff9b00c3e30 0x7ff9b00c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:18.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9af7fe700 1 --2- 172.21.15.7:0/450561556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b014fc50 0x7ff9b00c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:18.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/450561556 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979485966 0 0) 0x7ff9b00ad2b0 con 0x7ff9b006d240 2023-03-03T17:57:18.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/450561556 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff99c0035b0 con 0x7ff9b006d240 2023-03-03T17:57:18.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9affff700 1 -- 172.21.15.7:0/450561556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b014fc50 msgr2=0x7ff9b00c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:57:18.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9affff700 1 --2- 172.21.15.7:0/450561556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b014fc50 0x7ff9b00c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:18.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9affff700 1 -- 172.21.15.7:0/450561556 >> v1:172.21.15.7:6789/0 conn(0x7ff9b006d240 legacy=0x7ff9b00c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:18.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/450561556 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3476262201 0 0) 0x7ff99c0035b0 con 0x7ff9b006d240 2023-03-03T17:57:18.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9affff700 1 -- 172.21.15.7:0/450561556 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b00ad520 con 0x7ff9b00c3e30 2023-03-03T17:57:18.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9affff700 1 --2- 172.21.15.7:0/450561556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b00c3e30 0x7ff9b00c42b0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009d90 tx=0x7ff9a800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c2bfb81666bbf8c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:57:18.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/450561556 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a800e070 con 0x7ff9b00c3e30 2023-03-03T17:57:18.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/450561556 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a8003a20 con 0x7ff9b00c3e30 2023-03-03T17:57:18.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.690+0000 7ff9ae7fc700 1 -- 172.21.15.7:0/450561556 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff998001670 con 0x7ff9b00c3e30 2023-03-03T17:57:18.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.691+0000 7ff9b6170700 1 -- 172.21.15.7:0/450561556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b00c3e30 msgr2=0x7ff9b00c42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:18.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.691+0000 7ff9b6170700 1 --2- 172.21.15.7:0/450561556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b00c3e30 0x7ff9b00c42b0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009d90 tx=0x7ff9a800b330 comp rx=0 tx=0).stop 2023-03-03T17:57:18.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.691+0000 7ff9b6170700 1 -- 172.21.15.7:0/450561556 shutdown_connections 2023-03-03T17:57:18.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.691+0000 7ff9b6170700 1 --2- 172.21.15.7:0/450561556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b00c3e30 0x7ff9b00c42b0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:18.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.691+0000 7ff9b6170700 1 --2- 172.21.15.7:0/450561556 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b014fc50 0x7ff9b00c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:18.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.691+0000 7ff9b6170700 1 -- 172.21.15.7:0/450561556 >> 172.21.15.7:0/450561556 conn(0x7ff9b006c110 msgr2=0x7ff9b006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:57:18.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.691+0000 7ff9b6170700 1 -- 172.21.15.7:0/450561556 shutdown_connections 2023-03-03T17:57:18.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.691+0000 7ff9b6170700 1 -- 172.21.15.7:0/450561556 wait complete. 2023-03-03T17:57:18.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.692+0000 7ff9b6170700 1 Processor -- start 2023-03-03T17:57:18.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.692+0000 7ff9b6170700 1 -- start start 2023-03-03T17:57:18.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9b6170700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b006d240 0x7ff9b0071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:18.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9b6170700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b00c3e30 0x7ff9b0148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:18.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9b6170700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b014fc50 0x7ff9b00c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:18.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9b6170700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b0070700 con 0x7ff9b014fc50 2023-03-03T17:57:18.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9b6170700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b01ed850 con 0x7ff9b006d240 2023-03-03T17:57:18.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9b6170700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b01edce0 con 0x7ff9b00c3e30 2023-03-03T17:57:18.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9affff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b014fc50 0x7ff9b00c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:18.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9affff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b014fc50 0x7ff9b00c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35268/0 (socket says 172.21.15.7:35268) 2023-03-03T17:57:18.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9affff700 1 -- 172.21.15.7:0/2960127900 learned_addr learned my addr 172.21.15.7:0/2960127900 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:57:18.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9aeffd700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b00c3e30 0x7ff9b0148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:18.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9af7fe700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b006d240 0x7ff9b0071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:18.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9affff700 1 -- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b00c3e30 msgr2=0x7ff9b0148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:18.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9affff700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b00c3e30 0x7ff9b0148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:18.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9affff700 1 -- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b006d240 msgr2=0x7ff9b0071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:18.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9affff700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b006d240 0x7ff9b0071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:18.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.693+0000 7ff9affff700 1 -- 172.21.15.7:0/2960127900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a8009a70 con 0x7ff9b014fc50 2023-03-03T17:57:18.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.694+0000 7ff9affff700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b014fc50 0x7ff9b00c64d0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7ff9b006cff0 tx=0x7ff9a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:57:18.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.694+0000 7ff9a7fff700 1 -- 172.21.15.7:0/2960127900 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8004600 con 0x7ff9b014fc50 2023-03-03T17:57:18.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.694+0000 7ff9a7fff700 1 -- 172.21.15.7:0/2960127900 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a80047a0 con 0x7ff9b014fc50 2023-03-03T17:57:18.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.694+0000 7ff9a7fff700 1 -- 172.21.15.7:0/2960127900 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8012670 con 0x7ff9b014fc50 2023-03-03T17:57:18.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.694+0000 7ff9b6170700 1 -- 172.21.15.7:0/2960127900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b0074220 con 0x7ff9b014fc50 2023-03-03T17:57:18.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.694+0000 7ff9b6170700 1 -- 172.21.15.7:0/2960127900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b00c2490 con 0x7ff9b014fc50 2023-03-03T17:57:18.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.695+0000 7ff9b6170700 1 -- 172.21.15.7:0/2960127900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b0156690 con 0x7ff9b014fc50 2023-03-03T17:57:18.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.695+0000 7ff9a7fff700 1 -- 172.21.15.7:0/2960127900 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff9a8023030 con 0x7ff9b014fc50 2023-03-03T17:57:18.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.696+0000 7ff9a7fff700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9900682f0 0x7ff99006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:18.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.696+0000 7ff9a7fff700 1 -- 172.21.15.7:0/2960127900 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff9a8087dd0 con 0x7ff9b014fc50 2023-03-03T17:57:18.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.696+0000 7ff9af7fe700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9900682f0 0x7ff99006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:18.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.696+0000 7ff9af7fe700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9900682f0 0x7ff99006a7a0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7ff9b01df350 tx=0x7ff9a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:57:18.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.700+0000 7ff9a7fff700 1 -- 172.21.15.7:0/2960127900 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff9a8054080 con 0x7ff9b014fc50 2023-03-03T17:57:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.936+0000 7ff9b6170700 1 -- 172.21.15.7:0/2960127900 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9b0074bb0 con 0x7ff9b014fc50 2023-03-03T17:57:18.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.937+0000 7ff9a7fff700 1 -- 172.21.15.7:0/2960127900 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff9a80576a0 con 0x7ff9b014fc50 2023-03-03T17:57:18.937 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:57:18.937 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:57:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 -- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9900682f0 msgr2=0x7ff99006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9900682f0 0x7ff99006a7a0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7ff9b01df350 tx=0x7ff9a0008040 comp rx=0 tx=0).stop 2023-03-03T17:57:18.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 -- 172.21.15.7:0/2960127900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b014fc50 msgr2=0x7ff9b00c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b014fc50 0x7ff9b00c64d0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7ff9b006cff0 tx=0x7ff9a8004ad0 comp rx=0 tx=0).stop 2023-03-03T17:57:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 -- 172.21.15.7:0/2960127900 shutdown_connections 2023-03-03T17:57:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9900682f0 0x7ff99006a7a0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b014fc50 0x7ff9b00c64d0 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b006d240 0x7ff9b0071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 --2- 172.21.15.7:0/2960127900 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b00c3e30 0x7ff9b0148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:18.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 -- 172.21.15.7:0/2960127900 >> 172.21.15.7:0/2960127900 conn(0x7ff9b006c110 msgr2=0x7ff9b006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:57:18.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 -- 172.21.15.7:0/2960127900 shutdown_connections 2023-03-03T17:57:18.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:18.939+0000 7ff9b6170700 1 -- 172.21.15.7:0/2960127900 wait complete. 2023-03-03T17:57:18.940 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:57:18.950 INFO:tasks.mds_thrash.fs.[b]:waiting for 70 secs before thrashing 2023-03-03T17:57:18.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:57:19.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.119+0000 7fd68afec700 1 Processor -- start 2023-03-03T17:57:19.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.119+0000 7fd68afec700 1 -- start start 2023-03-03T17:57:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd68afec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd684151cd0 0x7fd684154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd68afec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6840c2e30 0x7fd684158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd68afec700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd684157ff0 con 0x7fd68406c6c0 2023-03-03T17:57:19.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd68afec700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68406d640 con 0x7fd684151cd0 2023-03-03T17:57:19.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd68afec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68406cda0 con 0x7fd6840c2e30 2023-03-03T17:57:19.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd688d88700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd684151cd0 0x7fd684154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:19.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd688d88700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd684151cd0 0x7fd684154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35272/0 (socket says 172.21.15.7:35272) 2023-03-03T17:57:19.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd688d88700 1 -- 172.21.15.7:0/4294014416 learned_addr learned my addr 172.21.15.7:0/4294014416 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:57:19.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd688d88700 1 -- 172.21.15.7:0/4294014416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6840c2e30 msgr2=0x7fd684158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:57:19.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd688d88700 1 --2- 172.21.15.7:0/4294014416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6840c2e30 0x7fd684158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:19.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd688d88700 1 -- 172.21.15.7:0/4294014416 >> v1:172.21.15.143:6789/0 conn(0x7fd68406c6c0 legacy=0x7fd6840736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:57:19.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd688d88700 1 -- 172.21.15.7:0/4294014416 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68406d080 con 0x7fd684151cd0 2023-03-03T17:57:19.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.120+0000 7fd688d88700 1 --2- 172.21.15.7:0/4294014416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 0x7fd684154510 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7fd6780098f0 tx=0x7fd678009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efba939d457c0c11 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:57:19.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd6837fe700 1 -- 172.21.15.7:0/4294014416 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd678004230 con 0x7fd684151cd0 2023-03-03T17:57:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd6837fe700 1 -- 172.21.15.7:0/4294014416 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd678004390 con 0x7fd684151cd0 2023-03-03T17:57:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd6837fe700 1 -- 172.21.15.7:0/4294014416 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6780036a0 con 0x7fd684151cd0 2023-03-03T17:57:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd68afec700 1 -- 172.21.15.7:0/4294014416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 msgr2=0x7fd684154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:19.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd68afec700 1 --2- 172.21.15.7:0/4294014416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 0x7fd684154510 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7fd6780098f0 tx=0x7fd678009c00 comp rx=0 tx=0).stop 2023-03-03T17:57:19.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd68afec700 1 -- 172.21.15.7:0/4294014416 shutdown_connections 2023-03-03T17:57:19.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd68afec700 1 --2- 172.21.15.7:0/4294014416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 0x7fd684154510 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:19.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd68afec700 1 --2- 172.21.15.7:0/4294014416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6840c2e30 0x7fd684158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:19.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd68afec700 1 -- 172.21.15.7:0/4294014416 >> 172.21.15.7:0/4294014416 conn(0x7fd6840a94d0 msgr2=0x7fd6840a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:57:19.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd68afec700 1 -- 172.21.15.7:0/4294014416 shutdown_connections 2023-03-03T17:57:19.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.121+0000 7fd68afec700 1 -- 172.21.15.7:0/4294014416 wait complete. 2023-03-03T17:57:19.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.122+0000 7fd68afec700 1 Processor -- start 2023-03-03T17:57:19.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.122+0000 7fd68afec700 1 -- start start 2023-03-03T17:57:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd68afec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd68406c6c0 0x7fd684072340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd68afec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6840c2e30 0x7fd6841e1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:19.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd68afec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 0x7fd6840c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:19.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd68afec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd684157b40 con 0x7fd684151cd0 2023-03-03T17:57:19.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd68afec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6840c55d0 con 0x7fd68406c6c0 2023-03-03T17:57:19.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd68afec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68414d750 con 0x7fd6840c2e30 2023-03-03T17:57:19.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd689589700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 0x7fd6840c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:19.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd689589700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 0x7fd6840c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35280/0 (socket says 172.21.15.7:35280) 2023-03-03T17:57:19.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd689589700 1 -- 172.21.15.7:0/3927405306 learned_addr learned my addr 172.21.15.7:0/3927405306 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:57:19.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd688d88700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd68406c6c0 0x7fd684072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:19.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.123+0000 7fd683fff700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6840c2e30 0x7fd6841e1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:19.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd689589700 1 -- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6840c2e30 msgr2=0x7fd6841e1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:19.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd689589700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6840c2e30 0x7fd6841e1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:19.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd689589700 1 -- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd68406c6c0 msgr2=0x7fd684072340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:19.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd689589700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd68406c6c0 0x7fd684072340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:19.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd689589700 1 -- 172.21.15.7:0/3927405306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd678009550 con 0x7fd684151cd0 2023-03-03T17:57:19.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd689589700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 0x7fd6840c59f0 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00b9f0 tx=0x7fd67c00bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:57:19.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd6817fa700 1 -- 172.21.15.7:0/3927405306 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c00d8e0 con 0x7fd684151cd0 2023-03-03T17:57:19.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd6817fa700 1 -- 172.21.15.7:0/3927405306 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67c00da40 con 0x7fd684151cd0 2023-03-03T17:57:19.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd6817fa700 1 -- 172.21.15.7:0/3927405306 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c00ddf0 con 0x7fd684151cd0 2023-03-03T17:57:19.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd68afec700 1 -- 172.21.15.7:0/3927405306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd684157c80 con 0x7fd684151cd0 2023-03-03T17:57:19.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.124+0000 7fd68afec700 1 -- 172.21.15.7:0/3927405306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68406d360 con 0x7fd684151cd0 2023-03-03T17:57:19.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.125+0000 7fd68afec700 1 -- 172.21.15.7:0/3927405306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd684154a50 con 0x7fd684151cd0 2023-03-03T17:57:19.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.125+0000 7fd6817fa700 1 -- 172.21.15.7:0/3927405306 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd67c015420 con 0x7fd684151cd0 2023-03-03T17:57:19.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.126+0000 7fd6817fa700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 0x7fd66806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:57:19.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.126+0000 7fd6817fa700 1 -- 172.21.15.7:0/3927405306 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd67c011070 con 0x7fd684151cd0 2023-03-03T17:57:19.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.126+0000 7fd688d88700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 0x7fd66806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:57:19.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.127+0000 7fd688d88700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 0x7fd66806a7a0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fd678009130 tx=0x7fd678008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:57:19.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.130+0000 7fd6817fa700 1 -- 172.21.15.7:0/3927405306 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd67c04d5e0 con 0x7fd684151cd0 2023-03-03T17:57:19.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.364+0000 7fd68afec700 1 -- 172.21.15.7:0/3927405306 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6840c5710 con 0x7fd684151cd0 2023-03-03T17:57:19.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.365+0000 7fd6817fa700 1 -- 172.21.15.7:0/3927405306 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd67c0525a0 con 0x7fd684151cd0 2023-03-03T17:57:19.364 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:57:19.365 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:57:19.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 -- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 msgr2=0x7fd66806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:19.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 0x7fd66806a7a0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fd678009130 tx=0x7fd678008030 comp rx=0 tx=0).stop 2023-03-03T17:57:19.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 -- 172.21.15.7:0/3927405306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 msgr2=0x7fd6840c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:57:19.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 0x7fd6840c59f0 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00b9f0 tx=0x7fd67c00bd00 comp rx=0 tx=0).stop 2023-03-03T17:57:19.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 -- 172.21.15.7:0/3927405306 shutdown_connections 2023-03-03T17:57:19.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6680682f0 0x7fd66806a7a0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:19.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd684151cd0 0x7fd6840c59f0 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:19.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd68406c6c0 0x7fd684072340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:19.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 --2- 172.21.15.7:0/3927405306 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6840c2e30 0x7fd6841e1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:57:19.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 -- 172.21.15.7:0/3927405306 >> 172.21.15.7:0/3927405306 conn(0x7fd6840a94d0 msgr2=0x7fd684152e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:57:19.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 -- 172.21.15.7:0/3927405306 shutdown_connections 2023-03-03T17:57:19.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:57:19.367+0000 7fd68afec700 1 -- 172.21.15.7:0/3927405306 wait complete. 2023-03-03T17:57:19.369 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:57:19.378 INFO:tasks.mds_thrash.fs.[a]:waiting for 102 secs before thrashing 2023-03-03T17:57:32.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:57:32.927 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:03.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:03.007 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:28.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:58:29.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.092+0000 7fa842577700 1 Processor -- start 2023-03-03T17:58:29.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.092+0000 7fa842577700 1 -- start start 2023-03-03T17:58:29.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa842577700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83c151b20 0x7fa83c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:58:29.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa842577700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa83c0c2950 0x7fa83c157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:58:29.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa842577700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa83c073c30 0x7fa83c158020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:58:29.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa842577700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83c06c2f0 con 0x7fa83c073c30 2023-03-03T17:58:29.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa842577700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83c06cb70 con 0x7fa83c0c2950 2023-03-03T17:58:29.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa842577700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c06ce50 con 0x7fa83c151b20 2023-03-03T17:58:29.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa840b14700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa83c073c30 0x7fa83c158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:58:29.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa840b14700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa83c073c30 0x7fa83c158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35314/0 (socket says 172.21.15.7:35314) 2023-03-03T17:58:29.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa840b14700 1 -- 172.21.15.7:0/4131867353 learned_addr learned my addr 172.21.15.7:0/4131867353 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:58:29.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa83bfff700 1 --2- 172.21.15.7:0/4131867353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83c151b20 0x7fa83c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:58:29.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa83b7fe700 1 --2- 172.21.15.7:0/4131867353 >> v2:172.21.15.143:3300/0 conn(0x7fa83c0c2950 0x7fa83c157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:58:29.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa840b14700 1 -- 172.21.15.7:0/4131867353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83c151b20 msgr2=0x7fa83c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:58:29.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa840b14700 1 --2- 172.21.15.7:0/4131867353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83c151b20 0x7fa83c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa840b14700 1 -- 172.21.15.7:0/4131867353 >> v2:172.21.15.143:3300/0 conn(0x7fa83c0c2950 msgr2=0x7fa83c157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:58:29.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa840b14700 1 --2- 172.21.15.7:0/4131867353 >> v2:172.21.15.143:3300/0 conn(0x7fa83c0c2950 0x7fa83c157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa840b14700 1 -- 172.21.15.7:0/4131867353 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c06d130 con 0x7fa83c073c30 2023-03-03T17:58:29.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.093+0000 7fa840b14700 1 --2- 172.21.15.7:0/4131867353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c073c30 0x7fa83c158020 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fa83c06b7e0 tx=0x7fa83400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660932a2339339bc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:58:29.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa83affd700 1 -- 172.21.15.7:0/4131867353 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83400e070 con 0x7fa83c073c30 2023-03-03T17:58:29.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa83affd700 1 -- 172.21.15.7:0/4131867353 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83400b9c0 con 0x7fa83c073c30 2023-03-03T17:58:29.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa83affd700 1 -- 172.21.15.7:0/4131867353 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83400bd70 con 0x7fa83c073c30 2023-03-03T17:58:29.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa842577700 1 -- 172.21.15.7:0/4131867353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c073c30 msgr2=0x7fa83c158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:58:29.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa842577700 1 --2- 172.21.15.7:0/4131867353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c073c30 0x7fa83c158020 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fa83c06b7e0 tx=0x7fa83400b330 comp rx=0 tx=0).stop 2023-03-03T17:58:29.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa842577700 1 -- 172.21.15.7:0/4131867353 shutdown_connections 2023-03-03T17:58:29.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa842577700 1 --2- 172.21.15.7:0/4131867353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c073c30 0x7fa83c158020 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa842577700 1 --2- 172.21.15.7:0/4131867353 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83c151b20 0x7fa83c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa842577700 1 --2- 172.21.15.7:0/4131867353 >> v2:172.21.15.143:3300/0 conn(0x7fa83c0c2950 0x7fa83c157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa842577700 1 -- 172.21.15.7:0/4131867353 >> 172.21.15.7:0/4131867353 conn(0x7fa83c0a94d0 msgr2=0x7fa83c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:58:29.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa842577700 1 -- 172.21.15.7:0/4131867353 shutdown_connections 2023-03-03T17:58:29.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.094+0000 7fa842577700 1 -- 172.21.15.7:0/4131867353 wait complete. 2023-03-03T17:58:29.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.095+0000 7fa842577700 1 Processor -- start 2023-03-03T17:58:29.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.096+0000 7fa842577700 1 -- start start 2023-03-03T17:58:29.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.096+0000 7fa842577700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83c073c30 0x7fa83c0aa440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:58:29.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.096+0000 7fa842577700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c0c2950 0x7fa83c153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:58:29.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.096+0000 7fa842577700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83c151b20 0x7fa83c153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:58:29.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.096+0000 7fa842577700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c1553b0 con 0x7fa83c0c2950 2023-03-03T17:58:29.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.096+0000 7fa842577700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0abd00 con 0x7fa83c073c30 2023-03-03T17:58:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.096+0000 7fa842577700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83c0acf30 con 0x7fa83c151b20 2023-03-03T17:58:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.096+0000 7fa83b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c0c2950 0x7fa83c153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:58:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.096+0000 7fa83b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c0c2950 0x7fa83c153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35318/0 (socket says 172.21.15.7:35318) 2023-03-03T17:58:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.096+0000 7fa83b7fe700 1 -- 172.21.15.7:0/566695436 learned_addr learned my addr 172.21.15.7:0/566695436 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:58:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa840b14700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83c151b20 0x7fa83c153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:58:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa83bfff700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83c073c30 0x7fa83c0aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:58:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa83b7fe700 1 -- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83c151b20 msgr2=0x7fa83c153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:58:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa83b7fe700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83c151b20 0x7fa83c153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa83b7fe700 1 -- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83c073c30 msgr2=0x7fa83c0aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:58:29.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa83b7fe700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83c073c30 0x7fa83c0aa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa83b7fe700 1 -- 172.21.15.7:0/566695436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa834009b30 con 0x7fa83c0c2950 2023-03-03T17:58:29.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa83bfff700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83c073c30 0x7fa83c0aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T17:58:29.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa83b7fe700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c0c2950 0x7fa83c153150 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00d800 tx=0x7fa82c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:58:29.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa838ff9700 1 -- 172.21.15.7:0/566695436 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0125f0 con 0x7fa83c0c2950 2023-03-03T17:58:29.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa838ff9700 1 -- 172.21.15.7:0/566695436 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82c012750 con 0x7fa83c0c2950 2023-03-03T17:58:29.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa842577700 1 -- 172.21.15.7:0/566695436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa83c151ef0 con 0x7fa83c0c2950 2023-03-03T17:58:29.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa838ff9700 1 -- 172.21.15.7:0/566695436 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c012b00 con 0x7fa83c0c2950 2023-03-03T17:58:29.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.097+0000 7fa842577700 1 -- 172.21.15.7:0/566695436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa83c06d410 con 0x7fa83c0c2950 2023-03-03T17:58:29.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.098+0000 7fa842577700 1 -- 172.21.15.7:0/566695436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa83c06bea0 con 0x7fa83c0c2950 2023-03-03T17:58:29.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.099+0000 7fa838ff9700 1 -- 172.21.15.7:0/566695436 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa82c012c60 con 0x7fa83c0c2950 2023-03-03T17:58:29.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.099+0000 7fa838ff9700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 0x7fa81c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:58:29.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.099+0000 7fa838ff9700 1 -- 172.21.15.7:0/566695436 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa82c0865b0 con 0x7fa83c0c2950 2023-03-03T17:58:29.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.099+0000 7fa83bfff700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 0x7fa81c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:58:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.100+0000 7fa83bfff700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 0x7fa81c06a7a0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0abbc0 tx=0x7fa824008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:58:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.103+0000 7fa838ff9700 1 -- 172.21.15.7:0/566695436 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa82c052950 con 0x7fa83c0c2950 2023-03-03T17:58:29.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.341+0000 7fa842577700 1 -- 172.21.15.7:0/566695436 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa83c074bb0 con 0x7fa83c0c2950 2023-03-03T17:58:29.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.342+0000 7fa838ff9700 1 -- 172.21.15.7:0/566695436 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa82c055f70 con 0x7fa83c0c2950 2023-03-03T17:58:29.342 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:58:29.342 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:58:29.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 -- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 msgr2=0x7fa81c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:58:29.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 0x7fa81c06a7a0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0abbc0 tx=0x7fa824008040 comp rx=0 tx=0).stop 2023-03-03T17:58:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 -- 172.21.15.7:0/566695436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c0c2950 msgr2=0x7fa83c153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:58:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c0c2950 0x7fa83c153150 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00d800 tx=0x7fa82c00db10 comp rx=0 tx=0).stop 2023-03-03T17:58:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 -- 172.21.15.7:0/566695436 shutdown_connections 2023-03-03T17:58:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83c151b20 0x7fa83c153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83c073c30 0x7fa83c0aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 0x7fa81c06a7a0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 --2- 172.21.15.7:0/566695436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83c0c2950 0x7fa83c153150 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:58:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 -- 172.21.15.7:0/566695436 >> 172.21.15.7:0/566695436 conn(0x7fa83c0a94d0 msgr2=0x7fa83c0c6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:58:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 -- 172.21.15.7:0/566695436 shutdown_connections 2023-03-03T17:58:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:58:29.344+0000 7fa842577700 1 -- 172.21.15.7:0/566695436 wait complete. 2023-03-03T17:58:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:58:29.355 INFO:tasks.mds_thrash.fs.[b]:waiting for 102 secs before thrashing 2023-03-03T17:58:33.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:58:33.071 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:01.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T17:59:01.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.519+0000 7fd583fec700 1 Processor -- start 2023-03-03T17:59:01.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.519+0000 7fd583fec700 1 -- start start 2023-03-03T17:59:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd583fec700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd57c14e560 0x7fd57c071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:59:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd583fec700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd57c0c43d0 0x7fd57c0c4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:59:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd583fec700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd57c072920 con 0x7fd57c154590 2023-03-03T17:59:01.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd583fec700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c06bef0 con 0x7fd57c0c43d0 2023-03-03T17:59:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd583fec700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57c06b910 con 0x7fd57c14e560 2023-03-03T17:59:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd581587700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd57c0c43d0 0x7fd57c0c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:59:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd582589700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd57c154590 0x7fd57c1549b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57554/0 (socket says 172.21.15.7:57554) 2023-03-03T17:59:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd582589700 1 -- 172.21.15.7:0/1702930744 learned_addr learned my addr 172.21.15.7:0/1702930744 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T17:59:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd581d88700 1 --2- 172.21.15.7:0/1702930744 >> v2:172.21.15.143:3300/0 conn(0x7fd57c14e560 0x7fd57c071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:59:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd581587700 1 -- 172.21.15.7:0/1702930744 >> v2:172.21.15.143:3300/0 conn(0x7fd57c14e560 msgr2=0x7fd57c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T17:59:01.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd581587700 1 --2- 172.21.15.7:0/1702930744 >> v2:172.21.15.143:3300/0 conn(0x7fd57c14e560 0x7fd57c071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:59:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd581587700 1 -- 172.21.15.7:0/1702930744 >> v1:172.21.15.7:6789/0 conn(0x7fd57c154590 legacy=0x7fd57c1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:59:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd580d86700 1 -- 172.21.15.7:0/1702930744 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328713070 0 0) 0x7fd574001490 con 0x7fd57c154590 2023-03-03T17:59:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd581587700 1 -- 172.21.15.7:0/1702930744 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0aacc0 con 0x7fd57c0c43d0 2023-03-03T17:59:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.520+0000 7fd581587700 1 --2- 172.21.15.7:0/1702930744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c43d0 0x7fd57c0c4850 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0c5e80 tx=0x7fd578009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54f66078a2c1bd6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:59:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd580d86700 1 -- 172.21.15.7:0/1702930744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578011830 con 0x7fd57c0c43d0 2023-03-03T17:59:01.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd580d86700 1 -- 172.21.15.7:0/1702930744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd578011990 con 0x7fd57c0c43d0 2023-03-03T17:59:01.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd580d86700 1 -- 172.21.15.7:0/1702930744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578011d20 con 0x7fd57c0c43d0 2023-03-03T17:59:01.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd583fec700 1 -- 172.21.15.7:0/1702930744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c43d0 msgr2=0x7fd57c0c4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:59:01.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd583fec700 1 --2- 172.21.15.7:0/1702930744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c43d0 0x7fd57c0c4850 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0c5e80 tx=0x7fd578009b20 comp rx=0 tx=0).stop 2023-03-03T17:59:01.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd583fec700 1 -- 172.21.15.7:0/1702930744 shutdown_connections 2023-03-03T17:59:01.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd583fec700 1 --2- 172.21.15.7:0/1702930744 >> v2:172.21.15.143:3300/0 conn(0x7fd57c14e560 0x7fd57c071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:59:01.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd583fec700 1 --2- 172.21.15.7:0/1702930744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c0c43d0 0x7fd57c0c4850 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:59:01.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd583fec700 1 -- 172.21.15.7:0/1702930744 >> 172.21.15.7:0/1702930744 conn(0x7fd57c0736f0 msgr2=0x7fd57c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:59:01.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd583fec700 1 -- 172.21.15.7:0/1702930744 shutdown_connections 2023-03-03T17:59:01.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.521+0000 7fd583fec700 1 -- 172.21.15.7:0/1702930744 wait complete. 2023-03-03T17:59:01.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.522+0000 7fd583fec700 1 Processor -- start 2023-03-03T17:59:01.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.522+0000 7fd583fec700 1 -- start start 2023-03-03T17:59:01.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd583fec700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd57c0c43d0 0x7fd57c14a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:59:01.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd583fec700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c14e560 0x7fd57c1ed430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:59:01.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd583fec700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c154590 0x7fd57c1ed970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:59:01.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd583fec700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c1e0b60 con 0x7fd57c154590 2023-03-03T17:59:01.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd583fec700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c1dfde0 con 0x7fd57c0c43d0 2023-03-03T17:59:01.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd583fec700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c14dc10 con 0x7fd57c14e560 2023-03-03T17:59:01.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd582589700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c154590 0x7fd57c1ed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:59:01.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd582589700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c154590 0x7fd57c1ed970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35348/0 (socket says 172.21.15.7:35348) 2023-03-03T17:59:01.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd582589700 1 -- 172.21.15.7:0/2591390835 learned_addr learned my addr 172.21.15.7:0/2591390835 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T17:59:01.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd581d88700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd57c0c43d0 0x7fd57c14a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:59:01.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.523+0000 7fd581587700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c14e560 0x7fd57c1ed430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:59:01.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd582589700 1 -- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c14e560 msgr2=0x7fd57c1ed430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:59:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd582589700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c14e560 0x7fd57c1ed430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:59:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd582589700 1 -- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd57c0c43d0 msgr2=0x7fd57c14a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:59:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd582589700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd57c0c43d0 0x7fd57c14a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:59:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd582589700 1 -- 172.21.15.7:0/2591390835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5780094a0 con 0x7fd57c154590 2023-03-03T17:59:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd582589700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c154590 0x7fd57c1ed970 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7fd57400cba0 tx=0x7fd57400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:59:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd5727fc700 1 -- 172.21.15.7:0/2591390835 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57400aaa0 con 0x7fd57c154590 2023-03-03T17:59:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd5727fc700 1 -- 172.21.15.7:0/2591390835 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57400ac20 con 0x7fd57c154590 2023-03-03T17:59:01.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd583fec700 1 -- 172.21.15.7:0/2591390835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c0aa970 con 0x7fd57c154590 2023-03-03T17:59:01.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd5727fc700 1 -- 172.21.15.7:0/2591390835 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5740070d0 con 0x7fd57c154590 2023-03-03T17:59:01.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.524+0000 7fd583fec700 1 -- 172.21.15.7:0/2591390835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c06bb80 con 0x7fd57c154590 2023-03-03T17:59:01.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.525+0000 7fd583fec700 1 -- 172.21.15.7:0/2591390835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c0aafa0 con 0x7fd57c154590 2023-03-03T17:59:01.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.525+0000 7fd5727fc700 1 -- 172.21.15.7:0/2591390835 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd57401a030 con 0x7fd57c154590 2023-03-03T17:59:01.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.526+0000 7fd5727fc700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 0x7fd56006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T17:59:01.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.526+0000 7fd5727fc700 1 -- 172.21.15.7:0/2591390835 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd574085c20 con 0x7fd57c154590 2023-03-03T17:59:01.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.526+0000 7fd581d88700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 0x7fd56006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T17:59:01.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.526+0000 7fd581d88700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 0x7fd56006a7a0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fd56c006b20 tx=0x7fd56c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T17:59:01.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.530+0000 7fd5727fc700 1 -- 172.21.15.7:0/2591390835 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd574051ed0 con 0x7fd57c154590 2023-03-03T17:59:01.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.765+0000 7fd583fec700 1 -- 172.21.15.7:0/2591390835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd57c074bb0 con 0x7fd57c154590 2023-03-03T17:59:01.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.766+0000 7fd5727fc700 1 -- 172.21.15.7:0/2591390835 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd574018020 con 0x7fd57c154590 2023-03-03T17:59:01.766 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T17:59:01.766 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T17:59:01.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 -- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 msgr2=0x7fd56006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:59:01.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 0x7fd56006a7a0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fd56c006b20 tx=0x7fd56c008040 comp rx=0 tx=0).stop 2023-03-03T17:59:01.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 -- 172.21.15.7:0/2591390835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c154590 msgr2=0x7fd57c1ed970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T17:59:01.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c154590 0x7fd57c1ed970 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7fd57400cba0 tx=0x7fd57400ceb0 comp rx=0 tx=0).stop 2023-03-03T17:59:01.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 -- 172.21.15.7:0/2591390835 shutdown_connections 2023-03-03T17:59:01.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5600682f0 0x7fd56006a7a0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:59:01.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd57c154590 0x7fd57c1ed970 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:59:01.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd57c0c43d0 0x7fd57c14a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:59:01.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 --2- 172.21.15.7:0/2591390835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd57c14e560 0x7fd57c1ed430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T17:59:01.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 -- 172.21.15.7:0/2591390835 >> 172.21.15.7:0/2591390835 conn(0x7fd57c0736f0 msgr2=0x7fd57c0aa060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T17:59:01.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 -- 172.21.15.7:0/2591390835 shutdown_connections 2023-03-03T17:59:01.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T17:59:01.768+0000 7fd583fec700 1 -- 172.21.15.7:0/2591390835 wait complete. 2023-03-03T17:59:01.769 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T17:59:01.779 INFO:tasks.mds_thrash.fs.[a]:waiting for 99 secs before thrashing 2023-03-03T17:59:03.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:03.159 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:33.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:33.228 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T17:59:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T17:59:59.998+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T18:00:03.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:03.311 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:11.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:00:11.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.496+0000 7f35b1403700 1 Processor -- start 2023-03-03T18:00:11.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.496+0000 7f35b1403700 1 -- start start 2023-03-03T18:00:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35b1403700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac0c2e30 0x7f35ac158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35b1403700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f35ac06c6c0 0x7f35ac0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:11.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35b1403700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ac157ff0 con 0x7f35ac151cd0 2023-03-03T18:00:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35b1403700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ac06d640 con 0x7f35ac06c6c0 2023-03-03T18:00:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35b1403700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac06cda0 con 0x7f35ac0c2e30 2023-03-03T18:00:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35ab7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f35ac06c6c0 0x7f35ac0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35aa7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac0c2e30 0x7f35ac158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35ab7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f35ac06c6c0 0x7f35ac0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:59404/0 (socket says 172.21.15.7:59404) 2023-03-03T18:00:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35aaffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f35ac151cd0 0x7f35ac154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:32894/0 (socket says 172.21.15.7:32894) 2023-03-03T18:00:11.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35ab7fe700 1 -- 172.21.15.7:0/4220325455 learned_addr learned my addr 172.21.15.7:0/4220325455 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:00:11.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35ab7fe700 1 -- 172.21.15.7:0/4220325455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac0c2e30 msgr2=0x7f35ac158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:00:11.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35ab7fe700 1 --2- 172.21.15.7:0/4220325455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac0c2e30 0x7f35ac158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35ab7fe700 1 -- 172.21.15.7:0/4220325455 >> v1:172.21.15.143:6789/0 conn(0x7f35ac151cd0 legacy=0x7f35ac154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:11.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35ab7fe700 1 -- 172.21.15.7:0/4220325455 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac06d080 con 0x7f35ac06c6c0 2023-03-03T18:00:11.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.497+0000 7f35aa7fc700 1 --2- 172.21.15.7:0/4220325455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac0c2e30 0x7f35ac158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:00:11.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.498+0000 7f35ab7fe700 1 --2- 172.21.15.7:0/4220325455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ac06c6c0 0x7f35ac0736f0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009e50 tx=0x7f35a400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cf2fd7775f22cdc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:00:11.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.498+0000 7f35a9ffb700 1 -- 172.21.15.7:0/4220325455 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a400e070 con 0x7f35ac06c6c0 2023-03-03T18:00:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.498+0000 7f35a9ffb700 1 -- 172.21.15.7:0/4220325455 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a400b910 con 0x7f35ac06c6c0 2023-03-03T18:00:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.498+0000 7f35a9ffb700 1 -- 172.21.15.7:0/4220325455 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a400bcc0 con 0x7f35ac06c6c0 2023-03-03T18:00:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.499+0000 7f35b1403700 1 -- 172.21.15.7:0/4220325455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ac06c6c0 msgr2=0x7f35ac0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.499+0000 7f35b1403700 1 --2- 172.21.15.7:0/4220325455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ac06c6c0 0x7f35ac0736f0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009e50 tx=0x7f35a400b330 comp rx=0 tx=0).stop 2023-03-03T18:00:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.499+0000 7f35b1403700 1 -- 172.21.15.7:0/4220325455 shutdown_connections 2023-03-03T18:00:11.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.499+0000 7f35b1403700 1 --2- 172.21.15.7:0/4220325455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ac06c6c0 0x7f35ac0736f0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.499+0000 7f35b1403700 1 --2- 172.21.15.7:0/4220325455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac0c2e30 0x7f35ac158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.499+0000 7f35b1403700 1 -- 172.21.15.7:0/4220325455 >> 172.21.15.7:0/4220325455 conn(0x7f35ac0a94d0 msgr2=0x7f35ac0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:00:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.499+0000 7f35b1403700 1 -- 172.21.15.7:0/4220325455 shutdown_connections 2023-03-03T18:00:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.499+0000 7f35b1403700 1 -- 172.21.15.7:0/4220325455 wait complete. 2023-03-03T18:00:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.500+0000 7f35b1403700 1 Processor -- start 2023-03-03T18:00:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.500+0000 7f35b1403700 1 -- start start 2023-03-03T18:00:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35b1403700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac06c6c0 0x7f35ac1ea770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:11.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35b1403700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ac0c2e30 0x7f35ac072000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35b1403700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ac151cd0 0x7f35ac1ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35b1403700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac153470 con 0x7f35ac0c2e30 2023-03-03T18:00:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35b1403700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac156600 con 0x7f35ac151cd0 2023-03-03T18:00:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35b1403700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac1edf10 con 0x7f35ac06c6c0 2023-03-03T18:00:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35aa7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ac0c2e30 0x7f35ac072000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35aa7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ac0c2e30 0x7f35ac072000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35402/0 (socket says 172.21.15.7:35402) 2023-03-03T18:00:11.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35aa7fc700 1 -- 172.21.15.7:0/531285835 learned_addr learned my addr 172.21.15.7:0/531285835 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:00:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35aaffd700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac06c6c0 0x7f35ac1ea770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.501+0000 7f35ab7fe700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ac151cd0 0x7f35ac1ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35aa7fc700 1 -- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac06c6c0 msgr2=0x7f35ac1ea770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35aa7fc700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac06c6c0 0x7f35ac1ea770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35aa7fc700 1 -- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ac151cd0 msgr2=0x7f35ac1ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35aa7fc700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ac151cd0 0x7f35ac1ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35aa7fc700 1 -- 172.21.15.7:0/531285835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a00094a0 con 0x7f35ac0c2e30 2023-03-03T18:00:11.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35aa7fc700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ac0c2e30 0x7f35ac072000 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f35a000ebe0 tx=0x7f35a000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:00:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35937fe700 1 -- 172.21.15.7:0/531285835 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0009760 con 0x7f35ac0c2e30 2023-03-03T18:00:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35937fe700 1 -- 172.21.15.7:0/531285835 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a000c850 con 0x7f35ac0c2e30 2023-03-03T18:00:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35937fe700 1 -- 172.21.15.7:0/531285835 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a000cbe0 con 0x7f35ac0c2e30 2023-03-03T18:00:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35b1403700 1 -- 172.21.15.7:0/531285835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a4009b30 con 0x7f35ac0c2e30 2023-03-03T18:00:11.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.502+0000 7f35b1403700 1 -- 172.21.15.7:0/531285835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ac157c80 con 0x7f35ac0c2e30 2023-03-03T18:00:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.503+0000 7f35b1403700 1 -- 172.21.15.7:0/531285835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ac06d360 con 0x7f35ac0c2e30 2023-03-03T18:00:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.503+0000 7f35937fe700 1 -- 172.21.15.7:0/531285835 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f35a0019030 con 0x7f35ac0c2e30 2023-03-03T18:00:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.504+0000 7f35937fe700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35980682f0 0x7f359806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:11.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.504+0000 7f35937fe700 1 -- 172.21.15.7:0/531285835 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f35a0085f90 con 0x7f35ac0c2e30 2023-03-03T18:00:11.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.504+0000 7f35aaffd700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35980682f0 0x7f359806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:11.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.505+0000 7f35aaffd700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35980682f0 0x7f359806a7a0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f359c000f60 tx=0x7f359c007ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:00:11.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.508+0000 7f35937fe700 1 -- 172.21.15.7:0/531285835 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f35a0052330 con 0x7f35ac0c2e30 2023-03-03T18:00:11.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.746+0000 7f35b1403700 1 -- 172.21.15.7:0/531285835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35ac1589c0 con 0x7f35ac0c2e30 2023-03-03T18:00:11.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.746+0000 7f35937fe700 1 -- 172.21.15.7:0/531285835 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f35a0055950 con 0x7f35ac0c2e30 2023-03-03T18:00:11.746 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:00:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:00:11.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 -- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35980682f0 msgr2=0x7f359806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:11.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35980682f0 0x7f359806a7a0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f359c000f60 tx=0x7f359c007ed0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 -- 172.21.15.7:0/531285835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ac0c2e30 msgr2=0x7f35ac072000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:11.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ac0c2e30 0x7f35ac072000 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f35a000ebe0 tx=0x7f35a000eef0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 -- 172.21.15.7:0/531285835 shutdown_connections 2023-03-03T18:00:11.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35ac151cd0 0x7f35ac1ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35ac06c6c0 0x7f35ac1ea770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f35980682f0 0x7f359806a7a0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 --2- 172.21.15.7:0/531285835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35ac0c2e30 0x7f35ac072000 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:11.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 -- 172.21.15.7:0/531285835 >> 172.21.15.7:0/531285835 conn(0x7f35ac0a94d0 msgr2=0x7f35ac065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:00:11.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.748+0000 7f35b1403700 1 -- 172.21.15.7:0/531285835 shutdown_connections 2023-03-03T18:00:11.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:11.749+0000 7f35b1403700 1 -- 172.21.15.7:0/531285835 wait complete. 2023-03-03T18:00:11.750 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:00:11.760 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before thrashing 2023-03-03T18:00:15.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:00:15.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.903+0000 7fd3ecd9e700 1 Processor -- start 2023-03-03T18:00:15.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.903+0000 7fd3ecd9e700 1 -- start start 2023-03-03T18:00:15.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3ecd9e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd3e80719d0 0x7fd3e81522f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:15.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3ecd9e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e814fd90 con 0x7fd3e80ade30 2023-03-03T18:00:15.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3ecd9e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e80c4790 con 0x7fd3e814f9c0 2023-03-03T18:00:15.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3ecd9e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e80c32b0 con 0x7fd3e80719d0 2023-03-03T18:00:15.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd3e80ade30 0x7fd3e8152830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57624/0 (socket says 172.21.15.7:57624) 2023-03-03T18:00:15.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e7fff700 1 -- 172.21.15.7:0/623061299 learned_addr learned my addr 172.21.15.7:0/623061299 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:00:15.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e6ffd700 1 --2- 172.21.15.7:0/623061299 >> v2:172.21.15.143:3300/0 conn(0x7fd3e80719d0 0x7fd3e81522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:15.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632945067 0 0) 0x7fd3e0001490 con 0x7fd3e80ade30 2023-03-03T18:00:15.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d40035b0 con 0x7fd3e80ade30 2023-03-03T18:00:15.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855424215 0 0) 0x7fd3d40035b0 con 0x7fd3e80ade30 2023-03-03T18:00:15.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 >> v2:172.21.15.143:3300/0 conn(0x7fd3e80719d0 msgr2=0x7fd3e81522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:00:15.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 --2- 172.21.15.7:0/623061299 >> v2:172.21.15.143:3300/0 conn(0x7fd3e80719d0 0x7fd3e81522f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:15.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 >> v1:172.21.15.143:6789/0 conn(0x7fd3e814f9c0 legacy=0x7fd3e80c4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:15.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e8071f40 con 0x7fd3e80ade30 2023-03-03T18:00:15.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd3e0001b80 con 0x7fd3e80ade30 2023-03-03T18:00:15.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 >> v1:172.21.15.7:6789/0 conn(0x7fd3e80ade30 legacy=0x7fd3e8152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:15.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3d4003ae0 0x7fd3d4003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:15.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3d4004850 0x7fd3d4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:15.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3d4005320 0x7fd3d400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:15.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80c32b0 con 0x7fd3d4005320 2023-03-03T18:00:15.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d400c200 con 0x7fd3d4003ae0 2023-03-03T18:00:15.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.905+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d400c3c0 con 0x7fd3d4004850 2023-03-03T18:00:15.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e7fff700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3d4005320 0x7fd3d400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:15.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e77fe700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3d4003ae0 0x7fd3d4003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:15.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e6ffd700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3d4004850 0x7fd3d4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:15.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e7fff700 1 -- 172.21.15.7:0/623061299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3d4004850 msgr2=0x7fd3d4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:00:15.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e7fff700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3d4004850 0x7fd3d4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:15.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e7fff700 1 -- 172.21.15.7:0/623061299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3d4003ae0 msgr2=0x7fd3d4003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:00:15.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e7fff700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3d4003ae0 0x7fd3d4003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:15.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e7fff700 1 -- 172.21.15.7:0/623061299 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d40035b0 con 0x7fd3d4005320 2023-03-03T18:00:15.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e7fff700 1 -- 172.21.15.7:0/623061299 reap_dead start 2023-03-03T18:00:15.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e7fff700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3d4005320 0x7fd3d400ba10 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000a830 tx=0x7fd3d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9ab376c28e88ea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:00:15.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e000a490 con 0x7fd3d4005320 2023-03-03T18:00:15.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e0012620 con 0x7fd3d4005320 2023-03-03T18:00:15.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.906+0000 7fd3e67fc700 1 -- 172.21.15.7:0/623061299 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e00129d0 con 0x7fd3d4005320 2023-03-03T18:00:15.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.907+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/623061299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3d4005320 msgr2=0x7fd3d400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:15.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.907+0000 7fd3ecd9e700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3d4005320 0x7fd3d400ba10 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000a830 tx=0x7fd3d4005880 comp rx=0 tx=0).stop 2023-03-03T18:00:15.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.907+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/623061299 shutdown_connections 2023-03-03T18:00:15.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.907+0000 7fd3ecd9e700 1 --2- 172.21.15.7:0/623061299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3d4005320 0x7fd3d400ba10 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:15.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.907+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/623061299 >> 172.21.15.7:0/623061299 conn(0x7fd3e806c4c0 msgr2=0x7fd3e806c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:00:15.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.907+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/623061299 shutdown_connections 2023-03-03T18:00:15.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.907+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/623061299 wait complete. 2023-03-03T18:00:15.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.908+0000 7fd3ecd9e700 1 Processor -- start 2023-03-03T18:00:15.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.908+0000 7fd3ecd9e700 1 -- start start 2023-03-03T18:00:15.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.909+0000 7fd3ecd9e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3d4005320 0x7fd3e80c1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:15.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.909+0000 7fd3ecd9e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3e8150140 0x7fd3e814ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:15.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.909+0000 7fd3ecd9e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e806bfb0 0x7fd3e814b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:15.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.909+0000 7fd3ecd9e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e81535b0 con 0x7fd3e8150140 2023-03-03T18:00:15.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.909+0000 7fd3ecd9e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e8152e90 con 0x7fd3d4005320 2023-03-03T18:00:15.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.909+0000 7fd3ecd9e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e81edb00 con 0x7fd3e806bfb0 2023-03-03T18:00:15.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.909+0000 7fd3e6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3e8150140 0x7fd3e814ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:15.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.909+0000 7fd3e6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3e8150140 0x7fd3e814ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35422/0 (socket says 172.21.15.7:35422) 2023-03-03T18:00:15.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.909+0000 7fd3e6ffd700 1 -- 172.21.15.7:0/2767921055 learned_addr learned my addr 172.21.15.7:0/2767921055 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:00:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.909+0000 7fd3e77fe700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3d4005320 0x7fd3e80c1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3e7fff700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e806bfb0 0x7fd3e814b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3e6ffd700 1 -- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e806bfb0 msgr2=0x7fd3e814b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:15.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3e6ffd700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e806bfb0 0x7fd3e814b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3e6ffd700 1 -- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3d4005320 msgr2=0x7fd3e80c1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3e6ffd700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3d4005320 0x7fd3e80c1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3e6ffd700 1 -- 172.21.15.7:0/2767921055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e8156700 con 0x7fd3e8150140 2023-03-03T18:00:15.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3e6ffd700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3e8150140 0x7fd3e814ac20 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7fd3d80119a0 tx=0x7fd3d8011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:00:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3dffff700 1 -- 172.21.15.7:0/2767921055 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d800f940 con 0x7fd3e8150140 2023-03-03T18:00:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3dffff700 1 -- 172.21.15.7:0/2767921055 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d800faa0 con 0x7fd3e8150140 2023-03-03T18:00:15.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/2767921055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e8156420 con 0x7fd3e8150140 2023-03-03T18:00:15.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3dffff700 1 -- 172.21.15.7:0/2767921055 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d800fe30 con 0x7fd3e8150140 2023-03-03T18:00:15.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.910+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/2767921055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e80c4390 con 0x7fd3e8150140 2023-03-03T18:00:15.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.911+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/2767921055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e8061f40 con 0x7fd3e8150140 2023-03-03T18:00:15.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.912+0000 7fd3dffff700 1 -- 172.21.15.7:0/2767921055 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd3d8014440 con 0x7fd3e8150140 2023-03-03T18:00:15.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.912+0000 7fd3dffff700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3c80682f0 0x7fd3c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:15.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.912+0000 7fd3dffff700 1 -- 172.21.15.7:0/2767921055 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd3d8086960 con 0x7fd3e8150140 2023-03-03T18:00:15.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.912+0000 7fd3e77fe700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3c80682f0 0x7fd3c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:15.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.913+0000 7fd3e77fe700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3c80682f0 0x7fd3c806a7a0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0007fd0 tx=0x7fd3d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:00:15.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:15.916+0000 7fd3dffff700 1 -- 172.21.15.7:0/2767921055 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd3d8052ca0 con 0x7fd3e8150140 2023-03-03T18:00:16.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.156+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/2767921055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3e80717b0 con 0x7fd3e8150140 2023-03-03T18:00:16.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.156+0000 7fd3dffff700 1 -- 172.21.15.7:0/2767921055 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd3d80562c0 con 0x7fd3e8150140 2023-03-03T18:00:16.156 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:00:16.156 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:00:16.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.158+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3c80682f0 msgr2=0x7fd3c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:16.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.158+0000 7fd3ecd9e700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3c80682f0 0x7fd3c806a7a0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0007fd0 tx=0x7fd3d0009040 comp rx=0 tx=0).stop 2023-03-03T18:00:16.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.158+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/2767921055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3e8150140 msgr2=0x7fd3e814ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:16.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.158+0000 7fd3ecd9e700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3e8150140 0x7fd3e814ac20 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7fd3d80119a0 tx=0x7fd3d8011d60 comp rx=0 tx=0).stop 2023-03-03T18:00:16.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.158+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/2767921055 shutdown_connections 2023-03-03T18:00:16.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.158+0000 7fd3ecd9e700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3e806bfb0 0x7fd3e814b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:16.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.158+0000 7fd3ecd9e700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3d4005320 0x7fd3e80c1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:16.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.158+0000 7fd3ecd9e700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd3c80682f0 0x7fd3c806a7a0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:16.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.158+0000 7fd3ecd9e700 1 --2- 172.21.15.7:0/2767921055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3e8150140 0x7fd3e814ac20 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:16.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.158+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/2767921055 >> 172.21.15.7:0/2767921055 conn(0x7fd3e806c4c0 msgr2=0x7fd3e80c4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:00:16.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.159+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/2767921055 shutdown_connections 2023-03-03T18:00:16.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:16.159+0000 7fd3ecd9e700 1 -- 172.21.15.7:0/2767921055 wait complete. 2023-03-03T18:00:16.160 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:00:16.170 INFO:tasks.mds_thrash.fs.[b]:waiting for 101 secs before thrashing 2023-03-03T18:00:33.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:33.371 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:00:40.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:00:40.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.921+0000 7f59d081b700 1 Processor -- start 2023-03-03T18:00:40.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.921+0000 7f59d081b700 1 -- start start 2023-03-03T18:00:40.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.922+0000 7f59d081b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59cc14fda0 0x7f59cc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:40.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.922+0000 7f59d081b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc0abe20 0x7f59cc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:40.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.922+0000 7f59d081b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc0c2f30 con 0x7f59cc14f9d0 2023-03-03T18:00:40.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.922+0000 7f59d081b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59cc0c31d0 con 0x7f59cc14fda0 2023-03-03T18:00:40.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.922+0000 7f59d081b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc1524a0 con 0x7f59cc0abe20 2023-03-03T18:00:40.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.922+0000 7f59cad9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc0abe20 0x7f59cc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:40.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.922+0000 7f59c9d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f59cc14fda0 0x7f59cc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:40.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.922+0000 7f59ca59c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f59cc14f9d0 0x7f59cc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:32950/0 (socket says 172.21.15.7:32950) 2023-03-03T18:00:40.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.922+0000 7f59ca59c700 1 -- 172.21.15.7:0/3718252109 learned_addr learned my addr 172.21.15.7:0/3718252109 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:00:40.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.923+0000 7f59c9d9b700 1 -- 172.21.15.7:0/3718252109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc0abe20 msgr2=0x7f59cc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:00:40.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.923+0000 7f59c9d9b700 1 --2- 172.21.15.7:0/3718252109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc0abe20 0x7f59cc1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:40.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.923+0000 7f59c9d9b700 1 -- 172.21.15.7:0/3718252109 >> v1:172.21.15.143:6789/0 conn(0x7f59cc14f9d0 legacy=0x7f59cc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:40.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.923+0000 7f59c9d9b700 1 -- 172.21.15.7:0/3718252109 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc0c34b0 con 0x7f59cc14fda0 2023-03-03T18:00:40.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.923+0000 7f59c959a700 1 -- 172.21.15.7:0/3718252109 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370551630 0 0) 0x7f59cc0c2f30 con 0x7f59cc14f9d0 2023-03-03T18:00:40.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.923+0000 7f59cad9d700 1 --2- 172.21.15.7:0/3718252109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc0abe20 0x7f59cc1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:00:40.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.923+0000 7f59c9d9b700 1 --2- 172.21.15.7:0/3718252109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59cc14fda0 0x7f59cc156430 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009840 tx=0x7f59bc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=566a45e4dc7dc26f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:00:40.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.923+0000 7f59c959a700 1 -- 172.21.15.7:0/3718252109 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59bc011830 con 0x7f59cc14fda0 2023-03-03T18:00:40.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.923+0000 7f59c959a700 1 -- 172.21.15.7:0/3718252109 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59bc011990 con 0x7f59cc14fda0 2023-03-03T18:00:40.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.923+0000 7f59c959a700 1 -- 172.21.15.7:0/3718252109 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59bc011d20 con 0x7f59cc14fda0 2023-03-03T18:00:40.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.924+0000 7f59d081b700 1 -- 172.21.15.7:0/3718252109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59cc14fda0 msgr2=0x7f59cc156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:40.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.924+0000 7f59d081b700 1 --2- 172.21.15.7:0/3718252109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59cc14fda0 0x7f59cc156430 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009840 tx=0x7f59bc009b50 comp rx=0 tx=0).stop 2023-03-03T18:00:40.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.924+0000 7f59d081b700 1 -- 172.21.15.7:0/3718252109 shutdown_connections 2023-03-03T18:00:40.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.924+0000 7f59d081b700 1 --2- 172.21.15.7:0/3718252109 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc0abe20 0x7f59cc1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:40.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.924+0000 7f59d081b700 1 --2- 172.21.15.7:0/3718252109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59cc14fda0 0x7f59cc156430 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:40.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.924+0000 7f59d081b700 1 -- 172.21.15.7:0/3718252109 >> 172.21.15.7:0/3718252109 conn(0x7f59cc0c8900 msgr2=0x7f59cc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:00:40.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.924+0000 7f59d081b700 1 -- 172.21.15.7:0/3718252109 shutdown_connections 2023-03-03T18:00:40.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.924+0000 7f59d081b700 1 -- 172.21.15.7:0/3718252109 wait complete. 2023-03-03T18:00:40.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.925+0000 7f59d081b700 1 Processor -- start 2023-03-03T18:00:40.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.925+0000 7f59d081b700 1 -- start start 2023-03-03T18:00:40.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59d081b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59cc0abe20 0x7f59cc14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:40.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59d081b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc14f9d0 0x7f59cc1ece90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:40.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59d081b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59cc14fda0 0x7f59cc1ed3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:40.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59d081b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc1eca00 con 0x7f59cc0abe20 2023-03-03T18:00:40.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59d081b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc1ea030 con 0x7f59cc14fda0 2023-03-03T18:00:40.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59d081b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc1ea310 con 0x7f59cc14f9d0 2023-03-03T18:00:40.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59ca59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59cc0abe20 0x7f59cc14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:40.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59ca59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59cc0abe20 0x7f59cc14d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35456/0 (socket says 172.21.15.7:35456) 2023-03-03T18:00:40.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59ca59c700 1 -- 172.21.15.7:0/3771073509 learned_addr learned my addr 172.21.15.7:0/3771073509 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:00:40.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59c9d9b700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc14f9d0 0x7f59cc1ece90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:40.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59cad9d700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59cc14fda0 0x7f59cc1ed3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:40.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59ca59c700 1 -- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc14f9d0 msgr2=0x7f59cc1ece90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:40.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59ca59c700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc14f9d0 0x7f59cc1ece90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:40.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.926+0000 7f59ca59c700 1 -- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59cc14fda0 msgr2=0x7f59cc1ed3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:40.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.927+0000 7f59ca59c700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59cc14fda0 0x7f59cc1ed3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:40.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.927+0000 7f59ca59c700 1 -- 172.21.15.7:0/3771073509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c4009a70 con 0x7f59cc0abe20 2023-03-03T18:00:40.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.927+0000 7f59ca59c700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59cc0abe20 0x7f59cc14d080 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f59c00011e0 tx=0x7f59c000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:00:40.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.927+0000 7f59baffd700 1 -- 172.21.15.7:0/3771073509 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0004c80 con 0x7f59cc0abe20 2023-03-03T18:00:40.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.927+0000 7f59baffd700 1 -- 172.21.15.7:0/3771073509 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c0004de0 con 0x7f59cc0abe20 2023-03-03T18:00:40.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.927+0000 7f59baffd700 1 -- 172.21.15.7:0/3771073509 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c0007310 con 0x7f59cc0abe20 2023-03-03T18:00:40.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.927+0000 7f59d081b700 1 -- 172.21.15.7:0/3771073509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59bc0094a0 con 0x7f59cc0abe20 2023-03-03T18:00:40.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.927+0000 7f59d081b700 1 -- 172.21.15.7:0/3771073509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc0ac350 con 0x7f59cc0abe20 2023-03-03T18:00:40.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.928+0000 7f59d081b700 1 -- 172.21.15.7:0/3771073509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59cc0ab630 con 0x7f59cc0abe20 2023-03-03T18:00:40.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.928+0000 7f59baffd700 1 -- 172.21.15.7:0/3771073509 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f59c001a020 con 0x7f59cc0abe20 2023-03-03T18:00:40.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.929+0000 7f59baffd700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59ac0682f0 0x7f59ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:00:40.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.929+0000 7f59baffd700 1 -- 172.21.15.7:0/3771073509 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f59c0085c20 con 0x7f59cc0abe20 2023-03-03T18:00:40.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.929+0000 7f59c9d9b700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59ac0682f0 0x7f59ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:00:40.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.929+0000 7f59c9d9b700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59ac0682f0 0x7f59ac06a7a0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f59cc1501d0 tx=0x7f59bc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:00:40.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:40.933+0000 7f59baffd700 1 -- 172.21.15.7:0/3771073509 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f59c0051fc0 con 0x7f59cc0abe20 2023-03-03T18:00:41.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.170+0000 7f59d081b700 1 -- 172.21.15.7:0/3771073509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59cc071fa0 con 0x7f59cc0abe20 2023-03-03T18:00:41.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.170+0000 7f59baffd700 1 -- 172.21.15.7:0/3771073509 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f59c00555e0 con 0x7f59cc0abe20 2023-03-03T18:00:41.170 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:00:41.170 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:00:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 -- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59ac0682f0 msgr2=0x7f59ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59ac0682f0 0x7f59ac06a7a0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f59cc1501d0 tx=0x7f59bc008030 comp rx=0 tx=0).stop 2023-03-03T18:00:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 -- 172.21.15.7:0/3771073509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59cc0abe20 msgr2=0x7f59cc14d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:00:41.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59cc0abe20 0x7f59cc14d080 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f59c00011e0 tx=0x7f59c000ca30 comp rx=0 tx=0).stop 2023-03-03T18:00:41.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 -- 172.21.15.7:0/3771073509 shutdown_connections 2023-03-03T18:00:41.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f59cc14fda0 0x7f59cc1ed3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:41.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f59ac0682f0 0x7f59ac06a7a0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:41.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59cc0abe20 0x7f59cc14d080 unknown :-1 s=CLOSED pgs=3078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:41.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 --2- 172.21.15.7:0/3771073509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f59cc14f9d0 0x7f59cc1ece90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:00:41.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 -- 172.21.15.7:0/3771073509 >> 172.21.15.7:0/3771073509 conn(0x7f59cc0c8900 msgr2=0x7f59cc1528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:00:41.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 -- 172.21.15.7:0/3771073509 shutdown_connections 2023-03-03T18:00:41.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:00:41.172+0000 7f59d081b700 1 -- 172.21.15.7:0/3771073509 wait complete. 2023-03-03T18:00:41.174 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:00:41.183 INFO:tasks.mds_thrash.fs.[a]:waiting for 34 secs before thrashing 2023-03-03T18:01:03.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:03.445 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:15.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:01:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.324+0000 7f3d4338a700 1 Processor -- start 2023-03-03T18:01:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.325+0000 7f3d4338a700 1 -- start start 2023-03-03T18:01:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.325+0000 7f3d4338a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3d3c0c00e0 0x7f3d3c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.325+0000 7f3d4338a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d3c154ff0 0x7f3d3c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.325+0000 7f3d4338a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c0c7310 con 0x7f3d3c0caa30 2023-03-03T18:01:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.325+0000 7f3d4338a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3c155430 con 0x7f3d3c154ff0 2023-03-03T18:01:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.325+0000 7f3d4338a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3c0bfd10 con 0x7f3d3c0c00e0 2023-03-03T18:01:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d41927700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d3c154ff0 0x7f3d3c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d41927700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d3c154ff0 0x7f3d3c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35476/0 (socket says 172.21.15.7:35476) 2023-03-03T18:01:15.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d41927700 1 -- 172.21.15.7:0/1411808351 learned_addr learned my addr 172.21.15.7:0/1411808351 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:01:15.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d40925700 1 --2- 172.21.15.7:0/1411808351 >> v2:172.21.15.143:3300/0 conn(0x7f3d3c0c00e0 0x7f3d3c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:15.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d41927700 1 -- 172.21.15.7:0/1411808351 >> v2:172.21.15.143:3300/0 conn(0x7f3d3c0c00e0 msgr2=0x7f3d3c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:01:15.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d41927700 1 --2- 172.21.15.7:0/1411808351 >> v2:172.21.15.143:3300/0 conn(0x7f3d3c0c00e0 0x7f3d3c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:15.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d41927700 1 -- 172.21.15.7:0/1411808351 >> v1:172.21.15.143:6789/0 conn(0x7f3d3c0caa30 legacy=0x7f3d3c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:15.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d41927700 1 -- 172.21.15.7:0/1411808351 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c0bfa30 con 0x7f3d3c154ff0 2023-03-03T18:01:15.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d41927700 1 --2- 172.21.15.7:0/1411808351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c154ff0 0x7f3d3c0cae00 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0ca910 tx=0x7f3d3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff904af03612934d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:01:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d2ffff700 1 -- 172.21.15.7:0/1411808351 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3400e070 con 0x7f3d3c154ff0 2023-03-03T18:01:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d2ffff700 1 -- 172.21.15.7:0/1411808351 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3400b910 con 0x7f3d3c154ff0 2023-03-03T18:01:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.326+0000 7f3d2ffff700 1 -- 172.21.15.7:0/1411808351 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3400bcc0 con 0x7f3d3c154ff0 2023-03-03T18:01:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.327+0000 7f3d4338a700 1 -- 172.21.15.7:0/1411808351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c154ff0 msgr2=0x7f3d3c0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.327+0000 7f3d4338a700 1 --2- 172.21.15.7:0/1411808351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c154ff0 0x7f3d3c0cae00 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0ca910 tx=0x7f3d3400b330 comp rx=0 tx=0).stop 2023-03-03T18:01:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.327+0000 7f3d4338a700 1 -- 172.21.15.7:0/1411808351 shutdown_connections 2023-03-03T18:01:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.327+0000 7f3d4338a700 1 --2- 172.21.15.7:0/1411808351 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c154ff0 0x7f3d3c0cae00 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.327+0000 7f3d4338a700 1 --2- 172.21.15.7:0/1411808351 >> v2:172.21.15.143:3300/0 conn(0x7f3d3c0c00e0 0x7f3d3c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.327+0000 7f3d4338a700 1 -- 172.21.15.7:0/1411808351 >> 172.21.15.7:0/1411808351 conn(0x7f3d3c066e40 msgr2=0x7f3d3c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:01:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.327+0000 7f3d4338a700 1 -- 172.21.15.7:0/1411808351 shutdown_connections 2023-03-03T18:01:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.327+0000 7f3d4338a700 1 -- 172.21.15.7:0/1411808351 wait complete. 2023-03-03T18:01:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.328+0000 7f3d4338a700 1 Processor -- start 2023-03-03T18:01:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.328+0000 7f3d4338a700 1 -- start start 2023-03-03T18:01:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d4338a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 0x7f3d3c157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d4338a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0caa30 0x7f3d3c13ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d4338a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c154ff0 0x7f3d3c13e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d4338a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c156630 con 0x7f3d3c0caa30 2023-03-03T18:01:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d4338a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c13e870 con 0x7f3d3c0c00e0 2023-03-03T18:01:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d4338a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c1ea280 con 0x7f3d3c154ff0 2023-03-03T18:01:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d40925700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0caa30 0x7f3d3c13ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d40925700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0caa30 0x7f3d3c13ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35480/0 (socket says 172.21.15.7:35480) 2023-03-03T18:01:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d40925700 1 -- 172.21.15.7:0/3486085413 learned_addr learned my addr 172.21.15.7:0/3486085413 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:01:15.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d41126700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 0x7f3d3c157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:15.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d41927700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c154ff0 0x7f3d3c13e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:15.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d40925700 1 -- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c154ff0 msgr2=0x7f3d3c13e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:15.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d40925700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c154ff0 0x7f3d3c13e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:15.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d40925700 1 -- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 msgr2=0x7f3d3c157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:15.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d40925700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 0x7f3d3c157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:15.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d40925700 1 -- 172.21.15.7:0/3486085413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34009b30 con 0x7f3d3c0caa30 2023-03-03T18:01:15.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.329+0000 7f3d41927700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c154ff0 0x7f3d3c13e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:01:15.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.330+0000 7f3d40925700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0caa30 0x7f3d3c13ddf0 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000e990 tx=0x7f3d3000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:01:15.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.330+0000 7f3d2dffb700 1 -- 172.21.15.7:0/3486085413 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d300096f0 con 0x7f3d3c0caa30 2023-03-03T18:01:15.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.330+0000 7f3d2dffb700 1 -- 172.21.15.7:0/3486085413 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3000c7f0 con 0x7f3d3c0caa30 2023-03-03T18:01:15.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.330+0000 7f3d2dffb700 1 -- 172.21.15.7:0/3486085413 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3000eee0 con 0x7f3d3c0caa30 2023-03-03T18:01:15.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.330+0000 7f3d4338a700 1 -- 172.21.15.7:0/3486085413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c1547f0 con 0x7f3d3c0caa30 2023-03-03T18:01:15.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.330+0000 7f3d4338a700 1 -- 172.21.15.7:0/3486085413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3c154510 con 0x7f3d3c0caa30 2023-03-03T18:01:15.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.331+0000 7f3d4338a700 1 -- 172.21.15.7:0/3486085413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3c154ad0 con 0x7f3d3c0caa30 2023-03-03T18:01:15.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.331+0000 7f3d2dffb700 1 -- 172.21.15.7:0/3486085413 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d30023070 con 0x7f3d3c0caa30 2023-03-03T18:01:15.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.332+0000 7f3d2dffb700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d200682f0 0x7f3d2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:15.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.332+0000 7f3d2dffb700 1 -- 172.21.15.7:0/3486085413 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3d300863a0 con 0x7f3d3c0caa30 2023-03-03T18:01:15.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.332+0000 7f3d41126700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d200682f0 0x7f3d2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:15.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.333+0000 7f3d41126700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d200682f0 0x7f3d2006a7a0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f3d380018a0 tx=0x7f3d38009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:01:15.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.336+0000 7f3d2dffb700 1 -- 172.21.15.7:0/3486085413 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d300526e0 con 0x7f3d3c0caa30 2023-03-03T18:01:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.577+0000 7f3d4338a700 1 -- 172.21.15.7:0/3486085413 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d3c074bb0 con 0x7f3d3c0caa30 2023-03-03T18:01:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.578+0000 7f3d2dffb700 1 -- 172.21.15.7:0/3486085413 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3d30055d00 con 0x7f3d3c0caa30 2023-03-03T18:01:15.578 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:01:15.578 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:01:15.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 -- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d200682f0 msgr2=0x7f3d2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:15.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d200682f0 0x7f3d2006a7a0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f3d380018a0 tx=0x7f3d38009040 comp rx=0 tx=0).stop 2023-03-03T18:01:15.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 -- 172.21.15.7:0/3486085413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0caa30 msgr2=0x7f3d3c13ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:15.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0caa30 0x7f3d3c13ddf0 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000e990 tx=0x7f3d3000eca0 comp rx=0 tx=0).stop 2023-03-03T18:01:15.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 -- 172.21.15.7:0/3486085413 shutdown_connections 2023-03-03T18:01:15.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d3c154ff0 0x7f3d3c13e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:15.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d3c0c00e0 0x7f3d3c157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:15.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d200682f0 0x7f3d2006a7a0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:15.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 --2- 172.21.15.7:0/3486085413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d3c0caa30 0x7f3d3c13ddf0 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:15.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 -- 172.21.15.7:0/3486085413 >> 172.21.15.7:0/3486085413 conn(0x7f3d3c066e40 msgr2=0x7f3d3c0c99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:01:15.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 -- 172.21.15.7:0/3486085413 shutdown_connections 2023-03-03T18:01:15.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:15.580+0000 7f3d4338a700 1 -- 172.21.15.7:0/3486085413 wait complete. 2023-03-03T18:01:15.581 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:01:15.591 INFO:tasks.mds_thrash.fs.[a]:waiting for 61 secs before thrashing 2023-03-03T18:01:33.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:33.510 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:01:57.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:01:57.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.310+0000 7f607ec26700 1 Processor -- start 2023-03-03T18:01:57.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.310+0000 7f607ec26700 1 -- start start 2023-03-03T18:01:57.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.311+0000 7f607ec26700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f60780caa30 0x7f60780c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:57.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.311+0000 7f607ec26700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f60780c00e0 0x7f6078154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:57.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.311+0000 7f607ec26700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60780c7310 con 0x7f60780cae00 2023-03-03T18:01:57.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.311+0000 7f607ec26700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60780cb240 con 0x7f60780caa30 2023-03-03T18:01:57.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.311+0000 7f607ec26700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60780bfd10 con 0x7f60780c00e0 2023-03-03T18:01:57.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.311+0000 7f607c9c2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f60780caa30 0x7f60780c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:57.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.311+0000 7f607d1c3700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f60780cae00 0x7f60780c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57734/0 (socket says 172.21.15.7:57734) 2023-03-03T18:01:57.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.311+0000 7f607d1c3700 1 -- 172.21.15.7:0/3448891618 learned_addr learned my addr 172.21.15.7:0/3448891618 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:01:57.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.312+0000 7f6077fff700 1 --2- 172.21.15.7:0/3448891618 >> v2:172.21.15.143:3300/0 conn(0x7f60780c00e0 0x7f6078154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:57.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.312+0000 7f607c9c2700 1 -- 172.21.15.7:0/3448891618 >> v2:172.21.15.143:3300/0 conn(0x7f60780c00e0 msgr2=0x7f6078154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:01:57.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.312+0000 7f607c9c2700 1 --2- 172.21.15.7:0/3448891618 >> v2:172.21.15.143:3300/0 conn(0x7f60780c00e0 0x7f6078154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:57.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.312+0000 7f607c9c2700 1 -- 172.21.15.7:0/3448891618 >> v1:172.21.15.7:6789/0 conn(0x7f60780cae00 legacy=0x7f60780c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:57.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.312+0000 7f607c9c2700 1 -- 172.21.15.7:0/3448891618 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60780bfa30 con 0x7f60780caa30 2023-03-03T18:01:57.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.312+0000 7f60777fe700 1 -- 172.21.15.7:0/3448891618 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371620227 0 0) 0x7f60780c7310 con 0x7f60780cae00 2023-03-03T18:01:57.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.312+0000 7f607c9c2700 1 --2- 172.21.15.7:0/3448891618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 0x7f60780c5c60 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7f60780ca910 tx=0x7f606c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c8bcb22497521a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:01:57.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.312+0000 7f60777fe700 1 -- 172.21.15.7:0/3448891618 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c011830 con 0x7f60780caa30 2023-03-03T18:01:57.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.312+0000 7f60777fe700 1 -- 172.21.15.7:0/3448891618 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606c011990 con 0x7f60780caa30 2023-03-03T18:01:57.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.312+0000 7f60777fe700 1 -- 172.21.15.7:0/3448891618 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c011d20 con 0x7f60780caa30 2023-03-03T18:01:57.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.313+0000 7f607ec26700 1 -- 172.21.15.7:0/3448891618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 msgr2=0x7f60780c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:57.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.313+0000 7f607ec26700 1 --2- 172.21.15.7:0/3448891618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 0x7f60780c5c60 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7f60780ca910 tx=0x7f606c009b20 comp rx=0 tx=0).stop 2023-03-03T18:01:57.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.313+0000 7f607ec26700 1 -- 172.21.15.7:0/3448891618 shutdown_connections 2023-03-03T18:01:57.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.313+0000 7f607ec26700 1 --2- 172.21.15.7:0/3448891618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 0x7f60780c5c60 unknown :-1 s=CLOSED pgs=3082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:57.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.313+0000 7f607ec26700 1 --2- 172.21.15.7:0/3448891618 >> v2:172.21.15.143:3300/0 conn(0x7f60780c00e0 0x7f6078154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:57.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.313+0000 7f607ec26700 1 -- 172.21.15.7:0/3448891618 >> 172.21.15.7:0/3448891618 conn(0x7f6078066e40 msgr2=0x7f6078071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:01:57.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.313+0000 7f607ec26700 1 -- 172.21.15.7:0/3448891618 shutdown_connections 2023-03-03T18:01:57.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.313+0000 7f607ec26700 1 -- 172.21.15.7:0/3448891618 wait complete. 2023-03-03T18:01:57.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.314+0000 7f607ec26700 1 Processor -- start 2023-03-03T18:01:57.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.314+0000 7f607ec26700 1 -- start start 2023-03-03T18:01:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.315+0000 7f607ec26700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60780c00e0 0x7f6078156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.315+0000 7f607ec26700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 0x7f60781562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.315+0000 7f607ec26700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60780cae00 0x7f60780c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.315+0000 7f607ec26700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60781e9bb0 con 0x7f60780caa30 2023-03-03T18:01:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.315+0000 7f607ec26700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60781567e0 con 0x7f60780cae00 2023-03-03T18:01:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.315+0000 7f607ec26700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60781ea9d0 con 0x7f60780c00e0 2023-03-03T18:01:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.315+0000 7f6077fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 0x7f60781562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:57.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.315+0000 7f6077fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 0x7f60781562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35524/0 (socket says 172.21.15.7:35524) 2023-03-03T18:01:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.315+0000 7f6077fff700 1 -- 172.21.15.7:0/359436357 learned_addr learned my addr 172.21.15.7:0/359436357 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:01:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f607d1c3700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60780cae00 0x7f60780c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f607c9c2700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60780c00e0 0x7f6078156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f6077fff700 1 -- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60780c00e0 msgr2=0x7f6078156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f6077fff700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60780c00e0 0x7f6078156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f6077fff700 1 -- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60780cae00 msgr2=0x7f60780c6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:57.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f6077fff700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60780cae00 0x7f60780c6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:57.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f6077fff700 1 -- 172.21.15.7:0/359436357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c0094a0 con 0x7f60780caa30 2023-03-03T18:01:57.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f607d1c3700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60780cae00 0x7f60780c6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:01:57.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f6077fff700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 0x7f60781562a0 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f606400e760 tx=0x7f606400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:01:57.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f60757fa700 1 -- 172.21.15.7:0/359436357 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064013610 con 0x7f60780caa30 2023-03-03T18:01:57.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f60757fa700 1 -- 172.21.15.7:0/359436357 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6064013770 con 0x7f60780caa30 2023-03-03T18:01:57.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f607ec26700 1 -- 172.21.15.7:0/359436357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60781547f0 con 0x7f60780caa30 2023-03-03T18:01:57.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f60757fa700 1 -- 172.21.15.7:0/359436357 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064013b20 con 0x7f60780caa30 2023-03-03T18:01:57.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.316+0000 7f607ec26700 1 -- 172.21.15.7:0/359436357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6078154510 con 0x7f60780caa30 2023-03-03T18:01:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.317+0000 7f607ec26700 1 -- 172.21.15.7:0/359436357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6078154ad0 con 0x7f60780caa30 2023-03-03T18:01:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.318+0000 7f60757fa700 1 -- 172.21.15.7:0/359436357 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6064019030 con 0x7f60780caa30 2023-03-03T18:01:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.318+0000 7f60757fa700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f605c0682f0 0x7f605c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:01:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.318+0000 7f60757fa700 1 -- 172.21.15.7:0/359436357 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6064085af0 con 0x7f60780caa30 2023-03-03T18:01:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.318+0000 7f607c9c2700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f605c0682f0 0x7f605c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:01:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.319+0000 7f607c9c2700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f605c0682f0 0x7f605c06a7a0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f606c000c00 tx=0x7f606c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:01:57.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.323+0000 7f60757fa700 1 -- 172.21.15.7:0/359436357 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6064051e90 con 0x7f60780caa30 2023-03-03T18:01:57.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.557+0000 7f607ec26700 1 -- 172.21.15.7:0/359436357 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6078074bb0 con 0x7f60780caa30 2023-03-03T18:01:57.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.558+0000 7f60757fa700 1 -- 172.21.15.7:0/359436357 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f60640554b0 con 0x7f60780caa30 2023-03-03T18:01:57.559 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:01:57.559 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:01:57.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 -- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f605c0682f0 msgr2=0x7f605c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:57.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f605c0682f0 0x7f605c06a7a0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f606c000c00 tx=0x7f606c008030 comp rx=0 tx=0).stop 2023-03-03T18:01:57.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 -- 172.21.15.7:0/359436357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 msgr2=0x7f60781562a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:01:57.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 0x7f60781562a0 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f606400e760 tx=0x7f606400eb20 comp rx=0 tx=0).stop 2023-03-03T18:01:57.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 -- 172.21.15.7:0/359436357 shutdown_connections 2023-03-03T18:01:57.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f60780cae00 0x7f60780c6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:57.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f60780c00e0 0x7f6078156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:57.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f605c0682f0 0x7f605c06a7a0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:57.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 --2- 172.21.15.7:0/359436357 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f60780caa30 0x7f60781562a0 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:01:57.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 -- 172.21.15.7:0/359436357 >> 172.21.15.7:0/359436357 conn(0x7f6078066e40 msgr2=0x7f6078065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:01:57.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 -- 172.21.15.7:0/359436357 shutdown_connections 2023-03-03T18:01:57.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:01:57.560+0000 7f607ec26700 1 -- 172.21.15.7:0/359436357 wait complete. 2023-03-03T18:01:57.563 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:01:57.573 INFO:tasks.mds_thrash.fs.[b]:waiting for 46 secs before thrashing 2023-03-03T18:02:03.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:03.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:16.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:02:16.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.730+0000 7f3cbd51a700 1 Processor -- start 2023-03-03T18:02:16.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.730+0000 7f3cbd51a700 1 -- start start 2023-03-03T18:02:16.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cbd51a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3cb80c4630 0x7f3cb806e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:16.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cbd51a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb806e730 con 0x7f3cb8151b00 2023-03-03T18:02:16.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cbd51a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb81583e0 con 0x7f3cb8154420 2023-03-03T18:02:16.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cbd51a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cb8157ab0 con 0x7f3cb80c4630 2023-03-03T18:02:16.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb6ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3cb8151b00 0x7f3cb80a94c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57754/0 (socket says 172.21.15.7:57754) 2023-03-03T18:02:16.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb6ffd700 1 -- 172.21.15.7:0/1415778410 learned_addr learned my addr 172.21.15.7:0/1415778410 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:02:16.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476194748 0 0) 0x7f3cb806e730 con 0x7f3cb8151b00 2023-03-03T18:02:16.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb67fc700 1 --2- 172.21.15.7:0/1415778410 >> v2:172.21.15.143:3300/0 conn(0x7f3cb80c4630 0x7f3cb806e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:16.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca00035b0 con 0x7f3cb8151b00 2023-03-03T18:02:16.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776926338 0 0) 0x7f3ca00035b0 con 0x7f3cb8151b00 2023-03-03T18:02:16.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 >> v2:172.21.15.143:3300/0 conn(0x7f3cb80c4630 msgr2=0x7f3cb806e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:02:16.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 --2- 172.21.15.7:0/1415778410 >> v2:172.21.15.143:3300/0 conn(0x7f3cb80c4630 0x7f3cb806e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:16.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 >> v1:172.21.15.143:6789/0 conn(0x7f3cb8154420 legacy=0x7f3cb81548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:16.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8157d90 con 0x7f3cb8151b00 2023-03-03T18:02:16.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f3ca8001670 con 0x7f3cb8151b00 2023-03-03T18:02:16.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 >> v1:172.21.15.7:6789/0 conn(0x7f3cb8151b00 legacy=0x7f3cb80a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:16.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ca0003a70 0x7f3ca0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:16.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3ca0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:16.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.731+0000 7f3cb5ffb700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ca0005280 0x7f3ca000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:16.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8157ab0 con 0x7f3ca0004850 2023-03-03T18:02:16.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca000c1b0 con 0x7f3ca0003a70 2023-03-03T18:02:16.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca000c370 con 0x7f3ca0005280 2023-03-03T18:02:16.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb67fc700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3ca0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:16.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb6ffd700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ca0003a70 0x7f3ca0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:16.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb77fe700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ca0005280 0x7f3ca000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:16.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb67fc700 1 -- 172.21.15.7:0/1415778410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ca0005280 msgr2=0x7f3ca000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:02:16.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb67fc700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ca0005280 0x7f3ca000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:16.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb67fc700 1 -- 172.21.15.7:0/1415778410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ca0003a70 msgr2=0x7f3ca0003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:02:16.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb67fc700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ca0003a70 0x7f3ca0003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:16.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb67fc700 1 -- 172.21.15.7:0/1415778410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca00035b0 con 0x7f3ca0004850 2023-03-03T18:02:16.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb77fe700 1 -- 172.21.15.7:0/1415778410 reap_dead start 2023-03-03T18:02:16.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb67fc700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3ca0004cc0 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009b60 tx=0x7f3ca0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d2b7ecd90adc4d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:02:16.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac00e6c0 con 0x7f3ca0004850 2023-03-03T18:02:16.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cac00e820 con 0x7f3ca0004850 2023-03-03T18:02:16.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.732+0000 7f3cb5ffb700 1 -- 172.21.15.7:0/1415778410 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cac00eb90 con 0x7f3ca0004850 2023-03-03T18:02:16.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.733+0000 7f3cbd51a700 1 -- 172.21.15.7:0/1415778410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 msgr2=0x7f3ca0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:16.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.733+0000 7f3cbd51a700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3ca0004cc0 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009b60 tx=0x7f3ca0005880 comp rx=0 tx=0).stop 2023-03-03T18:02:16.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.733+0000 7f3cbd51a700 1 -- 172.21.15.7:0/1415778410 shutdown_connections 2023-03-03T18:02:16.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.733+0000 7f3cbd51a700 1 --2- 172.21.15.7:0/1415778410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3ca0004cc0 unknown :-1 s=CLOSED pgs=3085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:16.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.733+0000 7f3cbd51a700 1 -- 172.21.15.7:0/1415778410 >> 172.21.15.7:0/1415778410 conn(0x7f3cb80c84d0 msgr2=0x7f3cb80c88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:02:16.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.733+0000 7f3cbd51a700 1 -- 172.21.15.7:0/1415778410 shutdown_connections 2023-03-03T18:02:16.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.733+0000 7f3cbd51a700 1 -- 172.21.15.7:0/1415778410 wait complete. 2023-03-03T18:02:16.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.734+0000 7f3cbd51a700 1 Processor -- start 2023-03-03T18:02:16.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.734+0000 7f3cbd51a700 1 -- start start 2023-03-03T18:02:16.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cbd51a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3cb8152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:16.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cbd51a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb814e280 0x7f3cb806eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:16.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cbd51a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb8156fa0 0x7f3cb81ed9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:16.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cbd51a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8156d50 con 0x7f3ca0004850 2023-03-03T18:02:16.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cbd51a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb806be30 con 0x7f3cb814e280 2023-03-03T18:02:16.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cbd51a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb81ed1d0 con 0x7f3cb8156fa0 2023-03-03T18:02:16.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3cb8152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:16.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3cb8152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35556/0 (socket says 172.21.15.7:35556) 2023-03-03T18:02:16.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb6ffd700 1 -- 172.21.15.7:0/2814797058 learned_addr learned my addr 172.21.15.7:0/2814797058 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:02:16.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb67fc700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb814e280 0x7f3cb806eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:16.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb77fe700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb8156fa0 0x7f3cb81ed9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:16.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb6ffd700 1 -- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb8156fa0 msgr2=0x7f3cb81ed9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:16.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb6ffd700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb8156fa0 0x7f3cb81ed9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:16.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb6ffd700 1 -- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb814e280 msgr2=0x7f3cb806eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:16.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb6ffd700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb814e280 0x7f3cb806eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:16.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb6ffd700 1 -- 172.21.15.7:0/2814797058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8158870 con 0x7f3ca0004850 2023-03-03T18:02:16.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb77fe700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb8156fa0 0x7f3cb81ed9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:02:16.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.735+0000 7f3cb67fc700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb814e280 0x7f3cb806eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:02:16.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.736+0000 7f3cb6ffd700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3cb8152290 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8007290 tx=0x7f3ca8007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:02:16.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.736+0000 7f3c9f7fe700 1 -- 172.21.15.7:0/2814797058 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8001490 con 0x7f3ca0004850 2023-03-03T18:02:16.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.736+0000 7f3c9f7fe700 1 -- 172.21.15.7:0/2814797058 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca800f770 con 0x7f3ca0004850 2023-03-03T18:02:16.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.736+0000 7f3c9f7fe700 1 -- 172.21.15.7:0/2814797058 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca800fb20 con 0x7f3ca0004850 2023-03-03T18:02:16.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.736+0000 7f3cbd51a700 1 -- 172.21.15.7:0/2814797058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb8158070 con 0x7f3ca0004850 2023-03-03T18:02:16.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.736+0000 7f3cbd51a700 1 -- 172.21.15.7:0/2814797058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb80c3c30 con 0x7f3ca0004850 2023-03-03T18:02:16.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.737+0000 7f3cbd51a700 1 -- 172.21.15.7:0/2814797058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb81ed540 con 0x7f3ca0004850 2023-03-03T18:02:16.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.737+0000 7f3c9f7fe700 1 -- 172.21.15.7:0/2814797058 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3ca800fc80 con 0x7f3ca0004850 2023-03-03T18:02:16.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.738+0000 7f3c9f7fe700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ca40682f0 0x7f3ca406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:16.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.738+0000 7f3c9f7fe700 1 -- 172.21.15.7:0/2814797058 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3ca80862e0 con 0x7f3ca0004850 2023-03-03T18:02:16.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.738+0000 7f3cb67fc700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ca40682f0 0x7f3ca406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:16.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.738+0000 7f3cb67fc700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ca40682f0 0x7f3ca406a7a0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0090b0 tx=0x7f3cac01e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:02:16.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.742+0000 7f3c9f7fe700 1 -- 172.21.15.7:0/2814797058 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3ca8052620 con 0x7f3ca0004850 2023-03-03T18:02:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.980+0000 7f3cbd51a700 1 -- 172.21.15.7:0/2814797058 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cb8074bb0 con 0x7f3ca0004850 2023-03-03T18:02:16.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.980+0000 7f3c9f7fe700 1 -- 172.21.15.7:0/2814797058 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3ca8055c40 con 0x7f3ca0004850 2023-03-03T18:02:16.980 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:02:16.980 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:02:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 -- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ca40682f0 msgr2=0x7f3ca406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ca40682f0 0x7f3ca406a7a0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0090b0 tx=0x7f3cac01e040 comp rx=0 tx=0).stop 2023-03-03T18:02:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 -- 172.21.15.7:0/2814797058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 msgr2=0x7f3cb8152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3cb8152290 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8007290 tx=0x7f3ca8007650 comp rx=0 tx=0).stop 2023-03-03T18:02:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 -- 172.21.15.7:0/2814797058 shutdown_connections 2023-03-03T18:02:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cb8156fa0 0x7f3cb81ed9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ca40682f0 0x7f3ca406a7a0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca0004850 0x7f3cb8152290 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 --2- 172.21.15.7:0/2814797058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cb814e280 0x7f3cb806eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 -- 172.21.15.7:0/2814797058 >> 172.21.15.7:0/2814797058 conn(0x7f3cb80c84d0 msgr2=0x7f3cb8071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:02:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 -- 172.21.15.7:0/2814797058 shutdown_connections 2023-03-03T18:02:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:16.982+0000 7f3cbd51a700 1 -- 172.21.15.7:0/2814797058 wait complete. 2023-03-03T18:02:16.984 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:02:16.994 INFO:tasks.mds_thrash.fs.[a]:waiting for 22 secs before thrashing 2023-03-03T18:02:33.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:33.652 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:02:38.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:02:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.134+0000 7f02d1214700 1 Processor -- start 2023-03-03T18:02:39.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.134+0000 7f02d1214700 1 -- start start 2023-03-03T18:02:39.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.135+0000 7f02d1214700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc1530d0 0x7f02cc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:39.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.135+0000 7f02d1214700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02cc0c2f30 con 0x7f02cc14f9d0 2023-03-03T18:02:39.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.135+0000 7f02d1214700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02cc0c31d0 con 0x7f02cc14fda0 2023-03-03T18:02:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.135+0000 7f02d1214700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc1524a0 con 0x7f02cc1530d0 2023-03-03T18:02:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.135+0000 7f02cad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f02cc14f9d0 0x7f02cc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57774/0 (socket says 172.21.15.7:57774) 2023-03-03T18:02:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.135+0000 7f02cad9d700 1 -- 172.21.15.7:0/3219826088 learned_addr learned my addr 172.21.15.7:0/3219826088 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:02:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.135+0000 7f02cb59e700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc1530d0 0x7f02cc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.135+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552104646 0 0) 0x7f02cc0c2f30 con 0x7f02cc14f9d0 2023-03-03T18:02:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b40035b0 con 0x7f02cc14f9d0 2023-03-03T18:02:39.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562368761 0 0) 0x7f02cc0c31d0 con 0x7f02cc14fda0 2023-03-03T18:02:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02cc0c2f30 con 0x7f02cc14fda0 2023-03-03T18:02:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157471760 0 0) 0x7f02b40035b0 con 0x7f02cc14f9d0 2023-03-03T18:02:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc1530d0 msgr2=0x7f02cc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:02:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc1530d0 0x7f02cc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 >> v1:172.21.15.143:6789/0 conn(0x7f02cc14fda0 legacy=0x7f02cc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc0c34b0 con 0x7f02cc14f9d0 2023-03-03T18:02:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f02c0001670 con 0x7f02cc14f9d0 2023-03-03T18:02:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 >> v1:172.21.15.7:6789/0 conn(0x7f02cc14f9d0 legacy=0x7f02cc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc1530d0 0x7f02b4003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02b40041e0 0x7f02b4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02b4004de0 0x7f02b4005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc1524a0 con 0x7f02b40041e0 2023-03-03T18:02:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b400bcc0 con 0x7f02b4004de0 2023-03-03T18:02:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b400be80 con 0x7f02cc1530d0 2023-03-03T18:02:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02ca59c700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02b40041e0 0x7f02b4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02cad9d700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc1530d0 0x7f02b4003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02cb59e700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02b4004de0 0x7f02b4005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02ca59c700 1 -- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc1530d0 msgr2=0x7f02b4003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:02:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02ca59c700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc1530d0 0x7f02b4003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:39.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02ca59c700 1 -- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02b4004de0 msgr2=0x7f02b4005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:02:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02ca59c700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02b4004de0 0x7f02b4005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.136+0000 7f02ca59c700 1 -- 172.21.15.7:0/3219826088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b40035b0 con 0x7f02b40041e0 2023-03-03T18:02:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02ca59c700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02b40041e0 0x7f02b4004850 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f02cc1501d0 tx=0x7f02b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768b347d001ae4f9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:02:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc001d80 con 0x7f02b40041e0 2023-03-03T18:02:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02bc0052d0 con 0x7f02b40041e0 2023-03-03T18:02:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02c9d9b700 1 -- 172.21.15.7:0/3219826088 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc001470 con 0x7f02b40041e0 2023-03-03T18:02:39.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02d1214700 1 -- 172.21.15.7:0/3219826088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02b40041e0 msgr2=0x7f02b4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02d1214700 1 --2- 172.21.15.7:0/3219826088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02b40041e0 0x7f02b4004850 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7f02cc1501d0 tx=0x7f02b4005880 comp rx=0 tx=0).stop 2023-03-03T18:02:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02cb59e700 1 -- 172.21.15.7:0/3219826088 reap_dead start 2023-03-03T18:02:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02d1214700 1 -- 172.21.15.7:0/3219826088 shutdown_connections 2023-03-03T18:02:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02d1214700 1 -- 172.21.15.7:0/3219826088 >> 172.21.15.7:0/3219826088 conn(0x7f02cc0c8900 msgr2=0x7f02cc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:02:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02d1214700 1 -- 172.21.15.7:0/3219826088 shutdown_connections 2023-03-03T18:02:39.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.137+0000 7f02d1214700 1 -- 172.21.15.7:0/3219826088 wait complete. 2023-03-03T18:02:39.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.139+0000 7f02d1214700 1 Processor -- start 2023-03-03T18:02:39.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.139+0000 7f02d1214700 1 -- start start 2023-03-03T18:02:39.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02d1214700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02cc0c1a80 0x7f02cc1ead50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:39.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02d1214700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc0c1e50 0x7f02cc06f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02d1214700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02cc1ed730 0x7f02cc06d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02d1214700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc06f6b0 con 0x7f02cc0c1a80 2023-03-03T18:02:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02d1214700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc06f8e0 con 0x7f02cc1ed730 2023-03-03T18:02:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02d1214700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc06fbc0 con 0x7f02cc0c1e50 2023-03-03T18:02:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02cc0c1a80 0x7f02cc1ead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02cc0c1a80 0x7f02cc1ead50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35574/0 (socket says 172.21.15.7:35574) 2023-03-03T18:02:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cad9d700 1 -- 172.21.15.7:0/169995212 learned_addr learned my addr 172.21.15.7:0/169995212 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:02:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cb59e700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02cc1ed730 0x7f02cc06d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02ca59c700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc0c1e50 0x7f02cc06f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cad9d700 1 -- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc0c1e50 msgr2=0x7f02cc06f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cad9d700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc0c1e50 0x7f02cc06f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cad9d700 1 -- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02cc1ed730 msgr2=0x7f02cc06d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cad9d700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02cc1ed730 0x7f02cc06d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:39.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cad9d700 1 -- 172.21.15.7:0/169995212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc0ac350 con 0x7f02cc0c1a80 2023-03-03T18:02:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cb59e700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02cc1ed730 0x7f02cc06d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:02:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02cad9d700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02cc0c1a80 0x7f02cc1ead50 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f02c00011b0 tx=0x7f02c00073f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:02:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.140+0000 7f02b37fe700 1 -- 172.21.15.7:0/169995212 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0011580 con 0x7f02cc0c1a80 2023-03-03T18:02:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.141+0000 7f02b37fe700 1 -- 172.21.15.7:0/169995212 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c00116e0 con 0x7f02cc0c1a80 2023-03-03T18:02:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.141+0000 7f02b37fe700 1 -- 172.21.15.7:0/169995212 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0011a90 con 0x7f02cc0c1a80 2023-03-03T18:02:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.141+0000 7f02d1214700 1 -- 172.21.15.7:0/169995212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02cc0ab630 con 0x7f02cc0c1a80 2023-03-03T18:02:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.141+0000 7f02d1214700 1 -- 172.21.15.7:0/169995212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02cc071fa0 con 0x7f02cc0c1a80 2023-03-03T18:02:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.142+0000 7f02d1214700 1 -- 172.21.15.7:0/169995212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02cc074bb0 con 0x7f02cc0c1a80 2023-03-03T18:02:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.142+0000 7f02b37fe700 1 -- 172.21.15.7:0/169995212 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f02c0011bf0 con 0x7f02cc0c1a80 2023-03-03T18:02:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.142+0000 7f02b37fe700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02b80682f0 0x7f02b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.142+0000 7f02b37fe700 1 -- 172.21.15.7:0/169995212 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f02c0086350 con 0x7f02cc0c1a80 2023-03-03T18:02:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.143+0000 7f02ca59c700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02b80682f0 0x7f02b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.143+0000 7f02ca59c700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02b80682f0 0x7f02b806a7a0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f02cc063f00 tx=0x7f02bc00a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:02:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.147+0000 7f02b37fe700 1 -- 172.21.15.7:0/169995212 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f02c0052690 con 0x7f02cc0c1a80 2023-03-03T18:02:39.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.385+0000 7f02d1214700 1 -- 172.21.15.7:0/169995212 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02cc0c2c70 con 0x7f02cc0c1a80 2023-03-03T18:02:39.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.385+0000 7f02b37fe700 1 -- 172.21.15.7:0/169995212 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f02c0055cb0 con 0x7f02cc0c1a80 2023-03-03T18:02:39.385 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:02:39.385 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:02:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 -- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02b80682f0 msgr2=0x7f02b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02b80682f0 0x7f02b806a7a0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f02cc063f00 tx=0x7f02bc00a000 comp rx=0 tx=0).stop 2023-03-03T18:02:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 -- 172.21.15.7:0/169995212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02cc0c1a80 msgr2=0x7f02cc1ead50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02cc0c1a80 0x7f02cc1ead50 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7f02c00011b0 tx=0x7f02c00073f0 comp rx=0 tx=0).stop 2023-03-03T18:02:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 -- 172.21.15.7:0/169995212 shutdown_connections 2023-03-03T18:02:39.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02cc1ed730 0x7f02cc06d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02b80682f0 0x7f02b806a7a0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02cc0c1a80 0x7f02cc1ead50 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 --2- 172.21.15.7:0/169995212 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f02cc0c1e50 0x7f02cc06f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 -- 172.21.15.7:0/169995212 >> 172.21.15.7:0/169995212 conn(0x7f02cc0c8900 msgr2=0x7f02cc151af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:02:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 -- 172.21.15.7:0/169995212 shutdown_connections 2023-03-03T18:02:39.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:39.387+0000 7f02d1214700 1 -- 172.21.15.7:0/169995212 wait complete. 2023-03-03T18:02:39.389 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:02:39.398 INFO:tasks.mds_thrash.fs.[a]:waiting for 117 secs before thrashing 2023-03-03T18:02:43.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:02:43.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.713+0000 7f55c3d4a700 1 Processor -- start 2023-03-03T18:02:43.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.713+0000 7f55c3d4a700 1 -- start start 2023-03-03T18:02:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.714+0000 7f55c3d4a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55bc0c8790 0x7f55bc0a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.714+0000 7f55c3d4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc06efd0 0x7f55bc0c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.714+0000 7f55c3d4a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55bc0aa0d0 con 0x7f55bc0aa560 2023-03-03T18:02:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.714+0000 7f55c3d4a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55bc071ec0 con 0x7f55bc0c8790 2023-03-03T18:02:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.714+0000 7f55c3d4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc0a9aa0 con 0x7f55bc06efd0 2023-03-03T18:02:43.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.714+0000 7f55c22e7700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f55bc0aa560 0x7f55bc1557e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57800/0 (socket says 172.21.15.7:57800) 2023-03-03T18:02:43.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.714+0000 7f55c22e7700 1 -- 172.21.15.7:0/4080693820 learned_addr learned my addr 172.21.15.7:0/4080693820 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:02:43.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c1ae6700 1 --2- 172.21.15.7:0/4080693820 >> v2:172.21.15.143:3300/0 conn(0x7f55bc0c8790 0x7f55bc0a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:43.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c12e5700 1 --2- 172.21.15.7:0/4080693820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc06efd0 0x7f55bc0c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:43.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c0ae4700 1 -- 172.21.15.7:0/4080693820 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058173535 0 0) 0x7f55bc0aa0d0 con 0x7f55bc0aa560 2023-03-03T18:02:43.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c0ae4700 1 -- 172.21.15.7:0/4080693820 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a80035b0 con 0x7f55bc0aa560 2023-03-03T18:02:43.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c0ae4700 1 -- 172.21.15.7:0/4080693820 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362291972 0 0) 0x7f55a80035b0 con 0x7f55bc0aa560 2023-03-03T18:02:43.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c12e5700 1 -- 172.21.15.7:0/4080693820 >> v2:172.21.15.143:3300/0 conn(0x7f55bc0c8790 msgr2=0x7f55bc0a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:02:43.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c12e5700 1 --2- 172.21.15.7:0/4080693820 >> v2:172.21.15.143:3300/0 conn(0x7f55bc0c8790 0x7f55bc0a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c12e5700 1 -- 172.21.15.7:0/4080693820 >> v1:172.21.15.7:6789/0 conn(0x7f55bc0aa560 legacy=0x7f55bc1557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c12e5700 1 -- 172.21.15.7:0/4080693820 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc0c2710 con 0x7f55bc06efd0 2023-03-03T18:02:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c1ae6700 1 --2- 172.21.15.7:0/4080693820 >> v2:172.21.15.143:3300/0 conn(0x7f55bc0c8790 0x7f55bc0a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:02:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c12e5700 1 --2- 172.21.15.7:0/4080693820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc06efd0 0x7f55bc0c8b60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009840 tx=0x7f55b8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f605a77202558b72 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:02:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c0ae4700 1 -- 172.21.15.7:0/4080693820 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b40019f0 con 0x7f55bc06efd0 2023-03-03T18:02:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c0ae4700 1 -- 172.21.15.7:0/4080693820 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b8004030 con 0x7f55bc06efd0 2023-03-03T18:02:43.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.715+0000 7f55c0ae4700 1 -- 172.21.15.7:0/4080693820 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b80043c0 con 0x7f55bc06efd0 2023-03-03T18:02:43.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.716+0000 7f55c3d4a700 1 -- 172.21.15.7:0/4080693820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc06efd0 msgr2=0x7f55bc0c8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:43.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.716+0000 7f55c3d4a700 1 --2- 172.21.15.7:0/4080693820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc06efd0 0x7f55bc0c8b60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009840 tx=0x7f55b8009b50 comp rx=0 tx=0).stop 2023-03-03T18:02:43.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.716+0000 7f55c3d4a700 1 -- 172.21.15.7:0/4080693820 shutdown_connections 2023-03-03T18:02:43.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.716+0000 7f55c3d4a700 1 --2- 172.21.15.7:0/4080693820 >> v2:172.21.15.143:3300/0 conn(0x7f55bc0c8790 0x7f55bc0a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:43.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.716+0000 7f55c3d4a700 1 --2- 172.21.15.7:0/4080693820 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc06efd0 0x7f55bc0c8b60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:43.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.716+0000 7f55c3d4a700 1 -- 172.21.15.7:0/4080693820 >> 172.21.15.7:0/4080693820 conn(0x7f55bc0736f0 msgr2=0x7f55bc073af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:02:43.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.716+0000 7f55c3d4a700 1 -- 172.21.15.7:0/4080693820 shutdown_connections 2023-03-03T18:02:43.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.716+0000 7f55c3d4a700 1 -- 172.21.15.7:0/4080693820 wait complete. 2023-03-03T18:02:43.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.717+0000 7f55c3d4a700 1 Processor -- start 2023-03-03T18:02:43.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.717+0000 7f55c3d4a700 1 -- start start 2023-03-03T18:02:43.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c3d4a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55bc06efd0 0x7f55bc1e96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:43.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c3d4a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55bc0aa560 0x7f55bc148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:43.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c3d4a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc0c8790 0x7f55bc141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:43.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c3d4a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc146c00 con 0x7f55bc0aa560 2023-03-03T18:02:43.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c3d4a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc146ea0 con 0x7f55bc06efd0 2023-03-03T18:02:43.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c3d4a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55bc148870 con 0x7f55bc0c8790 2023-03-03T18:02:43.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c12e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55bc0aa560 0x7f55bc148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:43.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c12e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55bc0aa560 0x7f55bc148d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35590/0 (socket says 172.21.15.7:35590) 2023-03-03T18:02:43.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c12e5700 1 -- 172.21.15.7:0/3426392524 learned_addr learned my addr 172.21.15.7:0/3426392524 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:02:43.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c22e7700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc0c8790 0x7f55bc141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:43.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c1ae6700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55bc06efd0 0x7f55bc1e96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c12e5700 1 -- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc0c8790 msgr2=0x7f55bc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c12e5700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc0c8790 0x7f55bc141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c12e5700 1 -- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55bc06efd0 msgr2=0x7f55bc1e96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:43.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c12e5700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55bc06efd0 0x7f55bc1e96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c12e5700 1 -- 172.21.15.7:0/3426392524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac0094a0 con 0x7f55bc0aa560 2023-03-03T18:02:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c22e7700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc0c8790 0x7f55bc141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:02:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.718+0000 7f55c12e5700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55bc0aa560 0x7f55bc148d90 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f55b800a010 tx=0x7f55b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:02:43.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.719+0000 7f55b27fc700 1 -- 172.21.15.7:0/3426392524 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b8003b90 con 0x7f55bc0aa560 2023-03-03T18:02:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.719+0000 7f55b27fc700 1 -- 172.21.15.7:0/3426392524 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b801f070 con 0x7f55bc0aa560 2023-03-03T18:02:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.719+0000 7f55c3d4a700 1 -- 172.21.15.7:0/3426392524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b80094a0 con 0x7f55bc0aa560 2023-03-03T18:02:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.719+0000 7f55b27fc700 1 -- 172.21.15.7:0/3426392524 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b8011630 con 0x7f55bc0aa560 2023-03-03T18:02:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.719+0000 7f55c3d4a700 1 -- 172.21.15.7:0/3426392524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55bc0a9d60 con 0x7f55bc0aa560 2023-03-03T18:02:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.720+0000 7f55c3d4a700 1 -- 172.21.15.7:0/3426392524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55bc0c29f0 con 0x7f55bc0aa560 2023-03-03T18:02:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.720+0000 7f55b27fc700 1 -- 172.21.15.7:0/3426392524 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f55b802a020 con 0x7f55bc0aa560 2023-03-03T18:02:43.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.720+0000 7f55b27fc700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55a00682f0 0x7f55a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:02:43.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.721+0000 7f55b27fc700 1 -- 172.21.15.7:0/3426392524 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f55b8087560 con 0x7f55bc0aa560 2023-03-03T18:02:43.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.721+0000 7f55c1ae6700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55a00682f0 0x7f55a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:02:43.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.721+0000 7f55c1ae6700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55a00682f0 0x7f55a006a7a0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f55bc1e74f0 tx=0x7f55ac009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:02:43.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.725+0000 7f55b27fc700 1 -- 172.21.15.7:0/3426392524 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f55b8053880 con 0x7f55bc0aa560 2023-03-03T18:02:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.960+0000 7f55c3d4a700 1 -- 172.21.15.7:0/3426392524 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55bc14c8c0 con 0x7f55bc0aa560 2023-03-03T18:02:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.961+0000 7f55b27fc700 1 -- 172.21.15.7:0/3426392524 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f55b8056ea0 con 0x7f55bc0aa560 2023-03-03T18:02:43.961 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:02:43.961 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:02:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 -- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55a00682f0 msgr2=0x7f55a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:43.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55a00682f0 0x7f55a006a7a0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f55bc1e74f0 tx=0x7f55ac009210 comp rx=0 tx=0).stop 2023-03-03T18:02:43.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 -- 172.21.15.7:0/3426392524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55bc0aa560 msgr2=0x7f55bc148d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:02:43.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55bc0aa560 0x7f55bc148d90 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f55b800a010 tx=0x7f55b8004750 comp rx=0 tx=0).stop 2023-03-03T18:02:43.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 -- 172.21.15.7:0/3426392524 shutdown_connections 2023-03-03T18:02:43.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55bc0c8790 0x7f55bc141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:43.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55bc06efd0 0x7f55bc1e96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:43.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55a00682f0 0x7f55a006a7a0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 --2- 172.21.15.7:0/3426392524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55bc0aa560 0x7f55bc148d90 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:02:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 -- 172.21.15.7:0/3426392524 >> 172.21.15.7:0/3426392524 conn(0x7f55bc0736f0 msgr2=0x7f55bc1463d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:02:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 -- 172.21.15.7:0/3426392524 shutdown_connections 2023-03-03T18:02:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:02:43.963+0000 7f55c3d4a700 1 -- 172.21.15.7:0/3426392524 wait complete. 2023-03-03T18:02:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:02:43.974 INFO:tasks.mds_thrash.fs.[b]:waiting for 24 secs before thrashing 2023-03-03T18:03:03.712 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:03.716 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:07.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:03:08.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.114+0000 7f87b8d83700 1 Processor -- start 2023-03-03T18:03:08.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.114+0000 7f87b8d83700 1 -- start start 2023-03-03T18:03:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.115+0000 7f87b8d83700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f87b40ac220 0x7f87b40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:03:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.115+0000 7f87b8d83700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b40ac900 con 0x7f87b4073c30 2023-03-03T18:03:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.115+0000 7f87b8d83700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b40acb50 con 0x7f87b4152b60 2023-03-03T18:03:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.115+0000 7f87b8d83700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b4071e30 con 0x7f87b40ac220 2023-03-03T18:03:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.115+0000 7f87b1d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f87b4073c30 0x7f87b4152450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57830/0 (socket says 172.21.15.7:57830) 2023-03-03T18:03:08.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.115+0000 7f87b1d9b700 1 -- 172.21.15.7:0/3078903968 learned_addr learned my addr 172.21.15.7:0/3078903968 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:03:08.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.115+0000 7f87b259c700 1 --2- 172.21.15.7:0/3078903968 >> v2:172.21.15.143:3300/0 conn(0x7f87b40ac220 0x7f87b40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:03:08.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.115+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393325643 0 0) 0x7f87b40ac900 con 0x7f87b4073c30 2023-03-03T18:03:08.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.115+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f879c0035b0 con 0x7f87b4073c30 2023-03-03T18:03:08.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1475594809 0 0) 0x7f879c0035b0 con 0x7f87b4073c30 2023-03-03T18:03:08.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 >> v2:172.21.15.143:3300/0 conn(0x7f87b40ac220 msgr2=0x7f87b40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:03:08.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 --2- 172.21.15.7:0/3078903968 >> v2:172.21.15.143:3300/0 conn(0x7f87b40ac220 0x7f87b40736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 >> v1:172.21.15.143:6789/0 conn(0x7f87b4152b60 legacy=0x7f87b415ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:03:08.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b41559c0 con 0x7f87b4073c30 2023-03-03T18:03:08.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f87a4001670 con 0x7f87b4073c30 2023-03-03T18:03:08.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 >> v1:172.21.15.7:6789/0 conn(0x7f87b4073c30 legacy=0x7f87b4152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:03:08.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f879c003a70 0x7f879c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:03:08.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f879c004850 0x7f879c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:03:08.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f879c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:03:08.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b4071e30 con 0x7f879c005250 2023-03-03T18:03:08.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c00c190 con 0x7f879c004850 2023-03-03T18:03:08.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879c00c350 con 0x7f879c003a70 2023-03-03T18:03:08.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b2d9d700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f879c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:03:08.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b259c700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f879c003a70 0x7f879c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:03:08.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b1d9b700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f879c004850 0x7f879c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:03:08.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b2d9d700 1 -- 172.21.15.7:0/3078903968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f879c003a70 msgr2=0x7f879c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:03:08.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b2d9d700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f879c003a70 0x7f879c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b2d9d700 1 -- 172.21.15.7:0/3078903968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f879c004850 msgr2=0x7f879c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:03:08.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b2d9d700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f879c004850 0x7f879c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b2d9d700 1 -- 172.21.15.7:0/3078903968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c0035b0 con 0x7f879c005250 2023-03-03T18:03:08.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b1d9b700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f879c004850 0x7f879c004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:03:08.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b2d9d700 1 -- 172.21.15.7:0/3078903968 reap_dead start 2023-03-03T18:03:08.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.116+0000 7f87b2d9d700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f879c00ba10 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7f87ac00c160 tx=0x7f879c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6d3d67c804a39 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:03:08.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.117+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac001db0 con 0x7f879c005250 2023-03-03T18:03:08.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.117+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ac00cca0 con 0x7f879c005250 2023-03-03T18:03:08.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.117+0000 7f87b159a700 1 -- 172.21.15.7:0/3078903968 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ac003c00 con 0x7f879c005250 2023-03-03T18:03:08.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.117+0000 7f87b8d83700 1 -- 172.21.15.7:0/3078903968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 msgr2=0x7f879c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:03:08.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.117+0000 7f87b8d83700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f879c00ba10 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7f87ac00c160 tx=0x7f879c005880 comp rx=0 tx=0).stop 2023-03-03T18:03:08.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.117+0000 7f87b8d83700 1 -- 172.21.15.7:0/3078903968 shutdown_connections 2023-03-03T18:03:08.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.117+0000 7f87b8d83700 1 --2- 172.21.15.7:0/3078903968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f879c00ba10 unknown :-1 s=CLOSED pgs=3093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.117+0000 7f87b8d83700 1 -- 172.21.15.7:0/3078903968 >> 172.21.15.7:0/3078903968 conn(0x7f87b40a94d0 msgr2=0x7f87b40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:03:08.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.117+0000 7f87b8d83700 1 -- 172.21.15.7:0/3078903968 shutdown_connections 2023-03-03T18:03:08.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.117+0000 7f87b8d83700 1 -- 172.21.15.7:0/3078903968 wait complete. 2023-03-03T18:03:08.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.119+0000 7f87b8d83700 1 Processor -- start 2023-03-03T18:03:08.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.119+0000 7f87b8d83700 1 -- start start 2023-03-03T18:03:08.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.119+0000 7f87b8d83700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f87b4154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:03:08.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.119+0000 7f87b8d83700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b40be020 0x7f87b40bd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:03:08.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.119+0000 7f87b8d83700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b40bd9c0 0x7f87b40bae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:03:08.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.119+0000 7f87b8d83700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b40c0580 con 0x7f879c005250 2023-03-03T18:03:08.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b8d83700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b414ebb0 con 0x7f87b40bd9c0 2023-03-03T18:03:08.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b8d83700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b40c0a10 con 0x7f87b40be020 2023-03-03T18:03:08.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f87b4154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:03:08.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f87b4154160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35630/0 (socket says 172.21.15.7:35630) 2023-03-03T18:03:08.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b259c700 1 -- 172.21.15.7:0/2560129193 learned_addr learned my addr 172.21.15.7:0/2560129193 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:03:08.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b2d9d700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b40bd9c0 0x7f87b40bae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:03:08.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b1d9b700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b40be020 0x7f87b40bd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:03:08.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b259c700 1 -- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b40be020 msgr2=0x7f87b40bd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:03:08.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b259c700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b40be020 0x7f87b40bd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b259c700 1 -- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b40bd9c0 msgr2=0x7f87b40bae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:03:08.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b259c700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b40bd9c0 0x7f87b40bae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b259c700 1 -- 172.21.15.7:0/2560129193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b4155f80 con 0x7f879c005250 2023-03-03T18:03:08.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b1d9b700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b40be020 0x7f87b40bd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:03:08.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b259c700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f87b4154160 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f87a80119a0 tx=0x7f87a8011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:03:08.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87a2ffd700 1 -- 172.21.15.7:0/2560129193 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800f940 con 0x7f879c005250 2023-03-03T18:03:08.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87a2ffd700 1 -- 172.21.15.7:0/2560129193 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a800faa0 con 0x7f879c005250 2023-03-03T18:03:08.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b8d83700 1 -- 172.21.15.7:0/2560129193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b4155ca0 con 0x7f879c005250 2023-03-03T18:03:08.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87a2ffd700 1 -- 172.21.15.7:0/2560129193 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800fe30 con 0x7f879c005250 2023-03-03T18:03:08.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.120+0000 7f87b8d83700 1 -- 172.21.15.7:0/2560129193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b4156260 con 0x7f879c005250 2023-03-03T18:03:08.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.121+0000 7f87b8d83700 1 -- 172.21.15.7:0/2560129193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b4074bb0 con 0x7f879c005250 2023-03-03T18:03:08.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.122+0000 7f87a2ffd700 1 -- 172.21.15.7:0/2560129193 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f87a8014440 con 0x7f879c005250 2023-03-03T18:03:08.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.122+0000 7f87a2ffd700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87940682f0 0x7f879406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:03:08.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.122+0000 7f87a2ffd700 1 -- 172.21.15.7:0/2560129193 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f87a8086910 con 0x7f879c005250 2023-03-03T18:03:08.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.122+0000 7f87b1d9b700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87940682f0 0x7f879406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:03:08.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.123+0000 7f87b1d9b700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87940682f0 0x7f879406a7a0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f87b414f620 tx=0x7f87a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:03:08.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.126+0000 7f87a2ffd700 1 -- 172.21.15.7:0/2560129193 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f87a8052c50 con 0x7f879c005250 2023-03-03T18:03:08.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.365+0000 7f87b8d83700 1 -- 172.21.15.7:0/2560129193 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87b40a9fb0 con 0x7f879c005250 2023-03-03T18:03:08.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.365+0000 7f87a2ffd700 1 -- 172.21.15.7:0/2560129193 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f87a8056270 con 0x7f879c005250 2023-03-03T18:03:08.366 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:03:08.367 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:03:08.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.367+0000 7f87b8d83700 1 -- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87940682f0 msgr2=0x7f879406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:03:08.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.367+0000 7f87b8d83700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87940682f0 0x7f879406a7a0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f87b414f620 tx=0x7f87a4009040 comp rx=0 tx=0).stop 2023-03-03T18:03:08.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.367+0000 7f87b8d83700 1 -- 172.21.15.7:0/2560129193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 msgr2=0x7f87b4154160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:03:08.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.367+0000 7f87b8d83700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f87b4154160 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f87a80119a0 tx=0x7f87a8011cb0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.367+0000 7f87b8d83700 1 -- 172.21.15.7:0/2560129193 shutdown_connections 2023-03-03T18:03:08.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.367+0000 7f87b8d83700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87b40bd9c0 0x7f87b40bae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.367+0000 7f87b8d83700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87940682f0 0x7f879406a7a0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.367+0000 7f87b8d83700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f879c005250 0x7f87b4154160 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.367+0000 7f87b8d83700 1 --2- 172.21.15.7:0/2560129193 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f87b40be020 0x7f87b40bd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:03:08.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.368+0000 7f87b8d83700 1 -- 172.21.15.7:0/2560129193 >> 172.21.15.7:0/2560129193 conn(0x7f87b40a94d0 msgr2=0x7f87b41523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:03:08.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.368+0000 7f87b8d83700 1 -- 172.21.15.7:0/2560129193 shutdown_connections 2023-03-03T18:03:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:03:08.368+0000 7f87b8d83700 1 -- 172.21.15.7:0/2560129193 wait complete. 2023-03-03T18:03:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:03:08.380 INFO:tasks.mds_thrash.fs.[b]:waiting for 102 secs before thrashing 2023-03-03T18:03:33.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:03:33.776 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:03.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:03.851 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:33.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:33.945 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:04:36.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.539+0000 7f13e63ef700 1 Processor -- start 2023-03-03T18:04:36.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.539+0000 7f13e63ef700 1 -- start start 2023-03-03T18:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13e63ef700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f13e0072250 0x7f13e014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13e63ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13e00c1ce0 0x7f13e014e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:36.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13e63ef700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e01510d0 con 0x7f13e0071e80 2023-03-03T18:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13e63ef700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e00c3740 con 0x7f13e0072250 2023-03-03T18:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13e63ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e0070670 con 0x7f13e00c1ce0 2023-03-03T18:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13df7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f13e0072250 0x7f13e014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13dffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f13e0071e80 0x7f13e0151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57908/0 (socket says 172.21.15.7:57908) 2023-03-03T18:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13dffff700 1 -- 172.21.15.7:0/2960611988 learned_addr learned my addr 172.21.15.7:0/2960611988 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13e498c700 1 --2- 172.21.15.7:0/2960611988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13e00c1ce0 0x7f13e014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:36.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13df7fe700 1 -- 172.21.15.7:0/2960611988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13e00c1ce0 msgr2=0x7f13e014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:04:36.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13df7fe700 1 --2- 172.21.15.7:0/2960611988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13e00c1ce0 0x7f13e014e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:36.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13df7fe700 1 -- 172.21.15.7:0/2960611988 >> v1:172.21.15.7:6789/0 conn(0x7f13e0071e80 legacy=0x7f13e0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:36.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13deffd700 1 -- 172.21.15.7:0/2960611988 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059045610 0 0) 0x7f13e01510d0 con 0x7f13e0071e80 2023-03-03T18:04:36.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13df7fe700 1 -- 172.21.15.7:0/2960611988 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00c39e0 con 0x7f13e0072250 2023-03-03T18:04:36.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.540+0000 7f13df7fe700 1 --2- 172.21.15.7:0/2960611988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0072250 0x7f13e014e1f0 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f13c8009840 tx=0x7f13c8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c39c4d1b86cc67f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:04:36.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13deffd700 1 -- 172.21.15.7:0/2960611988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8011830 con 0x7f13e0072250 2023-03-03T18:04:36.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13deffd700 1 -- 172.21.15.7:0/2960611988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c8011990 con 0x7f13e0072250 2023-03-03T18:04:36.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13deffd700 1 -- 172.21.15.7:0/2960611988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8011d20 con 0x7f13e0072250 2023-03-03T18:04:36.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13e63ef700 1 -- 172.21.15.7:0/2960611988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0072250 msgr2=0x7f13e014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:36.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13e63ef700 1 --2- 172.21.15.7:0/2960611988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0072250 0x7f13e014e1f0 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f13c8009840 tx=0x7f13c8009b50 comp rx=0 tx=0).stop 2023-03-03T18:04:36.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13e63ef700 1 -- 172.21.15.7:0/2960611988 shutdown_connections 2023-03-03T18:04:36.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13e63ef700 1 --2- 172.21.15.7:0/2960611988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13e00c1ce0 0x7f13e014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:36.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13e63ef700 1 --2- 172.21.15.7:0/2960611988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0072250 0x7f13e014e1f0 unknown :-1 s=CLOSED pgs=3096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:36.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13e63ef700 1 -- 172.21.15.7:0/2960611988 >> 172.21.15.7:0/2960611988 conn(0x7f13e00a94d0 msgr2=0x7f13e00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:04:36.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13e63ef700 1 -- 172.21.15.7:0/2960611988 shutdown_connections 2023-03-03T18:04:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.541+0000 7f13e63ef700 1 -- 172.21.15.7:0/2960611988 wait complete. 2023-03-03T18:04:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.542+0000 7f13e63ef700 1 Processor -- start 2023-03-03T18:04:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.543+0000 7f13e63ef700 1 -- start start 2023-03-03T18:04:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.543+0000 7f13e63ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0071e80 0x7f13e014a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.543+0000 7f13e63ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13e0072250 0x7f13e0146a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.543+0000 7f13e63ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13e00c1ce0 0x7f13e01e9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:36.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.543+0000 7f13e63ef700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e00c7010 con 0x7f13e0071e80 2023-03-03T18:04:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.543+0000 7f13e63ef700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e00c6d30 con 0x7f13e0072250 2023-03-03T18:04:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.543+0000 7f13e63ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e0150360 con 0x7f13e00c1ce0 2023-03-03T18:04:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.543+0000 7f13dffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0071e80 0x7f13e014a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.543+0000 7f13dffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0071e80 0x7f13e014a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35702/0 (socket says 172.21.15.7:35702) 2023-03-03T18:04:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.543+0000 7f13dffff700 1 -- 172.21.15.7:0/3861133931 learned_addr learned my addr 172.21.15.7:0/3861133931 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:04:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13e498c700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13e00c1ce0 0x7f13e01e9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:36.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13df7fe700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13e0072250 0x7f13e0146a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:36.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13dffff700 1 -- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13e00c1ce0 msgr2=0x7f13e01e9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:36.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13dffff700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13e00c1ce0 0x7f13e01e9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:36.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13dffff700 1 -- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13e0072250 msgr2=0x7f13e0146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:36.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13dffff700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13e0072250 0x7f13e0146a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:36.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13dffff700 1 -- 172.21.15.7:0/3861133931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c80094a0 con 0x7f13e0071e80 2023-03-03T18:04:36.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13dffff700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0071e80 0x7f13e014a350 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f13d00011e0 tx=0x7f13d000ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:04:36.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13dcff9700 1 -- 172.21.15.7:0/3861133931 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0004c80 con 0x7f13e0071e80 2023-03-03T18:04:36.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13dcff9700 1 -- 172.21.15.7:0/3861133931 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0004e00 con 0x7f13e0071e80 2023-03-03T18:04:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13dcff9700 1 -- 172.21.15.7:0/3861133931 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0007310 con 0x7f13e0071e80 2023-03-03T18:04:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13e63ef700 1 -- 172.21.15.7:0/3861133931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e0151f30 con 0x7f13e0071e80 2023-03-03T18:04:36.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.544+0000 7f13e63ef700 1 -- 172.21.15.7:0/3861133931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e0151300 con 0x7f13e0071e80 2023-03-03T18:04:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.545+0000 7f13e63ef700 1 -- 172.21.15.7:0/3861133931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e006c9a0 con 0x7f13e0071e80 2023-03-03T18:04:36.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.546+0000 7f13dcff9700 1 -- 172.21.15.7:0/3861133931 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f13d001a020 con 0x7f13e0071e80 2023-03-03T18:04:36.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.546+0000 7f13dcff9700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13c00682f0 0x7f13c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:36.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.546+0000 7f13dcff9700 1 -- 172.21.15.7:0/3861133931 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f13d0085c30 con 0x7f13e0071e80 2023-03-03T18:04:36.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.546+0000 7f13df7fe700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13c00682f0 0x7f13c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:36.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.547+0000 7f13df7fe700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13c00682f0 0x7f13c006a7a0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f13e00c1780 tx=0x7f13c8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:04:36.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.551+0000 7f13dcff9700 1 -- 172.21.15.7:0/3861133931 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13d0051fd0 con 0x7f13e0071e80 2023-03-03T18:04:36.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.791+0000 7f13e63ef700 1 -- 172.21.15.7:0/3861133931 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13e01501a0 con 0x7f13e0071e80 2023-03-03T18:04:36.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.792+0000 7f13dcff9700 1 -- 172.21.15.7:0/3861133931 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f13d00555f0 con 0x7f13e0071e80 2023-03-03T18:04:36.792 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:04:36.792 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:04:36.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 -- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13c00682f0 msgr2=0x7f13c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:36.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13c00682f0 0x7f13c006a7a0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f13e00c1780 tx=0x7f13c8008030 comp rx=0 tx=0).stop 2023-03-03T18:04:36.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 -- 172.21.15.7:0/3861133931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0071e80 msgr2=0x7f13e014a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0071e80 0x7f13e014a350 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f13d00011e0 tx=0x7f13d000ca30 comp rx=0 tx=0).stop 2023-03-03T18:04:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 -- 172.21.15.7:0/3861133931 shutdown_connections 2023-03-03T18:04:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13e00c1ce0 0x7f13e01e9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f13c00682f0 0x7f13c006a7a0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13e0071e80 0x7f13e014a350 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 --2- 172.21.15.7:0/3861133931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13e0072250 0x7f13e0146a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 -- 172.21.15.7:0/3861133931 >> 172.21.15.7:0/3861133931 conn(0x7f13e00a94d0 msgr2=0x7f13e0066130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:04:36.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 -- 172.21.15.7:0/3861133931 shutdown_connections 2023-03-03T18:04:36.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:36.794+0000 7f13e63ef700 1 -- 172.21.15.7:0/3861133931 wait complete. 2023-03-03T18:04:36.796 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:04:36.806 INFO:tasks.mds_thrash.fs.[a]:waiting for 88 secs before thrashing 2023-03-03T18:04:50.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:04:50.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.520+0000 7fe0e0534700 1 Processor -- start 2023-03-03T18:04:50.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.520+0000 7fe0e0534700 1 -- start start 2023-03-03T18:04:50.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0e0534700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe0d806bea0 0x7fe0d806c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:50.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0e0534700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe0d8157dc0 0x7fe0d8158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:50.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0e0534700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d806d780 con 0x7fe0d8151ab0 2023-03-03T18:04:50.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0e0534700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d806c830 con 0x7fe0d806bea0 2023-03-03T18:04:50.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0e0534700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d806cb70 con 0x7fe0d8157dc0 2023-03-03T18:04:50.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0ddacf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe0d806bea0 0x7fe0d806c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:50.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0ddacf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe0d806bea0 0x7fe0d806c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35726/0 (socket says 172.21.15.7:35726) 2023-03-03T18:04:50.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0ddacf700 1 -- 172.21.15.7:0/427396644 learned_addr learned my addr 172.21.15.7:0/427396644 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:04:50.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0dead1700 1 --2- 172.21.15.7:0/427396644 >> v2:172.21.15.143:3300/0 conn(0x7fe0d8157dc0 0x7fe0d8158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:50.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0ddacf700 1 -- 172.21.15.7:0/427396644 >> v2:172.21.15.143:3300/0 conn(0x7fe0d8157dc0 msgr2=0x7fe0d8158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:04:50.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0ddacf700 1 --2- 172.21.15.7:0/427396644 >> v2:172.21.15.143:3300/0 conn(0x7fe0d8157dc0 0x7fe0d8158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:50.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.521+0000 7fe0ddacf700 1 -- 172.21.15.7:0/427396644 >> v1:172.21.15.143:6789/0 conn(0x7fe0d8151ab0 legacy=0x7fe0d80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:50.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.522+0000 7fe0ddacf700 1 -- 172.21.15.7:0/427396644 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d806ce50 con 0x7fe0d806bea0 2023-03-03T18:04:50.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.522+0000 7fe0ddacf700 1 --2- 172.21.15.7:0/427396644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d806bea0 0x7fe0d806c2f0 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40098f0 tx=0x7fe0d4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd1640f84bca448 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:04:50.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.522+0000 7fe0dd2ce700 1 -- 172.21.15.7:0/427396644 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d400b860 con 0x7fe0d806bea0 2023-03-03T18:04:50.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.522+0000 7fe0dd2ce700 1 -- 172.21.15.7:0/427396644 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d400b9c0 con 0x7fe0d806bea0 2023-03-03T18:04:50.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.522+0000 7fe0dd2ce700 1 -- 172.21.15.7:0/427396644 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d400bd50 con 0x7fe0d806bea0 2023-03-03T18:04:50.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.522+0000 7fe0e0534700 1 -- 172.21.15.7:0/427396644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d806bea0 msgr2=0x7fe0d806c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:50.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.522+0000 7fe0e0534700 1 --2- 172.21.15.7:0/427396644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d806bea0 0x7fe0d806c2f0 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40098f0 tx=0x7fe0d4009c00 comp rx=0 tx=0).stop 2023-03-03T18:04:50.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.523+0000 7fe0e0534700 1 -- 172.21.15.7:0/427396644 shutdown_connections 2023-03-03T18:04:50.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.523+0000 7fe0e0534700 1 --2- 172.21.15.7:0/427396644 >> v2:172.21.15.143:3300/0 conn(0x7fe0d8157dc0 0x7fe0d8158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:50.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.523+0000 7fe0e0534700 1 --2- 172.21.15.7:0/427396644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d806bea0 0x7fe0d806c2f0 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:50.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.523+0000 7fe0e0534700 1 -- 172.21.15.7:0/427396644 >> 172.21.15.7:0/427396644 conn(0x7fe0d80a94d0 msgr2=0x7fe0d80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:04:50.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.523+0000 7fe0e0534700 1 -- 172.21.15.7:0/427396644 shutdown_connections 2023-03-03T18:04:50.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.523+0000 7fe0e0534700 1 -- 172.21.15.7:0/427396644 wait complete. 2023-03-03T18:04:50.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.524+0000 7fe0e0534700 1 Processor -- start 2023-03-03T18:04:50.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.524+0000 7fe0e0534700 1 -- start start 2023-03-03T18:04:50.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.524+0000 7fe0e0534700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0d806bea0 0x7fe0d81ead80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:50.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0e0534700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d8151ab0 0x7fe0d81e0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:50.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0e0534700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0d8157dc0 0x7fe0d81de210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:50.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0e0534700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d81eb2c0 con 0x7fe0d8151ab0 2023-03-03T18:04:50.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0e0534700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d81ea8f0 con 0x7fe0d806bea0 2023-03-03T18:04:50.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0e0534700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d81ea300 con 0x7fe0d8157dc0 2023-03-03T18:04:50.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ddacf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d8151ab0 0x7fe0d81e0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:50.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ddacf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d8151ab0 0x7fe0d81e0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35732/0 (socket says 172.21.15.7:35732) 2023-03-03T18:04:50.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ddacf700 1 -- 172.21.15.7:0/1813273027 learned_addr learned my addr 172.21.15.7:0/1813273027 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:04:50.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ddacf700 1 -- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0d8157dc0 msgr2=0x7fe0d81de210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:50.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0de2d0700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0d806bea0 0x7fe0d81ead80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:50.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0dead1700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0d8157dc0 0x7fe0d81de210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:50.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ddacf700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0d8157dc0 0x7fe0d81de210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:50.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ddacf700 1 -- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0d806bea0 msgr2=0x7fe0d81ead80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:50.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ddacf700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0d806bea0 0x7fe0d81ead80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:50.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ddacf700 1 -- 172.21.15.7:0/1813273027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d4009550 con 0x7fe0d8151ab0 2023-03-03T18:04:50.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0de2d0700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0d806bea0 0x7fe0d81ead80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:04:50.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0dead1700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0d8157dc0 0x7fe0d81de210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:04:50.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ddacf700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d8151ab0 0x7fe0d81e0230 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7fe0d806bb50 tx=0x7fe0d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:04:50.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ceffd700 1 -- 172.21.15.7:0/1813273027 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4004400 con 0x7fe0d8151ab0 2023-03-03T18:04:50.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ceffd700 1 -- 172.21.15.7:0/1813273027 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d401f070 con 0x7fe0d8151ab0 2023-03-03T18:04:50.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0e0534700 1 -- 172.21.15.7:0/1813273027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d806d410 con 0x7fe0d8151ab0 2023-03-03T18:04:50.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0ceffd700 1 -- 172.21.15.7:0/1813273027 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d40038c0 con 0x7fe0d8151ab0 2023-03-03T18:04:50.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.525+0000 7fe0e0534700 1 -- 172.21.15.7:0/1813273027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d806d130 con 0x7fe0d8151ab0 2023-03-03T18:04:50.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.526+0000 7fe0e0534700 1 -- 172.21.15.7:0/1813273027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d8073e00 con 0x7fe0d8151ab0 2023-03-03T18:04:50.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.527+0000 7fe0ceffd700 1 -- 172.21.15.7:0/1813273027 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe0d402a020 con 0x7fe0d8151ab0 2023-03-03T18:04:50.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.527+0000 7fe0ceffd700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0bc068320 0x7fe0bc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:04:50.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.527+0000 7fe0ceffd700 1 -- 172.21.15.7:0/1813273027 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe0d40876f0 con 0x7fe0d8151ab0 2023-03-03T18:04:50.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.527+0000 7fe0de2d0700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0bc068320 0x7fe0bc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:04:50.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.528+0000 7fe0de2d0700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0bc068320 0x7fe0bc06a7d0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8000f60 tx=0x7fe0c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:04:50.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.531+0000 7fe0ceffd700 1 -- 172.21.15.7:0/1813273027 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe0d4053a10 con 0x7fe0d8151ab0 2023-03-03T18:04:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.767+0000 7fe0e0534700 1 -- 172.21.15.7:0/1813273027 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0d8074bb0 con 0x7fe0d8151ab0 2023-03-03T18:04:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.767+0000 7fe0ceffd700 1 -- 172.21.15.7:0/1813273027 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe0d4057030 con 0x7fe0d8151ab0 2023-03-03T18:04:50.767 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:04:50.767 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:04:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.769+0000 7fe0e0534700 1 -- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0bc068320 msgr2=0x7fe0bc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.769+0000 7fe0e0534700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0bc068320 0x7fe0bc06a7d0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8000f60 tx=0x7fe0c8009040 comp rx=0 tx=0).stop 2023-03-03T18:04:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.769+0000 7fe0e0534700 1 -- 172.21.15.7:0/1813273027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d8151ab0 msgr2=0x7fe0d81e0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:04:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.769+0000 7fe0e0534700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d8151ab0 0x7fe0d81e0230 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7fe0d806bb50 tx=0x7fe0d4004750 comp rx=0 tx=0).stop 2023-03-03T18:04:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.769+0000 7fe0e0534700 1 -- 172.21.15.7:0/1813273027 shutdown_connections 2023-03-03T18:04:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.769+0000 7fe0e0534700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0d8157dc0 0x7fe0d81de210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.769+0000 7fe0e0534700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0d806bea0 0x7fe0d81ead80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.769+0000 7fe0e0534700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0bc068320 0x7fe0bc06a7d0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.769+0000 7fe0e0534700 1 --2- 172.21.15.7:0/1813273027 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0d8151ab0 0x7fe0d81e0230 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:04:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.769+0000 7fe0e0534700 1 -- 172.21.15.7:0/1813273027 >> 172.21.15.7:0/1813273027 conn(0x7fe0d80a94d0 msgr2=0x7fe0d8150120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:04:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.770+0000 7fe0e0534700 1 -- 172.21.15.7:0/1813273027 shutdown_connections 2023-03-03T18:04:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:04:50.770+0000 7fe0e0534700 1 -- 172.21.15.7:0/1813273027 wait complete. 2023-03-03T18:04:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:04:50.781 INFO:tasks.mds_thrash.fs.[b]:waiting for 72 secs before thrashing 2023-03-03T18:05:04.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:04.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:34.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:05:34.097 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:02.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:06:02.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.922+0000 7fd77fcdb700 1 Processor -- start 2023-03-03T18:06:02.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.922+0000 7fd77fcdb700 1 -- start start 2023-03-03T18:06:02.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.923+0000 7fd77fcdb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd7781550c0 0x7fd77815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:02.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.923+0000 7fd77fcdb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7780c7a60 con 0x7fd77806d870 2023-03-03T18:06:02.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.923+0000 7fd77fcdb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7780c3620 con 0x7fd7780c1da0 2023-03-03T18:06:02.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.923+0000 7fd77fcdb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7781554c0 con 0x7fd7781550c0 2023-03-03T18:06:02.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.923+0000 7fd77da77700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd77806d870 0x7fd7780c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57996/0 (socket says 172.21.15.7:57996) 2023-03-03T18:06:02.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.923+0000 7fd77da77700 1 -- 172.21.15.7:0/1292384912 learned_addr learned my addr 172.21.15.7:0/1292384912 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:06:02.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77e278700 1 --2- 172.21.15.7:0/1292384912 >> v2:172.21.15.143:3300/0 conn(0x7fd7781550c0 0x7fd77815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:02.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77ca75700 1 -- 172.21.15.7:0/1292384912 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695428667 0 0) 0x7fd7780c7a60 con 0x7fd77806d870 2023-03-03T18:06:02.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77ca75700 1 -- 172.21.15.7:0/1292384912 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7640035b0 con 0x7fd77806d870 2023-03-03T18:06:02.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77ca75700 1 -- 172.21.15.7:0/1292384912 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564888259 0 0) 0x7fd7780c3620 con 0x7fd7780c1da0 2023-03-03T18:06:02.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77ca75700 1 -- 172.21.15.7:0/1292384912 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7780c7a60 con 0x7fd7780c1da0 2023-03-03T18:06:02.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77ca75700 1 -- 172.21.15.7:0/1292384912 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136008094 0 0) 0x7fd7640035b0 con 0x7fd77806d870 2023-03-03T18:06:02.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77e278700 1 -- 172.21.15.7:0/1292384912 >> v1:172.21.15.143:6789/0 conn(0x7fd7780c1da0 legacy=0x7fd7780c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:02.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77e278700 1 -- 172.21.15.7:0/1292384912 >> v1:172.21.15.7:6789/0 conn(0x7fd77806d870 legacy=0x7fd7780c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:02.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77e278700 1 -- 172.21.15.7:0/1292384912 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7781545e0 con 0x7fd7781550c0 2023-03-03T18:06:02.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77e278700 1 --2- 172.21.15.7:0/1292384912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7781550c0 0x7fd77815fe60 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd770009d90 tx=0x7fd77000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=870ee424290eb47c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:02.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77ca75700 1 -- 172.21.15.7:0/1292384912 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77000e070 con 0x7fd7781550c0 2023-03-03T18:06:02.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77ca75700 1 -- 172.21.15.7:0/1292384912 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd770003a20 con 0x7fd7781550c0 2023-03-03T18:06:02.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.924+0000 7fd77ca75700 1 -- 172.21.15.7:0/1292384912 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768001670 con 0x7fd7781550c0 2023-03-03T18:06:02.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.925+0000 7fd77fcdb700 1 -- 172.21.15.7:0/1292384912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7781550c0 msgr2=0x7fd77815fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:02.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.925+0000 7fd77fcdb700 1 --2- 172.21.15.7:0/1292384912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7781550c0 0x7fd77815fe60 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd770009d90 tx=0x7fd77000b330 comp rx=0 tx=0).stop 2023-03-03T18:06:02.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.925+0000 7fd77fcdb700 1 -- 172.21.15.7:0/1292384912 shutdown_connections 2023-03-03T18:06:02.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.925+0000 7fd77fcdb700 1 --2- 172.21.15.7:0/1292384912 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7781550c0 0x7fd77815fe60 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:02.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.925+0000 7fd77fcdb700 1 -- 172.21.15.7:0/1292384912 >> 172.21.15.7:0/1292384912 conn(0x7fd77806c110 msgr2=0x7fd77806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:06:02.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.925+0000 7fd77fcdb700 1 -- 172.21.15.7:0/1292384912 shutdown_connections 2023-03-03T18:06:02.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.925+0000 7fd77fcdb700 1 -- 172.21.15.7:0/1292384912 wait complete. 2023-03-03T18:06:02.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.926+0000 7fd77fcdb700 1 Processor -- start 2023-03-03T18:06:02.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.926+0000 7fd77fcdb700 1 -- start start 2023-03-03T18:06:02.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77fcdb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd77806d870 0x7fd7780c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:02.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77fcdb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7780c1da0 0x7fd77813cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:02.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77fcdb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7781550c0 0x7fd77813d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:02.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77fcdb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd778157890 con 0x7fd7781550c0 2023-03-03T18:06:02.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77fcdb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7781588f0 con 0x7fd7780c1da0 2023-03-03T18:06:02.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77fcdb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd778157b30 con 0x7fd77806d870 2023-03-03T18:06:02.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77e278700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7781550c0 0x7fd77813d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:02.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77e278700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7781550c0 0x7fd77813d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35792/0 (socket says 172.21.15.7:35792) 2023-03-03T18:06:02.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77e278700 1 -- 172.21.15.7:0/2739558729 learned_addr learned my addr 172.21.15.7:0/2739558729 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:06:02.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77d276700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7780c1da0 0x7fd77813cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:02.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77da77700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd77806d870 0x7fd7780c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:02.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77e278700 1 -- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd77806d870 msgr2=0x7fd7780c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:02.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77e278700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd77806d870 0x7fd7780c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:02.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77e278700 1 -- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7780c1da0 msgr2=0x7fd77813cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:02.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77e278700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7780c1da0 0x7fd77813cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:02.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77e278700 1 -- 172.21.15.7:0/2739558729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd770009a70 con 0x7fd7781550c0 2023-03-03T18:06:02.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77da77700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd77806d870 0x7fd7780c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:06:02.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77d276700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7780c1da0 0x7fd77813cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:06:02.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.927+0000 7fd77e278700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7781550c0 0x7fd77813d130 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7fd77000b370 tx=0x7fd770004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:02.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.928+0000 7fd76e7fc700 1 -- 172.21.15.7:0/2739558729 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7700045c0 con 0x7fd7781550c0 2023-03-03T18:06:02.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.928+0000 7fd76e7fc700 1 -- 172.21.15.7:0/2739558729 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd770004740 con 0x7fd7781550c0 2023-03-03T18:06:02.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.928+0000 7fd77fcdb700 1 -- 172.21.15.7:0/2739558729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd778154ba0 con 0x7fd7781550c0 2023-03-03T18:06:02.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.928+0000 7fd76e7fc700 1 -- 172.21.15.7:0/2739558729 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77001d470 con 0x7fd7781550c0 2023-03-03T18:06:02.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.928+0000 7fd77fcdb700 1 -- 172.21.15.7:0/2739558729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7781548c0 con 0x7fd7781550c0 2023-03-03T18:06:02.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.929+0000 7fd77fcdb700 1 -- 172.21.15.7:0/2739558729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77806dc40 con 0x7fd7781550c0 2023-03-03T18:06:02.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.929+0000 7fd76e7fc700 1 -- 172.21.15.7:0/2739558729 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd770018070 con 0x7fd7781550c0 2023-03-03T18:06:02.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.929+0000 7fd76e7fc700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd75c0682f0 0x7fd75c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:02.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.930+0000 7fd76e7fc700 1 -- 172.21.15.7:0/2739558729 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd770087d20 con 0x7fd7781550c0 2023-03-03T18:06:02.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.930+0000 7fd77da77700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd75c0682f0 0x7fd75c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:02.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.930+0000 7fd77da77700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd75c0682f0 0x7fd75c06a7a0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fd768000fd0 tx=0x7fd768009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:02.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:02.934+0000 7fd76e7fc700 1 -- 172.21.15.7:0/2739558729 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd770054040 con 0x7fd7781550c0 2023-03-03T18:06:03.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.175+0000 7fd77fcdb700 1 -- 172.21.15.7:0/2739558729 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd778074bb0 con 0x7fd7781550c0 2023-03-03T18:06:03.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.175+0000 7fd76e7fc700 1 -- 172.21.15.7:0/2739558729 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd770057660 con 0x7fd7781550c0 2023-03-03T18:06:03.176 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:06:03.176 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:06:03.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 -- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd75c0682f0 msgr2=0x7fd75c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:03.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd75c0682f0 0x7fd75c06a7a0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fd768000fd0 tx=0x7fd768009040 comp rx=0 tx=0).stop 2023-03-03T18:06:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 -- 172.21.15.7:0/2739558729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7781550c0 msgr2=0x7fd77813d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7781550c0 0x7fd77813d130 secure :-1 s=READY pgs=3101 cs=0 l=1 rev1=1 crypto rx=0x7fd77000b370 tx=0x7fd770004ad0 comp rx=0 tx=0).stop 2023-03-03T18:06:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 -- 172.21.15.7:0/2739558729 shutdown_connections 2023-03-03T18:06:03.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd75c0682f0 0x7fd75c06a7a0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7781550c0 0x7fd77813d130 unknown :-1 s=CLOSED pgs=3101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd77806d870 0x7fd7780c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 --2- 172.21.15.7:0/2739558729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7780c1da0 0x7fd77813cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:03.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 -- 172.21.15.7:0/2739558729 >> 172.21.15.7:0/2739558729 conn(0x7fd77806c110 msgr2=0x7fd77806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:06:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.177+0000 7fd77fcdb700 1 -- 172.21.15.7:0/2739558729 shutdown_connections 2023-03-03T18:06:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:03.178+0000 7fd77fcdb700 1 -- 172.21.15.7:0/2739558729 wait complete. 2023-03-03T18:06:03.180 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:06:03.189 INFO:tasks.mds_thrash.fs.[b]:waiting for 29 secs before thrashing 2023-03-03T18:06:04.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:04.171 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:04.807 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:06:04.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.947+0000 7f2170db7700 1 Processor -- start 2023-03-03T18:06:04.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.947+0000 7f2170db7700 1 -- start start 2023-03-03T18:06:04.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f2170db7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f216c14b0d0 0x7f216c071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:04.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f2170db7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c071d40 0x7f216c158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:04.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f2170db7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216c0a9c20 con 0x7f216c0aad20 2023-03-03T18:06:04.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f2170db7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216c154f10 con 0x7f216c14b0d0 2023-03-03T18:06:04.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f2170db7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0ac3c0 con 0x7f216c071d40 2023-03-03T18:06:04.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f216a59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f216c14b0d0 0x7f216c071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:04.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f2169d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f216c0aad20 0x7f216c157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58018/0 (socket says 172.21.15.7:58018) 2023-03-03T18:06:04.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f2169d9b700 1 -- 172.21.15.7:0/1644835284 learned_addr learned my addr 172.21.15.7:0/1644835284 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:06:04.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f216ad9d700 1 --2- 172.21.15.7:0/1644835284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c071d40 0x7f216c158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:04.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f216a59c700 1 -- 172.21.15.7:0/1644835284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c071d40 msgr2=0x7f216c158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:06:04.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f216a59c700 1 --2- 172.21.15.7:0/1644835284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c071d40 0x7f216c158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:04.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f216a59c700 1 -- 172.21.15.7:0/1644835284 >> v1:172.21.15.7:6789/0 conn(0x7f216c0aad20 legacy=0x7f216c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:04.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f216a59c700 1 -- 172.21.15.7:0/1644835284 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c06fc70 con 0x7f216c14b0d0 2023-03-03T18:06:04.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f216959a700 1 -- 172.21.15.7:0/1644835284 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086013530 0 0) 0x7f216c0a9c20 con 0x7f216c0aad20 2023-03-03T18:06:04.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f216a59c700 1 --2- 172.21.15.7:0/1644835284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 0x7f216c071800 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7f215c009840 tx=0x7f215c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e931f3a9f839faa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:04.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.948+0000 7f216959a700 1 -- 172.21.15.7:0/1644835284 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c011830 con 0x7f216c14b0d0 2023-03-03T18:06:04.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.949+0000 7f216959a700 1 -- 172.21.15.7:0/1644835284 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215c011990 con 0x7f216c14b0d0 2023-03-03T18:06:04.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.949+0000 7f216959a700 1 -- 172.21.15.7:0/1644835284 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c011d20 con 0x7f216c14b0d0 2023-03-03T18:06:04.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.949+0000 7f2170db7700 1 -- 172.21.15.7:0/1644835284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 msgr2=0x7f216c071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:04.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.949+0000 7f2170db7700 1 --2- 172.21.15.7:0/1644835284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 0x7f216c071800 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7f215c009840 tx=0x7f215c009b50 comp rx=0 tx=0).stop 2023-03-03T18:06:04.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.949+0000 7f2170db7700 1 -- 172.21.15.7:0/1644835284 shutdown_connections 2023-03-03T18:06:04.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.949+0000 7f2170db7700 1 --2- 172.21.15.7:0/1644835284 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c071d40 0x7f216c158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:04.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.949+0000 7f2170db7700 1 --2- 172.21.15.7:0/1644835284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 0x7f216c071800 unknown :-1 s=CLOSED pgs=3103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:04.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.949+0000 7f2170db7700 1 -- 172.21.15.7:0/1644835284 >> 172.21.15.7:0/1644835284 conn(0x7f216c0c8900 msgr2=0x7f216c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:06:04.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.949+0000 7f2170db7700 1 -- 172.21.15.7:0/1644835284 shutdown_connections 2023-03-03T18:06:04.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.949+0000 7f2170db7700 1 -- 172.21.15.7:0/1644835284 wait complete. 2023-03-03T18:06:04.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.951+0000 7f2170db7700 1 Processor -- start 2023-03-03T18:06:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.951+0000 7f2170db7700 1 -- start start 2023-03-03T18:06:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f2170db7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c071d40 0x7f216c1de4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f2170db7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0aad20 0x7f216c1568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f2170db7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 0x7f216c1458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f2170db7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c1e1610 con 0x7f216c14b0d0 2023-03-03T18:06:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f2170db7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c1dfae0 con 0x7f216c071d40 2023-03-03T18:06:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f2170db7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c1e0cb0 con 0x7f216c0aad20 2023-03-03T18:06:04.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f216ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 0x7f216c1458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f216ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 0x7f216c1458a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35814/0 (socket says 172.21.15.7:35814) 2023-03-03T18:06:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f216ad9d700 1 -- 172.21.15.7:0/1936407050 learned_addr learned my addr 172.21.15.7:0/1936407050 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:06:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f2169d9b700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0aad20 0x7f216c1568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f216a59c700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c071d40 0x7f216c1de4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f216ad9d700 1 -- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0aad20 msgr2=0x7f216c1568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f216ad9d700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0aad20 0x7f216c1568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:04.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f216ad9d700 1 -- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c071d40 msgr2=0x7f216c1de4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f216ad9d700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c071d40 0x7f216c1de4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f216ad9d700 1 -- 172.21.15.7:0/1936407050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c0094a0 con 0x7f216c14b0d0 2023-03-03T18:06:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.952+0000 7f216a59c700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c071d40 0x7f216c1de4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:06:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.953+0000 7f216ad9d700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 0x7f216c1458a0 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7f216400efc0 tx=0x7f2164009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.953+0000 7f215affd700 1 -- 172.21.15.7:0/1936407050 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2164009b20 con 0x7f216c14b0d0 2023-03-03T18:06:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.953+0000 7f215affd700 1 -- 172.21.15.7:0/1936407050 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216400f040 con 0x7f216c14b0d0 2023-03-03T18:06:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.953+0000 7f2170db7700 1 -- 172.21.15.7:0/1936407050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216c06f6b0 con 0x7f216c14b0d0 2023-03-03T18:06:04.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.953+0000 7f215affd700 1 -- 172.21.15.7:0/1936407050 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216400cd40 con 0x7f216c14b0d0 2023-03-03T18:06:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.953+0000 7f2170db7700 1 -- 172.21.15.7:0/1936407050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c155160 con 0x7f216c14b0d0 2023-03-03T18:06:04.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.954+0000 7f2170db7700 1 -- 172.21.15.7:0/1936407050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c06f990 con 0x7f216c14b0d0 2023-03-03T18:06:04.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.954+0000 7f215affd700 1 -- 172.21.15.7:0/1936407050 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2164014480 con 0x7f216c14b0d0 2023-03-03T18:06:04.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.955+0000 7f215affd700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f214c0682f0 0x7f214c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:04.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.955+0000 7f215affd700 1 -- 172.21.15.7:0/1936407050 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2164086d60 con 0x7f216c14b0d0 2023-03-03T18:06:04.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.955+0000 7f216a59c700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f214c0682f0 0x7f214c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:04.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.956+0000 7f216a59c700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f214c0682f0 0x7f214c06a7a0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f215c000c00 tx=0x7f215c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:04.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:04.959+0000 7f215affd700 1 -- 172.21.15.7:0/1936407050 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f21640530a0 con 0x7f216c14b0d0 2023-03-03T18:06:05.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.196+0000 7f2170db7700 1 -- 172.21.15.7:0/1936407050 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f216c1e1750 con 0x7f216c14b0d0 2023-03-03T18:06:05.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.197+0000 7f215affd700 1 -- 172.21.15.7:0/1936407050 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f216c1e1750 con 0x7f216c14b0d0 2023-03-03T18:06:05.197 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:06:05.197 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:06:05.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 -- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f214c0682f0 msgr2=0x7f214c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:05.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f214c0682f0 0x7f214c06a7a0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f215c000c00 tx=0x7f215c008030 comp rx=0 tx=0).stop 2023-03-03T18:06:05.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 -- 172.21.15.7:0/1936407050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 msgr2=0x7f216c1458a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:05.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 0x7f216c1458a0 secure :-1 s=READY pgs=3104 cs=0 l=1 rev1=1 crypto rx=0x7f216400efc0 tx=0x7f2164009f60 comp rx=0 tx=0).stop 2023-03-03T18:06:05.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 -- 172.21.15.7:0/1936407050 shutdown_connections 2023-03-03T18:06:05.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f214c0682f0 0x7f214c06a7a0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:05.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f216c14b0d0 0x7f216c1458a0 unknown :-1 s=CLOSED pgs=3104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:05.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f216c071d40 0x7f216c1de4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:05.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 --2- 172.21.15.7:0/1936407050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f216c0aad20 0x7f216c1568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:05.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 -- 172.21.15.7:0/1936407050 >> 172.21.15.7:0/1936407050 conn(0x7f216c0c8900 msgr2=0x7f216c1469b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:06:05.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 -- 172.21.15.7:0/1936407050 shutdown_connections 2023-03-03T18:06:05.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:05.199+0000 7f2170db7700 1 -- 172.21.15.7:0/1936407050 wait complete. 2023-03-03T18:06:05.200 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:06:05.211 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2023-03-03T18:06:32.190 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:06:32.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:06:32.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.331+0000 7fef08c81700 1 Processor -- start 2023-03-03T18:06:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.331+0000 7fef08c81700 1 -- start start 2023-03-03T18:06:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.332+0000 7fef08c81700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef04155410 0x7fef040bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.332+0000 7fef08c81700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 0x7fef040bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.332+0000 7fef08c81700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef040c08d0 con 0x7fef04151e80 2023-03-03T18:06:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.332+0000 7fef08c81700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef04152250 con 0x7fef04155410 2023-03-03T18:06:32.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.332+0000 7fef08c81700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef040c0fb0 con 0x7fef040c5b00 2023-03-03T18:06:32.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.332+0000 7fef02d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 0x7fef040bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.332+0000 7fef01d9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef04155410 0x7fef040bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.332+0000 7fef02d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 0x7fef040bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:43912/0 (socket says 172.21.15.7:43912) 2023-03-03T18:06:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.332+0000 7fef0259c700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fef04151e80 0x7fef040c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:33336/0 (socket says 172.21.15.7:33336) 2023-03-03T18:06:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.332+0000 7fef02d9d700 1 -- 172.21.15.7:0/4176718554 learned_addr learned my addr 172.21.15.7:0/4176718554 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:06:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.333+0000 7fef02d9d700 1 -- 172.21.15.7:0/4176718554 >> v2:172.21.15.143:3300/0 conn(0x7fef04155410 msgr2=0x7fef040bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:06:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.333+0000 7fef02d9d700 1 --2- 172.21.15.7:0/4176718554 >> v2:172.21.15.143:3300/0 conn(0x7fef04155410 0x7fef040bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.333+0000 7fef02d9d700 1 -- 172.21.15.7:0/4176718554 >> v1:172.21.15.143:6789/0 conn(0x7fef04151e80 legacy=0x7fef040c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.333+0000 7fef02d9d700 1 -- 172.21.15.7:0/4176718554 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef04154930 con 0x7fef040c5b00 2023-03-03T18:06:32.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.333+0000 7fef02d9d700 1 --2- 172.21.15.7:0/4176718554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 0x7fef040bfe20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feefc009e50 tx=0x7feefc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71f355607508cc86 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.333+0000 7fef0159a700 1 -- 172.21.15.7:0/4176718554 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc00e070 con 0x7fef040c5b00 2023-03-03T18:06:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.333+0000 7fef0159a700 1 -- 172.21.15.7:0/4176718554 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feefc00b910 con 0x7fef040c5b00 2023-03-03T18:06:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.333+0000 7fef0159a700 1 -- 172.21.15.7:0/4176718554 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feefc00bcc0 con 0x7fef040c5b00 2023-03-03T18:06:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.334+0000 7fef08c81700 1 -- 172.21.15.7:0/4176718554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 msgr2=0x7fef040bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.334+0000 7fef08c81700 1 --2- 172.21.15.7:0/4176718554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 0x7fef040bfe20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feefc009e50 tx=0x7feefc00b330 comp rx=0 tx=0).stop 2023-03-03T18:06:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.334+0000 7fef08c81700 1 -- 172.21.15.7:0/4176718554 shutdown_connections 2023-03-03T18:06:32.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.334+0000 7fef08c81700 1 --2- 172.21.15.7:0/4176718554 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 0x7fef040bfe20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.334+0000 7fef08c81700 1 --2- 172.21.15.7:0/4176718554 >> v2:172.21.15.143:3300/0 conn(0x7fef04155410 0x7fef040bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.334+0000 7fef08c81700 1 -- 172.21.15.7:0/4176718554 >> 172.21.15.7:0/4176718554 conn(0x7fef04071630 msgr2=0x7fef0406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:06:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.334+0000 7fef08c81700 1 -- 172.21.15.7:0/4176718554 shutdown_connections 2023-03-03T18:06:32.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.334+0000 7fef08c81700 1 -- 172.21.15.7:0/4176718554 wait complete. 2023-03-03T18:06:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.336+0000 7fef08c81700 1 Processor -- start 2023-03-03T18:06:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.336+0000 7fef08c81700 1 -- start start 2023-03-03T18:06:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef08c81700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 0x7fef04150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef08c81700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04151e80 0x7fef041df420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef08c81700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef04155410 0x7fef041df960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef08c81700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef040c8f90 con 0x7fef04151e80 2023-03-03T18:06:32.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef08c81700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0414bb80 con 0x7fef04155410 2023-03-03T18:06:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef08c81700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef040bfa00 con 0x7fef040c5b00 2023-03-03T18:06:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef01d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04151e80 0x7fef041df420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef01d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04151e80 0x7fef041df420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35844/0 (socket says 172.21.15.7:35844) 2023-03-03T18:06:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef01d9b700 1 -- 172.21.15.7:0/3486349797 learned_addr learned my addr 172.21.15.7:0/3486349797 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:06:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef0259c700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 0x7fef04150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef02d9d700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef04155410 0x7fef041df960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef01d9b700 1 -- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 msgr2=0x7fef04150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef01d9b700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 0x7fef04150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef01d9b700 1 -- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef04155410 msgr2=0x7fef041df960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef01d9b700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef04155410 0x7fef041df960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef01d9b700 1 -- 172.21.15.7:0/3486349797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feefc009b30 con 0x7fef04151e80 2023-03-03T18:06:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.337+0000 7fef01d9b700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04151e80 0x7fef041df420 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7feef800e9e0 tx=0x7feef800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.338+0000 7feef2ffd700 1 -- 172.21.15.7:0/3486349797 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef800c7f0 con 0x7fef04151e80 2023-03-03T18:06:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.338+0000 7feef2ffd700 1 -- 172.21.15.7:0/3486349797 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef800c950 con 0x7fef04151e80 2023-03-03T18:06:32.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.338+0000 7fef08c81700 1 -- 172.21.15.7:0/3486349797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef04154ef0 con 0x7fef04151e80 2023-03-03T18:06:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.338+0000 7feef2ffd700 1 -- 172.21.15.7:0/3486349797 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef800ccc0 con 0x7fef04151e80 2023-03-03T18:06:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.338+0000 7fef08c81700 1 -- 172.21.15.7:0/3486349797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef04154c10 con 0x7fef04151e80 2023-03-03T18:06:32.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.339+0000 7fef08c81700 1 -- 172.21.15.7:0/3486349797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef040c5220 con 0x7fef04151e80 2023-03-03T18:06:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.340+0000 7feef2ffd700 1 -- 172.21.15.7:0/3486349797 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feef8023070 con 0x7fef04151e80 2023-03-03T18:06:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.340+0000 7feef2ffd700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 0x7feee406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.340+0000 7feef2ffd700 1 -- 172.21.15.7:0/3486349797 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7feef8086450 con 0x7fef04151e80 2023-03-03T18:06:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.341+0000 7fef0259c700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 0x7feee406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.341+0000 7fef0259c700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 0x7feee406a7a0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7feef4000f60 tx=0x7feef4007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:32.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.345+0000 7feef2ffd700 1 -- 172.21.15.7:0/3486349797 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feef80527f0 con 0x7fef04151e80 2023-03-03T18:06:32.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.368+0000 7fca84f23700 1 Processor -- start 2023-03-03T18:06:32.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.368+0000 7fca84f23700 1 -- start start 2023-03-03T18:06:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.369+0000 7fca84f23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca80151e80 0x7fca800c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.369+0000 7fca84f23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca80155410 0x7fca800bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.369+0000 7fca84f23700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca800c08d0 con 0x7fca800c6fe0 2023-03-03T18:06:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.369+0000 7fca84f23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca80152250 con 0x7fca80155410 2023-03-03T18:06:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.369+0000 7fca84f23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca800c0fb0 con 0x7fca80151e80 2023-03-03T18:06:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.369+0000 7fca7ed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fca800c6fe0 0x7fca800bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58066/0 (socket says 172.21.15.7:58066) 2023-03-03T18:06:32.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.369+0000 7fca7ed9d700 1 -- 172.21.15.7:0/1150182531 learned_addr learned my addr 172.21.15.7:0/1150182531 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:06:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7dd9b700 1 --2- 172.21.15.7:0/1150182531 >> v2:172.21.15.143:3300/0 conn(0x7fca80155410 0x7fca800bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7e59c700 1 --2- 172.21.15.7:0/1150182531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca80151e80 0x7fca800c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7d59a700 1 -- 172.21.15.7:0/1150182531 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265609070 0 0) 0x7fca800c08d0 con 0x7fca800c6fe0 2023-03-03T18:06:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7d59a700 1 -- 172.21.15.7:0/1150182531 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca680035b0 con 0x7fca800c6fe0 2023-03-03T18:06:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7e59c700 1 -- 172.21.15.7:0/1150182531 >> v2:172.21.15.143:3300/0 conn(0x7fca80155410 msgr2=0x7fca800bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:06:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7e59c700 1 --2- 172.21.15.7:0/1150182531 >> v2:172.21.15.143:3300/0 conn(0x7fca80155410 0x7fca800bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7e59c700 1 -- 172.21.15.7:0/1150182531 >> v1:172.21.15.7:6789/0 conn(0x7fca800c6fe0 legacy=0x7fca800bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7e59c700 1 -- 172.21.15.7:0/1150182531 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca80154930 con 0x7fca80151e80 2023-03-03T18:06:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7e59c700 1 --2- 172.21.15.7:0/1150182531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca80151e80 0x7fca800c6aa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fca70009840 tx=0x7fca70009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=459bdd0ce12c991c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7d59a700 1 -- 172.21.15.7:0/1150182531 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70011830 con 0x7fca80151e80 2023-03-03T18:06:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7d59a700 1 -- 172.21.15.7:0/1150182531 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca70011990 con 0x7fca80151e80 2023-03-03T18:06:32.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.370+0000 7fca7d59a700 1 -- 172.21.15.7:0/1150182531 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70011d20 con 0x7fca80151e80 2023-03-03T18:06:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.371+0000 7fca84f23700 1 -- 172.21.15.7:0/1150182531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca80151e80 msgr2=0x7fca800c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.371+0000 7fca84f23700 1 --2- 172.21.15.7:0/1150182531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca80151e80 0x7fca800c6aa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fca70009840 tx=0x7fca70009b50 comp rx=0 tx=0).stop 2023-03-03T18:06:32.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.371+0000 7fca84f23700 1 -- 172.21.15.7:0/1150182531 shutdown_connections 2023-03-03T18:06:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.371+0000 7fca84f23700 1 --2- 172.21.15.7:0/1150182531 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca80151e80 0x7fca800c6aa0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.371+0000 7fca84f23700 1 --2- 172.21.15.7:0/1150182531 >> v2:172.21.15.143:3300/0 conn(0x7fca80155410 0x7fca800bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.371+0000 7fca84f23700 1 -- 172.21.15.7:0/1150182531 >> 172.21.15.7:0/1150182531 conn(0x7fca80071630 msgr2=0x7fca8006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:06:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.371+0000 7fca84f23700 1 -- 172.21.15.7:0/1150182531 shutdown_connections 2023-03-03T18:06:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.371+0000 7fca84f23700 1 -- 172.21.15.7:0/1150182531 wait complete. 2023-03-03T18:06:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.372+0000 7fca84f23700 1 Processor -- start 2023-03-03T18:06:32.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.372+0000 7fca84f23700 1 -- start start 2023-03-03T18:06:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca84f23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca800c6fe0 0x7fca800c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca84f23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca80151e80 0x7fca801e09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca84f23700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca80155410 0x7fca801e0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca84f23700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca801df9b0 con 0x7fca80155410 2023-03-03T18:06:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca84f23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca801588b0 con 0x7fca80151e80 2023-03-03T18:06:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca84f23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca80158b90 con 0x7fca800c6fe0 2023-03-03T18:06:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca7ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca80155410 0x7fca801e0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca7ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca80155410 0x7fca801e0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35860/0 (socket says 172.21.15.7:35860) 2023-03-03T18:06:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca7ed9d700 1 -- 172.21.15.7:0/828331388 learned_addr learned my addr 172.21.15.7:0/828331388 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:06:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca7dd9b700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca80151e80 0x7fca801e09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca7e59c700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca800c6fe0 0x7fca800c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca7ed9d700 1 -- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca800c6fe0 msgr2=0x7fca800c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca7ed9d700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca800c6fe0 0x7fca800c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca7ed9d700 1 -- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca80151e80 msgr2=0x7fca801e09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.373+0000 7fca7ed9d700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca80151e80 0x7fca801e09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.374+0000 7fca7ed9d700 1 -- 172.21.15.7:0/828331388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700094a0 con 0x7fca80155410 2023-03-03T18:06:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.374+0000 7fca7ed9d700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca80155410 0x7fca801e0f20 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7fca7800caa0 tx=0x7fca7800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.374+0000 7fca6effd700 1 -- 172.21.15.7:0/828331388 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7800a9e0 con 0x7fca80155410 2023-03-03T18:06:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.374+0000 7fca6effd700 1 -- 172.21.15.7:0/828331388 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7800ab40 con 0x7fca80155410 2023-03-03T18:06:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.374+0000 7fca84f23700 1 -- 172.21.15.7:0/828331388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca80154ef0 con 0x7fca80155410 2023-03-03T18:06:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.374+0000 7fca6effd700 1 -- 172.21.15.7:0/828331388 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78012400 con 0x7fca80155410 2023-03-03T18:06:32.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.374+0000 7fca84f23700 1 -- 172.21.15.7:0/828331388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca80154c10 con 0x7fca80155410 2023-03-03T18:06:32.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.375+0000 7fca84f23700 1 -- 172.21.15.7:0/828331388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca800c5220 con 0x7fca80155410 2023-03-03T18:06:32.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.375+0000 7fca6effd700 1 -- 172.21.15.7:0/828331388 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fca78019030 con 0x7fca80155410 2023-03-03T18:06:32.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.376+0000 7fca6effd700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 0x7fca6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:32.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.376+0000 7fca6effd700 1 -- 172.21.15.7:0/828331388 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fca78085c00 con 0x7fca80155410 2023-03-03T18:06:32.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.376+0000 7fca7e59c700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 0x7fca6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:32.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.377+0000 7fca7e59c700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 0x7fca6006a7a0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fca70009fd0 tx=0x7fca70008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:32.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.380+0000 7fca6effd700 1 -- 172.21.15.7:0/828331388 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fca7808b020 con 0x7fca80155410 2023-03-03T18:06:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.584+0000 7fef08c81700 1 -- 172.21.15.7:0/3486349797 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef041df140 con 0x7fef04151e80 2023-03-03T18:06:32.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.587+0000 7feef2ffd700 1 -- 172.21.15.7:0/3486349797 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7feef8055e10 con 0x7fef04151e80 2023-03-03T18:06:32.587 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:06:32.587 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:06:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 -- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 msgr2=0x7feee406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 0x7feee406a7a0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7feef4000f60 tx=0x7feef4007d30 comp rx=0 tx=0).stop 2023-03-03T18:06:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 -- 172.21.15.7:0/3486349797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04151e80 msgr2=0x7fef041df420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04151e80 0x7fef041df420 secure :-1 s=READY pgs=3105 cs=0 l=1 rev1=1 crypto rx=0x7feef800e9e0 tx=0x7feef800eda0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 -- 172.21.15.7:0/3486349797 shutdown_connections 2023-03-03T18:06:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef04155410 0x7fef041df960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef040c5b00 0x7fef04150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 0x7feee406a7a0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 --2- 172.21.15.7:0/3486349797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef04151e80 0x7fef041df420 unknown :-1 s=CLOSED pgs=3105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 -- 172.21.15.7:0/3486349797 >> 172.21.15.7:0/3486349797 conn(0x7fef04071630 msgr2=0x7fef040c3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:06:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 -- 172.21.15.7:0/3486349797 shutdown_connections 2023-03-03T18:06:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.589+0000 7fef08c81700 1 -- 172.21.15.7:0/3486349797 wait complete. 2023-03-03T18:06:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:06:32.601 INFO:tasks.mds_thrash.fs.[b]:waiting for 26 secs before thrashing 2023-03-03T18:06:32.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.622+0000 7fca84f23700 1 -- 172.21.15.7:0/828331388 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca800c0af0 con 0x7fca80155410 2023-03-03T18:06:32.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.623+0000 7fca6effd700 1 -- 172.21.15.7:0/828331388 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fca78052060 con 0x7fca80155410 2023-03-03T18:06:32.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:06:32.624 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:06:32.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 -- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 msgr2=0x7fca6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 0x7fca6006a7a0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fca70009fd0 tx=0x7fca70008030 comp rx=0 tx=0).stop 2023-03-03T18:06:32.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 -- 172.21.15.7:0/828331388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca80155410 msgr2=0x7fca801e0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:32.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca80155410 0x7fca801e0f20 secure :-1 s=READY pgs=3107 cs=0 l=1 rev1=1 crypto rx=0x7fca7800caa0 tx=0x7fca7800cdb0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 -- 172.21.15.7:0/828331388 shutdown_connections 2023-03-03T18:06:32.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 0x7fca6006a7a0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca80155410 0x7fca801e0f20 unknown :-1 s=CLOSED pgs=3107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca800c6fe0 0x7fca800c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 --2- 172.21.15.7:0/828331388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca80151e80 0x7fca801e09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:32.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 -- 172.21.15.7:0/828331388 >> 172.21.15.7:0/828331388 conn(0x7fca80071630 msgr2=0x7fca800c3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:06:32.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 -- 172.21.15.7:0/828331388 shutdown_connections 2023-03-03T18:06:32.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:32.625+0000 7fca84f23700 1 -- 172.21.15.7:0/828331388 wait complete. 2023-03-03T18:06:32.627 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:06:32.637 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2023-03-03T18:06:34.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:34.235 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:06:58.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:06:58.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.741+0000 7fb15ad23700 1 Processor -- start 2023-03-03T18:06:58.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.741+0000 7fb15ad23700 1 -- start start 2023-03-03T18:06:58.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.741+0000 7fb15ad23700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb15414f9d0 0x7fb154156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.741+0000 7fb15ad23700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb1540abe20 0x7fb1541529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.741+0000 7fb15ad23700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1540c2f30 con 0x7fb15414fda0 2023-03-03T18:06:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.741+0000 7fb15ad23700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1540c31d0 con 0x7fb1540abe20 2023-03-03T18:06:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb15ad23700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1541524a0 con 0x7fb15414f9d0 2023-03-03T18:06:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb1592c0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb1540abe20 0x7fb1541529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb153fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb15414fda0 0x7fb154071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58102/0 (socket says 172.21.15.7:58102) 2023-03-03T18:06:58.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb153fff700 1 -- 172.21.15.7:0/2859224404 learned_addr learned my addr 172.21.15.7:0/2859224404 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:06:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb158abf700 1 --2- 172.21.15.7:0/2859224404 >> v2:172.21.15.143:3300/0 conn(0x7fb15414f9d0 0x7fb154156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb1592c0700 1 -- 172.21.15.7:0/2859224404 >> v2:172.21.15.143:3300/0 conn(0x7fb15414f9d0 msgr2=0x7fb154156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:06:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb1592c0700 1 --2- 172.21.15.7:0/2859224404 >> v2:172.21.15.143:3300/0 conn(0x7fb15414f9d0 0x7fb154156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb1592c0700 1 -- 172.21.15.7:0/2859224404 >> v1:172.21.15.7:6789/0 conn(0x7fb15414fda0 legacy=0x7fb154071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb1592c0700 1 -- 172.21.15.7:0/2859224404 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1540c34b0 con 0x7fb1540abe20 2023-03-03T18:06:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb1537fe700 1 -- 172.21.15.7:0/2859224404 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808097455 0 0) 0x7fb1540c2f30 con 0x7fb15414fda0 2023-03-03T18:06:58.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb158abf700 1 --2- 172.21.15.7:0/2859224404 >> v2:172.21.15.143:3300/0 conn(0x7fb15414f9d0 0x7fb154156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:06:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb1592c0700 1 --2- 172.21.15.7:0/2859224404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1540abe20 0x7fb1541529c0 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7fb1540710a0 tx=0x7fb14c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e06680a953b46f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb1537fe700 1 -- 172.21.15.7:0/2859224404 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c00e070 con 0x7fb1540abe20 2023-03-03T18:06:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb1537fe700 1 -- 172.21.15.7:0/2859224404 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14c00bbd0 con 0x7fb1540abe20 2023-03-03T18:06:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.742+0000 7fb1537fe700 1 -- 172.21.15.7:0/2859224404 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c012430 con 0x7fb1540abe20 2023-03-03T18:06:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.743+0000 7fb15ad23700 1 -- 172.21.15.7:0/2859224404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1540abe20 msgr2=0x7fb1541529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:58.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.743+0000 7fb15ad23700 1 --2- 172.21.15.7:0/2859224404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1540abe20 0x7fb1541529c0 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7fb1540710a0 tx=0x7fb14c00b330 comp rx=0 tx=0).stop 2023-03-03T18:06:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.743+0000 7fb15ad23700 1 -- 172.21.15.7:0/2859224404 shutdown_connections 2023-03-03T18:06:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.743+0000 7fb15ad23700 1 --2- 172.21.15.7:0/2859224404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1540abe20 0x7fb1541529c0 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.743+0000 7fb15ad23700 1 --2- 172.21.15.7:0/2859224404 >> v2:172.21.15.143:3300/0 conn(0x7fb15414f9d0 0x7fb154156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.743+0000 7fb15ad23700 1 -- 172.21.15.7:0/2859224404 >> 172.21.15.7:0/2859224404 conn(0x7fb1540c8900 msgr2=0x7fb1540c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:06:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.743+0000 7fb15ad23700 1 -- 172.21.15.7:0/2859224404 shutdown_connections 2023-03-03T18:06:58.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.743+0000 7fb15ad23700 1 -- 172.21.15.7:0/2859224404 wait complete. 2023-03-03T18:06:58.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.744+0000 7fb15ad23700 1 Processor -- start 2023-03-03T18:06:58.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.744+0000 7fb15ad23700 1 -- start start 2023-03-03T18:06:58.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb15ad23700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1540abe20 0x7fb1541e7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:58.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb15ad23700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb15414f9d0 0x7fb1541dc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:58.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb15ad23700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb15414fda0 0x7fb15406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:58.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb15ad23700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541deae0 con 0x7fb15414f9d0 2023-03-03T18:06:58.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb15ad23700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb154155bf0 con 0x7fb1540abe20 2023-03-03T18:06:58.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb15ad23700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541dcd70 con 0x7fb15414fda0 2023-03-03T18:06:58.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb153fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb15414f9d0 0x7fb1541dc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:58.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb153fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb15414f9d0 0x7fb1541dc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35896/0 (socket says 172.21.15.7:35896) 2023-03-03T18:06:58.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb153fff700 1 -- 172.21.15.7:0/4273278932 learned_addr learned my addr 172.21.15.7:0/4273278932 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:06:58.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb158abf700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1540abe20 0x7fb1541e7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:58.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.745+0000 7fb1592c0700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb15414fda0 0x7fb15406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:58.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb153fff700 1 -- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb15414fda0 msgr2=0x7fb15406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:58.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb153fff700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb15414fda0 0x7fb15406ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:58.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb153fff700 1 -- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1540abe20 msgr2=0x7fb1541e7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:58.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb153fff700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1540abe20 0x7fb1541e7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:58.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb153fff700 1 -- 172.21.15.7:0/4273278932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c009a70 con 0x7fb15414f9d0 2023-03-03T18:06:58.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb158abf700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1540abe20 0x7fb1541e7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:06:58.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb153fff700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb15414f9d0 0x7fb1541dc830 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7fb14000c540 tx=0x7fb14000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:58.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb1517fa700 1 -- 172.21.15.7:0/4273278932 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb140004c80 con 0x7fb15414f9d0 2023-03-03T18:06:58.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb1517fa700 1 -- 172.21.15.7:0/4273278932 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb140004de0 con 0x7fb15414f9d0 2023-03-03T18:06:58.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb15ad23700 1 -- 172.21.15.7:0/4273278932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1540ac350 con 0x7fb15414f9d0 2023-03-03T18:06:58.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb1517fa700 1 -- 172.21.15.7:0/4273278932 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb140007310 con 0x7fb15414f9d0 2023-03-03T18:06:58.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.746+0000 7fb15ad23700 1 -- 172.21.15.7:0/4273278932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1540ab630 con 0x7fb15414f9d0 2023-03-03T18:06:58.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.747+0000 7fb15ad23700 1 -- 172.21.15.7:0/4273278932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb154071fa0 con 0x7fb15414f9d0 2023-03-03T18:06:58.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.747+0000 7fb1517fa700 1 -- 172.21.15.7:0/4273278932 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb14001a020 con 0x7fb15414f9d0 2023-03-03T18:06:58.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.748+0000 7fb1517fa700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1380682f0 0x7fb13806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:06:58.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.748+0000 7fb1517fa700 1 -- 172.21.15.7:0/4273278932 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb140085d60 con 0x7fb15414f9d0 2023-03-03T18:06:58.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.748+0000 7fb158abf700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1380682f0 0x7fb13806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:06:58.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.749+0000 7fb158abf700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1380682f0 0x7fb13806a7a0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fb148006fd0 tx=0x7fb148008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:06:58.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.752+0000 7fb1517fa700 1 -- 172.21.15.7:0/4273278932 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb140052100 con 0x7fb15414f9d0 2023-03-03T18:06:58.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.989+0000 7fb15ad23700 1 -- 172.21.15.7:0/4273278932 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb154074bb0 con 0x7fb15414f9d0 2023-03-03T18:06:58.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.989+0000 7fb1517fa700 1 -- 172.21.15.7:0/4273278932 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb140055720 con 0x7fb15414f9d0 2023-03-03T18:06:58.989 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:06:58.989 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:06:58.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 -- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1380682f0 msgr2=0x7fb13806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:58.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1380682f0 0x7fb13806a7a0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fb148006fd0 tx=0x7fb148008040 comp rx=0 tx=0).stop 2023-03-03T18:06:58.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 -- 172.21.15.7:0/4273278932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb15414f9d0 msgr2=0x7fb1541dc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:06:58.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb15414f9d0 0x7fb1541dc830 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7fb14000c540 tx=0x7fb14000c900 comp rx=0 tx=0).stop 2023-03-03T18:06:58.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 -- 172.21.15.7:0/4273278932 shutdown_connections 2023-03-03T18:06:58.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb15414fda0 0x7fb15406ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:58.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1540abe20 0x7fb1541e7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:58.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb1380682f0 0x7fb13806a7a0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:58.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 --2- 172.21.15.7:0/4273278932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb15414f9d0 0x7fb1541dc830 unknown :-1 s=CLOSED pgs=3110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:06:58.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 -- 172.21.15.7:0/4273278932 >> 172.21.15.7:0/4273278932 conn(0x7fb1540c8900 msgr2=0x7fb1541528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:06:58.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 -- 172.21.15.7:0/4273278932 shutdown_connections 2023-03-03T18:06:58.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:06:58.991+0000 7fb15ad23700 1 -- 172.21.15.7:0/4273278932 wait complete. 2023-03-03T18:06:58.993 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:06:59.003 INFO:tasks.mds_thrash.fs.[b]:waiting for 89 secs before thrashing 2023-03-03T18:07:04.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:04.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:34.390 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:34.394 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:07:51.638 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:07:51.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.777+0000 7fcba0848700 1 Processor -- start 2023-03-03T18:07:51.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.777+0000 7fcba0848700 1 -- start start 2023-03-03T18:07:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcba0848700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcb9c14b0d0 0x7fcb9c071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:07:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcba0848700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 0x7fcb9c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:07:51.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcba0848700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcb9c071d40 0x7fcb9c158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:07:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcba0848700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9c0a9c20 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcba0848700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9c154f10 con 0x7fcb9c071d40 2023-03-03T18:07:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcba0848700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0ac3c0 con 0x7fcb9c0aad20 2023-03-03T18:07:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb9a59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcb9c14b0d0 0x7fcb9c071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:07:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb9a59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcb9c14b0d0 0x7fcb9c071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35938/0 (socket says 172.21.15.7:35938) 2023-03-03T18:07:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb9a59c700 1 -- 172.21.15.7:0/3399631800 learned_addr learned my addr 172.21.15.7:0/3399631800 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:07:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb99d9b700 1 --2- 172.21.15.7:0/3399631800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 0x7fcb9c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:07:51.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb9a59c700 1 -- 172.21.15.7:0/3399631800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 msgr2=0x7fcb9c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:07:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb9ad9d700 1 --2- 172.21.15.7:0/3399631800 >> v2:172.21.15.143:3300/0 conn(0x7fcb9c071d40 0x7fcb9c158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:07:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb9a59c700 1 --2- 172.21.15.7:0/3399631800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 0x7fcb9c157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb9a59c700 1 -- 172.21.15.7:0/3399631800 >> v2:172.21.15.143:3300/0 conn(0x7fcb9c071d40 msgr2=0x7fcb9c158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:07:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb9a59c700 1 --2- 172.21.15.7:0/3399631800 >> v2:172.21.15.143:3300/0 conn(0x7fcb9c071d40 0x7fcb9c158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb9a59c700 1 -- 172.21.15.7:0/3399631800 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c06fc70 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb99d9b700 1 --2- 172.21.15.7:0/3399631800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 0x7fcb9c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:07:51.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.778+0000 7fcb9ad9d700 1 --2- 172.21.15.7:0/3399631800 >> v2:172.21.15.143:3300/0 conn(0x7fcb9c071d40 0x7fcb9c158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:07:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcb9a59c700 1 --2- 172.21.15.7:0/3399631800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 0x7fcb9c071800 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0a98d0 tx=0x7fcb90009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c1ef43e97fad05 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:07:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcb9959a700 1 -- 172.21.15.7:0/3399631800 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9000b870 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcb9959a700 1 -- 172.21.15.7:0/3399631800 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9000b9d0 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcb9959a700 1 -- 172.21.15.7:0/3399631800 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9000bd60 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcba0848700 1 -- 172.21.15.7:0/3399631800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 msgr2=0x7fcb9c071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:07:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcba0848700 1 --2- 172.21.15.7:0/3399631800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 0x7fcb9c071800 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0a98d0 tx=0x7fcb90009bd0 comp rx=0 tx=0).stop 2023-03-03T18:07:51.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcba0848700 1 -- 172.21.15.7:0/3399631800 shutdown_connections 2023-03-03T18:07:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcba0848700 1 --2- 172.21.15.7:0/3399631800 >> v2:172.21.15.143:3300/0 conn(0x7fcb9c071d40 0x7fcb9c158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcba0848700 1 --2- 172.21.15.7:0/3399631800 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 0x7fcb9c071800 unknown :-1 s=CLOSED pgs=3111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcba0848700 1 --2- 172.21.15.7:0/3399631800 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 0x7fcb9c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcba0848700 1 -- 172.21.15.7:0/3399631800 >> 172.21.15.7:0/3399631800 conn(0x7fcb9c0c8900 msgr2=0x7fcb9c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:07:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.779+0000 7fcba0848700 1 -- 172.21.15.7:0/3399631800 shutdown_connections 2023-03-03T18:07:51.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.780+0000 7fcba0848700 1 -- 172.21.15.7:0/3399631800 wait complete. 2023-03-03T18:07:51.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.781+0000 7fcba0848700 1 Processor -- start 2023-03-03T18:07:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.781+0000 7fcba0848700 1 -- start start 2023-03-03T18:07:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcba0848700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c071d40 0x7fcb9c156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:07:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcba0848700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 0x7fcb9c152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:07:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcba0848700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 0x7fcb9c153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:07:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcba0848700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c1de480 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcba0848700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c1edb80 con 0x7fcb9c071d40 2023-03-03T18:07:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcba0848700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c1ede60 con 0x7fcb9c0aad20 2023-03-03T18:07:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb9ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 0x7fcb9c153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:07:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb9ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 0x7fcb9c153110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35948/0 (socket says 172.21.15.7:35948) 2023-03-03T18:07:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb9ad9d700 1 -- 172.21.15.7:0/3352349693 learned_addr learned my addr 172.21.15.7:0/3352349693 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:07:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb99d9b700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 0x7fcb9c152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:07:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb9a59c700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c071d40 0x7fcb9c156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:07:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb9ad9d700 1 -- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 msgr2=0x7fcb9c152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:07:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb9ad9d700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 0x7fcb9c152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb9ad9d700 1 -- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c071d40 msgr2=0x7fcb9c156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:07:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb9ad9d700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c071d40 0x7fcb9c156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb9ad9d700 1 -- 172.21.15.7:0/3352349693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb90009550 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb9ad9d700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 0x7fcb9c153110 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400cd10 tx=0x7fcb9400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:07:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb8affd700 1 -- 172.21.15.7:0/3352349693 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9400ce60 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcb8affd700 1 -- 172.21.15.7:0/3352349693 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9400f480 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.782+0000 7fcba0848700 1 -- 172.21.15.7:0/3352349693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9c06f6b0 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.783+0000 7fcb8affd700 1 -- 172.21.15.7:0/3352349693 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9400f810 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.783+0000 7fcba0848700 1 -- 172.21.15.7:0/3352349693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9c155160 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.784+0000 7fcba0848700 1 -- 172.21.15.7:0/3352349693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9c06f990 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.784+0000 7fcb8affd700 1 -- 172.21.15.7:0/3352349693 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcb94009030 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.784+0000 7fcb8affd700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb7c0682f0 0x7fcb7c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:07:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.784+0000 7fcb8affd700 1 -- 172.21.15.7:0/3352349693 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcb94086910 con 0x7fcb9c14b0d0 2023-03-03T18:07:51.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.785+0000 7fcb9a59c700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb7c0682f0 0x7fcb7c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:07:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.785+0000 7fcb9a59c700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb7c0682f0 0x7fcb7c06a7a0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fcb90000c00 tx=0x7fcb90008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:07:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:51.789+0000 7fcb8affd700 1 -- 172.21.15.7:0/3352349693 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcb94052cb0 con 0x7fcb9c14b0d0 2023-03-03T18:07:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.032+0000 7fcba0848700 1 -- 172.21.15.7:0/3352349693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb9c074bb0 con 0x7fcb9c14b0d0 2023-03-03T18:07:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.033+0000 7fcb8affd700 1 -- 172.21.15.7:0/3352349693 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcb940562d0 con 0x7fcb9c14b0d0 2023-03-03T18:07:52.033 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:07:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:07:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.034+0000 7fcba0848700 1 -- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb7c0682f0 msgr2=0x7fcb7c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:07:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.034+0000 7fcba0848700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb7c0682f0 0x7fcb7c06a7a0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fcb90000c00 tx=0x7fcb90008030 comp rx=0 tx=0).stop 2023-03-03T18:07:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.035+0000 7fcba0848700 1 -- 172.21.15.7:0/3352349693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 msgr2=0x7fcb9c153110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:07:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.035+0000 7fcba0848700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 0x7fcb9c153110 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400cd10 tx=0x7fcb9400a560 comp rx=0 tx=0).stop 2023-03-03T18:07:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.035+0000 7fcba0848700 1 -- 172.21.15.7:0/3352349693 shutdown_connections 2023-03-03T18:07:52.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.035+0000 7fcba0848700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcb7c0682f0 0x7fcb7c06a7a0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.035+0000 7fcba0848700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb9c14b0d0 0x7fcb9c153110 unknown :-1 s=CLOSED pgs=3112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.035+0000 7fcba0848700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb9c071d40 0x7fcb9c156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.035+0000 7fcba0848700 1 --2- 172.21.15.7:0/3352349693 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcb9c0aad20 0x7fcb9c152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:07:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.035+0000 7fcba0848700 1 -- 172.21.15.7:0/3352349693 >> 172.21.15.7:0/3352349693 conn(0x7fcb9c0c8900 msgr2=0x7fcb9c0c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:07:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.035+0000 7fcba0848700 1 -- 172.21.15.7:0/3352349693 shutdown_connections 2023-03-03T18:07:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:07:52.035+0000 7fcba0848700 1 -- 172.21.15.7:0/3352349693 wait complete. 2023-03-03T18:07:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:07:52.047 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2023-03-03T18:08:04.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:08:04.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:08:28.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:08:28.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.144+0000 7f68b0de6700 1 Processor -- start 2023-03-03T18:08:28.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.144+0000 7f68b0de6700 1 -- start start 2023-03-03T18:08:28.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68b0de6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68ac14b020 0x7f68ac0717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:08:28.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68b0de6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f68ac0aad20 0x7f68ac157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:08:28.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68b0de6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68ac071d20 0x7f68ac1581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:08:28.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68b0de6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ac0728b0 con 0x7f68ac071d20 2023-03-03T18:08:28.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68b0de6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ac155070 con 0x7f68ac0aad20 2023-03-03T18:08:28.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68b0de6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac06f5b0 con 0x7f68ac14b020 2023-03-03T18:08:28.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68aad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68ac071d20 0x7f68ac1581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:08:28.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68aad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68ac071d20 0x7f68ac1581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36002/0 (socket says 172.21.15.7:36002) 2023-03-03T18:08:28.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68aad9d700 1 -- 172.21.15.7:0/2017472529 learned_addr learned my addr 172.21.15.7:0/2017472529 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:08:28.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68a9d9b700 1 --2- 172.21.15.7:0/2017472529 >> v2:172.21.15.143:3300/0 conn(0x7f68ac0aad20 0x7f68ac157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:08:28.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68aa59c700 1 --2- 172.21.15.7:0/2017472529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68ac14b020 0x7f68ac0717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:08:28.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68aad9d700 1 -- 172.21.15.7:0/2017472529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68ac14b020 msgr2=0x7f68ac0717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:08:28.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68aad9d700 1 --2- 172.21.15.7:0/2017472529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68ac14b020 0x7f68ac0717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68aad9d700 1 -- 172.21.15.7:0/2017472529 >> v2:172.21.15.143:3300/0 conn(0x7f68ac0aad20 msgr2=0x7f68ac157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:08:28.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68aad9d700 1 --2- 172.21.15.7:0/2017472529 >> v2:172.21.15.143:3300/0 conn(0x7f68ac0aad20 0x7f68ac157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.145+0000 7f68aad9d700 1 -- 172.21.15.7:0/2017472529 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac06f890 con 0x7f68ac071d20 2023-03-03T18:08:28.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68aad9d700 1 --2- 172.21.15.7:0/2017472529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 0x7f68ac1581e0 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0c28b0 tx=0x7f68a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa514619d2600bc3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:08:28.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68a959a700 1 -- 172.21.15.7:0/2017472529 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a400e070 con 0x7f68ac071d20 2023-03-03T18:08:28.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68a959a700 1 -- 172.21.15.7:0/2017472529 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a400b910 con 0x7f68ac071d20 2023-03-03T18:08:28.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68a959a700 1 -- 172.21.15.7:0/2017472529 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a400bcc0 con 0x7f68ac071d20 2023-03-03T18:08:28.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68b0de6700 1 -- 172.21.15.7:0/2017472529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 msgr2=0x7f68ac1581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:08:28.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68b0de6700 1 --2- 172.21.15.7:0/2017472529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 0x7f68ac1581e0 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0c28b0 tx=0x7f68a400b330 comp rx=0 tx=0).stop 2023-03-03T18:08:28.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68b0de6700 1 -- 172.21.15.7:0/2017472529 shutdown_connections 2023-03-03T18:08:28.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68b0de6700 1 --2- 172.21.15.7:0/2017472529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 0x7f68ac1581e0 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68b0de6700 1 --2- 172.21.15.7:0/2017472529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68ac14b020 0x7f68ac0717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68b0de6700 1 --2- 172.21.15.7:0/2017472529 >> v2:172.21.15.143:3300/0 conn(0x7f68ac0aad20 0x7f68ac157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68b0de6700 1 -- 172.21.15.7:0/2017472529 >> 172.21.15.7:0/2017472529 conn(0x7f68ac0c8900 msgr2=0x7f68ac0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:08:28.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68b0de6700 1 -- 172.21.15.7:0/2017472529 shutdown_connections 2023-03-03T18:08:28.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.146+0000 7f68b0de6700 1 -- 172.21.15.7:0/2017472529 wait complete. 2023-03-03T18:08:28.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.148+0000 7f68b0de6700 1 Processor -- start 2023-03-03T18:08:28.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.148+0000 7f68b0de6700 1 -- start start 2023-03-03T18:08:28.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.148+0000 7f68b0de6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 0x7f68ac1e1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:08:28.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.148+0000 7f68b0de6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68ac0aad20 0x7f68ac14c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:08:28.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.148+0000 7f68b0de6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68ac14b020 0x7f68ac14c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:08:28.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.148+0000 7f68b0de6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1dfd30 con 0x7f68ac071d20 2023-03-03T18:08:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.148+0000 7f68b0de6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0ad240 con 0x7f68ac0aad20 2023-03-03T18:08:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.148+0000 7f68b0de6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac153020 con 0x7f68ac14b020 2023-03-03T18:08:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68aa59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 0x7f68ac1e1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:08:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68aa59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 0x7f68ac1e1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36004/0 (socket says 172.21.15.7:36004) 2023-03-03T18:08:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68aa59c700 1 -- 172.21.15.7:0/3829957359 learned_addr learned my addr 172.21.15.7:0/3829957359 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:08:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68a9d9b700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68ac0aad20 0x7f68ac14c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:08:28.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68aa59c700 1 -- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68ac14b020 msgr2=0x7f68ac14c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:08:28.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68aad9d700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68ac14b020 0x7f68ac14c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:08:28.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68aa59c700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68ac14b020 0x7f68ac14c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68aa59c700 1 -- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68ac0aad20 msgr2=0x7f68ac14c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:08:28.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68aa59c700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68ac0aad20 0x7f68ac14c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68aa59c700 1 -- 172.21.15.7:0/3829957359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a4009b30 con 0x7f68ac071d20 2023-03-03T18:08:28.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68a9d9b700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68ac0aad20 0x7f68ac14c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:08:28.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68aa59c700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 0x7f68ac1e1110 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7f689c00e9b0 tx=0x7f689c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:08:28.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f689affd700 1 -- 172.21.15.7:0/3829957359 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c009710 con 0x7f68ac071d20 2023-03-03T18:08:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f689affd700 1 -- 172.21.15.7:0/3829957359 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c00c850 con 0x7f68ac071d20 2023-03-03T18:08:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68b0de6700 1 -- 172.21.15.7:0/3829957359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac155350 con 0x7f68ac071d20 2023-03-03T18:08:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f689affd700 1 -- 172.21.15.7:0/3829957359 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c00cbc0 con 0x7f68ac071d20 2023-03-03T18:08:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.149+0000 7f68b0de6700 1 -- 172.21.15.7:0/3829957359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac06fb70 con 0x7f68ac071d20 2023-03-03T18:08:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.150+0000 7f68b0de6700 1 -- 172.21.15.7:0/3829957359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ac158720 con 0x7f68ac071d20 2023-03-03T18:08:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.151+0000 7f689affd700 1 -- 172.21.15.7:0/3829957359 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f689c019030 con 0x7f68ac071d20 2023-03-03T18:08:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.151+0000 7f689affd700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f688c0682f0 0x7f688c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:08:28.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.151+0000 7f689affd700 1 -- 172.21.15.7:0/3829957359 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f689c086170 con 0x7f68ac071d20 2023-03-03T18:08:28.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.151+0000 7f68a9d9b700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f688c0682f0 0x7f688c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:08:28.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.152+0000 7f68a9d9b700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f688c0682f0 0x7f688c06a7a0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f68ac14ac30 tx=0x7f68a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:08:28.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.155+0000 7f689affd700 1 -- 172.21.15.7:0/3829957359 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f689c052510 con 0x7f68ac071d20 2023-03-03T18:08:28.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.390+0000 7f68b0de6700 1 -- 172.21.15.7:0/3829957359 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68ac14b3f0 con 0x7f68ac071d20 2023-03-03T18:08:28.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.391+0000 7f689affd700 1 -- 172.21.15.7:0/3829957359 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f68ac14b3f0 con 0x7f68ac071d20 2023-03-03T18:08:28.392 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:08:28.392 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:08:28.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 -- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f688c0682f0 msgr2=0x7f688c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:08:28.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f688c0682f0 0x7f688c06a7a0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f68ac14ac30 tx=0x7f68a0008040 comp rx=0 tx=0).stop 2023-03-03T18:08:28.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 -- 172.21.15.7:0/3829957359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 msgr2=0x7f68ac1e1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:08:28.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 0x7f68ac1e1110 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7f689c00e9b0 tx=0x7f689c00ed70 comp rx=0 tx=0).stop 2023-03-03T18:08:28.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 -- 172.21.15.7:0/3829957359 shutdown_connections 2023-03-03T18:08:28.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68ac14b020 0x7f68ac14c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f688c0682f0 0x7f688c06a7a0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68ac071d20 0x7f68ac1e1110 unknown :-1 s=CLOSED pgs=3115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 --2- 172.21.15.7:0/3829957359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68ac0aad20 0x7f68ac14c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:08:28.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 -- 172.21.15.7:0/3829957359 >> 172.21.15.7:0/3829957359 conn(0x7f68ac0c8900 msgr2=0x7f68ac0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:08:28.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 -- 172.21.15.7:0/3829957359 shutdown_connections 2023-03-03T18:08:28.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:08:28.393+0000 7f68b0de6700 1 -- 172.21.15.7:0/3829957359 wait complete. 2023-03-03T18:08:28.396 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:08:28.406 INFO:tasks.mds_thrash.fs.[b]:waiting for 53 secs before thrashing 2023-03-03T18:08:34.536 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:08:34.539 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:09:04.611 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:09:04.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:09:11.048 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:09:11.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.187+0000 7fc3ee87f700 1 Processor -- start 2023-03-03T18:09:11.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.188+0000 7fc3ee87f700 1 -- start start 2023-03-03T18:09:11.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.188+0000 7fc3ee87f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e8071e80 0x7fc3e8151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:11.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.188+0000 7fc3ee87f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e81510d0 con 0x7fc3e814e900 2023-03-03T18:09:11.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.188+0000 7fc3ee87f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80c3740 con 0x7fc3e8072250 2023-03-03T18:09:11.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.188+0000 7fc3ee87f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8070670 con 0x7fc3e8071e80 2023-03-03T18:09:11.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.188+0000 7fc3ece1c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc3e814e900 0x7fc3e815ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58252/0 (socket says 172.21.15.7:58252) 2023-03-03T18:09:11.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.188+0000 7fc3ece1c700 1 -- 172.21.15.7:0/82382895 learned_addr learned my addr 172.21.15.7:0/82382895 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:09:11.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963573764 0 0) 0x7fc3e81510d0 con 0x7fc3e814e900 2023-03-03T18:09:11.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e7fff700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e8071e80 0x7fc3e8151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:11.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d40035b0 con 0x7fc3e814e900 2023-03-03T18:09:11.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1130521368 0 0) 0x7fc3d40035b0 con 0x7fc3e814e900 2023-03-03T18:09:11.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e8071e80 msgr2=0x7fc3e8151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:09:11.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e8071e80 0x7fc3e8151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:11.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 >> v1:172.21.15.143:6789/0 conn(0x7fc3e8072250 legacy=0x7fc3e814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:11.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80c39e0 con 0x7fc3e814e900 2023-03-03T18:09:11.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc3e00019f0 con 0x7fc3e814e900 2023-03-03T18:09:11.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 >> v1:172.21.15.7:6789/0 conn(0x7fc3e814e900 legacy=0x7fc3e815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:11.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3d4003aa0 0x7fc3d4003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:11.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e8071e80 0x7fc3d4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:11.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d4004d90 0x7fc3d4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:11.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8070670 con 0x7fc3d4004d90 2023-03-03T18:09:11.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d400bba0 con 0x7fc3d4003aa0 2023-03-03T18:09:11.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d400bd60 con 0x7fc3e8071e80 2023-03-03T18:09:11.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3ece1c700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d4004d90 0x7fc3d4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:11.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e77fe700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e8071e80 0x7fc3d4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:11.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.189+0000 7fc3e7fff700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3d4003aa0 0x7fc3d4003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ece1c700 1 -- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e8071e80 msgr2=0x7fc3d4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:09:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ece1c700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3e8071e80 0x7fc3d4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ece1c700 1 -- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3d4003aa0 msgr2=0x7fc3d4003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:09:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ece1c700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3d4003aa0 0x7fc3d4003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ece1c700 1 -- 172.21.15.7:0/82382895 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d40035b0 con 0x7fc3d4004d90 2023-03-03T18:09:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ece1c700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d4004d90 0x7fc3d4005200 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000a830 tx=0x7fc3d4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae14f4578b1a278c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:09:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e000a2f0 con 0x7fc3d4004d90 2023-03-03T18:09:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e0012600 con 0x7fc3d4004d90 2023-03-03T18:09:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3e6ffd700 1 -- 172.21.15.7:0/82382895 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00129b0 con 0x7fc3d4004d90 2023-03-03T18:09:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ee87f700 1 -- 172.21.15.7:0/82382895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d4004d90 msgr2=0x7fc3d4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ee87f700 1 --2- 172.21.15.7:0/82382895 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3d4004d90 0x7fc3d4005200 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000a830 tx=0x7fc3d4005880 comp rx=0 tx=0).stop 2023-03-03T18:09:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ece1c700 1 -- 172.21.15.7:0/82382895 reap_dead start 2023-03-03T18:09:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ee87f700 1 -- 172.21.15.7:0/82382895 shutdown_connections 2023-03-03T18:09:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.190+0000 7fc3ee87f700 1 -- 172.21.15.7:0/82382895 >> 172.21.15.7:0/82382895 conn(0x7fc3e80a94d0 msgr2=0x7fc3e80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:09:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.191+0000 7fc3ee87f700 1 -- 172.21.15.7:0/82382895 shutdown_connections 2023-03-03T18:09:11.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.191+0000 7fc3ee87f700 1 -- 172.21.15.7:0/82382895 wait complete. 2023-03-03T18:09:11.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.192+0000 7fc3ee87f700 1 Processor -- start 2023-03-03T18:09:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.192+0000 7fc3ee87f700 1 -- start start 2023-03-03T18:09:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ee87f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3d4004d90 0x7fc3e81eb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ee87f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e81ebc00 0x7fc3e80ab230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ee87f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e8144740 0x7fc3e80ab770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ee87f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80c5c80 con 0x7fc3e8144740 2023-03-03T18:09:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ee87f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80aae10 con 0x7fc3e81ebc00 2023-03-03T18:09:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ee87f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e81ec390 con 0x7fc3d4004d90 2023-03-03T18:09:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ece1c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e8144740 0x7fc3e80ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ece1c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e8144740 0x7fc3e80ab770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36052/0 (socket says 172.21.15.7:36052) 2023-03-03T18:09:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ece1c700 1 -- 172.21.15.7:0/2403864627 learned_addr learned my addr 172.21.15.7:0/2403864627 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:09:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3e77fe700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e81ebc00 0x7fc3e80ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3e7fff700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3d4004d90 0x7fc3e81eb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ece1c700 1 -- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3d4004d90 msgr2=0x7fc3e81eb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ece1c700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3d4004d90 0x7fc3e81eb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ece1c700 1 -- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e81ebc00 msgr2=0x7fc3e80ab230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ece1c700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e81ebc00 0x7fc3e80ab230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ece1c700 1 -- 172.21.15.7:0/2403864627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e806c9a0 con 0x7fc3e8144740 2023-03-03T18:09:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3e77fe700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e81ebc00 0x7fc3e80ab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:09:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ece1c700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e8144740 0x7fc3e80ab770 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000a800 tx=0x7fc3e0001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:09:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3e4ff9700 1 -- 172.21.15.7:0/2403864627 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00011c0 con 0x7fc3e8144740 2023-03-03T18:09:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3e4ff9700 1 -- 172.21.15.7:0/2403864627 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e00017b0 con 0x7fc3e8144740 2023-03-03T18:09:11.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ee87f700 1 -- 172.21.15.7:0/2403864627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e8151300 con 0x7fc3e8144740 2023-03-03T18:09:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3e4ff9700 1 -- 172.21.15.7:0/2403864627 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e001e430 con 0x7fc3e8144740 2023-03-03T18:09:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.193+0000 7fc3ee87f700 1 -- 172.21.15.7:0/2403864627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e806cb30 con 0x7fc3e8144740 2023-03-03T18:09:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.194+0000 7fc3ee87f700 1 -- 172.21.15.7:0/2403864627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e80aaf50 con 0x7fc3e8144740 2023-03-03T18:09:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.195+0000 7fc3e4ff9700 1 -- 172.21.15.7:0/2403864627 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc3e0024030 con 0x7fc3e8144740 2023-03-03T18:09:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.195+0000 7fc3e4ff9700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 0x7fc3c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.195+0000 7fc3e4ff9700 1 -- 172.21.15.7:0/2403864627 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc3e0088710 con 0x7fc3e8144740 2023-03-03T18:09:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.195+0000 7fc3e7fff700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 0x7fc3c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.196+0000 7fc3e7fff700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 0x7fc3c806a7a0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009c60 tx=0x7fc3d8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:09:11.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.200+0000 7fc3e4ff9700 1 -- 172.21.15.7:0/2403864627 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc3e80aaf50 con 0x7fc3e8144740 2023-03-03T18:09:11.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.435+0000 7fc3ee87f700 1 -- 172.21.15.7:0/2403864627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3e80aaf50 con 0x7fc3e8144740 2023-03-03T18:09:11.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.436+0000 7fc3e4ff9700 1 -- 172.21.15.7:0/2403864627 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc3e002c090 con 0x7fc3e8144740 2023-03-03T18:09:11.435 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:09:11.436 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:09:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.437+0000 7fc3ee87f700 1 -- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 msgr2=0x7fc3c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.437+0000 7fc3ee87f700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 0x7fc3c806a7a0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009c60 tx=0x7fc3d8009210 comp rx=0 tx=0).stop 2023-03-03T18:09:11.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.437+0000 7fc3ee87f700 1 -- 172.21.15.7:0/2403864627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e8144740 msgr2=0x7fc3e80ab770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.437+0000 7fc3ee87f700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e8144740 0x7fc3e80ab770 secure :-1 s=READY pgs=3118 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000a800 tx=0x7fc3e0001d80 comp rx=0 tx=0).stop 2023-03-03T18:09:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.437+0000 7fc3ee87f700 1 -- 172.21.15.7:0/2403864627 shutdown_connections 2023-03-03T18:09:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.437+0000 7fc3ee87f700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3c80682f0 0x7fc3c806a7a0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.437+0000 7fc3ee87f700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e8144740 0x7fc3e80ab770 unknown :-1 s=CLOSED pgs=3118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.437+0000 7fc3ee87f700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3d4004d90 0x7fc3e81eb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.437+0000 7fc3ee87f700 1 --2- 172.21.15.7:0/2403864627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3e81ebc00 0x7fc3e80ab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:11.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.437+0000 7fc3ee87f700 1 -- 172.21.15.7:0/2403864627 >> 172.21.15.7:0/2403864627 conn(0x7fc3e80a94d0 msgr2=0x7fc3e8151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:09:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.438+0000 7fc3ee87f700 1 -- 172.21.15.7:0/2403864627 shutdown_connections 2023-03-03T18:09:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:11.438+0000 7fc3ee87f700 1 -- 172.21.15.7:0/2403864627 wait complete. 2023-03-03T18:09:11.439 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:09:11.449 INFO:tasks.mds_thrash.fs.[a]:waiting for 71 secs before thrashing 2023-03-03T18:09:21.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:09:21.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.546+0000 7f9e6d45c700 1 Processor -- start 2023-03-03T18:09:21.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.546+0000 7f9e6d45c700 1 -- start start 2023-03-03T18:09:21.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.547+0000 7f9e6d45c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9e680c5b00 0x7f9e680bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.547+0000 7f9e6d45c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e680c08d0 con 0x7f9e68155410 2023-03-03T18:09:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.547+0000 7f9e6d45c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e68154930 con 0x7f9e68151e80 2023-03-03T18:09:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.547+0000 7f9e6d45c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e68152250 con 0x7f9e680c5b00 2023-03-03T18:09:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.547+0000 7f9e677fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9e680c5b00 0x7f9e680bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.547+0000 7f9e667fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9e68155410 0x7f9e680bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58276/0 (socket says 172.21.15.7:58276) 2023-03-03T18:09:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.547+0000 7f9e667fc700 1 -- 172.21.15.7:0/32349473 learned_addr learned my addr 172.21.15.7:0/32349473 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:09:21.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.547+0000 7f9e65ffb700 1 -- 172.21.15.7:0/32349473 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993377403 0 0) 0x7f9e680c08d0 con 0x7f9e68155410 2023-03-03T18:09:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.547+0000 7f9e65ffb700 1 -- 172.21.15.7:0/32349473 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e500035b0 con 0x7f9e68155410 2023-03-03T18:09:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e677fe700 1 -- 172.21.15.7:0/32349473 >> v1:172.21.15.143:6789/0 conn(0x7f9e68151e80 legacy=0x7f9e680c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:09:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e677fe700 1 -- 172.21.15.7:0/32349473 >> v1:172.21.15.7:6789/0 conn(0x7f9e68155410 legacy=0x7f9e680bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e677fe700 1 -- 172.21.15.7:0/32349473 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e68154c10 con 0x7f9e680c5b00 2023-03-03T18:09:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e677fe700 1 --2- 172.21.15.7:0/32349473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e680c5b00 0x7f9e680bfff0 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009d90 tx=0x7f9e6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30c5ba0d4794883 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:09:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e65ffb700 1 -- 172.21.15.7:0/32349473 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6000e070 con 0x7f9e680c5b00 2023-03-03T18:09:21.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e65ffb700 1 -- 172.21.15.7:0/32349473 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6000bae0 con 0x7f9e680c5b00 2023-03-03T18:09:21.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e65ffb700 1 -- 172.21.15.7:0/32349473 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6000be90 con 0x7f9e680c5b00 2023-03-03T18:09:21.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e6d45c700 1 -- 172.21.15.7:0/32349473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e680c5b00 msgr2=0x7f9e680bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:21.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e6d45c700 1 --2- 172.21.15.7:0/32349473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e680c5b00 0x7f9e680bfff0 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009d90 tx=0x7f9e6000b330 comp rx=0 tx=0).stop 2023-03-03T18:09:21.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e6d45c700 1 -- 172.21.15.7:0/32349473 shutdown_connections 2023-03-03T18:09:21.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e6d45c700 1 --2- 172.21.15.7:0/32349473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e680c5b00 0x7f9e680bfff0 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:21.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.548+0000 7f9e6d45c700 1 -- 172.21.15.7:0/32349473 >> 172.21.15.7:0/32349473 conn(0x7f9e68071630 msgr2=0x7f9e6806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:09:21.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.549+0000 7f9e6d45c700 1 -- 172.21.15.7:0/32349473 shutdown_connections 2023-03-03T18:09:21.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.549+0000 7f9e6d45c700 1 -- 172.21.15.7:0/32349473 wait complete. 2023-03-03T18:09:21.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.550+0000 7f9e6d45c700 1 Processor -- start 2023-03-03T18:09:21.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.550+0000 7f9e6d45c700 1 -- start start 2023-03-03T18:09:21.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.550+0000 7f9e6d45c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e680c5b00 0x7f9e681e1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:21.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e6d45c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e68151e80 0x7f9e680ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:21.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e6d45c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e68155410 0x7f9e68149390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:21.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e6d45c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e681e1770 con 0x7f9e68151e80 2023-03-03T18:09:21.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e6d45c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e68158130 con 0x7f9e68155410 2023-03-03T18:09:21.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e6d45c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e68157090 con 0x7f9e680c5b00 2023-03-03T18:09:21.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e667fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e68151e80 0x7f9e680ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:21.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e667fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e68151e80 0x7f9e680ca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36070/0 (socket says 172.21.15.7:36070) 2023-03-03T18:09:21.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e667fc700 1 -- 172.21.15.7:0/3928139136 learned_addr learned my addr 172.21.15.7:0/3928139136 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:09:21.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e677fe700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e68155410 0x7f9e68149390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:21.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e66ffd700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e680c5b00 0x7f9e681e1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:21.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e667fc700 1 -- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e680c5b00 msgr2=0x7f9e681e1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:21.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e667fc700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e680c5b00 0x7f9e681e1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:21.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e667fc700 1 -- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e68155410 msgr2=0x7f9e68149390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:21.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e667fc700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e68155410 0x7f9e68149390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:21.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e667fc700 1 -- 172.21.15.7:0/3928139136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e60009a70 con 0x7f9e68151e80 2023-03-03T18:09:21.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e667fc700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e68151e80 0x7f9e680ca220 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800c500 tx=0x7f9e5800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:09:21.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e4f7fe700 1 -- 172.21.15.7:0/3928139136 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58005e30 con 0x7f9e68151e80 2023-03-03T18:09:21.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e4f7fe700 1 -- 172.21.15.7:0/3928139136 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e580070d0 con 0x7f9e68151e80 2023-03-03T18:09:21.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e4f7fe700 1 -- 172.21.15.7:0/3928139136 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58007480 con 0x7f9e68151e80 2023-03-03T18:09:21.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e6d45c700 1 -- 172.21.15.7:0/3928139136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e680c5220 con 0x7f9e68151e80 2023-03-03T18:09:21.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.551+0000 7f9e6d45c700 1 -- 172.21.15.7:0/3928139136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e68154ef0 con 0x7f9e68151e80 2023-03-03T18:09:21.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.553+0000 7f9e6d45c700 1 -- 172.21.15.7:0/3928139136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e680c0530 con 0x7f9e68151e80 2023-03-03T18:09:21.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.553+0000 7f9e4f7fe700 1 -- 172.21.15.7:0/3928139136 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9e58019030 con 0x7f9e68151e80 2023-03-03T18:09:21.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.553+0000 7f9e4f7fe700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e540682f0 0x7f9e5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:09:21.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.553+0000 7f9e4f7fe700 1 -- 172.21.15.7:0/3928139136 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9e58085520 con 0x7f9e68151e80 2023-03-03T18:09:21.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.553+0000 7f9e66ffd700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e540682f0 0x7f9e5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:09:21.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.554+0000 7f9e66ffd700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e540682f0 0x7f9e5406a7a0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00a7d0 tx=0x7f9e5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:09:21.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.557+0000 7f9e4f7fe700 1 -- 172.21.15.7:0/3928139136 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9e580518c0 con 0x7f9e68151e80 2023-03-03T18:09:21.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.794+0000 7f9e6d45c700 1 -- 172.21.15.7:0/3928139136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e68074bb0 con 0x7f9e68151e80 2023-03-03T18:09:21.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.794+0000 7f9e4f7fe700 1 -- 172.21.15.7:0/3928139136 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9e58054ee0 con 0x7f9e68151e80 2023-03-03T18:09:21.798 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:09:21.799 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:09:21.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 -- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e540682f0 msgr2=0x7f9e5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:21.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e540682f0 0x7f9e5406a7a0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00a7d0 tx=0x7f9e5c008040 comp rx=0 tx=0).stop 2023-03-03T18:09:21.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 -- 172.21.15.7:0/3928139136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e68151e80 msgr2=0x7f9e680ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:09:21.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e68151e80 0x7f9e680ca220 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800c500 tx=0x7f9e5800c8c0 comp rx=0 tx=0).stop 2023-03-03T18:09:21.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 -- 172.21.15.7:0/3928139136 shutdown_connections 2023-03-03T18:09:21.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e68155410 0x7f9e68149390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:21.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e680c5b00 0x7f9e681e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:21.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e540682f0 0x7f9e5406a7a0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:21.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 --2- 172.21.15.7:0/3928139136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e68151e80 0x7f9e680ca220 unknown :-1 s=CLOSED pgs=3121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:09:21.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 -- 172.21.15.7:0/3928139136 >> 172.21.15.7:0/3928139136 conn(0x7f9e68071630 msgr2=0x7f9e68153f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:09:21.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 -- 172.21.15.7:0/3928139136 shutdown_connections 2023-03-03T18:09:21.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:09:21.796+0000 7f9e6d45c700 1 -- 172.21.15.7:0/3928139136 wait complete. 2023-03-03T18:09:21.803 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:09:21.808 INFO:tasks.mds_thrash.fs.[b]:waiting for 96 secs before thrashing 2023-03-03T18:09:34.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:09:34.744 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:09:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T18:09:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T18:10:04.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:10:04.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:10:22.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:10:22.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.589+0000 7ff925acb700 1 Processor -- start 2023-03-03T18:10:22.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.589+0000 7ff925acb700 1 -- start start 2023-03-03T18:10:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff925acb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff920151d40 0x7ff9200736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff925acb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92006d430 con 0x7ff9200c2930 2023-03-03T18:10:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff925acb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92006cb00 con 0x7ff920073c30 2023-03-03T18:10:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff925acb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92006c2a0 con 0x7ff920151d40 2023-03-03T18:10:22.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff91effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff9200c2930 0x7ff920157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58336/0 (socket says 172.21.15.7:58336) 2023-03-03T18:10:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff91effd700 1 -- 172.21.15.7:0/1317501893 learned_addr learned my addr 172.21.15.7:0/1317501893 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:10:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff91f7fe700 1 --2- 172.21.15.7:0/1317501893 >> v2:172.21.15.143:3300/0 conn(0x7ff920151d40 0x7ff9200736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:10:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff91e7fc700 1 -- 172.21.15.7:0/1317501893 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758659841 0 0) 0x7ff92006d430 con 0x7ff9200c2930 2023-03-03T18:10:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff91e7fc700 1 -- 172.21.15.7:0/1317501893 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff90c0035b0 con 0x7ff9200c2930 2023-03-03T18:10:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff91f7fe700 1 -- 172.21.15.7:0/1317501893 >> v1:172.21.15.143:6789/0 conn(0x7ff920073c30 legacy=0x7ff9201581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff91f7fe700 1 -- 172.21.15.7:0/1317501893 >> v1:172.21.15.7:6789/0 conn(0x7ff9200c2930 legacy=0x7ff920157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:22.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.590+0000 7ff91f7fe700 1 -- 172.21.15.7:0/1317501893 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92006cde0 con 0x7ff920151d40 2023-03-03T18:10:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff91f7fe700 1 --2- 172.21.15.7:0/1317501893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff920151d40 0x7ff9200736f0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff910009870 tx=0x7ff910009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bed11fc62bbc6c9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:10:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff91e7fc700 1 -- 172.21.15.7:0/1317501893 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff910011830 con 0x7ff920151d40 2023-03-03T18:10:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff91e7fc700 1 -- 172.21.15.7:0/1317501893 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff910011990 con 0x7ff920151d40 2023-03-03T18:10:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff91e7fc700 1 -- 172.21.15.7:0/1317501893 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff910011d20 con 0x7ff920151d40 2023-03-03T18:10:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff925acb700 1 -- 172.21.15.7:0/1317501893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff920151d40 msgr2=0x7ff9200736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff925acb700 1 --2- 172.21.15.7:0/1317501893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff920151d40 0x7ff9200736f0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff910009870 tx=0x7ff910009b80 comp rx=0 tx=0).stop 2023-03-03T18:10:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff925acb700 1 -- 172.21.15.7:0/1317501893 shutdown_connections 2023-03-03T18:10:22.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff925acb700 1 --2- 172.21.15.7:0/1317501893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff920151d40 0x7ff9200736f0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:22.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff925acb700 1 -- 172.21.15.7:0/1317501893 >> 172.21.15.7:0/1317501893 conn(0x7ff9200a94d0 msgr2=0x7ff9200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:10:22.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff925acb700 1 -- 172.21.15.7:0/1317501893 shutdown_connections 2023-03-03T18:10:22.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.591+0000 7ff925acb700 1 -- 172.21.15.7:0/1317501893 wait complete. 2023-03-03T18:10:22.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.592+0000 7ff925acb700 1 Processor -- start 2023-03-03T18:10:22.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.593+0000 7ff925acb700 1 -- start start 2023-03-03T18:10:22.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff925acb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff920073c30 0x7ff9200ab2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:22.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff925acb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9200c2930 0x7ff9201ea840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:22.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff925acb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff920151d40 0x7ff9200c0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:22.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff925acb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9201ead80 con 0x7ff920073c30 2023-03-03T18:10:22.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff925acb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200ac6a0 con 0x7ff9200c2930 2023-03-03T18:10:22.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff925acb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200ad150 con 0x7ff920151d40 2023-03-03T18:10:22.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff91f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff920073c30 0x7ff9200ab2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:10:22.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff91f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff920073c30 0x7ff9200ab2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36128/0 (socket says 172.21.15.7:36128) 2023-03-03T18:10:22.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff91f7fe700 1 -- 172.21.15.7:0/3810401969 learned_addr learned my addr 172.21.15.7:0/3810401969 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:10:22.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff91f7fe700 1 -- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff920151d40 msgr2=0x7ff9200c0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:22.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff91f7fe700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff920151d40 0x7ff9200c0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:22.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff91f7fe700 1 -- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9200c2930 msgr2=0x7ff9201ea840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:22.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff91f7fe700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9200c2930 0x7ff9201ea840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:22.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff91f7fe700 1 -- 172.21.15.7:0/3810401969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9100094a0 con 0x7ff920073c30 2023-03-03T18:10:22.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff91f7fe700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff920073c30 0x7ff9200ab2c0 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 crypto rx=0x7ff910000c00 tx=0x7ff910004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:10:22.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff917fff700 1 -- 172.21.15.7:0/3810401969 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9100036a0 con 0x7ff920073c30 2023-03-03T18:10:22.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff917fff700 1 -- 172.21.15.7:0/3810401969 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91001e070 con 0x7ff920073c30 2023-03-03T18:10:22.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff917fff700 1 -- 172.21.15.7:0/3810401969 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff910003a20 con 0x7ff920073c30 2023-03-03T18:10:22.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.594+0000 7ff925acb700 1 -- 172.21.15.7:0/3810401969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92006bea0 con 0x7ff920073c30 2023-03-03T18:10:22.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.595+0000 7ff925acb700 1 -- 172.21.15.7:0/3810401969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92006d0c0 con 0x7ff920073c30 2023-03-03T18:10:22.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.595+0000 7ff925acb700 1 -- 172.21.15.7:0/3810401969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92006c030 con 0x7ff920073c30 2023-03-03T18:10:22.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.596+0000 7ff917fff700 1 -- 172.21.15.7:0/3810401969 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff91000f050 con 0x7ff920073c30 2023-03-03T18:10:22.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.596+0000 7ff917fff700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 0x7ff90006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:22.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.596+0000 7ff917fff700 1 -- 172.21.15.7:0/3810401969 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff910020070 con 0x7ff920073c30 2023-03-03T18:10:22.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.597+0000 7ff91effd700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 0x7ff90006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:10:22.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.597+0000 7ff91effd700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 0x7ff90006a7a0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7ff9200bdf50 tx=0x7ff9080079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:10:22.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.601+0000 7ff917fff700 1 -- 172.21.15.7:0/3810401969 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff910053930 con 0x7ff920073c30 2023-03-03T18:10:22.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.840+0000 7ff925acb700 1 -- 172.21.15.7:0/3810401969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff920074bb0 con 0x7ff920073c30 2023-03-03T18:10:22.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.840+0000 7ff917fff700 1 -- 172.21.15.7:0/3810401969 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff910015090 con 0x7ff920073c30 2023-03-03T18:10:22.840 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:10:22.841 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:10:22.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 -- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 msgr2=0x7ff90006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:22.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 0x7ff90006a7a0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7ff9200bdf50 tx=0x7ff9080079c0 comp rx=0 tx=0).stop 2023-03-03T18:10:22.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 -- 172.21.15.7:0/3810401969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff920073c30 msgr2=0x7ff9200ab2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:22.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff920073c30 0x7ff9200ab2c0 secure :-1 s=READY pgs=3124 cs=0 l=1 rev1=1 crypto rx=0x7ff910000c00 tx=0x7ff910004750 comp rx=0 tx=0).stop 2023-03-03T18:10:22.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 -- 172.21.15.7:0/3810401969 shutdown_connections 2023-03-03T18:10:22.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff920151d40 0x7ff9200c0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:22.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 0x7ff90006a7a0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:22.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff920073c30 0x7ff9200ab2c0 unknown :-1 s=CLOSED pgs=3124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:22.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 --2- 172.21.15.7:0/3810401969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9200c2930 0x7ff9201ea840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:22.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 -- 172.21.15.7:0/3810401969 >> 172.21.15.7:0/3810401969 conn(0x7ff9200a94d0 msgr2=0x7ff92014f560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:10:22.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 -- 172.21.15.7:0/3810401969 shutdown_connections 2023-03-03T18:10:22.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:22.843+0000 7ff925acb700 1 -- 172.21.15.7:0/3810401969 wait complete. 2023-03-03T18:10:22.844 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:10:22.855 INFO:tasks.mds_thrash.fs.[a]:waiting for 108 secs before thrashing 2023-03-03T18:10:34.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:10:34.979 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:10:57.810 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:10:57.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.949+0000 7f2ebfdf3700 1 Processor -- start 2023-03-03T18:10:57.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.949+0000 7f2ebfdf3700 1 -- start start 2023-03-03T18:10:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebfdf3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2eb814f9d0 0x7f2eb8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebfdf3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2eb80abe20 0x7f2eb81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebfdf3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb80c2f30 con 0x7f2eb814fda0 2023-03-03T18:10:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebfdf3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb80c31d0 con 0x7f2eb80abe20 2023-03-03T18:10:57.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebfdf3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb81524a0 con 0x7f2eb814f9d0 2023-03-03T18:10:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebe390700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2eb80abe20 0x7f2eb81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:10:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebe390700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2eb80abe20 0x7f2eb81529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36166/0 (socket says 172.21.15.7:36166) 2023-03-03T18:10:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebe390700 1 -- 172.21.15.7:0/416452276 learned_addr learned my addr 172.21.15.7:0/416452276 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:10:57.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebdb8f700 1 --2- 172.21.15.7:0/416452276 >> v2:172.21.15.143:3300/0 conn(0x7f2eb814f9d0 0x7f2eb8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:10:57.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebe390700 1 -- 172.21.15.7:0/416452276 >> v2:172.21.15.143:3300/0 conn(0x7f2eb814f9d0 msgr2=0x7f2eb8156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:10:57.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebe390700 1 --2- 172.21.15.7:0/416452276 >> v2:172.21.15.143:3300/0 conn(0x7f2eb814f9d0 0x7f2eb8156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:57.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebe390700 1 -- 172.21.15.7:0/416452276 >> v1:172.21.15.143:6789/0 conn(0x7f2eb814fda0 legacy=0x7f2eb8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:10:57.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebe390700 1 -- 172.21.15.7:0/416452276 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb80c34b0 con 0x7f2eb80abe20 2023-03-03T18:10:57.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.950+0000 7f2ebe390700 1 --2- 172.21.15.7:0/416452276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb80abe20 0x7f2eb81529c0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80710a0 tx=0x7f2eb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2699fdcbf5636bdd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:10:57.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebcb8d700 1 -- 172.21.15.7:0/416452276 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000e070 con 0x7f2eb80abe20 2023-03-03T18:10:57.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebcb8d700 1 -- 172.21.15.7:0/416452276 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb0003a20 con 0x7f2eb80abe20 2023-03-03T18:10:57.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebcb8d700 1 -- 172.21.15.7:0/416452276 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0003dd0 con 0x7f2eb80abe20 2023-03-03T18:10:57.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/416452276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb80abe20 msgr2=0x7f2eb81529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:57.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebfdf3700 1 --2- 172.21.15.7:0/416452276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb80abe20 0x7f2eb81529c0 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80710a0 tx=0x7f2eb000b330 comp rx=0 tx=0).stop 2023-03-03T18:10:57.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/416452276 shutdown_connections 2023-03-03T18:10:57.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebfdf3700 1 --2- 172.21.15.7:0/416452276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb80abe20 0x7f2eb81529c0 unknown :-1 s=CLOSED pgs=3125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:57.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebfdf3700 1 --2- 172.21.15.7:0/416452276 >> v2:172.21.15.143:3300/0 conn(0x7f2eb814f9d0 0x7f2eb8156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:57.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/416452276 >> 172.21.15.7:0/416452276 conn(0x7f2eb80c8900 msgr2=0x7f2eb80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:10:57.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/416452276 shutdown_connections 2023-03-03T18:10:57.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.951+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/416452276 wait complete. 2023-03-03T18:10:57.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.952+0000 7f2ebfdf3700 1 Processor -- start 2023-03-03T18:10:57.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.952+0000 7f2ebfdf3700 1 -- start start 2023-03-03T18:10:57.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.953+0000 7f2ebfdf3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80abe20 0x7f2eb80c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:57.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.953+0000 7f2ebfdf3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb814f9d0 0x7f2eb81ddaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:57.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.953+0000 7f2ebfdf3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eb814fda0 0x7f2eb814ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:57.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.953+0000 7f2ebfdf3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81dd680 con 0x7f2eb814f9d0 2023-03-03T18:10:57.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.953+0000 7f2ebfdf3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81dc100 con 0x7f2eb80abe20 2023-03-03T18:10:57.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.953+0000 7f2ebfdf3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81ecc00 con 0x7f2eb814fda0 2023-03-03T18:10:57.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.953+0000 7f2ebd38e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb814f9d0 0x7f2eb81ddaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:10:57.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.953+0000 7f2ebd38e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb814f9d0 0x7f2eb81ddaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36170/0 (socket says 172.21.15.7:36170) 2023-03-03T18:10:57.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.953+0000 7f2ebd38e700 1 -- 172.21.15.7:0/3499929521 learned_addr learned my addr 172.21.15.7:0/3499929521 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:10:57.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.953+0000 7f2ebdb8f700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80abe20 0x7f2eb80c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:10:57.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2ebe390700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eb814fda0 0x7f2eb814ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:10:57.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2ebd38e700 1 -- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eb814fda0 msgr2=0x7f2eb814ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:57.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2ebd38e700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eb814fda0 0x7f2eb814ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:57.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2ebd38e700 1 -- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80abe20 msgr2=0x7f2eb80c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:57.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2ebd38e700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80abe20 0x7f2eb80c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:57.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2ebd38e700 1 -- 172.21.15.7:0/3499929521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb0009b30 con 0x7f2eb814f9d0 2023-03-03T18:10:57.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2ebd38e700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb814f9d0 0x7f2eb81ddaa0 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800c4a0 tx=0x7f2ea800c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:10:57.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2eae7fc700 1 -- 172.21.15.7:0/3499929521 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8011790 con 0x7f2eb814f9d0 2023-03-03T18:10:57.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2eae7fc700 1 -- 172.21.15.7:0/3499929521 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea80118f0 con 0x7f2eb814f9d0 2023-03-03T18:10:57.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/3499929521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb80ac350 con 0x7f2eb814f9d0 2023-03-03T18:10:57.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2eae7fc700 1 -- 172.21.15.7:0/3499929521 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8011ca0 con 0x7f2eb814f9d0 2023-03-03T18:10:57.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.954+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/3499929521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb80ab630 con 0x7f2eb814f9d0 2023-03-03T18:10:57.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.955+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/3499929521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb8071fa0 con 0x7f2eb814f9d0 2023-03-03T18:10:57.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.955+0000 7f2eae7fc700 1 -- 172.21.15.7:0/3499929521 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2ea8010050 con 0x7f2eb814f9d0 2023-03-03T18:10:57.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.956+0000 7f2eae7fc700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e9c0682f0 0x7f2e9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:10:57.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.956+0000 7f2eae7fc700 1 -- 172.21.15.7:0/3499929521 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2ea8085dd0 con 0x7f2eb814f9d0 2023-03-03T18:10:57.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.956+0000 7f2ebdb8f700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e9c0682f0 0x7f2e9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:10:57.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.957+0000 7f2ebdb8f700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e9c0682f0 0x7f2e9c06a7a0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4006b20 tx=0x7f2eb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:10:57.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:57.960+0000 7f2eae7fc700 1 -- 172.21.15.7:0/3499929521 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2ea8052170 con 0x7f2eb814f9d0 2023-03-03T18:10:58.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.200+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/3499929521 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2eb81eca60 con 0x7f2eb814f9d0 2023-03-03T18:10:58.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.201+0000 7f2eae7fc700 1 -- 172.21.15.7:0/3499929521 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2eb81eca60 con 0x7f2eb814f9d0 2023-03-03T18:10:58.201 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:10:58.201 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:10:58.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e9c0682f0 msgr2=0x7f2e9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:58.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e9c0682f0 0x7f2e9c06a7a0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4006b20 tx=0x7f2eb4008040 comp rx=0 tx=0).stop 2023-03-03T18:10:58.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/3499929521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb814f9d0 msgr2=0x7f2eb81ddaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:10:58.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb814f9d0 0x7f2eb81ddaa0 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800c4a0 tx=0x7f2ea800c7b0 comp rx=0 tx=0).stop 2023-03-03T18:10:58.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/3499929521 shutdown_connections 2023-03-03T18:10:58.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2eb814fda0 0x7f2eb814ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:58.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2eb80abe20 0x7f2eb80c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:58.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e9c0682f0 0x7f2e9c06a7a0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:58.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 --2- 172.21.15.7:0/3499929521 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2eb814f9d0 0x7f2eb81ddaa0 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:10:58.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/3499929521 >> 172.21.15.7:0/3499929521 conn(0x7f2eb80c8900 msgr2=0x7f2eb80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:10:58.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/3499929521 shutdown_connections 2023-03-03T18:10:58.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:10:58.203+0000 7f2ebfdf3700 1 -- 172.21.15.7:0/3499929521 wait complete. 2023-03-03T18:10:58.205 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:10:58.214 INFO:tasks.mds_thrash.fs.[b]:waiting for 94 secs before thrashing 2023-03-03T18:11:05.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:11:05.057 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:11:35.118 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:11:35.121 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:12:05.173 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:12:05.176 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:12:10.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:12:10.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.995+0000 7ff059814700 1 Processor -- start 2023-03-03T18:12:10.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.995+0000 7ff059814700 1 -- start start 2023-03-03T18:12:10.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff059814700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0540caa30 0x7ff0540c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:10.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff059814700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0540c00e0 0x7ff054154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:10.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff059814700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0540cb280 con 0x7ff0540cae00 2023-03-03T18:12:10.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff059814700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0540c7310 con 0x7ff0540c00e0 2023-03-03T18:12:10.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff059814700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0540bfd10 con 0x7ff0540caa30 2023-03-03T18:12:10.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff0527fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0540c00e0 0x7ff054154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:10.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff0537fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff0540cae00 0x7ff0540c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58430/0 (socket says 172.21.15.7:58430) 2023-03-03T18:12:10.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff0537fe700 1 -- 172.21.15.7:0/1694918362 learned_addr learned my addr 172.21.15.7:0/1694918362 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:12:10.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff0527fc700 1 -- 172.21.15.7:0/1694918362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0540caa30 msgr2=0x7ff0540c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:12:10.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff0527fc700 1 --2- 172.21.15.7:0/1694918362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0540caa30 0x7ff0540c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:10.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff0527fc700 1 -- 172.21.15.7:0/1694918362 >> v1:172.21.15.7:6789/0 conn(0x7ff0540cae00 legacy=0x7ff0540c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:10.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff051ffb700 1 -- 172.21.15.7:0/1694918362 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029670746 0 0) 0x7ff0540cb280 con 0x7ff0540cae00 2023-03-03T18:12:10.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff0527fc700 1 -- 172.21.15.7:0/1694918362 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0540bfa30 con 0x7ff0540c00e0 2023-03-03T18:12:10.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.996+0000 7ff0527fc700 1 --2- 172.21.15.7:0/1694918362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 0x7ff054154ff0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7ff044009820 tx=0x7ff044009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4dcf14e375c6770 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:12:10.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff051ffb700 1 -- 172.21.15.7:0/1694918362 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff044005e30 con 0x7ff0540c00e0 2023-03-03T18:12:10.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff051ffb700 1 -- 172.21.15.7:0/1694918362 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff044006030 con 0x7ff0540c00e0 2023-03-03T18:12:10.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff051ffb700 1 -- 172.21.15.7:0/1694918362 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0440063e0 con 0x7ff0540c00e0 2023-03-03T18:12:10.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff059814700 1 -- 172.21.15.7:0/1694918362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 msgr2=0x7ff054154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:10.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff059814700 1 --2- 172.21.15.7:0/1694918362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 0x7ff054154ff0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7ff044009820 tx=0x7ff044009b30 comp rx=0 tx=0).stop 2023-03-03T18:12:10.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff059814700 1 -- 172.21.15.7:0/1694918362 shutdown_connections 2023-03-03T18:12:10.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff059814700 1 --2- 172.21.15.7:0/1694918362 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0540caa30 0x7ff0540c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:10.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff059814700 1 --2- 172.21.15.7:0/1694918362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 0x7ff054154ff0 unknown :-1 s=CLOSED pgs=3128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:11.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff059814700 1 -- 172.21.15.7:0/1694918362 >> 172.21.15.7:0/1694918362 conn(0x7ff054066e40 msgr2=0x7ff054071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:12:11.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff059814700 1 -- 172.21.15.7:0/1694918362 shutdown_connections 2023-03-03T18:12:11.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.997+0000 7ff059814700 1 -- 172.21.15.7:0/1694918362 wait complete. 2023-03-03T18:12:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.998+0000 7ff059814700 1 Processor -- start 2023-03-03T18:12:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff059814700 1 -- start start 2023-03-03T18:12:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff059814700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 0x7ff0540c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff059814700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0540caa30 0x7ff0540c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff059814700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0540cae00 0x7ff054156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff059814700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0541eaba0 con 0x7ff0540c00e0 2023-03-03T18:12:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff059814700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff054156820 con 0x7ff0540caa30 2023-03-03T18:12:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff059814700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff054156b00 con 0x7ff0540cae00 2023-03-03T18:12:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff052ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 0x7ff0540c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff052ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 0x7ff0540c26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36222/0 (socket says 172.21.15.7:36222) 2023-03-03T18:12:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff052ffd700 1 -- 172.21.15.7:0/3129387781 learned_addr learned my addr 172.21.15.7:0/3129387781 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:12:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:10.999+0000 7ff0537fe700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0540cae00 0x7ff054156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:11.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff0527fc700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0540caa30 0x7ff0540c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:11.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff052ffd700 1 -- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0540cae00 msgr2=0x7ff054156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:11.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff052ffd700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0540cae00 0x7ff054156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:11.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff052ffd700 1 -- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0540caa30 msgr2=0x7ff0540c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:11.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff052ffd700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0540caa30 0x7ff0540c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:11.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff052ffd700 1 -- 172.21.15.7:0/3129387781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff044009430 con 0x7ff0540c00e0 2023-03-03T18:12:11.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff052ffd700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 0x7ff0540c26d0 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7ff04800b560 tx=0x7ff04800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:12:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff03bfff700 1 -- 172.21.15.7:0/3129387781 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048011830 con 0x7ff0540c00e0 2023-03-03T18:12:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff03bfff700 1 -- 172.21.15.7:0/3129387781 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff048011990 con 0x7ff0540c00e0 2023-03-03T18:12:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff059814700 1 -- 172.21.15.7:0/3129387781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0541547f0 con 0x7ff0540c00e0 2023-03-03T18:12:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff03bfff700 1 -- 172.21.15.7:0/3129387781 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048011d40 con 0x7ff0540c00e0 2023-03-03T18:12:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.000+0000 7ff059814700 1 -- 172.21.15.7:0/3129387781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff054154510 con 0x7ff0540c00e0 2023-03-03T18:12:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.001+0000 7ff059814700 1 -- 172.21.15.7:0/3129387781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff054154ad0 con 0x7ff0540c00e0 2023-03-03T18:12:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.002+0000 7ff03bfff700 1 -- 172.21.15.7:0/3129387781 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff048021070 con 0x7ff0540c00e0 2023-03-03T18:12:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.002+0000 7ff03bfff700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0400682f0 0x7ff04006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.002+0000 7ff03bfff700 1 -- 172.21.15.7:0/3129387781 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff048085f40 con 0x7ff0540c00e0 2023-03-03T18:12:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.002+0000 7ff0527fc700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0400682f0 0x7ff04006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:11.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.003+0000 7ff0527fc700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0400682f0 0x7ff04006a7a0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7ff044013040 tx=0x7ff044008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:12:11.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.007+0000 7ff03bfff700 1 -- 172.21.15.7:0/3129387781 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff048052280 con 0x7ff0540c00e0 2023-03-03T18:12:11.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.244+0000 7ff059814700 1 -- 172.21.15.7:0/3129387781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0540c2c10 con 0x7ff0540c00e0 2023-03-03T18:12:11.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.244+0000 7ff03bfff700 1 -- 172.21.15.7:0/3129387781 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff048015090 con 0x7ff0540c00e0 2023-03-03T18:12:11.244 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:12:11.245 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:12:11.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 -- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0400682f0 msgr2=0x7ff04006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:11.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0400682f0 0x7ff04006a7a0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7ff044013040 tx=0x7ff044008070 comp rx=0 tx=0).stop 2023-03-03T18:12:11.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 -- 172.21.15.7:0/3129387781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 msgr2=0x7ff0540c26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:11.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 0x7ff0540c26d0 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7ff04800b560 tx=0x7ff04800b870 comp rx=0 tx=0).stop 2023-03-03T18:12:11.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 -- 172.21.15.7:0/3129387781 shutdown_connections 2023-03-03T18:12:11.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0540cae00 0x7ff054156f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:11.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0400682f0 0x7ff04006a7a0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:11.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0540c00e0 0x7ff0540c26d0 unknown :-1 s=CLOSED pgs=3129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:11.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 --2- 172.21.15.7:0/3129387781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0540caa30 0x7ff0540c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:11.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 -- 172.21.15.7:0/3129387781 >> 172.21.15.7:0/3129387781 conn(0x7ff054066e40 msgr2=0x7ff05406b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:12:11.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 -- 172.21.15.7:0/3129387781 shutdown_connections 2023-03-03T18:12:11.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:11.246+0000 7ff059814700 1 -- 172.21.15.7:0/3129387781 wait complete. 2023-03-03T18:12:11.248 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:12:11.258 INFO:tasks.mds_thrash.fs.[a]:waiting for 44 secs before thrashing 2023-03-03T18:12:32.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:12:32.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.351+0000 7f8c705ce700 1 Processor -- start 2023-03-03T18:12:32.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.351+0000 7f8c705ce700 1 -- start start 2023-03-03T18:12:32.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.352+0000 7f8c705ce700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8c680c17f0 0x7f8c6806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:32.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.352+0000 7f8c705ce700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6814ebe0 con 0x7f8c680c6010 2023-03-03T18:12:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.352+0000 7f8c705ce700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c680c2e70 con 0x7f8c680c5190 2023-03-03T18:12:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.352+0000 7f8c705ce700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c68152100 con 0x7f8c680c17f0 2023-03-03T18:12:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.352+0000 7f8c6db69700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8c680c17f0 0x7f8c6806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.352+0000 7f8c6e36a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8c680c6010 0x7f8c680c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58466/0 (socket says 172.21.15.7:58466) 2023-03-03T18:12:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.352+0000 7f8c6e36a700 1 -- 172.21.15.7:0/905161670 learned_addr learned my addr 172.21.15.7:0/905161670 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:12:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.352+0000 7f8c6db69700 1 -- 172.21.15.7:0/905161670 >> v1:172.21.15.143:6789/0 conn(0x7f8c680c5190 legacy=0x7f8c6806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:12:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.352+0000 7f8c6db69700 1 -- 172.21.15.7:0/905161670 >> v1:172.21.15.7:6789/0 conn(0x7f8c680c6010 legacy=0x7f8c680c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.352+0000 7f8c6db69700 1 -- 172.21.15.7:0/905161670 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c680aa680 con 0x7f8c680c17f0 2023-03-03T18:12:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c6db69700 1 --2- 172.21.15.7:0/905161670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c17f0 0x7f8c6806d6a0 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009840 tx=0x7f8c64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ec63d06b6a500d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:12:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c6d368700 1 -- 172.21.15.7:0/905161670 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64004230 con 0x7f8c680c17f0 2023-03-03T18:12:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c6d368700 1 -- 172.21.15.7:0/905161670 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c64004390 con 0x7f8c680c17f0 2023-03-03T18:12:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c6d368700 1 -- 172.21.15.7:0/905161670 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640036a0 con 0x7f8c680c17f0 2023-03-03T18:12:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c705ce700 1 -- 172.21.15.7:0/905161670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c17f0 msgr2=0x7f8c6806d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c705ce700 1 --2- 172.21.15.7:0/905161670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c17f0 0x7f8c6806d6a0 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009840 tx=0x7f8c64009b50 comp rx=0 tx=0).stop 2023-03-03T18:12:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c705ce700 1 -- 172.21.15.7:0/905161670 shutdown_connections 2023-03-03T18:12:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c705ce700 1 --2- 172.21.15.7:0/905161670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c17f0 0x7f8c6806d6a0 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c705ce700 1 -- 172.21.15.7:0/905161670 >> 172.21.15.7:0/905161670 conn(0x7f8c680a9430 msgr2=0x7f8c680c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:12:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c705ce700 1 -- 172.21.15.7:0/905161670 shutdown_connections 2023-03-03T18:12:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.353+0000 7f8c705ce700 1 -- 172.21.15.7:0/905161670 wait complete. 2023-03-03T18:12:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c705ce700 1 Processor -- start 2023-03-03T18:12:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c705ce700 1 -- start start 2023-03-03T18:12:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c705ce700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c680c17f0 0x7f8c681ded90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c705ce700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c680c5190 0x7f8c681ed3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c705ce700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c6010 0x7f8c681e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c705ce700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c680a73b0 con 0x7f8c680c6010 2023-03-03T18:12:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c705ce700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c68152dd0 con 0x7f8c680c17f0 2023-03-03T18:12:32.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c705ce700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c68153070 con 0x7f8c680c5190 2023-03-03T18:12:32.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c6eb6b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c6010 0x7f8c681e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:32.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c6eb6b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c6010 0x7f8c681e9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36264/0 (socket says 172.21.15.7:36264) 2023-03-03T18:12:32.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.355+0000 7f8c6eb6b700 1 -- 172.21.15.7:0/1643495889 learned_addr learned my addr 172.21.15.7:0/1643495889 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:12:32.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c6e36a700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c680c17f0 0x7f8c681ded90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:32.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c6db69700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c680c5190 0x7f8c681ed3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:32.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c6eb6b700 1 -- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c680c5190 msgr2=0x7f8c681ed3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:32.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c6eb6b700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c680c5190 0x7f8c681ed3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:32.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c6eb6b700 1 -- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c680c17f0 msgr2=0x7f8c681ded90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:32.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c6eb6b700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c680c17f0 0x7f8c681ded90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:32.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c6eb6b700 1 -- 172.21.15.7:0/1643495889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c640094a0 con 0x7f8c680c6010 2023-03-03T18:12:32.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c6eb6b700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c6010 0x7f8c681e9dc0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000cc50 tx=0x7f8c6000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:12:32.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c5effd700 1 -- 172.21.15.7:0/1643495889 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6000ad80 con 0x7f8c680c6010 2023-03-03T18:12:32.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c5effd700 1 -- 172.21.15.7:0/1643495889 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6000ce50 con 0x7f8c680c6010 2023-03-03T18:12:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c5effd700 1 -- 172.21.15.7:0/1643495889 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c60003c40 con 0x7f8c680c6010 2023-03-03T18:12:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c705ce700 1 -- 172.21.15.7:0/1643495889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c680c2b00 con 0x7f8c680c6010 2023-03-03T18:12:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.356+0000 7f8c705ce700 1 -- 172.21.15.7:0/1643495889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c680aa960 con 0x7f8c680c6010 2023-03-03T18:12:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.357+0000 7f8c705ce700 1 -- 172.21.15.7:0/1643495889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c680c32e0 con 0x7f8c680c6010 2023-03-03T18:12:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.357+0000 7f8c5effd700 1 -- 172.21.15.7:0/1643495889 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c60010050 con 0x7f8c680c6010 2023-03-03T18:12:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.358+0000 7f8c5effd700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c4c0682f0 0x7f8c4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.358+0000 7f8c5effd700 1 -- 172.21.15.7:0/1643495889 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8c60086160 con 0x7f8c680c6010 2023-03-03T18:12:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.358+0000 7f8c6e36a700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c4c0682f0 0x7f8c4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:32.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.358+0000 7f8c6e36a700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c4c0682f0 0x7f8c4c06a7a0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f8c58000f60 tx=0x7f8c58007d90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:12:32.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.362+0000 7f8c5effd700 1 -- 172.21.15.7:0/1643495889 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c60052480 con 0x7f8c680c6010 2023-03-03T18:12:32.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.601+0000 7f8c705ce700 1 -- 172.21.15.7:0/1643495889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c68074bb0 con 0x7f8c680c6010 2023-03-03T18:12:32.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.601+0000 7f8c5effd700 1 -- 172.21.15.7:0/1643495889 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8c60055aa0 con 0x7f8c680c6010 2023-03-03T18:12:32.601 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:12:32.601 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:12:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 -- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c4c0682f0 msgr2=0x7f8c4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c4c0682f0 0x7f8c4c06a7a0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f8c58000f60 tx=0x7f8c58007d90 comp rx=0 tx=0).stop 2023-03-03T18:12:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 -- 172.21.15.7:0/1643495889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c6010 msgr2=0x7f8c681e9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c6010 0x7f8c681e9dc0 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000cc50 tx=0x7f8c6000a340 comp rx=0 tx=0).stop 2023-03-03T18:12:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 -- 172.21.15.7:0/1643495889 shutdown_connections 2023-03-03T18:12:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c4c0682f0 0x7f8c4c06a7a0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c680c6010 0x7f8c681e9dc0 unknown :-1 s=CLOSED pgs=3132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c680c17f0 0x7f8c681ded90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 --2- 172.21.15.7:0/1643495889 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c680c5190 0x7f8c681ed3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 -- 172.21.15.7:0/1643495889 >> 172.21.15.7:0/1643495889 conn(0x7f8c680a9430 msgr2=0x7f8c680a6740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:12:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 -- 172.21.15.7:0/1643495889 shutdown_connections 2023-03-03T18:12:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:32.603+0000 7f8c705ce700 1 -- 172.21.15.7:0/1643495889 wait complete. 2023-03-03T18:12:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:12:32.615 INFO:tasks.mds_thrash.fs.[b]:waiting for 50 secs before thrashing 2023-03-03T18:12:35.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:12:35.240 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:12:55.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:12:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.398+0000 7f32e5053700 1 Processor -- start 2023-03-03T18:12:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.398+0000 7f32e5053700 1 -- start start 2023-03-03T18:12:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.399+0000 7f32e5053700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f32e0071e80 0x7f32e0151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.399+0000 7f32e5053700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f32e0072250 0x7f32e014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.399+0000 7f32e5053700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e01510d0 con 0x7f32e00c1ce0 2023-03-03T18:12:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.399+0000 7f32e5053700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e00c3740 con 0x7f32e0072250 2023-03-03T18:12:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.399+0000 7f32e5053700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e0070670 con 0x7f32e0071e80 2023-03-03T18:12:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.399+0000 7f32de59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f32e0072250 0x7f32e014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.399+0000 7f32de59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f32e0072250 0x7f32e014e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36276/0 (socket says 172.21.15.7:36276) 2023-03-03T18:12:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.399+0000 7f32de59c700 1 -- 172.21.15.7:0/3593110827 learned_addr learned my addr 172.21.15.7:0/3593110827 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:12:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.400+0000 7f32ded9d700 1 --2- 172.21.15.7:0/3593110827 >> v2:172.21.15.143:3300/0 conn(0x7f32e0071e80 0x7f32e0151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.400+0000 7f32de59c700 1 -- 172.21.15.7:0/3593110827 >> v2:172.21.15.143:3300/0 conn(0x7f32e0071e80 msgr2=0x7f32e0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:12:55.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.400+0000 7f32de59c700 1 --2- 172.21.15.7:0/3593110827 >> v2:172.21.15.143:3300/0 conn(0x7f32e0071e80 0x7f32e0151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:55.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.400+0000 7f32de59c700 1 -- 172.21.15.7:0/3593110827 >> v1:172.21.15.143:6789/0 conn(0x7f32e00c1ce0 legacy=0x7f32e01546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:55.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.400+0000 7f32de59c700 1 -- 172.21.15.7:0/3593110827 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00c39e0 con 0x7f32e0072250 2023-03-03T18:12:55.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.400+0000 7f32de59c700 1 --2- 172.21.15.7:0/3593110827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0072250 0x7f32e014e1f0 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7f32d400bb90 tx=0x7f32d400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e1109ca0924b89 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:12:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.400+0000 7f32ddd9b700 1 -- 172.21.15.7:0/3593110827 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d400dd10 con 0x7f32e0072250 2023-03-03T18:12:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.400+0000 7f32ddd9b700 1 -- 172.21.15.7:0/3593110827 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d400de70 con 0x7f32e0072250 2023-03-03T18:12:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.400+0000 7f32ddd9b700 1 -- 172.21.15.7:0/3593110827 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d4010670 con 0x7f32e0072250 2023-03-03T18:12:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.401+0000 7f32e5053700 1 -- 172.21.15.7:0/3593110827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0072250 msgr2=0x7f32e014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:55.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.401+0000 7f32e5053700 1 --2- 172.21.15.7:0/3593110827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0072250 0x7f32e014e1f0 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7f32d400bb90 tx=0x7f32d400bea0 comp rx=0 tx=0).stop 2023-03-03T18:12:55.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.401+0000 7f32e5053700 1 -- 172.21.15.7:0/3593110827 shutdown_connections 2023-03-03T18:12:55.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.401+0000 7f32e5053700 1 --2- 172.21.15.7:0/3593110827 >> v2:172.21.15.143:3300/0 conn(0x7f32e0071e80 0x7f32e0151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:55.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.401+0000 7f32e5053700 1 --2- 172.21.15.7:0/3593110827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0072250 0x7f32e014e1f0 unknown :-1 s=CLOSED pgs=3133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:55.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.401+0000 7f32e5053700 1 -- 172.21.15.7:0/3593110827 >> 172.21.15.7:0/3593110827 conn(0x7f32e00a94d0 msgr2=0x7f32e00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:12:55.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.401+0000 7f32e5053700 1 -- 172.21.15.7:0/3593110827 shutdown_connections 2023-03-03T18:12:55.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.401+0000 7f32e5053700 1 -- 172.21.15.7:0/3593110827 wait complete. 2023-03-03T18:12:55.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.402+0000 7f32e5053700 1 Processor -- start 2023-03-03T18:12:55.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.402+0000 7f32e5053700 1 -- start start 2023-03-03T18:12:55.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32e5053700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0071e80 0x7f32e014cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:55.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32e5053700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32e0072250 0x7f32e0149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:55.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32e5053700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e00c1ce0 0x7f32e0149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:55.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32e5053700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00ba390 con 0x7f32e0071e80 2023-03-03T18:12:55.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32e5053700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e01db460 con 0x7f32e00c1ce0 2023-03-03T18:12:55.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32e5053700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00ba8c0 con 0x7f32e0072250 2023-03-03T18:12:55.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32ded9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0071e80 0x7f32e014cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:55.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32ded9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0071e80 0x7f32e014cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36280/0 (socket says 172.21.15.7:36280) 2023-03-03T18:12:55.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32ded9d700 1 -- 172.21.15.7:0/1683020443 learned_addr learned my addr 172.21.15.7:0/1683020443 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:12:55.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32de59c700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32e0072250 0x7f32e0149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:55.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.403+0000 7f32df59e700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e00c1ce0 0x7f32e0149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:55.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32ded9d700 1 -- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32e0072250 msgr2=0x7f32e0149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:55.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32ded9d700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32e0072250 0x7f32e0149930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:55.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32ded9d700 1 -- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e00c1ce0 msgr2=0x7f32e0149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:55.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32ded9d700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e00c1ce0 0x7f32e0149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:55.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32ded9d700 1 -- 172.21.15.7:0/1683020443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d400b7f0 con 0x7f32e0071e80 2023-03-03T18:12:55.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32ded9d700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0071e80 0x7f32e014cb70 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f32d000e810 tx=0x7f32d000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:12:55.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32c77fe700 1 -- 172.21.15.7:0/1683020443 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d000c070 con 0x7f32e0071e80 2023-03-03T18:12:55.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32c77fe700 1 -- 172.21.15.7:0/1683020443 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d0013610 con 0x7f32e0071e80 2023-03-03T18:12:55.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32e5053700 1 -- 172.21.15.7:0/1683020443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e006c9a0 con 0x7f32e0071e80 2023-03-03T18:12:55.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32c77fe700 1 -- 172.21.15.7:0/1683020443 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00139c0 con 0x7f32e0071e80 2023-03-03T18:12:55.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.404+0000 7f32e5053700 1 -- 172.21.15.7:0/1683020443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e0151300 con 0x7f32e0071e80 2023-03-03T18:12:55.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.405+0000 7f32e5053700 1 -- 172.21.15.7:0/1683020443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e006cb30 con 0x7f32e0071e80 2023-03-03T18:12:55.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.406+0000 7f32c77fe700 1 -- 172.21.15.7:0/1683020443 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f32d0023070 con 0x7f32e0071e80 2023-03-03T18:12:55.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.406+0000 7f32c77fe700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32cc0682f0 0x7f32cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:12:55.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.406+0000 7f32c77fe700 1 -- 172.21.15.7:0/1683020443 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f32d0085eb0 con 0x7f32e0071e80 2023-03-03T18:12:55.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.406+0000 7f32de59c700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32cc0682f0 0x7f32cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:12:55.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.407+0000 7f32de59c700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32cc0682f0 0x7f32cc06a7a0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f32e00c1780 tx=0x7f32d401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:12:55.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.410+0000 7f32c77fe700 1 -- 172.21.15.7:0/1683020443 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f32d00521d0 con 0x7f32e0071e80 2023-03-03T18:12:55.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.651+0000 7f32e5053700 1 -- 172.21.15.7:0/1683020443 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32e01db7f0 con 0x7f32e0071e80 2023-03-03T18:12:55.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.651+0000 7f32c77fe700 1 -- 172.21.15.7:0/1683020443 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f32d000f040 con 0x7f32e0071e80 2023-03-03T18:12:55.652 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:12:55.652 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:12:55.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 -- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32cc0682f0 msgr2=0x7f32cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:55.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32cc0682f0 0x7f32cc06a7a0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f32e00c1780 tx=0x7f32d401c040 comp rx=0 tx=0).stop 2023-03-03T18:12:55.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 -- 172.21.15.7:0/1683020443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0071e80 msgr2=0x7f32e014cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:12:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0071e80 0x7f32e014cb70 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7f32d000e810 tx=0x7f32d000eb20 comp rx=0 tx=0).stop 2023-03-03T18:12:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 -- 172.21.15.7:0/1683020443 shutdown_connections 2023-03-03T18:12:55.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e00c1ce0 0x7f32e0149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32cc0682f0 0x7f32cc06a7a0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32e0071e80 0x7f32e014cb70 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 --2- 172.21.15.7:0/1683020443 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32e0072250 0x7f32e0149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:12:55.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 -- 172.21.15.7:0/1683020443 >> 172.21.15.7:0/1683020443 conn(0x7f32e00a94d0 msgr2=0x7f32e00c2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:12:55.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.653+0000 7f32e5053700 1 -- 172.21.15.7:0/1683020443 shutdown_connections 2023-03-03T18:12:55.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:12:55.654+0000 7f32e5053700 1 -- 172.21.15.7:0/1683020443 wait complete. 2023-03-03T18:12:55.657 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:12:55.666 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T18:13:05.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:13:05.326 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:13:22.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:13:22.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.755+0000 7f23827b8700 1 Processor -- start 2023-03-03T18:13:22.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.756+0000 7f23827b8700 1 -- start start 2023-03-03T18:13:22.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.756+0000 7f23827b8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c14f9d0 0x7f237c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.756+0000 7f23827b8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f237c14fda0 0x7f237c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.756+0000 7f23827b8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c0c2f30 con 0x7f237c0abe20 2023-03-03T18:13:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.756+0000 7f23827b8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c0c31d0 con 0x7f237c14fda0 2023-03-03T18:13:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.756+0000 7f23827b8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1524a0 con 0x7f237c14f9d0 2023-03-03T18:13:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237b7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f237c14fda0 0x7f237c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:22.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c14f9d0 0x7f237c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237b7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f237c14fda0 0x7f237c071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:60324/0 (socket says 172.21.15.7:60324) 2023-03-03T18:13:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237b7fe700 1 -- 172.21.15.7:0/2589819823 learned_addr learned my addr 172.21.15.7:0/2589819823 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:13:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f2380d55700 1 --1- 172.21.15.7:0/2589819823 >> v1:172.21.15.143:6789/0 conn(0x7f237c0abe20 0x7f237c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:33820/0 (socket says 172.21.15.7:33820) 2023-03-03T18:13:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237bfff700 1 -- 172.21.15.7:0/2589819823 >> v2:172.21.15.143:3300/0 conn(0x7f237c14fda0 msgr2=0x7f237c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:13:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237bfff700 1 --2- 172.21.15.7:0/2589819823 >> v2:172.21.15.143:3300/0 conn(0x7f237c14fda0 0x7f237c071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237bfff700 1 -- 172.21.15.7:0/2589819823 >> v1:172.21.15.143:6789/0 conn(0x7f237c0abe20 legacy=0x7f237c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:22.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237bfff700 1 -- 172.21.15.7:0/2589819823 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c0c34b0 con 0x7f237c14f9d0 2023-03-03T18:13:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237affd700 1 -- 172.21.15.7:0/2589819823 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195787583 0 0) 0x7f237c0c2f30 con 0x7f237c0abe20 2023-03-03T18:13:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237b7fe700 1 --2- 172.21.15.7:0/2589819823 >> v2:172.21.15.143:3300/0 conn(0x7f237c14fda0 0x7f237c071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:13:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.757+0000 7f237bfff700 1 --2- 172.21.15.7:0/2589819823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c14f9d0 0x7f237c156430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f236c009840 tx=0x7f236c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51e6bf94c0bdfaae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:13:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f237affd700 1 -- 172.21.15.7:0/2589819823 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c011830 con 0x7f237c14f9d0 2023-03-03T18:13:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f237affd700 1 -- 172.21.15.7:0/2589819823 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c011990 con 0x7f237c14f9d0 2023-03-03T18:13:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f237affd700 1 -- 172.21.15.7:0/2589819823 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c011d20 con 0x7f237c14f9d0 2023-03-03T18:13:22.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f23827b8700 1 -- 172.21.15.7:0/2589819823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c14f9d0 msgr2=0x7f237c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f23827b8700 1 --2- 172.21.15.7:0/2589819823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c14f9d0 0x7f237c156430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f236c009840 tx=0x7f236c009b50 comp rx=0 tx=0).stop 2023-03-03T18:13:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f23827b8700 1 -- 172.21.15.7:0/2589819823 shutdown_connections 2023-03-03T18:13:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f23827b8700 1 --2- 172.21.15.7:0/2589819823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c14f9d0 0x7f237c156430 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f23827b8700 1 --2- 172.21.15.7:0/2589819823 >> v2:172.21.15.143:3300/0 conn(0x7f237c14fda0 0x7f237c071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f23827b8700 1 -- 172.21.15.7:0/2589819823 >> 172.21.15.7:0/2589819823 conn(0x7f237c0c8900 msgr2=0x7f237c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:13:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f23827b8700 1 -- 172.21.15.7:0/2589819823 shutdown_connections 2023-03-03T18:13:22.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.758+0000 7f23827b8700 1 -- 172.21.15.7:0/2589819823 wait complete. 2023-03-03T18:13:22.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.759+0000 7f23827b8700 1 Processor -- start 2023-03-03T18:13:22.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f23827b8700 1 -- start start 2023-03-03T18:13:22.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f23827b8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c0abe20 0x7f237c1e7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:22.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f23827b8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c14f9d0 0x7f237c0c20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:22.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f23827b8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f237c14fda0 0x7f237c14c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:22.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f23827b8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1dbc70 con 0x7f237c14fda0 2023-03-03T18:13:22.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f23827b8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1dbf50 con 0x7f237c14f9d0 2023-03-03T18:13:22.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f23827b8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c1dc230 con 0x7f237c0abe20 2023-03-03T18:13:22.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f2380d55700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f237c14fda0 0x7f237c14c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:22.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f2380d55700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f237c14fda0 0x7f237c14c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36328/0 (socket says 172.21.15.7:36328) 2023-03-03T18:13:22.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f2380d55700 1 -- 172.21.15.7:0/2590991455 learned_addr learned my addr 172.21.15.7:0/2590991455 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:13:22.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f237bfff700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c0abe20 0x7f237c1e7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:22.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.760+0000 7f237b7fe700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c14f9d0 0x7f237c0c20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:22.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f2380d55700 1 -- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c0abe20 msgr2=0x7f237c1e7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:22.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f2380d55700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c0abe20 0x7f237c1e7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:22.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f2380d55700 1 -- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c14f9d0 msgr2=0x7f237c0c20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:22.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f2380d55700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c14f9d0 0x7f237c0c20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:22.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f2380d55700 1 -- 172.21.15.7:0/2590991455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2370009550 con 0x7f237c14fda0 2023-03-03T18:13:22.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f237b7fe700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c14f9d0 0x7f237c0c20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:13:22.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f237bfff700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c0abe20 0x7f237c1e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:13:22.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f2380d55700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f237c14fda0 0x7f237c14c9a0 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7f237400cba0 tx=0x7f237400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:13:22.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f2378ff9700 1 -- 172.21.15.7:0/2590991455 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237400aaa0 con 0x7f237c14fda0 2023-03-03T18:13:22.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f2378ff9700 1 -- 172.21.15.7:0/2590991455 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237400ac20 con 0x7f237c14fda0 2023-03-03T18:13:22.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f23827b8700 1 -- 172.21.15.7:0/2590991455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c0094a0 con 0x7f237c14fda0 2023-03-03T18:13:22.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f2378ff9700 1 -- 172.21.15.7:0/2590991455 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23740070d0 con 0x7f237c14fda0 2023-03-03T18:13:22.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.761+0000 7f23827b8700 1 -- 172.21.15.7:0/2590991455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c0ac350 con 0x7f237c14fda0 2023-03-03T18:13:22.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.762+0000 7f23827b8700 1 -- 172.21.15.7:0/2590991455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237c0ab630 con 0x7f237c14fda0 2023-03-03T18:13:22.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.763+0000 7f2378ff9700 1 -- 172.21.15.7:0/2590991455 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f237401a030 con 0x7f237c14fda0 2023-03-03T18:13:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.764+0000 7f2378ff9700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23680682f0 0x7f236806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.764+0000 7f2378ff9700 1 -- 172.21.15.7:0/2590991455 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2374086000 con 0x7f237c14fda0 2023-03-03T18:13:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.764+0000 7f237bfff700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23680682f0 0x7f236806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.765+0000 7f237bfff700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23680682f0 0x7f236806a7a0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f236c0090b0 tx=0x7f236c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:13:22.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:22.767+0000 7f2378ff9700 1 -- 172.21.15.7:0/2590991455 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2374052440 con 0x7f237c14fda0 2023-03-03T18:13:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.006+0000 7f23827b8700 1 -- 172.21.15.7:0/2590991455 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f237c0c3e10 con 0x7f237c14fda0 2023-03-03T18:13:23.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.007+0000 7f2378ff9700 1 -- 172.21.15.7:0/2590991455 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2374018020 con 0x7f237c14fda0 2023-03-03T18:13:23.007 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:13:23.008 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:13:23.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 -- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23680682f0 msgr2=0x7f236806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:23.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23680682f0 0x7f236806a7a0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f236c0090b0 tx=0x7f236c008030 comp rx=0 tx=0).stop 2023-03-03T18:13:23.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 -- 172.21.15.7:0/2590991455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f237c14fda0 msgr2=0x7f237c14c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:23.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f237c14fda0 0x7f237c14c9a0 secure :-1 s=READY pgs=3135 cs=0 l=1 rev1=1 crypto rx=0x7f237400cba0 tx=0x7f237400ceb0 comp rx=0 tx=0).stop 2023-03-03T18:13:23.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 -- 172.21.15.7:0/2590991455 shutdown_connections 2023-03-03T18:13:23.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23680682f0 0x7f236806a7a0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:23.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f237c14fda0 0x7f237c14c9a0 unknown :-1 s=CLOSED pgs=3135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:23.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f237c0abe20 0x7f237c1e7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:23.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 --2- 172.21.15.7:0/2590991455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f237c14f9d0 0x7f237c0c20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:23.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 -- 172.21.15.7:0/2590991455 >> 172.21.15.7:0/2590991455 conn(0x7f237c0c8900 msgr2=0x7f237c1528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:13:23.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 -- 172.21.15.7:0/2590991455 shutdown_connections 2023-03-03T18:13:23.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:23.009+0000 7f23827b8700 1 -- 172.21.15.7:0/2590991455 wait complete. 2023-03-03T18:13:23.013 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:13:23.021 INFO:tasks.mds_thrash.fs.[b]:waiting for 47 secs before thrashing 2023-03-03T18:13:35.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:13:35.390 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:13:59.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:13:59.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.806+0000 7f803b9e4700 1 Processor -- start 2023-03-03T18:13:59.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.806+0000 7f803b9e4700 1 -- start start 2023-03-03T18:13:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f803b9e4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8034152b60 0x7f803415dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:59.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f803b9e4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80340ac900 con 0x7f80340ace30 2023-03-03T18:13:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f803b9e4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80340acb50 con 0x7f80340ac220 2023-03-03T18:13:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f803b9e4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8034071e30 con 0x7f8034152b60 2023-03-03T18:13:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f8038f7f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f80340ace30 0x7f8034152450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58562/0 (socket says 172.21.15.7:58562) 2023-03-03T18:13:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f8038f7f700 1 -- 172.21.15.7:0/3136856585 learned_addr learned my addr 172.21.15.7:0/3136856585 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:13:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f8039f81700 1 --2- 172.21.15.7:0/3136856585 >> v2:172.21.15.143:3300/0 conn(0x7f8034152b60 0x7f803415dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f802bfff700 1 -- 172.21.15.7:0/3136856585 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27506878 0 0) 0x7f80340ac900 con 0x7f80340ace30 2023-03-03T18:13:59.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f802bfff700 1 -- 172.21.15.7:0/3136856585 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80200035b0 con 0x7f80340ace30 2023-03-03T18:13:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f8039f81700 1 -- 172.21.15.7:0/3136856585 >> v1:172.21.15.143:6789/0 conn(0x7f80340ac220 legacy=0x7f80340736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f802bfff700 1 -- 172.21.15.7:0/3136856585 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 777677959 0 0) 0x7f80200035b0 con 0x7f80340ace30 2023-03-03T18:13:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f8039f81700 1 -- 172.21.15.7:0/3136856585 >> v1:172.21.15.7:6789/0 conn(0x7f80340ace30 legacy=0x7f8034152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.807+0000 7f8039f81700 1 -- 172.21.15.7:0/3136856585 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80341559c0 con 0x7f8034152b60 2023-03-03T18:13:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.808+0000 7f8039f81700 1 --2- 172.21.15.7:0/3136856585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8034152b60 0x7f803415dd00 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f802c009d90 tx=0x7f802c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b93317fc2e0defa6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:13:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.808+0000 7f802bfff700 1 -- 172.21.15.7:0/3136856585 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c00e070 con 0x7f8034152b60 2023-03-03T18:13:59.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.808+0000 7f802bfff700 1 -- 172.21.15.7:0/3136856585 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802c003a20 con 0x7f8034152b60 2023-03-03T18:13:59.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.808+0000 7f802bfff700 1 -- 172.21.15.7:0/3136856585 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8024001670 con 0x7f8034152b60 2023-03-03T18:13:59.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.808+0000 7f803b9e4700 1 -- 172.21.15.7:0/3136856585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8034152b60 msgr2=0x7f803415dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.808+0000 7f803b9e4700 1 --2- 172.21.15.7:0/3136856585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8034152b60 0x7f803415dd00 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f802c009d90 tx=0x7f802c00b330 comp rx=0 tx=0).stop 2023-03-03T18:13:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.808+0000 7f803b9e4700 1 -- 172.21.15.7:0/3136856585 shutdown_connections 2023-03-03T18:13:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.808+0000 7f803b9e4700 1 --2- 172.21.15.7:0/3136856585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8034152b60 0x7f803415dd00 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.808+0000 7f803b9e4700 1 -- 172.21.15.7:0/3136856585 >> 172.21.15.7:0/3136856585 conn(0x7f80340a94d0 msgr2=0x7f80340a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:13:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.809+0000 7f803b9e4700 1 -- 172.21.15.7:0/3136856585 shutdown_connections 2023-03-03T18:13:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.809+0000 7f803b9e4700 1 -- 172.21.15.7:0/3136856585 wait complete. 2023-03-03T18:13:59.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.810+0000 7f803b9e4700 1 Processor -- start 2023-03-03T18:13:59.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.810+0000 7f803b9e4700 1 -- start start 2023-03-03T18:13:59.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.810+0000 7f803b9e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80340ac220 0x7f8034154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:59.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.810+0000 7f803b9e4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80340ace30 0x7f80341eb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:59.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.810+0000 7f803b9e4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8034152b60 0x7f80341eb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:59.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.810+0000 7f803b9e4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80341dd490 con 0x7f8034152b60 2023-03-03T18:13:59.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.810+0000 7f803b9e4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80340c7010 con 0x7f80340ac220 2023-03-03T18:13:59.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.810+0000 7f803b9e4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80341dc2a0 con 0x7f80340ace30 2023-03-03T18:13:59.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039f81700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8034152b60 0x7f80341eb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:59.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039f81700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8034152b60 0x7f80341eb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36358/0 (socket says 172.21.15.7:36358) 2023-03-03T18:13:59.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039f81700 1 -- 172.21.15.7:0/3266908404 learned_addr learned my addr 172.21.15.7:0/3266908404 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:13:59.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8038f7f700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80340ace30 0x7f80341eb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:59.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039780700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80340ac220 0x7f8034154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:59.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039f81700 1 -- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80340ace30 msgr2=0x7f80341eb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:59.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039f81700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80340ace30 0x7f80341eb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:59.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039f81700 1 -- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80340ac220 msgr2=0x7f8034154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:13:59.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039f81700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80340ac220 0x7f8034154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:13:59.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039f81700 1 -- 172.21.15.7:0/3266908404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c009a70 con 0x7f8034152b60 2023-03-03T18:13:59.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039780700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80340ac220 0x7f8034154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:13:59.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8038f7f700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80340ace30 0x7f80341eb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:13:59.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8039f81700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8034152b60 0x7f80341eb890 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f802c00b370 tx=0x7f802c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:13:59.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8029ffb700 1 -- 172.21.15.7:0/3266908404 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c0046f0 con 0x7f8034152b60 2023-03-03T18:13:59.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8029ffb700 1 -- 172.21.15.7:0/3266908404 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802c004850 con 0x7f8034152b60 2023-03-03T18:13:59.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f803b9e4700 1 -- 172.21.15.7:0/3266908404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8034155f80 con 0x7f8034152b60 2023-03-03T18:13:59.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.811+0000 7f8029ffb700 1 -- 172.21.15.7:0/3266908404 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c012670 con 0x7f8034152b60 2023-03-03T18:13:59.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.812+0000 7f803b9e4700 1 -- 172.21.15.7:0/3266908404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8034155ca0 con 0x7f8034152b60 2023-03-03T18:13:59.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.812+0000 7f803b9e4700 1 -- 172.21.15.7:0/3266908404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8034156260 con 0x7f8034152b60 2023-03-03T18:13:59.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.813+0000 7f8029ffb700 1 -- 172.21.15.7:0/3266908404 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f802c023030 con 0x7f8034152b60 2023-03-03T18:13:59.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.813+0000 7f8029ffb700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8018068320 0x7f801806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:13:59.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.813+0000 7f8029ffb700 1 -- 172.21.15.7:0/3266908404 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f802c0875d0 con 0x7f8034152b60 2023-03-03T18:13:59.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.814+0000 7f8039780700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8018068320 0x7f801806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:13:59.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.814+0000 7f8039780700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8018068320 0x7f801806a7d0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f8030000f60 tx=0x7f8030009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:13:59.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:13:59.818+0000 7f8029ffb700 1 -- 172.21.15.7:0/3266908404 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f802c0538f0 con 0x7f8034152b60 2023-03-03T18:14:00.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.058+0000 7f803b9e4700 1 -- 172.21.15.7:0/3266908404 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80341eb070 con 0x7f8034152b60 2023-03-03T18:14:00.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.059+0000 7f8029ffb700 1 -- 172.21.15.7:0/3266908404 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f802c056f10 con 0x7f8034152b60 2023-03-03T18:14:00.059 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:14:00.059 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:14:00.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 -- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8018068320 msgr2=0x7f801806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:00.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8018068320 0x7f801806a7d0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f8030000f60 tx=0x7f8030009040 comp rx=0 tx=0).stop 2023-03-03T18:14:00.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 -- 172.21.15.7:0/3266908404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8034152b60 msgr2=0x7f80341eb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:00.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8034152b60 0x7f80341eb890 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f802c00b370 tx=0x7f802c004ad0 comp rx=0 tx=0).stop 2023-03-03T18:14:00.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 -- 172.21.15.7:0/3266908404 shutdown_connections 2023-03-03T18:14:00.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8018068320 0x7f801806a7d0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:00.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8034152b60 0x7f80341eb890 unknown :-1 s=CLOSED pgs=3137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:00.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80340ac220 0x7f8034154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:00.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 --2- 172.21.15.7:0/3266908404 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80340ace30 0x7f80341eb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:00.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 -- 172.21.15.7:0/3266908404 >> 172.21.15.7:0/3266908404 conn(0x7f80340a94d0 msgr2=0x7f80340c4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:14:00.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 -- 172.21.15.7:0/3266908404 shutdown_connections 2023-03-03T18:14:00.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:00.061+0000 7f803b9e4700 1 -- 172.21.15.7:0/3266908404 wait complete. 2023-03-03T18:14:00.063 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:14:00.073 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T18:14:05.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:14:05.471 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:14:10.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:14:10.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.161+0000 7f1f44738700 1 Processor -- start 2023-03-03T18:14:10.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.161+0000 7f1f44738700 1 -- start start 2023-03-03T18:14:10.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.162+0000 7f1f44738700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f3c14f9d0 0x7f1f3c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:10.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.162+0000 7f1f44738700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f3c0abe20 0x7f1f3c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:10.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.162+0000 7f1f44738700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c0c2f30 con 0x7f1f3c14fda0 2023-03-03T18:14:10.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.162+0000 7f1f44738700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c0c31d0 con 0x7f1f3c14f9d0 2023-03-03T18:14:10.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.162+0000 7f1f44738700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c1524a0 con 0x7f1f3c0abe20 2023-03-03T18:14:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f424d4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f3c14f9d0 0x7f1f3c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f424d4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f3c14f9d0 0x7f1f3c156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36364/0 (socket says 172.21.15.7:36364) 2023-03-03T18:14:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f424d4700 1 -- 172.21.15.7:0/1074062961 learned_addr learned my addr 172.21.15.7:0/1074062961 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:14:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f42cd5700 1 --2- 172.21.15.7:0/1074062961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f3c0abe20 0x7f1f3c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f424d4700 1 -- 172.21.15.7:0/1074062961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f3c0abe20 msgr2=0x7f1f3c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:14:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f424d4700 1 --2- 172.21.15.7:0/1074062961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f3c0abe20 0x7f1f3c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f424d4700 1 -- 172.21.15.7:0/1074062961 >> v1:172.21.15.143:6789/0 conn(0x7f1f3c14fda0 legacy=0x7f1f3c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:10.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f424d4700 1 -- 172.21.15.7:0/1074062961 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c0c34b0 con 0x7f1f3c14f9d0 2023-03-03T18:14:10.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f424d4700 1 --2- 172.21.15.7:0/1074062961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c14f9d0 0x7f1f3c156430 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f1f380098f0 tx=0x7f1f38009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880efe2b8068122e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:14:10.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f414d2700 1 -- 172.21.15.7:0/1074062961 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38011830 con 0x7f1f3c14f9d0 2023-03-03T18:14:10.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f414d2700 1 -- 172.21.15.7:0/1074062961 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f38011990 con 0x7f1f3c14f9d0 2023-03-03T18:14:10.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.163+0000 7f1f414d2700 1 -- 172.21.15.7:0/1074062961 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38011d20 con 0x7f1f3c14f9d0 2023-03-03T18:14:10.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.164+0000 7f1f44738700 1 -- 172.21.15.7:0/1074062961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c14f9d0 msgr2=0x7f1f3c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:10.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.164+0000 7f1f44738700 1 --2- 172.21.15.7:0/1074062961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c14f9d0 0x7f1f3c156430 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f1f380098f0 tx=0x7f1f38009c00 comp rx=0 tx=0).stop 2023-03-03T18:14:10.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.164+0000 7f1f44738700 1 -- 172.21.15.7:0/1074062961 shutdown_connections 2023-03-03T18:14:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.164+0000 7f1f44738700 1 --2- 172.21.15.7:0/1074062961 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f3c0abe20 0x7f1f3c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.164+0000 7f1f44738700 1 --2- 172.21.15.7:0/1074062961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c14f9d0 0x7f1f3c156430 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.164+0000 7f1f44738700 1 -- 172.21.15.7:0/1074062961 >> 172.21.15.7:0/1074062961 conn(0x7f1f3c0c8900 msgr2=0x7f1f3c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:14:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.164+0000 7f1f44738700 1 -- 172.21.15.7:0/1074062961 shutdown_connections 2023-03-03T18:14:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.164+0000 7f1f44738700 1 -- 172.21.15.7:0/1074062961 wait complete. 2023-03-03T18:14:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.165+0000 7f1f44738700 1 Processor -- start 2023-03-03T18:14:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.165+0000 7f1f44738700 1 -- start start 2023-03-03T18:14:10.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f44738700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c0abe20 0x7f1f3c0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f44738700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f3c14f9d0 0x7f1f3c14b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f44738700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f3c14fda0 0x7f1f3c14b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f44738700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c1df600 con 0x7f1f3c0abe20 2023-03-03T18:14:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f44738700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c1eaa40 con 0x7f1f3c14fda0 2023-03-03T18:14:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f44738700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c14c7d0 con 0x7f1f3c14f9d0 2023-03-03T18:14:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f424d4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c0abe20 0x7f1f3c0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:10.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f424d4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c0abe20 0x7f1f3c0c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36370/0 (socket says 172.21.15.7:36370) 2023-03-03T18:14:10.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f424d4700 1 -- 172.21.15.7:0/3086665104 learned_addr learned my addr 172.21.15.7:0/3086665104 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:14:10.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f42cd5700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f3c14fda0 0x7f1f3c14b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:10.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f41cd3700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f3c14f9d0 0x7f1f3c14b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:10.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f424d4700 1 -- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f3c14f9d0 msgr2=0x7f1f3c14b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:10.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f424d4700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f3c14f9d0 0x7f1f3c14b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:10.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f424d4700 1 -- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f3c14fda0 msgr2=0x7f1f3c14b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:10.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f424d4700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f3c14fda0 0x7f1f3c14b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:10.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.166+0000 7f1f424d4700 1 -- 172.21.15.7:0/3086665104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f38009550 con 0x7f1f3c0abe20 2023-03-03T18:14:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.167+0000 7f1f424d4700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c0abe20 0x7f1f3c0c1a50 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f1f38000c00 tx=0x7f1f38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:14:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.167+0000 7f1f32ffd700 1 -- 172.21.15.7:0/3086665104 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f380036a0 con 0x7f1f3c0abe20 2023-03-03T18:14:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.167+0000 7f1f32ffd700 1 -- 172.21.15.7:0/3086665104 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f38003820 con 0x7f1f3c0abe20 2023-03-03T18:14:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.167+0000 7f1f44738700 1 -- 172.21.15.7:0/3086665104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c0ac350 con 0x7f1f3c0abe20 2023-03-03T18:14:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.167+0000 7f1f32ffd700 1 -- 172.21.15.7:0/3086665104 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f38003bd0 con 0x7f1f3c0abe20 2023-03-03T18:14:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.167+0000 7f1f44738700 1 -- 172.21.15.7:0/3086665104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c0ab630 con 0x7f1f3c0abe20 2023-03-03T18:14:10.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.168+0000 7f1f44738700 1 -- 172.21.15.7:0/3086665104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c071fa0 con 0x7f1f3c0abe20 2023-03-03T18:14:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.168+0000 7f1f32ffd700 1 -- 172.21.15.7:0/3086665104 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f3800f050 con 0x7f1f3c0abe20 2023-03-03T18:14:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.169+0000 7f1f32ffd700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f200682f0 0x7f1f2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:10.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.169+0000 7f1f32ffd700 1 -- 172.21.15.7:0/3086665104 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1f38087790 con 0x7f1f3c0abe20 2023-03-03T18:14:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.169+0000 7f1f41cd3700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f200682f0 0x7f1f2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.169+0000 7f1f41cd3700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f200682f0 0x7f1f2006a7a0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c06f650 tx=0x7f1f2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:14:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.173+0000 7f1f32ffd700 1 -- 172.21.15.7:0/3086665104 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f38053b30 con 0x7f1f3c0abe20 2023-03-03T18:14:10.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.411+0000 7f1f44738700 1 -- 172.21.15.7:0/3086665104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f3c074bb0 con 0x7f1f3c0abe20 2023-03-03T18:14:10.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.411+0000 7f1f32ffd700 1 -- 172.21.15.7:0/3086665104 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1f38015090 con 0x7f1f3c0abe20 2023-03-03T18:14:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:14:10.412 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:14:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 -- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f200682f0 msgr2=0x7f1f2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f200682f0 0x7f1f2006a7a0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c06f650 tx=0x7f1f2c009040 comp rx=0 tx=0).stop 2023-03-03T18:14:10.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 -- 172.21.15.7:0/3086665104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c0abe20 msgr2=0x7f1f3c0c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c0abe20 0x7f1f3c0c1a50 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f1f38000c00 tx=0x7f1f38004750 comp rx=0 tx=0).stop 2023-03-03T18:14:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 -- 172.21.15.7:0/3086665104 shutdown_connections 2023-03-03T18:14:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f3c14fda0 0x7f1f3c14b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1f200682f0 0x7f1f2006a7a0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f3c0abe20 0x7f1f3c0c1a50 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 --2- 172.21.15.7:0/3086665104 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f3c14f9d0 0x7f1f3c14b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 -- 172.21.15.7:0/3086665104 >> 172.21.15.7:0/3086665104 conn(0x7f1f3c0c8900 msgr2=0x7f1f3c0c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:14:10.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 -- 172.21.15.7:0/3086665104 shutdown_connections 2023-03-03T18:14:10.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:10.413+0000 7f1f44738700 1 -- 172.21.15.7:0/3086665104 wait complete. 2023-03-03T18:14:10.415 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:14:10.425 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-03-03T18:14:35.537 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:14:35.541 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:14:42.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:14:42.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.212+0000 7f150e0d9700 1 Processor -- start 2023-03-03T18:14:42.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.212+0000 7f150e0d9700 1 -- start start 2023-03-03T18:14:42.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.213+0000 7f150e0d9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f150806d870 0x7f15081550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.213+0000 7f150e0d9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f15080c3e30 0x7f15080c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.213+0000 7f150e0d9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15080c7a60 con 0x7f15080c1da0 2023-03-03T18:14:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.213+0000 7f150e0d9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15080c2af0 con 0x7f15080c3e30 2023-03-03T18:14:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.213+0000 7f150e0d9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15080c3620 con 0x7f150806d870 2023-03-03T18:14:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.213+0000 7f1507fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f15080c3e30 0x7f15080c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.213+0000 7f1506ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f15080c1da0 0x7f15080c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58618/0 (socket says 172.21.15.7:58618) 2023-03-03T18:14:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.213+0000 7f1507fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f15080c3e30 0x7f15080c4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36410/0 (socket says 172.21.15.7:36410) 2023-03-03T18:14:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.213+0000 7f1506ffd700 1 -- 172.21.15.7:0/3141591613 learned_addr learned my addr 172.21.15.7:0/3141591613 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:14:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.214+0000 7f15077fe700 1 --2- 172.21.15.7:0/3141591613 >> v2:172.21.15.143:3300/0 conn(0x7f150806d870 0x7f15081550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.214+0000 7f1507fff700 1 -- 172.21.15.7:0/3141591613 >> v2:172.21.15.143:3300/0 conn(0x7f150806d870 msgr2=0x7f15081550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:14:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.214+0000 7f1507fff700 1 --2- 172.21.15.7:0/3141591613 >> v2:172.21.15.143:3300/0 conn(0x7f150806d870 0x7f15081550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.214+0000 7f1507fff700 1 -- 172.21.15.7:0/3141591613 >> v1:172.21.15.7:6789/0 conn(0x7f15080c1da0 legacy=0x7f15080c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.214+0000 7f15067fc700 1 -- 172.21.15.7:0/3141591613 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386976320 0 0) 0x7f15080c7a60 con 0x7f15080c1da0 2023-03-03T18:14:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.214+0000 7f1507fff700 1 -- 172.21.15.7:0/3141591613 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15081545e0 con 0x7f15080c3e30 2023-03-03T18:14:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.214+0000 7f1507fff700 1 --2- 172.21.15.7:0/3141591613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c3e30 0x7f15080c4230 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7f1508073880 tx=0x7f150000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=939a137802826ba2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:14:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.214+0000 7f15067fc700 1 -- 172.21.15.7:0/3141591613 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150000e070 con 0x7f15080c3e30 2023-03-03T18:14:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.214+0000 7f15067fc700 1 -- 172.21.15.7:0/3141591613 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150000bbd0 con 0x7f15080c3e30 2023-03-03T18:14:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.214+0000 7f15067fc700 1 -- 172.21.15.7:0/3141591613 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500012430 con 0x7f15080c3e30 2023-03-03T18:14:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.215+0000 7f150e0d9700 1 -- 172.21.15.7:0/3141591613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c3e30 msgr2=0x7f15080c4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.215+0000 7f150e0d9700 1 --2- 172.21.15.7:0/3141591613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c3e30 0x7f15080c4230 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7f1508073880 tx=0x7f150000b330 comp rx=0 tx=0).stop 2023-03-03T18:14:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.215+0000 7f150e0d9700 1 -- 172.21.15.7:0/3141591613 shutdown_connections 2023-03-03T18:14:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.215+0000 7f150e0d9700 1 --2- 172.21.15.7:0/3141591613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c3e30 0x7f15080c4230 unknown :-1 s=CLOSED pgs=3141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.215+0000 7f150e0d9700 1 --2- 172.21.15.7:0/3141591613 >> v2:172.21.15.143:3300/0 conn(0x7f150806d870 0x7f15081550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.215+0000 7f150e0d9700 1 -- 172.21.15.7:0/3141591613 >> 172.21.15.7:0/3141591613 conn(0x7f150806c110 msgr2=0x7f150806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:14:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.215+0000 7f150e0d9700 1 -- 172.21.15.7:0/3141591613 shutdown_connections 2023-03-03T18:14:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.215+0000 7f150e0d9700 1 -- 172.21.15.7:0/3141591613 wait complete. 2023-03-03T18:14:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.216+0000 7f150e0d9700 1 Processor -- start 2023-03-03T18:14:42.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.216+0000 7f150e0d9700 1 -- start start 2023-03-03T18:14:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f150e0d9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f150806d870 0x7f15080aaa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f150e0d9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c1da0 0x7f15081e98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f150e0d9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15080c3e30 0x7f15080ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f150e0d9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15080ab710 con 0x7f15080c1da0 2023-03-03T18:14:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f150e0d9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15080ab070 con 0x7f150806d870 2023-03-03T18:14:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f150e0d9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15081ed710 con 0x7f15080c3e30 2023-03-03T18:14:42.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f1506ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c1da0 0x7f15081e98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f1506ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c1da0 0x7f15081e98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36414/0 (socket says 172.21.15.7:36414) 2023-03-03T18:14:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f1506ffd700 1 -- 172.21.15.7:0/1878642405 learned_addr learned my addr 172.21.15.7:0/1878642405 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:14:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f15077fe700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f150806d870 0x7f15080aaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f1507fff700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15080c3e30 0x7f15080ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f1506ffd700 1 -- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15080c3e30 msgr2=0x7f15080ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f1506ffd700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15080c3e30 0x7f15080ac520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:42.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f1506ffd700 1 -- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f150806d870 msgr2=0x7f15080aaa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f1506ffd700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f150806d870 0x7f15080aaa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f1506ffd700 1 -- 172.21.15.7:0/1878642405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1500009a70 con 0x7f15080c1da0 2023-03-03T18:14:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f15077fe700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f150806d870 0x7f15080aaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:14:42.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.217+0000 7f1506ffd700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c1da0 0x7f15081e98d0 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0011e0 tx=0x7f14fc00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:14:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.218+0000 7f14effff700 1 -- 172.21.15.7:0/1878642405 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc004c80 con 0x7f15080c1da0 2023-03-03T18:14:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.218+0000 7f14effff700 1 -- 172.21.15.7:0/1878642405 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc004e00 con 0x7f15080c1da0 2023-03-03T18:14:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.218+0000 7f150e0d9700 1 -- 172.21.15.7:0/1878642405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1508154ba0 con 0x7f15080c1da0 2023-03-03T18:14:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.218+0000 7f14effff700 1 -- 172.21.15.7:0/1878642405 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc007310 con 0x7f15080c1da0 2023-03-03T18:14:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.218+0000 7f150e0d9700 1 -- 172.21.15.7:0/1878642405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15081548c0 con 0x7f15080c1da0 2023-03-03T18:14:42.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.219+0000 7f150e0d9700 1 -- 172.21.15.7:0/1878642405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150806dc40 con 0x7f15080c1da0 2023-03-03T18:14:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.219+0000 7f14effff700 1 -- 172.21.15.7:0/1878642405 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f14fc01a020 con 0x7f15080c1da0 2023-03-03T18:14:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.219+0000 7f14effff700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14f40682f0 0x7f14f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:14:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.220+0000 7f14effff700 1 -- 172.21.15.7:0/1878642405 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f14fc085d50 con 0x7f15080c1da0 2023-03-03T18:14:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.220+0000 7f15077fe700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14f40682f0 0x7f14f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:14:42.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.220+0000 7f15077fe700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14f40682f0 0x7f14f406a7a0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f14f8006fd0 tx=0x7f14f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:14:42.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.224+0000 7f14effff700 1 -- 172.21.15.7:0/1878642405 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f14fc0520f0 con 0x7f15080c1da0 2023-03-03T18:14:42.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.460+0000 7f150e0d9700 1 -- 172.21.15.7:0/1878642405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1508074bb0 con 0x7f15080c1da0 2023-03-03T18:14:42.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.460+0000 7f14effff700 1 -- 172.21.15.7:0/1878642405 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f14fc055710 con 0x7f15080c1da0 2023-03-03T18:14:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:14:42.460 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:14:42.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.462+0000 7f150e0d9700 1 -- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14f40682f0 msgr2=0x7f14f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:42.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.462+0000 7f150e0d9700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14f40682f0 0x7f14f406a7a0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f14f8006fd0 tx=0x7f14f8008040 comp rx=0 tx=0).stop 2023-03-03T18:14:42.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.462+0000 7f150e0d9700 1 -- 172.21.15.7:0/1878642405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c1da0 msgr2=0x7f15081e98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:14:42.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.462+0000 7f150e0d9700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c1da0 0x7f15081e98d0 secure :-1 s=READY pgs=3142 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0011e0 tx=0x7f14fc00cae0 comp rx=0 tx=0).stop 2023-03-03T18:14:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.462+0000 7f150e0d9700 1 -- 172.21.15.7:0/1878642405 shutdown_connections 2023-03-03T18:14:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.462+0000 7f150e0d9700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f15080c3e30 0x7f15080ac520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.462+0000 7f150e0d9700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f150806d870 0x7f15080aaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.462+0000 7f150e0d9700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14f40682f0 0x7f14f406a7a0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.462+0000 7f150e0d9700 1 --2- 172.21.15.7:0/1878642405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15080c1da0 0x7f15081e98d0 unknown :-1 s=CLOSED pgs=3142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:14:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.462+0000 7f150e0d9700 1 -- 172.21.15.7:0/1878642405 >> 172.21.15.7:0/1878642405 conn(0x7f150806c110 msgr2=0x7f150814e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:14:42.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.463+0000 7f150e0d9700 1 -- 172.21.15.7:0/1878642405 shutdown_connections 2023-03-03T18:14:42.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:14:42.463+0000 7f150e0d9700 1 -- 172.21.15.7:0/1878642405 wait complete. 2023-03-03T18:14:42.464 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:14:42.474 INFO:tasks.mds_thrash.fs.[a]:waiting for 34 secs before thrashing 2023-03-03T18:15:05.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:15:05.613 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:15:10.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:15:10.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.565+0000 7f7a792aa700 1 Processor -- start 2023-03-03T18:15:10.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.565+0000 7f7a792aa700 1 -- start start 2023-03-03T18:15:10.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a792aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 0x7f7a740c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:10.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a792aa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7a740c7590 0x7f7a7415ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:10.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a792aa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a740c6cf0 con 0x7f7a74151cc0 2023-03-03T18:15:10.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a792aa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a74152090 con 0x7f7a740c7590 2023-03-03T18:15:10.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a792aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a741548f0 con 0x7f7a740b00b0 2023-03-03T18:15:10.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a72ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7a74151cc0 0x7f7a740ca6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58654/0 (socket says 172.21.15.7:58654) 2023-03-03T18:15:10.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a72ffd700 1 -- 172.21.15.7:0/2920545067 learned_addr learned my addr 172.21.15.7:0/2920545067 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:15:10.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a737fe700 1 --2- 172.21.15.7:0/2920545067 >> v2:172.21.15.7:3300/0 conn(0x7f7a740c7590 0x7f7a7415ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:10.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a71ffb700 1 -- 172.21.15.7:0/2920545067 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577275811 0 0) 0x7f7a740c6cf0 con 0x7f7a74151cc0 2023-03-03T18:15:10.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a71ffb700 1 -- 172.21.15.7:0/2920545067 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c0035b0 con 0x7f7a74151cc0 2023-03-03T18:15:10.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a727fc700 1 --2- 172.21.15.7:0/2920545067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 0x7f7a740c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:10.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a737fe700 1 -- 172.21.15.7:0/2920545067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 msgr2=0x7f7a740c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:15:10.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a737fe700 1 --2- 172.21.15.7:0/2920545067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 0x7f7a740c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a737fe700 1 -- 172.21.15.7:0/2920545067 >> v1:172.21.15.7:6789/0 conn(0x7f7a74151cc0 legacy=0x7f7a740ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:10.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a71ffb700 1 -- 172.21.15.7:0/2920545067 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 999212427 0 0) 0x7f7a5c0035b0 con 0x7f7a74151cc0 2023-03-03T18:15:10.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a737fe700 1 -- 172.21.15.7:0/2920545067 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a74154bd0 con 0x7f7a740c7590 2023-03-03T18:15:10.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a737fe700 1 --2- 172.21.15.7:0/2920545067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 0x7f7a7415ef20 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009d90 tx=0x7f7a6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0e3936b76785a2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:10.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.566+0000 7f7a71ffb700 1 -- 172.21.15.7:0/2920545067 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c00e070 con 0x7f7a740c7590 2023-03-03T18:15:10.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.567+0000 7f7a71ffb700 1 -- 172.21.15.7:0/2920545067 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c003a20 con 0x7f7a740c7590 2023-03-03T18:15:10.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.567+0000 7f7a71ffb700 1 -- 172.21.15.7:0/2920545067 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a64001670 con 0x7f7a740c7590 2023-03-03T18:15:10.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.567+0000 7f7a792aa700 1 -- 172.21.15.7:0/2920545067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 msgr2=0x7f7a7415ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:10.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.567+0000 7f7a792aa700 1 --2- 172.21.15.7:0/2920545067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 0x7f7a7415ef20 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009d90 tx=0x7f7a6c00b330 comp rx=0 tx=0).stop 2023-03-03T18:15:10.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.567+0000 7f7a792aa700 1 -- 172.21.15.7:0/2920545067 shutdown_connections 2023-03-03T18:15:10.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.567+0000 7f7a792aa700 1 --2- 172.21.15.7:0/2920545067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 0x7f7a7415ef20 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.567+0000 7f7a792aa700 1 --2- 172.21.15.7:0/2920545067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 0x7f7a740c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.567+0000 7f7a792aa700 1 -- 172.21.15.7:0/2920545067 >> 172.21.15.7:0/2920545067 conn(0x7f7a7406b980 msgr2=0x7f7a74071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:15:10.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.567+0000 7f7a792aa700 1 -- 172.21.15.7:0/2920545067 shutdown_connections 2023-03-03T18:15:10.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.567+0000 7f7a792aa700 1 -- 172.21.15.7:0/2920545067 wait complete. 2023-03-03T18:15:10.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.569+0000 7f7a792aa700 1 Processor -- start 2023-03-03T18:15:10.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.569+0000 7f7a792aa700 1 -- start start 2023-03-03T18:15:10.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a792aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 0x7f7a740bfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:10.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a792aa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 0x7f7a741ed530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:10.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a792aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a74151cc0 0x7f7a741eda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:10.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a792aa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a741edfb0 con 0x7f7a740c7590 2023-03-03T18:15:10.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a792aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a741ed050 con 0x7f7a74151cc0 2023-03-03T18:15:10.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a792aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a74144290 con 0x7f7a740b00b0 2023-03-03T18:15:10.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a727fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 0x7f7a741ed530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:10.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a727fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 0x7f7a741ed530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36450/0 (socket says 172.21.15.7:36450) 2023-03-03T18:15:10.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a727fc700 1 -- 172.21.15.7:0/2295303407 learned_addr learned my addr 172.21.15.7:0/2295303407 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:15:10.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a72ffd700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 0x7f7a740bfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:10.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a737fe700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a74151cc0 0x7f7a741eda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:10.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a727fc700 1 -- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 msgr2=0x7f7a740bfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:10.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a727fc700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 0x7f7a740bfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a727fc700 1 -- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a74151cc0 msgr2=0x7f7a741eda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:10.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a727fc700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a74151cc0 0x7f7a741eda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a727fc700 1 -- 172.21.15.7:0/2295303407 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c009a70 con 0x7f7a740c7590 2023-03-03T18:15:10.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.570+0000 7f7a72ffd700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 0x7f7a740bfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:15:10.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.571+0000 7f7a727fc700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 0x7f7a741ed530 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800e810 tx=0x7f7a6800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:10.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.571+0000 7f7a5b7fe700 1 -- 172.21.15.7:0/2295303407 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68013610 con 0x7f7a740c7590 2023-03-03T18:15:10.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.571+0000 7f7a5b7fe700 1 -- 172.21.15.7:0/2295303407 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a68013770 con 0x7f7a740c7590 2023-03-03T18:15:10.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.571+0000 7f7a792aa700 1 -- 172.21.15.7:0/2295303407 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a74155190 con 0x7f7a740c7590 2023-03-03T18:15:10.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.571+0000 7f7a5b7fe700 1 -- 172.21.15.7:0/2295303407 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a68013b20 con 0x7f7a740c7590 2023-03-03T18:15:10.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.571+0000 7f7a792aa700 1 -- 172.21.15.7:0/2295303407 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a74154eb0 con 0x7f7a740c7590 2023-03-03T18:15:10.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.572+0000 7f7a792aa700 1 -- 172.21.15.7:0/2295303407 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a741545a0 con 0x7f7a740c7590 2023-03-03T18:15:10.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.572+0000 7f7a5b7fe700 1 -- 172.21.15.7:0/2295303407 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7a68023070 con 0x7f7a740c7590 2023-03-03T18:15:10.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.573+0000 7f7a5b7fe700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a600682f0 0x7f7a6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:10.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.573+0000 7f7a5b7fe700 1 -- 172.21.15.7:0/2295303407 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7a68085ec0 con 0x7f7a740c7590 2023-03-03T18:15:10.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.573+0000 7f7a72ffd700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a600682f0 0x7f7a6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:10.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.574+0000 7f7a72ffd700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a600682f0 0x7f7a6006a7a0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f7a64001eb0 tx=0x7f7a64001ce0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:10.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.577+0000 7f7a5b7fe700 1 -- 172.21.15.7:0/2295303407 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7a680521e0 con 0x7f7a740c7590 2023-03-03T18:15:10.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.816+0000 7f7a792aa700 1 -- 172.21.15.7:0/2295303407 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a741ed380 con 0x7f7a740c7590 2023-03-03T18:15:10.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.817+0000 7f7a5b7fe700 1 -- 172.21.15.7:0/2295303407 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7a68055800 con 0x7f7a740c7590 2023-03-03T18:15:10.817 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:15:10.817 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:15:10.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 -- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a600682f0 msgr2=0x7f7a6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:10.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a600682f0 0x7f7a6006a7a0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f7a64001eb0 tx=0x7f7a64001ce0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 -- 172.21.15.7:0/2295303407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 msgr2=0x7f7a741ed530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:10.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 0x7f7a741ed530 secure :-1 s=READY pgs=3145 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800e810 tx=0x7f7a6800ebd0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 -- 172.21.15.7:0/2295303407 shutdown_connections 2023-03-03T18:15:10.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7a74151cc0 0x7f7a741eda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7a740b00b0 0x7f7a740bfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7a600682f0 0x7f7a6006a7a0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 --2- 172.21.15.7:0/2295303407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a740c7590 0x7f7a741ed530 unknown :-1 s=CLOSED pgs=3145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:10.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 -- 172.21.15.7:0/2295303407 >> 172.21.15.7:0/2295303407 conn(0x7f7a7406b980 msgr2=0x7f7a740c6270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:15:10.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 -- 172.21.15.7:0/2295303407 shutdown_connections 2023-03-03T18:15:10.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:10.819+0000 7f7a792aa700 1 -- 172.21.15.7:0/2295303407 wait complete. 2023-03-03T18:15:10.820 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:15:10.831 INFO:tasks.mds_thrash.fs.[b]:waiting for 113 secs before thrashing 2023-03-03T18:15:16.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:15:16.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.615+0000 7f647b627700 1 Processor -- start 2023-03-03T18:15:16.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.615+0000 7f647b627700 1 -- start start 2023-03-03T18:15:16.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.616+0000 7f647b627700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f64741530d0 0x7f6474156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.616+0000 7f647b627700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64740c2f30 con 0x7f647414fda0 2023-03-03T18:15:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.616+0000 7f647b627700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64740c31d0 con 0x7f647414f9d0 2023-03-03T18:15:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.616+0000 7f647b627700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64741524a0 con 0x7f64741530d0 2023-03-03T18:15:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.616+0000 7f6478bc2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f647414fda0 0x7f64741529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58678/0 (socket says 172.21.15.7:58678) 2023-03-03T18:15:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.616+0000 7f6478bc2700 1 -- 172.21.15.7:0/2139802523 learned_addr learned my addr 172.21.15.7:0/2139802523 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:15:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6479bc4700 1 --2- 172.21.15.7:0/2139802523 >> v2:172.21.15.143:3300/0 conn(0x7f64741530d0 0x7f6474156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6467fff700 1 -- 172.21.15.7:0/2139802523 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584440423 0 0) 0x7f64740c2f30 con 0x7f647414fda0 2023-03-03T18:15:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6467fff700 1 -- 172.21.15.7:0/2139802523 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64600035b0 con 0x7f647414fda0 2023-03-03T18:15:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6479bc4700 1 -- 172.21.15.7:0/2139802523 >> v1:172.21.15.143:6789/0 conn(0x7f647414f9d0 legacy=0x7f6474071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6467fff700 1 -- 172.21.15.7:0/2139802523 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914991007 0 0) 0x7f64740c31d0 con 0x7f647414f9d0 2023-03-03T18:15:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6479bc4700 1 -- 172.21.15.7:0/2139802523 >> v1:172.21.15.7:6789/0 conn(0x7f647414fda0 legacy=0x7f64741529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6479bc4700 1 -- 172.21.15.7:0/2139802523 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64740c34b0 con 0x7f64741530d0 2023-03-03T18:15:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6479bc4700 1 --2- 172.21.15.7:0/2139802523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64741530d0 0x7f6474156430 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f646c009d90 tx=0x7f646c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7da9369569da10db server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6467fff700 1 -- 172.21.15.7:0/2139802523 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c00e070 con 0x7f64741530d0 2023-03-03T18:15:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6467fff700 1 -- 172.21.15.7:0/2139802523 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c00bb20 con 0x7f64741530d0 2023-03-03T18:15:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.617+0000 7f6467fff700 1 -- 172.21.15.7:0/2139802523 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c00be90 con 0x7f64741530d0 2023-03-03T18:15:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.618+0000 7f647b627700 1 -- 172.21.15.7:0/2139802523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64741530d0 msgr2=0x7f6474156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.618+0000 7f647b627700 1 --2- 172.21.15.7:0/2139802523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64741530d0 0x7f6474156430 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f646c009d90 tx=0x7f646c00b330 comp rx=0 tx=0).stop 2023-03-03T18:15:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.618+0000 7f647b627700 1 -- 172.21.15.7:0/2139802523 shutdown_connections 2023-03-03T18:15:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.618+0000 7f647b627700 1 --2- 172.21.15.7:0/2139802523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64741530d0 0x7f6474156430 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.618+0000 7f647b627700 1 -- 172.21.15.7:0/2139802523 >> 172.21.15.7:0/2139802523 conn(0x7f64740c8900 msgr2=0x7f64740c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:15:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.618+0000 7f647b627700 1 -- 172.21.15.7:0/2139802523 shutdown_connections 2023-03-03T18:15:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.618+0000 7f647b627700 1 -- 172.21.15.7:0/2139802523 wait complete. 2023-03-03T18:15:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.619+0000 7f647b627700 1 Processor -- start 2023-03-03T18:15:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.619+0000 7f647b627700 1 -- start start 2023-03-03T18:15:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f647b627700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f647414f9d0 0x7f647406f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f647b627700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f647414fda0 0x7f64740c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f647b627700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64741530d0 0x7f64741e71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f647b627700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647406c990 con 0x7f647414f9d0 2023-03-03T18:15:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f647b627700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64741538d0 con 0x7f647414fda0 2023-03-03T18:15:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f647b627700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64741eb250 con 0x7f64741530d0 2023-03-03T18:15:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f64793c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f647414f9d0 0x7f647406f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f64793c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f647414f9d0 0x7f647406f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36470/0 (socket says 172.21.15.7:36470) 2023-03-03T18:15:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f64793c3700 1 -- 172.21.15.7:0/529753055 learned_addr learned my addr 172.21.15.7:0/529753055 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:15:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f6478bc2700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f647414fda0 0x7f64740c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f6479bc4700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64741530d0 0x7f64741e71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f64793c3700 1 -- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64741530d0 msgr2=0x7f64741e71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f64793c3700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64741530d0 0x7f64741e71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f64793c3700 1 -- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f647414fda0 msgr2=0x7f64740c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f64793c3700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f647414fda0 0x7f64740c7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.620+0000 7f64793c3700 1 -- 172.21.15.7:0/529753055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c009a70 con 0x7f647414f9d0 2023-03-03T18:15:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.621+0000 7f64793c3700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f647414f9d0 0x7f647406f040 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f647000c550 tx=0x7f647000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.621+0000 7f6465ffb700 1 -- 172.21.15.7:0/529753055 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700054e0 con 0x7f647414f9d0 2023-03-03T18:15:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.621+0000 7f6465ffb700 1 -- 172.21.15.7:0/529753055 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6470003680 con 0x7f647414f9d0 2023-03-03T18:15:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.621+0000 7f647b627700 1 -- 172.21.15.7:0/529753055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64740ac350 con 0x7f647414f9d0 2023-03-03T18:15:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.621+0000 7f6465ffb700 1 -- 172.21.15.7:0/529753055 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470003a30 con 0x7f647414f9d0 2023-03-03T18:15:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.621+0000 7f647b627700 1 -- 172.21.15.7:0/529753055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64740ab630 con 0x7f647414f9d0 2023-03-03T18:15:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.622+0000 7f647b627700 1 -- 172.21.15.7:0/529753055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6474071fa0 con 0x7f647414f9d0 2023-03-03T18:15:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.622+0000 7f6465ffb700 1 -- 172.21.15.7:0/529753055 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6470019030 con 0x7f647414f9d0 2023-03-03T18:15:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.623+0000 7f6465ffb700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64580682f0 0x7f645806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.623+0000 7f6465ffb700 1 -- 172.21.15.7:0/529753055 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6470085410 con 0x7f647414f9d0 2023-03-03T18:15:16.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.623+0000 7f6478bc2700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64580682f0 0x7f645806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:16.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.624+0000 7f6478bc2700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64580682f0 0x7f645806a7a0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f6468007fd0 tx=0x7f6468009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:16.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.627+0000 7f6465ffb700 1 -- 172.21.15.7:0/529753055 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f64700517b0 con 0x7f647414f9d0 2023-03-03T18:15:16.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.867+0000 7f647b627700 1 -- 172.21.15.7:0/529753055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64740a9df0 con 0x7f647414f9d0 2023-03-03T18:15:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.868+0000 7f6465ffb700 1 -- 172.21.15.7:0/529753055 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6470017020 con 0x7f647414f9d0 2023-03-03T18:15:16.868 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:15:16.868 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:15:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 -- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64580682f0 msgr2=0x7f645806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64580682f0 0x7f645806a7a0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f6468007fd0 tx=0x7f6468009040 comp rx=0 tx=0).stop 2023-03-03T18:15:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 -- 172.21.15.7:0/529753055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f647414f9d0 msgr2=0x7f647406f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f647414f9d0 0x7f647406f040 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7f647000c550 tx=0x7f647000c860 comp rx=0 tx=0).stop 2023-03-03T18:15:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 -- 172.21.15.7:0/529753055 shutdown_connections 2023-03-03T18:15:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64741530d0 0x7f64741e71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64580682f0 0x7f645806a7a0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f647414f9d0 0x7f647406f040 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 --2- 172.21.15.7:0/529753055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f647414fda0 0x7f64740c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 -- 172.21.15.7:0/529753055 >> 172.21.15.7:0/529753055 conn(0x7f64740c8900 msgr2=0x7f64740c3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:15:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 -- 172.21.15.7:0/529753055 shutdown_connections 2023-03-03T18:15:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:16.870+0000 7f647b627700 1 -- 172.21.15.7:0/529753055 wait complete. 2023-03-03T18:15:16.872 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:15:16.881 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T18:15:26.882 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:15:27.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.021+0000 7fbdd65be700 1 Processor -- start 2023-03-03T18:15:27.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.021+0000 7fbdd65be700 1 -- start start 2023-03-03T18:15:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdd65be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdd006a2b0 0x7fbdd006a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdd65be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00c5cb0 0x7fbdd015cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdd65be700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd01558f0 con 0x7fbdd00a6370 2023-03-03T18:15:27.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdd65be700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd006bab0 con 0x7fbdd006a2b0 2023-03-03T18:15:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdd65be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00a6180 con 0x7fbdd00c5cb0 2023-03-03T18:15:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdcf7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbdd00a6370 0x7fbdd0155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58696/0 (socket says 172.21.15.7:58696) 2023-03-03T18:15:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdcf7fe700 1 -- 172.21.15.7:0/1449910490 learned_addr learned my addr 172.21.15.7:0/1449910490 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:15:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdcffff700 1 --2- 172.21.15.7:0/1449910490 >> v2:172.21.15.143:3300/0 conn(0x7fbdd006a2b0 0x7fbdd006a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdd4b5b700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00c5cb0 0x7fbdd015cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2099945423 0 0) 0x7fbdd01558f0 con 0x7fbdd00a6370 2023-03-03T18:15:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdbc0035b0 con 0x7fbdd00a6370 2023-03-03T18:15:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1672270518 0 0) 0x7fbdbc0035b0 con 0x7fbdd00a6370 2023-03-03T18:15:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00c5cb0 msgr2=0x7fbdd015cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:15:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdceffd700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00c5cb0 0x7fbdd015cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 >> v2:172.21.15.143:3300/0 conn(0x7fbdd006a2b0 msgr2=0x7fbdd006a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:15:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.022+0000 7fbdceffd700 1 --2- 172.21.15.7:0/1449910490 >> v2:172.21.15.143:3300/0 conn(0x7fbdd006a2b0 0x7fbdd006a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd00a5ea0 con 0x7fbdd00a6370 2023-03-03T18:15:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fbdb8001670 con 0x7fbdd00a6370 2023-03-03T18:15:27.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 >> v1:172.21.15.7:6789/0 conn(0x7fbdd00a6370 legacy=0x7fbdd0155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00c5cb0 0x7fbdbc003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc003ff0 0x7fbdbc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc004d90 0x7fbdbc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00a6180 con 0x7fbdbc004d90 2023-03-03T18:15:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc00ba40 con 0x7fbdbc003ff0 2023-03-03T18:15:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc00bc00 con 0x7fbdd00c5cb0 2023-03-03T18:15:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdd4b5b700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc004d90 0x7fbdbc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:27.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdcf7fe700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc003ff0 0x7fbdbc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdcffff700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00c5cb0 0x7fbdbc003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdd4b5b700 1 -- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00c5cb0 msgr2=0x7fbdbc003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:15:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdd4b5b700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00c5cb0 0x7fbdbc003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdd4b5b700 1 -- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc003ff0 msgr2=0x7fbdbc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:15:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdd4b5b700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc003ff0 0x7fbdbc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdd4b5b700 1 -- 172.21.15.7:0/1449910490 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc0035b0 con 0x7fbdbc004d90 2023-03-03T18:15:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdcffff700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd00c5cb0 0x7fbdbc003a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:15:27.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdd4b5b700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc004d90 0x7fbdbc005200 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009f90 tx=0x7fbdbc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c1a97a57b6c20f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc800ec30 con 0x7fbdbc004d90 2023-03-03T18:15:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc800f040 con 0x7fbdbc004d90 2023-03-03T18:15:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.023+0000 7fbdceffd700 1 -- 172.21.15.7:0/1449910490 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8014480 con 0x7fbdbc004d90 2023-03-03T18:15:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.024+0000 7fbdd65be700 1 -- 172.21.15.7:0/1449910490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc004d90 msgr2=0x7fbdbc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.024+0000 7fbdd65be700 1 --2- 172.21.15.7:0/1449910490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc004d90 0x7fbdbc005200 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009f90 tx=0x7fbdbc005880 comp rx=0 tx=0).stop 2023-03-03T18:15:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.024+0000 7fbdd4b5b700 1 -- 172.21.15.7:0/1449910490 reap_dead start 2023-03-03T18:15:27.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.024+0000 7fbdd65be700 1 -- 172.21.15.7:0/1449910490 shutdown_connections 2023-03-03T18:15:27.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.024+0000 7fbdd65be700 1 -- 172.21.15.7:0/1449910490 >> 172.21.15.7:0/1449910490 conn(0x7fbdd00a9430 msgr2=0x7fbdd00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:15:27.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.024+0000 7fbdd65be700 1 -- 172.21.15.7:0/1449910490 shutdown_connections 2023-03-03T18:15:27.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.024+0000 7fbdd65be700 1 -- 172.21.15.7:0/1449910490 wait complete. 2023-03-03T18:15:27.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.025+0000 7fbdd65be700 1 Processor -- start 2023-03-03T18:15:27.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.025+0000 7fbdd65be700 1 -- start start 2023-03-03T18:15:27.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdd65be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd014f240 0x7fbdd00c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:27.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdd65be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd0072450 0x7fbdd01edb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:27.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdd65be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd00bfe70 0x7fbdd00c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:27.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdd65be700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0154070 con 0x7fbdd00bfe70 2023-03-03T18:15:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdd65be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd01ee070 con 0x7fbdd0072450 2023-03-03T18:15:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdd65be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00a9bf0 con 0x7fbdd014f240 2023-03-03T18:15:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdd4b5b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd00bfe70 0x7fbdd00c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdd4b5b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd00bfe70 0x7fbdd00c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36500/0 (socket says 172.21.15.7:36500) 2023-03-03T18:15:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdd4b5b700 1 -- 172.21.15.7:0/3471039400 learned_addr learned my addr 172.21.15.7:0/3471039400 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:15:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdcf7fe700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd0072450 0x7fbdd01edb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.026+0000 7fbdcffff700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd014f240 0x7fbdd00c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:27.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdd4b5b700 1 -- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd014f240 msgr2=0x7fbdd00c0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdd4b5b700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd014f240 0x7fbdd00c0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdd4b5b700 1 -- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd0072450 msgr2=0x7fbdd01edb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdd4b5b700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd0072450 0x7fbdd01edb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdd4b5b700 1 -- 172.21.15.7:0/3471039400 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd0153160 con 0x7fbdd00bfe70 2023-03-03T18:15:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdcffff700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd014f240 0x7fbdd00c0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:15:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdd4b5b700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd00bfe70 0x7fbdd00c02e0 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800e700 tx=0x7fbdc8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdccff9700 1 -- 172.21.15.7:0/3471039400 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8015cb0 con 0x7fbdd00bfe70 2023-03-03T18:15:27.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdccff9700 1 -- 172.21.15.7:0/3471039400 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc8015e30 con 0x7fbdd00bfe70 2023-03-03T18:15:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdd65be700 1 -- 172.21.15.7:0/3471039400 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd0152e80 con 0x7fbdd00bfe70 2023-03-03T18:15:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdccff9700 1 -- 172.21.15.7:0/3471039400 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc80276b0 con 0x7fbdd00bfe70 2023-03-03T18:15:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.027+0000 7fbdd65be700 1 -- 172.21.15.7:0/3471039400 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd00c1500 con 0x7fbdd00bfe70 2023-03-03T18:15:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.028+0000 7fbdd65be700 1 -- 172.21.15.7:0/3471039400 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd0074bb0 con 0x7fbdd00bfe70 2023-03-03T18:15:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.028+0000 7fbdccff9700 1 -- 172.21.15.7:0/3471039400 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbdc801e430 con 0x7fbdd00bfe70 2023-03-03T18:15:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.029+0000 7fbdccff9700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb00682f0 0x7fbdb006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:27.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.029+0000 7fbdccff9700 1 -- 172.21.15.7:0/3471039400 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbdc80881c0 con 0x7fbdd00bfe70 2023-03-03T18:15:27.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.029+0000 7fbdcffff700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb00682f0 0x7fbdb006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:27.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.030+0000 7fbdcffff700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb00682f0 0x7fbdb006a7a0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0009c60 tx=0x7fbdc0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:27.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.033+0000 7fbdccff9700 1 -- 172.21.15.7:0/3471039400 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbdc80545b0 con 0x7fbdd00bfe70 2023-03-03T18:15:27.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.272+0000 7fbdd65be700 1 -- 172.21.15.7:0/3471039400 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdd00c0b70 con 0x7fbdd00bfe70 2023-03-03T18:15:27.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.273+0000 7fbdccff9700 1 -- 172.21.15.7:0/3471039400 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbdc8018090 con 0x7fbdd00bfe70 2023-03-03T18:15:27.273 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:15:27.273 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:15:27.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 -- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb00682f0 msgr2=0x7fbdb006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:27.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb00682f0 0x7fbdb006a7a0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0009c60 tx=0x7fbdc0008040 comp rx=0 tx=0).stop 2023-03-03T18:15:27.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 -- 172.21.15.7:0/3471039400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd00bfe70 msgr2=0x7fbdd00c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:27.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd00bfe70 0x7fbdd00c02e0 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800e700 tx=0x7fbdc8009750 comp rx=0 tx=0).stop 2023-03-03T18:15:27.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 -- 172.21.15.7:0/3471039400 shutdown_connections 2023-03-03T18:15:27.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb00682f0 0x7fbdb006a7a0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:27.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd00bfe70 0x7fbdd00c02e0 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:27.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd014f240 0x7fbdd00c0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:27.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 --2- 172.21.15.7:0/3471039400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdd0072450 0x7fbdd01edb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:27.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 -- 172.21.15.7:0/3471039400 >> 172.21.15.7:0/3471039400 conn(0x7fbdd00a9430 msgr2=0x7fbdd00a8070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:15:27.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 -- 172.21.15.7:0/3471039400 shutdown_connections 2023-03-03T18:15:27.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:27.275+0000 7fbdd65be700 1 -- 172.21.15.7:0/3471039400 wait complete. 2023-03-03T18:15:27.276 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:15:27.286 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before thrashing 2023-03-03T18:15:35.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:15:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.426+0000 7fe09e263700 1 Processor -- start 2023-03-03T18:15:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.426+0000 7fe09e263700 1 -- start start 2023-03-03T18:15:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.427+0000 7fe09e263700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe09806b6f0 0x7fe098155120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.427+0000 7fe09e263700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe098157bb0 con 0x7fe0980c7a30 2023-03-03T18:15:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.427+0000 7fe09e263700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09806a2f0 con 0x7fe0980c7e00 2023-03-03T18:15:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.427+0000 7fe09e263700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09806ab00 con 0x7fe09806b6f0 2023-03-03T18:15:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.427+0000 7fe09c800700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe09806b6f0 0x7fe098155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.427+0000 7fe097fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe0980c7a30 0x7fe09814e160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58742/0 (socket says 172.21.15.7:58742) 2023-03-03T18:15:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.427+0000 7fe097fff700 1 -- 172.21.15.7:0/2365093183 learned_addr learned my addr 172.21.15.7:0/2365093183 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:15:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.428+0000 7fe09c800700 1 -- 172.21.15.7:0/2365093183 >> v1:172.21.15.143:6789/0 conn(0x7fe0980c7e00 legacy=0x7fe098154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.428+0000 7fe096ffd700 1 -- 172.21.15.7:0/2365093183 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769030459 0 0) 0x7fe098157bb0 con 0x7fe0980c7a30 2023-03-03T18:15:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.428+0000 7fe09c800700 1 -- 172.21.15.7:0/2365093183 >> v1:172.21.15.7:6789/0 conn(0x7fe0980c7a30 legacy=0x7fe09814e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.428+0000 7fe09c800700 1 -- 172.21.15.7:0/2365093183 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0980bfbb0 con 0x7fe09806b6f0 2023-03-03T18:15:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.428+0000 7fe09c800700 1 --2- 172.21.15.7:0/2365093183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe09806b6f0 0x7fe098155120 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7fe090009d90 tx=0x7fe09000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42747185cbc16bda server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.428+0000 7fe096ffd700 1 -- 172.21.15.7:0/2365093183 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09000e070 con 0x7fe09806b6f0 2023-03-03T18:15:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.428+0000 7fe096ffd700 1 -- 172.21.15.7:0/2365093183 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09000bbd0 con 0x7fe09806b6f0 2023-03-03T18:15:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.428+0000 7fe096ffd700 1 -- 172.21.15.7:0/2365093183 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe090012430 con 0x7fe09806b6f0 2023-03-03T18:15:35.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.429+0000 7fe09e263700 1 -- 172.21.15.7:0/2365093183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe09806b6f0 msgr2=0x7fe098155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.429+0000 7fe09e263700 1 --2- 172.21.15.7:0/2365093183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe09806b6f0 0x7fe098155120 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7fe090009d90 tx=0x7fe09000b330 comp rx=0 tx=0).stop 2023-03-03T18:15:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.429+0000 7fe09e263700 1 -- 172.21.15.7:0/2365093183 shutdown_connections 2023-03-03T18:15:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.429+0000 7fe09e263700 1 --2- 172.21.15.7:0/2365093183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe09806b6f0 0x7fe098155120 unknown :-1 s=CLOSED pgs=3152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.429+0000 7fe09e263700 1 -- 172.21.15.7:0/2365093183 >> 172.21.15.7:0/2365093183 conn(0x7fe098073d20 msgr2=0x7fe0980716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:15:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.429+0000 7fe09e263700 1 -- 172.21.15.7:0/2365093183 shutdown_connections 2023-03-03T18:15:35.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.429+0000 7fe09e263700 1 -- 172.21.15.7:0/2365093183 wait complete. 2023-03-03T18:15:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.430+0000 7fe09e263700 1 Processor -- start 2023-03-03T18:15:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.430+0000 7fe09e263700 1 -- start start 2023-03-03T18:15:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe09e263700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0980c7a30 0x7fe0981ecef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe09e263700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0980c7e00 0x7fe0980c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe09e263700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe098147570 0x7fe098145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:35.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe09e263700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe098145060 con 0x7fe0980c7e00 2023-03-03T18:15:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe09e263700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0980c1d80 con 0x7fe098147570 2023-03-03T18:15:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe09e263700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0980c9790 con 0x7fe0980c7a30 2023-03-03T18:15:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe0977fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0980c7e00 0x7fe0980c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe0977fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0980c7e00 0x7fe0980c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36538/0 (socket says 172.21.15.7:36538) 2023-03-03T18:15:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe0977fe700 1 -- 172.21.15.7:0/2733219328 learned_addr learned my addr 172.21.15.7:0/2733219328 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:15:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe09c800700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe098147570 0x7fe098145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:35.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe097fff700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0980c7a30 0x7fe0981ecef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe0977fe700 1 -- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0980c7a30 msgr2=0x7fe0981ecef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe0977fe700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0980c7a30 0x7fe0981ecef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe0977fe700 1 -- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe098147570 msgr2=0x7fe098145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe0977fe700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe098147570 0x7fe098145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe0977fe700 1 -- 172.21.15.7:0/2733219328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe090009a70 con 0x7fe0980c7e00 2023-03-03T18:15:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe09c800700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe098147570 0x7fe098145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:15:35.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe097fff700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0980c7a30 0x7fe0981ecef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:15:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.431+0000 7fe0977fe700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0980c7e00 0x7fe0980c24e0 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00c420 tx=0x7fe08c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.432+0000 7fe094ff9700 1 -- 172.21.15.7:0/2733219328 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0052d0 con 0x7fe0980c7e00 2023-03-03T18:15:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.432+0000 7fe094ff9700 1 -- 172.21.15.7:0/2733219328 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c005430 con 0x7fe0980c7e00 2023-03-03T18:15:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.432+0000 7fe09e263700 1 -- 172.21.15.7:0/2733219328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09806a820 con 0x7fe0980c7e00 2023-03-03T18:15:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.432+0000 7fe094ff9700 1 -- 172.21.15.7:0/2733219328 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0038c0 con 0x7fe0980c7e00 2023-03-03T18:15:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.432+0000 7fe09e263700 1 -- 172.21.15.7:0/2733219328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09806a540 con 0x7fe0980c7e00 2023-03-03T18:15:35.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.433+0000 7fe09e263700 1 -- 172.21.15.7:0/2733219328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe09814e870 con 0x7fe0980c7e00 2023-03-03T18:15:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.433+0000 7fe094ff9700 1 -- 172.21.15.7:0/2733219328 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe08c019030 con 0x7fe0980c7e00 2023-03-03T18:15:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.433+0000 7fe094ff9700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0840682f0 0x7fe08406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:15:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.434+0000 7fe094ff9700 1 -- 172.21.15.7:0/2733219328 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe08c085430 con 0x7fe0980c7e00 2023-03-03T18:15:35.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.434+0000 7fe097fff700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0840682f0 0x7fe08406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:15:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.434+0000 7fe097fff700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0840682f0 0x7fe08406a7a0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fe088000c00 tx=0x7fe088009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:15:35.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.438+0000 7fe094ff9700 1 -- 172.21.15.7:0/2733219328 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe08c0517d0 con 0x7fe0980c7e00 2023-03-03T18:15:35.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.678+0000 7fe09e263700 1 -- 172.21.15.7:0/2733219328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0981451a0 con 0x7fe0980c7e00 2023-03-03T18:15:35.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.679+0000 7fe094ff9700 1 -- 172.21.15.7:0/2733219328 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe0981451a0 con 0x7fe0980c7e00 2023-03-03T18:15:35.679 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:15:35.679 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:15:35.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.680+0000 7fe09e263700 1 -- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0840682f0 msgr2=0x7fe08406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:35.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.680+0000 7fe09e263700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0840682f0 0x7fe08406a7a0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fe088000c00 tx=0x7fe088009040 comp rx=0 tx=0).stop 2023-03-03T18:15:35.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.681+0000 7fe09e263700 1 -- 172.21.15.7:0/2733219328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0980c7e00 msgr2=0x7fe0980c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:15:35.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.681+0000 7fe09e263700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0980c7e00 0x7fe0980c24e0 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00c420 tx=0x7fe08c00c7e0 comp rx=0 tx=0).stop 2023-03-03T18:15:35.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.681+0000 7fe09e263700 1 -- 172.21.15.7:0/2733219328 shutdown_connections 2023-03-03T18:15:35.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.681+0000 7fe09e263700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe098147570 0x7fe098145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:35.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.681+0000 7fe09e263700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0980c7a30 0x7fe0981ecef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:35.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.681+0000 7fe09e263700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0840682f0 0x7fe08406a7a0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:35.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.681+0000 7fe09e263700 1 --2- 172.21.15.7:0/2733219328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe0980c7e00 0x7fe0980c24e0 unknown :-1 s=CLOSED pgs=3153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:15:35.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.681+0000 7fe09e263700 1 -- 172.21.15.7:0/2733219328 >> 172.21.15.7:0/2733219328 conn(0x7fe09806b6f0 msgr2=0x7fe098154910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:15:35.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.681+0000 7fe09e263700 1 -- 172.21.15.7:0/2733219328 shutdown_connections 2023-03-03T18:15:35.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:15:35.681+0000 7fe09e263700 1 -- 172.21.15.7:0/2733219328 wait complete. 2023-03-03T18:15:35.683 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:15:35.693 INFO:tasks.mds_thrash.fs.[a]:waiting for 119 secs before thrashing 2023-03-03T18:15:35.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:15:35.743 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:16:05.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:16:05.864 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:16:36.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:16:36.005 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:17:03.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:17:03.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.973+0000 7f68e03ac700 1 Processor -- start 2023-03-03T18:17:03.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.973+0000 7f68e03ac700 1 -- start start 2023-03-03T18:17:03.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68e03ac700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68d81586f0 0x7f68d806d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:03.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68e03ac700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d8157b50 con 0x7f68d806c560 2023-03-03T18:17:03.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68e03ac700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d806c240 con 0x7f68d8155090 2023-03-03T18:17:03.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68e03ac700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d80c33b0 con 0x7f68d81586f0 2023-03-03T18:17:03.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68de949700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f68d81586f0 0x7f68d806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:03.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68dd947700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f68d806c560 0x7f68d8157fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58820/0 (socket says 172.21.15.7:58820) 2023-03-03T18:17:03.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68dd947700 1 -- 172.21.15.7:0/1835824827 learned_addr learned my addr 172.21.15.7:0/1835824827 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:17:03.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68de949700 1 -- 172.21.15.7:0/1835824827 >> v1:172.21.15.143:6789/0 conn(0x7f68d8155090 legacy=0x7f68d80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:17:03.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68de949700 1 -- 172.21.15.7:0/1835824827 >> v1:172.21.15.7:6789/0 conn(0x7f68d806c560 legacy=0x7f68d8157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:03.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68dd146700 1 -- 172.21.15.7:0/1835824827 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104691813 0 0) 0x7f68d8157b50 con 0x7f68d806c560 2023-03-03T18:17:03.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.974+0000 7f68de949700 1 -- 172.21.15.7:0/1835824827 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d80c3690 con 0x7f68d81586f0 2023-03-03T18:17:03.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68de949700 1 --2- 172.21.15.7:0/1835824827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d81586f0 0x7f68d806d450 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f68d0009d90 tx=0x7f68d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59bced899e651bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:17:03.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68dd146700 1 -- 172.21.15.7:0/1835824827 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d000e070 con 0x7f68d81586f0 2023-03-03T18:17:03.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68dd146700 1 -- 172.21.15.7:0/1835824827 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d000bbd0 con 0x7f68d81586f0 2023-03-03T18:17:03.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68dd146700 1 -- 172.21.15.7:0/1835824827 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d0003a20 con 0x7f68d81586f0 2023-03-03T18:17:03.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68e03ac700 1 -- 172.21.15.7:0/1835824827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d81586f0 msgr2=0x7f68d806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:03.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68e03ac700 1 --2- 172.21.15.7:0/1835824827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d81586f0 0x7f68d806d450 secure :-1 s=READY pgs=3156 cs=0 l=1 rev1=1 crypto rx=0x7f68d0009d90 tx=0x7f68d000b330 comp rx=0 tx=0).stop 2023-03-03T18:17:03.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68e03ac700 1 -- 172.21.15.7:0/1835824827 shutdown_connections 2023-03-03T18:17:03.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68e03ac700 1 --2- 172.21.15.7:0/1835824827 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d81586f0 0x7f68d806d450 unknown :-1 s=CLOSED pgs=3156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:03.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68e03ac700 1 -- 172.21.15.7:0/1835824827 >> 172.21.15.7:0/1835824827 conn(0x7f68d80a94d0 msgr2=0x7f68d80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:17:03.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68e03ac700 1 -- 172.21.15.7:0/1835824827 shutdown_connections 2023-03-03T18:17:03.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.975+0000 7f68e03ac700 1 -- 172.21.15.7:0/1835824827 wait complete. 2023-03-03T18:17:03.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.976+0000 7f68e03ac700 1 Processor -- start 2023-03-03T18:17:03.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.977+0000 7f68e03ac700 1 -- start start 2023-03-03T18:17:03.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.977+0000 7f68e03ac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d806c560 0x7f68d81ed870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:03.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.977+0000 7f68e03ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d8155090 0x7f68d80c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:03.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.977+0000 7f68e03ac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d81586f0 0x7f68d80be6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:03.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.977+0000 7f68e03ac700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d80c1280 con 0x7f68d806c560 2023-03-03T18:17:03.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.977+0000 7f68e03ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d80c0c40 con 0x7f68d8155090 2023-03-03T18:17:03.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.977+0000 7f68e03ac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d80bd0f0 con 0x7f68d81586f0 2023-03-03T18:17:03.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.977+0000 7f68de148700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d806c560 0x7f68d81ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:03.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.977+0000 7f68de148700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d806c560 0x7f68d81ed870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36612/0 (socket says 172.21.15.7:36612) 2023-03-03T18:17:03.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.977+0000 7f68de148700 1 -- 172.21.15.7:0/3408738570 learned_addr learned my addr 172.21.15.7:0/3408738570 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:17:03.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68dd947700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d8155090 0x7f68d80c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:03.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68de949700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d81586f0 0x7f68d80be6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:03.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68de148700 1 -- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d81586f0 msgr2=0x7f68d80be6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:03.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68de148700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d81586f0 0x7f68d80be6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:03.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68de148700 1 -- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d8155090 msgr2=0x7f68d80c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:03.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68de148700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d8155090 0x7f68d80c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:03.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68de148700 1 -- 172.21.15.7:0/3408738570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d0009a70 con 0x7f68d806c560 2023-03-03T18:17:03.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68dd947700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d8155090 0x7f68d80c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:17:03.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68de148700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d806c560 0x7f68d81ed870 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7f68d400c4d0 tx=0x7f68d400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:17:03.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68ceffd700 1 -- 172.21.15.7:0/3408738570 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d4011790 con 0x7f68d806c560 2023-03-03T18:17:03.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68ceffd700 1 -- 172.21.15.7:0/3408738570 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d40118f0 con 0x7f68d806c560 2023-03-03T18:17:03.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68e03ac700 1 -- 172.21.15.7:0/3408738570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d81545e0 con 0x7f68d806c560 2023-03-03T18:17:03.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68ceffd700 1 -- 172.21.15.7:0/3408738570 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d4011ca0 con 0x7f68d806c560 2023-03-03T18:17:03.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.978+0000 7f68e03ac700 1 -- 172.21.15.7:0/3408738570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d80c3970 con 0x7f68d806c560 2023-03-03T18:17:03.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.979+0000 7f68e03ac700 1 -- 172.21.15.7:0/3408738570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d8155460 con 0x7f68d806c560 2023-03-03T18:17:03.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.979+0000 7f68ceffd700 1 -- 172.21.15.7:0/3408738570 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f68d4010050 con 0x7f68d806c560 2023-03-03T18:17:03.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.980+0000 7f68ceffd700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68bc0682f0 0x7f68bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:03.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.980+0000 7f68ceffd700 1 -- 172.21.15.7:0/3408738570 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f68d4085e40 con 0x7f68d806c560 2023-03-03T18:17:03.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.980+0000 7f68dd947700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68bc0682f0 0x7f68bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:03.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.981+0000 7f68dd947700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68bc0682f0 0x7f68bc06a7a0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f68d80c2d70 tx=0x7f68c8007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:17:03.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:03.984+0000 7f68ceffd700 1 -- 172.21.15.7:0/3408738570 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68d40521e0 con 0x7f68d806c560 2023-03-03T18:17:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.226+0000 7f68e03ac700 1 -- 172.21.15.7:0/3408738570 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68d8074bb0 con 0x7f68d806c560 2023-03-03T18:17:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.226+0000 7f68ceffd700 1 -- 172.21.15.7:0/3408738570 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f68d4085020 con 0x7f68d806c560 2023-03-03T18:17:04.227 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:17:04.228 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:17:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 -- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68bc0682f0 msgr2=0x7f68bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68bc0682f0 0x7f68bc06a7a0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f68d80c2d70 tx=0x7f68c8007ca0 comp rx=0 tx=0).stop 2023-03-03T18:17:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 -- 172.21.15.7:0/3408738570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d806c560 msgr2=0x7f68d81ed870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d806c560 0x7f68d81ed870 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7f68d400c4d0 tx=0x7f68d400c890 comp rx=0 tx=0).stop 2023-03-03T18:17:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 -- 172.21.15.7:0/3408738570 shutdown_connections 2023-03-03T18:17:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d81586f0 0x7f68d80be6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68bc0682f0 0x7f68bc06a7a0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d806c560 0x7f68d81ed870 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 --2- 172.21.15.7:0/3408738570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d8155090 0x7f68d80c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 -- 172.21.15.7:0/3408738570 >> 172.21.15.7:0/3408738570 conn(0x7f68d80a94d0 msgr2=0x7f68d8151920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:17:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 -- 172.21.15.7:0/3408738570 shutdown_connections 2023-03-03T18:17:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:04.229+0000 7f68e03ac700 1 -- 172.21.15.7:0/3408738570 wait complete. 2023-03-03T18:17:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:17:04.242 INFO:tasks.mds_thrash.fs.[b]:waiting for 112 secs before thrashing 2023-03-03T18:17:06.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:17:06.132 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:17:34.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:17:34.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.833+0000 7fbb90605700 1 Processor -- start 2023-03-03T18:17:34.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.833+0000 7fbb90605700 1 -- start start 2023-03-03T18:17:34.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.834+0000 7fbb90605700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 0x7fbb880736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:34.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.834+0000 7fbb90605700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb88073c30 0x7fbb8806bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:34.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.834+0000 7fbb90605700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8806d780 con 0x7fbb8806c3e0 2023-03-03T18:17:34.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.834+0000 7fbb90605700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb8806cb70 con 0x7fbb88073c30 2023-03-03T18:17:34.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.834+0000 7fbb90605700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8806ce50 con 0x7fbb88151ab0 2023-03-03T18:17:34.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.834+0000 7fbb8dba0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb88073c30 0x7fbb8806bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:34.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.834+0000 7fbb8eba2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbb8806c3e0 0x7fbb88157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58864/0 (socket says 172.21.15.7:58864) 2023-03-03T18:17:34.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.834+0000 7fbb8eba2700 1 -- 172.21.15.7:0/3215592978 learned_addr learned my addr 172.21.15.7:0/3215592978 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:17:34.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.834+0000 7fbb8e3a1700 1 --2- 172.21.15.7:0/3215592978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 0x7fbb880736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:34.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.835+0000 7fbb8dba0700 1 -- 172.21.15.7:0/3215592978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 msgr2=0x7fbb880736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:17:34.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.835+0000 7fbb8dba0700 1 --2- 172.21.15.7:0/3215592978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 0x7fbb880736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:34.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.835+0000 7fbb8dba0700 1 -- 172.21.15.7:0/3215592978 >> v1:172.21.15.7:6789/0 conn(0x7fbb8806c3e0 legacy=0x7fbb88157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:34.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.835+0000 7fbb8dba0700 1 -- 172.21.15.7:0/3215592978 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8806d130 con 0x7fbb88073c30 2023-03-03T18:17:34.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.835+0000 7fbb8d39f700 1 -- 172.21.15.7:0/3215592978 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619756164 0 0) 0x7fbb8806d780 con 0x7fbb8806c3e0 2023-03-03T18:17:34.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.835+0000 7fbb8e3a1700 1 --2- 172.21.15.7:0/3215592978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 0x7fbb880736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:17:34.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.835+0000 7fbb8dba0700 1 --2- 172.21.15.7:0/3215592978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 0x7fbb8806bea0 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009840 tx=0x7fbb84009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69cbf0d5f169f71 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:17:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.835+0000 7fbb8d39f700 1 -- 172.21.15.7:0/3215592978 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84011830 con 0x7fbb88073c30 2023-03-03T18:17:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.835+0000 7fbb8d39f700 1 -- 172.21.15.7:0/3215592978 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb84011990 con 0x7fbb88073c30 2023-03-03T18:17:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.835+0000 7fbb8d39f700 1 -- 172.21.15.7:0/3215592978 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84011d20 con 0x7fbb88073c30 2023-03-03T18:17:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.836+0000 7fbb90605700 1 -- 172.21.15.7:0/3215592978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 msgr2=0x7fbb8806bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.836+0000 7fbb90605700 1 --2- 172.21.15.7:0/3215592978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 0x7fbb8806bea0 secure :-1 s=READY pgs=3159 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009840 tx=0x7fbb84009b50 comp rx=0 tx=0).stop 2023-03-03T18:17:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.836+0000 7fbb90605700 1 -- 172.21.15.7:0/3215592978 shutdown_connections 2023-03-03T18:17:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.836+0000 7fbb90605700 1 --2- 172.21.15.7:0/3215592978 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 0x7fbb880736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:34.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.836+0000 7fbb90605700 1 --2- 172.21.15.7:0/3215592978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 0x7fbb8806bea0 unknown :-1 s=CLOSED pgs=3159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:34.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.836+0000 7fbb90605700 1 -- 172.21.15.7:0/3215592978 >> 172.21.15.7:0/3215592978 conn(0x7fbb880a94d0 msgr2=0x7fbb880a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:17:34.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.836+0000 7fbb90605700 1 -- 172.21.15.7:0/3215592978 shutdown_connections 2023-03-03T18:17:34.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.836+0000 7fbb90605700 1 -- 172.21.15.7:0/3215592978 wait complete. 2023-03-03T18:17:34.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.837+0000 7fbb90605700 1 Processor -- start 2023-03-03T18:17:34.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.837+0000 7fbb90605700 1 -- start start 2023-03-03T18:17:34.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb90605700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb8806c3e0 0x7fbb881e9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:34.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb90605700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 0x7fbb880c0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb90605700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 0x7fbb880c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb90605700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880c0e40 con 0x7fbb88073c30 2023-03-03T18:17:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb90605700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880ab150 con 0x7fbb8806c3e0 2023-03-03T18:17:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb90605700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb880aa540 con 0x7fbb88151ab0 2023-03-03T18:17:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8dba0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 0x7fbb880c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8dba0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 0x7fbb880c0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36656/0 (socket says 172.21.15.7:36656) 2023-03-03T18:17:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8dba0700 1 -- 172.21.15.7:0/2865659770 learned_addr learned my addr 172.21.15.7:0/2865659770 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:17:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8eba2700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 0x7fbb880c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8e3a1700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb8806c3e0 0x7fbb881e9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8dba0700 1 -- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 msgr2=0x7fbb880c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8dba0700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 0x7fbb880c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8dba0700 1 -- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb8806c3e0 msgr2=0x7fbb881e9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8dba0700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb8806c3e0 0x7fbb881e9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8dba0700 1 -- 172.21.15.7:0/2865659770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb840094a0 con 0x7fbb88073c30 2023-03-03T18:17:34.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb8dba0700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 0x7fbb880c0900 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7fbb840090b0 tx=0x7fbb84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:17:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb7effd700 1 -- 172.21.15.7:0/2865659770 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb840038a0 con 0x7fbb88073c30 2023-03-03T18:17:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb7effd700 1 -- 172.21.15.7:0/2865659770 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8401f070 con 0x7fbb88073c30 2023-03-03T18:17:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb90605700 1 -- 172.21.15.7:0/2865659770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb88151e80 con 0x7fbb88073c30 2023-03-03T18:17:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb7effd700 1 -- 172.21.15.7:0/2865659770 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84003c20 con 0x7fbb88073c30 2023-03-03T18:17:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.838+0000 7fbb90605700 1 -- 172.21.15.7:0/2865659770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8806d410 con 0x7fbb88073c30 2023-03-03T18:17:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.839+0000 7fbb90605700 1 -- 172.21.15.7:0/2865659770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb88152010 con 0x7fbb88073c30 2023-03-03T18:17:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.840+0000 7fbb7effd700 1 -- 172.21.15.7:0/2865659770 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbb8400f050 con 0x7fbb88073c30 2023-03-03T18:17:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.840+0000 7fbb7effd700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb6c0682f0 0x7fbb6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.840+0000 7fbb7effd700 1 -- 172.21.15.7:0/2865659770 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbb84021070 con 0x7fbb88073c30 2023-03-03T18:17:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.840+0000 7fbb8e3a1700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb6c0682f0 0x7fbb6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.841+0000 7fbb8e3a1700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb6c0682f0 0x7fbb6c06a7a0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fbb780098f0 tx=0x7fbb78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:17:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:34.845+0000 7fbb7effd700 1 -- 172.21.15.7:0/2865659770 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbb84053e80 con 0x7fbb88073c30 2023-03-03T18:17:35.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.082+0000 7fbb90605700 1 -- 172.21.15.7:0/2865659770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb88074bb0 con 0x7fbb88073c30 2023-03-03T18:17:35.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.083+0000 7fbb7effd700 1 -- 172.21.15.7:0/2865659770 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbb840574a0 con 0x7fbb88073c30 2023-03-03T18:17:35.083 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:17:35.083 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:17:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.084+0000 7fbb90605700 1 -- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb6c0682f0 msgr2=0x7fbb6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.084+0000 7fbb90605700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb6c0682f0 0x7fbb6c06a7a0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fbb780098f0 tx=0x7fbb78008040 comp rx=0 tx=0).stop 2023-03-03T18:17:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.084+0000 7fbb90605700 1 -- 172.21.15.7:0/2865659770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 msgr2=0x7fbb880c0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.084+0000 7fbb90605700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 0x7fbb880c0900 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7fbb840090b0 tx=0x7fbb84004750 comp rx=0 tx=0).stop 2023-03-03T18:17:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.084+0000 7fbb90605700 1 -- 172.21.15.7:0/2865659770 shutdown_connections 2023-03-03T18:17:35.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.084+0000 7fbb90605700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb88151ab0 0x7fbb880c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.084+0000 7fbb90605700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb8806c3e0 0x7fbb881e9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.085+0000 7fbb90605700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb6c0682f0 0x7fbb6c06a7a0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.085+0000 7fbb90605700 1 --2- 172.21.15.7:0/2865659770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb88073c30 0x7fbb880c0900 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.085+0000 7fbb90605700 1 -- 172.21.15.7:0/2865659770 >> 172.21.15.7:0/2865659770 conn(0x7fbb880a94d0 msgr2=0x7fbb88071640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:17:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.085+0000 7fbb90605700 1 -- 172.21.15.7:0/2865659770 shutdown_connections 2023-03-03T18:17:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:35.085+0000 7fbb90605700 1 -- 172.21.15.7:0/2865659770 wait complete. 2023-03-03T18:17:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:17:35.097 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before thrashing 2023-03-03T18:17:36.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:17:36.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:17:44.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:17:44.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.238+0000 7f05d5fd7700 1 Processor -- start 2023-03-03T18:17:44.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.238+0000 7f05d5fd7700 1 -- start start 2023-03-03T18:17:44.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.238+0000 7f05d5fd7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05d014b020 0x7f05d00717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:44.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05d5fd7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f05d00aad20 0x7f05d0157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:44.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05d5fd7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f05d0071d20 0x7f05d01581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:44.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05d5fd7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d00728b0 con 0x7f05d0071d20 2023-03-03T18:17:44.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05d5fd7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d0155070 con 0x7f05d00aad20 2023-03-03T18:17:44.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05d5fd7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d006f5b0 con 0x7f05d014b020 2023-03-03T18:17:44.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05cffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f05d0071d20 0x7f05d01581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:44.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05cffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f05d0071d20 0x7f05d01581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36670/0 (socket says 172.21.15.7:36670) 2023-03-03T18:17:44.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05cffff700 1 -- 172.21.15.7:0/4007694245 learned_addr learned my addr 172.21.15.7:0/4007694245 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:17:44.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05cf7fe700 1 --2- 172.21.15.7:0/4007694245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05d014b020 0x7f05d00717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:44.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05ceffd700 1 --2- 172.21.15.7:0/4007694245 >> v2:172.21.15.143:3300/0 conn(0x7f05d00aad20 0x7f05d0157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:44.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05cffff700 1 -- 172.21.15.7:0/4007694245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05d014b020 msgr2=0x7f05d00717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:17:44.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05cffff700 1 --2- 172.21.15.7:0/4007694245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05d014b020 0x7f05d00717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05cffff700 1 -- 172.21.15.7:0/4007694245 >> v2:172.21.15.143:3300/0 conn(0x7f05d00aad20 msgr2=0x7f05d0157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:17:44.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05cffff700 1 --2- 172.21.15.7:0/4007694245 >> v2:172.21.15.143:3300/0 conn(0x7f05d00aad20 0x7f05d0157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05cffff700 1 -- 172.21.15.7:0/4007694245 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d006f890 con 0x7f05d0071d20 2023-03-03T18:17:44.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05cffff700 1 --2- 172.21.15.7:0/4007694245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d0071d20 0x7f05d01581e0 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f05d00c28b0 tx=0x7f05c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d058aa2798d02f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:17:44.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05ce7fc700 1 -- 172.21.15.7:0/4007694245 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c800e070 con 0x7f05d0071d20 2023-03-03T18:17:44.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05ce7fc700 1 -- 172.21.15.7:0/4007694245 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c800b910 con 0x7f05d0071d20 2023-03-03T18:17:44.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.239+0000 7f05ce7fc700 1 -- 172.21.15.7:0/4007694245 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c800bcc0 con 0x7f05d0071d20 2023-03-03T18:17:44.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.240+0000 7f05d5fd7700 1 -- 172.21.15.7:0/4007694245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d0071d20 msgr2=0x7f05d01581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:44.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.240+0000 7f05d5fd7700 1 --2- 172.21.15.7:0/4007694245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d0071d20 0x7f05d01581e0 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f05d00c28b0 tx=0x7f05c800b330 comp rx=0 tx=0).stop 2023-03-03T18:17:44.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.240+0000 7f05d5fd7700 1 -- 172.21.15.7:0/4007694245 shutdown_connections 2023-03-03T18:17:44.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.240+0000 7f05d5fd7700 1 --2- 172.21.15.7:0/4007694245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d0071d20 0x7f05d01581e0 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.240+0000 7f05d5fd7700 1 --2- 172.21.15.7:0/4007694245 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05d014b020 0x7f05d00717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.240+0000 7f05d5fd7700 1 --2- 172.21.15.7:0/4007694245 >> v2:172.21.15.143:3300/0 conn(0x7f05d00aad20 0x7f05d0157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.240+0000 7f05d5fd7700 1 -- 172.21.15.7:0/4007694245 >> 172.21.15.7:0/4007694245 conn(0x7f05d00c8900 msgr2=0x7f05d00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:17:44.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.240+0000 7f05d5fd7700 1 -- 172.21.15.7:0/4007694245 shutdown_connections 2023-03-03T18:17:44.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.240+0000 7f05d5fd7700 1 -- 172.21.15.7:0/4007694245 wait complete. 2023-03-03T18:17:44.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.241+0000 7f05d5fd7700 1 Processor -- start 2023-03-03T18:17:44.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.241+0000 7f05d5fd7700 1 -- start start 2023-03-03T18:17:44.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05d5fd7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05d0071d20 0x7f05d0148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:44.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05d5fd7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d00aad20 0x7f05d014d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:44.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05d5fd7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d014b020 0x7f05d0147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:44.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05d5fd7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d014e070 con 0x7f05d014b020 2023-03-03T18:17:44.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05d5fd7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d014e310 con 0x7f05d00aad20 2023-03-03T18:17:44.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05d5fd7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d0151e10 con 0x7f05d0071d20 2023-03-03T18:17:44.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05cffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d014b020 0x7f05d0147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:44.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05cffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d014b020 0x7f05d0147b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36676/0 (socket says 172.21.15.7:36676) 2023-03-03T18:17:44.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05cffff700 1 -- 172.21.15.7:0/2020238568 learned_addr learned my addr 172.21.15.7:0/2020238568 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:17:44.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05cf7fe700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05d0071d20 0x7f05d0148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:44.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.242+0000 7f05ceffd700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d00aad20 0x7f05d014d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:44.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05cffff700 1 -- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05d0071d20 msgr2=0x7f05d0148ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:44.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05cffff700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05d0071d20 0x7f05d0148ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05cffff700 1 -- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d00aad20 msgr2=0x7f05d014d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:44.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05cffff700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d00aad20 0x7f05d014d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05cffff700 1 -- 172.21.15.7:0/2020238568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c8009b30 con 0x7f05d014b020 2023-03-03T18:17:44.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05cffff700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d014b020 0x7f05d0147b80 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7f05d00c2b10 tx=0x7f05c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:17:44.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05b7fff700 1 -- 172.21.15.7:0/2020238568 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c80099b0 con 0x7f05d014b020 2023-03-03T18:17:44.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05b7fff700 1 -- 172.21.15.7:0/2020238568 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c800ce80 con 0x7f05d014b020 2023-03-03T18:17:44.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05d5fd7700 1 -- 172.21.15.7:0/2020238568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d0155350 con 0x7f05d014b020 2023-03-03T18:17:44.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05b7fff700 1 -- 172.21.15.7:0/2020238568 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c8004490 con 0x7f05d014b020 2023-03-03T18:17:44.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.243+0000 7f05d5fd7700 1 -- 172.21.15.7:0/2020238568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d006fb70 con 0x7f05d014b020 2023-03-03T18:17:44.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.244+0000 7f05d5fd7700 1 -- 172.21.15.7:0/2020238568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d0158720 con 0x7f05d014b020 2023-03-03T18:17:44.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.244+0000 7f05b7fff700 1 -- 172.21.15.7:0/2020238568 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f05c80045f0 con 0x7f05d014b020 2023-03-03T18:17:44.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.245+0000 7f05b7fff700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05bc0682f0 0x7f05bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:17:44.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.245+0000 7f05b7fff700 1 -- 172.21.15.7:0/2020238568 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f05c8087350 con 0x7f05d014b020 2023-03-03T18:17:44.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.245+0000 7f05cf7fe700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05bc0682f0 0x7f05bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:17:44.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.246+0000 7f05cf7fe700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05bc0682f0 0x7f05bc06a7a0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f05c0009be0 tx=0x7f05c0009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:17:44.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.249+0000 7f05b7fff700 1 -- 172.21.15.7:0/2020238568 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f05c80537a0 con 0x7f05d014b020 2023-03-03T18:17:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.483+0000 7f05d5fd7700 1 -- 172.21.15.7:0/2020238568 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05d01522d0 con 0x7f05d014b020 2023-03-03T18:17:44.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.483+0000 7f05b7fff700 1 -- 172.21.15.7:0/2020238568 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f05c801f090 con 0x7f05d014b020 2023-03-03T18:17:44.484 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:17:44.484 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:17:44.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 -- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05bc0682f0 msgr2=0x7f05bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:44.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05bc0682f0 0x7f05bc06a7a0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f05c0009be0 tx=0x7f05c0009210 comp rx=0 tx=0).stop 2023-03-03T18:17:44.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 -- 172.21.15.7:0/2020238568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d014b020 msgr2=0x7f05d0147b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:17:44.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d014b020 0x7f05d0147b80 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7f05d00c2b10 tx=0x7f05c8004ad0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 -- 172.21.15.7:0/2020238568 shutdown_connections 2023-03-03T18:17:44.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f05bc0682f0 0x7f05bc06a7a0 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05d014b020 0x7f05d0147b80 unknown :-1 s=CLOSED pgs=3162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f05d0071d20 0x7f05d0148ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 --2- 172.21.15.7:0/2020238568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05d00aad20 0x7f05d014d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:17:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 -- 172.21.15.7:0/2020238568 >> 172.21.15.7:0/2020238568 conn(0x7f05d00c8900 msgr2=0x7f05d00acac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:17:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 -- 172.21.15.7:0/2020238568 shutdown_connections 2023-03-03T18:17:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:17:44.485+0000 7f05d5fd7700 1 -- 172.21.15.7:0/2020238568 wait complete. 2023-03-03T18:17:44.487 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:17:44.498 INFO:tasks.mds_thrash.fs.[a]:waiting for 82 secs before thrashing 2023-03-03T18:18:06.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:18:06.392 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:18:36.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:18:36.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:18:56.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:18:56.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.383+0000 7f78c82bd700 1 Processor -- start 2023-03-03T18:18:56.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.383+0000 7f78c82bd700 1 -- start start 2023-03-03T18:18:56.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c82bd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f78c013a0f0 0x7f78c00c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:18:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c82bd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f78c00c1da0 0x7f78c006d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:18:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c82bd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c006de90 con 0x7f78c01586c0 2023-03-03T18:18:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c82bd700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c0154840 con 0x7f78c013a0f0 2023-03-03T18:18:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c82bd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c0154560 con 0x7f78c00c1da0 2023-03-03T18:18:56.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c6059700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f78c013a0f0 0x7f78c00c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:18:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c6059700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f78c013a0f0 0x7f78c00c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36716/0 (socket says 172.21.15.7:36716) 2023-03-03T18:18:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c6059700 1 -- 172.21.15.7:0/938614163 learned_addr learned my addr 172.21.15.7:0/938614163 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:18:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c685a700 1 --2- 172.21.15.7:0/938614163 >> v2:172.21.15.143:3300/0 conn(0x7f78c00c1da0 0x7f78c006d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:18:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c6059700 1 -- 172.21.15.7:0/938614163 >> v2:172.21.15.143:3300/0 conn(0x7f78c00c1da0 msgr2=0x7f78c006d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:18:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c6059700 1 --2- 172.21.15.7:0/938614163 >> v2:172.21.15.143:3300/0 conn(0x7f78c00c1da0 0x7f78c006d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:18:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c6059700 1 -- 172.21.15.7:0/938614163 >> v1:172.21.15.143:6789/0 conn(0x7f78c01586c0 legacy=0x7f78c006d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:18:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.384+0000 7f78c6059700 1 -- 172.21.15.7:0/938614163 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c0157be0 con 0x7f78c013a0f0 2023-03-03T18:18:56.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c6059700 1 --2- 172.21.15.7:0/938614163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c013a0f0 0x7f78c00c07d0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7f78c0070000 tx=0x7f78bc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8eff4f63fe490f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:18:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c5057700 1 -- 172.21.15.7:0/938614163 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc011830 con 0x7f78c013a0f0 2023-03-03T18:18:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c5057700 1 -- 172.21.15.7:0/938614163 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78bc011990 con 0x7f78c013a0f0 2023-03-03T18:18:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c5057700 1 -- 172.21.15.7:0/938614163 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc011d20 con 0x7f78c013a0f0 2023-03-03T18:18:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c82bd700 1 -- 172.21.15.7:0/938614163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c013a0f0 msgr2=0x7f78c00c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:18:56.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c82bd700 1 --2- 172.21.15.7:0/938614163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c013a0f0 0x7f78c00c07d0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7f78c0070000 tx=0x7f78bc009bd0 comp rx=0 tx=0).stop 2023-03-03T18:18:56.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c82bd700 1 -- 172.21.15.7:0/938614163 shutdown_connections 2023-03-03T18:18:56.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c82bd700 1 --2- 172.21.15.7:0/938614163 >> v2:172.21.15.143:3300/0 conn(0x7f78c00c1da0 0x7f78c006d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:18:56.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c82bd700 1 --2- 172.21.15.7:0/938614163 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c013a0f0 0x7f78c00c07d0 unknown :-1 s=CLOSED pgs=3163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:18:56.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c82bd700 1 -- 172.21.15.7:0/938614163 >> 172.21.15.7:0/938614163 conn(0x7f78c006c110 msgr2=0x7f78c006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:18:56.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c82bd700 1 -- 172.21.15.7:0/938614163 shutdown_connections 2023-03-03T18:18:56.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.385+0000 7f78c82bd700 1 -- 172.21.15.7:0/938614163 wait complete. 2023-03-03T18:18:56.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.387+0000 7f78c82bd700 1 Processor -- start 2023-03-03T18:18:56.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.387+0000 7f78c82bd700 1 -- start start 2023-03-03T18:18:56.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c82bd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c00c1da0 0x7f78c00c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:18:56.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c82bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78c013a0f0 0x7f78c00c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:18:56.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c82bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78c01586c0 0x7f78c00702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:18:56.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c82bd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c00c5200 con 0x7f78c00c1da0 2023-03-03T18:18:56.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c82bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c01eadb0 con 0x7f78c01586c0 2023-03-03T18:18:56.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c82bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c00c77d0 con 0x7f78c013a0f0 2023-03-03T18:18:56.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c6059700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c00c1da0 0x7f78c00c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:18:56.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c6059700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c00c1da0 0x7f78c00c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36722/0 (socket says 172.21.15.7:36722) 2023-03-03T18:18:56.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c6059700 1 -- 172.21.15.7:0/1061679871 learned_addr learned my addr 172.21.15.7:0/1061679871 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:18:56.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c5858700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78c013a0f0 0x7f78c00c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:18:56.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c685a700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78c01586c0 0x7f78c00702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:18:56.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c6059700 1 -- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78c013a0f0 msgr2=0x7f78c00c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:18:56.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c6059700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78c013a0f0 0x7f78c00c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:18:56.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c6059700 1 -- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78c01586c0 msgr2=0x7f78c00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:18:56.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c6059700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78c01586c0 0x7f78c00702b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:18:56.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c6059700 1 -- 172.21.15.7:0/1061679871 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc009550 con 0x7f78c00c1da0 2023-03-03T18:18:56.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.388+0000 7f78c685a700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78c01586c0 0x7f78c00702b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:18:56.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.389+0000 7f78c6059700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c00c1da0 0x7f78c00c6d50 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7f78bc000c00 tx=0x7f78bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:18:56.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.389+0000 7f78b6ffd700 1 -- 172.21.15.7:0/1061679871 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc0036a0 con 0x7f78c00c1da0 2023-03-03T18:18:56.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.389+0000 7f78b6ffd700 1 -- 172.21.15.7:0/1061679871 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78bc003820 con 0x7f78c00c1da0 2023-03-03T18:18:56.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.389+0000 7f78c82bd700 1 -- 172.21.15.7:0/1061679871 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c01581a0 con 0x7f78c00c1da0 2023-03-03T18:18:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.389+0000 7f78b6ffd700 1 -- 172.21.15.7:0/1061679871 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc003bd0 con 0x7f78c00c1da0 2023-03-03T18:18:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.389+0000 7f78c82bd700 1 -- 172.21.15.7:0/1061679871 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c0157ec0 con 0x7f78c00c1da0 2023-03-03T18:18:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.390+0000 7f78c82bd700 1 -- 172.21.15.7:0/1061679871 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c006e1a0 con 0x7f78c00c1da0 2023-03-03T18:18:56.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.390+0000 7f78b6ffd700 1 -- 172.21.15.7:0/1061679871 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f78bc00f050 con 0x7f78c00c1da0 2023-03-03T18:18:56.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.391+0000 7f78b6ffd700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78a40682f0 0x7f78a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:18:56.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.391+0000 7f78b6ffd700 1 -- 172.21.15.7:0/1061679871 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f78bc087850 con 0x7f78c00c1da0 2023-03-03T18:18:56.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.391+0000 7f78c5858700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78a40682f0 0x7f78a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:18:56.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.391+0000 7f78c5858700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78a40682f0 0x7f78a406a7a0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f78c0139120 tx=0x7f78b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:18:56.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.395+0000 7f78b6ffd700 1 -- 172.21.15.7:0/1061679871 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f78bc053b00 con 0x7f78c00c1da0 2023-03-03T18:18:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.633+0000 7f78c82bd700 1 -- 172.21.15.7:0/1061679871 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78c0152130 con 0x7f78c00c1da0 2023-03-03T18:18:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.634+0000 7f78b6ffd700 1 -- 172.21.15.7:0/1061679871 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f78bc057120 con 0x7f78c00c1da0 2023-03-03T18:18:56.634 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:18:56.634 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:18:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 -- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78a40682f0 msgr2=0x7f78a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:18:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78a40682f0 0x7f78a406a7a0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f78c0139120 tx=0x7f78b0009040 comp rx=0 tx=0).stop 2023-03-03T18:18:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 -- 172.21.15.7:0/1061679871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c00c1da0 msgr2=0x7f78c00c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:18:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c00c1da0 0x7f78c00c6d50 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7f78bc000c00 tx=0x7f78bc004750 comp rx=0 tx=0).stop 2023-03-03T18:18:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 -- 172.21.15.7:0/1061679871 shutdown_connections 2023-03-03T18:18:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78c01586c0 0x7f78c00702b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:18:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78a40682f0 0x7f78a406a7a0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:18:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78c00c1da0 0x7f78c00c6d50 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:18:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 --2- 172.21.15.7:0/1061679871 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78c013a0f0 0x7f78c00c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:18:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 -- 172.21.15.7:0/1061679871 >> 172.21.15.7:0/1061679871 conn(0x7f78c006c110 msgr2=0x7f78c006d220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:18:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 -- 172.21.15.7:0/1061679871 shutdown_connections 2023-03-03T18:18:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:18:56.636+0000 7f78c82bd700 1 -- 172.21.15.7:0/1061679871 wait complete. 2023-03-03T18:18:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:18:56.648 INFO:tasks.mds_thrash.fs.[b]:waiting for 57 secs before thrashing 2023-03-03T18:19:06.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:19:06.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:19:06.545 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:19:06.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.650+0000 7fb69fe27700 1 Processor -- start 2023-03-03T18:19:06.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.650+0000 7fb69fe27700 1 -- start start 2023-03-03T18:19:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.651+0000 7fb69fe27700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb6980c2930 0x7fb698157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.651+0000 7fb69fe27700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69806d430 con 0x7fb69806c480 2023-03-03T18:19:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.651+0000 7fb69fe27700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6981586e0 con 0x7fb698151d40 2023-03-03T18:19:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.651+0000 7fb69fe27700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb69806c2a0 con 0x7fb6980c2930 2023-03-03T18:19:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.651+0000 7fb69e3c4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb69806c480 0x7fb698157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58948/0 (socket says 172.21.15.7:58948) 2023-03-03T18:19:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.651+0000 7fb69e3c4700 1 -- 172.21.15.7:0/2059579695 learned_addr learned my addr 172.21.15.7:0/2059579695 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:19:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.651+0000 7fb69d3c2700 1 --2- 172.21.15.7:0/2059579695 >> v2:172.21.15.143:3300/0 conn(0x7fb6980c2930 0x7fb698157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705868954 0 0) 0x7fb69806d430 con 0x7fb69806c480 2023-03-03T18:19:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6840035b0 con 0x7fb69806c480 2023-03-03T18:19:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2214730443 0 0) 0x7fb6840035b0 con 0x7fb69806c480 2023-03-03T18:19:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 >> v2:172.21.15.143:3300/0 conn(0x7fb6980c2930 msgr2=0x7fb698157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:19:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 --2- 172.21.15.7:0/2059579695 >> v2:172.21.15.143:3300/0 conn(0x7fb6980c2930 0x7fb698157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 >> v1:172.21.15.143:6789/0 conn(0x7fb698151d40 legacy=0x7fb6980736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69806cb00 con 0x7fb69806c480 2023-03-03T18:19:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb6900019f0 con 0x7fb69806c480 2023-03-03T18:19:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 >> v1:172.21.15.7:6789/0 conn(0x7fb69806c480 legacy=0x7fb698157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb684003a70 0x7fb684003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb684004850 0x7fb684004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb684005320 0x7fb68400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69806c2a0 con 0x7fb684005320 2023-03-03T18:19:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68400c250 con 0x7fb684004850 2023-03-03T18:19:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68400c410 con 0x7fb684003a70 2023-03-03T18:19:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69e3c4700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb684005320 0x7fb68400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69d3c2700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb684004850 0x7fb684004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69dbc3700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb684003a70 0x7fb684003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:06.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69e3c4700 1 -- 172.21.15.7:0/2059579695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb684003a70 msgr2=0x7fb684003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:19:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69e3c4700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb684003a70 0x7fb684003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69e3c4700 1 -- 172.21.15.7:0/2059579695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb684004850 msgr2=0x7fb684004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:19:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69e3c4700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb684004850 0x7fb684004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.652+0000 7fb69e3c4700 1 -- 172.21.15.7:0/2059579695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6840035b0 con 0x7fb684005320 2023-03-03T18:19:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69e3c4700 1 -- 172.21.15.7:0/2059579695 reap_dead start 2023-03-03T18:19:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69e3c4700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb684005320 0x7fb68400ba10 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7fb6900013d0 tx=0x7fb684005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671d807067426409 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:19:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690012600 con 0x7fb684005320 2023-03-03T18:19:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6900127a0 con 0x7fb684005320 2023-03-03T18:19:06.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69cbc1700 1 -- 172.21.15.7:0/2059579695 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690012b50 con 0x7fb684005320 2023-03-03T18:19:06.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69fe27700 1 -- 172.21.15.7:0/2059579695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb684005320 msgr2=0x7fb68400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:06.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69fe27700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb684005320 0x7fb68400ba10 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7fb6900013d0 tx=0x7fb684005880 comp rx=0 tx=0).stop 2023-03-03T18:19:06.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69fe27700 1 -- 172.21.15.7:0/2059579695 shutdown_connections 2023-03-03T18:19:06.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69fe27700 1 --2- 172.21.15.7:0/2059579695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb684005320 0x7fb68400ba10 unknown :-1 s=CLOSED pgs=3166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69fe27700 1 -- 172.21.15.7:0/2059579695 >> 172.21.15.7:0/2059579695 conn(0x7fb6980a94d0 msgr2=0x7fb6980a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:19:06.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69fe27700 1 -- 172.21.15.7:0/2059579695 shutdown_connections 2023-03-03T18:19:06.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.653+0000 7fb69fe27700 1 -- 172.21.15.7:0/2059579695 wait complete. 2023-03-03T18:19:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69fe27700 1 Processor -- start 2023-03-03T18:19:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69fe27700 1 -- start start 2023-03-03T18:19:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69fe27700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb684005320 0x7fb6980c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69fe27700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb698157a60 0x7fb6981ed3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:06.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69fe27700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6981ddaf0 0x7fb6981ed8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69fe27700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6981ecf20 con 0x7fb698157a60 2023-03-03T18:19:06.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69fe27700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6981ede30 con 0x7fb684005320 2023-03-03T18:19:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69fe27700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb698156eb0 con 0x7fb6981ddaf0 2023-03-03T18:19:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69d3c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb698157a60 0x7fb6981ed3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69d3c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb698157a60 0x7fb6981ed3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36746/0 (socket says 172.21.15.7:36746) 2023-03-03T18:19:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.655+0000 7fb69d3c2700 1 -- 172.21.15.7:0/2729919746 learned_addr learned my addr 172.21.15.7:0/2729919746 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:19:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb69dbc3700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb684005320 0x7fb6980c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb69e3c4700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6981ddaf0 0x7fb6981ed8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:06.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb69d3c2700 1 -- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6981ddaf0 msgr2=0x7fb6981ed8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb69d3c2700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6981ddaf0 0x7fb6981ed8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb69d3c2700 1 -- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb684005320 msgr2=0x7fb6980c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb69d3c2700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb684005320 0x7fb6980c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb69d3c2700 1 -- 172.21.15.7:0/2729919746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69806d0c0 con 0x7fb698157a60 2023-03-03T18:19:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb69d3c2700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb698157a60 0x7fb6981ed3b0 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7fb6940119d0 tx=0x7fb694011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:19:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb68e7fc700 1 -- 172.21.15.7:0/2729919746 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb694011e70 con 0x7fb698157a60 2023-03-03T18:19:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb68e7fc700 1 -- 172.21.15.7:0/2729919746 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69400f940 con 0x7fb698157a60 2023-03-03T18:19:06.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb69fe27700 1 -- 172.21.15.7:0/2729919746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69806cde0 con 0x7fb698157a60 2023-03-03T18:19:06.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb68e7fc700 1 -- 172.21.15.7:0/2729919746 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69400fcd0 con 0x7fb698157a60 2023-03-03T18:19:06.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.656+0000 7fb69fe27700 1 -- 172.21.15.7:0/2729919746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69806bea0 con 0x7fb698157a60 2023-03-03T18:19:06.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.657+0000 7fb69fe27700 1 -- 172.21.15.7:0/2729919746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb698074bb0 con 0x7fb698157a60 2023-03-03T18:19:06.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.658+0000 7fb68e7fc700 1 -- 172.21.15.7:0/2729919746 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb69401d400 con 0x7fb698157a60 2023-03-03T18:19:06.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.658+0000 7fb68e7fc700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb67c068320 0x7fb67c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:06.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.658+0000 7fb68e7fc700 1 -- 172.21.15.7:0/2729919746 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb694086860 con 0x7fb698157a60 2023-03-03T18:19:06.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.658+0000 7fb69dbc3700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb67c068320 0x7fb67c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.659+0000 7fb69dbc3700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb67c068320 0x7fb67c06a7d0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fb688000c00 tx=0x7fb688009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:19:06.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.662+0000 7fb68e7fc700 1 -- 172.21.15.7:0/2729919746 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb694052ba0 con 0x7fb698157a60 2023-03-03T18:19:06.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.899+0000 7fb69fe27700 1 -- 172.21.15.7:0/2729919746 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6981ed060 con 0x7fb698157a60 2023-03-03T18:19:06.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.900+0000 7fb68e7fc700 1 -- 172.21.15.7:0/2729919746 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb6940561c0 con 0x7fb698157a60 2023-03-03T18:19:06.901 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:19:06.901 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:19:06.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 -- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb67c068320 msgr2=0x7fb67c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:06.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb67c068320 0x7fb67c06a7d0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fb688000c00 tx=0x7fb688009040 comp rx=0 tx=0).stop 2023-03-03T18:19:06.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 -- 172.21.15.7:0/2729919746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb698157a60 msgr2=0x7fb6981ed3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:06.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb698157a60 0x7fb6981ed3b0 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7fb6940119d0 tx=0x7fb694011ce0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 -- 172.21.15.7:0/2729919746 shutdown_connections 2023-03-03T18:19:06.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb6981ddaf0 0x7fb6981ed8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb684005320 0x7fb6980c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb67c068320 0x7fb67c06a7d0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 --2- 172.21.15.7:0/2729919746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb698157a60 0x7fb6981ed3b0 unknown :-1 s=CLOSED pgs=3167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:06.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 -- 172.21.15.7:0/2729919746 >> 172.21.15.7:0/2729919746 conn(0x7fb6980a94d0 msgr2=0x7fb69814eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:19:06.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 -- 172.21.15.7:0/2729919746 shutdown_connections 2023-03-03T18:19:06.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:06.902+0000 7fb69fe27700 1 -- 172.21.15.7:0/2729919746 wait complete. 2023-03-03T18:19:06.905 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:19:06.915 INFO:tasks.mds_thrash.fs.[a]:waiting for 48 secs before thrashing 2023-03-03T18:19:36.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:19:36.674 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:19:53.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:19:53.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.786+0000 7fe270229700 1 Processor -- start 2023-03-03T18:19:53.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.786+0000 7fe270229700 1 -- start start 2023-03-03T18:19:53.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe270229700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe268072470 0x7fe26806b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:53.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe270229700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe268070360 con 0x7fe26814b460 2023-03-03T18:19:53.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe270229700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26806a5e0 con 0x7fe268071990 2023-03-03T18:19:53.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe270229700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2681521a0 con 0x7fe268072470 2023-03-03T18:19:53.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe26d7c4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe26814b460 0x7fe26806aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58996/0 (socket says 172.21.15.7:58996) 2023-03-03T18:19:53.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe26d7c4700 1 -- 172.21.15.7:0/3612036878 learned_addr learned my addr 172.21.15.7:0/3612036878 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:19:53.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660781098 0 0) 0x7fe268070360 con 0x7fe26814b460 2023-03-03T18:19:53.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2540035b0 con 0x7fe26814b460 2023-03-03T18:19:53.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe26e7c6700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe268072470 0x7fe26806b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:53.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1255536073 0 0) 0x7fe2540035b0 con 0x7fe26814b460 2023-03-03T18:19:53.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe268072470 msgr2=0x7fe26806b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:19:53.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.787+0000 7fe26cfc3700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe268072470 0x7fe26806b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:53.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 >> v1:172.21.15.143:6789/0 conn(0x7fe268071990 legacy=0x7fe268071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:53.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2680c38b0 con 0x7fe26814b460 2023-03-03T18:19:53.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe264001670 con 0x7fe26814b460 2023-03-03T18:19:53.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 >> v1:172.21.15.7:6789/0 conn(0x7fe26814b460 legacy=0x7fe26806aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:53.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe268072470 0x7fe254003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:53.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 0x7fe254004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:53.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe254004d90 0x7fe254005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:53.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2681521a0 con 0x7fe254004020 2023-03-03T18:19:53.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25400bc00 con 0x7fe254004d90 2023-03-03T18:19:53.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25400dfd0 con 0x7fe268072470 2023-03-03T18:19:53.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26d7c4700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 0x7fe254004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:53.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26dfc5700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe268072470 0x7fe254003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:53.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26e7c6700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe254004d90 0x7fe254005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:53.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26d7c4700 1 -- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe268072470 msgr2=0x7fe254003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:19:53.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26d7c4700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe268072470 0x7fe254003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:53.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26d7c4700 1 -- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe254004d90 msgr2=0x7fe254005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:19:53.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26d7c4700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe254004d90 0x7fe254005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:53.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26d7c4700 1 -- 172.21.15.7:0/3612036878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2540035b0 con 0x7fe254004020 2023-03-03T18:19:53.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26e7c6700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe254004d90 0x7fe254005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:19:53.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26d7c4700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 0x7fe254004850 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7fe26400c210 tx=0x7fe254005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76330ac6494a098 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:19:53.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264007c10 con 0x7fe254004020 2023-03-03T18:19:53.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26400cdb0 con 0x7fe254004020 2023-03-03T18:19:53.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.788+0000 7fe26cfc3700 1 -- 172.21.15.7:0/3612036878 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264005ba0 con 0x7fe254004020 2023-03-03T18:19:53.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.789+0000 7fe270229700 1 -- 172.21.15.7:0/3612036878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 msgr2=0x7fe254004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:53.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.789+0000 7fe270229700 1 --2- 172.21.15.7:0/3612036878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 0x7fe254004850 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7fe26400c210 tx=0x7fe254005880 comp rx=0 tx=0).stop 2023-03-03T18:19:53.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.789+0000 7fe26e7c6700 1 -- 172.21.15.7:0/3612036878 reap_dead start 2023-03-03T18:19:53.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.789+0000 7fe270229700 1 -- 172.21.15.7:0/3612036878 shutdown_connections 2023-03-03T18:19:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.789+0000 7fe270229700 1 -- 172.21.15.7:0/3612036878 >> 172.21.15.7:0/3612036878 conn(0x7fe2680a94d0 msgr2=0x7fe2680a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:19:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.789+0000 7fe270229700 1 -- 172.21.15.7:0/3612036878 shutdown_connections 2023-03-03T18:19:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.789+0000 7fe270229700 1 -- 172.21.15.7:0/3612036878 wait complete. 2023-03-03T18:19:53.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.791+0000 7fe270229700 1 Processor -- start 2023-03-03T18:19:53.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.791+0000 7fe270229700 1 -- start start 2023-03-03T18:19:53.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe270229700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 0x7fe2681502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:53.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe270229700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2681dcb40 0x7fe2681dcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:53.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe270229700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2681dd4f0 0x7fe268144870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:53.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe270229700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe268145870 con 0x7fe254004020 2023-03-03T18:19:53.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe270229700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2681dc570 con 0x7fe2681dd4f0 2023-03-03T18:19:53.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe270229700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2681da980 con 0x7fe2681dcb40 2023-03-03T18:19:53.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26dfc5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 0x7fe2681502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:53.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26dfc5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 0x7fe2681502b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36794/0 (socket says 172.21.15.7:36794) 2023-03-03T18:19:53.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26dfc5700 1 -- 172.21.15.7:0/3682324491 learned_addr learned my addr 172.21.15.7:0/3682324491 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:19:53.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26dfc5700 1 -- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2681dcb40 msgr2=0x7fe2681dcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:53.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26e7c6700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2681dd4f0 0x7fe268144870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:53.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26d7c4700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2681dcb40 0x7fe2681dcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:53.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26dfc5700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2681dcb40 0x7fe2681dcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:53.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26dfc5700 1 -- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2681dd4f0 msgr2=0x7fe268144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:53.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26dfc5700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2681dd4f0 0x7fe268144870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:53.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26dfc5700 1 -- 172.21.15.7:0/3682324491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2680c3b50 con 0x7fe254004020 2023-03-03T18:19:53.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26e7c6700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2681dd4f0 0x7fe268144870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:19:53.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26d7c4700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2681dcb40 0x7fe2681dcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:19:53.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe26dfc5700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 0x7fe2681502b0 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7fe25800f820 tx=0x7fe25800fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:19:53.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe25e7fc700 1 -- 172.21.15.7:0/3682324491 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2580052e0 con 0x7fe254004020 2023-03-03T18:19:53.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe25e7fc700 1 -- 172.21.15.7:0/3682324491 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe258005440 con 0x7fe254004020 2023-03-03T18:19:53.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe270229700 1 -- 172.21.15.7:0/3682324491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26806a2b0 con 0x7fe254004020 2023-03-03T18:19:53.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.792+0000 7fe25e7fc700 1 -- 172.21.15.7:0/3682324491 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2580038c0 con 0x7fe254004020 2023-03-03T18:19:53.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.793+0000 7fe270229700 1 -- 172.21.15.7:0/3682324491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe268151c00 con 0x7fe254004020 2023-03-03T18:19:53.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.794+0000 7fe270229700 1 -- 172.21.15.7:0/3682324491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe268074bb0 con 0x7fe254004020 2023-03-03T18:19:53.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.794+0000 7fe25e7fc700 1 -- 172.21.15.7:0/3682324491 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe258003a20 con 0x7fe254004020 2023-03-03T18:19:53.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.794+0000 7fe25e7fc700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe24c0682f0 0x7fe24c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:53.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.794+0000 7fe25e7fc700 1 -- 172.21.15.7:0/3682324491 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe258086130 con 0x7fe254004020 2023-03-03T18:19:53.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.795+0000 7fe26d7c4700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe24c0682f0 0x7fe24c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:53.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.795+0000 7fe26d7c4700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe24c0682f0 0x7fe24c06a7a0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fe2681510b0 tx=0x7fe264007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:19:53.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:53.799+0000 7fe25e7fc700 1 -- 172.21.15.7:0/3682324491 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe2580524d0 con 0x7fe254004020 2023-03-03T18:19:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.039+0000 7fe270229700 1 -- 172.21.15.7:0/3682324491 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe268154460 con 0x7fe254004020 2023-03-03T18:19:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.040+0000 7fe25e7fc700 1 -- 172.21.15.7:0/3682324491 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe258003cb0 con 0x7fe254004020 2023-03-03T18:19:54.040 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:19:54.040 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:19:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 -- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe24c0682f0 msgr2=0x7fe24c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:54.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe24c0682f0 0x7fe24c06a7a0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fe2681510b0 tx=0x7fe264007000 comp rx=0 tx=0).stop 2023-03-03T18:19:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 -- 172.21.15.7:0/3682324491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 msgr2=0x7fe2681502b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 0x7fe2681502b0 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7fe25800f820 tx=0x7fe25800fbe0 comp rx=0 tx=0).stop 2023-03-03T18:19:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 -- 172.21.15.7:0/3682324491 shutdown_connections 2023-03-03T18:19:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe2681dd4f0 0x7fe268144870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe24c0682f0 0x7fe24c06a7a0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe254004020 0x7fe2681502b0 unknown :-1 s=CLOSED pgs=3170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:54.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 --2- 172.21.15.7:0/3682324491 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe2681dcb40 0x7fe2681dcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 -- 172.21.15.7:0/3682324491 >> 172.21.15.7:0/3682324491 conn(0x7fe2680a94d0 msgr2=0x7fe2680c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:19:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 -- 172.21.15.7:0/3682324491 shutdown_connections 2023-03-03T18:19:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:54.042+0000 7fe270229700 1 -- 172.21.15.7:0/3682324491 wait complete. 2023-03-03T18:19:54.044 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:19:54.054 INFO:tasks.mds_thrash.fs.[b]:waiting for 77 secs before thrashing 2023-03-03T18:19:54.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:19:55.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.052+0000 7fdbd2118700 1 Processor -- start 2023-03-03T18:19:55.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.052+0000 7fdbd2118700 1 -- start start 2023-03-03T18:19:55.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.053+0000 7fdbd2118700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbcc06a2b0 0x7fdbcc06a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:55.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.053+0000 7fdbd2118700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdbcc0c5cb0 0x7fdbcc15cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:55.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.053+0000 7fdbd2118700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbcc1558f0 con 0x7fdbcc0a6370 2023-03-03T18:19:55.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.053+0000 7fdbd2118700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbcc06bab0 con 0x7fdbcc0c5cb0 2023-03-03T18:19:55.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.053+0000 7fdbd2118700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc0a6180 con 0x7fdbcc06a2b0 2023-03-03T18:19:55.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.053+0000 7fdbcaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdbcc0a6370 0x7fdbcc155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59016/0 (socket says 172.21.15.7:59016) 2023-03-03T18:19:55.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.053+0000 7fdbcaffd700 1 -- 172.21.15.7:0/2291858501 learned_addr learned my addr 172.21.15.7:0/2291858501 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:19:55.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278566078 0 0) 0x7fdbcc1558f0 con 0x7fdbcc0a6370 2023-03-03T18:19:55.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbcb7fe700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbcc06a2b0 0x7fdbcc06a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbb80035b0 con 0x7fdbcc0a6370 2023-03-03T18:19:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbcbfff700 1 --2- 172.21.15.7:0/2291858501 >> v2:172.21.15.143:3300/0 conn(0x7fdbcc0c5cb0 0x7fdbcc15cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1816580302 0 0) 0x7fdbb80035b0 con 0x7fdbcc0a6370 2023-03-03T18:19:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbcc06a2b0 msgr2=0x7fdbcc06a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:19:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbcc06a2b0 0x7fdbcc06a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 >> v2:172.21.15.143:3300/0 conn(0x7fdbcc0c5cb0 msgr2=0x7fdbcc15cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:19:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 --2- 172.21.15.7:0/2291858501 >> v2:172.21.15.143:3300/0 conn(0x7fdbcc0c5cb0 0x7fdbcc15cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:55.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbcc0a5ea0 con 0x7fdbcc0a6370 2023-03-03T18:19:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fdbb4001670 con 0x7fdbcc0a6370 2023-03-03T18:19:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 >> v1:172.21.15.7:6789/0 conn(0x7fdbcc0a6370 legacy=0x7fdbcc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbb8003a70 0x7fdbb8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbcc06a2b0 0x7fdbb8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbb8004d90 0x7fdbb8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc0a6180 con 0x7fdbb8003a70 2023-03-03T18:19:55.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb800ba10 con 0x7fdbb8004d90 2023-03-03T18:19:55.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb800bba0 con 0x7fdbcc06a2b0 2023-03-03T18:19:55.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbcb7fe700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbb8003a70 0x7fdbb8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:55.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbcaffd700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbcc06a2b0 0x7fdbb8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:55.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbcbfff700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbb8004d90 0x7fdbb8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:55.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbcb7fe700 1 -- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbcc06a2b0 msgr2=0x7fdbb8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:19:55.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.054+0000 7fdbcb7fe700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbcc06a2b0 0x7fdbb8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:55.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbcb7fe700 1 -- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbb8004d90 msgr2=0x7fdbb8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:19:55.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbcb7fe700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbb8004d90 0x7fdbb8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbcb7fe700 1 -- 172.21.15.7:0/2291858501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb80035b0 con 0x7fdbb8003a70 2023-03-03T18:19:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbcb7fe700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbb8003a70 0x7fdbb8003e80 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009b90 tx=0x7fdbb8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e7813b8e108895 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:19:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc00e6c0 con 0x7fdbb8003a70 2023-03-03T18:19:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc00e820 con 0x7fdbb8003a70 2023-03-03T18:19:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbca7fc700 1 -- 172.21.15.7:0/2291858501 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc00eb90 con 0x7fdbb8003a70 2023-03-03T18:19:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbd2118700 1 -- 172.21.15.7:0/2291858501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbb8003a70 msgr2=0x7fdbb8003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:55.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbd2118700 1 --2- 172.21.15.7:0/2291858501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbb8003a70 0x7fdbb8003e80 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009b90 tx=0x7fdbb8005880 comp rx=0 tx=0).stop 2023-03-03T18:19:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbcbfff700 1 -- 172.21.15.7:0/2291858501 reap_dead start 2023-03-03T18:19:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbd2118700 1 -- 172.21.15.7:0/2291858501 shutdown_connections 2023-03-03T18:19:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbd2118700 1 -- 172.21.15.7:0/2291858501 >> 172.21.15.7:0/2291858501 conn(0x7fdbcc0a9430 msgr2=0x7fdbcc0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:19:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbd2118700 1 -- 172.21.15.7:0/2291858501 shutdown_connections 2023-03-03T18:19:55.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.055+0000 7fdbd2118700 1 -- 172.21.15.7:0/2291858501 wait complete. 2023-03-03T18:19:55.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.057+0000 7fdbd2118700 1 Processor -- start 2023-03-03T18:19:55.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.057+0000 7fdbd2118700 1 -- start start 2023-03-03T18:19:55.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.057+0000 7fdbd2118700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbb8003a70 0x7fdbcc1507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbd2118700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbcc1411d0 0x7fdbcc1e82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbd2118700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbcc06ff50 0x7fdbcc14cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbd2118700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc150310 con 0x7fdbcc06ff50 2023-03-03T18:19:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbd2118700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc151c70 con 0x7fdbcc1411d0 2023-03-03T18:19:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbd2118700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc1e8810 con 0x7fdbb8003a70 2023-03-03T18:19:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbcc06ff50 0x7fdbcc14cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:55.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbcc06ff50 0x7fdbcc14cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36818/0 (socket says 172.21.15.7:36818) 2023-03-03T18:19:55.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcbfff700 1 -- 172.21.15.7:0/3306907818 learned_addr learned my addr 172.21.15.7:0/3306907818 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:19:55.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcaffd700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbcc1411d0 0x7fdbcc1e82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:55.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcb7fe700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbb8003a70 0x7fdbcc1507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:55.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcbfff700 1 -- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbb8003a70 msgr2=0x7fdbcc1507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:55.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcbfff700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbb8003a70 0x7fdbcc1507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcbfff700 1 -- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbcc1411d0 msgr2=0x7fdbcc1e82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcbfff700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbcc1411d0 0x7fdbcc1e82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcbfff700 1 -- 172.21.15.7:0/3306907818 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbcc153160 con 0x7fdbcc06ff50 2023-03-03T18:19:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcb7fe700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbb8003a70 0x7fdbcc1507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:19:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcaffd700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbcc1411d0 0x7fdbcc1e82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:19:55.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbcbfff700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbcc06ff50 0x7fdbcc14cfc0 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0c7be0 tx=0x7fdbc4011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:19:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbc3fff700 1 -- 172.21.15.7:0/3306907818 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc400e040 con 0x7fdbcc06ff50 2023-03-03T18:19:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbc3fff700 1 -- 172.21.15.7:0/3306907818 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc400f040 con 0x7fdbcc06ff50 2023-03-03T18:19:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbc3fff700 1 -- 172.21.15.7:0/3306907818 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc400fd50 con 0x7fdbcc06ff50 2023-03-03T18:19:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbd2118700 1 -- 172.21.15.7:0/3306907818 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbcc152e80 con 0x7fdbcc06ff50 2023-03-03T18:19:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.058+0000 7fdbd2118700 1 -- 172.21.15.7:0/3306907818 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbcc0c1500 con 0x7fdbcc06ff50 2023-03-03T18:19:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.059+0000 7fdbd2118700 1 -- 172.21.15.7:0/3306907818 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbcc074bb0 con 0x7fdbcc06ff50 2023-03-03T18:19:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.060+0000 7fdbc3fff700 1 -- 172.21.15.7:0/3306907818 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdbc4015430 con 0x7fdbcc06ff50 2023-03-03T18:19:55.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.060+0000 7fdbc3fff700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbac0682f0 0x7fdbac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:19:55.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.060+0000 7fdbc3fff700 1 -- 172.21.15.7:0/3306907818 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdbc4086b20 con 0x7fdbcc06ff50 2023-03-03T18:19:55.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.060+0000 7fdbcb7fe700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbac0682f0 0x7fdbac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:19:55.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.061+0000 7fdbcb7fe700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbac0682f0 0x7fdbac06a7a0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0c1380 tx=0x7fdbbc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:19:55.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.065+0000 7fdbc3fff700 1 -- 172.21.15.7:0/3306907818 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdbc4052e60 con 0x7fdbcc06ff50 2023-03-03T18:19:55.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.300+0000 7fdbd2118700 1 -- 172.21.15.7:0/3306907818 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbcc1520d0 con 0x7fdbcc06ff50 2023-03-03T18:19:55.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.301+0000 7fdbc3fff700 1 -- 172.21.15.7:0/3306907818 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdbc4056480 con 0x7fdbcc06ff50 2023-03-03T18:19:55.301 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:19:55.301 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:19:55.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 -- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbac0682f0 msgr2=0x7fdbac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:55.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbac0682f0 0x7fdbac06a7a0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0c1380 tx=0x7fdbbc008030 comp rx=0 tx=0).stop 2023-03-03T18:19:55.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 -- 172.21.15.7:0/3306907818 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbcc06ff50 msgr2=0x7fdbcc14cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:19:55.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbcc06ff50 0x7fdbcc14cfc0 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0c7be0 tx=0x7fdbc4011f80 comp rx=0 tx=0).stop 2023-03-03T18:19:55.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 -- 172.21.15.7:0/3306907818 shutdown_connections 2023-03-03T18:19:55.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbac0682f0 0x7fdbac06a7a0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:55.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbcc06ff50 0x7fdbcc14cfc0 unknown :-1 s=CLOSED pgs=3173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:55.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbb8003a70 0x7fdbcc1507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:55.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 --2- 172.21.15.7:0/3306907818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbcc1411d0 0x7fdbcc1e82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:19:55.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 -- 172.21.15.7:0/3306907818 >> 172.21.15.7:0/3306907818 conn(0x7fdbcc0a9430 msgr2=0x7fdbcc155c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:19:55.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 -- 172.21.15.7:0/3306907818 shutdown_connections 2023-03-03T18:19:55.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:19:55.303+0000 7fdbd2118700 1 -- 172.21.15.7:0/3306907818 wait complete. 2023-03-03T18:19:55.305 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:19:55.316 INFO:tasks.mds_thrash.fs.[a]:waiting for 64 secs before thrashing 2023-03-03T18:19:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T18:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T18:20:06.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:20:06.762 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:20:36.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:20:36.883 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:20:59.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:20:59.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.464+0000 7f09461c1700 1 Processor -- start 2023-03-03T18:20:59.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.464+0000 7f09461c1700 1 -- start start 2023-03-03T18:20:59.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f09461c1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f094014f860 0x7f09401562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:20:59.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f09461c1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f09400c27c0 0x7f09400736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:20:59.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f09461c1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0940155e60 con 0x7f09400c20d0 2023-03-03T18:20:59.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f09461c1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f094006d350 con 0x7f094014f860 2023-03-03T18:20:59.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f09461c1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09401558e0 con 0x7f09400c27c0 2023-03-03T18:20:59.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f093f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f094014f860 0x7f09401562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:20:59.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f093f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f094014f860 0x7f09401562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36862/0 (socket says 172.21.15.7:36862) 2023-03-03T18:20:59.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f093f7fe700 1 -- 172.21.15.7:0/2490742052 learned_addr learned my addr 172.21.15.7:0/2490742052 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:20:59.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f093ffff700 1 --2- 172.21.15.7:0/2490742052 >> v2:172.21.15.143:3300/0 conn(0x7f09400c27c0 0x7f09400736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:20:59.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f093f7fe700 1 -- 172.21.15.7:0/2490742052 >> v2:172.21.15.143:3300/0 conn(0x7f09400c27c0 msgr2=0x7f09400736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:20:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f093f7fe700 1 --2- 172.21.15.7:0/2490742052 >> v2:172.21.15.143:3300/0 conn(0x7f09400c27c0 0x7f09400736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f093f7fe700 1 -- 172.21.15.7:0/2490742052 >> v1:172.21.15.143:6789/0 conn(0x7f09400c20d0 legacy=0x7f0940152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:20:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.465+0000 7f093f7fe700 1 -- 172.21.15.7:0/2490742052 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094006cd90 con 0x7f094014f860 2023-03-03T18:20:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f093f7fe700 1 --2- 172.21.15.7:0/2490742052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 0x7f09401562f0 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 crypto rx=0x7f094006bcb0 tx=0x7f0930009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e007c1e8ba7bbea4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:20:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f093e7fc700 1 -- 172.21.15.7:0/2490742052 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930004230 con 0x7f094014f860 2023-03-03T18:20:59.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f093e7fc700 1 -- 172.21.15.7:0/2490742052 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0930004390 con 0x7f094014f860 2023-03-03T18:20:59.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f093e7fc700 1 -- 172.21.15.7:0/2490742052 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09300036a0 con 0x7f094014f860 2023-03-03T18:20:59.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f09461c1700 1 -- 172.21.15.7:0/2490742052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 msgr2=0x7f09401562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:20:59.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f09461c1700 1 --2- 172.21.15.7:0/2490742052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 0x7f09401562f0 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 crypto rx=0x7f094006bcb0 tx=0x7f0930009bd0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f09461c1700 1 -- 172.21.15.7:0/2490742052 shutdown_connections 2023-03-03T18:20:59.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f09461c1700 1 --2- 172.21.15.7:0/2490742052 >> v2:172.21.15.143:3300/0 conn(0x7f09400c27c0 0x7f09400736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f09461c1700 1 --2- 172.21.15.7:0/2490742052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 0x7f09401562f0 unknown :-1 s=CLOSED pgs=3174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f09461c1700 1 -- 172.21.15.7:0/2490742052 >> 172.21.15.7:0/2490742052 conn(0x7f09400a94d0 msgr2=0x7f09400a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:20:59.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f09461c1700 1 -- 172.21.15.7:0/2490742052 shutdown_connections 2023-03-03T18:20:59.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.466+0000 7f09461c1700 1 -- 172.21.15.7:0/2490742052 wait complete. 2023-03-03T18:20:59.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.468+0000 7f09461c1700 1 Processor -- start 2023-03-03T18:20:59.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.468+0000 7f09461c1700 1 -- start start 2023-03-03T18:20:59.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.468+0000 7f09461c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09400c20d0 0x7f09400c1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:20:59.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.468+0000 7f09461c1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c27c0 0x7f0940066220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:20:59.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.468+0000 7f09461c1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 0x7f09401e80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:20:59.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.468+0000 7f09461c1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09401eb6b0 con 0x7f094014f860 2023-03-03T18:20:59.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.468+0000 7f09461c1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09400bf230 con 0x7f09400c20d0 2023-03-03T18:20:59.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.468+0000 7f09461c1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09401eb950 con 0x7f09400c27c0 2023-03-03T18:20:59.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 0x7f09401e80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:20:59.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 0x7f09401e80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36872/0 (socket says 172.21.15.7:36872) 2023-03-03T18:20:59.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093ffff700 1 -- 172.21.15.7:0/2518023949 learned_addr learned my addr 172.21.15.7:0/2518023949 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:20:59.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093f7fe700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09400c20d0 0x7f09400c1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:20:59.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093effd700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c27c0 0x7f0940066220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:20:59.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093ffff700 1 -- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c27c0 msgr2=0x7f0940066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:20:59.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093ffff700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c27c0 0x7f0940066220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093ffff700 1 -- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09400c20d0 msgr2=0x7f09400c1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:20:59.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093ffff700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09400c20d0 0x7f09400c1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093ffff700 1 -- 172.21.15.7:0/2518023949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0930009550 con 0x7f094014f860 2023-03-03T18:20:59.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f093ffff700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 0x7f09401e80c0 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f0938007e40 tx=0x7f093800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:20:59.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f0937fff700 1 -- 172.21.15.7:0/2518023949 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093800ed30 con 0x7f094014f860 2023-03-03T18:20:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f0937fff700 1 -- 172.21.15.7:0/2518023949 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0938014420 con 0x7f094014f860 2023-03-03T18:20:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f09461c1700 1 -- 172.21.15.7:0/2518023949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0940155af0 con 0x7f094014f860 2023-03-03T18:20:59.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f0937fff700 1 -- 172.21.15.7:0/2518023949 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09380147d0 con 0x7f094014f860 2023-03-03T18:20:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.469+0000 7f09461c1700 1 -- 172.21.15.7:0/2518023949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094006d070 con 0x7f094014f860 2023-03-03T18:20:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.470+0000 7f09461c1700 1 -- 172.21.15.7:0/2518023949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094014fc30 con 0x7f094014f860 2023-03-03T18:20:59.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.471+0000 7f0937fff700 1 -- 172.21.15.7:0/2518023949 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0938012050 con 0x7f094014f860 2023-03-03T18:20:59.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.471+0000 7f0937fff700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 0x7f092006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:20:59.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.472+0000 7f0937fff700 1 -- 172.21.15.7:0/2518023949 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0938086880 con 0x7f094014f860 2023-03-03T18:20:59.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.472+0000 7f093f7fe700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 0x7f092006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:20:59.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.472+0000 7f093f7fe700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 0x7f092006a7a0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f0930009130 tx=0x7f0930008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:20:59.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.476+0000 7f0937fff700 1 -- 172.21.15.7:0/2518023949 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0938052ab0 con 0x7f094014f860 2023-03-03T18:20:59.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.713+0000 7f09461c1700 1 -- 172.21.15.7:0/2518023949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0940074bb0 con 0x7f094014f860 2023-03-03T18:20:59.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.713+0000 7f0937fff700 1 -- 172.21.15.7:0/2518023949 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f09380560d0 con 0x7f094014f860 2023-03-03T18:20:59.713 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:20:59.714 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:20:59.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.715+0000 7f09461c1700 1 -- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 msgr2=0x7f092006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:20:59.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.715+0000 7f09461c1700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 0x7f092006a7a0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f0930009130 tx=0x7f0930008030 comp rx=0 tx=0).stop 2023-03-03T18:20:59.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.715+0000 7f09461c1700 1 -- 172.21.15.7:0/2518023949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 msgr2=0x7f09401e80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:20:59.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.715+0000 7f09461c1700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 0x7f09401e80c0 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f0938007e40 tx=0x7f093800e3c0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.715+0000 7f09461c1700 1 -- 172.21.15.7:0/2518023949 shutdown_connections 2023-03-03T18:20:59.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.715+0000 7f09461c1700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09200682f0 0x7f092006a7a0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.715+0000 7f09461c1700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f094014f860 0x7f09401e80c0 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.715+0000 7f09461c1700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09400c20d0 0x7f09400c1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.715+0000 7f09461c1700 1 --2- 172.21.15.7:0/2518023949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09400c27c0 0x7f0940066220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:20:59.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.715+0000 7f09461c1700 1 -- 172.21.15.7:0/2518023949 >> 172.21.15.7:0/2518023949 conn(0x7f09400a94d0 msgr2=0x7f09400a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:20:59.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.716+0000 7f09461c1700 1 -- 172.21.15.7:0/2518023949 shutdown_connections 2023-03-03T18:20:59.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:20:59.716+0000 7f09461c1700 1 -- 172.21.15.7:0/2518023949 wait complete. 2023-03-03T18:20:59.721 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:20:59.728 INFO:tasks.mds_thrash.fs.[a]:waiting for 22 secs before thrashing 2023-03-03T18:21:06.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:21:06.949 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:21:11.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:21:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.194+0000 7f62088f7700 1 Processor -- start 2023-03-03T18:21:11.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.194+0000 7f62088f7700 1 -- start start 2023-03-03T18:21:11.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f62088f7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6204072250 0x7f620414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:11.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f62088f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 0x7f620414e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:11.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f62088f7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62041510d0 con 0x7f6204071e80 2023-03-03T18:21:11.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f62088f7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62040c3740 con 0x7f6204072250 2023-03-03T18:21:11.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f62088f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6204070670 con 0x7f62040c1ce0 2023-03-03T18:21:11.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f6201d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6204072250 0x7f620414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:11.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f6201d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6204072250 0x7f620414e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36882/0 (socket says 172.21.15.7:36882) 2023-03-03T18:21:11.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f6201d9b700 1 -- 172.21.15.7:0/3078599919 learned_addr learned my addr 172.21.15.7:0/3078599919 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:21:11.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f6202d9d700 1 --2- 172.21.15.7:0/3078599919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 0x7f620414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:11.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f6201d9b700 1 -- 172.21.15.7:0/3078599919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 msgr2=0x7f620414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:21:11.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f6201d9b700 1 --2- 172.21.15.7:0/3078599919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 0x7f620414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:11.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f6201d9b700 1 -- 172.21.15.7:0/3078599919 >> v1:172.21.15.143:6789/0 conn(0x7f6204071e80 legacy=0x7f6204151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:21:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f6201d9b700 1 -- 172.21.15.7:0/3078599919 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62040c39e0 con 0x7f6204072250 2023-03-03T18:21:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.195+0000 7f6201d9b700 1 --2- 172.21.15.7:0/3078599919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 0x7f620414e1f0 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7f61f800bb90 tx=0x7f61f800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae8e0b33250a1e5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:21:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f620159a700 1 -- 172.21.15.7:0/3078599919 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f800dd10 con 0x7f6204072250 2023-03-03T18:21:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f620159a700 1 -- 172.21.15.7:0/3078599919 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f800de70 con 0x7f6204072250 2023-03-03T18:21:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f620159a700 1 -- 172.21.15.7:0/3078599919 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f8004230 con 0x7f6204072250 2023-03-03T18:21:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f62088f7700 1 -- 172.21.15.7:0/3078599919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 msgr2=0x7f620414e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:11.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f62088f7700 1 --2- 172.21.15.7:0/3078599919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 0x7f620414e1f0 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7f61f800bb90 tx=0x7f61f800bea0 comp rx=0 tx=0).stop 2023-03-03T18:21:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f62088f7700 1 -- 172.21.15.7:0/3078599919 shutdown_connections 2023-03-03T18:21:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f62088f7700 1 --2- 172.21.15.7:0/3078599919 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 0x7f620414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f62088f7700 1 --2- 172.21.15.7:0/3078599919 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 0x7f620414e1f0 unknown :-1 s=CLOSED pgs=3176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f62088f7700 1 -- 172.21.15.7:0/3078599919 >> 172.21.15.7:0/3078599919 conn(0x7f62040a94d0 msgr2=0x7f62040a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:21:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f62088f7700 1 -- 172.21.15.7:0/3078599919 shutdown_connections 2023-03-03T18:21:11.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.196+0000 7f62088f7700 1 -- 172.21.15.7:0/3078599919 wait complete. 2023-03-03T18:21:11.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f62088f7700 1 Processor -- start 2023-03-03T18:21:11.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f62088f7700 1 -- start start 2023-03-03T18:21:11.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f62088f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6204071e80 0x7f62041e6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:11.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f62088f7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 0x7f62040c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:11.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f62088f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 0x7f62041420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:11.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f62088f7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6204142da0 con 0x7f6204072250 2023-03-03T18:21:11.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f62088f7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62041e7060 con 0x7f6204071e80 2023-03-03T18:21:11.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f62088f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62041e60d0 con 0x7f62040c1ce0 2023-03-03T18:21:11.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f6201d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 0x7f62040c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:11.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f6201d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 0x7f62040c5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36886/0 (socket says 172.21.15.7:36886) 2023-03-03T18:21:11.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f6201d9b700 1 -- 172.21.15.7:0/3378451005 learned_addr learned my addr 172.21.15.7:0/3378451005 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:21:11.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f6202d9d700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 0x7f62041420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:11.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.198+0000 7f620259c700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6204071e80 0x7f62041e6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:11.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f6201d9b700 1 -- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 msgr2=0x7f62041420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:11.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f6201d9b700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 0x7f62041420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f6201d9b700 1 -- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6204071e80 msgr2=0x7f62041e6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f6201d9b700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6204071e80 0x7f62041e6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f6201d9b700 1 -- 172.21.15.7:0/3378451005 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f800b7f0 con 0x7f6204072250 2023-03-03T18:21:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f620259c700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6204071e80 0x7f62041e6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:21:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f6202d9d700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 0x7f62041420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:21:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f6201d9b700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 0x7f62040c5160 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 crypto rx=0x7f61f8004000 tx=0x7f61f8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:21:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f61f2ffd700 1 -- 172.21.15.7:0/3378451005 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f8020070 con 0x7f6204072250 2023-03-03T18:21:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f61f2ffd700 1 -- 172.21.15.7:0/3378451005 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f8003aa0 con 0x7f6204072250 2023-03-03T18:21:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f62088f7700 1 -- 172.21.15.7:0/3378451005 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6204151f30 con 0x7f6204072250 2023-03-03T18:21:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f61f2ffd700 1 -- 172.21.15.7:0/3378451005 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f8003e50 con 0x7f6204072250 2023-03-03T18:21:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.199+0000 7f62088f7700 1 -- 172.21.15.7:0/3378451005 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6204151300 con 0x7f6204072250 2023-03-03T18:21:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.200+0000 7f62088f7700 1 -- 172.21.15.7:0/3378451005 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620406c9a0 con 0x7f6204072250 2023-03-03T18:21:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.201+0000 7f61f2ffd700 1 -- 172.21.15.7:0/3378451005 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f61f8024410 con 0x7f6204072250 2023-03-03T18:21:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.201+0000 7f61f2ffd700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61e40682f0 0x7f61e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:11.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.201+0000 7f61f2ffd700 1 -- 172.21.15.7:0/3378451005 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f61f8089790 con 0x7f6204072250 2023-03-03T18:21:11.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.201+0000 7f620259c700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61e40682f0 0x7f61e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:11.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.202+0000 7f620259c700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61e40682f0 0x7f61e406a7a0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f61f4003fc0 tx=0x7f61f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:21:11.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.205+0000 7f61f2ffd700 1 -- 172.21.15.7:0/3378451005 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f61f8055ad0 con 0x7f6204072250 2023-03-03T18:21:11.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.446+0000 7f62088f7700 1 -- 172.21.15.7:0/3378451005 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62040c7100 con 0x7f6204072250 2023-03-03T18:21:11.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.447+0000 7f61f2ffd700 1 -- 172.21.15.7:0/3378451005 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f62040c7100 con 0x7f6204072250 2023-03-03T18:21:11.447 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:21:11.447 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:21:11.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 -- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61e40682f0 msgr2=0x7f61e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:11.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61e40682f0 0x7f61e406a7a0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f61f4003fc0 tx=0x7f61f4009040 comp rx=0 tx=0).stop 2023-03-03T18:21:11.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 -- 172.21.15.7:0/3378451005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 msgr2=0x7f62040c5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:11.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 0x7f62040c5160 secure :-1 s=READY pgs=3177 cs=0 l=1 rev1=1 crypto rx=0x7f61f8004000 tx=0x7f61f8004750 comp rx=0 tx=0).stop 2023-03-03T18:21:11.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 -- 172.21.15.7:0/3378451005 shutdown_connections 2023-03-03T18:21:11.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62040c1ce0 0x7f62041420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:11.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6204071e80 0x7f62041e6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:11.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61e40682f0 0x7f61e406a7a0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:11.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 --2- 172.21.15.7:0/3378451005 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6204072250 0x7f62040c5160 unknown :-1 s=CLOSED pgs=3177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:11.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 -- 172.21.15.7:0/3378451005 >> 172.21.15.7:0/3378451005 conn(0x7f62040a94d0 msgr2=0x7f620406d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:21:11.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 -- 172.21.15.7:0/3378451005 shutdown_connections 2023-03-03T18:21:11.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:11.449+0000 7f62088f7700 1 -- 172.21.15.7:0/3378451005 wait complete. 2023-03-03T18:21:11.453 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:21:11.461 INFO:tasks.mds_thrash.fs.[b]:waiting for 56 secs before thrashing 2023-03-03T18:21:21.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:21:21.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.868+0000 7f5de19fb700 1 Processor -- start 2023-03-03T18:21:21.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.868+0000 7f5de19fb700 1 -- start start 2023-03-03T18:21:21.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.869+0000 7f5de19fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ddc0aad20 0x7f5ddc157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:21.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.869+0000 7f5de19fb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5ddc0aa030 0x7f5ddc158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:21.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.869+0000 7f5de19fb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ddc0a9c20 con 0x7f5ddc14b0d0 2023-03-03T18:21:21.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.869+0000 7f5de19fb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ddc154f10 con 0x7f5ddc0aa030 2023-03-03T18:21:21.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.869+0000 7f5de19fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc0ac3c0 con 0x7f5ddc0aad20 2023-03-03T18:21:21.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.869+0000 7f5ddaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5ddc14b0d0 0x7f5ddc071800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59110/0 (socket says 172.21.15.7:59110) 2023-03-03T18:21:21.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.869+0000 7f5ddaffd700 1 -- 172.21.15.7:0/1768591707 learned_addr learned my addr 172.21.15.7:0/1768591707 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:21:21.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.869+0000 7f5ddb7fe700 1 --2- 172.21.15.7:0/1768591707 >> v2:172.21.15.7:3300/0 conn(0x7f5ddc0aa030 0x7f5ddc158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:21.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.869+0000 7f5dda7fc700 1 --2- 172.21.15.7:0/1768591707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ddc0aad20 0x7f5ddc157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:21.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.869+0000 7f5dd9ffb700 1 -- 172.21.15.7:0/1768591707 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703515139 0 0) 0x7f5ddc0a9c20 con 0x7f5ddc14b0d0 2023-03-03T18:21:21.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5dd9ffb700 1 -- 172.21.15.7:0/1768591707 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc80035b0 con 0x7f5ddc14b0d0 2023-03-03T18:21:21.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5ddb7fe700 1 -- 172.21.15.7:0/1768591707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ddc0aad20 msgr2=0x7f5ddc157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:21:21.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5ddb7fe700 1 --2- 172.21.15.7:0/1768591707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ddc0aad20 0x7f5ddc157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:21.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5ddb7fe700 1 -- 172.21.15.7:0/1768591707 >> v1:172.21.15.7:6789/0 conn(0x7f5ddc14b0d0 legacy=0x7f5ddc071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:21.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5ddb7fe700 1 -- 172.21.15.7:0/1768591707 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ddc06fc70 con 0x7f5ddc0aa030 2023-03-03T18:21:21.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5ddb7fe700 1 --2- 172.21.15.7:0/1768591707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aa030 0x7f5ddc158330 secure :-1 s=READY pgs=3179 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009d90 tx=0x7f5dd400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe2008d5da0078a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:21:21.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5dd9ffb700 1 -- 172.21.15.7:0/1768591707 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd400e070 con 0x7f5ddc0aa030 2023-03-03T18:21:21.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5dd9ffb700 1 -- 172.21.15.7:0/1768591707 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd400bb20 con 0x7f5ddc0aa030 2023-03-03T18:21:21.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5dd9ffb700 1 -- 172.21.15.7:0/1768591707 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4012430 con 0x7f5ddc0aa030 2023-03-03T18:21:21.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5de19fb700 1 -- 172.21.15.7:0/1768591707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aa030 msgr2=0x7f5ddc158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:21.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5de19fb700 1 --2- 172.21.15.7:0/1768591707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aa030 0x7f5ddc158330 secure :-1 s=READY pgs=3179 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009d90 tx=0x7f5dd400b330 comp rx=0 tx=0).stop 2023-03-03T18:21:21.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5de19fb700 1 -- 172.21.15.7:0/1768591707 shutdown_connections 2023-03-03T18:21:21.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5de19fb700 1 --2- 172.21.15.7:0/1768591707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aa030 0x7f5ddc158330 unknown :-1 s=CLOSED pgs=3179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:21.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.870+0000 7f5de19fb700 1 --2- 172.21.15.7:0/1768591707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ddc0aad20 0x7f5ddc157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:21.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.871+0000 7f5de19fb700 1 -- 172.21.15.7:0/1768591707 >> 172.21.15.7:0/1768591707 conn(0x7f5ddc0c8900 msgr2=0x7f5ddc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:21:21.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.871+0000 7f5de19fb700 1 -- 172.21.15.7:0/1768591707 shutdown_connections 2023-03-03T18:21:21.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.871+0000 7f5de19fb700 1 -- 172.21.15.7:0/1768591707 wait complete. 2023-03-03T18:21:21.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.872+0000 7f5de19fb700 1 Processor -- start 2023-03-03T18:21:21.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.872+0000 7f5de19fb700 1 -- start start 2023-03-03T18:21:21.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5de19fb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ddc0aa030 0x7f5ddc158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:21.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5de19fb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aad20 0x7f5ddc153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:21.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5de19fb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ddc14b0d0 0x7f5ddc153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:21.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5de19fb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc153dd0 con 0x7f5ddc0aad20 2023-03-03T18:21:21.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5de19fb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc1edc20 con 0x7f5ddc14b0d0 2023-03-03T18:21:21.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5de19fb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc145e60 con 0x7f5ddc0aa030 2023-03-03T18:21:21.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dda7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aad20 0x7f5ddc153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:21.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dda7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aad20 0x7f5ddc153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36906/0 (socket says 172.21.15.7:36906) 2023-03-03T18:21:21.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dda7fc700 1 -- 172.21.15.7:0/2191988739 learned_addr learned my addr 172.21.15.7:0/2191988739 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:21:21.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5ddaffd700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ddc0aa030 0x7f5ddc158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:21.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5ddb7fe700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ddc14b0d0 0x7f5ddc153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:21.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dda7fc700 1 -- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ddc0aa030 msgr2=0x7f5ddc158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:21.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dda7fc700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ddc0aa030 0x7f5ddc158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:21.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dda7fc700 1 -- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ddc14b0d0 msgr2=0x7f5ddc153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:21.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dda7fc700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ddc14b0d0 0x7f5ddc153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:21.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dda7fc700 1 -- 172.21.15.7:0/2191988739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd4009a70 con 0x7f5ddc0aad20 2023-03-03T18:21:21.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dda7fc700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aad20 0x7f5ddc153350 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00e870 tx=0x7f5dcc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:21:21.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dd3fff700 1 -- 172.21.15.7:0/2191988739 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc009590 con 0x7f5ddc0aad20 2023-03-03T18:21:21.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dd3fff700 1 -- 172.21.15.7:0/2191988739 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dcc00ee70 con 0x7f5ddc0aad20 2023-03-03T18:21:21.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5de19fb700 1 -- 172.21.15.7:0/2191988739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ddc06f6b0 con 0x7f5ddc0aad20 2023-03-03T18:21:21.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5dd3fff700 1 -- 172.21.15.7:0/2191988739 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc00ca50 con 0x7f5ddc0aad20 2023-03-03T18:21:21.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.873+0000 7f5de19fb700 1 -- 172.21.15.7:0/2191988739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ddc155160 con 0x7f5ddc0aad20 2023-03-03T18:21:21.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.874+0000 7f5de19fb700 1 -- 172.21.15.7:0/2191988739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ddc06f990 con 0x7f5ddc0aad20 2023-03-03T18:21:21.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.875+0000 7f5dd3fff700 1 -- 172.21.15.7:0/2191988739 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5dcc019030 con 0x7f5ddc0aad20 2023-03-03T18:21:21.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.875+0000 7f5dd3fff700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5dbc0682f0 0x7f5dbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:21:21.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.875+0000 7f5dd3fff700 1 -- 172.21.15.7:0/2191988739 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5dcc085f10 con 0x7f5ddc0aad20 2023-03-03T18:21:21.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.876+0000 7f5ddaffd700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5dbc0682f0 0x7f5dbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:21:21.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.876+0000 7f5ddaffd700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5dbc0682f0 0x7f5dbc06a7a0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40011e0 tx=0x7f5dc4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:21:21.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:21.880+0000 7f5dd3fff700 1 -- 172.21.15.7:0/2191988739 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5dcc0522b0 con 0x7f5ddc0aad20 2023-03-03T18:21:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.117+0000 7f5de19fb700 1 -- 172.21.15.7:0/2191988739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ddc1edd90 con 0x7f5ddc0aad20 2023-03-03T18:21:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.118+0000 7f5dd3fff700 1 -- 172.21.15.7:0/2191988739 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5dcc0558d0 con 0x7f5ddc0aad20 2023-03-03T18:21:22.119 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:21:22.120 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:21:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 -- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5dbc0682f0 msgr2=0x7f5dbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5dbc0682f0 0x7f5dbc06a7a0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40011e0 tx=0x7f5dc4009040 comp rx=0 tx=0).stop 2023-03-03T18:21:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 -- 172.21.15.7:0/2191988739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aad20 msgr2=0x7f5ddc153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:21:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aad20 0x7f5ddc153350 secure :-1 s=READY pgs=3180 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00e870 tx=0x7f5dcc00ec30 comp rx=0 tx=0).stop 2023-03-03T18:21:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 -- 172.21.15.7:0/2191988739 shutdown_connections 2023-03-03T18:21:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ddc14b0d0 0x7f5ddc153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5ddc0aa030 0x7f5ddc158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5dbc0682f0 0x7f5dbc06a7a0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 --2- 172.21.15.7:0/2191988739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ddc0aad20 0x7f5ddc153350 unknown :-1 s=CLOSED pgs=3180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:21:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 -- 172.21.15.7:0/2191988739 >> 172.21.15.7:0/2191988739 conn(0x7f5ddc0c8900 msgr2=0x7f5ddc0acd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:21:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 -- 172.21.15.7:0/2191988739 shutdown_connections 2023-03-03T18:21:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:21:22.120+0000 7f5de19fb700 1 -- 172.21.15.7:0/2191988739 wait complete. 2023-03-03T18:21:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:21:22.133 INFO:tasks.mds_thrash.fs.[a]:waiting for 96 secs before thrashing 2023-03-03T18:21:37.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:21:37.008 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:22:07.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:22:07.072 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:22:07.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:22:07.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.603+0000 7f8507da4700 1 Processor -- start 2023-03-03T18:22:07.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.603+0000 7f8507da4700 1 -- start start 2023-03-03T18:22:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8507da4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85000c2790 0x7f8500152d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8507da4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f85001546f0 0x7f8500154b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8507da4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850006c920 con 0x7f850014b340 2023-03-03T18:22:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8507da4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85000abd90 con 0x7f85000c2790 2023-03-03T18:22:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8507da4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85000c1300 con 0x7f85001546f0 2023-03-03T18:22:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f850533f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85000c2790 0x7f8500152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:07.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8505b40700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f850014b340 0x7f8500152630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59148/0 (socket says 172.21.15.7:59148) 2023-03-03T18:22:07.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f850533f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85000c2790 0x7f8500152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36938/0 (socket says 172.21.15.7:36938) 2023-03-03T18:22:07.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8505b40700 1 -- 172.21.15.7:0/525330116 learned_addr learned my addr 172.21.15.7:0/525330116 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:22:07.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8506341700 1 --2- 172.21.15.7:0/525330116 >> v2:172.21.15.143:3300/0 conn(0x7f85001546f0 0x7f8500154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:07.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f850533f700 1 -- 172.21.15.7:0/525330116 >> v2:172.21.15.143:3300/0 conn(0x7f85001546f0 msgr2=0x7f8500154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:22:07.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f850533f700 1 --2- 172.21.15.7:0/525330116 >> v2:172.21.15.143:3300/0 conn(0x7f85001546f0 0x7f8500154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:07.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f850533f700 1 -- 172.21.15.7:0/525330116 >> v1:172.21.15.7:6789/0 conn(0x7f850014b340 legacy=0x7f8500152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:07.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8504b3e700 1 -- 172.21.15.7:0/525330116 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520296013 0 0) 0x7f850006c920 con 0x7f850014b340 2023-03-03T18:22:07.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f850533f700 1 -- 172.21.15.7:0/525330116 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8500153280 con 0x7f85000c2790 2023-03-03T18:22:07.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f850533f700 1 --2- 172.21.15.7:0/525330116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85000c2790 0x7f8500152d40 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 crypto rx=0x7f850006ad90 tx=0x7f84f0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee8edcf010808f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:22:07.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8504b3e700 1 -- 172.21.15.7:0/525330116 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0011830 con 0x7f85000c2790 2023-03-03T18:22:07.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8504b3e700 1 -- 172.21.15.7:0/525330116 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f0011990 con 0x7f85000c2790 2023-03-03T18:22:07.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.604+0000 7f8504b3e700 1 -- 172.21.15.7:0/525330116 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0011d20 con 0x7f85000c2790 2023-03-03T18:22:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.605+0000 7f8507da4700 1 -- 172.21.15.7:0/525330116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85000c2790 msgr2=0x7f8500152d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.605+0000 7f8507da4700 1 --2- 172.21.15.7:0/525330116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85000c2790 0x7f8500152d40 secure :-1 s=READY pgs=3182 cs=0 l=1 rev1=1 crypto rx=0x7f850006ad90 tx=0x7f84f0009b20 comp rx=0 tx=0).stop 2023-03-03T18:22:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.605+0000 7f8507da4700 1 -- 172.21.15.7:0/525330116 shutdown_connections 2023-03-03T18:22:07.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.605+0000 7f8507da4700 1 --2- 172.21.15.7:0/525330116 >> v2:172.21.15.143:3300/0 conn(0x7f85001546f0 0x7f8500154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:07.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.605+0000 7f8507da4700 1 --2- 172.21.15.7:0/525330116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85000c2790 0x7f8500152d40 unknown :-1 s=CLOSED pgs=3182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:07.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.605+0000 7f8507da4700 1 -- 172.21.15.7:0/525330116 >> 172.21.15.7:0/525330116 conn(0x7f85000c62e0 msgr2=0x7f85000c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:22:07.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.605+0000 7f8507da4700 1 -- 172.21.15.7:0/525330116 shutdown_connections 2023-03-03T18:22:07.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.605+0000 7f8507da4700 1 -- 172.21.15.7:0/525330116 wait complete. 2023-03-03T18:22:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.606+0000 7f8507da4700 1 Processor -- start 2023-03-03T18:22:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.606+0000 7f8507da4700 1 -- start start 2023-03-03T18:22:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8507da4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85000c2790 0x7f85001412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8507da4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850014b340 0x7f85001455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8507da4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85001546f0 0x7f8500147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8507da4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8500141800 con 0x7f85001546f0 2023-03-03T18:22:07.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8507da4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8500146030 con 0x7f850014b340 2023-03-03T18:22:07.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8507da4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8500146310 con 0x7f85000c2790 2023-03-03T18:22:07.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8506341700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85001546f0 0x7f8500147480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8506341700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85001546f0 0x7f8500147480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36946/0 (socket says 172.21.15.7:36946) 2023-03-03T18:22:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8506341700 1 -- 172.21.15.7:0/3728171183 learned_addr learned my addr 172.21.15.7:0/3728171183 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:22:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f850533f700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850014b340 0x7f85001455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8505b40700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85000c2790 0x7f85001412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8506341700 1 -- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85000c2790 msgr2=0x7f85001412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:07.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8506341700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85000c2790 0x7f85001412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8506341700 1 -- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850014b340 msgr2=0x7f85001455e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8506341700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850014b340 0x7f85001455e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.607+0000 7f8506341700 1 -- 172.21.15.7:0/3728171183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f00094a0 con 0x7f85001546f0 2023-03-03T18:22:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.608+0000 7f8506341700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85001546f0 0x7f8500147480 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 crypto rx=0x7f84f8007e60 tx=0x7f84f800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:22:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.608+0000 7f84f67fc700 1 -- 172.21.15.7:0/3728171183 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f800ed30 con 0x7f85001546f0 2023-03-03T18:22:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.608+0000 7f84f67fc700 1 -- 172.21.15.7:0/3728171183 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f8014420 con 0x7f85001546f0 2023-03-03T18:22:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.608+0000 7f8507da4700 1 -- 172.21.15.7:0/3728171183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850006bff0 con 0x7f85001546f0 2023-03-03T18:22:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.608+0000 7f84f67fc700 1 -- 172.21.15.7:0/3728171183 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f80147d0 con 0x7f85001546f0 2023-03-03T18:22:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.608+0000 7f8507da4700 1 -- 172.21.15.7:0/3728171183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850006bd50 con 0x7f85001546f0 2023-03-03T18:22:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.609+0000 7f8507da4700 1 -- 172.21.15.7:0/3728171183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850006c2d0 con 0x7f85001546f0 2023-03-03T18:22:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.609+0000 7f84f67fc700 1 -- 172.21.15.7:0/3728171183 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f84f8012050 con 0x7f85001546f0 2023-03-03T18:22:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.609+0000 7f84f67fc700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 0x7f84e406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.610+0000 7f84f67fc700 1 -- 172.21.15.7:0/3728171183 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f84f8086620 con 0x7f85001546f0 2023-03-03T18:22:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.610+0000 7f8505b40700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 0x7f84e406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.610+0000 7f8505b40700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 0x7f84e406a7d0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f84fc000f60 tx=0x7f84fc007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:22:07.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.614+0000 7f84f67fc700 1 -- 172.21.15.7:0/3728171183 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f84f80529c0 con 0x7f85001546f0 2023-03-03T18:22:07.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.853+0000 7f8507da4700 1 -- 172.21.15.7:0/3728171183 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85001e91b0 con 0x7f85001546f0 2023-03-03T18:22:07.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.854+0000 7f84f67fc700 1 -- 172.21.15.7:0/3728171183 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f85001e91b0 con 0x7f85001546f0 2023-03-03T18:22:07.854 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:22:07.854 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:22:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 -- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 msgr2=0x7f84e406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:07.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 0x7f84e406a7d0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f84fc000f60 tx=0x7f84fc007e70 comp rx=0 tx=0).stop 2023-03-03T18:22:07.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 -- 172.21.15.7:0/3728171183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85001546f0 msgr2=0x7f8500147480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:07.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85001546f0 0x7f8500147480 secure :-1 s=READY pgs=3183 cs=0 l=1 rev1=1 crypto rx=0x7f84f8007e60 tx=0x7f84f800e3c0 comp rx=0 tx=0).stop 2023-03-03T18:22:07.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 -- 172.21.15.7:0/3728171183 shutdown_connections 2023-03-03T18:22:07.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f84e4068320 0x7f84e406a7d0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:07.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85001546f0 0x7f8500147480 unknown :-1 s=CLOSED pgs=3183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:07.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f85000c2790 0x7f85001412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:07.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 --2- 172.21.15.7:0/3728171183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f850014b340 0x7f85001455e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:07.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 -- 172.21.15.7:0/3728171183 >> 172.21.15.7:0/3728171183 conn(0x7f85000c62e0 msgr2=0x7f85000c14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:22:07.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 -- 172.21.15.7:0/3728171183 shutdown_connections 2023-03-03T18:22:07.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:07.856+0000 7f8507da4700 1 -- 172.21.15.7:0/3728171183 wait complete. 2023-03-03T18:22:07.860 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:22:07.870 INFO:tasks.mds_thrash.fs.[b]:waiting for 94 secs before thrashing 2023-03-03T18:22:37.131 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:22:37.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:22:58.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:22:58.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.275+0000 7faadc333700 1 Processor -- start 2023-03-03T18:22:58.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.275+0000 7faadc333700 1 -- start start 2023-03-03T18:22:58.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faadc333700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad4151d40 0x7faad40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:58.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faadc333700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faad40c2930 0x7faad4157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:58.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faadc333700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad406d430 con 0x7faad4073c30 2023-03-03T18:22:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faadc333700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad406c2a0 con 0x7faad40c2930 2023-03-03T18:22:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faadc333700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad41586e0 con 0x7faad4151d40 2023-03-03T18:22:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faada0cf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad4151d40 0x7faad40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faad98ce700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faad40c2930 0x7faad4157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faada0cf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad4151d40 0x7faad40736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:45040/0 (socket says 172.21.15.7:45040) 2023-03-03T18:22:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faada0cf700 1 -- 172.21.15.7:0/2369300287 learned_addr learned my addr 172.21.15.7:0/2369300287 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:22:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faad90cd700 1 -- 172.21.15.7:0/2369300287 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680314339 0 0) 0x7faad406d430 con 0x7faad4073c30 2023-03-03T18:22:58.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faad90cd700 1 -- 172.21.15.7:0/2369300287 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faac00035b0 con 0x7faad4073c30 2023-03-03T18:22:58.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faada0cf700 1 -- 172.21.15.7:0/2369300287 >> v2:172.21.15.143:3300/0 conn(0x7faad40c2930 msgr2=0x7faad4157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:22:58.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faada0cf700 1 --2- 172.21.15.7:0/2369300287 >> v2:172.21.15.143:3300/0 conn(0x7faad40c2930 0x7faad4157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:58.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faada0cf700 1 -- 172.21.15.7:0/2369300287 >> v1:172.21.15.143:6789/0 conn(0x7faad4073c30 legacy=0x7faad4157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:58.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faada0cf700 1 -- 172.21.15.7:0/2369300287 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad406cb00 con 0x7faad4151d40 2023-03-03T18:22:58.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.276+0000 7faad98ce700 1 --2- 172.21.15.7:0/2369300287 >> v2:172.21.15.143:3300/0 conn(0x7faad40c2930 0x7faad4157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:22:58.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faada0cf700 1 --2- 172.21.15.7:0/2369300287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad4151d40 0x7faad40736f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faac40098f0 tx=0x7faac4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a748c18296107001 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:22:58.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faad90cd700 1 -- 172.21.15.7:0/2369300287 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4011830 con 0x7faad4151d40 2023-03-03T18:22:58.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faad90cd700 1 -- 172.21.15.7:0/2369300287 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac4011990 con 0x7faad4151d40 2023-03-03T18:22:58.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faad90cd700 1 -- 172.21.15.7:0/2369300287 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4011d20 con 0x7faad4151d40 2023-03-03T18:22:58.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faadc333700 1 -- 172.21.15.7:0/2369300287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad4151d40 msgr2=0x7faad40736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:58.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faadc333700 1 --2- 172.21.15.7:0/2369300287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad4151d40 0x7faad40736f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faac40098f0 tx=0x7faac4009c00 comp rx=0 tx=0).stop 2023-03-03T18:22:58.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faadc333700 1 -- 172.21.15.7:0/2369300287 shutdown_connections 2023-03-03T18:22:58.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faadc333700 1 --2- 172.21.15.7:0/2369300287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad4151d40 0x7faad40736f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:58.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faadc333700 1 --2- 172.21.15.7:0/2369300287 >> v2:172.21.15.143:3300/0 conn(0x7faad40c2930 0x7faad4157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:58.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faadc333700 1 -- 172.21.15.7:0/2369300287 >> 172.21.15.7:0/2369300287 conn(0x7faad40a94d0 msgr2=0x7faad40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:22:58.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.277+0000 7faadc333700 1 -- 172.21.15.7:0/2369300287 shutdown_connections 2023-03-03T18:22:58.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.278+0000 7faadc333700 1 -- 172.21.15.7:0/2369300287 wait complete. 2023-03-03T18:22:58.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.279+0000 7faadc333700 1 Processor -- start 2023-03-03T18:22:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.279+0000 7faadc333700 1 -- start start 2023-03-03T18:22:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.279+0000 7faadc333700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faad4073c30 0x7faad414e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.279+0000 7faadc333700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40c2930 0x7faad40aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.279+0000 7faadc333700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad4151d40 0x7faad41eda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.279+0000 7faadc333700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad40ab870 con 0x7faad4073c30 2023-03-03T18:22:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.279+0000 7faadc333700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad414ed60 con 0x7faad4151d40 2023-03-03T18:22:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.279+0000 7faadc333700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad4154cf0 con 0x7faad40c2930 2023-03-03T18:22:58.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.279+0000 7faada0cf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faad4073c30 0x7faad414e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:58.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faada0cf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faad4073c30 0x7faad414e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36970/0 (socket says 172.21.15.7:36970) 2023-03-03T18:22:58.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faada0cf700 1 -- 172.21.15.7:0/481485276 learned_addr learned my addr 172.21.15.7:0/481485276 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:22:58.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faad98ce700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40c2930 0x7faad40aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:58.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faada8d0700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad4151d40 0x7faad41eda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:58.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faada0cf700 1 -- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40c2930 msgr2=0x7faad40aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:58.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faada0cf700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40c2930 0x7faad40aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:58.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faada0cf700 1 -- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad4151d40 msgr2=0x7faad41eda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:58.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faada0cf700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad4151d40 0x7faad41eda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:58.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faada0cf700 1 -- 172.21.15.7:0/481485276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad00094a0 con 0x7faad4073c30 2023-03-03T18:22:58.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faada8d0700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad4151d40 0x7faad41eda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:22:58.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faada0cf700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faad4073c30 0x7faad414e820 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 crypto rx=0x7faad4158820 tx=0x7faac4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:22:58.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faacaffd700 1 -- 172.21.15.7:0/481485276 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac4004450 con 0x7faad4073c30 2023-03-03T18:22:58.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faacaffd700 1 -- 172.21.15.7:0/481485276 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac40045d0 con 0x7faad4073c30 2023-03-03T18:22:58.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faacaffd700 1 -- 172.21.15.7:0/481485276 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac40038e0 con 0x7faad4073c30 2023-03-03T18:22:58.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faadc333700 1 -- 172.21.15.7:0/481485276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac4009550 con 0x7faad4073c30 2023-03-03T18:22:58.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.280+0000 7faadc333700 1 -- 172.21.15.7:0/481485276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad406d0c0 con 0x7faad4073c30 2023-03-03T18:22:58.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.281+0000 7faadc333700 1 -- 172.21.15.7:0/481485276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad406cde0 con 0x7faad4073c30 2023-03-03T18:22:58.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.282+0000 7faacaffd700 1 -- 172.21.15.7:0/481485276 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faac402a020 con 0x7faad4073c30 2023-03-03T18:22:58.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.282+0000 7faacaffd700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faab80682f0 0x7faab806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:22:58.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.282+0000 7faacaffd700 1 -- 172.21.15.7:0/481485276 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faac4087420 con 0x7faad4073c30 2023-03-03T18:22:58.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.282+0000 7faad98ce700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faab80682f0 0x7faab806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:22:58.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.283+0000 7faad98ce700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faab80682f0 0x7faab806a7a0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7faad40c3cd0 tx=0x7faad0009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:22:58.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.286+0000 7faacaffd700 1 -- 172.21.15.7:0/481485276 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faac4053740 con 0x7faad4073c30 2023-03-03T18:22:58.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.525+0000 7faadc333700 1 -- 172.21.15.7:0/481485276 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faad406bea0 con 0x7faad4073c30 2023-03-03T18:22:58.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.526+0000 7faacaffd700 1 -- 172.21.15.7:0/481485276 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faac4056d60 con 0x7faad4073c30 2023-03-03T18:22:58.527 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:22:58.527 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:22:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 -- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faab80682f0 msgr2=0x7faab806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faab80682f0 0x7faab806a7a0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7faad40c3cd0 tx=0x7faad0009210 comp rx=0 tx=0).stop 2023-03-03T18:22:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 -- 172.21.15.7:0/481485276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faad4073c30 msgr2=0x7faad414e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:22:58.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faad4073c30 0x7faad414e820 secure :-1 s=READY pgs=3185 cs=0 l=1 rev1=1 crypto rx=0x7faad4158820 tx=0x7faac4004750 comp rx=0 tx=0).stop 2023-03-03T18:22:58.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 -- 172.21.15.7:0/481485276 shutdown_connections 2023-03-03T18:22:58.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faad4151d40 0x7faad41eda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:58.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faab80682f0 0x7faab806a7a0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:58.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faad4073c30 0x7faad414e820 unknown :-1 s=CLOSED pgs=3185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:58.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 --2- 172.21.15.7:0/481485276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faad40c2930 0x7faad40aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:22:58.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 -- 172.21.15.7:0/481485276 >> 172.21.15.7:0/481485276 conn(0x7faad40a94d0 msgr2=0x7faad4157200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:22:58.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 -- 172.21.15.7:0/481485276 shutdown_connections 2023-03-03T18:22:58.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:22:58.528+0000 7faadc333700 1 -- 172.21.15.7:0/481485276 wait complete. 2023-03-03T18:22:58.531 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:22:58.540 INFO:tasks.mds_thrash.fs.[a]:waiting for 43 secs before thrashing 2023-03-03T18:23:07.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:23:07.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:23:37.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:23:37.378 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:23:41.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:23:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.682+0000 7fb01d93b700 1 Processor -- start 2023-03-03T18:23:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.682+0000 7fb01d93b700 1 -- start start 2023-03-03T18:23:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.682+0000 7fb01d93b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0180caa30 0x7fb0180c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.682+0000 7fb01d93b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb018154ff0 0x7fb0180cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.682+0000 7fb01d93b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0180c7310 con 0x7fb0180c00e0 2023-03-03T18:23:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.682+0000 7fb01d93b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180bfd10 con 0x7fb0180caa30 2023-03-03T18:23:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.682+0000 7fb01d93b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180bfa30 con 0x7fb018154ff0 2023-03-03T18:23:41.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb016ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0180caa30 0x7fb0180c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb016ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0180caa30 0x7fb0180c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37012/0 (socket says 172.21.15.7:37012) 2023-03-03T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb016ffd700 1 -- 172.21.15.7:0/1955463417 learned_addr learned my addr 172.21.15.7:0/1955463417 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb0177fe700 1 --2- 172.21.15.7:0/1955463417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb018154ff0 0x7fb0180cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb016ffd700 1 -- 172.21.15.7:0/1955463417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb018154ff0 msgr2=0x7fb0180cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb016ffd700 1 --2- 172.21.15.7:0/1955463417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb018154ff0 0x7fb0180cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb016ffd700 1 -- 172.21.15.7:0/1955463417 >> v1:172.21.15.143:6789/0 conn(0x7fb0180c00e0 legacy=0x7fb0180c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:23:41.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb016ffd700 1 -- 172.21.15.7:0/1955463417 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb018154510 con 0x7fb0180caa30 2023-03-03T18:23:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb0177fe700 1 --2- 172.21.15.7:0/1955463417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb018154ff0 0x7fb0180cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:23:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb016ffd700 1 --2- 172.21.15.7:0/1955463417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180caa30 0x7fb0180c5c60 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 crypto rx=0x7fb0080098f0 tx=0x7fb008009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3dda0704734dc1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:23:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb015ffb700 1 -- 172.21.15.7:0/1955463417 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008004230 con 0x7fb0180caa30 2023-03-03T18:23:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb015ffb700 1 -- 172.21.15.7:0/1955463417 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb008004390 con 0x7fb0180caa30 2023-03-03T18:23:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.683+0000 7fb015ffb700 1 -- 172.21.15.7:0/1955463417 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0080036a0 con 0x7fb0180caa30 2023-03-03T18:23:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.684+0000 7fb01d93b700 1 -- 172.21.15.7:0/1955463417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180caa30 msgr2=0x7fb0180c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.684+0000 7fb01d93b700 1 --2- 172.21.15.7:0/1955463417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180caa30 0x7fb0180c5c60 secure :-1 s=READY pgs=3186 cs=0 l=1 rev1=1 crypto rx=0x7fb0080098f0 tx=0x7fb008009c00 comp rx=0 tx=0).stop 2023-03-03T18:23:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.684+0000 7fb01d93b700 1 -- 172.21.15.7:0/1955463417 shutdown_connections 2023-03-03T18:23:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.684+0000 7fb01d93b700 1 --2- 172.21.15.7:0/1955463417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb018154ff0 0x7fb0180cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.684+0000 7fb01d93b700 1 --2- 172.21.15.7:0/1955463417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180caa30 0x7fb0180c5c60 unknown :-1 s=CLOSED pgs=3186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.684+0000 7fb01d93b700 1 -- 172.21.15.7:0/1955463417 >> 172.21.15.7:0/1955463417 conn(0x7fb018066e40 msgr2=0x7fb018071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:23:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.684+0000 7fb01d93b700 1 -- 172.21.15.7:0/1955463417 shutdown_connections 2023-03-03T18:23:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.684+0000 7fb01d93b700 1 -- 172.21.15.7:0/1955463417 wait complete. 2023-03-03T18:23:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.685+0000 7fb01d93b700 1 Processor -- start 2023-03-03T18:23:41.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.686+0000 7fb01d93b700 1 -- start start 2023-03-03T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb01d93b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180c00e0 0x7fb01814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb01d93b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0180caa30 0x7fb018156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb01d93b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb018154ff0 0x7fb0181572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb01d93b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01814ca70 con 0x7fb0180c00e0 2023-03-03T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb01d93b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0181de0e0 con 0x7fb018154ff0 2023-03-03T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb01d93b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb018156920 con 0x7fb0180caa30 2023-03-03T18:23:41.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb016ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180c00e0 0x7fb01814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb016ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180c00e0 0x7fb01814fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37018/0 (socket says 172.21.15.7:37018) 2023-03-03T18:23:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb016ffd700 1 -- 172.21.15.7:0/1426811987 learned_addr learned my addr 172.21.15.7:0/1426811987 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:23:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb0177fe700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb018154ff0 0x7fb0181572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb0167fc700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0180caa30 0x7fb018156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb016ffd700 1 -- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0180caa30 msgr2=0x7fb018156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb016ffd700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0180caa30 0x7fb018156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb016ffd700 1 -- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb018154ff0 msgr2=0x7fb0181572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:41.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb016ffd700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb018154ff0 0x7fb0181572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb016ffd700 1 -- 172.21.15.7:0/1426811987 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb008009550 con 0x7fb0180c00e0 2023-03-03T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb0177fe700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb018154ff0 0x7fb0181572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb016ffd700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180c00e0 0x7fb01814fe10 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7fb008009130 tx=0x7fb008004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7faffffff700 1 -- 172.21.15.7:0/1426811987 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008003e30 con 0x7fb0180c00e0 2023-03-03T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7faffffff700 1 -- 172.21.15.7:0/1426811987 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00801f070 con 0x7fb0180c00e0 2023-03-03T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7fb01d93b700 1 -- 172.21.15.7:0/1426811987 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb018154ad0 con 0x7fb0180c00e0 2023-03-03T18:23:41.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.687+0000 7faffffff700 1 -- 172.21.15.7:0/1426811987 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00800b8a0 con 0x7fb0180c00e0 2023-03-03T18:23:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.688+0000 7fb01d93b700 1 -- 172.21.15.7:0/1426811987 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0181547f0 con 0x7fb0180c00e0 2023-03-03T18:23:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.689+0000 7fb01d93b700 1 -- 172.21.15.7:0/1426811987 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0180bf650 con 0x7fb0180c00e0 2023-03-03T18:23:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.689+0000 7faffffff700 1 -- 172.21.15.7:0/1426811987 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb00802a020 con 0x7fb0180c00e0 2023-03-03T18:23:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.689+0000 7faffffff700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0040682f0 0x7fb00406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.690+0000 7faffffff700 1 -- 172.21.15.7:0/1426811987 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb008087990 con 0x7fb0180c00e0 2023-03-03T18:23:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.690+0000 7fb0167fc700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0040682f0 0x7fb00406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.690+0000 7fb0167fc700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0040682f0 0x7fb00406a7a0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fb01814f8a0 tx=0x7fb00c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:23:41.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.694+0000 7faffffff700 1 -- 172.21.15.7:0/1426811987 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb008053d30 con 0x7fb0180c00e0 2023-03-03T18:23:41.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:23:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.941+0000 7fb01d93b700 1 -- 172.21.15.7:0/1426811987 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0180bf650 con 0x7fb0180c00e0 2023-03-03T18:23:41.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.941+0000 7faffffff700 1 -- 172.21.15.7:0/1426811987 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb008057350 con 0x7fb0180c00e0 2023-03-03T18:23:41.942 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:23:41.942 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:23:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 -- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0040682f0 msgr2=0x7fb00406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:41.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0040682f0 0x7fb00406a7a0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fb01814f8a0 tx=0x7fb00c009040 comp rx=0 tx=0).stop 2023-03-03T18:23:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 -- 172.21.15.7:0/1426811987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180c00e0 msgr2=0x7fb01814fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180c00e0 0x7fb01814fe10 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7fb008009130 tx=0x7fb008004750 comp rx=0 tx=0).stop 2023-03-03T18:23:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 -- 172.21.15.7:0/1426811987 shutdown_connections 2023-03-03T18:23:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb018154ff0 0x7fb0181572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb0040682f0 0x7fb00406a7a0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0180c00e0 0x7fb01814fe10 unknown :-1 s=CLOSED pgs=3187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:41.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 --2- 172.21.15.7:0/1426811987 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb0180caa30 0x7fb018156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 -- 172.21.15.7:0/1426811987 >> 172.21.15.7:0/1426811987 conn(0x7fb018066e40 msgr2=0x7fb0180c6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 -- 172.21.15.7:0/1426811987 shutdown_connections 2023-03-03T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.943+0000 7fb01d93b700 1 -- 172.21.15.7:0/1426811987 wait complete. 2023-03-03T18:23:41.945 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:23:41.956 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before thrashing 2023-03-03T18:23:41.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.998+0000 7efce43d7700 1 Processor -- start 2023-03-03T18:23:41.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.998+0000 7efce43d7700 1 -- start start 2023-03-03T18:23:41.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.999+0000 7efce43d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc14fda0 0x7efcdc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:42.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.999+0000 7efce43d7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efcdc0abe20 0x7efcdc1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:42.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.999+0000 7efce43d7700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc0c2f30 con 0x7efcdc14f9d0 2023-03-03T18:23:42.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.999+0000 7efce43d7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcdc0c31d0 con 0x7efcdc0abe20 2023-03-03T18:23:42.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:41.999+0000 7efce43d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc1524a0 con 0x7efcdc14fda0 2023-03-03T18:23:42.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce2974700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efcdc0abe20 0x7efcdc1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:42.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce1972700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc14fda0 0x7efcdc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:42.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce1972700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc14fda0 0x7efcdc156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:45100/0 (socket says 172.21.15.7:45100) 2023-03-03T18:23:42.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce1972700 1 -- 172.21.15.7:0/526365311 learned_addr learned my addr 172.21.15.7:0/526365311 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:23:42.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce2974700 1 -- 172.21.15.7:0/526365311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc14fda0 msgr2=0x7efcdc156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:23:42.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce2974700 1 --2- 172.21.15.7:0/526365311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc14fda0 0x7efcdc156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:42.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce2974700 1 -- 172.21.15.7:0/526365311 >> v1:172.21.15.143:6789/0 conn(0x7efcdc14f9d0 legacy=0x7efcdc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:42.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce2974700 1 -- 172.21.15.7:0/526365311 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0c34b0 con 0x7efcdc0abe20 2023-03-03T18:23:42.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce1171700 1 -- 172.21.15.7:0/526365311 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298408291 0 0) 0x7efcdc0c2f30 con 0x7efcdc14f9d0 2023-03-03T18:23:42.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce2974700 1 --2- 172.21.15.7:0/526365311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcdc0abe20 0x7efcdc1529c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009d90 tx=0x7efcd400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc32c32e2090a94d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:23:42.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce1171700 1 -- 172.21.15.7:0/526365311 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd400e070 con 0x7efcdc0abe20 2023-03-03T18:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce1171700 1 -- 172.21.15.7:0/526365311 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd400bb20 con 0x7efcdc0abe20 2023-03-03T18:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.000+0000 7efce1171700 1 -- 172.21.15.7:0/526365311 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4012430 con 0x7efcdc0abe20 2023-03-03T18:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.001+0000 7efce43d7700 1 -- 172.21.15.7:0/526365311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcdc0abe20 msgr2=0x7efcdc1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.001+0000 7efce43d7700 1 --2- 172.21.15.7:0/526365311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcdc0abe20 0x7efcdc1529c0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009d90 tx=0x7efcd400b330 comp rx=0 tx=0).stop 2023-03-03T18:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.001+0000 7efce43d7700 1 -- 172.21.15.7:0/526365311 shutdown_connections 2023-03-03T18:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.001+0000 7efce43d7700 1 --2- 172.21.15.7:0/526365311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcdc0abe20 0x7efcdc1529c0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:42.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.001+0000 7efce43d7700 1 --2- 172.21.15.7:0/526365311 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc14fda0 0x7efcdc156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.001+0000 7efce43d7700 1 -- 172.21.15.7:0/526365311 >> 172.21.15.7:0/526365311 conn(0x7efcdc0c8900 msgr2=0x7efcdc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.001+0000 7efce43d7700 1 -- 172.21.15.7:0/526365311 shutdown_connections 2023-03-03T18:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.001+0000 7efce43d7700 1 -- 172.21.15.7:0/526365311 wait complete. 2023-03-03T18:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.002+0000 7efce43d7700 1 Processor -- start 2023-03-03T18:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.002+0000 7efce43d7700 1 -- start start 2023-03-03T18:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce43d7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc0abe20 0x7efcdc06f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce43d7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcdc14f9d0 0x7efcdc06f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:42.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce43d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcdc14fda0 0x7efcdc14b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:42.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce43d7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc0bddd0 con 0x7efcdc14fda0 2023-03-03T18:23:42.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce43d7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc06eea0 con 0x7efcdc14f9d0 2023-03-03T18:23:42.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce43d7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc06d400 con 0x7efcdc0abe20 2023-03-03T18:23:42.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce2974700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcdc14fda0 0x7efcdc14b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:42.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce2974700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcdc14fda0 0x7efcdc14b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37036/0 (socket says 172.21.15.7:37036) 2023-03-03T18:23:42.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce2974700 1 -- 172.21.15.7:0/1931088647 learned_addr learned my addr 172.21.15.7:0/1931088647 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:23:42.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce1972700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcdc14f9d0 0x7efcdc06f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:42.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce2173700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc0abe20 0x7efcdc06f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:42.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce2974700 1 -- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc0abe20 msgr2=0x7efcdc06f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:42.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce2974700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc0abe20 0x7efcdc06f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:42.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce2974700 1 -- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcdc14f9d0 msgr2=0x7efcdc06f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:42.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce2974700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcdc14f9d0 0x7efcdc06f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:42.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.003+0000 7efce2974700 1 -- 172.21.15.7:0/1931088647 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd4009a70 con 0x7efcdc14fda0 2023-03-03T18:23:42.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.004+0000 7efce2974700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcdc14fda0 0x7efcdc14b090 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7efcd4014040 tx=0x7efcd4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:23:42.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.004+0000 7efcd2ffd700 1 -- 172.21.15.7:0/1931088647 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4012430 con 0x7efcdc14fda0 2023-03-03T18:23:42.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.004+0000 7efcd2ffd700 1 -- 172.21.15.7:0/1931088647 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd4020070 con 0x7efcdc14fda0 2023-03-03T18:23:42.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.004+0000 7efce43d7700 1 -- 172.21.15.7:0/1931088647 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcdc0ac350 con 0x7efcdc14fda0 2023-03-03T18:23:42.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.004+0000 7efcd2ffd700 1 -- 172.21.15.7:0/1931088647 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4004280 con 0x7efcdc14fda0 2023-03-03T18:23:42.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.004+0000 7efce43d7700 1 -- 172.21.15.7:0/1931088647 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcdc0ab630 con 0x7efcdc14fda0 2023-03-03T18:23:42.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.005+0000 7efce43d7700 1 -- 172.21.15.7:0/1931088647 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcdc071fa0 con 0x7efcdc14fda0 2023-03-03T18:23:42.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.005+0000 7efcd2ffd700 1 -- 172.21.15.7:0/1931088647 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efcd40043e0 con 0x7efcdc14fda0 2023-03-03T18:23:42.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.006+0000 7efcd2ffd700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcc00682f0 0x7efcc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:42.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.006+0000 7efcd2ffd700 1 -- 172.21.15.7:0/1931088647 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efcd4022070 con 0x7efcdc14fda0 2023-03-03T18:23:42.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.006+0000 7efce2173700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcc00682f0 0x7efcc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:42.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.006+0000 7efce2173700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcc00682f0 0x7efcc006a7a0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7efccc000f60 tx=0x7efccc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:23:42.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.010+0000 7efcd2ffd700 1 -- 172.21.15.7:0/1931088647 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efcd4054850 con 0x7efcdc14fda0 2023-03-03T18:23:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.249+0000 7efce43d7700 1 -- 172.21.15.7:0/1931088647 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcdc074bb0 con 0x7efcdc14fda0 2023-03-03T18:23:42.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.249+0000 7efcd2ffd700 1 -- 172.21.15.7:0/1931088647 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efcd4057e70 con 0x7efcdc14fda0 2023-03-03T18:23:42.250 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:23:42.250 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:23:42.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 -- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcc00682f0 msgr2=0x7efcc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcc00682f0 0x7efcc006a7a0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7efccc000f60 tx=0x7efccc009040 comp rx=0 tx=0).stop 2023-03-03T18:23:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 -- 172.21.15.7:0/1931088647 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcdc14fda0 msgr2=0x7efcdc14b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcdc14fda0 0x7efcdc14b090 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7efcd4014040 tx=0x7efcd4004ad0 comp rx=0 tx=0).stop 2023-03-03T18:23:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 -- 172.21.15.7:0/1931088647 shutdown_connections 2023-03-03T18:23:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efcc00682f0 0x7efcc006a7a0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:42.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcdc14fda0 0x7efcdc14b090 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efcdc0abe20 0x7efcdc06f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 --2- 172.21.15.7:0/1931088647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efcdc14f9d0 0x7efcdc06f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 -- 172.21.15.7:0/1931088647 >> 172.21.15.7:0/1931088647 conn(0x7efcdc0c8900 msgr2=0x7efcdc14efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:23:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 -- 172.21.15.7:0/1931088647 shutdown_connections 2023-03-03T18:23:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:42.252+0000 7efce43d7700 1 -- 172.21.15.7:0/1931088647 wait complete. 2023-03-03T18:23:42.253 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:23:42.263 INFO:tasks.mds_thrash.fs.[b]:waiting for 113 secs before thrashing 2023-03-03T18:23:49.957 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:23:50.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.096+0000 7f092a753700 1 Processor -- start 2023-03-03T18:23:50.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.096+0000 7f092a753700 1 -- start start 2023-03-03T18:23:50.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f092a753700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f09240c1da0 0x7f0924156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:50.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f092a753700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 0x7f092415cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:50.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f092a753700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0924155840 con 0x7f092414fb00 2023-03-03T18:23:50.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f092a753700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0924072940 con 0x7f09240c1da0 2023-03-03T18:23:50.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f092a753700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09241567d0 con 0x7f09240740f0 2023-03-03T18:23:50.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f09237fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f09240c1da0 0x7f0924156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:50.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f0923fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f092414fb00 0x7f09240c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59260/0 (socket says 172.21.15.7:59260) 2023-03-03T18:23:50.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f0923fff700 1 -- 172.21.15.7:0/1184803736 learned_addr learned my addr 172.21.15.7:0/1184803736 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:23:50.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f0928cf0700 1 --2- 172.21.15.7:0/1184803736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 0x7f092415cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:50.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f09237fe700 1 -- 172.21.15.7:0/1184803736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 msgr2=0x7f092415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:23:50.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f09237fe700 1 --2- 172.21.15.7:0/1184803736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 0x7f092415cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:50.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f09237fe700 1 -- 172.21.15.7:0/1184803736 >> v1:172.21.15.7:6789/0 conn(0x7f092414fb00 legacy=0x7f09240c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:50.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f09237fe700 1 -- 172.21.15.7:0/1184803736 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09240c2330 con 0x7f09240c1da0 2023-03-03T18:23:50.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f0922ffd700 1 -- 172.21.15.7:0/1184803736 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278275140 0 0) 0x7f0924155840 con 0x7f092414fb00 2023-03-03T18:23:50.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.097+0000 7f09237fe700 1 --2- 172.21.15.7:0/1184803736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09240c1da0 0x7f0924156290 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7f090c009840 tx=0x7f090c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef891a7a66321ec0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:23:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f0922ffd700 1 -- 172.21.15.7:0/1184803736 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c011830 con 0x7f09240c1da0 2023-03-03T18:23:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f0922ffd700 1 -- 172.21.15.7:0/1184803736 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090c011990 con 0x7f09240c1da0 2023-03-03T18:23:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f0922ffd700 1 -- 172.21.15.7:0/1184803736 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090c011d20 con 0x7f09240c1da0 2023-03-03T18:23:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f092a753700 1 -- 172.21.15.7:0/1184803736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09240c1da0 msgr2=0x7f0924156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f092a753700 1 --2- 172.21.15.7:0/1184803736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09240c1da0 0x7f0924156290 secure :-1 s=READY pgs=3190 cs=0 l=1 rev1=1 crypto rx=0x7f090c009840 tx=0x7f090c009b50 comp rx=0 tx=0).stop 2023-03-03T18:23:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f092a753700 1 -- 172.21.15.7:0/1184803736 shutdown_connections 2023-03-03T18:23:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f092a753700 1 --2- 172.21.15.7:0/1184803736 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 0x7f092415cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f092a753700 1 --2- 172.21.15.7:0/1184803736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09240c1da0 0x7f0924156290 unknown :-1 s=CLOSED pgs=3190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f092a753700 1 -- 172.21.15.7:0/1184803736 >> 172.21.15.7:0/1184803736 conn(0x7f092406c110 msgr2=0x7f092406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:23:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f092a753700 1 -- 172.21.15.7:0/1184803736 shutdown_connections 2023-03-03T18:23:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.098+0000 7f092a753700 1 -- 172.21.15.7:0/1184803736 wait complete. 2023-03-03T18:23:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.100+0000 7f092a753700 1 Processor -- start 2023-03-03T18:23:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.100+0000 7f092a753700 1 -- start start 2023-03-03T18:23:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f092a753700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 0x7f09241e8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f092a753700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09240c1da0 0x7f09241dbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f092a753700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f092414fb00 0x7f092414add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f092a753700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09241dd0a0 con 0x7f092414fb00 2023-03-03T18:23:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f092a753700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09241dd4b0 con 0x7f09240c1da0 2023-03-03T18:23:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f092a753700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09241de5a0 con 0x7f09240740f0 2023-03-03T18:23:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0928cf0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f092414fb00 0x7f092414add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0928cf0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f092414fb00 0x7f092414add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37058/0 (socket says 172.21.15.7:37058) 2023-03-03T18:23:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0928cf0700 1 -- 172.21.15.7:0/2506002711 learned_addr learned my addr 172.21.15.7:0/2506002711 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:23:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0923fff700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 0x7f09241e8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f09237fe700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09240c1da0 0x7f09241dbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0928cf0700 1 -- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 msgr2=0x7f09241e8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0928cf0700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 0x7f09241e8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0928cf0700 1 -- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09240c1da0 msgr2=0x7f09241dbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0928cf0700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09240c1da0 0x7f09241dbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0928cf0700 1 -- 172.21.15.7:0/2506002711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c0094a0 con 0x7f092414fb00 2023-03-03T18:23:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0923fff700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 0x7f09241e8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:23:50.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f09237fe700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09240c1da0 0x7f09241dbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:23:50.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.101+0000 7f0928cf0700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f092414fb00 0x7f092414add0 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7f091c007e90 tx=0x7f091c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:23:50.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.102+0000 7f0920ff9700 1 -- 172.21.15.7:0/2506002711 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c00ede0 con 0x7f092414fb00 2023-03-03T18:23:50.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.102+0000 7f0920ff9700 1 -- 172.21.15.7:0/2506002711 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091c014420 con 0x7f092414fb00 2023-03-03T18:23:50.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.102+0000 7f092a753700 1 -- 172.21.15.7:0/2506002711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0924153090 con 0x7f092414fb00 2023-03-03T18:23:50.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.102+0000 7f0920ff9700 1 -- 172.21.15.7:0/2506002711 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c0147d0 con 0x7f092414fb00 2023-03-03T18:23:50.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.102+0000 7f092a753700 1 -- 172.21.15.7:0/2506002711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0924152db0 con 0x7f092414fb00 2023-03-03T18:23:50.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.103+0000 7f092a753700 1 -- 172.21.15.7:0/2506002711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0924155a90 con 0x7f092414fb00 2023-03-03T18:23:50.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.103+0000 7f0920ff9700 1 -- 172.21.15.7:0/2506002711 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f091c012050 con 0x7f092414fb00 2023-03-03T18:23:50.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.104+0000 7f0920ff9700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09040682f0 0x7f090406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:23:50.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.104+0000 7f0920ff9700 1 -- 172.21.15.7:0/2506002711 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f091c086780 con 0x7f092414fb00 2023-03-03T18:23:50.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.104+0000 7f0923fff700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09040682f0 0x7f090406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:23:50.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.105+0000 7f0923fff700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09040682f0 0x7f090406a7a0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f0914000f60 tx=0x7f0914007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:23:50.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.108+0000 7f0920ff9700 1 -- 172.21.15.7:0/2506002711 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f091c052b20 con 0x7f092414fb00 2023-03-03T18:23:50.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.348+0000 7f092a753700 1 -- 172.21.15.7:0/2506002711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f092406e410 con 0x7f092414fb00 2023-03-03T18:23:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.348+0000 7f0920ff9700 1 -- 172.21.15.7:0/2506002711 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f091c056140 con 0x7f092414fb00 2023-03-03T18:23:50.348 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:23:50.349 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:23:50.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.350+0000 7f092a753700 1 -- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09040682f0 msgr2=0x7f090406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:50.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.350+0000 7f092a753700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09040682f0 0x7f090406a7a0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f0914000f60 tx=0x7f0914007d30 comp rx=0 tx=0).stop 2023-03-03T18:23:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.350+0000 7f092a753700 1 -- 172.21.15.7:0/2506002711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f092414fb00 msgr2=0x7f092414add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:23:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.350+0000 7f092a753700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f092414fb00 0x7f092414add0 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7f091c007e90 tx=0x7f091c00e3c0 comp rx=0 tx=0).stop 2023-03-03T18:23:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.350+0000 7f092a753700 1 -- 172.21.15.7:0/2506002711 shutdown_connections 2023-03-03T18:23:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.350+0000 7f092a753700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09040682f0 0x7f090406a7a0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.350+0000 7f092a753700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f092414fb00 0x7f092414add0 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.350+0000 7f092a753700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09240740f0 0x7f09241e8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:50.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.350+0000 7f092a753700 1 --2- 172.21.15.7:0/2506002711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09240c1da0 0x7f09241dbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:23:50.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.350+0000 7f092a753700 1 -- 172.21.15.7:0/2506002711 >> 172.21.15.7:0/2506002711 conn(0x7f092406c110 msgr2=0x7f092414f340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:23:50.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.351+0000 7f092a753700 1 -- 172.21.15.7:0/2506002711 shutdown_connections 2023-03-03T18:23:50.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:23:50.351+0000 7f092a753700 1 -- 172.21.15.7:0/2506002711 wait complete. 2023-03-03T18:23:50.352 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:23:50.363 INFO:tasks.mds_thrash.fs.[a]:waiting for 91 secs before thrashing 2023-03-03T18:24:07.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:24:07.499 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:24:37.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:24:37.636 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:25:07.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:25:07.716 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:25:21.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:25:21.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.502+0000 7f67c4a27700 1 Processor -- start 2023-03-03T18:25:21.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.502+0000 7f67c4a27700 1 -- start start 2023-03-03T18:25:21.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.503+0000 7f67c4a27700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f67c00c00e0 0x7f67c00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:21.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.503+0000 7f67c4a27700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c00c7310 con 0x7f67c0154ff0 2023-03-03T18:25:21.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.503+0000 7f67c4a27700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c00bfd10 con 0x7f67c00caa30 2023-03-03T18:25:21.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.503+0000 7f67c4a27700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c0155430 con 0x7f67c00c00e0 2023-03-03T18:25:21.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.503+0000 7f67bdd9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f67c00c00e0 0x7f67c00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:21.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.503+0000 7f67bed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f67c0154ff0 0x7f67c015ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59352/0 (socket says 172.21.15.7:59352) 2023-03-03T18:25:21.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.503+0000 7f67bed9d700 1 -- 172.21.15.7:0/738225206 learned_addr learned my addr 172.21.15.7:0/738225206 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:25:21.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.504+0000 7f67bd59a700 1 -- 172.21.15.7:0/738225206 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30899037 0 0) 0x7f67c00c7310 con 0x7f67c0154ff0 2023-03-03T18:25:21.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.504+0000 7f67bd59a700 1 -- 172.21.15.7:0/738225206 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67a80035b0 con 0x7f67c0154ff0 2023-03-03T18:25:21.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.504+0000 7f67bdd9b700 1 -- 172.21.15.7:0/738225206 >> v1:172.21.15.143:6789/0 conn(0x7f67c00caa30 legacy=0x7f67c00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:21.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.504+0000 7f67bdd9b700 1 -- 172.21.15.7:0/738225206 >> v1:172.21.15.7:6789/0 conn(0x7f67c0154ff0 legacy=0x7f67c015ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:21.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.504+0000 7f67bdd9b700 1 -- 172.21.15.7:0/738225206 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c00bfa30 con 0x7f67c00c00e0 2023-03-03T18:25:21.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.504+0000 7f67bdd9b700 1 --2- 172.21.15.7:0/738225206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00c00e0 0x7f67c00c5c60 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009820 tx=0x7f67b0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8eed7958d5e8c30 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:25:21.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.504+0000 7f67bd59a700 1 -- 172.21.15.7:0/738225206 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b0005e30 con 0x7f67c00c00e0 2023-03-03T18:25:21.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.504+0000 7f67bd59a700 1 -- 172.21.15.7:0/738225206 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b0006030 con 0x7f67c00c00e0 2023-03-03T18:25:21.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.504+0000 7f67bd59a700 1 -- 172.21.15.7:0/738225206 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b00063c0 con 0x7f67c00c00e0 2023-03-03T18:25:21.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.505+0000 7f67c4a27700 1 -- 172.21.15.7:0/738225206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00c00e0 msgr2=0x7f67c00c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:21.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.505+0000 7f67c4a27700 1 --2- 172.21.15.7:0/738225206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00c00e0 0x7f67c00c5c60 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009820 tx=0x7f67b0009b30 comp rx=0 tx=0).stop 2023-03-03T18:25:21.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.505+0000 7f67c4a27700 1 -- 172.21.15.7:0/738225206 shutdown_connections 2023-03-03T18:25:21.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.505+0000 7f67c4a27700 1 --2- 172.21.15.7:0/738225206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00c00e0 0x7f67c00c5c60 unknown :-1 s=CLOSED pgs=3193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:21.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.505+0000 7f67c4a27700 1 -- 172.21.15.7:0/738225206 >> 172.21.15.7:0/738225206 conn(0x7f67c0066e40 msgr2=0x7f67c0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:25:21.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.505+0000 7f67c4a27700 1 -- 172.21.15.7:0/738225206 shutdown_connections 2023-03-03T18:25:21.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.505+0000 7f67c4a27700 1 -- 172.21.15.7:0/738225206 wait complete. 2023-03-03T18:25:21.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.506+0000 7f67c4a27700 1 Processor -- start 2023-03-03T18:25:21.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.506+0000 7f67c4a27700 1 -- start start 2023-03-03T18:25:21.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67c4a27700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67c00c00e0 0x7f67c01eaae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:21.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67c4a27700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00caa30 0x7f67c014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:21.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67c4a27700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67c0154ff0 0x7f67c0155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:21.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67c4a27700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01ea650 con 0x7f67c00caa30 2023-03-03T18:25:21.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67c4a27700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01584b0 con 0x7f67c00c00e0 2023-03-03T18:25:21.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67c4a27700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01ea000 con 0x7f67c0154ff0 2023-03-03T18:25:21.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67bdd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00caa30 0x7f67c014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:21.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67bdd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00caa30 0x7f67c014b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37144/0 (socket says 172.21.15.7:37144) 2023-03-03T18:25:21.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67bdd9b700 1 -- 172.21.15.7:0/1882311200 learned_addr learned my addr 172.21.15.7:0/1882311200 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:25:21.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67bed9d700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67c0154ff0 0x7f67c0155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:21.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67be59c700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67c00c00e0 0x7f67c01eaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:21.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67bdd9b700 1 -- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67c0154ff0 msgr2=0x7f67c0155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:21.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67bdd9b700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67c0154ff0 0x7f67c0155790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:21.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67bdd9b700 1 -- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67c00c00e0 msgr2=0x7f67c01eaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:21.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67bdd9b700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67c00c00e0 0x7f67c01eaae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:21.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67bdd9b700 1 -- 172.21.15.7:0/1882311200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b0009430 con 0x7f67c00caa30 2023-03-03T18:25:21.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.507+0000 7f67bdd9b700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00caa30 0x7f67c014b0f0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009fd0 tx=0x7f67b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:25:21.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.508+0000 7f67aeffd700 1 -- 172.21.15.7:0/1882311200 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b0004240 con 0x7f67c00caa30 2023-03-03T18:25:21.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.508+0000 7f67aeffd700 1 -- 172.21.15.7:0/1882311200 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b00043a0 con 0x7f67c00caa30 2023-03-03T18:25:21.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.508+0000 7f67c4a27700 1 -- 172.21.15.7:0/1882311200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c01547f0 con 0x7f67c00caa30 2023-03-03T18:25:21.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.508+0000 7f67aeffd700 1 -- 172.21.15.7:0/1882311200 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b00036a0 con 0x7f67c00caa30 2023-03-03T18:25:21.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.508+0000 7f67c4a27700 1 -- 172.21.15.7:0/1882311200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c0154510 con 0x7f67c00caa30 2023-03-03T18:25:21.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.509+0000 7f67c4a27700 1 -- 172.21.15.7:0/1882311200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c0154ad0 con 0x7f67c00caa30 2023-03-03T18:25:21.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.509+0000 7f67aeffd700 1 -- 172.21.15.7:0/1882311200 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f67b000f050 con 0x7f67c00caa30 2023-03-03T18:25:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.509+0000 7f67aeffd700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67a00682f0 0x7f67a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.510+0000 7f67aeffd700 1 -- 172.21.15.7:0/1882311200 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f67b0087ca0 con 0x7f67c00caa30 2023-03-03T18:25:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.510+0000 7f67be59c700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67a00682f0 0x7f67a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.510+0000 7f67be59c700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67a00682f0 0x7f67a006a7a0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f67b4000f60 tx=0x7f67b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:25:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.514+0000 7f67aeffd700 1 -- 172.21.15.7:0/1882311200 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f67b008c070 con 0x7f67c00caa30 2023-03-03T18:25:21.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.753+0000 7f67c4a27700 1 -- 172.21.15.7:0/1882311200 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67c0074bb0 con 0x7f67c00caa30 2023-03-03T18:25:21.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.753+0000 7f67aeffd700 1 -- 172.21.15.7:0/1882311200 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f67b002a090 con 0x7f67c00caa30 2023-03-03T18:25:21.753 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:25:21.754 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:25:21.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 -- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67a00682f0 msgr2=0x7f67a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:21.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67a00682f0 0x7f67a006a7a0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f67b4000f60 tx=0x7f67b4009040 comp rx=0 tx=0).stop 2023-03-03T18:25:21.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 -- 172.21.15.7:0/1882311200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00caa30 msgr2=0x7f67c014b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:21.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00caa30 0x7f67c014b0f0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009fd0 tx=0x7f67b0004750 comp rx=0 tx=0).stop 2023-03-03T18:25:21.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 -- 172.21.15.7:0/1882311200 shutdown_connections 2023-03-03T18:25:21.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f67c0154ff0 0x7f67c0155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:21.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67c00c00e0 0x7f67c01eaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:21.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f67a00682f0 0x7f67a006a7a0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:21.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 --2- 172.21.15.7:0/1882311200 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67c00caa30 0x7f67c014b0f0 unknown :-1 s=CLOSED pgs=3194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:21.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 -- 172.21.15.7:0/1882311200 >> 172.21.15.7:0/1882311200 conn(0x7f67c0066e40 msgr2=0x7f67c006b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:25:21.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 -- 172.21.15.7:0/1882311200 shutdown_connections 2023-03-03T18:25:21.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:21.755+0000 7f67c4a27700 1 -- 172.21.15.7:0/1882311200 wait complete. 2023-03-03T18:25:21.757 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:25:21.767 INFO:tasks.mds_thrash.fs.[a]:waiting for 108 secs before thrashing 2023-03-03T18:25:35.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:25:35.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.404+0000 7f960ad60700 1 Processor -- start 2023-03-03T18:25:35.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.404+0000 7f960ad60700 1 -- start start 2023-03-03T18:25:35.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f960ad60700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9604151cd0 0x7f9604154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:35.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f960ad60700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c2e30 0x7f9604158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:35.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f960ad60700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9604157ff0 con 0x7f960406c6c0 2023-03-03T18:25:35.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f960ad60700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f960406d640 con 0x7f9604151cd0 2023-03-03T18:25:35.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f960ad60700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960406cda0 con 0x7f96040c2e30 2023-03-03T18:25:35.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f96092fd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f960406c6c0 0x7f96040736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59378/0 (socket says 172.21.15.7:59378) 2023-03-03T18:25:35.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f96092fd700 1 -- 172.21.15.7:0/852480046 learned_addr learned my addr 172.21.15.7:0/852480046 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:25:35.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f9603fff700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c2e30 0x7f9604158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:35.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f9608afc700 1 --2- 172.21.15.7:0/852480046 >> v2:172.21.15.143:3300/0 conn(0x7f9604151cd0 0x7f9604154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:35.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148324185 0 0) 0x7f9604157ff0 con 0x7f960406c6c0 2023-03-03T18:25:35.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f40035b0 con 0x7f960406c6c0 2023-03-03T18:25:35.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2472886429 0 0) 0x7f95f40035b0 con 0x7f960406c6c0 2023-03-03T18:25:35.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c2e30 msgr2=0x7f9604158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:25:35.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f96037fe700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c2e30 0x7f9604158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:35.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 >> v2:172.21.15.143:3300/0 conn(0x7f9604151cd0 msgr2=0x7f9604154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:25:35.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f96037fe700 1 --2- 172.21.15.7:0/852480046 >> v2:172.21.15.143:3300/0 conn(0x7f9604151cd0 0x7f9604154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:35.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.405+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960406d080 con 0x7f960406c6c0 2023-03-03T18:25:35.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f95fc0019f0 con 0x7f960406c6c0 2023-03-03T18:25:35.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 >> v1:172.21.15.7:6789/0 conn(0x7f960406c6c0 legacy=0x7f96040736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:35.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95f4003a70 0x7f95f4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:35.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 0x7f95f4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:35.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c2e30 0x7f95f4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:35.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960406d640 con 0x7f95f4004850 2023-03-03T18:25:35.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960406cda0 con 0x7f95f4003a70 2023-03-03T18:25:35.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f400bb10 con 0x7f96040c2e30 2023-03-03T18:25:35.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f9603fff700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 0x7f95f4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:35.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f9608afc700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95f4003a70 0x7f95f4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:35.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96092fd700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c2e30 0x7f95f4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:35.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f9603fff700 1 -- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c2e30 msgr2=0x7f95f4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:25:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f9603fff700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c2e30 0x7f95f4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f9603fff700 1 -- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95f4003a70 msgr2=0x7f95f4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:25:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f9603fff700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f95f4003a70 0x7f95f4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f9603fff700 1 -- 172.21.15.7:0/852480046 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f40035b0 con 0x7f95f4004850 2023-03-03T18:25:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f9603fff700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 0x7f95f4004cc0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7f95f0014f30 tx=0x7f95f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce43047aeaada09e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:25:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f0019070 con 0x7f95f4004850 2023-03-03T18:25:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f0009dd0 con 0x7f95f4004850 2023-03-03T18:25:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.406+0000 7f96037fe700 1 -- 172.21.15.7:0/852480046 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f001d610 con 0x7f95f4004850 2023-03-03T18:25:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.407+0000 7f960ad60700 1 -- 172.21.15.7:0/852480046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 msgr2=0x7f95f4004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.407+0000 7f960ad60700 1 --2- 172.21.15.7:0/852480046 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 0x7f95f4004cc0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7f95f0014f30 tx=0x7f95f4005880 comp rx=0 tx=0).stop 2023-03-03T18:25:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.407+0000 7f96092fd700 1 -- 172.21.15.7:0/852480046 reap_dead start 2023-03-03T18:25:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.407+0000 7f960ad60700 1 -- 172.21.15.7:0/852480046 shutdown_connections 2023-03-03T18:25:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.407+0000 7f960ad60700 1 -- 172.21.15.7:0/852480046 >> 172.21.15.7:0/852480046 conn(0x7f96040a94d0 msgr2=0x7f96040a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:25:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.407+0000 7f960ad60700 1 -- 172.21.15.7:0/852480046 shutdown_connections 2023-03-03T18:25:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.407+0000 7f960ad60700 1 -- 172.21.15.7:0/852480046 wait complete. 2023-03-03T18:25:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.409+0000 7f960ad60700 1 Processor -- start 2023-03-03T18:25:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.409+0000 7f960ad60700 1 -- start start 2023-03-03T18:25:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f960ad60700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 0x7f96040c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f960ad60700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c51f0 0x7f960414e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f960ad60700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f960414e6e0 0x7f96041463d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f960ad60700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96041e5bf0 con 0x7f95f4004850 2023-03-03T18:25:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f960ad60700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96041e6bc0 con 0x7f960414e6e0 2023-03-03T18:25:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f960ad60700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96041e6730 con 0x7f96040c51f0 2023-03-03T18:25:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f9608afc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 0x7f96040c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f9608afc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 0x7f96040c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37174/0 (socket says 172.21.15.7:37174) 2023-03-03T18:25:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f9608afc700 1 -- 172.21.15.7:0/2816330536 learned_addr learned my addr 172.21.15.7:0/2816330536 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:25:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f96092fd700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f960414e6e0 0x7f96041463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f9603fff700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c51f0 0x7f960414e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f9608afc700 1 -- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c51f0 msgr2=0x7f960414e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f9608afc700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c51f0 0x7f960414e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f9608afc700 1 -- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f960414e6e0 msgr2=0x7f96041463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f9608afc700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f960414e6e0 0x7f96041463d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f9608afc700 1 -- 172.21.15.7:0/2816330536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9604157c80 con 0x7f95f4004850 2023-03-03T18:25:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.410+0000 7f9608afc700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 0x7f96040c6c30 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f95f8011aa0 tx=0x7f95f8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:25:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.411+0000 7f96017fa700 1 -- 172.21.15.7:0/2816330536 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f800f980 con 0x7f95f4004850 2023-03-03T18:25:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.411+0000 7f96017fa700 1 -- 172.21.15.7:0/2816330536 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f800fae0 con 0x7f95f4004850 2023-03-03T18:25:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.411+0000 7f960ad60700 1 -- 172.21.15.7:0/2816330536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960406d360 con 0x7f95f4004850 2023-03-03T18:25:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.411+0000 7f96017fa700 1 -- 172.21.15.7:0/2816330536 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f800fe90 con 0x7f95f4004850 2023-03-03T18:25:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.411+0000 7f960ad60700 1 -- 172.21.15.7:0/2816330536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9604154a50 con 0x7f95f4004850 2023-03-03T18:25:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.412+0000 7f960ad60700 1 -- 172.21.15.7:0/2816330536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96041e6a10 con 0x7f95f4004850 2023-03-03T18:25:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.412+0000 7f96017fa700 1 -- 172.21.15.7:0/2816330536 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f95f8014440 con 0x7f95f4004850 2023-03-03T18:25:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.413+0000 7f96017fa700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95e80682f0 0x7f95e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.413+0000 7f96017fa700 1 -- 172.21.15.7:0/2816330536 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f95f8086b10 con 0x7f95f4004850 2023-03-03T18:25:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.413+0000 7f9603fff700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95e80682f0 0x7f95e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.414+0000 7f9603fff700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95e80682f0 0x7f95e806a7a0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f95f0020040 tx=0x7f95f0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:25:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.417+0000 7f96017fa700 1 -- 172.21.15.7:0/2816330536 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f95f8052d60 con 0x7f95f4004850 2023-03-03T18:25:35.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.659+0000 7f960ad60700 1 -- 172.21.15.7:0/2816330536 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9604074bb0 con 0x7f95f4004850 2023-03-03T18:25:35.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.659+0000 7f96017fa700 1 -- 172.21.15.7:0/2816330536 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f95f8056380 con 0x7f95f4004850 2023-03-03T18:25:35.660 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:25:35.661 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:25:35.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.661+0000 7f960ad60700 1 -- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95e80682f0 msgr2=0x7f95e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:35.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.661+0000 7f960ad60700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95e80682f0 0x7f95e806a7a0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f95f0020040 tx=0x7f95f0008030 comp rx=0 tx=0).stop 2023-03-03T18:25:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.661+0000 7f960ad60700 1 -- 172.21.15.7:0/2816330536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 msgr2=0x7f96040c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.661+0000 7f960ad60700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 0x7f96040c6c30 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f95f8011aa0 tx=0x7f95f8011e60 comp rx=0 tx=0).stop 2023-03-03T18:25:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.661+0000 7f960ad60700 1 -- 172.21.15.7:0/2816330536 shutdown_connections 2023-03-03T18:25:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.661+0000 7f960ad60700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f960414e6e0 0x7f96041463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.661+0000 7f960ad60700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f95e80682f0 0x7f95e806a7a0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.661+0000 7f960ad60700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95f4004850 0x7f96040c6c30 unknown :-1 s=CLOSED pgs=3197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.661+0000 7f960ad60700 1 --2- 172.21.15.7:0/2816330536 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f96040c51f0 0x7f960414e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:35.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.661+0000 7f960ad60700 1 -- 172.21.15.7:0/2816330536 >> 172.21.15.7:0/2816330536 conn(0x7f96040a94d0 msgr2=0x7f9604150bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:25:35.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.662+0000 7f960ad60700 1 -- 172.21.15.7:0/2816330536 shutdown_connections 2023-03-03T18:25:35.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:35.662+0000 7f960ad60700 1 -- 172.21.15.7:0/2816330536 wait complete. 2023-03-03T18:25:35.664 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:25:35.674 INFO:tasks.mds_thrash.fs.[b]:waiting for 23 secs before thrashing 2023-03-03T18:25:37.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:25:37.780 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:25:58.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:25:58.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.816+0000 7f7cf55c8700 1 Processor -- start 2023-03-03T18:25:58.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.816+0000 7f7cf55c8700 1 -- start start 2023-03-03T18:25:58.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cf55c8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cf00caa30 0x7f7cf00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:58.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cf55c8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7cf0154ff0 0x7f7cf00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:58.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cf55c8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf00c7310 con 0x7f7cf00c00e0 2023-03-03T18:25:58.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cf55c8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf00bfd10 con 0x7f7cf0154ff0 2023-03-03T18:25:58.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cf55c8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf00bfa30 con 0x7f7cf00caa30 2023-03-03T18:25:58.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cef7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7cf0154ff0 0x7f7cf00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:58.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cee7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7cf00c00e0 0x7f7cf00c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59410/0 (socket says 172.21.15.7:59410) 2023-03-03T18:25:58.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cee7fc700 1 -- 172.21.15.7:0/2632569578 learned_addr learned my addr 172.21.15.7:0/2632569578 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:25:58.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7ceeffd700 1 --2- 172.21.15.7:0/2632569578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cf00caa30 0x7f7cf00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:58.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cedffb700 1 -- 172.21.15.7:0/2632569578 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335434787 0 0) 0x7f7cf00c7310 con 0x7f7cf00c00e0 2023-03-03T18:25:58.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cedffb700 1 -- 172.21.15.7:0/2632569578 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd80035b0 con 0x7f7cf00c00e0 2023-03-03T18:25:58.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cef7fe700 1 -- 172.21.15.7:0/2632569578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cf00caa30 msgr2=0x7f7cf00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:25:58.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cef7fe700 1 --2- 172.21.15.7:0/2632569578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cf00caa30 0x7f7cf00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:58.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cef7fe700 1 -- 172.21.15.7:0/2632569578 >> v1:172.21.15.7:6789/0 conn(0x7f7cf00c00e0 legacy=0x7f7cf00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:58.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.817+0000 7f7cef7fe700 1 -- 172.21.15.7:0/2632569578 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf0154510 con 0x7f7cf0154ff0 2023-03-03T18:25:58.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cef7fe700 1 --2- 172.21.15.7:0/2632569578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf0154ff0 0x7f7cf00cae00 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009d90 tx=0x7f7ce800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d3af605ff271e2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:25:58.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cedffb700 1 -- 172.21.15.7:0/2632569578 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800e070 con 0x7f7cf0154ff0 2023-03-03T18:25:58.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cedffb700 1 -- 172.21.15.7:0/2632569578 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce800bb20 con 0x7f7cf0154ff0 2023-03-03T18:25:58.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cedffb700 1 -- 172.21.15.7:0/2632569578 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8012430 con 0x7f7cf0154ff0 2023-03-03T18:25:58.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cf55c8700 1 -- 172.21.15.7:0/2632569578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf0154ff0 msgr2=0x7f7cf00cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:58.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cf55c8700 1 --2- 172.21.15.7:0/2632569578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf0154ff0 0x7f7cf00cae00 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009d90 tx=0x7f7ce800b330 comp rx=0 tx=0).stop 2023-03-03T18:25:58.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cf55c8700 1 -- 172.21.15.7:0/2632569578 shutdown_connections 2023-03-03T18:25:58.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cf55c8700 1 --2- 172.21.15.7:0/2632569578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf0154ff0 0x7f7cf00cae00 unknown :-1 s=CLOSED pgs=3199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:58.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cf55c8700 1 --2- 172.21.15.7:0/2632569578 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cf00caa30 0x7f7cf00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:58.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cf55c8700 1 -- 172.21.15.7:0/2632569578 >> 172.21.15.7:0/2632569578 conn(0x7f7cf0066e40 msgr2=0x7f7cf0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:25:58.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cf55c8700 1 -- 172.21.15.7:0/2632569578 shutdown_connections 2023-03-03T18:25:58.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.818+0000 7f7cf55c8700 1 -- 172.21.15.7:0/2632569578 wait complete. 2023-03-03T18:25:58.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.820+0000 7f7cf55c8700 1 Processor -- start 2023-03-03T18:25:58.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.820+0000 7f7cf55c8700 1 -- start start 2023-03-03T18:25:58.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.820+0000 7f7cf55c8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cf00c00e0 0x7f7cf01ed330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:58.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.820+0000 7f7cf55c8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf00caa30 0x7f7cf00c39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:58.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.820+0000 7f7cf55c8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cf0154ff0 0x7f7cf00c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:58.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.820+0000 7f7cf55c8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01569d0 con 0x7f7cf00caa30 2023-03-03T18:25:58.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.820+0000 7f7cf55c8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0156480 con 0x7f7cf0154ff0 2023-03-03T18:25:58.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.820+0000 7f7cf55c8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf00c4450 con 0x7f7cf00c00e0 2023-03-03T18:25:58.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cee7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf00caa30 0x7f7cf00c39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:58.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cee7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf00caa30 0x7f7cf00c39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37204/0 (socket says 172.21.15.7:37204) 2023-03-03T18:25:58.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cee7fc700 1 -- 172.21.15.7:0/253980303 learned_addr learned my addr 172.21.15.7:0/253980303 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:25:58.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7ceeffd700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cf00c00e0 0x7f7cf01ed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:58.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cef7fe700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cf0154ff0 0x7f7cf00c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:58.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cee7fc700 1 -- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cf00c00e0 msgr2=0x7f7cf01ed330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:58.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cee7fc700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cf00c00e0 0x7f7cf01ed330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:58.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cee7fc700 1 -- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cf0154ff0 msgr2=0x7f7cf00c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:58.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cee7fc700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cf0154ff0 0x7f7cf00c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:58.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cee7fc700 1 -- 172.21.15.7:0/253980303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce8009a70 con 0x7f7cf00caa30 2023-03-03T18:25:58.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cef7fe700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cf0154ff0 0x7f7cf00c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:25:58.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cee7fc700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf00caa30 0x7f7cf00c39d0 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0001260 tx=0x7f7ce000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:25:58.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cd77fe700 1 -- 172.21.15.7:0/253980303 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00074e0 con 0x7f7cf00caa30 2023-03-03T18:25:58.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cd77fe700 1 -- 172.21.15.7:0/253980303 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce0007640 con 0x7f7cf00caa30 2023-03-03T18:25:58.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cf55c8700 1 -- 172.21.15.7:0/253980303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf0154ad0 con 0x7f7cf00caa30 2023-03-03T18:25:58.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cd77fe700 1 -- 172.21.15.7:0/253980303 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0005a50 con 0x7f7cf00caa30 2023-03-03T18:25:58.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.821+0000 7f7cf55c8700 1 -- 172.21.15.7:0/253980303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf01547f0 con 0x7f7cf00caa30 2023-03-03T18:25:58.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.822+0000 7f7cf55c8700 1 -- 172.21.15.7:0/253980303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf00bf650 con 0x7f7cf00caa30 2023-03-03T18:25:58.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.823+0000 7f7cd77fe700 1 -- 172.21.15.7:0/253980303 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7ce0019030 con 0x7f7cf00caa30 2023-03-03T18:25:58.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.823+0000 7f7cd77fe700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cdc068320 0x7f7cdc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:25:58.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.823+0000 7f7cd77fe700 1 -- 172.21.15.7:0/253980303 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7ce00854e0 con 0x7f7cf00caa30 2023-03-03T18:25:58.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.823+0000 7f7ceeffd700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cdc068320 0x7f7cdc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:25:58.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.824+0000 7f7ceeffd700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cdc068320 0x7f7cdc06a7d0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009ab0 tx=0x7f7ce4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:25:58.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:58.828+0000 7f7cd77fe700 1 -- 172.21.15.7:0/253980303 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7ce0051880 con 0x7f7cf00caa30 2023-03-03T18:25:59.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.067+0000 7f7cf55c8700 1 -- 172.21.15.7:0/253980303 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cf004e6e0 con 0x7f7cf00caa30 2023-03-03T18:25:59.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.067+0000 7f7cd77fe700 1 -- 172.21.15.7:0/253980303 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7ce0007e60 con 0x7f7cf00caa30 2023-03-03T18:25:59.067 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:25:59.067 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:25:59.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 -- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cdc068320 msgr2=0x7f7cdc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:59.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cdc068320 0x7f7cdc06a7d0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009ab0 tx=0x7f7ce4008040 comp rx=0 tx=0).stop 2023-03-03T18:25:59.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 -- 172.21.15.7:0/253980303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf00caa30 msgr2=0x7f7cf00c39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:25:59.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf00caa30 0x7f7cf00c39d0 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0001260 tx=0x7f7ce000c860 comp rx=0 tx=0).stop 2023-03-03T18:25:59.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 -- 172.21.15.7:0/253980303 shutdown_connections 2023-03-03T18:25:59.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cf0154ff0 0x7f7cf00c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:59.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7cf00c00e0 0x7f7cf01ed330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:59.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7cdc068320 0x7f7cdc06a7d0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:59.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 --2- 172.21.15.7:0/253980303 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7cf00caa30 0x7f7cf00c39d0 unknown :-1 s=CLOSED pgs=3200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:25:59.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 -- 172.21.15.7:0/253980303 >> 172.21.15.7:0/253980303 conn(0x7f7cf0066e40 msgr2=0x7f7cf00c67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:25:59.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 -- 172.21.15.7:0/253980303 shutdown_connections 2023-03-03T18:25:59.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:25:59.069+0000 7f7cf55c8700 1 -- 172.21.15.7:0/253980303 wait complete. 2023-03-03T18:25:59.071 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:25:59.081 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2023-03-03T18:26:07.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:26:07.867 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:26:38.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:26:38.025 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:27:08.132 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:27:08.136 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:27:09.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:27:09.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.907+0000 7f1f21af3700 1 Processor -- start 2023-03-03T18:27:09.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.907+0000 7f1f21af3700 1 -- start start 2023-03-03T18:27:09.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.908+0000 7f1f21af3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1f1c14e900 0x7f1f1c15ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:09.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.908+0000 7f1f21af3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c1510d0 con 0x7f1f1c072250 2023-03-03T18:27:09.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.908+0000 7f1f21af3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c0c3740 con 0x7f1f1c071e80 2023-03-03T18:27:09.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.908+0000 7f1f21af3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f1c070670 con 0x7f1f1c14e900 2023-03-03T18:27:09.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.908+0000 7f1f1affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1f1c072250 0x7f1f1c14e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59468/0 (socket says 172.21.15.7:59468) 2023-03-03T18:27:09.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.908+0000 7f1f1affd700 1 -- 172.21.15.7:0/2696172358 learned_addr learned my addr 172.21.15.7:0/2696172358 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:27:09.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.908+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129993039 0 0) 0x7f1f1c1510d0 con 0x7f1f1c072250 2023-03-03T18:27:09.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.908+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f080035b0 con 0x7f1f1c072250 2023-03-03T18:27:09.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.908+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/2696172358 >> v2:172.21.15.143:3300/0 conn(0x7f1f1c14e900 0x7f1f1c15ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:09.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4292401107 0 0) 0x7f1f080035b0 con 0x7f1f1c072250 2023-03-03T18:27:09.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 >> v2:172.21.15.143:3300/0 conn(0x7f1f1c14e900 msgr2=0x7f1f1c15ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:27:09.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 --2- 172.21.15.7:0/2696172358 >> v2:172.21.15.143:3300/0 conn(0x7f1f1c14e900 0x7f1f1c15ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:09.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 >> v1:172.21.15.143:6789/0 conn(0x7f1f1c071e80 legacy=0x7f1f1c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:09.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c0c39e0 con 0x7f1f1c072250 2023-03-03T18:27:09.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f1f04001670 con 0x7f1f1c072250 2023-03-03T18:27:09.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 >> v1:172.21.15.7:6789/0 conn(0x7f1f1c072250 legacy=0x7f1f1c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:09.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f08003a70 0x7f1f08003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:09.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f08004850 0x7f1f08004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:09.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f08005250 0x7f1f0800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:09.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c070670 con 0x7f1f08005250 2023-03-03T18:27:09.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0800c190 con 0x7f1f08003a70 2023-03-03T18:27:09.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0800c350 con 0x7f1f08004850 2023-03-03T18:27:09.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f08005250 0x7f1f0800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:09.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1b7fe700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f08003a70 0x7f1f08003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:09.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1affd700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f08004850 0x7f1f08004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:09.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1bfff700 1 -- 172.21.15.7:0/2696172358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f08004850 msgr2=0x7f1f08004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:27:09.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f08004850 0x7f1f08004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:09.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1bfff700 1 -- 172.21.15.7:0/2696172358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f08003a70 msgr2=0x7f1f08003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:27:09.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f08003a70 0x7f1f08003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:09.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1bfff700 1 -- 172.21.15.7:0/2696172358 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f080035b0 con 0x7f1f08005250 2023-03-03T18:27:09.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1bfff700 1 -- 172.21.15.7:0/2696172358 reap_dead start 2023-03-03T18:27:09.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f08005250 0x7f1f0800ba10 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0713e0 tx=0x7f1f08005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ccf0abf868ef0d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:27:09.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.909+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14003c20 con 0x7f1f08005250 2023-03-03T18:27:09.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.910+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1400f040 con 0x7f1f08005250 2023-03-03T18:27:09.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.910+0000 7f1f1a7fc700 1 -- 172.21.15.7:0/2696172358 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1400e9d0 con 0x7f1f08005250 2023-03-03T18:27:09.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.910+0000 7f1f21af3700 1 -- 172.21.15.7:0/2696172358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f08005250 msgr2=0x7f1f0800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:09.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.910+0000 7f1f21af3700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f08005250 0x7f1f0800ba10 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0713e0 tx=0x7f1f08005880 comp rx=0 tx=0).stop 2023-03-03T18:27:09.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.910+0000 7f1f21af3700 1 -- 172.21.15.7:0/2696172358 shutdown_connections 2023-03-03T18:27:09.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.910+0000 7f1f21af3700 1 --2- 172.21.15.7:0/2696172358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f08005250 0x7f1f0800ba10 unknown :-1 s=CLOSED pgs=3203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:09.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.910+0000 7f1f21af3700 1 -- 172.21.15.7:0/2696172358 >> 172.21.15.7:0/2696172358 conn(0x7f1f1c0a94d0 msgr2=0x7f1f1c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:27:09.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.910+0000 7f1f21af3700 1 -- 172.21.15.7:0/2696172358 shutdown_connections 2023-03-03T18:27:09.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.910+0000 7f1f21af3700 1 -- 172.21.15.7:0/2696172358 wait complete. 2023-03-03T18:27:09.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.912+0000 7f1f21af3700 1 Processor -- start 2023-03-03T18:27:09.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.912+0000 7f1f21af3700 1 -- start start 2023-03-03T18:27:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f21af3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f08005250 0x7f1f1c1eae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f21af3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c0ac6b0 0x7f1f1c0c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f21af3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ace80 0x7f1f1c1e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f21af3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c1ea9f0 con 0x7f1f1c0ace80 2023-03-03T18:27:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f21af3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0aa1d0 con 0x7f1f1c0ac6b0 2023-03-03T18:27:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f21af3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c1ea650 con 0x7f1f08005250 2023-03-03T18:27:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ace80 0x7f1f1c1e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:09.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ace80 0x7f1f1c1e6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37270/0 (socket says 172.21.15.7:37270) 2023-03-03T18:27:09.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1bfff700 1 -- 172.21.15.7:0/1710962397 learned_addr learned my addr 172.21.15.7:0/1710962397 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:27:09.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1affd700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c0ac6b0 0x7f1f1c0c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:09.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1b7fe700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f08005250 0x7f1f1c1eae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:09.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1bfff700 1 -- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f08005250 msgr2=0x7f1f1c1eae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:09.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f08005250 0x7f1f1c1eae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:09.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1bfff700 1 -- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c0ac6b0 msgr2=0x7f1f1c0c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:09.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c0ac6b0 0x7f1f1c0c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:09.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1bfff700 1 -- 172.21.15.7:0/1710962397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c06c9a0 con 0x7f1f1c0ace80 2023-03-03T18:27:09.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1b7fe700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f08005250 0x7f1f1c1eae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:27:09.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.913+0000 7f1f1bfff700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ace80 0x7f1f1c1e6f40 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c072670 tx=0x7f1f14009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:27:09.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.914+0000 7f1f13fff700 1 -- 172.21.15.7:0/1710962397 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f14014d20 con 0x7f1f1c0ace80 2023-03-03T18:27:09.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.914+0000 7f1f13fff700 1 -- 172.21.15.7:0/1710962397 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f14009410 con 0x7f1f1c0ace80 2023-03-03T18:27:09.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.914+0000 7f1f21af3700 1 -- 172.21.15.7:0/1710962397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c151300 con 0x7f1f1c0ace80 2023-03-03T18:27:09.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.914+0000 7f1f13fff700 1 -- 172.21.15.7:0/1710962397 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1401d630 con 0x7f1f1c0ace80 2023-03-03T18:27:09.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.914+0000 7f1f21af3700 1 -- 172.21.15.7:0/1710962397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c06cb30 con 0x7f1f1c0ace80 2023-03-03T18:27:09.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.915+0000 7f1f21af3700 1 -- 172.21.15.7:0/1710962397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c1ea4b0 con 0x7f1f1c0ace80 2023-03-03T18:27:09.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.915+0000 7f1f13fff700 1 -- 172.21.15.7:0/1710962397 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1f1402f070 con 0x7f1f1c0ace80 2023-03-03T18:27:09.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.916+0000 7f1f13fff700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1efc0682f0 0x7f1efc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:09.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.916+0000 7f1f13fff700 1 -- 172.21.15.7:0/1710962397 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1f140876d0 con 0x7f1f1c0ace80 2023-03-03T18:27:09.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.916+0000 7f1f1b7fe700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1efc0682f0 0x7f1efc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:09.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.917+0000 7f1f1b7fe700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1efc0682f0 0x7f1efc06a7a0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c151110 tx=0x7f1f0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:27:09.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:09.920+0000 7f1f13fff700 1 -- 172.21.15.7:0/1710962397 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1f1c1ea4b0 con 0x7f1f1c0ace80 2023-03-03T18:27:10.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.161+0000 7f1f21af3700 1 -- 172.21.15.7:0/1710962397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f1c1ea4b0 con 0x7f1f1c0ace80 2023-03-03T18:27:10.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.161+0000 7f1f13fff700 1 -- 172.21.15.7:0/1710962397 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1f1c1ea4b0 con 0x7f1f1c0ace80 2023-03-03T18:27:10.161 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:27:10.162 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:27:10.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 -- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1efc0682f0 msgr2=0x7f1efc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:10.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1efc0682f0 0x7f1efc06a7a0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c151110 tx=0x7f1f0c009040 comp rx=0 tx=0).stop 2023-03-03T18:27:10.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 -- 172.21.15.7:0/1710962397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ace80 msgr2=0x7f1f1c1e6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ace80 0x7f1f1c1e6f40 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c072670 tx=0x7f1f14009750 comp rx=0 tx=0).stop 2023-03-03T18:27:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 -- 172.21.15.7:0/1710962397 shutdown_connections 2023-03-03T18:27:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1efc0682f0 0x7f1efc06a7a0 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f1c0ace80 0x7f1f1c1e6f40 unknown :-1 s=CLOSED pgs=3204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f08005250 0x7f1f1c1eae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 --2- 172.21.15.7:0/1710962397 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f1c0ac6b0 0x7f1f1c0c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 -- 172.21.15.7:0/1710962397 >> 172.21.15.7:0/1710962397 conn(0x7f1f1c0a94d0 msgr2=0x7f1f1c06db00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:27:10.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 -- 172.21.15.7:0/1710962397 shutdown_connections 2023-03-03T18:27:10.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:10.163+0000 7f1f21af3700 1 -- 172.21.15.7:0/1710962397 wait complete. 2023-03-03T18:27:10.165 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:27:10.175 INFO:tasks.mds_thrash.fs.[a]:waiting for 114 secs before thrashing 2023-03-03T18:27:25.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:27:25.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.223+0000 7f94c53c5700 1 Processor -- start 2023-03-03T18:27:25.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.223+0000 7f94c53c5700 1 -- start start 2023-03-03T18:27:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94c53c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00caa30 0x7f94c00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94c53c5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94c0154ff0 0x7f94c00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94c53c5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c00c7310 con 0x7f94c00c00e0 2023-03-03T18:27:25.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94c53c5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c00bfd10 con 0x7f94c0154ff0 2023-03-03T18:27:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94c53c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c00bfa30 con 0x7f94c00caa30 2023-03-03T18:27:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94bf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94c0154ff0 0x7f94c00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94bf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f94c0154ff0 0x7f94c00cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37282/0 (socket says 172.21.15.7:37282) 2023-03-03T18:27:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94bf7fe700 1 -- 172.21.15.7:0/3248462168 learned_addr learned my addr 172.21.15.7:0/3248462168 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:27:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94beffd700 1 --2- 172.21.15.7:0/3248462168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00caa30 0x7f94c00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94bf7fe700 1 -- 172.21.15.7:0/3248462168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00caa30 msgr2=0x7f94c00c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:27:25.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94bf7fe700 1 --2- 172.21.15.7:0/3248462168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00caa30 0x7f94c00c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94bf7fe700 1 -- 172.21.15.7:0/3248462168 >> v1:172.21.15.143:6789/0 conn(0x7f94c00c00e0 legacy=0x7f94c00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94bf7fe700 1 -- 172.21.15.7:0/3248462168 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c0154510 con 0x7f94c0154ff0 2023-03-03T18:27:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.224+0000 7f94beffd700 1 --2- 172.21.15.7:0/3248462168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00caa30 0x7f94c00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:27:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94bf7fe700 1 --2- 172.21.15.7:0/3248462168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c0154ff0 0x7f94c00cae00 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009e50 tx=0x7f94b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1115bd0f11b8c7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:27:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94bdffb700 1 -- 172.21.15.7:0/3248462168 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b800e070 con 0x7f94c0154ff0 2023-03-03T18:27:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94bdffb700 1 -- 172.21.15.7:0/3248462168 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b800b9c0 con 0x7f94c0154ff0 2023-03-03T18:27:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94bdffb700 1 -- 172.21.15.7:0/3248462168 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b800bd70 con 0x7f94c0154ff0 2023-03-03T18:27:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94c53c5700 1 -- 172.21.15.7:0/3248462168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c0154ff0 msgr2=0x7f94c00cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94c53c5700 1 --2- 172.21.15.7:0/3248462168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c0154ff0 0x7f94c00cae00 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009e50 tx=0x7f94b800b330 comp rx=0 tx=0).stop 2023-03-03T18:27:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94c53c5700 1 -- 172.21.15.7:0/3248462168 shutdown_connections 2023-03-03T18:27:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94c53c5700 1 --2- 172.21.15.7:0/3248462168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c0154ff0 0x7f94c00cae00 unknown :-1 s=CLOSED pgs=3205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94c53c5700 1 --2- 172.21.15.7:0/3248462168 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00caa30 0x7f94c00c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94c53c5700 1 -- 172.21.15.7:0/3248462168 >> 172.21.15.7:0/3248462168 conn(0x7f94c0066e40 msgr2=0x7f94c0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:27:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94c53c5700 1 -- 172.21.15.7:0/3248462168 shutdown_connections 2023-03-03T18:27:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.225+0000 7f94c53c5700 1 -- 172.21.15.7:0/3248462168 wait complete. 2023-03-03T18:27:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.227+0000 7f94c53c5700 1 Processor -- start 2023-03-03T18:27:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.227+0000 7f94c53c5700 1 -- start start 2023-03-03T18:27:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.227+0000 7f94c53c5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00c00e0 0x7f94c01557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.227+0000 7f94c53c5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c00caa30 0x7f94c0157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.227+0000 7f94c53c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c0154ff0 0x7f94c0157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.227+0000 7f94c53c5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c014d410 con 0x7f94c00caa30 2023-03-03T18:27:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.227+0000 7f94c53c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c0158920 con 0x7f94c0154ff0 2023-03-03T18:27:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.227+0000 7f94c53c5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c01eaa80 con 0x7f94c00c00e0 2023-03-03T18:27:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94be7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c00caa30 0x7f94c0157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94be7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c00caa30 0x7f94c0157130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37286/0 (socket says 172.21.15.7:37286) 2023-03-03T18:27:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94be7fc700 1 -- 172.21.15.7:0/3294160323 learned_addr learned my addr 172.21.15.7:0/3294160323 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:27:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94bf7fe700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c0154ff0 0x7f94c0157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94beffd700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00c00e0 0x7f94c01557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94be7fc700 1 -- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00c00e0 msgr2=0x7f94c01557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94be7fc700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00c00e0 0x7f94c01557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94be7fc700 1 -- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c0154ff0 msgr2=0x7f94c0157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94be7fc700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c0154ff0 0x7f94c0157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94be7fc700 1 -- 172.21.15.7:0/3294160323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b8009b30 con 0x7f94c00caa30 2023-03-03T18:27:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94bf7fe700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c0154ff0 0x7f94c0157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:27:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94be7fc700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c00caa30 0x7f94c0157130 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7f94b000c4d0 tx=0x7f94b000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:27:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94a77fe700 1 -- 172.21.15.7:0/3294160323 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b00052d0 con 0x7f94c00caa30 2023-03-03T18:27:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94a77fe700 1 -- 172.21.15.7:0/3294160323 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0005430 con 0x7f94c00caa30 2023-03-03T18:27:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94c53c5700 1 -- 172.21.15.7:0/3294160323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c0154ad0 con 0x7f94c00caa30 2023-03-03T18:27:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94a77fe700 1 -- 172.21.15.7:0/3294160323 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b00038f0 con 0x7f94c00caa30 2023-03-03T18:27:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.228+0000 7f94c53c5700 1 -- 172.21.15.7:0/3294160323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c01547f0 con 0x7f94c00caa30 2023-03-03T18:27:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.229+0000 7f94c53c5700 1 -- 172.21.15.7:0/3294160323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c00bf650 con 0x7f94c00caa30 2023-03-03T18:27:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.230+0000 7f94a77fe700 1 -- 172.21.15.7:0/3294160323 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f94b0019030 con 0x7f94c00caa30 2023-03-03T18:27:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.230+0000 7f94a77fe700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94ac0682c0 0x7f94ac06a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:27:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.230+0000 7f94a77fe700 1 -- 172.21.15.7:0/3294160323 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f94b00854e0 con 0x7f94c00caa30 2023-03-03T18:27:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.230+0000 7f94beffd700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94ac0682c0 0x7f94ac06a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:27:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.231+0000 7f94beffd700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94ac0682c0 0x7f94ac06a770 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f94c0147ad0 tx=0x7f94b4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:27:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.234+0000 7f94a77fe700 1 -- 172.21.15.7:0/3294160323 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f94b0051790 con 0x7f94c00caa30 2023-03-03T18:27:25.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.470+0000 7f94c53c5700 1 -- 172.21.15.7:0/3294160323 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94c0074bb0 con 0x7f94c00caa30 2023-03-03T18:27:25.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.470+0000 7f94a77fe700 1 -- 172.21.15.7:0/3294160323 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f94b0054db0 con 0x7f94c00caa30 2023-03-03T18:27:25.471 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:27:25.471 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:27:25.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 -- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94ac0682c0 msgr2=0x7f94ac06a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:25.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94ac0682c0 0x7f94ac06a770 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f94c0147ad0 tx=0x7f94b4008040 comp rx=0 tx=0).stop 2023-03-03T18:27:25.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 -- 172.21.15.7:0/3294160323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c00caa30 msgr2=0x7f94c0157130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:27:25.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c00caa30 0x7f94c0157130 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7f94b000c4d0 tx=0x7f94b000c890 comp rx=0 tx=0).stop 2023-03-03T18:27:25.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 -- 172.21.15.7:0/3294160323 shutdown_connections 2023-03-03T18:27:25.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c0154ff0 0x7f94c0157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:25.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f94c00c00e0 0x7f94c01557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:25.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f94ac0682c0 0x7f94ac06a770 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:25.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 --2- 172.21.15.7:0/3294160323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f94c00caa30 0x7f94c0157130 unknown :-1 s=CLOSED pgs=3206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:27:25.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 -- 172.21.15.7:0/3294160323 >> 172.21.15.7:0/3294160323 conn(0x7f94c0066e40 msgr2=0x7f94c00736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:27:25.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 -- 172.21.15.7:0/3294160323 shutdown_connections 2023-03-03T18:27:25.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:27:25.473+0000 7f94c53c5700 1 -- 172.21.15.7:0/3294160323 wait complete. 2023-03-03T18:27:25.475 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:27:25.485 INFO:tasks.mds_thrash.fs.[b]:waiting for 91 secs before thrashing 2023-03-03T18:27:38.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:27:38.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:28:08.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:28:08.471 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:28:38.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:28:38.608 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:28:56.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:28:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.624+0000 7faa6e3f9700 1 Processor -- start 2023-03-03T18:28:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.624+0000 7faa6e3f9700 1 -- start start 2023-03-03T18:28:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.625+0000 7faa6e3f9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa680a9af0 0x7faa680aa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:28:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.625+0000 7faa6e3f9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa68155670 con 0x7faa680c5ee0 2023-03-03T18:28:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.625+0000 7faa6e3f9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa680c2b40 con 0x7faa68154e40 2023-03-03T18:28:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.625+0000 7faa6e3f9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa680aae70 con 0x7faa680a9af0 2023-03-03T18:28:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.625+0000 7faa67fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faa680c5ee0 0x7faa68154730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59570/0 (socket says 172.21.15.7:59570) 2023-03-03T18:28:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.625+0000 7faa67fff700 1 -- 172.21.15.7:0/3790144771 learned_addr learned my addr 172.21.15.7:0/3790144771 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:28:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.625+0000 7faa6c996700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa680a9af0 0x7faa680aa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:28:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389740746 0 0) 0x7faa68155670 con 0x7faa680c5ee0 2023-03-03T18:28:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa500035b0 con 0x7faa680c5ee0 2023-03-03T18:28:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960998003 0 0) 0x7faa680c2b40 con 0x7faa68154e40 2023-03-03T18:28:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa68155670 con 0x7faa68154e40 2023-03-03T18:28:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 787058480 0 0) 0x7faa500035b0 con 0x7faa680c5ee0 2023-03-03T18:28:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa680a9af0 msgr2=0x7faa680aa5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:28:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa680a9af0 0x7faa680aa5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:28:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 >> v1:172.21.15.143:6789/0 conn(0x7faa68154e40 legacy=0x7faa680c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:28:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa68155390 con 0x7faa680c5ee0 2023-03-03T18:28:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7faa58001670 con 0x7faa680c5ee0 2023-03-03T18:28:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 >> v1:172.21.15.7:6789/0 conn(0x7faa680c5ee0 legacy=0x7faa68154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:28:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa50003c30 0x7faa50004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:28:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa680a9af0 0x7faa50004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:28:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 0x7faa50005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:28:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5000bc70 con 0x7faa50004de0 2023-03-03T18:28:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5000bde0 con 0x7faa50003c30 2023-03-03T18:28:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5000bfa0 con 0x7faa680a9af0 2023-03-03T18:28:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa6c996700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 0x7faa50005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:28:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa67fff700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa50003c30 0x7faa50004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:28:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa677fe700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa680a9af0 0x7faa50004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:28:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa6c996700 1 -- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa680a9af0 msgr2=0x7faa50004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:28:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa6c996700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa680a9af0 0x7faa50004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:28:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa6c996700 1 -- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa50003c30 msgr2=0x7faa50004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:28:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa6c996700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa50003c30 0x7faa50004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:28:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.626+0000 7faa6c996700 1 -- 172.21.15.7:0/3790144771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa500035b0 con 0x7faa50004de0 2023-03-03T18:28:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa6c996700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 0x7faa50005250 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7faa60009f90 tx=0x7faa50005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88878ec0d5778e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:28:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6000ec30 con 0x7faa50004de0 2023-03-03T18:28:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6000f040 con 0x7faa50004de0 2023-03-03T18:28:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa66ffd700 1 -- 172.21.15.7:0/3790144771 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60014480 con 0x7faa50004de0 2023-03-03T18:28:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa6e3f9700 1 -- 172.21.15.7:0/3790144771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 msgr2=0x7faa50005250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:28:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa6e3f9700 1 --2- 172.21.15.7:0/3790144771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 0x7faa50005250 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7faa60009f90 tx=0x7faa50005880 comp rx=0 tx=0).stop 2023-03-03T18:28:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa6c996700 1 -- 172.21.15.7:0/3790144771 reap_dead start 2023-03-03T18:28:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa6e3f9700 1 -- 172.21.15.7:0/3790144771 shutdown_connections 2023-03-03T18:28:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa6e3f9700 1 -- 172.21.15.7:0/3790144771 >> 172.21.15.7:0/3790144771 conn(0x7faa680a9430 msgr2=0x7faa680c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:28:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa6e3f9700 1 -- 172.21.15.7:0/3790144771 shutdown_connections 2023-03-03T18:28:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.627+0000 7faa6e3f9700 1 -- 172.21.15.7:0/3790144771 wait complete. 2023-03-03T18:28:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.628+0000 7faa6e3f9700 1 Processor -- start 2023-03-03T18:28:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.629+0000 7faa6e3f9700 1 -- start start 2023-03-03T18:28:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.629+0000 7faa6e3f9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 0x7faa6814ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:28:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.629+0000 7faa6e3f9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa68070b50 0x7faa68070fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:28:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.629+0000 7faa6e3f9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa68150400 0x7faa68150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:28:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.629+0000 7faa6e3f9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6814ff70 con 0x7faa50004de0 2023-03-03T18:28:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.629+0000 7faa6e3f9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6814f9b0 con 0x7faa68070b50 2023-03-03T18:28:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.629+0000 7faa6e3f9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa680715a0 con 0x7faa68150400 2023-03-03T18:28:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.629+0000 7faa67fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 0x7faa6814ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:28:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.629+0000 7faa67fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 0x7faa6814ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37372/0 (socket says 172.21.15.7:37372) 2023-03-03T18:28:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.629+0000 7faa67fff700 1 -- 172.21.15.7:0/1268002041 learned_addr learned my addr 172.21.15.7:0/1268002041 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:28:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa6c996700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa68150400 0x7faa68150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:28:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa677fe700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa68070b50 0x7faa68070fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:28:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa67fff700 1 -- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa68150400 msgr2=0x7faa68150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:28:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa67fff700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa68150400 0x7faa68150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:28:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa67fff700 1 -- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa68070b50 msgr2=0x7faa68070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:28:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa67fff700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa68070b50 0x7faa68070fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:28:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa67fff700 1 -- 172.21.15.7:0/1268002041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680c1c00 con 0x7faa50004de0 2023-03-03T18:28:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa67fff700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 0x7faa6814ce80 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7faa58004f50 tx=0x7faa58007340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:28:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa64ff9700 1 -- 172.21.15.7:0/1268002041 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58012560 con 0x7faa50004de0 2023-03-03T18:28:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa64ff9700 1 -- 172.21.15.7:0/1268002041 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa580126e0 con 0x7faa50004de0 2023-03-03T18:28:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa6e3f9700 1 -- 172.21.15.7:0/1268002041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa680c1920 con 0x7faa50004de0 2023-03-03T18:28:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa64ff9700 1 -- 172.21.15.7:0/1268002041 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa58012a90 con 0x7faa50004de0 2023-03-03T18:28:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.630+0000 7faa6e3f9700 1 -- 172.21.15.7:0/1268002041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa680c62b0 con 0x7faa50004de0 2023-03-03T18:28:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.631+0000 7faa6e3f9700 1 -- 172.21.15.7:0/1268002041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa68074bb0 con 0x7faa50004de0 2023-03-03T18:28:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.632+0000 7faa64ff9700 1 -- 172.21.15.7:0/1268002041 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faa58012bf0 con 0x7faa50004de0 2023-03-03T18:28:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.632+0000 7faa64ff9700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa540682f0 0x7faa5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:28:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.632+0000 7faa64ff9700 1 -- 172.21.15.7:0/1268002041 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faa58086290 con 0x7faa50004de0 2023-03-03T18:28:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.632+0000 7faa677fe700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa540682f0 0x7faa5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:28:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.633+0000 7faa677fe700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa540682f0 0x7faa5406a7a0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0010e0 tx=0x7faa5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:28:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.637+0000 7faa64ff9700 1 -- 172.21.15.7:0/1268002041 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faa580524e0 con 0x7faa50004de0 2023-03-03T18:28:56.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.875+0000 7faa6e3f9700 1 -- 172.21.15.7:0/1268002041 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa680c15d0 con 0x7faa50004de0 2023-03-03T18:28:56.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.876+0000 7faa64ff9700 1 -- 172.21.15.7:0/1268002041 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faa58055b00 con 0x7faa50004de0 2023-03-03T18:28:56.877 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:28:56.878 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:28:56.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 -- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa540682f0 msgr2=0x7faa5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:28:56.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa540682f0 0x7faa5406a7a0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0010e0 tx=0x7faa5c009040 comp rx=0 tx=0).stop 2023-03-03T18:28:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 -- 172.21.15.7:0/1268002041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 msgr2=0x7faa6814ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:28:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 0x7faa6814ce80 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7faa58004f50 tx=0x7faa58007340 comp rx=0 tx=0).stop 2023-03-03T18:28:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 -- 172.21.15.7:0/1268002041 shutdown_connections 2023-03-03T18:28:56.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa68150400 0x7faa68150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:28:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa540682f0 0x7faa5406a7a0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:28:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa50004de0 0x7faa6814ce80 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:28:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 --2- 172.21.15.7:0/1268002041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa68070b50 0x7faa68070fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:28:56.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 -- 172.21.15.7:0/1268002041 >> 172.21.15.7:0/1268002041 conn(0x7faa680a9430 msgr2=0x7faa680c4980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:28:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 -- 172.21.15.7:0/1268002041 shutdown_connections 2023-03-03T18:28:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:28:56.878+0000 7faa6e3f9700 1 -- 172.21.15.7:0/1268002041 wait complete. 2023-03-03T18:28:56.882 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:28:56.891 INFO:tasks.mds_thrash.fs.[b]:waiting for 113 secs before thrashing 2023-03-03T18:29:04.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:29:04.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.314+0000 7f45662a9700 1 Processor -- start 2023-03-03T18:29:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.315+0000 7f45662a9700 1 -- start start 2023-03-03T18:29:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.315+0000 7f45662a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4560151ba0 0x7f45600aa720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:29:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.315+0000 7f45662a9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4560071200 0x7f45600aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:29:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.315+0000 7f45662a9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4560071cd0 con 0x7f456006f140 2023-03-03T18:29:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.315+0000 7f45662a9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4560072290 con 0x7f4560071200 2023-03-03T18:29:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.315+0000 7f45662a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4560151f70 con 0x7f4560151ba0 2023-03-03T18:29:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4560071200 0x7f45600aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:29:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4560071200 0x7f45600aac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37384/0 (socket says 172.21.15.7:37384) 2023-03-03T18:29:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455f7fe700 1 -- 172.21.15.7:0/3866595836 learned_addr learned my addr 172.21.15.7:0/3866595836 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:29:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455ffff700 1 --2- 172.21.15.7:0/3866595836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4560151ba0 0x7f45600aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:29:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455f7fe700 1 -- 172.21.15.7:0/3866595836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4560151ba0 msgr2=0x7f45600aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:29:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455f7fe700 1 --2- 172.21.15.7:0/3866595836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4560151ba0 0x7f45600aa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:29:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455f7fe700 1 -- 172.21.15.7:0/3866595836 >> v1:172.21.15.143:6789/0 conn(0x7f456006f140 legacy=0x7f4560154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:29:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455f7fe700 1 -- 172.21.15.7:0/3866595836 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4560072720 con 0x7f4560071200 2023-03-03T18:29:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455f7fe700 1 --2- 172.21.15.7:0/3866595836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560071200 0x7f45600aac60 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7f45540098f0 tx=0x7f4554009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=939bba3edb54762a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:29:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455effd700 1 -- 172.21.15.7:0/3866595836 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554004230 con 0x7f4560071200 2023-03-03T18:29:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455effd700 1 -- 172.21.15.7:0/3866595836 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4554004390 con 0x7f4560071200 2023-03-03T18:29:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.316+0000 7f455effd700 1 -- 172.21.15.7:0/3866595836 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45540036a0 con 0x7f4560071200 2023-03-03T18:29:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.317+0000 7f45662a9700 1 -- 172.21.15.7:0/3866595836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560071200 msgr2=0x7f45600aac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:29:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.317+0000 7f45662a9700 1 --2- 172.21.15.7:0/3866595836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560071200 0x7f45600aac60 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7f45540098f0 tx=0x7f4554009c00 comp rx=0 tx=0).stop 2023-03-03T18:29:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.317+0000 7f45662a9700 1 -- 172.21.15.7:0/3866595836 shutdown_connections 2023-03-03T18:29:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.317+0000 7f45662a9700 1 --2- 172.21.15.7:0/3866595836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4560151ba0 0x7f45600aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:29:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.317+0000 7f45662a9700 1 --2- 172.21.15.7:0/3866595836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560071200 0x7f45600aac60 unknown :-1 s=CLOSED pgs=3210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:29:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.317+0000 7f45662a9700 1 -- 172.21.15.7:0/3866595836 >> 172.21.15.7:0/3866595836 conn(0x7f45600736f0 msgr2=0x7f4560073af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:29:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.317+0000 7f45662a9700 1 -- 172.21.15.7:0/3866595836 shutdown_connections 2023-03-03T18:29:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.317+0000 7f45662a9700 1 -- 172.21.15.7:0/3866595836 wait complete. 2023-03-03T18:29:04.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.318+0000 7f45662a9700 1 Processor -- start 2023-03-03T18:29:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.318+0000 7f45662a9700 1 -- start start 2023-03-03T18:29:04.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f45662a9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f456006f140 0x7f456014a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:29:04.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f45662a9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4560071200 0x7f4560148f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:29:04.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f45662a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560151ba0 0x7f4560149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:29:04.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f45662a9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4560148200 con 0x7f4560151ba0 2023-03-03T18:29:04.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f45662a9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456006eb30 con 0x7f4560071200 2023-03-03T18:29:04.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f45662a9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45601ee260 con 0x7f456006f140 2023-03-03T18:29:04.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f4564846700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560151ba0 0x7f4560149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:29:04.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f4564846700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560151ba0 0x7f4560149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37390/0 (socket says 172.21.15.7:37390) 2023-03-03T18:29:04.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f4564846700 1 -- 172.21.15.7:0/452777880 learned_addr learned my addr 172.21.15.7:0/452777880 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:29:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f455ffff700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f456006f140 0x7f456014a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:29:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f455f7fe700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4560071200 0x7f4560148f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:29:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f4564846700 1 -- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f456006f140 msgr2=0x7f456014a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:29:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f4564846700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f456006f140 0x7f456014a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:29:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f4564846700 1 -- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4560071200 msgr2=0x7f4560148f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:29:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f4564846700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4560071200 0x7f4560148f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:29:04.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.319+0000 7f4564846700 1 -- 172.21.15.7:0/452777880 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4554009550 con 0x7f4560151ba0 2023-03-03T18:29:04.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.320+0000 7f4564846700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560151ba0 0x7f4560149470 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f455800b9e0 tx=0x7f455800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:29:04.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.320+0000 7f455cff9700 1 -- 172.21.15.7:0/452777880 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455800c9b0 con 0x7f4560151ba0 2023-03-03T18:29:04.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.320+0000 7f455cff9700 1 -- 172.21.15.7:0/452777880 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455800cb50 con 0x7f4560151ba0 2023-03-03T18:29:04.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.320+0000 7f455cff9700 1 -- 172.21.15.7:0/452777880 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558014440 con 0x7f4560151ba0 2023-03-03T18:29:04.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.320+0000 7f45662a9700 1 -- 172.21.15.7:0/452777880 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4560071950 con 0x7f4560151ba0 2023-03-03T18:29:04.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.320+0000 7f45662a9700 1 -- 172.21.15.7:0/452777880 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4560071670 con 0x7f4560151ba0 2023-03-03T18:29:04.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.321+0000 7f45662a9700 1 -- 172.21.15.7:0/452777880 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4560071f20 con 0x7f4560151ba0 2023-03-03T18:29:04.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.321+0000 7f455cff9700 1 -- 172.21.15.7:0/452777880 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f455801b030 con 0x7f4560151ba0 2023-03-03T18:29:04.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.321+0000 7f455cff9700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f454c0682f0 0x7f454c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:29:04.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.322+0000 7f455cff9700 1 -- 172.21.15.7:0/452777880 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4558085be0 con 0x7f4560151ba0 2023-03-03T18:29:04.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.322+0000 7f455ffff700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f454c0682f0 0x7f454c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:29:04.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.322+0000 7f455ffff700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f454c0682f0 0x7f454c06a7a0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f4550009930 tx=0x7f4550008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:29:04.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.326+0000 7f455cff9700 1 -- 172.21.15.7:0/452777880 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f455804d5e0 con 0x7f4560151ba0 2023-03-03T18:29:04.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.565+0000 7f45662a9700 1 -- 172.21.15.7:0/452777880 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4560074bb0 con 0x7f4560151ba0 2023-03-03T18:29:04.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.566+0000 7f455cff9700 1 -- 172.21.15.7:0/452777880 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4558052030 con 0x7f4560151ba0 2023-03-03T18:29:04.566 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:29:04.566 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:29:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 -- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f454c0682f0 msgr2=0x7f454c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:29:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f454c0682f0 0x7f454c06a7a0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f4550009930 tx=0x7f4550008040 comp rx=0 tx=0).stop 2023-03-03T18:29:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 -- 172.21.15.7:0/452777880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560151ba0 msgr2=0x7f4560149470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:29:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560151ba0 0x7f4560149470 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f455800b9e0 tx=0x7f455800bda0 comp rx=0 tx=0).stop 2023-03-03T18:29:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 -- 172.21.15.7:0/452777880 shutdown_connections 2023-03-03T18:29:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f454c0682f0 0x7f454c06a7a0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:29:04.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4560151ba0 0x7f4560149470 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:29:04.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f456006f140 0x7f456014a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:29:04.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 --2- 172.21.15.7:0/452777880 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4560071200 0x7f4560148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:29:04.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 -- 172.21.15.7:0/452777880 >> 172.21.15.7:0/452777880 conn(0x7f45600736f0 msgr2=0x7f456014eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:29:04.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 -- 172.21.15.7:0/452777880 shutdown_connections 2023-03-03T18:29:04.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:29:04.568+0000 7f45662a9700 1 -- 172.21.15.7:0/452777880 wait complete. 2023-03-03T18:29:04.570 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:29:04.580 INFO:tasks.mds_thrash.fs.[a]:waiting for 115 secs before thrashing 2023-03-03T18:29:08.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:29:08.738 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:29:38.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:29:38.884 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:29:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T18:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T18:30:09.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:30:09.031 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:30:39.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:30:39.106 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:30:49.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:30:50.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.031+0000 7fdc27cd9700 1 Processor -- start 2023-03-03T18:30:50.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.031+0000 7fdc27cd9700 1 -- start start 2023-03-03T18:30:50.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc27cd9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdc200aadb0 0x7fdc20157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:50.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc27cd9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aa030 0x7fdc20158390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:50.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc27cd9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc201588d0 con 0x7fdc2014e410 2023-03-03T18:30:50.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc27cd9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2006fce0 con 0x7fdc200aadb0 2023-03-03T18:30:50.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc27cd9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2006f510 con 0x7fdc200aa030 2023-03-03T18:30:50.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc25a75700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdc2014e410 0x7fdc200717c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59652/0 (socket says 172.21.15.7:59652) 2023-03-03T18:30:50.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc25a75700 1 -- 172.21.15.7:0/1512073113 learned_addr learned my addr 172.21.15.7:0/1512073113 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:30:50.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc25274700 1 --2- 172.21.15.7:0/1512073113 >> v2:172.21.15.143:3300/0 conn(0x7fdc200aadb0 0x7fdc20157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:50.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc26276700 1 --2- 172.21.15.7:0/1512073113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aa030 0x7fdc20158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:50.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc24a73700 1 -- 172.21.15.7:0/1512073113 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886363161 0 0) 0x7fdc201588d0 con 0x7fdc2014e410 2023-03-03T18:30:50.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc24a73700 1 -- 172.21.15.7:0/1512073113 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc0c0035b0 con 0x7fdc2014e410 2023-03-03T18:30:50.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc25274700 1 -- 172.21.15.7:0/1512073113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aa030 msgr2=0x7fdc20158390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:30:50.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc25274700 1 --2- 172.21.15.7:0/1512073113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aa030 0x7fdc20158390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:50.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc25274700 1 -- 172.21.15.7:0/1512073113 >> v1:172.21.15.7:6789/0 conn(0x7fdc2014e410 legacy=0x7fdc200717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:50.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc24a73700 1 -- 172.21.15.7:0/1512073113 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2422578131 0 0) 0x7fdc0c0035b0 con 0x7fdc2014e410 2023-03-03T18:30:50.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.032+0000 7fdc25274700 1 -- 172.21.15.7:0/1512073113 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc20154f00 con 0x7fdc200aadb0 2023-03-03T18:30:50.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc26276700 1 --2- 172.21.15.7:0/1512073113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aa030 0x7fdc20158390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:30:50.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc25274700 1 --2- 172.21.15.7:0/1512073113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc200aadb0 0x7fdc20157e50 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009840 tx=0x7fdc1c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d47ee1565a08476 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:30:50.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc24a73700 1 -- 172.21.15.7:0/1512073113 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10001670 con 0x7fdc200aadb0 2023-03-03T18:30:50.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc24a73700 1 -- 172.21.15.7:0/1512073113 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1c004030 con 0x7fdc200aadb0 2023-03-03T18:30:50.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc24a73700 1 -- 172.21.15.7:0/1512073113 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c0043c0 con 0x7fdc200aadb0 2023-03-03T18:30:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc27cd9700 1 -- 172.21.15.7:0/1512073113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc200aadb0 msgr2=0x7fdc20157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc27cd9700 1 --2- 172.21.15.7:0/1512073113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc200aadb0 0x7fdc20157e50 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009840 tx=0x7fdc1c009b50 comp rx=0 tx=0).stop 2023-03-03T18:30:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc27cd9700 1 -- 172.21.15.7:0/1512073113 shutdown_connections 2023-03-03T18:30:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc27cd9700 1 --2- 172.21.15.7:0/1512073113 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aa030 0x7fdc20158390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc27cd9700 1 --2- 172.21.15.7:0/1512073113 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc200aadb0 0x7fdc20157e50 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.033+0000 7fdc27cd9700 1 -- 172.21.15.7:0/1512073113 >> 172.21.15.7:0/1512073113 conn(0x7fdc200c8900 msgr2=0x7fdc200c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:30:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.034+0000 7fdc27cd9700 1 -- 172.21.15.7:0/1512073113 shutdown_connections 2023-03-03T18:30:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.034+0000 7fdc27cd9700 1 -- 172.21.15.7:0/1512073113 wait complete. 2023-03-03T18:30:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.035+0000 7fdc27cd9700 1 Processor -- start 2023-03-03T18:30:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.035+0000 7fdc27cd9700 1 -- start start 2023-03-03T18:30:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.035+0000 7fdc27cd9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc200aa030 0x7fdc201534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.035+0000 7fdc27cd9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aadb0 0x7fdc201e6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.035+0000 7fdc27cd9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014e410 0x7fdc201e99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:50.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc27cd9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc201504c0 con 0x7fdc2014e410 2023-03-03T18:30:50.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc27cd9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2014f760 con 0x7fdc200aa030 2023-03-03T18:30:50.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc27cd9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc20151400 con 0x7fdc200aadb0 2023-03-03T18:30:50.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc26276700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014e410 0x7fdc201e99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:50.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc26276700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014e410 0x7fdc201e99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37450/0 (socket says 172.21.15.7:37450) 2023-03-03T18:30:50.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc26276700 1 -- 172.21.15.7:0/900124959 learned_addr learned my addr 172.21.15.7:0/900124959 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:30:50.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc25a75700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc200aa030 0x7fdc201534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:50.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc25274700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aadb0 0x7fdc201e6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc26276700 1 -- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aadb0 msgr2=0x7fdc201e6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc26276700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aadb0 0x7fdc201e6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc26276700 1 -- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc200aa030 msgr2=0x7fdc201534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc26276700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc200aa030 0x7fdc201534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc26276700 1 -- 172.21.15.7:0/900124959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc18009a70 con 0x7fdc2014e410 2023-03-03T18:30:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc26276700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014e410 0x7fdc201e99f0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800efc0 tx=0x7fdc1800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:30:50.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc167fc700 1 -- 172.21.15.7:0/900124959 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800ca90 con 0x7fdc2014e410 2023-03-03T18:30:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc167fc700 1 -- 172.21.15.7:0/900124959 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1800f040 con 0x7fdc2014e410 2023-03-03T18:30:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc27cd9700 1 -- 172.21.15.7:0/900124959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c0094a0 con 0x7fdc2014e410 2023-03-03T18:30:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc167fc700 1 -- 172.21.15.7:0/900124959 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800ce10 con 0x7fdc2014e410 2023-03-03T18:30:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.036+0000 7fdc27cd9700 1 -- 172.21.15.7:0/900124959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2006f720 con 0x7fdc2014e410 2023-03-03T18:30:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.037+0000 7fdc27cd9700 1 -- 172.21.15.7:0/900124959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc201551e0 con 0x7fdc2014e410 2023-03-03T18:30:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.038+0000 7fdc167fc700 1 -- 172.21.15.7:0/900124959 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdc18014480 con 0x7fdc2014e410 2023-03-03T18:30:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.039+0000 7fdc167fc700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 0x7fdc0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:50.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.039+0000 7fdc167fc700 1 -- 172.21.15.7:0/900124959 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdc180868c0 con 0x7fdc2014e410 2023-03-03T18:30:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.039+0000 7fdc25a75700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 0x7fdc0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.039+0000 7fdc25a75700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 0x7fdc0406a7a0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fdc10000f60 tx=0x7fdc10001e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:30:50.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.043+0000 7fdc167fc700 1 -- 172.21.15.7:0/900124959 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdc18052c00 con 0x7fdc2014e410 2023-03-03T18:30:50.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.280+0000 7fdc27cd9700 1 -- 172.21.15.7:0/900124959 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc2006fa00 con 0x7fdc2014e410 2023-03-03T18:30:50.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.281+0000 7fdc167fc700 1 -- 172.21.15.7:0/900124959 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdc18009d70 con 0x7fdc2014e410 2023-03-03T18:30:50.281 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:30:50.281 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:30:50.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 -- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 msgr2=0x7fdc0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:50.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 0x7fdc0406a7a0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7fdc10000f60 tx=0x7fdc10001e00 comp rx=0 tx=0).stop 2023-03-03T18:30:50.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 -- 172.21.15.7:0/900124959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014e410 msgr2=0x7fdc201e99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:50.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014e410 0x7fdc201e99f0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800efc0 tx=0x7fdc1800c3a0 comp rx=0 tx=0).stop 2023-03-03T18:30:50.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 -- 172.21.15.7:0/900124959 shutdown_connections 2023-03-03T18:30:50.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdc040682f0 0x7fdc0406a7a0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:50.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdc2014e410 0x7fdc201e99f0 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:50.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc200aa030 0x7fdc201534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:50.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 --2- 172.21.15.7:0/900124959 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdc200aadb0 0x7fdc201e6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:50.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 -- 172.21.15.7:0/900124959 >> 172.21.15.7:0/900124959 conn(0x7fdc200c8900 msgr2=0x7fdc200c8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:30:50.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 -- 172.21.15.7:0/900124959 shutdown_connections 2023-03-03T18:30:50.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:50.283+0000 7fdc27cd9700 1 -- 172.21.15.7:0/900124959 wait complete. 2023-03-03T18:30:50.285 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:30:50.294 INFO:tasks.mds_thrash.fs.[b]:waiting for 40 secs before thrashing 2023-03-03T18:30:59.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:30:59.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.719+0000 7fc409f42700 1 Processor -- start 2023-03-03T18:30:59.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.719+0000 7fc409f42700 1 -- start start 2023-03-03T18:30:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.720+0000 7fc409f42700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 0x7fc404156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.720+0000 7fc409f42700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc4040abe20 0x7fc4041529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:59.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.720+0000 7fc409f42700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4040c2f30 con 0x7fc40414fda0 2023-03-03T18:30:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.720+0000 7fc409f42700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4040c31d0 con 0x7fc4040abe20 2023-03-03T18:30:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.720+0000 7fc409f42700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4041524a0 con 0x7fc40414f9d0 2023-03-03T18:30:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.720+0000 7fc403fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc4040abe20 0x7fc4041529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:59.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.720+0000 7fc4037fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 0x7fc404156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.720+0000 7fc403fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc4040abe20 0x7fc4041529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:33242/0 (socket says 172.21.15.7:33242) 2023-03-03T18:30:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.720+0000 7fc4037fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 0x7fc404156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:45536/0 (socket says 172.21.15.7:45536) 2023-03-03T18:30:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.720+0000 7fc403fff700 1 -- 172.21.15.7:0/1715144309 learned_addr learned my addr 172.21.15.7:0/1715144309 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:30:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.721+0000 7fc403fff700 1 -- 172.21.15.7:0/1715144309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 msgr2=0x7fc404156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:30:59.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.721+0000 7fc403fff700 1 --2- 172.21.15.7:0/1715144309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 0x7fc404156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.721+0000 7fc403fff700 1 -- 172.21.15.7:0/1715144309 >> v1:172.21.15.143:6789/0 conn(0x7fc40414fda0 legacy=0x7fc404071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.721+0000 7fc403fff700 1 -- 172.21.15.7:0/1715144309 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4040c34b0 con 0x7fc4040abe20 2023-03-03T18:30:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.721+0000 7fc4027fc700 1 -- 172.21.15.7:0/1715144309 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984828660 0 0) 0x7fc4040c2f30 con 0x7fc40414fda0 2023-03-03T18:30:59.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.721+0000 7fc4037fe700 1 --2- 172.21.15.7:0/1715144309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 0x7fc404156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:30:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.721+0000 7fc403fff700 1 --2- 172.21.15.7:0/1715144309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4040abe20 0x7fc4041529c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009e50 tx=0x7fc3fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=199817f73a3aeff3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:30:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.721+0000 7fc4027fc700 1 -- 172.21.15.7:0/1715144309 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc00e070 con 0x7fc4040abe20 2023-03-03T18:30:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.721+0000 7fc4027fc700 1 -- 172.21.15.7:0/1715144309 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc00b910 con 0x7fc4040abe20 2023-03-03T18:30:59.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.721+0000 7fc4027fc700 1 -- 172.21.15.7:0/1715144309 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc00bcc0 con 0x7fc4040abe20 2023-03-03T18:30:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.722+0000 7fc409f42700 1 -- 172.21.15.7:0/1715144309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4040abe20 msgr2=0x7fc4041529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.722+0000 7fc409f42700 1 --2- 172.21.15.7:0/1715144309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4040abe20 0x7fc4041529c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009e50 tx=0x7fc3fc00b330 comp rx=0 tx=0).stop 2023-03-03T18:30:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.722+0000 7fc409f42700 1 -- 172.21.15.7:0/1715144309 shutdown_connections 2023-03-03T18:30:59.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.722+0000 7fc409f42700 1 --2- 172.21.15.7:0/1715144309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4040abe20 0x7fc4041529c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:59.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.722+0000 7fc409f42700 1 --2- 172.21.15.7:0/1715144309 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 0x7fc404156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:59.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.722+0000 7fc409f42700 1 -- 172.21.15.7:0/1715144309 >> 172.21.15.7:0/1715144309 conn(0x7fc4040c8900 msgr2=0x7fc4040c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:30:59.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.722+0000 7fc409f42700 1 -- 172.21.15.7:0/1715144309 shutdown_connections 2023-03-03T18:30:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.722+0000 7fc409f42700 1 -- 172.21.15.7:0/1715144309 wait complete. 2023-03-03T18:30:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.723+0000 7fc409f42700 1 Processor -- start 2023-03-03T18:30:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.723+0000 7fc409f42700 1 -- start start 2023-03-03T18:30:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc409f42700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4040abe20 0x7fc40414e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:59.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc409f42700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 0x7fc404148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc409f42700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc40414fda0 0x7fc404148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc409f42700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc404153ca0 con 0x7fc40414fda0 2023-03-03T18:30:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc409f42700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4041493e0 con 0x7fc4040abe20 2023-03-03T18:30:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc409f42700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc404153700 con 0x7fc40414f9d0 2023-03-03T18:30:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc403fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc40414fda0 0x7fc404148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc403fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc40414fda0 0x7fc404148ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37474/0 (socket says 172.21.15.7:37474) 2023-03-03T18:30:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc403fff700 1 -- 172.21.15.7:0/1887300544 learned_addr learned my addr 172.21.15.7:0/1887300544 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:30:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc4037fe700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4040abe20 0x7fc40414e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc402ffd700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 0x7fc404148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc403fff700 1 -- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 msgr2=0x7fc404148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc403fff700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 0x7fc404148960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc403fff700 1 -- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4040abe20 msgr2=0x7fc40414e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc403fff700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4040abe20 0x7fc40414e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc403fff700 1 -- 172.21.15.7:0/1887300544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40094a0 con 0x7fc40414fda0 2023-03-03T18:30:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.724+0000 7fc4037fe700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4040abe20 0x7fc40414e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:30:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.725+0000 7fc403fff700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc40414fda0 0x7fc404148ea0 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7fc404072520 tx=0x7fc3fc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:30:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.725+0000 7fc3fbfff700 1 -- 172.21.15.7:0/1887300544 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc003ea0 con 0x7fc40414fda0 2023-03-03T18:30:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.725+0000 7fc3fbfff700 1 -- 172.21.15.7:0/1887300544 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc0099b0 con 0x7fc40414fda0 2023-03-03T18:30:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.725+0000 7fc409f42700 1 -- 172.21.15.7:0/1887300544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3fc009b30 con 0x7fc40414fda0 2023-03-03T18:30:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.725+0000 7fc3fbfff700 1 -- 172.21.15.7:0/1887300544 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc01b670 con 0x7fc40414fda0 2023-03-03T18:30:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.725+0000 7fc409f42700 1 -- 172.21.15.7:0/1887300544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4040ac350 con 0x7fc40414fda0 2023-03-03T18:30:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.726+0000 7fc409f42700 1 -- 172.21.15.7:0/1887300544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4040ab630 con 0x7fc40414fda0 2023-03-03T18:30:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.726+0000 7fc3fbfff700 1 -- 172.21.15.7:0/1887300544 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc3fc02b070 con 0x7fc40414fda0 2023-03-03T18:30:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.727+0000 7fc3fbfff700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3e4068320 0x7fc3e406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:30:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.727+0000 7fc3fbfff700 1 -- 172.21.15.7:0/1887300544 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc3fc087930 con 0x7fc40414fda0 2023-03-03T18:30:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.727+0000 7fc4037fe700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3e4068320 0x7fc3e406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:30:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.727+0000 7fc4037fe700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3e4068320 0x7fc3e406a7d0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009cf0 tx=0x7fc3f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:30:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.731+0000 7fc3fbfff700 1 -- 172.21.15.7:0/1887300544 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc3fc053cd0 con 0x7fc40414fda0 2023-03-03T18:30:59.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.969+0000 7fc409f42700 1 -- 172.21.15.7:0/1887300544 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc404071fa0 con 0x7fc40414fda0 2023-03-03T18:30:59.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.970+0000 7fc3fbfff700 1 -- 172.21.15.7:0/1887300544 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc3fc0572f0 con 0x7fc40414fda0 2023-03-03T18:30:59.971 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:30:59.971 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:30:59.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 -- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3e4068320 msgr2=0x7fc3e406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3e4068320 0x7fc3e406a7d0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009cf0 tx=0x7fc3f4008040 comp rx=0 tx=0).stop 2023-03-03T18:30:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 -- 172.21.15.7:0/1887300544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc40414fda0 msgr2=0x7fc404148ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:30:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc40414fda0 0x7fc404148ea0 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7fc404072520 tx=0x7fc3fc004ad0 comp rx=0 tx=0).stop 2023-03-03T18:30:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 -- 172.21.15.7:0/1887300544 shutdown_connections 2023-03-03T18:30:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3e4068320 0x7fc3e406a7d0 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:59.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc40414fda0 0x7fc404148ea0 unknown :-1 s=CLOSED pgs=3214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4040abe20 0x7fc40414e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 --2- 172.21.15.7:0/1887300544 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc40414f9d0 0x7fc404148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:30:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 -- 172.21.15.7:0/1887300544 >> 172.21.15.7:0/1887300544 conn(0x7fc4040c8900 msgr2=0x7fc404150e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:30:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 -- 172.21.15.7:0/1887300544 shutdown_connections 2023-03-03T18:30:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:30:59.972+0000 7fc409f42700 1 -- 172.21.15.7:0/1887300544 wait complete. 2023-03-03T18:30:59.974 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:30:59.985 INFO:tasks.mds_thrash.fs.[a]:waiting for 62 secs before thrashing 2023-03-03T18:31:09.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:31:09.240 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:31:30.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:31:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.434+0000 7f1fc6cab700 1 Processor -- start 2023-03-03T18:31:30.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.434+0000 7f1fc6cab700 1 -- start start 2023-03-03T18:31:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc6cab700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1fc0151d40 0x7f1fc00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:31:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc6cab700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1fc0073c30 0x7f1fc01581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:31:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc6cab700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc006d430 con 0x7f1fc00c2930 2023-03-03T18:31:30.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc6cab700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc006c2a0 con 0x7f1fc0151d40 2023-03-03T18:31:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc6cab700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc01586e0 con 0x7f1fc0073c30 2023-03-03T18:31:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc4a47700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1fc0151d40 0x7f1fc00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:31:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc4a47700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1fc0151d40 0x7f1fc00736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37498/0 (socket says 172.21.15.7:37498) 2023-03-03T18:31:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc4a47700 1 -- 172.21.15.7:0/2820487609 learned_addr learned my addr 172.21.15.7:0/2820487609 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:31:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc5248700 1 --2- 172.21.15.7:0/2820487609 >> v2:172.21.15.143:3300/0 conn(0x7f1fc0073c30 0x7f1fc01581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:31:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc4a47700 1 -- 172.21.15.7:0/2820487609 >> v2:172.21.15.143:3300/0 conn(0x7f1fc0073c30 msgr2=0x7f1fc01581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:31:30.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc4a47700 1 --2- 172.21.15.7:0/2820487609 >> v2:172.21.15.143:3300/0 conn(0x7f1fc0073c30 0x7f1fc01581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:31:30.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc4a47700 1 -- 172.21.15.7:0/2820487609 >> v1:172.21.15.143:6789/0 conn(0x7f1fc00c2930 legacy=0x7f1fc0157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:31:30.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.435+0000 7f1fc4a47700 1 -- 172.21.15.7:0/2820487609 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc006cb00 con 0x7f1fc0151d40 2023-03-03T18:31:30.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fc4a47700 1 --2- 172.21.15.7:0/2820487609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 0x7f1fc00736f0 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7f1fb40098f0 tx=0x7f1fb4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444fc1708f0e9ae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:31:30.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fbf7fe700 1 -- 172.21.15.7:0/2820487609 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4004230 con 0x7f1fc0151d40 2023-03-03T18:31:30.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fbf7fe700 1 -- 172.21.15.7:0/2820487609 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb4004390 con 0x7f1fc0151d40 2023-03-03T18:31:30.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fbf7fe700 1 -- 172.21.15.7:0/2820487609 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb40036a0 con 0x7f1fc0151d40 2023-03-03T18:31:30.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2820487609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 msgr2=0x7f1fc00736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:31:30.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fc6cab700 1 --2- 172.21.15.7:0/2820487609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 0x7f1fc00736f0 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7f1fb40098f0 tx=0x7f1fb4009c00 comp rx=0 tx=0).stop 2023-03-03T18:31:30.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2820487609 shutdown_connections 2023-03-03T18:31:30.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fc6cab700 1 --2- 172.21.15.7:0/2820487609 >> v2:172.21.15.143:3300/0 conn(0x7f1fc0073c30 0x7f1fc01581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:31:30.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fc6cab700 1 --2- 172.21.15.7:0/2820487609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 0x7f1fc00736f0 unknown :-1 s=CLOSED pgs=3215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:31:30.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2820487609 >> 172.21.15.7:0/2820487609 conn(0x7f1fc00a94d0 msgr2=0x7f1fc00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:31:30.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2820487609 shutdown_connections 2023-03-03T18:31:30.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.436+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2820487609 wait complete. 2023-03-03T18:31:30.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.438+0000 7f1fc6cab700 1 Processor -- start 2023-03-03T18:31:30.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.438+0000 7f1fc6cab700 1 -- start start 2023-03-03T18:31:30.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.438+0000 7f1fc6cab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fc0073c30 0x7f1fc00bde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:31:30.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.438+0000 7f1fc6cab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fc00c2930 0x7f1fc01485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:31:30.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.438+0000 7f1fc6cab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 0x7f1fc01df680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:31:30.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.438+0000 7f1fc6cab700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc01ddaf0 con 0x7f1fc0151d40 2023-03-03T18:31:30.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.438+0000 7f1fc6cab700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc01ead20 con 0x7f1fc0073c30 2023-03-03T18:31:30.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.438+0000 7f1fc6cab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc01eb000 con 0x7f1fc00c2930 2023-03-03T18:31:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc5248700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 0x7f1fc01df680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:31:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc5248700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 0x7f1fc01df680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37506/0 (socket says 172.21.15.7:37506) 2023-03-03T18:31:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc5248700 1 -- 172.21.15.7:0/2124915054 learned_addr learned my addr 172.21.15.7:0/2124915054 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:31:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fbffff700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fc00c2930 0x7f1fc01485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:31:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc4a47700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fc0073c30 0x7f1fc00bde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:31:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc5248700 1 -- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fc00c2930 msgr2=0x7f1fc01485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:31:30.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc5248700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fc00c2930 0x7f1fc01485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:31:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc5248700 1 -- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fc0073c30 msgr2=0x7f1fc00bde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:31:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc5248700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fc0073c30 0x7f1fc00bde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:31:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc5248700 1 -- 172.21.15.7:0/2124915054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4009550 con 0x7f1fc0151d40 2023-03-03T18:31:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc4a47700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fc0073c30 0x7f1fc00bde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:31:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc5248700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 0x7f1fc01df680 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800cc80 tx=0x7f1fb800a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:31:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fbd7fa700 1 -- 172.21.15.7:0/2124915054 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb800ce80 con 0x7f1fc0151d40 2023-03-03T18:31:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fbd7fa700 1 -- 172.21.15.7:0/2124915054 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb800acd0 con 0x7f1fc0151d40 2023-03-03T18:31:30.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2124915054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc006d0c0 con 0x7f1fc0151d40 2023-03-03T18:31:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fbd7fa700 1 -- 172.21.15.7:0/2124915054 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8003c60 con 0x7f1fc0151d40 2023-03-03T18:31:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.439+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2124915054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc006cde0 con 0x7f1fc0151d40 2023-03-03T18:31:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.440+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2124915054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc006bea0 con 0x7f1fc0151d40 2023-03-03T18:31:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.441+0000 7f1fbd7fa700 1 -- 172.21.15.7:0/2124915054 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1fb801f070 con 0x7f1fc0151d40 2023-03-03T18:31:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.441+0000 7f1fbd7fa700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fa40682f0 0x7f1fa406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:31:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.441+0000 7f1fbd7fa700 1 -- 172.21.15.7:0/2124915054 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1fb8086180 con 0x7f1fc0151d40 2023-03-03T18:31:30.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.441+0000 7f1fc4a47700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fa40682f0 0x7f1fa406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:31:30.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.442+0000 7f1fc4a47700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fa40682f0 0x7f1fa406a7a0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009c40 tx=0x7f1fb4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:31:30.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.446+0000 7f1fbd7fa700 1 -- 172.21.15.7:0/2124915054 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1fb8052520 con 0x7f1fc0151d40 2023-03-03T18:31:30.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.681+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2124915054 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fc0074bb0 con 0x7f1fc0151d40 2023-03-03T18:31:30.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.682+0000 7f1fbd7fa700 1 -- 172.21.15.7:0/2124915054 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1fb801d020 con 0x7f1fc0151d40 2023-03-03T18:31:30.683 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:31:30.683 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:31:30.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fa40682f0 msgr2=0x7f1fa406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:31:30.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fa40682f0 0x7f1fa406a7a0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009c40 tx=0x7f1fb4008030 comp rx=0 tx=0).stop 2023-03-03T18:31:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2124915054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 msgr2=0x7f1fc01df680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:31:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 0x7f1fc01df680 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800cc80 tx=0x7f1fb800a4d0 comp rx=0 tx=0).stop 2023-03-03T18:31:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2124915054 shutdown_connections 2023-03-03T18:31:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1fa40682f0 0x7f1fa406a7a0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:31:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1fc0151d40 0x7f1fc01df680 unknown :-1 s=CLOSED pgs=3216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:31:30.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1fc0073c30 0x7f1fc00bde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:31:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 --2- 172.21.15.7:0/2124915054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1fc00c2930 0x7f1fc01485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:31:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2124915054 >> 172.21.15.7:0/2124915054 conn(0x7f1fc00a94d0 msgr2=0x7f1fc01558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:31:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2124915054 shutdown_connections 2023-03-03T18:31:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:31:30.684+0000 7f1fc6cab700 1 -- 172.21.15.7:0/2124915054 wait complete. 2023-03-03T18:31:30.686 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:31:30.695 INFO:tasks.mds_thrash.fs.[b]:waiting for 85 secs before thrashing 2023-03-03T18:31:39.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:31:39.303 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:32:01.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:32:02.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.125+0000 7f5a381e5700 1 Processor -- start 2023-03-03T18:32:02.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.125+0000 7f5a381e5700 1 -- start start 2023-03-03T18:32:02.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a381e5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5a3014f9d0 0x7f5a30156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:02.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a381e5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300c2f30 con 0x7f5a300abe20 2023-03-03T18:32:02.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a381e5700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300c31d0 con 0x7f5a3014fda0 2023-03-03T18:32:02.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a381e5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a301524a0 con 0x7f5a3014f9d0 2023-03-03T18:32:02.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a36782700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5a300abe20 0x7f5a301529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59746/0 (socket says 172.21.15.7:59746) 2023-03-03T18:32:02.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a36782700 1 -- 172.21.15.7:0/1580792318 learned_addr learned my addr 172.21.15.7:0/1580792318 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:32:02.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a35f81700 1 --2- 172.21.15.7:0/1580792318 >> v2:172.21.15.143:3300/0 conn(0x7f5a3014f9d0 0x7f5a30156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:02.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512999749 0 0) 0x7f5a300c2f30 con 0x7f5a300abe20 2023-03-03T18:32:02.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a1c0035b0 con 0x7f5a300abe20 2023-03-03T18:32:02.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1362504626 0 0) 0x7f5a1c0035b0 con 0x7f5a300abe20 2023-03-03T18:32:02.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 >> v2:172.21.15.143:3300/0 conn(0x7f5a3014f9d0 msgr2=0x7f5a30156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:32:02.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a34f7f700 1 --2- 172.21.15.7:0/1580792318 >> v2:172.21.15.143:3300/0 conn(0x7f5a3014f9d0 0x7f5a30156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 >> v1:172.21.15.143:6789/0 conn(0x7f5a3014fda0 legacy=0x7f5a30071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:02.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300c34b0 con 0x7f5a300abe20 2023-03-03T18:32:02.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.126+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f5a280019f0 con 0x7f5a300abe20 2023-03-03T18:32:02.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 >> v1:172.21.15.7:6789/0 conn(0x7f5a300abe20 legacy=0x7f5a301529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:02.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a34f7f700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1c003a70 0x7f5a1c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:02.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a34f7f700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a1c004850 0x7f5a1c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:02.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a34f7f700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a1c005320 0x7f5a1c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:02.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a301524a0 con 0x7f5a1c003a70 2023-03-03T18:32:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c00c230 con 0x7f5a1c005320 2023-03-03T18:32:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c00c3f0 con 0x7f5a1c004850 2023-03-03T18:32:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a35f81700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1c003a70 0x7f5a1c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a35780700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a1c004850 0x7f5a1c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a36782700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a1c005320 0x7f5a1c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a35f81700 1 -- 172.21.15.7:0/1580792318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a1c004850 msgr2=0x7f5a1c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:32:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a35f81700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a1c004850 0x7f5a1c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a35f81700 1 -- 172.21.15.7:0/1580792318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a1c005320 msgr2=0x7f5a1c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:32:02.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a35f81700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a1c005320 0x7f5a1c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a35f81700 1 -- 172.21.15.7:0/1580792318 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c0035b0 con 0x7f5a1c003a70 2023-03-03T18:32:02.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a36782700 1 -- 172.21.15.7:0/1580792318 reap_dead start 2023-03-03T18:32:02.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a35780700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a1c004850 0x7f5a1c004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:32:02.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a35f81700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1c003a70 0x7f5a1c003e80 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009b60 tx=0x7f5a1c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7eba2981e53815f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:32:02.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00e6c0 con 0x7f5a1c003a70 2023-03-03T18:32:02.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c00e820 con 0x7f5a1c003a70 2023-03-03T18:32:02.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.127+0000 7f5a34f7f700 1 -- 172.21.15.7:0/1580792318 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00eb90 con 0x7f5a1c003a70 2023-03-03T18:32:02.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.128+0000 7f5a381e5700 1 -- 172.21.15.7:0/1580792318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1c003a70 msgr2=0x7f5a1c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:02.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.128+0000 7f5a381e5700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1c003a70 0x7f5a1c003e80 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009b60 tx=0x7f5a1c005880 comp rx=0 tx=0).stop 2023-03-03T18:32:02.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.128+0000 7f5a381e5700 1 -- 172.21.15.7:0/1580792318 shutdown_connections 2023-03-03T18:32:02.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.128+0000 7f5a381e5700 1 --2- 172.21.15.7:0/1580792318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a1c003a70 0x7f5a1c003e80 unknown :-1 s=CLOSED pgs=3218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.128+0000 7f5a381e5700 1 -- 172.21.15.7:0/1580792318 >> 172.21.15.7:0/1580792318 conn(0x7f5a300c8900 msgr2=0x7f5a300c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:32:02.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.128+0000 7f5a381e5700 1 -- 172.21.15.7:0/1580792318 shutdown_connections 2023-03-03T18:32:02.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.128+0000 7f5a381e5700 1 -- 172.21.15.7:0/1580792318 wait complete. 2023-03-03T18:32:02.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.129+0000 7f5a381e5700 1 Processor -- start 2023-03-03T18:32:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.129+0000 7f5a381e5700 1 -- start start 2023-03-03T18:32:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a381e5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a1c003a70 0x7f5a301dee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a381e5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a30154b90 0x7f5a300bd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a381e5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3014eea0 0x7f5a3014f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a381e5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a301eb650 con 0x7f5a3014eea0 2023-03-03T18:32:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a381e5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3014f880 con 0x7f5a30154b90 2023-03-03T18:32:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a381e5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a301ea9d0 con 0x7f5a1c003a70 2023-03-03T18:32:02.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a36782700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3014eea0 0x7f5a3014f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:02.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a36782700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3014eea0 0x7f5a3014f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37546/0 (socket says 172.21.15.7:37546) 2023-03-03T18:32:02.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a36782700 1 -- 172.21.15.7:0/4266268067 learned_addr learned my addr 172.21.15.7:0/4266268067 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:32:02.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a35780700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a30154b90 0x7f5a300bd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:02.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a35f81700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a1c003a70 0x7f5a301dee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:02.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a36782700 1 -- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a1c003a70 msgr2=0x7f5a301dee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:02.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a36782700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a1c003a70 0x7f5a301dee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a36782700 1 -- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a30154b90 msgr2=0x7f5a300bd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:02.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a36782700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a30154b90 0x7f5a300bd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.130+0000 7f5a36782700 1 -- 172.21.15.7:0/4266268067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300ac350 con 0x7f5a3014eea0 2023-03-03T18:32:02.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.131+0000 7f5a36782700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3014eea0 0x7f5a3014f340 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 crypto rx=0x7f5a28000f80 tx=0x7f5a2800a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:32:02.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.131+0000 7f5a267fc700 1 -- 172.21.15.7:0/4266268067 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2800ac80 con 0x7f5a3014eea0 2023-03-03T18:32:02.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.131+0000 7f5a267fc700 1 -- 172.21.15.7:0/4266268067 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2800ae00 con 0x7f5a3014eea0 2023-03-03T18:32:02.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.131+0000 7f5a381e5700 1 -- 172.21.15.7:0/4266268067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a300ab630 con 0x7f5a3014eea0 2023-03-03T18:32:02.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.131+0000 7f5a267fc700 1 -- 172.21.15.7:0/4266268067 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28013630 con 0x7f5a3014eea0 2023-03-03T18:32:02.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.131+0000 7f5a381e5700 1 -- 172.21.15.7:0/4266268067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a300c3e10 con 0x7f5a3014eea0 2023-03-03T18:32:02.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.132+0000 7f5a381e5700 1 -- 172.21.15.7:0/4266268067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a30074bb0 con 0x7f5a3014eea0 2023-03-03T18:32:02.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.133+0000 7f5a267fc700 1 -- 172.21.15.7:0/4266268067 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5a28023070 con 0x7f5a3014eea0 2023-03-03T18:32:02.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.133+0000 7f5a267fc700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a140682c0 0x7f5a1406a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:02.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.133+0000 7f5a267fc700 1 -- 172.21.15.7:0/4266268067 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5a28087030 con 0x7f5a3014eea0 2023-03-03T18:32:02.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.133+0000 7f5a35f81700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a140682c0 0x7f5a1406a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:02.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.134+0000 7f5a35f81700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a140682c0 0x7f5a1406a770 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0090b0 tx=0x7f5a2c01e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:32:02.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.137+0000 7f5a267fc700 1 -- 172.21.15.7:0/4266268067 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5a28053470 con 0x7f5a3014eea0 2023-03-03T18:32:02.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.376+0000 7f5a381e5700 1 -- 172.21.15.7:0/4266268067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a301eb790 con 0x7f5a3014eea0 2023-03-03T18:32:02.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.377+0000 7f5a267fc700 1 -- 172.21.15.7:0/4266268067 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5a301eb790 con 0x7f5a3014eea0 2023-03-03T18:32:02.377 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:32:02.378 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:32:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 -- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a140682c0 msgr2=0x7f5a1406a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a140682c0 0x7f5a1406a770 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0090b0 tx=0x7f5a2c01e040 comp rx=0 tx=0).stop 2023-03-03T18:32:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 -- 172.21.15.7:0/4266268067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3014eea0 msgr2=0x7f5a3014f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:02.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3014eea0 0x7f5a3014f340 secure :-1 s=READY pgs=3219 cs=0 l=1 rev1=1 crypto rx=0x7f5a28000f80 tx=0x7f5a2800a5f0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 -- 172.21.15.7:0/4266268067 shutdown_connections 2023-03-03T18:32:02.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5a140682c0 0x7f5a1406a770 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a3014eea0 0x7f5a3014f340 unknown :-1 s=CLOSED pgs=3219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5a1c003a70 0x7f5a301dee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 --2- 172.21.15.7:0/4266268067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a30154b90 0x7f5a300bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:02.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 -- 172.21.15.7:0/4266268067 >> 172.21.15.7:0/4266268067 conn(0x7f5a300c8900 msgr2=0x7f5a300c7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:32:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 -- 172.21.15.7:0/4266268067 shutdown_connections 2023-03-03T18:32:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:02.379+0000 7f5a381e5700 1 -- 172.21.15.7:0/4266268067 wait complete. 2023-03-03T18:32:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:32:02.391 INFO:tasks.mds_thrash.fs.[a]:waiting for 80 secs before thrashing 2023-03-03T18:32:09.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:32:09.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:32:39.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:32:39.443 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:32:55.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:32:55.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.836+0000 7f8e445e3700 1 Processor -- start 2023-03-03T18:32:55.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.836+0000 7f8e445e3700 1 -- start start 2023-03-03T18:32:55.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.837+0000 7f8e445e3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e3c072250 0x7f8e3c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:55.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.837+0000 7f8e445e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 0x7f8e3c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:55.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.837+0000 7f8e445e3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3c1510d0 con 0x7f8e3c071e80 2023-03-03T18:32:55.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.837+0000 7f8e445e3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e3c0c3740 con 0x7f8e3c072250 2023-03-03T18:32:55.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.837+0000 7f8e445e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c070670 con 0x7f8e3c0c1ce0 2023-03-03T18:32:55.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.837+0000 7f8e41b7e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e3c072250 0x7f8e3c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.837+0000 7f8e41b7e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e3c072250 0x7f8e3c14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37598/0 (socket says 172.21.15.7:37598) 2023-03-03T18:32:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.837+0000 7f8e41b7e700 1 -- 172.21.15.7:0/1839798079 learned_addr learned my addr 172.21.15.7:0/1839798079 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:32:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.838+0000 7f8e42b80700 1 --2- 172.21.15.7:0/1839798079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 0x7f8e3c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.838+0000 7f8e41b7e700 1 -- 172.21.15.7:0/1839798079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 msgr2=0x7f8e3c14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:32:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.838+0000 7f8e41b7e700 1 --2- 172.21.15.7:0/1839798079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 0x7f8e3c14e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.838+0000 7f8e41b7e700 1 -- 172.21.15.7:0/1839798079 >> v1:172.21.15.143:6789/0 conn(0x7f8e3c071e80 legacy=0x7f8e3c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:55.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.838+0000 7f8e41b7e700 1 -- 172.21.15.7:0/1839798079 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c0c39e0 con 0x7f8e3c072250 2023-03-03T18:32:55.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.838+0000 7f8e41b7e700 1 --2- 172.21.15.7:0/1839798079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 0x7f8e3c14e1f0 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 crypto rx=0x7f8e380098f0 tx=0x7f8e38009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c0620111c486e1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:32:55.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.838+0000 7f8e4137d700 1 -- 172.21.15.7:0/1839798079 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3800b8a0 con 0x7f8e3c072250 2023-03-03T18:32:55.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.838+0000 7f8e4137d700 1 -- 172.21.15.7:0/1839798079 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3800ba00 con 0x7f8e3c072250 2023-03-03T18:32:55.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.838+0000 7f8e4137d700 1 -- 172.21.15.7:0/1839798079 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3800bd90 con 0x7f8e3c072250 2023-03-03T18:32:55.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.839+0000 7f8e445e3700 1 -- 172.21.15.7:0/1839798079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 msgr2=0x7f8e3c14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:55.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.839+0000 7f8e445e3700 1 --2- 172.21.15.7:0/1839798079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 0x7f8e3c14e1f0 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 crypto rx=0x7f8e380098f0 tx=0x7f8e38009c00 comp rx=0 tx=0).stop 2023-03-03T18:32:55.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.839+0000 7f8e445e3700 1 -- 172.21.15.7:0/1839798079 shutdown_connections 2023-03-03T18:32:55.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.839+0000 7f8e445e3700 1 --2- 172.21.15.7:0/1839798079 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 0x7f8e3c14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:55.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.839+0000 7f8e445e3700 1 --2- 172.21.15.7:0/1839798079 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 0x7f8e3c14e1f0 unknown :-1 s=CLOSED pgs=3220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:55.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.839+0000 7f8e445e3700 1 -- 172.21.15.7:0/1839798079 >> 172.21.15.7:0/1839798079 conn(0x7f8e3c0a94d0 msgr2=0x7f8e3c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:32:55.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.839+0000 7f8e445e3700 1 -- 172.21.15.7:0/1839798079 shutdown_connections 2023-03-03T18:32:55.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.839+0000 7f8e445e3700 1 -- 172.21.15.7:0/1839798079 wait complete. 2023-03-03T18:32:55.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.840+0000 7f8e445e3700 1 Processor -- start 2023-03-03T18:32:55.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.840+0000 7f8e445e3700 1 -- start start 2023-03-03T18:32:55.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e445e3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3c071e80 0x7f8e3c1edda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:55.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e445e3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 0x7f8e3c146d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:55.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e445e3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 0x7f8e3c145120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:55.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e445e3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c0c6070 con 0x7f8e3c072250 2023-03-03T18:32:55.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e445e3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c145e50 con 0x7f8e3c071e80 2023-03-03T18:32:55.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e445e3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3c146130 con 0x7f8e3c0c1ce0 2023-03-03T18:32:55.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e41b7e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 0x7f8e3c146d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:55.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e41b7e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 0x7f8e3c146d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37604/0 (socket says 172.21.15.7:37604) 2023-03-03T18:32:55.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e41b7e700 1 -- 172.21.15.7:0/4273024808 learned_addr learned my addr 172.21.15.7:0/4273024808 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:32:55.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e4237f700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3c071e80 0x7f8e3c1edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:55.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e42b80700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 0x7f8e3c145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:55.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e41b7e700 1 -- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 msgr2=0x7f8e3c145120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:55.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e41b7e700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 0x7f8e3c145120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:55.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e41b7e700 1 -- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3c071e80 msgr2=0x7f8e3c1edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:55.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e41b7e700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3c071e80 0x7f8e3c1edda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:55.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e41b7e700 1 -- 172.21.15.7:0/4273024808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e38009550 con 0x7f8e3c072250 2023-03-03T18:32:55.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e42b80700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 0x7f8e3c145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:32:55.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e4237f700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3c071e80 0x7f8e3c1edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:32:55.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e41b7e700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 0x7f8e3c146d50 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f8e38000c00 tx=0x7f8e38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:32:55.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.841+0000 7f8e32ffd700 1 -- 172.21.15.7:0/4273024808 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38004480 con 0x7f8e3c072250 2023-03-03T18:32:55.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.842+0000 7f8e32ffd700 1 -- 172.21.15.7:0/4273024808 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3801f070 con 0x7f8e3c072250 2023-03-03T18:32:55.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.842+0000 7f8e445e3700 1 -- 172.21.15.7:0/4273024808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e3c151f30 con 0x7f8e3c072250 2023-03-03T18:32:55.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.842+0000 7f8e32ffd700 1 -- 172.21.15.7:0/4273024808 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e380038c0 con 0x7f8e3c072250 2023-03-03T18:32:55.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.842+0000 7f8e445e3700 1 -- 172.21.15.7:0/4273024808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e3c151300 con 0x7f8e3c072250 2023-03-03T18:32:55.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.843+0000 7f8e445e3700 1 -- 172.21.15.7:0/4273024808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e3c06c9a0 con 0x7f8e3c072250 2023-03-03T18:32:55.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.843+0000 7f8e32ffd700 1 -- 172.21.15.7:0/4273024808 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e3802a020 con 0x7f8e3c072250 2023-03-03T18:32:55.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.844+0000 7f8e32ffd700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e20068320 0x7f8e2006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:32:55.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.844+0000 7f8e32ffd700 1 -- 172.21.15.7:0/4273024808 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8e38087810 con 0x7f8e3c072250 2023-03-03T18:32:55.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.844+0000 7f8e4237f700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e20068320 0x7f8e2006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:32:55.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.844+0000 7f8e4237f700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e20068320 0x7f8e2006a7d0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c000f60 tx=0x7f8e2c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:32:55.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:55.848+0000 7f8e32ffd700 1 -- 172.21.15.7:0/4273024808 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e38053bb0 con 0x7f8e3c072250 2023-03-03T18:32:56.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.088+0000 7f8e445e3700 1 -- 172.21.15.7:0/4273024808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e3c074bb0 con 0x7f8e3c072250 2023-03-03T18:32:56.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.089+0000 7f8e32ffd700 1 -- 172.21.15.7:0/4273024808 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8e380571d0 con 0x7f8e3c072250 2023-03-03T18:32:56.089 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:32:56.089 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:32:56.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 -- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e20068320 msgr2=0x7f8e2006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:56.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e20068320 0x7f8e2006a7d0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c000f60 tx=0x7f8e2c009040 comp rx=0 tx=0).stop 2023-03-03T18:32:56.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 -- 172.21.15.7:0/4273024808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 msgr2=0x7f8e3c146d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:32:56.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 0x7f8e3c146d50 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f8e38000c00 tx=0x7f8e38004750 comp rx=0 tx=0).stop 2023-03-03T18:32:56.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 -- 172.21.15.7:0/4273024808 shutdown_connections 2023-03-03T18:32:56.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e3c0c1ce0 0x7f8e3c145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:56.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e3c071e80 0x7f8e3c1edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:56.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e20068320 0x7f8e2006a7d0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:56.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 --2- 172.21.15.7:0/4273024808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e3c072250 0x7f8e3c146d50 unknown :-1 s=CLOSED pgs=3221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:32:56.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 -- 172.21.15.7:0/4273024808 >> 172.21.15.7:0/4273024808 conn(0x7f8e3c0a94d0 msgr2=0x7f8e3c06d590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:32:56.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 -- 172.21.15.7:0/4273024808 shutdown_connections 2023-03-03T18:32:56.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:32:56.091+0000 7f8e445e3700 1 -- 172.21.15.7:0/4273024808 wait complete. 2023-03-03T18:32:56.092 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:32:56.103 INFO:tasks.mds_thrash.fs.[b]:waiting for 77 secs before thrashing 2023-03-03T18:33:09.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:33:09.514 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:33:22.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:33:22.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.533+0000 7fb008a12700 1 Processor -- start 2023-03-03T18:33:22.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.533+0000 7fb008a12700 1 -- start start 2023-03-03T18:33:22.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb008a12700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 0x7fb004156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:33:22.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb008a12700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0040740f0 0x7fb00415cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:33:22.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb008a12700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb004155840 con 0x7fb0040c1da0 2023-03-03T18:33:22.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb008a12700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb004072940 con 0x7fb0040740f0 2023-03-03T18:33:22.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb008a12700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0041567d0 con 0x7fb00414fb00 2023-03-03T18:33:22.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb002d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0040740f0 0x7fb00415cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:33:22.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb002d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb0040740f0 0x7fb00415cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37620/0 (socket says 172.21.15.7:37620) 2023-03-03T18:33:22.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb002d9d700 1 -- 172.21.15.7:0/290420252 learned_addr learned my addr 172.21.15.7:0/290420252 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:33:22.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb00259c700 1 --2- 172.21.15.7:0/290420252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 0x7fb004156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:33:22.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb002d9d700 1 -- 172.21.15.7:0/290420252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 msgr2=0x7fb004156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:33:22.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb002d9d700 1 --2- 172.21.15.7:0/290420252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 0x7fb004156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:33:22.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb002d9d700 1 -- 172.21.15.7:0/290420252 >> v1:172.21.15.143:6789/0 conn(0x7fb0040c1da0 legacy=0x7fb0040c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:33:22.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb002d9d700 1 -- 172.21.15.7:0/290420252 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0040c2330 con 0x7fb0040740f0 2023-03-03T18:33:22.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb002d9d700 1 --2- 172.21.15.7:0/290420252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040740f0 0x7fb00415cbd0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7faffc009e50 tx=0x7faffc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646a0417c6bee489 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:33:22.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.534+0000 7fb00159a700 1 -- 172.21.15.7:0/290420252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc00e070 con 0x7fb0040740f0 2023-03-03T18:33:22.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.535+0000 7fb00159a700 1 -- 172.21.15.7:0/290420252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faffc00b9c0 con 0x7fb0040740f0 2023-03-03T18:33:22.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.535+0000 7fb00159a700 1 -- 172.21.15.7:0/290420252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc00bd70 con 0x7fb0040740f0 2023-03-03T18:33:22.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.535+0000 7fb008a12700 1 -- 172.21.15.7:0/290420252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040740f0 msgr2=0x7fb00415cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:33:22.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.535+0000 7fb008a12700 1 --2- 172.21.15.7:0/290420252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040740f0 0x7fb00415cbd0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7faffc009e50 tx=0x7faffc00b330 comp rx=0 tx=0).stop 2023-03-03T18:33:22.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.535+0000 7fb008a12700 1 -- 172.21.15.7:0/290420252 shutdown_connections 2023-03-03T18:33:22.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.535+0000 7fb008a12700 1 --2- 172.21.15.7:0/290420252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040740f0 0x7fb00415cbd0 unknown :-1 s=CLOSED pgs=3222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:33:22.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.535+0000 7fb008a12700 1 --2- 172.21.15.7:0/290420252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 0x7fb004156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:33:22.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.535+0000 7fb008a12700 1 -- 172.21.15.7:0/290420252 >> 172.21.15.7:0/290420252 conn(0x7fb00406c110 msgr2=0x7fb00406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:33:22.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.535+0000 7fb008a12700 1 -- 172.21.15.7:0/290420252 shutdown_connections 2023-03-03T18:33:22.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.535+0000 7fb008a12700 1 -- 172.21.15.7:0/290420252 wait complete. 2023-03-03T18:33:22.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.536+0000 7fb008a12700 1 Processor -- start 2023-03-03T18:33:22.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.537+0000 7fb008a12700 1 -- start start 2023-03-03T18:33:22.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.537+0000 7fb008a12700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0040740f0 0x7fb004071800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:33:22.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.537+0000 7fb008a12700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040c1da0 0x7fb0040c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:33:22.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.537+0000 7fb008a12700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 0x7fb0040701e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:33:22.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.537+0000 7fb008a12700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb004150ad0 con 0x7fb0040c1da0 2023-03-03T18:33:22.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.537+0000 7fb008a12700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0041e9bc0 con 0x7fb0040740f0 2023-03-03T18:33:22.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.537+0000 7fb008a12700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb004071d40 con 0x7fb00414fb00 2023-03-03T18:33:22.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.537+0000 7fb001d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040c1da0 0x7fb0040c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:33:22.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.537+0000 7fb001d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040c1da0 0x7fb0040c0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37624/0 (socket says 172.21.15.7:37624) 2023-03-03T18:33:22.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.537+0000 7fb001d9b700 1 -- 172.21.15.7:0/1707647834 learned_addr learned my addr 172.21.15.7:0/1707647834 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:33:22.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb00259c700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0040740f0 0x7fb004071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:33:22.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb002d9d700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 0x7fb0040701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:33:22.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb001d9b700 1 -- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 msgr2=0x7fb0040701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:33:22.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb001d9b700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 0x7fb0040701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:33:22.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb001d9b700 1 -- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0040740f0 msgr2=0x7fb004071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:33:22.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb001d9b700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0040740f0 0x7fb004071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:33:22.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb001d9b700 1 -- 172.21.15.7:0/1707647834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc009b30 con 0x7fb0040c1da0 2023-03-03T18:33:22.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb002d9d700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 0x7fb0040701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:33:22.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb00259c700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0040740f0 0x7fb004071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:33:22.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb001d9b700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040c1da0 0x7fb0040c0a40 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7faff800c480 tx=0x7faff800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:33:22.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7faff2ffd700 1 -- 172.21.15.7:0/1707647834 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff80052d0 con 0x7fb0040c1da0 2023-03-03T18:33:22.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7faff2ffd700 1 -- 172.21.15.7:0/1707647834 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff8005430 con 0x7fb0040c1da0 2023-03-03T18:33:22.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb008a12700 1 -- 172.21.15.7:0/1707647834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb004153090 con 0x7fb0040c1da0 2023-03-03T18:33:22.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7faff2ffd700 1 -- 172.21.15.7:0/1707647834 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff80038c0 con 0x7fb0040c1da0 2023-03-03T18:33:22.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.538+0000 7fb008a12700 1 -- 172.21.15.7:0/1707647834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb004152db0 con 0x7fb0040c1da0 2023-03-03T18:33:22.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.539+0000 7fb008a12700 1 -- 172.21.15.7:0/1707647834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb004155a90 con 0x7fb0040c1da0 2023-03-03T18:33:22.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.540+0000 7faff2ffd700 1 -- 172.21.15.7:0/1707647834 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faff8019030 con 0x7fb0040c1da0 2023-03-03T18:33:22.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.540+0000 7faff2ffd700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe40682f0 0x7fafe406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:33:22.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.540+0000 7faff2ffd700 1 -- 172.21.15.7:0/1707647834 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faff8085420 con 0x7fb0040c1da0 2023-03-03T18:33:22.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.540+0000 7fb00259c700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe40682f0 0x7fafe406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:33:22.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.541+0000 7fb00259c700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe40682f0 0x7fafe406a7a0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7faff4009820 tx=0x7faff4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:33:22.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.545+0000 7faff2ffd700 1 -- 172.21.15.7:0/1707647834 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faff80517c0 con 0x7fb0040c1da0 2023-03-03T18:33:22.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.783+0000 7fb008a12700 1 -- 172.21.15.7:0/1707647834 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb00406e410 con 0x7fb0040c1da0 2023-03-03T18:33:22.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.784+0000 7faff2ffd700 1 -- 172.21.15.7:0/1707647834 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faff8054de0 con 0x7fb0040c1da0 2023-03-03T18:33:22.784 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:33:22.784 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:33:22.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 -- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe40682f0 msgr2=0x7fafe406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:33:22.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe40682f0 0x7fafe406a7a0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7faff4009820 tx=0x7faff4008040 comp rx=0 tx=0).stop 2023-03-03T18:33:22.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 -- 172.21.15.7:0/1707647834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040c1da0 msgr2=0x7fb0040c0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:33:22.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040c1da0 0x7fb0040c0a40 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7faff800c480 tx=0x7faff800c840 comp rx=0 tx=0).stop 2023-03-03T18:33:22.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 -- 172.21.15.7:0/1707647834 shutdown_connections 2023-03-03T18:33:22.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb00414fb00 0x7fb0040701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:33:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0040740f0 0x7fb004071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:33:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fafe40682f0 0x7fafe406a7a0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:33:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 --2- 172.21.15.7:0/1707647834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0040c1da0 0x7fb0040c0a40 unknown :-1 s=CLOSED pgs=3223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:33:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 -- 172.21.15.7:0/1707647834 >> 172.21.15.7:0/1707647834 conn(0x7fb00406c110 msgr2=0x7fb004154fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:33:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 -- 172.21.15.7:0/1707647834 shutdown_connections 2023-03-03T18:33:22.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:33:22.786+0000 7fb008a12700 1 -- 172.21.15.7:0/1707647834 wait complete. 2023-03-03T18:33:22.788 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:33:22.797 INFO:tasks.mds_thrash.fs.[a]:waiting for 55 secs before thrashing 2023-03-03T18:33:39.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:33:39.580 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:34:09.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:34:09.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:34:13.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:34:13.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.243+0000 7f2503c71700 1 Processor -- start 2023-03-03T18:34:13.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.243+0000 7f2503c71700 1 -- start start 2023-03-03T18:34:13.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.244+0000 7f2503c71700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f24fc06d870 0x7f24fc1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:13.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.244+0000 7f2503c71700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f24fc0c3e30 0x7f24fc0c4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:13.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.244+0000 7f2503c71700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24fc0c7a60 con 0x7f24fc0c1da0 2023-03-03T18:34:13.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.244+0000 7f2503c71700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24fc0c2af0 con 0x7f24fc06d870 2023-03-03T18:34:13.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.244+0000 7f2503c71700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24fc0c3620 con 0x7f24fc0c3e30 2023-03-03T18:34:13.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.244+0000 7f2501a0d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f24fc06d870 0x7f24fc1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:13.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.244+0000 7f250120c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f24fc0c1da0 0x7f24fc0c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59862/0 (socket says 172.21.15.7:59862) 2023-03-03T18:34:13.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.244+0000 7f250120c700 1 -- 172.21.15.7:0/2350628148 learned_addr learned my addr 172.21.15.7:0/2350628148 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:34:13.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.244+0000 7f2501a0d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f24fc06d870 0x7f24fc1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37650/0 (socket says 172.21.15.7:37650) 2023-03-03T18:34:13.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.244+0000 7f250220e700 1 --2- 172.21.15.7:0/2350628148 >> v2:172.21.15.143:3300/0 conn(0x7f24fc0c3e30 0x7f24fc0c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:13.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2501a0d700 1 -- 172.21.15.7:0/2350628148 >> v2:172.21.15.143:3300/0 conn(0x7f24fc0c3e30 msgr2=0x7f24fc0c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:34:13.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2501a0d700 1 --2- 172.21.15.7:0/2350628148 >> v2:172.21.15.143:3300/0 conn(0x7f24fc0c3e30 0x7f24fc0c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:13.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2501a0d700 1 -- 172.21.15.7:0/2350628148 >> v1:172.21.15.7:6789/0 conn(0x7f24fc0c1da0 legacy=0x7f24fc0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:13.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2501a0d700 1 -- 172.21.15.7:0/2350628148 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc1545e0 con 0x7f24fc06d870 2023-03-03T18:34:13.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2500a0b700 1 -- 172.21.15.7:0/2350628148 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057582992 0 0) 0x7f24fc0c7a60 con 0x7f24fc0c1da0 2023-03-03T18:34:13.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2501a0d700 1 --2- 172.21.15.7:0/2350628148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 0x7f24fc1550c0 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 crypto rx=0x7f24fc073830 tx=0x7f24ec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb7f7e8c909fdb57 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:13.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2500a0b700 1 -- 172.21.15.7:0/2350628148 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec011830 con 0x7f24fc06d870 2023-03-03T18:34:13.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2500a0b700 1 -- 172.21.15.7:0/2350628148 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ec011990 con 0x7f24fc06d870 2023-03-03T18:34:13.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2500a0b700 1 -- 172.21.15.7:0/2350628148 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec011d20 con 0x7f24fc06d870 2023-03-03T18:34:13.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2503c71700 1 -- 172.21.15.7:0/2350628148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 msgr2=0x7f24fc1550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:13.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2503c71700 1 --2- 172.21.15.7:0/2350628148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 0x7f24fc1550c0 secure :-1 s=READY pgs=3225 cs=0 l=1 rev1=1 crypto rx=0x7f24fc073830 tx=0x7f24ec009b20 comp rx=0 tx=0).stop 2023-03-03T18:34:13.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2503c71700 1 -- 172.21.15.7:0/2350628148 shutdown_connections 2023-03-03T18:34:13.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2503c71700 1 --2- 172.21.15.7:0/2350628148 >> v2:172.21.15.143:3300/0 conn(0x7f24fc0c3e30 0x7f24fc0c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:13.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.245+0000 7f2503c71700 1 --2- 172.21.15.7:0/2350628148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 0x7f24fc1550c0 unknown :-1 s=CLOSED pgs=3225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:13.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.246+0000 7f2503c71700 1 -- 172.21.15.7:0/2350628148 >> 172.21.15.7:0/2350628148 conn(0x7f24fc06c110 msgr2=0x7f24fc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:34:13.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.246+0000 7f2503c71700 1 -- 172.21.15.7:0/2350628148 shutdown_connections 2023-03-03T18:34:13.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.246+0000 7f2503c71700 1 -- 172.21.15.7:0/2350628148 wait complete. 2023-03-03T18:34:13.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.247+0000 7f2503c71700 1 Processor -- start 2023-03-03T18:34:13.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.247+0000 7f2503c71700 1 -- start start 2023-03-03T18:34:13.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2503c71700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 0x7f24fc0ab790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:13.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2503c71700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24fc0c1da0 0x7f24fc155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:13.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2503c71700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24fc0c3e30 0x7f24fc142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:13.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2503c71700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc157990 con 0x7f24fc06d870 2023-03-03T18:34:13.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2503c71700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc0ac5d0 con 0x7f24fc0c3e30 2023-03-03T18:34:13.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2503c71700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc0acde0 con 0x7f24fc0c1da0 2023-03-03T18:34:13.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2501a0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 0x7f24fc0ab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:13.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2501a0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 0x7f24fc0ab790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37654/0 (socket says 172.21.15.7:37654) 2023-03-03T18:34:13.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2501a0d700 1 -- 172.21.15.7:0/1233087319 learned_addr learned my addr 172.21.15.7:0/1233087319 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:34:13.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f250120c700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24fc0c1da0 0x7f24fc155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:13.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f250220e700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24fc0c3e30 0x7f24fc142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:13.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2501a0d700 1 -- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24fc0c1da0 msgr2=0x7f24fc155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:13.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2501a0d700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24fc0c1da0 0x7f24fc155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:13.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2501a0d700 1 -- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24fc0c3e30 msgr2=0x7f24fc142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:13.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2501a0d700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24fc0c3e30 0x7f24fc142800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:13.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2501a0d700 1 -- 172.21.15.7:0/1233087319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec0094a0 con 0x7f24fc06d870 2023-03-03T18:34:13.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f250120c700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24fc0c1da0 0x7f24fc155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:34:13.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2501a0d700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 0x7f24fc0ab790 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0090b0 tx=0x7f24ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:13.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f24f27fc700 1 -- 172.21.15.7:0/1233087319 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec0036a0 con 0x7f24fc06d870 2023-03-03T18:34:13.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f24f27fc700 1 -- 172.21.15.7:0/1233087319 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ec01f070 con 0x7f24fc06d870 2023-03-03T18:34:13.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f2503c71700 1 -- 172.21.15.7:0/1233087319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24fc154ba0 con 0x7f24fc06d870 2023-03-03T18:34:13.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.248+0000 7f24f27fc700 1 -- 172.21.15.7:0/1233087319 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec003a00 con 0x7f24fc06d870 2023-03-03T18:34:13.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.249+0000 7f2503c71700 1 -- 172.21.15.7:0/1233087319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24fc1548c0 con 0x7f24fc06d870 2023-03-03T18:34:13.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.250+0000 7f2503c71700 1 -- 172.21.15.7:0/1233087319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24fc06dc40 con 0x7f24fc06d870 2023-03-03T18:34:13.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.250+0000 7f24f27fc700 1 -- 172.21.15.7:0/1233087319 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f24ec00f050 con 0x7f24fc06d870 2023-03-03T18:34:13.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.250+0000 7f24f27fc700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e00682f0 0x7f24e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:13.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.250+0000 7f24f27fc700 1 -- 172.21.15.7:0/1233087319 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f24ec021070 con 0x7f24fc06d870 2023-03-03T18:34:13.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.251+0000 7f250120c700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e00682f0 0x7f24e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:13.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.251+0000 7f250120c700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e00682f0 0x7f24e006a7a0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f24fc0ad3f0 tx=0x7f24f8007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:13.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.255+0000 7f24f27fc700 1 -- 172.21.15.7:0/1233087319 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24ec053de0 con 0x7f24fc06d870 2023-03-03T18:34:13.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.494+0000 7f2503c71700 1 -- 172.21.15.7:0/1233087319 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24fc074bb0 con 0x7f24fc06d870 2023-03-03T18:34:13.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.494+0000 7f24f27fc700 1 -- 172.21.15.7:0/1233087319 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f24ec057400 con 0x7f24fc06d870 2023-03-03T18:34:13.495 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:34:13.495 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:34:13.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 -- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e00682f0 msgr2=0x7f24e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:13.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e00682f0 0x7f24e006a7a0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f24fc0ad3f0 tx=0x7f24f8007ca0 comp rx=0 tx=0).stop 2023-03-03T18:34:13.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 -- 172.21.15.7:0/1233087319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 msgr2=0x7f24fc0ab790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:13.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 0x7f24fc0ab790 secure :-1 s=READY pgs=3226 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0090b0 tx=0x7f24ec004750 comp rx=0 tx=0).stop 2023-03-03T18:34:13.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 -- 172.21.15.7:0/1233087319 shutdown_connections 2023-03-03T18:34:13.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24fc0c3e30 0x7f24fc142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:13.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24e00682f0 0x7f24e006a7a0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:13.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24fc06d870 0x7f24fc0ab790 unknown :-1 s=CLOSED pgs=3226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:13.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 --2- 172.21.15.7:0/1233087319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24fc0c1da0 0x7f24fc155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:13.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 -- 172.21.15.7:0/1233087319 >> 172.21.15.7:0/1233087319 conn(0x7f24fc06c110 msgr2=0x7f24fc070910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:34:13.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 -- 172.21.15.7:0/1233087319 shutdown_connections 2023-03-03T18:34:13.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:13.496+0000 7f2503c71700 1 -- 172.21.15.7:0/1233087319 wait complete. 2023-03-03T18:34:13.499 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:34:13.509 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before thrashing 2023-03-03T18:34:17.510 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:34:17.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.649+0000 7f0470b3c700 1 Processor -- start 2023-03-03T18:34:17.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.649+0000 7f0470b3c700 1 -- start start 2023-03-03T18:34:17.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f0470b3c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f046c14fc50 0x7f046c0c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f0470b3c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046c0ad2b0 con 0x7f046c06d240 2023-03-03T18:34:17.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f0470b3c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046c156420 con 0x7f046c0c3e30 2023-03-03T18:34:17.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f0470b3c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046c0c30c0 con 0x7f046c14fc50 2023-03-03T18:34:17.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f046a59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f046c14fc50 0x7f046c0c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f0469d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f046c06d240 0x7f046c0c27b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59876/0 (socket says 172.21.15.7:59876) 2023-03-03T18:34:17.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f0469d9b700 1 -- 172.21.15.7:0/4099988894 learned_addr learned my addr 172.21.15.7:0/4099988894 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:34:17.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f046a59c700 1 -- 172.21.15.7:0/4099988894 >> v1:172.21.15.143:6789/0 conn(0x7f046c0c3e30 legacy=0x7f046c15cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f046a59c700 1 -- 172.21.15.7:0/4099988894 >> v1:172.21.15.7:6789/0 conn(0x7f046c06d240 legacy=0x7f046c0c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f046a59c700 1 -- 172.21.15.7:0/4099988894 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c0ad520 con 0x7f046c14fc50 2023-03-03T18:34:17.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f046a59c700 1 --2- 172.21.15.7:0/4099988894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c14fc50 0x7f046c0c1f50 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7f045c009840 tx=0x7f045c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed6ed1a6576810a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:17.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f046959a700 1 -- 172.21.15.7:0/4099988894 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c011830 con 0x7f046c14fc50 2023-03-03T18:34:17.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f046959a700 1 -- 172.21.15.7:0/4099988894 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c011990 con 0x7f046c14fc50 2023-03-03T18:34:17.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.650+0000 7f046959a700 1 -- 172.21.15.7:0/4099988894 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c011d20 con 0x7f046c14fc50 2023-03-03T18:34:17.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.651+0000 7f0470b3c700 1 -- 172.21.15.7:0/4099988894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c14fc50 msgr2=0x7f046c0c1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.651+0000 7f0470b3c700 1 --2- 172.21.15.7:0/4099988894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c14fc50 0x7f046c0c1f50 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 crypto rx=0x7f045c009840 tx=0x7f045c009b50 comp rx=0 tx=0).stop 2023-03-03T18:34:17.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.651+0000 7f0470b3c700 1 -- 172.21.15.7:0/4099988894 shutdown_connections 2023-03-03T18:34:17.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.651+0000 7f0470b3c700 1 --2- 172.21.15.7:0/4099988894 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c14fc50 0x7f046c0c1f50 unknown :-1 s=CLOSED pgs=3228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.651+0000 7f0470b3c700 1 -- 172.21.15.7:0/4099988894 >> 172.21.15.7:0/4099988894 conn(0x7f046c06c110 msgr2=0x7f046c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:34:17.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.651+0000 7f0470b3c700 1 -- 172.21.15.7:0/4099988894 shutdown_connections 2023-03-03T18:34:17.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.651+0000 7f0470b3c700 1 -- 172.21.15.7:0/4099988894 wait complete. 2023-03-03T18:34:17.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.652+0000 7f0470b3c700 1 Processor -- start 2023-03-03T18:34:17.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.653+0000 7f0470b3c700 1 -- start start 2023-03-03T18:34:17.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.653+0000 7f0470b3c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f046c06d240 0x7f046c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.653+0000 7f0470b3c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c0c3e30 0x7f046c0c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.653+0000 7f0470b3c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f046c14fc50 0x7f046c1ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.653+0000 7f0470b3c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c1ea2a0 con 0x7f046c0c3e30 2023-03-03T18:34:17.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.653+0000 7f0470b3c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c1dd920 con 0x7f046c06d240 2023-03-03T18:34:17.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.653+0000 7f0470b3c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c148af0 con 0x7f046c14fc50 2023-03-03T18:34:17.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.653+0000 7f0469d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c0c3e30 0x7f046c0c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.653+0000 7f0469d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c0c3e30 0x7f046c0c0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37672/0 (socket says 172.21.15.7:37672) 2023-03-03T18:34:17.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.653+0000 7f0469d9b700 1 -- 172.21.15.7:0/2629607997 learned_addr learned my addr 172.21.15.7:0/2629607997 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:34:17.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f046a59c700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f046c06d240 0x7f046c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f046ad9d700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f046c14fc50 0x7f046c1ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0469d9b700 1 -- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f046c14fc50 msgr2=0x7f046c1ea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0469d9b700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f046c14fc50 0x7f046c1ea6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0469d9b700 1 -- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f046c06d240 msgr2=0x7f046c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0469d9b700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f046c06d240 0x7f046c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0469d9b700 1 -- 172.21.15.7:0/2629607997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c0094a0 con 0x7f046c0c3e30 2023-03-03T18:34:17.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f046ad9d700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f046c14fc50 0x7f046c1ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:34:17.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0469d9b700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c0c3e30 0x7f046c0c0a40 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 crypto rx=0x7f045400c480 tx=0x7f045400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:17.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0462ffd700 1 -- 172.21.15.7:0/2629607997 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454005d70 con 0x7f046c0c3e30 2023-03-03T18:34:17.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0462ffd700 1 -- 172.21.15.7:0/2629607997 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04540070d0 con 0x7f046c0c3e30 2023-03-03T18:34:17.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0470b3c700 1 -- 172.21.15.7:0/2629607997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046c074220 con 0x7f046c0c3e30 2023-03-03T18:34:17.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0462ffd700 1 -- 172.21.15.7:0/2629607997 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0454007440 con 0x7f046c0c3e30 2023-03-03T18:34:17.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.654+0000 7f0470b3c700 1 -- 172.21.15.7:0/2629607997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046c0c2490 con 0x7f046c0c3e30 2023-03-03T18:34:17.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.655+0000 7f0470b3c700 1 -- 172.21.15.7:0/2629607997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046c156690 con 0x7f046c0c3e30 2023-03-03T18:34:17.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.656+0000 7f0462ffd700 1 -- 172.21.15.7:0/2629607997 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0454019030 con 0x7f046c0c3e30 2023-03-03T18:34:17.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.656+0000 7f0462ffd700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f044c0682f0 0x7f044c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.656+0000 7f0462ffd700 1 -- 172.21.15.7:0/2629607997 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0454085470 con 0x7f046c0c3e30 2023-03-03T18:34:17.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.656+0000 7f046a59c700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f044c0682f0 0x7f044c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.657+0000 7f046a59c700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f044c0682f0 0x7f044c06a7a0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f045c0090b0 tx=0x7f045c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:17.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.661+0000 7f0462ffd700 1 -- 172.21.15.7:0/2629607997 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f045404b5e0 con 0x7f046c0c3e30 2023-03-03T18:34:17.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:34:17.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.901+0000 7f0470b3c700 1 -- 172.21.15.7:0/2629607997 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f046c1ea3e0 con 0x7f046c0c3e30 2023-03-03T18:34:17.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.902+0000 7f0462ffd700 1 -- 172.21.15.7:0/2629607997 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f046c1ea3e0 con 0x7f046c0c3e30 2023-03-03T18:34:17.903 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:34:17.904 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:34:17.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 -- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f044c0682f0 msgr2=0x7f044c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f044c0682f0 0x7f044c06a7a0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f045c0090b0 tx=0x7f045c008030 comp rx=0 tx=0).stop 2023-03-03T18:34:17.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 -- 172.21.15.7:0/2629607997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c0c3e30 msgr2=0x7f046c0c0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c0c3e30 0x7f046c0c0a40 secure :-1 s=READY pgs=3229 cs=0 l=1 rev1=1 crypto rx=0x7f045400c480 tx=0x7f045400c840 comp rx=0 tx=0).stop 2023-03-03T18:34:17.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 -- 172.21.15.7:0/2629607997 shutdown_connections 2023-03-03T18:34:17.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f046c14fc50 0x7f046c1ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f046c06d240 0x7f046c0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f044c0682f0 0x7f044c06a7a0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 --2- 172.21.15.7:0/2629607997 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f046c0c3e30 0x7f046c0c0a40 unknown :-1 s=CLOSED pgs=3229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 -- 172.21.15.7:0/2629607997 >> 172.21.15.7:0/2629607997 conn(0x7f046c06c110 msgr2=0x7f046c0c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:34:17.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 -- 172.21.15.7:0/2629607997 shutdown_connections 2023-03-03T18:34:17.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.904+0000 7f0470b3c700 1 -- 172.21.15.7:0/2629607997 wait complete. 2023-03-03T18:34:17.907 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:34:17.918 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:34:17.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.931+0000 7f78f9aad700 1 Processor -- start 2023-03-03T18:34:17.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.931+0000 7f78f9aad700 1 -- start start 2023-03-03T18:34:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.932+0000 7f78f9aad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c65c0 0x7f78f40c2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.932+0000 7f78f9aad700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f40a65b0 con 0x7f78f414de50 2023-03-03T18:34:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.932+0000 7f78f9aad700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f41541f0 con 0x7f78f40a9af0 2023-03-03T18:34:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.932+0000 7f78f9aad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f4154e00 con 0x7f78f40c65c0 2023-03-03T18:34:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.932+0000 7f78f37fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f78f414de50 0x7f78f406a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59890/0 (socket says 172.21.15.7:59890) 2023-03-03T18:34:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.932+0000 7f78f37fe700 1 -- 172.21.15.7:0/2990964859 learned_addr learned my addr 172.21.15.7:0/2990964859 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:34:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.932+0000 7f78f2ffd700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c65c0 0x7f78f40c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.932+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779359436 0 0) 0x7f78f40a65b0 con 0x7f78f414de50 2023-03-03T18:34:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.932+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78dc0035b0 con 0x7f78f414de50 2023-03-03T18:34:17.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 569285020 0 0) 0x7f78dc0035b0 con 0x7f78f414de50 2023-03-03T18:34:17.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c65c0 msgr2=0x7f78f40c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:34:17.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c65c0 0x7f78f40c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 >> v1:172.21.15.143:6789/0 conn(0x7f78f40a9af0 legacy=0x7f78f40c2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f41544d0 con 0x7f78f414de50 2023-03-03T18:34:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f78e4001670 con 0x7f78f414de50 2023-03-03T18:34:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 >> v1:172.21.15.7:6789/0 conn(0x7f78f414de50 legacy=0x7f78f406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 0x7f78dc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78dc004850 0x7f78dc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78dc005200 0x7f78dc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f4154e00 con 0x7f78dc004850 2023-03-03T18:34:17.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc00bfd0 con 0x7f78dc003aa0 2023-03-03T18:34:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc00c190 con 0x7f78dc005200 2023-03-03T18:34:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f2ffd700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c65c0 0x7f78f40c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:34:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f37fe700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 0x7f78dc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f2ffd700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78dc004850 0x7f78dc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f3fff700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78dc005200 0x7f78dc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f37fe700 1 -- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78dc005200 msgr2=0x7f78dc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:34:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f37fe700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78dc005200 0x7f78dc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f37fe700 1 -- 172.21.15.7:0/2990964859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78dc004850 msgr2=0x7f78dc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:34:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f37fe700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78dc004850 0x7f78dc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f37fe700 1 -- 172.21.15.7:0/2990964859 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78dc0035b0 con 0x7f78dc003aa0 2023-03-03T18:34:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.933+0000 7f78f2ffd700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78dc004850 0x7f78dc004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:34:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f37fe700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 0x7f78dc003e70 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f78e400c2f0 tx=0x7f78dc005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62efbd90c74f6aff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:17.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e400cd70 con 0x7f78dc003aa0 2023-03-03T18:34:17.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e4005960 con 0x7f78dc003aa0 2023-03-03T18:34:17.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f27fc700 1 -- 172.21.15.7:0/2990964859 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e4005d10 con 0x7f78dc003aa0 2023-03-03T18:34:17.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f9aad700 1 -- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 msgr2=0x7f78dc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f9aad700 1 --2- 172.21.15.7:0/2990964859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 0x7f78dc003e70 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f78e400c2f0 tx=0x7f78dc005880 comp rx=0 tx=0).stop 2023-03-03T18:34:17.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f3fff700 1 -- 172.21.15.7:0/2990964859 reap_dead start 2023-03-03T18:34:17.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f9aad700 1 -- 172.21.15.7:0/2990964859 shutdown_connections 2023-03-03T18:34:17.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f9aad700 1 -- 172.21.15.7:0/2990964859 >> 172.21.15.7:0/2990964859 conn(0x7f78f40a9430 msgr2=0x7f78f40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:34:17.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f9aad700 1 -- 172.21.15.7:0/2990964859 shutdown_connections 2023-03-03T18:34:17.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.934+0000 7f78f9aad700 1 -- 172.21.15.7:0/2990964859 wait complete. 2023-03-03T18:34:17.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.936+0000 7f78f9aad700 1 Processor -- start 2023-03-03T18:34:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.936+0000 7f78f9aad700 1 -- start start 2023-03-03T18:34:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.936+0000 7f78f9aad700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 0x7f78f40c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.936+0000 7f78f9aad700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c3ae0 0x7f78f406beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.936+0000 7f78f9aad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78f406c3f0 0x7f78f41e8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.936+0000 7f78f9aad700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f414a330 con 0x7f78f406c3f0 2023-03-03T18:34:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.936+0000 7f78f9aad700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f414a750 con 0x7f78dc003aa0 2023-03-03T18:34:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.936+0000 7f78f9aad700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f41499d0 con 0x7f78f40c3ae0 2023-03-03T18:34:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78f406c3f0 0x7f78f41e8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78f406c3f0 0x7f78f41e8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37694/0 (socket says 172.21.15.7:37694) 2023-03-03T18:34:17.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f3fff700 1 -- 172.21.15.7:0/912863504 learned_addr learned my addr 172.21.15.7:0/912863504 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:34:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f2ffd700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c3ae0 0x7f78f406beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f37fe700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 0x7f78f40c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f3fff700 1 -- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c3ae0 msgr2=0x7f78f406beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f3fff700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c3ae0 0x7f78f406beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f3fff700 1 -- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 msgr2=0x7f78f40c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f3fff700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 0x7f78f40c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f3fff700 1 -- 172.21.15.7:0/912863504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f4154a90 con 0x7f78f406c3f0 2023-03-03T18:34:17.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f37fe700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 0x7f78f40c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:34:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f2ffd700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c3ae0 0x7f78f406beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:34:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f3fff700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78f406c3f0 0x7f78f41e8a10 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7f78f40c60a0 tx=0x7f78ec00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78dbfff700 1 -- 172.21.15.7:0/912863504 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec00dc60 con 0x7f78f406c3f0 2023-03-03T18:34:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78dbfff700 1 -- 172.21.15.7:0/912863504 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ec00ddc0 con 0x7f78f406c3f0 2023-03-03T18:34:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78dbfff700 1 -- 172.21.15.7:0/912863504 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec003c20 con 0x7f78f406c3f0 2023-03-03T18:34:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f9aad700 1 -- 172.21.15.7:0/912863504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f41547b0 con 0x7f78f406c3f0 2023-03-03T18:34:17.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.937+0000 7f78f9aad700 1 -- 172.21.15.7:0/912863504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f4153e80 con 0x7f78f406c3f0 2023-03-03T18:34:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.938+0000 7f78f9aad700 1 -- 172.21.15.7:0/912863504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f414a470 con 0x7f78f406c3f0 2023-03-03T18:34:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.939+0000 7f78dbfff700 1 -- 172.21.15.7:0/912863504 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f78ec0050d0 con 0x7f78f406c3f0 2023-03-03T18:34:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.939+0000 7f78dbfff700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78e00682f0 0x7f78e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.939+0000 7f78dbfff700 1 -- 172.21.15.7:0/912863504 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f78ec0864a0 con 0x7f78f406c3f0 2023-03-03T18:34:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.940+0000 7f78f37fe700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78e00682f0 0x7f78e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.940+0000 7f78f37fe700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78e00682f0 0x7f78e006a7a0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f78e4015040 tx=0x7f78e4007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:17.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:17.944+0000 7f78dbfff700 1 -- 172.21.15.7:0/912863504 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f78f414a470 con 0x7f78f406c3f0 2023-03-03T18:34:18.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.043+0000 7fe7ed96e700 1 Processor -- start 2023-03-03T18:34:18.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.043+0000 7fe7ed96e700 1 -- start start 2023-03-03T18:34:18.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7ed96e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7e8151e80 0x7fe7e80c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:18.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7ed96e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe7e8155410 0x7fe7e80bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:18.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7ed96e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e80c6fe0 0x7fe7e80bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:18.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7ed96e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e8152250 con 0x7fe7e8151e80 2023-03-03T18:34:18.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7ed96e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e80c0fb0 con 0x7fe7e8155410 2023-03-03T18:34:18.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7ed96e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e8154740 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7e8151e80 0x7fe7e80c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7e8151e80 0x7fe7e80c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37698/0 (socket says 172.21.15.7:37698) 2023-03-03T18:34:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/1028254326 learned_addr learned my addr 172.21.15.7:0/1028254326 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:34:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e67fc700 1 --2- 172.21.15.7:0/1028254326 >> v2:172.21.15.143:3300/0 conn(0x7fe7e8155410 0x7fe7e80bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e77fe700 1 --2- 172.21.15.7:0/1028254326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e80c6fe0 0x7fe7e80bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/1028254326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e80c6fe0 msgr2=0x7fe7e80bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:34:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e6ffd700 1 --2- 172.21.15.7:0/1028254326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e80c6fe0 0x7fe7e80bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/1028254326 >> v2:172.21.15.143:3300/0 conn(0x7fe7e8155410 msgr2=0x7fe7e80bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:34:18.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e6ffd700 1 --2- 172.21.15.7:0/1028254326 >> v2:172.21.15.143:3300/0 conn(0x7fe7e8155410 0x7fe7e80bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/1028254326 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e80c08d0 con 0x7fe7e8151e80 2023-03-03T18:34:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.044+0000 7fe7e67fc700 1 --2- 172.21.15.7:0/1028254326 >> v2:172.21.15.143:3300/0 conn(0x7fe7e8155410 0x7fe7e80bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:34:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.045+0000 7fe7e6ffd700 1 --2- 172.21.15.7:0/1028254326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e8151e80 0x7fe7e80c6aa0 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7fe7e806c380 tx=0x7fe7d8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911c9cae1674b4b0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.045+0000 7fe7e5ffb700 1 -- 172.21.15.7:0/1028254326 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d800b870 con 0x7fe7e8151e80 2023-03-03T18:34:18.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.045+0000 7fe7e5ffb700 1 -- 172.21.15.7:0/1028254326 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d800b9d0 con 0x7fe7e8151e80 2023-03-03T18:34:18.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.045+0000 7fe7e5ffb700 1 -- 172.21.15.7:0/1028254326 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d800bd60 con 0x7fe7e8151e80 2023-03-03T18:34:18.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.045+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1028254326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e8151e80 msgr2=0x7fe7e80c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:18.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.045+0000 7fe7ed96e700 1 --2- 172.21.15.7:0/1028254326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e8151e80 0x7fe7e80c6aa0 secure :-1 s=READY pgs=3232 cs=0 l=1 rev1=1 crypto rx=0x7fe7e806c380 tx=0x7fe7d8009bd0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.046+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1028254326 shutdown_connections 2023-03-03T18:34:18.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.046+0000 7fe7ed96e700 1 --2- 172.21.15.7:0/1028254326 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e80c6fe0 0x7fe7e80bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.046+0000 7fe7ed96e700 1 --2- 172.21.15.7:0/1028254326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e8151e80 0x7fe7e80c6aa0 unknown :-1 s=CLOSED pgs=3232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.046+0000 7fe7ed96e700 1 --2- 172.21.15.7:0/1028254326 >> v2:172.21.15.143:3300/0 conn(0x7fe7e8155410 0x7fe7e80bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.046+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1028254326 >> 172.21.15.7:0/1028254326 conn(0x7fe7e8071630 msgr2=0x7fe7e806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:34:18.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.046+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1028254326 shutdown_connections 2023-03-03T18:34:18.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.046+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1028254326 wait complete. 2023-03-03T18:34:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.047+0000 7fe7ed96e700 1 Processor -- start 2023-03-03T18:34:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.047+0000 7fe7ed96e700 1 -- start start 2023-03-03T18:34:18.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7ed96e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e80c6fe0 0x7fe7e80c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:18.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7ed96e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e8151e80 0x7fe7e81ea530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:18.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7ed96e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e8155410 0x7fe7e814ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:18.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7ed96e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e80c9220 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7ed96e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e80ca5b0 con 0x7fe7e8155410 2023-03-03T18:34:18.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7ed96e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e8147d00 con 0x7fe7e8151e80 2023-03-03T18:34:18.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e80c6fe0 0x7fe7e80c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:18.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e80c6fe0 0x7fe7e80c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37704/0 (socket says 172.21.15.7:37704) 2023-03-03T18:34:18.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/1540158569 learned_addr learned my addr 172.21.15.7:0/1540158569 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:34:18.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e77fe700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e8155410 0x7fe7e814ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:18.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e67fc700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e8151e80 0x7fe7e81ea530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:18.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e8151e80 msgr2=0x7fe7e81ea530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:18.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e6ffd700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e8151e80 0x7fe7e81ea530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e8155410 msgr2=0x7fe7e814ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:18.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e6ffd700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e8155410 0x7fe7e814ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e6ffd700 1 -- 172.21.15.7:0/1540158569 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d8009550 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e77fe700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e8155410 0x7fe7e814ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:34:18.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e67fc700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e8151e80 0x7fe7e81ea530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:34:18.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7e6ffd700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e80c6fe0 0x7fe7e80c8760 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8014040 tx=0x7fe7d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:18.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7cffff700 1 -- 172.21.15.7:0/1540158569 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8004520 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7cffff700 1 -- 172.21.15.7:0/1540158569 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d801e070 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7cffff700 1 -- 172.21.15.7:0/1540158569 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80038e0 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.048+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1540158569 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e8154c10 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.049+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1540158569 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e8154930 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.049+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1540158569 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e8154ef0 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.050+0000 7fe7cffff700 1 -- 172.21.15.7:0/1540158569 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe7d800f050 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.050+0000 7fe7cffff700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d40682f0 0x7fe7d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:34:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.050+0000 7fe7cffff700 1 -- 172.21.15.7:0/1540158569 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe7d8020070 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.050+0000 7fe7e67fc700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d40682f0 0x7fe7d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:34:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.051+0000 7fe7e67fc700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d40682f0 0x7fe7d406a7a0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80ca310 tx=0x7fe7dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:34:18.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.055+0000 7fe7cffff700 1 -- 172.21.15.7:0/1540158569 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe7d8053ba0 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.182+0000 7f78f9aad700 1 -- 172.21.15.7:0/912863504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78f414a470 con 0x7f78f406c3f0 2023-03-03T18:34:18.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.183+0000 7f78dbfff700 1 -- 172.21.15.7:0/912863504 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f78f414a470 con 0x7f78f406c3f0 2023-03-03T18:34:18.184 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:34:18.184 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:34:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 -- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78e00682f0 msgr2=0x7f78e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78e00682f0 0x7f78e006a7a0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f78e4015040 tx=0x7f78e4007000 comp rx=0 tx=0).stop 2023-03-03T18:34:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 -- 172.21.15.7:0/912863504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78f406c3f0 msgr2=0x7f78f41e8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78f406c3f0 0x7f78f41e8a10 secure :-1 s=READY pgs=3231 cs=0 l=1 rev1=1 crypto rx=0x7f78f40c60a0 tx=0x7f78ec00d430 comp rx=0 tx=0).stop 2023-03-03T18:34:18.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 -- 172.21.15.7:0/912863504 shutdown_connections 2023-03-03T18:34:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78e00682f0 0x7f78e006a7a0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78f406c3f0 0x7f78f41e8a10 unknown :-1 s=CLOSED pgs=3231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78dc003aa0 0x7f78f40c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 --2- 172.21.15.7:0/912863504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78f40c3ae0 0x7f78f406beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 -- 172.21.15.7:0/912863504 >> 172.21.15.7:0/912863504 conn(0x7f78f40a9430 msgr2=0x7f78f414b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:34:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 -- 172.21.15.7:0/912863504 shutdown_connections 2023-03-03T18:34:18.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.185+0000 7f78f9aad700 1 -- 172.21.15.7:0/912863504 wait complete. 2023-03-03T18:34:18.188 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:34:18.198 INFO:tasks.mds_thrash.fs.[a]:waiting for 76 secs before thrashing 2023-03-03T18:34:18.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.292+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1540158569 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7e8074bb0 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.293+0000 7fe7cffff700 1 -- 172.21.15.7:0/1540158569 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe7d80571c0 con 0x7fe7e80c6fe0 2023-03-03T18:34:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:34:18.294 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:34:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d40682f0 msgr2=0x7fe7d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d40682f0 0x7fe7d406a7a0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80ca310 tx=0x7fe7dc008040 comp rx=0 tx=0).stop 2023-03-03T18:34:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1540158569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e80c6fe0 msgr2=0x7fe7e80c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:34:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e80c6fe0 0x7fe7e80c8760 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8014040 tx=0x7fe7d8004750 comp rx=0 tx=0).stop 2023-03-03T18:34:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1540158569 shutdown_connections 2023-03-03T18:34:18.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe7e8155410 0x7fe7e814ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe7d40682f0 0x7fe7d406a7a0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7e80c6fe0 0x7fe7e80c8760 unknown :-1 s=CLOSED pgs=3233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 --2- 172.21.15.7:0/1540158569 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe7e8151e80 0x7fe7e81ea530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:34:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1540158569 >> 172.21.15.7:0/1540158569 conn(0x7fe7e8071630 msgr2=0x7fe7e80c3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:34:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1540158569 shutdown_connections 2023-03-03T18:34:18.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:34:18.295+0000 7fe7ed96e700 1 -- 172.21.15.7:0/1540158569 wait complete. 2023-03-03T18:34:18.298 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:34:18.308 INFO:tasks.mds_thrash.fs.[b]:waiting for 104 secs before thrashing 2023-03-03T18:34:39.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:34:39.730 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:35:09.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:35:09.803 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:35:34.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:35:34.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.339+0000 7fd155567700 1 Processor -- start 2023-03-03T18:35:34.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.339+0000 7fd155567700 1 -- start start 2023-03-03T18:35:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd155567700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd150151dc0 0x7fd1500ca630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:35:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd155567700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd1500cab70 0x7fd150158710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:35:34.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd155567700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd150152190 con 0x7fd1500c6980 2023-03-03T18:35:34.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd155567700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1500c6310 con 0x7fd1500cab70 2023-03-03T18:35:34.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd155567700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd150158c50 con 0x7fd150151dc0 2023-03-03T18:35:34.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd1500cab70 0x7fd150158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:35:34.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14e7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd1500c6980 0x7fd150158000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59996/0 (socket says 172.21.15.7:59996) 2023-03-03T18:35:34.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14e7fc700 1 -- 172.21.15.7:0/1183792592 learned_addr learned my addr 172.21.15.7:0/1183792592 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:35:34.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14dffb700 1 -- 172.21.15.7:0/1183792592 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935690400 0 0) 0x7fd150152190 con 0x7fd1500c6980 2023-03-03T18:35:34.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14effd700 1 --2- 172.21.15.7:0/1183792592 >> v2:172.21.15.143:3300/0 conn(0x7fd150151dc0 0x7fd1500ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:35:34.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14dffb700 1 -- 172.21.15.7:0/1183792592 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1380035b0 con 0x7fd1500c6980 2023-03-03T18:35:34.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14f7fe700 1 -- 172.21.15.7:0/1183792592 >> v2:172.21.15.143:3300/0 conn(0x7fd150151dc0 msgr2=0x7fd1500ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:35:34.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14f7fe700 1 --2- 172.21.15.7:0/1183792592 >> v2:172.21.15.143:3300/0 conn(0x7fd150151dc0 0x7fd1500ca630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:35:34.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14f7fe700 1 -- 172.21.15.7:0/1183792592 >> v1:172.21.15.7:6789/0 conn(0x7fd1500c6980 legacy=0x7fd150158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:35:34.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14f7fe700 1 -- 172.21.15.7:0/1183792592 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd150155160 con 0x7fd1500cab70 2023-03-03T18:35:34.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.340+0000 7fd14f7fe700 1 --2- 172.21.15.7:0/1183792592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1500cab70 0x7fd150158710 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 crypto rx=0x7fd1500c7080 tx=0x7fd14800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5601a87dba2e6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:35:34.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd14dffb700 1 -- 172.21.15.7:0/1183792592 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14800e070 con 0x7fd1500cab70 2023-03-03T18:35:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd14dffb700 1 -- 172.21.15.7:0/1183792592 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14800bb20 con 0x7fd1500cab70 2023-03-03T18:35:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd14dffb700 1 -- 172.21.15.7:0/1183792592 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148012430 con 0x7fd1500cab70 2023-03-03T18:35:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd155567700 1 -- 172.21.15.7:0/1183792592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1500cab70 msgr2=0x7fd150158710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:35:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd155567700 1 --2- 172.21.15.7:0/1183792592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1500cab70 0x7fd150158710 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 crypto rx=0x7fd1500c7080 tx=0x7fd14800b330 comp rx=0 tx=0).stop 2023-03-03T18:35:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd155567700 1 -- 172.21.15.7:0/1183792592 shutdown_connections 2023-03-03T18:35:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd155567700 1 --2- 172.21.15.7:0/1183792592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd1500cab70 0x7fd150158710 unknown :-1 s=CLOSED pgs=3236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:35:34.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd155567700 1 --2- 172.21.15.7:0/1183792592 >> v2:172.21.15.143:3300/0 conn(0x7fd150151dc0 0x7fd1500ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:35:34.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd155567700 1 -- 172.21.15.7:0/1183792592 >> 172.21.15.7:0/1183792592 conn(0x7fd15006b430 msgr2=0x7fd150071490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:35:34.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd155567700 1 -- 172.21.15.7:0/1183792592 shutdown_connections 2023-03-03T18:35:34.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.341+0000 7fd155567700 1 -- 172.21.15.7:0/1183792592 wait complete. 2023-03-03T18:35:34.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.342+0000 7fd155567700 1 Processor -- start 2023-03-03T18:35:34.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.342+0000 7fd155567700 1 -- start start 2023-03-03T18:35:34.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd155567700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1500c6980 0x7fd1501529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:35:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd155567700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1500cab70 0x7fd1500aa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:35:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd155567700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd150151dc0 0x7fd1500af240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:35:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd155567700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd150155720 con 0x7fd150151dc0 2023-03-03T18:35:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd155567700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1501ea780 con 0x7fd1500c6980 2023-03-03T18:35:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd155567700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1500ab5e0 con 0x7fd1500cab70 2023-03-03T18:35:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd14f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd150151dc0 0x7fd1500af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:35:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd14f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd150151dc0 0x7fd1500af240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37792/0 (socket says 172.21.15.7:37792) 2023-03-03T18:35:34.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd14f7fe700 1 -- 172.21.15.7:0/2340339741 learned_addr learned my addr 172.21.15.7:0/2340339741 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:35:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd14effd700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1500c6980 0x7fd1501529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:35:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.343+0000 7fd14e7fc700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1500cab70 0x7fd1500aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:35:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd14f7fe700 1 -- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1500cab70 msgr2=0x7fd1500aa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:35:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd14f7fe700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1500cab70 0x7fd1500aa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:35:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd14f7fe700 1 -- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1500c6980 msgr2=0x7fd1501529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:35:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd14f7fe700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1500c6980 0x7fd1501529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:35:34.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd14f7fe700 1 -- 172.21.15.7:0/2340339741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148009a70 con 0x7fd150151dc0 2023-03-03T18:35:34.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd14effd700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1500c6980 0x7fd1501529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:35:34.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd14f7fe700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd150151dc0 0x7fd1500af240 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7fd1500c8220 tx=0x7fd148004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:35:34.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd1377fe700 1 -- 172.21.15.7:0/2340339741 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148012430 con 0x7fd150151dc0 2023-03-03T18:35:34.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd1377fe700 1 -- 172.21.15.7:0/2340339741 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd148020070 con 0x7fd150151dc0 2023-03-03T18:35:34.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd1377fe700 1 -- 172.21.15.7:0/2340339741 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148004280 con 0x7fd150151dc0 2023-03-03T18:35:34.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd155567700 1 -- 172.21.15.7:0/2340339741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1501553b0 con 0x7fd150151dc0 2023-03-03T18:35:34.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.344+0000 7fd155567700 1 -- 172.21.15.7:0/2340339741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1500b0640 con 0x7fd150151dc0 2023-03-03T18:35:34.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.345+0000 7fd155567700 1 -- 172.21.15.7:0/2340339741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd150074bb0 con 0x7fd150151dc0 2023-03-03T18:35:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.346+0000 7fd1377fe700 1 -- 172.21.15.7:0/2340339741 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd14802e070 con 0x7fd150151dc0 2023-03-03T18:35:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.346+0000 7fd1377fe700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c068320 0x7fd13c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:35:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.346+0000 7fd1377fe700 1 -- 172.21.15.7:0/2340339741 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd148022070 con 0x7fd150151dc0 2023-03-03T18:35:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.346+0000 7fd14effd700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c068320 0x7fd13c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:35:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.347+0000 7fd14effd700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c068320 0x7fd13c06a7d0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fd144006fd0 tx=0x7fd144008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:35:34.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.350+0000 7fd1377fe700 1 -- 172.21.15.7:0/2340339741 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd1480541e0 con 0x7fd150151dc0 2023-03-03T18:35:34.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.588+0000 7fd155567700 1 -- 172.21.15.7:0/2340339741 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd15006bec0 con 0x7fd150151dc0 2023-03-03T18:35:34.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.589+0000 7fd1377fe700 1 -- 172.21.15.7:0/2340339741 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd14802c090 con 0x7fd150151dc0 2023-03-03T18:35:34.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:35:34.589 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:35:34.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 -- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c068320 msgr2=0x7fd13c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:35:34.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c068320 0x7fd13c06a7d0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fd144006fd0 tx=0x7fd144008040 comp rx=0 tx=0).stop 2023-03-03T18:35:34.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 -- 172.21.15.7:0/2340339741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd150151dc0 msgr2=0x7fd1500af240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:35:34.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd150151dc0 0x7fd1500af240 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7fd1500c8220 tx=0x7fd148004ad0 comp rx=0 tx=0).stop 2023-03-03T18:35:34.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 -- 172.21.15.7:0/2340339741 shutdown_connections 2023-03-03T18:35:34.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd13c068320 0x7fd13c06a7d0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:35:34.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd150151dc0 0x7fd1500af240 unknown :-1 s=CLOSED pgs=3237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:35:34.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1500c6980 0x7fd1501529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:35:34.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 --2- 172.21.15.7:0/2340339741 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd1500cab70 0x7fd1500aa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:35:34.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 -- 172.21.15.7:0/2340339741 >> 172.21.15.7:0/2340339741 conn(0x7fd15006b430 msgr2=0x7fd150151000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:35:34.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 -- 172.21.15.7:0/2340339741 shutdown_connections 2023-03-03T18:35:34.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:35:34.591+0000 7fd155567700 1 -- 172.21.15.7:0/2340339741 wait complete. 2023-03-03T18:35:34.593 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:35:34.602 INFO:tasks.mds_thrash.fs.[a]:waiting for 31 secs before thrashing 2023-03-03T18:35:39.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:35:39.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:36:02.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:36:02.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.448+0000 7f68c117c700 1 Processor -- start 2023-03-03T18:36:02.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.448+0000 7f68c117c700 1 -- start start 2023-03-03T18:36:02.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68c117c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f68bc0c1da0 0x7f68bc139d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:02.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68c117c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68bc0c7b90 con 0x7f68bc0c8020 2023-03-03T18:36:02.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68c117c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68bc154640 con 0x7f68bc06d4b0 2023-03-03T18:36:02.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68c117c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68bc0c7330 con 0x7f68bc0c1da0 2023-03-03T18:36:02.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68ba59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f68bc0c8020 0x7f68bc0c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60028/0 (socket says 172.21.15.7:60028) 2023-03-03T18:36:02.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68ba59c700 1 -- 172.21.15.7:0/3730330405 learned_addr learned my addr 172.21.15.7:0/3730330405 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:36:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168343818 0 0) 0x7f68bc0c7b90 con 0x7f68bc0c8020 2023-03-03T18:36:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a40035b0 con 0x7f68bc0c8020 2023-03-03T18:36:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68bb59e700 1 --2- 172.21.15.7:0/3730330405 >> v2:172.21.15.143:3300/0 conn(0x7f68bc0c1da0 0x7f68bc139d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:02.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3025306830 0 0) 0x7f68a40035b0 con 0x7f68bc0c8020 2023-03-03T18:36:02.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 >> v2:172.21.15.143:3300/0 conn(0x7f68bc0c1da0 msgr2=0x7f68bc139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:36:02.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 --2- 172.21.15.7:0/3730330405 >> v2:172.21.15.143:3300/0 conn(0x7f68bc0c1da0 0x7f68bc139d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:02.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 >> v1:172.21.15.143:6789/0 conn(0x7f68bc06d4b0 legacy=0x7f68bc06d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:02.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68bc0c7540 con 0x7f68bc0c8020 2023-03-03T18:36:02.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f68ac001670 con 0x7f68bc0c8020 2023-03-03T18:36:02.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 >> v1:172.21.15.7:6789/0 conn(0x7f68bc0c8020 legacy=0x7f68bc0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:02.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4003a70 0x7f68a4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:02.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.449+0000 7f68b9d9b700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4004850 0x7f68a4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:02.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68b9d9b700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68a4005250 0x7f68a400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:02.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0c7330 con 0x7f68a4003a70 2023-03-03T18:36:02.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a400c190 con 0x7f68a4004850 2023-03-03T18:36:02.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68a400c350 con 0x7f68a4005250 2023-03-03T18:36:02.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68bad9d700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4003a70 0x7f68a4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:02.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68ba59c700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4004850 0x7f68a4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:02.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68bb59e700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68a4005250 0x7f68a400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:02.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68bad9d700 1 -- 172.21.15.7:0/3730330405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68a4005250 msgr2=0x7f68a400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:36:02.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68bad9d700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68a4005250 0x7f68a400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:02.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68bad9d700 1 -- 172.21.15.7:0/3730330405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4004850 msgr2=0x7f68a4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:36:02.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68bad9d700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4004850 0x7f68a4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:02.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68bad9d700 1 -- 172.21.15.7:0/3730330405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a40035b0 con 0x7f68a4003a70 2023-03-03T18:36:02.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68bb59e700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68a4005250 0x7f68a400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:36:02.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68bb59e700 1 -- 172.21.15.7:0/3730330405 reap_dead start 2023-03-03T18:36:02.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68bad9d700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4003a70 0x7f68a4003e80 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 crypto rx=0x7f68b0007fd0 tx=0x7f68a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22b0cf6e8169649 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:36:02.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b00012f0 con 0x7f68a4003a70 2023-03-03T18:36:02.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b0005310 con 0x7f68a4003a70 2023-03-03T18:36:02.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.450+0000 7f68b9d9b700 1 -- 172.21.15.7:0/3730330405 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0006860 con 0x7f68a4003a70 2023-03-03T18:36:02.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.451+0000 7f68c117c700 1 -- 172.21.15.7:0/3730330405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4003a70 msgr2=0x7f68a4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:02.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.451+0000 7f68c117c700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4003a70 0x7f68a4003e80 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 crypto rx=0x7f68b0007fd0 tx=0x7f68a4005880 comp rx=0 tx=0).stop 2023-03-03T18:36:02.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.451+0000 7f68c117c700 1 -- 172.21.15.7:0/3730330405 shutdown_connections 2023-03-03T18:36:02.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.451+0000 7f68c117c700 1 --2- 172.21.15.7:0/3730330405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68a4003a70 0x7f68a4003e80 unknown :-1 s=CLOSED pgs=3239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:02.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.451+0000 7f68c117c700 1 -- 172.21.15.7:0/3730330405 >> 172.21.15.7:0/3730330405 conn(0x7f68bc06c110 msgr2=0x7f68bc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:36:02.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.451+0000 7f68c117c700 1 -- 172.21.15.7:0/3730330405 shutdown_connections 2023-03-03T18:36:02.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.451+0000 7f68c117c700 1 -- 172.21.15.7:0/3730330405 wait complete. 2023-03-03T18:36:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.452+0000 7f68c117c700 1 Processor -- start 2023-03-03T18:36:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68c117c700 1 -- start start 2023-03-03T18:36:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68c117c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4003a70 0x7f68bc1ed050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68c117c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc139b50 0x7f68bc13ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:02.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68c117c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68bc13d820 0x7f68bc13b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:02.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68c117c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc0acc80 con 0x7f68bc139b50 2023-03-03T18:36:02.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68c117c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc13c730 con 0x7f68a4003a70 2023-03-03T18:36:02.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68c117c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc13a670 con 0x7f68bc13d820 2023-03-03T18:36:02.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68ba59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc139b50 0x7f68bc13ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:02.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68ba59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc139b50 0x7f68bc13ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37828/0 (socket says 172.21.15.7:37828) 2023-03-03T18:36:02.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68ba59c700 1 -- 172.21.15.7:0/638098613 learned_addr learned my addr 172.21.15.7:0/638098613 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:36:02.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68bad9d700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4003a70 0x7f68bc1ed050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:02.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.453+0000 7f68bb59e700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68bc13d820 0x7f68bc13b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:02.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68ba59c700 1 -- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68bc13d820 msgr2=0x7f68bc13b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:02.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68ba59c700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68bc13d820 0x7f68bc13b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:02.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68ba59c700 1 -- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4003a70 msgr2=0x7f68bc1ed050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:02.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68ba59c700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4003a70 0x7f68bc1ed050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:02.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68ba59c700 1 -- 172.21.15.7:0/638098613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68bc06e210 con 0x7f68bc139b50 2023-03-03T18:36:02.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68ba59c700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc139b50 0x7f68bc13ab00 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0072f0 tx=0x7f68ac007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:36:02.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68a37fe700 1 -- 172.21.15.7:0/638098613 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac011580 con 0x7f68bc139b50 2023-03-03T18:36:02.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68a37fe700 1 -- 172.21.15.7:0/638098613 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac011700 con 0x7f68bc139b50 2023-03-03T18:36:02.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68c117c700 1 -- 172.21.15.7:0/638098613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68bc0c7820 con 0x7f68bc139b50 2023-03-03T18:36:02.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68a37fe700 1 -- 172.21.15.7:0/638098613 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac011ab0 con 0x7f68bc139b50 2023-03-03T18:36:02.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.454+0000 7f68c117c700 1 -- 172.21.15.7:0/638098613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68bc06d880 con 0x7f68bc139b50 2023-03-03T18:36:02.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.455+0000 7f68c117c700 1 -- 172.21.15.7:0/638098613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68bc074bb0 con 0x7f68bc139b50 2023-03-03T18:36:02.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.455+0000 7f68a37fe700 1 -- 172.21.15.7:0/638098613 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f68ac01b030 con 0x7f68bc139b50 2023-03-03T18:36:02.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.456+0000 7f68a37fe700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68a8068320 0x7f68a806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:02.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.456+0000 7f68a37fe700 1 -- 172.21.15.7:0/638098613 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f68ac085a20 con 0x7f68bc139b50 2023-03-03T18:36:02.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.456+0000 7f68bad9d700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68a8068320 0x7f68a806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:02.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.457+0000 7f68bad9d700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68a8068320 0x7f68a806a7d0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f68b0007fd0 tx=0x7f68b0011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:36:02.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.460+0000 7f68a37fe700 1 -- 172.21.15.7:0/638098613 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68ac051dc0 con 0x7f68bc139b50 2023-03-03T18:36:02.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.702+0000 7f68c117c700 1 -- 172.21.15.7:0/638098613 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68bc0c14e0 con 0x7f68bc139b50 2023-03-03T18:36:02.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.702+0000 7f68a37fe700 1 -- 172.21.15.7:0/638098613 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f68ac019090 con 0x7f68bc139b50 2023-03-03T18:36:02.703 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:36:02.703 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:36:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.704+0000 7f68c117c700 1 -- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68a8068320 msgr2=0x7f68a806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.704+0000 7f68c117c700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68a8068320 0x7f68a806a7d0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f68b0007fd0 tx=0x7f68b0011000 comp rx=0 tx=0).stop 2023-03-03T18:36:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.704+0000 7f68c117c700 1 -- 172.21.15.7:0/638098613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc139b50 msgr2=0x7f68bc13ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.704+0000 7f68c117c700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc139b50 0x7f68bc13ab00 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0072f0 tx=0x7f68ac007600 comp rx=0 tx=0).stop 2023-03-03T18:36:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.704+0000 7f68c117c700 1 -- 172.21.15.7:0/638098613 shutdown_connections 2023-03-03T18:36:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.704+0000 7f68c117c700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68bc13d820 0x7f68bc13b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.704+0000 7f68c117c700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68a4003a70 0x7f68bc1ed050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.704+0000 7f68c117c700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68a8068320 0x7f68a806a7d0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.704+0000 7f68c117c700 1 --2- 172.21.15.7:0/638098613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68bc139b50 0x7f68bc13ab00 unknown :-1 s=CLOSED pgs=3240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.704+0000 7f68c117c700 1 -- 172.21.15.7:0/638098613 >> 172.21.15.7:0/638098613 conn(0x7f68bc06c110 msgr2=0x7f68bc139070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:36:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.705+0000 7f68c117c700 1 -- 172.21.15.7:0/638098613 shutdown_connections 2023-03-03T18:36:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:02.705+0000 7f68c117c700 1 -- 172.21.15.7:0/638098613 wait complete. 2023-03-03T18:36:02.706 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:36:02.716 INFO:tasks.mds_thrash.fs.[b]:waiting for 74 secs before thrashing 2023-03-03T18:36:05.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:36:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.743+0000 7ff2f50be700 1 Processor -- start 2023-03-03T18:36:05.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.743+0000 7ff2f50be700 1 -- start start 2023-03-03T18:36:05.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.743+0000 7ff2f50be700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff2f00aa030 0x7ff2f0158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:05.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.743+0000 7ff2f50be700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f00a9c20 con 0x7ff2f00aad20 2023-03-03T18:36:05.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.743+0000 7ff2f50be700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f0154f10 con 0x7ff2f014b0d0 2023-03-03T18:36:05.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.743+0000 7ff2f50be700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f00ac3c0 con 0x7ff2f00aa030 2023-03-03T18:36:05.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.744+0000 7ff2ee59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff2f00aad20 0x7ff2f0157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60048/0 (socket says 172.21.15.7:60048) 2023-03-03T18:36:05.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.744+0000 7ff2ee59c700 1 -- 172.21.15.7:0/4266582312 learned_addr learned my addr 172.21.15.7:0/4266582312 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:36:05.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.744+0000 7ff2ef59e700 1 --2- 172.21.15.7:0/4266582312 >> v2:172.21.15.143:3300/0 conn(0x7ff2f00aa030 0x7ff2f0158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:05.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.744+0000 7ff2edd9b700 1 -- 172.21.15.7:0/4266582312 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788483816 0 0) 0x7ff2f00a9c20 con 0x7ff2f00aad20 2023-03-03T18:36:05.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.744+0000 7ff2edd9b700 1 -- 172.21.15.7:0/4266582312 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d80035b0 con 0x7ff2f00aad20 2023-03-03T18:36:05.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.744+0000 7ff2edd9b700 1 -- 172.21.15.7:0/4266582312 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3298502578 0 0) 0x7ff2d80035b0 con 0x7ff2f00aad20 2023-03-03T18:36:05.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.744+0000 7ff2ef59e700 1 -- 172.21.15.7:0/4266582312 >> v1:172.21.15.143:6789/0 conn(0x7ff2f014b0d0 legacy=0x7ff2f0071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:05.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.744+0000 7ff2ef59e700 1 -- 172.21.15.7:0/4266582312 >> v1:172.21.15.7:6789/0 conn(0x7ff2f00aad20 legacy=0x7ff2f0157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:05.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.744+0000 7ff2ef59e700 1 -- 172.21.15.7:0/4266582312 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f006fc70 con 0x7ff2f00aa030 2023-03-03T18:36:05.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.744+0000 7ff2ef59e700 1 --2- 172.21.15.7:0/4266582312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f00aa030 0x7ff2f0158500 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8009d90 tx=0x7ff2e800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cc2d25b01f6c50d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:36:05.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.745+0000 7ff2edd9b700 1 -- 172.21.15.7:0/4266582312 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e800e070 con 0x7ff2f00aa030 2023-03-03T18:36:05.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.745+0000 7ff2edd9b700 1 -- 172.21.15.7:0/4266582312 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e8003a20 con 0x7ff2f00aa030 2023-03-03T18:36:05.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.745+0000 7ff2edd9b700 1 -- 172.21.15.7:0/4266582312 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4001670 con 0x7ff2f00aa030 2023-03-03T18:36:05.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.745+0000 7ff2f50be700 1 -- 172.21.15.7:0/4266582312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f00aa030 msgr2=0x7ff2f0158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:05.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.745+0000 7ff2f50be700 1 --2- 172.21.15.7:0/4266582312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f00aa030 0x7ff2f0158500 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8009d90 tx=0x7ff2e800b330 comp rx=0 tx=0).stop 2023-03-03T18:36:05.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.745+0000 7ff2f50be700 1 -- 172.21.15.7:0/4266582312 shutdown_connections 2023-03-03T18:36:05.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.745+0000 7ff2f50be700 1 --2- 172.21.15.7:0/4266582312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f00aa030 0x7ff2f0158500 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:05.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.745+0000 7ff2f50be700 1 -- 172.21.15.7:0/4266582312 >> 172.21.15.7:0/4266582312 conn(0x7ff2f00c8900 msgr2=0x7ff2f00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:36:05.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.745+0000 7ff2f50be700 1 -- 172.21.15.7:0/4266582312 shutdown_connections 2023-03-03T18:36:05.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.745+0000 7ff2f50be700 1 -- 172.21.15.7:0/4266582312 wait complete. 2023-03-03T18:36:05.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2f50be700 1 Processor -- start 2023-03-03T18:36:05.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2f50be700 1 -- start start 2023-03-03T18:36:05.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2f50be700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2f00aa030 0x7ff2f0153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:05.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2f50be700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2f00aad20 0x7ff2f0157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:05.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2f50be700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f014b0d0 0x7ff2f00c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:05.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2f50be700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f00c6440 con 0x7ff2f00aad20 2023-03-03T18:36:05.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2f50be700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f0146850 con 0x7ff2f014b0d0 2023-03-03T18:36:05.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2f50be700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f0146570 con 0x7ff2f00aa030 2023-03-03T18:36:05.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2ee59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2f00aad20 0x7ff2f0157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:05.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2ee59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2f00aad20 0x7ff2f0157ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37842/0 (socket says 172.21.15.7:37842) 2023-03-03T18:36:05.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.747+0000 7ff2ee59c700 1 -- 172.21.15.7:0/2713504108 learned_addr learned my addr 172.21.15.7:0/2713504108 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:36:05.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2eed9d700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2f00aa030 0x7ff2f0153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:05.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2ef59e700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f014b0d0 0x7ff2f00c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:05.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2ee59c700 1 -- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2f00aa030 msgr2=0x7ff2f0153690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:05.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2ee59c700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2f00aa030 0x7ff2f0153690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:05.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2ee59c700 1 -- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f014b0d0 msgr2=0x7ff2f00c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:05.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2ee59c700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f014b0d0 0x7ff2f00c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:05.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2ee59c700 1 -- 172.21.15.7:0/2713504108 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e8009a70 con 0x7ff2f00aad20 2023-03-03T18:36:05.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2ee59c700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2f00aad20 0x7ff2f0157ed0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400c730 tx=0x7ff2e400ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:36:05.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2d77fe700 1 -- 172.21.15.7:0/2713504108 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4004c80 con 0x7ff2f00aad20 2023-03-03T18:36:05.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2d77fe700 1 -- 172.21.15.7:0/2713504108 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e4004de0 con 0x7ff2f00aad20 2023-03-03T18:36:05.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2d77fe700 1 -- 172.21.15.7:0/2713504108 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4007310 con 0x7ff2f00aad20 2023-03-03T18:36:05.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2f50be700 1 -- 172.21.15.7:0/2713504108 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f006f6b0 con 0x7ff2f00aad20 2023-03-03T18:36:05.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.748+0000 7ff2f50be700 1 -- 172.21.15.7:0/2713504108 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f0155160 con 0x7ff2f00aad20 2023-03-03T18:36:05.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.749+0000 7ff2f50be700 1 -- 172.21.15.7:0/2713504108 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f006f990 con 0x7ff2f00aad20 2023-03-03T18:36:05.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.750+0000 7ff2d77fe700 1 -- 172.21.15.7:0/2713504108 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff2e401a020 con 0x7ff2f00aad20 2023-03-03T18:36:05.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.750+0000 7ff2d77fe700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2dc068320 0x7ff2dc06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:36:05.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.750+0000 7ff2d77fe700 1 -- 172.21.15.7:0/2713504108 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff2e4085d50 con 0x7ff2f00aad20 2023-03-03T18:36:05.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.750+0000 7ff2eed9d700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2dc068320 0x7ff2dc06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:36:05.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.751+0000 7ff2eed9d700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2dc068320 0x7ff2dc06a7d0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0007fd0 tx=0x7ff2e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:36:05.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.754+0000 7ff2d77fe700 1 -- 172.21.15.7:0/2713504108 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff2e40520f0 con 0x7ff2f00aad20 2023-03-03T18:36:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.993+0000 7ff2f50be700 1 -- 172.21.15.7:0/2713504108 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2f004e6e0 con 0x7ff2f00aad20 2023-03-03T18:36:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.993+0000 7ff2d77fe700 1 -- 172.21.15.7:0/2713504108 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff2e4055710 con 0x7ff2f00aad20 2023-03-03T18:36:05.994 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:36:05.994 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:36:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.995+0000 7ff2f50be700 1 -- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2dc068320 msgr2=0x7ff2dc06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.995+0000 7ff2f50be700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2dc068320 0x7ff2dc06a7d0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0007fd0 tx=0x7ff2e0009040 comp rx=0 tx=0).stop 2023-03-03T18:36:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.995+0000 7ff2f50be700 1 -- 172.21.15.7:0/2713504108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2f00aad20 msgr2=0x7ff2f0157ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:36:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.995+0000 7ff2f50be700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2f00aad20 0x7ff2f0157ed0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400c730 tx=0x7ff2e400ca40 comp rx=0 tx=0).stop 2023-03-03T18:36:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.996+0000 7ff2f50be700 1 -- 172.21.15.7:0/2713504108 shutdown_connections 2023-03-03T18:36:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.996+0000 7ff2f50be700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2f014b0d0 0x7ff2f00c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.996+0000 7ff2f50be700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2f00aa030 0x7ff2f0153690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:05.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.996+0000 7ff2f50be700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2dc068320 0x7ff2dc06a7d0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.996+0000 7ff2f50be700 1 --2- 172.21.15.7:0/2713504108 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2f00aad20 0x7ff2f0157ed0 unknown :-1 s=CLOSED pgs=3242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:36:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.996+0000 7ff2f50be700 1 -- 172.21.15.7:0/2713504108 >> 172.21.15.7:0/2713504108 conn(0x7ff2f00c8900 msgr2=0x7ff2f00c4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:36:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.996+0000 7ff2f50be700 1 -- 172.21.15.7:0/2713504108 shutdown_connections 2023-03-03T18:36:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:36:05.996+0000 7ff2f50be700 1 -- 172.21.15.7:0/2713504108 wait complete. 2023-03-03T18:36:05.998 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:36:06.008 INFO:tasks.mds_thrash.fs.[a]:waiting for 93 secs before thrashing 2023-03-03T18:36:09.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:36:09.935 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:36:39.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:36:40.001 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:37:10.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:37:10.071 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:37:16.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:37:16.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.857+0000 7f61c734b700 1 Processor -- start 2023-03-03T18:37:16.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.857+0000 7f61c734b700 1 -- start start 2023-03-03T18:37:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61c734b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61c006c480 0x7f61c01581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61c734b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c006d430 con 0x7f61c00c2930 2023-03-03T18:37:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61c734b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c01586e0 con 0x7f61c0151d40 2023-03-03T18:37:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61c734b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c006c2a0 con 0x7f61c006c480 2023-03-03T18:37:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61c48e6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f61c00c2930 0x7f61c0157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60120/0 (socket says 172.21.15.7:60120) 2023-03-03T18:37:16.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61c48e6700 1 -- 172.21.15.7:0/3516932337 learned_addr learned my addr 172.21.15.7:0/3516932337 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:37:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61c58e8700 1 --2- 172.21.15.7:0/3516932337 >> v2:172.21.15.143:3300/0 conn(0x7f61c006c480 0x7f61c01581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876401583 0 0) 0x7f61c006d430 con 0x7f61c00c2930 2023-03-03T18:37:16.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ac0035b0 con 0x7f61c00c2930 2023-03-03T18:37:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1225968457 0 0) 0x7f61ac0035b0 con 0x7f61c00c2930 2023-03-03T18:37:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 >> v2:172.21.15.143:3300/0 conn(0x7f61c006c480 msgr2=0x7f61c01581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:37:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61b7fff700 1 --2- 172.21.15.7:0/3516932337 >> v2:172.21.15.143:3300/0 conn(0x7f61c006c480 0x7f61c01581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:16.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 >> v1:172.21.15.143:6789/0 conn(0x7f61c0151d40 legacy=0x7f61c00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c006cb00 con 0x7f61c00c2930 2023-03-03T18:37:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.858+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f61b0001670 con 0x7f61c00c2930 2023-03-03T18:37:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 >> v1:172.21.15.7:6789/0 conn(0x7f61c00c2930 legacy=0x7f61c0157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61b7fff700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61ac003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61b7fff700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ac004850 0x7f61ac004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61b7fff700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ac005400 0x7f61ac00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:16.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c006c2a0 con 0x7f61ac003af0 2023-03-03T18:37:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac00c2a0 con 0x7f61ac005400 2023-03-03T18:37:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac00c460 con 0x7f61ac004850 2023-03-03T18:37:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61c50e7700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61ac003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61c48e6700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ac004850 0x7f61ac004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61c58e8700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ac005400 0x7f61ac00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61c50e7700 1 -- 172.21.15.7:0/3516932337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ac004850 msgr2=0x7f61ac004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:37:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61c50e7700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61ac004850 0x7f61ac004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:16.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61c50e7700 1 -- 172.21.15.7:0/3516932337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ac005400 msgr2=0x7f61ac00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:37:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61c50e7700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61ac005400 0x7f61ac00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61c58e8700 1 -- 172.21.15.7:0/3516932337 reap_dead start 2023-03-03T18:37:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61c50e7700 1 -- 172.21.15.7:0/3516932337 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac0035b0 con 0x7f61ac003af0 2023-03-03T18:37:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61c50e7700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61ac003f00 secure :-1 s=READY pgs=3244 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0017c0 tx=0x7f61ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26079320de7ad333 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:37:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0052d0 con 0x7f61ac003af0 2023-03-03T18:37:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc005450 con 0x7f61ac003af0 2023-03-03T18:37:16.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.859+0000 7f61b7fff700 1 -- 172.21.15.7:0/3516932337 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0038c0 con 0x7f61ac003af0 2023-03-03T18:37:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.860+0000 7f61c734b700 1 -- 172.21.15.7:0/3516932337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 msgr2=0x7f61ac003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.860+0000 7f61c734b700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61ac003f00 secure :-1 s=READY pgs=3244 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0017c0 tx=0x7f61ac005880 comp rx=0 tx=0).stop 2023-03-03T18:37:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.860+0000 7f61c734b700 1 -- 172.21.15.7:0/3516932337 shutdown_connections 2023-03-03T18:37:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.860+0000 7f61c734b700 1 --2- 172.21.15.7:0/3516932337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61ac003f00 unknown :-1 s=CLOSED pgs=3244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.860+0000 7f61c734b700 1 -- 172.21.15.7:0/3516932337 >> 172.21.15.7:0/3516932337 conn(0x7f61c00a94d0 msgr2=0x7f61c00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:37:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.860+0000 7f61c734b700 1 -- 172.21.15.7:0/3516932337 shutdown_connections 2023-03-03T18:37:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.860+0000 7f61c734b700 1 -- 172.21.15.7:0/3516932337 wait complete. 2023-03-03T18:37:16.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.861+0000 7f61c734b700 1 Processor -- start 2023-03-03T18:37:16.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.861+0000 7f61c734b700 1 -- start start 2023-03-03T18:37:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c734b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61c00c1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c734b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61c00c5b70 0x7f61c00c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c734b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61c00c5fe0 0x7f61c00c0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c734b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c014bb10 con 0x7f61ac003af0 2023-03-03T18:37:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c734b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c014b620 con 0x7f61c00c5fe0 2023-03-03T18:37:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c734b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c00bfa00 con 0x7f61c00c5b70 2023-03-03T18:37:16.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c50e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61c00c1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:16.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c50e7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61c00c1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37920/0 (socket says 172.21.15.7:37920) 2023-03-03T18:37:16.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c50e7700 1 -- 172.21.15.7:0/2942482557 learned_addr learned my addr 172.21.15.7:0/2942482557 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:37:16.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c58e8700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61c00c5fe0 0x7f61c00c0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:16.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c48e6700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61c00c5b70 0x7f61c00c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:16.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c50e7700 1 -- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61c00c5b70 msgr2=0x7f61c00c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:16.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c50e7700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61c00c5b70 0x7f61c00c00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:16.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c50e7700 1 -- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61c00c5fe0 msgr2=0x7f61c00c0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c50e7700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61c00c5fe0 0x7f61c00c0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c50e7700 1 -- 172.21.15.7:0/2942482557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c006d0c0 con 0x7f61ac003af0 2023-03-03T18:37:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c48e6700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61c00c5b70 0x7f61c00c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:37:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.862+0000 7f61c58e8700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61c00c5fe0 0x7f61c00c0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:37:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.863+0000 7f61c50e7700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61c00c1100 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 crypto rx=0x7f61bc005040 tx=0x7f61bc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:37:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.863+0000 7f61b5ffb700 1 -- 172.21.15.7:0/2942482557 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc001290 con 0x7f61ac003af0 2023-03-03T18:37:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.863+0000 7f61b5ffb700 1 -- 172.21.15.7:0/2942482557 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc007040 con 0x7f61ac003af0 2023-03-03T18:37:16.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.863+0000 7f61c734b700 1 -- 172.21.15.7:0/2942482557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c006cde0 con 0x7f61ac003af0 2023-03-03T18:37:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.863+0000 7f61b5ffb700 1 -- 172.21.15.7:0/2942482557 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0073f0 con 0x7f61ac003af0 2023-03-03T18:37:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.863+0000 7f61c734b700 1 -- 172.21.15.7:0/2942482557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c006bea0 con 0x7f61ac003af0 2023-03-03T18:37:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.864+0000 7f61c734b700 1 -- 172.21.15.7:0/2942482557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c0074bb0 con 0x7f61ac003af0 2023-03-03T18:37:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.864+0000 7f61b5ffb700 1 -- 172.21.15.7:0/2942482557 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f61bc02d070 con 0x7f61ac003af0 2023-03-03T18:37:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.865+0000 7f61b5ffb700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61a40682f0 0x7f61a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.865+0000 7f61b5ffb700 1 -- 172.21.15.7:0/2942482557 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f61bc086a10 con 0x7f61ac003af0 2023-03-03T18:37:16.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.865+0000 7f61c48e6700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61a40682f0 0x7f61a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:16.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.866+0000 7f61c48e6700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61a40682f0 0x7f61a406a7a0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f61c0155630 tx=0x7f61b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:37:16.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:16.869+0000 7f61b5ffb700 1 -- 172.21.15.7:0/2942482557 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f61bc052db0 con 0x7f61ac003af0 2023-03-03T18:37:17.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.109+0000 7f61c734b700 1 -- 172.21.15.7:0/2942482557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61c006df40 con 0x7f61ac003af0 2023-03-03T18:37:17.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.110+0000 7f61b5ffb700 1 -- 172.21.15.7:0/2942482557 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f61bc021090 con 0x7f61ac003af0 2023-03-03T18:37:17.110 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:37:17.110 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:37:17.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 -- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61a40682f0 msgr2=0x7f61a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:17.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61a40682f0 0x7f61a406a7a0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f61c0155630 tx=0x7f61b0009040 comp rx=0 tx=0).stop 2023-03-03T18:37:17.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 -- 172.21.15.7:0/2942482557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 msgr2=0x7f61c00c1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:17.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61c00c1100 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 crypto rx=0x7f61bc005040 tx=0x7f61bc001120 comp rx=0 tx=0).stop 2023-03-03T18:37:17.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 -- 172.21.15.7:0/2942482557 shutdown_connections 2023-03-03T18:37:17.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61c00c5fe0 0x7f61c00c0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:17.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f61a40682f0 0x7f61a406a7a0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:17.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61ac003af0 0x7f61c00c1100 unknown :-1 s=CLOSED pgs=3245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:17.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 --2- 172.21.15.7:0/2942482557 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61c00c5b70 0x7f61c00c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:17.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 -- 172.21.15.7:0/2942482557 >> 172.21.15.7:0/2942482557 conn(0x7f61c00a94d0 msgr2=0x7f61c0157f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:37:17.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 -- 172.21.15.7:0/2942482557 shutdown_connections 2023-03-03T18:37:17.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:17.112+0000 7f61c734b700 1 -- 172.21.15.7:0/2942482557 wait complete. 2023-03-03T18:37:17.114 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:37:17.124 INFO:tasks.mds_thrash.fs.[b]:waiting for 117 secs before thrashing 2023-03-03T18:37:39.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:37:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.148+0000 7f0307652700 1 Processor -- start 2023-03-03T18:37:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.148+0000 7f0307652700 1 -- start start 2023-03-03T18:37:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f0307652700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0300151e80 0x7f03000c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f0307652700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0300155410 0x7f03000bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f0307652700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000c08d0 con 0x7f03000c6fe0 2023-03-03T18:37:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f0307652700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0300152250 con 0x7f0300151e80 2023-03-03T18:37:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f0307652700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03000c0fb0 con 0x7f0300155410 2023-03-03T18:37:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f03053ee700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0300151e80 0x7f03000c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f03053ee700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0300151e80 0x7f03000c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37948/0 (socket says 172.21.15.7:37948) 2023-03-03T18:37:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f03053ee700 1 -- 172.21.15.7:0/2996522065 learned_addr learned my addr 172.21.15.7:0/2996522065 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:37:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f0304bed700 1 --2- 172.21.15.7:0/2996522065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0300155410 0x7f03000bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f03053ee700 1 -- 172.21.15.7:0/2996522065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0300155410 msgr2=0x7f03000bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:37:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f03053ee700 1 --2- 172.21.15.7:0/2996522065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0300155410 0x7f03000bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f03053ee700 1 -- 172.21.15.7:0/2996522065 >> v1:172.21.15.143:6789/0 conn(0x7f03000c6fe0 legacy=0x7f03000bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f03053ee700 1 -- 172.21.15.7:0/2996522065 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0300154930 con 0x7f0300151e80 2023-03-03T18:37:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f0304bed700 1 --2- 172.21.15.7:0/2996522065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0300155410 0x7f03000bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:37:39.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.149+0000 7f03053ee700 1 --2- 172.21.15.7:0/2996522065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300151e80 0x7f03000c6aa0 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009920 tx=0x7f02fc009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dee9d6c21891bc1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:37:39.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f02f7fff700 1 -- 172.21.15.7:0/2996522065 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc00ba20 con 0x7f0300151e80 2023-03-03T18:37:39.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f02f7fff700 1 -- 172.21.15.7:0/2996522065 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc00bb80 con 0x7f0300151e80 2023-03-03T18:37:39.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f02f7fff700 1 -- 172.21.15.7:0/2996522065 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc009e70 con 0x7f0300151e80 2023-03-03T18:37:39.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f0307652700 1 -- 172.21.15.7:0/2996522065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300151e80 msgr2=0x7f03000c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:39.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f0307652700 1 --2- 172.21.15.7:0/2996522065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300151e80 0x7f03000c6aa0 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009920 tx=0x7f02fc009c30 comp rx=0 tx=0).stop 2023-03-03T18:37:39.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f0307652700 1 -- 172.21.15.7:0/2996522065 shutdown_connections 2023-03-03T18:37:39.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f0307652700 1 --2- 172.21.15.7:0/2996522065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300151e80 0x7f03000c6aa0 unknown :-1 s=CLOSED pgs=3246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:39.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f0307652700 1 --2- 172.21.15.7:0/2996522065 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0300155410 0x7f03000bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:39.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f0307652700 1 -- 172.21.15.7:0/2996522065 >> 172.21.15.7:0/2996522065 conn(0x7f0300071630 msgr2=0x7f030006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:37:39.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f0307652700 1 -- 172.21.15.7:0/2996522065 shutdown_connections 2023-03-03T18:37:39.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.150+0000 7f0307652700 1 -- 172.21.15.7:0/2996522065 wait complete. 2023-03-03T18:37:39.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0307652700 1 Processor -- start 2023-03-03T18:37:39.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0307652700 1 -- start start 2023-03-03T18:37:39.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0307652700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03000c6fe0 0x7f0300145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:39.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0307652700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0300151e80 0x7f03001ddd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:39.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0307652700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300155410 0x7f0300146cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:39.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0307652700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001e0c70 con 0x7f0300155410 2023-03-03T18:37:39.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0307652700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03001df6c0 con 0x7f0300151e80 2023-03-03T18:37:39.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0307652700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03001e06b0 con 0x7f03000c6fe0 2023-03-03T18:37:39.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0305bef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300155410 0x7f0300146cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:39.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0305bef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300155410 0x7f0300146cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37958/0 (socket says 172.21.15.7:37958) 2023-03-03T18:37:39.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.152+0000 7f0305bef700 1 -- 172.21.15.7:0/117620836 learned_addr learned my addr 172.21.15.7:0/117620836 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:37:39.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f03053ee700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03000c6fe0 0x7f0300145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:39.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f0304bed700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0300151e80 0x7f03001ddd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:39.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f0305bef700 1 -- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03000c6fe0 msgr2=0x7f0300145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:39.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f0305bef700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03000c6fe0 0x7f0300145f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:39.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f0305bef700 1 -- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0300151e80 msgr2=0x7f03001ddd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:39.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f0305bef700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0300151e80 0x7f03001ddd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:39.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f0305bef700 1 -- 172.21.15.7:0/117620836 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc009550 con 0x7f0300155410 2023-03-03T18:37:39.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f03053ee700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03000c6fe0 0x7f0300145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:37:39.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f0304bed700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0300151e80 0x7f03001ddd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:37:39.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f0305bef700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300155410 0x7f0300146cd0 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f02f800cb80 tx=0x7f02f800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:37:39.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f02f5ffb700 1 -- 172.21.15.7:0/117620836 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8003c00 con 0x7f0300155410 2023-03-03T18:37:39.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f02f5ffb700 1 -- 172.21.15.7:0/117620836 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f8003d60 con 0x7f0300155410 2023-03-03T18:37:39.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f0307652700 1 -- 172.21.15.7:0/117620836 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0300154ef0 con 0x7f0300155410 2023-03-03T18:37:39.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f02f5ffb700 1 -- 172.21.15.7:0/117620836 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f800abb0 con 0x7f0300155410 2023-03-03T18:37:39.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.153+0000 7f0307652700 1 -- 172.21.15.7:0/117620836 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0300154c10 con 0x7f0300155410 2023-03-03T18:37:39.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.154+0000 7f0307652700 1 -- 172.21.15.7:0/117620836 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03000c8df0 con 0x7f0300155410 2023-03-03T18:37:39.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.155+0000 7f02f5ffb700 1 -- 172.21.15.7:0/117620836 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f02f8010050 con 0x7f0300155410 2023-03-03T18:37:39.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.155+0000 7f02f5ffb700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02e40682f0 0x7f02e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:37:39.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.155+0000 7f02f5ffb700 1 -- 172.21.15.7:0/117620836 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f02f8086000 con 0x7f0300155410 2023-03-03T18:37:39.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.155+0000 7f03053ee700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02e40682f0 0x7f02e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:37:39.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.156+0000 7f03053ee700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02e40682f0 0x7f02e406a7a0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f02fc014040 tx=0x7f02fc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:37:39.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.160+0000 7f02f5ffb700 1 -- 172.21.15.7:0/117620836 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f02f80523a0 con 0x7f0300155410 2023-03-03T18:37:39.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.395+0000 7f0307652700 1 -- 172.21.15.7:0/117620836 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03000c8fa0 con 0x7f0300155410 2023-03-03T18:37:39.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.395+0000 7f02f5ffb700 1 -- 172.21.15.7:0/117620836 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f02f80559c0 con 0x7f0300155410 2023-03-03T18:37:39.396 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:37:39.396 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:37:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 -- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02e40682f0 msgr2=0x7f02e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02e40682f0 0x7f02e406a7a0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f02fc014040 tx=0x7f02fc008030 comp rx=0 tx=0).stop 2023-03-03T18:37:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 -- 172.21.15.7:0/117620836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300155410 msgr2=0x7f0300146cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:37:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300155410 0x7f0300146cd0 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f02f800cb80 tx=0x7f02f800cf40 comp rx=0 tx=0).stop 2023-03-03T18:37:39.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 -- 172.21.15.7:0/117620836 shutdown_connections 2023-03-03T18:37:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02e40682f0 0x7f02e406a7a0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0300155410 0x7f0300146cd0 unknown :-1 s=CLOSED pgs=3247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f03000c6fe0 0x7f0300145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 --2- 172.21.15.7:0/117620836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0300151e80 0x7f03001ddd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:37:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 -- 172.21.15.7:0/117620836 >> 172.21.15.7:0/117620836 conn(0x7f0300071630 msgr2=0x7f03000c2210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:37:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 -- 172.21.15.7:0/117620836 shutdown_connections 2023-03-03T18:37:39.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:37:39.397+0000 7f0307652700 1 -- 172.21.15.7:0/117620836 wait complete. 2023-03-03T18:37:39.400 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:37:39.410 INFO:tasks.mds_thrash.fs.[a]:waiting for 62 secs before thrashing 2023-03-03T18:37:40.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:37:40.212 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:38:10.346 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:38:10.350 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:38:40.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:38:40.487 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:38:41.411 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:38:41.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.549+0000 7f5085bbc700 1 Processor -- start 2023-03-03T18:38:41.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.549+0000 7f5085bbc700 1 -- start start 2023-03-03T18:38:41.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f5085bbc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f508006d870 0x7f50801550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:38:41.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f5085bbc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50800c1da0 0x7f50800c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:38:41.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f5085bbc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f50800c3e30 0x7f50800c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:38:41.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f5085bbc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50800c2af0 con 0x7f50800c1da0 2023-03-03T18:38:41.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f5085bbc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50800c3620 con 0x7f50800c3e30 2023-03-03T18:38:41.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f5085bbc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50801545e0 con 0x7f508006d870 2023-03-03T18:38:41.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50800c1da0 0x7f50800c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:38:41.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50800c1da0 0x7f50800c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38002/0 (socket says 172.21.15.7:38002) 2023-03-03T18:38:41.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507effd700 1 -- 172.21.15.7:0/615874054 learned_addr learned my addr 172.21.15.7:0/615874054 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:38:41.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507effd700 1 -- 172.21.15.7:0/615874054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f508006d870 msgr2=0x7f50801550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:38:41.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507ffff700 1 --2- 172.21.15.7:0/615874054 >> v2:172.21.15.143:3300/0 conn(0x7f50800c3e30 0x7f50800c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:38:41.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507f7fe700 1 --2- 172.21.15.7:0/615874054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f508006d870 0x7f50801550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:38:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507effd700 1 --2- 172.21.15.7:0/615874054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f508006d870 0x7f50801550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507effd700 1 -- 172.21.15.7:0/615874054 >> v2:172.21.15.143:3300/0 conn(0x7f50800c3e30 msgr2=0x7f50800c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:38:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507effd700 1 --2- 172.21.15.7:0/615874054 >> v2:172.21.15.143:3300/0 conn(0x7f50800c3e30 0x7f50800c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507effd700 1 -- 172.21.15.7:0/615874054 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50801548c0 con 0x7f50800c1da0 2023-03-03T18:38:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507ffff700 1 --2- 172.21.15.7:0/615874054 >> v2:172.21.15.143:3300/0 conn(0x7f50800c3e30 0x7f50800c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:38:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.550+0000 7f507effd700 1 --2- 172.21.15.7:0/615874054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50800c1da0 0x7f50800c7350 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f508006be80 tx=0x7f5068009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d8726a55210b76 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:38:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f507e7fc700 1 -- 172.21.15.7:0/615874054 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506800b870 con 0x7f50800c1da0 2023-03-03T18:38:41.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f507e7fc700 1 -- 172.21.15.7:0/615874054 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506800b9d0 con 0x7f50800c1da0 2023-03-03T18:38:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f507e7fc700 1 -- 172.21.15.7:0/615874054 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506800bd60 con 0x7f50800c1da0 2023-03-03T18:38:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f5085bbc700 1 -- 172.21.15.7:0/615874054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50800c1da0 msgr2=0x7f50800c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:38:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f5085bbc700 1 --2- 172.21.15.7:0/615874054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50800c1da0 0x7f50800c7350 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f508006be80 tx=0x7f5068009bd0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f5085bbc700 1 -- 172.21.15.7:0/615874054 shutdown_connections 2023-03-03T18:38:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f5085bbc700 1 --2- 172.21.15.7:0/615874054 >> v2:172.21.15.143:3300/0 conn(0x7f50800c3e30 0x7f50800c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f5085bbc700 1 --2- 172.21.15.7:0/615874054 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f508006d870 0x7f50801550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f5085bbc700 1 --2- 172.21.15.7:0/615874054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50800c1da0 0x7f50800c7350 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f5085bbc700 1 -- 172.21.15.7:0/615874054 >> 172.21.15.7:0/615874054 conn(0x7f508006c110 msgr2=0x7f508006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:38:41.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f5085bbc700 1 -- 172.21.15.7:0/615874054 shutdown_connections 2023-03-03T18:38:41.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.551+0000 7f5085bbc700 1 -- 172.21.15.7:0/615874054 wait complete. 2023-03-03T18:38:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.552+0000 7f5085bbc700 1 Processor -- start 2023-03-03T18:38:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f5085bbc700 1 -- start start 2023-03-03T18:38:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f5085bbc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f508006d870 0x7f5080158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:38:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f5085bbc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50800c1da0 0x7f5080152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:38:41.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f5085bbc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50800c3e30 0x7f50801ecf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:38:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f5085bbc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5080157510 con 0x7f508006d870 2023-03-03T18:38:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f5085bbc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50801e9830 con 0x7f50800c1da0 2023-03-03T18:38:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f5085bbc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50801e9b10 con 0x7f50800c3e30 2023-03-03T18:38:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f507f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f508006d870 0x7f5080158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:38:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f507f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f508006d870 0x7f5080158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38006/0 (socket says 172.21.15.7:38006) 2023-03-03T18:38:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f507f7fe700 1 -- 172.21.15.7:0/3325395601 learned_addr learned my addr 172.21.15.7:0/3325395601 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:38:41.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f507effd700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50800c1da0 0x7f5080152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:38:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.553+0000 7f507ffff700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50800c3e30 0x7f50801ecf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:38:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f507f7fe700 1 -- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50800c3e30 msgr2=0x7f50801ecf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:38:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f507f7fe700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50800c3e30 0x7f50801ecf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f507f7fe700 1 -- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50800c1da0 msgr2=0x7f5080152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:38:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f507f7fe700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50800c1da0 0x7f5080152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f507f7fe700 1 -- 172.21.15.7:0/3325395601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5068009550 con 0x7f508006d870 2023-03-03T18:38:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f507f7fe700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f508006d870 0x7f5080158320 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f507000c7f0 tx=0x7f507000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:38:41.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f5077fff700 1 -- 172.21.15.7:0/3325395601 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070004220 con 0x7f508006d870 2023-03-03T18:38:41.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f5077fff700 1 -- 172.21.15.7:0/3325395601 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5070004380 con 0x7f508006d870 2023-03-03T18:38:41.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f5085bbc700 1 -- 172.21.15.7:0/3325395601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508006dc40 con 0x7f508006d870 2023-03-03T18:38:41.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f5077fff700 1 -- 172.21.15.7:0/3325395601 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50700036a0 con 0x7f508006d870 2023-03-03T18:38:41.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.554+0000 7f5085bbc700 1 -- 172.21.15.7:0/3325395601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5080154ba0 con 0x7f508006d870 2023-03-03T18:38:41.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.555+0000 7f5085bbc700 1 -- 172.21.15.7:0/3325395601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50800c38e0 con 0x7f508006d870 2023-03-03T18:38:41.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.555+0000 7f5077fff700 1 -- 172.21.15.7:0/3325395601 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5070020070 con 0x7f508006d870 2023-03-03T18:38:41.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.556+0000 7f5077fff700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50600682f0 0x7f506006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:38:41.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.556+0000 7f5077fff700 1 -- 172.21.15.7:0/3325395601 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5070085b30 con 0x7f508006d870 2023-03-03T18:38:41.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.556+0000 7f507effd700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50600682f0 0x7f506006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:38:41.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.557+0000 7f507effd700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50600682f0 0x7f506006a7a0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f5068000c00 tx=0x7f5068008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:38:41.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.560+0000 7f5077fff700 1 -- 172.21.15.7:0/3325395601 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5070051e40 con 0x7f508006d870 2023-03-03T18:38:41.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.798+0000 7f5085bbc700 1 -- 172.21.15.7:0/3325395601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5080074bb0 con 0x7f508006d870 2023-03-03T18:38:41.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.799+0000 7f5077fff700 1 -- 172.21.15.7:0/3325395601 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f507001e090 con 0x7f508006d870 2023-03-03T18:38:41.800 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:38:41.800 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:38:41.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 -- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50600682f0 msgr2=0x7f506006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:38:41.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50600682f0 0x7f506006a7a0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f5068000c00 tx=0x7f5068008030 comp rx=0 tx=0).stop 2023-03-03T18:38:41.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 -- 172.21.15.7:0/3325395601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f508006d870 msgr2=0x7f5080158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:38:41.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f508006d870 0x7f5080158320 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f507000c7f0 tx=0x7f507000cb00 comp rx=0 tx=0).stop 2023-03-03T18:38:41.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 -- 172.21.15.7:0/3325395601 shutdown_connections 2023-03-03T18:38:41.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50800c3e30 0x7f50801ecf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50600682f0 0x7f506006a7a0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f508006d870 0x7f5080158320 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 --2- 172.21.15.7:0/3325395601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50800c1da0 0x7f5080152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:38:41.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 -- 172.21.15.7:0/3325395601 >> 172.21.15.7:0/3325395601 conn(0x7f508006c110 msgr2=0x7f508014ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:38:41.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 -- 172.21.15.7:0/3325395601 shutdown_connections 2023-03-03T18:38:41.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:38:41.801+0000 7f5085bbc700 1 -- 172.21.15.7:0/3325395601 wait complete. 2023-03-03T18:38:41.804 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:38:41.814 INFO:tasks.mds_thrash.fs.[a]:waiting for 25 secs before thrashing 2023-03-03T18:39:06.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:39:06.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.953+0000 7fc956f16700 1 Processor -- start 2023-03-03T18:39:06.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.954+0000 7fc956f16700 1 -- start start 2023-03-03T18:39:06.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.954+0000 7fc956f16700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc95014f9d0 0x7fc950156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.954+0000 7fc956f16700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9500abe20 0x7fc9501529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.954+0000 7fc956f16700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9500c2f30 con 0x7fc95014fda0 2023-03-03T18:39:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.954+0000 7fc956f16700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9500c31d0 con 0x7fc9500abe20 2023-03-03T18:39:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.954+0000 7fc956f16700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9501524a0 con 0x7fc95014f9d0 2023-03-03T18:39:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc9554b3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9500abe20 0x7fc9501529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc9554b3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9500abe20 0x7fc9501529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38032/0 (socket says 172.21.15.7:38032) 2023-03-03T18:39:06.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc9554b3700 1 -- 172.21.15.7:0/3143104505 learned_addr learned my addr 172.21.15.7:0/3143104505 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:39:06.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc954cb2700 1 --2- 172.21.15.7:0/3143104505 >> v2:172.21.15.143:3300/0 conn(0x7fc95014f9d0 0x7fc950156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:06.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc9554b3700 1 -- 172.21.15.7:0/3143104505 >> v2:172.21.15.143:3300/0 conn(0x7fc95014f9d0 msgr2=0x7fc950156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:39:06.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc9554b3700 1 --2- 172.21.15.7:0/3143104505 >> v2:172.21.15.143:3300/0 conn(0x7fc95014f9d0 0x7fc950156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:06.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc9554b3700 1 -- 172.21.15.7:0/3143104505 >> v1:172.21.15.143:6789/0 conn(0x7fc95014fda0 legacy=0x7fc950071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:39:06.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc9554b3700 1 -- 172.21.15.7:0/3143104505 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9500c34b0 con 0x7fc9500abe20 2023-03-03T18:39:06.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc9554b3700 1 --2- 172.21.15.7:0/3143104505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9500abe20 0x7fc9501529c0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7fc9500710a0 tx=0x7fc94800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9119f6a23ab9d98 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:39:06.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc94f7fe700 1 -- 172.21.15.7:0/3143104505 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94800e070 con 0x7fc9500abe20 2023-03-03T18:39:06.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc94f7fe700 1 -- 172.21.15.7:0/3143104505 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc948003a20 con 0x7fc9500abe20 2023-03-03T18:39:06.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.955+0000 7fc94f7fe700 1 -- 172.21.15.7:0/3143104505 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948003dd0 con 0x7fc9500abe20 2023-03-03T18:39:06.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.956+0000 7fc956f16700 1 -- 172.21.15.7:0/3143104505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9500abe20 msgr2=0x7fc9501529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:06.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.956+0000 7fc956f16700 1 --2- 172.21.15.7:0/3143104505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9500abe20 0x7fc9501529c0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7fc9500710a0 tx=0x7fc94800b330 comp rx=0 tx=0).stop 2023-03-03T18:39:06.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.956+0000 7fc956f16700 1 -- 172.21.15.7:0/3143104505 shutdown_connections 2023-03-03T18:39:06.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.956+0000 7fc956f16700 1 --2- 172.21.15.7:0/3143104505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9500abe20 0x7fc9501529c0 unknown :-1 s=CLOSED pgs=3250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:06.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.956+0000 7fc956f16700 1 --2- 172.21.15.7:0/3143104505 >> v2:172.21.15.143:3300/0 conn(0x7fc95014f9d0 0x7fc950156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:06.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.956+0000 7fc956f16700 1 -- 172.21.15.7:0/3143104505 >> 172.21.15.7:0/3143104505 conn(0x7fc9500c8900 msgr2=0x7fc9500c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:39:06.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.956+0000 7fc956f16700 1 -- 172.21.15.7:0/3143104505 shutdown_connections 2023-03-03T18:39:06.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.956+0000 7fc956f16700 1 -- 172.21.15.7:0/3143104505 wait complete. 2023-03-03T18:39:06.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.957+0000 7fc956f16700 1 Processor -- start 2023-03-03T18:39:06.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.957+0000 7fc956f16700 1 -- start start 2023-03-03T18:39:06.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc956f16700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9500abe20 0x7fc9501db6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:06.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc956f16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc95014f9d0 0x7fc9501e7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:06.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc956f16700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95014fda0 0x7fc9500c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:06.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc956f16700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500c25e0 con 0x7fc95014fda0 2023-03-03T18:39:06.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc956f16700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9501dbc20 con 0x7fc9500abe20 2023-03-03T18:39:06.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc956f16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500c2020 con 0x7fc95014f9d0 2023-03-03T18:39:06.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc9554b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95014fda0 0x7fc9500c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:06.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc9554b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95014fda0 0x7fc9500c6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38038/0 (socket says 172.21.15.7:38038) 2023-03-03T18:39:06.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc9554b3700 1 -- 172.21.15.7:0/1368231002 learned_addr learned my addr 172.21.15.7:0/1368231002 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:39:06.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc954cb2700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9500abe20 0x7fc9501db6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:06.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc94ffff700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc95014f9d0 0x7fc9501e7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:06.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.958+0000 7fc9554b3700 1 -- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc95014f9d0 msgr2=0x7fc9501e7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:06.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.959+0000 7fc9554b3700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc95014f9d0 0x7fc9501e7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:06.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.959+0000 7fc9554b3700 1 -- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9500abe20 msgr2=0x7fc9501db6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:06.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.959+0000 7fc9554b3700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9500abe20 0x7fc9501db6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:06.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.959+0000 7fc9554b3700 1 -- 172.21.15.7:0/1368231002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc948009b30 con 0x7fc95014fda0 2023-03-03T18:39:06.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.959+0000 7fc9554b3700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95014fda0 0x7fc9500c6860 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7fc94800b370 tx=0x7fc948004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:39:06.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.959+0000 7fc94d7fa700 1 -- 172.21.15.7:0/1368231002 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948004440 con 0x7fc95014fda0 2023-03-03T18:39:06.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.959+0000 7fc94d7fa700 1 -- 172.21.15.7:0/1368231002 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9480092e0 con 0x7fc95014fda0 2023-03-03T18:39:06.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.959+0000 7fc94d7fa700 1 -- 172.21.15.7:0/1368231002 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948004800 con 0x7fc95014fda0 2023-03-03T18:39:06.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.959+0000 7fc956f16700 1 -- 172.21.15.7:0/1368231002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9500ac350 con 0x7fc95014fda0 2023-03-03T18:39:06.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.959+0000 7fc956f16700 1 -- 172.21.15.7:0/1368231002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9500ab630 con 0x7fc95014fda0 2023-03-03T18:39:06.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.960+0000 7fc956f16700 1 -- 172.21.15.7:0/1368231002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc950071fa0 con 0x7fc95014fda0 2023-03-03T18:39:06.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.960+0000 7fc94d7fa700 1 -- 172.21.15.7:0/1368231002 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc948022030 con 0x7fc95014fda0 2023-03-03T18:39:06.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.961+0000 7fc94d7fa700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9340682f0 0x7fc93406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:06.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.961+0000 7fc94d7fa700 1 -- 172.21.15.7:0/1368231002 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc948087240 con 0x7fc95014fda0 2023-03-03T18:39:06.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.961+0000 7fc954cb2700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9340682f0 0x7fc93406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:06.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.962+0000 7fc954cb2700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9340682f0 0x7fc93406a7a0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fc9500c27b0 tx=0x7fc944008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:39:06.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:06.965+0000 7fc94d7fa700 1 -- 172.21.15.7:0/1368231002 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc9480535e0 con 0x7fc95014fda0 2023-03-03T18:39:07.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.207+0000 7fc956f16700 1 -- 172.21.15.7:0/1368231002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc950074bb0 con 0x7fc95014fda0 2023-03-03T18:39:07.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.207+0000 7fc94d7fa700 1 -- 172.21.15.7:0/1368231002 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc948056c00 con 0x7fc95014fda0 2023-03-03T18:39:07.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:39:07.207 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:39:07.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 -- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9340682f0 msgr2=0x7fc93406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:07.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9340682f0 0x7fc93406a7a0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fc9500c27b0 tx=0x7fc944008040 comp rx=0 tx=0).stop 2023-03-03T18:39:07.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 -- 172.21.15.7:0/1368231002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95014fda0 msgr2=0x7fc9500c6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95014fda0 0x7fc9500c6860 secure :-1 s=READY pgs=3251 cs=0 l=1 rev1=1 crypto rx=0x7fc94800b370 tx=0x7fc948004ad0 comp rx=0 tx=0).stop 2023-03-03T18:39:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 -- 172.21.15.7:0/1368231002 shutdown_connections 2023-03-03T18:39:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9340682f0 0x7fc93406a7a0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc95014fda0 0x7fc9500c6860 unknown :-1 s=CLOSED pgs=3251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9500abe20 0x7fc9501db6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 --2- 172.21.15.7:0/1368231002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc95014f9d0 0x7fc9501e7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 -- 172.21.15.7:0/1368231002 >> 172.21.15.7:0/1368231002 conn(0x7fc9500c8900 msgr2=0x7fc9500c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:39:07.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 -- 172.21.15.7:0/1368231002 shutdown_connections 2023-03-03T18:39:07.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:07.209+0000 7fc956f16700 1 -- 172.21.15.7:0/1368231002 wait complete. 2023-03-03T18:39:07.211 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:39:07.221 INFO:tasks.mds_thrash.fs.[a]:waiting for 110 secs before thrashing 2023-03-03T18:39:10.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:39:10.550 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:39:14.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:39:14.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.265+0000 7f523c93a700 1 Processor -- start 2023-03-03T18:39:14.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.265+0000 7f523c93a700 1 -- start start 2023-03-03T18:39:14.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.266+0000 7f523c93a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f523806c480 0x7f52381581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:14.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.266+0000 7f523c93a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523806cb70 con 0x7f5238151b20 2023-03-03T18:39:14.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.266+0000 7f523c93a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523806ce50 con 0x7f52380c2950 2023-03-03T18:39:14.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.266+0000 7f523c93a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523806c2f0 con 0x7f523806c480 2023-03-03T18:39:14.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.266+0000 7f523659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5238151b20 0x7f52380736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60254/0 (socket says 172.21.15.7:60254) 2023-03-03T18:39:14.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.266+0000 7f523659c700 1 -- 172.21.15.7:0/1782787130 learned_addr learned my addr 172.21.15.7:0/1782787130 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:39:14.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.266+0000 7f5236d9d700 1 --2- 172.21.15.7:0/1782787130 >> v2:172.21.15.143:3300/0 conn(0x7f523806c480 0x7f52381581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:14.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f523559a700 1 -- 172.21.15.7:0/1782787130 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865608012 0 0) 0x7f523806cb70 con 0x7f5238151b20 2023-03-03T18:39:14.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f523559a700 1 -- 172.21.15.7:0/1782787130 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52240035b0 con 0x7f5238151b20 2023-03-03T18:39:14.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f523559a700 1 -- 172.21.15.7:0/1782787130 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815522761 0 0) 0x7f523806ce50 con 0x7f52380c2950 2023-03-03T18:39:14.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f523559a700 1 -- 172.21.15.7:0/1782787130 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f523806cb70 con 0x7f52380c2950 2023-03-03T18:39:14.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f5236d9d700 1 -- 172.21.15.7:0/1782787130 >> v1:172.21.15.143:6789/0 conn(0x7f52380c2950 legacy=0x7f5238157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:14.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f5236d9d700 1 -- 172.21.15.7:0/1782787130 >> v1:172.21.15.7:6789/0 conn(0x7f5238151b20 legacy=0x7f52380736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:14.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f5236d9d700 1 -- 172.21.15.7:0/1782787130 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523806d130 con 0x7f523806c480 2023-03-03T18:39:14.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f5236d9d700 1 --2- 172.21.15.7:0/1782787130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f523806c480 0x7f52381581f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5230009d90 tx=0x7f523000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b739b7f3ffd3d1aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:39:14.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f523559a700 1 -- 172.21.15.7:0/1782787130 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523000e070 con 0x7f523806c480 2023-03-03T18:39:14.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f523559a700 1 -- 172.21.15.7:0/1782787130 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523000bb20 con 0x7f523806c480 2023-03-03T18:39:14.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.267+0000 7f523559a700 1 -- 172.21.15.7:0/1782787130 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5230012430 con 0x7f523806c480 2023-03-03T18:39:14.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.268+0000 7f523c93a700 1 -- 172.21.15.7:0/1782787130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f523806c480 msgr2=0x7f52381581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:14.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.268+0000 7f523c93a700 1 --2- 172.21.15.7:0/1782787130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f523806c480 0x7f52381581f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5230009d90 tx=0x7f523000b330 comp rx=0 tx=0).stop 2023-03-03T18:39:14.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.268+0000 7f523c93a700 1 -- 172.21.15.7:0/1782787130 shutdown_connections 2023-03-03T18:39:14.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.268+0000 7f523c93a700 1 --2- 172.21.15.7:0/1782787130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f523806c480 0x7f52381581f0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:14.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.268+0000 7f523c93a700 1 -- 172.21.15.7:0/1782787130 >> 172.21.15.7:0/1782787130 conn(0x7f52380a94d0 msgr2=0x7f52380a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:39:14.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.268+0000 7f523c93a700 1 -- 172.21.15.7:0/1782787130 shutdown_connections 2023-03-03T18:39:14.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.268+0000 7f523c93a700 1 -- 172.21.15.7:0/1782787130 wait complete. 2023-03-03T18:39:14.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.269+0000 7f523c93a700 1 Processor -- start 2023-03-03T18:39:14.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.269+0000 7f523c93a700 1 -- start start 2023-03-03T18:39:14.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f523c93a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f523806c480 0x7f5238156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:14.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f523c93a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52380c2950 0x7f5238157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:14.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f523c93a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5238151b20 0x7f5238149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:14.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f523c93a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5238156df0 con 0x7f52380c2950 2023-03-03T18:39:14.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f523c93a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5238156bd0 con 0x7f523806c480 2023-03-03T18:39:14.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f523c93a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52381523e0 con 0x7f5238151b20 2023-03-03T18:39:14.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f5235d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52380c2950 0x7f5238157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:14.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f5235d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52380c2950 0x7f5238157240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38048/0 (socket says 172.21.15.7:38048) 2023-03-03T18:39:14.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f5235d9b700 1 -- 172.21.15.7:0/3225504602 learned_addr learned my addr 172.21.15.7:0/3225504602 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:39:14.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f523659c700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f523806c480 0x7f5238156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:14.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f5236d9d700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5238151b20 0x7f5238149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:14.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f5235d9b700 1 -- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5238151b20 msgr2=0x7f5238149a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:14.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.270+0000 7f5235d9b700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5238151b20 0x7f5238149a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:14.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.271+0000 7f5235d9b700 1 -- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f523806c480 msgr2=0x7f5238156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:14.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.271+0000 7f5235d9b700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f523806c480 0x7f5238156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:14.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.271+0000 7f5235d9b700 1 -- 172.21.15.7:0/3225504602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5230009a70 con 0x7f52380c2950 2023-03-03T18:39:14.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.271+0000 7f5235d9b700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52380c2950 0x7f5238157240 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7f522800c530 tx=0x7f522800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:39:14.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.271+0000 7f522effd700 1 -- 172.21.15.7:0/3225504602 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522800cea0 con 0x7f52380c2950 2023-03-03T18:39:14.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.271+0000 7f522effd700 1 -- 172.21.15.7:0/3225504602 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52280052f0 con 0x7f52380c2950 2023-03-03T18:39:14.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.271+0000 7f522effd700 1 -- 172.21.15.7:0/3225504602 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228003680 con 0x7f52380c2950 2023-03-03T18:39:14.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.271+0000 7f523c93a700 1 -- 172.21.15.7:0/3225504602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5238158730 con 0x7f52380c2950 2023-03-03T18:39:14.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.271+0000 7f523c93a700 1 -- 172.21.15.7:0/3225504602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523806d410 con 0x7f52380c2950 2023-03-03T18:39:14.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.272+0000 7f523c93a700 1 -- 172.21.15.7:0/3225504602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5238151ef0 con 0x7f52380c2950 2023-03-03T18:39:14.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.272+0000 7f522effd700 1 -- 172.21.15.7:0/3225504602 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5228019030 con 0x7f52380c2950 2023-03-03T18:39:14.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.273+0000 7f522effd700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52180682f0 0x7f521806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:14.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.273+0000 7f522effd700 1 -- 172.21.15.7:0/3225504602 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5228085480 con 0x7f52380c2950 2023-03-03T18:39:14.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.273+0000 7f523659c700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52180682f0 0x7f521806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:14.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.274+0000 7f523659c700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52180682f0 0x7f521806a7a0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f5220007fd0 tx=0x7f5220007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:39:14.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.277+0000 7f522effd700 1 -- 172.21.15.7:0/3225504602 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5228051730 con 0x7f52380c2950 2023-03-03T18:39:14.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.516+0000 7f523c93a700 1 -- 172.21.15.7:0/3225504602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5238154410 con 0x7f52380c2950 2023-03-03T18:39:14.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.517+0000 7f522effd700 1 -- 172.21.15.7:0/3225504602 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5228054d50 con 0x7f52380c2950 2023-03-03T18:39:14.517 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:39:14.518 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:39:14.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 -- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52180682f0 msgr2=0x7f521806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:14.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52180682f0 0x7f521806a7a0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f5220007fd0 tx=0x7f5220007dc0 comp rx=0 tx=0).stop 2023-03-03T18:39:14.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 -- 172.21.15.7:0/3225504602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52380c2950 msgr2=0x7f5238157240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:14.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52380c2950 0x7f5238157240 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7f522800c530 tx=0x7f522800c840 comp rx=0 tx=0).stop 2023-03-03T18:39:14.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 -- 172.21.15.7:0/3225504602 shutdown_connections 2023-03-03T18:39:14.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5238151b20 0x7f5238149a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:14.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f523806c480 0x7f5238156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:14.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f52180682f0 0x7f521806a7a0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:14.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 --2- 172.21.15.7:0/3225504602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52380c2950 0x7f5238157240 unknown :-1 s=CLOSED pgs=3253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:14.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 -- 172.21.15.7:0/3225504602 >> 172.21.15.7:0/3225504602 conn(0x7f52380a94d0 msgr2=0x7f52380c4730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:39:14.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 -- 172.21.15.7:0/3225504602 shutdown_connections 2023-03-03T18:39:14.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:14.519+0000 7f523c93a700 1 -- 172.21.15.7:0/3225504602 wait complete. 2023-03-03T18:39:14.521 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:39:14.532 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before thrashing 2023-03-03T18:39:20.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:39:20.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.672+0000 7f5f61373700 1 Processor -- start 2023-03-03T18:39:20.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.673+0000 7f5f61373700 1 -- start start 2023-03-03T18:39:20.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.673+0000 7f5f61373700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f5c151b00 0x7f5f5c06e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:20.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.673+0000 7f5f61373700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f5c154420 0x7f5f5c1548a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:20.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.673+0000 7f5f61373700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c06e730 con 0x7f5f5c0c4630 2023-03-03T18:39:20.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.673+0000 7f5f61373700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5c1583e0 con 0x7f5f5c151b00 2023-03-03T18:39:20.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.673+0000 7f5f61373700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c157ab0 con 0x7f5f5c154420 2023-03-03T18:39:20.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.673+0000 7f5f5a7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5f5c0c4630 0x7f5f5c0a94c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60270/0 (socket says 172.21.15.7:60270) 2023-03-03T18:39:20.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.673+0000 7f5f5affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f5c151b00 0x7f5f5c06e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:20.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f5a7fc700 1 -- 172.21.15.7:0/3795148187 learned_addr learned my addr 172.21.15.7:0/3795148187 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:39:20.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f5b7fe700 1 --2- 172.21.15.7:0/3795148187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f5c154420 0x7f5f5c1548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:20.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f59ffb700 1 -- 172.21.15.7:0/3795148187 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536826989 0 0) 0x7f5f5c06e730 con 0x7f5f5c0c4630 2023-03-03T18:39:20.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f59ffb700 1 -- 172.21.15.7:0/3795148187 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f480035b0 con 0x7f5f5c0c4630 2023-03-03T18:39:20.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f5affd700 1 -- 172.21.15.7:0/3795148187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f5c154420 msgr2=0x7f5f5c1548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:39:20.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f5affd700 1 --2- 172.21.15.7:0/3795148187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f5c154420 0x7f5f5c1548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:20.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f5affd700 1 -- 172.21.15.7:0/3795148187 >> v1:172.21.15.7:6789/0 conn(0x7f5f5c0c4630 legacy=0x7f5f5c0a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:20.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f5affd700 1 -- 172.21.15.7:0/3795148187 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c157d90 con 0x7f5f5c151b00 2023-03-03T18:39:20.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f5affd700 1 --2- 172.21.15.7:0/3795148187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c151b00 0x7f5f5c06e150 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009840 tx=0x7f5f4c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca55afe3d5c9a24b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:39:20.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f59ffb700 1 -- 172.21.15.7:0/3795148187 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c011830 con 0x7f5f5c151b00 2023-03-03T18:39:20.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f59ffb700 1 -- 172.21.15.7:0/3795148187 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c011990 con 0x7f5f5c151b00 2023-03-03T18:39:20.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.674+0000 7f5f59ffb700 1 -- 172.21.15.7:0/3795148187 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c011d20 con 0x7f5f5c151b00 2023-03-03T18:39:20.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.675+0000 7f5f61373700 1 -- 172.21.15.7:0/3795148187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c151b00 msgr2=0x7f5f5c06e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:20.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.675+0000 7f5f61373700 1 --2- 172.21.15.7:0/3795148187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c151b00 0x7f5f5c06e150 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009840 tx=0x7f5f4c009b50 comp rx=0 tx=0).stop 2023-03-03T18:39:20.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.675+0000 7f5f61373700 1 -- 172.21.15.7:0/3795148187 shutdown_connections 2023-03-03T18:39:20.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.675+0000 7f5f61373700 1 --2- 172.21.15.7:0/3795148187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f5c154420 0x7f5f5c1548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:20.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.675+0000 7f5f61373700 1 --2- 172.21.15.7:0/3795148187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c151b00 0x7f5f5c06e150 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:20.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.675+0000 7f5f61373700 1 -- 172.21.15.7:0/3795148187 >> 172.21.15.7:0/3795148187 conn(0x7f5f5c0c84d0 msgr2=0x7f5f5c0c88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:39:20.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.675+0000 7f5f61373700 1 -- 172.21.15.7:0/3795148187 shutdown_connections 2023-03-03T18:39:20.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.675+0000 7f5f61373700 1 -- 172.21.15.7:0/3795148187 wait complete. 2023-03-03T18:39:20.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.676+0000 7f5f61373700 1 Processor -- start 2023-03-03T18:39:20.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.676+0000 7f5f61373700 1 -- start start 2023-03-03T18:39:20.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f61373700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f5c0c4630 0x7f5f5c06eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:20.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f61373700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f5c151b00 0x7f5f5c1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:20.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f61373700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c154420 0x7f5f5c152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:20.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f61373700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c153230 con 0x7f5f5c154420 2023-03-03T18:39:20.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f61373700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c152320 con 0x7f5f5c151b00 2023-03-03T18:39:20.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f61373700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c156bf0 con 0x7f5f5c0c4630 2023-03-03T18:39:20.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f5b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c154420 0x7f5f5c152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:20.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f5b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c154420 0x7f5f5c152cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38068/0 (socket says 172.21.15.7:38068) 2023-03-03T18:39:20.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f5b7fe700 1 -- 172.21.15.7:0/1274767606 learned_addr learned my addr 172.21.15.7:0/1274767606 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:39:20.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f5affd700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f5c0c4630 0x7f5f5c06eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:20.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f5a7fc700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f5c151b00 0x7f5f5c1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:20.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f5b7fe700 1 -- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f5c0c4630 msgr2=0x7f5f5c06eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:20.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f5b7fe700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f5c0c4630 0x7f5f5c06eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:20.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f5b7fe700 1 -- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f5c151b00 msgr2=0x7f5f5c1527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:20.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f5b7fe700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f5c151b00 0x7f5f5c1527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:20.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.677+0000 7f5f5b7fe700 1 -- 172.21.15.7:0/1274767606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4c0094a0 con 0x7f5f5c154420 2023-03-03T18:39:20.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.678+0000 7f5f5b7fe700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c154420 0x7f5f5c152cf0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7f5f54007e40 tx=0x7f5f5400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:39:20.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.678+0000 7f5f537fe700 1 -- 172.21.15.7:0/1274767606 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400eea0 con 0x7f5f5c154420 2023-03-03T18:39:20.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.678+0000 7f5f537fe700 1 -- 172.21.15.7:0/1274767606 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f54014420 con 0x7f5f5c154420 2023-03-03T18:39:20.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.678+0000 7f5f61373700 1 -- 172.21.15.7:0/1274767606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c158870 con 0x7f5f5c154420 2023-03-03T18:39:20.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.678+0000 7f5f537fe700 1 -- 172.21.15.7:0/1274767606 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f540147b0 con 0x7f5f5c154420 2023-03-03T18:39:20.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.678+0000 7f5f61373700 1 -- 172.21.15.7:0/1274767606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c158070 con 0x7f5f5c154420 2023-03-03T18:39:20.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.679+0000 7f5f61373700 1 -- 172.21.15.7:0/1274767606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5c0c3c30 con 0x7f5f5c154420 2023-03-03T18:39:20.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.679+0000 7f5f537fe700 1 -- 172.21.15.7:0/1274767606 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5f54012050 con 0x7f5f5c154420 2023-03-03T18:39:20.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.680+0000 7f5f537fe700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f3c0682f0 0x7f5f3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:39:20.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.680+0000 7f5f537fe700 1 -- 172.21.15.7:0/1274767606 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5f54086820 con 0x7f5f5c154420 2023-03-03T18:39:20.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.680+0000 7f5f5affd700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f3c0682f0 0x7f5f3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:39:20.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.680+0000 7f5f5affd700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f3c0682f0 0x7f5f3c06a7a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c000c00 tx=0x7f5f4c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:39:20.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.684+0000 7f5f537fe700 1 -- 172.21.15.7:0/1274767606 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5f54052bc0 con 0x7f5f5c154420 2023-03-03T18:39:20.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.924+0000 7f5f61373700 1 -- 172.21.15.7:0/1274767606 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f5c0aace0 con 0x7f5f5c154420 2023-03-03T18:39:20.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.924+0000 7f5f537fe700 1 -- 172.21.15.7:0/1274767606 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5f540561e0 con 0x7f5f5c154420 2023-03-03T18:39:20.925 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:39:20.925 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:39:20.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 -- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f3c0682f0 msgr2=0x7f5f3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:20.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f3c0682f0 0x7f5f3c06a7a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c000c00 tx=0x7f5f4c01c040 comp rx=0 tx=0).stop 2023-03-03T18:39:20.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 -- 172.21.15.7:0/1274767606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c154420 msgr2=0x7f5f5c152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:39:20.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c154420 0x7f5f5c152cf0 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7f5f54007e40 tx=0x7f5f5400e3c0 comp rx=0 tx=0).stop 2023-03-03T18:39:20.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 -- 172.21.15.7:0/1274767606 shutdown_connections 2023-03-03T18:39:20.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5f3c0682f0 0x7f5f3c06a7a0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:20.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f5c154420 0x7f5f5c152cf0 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:20.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5f5c0c4630 0x7f5f5c06eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:20.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 --2- 172.21.15.7:0/1274767606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f5c151b00 0x7f5f5c1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:39:20.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 -- 172.21.15.7:0/1274767606 >> 172.21.15.7:0/1274767606 conn(0x7f5f5c0c84d0 msgr2=0x7f5f5c0c17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:39:20.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 -- 172.21.15.7:0/1274767606 shutdown_connections 2023-03-03T18:39:20.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:39:20.927+0000 7f5f61373700 1 -- 172.21.15.7:0/1274767606 wait complete. 2023-03-03T18:39:20.928 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:39:20.939 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2023-03-03T18:39:40.694 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:39:40.697 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:39:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T18:39:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T18:40:10.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:40:10.828 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:40:40.946 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:40:40.950 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:40:46.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:40:47.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.079+0000 7fa71d0bf700 1 Processor -- start 2023-03-03T18:40:47.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.079+0000 7fa71d0bf700 1 -- start start 2023-03-03T18:40:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.079+0000 7fa71d0bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7180c2e30 0x7fa718158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:47.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.079+0000 7fa71d0bf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa718157ff0 con 0x7fa71806c6c0 2023-03-03T18:40:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.079+0000 7fa71d0bf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71806d640 con 0x7fa718151cd0 2023-03-03T18:40:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.079+0000 7fa71d0bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa71806cda0 con 0x7fa7180c2e30 2023-03-03T18:40:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa71759e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa71806c6c0 0x7fa7180736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60350/0 (socket says 172.21.15.7:60350) 2023-03-03T18:40:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa71759e700 1 -- 172.21.15.7:0/651245130 learned_addr learned my addr 172.21.15.7:0/651245130 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:40:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa71659c700 1 --2- 172.21.15.7:0/651245130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7180c2e30 0x7fa718158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa715d9b700 1 -- 172.21.15.7:0/651245130 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020185376 0 0) 0x7fa718157ff0 con 0x7fa71806c6c0 2023-03-03T18:40:47.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa715d9b700 1 -- 172.21.15.7:0/651245130 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7000035b0 con 0x7fa71806c6c0 2023-03-03T18:40:47.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa715d9b700 1 -- 172.21.15.7:0/651245130 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721802136 0 0) 0x7fa71806d640 con 0x7fa718151cd0 2023-03-03T18:40:47.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa715d9b700 1 -- 172.21.15.7:0/651245130 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa718157ff0 con 0x7fa718151cd0 2023-03-03T18:40:47.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa71659c700 1 -- 172.21.15.7:0/651245130 >> v1:172.21.15.143:6789/0 conn(0x7fa718151cd0 legacy=0x7fa718154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:47.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa71659c700 1 -- 172.21.15.7:0/651245130 >> v1:172.21.15.7:6789/0 conn(0x7fa71806c6c0 legacy=0x7fa7180736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:47.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa71659c700 1 -- 172.21.15.7:0/651245130 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71806d080 con 0x7fa7180c2e30 2023-03-03T18:40:47.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa71659c700 1 --2- 172.21.15.7:0/651245130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7180c2e30 0x7fa718158480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009870 tx=0x7fa70c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b984ac4a84a32f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:40:47.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.080+0000 7fa715d9b700 1 -- 172.21.15.7:0/651245130 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c011830 con 0x7fa7180c2e30 2023-03-03T18:40:47.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.081+0000 7fa715d9b700 1 -- 172.21.15.7:0/651245130 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70c011990 con 0x7fa7180c2e30 2023-03-03T18:40:47.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.081+0000 7fa715d9b700 1 -- 172.21.15.7:0/651245130 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c011d20 con 0x7fa7180c2e30 2023-03-03T18:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.081+0000 7fa71d0bf700 1 -- 172.21.15.7:0/651245130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7180c2e30 msgr2=0x7fa718158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.081+0000 7fa71d0bf700 1 --2- 172.21.15.7:0/651245130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7180c2e30 0x7fa718158480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa70c009870 tx=0x7fa70c009b80 comp rx=0 tx=0).stop 2023-03-03T18:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.081+0000 7fa71d0bf700 1 -- 172.21.15.7:0/651245130 shutdown_connections 2023-03-03T18:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.081+0000 7fa71d0bf700 1 --2- 172.21.15.7:0/651245130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7180c2e30 0x7fa718158480 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.081+0000 7fa71d0bf700 1 -- 172.21.15.7:0/651245130 >> 172.21.15.7:0/651245130 conn(0x7fa7180a94d0 msgr2=0x7fa7180a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:40:47.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.081+0000 7fa71d0bf700 1 -- 172.21.15.7:0/651245130 shutdown_connections 2023-03-03T18:40:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.081+0000 7fa71d0bf700 1 -- 172.21.15.7:0/651245130 wait complete. 2023-03-03T18:40:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.083+0000 7fa71d0bf700 1 Processor -- start 2023-03-03T18:40:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.083+0000 7fa71d0bf700 1 -- start start 2023-03-03T18:40:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.083+0000 7fa71d0bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa71806c6c0 0x7fa7181edda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.083+0000 7fa71d0bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7180c2e30 0x7fa718150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.083+0000 7fa71d0bf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa718151cd0 0x7fa7181ea2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.083+0000 7fa71d0bf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7180aa540 con 0x7fa718151cd0 2023-03-03T18:40:47.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.083+0000 7fa71d0bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7181ee2e0 con 0x7fa7180c2e30 2023-03-03T18:40:47.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.083+0000 7fa71d0bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7181521e0 con 0x7fa71806c6c0 2023-03-03T18:40:47.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71759e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa718151cd0 0x7fa7181ea2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:47.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71759e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa718151cd0 0x7fa7181ea2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38144/0 (socket says 172.21.15.7:38144) 2023-03-03T18:40:47.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71759e700 1 -- 172.21.15.7:0/4182259767 learned_addr learned my addr 172.21.15.7:0/4182259767 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:40:47.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71659c700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7180c2e30 0x7fa718150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:47.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71759e700 1 -- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa71806c6c0 msgr2=0x7fa7181edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:47.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa716d9d700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa71806c6c0 0x7fa7181edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:47.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71759e700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa71806c6c0 0x7fa7181edda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:47.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71759e700 1 -- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7180c2e30 msgr2=0x7fa718150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:47.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71759e700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7180c2e30 0x7fa718150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:47.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71759e700 1 -- 172.21.15.7:0/4182259767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70c0094a0 con 0x7fa718151cd0 2023-03-03T18:40:47.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71659c700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7180c2e30 0x7fa718150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:40:47.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa716d9d700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa71806c6c0 0x7fa7181edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:40:47.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71759e700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa718151cd0 0x7fa7181ea2a0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7fa71000c500 tx=0x7fa71000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:40:47.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa6ff7fe700 1 -- 172.21.15.7:0/4182259767 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710005d70 con 0x7fa718151cd0 2023-03-03T18:40:47.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa6ff7fe700 1 -- 172.21.15.7:0/4182259767 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7100070d0 con 0x7fa718151cd0 2023-03-03T18:40:47.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71d0bf700 1 -- 172.21.15.7:0/4182259767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa718157c80 con 0x7fa718151cd0 2023-03-03T18:40:47.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa6ff7fe700 1 -- 172.21.15.7:0/4182259767 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710007460 con 0x7fa718151cd0 2023-03-03T18:40:47.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.084+0000 7fa71d0bf700 1 -- 172.21.15.7:0/4182259767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71806d360 con 0x7fa718151cd0 2023-03-03T18:40:47.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.085+0000 7fa71d0bf700 1 -- 172.21.15.7:0/4182259767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7181589c0 con 0x7fa718151cd0 2023-03-03T18:40:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.086+0000 7fa6ff7fe700 1 -- 172.21.15.7:0/4182259767 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa710019030 con 0x7fa718151cd0 2023-03-03T18:40:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.086+0000 7fa6ff7fe700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7040682f0 0x7fa70406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.086+0000 7fa6ff7fe700 1 -- 172.21.15.7:0/4182259767 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa7100854c0 con 0x7fa718151cd0 2023-03-03T18:40:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.086+0000 7fa716d9d700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7040682f0 0x7fa70406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:47.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.087+0000 7fa716d9d700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7040682f0 0x7fa70406a7a0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7fa7080015a0 tx=0x7fa708009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:40:47.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.091+0000 7fa6ff7fe700 1 -- 172.21.15.7:0/4182259767 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa710051860 con 0x7fa718151cd0 2023-03-03T18:40:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.328+0000 7fa71d0bf700 1 -- 172.21.15.7:0/4182259767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa718074bb0 con 0x7fa718151cd0 2023-03-03T18:40:47.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.328+0000 7fa6ff7fe700 1 -- 172.21.15.7:0/4182259767 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa710054e80 con 0x7fa718151cd0 2023-03-03T18:40:47.330 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:40:47.330 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:40:47.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.330+0000 7fa71d0bf700 1 -- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7040682f0 msgr2=0x7fa70406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:47.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.330+0000 7fa71d0bf700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7040682f0 0x7fa70406a7a0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7fa7080015a0 tx=0x7fa708009040 comp rx=0 tx=0).stop 2023-03-03T18:40:47.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.330+0000 7fa71d0bf700 1 -- 172.21.15.7:0/4182259767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa718151cd0 msgr2=0x7fa7181ea2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:47.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.330+0000 7fa71d0bf700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa718151cd0 0x7fa7181ea2a0 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7fa71000c500 tx=0x7fa71000c8c0 comp rx=0 tx=0).stop 2023-03-03T18:40:47.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.330+0000 7fa71d0bf700 1 -- 172.21.15.7:0/4182259767 shutdown_connections 2023-03-03T18:40:47.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.330+0000 7fa71d0bf700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7040682f0 0x7fa70406a7a0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:47.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.330+0000 7fa71d0bf700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa718151cd0 0x7fa7181ea2a0 unknown :-1 s=CLOSED pgs=3258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:47.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.330+0000 7fa71d0bf700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa71806c6c0 0x7fa7181edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:47.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.330+0000 7fa71d0bf700 1 --2- 172.21.15.7:0/4182259767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa7180c2e30 0x7fa718150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:47.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.330+0000 7fa71d0bf700 1 -- 172.21.15.7:0/4182259767 >> 172.21.15.7:0/4182259767 conn(0x7fa7180a94d0 msgr2=0x7fa71814f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:40:47.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.331+0000 7fa71d0bf700 1 -- 172.21.15.7:0/4182259767 shutdown_connections 2023-03-03T18:40:47.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:47.331+0000 7fa71d0bf700 1 -- 172.21.15.7:0/4182259767 wait complete. 2023-03-03T18:40:47.333 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:40:47.343 INFO:tasks.mds_thrash.fs.[b]:waiting for 80 secs before thrashing 2023-03-03T18:40:57.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:40:57.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.363+0000 7f8f38aaa700 1 Processor -- start 2023-03-03T18:40:57.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.363+0000 7f8f38aaa700 1 -- start start 2023-03-03T18:40:57.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.364+0000 7f8f38aaa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8f3414ae20 0x7f8f34154e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:57.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.364+0000 7f8f38aaa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340c1ac0 0x7f8f340c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:57.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.364+0000 7f8f38aaa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f341553b0 con 0x7f8f340aabc0 2023-03-03T18:40:57.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.364+0000 7f8f38aaa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f340aa150 con 0x7f8f3414ae20 2023-03-03T18:40:57.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.364+0000 7f8f38aaa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3406f5b0 con 0x7f8f340c1ac0 2023-03-03T18:40:57.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f32d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340c1ac0 0x7f8f340c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:57.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3259c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8f3414ae20 0x7f8f34154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:57.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3259c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8f3414ae20 0x7f8f34154e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:33922/0 (socket says 172.21.15.7:33922) 2023-03-03T18:40:57.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f32d9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340c1ac0 0x7f8f340c1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:46224/0 (socket says 172.21.15.7:46224) 2023-03-03T18:40:57.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f31d9b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8f340aabc0 0x7f8f34071840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:35648/0 (socket says 172.21.15.7:35648) 2023-03-03T18:40:57.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3259c700 1 -- 172.21.15.7:0/1444343530 learned_addr learned my addr 172.21.15.7:0/1444343530 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:40:57.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3259c700 1 -- 172.21.15.7:0/1444343530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340c1ac0 msgr2=0x7f8f340c1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:40:57.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3259c700 1 --2- 172.21.15.7:0/1444343530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340c1ac0 0x7f8f340c1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:57.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3259c700 1 -- 172.21.15.7:0/1444343530 >> v1:172.21.15.143:6789/0 conn(0x7f8f340aabc0 legacy=0x7f8f34071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:57.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3259c700 1 -- 172.21.15.7:0/1444343530 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3406f890 con 0x7f8f3414ae20 2023-03-03T18:40:57.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f32d9d700 1 --2- 172.21.15.7:0/1444343530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340c1ac0 0x7f8f340c1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:40:57.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3259c700 1 --2- 172.21.15.7:0/1444343530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f3414ae20 0x7f8f34154e70 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8f240098f0 tx=0x7f8f24009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3215da22a8f16c25 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:40:57.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3159a700 1 -- 172.21.15.7:0/1444343530 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24011830 con 0x7f8f3414ae20 2023-03-03T18:40:57.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3159a700 1 -- 172.21.15.7:0/1444343530 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f24011990 con 0x7f8f3414ae20 2023-03-03T18:40:57.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.365+0000 7f8f3159a700 1 -- 172.21.15.7:0/1444343530 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f24011d20 con 0x7f8f3414ae20 2023-03-03T18:40:57.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.366+0000 7f8f38aaa700 1 -- 172.21.15.7:0/1444343530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f3414ae20 msgr2=0x7f8f34154e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:57.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.366+0000 7f8f38aaa700 1 --2- 172.21.15.7:0/1444343530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f3414ae20 0x7f8f34154e70 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8f240098f0 tx=0x7f8f24009c00 comp rx=0 tx=0).stop 2023-03-03T18:40:57.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.366+0000 7f8f38aaa700 1 -- 172.21.15.7:0/1444343530 shutdown_connections 2023-03-03T18:40:57.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.366+0000 7f8f38aaa700 1 --2- 172.21.15.7:0/1444343530 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340c1ac0 0x7f8f340c1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:57.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.366+0000 7f8f38aaa700 1 --2- 172.21.15.7:0/1444343530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f3414ae20 0x7f8f34154e70 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:57.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.366+0000 7f8f38aaa700 1 -- 172.21.15.7:0/1444343530 >> 172.21.15.7:0/1444343530 conn(0x7f8f340c8900 msgr2=0x7f8f340c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:40:57.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.366+0000 7f8f38aaa700 1 -- 172.21.15.7:0/1444343530 shutdown_connections 2023-03-03T18:40:57.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.366+0000 7f8f38aaa700 1 -- 172.21.15.7:0/1444343530 wait complete. 2023-03-03T18:40:57.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.368+0000 7f8f38aaa700 1 Processor -- start 2023-03-03T18:40:57.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.368+0000 7f8f38aaa700 1 -- start start 2023-03-03T18:40:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f38aaa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340aabc0 0x7f8f341582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f38aaa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f340c1ac0 0x7f8f34147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f38aaa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f3414ae20 0x7f8f3414def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f38aaa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f341511d0 con 0x7f8f340c1ac0 2023-03-03T18:40:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f38aaa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f341508e0 con 0x7f8f3414ae20 2023-03-03T18:40:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f38aaa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f34150b80 con 0x7f8f340aabc0 2023-03-03T18:40:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f31d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f340c1ac0 0x7f8f34147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f31d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f340c1ac0 0x7f8f34147470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38156/0 (socket says 172.21.15.7:38156) 2023-03-03T18:40:57.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f31d9b700 1 -- 172.21.15.7:0/2200243420 learned_addr learned my addr 172.21.15.7:0/2200243420 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:40:57.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f3259c700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340aabc0 0x7f8f341582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:57.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f32d9d700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f3414ae20 0x7f8f3414def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:57.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f31d9b700 1 -- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340aabc0 msgr2=0x7f8f341582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:57.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f31d9b700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340aabc0 0x7f8f341582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:57.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f31d9b700 1 -- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f3414ae20 msgr2=0x7f8f3414def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:57.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f31d9b700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f3414ae20 0x7f8f3414def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:57.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f31d9b700 1 -- 172.21.15.7:0/2200243420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c009a70 con 0x7f8f340c1ac0 2023-03-03T18:40:57.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f3259c700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340aabc0 0x7f8f341582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:40:57.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f32d9d700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f3414ae20 0x7f8f3414def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:40:57.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.369+0000 7f8f31d9b700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f340c1ac0 0x7f8f34147470 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00c500 tx=0x7f8f1c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:40:57.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.370+0000 7f8f2affd700 1 -- 172.21.15.7:0/2200243420 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c005cb0 con 0x7f8f340c1ac0 2023-03-03T18:40:57.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.370+0000 7f8f2affd700 1 -- 172.21.15.7:0/2200243420 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1c005e10 con 0x7f8f340c1ac0 2023-03-03T18:40:57.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.370+0000 7f8f38aaa700 1 -- 172.21.15.7:0/2200243420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f24009550 con 0x7f8f340c1ac0 2023-03-03T18:40:57.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.370+0000 7f8f2affd700 1 -- 172.21.15.7:0/2200243420 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c0072d0 con 0x7f8f340c1ac0 2023-03-03T18:40:57.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.370+0000 7f8f38aaa700 1 -- 172.21.15.7:0/2200243420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f340aa430 con 0x7f8f340c1ac0 2023-03-03T18:40:57.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.371+0000 7f8f38aaa700 1 -- 172.21.15.7:0/2200243420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3406fb70 con 0x7f8f340c1ac0 2023-03-03T18:40:57.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.371+0000 7f8f2affd700 1 -- 172.21.15.7:0/2200243420 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8f1c019030 con 0x7f8f340c1ac0 2023-03-03T18:40:57.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.371+0000 7f8f2affd700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f140682f0 0x7f8f1406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:40:57.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.372+0000 7f8f2affd700 1 -- 172.21.15.7:0/2200243420 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8f1c085330 con 0x7f8f340c1ac0 2023-03-03T18:40:57.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.372+0000 7f8f3259c700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f140682f0 0x7f8f1406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:40:57.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.372+0000 7f8f3259c700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f140682f0 0x7f8f1406a7a0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009c40 tx=0x7f8f24008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:40:57.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.376+0000 7f8f2affd700 1 -- 172.21.15.7:0/2200243420 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8f1c04b5e0 con 0x7f8f340c1ac0 2023-03-03T18:40:57.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.616+0000 7f8f38aaa700 1 -- 172.21.15.7:0/2200243420 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f3414b1f0 con 0x7f8f340c1ac0 2023-03-03T18:40:57.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.617+0000 7f8f2affd700 1 -- 172.21.15.7:0/2200243420 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8f1c017090 con 0x7f8f340c1ac0 2023-03-03T18:40:57.617 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:40:57.617 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:40:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 -- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f140682f0 msgr2=0x7f8f1406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f140682f0 0x7f8f1406a7a0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f8f24009c40 tx=0x7f8f24008030 comp rx=0 tx=0).stop 2023-03-03T18:40:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 -- 172.21.15.7:0/2200243420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f340c1ac0 msgr2=0x7f8f34147470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:40:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f340c1ac0 0x7f8f34147470 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00c500 tx=0x7f8f1c00c8c0 comp rx=0 tx=0).stop 2023-03-03T18:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 -- 172.21.15.7:0/2200243420 shutdown_connections 2023-03-03T18:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f3414ae20 0x7f8f3414def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8f340aabc0 0x7f8f341582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8f140682f0 0x7f8f1406a7a0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 --2- 172.21.15.7:0/2200243420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f340c1ac0 0x7f8f34147470 unknown :-1 s=CLOSED pgs=3259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:40:57.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 -- 172.21.15.7:0/2200243420 >> 172.21.15.7:0/2200243420 conn(0x7f8f340c8900 msgr2=0x7f8f34152700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:40:57.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 -- 172.21.15.7:0/2200243420 shutdown_connections 2023-03-03T18:40:57.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:40:57.619+0000 7f8f38aaa700 1 -- 172.21.15.7:0/2200243420 wait complete. 2023-03-03T18:40:57.621 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:40:57.631 INFO:tasks.mds_thrash.fs.[a]:waiting for 91 secs before thrashing 2023-03-03T18:41:11.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:41:11.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:41:41.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:41:41.251 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:42:07.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:42:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.485+0000 7fc0338c6700 1 Processor -- start 2023-03-03T18:42:07.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.485+0000 7fc0338c6700 1 -- start start 2023-03-03T18:42:07.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc0338c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c0caa30 0x7fc02c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc0338c6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0c7310 con 0x7fc02c0c00e0 2023-03-03T18:42:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc0338c6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02c0bfd10 con 0x7fc02c154ff0 2023-03-03T18:42:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc0338c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c0bfa30 con 0x7fc02c0caa30 2023-03-03T18:42:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc030e61700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc02c0c00e0 0x7fc02c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60458/0 (socket says 172.21.15.7:60458) 2023-03-03T18:42:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc030e61700 1 -- 172.21.15.7:0/3868635085 learned_addr learned my addr 172.21.15.7:0/3868635085 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:42:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc031662700 1 --2- 172.21.15.7:0/3868635085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c0caa30 0x7fc02c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:07.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc023fff700 1 -- 172.21.15.7:0/3868635085 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828041883 0 0) 0x7fc02c0c7310 con 0x7fc02c0c00e0 2023-03-03T18:42:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc023fff700 1 -- 172.21.15.7:0/3868635085 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0180035b0 con 0x7fc02c0c00e0 2023-03-03T18:42:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc023fff700 1 -- 172.21.15.7:0/3868635085 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104988936 0 0) 0x7fc02c0bfd10 con 0x7fc02c154ff0 2023-03-03T18:42:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc023fff700 1 -- 172.21.15.7:0/3868635085 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc02c0c7310 con 0x7fc02c154ff0 2023-03-03T18:42:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc031662700 1 -- 172.21.15.7:0/3868635085 >> v1:172.21.15.143:6789/0 conn(0x7fc02c154ff0 legacy=0x7fc02c15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc031662700 1 -- 172.21.15.7:0/3868635085 >> v1:172.21.15.7:6789/0 conn(0x7fc02c0c00e0 legacy=0x7fc02c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc023fff700 1 -- 172.21.15.7:0/3868635085 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3180639596 0 0) 0x7fc0180035b0 con 0x7fc02c0c00e0 2023-03-03T18:42:07.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.486+0000 7fc031662700 1 -- 172.21.15.7:0/3868635085 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c154510 con 0x7fc02c0caa30 2023-03-03T18:42:07.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc031662700 1 --2- 172.21.15.7:0/3868635085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c0caa30 0x7fc02c0c5c60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc028009870 tx=0x7fc028009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b9f42643b04659a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:07.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc023fff700 1 -- 172.21.15.7:0/3868635085 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01c0017f0 con 0x7fc02c0caa30 2023-03-03T18:42:07.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc023fff700 1 -- 172.21.15.7:0/3868635085 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc028004030 con 0x7fc02c0caa30 2023-03-03T18:42:07.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc023fff700 1 -- 172.21.15.7:0/3868635085 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0280043c0 con 0x7fc02c0caa30 2023-03-03T18:42:07.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc0338c6700 1 -- 172.21.15.7:0/3868635085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c0caa30 msgr2=0x7fc02c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:07.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc0338c6700 1 --2- 172.21.15.7:0/3868635085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c0caa30 0x7fc02c0c5c60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc028009870 tx=0x7fc028009b80 comp rx=0 tx=0).stop 2023-03-03T18:42:07.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc0338c6700 1 -- 172.21.15.7:0/3868635085 shutdown_connections 2023-03-03T18:42:07.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc0338c6700 1 --2- 172.21.15.7:0/3868635085 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c0caa30 0x7fc02c0c5c60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:07.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc0338c6700 1 -- 172.21.15.7:0/3868635085 >> 172.21.15.7:0/3868635085 conn(0x7fc02c066e40 msgr2=0x7fc02c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:42:07.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc0338c6700 1 -- 172.21.15.7:0/3868635085 shutdown_connections 2023-03-03T18:42:07.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.487+0000 7fc0338c6700 1 -- 172.21.15.7:0/3868635085 wait complete. 2023-03-03T18:42:07.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc0338c6700 1 Processor -- start 2023-03-03T18:42:07.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc0338c6700 1 -- start start 2023-03-03T18:42:07.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc0338c6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c00e0 0x7fc02c147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:07.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc0338c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0caa30 0x7fc02c0c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:07.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc0338c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c154ff0 0x7fc02c146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:07.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc0338c6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c148450 con 0x7fc02c0c00e0 2023-03-03T18:42:07.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc0338c6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c147850 con 0x7fc02c0caa30 2023-03-03T18:42:07.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc0338c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc02c1ed740 con 0x7fc02c154ff0 2023-03-03T18:42:07.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc031662700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c00e0 0x7fc02c147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:07.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc031662700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c00e0 0x7fc02c147310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38250/0 (socket says 172.21.15.7:38250) 2023-03-03T18:42:07.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc031662700 1 -- 172.21.15.7:0/3413680356 learned_addr learned my addr 172.21.15.7:0/3413680356 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:42:07.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc030e61700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0caa30 0x7fc02c0c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:07.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.489+0000 7fc031e63700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c154ff0 0x7fc02c146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:07.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc031662700 1 -- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c154ff0 msgr2=0x7fc02c146660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:07.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc031662700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c154ff0 0x7fc02c146660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:07.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc031662700 1 -- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0caa30 msgr2=0x7fc02c0c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:07.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc031662700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0caa30 0x7fc02c0c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:07.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc031662700 1 -- 172.21.15.7:0/3413680356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0280094a0 con 0x7fc02c0c00e0 2023-03-03T18:42:07.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc031662700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c00e0 0x7fc02c147310 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7fc028000c00 tx=0x7fc028004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:07.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc021ffb700 1 -- 172.21.15.7:0/3413680356 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028003e60 con 0x7fc02c0c00e0 2023-03-03T18:42:07.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc0338c6700 1 -- 172.21.15.7:0/3413680356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02c154ad0 con 0x7fc02c0c00e0 2023-03-03T18:42:07.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc021ffb700 1 -- 172.21.15.7:0/3413680356 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02801f070 con 0x7fc02c0c00e0 2023-03-03T18:42:07.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc021ffb700 1 -- 172.21.15.7:0/3413680356 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc028011850 con 0x7fc02c0c00e0 2023-03-03T18:42:07.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.490+0000 7fc0338c6700 1 -- 172.21.15.7:0/3413680356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02c1547f0 con 0x7fc02c0c00e0 2023-03-03T18:42:07.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.491+0000 7fc0338c6700 1 -- 172.21.15.7:0/3413680356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02c0bf650 con 0x7fc02c0c00e0 2023-03-03T18:42:07.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.492+0000 7fc021ffb700 1 -- 172.21.15.7:0/3413680356 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc028017020 con 0x7fc02c0c00e0 2023-03-03T18:42:07.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.492+0000 7fc021ffb700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc010068320 0x7fc01006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:07.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.492+0000 7fc021ffb700 1 -- 172.21.15.7:0/3413680356 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc028021070 con 0x7fc02c0c00e0 2023-03-03T18:42:07.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.493+0000 7fc030e61700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc010068320 0x7fc01006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:07.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.493+0000 7fc030e61700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc010068320 0x7fc01006a7d0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fc02c1e0b00 tx=0x7fc01c000bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:07.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.497+0000 7fc021ffb700 1 -- 172.21.15.7:0/3413680356 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc028053ce0 con 0x7fc02c0c00e0 2023-03-03T18:42:07.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.741+0000 7fc0338c6700 1 -- 172.21.15.7:0/3413680356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc02c074bb0 con 0x7fc02c0c00e0 2023-03-03T18:42:07.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.742+0000 7fc021ffb700 1 -- 172.21.15.7:0/3413680356 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc028057300 con 0x7fc02c0c00e0 2023-03-03T18:42:07.742 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:42:07.743 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:42:07.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 -- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc010068320 msgr2=0x7fc01006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:07.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc010068320 0x7fc01006a7d0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fc02c1e0b00 tx=0x7fc01c000bc0 comp rx=0 tx=0).stop 2023-03-03T18:42:07.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 -- 172.21.15.7:0/3413680356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c00e0 msgr2=0x7fc02c147310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:07.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c00e0 0x7fc02c147310 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7fc028000c00 tx=0x7fc028004750 comp rx=0 tx=0).stop 2023-03-03T18:42:07.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 -- 172.21.15.7:0/3413680356 shutdown_connections 2023-03-03T18:42:07.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc02c154ff0 0x7fc02c146660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:07.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc010068320 0x7fc01006a7d0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:07.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc02c0c00e0 0x7fc02c147310 unknown :-1 s=CLOSED pgs=3261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:07.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 --2- 172.21.15.7:0/3413680356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc02c0caa30 0x7fc02c0c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:07.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 -- 172.21.15.7:0/3413680356 >> 172.21.15.7:0/3413680356 conn(0x7fc02c066e40 msgr2=0x7fc02c0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:42:07.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 -- 172.21.15.7:0/3413680356 shutdown_connections 2023-03-03T18:42:07.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:07.744+0000 7fc0338c6700 1 -- 172.21.15.7:0/3413680356 wait complete. 2023-03-03T18:42:07.746 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:42:07.756 INFO:tasks.mds_thrash.fs.[b]:waiting for 43 secs before thrashing 2023-03-03T18:42:11.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:42:11.355 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:42:28.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:42:28.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.772+0000 7f4e71ea6700 1 Processor -- start 2023-03-03T18:42:28.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.772+0000 7f4e71ea6700 1 -- start start 2023-03-03T18:42:28.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.772+0000 7f4e71ea6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e6c0caa30 0x7f4e6c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:28.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.772+0000 7f4e71ea6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e6c0c00e0 0x7f4e6c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:28.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.772+0000 7f4e71ea6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6c0c7310 con 0x7f4e6c0cae00 2023-03-03T18:42:28.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.772+0000 7f4e71ea6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6c0cb240 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.772+0000 7f4e71ea6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e6c0bfd10 con 0x7f4e6c0caa30 2023-03-03T18:42:28.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e6c0c00e0 0x7f4e6c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:28.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e6c0c00e0 0x7f4e6c154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38270/0 (socket says 172.21.15.7:38270) 2023-03-03T18:42:28.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6affd700 1 -- 172.21.15.7:0/783145077 learned_addr learned my addr 172.21.15.7:0/783145077 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:42:28.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6b7fe700 1 --2- 172.21.15.7:0/783145077 >> v2:172.21.15.143:3300/0 conn(0x7f4e6c0caa30 0x7f4e6c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:28.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6affd700 1 -- 172.21.15.7:0/783145077 >> v2:172.21.15.143:3300/0 conn(0x7f4e6c0caa30 msgr2=0x7f4e6c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:42:28.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6affd700 1 --2- 172.21.15.7:0/783145077 >> v2:172.21.15.143:3300/0 conn(0x7f4e6c0caa30 0x7f4e6c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:28.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6affd700 1 -- 172.21.15.7:0/783145077 >> v1:172.21.15.143:6789/0 conn(0x7f4e6c0cae00 legacy=0x7f4e6c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:28.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6affd700 1 -- 172.21.15.7:0/783145077 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6c0bfa30 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6affd700 1 --2- 172.21.15.7:0/783145077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 0x7f4e6c154ff0 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0c81b0 tx=0x7f4e60009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5b54795aa1fae7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6a7fc700 1 -- 172.21.15.7:0/783145077 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e60005e30 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6a7fc700 1 -- 172.21.15.7:0/783145077 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e60004030 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.773+0000 7f4e6a7fc700 1 -- 172.21.15.7:0/783145077 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e600043e0 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.774+0000 7f4e71ea6700 1 -- 172.21.15.7:0/783145077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 msgr2=0x7f4e6c154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.774+0000 7f4e71ea6700 1 --2- 172.21.15.7:0/783145077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 0x7f4e6c154ff0 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0c81b0 tx=0x7f4e60009bd0 comp rx=0 tx=0).stop 2023-03-03T18:42:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.774+0000 7f4e71ea6700 1 -- 172.21.15.7:0/783145077 shutdown_connections 2023-03-03T18:42:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.774+0000 7f4e71ea6700 1 --2- 172.21.15.7:0/783145077 >> v2:172.21.15.143:3300/0 conn(0x7f4e6c0caa30 0x7f4e6c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.774+0000 7f4e71ea6700 1 --2- 172.21.15.7:0/783145077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 0x7f4e6c154ff0 unknown :-1 s=CLOSED pgs=3262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:28.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.774+0000 7f4e71ea6700 1 -- 172.21.15.7:0/783145077 >> 172.21.15.7:0/783145077 conn(0x7f4e6c066e40 msgr2=0x7f4e6c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:42:28.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.774+0000 7f4e71ea6700 1 -- 172.21.15.7:0/783145077 shutdown_connections 2023-03-03T18:42:28.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.774+0000 7f4e71ea6700 1 -- 172.21.15.7:0/783145077 wait complete. 2023-03-03T18:42:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.775+0000 7f4e71ea6700 1 Processor -- start 2023-03-03T18:42:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.775+0000 7f4e71ea6700 1 -- start start 2023-03-03T18:42:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e71ea6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 0x7f4e6c155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e71ea6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6c0caa30 0x7f4e6c157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e71ea6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6c0cae00 0x7f4e6c157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:28.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e71ea6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c1582b0 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e71ea6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c146d30 con 0x7f4e6c0cae00 2023-03-03T18:42:28.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e71ea6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c1ea790 con 0x7f4e6c0caa30 2023-03-03T18:42:28.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 0x7f4e6c155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:28.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 0x7f4e6c155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38276/0 (socket says 172.21.15.7:38276) 2023-03-03T18:42:28.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6b7fe700 1 -- 172.21.15.7:0/3647301645 learned_addr learned my addr 172.21.15.7:0/3647301645 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:42:28.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6bfff700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6c0cae00 0x7f4e6c157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:28.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6affd700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6c0caa30 0x7f4e6c157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:28.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6b7fe700 1 -- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6c0caa30 msgr2=0x7f4e6c157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:28.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6b7fe700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6c0caa30 0x7f4e6c157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:28.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6b7fe700 1 -- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6c0cae00 msgr2=0x7f4e6c157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:28.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6b7fe700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6c0cae00 0x7f4e6c157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:28.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6b7fe700 1 -- 172.21.15.7:0/3647301645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e60009500 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.776+0000 7f4e6affd700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6c0caa30 0x7f4e6c157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:42:28.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.777+0000 7f4e6b7fe700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 0x7f4e6c155c50 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00e810 tx=0x7f4e5c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:28.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.777+0000 7f4e53fff700 1 -- 172.21.15.7:0/3647301645 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c013610 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.777+0000 7f4e53fff700 1 -- 172.21.15.7:0/3647301645 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c0137b0 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.777+0000 7f4e71ea6700 1 -- 172.21.15.7:0/3647301645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c1547f0 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.777+0000 7f4e53fff700 1 -- 172.21.15.7:0/3647301645 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c013b60 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.777+0000 7f4e71ea6700 1 -- 172.21.15.7:0/3647301645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c154510 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.778+0000 7f4e71ea6700 1 -- 172.21.15.7:0/3647301645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c154ad0 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.778+0000 7f4e53fff700 1 -- 172.21.15.7:0/3647301645 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4e5c023070 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.779+0000 7f4e53fff700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e580682f0 0x7f4e5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:28.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.779+0000 7f4e53fff700 1 -- 172.21.15.7:0/3647301645 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4e5c085ed0 con 0x7f4e6c0c00e0 2023-03-03T18:42:28.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.779+0000 7f4e6affd700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e580682f0 0x7f4e5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:28.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.780+0000 7f4e6affd700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e580682f0 0x7f4e5806a7a0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0c81b0 tx=0x7f4e60008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:28.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:28.783+0000 7f4e53fff700 1 -- 172.21.15.7:0/3647301645 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4e5c052180 con 0x7f4e6c0c00e0 2023-03-03T18:42:29.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.021+0000 7f4e71ea6700 1 -- 172.21.15.7:0/3647301645 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e6c074bb0 con 0x7f4e6c0c00e0 2023-03-03T18:42:29.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.022+0000 7f4e53fff700 1 -- 172.21.15.7:0/3647301645 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4e5c0557a0 con 0x7f4e6c0c00e0 2023-03-03T18:42:29.022 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:42:29.022 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:42:29.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 -- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e580682f0 msgr2=0x7f4e5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:29.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e580682f0 0x7f4e5806a7a0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0c81b0 tx=0x7f4e60008070 comp rx=0 tx=0).stop 2023-03-03T18:42:29.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 -- 172.21.15.7:0/3647301645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 msgr2=0x7f4e6c155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:29.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 0x7f4e6c155c50 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00e810 tx=0x7f4e5c00eb20 comp rx=0 tx=0).stop 2023-03-03T18:42:29.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 -- 172.21.15.7:0/3647301645 shutdown_connections 2023-03-03T18:42:29.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6c0cae00 0x7f4e6c157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:29.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e580682f0 0x7f4e5806a7a0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:29.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e6c0c00e0 0x7f4e6c155c50 unknown :-1 s=CLOSED pgs=3263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:29.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 --2- 172.21.15.7:0/3647301645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6c0caa30 0x7f4e6c157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:29.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 -- 172.21.15.7:0/3647301645 >> 172.21.15.7:0/3647301645 conn(0x7f4e6c066e40 msgr2=0x7f4e6c0736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:42:29.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 -- 172.21.15.7:0/3647301645 shutdown_connections 2023-03-03T18:42:29.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:29.024+0000 7f4e71ea6700 1 -- 172.21.15.7:0/3647301645 wait complete. 2023-03-03T18:42:29.026 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:42:29.036 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2023-03-03T18:42:41.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:42:41.434 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:42:50.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:42:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.896+0000 7f5c6622c700 1 Processor -- start 2023-03-03T18:42:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.896+0000 7f5c6622c700 1 -- start start 2023-03-03T18:42:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c6622c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5c6006bea0 0x7f5c6006c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c6622c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c60157dc0 0x7f5c60158240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c6622c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6006d780 con 0x7f5c60151ab0 2023-03-03T18:42:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c6622c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c6006c830 con 0x7f5c6006bea0 2023-03-03T18:42:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c6622c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6006cb70 con 0x7f5c60157dc0 2023-03-03T18:42:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5c6006bea0 0x7f5c6006c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5c6006bea0 0x7f5c6006c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38306/0 (socket says 172.21.15.7:38306) 2023-03-03T18:42:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5effd700 1 -- 172.21.15.7:0/2719717441 learned_addr learned my addr 172.21.15.7:0/2719717441 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:42:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5ffff700 1 --2- 172.21.15.7:0/2719717441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c60157dc0 0x7f5c60158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5effd700 1 -- 172.21.15.7:0/2719717441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c60157dc0 msgr2=0x7f5c60158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:42:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5effd700 1 --2- 172.21.15.7:0/2719717441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c60157dc0 0x7f5c60158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5effd700 1 -- 172.21.15.7:0/2719717441 >> v1:172.21.15.143:6789/0 conn(0x7f5c60151ab0 legacy=0x7f5c600736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5effd700 1 -- 172.21.15.7:0/2719717441 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6006ce50 con 0x7f5c6006bea0 2023-03-03T18:42:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5effd700 1 --2- 172.21.15.7:0/2719717441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c6006bea0 0x7f5c6006c2f0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f5c540098f0 tx=0x7f5c54009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a133f3fe31a807 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5e7fc700 1 -- 172.21.15.7:0/2719717441 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5400b8a0 con 0x7f5c6006bea0 2023-03-03T18:42:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5e7fc700 1 -- 172.21.15.7:0/2719717441 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5400ba00 con 0x7f5c6006bea0 2023-03-03T18:42:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.897+0000 7f5c5e7fc700 1 -- 172.21.15.7:0/2719717441 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5400bd90 con 0x7f5c6006bea0 2023-03-03T18:42:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.898+0000 7f5c6622c700 1 -- 172.21.15.7:0/2719717441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c6006bea0 msgr2=0x7f5c6006c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.898+0000 7f5c6622c700 1 --2- 172.21.15.7:0/2719717441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c6006bea0 0x7f5c6006c2f0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f5c540098f0 tx=0x7f5c54009c00 comp rx=0 tx=0).stop 2023-03-03T18:42:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.898+0000 7f5c6622c700 1 -- 172.21.15.7:0/2719717441 shutdown_connections 2023-03-03T18:42:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.898+0000 7f5c6622c700 1 --2- 172.21.15.7:0/2719717441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c60157dc0 0x7f5c60158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.898+0000 7f5c6622c700 1 --2- 172.21.15.7:0/2719717441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c6006bea0 0x7f5c6006c2f0 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.898+0000 7f5c6622c700 1 -- 172.21.15.7:0/2719717441 >> 172.21.15.7:0/2719717441 conn(0x7f5c600a94d0 msgr2=0x7f5c600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:42:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.898+0000 7f5c6622c700 1 -- 172.21.15.7:0/2719717441 shutdown_connections 2023-03-03T18:42:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.898+0000 7f5c6622c700 1 -- 172.21.15.7:0/2719717441 wait complete. 2023-03-03T18:42:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.899+0000 7f5c6622c700 1 Processor -- start 2023-03-03T18:42:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.899+0000 7f5c6622c700 1 -- start start 2023-03-03T18:42:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c6622c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c6006bea0 0x7f5c6014e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c6622c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c60151ab0 0x7f5c6014ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c6622c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c60157dc0 0x7f5c60145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c6622c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601ee350 con 0x7f5c60151ab0 2023-03-03T18:42:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c6622c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601eddd0 con 0x7f5c60157dc0 2023-03-03T18:42:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c6622c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c600c5880 con 0x7f5c6006bea0 2023-03-03T18:42:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c5effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c60151ab0 0x7f5c6014ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c5effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c60151ab0 0x7f5c6014ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38312/0 (socket says 172.21.15.7:38312) 2023-03-03T18:42:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c5effd700 1 -- 172.21.15.7:0/4288565695 learned_addr learned my addr 172.21.15.7:0/4288565695 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:42:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c5f7fe700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c6006bea0 0x7f5c6014e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c5ffff700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c60157dc0 0x7f5c60145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c5effd700 1 -- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c6006bea0 msgr2=0x7f5c6014e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c5effd700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c6006bea0 0x7f5c6014e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c5effd700 1 -- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c60157dc0 msgr2=0x7f5c60145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.900+0000 7f5c5effd700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c60157dc0 0x7f5c60145ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:50.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.901+0000 7f5c5effd700 1 -- 172.21.15.7:0/4288565695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c54009550 con 0x7f5c60151ab0 2023-03-03T18:42:50.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.901+0000 7f5c5ffff700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c60157dc0 0x7f5c60145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:42:50.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.901+0000 7f5c5effd700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c60151ab0 0x7f5c6014ae90 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f5c54000c00 tx=0x7f5c54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:50.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.901+0000 7f5c47fff700 1 -- 172.21.15.7:0/4288565695 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c54004380 con 0x7f5c60151ab0 2023-03-03T18:42:50.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.901+0000 7f5c47fff700 1 -- 172.21.15.7:0/4288565695 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5401f070 con 0x7f5c60151ab0 2023-03-03T18:42:50.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.901+0000 7f5c6622c700 1 -- 172.21.15.7:0/4288565695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c6006d410 con 0x7f5c60151ab0 2023-03-03T18:42:50.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.901+0000 7f5c47fff700 1 -- 172.21.15.7:0/4288565695 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c540036a0 con 0x7f5c60151ab0 2023-03-03T18:42:50.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.901+0000 7f5c6622c700 1 -- 172.21.15.7:0/4288565695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6006d130 con 0x7f5c60151ab0 2023-03-03T18:42:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.902+0000 7f5c6622c700 1 -- 172.21.15.7:0/4288565695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c60073e00 con 0x7f5c60151ab0 2023-03-03T18:42:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.902+0000 7f5c47fff700 1 -- 172.21.15.7:0/4288565695 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5c5402a020 con 0x7f5c60151ab0 2023-03-03T18:42:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.903+0000 7f5c47fff700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c4c0682f0 0x7f5c4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.903+0000 7f5c47fff700 1 -- 172.21.15.7:0/4288565695 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5c54087660 con 0x7f5c60151ab0 2023-03-03T18:42:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.903+0000 7f5c5f7fe700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c4c0682f0 0x7f5c4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:50.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.904+0000 7f5c5f7fe700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c4c0682f0 0x7f5c4c06a7a0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f5c6014f130 tx=0x7f5c50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:50.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:50.907+0000 7f5c47fff700 1 -- 172.21.15.7:0/4288565695 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5c54053a00 con 0x7f5c60151ab0 2023-03-03T18:42:51.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.144+0000 7f5c6622c700 1 -- 172.21.15.7:0/4288565695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c60074bb0 con 0x7f5c60151ab0 2023-03-03T18:42:51.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.145+0000 7f5c47fff700 1 -- 172.21.15.7:0/4288565695 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5c54057020 con 0x7f5c60151ab0 2023-03-03T18:42:51.145 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:42:51.145 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:42:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 -- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c4c0682f0 msgr2=0x7f5c4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c4c0682f0 0x7f5c4c06a7a0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f5c6014f130 tx=0x7f5c50009040 comp rx=0 tx=0).stop 2023-03-03T18:42:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 -- 172.21.15.7:0/4288565695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c60151ab0 msgr2=0x7f5c6014ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c60151ab0 0x7f5c6014ae90 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f5c54000c00 tx=0x7f5c54004750 comp rx=0 tx=0).stop 2023-03-03T18:42:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 -- 172.21.15.7:0/4288565695 shutdown_connections 2023-03-03T18:42:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c60157dc0 0x7f5c60145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c6006bea0 0x7f5c6014e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c4c0682f0 0x7f5c4c06a7a0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 --2- 172.21.15.7:0/4288565695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c60151ab0 0x7f5c6014ae90 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 -- 172.21.15.7:0/4288565695 >> 172.21.15.7:0/4288565695 conn(0x7f5c600a94d0 msgr2=0x7f5c60150dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:42:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 -- 172.21.15.7:0/4288565695 shutdown_connections 2023-03-03T18:42:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:51.147+0000 7f5c6622c700 1 -- 172.21.15.7:0/4288565695 wait complete. 2023-03-03T18:42:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:42:51.159 INFO:tasks.mds_thrash.fs.[b]:waiting for 69 secs before thrashing 2023-03-03T18:42:56.037 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:42:56.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.175+0000 7f8816513700 1 Processor -- start 2023-03-03T18:42:56.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.175+0000 7f8816513700 1 -- start start 2023-03-03T18:42:56.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f8816513700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f88100aae30 0x7f88100beab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:56.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f8816513700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88100beff0 0x7f88100c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:56.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f8816513700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100c49e0 0x7f881006e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:56.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f8816513700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88100c4de0 con 0x7f88100beff0 2023-03-03T18:42:56.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f8816513700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88100c3020 con 0x7f88100aae30 2023-03-03T18:42:56.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f8816513700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88101552d0 con 0x7f88100c49e0 2023-03-03T18:42:56.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f880f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88100beff0 0x7f88100c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:56.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f880f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f88100beff0 0x7f88100c44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38320/0 (socket says 172.21.15.7:38320) 2023-03-03T18:42:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f880f7fe700 1 -- 172.21.15.7:0/1734651549 learned_addr learned my addr 172.21.15.7:0/1734651549 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:42:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f880ffff700 1 --2- 172.21.15.7:0/1734651549 >> v2:172.21.15.143:3300/0 conn(0x7f88100aae30 0x7f88100beab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f8814ab0700 1 --2- 172.21.15.7:0/1734651549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100c49e0 0x7f881006e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f880f7fe700 1 -- 172.21.15.7:0/1734651549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100c49e0 msgr2=0x7f881006e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:42:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f880f7fe700 1 --2- 172.21.15.7:0/1734651549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100c49e0 0x7f881006e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f880f7fe700 1 -- 172.21.15.7:0/1734651549 >> v2:172.21.15.143:3300/0 conn(0x7f88100aae30 msgr2=0x7f88100beab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:42:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f880f7fe700 1 --2- 172.21.15.7:0/1734651549 >> v2:172.21.15.143:3300/0 conn(0x7f88100aae30 0x7f88100beab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f880f7fe700 1 -- 172.21.15.7:0/1734651549 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88101546e0 con 0x7f88100beff0 2023-03-03T18:42:56.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f8814ab0700 1 --2- 172.21.15.7:0/1734651549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100c49e0 0x7f881006e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:42:56.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.176+0000 7f880ffff700 1 --2- 172.21.15.7:0/1734651549 >> v2:172.21.15.143:3300/0 conn(0x7f88100aae30 0x7f88100beab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:42:56.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f880f7fe700 1 --2- 172.21.15.7:0/1734651549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100beff0 0x7f88100c44a0 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f8810066d70 tx=0x7f8804009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22578c20ba2e197 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:56.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f880effd700 1 -- 172.21.15.7:0/1734651549 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880400b860 con 0x7f88100beff0 2023-03-03T18:42:56.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f880effd700 1 -- 172.21.15.7:0/1734651549 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880400b9c0 con 0x7f88100beff0 2023-03-03T18:42:56.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f880effd700 1 -- 172.21.15.7:0/1734651549 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880400bd50 con 0x7f88100beff0 2023-03-03T18:42:56.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f8816513700 1 -- 172.21.15.7:0/1734651549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100beff0 msgr2=0x7f88100c44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:56.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f8816513700 1 --2- 172.21.15.7:0/1734651549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100beff0 0x7f88100c44a0 secure :-1 s=READY pgs=3266 cs=0 l=1 rev1=1 crypto rx=0x7f8810066d70 tx=0x7f8804009c00 comp rx=0 tx=0).stop 2023-03-03T18:42:56.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f8816513700 1 -- 172.21.15.7:0/1734651549 shutdown_connections 2023-03-03T18:42:56.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f8816513700 1 --2- 172.21.15.7:0/1734651549 >> v2:172.21.15.143:3300/0 conn(0x7f88100aae30 0x7f88100beab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f8816513700 1 --2- 172.21.15.7:0/1734651549 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100c49e0 0x7f881006e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f8816513700 1 --2- 172.21.15.7:0/1734651549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100beff0 0x7f88100c44a0 unknown :-1 s=CLOSED pgs=3266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f8816513700 1 -- 172.21.15.7:0/1734651549 >> 172.21.15.7:0/1734651549 conn(0x7f88100c8510 msgr2=0x7f88100c8910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:42:56.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f8816513700 1 -- 172.21.15.7:0/1734651549 shutdown_connections 2023-03-03T18:42:56.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.177+0000 7f8816513700 1 -- 172.21.15.7:0/1734651549 wait complete. 2023-03-03T18:42:56.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.179+0000 7f8816513700 1 Processor -- start 2023-03-03T18:42:56.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.179+0000 7f8816513700 1 -- start start 2023-03-03T18:42:56.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.179+0000 7f8816513700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100aae30 0x7f881014fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:56.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f8816513700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100beff0 0x7f881014c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:56.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f8816513700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88100c49e0 0x7f881014c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:56.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f8816513700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100ac180 con 0x7f88100aae30 2023-03-03T18:42:56.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f8816513700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88101ea6a0 con 0x7f88100c49e0 2023-03-03T18:42:56.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f8816513700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88101ea980 con 0x7f88100beff0 2023-03-03T18:42:56.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100aae30 0x7f881014fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:56.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100aae30 0x7f881014fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38326/0 (socket says 172.21.15.7:38326) 2023-03-03T18:42:56.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880ffff700 1 -- 172.21.15.7:0/2243318402 learned_addr learned my addr 172.21.15.7:0/2243318402 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:42:56.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880f7fe700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100beff0 0x7f881014c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:56.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f8814ab0700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88100c49e0 0x7f881014c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:56.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880ffff700 1 -- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100beff0 msgr2=0x7f881014c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:56.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880ffff700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100beff0 0x7f881014c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880ffff700 1 -- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88100c49e0 msgr2=0x7f881014c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:56.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880ffff700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88100c49e0 0x7f881014c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880ffff700 1 -- 172.21.15.7:0/2243318402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8804009550 con 0x7f88100aae30 2023-03-03T18:42:56.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880f7fe700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100beff0 0x7f881014c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:42:56.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f8814ab0700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88100c49e0 0x7f881014c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:42:56.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880ffff700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100aae30 0x7f881014fd40 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f880000d800 tx=0x7f880000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:56.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880cff9700 1 -- 172.21.15.7:0/2243318402 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800004030 con 0x7f88100aae30 2023-03-03T18:42:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880cff9700 1 -- 172.21.15.7:0/2243318402 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8800004190 con 0x7f88100aae30 2023-03-03T18:42:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f8816513700 1 -- 172.21.15.7:0/2243318402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8810154c80 con 0x7f88100aae30 2023-03-03T18:42:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.180+0000 7f880cff9700 1 -- 172.21.15.7:0/2243318402 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800004520 con 0x7f88100aae30 2023-03-03T18:42:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.181+0000 7f8816513700 1 -- 172.21.15.7:0/2243318402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88101549a0 con 0x7f88100aae30 2023-03-03T18:42:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.182+0000 7f8816513700 1 -- 172.21.15.7:0/2243318402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8810154f60 con 0x7f88100aae30 2023-03-03T18:42:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.182+0000 7f880cff9700 1 -- 172.21.15.7:0/2243318402 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f880001a020 con 0x7f88100aae30 2023-03-03T18:42:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.182+0000 7f880cff9700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87fc0682f0 0x7f87fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:42:56.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.183+0000 7f880cff9700 1 -- 172.21.15.7:0/2243318402 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f88000865d0 con 0x7f88100aae30 2023-03-03T18:42:56.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.183+0000 7f880f7fe700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87fc0682f0 0x7f87fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:42:56.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.183+0000 7f880f7fe700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87fc0682f0 0x7f87fc06a7a0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f88100be6a0 tx=0x7f8804008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:42:56.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.187+0000 7f880cff9700 1 -- 172.21.15.7:0/2243318402 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8800052970 con 0x7f88100aae30 2023-03-03T18:42:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.424+0000 7f8816513700 1 -- 172.21.15.7:0/2243318402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88100bece0 con 0x7f88100aae30 2023-03-03T18:42:56.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.424+0000 7f880cff9700 1 -- 172.21.15.7:0/2243318402 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8800055f90 con 0x7f88100aae30 2023-03-03T18:42:56.426 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:42:56.426 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:42:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.426+0000 7f8816513700 1 -- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87fc0682f0 msgr2=0x7f87fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:56.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.426+0000 7f8816513700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87fc0682f0 0x7f87fc06a7a0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f88100be6a0 tx=0x7f8804008030 comp rx=0 tx=0).stop 2023-03-03T18:42:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.426+0000 7f8816513700 1 -- 172.21.15.7:0/2243318402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100aae30 msgr2=0x7f881014fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:42:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.426+0000 7f8816513700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100aae30 0x7f881014fd40 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f880000d800 tx=0x7f880000dbc0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.426+0000 7f8816513700 1 -- 172.21.15.7:0/2243318402 shutdown_connections 2023-03-03T18:42:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.426+0000 7f8816513700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88100c49e0 0x7f881014c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.426+0000 7f8816513700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87fc0682f0 0x7f87fc06a7a0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.426+0000 7f8816513700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f88100aae30 0x7f881014fd40 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.426+0000 7f8816513700 1 --2- 172.21.15.7:0/2243318402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f88100beff0 0x7f881014c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:42:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.426+0000 7f8816513700 1 -- 172.21.15.7:0/2243318402 >> 172.21.15.7:0/2243318402 conn(0x7f88100c8510 msgr2=0x7f88100a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:42:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.427+0000 7f8816513700 1 -- 172.21.15.7:0/2243318402 shutdown_connections 2023-03-03T18:42:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:42:56.427+0000 7f8816513700 1 -- 172.21.15.7:0/2243318402 wait complete. 2023-03-03T18:42:56.429 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:42:56.439 INFO:tasks.mds_thrash.fs.[a]:waiting for 111 secs before thrashing 2023-03-03T18:43:11.504 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:43:11.506 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:43:41.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:43:41.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:44:00.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:44:00.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.298+0000 7ff0d59bb700 1 Processor -- start 2023-03-03T18:44:00.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.298+0000 7ff0d59bb700 1 -- start start 2023-03-03T18:44:00.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.298+0000 7ff0d59bb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff0d00c7a30 0x7ff0d014e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:00.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.298+0000 7ff0d59bb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0d006b6f0 0x7ff0d014e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:00.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.298+0000 7ff0d59bb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d0157bb0 con 0x7ff0d00c7e00 2023-03-03T18:44:00.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.298+0000 7ff0d59bb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d006ab00 con 0x7ff0d006b6f0 2023-03-03T18:44:00.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.298+0000 7ff0d59bb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d01545c0 con 0x7ff0d00c7a30 2023-03-03T18:44:00.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0cf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0d006b6f0 0x7ff0d014e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:00.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0ce7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff0d00c7e00 0x7ff0d0154a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60594/0 (socket says 172.21.15.7:60594) 2023-03-03T18:44:00.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0ce7fc700 1 -- 172.21.15.7:0/561562825 learned_addr learned my addr 172.21.15.7:0/561562825 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:44:00.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0ceffd700 1 --2- 172.21.15.7:0/561562825 >> v2:172.21.15.143:3300/0 conn(0x7ff0d00c7a30 0x7ff0d014e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:00.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0cf7fe700 1 -- 172.21.15.7:0/561562825 >> v2:172.21.15.143:3300/0 conn(0x7ff0d00c7a30 msgr2=0x7ff0d014e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:44:00.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0cf7fe700 1 --2- 172.21.15.7:0/561562825 >> v2:172.21.15.143:3300/0 conn(0x7ff0d00c7a30 0x7ff0d014e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:00.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0cf7fe700 1 -- 172.21.15.7:0/561562825 >> v1:172.21.15.7:6789/0 conn(0x7ff0d00c7e00 legacy=0x7ff0d0154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:00.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0cdffb700 1 -- 172.21.15.7:0/561562825 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909246414 0 0) 0x7ff0d0157bb0 con 0x7ff0d00c7e00 2023-03-03T18:44:00.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0cf7fe700 1 -- 172.21.15.7:0/561562825 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d006a2f0 con 0x7ff0d006b6f0 2023-03-03T18:44:00.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0cf7fe700 1 --2- 172.21.15.7:0/561562825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d006b6f0 0x7ff0d014e6a0 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7ff0d008c2a0 tx=0x7ff0c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff14f7a3f6cc5931 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:44:00.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0cdffb700 1 -- 172.21.15.7:0/561562825 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c800e070 con 0x7ff0d006b6f0 2023-03-03T18:44:00.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0cdffb700 1 -- 172.21.15.7:0/561562825 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c800bbd0 con 0x7ff0d006b6f0 2023-03-03T18:44:00.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.299+0000 7ff0cdffb700 1 -- 172.21.15.7:0/561562825 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8012430 con 0x7ff0d006b6f0 2023-03-03T18:44:00.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.300+0000 7ff0d59bb700 1 -- 172.21.15.7:0/561562825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d006b6f0 msgr2=0x7ff0d014e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:00.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.300+0000 7ff0d59bb700 1 --2- 172.21.15.7:0/561562825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d006b6f0 0x7ff0d014e6a0 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7ff0d008c2a0 tx=0x7ff0c800b330 comp rx=0 tx=0).stop 2023-03-03T18:44:00.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.300+0000 7ff0d59bb700 1 -- 172.21.15.7:0/561562825 shutdown_connections 2023-03-03T18:44:00.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.300+0000 7ff0d59bb700 1 --2- 172.21.15.7:0/561562825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d006b6f0 0x7ff0d014e6a0 unknown :-1 s=CLOSED pgs=3269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:00.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.300+0000 7ff0d59bb700 1 --2- 172.21.15.7:0/561562825 >> v2:172.21.15.143:3300/0 conn(0x7ff0d00c7a30 0x7ff0d014e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:00.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.300+0000 7ff0d59bb700 1 -- 172.21.15.7:0/561562825 >> 172.21.15.7:0/561562825 conn(0x7ff0d0073d20 msgr2=0x7ff0d00716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:44:00.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.300+0000 7ff0d59bb700 1 -- 172.21.15.7:0/561562825 shutdown_connections 2023-03-03T18:44:00.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.300+0000 7ff0d59bb700 1 -- 172.21.15.7:0/561562825 wait complete. 2023-03-03T18:44:00.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.301+0000 7ff0d59bb700 1 Processor -- start 2023-03-03T18:44:00.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.301+0000 7ff0d59bb700 1 -- start start 2023-03-03T18:44:00.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0d59bb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d00c7a30 0x7ff0d00c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:00.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0d59bb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d00c7e00 0x7ff0d014d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:00.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0d59bb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d01ed800 0x7ff0d01ea0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:00.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0d59bb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d01edd00 con 0x7ff0d01ed800 2023-03-03T18:44:00.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0d59bb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d01e9c60 con 0x7ff0d00c7a30 2023-03-03T18:44:00.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0d59bb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d0150e30 con 0x7ff0d00c7e00 2023-03-03T18:44:00.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d01ed800 0x7ff0d01ea0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:00.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d01ed800 0x7ff0d01ea0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38390/0 (socket says 172.21.15.7:38390) 2023-03-03T18:44:00.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0cf7fe700 1 -- 172.21.15.7:0/2206841830 learned_addr learned my addr 172.21.15.7:0/2206841830 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:44:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0ceffd700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d00c7a30 0x7ff0d00c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0ce7fc700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d00c7e00 0x7ff0d014d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0cf7fe700 1 -- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d00c7e00 msgr2=0x7ff0d014d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.302+0000 7ff0cf7fe700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d00c7e00 0x7ff0d014d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.303+0000 7ff0cf7fe700 1 -- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d00c7a30 msgr2=0x7ff0d00c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.303+0000 7ff0cf7fe700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d00c7a30 0x7ff0d00c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:00.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.303+0000 7ff0cf7fe700 1 -- 172.21.15.7:0/2206841830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c8009a70 con 0x7ff0d01ed800 2023-03-03T18:44:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.303+0000 7ff0cf7fe700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d01ed800 0x7ff0d01ea0f0 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8014040 tx=0x7ff0c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:44:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.303+0000 7ff0b7fff700 1 -- 172.21.15.7:0/2206841830 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8012430 con 0x7ff0d01ed800 2023-03-03T18:44:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.303+0000 7ff0b7fff700 1 -- 172.21.15.7:0/2206841830 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c8020070 con 0x7ff0d01ed800 2023-03-03T18:44:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.303+0000 7ff0d59bb700 1 -- 172.21.15.7:0/2206841830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d006a540 con 0x7ff0d01ed800 2023-03-03T18:44:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.303+0000 7ff0b7fff700 1 -- 172.21.15.7:0/2206841830 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8004280 con 0x7ff0d01ed800 2023-03-03T18:44:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.303+0000 7ff0d59bb700 1 -- 172.21.15.7:0/2206841830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d00bfbb0 con 0x7ff0d01ed800 2023-03-03T18:44:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.304+0000 7ff0d59bb700 1 -- 172.21.15.7:0/2206841830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d006a820 con 0x7ff0d01ed800 2023-03-03T18:44:00.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.304+0000 7ff0b7fff700 1 -- 172.21.15.7:0/2206841830 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff0c80043e0 con 0x7ff0d01ed800 2023-03-03T18:44:00.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.305+0000 7ff0b7fff700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0bc0682f0 0x7ff0bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:00.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.305+0000 7ff0b7fff700 1 -- 172.21.15.7:0/2206841830 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff0c8022070 con 0x7ff0d01ed800 2023-03-03T18:44:00.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.305+0000 7ff0ceffd700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0bc0682f0 0x7ff0bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:00.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.306+0000 7ff0ceffd700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0bc0682f0 0x7ff0bc06a7a0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0006fd0 tx=0x7ff0c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:44:00.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.309+0000 7ff0b7fff700 1 -- 172.21.15.7:0/2206841830 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff0c8054d20 con 0x7ff0d01ed800 2023-03-03T18:44:00.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.549+0000 7ff0d59bb700 1 -- 172.21.15.7:0/2206841830 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0d0150c90 con 0x7ff0d01ed800 2023-03-03T18:44:00.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.550+0000 7ff0b7fff700 1 -- 172.21.15.7:0/2206841830 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff0d0150c90 con 0x7ff0d01ed800 2023-03-03T18:44:00.551 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:44:00.551 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:44:00.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.551+0000 7ff0d59bb700 1 -- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0bc0682f0 msgr2=0x7ff0bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:00.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0bc0682f0 0x7ff0bc06a7a0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0006fd0 tx=0x7ff0c0008040 comp rx=0 tx=0).stop 2023-03-03T18:44:00.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 -- 172.21.15.7:0/2206841830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d01ed800 msgr2=0x7ff0d01ea0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:00.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d01ed800 0x7ff0d01ea0f0 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8014040 tx=0x7ff0c8004ad0 comp rx=0 tx=0).stop 2023-03-03T18:44:00.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 -- 172.21.15.7:0/2206841830 shutdown_connections 2023-03-03T18:44:00.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0bc0682f0 0x7ff0bc06a7a0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:00.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0d01ed800 0x7ff0d01ea0f0 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:00.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0d00c7a30 0x7ff0d00c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:00.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 --2- 172.21.15.7:0/2206841830 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0d00c7e00 0x7ff0d014d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:00.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 -- 172.21.15.7:0/2206841830 >> 172.21.15.7:0/2206841830 conn(0x7ff0d006b6f0 msgr2=0x7ff0d014df30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:44:00.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 -- 172.21.15.7:0/2206841830 shutdown_connections 2023-03-03T18:44:00.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:00.552+0000 7ff0d59bb700 1 -- 172.21.15.7:0/2206841830 wait complete. 2023-03-03T18:44:00.554 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:44:00.564 INFO:tasks.mds_thrash.fs.[b]:waiting for 74 secs before thrashing 2023-03-03T18:44:11.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:44:11.698 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:44:41.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:44:41.787 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:44:47.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:44:47.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.580+0000 7f8998d66700 1 Processor -- start 2023-03-03T18:44:47.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.580+0000 7f8998d66700 1 -- start start 2023-03-03T18:44:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.581+0000 7f8998d66700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8994155390 0x7f89940c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.581+0000 7f8998d66700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f899406e020 0x7f8994071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.581+0000 7f8998d66700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8994072520 0x7f8994151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.581+0000 7f8998d66700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8994070b90 con 0x7f899406e020 2023-03-03T18:44:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.581+0000 7f8998d66700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8994151320 con 0x7f8994155390 2023-03-03T18:44:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.581+0000 7f8998d66700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89940c4780 con 0x7f8994072520 2023-03-03T18:44:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.581+0000 7f8991d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f899406e020 0x7f8994071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.581+0000 7f8991d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f899406e020 0x7f8994071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38460/0 (socket says 172.21.15.7:38460) 2023-03-03T18:44:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.581+0000 7f8991d9b700 1 -- 172.21.15.7:0/3668285833 learned_addr learned my addr 172.21.15.7:0/3668285833 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:44:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f8992d9d700 1 --2- 172.21.15.7:0/3668285833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8994072520 0x7f8994151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f899259c700 1 --2- 172.21.15.7:0/3668285833 >> v2:172.21.15.143:3300/0 conn(0x7f8994155390 0x7f89940c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f8991d9b700 1 -- 172.21.15.7:0/3668285833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8994072520 msgr2=0x7f8994151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:44:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f8991d9b700 1 --2- 172.21.15.7:0/3668285833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8994072520 0x7f8994151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f8991d9b700 1 -- 172.21.15.7:0/3668285833 >> v2:172.21.15.143:3300/0 conn(0x7f8994155390 msgr2=0x7f89940c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:44:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f8991d9b700 1 --2- 172.21.15.7:0/3668285833 >> v2:172.21.15.143:3300/0 conn(0x7f8994155390 0x7f89940c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f8991d9b700 1 -- 172.21.15.7:0/3668285833 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8994154910 con 0x7f899406e020 2023-03-03T18:44:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f899259c700 1 --2- 172.21.15.7:0/3668285833 >> v2:172.21.15.143:3300/0 conn(0x7f8994155390 0x7f89940c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:44:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f8991d9b700 1 --2- 172.21.15.7:0/3668285833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 0x7f8994071fe0 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7f899406d700 tx=0x7f8984009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2bb41bcbcc82048 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:44:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f899159a700 1 -- 172.21.15.7:0/3668285833 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984011830 con 0x7f899406e020 2023-03-03T18:44:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f899159a700 1 -- 172.21.15.7:0/3668285833 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8984011990 con 0x7f899406e020 2023-03-03T18:44:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.582+0000 7f899159a700 1 -- 172.21.15.7:0/3668285833 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984011d20 con 0x7f899406e020 2023-03-03T18:44:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.583+0000 7f8998d66700 1 -- 172.21.15.7:0/3668285833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 msgr2=0x7f8994071fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.583+0000 7f8998d66700 1 --2- 172.21.15.7:0/3668285833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 0x7f8994071fe0 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7f899406d700 tx=0x7f8984009c00 comp rx=0 tx=0).stop 2023-03-03T18:44:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.583+0000 7f8998d66700 1 -- 172.21.15.7:0/3668285833 shutdown_connections 2023-03-03T18:44:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.583+0000 7f8998d66700 1 --2- 172.21.15.7:0/3668285833 >> v2:172.21.15.143:3300/0 conn(0x7f8994155390 0x7f89940c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.583+0000 7f8998d66700 1 --2- 172.21.15.7:0/3668285833 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8994072520 0x7f8994151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.583+0000 7f8998d66700 1 --2- 172.21.15.7:0/3668285833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 0x7f8994071fe0 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.583+0000 7f8998d66700 1 -- 172.21.15.7:0/3668285833 >> 172.21.15.7:0/3668285833 conn(0x7f899406c110 msgr2=0x7f899406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:44:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.583+0000 7f8998d66700 1 -- 172.21.15.7:0/3668285833 shutdown_connections 2023-03-03T18:44:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.583+0000 7f8998d66700 1 -- 172.21.15.7:0/3668285833 wait complete. 2023-03-03T18:44:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.585+0000 7f8998d66700 1 Processor -- start 2023-03-03T18:44:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.585+0000 7f8998d66700 1 -- start start 2023-03-03T18:44:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8998d66700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 0x7f89941eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8998d66700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8994072520 0x7f899414de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8998d66700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8994155390 0x7f89940abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8998d66700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89940ab6d0 con 0x7f899406e020 2023-03-03T18:44:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8998d66700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89941eb670 con 0x7f8994072520 2023-03-03T18:44:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8998d66700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8994152f00 con 0x7f8994155390 2023-03-03T18:44:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f899259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 0x7f89941eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f899259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 0x7f89941eb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38466/0 (socket says 172.21.15.7:38466) 2023-03-03T18:44:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f899259c700 1 -- 172.21.15.7:0/2459510666 learned_addr learned my addr 172.21.15.7:0/2459510666 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:44:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8992d9d700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8994155390 0x7f89940abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8991d9b700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8994072520 0x7f899414de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f899259c700 1 -- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8994155390 msgr2=0x7f89940abb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f899259c700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8994155390 0x7f89940abb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f899259c700 1 -- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8994072520 msgr2=0x7f899414de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f899259c700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8994072520 0x7f899414de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f899259c700 1 -- 172.21.15.7:0/2459510666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984009550 con 0x7f899406e020 2023-03-03T18:44:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8991d9b700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8994072520 0x7f899414de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:44:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f899259c700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 0x7f89941eb130 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f898800e810 tx=0x7f898800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:44:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8982ffd700 1 -- 172.21.15.7:0/2459510666 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898800c020 con 0x7f899406e020 2023-03-03T18:44:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8982ffd700 1 -- 172.21.15.7:0/2459510666 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8988004250 con 0x7f899406e020 2023-03-03T18:44:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8998d66700 1 -- 172.21.15.7:0/2459510666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8994070540 con 0x7f899406e020 2023-03-03T18:44:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8982ffd700 1 -- 172.21.15.7:0/2459510666 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89880036a0 con 0x7f899406e020 2023-03-03T18:44:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.586+0000 7f8998d66700 1 -- 172.21.15.7:0/2459510666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89940702a0 con 0x7f899406e020 2023-03-03T18:44:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.587+0000 7f8998d66700 1 -- 172.21.15.7:0/2459510666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8994070820 con 0x7f899406e020 2023-03-03T18:44:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.588+0000 7f8982ffd700 1 -- 172.21.15.7:0/2459510666 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8988022070 con 0x7f899406e020 2023-03-03T18:44:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.588+0000 7f8982ffd700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89740682f0 0x7f897406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:44:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.588+0000 7f8982ffd700 1 -- 172.21.15.7:0/2459510666 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8988085eb0 con 0x7f899406e020 2023-03-03T18:44:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.589+0000 7f8991d9b700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89740682f0 0x7f897406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:44:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.589+0000 7f8991d9b700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89740682f0 0x7f897406a7a0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f89940c2550 tx=0x7f8984008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:44:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.593+0000 7f8982ffd700 1 -- 172.21.15.7:0/2459510666 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8988052250 con 0x7f899406e020 2023-03-03T18:44:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.829+0000 7f8998d66700 1 -- 172.21.15.7:0/2459510666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8994074bb0 con 0x7f899406e020 2023-03-03T18:44:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.830+0000 7f8982ffd700 1 -- 172.21.15.7:0/2459510666 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f898800f040 con 0x7f899406e020 2023-03-03T18:44:47.830 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:44:47.830 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:44:47.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 -- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89740682f0 msgr2=0x7f897406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:47.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89740682f0 0x7f897406a7a0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f89940c2550 tx=0x7f8984008030 comp rx=0 tx=0).stop 2023-03-03T18:44:47.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 -- 172.21.15.7:0/2459510666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 msgr2=0x7f89941eb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:44:47.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 0x7f89941eb130 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7f898800e810 tx=0x7f898800ebd0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 -- 172.21.15.7:0/2459510666 shutdown_connections 2023-03-03T18:44:47.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8994155390 0x7f89940abb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f89740682f0 0x7f897406a7a0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f899406e020 0x7f89941eb130 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 --2- 172.21.15.7:0/2459510666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8994072520 0x7f899414de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:44:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 -- 172.21.15.7:0/2459510666 >> 172.21.15.7:0/2459510666 conn(0x7f899406c110 msgr2=0x7f899406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:44:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 -- 172.21.15.7:0/2459510666 shutdown_connections 2023-03-03T18:44:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:44:47.832+0000 7f8998d66700 1 -- 172.21.15.7:0/2459510666 wait complete. 2023-03-03T18:44:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:44:47.843 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2023-03-03T18:45:11.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:45:11.859 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:45:14.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:45:14.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.704+0000 7fdbeb8bb700 1 Processor -- start 2023-03-03T18:45:14.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.704+0000 7fdbeb8bb700 1 -- start start 2023-03-03T18:45:14.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbeb8bb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdbe4152e30 0x7fdbe40c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbeb8bb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdbe40c4910 0x7fdbe4156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbeb8bb700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe41523b0 con 0x7fdbe40beab0 2023-03-03T18:45:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbeb8bb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe4072010 con 0x7fdbe4152e30 2023-03-03T18:45:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbeb8bb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe40aaaf0 con 0x7fdbe40c4910 2023-03-03T18:45:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbe9657700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdbe4152e30 0x7fdbe40c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbe9657700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdbe4152e30 0x7fdbe40c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38494/0 (socket says 172.21.15.7:38494) 2023-03-03T18:45:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbe9657700 1 -- 172.21.15.7:0/1878118207 learned_addr learned my addr 172.21.15.7:0/1878118207 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:45:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbe9e58700 1 --2- 172.21.15.7:0/1878118207 >> v2:172.21.15.143:3300/0 conn(0x7fdbe40c4910 0x7fdbe4156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbe9657700 1 -- 172.21.15.7:0/1878118207 >> v2:172.21.15.143:3300/0 conn(0x7fdbe40c4910 msgr2=0x7fdbe4156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:45:14.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbe9657700 1 --2- 172.21.15.7:0/1878118207 >> v2:172.21.15.143:3300/0 conn(0x7fdbe40c4910 0x7fdbe4156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbe9657700 1 -- 172.21.15.7:0/1878118207 >> v1:172.21.15.143:6789/0 conn(0x7fdbe40beab0 legacy=0x7fdbe4155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:45:14.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.705+0000 7fdbe9657700 1 -- 172.21.15.7:0/1878118207 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe40aadd0 con 0x7fdbe4152e30 2023-03-03T18:45:14.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbe9657700 1 --2- 172.21.15.7:0/1878118207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe4152e30 0x7fdbe40c43d0 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40c3530 tx=0x7fdbd8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6999bd74362ff7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:45:14.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbd7fff700 1 -- 172.21.15.7:0/1878118207 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8004230 con 0x7fdbe4152e30 2023-03-03T18:45:14.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbd7fff700 1 -- 172.21.15.7:0/1878118207 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd8004390 con 0x7fdbe4152e30 2023-03-03T18:45:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbd7fff700 1 -- 172.21.15.7:0/1878118207 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd80036a0 con 0x7fdbe4152e30 2023-03-03T18:45:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/1878118207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe4152e30 msgr2=0x7fdbe40c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbeb8bb700 1 --2- 172.21.15.7:0/1878118207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe4152e30 0x7fdbe40c43d0 secure :-1 s=READY pgs=3274 cs=0 l=1 rev1=1 crypto rx=0x7fdbe40c3530 tx=0x7fdbd8009bd0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/1878118207 shutdown_connections 2023-03-03T18:45:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbeb8bb700 1 --2- 172.21.15.7:0/1878118207 >> v2:172.21.15.143:3300/0 conn(0x7fdbe40c4910 0x7fdbe4156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbeb8bb700 1 --2- 172.21.15.7:0/1878118207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe4152e30 0x7fdbe40c43d0 unknown :-1 s=CLOSED pgs=3274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/1878118207 >> 172.21.15.7:0/1878118207 conn(0x7fdbe40736f0 msgr2=0x7fdbe4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:45:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/1878118207 shutdown_connections 2023-03-03T18:45:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.706+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/1878118207 wait complete. 2023-03-03T18:45:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.708+0000 7fdbeb8bb700 1 Processor -- start 2023-03-03T18:45:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.708+0000 7fdbeb8bb700 1 -- start start 2023-03-03T18:45:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbeb8bb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe40beab0 0x7fdbe4156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbeb8bb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40c4910 0x7fdbe41e7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbeb8bb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe4152e30 0x7fdbe41e8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbeb8bb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe414b9d0 con 0x7fdbe40beab0 2023-03-03T18:45:14.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbeb8bb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe41ec320 con 0x7fdbe4152e30 2023-03-03T18:45:14.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbeb8bb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe41ebe90 con 0x7fdbe40c4910 2023-03-03T18:45:14.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbe9657700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe40beab0 0x7fdbe4156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbe9657700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe40beab0 0x7fdbe4156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38500/0 (socket says 172.21.15.7:38500) 2023-03-03T18:45:14.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbe9657700 1 -- 172.21.15.7:0/830992811 learned_addr learned my addr 172.21.15.7:0/830992811 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:45:14.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbe9e58700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe4152e30 0x7fdbe41e8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbe8e56700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40c4910 0x7fdbe41e7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbe9657700 1 -- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40c4910 msgr2=0x7fdbe41e7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbe9657700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40c4910 0x7fdbe41e7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbe9657700 1 -- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe4152e30 msgr2=0x7fdbe41e8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbe9657700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe4152e30 0x7fdbe41e8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.709+0000 7fdbe9657700 1 -- 172.21.15.7:0/830992811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd8009550 con 0x7fdbe40beab0 2023-03-03T18:45:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.710+0000 7fdbe9657700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe40beab0 0x7fdbe4156080 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009130 tx=0x7fdbd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:45:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.710+0000 7fdbd5ffb700 1 -- 172.21.15.7:0/830992811 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8003d40 con 0x7fdbe40beab0 2023-03-03T18:45:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.710+0000 7fdbd5ffb700 1 -- 172.21.15.7:0/830992811 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd801f070 con 0x7fdbe40beab0 2023-03-03T18:45:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.710+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/830992811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe4153200 con 0x7fdbe40beab0 2023-03-03T18:45:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.710+0000 7fdbd5ffb700 1 -- 172.21.15.7:0/830992811 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd8011870 con 0x7fdbe40beab0 2023-03-03T18:45:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.710+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/830992811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe4071c70 con 0x7fdbe40beab0 2023-03-03T18:45:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.711+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/830992811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe4153390 con 0x7fdbe40beab0 2023-03-03T18:45:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.711+0000 7fdbd5ffb700 1 -- 172.21.15.7:0/830992811 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdbd8017020 con 0x7fdbe40beab0 2023-03-03T18:45:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.712+0000 7fdbd5ffb700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbc80682f0 0x7fdbc806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.712+0000 7fdbd5ffb700 1 -- 172.21.15.7:0/830992811 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdbd80879a0 con 0x7fdbe40beab0 2023-03-03T18:45:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.712+0000 7fdbe8e56700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbc80682f0 0x7fdbc806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.713+0000 7fdbe8e56700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbc80682f0 0x7fdbc806a7a0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4154320 tx=0x7fdbe0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:45:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.716+0000 7fdbd5ffb700 1 -- 172.21.15.7:0/830992811 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdbd8053c50 con 0x7fdbe40beab0 2023-03-03T18:45:14.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:45:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.957+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/830992811 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbe41ec170 con 0x7fdbe40beab0 2023-03-03T18:45:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.958+0000 7fdbd5ffb700 1 -- 172.21.15.7:0/830992811 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdbd8057270 con 0x7fdbe40beab0 2023-03-03T18:45:14.959 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:45:14.960 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:45:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbc80682f0 msgr2=0x7fdbc806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbc80682f0 0x7fdbc806a7a0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4154320 tx=0x7fdbe0008040 comp rx=0 tx=0).stop 2023-03-03T18:45:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/830992811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe40beab0 msgr2=0x7fdbe4156080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe40beab0 0x7fdbe4156080 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009130 tx=0x7fdbd8004750 comp rx=0 tx=0).stop 2023-03-03T18:45:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/830992811 shutdown_connections 2023-03-03T18:45:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbe4152e30 0x7fdbe41e8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdbc80682f0 0x7fdbc806a7a0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdbe40beab0 0x7fdbe4156080 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 --2- 172.21.15.7:0/830992811 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdbe40c4910 0x7fdbe41e7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/830992811 >> 172.21.15.7:0/830992811 conn(0x7fdbe40736f0 msgr2=0x7fdbe40c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:45:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/830992811 shutdown_connections 2023-03-03T18:45:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.960+0000 7fdbeb8bb700 1 -- 172.21.15.7:0/830992811 wait complete. 2023-03-03T18:45:14.963 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:45:14.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.969+0000 7ff30122a700 1 Processor -- start 2023-03-03T18:45:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.969+0000 7ff30122a700 1 -- start start 2023-03-03T18:45:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff30122a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff2fc0c00e0 0x7ff2fc0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff30122a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc154ff0 0x7ff2fc0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff30122a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc0c7310 con 0x7ff2fc0caa30 2023-03-03T18:45:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff30122a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2fc0bfd10 con 0x7ff2fc0c00e0 2023-03-03T18:45:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff30122a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc0bfa30 con 0x7ff2fc154ff0 2023-03-03T18:45:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2fad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff2fc0caa30 0x7ff2fc0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60720/0 (socket says 172.21.15.7:60720) 2023-03-03T18:45:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2fad9d700 1 -- 172.21.15.7:0/1903246248 learned_addr learned my addr 172.21.15.7:0/1903246248 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:45:14.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2fb59e700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc154ff0 0x7ff2fc0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452575736 0 0) 0x7ff2fc0c7310 con 0x7ff2fc0caa30 2023-03-03T18:45:14.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2fa59c700 1 --2- 172.21.15.7:0/1903246248 >> v2:172.21.15.143:3300/0 conn(0x7ff2fc0c00e0 0x7ff2fc0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e40035b0 con 0x7ff2fc0caa30 2023-03-03T18:45:14.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1005240529 0 0) 0x7ff2e40035b0 con 0x7ff2fc0caa30 2023-03-03T18:45:14.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc154ff0 msgr2=0x7ff2fc0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:45:14.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc154ff0 0x7ff2fc0cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 >> v2:172.21.15.143:3300/0 conn(0x7ff2fc0c00e0 msgr2=0x7ff2fc0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:45:14.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 --2- 172.21.15.7:0/1903246248 >> v2:172.21.15.143:3300/0 conn(0x7ff2fc0c00e0 0x7ff2fc0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc154510 con 0x7ff2fc0caa30 2023-03-03T18:45:14.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff2ec001670 con 0x7ff2fc0caa30 2023-03-03T18:45:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 >> v1:172.21.15.7:6789/0 conn(0x7ff2fc0caa30 legacy=0x7ff2fc0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2e4003a70 0x7ff2e4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc154ff0 0x7ff2e4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2e4004d90 0x7ff2e4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc0bfd10 con 0x7ff2e4004d90 2023-03-03T18:45:14.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc0bfa30 con 0x7ff2e4003a70 2023-03-03T18:45:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.970+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e400bac0 con 0x7ff2fc154ff0 2023-03-03T18:45:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2fb59e700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2e4004d90 0x7ff2e4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2fad9d700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2e4003a70 0x7ff2e4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2fa59c700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc154ff0 0x7ff2e4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2fb59e700 1 -- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc154ff0 msgr2=0x7ff2e4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:45:14.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2fb59e700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc154ff0 0x7ff2e4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2fb59e700 1 -- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2e4003a70 msgr2=0x7ff2e4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:45:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2fb59e700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2e4003a70 0x7ff2e4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2fb59e700 1 -- 172.21.15.7:0/1903246248 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e40035b0 con 0x7ff2e4004d90 2023-03-03T18:45:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2fad9d700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2e4003a70 0x7ff2e4003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:45:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2fb59e700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2e4004d90 0x7ff2e4005200 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4009fc0 tx=0x7ff2e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec4dcf521f9e0ae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:45:14.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f400ec30 con 0x7ff2e4004d90 2023-03-03T18:45:14.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f400f040 con 0x7ff2e4004d90 2023-03-03T18:45:14.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.971+0000 7ff2f9d9b700 1 -- 172.21.15.7:0/1903246248 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4014480 con 0x7ff2e4004d90 2023-03-03T18:45:14.980 INFO:tasks.mds_thrash.fs.[b]:waiting for 92 secs before thrashing 2023-03-03T18:45:14.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.972+0000 7ff30122a700 1 -- 172.21.15.7:0/1903246248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2e4004d90 msgr2=0x7ff2e4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:14.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.972+0000 7ff30122a700 1 --2- 172.21.15.7:0/1903246248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2e4004d90 0x7ff2e4005200 secure :-1 s=READY pgs=3277 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4009fc0 tx=0x7ff2e4005880 comp rx=0 tx=0).stop 2023-03-03T18:45:14.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.972+0000 7ff2fb59e700 1 -- 172.21.15.7:0/1903246248 reap_dead start 2023-03-03T18:45:14.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.972+0000 7ff30122a700 1 -- 172.21.15.7:0/1903246248 shutdown_connections 2023-03-03T18:45:14.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.972+0000 7ff30122a700 1 -- 172.21.15.7:0/1903246248 >> 172.21.15.7:0/1903246248 conn(0x7ff2fc066e40 msgr2=0x7ff2fc071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:45:14.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.972+0000 7ff30122a700 1 -- 172.21.15.7:0/1903246248 shutdown_connections 2023-03-03T18:45:14.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.972+0000 7ff30122a700 1 -- 172.21.15.7:0/1903246248 wait complete. 2023-03-03T18:45:14.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.973+0000 7ff30122a700 1 Processor -- start 2023-03-03T18:45:14.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff30122a700 1 -- start start 2023-03-03T18:45:14.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff30122a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2fc156020 0x7ff2fc1ed920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff30122a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc1563f0 0x7ff2fc0c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff30122a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2fc157c40 0x7ff2fc0c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff30122a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc156e40 con 0x7ff2fc157c40 2023-03-03T18:45:14.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff30122a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc14d5d0 con 0x7ff2fc156020 2023-03-03T18:45:14.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff30122a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc1580b0 con 0x7ff2fc1563f0 2023-03-03T18:45:14.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff2fb59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2fc157c40 0x7ff2fc0c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff2fb59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2fc157c40 0x7ff2fc0c1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38524/0 (socket says 172.21.15.7:38524) 2023-03-03T18:45:14.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff2fb59e700 1 -- 172.21.15.7:0/2367489669 learned_addr learned my addr 172.21.15.7:0/2367489669 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:45:14.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff2fad9d700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2fc156020 0x7ff2fc1ed920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.974+0000 7ff2fa59c700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc1563f0 0x7ff2fc0c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff2fb59e700 1 -- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc1563f0 msgr2=0x7ff2fc0c19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:14.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff2fb59e700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc1563f0 0x7ff2fc0c19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff2fb59e700 1 -- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2fc156020 msgr2=0x7ff2fc1ed920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:14.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff2fb59e700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2fc156020 0x7ff2fc1ed920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:14.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff2fb59e700 1 -- 172.21.15.7:0/2367489669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc154ad0 con 0x7ff2fc157c40 2023-03-03T18:45:14.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff2fb59e700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2fc157c40 0x7ff2fc0c1f20 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4016000 tx=0x7ff2f4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:45:14.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff2e37fe700 1 -- 172.21.15.7:0/2367489669 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4007000 con 0x7ff2fc157c40 2023-03-03T18:45:14.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff2e37fe700 1 -- 172.21.15.7:0/2367489669 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f4003e30 con 0x7ff2fc157c40 2023-03-03T18:45:14.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff30122a700 1 -- 172.21.15.7:0/2367489669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2fc1547f0 con 0x7ff2fc157c40 2023-03-03T18:45:14.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff2e37fe700 1 -- 172.21.15.7:0/2367489669 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4015a70 con 0x7ff2fc157c40 2023-03-03T18:45:14.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.975+0000 7ff30122a700 1 -- 172.21.15.7:0/2367489669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2fc0bf650 con 0x7ff2fc157c40 2023-03-03T18:45:14.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.976+0000 7ff30122a700 1 -- 172.21.15.7:0/2367489669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2fc074bb0 con 0x7ff2fc157c40 2023-03-03T18:45:14.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.976+0000 7ff2e37fe700 1 -- 172.21.15.7:0/2367489669 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff2f4015bd0 con 0x7ff2fc157c40 2023-03-03T18:45:14.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.977+0000 7ff2e37fe700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2e80682f0 0x7ff2e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.977+0000 7ff2e37fe700 1 -- 172.21.15.7:0/2367489669 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff2f4087f40 con 0x7ff2fc157c40 2023-03-03T18:45:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.977+0000 7ff2fad9d700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2e80682f0 0x7ff2e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.977+0000 7ff2fad9d700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2e80682f0 0x7ff2e806a7a0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc1555e0 tx=0x7ff2ec001e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:45:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:14.981+0000 7ff2e37fe700 1 -- 172.21.15.7:0/2367489669 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff2f40543a0 con 0x7ff2fc157c40 2023-03-03T18:45:15.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.218+0000 7ff30122a700 1 -- 172.21.15.7:0/2367489669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2fc0c5030 con 0x7ff2fc157c40 2023-03-03T18:45:15.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.218+0000 7ff2e37fe700 1 -- 172.21.15.7:0/2367489669 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff2f4015e60 con 0x7ff2fc157c40 2023-03-03T18:45:15.219 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:45:15.219 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:45:15.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 -- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2e80682f0 msgr2=0x7ff2e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:15.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2e80682f0 0x7ff2e806a7a0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc1555e0 tx=0x7ff2ec001e00 comp rx=0 tx=0).stop 2023-03-03T18:45:15.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 -- 172.21.15.7:0/2367489669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2fc157c40 msgr2=0x7ff2fc0c1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:15.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2fc157c40 0x7ff2fc0c1f20 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4016000 tx=0x7ff2f4009750 comp rx=0 tx=0).stop 2023-03-03T18:45:15.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 -- 172.21.15.7:0/2367489669 shutdown_connections 2023-03-03T18:45:15.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2e80682f0 0x7ff2e806a7a0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:15.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2fc157c40 0x7ff2fc0c1f20 unknown :-1 s=CLOSED pgs=3278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:15.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff2fc156020 0x7ff2fc1ed920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:15.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 --2- 172.21.15.7:0/2367489669 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff2fc1563f0 0x7ff2fc0c19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:15.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 -- 172.21.15.7:0/2367489669 >> 172.21.15.7:0/2367489669 conn(0x7ff2fc066e40 msgr2=0x7ff2fc0c6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:45:15.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 -- 172.21.15.7:0/2367489669 shutdown_connections 2023-03-03T18:45:15.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:15.220+0000 7ff30122a700 1 -- 172.21.15.7:0/2367489669 wait complete. 2023-03-03T18:45:15.224 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:45:15.233 INFO:tasks.mds_thrash.fs.[a]:waiting for 13 secs before thrashing 2023-03-03T18:45:28.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:45:28.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.373+0000 7f2bf8d3d700 1 Processor -- start 2023-03-03T18:45:28.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.373+0000 7f2bf8d3d700 1 -- start start 2023-03-03T18:45:28.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.374+0000 7f2bf8d3d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2bf406d4b0 0x7f2bf40c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:28.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.374+0000 7f2bf8d3d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2bf406d880 0x7f2bf40c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:28.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.374+0000 7f2bf8d3d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf40c7c20 con 0x7f2bf40c0d10 2023-03-03T18:45:28.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.374+0000 7f2bf8d3d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf41544c0 con 0x7f2bf406d880 2023-03-03T18:45:28.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.374+0000 7f2bf8d3d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf413a600 con 0x7f2bf406d4b0 2023-03-03T18:45:28.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.374+0000 7f2bf2d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2bf40c0d10 0x7f2bf4138ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60746/0 (socket says 172.21.15.7:60746) 2023-03-03T18:45:28.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.374+0000 7f2bf2d9d700 1 -- 172.21.15.7:0/3106627240 learned_addr learned my addr 172.21.15.7:0/3106627240 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:45:28.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.374+0000 7f2bf1d9b700 1 --2- 172.21.15.7:0/3106627240 >> v2:172.21.15.7:3300/0 conn(0x7f2bf406d880 0x7f2bf40c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:28.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.374+0000 7f2bf259c700 1 --2- 172.21.15.7:0/3106627240 >> v2:172.21.15.143:3300/0 conn(0x7f2bf406d4b0 0x7f2bf40c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:28.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.375+0000 7f2bf159a700 1 -- 172.21.15.7:0/3106627240 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280747444 0 0) 0x7f2bf40c7c20 con 0x7f2bf40c0d10 2023-03-03T18:45:28.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.375+0000 7f2bf159a700 1 -- 172.21.15.7:0/3106627240 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bdc0035b0 con 0x7f2bf40c0d10 2023-03-03T18:45:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.375+0000 7f2bf1d9b700 1 -- 172.21.15.7:0/3106627240 >> v2:172.21.15.143:3300/0 conn(0x7f2bf406d4b0 msgr2=0x7f2bf40c1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:45:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.375+0000 7f2bf1d9b700 1 --2- 172.21.15.7:0/3106627240 >> v2:172.21.15.143:3300/0 conn(0x7f2bf406d4b0 0x7f2bf40c1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.375+0000 7f2bf1d9b700 1 -- 172.21.15.7:0/3106627240 >> v1:172.21.15.7:6789/0 conn(0x7f2bf40c0d10 legacy=0x7f2bf4138ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.375+0000 7f2bf1d9b700 1 -- 172.21.15.7:0/3106627240 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf40c1170 con 0x7f2bf406d880 2023-03-03T18:45:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.375+0000 7f2bf1d9b700 1 --2- 172.21.15.7:0/3106627240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 0x7f2bf40c07d0 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40aa150 tx=0x7f2be4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b3ba4ec64a51cb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:45:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.375+0000 7f2bf159a700 1 -- 172.21.15.7:0/3106627240 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4011830 con 0x7f2bf406d880 2023-03-03T18:45:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.375+0000 7f2bf159a700 1 -- 172.21.15.7:0/3106627240 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be4011990 con 0x7f2bf406d880 2023-03-03T18:45:28.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.375+0000 7f2bf159a700 1 -- 172.21.15.7:0/3106627240 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4011d20 con 0x7f2bf406d880 2023-03-03T18:45:28.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.376+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/3106627240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 msgr2=0x7f2bf40c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:28.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.376+0000 7f2bf8d3d700 1 --2- 172.21.15.7:0/3106627240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 0x7f2bf40c07d0 secure :-1 s=READY pgs=3280 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40aa150 tx=0x7f2be4009b20 comp rx=0 tx=0).stop 2023-03-03T18:45:28.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.376+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/3106627240 shutdown_connections 2023-03-03T18:45:28.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.376+0000 7f2bf8d3d700 1 --2- 172.21.15.7:0/3106627240 >> v2:172.21.15.143:3300/0 conn(0x7f2bf406d4b0 0x7f2bf40c1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:28.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.376+0000 7f2bf8d3d700 1 --2- 172.21.15.7:0/3106627240 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 0x7f2bf40c07d0 unknown :-1 s=CLOSED pgs=3280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.376+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/3106627240 >> 172.21.15.7:0/3106627240 conn(0x7f2bf406c110 msgr2=0x7f2bf406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:45:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.376+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/3106627240 shutdown_connections 2023-03-03T18:45:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.376+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/3106627240 wait complete. 2023-03-03T18:45:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.377+0000 7f2bf8d3d700 1 Processor -- start 2023-03-03T18:45:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.377+0000 7f2bf8d3d700 1 -- start start 2023-03-03T18:45:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf8d3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf406d4b0 0x7f2bf41e9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf8d3d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 0x7f2bf4134a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:28.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf8d3d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf40c0d10 0x7f2bf40aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:28.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf8d3d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf40ab420 con 0x7f2bf406d880 2023-03-03T18:45:28.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf8d3d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf4135210 con 0x7f2bf40c0d10 2023-03-03T18:45:28.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf8d3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf4134f90 con 0x7f2bf406d4b0 2023-03-03T18:45:28.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 0x7f2bf4134a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:28.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 0x7f2bf4134a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38540/0 (socket says 172.21.15.7:38540) 2023-03-03T18:45:28.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf1d9b700 1 -- 172.21.15.7:0/758600429 learned_addr learned my addr 172.21.15.7:0/758600429 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:45:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf259c700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf406d4b0 0x7f2bf41e9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf2d9d700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf40c0d10 0x7f2bf40aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf1d9b700 1 -- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf406d4b0 msgr2=0x7f2bf41e9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf1d9b700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf406d4b0 0x7f2bf41e9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf1d9b700 1 -- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf40c0d10 msgr2=0x7f2bf40aa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf1d9b700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf40c0d10 0x7f2bf40aa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf1d9b700 1 -- 172.21.15.7:0/758600429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be40094a0 con 0x7f2bf406d880 2023-03-03T18:45:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.378+0000 7f2bf259c700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf406d4b0 0x7f2bf41e9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:45:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.379+0000 7f2bf1d9b700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 0x7f2bf4134a50 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7f2be4000c00 tx=0x7f2be4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:45:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.379+0000 7f2be2ffd700 1 -- 172.21.15.7:0/758600429 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be40036a0 con 0x7f2bf406d880 2023-03-03T18:45:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.379+0000 7f2be2ffd700 1 -- 172.21.15.7:0/758600429 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be401e070 con 0x7f2bf406d880 2023-03-03T18:45:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.379+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/758600429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf4154740 con 0x7f2bf406d880 2023-03-03T18:45:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.379+0000 7f2be2ffd700 1 -- 172.21.15.7:0/758600429 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4003a20 con 0x7f2bf406d880 2023-03-03T18:45:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.379+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/758600429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf40c7310 con 0x7f2bf406d880 2023-03-03T18:45:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.380+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/758600429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf40c75d0 con 0x7f2bf406d880 2023-03-03T18:45:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.380+0000 7f2be2ffd700 1 -- 172.21.15.7:0/758600429 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2be400f050 con 0x7f2bf406d880 2023-03-03T18:45:28.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.380+0000 7f2be2ffd700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 0x7f2bd406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:45:28.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.381+0000 7f2be2ffd700 1 -- 172.21.15.7:0/758600429 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2be4086f30 con 0x7f2bf406d880 2023-03-03T18:45:28.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.381+0000 7f2bf259c700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 0x7f2bd406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:45:28.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.381+0000 7f2bf259c700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 0x7f2bd406a7a0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40725b0 tx=0x7f2be8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:45:28.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.385+0000 7f2be2ffd700 1 -- 172.21.15.7:0/758600429 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2be40532d0 con 0x7f2bf406d880 2023-03-03T18:45:28.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.619+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/758600429 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bf40c1ca0 con 0x7f2bf406d880 2023-03-03T18:45:28.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.620+0000 7f2be2ffd700 1 -- 172.21.15.7:0/758600429 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2be4020090 con 0x7f2bf406d880 2023-03-03T18:45:28.620 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:45:28.620 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:45:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 msgr2=0x7f2bd406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 0x7f2bd406a7a0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40725b0 tx=0x7f2be8008040 comp rx=0 tx=0).stop 2023-03-03T18:45:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/758600429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 msgr2=0x7f2bf4134a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:45:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 0x7f2bf4134a50 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7f2be4000c00 tx=0x7f2be4004750 comp rx=0 tx=0).stop 2023-03-03T18:45:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/758600429 shutdown_connections 2023-03-03T18:45:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bf40c0d10 0x7f2bf40aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bf406d4b0 0x7f2bf41e9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bd40682f0 0x7f2bd406a7a0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 --2- 172.21.15.7:0/758600429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bf406d880 0x7f2bf4134a50 unknown :-1 s=CLOSED pgs=3281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:45:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/758600429 >> 172.21.15.7:0/758600429 conn(0x7f2bf406c110 msgr2=0x7f2bf406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:45:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/758600429 shutdown_connections 2023-03-03T18:45:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:45:28.622+0000 7f2bf8d3d700 1 -- 172.21.15.7:0/758600429 wait complete. 2023-03-03T18:45:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:45:28.634 INFO:tasks.mds_thrash.fs.[a]:waiting for 98 secs before thrashing 2023-03-03T18:45:41.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:45:41.923 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:46:11.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:46:11.995 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:46:42.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:46:42.065 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:46:46.982 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:46:47.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.122+0000 7fb82a87b700 1 Processor -- start 2023-03-03T18:46:47.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.122+0000 7fb82a87b700 1 -- start start 2023-03-03T18:46:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.123+0000 7fb82a87b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb8241530d0 0x7fb824156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:46:47.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.123+0000 7fb82a87b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8240c2f30 con 0x7fb82414fda0 2023-03-03T18:46:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.123+0000 7fb82a87b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8240c31d0 con 0x7fb82414f9d0 2023-03-03T18:46:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.123+0000 7fb82a87b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8241524a0 con 0x7fb8241530d0 2023-03-03T18:46:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.123+0000 7fb8237fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb82414fda0 0x7fb8241529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60814/0 (socket says 172.21.15.7:60814) 2023-03-03T18:46:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.123+0000 7fb8237fe700 1 -- 172.21.15.7:0/2823533031 learned_addr learned my addr 172.21.15.7:0/2823533031 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:46:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.123+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272498917 0 0) 0x7fb8240c2f30 con 0x7fb82414fda0 2023-03-03T18:46:47.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.123+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8100035b0 con 0x7fb82414fda0 2023-03-03T18:46:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.123+0000 7fb828e18700 1 --2- 172.21.15.7:0/2823533031 >> v2:172.21.15.143:3300/0 conn(0x7fb8241530d0 0x7fb824156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:46:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.123+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4129061309 0 0) 0x7fb8100035b0 con 0x7fb82414fda0 2023-03-03T18:46:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 >> v2:172.21.15.143:3300/0 conn(0x7fb8241530d0 msgr2=0x7fb824156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:46:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 --2- 172.21.15.7:0/2823533031 >> v2:172.21.15.143:3300/0 conn(0x7fb8241530d0 0x7fb824156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 >> v1:172.21.15.143:6789/0 conn(0x7fb82414f9d0 legacy=0x7fb824071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:46:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8240c34b0 con 0x7fb82414fda0 2023-03-03T18:46:47.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb80c001670 con 0x7fb82414fda0 2023-03-03T18:46:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 >> v1:172.21.15.7:6789/0 conn(0x7fb82414fda0 legacy=0x7fb8241529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:46:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb810003a70 0x7fb810003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:46:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb810004850 0x7fb810004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:46:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb810005400 0x7fb81000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:46:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241524a0 con 0x7fb810004850 2023-03-03T18:46:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81000c2d0 con 0x7fb810005400 2023-03-03T18:46:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81000c490 con 0x7fb810003a70 2023-03-03T18:46:47.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb828e18700 1 --2- 172.21.15.7:0/2823533031 >> v2:172.21.15.143:3300/0 conn(0x7fb8241530d0 0x7fb824156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:46:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb8237fe700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb810004850 0x7fb810004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:46:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb823fff700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb810003a70 0x7fb810003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:46:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb828e18700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb810005400 0x7fb81000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:46:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb8237fe700 1 -- 172.21.15.7:0/2823533031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb810003a70 msgr2=0x7fb810003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:46:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb8237fe700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb810003a70 0x7fb810003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb8237fe700 1 -- 172.21.15.7:0/2823533031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb810005400 msgr2=0x7fb81000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:46:47.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb8237fe700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb810005400 0x7fb81000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb8237fe700 1 -- 172.21.15.7:0/2823533031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8100035b0 con 0x7fb810004850 2023-03-03T18:46:47.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb828e18700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb810005400 0x7fb81000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:46:47.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb823fff700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb810003a70 0x7fb810003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:46:47.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb828e18700 1 -- 172.21.15.7:0/2823533031 reap_dead start 2023-03-03T18:46:47.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb8237fe700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb810004850 0x7fb810004cc0 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00c210 tx=0x7fb810005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865fe7a2eb43b37f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:46:47.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c007c10 con 0x7fb810004850 2023-03-03T18:46:47.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80c00cdd0 con 0x7fb810004850 2023-03-03T18:46:47.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.124+0000 7fb822ffd700 1 -- 172.21.15.7:0/2823533031 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c005ba0 con 0x7fb810004850 2023-03-03T18:46:47.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.125+0000 7fb82a87b700 1 -- 172.21.15.7:0/2823533031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb810004850 msgr2=0x7fb810004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:46:47.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.125+0000 7fb82a87b700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb810004850 0x7fb810004cc0 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00c210 tx=0x7fb810005880 comp rx=0 tx=0).stop 2023-03-03T18:46:47.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.125+0000 7fb82a87b700 1 -- 172.21.15.7:0/2823533031 shutdown_connections 2023-03-03T18:46:47.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.125+0000 7fb82a87b700 1 --2- 172.21.15.7:0/2823533031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb810004850 0x7fb810004cc0 unknown :-1 s=CLOSED pgs=3283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.125+0000 7fb82a87b700 1 -- 172.21.15.7:0/2823533031 >> 172.21.15.7:0/2823533031 conn(0x7fb8240c8900 msgr2=0x7fb8240c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:46:47.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.125+0000 7fb82a87b700 1 -- 172.21.15.7:0/2823533031 shutdown_connections 2023-03-03T18:46:47.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.125+0000 7fb82a87b700 1 -- 172.21.15.7:0/2823533031 wait complete. 2023-03-03T18:46:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.127+0000 7fb82a87b700 1 Processor -- start 2023-03-03T18:46:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.127+0000 7fb82a87b700 1 -- start start 2023-03-03T18:46:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb82a87b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb810004850 0x7fb82406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:46:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb82a87b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8240bd3d0 0x7fb8240c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:46:47.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb82a87b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb82414f0a0 0x7fb82406c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:46:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb82a87b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241ec080 con 0x7fb82414f0a0 2023-03-03T18:46:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb82a87b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8241ebb20 con 0x7fb810004850 2023-03-03T18:46:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb82a87b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8240bdce0 con 0x7fb8240bd3d0 2023-03-03T18:46:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb828e18700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb82414f0a0 0x7fb82406c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:46:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb828e18700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb82414f0a0 0x7fb82406c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38618/0 (socket says 172.21.15.7:38618) 2023-03-03T18:46:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb828e18700 1 -- 172.21.15.7:0/3280042015 learned_addr learned my addr 172.21.15.7:0/3280042015 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:46:47.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb823fff700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb810004850 0x7fb82406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:46:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb8237fe700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8240bd3d0 0x7fb8240c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:46:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb828e18700 1 -- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8240bd3d0 msgr2=0x7fb8240c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:46:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb828e18700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8240bd3d0 0x7fb8240c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb828e18700 1 -- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb810004850 msgr2=0x7fb82406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:46:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb828e18700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb810004850 0x7fb82406def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb828e18700 1 -- 172.21.15.7:0/3280042015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8240ac350 con 0x7fb82414f0a0 2023-03-03T18:46:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.128+0000 7fb828e18700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb82414f0a0 0x7fb82406c4c0 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7fb8240ab150 tx=0x7fb81c0048c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:46:47.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.129+0000 7fb820ff9700 1 -- 172.21.15.7:0/3280042015 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c007b80 con 0x7fb82414f0a0 2023-03-03T18:46:47.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.129+0000 7fb820ff9700 1 -- 172.21.15.7:0/3280042015 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c00f050 con 0x7fb82414f0a0 2023-03-03T18:46:47.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.129+0000 7fb820ff9700 1 -- 172.21.15.7:0/3280042015 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c003e30 con 0x7fb82414f0a0 2023-03-03T18:46:47.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.129+0000 7fb82a87b700 1 -- 172.21.15.7:0/3280042015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8240ab630 con 0x7fb82414f0a0 2023-03-03T18:46:47.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.129+0000 7fb82a87b700 1 -- 172.21.15.7:0/3280042015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8240c3e10 con 0x7fb82414f0a0 2023-03-03T18:46:47.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.130+0000 7fb82a87b700 1 -- 172.21.15.7:0/3280042015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82406dd40 con 0x7fb82414f0a0 2023-03-03T18:46:47.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.130+0000 7fb820ff9700 1 -- 172.21.15.7:0/3280042015 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb81c01e460 con 0x7fb82414f0a0 2023-03-03T18:46:47.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.130+0000 7fb820ff9700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8040682f0 0x7fb80406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:46:47.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.130+0000 7fb820ff9700 1 -- 172.21.15.7:0/3280042015 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb81c086570 con 0x7fb82414f0a0 2023-03-03T18:46:47.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.131+0000 7fb823fff700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8040682f0 0x7fb80406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:46:47.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.131+0000 7fb823fff700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8040682f0 0x7fb80406a7a0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fb814000c00 tx=0x7fb814009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:46:47.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.135+0000 7fb820ff9700 1 -- 172.21.15.7:0/3280042015 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb81c0528b0 con 0x7fb82414f0a0 2023-03-03T18:46:47.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.377+0000 7fb82a87b700 1 -- 172.21.15.7:0/3280042015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb824074bb0 con 0x7fb82414f0a0 2023-03-03T18:46:47.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.378+0000 7fb820ff9700 1 -- 172.21.15.7:0/3280042015 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb81c055ed0 con 0x7fb82414f0a0 2023-03-03T18:46:47.378 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:46:47.378 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:46:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 -- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8040682f0 msgr2=0x7fb80406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:46:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8040682f0 0x7fb80406a7a0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fb814000c00 tx=0x7fb814009040 comp rx=0 tx=0).stop 2023-03-03T18:46:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 -- 172.21.15.7:0/3280042015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb82414f0a0 msgr2=0x7fb82406c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:46:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb82414f0a0 0x7fb82406c4c0 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7fb8240ab150 tx=0x7fb81c0048c0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 -- 172.21.15.7:0/3280042015 shutdown_connections 2023-03-03T18:46:47.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8040682f0 0x7fb80406a7a0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb82414f0a0 0x7fb82406c4c0 unknown :-1 s=CLOSED pgs=3284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb810004850 0x7fb82406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 --2- 172.21.15.7:0/3280042015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb8240bd3d0 0x7fb8240c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:46:47.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 -- 172.21.15.7:0/3280042015 >> 172.21.15.7:0/3280042015 conn(0x7fb8240c8900 msgr2=0x7fb824150c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:46:47.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 -- 172.21.15.7:0/3280042015 shutdown_connections 2023-03-03T18:46:47.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:46:47.380+0000 7fb82a87b700 1 -- 172.21.15.7:0/3280042015 wait complete. 2023-03-03T18:46:47.382 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:46:47.392 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2023-03-03T18:47:06.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:47:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.776+0000 7f29aef9d700 1 Processor -- start 2023-03-03T18:47:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.776+0000 7f29aef9d700 1 -- start start 2023-03-03T18:47:06.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29aef9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29a806d870 0x7f29a81550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:06.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29aef9d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a80c7a60 con 0x7f29a80c1da0 2023-03-03T18:47:06.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29aef9d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a80c3620 con 0x7f29a80c3e30 2023-03-03T18:47:06.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29aef9d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a80c2af0 con 0x7f29a806d870 2023-03-03T18:47:06.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29a7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f29a80c1da0 0x7f29a80c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60846/0 (socket says 172.21.15.7:60846) 2023-03-03T18:47:06.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29a7fff700 1 -- 172.21.15.7:0/1250772028 learned_addr learned my addr 172.21.15.7:0/1250772028 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:47:06.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29acd39700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29a806d870 0x7f29a81550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:06.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684733509 0 0) 0x7f29a80c7a60 con 0x7f29a80c1da0 2023-03-03T18:47:06.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29980035b0 con 0x7f29a80c1da0 2023-03-03T18:47:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1922934970 0 0) 0x7f29980035b0 con 0x7f29a80c1da0 2023-03-03T18:47:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29a806d870 msgr2=0x7f29a81550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:47:06.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29a77fe700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29a806d870 0x7f29a81550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 >> v1:172.21.15.143:6789/0 conn(0x7f29a80c3e30 legacy=0x7f29a815ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a81545e0 con 0x7f29a80c1da0 2023-03-03T18:47:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.777+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f2994001670 con 0x7f29a80c1da0 2023-03-03T18:47:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 >> v1:172.21.15.7:6789/0 conn(0x7f29a80c1da0 legacy=0x7f29a80c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a77fe700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2998003aa0 0x7f2998003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a77fe700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2998004850 0x7f2998004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a77fe700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29a806d870 0x7f2998005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:06.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299800bac0 con 0x7f2998003aa0 2023-03-03T18:47:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299800bc30 con 0x7f2998004850 2023-03-03T18:47:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299800bdf0 con 0x7f29a806d870 2023-03-03T18:47:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29acd39700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2998003aa0 0x7f2998003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a7fff700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2998004850 0x7f2998004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29ad53a700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29a806d870 0x7f2998005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29acd39700 1 -- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29a806d870 msgr2=0x7f2998005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:47:06.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29acd39700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29a806d870 0x7f2998005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29acd39700 1 -- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2998004850 msgr2=0x7f2998004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:47:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29acd39700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2998004850 0x7f2998004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29acd39700 1 -- 172.21.15.7:0/1250772028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29980035b0 con 0x7f2998003aa0 2023-03-03T18:47:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29ad53a700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f29a806d870 0x7f2998005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:47:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29acd39700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2998003aa0 0x7f2998003eb0 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7f299c009b60 tx=0x7f2998005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc92ecf031dd3de server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:47:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c00e6c0 con 0x7f2998003aa0 2023-03-03T18:47:06.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c00e820 con 0x7f2998003aa0 2023-03-03T18:47:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.778+0000 7f29a77fe700 1 -- 172.21.15.7:0/1250772028 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c00eb90 con 0x7f2998003aa0 2023-03-03T18:47:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.779+0000 7f29aef9d700 1 -- 172.21.15.7:0/1250772028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2998003aa0 msgr2=0x7f2998003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.779+0000 7f29aef9d700 1 --2- 172.21.15.7:0/1250772028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2998003aa0 0x7f2998003eb0 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7f299c009b60 tx=0x7f2998005880 comp rx=0 tx=0).stop 2023-03-03T18:47:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.779+0000 7f29ad53a700 1 -- 172.21.15.7:0/1250772028 reap_dead start 2023-03-03T18:47:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.779+0000 7f29aef9d700 1 -- 172.21.15.7:0/1250772028 shutdown_connections 2023-03-03T18:47:06.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.779+0000 7f29aef9d700 1 -- 172.21.15.7:0/1250772028 >> 172.21.15.7:0/1250772028 conn(0x7f29a806c110 msgr2=0x7f29a806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:47:06.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.779+0000 7f29aef9d700 1 -- 172.21.15.7:0/1250772028 shutdown_connections 2023-03-03T18:47:06.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.779+0000 7f29aef9d700 1 -- 172.21.15.7:0/1250772028 wait complete. 2023-03-03T18:47:06.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.780+0000 7f29aef9d700 1 Processor -- start 2023-03-03T18:47:06.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.780+0000 7f29aef9d700 1 -- start start 2023-03-03T18:47:06.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29aef9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2998003aa0 0x7f29a81ea0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:06.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29aef9d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29a80716b0 0x7f29a81ed1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:06.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29aef9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29a81ed6f0 0x7f29a8135730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:06.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29aef9d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a8157d90 con 0x7f29a81ed6f0 2023-03-03T18:47:06.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29aef9d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a8157a20 con 0x7f29a80716b0 2023-03-03T18:47:06.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29aef9d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a81ecd20 con 0x7f2998003aa0 2023-03-03T18:47:06.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29ad53a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29a81ed6f0 0x7f29a8135730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:06.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29ad53a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29a81ed6f0 0x7f29a8135730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38648/0 (socket says 172.21.15.7:38648) 2023-03-03T18:47:06.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29ad53a700 1 -- 172.21.15.7:0/1355063501 learned_addr learned my addr 172.21.15.7:0/1355063501 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:47:06.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29a7fff700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29a80716b0 0x7f29a81ed1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29acd39700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2998003aa0 0x7f29a81ea0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29ad53a700 1 -- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2998003aa0 msgr2=0x7f29a81ea0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29ad53a700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2998003aa0 0x7f29a81ea0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29ad53a700 1 -- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29a80716b0 msgr2=0x7f29a81ed1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29ad53a700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29a80716b0 0x7f29a81ed1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.781+0000 7f29ad53a700 1 -- 172.21.15.7:0/1355063501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a8154ba0 con 0x7f29a81ed6f0 2023-03-03T18:47:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.782+0000 7f29ad53a700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29a81ed6f0 0x7f29a8135730 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f29a8073830 tx=0x7f29a000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:47:06.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.782+0000 7f29a57fa700 1 -- 172.21.15.7:0/1355063501 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0003c10 con 0x7f29a81ed6f0 2023-03-03T18:47:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.782+0000 7f29a57fa700 1 -- 172.21.15.7:0/1355063501 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a0003d90 con 0x7f29a81ed6f0 2023-03-03T18:47:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.782+0000 7f29aef9d700 1 -- 172.21.15.7:0/1355063501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a81548c0 con 0x7f29a81ed6f0 2023-03-03T18:47:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.782+0000 7f29a57fa700 1 -- 172.21.15.7:0/1355063501 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0005310 con 0x7f29a81ed6f0 2023-03-03T18:47:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.782+0000 7f29aef9d700 1 -- 172.21.15.7:0/1355063501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a806dc40 con 0x7f29a81ed6f0 2023-03-03T18:47:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.783+0000 7f29aef9d700 1 -- 172.21.15.7:0/1355063501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a81edb70 con 0x7f29a81ed6f0 2023-03-03T18:47:06.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.783+0000 7f29a57fa700 1 -- 172.21.15.7:0/1355063501 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f29a0005470 con 0x7f29a81ed6f0 2023-03-03T18:47:06.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.784+0000 7f29a57fa700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f298c0682f0 0x7f298c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.784+0000 7f29a57fa700 1 -- 172.21.15.7:0/1355063501 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f29a0086310 con 0x7f29a81ed6f0 2023-03-03T18:47:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.784+0000 7f29acd39700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f298c0682f0 0x7f298c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.785+0000 7f29acd39700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f298c0682f0 0x7f298c06a7a0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f299c0090b0 tx=0x7f299c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:47:06.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:06.788+0000 7f29a57fa700 1 -- 172.21.15.7:0/1355063501 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f29a00526b0 con 0x7f29a81ed6f0 2023-03-03T18:47:07.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.027+0000 7f29aef9d700 1 -- 172.21.15.7:0/1355063501 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29a8074bb0 con 0x7f29a81ed6f0 2023-03-03T18:47:07.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.028+0000 7f29a57fa700 1 -- 172.21.15.7:0/1355063501 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f29a0055cd0 con 0x7f29a81ed6f0 2023-03-03T18:47:07.031 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:47:07.031 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:47:07.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 -- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f298c0682f0 msgr2=0x7f298c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:07.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f298c0682f0 0x7f298c06a7a0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f299c0090b0 tx=0x7f299c008030 comp rx=0 tx=0).stop 2023-03-03T18:47:07.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 -- 172.21.15.7:0/1355063501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29a81ed6f0 msgr2=0x7f29a8135730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:07.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29a81ed6f0 0x7f29a8135730 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f29a8073830 tx=0x7f29a000ee50 comp rx=0 tx=0).stop 2023-03-03T18:47:07.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 -- 172.21.15.7:0/1355063501 shutdown_connections 2023-03-03T18:47:07.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f298c0682f0 0x7f298c06a7a0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:07.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29a81ed6f0 0x7f29a8135730 unknown :-1 s=CLOSED pgs=3287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:07.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2998003aa0 0x7f29a81ea0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:07.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 --2- 172.21.15.7:0/1355063501 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29a80716b0 0x7f29a81ed1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:07.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 -- 172.21.15.7:0/1355063501 >> 172.21.15.7:0/1355063501 conn(0x7f29a806c110 msgr2=0x7f29a806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:47:07.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 -- 172.21.15.7:0/1355063501 shutdown_connections 2023-03-03T18:47:07.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:07.030+0000 7f29aef9d700 1 -- 172.21.15.7:0/1355063501 wait complete. 2023-03-03T18:47:07.037 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:47:07.042 INFO:tasks.mds_thrash.fs.[a]:waiting for 39 secs before thrashing 2023-03-03T18:47:12.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:47:12.135 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:47:14.394 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:47:14.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.533+0000 7f2865682700 1 Processor -- start 2023-03-03T18:47:14.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.533+0000 7f2865682700 1 -- start start 2023-03-03T18:47:14.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.533+0000 7f2865682700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f28600c7e00 0x7f2860154a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:14.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.533+0000 7f2865682700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f286006b6f0 0x7f2860154f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:14.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.533+0000 7f2865682700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2860157bb0 con 0x7f28600c7a30 2023-03-03T18:47:14.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.533+0000 7f2865682700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286006ab00 con 0x7f286006b6f0 2023-03-03T18:47:14.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.533+0000 7f2865682700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28601545c0 con 0x7f28600c7e00 2023-03-03T18:47:14.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f286006b6f0 0x7f2860154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:14.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f286006b6f0 0x7f2860154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38656/0 (socket says 172.21.15.7:38656) 2023-03-03T18:47:14.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285f7fe700 1 -- 172.21.15.7:0/822188078 learned_addr learned my addr 172.21.15.7:0/822188078 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:47:14.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285e7fc700 1 --2- 172.21.15.7:0/822188078 >> v2:172.21.15.143:3300/0 conn(0x7f28600c7e00 0x7f2860154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:14.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285f7fe700 1 -- 172.21.15.7:0/822188078 >> v2:172.21.15.143:3300/0 conn(0x7f28600c7e00 msgr2=0x7f2860154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:47:14.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285f7fe700 1 --2- 172.21.15.7:0/822188078 >> v2:172.21.15.143:3300/0 conn(0x7f28600c7e00 0x7f2860154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:14.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285f7fe700 1 -- 172.21.15.7:0/822188078 >> v1:172.21.15.143:6789/0 conn(0x7f28600c7a30 legacy=0x7f286014e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285f7fe700 1 -- 172.21.15.7:0/822188078 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286006a2f0 con 0x7f286006b6f0 2023-03-03T18:47:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285f7fe700 1 --2- 172.21.15.7:0/822188078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f286006b6f0 0x7f2860154f50 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7f286008c770 tx=0x7f285800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7578bbeb3d119551 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:47:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285dffb700 1 -- 172.21.15.7:0/822188078 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285800e070 con 0x7f286006b6f0 2023-03-03T18:47:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285dffb700 1 -- 172.21.15.7:0/822188078 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285800b910 con 0x7f286006b6f0 2023-03-03T18:47:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.534+0000 7f285dffb700 1 -- 172.21.15.7:0/822188078 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285800bcc0 con 0x7f286006b6f0 2023-03-03T18:47:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.535+0000 7f2865682700 1 -- 172.21.15.7:0/822188078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f286006b6f0 msgr2=0x7f2860154f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:14.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.535+0000 7f2865682700 1 --2- 172.21.15.7:0/822188078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f286006b6f0 0x7f2860154f50 secure :-1 s=READY pgs=3288 cs=0 l=1 rev1=1 crypto rx=0x7f286008c770 tx=0x7f285800b330 comp rx=0 tx=0).stop 2023-03-03T18:47:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.535+0000 7f2865682700 1 -- 172.21.15.7:0/822188078 shutdown_connections 2023-03-03T18:47:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.535+0000 7f2865682700 1 --2- 172.21.15.7:0/822188078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f286006b6f0 0x7f2860154f50 unknown :-1 s=CLOSED pgs=3288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.535+0000 7f2865682700 1 --2- 172.21.15.7:0/822188078 >> v2:172.21.15.143:3300/0 conn(0x7f28600c7e00 0x7f2860154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.535+0000 7f2865682700 1 -- 172.21.15.7:0/822188078 >> 172.21.15.7:0/822188078 conn(0x7f2860073d20 msgr2=0x7f28600716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:47:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.535+0000 7f2865682700 1 -- 172.21.15.7:0/822188078 shutdown_connections 2023-03-03T18:47:14.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.535+0000 7f2865682700 1 -- 172.21.15.7:0/822188078 wait complete. 2023-03-03T18:47:14.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.537+0000 7f2865682700 1 Processor -- start 2023-03-03T18:47:14.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.537+0000 7f2865682700 1 -- start start 2023-03-03T18:47:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f2865682700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28600c7a30 0x7f28601ea750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f2865682700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28600c7e00 0x7f28601edba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f2865682700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2860157280 0x7f28600c2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f2865682700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28600c1ea0 con 0x7f28600c7a30 2023-03-03T18:47:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f2865682700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28601e9cb0 con 0x7f2860157280 2023-03-03T18:47:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f2865682700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28601ee0e0 con 0x7f28600c7e00 2023-03-03T18:47:14.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28600c7a30 0x7f28601ea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:14.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28600c7a30 0x7f28601ea750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38658/0 (socket says 172.21.15.7:38658) 2023-03-03T18:47:14.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285effd700 1 -- 172.21.15.7:0/1726745434 learned_addr learned my addr 172.21.15.7:0/1726745434 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:47:14.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285f7fe700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2860157280 0x7f28600c2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:14.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285effd700 1 -- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28600c7e00 msgr2=0x7f28601edba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:14.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285effd700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28600c7e00 0x7f28601edba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:14.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285effd700 1 -- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2860157280 msgr2=0x7f28600c2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:14.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285effd700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2860157280 0x7f28600c2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:14.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285effd700 1 -- 172.21.15.7:0/1726745434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2858009b30 con 0x7f28600c7a30 2023-03-03T18:47:14.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285f7fe700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2860157280 0x7f28600c2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:47:14.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f285effd700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28600c7a30 0x7f28601ea750 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7f285000c4d0 tx=0x7f285000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:47:14.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f28477fe700 1 -- 172.21.15.7:0/1726745434 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28500052d0 con 0x7f28600c7a30 2023-03-03T18:47:14.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.538+0000 7f28477fe700 1 -- 172.21.15.7:0/1726745434 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2850005430 con 0x7f28600c7a30 2023-03-03T18:47:14.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.539+0000 7f28477fe700 1 -- 172.21.15.7:0/1726745434 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28500038f0 con 0x7f28600c7a30 2023-03-03T18:47:14.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.539+0000 7f2865682700 1 -- 172.21.15.7:0/1726745434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286006a540 con 0x7f28600c7a30 2023-03-03T18:47:14.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.539+0000 7f2865682700 1 -- 172.21.15.7:0/1726745434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28600bfbb0 con 0x7f28600c7a30 2023-03-03T18:47:14.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.539+0000 7f2865682700 1 -- 172.21.15.7:0/1726745434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286006a820 con 0x7f28600c7a30 2023-03-03T18:47:14.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.540+0000 7f28477fe700 1 -- 172.21.15.7:0/1726745434 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2850019030 con 0x7f28600c7a30 2023-03-03T18:47:14.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.540+0000 7f28477fe700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f284c0682f0 0x7f284c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:14.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.540+0000 7f28477fe700 1 -- 172.21.15.7:0/1726745434 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f28500854d0 con 0x7f28600c7a30 2023-03-03T18:47:14.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.541+0000 7f285e7fc700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f284c0682f0 0x7f284c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:14.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.541+0000 7f285e7fc700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f284c0682f0 0x7f284c06a7a0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f2854006b20 tx=0x7f2854006ab0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:47:14.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.545+0000 7f28477fe700 1 -- 172.21.15.7:0/1726745434 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2850051810 con 0x7f28600c7a30 2023-03-03T18:47:14.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.788+0000 7f2865682700 1 -- 172.21.15.7:0/1726745434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2860074bb0 con 0x7f28600c7a30 2023-03-03T18:47:14.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.788+0000 7f28477fe700 1 -- 172.21.15.7:0/1726745434 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2850054e30 con 0x7f28600c7a30 2023-03-03T18:47:14.790 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:47:14.790 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:47:14.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.790+0000 7f2865682700 1 -- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f284c0682f0 msgr2=0x7f284c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:14.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.790+0000 7f2865682700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f284c0682f0 0x7f284c06a7a0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f2854006b20 tx=0x7f2854006ab0 comp rx=0 tx=0).stop 2023-03-03T18:47:14.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.790+0000 7f2865682700 1 -- 172.21.15.7:0/1726745434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28600c7a30 msgr2=0x7f28601ea750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:14.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.790+0000 7f2865682700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28600c7a30 0x7f28601ea750 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7f285000c4d0 tx=0x7f285000c890 comp rx=0 tx=0).stop 2023-03-03T18:47:14.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.791+0000 7f2865682700 1 -- 172.21.15.7:0/1726745434 shutdown_connections 2023-03-03T18:47:14.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.791+0000 7f2865682700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2860157280 0x7f28600c2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:14.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.791+0000 7f2865682700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f284c0682f0 0x7f284c06a7a0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:14.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.791+0000 7f2865682700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28600c7a30 0x7f28601ea750 unknown :-1 s=CLOSED pgs=3289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:14.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.791+0000 7f2865682700 1 --2- 172.21.15.7:0/1726745434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f28600c7e00 0x7f28601edba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:14.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.791+0000 7f2865682700 1 -- 172.21.15.7:0/1726745434 >> 172.21.15.7:0/1726745434 conn(0x7f286006b6f0 msgr2=0x7f2860071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:47:14.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.791+0000 7f2865682700 1 -- 172.21.15.7:0/1726745434 shutdown_connections 2023-03-03T18:47:14.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:14.791+0000 7f2865682700 1 -- 172.21.15.7:0/1726745434 wait complete. 2023-03-03T18:47:14.793 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:47:14.804 INFO:tasks.mds_thrash.fs.[b]:waiting for 95 secs before thrashing 2023-03-03T18:47:42.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:47:42.209 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:47:46.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:47:46.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.182+0000 7fdb5885d700 1 Processor -- start 2023-03-03T18:47:46.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.182+0000 7fdb5885d700 1 -- start start 2023-03-03T18:47:46.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.183+0000 7fdb5885d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb54151d40 0x7fdb540736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:46.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.183+0000 7fdb5885d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 0x7fdb541581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:46.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.183+0000 7fdb5885d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5406d430 con 0x7fdb540c2930 2023-03-03T18:47:46.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.183+0000 7fdb5885d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5406c2a0 con 0x7fdb54151d40 2023-03-03T18:47:46.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.183+0000 7fdb5885d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb541586e0 con 0x7fdb54073c30 2023-03-03T18:47:46.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.183+0000 7fdb5259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb54151d40 0x7fdb540736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:46.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.183+0000 7fdb5259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb54151d40 0x7fdb540736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38688/0 (socket says 172.21.15.7:38688) 2023-03-03T18:47:46.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.183+0000 7fdb5259c700 1 -- 172.21.15.7:0/469271541 learned_addr learned my addr 172.21.15.7:0/469271541 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:47:46.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.184+0000 7fdb52d9d700 1 --2- 172.21.15.7:0/469271541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 0x7fdb541581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:46.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.184+0000 7fdb5259c700 1 -- 172.21.15.7:0/469271541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 msgr2=0x7fdb541581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:47:46.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.184+0000 7fdb5259c700 1 --2- 172.21.15.7:0/469271541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 0x7fdb541581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:46.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.184+0000 7fdb5259c700 1 -- 172.21.15.7:0/469271541 >> v1:172.21.15.143:6789/0 conn(0x7fdb540c2930 legacy=0x7fdb54157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:47:46.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.184+0000 7fdb5259c700 1 -- 172.21.15.7:0/469271541 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5406cb00 con 0x7fdb54151d40 2023-03-03T18:47:46.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.184+0000 7fdb5259c700 1 --2- 172.21.15.7:0/469271541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 0x7fdb540736f0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7fdb480098f0 tx=0x7fdb48009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83cb62332c44f78 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:47:46.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.184+0000 7fdb5159a700 1 -- 172.21.15.7:0/469271541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48004230 con 0x7fdb54151d40 2023-03-03T18:47:46.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.184+0000 7fdb5159a700 1 -- 172.21.15.7:0/469271541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb48004390 con 0x7fdb54151d40 2023-03-03T18:47:46.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.184+0000 7fdb5159a700 1 -- 172.21.15.7:0/469271541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb480036a0 con 0x7fdb54151d40 2023-03-03T18:47:46.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.185+0000 7fdb5885d700 1 -- 172.21.15.7:0/469271541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 msgr2=0x7fdb540736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:46.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.185+0000 7fdb5885d700 1 --2- 172.21.15.7:0/469271541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 0x7fdb540736f0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7fdb480098f0 tx=0x7fdb48009c00 comp rx=0 tx=0).stop 2023-03-03T18:47:46.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.185+0000 7fdb5885d700 1 -- 172.21.15.7:0/469271541 shutdown_connections 2023-03-03T18:47:46.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.185+0000 7fdb5885d700 1 --2- 172.21.15.7:0/469271541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 0x7fdb541581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:46.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.185+0000 7fdb5885d700 1 --2- 172.21.15.7:0/469271541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 0x7fdb540736f0 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.185+0000 7fdb5885d700 1 -- 172.21.15.7:0/469271541 >> 172.21.15.7:0/469271541 conn(0x7fdb540a94d0 msgr2=0x7fdb540a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:47:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.185+0000 7fdb5885d700 1 -- 172.21.15.7:0/469271541 shutdown_connections 2023-03-03T18:47:46.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.185+0000 7fdb5885d700 1 -- 172.21.15.7:0/469271541 wait complete. 2023-03-03T18:47:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.186+0000 7fdb5885d700 1 Processor -- start 2023-03-03T18:47:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.186+0000 7fdb5885d700 1 -- start start 2023-03-03T18:47:46.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb5885d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 0x7fdb5414ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb5885d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb540c2930 0x7fdb540bf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb5885d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 0x7fdb540bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb5885d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540c0a30 con 0x7fdb54151d40 2023-03-03T18:47:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb5885d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb541536c0 con 0x7fdb540c2930 2023-03-03T18:47:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb5885d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb54153370 con 0x7fdb54073c30 2023-03-03T18:47:46.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb52d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 0x7fdb540bfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:46.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb52d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 0x7fdb540bfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38698/0 (socket says 172.21.15.7:38698) 2023-03-03T18:47:46.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb52d9d700 1 -- 172.21.15.7:0/335518847 learned_addr learned my addr 172.21.15.7:0/335518847 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:47:46.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb51d9b700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb540c2930 0x7fdb540bf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:46.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb5259c700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 0x7fdb5414ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:46.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb52d9d700 1 -- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 msgr2=0x7fdb5414ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:46.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb52d9d700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 0x7fdb5414ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:46.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb52d9d700 1 -- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb540c2930 msgr2=0x7fdb540bf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb52d9d700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb540c2930 0x7fdb540bf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb52d9d700 1 -- 172.21.15.7:0/335518847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb48009550 con 0x7fdb54151d40 2023-03-03T18:47:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb5259c700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 0x7fdb5414ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:47:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.187+0000 7fdb52d9d700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 0x7fdb540bfde0 secure :-1 s=READY pgs=3291 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009fd0 tx=0x7fdb4c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:47:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.188+0000 7fdb42ffd700 1 -- 172.21.15.7:0/335518847 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c00ede0 con 0x7fdb54151d40 2023-03-03T18:47:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.188+0000 7fdb42ffd700 1 -- 172.21.15.7:0/335518847 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c014420 con 0x7fdb54151d40 2023-03-03T18:47:46.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.188+0000 7fdb5885d700 1 -- 172.21.15.7:0/335518847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5406d0c0 con 0x7fdb54151d40 2023-03-03T18:47:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.188+0000 7fdb42ffd700 1 -- 172.21.15.7:0/335518847 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0147d0 con 0x7fdb54151d40 2023-03-03T18:47:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.188+0000 7fdb5885d700 1 -- 172.21.15.7:0/335518847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5406cde0 con 0x7fdb54151d40 2023-03-03T18:47:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.189+0000 7fdb5885d700 1 -- 172.21.15.7:0/335518847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5406bea0 con 0x7fdb54151d40 2023-03-03T18:47:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.189+0000 7fdb42ffd700 1 -- 172.21.15.7:0/335518847 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdb4c012050 con 0x7fdb54151d40 2023-03-03T18:47:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.190+0000 7fdb42ffd700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb340682f0 0x7fdb3406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:47:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.190+0000 7fdb42ffd700 1 -- 172.21.15.7:0/335518847 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdb4c086740 con 0x7fdb54151d40 2023-03-03T18:47:46.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.190+0000 7fdb5259c700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb340682f0 0x7fdb3406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:47:46.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.190+0000 7fdb5259c700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb340682f0 0x7fdb3406a7a0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fdb5414c620 tx=0x7fdb48008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:47:46.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.194+0000 7fdb42ffd700 1 -- 172.21.15.7:0/335518847 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdb4c052ae0 con 0x7fdb54151d40 2023-03-03T18:47:46.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.434+0000 7fdb5885d700 1 -- 172.21.15.7:0/335518847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb54074bb0 con 0x7fdb54151d40 2023-03-03T18:47:46.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.434+0000 7fdb42ffd700 1 -- 172.21.15.7:0/335518847 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdb4c056100 con 0x7fdb54151d40 2023-03-03T18:47:46.437 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:47:46.437 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:47:46.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 -- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb340682f0 msgr2=0x7fdb3406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb340682f0 0x7fdb3406a7a0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fdb5414c620 tx=0x7fdb48008030 comp rx=0 tx=0).stop 2023-03-03T18:47:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 -- 172.21.15.7:0/335518847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 msgr2=0x7fdb540bfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:47:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 0x7fdb540bfde0 secure :-1 s=READY pgs=3291 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009fd0 tx=0x7fdb4c00e3c0 comp rx=0 tx=0).stop 2023-03-03T18:47:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 -- 172.21.15.7:0/335518847 shutdown_connections 2023-03-03T18:47:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb340682f0 0x7fdb3406a7a0 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:46.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb54151d40 0x7fdb540bfde0 unknown :-1 s=CLOSED pgs=3291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb54073c30 0x7fdb5414ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 --2- 172.21.15.7:0/335518847 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb540c2930 0x7fdb540bf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:47:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 -- 172.21.15.7:0/335518847 >> 172.21.15.7:0/335518847 conn(0x7fdb540a94d0 msgr2=0x7fdb540c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:47:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 -- 172.21.15.7:0/335518847 shutdown_connections 2023-03-03T18:47:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:47:46.436+0000 7fdb5885d700 1 -- 172.21.15.7:0/335518847 wait complete. 2023-03-03T18:47:46.440 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:47:46.450 INFO:tasks.mds_thrash.fs.[a]:waiting for 35 secs before thrashing 2023-03-03T18:48:12.272 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:48:12.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:48:21.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:48:21.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.589+0000 7f9e68c8a700 1 Processor -- start 2023-03-03T18:48:21.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.589+0000 7f9e68c8a700 1 -- start start 2023-03-03T18:48:21.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e68c8a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9e6414e900 0x7f9e6415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:21.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e68c8a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e641510d0 con 0x7f9e64071e80 2023-03-03T18:48:21.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e68c8a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e640c3740 con 0x7f9e64072250 2023-03-03T18:48:21.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e68c8a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e64070670 con 0x7f9e6414e900 2023-03-03T18:48:21.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e62d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9e6414e900 0x7f9e6415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:21.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e6259c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9e64071e80 0x7f9e64151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60936/0 (socket says 172.21.15.7:60936) 2023-03-03T18:48:21.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e6259c700 1 -- 172.21.15.7:0/2594468769 learned_addr learned my addr 172.21.15.7:0/2594468769 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:48:21.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e62d9d700 1 -- 172.21.15.7:0/2594468769 >> v1:172.21.15.143:6789/0 conn(0x7f9e64072250 legacy=0x7f9e6414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:48:21.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e62d9d700 1 -- 172.21.15.7:0/2594468769 >> v1:172.21.15.7:6789/0 conn(0x7f9e64071e80 legacy=0x7f9e64151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:21.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e6159a700 1 -- 172.21.15.7:0/2594468769 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976829948 0 0) 0x7f9e641510d0 con 0x7f9e64071e80 2023-03-03T18:48:21.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e62d9d700 1 -- 172.21.15.7:0/2594468769 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e640c39e0 con 0x7f9e6414e900 2023-03-03T18:48:21.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e62d9d700 1 --2- 172.21.15.7:0/2594468769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 0x7f9e6415ca60 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009d90 tx=0x7f9e5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda278059464e582 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:21.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e6159a700 1 -- 172.21.15.7:0/2594468769 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c00e070 con 0x7f9e6414e900 2023-03-03T18:48:21.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e6159a700 1 -- 172.21.15.7:0/2594468769 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c00bb90 con 0x7f9e6414e900 2023-03-03T18:48:21.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.590+0000 7f9e6159a700 1 -- 172.21.15.7:0/2594468769 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c003a20 con 0x7f9e6414e900 2023-03-03T18:48:21.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.591+0000 7f9e68c8a700 1 -- 172.21.15.7:0/2594468769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 msgr2=0x7f9e6415ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:21.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.591+0000 7f9e68c8a700 1 --2- 172.21.15.7:0/2594468769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 0x7f9e6415ca60 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009d90 tx=0x7f9e5c00b330 comp rx=0 tx=0).stop 2023-03-03T18:48:21.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.591+0000 7f9e68c8a700 1 -- 172.21.15.7:0/2594468769 shutdown_connections 2023-03-03T18:48:21.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.591+0000 7f9e68c8a700 1 --2- 172.21.15.7:0/2594468769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 0x7f9e6415ca60 unknown :-1 s=CLOSED pgs=3293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:21.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.591+0000 7f9e68c8a700 1 -- 172.21.15.7:0/2594468769 >> 172.21.15.7:0/2594468769 conn(0x7f9e640a94d0 msgr2=0x7f9e640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:48:21.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.591+0000 7f9e68c8a700 1 -- 172.21.15.7:0/2594468769 shutdown_connections 2023-03-03T18:48:21.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.591+0000 7f9e68c8a700 1 -- 172.21.15.7:0/2594468769 wait complete. 2023-03-03T18:48:21.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.592+0000 7f9e68c8a700 1 Processor -- start 2023-03-03T18:48:21.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.593+0000 7f9e68c8a700 1 -- start start 2023-03-03T18:48:21.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.593+0000 7f9e68c8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e64071e80 0x7f9e6414c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:21.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.593+0000 7f9e68c8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e64072250 0x7f9e640c09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:21.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.593+0000 7f9e68c8a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 0x7f9e640c0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:21.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.593+0000 7f9e68c8a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e641edf80 con 0x7f9e6414e900 2023-03-03T18:48:21.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.593+0000 7f9e68c8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6414ac80 con 0x7f9e64072250 2023-03-03T18:48:21.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.593+0000 7f9e68c8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e641ed390 con 0x7f9e64071e80 2023-03-03T18:48:21.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.593+0000 7f9e62d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 0x7f9e640c0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:21.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.593+0000 7f9e62d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 0x7f9e640c0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38734/0 (socket says 172.21.15.7:38734) 2023-03-03T18:48:21.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.593+0000 7f9e62d9d700 1 -- 172.21.15.7:0/3321613789 learned_addr learned my addr 172.21.15.7:0/3321613789 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:48:21.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e6259c700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e64071e80 0x7f9e6414c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:21.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e61d9b700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e64072250 0x7f9e640c09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:21.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e62d9d700 1 -- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e64071e80 msgr2=0x7f9e6414c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:21.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e62d9d700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e64071e80 0x7f9e6414c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:21.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e62d9d700 1 -- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e64072250 msgr2=0x7f9e640c09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:21.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e62d9d700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e64072250 0x7f9e640c09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:21.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e62d9d700 1 -- 172.21.15.7:0/3321613789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c009a70 con 0x7f9e6414e900 2023-03-03T18:48:21.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e6259c700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e64071e80 0x7f9e6414c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:48:21.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e62d9d700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 0x7f9e640c0f20 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c004000 tx=0x7f9e5c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:21.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e52ffd700 1 -- 172.21.15.7:0/3321613789 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c012630 con 0x7f9e6414e900 2023-03-03T18:48:21.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e52ffd700 1 -- 172.21.15.7:0/3321613789 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c012790 con 0x7f9e6414e900 2023-03-03T18:48:21.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e68c8a700 1 -- 172.21.15.7:0/3321613789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e64151f30 con 0x7f9e6414e900 2023-03-03T18:48:21.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e52ffd700 1 -- 172.21.15.7:0/3321613789 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c012b40 con 0x7f9e6414e900 2023-03-03T18:48:21.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.594+0000 7f9e68c8a700 1 -- 172.21.15.7:0/3321613789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e64151300 con 0x7f9e6414e900 2023-03-03T18:48:21.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.595+0000 7f9e68c8a700 1 -- 172.21.15.7:0/3321613789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6406c9a0 con 0x7f9e6414e900 2023-03-03T18:48:21.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.596+0000 7f9e52ffd700 1 -- 172.21.15.7:0/3321613789 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9e5c012ca0 con 0x7f9e6414e900 2023-03-03T18:48:21.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.596+0000 7f9e52ffd700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e440682f0 0x7f9e4406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:21.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.596+0000 7f9e52ffd700 1 -- 172.21.15.7:0/3321613789 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9e5c088820 con 0x7f9e6414e900 2023-03-03T18:48:21.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.596+0000 7f9e6259c700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e440682f0 0x7f9e4406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:21.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.597+0000 7f9e6259c700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e440682f0 0x7f9e4406a7a0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f9e54000f60 tx=0x7f9e54007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:21.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.600+0000 7f9e52ffd700 1 -- 172.21.15.7:0/3321613789 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9e5c054bc0 con 0x7f9e6414e900 2023-03-03T18:48:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.837+0000 7f9e68c8a700 1 -- 172.21.15.7:0/3321613789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e64074bb0 con 0x7f9e6414e900 2023-03-03T18:48:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.838+0000 7f9e52ffd700 1 -- 172.21.15.7:0/3321613789 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9e5c0581e0 con 0x7f9e6414e900 2023-03-03T18:48:21.839 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:48:21.839 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:48:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 -- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e440682f0 msgr2=0x7f9e4406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e440682f0 0x7f9e4406a7a0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f9e54000f60 tx=0x7f9e54007ca0 comp rx=0 tx=0).stop 2023-03-03T18:48:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 -- 172.21.15.7:0/3321613789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 msgr2=0x7f9e640c0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 0x7f9e640c0f20 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c004000 tx=0x7f9e5c004ad0 comp rx=0 tx=0).stop 2023-03-03T18:48:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 -- 172.21.15.7:0/3321613789 shutdown_connections 2023-03-03T18:48:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9e440682f0 0x7f9e4406a7a0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9e6414e900 0x7f9e640c0f20 unknown :-1 s=CLOSED pgs=3294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9e64071e80 0x7f9e6414c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 --2- 172.21.15.7:0/3321613789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e64072250 0x7f9e640c09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 -- 172.21.15.7:0/3321613789 >> 172.21.15.7:0/3321613789 conn(0x7f9e640a94d0 msgr2=0x7f9e640c1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:48:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 -- 172.21.15.7:0/3321613789 shutdown_connections 2023-03-03T18:48:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:21.840+0000 7f9e68c8a700 1 -- 172.21.15.7:0/3321613789 wait complete. 2023-03-03T18:48:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:48:21.852 INFO:tasks.mds_thrash.fs.[a]:waiting for 29 secs before thrashing 2023-03-03T18:48:42.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:48:42.351 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:48:49.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:48:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.944+0000 7fb5b1ef2700 1 Processor -- start 2023-03-03T18:48:49.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.944+0000 7fb5b1ef2700 1 -- start start 2023-03-03T18:48:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5b1ef2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac151de0 0x7fb5ac0c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5b1ef2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5ac0c5140 0x7fb5ac06a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5b1ef2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac06dad0 con 0x7fb5ac0c1a40 2023-03-03T18:48:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5b1ef2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac06e370 con 0x7fb5ac0c5140 2023-03-03T18:48:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5b1ef2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac06ddb0 con 0x7fb5ac151de0 2023-03-03T18:48:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5aaffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb5ac0c1a40 0x7fb5ac157ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60978/0 (socket says 172.21.15.7:60978) 2023-03-03T18:48:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5aaffd700 1 -- 172.21.15.7:0/2868088347 learned_addr learned my addr 172.21.15.7:0/2868088347 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:48:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5ab7fe700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac151de0 0x7fb5ac0c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5abfff700 1 --2- 172.21.15.7:0/2868088347 >> v2:172.21.15.143:3300/0 conn(0x7fb5ac0c5140 0x7fb5ac06a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633164075 0 0) 0x7fb5ac06dad0 con 0x7fb5ac0c1a40 2023-03-03T18:48:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.945+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5980035b0 con 0x7fb5ac0c1a40 2023-03-03T18:48:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2260965282 0 0) 0x7fb5980035b0 con 0x7fb5ac0c1a40 2023-03-03T18:48:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac151de0 msgr2=0x7fb5ac0c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:48:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac151de0 0x7fb5ac0c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 >> v2:172.21.15.143:3300/0 conn(0x7fb5ac0c5140 msgr2=0x7fb5ac06a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:48:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 --2- 172.21.15.7:0/2868088347 >> v2:172.21.15.143:3300/0 conn(0x7fb5ac0c5140 0x7fb5ac06a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac06e090 con 0x7fb5ac0c1a40 2023-03-03T18:48:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb594001670 con 0x7fb5ac0c1a40 2023-03-03T18:48:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 >> v1:172.21.15.7:6789/0 conn(0x7fb5ac0c1a40 legacy=0x7fb5ac157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 0x7fb598003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb598004850 0x7fb598004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac151de0 0x7fb598005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac06e370 con 0x7fb598003a70 2023-03-03T18:48:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac06ddb0 con 0x7fb598004850 2023-03-03T18:48:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59800ba10 con 0x7fb5ac151de0 2023-03-03T18:48:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5ab7fe700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 0x7fb598003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5aaffd700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb598004850 0x7fb598004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5abfff700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac151de0 0x7fb598005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5ab7fe700 1 -- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac151de0 msgr2=0x7fb598005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:48:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5ab7fe700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac151de0 0x7fb598005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5ab7fe700 1 -- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb598004850 msgr2=0x7fb598004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:48:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5ab7fe700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb598004850 0x7fb598004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5ab7fe700 1 -- 172.21.15.7:0/2868088347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5980035b0 con 0x7fb598003a70 2023-03-03T18:48:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.946+0000 7fb5ab7fe700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 0x7fb598003e80 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009b60 tx=0x7fb598005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c914eeb1d63b188 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.947+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c00e6c0 con 0x7fb598003a70 2023-03-03T18:48:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.947+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c00e820 con 0x7fb598003a70 2023-03-03T18:48:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.947+0000 7fb5aa7fc700 1 -- 172.21.15.7:0/2868088347 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c00eb90 con 0x7fb598003a70 2023-03-03T18:48:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.947+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/2868088347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 msgr2=0x7fb598003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.947+0000 7fb5b1ef2700 1 --2- 172.21.15.7:0/2868088347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 0x7fb598003e80 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009b60 tx=0x7fb598005880 comp rx=0 tx=0).stop 2023-03-03T18:48:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.947+0000 7fb5abfff700 1 -- 172.21.15.7:0/2868088347 reap_dead start 2023-03-03T18:48:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.947+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/2868088347 shutdown_connections 2023-03-03T18:48:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.947+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/2868088347 >> 172.21.15.7:0/2868088347 conn(0x7fb5ac0a9430 msgr2=0x7fb5ac0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:48:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.947+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/2868088347 shutdown_connections 2023-03-03T18:48:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.947+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/2868088347 wait complete. 2023-03-03T18:48:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.948+0000 7fb5b1ef2700 1 Processor -- start 2023-03-03T18:48:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5b1ef2700 1 -- start start 2023-03-03T18:48:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5b1ef2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 0x7fb5ac0aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5b1ef2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ac14dd10 0x7fb5ac148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5b1ef2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac148ea0 0x7fb5ac14a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5b1ef2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac14b2f0 con 0x7fb598003a70 2023-03-03T18:48:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5b1ef2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0ab1c0 con 0x7fb5ac14dd10 2023-03-03T18:48:49.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5b1ef2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0a9b90 con 0x7fb5ac148ea0 2023-03-03T18:48:49.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5ab7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 0x7fb5ac0aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:49.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5ab7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 0x7fb5ac0aaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38776/0 (socket says 172.21.15.7:38776) 2023-03-03T18:48:49.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5ab7fe700 1 -- 172.21.15.7:0/4166721187 learned_addr learned my addr 172.21.15.7:0/4166721187 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:48:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5abfff700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac148ea0 0x7fb5ac14a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.949+0000 7fb5aaffd700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ac14dd10 0x7fb5ac148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5ab7fe700 1 -- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac148ea0 msgr2=0x7fb5ac14a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5ab7fe700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac148ea0 0x7fb5ac14a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5ab7fe700 1 -- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ac14dd10 msgr2=0x7fb5ac148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5ab7fe700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ac14dd10 0x7fb5ac148960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5ab7fe700 1 -- 172.21.15.7:0/4166721187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac066de0 con 0x7fb598003a70 2023-03-03T18:48:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5ab7fe700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 0x7fb5ac0aaac0 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0090b0 tx=0x7fb59c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5a3fff700 1 -- 172.21.15.7:0/4166721187 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c021070 con 0x7fb598003a70 2023-03-03T18:48:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5a3fff700 1 -- 172.21.15.7:0/4166721187 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c0045d0 con 0x7fb598003a70 2023-03-03T18:48:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/4166721187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ac0c4c20 con 0x7fb598003a70 2023-03-03T18:48:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5a3fff700 1 -- 172.21.15.7:0/4166721187 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c00e6c0 con 0x7fb598003a70 2023-03-03T18:48:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.950+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/4166721187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac1582b0 con 0x7fb598003a70 2023-03-03T18:48:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.951+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/4166721187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac0aa7a0 con 0x7fb598003a70 2023-03-03T18:48:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.951+0000 7fb5a3fff700 1 -- 172.21.15.7:0/4166721187 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb59c02e030 con 0x7fb598003a70 2023-03-03T18:48:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.952+0000 7fb5a3fff700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb58c0682f0 0x7fb58c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.952+0000 7fb5a3fff700 1 -- 172.21.15.7:0/4166721187 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb59c0874d0 con 0x7fb598003a70 2023-03-03T18:48:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.952+0000 7fb5aaffd700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb58c0682f0 0x7fb58c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.953+0000 7fb5aaffd700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb58c0682f0 0x7fb58c06a7a0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fb594000c00 tx=0x7fb594009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:49.956+0000 7fb5a3fff700 1 -- 172.21.15.7:0/4166721187 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb59c053870 con 0x7fb598003a70 2023-03-03T18:48:50.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.191+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/4166721187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5ac0c1060 con 0x7fb598003a70 2023-03-03T18:48:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.192+0000 7fb5a3fff700 1 -- 172.21.15.7:0/4166721187 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb59c056e90 con 0x7fb598003a70 2023-03-03T18:48:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:48:50.193 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:48:50.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb58c0682f0 msgr2=0x7fb58c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:50.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb58c0682f0 0x7fb58c06a7a0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fb594000c00 tx=0x7fb594009040 comp rx=0 tx=0).stop 2023-03-03T18:48:50.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/4166721187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 msgr2=0x7fb5ac0aaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 0x7fb5ac0aaac0 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0090b0 tx=0x7fb59c0047c0 comp rx=0 tx=0).stop 2023-03-03T18:48:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/4166721187 shutdown_connections 2023-03-03T18:48:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ac148ea0 0x7fb5ac14a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb58c0682f0 0x7fb58c06a7a0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb598003a70 0x7fb5ac0aaac0 unknown :-1 s=CLOSED pgs=3297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 --2- 172.21.15.7:0/4166721187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ac14dd10 0x7fb5ac148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/4166721187 >> 172.21.15.7:0/4166721187 conn(0x7fb5ac0a9430 msgr2=0x7fb5ac0bfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:48:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/4166721187 shutdown_connections 2023-03-03T18:48:50.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.194+0000 7fb5b1ef2700 1 -- 172.21.15.7:0/4166721187 wait complete. 2023-03-03T18:48:50.196 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:48:50.206 INFO:tasks.mds_thrash.fs.[b]:waiting for 42 secs before thrashing 2023-03-03T18:48:50.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:48:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.989+0000 7fe5afeb5700 1 Processor -- start 2023-03-03T18:48:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.989+0000 7fe5afeb5700 1 -- start start 2023-03-03T18:48:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.990+0000 7fe5afeb5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5a80c2e30 0x7fe5a8158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.990+0000 7fe5afeb5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe5a806c6c0 0x7fe5a80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.990+0000 7fe5afeb5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a8157ff0 con 0x7fe5a8151cd0 2023-03-03T18:48:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.990+0000 7fe5afeb5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a806d640 con 0x7fe5a80c2e30 2023-03-03T18:48:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.990+0000 7fe5afeb5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a806cda0 con 0x7fe5a806c6c0 2023-03-03T18:48:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.990+0000 7fe5ad450700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5a80c2e30 0x7fe5a8158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.990+0000 7fe5adc51700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe5a8151cd0 0x7fe5a8154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60996/0 (socket says 172.21.15.7:60996) 2023-03-03T18:48:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.990+0000 7fe5adc51700 1 -- 172.21.15.7:0/2321625463 learned_addr learned my addr 172.21.15.7:0/2321625463 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:48:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.990+0000 7fe5ae452700 1 --2- 172.21.15.7:0/2321625463 >> v2:172.21.15.143:3300/0 conn(0x7fe5a806c6c0 0x7fe5a80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5ad450700 1 -- 172.21.15.7:0/2321625463 >> v2:172.21.15.143:3300/0 conn(0x7fe5a806c6c0 msgr2=0x7fe5a80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:48:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5ad450700 1 --2- 172.21.15.7:0/2321625463 >> v2:172.21.15.143:3300/0 conn(0x7fe5a806c6c0 0x7fe5a80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5ad450700 1 -- 172.21.15.7:0/2321625463 >> v1:172.21.15.7:6789/0 conn(0x7fe5a8151cd0 legacy=0x7fe5a8154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5ad450700 1 -- 172.21.15.7:0/2321625463 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a806d080 con 0x7fe5a80c2e30 2023-03-03T18:48:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5acc4f700 1 -- 172.21.15.7:0/2321625463 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894504418 0 0) 0x7fe5a8157ff0 con 0x7fe5a8151cd0 2023-03-03T18:48:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5ad450700 1 --2- 172.21.15.7:0/2321625463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 0x7fe5a8158480 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80c38a0 tx=0x7fe5a4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5256078752b9deb9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5acc4f700 1 -- 172.21.15.7:0/2321625463 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4011830 con 0x7fe5a80c2e30 2023-03-03T18:48:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5acc4f700 1 -- 172.21.15.7:0/2321625463 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a4011990 con 0x7fe5a80c2e30 2023-03-03T18:48:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5acc4f700 1 -- 172.21.15.7:0/2321625463 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4011d20 con 0x7fe5a80c2e30 2023-03-03T18:48:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5afeb5700 1 -- 172.21.15.7:0/2321625463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 msgr2=0x7fe5a8158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.991+0000 7fe5afeb5700 1 --2- 172.21.15.7:0/2321625463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 0x7fe5a8158480 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80c38a0 tx=0x7fe5a4009b20 comp rx=0 tx=0).stop 2023-03-03T18:48:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.992+0000 7fe5afeb5700 1 -- 172.21.15.7:0/2321625463 shutdown_connections 2023-03-03T18:48:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.992+0000 7fe5afeb5700 1 --2- 172.21.15.7:0/2321625463 >> v2:172.21.15.143:3300/0 conn(0x7fe5a806c6c0 0x7fe5a80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.992+0000 7fe5afeb5700 1 --2- 172.21.15.7:0/2321625463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 0x7fe5a8158480 unknown :-1 s=CLOSED pgs=3299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.992+0000 7fe5afeb5700 1 -- 172.21.15.7:0/2321625463 >> 172.21.15.7:0/2321625463 conn(0x7fe5a80a94d0 msgr2=0x7fe5a80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:48:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.992+0000 7fe5afeb5700 1 -- 172.21.15.7:0/2321625463 shutdown_connections 2023-03-03T18:48:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.992+0000 7fe5afeb5700 1 -- 172.21.15.7:0/2321625463 wait complete. 2023-03-03T18:48:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.993+0000 7fe5afeb5700 1 Processor -- start 2023-03-03T18:48:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.993+0000 7fe5afeb5700 1 -- start start 2023-03-03T18:48:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.993+0000 7fe5afeb5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5a806c6c0 0x7fe5a8153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.993+0000 7fe5afeb5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 0x7fe5a80bf820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:50.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.993+0000 7fe5afeb5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a8151cd0 0x7fe5a80c01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.993+0000 7fe5afeb5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80bf390 con 0x7fe5a80c2e30 2023-03-03T18:48:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.993+0000 7fe5afeb5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a80bed80 con 0x7fe5a8151cd0 2023-03-03T18:48:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.993+0000 7fe5afeb5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a81e1830 con 0x7fe5a806c6c0 2023-03-03T18:48:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ad450700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 0x7fe5a80bf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ad450700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 0x7fe5a80bf820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38792/0 (socket says 172.21.15.7:38792) 2023-03-03T18:48:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ad450700 1 -- 172.21.15.7:0/1545097791 learned_addr learned my addr 172.21.15.7:0/1545097791 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:48:50.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ae452700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a8151cd0 0x7fe5a80c01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5adc51700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5a806c6c0 0x7fe5a8153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ad450700 1 -- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5a806c6c0 msgr2=0x7fe5a8153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ad450700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5a806c6c0 0x7fe5a8153230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ad450700 1 -- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a8151cd0 msgr2=0x7fe5a80c01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ad450700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a8151cd0 0x7fe5a80c01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ad450700 1 -- 172.21.15.7:0/1545097791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40094a0 con 0x7fe5a80c2e30 2023-03-03T18:48:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ae452700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a8151cd0 0x7fe5a80c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:48:50.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5ad450700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 0x7fe5a80bf820 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4014040 tx=0x7fe5a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe59e7fc700 1 -- 172.21.15.7:0/1545097791 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40038a0 con 0x7fe5a80c2e30 2023-03-03T18:48:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe59e7fc700 1 -- 172.21.15.7:0/1545097791 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a401e070 con 0x7fe5a80c2e30 2023-03-03T18:48:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe59e7fc700 1 -- 172.21.15.7:0/1545097791 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4003c20 con 0x7fe5a80c2e30 2023-03-03T18:48:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5afeb5700 1 -- 172.21.15.7:0/1545097791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a8157c80 con 0x7fe5a80c2e30 2023-03-03T18:48:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.994+0000 7fe5afeb5700 1 -- 172.21.15.7:0/1545097791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a806d360 con 0x7fe5a80c2e30 2023-03-03T18:48:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.995+0000 7fe5afeb5700 1 -- 172.21.15.7:0/1545097791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a81589c0 con 0x7fe5a80c2e30 2023-03-03T18:48:51.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.996+0000 7fe59e7fc700 1 -- 172.21.15.7:0/1545097791 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe5a400f050 con 0x7fe5a80c2e30 2023-03-03T18:48:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.996+0000 7fe59e7fc700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe58c0682f0 0x7fe58c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.996+0000 7fe59e7fc700 1 -- 172.21.15.7:0/1545097791 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe5a4087190 con 0x7fe5a80c2e30 2023-03-03T18:48:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.996+0000 7fe5adc51700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe58c0682f0 0x7fe58c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:50.997+0000 7fe5adc51700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe58c0682f0 0x7fe58c06a7a0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fe598000f60 tx=0x7fe598009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:51.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.000+0000 7fe59e7fc700 1 -- 172.21.15.7:0/1545097791 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe5a4053530 con 0x7fe5a80c2e30 2023-03-03T18:48:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.236+0000 7fe5afeb5700 1 -- 172.21.15.7:0/1545097791 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5a8074bb0 con 0x7fe5a80c2e30 2023-03-03T18:48:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.236+0000 7fe59e7fc700 1 -- 172.21.15.7:0/1545097791 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe5a4020090 con 0x7fe5a80c2e30 2023-03-03T18:48:51.237 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:48:51.237 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:48:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.238+0000 7fe5afeb5700 1 -- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe58c0682f0 msgr2=0x7fe58c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.238+0000 7fe5afeb5700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe58c0682f0 0x7fe58c06a7a0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fe598000f60 tx=0x7fe598009040 comp rx=0 tx=0).stop 2023-03-03T18:48:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.238+0000 7fe5afeb5700 1 -- 172.21.15.7:0/1545097791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 msgr2=0x7fe5a80bf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.238+0000 7fe5afeb5700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 0x7fe5a80bf820 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4014040 tx=0x7fe5a4004750 comp rx=0 tx=0).stop 2023-03-03T18:48:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.238+0000 7fe5afeb5700 1 -- 172.21.15.7:0/1545097791 shutdown_connections 2023-03-03T18:48:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.239+0000 7fe5afeb5700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe5a8151cd0 0x7fe5a80c01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.239+0000 7fe5afeb5700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe5a806c6c0 0x7fe5a8153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.239+0000 7fe5afeb5700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe58c0682f0 0x7fe58c06a7a0 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.239+0000 7fe5afeb5700 1 --2- 172.21.15.7:0/1545097791 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5a80c2e30 0x7fe5a80bf820 unknown :-1 s=CLOSED pgs=3300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.239+0000 7fe5afeb5700 1 -- 172.21.15.7:0/1545097791 >> 172.21.15.7:0/1545097791 conn(0x7fe5a80a94d0 msgr2=0x7fe5a8070fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:48:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.239+0000 7fe5afeb5700 1 -- 172.21.15.7:0/1545097791 shutdown_connections 2023-03-03T18:48:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:51.239+0000 7fe5afeb5700 1 -- 172.21.15.7:0/1545097791 wait complete. 2023-03-03T18:48:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:48:51.250 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before thrashing 2023-03-03T18:48:55.252 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:48:55.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.391+0000 7fca34245700 1 Processor -- start 2023-03-03T18:48:55.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.391+0000 7fca34245700 1 -- start start 2023-03-03T18:48:55.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.392+0000 7fca34245700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14b0d0 0x7fca2c071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:55.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.392+0000 7fca34245700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca2c0aad20 0x7fca2c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:55.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.392+0000 7fca34245700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2c0a9c20 con 0x7fca2c071d40 2023-03-03T18:48:55.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.392+0000 7fca34245700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2c154f10 con 0x7fca2c0aad20 2023-03-03T18:48:55.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.392+0000 7fca34245700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c0ac3c0 con 0x7fca2c14b0d0 2023-03-03T18:48:55.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca317e0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca2c0aad20 0x7fca2c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:55.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca327e2700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fca2c071d40 0x7fca2c158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:36300/0 (socket says 172.21.15.7:36300) 2023-03-03T18:48:55.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca327e2700 1 -- 172.21.15.7:0/1119470410 learned_addr learned my addr 172.21.15.7:0/1119470410 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:48:55.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca317e0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca2c0aad20 0x7fca2c157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:34572/0 (socket says 172.21.15.7:34572) 2023-03-03T18:48:55.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca31fe1700 1 --2- 172.21.15.7:0/1119470410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14b0d0 0x7fca2c071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:55.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca30fdf700 1 -- 172.21.15.7:0/1119470410 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336724874 0 0) 0x7fca2c0a9c20 con 0x7fca2c071d40 2023-03-03T18:48:55.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca30fdf700 1 -- 172.21.15.7:0/1119470410 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca180035b0 con 0x7fca2c071d40 2023-03-03T18:48:55.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca31fe1700 1 -- 172.21.15.7:0/1119470410 >> v2:172.21.15.143:3300/0 conn(0x7fca2c0aad20 msgr2=0x7fca2c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:48:55.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca31fe1700 1 --2- 172.21.15.7:0/1119470410 >> v2:172.21.15.143:3300/0 conn(0x7fca2c0aad20 0x7fca2c157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca31fe1700 1 -- 172.21.15.7:0/1119470410 >> v1:172.21.15.143:6789/0 conn(0x7fca2c071d40 legacy=0x7fca2c158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca31fe1700 1 -- 172.21.15.7:0/1119470410 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c06fc70 con 0x7fca2c14b0d0 2023-03-03T18:48:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.393+0000 7fca317e0700 1 --2- 172.21.15.7:0/1119470410 >> v2:172.21.15.143:3300/0 conn(0x7fca2c0aad20 0x7fca2c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:48:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca31fe1700 1 --2- 172.21.15.7:0/1119470410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14b0d0 0x7fca2c071800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009840 tx=0x7fca1c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecc2434cd89ba040 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca30fdf700 1 -- 172.21.15.7:0/1119470410 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1c011830 con 0x7fca2c14b0d0 2023-03-03T18:48:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca30fdf700 1 -- 172.21.15.7:0/1119470410 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1c011990 con 0x7fca2c14b0d0 2023-03-03T18:48:55.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca30fdf700 1 -- 172.21.15.7:0/1119470410 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1c011d20 con 0x7fca2c14b0d0 2023-03-03T18:48:55.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca34245700 1 -- 172.21.15.7:0/1119470410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14b0d0 msgr2=0x7fca2c071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca34245700 1 --2- 172.21.15.7:0/1119470410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14b0d0 0x7fca2c071800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009840 tx=0x7fca1c009b50 comp rx=0 tx=0).stop 2023-03-03T18:48:55.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca34245700 1 -- 172.21.15.7:0/1119470410 shutdown_connections 2023-03-03T18:48:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca34245700 1 --2- 172.21.15.7:0/1119470410 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c14b0d0 0x7fca2c071800 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca34245700 1 --2- 172.21.15.7:0/1119470410 >> v2:172.21.15.143:3300/0 conn(0x7fca2c0aad20 0x7fca2c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca34245700 1 -- 172.21.15.7:0/1119470410 >> 172.21.15.7:0/1119470410 conn(0x7fca2c0c8900 msgr2=0x7fca2c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:48:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca34245700 1 -- 172.21.15.7:0/1119470410 shutdown_connections 2023-03-03T18:48:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.394+0000 7fca34245700 1 -- 172.21.15.7:0/1119470410 wait complete. 2023-03-03T18:48:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca34245700 1 Processor -- start 2023-03-03T18:48:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca34245700 1 -- start start 2023-03-03T18:48:55.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca34245700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c071d40 0x7fca2c0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca34245700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c0aad20 0x7fca2c1eafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca34245700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2c14b0d0 0x7fca2c1eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca34245700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c1573a0 con 0x7fca2c14b0d0 2023-03-03T18:48:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca34245700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c1eba60 con 0x7fca2c071d40 2023-03-03T18:48:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca34245700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c157830 con 0x7fca2c0aad20 2023-03-03T18:48:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca327e2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2c14b0d0 0x7fca2c1eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:55.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca31fe1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c071d40 0x7fca2c0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca317e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c0aad20 0x7fca2c1eafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca31fe1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c071d40 0x7fca2c0c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:34576/0 (socket says 172.21.15.7:34576) 2023-03-03T18:48:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.396+0000 7fca31fe1700 1 -- 172.21.15.7:0/1081827097 learned_addr learned my addr 172.21.15.7:0/1081827097 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:48:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca327e2700 1 -- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c0aad20 msgr2=0x7fca2c1eafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca327e2700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c0aad20 0x7fca2c1eafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:55.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca327e2700 1 -- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c071d40 msgr2=0x7fca2c0c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca327e2700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c071d40 0x7fca2c0c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca327e2700 1 -- 172.21.15.7:0/1081827097 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca280094a0 con 0x7fca2c14b0d0 2023-03-03T18:48:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca317e0700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c0aad20 0x7fca2c1eafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:48:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca327e2700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2c14b0d0 0x7fca2c1eb520 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 crypto rx=0x7fca2400caa0 tx=0x7fca2400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:55.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca227fc700 1 -- 172.21.15.7:0/1081827097 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2400a9e0 con 0x7fca2c14b0d0 2023-03-03T18:48:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca227fc700 1 -- 172.21.15.7:0/1081827097 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2400ab40 con 0x7fca2c14b0d0 2023-03-03T18:48:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca34245700 1 -- 172.21.15.7:0/1081827097 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c0094a0 con 0x7fca2c14b0d0 2023-03-03T18:48:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca227fc700 1 -- 172.21.15.7:0/1081827097 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24012400 con 0x7fca2c14b0d0 2023-03-03T18:48:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.397+0000 7fca34245700 1 -- 172.21.15.7:0/1081827097 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c06f6b0 con 0x7fca2c14b0d0 2023-03-03T18:48:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.398+0000 7fca34245700 1 -- 172.21.15.7:0/1081827097 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c155160 con 0x7fca2c14b0d0 2023-03-03T18:48:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.399+0000 7fca227fc700 1 -- 172.21.15.7:0/1081827097 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fca24019030 con 0x7fca2c14b0d0 2023-03-03T18:48:55.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.399+0000 7fca227fc700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca100682f0 0x7fca1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:48:55.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.399+0000 7fca227fc700 1 -- 172.21.15.7:0/1081827097 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fca24085a10 con 0x7fca2c14b0d0 2023-03-03T18:48:55.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.399+0000 7fca31fe1700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca100682f0 0x7fca1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:48:55.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.400+0000 7fca31fe1700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca100682f0 0x7fca1006a7a0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fca2c1540f0 tx=0x7fca1c01d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:48:55.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.403+0000 7fca227fc700 1 -- 172.21.15.7:0/1081827097 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fca2404b5e0 con 0x7fca2c14b0d0 2023-03-03T18:48:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.641+0000 7fca34245700 1 -- 172.21.15.7:0/1081827097 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca2c06f990 con 0x7fca2c14b0d0 2023-03-03T18:48:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.641+0000 7fca227fc700 1 -- 172.21.15.7:0/1081827097 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fca24051e60 con 0x7fca2c14b0d0 2023-03-03T18:48:55.642 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:48:55.642 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:48:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.643+0000 7fca34245700 1 -- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca100682f0 msgr2=0x7fca1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.643+0000 7fca34245700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca100682f0 0x7fca1006a7a0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fca2c1540f0 tx=0x7fca1c01d040 comp rx=0 tx=0).stop 2023-03-03T18:48:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.643+0000 7fca34245700 1 -- 172.21.15.7:0/1081827097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2c14b0d0 msgr2=0x7fca2c1eb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:48:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.643+0000 7fca34245700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2c14b0d0 0x7fca2c1eb520 secure :-1 s=READY pgs=3301 cs=0 l=1 rev1=1 crypto rx=0x7fca2400caa0 tx=0x7fca2400cdb0 comp rx=0 tx=0).stop 2023-03-03T18:48:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.644+0000 7fca34245700 1 -- 172.21.15.7:0/1081827097 shutdown_connections 2023-03-03T18:48:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.644+0000 7fca34245700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca100682f0 0x7fca1006a7a0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.644+0000 7fca34245700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2c14b0d0 0x7fca2c1eb520 unknown :-1 s=CLOSED pgs=3301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.644+0000 7fca34245700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca2c071d40 0x7fca2c0c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.644+0000 7fca34245700 1 --2- 172.21.15.7:0/1081827097 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2c0aad20 0x7fca2c1eafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:48:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.644+0000 7fca34245700 1 -- 172.21.15.7:0/1081827097 >> 172.21.15.7:0/1081827097 conn(0x7fca2c0c8900 msgr2=0x7fca2c0acaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:48:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.644+0000 7fca34245700 1 -- 172.21.15.7:0/1081827097 shutdown_connections 2023-03-03T18:48:55.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:48:55.644+0000 7fca34245700 1 -- 172.21.15.7:0/1081827097 wait complete. 2023-03-03T18:48:55.646 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:48:55.655 INFO:tasks.mds_thrash.fs.[a]:waiting for 39 secs before thrashing 2023-03-03T18:49:12.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:49:12.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:49:32.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:49:32.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.345+0000 7feaef609700 1 Processor -- start 2023-03-03T18:49:32.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.345+0000 7feaef609700 1 -- start start 2023-03-03T18:49:32.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.346+0000 7feaef609700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7feae80c2930 0x7feae8157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:32.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.346+0000 7feaef609700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae806c480 0x7feae8157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:32.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.346+0000 7feaef609700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae806d430 con 0x7feae8151d40 2023-03-03T18:49:32.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.346+0000 7feaef609700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae806c2a0 con 0x7feae80c2930 2023-03-03T18:49:32.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.346+0000 7feaef609700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae806cb00 con 0x7feae806c480 2023-03-03T18:49:32.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.346+0000 7feaed3a5700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7feae8151d40 0x7feae80736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:32816/0 (socket says 172.21.15.7:32816) 2023-03-03T18:49:32.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.346+0000 7feaed3a5700 1 -- 172.21.15.7:0/2281564760 learned_addr learned my addr 172.21.15.7:0/2281564760 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:49:32.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.346+0000 7feaedba6700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae806c480 0x7feae8157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:32.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.346+0000 7feaecba4700 1 --2- 172.21.15.7:0/2281564760 >> v2:172.21.15.143:3300/0 conn(0x7feae80c2930 0x7feae8157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:32.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.346+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510876726 0 0) 0x7feae806d430 con 0x7feae8151d40 2023-03-03T18:49:32.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead40035b0 con 0x7feae8151d40 2023-03-03T18:49:32.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1621298631 0 0) 0x7fead40035b0 con 0x7feae8151d40 2023-03-03T18:49:32.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae806c480 msgr2=0x7feae8157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:49:32.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae806c480 0x7feae8157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:32.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 >> v2:172.21.15.143:3300/0 conn(0x7feae80c2930 msgr2=0x7feae8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:49:32.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 --2- 172.21.15.7:0/2281564760 >> v2:172.21.15.143:3300/0 conn(0x7feae80c2930 0x7feae8157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:32.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae806cde0 con 0x7feae8151d40 2023-03-03T18:49:32.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fead8001670 con 0x7feae8151d40 2023-03-03T18:49:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 >> v1:172.21.15.7:6789/0 conn(0x7feae8151d40 legacy=0x7feae80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fead4003a70 0x7fead4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae806c480 0x7fead4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fead4004d90 0x7fead4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae806c2a0 con 0x7fead4004d90 2023-03-03T18:49:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae806cb00 con 0x7fead4003a70 2023-03-03T18:49:32.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead400bb40 con 0x7feae806c480 2023-03-03T18:49:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feaedba6700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fead4004d90 0x7fead4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feaed3a5700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fead4003a70 0x7fead4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feaecba4700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae806c480 0x7fead4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feaedba6700 1 -- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae806c480 msgr2=0x7fead4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:49:32.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feaedba6700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae806c480 0x7fead4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:32.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feaedba6700 1 -- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fead4003a70 msgr2=0x7fead4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:49:32.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feaedba6700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fead4003a70 0x7fead4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:32.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.347+0000 7feaedba6700 1 -- 172.21.15.7:0/2281564760 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead40035b0 con 0x7fead4004d90 2023-03-03T18:49:32.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feaedba6700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fead4004d90 0x7fead4005200 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7feae0009fc0 tx=0x7fead4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5617de089985e6dd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:49:32.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae000ec30 con 0x7fead4004d90 2023-03-03T18:49:32.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae000f040 con 0x7fead4004d90 2023-03-03T18:49:32.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feadffff700 1 -- 172.21.15.7:0/2281564760 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae0014480 con 0x7fead4004d90 2023-03-03T18:49:32.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feaef609700 1 -- 172.21.15.7:0/2281564760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fead4004d90 msgr2=0x7fead4005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:32.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feaef609700 1 --2- 172.21.15.7:0/2281564760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fead4004d90 0x7fead4005200 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7feae0009fc0 tx=0x7fead4005880 comp rx=0 tx=0).stop 2023-03-03T18:49:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feaedba6700 1 -- 172.21.15.7:0/2281564760 reap_dead start 2023-03-03T18:49:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feaef609700 1 -- 172.21.15.7:0/2281564760 shutdown_connections 2023-03-03T18:49:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feaef609700 1 -- 172.21.15.7:0/2281564760 >> 172.21.15.7:0/2281564760 conn(0x7feae80a94d0 msgr2=0x7feae80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:49:32.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feaef609700 1 -- 172.21.15.7:0/2281564760 shutdown_connections 2023-03-03T18:49:32.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.348+0000 7feaef609700 1 -- 172.21.15.7:0/2281564760 wait complete. 2023-03-03T18:49:32.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.350+0000 7feaef609700 1 Processor -- start 2023-03-03T18:49:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.350+0000 7feaef609700 1 -- start start 2023-03-03T18:49:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.350+0000 7feaef609700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae80c6420 0x7feae8155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.350+0000 7feaef609700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae80c67f0 0x7feae81ea1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.350+0000 7feaef609700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae81ea6e0 0x7feae80be430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.350+0000 7feaef609700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae8152f10 con 0x7feae80c67f0 2023-03-03T18:49:32.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.350+0000 7feaef609700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae81531f0 con 0x7feae80c6420 2023-03-03T18:49:32.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.350+0000 7feaef609700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae81534d0 con 0x7feae81ea6e0 2023-03-03T18:49:32.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.350+0000 7feaecba4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae80c67f0 0x7feae81ea1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:32.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaecba4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae80c67f0 0x7feae81ea1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38850/0 (socket says 172.21.15.7:38850) 2023-03-03T18:49:32.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaecba4700 1 -- 172.21.15.7:0/691098232 learned_addr learned my addr 172.21.15.7:0/691098232 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:49:32.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaedba6700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae81ea6e0 0x7feae80be430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:32.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaecba4700 1 -- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae81ea6e0 msgr2=0x7feae80be430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:32.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaed3a5700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae80c6420 0x7feae8155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:32.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaecba4700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae81ea6e0 0x7feae80be430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:32.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaecba4700 1 -- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae80c6420 msgr2=0x7feae8155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:32.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaecba4700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae80c6420 0x7feae8155140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:32.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaecba4700 1 -- 172.21.15.7:0/691098232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae8073e00 con 0x7feae80c67f0 2023-03-03T18:49:32.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaedba6700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae81ea6e0 0x7feae80be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:49:32.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaecba4700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae80c67f0 0x7feae81ea1a0 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7feae4011aa0 tx=0x7feae4011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:49:32.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaddffb700 1 -- 172.21.15.7:0/691098232 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae400f9e0 con 0x7feae80c67f0 2023-03-03T18:49:32.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaddffb700 1 -- 172.21.15.7:0/691098232 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae400fb40 con 0x7feae80c67f0 2023-03-03T18:49:32.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaddffb700 1 -- 172.21.15.7:0/691098232 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae4014440 con 0x7feae80c67f0 2023-03-03T18:49:32.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaef609700 1 -- 172.21.15.7:0/691098232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae806d0c0 con 0x7feae80c67f0 2023-03-03T18:49:32.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.351+0000 7feaef609700 1 -- 172.21.15.7:0/691098232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae806bea0 con 0x7feae80c67f0 2023-03-03T18:49:32.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.352+0000 7feaef609700 1 -- 172.21.15.7:0/691098232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae8074bb0 con 0x7feae80c67f0 2023-03-03T18:49:32.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.353+0000 7feaddffb700 1 -- 172.21.15.7:0/691098232 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feae40145a0 con 0x7feae80c67f0 2023-03-03T18:49:32.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.353+0000 7feaddffb700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feacc0682f0 0x7feacc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:32.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.354+0000 7feaddffb700 1 -- 172.21.15.7:0/691098232 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7feae4086af0 con 0x7feae80c67f0 2023-03-03T18:49:32.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.354+0000 7feaed3a5700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feacc0682f0 0x7feacc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:32.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.354+0000 7feaed3a5700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feacc0682f0 0x7feacc06a7a0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7fead8001120 tx=0x7fead8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:49:32.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.358+0000 7feaddffb700 1 -- 172.21.15.7:0/691098232 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feae4052e30 con 0x7feae80c67f0 2023-03-03T18:49:32.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.599+0000 7feaef609700 1 -- 172.21.15.7:0/691098232 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feae80a9fb0 con 0x7feae80c67f0 2023-03-03T18:49:32.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.599+0000 7feaddffb700 1 -- 172.21.15.7:0/691098232 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7feae4056450 con 0x7feae80c67f0 2023-03-03T18:49:32.601 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:49:32.601 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:49:32.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.601+0000 7feaef609700 1 -- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feacc0682f0 msgr2=0x7feacc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.601+0000 7feaef609700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feacc0682f0 0x7feacc06a7a0 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7fead8001120 tx=0x7fead8009040 comp rx=0 tx=0).stop 2023-03-03T18:49:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.601+0000 7feaef609700 1 -- 172.21.15.7:0/691098232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae80c67f0 msgr2=0x7feae81ea1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.601+0000 7feaef609700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae80c67f0 0x7feae81ea1a0 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7feae4011aa0 tx=0x7feae4011e60 comp rx=0 tx=0).stop 2023-03-03T18:49:32.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.601+0000 7feaef609700 1 -- 172.21.15.7:0/691098232 shutdown_connections 2023-03-03T18:49:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.601+0000 7feaef609700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae81ea6e0 0x7feae80be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.601+0000 7feaef609700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae80c6420 0x7feae8155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.602+0000 7feaef609700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feacc0682f0 0x7feacc06a7a0 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.602+0000 7feaef609700 1 --2- 172.21.15.7:0/691098232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae80c67f0 0x7feae81ea1a0 unknown :-1 s=CLOSED pgs=3304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:32.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.602+0000 7feaef609700 1 -- 172.21.15.7:0/691098232 >> 172.21.15.7:0/691098232 conn(0x7feae80a94d0 msgr2=0x7feae81582f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:49:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.602+0000 7feaef609700 1 -- 172.21.15.7:0/691098232 shutdown_connections 2023-03-03T18:49:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:32.602+0000 7feaef609700 1 -- 172.21.15.7:0/691098232 wait complete. 2023-03-03T18:49:32.605 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:49:32.615 INFO:tasks.mds_thrash.fs.[b]:waiting for 57 secs before thrashing 2023-03-03T18:49:34.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:49:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.791+0000 7fcde30f7700 1 Processor -- start 2023-03-03T18:49:34.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.791+0000 7fcde30f7700 1 -- start start 2023-03-03T18:49:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde30f7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcddc06d4b0 0x7fcddc06d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:34.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde30f7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcddc0c8020 0x7fcddc0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde30f7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcddc0c7b90 con 0x7fcddc0c1da0 2023-03-03T18:49:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde30f7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc0c7330 con 0x7fcddc06d4b0 2023-03-03T18:49:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde30f7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcddc154420 con 0x7fcddc0c8020 2023-03-03T18:49:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde1694700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcddc0c1da0 0x7fcddc139d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:32836/0 (socket says 172.21.15.7:32836) 2023-03-03T18:49:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde1694700 1 -- 172.21.15.7:0/4237213822 learned_addr learned my addr 172.21.15.7:0/4237213822 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:49:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde0e93700 1 --2- 172.21.15.7:0/4237213822 >> v2:172.21.15.7:3300/0 conn(0x7fcddc06d4b0 0x7fcddc06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcddbfff700 1 --2- 172.21.15.7:0/4237213822 >> v2:172.21.15.143:3300/0 conn(0x7fcddc0c8020 0x7fcddc0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:34.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcddb7fe700 1 -- 172.21.15.7:0/4237213822 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010275223 0 0) 0x7fcddc0c7b90 con 0x7fcddc0c1da0 2023-03-03T18:49:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcddb7fe700 1 -- 172.21.15.7:0/4237213822 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdcc0035b0 con 0x7fcddc0c1da0 2023-03-03T18:49:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde0e93700 1 -- 172.21.15.7:0/4237213822 >> v2:172.21.15.143:3300/0 conn(0x7fcddc0c8020 msgr2=0x7fcddc0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:49:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde0e93700 1 --2- 172.21.15.7:0/4237213822 >> v2:172.21.15.143:3300/0 conn(0x7fcddc0c8020 0x7fcddc0c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde0e93700 1 -- 172.21.15.7:0/4237213822 >> v1:172.21.15.7:6789/0 conn(0x7fcddc0c1da0 legacy=0x7fcddc139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.792+0000 7fcde0e93700 1 -- 172.21.15.7:0/4237213822 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc154640 con 0x7fcddc06d4b0 2023-03-03T18:49:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcde0e93700 1 --2- 172.21.15.7:0/4237213822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc06d4b0 0x7fcddc06d880 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7fcddc06fcb0 tx=0x7fcdd0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbcc5ed15d9b6ae2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:49:34.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcddb7fe700 1 -- 172.21.15.7:0/4237213822 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0011830 con 0x7fcddc06d4b0 2023-03-03T18:49:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcddb7fe700 1 -- 172.21.15.7:0/4237213822 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd0011990 con 0x7fcddc06d4b0 2023-03-03T18:49:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcddb7fe700 1 -- 172.21.15.7:0/4237213822 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0011d20 con 0x7fcddc06d4b0 2023-03-03T18:49:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcde30f7700 1 -- 172.21.15.7:0/4237213822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc06d4b0 msgr2=0x7fcddc06d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcde30f7700 1 --2- 172.21.15.7:0/4237213822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc06d4b0 0x7fcddc06d880 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7fcddc06fcb0 tx=0x7fcdd0009b20 comp rx=0 tx=0).stop 2023-03-03T18:49:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcde30f7700 1 -- 172.21.15.7:0/4237213822 shutdown_connections 2023-03-03T18:49:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcde30f7700 1 --2- 172.21.15.7:0/4237213822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc06d4b0 0x7fcddc06d880 unknown :-1 s=CLOSED pgs=3306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:34.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcde30f7700 1 --2- 172.21.15.7:0/4237213822 >> v2:172.21.15.143:3300/0 conn(0x7fcddc0c8020 0x7fcddc0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcde30f7700 1 -- 172.21.15.7:0/4237213822 >> 172.21.15.7:0/4237213822 conn(0x7fcddc06c110 msgr2=0x7fcddc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:49:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcde30f7700 1 -- 172.21.15.7:0/4237213822 shutdown_connections 2023-03-03T18:49:34.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.793+0000 7fcde30f7700 1 -- 172.21.15.7:0/4237213822 wait complete. 2023-03-03T18:49:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.795+0000 7fcde30f7700 1 Processor -- start 2023-03-03T18:49:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.795+0000 7fcde30f7700 1 -- start start 2023-03-03T18:49:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.795+0000 7fcde30f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcddc06d4b0 0x7fcddc0c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.795+0000 7fcde30f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcddc0c1da0 0x7fcddc135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:34.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.795+0000 7fcde30f7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc0c8020 0x7fcddc1364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.795+0000 7fcde30f7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc0ad440 con 0x7fcddc0c8020 2023-03-03T18:49:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.795+0000 7fcde30f7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc139b50 con 0x7fcddc06d4b0 2023-03-03T18:49:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.795+0000 7fcde30f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc136c50 con 0x7fcddc0c1da0 2023-03-03T18:49:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde1694700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc0c8020 0x7fcddc1364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde1694700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc0c8020 0x7fcddc1364f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38866/0 (socket says 172.21.15.7:38866) 2023-03-03T18:49:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde1694700 1 -- 172.21.15.7:0/2392928810 learned_addr learned my addr 172.21.15.7:0/2392928810 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:49:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcddbfff700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcddc0c1da0 0x7fcddc135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde0e93700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcddc06d4b0 0x7fcddc0c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde1694700 1 -- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcddc0c1da0 msgr2=0x7fcddc135fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde1694700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcddc0c1da0 0x7fcddc135fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde1694700 1 -- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcddc06d4b0 msgr2=0x7fcddc0c37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde1694700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcddc06d4b0 0x7fcddc0c37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde1694700 1 -- 172.21.15.7:0/2392928810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd00094a0 con 0x7fcddc0c8020 2023-03-03T18:49:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcddbfff700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcddc0c1da0 0x7fcddc135fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:49:34.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde1694700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc0c8020 0x7fcddc1364f0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400ca70 tx=0x7fcdd400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:49:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcdd97fa700 1 -- 172.21.15.7:0/2392928810 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd400aa40 con 0x7fcddc0c8020 2023-03-03T18:49:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcdd97fa700 1 -- 172.21.15.7:0/2392928810 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd400aba0 con 0x7fcddc0c8020 2023-03-03T18:49:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcdd97fa700 1 -- 172.21.15.7:0/2392928810 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4012400 con 0x7fcddc0c8020 2023-03-03T18:49:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde30f7700 1 -- 172.21.15.7:0/2392928810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcddc0c7820 con 0x7fcddc0c8020 2023-03-03T18:49:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.796+0000 7fcde30f7700 1 -- 172.21.15.7:0/2392928810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcddc0c7540 con 0x7fcddc0c8020 2023-03-03T18:49:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.797+0000 7fcde30f7700 1 -- 172.21.15.7:0/2392928810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcddc06ddc0 con 0x7fcddc0c8020 2023-03-03T18:49:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.798+0000 7fcdd97fa700 1 -- 172.21.15.7:0/2392928810 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcdd4019030 con 0x7fcddc0c8020 2023-03-03T18:49:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.798+0000 7fcdd97fa700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc00682f0 0x7fcdc006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:49:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.798+0000 7fcdd97fa700 1 -- 172.21.15.7:0/2392928810 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcdd4085b70 con 0x7fcddc0c8020 2023-03-03T18:49:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.798+0000 7fcde0e93700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc00682f0 0x7fcdc006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:49:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.799+0000 7fcde0e93700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc00682f0 0x7fcdc006a7a0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00090b0 tx=0x7fcdd001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:49:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:34.803+0000 7fcdd97fa700 1 -- 172.21.15.7:0/2392928810 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcdd404b5e0 con 0x7fcddc0c8020 2023-03-03T18:49:35.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.046+0000 7fcde30f7700 1 -- 172.21.15.7:0/2392928810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcddc074bb0 con 0x7fcddc0c8020 2023-03-03T18:49:35.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.046+0000 7fcdd97fa700 1 -- 172.21.15.7:0/2392928810 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcdd4051fc0 con 0x7fcddc0c8020 2023-03-03T18:49:35.048 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:49:35.048 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:49:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 -- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc00682f0 msgr2=0x7fcdc006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc00682f0 0x7fcdc006a7a0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00090b0 tx=0x7fcdd001c040 comp rx=0 tx=0).stop 2023-03-03T18:49:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 -- 172.21.15.7:0/2392928810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc0c8020 msgr2=0x7fcddc1364f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:49:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc0c8020 0x7fcddc1364f0 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400ca70 tx=0x7fcdd400ce30 comp rx=0 tx=0).stop 2023-03-03T18:49:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 -- 172.21.15.7:0/2392928810 shutdown_connections 2023-03-03T18:49:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcdc00682f0 0x7fcdc006a7a0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:35.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcddc0c8020 0x7fcddc1364f0 unknown :-1 s=CLOSED pgs=3307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcddc06d4b0 0x7fcddc0c37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 --2- 172.21.15.7:0/2392928810 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcddc0c1da0 0x7fcddc135fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:49:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 -- 172.21.15.7:0/2392928810 >> 172.21.15.7:0/2392928810 conn(0x7fcddc06c110 msgr2=0x7fcddc1549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:49:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 -- 172.21.15.7:0/2392928810 shutdown_connections 2023-03-03T18:49:35.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:49:35.048+0000 7fcde30f7700 1 -- 172.21.15.7:0/2392928810 wait complete. 2023-03-03T18:49:35.052 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:49:35.062 INFO:tasks.mds_thrash.fs.[a]:waiting for 36 secs before thrashing 2023-03-03T18:49:42.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:49:42.512 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:50:00.009 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T18:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T18:50:11.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:50:11.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.203+0000 7fe3d8088700 1 Processor -- start 2023-03-03T18:50:11.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.204+0000 7fe3d8088700 1 -- start start 2023-03-03T18:50:11.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.204+0000 7fe3d8088700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe3d0155410 0x7fe3d00bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:11.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.204+0000 7fe3d8088700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d00c5b00 0x7fe3d00bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:11.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.204+0000 7fe3d8088700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d00c08d0 con 0x7fe3d0151e80 2023-03-03T18:50:11.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.204+0000 7fe3d8088700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d0152250 con 0x7fe3d0155410 2023-03-03T18:50:11.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.204+0000 7fe3d8088700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d00c0fb0 con 0x7fe3d00c5b00 2023-03-03T18:50:11.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d5e24700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe3d0151e80 0x7fe3d00c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:32870/0 (socket says 172.21.15.7:32870) 2023-03-03T18:50:11.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d5e24700 1 -- 172.21.15.7:0/1856197507 learned_addr learned my addr 172.21.15.7:0/1856197507 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:50:11.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d6625700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d00c5b00 0x7fe3d00bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:11.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d5623700 1 --2- 172.21.15.7:0/1856197507 >> v2:172.21.15.143:3300/0 conn(0x7fe3d0155410 0x7fe3d00bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639124569 0 0) 0x7fe3d00c08d0 con 0x7fe3d0151e80 2023-03-03T18:50:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3bc0035b0 con 0x7fe3d0151e80 2023-03-03T18:50:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2038093100 0 0) 0x7fe3bc0035b0 con 0x7fe3d0151e80 2023-03-03T18:50:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d00c5b00 msgr2=0x7fe3d00bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:50:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d00c5b00 0x7fe3d00bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 >> v2:172.21.15.143:3300/0 conn(0x7fe3d0155410 msgr2=0x7fe3d00bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:50:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 --2- 172.21.15.7:0/1856197507 >> v2:172.21.15.143:3300/0 conn(0x7fe3d0155410 0x7fe3d00bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:11.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d0154930 con 0x7fe3d0151e80 2023-03-03T18:50:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fe3cc001670 con 0x7fe3d0151e80 2023-03-03T18:50:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 >> v1:172.21.15.7:6789/0 conn(0x7fe3d0151e80 legacy=0x7fe3d00c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3bc003a70 0x7fe3bc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d00c5b00 0x7fe3bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3bc004d90 0x7fe3bc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc00ba10 con 0x7fe3bc004d90 2023-03-03T18:50:11.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc00bb50 con 0x7fe3bc003a70 2023-03-03T18:50:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc00bc90 con 0x7fe3d00c5b00 2023-03-03T18:50:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d6625700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3bc004d90 0x7fe3bc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d5e24700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3bc003a70 0x7fe3bc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.205+0000 7fe3d5623700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d00c5b00 0x7fe3bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d6625700 1 -- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d00c5b00 msgr2=0x7fe3bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:50:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d6625700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d00c5b00 0x7fe3bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d6625700 1 -- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3bc003a70 msgr2=0x7fe3bc003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:50:11.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d6625700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3bc003a70 0x7fe3bc003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d6625700 1 -- 172.21.15.7:0/1856197507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0035b0 con 0x7fe3bc004d90 2023-03-03T18:50:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d6625700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3bc004d90 0x7fe3bc005200 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009fc0 tx=0x7fe3bc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e761fdbc0ab953 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:50:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c800ec30 con 0x7fe3bc004d90 2023-03-03T18:50:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c800f040 con 0x7fe3bc004d90 2023-03-03T18:50:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d4e22700 1 -- 172.21.15.7:0/1856197507 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8014480 con 0x7fe3bc004d90 2023-03-03T18:50:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d8088700 1 -- 172.21.15.7:0/1856197507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3bc004d90 msgr2=0x7fe3bc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.206+0000 7fe3d8088700 1 --2- 172.21.15.7:0/1856197507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3bc004d90 0x7fe3bc005200 secure :-1 s=READY pgs=3309 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009fc0 tx=0x7fe3bc005880 comp rx=0 tx=0).stop 2023-03-03T18:50:11.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.207+0000 7fe3d6625700 1 -- 172.21.15.7:0/1856197507 reap_dead start 2023-03-03T18:50:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.207+0000 7fe3d8088700 1 -- 172.21.15.7:0/1856197507 shutdown_connections 2023-03-03T18:50:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.207+0000 7fe3d8088700 1 -- 172.21.15.7:0/1856197507 >> 172.21.15.7:0/1856197507 conn(0x7fe3d0071630 msgr2=0x7fe3d006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:50:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.207+0000 7fe3d8088700 1 -- 172.21.15.7:0/1856197507 shutdown_connections 2023-03-03T18:50:11.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.207+0000 7fe3d8088700 1 -- 172.21.15.7:0/1856197507 wait complete. 2023-03-03T18:50:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.208+0000 7fe3d8088700 1 Processor -- start 2023-03-03T18:50:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.208+0000 7fe3d8088700 1 -- start start 2023-03-03T18:50:11.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d8088700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3bc004d90 0x7fe3d01527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d8088700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d0152d20 0x7fe3d0157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d8088700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3d01583d0 0x7fe3d014bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d8088700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d0157090 con 0x7fe3d01583d0 2023-03-03T18:50:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d8088700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d01ede60 con 0x7fe3bc004d90 2023-03-03T18:50:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d8088700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d01ee390 con 0x7fe3d0152d20 2023-03-03T18:50:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d6625700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3d01583d0 0x7fe3d014bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d6625700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3d01583d0 0x7fe3d014bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38906/0 (socket says 172.21.15.7:38906) 2023-03-03T18:50:11.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d6625700 1 -- 172.21.15.7:0/1671384854 learned_addr learned my addr 172.21.15.7:0/1671384854 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:50:11.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d5e24700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3bc004d90 0x7fe3d01527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:11.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d5623700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d0152d20 0x7fe3d0157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:11.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d6625700 1 -- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d0152d20 msgr2=0x7fe3d0157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:11.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d6625700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d0152d20 0x7fe3d0157e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:11.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d6625700 1 -- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3bc004d90 msgr2=0x7fe3d01527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:11.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d6625700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3bc004d90 0x7fe3d01527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:11.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d6625700 1 -- 172.21.15.7:0/1671384854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d0154ef0 con 0x7fe3d01583d0 2023-03-03T18:50:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d6625700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3d01583d0 0x7fe3d014bbb0 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800e700 tx=0x7fe3c8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:50:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3c67fc700 1 -- 172.21.15.7:0/1671384854 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8007000 con 0x7fe3d01583d0 2023-03-03T18:50:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3c67fc700 1 -- 172.21.15.7:0/1671384854 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c8015b60 con 0x7fe3d01583d0 2023-03-03T18:50:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3d8088700 1 -- 172.21.15.7:0/1671384854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d0154c10 con 0x7fe3d01583d0 2023-03-03T18:50:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.209+0000 7fe3c67fc700 1 -- 172.21.15.7:0/1671384854 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c801d420 con 0x7fe3d01583d0 2023-03-03T18:50:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.210+0000 7fe3d8088700 1 -- 172.21.15.7:0/1671384854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d00c5220 con 0x7fe3d01583d0 2023-03-03T18:50:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.210+0000 7fe3d8088700 1 -- 172.21.15.7:0/1671384854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d01ee110 con 0x7fe3d01583d0 2023-03-03T18:50:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.211+0000 7fe3c67fc700 1 -- 172.21.15.7:0/1671384854 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe3c802d030 con 0x7fe3d01583d0 2023-03-03T18:50:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.211+0000 7fe3c67fc700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3b40682f0 0x7fe3b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.211+0000 7fe3c67fc700 1 -- 172.21.15.7:0/1671384854 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe3c80877c0 con 0x7fe3d01583d0 2023-03-03T18:50:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.211+0000 7fe3d5e24700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3b40682f0 0x7fe3b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.212+0000 7fe3d5e24700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3b40682f0 0x7fe3b406a7a0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc001050 tx=0x7fe3cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:50:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.216+0000 7fe3c67fc700 1 -- 172.21.15.7:0/1671384854 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe3c8053b60 con 0x7fe3d01583d0 2023-03-03T18:50:11.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.456+0000 7fe3d8088700 1 -- 172.21.15.7:0/1671384854 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3d0074bb0 con 0x7fe3d01583d0 2023-03-03T18:50:11.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.457+0000 7fe3c67fc700 1 -- 172.21.15.7:0/1671384854 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe3c8057180 con 0x7fe3d01583d0 2023-03-03T18:50:11.458 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:50:11.458 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:50:11.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 -- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3b40682f0 msgr2=0x7fe3b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:11.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3b40682f0 0x7fe3b406a7a0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc001050 tx=0x7fe3cc009040 comp rx=0 tx=0).stop 2023-03-03T18:50:11.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 -- 172.21.15.7:0/1671384854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3d01583d0 msgr2=0x7fe3d014bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:11.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3d01583d0 0x7fe3d014bbb0 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800e700 tx=0x7fe3c8009750 comp rx=0 tx=0).stop 2023-03-03T18:50:11.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 -- 172.21.15.7:0/1671384854 shutdown_connections 2023-03-03T18:50:11.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3b40682f0 0x7fe3b406a7a0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:11.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3d01583d0 0x7fe3d014bbb0 unknown :-1 s=CLOSED pgs=3310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:11.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe3bc004d90 0x7fe3d01527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:11.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 --2- 172.21.15.7:0/1671384854 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe3d0152d20 0x7fe3d0157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:11.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 -- 172.21.15.7:0/1671384854 >> 172.21.15.7:0/1671384854 conn(0x7fe3d0071630 msgr2=0x7fe3d0151480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:50:11.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 -- 172.21.15.7:0/1671384854 shutdown_connections 2023-03-03T18:50:11.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:11.459+0000 7fe3d8088700 1 -- 172.21.15.7:0/1671384854 wait complete. 2023-03-03T18:50:11.465 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:50:11.471 INFO:tasks.mds_thrash.fs.[a]:waiting for 61 secs before thrashing 2023-03-03T18:50:12.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:50:12.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:50:29.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:50:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.756+0000 7fa78d86a700 1 Processor -- start 2023-03-03T18:50:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.756+0000 7fa78d86a700 1 -- start start 2023-03-03T18:50:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.756+0000 7fa78d86a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7880b00b0 0x7fa7880c7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.756+0000 7fa78d86a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa7880c7590 0x7fa78815ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.756+0000 7fa78d86a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7880c6cf0 con 0x7fa788151cc0 2023-03-03T18:50:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.756+0000 7fa78d86a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa788152090 con 0x7fa7880b00b0 2023-03-03T18:50:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.756+0000 7fa78d86a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7881548f0 con 0x7fa7880c7590 2023-03-03T18:50:29.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa7867fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa7880b00b0 0x7fa7880c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:29.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa786ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa788151cc0 0x7fa7880ca6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:32898/0 (socket says 172.21.15.7:32898) 2023-03-03T18:50:29.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa786ffd700 1 -- 172.21.15.7:0/1592721208 learned_addr learned my addr 172.21.15.7:0/1592721208 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:50:29.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa7877fe700 1 --2- 172.21.15.7:0/1592721208 >> v2:172.21.15.143:3300/0 conn(0x7fa7880c7590 0x7fa78815ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:29.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa7867fc700 1 -- 172.21.15.7:0/1592721208 >> v2:172.21.15.143:3300/0 conn(0x7fa7880c7590 msgr2=0x7fa78815ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:50:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa7867fc700 1 --2- 172.21.15.7:0/1592721208 >> v2:172.21.15.143:3300/0 conn(0x7fa7880c7590 0x7fa78815ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa7867fc700 1 -- 172.21.15.7:0/1592721208 >> v1:172.21.15.7:6789/0 conn(0x7fa788151cc0 legacy=0x7fa7880ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa7867fc700 1 -- 172.21.15.7:0/1592721208 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788154bd0 con 0x7fa7880b00b0 2023-03-03T18:50:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa7877fe700 1 --2- 172.21.15.7:0/1592721208 >> v2:172.21.15.143:3300/0 conn(0x7fa7880c7590 0x7fa78815ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:50:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa7867fc700 1 --2- 172.21.15.7:0/1592721208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 0x7fa7880c7050 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7fa7880c1700 tx=0x7fa77c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbcf3f53f753abd4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:50:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa785ffb700 1 -- 172.21.15.7:0/1592721208 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c011830 con 0x7fa7880b00b0 2023-03-03T18:50:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa785ffb700 1 -- 172.21.15.7:0/1592721208 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77c011990 con 0x7fa7880b00b0 2023-03-03T18:50:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.757+0000 7fa785ffb700 1 -- 172.21.15.7:0/1592721208 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c011d20 con 0x7fa7880b00b0 2023-03-03T18:50:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.758+0000 7fa78d86a700 1 -- 172.21.15.7:0/1592721208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 msgr2=0x7fa7880c7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.758+0000 7fa78d86a700 1 --2- 172.21.15.7:0/1592721208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 0x7fa7880c7050 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7fa7880c1700 tx=0x7fa77c009b20 comp rx=0 tx=0).stop 2023-03-03T18:50:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.758+0000 7fa78d86a700 1 -- 172.21.15.7:0/1592721208 shutdown_connections 2023-03-03T18:50:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.758+0000 7fa78d86a700 1 --2- 172.21.15.7:0/1592721208 >> v2:172.21.15.143:3300/0 conn(0x7fa7880c7590 0x7fa78815ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.758+0000 7fa78d86a700 1 --2- 172.21.15.7:0/1592721208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 0x7fa7880c7050 unknown :-1 s=CLOSED pgs=3312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.758+0000 7fa78d86a700 1 -- 172.21.15.7:0/1592721208 >> 172.21.15.7:0/1592721208 conn(0x7fa78806b980 msgr2=0x7fa788071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:50:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.758+0000 7fa78d86a700 1 -- 172.21.15.7:0/1592721208 shutdown_connections 2023-03-03T18:50:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.758+0000 7fa78d86a700 1 -- 172.21.15.7:0/1592721208 wait complete. 2023-03-03T18:50:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.759+0000 7fa78d86a700 1 Processor -- start 2023-03-03T18:50:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.759+0000 7fa78d86a700 1 -- start start 2023-03-03T18:50:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa78d86a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 0x7fa78814e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa78d86a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7880c7590 0x7fa78814ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa78d86a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa788151cc0 0x7fa7881490c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa78d86a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa788157000 con 0x7fa7880b00b0 2023-03-03T18:50:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa78d86a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880c4b00 con 0x7fa788151cc0 2023-03-03T18:50:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa78d86a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa788155d30 con 0x7fa7880c7590 2023-03-03T18:50:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa786ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 0x7fa78814e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:29.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa786ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 0x7fa78814e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38924/0 (socket says 172.21.15.7:38924) 2023-03-03T18:50:29.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa786ffd700 1 -- 172.21.15.7:0/622978071 learned_addr learned my addr 172.21.15.7:0/622978071 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:50:29.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa7877fe700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa788151cc0 0x7fa7881490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:29.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa7867fc700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7880c7590 0x7fa78814ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:29.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa786ffd700 1 -- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7880c7590 msgr2=0x7fa78814ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:29.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa786ffd700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7880c7590 0x7fa78814ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:29.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa786ffd700 1 -- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa788151cc0 msgr2=0x7fa7881490c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:29.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa786ffd700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa788151cc0 0x7fa7881490c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:29.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.760+0000 7fa786ffd700 1 -- 172.21.15.7:0/622978071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c0094a0 con 0x7fa7880b00b0 2023-03-03T18:50:29.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.761+0000 7fa786ffd700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 0x7fa78814e7f0 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7fa77800c4b0 tx=0x7fa77800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:50:29.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.761+0000 7fa76ffff700 1 -- 172.21.15.7:0/622978071 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778005d70 con 0x7fa7880b00b0 2023-03-03T18:50:29.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.761+0000 7fa76ffff700 1 -- 172.21.15.7:0/622978071 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7780070d0 con 0x7fa7880b00b0 2023-03-03T18:50:29.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.761+0000 7fa78d86a700 1 -- 172.21.15.7:0/622978071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa788155190 con 0x7fa7880b00b0 2023-03-03T18:50:29.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.761+0000 7fa76ffff700 1 -- 172.21.15.7:0/622978071 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778007440 con 0x7fa7880b00b0 2023-03-03T18:50:29.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.761+0000 7fa78d86a700 1 -- 172.21.15.7:0/622978071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa788154eb0 con 0x7fa7880b00b0 2023-03-03T18:50:29.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.762+0000 7fa78d86a700 1 -- 172.21.15.7:0/622978071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7881545a0 con 0x7fa7880b00b0 2023-03-03T18:50:29.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.763+0000 7fa76ffff700 1 -- 172.21.15.7:0/622978071 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa778019030 con 0x7fa7880b00b0 2023-03-03T18:50:29.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.763+0000 7fa76ffff700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7740682f0 0x7fa77406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:50:29.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.763+0000 7fa76ffff700 1 -- 172.21.15.7:0/622978071 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa7780857e0 con 0x7fa7880b00b0 2023-03-03T18:50:29.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.763+0000 7fa7867fc700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7740682f0 0x7fa77406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:50:29.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.764+0000 7fa7867fc700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7740682f0 0x7fa77406a7a0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fa7880cb170 tx=0x7fa77c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:50:29.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:29.767+0000 7fa76ffff700 1 -- 172.21.15.7:0/622978071 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa778051cd0 con 0x7fa7880b00b0 2023-03-03T18:50:30.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.001+0000 7fa78d86a700 1 -- 172.21.15.7:0/622978071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa788074bb0 con 0x7fa7880b00b0 2023-03-03T18:50:30.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.002+0000 7fa76ffff700 1 -- 172.21.15.7:0/622978071 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa7780552f0 con 0x7fa7880b00b0 2023-03-03T18:50:30.003 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:50:30.004 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:50:30.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 -- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7740682f0 msgr2=0x7fa77406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:30.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7740682f0 0x7fa77406a7a0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fa7880cb170 tx=0x7fa77c008030 comp rx=0 tx=0).stop 2023-03-03T18:50:30.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 -- 172.21.15.7:0/622978071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 msgr2=0x7fa78814e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:50:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 0x7fa78814e7f0 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7fa77800c4b0 tx=0x7fa77800c870 comp rx=0 tx=0).stop 2023-03-03T18:50:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 -- 172.21.15.7:0/622978071 shutdown_connections 2023-03-03T18:50:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa788151cc0 0x7fa7881490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa7740682f0 0x7fa77406a7a0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7880b00b0 0x7fa78814e7f0 unknown :-1 s=CLOSED pgs=3313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 --2- 172.21.15.7:0/622978071 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa7880c7590 0x7fa78814ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:50:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 -- 172.21.15.7:0/622978071 >> 172.21.15.7:0/622978071 conn(0x7fa78806b980 msgr2=0x7fa78814a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:50:30.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 -- 172.21.15.7:0/622978071 shutdown_connections 2023-03-03T18:50:30.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:50:30.004+0000 7fa78d86a700 1 -- 172.21.15.7:0/622978071 wait complete. 2023-03-03T18:50:30.007 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:50:30.018 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2023-03-03T18:50:42.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:50:42.655 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:51:12.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:51:12.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.612+0000 7f0eae082700 1 Processor -- start 2023-03-03T18:51:12.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.612+0000 7f0eae082700 1 -- start start 2023-03-03T18:51:12.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.612+0000 7f0eae082700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814f9d0 0x7f0ea8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.612+0000 7f0eae082700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0ea80abe20 0x7f0ea81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.612+0000 7f0eae082700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea80c2f30 con 0x7f0ea814fda0 2023-03-03T18:51:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.612+0000 7f0eae082700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea80c31d0 con 0x7f0ea80abe20 2023-03-03T18:51:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.612+0000 7f0eae082700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea81524a0 con 0x7f0ea814f9d0 2023-03-03T18:51:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea77fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814f9d0 0x7f0ea8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea7fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0ea80abe20 0x7f0ea81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:12.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea7fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0ea80abe20 0x7f0ea81529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:34724/0 (socket says 172.21.15.7:34724) 2023-03-03T18:51:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea6ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f0ea814fda0 0x7f0ea8071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:36446/0 (socket says 172.21.15.7:36446) 2023-03-03T18:51:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea77fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814f9d0 0x7f0ea8156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:47018/0 (socket says 172.21.15.7:47018) 2023-03-03T18:51:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea7fff700 1 -- 172.21.15.7:0/1776212096 learned_addr learned my addr 172.21.15.7:0/1776212096 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:51:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea7fff700 1 -- 172.21.15.7:0/1776212096 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814f9d0 msgr2=0x7f0ea8156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:51:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea7fff700 1 --2- 172.21.15.7:0/1776212096 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814f9d0 0x7f0ea8156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea7fff700 1 -- 172.21.15.7:0/1776212096 >> v1:172.21.15.143:6789/0 conn(0x7f0ea814fda0 legacy=0x7f0ea8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:12.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea7fff700 1 -- 172.21.15.7:0/1776212096 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea80c34b0 con 0x7f0ea80abe20 2023-03-03T18:51:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea67fc700 1 -- 172.21.15.7:0/1776212096 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284947419 0 0) 0x7f0ea80c2f30 con 0x7f0ea814fda0 2023-03-03T18:51:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea77fe700 1 --2- 172.21.15.7:0/1776212096 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814f9d0 0x7f0ea8156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:51:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea7fff700 1 --2- 172.21.15.7:0/1776212096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ea80abe20 0x7f0ea81529c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009e50 tx=0x7f0ea000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc43e42da9675432 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:51:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.613+0000 7f0ea67fc700 1 -- 172.21.15.7:0/1776212096 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea000e070 con 0x7f0ea80abe20 2023-03-03T18:51:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.614+0000 7f0ea67fc700 1 -- 172.21.15.7:0/1776212096 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea000b910 con 0x7f0ea80abe20 2023-03-03T18:51:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.614+0000 7f0ea67fc700 1 -- 172.21.15.7:0/1776212096 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea000bcc0 con 0x7f0ea80abe20 2023-03-03T18:51:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.614+0000 7f0eae082700 1 -- 172.21.15.7:0/1776212096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ea80abe20 msgr2=0x7f0ea81529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:12.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.614+0000 7f0eae082700 1 --2- 172.21.15.7:0/1776212096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ea80abe20 0x7f0ea81529c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0009e50 tx=0x7f0ea000b330 comp rx=0 tx=0).stop 2023-03-03T18:51:12.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.614+0000 7f0eae082700 1 -- 172.21.15.7:0/1776212096 shutdown_connections 2023-03-03T18:51:12.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.614+0000 7f0eae082700 1 --2- 172.21.15.7:0/1776212096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ea80abe20 0x7f0ea81529c0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:12.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.614+0000 7f0eae082700 1 --2- 172.21.15.7:0/1776212096 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814f9d0 0x7f0ea8156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:12.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.614+0000 7f0eae082700 1 -- 172.21.15.7:0/1776212096 >> 172.21.15.7:0/1776212096 conn(0x7f0ea80c8900 msgr2=0x7f0ea80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:51:12.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.614+0000 7f0eae082700 1 -- 172.21.15.7:0/1776212096 shutdown_connections 2023-03-03T18:51:12.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.614+0000 7f0eae082700 1 -- 172.21.15.7:0/1776212096 wait complete. 2023-03-03T18:51:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.615+0000 7f0eae082700 1 Processor -- start 2023-03-03T18:51:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0eae082700 1 -- start start 2023-03-03T18:51:12.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0eae082700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ea80abe20 0x7f0ea80c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0eae082700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ea814f9d0 0x7f0ea814b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0eae082700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814fda0 0x7f0ea814cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0eae082700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea81e7df0 con 0x7f0ea814f9d0 2023-03-03T18:51:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0eae082700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea814b680 con 0x7f0ea80abe20 2023-03-03T18:51:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0eae082700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea80bd930 con 0x7f0ea814fda0 2023-03-03T18:51:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0ea6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ea814f9d0 0x7f0ea814b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:12.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0ea6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ea814f9d0 0x7f0ea814b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38954/0 (socket says 172.21.15.7:38954) 2023-03-03T18:51:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0ea6ffd700 1 -- 172.21.15.7:0/3807782179 learned_addr learned my addr 172.21.15.7:0/3807782179 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:51:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0ea77fe700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ea80abe20 0x7f0ea80c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.616+0000 7f0ea7fff700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814fda0 0x7f0ea814cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0ea6ffd700 1 -- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814fda0 msgr2=0x7f0ea814cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0ea6ffd700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814fda0 0x7f0ea814cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0ea6ffd700 1 -- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ea80abe20 msgr2=0x7f0ea80c1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:12.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0ea6ffd700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ea80abe20 0x7f0ea80c1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:12.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0ea6ffd700 1 -- 172.21.15.7:0/3807782179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980094a0 con 0x7f0ea814f9d0 2023-03-03T18:51:12.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0ea6ffd700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ea814f9d0 0x7f0ea814b140 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000c540 tx=0x7f0e9000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:51:12.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0e9ffff700 1 -- 172.21.15.7:0/3807782179 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e90004c80 con 0x7f0ea814f9d0 2023-03-03T18:51:12.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0e9ffff700 1 -- 172.21.15.7:0/3807782179 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e90004de0 con 0x7f0ea814f9d0 2023-03-03T18:51:12.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0e9ffff700 1 -- 172.21.15.7:0/3807782179 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e90007310 con 0x7f0ea814f9d0 2023-03-03T18:51:12.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0eae082700 1 -- 172.21.15.7:0/3807782179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea0009b30 con 0x7f0ea814f9d0 2023-03-03T18:51:12.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.617+0000 7f0eae082700 1 -- 172.21.15.7:0/3807782179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea80ac350 con 0x7f0ea814f9d0 2023-03-03T18:51:12.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.618+0000 7f0eae082700 1 -- 172.21.15.7:0/3807782179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea80ab630 con 0x7f0ea814f9d0 2023-03-03T18:51:12.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.619+0000 7f0e9ffff700 1 -- 172.21.15.7:0/3807782179 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0e9001a020 con 0x7f0ea814f9d0 2023-03-03T18:51:12.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.619+0000 7f0e9ffff700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e880682f0 0x7f0e8806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:12.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.619+0000 7f0e9ffff700 1 -- 172.21.15.7:0/3807782179 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0e90085bb0 con 0x7f0ea814f9d0 2023-03-03T18:51:12.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.619+0000 7f0ea77fe700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e880682f0 0x7f0e8806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:12.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.620+0000 7f0ea77fe700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e880682f0 0x7f0e8806a7a0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f0e98009d70 tx=0x7f0e98009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:51:12.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.623+0000 7f0e9ffff700 1 -- 172.21.15.7:0/3807782179 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0e9008c080 con 0x7f0ea814f9d0 2023-03-03T18:51:12.745 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:51:12.747 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:51:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.866+0000 7f0eae082700 1 -- 172.21.15.7:0/3807782179 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ea8071fa0 con 0x7f0ea814f9d0 2023-03-03T18:51:12.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.866+0000 7f0e9ffff700 1 -- 172.21.15.7:0/3807782179 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0e90051f50 con 0x7f0ea814f9d0 2023-03-03T18:51:12.868 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:51:12.868 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:51:12.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.868+0000 7f0eae082700 1 -- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e880682f0 msgr2=0x7f0e8806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:12.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e880682f0 0x7f0e8806a7a0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f0e98009d70 tx=0x7f0e98009210 comp rx=0 tx=0).stop 2023-03-03T18:51:12.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 -- 172.21.15.7:0/3807782179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ea814f9d0 msgr2=0x7f0ea814b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:12.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ea814f9d0 0x7f0ea814b140 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000c540 tx=0x7f0e9000c850 comp rx=0 tx=0).stop 2023-03-03T18:51:12.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 -- 172.21.15.7:0/3807782179 shutdown_connections 2023-03-03T18:51:12.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0ea814fda0 0x7f0ea814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:12.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ea80abe20 0x7f0ea80c1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:12.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0e880682f0 0x7f0e8806a7a0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:12.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 --2- 172.21.15.7:0/3807782179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0ea814f9d0 0x7f0ea814b140 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:12.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 -- 172.21.15.7:0/3807782179 >> 172.21.15.7:0/3807782179 conn(0x7f0ea80c8900 msgr2=0x7f0ea8152ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:51:12.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 -- 172.21.15.7:0/3807782179 shutdown_connections 2023-03-03T18:51:12.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:12.869+0000 7f0eae082700 1 -- 172.21.15.7:0/3807782179 wait complete. 2023-03-03T18:51:12.871 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:51:12.880 INFO:tasks.mds_thrash.fs.[a]:waiting for 113 secs before thrashing 2023-03-03T18:51:42.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:51:42.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:51:56.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:51:56.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.159+0000 7f907ac81700 1 Processor -- start 2023-03-03T18:51:56.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.159+0000 7f907ac81700 1 -- start start 2023-03-03T18:51:56.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.160+0000 7f907ac81700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9074072250 0x7f907414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:56.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.160+0000 7f907ac81700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f90740c1ce0 0x7f907414e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:56.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.160+0000 7f907ac81700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90741510d0 con 0x7f9074071e80 2023-03-03T18:51:56.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.160+0000 7f907ac81700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90740c3740 con 0x7f90740c1ce0 2023-03-03T18:51:56.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.160+0000 7f907ac81700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9074070670 con 0x7f9074072250 2023-03-03T18:51:56.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.160+0000 7f9073fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9074072250 0x7f907414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:56.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.160+0000 7f907921e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f90740c1ce0 0x7f907414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:56.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.160+0000 7f9073fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9074072250 0x7f907414e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:47054/0 (socket says 172.21.15.7:47054) 2023-03-03T18:51:56.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.160+0000 7f9073fff700 1 -- 172.21.15.7:0/2208561798 learned_addr learned my addr 172.21.15.7:0/2208561798 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:51:56.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.161+0000 7f9073fff700 1 -- 172.21.15.7:0/2208561798 >> v2:172.21.15.143:3300/0 conn(0x7f90740c1ce0 msgr2=0x7f907414e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:51:56.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.161+0000 7f9073fff700 1 --2- 172.21.15.7:0/2208561798 >> v2:172.21.15.143:3300/0 conn(0x7f90740c1ce0 0x7f907414e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:56.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.161+0000 7f9073fff700 1 -- 172.21.15.7:0/2208561798 >> v1:172.21.15.143:6789/0 conn(0x7f9074071e80 legacy=0x7f9074151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:56.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.161+0000 7f9073fff700 1 -- 172.21.15.7:0/2208561798 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90740c39e0 con 0x7f9074072250 2023-03-03T18:51:56.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.161+0000 7f907921e700 1 --2- 172.21.15.7:0/2208561798 >> v2:172.21.15.143:3300/0 conn(0x7f90740c1ce0 0x7f907414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:51:56.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.161+0000 7f9073fff700 1 --2- 172.21.15.7:0/2208561798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9074072250 0x7f907414e1f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f90600098f0 tx=0x7f9060009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3ecdef61894110a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:51:56.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.161+0000 7f90737fe700 1 -- 172.21.15.7:0/2208561798 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060011830 con 0x7f9074072250 2023-03-03T18:51:56.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.161+0000 7f90737fe700 1 -- 172.21.15.7:0/2208561798 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9060011990 con 0x7f9074072250 2023-03-03T18:51:56.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.161+0000 7f90737fe700 1 -- 172.21.15.7:0/2208561798 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060011d20 con 0x7f9074072250 2023-03-03T18:51:56.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.161+0000 7f907ac81700 1 -- 172.21.15.7:0/2208561798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9074072250 msgr2=0x7f907414e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:56.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.162+0000 7f907ac81700 1 --2- 172.21.15.7:0/2208561798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9074072250 0x7f907414e1f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f90600098f0 tx=0x7f9060009c00 comp rx=0 tx=0).stop 2023-03-03T18:51:56.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.162+0000 7f907ac81700 1 -- 172.21.15.7:0/2208561798 shutdown_connections 2023-03-03T18:51:56.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.162+0000 7f907ac81700 1 --2- 172.21.15.7:0/2208561798 >> v2:172.21.15.143:3300/0 conn(0x7f90740c1ce0 0x7f907414e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:56.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.162+0000 7f907ac81700 1 --2- 172.21.15.7:0/2208561798 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9074072250 0x7f907414e1f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:56.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.162+0000 7f907ac81700 1 -- 172.21.15.7:0/2208561798 >> 172.21.15.7:0/2208561798 conn(0x7f90740a94d0 msgr2=0x7f90740a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:51:56.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.162+0000 7f907ac81700 1 -- 172.21.15.7:0/2208561798 shutdown_connections 2023-03-03T18:51:56.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.162+0000 7f907ac81700 1 -- 172.21.15.7:0/2208561798 wait complete. 2023-03-03T18:51:56.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.163+0000 7f907ac81700 1 Processor -- start 2023-03-03T18:51:56.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.163+0000 7f907ac81700 1 -- start start 2023-03-03T18:51:56.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f907ac81700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9074071e80 0x7f90740c02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:56.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f907ac81700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9074072250 0x7f90740abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:56.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f907ac81700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90740c1ce0 0x7f90740ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:56.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f907ac81700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90741ee2f0 con 0x7f9074072250 2023-03-03T18:51:56.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f907ac81700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90740ac8e0 con 0x7f9074071e80 2023-03-03T18:51:56.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f907ac81700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90741ed530 con 0x7f90740c1ce0 2023-03-03T18:51:56.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f9073fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9074072250 0x7f90740abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:56.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f9073fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9074072250 0x7f90740abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38988/0 (socket says 172.21.15.7:38988) 2023-03-03T18:51:56.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f9073fff700 1 -- 172.21.15.7:0/3551451234 learned_addr learned my addr 172.21.15.7:0/3551451234 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:51:56.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f907921e700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90740c1ce0 0x7f90740ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:56.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f9078a1d700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9074071e80 0x7f90740c02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:56.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f9073fff700 1 -- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90740c1ce0 msgr2=0x7f90740ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:56.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f9073fff700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90740c1ce0 0x7f90740ac3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:56.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f9073fff700 1 -- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9074071e80 msgr2=0x7f90740c02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:56.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f9073fff700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9074071e80 0x7f90740c02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:56.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f9073fff700 1 -- 172.21.15.7:0/3551451234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906c009a70 con 0x7f9074072250 2023-03-03T18:51:56.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.164+0000 7f9073fff700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9074072250 0x7f90740abe60 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f9060000c00 tx=0x7f9060004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:51:56.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.165+0000 7f90717fa700 1 -- 172.21.15.7:0/3551451234 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060004570 con 0x7f9074072250 2023-03-03T18:51:56.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.165+0000 7f90717fa700 1 -- 172.21.15.7:0/3551451234 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906001f070 con 0x7f9074072250 2023-03-03T18:51:56.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.165+0000 7f907ac81700 1 -- 172.21.15.7:0/3551451234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9060009550 con 0x7f9074072250 2023-03-03T18:51:56.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.165+0000 7f90717fa700 1 -- 172.21.15.7:0/3551451234 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90600038c0 con 0x7f9074072250 2023-03-03T18:51:56.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.165+0000 7f907ac81700 1 -- 172.21.15.7:0/3551451234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9074151f30 con 0x7f9074072250 2023-03-03T18:51:56.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.166+0000 7f907ac81700 1 -- 172.21.15.7:0/3551451234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9074151300 con 0x7f9074072250 2023-03-03T18:51:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.166+0000 7f90717fa700 1 -- 172.21.15.7:0/3551451234 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f906002a020 con 0x7f9074072250 2023-03-03T18:51:56.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.166+0000 7f90717fa700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90580682f0 0x7f905806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:51:56.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.167+0000 7f90717fa700 1 -- 172.21.15.7:0/3551451234 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f90600877a0 con 0x7f9074072250 2023-03-03T18:51:56.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.167+0000 7f9078a1d700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90580682f0 0x7f905806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:51:56.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.167+0000 7f9078a1d700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90580682f0 0x7f905806a7a0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f9068000f60 tx=0x7f9068009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:51:56.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.171+0000 7f90717fa700 1 -- 172.21.15.7:0/3551451234 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9060053b40 con 0x7f9074072250 2023-03-03T18:51:56.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.410+0000 7f907ac81700 1 -- 172.21.15.7:0/3551451234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90741ee140 con 0x7f9074072250 2023-03-03T18:51:56.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.410+0000 7f90717fa700 1 -- 172.21.15.7:0/3551451234 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9060057160 con 0x7f9074072250 2023-03-03T18:51:56.411 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:51:56.411 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:51:56.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 -- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90580682f0 msgr2=0x7f905806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:56.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90580682f0 0x7f905806a7a0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f9068000f60 tx=0x7f9068009040 comp rx=0 tx=0).stop 2023-03-03T18:51:56.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 -- 172.21.15.7:0/3551451234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9074072250 msgr2=0x7f90740abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:51:56.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9074072250 0x7f90740abe60 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f9060000c00 tx=0x7f9060004750 comp rx=0 tx=0).stop 2023-03-03T18:51:56.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 -- 172.21.15.7:0/3551451234 shutdown_connections 2023-03-03T18:51:56.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90740c1ce0 0x7f90740ac3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:56.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9074071e80 0x7f90740c02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:56.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f90580682f0 0x7f905806a7a0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:56.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 --2- 172.21.15.7:0/3551451234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9074072250 0x7f90740abe60 unknown :-1 s=CLOSED pgs=3315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:51:56.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 -- 172.21.15.7:0/3551451234 >> 172.21.15.7:0/3551451234 conn(0x7f90740a94d0 msgr2=0x7f907414dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:51:56.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 -- 172.21.15.7:0/3551451234 shutdown_connections 2023-03-03T18:51:56.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:51:56.412+0000 7f907ac81700 1 -- 172.21.15.7:0/3551451234 wait complete. 2023-03-03T18:51:56.414 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:51:56.424 INFO:tasks.mds_thrash.fs.[b]:waiting for 92 secs before thrashing 2023-03-03T18:52:12.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:52:12.876 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:52:42.936 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:52:42.940 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:53:05.882 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:53:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.022+0000 7f441be61700 1 Processor -- start 2023-03-03T18:53:06.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.022+0000 7f441be61700 1 -- start start 2023-03-03T18:53:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.022+0000 7f441be61700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f441406a2b0 0x7f441406a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f441be61700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44141529e0 con 0x7f44140a61d0 2023-03-03T18:53:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f441be61700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44141523d0 con 0x7f4414152e70 2023-03-03T18:53:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f441be61700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44141521c0 con 0x7f441406a2b0 2023-03-03T18:53:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f441a3fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f44140a61d0 0x7f441415cb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33016/0 (socket says 172.21.15.7:33016) 2023-03-03T18:53:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f441a3fe700 1 -- 172.21.15.7:0/3145347002 learned_addr learned my addr 172.21.15.7:0/3145347002 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:53:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4419bfd700 1 --2- 172.21.15.7:0/3145347002 >> v2:172.21.15.143:3300/0 conn(0x7f441406a2b0 0x7f441406a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015651210 0 0) 0x7f44141529e0 con 0x7f44140a61d0 2023-03-03T18:53:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43fc0035b0 con 0x7f44140a61d0 2023-03-03T18:53:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281149210 0 0) 0x7f44141523d0 con 0x7f4414152e70 2023-03-03T18:53:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44141529e0 con 0x7f4414152e70 2023-03-03T18:53:06.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4058652301 0 0) 0x7f43fc0035b0 con 0x7f44140a61d0 2023-03-03T18:53:06.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 >> v2:172.21.15.143:3300/0 conn(0x7f441406a2b0 msgr2=0x7f441406a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:53:06.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 --2- 172.21.15.7:0/3145347002 >> v2:172.21.15.143:3300/0 conn(0x7f441406a2b0 0x7f441406a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:06.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 >> v1:172.21.15.143:6789/0 conn(0x7f4414152e70 legacy=0x7f44140c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:06.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44141561c0 con 0x7f44140a61d0 2023-03-03T18:53:06.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f44040019f0 con 0x7f44140a61d0 2023-03-03T18:53:06.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4419bfd700 1 --2- 172.21.15.7:0/3145347002 >> v2:172.21.15.143:3300/0 conn(0x7f441406a2b0 0x7f441406a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:53:06.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 >> v1:172.21.15.7:6789/0 conn(0x7f44140a61d0 legacy=0x7f441415cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:06.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43fc003c00 0x7f43fc003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:06.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43fc004850 0x7f43fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:06.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43fc005250 0x7f43fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:06.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44141521c0 con 0x7f43fc005250 2023-03-03T18:53:06.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc00c160 con 0x7f43fc003c00 2023-03-03T18:53:06.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.023+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc00c320 con 0x7f43fc004850 2023-03-03T18:53:06.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f441a3fe700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43fc005250 0x7f43fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f4419bfd700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43fc003c00 0x7f43fc003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f44193fc700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43fc004850 0x7f43fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f441a3fe700 1 -- 172.21.15.7:0/3145347002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43fc004850 msgr2=0x7f43fc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:53:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f441a3fe700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f43fc004850 0x7f43fc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f441a3fe700 1 -- 172.21.15.7:0/3145347002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43fc003c00 msgr2=0x7f43fc003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:53:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f441a3fe700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43fc003c00 0x7f43fc003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f441a3fe700 1 -- 172.21.15.7:0/3145347002 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc0035b0 con 0x7f43fc005250 2023-03-03T18:53:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f4419bfd700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43fc003c00 0x7f43fc003ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:53:06.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f441a3fe700 1 -- 172.21.15.7:0/3145347002 reap_dead start 2023-03-03T18:53:06.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f441a3fe700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43fc005250 0x7f43fc00ba10 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f440400a7a0 tx=0x7f43fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae47ed8d2f5c91b7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:53:06.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4404012600 con 0x7f43fc005250 2023-03-03T18:53:06.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4404012780 con 0x7f43fc005250 2023-03-03T18:53:06.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.024+0000 7f4418bfb700 1 -- 172.21.15.7:0/3145347002 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4404012b30 con 0x7f43fc005250 2023-03-03T18:53:06.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.025+0000 7f441be61700 1 -- 172.21.15.7:0/3145347002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43fc005250 msgr2=0x7f43fc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:06.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.025+0000 7f441be61700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43fc005250 0x7f43fc00ba10 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f440400a7a0 tx=0x7f43fc005880 comp rx=0 tx=0).stop 2023-03-03T18:53:06.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.025+0000 7f441be61700 1 -- 172.21.15.7:0/3145347002 shutdown_connections 2023-03-03T18:53:06.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.025+0000 7f441be61700 1 --2- 172.21.15.7:0/3145347002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f43fc005250 0x7f43fc00ba10 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:06.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.025+0000 7f441be61700 1 -- 172.21.15.7:0/3145347002 >> 172.21.15.7:0/3145347002 conn(0x7f44140a9430 msgr2=0x7f44140c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:53:06.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.025+0000 7f441be61700 1 -- 172.21.15.7:0/3145347002 shutdown_connections 2023-03-03T18:53:06.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.025+0000 7f441be61700 1 -- 172.21.15.7:0/3145347002 wait complete. 2023-03-03T18:53:06.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.026+0000 7f441be61700 1 Processor -- start 2023-03-03T18:53:06.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.026+0000 7f441be61700 1 -- start start 2023-03-03T18:53:06.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441be61700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43fc005250 0x7f44141ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:06.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441be61700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f441414f3d0 0x7f44140c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:06.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441be61700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44141ea1f0 0x7f441414e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:06.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441be61700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4414154060 con 0x7f44141ea1f0 2023-03-03T18:53:06.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441be61700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44141ed760 con 0x7f43fc005250 2023-03-03T18:53:06.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441be61700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4414154300 con 0x7f441414f3d0 2023-03-03T18:53:06.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441a3fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44141ea1f0 0x7f441414e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:06.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441a3fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44141ea1f0 0x7f441414e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39048/0 (socket says 172.21.15.7:39048) 2023-03-03T18:53:06.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441a3fe700 1 -- 172.21.15.7:0/3328026878 learned_addr learned my addr 172.21.15.7:0/3328026878 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:53:06.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f4419bfd700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43fc005250 0x7f44141ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:06.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f44193fc700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f441414f3d0 0x7f44140c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:06.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441a3fe700 1 -- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f441414f3d0 msgr2=0x7f44140c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:06.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441a3fe700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f441414f3d0 0x7f44140c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:06.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441a3fe700 1 -- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43fc005250 msgr2=0x7f44141ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:06.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441a3fe700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43fc005250 0x7f44141ee220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:06.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.027+0000 7f441a3fe700 1 -- 172.21.15.7:0/3328026878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4414152670 con 0x7f44141ea1f0 2023-03-03T18:53:06.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.028+0000 7f441a3fe700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44141ea1f0 0x7f441414e7a0 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7f441406f7f0 tx=0x7f4404001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:53:06.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.028+0000 7f44127fc700 1 -- 172.21.15.7:0/3328026878 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4404012d00 con 0x7f44141ea1f0 2023-03-03T18:53:06.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.028+0000 7f44127fc700 1 -- 172.21.15.7:0/3328026878 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4404012e80 con 0x7f44141ea1f0 2023-03-03T18:53:06.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.028+0000 7f441be61700 1 -- 172.21.15.7:0/3328026878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44141564a0 con 0x7f44141ea1f0 2023-03-03T18:53:06.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.028+0000 7f44127fc700 1 -- 172.21.15.7:0/3328026878 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440401e670 con 0x7f44141ea1f0 2023-03-03T18:53:06.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.028+0000 7f441be61700 1 -- 172.21.15.7:0/3328026878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44140c5cb0 con 0x7f44141ea1f0 2023-03-03T18:53:06.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.029+0000 7f441be61700 1 -- 172.21.15.7:0/3328026878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44141ee070 con 0x7f44141ea1f0 2023-03-03T18:53:06.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.029+0000 7f44127fc700 1 -- 172.21.15.7:0/3328026878 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4404010070 con 0x7f44141ea1f0 2023-03-03T18:53:06.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.030+0000 7f44127fc700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44000682f0 0x7f440006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:06.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.030+0000 7f44127fc700 1 -- 172.21.15.7:0/3328026878 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4404087790 con 0x7f44141ea1f0 2023-03-03T18:53:06.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.030+0000 7f4419bfd700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44000682f0 0x7f440006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:06.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.031+0000 7f4419bfd700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44000682f0 0x7f440006a7a0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f440c009d70 tx=0x7f440c009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:53:06.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.034+0000 7f44127fc700 1 -- 172.21.15.7:0/3328026878 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4404053bf0 con 0x7f44141ea1f0 2023-03-03T18:53:06.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.274+0000 7f441be61700 1 -- 172.21.15.7:0/3328026878 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4414074bb0 con 0x7f44141ea1f0 2023-03-03T18:53:06.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.274+0000 7f44127fc700 1 -- 172.21.15.7:0/3328026878 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4404017090 con 0x7f44141ea1f0 2023-03-03T18:53:06.275 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:53:06.275 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:53:06.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 -- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44000682f0 msgr2=0x7f440006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:06.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44000682f0 0x7f440006a7a0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f440c009d70 tx=0x7f440c009210 comp rx=0 tx=0).stop 2023-03-03T18:53:06.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 -- 172.21.15.7:0/3328026878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44141ea1f0 msgr2=0x7f441414e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:06.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44141ea1f0 0x7f441414e7a0 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7f441406f7f0 tx=0x7f4404001d00 comp rx=0 tx=0).stop 2023-03-03T18:53:06.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 -- 172.21.15.7:0/3328026878 shutdown_connections 2023-03-03T18:53:06.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44000682f0 0x7f440006a7a0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:06.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44141ea1f0 0x7f441414e7a0 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:06.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f43fc005250 0x7f44141ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:06.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 --2- 172.21.15.7:0/3328026878 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f441414f3d0 0x7f44140c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:06.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 -- 172.21.15.7:0/3328026878 >> 172.21.15.7:0/3328026878 conn(0x7f44140a9430 msgr2=0x7f441406d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:53:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 -- 172.21.15.7:0/3328026878 shutdown_connections 2023-03-03T18:53:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:06.277+0000 7f441be61700 1 -- 172.21.15.7:0/3328026878 wait complete. 2023-03-03T18:53:06.279 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:53:06.290 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2023-03-03T18:53:13.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:53:13.018 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:53:21.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:53:21.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.433+0000 7ff227927700 1 Processor -- start 2023-03-03T18:53:21.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.433+0000 7ff227927700 1 -- start start 2023-03-03T18:53:21.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff227927700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff220151ab0 0x7ff2200736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:21.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff227927700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220073c30 0x7ff22006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:21.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff227927700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff22006c3e0 0x7ff220157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:21.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff227927700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22006d780 con 0x7ff22006c3e0 2023-03-03T18:53:21.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff227927700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22006cb70 con 0x7ff220151ab0 2023-03-03T18:53:21.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff227927700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22006ce50 con 0x7ff220073c30 2023-03-03T18:53:21.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff225ec4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff22006c3e0 0x7ff220157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:21.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff225ec4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff22006c3e0 0x7ff220157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39066/0 (socket says 172.21.15.7:39066) 2023-03-03T18:53:21.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff225ec4700 1 -- 172.21.15.7:0/111446999 learned_addr learned my addr 172.21.15.7:0/111446999 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:53:21.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff224ec2700 1 --2- 172.21.15.7:0/111446999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220073c30 0x7ff22006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:21.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff2256c3700 1 --2- 172.21.15.7:0/111446999 >> v2:172.21.15.143:3300/0 conn(0x7ff220151ab0 0x7ff2200736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:21.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff225ec4700 1 -- 172.21.15.7:0/111446999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220073c30 msgr2=0x7ff22006bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:53:21.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff225ec4700 1 --2- 172.21.15.7:0/111446999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220073c30 0x7ff22006bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff225ec4700 1 -- 172.21.15.7:0/111446999 >> v2:172.21.15.143:3300/0 conn(0x7ff220151ab0 msgr2=0x7ff2200736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:53:21.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff225ec4700 1 --2- 172.21.15.7:0/111446999 >> v2:172.21.15.143:3300/0 conn(0x7ff220151ab0 0x7ff2200736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff225ec4700 1 -- 172.21.15.7:0/111446999 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22006d130 con 0x7ff22006c3e0 2023-03-03T18:53:21.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff224ec2700 1 --2- 172.21.15.7:0/111446999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220073c30 0x7ff22006bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:53:21.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.434+0000 7ff225ec4700 1 --2- 172.21.15.7:0/111446999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff22006c3e0 0x7ff220157dc0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7ff22006bb20 tx=0x7ff21800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35fcb4f7ebab898 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:53:21.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff217fff700 1 -- 172.21.15.7:0/111446999 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21800e070 con 0x7ff22006c3e0 2023-03-03T18:53:21.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff217fff700 1 -- 172.21.15.7:0/111446999 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21800b9c0 con 0x7ff22006c3e0 2023-03-03T18:53:21.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff217fff700 1 -- 172.21.15.7:0/111446999 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21800bd70 con 0x7ff22006c3e0 2023-03-03T18:53:21.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff227927700 1 -- 172.21.15.7:0/111446999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff22006c3e0 msgr2=0x7ff220157dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:21.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff227927700 1 --2- 172.21.15.7:0/111446999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff22006c3e0 0x7ff220157dc0 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7ff22006bb20 tx=0x7ff21800b330 comp rx=0 tx=0).stop 2023-03-03T18:53:21.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff227927700 1 -- 172.21.15.7:0/111446999 shutdown_connections 2023-03-03T18:53:21.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff227927700 1 --2- 172.21.15.7:0/111446999 >> v2:172.21.15.143:3300/0 conn(0x7ff220151ab0 0x7ff2200736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff227927700 1 --2- 172.21.15.7:0/111446999 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff22006c3e0 0x7ff220157dc0 unknown :-1 s=CLOSED pgs=3319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff227927700 1 --2- 172.21.15.7:0/111446999 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220073c30 0x7ff22006bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff227927700 1 -- 172.21.15.7:0/111446999 >> 172.21.15.7:0/111446999 conn(0x7ff2200a94d0 msgr2=0x7ff2200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:53:21.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff227927700 1 -- 172.21.15.7:0/111446999 shutdown_connections 2023-03-03T18:53:21.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.435+0000 7ff227927700 1 -- 172.21.15.7:0/111446999 wait complete. 2023-03-03T18:53:21.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.436+0000 7ff227927700 1 Processor -- start 2023-03-03T18:53:21.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.436+0000 7ff227927700 1 -- start start 2023-03-03T18:53:21.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.437+0000 7ff227927700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff22006c3e0 0x7ff220149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:21.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.437+0000 7ff227927700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220073c30 0x7ff2200c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:21.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.437+0000 7ff227927700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220151ab0 0x7ff22014e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:21.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.437+0000 7ff227927700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2200acd60 con 0x7ff220073c30 2023-03-03T18:53:21.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.437+0000 7ff227927700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff220150210 con 0x7ff22006c3e0 2023-03-03T18:53:21.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.437+0000 7ff227927700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2200aca30 con 0x7ff220151ab0 2023-03-03T18:53:21.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.437+0000 7ff224ec2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220073c30 0x7ff2200c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:21.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.437+0000 7ff224ec2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220073c30 0x7ff2200c3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39070/0 (socket says 172.21.15.7:39070) 2023-03-03T18:53:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.437+0000 7ff224ec2700 1 -- 172.21.15.7:0/2278454686 learned_addr learned my addr 172.21.15.7:0/2278454686 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:53:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff2256c3700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff22006c3e0 0x7ff220149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff225ec4700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220151ab0 0x7ff22014e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff224ec2700 1 -- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220151ab0 msgr2=0x7ff22014e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff224ec2700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220151ab0 0x7ff22014e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff224ec2700 1 -- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff22006c3e0 msgr2=0x7ff220149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff224ec2700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff22006c3e0 0x7ff220149890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff224ec2700 1 -- 172.21.15.7:0/2278454686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff218009b30 con 0x7ff220073c30 2023-03-03T18:53:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff224ec2700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220073c30 0x7ff2200c3de0 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 crypto rx=0x7ff21000e810 tx=0x7ff21000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:53:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff215ffb700 1 -- 172.21.15.7:0/2278454686 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff210013610 con 0x7ff220073c30 2023-03-03T18:53:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff215ffb700 1 -- 172.21.15.7:0/2278454686 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff210013770 con 0x7ff220073c30 2023-03-03T18:53:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff215ffb700 1 -- 172.21.15.7:0/2278454686 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff210013b20 con 0x7ff220073c30 2023-03-03T18:53:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff227927700 1 -- 172.21.15.7:0/2278454686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff220151e80 con 0x7ff220073c30 2023-03-03T18:53:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.438+0000 7ff227927700 1 -- 172.21.15.7:0/2278454686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22006d410 con 0x7ff220073c30 2023-03-03T18:53:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.439+0000 7ff227927700 1 -- 172.21.15.7:0/2278454686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff220152010 con 0x7ff220073c30 2023-03-03T18:53:21.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.440+0000 7ff215ffb700 1 -- 172.21.15.7:0/2278454686 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff210023070 con 0x7ff220073c30 2023-03-03T18:53:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.440+0000 7ff215ffb700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2040682f0 0x7ff20406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.440+0000 7ff215ffb700 1 -- 172.21.15.7:0/2278454686 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff210085ea0 con 0x7ff220073c30 2023-03-03T18:53:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.440+0000 7ff2256c3700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2040682f0 0x7ff20406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.441+0000 7ff2256c3700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2040682f0 0x7ff20406a7a0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0096f0 tx=0x7ff21c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:53:21.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.444+0000 7ff215ffb700 1 -- 172.21.15.7:0/2278454686 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff2100521c0 con 0x7ff220073c30 2023-03-03T18:53:21.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.680+0000 7ff227927700 1 -- 172.21.15.7:0/2278454686 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff220074bb0 con 0x7ff220073c30 2023-03-03T18:53:21.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.680+0000 7ff215ffb700 1 -- 172.21.15.7:0/2278454686 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff2100557e0 con 0x7ff220073c30 2023-03-03T18:53:21.681 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:53:21.681 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:53:21.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 -- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2040682f0 msgr2=0x7ff20406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:21.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2040682f0 0x7ff20406a7a0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0096f0 tx=0x7ff21c008040 comp rx=0 tx=0).stop 2023-03-03T18:53:21.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 -- 172.21.15.7:0/2278454686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220073c30 msgr2=0x7ff2200c3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:21.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220073c30 0x7ff2200c3de0 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 crypto rx=0x7ff21000e810 tx=0x7ff21000ebd0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 -- 172.21.15.7:0/2278454686 shutdown_connections 2023-03-03T18:53:21.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff220151ab0 0x7ff22014e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff22006c3e0 0x7ff220149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff2040682f0 0x7ff20406a7a0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 --2- 172.21.15.7:0/2278454686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff220073c30 0x7ff2200c3de0 unknown :-1 s=CLOSED pgs=3320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:21.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 -- 172.21.15.7:0/2278454686 >> 172.21.15.7:0/2278454686 conn(0x7ff2200a94d0 msgr2=0x7ff220155bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:53:21.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 -- 172.21.15.7:0/2278454686 shutdown_connections 2023-03-03T18:53:21.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:21.683+0000 7ff227927700 1 -- 172.21.15.7:0/2278454686 wait complete. 2023-03-03T18:53:21.686 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:53:21.694 INFO:tasks.mds_thrash.fs.[a]:waiting for 84 secs before thrashing 2023-03-03T18:53:28.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:53:28.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.564+0000 7f4cc9e3d700 1 Processor -- start 2023-03-03T18:53:28.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.564+0000 7f4cc9e3d700 1 -- start start 2023-03-03T18:53:28.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.565+0000 7f4cc9e3d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4cc4151cd0 0x7f4cc4154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:28.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.565+0000 7f4cc9e3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc40c2e30 0x7f4cc4158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:28.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.565+0000 7f4cc9e3d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc4157ff0 con 0x7f4cc406c6c0 2023-03-03T18:53:28.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.565+0000 7f4cc9e3d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc406d640 con 0x7f4cc4151cd0 2023-03-03T18:53:28.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.565+0000 7f4cc9e3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc406cda0 con 0x7f4cc40c2e30 2023-03-03T18:53:28.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.565+0000 7f4cc3fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4cc406c6c0 0x7f4cc40736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33064/0 (socket says 172.21.15.7:33064) 2023-03-03T18:53:28.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.565+0000 7f4cc3fff700 1 -- 172.21.15.7:0/378084888 learned_addr learned my addr 172.21.15.7:0/378084888 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:53:28.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.565+0000 7f4cc37fe700 1 --2- 172.21.15.7:0/378084888 >> v2:172.21.15.143:3300/0 conn(0x7f4cc4151cd0 0x7f4cc4154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:28.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc2ffd700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc40c2e30 0x7f4cc4158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:28.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432612076 0 0) 0x7f4cc4157ff0 con 0x7f4cc406c6c0 2023-03-03T18:53:28.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb00035b0 con 0x7f4cc406c6c0 2023-03-03T18:53:28.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1203247433 0 0) 0x7f4cb00035b0 con 0x7f4cc406c6c0 2023-03-03T18:53:28.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc40c2e30 msgr2=0x7f4cc4158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:53:28.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc40c2e30 0x7f4cc4158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:28.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 >> v2:172.21.15.143:3300/0 conn(0x7f4cc4151cd0 msgr2=0x7f4cc4154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:53:28.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 --2- 172.21.15.7:0/378084888 >> v2:172.21.15.143:3300/0 conn(0x7f4cc4151cd0 0x7f4cc4154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:28.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc406d080 con 0x7f4cc406c6c0 2023-03-03T18:53:28.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f4cbc0019f0 con 0x7f4cc406c6c0 2023-03-03T18:53:28.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 >> v1:172.21.15.7:6789/0 conn(0x7f4cc406c6c0 legacy=0x7f4cc40736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:28.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cb0003a70 0x7f4cb0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:28.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 0x7f4cb0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:28.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc40c2e30 0x7f4cb0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:28.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc406d640 con 0x7f4cb0004850 2023-03-03T18:53:28.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc406cda0 con 0x7f4cb0003a70 2023-03-03T18:53:28.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb000bb10 con 0x7f4cc40c2e30 2023-03-03T18:53:28.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc37fe700 1 --2- 172.21.15.7:0/378084888 >> v2:172.21.15.143:3300/0 conn(0x7f4cc4151cd0 0x7f4cc4154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:53:28.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc2ffd700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 0x7f4cb0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:28.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc3fff700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc40c2e30 0x7f4cb0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:28.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc2ffd700 1 -- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc40c2e30 msgr2=0x7f4cb0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:53:28.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc2ffd700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc40c2e30 0x7f4cb0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:28.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc37fe700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cb0003a70 0x7f4cb0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc2ffd700 1 -- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cb0003a70 msgr2=0x7f4cb0003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:53:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc2ffd700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cb0003a70 0x7f4cb0003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc2ffd700 1 -- 172.21.15.7:0/378084888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb00035b0 con 0x7f4cb0004850 2023-03-03T18:53:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.566+0000 7f4cc37fe700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cb0003a70 0x7f4cb0003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:53:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc2ffd700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 0x7f4cb0004cc0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009b90 tx=0x7f4cb0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46f2f11ed3c456a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:53:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400e6c0 con 0x7f4cb0004850 2023-03-03T18:53:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb400e820 con 0x7f4cb0004850 2023-03-03T18:53:28.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc27fc700 1 -- 172.21.15.7:0/378084888 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400eb90 con 0x7f4cb0004850 2023-03-03T18:53:28.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/378084888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 msgr2=0x7f4cb0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:28.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc9e3d700 1 --2- 172.21.15.7:0/378084888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 0x7f4cb0004cc0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009b90 tx=0x7f4cb0005880 comp rx=0 tx=0).stop 2023-03-03T18:53:28.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc3fff700 1 -- 172.21.15.7:0/378084888 reap_dead start 2023-03-03T18:53:28.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/378084888 shutdown_connections 2023-03-03T18:53:28.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/378084888 >> 172.21.15.7:0/378084888 conn(0x7f4cc40a94d0 msgr2=0x7f4cc40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:53:28.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/378084888 shutdown_connections 2023-03-03T18:53:28.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.567+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/378084888 wait complete. 2023-03-03T18:53:28.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.569+0000 7f4cc9e3d700 1 Processor -- start 2023-03-03T18:53:28.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.569+0000 7f4cc9e3d700 1 -- start start 2023-03-03T18:53:28.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.569+0000 7f4cc9e3d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 0x7f4cc414dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:28.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.569+0000 7f4cc9e3d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cc40707e0 0x7f4cc4147500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:28.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc9e3d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc4147a40 0x7f4cc41dec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:28.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc9e3d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc41e0730 con 0x7f4cb0004850 2023-03-03T18:53:28.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc9e3d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc4072420 con 0x7f4cc40707e0 2023-03-03T18:53:28.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc9e3d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc4147eb0 con 0x7f4cc4147a40 2023-03-03T18:53:28.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 0x7f4cc414dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:28.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 0x7f4cc414dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39092/0 (socket says 172.21.15.7:39092) 2023-03-03T18:53:28.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc37fe700 1 -- 172.21.15.7:0/76962694 learned_addr learned my addr 172.21.15.7:0/76962694 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:53:28.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc2ffd700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cc40707e0 0x7f4cc4147500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:28.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc3fff700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc4147a40 0x7f4cc41dec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:28.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc37fe700 1 -- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc4147a40 msgr2=0x7f4cc41dec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:28.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc37fe700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc4147a40 0x7f4cc41dec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:28.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc37fe700 1 -- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cc40707e0 msgr2=0x7f4cc4147500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:28.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc37fe700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cc40707e0 0x7f4cc4147500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:28.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc37fe700 1 -- 172.21.15.7:0/76962694 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc4157c80 con 0x7f4cb0004850 2023-03-03T18:53:28.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc37fe700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 0x7f4cc414dbd0 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7f4cac010c00 tx=0x7f4cac010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:53:28.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cbbfff700 1 -- 172.21.15.7:0/76962694 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac00eab0 con 0x7f4cb0004850 2023-03-03T18:53:28.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cbbfff700 1 -- 172.21.15.7:0/76962694 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cac00ec10 con 0x7f4cb0004850 2023-03-03T18:53:28.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/76962694 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc406d360 con 0x7f4cb0004850 2023-03-03T18:53:28.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cbbfff700 1 -- 172.21.15.7:0/76962694 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac01c400 con 0x7f4cb0004850 2023-03-03T18:53:28.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.570+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/76962694 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc4154a50 con 0x7f4cb0004850 2023-03-03T18:53:28.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.571+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/76962694 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc4072700 con 0x7f4cb0004850 2023-03-03T18:53:28.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.572+0000 7f4cbbfff700 1 -- 172.21.15.7:0/76962694 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4cac012040 con 0x7f4cb0004850 2023-03-03T18:53:28.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.572+0000 7f4cbbfff700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ca40682f0 0x7f4ca406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:53:28.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.572+0000 7f4cbbfff700 1 -- 172.21.15.7:0/76962694 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4cac0865d0 con 0x7f4cb0004850 2023-03-03T18:53:28.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.572+0000 7f4cc2ffd700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ca40682f0 0x7f4ca406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:53:28.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.573+0000 7f4cc2ffd700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ca40682f0 0x7f4ca406a7a0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40090b0 tx=0x7f4cb4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:53:28.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.577+0000 7f4cbbfff700 1 -- 172.21.15.7:0/76962694 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4cac052910 con 0x7f4cb0004850 2023-03-03T18:53:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.814+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/76962694 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cc4074bb0 con 0x7f4cb0004850 2023-03-03T18:53:28.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.814+0000 7f4cbbfff700 1 -- 172.21.15.7:0/76962694 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4cac055f30 con 0x7f4cb0004850 2023-03-03T18:53:28.814 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:53:28.815 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:53:28.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ca40682f0 msgr2=0x7f4ca406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:28.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ca40682f0 0x7f4ca406a7a0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40090b0 tx=0x7f4cb4008030 comp rx=0 tx=0).stop 2023-03-03T18:53:28.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/76962694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 msgr2=0x7f4cc414dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:53:28.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 0x7f4cc414dbd0 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7f4cac010c00 tx=0x7f4cac010f10 comp rx=0 tx=0).stop 2023-03-03T18:53:28.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/76962694 shutdown_connections 2023-03-03T18:53:28.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4cc4147a40 0x7f4cc41dec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:28.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ca40682f0 0x7f4ca406a7a0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:28.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4cb0004850 0x7f4cc414dbd0 unknown :-1 s=CLOSED pgs=3323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:28.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 --2- 172.21.15.7:0/76962694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cc40707e0 0x7f4cc4147500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:53:28.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/76962694 >> 172.21.15.7:0/76962694 conn(0x7f4cc40a94d0 msgr2=0x7f4cc40c2420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:53:28.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/76962694 shutdown_connections 2023-03-03T18:53:28.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:53:28.816+0000 7f4cc9e3d700 1 -- 172.21.15.7:0/76962694 wait complete. 2023-03-03T18:53:28.818 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:53:28.828 INFO:tasks.mds_thrash.fs.[b]:waiting for 108 secs before thrashing 2023-03-03T18:53:43.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:53:43.074 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:54:13.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:54:13.146 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:54:43.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:54:43.218 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:54:45.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:54:45.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.834+0000 7f31d25f3700 1 Processor -- start 2023-03-03T18:54:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.834+0000 7f31d25f3700 1 -- start start 2023-03-03T18:54:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31d25f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31cc0c17f0 0x7f31cc06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:54:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31d25f3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f31cc0c5190 0x7f31cc06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:54:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31d25f3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31cc14ebe0 con 0x7f31cc0c6010 2023-03-03T18:54:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31d25f3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31cc152100 con 0x7f31cc0c5190 2023-03-03T18:54:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31d25f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0c2e70 con 0x7f31cc0c17f0 2023-03-03T18:54:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31cbfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f31cc0c6010 0x7f31cc0c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33112/0 (socket says 172.21.15.7:33112) 2023-03-03T18:54:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31cbfff700 1 -- 172.21.15.7:0/736050746 learned_addr learned my addr 172.21.15.7:0/736050746 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:54:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31d0b90700 1 --2- 172.21.15.7:0/736050746 >> v2:172.21.15.7:3300/0 conn(0x7f31cc0c5190 0x7f31cc06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:54:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31caffd700 1 -- 172.21.15.7:0/736050746 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982915379 0 0) 0x7f31cc14ebe0 con 0x7f31cc0c6010 2023-03-03T18:54:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31caffd700 1 -- 172.21.15.7:0/736050746 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b40035b0 con 0x7f31cc0c6010 2023-03-03T18:54:45.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.835+0000 7f31cb7fe700 1 --2- 172.21.15.7:0/736050746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31cc0c17f0 0x7f31cc06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:54:45.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d0b90700 1 -- 172.21.15.7:0/736050746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31cc0c17f0 msgr2=0x7f31cc06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:54:45.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d0b90700 1 --2- 172.21.15.7:0/736050746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31cc0c17f0 0x7f31cc06d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:54:45.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d0b90700 1 -- 172.21.15.7:0/736050746 >> v1:172.21.15.7:6789/0 conn(0x7f31cc0c6010 legacy=0x7f31cc0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:54:45.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d0b90700 1 -- 172.21.15.7:0/736050746 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc0aa680 con 0x7f31cc0c5190 2023-03-03T18:54:45.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d0b90700 1 --2- 172.21.15.7:0/736050746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c5190 0x7f31cc06dbe0 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009840 tx=0x7f31c4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7cd97aed60f2f50 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:54:45.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31caffd700 1 -- 172.21.15.7:0/736050746 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4011830 con 0x7f31cc0c5190 2023-03-03T18:54:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31caffd700 1 -- 172.21.15.7:0/736050746 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c4011990 con 0x7f31cc0c5190 2023-03-03T18:54:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31caffd700 1 -- 172.21.15.7:0/736050746 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4011d20 con 0x7f31cc0c5190 2023-03-03T18:54:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d25f3700 1 -- 172.21.15.7:0/736050746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c5190 msgr2=0x7f31cc06dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:54:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d25f3700 1 --2- 172.21.15.7:0/736050746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c5190 0x7f31cc06dbe0 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009840 tx=0x7f31c4009b50 comp rx=0 tx=0).stop 2023-03-03T18:54:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d25f3700 1 -- 172.21.15.7:0/736050746 shutdown_connections 2023-03-03T18:54:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d25f3700 1 --2- 172.21.15.7:0/736050746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c5190 0x7f31cc06dbe0 unknown :-1 s=CLOSED pgs=3325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:54:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d25f3700 1 --2- 172.21.15.7:0/736050746 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31cc0c17f0 0x7f31cc06d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:54:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.836+0000 7f31d25f3700 1 -- 172.21.15.7:0/736050746 >> 172.21.15.7:0/736050746 conn(0x7f31cc0a9430 msgr2=0x7f31cc0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:54:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.837+0000 7f31d25f3700 1 -- 172.21.15.7:0/736050746 shutdown_connections 2023-03-03T18:54:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.837+0000 7f31d25f3700 1 -- 172.21.15.7:0/736050746 wait complete. 2023-03-03T18:54:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.838+0000 7f31d25f3700 1 Processor -- start 2023-03-03T18:54:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.838+0000 7f31d25f3700 1 -- start start 2023-03-03T18:54:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d25f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31cc0c17f0 0x7f31cc0a7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:54:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d25f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31cc0c5190 0x7f31cc14ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:54:45.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d25f3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c6010 0x7f31cc14d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:54:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d25f3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0bbfe0 con 0x7f31cc0c6010 2023-03-03T18:54:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d25f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0722a0 con 0x7f31cc0c5190 2023-03-03T18:54:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d25f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc072580 con 0x7f31cc0c17f0 2023-03-03T18:54:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d0b90700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c6010 0x7f31cc14d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:54:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d0b90700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c6010 0x7f31cc14d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39142/0 (socket says 172.21.15.7:39142) 2023-03-03T18:54:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d0b90700 1 -- 172.21.15.7:0/703556971 learned_addr learned my addr 172.21.15.7:0/703556971 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:54:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31cb7fe700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31cc0c5190 0x7f31cc14ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:54:45.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31cbfff700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31cc0c17f0 0x7f31cc0a7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:54:45.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d0b90700 1 -- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31cc0c17f0 msgr2=0x7f31cc0a7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:54:45.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d0b90700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31cc0c17f0 0x7f31cc0a7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:54:45.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d0b90700 1 -- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31cc0c5190 msgr2=0x7f31cc14ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:54:45.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d0b90700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31cc0c5190 0x7f31cc14ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:54:45.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d0b90700 1 -- 172.21.15.7:0/703556971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c40094a0 con 0x7f31cc0c6010 2023-03-03T18:54:45.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d0b90700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c6010 0x7f31cc14d3a0 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009b90 tx=0x7f31c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:54:45.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31c8ff9700 1 -- 172.21.15.7:0/703556971 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c40038a0 con 0x7f31cc0c6010 2023-03-03T18:54:45.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31c8ff9700 1 -- 172.21.15.7:0/703556971 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c401f070 con 0x7f31cc0c6010 2023-03-03T18:54:45.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31d25f3700 1 -- 172.21.15.7:0/703556971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31cc0c2b00 con 0x7f31cc0c6010 2023-03-03T18:54:45.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.839+0000 7f31c8ff9700 1 -- 172.21.15.7:0/703556971 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4003c60 con 0x7f31cc0c6010 2023-03-03T18:54:45.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.840+0000 7f31d25f3700 1 -- 172.21.15.7:0/703556971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc0aa960 con 0x7f31cc0c6010 2023-03-03T18:54:45.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.840+0000 7f31d25f3700 1 -- 172.21.15.7:0/703556971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31cc0c3300 con 0x7f31cc0c6010 2023-03-03T18:54:45.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.841+0000 7f31c8ff9700 1 -- 172.21.15.7:0/703556971 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f31c400f050 con 0x7f31cc0c6010 2023-03-03T18:54:45.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.841+0000 7f31c8ff9700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31b80682f0 0x7f31b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:54:45.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.841+0000 7f31c8ff9700 1 -- 172.21.15.7:0/703556971 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f31c4021070 con 0x7f31cc0c6010 2023-03-03T18:54:45.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.842+0000 7f31cbfff700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31b80682f0 0x7f31b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:54:45.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.842+0000 7f31cbfff700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31b80682f0 0x7f31b806a7a0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f31bc001560 tx=0x7f31bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:54:45.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:45.846+0000 7f31c8ff9700 1 -- 172.21.15.7:0/703556971 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f31c4053de0 con 0x7f31cc0c6010 2023-03-03T18:54:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.086+0000 7f31d25f3700 1 -- 172.21.15.7:0/703556971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31cc0bba60 con 0x7f31cc0c6010 2023-03-03T18:54:46.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.087+0000 7f31c8ff9700 1 -- 172.21.15.7:0/703556971 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f31cc0bba60 con 0x7f31cc0c6010 2023-03-03T18:54:46.088 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:54:46.088 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:54:46.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.089+0000 7f31d25f3700 1 -- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31b80682f0 msgr2=0x7f31b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:54:46.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.089+0000 7f31d25f3700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31b80682f0 0x7f31b806a7a0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f31bc001560 tx=0x7f31bc009040 comp rx=0 tx=0).stop 2023-03-03T18:54:46.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.089+0000 7f31d25f3700 1 -- 172.21.15.7:0/703556971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c6010 msgr2=0x7f31cc14d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:54:46.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.089+0000 7f31d25f3700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c6010 0x7f31cc14d3a0 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009b90 tx=0x7f31c4004750 comp rx=0 tx=0).stop 2023-03-03T18:54:46.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.089+0000 7f31d25f3700 1 -- 172.21.15.7:0/703556971 shutdown_connections 2023-03-03T18:54:46.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.089+0000 7f31d25f3700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f31b80682f0 0x7f31b806a7a0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:54:46.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.089+0000 7f31d25f3700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31cc0c6010 0x7f31cc14d3a0 unknown :-1 s=CLOSED pgs=3326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:54:46.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.089+0000 7f31d25f3700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f31cc0c17f0 0x7f31cc0a7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:54:46.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.089+0000 7f31d25f3700 1 --2- 172.21.15.7:0/703556971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f31cc0c5190 0x7f31cc14ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:54:46.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.089+0000 7f31d25f3700 1 -- 172.21.15.7:0/703556971 >> 172.21.15.7:0/703556971 conn(0x7f31cc0a9430 msgr2=0x7f31cc0c1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:54:46.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.090+0000 7f31d25f3700 1 -- 172.21.15.7:0/703556971 shutdown_connections 2023-03-03T18:54:46.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:54:46.090+0000 7f31d25f3700 1 -- 172.21.15.7:0/703556971 wait complete. 2023-03-03T18:54:46.092 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:54:46.106 INFO:tasks.mds_thrash.fs.[a]:waiting for 29 secs before thrashing 2023-03-03T18:55:13.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:55:13.298 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:55:15.107 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:55:15.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.247+0000 7fde54d78700 1 Processor -- start 2023-03-03T18:55:15.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.247+0000 7fde54d78700 1 -- start start 2023-03-03T18:55:15.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde54d78700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde5014f9d0 0x7fde50156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:15.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde54d78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014fda0 0x7fde50071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde54d78700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde500c2f30 con 0x7fde500abe20 2023-03-03T18:55:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde54d78700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde500c31d0 con 0x7fde5014f9d0 2023-03-03T18:55:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde54d78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde501524a0 con 0x7fde5014fda0 2023-03-03T18:55:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde4dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014fda0 0x7fde50071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde4e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde5014f9d0 0x7fde50156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde4e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde5014f9d0 0x7fde50156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:34956/0 (socket says 172.21.15.7:34956) 2023-03-03T18:55:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde4dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014fda0 0x7fde50071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:47254/0 (socket says 172.21.15.7:47254) 2023-03-03T18:55:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde4ed9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fde500abe20 0x7fde501529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:36686/0 (socket says 172.21.15.7:36686) 2023-03-03T18:55:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.248+0000 7fde4e59c700 1 -- 172.21.15.7:0/3273163668 learned_addr learned my addr 172.21.15.7:0/3273163668 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:55:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.249+0000 7fde4dd9b700 1 -- 172.21.15.7:0/3273163668 >> v2:172.21.15.143:3300/0 conn(0x7fde5014f9d0 msgr2=0x7fde50156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:55:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.249+0000 7fde4dd9b700 1 --2- 172.21.15.7:0/3273163668 >> v2:172.21.15.143:3300/0 conn(0x7fde5014f9d0 0x7fde50156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.249+0000 7fde4dd9b700 1 -- 172.21.15.7:0/3273163668 >> v1:172.21.15.143:6789/0 conn(0x7fde500abe20 legacy=0x7fde501529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.249+0000 7fde4dd9b700 1 -- 172.21.15.7:0/3273163668 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde500c34b0 con 0x7fde5014fda0 2023-03-03T18:55:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.249+0000 7fde4e59c700 1 --2- 172.21.15.7:0/3273163668 >> v2:172.21.15.143:3300/0 conn(0x7fde5014f9d0 0x7fde50156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T18:55:15.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.249+0000 7fde4dd9b700 1 --2- 172.21.15.7:0/3273163668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014fda0 0x7fde50071890 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fde44009840 tx=0x7fde44009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbac93c8af85b2dd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:55:15.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.249+0000 7fde4d59a700 1 -- 172.21.15.7:0/3273163668 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44011830 con 0x7fde5014fda0 2023-03-03T18:55:15.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.249+0000 7fde4d59a700 1 -- 172.21.15.7:0/3273163668 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde44011990 con 0x7fde5014fda0 2023-03-03T18:55:15.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.249+0000 7fde4d59a700 1 -- 172.21.15.7:0/3273163668 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44011d20 con 0x7fde5014fda0 2023-03-03T18:55:15.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.250+0000 7fde54d78700 1 -- 172.21.15.7:0/3273163668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014fda0 msgr2=0x7fde50071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:15.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.250+0000 7fde54d78700 1 --2- 172.21.15.7:0/3273163668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014fda0 0x7fde50071890 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fde44009840 tx=0x7fde44009b50 comp rx=0 tx=0).stop 2023-03-03T18:55:15.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.250+0000 7fde54d78700 1 -- 172.21.15.7:0/3273163668 shutdown_connections 2023-03-03T18:55:15.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.250+0000 7fde54d78700 1 --2- 172.21.15.7:0/3273163668 >> v2:172.21.15.143:3300/0 conn(0x7fde5014f9d0 0x7fde50156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:15.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.250+0000 7fde54d78700 1 --2- 172.21.15.7:0/3273163668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014fda0 0x7fde50071890 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:15.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.250+0000 7fde54d78700 1 -- 172.21.15.7:0/3273163668 >> 172.21.15.7:0/3273163668 conn(0x7fde500c8900 msgr2=0x7fde500c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:55:15.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.250+0000 7fde54d78700 1 -- 172.21.15.7:0/3273163668 shutdown_connections 2023-03-03T18:55:15.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.250+0000 7fde54d78700 1 -- 172.21.15.7:0/3273163668 wait complete. 2023-03-03T18:55:15.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.251+0000 7fde54d78700 1 Processor -- start 2023-03-03T18:55:15.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.251+0000 7fde54d78700 1 -- start start 2023-03-03T18:55:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde54d78700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde500abe20 0x7fde501e7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde54d78700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014f9d0 0x7fde5014c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde54d78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5014fda0 0x7fde5014c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde54d78700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde501dcd70 con 0x7fde500abe20 2023-03-03T18:55:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde54d78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5014cc00 con 0x7fde5014fda0 2023-03-03T18:55:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde54d78700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde50155b80 con 0x7fde5014f9d0 2023-03-03T18:55:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde4e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde500abe20 0x7fde501e7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde4e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde500abe20 0x7fde501e7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39188/0 (socket says 172.21.15.7:39188) 2023-03-03T18:55:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde4e59c700 1 -- 172.21.15.7:0/1336028903 learned_addr learned my addr 172.21.15.7:0/1336028903 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:55:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde4ed9d700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5014fda0 0x7fde5014c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde4dd9b700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014f9d0 0x7fde5014c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde4e59c700 1 -- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014f9d0 msgr2=0x7fde5014c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde4e59c700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014f9d0 0x7fde5014c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde4e59c700 1 -- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5014fda0 msgr2=0x7fde5014c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde4e59c700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5014fda0 0x7fde5014c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.252+0000 7fde4e59c700 1 -- 172.21.15.7:0/1336028903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40009550 con 0x7fde500abe20 2023-03-03T18:55:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.253+0000 7fde4e59c700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde500abe20 0x7fde501e7360 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7fde4000ebe0 tx=0x7fde4000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:55:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.253+0000 7fde3effd700 1 -- 172.21.15.7:0/1336028903 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4000c850 con 0x7fde500abe20 2023-03-03T18:55:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.253+0000 7fde3effd700 1 -- 172.21.15.7:0/1336028903 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4000c9b0 con 0x7fde500abe20 2023-03-03T18:55:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.253+0000 7fde54d78700 1 -- 172.21.15.7:0/1336028903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde440094a0 con 0x7fde500abe20 2023-03-03T18:55:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.253+0000 7fde3effd700 1 -- 172.21.15.7:0/1336028903 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4000cd40 con 0x7fde500abe20 2023-03-03T18:55:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.253+0000 7fde54d78700 1 -- 172.21.15.7:0/1336028903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde500ac350 con 0x7fde500abe20 2023-03-03T18:55:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.254+0000 7fde54d78700 1 -- 172.21.15.7:0/1336028903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde500ab630 con 0x7fde500abe20 2023-03-03T18:55:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.254+0000 7fde3effd700 1 -- 172.21.15.7:0/1336028903 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fde40019030 con 0x7fde500abe20 2023-03-03T18:55:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.254+0000 7fde3effd700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde300682f0 0x7fde3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.255+0000 7fde3effd700 1 -- 172.21.15.7:0/1336028903 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fde40086160 con 0x7fde500abe20 2023-03-03T18:55:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.255+0000 7fde4dd9b700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde300682f0 0x7fde3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.255+0000 7fde4dd9b700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde300682f0 0x7fde3006a7a0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fde501501d0 tx=0x7fde4401d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:55:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.259+0000 7fde3effd700 1 -- 172.21.15.7:0/1336028903 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fde40052500 con 0x7fde500abe20 2023-03-03T18:55:15.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.499+0000 7fde54d78700 1 -- 172.21.15.7:0/1336028903 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde500c3e10 con 0x7fde500abe20 2023-03-03T18:55:15.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.499+0000 7fde3effd700 1 -- 172.21.15.7:0/1336028903 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fde40055b20 con 0x7fde500abe20 2023-03-03T18:55:15.500 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:55:15.500 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:55:15.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 -- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde300682f0 msgr2=0x7fde3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:15.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde300682f0 0x7fde3006a7a0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fde501501d0 tx=0x7fde4401d040 comp rx=0 tx=0).stop 2023-03-03T18:55:15.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 -- 172.21.15.7:0/1336028903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde500abe20 msgr2=0x7fde501e7360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:15.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde500abe20 0x7fde501e7360 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7fde4000ebe0 tx=0x7fde4000eef0 comp rx=0 tx=0).stop 2023-03-03T18:55:15.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 -- 172.21.15.7:0/1336028903 shutdown_connections 2023-03-03T18:55:15.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde5014fda0 0x7fde5014c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:15.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde300682f0 0x7fde3006a7a0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:15.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde500abe20 0x7fde501e7360 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:15.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 --2- 172.21.15.7:0/1336028903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde5014f9d0 0x7fde5014c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:15.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 -- 172.21.15.7:0/1336028903 >> 172.21.15.7:0/1336028903 conn(0x7fde500c8900 msgr2=0x7fde500c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:55:15.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 -- 172.21.15.7:0/1336028903 shutdown_connections 2023-03-03T18:55:15.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:15.501+0000 7fde54d78700 1 -- 172.21.15.7:0/1336028903 wait complete. 2023-03-03T18:55:15.503 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:55:15.513 INFO:tasks.mds_thrash.fs.[a]:waiting for 109 secs before thrashing 2023-03-03T18:55:16.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:55:16.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.969+0000 7faed3128700 1 Processor -- start 2023-03-03T18:55:16.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.969+0000 7faed3128700 1 -- start start 2023-03-03T18:55:16.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faed3128700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faecc1530d0 0x7faecc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faed3128700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0c2f30 con 0x7faecc14f9d0 2023-03-03T18:55:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faed3128700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0c31d0 con 0x7faecc14fda0 2023-03-03T18:55:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faed3128700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc1524a0 con 0x7faecc1530d0 2023-03-03T18:55:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faed16c5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faecc1530d0 0x7faecc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faed0ec4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faecc14f9d0 0x7faecc071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33176/0 (socket says 172.21.15.7:33176) 2023-03-03T18:55:16.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faed0ec4700 1 -- 172.21.15.7:0/2933658008 learned_addr learned my addr 172.21.15.7:0/2933658008 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:55:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faed16c5700 1 -- 172.21.15.7:0/2933658008 >> v1:172.21.15.143:6789/0 conn(0x7faecc14fda0 legacy=0x7faecc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:55:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faed16c5700 1 -- 172.21.15.7:0/2933658008 >> v1:172.21.15.7:6789/0 conn(0x7faecc14f9d0 legacy=0x7faecc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faed16c5700 1 -- 172.21.15.7:0/2933658008 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0c34b0 con 0x7faecc1530d0 2023-03-03T18:55:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.970+0000 7faecb7fe700 1 -- 172.21.15.7:0/2933658008 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483321188 0 0) 0x7faecc0c2f30 con 0x7faecc14f9d0 2023-03-03T18:55:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.971+0000 7faed16c5700 1 --2- 172.21.15.7:0/2933658008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc1530d0 0x7faecc156430 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7faec4009d90 tx=0x7faec400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8822e5123550c463 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:55:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.971+0000 7faecb7fe700 1 -- 172.21.15.7:0/2933658008 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec400e070 con 0x7faecc1530d0 2023-03-03T18:55:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.971+0000 7faecb7fe700 1 -- 172.21.15.7:0/2933658008 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec400bbd0 con 0x7faecc1530d0 2023-03-03T18:55:16.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.971+0000 7faecb7fe700 1 -- 172.21.15.7:0/2933658008 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4012430 con 0x7faecc1530d0 2023-03-03T18:55:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.971+0000 7faed3128700 1 -- 172.21.15.7:0/2933658008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc1530d0 msgr2=0x7faecc156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:16.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.971+0000 7faed3128700 1 --2- 172.21.15.7:0/2933658008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc1530d0 0x7faecc156430 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7faec4009d90 tx=0x7faec400b330 comp rx=0 tx=0).stop 2023-03-03T18:55:16.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.971+0000 7faed3128700 1 -- 172.21.15.7:0/2933658008 shutdown_connections 2023-03-03T18:55:16.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.971+0000 7faed3128700 1 --2- 172.21.15.7:0/2933658008 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc1530d0 0x7faecc156430 unknown :-1 s=CLOSED pgs=3329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:16.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.971+0000 7faed3128700 1 -- 172.21.15.7:0/2933658008 >> 172.21.15.7:0/2933658008 conn(0x7faecc0c8900 msgr2=0x7faecc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:55:16.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.971+0000 7faed3128700 1 -- 172.21.15.7:0/2933658008 shutdown_connections 2023-03-03T18:55:16.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.972+0000 7faed3128700 1 -- 172.21.15.7:0/2933658008 wait complete. 2023-03-03T18:55:16.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.973+0000 7faed3128700 1 Processor -- start 2023-03-03T18:55:16.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.973+0000 7faed3128700 1 -- start start 2023-03-03T18:55:16.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.973+0000 7faed3128700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faecc14f9d0 0x7faecc154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faed3128700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc14fda0 0x7faecc1dc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faed3128700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc1530d0 0x7faecc06c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faed3128700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc06dd90 con 0x7faecc14fda0 2023-03-03T18:55:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faed3128700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc06d760 con 0x7faecc1530d0 2023-03-03T18:55:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faed3128700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc1db6e0 con 0x7faecc14f9d0 2023-03-03T18:55:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faecbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc14fda0 0x7faecc1dc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:16.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faecbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc14fda0 0x7faecc1dc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39204/0 (socket says 172.21.15.7:39204) 2023-03-03T18:55:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faecbfff700 1 -- 172.21.15.7:0/2315123520 learned_addr learned my addr 172.21.15.7:0/2315123520 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:55:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faed16c5700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc1530d0 0x7faecc06c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faed0ec4700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faecc14f9d0 0x7faecc154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faecbfff700 1 -- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faecc14f9d0 msgr2=0x7faecc154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faecbfff700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faecc14f9d0 0x7faecc154870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faecbfff700 1 -- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc1530d0 msgr2=0x7faecc06c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faecbfff700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc1530d0 0x7faecc06c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:16.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faecbfff700 1 -- 172.21.15.7:0/2315123520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec4009a70 con 0x7faecc14fda0 2023-03-03T18:55:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faed16c5700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc1530d0 0x7faecc06c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:55:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faecbfff700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc14fda0 0x7faecc1dc0c0 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 crypto rx=0x7faeb800c4e0 tx=0x7faeb800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:55:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faec97fa700 1 -- 172.21.15.7:0/2315123520 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb80052d0 con 0x7faecc14fda0 2023-03-03T18:55:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faec97fa700 1 -- 172.21.15.7:0/2315123520 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb8005430 con 0x7faecc14fda0 2023-03-03T18:55:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faed3128700 1 -- 172.21.15.7:0/2315123520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc0ac350 con 0x7faecc14fda0 2023-03-03T18:55:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faec97fa700 1 -- 172.21.15.7:0/2315123520 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb80038c0 con 0x7faecc14fda0 2023-03-03T18:55:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.974+0000 7faed3128700 1 -- 172.21.15.7:0/2315123520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc0ab630 con 0x7faecc14fda0 2023-03-03T18:55:16.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.975+0000 7faed3128700 1 -- 172.21.15.7:0/2315123520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc071fa0 con 0x7faecc14fda0 2023-03-03T18:55:16.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.976+0000 7faec97fa700 1 -- 172.21.15.7:0/2315123520 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faeb8019030 con 0x7faecc14fda0 2023-03-03T18:55:16.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.976+0000 7faec97fa700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb00682f0 0x7faeb006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:55:16.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.976+0000 7faec97fa700 1 -- 172.21.15.7:0/2315123520 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faeb80854b0 con 0x7faecc14fda0 2023-03-03T18:55:16.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.977+0000 7faed0ec4700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb00682f0 0x7faeb006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:55:16.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.977+0000 7faed0ec4700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb00682f0 0x7faeb006a7a0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7faec0007fd0 tx=0x7faec0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:55:16.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:16.981+0000 7faec97fa700 1 -- 172.21.15.7:0/2315123520 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faeb8051850 con 0x7faecc14fda0 2023-03-03T18:55:17.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.222+0000 7faed3128700 1 -- 172.21.15.7:0/2315123520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faecc06d260 con 0x7faecc14fda0 2023-03-03T18:55:17.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.222+0000 7faec97fa700 1 -- 172.21.15.7:0/2315123520 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faecc06d260 con 0x7faecc14fda0 2023-03-03T18:55:17.223 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:55:17.224 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:55:17.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 -- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb00682f0 msgr2=0x7faeb006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:17.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb00682f0 0x7faeb006a7a0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7faec0007fd0 tx=0x7faec0009040 comp rx=0 tx=0).stop 2023-03-03T18:55:17.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 -- 172.21.15.7:0/2315123520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc14fda0 msgr2=0x7faecc1dc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:55:17.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc14fda0 0x7faecc1dc0c0 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 crypto rx=0x7faeb800c4e0 tx=0x7faeb800c8a0 comp rx=0 tx=0).stop 2023-03-03T18:55:17.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 -- 172.21.15.7:0/2315123520 shutdown_connections 2023-03-03T18:55:17.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc1530d0 0x7faecc06c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:17.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faecc14f9d0 0x7faecc154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:17.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faeb00682f0 0x7faeb006a7a0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:17.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 --2- 172.21.15.7:0/2315123520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faecc14fda0 0x7faecc1dc0c0 unknown :-1 s=CLOSED pgs=3330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:55:17.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 -- 172.21.15.7:0/2315123520 >> 172.21.15.7:0/2315123520 conn(0x7faecc0c8900 msgr2=0x7faecc14f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:55:17.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 -- 172.21.15.7:0/2315123520 shutdown_connections 2023-03-03T18:55:17.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:55:17.225+0000 7faed3128700 1 -- 172.21.15.7:0/2315123520 wait complete. 2023-03-03T18:55:17.229 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:55:17.237 INFO:tasks.mds_thrash.fs.[b]:waiting for 76 secs before thrashing 2023-03-03T18:55:43.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:55:43.361 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:56:13.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:56:13.433 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:56:33.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:56:33.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.382+0000 7f568348a700 1 Processor -- start 2023-03-03T18:56:33.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.382+0000 7f568348a700 1 -- start start 2023-03-03T18:56:33.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f568348a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f567c0ab0c0 0x7f567c0c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:56:33.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f568348a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f567c06a720 0x7f567c158150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:56:33.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f568348a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c158690 con 0x7f567c06a2b0 2023-03-03T18:56:33.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f568348a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c154530 con 0x7f567c06a720 2023-03-03T18:56:33.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f568348a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c154810 con 0x7f567c0ab0c0 2023-03-03T18:56:33.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f5681a27700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f567c06a720 0x7f567c158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:56:33.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f5681a27700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f567c06a720 0x7f567c158150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39260/0 (socket says 172.21.15.7:39260) 2023-03-03T18:56:33.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f5681a27700 1 -- 172.21.15.7:0/1849542948 learned_addr learned my addr 172.21.15.7:0/1849542948 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:56:33.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f5681226700 1 --2- 172.21.15.7:0/1849542948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f567c0ab0c0 0x7f567c0c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:56:33.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f5681a27700 1 -- 172.21.15.7:0/1849542948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f567c0ab0c0 msgr2=0x7f567c0c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:56:33.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f5681a27700 1 --2- 172.21.15.7:0/1849542948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f567c0ab0c0 0x7f567c0c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:56:33.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f5681a27700 1 -- 172.21.15.7:0/1849542948 >> v1:172.21.15.143:6789/0 conn(0x7f567c06a2b0 legacy=0x7f567c157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:56:33.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.383+0000 7f5681a27700 1 -- 172.21.15.7:0/1849542948 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c154af0 con 0x7f567c06a720 2023-03-03T18:56:33.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f5681a27700 1 --2- 172.21.15.7:0/1849542948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 0x7f567c158150 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7f5674009e50 tx=0x7f567400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73928c2b6656118 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:56:33.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f566ffff700 1 -- 172.21.15.7:0/1849542948 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567400e070 con 0x7f567c06a720 2023-03-03T18:56:33.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f566ffff700 1 -- 172.21.15.7:0/1849542948 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5674003a20 con 0x7f567c06a720 2023-03-03T18:56:33.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f566ffff700 1 -- 172.21.15.7:0/1849542948 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5674003dd0 con 0x7f567c06a720 2023-03-03T18:56:33.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f568348a700 1 -- 172.21.15.7:0/1849542948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 msgr2=0x7f567c158150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:56:33.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f568348a700 1 --2- 172.21.15.7:0/1849542948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 0x7f567c158150 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7f5674009e50 tx=0x7f567400b330 comp rx=0 tx=0).stop 2023-03-03T18:56:33.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f568348a700 1 -- 172.21.15.7:0/1849542948 shutdown_connections 2023-03-03T18:56:33.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f568348a700 1 --2- 172.21.15.7:0/1849542948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 0x7f567c158150 unknown :-1 s=CLOSED pgs=3331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:56:33.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f568348a700 1 --2- 172.21.15.7:0/1849542948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f567c0ab0c0 0x7f567c0c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:56:33.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f568348a700 1 -- 172.21.15.7:0/1849542948 >> 172.21.15.7:0/1849542948 conn(0x7f567c0a9430 msgr2=0x7f567c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:56:33.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f568348a700 1 -- 172.21.15.7:0/1849542948 shutdown_connections 2023-03-03T18:56:33.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.384+0000 7f568348a700 1 -- 172.21.15.7:0/1849542948 wait complete. 2023-03-03T18:56:33.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f568348a700 1 Processor -- start 2023-03-03T18:56:33.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f568348a700 1 -- start start 2023-03-03T18:56:33.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f568348a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f567c06a2b0 0x7f567c150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:56:33.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f568348a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 0x7f567c156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:56:33.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f568348a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f567c0ab0c0 0x7f567c1e0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:56:33.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f568348a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c1eb080 con 0x7f567c06a720 2023-03-03T18:56:33.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f568348a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c06d950 con 0x7f567c06a2b0 2023-03-03T18:56:33.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f568348a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c06c090 con 0x7f567c0ab0c0 2023-03-03T18:56:33.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f5680a25700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 0x7f567c156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:56:33.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f5680a25700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 0x7f567c156af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39264/0 (socket says 172.21.15.7:39264) 2023-03-03T18:56:33.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f5680a25700 1 -- 172.21.15.7:0/3336001984 learned_addr learned my addr 172.21.15.7:0/3336001984 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:56:33.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.386+0000 7f5681226700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f567c06a2b0 0x7f567c150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:56:33.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f5681a27700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f567c0ab0c0 0x7f567c1e0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:56:33.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f5680a25700 1 -- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f567c0ab0c0 msgr2=0x7f567c1e0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:56:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f5680a25700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f567c0ab0c0 0x7f567c1e0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:56:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f5680a25700 1 -- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f567c06a2b0 msgr2=0x7f567c150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:56:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f5680a25700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f567c06a2b0 0x7f567c150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:56:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f5680a25700 1 -- 172.21.15.7:0/3336001984 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5674009b30 con 0x7f567c06a720 2023-03-03T18:56:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f5680a25700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 0x7f567c156af0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f567000b4a0 tx=0x7f567000b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:56:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f566dffb700 1 -- 172.21.15.7:0/3336001984 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670011830 con 0x7f567c06a720 2023-03-03T18:56:33.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f566dffb700 1 -- 172.21.15.7:0/3336001984 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5670011990 con 0x7f567c06a720 2023-03-03T18:56:33.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f566dffb700 1 -- 172.21.15.7:0/3336001984 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670011d40 con 0x7f567c06a720 2023-03-03T18:56:33.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f568348a700 1 -- 172.21.15.7:0/3336001984 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567c1550b0 con 0x7f567c06a720 2023-03-03T18:56:33.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.387+0000 7f568348a700 1 -- 172.21.15.7:0/3336001984 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c154dd0 con 0x7f567c06a720 2023-03-03T18:56:33.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.388+0000 7f568348a700 1 -- 172.21.15.7:0/3336001984 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c0ab490 con 0x7f567c06a720 2023-03-03T18:56:33.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.389+0000 7f566dffb700 1 -- 172.21.15.7:0/3336001984 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f567000f050 con 0x7f567c06a720 2023-03-03T18:56:33.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.389+0000 7f566dffb700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f56600682f0 0x7f566006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:56:33.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.389+0000 7f566dffb700 1 -- 172.21.15.7:0/3336001984 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5670085c30 con 0x7f567c06a720 2023-03-03T18:56:33.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.389+0000 7f5681226700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f56600682f0 0x7f566006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:56:33.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.390+0000 7f5681226700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f56600682f0 0x7f566006a7a0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f567c1573b0 tx=0x7f5678008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:56:33.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.393+0000 7f566dffb700 1 -- 172.21.15.7:0/3336001984 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5670051fd0 con 0x7f567c06a720 2023-03-03T18:56:33.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.631+0000 7f568348a700 1 -- 172.21.15.7:0/3336001984 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f567c074bb0 con 0x7f567c06a720 2023-03-03T18:56:33.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.632+0000 7f566dffb700 1 -- 172.21.15.7:0/3336001984 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f56700555f0 con 0x7f567c06a720 2023-03-03T18:56:33.632 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:56:33.633 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:56:33.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 -- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f56600682f0 msgr2=0x7f566006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:56:33.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f56600682f0 0x7f566006a7a0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f567c1573b0 tx=0x7f5678008040 comp rx=0 tx=0).stop 2023-03-03T18:56:33.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 -- 172.21.15.7:0/3336001984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 msgr2=0x7f567c156af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:56:33.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 0x7f567c156af0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f567000b4a0 tx=0x7f567000b7b0 comp rx=0 tx=0).stop 2023-03-03T18:56:33.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 -- 172.21.15.7:0/3336001984 shutdown_connections 2023-03-03T18:56:33.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f567c0ab0c0 0x7f567c1e0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:56:33.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f567c06a2b0 0x7f567c150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:56:33.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f56600682f0 0x7f566006a7a0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:56:33.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 --2- 172.21.15.7:0/3336001984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f567c06a720 0x7f567c156af0 unknown :-1 s=CLOSED pgs=3332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:56:33.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 -- 172.21.15.7:0/3336001984 >> 172.21.15.7:0/3336001984 conn(0x7f567c0a9430 msgr2=0x7f567c0c5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:56:33.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 -- 172.21.15.7:0/3336001984 shutdown_connections 2023-03-03T18:56:33.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:56:33.634+0000 7f568348a700 1 -- 172.21.15.7:0/3336001984 wait complete. 2023-03-03T18:56:33.636 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:56:33.646 INFO:tasks.mds_thrash.fs.[b]:waiting for 39 secs before thrashing 2023-03-03T18:56:43.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:56:43.504 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:57:04.514 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:57:04.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.652+0000 7feae5a2d700 1 Processor -- start 2023-03-03T18:57:04.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.653+0000 7feae5a2d700 1 -- start start 2023-03-03T18:57:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.653+0000 7feae5a2d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feae00c00e0 0x7feae00c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.653+0000 7feae5a2d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae0154ff0 0x7feae00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.653+0000 7feae5a2d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae00c7310 con 0x7feae00caa30 2023-03-03T18:57:04.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.653+0000 7feae5a2d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae00bfd10 con 0x7feae00c00e0 2023-03-03T18:57:04.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.653+0000 7feae5a2d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae00bfa30 con 0x7feae0154ff0 2023-03-03T18:57:04.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.653+0000 7feadeffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7feae00caa30 0x7feae00c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33280/0 (socket says 172.21.15.7:33280) 2023-03-03T18:57:04.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.653+0000 7feadeffd700 1 -- 172.21.15.7:0/3629563006 learned_addr learned my addr 172.21.15.7:0/3629563006 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:57:04.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.653+0000 7feade7fc700 1 --2- 172.21.15.7:0/3629563006 >> v2:172.21.15.7:3300/0 conn(0x7feae00c00e0 0x7feae00c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:04.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.654+0000 7feadf7fe700 1 --2- 172.21.15.7:0/3629563006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae0154ff0 0x7feae00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:04.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.654+0000 7feade7fc700 1 -- 172.21.15.7:0/3629563006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae0154ff0 msgr2=0x7feae00cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:57:04.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.654+0000 7feade7fc700 1 --2- 172.21.15.7:0/3629563006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae0154ff0 0x7feae00cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:04.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.654+0000 7feade7fc700 1 -- 172.21.15.7:0/3629563006 >> v1:172.21.15.7:6789/0 conn(0x7feae00caa30 legacy=0x7feae00c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:04.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.654+0000 7feade7fc700 1 -- 172.21.15.7:0/3629563006 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae0154510 con 0x7feae00c00e0 2023-03-03T18:57:04.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.654+0000 7feaddffb700 1 -- 172.21.15.7:0/3629563006 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254214475 0 0) 0x7feae00c7310 con 0x7feae00caa30 2023-03-03T18:57:04.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.654+0000 7feade7fc700 1 --2- 172.21.15.7:0/3629563006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00c00e0 0x7feae00c5c60 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7feac8009820 tx=0x7feac8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5864cf4b24bae1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:57:04.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.654+0000 7feaddffb700 1 -- 172.21.15.7:0/3629563006 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac8005e30 con 0x7feae00c00e0 2023-03-03T18:57:04.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.654+0000 7feaddffb700 1 -- 172.21.15.7:0/3629563006 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac8006030 con 0x7feae00c00e0 2023-03-03T18:57:04.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.654+0000 7feaddffb700 1 -- 172.21.15.7:0/3629563006 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac80063e0 con 0x7feae00c00e0 2023-03-03T18:57:04.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.655+0000 7feae5a2d700 1 -- 172.21.15.7:0/3629563006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00c00e0 msgr2=0x7feae00c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:04.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.655+0000 7feae5a2d700 1 --2- 172.21.15.7:0/3629563006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00c00e0 0x7feae00c5c60 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7feac8009820 tx=0x7feac8009b30 comp rx=0 tx=0).stop 2023-03-03T18:57:04.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.655+0000 7feae5a2d700 1 -- 172.21.15.7:0/3629563006 shutdown_connections 2023-03-03T18:57:04.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.655+0000 7feae5a2d700 1 --2- 172.21.15.7:0/3629563006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae0154ff0 0x7feae00cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:04.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.655+0000 7feae5a2d700 1 --2- 172.21.15.7:0/3629563006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00c00e0 0x7feae00c5c60 unknown :-1 s=CLOSED pgs=3335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:04.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.655+0000 7feae5a2d700 1 -- 172.21.15.7:0/3629563006 >> 172.21.15.7:0/3629563006 conn(0x7feae0066e40 msgr2=0x7feae0071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:57:04.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.655+0000 7feae5a2d700 1 -- 172.21.15.7:0/3629563006 shutdown_connections 2023-03-03T18:57:04.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.655+0000 7feae5a2d700 1 -- 172.21.15.7:0/3629563006 wait complete. 2023-03-03T18:57:04.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.656+0000 7feae5a2d700 1 Processor -- start 2023-03-03T18:57:04.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.656+0000 7feae5a2d700 1 -- start start 2023-03-03T18:57:04.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feae5a2d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae00c00e0 0x7feae01ea120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:04.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feae5a2d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00caa30 0x7feae00c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:04.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feae5a2d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae0154ff0 0x7feae00c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:04.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feae5a2d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae00c3db0 con 0x7feae00caa30 2023-03-03T18:57:04.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feae5a2d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae0158520 con 0x7feae00c00e0 2023-03-03T18:57:04.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feae5a2d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae01ea660 con 0x7feae0154ff0 2023-03-03T18:57:04.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feade7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00caa30 0x7feae00c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:04.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feade7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00caa30 0x7feae00c23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39308/0 (socket says 172.21.15.7:39308) 2023-03-03T18:57:04.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feade7fc700 1 -- 172.21.15.7:0/420375897 learned_addr learned my addr 172.21.15.7:0/420375897 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:57:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feadf7fe700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae0154ff0 0x7feae00c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feadeffd700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae00c00e0 0x7feae01ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feade7fc700 1 -- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae0154ff0 msgr2=0x7feae00c41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feade7fc700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae0154ff0 0x7feae00c41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feade7fc700 1 -- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae00c00e0 msgr2=0x7feae01ea120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feade7fc700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae00c00e0 0x7feae01ea120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:04.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feade7fc700 1 -- 172.21.15.7:0/420375897 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac8009430 con 0x7feae00caa30 2023-03-03T18:57:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.657+0000 7feade7fc700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00caa30 0x7feae00c23d0 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7feac8013040 tx=0x7feac8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:57:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.658+0000 7fead7fff700 1 -- 172.21.15.7:0/420375897 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac8025430 con 0x7feae00caa30 2023-03-03T18:57:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.658+0000 7fead7fff700 1 -- 172.21.15.7:0/420375897 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac8020070 con 0x7feae00caa30 2023-03-03T18:57:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.658+0000 7feae5a2d700 1 -- 172.21.15.7:0/420375897 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae0154ad0 con 0x7feae00caa30 2023-03-03T18:57:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.658+0000 7fead7fff700 1 -- 172.21.15.7:0/420375897 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac8025810 con 0x7feae00caa30 2023-03-03T18:57:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.658+0000 7feae5a2d700 1 -- 172.21.15.7:0/420375897 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae01547f0 con 0x7feae00caa30 2023-03-03T18:57:04.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.659+0000 7feae5a2d700 1 -- 172.21.15.7:0/420375897 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae00bf650 con 0x7feae00caa30 2023-03-03T18:57:04.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.659+0000 7fead7fff700 1 -- 172.21.15.7:0/420375897 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feac802b020 con 0x7feae00caa30 2023-03-03T18:57:04.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.659+0000 7fead7fff700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac00682f0 0x7feac006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:04.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.660+0000 7fead7fff700 1 -- 172.21.15.7:0/420375897 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7feac8087a30 con 0x7feae00caa30 2023-03-03T18:57:04.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.660+0000 7feadeffd700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac00682f0 0x7feac006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:04.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.660+0000 7feadeffd700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac00682f0 0x7feac006a7a0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fead0000f60 tx=0x7fead0007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:57:04.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.664+0000 7fead7fff700 1 -- 172.21.15.7:0/420375897 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feac8053dd0 con 0x7feae00caa30 2023-03-03T18:57:04.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.903+0000 7feae5a2d700 1 -- 172.21.15.7:0/420375897 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feae00c3ef0 con 0x7feae00caa30 2023-03-03T18:57:04.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.904+0000 7fead7fff700 1 -- 172.21.15.7:0/420375897 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7feae00c3ef0 con 0x7feae00caa30 2023-03-03T18:57:04.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:57:04.905 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:57:04.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 -- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac00682f0 msgr2=0x7feac006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:04.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac00682f0 0x7feac006a7a0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fead0000f60 tx=0x7fead0007be0 comp rx=0 tx=0).stop 2023-03-03T18:57:04.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 -- 172.21.15.7:0/420375897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00caa30 msgr2=0x7feae00c23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:04.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00caa30 0x7feae00c23d0 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7feac8013040 tx=0x7feac8004750 comp rx=0 tx=0).stop 2023-03-03T18:57:04.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 -- 172.21.15.7:0/420375897 shutdown_connections 2023-03-03T18:57:04.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7feae0154ff0 0x7feae00c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:04.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feae00c00e0 0x7feae01ea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:04.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feac00682f0 0x7feac006a7a0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:04.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 --2- 172.21.15.7:0/420375897 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feae00caa30 0x7feae00c23d0 unknown :-1 s=CLOSED pgs=3336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:04.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 -- 172.21.15.7:0/420375897 >> 172.21.15.7:0/420375897 conn(0x7feae0066e40 msgr2=0x7feae01536d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:57:04.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 -- 172.21.15.7:0/420375897 shutdown_connections 2023-03-03T18:57:04.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:04.906+0000 7feae5a2d700 1 -- 172.21.15.7:0/420375897 wait complete. 2023-03-03T18:57:04.909 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:57:04.919 INFO:tasks.mds_thrash.fs.[a]:waiting for 63 secs before thrashing 2023-03-03T18:57:12.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:57:12.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.788+0000 7f6713250700 1 Processor -- start 2023-03-03T18:57:12.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.788+0000 7f6713250700 1 -- start start 2023-03-03T18:57:12.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f6713250700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f670c1530d0 0x7f670c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f6713250700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670c0c2f30 con 0x7f670c14fda0 2023-03-03T18:57:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f6713250700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670c0c31d0 con 0x7f670c14f9d0 2023-03-03T18:57:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f6713250700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670c1524a0 con 0x7f670c1530d0 2023-03-03T18:57:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f67117ed700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f670c1530d0 0x7f670c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f670bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f670c14fda0 0x7f670c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33300/0 (socket says 172.21.15.7:33300) 2023-03-03T18:57:12.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f670bfff700 1 -- 172.21.15.7:0/3539207732 learned_addr learned my addr 172.21.15.7:0/3539207732 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:57:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f67117ed700 1 -- 172.21.15.7:0/3539207732 >> v1:172.21.15.143:6789/0 conn(0x7f670c14f9d0 legacy=0x7f670c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f67117ed700 1 -- 172.21.15.7:0/3539207732 >> v1:172.21.15.7:6789/0 conn(0x7f670c14fda0 legacy=0x7f670c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f67117ed700 1 -- 172.21.15.7:0/3539207732 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c0c34b0 con 0x7f670c1530d0 2023-03-03T18:57:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.789+0000 7f670b7fe700 1 -- 172.21.15.7:0/3539207732 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246323710 0 0) 0x7f670c0c2f30 con 0x7f670c14fda0 2023-03-03T18:57:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f67117ed700 1 --2- 172.21.15.7:0/3539207732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 0x7f670c156430 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 crypto rx=0x7f6704009d90 tx=0x7f670400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1de64aad7fc0d5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:57:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f670b7fe700 1 -- 172.21.15.7:0/3539207732 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400e070 con 0x7f670c1530d0 2023-03-03T18:57:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f670b7fe700 1 -- 172.21.15.7:0/3539207732 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670400bbd0 con 0x7f670c1530d0 2023-03-03T18:57:12.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f670b7fe700 1 -- 172.21.15.7:0/3539207732 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704012430 con 0x7f670c1530d0 2023-03-03T18:57:12.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f6713250700 1 -- 172.21.15.7:0/3539207732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 msgr2=0x7f670c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:12.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f6713250700 1 --2- 172.21.15.7:0/3539207732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 0x7f670c156430 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 crypto rx=0x7f6704009d90 tx=0x7f670400b330 comp rx=0 tx=0).stop 2023-03-03T18:57:12.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f6713250700 1 -- 172.21.15.7:0/3539207732 shutdown_connections 2023-03-03T18:57:12.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f6713250700 1 --2- 172.21.15.7:0/3539207732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 0x7f670c156430 unknown :-1 s=CLOSED pgs=3338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:12.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f6713250700 1 -- 172.21.15.7:0/3539207732 >> 172.21.15.7:0/3539207732 conn(0x7f670c0c8900 msgr2=0x7f670c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:57:12.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f6713250700 1 -- 172.21.15.7:0/3539207732 shutdown_connections 2023-03-03T18:57:12.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.790+0000 7f6713250700 1 -- 172.21.15.7:0/3539207732 wait complete. 2023-03-03T18:57:12.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.792+0000 7f6713250700 1 Processor -- start 2023-03-03T18:57:12.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.792+0000 7f6713250700 1 -- start start 2023-03-03T18:57:12.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f6713250700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670c14f9d0 0x7f670c1216f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:12.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f6713250700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c14fda0 0x7f670c0c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:12.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f6713250700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 0x7f670c06e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:12.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f6713250700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c1221a0 con 0x7f670c1530d0 2023-03-03T18:57:12.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f6713250700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c145f90 con 0x7f670c14fda0 2023-03-03T18:57:12.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f6713250700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670c1e9a70 con 0x7f670c14f9d0 2023-03-03T18:57:12.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f67117ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 0x7f670c06e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:12.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f67117ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 0x7f670c06e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39328/0 (socket says 172.21.15.7:39328) 2023-03-03T18:57:12.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f67117ed700 1 -- 172.21.15.7:0/1174002805 learned_addr learned my addr 172.21.15.7:0/1174002805 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:57:12.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f670bfff700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c14fda0 0x7f670c0c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:12.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f6710fec700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670c14f9d0 0x7f670c1216f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:12.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.793+0000 7f67117ed700 1 -- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670c14f9d0 msgr2=0x7f670c1216f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:12.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f67117ed700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670c14f9d0 0x7f670c1216f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:12.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f67117ed700 1 -- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c14fda0 msgr2=0x7f670c0c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:12.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f67117ed700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c14fda0 0x7f670c0c7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:12.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f67117ed700 1 -- 172.21.15.7:0/1174002805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6704009a70 con 0x7f670c1530d0 2023-03-03T18:57:12.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f6710fec700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670c14f9d0 0x7f670c1216f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:57:12.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f670bfff700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c14fda0 0x7f670c0c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:57:12.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f67117ed700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 0x7f670c06e690 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7f6704014040 tx=0x7f6704004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:57:12.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f67097fa700 1 -- 172.21.15.7:0/1174002805 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704012430 con 0x7f670c1530d0 2023-03-03T18:57:12.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f67097fa700 1 -- 172.21.15.7:0/1174002805 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6704020070 con 0x7f670c1530d0 2023-03-03T18:57:12.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f6713250700 1 -- 172.21.15.7:0/1174002805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670c0ac350 con 0x7f670c1530d0 2023-03-03T18:57:12.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f67097fa700 1 -- 172.21.15.7:0/1174002805 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670401bce0 con 0x7f670c1530d0 2023-03-03T18:57:12.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.794+0000 7f6713250700 1 -- 172.21.15.7:0/1174002805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670c0ab630 con 0x7f670c1530d0 2023-03-03T18:57:12.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.795+0000 7f6713250700 1 -- 172.21.15.7:0/1174002805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670c071fa0 con 0x7f670c1530d0 2023-03-03T18:57:12.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.795+0000 7f67097fa700 1 -- 172.21.15.7:0/1174002805 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6704026400 con 0x7f670c1530d0 2023-03-03T18:57:12.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.796+0000 7f67097fa700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66f00682f0 0x7f66f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:12.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.796+0000 7f67097fa700 1 -- 172.21.15.7:0/1174002805 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6704088320 con 0x7f670c1530d0 2023-03-03T18:57:12.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.796+0000 7f6710fec700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66f00682f0 0x7f66f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:12.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.797+0000 7f6710fec700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66f00682f0 0x7f66f006a7a0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f6700000f60 tx=0x7f6700009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:57:12.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:12.800+0000 7f67097fa700 1 -- 172.21.15.7:0/1174002805 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6704054660 con 0x7f670c1530d0 2023-03-03T18:57:13.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.044+0000 7f6713250700 1 -- 172.21.15.7:0/1174002805 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f670c074bb0 con 0x7f670c1530d0 2023-03-03T18:57:13.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.045+0000 7f67097fa700 1 -- 172.21.15.7:0/1174002805 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6704057c80 con 0x7f670c1530d0 2023-03-03T18:57:13.045 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:57:13.045 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:57:13.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 -- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66f00682f0 msgr2=0x7f66f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:13.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66f00682f0 0x7f66f006a7a0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f6700000f60 tx=0x7f6700009040 comp rx=0 tx=0).stop 2023-03-03T18:57:13.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 -- 172.21.15.7:0/1174002805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 msgr2=0x7f670c06e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:13.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 0x7f670c06e690 secure :-1 s=READY pgs=3339 cs=0 l=1 rev1=1 crypto rx=0x7f6704014040 tx=0x7f6704004ad0 comp rx=0 tx=0).stop 2023-03-03T18:57:13.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 -- 172.21.15.7:0/1174002805 shutdown_connections 2023-03-03T18:57:13.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f66f00682f0 0x7f66f006a7a0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:13.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f670c1530d0 0x7f670c06e690 unknown :-1 s=CLOSED pgs=3339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:13.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f670c14f9d0 0x7f670c1216f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:13.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 --2- 172.21.15.7:0/1174002805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f670c14fda0 0x7f670c0c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:13.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 -- 172.21.15.7:0/1174002805 >> 172.21.15.7:0/1174002805 conn(0x7f670c0c8900 msgr2=0x7f670c14ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:57:13.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 -- 172.21.15.7:0/1174002805 shutdown_connections 2023-03-03T18:57:13.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:13.047+0000 7f6713250700 1 -- 172.21.15.7:0/1174002805 wait complete. 2023-03-03T18:57:13.049 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:57:13.059 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T18:57:13.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:57:13.583 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:57:24.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:57:24.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.199+0000 7fd80080d700 1 Processor -- start 2023-03-03T18:57:24.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.199+0000 7fd80080d700 1 -- start start 2023-03-03T18:57:24.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.200+0000 7fd80080d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd7fc06c480 0x7fd7fc1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.200+0000 7fd80080d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7fc06cb70 con 0x7fd7fc0c2950 2023-03-03T18:57:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.200+0000 7fd80080d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7fc06ce50 con 0x7fd7fc151b20 2023-03-03T18:57:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.200+0000 7fd80080d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7fc06c2f0 con 0x7fd7fc06c480 2023-03-03T18:57:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f9d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd7fc0c2950 0x7fd7fc157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33316/0 (socket says 172.21.15.7:33316) 2023-03-03T18:57:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f9d9b700 1 -- 172.21.15.7:0/2904294738 learned_addr learned my addr 172.21.15.7:0/2904294738 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:57:24.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7fad9d700 1 --2- 172.21.15.7:0/2904294738 >> v2:172.21.15.143:3300/0 conn(0x7fd7fc06c480 0x7fd7fc1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479636384 0 0) 0x7fd7fc06cb70 con 0x7fd7fc0c2950 2023-03-03T18:57:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7e40035b0 con 0x7fd7fc0c2950 2023-03-03T18:57:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355608929 0 0) 0x7fd7e40035b0 con 0x7fd7fc0c2950 2023-03-03T18:57:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 >> v2:172.21.15.143:3300/0 conn(0x7fd7fc06c480 msgr2=0x7fd7fc1581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:57:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 --2- 172.21.15.7:0/2904294738 >> v2:172.21.15.143:3300/0 conn(0x7fd7fc06c480 0x7fd7fc1581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 >> v1:172.21.15.143:6789/0 conn(0x7fd7fc151b20 legacy=0x7fd7fc0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:24.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7fc06d130 con 0x7fd7fc0c2950 2023-03-03T18:57:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd7f0001670 con 0x7fd7fc0c2950 2023-03-03T18:57:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 >> v1:172.21.15.7:6789/0 conn(0x7fd7fc0c2950 legacy=0x7fd7fc157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e4003a70 0x7fd7e4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7e4004850 0x7fd7e4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7e4005250 0x7fd7e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc06c2f0 con 0x7fd7e4003a70 2023-03-03T18:57:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e400c190 con 0x7fd7e4005250 2023-03-03T18:57:24.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e400c350 con 0x7fd7e4004850 2023-03-03T18:57:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7fa59c700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e4003a70 0x7fd7e4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.201+0000 7fd7f9d9b700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7e4004850 0x7fd7e4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7fad9d700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7e4005250 0x7fd7e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7fa59c700 1 -- 172.21.15.7:0/2904294738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7e4004850 msgr2=0x7fd7e4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:57:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7fa59c700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7e4004850 0x7fd7e4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7fa59c700 1 -- 172.21.15.7:0/2904294738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7e4005250 msgr2=0x7fd7e400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:57:24.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7fa59c700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7e4005250 0x7fd7e400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7fa59c700 1 -- 172.21.15.7:0/2904294738 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e40035b0 con 0x7fd7e4003a70 2023-03-03T18:57:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7fad9d700 1 -- 172.21.15.7:0/2904294738 reap_dead start 2023-03-03T18:57:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7fa59c700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e4003a70 0x7fd7e4003e80 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec007fd0 tx=0x7fd7e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70990096d7bdbe6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:57:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec0052d0 con 0x7fd7e4003a70 2023-03-03T18:57:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ec005450 con 0x7fd7e4003a70 2023-03-03T18:57:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd7f959a700 1 -- 172.21.15.7:0/2904294738 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec0038c0 con 0x7fd7e4003a70 2023-03-03T18:57:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd80080d700 1 -- 172.21.15.7:0/2904294738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e4003a70 msgr2=0x7fd7e4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:24.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.202+0000 7fd80080d700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e4003a70 0x7fd7e4003e80 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec007fd0 tx=0x7fd7e4005880 comp rx=0 tx=0).stop 2023-03-03T18:57:24.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.203+0000 7fd80080d700 1 -- 172.21.15.7:0/2904294738 shutdown_connections 2023-03-03T18:57:24.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.203+0000 7fd80080d700 1 --2- 172.21.15.7:0/2904294738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7e4003a70 0x7fd7e4003e80 unknown :-1 s=CLOSED pgs=3341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:24.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.203+0000 7fd80080d700 1 -- 172.21.15.7:0/2904294738 >> 172.21.15.7:0/2904294738 conn(0x7fd7fc0a94d0 msgr2=0x7fd7fc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:57:24.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.203+0000 7fd80080d700 1 -- 172.21.15.7:0/2904294738 shutdown_connections 2023-03-03T18:57:24.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.203+0000 7fd80080d700 1 -- 172.21.15.7:0/2904294738 wait complete. 2023-03-03T18:57:24.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.204+0000 7fd80080d700 1 Processor -- start 2023-03-03T18:57:24.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.204+0000 7fd80080d700 1 -- start start 2023-03-03T18:57:24.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd80080d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7e4003a70 0x7fd7fc1ddeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:24.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd80080d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0c6320 0x7fd7fc1ed6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:24.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd80080d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc1e9fc0 0x7fd7fc148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:24.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd80080d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc1dff40 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd80080d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc0c5800 con 0x7fd7fc0c6320 2023-03-03T18:57:24.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd80080d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc1edc10 con 0x7fd7e4003a70 2023-03-03T18:57:24.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc1e9fc0 0x7fd7fc148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:24.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc1e9fc0 0x7fd7fc148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39350/0 (socket says 172.21.15.7:39350) 2023-03-03T18:57:24.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fad9d700 1 -- 172.21.15.7:0/356792516 learned_addr learned my addr 172.21.15.7:0/356792516 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:57:24.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fa59c700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7e4003a70 0x7fd7fc1ddeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:24.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7f9d9b700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0c6320 0x7fd7fc1ed6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:24.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fad9d700 1 -- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7e4003a70 msgr2=0x7fd7fc1ddeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:24.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fad9d700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7e4003a70 0x7fd7fc1ddeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:24.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fad9d700 1 -- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0c6320 msgr2=0x7fd7fc1ed6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:24.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fad9d700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0c6320 0x7fd7fc1ed6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:24.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fad9d700 1 -- 172.21.15.7:0/356792516 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7fc151ef0 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fa59c700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7e4003a70 0x7fd7fc1ddeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:57:24.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7f9d9b700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0c6320 0x7fd7fc1ed6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:57:24.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.205+0000 7fd7fad9d700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc1e9fc0 0x7fd7fc148130 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0be3c0 tx=0x7fd7f4011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:57:24.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.206+0000 7fd7eaffd700 1 -- 172.21.15.7:0/356792516 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f400e040 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.206+0000 7fd7eaffd700 1 -- 172.21.15.7:0/356792516 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f400f040 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.206+0000 7fd7eaffd700 1 -- 172.21.15.7:0/356792516 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f400fd80 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.206+0000 7fd80080d700 1 -- 172.21.15.7:0/356792516 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7fc06d410 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.206+0000 7fd80080d700 1 -- 172.21.15.7:0/356792516 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7fc06bea0 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.207+0000 7fd80080d700 1 -- 172.21.15.7:0/356792516 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7fc074bb0 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.207+0000 7fd7eaffd700 1 -- 172.21.15.7:0/356792516 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd7f4022460 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.207+0000 7fd7eaffd700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7dc0682f0 0x7fd7dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:57:24.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.207+0000 7fd7eaffd700 1 -- 172.21.15.7:0/356792516 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd7f40867f0 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.208+0000 7fd7fa59c700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7dc0682f0 0x7fd7dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:57:24.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.208+0000 7fd7fa59c700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7dc0682f0 0x7fd7dc06a7a0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0c7110 tx=0x7fd7ec009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:57:24.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.212+0000 7fd7eaffd700 1 -- 172.21.15.7:0/356792516 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7f4052b30 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.449+0000 7fd80080d700 1 -- 172.21.15.7:0/356792516 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7fc0c26a0 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.449+0000 7fd7eaffd700 1 -- 172.21.15.7:0/356792516 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd7f4056150 con 0x7fd7fc1e9fc0 2023-03-03T18:57:24.450 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:57:24.450 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:57:24.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 -- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7dc0682f0 msgr2=0x7fd7dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7dc0682f0 0x7fd7dc06a7a0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0c7110 tx=0x7fd7ec009030 comp rx=0 tx=0).stop 2023-03-03T18:57:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 -- 172.21.15.7:0/356792516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc1e9fc0 msgr2=0x7fd7fc148130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:57:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc1e9fc0 0x7fd7fc148130 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0be3c0 tx=0x7fd7f4011f80 comp rx=0 tx=0).stop 2023-03-03T18:57:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 -- 172.21.15.7:0/356792516 shutdown_connections 2023-03-03T18:57:24.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7dc0682f0 0x7fd7dc06a7a0 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:24.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7fc1e9fc0 0x7fd7fc148130 unknown :-1 s=CLOSED pgs=3342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:24.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7e4003a70 0x7fd7fc1ddeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:24.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 --2- 172.21.15.7:0/356792516 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0c6320 0x7fd7fc1ed6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:57:24.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 -- 172.21.15.7:0/356792516 >> 172.21.15.7:0/356792516 conn(0x7fd7fc0a94d0 msgr2=0x7fd7fc14ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:57:24.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.451+0000 7fd80080d700 1 -- 172.21.15.7:0/356792516 shutdown_connections 2023-03-03T18:57:24.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:57:24.452+0000 7fd80080d700 1 -- 172.21.15.7:0/356792516 wait complete. 2023-03-03T18:57:24.455 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:57:24.464 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-03-03T18:57:43.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:57:43.646 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:58:07.920 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:58:08.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.059+0000 7f2b2b705700 1 Processor -- start 2023-03-03T18:58:08.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.060+0000 7f2b2b705700 1 -- start start 2023-03-03T18:58:08.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.060+0000 7f2b2b705700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2b240caa30 0x7f2b240c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:08.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.060+0000 7f2b2b705700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b240c00e0 0x7f2b24154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:08.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.060+0000 7f2b2b705700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b240cb280 con 0x7f2b240cae00 2023-03-03T18:58:08.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.060+0000 7f2b2b705700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b240c7310 con 0x7f2b240caa30 2023-03-03T18:58:08.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.060+0000 7f2b2b705700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b240bfd10 con 0x7f2b240c00e0 2023-03-03T18:58:08.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b294a1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2b240caa30 0x7f2b240c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:08.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b294a1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2b240caa30 0x7f2b240c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39392/0 (socket says 172.21.15.7:39392) 2023-03-03T18:58:08.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b294a1700 1 -- 172.21.15.7:0/1993435469 learned_addr learned my addr 172.21.15.7:0/1993435469 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:58:08.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b28ca0700 1 --2- 172.21.15.7:0/1993435469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b240c00e0 0x7f2b24154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:08.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b294a1700 1 -- 172.21.15.7:0/1993435469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b240c00e0 msgr2=0x7f2b24154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:58:08.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b294a1700 1 --2- 172.21.15.7:0/1993435469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b240c00e0 0x7f2b24154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:08.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b294a1700 1 -- 172.21.15.7:0/1993435469 >> v1:172.21.15.143:6789/0 conn(0x7f2b240cae00 legacy=0x7f2b240c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:08.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b294a1700 1 -- 172.21.15.7:0/1993435469 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b240bfa30 con 0x7f2b240caa30 2023-03-03T18:58:08.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b294a1700 1 --2- 172.21.15.7:0/1993435469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 0x7f2b240c5c60 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7f2b200098f0 tx=0x7f2b20009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601e8317e5e89621 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:58:08.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b1bfff700 1 -- 172.21.15.7:0/1993435469 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20011830 con 0x7f2b240caa30 2023-03-03T18:58:08.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b1bfff700 1 -- 172.21.15.7:0/1993435469 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b20011990 con 0x7f2b240caa30 2023-03-03T18:58:08.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.061+0000 7f2b1bfff700 1 -- 172.21.15.7:0/1993435469 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20011d20 con 0x7f2b240caa30 2023-03-03T18:58:08.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.062+0000 7f2b2b705700 1 -- 172.21.15.7:0/1993435469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 msgr2=0x7f2b240c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:08.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.062+0000 7f2b2b705700 1 --2- 172.21.15.7:0/1993435469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 0x7f2b240c5c60 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7f2b200098f0 tx=0x7f2b20009c00 comp rx=0 tx=0).stop 2023-03-03T18:58:08.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.062+0000 7f2b2b705700 1 -- 172.21.15.7:0/1993435469 shutdown_connections 2023-03-03T18:58:08.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.062+0000 7f2b2b705700 1 --2- 172.21.15.7:0/1993435469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 0x7f2b240c5c60 unknown :-1 s=CLOSED pgs=3344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:08.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.062+0000 7f2b2b705700 1 --2- 172.21.15.7:0/1993435469 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b240c00e0 0x7f2b24154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:08.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.062+0000 7f2b2b705700 1 -- 172.21.15.7:0/1993435469 >> 172.21.15.7:0/1993435469 conn(0x7f2b24066e40 msgr2=0x7f2b24071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:58:08.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.062+0000 7f2b2b705700 1 -- 172.21.15.7:0/1993435469 shutdown_connections 2023-03-03T18:58:08.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.062+0000 7f2b2b705700 1 -- 172.21.15.7:0/1993435469 wait complete. 2023-03-03T18:58:08.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.063+0000 7f2b2b705700 1 Processor -- start 2023-03-03T18:58:08.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.063+0000 7f2b2b705700 1 -- start start 2023-03-03T18:58:08.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b2b705700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b240c00e0 0x7f2b2414cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:08.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b2b705700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 0x7f2b241ea220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:08.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b2b705700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b240cae00 0x7f2b24152c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:08.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b2b705700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b241ecff0 con 0x7f2b240caa30 2023-03-03T18:58:08.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b2b705700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b240c2630 con 0x7f2b240c00e0 2023-03-03T18:58:08.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b2b705700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b241eccc0 con 0x7f2b240cae00 2023-03-03T18:58:08.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b28ca0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 0x7f2b241ea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:08.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b28ca0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 0x7f2b241ea220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39400/0 (socket says 172.21.15.7:39400) 2023-03-03T18:58:08.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b28ca0700 1 -- 172.21.15.7:0/1110450121 learned_addr learned my addr 172.21.15.7:0/1110450121 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:58:08.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b29ca2700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b240cae00 0x7f2b24152c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:08.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b294a1700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b240c00e0 0x7f2b2414cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:08.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b28ca0700 1 -- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b240cae00 msgr2=0x7f2b24152c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:08.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b28ca0700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b240cae00 0x7f2b24152c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:08.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b28ca0700 1 -- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b240c00e0 msgr2=0x7f2b2414cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:08.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b28ca0700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b240c00e0 0x7f2b2414cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:08.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b28ca0700 1 -- 172.21.15.7:0/1110450121 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b20009550 con 0x7f2b240caa30 2023-03-03T18:58:08.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.064+0000 7f2b294a1700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b240c00e0 0x7f2b2414cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:58:08.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.065+0000 7f2b28ca0700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 0x7f2b241ea220 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400e990 tx=0x7f2b1400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:58:08.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.065+0000 7f2b19ffb700 1 -- 172.21.15.7:0/1110450121 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b140096f0 con 0x7f2b240caa30 2023-03-03T18:58:08.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.065+0000 7f2b19ffb700 1 -- 172.21.15.7:0/1110450121 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1400c850 con 0x7f2b240caa30 2023-03-03T18:58:08.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.065+0000 7f2b2b705700 1 -- 172.21.15.7:0/1110450121 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b241547f0 con 0x7f2b240caa30 2023-03-03T18:58:08.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.065+0000 7f2b19ffb700 1 -- 172.21.15.7:0/1110450121 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1400eee0 con 0x7f2b240caa30 2023-03-03T18:58:08.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.065+0000 7f2b2b705700 1 -- 172.21.15.7:0/1110450121 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b24154510 con 0x7f2b240caa30 2023-03-03T18:58:08.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.066+0000 7f2b2b705700 1 -- 172.21.15.7:0/1110450121 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b24154ad0 con 0x7f2b240caa30 2023-03-03T18:58:08.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.066+0000 7f2b19ffb700 1 -- 172.21.15.7:0/1110450121 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2b14023070 con 0x7f2b240caa30 2023-03-03T18:58:08.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.067+0000 7f2b19ffb700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b080682f0 0x7f2b0806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:08.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.067+0000 7f2b19ffb700 1 -- 172.21.15.7:0/1110450121 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2b14086400 con 0x7f2b240caa30 2023-03-03T18:58:08.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.067+0000 7f2b294a1700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b080682f0 0x7f2b0806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:08.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.068+0000 7f2b294a1700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b080682f0 0x7f2b0806a7a0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f2b20000c00 tx=0x7f2b20008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:58:08.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.071+0000 7f2b19ffb700 1 -- 172.21.15.7:0/1110450121 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2b140527a0 con 0x7f2b240caa30 2023-03-03T18:58:08.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.308+0000 7f2b2b705700 1 -- 172.21.15.7:0/1110450121 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b24074bb0 con 0x7f2b240caa30 2023-03-03T18:58:08.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.308+0000 7f2b19ffb700 1 -- 172.21.15.7:0/1110450121 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2b14055dc0 con 0x7f2b240caa30 2023-03-03T18:58:08.309 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:58:08.310 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:58:08.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 -- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b080682f0 msgr2=0x7f2b0806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:08.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b080682f0 0x7f2b0806a7a0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f2b20000c00 tx=0x7f2b20008030 comp rx=0 tx=0).stop 2023-03-03T18:58:08.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 -- 172.21.15.7:0/1110450121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 msgr2=0x7f2b241ea220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 0x7f2b241ea220 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400e990 tx=0x7f2b1400eca0 comp rx=0 tx=0).stop 2023-03-03T18:58:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 -- 172.21.15.7:0/1110450121 shutdown_connections 2023-03-03T18:58:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2b240cae00 0x7f2b24152c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2b240c00e0 0x7f2b2414cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2b080682f0 0x7f2b0806a7a0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 --2- 172.21.15.7:0/1110450121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2b240caa30 0x7f2b241ea220 unknown :-1 s=CLOSED pgs=3345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 -- 172.21.15.7:0/1110450121 >> 172.21.15.7:0/1110450121 conn(0x7f2b24066e40 msgr2=0x7f2b240c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:58:08.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 -- 172.21.15.7:0/1110450121 shutdown_connections 2023-03-03T18:58:08.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:08.310+0000 7f2b2b705700 1 -- 172.21.15.7:0/1110450121 wait complete. 2023-03-03T18:58:08.313 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:58:08.323 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before thrashing 2023-03-03T18:58:13.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:58:13.719 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:58:14.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:58:14.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.463+0000 7fdb432c3700 1 Processor -- start 2023-03-03T18:58:14.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.464+0000 7fdb432c3700 1 -- start start 2023-03-03T18:58:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.464+0000 7fdb432c3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdb3c072250 0x7fdb3c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.464+0000 7fdb432c3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb3c0c1ce0 0x7fdb3c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.464+0000 7fdb432c3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c1510d0 con 0x7fdb3c071e80 2023-03-03T18:58:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.464+0000 7fdb432c3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c0c3740 con 0x7fdb3c0c1ce0 2023-03-03T18:58:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.464+0000 7fdb432c3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb3c070670 con 0x7fdb3c072250 2023-03-03T18:58:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.464+0000 7fdb41860700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb3c0c1ce0 0x7fdb3c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:14.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.464+0000 7fdb4105f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdb3c071e80 0x7fdb3c151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33388/0 (socket says 172.21.15.7:33388) 2023-03-03T18:58:14.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.464+0000 7fdb41860700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdb3c0c1ce0 0x7fdb3c14e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39412/0 (socket says 172.21.15.7:39412) 2023-03-03T18:58:14.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.464+0000 7fdb4105f700 1 -- 172.21.15.7:0/436707886 learned_addr learned my addr 172.21.15.7:0/436707886 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:58:14.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.465+0000 7fdb4085e700 1 --2- 172.21.15.7:0/436707886 >> v2:172.21.15.143:3300/0 conn(0x7fdb3c072250 0x7fdb3c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:14.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.465+0000 7fdb41860700 1 -- 172.21.15.7:0/436707886 >> v2:172.21.15.143:3300/0 conn(0x7fdb3c072250 msgr2=0x7fdb3c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:58:14.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.465+0000 7fdb41860700 1 --2- 172.21.15.7:0/436707886 >> v2:172.21.15.143:3300/0 conn(0x7fdb3c072250 0x7fdb3c14e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:14.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.465+0000 7fdb41860700 1 -- 172.21.15.7:0/436707886 >> v1:172.21.15.7:6789/0 conn(0x7fdb3c071e80 legacy=0x7fdb3c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:14.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.465+0000 7fdb41860700 1 -- 172.21.15.7:0/436707886 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0c39e0 con 0x7fdb3c0c1ce0 2023-03-03T18:58:14.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.465+0000 7fdb2ffff700 1 -- 172.21.15.7:0/436707886 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803285357 0 0) 0x7fdb3c1510d0 con 0x7fdb3c071e80 2023-03-03T18:58:14.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.465+0000 7fdb41860700 1 --2- 172.21.15.7:0/436707886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c0c1ce0 0x7fdb3c14e730 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c06b520 tx=0x7fdb3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3756074204959b23 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:58:14.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.465+0000 7fdb2ffff700 1 -- 172.21.15.7:0/436707886 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3400e070 con 0x7fdb3c0c1ce0 2023-03-03T18:58:14.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.465+0000 7fdb2ffff700 1 -- 172.21.15.7:0/436707886 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3400bb20 con 0x7fdb3c0c1ce0 2023-03-03T18:58:14.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.465+0000 7fdb2ffff700 1 -- 172.21.15.7:0/436707886 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34012430 con 0x7fdb3c0c1ce0 2023-03-03T18:58:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.466+0000 7fdb432c3700 1 -- 172.21.15.7:0/436707886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c0c1ce0 msgr2=0x7fdb3c14e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.466+0000 7fdb432c3700 1 --2- 172.21.15.7:0/436707886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c0c1ce0 0x7fdb3c14e730 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c06b520 tx=0x7fdb3400b330 comp rx=0 tx=0).stop 2023-03-03T18:58:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.466+0000 7fdb432c3700 1 -- 172.21.15.7:0/436707886 shutdown_connections 2023-03-03T18:58:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.466+0000 7fdb432c3700 1 --2- 172.21.15.7:0/436707886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c0c1ce0 0x7fdb3c14e730 unknown :-1 s=CLOSED pgs=3347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.466+0000 7fdb432c3700 1 --2- 172.21.15.7:0/436707886 >> v2:172.21.15.143:3300/0 conn(0x7fdb3c072250 0x7fdb3c14e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.466+0000 7fdb432c3700 1 -- 172.21.15.7:0/436707886 >> 172.21.15.7:0/436707886 conn(0x7fdb3c0a94d0 msgr2=0x7fdb3c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:58:14.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.466+0000 7fdb432c3700 1 -- 172.21.15.7:0/436707886 shutdown_connections 2023-03-03T18:58:14.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.466+0000 7fdb432c3700 1 -- 172.21.15.7:0/436707886 wait complete. 2023-03-03T18:58:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.467+0000 7fdb432c3700 1 Processor -- start 2023-03-03T18:58:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.467+0000 7fdb432c3700 1 -- start start 2023-03-03T18:58:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb432c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c071e80 0x7fdb3c1e8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb432c3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb3c072250 0x7fdb3c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:14.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb432c3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb3c0c1ce0 0x7fdb3c145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb432c3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c145430 con 0x7fdb3c071e80 2023-03-03T18:58:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb432c3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c144420 con 0x7fdb3c072250 2023-03-03T18:58:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb432c3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3c144670 con 0x7fdb3c0c1ce0 2023-03-03T18:58:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb4105f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c071e80 0x7fdb3c1e8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb4105f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c071e80 0x7fdb3c1e8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39414/0 (socket says 172.21.15.7:39414) 2023-03-03T18:58:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb4105f700 1 -- 172.21.15.7:0/3830520504 learned_addr learned my addr 172.21.15.7:0/3830520504 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:58:14.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb41860700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb3c0c1ce0 0x7fdb3c145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:14.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb4085e700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb3c072250 0x7fdb3c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:14.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb4105f700 1 -- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb3c0c1ce0 msgr2=0x7fdb3c145c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:14.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb4105f700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb3c0c1ce0 0x7fdb3c145c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:14.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb4105f700 1 -- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb3c072250 msgr2=0x7fdb3c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:14.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb4105f700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb3c072250 0x7fdb3c0736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:14.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb4105f700 1 -- 172.21.15.7:0/3830520504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb34009a70 con 0x7fdb3c071e80 2023-03-03T18:58:14.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.468+0000 7fdb4105f700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c071e80 0x7fdb3c1e8760 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7fdb380011e0 tx=0x7fdb3800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:58:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.469+0000 7fdb2dffb700 1 -- 172.21.15.7:0/3830520504 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38004c80 con 0x7fdb3c071e80 2023-03-03T18:58:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.469+0000 7fdb2dffb700 1 -- 172.21.15.7:0/3830520504 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb38004e00 con 0x7fdb3c071e80 2023-03-03T18:58:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.469+0000 7fdb432c3700 1 -- 172.21.15.7:0/3830520504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c151f30 con 0x7fdb3c071e80 2023-03-03T18:58:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.469+0000 7fdb2dffb700 1 -- 172.21.15.7:0/3830520504 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb38007310 con 0x7fdb3c071e80 2023-03-03T18:58:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.469+0000 7fdb432c3700 1 -- 172.21.15.7:0/3830520504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c151300 con 0x7fdb3c071e80 2023-03-03T18:58:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.470+0000 7fdb432c3700 1 -- 172.21.15.7:0/3830520504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c06c9a0 con 0x7fdb3c071e80 2023-03-03T18:58:14.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.470+0000 7fdb2dffb700 1 -- 172.21.15.7:0/3830520504 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdb3801a020 con 0x7fdb3c071e80 2023-03-03T18:58:14.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.471+0000 7fdb2dffb700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb200682f0 0x7fdb2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:58:14.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.471+0000 7fdb2dffb700 1 -- 172.21.15.7:0/3830520504 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdb38085e20 con 0x7fdb3c071e80 2023-03-03T18:58:14.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.471+0000 7fdb4085e700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb200682f0 0x7fdb2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:58:14.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.472+0000 7fdb4085e700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb200682f0 0x7fdb2006a7a0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0747a0 tx=0x7fdb30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:58:14.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.475+0000 7fdb2dffb700 1 -- 172.21.15.7:0/3830520504 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdb380521c0 con 0x7fdb3c071e80 2023-03-03T18:58:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.718+0000 7fdb432c3700 1 -- 172.21.15.7:0/3830520504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb3c074bb0 con 0x7fdb3c071e80 2023-03-03T18:58:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.718+0000 7fdb2dffb700 1 -- 172.21.15.7:0/3830520504 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdb380557e0 con 0x7fdb3c071e80 2023-03-03T18:58:14.719 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:58:14.719 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:58:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 -- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb200682f0 msgr2=0x7fdb2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb200682f0 0x7fdb2006a7a0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0747a0 tx=0x7fdb30008040 comp rx=0 tx=0).stop 2023-03-03T18:58:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 -- 172.21.15.7:0/3830520504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c071e80 msgr2=0x7fdb3c1e8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:58:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c071e80 0x7fdb3c1e8760 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7fdb380011e0 tx=0x7fdb3800cae0 comp rx=0 tx=0).stop 2023-03-03T18:58:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 -- 172.21.15.7:0/3830520504 shutdown_connections 2023-03-03T18:58:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdb3c0c1ce0 0x7fdb3c145c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdb200682f0 0x7fdb2006a7a0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb3c071e80 0x7fdb3c1e8760 unknown :-1 s=CLOSED pgs=3348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 --2- 172.21.15.7:0/3830520504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb3c072250 0x7fdb3c0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:58:14.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 -- 172.21.15.7:0/3830520504 >> 172.21.15.7:0/3830520504 conn(0x7fdb3c0a94d0 msgr2=0x7fdb3c0660d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:58:14.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 -- 172.21.15.7:0/3830520504 shutdown_connections 2023-03-03T18:58:14.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:58:14.720+0000 7fdb432c3700 1 -- 172.21.15.7:0/3830520504 wait complete. 2023-03-03T18:58:14.723 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:58:14.732 INFO:tasks.mds_thrash.fs.[a]:waiting for 76 secs before thrashing 2023-03-03T18:58:43.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:58:43.782 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:59:13.465 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:59:13.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.604+0000 7fa2175aa700 1 Processor -- start 2023-03-03T18:59:13.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.604+0000 7fa2175aa700 1 -- start start 2023-03-03T18:59:13.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa2175aa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa2100aad20 0x7fa210157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:13.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa2175aa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2100a9c20 con 0x7fa2100aa030 2023-03-03T18:59:13.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa2175aa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa210154f10 con 0x7fa21014b0d0 2023-03-03T18:59:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa2175aa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2100ac3c0 con 0x7fa2100aad20 2023-03-03T18:59:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa214b45700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa2100aad20 0x7fa210157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa215b47700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa2100aa030 0x7fa210158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33438/0 (socket says 172.21.15.7:33438) 2023-03-03T18:59:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa215b47700 1 -- 172.21.15.7:0/442569738 learned_addr learned my addr 172.21.15.7:0/442569738 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:59:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa214b45700 1 -- 172.21.15.7:0/442569738 >> v1:172.21.15.143:6789/0 conn(0x7fa21014b0d0 legacy=0x7fa210071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:59:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa214b45700 1 -- 172.21.15.7:0/442569738 >> v1:172.21.15.7:6789/0 conn(0x7fa2100aa030 legacy=0x7fa210158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa207fff700 1 -- 172.21.15.7:0/442569738 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026381389 0 0) 0x7fa2100a9c20 con 0x7fa2100aa030 2023-03-03T18:59:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa214b45700 1 -- 172.21.15.7:0/442569738 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21006fc70 con 0x7fa2100aad20 2023-03-03T18:59:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa214b45700 1 --2- 172.21.15.7:0/442569738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100aad20 0x7fa210157df0 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7fa200009840 tx=0x7fa200009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1217ea1f78aa34 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:59:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa207fff700 1 -- 172.21.15.7:0/442569738 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200004230 con 0x7fa2100aad20 2023-03-03T18:59:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa207fff700 1 -- 172.21.15.7:0/442569738 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa200004390 con 0x7fa2100aad20 2023-03-03T18:59:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.605+0000 7fa207fff700 1 -- 172.21.15.7:0/442569738 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2000036a0 con 0x7fa2100aad20 2023-03-03T18:59:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.606+0000 7fa2175aa700 1 -- 172.21.15.7:0/442569738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100aad20 msgr2=0x7fa210157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.606+0000 7fa2175aa700 1 --2- 172.21.15.7:0/442569738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100aad20 0x7fa210157df0 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7fa200009840 tx=0x7fa200009b50 comp rx=0 tx=0).stop 2023-03-03T18:59:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.606+0000 7fa2175aa700 1 -- 172.21.15.7:0/442569738 shutdown_connections 2023-03-03T18:59:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.606+0000 7fa2175aa700 1 --2- 172.21.15.7:0/442569738 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2100aad20 0x7fa210157df0 unknown :-1 s=CLOSED pgs=3350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.606+0000 7fa2175aa700 1 -- 172.21.15.7:0/442569738 >> 172.21.15.7:0/442569738 conn(0x7fa2100c8900 msgr2=0x7fa2100c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:59:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.606+0000 7fa2175aa700 1 -- 172.21.15.7:0/442569738 shutdown_connections 2023-03-03T18:59:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.606+0000 7fa2175aa700 1 -- 172.21.15.7:0/442569738 wait complete. 2023-03-03T18:59:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.607+0000 7fa2175aa700 1 Processor -- start 2023-03-03T18:59:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.607+0000 7fa2175aa700 1 -- start start 2023-03-03T18:59:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa2175aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2100aa030 0x7fa2101df540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa2175aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2100aad20 0x7fa2101dfa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa2175aa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa21014b0d0 0x7fa2100c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa2175aa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2100c8260 con 0x7fa21014b0d0 2023-03-03T18:59:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa2175aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2101e9a10 con 0x7fa2100aa030 2023-03-03T18:59:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa2175aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2101e0400 con 0x7fa2100aad20 2023-03-03T18:59:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa215b47700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa21014b0d0 0x7fa2100c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa215b47700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa21014b0d0 0x7fa2100c6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39464/0 (socket says 172.21.15.7:39464) 2023-03-03T18:59:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa215b47700 1 -- 172.21.15.7:0/4169617627 learned_addr learned my addr 172.21.15.7:0/4169617627 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:59:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa214b45700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2100aad20 0x7fa2101dfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa215b47700 1 -- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2100aad20 msgr2=0x7fa2101dfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa215b47700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2100aad20 0x7fa2101dfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa215346700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2100aa030 0x7fa2101df540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa215b47700 1 -- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2100aa030 msgr2=0x7fa2101df540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa215b47700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2100aa030 0x7fa2101df540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.608+0000 7fa215b47700 1 -- 172.21.15.7:0/4169617627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2000094a0 con 0x7fa21014b0d0 2023-03-03T18:59:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.609+0000 7fa215b47700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa21014b0d0 0x7fa2100c6310 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7fa20800cb60 tx=0x7fa20800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:59:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.609+0000 7fa205ffb700 1 -- 172.21.15.7:0/4169617627 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20800ab50 con 0x7fa21014b0d0 2023-03-03T18:59:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.609+0000 7fa205ffb700 1 -- 172.21.15.7:0/4169617627 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20800acb0 con 0x7fa21014b0d0 2023-03-03T18:59:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.609+0000 7fa2175aa700 1 -- 172.21.15.7:0/4169617627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21006f6b0 con 0x7fa21014b0d0 2023-03-03T18:59:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.609+0000 7fa205ffb700 1 -- 172.21.15.7:0/4169617627 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa208007310 con 0x7fa21014b0d0 2023-03-03T18:59:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.609+0000 7fa2175aa700 1 -- 172.21.15.7:0/4169617627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa210155160 con 0x7fa21014b0d0 2023-03-03T18:59:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.610+0000 7fa2175aa700 1 -- 172.21.15.7:0/4169617627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21006f990 con 0x7fa21014b0d0 2023-03-03T18:59:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.610+0000 7fa205ffb700 1 -- 172.21.15.7:0/4169617627 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa20801a030 con 0x7fa21014b0d0 2023-03-03T18:59:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.611+0000 7fa205ffb700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 0x7fa1f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.611+0000 7fa205ffb700 1 -- 172.21.15.7:0/4169617627 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa208085e00 con 0x7fa21014b0d0 2023-03-03T18:59:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.611+0000 7fa215346700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 0x7fa1f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.612+0000 7fa215346700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 0x7fa1f406a7a0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fa20c003e60 tx=0x7fa20c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:59:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.615+0000 7fa205ffb700 1 -- 172.21.15.7:0/4169617627 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa2080521a0 con 0x7fa21014b0d0 2023-03-03T18:59:13.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:59:13.854 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:59:13.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.856+0000 7fa2175aa700 1 -- 172.21.15.7:0/4169617627 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa210074bb0 con 0x7fa21014b0d0 2023-03-03T18:59:13.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.856+0000 7fa205ffb700 1 -- 172.21.15.7:0/4169617627 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa208018090 con 0x7fa21014b0d0 2023-03-03T18:59:13.858 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:59:13.858 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:59:13.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.858+0000 7fa2175aa700 1 -- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 msgr2=0x7fa1f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:13.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.858+0000 7fa2175aa700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 0x7fa1f406a7a0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fa20c003e60 tx=0x7fa20c009040 comp rx=0 tx=0).stop 2023-03-03T18:59:13.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.858+0000 7fa2175aa700 1 -- 172.21.15.7:0/4169617627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa21014b0d0 msgr2=0x7fa2100c6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:13.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.858+0000 7fa2175aa700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa21014b0d0 0x7fa2100c6310 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7fa20800cb60 tx=0x7fa20800cf20 comp rx=0 tx=0).stop 2023-03-03T18:59:13.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.858+0000 7fa2175aa700 1 -- 172.21.15.7:0/4169617627 shutdown_connections 2023-03-03T18:59:13.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.858+0000 7fa2175aa700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1f40682f0 0x7fa1f406a7a0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:13.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.858+0000 7fa2175aa700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa21014b0d0 0x7fa2100c6310 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:13.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.858+0000 7fa2175aa700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2100aa030 0x7fa2101df540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:13.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.858+0000 7fa2175aa700 1 --2- 172.21.15.7:0/4169617627 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2100aad20 0x7fa2101dfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:13.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.858+0000 7fa2175aa700 1 -- 172.21.15.7:0/4169617627 >> 172.21.15.7:0/4169617627 conn(0x7fa2100c8900 msgr2=0x7fa2100aca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:59:13.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.859+0000 7fa2175aa700 1 -- 172.21.15.7:0/4169617627 shutdown_connections 2023-03-03T18:59:13.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:13.859+0000 7fa2175aa700 1 -- 172.21.15.7:0/4169617627 wait complete. 2023-03-03T18:59:13.861 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:59:13.873 INFO:tasks.mds_thrash.fs.[b]:waiting for 93 secs before thrashing 2023-03-03T18:59:30.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:59:30.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.884+0000 7fdf0e72b700 1 Processor -- start 2023-03-03T18:59:30.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.884+0000 7fdf0e72b700 1 -- start start 2023-03-03T18:59:30.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf0e72b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf0806c480 0x7fdf081581f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:30.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf0e72b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0806cb70 con 0x7fdf080c2950 2023-03-03T18:59:30.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf0e72b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0806ce50 con 0x7fdf08151b20 2023-03-03T18:59:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf0e72b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf0806c2f0 con 0x7fdf0806c480 2023-03-03T18:59:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf0ccc8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdf0806c480 0x7fdf081581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf077fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdf080c2950 0x7fdf08157ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33464/0 (socket says 172.21.15.7:33464) 2023-03-03T18:59:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf077fe700 1 -- 172.21.15.7:0/3839013570 learned_addr learned my addr 172.21.15.7:0/3839013570 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:59:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf0ccc8700 1 -- 172.21.15.7:0/3839013570 >> v1:172.21.15.143:6789/0 conn(0x7fdf08151b20 legacy=0x7fdf080736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T18:59:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf0ccc8700 1 -- 172.21.15.7:0/3839013570 >> v1:172.21.15.7:6789/0 conn(0x7fdf080c2950 legacy=0x7fdf08157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf0ccc8700 1 -- 172.21.15.7:0/3839013570 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0806d130 con 0x7fdf0806c480 2023-03-03T18:59:30.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.885+0000 7fdf06ffd700 1 -- 172.21.15.7:0/3839013570 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308514844 0 0) 0x7fdf0806cb70 con 0x7fdf080c2950 2023-03-03T18:59:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf0ccc8700 1 --2- 172.21.15.7:0/3839013570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf0806c480 0x7fdf081581f0 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009d90 tx=0x7fdf0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7be29cecc608109 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:59:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf06ffd700 1 -- 172.21.15.7:0/3839013570 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0000e070 con 0x7fdf0806c480 2023-03-03T18:59:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf06ffd700 1 -- 172.21.15.7:0/3839013570 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0000bb90 con 0x7fdf0806c480 2023-03-03T18:59:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf06ffd700 1 -- 172.21.15.7:0/3839013570 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00003a20 con 0x7fdf0806c480 2023-03-03T18:59:30.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3839013570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf0806c480 msgr2=0x7fdf081581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:30.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf0e72b700 1 --2- 172.21.15.7:0/3839013570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf0806c480 0x7fdf081581f0 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009d90 tx=0x7fdf0000b330 comp rx=0 tx=0).stop 2023-03-03T18:59:30.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3839013570 shutdown_connections 2023-03-03T18:59:30.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf0e72b700 1 --2- 172.21.15.7:0/3839013570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf0806c480 0x7fdf081581f0 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:30.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3839013570 >> 172.21.15.7:0/3839013570 conn(0x7fdf080a94d0 msgr2=0x7fdf080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:59:30.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3839013570 shutdown_connections 2023-03-03T18:59:30.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.886+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3839013570 wait complete. 2023-03-03T18:59:30.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.887+0000 7fdf0e72b700 1 Processor -- start 2023-03-03T18:59:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.888+0000 7fdf0e72b700 1 -- start start 2023-03-03T18:59:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.888+0000 7fdf0e72b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf0806c480 0x7fdf0814cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.888+0000 7fdf0e72b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf080c2950 0x7fdf081562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:30.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.888+0000 7fdf0e72b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08151b20 0x7fdf081533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.888+0000 7fdf0e72b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf081578d0 con 0x7fdf08151b20 2023-03-03T18:59:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.888+0000 7fdf0e72b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0814d1c0 con 0x7fdf0806c480 2023-03-03T18:59:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.888+0000 7fdf0e72b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf081575f0 con 0x7fdf080c2950 2023-03-03T18:59:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.888+0000 7fdf0ccc8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08151b20 0x7fdf081533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.888+0000 7fdf0ccc8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08151b20 0x7fdf081533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39494/0 (socket says 172.21.15.7:39494) 2023-03-03T18:59:30.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.888+0000 7fdf0ccc8700 1 -- 172.21.15.7:0/3014403211 learned_addr learned my addr 172.21.15.7:0/3014403211 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:59:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf07fff700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf0806c480 0x7fdf0814cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf077fe700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf080c2950 0x7fdf081562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf0ccc8700 1 -- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf080c2950 msgr2=0x7fdf081562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf0ccc8700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf080c2950 0x7fdf081562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf0ccc8700 1 -- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf0806c480 msgr2=0x7fdf0814cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf0ccc8700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf0806c480 0x7fdf0814cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf0ccc8700 1 -- 172.21.15.7:0/3014403211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf00009a70 con 0x7fdf08151b20 2023-03-03T18:59:30.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf07fff700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf0806c480 0x7fdf0814cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:59:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf077fe700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf080c2950 0x7fdf081562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T18:59:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf0ccc8700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08151b20 0x7fdf081533e0 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7fdf0000b730 tx=0x7fdf00004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:59:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf04ff9700 1 -- 172.21.15.7:0/3014403211 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00012840 con 0x7fdf08151b20 2023-03-03T18:59:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf04ff9700 1 -- 172.21.15.7:0/3014403211 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf000129a0 con 0x7fdf08151b20 2023-03-03T18:59:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3014403211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf08158730 con 0x7fdf08151b20 2023-03-03T18:59:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf04ff9700 1 -- 172.21.15.7:0/3014403211 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf00012d50 con 0x7fdf08151b20 2023-03-03T18:59:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.889+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3014403211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0806d410 con 0x7fdf08151b20 2023-03-03T18:59:30.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.890+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3014403211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf08151ef0 con 0x7fdf08151b20 2023-03-03T18:59:30.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.891+0000 7fdf04ff9700 1 -- 172.21.15.7:0/3014403211 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdf0001d420 con 0x7fdf08151b20 2023-03-03T18:59:30.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.891+0000 7fdf04ff9700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee80682f0 0x7fdee806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:30.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.891+0000 7fdf04ff9700 1 -- 172.21.15.7:0/3014403211 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdf00088940 con 0x7fdf08151b20 2023-03-03T18:59:30.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.891+0000 7fdf07fff700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee80682f0 0x7fdee806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:30.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.892+0000 7fdf07fff700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee80682f0 0x7fdee806a7a0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7fdef0006fd0 tx=0x7fdef0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:59:30.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:30.895+0000 7fdf04ff9700 1 -- 172.21.15.7:0/3014403211 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdf00054c80 con 0x7fdf08151b20 2023-03-03T18:59:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.132+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3014403211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf08074bb0 con 0x7fdf08151b20 2023-03-03T18:59:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.132+0000 7fdf04ff9700 1 -- 172.21.15.7:0/3014403211 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdf000582a0 con 0x7fdf08151b20 2023-03-03T18:59:31.133 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:59:31.133 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:59:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.134+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee80682f0 msgr2=0x7fdee806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:31.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.134+0000 7fdf0e72b700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee80682f0 0x7fdee806a7a0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7fdef0006fd0 tx=0x7fdef0008040 comp rx=0 tx=0).stop 2023-03-03T18:59:31.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.134+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3014403211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08151b20 msgr2=0x7fdf081533e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:31.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.134+0000 7fdf0e72b700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08151b20 0x7fdf081533e0 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7fdf0000b730 tx=0x7fdf00004ad0 comp rx=0 tx=0).stop 2023-03-03T18:59:31.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.134+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3014403211 shutdown_connections 2023-03-03T18:59:31.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.134+0000 7fdf0e72b700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdee80682f0 0x7fdee806a7a0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:31.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.134+0000 7fdf0e72b700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf08151b20 0x7fdf081533e0 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.134+0000 7fdf0e72b700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdf0806c480 0x7fdf0814cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.134+0000 7fdf0e72b700 1 --2- 172.21.15.7:0/3014403211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdf080c2950 0x7fdf081562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.134+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3014403211 >> 172.21.15.7:0/3014403211 conn(0x7fdf080a94d0 msgr2=0x7fdf080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:59:31.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.135+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3014403211 shutdown_connections 2023-03-03T18:59:31.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:31.135+0000 7fdf0e72b700 1 -- 172.21.15.7:0/3014403211 wait complete. 2023-03-03T18:59:31.137 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:59:31.146 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-03-03T18:59:43.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:59:43.909 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T18:59:57.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T18:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.284+0000 7fd2e6bd8700 1 Processor -- start 2023-03-03T18:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.284+0000 7fd2e6bd8700 1 -- start start 2023-03-03T18:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.285+0000 7fd2e6bd8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd2e006c480 0x7fd2e01581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.285+0000 7fd2e6bd8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e006d430 con 0x7fd2e00c2930 2023-03-03T18:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.286+0000 7fd2e6bd8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e01586e0 con 0x7fd2e0151d40 2023-03-03T18:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.286+0000 7fd2e6bd8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e006c2a0 con 0x7fd2e006c480 2023-03-03T18:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.286+0000 7fd2e5175700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd2e006c480 0x7fd2e01581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.286+0000 7fd2dffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd2e00c2930 0x7fd2e0157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33512/0 (socket says 172.21.15.7:33512) 2023-03-03T18:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.286+0000 7fd2dffff700 1 -- 172.21.15.7:0/1032250258 learned_addr learned my addr 172.21.15.7:0/1032250258 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T18:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.286+0000 7fd2e5175700 1 -- 172.21.15.7:0/1032250258 >> v1:172.21.15.143:6789/0 conn(0x7fd2e0151d40 legacy=0x7fd2e00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.286+0000 7fd2e5175700 1 -- 172.21.15.7:0/1032250258 >> v1:172.21.15.7:6789/0 conn(0x7fd2e00c2930 legacy=0x7fd2e0157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.286+0000 7fd2e5175700 1 -- 172.21.15.7:0/1032250258 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e006cb00 con 0x7fd2e006c480 2023-03-03T18:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.286+0000 7fd2e5175700 1 --2- 172.21.15.7:0/1032250258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e006c480 0x7fd2e01581a0 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009dc0 tx=0x7fd2d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9303456d2b7af94 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.287+0000 7fd2df7fe700 1 -- 172.21.15.7:0/1032250258 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d800e070 con 0x7fd2e006c480 2023-03-03T18:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.287+0000 7fd2df7fe700 1 -- 172.21.15.7:0/1032250258 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d800bb90 con 0x7fd2e006c480 2023-03-03T18:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.287+0000 7fd2df7fe700 1 -- 172.21.15.7:0/1032250258 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8012430 con 0x7fd2e006c480 2023-03-03T18:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.288+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/1032250258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e006c480 msgr2=0x7fd2e01581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.288+0000 7fd2e6bd8700 1 --2- 172.21.15.7:0/1032250258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e006c480 0x7fd2e01581a0 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009dc0 tx=0x7fd2d800b330 comp rx=0 tx=0).stop 2023-03-03T18:59:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.288+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/1032250258 shutdown_connections 2023-03-03T18:59:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.288+0000 7fd2e6bd8700 1 --2- 172.21.15.7:0/1032250258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e006c480 0x7fd2e01581a0 unknown :-1 s=CLOSED pgs=3356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.288+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/1032250258 >> 172.21.15.7:0/1032250258 conn(0x7fd2e00a94d0 msgr2=0x7fd2e00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:59:57.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.288+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/1032250258 shutdown_connections 2023-03-03T18:59:57.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.288+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/1032250258 wait complete. 2023-03-03T18:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.289+0000 7fd2e6bd8700 1 Processor -- start 2023-03-03T18:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.289+0000 7fd2e6bd8700 1 -- start start 2023-03-03T18:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e6bd8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e006c480 0x7fd2e0150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e6bd8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e00c2930 0x7fd2e00ab740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e6bd8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e0151d40 0x7fd2e00abc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e6bd8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e00bec00 con 0x7fd2e0151d40 2023-03-03T18:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e6bd8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e00be430 con 0x7fd2e006c480 2023-03-03T18:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e6bd8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e00ab2d0 con 0x7fd2e00c2930 2023-03-03T18:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e5175700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e0151d40 0x7fd2e00abc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e5175700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e0151d40 0x7fd2e00abc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39540/0 (socket says 172.21.15.7:39540) 2023-03-03T18:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e5175700 1 -- 172.21.15.7:0/437059452 learned_addr learned my addr 172.21.15.7:0/437059452 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T18:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e4974700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e006c480 0x7fd2e0150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2dffff700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e00c2930 0x7fd2e00ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e5175700 1 -- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e00c2930 msgr2=0x7fd2e00ab740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e5175700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e00c2930 0x7fd2e00ab740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:57.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e5175700 1 -- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e006c480 msgr2=0x7fd2e0150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:57.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e5175700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e006c480 0x7fd2e0150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:57.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e5175700 1 -- 172.21.15.7:0/437059452 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d8009a70 con 0x7fd2e0151d40 2023-03-03T18:59:57.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2dffff700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e00c2930 0x7fd2e00ab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T18:59:57.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.290+0000 7fd2e5175700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e0151d40 0x7fd2e00abc80 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7fd2d800b370 tx=0x7fd2d8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:59:57.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.291+0000 7fd2dd7fa700 1 -- 172.21.15.7:0/437059452 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80048b0 con 0x7fd2e0151d40 2023-03-03T18:59:57.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.291+0000 7fd2dd7fa700 1 -- 172.21.15.7:0/437059452 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d8020070 con 0x7fd2e0151d40 2023-03-03T18:59:57.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.291+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/437059452 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e006d0c0 con 0x7fd2e0151d40 2023-03-03T18:59:57.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.291+0000 7fd2dd7fa700 1 -- 172.21.15.7:0/437059452 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80276c0 con 0x7fd2e0151d40 2023-03-03T18:59:57.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.291+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/437059452 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e006cde0 con 0x7fd2e0151d40 2023-03-03T18:59:57.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.292+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/437059452 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e0073e00 con 0x7fd2e0151d40 2023-03-03T18:59:57.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.292+0000 7fd2dd7fa700 1 -- 172.21.15.7:0/437059452 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd2d8012430 con 0x7fd2e0151d40 2023-03-03T18:59:57.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.293+0000 7fd2dd7fa700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2c40682f0 0x7fd2c406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T18:59:57.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.293+0000 7fd2dd7fa700 1 -- 172.21.15.7:0/437059452 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd2d8022070 con 0x7fd2e0151d40 2023-03-03T18:59:57.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.293+0000 7fd2e4974700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2c40682f0 0x7fd2c406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T18:59:57.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.293+0000 7fd2e4974700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2c40682f0 0x7fd2c406a7a0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00bdde0 tx=0x7fd2d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T18:59:57.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.297+0000 7fd2dd7fa700 1 -- 172.21.15.7:0/437059452 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd2d80549c0 con 0x7fd2e0151d40 2023-03-03T18:59:57.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.536+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/437059452 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2e0074bb0 con 0x7fd2e0151d40 2023-03-03T18:59:57.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.537+0000 7fd2dd7fa700 1 -- 172.21.15.7:0/437059452 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd2d8057fe0 con 0x7fd2e0151d40 2023-03-03T18:59:57.537 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T18:59:57.538 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T18:59:57.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2c40682f0 msgr2=0x7fd2c406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:57.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2c40682f0 0x7fd2c406a7a0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00bdde0 tx=0x7fd2d4009040 comp rx=0 tx=0).stop 2023-03-03T18:59:57.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/437059452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e0151d40 msgr2=0x7fd2e00abc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T18:59:57.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e0151d40 0x7fd2e00abc80 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7fd2d800b370 tx=0x7fd2d8004ad0 comp rx=0 tx=0).stop 2023-03-03T18:59:57.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/437059452 shutdown_connections 2023-03-03T18:59:57.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd2c40682f0 0x7fd2c406a7a0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:57.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2e0151d40 0x7fd2e00abc80 unknown :-1 s=CLOSED pgs=3357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:57.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2e006c480 0x7fd2e0150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 --2- 172.21.15.7:0/437059452 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2e00c2930 0x7fd2e00ab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T18:59:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/437059452 >> 172.21.15.7:0/437059452 conn(0x7fd2e00a94d0 msgr2=0x7fd2e0071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T18:59:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/437059452 shutdown_connections 2023-03-03T18:59:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T18:59:57.539+0000 7fd2e6bd8700 1 -- 172.21.15.7:0/437059452 wait complete. 2023-03-03T18:59:57.541 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T18:59:57.551 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before thrashing 2023-03-03T19:00:00.001 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T19:00:00.000+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T19:00:03.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:00:03.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.692+0000 7fc4e3cdc700 1 Processor -- start 2023-03-03T19:00:03.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.693+0000 7fc4e3cdc700 1 -- start start 2023-03-03T19:00:03.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.693+0000 7fc4e3cdc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc4dc155390 0x7fc4dc0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:03.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.693+0000 7fc4e3cdc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc154910 con 0x7fc4dc151770 2023-03-03T19:00:03.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.693+0000 7fc4e3cdc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4dc1522e0 con 0x7fc4dc06e020 2023-03-03T19:00:03.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.693+0000 7fc4e3cdc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4dc070b90 con 0x7fc4dc155390 2023-03-03T19:00:03.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e1a78700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc4dc155390 0x7fc4dc0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:03.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e2279700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc4dc151770 0x7fc4dc151bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33528/0 (socket says 172.21.15.7:33528) 2023-03-03T19:00:03.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e1a78700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc4dc155390 0x7fc4dc0c3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39544/0 (socket says 172.21.15.7:39544) 2023-03-03T19:00:03.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e2279700 1 -- 172.21.15.7:0/2787191629 learned_addr learned my addr 172.21.15.7:0/2787191629 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:00:03.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e1a78700 1 -- 172.21.15.7:0/2787191629 >> v1:172.21.15.143:6789/0 conn(0x7fc4dc06e020 legacy=0x7fc4dc071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:00:03.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e1a78700 1 -- 172.21.15.7:0/2787191629 >> v1:172.21.15.7:6789/0 conn(0x7fc4dc151770 legacy=0x7fc4dc151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:03.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e1a78700 1 -- 172.21.15.7:0/2787191629 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc0702a0 con 0x7fc4dc155390 2023-03-03T19:00:03.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e0a76700 1 -- 172.21.15.7:0/2787191629 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999861333 0 0) 0x7fc4dc154910 con 0x7fc4dc151770 2023-03-03T19:00:03.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e1a78700 1 --2- 172.21.15.7:0/2787191629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc155390 0x7fc4dc0c3e30 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009840 tx=0x7fc4cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7600c6aa82227d0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:00:03.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e0a76700 1 -- 172.21.15.7:0/2787191629 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc004230 con 0x7fc4dc155390 2023-03-03T19:00:03.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e0a76700 1 -- 172.21.15.7:0/2787191629 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4cc004390 con 0x7fc4dc155390 2023-03-03T19:00:03.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.694+0000 7fc4e0a76700 1 -- 172.21.15.7:0/2787191629 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc0036a0 con 0x7fc4dc155390 2023-03-03T19:00:03.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.695+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/2787191629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc155390 msgr2=0x7fc4dc0c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:03.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.695+0000 7fc4e3cdc700 1 --2- 172.21.15.7:0/2787191629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc155390 0x7fc4dc0c3e30 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009840 tx=0x7fc4cc009b50 comp rx=0 tx=0).stop 2023-03-03T19:00:03.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.695+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/2787191629 shutdown_connections 2023-03-03T19:00:03.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.695+0000 7fc4e3cdc700 1 --2- 172.21.15.7:0/2787191629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc155390 0x7fc4dc0c3e30 unknown :-1 s=CLOSED pgs=3359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:03.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.695+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/2787191629 >> 172.21.15.7:0/2787191629 conn(0x7fc4dc06c110 msgr2=0x7fc4dc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:00:03.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.695+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/2787191629 shutdown_connections 2023-03-03T19:00:03.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.695+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/2787191629 wait complete. 2023-03-03T19:00:03.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.696+0000 7fc4e3cdc700 1 Processor -- start 2023-03-03T19:00:03.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.696+0000 7fc4e3cdc700 1 -- start start 2023-03-03T19:00:03.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e3cdc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc06e020 0x7fc4dc0c0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:03.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e3cdc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc151770 0x7fc4dc0c0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:03.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e3cdc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc155390 0x7fc4dc1e92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:03.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e3cdc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0bcc40 con 0x7fc4dc06e020 2023-03-03T19:00:03.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e3cdc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc151420 con 0x7fc4dc151770 2023-03-03T19:00:03.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e3cdc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc150e60 con 0x7fc4dc155390 2023-03-03T19:00:03.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e1a78700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc06e020 0x7fc4dc0c0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:03.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e1a78700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc06e020 0x7fc4dc0c0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39550/0 (socket says 172.21.15.7:39550) 2023-03-03T19:00:03.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e1a78700 1 -- 172.21.15.7:0/4139160541 learned_addr learned my addr 172.21.15.7:0/4139160541 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:00:03.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e2279700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc155390 0x7fc4dc1e92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:03.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e1277700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc151770 0x7fc4dc0c0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:03.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e1a78700 1 -- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc155390 msgr2=0x7fc4dc1e92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:03.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e1a78700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc155390 0x7fc4dc1e92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:03.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e1a78700 1 -- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc151770 msgr2=0x7fc4dc0c0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:03.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e1a78700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc151770 0x7fc4dc0c0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e1a78700 1 -- 172.21.15.7:0/4139160541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc0094a0 con 0x7fc4dc06e020 2023-03-03T19:00:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.697+0000 7fc4e1277700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc151770 0x7fc4dc0c0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:00:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.698+0000 7fc4e1a78700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc06e020 0x7fc4dc0c0530 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0c4a90 tx=0x7fc4cc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:00:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.698+0000 7fc4d27fc700 1 -- 172.21.15.7:0/4139160541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc011630 con 0x7fc4dc06e020 2023-03-03T19:00:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.698+0000 7fc4d27fc700 1 -- 172.21.15.7:0/4139160541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4cc011790 con 0x7fc4dc06e020 2023-03-03T19:00:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.698+0000 7fc4d27fc700 1 -- 172.21.15.7:0/4139160541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc011b40 con 0x7fc4dc06e020 2023-03-03T19:00:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.698+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/4139160541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc070820 con 0x7fc4dc06e020 2023-03-03T19:00:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.698+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/4139160541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc070540 con 0x7fc4dc06e020 2023-03-03T19:00:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.699+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/4139160541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc155760 con 0x7fc4dc06e020 2023-03-03T19:00:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.699+0000 7fc4d27fc700 1 -- 172.21.15.7:0/4139160541 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc4cc017020 con 0x7fc4dc06e020 2023-03-03T19:00:03.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.700+0000 7fc4d27fc700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4c00682f0 0x7fc4c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:03.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.700+0000 7fc4d27fc700 1 -- 172.21.15.7:0/4139160541 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc4cc0877c0 con 0x7fc4dc06e020 2023-03-03T19:00:03.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.700+0000 7fc4e1277700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4c00682f0 0x7fc4c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:03.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.700+0000 7fc4e1277700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4c00682f0 0x7fc4c006a7a0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc14e9e0 tx=0x7fc4d8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:00:03.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.704+0000 7fc4d27fc700 1 -- 172.21.15.7:0/4139160541 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc4cc053b60 con 0x7fc4dc06e020 2023-03-03T19:00:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.940+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/4139160541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4dc1eba70 con 0x7fc4dc06e020 2023-03-03T19:00:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.940+0000 7fc4d27fc700 1 -- 172.21.15.7:0/4139160541 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc4cc057180 con 0x7fc4dc06e020 2023-03-03T19:00:03.941 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:00:03.941 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:00:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4c00682f0 msgr2=0x7fc4c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4c00682f0 0x7fc4c006a7a0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc14e9e0 tx=0x7fc4d8009040 comp rx=0 tx=0).stop 2023-03-03T19:00:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/4139160541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc06e020 msgr2=0x7fc4dc0c0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc06e020 0x7fc4dc0c0530 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0c4a90 tx=0x7fc4cc0047c0 comp rx=0 tx=0).stop 2023-03-03T19:00:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/4139160541 shutdown_connections 2023-03-03T19:00:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc4dc155390 0x7fc4dc1e92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc4c00682f0 0x7fc4c006a7a0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4dc06e020 0x7fc4dc0c0530 unknown :-1 s=CLOSED pgs=3360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 --2- 172.21.15.7:0/4139160541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4dc151770 0x7fc4dc0c0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/4139160541 >> 172.21.15.7:0/4139160541 conn(0x7fc4dc06c110 msgr2=0x7fc4dc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:00:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/4139160541 shutdown_connections 2023-03-03T19:00:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:03.943+0000 7fc4e3cdc700 1 -- 172.21.15.7:0/4139160541 wait complete. 2023-03-03T19:00:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:00:03.954 INFO:tasks.mds_thrash.fs.[a]:waiting for 114 secs before thrashing 2023-03-03T19:00:13.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:00:13.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:00:44.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:00:44.058 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:00:46.874 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:00:47.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.013+0000 7ff91a16c700 1 Processor -- start 2023-03-03T19:00:47.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.013+0000 7ff91a16c700 1 -- start start 2023-03-03T19:00:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff91a16c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff9140c5cb0 0x7ff91406a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff91a16c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9141558f0 con 0x7ff91406a2b0 2023-03-03T19:00:47.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff91a16c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9140a6180 con 0x7ff914156490 2023-03-03T19:00:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff91a16c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9140c3290 con 0x7ff9140c5cb0 2023-03-03T19:00:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9137fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff91406a2b0 0x7ff914155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33560/0 (socket says 172.21.15.7:33560) 2023-03-03T19:00:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9137fe700 1 -- 172.21.15.7:0/2171111024 learned_addr learned my addr 172.21.15.7:0/2171111024 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:00:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff912ffd700 1 --2- 172.21.15.7:0/2171111024 >> v2:172.21.15.143:3300/0 conn(0x7ff9140c5cb0 0x7ff91406a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938082709 0 0) 0x7ff9141558f0 con 0x7ff91406a2b0 2023-03-03T19:00:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8fc0035b0 con 0x7ff91406a2b0 2023-03-03T19:00:47.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691420905 0 0) 0x7ff9140a6180 con 0x7ff914156490 2023-03-03T19:00:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9141558f0 con 0x7ff914156490 2023-03-03T19:00:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3984111283 0 0) 0x7ff8fc0035b0 con 0x7ff91406a2b0 2023-03-03T19:00:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 >> v2:172.21.15.143:3300/0 conn(0x7ff9140c5cb0 msgr2=0x7ff91406a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:00:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 --2- 172.21.15.7:0/2171111024 >> v2:172.21.15.143:3300/0 conn(0x7ff9140c5cb0 0x7ff91406a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 >> v1:172.21.15.143:6789/0 conn(0x7ff914156490 legacy=0x7ff91415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9140a5ea0 con 0x7ff91406a2b0 2023-03-03T19:00:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff904001670 con 0x7ff91406a2b0 2023-03-03T19:00:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 >> v1:172.21.15.7:6789/0 conn(0x7ff91406a2b0 legacy=0x7ff914155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc003c00 0x7ff8fc003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8fc004850 0x7ff8fc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8fc005250 0x7ff8fc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc00c130 con 0x7ff8fc005250 2023-03-03T19:00:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.014+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc00c2a0 con 0x7ff8fc004850 2023-03-03T19:00:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc00c460 con 0x7ff8fc003c00 2023-03-03T19:00:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff913fff700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8fc005250 0x7ff8fc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:47.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff9137fe700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc003c00 0x7ff8fc003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:47.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff912ffd700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8fc004850 0x7ff8fc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:47.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff913fff700 1 -- 172.21.15.7:0/2171111024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc003c00 msgr2=0x7ff8fc003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:00:47.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff913fff700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc003c00 0x7ff8fc003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:47.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff913fff700 1 -- 172.21.15.7:0/2171111024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8fc004850 msgr2=0x7ff8fc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:00:47.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff913fff700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8fc004850 0x7ff8fc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:47.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff913fff700 1 -- 172.21.15.7:0/2171111024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0035b0 con 0x7ff8fc005250 2023-03-03T19:00:47.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff912ffd700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8fc004850 0x7ff8fc004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:00:47.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff913fff700 1 -- 172.21.15.7:0/2171111024 reap_dead start 2023-03-03T19:00:47.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff913fff700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8fc005250 0x7ff8fc00ba10 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7ff90c00c320 tx=0x7ff8fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc97e0977bf1484 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:00:47.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90c001f20 con 0x7ff8fc005250 2023-03-03T19:00:47.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90c00ce80 con 0x7ff8fc005250 2023-03-03T19:00:47.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.015+0000 7ff9127fc700 1 -- 172.21.15.7:0/2171111024 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90c003c40 con 0x7ff8fc005250 2023-03-03T19:00:47.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.016+0000 7ff91a16c700 1 -- 172.21.15.7:0/2171111024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8fc005250 msgr2=0x7ff8fc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:47.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.016+0000 7ff91a16c700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8fc005250 0x7ff8fc00ba10 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7ff90c00c320 tx=0x7ff8fc005880 comp rx=0 tx=0).stop 2023-03-03T19:00:47.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.016+0000 7ff91a16c700 1 -- 172.21.15.7:0/2171111024 shutdown_connections 2023-03-03T19:00:47.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.016+0000 7ff91a16c700 1 --2- 172.21.15.7:0/2171111024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8fc005250 0x7ff8fc00ba10 unknown :-1 s=CLOSED pgs=3362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:47.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.016+0000 7ff91a16c700 1 -- 172.21.15.7:0/2171111024 >> 172.21.15.7:0/2171111024 conn(0x7ff9140a9430 msgr2=0x7ff9140c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:00:47.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.016+0000 7ff91a16c700 1 -- 172.21.15.7:0/2171111024 shutdown_connections 2023-03-03T19:00:47.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.016+0000 7ff91a16c700 1 -- 172.21.15.7:0/2171111024 wait complete. 2023-03-03T19:00:47.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.017+0000 7ff91a16c700 1 Processor -- start 2023-03-03T19:00:47.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.017+0000 7ff91a16c700 1 -- start start 2023-03-03T19:00:47.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff91a16c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc005250 0x7ff914071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:47.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff91a16c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff914149400 0x7ff9140a74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:47.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff91a16c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9141ed940 0x7ff914148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:47.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff91a16c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9140704e0 con 0x7ff9141ed940 2023-03-03T19:00:47.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff91a16c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9140aa400 con 0x7ff914149400 2023-03-03T19:00:47.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff91a16c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9140712e0 con 0x7ff8fc005250 2023-03-03T19:00:47.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff913fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9141ed940 0x7ff914148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:47.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff913fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9141ed940 0x7ff914148960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39596/0 (socket says 172.21.15.7:39596) 2023-03-03T19:00:47.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff913fff700 1 -- 172.21.15.7:0/2646202381 learned_addr learned my addr 172.21.15.7:0/2646202381 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:00:47.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff9137fe700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc005250 0x7ff914071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:47.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff912ffd700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff914149400 0x7ff9140a74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:47.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff913fff700 1 -- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc005250 msgr2=0x7ff914071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:47.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff913fff700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc005250 0x7ff914071a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:47.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff913fff700 1 -- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff914149400 msgr2=0x7ff9140a74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:47.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff913fff700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff914149400 0x7ff9140a74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:47.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.018+0000 7ff913fff700 1 -- 172.21.15.7:0/2646202381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff914153160 con 0x7ff9141ed940 2023-03-03T19:00:47.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.019+0000 7ff913fff700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9141ed940 0x7ff914148960 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7ff90c00c910 tx=0x7ff90c001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:00:47.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.019+0000 7ff8fbfff700 1 -- 172.21.15.7:0/2646202381 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90c020070 con 0x7ff9141ed940 2023-03-03T19:00:47.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.019+0000 7ff8fbfff700 1 -- 172.21.15.7:0/2646202381 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90c00cc80 con 0x7ff9141ed940 2023-03-03T19:00:47.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.019+0000 7ff91a16c700 1 -- 172.21.15.7:0/2646202381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff914152e80 con 0x7ff9141ed940 2023-03-03T19:00:47.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.019+0000 7ff8fbfff700 1 -- 172.21.15.7:0/2646202381 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90c005290 con 0x7ff9141ed940 2023-03-03T19:00:47.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.019+0000 7ff91a16c700 1 -- 172.21.15.7:0/2646202381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9140c68e0 con 0x7ff9141ed940 2023-03-03T19:00:47.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.020+0000 7ff91a16c700 1 -- 172.21.15.7:0/2646202381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9140a9bf0 con 0x7ff9141ed940 2023-03-03T19:00:47.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.020+0000 7ff8fbfff700 1 -- 172.21.15.7:0/2646202381 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff90c019040 con 0x7ff9141ed940 2023-03-03T19:00:47.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.021+0000 7ff8fbfff700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 0x7ff90006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:00:47.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.021+0000 7ff8fbfff700 1 -- 172.21.15.7:0/2646202381 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff90c003e40 con 0x7ff9141ed940 2023-03-03T19:00:47.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.021+0000 7ff9137fe700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 0x7ff90006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:00:47.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.021+0000 7ff9137fe700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 0x7ff90006a7a0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff904000c00 tx=0x7ff904001e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:00:47.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.025+0000 7ff8fbfff700 1 -- 172.21.15.7:0/2646202381 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff90c052060 con 0x7ff9141ed940 2023-03-03T19:00:47.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.261+0000 7ff91a16c700 1 -- 172.21.15.7:0/2646202381 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff914074bb0 con 0x7ff9141ed940 2023-03-03T19:00:47.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.262+0000 7ff8fbfff700 1 -- 172.21.15.7:0/2646202381 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff90c002080 con 0x7ff9141ed940 2023-03-03T19:00:47.262 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:00:47.263 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:00:47.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 -- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 msgr2=0x7ff90006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:47.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 0x7ff90006a7a0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff904000c00 tx=0x7ff904001e00 comp rx=0 tx=0).stop 2023-03-03T19:00:47.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 -- 172.21.15.7:0/2646202381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9141ed940 msgr2=0x7ff914148960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:00:47.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9141ed940 0x7ff914148960 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7ff90c00c910 tx=0x7ff90c001650 comp rx=0 tx=0).stop 2023-03-03T19:00:47.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 -- 172.21.15.7:0/2646202381 shutdown_connections 2023-03-03T19:00:47.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff9000682f0 0x7ff90006a7a0 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:47.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9141ed940 0x7ff914148960 unknown :-1 s=CLOSED pgs=3363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:47.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff8fc005250 0x7ff914071a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:47.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 --2- 172.21.15.7:0/2646202381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff914149400 0x7ff9140a74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:00:47.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 -- 172.21.15.7:0/2646202381 >> 172.21.15.7:0/2646202381 conn(0x7ff9140a9430 msgr2=0x7ff9140c4980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:00:47.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 -- 172.21.15.7:0/2646202381 shutdown_connections 2023-03-03T19:00:47.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:00:47.264+0000 7ff91a16c700 1 -- 172.21.15.7:0/2646202381 wait complete. 2023-03-03T19:00:47.266 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:00:47.276 INFO:tasks.mds_thrash.fs.[b]:waiting for 110 secs before thrashing 2023-03-03T19:01:14.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:01:14.177 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:01:44.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:01:44.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:01:57.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:01:58.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.096+0000 7f10700a7700 1 Processor -- start 2023-03-03T19:01:58.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.097+0000 7f10700a7700 1 -- start start 2023-03-03T19:01:58.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.097+0000 7f10700a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10680aad20 0x7f1068157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:01:58.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.097+0000 7f10700a7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f10680aa030 0x7f1068158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:01:58.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.097+0000 7f10700a7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10680a9c20 con 0x7f106814b0d0 2023-03-03T19:01:58.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.097+0000 7f10700a7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1068154f10 con 0x7f10680aa030 2023-03-03T19:01:58.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.097+0000 7f10700a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10680ac3c0 con 0x7f10680aad20 2023-03-03T19:01:58.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106e644700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f10680aa030 0x7f1068158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:01:58.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106de43700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f106814b0d0 0x7f1068071800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33628/0 (socket says 172.21.15.7:33628) 2023-03-03T19:01:58.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106de43700 1 -- 172.21.15.7:0/3618462486 learned_addr learned my addr 172.21.15.7:0/3618462486 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:01:58.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106d642700 1 --2- 172.21.15.7:0/3618462486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10680aad20 0x7f1068157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:01:58.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106e644700 1 -- 172.21.15.7:0/3618462486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10680aad20 msgr2=0x7f1068157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:01:58.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106e644700 1 --2- 172.21.15.7:0/3618462486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10680aad20 0x7f1068157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:01:58.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106e644700 1 -- 172.21.15.7:0/3618462486 >> v1:172.21.15.7:6789/0 conn(0x7f106814b0d0 legacy=0x7f1068071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:01:58.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106e644700 1 -- 172.21.15.7:0/3618462486 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106806fc70 con 0x7f10680aa030 2023-03-03T19:01:58.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106ce41700 1 -- 172.21.15.7:0/3618462486 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781633175 0 0) 0x7f10680a9c20 con 0x7f106814b0d0 2023-03-03T19:01:58.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106e644700 1 --2- 172.21.15.7:0/3618462486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 0x7f1068158330 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f1060009d90 tx=0x7f106000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d2b5c32feba646 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:01:58.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106ce41700 1 -- 172.21.15.7:0/3618462486 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106000e070 con 0x7f10680aa030 2023-03-03T19:01:58.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106ce41700 1 -- 172.21.15.7:0/3618462486 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106000bbd0 con 0x7f10680aa030 2023-03-03T19:01:58.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.098+0000 7f106ce41700 1 -- 172.21.15.7:0/3618462486 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060012430 con 0x7f10680aa030 2023-03-03T19:01:58.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.099+0000 7f10700a7700 1 -- 172.21.15.7:0/3618462486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 msgr2=0x7f1068158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:01:58.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.099+0000 7f10700a7700 1 --2- 172.21.15.7:0/3618462486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 0x7f1068158330 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f1060009d90 tx=0x7f106000b330 comp rx=0 tx=0).stop 2023-03-03T19:01:58.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.099+0000 7f10700a7700 1 -- 172.21.15.7:0/3618462486 shutdown_connections 2023-03-03T19:01:58.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.099+0000 7f10700a7700 1 --2- 172.21.15.7:0/3618462486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 0x7f1068158330 unknown :-1 s=CLOSED pgs=3365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:01:58.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.099+0000 7f10700a7700 1 --2- 172.21.15.7:0/3618462486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f10680aad20 0x7f1068157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:01:58.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.099+0000 7f10700a7700 1 -- 172.21.15.7:0/3618462486 >> 172.21.15.7:0/3618462486 conn(0x7f10680c8900 msgr2=0x7f10680c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:01:58.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.099+0000 7f10700a7700 1 -- 172.21.15.7:0/3618462486 shutdown_connections 2023-03-03T19:01:58.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.099+0000 7f10700a7700 1 -- 172.21.15.7:0/3618462486 wait complete. 2023-03-03T19:01:58.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.100+0000 7f10700a7700 1 Processor -- start 2023-03-03T19:01:58.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.100+0000 7f10700a7700 1 -- start start 2023-03-03T19:01:58.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f10700a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 0x7f1068145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:01:58.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f10700a7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10680aad20 0x7f10681568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:01:58.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f10700a7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106814b0d0 0x7f10681ed920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:01:58.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f10700a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10681526e0 con 0x7f10680aa030 2023-03-03T19:01:58.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f10700a7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10681529c0 con 0x7f10680aad20 2023-03-03T19:01:58.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f10700a7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1068152ca0 con 0x7f106814b0d0 2023-03-03T19:01:58.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106de43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 0x7f1068145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:01:58.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106de43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 0x7f1068145930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39654/0 (socket says 172.21.15.7:39654) 2023-03-03T19:01:58.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106de43700 1 -- 172.21.15.7:0/61456015 learned_addr learned my addr 172.21.15.7:0/61456015 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:01:58.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106e644700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106814b0d0 0x7f10681ed920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:01:58.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106d642700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10680aad20 0x7f10681568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:01:58.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106de43700 1 -- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106814b0d0 msgr2=0x7f10681ed920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:01:58.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106de43700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106814b0d0 0x7f10681ed920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:01:58.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106de43700 1 -- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10680aad20 msgr2=0x7f10681568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:01:58.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106de43700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10680aad20 0x7f10681568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:01:58.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106de43700 1 -- 172.21.15.7:0/61456015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1060009a70 con 0x7f10680aa030 2023-03-03T19:01:58.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.101+0000 7f106d642700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10680aad20 0x7f10681568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:01:58.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.102+0000 7f106de43700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 0x7f1068145930 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7f10640011e0 tx=0x7f106400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:01:58.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.102+0000 7f105a7fc700 1 -- 172.21.15.7:0/61456015 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064004c80 con 0x7f10680aa030 2023-03-03T19:01:58.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.102+0000 7f105a7fc700 1 -- 172.21.15.7:0/61456015 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1064004de0 con 0x7f10680aa030 2023-03-03T19:01:58.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.102+0000 7f10700a7700 1 -- 172.21.15.7:0/61456015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106806f6b0 con 0x7f10680aa030 2023-03-03T19:01:58.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.102+0000 7f105a7fc700 1 -- 172.21.15.7:0/61456015 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064007310 con 0x7f10680aa030 2023-03-03T19:01:58.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.102+0000 7f10700a7700 1 -- 172.21.15.7:0/61456015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1068155160 con 0x7f10680aa030 2023-03-03T19:01:58.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.103+0000 7f10700a7700 1 -- 172.21.15.7:0/61456015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106806f990 con 0x7f10680aa030 2023-03-03T19:01:58.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.103+0000 7f105a7fc700 1 -- 172.21.15.7:0/61456015 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f106401a020 con 0x7f10680aa030 2023-03-03T19:01:58.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.103+0000 7f105a7fc700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f104c0682f0 0x7f104c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:01:58.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.104+0000 7f105a7fc700 1 -- 172.21.15.7:0/61456015 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1064085d00 con 0x7f10680aa030 2023-03-03T19:01:58.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.104+0000 7f106d642700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f104c0682f0 0x7f104c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:01:58.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.104+0000 7f106d642700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f104c0682f0 0x7f104c06a7a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f10680ac870 tx=0x7f105c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:01:58.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.108+0000 7f105a7fc700 1 -- 172.21.15.7:0/61456015 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1064089030 con 0x7f10680aa030 2023-03-03T19:01:58.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.343+0000 7f10700a7700 1 -- 172.21.15.7:0/61456015 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1068153190 con 0x7f10680aa030 2023-03-03T19:01:58.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.344+0000 7f105a7fc700 1 -- 172.21.15.7:0/61456015 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f10640520a0 con 0x7f10680aa030 2023-03-03T19:01:58.344 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:01:58.345 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:01:58.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 -- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f104c0682f0 msgr2=0x7f104c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:01:58.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f104c0682f0 0x7f104c06a7a0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f10680ac870 tx=0x7f105c008040 comp rx=0 tx=0).stop 2023-03-03T19:01:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 -- 172.21.15.7:0/61456015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 msgr2=0x7f1068145930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:01:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 0x7f1068145930 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7f10640011e0 tx=0x7f106400ca30 comp rx=0 tx=0).stop 2023-03-03T19:01:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 -- 172.21.15.7:0/61456015 shutdown_connections 2023-03-03T19:01:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f106814b0d0 0x7f10681ed920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:01:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f104c0682f0 0x7f104c06a7a0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:01:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f10680aa030 0x7f1068145930 unknown :-1 s=CLOSED pgs=3366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:01:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 --2- 172.21.15.7:0/61456015 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10680aad20 0x7f10681568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:01:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 -- 172.21.15.7:0/61456015 >> 172.21.15.7:0/61456015 conn(0x7f10680c8900 msgr2=0x7f1068148490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:01:58.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 -- 172.21.15.7:0/61456015 shutdown_connections 2023-03-03T19:01:58.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:01:58.346+0000 7f10700a7700 1 -- 172.21.15.7:0/61456015 wait complete. 2023-03-03T19:01:58.348 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:01:58.358 INFO:tasks.mds_thrash.fs.[a]:waiting for 24 secs before thrashing 2023-03-03T19:02:14.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:02:14.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:02:22.359 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:02:22.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.498+0000 7fe9d6024700 1 Processor -- start 2023-03-03T19:02:22.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.498+0000 7fe9d6024700 1 -- start start 2023-03-03T19:02:22.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.499+0000 7fe9d6024700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe9d0155390 0x7fe9d00c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:22.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.499+0000 7fe9d6024700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe9d0151770 0x7fe9d0151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:22.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.499+0000 7fe9d6024700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d0154910 con 0x7fe9d006e020 2023-03-03T19:02:22.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.499+0000 7fe9d6024700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d0070b90 con 0x7fe9d0151770 2023-03-03T19:02:22.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.499+0000 7fe9d6024700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d0151320 con 0x7fe9d0155390 2023-03-03T19:02:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.499+0000 7fe9cffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe9d0151770 0x7fe9d0151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.499+0000 7fe9ceffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe9d006e020 0x7fe9d0071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33664/0 (socket says 172.21.15.7:33664) 2023-03-03T19:02:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.499+0000 7fe9ceffd700 1 -- 172.21.15.7:0/3899710936 learned_addr learned my addr 172.21.15.7:0/3899710936 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:02:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.499+0000 7fe9cf7fe700 1 --2- 172.21.15.7:0/3899710936 >> v2:172.21.15.143:3300/0 conn(0x7fe9d0155390 0x7fe9d00c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9cffff700 1 -- 172.21.15.7:0/3899710936 >> v2:172.21.15.143:3300/0 conn(0x7fe9d0155390 msgr2=0x7fe9d00c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:02:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9cffff700 1 --2- 172.21.15.7:0/3899710936 >> v2:172.21.15.143:3300/0 conn(0x7fe9d0155390 0x7fe9d00c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9cffff700 1 -- 172.21.15.7:0/3899710936 >> v1:172.21.15.7:6789/0 conn(0x7fe9d006e020 legacy=0x7fe9d0071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:22.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9cffff700 1 -- 172.21.15.7:0/3899710936 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d00702a0 con 0x7fe9d0151770 2023-03-03T19:02:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9ce7fc700 1 -- 172.21.15.7:0/3899710936 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482270148 0 0) 0x7fe9d0154910 con 0x7fe9d006e020 2023-03-03T19:02:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9cffff700 1 --2- 172.21.15.7:0/3899710936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d0151770 0x7fe9d0151bd0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7fe9d006d700 tx=0x7fe9c8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322dc5feed8d6f7b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:02:22.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9ce7fc700 1 -- 172.21.15.7:0/3899710936 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8011830 con 0x7fe9d0151770 2023-03-03T19:02:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9ce7fc700 1 -- 172.21.15.7:0/3899710936 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c8011990 con 0x7fe9d0151770 2023-03-03T19:02:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9ce7fc700 1 -- 172.21.15.7:0/3899710936 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8011d20 con 0x7fe9d0151770 2023-03-03T19:02:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9d6024700 1 -- 172.21.15.7:0/3899710936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d0151770 msgr2=0x7fe9d0151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.500+0000 7fe9d6024700 1 --2- 172.21.15.7:0/3899710936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d0151770 0x7fe9d0151bd0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7fe9d006d700 tx=0x7fe9c8009b20 comp rx=0 tx=0).stop 2023-03-03T19:02:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.501+0000 7fe9d6024700 1 -- 172.21.15.7:0/3899710936 shutdown_connections 2023-03-03T19:02:22.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.501+0000 7fe9d6024700 1 --2- 172.21.15.7:0/3899710936 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d0151770 0x7fe9d0151bd0 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:22.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.501+0000 7fe9d6024700 1 --2- 172.21.15.7:0/3899710936 >> v2:172.21.15.143:3300/0 conn(0x7fe9d0155390 0x7fe9d00c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:22.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.501+0000 7fe9d6024700 1 -- 172.21.15.7:0/3899710936 >> 172.21.15.7:0/3899710936 conn(0x7fe9d006c110 msgr2=0x7fe9d006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:02:22.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.501+0000 7fe9d6024700 1 -- 172.21.15.7:0/3899710936 shutdown_connections 2023-03-03T19:02:22.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.501+0000 7fe9d6024700 1 -- 172.21.15.7:0/3899710936 wait complete. 2023-03-03T19:02:22.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.502+0000 7fe9d6024700 1 Processor -- start 2023-03-03T19:02:22.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.502+0000 7fe9d6024700 1 -- start start 2023-03-03T19:02:22.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9d6024700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d006e020 0x7fe9d01491c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:22.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9d6024700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d0151770 0x7fe9d00c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:22.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9d6024700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9d0155390 0x7fe9d00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:22.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9d6024700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d00c5a50 con 0x7fe9d006e020 2023-03-03T19:02:22.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9d6024700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d014c980 con 0x7fe9d0151770 2023-03-03T19:02:22.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9d6024700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d0149700 con 0x7fe9d0155390 2023-03-03T19:02:22.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d006e020 0x7fe9d01491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:22.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d006e020 0x7fe9d01491c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39686/0 (socket says 172.21.15.7:39686) 2023-03-03T19:02:22.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9cf7fe700 1 -- 172.21.15.7:0/2665785953 learned_addr learned my addr 172.21.15.7:0/2665785953 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:02:22.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9cf7fe700 1 -- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9d0155390 msgr2=0x7fe9d00736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:22.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9cffff700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9d0155390 0x7fe9d00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:22.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9ceffd700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d0151770 0x7fe9d00c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:22.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9cf7fe700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9d0155390 0x7fe9d00736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:22.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9cf7fe700 1 -- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d0151770 msgr2=0x7fe9d00c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:22.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9cf7fe700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d0151770 0x7fe9d00c5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:22.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9cf7fe700 1 -- 172.21.15.7:0/2665785953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80094a0 con 0x7fe9d006e020 2023-03-03T19:02:22.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.503+0000 7fe9cf7fe700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d006e020 0x7fe9d01491c0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000efc0 tx=0x7fe9c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:02:22.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.504+0000 7fe9b7fff700 1 -- 172.21.15.7:0/2665785953 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0009b80 con 0x7fe9d006e020 2023-03-03T19:02:22.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.504+0000 7fe9b7fff700 1 -- 172.21.15.7:0/2665785953 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c000f040 con 0x7fe9d006e020 2023-03-03T19:02:22.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.504+0000 7fe9b7fff700 1 -- 172.21.15.7:0/2665785953 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0014480 con 0x7fe9d006e020 2023-03-03T19:02:22.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.504+0000 7fe9d6024700 1 -- 172.21.15.7:0/2665785953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d0070820 con 0x7fe9d006e020 2023-03-03T19:02:22.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.504+0000 7fe9d6024700 1 -- 172.21.15.7:0/2665785953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d0070540 con 0x7fe9d006e020 2023-03-03T19:02:22.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.505+0000 7fe9d6024700 1 -- 172.21.15.7:0/2665785953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d0155760 con 0x7fe9d006e020 2023-03-03T19:02:22.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.505+0000 7fe9b7fff700 1 -- 172.21.15.7:0/2665785953 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe9c00145e0 con 0x7fe9d006e020 2023-03-03T19:02:22.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.505+0000 7fe9b7fff700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9bc0682f0 0x7fe9bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:22.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.506+0000 7fe9b7fff700 1 -- 172.21.15.7:0/2665785953 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe9c0086b60 con 0x7fe9d006e020 2023-03-03T19:02:22.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.506+0000 7fe9ceffd700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9bc0682f0 0x7fe9bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:22.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.506+0000 7fe9ceffd700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9bc0682f0 0x7fe9bc06a7a0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fe9d01e5610 tx=0x7fe9c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:02:22.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.510+0000 7fe9b7fff700 1 -- 172.21.15.7:0/2665785953 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe9c0052fb0 con 0x7fe9d006e020 2023-03-03T19:02:22.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.746+0000 7fe9d6024700 1 -- 172.21.15.7:0/2665785953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9d006fa00 con 0x7fe9d006e020 2023-03-03T19:02:22.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.746+0000 7fe9b7fff700 1 -- 172.21.15.7:0/2665785953 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe9c00565d0 con 0x7fe9d006e020 2023-03-03T19:02:22.748 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:02:22.748 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:02:22.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 -- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9bc0682f0 msgr2=0x7fe9bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:22.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9bc0682f0 0x7fe9bc06a7a0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fe9d01e5610 tx=0x7fe9c4009040 comp rx=0 tx=0).stop 2023-03-03T19:02:22.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 -- 172.21.15.7:0/2665785953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d006e020 msgr2=0x7fe9d01491c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:22.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d006e020 0x7fe9d01491c0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000efc0 tx=0x7fe9c000c3a0 comp rx=0 tx=0).stop 2023-03-03T19:02:22.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 -- 172.21.15.7:0/2665785953 shutdown_connections 2023-03-03T19:02:22.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9d0155390 0x7fe9d00736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:22.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe9bc0682f0 0x7fe9bc06a7a0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:22.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9d006e020 0x7fe9d01491c0 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:22.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 --2- 172.21.15.7:0/2665785953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9d0151770 0x7fe9d00c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:22.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 -- 172.21.15.7:0/2665785953 >> 172.21.15.7:0/2665785953 conn(0x7fe9d006c110 msgr2=0x7fe9d006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:02:22.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 -- 172.21.15.7:0/2665785953 shutdown_connections 2023-03-03T19:02:22.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:22.748+0000 7fe9d6024700 1 -- 172.21.15.7:0/2665785953 wait complete. 2023-03-03T19:02:22.752 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:02:22.762 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before thrashing 2023-03-03T19:02:28.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:02:28.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.900+0000 7ffaf3e46700 1 Processor -- start 2023-03-03T19:02:28.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.901+0000 7ffaf3e46700 1 -- start start 2023-03-03T19:02:28.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.901+0000 7ffaf3e46700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ffaec0ac220 0x7ffaec0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:28.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.901+0000 7ffaf3e46700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffaec073c30 0x7ffaec152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:28.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.901+0000 7ffaf3e46700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffaec0ace30 0x7ffaec152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:28.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.901+0000 7ffaf3e46700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec071e30 con 0x7ffaec073c30 2023-03-03T19:02:28.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.901+0000 7ffaf3e46700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec152ed0 con 0x7ffaec0ac220 2023-03-03T19:02:28.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.901+0000 7ffaf3e46700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0ac900 con 0x7ffaec0ace30 2023-03-03T19:02:28.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf13e1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffaec073c30 0x7ffaec152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:28.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf13e1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffaec073c30 0x7ffaec152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39694/0 (socket says 172.21.15.7:39694) 2023-03-03T19:02:28.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf13e1700 1 -- 172.21.15.7:0/492116933 learned_addr learned my addr 172.21.15.7:0/492116933 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:02:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf1be2700 1 --2- 172.21.15.7:0/492116933 >> v2:172.21.15.143:3300/0 conn(0x7ffaec0ac220 0x7ffaec0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf23e3700 1 --2- 172.21.15.7:0/492116933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffaec0ace30 0x7ffaec152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf13e1700 1 -- 172.21.15.7:0/492116933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffaec0ace30 msgr2=0x7ffaec152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:02:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf13e1700 1 --2- 172.21.15.7:0/492116933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffaec0ace30 0x7ffaec152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf13e1700 1 -- 172.21.15.7:0/492116933 >> v2:172.21.15.143:3300/0 conn(0x7ffaec0ac220 msgr2=0x7ffaec0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:02:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf13e1700 1 --2- 172.21.15.7:0/492116933 >> v2:172.21.15.143:3300/0 conn(0x7ffaec0ac220 0x7ffaec0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf13e1700 1 -- 172.21.15.7:0/492116933 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec0acb50 con 0x7ffaec073c30 2023-03-03T19:02:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf13e1700 1 --2- 172.21.15.7:0/492116933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 0x7ffaec152450 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7ffaec06fbf0 tx=0x7ffae0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d379519635d857e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:02:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf0be0700 1 -- 172.21.15.7:0/492116933 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0011830 con 0x7ffaec073c30 2023-03-03T19:02:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf0be0700 1 -- 172.21.15.7:0/492116933 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae0011990 con 0x7ffaec073c30 2023-03-03T19:02:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.902+0000 7ffaf0be0700 1 -- 172.21.15.7:0/492116933 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae0011d20 con 0x7ffaec073c30 2023-03-03T19:02:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.903+0000 7ffaf3e46700 1 -- 172.21.15.7:0/492116933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 msgr2=0x7ffaec152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.903+0000 7ffaf3e46700 1 --2- 172.21.15.7:0/492116933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 0x7ffaec152450 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7ffaec06fbf0 tx=0x7ffae0009bd0 comp rx=0 tx=0).stop 2023-03-03T19:02:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.903+0000 7ffaf3e46700 1 -- 172.21.15.7:0/492116933 shutdown_connections 2023-03-03T19:02:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.903+0000 7ffaf3e46700 1 --2- 172.21.15.7:0/492116933 >> v2:172.21.15.143:3300/0 conn(0x7ffaec0ac220 0x7ffaec0736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.903+0000 7ffaf3e46700 1 --2- 172.21.15.7:0/492116933 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffaec0ace30 0x7ffaec152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.903+0000 7ffaf3e46700 1 --2- 172.21.15.7:0/492116933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 0x7ffaec152450 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.903+0000 7ffaf3e46700 1 -- 172.21.15.7:0/492116933 >> 172.21.15.7:0/492116933 conn(0x7ffaec0a94d0 msgr2=0x7ffaec0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:02:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.903+0000 7ffaf3e46700 1 -- 172.21.15.7:0/492116933 shutdown_connections 2023-03-03T19:02:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.903+0000 7ffaf3e46700 1 -- 172.21.15.7:0/492116933 wait complete. 2023-03-03T19:02:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.905+0000 7ffaf3e46700 1 Processor -- start 2023-03-03T19:02:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.905+0000 7ffaf3e46700 1 -- start start 2023-03-03T19:02:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.905+0000 7ffaf3e46700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 0x7ffaec0ba3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.905+0000 7ffaf3e46700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffaec0ac220 0x7ffaec0bea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.905+0000 7ffaf3e46700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffaec0ace30 0x7ffaec1e6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.905+0000 7ffaf3e46700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0c0050 con 0x7ffaec073c30 2023-03-03T19:02:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.905+0000 7ffaf3e46700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0bfd20 con 0x7ffaec0ac220 2023-03-03T19:02:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.905+0000 7ffaf3e46700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec1edcb0 con 0x7ffaec0ace30 2023-03-03T19:02:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.905+0000 7ffaf1be2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 0x7ffaec0ba3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf1be2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 0x7ffaec0ba3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39700/0 (socket says 172.21.15.7:39700) 2023-03-03T19:02:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf1be2700 1 -- 172.21.15.7:0/1920338706 learned_addr learned my addr 172.21.15.7:0/1920338706 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:02:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf13e1700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffaec0ac220 0x7ffaec0bea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf1be2700 1 -- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffaec0ace30 msgr2=0x7ffaec1e6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf1be2700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffaec0ace30 0x7ffaec1e6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf1be2700 1 -- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffaec0ac220 msgr2=0x7ffaec0bea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf1be2700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffaec0ac220 0x7ffaec0bea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf1be2700 1 -- 172.21.15.7:0/1920338706 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae0009550 con 0x7ffaec073c30 2023-03-03T19:02:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf1be2700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 0x7ffaec0ba3e0 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7ffae800e7f0 tx=0x7ffae800ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:02:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffade7fc700 1 -- 172.21.15.7:0/1920338706 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8009590 con 0x7ffaec073c30 2023-03-03T19:02:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffade7fc700 1 -- 172.21.15.7:0/1920338706 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae8013680 con 0x7ffaec073c30 2023-03-03T19:02:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffade7fc700 1 -- 172.21.15.7:0/1920338706 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae80139f0 con 0x7ffaec073c30 2023-03-03T19:02:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf3e46700 1 -- 172.21.15.7:0/1920338706 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec155ca0 con 0x7ffaec073c30 2023-03-03T19:02:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.906+0000 7ffaf3e46700 1 -- 172.21.15.7:0/1920338706 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec1559c0 con 0x7ffaec073c30 2023-03-03T19:02:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.907+0000 7ffaf3e46700 1 -- 172.21.15.7:0/1920338706 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec155f80 con 0x7ffaec073c30 2023-03-03T19:02:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.908+0000 7ffade7fc700 1 -- 172.21.15.7:0/1920338706 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffae8023070 con 0x7ffaec073c30 2023-03-03T19:02:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.908+0000 7ffade7fc700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffad00682f0 0x7ffad006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.908+0000 7ffade7fc700 1 -- 172.21.15.7:0/1920338706 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ffae80862a0 con 0x7ffaec073c30 2023-03-03T19:02:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.908+0000 7ffaf13e1700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffad00682f0 0x7ffad006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.909+0000 7ffaf13e1700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffad00682f0 0x7ffad006a7a0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0c1950 tx=0x7ffae0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:02:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:28.913+0000 7ffade7fc700 1 -- 172.21.15.7:0/1920338706 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ffae80525e0 con 0x7ffaec073c30 2023-03-03T19:02:29.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.151+0000 7ffaf3e46700 1 -- 172.21.15.7:0/1920338706 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaec074bb0 con 0x7ffaec073c30 2023-03-03T19:02:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.152+0000 7ffade7fc700 1 -- 172.21.15.7:0/1920338706 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ffae8055c00 con 0x7ffaec073c30 2023-03-03T19:02:29.153 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:02:29.154 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:02:29.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 -- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffad00682f0 msgr2=0x7ffad006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:29.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffad00682f0 0x7ffad006a7a0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0c1950 tx=0x7ffae0008030 comp rx=0 tx=0).stop 2023-03-03T19:02:29.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 -- 172.21.15.7:0/1920338706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 msgr2=0x7ffaec0ba3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:29.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 0x7ffaec0ba3e0 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7ffae800e7f0 tx=0x7ffae800ebb0 comp rx=0 tx=0).stop 2023-03-03T19:02:29.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 -- 172.21.15.7:0/1920338706 shutdown_connections 2023-03-03T19:02:29.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ffaec0ace30 0x7ffaec1e6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:29.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ffad00682f0 0x7ffad006a7a0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:29.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffaec073c30 0x7ffaec0ba3e0 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:29.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 --2- 172.21.15.7:0/1920338706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffaec0ac220 0x7ffaec0bea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:29.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 -- 172.21.15.7:0/1920338706 >> 172.21.15.7:0/1920338706 conn(0x7ffaec0a94d0 msgr2=0x7ffaec1527c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:02:29.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 -- 172.21.15.7:0/1920338706 shutdown_connections 2023-03-03T19:02:29.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:29.154+0000 7ffaf3e46700 1 -- 172.21.15.7:0/1920338706 wait complete. 2023-03-03T19:02:29.157 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:02:29.167 INFO:tasks.mds_thrash.fs.[a]:waiting for 32 secs before thrashing 2023-03-03T19:02:37.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:02:37.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.417+0000 7fac47329700 1 Processor -- start 2023-03-03T19:02:37.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.417+0000 7fac47329700 1 -- start start 2023-03-03T19:02:37.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.417+0000 7fac47329700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac4014b0d0 0x7fac40071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:37.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.417+0000 7fac47329700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fac40071d40 0x7fac40158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:37.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac47329700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac400a9c20 con 0x7fac400aad20 2023-03-03T19:02:37.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac47329700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac40154f10 con 0x7fac4014b0d0 2023-03-03T19:02:37.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac47329700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac400ac3c0 con 0x7fac40071d40 2023-03-03T19:02:37.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac450c5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac4014b0d0 0x7fac40071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:37.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac450c5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fac4014b0d0 0x7fac40071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39716/0 (socket says 172.21.15.7:39716) 2023-03-03T19:02:37.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac450c5700 1 -- 172.21.15.7:0/3144180025 learned_addr learned my addr 172.21.15.7:0/3144180025 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:02:37.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac458c6700 1 --2- 172.21.15.7:0/3144180025 >> v2:172.21.15.143:3300/0 conn(0x7fac40071d40 0x7fac40158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:37.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac450c5700 1 -- 172.21.15.7:0/3144180025 >> v2:172.21.15.143:3300/0 conn(0x7fac40071d40 msgr2=0x7fac40158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:02:37.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac450c5700 1 --2- 172.21.15.7:0/3144180025 >> v2:172.21.15.143:3300/0 conn(0x7fac40071d40 0x7fac40158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:37.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac450c5700 1 -- 172.21.15.7:0/3144180025 >> v1:172.21.15.143:6789/0 conn(0x7fac400aad20 legacy=0x7fac40157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:02:37.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac450c5700 1 -- 172.21.15.7:0/3144180025 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4006fc70 con 0x7fac4014b0d0 2023-03-03T19:02:37.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac450c5700 1 --2- 172.21.15.7:0/3144180025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4014b0d0 0x7fac40071800 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7fac400a98d0 tx=0x7fac3c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca282d8cfef2c57 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:02:37.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac33fff700 1 -- 172.21.15.7:0/3144180025 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c004230 con 0x7fac4014b0d0 2023-03-03T19:02:37.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac33fff700 1 -- 172.21.15.7:0/3144180025 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c004390 con 0x7fac4014b0d0 2023-03-03T19:02:37.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.418+0000 7fac33fff700 1 -- 172.21.15.7:0/3144180025 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0036a0 con 0x7fac4014b0d0 2023-03-03T19:02:37.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.419+0000 7fac47329700 1 -- 172.21.15.7:0/3144180025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4014b0d0 msgr2=0x7fac40071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:37.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.419+0000 7fac47329700 1 --2- 172.21.15.7:0/3144180025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4014b0d0 0x7fac40071800 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7fac400a98d0 tx=0x7fac3c009bd0 comp rx=0 tx=0).stop 2023-03-03T19:02:37.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.419+0000 7fac47329700 1 -- 172.21.15.7:0/3144180025 shutdown_connections 2023-03-03T19:02:37.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.419+0000 7fac47329700 1 --2- 172.21.15.7:0/3144180025 >> v2:172.21.15.143:3300/0 conn(0x7fac40071d40 0x7fac40158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:37.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.419+0000 7fac47329700 1 --2- 172.21.15.7:0/3144180025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac4014b0d0 0x7fac40071800 unknown :-1 s=CLOSED pgs=3372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:37.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.419+0000 7fac47329700 1 -- 172.21.15.7:0/3144180025 >> 172.21.15.7:0/3144180025 conn(0x7fac400c8900 msgr2=0x7fac400c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:02:37.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.419+0000 7fac47329700 1 -- 172.21.15.7:0/3144180025 shutdown_connections 2023-03-03T19:02:37.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.419+0000 7fac47329700 1 -- 172.21.15.7:0/3144180025 wait complete. 2023-03-03T19:02:37.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.420+0000 7fac47329700 1 Processor -- start 2023-03-03T19:02:37.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.420+0000 7fac47329700 1 -- start start 2023-03-03T19:02:37.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac47329700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac40071d40 0x7fac40156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:37.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac47329700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac400aad20 0x7fac40141da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:37.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac47329700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac4014b0d0 0x7fac401422e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:37.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac47329700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac40152510 con 0x7fac400aad20 2023-03-03T19:02:37.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac47329700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac401527f0 con 0x7fac4014b0d0 2023-03-03T19:02:37.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac47329700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac40152ad0 con 0x7fac40071d40 2023-03-03T19:02:37.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac448c4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac400aad20 0x7fac40141da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:37.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac448c4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac400aad20 0x7fac40141da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39724/0 (socket says 172.21.15.7:39724) 2023-03-03T19:02:37.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac448c4700 1 -- 172.21.15.7:0/1588337915 learned_addr learned my addr 172.21.15.7:0/1588337915 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:02:37.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac458c6700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac4014b0d0 0x7fac401422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:37.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac450c5700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac40071d40 0x7fac40156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:37.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac448c4700 1 -- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac40071d40 msgr2=0x7fac40156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:37.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac448c4700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac40071d40 0x7fac40156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:37.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac448c4700 1 -- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac4014b0d0 msgr2=0x7fac401422e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:37.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac448c4700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac4014b0d0 0x7fac401422e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:37.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.421+0000 7fac448c4700 1 -- 172.21.15.7:0/1588337915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c009550 con 0x7fac400aad20 2023-03-03T19:02:37.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.422+0000 7fac448c4700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac400aad20 0x7fac40141da0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7fac3400b450 tx=0x7fac3400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:02:37.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.422+0000 7fac31ffb700 1 -- 172.21.15.7:0/1588337915 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac34011830 con 0x7fac400aad20 2023-03-03T19:02:37.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.422+0000 7fac31ffb700 1 -- 172.21.15.7:0/1588337915 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac34011990 con 0x7fac400aad20 2023-03-03T19:02:37.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.422+0000 7fac47329700 1 -- 172.21.15.7:0/1588337915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4006f6b0 con 0x7fac400aad20 2023-03-03T19:02:37.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.422+0000 7fac31ffb700 1 -- 172.21.15.7:0/1588337915 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac34011d40 con 0x7fac400aad20 2023-03-03T19:02:37.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.422+0000 7fac47329700 1 -- 172.21.15.7:0/1588337915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac40155160 con 0x7fac400aad20 2023-03-03T19:02:37.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.423+0000 7fac47329700 1 -- 172.21.15.7:0/1588337915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4006f990 con 0x7fac400aad20 2023-03-03T19:02:37.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.423+0000 7fac31ffb700 1 -- 172.21.15.7:0/1588337915 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fac3400f050 con 0x7fac400aad20 2023-03-03T19:02:37.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.424+0000 7fac31ffb700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac240682c0 0x7fac2406a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:02:37.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.424+0000 7fac31ffb700 1 -- 172.21.15.7:0/1588337915 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fac34085de0 con 0x7fac400aad20 2023-03-03T19:02:37.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.424+0000 7fac450c5700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac240682c0 0x7fac2406a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:02:37.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.425+0000 7fac450c5700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac240682c0 0x7fac2406a770 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009130 tx=0x7fac3c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:02:37.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.428+0000 7fac31ffb700 1 -- 172.21.15.7:0/1588337915 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fac34052090 con 0x7fac400aad20 2023-03-03T19:02:37.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.667+0000 7fac47329700 1 -- 172.21.15.7:0/1588337915 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac400c60e0 con 0x7fac400aad20 2023-03-03T19:02:37.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.667+0000 7fac31ffb700 1 -- 172.21.15.7:0/1588337915 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fac34015020 con 0x7fac400aad20 2023-03-03T19:02:37.668 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:02:37.669 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:02:37.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.669+0000 7fac47329700 1 -- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac240682c0 msgr2=0x7fac2406a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:37.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.669+0000 7fac47329700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac240682c0 0x7fac2406a770 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009130 tx=0x7fac3c008030 comp rx=0 tx=0).stop 2023-03-03T19:02:37.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.669+0000 7fac47329700 1 -- 172.21.15.7:0/1588337915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac400aad20 msgr2=0x7fac40141da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:02:37.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.669+0000 7fac47329700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac400aad20 0x7fac40141da0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7fac3400b450 tx=0x7fac3400b810 comp rx=0 tx=0).stop 2023-03-03T19:02:37.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.670+0000 7fac47329700 1 -- 172.21.15.7:0/1588337915 shutdown_connections 2023-03-03T19:02:37.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.670+0000 7fac47329700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac4014b0d0 0x7fac401422e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:37.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.670+0000 7fac47329700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac40071d40 0x7fac40156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:37.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.670+0000 7fac47329700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac240682c0 0x7fac2406a770 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:37.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.670+0000 7fac47329700 1 --2- 172.21.15.7:0/1588337915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac400aad20 0x7fac40141da0 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:02:37.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.670+0000 7fac47329700 1 -- 172.21.15.7:0/1588337915 >> 172.21.15.7:0/1588337915 conn(0x7fac400c8900 msgr2=0x7fac400c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:02:37.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.670+0000 7fac47329700 1 -- 172.21.15.7:0/1588337915 shutdown_connections 2023-03-03T19:02:37.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:02:37.670+0000 7fac47329700 1 -- 172.21.15.7:0/1588337915 wait complete. 2023-03-03T19:02:37.675 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:02:37.682 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T19:02:44.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:02:44.374 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:03:01.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:03:01.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.308+0000 7fa20040d700 1 Processor -- start 2023-03-03T19:03:01.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.308+0000 7fa20040d700 1 -- start start 2023-03-03T19:03:01.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa20040d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa1f80c17f0 0x7fa1f806d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:01.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa20040d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f814ebe0 con 0x7fa1f80c6010 2023-03-03T19:03:01.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa20040d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f80c2e70 con 0x7fa1f80c5190 2023-03-03T19:03:01.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa20040d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f8152100 con 0x7fa1f80c17f0 2023-03-03T19:03:01.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fe1a9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa1f80c6010 0x7fa1f80c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33734/0 (socket says 172.21.15.7:33734) 2023-03-03T19:03:01.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fe1a9700 1 -- 172.21.15.7:0/2104163640 learned_addr learned my addr 172.21.15.7:0/2104163640 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:03:01.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd9a8700 1 --2- 172.21.15.7:0/2104163640 >> v2:172.21.15.143:3300/0 conn(0x7fa1f80c17f0 0x7fa1f806d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:01.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180022710 0 0) 0x7fa1f814ebe0 con 0x7fa1f80c6010 2023-03-03T19:03:01.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1e40035b0 con 0x7fa1f80c6010 2023-03-03T19:03:01.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1033435078 0 0) 0x7fa1e40035b0 con 0x7fa1f80c6010 2023-03-03T19:03:01.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 >> v2:172.21.15.143:3300/0 conn(0x7fa1f80c17f0 msgr2=0x7fa1f806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:03:01.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd1a7700 1 --2- 172.21.15.7:0/2104163640 >> v2:172.21.15.143:3300/0 conn(0x7fa1f80c17f0 0x7fa1f806d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 >> v1:172.21.15.143:6789/0 conn(0x7fa1f80c5190 legacy=0x7fa1f806dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:01.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f80aa680 con 0x7fa1f80c6010 2023-03-03T19:03:01.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fa1f4001670 con 0x7fa1f80c6010 2023-03-03T19:03:01.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 >> v1:172.21.15.7:6789/0 conn(0x7fa1f80c6010 legacy=0x7fa1f80c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:01.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.309+0000 7fa1fd1a7700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1e4003a70 0x7fa1e4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:01.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fd1a7700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1e4004850 0x7fa1e4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:01.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fd1a7700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1e400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:01.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f8152100 con 0x7fa1e4005250 2023-03-03T19:03:01.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e400c190 con 0x7fa1e4004850 2023-03-03T19:03:01.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e400c350 con 0x7fa1e4003a70 2023-03-03T19:03:01.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fe9aa700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1e400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:01.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fd9a8700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1e4004850 0x7fa1e4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:01.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fe1a9700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1e4003a70 0x7fa1e4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:01.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fe9aa700 1 -- 172.21.15.7:0/2104163640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1e4003a70 msgr2=0x7fa1e4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:03:01.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fe9aa700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1e4003a70 0x7fa1e4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fe9aa700 1 -- 172.21.15.7:0/2104163640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1e4004850 msgr2=0x7fa1e4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:03:01.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fe9aa700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1e4004850 0x7fa1e4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fe9aa700 1 -- 172.21.15.7:0/2104163640 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e40035b0 con 0x7fa1e4005250 2023-03-03T19:03:01.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fe9aa700 1 -- 172.21.15.7:0/2104163640 reap_dead start 2023-03-03T19:03:01.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fe9aa700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1e400ba10 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000c160 tx=0x7fa1e4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db89b2c0ec50081 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:03:01.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0001db0 con 0x7fa1e4005250 2023-03-03T19:03:01.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f000cca0 con 0x7fa1e4005250 2023-03-03T19:03:01.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.310+0000 7fa1fd1a7700 1 -- 172.21.15.7:0/2104163640 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0003c20 con 0x7fa1e4005250 2023-03-03T19:03:01.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.311+0000 7fa20040d700 1 -- 172.21.15.7:0/2104163640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 msgr2=0x7fa1e400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:01.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.311+0000 7fa20040d700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1e400ba10 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000c160 tx=0x7fa1e4005880 comp rx=0 tx=0).stop 2023-03-03T19:03:01.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.311+0000 7fa20040d700 1 -- 172.21.15.7:0/2104163640 shutdown_connections 2023-03-03T19:03:01.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.311+0000 7fa20040d700 1 --2- 172.21.15.7:0/2104163640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1e400ba10 unknown :-1 s=CLOSED pgs=3375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.311+0000 7fa20040d700 1 -- 172.21.15.7:0/2104163640 >> 172.21.15.7:0/2104163640 conn(0x7fa1f80a9430 msgr2=0x7fa1f80c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:03:01.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.311+0000 7fa20040d700 1 -- 172.21.15.7:0/2104163640 shutdown_connections 2023-03-03T19:03:01.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.311+0000 7fa20040d700 1 -- 172.21.15.7:0/2104163640 wait complete. 2023-03-03T19:03:01.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.312+0000 7fa20040d700 1 Processor -- start 2023-03-03T19:03:01.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.312+0000 7fa20040d700 1 -- start start 2023-03-03T19:03:01.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa20040d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1f81ece80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:01.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa20040d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1f80a5ed0 0x7fa1f8071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:01.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa20040d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1f8154130 0x7fa1f81e6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:01.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa20040d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f814cf00 con 0x7fa1e4005250 2023-03-03T19:03:01.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa20040d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81539e0 con 0x7fa1f80a5ed0 2023-03-03T19:03:01.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa20040d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f814c410 con 0x7fa1f8154130 2023-03-03T19:03:01.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa1fe1a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1f81ece80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:01.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa1fe1a9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1f81ece80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39766/0 (socket says 172.21.15.7:39766) 2023-03-03T19:03:01.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa1fe1a9700 1 -- 172.21.15.7:0/1282521654 learned_addr learned my addr 172.21.15.7:0/1282521654 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:03:01.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa1fd9a8700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1f80a5ed0 0x7fa1f8071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:01.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa1fe9aa700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1f8154130 0x7fa1f81e6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:01.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa1fe1a9700 1 -- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1f8154130 msgr2=0x7fa1f81e6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa1fe1a9700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1f8154130 0x7fa1f81e6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa1fe1a9700 1 -- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1f80a5ed0 msgr2=0x7fa1f8071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa1fe1a9700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1f80a5ed0 0x7fa1f8071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.313+0000 7fa1fe1a9700 1 -- 172.21.15.7:0/1282521654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f80c2b00 con 0x7fa1e4005250 2023-03-03T19:03:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.314+0000 7fa1fe1a9700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1f81ece80 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40072c0 tx=0x7fa1f40075d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:03:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.314+0000 7fa1eeffd700 1 -- 172.21.15.7:0/1282521654 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4012560 con 0x7fa1e4005250 2023-03-03T19:03:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.314+0000 7fa1eeffd700 1 -- 172.21.15.7:0/1282521654 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f40126e0 con 0x7fa1e4005250 2023-03-03T19:03:01.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.314+0000 7fa20040d700 1 -- 172.21.15.7:0/1282521654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f80aa960 con 0x7fa1e4005250 2023-03-03T19:03:01.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.314+0000 7fa1eeffd700 1 -- 172.21.15.7:0/1282521654 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4012a90 con 0x7fa1e4005250 2023-03-03T19:03:01.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.314+0000 7fa20040d700 1 -- 172.21.15.7:0/1282521654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f80c3300 con 0x7fa1e4005250 2023-03-03T19:03:01.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.315+0000 7fa20040d700 1 -- 172.21.15.7:0/1282521654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f8074bb0 con 0x7fa1e4005250 2023-03-03T19:03:01.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.315+0000 7fa1eeffd700 1 -- 172.21.15.7:0/1282521654 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa1f4012bf0 con 0x7fa1e4005250 2023-03-03T19:03:01.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.316+0000 7fa1eeffd700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1dc0682f0 0x7fa1dc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:01.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.316+0000 7fa1eeffd700 1 -- 172.21.15.7:0/1282521654 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa1f4086100 con 0x7fa1e4005250 2023-03-03T19:03:01.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.316+0000 7fa1fd9a8700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1dc0682f0 0x7fa1dc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.317+0000 7fa1fd9a8700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1dc0682f0 0x7fa1dc06a7a0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80c45f0 tx=0x7fa1e8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:03:01.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.320+0000 7fa1eeffd700 1 -- 172.21.15.7:0/1282521654 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa1f40524a0 con 0x7fa1e4005250 2023-03-03T19:03:01.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.557+0000 7fa20040d700 1 -- 172.21.15.7:0/1282521654 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1f80c15d0 con 0x7fa1e4005250 2023-03-03T19:03:01.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.557+0000 7fa1eeffd700 1 -- 172.21.15.7:0/1282521654 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa1f4055ac0 con 0x7fa1e4005250 2023-03-03T19:03:01.558 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:03:01.558 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:03:01.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 -- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1dc0682f0 msgr2=0x7fa1dc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:01.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1dc0682f0 0x7fa1dc06a7a0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80c45f0 tx=0x7fa1e8009210 comp rx=0 tx=0).stop 2023-03-03T19:03:01.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 -- 172.21.15.7:0/1282521654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 msgr2=0x7fa1f81ece80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:01.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1f81ece80 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40072c0 tx=0x7fa1f40075d0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 -- 172.21.15.7:0/1282521654 shutdown_connections 2023-03-03T19:03:01.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa1f8154130 0x7fa1f81e6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa1dc0682f0 0x7fa1dc06a7a0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa1e4005250 0x7fa1f81ece80 unknown :-1 s=CLOSED pgs=3376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 --2- 172.21.15.7:0/1282521654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa1f80a5ed0 0x7fa1f8071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:01.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 -- 172.21.15.7:0/1282521654 >> 172.21.15.7:0/1282521654 conn(0x7fa1f80a9430 msgr2=0x7fa1f80a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:03:01.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.559+0000 7fa20040d700 1 -- 172.21.15.7:0/1282521654 shutdown_connections 2023-03-03T19:03:01.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:01.560+0000 7fa20040d700 1 -- 172.21.15.7:0/1282521654 wait complete. 2023-03-03T19:03:01.561 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:03:01.572 INFO:tasks.mds_thrash.fs.[a]:waiting for 71 secs before thrashing 2023-03-03T19:03:14.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:03:14.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:03:41.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:03:41.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.823+0000 7f0458503700 1 Processor -- start 2023-03-03T19:03:41.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.823+0000 7f0458503700 1 -- start start 2023-03-03T19:03:41.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.824+0000 7f0458503700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f045014b0d0 0x7f0450071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:41.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.824+0000 7f0458503700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 0x7f0450157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:41.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.824+0000 7f0458503700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0450071d40 0x7f0450158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:41.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.824+0000 7f0458503700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04500a9c20 con 0x7f045014b0d0 2023-03-03T19:03:41.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.824+0000 7f0458503700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0450154f10 con 0x7f0450071d40 2023-03-03T19:03:41.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.824+0000 7f0458503700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04500ac3c0 con 0x7f04500aad20 2023-03-03T19:03:41.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045629f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f045014b0d0 0x7f0450071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:41.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045629f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f045014b0d0 0x7f0450071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39812/0 (socket says 172.21.15.7:39812) 2023-03-03T19:03:41.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045629f700 1 -- 172.21.15.7:0/1329061996 learned_addr learned my addr 172.21.15.7:0/1329061996 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:03:41.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f0455a9e700 1 --2- 172.21.15.7:0/1329061996 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 0x7f0450157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f0456aa0700 1 --2- 172.21.15.7:0/1329061996 >> v2:172.21.15.143:3300/0 conn(0x7f0450071d40 0x7f0450158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045629f700 1 -- 172.21.15.7:0/1329061996 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 msgr2=0x7f0450157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:03:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045629f700 1 --2- 172.21.15.7:0/1329061996 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 0x7f0450157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045629f700 1 -- 172.21.15.7:0/1329061996 >> v2:172.21.15.143:3300/0 conn(0x7f0450071d40 msgr2=0x7f0450158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:03:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045629f700 1 --2- 172.21.15.7:0/1329061996 >> v2:172.21.15.143:3300/0 conn(0x7f0450071d40 0x7f0450158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045629f700 1 -- 172.21.15.7:0/1329061996 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045006fc70 con 0x7f045014b0d0 2023-03-03T19:03:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045629f700 1 --2- 172.21.15.7:0/1329061996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f045014b0d0 0x7f0450071800 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7f04500a98d0 tx=0x7f0440009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f03f03efb2a602 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:03:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045529d700 1 -- 172.21.15.7:0/1329061996 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440011830 con 0x7f045014b0d0 2023-03-03T19:03:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045529d700 1 -- 172.21.15.7:0/1329061996 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0440011990 con 0x7f045014b0d0 2023-03-03T19:03:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.825+0000 7f045529d700 1 -- 172.21.15.7:0/1329061996 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440011d20 con 0x7f045014b0d0 2023-03-03T19:03:41.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.826+0000 7f0458503700 1 -- 172.21.15.7:0/1329061996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f045014b0d0 msgr2=0x7f0450071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:41.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.826+0000 7f0458503700 1 --2- 172.21.15.7:0/1329061996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f045014b0d0 0x7f0450071800 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7f04500a98d0 tx=0x7f0440009bd0 comp rx=0 tx=0).stop 2023-03-03T19:03:41.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.826+0000 7f0458503700 1 -- 172.21.15.7:0/1329061996 shutdown_connections 2023-03-03T19:03:41.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.826+0000 7f0458503700 1 --2- 172.21.15.7:0/1329061996 >> v2:172.21.15.143:3300/0 conn(0x7f0450071d40 0x7f0450158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:41.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.826+0000 7f0458503700 1 --2- 172.21.15.7:0/1329061996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f045014b0d0 0x7f0450071800 unknown :-1 s=CLOSED pgs=3377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:41.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.826+0000 7f0458503700 1 --2- 172.21.15.7:0/1329061996 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 0x7f0450157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:41.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.826+0000 7f0458503700 1 -- 172.21.15.7:0/1329061996 >> 172.21.15.7:0/1329061996 conn(0x7f04500c8900 msgr2=0x7f04500c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:03:41.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.826+0000 7f0458503700 1 -- 172.21.15.7:0/1329061996 shutdown_connections 2023-03-03T19:03:41.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.826+0000 7f0458503700 1 -- 172.21.15.7:0/1329061996 wait complete. 2023-03-03T19:03:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.828+0000 7f0458503700 1 Processor -- start 2023-03-03T19:03:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.828+0000 7f0458503700 1 -- start start 2023-03-03T19:03:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0458503700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0450071d40 0x7f04501ed9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0458503700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 0x7f0450157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0458503700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f045014b0d0 0x7f0450153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0458503700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04501ecfd0 con 0x7f0450071d40 2023-03-03T19:03:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0458503700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04501edf20 con 0x7f045014b0d0 2023-03-03T19:03:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0458503700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04501ed550 con 0x7f04500aad20 2023-03-03T19:03:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f045629f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0450071d40 0x7f04501ed9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f045629f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0450071d40 0x7f04501ed9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39816/0 (socket says 172.21.15.7:39816) 2023-03-03T19:03:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f045629f700 1 -- 172.21.15.7:0/367481764 learned_addr learned my addr 172.21.15.7:0/367481764 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:03:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0456aa0700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f045014b0d0 0x7f0450153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0455a9e700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 0x7f0450157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f045629f700 1 -- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 msgr2=0x7f0450157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f045629f700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 0x7f0450157210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f045629f700 1 -- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f045014b0d0 msgr2=0x7f0450153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f045629f700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f045014b0d0 0x7f0450153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f045629f700 1 -- 172.21.15.7:0/367481764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0440009550 con 0x7f0450071d40 2023-03-03T19:03:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0455a9e700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 0x7f0450157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:03:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0456aa0700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f045014b0d0 0x7f0450153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:03:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f045629f700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0450071d40 0x7f04501ed9e0 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7f0440000c00 tx=0x7f0440004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:03:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0446ffd700 1 -- 172.21.15.7:0/367481764 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440004300 con 0x7f0450071d40 2023-03-03T19:03:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0446ffd700 1 -- 172.21.15.7:0/367481764 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044001f070 con 0x7f0450071d40 2023-03-03T19:03:41.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0458503700 1 -- 172.21.15.7:0/367481764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045006f6b0 con 0x7f0450071d40 2023-03-03T19:03:41.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0446ffd700 1 -- 172.21.15.7:0/367481764 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04400036a0 con 0x7f0450071d40 2023-03-03T19:03:41.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.829+0000 7f0458503700 1 -- 172.21.15.7:0/367481764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0450155160 con 0x7f0450071d40 2023-03-03T19:03:41.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.830+0000 7f0458503700 1 -- 172.21.15.7:0/367481764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045006f990 con 0x7f0450071d40 2023-03-03T19:03:41.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.831+0000 7f0446ffd700 1 -- 172.21.15.7:0/367481764 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f044002a020 con 0x7f0450071d40 2023-03-03T19:03:41.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.831+0000 7f0446ffd700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04340682f0 0x7f043406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:41.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.831+0000 7f0446ffd700 1 -- 172.21.15.7:0/367481764 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0440087570 con 0x7f0450071d40 2023-03-03T19:03:41.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.832+0000 7f0455a9e700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04340682f0 0x7f043406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:41.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.832+0000 7f0455a9e700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04340682f0 0x7f043406a7a0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f04501e19c0 tx=0x7f044c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:03:41.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:41.836+0000 7f0446ffd700 1 -- 172.21.15.7:0/367481764 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0440053910 con 0x7f0450071d40 2023-03-03T19:03:42.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.074+0000 7f0458503700 1 -- 172.21.15.7:0/367481764 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04501df4d0 con 0x7f0450071d40 2023-03-03T19:03:42.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.074+0000 7f0446ffd700 1 -- 172.21.15.7:0/367481764 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0440056f30 con 0x7f0450071d40 2023-03-03T19:03:42.075 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:03:42.075 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:03:42.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 -- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04340682f0 msgr2=0x7f043406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:42.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04340682f0 0x7f043406a7a0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f04501e19c0 tx=0x7f044c008040 comp rx=0 tx=0).stop 2023-03-03T19:03:42.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 -- 172.21.15.7:0/367481764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0450071d40 msgr2=0x7f04501ed9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:42.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0450071d40 0x7f04501ed9e0 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7f0440000c00 tx=0x7f0440004750 comp rx=0 tx=0).stop 2023-03-03T19:03:42.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 -- 172.21.15.7:0/367481764 shutdown_connections 2023-03-03T19:03:42.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f045014b0d0 0x7f0450153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:42.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f04340682f0 0x7f043406a7a0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:42.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0450071d40 0x7f04501ed9e0 unknown :-1 s=CLOSED pgs=3378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:42.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 --2- 172.21.15.7:0/367481764 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f04500aad20 0x7f0450157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:42.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 -- 172.21.15.7:0/367481764 >> 172.21.15.7:0/367481764 conn(0x7f04500c8900 msgr2=0x7f04501584b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:03:42.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 -- 172.21.15.7:0/367481764 shutdown_connections 2023-03-03T19:03:42.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:42.076+0000 7f0458503700 1 -- 172.21.15.7:0/367481764 wait complete. 2023-03-03T19:03:42.078 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:03:42.088 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T19:03:44.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:03:44.526 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:03:53.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:03:53.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.228+0000 7f796fe69700 1 Processor -- start 2023-03-03T19:03:53.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.228+0000 7f796fe69700 1 -- start start 2023-03-03T19:03:53.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.229+0000 7f796fe69700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f79680c00e0 0x7f79680c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:53.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.229+0000 7f796fe69700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79680c7310 con 0x7f79680caa30 2023-03-03T19:03:53.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.229+0000 7f796fe69700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79680bfd10 con 0x7f7968154ff0 2023-03-03T19:03:53.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.229+0000 7f796fe69700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7968155430 con 0x7f79680c00e0 2023-03-03T19:03:53.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.229+0000 7f796dc05700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f79680caa30 0x7f79680c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33816/0 (socket says 172.21.15.7:33816) 2023-03-03T19:03:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.229+0000 7f796dc05700 1 -- 172.21.15.7:0/2290414551 learned_addr learned my addr 172.21.15.7:0/2290414551 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:03:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.230+0000 7f796d404700 1 --2- 172.21.15.7:0/2290414551 >> v2:172.21.15.143:3300/0 conn(0x7f79680c00e0 0x7f79680c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.230+0000 7f796cc03700 1 -- 172.21.15.7:0/2290414551 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845808379 0 0) 0x7f79680c7310 con 0x7f79680caa30 2023-03-03T19:03:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.230+0000 7f796cc03700 1 -- 172.21.15.7:0/2290414551 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79540035b0 con 0x7f79680caa30 2023-03-03T19:03:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.230+0000 7f796cc03700 1 -- 172.21.15.7:0/2290414551 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013794819 0 0) 0x7f79680bfd10 con 0x7f7968154ff0 2023-03-03T19:03:53.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.230+0000 7f796cc03700 1 -- 172.21.15.7:0/2290414551 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79680c7310 con 0x7f7968154ff0 2023-03-03T19:03:53.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.230+0000 7f796d404700 1 -- 172.21.15.7:0/2290414551 >> v1:172.21.15.143:6789/0 conn(0x7f7968154ff0 legacy=0x7f796815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:53.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.230+0000 7f796d404700 1 -- 172.21.15.7:0/2290414551 >> v1:172.21.15.7:6789/0 conn(0x7f79680caa30 legacy=0x7f79680c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:53.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.230+0000 7f796d404700 1 -- 172.21.15.7:0/2290414551 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680bfa30 con 0x7f79680c00e0 2023-03-03T19:03:53.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.230+0000 7f796d404700 1 --2- 172.21.15.7:0/2290414551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79680c00e0 0x7f79680c5c60 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f795c009820 tx=0x7f795c009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aae82316b1116480 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:03:53.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.230+0000 7f796cc03700 1 -- 172.21.15.7:0/2290414551 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c005e30 con 0x7f79680c00e0 2023-03-03T19:03:53.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.231+0000 7f796cc03700 1 -- 172.21.15.7:0/2290414551 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795c006030 con 0x7f79680c00e0 2023-03-03T19:03:53.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.231+0000 7f796cc03700 1 -- 172.21.15.7:0/2290414551 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c0063c0 con 0x7f79680c00e0 2023-03-03T19:03:53.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.231+0000 7f796fe69700 1 -- 172.21.15.7:0/2290414551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79680c00e0 msgr2=0x7f79680c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:53.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.231+0000 7f796fe69700 1 --2- 172.21.15.7:0/2290414551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79680c00e0 0x7f79680c5c60 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f795c009820 tx=0x7f795c009b30 comp rx=0 tx=0).stop 2023-03-03T19:03:53.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.231+0000 7f796fe69700 1 -- 172.21.15.7:0/2290414551 shutdown_connections 2023-03-03T19:03:53.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.231+0000 7f796fe69700 1 --2- 172.21.15.7:0/2290414551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79680c00e0 0x7f79680c5c60 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:53.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.231+0000 7f796fe69700 1 -- 172.21.15.7:0/2290414551 >> 172.21.15.7:0/2290414551 conn(0x7f7968066e40 msgr2=0x7f7968071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:03:53.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.231+0000 7f796fe69700 1 -- 172.21.15.7:0/2290414551 shutdown_connections 2023-03-03T19:03:53.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.231+0000 7f796fe69700 1 -- 172.21.15.7:0/2290414551 wait complete. 2023-03-03T19:03:53.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.232+0000 7f796fe69700 1 Processor -- start 2023-03-03T19:03:53.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.232+0000 7f796fe69700 1 -- start start 2023-03-03T19:03:53.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796fe69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79680c00e0 0x7f79680c25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:53.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796fe69700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79680caa30 0x7f796814ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:53.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796fe69700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7968154ff0 0x7f79681ed8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:53.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796fe69700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79681ed130 con 0x7f7968154ff0 2023-03-03T19:03:53.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796fe69700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79680c1ac0 con 0x7f79680caa30 2023-03-03T19:03:53.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796fe69700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7968158530 con 0x7f79680c00e0 2023-03-03T19:03:53.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796e406700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7968154ff0 0x7f79681ed8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:53.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796e406700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7968154ff0 0x7f79681ed8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39846/0 (socket says 172.21.15.7:39846) 2023-03-03T19:03:53.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796e406700 1 -- 172.21.15.7:0/291055558 learned_addr learned my addr 172.21.15.7:0/291055558 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:03:53.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796dc05700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79680c00e0 0x7f79680c25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:53.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.233+0000 7f796d404700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79680caa30 0x7f796814ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:53.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f796e406700 1 -- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79680c00e0 msgr2=0x7f79680c25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:53.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f796e406700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79680c00e0 0x7f79680c25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:53.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f796e406700 1 -- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79680caa30 msgr2=0x7f796814ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:53.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f796e406700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79680caa30 0x7f796814ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:53.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f796e406700 1 -- 172.21.15.7:0/291055558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c009430 con 0x7f7968154ff0 2023-03-03T19:03:53.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f796d404700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79680caa30 0x7f796814ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:03:53.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f796dc05700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79680c00e0 0x7f79680c25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:03:53.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f796e406700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7968154ff0 0x7f79681ed8a0 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7f796000cc20 tx=0x7f796000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:03:53.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f795a7fc700 1 -- 172.21.15.7:0/291055558 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796000ac20 con 0x7f7968154ff0 2023-03-03T19:03:53.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f795a7fc700 1 -- 172.21.15.7:0/291055558 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796000ad80 con 0x7f7968154ff0 2023-03-03T19:03:53.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f795a7fc700 1 -- 172.21.15.7:0/291055558 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960003c20 con 0x7f7968154ff0 2023-03-03T19:03:53.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f796fe69700 1 -- 172.21.15.7:0/291055558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79681547f0 con 0x7f7968154ff0 2023-03-03T19:03:53.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.234+0000 7f796fe69700 1 -- 172.21.15.7:0/291055558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7968154510 con 0x7f7968154ff0 2023-03-03T19:03:53.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.235+0000 7f796fe69700 1 -- 172.21.15.7:0/291055558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7968154ad0 con 0x7f7968154ff0 2023-03-03T19:03:53.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.236+0000 7f795a7fc700 1 -- 172.21.15.7:0/291055558 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7960010050 con 0x7f7968154ff0 2023-03-03T19:03:53.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.236+0000 7f795a7fc700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f794c0682f0 0x7f794c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:03:53.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.236+0000 7f795a7fc700 1 -- 172.21.15.7:0/291055558 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f79600862a0 con 0x7f7968154ff0 2023-03-03T19:03:53.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.236+0000 7f796dc05700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f794c0682f0 0x7f794c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:03:53.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.237+0000 7f796dc05700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f794c0682f0 0x7f794c06a7a0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f79640011e0 tx=0x7f7964009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:03:53.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.240+0000 7f795a7fc700 1 -- 172.21.15.7:0/291055558 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7960052550 con 0x7f7968154ff0 2023-03-03T19:03:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.480+0000 7f796fe69700 1 -- 172.21.15.7:0/291055558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79680c0f90 con 0x7f7968154ff0 2023-03-03T19:03:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.480+0000 7f795a7fc700 1 -- 172.21.15.7:0/291055558 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f796005b090 con 0x7f7968154ff0 2023-03-03T19:03:53.481 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:03:53.481 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:03:53.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 -- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f794c0682f0 msgr2=0x7f794c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:53.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f794c0682f0 0x7f794c06a7a0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f79640011e0 tx=0x7f7964009040 comp rx=0 tx=0).stop 2023-03-03T19:03:53.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 -- 172.21.15.7:0/291055558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7968154ff0 msgr2=0x7f79681ed8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:03:53.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7968154ff0 0x7f79681ed8a0 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7f796000cc20 tx=0x7f796000a060 comp rx=0 tx=0).stop 2023-03-03T19:03:53.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 -- 172.21.15.7:0/291055558 shutdown_connections 2023-03-03T19:03:53.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f794c0682f0 0x7f794c06a7a0 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:53.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7968154ff0 0x7f79681ed8a0 unknown :-1 s=CLOSED pgs=3380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:53.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79680c00e0 0x7f79680c25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:53.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 --2- 172.21.15.7:0/291055558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79680caa30 0x7f796814ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:03:53.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 -- 172.21.15.7:0/291055558 >> 172.21.15.7:0/291055558 conn(0x7f7968066e40 msgr2=0x7f7968153740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:03:53.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.482+0000 7f796fe69700 1 -- 172.21.15.7:0/291055558 shutdown_connections 2023-03-03T19:03:53.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:03:53.483+0000 7f796fe69700 1 -- 172.21.15.7:0/291055558 wait complete. 2023-03-03T19:03:53.484 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:03:53.495 INFO:tasks.mds_thrash.fs.[b]:waiting for 79 secs before thrashing 2023-03-03T19:04:12.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:04:12.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.714+0000 7f1d7154a700 1 Processor -- start 2023-03-03T19:04:12.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.714+0000 7f1d7154a700 1 -- start start 2023-03-03T19:04:12.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.714+0000 7f1d7154a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1d6c0c00e0 0x7f1d6c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:04:12.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.714+0000 7f1d7154a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c154ff0 0x7f1d6c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:04:12.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.714+0000 7f1d7154a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6c0c7310 con 0x7f1d6c0caa30 2023-03-03T19:04:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.714+0000 7f1d7154a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c0bfd10 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.714+0000 7f1d7154a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0bfa30 con 0x7f1d6c154ff0 2023-03-03T19:04:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d6b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c154ff0 0x7f1d6c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:04:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d6a7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1d6c0c00e0 0x7f1d6c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:04:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d6affd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1d6c0caa30 0x7f1d6c0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:37358/0 (socket says 172.21.15.7:37358) 2023-03-03T19:04:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d6a7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1d6c0c00e0 0x7f1d6c0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:35634/0 (socket says 172.21.15.7:35634) 2023-03-03T19:04:12.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d6b7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c154ff0 0x7f1d6c0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:47934/0 (socket says 172.21.15.7:47934) 2023-03-03T19:04:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d6affd700 1 -- 172.21.15.7:0/1736625006 learned_addr learned my addr 172.21.15.7:0/1736625006 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:04:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d6a7fc700 1 -- 172.21.15.7:0/1736625006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c154ff0 msgr2=0x7f1d6c0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:04:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d6a7fc700 1 --2- 172.21.15.7:0/1736625006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c154ff0 0x7f1d6c0cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:04:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d6a7fc700 1 -- 172.21.15.7:0/1736625006 >> v1:172.21.15.143:6789/0 conn(0x7f1d6c0caa30 legacy=0x7f1d6c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:04:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d69ffb700 1 -- 172.21.15.7:0/1736625006 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972416682 0 0) 0x7f1d6c0c7310 con 0x7f1d6c0caa30 2023-03-03T19:04:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.715+0000 7f1d6a7fc700 1 -- 172.21.15.7:0/1736625006 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c154510 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d6a7fc700 1 --2- 172.21.15.7:0/1736625006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d6c0c00e0 0x7f1d6c0c5c60 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009820 tx=0x7f1d60009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5513cfa13afb070d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:04:12.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d69ffb700 1 -- 172.21.15.7:0/1736625006 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d60005e30 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d69ffb700 1 -- 172.21.15.7:0/1736625006 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d60006030 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d69ffb700 1 -- 172.21.15.7:0/1736625006 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d600063c0 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d7154a700 1 -- 172.21.15.7:0/1736625006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d6c0c00e0 msgr2=0x7f1d6c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:04:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d7154a700 1 --2- 172.21.15.7:0/1736625006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d6c0c00e0 0x7f1d6c0c5c60 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009820 tx=0x7f1d60009b30 comp rx=0 tx=0).stop 2023-03-03T19:04:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d7154a700 1 -- 172.21.15.7:0/1736625006 shutdown_connections 2023-03-03T19:04:12.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d7154a700 1 --2- 172.21.15.7:0/1736625006 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c154ff0 0x7f1d6c0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:04:12.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d7154a700 1 --2- 172.21.15.7:0/1736625006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d6c0c00e0 0x7f1d6c0c5c60 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:04:12.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d7154a700 1 -- 172.21.15.7:0/1736625006 >> 172.21.15.7:0/1736625006 conn(0x7f1d6c066e40 msgr2=0x7f1d6c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:04:12.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d7154a700 1 -- 172.21.15.7:0/1736625006 shutdown_connections 2023-03-03T19:04:12.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.716+0000 7f1d7154a700 1 -- 172.21.15.7:0/1736625006 wait complete. 2023-03-03T19:04:12.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.718+0000 7f1d7154a700 1 Processor -- start 2023-03-03T19:04:12.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.718+0000 7f1d7154a700 1 -- start start 2023-03-03T19:04:12.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.718+0000 7f1d7154a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d6c0c00e0 0x7f1d6c1ea120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:04:12.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.718+0000 7f1d7154a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c0caa30 0x7f1d6c0c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:04:12.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.718+0000 7f1d7154a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d6c154ff0 0x7f1d6c0c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:04:12.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.718+0000 7f1d7154a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0c3db0 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.718+0000 7f1d7154a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c158520 con 0x7f1d6c154ff0 2023-03-03T19:04:12.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.718+0000 7f1d7154a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c1ea660 con 0x7f1d6c0caa30 2023-03-03T19:04:12.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d6c0c00e0 0x7f1d6c1ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:04:12.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d6c0c00e0 0x7f1d6c1ea120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39864/0 (socket says 172.21.15.7:39864) 2023-03-03T19:04:12.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6affd700 1 -- 172.21.15.7:0/3306779996 learned_addr learned my addr 172.21.15.7:0/3306779996 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:04:12.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6a7fc700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c0caa30 0x7f1d6c0c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:04:12.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6b7fe700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d6c154ff0 0x7f1d6c0c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:04:12.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6affd700 1 -- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c0caa30 msgr2=0x7f1d6c0c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:04:12.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6affd700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c0caa30 0x7f1d6c0c23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:04:12.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6affd700 1 -- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d6c154ff0 msgr2=0x7f1d6c0c41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:04:12.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6affd700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d6c154ff0 0x7f1d6c0c41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:04:12.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6affd700 1 -- 172.21.15.7:0/3306779996 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d60009430 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6b7fe700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d6c154ff0 0x7f1d6c0c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:04:12.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d6affd700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d6c0c00e0 0x7f1d6c1ea120 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0011e0 tx=0x7f1d5c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:04:12.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d537fe700 1 -- 172.21.15.7:0/3306779996 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c004c80 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d537fe700 1 -- 172.21.15.7:0/3306779996 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5c004e00 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d7154a700 1 -- 172.21.15.7:0/3306779996 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c154ad0 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d537fe700 1 -- 172.21.15.7:0/3306779996 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c007310 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.719+0000 7f1d7154a700 1 -- 172.21.15.7:0/3306779996 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c1547f0 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.721+0000 7f1d7154a700 1 -- 172.21.15.7:0/3306779996 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6c0bf650 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.721+0000 7f1d537fe700 1 -- 172.21.15.7:0/3306779996 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1d5c01a020 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.721+0000 7f1d537fe700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d580682f0 0x7f1d5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:04:12.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.721+0000 7f1d537fe700 1 -- 172.21.15.7:0/3306779996 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1d5c085d30 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.722+0000 7f1d6a7fc700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d580682f0 0x7f1d5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:04:12.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.722+0000 7f1d6a7fc700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d580682f0 0x7f1d5806a7a0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f1d60000c00 tx=0x7f1d60008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:04:12.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.726+0000 7f1d537fe700 1 -- 172.21.15.7:0/3306779996 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1d5c051fe0 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.966+0000 7f1d7154a700 1 -- 172.21.15.7:0/3306779996 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d6c0c3ef0 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.967+0000 7f1d537fe700 1 -- 172.21.15.7:0/3306779996 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1d6c0c3ef0 con 0x7f1d6c0c00e0 2023-03-03T19:04:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:04:12.967 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:04:12.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 -- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d580682f0 msgr2=0x7f1d5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:04:12.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d580682f0 0x7f1d5806a7a0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f1d60000c00 tx=0x7f1d60008070 comp rx=0 tx=0).stop 2023-03-03T19:04:12.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 -- 172.21.15.7:0/3306779996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d6c0c00e0 msgr2=0x7f1d6c1ea120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:04:12.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d6c0c00e0 0x7f1d6c1ea120 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0011e0 tx=0x7f1d5c00ca30 comp rx=0 tx=0).stop 2023-03-03T19:04:12.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 -- 172.21.15.7:0/3306779996 shutdown_connections 2023-03-03T19:04:12.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1d6c154ff0 0x7f1d6c0c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:04:12.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1d580682f0 0x7f1d5806a7a0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:04:12.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d6c0c00e0 0x7f1d6c1ea120 unknown :-1 s=CLOSED pgs=3381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:04:12.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 --2- 172.21.15.7:0/3306779996 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1d6c0caa30 0x7f1d6c0c23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:04:12.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 -- 172.21.15.7:0/3306779996 >> 172.21.15.7:0/3306779996 conn(0x7f1d6c066e40 msgr2=0x7f1d6c1536d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:04:12.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 -- 172.21.15.7:0/3306779996 shutdown_connections 2023-03-03T19:04:12.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:04:12.969+0000 7f1d7154a700 1 -- 172.21.15.7:0/3306779996 wait complete. 2023-03-03T19:04:12.971 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:04:12.981 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T19:04:14.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:04:14.606 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:04:44.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:04:44.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:05:12.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:05:12.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.635+0000 7fbe04f90700 1 Processor -- start 2023-03-03T19:05:12.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.635+0000 7fbe04f90700 1 -- start start 2023-03-03T19:05:12.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.636+0000 7fbe04f90700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe0014b020 0x7fbe000717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:12.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.636+0000 7fbe04f90700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe000728b0 con 0x7fbe00071d20 2023-03-03T19:05:12.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.636+0000 7fbe04f90700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe00155070 con 0x7fbe000aad20 2023-03-03T19:05:12.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.636+0000 7fbe04f90700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0006f5b0 con 0x7fbe0014b020 2023-03-03T19:05:12.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.636+0000 7fbdfed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbe00071d20 0x7fbe001583b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33880/0 (socket says 172.21.15.7:33880) 2023-03-03T19:05:12.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.636+0000 7fbdfed9d700 1 -- 172.21.15.7:0/2178493643 learned_addr learned my addr 172.21.15.7:0/2178493643 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:05:12.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.636+0000 7fbdfe59c700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe0014b020 0x7fbe000717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:12.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.636+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440413056 0 0) 0x7fbe000728b0 con 0x7fbe00071d20 2023-03-03T19:05:12.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde80035b0 con 0x7fbe00071d20 2023-03-03T19:05:12.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558617325 0 0) 0x7fbe00155070 con 0x7fbe000aad20 2023-03-03T19:05:12.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe000728b0 con 0x7fbe000aad20 2023-03-03T19:05:12.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3295094498 0 0) 0x7fbde80035b0 con 0x7fbe00071d20 2023-03-03T19:05:12.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe0014b020 msgr2=0x7fbe000717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:05:12.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe0014b020 0x7fbe000717e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:12.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 >> v1:172.21.15.143:6789/0 conn(0x7fbe000aad20 legacy=0x7fbe00157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:12.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0006f890 con 0x7fbe00071d20 2023-03-03T19:05:12.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fbdf80019f0 con 0x7fbe00071d20 2023-03-03T19:05:12.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 >> v1:172.21.15.7:6789/0 conn(0x7fbe00071d20 legacy=0x7fbe001583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:12.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe0014b020 0x7fbde8003c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:12.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde80041e0 0x7fbde8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:12.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbde8004de0 0x7fbde8005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:12.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0006f5b0 con 0x7fbde80041e0 2023-03-03T19:05:12.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde800bcc0 con 0x7fbde8004de0 2023-03-03T19:05:12.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde800be80 con 0x7fbe0014b020 2023-03-03T19:05:12.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfdd9b700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde80041e0 0x7fbde8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:12.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfe59c700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe0014b020 0x7fbde8003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:12.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfed9d700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbde8004de0 0x7fbde8005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:12.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfdd9b700 1 -- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe0014b020 msgr2=0x7fbde8003c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:05:12.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfdd9b700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbe0014b020 0x7fbde8003c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:12.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfdd9b700 1 -- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbde8004de0 msgr2=0x7fbde8005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:05:12.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfdd9b700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbde8004de0 0x7fbde8005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:12.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.637+0000 7fbdfdd9b700 1 -- 172.21.15.7:0/2178493643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde80035b0 con 0x7fbde80041e0 2023-03-03T19:05:12.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbdfdd9b700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde80041e0 0x7fbde8004850 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7fbe000ab1a0 tx=0x7fbde8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d7dbb413d5a25 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:05:12.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0001430 con 0x7fbde80041e0 2023-03-03T19:05:12.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf00052d0 con 0x7fbde80041e0 2023-03-03T19:05:12.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbdfd59a700 1 -- 172.21.15.7:0/2178493643 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0001d40 con 0x7fbde80041e0 2023-03-03T19:05:12.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbe04f90700 1 -- 172.21.15.7:0/2178493643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde80041e0 msgr2=0x7fbde8004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:12.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbe04f90700 1 --2- 172.21.15.7:0/2178493643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbde80041e0 0x7fbde8004850 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7fbe000ab1a0 tx=0x7fbde8005880 comp rx=0 tx=0).stop 2023-03-03T19:05:12.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbdfed9d700 1 -- 172.21.15.7:0/2178493643 reap_dead start 2023-03-03T19:05:12.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbe04f90700 1 -- 172.21.15.7:0/2178493643 shutdown_connections 2023-03-03T19:05:12.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbe04f90700 1 -- 172.21.15.7:0/2178493643 >> 172.21.15.7:0/2178493643 conn(0x7fbe000c8900 msgr2=0x7fbe000c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:05:12.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbe04f90700 1 -- 172.21.15.7:0/2178493643 shutdown_connections 2023-03-03T19:05:12.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.638+0000 7fbe04f90700 1 -- 172.21.15.7:0/2178493643 wait complete. 2023-03-03T19:05:12.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.640+0000 7fbe04f90700 1 Processor -- start 2023-03-03T19:05:12.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.640+0000 7fbe04f90700 1 -- start start 2023-03-03T19:05:12.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.640+0000 7fbe04f90700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde80041e0 0x7fbe00156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:12.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.640+0000 7fbe04f90700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe0014ddf0 0x7fbe001536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:12.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.640+0000 7fbe04f90700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe00144b20 0x7fbe001e9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:12.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.640+0000 7fbe04f90700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0014e260 con 0x7fbe0014ddf0 2023-03-03T19:05:12.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.640+0000 7fbe04f90700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0014c9a0 con 0x7fbe00144b20 2023-03-03T19:05:12.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.640+0000 7fbe04f90700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe001edc10 con 0x7fbde80041e0 2023-03-03T19:05:12.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.640+0000 7fbdfdd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe0014ddf0 0x7fbe001536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:12.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfdd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe0014ddf0 0x7fbe001536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39910/0 (socket says 172.21.15.7:39910) 2023-03-03T19:05:12.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfdd9b700 1 -- 172.21.15.7:0/1614973695 learned_addr learned my addr 172.21.15.7:0/1614973695 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:05:12.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfe59c700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde80041e0 0x7fbe00156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:12.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfed9d700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe00144b20 0x7fbe001e9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:12.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfdd9b700 1 -- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde80041e0 msgr2=0x7fbe00156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:12.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfdd9b700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde80041e0 0x7fbe00156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:12.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfdd9b700 1 -- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe00144b20 msgr2=0x7fbe001e9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:12.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfdd9b700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe00144b20 0x7fbe001e9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:12.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfdd9b700 1 -- 172.21.15.7:0/1614973695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe00155350 con 0x7fbe0014ddf0 2023-03-03T19:05:12.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfed9d700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe00144b20 0x7fbe001e9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:05:12.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdfdd9b700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe0014ddf0 0x7fbe001536a0 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0006b10 tx=0x7fbdf00012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:05:12.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdeeffd700 1 -- 172.21.15.7:0/1614973695 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0001d40 con 0x7fbe0014ddf0 2023-03-03T19:05:12.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdeeffd700 1 -- 172.21.15.7:0/1614973695 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf00013d0 con 0x7fbe0014ddf0 2023-03-03T19:05:12.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbe04f90700 1 -- 172.21.15.7:0/1614973695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0006fb70 con 0x7fbe0014ddf0 2023-03-03T19:05:12.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbdeeffd700 1 -- 172.21.15.7:0/1614973695 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0007260 con 0x7fbe0014ddf0 2023-03-03T19:05:12.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.641+0000 7fbe04f90700 1 -- 172.21.15.7:0/1614973695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe000aa030 con 0x7fbe0014ddf0 2023-03-03T19:05:12.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.642+0000 7fbe04f90700 1 -- 172.21.15.7:0/1614973695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0014b8a0 con 0x7fbe0014ddf0 2023-03-03T19:05:12.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.643+0000 7fbdeeffd700 1 -- 172.21.15.7:0/1614973695 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbdf002e070 con 0x7fbe0014ddf0 2023-03-03T19:05:12.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.643+0000 7fbdeeffd700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbde00682f0 0x7fbde006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:12.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.643+0000 7fbdeeffd700 1 -- 172.21.15.7:0/1614973695 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbdf0087240 con 0x7fbe0014ddf0 2023-03-03T19:05:12.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.643+0000 7fbdfe59c700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbde00682f0 0x7fbde006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:12.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.644+0000 7fbdfe59c700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbde00682f0 0x7fbde006a7a0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009b80 tx=0x7fbdf4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:05:12.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.648+0000 7fbdeeffd700 1 -- 172.21.15.7:0/1614973695 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbe0014b8a0 con 0x7fbe0014ddf0 2023-03-03T19:05:12.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.887+0000 7fbe04f90700 1 -- 172.21.15.7:0/1614973695 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe0014b880 con 0x7fbe0014ddf0 2023-03-03T19:05:12.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.887+0000 7fbdeeffd700 1 -- 172.21.15.7:0/1614973695 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbdf00535e0 con 0x7fbe0014ddf0 2023-03-03T19:05:12.888 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:05:12.888 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:05:12.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.889+0000 7fbe04f90700 1 -- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbde00682f0 msgr2=0x7fbde006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:12.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.889+0000 7fbe04f90700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbde00682f0 0x7fbde006a7a0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009b80 tx=0x7fbdf4009210 comp rx=0 tx=0).stop 2023-03-03T19:05:12.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.889+0000 7fbe04f90700 1 -- 172.21.15.7:0/1614973695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe0014ddf0 msgr2=0x7fbe001536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:12.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.889+0000 7fbe04f90700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe0014ddf0 0x7fbe001536a0 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0006b10 tx=0x7fbdf00012d0 comp rx=0 tx=0).stop 2023-03-03T19:05:12.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.889+0000 7fbe04f90700 1 -- 172.21.15.7:0/1614973695 shutdown_connections 2023-03-03T19:05:12.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.889+0000 7fbe04f90700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe00144b20 0x7fbe001e9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:12.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.889+0000 7fbe04f90700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbde80041e0 0x7fbe00156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:12.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.889+0000 7fbe04f90700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbde00682f0 0x7fbde006a7a0 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:12.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.889+0000 7fbe04f90700 1 --2- 172.21.15.7:0/1614973695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbe0014ddf0 0x7fbe001536a0 unknown :-1 s=CLOSED pgs=3384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:12.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.889+0000 7fbe04f90700 1 -- 172.21.15.7:0/1614973695 >> 172.21.15.7:0/1614973695 conn(0x7fbe000c8900 msgr2=0x7fbe001547c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:05:12.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.890+0000 7fbe04f90700 1 -- 172.21.15.7:0/1614973695 shutdown_connections 2023-03-03T19:05:12.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:12.890+0000 7fbe04f90700 1 -- 172.21.15.7:0/1614973695 wait complete. 2023-03-03T19:05:12.891 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:05:12.902 INFO:tasks.mds_thrash.fs.[b]:waiting for 96 secs before thrashing 2023-03-03T19:05:14.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:05:14.733 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:05:18.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:05:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.122+0000 7f3d00b55700 1 Processor -- start 2023-03-03T19:05:19.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.122+0000 7f3d00b55700 1 -- start start 2023-03-03T19:05:19.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3d00b55700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3cfc14f9d0 0x7f3cfc156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:19.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3d00b55700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc0c2f30 con 0x7f3cfc0abe20 2023-03-03T19:05:19.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3d00b55700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc0c31d0 con 0x7f3cfc14fda0 2023-03-03T19:05:19.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3d00b55700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc1524a0 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3cfa59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3cfc14f9d0 0x7f3cfc156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:19.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3cfad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3cfc0abe20 0x7f3cfc1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33904/0 (socket says 172.21.15.7:33904) 2023-03-03T19:05:19.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3cfad9d700 1 -- 172.21.15.7:0/4046643229 learned_addr learned my addr 172.21.15.7:0/4046643229 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:05:19.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3cfa59c700 1 -- 172.21.15.7:0/4046643229 >> v1:172.21.15.143:6789/0 conn(0x7f3cfc14fda0 legacy=0x7f3cfc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:19.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3cfa59c700 1 -- 172.21.15.7:0/4046643229 >> v1:172.21.15.7:6789/0 conn(0x7f3cfc0abe20 legacy=0x7f3cfc1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:19.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3cf959a700 1 -- 172.21.15.7:0/4046643229 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863729427 0 0) 0x7f3cfc0c2f30 con 0x7f3cfc0abe20 2023-03-03T19:05:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3cfa59c700 1 -- 172.21.15.7:0/4046643229 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc0c34b0 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.123+0000 7f3cfa59c700 1 --2- 172.21.15.7:0/4046643229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 0x7f3cfc156430 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009840 tx=0x7f3cec009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932b5c703ad8ac86 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:05:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.124+0000 7f3cf959a700 1 -- 172.21.15.7:0/4046643229 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec011830 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.124+0000 7f3cf959a700 1 -- 172.21.15.7:0/4046643229 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec011990 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.124+0000 7f3cf959a700 1 -- 172.21.15.7:0/4046643229 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec011d20 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.124+0000 7f3d00b55700 1 -- 172.21.15.7:0/4046643229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 msgr2=0x7f3cfc156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.124+0000 7f3d00b55700 1 --2- 172.21.15.7:0/4046643229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 0x7f3cfc156430 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009840 tx=0x7f3cec009b50 comp rx=0 tx=0).stop 2023-03-03T19:05:19.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.124+0000 7f3d00b55700 1 -- 172.21.15.7:0/4046643229 shutdown_connections 2023-03-03T19:05:19.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.124+0000 7f3d00b55700 1 --2- 172.21.15.7:0/4046643229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 0x7f3cfc156430 unknown :-1 s=CLOSED pgs=3386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:19.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.124+0000 7f3d00b55700 1 -- 172.21.15.7:0/4046643229 >> 172.21.15.7:0/4046643229 conn(0x7f3cfc0c8900 msgr2=0x7f3cfc0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:05:19.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.124+0000 7f3d00b55700 1 -- 172.21.15.7:0/4046643229 shutdown_connections 2023-03-03T19:05:19.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.124+0000 7f3d00b55700 1 -- 172.21.15.7:0/4046643229 wait complete. 2023-03-03T19:05:19.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.125+0000 7f3d00b55700 1 Processor -- start 2023-03-03T19:05:19.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3d00b55700 1 -- start start 2023-03-03T19:05:19.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3d00b55700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cfc0abe20 0x7f3cfc14a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:19.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3d00b55700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 0x7f3cfc14c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:19.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3d00b55700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cfc14fda0 0x7f3cfc14cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:19.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3d00b55700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc147600 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3d00b55700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc147850 con 0x7f3cfc0abe20 2023-03-03T19:05:19.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3d00b55700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc149260 con 0x7f3cfc14fda0 2023-03-03T19:05:19.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3cf9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 0x7f3cfc14c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:19.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3cf9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 0x7f3cfc14c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39928/0 (socket says 172.21.15.7:39928) 2023-03-03T19:05:19.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3cf9d9b700 1 -- 172.21.15.7:0/1695719151 learned_addr learned my addr 172.21.15.7:0/1695719151 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:05:19.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3cfa59c700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cfc0abe20 0x7f3cfc14a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:19.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.126+0000 7f3cfad9d700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cfc14fda0 0x7f3cfc14cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:19.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3cf9d9b700 1 -- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cfc14fda0 msgr2=0x7f3cfc14cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:19.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3cf9d9b700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cfc14fda0 0x7f3cfc14cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:19.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3cf9d9b700 1 -- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cfc0abe20 msgr2=0x7f3cfc14a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:19.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3cf9d9b700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cfc0abe20 0x7f3cfc14a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:19.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3cf9d9b700 1 -- 172.21.15.7:0/1695719151 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec0094a0 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3cf9d9b700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 0x7f3cfc14c6d0 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000c4b0 tx=0x7f3cf000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:05:19.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3ceaffd700 1 -- 172.21.15.7:0/1695719151 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf00052d0 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3ceaffd700 1 -- 172.21.15.7:0/1695719151 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf0005430 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3d00b55700 1 -- 172.21.15.7:0/1695719151 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc0ac350 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3ceaffd700 1 -- 172.21.15.7:0/1695719151 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf00038c0 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.127+0000 7f3d00b55700 1 -- 172.21.15.7:0/1695719151 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc0ab630 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.128+0000 7f3d00b55700 1 -- 172.21.15.7:0/1695719151 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc071fa0 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.129+0000 7f3ceaffd700 1 -- 172.21.15.7:0/1695719151 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3cf0019030 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.129+0000 7f3ceaffd700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cdc0682f0 0x7f3cdc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.129+0000 7f3ceaffd700 1 -- 172.21.15.7:0/1695719151 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3cf00853c0 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.129+0000 7f3cfa59c700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cdc0682f0 0x7f3cdc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:19.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.130+0000 7f3cfa59c700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cdc0682f0 0x7f3cdc06a7a0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0090b0 tx=0x7f3cec01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:05:19.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.133+0000 7f3ceaffd700 1 -- 172.21.15.7:0/1695719151 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3cf0051700 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.370+0000 7f3d00b55700 1 -- 172.21.15.7:0/1695719151 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cfc074bb0 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.370+0000 7f3ceaffd700 1 -- 172.21.15.7:0/1695719151 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3cf0054d20 con 0x7f3cfc14f9d0 2023-03-03T19:05:19.371 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:05:19.371 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:05:19.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.372+0000 7f3d00b55700 1 -- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cdc0682f0 msgr2=0x7f3cdc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:19.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.372+0000 7f3d00b55700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cdc0682f0 0x7f3cdc06a7a0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f3cec0090b0 tx=0x7f3cec01c040 comp rx=0 tx=0).stop 2023-03-03T19:05:19.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.373+0000 7f3d00b55700 1 -- 172.21.15.7:0/1695719151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 msgr2=0x7f3cfc14c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:19.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.373+0000 7f3d00b55700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 0x7f3cfc14c6d0 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7f3cf000c4b0 tx=0x7f3cf000c7c0 comp rx=0 tx=0).stop 2023-03-03T19:05:19.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.373+0000 7f3d00b55700 1 -- 172.21.15.7:0/1695719151 shutdown_connections 2023-03-03T19:05:19.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.373+0000 7f3d00b55700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3cfc14fda0 0x7f3cfc14cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:19.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.373+0000 7f3d00b55700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3cfc0abe20 0x7f3cfc14a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:19.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.373+0000 7f3d00b55700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3cdc0682f0 0x7f3cdc06a7a0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:19.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.373+0000 7f3d00b55700 1 --2- 172.21.15.7:0/1695719151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3cfc14f9d0 0x7f3cfc14c6d0 unknown :-1 s=CLOSED pgs=3387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:19.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.373+0000 7f3d00b55700 1 -- 172.21.15.7:0/1695719151 >> 172.21.15.7:0/1695719151 conn(0x7f3cfc0c8900 msgr2=0x7f3cfc0c1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:05:19.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.373+0000 7f3d00b55700 1 -- 172.21.15.7:0/1695719151 shutdown_connections 2023-03-03T19:05:19.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:19.373+0000 7f3d00b55700 1 -- 172.21.15.7:0/1695719151 wait complete. 2023-03-03T19:05:19.374 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:05:19.385 INFO:tasks.mds_thrash.fs.[a]:waiting for 29 secs before thrashing 2023-03-03T19:05:44.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:05:44.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:05:48.386 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:05:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.525+0000 7fd940cb7700 1 Processor -- start 2023-03-03T19:05:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.525+0000 7fd940cb7700 1 -- start start 2023-03-03T19:05:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.526+0000 7fd940cb7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c072250 0x7fd93c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.526+0000 7fd940cb7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd93c0c1ce0 0x7fd93c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.526+0000 7fd940cb7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c1510d0 con 0x7fd93c071e80 2023-03-03T19:05:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.526+0000 7fd940cb7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93c0c3740 con 0x7fd93c0c1ce0 2023-03-03T19:05:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.526+0000 7fd940cb7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c070670 con 0x7fd93c072250 2023-03-03T19:05:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.526+0000 7fd93a59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd93c071e80 0x7fd93c151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33934/0 (socket says 172.21.15.7:33934) 2023-03-03T19:05:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.526+0000 7fd93a59c700 1 -- 172.21.15.7:0/2205714292 learned_addr learned my addr 172.21.15.7:0/2205714292 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:05:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd939d9b700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c072250 0x7fd93c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93ad9d700 1 --2- 172.21.15.7:0/2205714292 >> v2:172.21.15.143:3300/0 conn(0x7fd93c0c1ce0 0x7fd93c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144324910 0 0) 0x7fd93c1510d0 con 0x7fd93c071e80 2023-03-03T19:05:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9240035b0 con 0x7fd93c071e80 2023-03-03T19:05:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2884999834 0 0) 0x7fd9240035b0 con 0x7fd93c071e80 2023-03-03T19:05:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c072250 msgr2=0x7fd93c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:05:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c072250 0x7fd93c14e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 >> v2:172.21.15.143:3300/0 conn(0x7fd93c0c1ce0 msgr2=0x7fd93c14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:05:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 --2- 172.21.15.7:0/2205714292 >> v2:172.21.15.143:3300/0 conn(0x7fd93c0c1ce0 0x7fd93c14e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c0c39e0 con 0x7fd93c071e80 2023-03-03T19:05:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd930001670 con 0x7fd93c071e80 2023-03-03T19:05:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 >> v1:172.21.15.7:6789/0 conn(0x7fd93c071e80 legacy=0x7fd93c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c072250 0x7fd924003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd924003ff0 0x7fd924004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd924004d90 0x7fd924005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0c3740 con 0x7fd924003ff0 2023-03-03T19:05:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c070670 con 0x7fd924004d90 2023-03-03T19:05:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92400bac0 con 0x7fd93c072250 2023-03-03T19:05:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd939d9b700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd924003ff0 0x7fd924004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93a59c700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c072250 0x7fd924003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd93ad9d700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd924004d90 0x7fd924005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.527+0000 7fd939d9b700 1 -- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c072250 msgr2=0x7fd924003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:05:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd939d9b700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c072250 0x7fd924003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd939d9b700 1 -- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd924004d90 msgr2=0x7fd924005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:05:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd939d9b700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd924004d90 0x7fd924005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd939d9b700 1 -- 172.21.15.7:0/2205714292 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240035b0 con 0x7fd924003ff0 2023-03-03T19:05:48.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd939d9b700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd924003ff0 0x7fd924004850 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0c1780 tx=0x7fd924005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131e980daea731aa server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:05:48.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c00e6c0 con 0x7fd924003ff0 2023-03-03T19:05:48.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c00e820 con 0x7fd924003ff0 2023-03-03T19:05:48.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd93959a700 1 -- 172.21.15.7:0/2205714292 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c00eb90 con 0x7fd924003ff0 2023-03-03T19:05:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd940cb7700 1 -- 172.21.15.7:0/2205714292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd924003ff0 msgr2=0x7fd924004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd940cb7700 1 --2- 172.21.15.7:0/2205714292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd924003ff0 0x7fd924004850 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0c1780 tx=0x7fd924005880 comp rx=0 tx=0).stop 2023-03-03T19:05:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd93ad9d700 1 -- 172.21.15.7:0/2205714292 reap_dead start 2023-03-03T19:05:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd940cb7700 1 -- 172.21.15.7:0/2205714292 shutdown_connections 2023-03-03T19:05:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.528+0000 7fd940cb7700 1 -- 172.21.15.7:0/2205714292 >> 172.21.15.7:0/2205714292 conn(0x7fd93c0a94d0 msgr2=0x7fd93c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:05:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.529+0000 7fd940cb7700 1 -- 172.21.15.7:0/2205714292 shutdown_connections 2023-03-03T19:05:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.529+0000 7fd940cb7700 1 -- 172.21.15.7:0/2205714292 wait complete. 2023-03-03T19:05:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.530+0000 7fd940cb7700 1 Processor -- start 2023-03-03T19:05:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.530+0000 7fd940cb7700 1 -- start start 2023-03-03T19:05:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.531+0000 7fd940cb7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd924003ff0 0x7fd93c1439d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.531+0000 7fd940cb7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd93c1434a0 0x7fd93c145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.531+0000 7fd940cb7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c1428e0 0x7fd93c142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.531+0000 7fd940cb7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0ab1d0 con 0x7fd93c1434a0 2023-03-03T19:05:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.531+0000 7fd940cb7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0ad1e0 con 0x7fd924003ff0 2023-03-03T19:05:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.531+0000 7fd940cb7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0aa9c0 con 0x7fd93c1428e0 2023-03-03T19:05:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.531+0000 7fd939d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd93c1434a0 0x7fd93c145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.531+0000 7fd939d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd93c1434a0 0x7fd93c145a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39968/0 (socket says 172.21.15.7:39968) 2023-03-03T19:05:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.531+0000 7fd939d9b700 1 -- 172.21.15.7:0/582133504 learned_addr learned my addr 172.21.15.7:0/582133504 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:05:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd93a59c700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd924003ff0 0x7fd93c1439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd93ad9d700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c1428e0 0x7fd93c142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd939d9b700 1 -- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c1428e0 msgr2=0x7fd93c142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd939d9b700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c1428e0 0x7fd93c142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd939d9b700 1 -- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd924003ff0 msgr2=0x7fd93c1439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd939d9b700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd924003ff0 0x7fd93c1439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd939d9b700 1 -- 172.21.15.7:0/582133504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c151f30 con 0x7fd93c1434a0 2023-03-03T19:05:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd939d9b700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd93c1434a0 0x7fd93c145a00 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7fd92c000c00 tx=0x7fd92c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:05:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd92affd700 1 -- 172.21.15.7:0/582133504 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c021070 con 0x7fd93c1434a0 2023-03-03T19:05:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd92affd700 1 -- 172.21.15.7:0/582133504 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c004370 con 0x7fd93c1434a0 2023-03-03T19:05:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd92affd700 1 -- 172.21.15.7:0/582133504 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c0036a0 con 0x7fd93c1434a0 2023-03-03T19:05:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd940cb7700 1 -- 172.21.15.7:0/582133504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93c151300 con 0x7fd93c1434a0 2023-03-03T19:05:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.532+0000 7fd940cb7700 1 -- 172.21.15.7:0/582133504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c06c9a0 con 0x7fd93c1434a0 2023-03-03T19:05:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.533+0000 7fd940cb7700 1 -- 172.21.15.7:0/582133504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c074bb0 con 0x7fd93c1434a0 2023-03-03T19:05:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.534+0000 7fd92affd700 1 -- 172.21.15.7:0/582133504 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd92c02e030 con 0x7fd93c1434a0 2023-03-03T19:05:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.534+0000 7fd92affd700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd91c0682f0 0x7fd91c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:05:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.534+0000 7fd92affd700 1 -- 172.21.15.7:0/582133504 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd92c087340 con 0x7fd93c1434a0 2023-03-03T19:05:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.534+0000 7fd93a59c700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd91c0682f0 0x7fd91c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:05:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.535+0000 7fd93a59c700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd91c0682f0 0x7fd91c06a7a0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fd93c147c90 tx=0x7fd930007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:05:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.539+0000 7fd92affd700 1 -- 172.21.15.7:0/582133504 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd92c0536e0 con 0x7fd93c1434a0 2023-03-03T19:05:48.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.779+0000 7fd940cb7700 1 -- 172.21.15.7:0/582133504 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd93c0a9fb0 con 0x7fd93c1434a0 2023-03-03T19:05:48.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.780+0000 7fd92affd700 1 -- 172.21.15.7:0/582133504 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd92c056d00 con 0x7fd93c1434a0 2023-03-03T19:05:48.780 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:05:48.780 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:05:48.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 -- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd91c0682f0 msgr2=0x7fd91c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:48.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd91c0682f0 0x7fd91c06a7a0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fd93c147c90 tx=0x7fd930007980 comp rx=0 tx=0).stop 2023-03-03T19:05:48.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 -- 172.21.15.7:0/582133504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd93c1434a0 msgr2=0x7fd93c145a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:05:48.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd93c1434a0 0x7fd93c145a00 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7fd92c000c00 tx=0x7fd92c0047c0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 -- 172.21.15.7:0/582133504 shutdown_connections 2023-03-03T19:05:48.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd93c1428e0 0x7fd93c142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd924003ff0 0x7fd93c1439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd91c0682f0 0x7fd91c06a7a0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 --2- 172.21.15.7:0/582133504 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd93c1434a0 0x7fd93c145a00 unknown :-1 s=CLOSED pgs=3390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:05:48.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 -- 172.21.15.7:0/582133504 >> 172.21.15.7:0/582133504 conn(0x7fd93c0a94d0 msgr2=0x7fd93c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:05:48.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 -- 172.21.15.7:0/582133504 shutdown_connections 2023-03-03T19:05:48.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:05:48.782+0000 7fd940cb7700 1 -- 172.21.15.7:0/582133504 wait complete. 2023-03-03T19:05:48.784 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:05:48.794 INFO:tasks.mds_thrash.fs.[a]:waiting for 59 secs before thrashing 2023-03-03T19:06:14.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:06:14.871 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:06:44.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:06:44.934 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:06:47.795 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:06:47.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.933+0000 7fb4f0e7f700 1 Processor -- start 2023-03-03T19:06:47.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.933+0000 7fb4f0e7f700 1 -- start start 2023-03-03T19:06:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.934+0000 7fb4f0e7f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb4ec0caa30 0x7fb4ec0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.934+0000 7fb4f0e7f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c00e0 0x7fb4ec154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:47.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.934+0000 7fb4f0e7f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec0cb280 con 0x7fb4ec0cae00 2023-03-03T19:06:47.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.934+0000 7fb4f0e7f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4ec0c7310 con 0x7fb4ec0caa30 2023-03-03T19:06:47.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.934+0000 7fb4f0e7f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec0bfd10 con 0x7fb4ec0c00e0 2023-03-03T19:06:47.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4ea59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb4ec0caa30 0x7fb4ec0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:47.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4ea59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb4ec0caa30 0x7fb4ec0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40008/0 (socket says 172.21.15.7:40008) 2023-03-03T19:06:47.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4ea59c700 1 -- 172.21.15.7:0/2976963266 learned_addr learned my addr 172.21.15.7:0/2976963266 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:06:47.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4e9d9b700 1 --2- 172.21.15.7:0/2976963266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c00e0 0x7fb4ec154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:47.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4ea59c700 1 -- 172.21.15.7:0/2976963266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c00e0 msgr2=0x7fb4ec154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:06:47.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4ea59c700 1 --2- 172.21.15.7:0/2976963266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c00e0 0x7fb4ec154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:47.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4ea59c700 1 -- 172.21.15.7:0/2976963266 >> v1:172.21.15.143:6789/0 conn(0x7fb4ec0cae00 legacy=0x7fb4ec0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:06:47.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4ea59c700 1 -- 172.21.15.7:0/2976963266 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec0bfa30 con 0x7fb4ec0caa30 2023-03-03T19:06:47.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4ea59c700 1 --2- 172.21.15.7:0/2976963266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0caa30 0x7fb4ec0c5c60 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc0098f0 tx=0x7fb4dc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0984e887c394f4a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:06:47.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4e959a700 1 -- 172.21.15.7:0/2976963266 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc00b8a0 con 0x7fb4ec0caa30 2023-03-03T19:06:47.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4e959a700 1 -- 172.21.15.7:0/2976963266 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc00ba00 con 0x7fb4ec0caa30 2023-03-03T19:06:47.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.935+0000 7fb4e959a700 1 -- 172.21.15.7:0/2976963266 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc00bd90 con 0x7fb4ec0caa30 2023-03-03T19:06:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.936+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/2976963266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0caa30 msgr2=0x7fb4ec0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.936+0000 7fb4f0e7f700 1 --2- 172.21.15.7:0/2976963266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0caa30 0x7fb4ec0c5c60 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc0098f0 tx=0x7fb4dc009c00 comp rx=0 tx=0).stop 2023-03-03T19:06:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.936+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/2976963266 shutdown_connections 2023-03-03T19:06:47.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.936+0000 7fb4f0e7f700 1 --2- 172.21.15.7:0/2976963266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0caa30 0x7fb4ec0c5c60 unknown :-1 s=CLOSED pgs=3391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:47.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.936+0000 7fb4f0e7f700 1 --2- 172.21.15.7:0/2976963266 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c00e0 0x7fb4ec154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:47.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.936+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/2976963266 >> 172.21.15.7:0/2976963266 conn(0x7fb4ec066e40 msgr2=0x7fb4ec071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:06:47.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.936+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/2976963266 shutdown_connections 2023-03-03T19:06:47.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.936+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/2976963266 wait complete. 2023-03-03T19:06:47.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.938+0000 7fb4f0e7f700 1 Processor -- start 2023-03-03T19:06:47.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.938+0000 7fb4f0e7f700 1 -- start start 2023-03-03T19:06:47.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4f0e7f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c00e0 0x7fb4ec0c3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4f0e7f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec0caa30 0x7fb4ec0c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4f0e7f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0cae00 0x7fb4ec1dea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4f0e7f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec0c2040 con 0x7fb4ec0cae00 2023-03-03T19:06:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4f0e7f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec14dcd0 con 0x7fb4ec0caa30 2023-03-03T19:06:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4f0e7f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec14d540 con 0x7fb4ec0c00e0 2023-03-03T19:06:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4ead9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0cae00 0x7fb4ec1dea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4ead9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0cae00 0x7fb4ec1dea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40018/0 (socket says 172.21.15.7:40018) 2023-03-03T19:06:47.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4ead9d700 1 -- 172.21.15.7:0/408524180 learned_addr learned my addr 172.21.15.7:0/408524180 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:06:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4ea59c700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c00e0 0x7fb4ec0c3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4e9d9b700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec0caa30 0x7fb4ec0c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4ead9d700 1 -- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c00e0 msgr2=0x7fb4ec0c3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4ead9d700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c00e0 0x7fb4ec0c3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4ead9d700 1 -- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec0caa30 msgr2=0x7fb4ec0c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4ead9d700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec0caa30 0x7fb4ec0c4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:47.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4ead9d700 1 -- 172.21.15.7:0/408524180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc009550 con 0x7fb4ec0cae00 2023-03-03T19:06:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.939+0000 7fb4ead9d700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0cae00 0x7fb4ec1dea90 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400cb50 tx=0x7fb4e400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:06:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.940+0000 7fb4daffd700 1 -- 172.21.15.7:0/408524180 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e4003c00 con 0x7fb4ec0cae00 2023-03-03T19:06:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.940+0000 7fb4daffd700 1 -- 172.21.15.7:0/408524180 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e4003d60 con 0x7fb4ec0cae00 2023-03-03T19:06:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.940+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/408524180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ec1547f0 con 0x7fb4ec0cae00 2023-03-03T19:06:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.940+0000 7fb4daffd700 1 -- 172.21.15.7:0/408524180 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e400aa20 con 0x7fb4ec0cae00 2023-03-03T19:06:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.940+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/408524180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ec154510 con 0x7fb4ec0cae00 2023-03-03T19:06:47.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.941+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/408524180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec154ad0 con 0x7fb4ec0cae00 2023-03-03T19:06:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.941+0000 7fb4daffd700 1 -- 172.21.15.7:0/408524180 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb4e4010050 con 0x7fb4ec0cae00 2023-03-03T19:06:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.941+0000 7fb4daffd700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 0x7fb4cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.942+0000 7fb4daffd700 1 -- 172.21.15.7:0/408524180 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb4e4085fe0 con 0x7fb4ec0cae00 2023-03-03T19:06:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.942+0000 7fb4ea59c700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 0x7fb4cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.942+0000 7fb4ea59c700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 0x7fb4cc06a7a0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc000c00 tx=0x7fb4dc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:06:47.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:47.946+0000 7fb4daffd700 1 -- 172.21.15.7:0/408524180 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb4e4052380 con 0x7fb4ec0cae00 2023-03-03T19:06:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.186+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/408524180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4ec0c3150 con 0x7fb4ec0cae00 2023-03-03T19:06:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.187+0000 7fb4daffd700 1 -- 172.21.15.7:0/408524180 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb4e40559a0 con 0x7fb4ec0cae00 2023-03-03T19:06:48.188 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:06:48.188 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:06:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 msgr2=0x7fb4cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 0x7fb4cc06a7a0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc000c00 tx=0x7fb4dc008030 comp rx=0 tx=0).stop 2023-03-03T19:06:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/408524180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0cae00 msgr2=0x7fb4ec1dea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0cae00 0x7fb4ec1dea90 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7fb4e400cb50 tx=0x7fb4e400ce60 comp rx=0 tx=0).stop 2023-03-03T19:06:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/408524180 shutdown_connections 2023-03-03T19:06:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb4cc0682f0 0x7fb4cc06a7a0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4ec0cae00 0x7fb4ec1dea90 unknown :-1 s=CLOSED pgs=3392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb4ec0c00e0 0x7fb4ec0c3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 --2- 172.21.15.7:0/408524180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4ec0caa30 0x7fb4ec0c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/408524180 >> 172.21.15.7:0/408524180 conn(0x7fb4ec066e40 msgr2=0x7fb4ec1513c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:06:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/408524180 shutdown_connections 2023-03-03T19:06:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:48.189+0000 7fb4f0e7f700 1 -- 172.21.15.7:0/408524180 wait complete. 2023-03-03T19:06:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:06:48.202 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2023-03-03T19:06:48.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:06:49.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.040+0000 7fb28c762700 1 Processor -- start 2023-03-03T19:06:49.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.040+0000 7fb28c762700 1 -- start start 2023-03-03T19:06:49.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.041+0000 7fb28c762700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2840caa30 0x7fb2840c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:49.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.041+0000 7fb28c762700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb284154ff0 0x7fb2840cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:49.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.041+0000 7fb28c762700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2840c7310 con 0x7fb2840c00e0 2023-03-03T19:06:49.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.041+0000 7fb28c762700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb284155430 con 0x7fb2840caa30 2023-03-03T19:06:49.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.041+0000 7fb28c762700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2840bfd10 con 0x7fb284154ff0 2023-03-03T19:06:49.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.041+0000 7fb28a4fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2840caa30 0x7fb2840c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:49.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.041+0000 7fb28a4fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2840caa30 0x7fb2840c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40024/0 (socket says 172.21.15.7:40024) 2023-03-03T19:06:49.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.041+0000 7fb28a4fe700 1 -- 172.21.15.7:0/4187654752 learned_addr learned my addr 172.21.15.7:0/4187654752 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:06:49.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.041+0000 7fb28acff700 1 --2- 172.21.15.7:0/4187654752 >> v2:172.21.15.143:3300/0 conn(0x7fb284154ff0 0x7fb2840cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:49.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb28a4fe700 1 -- 172.21.15.7:0/4187654752 >> v2:172.21.15.143:3300/0 conn(0x7fb284154ff0 msgr2=0x7fb2840cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:06:49.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb28a4fe700 1 --2- 172.21.15.7:0/4187654752 >> v2:172.21.15.143:3300/0 conn(0x7fb284154ff0 0x7fb2840cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb28a4fe700 1 -- 172.21.15.7:0/4187654752 >> v1:172.21.15.143:6789/0 conn(0x7fb2840c00e0 legacy=0x7fb2840c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:49.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb28a4fe700 1 -- 172.21.15.7:0/4187654752 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2840bfa30 con 0x7fb2840caa30 2023-03-03T19:06:49.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb28acff700 1 --2- 172.21.15.7:0/4187654752 >> v2:172.21.15.143:3300/0 conn(0x7fb284154ff0 0x7fb2840cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:06:49.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb28a4fe700 1 --2- 172.21.15.7:0/4187654752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2840caa30 0x7fb2840c5c60 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7fb284071530 tx=0x7fb274009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc35c0a489259b1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:06:49.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb2894fc700 1 -- 172.21.15.7:0/4187654752 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27400b870 con 0x7fb2840caa30 2023-03-03T19:06:49.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb2894fc700 1 -- 172.21.15.7:0/4187654752 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27400b9d0 con 0x7fb2840caa30 2023-03-03T19:06:49.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb2894fc700 1 -- 172.21.15.7:0/4187654752 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27400bd60 con 0x7fb2840caa30 2023-03-03T19:06:49.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb28c762700 1 -- 172.21.15.7:0/4187654752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2840caa30 msgr2=0x7fb2840c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:49.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.042+0000 7fb28c762700 1 --2- 172.21.15.7:0/4187654752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2840caa30 0x7fb2840c5c60 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7fb284071530 tx=0x7fb274009bd0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.043+0000 7fb28c762700 1 -- 172.21.15.7:0/4187654752 shutdown_connections 2023-03-03T19:06:49.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.043+0000 7fb28c762700 1 --2- 172.21.15.7:0/4187654752 >> v2:172.21.15.143:3300/0 conn(0x7fb284154ff0 0x7fb2840cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.043+0000 7fb28c762700 1 --2- 172.21.15.7:0/4187654752 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2840caa30 0x7fb2840c5c60 unknown :-1 s=CLOSED pgs=3393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.043+0000 7fb28c762700 1 -- 172.21.15.7:0/4187654752 >> 172.21.15.7:0/4187654752 conn(0x7fb284066e40 msgr2=0x7fb284071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:06:49.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.043+0000 7fb28c762700 1 -- 172.21.15.7:0/4187654752 shutdown_connections 2023-03-03T19:06:49.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.043+0000 7fb28c762700 1 -- 172.21.15.7:0/4187654752 wait complete. 2023-03-03T19:06:49.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.044+0000 7fb28c762700 1 Processor -- start 2023-03-03T19:06:49.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.044+0000 7fb28c762700 1 -- start start 2023-03-03T19:06:49.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb28c762700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2840c00e0 0x7fb2841eb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:49.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb28c762700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2840caa30 0x7fb2841de0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:49.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb28c762700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb284154ff0 0x7fb284152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:49.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb28c762700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb284157b00 con 0x7fb284154ff0 2023-03-03T19:06:49.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb28c762700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2841e0a10 con 0x7fb2840caa30 2023-03-03T19:06:49.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb28c762700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2840c4c00 con 0x7fb2840c00e0 2023-03-03T19:06:49.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb28acff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb284154ff0 0x7fb284152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:49.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb28acff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb284154ff0 0x7fb284152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40032/0 (socket says 172.21.15.7:40032) 2023-03-03T19:06:49.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb28acff700 1 -- 172.21.15.7:0/1749905884 learned_addr learned my addr 172.21.15.7:0/1749905884 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:06:49.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb28a4fe700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2840c00e0 0x7fb2841eb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:49.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.045+0000 7fb289cfd700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2840caa30 0x7fb2841de0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:49.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb28acff700 1 -- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2840c00e0 msgr2=0x7fb2841eb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:49.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb28acff700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2840c00e0 0x7fb2841eb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb28acff700 1 -- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2840caa30 msgr2=0x7fb2841de0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:49.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb28acff700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2840caa30 0x7fb2841de0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb28acff700 1 -- 172.21.15.7:0/1749905884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274009550 con 0x7fb284154ff0 2023-03-03T19:06:49.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb28acff700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb284154ff0 0x7fb284152bf0 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 crypto rx=0x7fb27c007e70 tx=0x7fb27c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:06:49.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb27affd700 1 -- 172.21.15.7:0/1749905884 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c00dd20 con 0x7fb284154ff0 2023-03-03T19:06:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb27affd700 1 -- 172.21.15.7:0/1749905884 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb27c00f040 con 0x7fb284154ff0 2023-03-03T19:06:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb28c762700 1 -- 172.21.15.7:0/1749905884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2841547f0 con 0x7fb284154ff0 2023-03-03T19:06:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb27affd700 1 -- 172.21.15.7:0/1749905884 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c00dea0 con 0x7fb284154ff0 2023-03-03T19:06:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.046+0000 7fb28c762700 1 -- 172.21.15.7:0/1749905884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb284154510 con 0x7fb284154ff0 2023-03-03T19:06:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.047+0000 7fb28c762700 1 -- 172.21.15.7:0/1749905884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb284154ad0 con 0x7fb284154ff0 2023-03-03T19:06:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.047+0000 7fb27affd700 1 -- 172.21.15.7:0/1749905884 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb27c01c030 con 0x7fb284154ff0 2023-03-03T19:06:49.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.048+0000 7fb27affd700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2680682f0 0x7fb26806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:06:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.048+0000 7fb27affd700 1 -- 172.21.15.7:0/1749905884 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb27c085f90 con 0x7fb284154ff0 2023-03-03T19:06:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.048+0000 7fb28a4fe700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2680682f0 0x7fb26806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:06:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.049+0000 7fb28a4fe700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2680682f0 0x7fb26806a7a0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fb274000c00 tx=0x7fb274008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:06:49.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.052+0000 7fb27affd700 1 -- 172.21.15.7:0/1749905884 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb27c0522d0 con 0x7fb284154ff0 2023-03-03T19:06:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.293+0000 7fb28c762700 1 -- 172.21.15.7:0/1749905884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2840bff00 con 0x7fb284154ff0 2023-03-03T19:06:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.293+0000 7fb27affd700 1 -- 172.21.15.7:0/1749905884 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb27c0558f0 con 0x7fb284154ff0 2023-03-03T19:06:49.294 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:06:49.294 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:06:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 -- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2680682f0 msgr2=0x7fb26806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2680682f0 0x7fb26806a7a0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fb274000c00 tx=0x7fb274008030 comp rx=0 tx=0).stop 2023-03-03T19:06:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 -- 172.21.15.7:0/1749905884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb284154ff0 msgr2=0x7fb284152bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:06:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb284154ff0 0x7fb284152bf0 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 crypto rx=0x7fb27c007e70 tx=0x7fb27c00d3b0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 -- 172.21.15.7:0/1749905884 shutdown_connections 2023-03-03T19:06:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb2680682f0 0x7fb26806a7a0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb284154ff0 0x7fb284152bf0 unknown :-1 s=CLOSED pgs=3394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb2840c00e0 0x7fb2841eb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 --2- 172.21.15.7:0/1749905884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb2840caa30 0x7fb2841de0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:06:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 -- 172.21.15.7:0/1749905884 >> 172.21.15.7:0/1749905884 conn(0x7fb284066e40 msgr2=0x7fb2840c8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:06:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 -- 172.21.15.7:0/1749905884 shutdown_connections 2023-03-03T19:06:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:06:49.295+0000 7fb28c762700 1 -- 172.21.15.7:0/1749905884 wait complete. 2023-03-03T19:06:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:06:49.307 INFO:tasks.mds_thrash.fs.[b]:waiting for 47 secs before thrashing 2023-03-03T19:07:15.008 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:07:15.011 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:07:15.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:07:15.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.342+0000 7f7d42172700 1 Processor -- start 2023-03-03T19:07:15.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.342+0000 7f7d42172700 1 -- start start 2023-03-03T19:07:15.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.343+0000 7f7d42172700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7d3c071e80 0x7f7d3c151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:15.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.343+0000 7f7d42172700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3c072250 0x7f7d3c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:15.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.343+0000 7f7d42172700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3c1510d0 con 0x7f7d3c0c1ce0 2023-03-03T19:07:15.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.343+0000 7f7d42172700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3c0c3740 con 0x7f7d3c071e80 2023-03-03T19:07:15.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.343+0000 7f7d42172700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c070670 con 0x7f7d3c072250 2023-03-03T19:07:15.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.343+0000 7f7d3bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7d3c0c1ce0 0x7f7d3c1546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34048/0 (socket says 172.21.15.7:34048) 2023-03-03T19:07:15.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.343+0000 7f7d3bfff700 1 -- 172.21.15.7:0/1713741976 learned_addr learned my addr 172.21.15.7:0/1713741976 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:07:15.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3b7fe700 1 --2- 172.21.15.7:0/1713741976 >> v2:172.21.15.143:3300/0 conn(0x7f7d3c071e80 0x7f7d3c151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:15.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3affd700 1 --2- 172.21.15.7:0/1713741976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3c072250 0x7f7d3c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:15.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3a7fc700 1 -- 172.21.15.7:0/1713741976 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524868813 0 0) 0x7f7d3c1510d0 con 0x7f7d3c0c1ce0 2023-03-03T19:07:15.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3a7fc700 1 -- 172.21.15.7:0/1713741976 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d240057c0 con 0x7f7d3c0c1ce0 2023-03-03T19:07:15.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3b7fe700 1 -- 172.21.15.7:0/1713741976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3c072250 msgr2=0x7f7d3c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:07:15.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3b7fe700 1 --2- 172.21.15.7:0/1713741976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3c072250 0x7f7d3c14e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:15.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3b7fe700 1 -- 172.21.15.7:0/1713741976 >> v1:172.21.15.7:6789/0 conn(0x7f7d3c0c1ce0 legacy=0x7f7d3c1546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:15.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3b7fe700 1 -- 172.21.15.7:0/1713741976 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c0c39e0 con 0x7f7d3c071e80 2023-03-03T19:07:15.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3b7fe700 1 --2- 172.21.15.7:0/1713741976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d3c071e80 0x7f7d3c151820 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009840 tx=0x7f7d2c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b101a08b7fa28d94 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:07:15.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3a7fc700 1 -- 172.21.15.7:0/1713741976 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c011830 con 0x7f7d3c071e80 2023-03-03T19:07:15.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3a7fc700 1 -- 172.21.15.7:0/1713741976 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c011990 con 0x7f7d3c071e80 2023-03-03T19:07:15.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.344+0000 7f7d3a7fc700 1 -- 172.21.15.7:0/1713741976 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c011d00 con 0x7f7d3c071e80 2023-03-03T19:07:15.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.345+0000 7f7d42172700 1 -- 172.21.15.7:0/1713741976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d3c071e80 msgr2=0x7f7d3c151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:15.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.345+0000 7f7d42172700 1 --2- 172.21.15.7:0/1713741976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d3c071e80 0x7f7d3c151820 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009840 tx=0x7f7d2c009b50 comp rx=0 tx=0).stop 2023-03-03T19:07:15.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.345+0000 7f7d42172700 1 -- 172.21.15.7:0/1713741976 shutdown_connections 2023-03-03T19:07:15.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.345+0000 7f7d42172700 1 --2- 172.21.15.7:0/1713741976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d3c071e80 0x7f7d3c151820 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:15.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.345+0000 7f7d42172700 1 --2- 172.21.15.7:0/1713741976 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3c072250 0x7f7d3c14e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:15.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.345+0000 7f7d42172700 1 -- 172.21.15.7:0/1713741976 >> 172.21.15.7:0/1713741976 conn(0x7f7d3c0a94d0 msgr2=0x7f7d3c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:07:15.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.345+0000 7f7d42172700 1 -- 172.21.15.7:0/1713741976 shutdown_connections 2023-03-03T19:07:15.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.345+0000 7f7d42172700 1 -- 172.21.15.7:0/1713741976 wait complete. 2023-03-03T19:07:15.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.346+0000 7f7d42172700 1 Processor -- start 2023-03-03T19:07:15.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d42172700 1 -- start start 2023-03-03T19:07:15.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d42172700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3c071e80 0x7f7d3c142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:15.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d42172700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3c072250 0x7f7d3c0c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:15.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d42172700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d3c0c1ce0 0x7f7d3c1e8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:15.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d42172700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c0c0b20 con 0x7f7d3c072250 2023-03-03T19:07:15.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d42172700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c0c0230 con 0x7f7d3c0c1ce0 2023-03-03T19:07:15.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d42172700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c142520 con 0x7f7d3c071e80 2023-03-03T19:07:15.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d3affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3c072250 0x7f7d3c0c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:15.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d3affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3c072250 0x7f7d3c0c6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40072/0 (socket says 172.21.15.7:40072) 2023-03-03T19:07:15.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d3affd700 1 -- 172.21.15.7:0/1385976604 learned_addr learned my addr 172.21.15.7:0/1385976604 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:07:15.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d3b7fe700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3c071e80 0x7f7d3c142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:15.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.347+0000 7f7d3bfff700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d3c0c1ce0 0x7f7d3c1e8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:15.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d3affd700 1 -- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3c071e80 msgr2=0x7f7d3c142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d3affd700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3c071e80 0x7f7d3c142800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d3affd700 1 -- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d3c0c1ce0 msgr2=0x7f7d3c1e8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d3affd700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d3c0c1ce0 0x7f7d3c1e8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d3affd700 1 -- 172.21.15.7:0/1385976604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c0094a0 con 0x7f7d3c072250 2023-03-03T19:07:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d3affd700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3c072250 0x7f7d3c0c6550 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000e980 tx=0x7f7d3000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:07:15.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d23fff700 1 -- 172.21.15.7:0/1385976604 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30009710 con 0x7f7d3c072250 2023-03-03T19:07:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d23fff700 1 -- 172.21.15.7:0/1385976604 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3000c690 con 0x7f7d3c072250 2023-03-03T19:07:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d42172700 1 -- 172.21.15.7:0/1385976604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c06c9a0 con 0x7f7d3c072250 2023-03-03T19:07:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d23fff700 1 -- 172.21.15.7:0/1385976604 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3000ca20 con 0x7f7d3c072250 2023-03-03T19:07:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.348+0000 7f7d42172700 1 -- 172.21.15.7:0/1385976604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3c151300 con 0x7f7d3c072250 2023-03-03T19:07:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.349+0000 7f7d42172700 1 -- 172.21.15.7:0/1385976604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c06cb30 con 0x7f7d3c072250 2023-03-03T19:07:15.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.349+0000 7f7d23fff700 1 -- 172.21.15.7:0/1385976604 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7d30019030 con 0x7f7d3c072250 2023-03-03T19:07:15.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.350+0000 7f7d23fff700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d280682f0 0x7f7d2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:15.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.350+0000 7f7d23fff700 1 -- 172.21.15.7:0/1385976604 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7d30086010 con 0x7f7d3c072250 2023-03-03T19:07:15.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.350+0000 7f7d3b7fe700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d280682f0 0x7f7d2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:15.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.351+0000 7f7d3b7fe700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d280682f0 0x7f7d2806a7a0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c000c00 tx=0x7f7d2c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:07:15.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.354+0000 7f7d23fff700 1 -- 172.21.15.7:0/1385976604 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7d300523b0 con 0x7f7d3c072250 2023-03-03T19:07:15.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.593+0000 7f7d42172700 1 -- 172.21.15.7:0/1385976604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d3c074bb0 con 0x7f7d3c072250 2023-03-03T19:07:15.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.593+0000 7f7d23fff700 1 -- 172.21.15.7:0/1385976604 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7d300559d0 con 0x7f7d3c072250 2023-03-03T19:07:15.594 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:07:15.594 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:07:15.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.595+0000 7f7d42172700 1 -- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d280682f0 msgr2=0x7f7d2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:15.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.595+0000 7f7d42172700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d280682f0 0x7f7d2806a7a0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c000c00 tx=0x7f7d2c008030 comp rx=0 tx=0).stop 2023-03-03T19:07:15.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.595+0000 7f7d42172700 1 -- 172.21.15.7:0/1385976604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3c072250 msgr2=0x7f7d3c0c6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:15.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.595+0000 7f7d42172700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3c072250 0x7f7d3c0c6550 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000e980 tx=0x7f7d3000ec90 comp rx=0 tx=0).stop 2023-03-03T19:07:15.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.595+0000 7f7d42172700 1 -- 172.21.15.7:0/1385976604 shutdown_connections 2023-03-03T19:07:15.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.595+0000 7f7d42172700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d3c0c1ce0 0x7f7d3c1e8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:15.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.595+0000 7f7d42172700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7d3c071e80 0x7f7d3c142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:15.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.595+0000 7f7d42172700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7d280682f0 0x7f7d2806a7a0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:15.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.595+0000 7f7d42172700 1 --2- 172.21.15.7:0/1385976604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7d3c072250 0x7f7d3c0c6550 unknown :-1 s=CLOSED pgs=3396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:15.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.595+0000 7f7d42172700 1 -- 172.21.15.7:0/1385976604 >> 172.21.15.7:0/1385976604 conn(0x7f7d3c0a94d0 msgr2=0x7f7d3c147df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:07:15.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.596+0000 7f7d42172700 1 -- 172.21.15.7:0/1385976604 shutdown_connections 2023-03-03T19:07:15.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:15.596+0000 7f7d42172700 1 -- 172.21.15.7:0/1385976604 wait complete. 2023-03-03T19:07:15.598 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:07:15.608 INFO:tasks.mds_thrash.fs.[a]:waiting for 56 secs before thrashing 2023-03-03T19:07:36.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:07:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.447+0000 7fac71925700 1 Processor -- start 2023-03-03T19:07:36.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.447+0000 7fac71925700 1 -- start start 2023-03-03T19:07:36.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.448+0000 7fac71925700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fac6c0ac220 0x7fac6c0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.448+0000 7fac71925700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c0ac900 con 0x7fac6c073c30 2023-03-03T19:07:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.448+0000 7fac71925700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c0acb50 con 0x7fac6c152b60 2023-03-03T19:07:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.448+0000 7fac71925700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c071e30 con 0x7fac6c0ac220 2023-03-03T19:07:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.448+0000 7fac6a7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fac6c073c30 0x7fac6c152450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34072/0 (socket says 172.21.15.7:34072) 2023-03-03T19:07:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.448+0000 7fac6a7fc700 1 -- 172.21.15.7:0/3665175786 learned_addr learned my addr 172.21.15.7:0/3665175786 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:07:36.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.448+0000 7fac6affd700 1 --2- 172.21.15.7:0/3665175786 >> v2:172.21.15.143:3300/0 conn(0x7fac6c0ac220 0x7fac6c0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.448+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711326492 0 0) 0x7fac6c0ac900 con 0x7fac6c073c30 2023-03-03T19:07:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac580035b0 con 0x7fac6c073c30 2023-03-03T19:07:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2536331258 0 0) 0x7fac580035b0 con 0x7fac6c073c30 2023-03-03T19:07:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 >> v2:172.21.15.143:3300/0 conn(0x7fac6c0ac220 msgr2=0x7fac6c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:07:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 --2- 172.21.15.7:0/3665175786 >> v2:172.21.15.143:3300/0 conn(0x7fac6c0ac220 0x7fac6c0736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 >> v1:172.21.15.143:6789/0 conn(0x7fac6c152b60 legacy=0x7fac6c15ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c1559c0 con 0x7fac6c073c30 2023-03-03T19:07:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fac5c001670 con 0x7fac6c073c30 2023-03-03T19:07:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 >> v1:172.21.15.7:6789/0 conn(0x7fac6c073c30 legacy=0x7fac6c152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac58003a70 0x7fac58003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac58004850 0x7fac58004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac58005250 0x7fac5800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c071e30 con 0x7fac58005250 2023-03-03T19:07:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5800c190 con 0x7fac58003a70 2023-03-03T19:07:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5800c350 con 0x7fac58004850 2023-03-03T19:07:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac6b7fe700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac58005250 0x7fac5800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac6affd700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac58003a70 0x7fac58003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac6a7fc700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac58004850 0x7fac58004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac6b7fe700 1 -- 172.21.15.7:0/3665175786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac58004850 msgr2=0x7fac58004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:07:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac6b7fe700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac58004850 0x7fac58004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac6b7fe700 1 -- 172.21.15.7:0/3665175786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac58003a70 msgr2=0x7fac58003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:07:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac6b7fe700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac58003a70 0x7fac58003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac6b7fe700 1 -- 172.21.15.7:0/3665175786 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580035b0 con 0x7fac58005250 2023-03-03T19:07:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.449+0000 7fac6affd700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac58003a70 0x7fac58003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:07:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac6b7fe700 1 -- 172.21.15.7:0/3665175786 reap_dead start 2023-03-03T19:07:36.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac6b7fe700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac58005250 0x7fac5800ba10 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7fac6400c160 tx=0x7fac58005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83121d5428786e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:07:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac64001db0 con 0x7fac58005250 2023-03-03T19:07:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac6400cca0 con 0x7fac58005250 2023-03-03T19:07:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac69ffb700 1 -- 172.21.15.7:0/3665175786 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac64003c00 con 0x7fac58005250 2023-03-03T19:07:36.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac71925700 1 -- 172.21.15.7:0/3665175786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac58005250 msgr2=0x7fac5800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac71925700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac58005250 0x7fac5800ba10 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7fac6400c160 tx=0x7fac58005880 comp rx=0 tx=0).stop 2023-03-03T19:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac71925700 1 -- 172.21.15.7:0/3665175786 shutdown_connections 2023-03-03T19:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac71925700 1 --2- 172.21.15.7:0/3665175786 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac58005250 0x7fac5800ba10 unknown :-1 s=CLOSED pgs=3398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac71925700 1 -- 172.21.15.7:0/3665175786 >> 172.21.15.7:0/3665175786 conn(0x7fac6c0a94d0 msgr2=0x7fac6c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac71925700 1 -- 172.21.15.7:0/3665175786 shutdown_connections 2023-03-03T19:07:36.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.450+0000 7fac71925700 1 -- 172.21.15.7:0/3665175786 wait complete. 2023-03-03T19:07:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.452+0000 7fac71925700 1 Processor -- start 2023-03-03T19:07:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.452+0000 7fac71925700 1 -- start start 2023-03-03T19:07:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac71925700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac58005250 0x7fac6c0b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:36.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac71925700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac6c06deb0 0x7fac6c153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac71925700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac6c0bb740 0x7fac6c06ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac71925700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c1e9db0 con 0x7fac6c06deb0 2023-03-03T19:07:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac71925700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c1e9ad0 con 0x7fac58005250 2023-03-03T19:07:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac71925700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c06c1f0 con 0x7fac6c0bb740 2023-03-03T19:07:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac6a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac6c06deb0 0x7fac6c153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac6a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac6c06deb0 0x7fac6c153f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40106/0 (socket says 172.21.15.7:40106) 2023-03-03T19:07:36.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac6a7fc700 1 -- 172.21.15.7:0/3515691823 learned_addr learned my addr 172.21.15.7:0/3515691823 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:07:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac6b7fe700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac6c0bb740 0x7fac6c06ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.453+0000 7fac6affd700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac58005250 0x7fac6c0b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac6a7fc700 1 -- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac6c0bb740 msgr2=0x7fac6c06ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac6a7fc700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac6c0bb740 0x7fac6c06ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac6a7fc700 1 -- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac58005250 msgr2=0x7fac6c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac6a7fc700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac58005250 0x7fac6c0b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac6a7fc700 1 -- 172.21.15.7:0/3515691823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c155f80 con 0x7fac6c06deb0 2023-03-03T19:07:36.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac6a7fc700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac6c06deb0 0x7fac6c153f80 secure :-1 s=READY pgs=3399 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0072c0 tx=0x7fac5c007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:07:36.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac63fff700 1 -- 172.21.15.7:0/3515691823 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c00f850 con 0x7fac6c06deb0 2023-03-03T19:07:36.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac63fff700 1 -- 172.21.15.7:0/3515691823 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c00f9d0 con 0x7fac6c06deb0 2023-03-03T19:07:36.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac71925700 1 -- 172.21.15.7:0/3515691823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c155ca0 con 0x7fac6c06deb0 2023-03-03T19:07:36.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac63fff700 1 -- 172.21.15.7:0/3515691823 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c00fd80 con 0x7fac6c06deb0 2023-03-03T19:07:36.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.454+0000 7fac71925700 1 -- 172.21.15.7:0/3515691823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c156260 con 0x7fac6c06deb0 2023-03-03T19:07:36.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.455+0000 7fac71925700 1 -- 172.21.15.7:0/3515691823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c074bb0 con 0x7fac6c06deb0 2023-03-03T19:07:36.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.456+0000 7fac63fff700 1 -- 172.21.15.7:0/3515691823 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fac5c01d460 con 0x7fac6c06deb0 2023-03-03T19:07:36.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.456+0000 7fac63fff700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac4c0682f0 0x7fac4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:07:36.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.456+0000 7fac63fff700 1 -- 172.21.15.7:0/3515691823 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fac5c086220 con 0x7fac6c06deb0 2023-03-03T19:07:36.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.456+0000 7fac6affd700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac4c0682f0 0x7fac4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:07:36.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.457+0000 7fac6affd700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac4c0682f0 0x7fac4c06a7a0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fac54009cc0 tx=0x7fac54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:07:36.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.460+0000 7fac63fff700 1 -- 172.21.15.7:0/3515691823 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fac5c052560 con 0x7fac6c06deb0 2023-03-03T19:07:36.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.700+0000 7fac71925700 1 -- 172.21.15.7:0/3515691823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac6c0a9fb0 con 0x7fac6c06deb0 2023-03-03T19:07:36.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.701+0000 7fac63fff700 1 -- 172.21.15.7:0/3515691823 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fac5c055b80 con 0x7fac6c06deb0 2023-03-03T19:07:36.701 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:07:36.701 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:07:36.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 -- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac4c0682f0 msgr2=0x7fac4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:36.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac4c0682f0 0x7fac4c06a7a0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fac54009cc0 tx=0x7fac54008040 comp rx=0 tx=0).stop 2023-03-03T19:07:36.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 -- 172.21.15.7:0/3515691823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac6c06deb0 msgr2=0x7fac6c153f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:07:36.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac6c06deb0 0x7fac6c153f80 secure :-1 s=READY pgs=3399 cs=0 l=1 rev1=1 crypto rx=0x7fac5c0072c0 tx=0x7fac5c007680 comp rx=0 tx=0).stop 2023-03-03T19:07:36.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 -- 172.21.15.7:0/3515691823 shutdown_connections 2023-03-03T19:07:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fac6c0bb740 0x7fac6c06ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac58005250 0x7fac6c0b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fac4c0682f0 0x7fac4c06a7a0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 --2- 172.21.15.7:0/3515691823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fac6c06deb0 0x7fac6c153f80 unknown :-1 s=CLOSED pgs=3399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:07:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 -- 172.21.15.7:0/3515691823 >> 172.21.15.7:0/3515691823 conn(0x7fac6c0a94d0 msgr2=0x7fac6c1523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:07:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 -- 172.21.15.7:0/3515691823 shutdown_connections 2023-03-03T19:07:36.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:07:36.703+0000 7fac71925700 1 -- 172.21.15.7:0/3515691823 wait complete. 2023-03-03T19:07:36.705 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:07:36.716 INFO:tasks.mds_thrash.fs.[b]:waiting for 90 secs before thrashing 2023-03-03T19:07:45.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:07:45.077 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:08:11.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:08:11.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.748+0000 7fe8d78a3700 1 Processor -- start 2023-03-03T19:08:11.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.748+0000 7fe8d78a3700 1 -- start start 2023-03-03T19:08:11.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d78a3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe8d0151e80 0x7fe8d00c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:08:11.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d78a3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe8d00c6fe0 0x7fe8d00bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:08:11.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d78a3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00c08d0 con 0x7fe8d0155410 2023-03-03T19:08:11.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d78a3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d0152250 con 0x7fe8d00c6fe0 2023-03-03T19:08:11.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d78a3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8d00c0fb0 con 0x7fe8d0151e80 2023-03-03T19:08:11.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d5e40700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe8d00c6fe0 0x7fe8d00bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:08:11.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d5e40700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe8d00c6fe0 0x7fe8d00bfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40122/0 (socket says 172.21.15.7:40122) 2023-03-03T19:08:11.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d5e40700 1 -- 172.21.15.7:0/612792563 learned_addr learned my addr 172.21.15.7:0/612792563 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:08:11.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d563f700 1 --2- 172.21.15.7:0/612792563 >> v2:172.21.15.143:3300/0 conn(0x7fe8d0151e80 0x7fe8d00c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:08:11.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d5e40700 1 -- 172.21.15.7:0/612792563 >> v2:172.21.15.143:3300/0 conn(0x7fe8d0151e80 msgr2=0x7fe8d00c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:08:11.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d5e40700 1 --2- 172.21.15.7:0/612792563 >> v2:172.21.15.143:3300/0 conn(0x7fe8d0151e80 0x7fe8d00c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:08:11.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d5e40700 1 -- 172.21.15.7:0/612792563 >> v1:172.21.15.143:6789/0 conn(0x7fe8d0155410 legacy=0x7fe8d00bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:08:11.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.749+0000 7fe8d5e40700 1 -- 172.21.15.7:0/612792563 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0154930 con 0x7fe8d00c6fe0 2023-03-03T19:08:11.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.750+0000 7fe8d5e40700 1 --2- 172.21.15.7:0/612792563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d00c6fe0 0x7fe8d00bfff0 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00cb250 tx=0x7fe8c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b33e5198b3dfc14 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:08:11.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.750+0000 7fe8c7fff700 1 -- 172.21.15.7:0/612792563 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c800e070 con 0x7fe8d00c6fe0 2023-03-03T19:08:11.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.750+0000 7fe8c7fff700 1 -- 172.21.15.7:0/612792563 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c800b9c0 con 0x7fe8d00c6fe0 2023-03-03T19:08:11.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.750+0000 7fe8c7fff700 1 -- 172.21.15.7:0/612792563 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c800bd70 con 0x7fe8d00c6fe0 2023-03-03T19:08:11.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.750+0000 7fe8d78a3700 1 -- 172.21.15.7:0/612792563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d00c6fe0 msgr2=0x7fe8d00bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:08:11.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.750+0000 7fe8d78a3700 1 --2- 172.21.15.7:0/612792563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d00c6fe0 0x7fe8d00bfff0 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00cb250 tx=0x7fe8c800b330 comp rx=0 tx=0).stop 2023-03-03T19:08:11.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.750+0000 7fe8d78a3700 1 -- 172.21.15.7:0/612792563 shutdown_connections 2023-03-03T19:08:11.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.750+0000 7fe8d78a3700 1 --2- 172.21.15.7:0/612792563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d00c6fe0 0x7fe8d00bfff0 unknown :-1 s=CLOSED pgs=3400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:08:11.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.750+0000 7fe8d78a3700 1 --2- 172.21.15.7:0/612792563 >> v2:172.21.15.143:3300/0 conn(0x7fe8d0151e80 0x7fe8d00c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:08:11.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.750+0000 7fe8d78a3700 1 -- 172.21.15.7:0/612792563 >> 172.21.15.7:0/612792563 conn(0x7fe8d0071630 msgr2=0x7fe8d006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:08:11.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.751+0000 7fe8d78a3700 1 -- 172.21.15.7:0/612792563 shutdown_connections 2023-03-03T19:08:11.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.751+0000 7fe8d78a3700 1 -- 172.21.15.7:0/612792563 wait complete. 2023-03-03T19:08:11.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.752+0000 7fe8d78a3700 1 Processor -- start 2023-03-03T19:08:11.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.752+0000 7fe8d78a3700 1 -- start start 2023-03-03T19:08:11.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.752+0000 7fe8d78a3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d00c6fe0 0x7fe8d01585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:08:11.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.752+0000 7fe8d78a3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0151e80 0x7fe8d00c35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:08:11.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.752+0000 7fe8d78a3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d0155410 0x7fe8d0146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:08:11.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.752+0000 7fe8d78a3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0149f60 con 0x7fe8d0155410 2023-03-03T19:08:11.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.752+0000 7fe8d78a3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0158b00 con 0x7fe8d0151e80 2023-03-03T19:08:11.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.752+0000 7fe8d78a3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d0157680 con 0x7fe8d00c6fe0 2023-03-03T19:08:11.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d5e40700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d0155410 0x7fe8d0146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:08:11.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d5e40700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d0155410 0x7fe8d0146120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40128/0 (socket says 172.21.15.7:40128) 2023-03-03T19:08:11.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d5e40700 1 -- 172.21.15.7:0/389695301 learned_addr learned my addr 172.21.15.7:0/389695301 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:08:11.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d563f700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d00c6fe0 0x7fe8d01585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:08:11.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d4e3e700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0151e80 0x7fe8d00c35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:08:11.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d5e40700 1 -- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d00c6fe0 msgr2=0x7fe8d01585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:08:11.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d5e40700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d00c6fe0 0x7fe8d01585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:08:11.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d5e40700 1 -- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0151e80 msgr2=0x7fe8d00c35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:08:11.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d5e40700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0151e80 0x7fe8d00c35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:08:11.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d5e40700 1 -- 172.21.15.7:0/389695301 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c8009b30 con 0x7fe8d0155410 2023-03-03T19:08:11.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d5e40700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d0155410 0x7fe8d0146120 secure :-1 s=READY pgs=3401 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00cb4b0 tx=0x7fe8c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:08:11.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8c5ffb700 1 -- 172.21.15.7:0/389695301 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c8004880 con 0x7fe8d0155410 2023-03-03T19:08:11.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8c5ffb700 1 -- 172.21.15.7:0/389695301 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c80099b0 con 0x7fe8d0155410 2023-03-03T19:08:11.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8d78a3700 1 -- 172.21.15.7:0/389695301 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d0154ef0 con 0x7fe8d0155410 2023-03-03T19:08:11.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.753+0000 7fe8c5ffb700 1 -- 172.21.15.7:0/389695301 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c801c680 con 0x7fe8d0155410 2023-03-03T19:08:11.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.754+0000 7fe8d78a3700 1 -- 172.21.15.7:0/389695301 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d0154c10 con 0x7fe8d0155410 2023-03-03T19:08:11.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.754+0000 7fe8d78a3700 1 -- 172.21.15.7:0/389695301 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d00c5220 con 0x7fe8d0155410 2023-03-03T19:08:11.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.755+0000 7fe8c5ffb700 1 -- 172.21.15.7:0/389695301 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe8c802c070 con 0x7fe8d0155410 2023-03-03T19:08:11.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.755+0000 7fe8c5ffb700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8b40682f0 0x7fe8b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:08:11.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.755+0000 7fe8c5ffb700 1 -- 172.21.15.7:0/389695301 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe8c8087ba0 con 0x7fe8d0155410 2023-03-03T19:08:11.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.756+0000 7fe8d563f700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8b40682f0 0x7fe8b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:08:11.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.756+0000 7fe8d563f700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8b40682f0 0x7fe8b406a7a0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fe8d01534a0 tx=0x7fe8c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:08:11.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.760+0000 7fe8c5ffb700 1 -- 172.21.15.7:0/389695301 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe8c8053f40 con 0x7fe8d0155410 2023-03-03T19:08:11.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.995+0000 7fe8d78a3700 1 -- 172.21.15.7:0/389695301 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8d014a0a0 con 0x7fe8d0155410 2023-03-03T19:08:11.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.995+0000 7fe8c5ffb700 1 -- 172.21.15.7:0/389695301 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe8d014a0a0 con 0x7fe8d0155410 2023-03-03T19:08:11.996 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:08:11.996 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:08:11.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 -- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8b40682f0 msgr2=0x7fe8b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:08:11.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8b40682f0 0x7fe8b406a7a0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fe8d01534a0 tx=0x7fe8c0008040 comp rx=0 tx=0).stop 2023-03-03T19:08:11.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 -- 172.21.15.7:0/389695301 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d0155410 msgr2=0x7fe8d0146120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:08:11.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d0155410 0x7fe8d0146120 secure :-1 s=READY pgs=3401 cs=0 l=1 rev1=1 crypto rx=0x7fe8d00cb4b0 tx=0x7fe8c8004ad0 comp rx=0 tx=0).stop 2023-03-03T19:08:11.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 -- 172.21.15.7:0/389695301 shutdown_connections 2023-03-03T19:08:11.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe8b40682f0 0x7fe8b406a7a0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:08:11.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8d0155410 0x7fe8d0146120 unknown :-1 s=CLOSED pgs=3401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:08:11.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe8d00c6fe0 0x7fe8d01585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:08:11.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 --2- 172.21.15.7:0/389695301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8d0151e80 0x7fe8d00c35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:08:11.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 -- 172.21.15.7:0/389695301 >> 172.21.15.7:0/389695301 conn(0x7fe8d0071630 msgr2=0x7fe8d00c24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:08:11.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 -- 172.21.15.7:0/389695301 shutdown_connections 2023-03-03T19:08:11.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:08:11.997+0000 7fe8d78a3700 1 -- 172.21.15.7:0/389695301 wait complete. 2023-03-03T19:08:12.000 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:08:12.009 INFO:tasks.mds_thrash.fs.[a]:waiting for 100 secs before thrashing 2023-03-03T19:08:15.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:08:15.155 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:08:45.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:08:45.218 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:09:06.717 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:09:06.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.858+0000 7fa67f688700 1 Processor -- start 2023-03-03T19:09:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.858+0000 7fa67f688700 1 -- start start 2023-03-03T19:09:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67f688700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa678151e80 0x7fa6780c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67f688700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa6780c6fe0 0x7fa6780bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67f688700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6780c08d0 con 0x7fa678155410 2023-03-03T19:09:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67f688700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa678152250 con 0x7fa6780c6fe0 2023-03-03T19:09:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67f688700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6780c0fb0 con 0x7fa678151e80 2023-03-03T19:09:06.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67dc25700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa6780c6fe0 0x7fa6780bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67cc23700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa678155410 0x7fa6780bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34142/0 (socket says 172.21.15.7:34142) 2023-03-03T19:09:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67cc23700 1 -- 172.21.15.7:0/1956414098 learned_addr learned my addr 172.21.15.7:0/1956414098 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:09:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67d424700 1 --2- 172.21.15.7:0/1956414098 >> v2:172.21.15.143:3300/0 conn(0x7fa678151e80 0x7fa6780c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67dc25700 1 -- 172.21.15.7:0/1956414098 >> v2:172.21.15.143:3300/0 conn(0x7fa678151e80 msgr2=0x7fa6780c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:09:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67dc25700 1 --2- 172.21.15.7:0/1956414098 >> v2:172.21.15.143:3300/0 conn(0x7fa678151e80 0x7fa6780c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67dc25700 1 -- 172.21.15.7:0/1956414098 >> v1:172.21.15.7:6789/0 conn(0x7fa678155410 legacy=0x7fa6780bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:06.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67dc25700 1 -- 172.21.15.7:0/1956414098 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa678154930 con 0x7fa6780c6fe0 2023-03-03T19:09:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa66ffff700 1 -- 172.21.15.7:0/1956414098 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107253228 0 0) 0x7fa6780c08d0 con 0x7fa678155410 2023-03-03T19:09:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.859+0000 7fa67dc25700 1 --2- 172.21.15.7:0/1956414098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6780c6fe0 0x7fa6780bfff0 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7fa6780cb250 tx=0x7fa67000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2572b13883d7cd8b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:09:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa66ffff700 1 -- 172.21.15.7:0/1956414098 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67000e070 con 0x7fa6780c6fe0 2023-03-03T19:09:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa66ffff700 1 -- 172.21.15.7:0/1956414098 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67000bbd0 con 0x7fa6780c6fe0 2023-03-03T19:09:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa66ffff700 1 -- 172.21.15.7:0/1956414098 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa670012430 con 0x7fa6780c6fe0 2023-03-03T19:09:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa67f688700 1 -- 172.21.15.7:0/1956414098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6780c6fe0 msgr2=0x7fa6780bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa67f688700 1 --2- 172.21.15.7:0/1956414098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6780c6fe0 0x7fa6780bfff0 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7fa6780cb250 tx=0x7fa67000b330 comp rx=0 tx=0).stop 2023-03-03T19:09:06.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa67f688700 1 -- 172.21.15.7:0/1956414098 shutdown_connections 2023-03-03T19:09:06.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa67f688700 1 --2- 172.21.15.7:0/1956414098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6780c6fe0 0x7fa6780bfff0 unknown :-1 s=CLOSED pgs=3403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:06.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa67f688700 1 --2- 172.21.15.7:0/1956414098 >> v2:172.21.15.143:3300/0 conn(0x7fa678151e80 0x7fa6780c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:06.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa67f688700 1 -- 172.21.15.7:0/1956414098 >> 172.21.15.7:0/1956414098 conn(0x7fa678071630 msgr2=0x7fa67806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:09:06.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa67f688700 1 -- 172.21.15.7:0/1956414098 shutdown_connections 2023-03-03T19:09:06.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.860+0000 7fa67f688700 1 -- 172.21.15.7:0/1956414098 wait complete. 2023-03-03T19:09:06.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.861+0000 7fa67f688700 1 Processor -- start 2023-03-03T19:09:06.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.862+0000 7fa67f688700 1 -- start start 2023-03-03T19:09:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.862+0000 7fa67f688700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6780c6fe0 0x7fa67814e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.862+0000 7fa67f688700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa678151e80 0x7fa6780bf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.862+0000 7fa67f688700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa678155410 0x7fa6781eaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.862+0000 7fa67f688700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67814f830 con 0x7fa678151e80 2023-03-03T19:09:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.862+0000 7fa67f688700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6780c8ad0 con 0x7fa6780c6fe0 2023-03-03T19:09:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.862+0000 7fa67f688700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6781ea3c0 con 0x7fa678155410 2023-03-03T19:09:06.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.862+0000 7fa67cc23700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa678151e80 0x7fa6780bf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.862+0000 7fa67cc23700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa678151e80 0x7fa6780bf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40170/0 (socket says 172.21.15.7:40170) 2023-03-03T19:09:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.862+0000 7fa67cc23700 1 -- 172.21.15.7:0/246428801 learned_addr learned my addr 172.21.15.7:0/246428801 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:09:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67d424700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6780c6fe0 0x7fa67814e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67dc25700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa678155410 0x7fa6781eaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67cc23700 1 -- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa678155410 msgr2=0x7fa6781eaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67cc23700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa678155410 0x7fa6781eaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:06.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67cc23700 1 -- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6780c6fe0 msgr2=0x7fa67814e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67cc23700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6780c6fe0 0x7fa67814e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67cc23700 1 -- 172.21.15.7:0/246428801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa670009a70 con 0x7fa678151e80 2023-03-03T19:09:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67dc25700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa678155410 0x7fa6781eaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:09:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67cc23700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa678151e80 0x7fa6780bf6d0 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7fa66800c460 tx=0x7fa66800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:09:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa66dffb700 1 -- 172.21.15.7:0/246428801 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668004c80 con 0x7fa678151e80 2023-03-03T19:09:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa66dffb700 1 -- 172.21.15.7:0/246428801 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa668004e00 con 0x7fa678151e80 2023-03-03T19:09:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67f688700 1 -- 172.21.15.7:0/246428801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa678154ef0 con 0x7fa678151e80 2023-03-03T19:09:06.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa66dffb700 1 -- 172.21.15.7:0/246428801 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668007310 con 0x7fa678151e80 2023-03-03T19:09:06.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.863+0000 7fa67f688700 1 -- 172.21.15.7:0/246428801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa678154c10 con 0x7fa678151e80 2023-03-03T19:09:06.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.864+0000 7fa67f688700 1 -- 172.21.15.7:0/246428801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6780c5220 con 0x7fa678151e80 2023-03-03T19:09:06.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.865+0000 7fa66dffb700 1 -- 172.21.15.7:0/246428801 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa66801a020 con 0x7fa678151e80 2023-03-03T19:09:06.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.865+0000 7fa66dffb700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa65c0682f0 0x7fa65c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:06.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.865+0000 7fa66dffb700 1 -- 172.21.15.7:0/246428801 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa668085ca0 con 0x7fa678151e80 2023-03-03T19:09:06.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.865+0000 7fa67d424700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa65c0682f0 0x7fa65c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:06.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.866+0000 7fa67d424700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa65c0682f0 0x7fa65c06a7a0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fa674009930 tx=0x7fa674008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:09:06.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:06.869+0000 7fa66dffb700 1 -- 172.21.15.7:0/246428801 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa66808a070 con 0x7fa678151e80 2023-03-03T19:09:07.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.105+0000 7fa67f688700 1 -- 172.21.15.7:0/246428801 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6780c23a0 con 0x7fa678151e80 2023-03-03T19:09:07.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.106+0000 7fa66dffb700 1 -- 172.21.15.7:0/246428801 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa6780c23a0 con 0x7fa678151e80 2023-03-03T19:09:07.106 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:09:07.107 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:09:07.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 -- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa65c0682f0 msgr2=0x7fa65c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:07.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa65c0682f0 0x7fa65c06a7a0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fa674009930 tx=0x7fa674008040 comp rx=0 tx=0).stop 2023-03-03T19:09:07.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 -- 172.21.15.7:0/246428801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa678151e80 msgr2=0x7fa6780bf6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:07.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa678151e80 0x7fa6780bf6d0 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7fa66800c460 tx=0x7fa66800c820 comp rx=0 tx=0).stop 2023-03-03T19:09:07.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 -- 172.21.15.7:0/246428801 shutdown_connections 2023-03-03T19:09:07.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa678155410 0x7fa6781eaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:07.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6780c6fe0 0x7fa67814e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:07.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa65c0682f0 0x7fa65c06a7a0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:07.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 --2- 172.21.15.7:0/246428801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa678151e80 0x7fa6780bf6d0 unknown :-1 s=CLOSED pgs=3404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:07.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 -- 172.21.15.7:0/246428801 >> 172.21.15.7:0/246428801 conn(0x7fa678071630 msgr2=0x7fa6780c4330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:09:07.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 -- 172.21.15.7:0/246428801 shutdown_connections 2023-03-03T19:09:07.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:07.108+0000 7fa67f688700 1 -- 172.21.15.7:0/246428801 wait complete. 2023-03-03T19:09:07.110 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:09:07.120 INFO:tasks.mds_thrash.fs.[b]:waiting for 55 secs before thrashing 2023-03-03T19:09:15.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:09:15.282 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:09:45.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:09:45.346 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:09:52.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:09:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.148+0000 7fad1d9ba700 1 Processor -- start 2023-03-03T19:09:52.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.148+0000 7fad1d9ba700 1 -- start start 2023-03-03T19:09:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.148+0000 7fad1d9ba700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fad180caa30 0x7fad180c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad1d9ba700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad180c7310 con 0x7fad18154ff0 2023-03-03T19:09:52.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad1d9ba700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad180bfd10 con 0x7fad180c00e0 2023-03-03T19:09:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad1d9ba700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad18155430 con 0x7fad180caa30 2023-03-03T19:09:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad177fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fad18154ff0 0x7fad1815ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34192/0 (socket says 172.21.15.7:34192) 2023-03-03T19:09:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad177fe700 1 -- 172.21.15.7:0/694035269 learned_addr learned my addr 172.21.15.7:0/694035269 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:09:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad16ffd700 1 --2- 172.21.15.7:0/694035269 >> v2:172.21.15.143:3300/0 conn(0x7fad180caa30 0x7fad180c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554536637 0 0) 0x7fad180c7310 con 0x7fad18154ff0 2023-03-03T19:09:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad000035b0 con 0x7fad18154ff0 2023-03-03T19:09:52.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 92877476 0 0) 0x7fad000035b0 con 0x7fad18154ff0 2023-03-03T19:09:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 >> v2:172.21.15.143:3300/0 conn(0x7fad180caa30 msgr2=0x7fad180c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:09:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 --2- 172.21.15.7:0/694035269 >> v2:172.21.15.143:3300/0 conn(0x7fad180caa30 0x7fad180c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 >> v1:172.21.15.143:6789/0 conn(0x7fad180c00e0 legacy=0x7fad180c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad180bfa30 con 0x7fad18154ff0 2023-03-03T19:09:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fad100019f0 con 0x7fad18154ff0 2023-03-03T19:09:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 >> v1:172.21.15.7:6789/0 conn(0x7fad18154ff0 legacy=0x7fad1815ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad00003a70 0x7fad00003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:52.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad00004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad00005250 0x7fad0000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad18155430 con 0x7fad00004850 2023-03-03T19:09:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0000c190 con 0x7fad00003a70 2023-03-03T19:09:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0000c350 con 0x7fad00005250 2023-03-03T19:09:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.149+0000 7fad167fc700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad00004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad16ffd700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad00003a70 0x7fad00003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad177fe700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad00005250 0x7fad0000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:52.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad167fc700 1 -- 172.21.15.7:0/694035269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad00005250 msgr2=0x7fad0000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:09:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad167fc700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad00005250 0x7fad0000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad167fc700 1 -- 172.21.15.7:0/694035269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad00003a70 msgr2=0x7fad00003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:09:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad167fc700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad00003a70 0x7fad00003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad167fc700 1 -- 172.21.15.7:0/694035269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000035b0 con 0x7fad00004850 2023-03-03T19:09:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad177fe700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad00005250 0x7fad0000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:09:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad177fe700 1 -- 172.21.15.7:0/694035269 reap_dead start 2023-03-03T19:09:52.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad167fc700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad00004cc0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7fad0c001830 tx=0x7fad00005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f6b56ac6269565 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:09:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c001320 con 0x7fad00004850 2023-03-03T19:09:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0c005310 con 0x7fad00004850 2023-03-03T19:09:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.150+0000 7fad15ffb700 1 -- 172.21.15.7:0/694035269 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0068b0 con 0x7fad00004850 2023-03-03T19:09:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.151+0000 7fad1d9ba700 1 -- 172.21.15.7:0/694035269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 msgr2=0x7fad00004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.151+0000 7fad1d9ba700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad00004cc0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7fad0c001830 tx=0x7fad00005880 comp rx=0 tx=0).stop 2023-03-03T19:09:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.151+0000 7fad1d9ba700 1 -- 172.21.15.7:0/694035269 shutdown_connections 2023-03-03T19:09:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.151+0000 7fad1d9ba700 1 --2- 172.21.15.7:0/694035269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad00004cc0 unknown :-1 s=CLOSED pgs=3406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:52.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.151+0000 7fad1d9ba700 1 -- 172.21.15.7:0/694035269 >> 172.21.15.7:0/694035269 conn(0x7fad18066e40 msgr2=0x7fad18071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:09:52.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.151+0000 7fad1d9ba700 1 -- 172.21.15.7:0/694035269 shutdown_connections 2023-03-03T19:09:52.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.151+0000 7fad1d9ba700 1 -- 172.21.15.7:0/694035269 wait complete. 2023-03-03T19:09:52.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.152+0000 7fad1d9ba700 1 Processor -- start 2023-03-03T19:09:52.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.152+0000 7fad1d9ba700 1 -- start start 2023-03-03T19:09:52.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad1d9ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad1814d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:52.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad1d9ba700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad180c9a60 0x7fad181ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:52.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad1d9ba700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad18149f20 0x7fad1814a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:52.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad1d9ba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1814d570 con 0x7fad00004850 2023-03-03T19:09:52.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad1d9ba700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1814ded0 con 0x7fad18149f20 2023-03-03T19:09:52.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad1d9ba700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1814cf60 con 0x7fad180c9a60 2023-03-03T19:09:52.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad16ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad1814d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:52.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad16ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad1814d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40220/0 (socket says 172.21.15.7:40220) 2023-03-03T19:09:52.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad16ffd700 1 -- 172.21.15.7:0/3299073714 learned_addr learned my addr 172.21.15.7:0/3299073714 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:09:52.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad16ffd700 1 -- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad180c9a60 msgr2=0x7fad181ed3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:52.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad16ffd700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad180c9a60 0x7fad181ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:52.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad16ffd700 1 -- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad18149f20 msgr2=0x7fad1814a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:52.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad16ffd700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad18149f20 0x7fad1814a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:52.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad16ffd700 1 -- 172.21.15.7:0/3299073714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad181547f0 con 0x7fad00004850 2023-03-03T19:09:52.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.153+0000 7fad16ffd700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad1814d990 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7fad08011ad0 tx=0x7fad08011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:09:52.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.154+0000 7facfffff700 1 -- 172.21.15.7:0/3299073714 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0800f9e0 con 0x7fad00004850 2023-03-03T19:09:52.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.154+0000 7facfffff700 1 -- 172.21.15.7:0/3299073714 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0800fb40 con 0x7fad00004850 2023-03-03T19:09:52.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.154+0000 7fad1d9ba700 1 -- 172.21.15.7:0/3299073714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad18154510 con 0x7fad00004850 2023-03-03T19:09:52.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.154+0000 7facfffff700 1 -- 172.21.15.7:0/3299073714 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad08014440 con 0x7fad00004850 2023-03-03T19:09:52.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.154+0000 7fad1d9ba700 1 -- 172.21.15.7:0/3299073714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad18154ad0 con 0x7fad00004850 2023-03-03T19:09:52.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.155+0000 7fad1d9ba700 1 -- 172.21.15.7:0/3299073714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1804e6e0 con 0x7fad00004850 2023-03-03T19:09:52.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.155+0000 7facfffff700 1 -- 172.21.15.7:0/3299073714 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fad080145a0 con 0x7fad00004850 2023-03-03T19:09:52.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.155+0000 7facfffff700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad040682f0 0x7fad0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:09:52.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.155+0000 7facfffff700 1 -- 172.21.15.7:0/3299073714 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fad08086a20 con 0x7fad00004850 2023-03-03T19:09:52.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.156+0000 7fad167fc700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad040682f0 0x7fad0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:09:52.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.156+0000 7fad167fc700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad040682f0 0x7fad0406a7a0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fad0c001830 tx=0x7fad0c011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:09:52.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.160+0000 7facfffff700 1 -- 172.21.15.7:0/3299073714 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fad08052dc0 con 0x7fad00004850 2023-03-03T19:09:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.397+0000 7fad1d9ba700 1 -- 172.21.15.7:0/3299073714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad18154d80 con 0x7fad00004850 2023-03-03T19:09:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.398+0000 7facfffff700 1 -- 172.21.15.7:0/3299073714 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fad080563e0 con 0x7fad00004850 2023-03-03T19:09:52.399 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:09:52.399 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:09:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.399+0000 7fad1d9ba700 1 -- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad040682f0 msgr2=0x7fad0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.399+0000 7fad1d9ba700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad040682f0 0x7fad0406a7a0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fad0c001830 tx=0x7fad0c011000 comp rx=0 tx=0).stop 2023-03-03T19:09:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.399+0000 7fad1d9ba700 1 -- 172.21.15.7:0/3299073714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 msgr2=0x7fad1814d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:09:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.399+0000 7fad1d9ba700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad1814d990 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7fad08011ad0 tx=0x7fad08011e90 comp rx=0 tx=0).stop 2023-03-03T19:09:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.399+0000 7fad1d9ba700 1 -- 172.21.15.7:0/3299073714 shutdown_connections 2023-03-03T19:09:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.399+0000 7fad1d9ba700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad18149f20 0x7fad1814a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.399+0000 7fad1d9ba700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fad040682f0 0x7fad0406a7a0 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.399+0000 7fad1d9ba700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fad00004850 0x7fad1814d990 unknown :-1 s=CLOSED pgs=3407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.399+0000 7fad1d9ba700 1 --2- 172.21.15.7:0/3299073714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fad180c9a60 0x7fad181ed3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:09:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.400+0000 7fad1d9ba700 1 -- 172.21.15.7:0/3299073714 >> 172.21.15.7:0/3299073714 conn(0x7fad18066e40 msgr2=0x7fad18152f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:09:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.400+0000 7fad1d9ba700 1 -- 172.21.15.7:0/3299073714 shutdown_connections 2023-03-03T19:09:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:09:52.400+0000 7fad1d9ba700 1 -- 172.21.15.7:0/3299073714 wait complete. 2023-03-03T19:09:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:09:52.413 INFO:tasks.mds_thrash.fs.[a]:waiting for 97 secs before thrashing 2023-03-03T19:09:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T19:09:59.998+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T19:10:02.121 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:10:02.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.261+0000 7f24d580b700 1 Processor -- start 2023-03-03T19:10:02.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.261+0000 7f24d580b700 1 -- start start 2023-03-03T19:10:02.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24d580b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f24d014b340 0x7f24d0152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:10:02.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24d580b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d00c2790 0x7f24d0152b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:10:02.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24d580b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d006c920 con 0x7f24d0151140 2023-03-03T19:10:02.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24d580b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d00abd90 con 0x7f24d014b340 2023-03-03T19:10:02.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24d580b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d00c1300 con 0x7f24d00c2790 2023-03-03T19:10:02.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24cf7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f24d0151140 0x7f24d0151590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34226/0 (socket says 172.21.15.7:34226) 2023-03-03T19:10:02.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24cf7fe700 1 -- 172.21.15.7:0/332035842 learned_addr learned my addr 172.21.15.7:0/332035842 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:10:02.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24ceffd700 1 --2- 172.21.15.7:0/332035842 >> v2:172.21.15.143:3300/0 conn(0x7f24d014b340 0x7f24d0152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:10:02.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24ce7fc700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d00c2790 0x7f24d0152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:10:02.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500055604 0 0) 0x7f24d006c920 con 0x7f24d0151140 2023-03-03T19:10:02.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.262+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b80035b0 con 0x7f24d0151140 2023-03-03T19:10:02.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 430674732 0 0) 0x7f24b80035b0 con 0x7f24d0151140 2023-03-03T19:10:02.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d00c2790 msgr2=0x7f24d0152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:10:02.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d00c2790 0x7f24d0152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:10:02.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 >> v2:172.21.15.143:3300/0 conn(0x7f24d014b340 msgr2=0x7f24d0152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:10:02.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 --2- 172.21.15.7:0/332035842 >> v2:172.21.15.143:3300/0 conn(0x7f24d014b340 0x7f24d0152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:10:02.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d006bd50 con 0x7f24d0151140 2023-03-03T19:10:02.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f24c80019f0 con 0x7f24d0151140 2023-03-03T19:10:02.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 >> v1:172.21.15.7:6789/0 conn(0x7f24d0151140 legacy=0x7f24d0151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:10:02.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b8003a70 0x7f24b8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:10:02.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24b8004850 0x7f24b8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:10:02.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d00c2790 0x7f24b8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:10:02.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d00abd90 con 0x7f24b8004850 2023-03-03T19:10:02.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d00c1300 con 0x7f24b8003a70 2023-03-03T19:10:02.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b800bb10 con 0x7f24d00c2790 2023-03-03T19:10:02.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24ce7fc700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24b8004850 0x7f24b8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:10:02.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24ceffd700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b8003a70 0x7f24b8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:10:02.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24cf7fe700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d00c2790 0x7f24b8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:10:02.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24ce7fc700 1 -- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d00c2790 msgr2=0x7f24b8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:10:02.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24ce7fc700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d00c2790 0x7f24b8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:10:02.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24ce7fc700 1 -- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b8003a70 msgr2=0x7f24b8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:10:02.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24ce7fc700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b8003a70 0x7f24b8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:10:02.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24ce7fc700 1 -- 172.21.15.7:0/332035842 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b80035b0 con 0x7f24b8004850 2023-03-03T19:10:02.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.263+0000 7f24ce7fc700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24b8004850 0x7f24b8004cc0 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f24c0009b90 tx=0x7f24b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276a593b5067a7c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:10:02.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.264+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c000e6c0 con 0x7f24b8004850 2023-03-03T19:10:02.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.264+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c000e820 con 0x7f24b8004850 2023-03-03T19:10:02.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.264+0000 7f24cdffb700 1 -- 172.21.15.7:0/332035842 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c000eb90 con 0x7f24b8004850 2023-03-03T19:10:02.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.264+0000 7f24d580b700 1 -- 172.21.15.7:0/332035842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24b8004850 msgr2=0x7f24b8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:10:02.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.264+0000 7f24d580b700 1 --2- 172.21.15.7:0/332035842 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24b8004850 0x7f24b8004cc0 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f24c0009b90 tx=0x7f24b8005880 comp rx=0 tx=0).stop 2023-03-03T19:10:02.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.264+0000 7f24cf7fe700 1 -- 172.21.15.7:0/332035842 reap_dead start 2023-03-03T19:10:02.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.264+0000 7f24d580b700 1 -- 172.21.15.7:0/332035842 shutdown_connections 2023-03-03T19:10:02.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.264+0000 7f24d580b700 1 -- 172.21.15.7:0/332035842 >> 172.21.15.7:0/332035842 conn(0x7f24d00c62e0 msgr2=0x7f24d00c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:10:02.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.264+0000 7f24d580b700 1 -- 172.21.15.7:0/332035842 shutdown_connections 2023-03-03T19:10:02.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.264+0000 7f24d580b700 1 -- 172.21.15.7:0/332035842 wait complete. 2023-03-03T19:10:02.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.266+0000 7f24d580b700 1 Processor -- start 2023-03-03T19:10:02.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.266+0000 7f24d580b700 1 -- start start 2023-03-03T19:10:02.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24d580b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b8004850 0x7f24d00c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:10:02.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24d580b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0147930 0x7f24d0144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:10:02.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24d580b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24d0145fc0 0x7f24d01445c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:10:02.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24d580b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d0146da0 con 0x7f24d0145fc0 2023-03-03T19:10:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24d580b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d0147570 con 0x7f24b8004850 2023-03-03T19:10:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24d580b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d0146590 con 0x7f24d0147930 2023-03-03T19:10:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24d0145fc0 0x7f24d01445c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:10:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24d0145fc0 0x7f24d01445c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40258/0 (socket says 172.21.15.7:40258) 2023-03-03T19:10:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24cf7fe700 1 -- 172.21.15.7:0/1329515348 learned_addr learned my addr 172.21.15.7:0/1329515348 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:10:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24ce7fc700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0147930 0x7f24d0144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:10:02.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24ceffd700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b8004850 0x7f24d00c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:10:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24cf7fe700 1 -- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0147930 msgr2=0x7f24d0144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:10:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24cf7fe700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0147930 0x7f24d0144080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:10:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24cf7fe700 1 -- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b8004850 msgr2=0x7f24d00c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:10:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24cf7fe700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b8004850 0x7f24d00c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:10:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24cf7fe700 1 -- 172.21.15.7:0/1329515348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d006c2d0 con 0x7f24d0145fc0 2023-03-03T19:10:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24cf7fe700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24d0145fc0 0x7f24d01445c0 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f24d014ed10 tx=0x7f24c8007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:10:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24b7fff700 1 -- 172.21.15.7:0/1329515348 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8017070 con 0x7f24d0145fc0 2023-03-03T19:10:02.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24b7fff700 1 -- 172.21.15.7:0/1329515348 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c800ec80 con 0x7f24d0145fc0 2023-03-03T19:10:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24d580b700 1 -- 172.21.15.7:0/1329515348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d006bff0 con 0x7f24d0145fc0 2023-03-03T19:10:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24b7fff700 1 -- 172.21.15.7:0/1329515348 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8001f00 con 0x7f24d0145fc0 2023-03-03T19:10:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.267+0000 7f24d580b700 1 -- 172.21.15.7:0/1329515348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d006c5b0 con 0x7f24d0145fc0 2023-03-03T19:10:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.268+0000 7f24d580b700 1 -- 172.21.15.7:0/1329515348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d006fcb0 con 0x7f24d0145fc0 2023-03-03T19:10:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.269+0000 7f24b7fff700 1 -- 172.21.15.7:0/1329515348 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f24c8013420 con 0x7f24d0145fc0 2023-03-03T19:10:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.269+0000 7f24b7fff700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24bc0682f0 0x7f24bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:10:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.269+0000 7f24b7fff700 1 -- 172.21.15.7:0/1329515348 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f24c8086020 con 0x7f24d0145fc0 2023-03-03T19:10:02.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.270+0000 7f24ceffd700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24bc0682f0 0x7f24bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:10:02.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.270+0000 7f24ceffd700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24bc0682f0 0x7f24bc06a7a0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009c20 tx=0x7f24c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:10:02.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.275+0000 7f24b7fff700 1 -- 172.21.15.7:0/1329515348 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f24c8052360 con 0x7f24d0145fc0 2023-03-03T19:10:02.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.510+0000 7f24d580b700 1 -- 172.21.15.7:0/1329515348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24d0074bb0 con 0x7f24d0145fc0 2023-03-03T19:10:02.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.511+0000 7f24b7fff700 1 -- 172.21.15.7:0/1329515348 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f24c8055980 con 0x7f24d0145fc0 2023-03-03T19:10:02.512 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:10:02.512 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:10:02.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.512+0000 7f24d580b700 1 -- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24bc0682f0 msgr2=0x7f24bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:10:02.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.512+0000 7f24d580b700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24bc0682f0 0x7f24bc06a7a0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009c20 tx=0x7f24c4008040 comp rx=0 tx=0).stop 2023-03-03T19:10:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.512+0000 7f24d580b700 1 -- 172.21.15.7:0/1329515348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24d0145fc0 msgr2=0x7f24d01445c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:10:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.512+0000 7f24d580b700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24d0145fc0 0x7f24d01445c0 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f24d014ed10 tx=0x7f24c8007c90 comp rx=0 tx=0).stop 2023-03-03T19:10:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.513+0000 7f24d580b700 1 -- 172.21.15.7:0/1329515348 shutdown_connections 2023-03-03T19:10:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.513+0000 7f24d580b700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f24bc0682f0 0x7f24bc06a7a0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:10:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.513+0000 7f24d580b700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24d0145fc0 0x7f24d01445c0 unknown :-1 s=CLOSED pgs=3410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:10:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.513+0000 7f24d580b700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24b8004850 0x7f24d00c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:10:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.513+0000 7f24d580b700 1 --2- 172.21.15.7:0/1329515348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f24d0147930 0x7f24d0144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:10:02.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.513+0000 7f24d580b700 1 -- 172.21.15.7:0/1329515348 >> 172.21.15.7:0/1329515348 conn(0x7f24d00c62e0 msgr2=0x7f24d0153a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:10:02.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.513+0000 7f24d580b700 1 -- 172.21.15.7:0/1329515348 shutdown_connections 2023-03-03T19:10:02.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:10:02.513+0000 7f24d580b700 1 -- 172.21.15.7:0/1329515348 wait complete. 2023-03-03T19:10:02.518 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:10:02.526 INFO:tasks.mds_thrash.fs.[b]:waiting for 117 secs before thrashing 2023-03-03T19:10:15.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:10:15.434 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:10:45.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:10:45.497 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:11:15.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:11:15.585 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:11:29.414 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:11:29.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.551+0000 7f8a28c8a700 1 Processor -- start 2023-03-03T19:11:29.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.551+0000 7f8a28c8a700 1 -- start start 2023-03-03T19:11:29.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.552+0000 7f8a28c8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 0x7f8a24156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:29.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.552+0000 7f8a28c8a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a240abe20 0x7f8a241529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:29.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.552+0000 7f8a28c8a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a240c2f30 con 0x7f8a2414fda0 2023-03-03T19:11:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.552+0000 7f8a28c8a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a240c31d0 con 0x7f8a240abe20 2023-03-03T19:11:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.552+0000 7f8a28c8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a241524a0 con 0x7f8a2414f9d0 2023-03-03T19:11:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a22d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a240abe20 0x7f8a241529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a22d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a240abe20 0x7f8a241529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40340/0 (socket says 172.21.15.7:40340) 2023-03-03T19:11:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a22d9d700 1 -- 172.21.15.7:0/3094331988 learned_addr learned my addr 172.21.15.7:0/3094331988 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:11:29.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a2259c700 1 --2- 172.21.15.7:0/3094331988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 0x7f8a24156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a22d9d700 1 -- 172.21.15.7:0/3094331988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 msgr2=0x7f8a24156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:11:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a22d9d700 1 --2- 172.21.15.7:0/3094331988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 0x7f8a24156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a22d9d700 1 -- 172.21.15.7:0/3094331988 >> v1:172.21.15.143:6789/0 conn(0x7f8a2414fda0 legacy=0x7f8a24071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a22d9d700 1 -- 172.21.15.7:0/3094331988 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a240c34b0 con 0x7f8a240abe20 2023-03-03T19:11:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a22d9d700 1 --2- 172.21.15.7:0/3094331988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 0x7f8a241529c0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009e50 tx=0x7f8a1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e4564d782df454 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:11:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a2159a700 1 -- 172.21.15.7:0/3094331988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c00e070 con 0x7f8a240abe20 2023-03-03T19:11:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a2159a700 1 -- 172.21.15.7:0/3094331988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c00b9c0 con 0x7f8a240abe20 2023-03-03T19:11:29.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.553+0000 7f8a2159a700 1 -- 172.21.15.7:0/3094331988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c00bd70 con 0x7f8a240abe20 2023-03-03T19:11:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.554+0000 7f8a28c8a700 1 -- 172.21.15.7:0/3094331988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 msgr2=0x7f8a241529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.554+0000 7f8a28c8a700 1 --2- 172.21.15.7:0/3094331988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 0x7f8a241529c0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009e50 tx=0x7f8a1c00b330 comp rx=0 tx=0).stop 2023-03-03T19:11:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.554+0000 7f8a28c8a700 1 -- 172.21.15.7:0/3094331988 shutdown_connections 2023-03-03T19:11:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.554+0000 7f8a28c8a700 1 --2- 172.21.15.7:0/3094331988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 0x7f8a241529c0 unknown :-1 s=CLOSED pgs=3411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.554+0000 7f8a28c8a700 1 --2- 172.21.15.7:0/3094331988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 0x7f8a24156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:29.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.554+0000 7f8a28c8a700 1 -- 172.21.15.7:0/3094331988 >> 172.21.15.7:0/3094331988 conn(0x7f8a240c8900 msgr2=0x7f8a240c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:11:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.554+0000 7f8a28c8a700 1 -- 172.21.15.7:0/3094331988 shutdown_connections 2023-03-03T19:11:29.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.554+0000 7f8a28c8a700 1 -- 172.21.15.7:0/3094331988 wait complete. 2023-03-03T19:11:29.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.555+0000 7f8a28c8a700 1 Processor -- start 2023-03-03T19:11:29.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.555+0000 7f8a28c8a700 1 -- start start 2023-03-03T19:11:29.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a28c8a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 0x7f8a24152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:29.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a28c8a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 0x7f8a240bd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:29.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a28c8a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2414fda0 0x7f8a241ddd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:29.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a28c8a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240c6dc0 con 0x7f8a240abe20 2023-03-03T19:11:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a28c8a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2406e8d0 con 0x7f8a2414fda0 2023-03-03T19:11:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a28c8a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2406ebb0 con 0x7f8a2414f9d0 2023-03-03T19:11:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a2259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 0x7f8a24152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a2259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 0x7f8a24152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40342/0 (socket says 172.21.15.7:40342) 2023-03-03T19:11:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a2259c700 1 -- 172.21.15.7:0/2609374032 learned_addr learned my addr 172.21.15.7:0/2609374032 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:11:29.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a22d9d700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2414fda0 0x7f8a241ddd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a21d9b700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 0x7f8a240bd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:29.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a2259c700 1 -- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 msgr2=0x7f8a240bd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:29.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a2259c700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 0x7f8a240bd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:29.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a2259c700 1 -- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2414fda0 msgr2=0x7f8a241ddd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:29.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a2259c700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2414fda0 0x7f8a241ddd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:29.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a2259c700 1 -- 172.21.15.7:0/2609374032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c009b30 con 0x7f8a240abe20 2023-03-03T19:11:29.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a21d9b700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 0x7f8a240bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:11:29.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.556+0000 7f8a22d9d700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2414fda0 0x7f8a241ddd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:11:29.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.557+0000 7f8a2259c700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 0x7f8a24152b30 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400e810 tx=0x7f8a1400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:11:29.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.557+0000 7f8a1affd700 1 -- 172.21.15.7:0/2609374032 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14013610 con 0x7f8a240abe20 2023-03-03T19:11:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.557+0000 7f8a1affd700 1 -- 172.21.15.7:0/2609374032 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14013770 con 0x7f8a240abe20 2023-03-03T19:11:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.557+0000 7f8a28c8a700 1 -- 172.21.15.7:0/2609374032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a240ac350 con 0x7f8a240abe20 2023-03-03T19:11:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.557+0000 7f8a1affd700 1 -- 172.21.15.7:0/2609374032 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14013b20 con 0x7f8a240abe20 2023-03-03T19:11:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.557+0000 7f8a28c8a700 1 -- 172.21.15.7:0/2609374032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a240ab630 con 0x7f8a240abe20 2023-03-03T19:11:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.558+0000 7f8a28c8a700 1 -- 172.21.15.7:0/2609374032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a24071fa0 con 0x7f8a240abe20 2023-03-03T19:11:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.558+0000 7f8a1affd700 1 -- 172.21.15.7:0/2609374032 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a14023070 con 0x7f8a240abe20 2023-03-03T19:11:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.559+0000 7f8a1affd700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a040682f0 0x7f8a0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.559+0000 7f8a1affd700 1 -- 172.21.15.7:0/2609374032 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8a14085e80 con 0x7f8a240abe20 2023-03-03T19:11:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.559+0000 7f8a21d9b700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a040682f0 0x7f8a0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:29.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.560+0000 7f8a21d9b700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a040682f0 0x7f8a0406a7a0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f8a24156400 tx=0x7f8a0c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:11:29.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.563+0000 7f8a1affd700 1 -- 172.21.15.7:0/2609374032 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8a14052220 con 0x7f8a240abe20 2023-03-03T19:11:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.800+0000 7f8a28c8a700 1 -- 172.21.15.7:0/2609374032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a2406fc50 con 0x7f8a240abe20 2023-03-03T19:11:29.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.800+0000 7f8a1affd700 1 -- 172.21.15.7:0/2609374032 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8a14055840 con 0x7f8a240abe20 2023-03-03T19:11:29.801 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:11:29.801 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:11:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 -- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a040682f0 msgr2=0x7f8a0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:29.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a040682f0 0x7f8a0406a7a0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f8a24156400 tx=0x7f8a0c009040 comp rx=0 tx=0).stop 2023-03-03T19:11:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 -- 172.21.15.7:0/2609374032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 msgr2=0x7f8a24152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 0x7f8a24152b30 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400e810 tx=0x7f8a1400ebd0 comp rx=0 tx=0).stop 2023-03-03T19:11:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 -- 172.21.15.7:0/2609374032 shutdown_connections 2023-03-03T19:11:29.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a2414fda0 0x7f8a241ddd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:29.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a040682f0 0x7f8a0406a7a0 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:29.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a240abe20 0x7f8a24152b30 unknown :-1 s=CLOSED pgs=3412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:29.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 --2- 172.21.15.7:0/2609374032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a2414f9d0 0x7f8a240bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:29.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 -- 172.21.15.7:0/2609374032 >> 172.21.15.7:0/2609374032 conn(0x7f8a240c8900 msgr2=0x7f8a240c3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:11:29.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 -- 172.21.15.7:0/2609374032 shutdown_connections 2023-03-03T19:11:29.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:29.802+0000 7f8a28c8a700 1 -- 172.21.15.7:0/2609374032 wait complete. 2023-03-03T19:11:29.806 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:11:29.814 INFO:tasks.mds_thrash.fs.[a]:waiting for 34 secs before thrashing 2023-03-03T19:11:45.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:11:45.648 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:11:59.527 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:11:59.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.666+0000 7f06f8ab5700 1 Processor -- start 2023-03-03T19:11:59.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.666+0000 7f06f8ab5700 1 -- start start 2023-03-03T19:11:59.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.667+0000 7f06f8ab5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06f414fb00 0x7f06f4156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:59.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.667+0000 7f06f8ab5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c1da0 0x7f06f40c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:59.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.667+0000 7f06f8ab5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f4155840 con 0x7f06f40c4370 2023-03-03T19:11:59.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.667+0000 7f06f8ab5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f4072940 con 0x7f06f414fb00 2023-03-03T19:11:59.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.667+0000 7f06f8ab5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f41567d0 con 0x7f06f40c1da0 2023-03-03T19:11:59.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06f414fb00 0x7f06f4156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:59.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f2d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f06f40c4370 0x7f06f40c7370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34348/0 (socket says 172.21.15.7:34348) 2023-03-03T19:11:59.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f2d9d700 1 -- 172.21.15.7:0/1883938338 learned_addr learned my addr 172.21.15.7:0/1883938338 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:11:59.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f1d9b700 1 --2- 172.21.15.7:0/1883938338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c1da0 0x7f06f40c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:59.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f259c700 1 -- 172.21.15.7:0/1883938338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c1da0 msgr2=0x7f06f40c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:11:59.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f259c700 1 --2- 172.21.15.7:0/1883938338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c1da0 0x7f06f40c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:59.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f259c700 1 -- 172.21.15.7:0/1883938338 >> v1:172.21.15.7:6789/0 conn(0x7f06f40c4370 legacy=0x7f06f40c7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:59.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f259c700 1 -- 172.21.15.7:0/1883938338 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f40c2330 con 0x7f06f414fb00 2023-03-03T19:11:59.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f159a700 1 -- 172.21.15.7:0/1883938338 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753347370 0 0) 0x7f06f4155840 con 0x7f06f40c4370 2023-03-03T19:11:59.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f1d9b700 1 --2- 172.21.15.7:0/1883938338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c1da0 0x7f06f40c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:11:59.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f259c700 1 --2- 172.21.15.7:0/1883938338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f414fb00 0x7f06f4156290 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009840 tx=0x7f06e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db9f2f0bf11eccf server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:11:59.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f159a700 1 -- 172.21.15.7:0/1883938338 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4011830 con 0x7f06f414fb00 2023-03-03T19:11:59.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f159a700 1 -- 172.21.15.7:0/1883938338 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e4011990 con 0x7f06f414fb00 2023-03-03T19:11:59.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.668+0000 7f06f159a700 1 -- 172.21.15.7:0/1883938338 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4011d20 con 0x7f06f414fb00 2023-03-03T19:11:59.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.669+0000 7f06f8ab5700 1 -- 172.21.15.7:0/1883938338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f414fb00 msgr2=0x7f06f4156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:59.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.669+0000 7f06f8ab5700 1 --2- 172.21.15.7:0/1883938338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f414fb00 0x7f06f4156290 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009840 tx=0x7f06e4009b50 comp rx=0 tx=0).stop 2023-03-03T19:11:59.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.669+0000 7f06f8ab5700 1 -- 172.21.15.7:0/1883938338 shutdown_connections 2023-03-03T19:11:59.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.669+0000 7f06f8ab5700 1 --2- 172.21.15.7:0/1883938338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f414fb00 0x7f06f4156290 unknown :-1 s=CLOSED pgs=3414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:59.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.669+0000 7f06f8ab5700 1 --2- 172.21.15.7:0/1883938338 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c1da0 0x7f06f40c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:59.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.669+0000 7f06f8ab5700 1 -- 172.21.15.7:0/1883938338 >> 172.21.15.7:0/1883938338 conn(0x7f06f406c110 msgr2=0x7f06f406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:11:59.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.669+0000 7f06f8ab5700 1 -- 172.21.15.7:0/1883938338 shutdown_connections 2023-03-03T19:11:59.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.669+0000 7f06f8ab5700 1 -- 172.21.15.7:0/1883938338 wait complete. 2023-03-03T19:11:59.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.670+0000 7f06f8ab5700 1 Processor -- start 2023-03-03T19:11:59.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.670+0000 7f06f8ab5700 1 -- start start 2023-03-03T19:11:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f8ab5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f40c1da0 0x7f06f41e7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f8ab5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c4370 0x7f06f41db9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f8ab5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f414fb00 0x7f06f41eb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f8ab5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f41eb880 con 0x7f06f40c1da0 2023-03-03T19:11:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f8ab5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f41dc0d0 con 0x7f06f414fb00 2023-03-03T19:11:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f8ab5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f41dc3b0 con 0x7f06f40c4370 2023-03-03T19:11:59.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f40c1da0 0x7f06f41e7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f40c1da0 0x7f06f41e7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40372/0 (socket says 172.21.15.7:40372) 2023-03-03T19:11:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f259c700 1 -- 172.21.15.7:0/3829632130 learned_addr learned my addr 172.21.15.7:0/3829632130 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:11:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f1d9b700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c4370 0x7f06f41db9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f2d9d700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f414fb00 0x7f06f41eb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f259c700 1 -- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c4370 msgr2=0x7f06f41db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f259c700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c4370 0x7f06f41db9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f259c700 1 -- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f414fb00 msgr2=0x7f06f41eb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:59.677 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f259c700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f414fb00 0x7f06f41eb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:59.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f259c700 1 -- 172.21.15.7:0/3829632130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e40094a0 con 0x7f06f40c1da0 2023-03-03T19:11:59.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f2d9d700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f414fb00 0x7f06f41eb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:11:59.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.671+0000 7f06f1d9b700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c4370 0x7f06f41db9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:11:59.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.672+0000 7f06f259c700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f40c1da0 0x7f06f41e7ef0 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7f06e40090b0 tx=0x7f06e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:11:59.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.672+0000 7f06e2ffd700 1 -- 172.21.15.7:0/3829632130 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e40045f0 con 0x7f06f40c1da0 2023-03-03T19:11:59.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.672+0000 7f06e2ffd700 1 -- 172.21.15.7:0/3829632130 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e401f070 con 0x7f06f40c1da0 2023-03-03T19:11:59.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.672+0000 7f06e2ffd700 1 -- 172.21.15.7:0/3829632130 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4003b00 con 0x7f06f40c1da0 2023-03-03T19:11:59.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.672+0000 7f06f8ab5700 1 -- 172.21.15.7:0/3829632130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f4153090 con 0x7f06f40c1da0 2023-03-03T19:11:59.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.672+0000 7f06f8ab5700 1 -- 172.21.15.7:0/3829632130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f4152db0 con 0x7f06f40c1da0 2023-03-03T19:11:59.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.673+0000 7f06f8ab5700 1 -- 172.21.15.7:0/3829632130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f4155a90 con 0x7f06f40c1da0 2023-03-03T19:11:59.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.673+0000 7f06e2ffd700 1 -- 172.21.15.7:0/3829632130 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f06e400f050 con 0x7f06f40c1da0 2023-03-03T19:11:59.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.673+0000 7f06e2ffd700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06d40682f0 0x7f06d406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:11:59.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.674+0000 7f06e2ffd700 1 -- 172.21.15.7:0/3829632130 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f06e4021070 con 0x7f06f40c1da0 2023-03-03T19:11:59.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.674+0000 7f06f1d9b700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06d40682f0 0x7f06d406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:11:59.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.674+0000 7f06f1d9b700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06d40682f0 0x7f06d406a7a0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f06f40c73b0 tx=0x7f06e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:11:59.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.678+0000 7f06e2ffd700 1 -- 172.21.15.7:0/3829632130 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f06e4054070 con 0x7f06f40c1da0 2023-03-03T19:11:59.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.913+0000 7f06f8ab5700 1 -- 172.21.15.7:0/3829632130 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06f406e410 con 0x7f06f40c1da0 2023-03-03T19:11:59.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.913+0000 7f06e2ffd700 1 -- 172.21.15.7:0/3829632130 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f06e4024e90 con 0x7f06f40c1da0 2023-03-03T19:11:59.915 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:11:59.915 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:11:59.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 -- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06d40682f0 msgr2=0x7f06d406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:59.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06d40682f0 0x7f06d406a7a0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f06f40c73b0 tx=0x7f06e8008040 comp rx=0 tx=0).stop 2023-03-03T19:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 -- 172.21.15.7:0/3829632130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f40c1da0 msgr2=0x7f06f41e7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f40c1da0 0x7f06f41e7ef0 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7f06e40090b0 tx=0x7f06e4004750 comp rx=0 tx=0).stop 2023-03-03T19:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 -- 172.21.15.7:0/3829632130 shutdown_connections 2023-03-03T19:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f06f414fb00 0x7f06f41eb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f06d40682f0 0x7f06d406a7a0 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06f40c1da0 0x7f06f41e7ef0 unknown :-1 s=CLOSED pgs=3415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:59.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 --2- 172.21.15.7:0/3829632130 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f06f40c4370 0x7f06f41db9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:11:59.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 -- 172.21.15.7:0/3829632130 >> 172.21.15.7:0/3829632130 conn(0x7f06f406c110 msgr2=0x7f06f4151f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:11:59.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 -- 172.21.15.7:0/3829632130 shutdown_connections 2023-03-03T19:11:59.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:11:59.915+0000 7f06f8ab5700 1 -- 172.21.15.7:0/3829632130 wait complete. 2023-03-03T19:11:59.918 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:11:59.928 INFO:tasks.mds_thrash.fs.[b]:waiting for 15 secs before thrashing 2023-03-03T19:12:03.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:12:03.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.954+0000 7f8c67821700 1 Processor -- start 2023-03-03T19:12:03.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.954+0000 7f8c67821700 1 -- start start 2023-03-03T19:12:03.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c67821700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c6014b460 0x7f8c6006aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:03.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c67821700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c60070360 con 0x7f8c60072470 2023-03-03T19:12:03.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c67821700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6006a5e0 con 0x7f8c60071990 2023-03-03T19:12:03.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c67821700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c601521a0 con 0x7f8c6014b460 2023-03-03T19:12:03.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c65dbe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8c60072470 0x7f8c6006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34368/0 (socket says 172.21.15.7:34368) 2023-03-03T19:12:03.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c65dbe700 1 -- 172.21.15.7:0/3755186489 learned_addr learned my addr 172.21.15.7:0/3755186489 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:12:03.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c64dbc700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c6014b460 0x7f8c6006aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:03.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177282504 0 0) 0x7f8c60070360 con 0x7f8c60072470 2023-03-03T19:12:03.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c4c0035b0 con 0x7f8c60072470 2023-03-03T19:12:03.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556430378 0 0) 0x7f8c6006a5e0 con 0x7f8c60071990 2023-03-03T19:12:03.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c60070360 con 0x7f8c60071990 2023-03-03T19:12:03.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1704344319 0 0) 0x7f8c4c0035b0 con 0x7f8c60072470 2023-03-03T19:12:03.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c6014b460 msgr2=0x7f8c6006aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:12:03.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c6014b460 0x7f8c6006aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:03.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 >> v1:172.21.15.143:6789/0 conn(0x7f8c60071990 legacy=0x7f8c60071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:03.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c600c38b0 con 0x7f8c60072470 2023-03-03T19:12:03.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8c580019f0 con 0x7f8c60072470 2023-03-03T19:12:03.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 >> v1:172.21.15.7:6789/0 conn(0x7f8c60072470 legacy=0x7f8c6006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:03.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c4c003c30 0x7f8c4c004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:03.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c4c004850 0x7f8c4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:03.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c4c0051c0 0x7f8c4c00dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:03.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c601521a0 con 0x7f8c4c003c30 2023-03-03T19:12:03.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c00e190 con 0x7f8c4c0051c0 2023-03-03T19:12:03.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.955+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c00e350 con 0x7f8c4c004850 2023-03-03T19:12:03.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c655bd700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c4c003c30 0x7f8c4c004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:03.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c65dbe700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c4c0051c0 0x7f8c4c00dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:03.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c64dbc700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c4c004850 0x7f8c4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:03.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c655bd700 1 -- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c4c004850 msgr2=0x7f8c4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:12:03.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c655bd700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c4c004850 0x7f8c4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:03.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c655bd700 1 -- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c4c0051c0 msgr2=0x7f8c4c00dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:12:03.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c655bd700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c4c0051c0 0x7f8c4c00dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:03.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c655bd700 1 -- 172.21.15.7:0/3755186489 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c0035b0 con 0x7f8c4c003c30 2023-03-03T19:12:03.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c655bd700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c4c003c30 0x7f8c4c004020 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7f8c50001980 tx=0x7f8c4c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abd18cf1e5cce91 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:03.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c50001470 con 0x7f8c4c003c30 2023-03-03T19:12:03.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c500052d0 con 0x7f8c4c003c30 2023-03-03T19:12:03.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.956+0000 7f8c57fff700 1 -- 172.21.15.7:0/3755186489 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c50003680 con 0x7f8c4c003c30 2023-03-03T19:12:03.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.957+0000 7f8c67821700 1 -- 172.21.15.7:0/3755186489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c4c003c30 msgr2=0x7f8c4c004020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:03.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.957+0000 7f8c67821700 1 --2- 172.21.15.7:0/3755186489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c4c003c30 0x7f8c4c004020 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7f8c50001980 tx=0x7f8c4c005880 comp rx=0 tx=0).stop 2023-03-03T19:12:03.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.957+0000 7f8c65dbe700 1 -- 172.21.15.7:0/3755186489 reap_dead start 2023-03-03T19:12:03.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.957+0000 7f8c67821700 1 -- 172.21.15.7:0/3755186489 shutdown_connections 2023-03-03T19:12:03.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.957+0000 7f8c67821700 1 -- 172.21.15.7:0/3755186489 >> 172.21.15.7:0/3755186489 conn(0x7f8c600a94d0 msgr2=0x7f8c600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:12:03.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.957+0000 7f8c67821700 1 -- 172.21.15.7:0/3755186489 shutdown_connections 2023-03-03T19:12:03.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.957+0000 7f8c67821700 1 -- 172.21.15.7:0/3755186489 wait complete. 2023-03-03T19:12:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.959+0000 7f8c67821700 1 Processor -- start 2023-03-03T19:12:03.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.959+0000 7f8c67821700 1 -- start start 2023-03-03T19:12:03.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.959+0000 7f8c67821700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c4c003c30 0x7f8c601e6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:03.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c67821700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c600c45a0 0x7f8c600ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:03.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c67821700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c600acb70 0x7f8c600bced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:03.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c67821700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c601e66b0 con 0x7f8c600c45a0 2023-03-03T19:12:03.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c67821700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c600acfe0 con 0x7f8c600acb70 2023-03-03T19:12:03.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c67821700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c600bc9f0 con 0x7f8c4c003c30 2023-03-03T19:12:03.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c64dbc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c600c45a0 0x7f8c600ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:03.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c64dbc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c600c45a0 0x7f8c600ac630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40398/0 (socket says 172.21.15.7:40398) 2023-03-03T19:12:03.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c64dbc700 1 -- 172.21.15.7:0/3032397136 learned_addr learned my addr 172.21.15.7:0/3032397136 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:12:03.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c65dbe700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c600acb70 0x7f8c600bced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:03.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c655bd700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c4c003c30 0x7f8c601e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:03.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c64dbc700 1 -- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c4c003c30 msgr2=0x7f8c601e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:03.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c64dbc700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c4c003c30 0x7f8c601e6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:03.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c64dbc700 1 -- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c600acb70 msgr2=0x7f8c600bced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:03.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c64dbc700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c600acb70 0x7f8c600bced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:03.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c64dbc700 1 -- 172.21.15.7:0/3032397136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c600c3b50 con 0x7f8c600c45a0 2023-03-03T19:12:03.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c65dbe700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c600acb70 0x7f8c600bced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:12:03.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c64dbc700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c600c45a0 0x7f8c600ac630 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0119a0 tx=0x7f8c5c011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:03.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c55ffb700 1 -- 172.21.15.7:0/3032397136 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c00f8e0 con 0x7f8c600c45a0 2023-03-03T19:12:03.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c55ffb700 1 -- 172.21.15.7:0/3032397136 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5c00fa40 con 0x7f8c600c45a0 2023-03-03T19:12:03.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c67821700 1 -- 172.21.15.7:0/3032397136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6006a2b0 con 0x7f8c600c45a0 2023-03-03T19:12:03.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.960+0000 7f8c55ffb700 1 -- 172.21.15.7:0/3032397136 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c00fdd0 con 0x7f8c600c45a0 2023-03-03T19:12:03.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.961+0000 7f8c67821700 1 -- 172.21.15.7:0/3032397136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6006b6c0 con 0x7f8c600c45a0 2023-03-03T19:12:03.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.962+0000 7f8c67821700 1 -- 172.21.15.7:0/3032397136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c600bcd20 con 0x7f8c600c45a0 2023-03-03T19:12:03.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.962+0000 7f8c55ffb700 1 -- 172.21.15.7:0/3032397136 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c5c014440 con 0x7f8c600c45a0 2023-03-03T19:12:03.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.962+0000 7f8c55ffb700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c440682f0 0x7f8c4406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:03.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.963+0000 7f8c55ffb700 1 -- 172.21.15.7:0/3032397136 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8c5c086970 con 0x7f8c600c45a0 2023-03-03T19:12:03.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.963+0000 7f8c655bd700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c440682f0 0x7f8c4406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:03.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.963+0000 7f8c655bd700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c440682f0 0x7f8c4406a7a0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f8c50015040 tx=0x7f8c5000a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:03.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:03.967+0000 7f8c55ffb700 1 -- 172.21.15.7:0/3032397136 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c5c052cb0 con 0x7f8c600c45a0 2023-03-03T19:12:04.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.203+0000 7f8c67821700 1 -- 172.21.15.7:0/3032397136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c60074bb0 con 0x7f8c600c45a0 2023-03-03T19:12:04.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.204+0000 7f8c55ffb700 1 -- 172.21.15.7:0/3032397136 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8c5c0562d0 con 0x7f8c600c45a0 2023-03-03T19:12:04.206 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:12:04.206 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:12:04.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.206+0000 7f8c67821700 1 -- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c440682f0 msgr2=0x7f8c4406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:04.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.206+0000 7f8c67821700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c440682f0 0x7f8c4406a7a0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f8c50015040 tx=0x7f8c5000a000 comp rx=0 tx=0).stop 2023-03-03T19:12:04.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.206+0000 7f8c67821700 1 -- 172.21.15.7:0/3032397136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c600c45a0 msgr2=0x7f8c600ac630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:04.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.206+0000 7f8c67821700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c600c45a0 0x7f8c600ac630 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0119a0 tx=0x7f8c5c011d60 comp rx=0 tx=0).stop 2023-03-03T19:12:04.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.207+0000 7f8c67821700 1 -- 172.21.15.7:0/3032397136 shutdown_connections 2023-03-03T19:12:04.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.207+0000 7f8c67821700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c600acb70 0x7f8c600bced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:04.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.207+0000 7f8c67821700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c4c003c30 0x7f8c601e6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:04.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.207+0000 7f8c67821700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c440682f0 0x7f8c4406a7a0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:04.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.207+0000 7f8c67821700 1 --2- 172.21.15.7:0/3032397136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c600c45a0 0x7f8c600ac630 unknown :-1 s=CLOSED pgs=3418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:04.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.207+0000 7f8c67821700 1 -- 172.21.15.7:0/3032397136 >> 172.21.15.7:0/3032397136 conn(0x7f8c600a94d0 msgr2=0x7f8c600c6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:12:04.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.207+0000 7f8c67821700 1 -- 172.21.15.7:0/3032397136 shutdown_connections 2023-03-03T19:12:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:04.207+0000 7f8c67821700 1 -- 172.21.15.7:0/3032397136 wait complete. 2023-03-03T19:12:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:12:04.220 INFO:tasks.mds_thrash.fs.[a]:waiting for 18 secs before thrashing 2023-03-03T19:12:14.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:12:15.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.068+0000 7fdff75f8700 1 Processor -- start 2023-03-03T19:12:15.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.068+0000 7fdff75f8700 1 -- start start 2023-03-03T19:12:15.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff75f8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fdff014f9d0 0x7fdff0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:15.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff75f8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdff014fda0 0x7fdff0071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:15.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff75f8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff00c2f30 con 0x7fdff00abe20 2023-03-03T19:12:15.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff75f8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff00c31d0 con 0x7fdff014fda0 2023-03-03T19:12:15.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff75f8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff01524a0 con 0x7fdff014f9d0 2023-03-03T19:12:15.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff4b93700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdff014fda0 0x7fdff0071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:15.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff4b93700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fdff014fda0 0x7fdff0071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40410/0 (socket says 172.21.15.7:40410) 2023-03-03T19:12:15.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff4b93700 1 -- 172.21.15.7:0/903149562 learned_addr learned my addr 172.21.15.7:0/903149562 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:12:15.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff5394700 1 --2- 172.21.15.7:0/903149562 >> v2:172.21.15.143:3300/0 conn(0x7fdff014f9d0 0x7fdff0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:15.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff4b93700 1 -- 172.21.15.7:0/903149562 >> v2:172.21.15.143:3300/0 conn(0x7fdff014f9d0 msgr2=0x7fdff0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:12:15.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff4b93700 1 --2- 172.21.15.7:0/903149562 >> v2:172.21.15.143:3300/0 conn(0x7fdff014f9d0 0x7fdff0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:15.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff4b93700 1 -- 172.21.15.7:0/903149562 >> v1:172.21.15.143:6789/0 conn(0x7fdff00abe20 legacy=0x7fdff01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:12:15.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.069+0000 7fdff4b93700 1 -- 172.21.15.7:0/903149562 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff00c34b0 con 0x7fdff014fda0 2023-03-03T19:12:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdff4b93700 1 --2- 172.21.15.7:0/903149562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff014fda0 0x7fdff0071890 secure :-1 s=READY pgs=3419 cs=0 l=1 rev1=1 crypto rx=0x7fdff00710a0 tx=0x7fdfec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3758a98f5e662e0d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdfe7fff700 1 -- 172.21.15.7:0/903149562 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec004230 con 0x7fdff014fda0 2023-03-03T19:12:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdfe7fff700 1 -- 172.21.15.7:0/903149562 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec004390 con 0x7fdff014fda0 2023-03-03T19:12:15.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdfe7fff700 1 -- 172.21.15.7:0/903149562 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec0036a0 con 0x7fdff014fda0 2023-03-03T19:12:15.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdff75f8700 1 -- 172.21.15.7:0/903149562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff014fda0 msgr2=0x7fdff0071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:15.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdff75f8700 1 --2- 172.21.15.7:0/903149562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff014fda0 0x7fdff0071890 secure :-1 s=READY pgs=3419 cs=0 l=1 rev1=1 crypto rx=0x7fdff00710a0 tx=0x7fdfec009bd0 comp rx=0 tx=0).stop 2023-03-03T19:12:15.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdff75f8700 1 -- 172.21.15.7:0/903149562 shutdown_connections 2023-03-03T19:12:15.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdff75f8700 1 --2- 172.21.15.7:0/903149562 >> v2:172.21.15.143:3300/0 conn(0x7fdff014f9d0 0x7fdff0156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:15.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdff75f8700 1 --2- 172.21.15.7:0/903149562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff014fda0 0x7fdff0071890 unknown :-1 s=CLOSED pgs=3419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:15.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdff75f8700 1 -- 172.21.15.7:0/903149562 >> 172.21.15.7:0/903149562 conn(0x7fdff00c8900 msgr2=0x7fdff00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:12:15.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdff75f8700 1 -- 172.21.15.7:0/903149562 shutdown_connections 2023-03-03T19:12:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.070+0000 7fdff75f8700 1 -- 172.21.15.7:0/903149562 wait complete. 2023-03-03T19:12:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.071+0000 7fdff75f8700 1 Processor -- start 2023-03-03T19:12:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff75f8700 1 -- start start 2023-03-03T19:12:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff75f8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff00abe20 0x7fdff006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff75f8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdff014f9d0 0x7fdff014dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff75f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdff014fda0 0x7fdff0149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff75f8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff014e800 con 0x7fdff00abe20 2023-03-03T19:12:15.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff75f8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff0155630 con 0x7fdff014f9d0 2023-03-03T19:12:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff75f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff0155b60 con 0x7fdff014fda0 2023-03-03T19:12:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff5394700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff00abe20 0x7fdff006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff5394700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff00abe20 0x7fdff006a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40416/0 (socket says 172.21.15.7:40416) 2023-03-03T19:12:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff5394700 1 -- 172.21.15.7:0/15647703 learned_addr learned my addr 172.21.15.7:0/15647703 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:12:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff4b93700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdff014f9d0 0x7fdff014dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:15.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.072+0000 7fdff5b95700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdff014fda0 0x7fdff0149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdff5394700 1 -- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdff014fda0 msgr2=0x7fdff0149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdff5394700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdff014fda0 0x7fdff0149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdff5394700 1 -- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdff014f9d0 msgr2=0x7fdff014dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdff5394700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdff014f9d0 0x7fdff014dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdff5394700 1 -- 172.21.15.7:0/15647703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfec009550 con 0x7fdff00abe20 2023-03-03T19:12:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdff5b95700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdff014fda0 0x7fdff0149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:12:15.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdff5394700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff00abe20 0x7fdff006a2b0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000e810 tx=0x7fdfe000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdfe5ffb700 1 -- 172.21.15.7:0/15647703 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0013610 con 0x7fdff00abe20 2023-03-03T19:12:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdfe5ffb700 1 -- 172.21.15.7:0/15647703 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe0013770 con 0x7fdff00abe20 2023-03-03T19:12:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdff75f8700 1 -- 172.21.15.7:0/15647703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff00ac350 con 0x7fdff00abe20 2023-03-03T19:12:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdfe5ffb700 1 -- 172.21.15.7:0/15647703 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0013b20 con 0x7fdff00abe20 2023-03-03T19:12:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.073+0000 7fdff75f8700 1 -- 172.21.15.7:0/15647703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff00ab630 con 0x7fdff00abe20 2023-03-03T19:12:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.074+0000 7fdff75f8700 1 -- 172.21.15.7:0/15647703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff00c3e10 con 0x7fdff00abe20 2023-03-03T19:12:15.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.075+0000 7fdfe5ffb700 1 -- 172.21.15.7:0/15647703 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdfe0023070 con 0x7fdff00abe20 2023-03-03T19:12:15.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.075+0000 7fdfe5ffb700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdfd40682f0 0x7fdfd406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:15.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.075+0000 7fdfe5ffb700 1 -- 172.21.15.7:0/15647703 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdfe0085de0 con 0x7fdff00abe20 2023-03-03T19:12:15.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.076+0000 7fdff4b93700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdfd40682f0 0x7fdfd406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:15.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.076+0000 7fdff4b93700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdfd40682f0 0x7fdfd406a7a0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fdff01501d0 tx=0x7fdfec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:15.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.080+0000 7fdfe5ffb700 1 -- 172.21.15.7:0/15647703 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdfe0052100 con 0x7fdff00abe20 2023-03-03T19:12:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.321+0000 7fdff75f8700 1 -- 172.21.15.7:0/15647703 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdff00ac4e0 con 0x7fdff00abe20 2023-03-03T19:12:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.322+0000 7fdfe5ffb700 1 -- 172.21.15.7:0/15647703 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdfe0055720 con 0x7fdff00abe20 2023-03-03T19:12:15.322 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:12:15.323 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:12:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 -- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdfd40682f0 msgr2=0x7fdfd406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdfd40682f0 0x7fdfd406a7a0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fdff01501d0 tx=0x7fdfec008030 comp rx=0 tx=0).stop 2023-03-03T19:12:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 -- 172.21.15.7:0/15647703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff00abe20 msgr2=0x7fdff006a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff00abe20 0x7fdff006a2b0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7fdfe000e810 tx=0x7fdfe000eb20 comp rx=0 tx=0).stop 2023-03-03T19:12:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 -- 172.21.15.7:0/15647703 shutdown_connections 2023-03-03T19:12:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdff014fda0 0x7fdff0149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdfd40682f0 0x7fdfd406a7a0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdff00abe20 0x7fdff006a2b0 unknown :-1 s=CLOSED pgs=3420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 --2- 172.21.15.7:0/15647703 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdff014f9d0 0x7fdff014dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 -- 172.21.15.7:0/15647703 >> 172.21.15.7:0/15647703 conn(0x7fdff00c8900 msgr2=0x7fdff00c1870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:12:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 -- 172.21.15.7:0/15647703 shutdown_connections 2023-03-03T19:12:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:15.324+0000 7fdff75f8700 1 -- 172.21.15.7:0/15647703 wait complete. 2023-03-03T19:12:15.326 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:12:15.336 INFO:tasks.mds_thrash.fs.[b]:waiting for 16 secs before thrashing 2023-03-03T19:12:15.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:12:15.753 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:12:22.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:12:22.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.361+0000 7facea516700 1 Processor -- start 2023-03-03T19:12:22.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.361+0000 7facea516700 1 -- start start 2023-03-03T19:12:22.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.361+0000 7facea516700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face414b0d0 0x7face4071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:22.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.361+0000 7facea516700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7face4071d40 0x7face4158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.361+0000 7facea516700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face40a9c20 con 0x7face40aad20 2023-03-03T19:12:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.361+0000 7facea516700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face4154f10 con 0x7face4071d40 2023-03-03T19:12:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.361+0000 7facea516700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face40ac3c0 con 0x7face414b0d0 2023-03-03T19:12:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face414b0d0 0x7face4071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face8ab3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7face4071d40 0x7face4158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face8ab3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7face4071d40 0x7face4158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:36202/0 (socket says 172.21.15.7:36202) 2023-03-03T19:12:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face3fff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face414b0d0 0x7face4071800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:48500/0 (socket says 172.21.15.7:48500) 2023-03-03T19:12:22.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face37fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7face40aad20 0x7face4157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:37928/0 (socket says 172.21.15.7:37928) 2023-03-03T19:12:22.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face8ab3700 1 -- 172.21.15.7:0/1694897163 learned_addr learned my addr 172.21.15.7:0/1694897163 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:12:22.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face3fff700 1 -- 172.21.15.7:0/1694897163 >> v2:172.21.15.143:3300/0 conn(0x7face4071d40 msgr2=0x7face4158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:12:22.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face3fff700 1 --2- 172.21.15.7:0/1694897163 >> v2:172.21.15.143:3300/0 conn(0x7face4071d40 0x7face4158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:22.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face3fff700 1 -- 172.21.15.7:0/1694897163 >> v1:172.21.15.143:6789/0 conn(0x7face40aad20 legacy=0x7face4157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:22.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face3fff700 1 -- 172.21.15.7:0/1694897163 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face406fc70 con 0x7face414b0d0 2023-03-03T19:12:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.362+0000 7face8ab3700 1 --2- 172.21.15.7:0/1694897163 >> v2:172.21.15.143:3300/0 conn(0x7face4071d40 0x7face4158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:12:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7face3fff700 1 --2- 172.21.15.7:0/1694897163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face414b0d0 0x7face4071800 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7facd4009840 tx=0x7facd4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50e343c9ad630e50 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:22.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7face2ffd700 1 -- 172.21.15.7:0/1694897163 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4011830 con 0x7face414b0d0 2023-03-03T19:12:22.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7face2ffd700 1 -- 172.21.15.7:0/1694897163 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd4011990 con 0x7face414b0d0 2023-03-03T19:12:22.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7face2ffd700 1 -- 172.21.15.7:0/1694897163 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4011d20 con 0x7face414b0d0 2023-03-03T19:12:22.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7facea516700 1 -- 172.21.15.7:0/1694897163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face414b0d0 msgr2=0x7face4071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:22.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7facea516700 1 --2- 172.21.15.7:0/1694897163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face414b0d0 0x7face4071800 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7facd4009840 tx=0x7facd4009b50 comp rx=0 tx=0).stop 2023-03-03T19:12:22.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7facea516700 1 -- 172.21.15.7:0/1694897163 shutdown_connections 2023-03-03T19:12:22.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7facea516700 1 --2- 172.21.15.7:0/1694897163 >> v2:172.21.15.143:3300/0 conn(0x7face4071d40 0x7face4158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:22.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7facea516700 1 --2- 172.21.15.7:0/1694897163 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face414b0d0 0x7face4071800 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:22.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7facea516700 1 -- 172.21.15.7:0/1694897163 >> 172.21.15.7:0/1694897163 conn(0x7face40c8900 msgr2=0x7face40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:12:22.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7facea516700 1 -- 172.21.15.7:0/1694897163 shutdown_connections 2023-03-03T19:12:22.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.363+0000 7facea516700 1 -- 172.21.15.7:0/1694897163 wait complete. 2023-03-03T19:12:22.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.365+0000 7facea516700 1 Processor -- start 2023-03-03T19:12:22.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.365+0000 7facea516700 1 -- start start 2023-03-03T19:12:22.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.365+0000 7facea516700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4071d40 0x7face4156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:22.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.365+0000 7facea516700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40aad20 0x7face41ed150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:22.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.365+0000 7facea516700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face414b0d0 0x7face41ed690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:22.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.365+0000 7facea516700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face4153770 con 0x7face4071d40 2023-03-03T19:12:22.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.365+0000 7facea516700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face41e0000 con 0x7face414b0d0 2023-03-03T19:12:22.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.365+0000 7facea516700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face4153160 con 0x7face40aad20 2023-03-03T19:12:22.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4071d40 0x7face4156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:22.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4071d40 0x7face4156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40434/0 (socket says 172.21.15.7:40434) 2023-03-03T19:12:22.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face3fff700 1 -- 172.21.15.7:0/816064256 learned_addr learned my addr 172.21.15.7:0/816064256 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:12:22.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face8ab3700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face414b0d0 0x7face41ed690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:22.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face37fe700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40aad20 0x7face41ed150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:22.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face3fff700 1 -- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40aad20 msgr2=0x7face41ed150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:22.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face3fff700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40aad20 0x7face41ed150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:22.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face3fff700 1 -- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face414b0d0 msgr2=0x7face41ed690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:22.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face3fff700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face414b0d0 0x7face41ed690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:22.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face3fff700 1 -- 172.21.15.7:0/816064256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc009b30 con 0x7face4071d40 2023-03-03T19:12:22.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face3fff700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4071d40 0x7face4156b40 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7facd4000c00 tx=0x7facd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:22.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face0ff9700 1 -- 172.21.15.7:0/816064256 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4004510 con 0x7face4071d40 2023-03-03T19:12:22.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face0ff9700 1 -- 172.21.15.7:0/816064256 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd40036a0 con 0x7face4071d40 2023-03-03T19:12:22.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7facea516700 1 -- 172.21.15.7:0/816064256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd40094a0 con 0x7face4071d40 2023-03-03T19:12:22.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7face0ff9700 1 -- 172.21.15.7:0/816064256 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4003a30 con 0x7face4071d40 2023-03-03T19:12:22.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.366+0000 7facea516700 1 -- 172.21.15.7:0/816064256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face406f6b0 con 0x7face4071d40 2023-03-03T19:12:22.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.367+0000 7facea516700 1 -- 172.21.15.7:0/816064256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face4155160 con 0x7face4071d40 2023-03-03T19:12:22.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.368+0000 7face0ff9700 1 -- 172.21.15.7:0/816064256 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7facd400f050 con 0x7face4071d40 2023-03-03T19:12:22.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.368+0000 7face0ff9700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facd00682f0 0x7facd006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:22.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.368+0000 7face0ff9700 1 -- 172.21.15.7:0/816064256 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7facd40873e0 con 0x7face4071d40 2023-03-03T19:12:22.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.368+0000 7face37fe700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facd00682f0 0x7facd006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:22.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.369+0000 7face37fe700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facd00682f0 0x7facd006a7a0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7face40c44a0 tx=0x7facd8007ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:22.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.373+0000 7face0ff9700 1 -- 172.21.15.7:0/816064256 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7facd4053610 con 0x7face4071d40 2023-03-03T19:12:22.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.612+0000 7facea516700 1 -- 172.21.15.7:0/816064256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7face406f990 con 0x7face4071d40 2023-03-03T19:12:22.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.612+0000 7face0ff9700 1 -- 172.21.15.7:0/816064256 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7facd4015090 con 0x7face4071d40 2023-03-03T19:12:22.622 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:12:22.622 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:12:22.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.614+0000 7facea516700 1 -- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facd00682f0 msgr2=0x7facd006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:22.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.614+0000 7facea516700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facd00682f0 0x7facd006a7a0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7face40c44a0 tx=0x7facd8007ed0 comp rx=0 tx=0).stop 2023-03-03T19:12:22.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.614+0000 7facea516700 1 -- 172.21.15.7:0/816064256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4071d40 msgr2=0x7face4156b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:22.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.614+0000 7facea516700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4071d40 0x7face4156b40 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7facd4000c00 tx=0x7facd4004750 comp rx=0 tx=0).stop 2023-03-03T19:12:22.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.614+0000 7facea516700 1 -- 172.21.15.7:0/816064256 shutdown_connections 2023-03-03T19:12:22.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.614+0000 7facea516700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7face414b0d0 0x7face41ed690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:22.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.614+0000 7facea516700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7facd00682f0 0x7facd006a7a0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:22.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.614+0000 7facea516700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7face4071d40 0x7face4156b40 unknown :-1 s=CLOSED pgs=3421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:22.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.614+0000 7facea516700 1 --2- 172.21.15.7:0/816064256 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7face40aad20 0x7face41ed150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:22.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.614+0000 7facea516700 1 -- 172.21.15.7:0/816064256 >> 172.21.15.7:0/816064256 conn(0x7face40c8900 msgr2=0x7face40c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:12:22.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.615+0000 7facea516700 1 -- 172.21.15.7:0/816064256 shutdown_connections 2023-03-03T19:12:22.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:22.615+0000 7facea516700 1 -- 172.21.15.7:0/816064256 wait complete. 2023-03-03T19:12:22.625 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:12:22.626 INFO:tasks.mds_thrash.fs.[a]:waiting for 32 secs before thrashing 2023-03-03T19:12:31.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:12:31.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.476+0000 7fb79a62a700 1 Processor -- start 2023-03-03T19:12:31.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.476+0000 7fb79a62a700 1 -- start start 2023-03-03T19:12:31.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb79a62a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb794151ab0 0x7fb7940736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb79a62a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794073c30 0x7fb79406bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb79a62a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79406d780 con 0x7fb79406c3e0 2023-03-03T19:12:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb79a62a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb79406cb70 con 0x7fb794151ab0 2023-03-03T19:12:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb79a62a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79406ce50 con 0x7fb794073c30 2023-03-03T19:12:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb793fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb794151ab0 0x7fb7940736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:31.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb793fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb794151ab0 0x7fb7940736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40448/0 (socket says 172.21.15.7:40448) 2023-03-03T19:12:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb793fff700 1 -- 172.21.15.7:0/2512024816 learned_addr learned my addr 172.21.15.7:0/2512024816 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:12:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb7937fe700 1 --2- 172.21.15.7:0/2512024816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794073c30 0x7fb79406bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb793fff700 1 -- 172.21.15.7:0/2512024816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794073c30 msgr2=0x7fb79406bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:12:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb793fff700 1 --2- 172.21.15.7:0/2512024816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794073c30 0x7fb79406bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb793fff700 1 -- 172.21.15.7:0/2512024816 >> v1:172.21.15.143:6789/0 conn(0x7fb79406c3e0 legacy=0x7fb794157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:12:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb793fff700 1 -- 172.21.15.7:0/2512024816 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79406d130 con 0x7fb794151ab0 2023-03-03T19:12:31.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.477+0000 7fb7937fe700 1 --2- 172.21.15.7:0/2512024816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794073c30 0x7fb79406bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:12:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb793fff700 1 --2- 172.21.15.7:0/2512024816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794151ab0 0x7fb7940736f0 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7fb7840098f0 tx=0x7fb784009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5721cf297cc97e78 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb792ffd700 1 -- 172.21.15.7:0/2512024816 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784004230 con 0x7fb794151ab0 2023-03-03T19:12:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb792ffd700 1 -- 172.21.15.7:0/2512024816 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb784004390 con 0x7fb794151ab0 2023-03-03T19:12:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb792ffd700 1 -- 172.21.15.7:0/2512024816 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7840036a0 con 0x7fb794151ab0 2023-03-03T19:12:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb79a62a700 1 -- 172.21.15.7:0/2512024816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794151ab0 msgr2=0x7fb7940736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb79a62a700 1 --2- 172.21.15.7:0/2512024816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794151ab0 0x7fb7940736f0 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7fb7840098f0 tx=0x7fb784009c00 comp rx=0 tx=0).stop 2023-03-03T19:12:31.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb79a62a700 1 -- 172.21.15.7:0/2512024816 shutdown_connections 2023-03-03T19:12:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb79a62a700 1 --2- 172.21.15.7:0/2512024816 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794151ab0 0x7fb7940736f0 unknown :-1 s=CLOSED pgs=3422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb79a62a700 1 --2- 172.21.15.7:0/2512024816 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794073c30 0x7fb79406bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb79a62a700 1 -- 172.21.15.7:0/2512024816 >> 172.21.15.7:0/2512024816 conn(0x7fb7940a94d0 msgr2=0x7fb7940a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:12:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb79a62a700 1 -- 172.21.15.7:0/2512024816 shutdown_connections 2023-03-03T19:12:31.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.478+0000 7fb79a62a700 1 -- 172.21.15.7:0/2512024816 wait complete. 2023-03-03T19:12:31.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.479+0000 7fb79a62a700 1 Processor -- start 2023-03-03T19:12:31.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.480+0000 7fb79a62a700 1 -- start start 2023-03-03T19:12:31.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.480+0000 7fb79a62a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb79406c3e0 0x7fb794155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.480+0000 7fb79a62a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794073c30 0x7fb7941ed770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.480+0000 7fb79a62a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794151ab0 0x7fb7941edcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.480+0000 7fb79a62a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79414cbd0 con 0x7fb794073c30 2023-03-03T19:12:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.480+0000 7fb79a62a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7940ac230 con 0x7fb79406c3e0 2023-03-03T19:12:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.480+0000 7fb79a62a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7940ac850 con 0x7fb794151ab0 2023-03-03T19:12:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.480+0000 7fb7937fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794073c30 0x7fb7941ed770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:31.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb7937fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794073c30 0x7fb7941ed770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40458/0 (socket says 172.21.15.7:40458) 2023-03-03T19:12:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb7937fe700 1 -- 172.21.15.7:0/1931759985 learned_addr learned my addr 172.21.15.7:0/1931759985 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:12:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb793fff700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb79406c3e0 0x7fb794155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb798bc7700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794151ab0 0x7fb7941edcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb7937fe700 1 -- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794151ab0 msgr2=0x7fb7941edcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb7937fe700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794151ab0 0x7fb7941edcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb7937fe700 1 -- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb79406c3e0 msgr2=0x7fb794155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:31.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb7937fe700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb79406c3e0 0x7fb794155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb7937fe700 1 -- 172.21.15.7:0/1931759985 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb784009550 con 0x7fb794073c30 2023-03-03T19:12:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb798bc7700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794151ab0 0x7fb7941edcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:12:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb793fff700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb79406c3e0 0x7fb794155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:12:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb7937fe700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794073c30 0x7fb7941ed770 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7fb78800c750 tx=0x7fb78800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb790ff9700 1 -- 172.21.15.7:0/1931759985 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788011800 con 0x7fb794073c30 2023-03-03T19:12:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb790ff9700 1 -- 172.21.15.7:0/1931759985 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb788011960 con 0x7fb794073c30 2023-03-03T19:12:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb79a62a700 1 -- 172.21.15.7:0/1931759985 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb794151e80 con 0x7fb794073c30 2023-03-03T19:12:31.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb790ff9700 1 -- 172.21.15.7:0/1931759985 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788011cd0 con 0x7fb794073c30 2023-03-03T19:12:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.481+0000 7fb79a62a700 1 -- 172.21.15.7:0/1931759985 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb79406d410 con 0x7fb794073c30 2023-03-03T19:12:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.482+0000 7fb79a62a700 1 -- 172.21.15.7:0/1931759985 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb794152010 con 0x7fb794073c30 2023-03-03T19:12:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.483+0000 7fb790ff9700 1 -- 172.21.15.7:0/1931759985 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb788020070 con 0x7fb794073c30 2023-03-03T19:12:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.483+0000 7fb790ff9700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7800682f0 0x7fb78006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.483+0000 7fb790ff9700 1 -- 172.21.15.7:0/1931759985 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb788085ff0 con 0x7fb794073c30 2023-03-03T19:12:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.483+0000 7fb793fff700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7800682f0 0x7fb78006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:31.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.484+0000 7fb793fff700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7800682f0 0x7fb78006a7a0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7fb784009130 tx=0x7fb784008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:31.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.487+0000 7fb790ff9700 1 -- 172.21.15.7:0/1931759985 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb788052390 con 0x7fb794073c30 2023-03-03T19:12:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.726+0000 7fb79a62a700 1 -- 172.21.15.7:0/1931759985 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb794074bb0 con 0x7fb794073c30 2023-03-03T19:12:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.727+0000 7fb790ff9700 1 -- 172.21.15.7:0/1931759985 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb78801e090 con 0x7fb794073c30 2023-03-03T19:12:31.727 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:12:31.728 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:12:31.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 -- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7800682f0 msgr2=0x7fb78006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:31.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7800682f0 0x7fb78006a7a0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7fb784009130 tx=0x7fb784008030 comp rx=0 tx=0).stop 2023-03-03T19:12:31.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 -- 172.21.15.7:0/1931759985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794073c30 msgr2=0x7fb7941ed770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:31.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794073c30 0x7fb7941ed770 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7fb78800c750 tx=0x7fb78800cb10 comp rx=0 tx=0).stop 2023-03-03T19:12:31.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 -- 172.21.15.7:0/1931759985 shutdown_connections 2023-03-03T19:12:31.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb794151ab0 0x7fb7941edcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:31.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb79406c3e0 0x7fb794155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:31.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7800682f0 0x7fb78006a7a0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:31.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 --2- 172.21.15.7:0/1931759985 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb794073c30 0x7fb7941ed770 unknown :-1 s=CLOSED pgs=3423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:31.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 -- 172.21.15.7:0/1931759985 >> 172.21.15.7:0/1931759985 conn(0x7fb7940a94d0 msgr2=0x7fb7940a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:12:31.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 -- 172.21.15.7:0/1931759985 shutdown_connections 2023-03-03T19:12:31.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:31.729+0000 7fb79a62a700 1 -- 172.21.15.7:0/1931759985 wait complete. 2023-03-03T19:12:31.732 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:12:31.742 INFO:tasks.mds_thrash.fs.[b]:waiting for 33 secs before thrashing 2023-03-03T19:12:45.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:12:45.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:12:54.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:12:54.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.763+0000 7f21778c2700 1 Processor -- start 2023-03-03T19:12:54.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.763+0000 7f21778c2700 1 -- start start 2023-03-03T19:12:54.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f21778c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c1da0 0x7f21701550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:54.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f21778c2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21700c7a60 con 0x7f21700c3e30 2023-03-03T19:12:54.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f21778c2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21700c3620 con 0x7f217006d870 2023-03-03T19:12:54.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f21778c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21700c2af0 con 0x7f21700c1da0 2023-03-03T19:12:54.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f2175e5f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f21700c3e30 0x7f217015ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34458/0 (socket says 172.21.15.7:34458) 2023-03-03T19:12:54.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f2175e5f700 1 -- 172.21.15.7:0/2209648973 learned_addr learned my addr 172.21.15.7:0/2209648973 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:12:54.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f2174e5d700 1 --2- 172.21.15.7:0/2209648973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c1da0 0x7f21701550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:54.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f2167fff700 1 -- 172.21.15.7:0/2209648973 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260726501 0 0) 0x7f21700c7a60 con 0x7f21700c3e30 2023-03-03T19:12:54.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f2167fff700 1 -- 172.21.15.7:0/2209648973 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f215c0035b0 con 0x7f21700c3e30 2023-03-03T19:12:54.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f2167fff700 1 -- 172.21.15.7:0/2209648973 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758106417 0 0) 0x7f21700c3620 con 0x7f217006d870 2023-03-03T19:12:54.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f2167fff700 1 -- 172.21.15.7:0/2209648973 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21700c7a60 con 0x7f217006d870 2023-03-03T19:12:54.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f2167fff700 1 -- 172.21.15.7:0/2209648973 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 303173949 0 0) 0x7f215c0035b0 con 0x7f21700c3e30 2023-03-03T19:12:54.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f2174e5d700 1 -- 172.21.15.7:0/2209648973 >> v1:172.21.15.143:6789/0 conn(0x7f217006d870 legacy=0x7f21700c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:54.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.764+0000 7f2174e5d700 1 -- 172.21.15.7:0/2209648973 >> v1:172.21.15.7:6789/0 conn(0x7f21700c3e30 legacy=0x7f217015ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:54.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.765+0000 7f2174e5d700 1 -- 172.21.15.7:0/2209648973 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21701545e0 con 0x7f21700c1da0 2023-03-03T19:12:54.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.765+0000 7f2174e5d700 1 --2- 172.21.15.7:0/2209648973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c1da0 0x7f21701550c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2160009870 tx=0x7f2160009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1528b2d5f1478b37 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:54.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.765+0000 7f2167fff700 1 -- 172.21.15.7:0/2209648973 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680019f0 con 0x7f21700c1da0 2023-03-03T19:12:54.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.765+0000 7f2167fff700 1 -- 172.21.15.7:0/2209648973 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2160004030 con 0x7f21700c1da0 2023-03-03T19:12:54.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.765+0000 7f2167fff700 1 -- 172.21.15.7:0/2209648973 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600043c0 con 0x7f21700c1da0 2023-03-03T19:12:54.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.765+0000 7f21778c2700 1 -- 172.21.15.7:0/2209648973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c1da0 msgr2=0x7f21701550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:54.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.765+0000 7f21778c2700 1 --2- 172.21.15.7:0/2209648973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c1da0 0x7f21701550c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2160009870 tx=0x7f2160009b80 comp rx=0 tx=0).stop 2023-03-03T19:12:54.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.765+0000 7f21778c2700 1 -- 172.21.15.7:0/2209648973 shutdown_connections 2023-03-03T19:12:54.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.765+0000 7f21778c2700 1 --2- 172.21.15.7:0/2209648973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c1da0 0x7f21701550c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:54.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.765+0000 7f21778c2700 1 -- 172.21.15.7:0/2209648973 >> 172.21.15.7:0/2209648973 conn(0x7f217006c110 msgr2=0x7f217006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:12:54.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.766+0000 7f21778c2700 1 -- 172.21.15.7:0/2209648973 shutdown_connections 2023-03-03T19:12:54.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.766+0000 7f21778c2700 1 -- 172.21.15.7:0/2209648973 wait complete. 2023-03-03T19:12:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.767+0000 7f21778c2700 1 Processor -- start 2023-03-03T19:12:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.767+0000 7f21778c2700 1 -- start start 2023-03-03T19:12:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.767+0000 7f21778c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217006d870 0x7f21700c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.767+0000 7f21778c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21700c1da0 0x7f21701393a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:54.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.767+0000 7f21778c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c3e30 0x7f21701eb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:54.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.767+0000 7f21778c2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217013a610 con 0x7f217006d870 2023-03-03T19:12:54.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.767+0000 7f21778c2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2170139e80 con 0x7f21700c1da0 2023-03-03T19:12:54.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.767+0000 7f21778c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21701edc60 con 0x7f21700c3e30 2023-03-03T19:12:54.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.767+0000 7f217565e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217006d870 0x7f21700c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:54.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f217565e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217006d870 0x7f21700c6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40480/0 (socket says 172.21.15.7:40480) 2023-03-03T19:12:54.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f217565e700 1 -- 172.21.15.7:0/3442662759 learned_addr learned my addr 172.21.15.7:0/3442662759 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:12:54.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f2174e5d700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21700c1da0 0x7f21701393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:54.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f2175e5f700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c3e30 0x7f21701eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:54.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f217565e700 1 -- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c3e30 msgr2=0x7f21701eb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:54.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f217565e700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c3e30 0x7f21701eb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:54.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f217565e700 1 -- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21700c1da0 msgr2=0x7f21701393a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:54.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f217565e700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21700c1da0 0x7f21701393a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:54.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f217565e700 1 -- 172.21.15.7:0/3442662759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21600094a0 con 0x7f217006d870 2023-03-03T19:12:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f217565e700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217006d870 0x7f21700c6a70 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c520 tx=0x7f216c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f2165ffb700 1 -- 172.21.15.7:0/3442662759 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c0052d0 con 0x7f217006d870 2023-03-03T19:12:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f2165ffb700 1 -- 172.21.15.7:0/3442662759 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c005430 con 0x7f217006d870 2023-03-03T19:12:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f2165ffb700 1 -- 172.21.15.7:0/3442662759 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c0038c0 con 0x7f217006d870 2023-03-03T19:12:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f21778c2700 1 -- 172.21.15.7:0/3442662759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2170154ba0 con 0x7f217006d870 2023-03-03T19:12:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.768+0000 7f21778c2700 1 -- 172.21.15.7:0/3442662759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21701548c0 con 0x7f217006d870 2023-03-03T19:12:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.769+0000 7f21778c2700 1 -- 172.21.15.7:0/3442662759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217006dc40 con 0x7f217006d870 2023-03-03T19:12:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.770+0000 7f2165ffb700 1 -- 172.21.15.7:0/3442662759 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f216c019030 con 0x7f217006d870 2023-03-03T19:12:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.770+0000 7f2165ffb700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21540682f0 0x7f215406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:12:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.770+0000 7f2165ffb700 1 -- 172.21.15.7:0/3442662759 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f216c0855c0 con 0x7f217006d870 2023-03-03T19:12:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.770+0000 7f2174e5d700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21540682f0 0x7f215406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:12:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.771+0000 7f2174e5d700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21540682f0 0x7f215406a7a0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f21600090b0 tx=0x7f2160008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:12:54.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:54.774+0000 7f2165ffb700 1 -- 172.21.15.7:0/3442662759 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f216c051960 con 0x7f217006d870 2023-03-03T19:12:55.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.013+0000 7f21778c2700 1 -- 172.21.15.7:0/3442662759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2170069730 con 0x7f217006d870 2023-03-03T19:12:55.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.014+0000 7f2165ffb700 1 -- 172.21.15.7:0/3442662759 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f216c054f80 con 0x7f217006d870 2023-03-03T19:12:55.014 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:12:55.014 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:12:55.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 -- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21540682f0 msgr2=0x7f215406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:55.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21540682f0 0x7f215406a7a0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f21600090b0 tx=0x7f2160008030 comp rx=0 tx=0).stop 2023-03-03T19:12:55.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 -- 172.21.15.7:0/3442662759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217006d870 msgr2=0x7f21700c6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:12:55.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217006d870 0x7f21700c6a70 secure :-1 s=READY pgs=3425 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c520 tx=0x7f216c00c8e0 comp rx=0 tx=0).stop 2023-03-03T19:12:55.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 -- 172.21.15.7:0/3442662759 shutdown_connections 2023-03-03T19:12:55.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f21700c3e30 0x7f21701eb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:55.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21540682f0 0x7f215406a7a0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:55.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f217006d870 0x7f21700c6a70 unknown :-1 s=CLOSED pgs=3425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:55.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 --2- 172.21.15.7:0/3442662759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f21700c1da0 0x7f21701393a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:12:55.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 -- 172.21.15.7:0/3442662759 >> 172.21.15.7:0/3442662759 conn(0x7f217006c110 msgr2=0x7f21700c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:12:55.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 -- 172.21.15.7:0/3442662759 shutdown_connections 2023-03-03T19:12:55.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:12:55.016+0000 7f21778c2700 1 -- 172.21.15.7:0/3442662759 wait complete. 2023-03-03T19:12:55.018 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:12:55.028 INFO:tasks.mds_thrash.fs.[a]:waiting for 86 secs before thrashing 2023-03-03T19:13:04.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:13:04.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.880+0000 7fbdbbeb4700 1 Processor -- start 2023-03-03T19:13:04.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.880+0000 7fbdbbeb4700 1 -- start start 2023-03-03T19:13:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdbbeb4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbdb4151b20 0x7fbdb40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:13:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdbbeb4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdb40c2950 0x7fbdb4157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:13:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdbbeb4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb406cb70 con 0x7fbdb4073c30 2023-03-03T19:13:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdbbeb4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb406c2f0 con 0x7fbdb4151b20 2023-03-03T19:13:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdbbeb4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb406ce50 con 0x7fbdb40c2950 2023-03-03T19:13:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb9c50700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbdb4151b20 0x7fbdb40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:13:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdba451700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbdb4073c30 0x7fbdb4158020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34478/0 (socket says 172.21.15.7:34478) 2023-03-03T19:13:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdba451700 1 -- 172.21.15.7:0/2521212681 learned_addr learned my addr 172.21.15.7:0/2521212681 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:13:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb944f700 1 --2- 172.21.15.7:0/2521212681 >> v2:172.21.15.143:3300/0 conn(0x7fbdb40c2950 0x7fbdb4157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:13:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb9c50700 1 -- 172.21.15.7:0/2521212681 >> v2:172.21.15.143:3300/0 conn(0x7fbdb40c2950 msgr2=0x7fbdb4157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:13:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb9c50700 1 --2- 172.21.15.7:0/2521212681 >> v2:172.21.15.143:3300/0 conn(0x7fbdb40c2950 0x7fbdb4157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:13:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb9c50700 1 -- 172.21.15.7:0/2521212681 >> v1:172.21.15.7:6789/0 conn(0x7fbdb4073c30 legacy=0x7fbdb4158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:13:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb9c50700 1 -- 172.21.15.7:0/2521212681 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb406d130 con 0x7fbdb4151b20 2023-03-03T19:13:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb8c4e700 1 -- 172.21.15.7:0/2521212681 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457965789 0 0) 0x7fbdb406cb70 con 0x7fbdb4073c30 2023-03-03T19:13:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb944f700 1 --2- 172.21.15.7:0/2521212681 >> v2:172.21.15.143:3300/0 conn(0x7fbdb40c2950 0x7fbdb4157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:13:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb9c50700 1 --2- 172.21.15.7:0/2521212681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb4151b20 0x7fbdb40736f0 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7fbdb406b8d0 tx=0x7fbdb0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5044db7ef03ac8a5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:13:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb8c4e700 1 -- 172.21.15.7:0/2521212681 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0011830 con 0x7fbdb4151b20 2023-03-03T19:13:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.881+0000 7fbdb8c4e700 1 -- 172.21.15.7:0/2521212681 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb0011990 con 0x7fbdb4151b20 2023-03-03T19:13:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.882+0000 7fbdb8c4e700 1 -- 172.21.15.7:0/2521212681 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb0011d20 con 0x7fbdb4151b20 2023-03-03T19:13:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.882+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/2521212681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb4151b20 msgr2=0x7fbdb40736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:13:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.882+0000 7fbdbbeb4700 1 --2- 172.21.15.7:0/2521212681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb4151b20 0x7fbdb40736f0 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7fbdb406b8d0 tx=0x7fbdb0009b20 comp rx=0 tx=0).stop 2023-03-03T19:13:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.882+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/2521212681 shutdown_connections 2023-03-03T19:13:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.882+0000 7fbdbbeb4700 1 --2- 172.21.15.7:0/2521212681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb4151b20 0x7fbdb40736f0 unknown :-1 s=CLOSED pgs=3427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:13:04.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.882+0000 7fbdbbeb4700 1 --2- 172.21.15.7:0/2521212681 >> v2:172.21.15.143:3300/0 conn(0x7fbdb40c2950 0x7fbdb4157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:13:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.882+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/2521212681 >> 172.21.15.7:0/2521212681 conn(0x7fbdb40a94d0 msgr2=0x7fbdb40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:13:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.882+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/2521212681 shutdown_connections 2023-03-03T19:13:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.882+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/2521212681 wait complete. 2023-03-03T19:13:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.883+0000 7fbdbbeb4700 1 Processor -- start 2023-03-03T19:13:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.883+0000 7fbdbbeb4700 1 -- start start 2023-03-03T19:13:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdbbeb4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb4073c30 0x7fbdb40bef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:13:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdbbeb4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb40c2950 0x7fbdb40bf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:13:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdbbeb4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb4151b20 0x7fbdb40aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:13:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdbbeb4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb40bf480 con 0x7fbdb40c2950 2023-03-03T19:13:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdbbeb4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb41e97c0 con 0x7fbdb4151b20 2023-03-03T19:13:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdbbeb4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb41e9a60 con 0x7fbdb4073c30 2023-03-03T19:13:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdb944f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb40c2950 0x7fbdb40bf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:13:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdb944f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb40c2950 0x7fbdb40bf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40502/0 (socket says 172.21.15.7:40502) 2023-03-03T19:13:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdb944f700 1 -- 172.21.15.7:0/3517573676 learned_addr learned my addr 172.21.15.7:0/3517573676 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:13:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdb9c50700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb4073c30 0x7fbdb40bef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:13:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.884+0000 7fbdba451700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb4151b20 0x7fbdb40aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:13:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdb944f700 1 -- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb4073c30 msgr2=0x7fbdb40bef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:13:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdb944f700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb4073c30 0x7fbdb40bef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:13:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdb944f700 1 -- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb4151b20 msgr2=0x7fbdb40aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:13:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdb944f700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb4151b20 0x7fbdb40aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:13:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdb944f700 1 -- 172.21.15.7:0/3517573676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb00094a0 con 0x7fbdb40c2950 2023-03-03T19:13:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdba451700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb4151b20 0x7fbdb40aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:13:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdb944f700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb40c2950 0x7fbdb40bf610 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7fbda400c7f0 tx=0x7fbda400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:13:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdaa7fc700 1 -- 172.21.15.7:0/3517573676 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4011800 con 0x7fbdb40c2950 2023-03-03T19:13:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdaa7fc700 1 -- 172.21.15.7:0/3517573676 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda4011960 con 0x7fbdb40c2950 2023-03-03T19:13:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdaa7fc700 1 -- 172.21.15.7:0/3517573676 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda4011cf0 con 0x7fbdb40c2950 2023-03-03T19:13:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/3517573676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb4158730 con 0x7fbdb40c2950 2023-03-03T19:13:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.885+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/3517573676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb406d410 con 0x7fbdb40c2950 2023-03-03T19:13:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.886+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/3517573676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb4151ef0 con 0x7fbdb40c2950 2023-03-03T19:13:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.886+0000 7fbdaa7fc700 1 -- 172.21.15.7:0/3517573676 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbda4021070 con 0x7fbdb40c2950 2023-03-03T19:13:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.887+0000 7fbdaa7fc700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd980682f0 0x7fbd9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:13:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.887+0000 7fbdaa7fc700 1 -- 172.21.15.7:0/3517573676 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbda40860b0 con 0x7fbdb40c2950 2023-03-03T19:13:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.887+0000 7fbdb9c50700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd980682f0 0x7fbd9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:13:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.888+0000 7fbdb9c50700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd980682f0 0x7fbd9806a7a0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00090b0 tx=0x7fbdb0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:13:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:04.891+0000 7fbdaa7fc700 1 -- 172.21.15.7:0/3517573676 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbda40522e0 con 0x7fbdb40c2950 2023-03-03T19:13:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.133+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/3517573676 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdb41e9bd0 con 0x7fbdb40c2950 2023-03-03T19:13:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.133+0000 7fbdaa7fc700 1 -- 172.21.15.7:0/3517573676 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbda401f090 con 0x7fbdb40c2950 2023-03-03T19:13:05.135 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:13:05.135 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:13:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd980682f0 msgr2=0x7fbd9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:13:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd980682f0 0x7fbd9806a7a0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00090b0 tx=0x7fbdb0008030 comp rx=0 tx=0).stop 2023-03-03T19:13:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/3517573676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb40c2950 msgr2=0x7fbdb40bf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:13:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb40c2950 0x7fbdb40bf610 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7fbda400c7f0 tx=0x7fbda400cb00 comp rx=0 tx=0).stop 2023-03-03T19:13:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/3517573676 shutdown_connections 2023-03-03T19:13:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdb4151b20 0x7fbdb40aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:13:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdb4073c30 0x7fbdb40bef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:13:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbd980682f0 0x7fbd9806a7a0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:13:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 --2- 172.21.15.7:0/3517573676 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdb40c2950 0x7fbdb40bf610 unknown :-1 s=CLOSED pgs=3428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:13:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/3517573676 >> 172.21.15.7:0/3517573676 conn(0x7fbdb40a94d0 msgr2=0x7fbdb4070aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:13:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/3517573676 shutdown_connections 2023-03-03T19:13:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:13:05.136+0000 7fbdbbeb4700 1 -- 172.21.15.7:0/3517573676 wait complete. 2023-03-03T19:13:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:13:05.149 INFO:tasks.mds_thrash.fs.[b]:waiting for 118 secs before thrashing 2023-03-03T19:13:15.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:13:15.889 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:13:45.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:13:45.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:14:16.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:14:16.034 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:14:21.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:14:21.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.168+0000 7f648c665700 1 Processor -- start 2023-03-03T19:14:21.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.168+0000 7f648c665700 1 -- start start 2023-03-03T19:14:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f648c665700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f648414f9d0 0x7f6484156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:14:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f648c665700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f648414fda0 0x7f6484071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:14:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f648c665700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64840c2f30 con 0x7f64840abe20 2023-03-03T19:14:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f648c665700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64840c31d0 con 0x7f648414fda0 2023-03-03T19:14:21.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f648c665700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64841524a0 con 0x7f648414f9d0 2023-03-03T19:14:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f6489c00700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f648414fda0 0x7f6484071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:14:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f6489c00700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f648414fda0 0x7f6484071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40578/0 (socket says 172.21.15.7:40578) 2023-03-03T19:14:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f6489c00700 1 -- 172.21.15.7:0/1551604495 learned_addr learned my addr 172.21.15.7:0/1551604495 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:14:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f648a401700 1 --2- 172.21.15.7:0/1551604495 >> v2:172.21.15.143:3300/0 conn(0x7f648414f9d0 0x7f6484156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:14:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f6489c00700 1 -- 172.21.15.7:0/1551604495 >> v2:172.21.15.143:3300/0 conn(0x7f648414f9d0 msgr2=0x7f6484156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:14:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f6489c00700 1 --2- 172.21.15.7:0/1551604495 >> v2:172.21.15.143:3300/0 conn(0x7f648414f9d0 0x7f6484156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:14:21.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f6489c00700 1 -- 172.21.15.7:0/1551604495 >> v1:172.21.15.143:6789/0 conn(0x7f64840abe20 legacy=0x7f64841529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:14:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f6489c00700 1 -- 172.21.15.7:0/1551604495 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64840c34b0 con 0x7f648414fda0 2023-03-03T19:14:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f648a401700 1 --2- 172.21.15.7:0/1551604495 >> v2:172.21.15.143:3300/0 conn(0x7f648414f9d0 0x7f6484156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:14:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.169+0000 7f6489c00700 1 --2- 172.21.15.7:0/1551604495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414fda0 0x7f6484071890 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7f64840710a0 tx=0x7f6474009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1f0aff979da255 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:14:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.170+0000 7f64893ff700 1 -- 172.21.15.7:0/1551604495 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474004230 con 0x7f648414fda0 2023-03-03T19:14:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.170+0000 7f64893ff700 1 -- 172.21.15.7:0/1551604495 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6474004390 con 0x7f648414fda0 2023-03-03T19:14:21.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.170+0000 7f64893ff700 1 -- 172.21.15.7:0/1551604495 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64740036a0 con 0x7f648414fda0 2023-03-03T19:14:21.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.170+0000 7f648c665700 1 -- 172.21.15.7:0/1551604495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414fda0 msgr2=0x7f6484071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:14:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.170+0000 7f648c665700 1 --2- 172.21.15.7:0/1551604495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414fda0 0x7f6484071890 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7f64840710a0 tx=0x7f6474009bd0 comp rx=0 tx=0).stop 2023-03-03T19:14:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.170+0000 7f648c665700 1 -- 172.21.15.7:0/1551604495 shutdown_connections 2023-03-03T19:14:21.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.170+0000 7f648c665700 1 --2- 172.21.15.7:0/1551604495 >> v2:172.21.15.143:3300/0 conn(0x7f648414f9d0 0x7f6484156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:14:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.170+0000 7f648c665700 1 --2- 172.21.15.7:0/1551604495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414fda0 0x7f6484071890 unknown :-1 s=CLOSED pgs=3429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:14:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.170+0000 7f648c665700 1 -- 172.21.15.7:0/1551604495 >> 172.21.15.7:0/1551604495 conn(0x7f64840c8900 msgr2=0x7f64840c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:14:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.171+0000 7f648c665700 1 -- 172.21.15.7:0/1551604495 shutdown_connections 2023-03-03T19:14:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.171+0000 7f648c665700 1 -- 172.21.15.7:0/1551604495 wait complete. 2023-03-03T19:14:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.172+0000 7f648c665700 1 Processor -- start 2023-03-03T19:14:21.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.172+0000 7f648c665700 1 -- start start 2023-03-03T19:14:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f648c665700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64840abe20 0x7f648414a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:14:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f648c665700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414f9d0 0x7f648414c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:14:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f648c665700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f648414fda0 0x7f6484147fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:14:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f648c665700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648406f5d0 con 0x7f648414f9d0 2023-03-03T19:14:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f648c665700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6484149430 con 0x7f648414fda0 2023-03-03T19:14:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f648c665700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648414ca30 con 0x7f64840abe20 2023-03-03T19:14:21.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f6489c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414f9d0 0x7f648414c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:14:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f6489c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414f9d0 0x7f648414c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40584/0 (socket says 172.21.15.7:40584) 2023-03-03T19:14:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f6489c00700 1 -- 172.21.15.7:0/1929970629 learned_addr learned my addr 172.21.15.7:0/1929970629 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:14:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f6489c00700 1 -- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64840abe20 msgr2=0x7f648414a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:14:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f6489c00700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64840abe20 0x7f648414a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:14:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f6489c00700 1 -- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f648414fda0 msgr2=0x7f6484147fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:14:21.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f648ac02700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f648414fda0 0x7f6484147fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:14:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f6489c00700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f648414fda0 0x7f6484147fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:14:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f6489c00700 1 -- 172.21.15.7:0/1929970629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6474009550 con 0x7f648414f9d0 2023-03-03T19:14:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f6489c00700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414f9d0 0x7f648414c1d0 secure :-1 s=READY pgs=3430 cs=0 l=1 rev1=1 crypto rx=0x7f6474004000 tx=0x7f6474004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:14:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f647affd700 1 -- 172.21.15.7:0/1929970629 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474003d80 con 0x7f648414f9d0 2023-03-03T19:14:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f647affd700 1 -- 172.21.15.7:0/1929970629 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647401f070 con 0x7f648414f9d0 2023-03-03T19:14:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f647affd700 1 -- 172.21.15.7:0/1929970629 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474011870 con 0x7f648414f9d0 2023-03-03T19:14:21.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f648c665700 1 -- 172.21.15.7:0/1929970629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64840ac350 con 0x7f648414f9d0 2023-03-03T19:14:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.173+0000 7f648c665700 1 -- 172.21.15.7:0/1929970629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64840ab630 con 0x7f648414f9d0 2023-03-03T19:14:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.174+0000 7f648c665700 1 -- 172.21.15.7:0/1929970629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64840c3e10 con 0x7f648414f9d0 2023-03-03T19:14:21.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.175+0000 7f647affd700 1 -- 172.21.15.7:0/1929970629 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6474017020 con 0x7f648414f9d0 2023-03-03T19:14:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.175+0000 7f647affd700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64680682f0 0x7f646806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:14:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.175+0000 7f647affd700 1 -- 172.21.15.7:0/1929970629 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f64740879a0 con 0x7f648414f9d0 2023-03-03T19:14:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.175+0000 7f648a401700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64680682f0 0x7f646806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:14:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.176+0000 7f648a401700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64680682f0 0x7f646806a7a0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f6484150b30 tx=0x7f6480008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:14:21.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.180+0000 7f647affd700 1 -- 172.21.15.7:0/1929970629 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6474053c50 con 0x7f648414f9d0 2023-03-03T19:14:21.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.417+0000 7f648c665700 1 -- 172.21.15.7:0/1929970629 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f648404e6e0 con 0x7f648414f9d0 2023-03-03T19:14:21.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.417+0000 7f647affd700 1 -- 172.21.15.7:0/1929970629 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6474057270 con 0x7f648414f9d0 2023-03-03T19:14:21.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:14:21.419 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:14:21.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 -- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64680682f0 msgr2=0x7f646806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:14:21.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64680682f0 0x7f646806a7a0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f6484150b30 tx=0x7f6480008040 comp rx=0 tx=0).stop 2023-03-03T19:14:21.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 -- 172.21.15.7:0/1929970629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414f9d0 msgr2=0x7f648414c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:14:21.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414f9d0 0x7f648414c1d0 secure :-1 s=READY pgs=3430 cs=0 l=1 rev1=1 crypto rx=0x7f6474004000 tx=0x7f6474004750 comp rx=0 tx=0).stop 2023-03-03T19:14:21.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 -- 172.21.15.7:0/1929970629 shutdown_connections 2023-03-03T19:14:21.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f648414fda0 0x7f6484147fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:14:21.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f64840abe20 0x7f648414a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:14:21.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f64680682f0 0x7f646806a7a0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:14:21.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 --2- 172.21.15.7:0/1929970629 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f648414f9d0 0x7f648414c1d0 unknown :-1 s=CLOSED pgs=3430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:14:21.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 -- 172.21.15.7:0/1929970629 >> 172.21.15.7:0/1929970629 conn(0x7f64840c8900 msgr2=0x7f64840c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:14:21.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 -- 172.21.15.7:0/1929970629 shutdown_connections 2023-03-03T19:14:21.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:14:21.419+0000 7f648c665700 1 -- 172.21.15.7:0/1929970629 wait complete. 2023-03-03T19:14:21.422 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:14:21.432 INFO:tasks.mds_thrash.fs.[a]:waiting for 61 secs before thrashing 2023-03-03T19:14:46.094 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:14:46.097 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:15:03.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:15:03.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.288+0000 7f4ee0468700 1 Processor -- start 2023-03-03T19:15:03.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.288+0000 7f4ee0468700 1 -- start start 2023-03-03T19:15:03.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.289+0000 7f4ee0468700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4ed814fda0 0x7f4ed8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:03.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.289+0000 7f4ee0468700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80abe20 0x7f4ed81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:03.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.289+0000 7f4ee0468700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80c2f30 con 0x7f4ed814f9d0 2023-03-03T19:15:03.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.289+0000 7f4ee0468700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed80c31d0 con 0x7f4ed814fda0 2023-03-03T19:15:03.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.289+0000 7f4ee0468700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed81524a0 con 0x7f4ed80abe20 2023-03-03T19:15:03.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edea05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80abe20 0x7f4ed81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:03.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edea05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80abe20 0x7f4ed81529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:48696/0 (socket says 172.21.15.7:48696) 2023-03-03T19:15:03.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edea05700 1 -- 172.21.15.7:0/469036521 learned_addr learned my addr 172.21.15.7:0/469036521 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:15:03.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edda03700 1 --2- 172.21.15.7:0/469036521 >> v2:172.21.15.143:3300/0 conn(0x7f4ed814fda0 0x7f4ed8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:03.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edea05700 1 -- 172.21.15.7:0/469036521 >> v2:172.21.15.143:3300/0 conn(0x7f4ed814fda0 msgr2=0x7f4ed8156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:15:03.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edea05700 1 --2- 172.21.15.7:0/469036521 >> v2:172.21.15.143:3300/0 conn(0x7f4ed814fda0 0x7f4ed8156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:03.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edea05700 1 -- 172.21.15.7:0/469036521 >> v1:172.21.15.143:6789/0 conn(0x7f4ed814f9d0 legacy=0x7f4ed8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:03.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edea05700 1 -- 172.21.15.7:0/469036521 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80c34b0 con 0x7f4ed80abe20 2023-03-03T19:15:03.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edd202700 1 -- 172.21.15.7:0/469036521 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540562310 0 0) 0x7f4ed80c2f30 con 0x7f4ed814f9d0 2023-03-03T19:15:03.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edda03700 1 --2- 172.21.15.7:0/469036521 >> v2:172.21.15.143:3300/0 conn(0x7f4ed814fda0 0x7f4ed8156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:15:03.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.290+0000 7f4edea05700 1 --2- 172.21.15.7:0/469036521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80abe20 0x7f4ed81529c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009e50 tx=0x7f4ed000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df456ea091b1d30d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:15:03.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4edd202700 1 -- 172.21.15.7:0/469036521 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed000e070 con 0x7f4ed80abe20 2023-03-03T19:15:03.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4edd202700 1 -- 172.21.15.7:0/469036521 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed000b910 con 0x7f4ed80abe20 2023-03-03T19:15:03.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4edd202700 1 -- 172.21.15.7:0/469036521 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed000bcc0 con 0x7f4ed80abe20 2023-03-03T19:15:03.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4ee0468700 1 -- 172.21.15.7:0/469036521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80abe20 msgr2=0x7f4ed81529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:03.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4ee0468700 1 --2- 172.21.15.7:0/469036521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80abe20 0x7f4ed81529c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009e50 tx=0x7f4ed000b330 comp rx=0 tx=0).stop 2023-03-03T19:15:03.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4ee0468700 1 -- 172.21.15.7:0/469036521 shutdown_connections 2023-03-03T19:15:03.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4ee0468700 1 --2- 172.21.15.7:0/469036521 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed80abe20 0x7f4ed81529c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:03.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4ee0468700 1 --2- 172.21.15.7:0/469036521 >> v2:172.21.15.143:3300/0 conn(0x7f4ed814fda0 0x7f4ed8156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:03.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4ee0468700 1 -- 172.21.15.7:0/469036521 >> 172.21.15.7:0/469036521 conn(0x7f4ed80c8900 msgr2=0x7f4ed80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:15:03.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4ee0468700 1 -- 172.21.15.7:0/469036521 shutdown_connections 2023-03-03T19:15:03.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.291+0000 7f4ee0468700 1 -- 172.21.15.7:0/469036521 wait complete. 2023-03-03T19:15:03.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.293+0000 7f4ee0468700 1 Processor -- start 2023-03-03T19:15:03.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.293+0000 7f4ee0468700 1 -- start start 2023-03-03T19:15:03.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.293+0000 7f4ee0468700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed80abe20 0x7f4ed81e9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:03.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.293+0000 7f4ee0468700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed814f9d0 0x7f4ed81ed7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:03.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.293+0000 7f4ee0468700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed814fda0 0x7f4ed81537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:03.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.293+0000 7f4ee0468700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed814aa60 con 0x7f4ed814f9d0 2023-03-03T19:15:03.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.293+0000 7f4ee0468700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed81ecff0 con 0x7f4ed80abe20 2023-03-03T19:15:03.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.293+0000 7f4ee0468700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed814a640 con 0x7f4ed814fda0 2023-03-03T19:15:03.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.293+0000 7f4edda03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed814f9d0 0x7f4ed81ed7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:03.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.293+0000 7f4edda03700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed814f9d0 0x7f4ed81ed7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40628/0 (socket says 172.21.15.7:40628) 2023-03-03T19:15:03.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4edda03700 1 -- 172.21.15.7:0/4279866110 learned_addr learned my addr 172.21.15.7:0/4279866110 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:15:03.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4edea05700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed814fda0 0x7f4ed81537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:03.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4ede204700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed80abe20 0x7f4ed81e9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:03.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4edda03700 1 -- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed814fda0 msgr2=0x7f4ed81537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:03.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4edda03700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed814fda0 0x7f4ed81537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4edda03700 1 -- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed80abe20 msgr2=0x7f4ed81e9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4edda03700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed80abe20 0x7f4ed81e9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4edda03700 1 -- 172.21.15.7:0/4279866110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ecc0094a0 con 0x7f4ed814f9d0 2023-03-03T19:15:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4edda03700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed814f9d0 0x7f4ed81ed7b0 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00ebe0 tx=0x7f4ecc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:15:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4ecaffd700 1 -- 172.21.15.7:0/4279866110 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc0097a0 con 0x7f4ed814f9d0 2023-03-03T19:15:03.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4ecaffd700 1 -- 172.21.15.7:0/4279866110 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ecc00c850 con 0x7f4ed814f9d0 2023-03-03T19:15:03.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4ee0468700 1 -- 172.21.15.7:0/4279866110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed0009b30 con 0x7f4ed814f9d0 2023-03-03T19:15:03.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4ecaffd700 1 -- 172.21.15.7:0/4279866110 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc00cbe0 con 0x7f4ed814f9d0 2023-03-03T19:15:03.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.294+0000 7f4ee0468700 1 -- 172.21.15.7:0/4279866110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed80ac350 con 0x7f4ed814f9d0 2023-03-03T19:15:03.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.295+0000 7f4ee0468700 1 -- 172.21.15.7:0/4279866110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed80ab630 con 0x7f4ed814f9d0 2023-03-03T19:15:03.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.296+0000 7f4ecaffd700 1 -- 172.21.15.7:0/4279866110 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4ecc019030 con 0x7f4ed814f9d0 2023-03-03T19:15:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.296+0000 7f4ecaffd700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ebc0682f0 0x7f4ebc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.296+0000 7f4ecaffd700 1 -- 172.21.15.7:0/4279866110 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4ecc085cc0 con 0x7f4ed814f9d0 2023-03-03T19:15:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.296+0000 7f4ede204700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ebc0682f0 0x7f4ebc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.297+0000 7f4ede204700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ebc0682f0 0x7f4ebc06a7a0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4007fd0 tx=0x7f4ed4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:15:03.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.301+0000 7f4ecaffd700 1 -- 172.21.15.7:0/4279866110 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4ecc052060 con 0x7f4ed814f9d0 2023-03-03T19:15:03.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.539+0000 7f4ee0468700 1 -- 172.21.15.7:0/4279866110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ed8071fa0 con 0x7f4ed814f9d0 2023-03-03T19:15:03.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.539+0000 7f4ecaffd700 1 -- 172.21.15.7:0/4279866110 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4ecc055680 con 0x7f4ed814f9d0 2023-03-03T19:15:03.540 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:15:03.540 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:15:03.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.541+0000 7f4ee0468700 1 -- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ebc0682f0 msgr2=0x7f4ebc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:03.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.541+0000 7f4ee0468700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ebc0682f0 0x7f4ebc06a7a0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4007fd0 tx=0x7f4ed4009040 comp rx=0 tx=0).stop 2023-03-03T19:15:03.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.541+0000 7f4ee0468700 1 -- 172.21.15.7:0/4279866110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed814f9d0 msgr2=0x7f4ed81ed7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:03.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.541+0000 7f4ee0468700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed814f9d0 0x7f4ed81ed7b0 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc00ebe0 tx=0x7f4ecc00eef0 comp rx=0 tx=0).stop 2023-03-03T19:15:03.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.541+0000 7f4ee0468700 1 -- 172.21.15.7:0/4279866110 shutdown_connections 2023-03-03T19:15:03.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.541+0000 7f4ee0468700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ed814fda0 0x7f4ed81537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:03.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.541+0000 7f4ee0468700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ed80abe20 0x7f4ed81e9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:03.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.541+0000 7f4ee0468700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ebc0682f0 0x7f4ebc06a7a0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:03.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.541+0000 7f4ee0468700 1 --2- 172.21.15.7:0/4279866110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ed814f9d0 0x7f4ed81ed7b0 unknown :-1 s=CLOSED pgs=3431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:03.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.541+0000 7f4ee0468700 1 -- 172.21.15.7:0/4279866110 >> 172.21.15.7:0/4279866110 conn(0x7f4ed80c8900 msgr2=0x7f4ed80c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:15:03.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.542+0000 7f4ee0468700 1 -- 172.21.15.7:0/4279866110 shutdown_connections 2023-03-03T19:15:03.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:03.542+0000 7f4ee0468700 1 -- 172.21.15.7:0/4279866110 wait complete. 2023-03-03T19:15:03.543 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:15:03.554 INFO:tasks.mds_thrash.fs.[b]:waiting for 63 secs before thrashing 2023-03-03T19:15:16.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:15:16.185 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:15:22.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:15:22.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.572+0000 7f0a829e2700 1 Processor -- start 2023-03-03T19:15:22.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.572+0000 7f0a829e2700 1 -- start start 2023-03-03T19:15:22.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.573+0000 7f0a829e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a7c072250 0x7f0a7c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:22.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.573+0000 7f0a829e2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0a7c0c1ce0 0x7f0a7c14e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:22.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.573+0000 7f0a829e2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7c1510d0 con 0x7f0a7c071e80 2023-03-03T19:15:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.573+0000 7f0a829e2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7c0c3740 con 0x7f0a7c0c1ce0 2023-03-03T19:15:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.573+0000 7f0a829e2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c070670 con 0x7f0a7c072250 2023-03-03T19:15:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.573+0000 7f0a80f7f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0a7c0c1ce0 0x7f0a7c14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.573+0000 7f0a80f7f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0a7c0c1ce0 0x7f0a7c14e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40650/0 (socket says 172.21.15.7:40650) 2023-03-03T19:15:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.573+0000 7f0a80f7f700 1 -- 172.21.15.7:0/2469502365 learned_addr learned my addr 172.21.15.7:0/2469502365 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:15:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.574+0000 7f0a80f7f700 1 -- 172.21.15.7:0/2469502365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a7c072250 msgr2=0x7f0a7c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:15:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.574+0000 7f0a80f7f700 1 --2- 172.21.15.7:0/2469502365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a7c072250 0x7f0a7c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.574+0000 7f0a80f7f700 1 -- 172.21.15.7:0/2469502365 >> v1:172.21.15.143:6789/0 conn(0x7f0a7c071e80 legacy=0x7f0a7c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:15:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.574+0000 7f0a80f7f700 1 -- 172.21.15.7:0/2469502365 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7c0c39e0 con 0x7f0a7c0c1ce0 2023-03-03T19:15:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.574+0000 7f0a80f7f700 1 --2- 172.21.15.7:0/2469502365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c0c1ce0 0x7f0a7c14e730 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 crypto rx=0x7f0a74009e50 tx=0x7f0a7400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c58649ead0c3ac1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:15:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.574+0000 7f0a7affd700 1 -- 172.21.15.7:0/2469502365 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7400e070 con 0x7f0a7c0c1ce0 2023-03-03T19:15:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.574+0000 7f0a7affd700 1 -- 172.21.15.7:0/2469502365 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a74003a20 con 0x7f0a7c0c1ce0 2023-03-03T19:15:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.574+0000 7f0a7affd700 1 -- 172.21.15.7:0/2469502365 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74003dd0 con 0x7f0a7c0c1ce0 2023-03-03T19:15:22.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.575+0000 7f0a829e2700 1 -- 172.21.15.7:0/2469502365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c0c1ce0 msgr2=0x7f0a7c14e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:22.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.575+0000 7f0a829e2700 1 --2- 172.21.15.7:0/2469502365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c0c1ce0 0x7f0a7c14e730 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 crypto rx=0x7f0a74009e50 tx=0x7f0a7400b330 comp rx=0 tx=0).stop 2023-03-03T19:15:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.575+0000 7f0a829e2700 1 -- 172.21.15.7:0/2469502365 shutdown_connections 2023-03-03T19:15:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.575+0000 7f0a829e2700 1 --2- 172.21.15.7:0/2469502365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c0c1ce0 0x7f0a7c14e730 unknown :-1 s=CLOSED pgs=3432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.575+0000 7f0a829e2700 1 --2- 172.21.15.7:0/2469502365 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a7c072250 0x7f0a7c14e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.575+0000 7f0a829e2700 1 -- 172.21.15.7:0/2469502365 >> 172.21.15.7:0/2469502365 conn(0x7f0a7c0a94d0 msgr2=0x7f0a7c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:15:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.575+0000 7f0a829e2700 1 -- 172.21.15.7:0/2469502365 shutdown_connections 2023-03-03T19:15:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.575+0000 7f0a829e2700 1 -- 172.21.15.7:0/2469502365 wait complete. 2023-03-03T19:15:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.576+0000 7f0a829e2700 1 Processor -- start 2023-03-03T19:15:22.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.576+0000 7f0a829e2700 1 -- start start 2023-03-03T19:15:22.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a829e2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c071e80 0x7f0a7c0c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:22.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a829e2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a7c072250 0x7f0a7c0c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:22.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a829e2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a7c0c1ce0 0x7f0a7c0c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a829e2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c0c4950 con 0x7f0a7c071e80 2023-03-03T19:15:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a829e2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c14d600 con 0x7f0a7c072250 2023-03-03T19:15:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a829e2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c14c0a0 con 0x7f0a7c0c1ce0 2023-03-03T19:15:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c071e80 0x7f0a7c0c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c071e80 0x7f0a7c0c3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40652/0 (socket says 172.21.15.7:40652) 2023-03-03T19:15:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7bfff700 1 -- 172.21.15.7:0/1885456921 learned_addr learned my addr 172.21.15.7:0/1885456921 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:15:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a80f7f700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a7c0c1ce0 0x7f0a7c0c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7b7fe700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a7c072250 0x7f0a7c0c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7bfff700 1 -- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a7c0c1ce0 msgr2=0x7f0a7c0c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7bfff700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a7c0c1ce0 0x7f0a7c0c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7bfff700 1 -- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a7c072250 msgr2=0x7f0a7c0c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7bfff700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a7c072250 0x7f0a7c0c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7bfff700 1 -- 172.21.15.7:0/1885456921 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a74009b30 con 0x7f0a7c071e80 2023-03-03T19:15:22.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7b7fe700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a7c072250 0x7f0a7c0c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:15:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a80f7f700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a7c0c1ce0 0x7f0a7c0c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:15:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.577+0000 7f0a7bfff700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c071e80 0x7f0a7c0c3e90 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00c4a0 tx=0x7f0a6c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:15:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.578+0000 7f0a78ff9700 1 -- 172.21.15.7:0/1885456921 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0054e0 con 0x7f0a7c071e80 2023-03-03T19:15:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.578+0000 7f0a78ff9700 1 -- 172.21.15.7:0/1885456921 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c003680 con 0x7f0a7c071e80 2023-03-03T19:15:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.578+0000 7f0a829e2700 1 -- 172.21.15.7:0/1885456921 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7c151f30 con 0x7f0a7c071e80 2023-03-03T19:15:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.578+0000 7f0a78ff9700 1 -- 172.21.15.7:0/1885456921 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c003a30 con 0x7f0a7c071e80 2023-03-03T19:15:22.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.578+0000 7f0a829e2700 1 -- 172.21.15.7:0/1885456921 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c151300 con 0x7f0a7c071e80 2023-03-03T19:15:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.579+0000 7f0a829e2700 1 -- 172.21.15.7:0/1885456921 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c06c9a0 con 0x7f0a7c071e80 2023-03-03T19:15:22.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.579+0000 7f0a78ff9700 1 -- 172.21.15.7:0/1885456921 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0a6c019030 con 0x7f0a7c071e80 2023-03-03T19:15:22.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.579+0000 7f0a78ff9700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a5c0682f0 0x7f0a5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:22.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.579+0000 7f0a78ff9700 1 -- 172.21.15.7:0/1885456921 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0a6c085550 con 0x7f0a7c071e80 2023-03-03T19:15:22.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.580+0000 7f0a7b7fe700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a5c0682f0 0x7f0a5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:22.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.580+0000 7f0a7b7fe700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a5c0682f0 0x7f0a5c06a7a0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0c72b0 tx=0x7f0a64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:15:22.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.584+0000 7f0a78ff9700 1 -- 172.21.15.7:0/1885456921 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0a6c0518f0 con 0x7f0a7c071e80 2023-03-03T19:15:22.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.819+0000 7f0a829e2700 1 -- 172.21.15.7:0/1885456921 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a7c074bb0 con 0x7f0a7c071e80 2023-03-03T19:15:22.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.820+0000 7f0a78ff9700 1 -- 172.21.15.7:0/1885456921 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0a6c054f10 con 0x7f0a7c071e80 2023-03-03T19:15:22.821 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:15:22.822 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:15:22.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 -- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a5c0682f0 msgr2=0x7f0a5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:22.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a5c0682f0 0x7f0a5c06a7a0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0c72b0 tx=0x7f0a64008040 comp rx=0 tx=0).stop 2023-03-03T19:15:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 -- 172.21.15.7:0/1885456921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c071e80 msgr2=0x7f0a7c0c3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c071e80 0x7f0a7c0c3e90 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00c4a0 tx=0x7f0a6c00c860 comp rx=0 tx=0).stop 2023-03-03T19:15:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 -- 172.21.15.7:0/1885456921 shutdown_connections 2023-03-03T19:15:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0a7c0c1ce0 0x7f0a7c0c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0a5c0682f0 0x7f0a5c06a7a0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a7c071e80 0x7f0a7c0c3e90 unknown :-1 s=CLOSED pgs=3433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:22.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 --2- 172.21.15.7:0/1885456921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a7c072250 0x7f0a7c0c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 -- 172.21.15.7:0/1885456921 >> 172.21.15.7:0/1885456921 conn(0x7f0a7c0a94d0 msgr2=0x7f0a7c06d680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:15:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 -- 172.21.15.7:0/1885456921 shutdown_connections 2023-03-03T19:15:22.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:22.822+0000 7f0a829e2700 1 -- 172.21.15.7:0/1885456921 wait complete. 2023-03-03T19:15:22.826 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:15:22.835 INFO:tasks.mds_thrash.fs.[a]:waiting for 24 secs before thrashing 2023-03-03T19:15:46.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:15:46.269 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:15:46.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:15:46.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.975+0000 7fabbbd16700 1 Processor -- start 2023-03-03T19:15:46.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.975+0000 7fabbbd16700 1 -- start start 2023-03-03T19:15:46.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.976+0000 7fabbbd16700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabb414f9d0 0x7fabb4156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:46.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.976+0000 7fabbbd16700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb40c2f30 con 0x7fabb40abe20 2023-03-03T19:15:46.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.976+0000 7fabbbd16700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb40c31d0 con 0x7fabb414fda0 2023-03-03T19:15:46.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.976+0000 7fabbbd16700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb41524a0 con 0x7fabb414f9d0 2023-03-03T19:15:46.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.976+0000 7fabb9ab2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabb414f9d0 0x7fabb4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:46.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.976+0000 7fabba2b3700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fabb40abe20 0x7fabb41529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34664/0 (socket says 172.21.15.7:34664) 2023-03-03T19:15:46.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.976+0000 7fabba2b3700 1 -- 172.21.15.7:0/4172281355 learned_addr learned my addr 172.21.15.7:0/4172281355 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:15:46.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.977+0000 7fabb9ab2700 1 -- 172.21.15.7:0/4172281355 >> v1:172.21.15.143:6789/0 conn(0x7fabb414fda0 legacy=0x7fabb4071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:15:46.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.977+0000 7fabb9ab2700 1 -- 172.21.15.7:0/4172281355 >> v1:172.21.15.7:6789/0 conn(0x7fabb40abe20 legacy=0x7fabb41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:46.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.977+0000 7fabb9ab2700 1 -- 172.21.15.7:0/4172281355 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb40c34b0 con 0x7fabb414f9d0 2023-03-03T19:15:46.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.977+0000 7fabb8ab0700 1 -- 172.21.15.7:0/4172281355 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278475179 0 0) 0x7fabb40c2f30 con 0x7fabb40abe20 2023-03-03T19:15:46.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.977+0000 7fabb9ab2700 1 --2- 172.21.15.7:0/4172281355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 0x7fabb4156430 secure :-1 s=READY pgs=3435 cs=0 l=1 rev1=1 crypto rx=0x7faba4009840 tx=0x7faba4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fea602d292f5f7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:15:46.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.977+0000 7fabb8ab0700 1 -- 172.21.15.7:0/4172281355 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4004230 con 0x7fabb414f9d0 2023-03-03T19:15:46.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.977+0000 7fabb8ab0700 1 -- 172.21.15.7:0/4172281355 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba4004390 con 0x7fabb414f9d0 2023-03-03T19:15:46.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.977+0000 7fabb8ab0700 1 -- 172.21.15.7:0/4172281355 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba40036a0 con 0x7fabb414f9d0 2023-03-03T19:15:46.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.977+0000 7fabbbd16700 1 -- 172.21.15.7:0/4172281355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 msgr2=0x7fabb4156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:46.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.977+0000 7fabbbd16700 1 --2- 172.21.15.7:0/4172281355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 0x7fabb4156430 secure :-1 s=READY pgs=3435 cs=0 l=1 rev1=1 crypto rx=0x7faba4009840 tx=0x7faba4009b50 comp rx=0 tx=0).stop 2023-03-03T19:15:46.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.978+0000 7fabbbd16700 1 -- 172.21.15.7:0/4172281355 shutdown_connections 2023-03-03T19:15:46.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.978+0000 7fabbbd16700 1 --2- 172.21.15.7:0/4172281355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 0x7fabb4156430 unknown :-1 s=CLOSED pgs=3435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:46.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.978+0000 7fabbbd16700 1 -- 172.21.15.7:0/4172281355 >> 172.21.15.7:0/4172281355 conn(0x7fabb40c8900 msgr2=0x7fabb40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:15:46.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.978+0000 7fabbbd16700 1 -- 172.21.15.7:0/4172281355 shutdown_connections 2023-03-03T19:15:46.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.978+0000 7fabbbd16700 1 -- 172.21.15.7:0/4172281355 wait complete. 2023-03-03T19:15:46.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.979+0000 7fabbbd16700 1 Processor -- start 2023-03-03T19:15:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.979+0000 7fabbbd16700 1 -- start start 2023-03-03T19:15:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.979+0000 7fabbbd16700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabb40abe20 0x7fabb414abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.979+0000 7fabbbd16700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 0x7fabb406ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.979+0000 7fabbbd16700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabb414fda0 0x7fabb406d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.979+0000 7fabbbd16700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb40c24d0 con 0x7fabb414f9d0 2023-03-03T19:15:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.979+0000 7fabbbd16700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb41e8240 con 0x7fabb40abe20 2023-03-03T19:15:46.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.979+0000 7fabbbd16700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb4155bd0 con 0x7fabb414fda0 2023-03-03T19:15:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb92b1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 0x7fabb406ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb92b1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 0x7fabb406ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40690/0 (socket says 172.21.15.7:40690) 2023-03-03T19:15:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb92b1700 1 -- 172.21.15.7:0/168756013 learned_addr learned my addr 172.21.15.7:0/168756013 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:15:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb9ab2700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabb40abe20 0x7fabb414abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabba2b3700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabb414fda0 0x7fabb406d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb92b1700 1 -- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabb414fda0 msgr2=0x7fabb406d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:46.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb92b1700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabb414fda0 0x7fabb406d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:46.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb92b1700 1 -- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabb40abe20 msgr2=0x7fabb414abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:46.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb92b1700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabb40abe20 0x7fabb414abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:46.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb92b1700 1 -- 172.21.15.7:0/168756013 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40094a0 con 0x7fabb414f9d0 2023-03-03T19:15:46.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb9ab2700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabb40abe20 0x7fabb414abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:15:46.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabb92b1700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 0x7fabb406ce60 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 crypto rx=0x7fabb000c4d0 tx=0x7fabb000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:15:46.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabaa7fc700 1 -- 172.21.15.7:0/168756013 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb00054e0 con 0x7fabb414f9d0 2023-03-03T19:15:46.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabaa7fc700 1 -- 172.21.15.7:0/168756013 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb0003680 con 0x7fabb414f9d0 2023-03-03T19:15:46.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabaa7fc700 1 -- 172.21.15.7:0/168756013 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb0003a10 con 0x7fabb414f9d0 2023-03-03T19:15:46.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabbbd16700 1 -- 172.21.15.7:0/168756013 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb40ac350 con 0x7fabb414f9d0 2023-03-03T19:15:46.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.980+0000 7fabbbd16700 1 -- 172.21.15.7:0/168756013 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb40ab630 con 0x7fabb414f9d0 2023-03-03T19:15:46.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.981+0000 7fabbbd16700 1 -- 172.21.15.7:0/168756013 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb4071fa0 con 0x7fabb414f9d0 2023-03-03T19:15:46.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.982+0000 7fabaa7fc700 1 -- 172.21.15.7:0/168756013 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fabb0010070 con 0x7fabb414f9d0 2023-03-03T19:15:46.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.982+0000 7fabaa7fc700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab980682f0 0x7fab9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:15:46.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.982+0000 7fabaa7fc700 1 -- 172.21.15.7:0/168756013 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fabb00852b0 con 0x7fabb414f9d0 2023-03-03T19:15:46.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.983+0000 7fabb9ab2700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab980682f0 0x7fab9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:15:46.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.983+0000 7fabb9ab2700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab980682f0 0x7fab9806a7a0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7faba4000c00 tx=0x7faba401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:15:46.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:46.987+0000 7fabaa7fc700 1 -- 172.21.15.7:0/168756013 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fabb0051560 con 0x7fabb414f9d0 2023-03-03T19:15:47.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.227+0000 7fabbbd16700 1 -- 172.21.15.7:0/168756013 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabb40c1cb0 con 0x7fabb414f9d0 2023-03-03T19:15:47.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.228+0000 7fabaa7fc700 1 -- 172.21.15.7:0/168756013 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fabb40c1cb0 con 0x7fabb414f9d0 2023-03-03T19:15:47.229 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:15:47.229 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:15:47.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 -- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab980682f0 msgr2=0x7fab9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:47.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab980682f0 0x7fab9806a7a0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7faba4000c00 tx=0x7faba401c040 comp rx=0 tx=0).stop 2023-03-03T19:15:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 -- 172.21.15.7:0/168756013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 msgr2=0x7fabb406ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:15:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 0x7fabb406ce60 secure :-1 s=READY pgs=3436 cs=0 l=1 rev1=1 crypto rx=0x7fabb000c4d0 tx=0x7fabb000c890 comp rx=0 tx=0).stop 2023-03-03T19:15:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 -- 172.21.15.7:0/168756013 shutdown_connections 2023-03-03T19:15:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fabb414fda0 0x7fabb406d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fabb40abe20 0x7fabb414abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fab980682f0 0x7fab9806a7a0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 --2- 172.21.15.7:0/168756013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabb414f9d0 0x7fabb406ce60 unknown :-1 s=CLOSED pgs=3436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:15:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 -- 172.21.15.7:0/168756013 >> 172.21.15.7:0/168756013 conn(0x7fabb40c8900 msgr2=0x7fabb4151160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:15:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 -- 172.21.15.7:0/168756013 shutdown_connections 2023-03-03T19:15:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:15:47.230+0000 7fabbbd16700 1 -- 172.21.15.7:0/168756013 wait complete. 2023-03-03T19:15:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:15:47.242 INFO:tasks.mds_thrash.fs.[a]:waiting for 32 secs before thrashing 2023-03-03T19:16:06.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:16:06.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.694+0000 7f183e6fc700 1 Processor -- start 2023-03-03T19:16:06.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.694+0000 7f183e6fc700 1 -- start start 2023-03-03T19:16:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.695+0000 7f183e6fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1838071c30 0x7f18380c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.695+0000 7f183e6fc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18380713c0 con 0x7f1838155950 2023-03-03T19:16:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.695+0000 7f183e6fc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1838152c70 con 0x7f18380aac00 2023-03-03T19:16:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.695+0000 7f183e6fc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18380c4930 con 0x7f1838071c30 2023-03-03T19:16:06.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.695+0000 7f183cc99700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1838155950 0x7f183815cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34684/0 (socket says 172.21.15.7:34684) 2023-03-03T19:16:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.695+0000 7f183cc99700 1 -- 172.21.15.7:0/3393781582 learned_addr learned my addr 172.21.15.7:0/3393781582 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:16:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.695+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068640513 0 0) 0x7f18380713c0 con 0x7f1838155950 2023-03-03T19:16:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18240035b0 con 0x7f1838155950 2023-03-03T19:16:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2953922006 0 0) 0x7f18240035b0 con 0x7f1838155950 2023-03-03T19:16:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 >> v2:172.21.15.143:3300/0 conn(0x7f1838071c30 msgr2=0x7f18380c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:16:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 --2- 172.21.15.7:0/3393781582 >> v2:172.21.15.143:3300/0 conn(0x7f1838071c30 0x7f18380c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 >> v1:172.21.15.143:6789/0 conn(0x7f18380aac00 legacy=0x7f18380beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:16:06.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1838152360 con 0x7f1838155950 2023-03-03T19:16:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f18300019f0 con 0x7f1838155950 2023-03-03T19:16:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 >> v1:172.21.15.7:6789/0 conn(0x7f1838155950 legacy=0x7f183815cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1824003a70 0x7f1824003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1824004850 0x7f1824004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1824009a00 0x7f182400bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18380c4930 con 0x7f1824003a70 2023-03-03T19:16:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182400c3a0 con 0x7f1824009a00 2023-03-03T19:16:06.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182400c560 con 0x7f1824004850 2023-03-03T19:16:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1837fff700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1824003a70 0x7f1824003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f183cc99700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1824009a00 0x7f182400bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f18377fe700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1824004850 0x7f1824004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1837fff700 1 -- 172.21.15.7:0/3393781582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1824004850 msgr2=0x7f1824004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:16:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1837fff700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1824004850 0x7f1824004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1837fff700 1 -- 172.21.15.7:0/3393781582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1824009a00 msgr2=0x7f182400bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:16:06.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1837fff700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1824009a00 0x7f182400bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f1837fff700 1 -- 172.21.15.7:0/3393781582 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240035b0 con 0x7f1824003a70 2023-03-03T19:16:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f183cc99700 1 -- 172.21.15.7:0/3393781582 reap_dead start 2023-03-03T19:16:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.696+0000 7f18377fe700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1824004850 0x7f1824004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:16:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f1837fff700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1824003a70 0x7f1824003e60 secure :-1 s=READY pgs=3438 cs=0 l=1 rev1=1 crypto rx=0x7f18200016c0 tx=0x7f1824005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfcef5e459a9d37d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18200052d0 con 0x7f1824003a70 2023-03-03T19:16:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1820005430 con 0x7f1824003a70 2023-03-03T19:16:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f1836ffd700 1 -- 172.21.15.7:0/3393781582 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18200038c0 con 0x7f1824003a70 2023-03-03T19:16:06.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f183e6fc700 1 -- 172.21.15.7:0/3393781582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1824003a70 msgr2=0x7f1824003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f183e6fc700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1824003a70 0x7f1824003e60 secure :-1 s=READY pgs=3438 cs=0 l=1 rev1=1 crypto rx=0x7f18200016c0 tx=0x7f1824005880 comp rx=0 tx=0).stop 2023-03-03T19:16:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f183e6fc700 1 -- 172.21.15.7:0/3393781582 shutdown_connections 2023-03-03T19:16:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f183e6fc700 1 --2- 172.21.15.7:0/3393781582 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1824003a70 0x7f1824003e60 unknown :-1 s=CLOSED pgs=3438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f183e6fc700 1 -- 172.21.15.7:0/3393781582 >> 172.21.15.7:0/3393781582 conn(0x7f18380736f0 msgr2=0x7f1838073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:16:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f183e6fc700 1 -- 172.21.15.7:0/3393781582 shutdown_connections 2023-03-03T19:16:06.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.697+0000 7f183e6fc700 1 -- 172.21.15.7:0/3393781582 wait complete. 2023-03-03T19:16:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.698+0000 7f183e6fc700 1 Processor -- start 2023-03-03T19:16:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.699+0000 7f183e6fc700 1 -- start start 2023-03-03T19:16:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.699+0000 7f183e6fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1824003a70 0x7f183806bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.699+0000 7f183e6fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f183814b250 0x7f183814e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.699+0000 7f183e6fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f183814b6c0 0x7f183814eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.699+0000 7f183e6fc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18381ea470 con 0x7f183814b250 2023-03-03T19:16:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.699+0000 7f183e6fc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183814f0d0 con 0x7f1824003a70 2023-03-03T19:16:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.699+0000 7f183e6fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18381e9530 con 0x7f183814b6c0 2023-03-03T19:16:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.699+0000 7f18377fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f183814b250 0x7f183814e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.699+0000 7f18377fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f183814b250 0x7f183814e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40714/0 (socket says 172.21.15.7:40714) 2023-03-03T19:16:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.699+0000 7f18377fe700 1 -- 172.21.15.7:0/3772928558 learned_addr learned my addr 172.21.15.7:0/3772928558 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:16:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f1837fff700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1824003a70 0x7f183806bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f183cc99700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f183814b6c0 0x7f183814eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f18377fe700 1 -- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f183814b6c0 msgr2=0x7f183814eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f18377fe700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f183814b6c0 0x7f183814eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f18377fe700 1 -- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1824003a70 msgr2=0x7f183806bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f18377fe700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1824003a70 0x7f183806bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f18377fe700 1 -- 172.21.15.7:0/3772928558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1838152620 con 0x7f183814b250 2023-03-03T19:16:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f18377fe700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f183814b250 0x7f183814e650 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7f182800d4a0 tx=0x7f182800d7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f1834ff9700 1 -- 172.21.15.7:0/3772928558 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182800de20 con 0x7f183814b250 2023-03-03T19:16:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f1834ff9700 1 -- 172.21.15.7:0/3772928558 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18280125f0 con 0x7f183814b250 2023-03-03T19:16:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f183e6fc700 1 -- 172.21.15.7:0/3772928558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18380c4b80 con 0x7f183814b250 2023-03-03T19:16:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f1834ff9700 1 -- 172.21.15.7:0/3772928558 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1828012980 con 0x7f183814b250 2023-03-03T19:16:06.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.700+0000 7f183e6fc700 1 -- 172.21.15.7:0/3772928558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1838152900 con 0x7f183814b250 2023-03-03T19:16:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.701+0000 7f183e6fc700 1 -- 172.21.15.7:0/3772928558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1838074bb0 con 0x7f183814b250 2023-03-03T19:16:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.702+0000 7f1834ff9700 1 -- 172.21.15.7:0/3772928558 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1828012ae0 con 0x7f183814b250 2023-03-03T19:16:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.702+0000 7f1834ff9700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18180682f0 0x7f181806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.702+0000 7f1834ff9700 1 -- 172.21.15.7:0/3772928558 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1828086190 con 0x7f183814b250 2023-03-03T19:16:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.702+0000 7f1837fff700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18180682f0 0x7f181806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.703+0000 7f1837fff700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18180682f0 0x7f181806a7a0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f1820015040 tx=0x7f182000a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:06.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.706+0000 7f1834ff9700 1 -- 172.21.15.7:0/3772928558 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1828052530 con 0x7f183814b250 2023-03-03T19:16:06.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.946+0000 7f183e6fc700 1 -- 172.21.15.7:0/3772928558 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1838071a10 con 0x7f183814b250 2023-03-03T19:16:06.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.946+0000 7f1834ff9700 1 -- 172.21.15.7:0/3772928558 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1828055b50 con 0x7f183814b250 2023-03-03T19:16:06.947 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:16:06.947 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:16:06.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 -- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18180682f0 msgr2=0x7f181806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:06.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18180682f0 0x7f181806a7a0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f1820015040 tx=0x7f182000a000 comp rx=0 tx=0).stop 2023-03-03T19:16:06.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 -- 172.21.15.7:0/3772928558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f183814b250 msgr2=0x7f183814e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:06.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f183814b250 0x7f183814e650 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7f182800d4a0 tx=0x7f182800d7b0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 -- 172.21.15.7:0/3772928558 shutdown_connections 2023-03-03T19:16:06.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f183814b6c0 0x7f183814eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1824003a70 0x7f183806bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f18180682f0 0x7f181806a7a0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 --2- 172.21.15.7:0/3772928558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f183814b250 0x7f183814e650 unknown :-1 s=CLOSED pgs=3439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 -- 172.21.15.7:0/3772928558 >> 172.21.15.7:0/3772928558 conn(0x7f18380736f0 msgr2=0x7f1838151bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:16:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 -- 172.21.15.7:0/3772928558 shutdown_connections 2023-03-03T19:16:06.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:06.948+0000 7f183e6fc700 1 -- 172.21.15.7:0/3772928558 wait complete. 2023-03-03T19:16:06.951 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:16:06.961 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-03-03T19:16:16.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:16:16.353 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:16:19.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:16:19.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.381+0000 7f4c84eb0700 1 Processor -- start 2023-03-03T19:16:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.381+0000 7f4c84eb0700 1 -- start start 2023-03-03T19:16:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.382+0000 7f4c84eb0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c801550c0 0x7f4c8015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.382+0000 7f4c84eb0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c800c7a60 con 0x7f4c8006d870 2023-03-03T19:16:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.382+0000 7f4c84eb0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c800c3620 con 0x7f4c800c1da0 2023-03-03T19:16:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.382+0000 7f4c84eb0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c801554c0 con 0x7f4c801550c0 2023-03-03T19:16:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7ed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c801550c0 0x7f4c8015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7e59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4c8006d870 0x7f4c800c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34706/0 (socket says 172.21.15.7:34706) 2023-03-03T19:16:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7e59c700 1 -- 172.21.15.7:0/807031491 learned_addr learned my addr 172.21.15.7:0/807031491 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:16:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7ed9d700 1 -- 172.21.15.7:0/807031491 >> v1:172.21.15.143:6789/0 conn(0x7f4c800c1da0 legacy=0x7f4c800c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:16:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7d59a700 1 -- 172.21.15.7:0/807031491 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9438754 0 0) 0x7f4c800c7a60 con 0x7f4c8006d870 2023-03-03T19:16:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7ed9d700 1 -- 172.21.15.7:0/807031491 >> v1:172.21.15.7:6789/0 conn(0x7f4c8006d870 legacy=0x7f4c800c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7ed9d700 1 -- 172.21.15.7:0/807031491 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c801545e0 con 0x7f4c801550c0 2023-03-03T19:16:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7ed9d700 1 --2- 172.21.15.7:0/807031491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c801550c0 0x7f4c8015fe60 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009d90 tx=0x7f4c7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7943d394c428a0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7d59a700 1 -- 172.21.15.7:0/807031491 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7800e070 con 0x7f4c801550c0 2023-03-03T19:16:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7d59a700 1 -- 172.21.15.7:0/807031491 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7800bbd0 con 0x7f4c801550c0 2023-03-03T19:16:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.383+0000 7f4c7d59a700 1 -- 172.21.15.7:0/807031491 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78012430 con 0x7f4c801550c0 2023-03-03T19:16:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.384+0000 7f4c84eb0700 1 -- 172.21.15.7:0/807031491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c801550c0 msgr2=0x7f4c8015fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.384+0000 7f4c84eb0700 1 --2- 172.21.15.7:0/807031491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c801550c0 0x7f4c8015fe60 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009d90 tx=0x7f4c7800b330 comp rx=0 tx=0).stop 2023-03-03T19:16:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.384+0000 7f4c84eb0700 1 -- 172.21.15.7:0/807031491 shutdown_connections 2023-03-03T19:16:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.384+0000 7f4c84eb0700 1 --2- 172.21.15.7:0/807031491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c801550c0 0x7f4c8015fe60 unknown :-1 s=CLOSED pgs=3441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.384+0000 7f4c84eb0700 1 -- 172.21.15.7:0/807031491 >> 172.21.15.7:0/807031491 conn(0x7f4c8006c110 msgr2=0x7f4c8006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:16:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.384+0000 7f4c84eb0700 1 -- 172.21.15.7:0/807031491 shutdown_connections 2023-03-03T19:16:19.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.384+0000 7f4c84eb0700 1 -- 172.21.15.7:0/807031491 wait complete. 2023-03-03T19:16:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.385+0000 7f4c84eb0700 1 Processor -- start 2023-03-03T19:16:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.385+0000 7f4c84eb0700 1 -- start start 2023-03-03T19:16:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c84eb0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c8006d870 0x7f4c80150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c84eb0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c800c1da0 0x7f4c8013c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c84eb0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c801550c0 0x7f4c800ab690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c84eb0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c800ace70 con 0x7f4c800c1da0 2023-03-03T19:16:19.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c84eb0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c800aac40 con 0x7f4c801550c0 2023-03-03T19:16:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c84eb0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c800ab200 con 0x7f4c8006d870 2023-03-03T19:16:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c800c1da0 0x7f4c8013c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c800c1da0 0x7f4c8013c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40732/0 (socket says 172.21.15.7:40732) 2023-03-03T19:16:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7dd9b700 1 -- 172.21.15.7:0/954584896 learned_addr learned my addr 172.21.15.7:0/954584896 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:16:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7e59c700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c8006d870 0x7f4c80150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7ed9d700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c801550c0 0x7f4c800ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7dd9b700 1 -- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c8006d870 msgr2=0x7f4c80150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7dd9b700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c8006d870 0x7f4c80150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7dd9b700 1 -- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c801550c0 msgr2=0x7f4c800ab690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7dd9b700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c801550c0 0x7f4c800ab690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7dd9b700 1 -- 172.21.15.7:0/954584896 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c78009a70 con 0x7f4c800c1da0 2023-03-03T19:16:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.386+0000 7f4c7ed9d700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c801550c0 0x7f4c800ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:16:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.387+0000 7f4c7dd9b700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c800c1da0 0x7f4c8013c480 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400c510 tx=0x7f4c7400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.387+0000 7f4c6effd700 1 -- 172.21.15.7:0/954584896 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c740052d0 con 0x7f4c800c1da0 2023-03-03T19:16:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.387+0000 7f4c6effd700 1 -- 172.21.15.7:0/954584896 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c74005430 con 0x7f4c800c1da0 2023-03-03T19:16:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.387+0000 7f4c84eb0700 1 -- 172.21.15.7:0/954584896 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c80154ba0 con 0x7f4c800c1da0 2023-03-03T19:16:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.387+0000 7f4c6effd700 1 -- 172.21.15.7:0/954584896 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c740038c0 con 0x7f4c800c1da0 2023-03-03T19:16:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.387+0000 7f4c84eb0700 1 -- 172.21.15.7:0/954584896 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c801548c0 con 0x7f4c800c1da0 2023-03-03T19:16:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.388+0000 7f4c84eb0700 1 -- 172.21.15.7:0/954584896 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8006dc40 con 0x7f4c800c1da0 2023-03-03T19:16:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.388+0000 7f4c6effd700 1 -- 172.21.15.7:0/954584896 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4c74019030 con 0x7f4c800c1da0 2023-03-03T19:16:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.389+0000 7f4c6effd700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c600682f0 0x7f4c6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.389+0000 7f4c6effd700 1 -- 172.21.15.7:0/954584896 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4c74085490 con 0x7f4c800c1da0 2023-03-03T19:16:19.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.389+0000 7f4c7e59c700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c600682f0 0x7f4c6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:19.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.390+0000 7f4c7e59c700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c600682f0 0x7f4c6006a7a0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f4c70000f60 tx=0x7f4c70007d90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:19.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.393+0000 7f4c6effd700 1 -- 172.21.15.7:0/954584896 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4c74051830 con 0x7f4c800c1da0 2023-03-03T19:16:19.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.630+0000 7f4c84eb0700 1 -- 172.21.15.7:0/954584896 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c80074bb0 con 0x7f4c800c1da0 2023-03-03T19:16:19.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.630+0000 7f4c6effd700 1 -- 172.21.15.7:0/954584896 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4c74054e50 con 0x7f4c800c1da0 2023-03-03T19:16:19.632 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:16:19.632 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:16:19.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 -- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c600682f0 msgr2=0x7f4c6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:19.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c600682f0 0x7f4c6006a7a0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f4c70000f60 tx=0x7f4c70007d90 comp rx=0 tx=0).stop 2023-03-03T19:16:19.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 -- 172.21.15.7:0/954584896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c800c1da0 msgr2=0x7f4c8013c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:19.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c800c1da0 0x7f4c8013c480 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400c510 tx=0x7f4c7400c8d0 comp rx=0 tx=0).stop 2023-03-03T19:16:19.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 -- 172.21.15.7:0/954584896 shutdown_connections 2023-03-03T19:16:19.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c801550c0 0x7f4c800ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:19.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4c8006d870 0x7f4c80150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:19.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4c600682f0 0x7f4c6006a7a0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:19.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 --2- 172.21.15.7:0/954584896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c800c1da0 0x7f4c8013c480 unknown :-1 s=CLOSED pgs=3442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 -- 172.21.15.7:0/954584896 >> 172.21.15.7:0/954584896 conn(0x7f4c8006c110 msgr2=0x7f4c8006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:16:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 -- 172.21.15.7:0/954584896 shutdown_connections 2023-03-03T19:16:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:19.632+0000 7f4c84eb0700 1 -- 172.21.15.7:0/954584896 wait complete. 2023-03-03T19:16:19.635 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:16:19.645 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before thrashing 2023-03-03T19:16:21.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:16:21.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.784+0000 7fc8a0f4b700 1 Processor -- start 2023-03-03T19:16:21.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.785+0000 7fc8a0f4b700 1 -- start start 2023-03-03T19:16:21.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.785+0000 7fc8a0f4b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc89c14fda0 0x7fc89c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.785+0000 7fc8a0f4b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89c0c2f30 con 0x7fc89c0abe20 2023-03-03T19:16:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.785+0000 7fc8a0f4b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89c0c31d0 con 0x7fc89c14f9d0 2023-03-03T19:16:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.785+0000 7fc8a0f4b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc89c1524a0 con 0x7fc89c14fda0 2023-03-03T19:16:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc899d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc89c14fda0 0x7fc89c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc89ad9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc89c0abe20 0x7fc89c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34722/0 (socket says 172.21.15.7:34722) 2023-03-03T19:16:21.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc89ad9d700 1 -- 172.21.15.7:0/2319182518 learned_addr learned my addr 172.21.15.7:0/2319182518 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:16:21.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc899d9b700 1 -- 172.21.15.7:0/2319182518 >> v1:172.21.15.143:6789/0 conn(0x7fc89c14f9d0 legacy=0x7fc89c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:21.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc899d9b700 1 -- 172.21.15.7:0/2319182518 >> v1:172.21.15.7:6789/0 conn(0x7fc89c0abe20 legacy=0x7fc89c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:21.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc899d9b700 1 -- 172.21.15.7:0/2319182518 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c0c34b0 con 0x7fc89c14fda0 2023-03-03T19:16:21.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc89959a700 1 -- 172.21.15.7:0/2319182518 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126190356 0 0) 0x7fc89c0c2f30 con 0x7fc89c0abe20 2023-03-03T19:16:21.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc899d9b700 1 --2- 172.21.15.7:0/2319182518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 0x7fc89c156430 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7fc884009840 tx=0x7fc884009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97236ae2d02376b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:21.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc89959a700 1 -- 172.21.15.7:0/2319182518 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884011830 con 0x7fc89c14fda0 2023-03-03T19:16:21.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc89959a700 1 -- 172.21.15.7:0/2319182518 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc884011990 con 0x7fc89c14fda0 2023-03-03T19:16:21.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.786+0000 7fc89959a700 1 -- 172.21.15.7:0/2319182518 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884011d20 con 0x7fc89c14fda0 2023-03-03T19:16:21.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.787+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/2319182518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 msgr2=0x7fc89c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:21.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.787+0000 7fc8a0f4b700 1 --2- 172.21.15.7:0/2319182518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 0x7fc89c156430 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7fc884009840 tx=0x7fc884009b50 comp rx=0 tx=0).stop 2023-03-03T19:16:21.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.787+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/2319182518 shutdown_connections 2023-03-03T19:16:21.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.787+0000 7fc8a0f4b700 1 --2- 172.21.15.7:0/2319182518 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 0x7fc89c156430 unknown :-1 s=CLOSED pgs=3444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:21.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.787+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/2319182518 >> 172.21.15.7:0/2319182518 conn(0x7fc89c0c8900 msgr2=0x7fc89c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:16:21.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.787+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/2319182518 shutdown_connections 2023-03-03T19:16:21.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.787+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/2319182518 wait complete. 2023-03-03T19:16:21.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.788+0000 7fc8a0f4b700 1 Processor -- start 2023-03-03T19:16:21.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.788+0000 7fc8a0f4b700 1 -- start start 2023-03-03T19:16:21.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc8a0f4b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c0abe20 0x7fc89c1eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:21.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc8a0f4b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c14f9d0 0x7fc89c148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:21.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc8a0f4b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 0x7fc89c1490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:21.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc8a0f4b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c14c260 con 0x7fc89c14fda0 2023-03-03T19:16:21.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc8a0f4b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c06f6a0 con 0x7fc89c0abe20 2023-03-03T19:16:21.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc8a0f4b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c1eb960 con 0x7fc89c14f9d0 2023-03-03T19:16:21.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc89ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 0x7fc89c1490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:21.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc89ad9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 0x7fc89c1490d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40748/0 (socket says 172.21.15.7:40748) 2023-03-03T19:16:21.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc89ad9d700 1 -- 172.21.15.7:0/1021477405 learned_addr learned my addr 172.21.15.7:0/1021477405 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:16:21.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc89a59c700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c0abe20 0x7fc89c1eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:21.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.789+0000 7fc899d9b700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c14f9d0 0x7fc89c148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:21.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc89ad9d700 1 -- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c14f9d0 msgr2=0x7fc89c148b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:21.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc89ad9d700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c14f9d0 0x7fc89c148b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:21.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc89ad9d700 1 -- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c0abe20 msgr2=0x7fc89c1eb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:21.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc89ad9d700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c0abe20 0x7fc89c1eb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:21.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc89ad9d700 1 -- 172.21.15.7:0/1021477405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8840094a0 con 0x7fc89c14fda0 2023-03-03T19:16:21.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc89a59c700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c0abe20 0x7fc89c1eb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:16:21.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc89ad9d700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 0x7fc89c1490d0 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7fc89400cc10 tx=0x7fc8940021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:21.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc892ffd700 1 -- 172.21.15.7:0/1021477405 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89400ab80 con 0x7fc89c14fda0 2023-03-03T19:16:21.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc892ffd700 1 -- 172.21.15.7:0/1021477405 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89400ad00 con 0x7fc89c14fda0 2023-03-03T19:16:21.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/1021477405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89c0ac350 con 0x7fc89c14fda0 2023-03-03T19:16:21.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc892ffd700 1 -- 172.21.15.7:0/1021477405 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc894007310 con 0x7fc89c14fda0 2023-03-03T19:16:21.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.790+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/1021477405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89c0ab630 con 0x7fc89c14fda0 2023-03-03T19:16:21.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.791+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/1021477405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89c071fa0 con 0x7fc89c14fda0 2023-03-03T19:16:21.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.791+0000 7fc892ffd700 1 -- 172.21.15.7:0/1021477405 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc89401a030 con 0x7fc89c14fda0 2023-03-03T19:16:21.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.792+0000 7fc892ffd700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 0x7fc87c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:21.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.792+0000 7fc892ffd700 1 -- 172.21.15.7:0/1021477405 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc894085e80 con 0x7fc89c14fda0 2023-03-03T19:16:21.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.792+0000 7fc89a59c700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 0x7fc87c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:21.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.793+0000 7fc89a59c700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 0x7fc87c06a7a0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7fc89c155ae0 tx=0x7fc88c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:21.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:21.796+0000 7fc892ffd700 1 -- 172.21.15.7:0/1021477405 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc894052220 con 0x7fc89c14fda0 2023-03-03T19:16:22.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.033+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/1021477405 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc89c06fb80 con 0x7fc89c14fda0 2023-03-03T19:16:22.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.034+0000 7fc892ffd700 1 -- 172.21.15.7:0/1021477405 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc89c06fb80 con 0x7fc89c14fda0 2023-03-03T19:16:22.035 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:16:22.036 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:16:22.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 msgr2=0x7fc87c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 0x7fc87c06a7a0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7fc89c155ae0 tx=0x7fc88c009040 comp rx=0 tx=0).stop 2023-03-03T19:16:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/1021477405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 msgr2=0x7fc89c1490d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 0x7fc89c1490d0 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7fc89400cc10 tx=0x7fc8940021c0 comp rx=0 tx=0).stop 2023-03-03T19:16:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/1021477405 shutdown_connections 2023-03-03T19:16:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 0x7fc87c06a7a0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc89c14fda0 0x7fc89c1490d0 unknown :-1 s=CLOSED pgs=3445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:22.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc89c0abe20 0x7fc89c1eb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 --2- 172.21.15.7:0/1021477405 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc89c14f9d0 0x7fc89c148b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/1021477405 >> 172.21.15.7:0/1021477405 conn(0x7fc89c0c8900 msgr2=0x7fc89c151aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:16:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/1021477405 shutdown_connections 2023-03-03T19:16:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:22.036+0000 7fc8a0f4b700 1 -- 172.21.15.7:0/1021477405 wait complete. 2023-03-03T19:16:22.039 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:16:22.049 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-03-03T19:16:46.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:16:46.419 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:16:48.050 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:16:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.189+0000 7fae3dd43700 1 Processor -- start 2023-03-03T19:16:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.189+0000 7fae3dd43700 1 -- start start 2023-03-03T19:16:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae3dd43700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae3814b020 0x7fae380717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:48.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae3dd43700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fae380aad20 0x7fae38157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae3dd43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 0x7fae381581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae3dd43700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae380728b0 con 0x7fae3814b020 2023-03-03T19:16:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae3dd43700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae38155070 con 0x7fae380aad20 2023-03-03T19:16:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae3dd43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3806f5b0 con 0x7fae38071d20 2023-03-03T19:16:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae377fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae3814b020 0x7fae380717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae377fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fae3814b020 0x7fae380717e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40764/0 (socket says 172.21.15.7:40764) 2023-03-03T19:16:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae377fe700 1 -- 172.21.15.7:0/912393358 learned_addr learned my addr 172.21.15.7:0/912393358 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:16:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae37fff700 1 --2- 172.21.15.7:0/912393358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 0x7fae381581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.190+0000 7fae36ffd700 1 --2- 172.21.15.7:0/912393358 >> v2:172.21.15.143:3300/0 conn(0x7fae380aad20 0x7fae38157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae377fe700 1 -- 172.21.15.7:0/912393358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 msgr2=0x7fae381581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:16:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae377fe700 1 --2- 172.21.15.7:0/912393358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 0x7fae381581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae377fe700 1 -- 172.21.15.7:0/912393358 >> v2:172.21.15.143:3300/0 conn(0x7fae380aad20 msgr2=0x7fae38157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:16:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae377fe700 1 --2- 172.21.15.7:0/912393358 >> v2:172.21.15.143:3300/0 conn(0x7fae380aad20 0x7fae38157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae377fe700 1 -- 172.21.15.7:0/912393358 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3806f890 con 0x7fae3814b020 2023-03-03T19:16:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae37fff700 1 --2- 172.21.15.7:0/912393358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 0x7fae381581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:16:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae36ffd700 1 --2- 172.21.15.7:0/912393358 >> v2:172.21.15.143:3300/0 conn(0x7fae380aad20 0x7fae38157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:16:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae377fe700 1 --2- 172.21.15.7:0/912393358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae3814b020 0x7fae380717e0 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 crypto rx=0x7fae380a98d0 tx=0x7fae28009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd421814c77a63e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae367fc700 1 -- 172.21.15.7:0/912393358 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2800b870 con 0x7fae3814b020 2023-03-03T19:16:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae367fc700 1 -- 172.21.15.7:0/912393358 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2800b9d0 con 0x7fae3814b020 2023-03-03T19:16:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae367fc700 1 -- 172.21.15.7:0/912393358 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2800bd60 con 0x7fae3814b020 2023-03-03T19:16:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae3dd43700 1 -- 172.21.15.7:0/912393358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae3814b020 msgr2=0x7fae380717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.191+0000 7fae3dd43700 1 --2- 172.21.15.7:0/912393358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae3814b020 0x7fae380717e0 secure :-1 s=READY pgs=3446 cs=0 l=1 rev1=1 crypto rx=0x7fae380a98d0 tx=0x7fae28009bd0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.192+0000 7fae3dd43700 1 -- 172.21.15.7:0/912393358 shutdown_connections 2023-03-03T19:16:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.192+0000 7fae3dd43700 1 --2- 172.21.15.7:0/912393358 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 0x7fae381581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.192+0000 7fae3dd43700 1 --2- 172.21.15.7:0/912393358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae3814b020 0x7fae380717e0 unknown :-1 s=CLOSED pgs=3446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.192+0000 7fae3dd43700 1 --2- 172.21.15.7:0/912393358 >> v2:172.21.15.143:3300/0 conn(0x7fae380aad20 0x7fae38157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.192+0000 7fae3dd43700 1 -- 172.21.15.7:0/912393358 >> 172.21.15.7:0/912393358 conn(0x7fae380c8900 msgr2=0x7fae380c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:16:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.192+0000 7fae3dd43700 1 -- 172.21.15.7:0/912393358 shutdown_connections 2023-03-03T19:16:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.192+0000 7fae3dd43700 1 -- 172.21.15.7:0/912393358 wait complete. 2023-03-03T19:16:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.193+0000 7fae3dd43700 1 Processor -- start 2023-03-03T19:16:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.193+0000 7fae3dd43700 1 -- start start 2023-03-03T19:16:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae3dd43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 0x7fae381ed330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae3dd43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae380aad20 0x7fae381defe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae3dd43700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae3814b020 0x7fae381df520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae3dd43700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae381de8e0 con 0x7fae380aad20 2023-03-03T19:16:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae3dd43700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae381ddad0 con 0x7fae3814b020 2023-03-03T19:16:48.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae3dd43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae381deb50 con 0x7fae38071d20 2023-03-03T19:16:48.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae36ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae380aad20 0x7fae381defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:48.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae36ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae380aad20 0x7fae381defe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40772/0 (socket says 172.21.15.7:40772) 2023-03-03T19:16:48.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae36ffd700 1 -- 172.21.15.7:0/3568524668 learned_addr learned my addr 172.21.15.7:0/3568524668 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:16:48.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae377fe700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 0x7fae381ed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:48.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae37fff700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae3814b020 0x7fae381df520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:48.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae36ffd700 1 -- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 msgr2=0x7fae381ed330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae36ffd700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 0x7fae381ed330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae36ffd700 1 -- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae3814b020 msgr2=0x7fae381df520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae36ffd700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae3814b020 0x7fae381df520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae36ffd700 1 -- 172.21.15.7:0/3568524668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae28009550 con 0x7fae380aad20 2023-03-03T19:16:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae36ffd700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae380aad20 0x7fae381defe0 secure :-1 s=READY pgs=3447 cs=0 l=1 rev1=1 crypto rx=0x7fae2000c820 tx=0x7fae2000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.194+0000 7fae2ffff700 1 -- 172.21.15.7:0/3568524668 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20011800 con 0x7fae380aad20 2023-03-03T19:16:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.195+0000 7fae2ffff700 1 -- 172.21.15.7:0/3568524668 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae20011960 con 0x7fae380aad20 2023-03-03T19:16:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.195+0000 7fae3dd43700 1 -- 172.21.15.7:0/3568524668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae38155350 con 0x7fae380aad20 2023-03-03T19:16:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.195+0000 7fae2ffff700 1 -- 172.21.15.7:0/3568524668 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20011d10 con 0x7fae380aad20 2023-03-03T19:16:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.195+0000 7fae3dd43700 1 -- 172.21.15.7:0/3568524668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3806fb70 con 0x7fae380aad20 2023-03-03T19:16:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.195+0000 7fae3dd43700 1 -- 172.21.15.7:0/3568524668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae38158720 con 0x7fae380aad20 2023-03-03T19:16:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.196+0000 7fae2ffff700 1 -- 172.21.15.7:0/3568524668 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fae20017030 con 0x7fae380aad20 2023-03-03T19:16:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.196+0000 7fae2ffff700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae180682f0 0x7fae1806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:16:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.196+0000 7fae2ffff700 1 -- 172.21.15.7:0/3568524668 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fae20085b00 con 0x7fae380aad20 2023-03-03T19:16:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.197+0000 7fae377fe700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae180682f0 0x7fae1806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:16:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.197+0000 7fae377fe700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae180682f0 0x7fae1806a7a0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7fae28000c00 tx=0x7fae28008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:16:48.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.201+0000 7fae2ffff700 1 -- 172.21.15.7:0/3568524668 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fae20051ea0 con 0x7fae380aad20 2023-03-03T19:16:48.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.439+0000 7fae3dd43700 1 -- 172.21.15.7:0/3568524668 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae380c6110 con 0x7fae380aad20 2023-03-03T19:16:48.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.440+0000 7fae2ffff700 1 -- 172.21.15.7:0/3568524668 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fae200554c0 con 0x7fae380aad20 2023-03-03T19:16:48.440 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:16:48.440 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:16:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 -- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae180682f0 msgr2=0x7fae1806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae180682f0 0x7fae1806a7a0 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7fae28000c00 tx=0x7fae28008030 comp rx=0 tx=0).stop 2023-03-03T19:16:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 -- 172.21.15.7:0/3568524668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae380aad20 msgr2=0x7fae381defe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:16:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae380aad20 0x7fae381defe0 secure :-1 s=READY pgs=3447 cs=0 l=1 rev1=1 crypto rx=0x7fae2000c820 tx=0x7fae2000cb30 comp rx=0 tx=0).stop 2023-03-03T19:16:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 -- 172.21.15.7:0/3568524668 shutdown_connections 2023-03-03T19:16:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae3814b020 0x7fae381df520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fae38071d20 0x7fae381ed330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fae180682f0 0x7fae1806a7a0 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 --2- 172.21.15.7:0/3568524668 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae380aad20 0x7fae381defe0 unknown :-1 s=CLOSED pgs=3447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:16:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 -- 172.21.15.7:0/3568524668 >> 172.21.15.7:0/3568524668 conn(0x7fae380c8900 msgr2=0x7fae380acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:16:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 -- 172.21.15.7:0/3568524668 shutdown_connections 2023-03-03T19:16:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:16:48.442+0000 7fae3dd43700 1 -- 172.21.15.7:0/3568524668 wait complete. 2023-03-03T19:16:48.445 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:16:48.454 INFO:tasks.mds_thrash.fs.[a]:waiting for 106 secs before thrashing 2023-03-03T19:17:16.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:17:16.597 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:17:28.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:17:29.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.100+0000 7fe3a165a700 1 Processor -- start 2023-03-03T19:17:29.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.100+0000 7fe3a165a700 1 -- start start 2023-03-03T19:17:29.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe3a165a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 0x7fe39c0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:17:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe3a165a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe39c0c6fe0 0x7fe39c0bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:17:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe3a165a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c0c08d0 con 0x7fe39c155410 2023-03-03T19:17:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe3a165a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39c152250 con 0x7fe39c0c6fe0 2023-03-03T19:17:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe3a165a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c0c0fb0 con 0x7fe39c151e80 2023-03-03T19:17:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe39affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 0x7fe39c0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:17:29.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe39b7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe39c0c6fe0 0x7fe39c0bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:17:29.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe39b7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe39c0c6fe0 0x7fe39c0bfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:36580/0 (socket says 172.21.15.7:36580) 2023-03-03T19:17:29.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe39affd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 0x7fe39c0c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:48874/0 (socket says 172.21.15.7:48874) 2023-03-03T19:17:29.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe39b7fe700 1 -- 172.21.15.7:0/1226316169 learned_addr learned my addr 172.21.15.7:0/1226316169 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:17:29.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe39a7fc700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fe39c155410 0x7fe39c0bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:38302/0 (socket says 172.21.15.7:38302) 2023-03-03T19:17:29.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe39affd700 1 -- 172.21.15.7:0/1226316169 >> v2:172.21.15.143:3300/0 conn(0x7fe39c0c6fe0 msgr2=0x7fe39c0bfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:17:29.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe39affd700 1 --2- 172.21.15.7:0/1226316169 >> v2:172.21.15.143:3300/0 conn(0x7fe39c0c6fe0 0x7fe39c0bfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:17:29.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe39affd700 1 -- 172.21.15.7:0/1226316169 >> v1:172.21.15.143:6789/0 conn(0x7fe39c155410 legacy=0x7fe39c0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:17:29.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.101+0000 7fe39affd700 1 -- 172.21.15.7:0/1226316169 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c154930 con 0x7fe39c151e80 2023-03-03T19:17:29.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.102+0000 7fe39b7fe700 1 --2- 172.21.15.7:0/1226316169 >> v2:172.21.15.143:3300/0 conn(0x7fe39c0c6fe0 0x7fe39c0bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:17:29.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.102+0000 7fe39affd700 1 --2- 172.21.15.7:0/1226316169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 0x7fe39c0c6aa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe394009d90 tx=0x7fe39400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36915958a6bd1a00 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:17:29.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.102+0000 7fe399ffb700 1 -- 172.21.15.7:0/1226316169 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39400e070 con 0x7fe39c151e80 2023-03-03T19:17:29.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.102+0000 7fe399ffb700 1 -- 172.21.15.7:0/1226316169 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39400bb20 con 0x7fe39c151e80 2023-03-03T19:17:29.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.102+0000 7fe399ffb700 1 -- 172.21.15.7:0/1226316169 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394012430 con 0x7fe39c151e80 2023-03-03T19:17:29.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.103+0000 7fe3a165a700 1 -- 172.21.15.7:0/1226316169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 msgr2=0x7fe39c0c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:17:29.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.103+0000 7fe3a165a700 1 --2- 172.21.15.7:0/1226316169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 0x7fe39c0c6aa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe394009d90 tx=0x7fe39400b330 comp rx=0 tx=0).stop 2023-03-03T19:17:29.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.103+0000 7fe3a165a700 1 -- 172.21.15.7:0/1226316169 shutdown_connections 2023-03-03T19:17:29.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.103+0000 7fe3a165a700 1 --2- 172.21.15.7:0/1226316169 >> v2:172.21.15.143:3300/0 conn(0x7fe39c0c6fe0 0x7fe39c0bfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:17:29.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.103+0000 7fe3a165a700 1 --2- 172.21.15.7:0/1226316169 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 0x7fe39c0c6aa0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:17:29.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.103+0000 7fe3a165a700 1 -- 172.21.15.7:0/1226316169 >> 172.21.15.7:0/1226316169 conn(0x7fe39c071630 msgr2=0x7fe39c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:17:29.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.103+0000 7fe3a165a700 1 -- 172.21.15.7:0/1226316169 shutdown_connections 2023-03-03T19:17:29.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.103+0000 7fe3a165a700 1 -- 172.21.15.7:0/1226316169 wait complete. 2023-03-03T19:17:29.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.104+0000 7fe3a165a700 1 Processor -- start 2023-03-03T19:17:29.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.104+0000 7fe3a165a700 1 -- start start 2023-03-03T19:17:29.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe3a165a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe39c0c6fe0 0x7fe39c0c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:17:29.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe3a165a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 0x7fe39c0bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:17:29.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe3a165a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe39c155410 0x7fe39c14ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:17:29.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe3a165a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c14c2a0 con 0x7fe39c155410 2023-03-03T19:17:29.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe3a165a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1e0f60 con 0x7fe39c0c6fe0 2023-03-03T19:17:29.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe3a165a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c0c8aa0 con 0x7fe39c151e80 2023-03-03T19:17:29.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe39c155410 0x7fe39c14ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:17:29.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe39c155410 0x7fe39c14ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40812/0 (socket says 172.21.15.7:40812) 2023-03-03T19:17:29.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39b7fe700 1 -- 172.21.15.7:0/1485125009 learned_addr learned my addr 172.21.15.7:0/1485125009 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:17:29.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39a7fc700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 0x7fe39c0bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:17:29.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39affd700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe39c0c6fe0 0x7fe39c0c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:17:29.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39b7fe700 1 -- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 msgr2=0x7fe39c0bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:17:29.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39b7fe700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 0x7fe39c0bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:17:29.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39b7fe700 1 -- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe39c0c6fe0 msgr2=0x7fe39c0c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:17:29.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39b7fe700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe39c0c6fe0 0x7fe39c0c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:17:29.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39b7fe700 1 -- 172.21.15.7:0/1485125009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe390009550 con 0x7fe39c155410 2023-03-03T19:17:29.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.105+0000 7fe39b7fe700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe39c155410 0x7fe39c14ba00 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7fe39000e930 tx=0x7fe39000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:17:29.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.106+0000 7fe3837fe700 1 -- 172.21.15.7:0/1485125009 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390009700 con 0x7fe39c155410 2023-03-03T19:17:29.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.106+0000 7fe3a165a700 1 -- 172.21.15.7:0/1485125009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe394009a70 con 0x7fe39c155410 2023-03-03T19:17:29.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.106+0000 7fe3837fe700 1 -- 172.21.15.7:0/1485125009 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe390004020 con 0x7fe39c155410 2023-03-03T19:17:29.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.106+0000 7fe3837fe700 1 -- 172.21.15.7:0/1485125009 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3900043b0 con 0x7fe39c155410 2023-03-03T19:17:29.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.106+0000 7fe3a165a700 1 -- 172.21.15.7:0/1485125009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c154ef0 con 0x7fe39c155410 2023-03-03T19:17:29.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.107+0000 7fe3a165a700 1 -- 172.21.15.7:0/1485125009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39c154c10 con 0x7fe39c155410 2023-03-03T19:17:29.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.107+0000 7fe3837fe700 1 -- 172.21.15.7:0/1485125009 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe39000c850 con 0x7fe39c155410 2023-03-03T19:17:29.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.107+0000 7fe3837fe700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3880682f0 0x7fe38806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:17:29.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.108+0000 7fe3837fe700 1 -- 172.21.15.7:0/1485125009 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe390086150 con 0x7fe39c155410 2023-03-03T19:17:29.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.108+0000 7fe39affd700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3880682f0 0x7fe38806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:17:29.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.108+0000 7fe39affd700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3880682f0 0x7fe38806a7a0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fe394000f80 tx=0x7fe394011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:17:29.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.112+0000 7fe3837fe700 1 -- 172.21.15.7:0/1485125009 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe390052380 con 0x7fe39c155410 2023-03-03T19:17:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.348+0000 7fe3a165a700 1 -- 172.21.15.7:0/1485125009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe39c0c8c10 con 0x7fe39c155410 2023-03-03T19:17:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.349+0000 7fe3837fe700 1 -- 172.21.15.7:0/1485125009 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe390020020 con 0x7fe39c155410 2023-03-03T19:17:29.350 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:17:29.350 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:17:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.350+0000 7fe3a165a700 1 -- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3880682f0 msgr2=0x7fe38806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:17:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.350+0000 7fe3a165a700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3880682f0 0x7fe38806a7a0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fe394000f80 tx=0x7fe394011000 comp rx=0 tx=0).stop 2023-03-03T19:17:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.350+0000 7fe3a165a700 1 -- 172.21.15.7:0/1485125009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe39c155410 msgr2=0x7fe39c14ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:17:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.350+0000 7fe3a165a700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe39c155410 0x7fe39c14ba00 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7fe39000e930 tx=0x7fe39000ecf0 comp rx=0 tx=0).stop 2023-03-03T19:17:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.350+0000 7fe3a165a700 1 -- 172.21.15.7:0/1485125009 shutdown_connections 2023-03-03T19:17:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.350+0000 7fe3a165a700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe3880682f0 0x7fe38806a7a0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:17:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.350+0000 7fe3a165a700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe39c155410 0x7fe39c14ba00 unknown :-1 s=CLOSED pgs=3448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:17:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.350+0000 7fe3a165a700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe39c0c6fe0 0x7fe39c0c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:17:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.351+0000 7fe3a165a700 1 --2- 172.21.15.7:0/1485125009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe39c151e80 0x7fe39c0bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:17:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.351+0000 7fe3a165a700 1 -- 172.21.15.7:0/1485125009 >> 172.21.15.7:0/1485125009 conn(0x7fe39c071630 msgr2=0x7fe39c0c2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:17:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.351+0000 7fe3a165a700 1 -- 172.21.15.7:0/1485125009 shutdown_connections 2023-03-03T19:17:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:17:29.351+0000 7fe3a165a700 1 -- 172.21.15.7:0/1485125009 wait complete. 2023-03-03T19:17:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:17:29.364 INFO:tasks.mds_thrash.fs.[b]:waiting for 83 secs before thrashing 2023-03-03T19:17:46.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:17:46.660 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:18:16.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:18:16.723 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:18:34.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:18:34.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.594+0000 7f7baa34c700 1 Processor -- start 2023-03-03T19:18:34.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.594+0000 7f7baa34c700 1 -- start start 2023-03-03T19:18:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7baa34c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ba40c6940 0x7f7ba40c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7baa34c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ba40c5200 0x7f7ba40c5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7baa34c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba40c4a90 con 0x7f7ba4151ce0 2023-03-03T19:18:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7baa34c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba41520b0 con 0x7f7ba40c5200 2023-03-03T19:18:34.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7baa34c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba40c6600 con 0x7f7ba40c6940 2023-03-03T19:18:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7ba88e9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ba40c5200 0x7f7ba40c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7ba88e9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7ba40c5200 0x7f7ba40c5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40862/0 (socket says 172.21.15.7:40862) 2023-03-03T19:18:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7ba88e9700 1 -- 172.21.15.7:0/2923589174 learned_addr learned my addr 172.21.15.7:0/2923589174 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:18:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7ba37fe700 1 --2- 172.21.15.7:0/2923589174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ba40c6940 0x7f7ba40c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7ba88e9700 1 -- 172.21.15.7:0/2923589174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ba40c6940 msgr2=0x7f7ba40c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:18:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7ba88e9700 1 --2- 172.21.15.7:0/2923589174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ba40c6940 0x7f7ba40c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:34.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7ba88e9700 1 -- 172.21.15.7:0/2923589174 >> v1:172.21.15.143:6789/0 conn(0x7f7ba4151ce0 legacy=0x7f7ba4158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7ba88e9700 1 -- 172.21.15.7:0/2923589174 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba406d0a0 con 0x7f7ba40c5200 2023-03-03T19:18:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.595+0000 7f7ba88e9700 1 --2- 172.21.15.7:0/2923589174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 0x7f7ba40c5680 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009e50 tx=0x7f7b9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2cfb68cfa78b6f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:18:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7ba2ffd700 1 -- 172.21.15.7:0/2923589174 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c00e070 con 0x7f7ba40c5200 2023-03-03T19:18:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7ba2ffd700 1 -- 172.21.15.7:0/2923589174 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9c00b9c0 con 0x7f7ba40c5200 2023-03-03T19:18:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7ba2ffd700 1 -- 172.21.15.7:0/2923589174 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c00bd70 con 0x7f7ba40c5200 2023-03-03T19:18:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7baa34c700 1 -- 172.21.15.7:0/2923589174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 msgr2=0x7f7ba40c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:34.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7baa34c700 1 --2- 172.21.15.7:0/2923589174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 0x7f7ba40c5680 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009e50 tx=0x7f7b9c00b330 comp rx=0 tx=0).stop 2023-03-03T19:18:34.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7baa34c700 1 -- 172.21.15.7:0/2923589174 shutdown_connections 2023-03-03T19:18:34.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7baa34c700 1 --2- 172.21.15.7:0/2923589174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 0x7f7ba40c5680 unknown :-1 s=CLOSED pgs=3449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:34.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7baa34c700 1 --2- 172.21.15.7:0/2923589174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ba40c6940 0x7f7ba40c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:34.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7baa34c700 1 -- 172.21.15.7:0/2923589174 >> 172.21.15.7:0/2923589174 conn(0x7f7ba406a2b0 msgr2=0x7f7ba406a680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:18:34.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7baa34c700 1 -- 172.21.15.7:0/2923589174 shutdown_connections 2023-03-03T19:18:34.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.596+0000 7f7baa34c700 1 -- 172.21.15.7:0/2923589174 wait complete. 2023-03-03T19:18:34.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.597+0000 7f7baa34c700 1 Processor -- start 2023-03-03T19:18:34.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.598+0000 7f7baa34c700 1 -- start start 2023-03-03T19:18:34.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.598+0000 7f7baa34c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 0x7f7ba414c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:34.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.598+0000 7f7baa34c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ba40c6940 0x7f7ba40bfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:34.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.598+0000 7f7baa34c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba4151ce0 0x7f7ba4153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:34.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.598+0000 7f7baa34c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba4153680 con 0x7f7ba40c5200 2023-03-03T19:18:34.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.598+0000 7f7baa34c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba4155440 con 0x7f7ba4151ce0 2023-03-03T19:18:34.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.598+0000 7f7baa34c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba414c8c0 con 0x7f7ba40c6940 2023-03-03T19:18:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.598+0000 7f7ba3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 0x7f7ba414c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.598+0000 7f7ba3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 0x7f7ba414c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40864/0 (socket says 172.21.15.7:40864) 2023-03-03T19:18:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.598+0000 7f7ba3fff700 1 -- 172.21.15.7:0/2641679417 learned_addr learned my addr 172.21.15.7:0/2641679417 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:18:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba37fe700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ba40c6940 0x7f7ba40bfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:34.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba88e9700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba4151ce0 0x7f7ba4153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba3fff700 1 -- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ba40c6940 msgr2=0x7f7ba40bfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba3fff700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ba40c6940 0x7f7ba40bfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba3fff700 1 -- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba4151ce0 msgr2=0x7f7ba4153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba3fff700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba4151ce0 0x7f7ba4153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba3fff700 1 -- 172.21.15.7:0/2641679417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c009b30 con 0x7f7ba40c5200 2023-03-03T19:18:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba88e9700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba4151ce0 0x7f7ba4153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:18:34.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba3fff700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 0x7f7ba414c380 secure :-1 s=READY pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7f7b9400c510 tx=0x7f7b9400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:18:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba0ff9700 1 -- 172.21.15.7:0/2641679417 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b940052d0 con 0x7f7ba40c5200 2023-03-03T19:18:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba0ff9700 1 -- 172.21.15.7:0/2641679417 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b94005430 con 0x7f7ba40c5200 2023-03-03T19:18:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7baa34c700 1 -- 172.21.15.7:0/2641679417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba406d660 con 0x7f7ba40c5200 2023-03-03T19:18:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7ba0ff9700 1 -- 172.21.15.7:0/2641679417 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b940038c0 con 0x7f7ba40c5200 2023-03-03T19:18:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.599+0000 7f7baa34c700 1 -- 172.21.15.7:0/2641679417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba406d380 con 0x7f7ba40c5200 2023-03-03T19:18:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.600+0000 7f7baa34c700 1 -- 172.21.15.7:0/2641679417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba40c4ce0 con 0x7f7ba40c5200 2023-03-03T19:18:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.601+0000 7f7ba0ff9700 1 -- 172.21.15.7:0/2641679417 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7b94019030 con 0x7f7ba40c5200 2023-03-03T19:18:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.601+0000 7f7ba0ff9700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b900682f0 0x7f7b9006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.601+0000 7f7ba0ff9700 1 -- 172.21.15.7:0/2641679417 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7b94085460 con 0x7f7ba40c5200 2023-03-03T19:18:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.601+0000 7f7ba37fe700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b900682f0 0x7f7b9006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.602+0000 7f7ba37fe700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b900682f0 0x7f7b9006a7a0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f7ba40aa3d0 tx=0x7f7b98008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:18:34.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.605+0000 7f7ba0ff9700 1 -- 172.21.15.7:0/2641679417 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7b94051800 con 0x7f7ba40c5200 2023-03-03T19:18:34.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.840+0000 7f7baa34c700 1 -- 172.21.15.7:0/2641679417 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ba41538f0 con 0x7f7ba40c5200 2023-03-03T19:18:34.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.841+0000 7f7ba0ff9700 1 -- 172.21.15.7:0/2641679417 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7b94054e20 con 0x7f7ba40c5200 2023-03-03T19:18:34.842 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:18:34.842 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:18:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.842+0000 7f7baa34c700 1 -- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b900682f0 msgr2=0x7f7b9006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.842+0000 7f7baa34c700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b900682f0 0x7f7b9006a7a0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f7ba40aa3d0 tx=0x7f7b98008040 comp rx=0 tx=0).stop 2023-03-03T19:18:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.842+0000 7f7baa34c700 1 -- 172.21.15.7:0/2641679417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 msgr2=0x7f7ba414c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.842+0000 7f7baa34c700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 0x7f7ba414c380 secure :-1 s=READY pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7f7b9400c510 tx=0x7f7b9400c820 comp rx=0 tx=0).stop 2023-03-03T19:18:34.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.843+0000 7f7baa34c700 1 -- 172.21.15.7:0/2641679417 shutdown_connections 2023-03-03T19:18:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.843+0000 7f7baa34c700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba4151ce0 0x7f7ba4153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.843+0000 7f7baa34c700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7b900682f0 0x7f7b9006a7a0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.843+0000 7f7baa34c700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ba40c5200 0x7f7ba414c380 unknown :-1 s=CLOSED pgs=3450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.843+0000 7f7baa34c700 1 --2- 172.21.15.7:0/2641679417 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7ba40c6940 0x7f7ba40bfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.843+0000 7f7baa34c700 1 -- 172.21.15.7:0/2641679417 >> 172.21.15.7:0/2641679417 conn(0x7f7ba406a2b0 msgr2=0x7f7ba4065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:18:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.843+0000 7f7baa34c700 1 -- 172.21.15.7:0/2641679417 shutdown_connections 2023-03-03T19:18:34.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:34.843+0000 7f7baa34c700 1 -- 172.21.15.7:0/2641679417 wait complete. 2023-03-03T19:18:34.846 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:18:34.856 INFO:tasks.mds_thrash.fs.[a]:waiting for 93 secs before thrashing 2023-03-03T19:18:46.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:18:46.805 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:18:52.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:18:52.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.504+0000 7fa3db75c700 1 Processor -- start 2023-03-03T19:18:52.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.505+0000 7fa3db75c700 1 -- start start 2023-03-03T19:18:52.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.505+0000 7fa3db75c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d40c43d0 0x7fa3d40c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:52.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.505+0000 7fa3db75c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d4072920 con 0x7fa3d4154590 2023-03-03T19:18:52.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.505+0000 7fa3db75c700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3d406bef0 con 0x7fa3d414e560 2023-03-03T19:18:52.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.505+0000 7fa3db75c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d4072380 con 0x7fa3d40c43d0 2023-03-03T19:18:52.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.505+0000 7fa3d9cf9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa3d4154590 0x7fa3d41549f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34864/0 (socket says 172.21.15.7:34864) 2023-03-03T19:18:52.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3d9cf9700 1 -- 172.21.15.7:0/1732598063 learned_addr learned my addr 172.21.15.7:0/1732598063 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:18:52.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3d8cf7700 1 --2- 172.21.15.7:0/1732598063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d40c43d0 0x7fa3d40c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:52.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3c7fff700 1 -- 172.21.15.7:0/1732598063 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394279685 0 0) 0x7fa3cc001490 con 0x7fa3d4154590 2023-03-03T19:18:52.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3c7fff700 1 -- 172.21.15.7:0/1732598063 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c00035b0 con 0x7fa3d4154590 2023-03-03T19:18:52.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3c7fff700 1 -- 172.21.15.7:0/1732598063 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239449862 0 0) 0x7fa3d406bef0 con 0x7fa3d414e560 2023-03-03T19:18:52.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3c7fff700 1 -- 172.21.15.7:0/1732598063 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3cc001490 con 0x7fa3d414e560 2023-03-03T19:18:52.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3d8cf7700 1 -- 172.21.15.7:0/1732598063 >> v1:172.21.15.143:6789/0 conn(0x7fa3d414e560 legacy=0x7fa3d4071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:52.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3d8cf7700 1 -- 172.21.15.7:0/1732598063 >> v1:172.21.15.7:6789/0 conn(0x7fa3d4154590 legacy=0x7fa3d41549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:52.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3c7fff700 1 -- 172.21.15.7:0/1732598063 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1907166956 0 0) 0x7fa3c00035b0 con 0x7fa3d4154590 2023-03-03T19:18:52.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3d8cf7700 1 -- 172.21.15.7:0/1732598063 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d406b910 con 0x7fa3d40c43d0 2023-03-03T19:18:52.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3d8cf7700 1 --2- 172.21.15.7:0/1732598063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d40c43d0 0x7fa3d40c4820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009870 tx=0x7fa3d0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e73076cd20d0fd89 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:18:52.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.506+0000 7fa3c7fff700 1 -- 172.21.15.7:0/1732598063 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc001b80 con 0x7fa3d40c43d0 2023-03-03T19:18:52.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.507+0000 7fa3c7fff700 1 -- 172.21.15.7:0/1732598063 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d0004030 con 0x7fa3d40c43d0 2023-03-03T19:18:52.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.507+0000 7fa3c7fff700 1 -- 172.21.15.7:0/1732598063 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d00043c0 con 0x7fa3d40c43d0 2023-03-03T19:18:52.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.507+0000 7fa3db75c700 1 -- 172.21.15.7:0/1732598063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d40c43d0 msgr2=0x7fa3d40c4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:52.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.507+0000 7fa3db75c700 1 --2- 172.21.15.7:0/1732598063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d40c43d0 0x7fa3d40c4820 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009870 tx=0x7fa3d0009b80 comp rx=0 tx=0).stop 2023-03-03T19:18:52.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.507+0000 7fa3db75c700 1 -- 172.21.15.7:0/1732598063 shutdown_connections 2023-03-03T19:18:52.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.507+0000 7fa3db75c700 1 --2- 172.21.15.7:0/1732598063 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d40c43d0 0x7fa3d40c4820 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:52.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.507+0000 7fa3db75c700 1 -- 172.21.15.7:0/1732598063 >> 172.21.15.7:0/1732598063 conn(0x7fa3d40736f0 msgr2=0x7fa3d4073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:18:52.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.507+0000 7fa3db75c700 1 -- 172.21.15.7:0/1732598063 shutdown_connections 2023-03-03T19:18:52.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.507+0000 7fa3db75c700 1 -- 172.21.15.7:0/1732598063 wait complete. 2023-03-03T19:18:52.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.508+0000 7fa3db75c700 1 Processor -- start 2023-03-03T19:18:52.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.508+0000 7fa3db75c700 1 -- start start 2023-03-03T19:18:52.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3db75c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d40c43d0 0x7fa3d414c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:52.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3db75c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d414e560 0x7fa3d40c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:52.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3db75c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3d4154590 0x7fa3d40be420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:52.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3db75c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40abdf0 con 0x7fa3d4154590 2023-03-03T19:18:52.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3db75c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40ab460 con 0x7fa3d40c43d0 2023-03-03T19:18:52.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3db75c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d40c71e0 con 0x7fa3d414e560 2023-03-03T19:18:52.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3d9cf9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3d4154590 0x7fa3d40be420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:52.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3d9cf9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3d4154590 0x7fa3d40be420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40890/0 (socket says 172.21.15.7:40890) 2023-03-03T19:18:52.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3d9cf9700 1 -- 172.21.15.7:0/3432719845 learned_addr learned my addr 172.21.15.7:0/3432719845 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:18:52.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3d8cf7700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d414e560 0x7fa3d40c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:52.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.509+0000 7fa3d94f8700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d40c43d0 0x7fa3d414c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:52.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3d9cf9700 1 -- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d414e560 msgr2=0x7fa3d40c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:52.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3d9cf9700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d414e560 0x7fa3d40c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:52.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3d9cf9700 1 -- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d40c43d0 msgr2=0x7fa3d414c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:52.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3d9cf9700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d40c43d0 0x7fa3d414c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:52.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3d9cf9700 1 -- 172.21.15.7:0/3432719845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d00094a0 con 0x7fa3d4154590 2023-03-03T19:18:52.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3d9cf9700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3d4154590 0x7fa3d40be420 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc001ab0 tx=0x7fa3cc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:18:52.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3c5ffb700 1 -- 172.21.15.7:0/3432719845 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc00acf0 con 0x7fa3d4154590 2023-03-03T19:18:52.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3c5ffb700 1 -- 172.21.15.7:0/3432719845 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3cc00ae50 con 0x7fa3d4154590 2023-03-03T19:18:52.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3db75c700 1 -- 172.21.15.7:0/3432719845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d406bb80 con 0x7fa3d4154590 2023-03-03T19:18:52.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3c5ffb700 1 -- 172.21.15.7:0/3432719845 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc007310 con 0x7fa3d4154590 2023-03-03T19:18:52.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.510+0000 7fa3db75c700 1 -- 172.21.15.7:0/3432719845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d40aacc0 con 0x7fa3d4154590 2023-03-03T19:18:52.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.511+0000 7fa3db75c700 1 -- 172.21.15.7:0/3432719845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d4155100 con 0x7fa3d4154590 2023-03-03T19:18:52.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.512+0000 7fa3c5ffb700 1 -- 172.21.15.7:0/3432719845 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa3cc01a030 con 0x7fa3d4154590 2023-03-03T19:18:52.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.512+0000 7fa3c5ffb700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3b80682f0 0x7fa3b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:52.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.512+0000 7fa3c5ffb700 1 -- 172.21.15.7:0/3432719845 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa3cc085d80 con 0x7fa3d4154590 2023-03-03T19:18:52.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.512+0000 7fa3d94f8700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3b80682f0 0x7fa3b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:52.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.513+0000 7fa3d94f8700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3b80682f0 0x7fa3b806a7a0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80011e0 tx=0x7fa3c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:18:52.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.517+0000 7fa3c5ffb700 1 -- 172.21.15.7:0/3432719845 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa3cc0520c0 con 0x7fa3d4154590 2023-03-03T19:18:52.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.758+0000 7fa3db75c700 1 -- 172.21.15.7:0/3432719845 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3d40c7320 con 0x7fa3d4154590 2023-03-03T19:18:52.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.758+0000 7fa3c5ffb700 1 -- 172.21.15.7:0/3432719845 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa3cc059680 con 0x7fa3d4154590 2023-03-03T19:18:52.759 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:18:52.759 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:18:52.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.760+0000 7fa3db75c700 1 -- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3b80682f0 msgr2=0x7fa3b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:52.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.760+0000 7fa3db75c700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3b80682f0 0x7fa3b806a7a0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80011e0 tx=0x7fa3c8009040 comp rx=0 tx=0).stop 2023-03-03T19:18:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.760+0000 7fa3db75c700 1 -- 172.21.15.7:0/3432719845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3d4154590 msgr2=0x7fa3d40be420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.760+0000 7fa3db75c700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3d4154590 0x7fa3d40be420 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc001ab0 tx=0x7fa3cc00a340 comp rx=0 tx=0).stop 2023-03-03T19:18:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.761+0000 7fa3db75c700 1 -- 172.21.15.7:0/3432719845 shutdown_connections 2023-03-03T19:18:52.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.761+0000 7fa3db75c700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa3b80682f0 0x7fa3b806a7a0 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:52.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.761+0000 7fa3db75c700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa3d4154590 0x7fa3d40be420 unknown :-1 s=CLOSED pgs=3452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:52.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.761+0000 7fa3db75c700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d40c43d0 0x7fa3d414c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:52.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.761+0000 7fa3db75c700 1 --2- 172.21.15.7:0/3432719845 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa3d414e560 0x7fa3d40c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:52.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.761+0000 7fa3db75c700 1 -- 172.21.15.7:0/3432719845 >> 172.21.15.7:0/3432719845 conn(0x7fa3d40736f0 msgr2=0x7fa3d40a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:18:52.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.761+0000 7fa3db75c700 1 -- 172.21.15.7:0/3432719845 shutdown_connections 2023-03-03T19:18:52.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:52.761+0000 7fa3db75c700 1 -- 172.21.15.7:0/3432719845 wait complete. 2023-03-03T19:18:52.766 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:18:52.773 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before thrashing 2023-03-03T19:18:53.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:18:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.914+0000 7f81dcab0700 1 Processor -- start 2023-03-03T19:18:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.914+0000 7f81dcab0700 1 -- start start 2023-03-03T19:18:53.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81dcab0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81d8155390 0x7f81d80c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:53.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81dcab0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f81d8151770 0x7f81d8151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:53.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81dcab0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d8154910 con 0x7f81d806e020 2023-03-03T19:18:53.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81dcab0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d8070b90 con 0x7f81d8155390 2023-03-03T19:18:53.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81dcab0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d8151320 con 0x7f81d8151770 2023-03-03T19:18:53.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d5d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f81d806e020 0x7f81d8071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34874/0 (socket says 172.21.15.7:34874) 2023-03-03T19:18:53.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d659c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81d8155390 0x7f81d80c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:53.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d5d9b700 1 -- 172.21.15.7:0/3978546024 learned_addr learned my addr 172.21.15.7:0/3978546024 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:18:53.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d6d9d700 1 --2- 172.21.15.7:0/3978546024 >> v2:172.21.15.143:3300/0 conn(0x7f81d8151770 0x7f81d8151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:53.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d559a700 1 -- 172.21.15.7:0/3978546024 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054395987 0 0) 0x7f81d8154910 con 0x7f81d806e020 2023-03-03T19:18:53.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d559a700 1 -- 172.21.15.7:0/3978546024 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c00035b0 con 0x7f81d806e020 2023-03-03T19:18:53.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d659c700 1 -- 172.21.15.7:0/3978546024 >> v2:172.21.15.143:3300/0 conn(0x7f81d8151770 msgr2=0x7f81d8151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:18:53.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d659c700 1 --2- 172.21.15.7:0/3978546024 >> v2:172.21.15.143:3300/0 conn(0x7f81d8151770 0x7f81d8151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:53.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d659c700 1 -- 172.21.15.7:0/3978546024 >> v1:172.21.15.7:6789/0 conn(0x7f81d806e020 legacy=0x7f81d8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:53.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d659c700 1 -- 172.21.15.7:0/3978546024 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d80702a0 con 0x7f81d8155390 2023-03-03T19:18:53.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d659c700 1 --2- 172.21.15.7:0/3978546024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 0x7f81d80c3e30 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f81d806ffe0 tx=0x7f81cc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14a7097372589e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:18:53.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d559a700 1 -- 172.21.15.7:0/3978546024 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc011830 con 0x7f81d8155390 2023-03-03T19:18:53.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.915+0000 7f81d559a700 1 -- 172.21.15.7:0/3978546024 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc011990 con 0x7f81d8155390 2023-03-03T19:18:53.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.916+0000 7f81d559a700 1 -- 172.21.15.7:0/3978546024 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc011d20 con 0x7f81d8155390 2023-03-03T19:18:53.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.916+0000 7f81dcab0700 1 -- 172.21.15.7:0/3978546024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 msgr2=0x7f81d80c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:53.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.916+0000 7f81dcab0700 1 --2- 172.21.15.7:0/3978546024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 0x7f81d80c3e30 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f81d806ffe0 tx=0x7f81cc009b20 comp rx=0 tx=0).stop 2023-03-03T19:18:53.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.916+0000 7f81dcab0700 1 -- 172.21.15.7:0/3978546024 shutdown_connections 2023-03-03T19:18:53.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.916+0000 7f81dcab0700 1 --2- 172.21.15.7:0/3978546024 >> v2:172.21.15.143:3300/0 conn(0x7f81d8151770 0x7f81d8151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:53.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.916+0000 7f81dcab0700 1 --2- 172.21.15.7:0/3978546024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 0x7f81d80c3e30 unknown :-1 s=CLOSED pgs=3454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:53.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.916+0000 7f81dcab0700 1 -- 172.21.15.7:0/3978546024 >> 172.21.15.7:0/3978546024 conn(0x7f81d806c110 msgr2=0x7f81d806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:18:53.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.916+0000 7f81dcab0700 1 -- 172.21.15.7:0/3978546024 shutdown_connections 2023-03-03T19:18:53.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.916+0000 7f81dcab0700 1 -- 172.21.15.7:0/3978546024 wait complete. 2023-03-03T19:18:53.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.918+0000 7f81dcab0700 1 Processor -- start 2023-03-03T19:18:53.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.918+0000 7f81dcab0700 1 -- start start 2023-03-03T19:18:53.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81dcab0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81d806e020 0x7f81d81dbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:53.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81dcab0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81d8151770 0x7f81d8152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:53.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81dcab0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 0x7f81d8152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:53.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81dcab0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80c5ec0 con 0x7f81d8155390 2023-03-03T19:18:53.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81dcab0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80c5990 con 0x7f81d8151770 2023-03-03T19:18:53.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81dcab0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80c7120 con 0x7f81d806e020 2023-03-03T19:18:53.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 0x7f81d8152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:53.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d6d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 0x7f81d8152d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40904/0 (socket says 172.21.15.7:40904) 2023-03-03T19:18:53.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d6d9d700 1 -- 172.21.15.7:0/1128176660 learned_addr learned my addr 172.21.15.7:0/1128176660 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:18:53.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d5d9b700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81d8151770 0x7f81d8152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:53.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d659c700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81d806e020 0x7f81d81dbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:53.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d6d9d700 1 -- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81d806e020 msgr2=0x7f81d81dbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:53.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d6d9d700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81d806e020 0x7f81d81dbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d6d9d700 1 -- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81d8151770 msgr2=0x7f81d8152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d6d9d700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81d8151770 0x7f81d8152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d6d9d700 1 -- 172.21.15.7:0/1128176660 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc0094a0 con 0x7f81d8155390 2023-03-03T19:18:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d5d9b700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81d8151770 0x7f81d8152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:18:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.919+0000 7f81d6d9d700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 0x7f81d8152d50 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f81d000efc0 tx=0x7f81d000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:18:53.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.920+0000 7f81c6ffd700 1 -- 172.21.15.7:0/1128176660 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d000cd40 con 0x7f81d8155390 2023-03-03T19:18:53.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.920+0000 7f81c6ffd700 1 -- 172.21.15.7:0/1128176660 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d000f040 con 0x7f81d8155390 2023-03-03T19:18:53.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.920+0000 7f81dcab0700 1 -- 172.21.15.7:0/1128176660 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d8070820 con 0x7f81d8155390 2023-03-03T19:18:53.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.920+0000 7f81c6ffd700 1 -- 172.21.15.7:0/1128176660 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d0012670 con 0x7f81d8155390 2023-03-03T19:18:53.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.920+0000 7f81dcab0700 1 -- 172.21.15.7:0/1128176660 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d8070540 con 0x7f81d8155390 2023-03-03T19:18:53.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.921+0000 7f81dcab0700 1 -- 172.21.15.7:0/1128176660 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d8155760 con 0x7f81d8155390 2023-03-03T19:18:53.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.921+0000 7f81c6ffd700 1 -- 172.21.15.7:0/1128176660 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f81d00127d0 con 0x7f81d8155390 2023-03-03T19:18:53.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.922+0000 7f81c6ffd700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81b80682f0 0x7f81b806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:18:53.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.922+0000 7f81c6ffd700 1 -- 172.21.15.7:0/1128176660 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f81d0016070 con 0x7f81d8155390 2023-03-03T19:18:53.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.922+0000 7f81d659c700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81b80682f0 0x7f81b806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:18:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.922+0000 7f81d659c700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81b80682f0 0x7f81b806a7a0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0090b0 tx=0x7f81cc01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:18:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:53.926+0000 7f81c6ffd700 1 -- 172.21.15.7:0/1128176660 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81d00550c0 con 0x7f81d8155390 2023-03-03T19:18:54.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.165+0000 7f81dcab0700 1 -- 172.21.15.7:0/1128176660 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81d8074bb0 con 0x7f81d8155390 2023-03-03T19:18:54.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.165+0000 7f81c6ffd700 1 -- 172.21.15.7:0/1128176660 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f81d000cea0 con 0x7f81d8155390 2023-03-03T19:18:54.166 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:18:54.166 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:18:54.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.167+0000 7f81dcab0700 1 -- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81b80682f0 msgr2=0x7f81b806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:54.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.167+0000 7f81dcab0700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81b80682f0 0x7f81b806a7a0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0090b0 tx=0x7f81cc01c040 comp rx=0 tx=0).stop 2023-03-03T19:18:54.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.167+0000 7f81dcab0700 1 -- 172.21.15.7:0/1128176660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 msgr2=0x7f81d8152d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:18:54.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.167+0000 7f81dcab0700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 0x7f81d8152d50 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f81d000efc0 tx=0x7f81d000c3a0 comp rx=0 tx=0).stop 2023-03-03T19:18:54.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.168+0000 7f81dcab0700 1 -- 172.21.15.7:0/1128176660 shutdown_connections 2023-03-03T19:18:54.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.168+0000 7f81dcab0700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81b80682f0 0x7f81b806a7a0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:54.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.168+0000 7f81dcab0700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81d8155390 0x7f81d8152d50 unknown :-1 s=CLOSED pgs=3455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:54.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.168+0000 7f81dcab0700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f81d806e020 0x7f81d81dbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:54.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.168+0000 7f81dcab0700 1 --2- 172.21.15.7:0/1128176660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81d8151770 0x7f81d8152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:18:54.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.168+0000 7f81dcab0700 1 -- 172.21.15.7:0/1128176660 >> 172.21.15.7:0/1128176660 conn(0x7f81d806c110 msgr2=0x7f81d814e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:18:54.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.168+0000 7f81dcab0700 1 -- 172.21.15.7:0/1128176660 shutdown_connections 2023-03-03T19:18:54.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:18:54.168+0000 7f81dcab0700 1 -- 172.21.15.7:0/1128176660 wait complete. 2023-03-03T19:18:54.171 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:18:54.180 INFO:tasks.mds_thrash.fs.[b]:waiting for 78 secs before thrashing 2023-03-03T19:19:16.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:19:16.902 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:19:46.962 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:19:46.965 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:19:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T19:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T19:20:07.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:20:08.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.009+0000 7fbb6bb7c700 1 Processor -- start 2023-03-03T19:20:08.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.009+0000 7fbb6bb7c700 1 -- start start 2023-03-03T19:20:08.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb6bb7c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbb64071e80 0x7fbb64151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:08.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb6bb7c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb64072250 0x7fbb6414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:08.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb6bb7c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb641510d0 con 0x7fbb640c1ce0 2023-03-03T19:20:08.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb6bb7c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb640c3740 con 0x7fbb64072250 2023-03-03T19:20:08.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb6bb7c700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb64070670 con 0x7fbb64071e80 2023-03-03T19:20:08.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb69117700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb64072250 0x7fbb6414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:08.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb6a119700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbb640c1ce0 0x7fbb641546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34956/0 (socket says 172.21.15.7:34956) 2023-03-03T19:20:08.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb6a119700 1 -- 172.21.15.7:0/222509154 learned_addr learned my addr 172.21.15.7:0/222509154 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:20:08.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb69918700 1 --2- 172.21.15.7:0/222509154 >> v2:172.21.15.143:3300/0 conn(0x7fbb64071e80 0x7fbb64151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:08.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb69117700 1 -- 172.21.15.7:0/222509154 >> v2:172.21.15.143:3300/0 conn(0x7fbb64071e80 msgr2=0x7fbb64151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:20:08.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb69117700 1 --2- 172.21.15.7:0/222509154 >> v2:172.21.15.143:3300/0 conn(0x7fbb64071e80 0x7fbb64151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:08.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb69117700 1 -- 172.21.15.7:0/222509154 >> v1:172.21.15.7:6789/0 conn(0x7fbb640c1ce0 legacy=0x7fbb641546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb69117700 1 -- 172.21.15.7:0/222509154 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640c39e0 con 0x7fbb64072250 2023-03-03T19:20:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.010+0000 7fbb69918700 1 --2- 172.21.15.7:0/222509154 >> v2:172.21.15.143:3300/0 conn(0x7fbb64071e80 0x7fbb64151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:20:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb69117700 1 --2- 172.21.15.7:0/222509154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb64072250 0x7fbb6414e1f0 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009840 tx=0x7fbb60009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2021d32518d39bf4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:20:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb68916700 1 -- 172.21.15.7:0/222509154 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60011830 con 0x7fbb64072250 2023-03-03T19:20:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb68916700 1 -- 172.21.15.7:0/222509154 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60011990 con 0x7fbb64072250 2023-03-03T19:20:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb68916700 1 -- 172.21.15.7:0/222509154 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60011d20 con 0x7fbb64072250 2023-03-03T19:20:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/222509154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb64072250 msgr2=0x7fbb6414e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb6bb7c700 1 --2- 172.21.15.7:0/222509154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb64072250 0x7fbb6414e1f0 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009840 tx=0x7fbb60009b50 comp rx=0 tx=0).stop 2023-03-03T19:20:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/222509154 shutdown_connections 2023-03-03T19:20:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb6bb7c700 1 --2- 172.21.15.7:0/222509154 >> v2:172.21.15.143:3300/0 conn(0x7fbb64071e80 0x7fbb64151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb6bb7c700 1 --2- 172.21.15.7:0/222509154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb64072250 0x7fbb6414e1f0 unknown :-1 s=CLOSED pgs=3459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/222509154 >> 172.21.15.7:0/222509154 conn(0x7fbb640a94d0 msgr2=0x7fbb640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:20:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/222509154 shutdown_connections 2023-03-03T19:20:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.011+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/222509154 wait complete. 2023-03-03T19:20:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6bb7c700 1 Processor -- start 2023-03-03T19:20:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6bb7c700 1 -- start start 2023-03-03T19:20:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6bb7c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb64071e80 0x7fbb6415e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6bb7c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb64072250 0x7fbb64147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6bb7c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb640c1ce0 0x7fbb6414ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6bb7c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6415fb40 con 0x7fbb640c1ce0 2023-03-03T19:20:08.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6bb7c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640ac7c0 con 0x7fbb64072250 2023-03-03T19:20:08.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6bb7c700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6415ed40 con 0x7fbb64071e80 2023-03-03T19:20:08.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6a119700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb640c1ce0 0x7fbb6414ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:08.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6a119700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb640c1ce0 0x7fbb6414ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40982/0 (socket says 172.21.15.7:40982) 2023-03-03T19:20:08.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.013+0000 7fbb6a119700 1 -- 172.21.15.7:0/3051659649 learned_addr learned my addr 172.21.15.7:0/3051659649 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:20:08.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb69918700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb64071e80 0x7fbb6415e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:08.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb69117700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb64072250 0x7fbb64147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:08.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb6a119700 1 -- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb64071e80 msgr2=0x7fbb6415e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:08.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb6a119700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb64071e80 0x7fbb6415e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:08.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb6a119700 1 -- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb64072250 msgr2=0x7fbb64147b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:08.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb6a119700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb64072250 0x7fbb64147b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:08.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb6a119700 1 -- 172.21.15.7:0/3051659649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb600094a0 con 0x7fbb640c1ce0 2023-03-03T19:20:08.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb6a119700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb640c1ce0 0x7fbb6414ae90 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00cad0 tx=0x7fbb5c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:20:08.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb5a7fc700 1 -- 172.21.15.7:0/3051659649 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00a9a0 con 0x7fbb640c1ce0 2023-03-03T19:20:08.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb5a7fc700 1 -- 172.21.15.7:0/3051659649 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c00ab00 con 0x7fbb640c1ce0 2023-03-03T19:20:08.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/3051659649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6406c9a0 con 0x7fbb640c1ce0 2023-03-03T19:20:08.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb5a7fc700 1 -- 172.21.15.7:0/3051659649 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c00ae70 con 0x7fbb640c1ce0 2023-03-03T19:20:08.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.014+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/3051659649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb64151300 con 0x7fbb640c1ce0 2023-03-03T19:20:08.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.015+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/3051659649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6406cb30 con 0x7fbb640c1ce0 2023-03-03T19:20:08.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.015+0000 7fbb5a7fc700 1 -- 172.21.15.7:0/3051659649 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbb5c019030 con 0x7fbb640c1ce0 2023-03-03T19:20:08.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.016+0000 7fbb5a7fc700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb48068320 0x7fbb4806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:08.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.016+0000 7fbb5a7fc700 1 -- 172.21.15.7:0/3051659649 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbb5c0859c0 con 0x7fbb640c1ce0 2023-03-03T19:20:08.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.016+0000 7fbb69918700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb48068320 0x7fbb4806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:08.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.017+0000 7fbb69918700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb48068320 0x7fbb4806a7d0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7fbb54006fd0 tx=0x7fbb54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:20:08.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.020+0000 7fbb5a7fc700 1 -- 172.21.15.7:0/3051659649 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbb5c04b5e0 con 0x7fbb640c1ce0 2023-03-03T19:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.262+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/3051659649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb640aa280 con 0x7fbb640c1ce0 2023-03-03T19:20:08.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.262+0000 7fbb5a7fc700 1 -- 172.21.15.7:0/3051659649 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbb5c051db0 con 0x7fbb640c1ce0 2023-03-03T19:20:08.263 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:20:08.263 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb48068320 msgr2=0x7fbb4806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb48068320 0x7fbb4806a7d0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7fbb54006fd0 tx=0x7fbb54008040 comp rx=0 tx=0).stop 2023-03-03T19:20:08.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/3051659649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb640c1ce0 msgr2=0x7fbb6414ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:08.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb640c1ce0 0x7fbb6414ae90 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00cad0 tx=0x7fbb5c00cde0 comp rx=0 tx=0).stop 2023-03-03T19:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/3051659649 shutdown_connections 2023-03-03T19:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbb48068320 0x7fbb4806a7d0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb640c1ce0 0x7fbb6414ae90 unknown :-1 s=CLOSED pgs=3460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbb64071e80 0x7fbb6415e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:08.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 --2- 172.21.15.7:0/3051659649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb64072250 0x7fbb64147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:08.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/3051659649 >> 172.21.15.7:0/3051659649 conn(0x7fbb640a94d0 msgr2=0x7fbb6414dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:20:08.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/3051659649 shutdown_connections 2023-03-03T19:20:08.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:08.264+0000 7fbb6bb7c700 1 -- 172.21.15.7:0/3051659649 wait complete. 2023-03-03T19:20:08.268 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:20:08.277 INFO:tasks.mds_thrash.fs.[a]:waiting for 90 secs before thrashing 2023-03-03T19:20:12.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:20:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.321+0000 7f2e36358700 1 Processor -- start 2023-03-03T19:20:12.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.321+0000 7f2e36358700 1 -- start start 2023-03-03T19:20:12.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e36358700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e3014f9d0 0x7f2e30156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:12.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e36358700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 0x7f2e30071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:12.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e36358700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2e300abe20 0x7f2e301529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e36358700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300c2f30 con 0x7f2e3014f9d0 2023-03-03T19:20:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e36358700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300c31d0 con 0x7f2e300abe20 2023-03-03T19:20:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e36358700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e301524a0 con 0x7f2e3014fda0 2023-03-03T19:20:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e3014f9d0 0x7f2e30156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2e3014f9d0 0x7f2e30156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:40994/0 (socket says 172.21.15.7:40994) 2023-03-03T19:20:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2ffff700 1 -- 172.21.15.7:0/45450335 learned_addr learned my addr 172.21.15.7:0/45450335 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:20:12.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e348f5700 1 --2- 172.21.15.7:0/45450335 >> v2:172.21.15.143:3300/0 conn(0x7f2e300abe20 0x7f2e301529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2f7fe700 1 --2- 172.21.15.7:0/45450335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 0x7f2e30071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2ffff700 1 -- 172.21.15.7:0/45450335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 msgr2=0x7f2e30071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:20:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2ffff700 1 --2- 172.21.15.7:0/45450335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 0x7f2e30071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2ffff700 1 -- 172.21.15.7:0/45450335 >> v2:172.21.15.143:3300/0 conn(0x7f2e300abe20 msgr2=0x7f2e301529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:20:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2ffff700 1 --2- 172.21.15.7:0/45450335 >> v2:172.21.15.143:3300/0 conn(0x7f2e300abe20 0x7f2e301529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2ffff700 1 -- 172.21.15.7:0/45450335 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e300c34b0 con 0x7f2e3014f9d0 2023-03-03T19:20:12.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2f7fe700 1 --2- 172.21.15.7:0/45450335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 0x7f2e30071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:20:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2ffff700 1 --2- 172.21.15.7:0/45450335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 0x7f2e30156430 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f2e300c2d20 tx=0x7f2e20009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6448a0ec2796415 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:20:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2effd700 1 -- 172.21.15.7:0/45450335 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2000b870 con 0x7f2e3014f9d0 2023-03-03T19:20:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.322+0000 7f2e2effd700 1 -- 172.21.15.7:0/45450335 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2000b9d0 con 0x7f2e3014f9d0 2023-03-03T19:20:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.323+0000 7f2e2effd700 1 -- 172.21.15.7:0/45450335 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2000bd60 con 0x7f2e3014f9d0 2023-03-03T19:20:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.323+0000 7f2e36358700 1 -- 172.21.15.7:0/45450335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 msgr2=0x7f2e30156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.323+0000 7f2e36358700 1 --2- 172.21.15.7:0/45450335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 0x7f2e30156430 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f2e300c2d20 tx=0x7f2e20009bd0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.323+0000 7f2e36358700 1 -- 172.21.15.7:0/45450335 shutdown_connections 2023-03-03T19:20:12.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.323+0000 7f2e36358700 1 --2- 172.21.15.7:0/45450335 >> v2:172.21.15.143:3300/0 conn(0x7f2e300abe20 0x7f2e301529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.323+0000 7f2e36358700 1 --2- 172.21.15.7:0/45450335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 0x7f2e30156430 unknown :-1 s=CLOSED pgs=3461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.323+0000 7f2e36358700 1 --2- 172.21.15.7:0/45450335 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 0x7f2e30071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.323+0000 7f2e36358700 1 -- 172.21.15.7:0/45450335 >> 172.21.15.7:0/45450335 conn(0x7f2e300c8900 msgr2=0x7f2e300c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:20:12.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.323+0000 7f2e36358700 1 -- 172.21.15.7:0/45450335 shutdown_connections 2023-03-03T19:20:12.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.323+0000 7f2e36358700 1 -- 172.21.15.7:0/45450335 wait complete. 2023-03-03T19:20:12.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.324+0000 7f2e36358700 1 Processor -- start 2023-03-03T19:20:12.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e36358700 1 -- start start 2023-03-03T19:20:12.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e36358700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e300abe20 0x7f2e300c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:12.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e36358700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 0x7f2e301dde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:12.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e36358700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 0x7f2e301de380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:12.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e36358700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e301de8c0 con 0x7f2e3014f9d0 2023-03-03T19:20:12.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e36358700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e301e97f0 con 0x7f2e300abe20 2023-03-03T19:20:12.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e36358700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e301e9ad0 con 0x7f2e3014fda0 2023-03-03T19:20:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e2f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 0x7f2e301dde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e2f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 0x7f2e301dde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41002/0 (socket says 172.21.15.7:41002) 2023-03-03T19:20:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e2f7fe700 1 -- 172.21.15.7:0/571105826 learned_addr learned my addr 172.21.15.7:0/571105826 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:20:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e2ffff700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e300abe20 0x7f2e300c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.325+0000 7f2e348f5700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 0x7f2e301de380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e2f7fe700 1 -- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 msgr2=0x7f2e301de380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e2f7fe700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 0x7f2e301de380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e2f7fe700 1 -- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e300abe20 msgr2=0x7f2e300c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e2f7fe700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e300abe20 0x7f2e300c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e2f7fe700 1 -- 172.21.15.7:0/571105826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e20009550 con 0x7f2e3014f9d0 2023-03-03T19:20:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e2f7fe700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 0x7f2e301dde40 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400c7f0 tx=0x7f2e2400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:20:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e2cff9700 1 -- 172.21.15.7:0/571105826 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24011800 con 0x7f2e3014f9d0 2023-03-03T19:20:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e2cff9700 1 -- 172.21.15.7:0/571105826 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e24011960 con 0x7f2e3014f9d0 2023-03-03T19:20:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e36358700 1 -- 172.21.15.7:0/571105826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e300ac350 con 0x7f2e3014f9d0 2023-03-03T19:20:12.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e2cff9700 1 -- 172.21.15.7:0/571105826 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24011cf0 con 0x7f2e3014f9d0 2023-03-03T19:20:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.326+0000 7f2e36358700 1 -- 172.21.15.7:0/571105826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e300ab630 con 0x7f2e3014f9d0 2023-03-03T19:20:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.327+0000 7f2e36358700 1 -- 172.21.15.7:0/571105826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e300c3e10 con 0x7f2e3014f9d0 2023-03-03T19:20:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.328+0000 7f2e2cff9700 1 -- 172.21.15.7:0/571105826 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2e24021070 con 0x7f2e3014f9d0 2023-03-03T19:20:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.328+0000 7f2e2cff9700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e1c0682f0 0x7f2e1c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:20:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.328+0000 7f2e2cff9700 1 -- 172.21.15.7:0/571105826 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2e24085f30 con 0x7f2e3014f9d0 2023-03-03T19:20:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.328+0000 7f2e2ffff700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e1c0682f0 0x7f2e1c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:20:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.329+0000 7f2e2ffff700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e1c0682f0 0x7f2e1c06a7a0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f2e20000c00 tx=0x7f2e20008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:20:12.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.332+0000 7f2e2cff9700 1 -- 172.21.15.7:0/571105826 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2e240521f0 con 0x7f2e3014f9d0 2023-03-03T19:20:12.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.576+0000 7f2e36358700 1 -- 172.21.15.7:0/571105826 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e30074bb0 con 0x7f2e3014f9d0 2023-03-03T19:20:12.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.576+0000 7f2e2cff9700 1 -- 172.21.15.7:0/571105826 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2e24055810 con 0x7f2e3014f9d0 2023-03-03T19:20:12.577 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:20:12.577 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:20:12.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 -- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e1c0682f0 msgr2=0x7f2e1c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:12.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e1c0682f0 0x7f2e1c06a7a0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f2e20000c00 tx=0x7f2e20008030 comp rx=0 tx=0).stop 2023-03-03T19:20:12.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 -- 172.21.15.7:0/571105826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 msgr2=0x7f2e301dde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:20:12.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 0x7f2e301dde40 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f2e2400c7f0 tx=0x7f2e2400cb00 comp rx=0 tx=0).stop 2023-03-03T19:20:12.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 -- 172.21.15.7:0/571105826 shutdown_connections 2023-03-03T19:20:12.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2e3014fda0 0x7f2e301de380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e300abe20 0x7f2e300c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2e1c0682f0 0x7f2e1c06a7a0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 --2- 172.21.15.7:0/571105826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e3014f9d0 0x7f2e301dde40 unknown :-1 s=CLOSED pgs=3462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:20:12.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 -- 172.21.15.7:0/571105826 >> 172.21.15.7:0/571105826 conn(0x7f2e300c8900 msgr2=0x7f2e30155660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:20:12.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 -- 172.21.15.7:0/571105826 shutdown_connections 2023-03-03T19:20:12.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:20:12.578+0000 7f2e36358700 1 -- 172.21.15.7:0/571105826 wait complete. 2023-03-03T19:20:12.580 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:20:12.591 INFO:tasks.mds_thrash.fs.[b]:waiting for 51 secs before thrashing 2023-03-03T19:20:17.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:20:17.035 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:20:47.097 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:20:47.101 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:21:03.593 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:21:03.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.733+0000 7f613523d700 1 Processor -- start 2023-03-03T19:21:03.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.733+0000 7f613523d700 1 -- start start 2023-03-03T19:21:03.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f613523d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f61300c20d0 0x7f61301562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:03.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f613523d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f61300c27c0 0x7f61300736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:03.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f613523d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6130155e60 con 0x7f613014f860 2023-03-03T19:21:03.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f613523d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f613006d350 con 0x7f61300c27c0 2023-03-03T19:21:03.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f613523d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61301558e0 con 0x7f61300c20d0 2023-03-03T19:21:03.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f612f59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f61300c27c0 0x7f61300736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:03.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f612ed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f613014f860 0x7f6130152d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35028/0 (socket says 172.21.15.7:35028) 2023-03-03T19:21:03.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f612ed9d700 1 -- 172.21.15.7:0/3644855502 learned_addr learned my addr 172.21.15.7:0/3644855502 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:21:03.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f612e59c700 1 --2- 172.21.15.7:0/3644855502 >> v2:172.21.15.143:3300/0 conn(0x7f61300c20d0 0x7f61301562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:03.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f612f59e700 1 -- 172.21.15.7:0/3644855502 >> v2:172.21.15.143:3300/0 conn(0x7f61300c20d0 msgr2=0x7f61301562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:21:03.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f612f59e700 1 --2- 172.21.15.7:0/3644855502 >> v2:172.21.15.143:3300/0 conn(0x7f61300c20d0 0x7f61301562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:03.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f612f59e700 1 -- 172.21.15.7:0/3644855502 >> v1:172.21.15.7:6789/0 conn(0x7f613014f860 legacy=0x7f6130152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:03.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f612f59e700 1 -- 172.21.15.7:0/3644855502 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613006cd90 con 0x7f61300c27c0 2023-03-03T19:21:03.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f612e59c700 1 --2- 172.21.15.7:0/3644855502 >> v2:172.21.15.143:3300/0 conn(0x7f61300c20d0 0x7f61301562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:21:03.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.734+0000 7f612f59e700 1 --2- 172.21.15.7:0/3644855502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 0x7f61300736f0 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f6130072520 tx=0x7f612800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f557f90aefe22a6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:21:03.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f612dd9b700 1 -- 172.21.15.7:0/3644855502 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612800e070 con 0x7f61300c27c0 2023-03-03T19:21:03.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f612dd9b700 1 -- 172.21.15.7:0/3644855502 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612800bbd0 con 0x7f61300c27c0 2023-03-03T19:21:03.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f612dd9b700 1 -- 172.21.15.7:0/3644855502 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6128012430 con 0x7f61300c27c0 2023-03-03T19:21:03.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f613523d700 1 -- 172.21.15.7:0/3644855502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 msgr2=0x7f61300736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:03.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f613523d700 1 --2- 172.21.15.7:0/3644855502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 0x7f61300736f0 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f6130072520 tx=0x7f612800b330 comp rx=0 tx=0).stop 2023-03-03T19:21:03.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f613523d700 1 -- 172.21.15.7:0/3644855502 shutdown_connections 2023-03-03T19:21:03.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f613523d700 1 --2- 172.21.15.7:0/3644855502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 0x7f61300736f0 unknown :-1 s=CLOSED pgs=3464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:03.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f613523d700 1 --2- 172.21.15.7:0/3644855502 >> v2:172.21.15.143:3300/0 conn(0x7f61300c20d0 0x7f61301562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:03.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f613523d700 1 -- 172.21.15.7:0/3644855502 >> 172.21.15.7:0/3644855502 conn(0x7f61300a94d0 msgr2=0x7f61300a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:21:03.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f613523d700 1 -- 172.21.15.7:0/3644855502 shutdown_connections 2023-03-03T19:21:03.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.735+0000 7f613523d700 1 -- 172.21.15.7:0/3644855502 wait complete. 2023-03-03T19:21:03.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.737+0000 7f613523d700 1 Processor -- start 2023-03-03T19:21:03.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.737+0000 7f613523d700 1 -- start start 2023-03-03T19:21:03.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.737+0000 7f613523d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61300c20d0 0x7f6130066220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:03.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.737+0000 7f613523d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 0x7f6130149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:03.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.737+0000 7f613523d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f613014f860 0x7f613014a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:03.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.737+0000 7f613523d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613014aba0 con 0x7f61300c27c0 2023-03-03T19:21:03.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.737+0000 7f613523d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613014a8c0 con 0x7f613014f860 2023-03-03T19:21:03.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.737+0000 7f613523d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61300bf110 con 0x7f61300c20d0 2023-03-03T19:21:03.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 0x7f6130149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:03.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 0x7f6130149d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41056/0 (socket says 172.21.15.7:41056) 2023-03-03T19:21:03.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612e59c700 1 -- 172.21.15.7:0/4028219141 learned_addr learned my addr 172.21.15.7:0/4028219141 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:21:03.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612f59e700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f613014f860 0x7f613014a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:03.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612ed9d700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61300c20d0 0x7f6130066220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:03.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612e59c700 1 -- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61300c20d0 msgr2=0x7f6130066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:03.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612e59c700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61300c20d0 0x7f6130066220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:03.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612e59c700 1 -- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f613014f860 msgr2=0x7f613014a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:03.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612e59c700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f613014f860 0x7f613014a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:03.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612e59c700 1 -- 172.21.15.7:0/4028219141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6128009a70 con 0x7f61300c27c0 2023-03-03T19:21:03.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612f59e700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f613014f860 0x7f613014a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:21:03.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f612e59c700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 0x7f6130149d30 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f612000e810 tx=0x7f612000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:21:03.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f61177fe700 1 -- 172.21.15.7:0/4028219141 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120004230 con 0x7f61300c27c0 2023-03-03T19:21:03.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f61177fe700 1 -- 172.21.15.7:0/4028219141 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6120004390 con 0x7f61300c27c0 2023-03-03T19:21:03.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f613523d700 1 -- 172.21.15.7:0/4028219141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6130155af0 con 0x7f61300c27c0 2023-03-03T19:21:03.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f61177fe700 1 -- 172.21.15.7:0/4028219141 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61200036a0 con 0x7f61300c27c0 2023-03-03T19:21:03.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.738+0000 7f613523d700 1 -- 172.21.15.7:0/4028219141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613006d070 con 0x7f61300c27c0 2023-03-03T19:21:03.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.739+0000 7f613523d700 1 -- 172.21.15.7:0/4028219141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613014fc30 con 0x7f61300c27c0 2023-03-03T19:21:03.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.740+0000 7f61177fe700 1 -- 172.21.15.7:0/4028219141 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6120022070 con 0x7f61300c27c0 2023-03-03T19:21:03.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.740+0000 7f61177fe700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f611c0682f0 0x7f611c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:03.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.740+0000 7f61177fe700 1 -- 172.21.15.7:0/4028219141 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6120085c80 con 0x7f61300c27c0 2023-03-03T19:21:03.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.740+0000 7f612ed9d700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f611c0682f0 0x7f611c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:03.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.741+0000 7f612ed9d700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f611c0682f0 0x7f611c06a7a0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f6124005910 tx=0x7f6124009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:21:03.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.744+0000 7f61177fe700 1 -- 172.21.15.7:0/4028219141 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6120051f40 con 0x7f61300c27c0 2023-03-03T19:21:03.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.984+0000 7f613523d700 1 -- 172.21.15.7:0/4028219141 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6130074bb0 con 0x7f61300c27c0 2023-03-03T19:21:03.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.984+0000 7f61177fe700 1 -- 172.21.15.7:0/4028219141 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6120055560 con 0x7f61300c27c0 2023-03-03T19:21:03.985 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:21:03.985 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:21:03.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 -- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f611c0682f0 msgr2=0x7f611c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:03.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f611c0682f0 0x7f611c06a7a0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f6124005910 tx=0x7f6124009040 comp rx=0 tx=0).stop 2023-03-03T19:21:03.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 -- 172.21.15.7:0/4028219141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 msgr2=0x7f6130149d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:03.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 0x7f6130149d30 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f612000e810 tx=0x7f612000ebd0 comp rx=0 tx=0).stop 2023-03-03T19:21:03.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 -- 172.21.15.7:0/4028219141 shutdown_connections 2023-03-03T19:21:03.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f613014f860 0x7f613014a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:03.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f61300c20d0 0x7f6130066220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:03.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f611c0682f0 0x7f611c06a7a0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:03.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 --2- 172.21.15.7:0/4028219141 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61300c27c0 0x7f6130149d30 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:03.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 -- 172.21.15.7:0/4028219141 >> 172.21.15.7:0/4028219141 conn(0x7f61300a94d0 msgr2=0x7f61300c45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:21:03.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 -- 172.21.15.7:0/4028219141 shutdown_connections 2023-03-03T19:21:03.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:03.986+0000 7f613523d700 1 -- 172.21.15.7:0/4028219141 wait complete. 2023-03-03T19:21:03.988 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:21:03.998 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T19:21:17.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:21:17.171 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:21:38.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:21:38.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.417+0000 7f075dd14700 1 Processor -- start 2023-03-03T19:21:38.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.417+0000 7f075dd14700 1 -- start start 2023-03-03T19:21:38.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.417+0000 7f075dd14700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075814b460 0x7f075806aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:38.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.417+0000 7f075dd14700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0758072470 0x7f075806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:38.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.417+0000 7f075dd14700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0758070360 con 0x7f0758071990 2023-03-03T19:21:38.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.417+0000 7f075dd14700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07581521a0 con 0x7f0758072470 2023-03-03T19:21:38.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.417+0000 7f075dd14700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075806a5e0 con 0x7f075814b460 2023-03-03T19:21:38.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07577fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0758071990 0x7f0758071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35082/0 (socket says 172.21.15.7:35082) 2023-03-03T19:21:38.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07577fe700 1 -- 172.21.15.7:0/3456868968 learned_addr learned my addr 172.21.15.7:0/3456868968 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:21:38.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f0756ffd700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075814b460 0x7f075806aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:38.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f0757fff700 1 --2- 172.21.15.7:0/3456868968 >> v2:172.21.15.143:3300/0 conn(0x7f0758072470 0x7f075806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:38.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028384232 0 0) 0x7f0758070360 con 0x7f0758071990 2023-03-03T19:21:38.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07440035b0 con 0x7f0758071990 2023-03-03T19:21:38.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1382202124 0 0) 0x7f07440035b0 con 0x7f0758071990 2023-03-03T19:21:38.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075814b460 msgr2=0x7f075806aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:21:38.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075814b460 0x7f075806aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:38.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 >> v2:172.21.15.143:3300/0 conn(0x7f0758072470 msgr2=0x7f075806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:21:38.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 --2- 172.21.15.7:0/3456868968 >> v2:172.21.15.143:3300/0 conn(0x7f0758072470 0x7f075806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:38.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07580c38b0 con 0x7f0758071990 2023-03-03T19:21:38.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f0740001670 con 0x7f0758071990 2023-03-03T19:21:38.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f0757fff700 1 --2- 172.21.15.7:0/3456868968 >> v2:172.21.15.143:3300/0 conn(0x7f0758072470 0x7f075806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:21:38.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f0756ffd700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f075814b460 0x7f075806aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:21:38.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 >> v1:172.21.15.7:6789/0 conn(0x7f0758071990 legacy=0x7f0758071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:38.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0744003a70 0x7f0744003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:38.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 0x7f0744004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:38.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0744005200 0x7f074400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:38.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07581521a0 con 0x7f0744004850 2023-03-03T19:21:38.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075806a5e0 con 0x7f0744003a70 2023-03-03T19:21:38.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074400bf50 con 0x7f0744005200 2023-03-03T19:21:38.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f0756ffd700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 0x7f0744004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:38.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f07577fe700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0744003a70 0x7f0744003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:38.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.418+0000 7f0757fff700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0744005200 0x7f074400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:38.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f0756ffd700 1 -- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0744005200 msgr2=0x7f074400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:21:38.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f0756ffd700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0744005200 0x7f074400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:38.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f0756ffd700 1 -- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0744003a70 msgr2=0x7f0744003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:21:38.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f0756ffd700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0744003a70 0x7f0744003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:38.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f0756ffd700 1 -- 172.21.15.7:0/3456868968 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07480094a0 con 0x7f0744004850 2023-03-03T19:21:38.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f0756ffd700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 0x7f0744004cc0 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f0748009d60 tx=0x7f0744005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d80515dd0bb5125 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:21:38.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074800e9a0 con 0x7f0744004850 2023-03-03T19:21:38.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074800eb00 con 0x7f0744004850 2023-03-03T19:21:38.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f07567fc700 1 -- 172.21.15.7:0/3456868968 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074800ee90 con 0x7f0744004850 2023-03-03T19:21:38.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f075dd14700 1 -- 172.21.15.7:0/3456868968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 msgr2=0x7f0744004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:38.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.419+0000 7f075dd14700 1 --2- 172.21.15.7:0/3456868968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 0x7f0744004cc0 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f0748009d60 tx=0x7f0744005880 comp rx=0 tx=0).stop 2023-03-03T19:21:38.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.420+0000 7f0757fff700 1 -- 172.21.15.7:0/3456868968 reap_dead start 2023-03-03T19:21:38.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.420+0000 7f075dd14700 1 -- 172.21.15.7:0/3456868968 shutdown_connections 2023-03-03T19:21:38.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.420+0000 7f075dd14700 1 -- 172.21.15.7:0/3456868968 >> 172.21.15.7:0/3456868968 conn(0x7f07580a94d0 msgr2=0x7f07580a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:21:38.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.420+0000 7f075dd14700 1 -- 172.21.15.7:0/3456868968 shutdown_connections 2023-03-03T19:21:38.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.420+0000 7f075dd14700 1 -- 172.21.15.7:0/3456868968 wait complete. 2023-03-03T19:21:38.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.421+0000 7f075dd14700 1 Processor -- start 2023-03-03T19:21:38.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.422+0000 7f075dd14700 1 -- start start 2023-03-03T19:21:38.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.422+0000 7f075dd14700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 0x7f07580c1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:38.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.422+0000 7f075dd14700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07580ac890 0x7f07581eb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:38.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.422+0000 7f075dd14700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07581eb9a0 0x7f0758142620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:38.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.422+0000 7f075dd14700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07580c0a40 con 0x7f0744004850 2023-03-03T19:21:38.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.422+0000 7f075dd14700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0758155350 con 0x7f07581eb9a0 2023-03-03T19:21:38.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.422+0000 7f075dd14700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07581555a0 con 0x7f07580ac890 2023-03-03T19:21:38.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.422+0000 7f07577fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 0x7f07580c1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:38.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f07577fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 0x7f07580c1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41114/0 (socket says 172.21.15.7:41114) 2023-03-03T19:21:38.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f07577fe700 1 -- 172.21.15.7:0/2827756639 learned_addr learned my addr 172.21.15.7:0/2827756639 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:21:38.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f0757fff700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07581eb9a0 0x7f0758142620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:38.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f0756ffd700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07580ac890 0x7f07581eb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:38.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f07577fe700 1 -- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07580ac890 msgr2=0x7f07581eb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:38.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f07577fe700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07580ac890 0x7f07581eb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:38.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f07577fe700 1 -- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07581eb9a0 msgr2=0x7f0758142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:38.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f07577fe700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07581eb9a0 0x7f0758142620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:38.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f07577fe700 1 -- 172.21.15.7:0/2827756639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07580c3b50 con 0x7f0744004850 2023-03-03T19:21:38.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f0757fff700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07581eb9a0 0x7f0758142620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:21:38.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f07577fe700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 0x7f07580c1450 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7f0740007310 tx=0x7f0740007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:21:38.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f074ffff700 1 -- 172.21.15.7:0/2827756639 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0740007ca0 con 0x7f0744004850 2023-03-03T19:21:38.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f074ffff700 1 -- 172.21.15.7:0/2827756639 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074000f660 con 0x7f0744004850 2023-03-03T19:21:38.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f075dd14700 1 -- 172.21.15.7:0/2827756639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075806a2b0 con 0x7f0744004850 2023-03-03T19:21:38.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f074ffff700 1 -- 172.21.15.7:0/2827756639 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074000f9f0 con 0x7f0744004850 2023-03-03T19:21:38.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.423+0000 7f075dd14700 1 -- 172.21.15.7:0/2827756639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075806b4f0 con 0x7f0744004850 2023-03-03T19:21:38.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.424+0000 7f075dd14700 1 -- 172.21.15.7:0/2827756639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0758154f20 con 0x7f0744004850 2023-03-03T19:21:38.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.425+0000 7f074ffff700 1 -- 172.21.15.7:0/2827756639 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f074000fb50 con 0x7f0744004850 2023-03-03T19:21:38.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.425+0000 7f074ffff700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 0x7f073806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:21:38.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.425+0000 7f074ffff700 1 -- 172.21.15.7:0/2827756639 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0740086280 con 0x7f0744004850 2023-03-03T19:21:38.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.425+0000 7f0756ffd700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 0x7f073806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:21:38.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.426+0000 7f0756ffd700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 0x7f073806a7a0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f07480090b0 tx=0x7f0748008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:21:38.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.430+0000 7f074ffff700 1 -- 172.21.15.7:0/2827756639 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0758154f20 con 0x7f0744004850 2023-03-03T19:21:38.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.666+0000 7f075dd14700 1 -- 172.21.15.7:0/2827756639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0758154f20 con 0x7f0744004850 2023-03-03T19:21:38.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.666+0000 7f074ffff700 1 -- 172.21.15.7:0/2827756639 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0758154f20 con 0x7f0744004850 2023-03-03T19:21:38.667 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:21:38.667 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:21:38.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 -- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 msgr2=0x7f073806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 0x7f073806a7a0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f07480090b0 tx=0x7f0748008030 comp rx=0 tx=0).stop 2023-03-03T19:21:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 -- 172.21.15.7:0/2827756639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 msgr2=0x7f07580c1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:21:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 0x7f07580c1450 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7f0740007310 tx=0x7f0740007620 comp rx=0 tx=0).stop 2023-03-03T19:21:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 -- 172.21.15.7:0/2827756639 shutdown_connections 2023-03-03T19:21:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07581eb9a0 0x7f0758142620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f07380682f0 0x7f073806a7a0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:38.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0744004850 0x7f07580c1450 unknown :-1 s=CLOSED pgs=3468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:38.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 --2- 172.21.15.7:0/2827756639 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f07580ac890 0x7f07581eb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:21:38.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 -- 172.21.15.7:0/2827756639 >> 172.21.15.7:0/2827756639 conn(0x7f07580a94d0 msgr2=0x7f075806aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:21:38.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 -- 172.21.15.7:0/2827756639 shutdown_connections 2023-03-03T19:21:38.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:21:38.668+0000 7f075dd14700 1 -- 172.21.15.7:0/2827756639 wait complete. 2023-03-03T19:21:38.670 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:21:38.681 INFO:tasks.mds_thrash.fs.[a]:waiting for 51 secs before thrashing 2023-03-03T19:21:47.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:21:47.250 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:22:06.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:22:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.137+0000 7f9d90bea700 1 Processor -- start 2023-03-03T19:22:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.137+0000 7f9d90bea700 1 -- start start 2023-03-03T19:22:06.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d90bea700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9d8c14f9d0 0x7f9d8c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:06.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d90bea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 0x7f9d8c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:06.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d90bea700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8c0c2f30 con 0x7f9d8c14fda0 2023-03-03T19:22:06.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d90bea700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8c0c31d0 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d90bea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c1524a0 con 0x7f9d8c0abe20 2023-03-03T19:22:06.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8ad9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 0x7f9d8c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:06.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8a59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9d8c14f9d0 0x7f9d8c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:06.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8ad9d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 0x7f9d8c1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:49224/0 (socket says 172.21.15.7:49224) 2023-03-03T19:22:06.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d89d9b700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f9d8c14fda0 0x7f9d8c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:38650/0 (socket says 172.21.15.7:38650) 2023-03-03T19:22:06.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8ad9d700 1 -- 172.21.15.7:0/3434524836 learned_addr learned my addr 172.21.15.7:0/3434524836 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:22:06.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8959a700 1 -- 172.21.15.7:0/3434524836 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69514755 0 0) 0x7f9d8c0c2f30 con 0x7f9d8c14fda0 2023-03-03T19:22:06.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8959a700 1 -- 172.21.15.7:0/3434524836 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d780035b0 con 0x7f9d8c14fda0 2023-03-03T19:22:06.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8a59c700 1 -- 172.21.15.7:0/3434524836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 msgr2=0x7f9d8c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:22:06.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8a59c700 1 --2- 172.21.15.7:0/3434524836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 0x7f9d8c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:06.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8a59c700 1 -- 172.21.15.7:0/3434524836 >> v1:172.21.15.143:6789/0 conn(0x7f9d8c14fda0 legacy=0x7f9d8c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:06.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8a59c700 1 -- 172.21.15.7:0/3434524836 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8c0c34b0 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.138+0000 7f9d8ad9d700 1 --2- 172.21.15.7:0/3434524836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 0x7f9d8c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:22:06.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.139+0000 7f9d8a59c700 1 --2- 172.21.15.7:0/3434524836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d8c14f9d0 0x7f9d8c156430 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009d90 tx=0x7f9d8400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c2f6f91ebd061f4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:06.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.139+0000 7f9d8959a700 1 -- 172.21.15.7:0/3434524836 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8400e070 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.139+0000 7f9d8959a700 1 -- 172.21.15.7:0/3434524836 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d8400bb20 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.139+0000 7f9d8959a700 1 -- 172.21.15.7:0/3434524836 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84012430 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.139+0000 7f9d90bea700 1 -- 172.21.15.7:0/3434524836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d8c14f9d0 msgr2=0x7f9d8c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.139+0000 7f9d90bea700 1 --2- 172.21.15.7:0/3434524836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d8c14f9d0 0x7f9d8c156430 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009d90 tx=0x7f9d8400b330 comp rx=0 tx=0).stop 2023-03-03T19:22:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.139+0000 7f9d90bea700 1 -- 172.21.15.7:0/3434524836 shutdown_connections 2023-03-03T19:22:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.139+0000 7f9d90bea700 1 --2- 172.21.15.7:0/3434524836 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 0x7f9d8c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.139+0000 7f9d90bea700 1 --2- 172.21.15.7:0/3434524836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d8c14f9d0 0x7f9d8c156430 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.140+0000 7f9d90bea700 1 -- 172.21.15.7:0/3434524836 >> 172.21.15.7:0/3434524836 conn(0x7f9d8c0c8900 msgr2=0x7f9d8c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:22:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.140+0000 7f9d90bea700 1 -- 172.21.15.7:0/3434524836 shutdown_connections 2023-03-03T19:22:06.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.140+0000 7f9d90bea700 1 -- 172.21.15.7:0/3434524836 wait complete. 2023-03-03T19:22:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.141+0000 7f9d90bea700 1 Processor -- start 2023-03-03T19:22:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.141+0000 7f9d90bea700 1 -- start start 2023-03-03T19:22:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d90bea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 0x7f9d8c06d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d90bea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d8c14f9d0 0x7f9d8c154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d90bea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d8c14fda0 0x7f9d8c1ea9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d90bea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c14bb20 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d90bea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c1e82d0 con 0x7f9d8c14fda0 2023-03-03T19:22:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d90bea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c1e7f80 con 0x7f9d8c0abe20 2023-03-03T19:22:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d89d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d8c14f9d0 0x7f9d8c154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d89d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d8c14f9d0 0x7f9d8c154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41156/0 (socket says 172.21.15.7:41156) 2023-03-03T19:22:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d89d9b700 1 -- 172.21.15.7:0/1533623623 learned_addr learned my addr 172.21.15.7:0/1533623623 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:22:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d8ad9d700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d8c14fda0 0x7f9d8c1ea9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d8a59c700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 0x7f9d8c06d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d89d9b700 1 -- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 msgr2=0x7f9d8c06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d89d9b700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 0x7f9d8c06d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d89d9b700 1 -- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d8c14fda0 msgr2=0x7f9d8c1ea9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d89d9b700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d8c14fda0 0x7f9d8c1ea9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d89d9b700 1 -- 172.21.15.7:0/1533623623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c009550 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d89d9b700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d8c14f9d0 0x7f9d8c154750 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400c550 tx=0x7f9d7400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.142+0000 7f9d82ffd700 1 -- 172.21.15.7:0/1533623623 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74005cb0 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.143+0000 7f9d82ffd700 1 -- 172.21.15.7:0/1533623623 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d74005e10 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.143+0000 7f9d90bea700 1 -- 172.21.15.7:0/1533623623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d84009a70 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.143+0000 7f9d82ffd700 1 -- 172.21.15.7:0/1533623623 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d740072f0 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.143+0000 7f9d90bea700 1 -- 172.21.15.7:0/1533623623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c0ac350 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.144+0000 7f9d90bea700 1 -- 172.21.15.7:0/1533623623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c0ab630 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.144+0000 7f9d82ffd700 1 -- 172.21.15.7:0/1533623623 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9d74019030 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.144+0000 7f9d82ffd700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d6c0682f0 0x7f9d6c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:06.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.144+0000 7f9d82ffd700 1 -- 172.21.15.7:0/1533623623 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9d74085460 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.145+0000 7f9d8a59c700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d6c0682f0 0x7f9d6c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.145+0000 7f9d8a59c700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d6c0682f0 0x7f9d6c06a7a0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f9d84014040 tx=0x7f9d84011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.149+0000 7f9d82ffd700 1 -- 172.21.15.7:0/1533623623 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9d74051800 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.386+0000 7f9d90bea700 1 -- 172.21.15.7:0/1533623623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d8c071fa0 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.387+0000 7f9d82ffd700 1 -- 172.21.15.7:0/1533623623 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9d74054e20 con 0x7f9d8c14f9d0 2023-03-03T19:22:06.389 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:22:06.389 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:22:06.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 -- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d6c0682f0 msgr2=0x7f9d6c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:06.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d6c0682f0 0x7f9d6c06a7a0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f9d84014040 tx=0x7f9d84011000 comp rx=0 tx=0).stop 2023-03-03T19:22:06.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 -- 172.21.15.7:0/1533623623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d8c14f9d0 msgr2=0x7f9d8c154750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:06.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d8c14f9d0 0x7f9d8c154750 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400c550 tx=0x7f9d7400c860 comp rx=0 tx=0).stop 2023-03-03T19:22:06.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 -- 172.21.15.7:0/1533623623 shutdown_connections 2023-03-03T19:22:06.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d8c14fda0 0x7f9d8c1ea9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:06.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f9d8c0abe20 0x7f9d8c06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:06.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f9d6c0682f0 0x7f9d6c06a7a0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:06.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 --2- 172.21.15.7:0/1533623623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d8c14f9d0 0x7f9d8c154750 unknown :-1 s=CLOSED pgs=3469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:06.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 -- 172.21.15.7:0/1533623623 >> 172.21.15.7:0/1533623623 conn(0x7f9d8c0c8900 msgr2=0x7f9d8c151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:22:06.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 -- 172.21.15.7:0/1533623623 shutdown_connections 2023-03-03T19:22:06.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:06.389+0000 7f9d90bea700 1 -- 172.21.15.7:0/1533623623 wait complete. 2023-03-03T19:22:06.392 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:22:06.401 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T19:22:17.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:22:17.322 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:22:17.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:22:17.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.539+0000 7fe12696b700 1 Processor -- start 2023-03-03T19:22:17.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.539+0000 7fe12696b700 1 -- start start 2023-03-03T19:22:17.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe12696b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe12014fda0 0x7fe120156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:17.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe12696b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe1200abe20 0x7fe1201529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:17.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe12696b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1200c2f30 con 0x7fe12014f9d0 2023-03-03T19:22:17.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe12696b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1200c31d0 con 0x7fe12014fda0 2023-03-03T19:22:17.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe12696b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1201524a0 con 0x7fe1200abe20 2023-03-03T19:22:17.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe12014fda0 0x7fe120156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe12014fda0 0x7fe120156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41192/0 (socket says 172.21.15.7:41192) 2023-03-03T19:22:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11f7fe700 1 -- 172.21.15.7:0/662131062 learned_addr learned my addr 172.21.15.7:0/662131062 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:22:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe124f08700 1 --2- 172.21.15.7:0/662131062 >> v2:172.21.15.143:3300/0 conn(0x7fe1200abe20 0x7fe1201529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11f7fe700 1 -- 172.21.15.7:0/662131062 >> v2:172.21.15.143:3300/0 conn(0x7fe1200abe20 msgr2=0x7fe1201529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:22:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11f7fe700 1 --2- 172.21.15.7:0/662131062 >> v2:172.21.15.143:3300/0 conn(0x7fe1200abe20 0x7fe1201529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11f7fe700 1 -- 172.21.15.7:0/662131062 >> v1:172.21.15.143:6789/0 conn(0x7fe12014f9d0 legacy=0x7fe120071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:22:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11f7fe700 1 -- 172.21.15.7:0/662131062 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200c34b0 con 0x7fe12014fda0 2023-03-03T19:22:17.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11f7fe700 1 --2- 172.21.15.7:0/662131062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe120156430 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7fe120071230 tx=0x7fe114009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc19b462c6c03fe0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:17.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11effd700 1 -- 172.21.15.7:0/662131062 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114004230 con 0x7fe12014fda0 2023-03-03T19:22:17.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11effd700 1 -- 172.21.15.7:0/662131062 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe114004390 con 0x7fe12014fda0 2023-03-03T19:22:17.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.540+0000 7fe11effd700 1 -- 172.21.15.7:0/662131062 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140036a0 con 0x7fe12014fda0 2023-03-03T19:22:17.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.541+0000 7fe12696b700 1 -- 172.21.15.7:0/662131062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 msgr2=0x7fe120156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:17.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.541+0000 7fe12696b700 1 --2- 172.21.15.7:0/662131062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe120156430 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7fe120071230 tx=0x7fe114009bd0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.541+0000 7fe12696b700 1 -- 172.21.15.7:0/662131062 shutdown_connections 2023-03-03T19:22:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.541+0000 7fe12696b700 1 --2- 172.21.15.7:0/662131062 >> v2:172.21.15.143:3300/0 conn(0x7fe1200abe20 0x7fe1201529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.541+0000 7fe12696b700 1 --2- 172.21.15.7:0/662131062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe120156430 unknown :-1 s=CLOSED pgs=3470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.541+0000 7fe12696b700 1 -- 172.21.15.7:0/662131062 >> 172.21.15.7:0/662131062 conn(0x7fe1200c8900 msgr2=0x7fe1200c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:22:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.541+0000 7fe12696b700 1 -- 172.21.15.7:0/662131062 shutdown_connections 2023-03-03T19:22:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.541+0000 7fe12696b700 1 -- 172.21.15.7:0/662131062 wait complete. 2023-03-03T19:22:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.543+0000 7fe12696b700 1 Processor -- start 2023-03-03T19:22:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.543+0000 7fe12696b700 1 -- start start 2023-03-03T19:22:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe12696b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1200abe20 0x7fe12006f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe12696b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12014f9d0 0x7fe1201dc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe12696b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe12014b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe12696b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1201eb0d0 con 0x7fe12014fda0 2023-03-03T19:22:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe12696b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1201eb3b0 con 0x7fe12014f9d0 2023-03-03T19:22:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe12696b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1201eb690 con 0x7fe1200abe20 2023-03-03T19:22:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe124f08700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe12014b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe124f08700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe12014b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41200/0 (socket says 172.21.15.7:41200) 2023-03-03T19:22:17.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe124f08700 1 -- 172.21.15.7:0/1845493789 learned_addr learned my addr 172.21.15.7:0/1845493789 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:22:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe11f7fe700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12014f9d0 0x7fe1201dc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe11ffff700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1200abe20 0x7fe12006f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe124f08700 1 -- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1200abe20 msgr2=0x7fe12006f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe124f08700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1200abe20 0x7fe12006f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe124f08700 1 -- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12014f9d0 msgr2=0x7fe1201dc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe124f08700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12014f9d0 0x7fe1201dc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe124f08700 1 -- 172.21.15.7:0/1845493789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114009550 con 0x7fe12014fda0 2023-03-03T19:22:17.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe124f08700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe12014b6e0 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7fe118009fd0 tx=0x7fe11800e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.544+0000 7fe11cff9700 1 -- 172.21.15.7:0/1845493789 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11800ede0 con 0x7fe12014fda0 2023-03-03T19:22:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.545+0000 7fe11cff9700 1 -- 172.21.15.7:0/1845493789 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe118014420 con 0x7fe12014fda0 2023-03-03T19:22:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.545+0000 7fe12696b700 1 -- 172.21.15.7:0/1845493789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1200ac350 con 0x7fe12014fda0 2023-03-03T19:22:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.545+0000 7fe11cff9700 1 -- 172.21.15.7:0/1845493789 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1180147d0 con 0x7fe12014fda0 2023-03-03T19:22:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.545+0000 7fe12696b700 1 -- 172.21.15.7:0/1845493789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1200ab630 con 0x7fe12014fda0 2023-03-03T19:22:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.545+0000 7fe12696b700 1 -- 172.21.15.7:0/1845493789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe120071fa0 con 0x7fe12014fda0 2023-03-03T19:22:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.546+0000 7fe11cff9700 1 -- 172.21.15.7:0/1845493789 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe118012050 con 0x7fe12014fda0 2023-03-03T19:22:17.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.546+0000 7fe11cff9700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 0x7fe10c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.547+0000 7fe11cff9700 1 -- 172.21.15.7:0/1845493789 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe118086770 con 0x7fe12014fda0 2023-03-03T19:22:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.547+0000 7fe11ffff700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 0x7fe10c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.547+0000 7fe11ffff700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 0x7fe10c06a7a0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fe110004f30 tx=0x7fe110009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:17.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.551+0000 7fe11cff9700 1 -- 172.21.15.7:0/1845493789 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe118052b10 con 0x7fe12014fda0 2023-03-03T19:22:17.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.787+0000 7fe12696b700 1 -- 172.21.15.7:0/1845493789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1200abc00 con 0x7fe12014fda0 2023-03-03T19:22:17.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.788+0000 7fe11cff9700 1 -- 172.21.15.7:0/1845493789 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe118056130 con 0x7fe12014fda0 2023-03-03T19:22:17.788 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:22:17.789 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:22:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 -- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 msgr2=0x7fe10c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 0x7fe10c06a7a0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fe110004f30 tx=0x7fe110009040 comp rx=0 tx=0).stop 2023-03-03T19:22:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 -- 172.21.15.7:0/1845493789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 msgr2=0x7fe12014b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:17.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe12014b6e0 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7fe118009fd0 tx=0x7fe11800e3c0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 -- 172.21.15.7:0/1845493789 shutdown_connections 2023-03-03T19:22:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe10c0682f0 0x7fe10c06a7a0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe12014fda0 0x7fe12014b6e0 unknown :-1 s=CLOSED pgs=3471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe1200abe20 0x7fe12006f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 --2- 172.21.15.7:0/1845493789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe12014f9d0 0x7fe1201dc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 -- 172.21.15.7:0/1845493789 >> 172.21.15.7:0/1845493789 conn(0x7fe1200c8900 msgr2=0x7fe1201516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:22:17.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 -- 172.21.15.7:0/1845493789 shutdown_connections 2023-03-03T19:22:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:17.790+0000 7fe12696b700 1 -- 172.21.15.7:0/1845493789 wait complete. 2023-03-03T19:22:17.792 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:22:17.802 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before thrashing 2023-03-03T19:22:26.803 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:22:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.942+0000 7fca14929700 1 Processor -- start 2023-03-03T19:22:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.942+0000 7fca14929700 1 -- start start 2023-03-03T19:22:26.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca14929700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca100ac220 0x7fca100736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:26.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca14929700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca10073c30 0x7fca10152450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:26.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca14929700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca100ac900 con 0x7fca100ace30 2023-03-03T19:22:26.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca14929700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca10071e30 con 0x7fca100ac220 2023-03-03T19:22:26.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca14929700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca100acb50 con 0x7fca10073c30 2023-03-03T19:22:26.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca0dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca10073c30 0x7fca10152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:26.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca0e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca100ac220 0x7fca100736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:26.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca0e59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca100ac220 0x7fca100736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:36982/0 (socket says 172.21.15.7:36982) 2023-03-03T19:22:26.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca0dd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca10073c30 0x7fca10152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:49282/0 (socket says 172.21.15.7:49282) 2023-03-03T19:22:26.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca0e59c700 1 -- 172.21.15.7:0/1025583155 learned_addr learned my addr 172.21.15.7:0/1025583155 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:22:26.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.943+0000 7fca0ed9d700 1 --1- 172.21.15.7:0/1025583155 >> v1:172.21.15.143:6789/0 conn(0x7fca100ace30 0x7fca1006a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:38712/0 (socket says 172.21.15.7:38712) 2023-03-03T19:22:26.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca0dd9b700 1 -- 172.21.15.7:0/1025583155 >> v2:172.21.15.143:3300/0 conn(0x7fca100ac220 msgr2=0x7fca100736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:22:26.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca0dd9b700 1 --2- 172.21.15.7:0/1025583155 >> v2:172.21.15.143:3300/0 conn(0x7fca100ac220 0x7fca100736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:26.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca0dd9b700 1 -- 172.21.15.7:0/1025583155 >> v1:172.21.15.143:6789/0 conn(0x7fca100ace30 legacy=0x7fca1006a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:26.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca0dd9b700 1 -- 172.21.15.7:0/1025583155 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca101559c0 con 0x7fca10073c30 2023-03-03T19:22:26.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca0e59c700 1 --2- 172.21.15.7:0/1025583155 >> v2:172.21.15.143:3300/0 conn(0x7fca100ac220 0x7fca100736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:22:26.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca0dd9b700 1 --2- 172.21.15.7:0/1025583155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca10073c30 0x7fca10152450 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fca04009840 tx=0x7fca04009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1871c2d747d3789 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:26.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca0d59a700 1 -- 172.21.15.7:0/1025583155 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04011830 con 0x7fca10073c30 2023-03-03T19:22:26.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca0d59a700 1 -- 172.21.15.7:0/1025583155 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca04011990 con 0x7fca10073c30 2023-03-03T19:22:26.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca0d59a700 1 -- 172.21.15.7:0/1025583155 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04011d20 con 0x7fca10073c30 2023-03-03T19:22:26.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca14929700 1 -- 172.21.15.7:0/1025583155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca10073c30 msgr2=0x7fca10152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:26.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.944+0000 7fca14929700 1 --2- 172.21.15.7:0/1025583155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca10073c30 0x7fca10152450 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fca04009840 tx=0x7fca04009b50 comp rx=0 tx=0).stop 2023-03-03T19:22:26.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.945+0000 7fca14929700 1 -- 172.21.15.7:0/1025583155 shutdown_connections 2023-03-03T19:22:26.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.945+0000 7fca14929700 1 --2- 172.21.15.7:0/1025583155 >> v2:172.21.15.143:3300/0 conn(0x7fca100ac220 0x7fca100736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:26.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.945+0000 7fca14929700 1 --2- 172.21.15.7:0/1025583155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca10073c30 0x7fca10152450 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:26.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.945+0000 7fca14929700 1 -- 172.21.15.7:0/1025583155 >> 172.21.15.7:0/1025583155 conn(0x7fca100a94d0 msgr2=0x7fca100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:22:26.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.945+0000 7fca14929700 1 -- 172.21.15.7:0/1025583155 shutdown_connections 2023-03-03T19:22:26.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.945+0000 7fca14929700 1 -- 172.21.15.7:0/1025583155 wait complete. 2023-03-03T19:22:26.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.946+0000 7fca14929700 1 Processor -- start 2023-03-03T19:22:26.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.946+0000 7fca14929700 1 -- start start 2023-03-03T19:22:26.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca14929700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10073c30 0x7fca1006c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:26.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca14929700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca100ac220 0x7fca1006ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:26.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca14929700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca100ace30 0x7fca101514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:26.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca14929700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca10152340 con 0x7fca10073c30 2023-03-03T19:22:26.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca14929700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1014f6b0 con 0x7fca100ac220 2023-03-03T19:22:26.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca14929700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1006e100 con 0x7fca100ace30 2023-03-03T19:22:26.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10073c30 0x7fca1006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:26.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10073c30 0x7fca1006c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41214/0 (socket says 172.21.15.7:41214) 2023-03-03T19:22:26.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0e59c700 1 -- 172.21.15.7:0/1632867235 learned_addr learned my addr 172.21.15.7:0/1632867235 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:22:26.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0dd9b700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca100ac220 0x7fca1006ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:26.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0ed9d700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca100ace30 0x7fca101514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:26.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0e59c700 1 -- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca100ace30 msgr2=0x7fca101514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:26.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0e59c700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca100ace30 0x7fca101514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:26.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0e59c700 1 -- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca100ac220 msgr2=0x7fca1006ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:26.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0e59c700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca100ac220 0x7fca1006ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:26.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0e59c700 1 -- 172.21.15.7:0/1632867235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca00009550 con 0x7fca10073c30 2023-03-03T19:22:26.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0dd9b700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca100ac220 0x7fca1006ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:22:26.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fca0e59c700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10073c30 0x7fca1006c790 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7fca0000e9b0 tx=0x7fca0000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:26.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.947+0000 7fc9feffd700 1 -- 172.21.15.7:0/1632867235 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0000c850 con 0x7fca10073c30 2023-03-03T19:22:26.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.948+0000 7fc9feffd700 1 -- 172.21.15.7:0/1632867235 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0000c9b0 con 0x7fca10073c30 2023-03-03T19:22:26.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.948+0000 7fca14929700 1 -- 172.21.15.7:0/1632867235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca040094a0 con 0x7fca10073c30 2023-03-03T19:22:26.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.948+0000 7fc9feffd700 1 -- 172.21.15.7:0/1632867235 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0000cd20 con 0x7fca10073c30 2023-03-03T19:22:26.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.948+0000 7fca14929700 1 -- 172.21.15.7:0/1632867235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca10155f80 con 0x7fca10073c30 2023-03-03T19:22:26.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.948+0000 7fca14929700 1 -- 172.21.15.7:0/1632867235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca10155ca0 con 0x7fca10073c30 2023-03-03T19:22:26.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.949+0000 7fc9feffd700 1 -- 172.21.15.7:0/1632867235 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fca00019030 con 0x7fca10073c30 2023-03-03T19:22:26.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.949+0000 7fc9feffd700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9f00682f0 0x7fc9f006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:26.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.950+0000 7fc9feffd700 1 -- 172.21.15.7:0/1632867235 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fca00086150 con 0x7fca10073c30 2023-03-03T19:22:26.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.950+0000 7fca0dd9b700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9f00682f0 0x7fc9f006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:26.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.950+0000 7fca0dd9b700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9f00682f0 0x7fc9f006a7a0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fca04000c00 tx=0x7fca0401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:26.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:26.954+0000 7fc9feffd700 1 -- 172.21.15.7:0/1632867235 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fca000524f0 con 0x7fca10073c30 2023-03-03T19:22:27.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.193+0000 7fca14929700 1 -- 172.21.15.7:0/1632867235 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca1014f990 con 0x7fca10073c30 2023-03-03T19:22:27.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.193+0000 7fc9feffd700 1 -- 172.21.15.7:0/1632867235 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fca00055b10 con 0x7fca10073c30 2023-03-03T19:22:27.194 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:22:27.194 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:22:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.195+0000 7fca14929700 1 -- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9f00682f0 msgr2=0x7fc9f006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.195+0000 7fca14929700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9f00682f0 0x7fc9f006a7a0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7fca04000c00 tx=0x7fca0401c040 comp rx=0 tx=0).stop 2023-03-03T19:22:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.195+0000 7fca14929700 1 -- 172.21.15.7:0/1632867235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10073c30 msgr2=0x7fca1006c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.195+0000 7fca14929700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10073c30 0x7fca1006c790 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7fca0000e9b0 tx=0x7fca0000ed70 comp rx=0 tx=0).stop 2023-03-03T19:22:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.196+0000 7fca14929700 1 -- 172.21.15.7:0/1632867235 shutdown_connections 2023-03-03T19:22:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.196+0000 7fca14929700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca100ace30 0x7fca101514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.196+0000 7fca14929700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc9f00682f0 0x7fc9f006a7a0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.196+0000 7fca14929700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca10073c30 0x7fca1006c790 unknown :-1 s=CLOSED pgs=3472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.196+0000 7fca14929700 1 --2- 172.21.15.7:0/1632867235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca100ac220 0x7fca1006ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.196+0000 7fca14929700 1 -- 172.21.15.7:0/1632867235 >> 172.21.15.7:0/1632867235 conn(0x7fca100a94d0 msgr2=0x7fca100c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:22:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.196+0000 7fca14929700 1 -- 172.21.15.7:0/1632867235 shutdown_connections 2023-03-03T19:22:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:27.196+0000 7fca14929700 1 -- 172.21.15.7:0/1632867235 wait complete. 2023-03-03T19:22:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:22:27.209 INFO:tasks.mds_thrash.fs.[b]:waiting for 74 secs before thrashing 2023-03-03T19:22:29.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:22:29.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.820+0000 7f1504cee700 1 Processor -- start 2023-03-03T19:22:29.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.820+0000 7f1504cee700 1 -- start start 2023-03-03T19:22:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.821+0000 7f1504cee700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1500151e80 0x7f15000c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.821+0000 7f1504cee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 0x7f15000bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.821+0000 7f1504cee700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15000c08d0 con 0x7f15000c6fe0 2023-03-03T19:22:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.821+0000 7f1504cee700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1500152250 con 0x7f1500151e80 2023-03-03T19:22:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.821+0000 7f1504cee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15000c0fb0 con 0x7f1500155410 2023-03-03T19:22:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fe59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1500151e80 0x7f15000c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fdd9b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 0x7f15000bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fed9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f15000c6fe0 0x7f15000bfe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:38724/0 (socket says 172.21.15.7:38724) 2023-03-03T19:22:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fed9d700 1 -- 172.21.15.7:0/1423312879 learned_addr learned my addr 172.21.15.7:0/1423312879 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:22:29.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fd59a700 1 -- 172.21.15.7:0/1423312879 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320418169 0 0) 0x7f15000c08d0 con 0x7f15000c6fe0 2023-03-03T19:22:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fdd9b700 1 -- 172.21.15.7:0/1423312879 >> v2:172.21.15.143:3300/0 conn(0x7f1500151e80 msgr2=0x7f15000c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:22:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fdd9b700 1 --2- 172.21.15.7:0/1423312879 >> v2:172.21.15.143:3300/0 conn(0x7f1500151e80 0x7f15000c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fdd9b700 1 -- 172.21.15.7:0/1423312879 >> v1:172.21.15.143:6789/0 conn(0x7f15000c6fe0 legacy=0x7f15000bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fdd9b700 1 -- 172.21.15.7:0/1423312879 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1500154930 con 0x7f1500155410 2023-03-03T19:22:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fe59c700 1 --2- 172.21.15.7:0/1423312879 >> v2:172.21.15.143:3300/0 conn(0x7f1500151e80 0x7f15000c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:22:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.822+0000 7f14fdd9b700 1 --2- 172.21.15.7:0/1423312879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 0x7f15000bf8e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009840 tx=0x7f14f4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dc91aea24e837e7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f14fd59a700 1 -- 172.21.15.7:0/1423312879 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4011830 con 0x7f1500155410 2023-03-03T19:22:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f14fd59a700 1 -- 172.21.15.7:0/1423312879 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f4011990 con 0x7f1500155410 2023-03-03T19:22:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f14fd59a700 1 -- 172.21.15.7:0/1423312879 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4011d20 con 0x7f1500155410 2023-03-03T19:22:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f1504cee700 1 -- 172.21.15.7:0/1423312879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 msgr2=0x7f15000bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f1504cee700 1 --2- 172.21.15.7:0/1423312879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 0x7f15000bf8e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009840 tx=0x7f14f4009b50 comp rx=0 tx=0).stop 2023-03-03T19:22:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f1504cee700 1 -- 172.21.15.7:0/1423312879 shutdown_connections 2023-03-03T19:22:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f1504cee700 1 --2- 172.21.15.7:0/1423312879 >> v2:172.21.15.143:3300/0 conn(0x7f1500151e80 0x7f15000c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f1504cee700 1 --2- 172.21.15.7:0/1423312879 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 0x7f15000bf8e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f1504cee700 1 -- 172.21.15.7:0/1423312879 >> 172.21.15.7:0/1423312879 conn(0x7f1500071630 msgr2=0x7f150006b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:22:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f1504cee700 1 -- 172.21.15.7:0/1423312879 shutdown_connections 2023-03-03T19:22:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.823+0000 7f1504cee700 1 -- 172.21.15.7:0/1423312879 wait complete. 2023-03-03T19:22:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.825+0000 7f1504cee700 1 Processor -- start 2023-03-03T19:22:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.825+0000 7f1504cee700 1 -- start start 2023-03-03T19:22:29.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.825+0000 7f1504cee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15000c6fe0 0x7f150014db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:29.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.825+0000 7f1504cee700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1500151e80 0x7f15001df490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:29.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.825+0000 7f1504cee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 0x7f15001e12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.825+0000 7f1504cee700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15000afcf0 con 0x7f1500151e80 2023-03-03T19:22:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.825+0000 7f1504cee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150014d640 con 0x7f15000c6fe0 2023-03-03T19:22:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.825+0000 7f1504cee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1500157700 con 0x7f1500155410 2023-03-03T19:22:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.825+0000 7f14fdd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1500151e80 0x7f15001df490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fdd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1500151e80 0x7f15001df490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41230/0 (socket says 172.21.15.7:41230) 2023-03-03T19:22:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fdd9b700 1 -- 172.21.15.7:0/1547664498 learned_addr learned my addr 172.21.15.7:0/1547664498 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:22:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fe59c700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15000c6fe0 0x7f150014db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fed9d700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 0x7f15001e12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fdd9b700 1 -- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 msgr2=0x7f15001e12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fdd9b700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 0x7f15001e12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fdd9b700 1 -- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15000c6fe0 msgr2=0x7f150014db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fdd9b700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15000c6fe0 0x7f150014db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fdd9b700 1 -- 172.21.15.7:0/1547664498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f00094a0 con 0x7f1500151e80 2023-03-03T19:22:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fe59c700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15000c6fe0 0x7f150014db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:22:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14fdd9b700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1500151e80 0x7f15001df490 secure :-1 s=READY pgs=3473 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009fd0 tx=0x7f14f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14eeffd700 1 -- 172.21.15.7:0/1547664498 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f401b920 con 0x7f1500151e80 2023-03-03T19:22:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14eeffd700 1 -- 172.21.15.7:0/1547664498 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f401ba80 con 0x7f1500151e80 2023-03-03T19:22:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f1504cee700 1 -- 172.21.15.7:0/1547664498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f40094a0 con 0x7f1500151e80 2023-03-03T19:22:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f14eeffd700 1 -- 172.21.15.7:0/1547664498 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f401be10 con 0x7f1500151e80 2023-03-03T19:22:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.826+0000 7f1504cee700 1 -- 172.21.15.7:0/1547664498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1500154ef0 con 0x7f1500151e80 2023-03-03T19:22:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.827+0000 7f1504cee700 1 -- 172.21.15.7:0/1547664498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1500154c10 con 0x7f1500151e80 2023-03-03T19:22:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.828+0000 7f14eeffd700 1 -- 172.21.15.7:0/1547664498 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f14f402c030 con 0x7f1500151e80 2023-03-03T19:22:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.828+0000 7f14eeffd700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14e00682f0 0x7f14e006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:22:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.828+0000 7f14eeffd700 1 -- 172.21.15.7:0/1547664498 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f14f4087550 con 0x7f1500151e80 2023-03-03T19:22:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.828+0000 7f14fe59c700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14e00682f0 0x7f14e006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:22:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.829+0000 7f14fe59c700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14e00682f0 0x7f14e006a7a0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f14f0009dc0 tx=0x7f14f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:22:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:29.833+0000 7f14eeffd700 1 -- 172.21.15.7:0/1547664498 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f14f4053870 con 0x7f1500151e80 2023-03-03T19:22:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.071+0000 7f1504cee700 1 -- 172.21.15.7:0/1547664498 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15000c5220 con 0x7f1500151e80 2023-03-03T19:22:30.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.071+0000 7f14eeffd700 1 -- 172.21.15.7:0/1547664498 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f14f4015090 con 0x7f1500151e80 2023-03-03T19:22:30.072 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:22:30.072 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:22:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.073+0000 7f1504cee700 1 -- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14e00682f0 msgr2=0x7f14e006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.073+0000 7f1504cee700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14e00682f0 0x7f14e006a7a0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f14f0009dc0 tx=0x7f14f0008040 comp rx=0 tx=0).stop 2023-03-03T19:22:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.073+0000 7f1504cee700 1 -- 172.21.15.7:0/1547664498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1500151e80 msgr2=0x7f15001df490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:22:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.073+0000 7f1504cee700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1500151e80 0x7f15001df490 secure :-1 s=READY pgs=3473 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009fd0 tx=0x7f14f4004750 comp rx=0 tx=0).stop 2023-03-03T19:22:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.073+0000 7f1504cee700 1 -- 172.21.15.7:0/1547664498 shutdown_connections 2023-03-03T19:22:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.073+0000 7f1504cee700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1500155410 0x7f15001e12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:30.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.073+0000 7f1504cee700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f15000c6fe0 0x7f150014db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:30.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.073+0000 7f1504cee700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f14e00682f0 0x7f14e006a7a0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:30.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.073+0000 7f1504cee700 1 --2- 172.21.15.7:0/1547664498 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1500151e80 0x7f15001df490 unknown :-1 s=CLOSED pgs=3473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:22:30.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.073+0000 7f1504cee700 1 -- 172.21.15.7:0/1547664498 >> 172.21.15.7:0/1547664498 conn(0x7f1500071630 msgr2=0x7f1500154140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:22:30.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.074+0000 7f1504cee700 1 -- 172.21.15.7:0/1547664498 shutdown_connections 2023-03-03T19:22:30.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:22:30.074+0000 7f1504cee700 1 -- 172.21.15.7:0/1547664498 wait complete. 2023-03-03T19:22:30.076 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:22:30.086 INFO:tasks.mds_thrash.fs.[a]:waiting for 89 secs before thrashing 2023-03-03T19:22:47.382 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:22:47.386 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:23:17.455 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:23:17.458 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:23:41.211 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:23:41.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.349+0000 7f4f0d22d700 1 Processor -- start 2023-03-03T19:23:41.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.349+0000 7f4f0d22d700 1 -- start start 2023-03-03T19:23:41.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f0d22d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4f0806d870 0x7f4f081550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:41.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f0d22d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f080c7a60 con 0x7f4f080c1da0 2023-03-03T19:23:41.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f0d22d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f080c3620 con 0x7f4f080c3e30 2023-03-03T19:23:41.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f0d22d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f080c2af0 con 0x7f4f0806d870 2023-03-03T19:23:41.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f0659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4f080c1da0 0x7f4f080c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35280/0 (socket says 172.21.15.7:35280) 2023-03-03T19:23:41.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f0659c700 1 -- 172.21.15.7:0/3862146790 learned_addr learned my addr 172.21.15.7:0/3862146790 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:23:41.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f06d9d700 1 --2- 172.21.15.7:0/3862146790 >> v2:172.21.15.143:3300/0 conn(0x7f4f0806d870 0x7f4f081550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:41.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545604570 0 0) 0x7f4f080c7a60 con 0x7f4f080c1da0 2023-03-03T19:23:41.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef00035b0 con 0x7f4f080c1da0 2023-03-03T19:23:41.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 991443563 0 0) 0x7f4ef00035b0 con 0x7f4f080c1da0 2023-03-03T19:23:41.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 >> v2:172.21.15.143:3300/0 conn(0x7f4f0806d870 msgr2=0x7f4f081550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:23:41.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f05d9b700 1 --2- 172.21.15.7:0/3862146790 >> v2:172.21.15.143:3300/0 conn(0x7f4f0806d870 0x7f4f081550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:41.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 >> v1:172.21.15.143:6789/0 conn(0x7f4f080c3e30 legacy=0x7f4f0815ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:41.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f081545e0 con 0x7f4f080c1da0 2023-03-03T19:23:41.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.350+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f4ef8001670 con 0x7f4f080c1da0 2023-03-03T19:23:41.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 >> v1:172.21.15.7:6789/0 conn(0x7f4f080c1da0 legacy=0x7f4f080c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:41.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f05d9b700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef0003a70 0x7f4ef0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f05d9b700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef0004850 0x7f4ef0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f05d9b700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4ef000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef000c0e0 con 0x7f4ef0005200 2023-03-03T19:23:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef000c220 con 0x7f4ef0004850 2023-03-03T19:23:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef000c3e0 con 0x7f4ef0003a70 2023-03-03T19:23:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f0759e700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4ef000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f06d9d700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef0003a70 0x7f4ef0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:41.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f0659c700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef0004850 0x7f4ef0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f0759e700 1 -- 172.21.15.7:0/3862146790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef0003a70 msgr2=0x7f4ef0003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:23:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f0759e700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4ef0003a70 0x7f4ef0003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f0759e700 1 -- 172.21.15.7:0/3862146790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef0004850 msgr2=0x7f4ef0004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:23:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f0759e700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4ef0004850 0x7f4ef0004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f0759e700 1 -- 172.21.15.7:0/3862146790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef00035b0 con 0x7f4ef0005200 2023-03-03T19:23:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f0759e700 1 -- 172.21.15.7:0/3862146790 reap_dead start 2023-03-03T19:23:41.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f0759e700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4ef000ba10 secure :-1 s=READY pgs=3475 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000c190 tx=0x7f4ef0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37b8593d712c233 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:23:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00001db0 con 0x7f4ef0005200 2023-03-03T19:23:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0000ccd0 con 0x7f4ef0005200 2023-03-03T19:23:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.351+0000 7f4f05d9b700 1 -- 172.21.15.7:0/3862146790 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00003c40 con 0x7f4ef0005200 2023-03-03T19:23:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.352+0000 7f4f0d22d700 1 -- 172.21.15.7:0/3862146790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 msgr2=0x7f4ef000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.352+0000 7f4f0d22d700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4ef000ba10 secure :-1 s=READY pgs=3475 cs=0 l=1 rev1=1 crypto rx=0x7f4f0000c190 tx=0x7f4ef0005880 comp rx=0 tx=0).stop 2023-03-03T19:23:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.352+0000 7f4f0d22d700 1 -- 172.21.15.7:0/3862146790 shutdown_connections 2023-03-03T19:23:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.352+0000 7f4f0d22d700 1 --2- 172.21.15.7:0/3862146790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4ef000ba10 unknown :-1 s=CLOSED pgs=3475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:41.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.352+0000 7f4f0d22d700 1 -- 172.21.15.7:0/3862146790 >> 172.21.15.7:0/3862146790 conn(0x7f4f0806c110 msgr2=0x7f4f0806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:23:41.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.352+0000 7f4f0d22d700 1 -- 172.21.15.7:0/3862146790 shutdown_connections 2023-03-03T19:23:41.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.352+0000 7f4f0d22d700 1 -- 172.21.15.7:0/3862146790 wait complete. 2023-03-03T19:23:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.354+0000 7f4f0d22d700 1 Processor -- start 2023-03-03T19:23:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.354+0000 7f4f0d22d700 1 -- start start 2023-03-03T19:23:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f0d22d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4f081e9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f0d22d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f081eac20 0x7f4f080acd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:41.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f0d22d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f0814eb90 0x7f4f080ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f0d22d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080ab390 con 0x7f4ef0005200 2023-03-03T19:23:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f0d22d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080aa780 con 0x7f4f0814eb90 2023-03-03T19:23:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f0d22d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080aaa60 con 0x7f4f081eac20 2023-03-03T19:23:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f06d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4f081e9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f06d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4f081e9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41312/0 (socket says 172.21.15.7:41312) 2023-03-03T19:23:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f06d9d700 1 -- 172.21.15.7:0/2863081213 learned_addr learned my addr 172.21.15.7:0/2863081213 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:23:41.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f0659c700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f081eac20 0x7f4f080acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f0759e700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f0814eb90 0x7f4f080ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f06d9d700 1 -- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f081eac20 msgr2=0x7f4f080acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f06d9d700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f081eac20 0x7f4f080acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f06d9d700 1 -- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f0814eb90 msgr2=0x7f4f080ab820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f06d9d700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f0814eb90 0x7f4f080ab820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f06d9d700 1 -- 172.21.15.7:0/2863081213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f08154ba0 con 0x7f4ef0005200 2023-03-03T19:23:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f0659c700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f081eac20 0x7f4f080acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:23:41.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.355+0000 7f4f0759e700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f0814eb90 0x7f4f080ab820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:23:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.356+0000 7f4f06d9d700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4f081e9730 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0119a0 tx=0x7f4efc011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:23:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.356+0000 7f4eef7fe700 1 -- 172.21.15.7:0/2863081213 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc00f940 con 0x7f4ef0005200 2023-03-03T19:23:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.356+0000 7f4eef7fe700 1 -- 172.21.15.7:0/2863081213 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4efc00faa0 con 0x7f4ef0005200 2023-03-03T19:23:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.356+0000 7f4f0d22d700 1 -- 172.21.15.7:0/2863081213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f081548c0 con 0x7f4ef0005200 2023-03-03T19:23:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.356+0000 7f4eef7fe700 1 -- 172.21.15.7:0/2863081213 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4efc00fe30 con 0x7f4ef0005200 2023-03-03T19:23:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.356+0000 7f4f0d22d700 1 -- 172.21.15.7:0/2863081213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0806dc40 con 0x7f4ef0005200 2023-03-03T19:23:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.357+0000 7f4f0d22d700 1 -- 172.21.15.7:0/2863081213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f080ab670 con 0x7f4ef0005200 2023-03-03T19:23:41.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.357+0000 7f4eef7fe700 1 -- 172.21.15.7:0/2863081213 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4efc014440 con 0x7f4ef0005200 2023-03-03T19:23:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.358+0000 7f4eef7fe700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ef40682f0 0x7f4ef406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.358+0000 7f4eef7fe700 1 -- 172.21.15.7:0/2863081213 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4efc0869c0 con 0x7f4ef0005200 2023-03-03T19:23:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.358+0000 7f4f0659c700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ef40682f0 0x7f4ef406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.358+0000 7f4f0659c700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ef40682f0 0x7f4ef406a7a0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8001cb0 tx=0x7f4ef8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:23:41.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.362+0000 7f4eef7fe700 1 -- 172.21.15.7:0/2863081213 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4efc052d00 con 0x7f4ef0005200 2023-03-03T19:23:41.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.601+0000 7f4f0d22d700 1 -- 172.21.15.7:0/2863081213 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f08074bb0 con 0x7f4ef0005200 2023-03-03T19:23:41.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.602+0000 7f4eef7fe700 1 -- 172.21.15.7:0/2863081213 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4efc056320 con 0x7f4ef0005200 2023-03-03T19:23:41.602 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:23:41.602 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:23:41.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 -- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ef40682f0 msgr2=0x7f4ef406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:41.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ef40682f0 0x7f4ef406a7a0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8001cb0 tx=0x7f4ef8009040 comp rx=0 tx=0).stop 2023-03-03T19:23:41.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 -- 172.21.15.7:0/2863081213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 msgr2=0x7f4f081e9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:41.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4f081e9730 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0119a0 tx=0x7f4efc011d60 comp rx=0 tx=0).stop 2023-03-03T19:23:41.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 -- 172.21.15.7:0/2863081213 shutdown_connections 2023-03-03T19:23:41.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4f0814eb90 0x7f4f080ab820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:41.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4ef40682f0 0x7f4ef406a7a0 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:41.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4ef0005200 0x7f4f081e9730 unknown :-1 s=CLOSED pgs=3476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:41.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 --2- 172.21.15.7:0/2863081213 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4f081eac20 0x7f4f080acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:41.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 -- 172.21.15.7:0/2863081213 >> 172.21.15.7:0/2863081213 conn(0x7f4f0806c110 msgr2=0x7f4f08071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:23:41.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 -- 172.21.15.7:0/2863081213 shutdown_connections 2023-03-03T19:23:41.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:41.604+0000 7f4f0d22d700 1 -- 172.21.15.7:0/2863081213 wait complete. 2023-03-03T19:23:41.606 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:23:41.616 INFO:tasks.mds_thrash.fs.[b]:waiting for 25 secs before thrashing 2023-03-03T19:23:47.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:23:47.512 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:23:59.087 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:23:59.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.225+0000 7f44b30b3700 1 Processor -- start 2023-03-03T19:23:59.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.225+0000 7f44b30b3700 1 -- start start 2023-03-03T19:23:59.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b30b3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44ac071e80 0x7f44ac151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:59.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b30b3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f44ac14e900 0x7f44ac15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:59.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b30b3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ac1510d0 con 0x7f44ac072250 2023-03-03T19:23:59.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b30b3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44ac0c3740 con 0x7f44ac071e80 2023-03-03T19:23:59.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b30b3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44ac070670 con 0x7f44ac14e900 2023-03-03T19:23:59.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b0e4f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44ac071e80 0x7f44ac151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:59.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b0e4f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44ac071e80 0x7f44ac151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41338/0 (socket says 172.21.15.7:41338) 2023-03-03T19:23:59.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b0e4f700 1 -- 172.21.15.7:0/2153934981 learned_addr learned my addr 172.21.15.7:0/2153934981 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:23:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b1650700 1 --2- 172.21.15.7:0/2153934981 >> v2:172.21.15.143:3300/0 conn(0x7f44ac14e900 0x7f44ac15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b0e4f700 1 -- 172.21.15.7:0/2153934981 >> v2:172.21.15.143:3300/0 conn(0x7f44ac14e900 msgr2=0x7f44ac15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:23:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b0e4f700 1 --2- 172.21.15.7:0/2153934981 >> v2:172.21.15.143:3300/0 conn(0x7f44ac14e900 0x7f44ac15ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b0e4f700 1 -- 172.21.15.7:0/2153934981 >> v1:172.21.15.143:6789/0 conn(0x7f44ac072250 legacy=0x7f44ac14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:23:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b0e4f700 1 -- 172.21.15.7:0/2153934981 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ac0c39e0 con 0x7f44ac071e80 2023-03-03T19:23:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b1650700 1 --2- 172.21.15.7:0/2153934981 >> v2:172.21.15.143:3300/0 conn(0x7f44ac14e900 0x7f44ac15ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:23:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44b0e4f700 1 --2- 172.21.15.7:0/2153934981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 0x7f44ac151820 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 crypto rx=0x7f44ac070de0 tx=0x7f44a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3c7b9272b85f7f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:23:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.226+0000 7f44ab7fe700 1 -- 172.21.15.7:0/2153934981 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0004230 con 0x7f44ac071e80 2023-03-03T19:23:59.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.227+0000 7f44ab7fe700 1 -- 172.21.15.7:0/2153934981 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a0004390 con 0x7f44ac071e80 2023-03-03T19:23:59.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.227+0000 7f44ab7fe700 1 -- 172.21.15.7:0/2153934981 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a00036a0 con 0x7f44ac071e80 2023-03-03T19:23:59.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.227+0000 7f44b30b3700 1 -- 172.21.15.7:0/2153934981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 msgr2=0x7f44ac151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:59.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.227+0000 7f44b30b3700 1 --2- 172.21.15.7:0/2153934981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 0x7f44ac151820 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 crypto rx=0x7f44ac070de0 tx=0x7f44a0009bd0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.227+0000 7f44b30b3700 1 -- 172.21.15.7:0/2153934981 shutdown_connections 2023-03-03T19:23:59.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.227+0000 7f44b30b3700 1 --2- 172.21.15.7:0/2153934981 >> v2:172.21.15.143:3300/0 conn(0x7f44ac14e900 0x7f44ac15ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.227+0000 7f44b30b3700 1 --2- 172.21.15.7:0/2153934981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 0x7f44ac151820 unknown :-1 s=CLOSED pgs=3477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.227+0000 7f44b30b3700 1 -- 172.21.15.7:0/2153934981 >> 172.21.15.7:0/2153934981 conn(0x7f44ac0a94d0 msgr2=0x7f44ac0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:23:59.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.227+0000 7f44b30b3700 1 -- 172.21.15.7:0/2153934981 shutdown_connections 2023-03-03T19:23:59.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.227+0000 7f44b30b3700 1 -- 172.21.15.7:0/2153934981 wait complete. 2023-03-03T19:23:59.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.228+0000 7f44b30b3700 1 Processor -- start 2023-03-03T19:23:59.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.228+0000 7f44b30b3700 1 -- start start 2023-03-03T19:23:59.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.229+0000 7f44b30b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 0x7f44ac1edc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:59.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.229+0000 7f44b30b3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac072250 0x7f44ac0bf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:59.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.229+0000 7f44b30b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44ac14e900 0x7f44ac06d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:59.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.229+0000 7f44b30b3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac1dd0a0 con 0x7f44ac071e80 2023-03-03T19:23:59.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.229+0000 7f44b30b3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac1ee1d0 con 0x7f44ac072250 2023-03-03T19:23:59.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.229+0000 7f44b30b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac0c1340 con 0x7f44ac14e900 2023-03-03T19:23:59.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.229+0000 7f44b0e4f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 0x7f44ac1edc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:59.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.229+0000 7f44b0e4f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 0x7f44ac1edc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41342/0 (socket says 172.21.15.7:41342) 2023-03-03T19:23:59.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.229+0000 7f44b0e4f700 1 -- 172.21.15.7:0/3709097821 learned_addr learned my addr 172.21.15.7:0/3709097821 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:23:59.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44abfff700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac072250 0x7f44ac0bf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:59.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44b1650700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44ac14e900 0x7f44ac06d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:59.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44b0e4f700 1 -- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44ac14e900 msgr2=0x7f44ac06d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:59.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44b0e4f700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44ac14e900 0x7f44ac06d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44b0e4f700 1 -- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac072250 msgr2=0x7f44ac0bf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:59.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44b0e4f700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac072250 0x7f44ac0bf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44b0e4f700 1 -- 172.21.15.7:0/3709097821 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a0009550 con 0x7f44ac071e80 2023-03-03T19:23:59.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44b0e4f700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 0x7f44ac1edc90 secure :-1 s=READY pgs=3478 cs=0 l=1 rev1=1 crypto rx=0x7f44a0009130 tx=0x7f44a0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:23:59.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44a97fa700 1 -- 172.21.15.7:0/3709097821 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0003c90 con 0x7f44ac071e80 2023-03-03T19:23:59.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44a97fa700 1 -- 172.21.15.7:0/3709097821 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a001f070 con 0x7f44ac071e80 2023-03-03T19:23:59.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44b30b3700 1 -- 172.21.15.7:0/3709097821 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ac06c9a0 con 0x7f44ac071e80 2023-03-03T19:23:59.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44a97fa700 1 -- 172.21.15.7:0/3709097821 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0011870 con 0x7f44ac071e80 2023-03-03T19:23:59.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.230+0000 7f44b30b3700 1 -- 172.21.15.7:0/3709097821 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ac151300 con 0x7f44ac071e80 2023-03-03T19:23:59.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.231+0000 7f44b30b3700 1 -- 172.21.15.7:0/3709097821 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ac06cb30 con 0x7f44ac071e80 2023-03-03T19:23:59.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.232+0000 7f44a97fa700 1 -- 172.21.15.7:0/3709097821 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f44a0017020 con 0x7f44ac071e80 2023-03-03T19:23:59.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.232+0000 7f44a97fa700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44900682f0 0x7f449006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:23:59.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.232+0000 7f44a97fa700 1 -- 172.21.15.7:0/3709097821 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f44a0087800 con 0x7f44ac071e80 2023-03-03T19:23:59.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.232+0000 7f44abfff700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44900682f0 0x7f449006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:23:59.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.233+0000 7f44abfff700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44900682f0 0x7f449006a7a0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f44ac14d7e0 tx=0x7f4498007ec0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:23:59.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.236+0000 7f44a97fa700 1 -- 172.21.15.7:0/3709097821 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f44a0053ba0 con 0x7f44ac071e80 2023-03-03T19:23:59.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.472+0000 7f44b30b3700 1 -- 172.21.15.7:0/3709097821 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44ac074bb0 con 0x7f44ac071e80 2023-03-03T19:23:59.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.472+0000 7f44a97fa700 1 -- 172.21.15.7:0/3709097821 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f44a00571c0 con 0x7f44ac071e80 2023-03-03T19:23:59.474 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:23:59.474 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:23:59.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 -- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44900682f0 msgr2=0x7f449006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:59.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44900682f0 0x7f449006a7a0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f44ac14d7e0 tx=0x7f4498007ec0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 -- 172.21.15.7:0/3709097821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 msgr2=0x7f44ac1edc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:23:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 0x7f44ac1edc90 secure :-1 s=READY pgs=3478 cs=0 l=1 rev1=1 crypto rx=0x7f44a0009130 tx=0x7f44a0004750 comp rx=0 tx=0).stop 2023-03-03T19:23:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 -- 172.21.15.7:0/3709097821 shutdown_connections 2023-03-03T19:23:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f44ac14e900 0x7f44ac06d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f44900682f0 0x7f449006a7a0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44ac071e80 0x7f44ac1edc90 unknown :-1 s=CLOSED pgs=3478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 --2- 172.21.15.7:0/3709097821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac072250 0x7f44ac0bf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:23:59.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 -- 172.21.15.7:0/3709097821 >> 172.21.15.7:0/3709097821 conn(0x7f44ac0a94d0 msgr2=0x7f44ac151ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:23:59.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 -- 172.21.15.7:0/3709097821 shutdown_connections 2023-03-03T19:23:59.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:23:59.474+0000 7f44b30b3700 1 -- 172.21.15.7:0/3709097821 wait complete. 2023-03-03T19:23:59.477 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:23:59.487 INFO:tasks.mds_thrash.fs.[a]:waiting for 45 secs before thrashing 2023-03-03T19:24:06.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:24:06.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8edaeea700 1 Processor -- start 2023-03-03T19:24:06.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8edaeea700 1 -- start start 2023-03-03T19:24:06.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8edaeea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ed4151d40 0x7f8ed40736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8edaeea700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8ed4073c30 0x7f8ed41581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8edaeea700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed406d430 con 0x7f8ed40c2930 2023-03-03T19:24:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8edaeea700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed406c2a0 con 0x7f8ed4151d40 2023-03-03T19:24:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8edaeea700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed41586e0 con 0x7f8ed4073c30 2023-03-03T19:24:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8ed8c86700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ed4151d40 0x7f8ed40736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8ed8c86700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ed4151d40 0x7f8ed40736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41350/0 (socket says 172.21.15.7:41350) 2023-03-03T19:24:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8ed8c86700 1 -- 172.21.15.7:0/1108905176 learned_addr learned my addr 172.21.15.7:0/1108905176 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:24:06.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8ed9487700 1 --2- 172.21.15.7:0/1108905176 >> v2:172.21.15.143:3300/0 conn(0x7f8ed4073c30 0x7f8ed41581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:06.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8ed8c86700 1 -- 172.21.15.7:0/1108905176 >> v2:172.21.15.143:3300/0 conn(0x7f8ed4073c30 msgr2=0x7f8ed41581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:24:06.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8ed8c86700 1 --2- 172.21.15.7:0/1108905176 >> v2:172.21.15.143:3300/0 conn(0x7f8ed4073c30 0x7f8ed41581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:06.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8ed8c86700 1 -- 172.21.15.7:0/1108905176 >> v1:172.21.15.143:6789/0 conn(0x7f8ed40c2930 legacy=0x7f8ed4157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:24:06.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8ed8c86700 1 -- 172.21.15.7:0/1108905176 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed406cb00 con 0x7f8ed4151d40 2023-03-03T19:24:06.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.756+0000 7f8ed8c86700 1 --2- 172.21.15.7:0/1108905176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 0x7f8ed40736f0 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80098f0 tx=0x7f8ec8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967dc53b5c4adb9c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:06.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8ed37fe700 1 -- 172.21.15.7:0/1108905176 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8004230 con 0x7f8ed4151d40 2023-03-03T19:24:06.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8ed37fe700 1 -- 172.21.15.7:0/1108905176 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec8004390 con 0x7f8ed4151d40 2023-03-03T19:24:06.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8ed37fe700 1 -- 172.21.15.7:0/1108905176 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec80036a0 con 0x7f8ed4151d40 2023-03-03T19:24:06.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8edaeea700 1 -- 172.21.15.7:0/1108905176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 msgr2=0x7f8ed40736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:06.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8edaeea700 1 --2- 172.21.15.7:0/1108905176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 0x7f8ed40736f0 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80098f0 tx=0x7f8ec8009c00 comp rx=0 tx=0).stop 2023-03-03T19:24:06.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8edaeea700 1 -- 172.21.15.7:0/1108905176 shutdown_connections 2023-03-03T19:24:06.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8edaeea700 1 --2- 172.21.15.7:0/1108905176 >> v2:172.21.15.143:3300/0 conn(0x7f8ed4073c30 0x7f8ed41581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:06.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8edaeea700 1 --2- 172.21.15.7:0/1108905176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 0x7f8ed40736f0 unknown :-1 s=CLOSED pgs=3479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:06.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8edaeea700 1 -- 172.21.15.7:0/1108905176 >> 172.21.15.7:0/1108905176 conn(0x7f8ed40a94d0 msgr2=0x7f8ed40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:24:06.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8edaeea700 1 -- 172.21.15.7:0/1108905176 shutdown_connections 2023-03-03T19:24:06.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.757+0000 7f8edaeea700 1 -- 172.21.15.7:0/1108905176 wait complete. 2023-03-03T19:24:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.758+0000 7f8edaeea700 1 Processor -- start 2023-03-03T19:24:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.758+0000 7f8edaeea700 1 -- start start 2023-03-03T19:24:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8edaeea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ed4073c30 0x7f8ed40bde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:06.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8edaeea700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ed40c2930 0x7f8ed41485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:06.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8edaeea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 0x7f8ed41df680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:06.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8edaeea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed41ddaf0 con 0x7f8ed4151d40 2023-03-03T19:24:06.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8edaeea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed41ead20 con 0x7f8ed4073c30 2023-03-03T19:24:06.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8edaeea700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed41eb000 con 0x7f8ed40c2930 2023-03-03T19:24:06.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8ed9487700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 0x7f8ed41df680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:06.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8ed9487700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 0x7f8ed41df680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41358/0 (socket says 172.21.15.7:41358) 2023-03-03T19:24:06.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8ed9487700 1 -- 172.21.15.7:0/2757389164 learned_addr learned my addr 172.21.15.7:0/2757389164 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:24:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8ed8c86700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ed4073c30 0x7f8ed40bde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.759+0000 7f8ed3fff700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ed40c2930 0x7f8ed41485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8ed9487700 1 -- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ed40c2930 msgr2=0x7f8ed41485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8ed9487700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ed40c2930 0x7f8ed41485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8ed9487700 1 -- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ed4073c30 msgr2=0x7f8ed40bde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8ed9487700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ed4073c30 0x7f8ed40bde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8ed9487700 1 -- 172.21.15.7:0/2757389164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec8009550 con 0x7f8ed4151d40 2023-03-03T19:24:06.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8ed9487700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 0x7f8ed41df680 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009fd0 tx=0x7f8ecc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8ed17fa700 1 -- 172.21.15.7:0/2757389164 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc007ea0 con 0x7f8ed4151d40 2023-03-03T19:24:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8ed17fa700 1 -- 172.21.15.7:0/2757389164 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc00ed30 con 0x7f8ed4151d40 2023-03-03T19:24:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8ed17fa700 1 -- 172.21.15.7:0/2757389164 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc00eea0 con 0x7f8ed4151d40 2023-03-03T19:24:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8edaeea700 1 -- 172.21.15.7:0/2757389164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed406d0c0 con 0x7f8ed4151d40 2023-03-03T19:24:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.760+0000 7f8edaeea700 1 -- 172.21.15.7:0/2757389164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed406cde0 con 0x7f8ed4151d40 2023-03-03T19:24:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.761+0000 7f8edaeea700 1 -- 172.21.15.7:0/2757389164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed406bea0 con 0x7f8ed4151d40 2023-03-03T19:24:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.761+0000 7f8ed17fa700 1 -- 172.21.15.7:0/2757389164 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8ecc012050 con 0x7f8ed4151d40 2023-03-03T19:24:06.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.762+0000 7f8ed17fa700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb80682f0 0x7f8eb806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.762+0000 7f8ed17fa700 1 -- 172.21.15.7:0/2757389164 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8ecc0865d0 con 0x7f8ed4151d40 2023-03-03T19:24:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.762+0000 7f8ed8c86700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb80682f0 0x7f8eb806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.763+0000 7f8ed8c86700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb80682f0 0x7f8eb806a7a0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009130 tx=0x7f8ec8008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:06.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:06.768+0000 7f8ed17fa700 1 -- 172.21.15.7:0/2757389164 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8ecc052970 con 0x7f8ed4151d40 2023-03-03T19:24:07.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.002+0000 7f8edaeea700 1 -- 172.21.15.7:0/2757389164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ed41eb1c0 con 0x7f8ed4151d40 2023-03-03T19:24:07.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.002+0000 7f8ed17fa700 1 -- 172.21.15.7:0/2757389164 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8ed41eb1c0 con 0x7f8ed4151d40 2023-03-03T19:24:07.003 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:24:07.003 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:24:07.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 -- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb80682f0 msgr2=0x7f8eb806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:07.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb80682f0 0x7f8eb806a7a0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009130 tx=0x7f8ec8008030 comp rx=0 tx=0).stop 2023-03-03T19:24:07.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 -- 172.21.15.7:0/2757389164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 msgr2=0x7f8ed41df680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:07.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 0x7f8ed41df680 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009fd0 tx=0x7f8ecc00e3c0 comp rx=0 tx=0).stop 2023-03-03T19:24:07.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 -- 172.21.15.7:0/2757389164 shutdown_connections 2023-03-03T19:24:07.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8eb80682f0 0x7f8eb806a7a0 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:07.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed4151d40 0x7f8ed41df680 unknown :-1 s=CLOSED pgs=3480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:07.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8ed4073c30 0x7f8ed40bde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:07.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 --2- 172.21.15.7:0/2757389164 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8ed40c2930 0x7f8ed41485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:07.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 -- 172.21.15.7:0/2757389164 >> 172.21.15.7:0/2757389164 conn(0x7f8ed40a94d0 msgr2=0x7f8ed41558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:24:07.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 -- 172.21.15.7:0/2757389164 shutdown_connections 2023-03-03T19:24:07.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:07.004+0000 7f8edaeea700 1 -- 172.21.15.7:0/2757389164 wait complete. 2023-03-03T19:24:07.007 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:24:07.018 INFO:tasks.mds_thrash.fs.[b]:waiting for 33 secs before thrashing 2023-03-03T19:24:17.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:24:17.585 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:24:40.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:24:40.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.159+0000 7f4d4fc4e700 1 Processor -- start 2023-03-03T19:24:40.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.159+0000 7f4d4fc4e700 1 -- start start 2023-03-03T19:24:40.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.160+0000 7f4d4fc4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c1da0 0x7f4d4806d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:40.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.160+0000 7f4d4fc4e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4806de90 con 0x7f4d481586c0 2023-03-03T19:24:40.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.160+0000 7f4d4fc4e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d48154560 con 0x7f4d4813a0f0 2023-03-03T19:24:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.160+0000 7f4d4fc4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d48154840 con 0x7f4d480c1da0 2023-03-03T19:24:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.160+0000 7f4d4d1e9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4d481586c0 0x7f4d4806d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35380/0 (socket says 172.21.15.7:35380) 2023-03-03T19:24:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.160+0000 7f4d4d1e9700 1 -- 172.21.15.7:0/3433925342 learned_addr learned my addr 172.21.15.7:0/3433925342 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:24:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.160+0000 7f4d4e1eb700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c1da0 0x7f4d4806d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703503821 0 0) 0x7f4d38001110 con 0x7f4d481586c0 2023-03-03T19:24:40.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d340035b0 con 0x7f4d481586c0 2023-03-03T19:24:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3858759476 0 0) 0x7f4d340035b0 con 0x7f4d481586c0 2023-03-03T19:24:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c1da0 msgr2=0x7f4d4806d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:24:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c1da0 0x7f4d4806d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 >> v1:172.21.15.143:6789/0 conn(0x7f4d4813a0f0 legacy=0x7f4d480c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d48157be0 con 0x7f4d481586c0 2023-03-03T19:24:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f4d38001800 con 0x7f4d481586c0 2023-03-03T19:24:40.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 >> v1:172.21.15.7:6789/0 conn(0x7f4d481586c0 legacy=0x7f4d4806d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:40.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d34003aa0 0x7f4d34003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:40.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c1da0 0x7f4d34004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:40.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34004d90 0x7f4d34005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:40.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d48154840 con 0x7f4d34004d90 2023-03-03T19:24:40.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3400bcc0 con 0x7f4d34003aa0 2023-03-03T19:24:40.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3400be80 con 0x7f4d480c1da0 2023-03-03T19:24:40.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4e1eb700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34004d90 0x7f4d34005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:40.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4d9ea700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d34003aa0 0x7f4d34003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:40.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4d1e9700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c1da0 0x7f4d34004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:40.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4e1eb700 1 -- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c1da0 msgr2=0x7f4d34004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:24:40.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4e1eb700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c1da0 0x7f4d34004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:40.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4e1eb700 1 -- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d34003aa0 msgr2=0x7f4d34003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:24:40.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4e1eb700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d34003aa0 0x7f4d34003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:40.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.161+0000 7f4d4e1eb700 1 -- 172.21.15.7:0/3433925342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d340035b0 con 0x7f4d34004d90 2023-03-03T19:24:40.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4e1eb700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34004d90 0x7f4d34005200 secure :-1 s=READY pgs=3482 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009fc0 tx=0x7f4d34005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb84d2070c0bdb7 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:40.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4000ec30 con 0x7f4d34004d90 2023-03-03T19:24:40.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4000f040 con 0x7f4d34004d90 2023-03-03T19:24:40.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4c9e8700 1 -- 172.21.15.7:0/3433925342 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40014480 con 0x7f4d34004d90 2023-03-03T19:24:40.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/3433925342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34004d90 msgr2=0x7f4d34005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:40.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4fc4e700 1 --2- 172.21.15.7:0/3433925342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d34004d90 0x7f4d34005200 secure :-1 s=READY pgs=3482 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009fc0 tx=0x7f4d34005880 comp rx=0 tx=0).stop 2023-03-03T19:24:40.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4e1eb700 1 -- 172.21.15.7:0/3433925342 reap_dead start 2023-03-03T19:24:40.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/3433925342 shutdown_connections 2023-03-03T19:24:40.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/3433925342 >> 172.21.15.7:0/3433925342 conn(0x7f4d4806c110 msgr2=0x7f4d4806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:24:40.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/3433925342 shutdown_connections 2023-03-03T19:24:40.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.162+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/3433925342 wait complete. 2023-03-03T19:24:40.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4fc4e700 1 Processor -- start 2023-03-03T19:24:40.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4fc4e700 1 -- start start 2023-03-03T19:24:40.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4fc4e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d480c0a40 0x7f4d48071790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:40.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4fc4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c0e10 0x7f4d481e9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4fc4e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d481ea3d0 0x7f4d48136940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4fc4e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d48072370 con 0x7f4d481ea3d0 2023-03-03T19:24:40.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4fc4e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d480728a0 con 0x7f4d480c0a40 2023-03-03T19:24:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4fc4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d481edeb0 con 0x7f4d480c0e10 2023-03-03T19:24:40.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4e1eb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d481ea3d0 0x7f4d48136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:40.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4e1eb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d481ea3d0 0x7f4d48136940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41416/0 (socket says 172.21.15.7:41416) 2023-03-03T19:24:40.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.164+0000 7f4d4e1eb700 1 -- 172.21.15.7:0/266161541 learned_addr learned my addr 172.21.15.7:0/266161541 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:24:40.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d4d9ea700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d480c0a40 0x7f4d48071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:40.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d4d1e9700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c0e10 0x7f4d481e9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:40.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d4e1eb700 1 -- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c0e10 msgr2=0x7f4d481e9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:40.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d4e1eb700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c0e10 0x7f4d481e9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:40.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d4e1eb700 1 -- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d480c0a40 msgr2=0x7f4d48071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:40.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d4e1eb700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d480c0a40 0x7f4d48071790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:40.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d4e1eb700 1 -- 172.21.15.7:0/266161541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d481581a0 con 0x7f4d481ea3d0 2023-03-03T19:24:40.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d4e1eb700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d481ea3d0 0x7f4d48136940 secure :-1 s=READY pgs=3483 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400bc90 tx=0x7f4d40009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:40.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d3e7fc700 1 -- 172.21.15.7:0/266161541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40007000 con 0x7f4d481ea3d0 2023-03-03T19:24:40.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d3e7fc700 1 -- 172.21.15.7:0/266161541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d40014890 con 0x7f4d481ea3d0 2023-03-03T19:24:40.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/266161541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d48157ec0 con 0x7f4d481ea3d0 2023-03-03T19:24:40.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d3e7fc700 1 -- 172.21.15.7:0/266161541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d40014c40 con 0x7f4d481ea3d0 2023-03-03T19:24:40.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.165+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/266161541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4806e1a0 con 0x7f4d481ea3d0 2023-03-03T19:24:40.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.166+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/266161541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d48074bb0 con 0x7f4d481ea3d0 2023-03-03T19:24:40.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.167+0000 7f4d3e7fc700 1 -- 172.21.15.7:0/266161541 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4d4001e430 con 0x7f4d481ea3d0 2023-03-03T19:24:40.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.167+0000 7f4d3e7fc700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4d2c0682f0 0x7f4d2c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:40.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.167+0000 7f4d3e7fc700 1 -- 172.21.15.7:0/266161541 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4d40088000 con 0x7f4d481ea3d0 2023-03-03T19:24:40.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.167+0000 7f4d4d9ea700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4d2c0682f0 0x7f4d2c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:40.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.168+0000 7f4d4d9ea700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4d2c0682f0 0x7f4d2c06a7a0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f4d48155240 tx=0x7f4d44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:40.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.171+0000 7f4d3e7fc700 1 -- 172.21.15.7:0/266161541 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4d40054400 con 0x7f4d481ea3d0 2023-03-03T19:24:40.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.409+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/266161541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d480714b0 con 0x7f4d481ea3d0 2023-03-03T19:24:40.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.410+0000 7f4d3e7fc700 1 -- 172.21.15.7:0/266161541 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4d480714b0 con 0x7f4d481ea3d0 2023-03-03T19:24:40.410 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:24:40.411 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:24:40.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4d2c0682f0 msgr2=0x7f4d2c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:40.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4d2c0682f0 0x7f4d2c06a7a0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f4d48155240 tx=0x7f4d44009040 comp rx=0 tx=0).stop 2023-03-03T19:24:40.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/266161541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d481ea3d0 msgr2=0x7f4d48136940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:40.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d481ea3d0 0x7f4d48136940 secure :-1 s=READY pgs=3483 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400bc90 tx=0x7f4d40009750 comp rx=0 tx=0).stop 2023-03-03T19:24:40.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/266161541 shutdown_connections 2023-03-03T19:24:40.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4d2c0682f0 0x7f4d2c06a7a0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:40.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4d481ea3d0 0x7f4d48136940 unknown :-1 s=CLOSED pgs=3483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:40.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d480c0a40 0x7f4d48071790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:40.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 --2- 172.21.15.7:0/266161541 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4d480c0e10 0x7f4d481e9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:40.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/266161541 >> 172.21.15.7:0/266161541 conn(0x7f4d4806c110 msgr2=0x7f4d48139180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:24:40.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/266161541 shutdown_connections 2023-03-03T19:24:40.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:40.412+0000 7f4d4fc4e700 1 -- 172.21.15.7:0/266161541 wait complete. 2023-03-03T19:24:40.415 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:24:40.424 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before thrashing 2023-03-03T19:24:44.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:24:44.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.628+0000 7fa83f399700 1 Processor -- start 2023-03-03T19:24:44.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.628+0000 7fa83f399700 1 -- start start 2023-03-03T19:24:44.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83f399700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa83814fc50 0x7fa8380c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:44.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83f399700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa83806d240 0x7fa8380c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:44.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83f399700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8380ad2b0 con 0x7fa8380c3e30 2023-03-03T19:24:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83f399700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8380c30c0 con 0x7fa83814fc50 2023-03-03T19:24:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83f399700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa838156420 con 0x7fa83806d240 2023-03-03T19:24:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83d135700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa83814fc50 0x7fa8380c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83d936700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa8380c3e30 0x7fa8380c7390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35408/0 (socket says 172.21.15.7:35408) 2023-03-03T19:24:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83d936700 1 -- 172.21.15.7:0/729958192 learned_addr learned my addr 172.21.15.7:0/729958192 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:24:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83c934700 1 --2- 172.21.15.7:0/729958192 >> v2:172.21.15.143:3300/0 conn(0x7fa83806d240 0x7fa8380c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:44.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83d135700 1 -- 172.21.15.7:0/729958192 >> v2:172.21.15.143:3300/0 conn(0x7fa83806d240 msgr2=0x7fa8380c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:24:44.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83d135700 1 --2- 172.21.15.7:0/729958192 >> v2:172.21.15.143:3300/0 conn(0x7fa83806d240 0x7fa8380c27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:44.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83d135700 1 -- 172.21.15.7:0/729958192 >> v1:172.21.15.7:6789/0 conn(0x7fa8380c3e30 legacy=0x7fa8380c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:44.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa83d135700 1 -- 172.21.15.7:0/729958192 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8380ad520 con 0x7fa83814fc50 2023-03-03T19:24:44.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.629+0000 7fa82ffff700 1 -- 172.21.15.7:0/729958192 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423810381 0 0) 0x7fa8380ad2b0 con 0x7fa8380c3e30 2023-03-03T19:24:44.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa83c934700 1 --2- 172.21.15.7:0/729958192 >> v2:172.21.15.143:3300/0 conn(0x7fa83806d240 0x7fa8380c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:24:44.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa83d135700 1 --2- 172.21.15.7:0/729958192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83814fc50 0x7fa8380c1f50 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7fa8380acf10 tx=0x7fa828009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ef4aff4a8fd84a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:44.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa82ffff700 1 -- 172.21.15.7:0/729958192 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828011830 con 0x7fa83814fc50 2023-03-03T19:24:44.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa82ffff700 1 -- 172.21.15.7:0/729958192 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa828011990 con 0x7fa83814fc50 2023-03-03T19:24:44.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa82ffff700 1 -- 172.21.15.7:0/729958192 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828011d20 con 0x7fa83814fc50 2023-03-03T19:24:44.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa83f399700 1 -- 172.21.15.7:0/729958192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83814fc50 msgr2=0x7fa8380c1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:44.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa83f399700 1 --2- 172.21.15.7:0/729958192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83814fc50 0x7fa8380c1f50 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7fa8380acf10 tx=0x7fa828009b20 comp rx=0 tx=0).stop 2023-03-03T19:24:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa83f399700 1 -- 172.21.15.7:0/729958192 shutdown_connections 2023-03-03T19:24:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa83f399700 1 --2- 172.21.15.7:0/729958192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa83814fc50 0x7fa8380c1f50 unknown :-1 s=CLOSED pgs=3485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa83f399700 1 --2- 172.21.15.7:0/729958192 >> v2:172.21.15.143:3300/0 conn(0x7fa83806d240 0x7fa8380c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.630+0000 7fa83f399700 1 -- 172.21.15.7:0/729958192 >> 172.21.15.7:0/729958192 conn(0x7fa83806c110 msgr2=0x7fa83806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:24:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.631+0000 7fa83f399700 1 -- 172.21.15.7:0/729958192 shutdown_connections 2023-03-03T19:24:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.631+0000 7fa83f399700 1 -- 172.21.15.7:0/729958192 wait complete. 2023-03-03T19:24:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.632+0000 7fa83f399700 1 Processor -- start 2023-03-03T19:24:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.632+0000 7fa83f399700 1 -- start start 2023-03-03T19:24:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.632+0000 7fa83f399700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83806d240 0x7fa8381e8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.632+0000 7fa83f399700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8380c3e30 0x7fa83814e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.632+0000 7fa83f399700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83814fc50 0x7fa8380c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.632+0000 7fa83f399700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8381e5640 con 0x7fa8380c3e30 2023-03-03T19:24:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.632+0000 7fa83f399700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8381e8390 con 0x7fa83814fc50 2023-03-03T19:24:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.632+0000 7fa83f399700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8381e88c0 con 0x7fa83806d240 2023-03-03T19:24:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.632+0000 7fa83c934700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8380c3e30 0x7fa83814e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83c934700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8380c3e30 0x7fa83814e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41432/0 (socket says 172.21.15.7:41432) 2023-03-03T19:24:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83c934700 1 -- 172.21.15.7:0/593971255 learned_addr learned my addr 172.21.15.7:0/593971255 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:24:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83d135700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83806d240 0x7fa8381e8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83d936700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83814fc50 0x7fa8380c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83c934700 1 -- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83806d240 msgr2=0x7fa8381e8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83c934700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83806d240 0x7fa8381e8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83c934700 1 -- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83814fc50 msgr2=0x7fa8380c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83c934700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83814fc50 0x7fa8380c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83c934700 1 -- 172.21.15.7:0/593971255 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8280094a0 con 0x7fa8380c3e30 2023-03-03T19:24:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83c934700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8380c3e30 0x7fa83814e3b0 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7fa83400c820 tx=0x7fa83400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa82dffb700 1 -- 172.21.15.7:0/593971255 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834011800 con 0x7fa8380c3e30 2023-03-03T19:24:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa82dffb700 1 -- 172.21.15.7:0/593971255 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa834011960 con 0x7fa8380c3e30 2023-03-03T19:24:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83f399700 1 -- 172.21.15.7:0/593971255 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8380c2cf0 con 0x7fa8380c3e30 2023-03-03T19:24:44.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa82dffb700 1 -- 172.21.15.7:0/593971255 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834011d10 con 0x7fa8380c3e30 2023-03-03T19:24:44.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.633+0000 7fa83f399700 1 -- 172.21.15.7:0/593971255 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8380c2490 con 0x7fa8380c3e30 2023-03-03T19:24:44.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.634+0000 7fa83f399700 1 -- 172.21.15.7:0/593971255 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa838074220 con 0x7fa8380c3e30 2023-03-03T19:24:44.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.635+0000 7fa82dffb700 1 -- 172.21.15.7:0/593971255 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa834021070 con 0x7fa8380c3e30 2023-03-03T19:24:44.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.635+0000 7fa82dffb700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 0x7fa81c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:44.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.635+0000 7fa82dffb700 1 -- 172.21.15.7:0/593971255 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa834086030 con 0x7fa8380c3e30 2023-03-03T19:24:44.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.635+0000 7fa83d135700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 0x7fa81c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:44.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.636+0000 7fa83d135700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 0x7fa81c06a7a0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fa8280090b0 tx=0x7fa828008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:44.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.640+0000 7fa82dffb700 1 -- 172.21.15.7:0/593971255 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa834052350 con 0x7fa8380c3e30 2023-03-03T19:24:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.879+0000 7fa83f399700 1 -- 172.21.15.7:0/593971255 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa838154cf0 con 0x7fa8380c3e30 2023-03-03T19:24:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.879+0000 7fa82dffb700 1 -- 172.21.15.7:0/593971255 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa834055970 con 0x7fa8380c3e30 2023-03-03T19:24:44.880 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:24:44.880 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:24:44.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 -- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 msgr2=0x7fa81c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:44.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 0x7fa81c06a7a0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7fa8280090b0 tx=0x7fa828008030 comp rx=0 tx=0).stop 2023-03-03T19:24:44.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 -- 172.21.15.7:0/593971255 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8380c3e30 msgr2=0x7fa83814e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:44.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8380c3e30 0x7fa83814e3b0 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7fa83400c820 tx=0x7fa83400cbe0 comp rx=0 tx=0).stop 2023-03-03T19:24:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 -- 172.21.15.7:0/593971255 shutdown_connections 2023-03-03T19:24:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa83814fc50 0x7fa8380c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa83806d240 0x7fa8381e8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa81c0682f0 0x7fa81c06a7a0 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 --2- 172.21.15.7:0/593971255 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8380c3e30 0x7fa83814e3b0 unknown :-1 s=CLOSED pgs=3486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 -- 172.21.15.7:0/593971255 >> 172.21.15.7:0/593971255 conn(0x7fa83806c110 msgr2=0x7fa83806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:24:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 -- 172.21.15.7:0/593971255 shutdown_connections 2023-03-03T19:24:44.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:44.882+0000 7fa83f399700 1 -- 172.21.15.7:0/593971255 wait complete. 2023-03-03T19:24:44.884 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:24:44.894 INFO:tasks.mds_thrash.fs.[a]:waiting for 57 secs before thrashing 2023-03-03T19:24:47.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:24:47.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.564+0000 7faa492fc700 1 Processor -- start 2023-03-03T19:24:47.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.564+0000 7faa492fc700 1 -- start start 2023-03-03T19:24:47.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa492fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 0x7faa440c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:47.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa492fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7faa44154ff0 0x7faa440cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:47.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa492fc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa440c7310 con 0x7faa440c00e0 2023-03-03T19:24:47.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa492fc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa440bfd10 con 0x7faa44154ff0 2023-03-03T19:24:47.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa492fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa440bfa30 con 0x7faa440caa30 2023-03-03T19:24:47.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa427fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faa440c00e0 0x7faa440c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35418/0 (socket says 172.21.15.7:35418) 2023-03-03T19:24:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa427fc700 1 -- 172.21.15.7:0/272161159 learned_addr learned my addr 172.21.15.7:0/272161159 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:24:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa437fe700 1 --2- 172.21.15.7:0/272161159 >> v2:172.21.15.143:3300/0 conn(0x7faa44154ff0 0x7faa440cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa42ffd700 1 --2- 172.21.15.7:0/272161159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 0x7faa440c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa41ffb700 1 -- 172.21.15.7:0/272161159 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726164256 0 0) 0x7faa440c7310 con 0x7faa440c00e0 2023-03-03T19:24:47.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa41ffb700 1 -- 172.21.15.7:0/272161159 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa2c0035b0 con 0x7faa440c00e0 2023-03-03T19:24:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa41ffb700 1 -- 172.21.15.7:0/272161159 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3505324531 0 0) 0x7faa2c0035b0 con 0x7faa440c00e0 2023-03-03T19:24:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa42ffd700 1 -- 172.21.15.7:0/272161159 >> v2:172.21.15.143:3300/0 conn(0x7faa44154ff0 msgr2=0x7faa440cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:24:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa42ffd700 1 --2- 172.21.15.7:0/272161159 >> v2:172.21.15.143:3300/0 conn(0x7faa44154ff0 0x7faa440cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa42ffd700 1 -- 172.21.15.7:0/272161159 >> v1:172.21.15.7:6789/0 conn(0x7faa440c00e0 legacy=0x7faa440c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa42ffd700 1 -- 172.21.15.7:0/272161159 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa44154510 con 0x7faa440caa30 2023-03-03T19:24:47.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.565+0000 7faa437fe700 1 --2- 172.21.15.7:0/272161159 >> v2:172.21.15.143:3300/0 conn(0x7faa44154ff0 0x7faa440cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:24:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa42ffd700 1 --2- 172.21.15.7:0/272161159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 0x7faa440c5c60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faa38009840 tx=0x7faa38009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcfb10df9a384fca server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa41ffb700 1 -- 172.21.15.7:0/272161159 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa340017f0 con 0x7faa440caa30 2023-03-03T19:24:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa41ffb700 1 -- 172.21.15.7:0/272161159 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa38004030 con 0x7faa440caa30 2023-03-03T19:24:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa41ffb700 1 -- 172.21.15.7:0/272161159 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa380043c0 con 0x7faa440caa30 2023-03-03T19:24:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa492fc700 1 -- 172.21.15.7:0/272161159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 msgr2=0x7faa440c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa492fc700 1 --2- 172.21.15.7:0/272161159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 0x7faa440c5c60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faa38009840 tx=0x7faa38009b50 comp rx=0 tx=0).stop 2023-03-03T19:24:47.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa492fc700 1 -- 172.21.15.7:0/272161159 shutdown_connections 2023-03-03T19:24:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa492fc700 1 --2- 172.21.15.7:0/272161159 >> v2:172.21.15.143:3300/0 conn(0x7faa44154ff0 0x7faa440cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa492fc700 1 --2- 172.21.15.7:0/272161159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 0x7faa440c5c60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa492fc700 1 -- 172.21.15.7:0/272161159 >> 172.21.15.7:0/272161159 conn(0x7faa44066e40 msgr2=0x7faa44071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:24:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.566+0000 7faa492fc700 1 -- 172.21.15.7:0/272161159 shutdown_connections 2023-03-03T19:24:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.567+0000 7faa492fc700 1 -- 172.21.15.7:0/272161159 wait complete. 2023-03-03T19:24:47.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.568+0000 7faa492fc700 1 Processor -- start 2023-03-03T19:24:47.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.568+0000 7faa492fc700 1 -- start start 2023-03-03T19:24:47.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa492fc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa440c00e0 0x7faa44156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:47.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa492fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 0x7faa441ed690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:47.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa492fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa44154ff0 0x7faa440c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:47.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa492fc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa441edbd0 con 0x7faa44154ff0 2023-03-03T19:24:47.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa492fc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4414c5e0 con 0x7faa440c00e0 2023-03-03T19:24:47.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa492fc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa440c45b0 con 0x7faa440caa30 2023-03-03T19:24:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa44154ff0 0x7faa440c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa44154ff0 0x7faa440c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41446/0 (socket says 172.21.15.7:41446) 2023-03-03T19:24:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa437fe700 1 -- 172.21.15.7:0/933555745 learned_addr learned my addr 172.21.15.7:0/933555745 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:24:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa42ffd700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa440c00e0 0x7faa44156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.569+0000 7faa427fc700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 0x7faa441ed690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa437fe700 1 -- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 msgr2=0x7faa441ed690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa437fe700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 0x7faa441ed690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa437fe700 1 -- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa440c00e0 msgr2=0x7faa44156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa437fe700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa440c00e0 0x7faa44156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa437fe700 1 -- 172.21.15.7:0/933555745 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c009a70 con 0x7faa44154ff0 2023-03-03T19:24:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa437fe700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa44154ff0 0x7faa440c67c0 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00efc0 tx=0x7faa3c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa2b7fe700 1 -- 172.21.15.7:0/933555745 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c00ca90 con 0x7faa44154ff0 2023-03-03T19:24:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa2b7fe700 1 -- 172.21.15.7:0/933555745 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3c00f040 con 0x7faa44154ff0 2023-03-03T19:24:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa492fc700 1 -- 172.21.15.7:0/933555745 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa380094a0 con 0x7faa44154ff0 2023-03-03T19:24:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa2b7fe700 1 -- 172.21.15.7:0/933555745 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c00ce10 con 0x7faa44154ff0 2023-03-03T19:24:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.570+0000 7faa492fc700 1 -- 172.21.15.7:0/933555745 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa44154ad0 con 0x7faa44154ff0 2023-03-03T19:24:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.571+0000 7faa492fc700 1 -- 172.21.15.7:0/933555745 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa441547f0 con 0x7faa44154ff0 2023-03-03T19:24:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.572+0000 7faa2b7fe700 1 -- 172.21.15.7:0/933555745 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faa3c014480 con 0x7faa44154ff0 2023-03-03T19:24:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.572+0000 7faa2b7fe700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa300682f0 0x7faa3006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:24:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.572+0000 7faa2b7fe700 1 -- 172.21.15.7:0/933555745 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faa3c0868c0 con 0x7faa44154ff0 2023-03-03T19:24:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.572+0000 7faa42ffd700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa300682f0 0x7faa3006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:24:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.573+0000 7faa42ffd700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa300682f0 0x7faa3006a7a0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7faa380090b0 tx=0x7faa38008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:24:47.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.576+0000 7faa2b7fe700 1 -- 172.21.15.7:0/933555745 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faa3c052c00 con 0x7faa44154ff0 2023-03-03T19:24:47.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:24:47.646 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:24:47.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.818+0000 7faa492fc700 1 -- 172.21.15.7:0/933555745 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa440bf650 con 0x7faa44154ff0 2023-03-03T19:24:47.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.819+0000 7faa2b7fe700 1 -- 172.21.15.7:0/933555745 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faa3c009d70 con 0x7faa44154ff0 2023-03-03T19:24:47.820 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:24:47.820 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:24:47.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 -- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa300682f0 msgr2=0x7faa3006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:47.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa300682f0 0x7faa3006a7a0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7faa380090b0 tx=0x7faa38008030 comp rx=0 tx=0).stop 2023-03-03T19:24:47.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 -- 172.21.15.7:0/933555745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa44154ff0 msgr2=0x7faa440c67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:24:47.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa44154ff0 0x7faa440c67c0 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00efc0 tx=0x7faa3c00c3a0 comp rx=0 tx=0).stop 2023-03-03T19:24:47.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 -- 172.21.15.7:0/933555745 shutdown_connections 2023-03-03T19:24:47.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faa300682f0 0x7faa3006a7a0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:47.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa44154ff0 0x7faa440c67c0 unknown :-1 s=CLOSED pgs=3488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa440c00e0 0x7faa44156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 --2- 172.21.15.7:0/933555745 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faa440caa30 0x7faa441ed690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:24:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 -- 172.21.15.7:0/933555745 >> 172.21.15.7:0/933555745 conn(0x7faa44066e40 msgr2=0x7faa44152450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:24:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 -- 172.21.15.7:0/933555745 shutdown_connections 2023-03-03T19:24:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:24:47.821+0000 7faa492fc700 1 -- 172.21.15.7:0/933555745 wait complete. 2023-03-03T19:24:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:24:47.834 INFO:tasks.mds_thrash.fs.[b]:waiting for 102 secs before thrashing 2023-03-03T19:25:17.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:25:17.712 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:25:41.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:25:42.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.033+0000 7ff735206700 1 Processor -- start 2023-03-03T19:25:42.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.033+0000 7ff735206700 1 -- start start 2023-03-03T19:25:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff735206700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 0x7ff7300c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:25:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff735206700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff7300c00e0 0x7ff730154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:25:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff735206700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7300cb280 con 0x7ff7300cae00 2023-03-03T19:25:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff735206700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7300c7310 con 0x7ff7300c00e0 2023-03-03T19:25:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff735206700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7300bfd10 con 0x7ff7300caa30 2023-03-03T19:25:42.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff72e59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff7300c00e0 0x7ff730154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:25:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff72f59e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff7300cae00 0x7ff7300c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35466/0 (socket says 172.21.15.7:35466) 2023-03-03T19:25:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff72f59e700 1 -- 172.21.15.7:0/1046672851 learned_addr learned my addr 172.21.15.7:0/1046672851 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:25:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff72ed9d700 1 --2- 172.21.15.7:0/1046672851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 0x7ff7300c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:25:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff72e59c700 1 -- 172.21.15.7:0/1046672851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 msgr2=0x7ff7300c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:25:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff72e59c700 1 --2- 172.21.15.7:0/1046672851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 0x7ff7300c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:25:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff72e59c700 1 -- 172.21.15.7:0/1046672851 >> v1:172.21.15.7:6789/0 conn(0x7ff7300cae00 legacy=0x7ff7300c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:25:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff72dd9b700 1 -- 172.21.15.7:0/1046672851 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655809670 0 0) 0x7ff7300cb280 con 0x7ff7300cae00 2023-03-03T19:25:42.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff72e59c700 1 -- 172.21.15.7:0/1046672851 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7300bfa30 con 0x7ff7300c00e0 2023-03-03T19:25:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.034+0000 7ff72ed9d700 1 --2- 172.21.15.7:0/1046672851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 0x7ff7300c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:25:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff72e59c700 1 --2- 172.21.15.7:0/1046672851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 0x7ff730154ff0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7ff720009820 tx=0x7ff720009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44aad9df46e4fba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:25:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff72dd9b700 1 -- 172.21.15.7:0/1046672851 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff720005e30 con 0x7ff7300c00e0 2023-03-03T19:25:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff72dd9b700 1 -- 172.21.15.7:0/1046672851 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff720006030 con 0x7ff7300c00e0 2023-03-03T19:25:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff72dd9b700 1 -- 172.21.15.7:0/1046672851 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7200063e0 con 0x7ff7300c00e0 2023-03-03T19:25:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff735206700 1 -- 172.21.15.7:0/1046672851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 msgr2=0x7ff730154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:25:42.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff735206700 1 --2- 172.21.15.7:0/1046672851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 0x7ff730154ff0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7ff720009820 tx=0x7ff720009b30 comp rx=0 tx=0).stop 2023-03-03T19:25:42.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff735206700 1 -- 172.21.15.7:0/1046672851 shutdown_connections 2023-03-03T19:25:42.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff735206700 1 --2- 172.21.15.7:0/1046672851 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 0x7ff7300c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:25:42.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff735206700 1 --2- 172.21.15.7:0/1046672851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 0x7ff730154ff0 unknown :-1 s=CLOSED pgs=3490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:25:42.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff735206700 1 -- 172.21.15.7:0/1046672851 >> 172.21.15.7:0/1046672851 conn(0x7ff730066e40 msgr2=0x7ff730071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:25:42.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff735206700 1 -- 172.21.15.7:0/1046672851 shutdown_connections 2023-03-03T19:25:42.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.035+0000 7ff735206700 1 -- 172.21.15.7:0/1046672851 wait complete. 2023-03-03T19:25:42.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.037+0000 7ff735206700 1 Processor -- start 2023-03-03T19:25:42.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.037+0000 7ff735206700 1 -- start start 2023-03-03T19:25:42.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff735206700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 0x7ff7301ecf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:25:42.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff735206700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 0x7ff730157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:25:42.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff735206700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7300cae00 0x7ff7301584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:25:42.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff735206700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7300a9dc0 con 0x7ff7300c00e0 2023-03-03T19:25:42.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff735206700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7300aa010 con 0x7ff7300cae00 2023-03-03T19:25:42.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff735206700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7301ed7d0 con 0x7ff7300caa30 2023-03-03T19:25:42.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 0x7ff7301ecf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:25:42.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 0x7ff7301ecf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41488/0 (socket says 172.21.15.7:41488) 2023-03-03T19:25:42.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72ed9d700 1 -- 172.21.15.7:0/3223352486 learned_addr learned my addr 172.21.15.7:0/3223352486 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:25:42.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72e59c700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 0x7ff730157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:25:42.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72f59e700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7300cae00 0x7ff7301584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:25:42.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72ed9d700 1 -- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 msgr2=0x7ff730157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:25:42.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72ed9d700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 0x7ff730157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:25:42.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72ed9d700 1 -- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7300cae00 msgr2=0x7ff7301584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:25:42.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72ed9d700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7300cae00 0x7ff7301584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:25:42.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72ed9d700 1 -- 172.21.15.7:0/3223352486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff720009430 con 0x7ff7300c00e0 2023-03-03T19:25:42.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72e59c700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 0x7ff730157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:25:42.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72f59e700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7300cae00 0x7ff7301584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:25:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.038+0000 7ff72ed9d700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 0x7ff7301ecf00 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7ff72400c750 tx=0x7ff72400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:25:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.039+0000 7ff7177fe700 1 -- 172.21.15.7:0/3223352486 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724011800 con 0x7ff7300c00e0 2023-03-03T19:25:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.039+0000 7ff7177fe700 1 -- 172.21.15.7:0/3223352486 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff724011960 con 0x7ff7300c00e0 2023-03-03T19:25:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.039+0000 7ff735206700 1 -- 172.21.15.7:0/3223352486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7301547f0 con 0x7ff7300c00e0 2023-03-03T19:25:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.039+0000 7ff7177fe700 1 -- 172.21.15.7:0/3223352486 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724011cd0 con 0x7ff7300c00e0 2023-03-03T19:25:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.039+0000 7ff735206700 1 -- 172.21.15.7:0/3223352486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff730154510 con 0x7ff7300c00e0 2023-03-03T19:25:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.040+0000 7ff735206700 1 -- 172.21.15.7:0/3223352486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff730154ad0 con 0x7ff7300c00e0 2023-03-03T19:25:42.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.040+0000 7ff7177fe700 1 -- 172.21.15.7:0/3223352486 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff724020070 con 0x7ff7300c00e0 2023-03-03T19:25:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.040+0000 7ff7177fe700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff71c0682f0 0x7ff71c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:25:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.041+0000 7ff7177fe700 1 -- 172.21.15.7:0/3223352486 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff7240860e0 con 0x7ff7300c00e0 2023-03-03T19:25:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.041+0000 7ff72e59c700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff71c0682f0 0x7ff71c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:25:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.041+0000 7ff72e59c700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff71c0682f0 0x7ff71c06a7a0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7ff720013040 tx=0x7ff720008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:25:42.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.045+0000 7ff7177fe700 1 -- 172.21.15.7:0/3223352486 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff724052390 con 0x7ff7300c00e0 2023-03-03T19:25:42.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.283+0000 7ff735206700 1 -- 172.21.15.7:0/3223352486 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff730074bb0 con 0x7ff7300c00e0 2023-03-03T19:25:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.284+0000 7ff7177fe700 1 -- 172.21.15.7:0/3223352486 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff72401e090 con 0x7ff7300c00e0 2023-03-03T19:25:42.284 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:25:42.285 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:25:42.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 -- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff71c0682f0 msgr2=0x7ff71c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:25:42.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff71c0682f0 0x7ff71c06a7a0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7ff720013040 tx=0x7ff720008070 comp rx=0 tx=0).stop 2023-03-03T19:25:42.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 -- 172.21.15.7:0/3223352486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 msgr2=0x7ff7301ecf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:25:42.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 0x7ff7301ecf00 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7ff72400c750 tx=0x7ff72400cb10 comp rx=0 tx=0).stop 2023-03-03T19:25:42.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 -- 172.21.15.7:0/3223352486 shutdown_connections 2023-03-03T19:25:42.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff7300cae00 0x7ff7301584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:25:42.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff71c0682f0 0x7ff71c06a7a0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:25:42.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7300c00e0 0x7ff7301ecf00 unknown :-1 s=CLOSED pgs=3491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:25:42.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 --2- 172.21.15.7:0/3223352486 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff7300caa30 0x7ff730157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:25:42.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 -- 172.21.15.7:0/3223352486 >> 172.21.15.7:0/3223352486 conn(0x7ff730066e40 msgr2=0x7ff730152ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:25:42.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 -- 172.21.15.7:0/3223352486 shutdown_connections 2023-03-03T19:25:42.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:25:42.286+0000 7ff735206700 1 -- 172.21.15.7:0/3223352486 wait complete. 2023-03-03T19:25:42.288 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:25:42.298 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T19:25:47.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:25:47.791 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:26:17.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:26:17.889 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:26:29.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:26:29.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.975+0000 7f287e118700 1 Processor -- start 2023-03-03T19:26:29.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.975+0000 7f287e118700 1 -- start start 2023-03-03T19:26:29.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f287e118700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f287814f9d0 0x7f2878156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:29.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f287e118700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f287814fda0 0x7f2878071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:29.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f287e118700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28780abe20 0x7f28781529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:29.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f287e118700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28780c2f30 con 0x7f28780abe20 2023-03-03T19:26:29.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f287e118700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28780c31d0 con 0x7f287814f9d0 2023-03-03T19:26:29.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f287e118700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28781524a0 con 0x7f287814fda0 2023-03-03T19:26:29.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f2877fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28780abe20 0x7f28781529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:29.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f2877fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f28780abe20 0x7f28781529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41544/0 (socket says 172.21.15.7:41544) 2023-03-03T19:26:29.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f2877fff700 1 -- 172.21.15.7:0/1421736136 learned_addr learned my addr 172.21.15.7:0/1421736136 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:26:29.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f28777fe700 1 --2- 172.21.15.7:0/1421736136 >> v2:172.21.15.143:3300/0 conn(0x7f287814f9d0 0x7f2878156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:29.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f2876ffd700 1 --2- 172.21.15.7:0/1421736136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f287814fda0 0x7f2878071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:29.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f2877fff700 1 -- 172.21.15.7:0/1421736136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f287814fda0 msgr2=0x7f2878071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:26:29.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f2877fff700 1 --2- 172.21.15.7:0/1421736136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f287814fda0 0x7f2878071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:29.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f2877fff700 1 -- 172.21.15.7:0/1421736136 >> v2:172.21.15.143:3300/0 conn(0x7f287814f9d0 msgr2=0x7f2878156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:26:29.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f2877fff700 1 --2- 172.21.15.7:0/1421736136 >> v2:172.21.15.143:3300/0 conn(0x7f287814f9d0 0x7f2878156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:29.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f2877fff700 1 -- 172.21.15.7:0/1421736136 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28780c34b0 con 0x7f28780abe20 2023-03-03T19:26:29.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.976+0000 7f2877fff700 1 --2- 172.21.15.7:0/1421736136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28780abe20 0x7f28781529c0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7f28780710a0 tx=0x7f287000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51eef4ea766825ed server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:26:29.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f28767fc700 1 -- 172.21.15.7:0/1421736136 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287000e070 con 0x7f28780abe20 2023-03-03T19:26:29.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f28767fc700 1 -- 172.21.15.7:0/1421736136 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287000b910 con 0x7f28780abe20 2023-03-03T19:26:29.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f28767fc700 1 -- 172.21.15.7:0/1421736136 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287000bcc0 con 0x7f28780abe20 2023-03-03T19:26:29.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f287e118700 1 -- 172.21.15.7:0/1421736136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28780abe20 msgr2=0x7f28781529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:26:29.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f287e118700 1 --2- 172.21.15.7:0/1421736136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28780abe20 0x7f28781529c0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7f28780710a0 tx=0x7f287000b330 comp rx=0 tx=0).stop 2023-03-03T19:26:29.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f287e118700 1 -- 172.21.15.7:0/1421736136 shutdown_connections 2023-03-03T19:26:29.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f287e118700 1 --2- 172.21.15.7:0/1421736136 >> v2:172.21.15.143:3300/0 conn(0x7f287814f9d0 0x7f2878156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:29.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f287e118700 1 --2- 172.21.15.7:0/1421736136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f28780abe20 0x7f28781529c0 unknown :-1 s=CLOSED pgs=3492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:29.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f287e118700 1 --2- 172.21.15.7:0/1421736136 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f287814fda0 0x7f2878071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:29.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f287e118700 1 -- 172.21.15.7:0/1421736136 >> 172.21.15.7:0/1421736136 conn(0x7f28780c8900 msgr2=0x7f28780c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:26:29.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f287e118700 1 -- 172.21.15.7:0/1421736136 shutdown_connections 2023-03-03T19:26:29.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.977+0000 7f287e118700 1 -- 172.21.15.7:0/1421736136 wait complete. 2023-03-03T19:26:29.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.978+0000 7f287e118700 1 Processor -- start 2023-03-03T19:26:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.979+0000 7f287e118700 1 -- start start 2023-03-03T19:26:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.979+0000 7f287e118700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28780abe20 0x7f28780c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:29.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.979+0000 7f287e118700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f287814f9d0 0x7f287814d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.979+0000 7f287e118700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f287814fda0 0x7f2878149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.979+0000 7f287e118700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28781dd3b0 con 0x7f287814fda0 2023-03-03T19:26:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.979+0000 7f287e118700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28781ddc50 con 0x7f28780abe20 2023-03-03T19:26:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.979+0000 7f287e118700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28780c6780 con 0x7f287814f9d0 2023-03-03T19:26:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.979+0000 7f2877fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f287814fda0 0x7f2878149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f2877fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f287814fda0 0x7f2878149ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41550/0 (socket says 172.21.15.7:41550) 2023-03-03T19:26:29.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f2877fff700 1 -- 172.21.15.7:0/1732896973 learned_addr learned my addr 172.21.15.7:0/1732896973 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:26:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f28777fe700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28780abe20 0x7f28780c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f2876ffd700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f287814f9d0 0x7f287814d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f2877fff700 1 -- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f287814f9d0 msgr2=0x7f287814d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:26:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f2877fff700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f287814f9d0 0x7f287814d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f2877fff700 1 -- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28780abe20 msgr2=0x7f28780c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:26:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f2877fff700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28780abe20 0x7f28780c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:29.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f2877fff700 1 -- 172.21.15.7:0/1732896973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2870009b30 con 0x7f287814fda0 2023-03-03T19:26:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f2877fff700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f287814fda0 0x7f2878149ec0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7f2878072520 tx=0x7f2870004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:26:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f285ffff700 1 -- 172.21.15.7:0/1732896973 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28700099b0 con 0x7f287814fda0 2023-03-03T19:26:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f285ffff700 1 -- 172.21.15.7:0/1732896973 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287000ce80 con 0x7f287814fda0 2023-03-03T19:26:29.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f287e118700 1 -- 172.21.15.7:0/1732896973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28780ac350 con 0x7f287814fda0 2023-03-03T19:26:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f285ffff700 1 -- 172.21.15.7:0/1732896973 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870004490 con 0x7f287814fda0 2023-03-03T19:26:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.980+0000 7f287e118700 1 -- 172.21.15.7:0/1732896973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28780ab630 con 0x7f287814fda0 2023-03-03T19:26:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.981+0000 7f287e118700 1 -- 172.21.15.7:0/1732896973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28780c3e10 con 0x7f287814fda0 2023-03-03T19:26:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.982+0000 7f285ffff700 1 -- 172.21.15.7:0/1732896973 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f28700045f0 con 0x7f287814fda0 2023-03-03T19:26:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.982+0000 7f285ffff700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28640682f0 0x7f286406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.982+0000 7f285ffff700 1 -- 172.21.15.7:0/1732896973 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2870087360 con 0x7f287814fda0 2023-03-03T19:26:29.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.982+0000 7f28777fe700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28640682f0 0x7f286406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:29.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.983+0000 7f28777fe700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28640682f0 0x7f286406a7a0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f28781dc200 tx=0x7f2868009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:26:29.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:29.986+0000 7f285ffff700 1 -- 172.21.15.7:0/1732896973 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2870053730 con 0x7f287814fda0 2023-03-03T19:26:30.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.224+0000 7f287e118700 1 -- 172.21.15.7:0/1732896973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28780c60e0 con 0x7f287814fda0 2023-03-03T19:26:30.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.225+0000 7f285ffff700 1 -- 172.21.15.7:0/1732896973 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f287001f090 con 0x7f287814fda0 2023-03-03T19:26:30.226 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:26:30.226 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:26:30.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 -- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28640682f0 msgr2=0x7f286406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:26:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28640682f0 0x7f286406a7a0 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f28781dc200 tx=0x7f2868009210 comp rx=0 tx=0).stop 2023-03-03T19:26:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 -- 172.21.15.7:0/1732896973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f287814fda0 msgr2=0x7f2878149ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:26:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f287814fda0 0x7f2878149ec0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7f2878072520 tx=0x7f2870004ad0 comp rx=0 tx=0).stop 2023-03-03T19:26:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 -- 172.21.15.7:0/1732896973 shutdown_connections 2023-03-03T19:26:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f28640682f0 0x7f286406a7a0 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f287814fda0 0x7f2878149ec0 unknown :-1 s=CLOSED pgs=3493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28780abe20 0x7f28780c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 --2- 172.21.15.7:0/1732896973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f287814f9d0 0x7f287814d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 -- 172.21.15.7:0/1732896973 >> 172.21.15.7:0/1732896973 conn(0x7f28780c8900 msgr2=0x7f2878150200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:26:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 -- 172.21.15.7:0/1732896973 shutdown_connections 2023-03-03T19:26:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:30.227+0000 7f287e118700 1 -- 172.21.15.7:0/1732896973 wait complete. 2023-03-03T19:26:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:26:30.239 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-03-03T19:26:47.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:26:47.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:26:48.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:26:48.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.438+0000 7f2bd6b8d700 1 Processor -- start 2023-03-03T19:26:48.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.438+0000 7f2bd6b8d700 1 -- start start 2023-03-03T19:26:48.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bd6b8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 0x7f2bd00c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:48.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bd6b8d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2bd0071c30 0x7f2bd00beb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:48.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bd6b8d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2bd00bf0a0 0x7f2bd0155950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:48.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bd6b8d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd00713c0 con 0x7f2bd0071c30 2023-03-03T19:26:48.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bd6b8d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd0152c70 con 0x7f2bd00bf0a0 2023-03-03T19:26:48.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bd6b8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00c4930 con 0x7f2bd00aac00 2023-03-03T19:26:48.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bcffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2bd0071c30 0x7f2bd00beb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:48.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bcffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2bd0071c30 0x7f2bd00beb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41556/0 (socket says 172.21.15.7:41556) 2023-03-03T19:26:48.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bcffff700 1 -- 172.21.15.7:0/2818826290 learned_addr learned my addr 172.21.15.7:0/2818826290 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:26:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bd512a700 1 --2- 172.21.15.7:0/2818826290 >> v2:172.21.15.143:3300/0 conn(0x7f2bd00bf0a0 0x7f2bd0155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bd4929700 1 --2- 172.21.15.7:0/2818826290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 0x7f2bd00c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bcffff700 1 -- 172.21.15.7:0/2818826290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 msgr2=0x7f2bd00c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:26:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bcffff700 1 --2- 172.21.15.7:0/2818826290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 0x7f2bd00c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bcffff700 1 -- 172.21.15.7:0/2818826290 >> v2:172.21.15.143:3300/0 conn(0x7f2bd00bf0a0 msgr2=0x7f2bd0155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:26:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bcffff700 1 --2- 172.21.15.7:0/2818826290 >> v2:172.21.15.143:3300/0 conn(0x7f2bd00bf0a0 0x7f2bd0155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bcffff700 1 -- 172.21.15.7:0/2818826290 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd0152360 con 0x7f2bd0071c30 2023-03-03T19:26:48.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bd512a700 1 --2- 172.21.15.7:0/2818826290 >> v2:172.21.15.143:3300/0 conn(0x7f2bd00bf0a0 0x7f2bd0155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:26:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bcffff700 1 --2- 172.21.15.7:0/2818826290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 0x7f2bd00beb60 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f2bd00719e0 tx=0x7f2bbc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8d479a0a7b3497 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:26:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.439+0000 7f2bcf7fe700 1 -- 172.21.15.7:0/2818826290 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc00b870 con 0x7f2bd0071c30 2023-03-03T19:26:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bcf7fe700 1 -- 172.21.15.7:0/2818826290 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc00b9d0 con 0x7f2bd0071c30 2023-03-03T19:26:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bcf7fe700 1 -- 172.21.15.7:0/2818826290 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc00bd60 con 0x7f2bd0071c30 2023-03-03T19:26:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2818826290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 msgr2=0x7f2bd00beb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:26:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bd6b8d700 1 --2- 172.21.15.7:0/2818826290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 0x7f2bd00beb60 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f2bd00719e0 tx=0x7f2bbc009bd0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2818826290 shutdown_connections 2023-03-03T19:26:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bd6b8d700 1 --2- 172.21.15.7:0/2818826290 >> v2:172.21.15.143:3300/0 conn(0x7f2bd00bf0a0 0x7f2bd0155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bd6b8d700 1 --2- 172.21.15.7:0/2818826290 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 0x7f2bd00c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bd6b8d700 1 --2- 172.21.15.7:0/2818826290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 0x7f2bd00beb60 unknown :-1 s=CLOSED pgs=3494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2818826290 >> 172.21.15.7:0/2818826290 conn(0x7f2bd00736f0 msgr2=0x7f2bd0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:26:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2818826290 shutdown_connections 2023-03-03T19:26:48.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.440+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2818826290 wait complete. 2023-03-03T19:26:48.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.441+0000 7f2bd6b8d700 1 Processor -- start 2023-03-03T19:26:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.441+0000 7f2bd6b8d700 1 -- start start 2023-03-03T19:26:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.442+0000 7f2bd6b8d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 0x7f2bd01ed570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.442+0000 7f2bd6b8d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 0x7f2bd01edab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.442+0000 7f2bd6b8d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bd00bf0a0 0x7f2bd00c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.442+0000 7f2bd6b8d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd014f790 con 0x7f2bd0071c30 2023-03-03T19:26:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.442+0000 7f2bd6b8d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd01edff0 con 0x7f2bd00bf0a0 2023-03-03T19:26:48.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.442+0000 7f2bd6b8d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd00c7e40 con 0x7f2bd00aac00 2023-03-03T19:26:48.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.442+0000 7f2bd4929700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 0x7f2bd01ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:48.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.442+0000 7f2bd4929700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 0x7f2bd01ed570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41562/0 (socket says 172.21.15.7:41562) 2023-03-03T19:26:48.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.442+0000 7f2bd4929700 1 -- 172.21.15.7:0/2147518661 learned_addr learned my addr 172.21.15.7:0/2147518661 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:26:48.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bd512a700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bd00bf0a0 0x7f2bd00c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:48.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bcffff700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 0x7f2bd01edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:48.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bd4929700 1 -- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 msgr2=0x7f2bd01edab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:26:48.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bd4929700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 0x7f2bd01edab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bd4929700 1 -- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bd00bf0a0 msgr2=0x7f2bd00c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:26:48.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bd4929700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bd00bf0a0 0x7f2bd00c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bd4929700 1 -- 172.21.15.7:0/2147518661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc009550 con 0x7f2bd0071c30 2023-03-03T19:26:48.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bcffff700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 0x7f2bd01edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:26:48.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bd512a700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bd00bf0a0 0x7f2bd00c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:26:48.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bd4929700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 0x7f2bd01ed570 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400d830 tx=0x7f2bc400dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:26:48.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bcd7fa700 1 -- 172.21.15.7:0/2147518661 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc40125f0 con 0x7f2bd0071c30 2023-03-03T19:26:48.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bcd7fa700 1 -- 172.21.15.7:0/2147518661 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc4012750 con 0x7f2bd0071c30 2023-03-03T19:26:48.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2147518661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd0152620 con 0x7f2bd0071c30 2023-03-03T19:26:48.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bcd7fa700 1 -- 172.21.15.7:0/2147518661 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4012ae0 con 0x7f2bd0071c30 2023-03-03T19:26:48.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.443+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2147518661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd00c4b80 con 0x7f2bd0071c30 2023-03-03T19:26:48.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.444+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2147518661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd0152900 con 0x7f2bd0071c30 2023-03-03T19:26:48.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.445+0000 7f2bcd7fa700 1 -- 172.21.15.7:0/2147518661 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2bc4012c40 con 0x7f2bd0071c30 2023-03-03T19:26:48.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.445+0000 7f2bcd7fa700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bb40682f0 0x7f2bb406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:26:48.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.445+0000 7f2bcd7fa700 1 -- 172.21.15.7:0/2147518661 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2bc40865b0 con 0x7f2bd0071c30 2023-03-03T19:26:48.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.445+0000 7f2bcffff700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bb40682f0 0x7f2bb406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:26:48.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.446+0000 7f2bcffff700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bb40682f0 0x7f2bb406a7a0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc000c00 tx=0x7f2bbc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:26:48.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.449+0000 7f2bcd7fa700 1 -- 172.21.15.7:0/2147518661 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2bc4052950 con 0x7f2bd0071c30 2023-03-03T19:26:48.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.689+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2147518661 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bd0074bb0 con 0x7f2bd0071c30 2023-03-03T19:26:48.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.690+0000 7f2bcd7fa700 1 -- 172.21.15.7:0/2147518661 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2bc4012ed0 con 0x7f2bd0071c30 2023-03-03T19:26:48.690 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:26:48.690 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:26:48.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bb40682f0 msgr2=0x7f2bb406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:26:48.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bb40682f0 0x7f2bb406a7a0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc000c00 tx=0x7f2bbc008030 comp rx=0 tx=0).stop 2023-03-03T19:26:48.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2147518661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 msgr2=0x7f2bd01ed570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:26:48.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 0x7f2bd01ed570 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400d830 tx=0x7f2bc400dbf0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2147518661 shutdown_connections 2023-03-03T19:26:48.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bd00bf0a0 0x7f2bd00c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2bb40682f0 0x7f2bb406a7a0 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2bd0071c30 0x7f2bd01ed570 unknown :-1 s=CLOSED pgs=3495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 --2- 172.21.15.7:0/2147518661 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2bd00aac00 0x7f2bd01edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:26:48.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2147518661 >> 172.21.15.7:0/2147518661 conn(0x7f2bd00736f0 msgr2=0x7f2bd014c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:26:48.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2147518661 shutdown_connections 2023-03-03T19:26:48.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:26:48.692+0000 7f2bd6b8d700 1 -- 172.21.15.7:0/2147518661 wait complete. 2023-03-03T19:26:48.694 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:26:48.704 INFO:tasks.mds_thrash.fs.[a]:waiting for 111 secs before thrashing 2023-03-03T19:27:18.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:27:18.042 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:27:48.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:27:48.103 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:28:18.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:28:18.175 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:28:19.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:28:19.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.378+0000 7f8d32902700 1 Processor -- start 2023-03-03T19:28:19.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.378+0000 7f8d32902700 1 -- start start 2023-03-03T19:28:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.379+0000 7f8d32902700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d2c14b0d0 0x7f8d2c071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.379+0000 7f8d32902700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8d2c071d40 0x7f8d2c158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.379+0000 7f8d32902700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c0a9c20 con 0x7f8d2c0aad20 2023-03-03T19:28:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.379+0000 7f8d32902700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2c154f10 con 0x7f8d2c071d40 2023-03-03T19:28:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.379+0000 7f8d32902700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0ac3c0 con 0x7f8d2c14b0d0 2023-03-03T19:28:19.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.379+0000 7f8d2b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8d2c0aad20 0x7f8d2c157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35630/0 (socket says 172.21.15.7:35630) 2023-03-03T19:28:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.379+0000 7f8d2b7fe700 1 -- 172.21.15.7:0/69993122 learned_addr learned my addr 172.21.15.7:0/69993122 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:28:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2bfff700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d2c14b0d0 0x7f8d2c071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d30e9f700 1 --2- 172.21.15.7:0/69993122 >> v2:172.21.15.143:3300/0 conn(0x7f8d2c071d40 0x7f8d2c158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909239306 0 0) 0x7f8d2c0a9c20 con 0x7f8d2c0aad20 2023-03-03T19:28:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d140035b0 con 0x7f8d2c0aad20 2023-03-03T19:28:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3718784932 0 0) 0x7f8d140035b0 con 0x7f8d2c0aad20 2023-03-03T19:28:19.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d2c14b0d0 msgr2=0x7f8d2c071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:28:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d2c14b0d0 0x7f8d2c071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 >> v2:172.21.15.143:3300/0 conn(0x7f8d2c071d40 msgr2=0x7f8d2c158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:28:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 --2- 172.21.15.7:0/69993122 >> v2:172.21.15.143:3300/0 conn(0x7f8d2c071d40 0x7f8d2c158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c06fc70 con 0x7f8d2c0aad20 2023-03-03T19:28:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f8d20001670 con 0x7f8d2c0aad20 2023-03-03T19:28:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 >> v1:172.21.15.7:6789/0 conn(0x7f8d2c0aad20 legacy=0x7f8d2c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d14003a70 0x7f8d14003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:19.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d14004850 0x7f8d14004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d2c14b0d0 0x7f8d14005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c154f10 con 0x7f8d14004850 2023-03-03T19:28:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0ac3c0 con 0x7f8d14003a70 2023-03-03T19:28:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1400ba40 con 0x7f8d2c14b0d0 2023-03-03T19:28:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2b7fe700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d14004850 0x7f8d14004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2bfff700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d14003a70 0x7f8d14003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d30e9f700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d2c14b0d0 0x7f8d14005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:19.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2b7fe700 1 -- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d2c14b0d0 msgr2=0x7f8d14005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:28:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2b7fe700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d2c14b0d0 0x7f8d14005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2b7fe700 1 -- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d14003a70 msgr2=0x7f8d14003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:28:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2b7fe700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d14003a70 0x7f8d14003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.380+0000 7f8d2b7fe700 1 -- 172.21.15.7:0/69993122 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d140035b0 con 0x7f8d14004850 2023-03-03T19:28:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d2b7fe700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d14004850 0x7f8d14004cc0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000c370 tx=0x7f8d14005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fdeb99eaf9a691 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:28:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20007c60 con 0x7f8d14004850 2023-03-03T19:28:19.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d20005960 con 0x7f8d14004850 2023-03-03T19:28:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d2affd700 1 -- 172.21.15.7:0/69993122 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d20005d10 con 0x7f8d14004850 2023-03-03T19:28:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d32902700 1 -- 172.21.15.7:0/69993122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d14004850 msgr2=0x7f8d14004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d32902700 1 --2- 172.21.15.7:0/69993122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d14004850 0x7f8d14004cc0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000c370 tx=0x7f8d14005880 comp rx=0 tx=0).stop 2023-03-03T19:28:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d30e9f700 1 -- 172.21.15.7:0/69993122 reap_dead start 2023-03-03T19:28:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d32902700 1 -- 172.21.15.7:0/69993122 shutdown_connections 2023-03-03T19:28:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d32902700 1 -- 172.21.15.7:0/69993122 >> 172.21.15.7:0/69993122 conn(0x7f8d2c0c8900 msgr2=0x7f8d2c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:28:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d32902700 1 -- 172.21.15.7:0/69993122 shutdown_connections 2023-03-03T19:28:19.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.381+0000 7f8d32902700 1 -- 172.21.15.7:0/69993122 wait complete. 2023-03-03T19:28:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.383+0000 7f8d32902700 1 Processor -- start 2023-03-03T19:28:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.383+0000 7f8d32902700 1 -- start start 2023-03-03T19:28:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.383+0000 7f8d32902700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d14004850 0x7f8d2c1e9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.383+0000 7f8d32902700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d2c1478a0 0x7f8d2c153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.383+0000 7f8d32902700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d2c155880 0x7f8d2c145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.383+0000 7f8d32902700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c147410 con 0x7f8d2c155880 2023-03-03T19:28:19.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.383+0000 7f8d32902700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c147cf0 con 0x7f8d2c1478a0 2023-03-03T19:28:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.383+0000 7f8d32902700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c146800 con 0x7f8d14004850 2023-03-03T19:28:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d30e9f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d2c155880 0x7f8d2c145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d30e9f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d2c155880 0x7f8d2c145560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41664/0 (socket says 172.21.15.7:41664) 2023-03-03T19:28:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d30e9f700 1 -- 172.21.15.7:0/3746587905 learned_addr learned my addr 172.21.15.7:0/3746587905 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:28:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d2bfff700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d14004850 0x7f8d2c1e9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d2b7fe700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d2c1478a0 0x7f8d2c153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d30e9f700 1 -- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d14004850 msgr2=0x7f8d2c1e9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d30e9f700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d14004850 0x7f8d2c1e9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d30e9f700 1 -- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d2c1478a0 msgr2=0x7f8d2c153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d30e9f700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d2c1478a0 0x7f8d2c153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d30e9f700 1 -- 172.21.15.7:0/3746587905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c06f6b0 con 0x7f8d2c155880 2023-03-03T19:28:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d2bfff700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d14004850 0x7f8d2c1e9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:28:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d30e9f700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d2c155880 0x7f8d2c145560 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c070930 tx=0x7f8d2400f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:28:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d28ff9700 1 -- 172.21.15.7:0/3746587905 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2400e040 con 0x7f8d2c155880 2023-03-03T19:28:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d28ff9700 1 -- 172.21.15.7:0/3746587905 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2400f040 con 0x7f8d2c155880 2023-03-03T19:28:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d28ff9700 1 -- 172.21.15.7:0/3746587905 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d24015430 con 0x7f8d2c155880 2023-03-03T19:28:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d32902700 1 -- 172.21.15.7:0/3746587905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c155160 con 0x7f8d2c155880 2023-03-03T19:28:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.384+0000 7f8d32902700 1 -- 172.21.15.7:0/3746587905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c06f990 con 0x7f8d2c155880 2023-03-03T19:28:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.385+0000 7f8d32902700 1 -- 172.21.15.7:0/3746587905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c1476f0 con 0x7f8d2c155880 2023-03-03T19:28:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.386+0000 7f8d28ff9700 1 -- 172.21.15.7:0/3746587905 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8d24026070 con 0x7f8d2c155880 2023-03-03T19:28:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.386+0000 7f8d28ff9700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d18068320 0x7f8d1806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.386+0000 7f8d28ff9700 1 -- 172.21.15.7:0/3746587905 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8d240868f0 con 0x7f8d2c155880 2023-03-03T19:28:19.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.386+0000 7f8d2bfff700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d18068320 0x7f8d1806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:19.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.387+0000 7f8d2bfff700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d18068320 0x7f8d1806a7d0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c006dd0 tx=0x7f8d1c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:28:19.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.391+0000 7f8d28ff9700 1 -- 172.21.15.7:0/3746587905 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8d24052c90 con 0x7f8d2c155880 2023-03-03T19:28:19.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.626+0000 7f8d32902700 1 -- 172.21.15.7:0/3746587905 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d2c147550 con 0x7f8d2c155880 2023-03-03T19:28:19.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.627+0000 7f8d28ff9700 1 -- 172.21.15.7:0/3746587905 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8d2c147550 con 0x7f8d2c155880 2023-03-03T19:28:19.628 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:28:19.629 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:28:19.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 -- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d18068320 msgr2=0x7f8d1806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:19.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d18068320 0x7f8d1806a7d0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c006dd0 tx=0x7f8d1c008040 comp rx=0 tx=0).stop 2023-03-03T19:28:19.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 -- 172.21.15.7:0/3746587905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d2c155880 msgr2=0x7f8d2c145560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:19.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d2c155880 0x7f8d2c145560 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c070930 tx=0x7f8d2400f490 comp rx=0 tx=0).stop 2023-03-03T19:28:19.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 -- 172.21.15.7:0/3746587905 shutdown_connections 2023-03-03T19:28:19.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8d18068320 0x7f8d1806a7d0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:19.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d2c155880 0x7f8d2c145560 unknown :-1 s=CLOSED pgs=3498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:19.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8d14004850 0x7f8d2c1e9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:19.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 --2- 172.21.15.7:0/3746587905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d2c1478a0 0x7f8d2c153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:19.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 -- 172.21.15.7:0/3746587905 >> 172.21.15.7:0/3746587905 conn(0x7f8d2c0c8900 msgr2=0x7f8d2c1e0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:28:19.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 -- 172.21.15.7:0/3746587905 shutdown_connections 2023-03-03T19:28:19.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:19.629+0000 7f8d32902700 1 -- 172.21.15.7:0/3746587905 wait complete. 2023-03-03T19:28:19.632 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:28:19.643 INFO:tasks.mds_thrash.fs.[b]:waiting for 80 secs before thrashing 2023-03-03T19:28:39.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:28:39.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.844+0000 7f37ec4fe700 1 Processor -- start 2023-03-03T19:28:39.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.844+0000 7f37ec4fe700 1 -- start start 2023-03-03T19:28:39.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37ec4fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f37e40c6010 0x7f37e40c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:39.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37ec4fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f37e40c4fc0 0x7f37e406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:39.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37ec4fe700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37e414ebe0 con 0x7f37e40c17f0 2023-03-03T19:28:39.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37ec4fe700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e4152100 con 0x7f37e40c6010 2023-03-03T19:28:39.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37ec4fe700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e40c2e70 con 0x7f37e40c4fc0 2023-03-03T19:28:39.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37ea29a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f37e40c6010 0x7f37e40c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:39.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37e9a99700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f37e40c17f0 0x7f37e406d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35658/0 (socket says 172.21.15.7:35658) 2023-03-03T19:28:39.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37e9a99700 1 -- 172.21.15.7:0/79483810 learned_addr learned my addr 172.21.15.7:0/79483810 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:28:39.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37eaa9b700 1 --2- 172.21.15.7:0/79483810 >> v2:172.21.15.143:3300/0 conn(0x7f37e40c4fc0 0x7f37e406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:39.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37e9298700 1 -- 172.21.15.7:0/79483810 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430932035 0 0) 0x7f37e414ebe0 con 0x7f37e40c17f0 2023-03-03T19:28:39.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37ea29a700 1 -- 172.21.15.7:0/79483810 >> v2:172.21.15.143:3300/0 conn(0x7f37e40c4fc0 msgr2=0x7f37e406ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:28:39.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37ea29a700 1 --2- 172.21.15.7:0/79483810 >> v2:172.21.15.143:3300/0 conn(0x7f37e40c4fc0 0x7f37e406ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:39.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37ea29a700 1 -- 172.21.15.7:0/79483810 >> v1:172.21.15.7:6789/0 conn(0x7f37e40c17f0 legacy=0x7f37e406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:39.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.845+0000 7f37ea29a700 1 -- 172.21.15.7:0/79483810 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40aa680 con 0x7f37e40c6010 2023-03-03T19:28:39.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.846+0000 7f37ea29a700 1 --2- 172.21.15.7:0/79483810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 0x7f37e40c4a80 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7f37e406f490 tx=0x7f37d4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d08140d669a0a04 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:28:39.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.846+0000 7f37e9298700 1 -- 172.21.15.7:0/79483810 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4011830 con 0x7f37e40c6010 2023-03-03T19:28:39.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.846+0000 7f37e9298700 1 -- 172.21.15.7:0/79483810 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d4011990 con 0x7f37e40c6010 2023-03-03T19:28:39.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.846+0000 7f37e9298700 1 -- 172.21.15.7:0/79483810 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4011d20 con 0x7f37e40c6010 2023-03-03T19:28:39.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.846+0000 7f37ec4fe700 1 -- 172.21.15.7:0/79483810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 msgr2=0x7f37e40c4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:39.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.846+0000 7f37ec4fe700 1 --2- 172.21.15.7:0/79483810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 0x7f37e40c4a80 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7f37e406f490 tx=0x7f37d4009b20 comp rx=0 tx=0).stop 2023-03-03T19:28:39.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.846+0000 7f37ec4fe700 1 -- 172.21.15.7:0/79483810 shutdown_connections 2023-03-03T19:28:39.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.846+0000 7f37ec4fe700 1 --2- 172.21.15.7:0/79483810 >> v2:172.21.15.143:3300/0 conn(0x7f37e40c4fc0 0x7f37e406ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:39.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.846+0000 7f37ec4fe700 1 --2- 172.21.15.7:0/79483810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 0x7f37e40c4a80 unknown :-1 s=CLOSED pgs=3500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:39.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.846+0000 7f37ec4fe700 1 -- 172.21.15.7:0/79483810 >> 172.21.15.7:0/79483810 conn(0x7f37e40a9430 msgr2=0x7f37e40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:28:39.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.847+0000 7f37ec4fe700 1 -- 172.21.15.7:0/79483810 shutdown_connections 2023-03-03T19:28:39.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.847+0000 7f37ec4fe700 1 -- 172.21.15.7:0/79483810 wait complete. 2023-03-03T19:28:39.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.848+0000 7f37ec4fe700 1 Processor -- start 2023-03-03T19:28:39.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.848+0000 7f37ec4fe700 1 -- start start 2023-03-03T19:28:39.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.848+0000 7f37ec4fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37e40c17f0 0x7f37e40c0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:39.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.848+0000 7f37ec4fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37e40c4fc0 0x7f37e41db4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:39.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.848+0000 7f37ec4fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 0x7f37e414df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:39.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.848+0000 7f37ec4fe700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4153dd0 con 0x7f37e40c6010 2023-03-03T19:28:39.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.848+0000 7f37ec4fe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e40a7f90 con 0x7f37e40c4fc0 2023-03-03T19:28:39.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.848+0000 7f37ec4fe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e41da7b0 con 0x7f37e40c17f0 2023-03-03T19:28:39.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37eaa9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 0x7f37e414df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:39.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37eaa9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 0x7f37e414df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41688/0 (socket says 172.21.15.7:41688) 2023-03-03T19:28:39.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37eaa9b700 1 -- 172.21.15.7:0/2380987031 learned_addr learned my addr 172.21.15.7:0/2380987031 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:28:39.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37e9a99700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37e40c4fc0 0x7f37e41db4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:39.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37ea29a700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37e40c17f0 0x7f37e40c0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:39.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37eaa9b700 1 -- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37e40c17f0 msgr2=0x7f37e40c0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:39.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37eaa9b700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37e40c17f0 0x7f37e40c0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:39.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37eaa9b700 1 -- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37e40c4fc0 msgr2=0x7f37e41db4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:39.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37eaa9b700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37e40c4fc0 0x7f37e41db4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:39.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37eaa9b700 1 -- 172.21.15.7:0/2380987031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d40094a0 con 0x7f37e40c6010 2023-03-03T19:28:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37eaa9b700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 0x7f37e414df00 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f37dc007e40 tx=0x7f37dc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:28:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37daffd700 1 -- 172.21.15.7:0/2380987031 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00eea0 con 0x7f37e40c6010 2023-03-03T19:28:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37daffd700 1 -- 172.21.15.7:0/2380987031 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc014420 con 0x7f37e40c6010 2023-03-03T19:28:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37ec4fe700 1 -- 172.21.15.7:0/2380987031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e40c2b00 con 0x7f37e40c6010 2023-03-03T19:28:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37daffd700 1 -- 172.21.15.7:0/2380987031 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0147b0 con 0x7f37e40c6010 2023-03-03T19:28:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.849+0000 7f37ec4fe700 1 -- 172.21.15.7:0/2380987031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e40aa960 con 0x7f37e40c6010 2023-03-03T19:28:39.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.850+0000 7f37ec4fe700 1 -- 172.21.15.7:0/2380987031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e40c3300 con 0x7f37e40c6010 2023-03-03T19:28:39.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.851+0000 7f37daffd700 1 -- 172.21.15.7:0/2380987031 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f37dc012050 con 0x7f37e40c6010 2023-03-03T19:28:39.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.851+0000 7f37daffd700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37c80682f0 0x7f37c806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:28:39.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.851+0000 7f37daffd700 1 -- 172.21.15.7:0/2380987031 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f37dc086820 con 0x7f37e40c6010 2023-03-03T19:28:39.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.852+0000 7f37ea29a700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37c80682f0 0x7f37c806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:28:39.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.852+0000 7f37ea29a700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37c80682f0 0x7f37c806a7a0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f37d40090b0 tx=0x7f37d401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:28:39.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:39.856+0000 7f37daffd700 1 -- 172.21.15.7:0/2380987031 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f37dc052bc0 con 0x7f37e40c6010 2023-03-03T19:28:40.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.097+0000 7f37ec4fe700 1 -- 172.21.15.7:0/2380987031 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37e4074bb0 con 0x7f37e40c6010 2023-03-03T19:28:40.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.098+0000 7f37daffd700 1 -- 172.21.15.7:0/2380987031 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f37dc0561e0 con 0x7f37e40c6010 2023-03-03T19:28:40.099 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:28:40.099 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:28:40.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 -- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37c80682f0 msgr2=0x7f37c806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:40.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37c80682f0 0x7f37c806a7a0 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f37d40090b0 tx=0x7f37d401c040 comp rx=0 tx=0).stop 2023-03-03T19:28:40.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 -- 172.21.15.7:0/2380987031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 msgr2=0x7f37e414df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:28:40.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 0x7f37e414df00 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f37dc007e40 tx=0x7f37dc00e3c0 comp rx=0 tx=0).stop 2023-03-03T19:28:40.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 -- 172.21.15.7:0/2380987031 shutdown_connections 2023-03-03T19:28:40.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f37c80682f0 0x7f37c806a7a0 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:40.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e40c6010 0x7f37e414df00 unknown :-1 s=CLOSED pgs=3501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:40.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f37e40c17f0 0x7f37e40c0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:40.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 --2- 172.21.15.7:0/2380987031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37e40c4fc0 0x7f37e41db4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:28:40.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 -- 172.21.15.7:0/2380987031 >> 172.21.15.7:0/2380987031 conn(0x7f37e40a9430 msgr2=0x7f37e40c1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:28:40.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 -- 172.21.15.7:0/2380987031 shutdown_connections 2023-03-03T19:28:40.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:28:40.100+0000 7f37ec4fe700 1 -- 172.21.15.7:0/2380987031 wait complete. 2023-03-03T19:28:40.103 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:28:40.113 INFO:tasks.mds_thrash.fs.[a]:waiting for 25 secs before thrashing 2023-03-03T19:28:48.236 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:28:48.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:29:05.114 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:29:05.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.254+0000 7fcbef2ef700 1 Processor -- start 2023-03-03T19:29:05.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.254+0000 7fcbef2ef700 1 -- start start 2023-03-03T19:29:05.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbef2ef700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcbe8071990 0x7fcbe8071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:05.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbef2ef700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcbe814b460 0x7fcbe80722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:05.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbef2ef700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe8070360 con 0x7fcbe8151c00 2023-03-03T19:29:05.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbef2ef700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe81521a0 con 0x7fcbe814b460 2023-03-03T19:29:05.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbef2ef700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe806a5e0 con 0x7fcbe8071990 2023-03-03T19:29:05.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbec88a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcbe814b460 0x7fcbe80722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:05.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbec88a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcbe814b460 0x7fcbe80722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41716/0 (socket says 172.21.15.7:41716) 2023-03-03T19:29:05.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbec88a700 1 -- 172.21.15.7:0/4183976267 learned_addr learned my addr 172.21.15.7:0/4183976267 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:29:05.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbed08b700 1 --2- 172.21.15.7:0/4183976267 >> v2:172.21.15.143:3300/0 conn(0x7fcbe8071990 0x7fcbe8071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:05.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbec88a700 1 -- 172.21.15.7:0/4183976267 >> v2:172.21.15.143:3300/0 conn(0x7fcbe8071990 msgr2=0x7fcbe8071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:29:05.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbec88a700 1 --2- 172.21.15.7:0/4183976267 >> v2:172.21.15.143:3300/0 conn(0x7fcbe8071990 0x7fcbe8071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:05.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbec88a700 1 -- 172.21.15.7:0/4183976267 >> v1:172.21.15.143:6789/0 conn(0x7fcbe8151c00 legacy=0x7fcbe806aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:05.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbec88a700 1 -- 172.21.15.7:0/4183976267 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80c38b0 con 0x7fcbe814b460 2023-03-03T19:29:05.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.255+0000 7fcbec88a700 1 --2- 172.21.15.7:0/4183976267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe814b460 0x7fcbe80722a0 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80c1de0 tx=0x7fcbe4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76ee70ca64deea4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:29:05.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbdffff700 1 -- 172.21.15.7:0/4183976267 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe400b870 con 0x7fcbe814b460 2023-03-03T19:29:05.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbdffff700 1 -- 172.21.15.7:0/4183976267 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe400b9d0 con 0x7fcbe814b460 2023-03-03T19:29:05.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbdffff700 1 -- 172.21.15.7:0/4183976267 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe400bd60 con 0x7fcbe814b460 2023-03-03T19:29:05.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbef2ef700 1 -- 172.21.15.7:0/4183976267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe814b460 msgr2=0x7fcbe80722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:05.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbef2ef700 1 --2- 172.21.15.7:0/4183976267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe814b460 0x7fcbe80722a0 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80c1de0 tx=0x7fcbe4009bd0 comp rx=0 tx=0).stop 2023-03-03T19:29:05.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbef2ef700 1 -- 172.21.15.7:0/4183976267 shutdown_connections 2023-03-03T19:29:05.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbef2ef700 1 --2- 172.21.15.7:0/4183976267 >> v2:172.21.15.143:3300/0 conn(0x7fcbe8071990 0x7fcbe8071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:05.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbef2ef700 1 --2- 172.21.15.7:0/4183976267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe814b460 0x7fcbe80722a0 unknown :-1 s=CLOSED pgs=3502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:05.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbef2ef700 1 -- 172.21.15.7:0/4183976267 >> 172.21.15.7:0/4183976267 conn(0x7fcbe80a94d0 msgr2=0x7fcbe80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:29:05.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbef2ef700 1 -- 172.21.15.7:0/4183976267 shutdown_connections 2023-03-03T19:29:05.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.256+0000 7fcbef2ef700 1 -- 172.21.15.7:0/4183976267 wait complete. 2023-03-03T19:29:05.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.257+0000 7fcbef2ef700 1 Processor -- start 2023-03-03T19:29:05.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.257+0000 7fcbef2ef700 1 -- start start 2023-03-03T19:29:05.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbef2ef700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe8071990 0x7fcbe8152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:05.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbef2ef700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe8151c00 0x7fcbe81428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:05.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbef2ef700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe8154cf0 0x7fcbe813e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:05.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbef2ef700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe8152da0 con 0x7fcbe8151c00 2023-03-03T19:29:05.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbef2ef700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81eb620 con 0x7fcbe8071990 2023-03-03T19:29:05.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbef2ef700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe81eb830 con 0x7fcbe8154cf0 2023-03-03T19:29:05.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbec88a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe8151c00 0x7fcbe81428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:05.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbec88a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe8151c00 0x7fcbe81428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41722/0 (socket says 172.21.15.7:41722) 2023-03-03T19:29:05.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbec88a700 1 -- 172.21.15.7:0/1663148070 learned_addr learned my addr 172.21.15.7:0/1663148070 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:29:05.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbed08b700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe8071990 0x7fcbe8152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:05.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.258+0000 7fcbed88c700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe8154cf0 0x7fcbe813e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:05.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbec88a700 1 -- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe8154cf0 msgr2=0x7fcbe813e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:05.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbec88a700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe8154cf0 0x7fcbe813e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:05.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbec88a700 1 -- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe8071990 msgr2=0x7fcbe8152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:05.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbec88a700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe8071990 0x7fcbe8152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:05.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbec88a700 1 -- 172.21.15.7:0/1663148070 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe4009550 con 0x7fcbe8151c00 2023-03-03T19:29:05.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbed88c700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe8154cf0 0x7fcbe813e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:29:05.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbec88a700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe8151c00 0x7fcbe81428b0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7fcbe806bcf0 tx=0x7fcbe4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:29:05.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbddffb700 1 -- 172.21.15.7:0/1663148070 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4004310 con 0x7fcbe8151c00 2023-03-03T19:29:05.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbddffb700 1 -- 172.21.15.7:0/1663148070 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe401f070 con 0x7fcbe8151c00 2023-03-03T19:29:05.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbef2ef700 1 -- 172.21.15.7:0/1663148070 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe80c3b50 con 0x7fcbe8151c00 2023-03-03T19:29:05.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbddffb700 1 -- 172.21.15.7:0/1663148070 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40036a0 con 0x7fcbe8151c00 2023-03-03T19:29:05.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.259+0000 7fcbef2ef700 1 -- 172.21.15.7:0/1663148070 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe806a2b0 con 0x7fcbe8151c00 2023-03-03T19:29:05.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.260+0000 7fcbef2ef700 1 -- 172.21.15.7:0/1663148070 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe806b180 con 0x7fcbe8151c00 2023-03-03T19:29:05.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.261+0000 7fcbddffb700 1 -- 172.21.15.7:0/1663148070 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcbe400f050 con 0x7fcbe8151c00 2023-03-03T19:29:05.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.261+0000 7fcbddffb700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:05.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.261+0000 7fcbddffb700 1 -- 172.21.15.7:0/1663148070 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcbe40874d0 con 0x7fcbe8151c00 2023-03-03T19:29:05.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.261+0000 7fcbed08b700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:05.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.262+0000 7fcbed08b700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80c7050 tx=0x7fcbd8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:29:05.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.265+0000 7fcbddffb700 1 -- 172.21.15.7:0/1663148070 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcbe4053870 con 0x7fcbe8151c00 2023-03-03T19:29:05.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.506+0000 7fcbef2ef700 1 -- 172.21.15.7:0/1663148070 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbe8074bb0 con 0x7fcbe8151c00 2023-03-03T19:29:05.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.507+0000 7fcbddffb700 1 -- 172.21.15.7:0/1663148070 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcbe4056e90 con 0x7fcbe8151c00 2023-03-03T19:29:05.508 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:29:05.508 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:29:05.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 -- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 msgr2=0x7fcbcc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:05.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80c7050 tx=0x7fcbd8008040 comp rx=0 tx=0).stop 2023-03-03T19:29:05.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 -- 172.21.15.7:0/1663148070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe8151c00 msgr2=0x7fcbe81428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:05.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe8151c00 0x7fcbe81428b0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7fcbe806bcf0 tx=0x7fcbe4004750 comp rx=0 tx=0).stop 2023-03-03T19:29:05.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 -- 172.21.15.7:0/1663148070 shutdown_connections 2023-03-03T19:29:05.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcbe8154cf0 0x7fcbe813e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:05.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbe8071990 0x7fcbe8152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:05.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbcc0682f0 0x7fcbcc06a7a0 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:05.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 --2- 172.21.15.7:0/1663148070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbe8151c00 0x7fcbe81428b0 unknown :-1 s=CLOSED pgs=3503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:05.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 -- 172.21.15.7:0/1663148070 >> 172.21.15.7:0/1663148070 conn(0x7fcbe80a94d0 msgr2=0x7fcbe815b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:29:05.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 -- 172.21.15.7:0/1663148070 shutdown_connections 2023-03-03T19:29:05.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:05.509+0000 7fcbef2ef700 1 -- 172.21.15.7:0/1663148070 wait complete. 2023-03-03T19:29:05.512 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:29:05.521 INFO:tasks.mds_thrash.fs.[a]:waiting for 109 secs before thrashing 2023-03-03T19:29:18.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:29:18.311 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:29:39.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:29:39.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.782+0000 7fc3d0caf700 1 Processor -- start 2023-03-03T19:29:39.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.783+0000 7fc3d0caf700 1 -- start start 2023-03-03T19:29:39.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.783+0000 7fc3d0caf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3cc0c0ee0 0x7fc3cc0c1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:39.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.783+0000 7fc3d0caf700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc3cc06d240 0x7fc3cc06d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:39.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3d0caf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3cc0c2ff0 con 0x7fc3cc13a160 2023-03-03T19:29:39.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3d0caf700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3cc158410 con 0x7fc3cc06d240 2023-03-03T19:29:39.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3d0caf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc154450 con 0x7fc3cc0c0ee0 2023-03-03T19:29:39.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3ca59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc3cc13a160 0x7fc3cc0c07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35740/0 (socket says 172.21.15.7:35740) 2023-03-03T19:29:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3ca59c700 1 -- 172.21.15.7:0/3466836822 learned_addr learned my addr 172.21.15.7:0/3466836822 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:29:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3cad9d700 1 --2- 172.21.15.7:0/3466836822 >> v2:172.21.15.143:3300/0 conn(0x7fc3cc06d240 0x7fc3cc06d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c9d9b700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3cc0c0ee0 0x7fc3cc0c1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880742344 0 0) 0x7fc3bc001110 con 0x7fc3cc13a160 2023-03-03T19:29:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b80035b0 con 0x7fc3cc13a160 2023-03-03T19:29:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1567713291 0 0) 0x7fc3b80035b0 con 0x7fc3cc13a160 2023-03-03T19:29:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3cc0c0ee0 msgr2=0x7fc3cc0c1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:29:39.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3cc0c0ee0 0x7fc3cc0c1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:39.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 >> v2:172.21.15.143:3300/0 conn(0x7fc3cc06d240 msgr2=0x7fc3cc06d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:29:39.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 --2- 172.21.15.7:0/3466836822 >> v2:172.21.15.143:3300/0 conn(0x7fc3cc06d240 0x7fc3cc06d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:39.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc157ae0 con 0x7fc3cc13a160 2023-03-03T19:29:39.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fc3bc001800 con 0x7fc3cc13a160 2023-03-03T19:29:39.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 >> v1:172.21.15.7:6789/0 conn(0x7fc3cc13a160 legacy=0x7fc3cc0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:39.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3b8003a70 0x7fc3b8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:39.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b8004850 0x7fc3b8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:39.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3cc0c0ee0 0x7fc3b8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:39.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc158410 con 0x7fc3b8004850 2023-03-03T19:29:39.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc154450 con 0x7fc3b8003a70 2023-03-03T19:29:39.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.784+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b800ba10 con 0x7fc3cc0c0ee0 2023-03-03T19:29:39.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3c9d9b700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b8004850 0x7fc3b8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:39.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3ca59c700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3b8003a70 0x7fc3b8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:39.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3cad9d700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3cc0c0ee0 0x7fc3b8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3c9d9b700 1 -- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3cc0c0ee0 msgr2=0x7fc3b8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:29:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3c9d9b700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3cc0c0ee0 0x7fc3b8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3c9d9b700 1 -- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3b8003a70 msgr2=0x7fc3b8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:29:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3c9d9b700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3b8003a70 0x7fc3b8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3c9d9b700 1 -- 172.21.15.7:0/3466836822 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b80035b0 con 0x7fc3b8004850 2023-03-03T19:29:39.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3c9d9b700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b8004850 0x7fc3b8004cc0 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009b90 tx=0x7fc3b8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c6f49c06e2ad server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:29:39.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b400e6c0 con 0x7fc3b8004850 2023-03-03T19:29:39.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b400e820 con 0x7fc3b8004850 2023-03-03T19:29:39.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.785+0000 7fc3c959a700 1 -- 172.21.15.7:0/3466836822 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b400eb90 con 0x7fc3b8004850 2023-03-03T19:29:39.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.786+0000 7fc3d0caf700 1 -- 172.21.15.7:0/3466836822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b8004850 msgr2=0x7fc3b8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:39.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.786+0000 7fc3d0caf700 1 --2- 172.21.15.7:0/3466836822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3b8004850 0x7fc3b8004cc0 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009b90 tx=0x7fc3b8005880 comp rx=0 tx=0).stop 2023-03-03T19:29:39.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.786+0000 7fc3cad9d700 1 -- 172.21.15.7:0/3466836822 reap_dead start 2023-03-03T19:29:39.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.786+0000 7fc3d0caf700 1 -- 172.21.15.7:0/3466836822 shutdown_connections 2023-03-03T19:29:39.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.786+0000 7fc3d0caf700 1 -- 172.21.15.7:0/3466836822 >> 172.21.15.7:0/3466836822 conn(0x7fc3cc06c110 msgr2=0x7fc3cc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:29:39.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.786+0000 7fc3d0caf700 1 -- 172.21.15.7:0/3466836822 shutdown_connections 2023-03-03T19:29:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.786+0000 7fc3d0caf700 1 -- 172.21.15.7:0/3466836822 wait complete. 2023-03-03T19:29:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.787+0000 7fc3d0caf700 1 Processor -- start 2023-03-03T19:29:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.787+0000 7fc3d0caf700 1 -- start start 2023-03-03T19:29:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3d0caf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3b8004850 0x7fc3cc0aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3d0caf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3cc071af0 0x7fc3cc155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3d0caf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3cc1367e0 0x7fc3cc150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:39.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3d0caf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1ed500 con 0x7fc3cc071af0 2023-03-03T19:29:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3d0caf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc0aae10 con 0x7fc3cc1367e0 2023-03-03T19:29:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3d0caf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc070d30 con 0x7fc3b8004850 2023-03-03T19:29:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3c9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3cc071af0 0x7fc3cc155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3c9d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3cc071af0 0x7fc3cc155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41774/0 (socket says 172.21.15.7:41774) 2023-03-03T19:29:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3c9d9b700 1 -- 172.21.15.7:0/1650842817 learned_addr learned my addr 172.21.15.7:0/1650842817 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:29:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3ca59c700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3b8004850 0x7fc3cc0aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:39.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3c9d9b700 1 -- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3b8004850 msgr2=0x7fc3cc0aa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3cad9d700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3cc1367e0 0x7fc3cc150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3c9d9b700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3b8004850 0x7fc3cc0aa8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3c9d9b700 1 -- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3cc1367e0 msgr2=0x7fc3cc150ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3c9d9b700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3cc1367e0 0x7fc3cc150ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3c9d9b700 1 -- 172.21.15.7:0/1650842817 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc1580a0 con 0x7fc3cc071af0 2023-03-03T19:29:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.788+0000 7fc3c9d9b700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3cc071af0 0x7fc3cc155840 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8004710 tx=0x7fc3b40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:29:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.789+0000 7fc3c2ffd700 1 -- 172.21.15.7:0/1650842817 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b400e6c0 con 0x7fc3cc071af0 2023-03-03T19:29:39.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.789+0000 7fc3c2ffd700 1 -- 172.21.15.7:0/1650842817 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b4003ab0 con 0x7fc3cc071af0 2023-03-03T19:29:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.789+0000 7fc3d0caf700 1 -- 172.21.15.7:0/1650842817 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3cc157dc0 con 0x7fc3cc071af0 2023-03-03T19:29:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.789+0000 7fc3c2ffd700 1 -- 172.21.15.7:0/1650842817 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4003e60 con 0x7fc3cc071af0 2023-03-03T19:29:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.789+0000 7fc3d0caf700 1 -- 172.21.15.7:0/1650842817 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3cc1588a0 con 0x7fc3cc071af0 2023-03-03T19:29:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.790+0000 7fc3d0caf700 1 -- 172.21.15.7:0/1650842817 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3cc074bb0 con 0x7fc3cc071af0 2023-03-03T19:29:39.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.790+0000 7fc3c2ffd700 1 -- 172.21.15.7:0/1650842817 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc3b4025070 con 0x7fc3cc071af0 2023-03-03T19:29:39.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.791+0000 7fc3c2ffd700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3ac0682f0 0x7fc3ac06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:29:39.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.791+0000 7fc3c2ffd700 1 -- 172.21.15.7:0/1650842817 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc3b4087a00 con 0x7fc3cc071af0 2023-03-03T19:29:39.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.791+0000 7fc3ca59c700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3ac0682f0 0x7fc3ac06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:29:39.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.791+0000 7fc3ca59c700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3ac0682f0 0x7fc3ac06a7a0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc1352c0 tx=0x7fc3bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:29:39.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:39.795+0000 7fc3c2ffd700 1 -- 172.21.15.7:0/1650842817 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc3b4053cb0 con 0x7fc3cc071af0 2023-03-03T19:29:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.033+0000 7fc3d0caf700 1 -- 172.21.15.7:0/1650842817 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3cc155d80 con 0x7fc3cc071af0 2023-03-03T19:29:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.033+0000 7fc3c2ffd700 1 -- 172.21.15.7:0/1650842817 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc3b40572d0 con 0x7fc3cc071af0 2023-03-03T19:29:40.034 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:29:40.034 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:29:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 -- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3ac0682f0 msgr2=0x7fc3ac06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3ac0682f0 0x7fc3ac06a7a0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc1352c0 tx=0x7fc3bc009040 comp rx=0 tx=0).stop 2023-03-03T19:29:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 -- 172.21.15.7:0/1650842817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3cc071af0 msgr2=0x7fc3cc155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:29:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3cc071af0 0x7fc3cc155840 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8004710 tx=0x7fc3b40047c0 comp rx=0 tx=0).stop 2023-03-03T19:29:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 -- 172.21.15.7:0/1650842817 shutdown_connections 2023-03-03T19:29:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc3cc1367e0 0x7fc3cc150ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc3b8004850 0x7fc3cc0aa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc3ac0682f0 0x7fc3ac06a7a0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 --2- 172.21.15.7:0/1650842817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3cc071af0 0x7fc3cc155840 unknown :-1 s=CLOSED pgs=3506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:29:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 -- 172.21.15.7:0/1650842817 >> 172.21.15.7:0/1650842817 conn(0x7fc3cc06c110 msgr2=0x7fc3cc0c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:29:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 -- 172.21.15.7:0/1650842817 shutdown_connections 2023-03-03T19:29:40.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:29:40.035+0000 7fc3d0caf700 1 -- 172.21.15.7:0/1650842817 wait complete. 2023-03-03T19:29:40.038 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:29:40.048 INFO:tasks.mds_thrash.fs.[b]:waiting for 35 secs before thrashing 2023-03-03T19:29:48.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:29:48.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:29:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T19:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T19:30:15.049 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:30:15.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.187+0000 7fd207f2b700 1 Processor -- start 2023-03-03T19:30:15.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.188+0000 7fd207f2b700 1 -- start start 2023-03-03T19:30:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd207f2b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd20014fcb0 0x7fd200070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd207f2b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2000c2730 0x7fd2000736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd207f2b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd200073c30 0x7fd200155b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:15.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd207f2b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd200071ea0 con 0x7fd200073c30 2023-03-03T19:30:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd207f2b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2000c2eb0 con 0x7fd20014fcb0 2023-03-03T19:30:15.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd207f2b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2000c3100 con 0x7fd2000c2730 2023-03-03T19:30:15.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd2064c8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd200073c30 0x7fd200155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:15.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd2064c8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd200073c30 0x7fd200155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41814/0 (socket says 172.21.15.7:41814) 2023-03-03T19:30:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd2064c8700 1 -- 172.21.15.7:0/1850840392 learned_addr learned my addr 172.21.15.7:0/1850840392 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:30:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd205cc7700 1 --2- 172.21.15.7:0/1850840392 >> v2:172.21.15.143:3300/0 conn(0x7fd20014fcb0 0x7fd200070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd2054c6700 1 --2- 172.21.15.7:0/1850840392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2000c2730 0x7fd2000736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd2064c8700 1 -- 172.21.15.7:0/1850840392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2000c2730 msgr2=0x7fd2000736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:30:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd2064c8700 1 --2- 172.21.15.7:0/1850840392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2000c2730 0x7fd2000736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd2064c8700 1 -- 172.21.15.7:0/1850840392 >> v2:172.21.15.143:3300/0 conn(0x7fd20014fcb0 msgr2=0x7fd200070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:30:15.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd2064c8700 1 --2- 172.21.15.7:0/1850840392 >> v2:172.21.15.143:3300/0 conn(0x7fd20014fcb0 0x7fd200070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd2064c8700 1 -- 172.21.15.7:0/1850840392 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2000c33e0 con 0x7fd200073c30 2023-03-03T19:30:15.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.189+0000 7fd2064c8700 1 --2- 172.21.15.7:0/1850840392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd200073c30 0x7fd200155b90 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7fd20006e3e0 tx=0x7fd1f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524caea6078ac900 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:30:15.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd204cc5700 1 -- 172.21.15.7:0/1850840392 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f800e070 con 0x7fd200073c30 2023-03-03T19:30:15.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd204cc5700 1 -- 172.21.15.7:0/1850840392 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f800b9c0 con 0x7fd200073c30 2023-03-03T19:30:15.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd204cc5700 1 -- 172.21.15.7:0/1850840392 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f800bd70 con 0x7fd200073c30 2023-03-03T19:30:15.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd207f2b700 1 -- 172.21.15.7:0/1850840392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd200073c30 msgr2=0x7fd200155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:15.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd207f2b700 1 --2- 172.21.15.7:0/1850840392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd200073c30 0x7fd200155b90 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7fd20006e3e0 tx=0x7fd1f800b330 comp rx=0 tx=0).stop 2023-03-03T19:30:15.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd207f2b700 1 -- 172.21.15.7:0/1850840392 shutdown_connections 2023-03-03T19:30:15.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd207f2b700 1 --2- 172.21.15.7:0/1850840392 >> v2:172.21.15.143:3300/0 conn(0x7fd20014fcb0 0x7fd200070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd207f2b700 1 --2- 172.21.15.7:0/1850840392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd200073c30 0x7fd200155b90 unknown :-1 s=CLOSED pgs=3507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd207f2b700 1 --2- 172.21.15.7:0/1850840392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2000c2730 0x7fd2000736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd207f2b700 1 -- 172.21.15.7:0/1850840392 >> 172.21.15.7:0/1850840392 conn(0x7fd2000a94d0 msgr2=0x7fd2000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:30:15.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd207f2b700 1 -- 172.21.15.7:0/1850840392 shutdown_connections 2023-03-03T19:30:15.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.190+0000 7fd207f2b700 1 -- 172.21.15.7:0/1850840392 wait complete. 2023-03-03T19:30:15.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.192+0000 7fd207f2b700 1 Processor -- start 2023-03-03T19:30:15.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.192+0000 7fd207f2b700 1 -- start start 2023-03-03T19:30:15.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd207f2b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd200073c30 0x7fd2001ebb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:15.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd207f2b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2000c2730 0x7fd2000ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:15.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd207f2b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd20014fcb0 0x7fd2001477b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:15.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd207f2b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2001eb690 con 0x7fd20014fcb0 2023-03-03T19:30:15.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd207f2b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd200148000 con 0x7fd200073c30 2023-03-03T19:30:15.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd207f2b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2001482e0 con 0x7fd2000c2730 2023-03-03T19:30:15.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd2064c8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd20014fcb0 0x7fd2001477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:15.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd2064c8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd20014fcb0 0x7fd2001477b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41820/0 (socket says 172.21.15.7:41820) 2023-03-03T19:30:15.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd2064c8700 1 -- 172.21.15.7:0/991977743 learned_addr learned my addr 172.21.15.7:0/991977743 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:30:15.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd205cc7700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd200073c30 0x7fd2001ebb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:15.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd2054c6700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2000c2730 0x7fd2000ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:15.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd2064c8700 1 -- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2000c2730 msgr2=0x7fd2000ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:15.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd2064c8700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2000c2730 0x7fd2000ac170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd2064c8700 1 -- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd200073c30 msgr2=0x7fd2001ebb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:15.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd2064c8700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd200073c30 0x7fd2001ebb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.193+0000 7fd2064c8700 1 -- 172.21.15.7:0/991977743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f8009b30 con 0x7fd20014fcb0 2023-03-03T19:30:15.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.194+0000 7fd2064c8700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd20014fcb0 0x7fd2001477b0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7fd20006b9c0 tx=0x7fd1f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:30:15.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.194+0000 7fd1f27fc700 1 -- 172.21.15.7:0/991977743 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f80099b0 con 0x7fd20014fcb0 2023-03-03T19:30:15.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.194+0000 7fd1f27fc700 1 -- 172.21.15.7:0/991977743 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f8004250 con 0x7fd20014fcb0 2023-03-03T19:30:15.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.194+0000 7fd207f2b700 1 -- 172.21.15.7:0/991977743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2000c39a0 con 0x7fd20014fcb0 2023-03-03T19:30:15.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.194+0000 7fd1f27fc700 1 -- 172.21.15.7:0/991977743 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8004600 con 0x7fd20014fcb0 2023-03-03T19:30:15.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.194+0000 7fd207f2b700 1 -- 172.21.15.7:0/991977743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2000c36c0 con 0x7fd20014fcb0 2023-03-03T19:30:15.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.195+0000 7fd207f2b700 1 -- 172.21.15.7:0/991977743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2000708d0 con 0x7fd20014fcb0 2023-03-03T19:30:15.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.195+0000 7fd1f27fc700 1 -- 172.21.15.7:0/991977743 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd1f8004760 con 0x7fd20014fcb0 2023-03-03T19:30:15.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.196+0000 7fd1f27fc700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1e40682f0 0x7fd1e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:15.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.196+0000 7fd1f27fc700 1 -- 172.21.15.7:0/991977743 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd1f8087680 con 0x7fd20014fcb0 2023-03-03T19:30:15.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.196+0000 7fd205cc7700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1e40682f0 0x7fd1e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:15.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.197+0000 7fd205cc7700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1e40682f0 0x7fd1e406a7a0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc006b20 tx=0x7fd1fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:30:15.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.200+0000 7fd1f27fc700 1 -- 172.21.15.7:0/991977743 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd1f8053a20 con 0x7fd20014fcb0 2023-03-03T19:30:15.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.445+0000 7fd207f2b700 1 -- 172.21.15.7:0/991977743 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd200074bb0 con 0x7fd20014fcb0 2023-03-03T19:30:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.445+0000 7fd1f27fc700 1 -- 172.21.15.7:0/991977743 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd1f8057040 con 0x7fd20014fcb0 2023-03-03T19:30:15.446 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:30:15.446 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:30:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.447+0000 7fd207f2b700 1 -- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1e40682f0 msgr2=0x7fd1e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.447+0000 7fd207f2b700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1e40682f0 0x7fd1e406a7a0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc006b20 tx=0x7fd1fc008040 comp rx=0 tx=0).stop 2023-03-03T19:30:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.447+0000 7fd207f2b700 1 -- 172.21.15.7:0/991977743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd20014fcb0 msgr2=0x7fd2001477b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.447+0000 7fd207f2b700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd20014fcb0 0x7fd2001477b0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7fd20006b9c0 tx=0x7fd1f8004ad0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.447+0000 7fd207f2b700 1 -- 172.21.15.7:0/991977743 shutdown_connections 2023-03-03T19:30:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.447+0000 7fd207f2b700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd1e40682f0 0x7fd1e406a7a0 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.447+0000 7fd207f2b700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd20014fcb0 0x7fd2001477b0 unknown :-1 s=CLOSED pgs=3508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.447+0000 7fd207f2b700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd200073c30 0x7fd2001ebb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.447+0000 7fd207f2b700 1 --2- 172.21.15.7:0/991977743 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2000c2730 0x7fd2000ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.447+0000 7fd207f2b700 1 -- 172.21.15.7:0/991977743 >> 172.21.15.7:0/991977743 conn(0x7fd2000a94d0 msgr2=0x7fd200070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:30:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.448+0000 7fd207f2b700 1 -- 172.21.15.7:0/991977743 shutdown_connections 2023-03-03T19:30:15.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:15.448+0000 7fd207f2b700 1 -- 172.21.15.7:0/991977743 wait complete. 2023-03-03T19:30:15.450 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:30:15.460 INFO:tasks.mds_thrash.fs.[b]:waiting for 85 secs before thrashing 2023-03-03T19:30:18.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:30:18.455 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:30:48.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:30:48.581 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:30:54.523 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:30:54.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.661+0000 7f8c5ee7e700 1 Processor -- start 2023-03-03T19:30:54.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.661+0000 7f8c5ee7e700 1 -- start start 2023-03-03T19:30:54.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.662+0000 7f8c5ee7e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c580740f0 0x7f8c58156290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:54.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.662+0000 7f8c5ee7e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c58155840 con 0x7f8c5814fb00 2023-03-03T19:30:54.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.662+0000 7f8c5ee7e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c581567d0 con 0x7f8c580c1da0 2023-03-03T19:30:54.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.662+0000 7f8c5ee7e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c58072940 con 0x7f8c580740f0 2023-03-03T19:30:54.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.662+0000 7f8c5cc1a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8c5814fb00 0x7f8c580c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35818/0 (socket says 172.21.15.7:35818) 2023-03-03T19:30:54.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.662+0000 7f8c5cc1a700 1 -- 172.21.15.7:0/1434775287 learned_addr learned my addr 172.21.15.7:0/1434775287 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:30:54.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c5d41b700 1 --2- 172.21.15.7:0/1434775287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c580740f0 0x7f8c58156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:54.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c577fe700 1 -- 172.21.15.7:0/1434775287 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453044485 0 0) 0x7f8c58155840 con 0x7f8c5814fb00 2023-03-03T19:30:54.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c577fe700 1 -- 172.21.15.7:0/1434775287 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c480035b0 con 0x7f8c5814fb00 2023-03-03T19:30:54.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c577fe700 1 -- 172.21.15.7:0/1434775287 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642893365 0 0) 0x7f8c581567d0 con 0x7f8c580c1da0 2023-03-03T19:30:54.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c577fe700 1 -- 172.21.15.7:0/1434775287 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c58155840 con 0x7f8c580c1da0 2023-03-03T19:30:54.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c5d41b700 1 -- 172.21.15.7:0/1434775287 >> v1:172.21.15.143:6789/0 conn(0x7f8c580c1da0 legacy=0x7f8c5815abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:54.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c577fe700 1 -- 172.21.15.7:0/1434775287 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2066838844 0 0) 0x7f8c480035b0 con 0x7f8c5814fb00 2023-03-03T19:30:54.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c5d41b700 1 -- 172.21.15.7:0/1434775287 >> v1:172.21.15.7:6789/0 conn(0x7f8c5814fb00 legacy=0x7f8c580c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:54.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c5d41b700 1 -- 172.21.15.7:0/1434775287 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c580c2330 con 0x7f8c580740f0 2023-03-03T19:30:54.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c5d41b700 1 --2- 172.21.15.7:0/1434775287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c580740f0 0x7f8c58156290 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8c50009dc0 tx=0x7f8c5000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30b08d70eb6a6ae8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:30:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c577fe700 1 -- 172.21.15.7:0/1434775287 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5000e070 con 0x7f8c580740f0 2023-03-03T19:30:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c577fe700 1 -- 172.21.15.7:0/1434775287 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c50003a20 con 0x7f8c580740f0 2023-03-03T19:30:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.663+0000 7f8c577fe700 1 -- 172.21.15.7:0/1434775287 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c001670 con 0x7f8c580740f0 2023-03-03T19:30:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.664+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/1434775287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c580740f0 msgr2=0x7f8c58156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.664+0000 7f8c5ee7e700 1 --2- 172.21.15.7:0/1434775287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c580740f0 0x7f8c58156290 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8c50009dc0 tx=0x7f8c5000b330 comp rx=0 tx=0).stop 2023-03-03T19:30:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.664+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/1434775287 shutdown_connections 2023-03-03T19:30:54.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.664+0000 7f8c5ee7e700 1 --2- 172.21.15.7:0/1434775287 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c580740f0 0x7f8c58156290 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:54.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.664+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/1434775287 >> 172.21.15.7:0/1434775287 conn(0x7f8c5806c110 msgr2=0x7f8c5806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:30:54.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.664+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/1434775287 shutdown_connections 2023-03-03T19:30:54.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.664+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/1434775287 wait complete. 2023-03-03T19:30:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.665+0000 7f8c5ee7e700 1 Processor -- start 2023-03-03T19:30:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.665+0000 7f8c5ee7e700 1 -- start start 2023-03-03T19:30:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c5ee7e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c580740f0 0x7f8c581513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c5ee7e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c580c1da0 0x7f8c58147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:54.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c5ee7e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c5814fb00 0x7f8c58148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c5ee7e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5814e210 con 0x7f8c580740f0 2023-03-03T19:30:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c5ee7e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c581ddf70 con 0x7f8c580c1da0 2023-03-03T19:30:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c5ee7e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5814e460 con 0x7f8c5814fb00 2023-03-03T19:30:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c5cc1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c580740f0 0x7f8c581513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c5cc1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c580740f0 0x7f8c581513d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41844/0 (socket says 172.21.15.7:41844) 2023-03-03T19:30:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c5cc1a700 1 -- 172.21.15.7:0/3076553282 learned_addr learned my addr 172.21.15.7:0/3076553282 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:30:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c5d41b700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c5814fb00 0x7f8c58148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:54.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.666+0000 7f8c57fff700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c580c1da0 0x7f8c58147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c5cc1a700 1 -- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c5814fb00 msgr2=0x7f8c58148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c5cc1a700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c5814fb00 0x7f8c58148150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c5cc1a700 1 -- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c580c1da0 msgr2=0x7f8c58147c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c5cc1a700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c580c1da0 0x7f8c58147c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c5cc1a700 1 -- 172.21.15.7:0/3076553282 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c50009a70 con 0x7f8c580740f0 2023-03-03T19:30:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c5cc1a700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c580740f0 0x7f8c581513d0 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00c4d0 tx=0x7f8c4c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:30:54.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c557fa700 1 -- 172.21.15.7:0/3076553282 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c004c80 con 0x7f8c580740f0 2023-03-03T19:30:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c557fa700 1 -- 172.21.15.7:0/3076553282 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4c004e00 con 0x7f8c580740f0 2023-03-03T19:30:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/3076553282 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c58153090 con 0x7f8c580740f0 2023-03-03T19:30:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c557fa700 1 -- 172.21.15.7:0/3076553282 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c007310 con 0x7f8c580740f0 2023-03-03T19:30:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.667+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/3076553282 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c58152db0 con 0x7f8c580740f0 2023-03-03T19:30:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.668+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/3076553282 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c58155a90 con 0x7f8c580740f0 2023-03-03T19:30:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.668+0000 7f8c557fa700 1 -- 172.21.15.7:0/3076553282 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8c4c01a020 con 0x7f8c580740f0 2023-03-03T19:30:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.669+0000 7f8c557fa700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c3c0682f0 0x7f8c3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:30:54.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.669+0000 7f8c557fa700 1 -- 172.21.15.7:0/3076553282 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8c4c085cb0 con 0x7f8c580740f0 2023-03-03T19:30:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.669+0000 7f8c57fff700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c3c0682f0 0x7f8c3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:30:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.670+0000 7f8c57fff700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c3c0682f0 0x7f8c3c06a7a0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f8c581ddcf0 tx=0x7f8c44009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:30:54.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.673+0000 7f8c557fa700 1 -- 172.21.15.7:0/3076553282 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8c4c052050 con 0x7f8c580740f0 2023-03-03T19:30:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.912+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/3076553282 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c5806e410 con 0x7f8c580740f0 2023-03-03T19:30:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.912+0000 7f8c557fa700 1 -- 172.21.15.7:0/3076553282 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8c4c055670 con 0x7f8c580740f0 2023-03-03T19:30:54.920 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:30:54.920 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:30:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c3c0682f0 msgr2=0x7f8c3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c3c0682f0 0x7f8c3c06a7a0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f8c581ddcf0 tx=0x7f8c44009040 comp rx=0 tx=0).stop 2023-03-03T19:30:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/3076553282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c580740f0 msgr2=0x7f8c581513d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:30:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c580740f0 0x7f8c581513d0 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00c4d0 tx=0x7f8c4c00c7e0 comp rx=0 tx=0).stop 2023-03-03T19:30:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/3076553282 shutdown_connections 2023-03-03T19:30:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8c5814fb00 0x7f8c58148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8c3c0682f0 0x7f8c3c06a7a0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c580740f0 0x7f8c581513d0 unknown :-1 s=CLOSED pgs=3510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 --2- 172.21.15.7:0/3076553282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c580c1da0 0x7f8c58147c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:30:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/3076553282 >> 172.21.15.7:0/3076553282 conn(0x7f8c5806c110 msgr2=0x7f8c5814f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:30:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/3076553282 shutdown_connections 2023-03-03T19:30:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:30:54.914+0000 7f8c5ee7e700 1 -- 172.21.15.7:0/3076553282 wait complete. 2023-03-03T19:30:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:30:54.929 INFO:tasks.mds_thrash.fs.[a]:waiting for 61 secs before thrashing 2023-03-03T19:31:18.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:31:18.726 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:31:40.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:31:40.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.601+0000 7f2482938700 1 Processor -- start 2023-03-03T19:31:40.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.601+0000 7f2482938700 1 -- start start 2023-03-03T19:31:40.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f2482938700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f247c14fda0 0x7f247c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:40.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f2482938700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247c0c2f30 con 0x7f247c14f9d0 2023-03-03T19:31:40.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f2482938700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247c0c31d0 con 0x7f247c0abe20 2023-03-03T19:31:40.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f2482938700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247c1524a0 con 0x7f247c14fda0 2023-03-03T19:31:40.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f247b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f247c14fda0 0x7f247c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:40.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f247bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f247c14f9d0 0x7f247c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35878/0 (socket says 172.21.15.7:35878) 2023-03-03T19:31:40.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f247bfff700 1 -- 172.21.15.7:0/842799252 learned_addr learned my addr 172.21.15.7:0/842799252 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:31:40.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f247b7fe700 1 -- 172.21.15.7:0/842799252 >> v1:172.21.15.143:6789/0 conn(0x7f247c0abe20 legacy=0x7f247c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:31:40.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f247b7fe700 1 -- 172.21.15.7:0/842799252 >> v1:172.21.15.7:6789/0 conn(0x7f247c14f9d0 legacy=0x7f247c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:40.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f247b7fe700 1 -- 172.21.15.7:0/842799252 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c0c34b0 con 0x7f247c14fda0 2023-03-03T19:31:40.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f247affd700 1 -- 172.21.15.7:0/842799252 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847374008 0 0) 0x7f247c0c2f30 con 0x7f247c14f9d0 2023-03-03T19:31:40.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f247b7fe700 1 --2- 172.21.15.7:0/842799252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c14fda0 0x7f247c156430 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 crypto rx=0x7f2470009840 tx=0x7f2470009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6de3fadf6d8313f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:31:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.602+0000 7f247affd700 1 -- 172.21.15.7:0/842799252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470004230 con 0x7f247c14fda0 2023-03-03T19:31:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.603+0000 7f247affd700 1 -- 172.21.15.7:0/842799252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2470004390 con 0x7f247c14fda0 2023-03-03T19:31:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.603+0000 7f247affd700 1 -- 172.21.15.7:0/842799252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24700036a0 con 0x7f247c14fda0 2023-03-03T19:31:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.603+0000 7f2482938700 1 -- 172.21.15.7:0/842799252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c14fda0 msgr2=0x7f247c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:40.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.603+0000 7f2482938700 1 --2- 172.21.15.7:0/842799252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c14fda0 0x7f247c156430 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 crypto rx=0x7f2470009840 tx=0x7f2470009b50 comp rx=0 tx=0).stop 2023-03-03T19:31:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.603+0000 7f2482938700 1 -- 172.21.15.7:0/842799252 shutdown_connections 2023-03-03T19:31:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.603+0000 7f2482938700 1 --2- 172.21.15.7:0/842799252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c14fda0 0x7f247c156430 unknown :-1 s=CLOSED pgs=3512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.603+0000 7f2482938700 1 -- 172.21.15.7:0/842799252 >> 172.21.15.7:0/842799252 conn(0x7f247c0c8900 msgr2=0x7f247c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:31:40.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.603+0000 7f2482938700 1 -- 172.21.15.7:0/842799252 shutdown_connections 2023-03-03T19:31:40.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.603+0000 7f2482938700 1 -- 172.21.15.7:0/842799252 wait complete. 2023-03-03T19:31:40.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.604+0000 7f2482938700 1 Processor -- start 2023-03-03T19:31:40.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.605+0000 7f2482938700 1 -- start start 2023-03-03T19:31:40.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.605+0000 7f2482938700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c0abe20 0x7f247c06e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:40.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.605+0000 7f2482938700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f247c14f9d0 0x7f247c1def80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:40.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.605+0000 7f2482938700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f247c14fda0 0x7f247c14dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:40.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.605+0000 7f2482938700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c06f5a0 con 0x7f247c0abe20 2023-03-03T19:31:40.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.605+0000 7f2482938700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247c1e9760 con 0x7f247c14fda0 2023-03-03T19:31:40.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.605+0000 7f2482938700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247c1e9a40 con 0x7f247c14f9d0 2023-03-03T19:31:40.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.605+0000 7f247bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c0abe20 0x7f247c06e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:40.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.605+0000 7f247bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c0abe20 0x7f247c06e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41904/0 (socket says 172.21.15.7:41904) 2023-03-03T19:31:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.605+0000 7f247bfff700 1 -- 172.21.15.7:0/1551216159 learned_addr learned my addr 172.21.15.7:0/1551216159 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:31:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f2480ed5700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f247c14fda0 0x7f247c14dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f247b7fe700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f247c14f9d0 0x7f247c1def80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f247bfff700 1 -- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f247c14f9d0 msgr2=0x7f247c1def80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f247bfff700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f247c14f9d0 0x7f247c1def80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f247bfff700 1 -- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f247c14fda0 msgr2=0x7f247c14dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f247bfff700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f247c14fda0 0x7f247c14dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f247bfff700 1 -- 172.21.15.7:0/1551216159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24700094a0 con 0x7f247c0abe20 2023-03-03T19:31:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f2480ed5700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f247c14fda0 0x7f247c14dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:31:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f247bfff700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c0abe20 0x7f247c06e900 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f246c0011e0 tx=0x7f246c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:31:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f2478ff9700 1 -- 172.21.15.7:0/1551216159 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c004c80 con 0x7f247c0abe20 2023-03-03T19:31:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f2478ff9700 1 -- 172.21.15.7:0/1551216159 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246c004e00 con 0x7f247c0abe20 2023-03-03T19:31:40.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f2482938700 1 -- 172.21.15.7:0/1551216159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247c0ac350 con 0x7f247c0abe20 2023-03-03T19:31:40.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f2478ff9700 1 -- 172.21.15.7:0/1551216159 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c007310 con 0x7f247c0abe20 2023-03-03T19:31:40.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.606+0000 7f2482938700 1 -- 172.21.15.7:0/1551216159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247c0ab630 con 0x7f247c0abe20 2023-03-03T19:31:40.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.607+0000 7f2482938700 1 -- 172.21.15.7:0/1551216159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247c071fa0 con 0x7f247c0abe20 2023-03-03T19:31:40.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.608+0000 7f2478ff9700 1 -- 172.21.15.7:0/1551216159 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f246c01a020 con 0x7f247c0abe20 2023-03-03T19:31:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.608+0000 7f2478ff9700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2468068320 0x7f246806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.608+0000 7f2478ff9700 1 -- 172.21.15.7:0/1551216159 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f246c085db0 con 0x7f247c0abe20 2023-03-03T19:31:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.608+0000 7f247b7fe700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2468068320 0x7f246806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.609+0000 7f247b7fe700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2468068320 0x7f246806a7d0 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f247c1501d0 tx=0x7f247001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:31:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.612+0000 7f2478ff9700 1 -- 172.21.15.7:0/1551216159 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f246c052150 con 0x7f247c0abe20 2023-03-03T19:31:40.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.852+0000 7f2482938700 1 -- 172.21.15.7:0/1551216159 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f247c074bb0 con 0x7f247c0abe20 2023-03-03T19:31:40.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.853+0000 7f2478ff9700 1 -- 172.21.15.7:0/1551216159 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f246c055770 con 0x7f247c0abe20 2023-03-03T19:31:40.853 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:31:40.854 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:31:40.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 -- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2468068320 msgr2=0x7f246806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:40.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2468068320 0x7f246806a7d0 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f247c1501d0 tx=0x7f247001c040 comp rx=0 tx=0).stop 2023-03-03T19:31:40.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 -- 172.21.15.7:0/1551216159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c0abe20 msgr2=0x7f247c06e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:40.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c0abe20 0x7f247c06e900 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f246c0011e0 tx=0x7f246c00cae0 comp rx=0 tx=0).stop 2023-03-03T19:31:40.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 -- 172.21.15.7:0/1551216159 shutdown_connections 2023-03-03T19:31:40.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f247c14fda0 0x7f247c14dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:40.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2468068320 0x7f246806a7d0 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:40.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f247c0abe20 0x7f247c06e900 unknown :-1 s=CLOSED pgs=3513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:40.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 --2- 172.21.15.7:0/1551216159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f247c14f9d0 0x7f247c1def80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:40.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 -- 172.21.15.7:0/1551216159 >> 172.21.15.7:0/1551216159 conn(0x7f247c0c8900 msgr2=0x7f247c0c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:31:40.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 -- 172.21.15.7:0/1551216159 shutdown_connections 2023-03-03T19:31:40.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:40.855+0000 7f2482938700 1 -- 172.21.15.7:0/1551216159 wait complete. 2023-03-03T19:31:40.857 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:31:40.867 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-03-03T19:31:48.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:31:48.889 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:31:51.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:31:52.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.007+0000 7f2f9702d700 1 Processor -- start 2023-03-03T19:31:52.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.007+0000 7f2f9702d700 1 -- start start 2023-03-03T19:31:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f9702d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f2f9014fcb0 0x7f2f90070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f9702d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2f90155b90 0x7f2f90155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f9702d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f900c2eb0 con 0x7f2f900c2730 2023-03-03T19:31:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f9702d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f90071ea0 con 0x7f2f90155b90 2023-03-03T19:31:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f9702d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f900c3100 con 0x7f2f9014fcb0 2023-03-03T19:31:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f955ca700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2f90155b90 0x7f2f90155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f8ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2f900c2730 0x7f2f900736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35894/0 (socket says 172.21.15.7:35894) 2023-03-03T19:31:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f8ffff700 1 -- 172.21.15.7:0/699481896 learned_addr learned my addr 172.21.15.7:0/699481896 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:31:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f94dc9700 1 --2- 172.21.15.7:0/699481896 >> v2:172.21.15.143:3300/0 conn(0x7f2f9014fcb0 0x7f2f90070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f955ca700 1 -- 172.21.15.7:0/699481896 >> v2:172.21.15.143:3300/0 conn(0x7f2f9014fcb0 msgr2=0x7f2f90070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:31:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f955ca700 1 --2- 172.21.15.7:0/699481896 >> v2:172.21.15.143:3300/0 conn(0x7f2f9014fcb0 0x7f2f90070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f955ca700 1 -- 172.21.15.7:0/699481896 >> v1:172.21.15.7:6789/0 conn(0x7f2f900c2730 legacy=0x7f2f900736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f955ca700 1 -- 172.21.15.7:0/699481896 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f900c33e0 con 0x7f2f90155b90 2023-03-03T19:31:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f955ca700 1 --2- 172.21.15.7:0/699481896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 0x7f2f90155fb0 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7f2f9006e3e0 tx=0x7f2f8800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a738d0d8e94e2272 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:31:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.008+0000 7f2f8f7fe700 1 -- 172.21.15.7:0/699481896 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8800e070 con 0x7f2f90155b90 2023-03-03T19:31:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.009+0000 7f2f8f7fe700 1 -- 172.21.15.7:0/699481896 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8800bb20 con 0x7f2f90155b90 2023-03-03T19:31:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.009+0000 7f2f8f7fe700 1 -- 172.21.15.7:0/699481896 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f88012430 con 0x7f2f90155b90 2023-03-03T19:31:52.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.009+0000 7f2f9702d700 1 -- 172.21.15.7:0/699481896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 msgr2=0x7f2f90155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:52.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.009+0000 7f2f9702d700 1 --2- 172.21.15.7:0/699481896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 0x7f2f90155fb0 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7f2f9006e3e0 tx=0x7f2f8800b330 comp rx=0 tx=0).stop 2023-03-03T19:31:52.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.009+0000 7f2f9702d700 1 -- 172.21.15.7:0/699481896 shutdown_connections 2023-03-03T19:31:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.009+0000 7f2f9702d700 1 --2- 172.21.15.7:0/699481896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 0x7f2f90155fb0 unknown :-1 s=CLOSED pgs=3515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.009+0000 7f2f9702d700 1 --2- 172.21.15.7:0/699481896 >> v2:172.21.15.143:3300/0 conn(0x7f2f9014fcb0 0x7f2f90070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.009+0000 7f2f9702d700 1 -- 172.21.15.7:0/699481896 >> 172.21.15.7:0/699481896 conn(0x7f2f900a94d0 msgr2=0x7f2f900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:31:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.009+0000 7f2f9702d700 1 -- 172.21.15.7:0/699481896 shutdown_connections 2023-03-03T19:31:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.009+0000 7f2f9702d700 1 -- 172.21.15.7:0/699481896 wait complete. 2023-03-03T19:31:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.010+0000 7f2f9702d700 1 Processor -- start 2023-03-03T19:31:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.010+0000 7f2f9702d700 1 -- start start 2023-03-03T19:31:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f9702d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f900c2730 0x7f2f90146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f9702d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f9014fcb0 0x7f2f900ab510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f9702d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 0x7f2f900aba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f9702d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f900c6030 con 0x7f2f90155b90 2023-03-03T19:31:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f9702d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f900ab080 con 0x7f2f900c2730 2023-03-03T19:31:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f9702d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f900ac8f0 con 0x7f2f9014fcb0 2023-03-03T19:31:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f955ca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 0x7f2f900aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f955ca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 0x7f2f900aba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41924/0 (socket says 172.21.15.7:41924) 2023-03-03T19:31:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f955ca700 1 -- 172.21.15.7:0/1901515345 learned_addr learned my addr 172.21.15.7:0/1901515345 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:31:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f94dc9700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f900c2730 0x7f2f90146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.011+0000 7f2f8ffff700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f9014fcb0 0x7f2f900ab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f955ca700 1 -- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f9014fcb0 msgr2=0x7f2f900ab510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f955ca700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f9014fcb0 0x7f2f900ab510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f955ca700 1 -- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f900c2730 msgr2=0x7f2f90146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f955ca700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f900c2730 0x7f2f90146640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:52.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f955ca700 1 -- 172.21.15.7:0/1901515345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f88009a70 con 0x7f2f90155b90 2023-03-03T19:31:52.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f8ffff700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f9014fcb0 0x7f2f900ab510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:31:52.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f955ca700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 0x7f2f900aba50 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7f2f88014040 tx=0x7f2f88004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:31:52.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f8d7fa700 1 -- 172.21.15.7:0/1901515345 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f88012430 con 0x7f2f90155b90 2023-03-03T19:31:52.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f8d7fa700 1 -- 172.21.15.7:0/1901515345 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f88020070 con 0x7f2f90155b90 2023-03-03T19:31:52.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f9702d700 1 -- 172.21.15.7:0/1901515345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f900c39a0 con 0x7f2f90155b90 2023-03-03T19:31:52.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f8d7fa700 1 -- 172.21.15.7:0/1901515345 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f88004280 con 0x7f2f90155b90 2023-03-03T19:31:52.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.012+0000 7f2f9702d700 1 -- 172.21.15.7:0/1901515345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f900c36c0 con 0x7f2f90155b90 2023-03-03T19:31:52.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.013+0000 7f2f9702d700 1 -- 172.21.15.7:0/1901515345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f90073e00 con 0x7f2f90155b90 2023-03-03T19:31:52.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.014+0000 7f2f8d7fa700 1 -- 172.21.15.7:0/1901515345 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2f880043e0 con 0x7f2f90155b90 2023-03-03T19:31:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.014+0000 7f2f8d7fa700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f740682f0 0x7f2f7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.014+0000 7f2f8d7fa700 1 -- 172.21.15.7:0/1901515345 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2f88022070 con 0x7f2f90155b90 2023-03-03T19:31:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.014+0000 7f2f94dc9700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f740682f0 0x7f2f7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.015+0000 7f2f94dc9700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f740682f0 0x7f2f7406a7a0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009be0 tx=0x7f2f84009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:31:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.018+0000 7f2f8d7fa700 1 -- 172.21.15.7:0/1901515345 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2f88054680 con 0x7f2f90155b90 2023-03-03T19:31:52.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.262+0000 7f2f9702d700 1 -- 172.21.15.7:0/1901515345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f90074bb0 con 0x7f2f90155b90 2023-03-03T19:31:52.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.262+0000 7f2f8d7fa700 1 -- 172.21.15.7:0/1901515345 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2f88057ca0 con 0x7f2f90155b90 2023-03-03T19:31:52.263 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:31:52.263 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:31:52.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 -- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f740682f0 msgr2=0x7f2f7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:52.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f740682f0 0x7f2f7406a7a0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009be0 tx=0x7f2f84009210 comp rx=0 tx=0).stop 2023-03-03T19:31:52.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 -- 172.21.15.7:0/1901515345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 msgr2=0x7f2f900aba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:52.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 0x7f2f900aba50 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7f2f88014040 tx=0x7f2f88004ad0 comp rx=0 tx=0).stop 2023-03-03T19:31:52.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 -- 172.21.15.7:0/1901515345 shutdown_connections 2023-03-03T19:31:52.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2f740682f0 0x7f2f7406a7a0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:52.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2f90155b90 0x7f2f900aba50 unknown :-1 s=CLOSED pgs=3516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:52.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f900c2730 0x7f2f90146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:52.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 --2- 172.21.15.7:0/1901515345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2f9014fcb0 0x7f2f900ab510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:52.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 -- 172.21.15.7:0/1901515345 >> 172.21.15.7:0/1901515345 conn(0x7f2f900a94d0 msgr2=0x7f2f900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:31:52.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 -- 172.21.15.7:0/1901515345 shutdown_connections 2023-03-03T19:31:52.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:52.264+0000 7f2f9702d700 1 -- 172.21.15.7:0/1901515345 wait complete. 2023-03-03T19:31:52.267 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:31:52.277 INFO:tasks.mds_thrash.fs.[b]:waiting for 118 secs before thrashing 2023-03-03T19:31:55.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:31:56.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.070+0000 7fca269f7700 1 Processor -- start 2023-03-03T19:31:56.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.070+0000 7fca269f7700 1 -- start start 2023-03-03T19:31:56.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca269f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2014f9d0 0x7fca20156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:56.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca269f7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca2014fda0 0x7fca20071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:56.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca269f7700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca200abe20 0x7fca201529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:56.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca269f7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca200c2f30 con 0x7fca2014fda0 2023-03-03T19:31:56.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca269f7700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca200c31d0 con 0x7fca200abe20 2023-03-03T19:31:56.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca269f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca201524a0 con 0x7fca2014f9d0 2023-03-03T19:31:56.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca2014fda0 0x7fca20071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:56.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca2014fda0 0x7fca20071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41930/0 (socket says 172.21.15.7:41930) 2023-03-03T19:31:56.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1f7fe700 1 -- 172.21.15.7:0/1517531456 learned_addr learned my addr 172.21.15.7:0/1517531456 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:31:56.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca24f94700 1 --2- 172.21.15.7:0/1517531456 >> v2:172.21.15.143:3300/0 conn(0x7fca200abe20 0x7fca201529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:56.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1ffff700 1 --2- 172.21.15.7:0/1517531456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2014f9d0 0x7fca20156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:56.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1f7fe700 1 -- 172.21.15.7:0/1517531456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2014f9d0 msgr2=0x7fca20156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:31:56.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1f7fe700 1 --2- 172.21.15.7:0/1517531456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2014f9d0 0x7fca20156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1f7fe700 1 -- 172.21.15.7:0/1517531456 >> v2:172.21.15.143:3300/0 conn(0x7fca200abe20 msgr2=0x7fca201529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:31:56.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1f7fe700 1 --2- 172.21.15.7:0/1517531456 >> v2:172.21.15.143:3300/0 conn(0x7fca200abe20 0x7fca201529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1f7fe700 1 -- 172.21.15.7:0/1517531456 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca200c34b0 con 0x7fca2014fda0 2023-03-03T19:31:56.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1f7fe700 1 --2- 172.21.15.7:0/1517531456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014fda0 0x7fca20071890 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7fca200c2d20 tx=0x7fca10009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155d1e53d37017d9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:31:56.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.071+0000 7fca1effd700 1 -- 172.21.15.7:0/1517531456 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1000b870 con 0x7fca2014fda0 2023-03-03T19:31:56.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca1effd700 1 -- 172.21.15.7:0/1517531456 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1000b9d0 con 0x7fca2014fda0 2023-03-03T19:31:56.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca1effd700 1 -- 172.21.15.7:0/1517531456 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1000bd60 con 0x7fca2014fda0 2023-03-03T19:31:56.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca269f7700 1 -- 172.21.15.7:0/1517531456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014fda0 msgr2=0x7fca20071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:56.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca269f7700 1 --2- 172.21.15.7:0/1517531456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014fda0 0x7fca20071890 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7fca200c2d20 tx=0x7fca10009bd0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca269f7700 1 -- 172.21.15.7:0/1517531456 shutdown_connections 2023-03-03T19:31:56.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca269f7700 1 --2- 172.21.15.7:0/1517531456 >> v2:172.21.15.143:3300/0 conn(0x7fca200abe20 0x7fca201529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca269f7700 1 --2- 172.21.15.7:0/1517531456 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2014f9d0 0x7fca20156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca269f7700 1 --2- 172.21.15.7:0/1517531456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014fda0 0x7fca20071890 unknown :-1 s=CLOSED pgs=3517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca269f7700 1 -- 172.21.15.7:0/1517531456 >> 172.21.15.7:0/1517531456 conn(0x7fca200c8900 msgr2=0x7fca200c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:31:56.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca269f7700 1 -- 172.21.15.7:0/1517531456 shutdown_connections 2023-03-03T19:31:56.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.072+0000 7fca269f7700 1 -- 172.21.15.7:0/1517531456 wait complete. 2023-03-03T19:31:56.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.073+0000 7fca269f7700 1 Processor -- start 2023-03-03T19:31:56.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca269f7700 1 -- start start 2023-03-03T19:31:56.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca269f7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca200abe20 0x7fca201e9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:56.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca269f7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014f9d0 0x7fca201eaa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:56.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca269f7700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2014fda0 0x7fca201dc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:56.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca269f7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca201dc350 con 0x7fca2014f9d0 2023-03-03T19:31:56.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca269f7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca201561d0 con 0x7fca200abe20 2023-03-03T19:31:56.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca269f7700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca201db780 con 0x7fca2014fda0 2023-03-03T19:31:56.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca1f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014f9d0 0x7fca201eaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:56.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca1f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014f9d0 0x7fca201eaa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:41936/0 (socket says 172.21.15.7:41936) 2023-03-03T19:31:56.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca1f7fe700 1 -- 172.21.15.7:0/2568228672 learned_addr learned my addr 172.21.15.7:0/2568228672 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:31:56.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca1ffff700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca200abe20 0x7fca201e9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:56.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.074+0000 7fca24f94700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2014fda0 0x7fca201dc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:56.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca1f7fe700 1 -- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2014fda0 msgr2=0x7fca201dc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:56.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca1f7fe700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2014fda0 0x7fca201dc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca1f7fe700 1 -- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca200abe20 msgr2=0x7fca201e9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:56.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca1f7fe700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca200abe20 0x7fca201e9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca1f7fe700 1 -- 172.21.15.7:0/2568228672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10009550 con 0x7fca2014f9d0 2023-03-03T19:31:56.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca1f7fe700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014f9d0 0x7fca201eaa10 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7fca10000c00 tx=0x7fca10004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:31:56.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca1cff9700 1 -- 172.21.15.7:0/2568228672 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10004480 con 0x7fca2014f9d0 2023-03-03T19:31:56.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca1cff9700 1 -- 172.21.15.7:0/2568228672 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca100045e0 con 0x7fca2014f9d0 2023-03-03T19:31:56.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca1cff9700 1 -- 172.21.15.7:0/2568228672 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca100038e0 con 0x7fca2014f9d0 2023-03-03T19:31:56.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca269f7700 1 -- 172.21.15.7:0/2568228672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca200ac350 con 0x7fca2014f9d0 2023-03-03T19:31:56.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.075+0000 7fca269f7700 1 -- 172.21.15.7:0/2568228672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca200ab630 con 0x7fca2014f9d0 2023-03-03T19:31:56.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.076+0000 7fca269f7700 1 -- 172.21.15.7:0/2568228672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca200c3e10 con 0x7fca2014f9d0 2023-03-03T19:31:56.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.077+0000 7fca1cff9700 1 -- 172.21.15.7:0/2568228672 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fca1000f050 con 0x7fca2014f9d0 2023-03-03T19:31:56.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.077+0000 7fca1cff9700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca000682f0 0x7fca0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:31:56.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.077+0000 7fca1cff9700 1 -- 172.21.15.7:0/2568228672 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fca10087730 con 0x7fca2014f9d0 2023-03-03T19:31:56.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.077+0000 7fca1ffff700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca000682f0 0x7fca0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:31:56.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.078+0000 7fca1ffff700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca000682f0 0x7fca0006a7a0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fca2006fbb0 tx=0x7fca08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:31:56.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.082+0000 7fca1cff9700 1 -- 172.21.15.7:0/2568228672 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fca10053ad0 con 0x7fca2014f9d0 2023-03-03T19:31:56.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.318+0000 7fca269f7700 1 -- 172.21.15.7:0/2568228672 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca20074bb0 con 0x7fca2014f9d0 2023-03-03T19:31:56.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.319+0000 7fca1cff9700 1 -- 172.21.15.7:0/2568228672 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fca100570f0 con 0x7fca2014f9d0 2023-03-03T19:31:56.320 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:31:56.326 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:31:56.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 -- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca000682f0 msgr2=0x7fca0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:56.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca000682f0 0x7fca0006a7a0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fca2006fbb0 tx=0x7fca08008040 comp rx=0 tx=0).stop 2023-03-03T19:31:56.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 -- 172.21.15.7:0/2568228672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014f9d0 msgr2=0x7fca201eaa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:31:56.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014f9d0 0x7fca201eaa10 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7fca10000c00 tx=0x7fca10004750 comp rx=0 tx=0).stop 2023-03-03T19:31:56.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 -- 172.21.15.7:0/2568228672 shutdown_connections 2023-03-03T19:31:56.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca2014fda0 0x7fca201dc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca200abe20 0x7fca201e9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca000682f0 0x7fca0006a7a0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 --2- 172.21.15.7:0/2568228672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca2014f9d0 0x7fca201eaa10 unknown :-1 s=CLOSED pgs=3518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:31:56.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 -- 172.21.15.7:0/2568228672 >> 172.21.15.7:0/2568228672 conn(0x7fca200c8900 msgr2=0x7fca200c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:31:56.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 -- 172.21.15.7:0/2568228672 shutdown_connections 2023-03-03T19:31:56.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:31:56.321+0000 7fca269f7700 1 -- 172.21.15.7:0/2568228672 wait complete. 2023-03-03T19:31:56.329 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:31:56.334 INFO:tasks.mds_thrash.fs.[a]:waiting for 68 secs before thrashing 2023-03-03T19:32:19.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:32:19.035 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:32:49.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:32:49.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:33:04.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:33:04.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.474+0000 7f49a6553700 1 Processor -- start 2023-03-03T19:33:04.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.474+0000 7f49a6553700 1 -- start start 2023-03-03T19:33:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f49a6553700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f49a0151d60 0x7f49a00c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f49a6553700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f49a00bffe0 0x7f49a0158020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f49a6553700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a0157b90 con 0x7f49a0158560 2023-03-03T19:33:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f49a6553700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a00c5590 con 0x7f49a00bffe0 2023-03-03T19:33:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f49a6553700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a00bf6a0 con 0x7f49a0151d60 2023-03-03T19:33:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f499f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f49a00bffe0 0x7f49a0158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:04.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f49a4af0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f49a0158560 0x7f49a00c6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35986/0 (socket says 172.21.15.7:35986) 2023-03-03T19:33:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f49a4af0700 1 -- 172.21.15.7:0/3402952926 learned_addr learned my addr 172.21.15.7:0/3402952926 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:33:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f499ffff700 1 --2- 172.21.15.7:0/3402952926 >> v2:172.21.15.143:3300/0 conn(0x7f49a0151d60 0x7f49a00c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f499f7fe700 1 -- 172.21.15.7:0/3402952926 >> v2:172.21.15.143:3300/0 conn(0x7f49a0151d60 msgr2=0x7f49a00c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:33:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f499f7fe700 1 --2- 172.21.15.7:0/3402952926 >> v2:172.21.15.143:3300/0 conn(0x7f49a0151d60 0x7f49a00c4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f499f7fe700 1 -- 172.21.15.7:0/3402952926 >> v1:172.21.15.7:6789/0 conn(0x7f49a0158560 legacy=0x7f49a00c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:04.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.475+0000 7f499effd700 1 -- 172.21.15.7:0/3402952926 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057426959 0 0) 0x7f49a0157b90 con 0x7f49a0158560 2023-03-03T19:33:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f499f7fe700 1 -- 172.21.15.7:0/3402952926 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a00bf980 con 0x7f49a00bffe0 2023-03-03T19:33:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f499f7fe700 1 --2- 172.21.15.7:0/3402952926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 0x7f49a0158020 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f49a00ca570 tx=0x7f4994009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fcfa21271f5b9b9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:33:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f499effd700 1 -- 172.21.15.7:0/3402952926 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4994011830 con 0x7f49a00bffe0 2023-03-03T19:33:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f499effd700 1 -- 172.21.15.7:0/3402952926 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4994011990 con 0x7f49a00bffe0 2023-03-03T19:33:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f499effd700 1 -- 172.21.15.7:0/3402952926 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4994011d20 con 0x7f49a00bffe0 2023-03-03T19:33:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f49a6553700 1 -- 172.21.15.7:0/3402952926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 msgr2=0x7f49a0158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:04.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f49a6553700 1 --2- 172.21.15.7:0/3402952926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 0x7f49a0158020 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f49a00ca570 tx=0x7f4994009b20 comp rx=0 tx=0).stop 2023-03-03T19:33:04.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f49a6553700 1 -- 172.21.15.7:0/3402952926 shutdown_connections 2023-03-03T19:33:04.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f49a6553700 1 --2- 172.21.15.7:0/3402952926 >> v2:172.21.15.143:3300/0 conn(0x7f49a0151d60 0x7f49a00c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:04.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f49a6553700 1 --2- 172.21.15.7:0/3402952926 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 0x7f49a0158020 unknown :-1 s=CLOSED pgs=3520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:04.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f49a6553700 1 -- 172.21.15.7:0/3402952926 >> 172.21.15.7:0/3402952926 conn(0x7f49a0071530 msgr2=0x7f49a0071900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:33:04.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.476+0000 7f49a6553700 1 -- 172.21.15.7:0/3402952926 shutdown_connections 2023-03-03T19:33:04.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.477+0000 7f49a6553700 1 -- 172.21.15.7:0/3402952926 wait complete. 2023-03-03T19:33:04.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.478+0000 7f49a6553700 1 Processor -- start 2023-03-03T19:33:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.478+0000 7f49a6553700 1 -- start start 2023-03-03T19:33:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f49a6553700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 0x7f49a01ddb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f49a6553700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a0151d60 0x7f49a00c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f49a6553700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a0158560 0x7f49a014bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f49a6553700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00c2840 con 0x7f49a00bffe0 2023-03-03T19:33:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f49a6553700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a01eddb0 con 0x7f49a0151d60 2023-03-03T19:33:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f49a6553700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00c1a00 con 0x7f49a0158560 2023-03-03T19:33:04.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 0x7f49a01ddb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 0x7f49a01ddb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42008/0 (socket says 172.21.15.7:42008) 2023-03-03T19:33:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499ffff700 1 -- 172.21.15.7:0/2238407631 learned_addr learned my addr 172.21.15.7:0/2238407631 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:33:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499f7fe700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a0151d60 0x7f49a00c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f49a4af0700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a0158560 0x7f49a014bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499ffff700 1 -- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a0158560 msgr2=0x7f49a014bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499ffff700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a0158560 0x7f49a014bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:04.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499ffff700 1 -- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a0151d60 msgr2=0x7f49a00c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499ffff700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a0151d60 0x7f49a00c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499ffff700 1 -- 172.21.15.7:0/2238407631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49940094a0 con 0x7f49a00bffe0 2023-03-03T19:33:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499f7fe700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a0151d60 0x7f49a00c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:33:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499ffff700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 0x7f49a01ddb30 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f499000e840 tx=0x7f499000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:33:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499cff9700 1 -- 172.21.15.7:0/2238407631 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990013610 con 0x7f49a00bffe0 2023-03-03T19:33:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499cff9700 1 -- 172.21.15.7:0/2238407631 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4990013770 con 0x7f49a00bffe0 2023-03-03T19:33:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f49a6553700 1 -- 172.21.15.7:0/2238407631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a0154620 con 0x7f49a00bffe0 2023-03-03T19:33:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f499cff9700 1 -- 172.21.15.7:0/2238407631 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990013b20 con 0x7f49a00bffe0 2023-03-03T19:33:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.479+0000 7f49a6553700 1 -- 172.21.15.7:0/2238407631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a00bfc60 con 0x7f49a00bffe0 2023-03-03T19:33:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.480+0000 7f49a6553700 1 -- 172.21.15.7:0/2238407631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a01589c0 con 0x7f49a00bffe0 2023-03-03T19:33:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.481+0000 7f499cff9700 1 -- 172.21.15.7:0/2238407631 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4990023070 con 0x7f49a00bffe0 2023-03-03T19:33:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.481+0000 7f499cff9700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f498c068320 0x7f498c06a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.481+0000 7f499cff9700 1 -- 172.21.15.7:0/2238407631 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4990085f20 con 0x7f49a00bffe0 2023-03-03T19:33:04.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.481+0000 7f499f7fe700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f498c068320 0x7f498c06a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.482+0000 7f499f7fe700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f498c068320 0x7f498c06a7d0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f49940090b0 tx=0x7f499401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:33:04.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.486+0000 7f499cff9700 1 -- 172.21.15.7:0/2238407631 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4990052150 con 0x7f49a00bffe0 2023-03-03T19:33:04.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.723+0000 7f49a6553700 1 -- 172.21.15.7:0/2238407631 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49a0074bb0 con 0x7f49a00bffe0 2023-03-03T19:33:04.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.723+0000 7f499cff9700 1 -- 172.21.15.7:0/2238407631 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4990055770 con 0x7f49a00bffe0 2023-03-03T19:33:04.724 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:33:04.724 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:33:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 -- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f498c068320 msgr2=0x7f498c06a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f498c068320 0x7f498c06a7d0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f49940090b0 tx=0x7f499401c040 comp rx=0 tx=0).stop 2023-03-03T19:33:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 -- 172.21.15.7:0/2238407631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 msgr2=0x7f49a01ddb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 0x7f49a01ddb30 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f499000e840 tx=0x7f499000ec00 comp rx=0 tx=0).stop 2023-03-03T19:33:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 -- 172.21.15.7:0/2238407631 shutdown_connections 2023-03-03T19:33:04.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f49a0158560 0x7f49a014bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f498c068320 0x7f498c06a7d0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f49a00bffe0 0x7f49a01ddb30 unknown :-1 s=CLOSED pgs=3521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 --2- 172.21.15.7:0/2238407631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a0151d60 0x7f49a00c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 -- 172.21.15.7:0/2238407631 >> 172.21.15.7:0/2238407631 conn(0x7f49a0071530 msgr2=0x7f49a01527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:33:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.725+0000 7f49a6553700 1 -- 172.21.15.7:0/2238407631 shutdown_connections 2023-03-03T19:33:04.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:04.726+0000 7f49a6553700 1 -- 172.21.15.7:0/2238407631 wait complete. 2023-03-03T19:33:04.728 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:33:04.738 INFO:tasks.mds_thrash.fs.[a]:waiting for 106 secs before thrashing 2023-03-03T19:33:19.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:33:19.313 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:33:49.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:33:49.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:33:50.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:33:50.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.416+0000 7f8a6b2ae700 1 Processor -- start 2023-03-03T19:33:50.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.416+0000 7f8a6b2ae700 1 -- start start 2023-03-03T19:33:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.417+0000 7f8a6b2ae700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8a640a9af0 0x7f8a640c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.417+0000 7f8a6b2ae700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a640a65b0 con 0x7f8a6414de50 2023-03-03T19:33:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.417+0000 7f8a6b2ae700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a641541f0 con 0x7f8a640c65c0 2023-03-03T19:33:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.417+0000 7f8a6b2ae700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a64154e00 con 0x7f8a640a9af0 2023-03-03T19:33:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.417+0000 7f8a6904a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8a6414de50 0x7f8a6406a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36016/0 (socket says 172.21.15.7:36016) 2023-03-03T19:33:50.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.417+0000 7f8a6904a700 1 -- 172.21.15.7:0/1291145224 learned_addr learned my addr 172.21.15.7:0/1291145224 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:33:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.417+0000 7f8a6984b700 1 --2- 172.21.15.7:0/1291145224 >> v2:172.21.15.143:3300/0 conn(0x7f8a640a9af0 0x7f8a640c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.417+0000 7f8a5bfff700 1 -- 172.21.15.7:0/1291145224 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593281914 0 0) 0x7f8a640a65b0 con 0x7f8a6414de50 2023-03-03T19:33:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.417+0000 7f8a5bfff700 1 -- 172.21.15.7:0/1291145224 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a500035b0 con 0x7f8a6414de50 2023-03-03T19:33:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.418+0000 7f8a6984b700 1 -- 172.21.15.7:0/1291145224 >> v1:172.21.15.143:6789/0 conn(0x7f8a640c65c0 legacy=0x7f8a640c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.418+0000 7f8a5bfff700 1 -- 172.21.15.7:0/1291145224 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967622953 0 0) 0x7f8a641541f0 con 0x7f8a640c65c0 2023-03-03T19:33:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.418+0000 7f8a6984b700 1 -- 172.21.15.7:0/1291145224 >> v1:172.21.15.7:6789/0 conn(0x7f8a6414de50 legacy=0x7f8a6406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.418+0000 7f8a6984b700 1 -- 172.21.15.7:0/1291145224 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a641544d0 con 0x7f8a640a9af0 2023-03-03T19:33:50.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.418+0000 7f8a6984b700 1 --2- 172.21.15.7:0/1291145224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a640a9af0 0x7f8a640c2e30 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00aea0 tx=0x7f8a5c00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d4a6f2eee6a8677 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:33:50.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.418+0000 7f8a5bfff700 1 -- 172.21.15.7:0/1291145224 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c00f070 con 0x7f8a640a9af0 2023-03-03T19:33:50.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.418+0000 7f8a5bfff700 1 -- 172.21.15.7:0/1291145224 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5c00c910 con 0x7f8a640a9af0 2023-03-03T19:33:50.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.418+0000 7f8a5bfff700 1 -- 172.21.15.7:0/1291145224 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c00ccc0 con 0x7f8a640a9af0 2023-03-03T19:33:50.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.418+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/1291145224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a640a9af0 msgr2=0x7f8a640c2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:50.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.418+0000 7f8a6b2ae700 1 --2- 172.21.15.7:0/1291145224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a640a9af0 0x7f8a640c2e30 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c00aea0 tx=0x7f8a5c00c330 comp rx=0 tx=0).stop 2023-03-03T19:33:50.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.419+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/1291145224 shutdown_connections 2023-03-03T19:33:50.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.419+0000 7f8a6b2ae700 1 --2- 172.21.15.7:0/1291145224 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a640a9af0 0x7f8a640c2e30 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:50.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.419+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/1291145224 >> 172.21.15.7:0/1291145224 conn(0x7f8a640a9430 msgr2=0x7f8a640c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:33:50.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.419+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/1291145224 shutdown_connections 2023-03-03T19:33:50.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.419+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/1291145224 wait complete. 2023-03-03T19:33:50.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.420+0000 7f8a6b2ae700 1 Processor -- start 2023-03-03T19:33:50.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.420+0000 7f8a6b2ae700 1 -- start start 2023-03-03T19:33:50.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a6b2ae700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a640a9af0 0x7f8a640c3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:50.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a6b2ae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a640c65c0 0x7f8a641e7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:50.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a6b2ae700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6414de50 0x7f8a641e7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:50.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a6b2ae700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a641eb900 con 0x7f8a640c65c0 2023-03-03T19:33:50.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a6b2ae700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6406fbc0 con 0x7f8a6414de50 2023-03-03T19:33:50.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a6b2ae700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a641e82c0 con 0x7f8a640a9af0 2023-03-03T19:33:50.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a68849700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a640c65c0 0x7f8a641e7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:50.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a68849700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a640c65c0 0x7f8a641e7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42044/0 (socket says 172.21.15.7:42044) 2023-03-03T19:33:50.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a68849700 1 -- 172.21.15.7:0/3633244032 learned_addr learned my addr 172.21.15.7:0/3633244032 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:33:50.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a6904a700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a640a9af0 0x7f8a640c3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:50.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.421+0000 7f8a6984b700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6414de50 0x7f8a641e7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:50.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a68849700 1 -- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a640a9af0 msgr2=0x7f8a640c3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a68849700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a640a9af0 0x7f8a640c3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a68849700 1 -- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6414de50 msgr2=0x7f8a641e7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a68849700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6414de50 0x7f8a641e7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a68849700 1 -- 172.21.15.7:0/3633244032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c00ab80 con 0x7f8a640c65c0 2023-03-03T19:33:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a68849700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a640c65c0 0x7f8a641e7840 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000c520 tx=0x7f8a6000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:33:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a59ffb700 1 -- 172.21.15.7:0/3633244032 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a600054e0 con 0x7f8a640c65c0 2023-03-03T19:33:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a59ffb700 1 -- 172.21.15.7:0/3633244032 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a60003680 con 0x7f8a640c65c0 2023-03-03T19:33:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/3633244032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a64154a90 con 0x7f8a640c65c0 2023-03-03T19:33:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a59ffb700 1 -- 172.21.15.7:0/3633244032 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60003a30 con 0x7f8a640c65c0 2023-03-03T19:33:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.422+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/3633244032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a641547b0 con 0x7f8a640c65c0 2023-03-03T19:33:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.423+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/3633244032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6406fa20 con 0x7f8a640c65c0 2023-03-03T19:33:50.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.424+0000 7f8a59ffb700 1 -- 172.21.15.7:0/3633244032 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a60019030 con 0x7f8a640c65c0 2023-03-03T19:33:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.424+0000 7f8a59ffb700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a480682f0 0x7f8a4806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:33:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.424+0000 7f8a59ffb700 1 -- 172.21.15.7:0/3633244032 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8a600853d0 con 0x7f8a640c65c0 2023-03-03T19:33:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.424+0000 7f8a6904a700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a480682f0 0x7f8a4806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:33:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.425+0000 7f8a6904a700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a480682f0 0x7f8a4806a7a0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f8a54007fd0 tx=0x7f8a54009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:33:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.428+0000 7f8a59ffb700 1 -- 172.21.15.7:0/3633244032 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8a6406fa20 con 0x7f8a640c65c0 2023-03-03T19:33:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.667+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/3633244032 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a6406fa20 con 0x7f8a640c65c0 2023-03-03T19:33:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.668+0000 7f8a59ffb700 1 -- 172.21.15.7:0/3633244032 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8a6406fa20 con 0x7f8a640c65c0 2023-03-03T19:33:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:33:50.669 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:33:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a480682f0 msgr2=0x7f8a4806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a480682f0 0x7f8a4806a7a0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f8a54007fd0 tx=0x7f8a54009040 comp rx=0 tx=0).stop 2023-03-03T19:33:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/3633244032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a640c65c0 msgr2=0x7f8a641e7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:33:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a640c65c0 0x7f8a641e7840 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7f8a6000c520 tx=0x7f8a6000c830 comp rx=0 tx=0).stop 2023-03-03T19:33:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/3633244032 shutdown_connections 2023-03-03T19:33:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8a6414de50 0x7f8a641e7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8a640a9af0 0x7f8a640c3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8a480682f0 0x7f8a4806a7a0 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 --2- 172.21.15.7:0/3633244032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a640c65c0 0x7f8a641e7840 unknown :-1 s=CLOSED pgs=3523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:33:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/3633244032 >> 172.21.15.7:0/3633244032 conn(0x7f8a640a9430 msgr2=0x7f8a640c56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:33:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/3633244032 shutdown_connections 2023-03-03T19:33:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:33:50.670+0000 7f8a6b2ae700 1 -- 172.21.15.7:0/3633244032 wait complete. 2023-03-03T19:33:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:33:50.682 INFO:tasks.mds_thrash.fs.[b]:waiting for 18 secs before thrashing 2023-03-03T19:34:08.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:34:08.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.821+0000 7f4a834e9700 1 Processor -- start 2023-03-03T19:34:08.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.821+0000 7f4a834e9700 1 -- start start 2023-03-03T19:34:08.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a834e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 0x7f4a7c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:08.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a834e9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c0c2f30 con 0x7f4a7c0abe20 2023-03-03T19:34:08.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a834e9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a7c0c31d0 con 0x7f4a7c14fda0 2023-03-03T19:34:08.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a834e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c1524a0 con 0x7f4a7c14f9d0 2023-03-03T19:34:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a81a86700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4a7c0abe20 0x7f4a7c1529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36040/0 (socket says 172.21.15.7:36040) 2023-03-03T19:34:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a81a86700 1 -- 172.21.15.7:0/4100606050 learned_addr learned my addr 172.21.15.7:0/4100606050 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:34:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a81285700 1 --2- 172.21.15.7:0/4100606050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 0x7f4a7c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a73fff700 1 -- 172.21.15.7:0/4100606050 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011249047 0 0) 0x7f4a7c0c2f30 con 0x7f4a7c0abe20 2023-03-03T19:34:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a73fff700 1 -- 172.21.15.7:0/4100606050 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a680035b0 con 0x7f4a7c0abe20 2023-03-03T19:34:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a73fff700 1 -- 172.21.15.7:0/4100606050 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837818046 0 0) 0x7f4a7c0c31d0 con 0x7f4a7c14fda0 2023-03-03T19:34:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a73fff700 1 -- 172.21.15.7:0/4100606050 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c0c2f30 con 0x7f4a7c14fda0 2023-03-03T19:34:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a81285700 1 -- 172.21.15.7:0/4100606050 >> v1:172.21.15.143:6789/0 conn(0x7f4a7c14fda0 legacy=0x7f4a7c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a81285700 1 -- 172.21.15.7:0/4100606050 >> v1:172.21.15.7:6789/0 conn(0x7f4a7c0abe20 legacy=0x7f4a7c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.822+0000 7f4a81285700 1 -- 172.21.15.7:0/4100606050 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c0c34b0 con 0x7f4a7c14f9d0 2023-03-03T19:34:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.823+0000 7f4a81285700 1 --2- 172.21.15.7:0/4100606050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 0x7f4a7c156430 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009870 tx=0x7f4a78009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89d8db97d110593b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:34:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.823+0000 7f4a73fff700 1 -- 172.21.15.7:0/4100606050 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a78011830 con 0x7f4a7c14f9d0 2023-03-03T19:34:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.823+0000 7f4a73fff700 1 -- 172.21.15.7:0/4100606050 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a78011990 con 0x7f4a7c14f9d0 2023-03-03T19:34:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.823+0000 7f4a73fff700 1 -- 172.21.15.7:0/4100606050 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a78011d20 con 0x7f4a7c14f9d0 2023-03-03T19:34:08.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.823+0000 7f4a834e9700 1 -- 172.21.15.7:0/4100606050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 msgr2=0x7f4a7c156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:08.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.823+0000 7f4a834e9700 1 --2- 172.21.15.7:0/4100606050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 0x7f4a7c156430 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009870 tx=0x7f4a78009b80 comp rx=0 tx=0).stop 2023-03-03T19:34:08.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.823+0000 7f4a834e9700 1 -- 172.21.15.7:0/4100606050 shutdown_connections 2023-03-03T19:34:08.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.823+0000 7f4a834e9700 1 --2- 172.21.15.7:0/4100606050 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 0x7f4a7c156430 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:08.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.823+0000 7f4a834e9700 1 -- 172.21.15.7:0/4100606050 >> 172.21.15.7:0/4100606050 conn(0x7f4a7c0c8900 msgr2=0x7f4a7c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:34:08.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.824+0000 7f4a834e9700 1 -- 172.21.15.7:0/4100606050 shutdown_connections 2023-03-03T19:34:08.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.824+0000 7f4a834e9700 1 -- 172.21.15.7:0/4100606050 wait complete. 2023-03-03T19:34:08.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.825+0000 7f4a834e9700 1 Processor -- start 2023-03-03T19:34:08.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.825+0000 7f4a834e9700 1 -- start start 2023-03-03T19:34:08.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.825+0000 7f4a834e9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a7c0abe20 0x7f4a7c06bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.825+0000 7f4a834e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 0x7f4a7c1e9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.825+0000 7f4a834e9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a7c14fda0 0x7f4a7c150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.825+0000 7f4a834e9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c146d20 con 0x7f4a7c0abe20 2023-03-03T19:34:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.825+0000 7f4a834e9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c1dc200 con 0x7f4a7c14fda0 2023-03-03T19:34:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.825+0000 7f4a834e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c1ea240 con 0x7f4a7c14f9d0 2023-03-03T19:34:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.825+0000 7f4a81285700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a7c0abe20 0x7f4a7c06bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.825+0000 7f4a81285700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a7c0abe20 0x7f4a7c06bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42062/0 (socket says 172.21.15.7:42062) 2023-03-03T19:34:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a81285700 1 -- 172.21.15.7:0/1203256464 learned_addr learned my addr 172.21.15.7:0/1203256464 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:34:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a80a84700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 0x7f4a7c1e9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a81a86700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a7c14fda0 0x7f4a7c150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a81285700 1 -- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 msgr2=0x7f4a7c1e9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a81285700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 0x7f4a7c1e9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a81285700 1 -- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a7c14fda0 msgr2=0x7f4a7c150ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a81285700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a7c14fda0 0x7f4a7c150ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:08.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a81285700 1 -- 172.21.15.7:0/1203256464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a780094a0 con 0x7f4a7c0abe20 2023-03-03T19:34:08.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a81285700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a7c0abe20 0x7f4a7c06bde0 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7f4a78014040 tx=0x7f4a78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:34:08.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a71ffb700 1 -- 172.21.15.7:0/1203256464 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7801a8f0 con 0x7f4a7c0abe20 2023-03-03T19:34:08.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a71ffb700 1 -- 172.21.15.7:0/1203256464 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7801aa50 con 0x7f4a7c0abe20 2023-03-03T19:34:08.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a834e9700 1 -- 172.21.15.7:0/1203256464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7c0ac350 con 0x7f4a7c0abe20 2023-03-03T19:34:08.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.826+0000 7f4a71ffb700 1 -- 172.21.15.7:0/1203256464 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7801adc0 con 0x7f4a7c0abe20 2023-03-03T19:34:08.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.827+0000 7f4a834e9700 1 -- 172.21.15.7:0/1203256464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7c0ab630 con 0x7f4a7c0abe20 2023-03-03T19:34:08.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.828+0000 7f4a834e9700 1 -- 172.21.15.7:0/1203256464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7c071fa0 con 0x7f4a7c0abe20 2023-03-03T19:34:08.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.828+0000 7f4a71ffb700 1 -- 172.21.15.7:0/1203256464 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4a7800f050 con 0x7f4a7c0abe20 2023-03-03T19:34:08.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.828+0000 7f4a71ffb700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a600682f0 0x7f4a6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:08.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.829+0000 7f4a71ffb700 1 -- 172.21.15.7:0/1203256464 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4a780871c0 con 0x7f4a7c0abe20 2023-03-03T19:34:08.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.829+0000 7f4a80a84700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a600682f0 0x7f4a6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:08.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.829+0000 7f4a80a84700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a600682f0 0x7f4a6006a7a0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c0aaa30 tx=0x7f4a6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:34:08.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:08.833+0000 7f4a71ffb700 1 -- 172.21.15.7:0/1203256464 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4a780534e0 con 0x7f4a7c0abe20 2023-03-03T19:34:09.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.069+0000 7f4a834e9700 1 -- 172.21.15.7:0/1203256464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a7c074bb0 con 0x7f4a7c0abe20 2023-03-03T19:34:09.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.070+0000 7f4a71ffb700 1 -- 172.21.15.7:0/1203256464 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4a78015020 con 0x7f4a7c0abe20 2023-03-03T19:34:09.070 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:34:09.071 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:34:09.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 -- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a600682f0 msgr2=0x7f4a6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a600682f0 0x7f4a6006a7a0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c0aaa30 tx=0x7f4a6c009040 comp rx=0 tx=0).stop 2023-03-03T19:34:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 -- 172.21.15.7:0/1203256464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a7c0abe20 msgr2=0x7f4a7c06bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a7c0abe20 0x7f4a7c06bde0 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7f4a78014040 tx=0x7f4a78004750 comp rx=0 tx=0).stop 2023-03-03T19:34:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 -- 172.21.15.7:0/1203256464 shutdown_connections 2023-03-03T19:34:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a7c14fda0 0x7f4a7c150ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4a600682f0 0x7f4a6006a7a0 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:09.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4a7c0abe20 0x7f4a7c06bde0 unknown :-1 s=CLOSED pgs=3525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:09.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 --2- 172.21.15.7:0/1203256464 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4a7c14f9d0 0x7f4a7c1e9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:09.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 -- 172.21.15.7:0/1203256464 >> 172.21.15.7:0/1203256464 conn(0x7f4a7c0c8900 msgr2=0x7f4a7c0c4120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:34:09.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 -- 172.21.15.7:0/1203256464 shutdown_connections 2023-03-03T19:34:09.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:09.072+0000 7f4a834e9700 1 -- 172.21.15.7:0/1203256464 wait complete. 2023-03-03T19:34:09.074 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:34:09.084 INFO:tasks.mds_thrash.fs.[b]:waiting for 67 secs before thrashing 2023-03-03T19:34:19.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:34:19.545 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:34:49.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:34:49.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:34:50.739 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:34:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.877+0000 7f436522d700 1 Processor -- start 2023-03-03T19:34:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.877+0000 7f436522d700 1 -- start start 2023-03-03T19:34:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f436522d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f436006e020 0x7f4360071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f436522d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4360072520 0x7f4360151770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f436522d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4360154910 con 0x7f4360155390 2023-03-03T19:34:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f436522d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4360070b90 con 0x7f4360072520 2023-03-03T19:34:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f436522d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4360151320 con 0x7f436006e020 2023-03-03T19:34:50.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435f59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4360072520 0x7f4360151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:50.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435ed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4360155390 0x7f43600c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36110/0 (socket says 172.21.15.7:36110) 2023-03-03T19:34:50.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435ed9d700 1 -- 172.21.15.7:0/3811765436 learned_addr learned my addr 172.21.15.7:0/3811765436 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:34:50.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435f59e700 1 -- 172.21.15.7:0/3811765436 >> v2:172.21.15.143:3300/0 conn(0x7f436006e020 msgr2=0x7f4360071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:34:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435e59c700 1 --2- 172.21.15.7:0/3811765436 >> v2:172.21.15.143:3300/0 conn(0x7f436006e020 0x7f4360071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435f59e700 1 --2- 172.21.15.7:0/3811765436 >> v2:172.21.15.143:3300/0 conn(0x7f436006e020 0x7f4360071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435f59e700 1 -- 172.21.15.7:0/3811765436 >> v1:172.21.15.7:6789/0 conn(0x7f4360155390 legacy=0x7f43600c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435dd9b700 1 -- 172.21.15.7:0/3811765436 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196462590 0 0) 0x7f4360154910 con 0x7f4360155390 2023-03-03T19:34:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435f59e700 1 -- 172.21.15.7:0/3811765436 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43600c4540 con 0x7f4360072520 2023-03-03T19:34:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435e59c700 1 --2- 172.21.15.7:0/3811765436 >> v2:172.21.15.143:3300/0 conn(0x7f436006e020 0x7f4360071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:34:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.878+0000 7f435f59e700 1 --2- 172.21.15.7:0/3811765436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4360072520 0x7f4360151770 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7f43600ac9e0 tx=0x7f435800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bfc013b2ba27f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:34:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f435dd9b700 1 -- 172.21.15.7:0/3811765436 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435800e070 con 0x7f4360072520 2023-03-03T19:34:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f435dd9b700 1 -- 172.21.15.7:0/3811765436 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435800bbd0 con 0x7f4360072520 2023-03-03T19:34:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f435dd9b700 1 -- 172.21.15.7:0/3811765436 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4358012430 con 0x7f4360072520 2023-03-03T19:34:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f436522d700 1 -- 172.21.15.7:0/3811765436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4360072520 msgr2=0x7f4360151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f436522d700 1 --2- 172.21.15.7:0/3811765436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4360072520 0x7f4360151770 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7f43600ac9e0 tx=0x7f435800b330 comp rx=0 tx=0).stop 2023-03-03T19:34:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f436522d700 1 -- 172.21.15.7:0/3811765436 shutdown_connections 2023-03-03T19:34:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f436522d700 1 --2- 172.21.15.7:0/3811765436 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4360072520 0x7f4360151770 unknown :-1 s=CLOSED pgs=3527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f436522d700 1 --2- 172.21.15.7:0/3811765436 >> v2:172.21.15.143:3300/0 conn(0x7f436006e020 0x7f4360071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f436522d700 1 -- 172.21.15.7:0/3811765436 >> 172.21.15.7:0/3811765436 conn(0x7f436006c110 msgr2=0x7f436006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:34:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f436522d700 1 -- 172.21.15.7:0/3811765436 shutdown_connections 2023-03-03T19:34:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.879+0000 7f436522d700 1 -- 172.21.15.7:0/3811765436 wait complete. 2023-03-03T19:34:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.881+0000 7f436522d700 1 Processor -- start 2023-03-03T19:34:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.881+0000 7f436522d700 1 -- start start 2023-03-03T19:34:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.881+0000 7f436522d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436006e020 0x7f43600aae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.881+0000 7f436522d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4360072520 0x7f43600c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.881+0000 7f436522d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4360155390 0x7f43601ea720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.881+0000 7f436522d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43600abb80 con 0x7f436006e020 2023-03-03T19:34:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.881+0000 7f436522d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43600c5c10 con 0x7f4360155390 2023-03-03T19:34:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.881+0000 7f436522d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43600aa660 con 0x7f4360072520 2023-03-03T19:34:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436006e020 0x7f43600aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436006e020 0x7f43600aae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42136/0 (socket says 172.21.15.7:42136) 2023-03-03T19:34:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435ed9d700 1 -- 172.21.15.7:0/2338637100 learned_addr learned my addr 172.21.15.7:0/2338637100 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:34:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435e59c700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4360072520 0x7f43600c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435ed9d700 1 -- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4360072520 msgr2=0x7f43600c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435ed9d700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4360072520 0x7f43600c6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435f59e700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4360155390 0x7f43601ea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435ed9d700 1 -- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4360155390 msgr2=0x7f43601ea720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435ed9d700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4360155390 0x7f43601ea720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435ed9d700 1 -- 172.21.15.7:0/2338637100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4358009a70 con 0x7f436006e020 2023-03-03T19:34:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f435ed9d700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436006e020 0x7f43600aae40 secure :-1 s=READY pgs=3528 cs=0 l=1 rev1=1 crypto rx=0x7f435400c550 tx=0x7f435400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:34:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f43477fe700 1 -- 172.21.15.7:0/2338637100 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354004c80 con 0x7f436006e020 2023-03-03T19:34:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f43477fe700 1 -- 172.21.15.7:0/2338637100 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4354004e00 con 0x7f436006e020 2023-03-03T19:34:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f436522d700 1 -- 172.21.15.7:0/2338637100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4360070540 con 0x7f436006e020 2023-03-03T19:34:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f43477fe700 1 -- 172.21.15.7:0/2338637100 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4354007310 con 0x7f436006e020 2023-03-03T19:34:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.882+0000 7f436522d700 1 -- 172.21.15.7:0/2338637100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43600702a0 con 0x7f436006e020 2023-03-03T19:34:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.883+0000 7f436522d700 1 -- 172.21.15.7:0/2338637100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4360070820 con 0x7f436006e020 2023-03-03T19:34:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.884+0000 7f43477fe700 1 -- 172.21.15.7:0/2338637100 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f435401a020 con 0x7f436006e020 2023-03-03T19:34:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.884+0000 7f43477fe700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f434c0682f0 0x7f434c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:34:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.884+0000 7f43477fe700 1 -- 172.21.15.7:0/2338637100 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4354085da0 con 0x7f436006e020 2023-03-03T19:34:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.884+0000 7f435e59c700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f434c0682f0 0x7f434c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:34:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.885+0000 7f435e59c700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f434c0682f0 0x7f434c06a7a0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f4350006fd0 tx=0x7f4350008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:34:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:50.889+0000 7f43477fe700 1 -- 172.21.15.7:0/2338637100 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f43540521f0 con 0x7f436006e020 2023-03-03T19:34:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.130+0000 7f436522d700 1 -- 172.21.15.7:0/2338637100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4360074bb0 con 0x7f436006e020 2023-03-03T19:34:51.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.131+0000 7f43477fe700 1 -- 172.21.15.7:0/2338637100 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4354055810 con 0x7f436006e020 2023-03-03T19:34:51.132 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:34:51.133 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:34:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 -- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f434c0682f0 msgr2=0x7f434c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f434c0682f0 0x7f434c06a7a0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f4350006fd0 tx=0x7f4350008040 comp rx=0 tx=0).stop 2023-03-03T19:34:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 -- 172.21.15.7:0/2338637100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436006e020 msgr2=0x7f43600aae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:34:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436006e020 0x7f43600aae40 secure :-1 s=READY pgs=3528 cs=0 l=1 rev1=1 crypto rx=0x7f435400c550 tx=0x7f435400c910 comp rx=0 tx=0).stop 2023-03-03T19:34:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 -- 172.21.15.7:0/2338637100 shutdown_connections 2023-03-03T19:34:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4360155390 0x7f43601ea720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f434c0682f0 0x7f434c06a7a0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f436006e020 0x7f43600aae40 unknown :-1 s=CLOSED pgs=3528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 --2- 172.21.15.7:0/2338637100 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4360072520 0x7f43600c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:34:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 -- 172.21.15.7:0/2338637100 >> 172.21.15.7:0/2338637100 conn(0x7f436006c110 msgr2=0x7f436014e230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:34:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 -- 172.21.15.7:0/2338637100 shutdown_connections 2023-03-03T19:34:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:34:51.133+0000 7f436522d700 1 -- 172.21.15.7:0/2338637100 wait complete. 2023-03-03T19:34:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:34:51.146 INFO:tasks.mds_thrash.fs.[a]:waiting for 31 secs before thrashing 2023-03-03T19:35:16.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:35:16.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.225+0000 7f3f28df0700 1 Processor -- start 2023-03-03T19:35:16.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.225+0000 7f3f28df0700 1 -- start start 2023-03-03T19:35:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f28df0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3f24151ab0 0x7f3f240736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f28df0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f3f2406c5b0 0x7f3f24157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:16.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f28df0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2406d780 con 0x7f3f24073c30 2023-03-03T19:35:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f28df0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2406cb70 con 0x7f3f24151ab0 2023-03-03T19:35:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f28df0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2406ce50 con 0x7f3f2406c5b0 2023-03-03T19:35:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f2259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3f24151ab0 0x7f3f240736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f21d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3f24073c30 0x7f3f2406bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36160/0 (socket says 172.21.15.7:36160) 2023-03-03T19:35:16.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f21d9b700 1 -- 172.21.15.7:0/1591602214 learned_addr learned my addr 172.21.15.7:0/1591602214 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:35:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f22d9d700 1 --2- 172.21.15.7:0/1591602214 >> v2:172.21.15.143:3300/0 conn(0x7f3f2406c5b0 0x7f3f24157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f2259c700 1 -- 172.21.15.7:0/1591602214 >> v2:172.21.15.143:3300/0 conn(0x7f3f2406c5b0 msgr2=0x7f3f24157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:35:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f2259c700 1 --2- 172.21.15.7:0/1591602214 >> v2:172.21.15.143:3300/0 conn(0x7f3f2406c5b0 0x7f3f24157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f2259c700 1 -- 172.21.15.7:0/1591602214 >> v1:172.21.15.7:6789/0 conn(0x7f3f24073c30 legacy=0x7f3f2406bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f2159a700 1 -- 172.21.15.7:0/1591602214 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062020886 0 0) 0x7f3f2406d780 con 0x7f3f24073c30 2023-03-03T19:35:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f2259c700 1 -- 172.21.15.7:0/1591602214 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2406d130 con 0x7f3f24151ab0 2023-03-03T19:35:16.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f2259c700 1 --2- 172.21.15.7:0/1591602214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24151ab0 0x7f3f240736f0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7f3f24074080 tx=0x7f3f14009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39c61fad15b3d84 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:35:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.226+0000 7f3f2159a700 1 -- 172.21.15.7:0/1591602214 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14011830 con 0x7f3f24151ab0 2023-03-03T19:35:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.227+0000 7f3f2159a700 1 -- 172.21.15.7:0/1591602214 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f14011990 con 0x7f3f24151ab0 2023-03-03T19:35:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.227+0000 7f3f2159a700 1 -- 172.21.15.7:0/1591602214 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14011d20 con 0x7f3f24151ab0 2023-03-03T19:35:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.227+0000 7f3f28df0700 1 -- 172.21.15.7:0/1591602214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24151ab0 msgr2=0x7f3f240736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:16.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.227+0000 7f3f28df0700 1 --2- 172.21.15.7:0/1591602214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24151ab0 0x7f3f240736f0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7f3f24074080 tx=0x7f3f14009b20 comp rx=0 tx=0).stop 2023-03-03T19:35:16.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.227+0000 7f3f28df0700 1 -- 172.21.15.7:0/1591602214 shutdown_connections 2023-03-03T19:35:16.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.227+0000 7f3f28df0700 1 --2- 172.21.15.7:0/1591602214 >> v2:172.21.15.143:3300/0 conn(0x7f3f2406c5b0 0x7f3f24157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:16.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.227+0000 7f3f28df0700 1 --2- 172.21.15.7:0/1591602214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24151ab0 0x7f3f240736f0 unknown :-1 s=CLOSED pgs=3530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:16.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.227+0000 7f3f28df0700 1 -- 172.21.15.7:0/1591602214 >> 172.21.15.7:0/1591602214 conn(0x7f3f240a94d0 msgr2=0x7f3f240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:35:16.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.227+0000 7f3f28df0700 1 -- 172.21.15.7:0/1591602214 shutdown_connections 2023-03-03T19:35:16.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.227+0000 7f3f28df0700 1 -- 172.21.15.7:0/1591602214 wait complete. 2023-03-03T19:35:16.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.229+0000 7f3f28df0700 1 Processor -- start 2023-03-03T19:35:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.229+0000 7f3f28df0700 1 -- start start 2023-03-03T19:35:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.229+0000 7f3f28df0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f2406c5b0 0x7f3f2414f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.229+0000 7f3f28df0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24073c30 0x7f3f2414b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.229+0000 7f3f28df0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f24151ab0 0x7f3f2414b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.229+0000 7f3f28df0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f240bf530 con 0x7f3f24073c30 2023-03-03T19:35:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.229+0000 7f3f28df0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f240bff90 con 0x7f3f2406c5b0 2023-03-03T19:35:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.229+0000 7f3f28df0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2414c410 con 0x7f3f24151ab0 2023-03-03T19:35:16.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.229+0000 7f3f21d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24073c30 0x7f3f2414b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f21d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24073c30 0x7f3f2414b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42184/0 (socket says 172.21.15.7:42184) 2023-03-03T19:35:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f21d9b700 1 -- 172.21.15.7:0/2442384087 learned_addr learned my addr 172.21.15.7:0/2442384087 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:35:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f2259c700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f2406c5b0 0x7f3f2414f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f22d9d700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f24151ab0 0x7f3f2414b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f21d9b700 1 -- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f24151ab0 msgr2=0x7f3f2414b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f21d9b700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f24151ab0 0x7f3f2414b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f21d9b700 1 -- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f2406c5b0 msgr2=0x7f3f2414f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:16.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f21d9b700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f2406c5b0 0x7f3f2414f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f21d9b700 1 -- 172.21.15.7:0/2442384087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f140094a0 con 0x7f3f24073c30 2023-03-03T19:35:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f2259c700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f2406c5b0 0x7f3f2414f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:35:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f21d9b700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24073c30 0x7f3f2414b390 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c0011e0 tx=0x7f3f0c00cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:35:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f1affd700 1 -- 172.21.15.7:0/2442384087 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c004c80 con 0x7f3f24073c30 2023-03-03T19:35:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f1affd700 1 -- 172.21.15.7:0/2442384087 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c004e00 con 0x7f3f24073c30 2023-03-03T19:35:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f1affd700 1 -- 172.21.15.7:0/2442384087 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c007310 con 0x7f3f24073c30 2023-03-03T19:35:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f28df0700 1 -- 172.21.15.7:0/2442384087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f24151e80 con 0x7f3f24073c30 2023-03-03T19:35:16.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.230+0000 7f3f28df0700 1 -- 172.21.15.7:0/2442384087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2406d410 con 0x7f3f24073c30 2023-03-03T19:35:16.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.231+0000 7f3f28df0700 1 -- 172.21.15.7:0/2442384087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f24152010 con 0x7f3f24073c30 2023-03-03T19:35:16.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.232+0000 7f3f1affd700 1 -- 172.21.15.7:0/2442384087 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3f0c01a020 con 0x7f3f24073c30 2023-03-03T19:35:16.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.232+0000 7f3f1affd700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f040682f0 0x7f3f0406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:16.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.232+0000 7f3f1affd700 1 -- 172.21.15.7:0/2442384087 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3f0c085dd0 con 0x7f3f24073c30 2023-03-03T19:35:16.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.232+0000 7f3f2259c700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f040682f0 0x7f3f0406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:16.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.233+0000 7f3f2259c700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f040682f0 0x7f3f0406a7a0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f3f140090b0 tx=0x7f3f1401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:35:16.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.237+0000 7f3f1affd700 1 -- 172.21.15.7:0/2442384087 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3f0c052170 con 0x7f3f24073c30 2023-03-03T19:35:16.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.474+0000 7f3f28df0700 1 -- 172.21.15.7:0/2442384087 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f240c3ba0 con 0x7f3f24073c30 2023-03-03T19:35:16.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.474+0000 7f3f1affd700 1 -- 172.21.15.7:0/2442384087 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3f0c055790 con 0x7f3f24073c30 2023-03-03T19:35:16.475 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:35:16.475 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:35:16.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.476+0000 7f3f28df0700 1 -- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f040682f0 msgr2=0x7f3f0406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:16.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.476+0000 7f3f28df0700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f040682f0 0x7f3f0406a7a0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f3f140090b0 tx=0x7f3f1401c040 comp rx=0 tx=0).stop 2023-03-03T19:35:16.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.476+0000 7f3f28df0700 1 -- 172.21.15.7:0/2442384087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24073c30 msgr2=0x7f3f2414b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:16.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.476+0000 7f3f28df0700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24073c30 0x7f3f2414b390 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c0011e0 tx=0x7f3f0c00cae0 comp rx=0 tx=0).stop 2023-03-03T19:35:16.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.476+0000 7f3f28df0700 1 -- 172.21.15.7:0/2442384087 shutdown_connections 2023-03-03T19:35:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.476+0000 7f3f28df0700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3f24151ab0 0x7f3f2414b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.476+0000 7f3f28df0700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3f2406c5b0 0x7f3f2414f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.476+0000 7f3f28df0700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3f040682f0 0x7f3f0406a7a0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.476+0000 7f3f28df0700 1 --2- 172.21.15.7:0/2442384087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f24073c30 0x7f3f2414b390 unknown :-1 s=CLOSED pgs=3531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.476+0000 7f3f28df0700 1 -- 172.21.15.7:0/2442384087 >> 172.21.15.7:0/2442384087 conn(0x7f3f240a94d0 msgr2=0x7f3f240c3070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:35:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.477+0000 7f3f28df0700 1 -- 172.21.15.7:0/2442384087 shutdown_connections 2023-03-03T19:35:16.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:16.477+0000 7f3f28df0700 1 -- 172.21.15.7:0/2442384087 wait complete. 2023-03-03T19:35:16.479 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:35:16.489 INFO:tasks.mds_thrash.fs.[b]:waiting for 110 secs before thrashing 2023-03-03T19:35:19.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:35:19.698 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:35:22.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:35:22.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.288+0000 7fb7f49d1700 1 Processor -- start 2023-03-03T19:35:22.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.288+0000 7fb7f49d1700 1 -- start start 2023-03-03T19:35:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7f49d1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f0152e30 0x7fb7f00c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:22.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7f49d1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb7f00beab0 0x7fb7f00c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7f49d1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f01523b0 con 0x7fb7f0155d80 2023-03-03T19:35:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7f49d1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f0156930 con 0x7fb7f00beab0 2023-03-03T19:35:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7f49d1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f0072010 con 0x7fb7f0152e30 2023-03-03T19:35:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7edd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb7f00beab0 0x7fb7f00c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7ee59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f0152e30 0x7fb7f00c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7edd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb7f00beab0 0x7fb7f00c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:37974/0 (socket says 172.21.15.7:37974) 2023-03-03T19:35:22.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7eed9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb7f0155d80 0x7fb7f0156220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:39696/0 (socket says 172.21.15.7:39696) 2023-03-03T19:35:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7edd9b700 1 -- 172.21.15.7:0/33636319 learned_addr learned my addr 172.21.15.7:0/33636319 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:35:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.289+0000 7fb7ee59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f0152e30 0x7fb7f00c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:50268/0 (socket says 172.21.15.7:50268) 2023-03-03T19:35:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.290+0000 7fb7ee59c700 1 -- 172.21.15.7:0/33636319 >> v2:172.21.15.143:3300/0 conn(0x7fb7f00beab0 msgr2=0x7fb7f00c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:35:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.290+0000 7fb7ee59c700 1 --2- 172.21.15.7:0/33636319 >> v2:172.21.15.143:3300/0 conn(0x7fb7f00beab0 0x7fb7f00c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.290+0000 7fb7ee59c700 1 -- 172.21.15.7:0/33636319 >> v1:172.21.15.143:6789/0 conn(0x7fb7f0155d80 legacy=0x7fb7f0156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.290+0000 7fb7ee59c700 1 -- 172.21.15.7:0/33636319 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f00aaaf0 con 0x7fb7f0152e30 2023-03-03T19:35:22.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.290+0000 7fb7ee59c700 1 --2- 172.21.15.7:0/33636319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f0152e30 0x7fb7f00c43d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009840 tx=0x7fb7e4009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fd1a5b11a65331b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:35:22.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.290+0000 7fb7ed59a700 1 -- 172.21.15.7:0/33636319 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4011830 con 0x7fb7f0152e30 2023-03-03T19:35:22.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.290+0000 7fb7ed59a700 1 -- 172.21.15.7:0/33636319 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e4011990 con 0x7fb7f0152e30 2023-03-03T19:35:22.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.290+0000 7fb7ed59a700 1 -- 172.21.15.7:0/33636319 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4011d20 con 0x7fb7f0152e30 2023-03-03T19:35:22.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.291+0000 7fb7f49d1700 1 -- 172.21.15.7:0/33636319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f0152e30 msgr2=0x7fb7f00c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:22.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.291+0000 7fb7f49d1700 1 --2- 172.21.15.7:0/33636319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f0152e30 0x7fb7f00c43d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009840 tx=0x7fb7e4009b50 comp rx=0 tx=0).stop 2023-03-03T19:35:22.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.291+0000 7fb7f49d1700 1 -- 172.21.15.7:0/33636319 shutdown_connections 2023-03-03T19:35:22.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.291+0000 7fb7f49d1700 1 --2- 172.21.15.7:0/33636319 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f0152e30 0x7fb7f00c43d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:22.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.291+0000 7fb7f49d1700 1 --2- 172.21.15.7:0/33636319 >> v2:172.21.15.143:3300/0 conn(0x7fb7f00beab0 0x7fb7f00c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:22.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.291+0000 7fb7f49d1700 1 -- 172.21.15.7:0/33636319 >> 172.21.15.7:0/33636319 conn(0x7fb7f00736f0 msgr2=0x7fb7f0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:35:22.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.291+0000 7fb7f49d1700 1 -- 172.21.15.7:0/33636319 shutdown_connections 2023-03-03T19:35:22.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.291+0000 7fb7f49d1700 1 -- 172.21.15.7:0/33636319 wait complete. 2023-03-03T19:35:22.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.292+0000 7fb7f49d1700 1 Processor -- start 2023-03-03T19:35:22.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.292+0000 7fb7f49d1700 1 -- start start 2023-03-03T19:35:22.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7f49d1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f00beab0 0x7fb7f00c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:22.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7f49d1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7f0152e30 0x7fb7f014b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:22.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7f49d1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7f0155d80 0x7fb7f014b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:22.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7f49d1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f014bcd0 con 0x7fb7f0152e30 2023-03-03T19:35:22.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7f49d1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f01538e0 con 0x7fb7f0155d80 2023-03-03T19:35:22.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7f49d1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f006bc30 con 0x7fb7f00beab0 2023-03-03T19:35:22.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7edd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7f0152e30 0x7fb7f014b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:22.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7edd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7f0152e30 0x7fb7f014b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42204/0 (socket says 172.21.15.7:42204) 2023-03-03T19:35:22.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7edd9b700 1 -- 172.21.15.7:0/3487800770 learned_addr learned my addr 172.21.15.7:0/3487800770 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:35:22.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7eed9d700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7f0155d80 0x7fb7f014b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:22.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7ee59c700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f00beab0 0x7fb7f00c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:22.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7edd9b700 1 -- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f00beab0 msgr2=0x7fb7f00c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:22.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7edd9b700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f00beab0 0x7fb7f00c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:22.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7edd9b700 1 -- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7f0155d80 msgr2=0x7fb7f014b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:22.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7edd9b700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7f0155d80 0x7fb7f014b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:22.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.293+0000 7fb7edd9b700 1 -- 172.21.15.7:0/3487800770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e40094a0 con 0x7fb7f0152e30 2023-03-03T19:35:22.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.294+0000 7fb7edd9b700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7f0152e30 0x7fb7f014b250 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000e8b0 tx=0x7fb7e000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:35:22.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.294+0000 7fb7deffd700 1 -- 172.21.15.7:0/3487800770 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0009600 con 0x7fb7f0152e30 2023-03-03T19:35:22.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.294+0000 7fb7deffd700 1 -- 172.21.15.7:0/3487800770 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e0013680 con 0x7fb7f0152e30 2023-03-03T19:35:22.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.294+0000 7fb7f49d1700 1 -- 172.21.15.7:0/3487800770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f0071c70 con 0x7fb7f0152e30 2023-03-03T19:35:22.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.294+0000 7fb7deffd700 1 -- 172.21.15.7:0/3487800770 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0013a30 con 0x7fb7f0152e30 2023-03-03T19:35:22.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.294+0000 7fb7f49d1700 1 -- 172.21.15.7:0/3487800770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f00aadd0 con 0x7fb7f0152e30 2023-03-03T19:35:22.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.295+0000 7fb7f49d1700 1 -- 172.21.15.7:0/3487800770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f0153200 con 0x7fb7f0152e30 2023-03-03T19:35:22.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.295+0000 7fb7deffd700 1 -- 172.21.15.7:0/3487800770 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb7e0019030 con 0x7fb7f0152e30 2023-03-03T19:35:22.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.296+0000 7fb7deffd700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7d00682f0 0x7fb7d006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:35:22.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.296+0000 7fb7deffd700 1 -- 172.21.15.7:0/3487800770 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb7e0085ed0 con 0x7fb7f0152e30 2023-03-03T19:35:22.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.296+0000 7fb7ee59c700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7d00682f0 0x7fb7d006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:35:22.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.296+0000 7fb7ee59c700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7d00682f0 0x7fb7d006a7a0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4000c00 tx=0x7fb7e4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:35:22.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.300+0000 7fb7deffd700 1 -- 172.21.15.7:0/3487800770 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb7e0052270 con 0x7fb7f0152e30 2023-03-03T19:35:22.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.537+0000 7fb7f49d1700 1 -- 172.21.15.7:0/3487800770 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7f0074bb0 con 0x7fb7f0152e30 2023-03-03T19:35:22.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.537+0000 7fb7deffd700 1 -- 172.21.15.7:0/3487800770 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb7e0055890 con 0x7fb7f0152e30 2023-03-03T19:35:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:35:22.538 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:35:22.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.539+0000 7fb7f49d1700 1 -- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7d00682f0 msgr2=0x7fb7d006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:22.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.539+0000 7fb7f49d1700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7d00682f0 0x7fb7d006a7a0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4000c00 tx=0x7fb7e4008030 comp rx=0 tx=0).stop 2023-03-03T19:35:22.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.539+0000 7fb7f49d1700 1 -- 172.21.15.7:0/3487800770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7f0152e30 msgr2=0x7fb7f014b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:35:22.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.539+0000 7fb7f49d1700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7f0152e30 0x7fb7f014b250 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000e8b0 tx=0x7fb7e000ebc0 comp rx=0 tx=0).stop 2023-03-03T19:35:22.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.540+0000 7fb7f49d1700 1 -- 172.21.15.7:0/3487800770 shutdown_connections 2023-03-03T19:35:22.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.540+0000 7fb7f49d1700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7f0155d80 0x7fb7f014b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:22.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.540+0000 7fb7f49d1700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb7f00beab0 0x7fb7f00c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:22.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.540+0000 7fb7f49d1700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7d00682f0 0x7fb7d006a7a0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:22.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.540+0000 7fb7f49d1700 1 --2- 172.21.15.7:0/3487800770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7f0152e30 0x7fb7f014b250 unknown :-1 s=CLOSED pgs=3532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:35:22.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.540+0000 7fb7f49d1700 1 -- 172.21.15.7:0/3487800770 >> 172.21.15.7:0/3487800770 conn(0x7fb7f00736f0 msgr2=0x7fb7f00c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:35:22.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.540+0000 7fb7f49d1700 1 -- 172.21.15.7:0/3487800770 shutdown_connections 2023-03-03T19:35:22.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:35:22.540+0000 7fb7f49d1700 1 -- 172.21.15.7:0/3487800770 wait complete. 2023-03-03T19:35:22.542 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:35:22.552 INFO:tasks.mds_thrash.fs.[a]:waiting for 83 secs before thrashing 2023-03-03T19:35:49.792 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:35:49.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:36:19.863 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:36:19.868 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:36:45.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:36:45.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.692+0000 7f6ae251a700 1 Processor -- start 2023-03-03T19:36:45.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.692+0000 7f6ae251a700 1 -- start start 2023-03-03T19:36:45.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.693+0000 7f6ae251a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0beab0 0x7f6adc155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:36:45.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.693+0000 7f6ae251a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc1523b0 con 0x7f6adc152e30 2023-03-03T19:36:45.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.693+0000 7f6ae251a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc072010 con 0x7f6adc1562c0 2023-03-03T19:36:45.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.693+0000 7f6ae251a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc0aaaf0 con 0x7f6adc0beab0 2023-03-03T19:36:45.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.693+0000 7f6adbfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6adc152e30 0x7f6adc0c43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36246/0 (socket says 172.21.15.7:36246) 2023-03-03T19:36:45.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.693+0000 7f6adbfff700 1 -- 172.21.15.7:0/3470129808 learned_addr learned my addr 172.21.15.7:0/3470129808 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:36:45.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adb7fe700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0beab0 0x7f6adc155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:36:45.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479949972 0 0) 0x7f6adc1523b0 con 0x7f6adc152e30 2023-03-03T19:36:45.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac80035b0 con 0x7f6adc152e30 2023-03-03T19:36:45.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 449248072 0 0) 0x7f6ac80035b0 con 0x7f6adc152e30 2023-03-03T19:36:45.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0beab0 msgr2=0x7f6adc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:36:45.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0beab0 0x7f6adc155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:36:45.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 >> v1:172.21.15.143:6789/0 conn(0x7f6adc1562c0 legacy=0x7f6adc15cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:36:45.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc0aadd0 con 0x7f6adc152e30 2023-03-03T19:36:45.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f6ac4001670 con 0x7f6adc152e30 2023-03-03T19:36:45.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 >> v1:172.21.15.7:6789/0 conn(0x7f6adc152e30 legacy=0x7f6adc0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:36:45.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ac8003aa0 0x7f6ac8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:36:45.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0beab0 0x7f6ac8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:36:45.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ac8004d90 0x7f6ac8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:36:45.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc0aaaf0 con 0x7f6ac8004d90 2023-03-03T19:36:45.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac800bba0 con 0x7f6ac8003aa0 2023-03-03T19:36:45.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac800bd10 con 0x7f6adc0beab0 2023-03-03T19:36:45.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6ae0ab7700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ac8004d90 0x7f6ac8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:36:45.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adbfff700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ac8003aa0 0x7f6ac8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:36:45.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6adb7fe700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0beab0 0x7f6ac8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:36:45.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6ae0ab7700 1 -- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0beab0 msgr2=0x7f6ac8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:36:45.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6ae0ab7700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0beab0 0x7f6ac8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:36:45.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6ae0ab7700 1 -- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ac8003aa0 msgr2=0x7f6ac8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:36:45.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6ae0ab7700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6ac8003aa0 0x7f6ac8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:36:45.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.694+0000 7f6ae0ab7700 1 -- 172.21.15.7:0/3470129808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac80035b0 con 0x7f6ac8004d90 2023-03-03T19:36:45.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6ae0ab7700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ac8004d90 0x7f6ac8005200 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400c260 tx=0x7f6ac8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59368310aed7889 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:36:45.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4001e70 con 0x7f6ac8004d90 2023-03-03T19:36:45.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad400ce20 con 0x7f6ac8004d90 2023-03-03T19:36:45.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6adaffd700 1 -- 172.21.15.7:0/3470129808 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4012660 con 0x7f6ac8004d90 2023-03-03T19:36:45.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6ae251a700 1 -- 172.21.15.7:0/3470129808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ac8004d90 msgr2=0x7f6ac8005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:36:45.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6ae251a700 1 --2- 172.21.15.7:0/3470129808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ac8004d90 0x7f6ac8005200 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400c260 tx=0x7f6ac8005880 comp rx=0 tx=0).stop 2023-03-03T19:36:45.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6ae0ab7700 1 -- 172.21.15.7:0/3470129808 reap_dead start 2023-03-03T19:36:45.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6ae251a700 1 -- 172.21.15.7:0/3470129808 shutdown_connections 2023-03-03T19:36:45.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6ae251a700 1 -- 172.21.15.7:0/3470129808 >> 172.21.15.7:0/3470129808 conn(0x7f6adc0736f0 msgr2=0x7f6adc073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:36:45.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6ae251a700 1 -- 172.21.15.7:0/3470129808 shutdown_connections 2023-03-03T19:36:45.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.695+0000 7f6ae251a700 1 -- 172.21.15.7:0/3470129808 wait complete. 2023-03-03T19:36:45.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.697+0000 7f6ae251a700 1 Processor -- start 2023-03-03T19:36:45.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.697+0000 7f6ae251a700 1 -- start start 2023-03-03T19:36:45.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6ae251a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6adc1514a0 0x7f6adc151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:36:45.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6ae251a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc14b5c0 0x7f6adc0ab740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:36:45.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6ae251a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0abc80 0x7f6adc1eb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:36:45.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6ae251a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc1546e0 con 0x7f6adc14b5c0 2023-03-03T19:36:45.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6ae251a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc0ad560 con 0x7f6adc1514a0 2023-03-03T19:36:45.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6ae251a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc154400 con 0x7f6adc0abc80 2023-03-03T19:36:45.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6adb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc14b5c0 0x7f6adc0ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:36:45.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6adb7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc14b5c0 0x7f6adc0ab740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42280/0 (socket says 172.21.15.7:42280) 2023-03-03T19:36:45.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6adb7fe700 1 -- 172.21.15.7:0/1160522792 learned_addr learned my addr 172.21.15.7:0/1160522792 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:36:45.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6ae0ab7700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0abc80 0x7f6adc1eb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:36:45.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.698+0000 7f6adbfff700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6adc1514a0 0x7f6adc151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:36:45.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6adb7fe700 1 -- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0abc80 msgr2=0x7f6adc1eb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:36:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6adb7fe700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0abc80 0x7f6adc1eb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:36:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6adb7fe700 1 -- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6adc1514a0 msgr2=0x7f6adc151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:36:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6adb7fe700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6adc1514a0 0x7f6adc151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:36:45.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6adb7fe700 1 -- 172.21.15.7:0/1160522792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc153200 con 0x7f6adc14b5c0 2023-03-03T19:36:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6ae0ab7700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0abc80 0x7f6adc1eb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:36:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6adbfff700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6adc1514a0 0x7f6adc151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:36:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6adb7fe700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc14b5c0 0x7f6adc0ab740 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 crypto rx=0x7f6acc011aa0 tx=0x7f6acc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:36:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6ad8ff9700 1 -- 172.21.15.7:0/1160522792 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc00f980 con 0x7f6adc14b5c0 2023-03-03T19:36:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6ad8ff9700 1 -- 172.21.15.7:0/1160522792 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc00fae0 con 0x7f6adc14b5c0 2023-03-03T19:36:45.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6ae251a700 1 -- 172.21.15.7:0/1160522792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc071c70 con 0x7f6adc14b5c0 2023-03-03T19:36:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6ad8ff9700 1 -- 172.21.15.7:0/1160522792 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc00fe90 con 0x7f6adc14b5c0 2023-03-03T19:36:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.699+0000 7f6ae251a700 1 -- 172.21.15.7:0/1160522792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc153390 con 0x7f6adc14b5c0 2023-03-03T19:36:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.700+0000 7f6ae251a700 1 -- 172.21.15.7:0/1160522792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc1548c0 con 0x7f6adc14b5c0 2023-03-03T19:36:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.700+0000 7f6ad8ff9700 1 -- 172.21.15.7:0/1160522792 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6acc014440 con 0x7f6adc14b5c0 2023-03-03T19:36:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.701+0000 7f6ad8ff9700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6abc0682f0 0x7f6abc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:36:45.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.701+0000 7f6ad8ff9700 1 -- 172.21.15.7:0/1160522792 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6acc086a60 con 0x7f6adc14b5c0 2023-03-03T19:36:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.701+0000 7f6adbfff700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6abc0682f0 0x7f6abc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:36:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.702+0000 7f6adbfff700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6abc0682f0 0x7f6abc06a7a0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4000c00 tx=0x7f6ac4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:36:45.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.705+0000 7f6ad8ff9700 1 -- 172.21.15.7:0/1160522792 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6adc1548c0 con 0x7f6adc14b5c0 2023-03-03T19:36:45.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.943+0000 7f6ae251a700 1 -- 172.21.15.7:0/1160522792 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6adc1548c0 con 0x7f6adc14b5c0 2023-03-03T19:36:45.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.943+0000 7f6ad8ff9700 1 -- 172.21.15.7:0/1160522792 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6adc1548c0 con 0x7f6adc14b5c0 2023-03-03T19:36:45.944 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:36:45.944 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:36:45.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 -- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6abc0682f0 msgr2=0x7f6abc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:36:45.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6abc0682f0 0x7f6abc06a7a0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4000c00 tx=0x7f6ac4009040 comp rx=0 tx=0).stop 2023-03-03T19:36:45.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 -- 172.21.15.7:0/1160522792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc14b5c0 msgr2=0x7f6adc0ab740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:36:45.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc14b5c0 0x7f6adc0ab740 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 crypto rx=0x7f6acc011aa0 tx=0x7f6acc011e60 comp rx=0 tx=0).stop 2023-03-03T19:36:45.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 -- 172.21.15.7:0/1160522792 shutdown_connections 2023-03-03T19:36:45.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6adc0abc80 0x7f6adc1eb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:36:45.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6adc1514a0 0x7f6adc151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:36:45.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6abc0682f0 0x7f6abc06a7a0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:36:45.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 --2- 172.21.15.7:0/1160522792 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc14b5c0 0x7f6adc0ab740 unknown :-1 s=CLOSED pgs=3537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:36:45.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 -- 172.21.15.7:0/1160522792 >> 172.21.15.7:0/1160522792 conn(0x7f6adc0736f0 msgr2=0x7f6adc0c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:36:45.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 -- 172.21.15.7:0/1160522792 shutdown_connections 2023-03-03T19:36:45.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:36:45.945+0000 7f6ae251a700 1 -- 172.21.15.7:0/1160522792 wait complete. 2023-03-03T19:36:45.948 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:36:45.958 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-03-03T19:36:50.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:36:50.079 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:37:06.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:37:06.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.633+0000 7f83931ac700 1 Processor -- start 2023-03-03T19:37:06.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.633+0000 7f83931ac700 1 -- start start 2023-03-03T19:37:06.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f83931ac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 0x7f838c0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:06.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f83931ac700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f838c151770 0x7f838c151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:06.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f83931ac700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838c154910 con 0x7f838c06e020 2023-03-03T19:37:06.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f83931ac700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838c070b90 con 0x7f838c151770 2023-03-03T19:37:06.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f83931ac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c151320 con 0x7f838c155390 2023-03-03T19:37:06.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f8391749700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f838c151770 0x7f838c151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f838bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f838c06e020 0x7f838c071fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36270/0 (socket says 172.21.15.7:36270) 2023-03-03T19:37:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f838bfff700 1 -- 172.21.15.7:0/4179079628 learned_addr learned my addr 172.21.15.7:0/4179079628 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:37:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f8390f48700 1 --2- 172.21.15.7:0/4179079628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 0x7f838c0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f8391749700 1 -- 172.21.15.7:0/4179079628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 msgr2=0x7f838c0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:37:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f8391749700 1 --2- 172.21.15.7:0/4179079628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 0x7f838c0c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f8391749700 1 -- 172.21.15.7:0/4179079628 >> v1:172.21.15.7:6789/0 conn(0x7f838c06e020 legacy=0x7f838c071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.634+0000 7f838b7fe700 1 -- 172.21.15.7:0/4179079628 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329560359 0 0) 0x7f838c154910 con 0x7f838c06e020 2023-03-03T19:37:06.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f8391749700 1 -- 172.21.15.7:0/4179079628 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c0702a0 con 0x7f838c151770 2023-03-03T19:37:06.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f8390f48700 1 --2- 172.21.15.7:0/4179079628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 0x7f838c0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:37:06.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f8391749700 1 --2- 172.21.15.7:0/4179079628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 0x7f838c151bf0 secure :-1 s=READY pgs=3539 cs=0 l=1 rev1=1 crypto rx=0x7f8384009d90 tx=0x7f838400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7449e0d16c9063 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:37:06.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f838b7fe700 1 -- 172.21.15.7:0/4179079628 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838400e070 con 0x7f838c151770 2023-03-03T19:37:06.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f838b7fe700 1 -- 172.21.15.7:0/4179079628 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838400bbd0 con 0x7f838c151770 2023-03-03T19:37:06.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f838b7fe700 1 -- 172.21.15.7:0/4179079628 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384012430 con 0x7f838c151770 2023-03-03T19:37:06.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f83931ac700 1 -- 172.21.15.7:0/4179079628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 msgr2=0x7f838c151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:06.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f83931ac700 1 --2- 172.21.15.7:0/4179079628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 0x7f838c151bf0 secure :-1 s=READY pgs=3539 cs=0 l=1 rev1=1 crypto rx=0x7f8384009d90 tx=0x7f838400b330 comp rx=0 tx=0).stop 2023-03-03T19:37:06.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f83931ac700 1 -- 172.21.15.7:0/4179079628 shutdown_connections 2023-03-03T19:37:06.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f83931ac700 1 --2- 172.21.15.7:0/4179079628 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 0x7f838c151bf0 unknown :-1 s=CLOSED pgs=3539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:06.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f83931ac700 1 --2- 172.21.15.7:0/4179079628 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 0x7f838c0c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:06.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f83931ac700 1 -- 172.21.15.7:0/4179079628 >> 172.21.15.7:0/4179079628 conn(0x7f838c06c110 msgr2=0x7f838c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:37:06.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f83931ac700 1 -- 172.21.15.7:0/4179079628 shutdown_connections 2023-03-03T19:37:06.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.635+0000 7f83931ac700 1 -- 172.21.15.7:0/4179079628 wait complete. 2023-03-03T19:37:06.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.637+0000 7f83931ac700 1 Processor -- start 2023-03-03T19:37:06.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.637+0000 7f83931ac700 1 -- start start 2023-03-03T19:37:06.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.637+0000 7f83931ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f838c06e020 0x7f838c1ed8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:06.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.637+0000 7f83931ac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 0x7f838c073960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:06.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.637+0000 7f83931ac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 0x7f838c1434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:06.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.637+0000 7f83931ac700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c154880 con 0x7f838c151770 2023-03-03T19:37:06.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.637+0000 7f83931ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c14e140 con 0x7f838c06e020 2023-03-03T19:37:06.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.637+0000 7f83931ac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c14c0d0 con 0x7f838c155390 2023-03-03T19:37:06.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f838bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 0x7f838c073960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:06.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f838bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 0x7f838c073960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42296/0 (socket says 172.21.15.7:42296) 2023-03-03T19:37:06.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f838bfff700 1 -- 172.21.15.7:0/3134380739 learned_addr learned my addr 172.21.15.7:0/3134380739 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:37:06.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f8391749700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 0x7f838c1434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:06.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f8390f48700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f838c06e020 0x7f838c1ed8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:06.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f838bfff700 1 -- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 msgr2=0x7f838c1434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:06.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f838bfff700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 0x7f838c1434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:06.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f838bfff700 1 -- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f838c06e020 msgr2=0x7f838c1ed8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:06.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f838bfff700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f838c06e020 0x7f838c1ed8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:06.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f838bfff700 1 -- 172.21.15.7:0/3134380739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8384009a70 con 0x7f838c151770 2023-03-03T19:37:06.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f8390f48700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f838c06e020 0x7f838c1ed8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:37:06.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f838bfff700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 0x7f838c073960 secure :-1 s=READY pgs=3540 cs=0 l=1 rev1=1 crypto rx=0x7f837800c750 tx=0x7f837800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:37:06.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f83897fa700 1 -- 172.21.15.7:0/3134380739 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378004c80 con 0x7f838c151770 2023-03-03T19:37:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f83897fa700 1 -- 172.21.15.7:0/3134380739 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8378004e00 con 0x7f838c151770 2023-03-03T19:37:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f83931ac700 1 -- 172.21.15.7:0/3134380739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838c070820 con 0x7f838c151770 2023-03-03T19:37:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f83897fa700 1 -- 172.21.15.7:0/3134380739 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378007310 con 0x7f838c151770 2023-03-03T19:37:06.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.638+0000 7f83931ac700 1 -- 172.21.15.7:0/3134380739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838c070540 con 0x7f838c151770 2023-03-03T19:37:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.639+0000 7f83931ac700 1 -- 172.21.15.7:0/3134380739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838c155760 con 0x7f838c151770 2023-03-03T19:37:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.640+0000 7f83897fa700 1 -- 172.21.15.7:0/3134380739 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f837801a020 con 0x7f838c151770 2023-03-03T19:37:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.640+0000 7f83897fa700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83700682c0 0x7f837006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:06.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.640+0000 7f83897fa700 1 -- 172.21.15.7:0/3134380739 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8378085e10 con 0x7f838c151770 2023-03-03T19:37:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.640+0000 7f8390f48700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83700682c0 0x7f837006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.641+0000 7f8390f48700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83700682c0 0x7f837006a770 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f8380008910 tx=0x7f8380008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:37:06.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.645+0000 7f83897fa700 1 -- 172.21.15.7:0/3134380739 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8378052150 con 0x7f838c151770 2023-03-03T19:37:06.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.882+0000 7f83931ac700 1 -- 172.21.15.7:0/3134380739 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f838c1ede20 con 0x7f838c151770 2023-03-03T19:37:06.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.882+0000 7f83897fa700 1 -- 172.21.15.7:0/3134380739 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8378055770 con 0x7f838c151770 2023-03-03T19:37:06.883 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:37:06.883 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:37:06.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 -- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83700682c0 msgr2=0x7f837006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:06.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83700682c0 0x7f837006a770 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f8380008910 tx=0x7f8380008040 comp rx=0 tx=0).stop 2023-03-03T19:37:06.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 -- 172.21.15.7:0/3134380739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 msgr2=0x7f838c073960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:06.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 0x7f838c073960 secure :-1 s=READY pgs=3540 cs=0 l=1 rev1=1 crypto rx=0x7f837800c750 tx=0x7f837800cb10 comp rx=0 tx=0).stop 2023-03-03T19:37:06.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 -- 172.21.15.7:0/3134380739 shutdown_connections 2023-03-03T19:37:06.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f838c155390 0x7f838c1434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:06.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f838c06e020 0x7f838c1ed8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:06.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83700682c0 0x7f837006a770 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:06.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 --2- 172.21.15.7:0/3134380739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f838c151770 0x7f838c073960 unknown :-1 s=CLOSED pgs=3540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:06.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 -- 172.21.15.7:0/3134380739 >> 172.21.15.7:0/3134380739 conn(0x7f838c06c110 msgr2=0x7f838c0c25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:37:06.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 -- 172.21.15.7:0/3134380739 shutdown_connections 2023-03-03T19:37:06.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:06.884+0000 7f83931ac700 1 -- 172.21.15.7:0/3134380739 wait complete. 2023-03-03T19:37:06.887 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:37:06.896 INFO:tasks.mds_thrash.fs.[b]:waiting for 39 secs before thrashing 2023-03-03T19:37:20.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:37:20.175 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:37:34.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:37:35.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.097+0000 7fd0faff5700 1 Processor -- start 2023-03-03T19:37:35.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.097+0000 7fd0faff5700 1 -- start start 2023-03-03T19:37:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0faff5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0f406d870 0x7fd0f41550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0faff5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd0f40c1da0 0x7fd0f40c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:35.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0faff5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd0f40c3e30 0x7fd0f40c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0faff5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f40c2af0 con 0x7fd0f40c1da0 2023-03-03T19:37:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0faff5700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0f40c3620 con 0x7fd0f40c3e30 2023-03-03T19:37:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0faff5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f41545e0 con 0x7fd0f406d870 2023-03-03T19:37:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0f3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd0f40c1da0 0x7fd0f40c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0f3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd0f40c1da0 0x7fd0f40c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42310/0 (socket says 172.21.15.7:42310) 2023-03-03T19:37:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0f3fff700 1 -- 172.21.15.7:0/2938466128 learned_addr learned my addr 172.21.15.7:0/2938466128 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:37:35.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0f9592700 1 --2- 172.21.15.7:0/2938466128 >> v2:172.21.15.143:3300/0 conn(0x7fd0f40c3e30 0x7fd0f40c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.098+0000 7fd0f8d91700 1 --2- 172.21.15.7:0/2938466128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0f406d870 0x7fd0f41550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0f3fff700 1 -- 172.21.15.7:0/2938466128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0f406d870 msgr2=0x7fd0f41550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:37:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0f3fff700 1 --2- 172.21.15.7:0/2938466128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0f406d870 0x7fd0f41550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0f3fff700 1 -- 172.21.15.7:0/2938466128 >> v2:172.21.15.143:3300/0 conn(0x7fd0f40c3e30 msgr2=0x7fd0f40c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:37:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0f3fff700 1 --2- 172.21.15.7:0/2938466128 >> v2:172.21.15.143:3300/0 conn(0x7fd0f40c3e30 0x7fd0f40c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0f3fff700 1 -- 172.21.15.7:0/2938466128 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f41548c0 con 0x7fd0f40c1da0 2023-03-03T19:37:35.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0f3fff700 1 --2- 172.21.15.7:0/2938466128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f40c1da0 0x7fd0f40c7350 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7fd0f406be80 tx=0x7fd0e0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b90acc0e03b0d7e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:37:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0f37fe700 1 -- 172.21.15.7:0/2938466128 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0011830 con 0x7fd0f40c1da0 2023-03-03T19:37:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0f37fe700 1 -- 172.21.15.7:0/2938466128 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e0011990 con 0x7fd0f40c1da0 2023-03-03T19:37:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0f37fe700 1 -- 172.21.15.7:0/2938466128 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0011d20 con 0x7fd0f40c1da0 2023-03-03T19:37:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0faff5700 1 -- 172.21.15.7:0/2938466128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f40c1da0 msgr2=0x7fd0f40c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.099+0000 7fd0faff5700 1 --2- 172.21.15.7:0/2938466128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f40c1da0 0x7fd0f40c7350 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7fd0f406be80 tx=0x7fd0e0009bd0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.100+0000 7fd0faff5700 1 -- 172.21.15.7:0/2938466128 shutdown_connections 2023-03-03T19:37:35.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.100+0000 7fd0faff5700 1 --2- 172.21.15.7:0/2938466128 >> v2:172.21.15.143:3300/0 conn(0x7fd0f40c3e30 0x7fd0f40c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.100+0000 7fd0faff5700 1 --2- 172.21.15.7:0/2938466128 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0f406d870 0x7fd0f41550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.100+0000 7fd0faff5700 1 --2- 172.21.15.7:0/2938466128 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f40c1da0 0x7fd0f40c7350 unknown :-1 s=CLOSED pgs=3541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.100+0000 7fd0faff5700 1 -- 172.21.15.7:0/2938466128 >> 172.21.15.7:0/2938466128 conn(0x7fd0f406c110 msgr2=0x7fd0f406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:37:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.100+0000 7fd0faff5700 1 -- 172.21.15.7:0/2938466128 shutdown_connections 2023-03-03T19:37:35.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.100+0000 7fd0faff5700 1 -- 172.21.15.7:0/2938466128 wait complete. 2023-03-03T19:37:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.101+0000 7fd0faff5700 1 Processor -- start 2023-03-03T19:37:35.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.101+0000 7fd0faff5700 1 -- start start 2023-03-03T19:37:35.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0faff5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f406d870 0x7fd0f40aa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:35.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0faff5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0f40c1da0 0x7fd0f4147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:35.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0faff5700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0f40c3e30 0x7fd0f41e9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:35.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0faff5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f40acd30 con 0x7fd0f406d870 2023-03-03T19:37:35.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0faff5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f41469d0 con 0x7fd0f40c1da0 2023-03-03T19:37:35.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0faff5700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f4071010 con 0x7fd0f40c3e30 2023-03-03T19:37:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f8d91700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f406d870 0x7fd0f40aa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f8d91700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f406d870 0x7fd0f40aa450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42312/0 (socket says 172.21.15.7:42312) 2023-03-03T19:37:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f8d91700 1 -- 172.21.15.7:0/2893192930 learned_addr learned my addr 172.21.15.7:0/2893192930 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:37:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f9592700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0f40c3e30 0x7fd0f41e9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f3fff700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0f40c1da0 0x7fd0f4147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f8d91700 1 -- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0f40c3e30 msgr2=0x7fd0f41e9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f8d91700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0f40c3e30 0x7fd0f41e9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f8d91700 1 -- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0f40c1da0 msgr2=0x7fd0f4147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f8d91700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0f40c1da0 0x7fd0f4147790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f8d91700 1 -- 172.21.15.7:0/2893192930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e0009550 con 0x7fd0f406d870 2023-03-03T19:37:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f3fff700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0f40c1da0 0x7fd0f4147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:37:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f8d91700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f406d870 0x7fd0f40aa450 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800e960 tx=0x7fd0e800ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:37:35.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.102+0000 7fd0f17fa700 1 -- 172.21.15.7:0/2893192930 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8009690 con 0x7fd0f406d870 2023-03-03T19:37:35.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.103+0000 7fd0f17fa700 1 -- 172.21.15.7:0/2893192930 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e800c850 con 0x7fd0f406d870 2023-03-03T19:37:35.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.103+0000 7fd0faff5700 1 -- 172.21.15.7:0/2893192930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f406dc40 con 0x7fd0f406d870 2023-03-03T19:37:35.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.103+0000 7fd0f17fa700 1 -- 172.21.15.7:0/2893192930 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e800cbc0 con 0x7fd0f406d870 2023-03-03T19:37:35.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.103+0000 7fd0faff5700 1 -- 172.21.15.7:0/2893192930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f4154ba0 con 0x7fd0f406d870 2023-03-03T19:37:35.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.104+0000 7fd0faff5700 1 -- 172.21.15.7:0/2893192930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f40c38e0 con 0x7fd0f406d870 2023-03-03T19:37:35.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.104+0000 7fd0f17fa700 1 -- 172.21.15.7:0/2893192930 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd0e8019030 con 0x7fd0f406d870 2023-03-03T19:37:35.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.104+0000 7fd0f17fa700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0d80682f0 0x7fd0d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:35.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.105+0000 7fd0f17fa700 1 -- 172.21.15.7:0/2893192930 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd0e80860c0 con 0x7fd0f406d870 2023-03-03T19:37:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.105+0000 7fd0f3fff700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0d80682f0 0x7fd0d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.105+0000 7fd0f3fff700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0d80682f0 0x7fd0d806a7a0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4073e50 tx=0x7fd0e0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:37:35.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.109+0000 7fd0f17fa700 1 -- 172.21.15.7:0/2893192930 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd0e8052460 con 0x7fd0f406d870 2023-03-03T19:37:35.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.345+0000 7fd0faff5700 1 -- 172.21.15.7:0/2893192930 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0f41ea740 con 0x7fd0f406d870 2023-03-03T19:37:35.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.345+0000 7fd0f17fa700 1 -- 172.21.15.7:0/2893192930 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd0e8055a80 con 0x7fd0f406d870 2023-03-03T19:37:35.346 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:37:35.347 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:37:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 -- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0d80682f0 msgr2=0x7fd0d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0d80682f0 0x7fd0d806a7a0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4073e50 tx=0x7fd0e0008030 comp rx=0 tx=0).stop 2023-03-03T19:37:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 -- 172.21.15.7:0/2893192930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f406d870 msgr2=0x7fd0f40aa450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:35.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f406d870 0x7fd0f40aa450 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800e960 tx=0x7fd0e800ed20 comp rx=0 tx=0).stop 2023-03-03T19:37:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 -- 172.21.15.7:0/2893192930 shutdown_connections 2023-03-03T19:37:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd0f40c3e30 0x7fd0f41e9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd0d80682f0 0x7fd0d806a7a0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd0f406d870 0x7fd0f40aa450 unknown :-1 s=CLOSED pgs=3542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 --2- 172.21.15.7:0/2893192930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0f40c1da0 0x7fd0f4147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:35.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 -- 172.21.15.7:0/2893192930 >> 172.21.15.7:0/2893192930 conn(0x7fd0f406c110 msgr2=0x7fd0f414ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:37:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.347+0000 7fd0faff5700 1 -- 172.21.15.7:0/2893192930 shutdown_connections 2023-03-03T19:37:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:35.348+0000 7fd0faff5700 1 -- 172.21.15.7:0/2893192930 wait complete. 2023-03-03T19:37:35.350 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:37:35.360 INFO:tasks.mds_thrash.fs.[a]:waiting for 87 secs before thrashing 2023-03-03T19:37:45.897 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:37:46.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.036+0000 7f5773135700 1 Processor -- start 2023-03-03T19:37:46.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.036+0000 7f5773135700 1 -- start start 2023-03-03T19:37:46.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f5773135700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c14f9d0 0x7f576c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:46.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f5773135700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f576c0abe20 0x7f576c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:46.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f5773135700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576c0c2f30 con 0x7f576c14fda0 2023-03-03T19:37:46.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f5773135700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576c0c31d0 con 0x7f576c0abe20 2023-03-03T19:37:46.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f5773135700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c1524a0 con 0x7f576c14f9d0 2023-03-03T19:37:46.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f576bfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f576c14fda0 0x7f576c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36316/0 (socket says 172.21.15.7:36316) 2023-03-03T19:37:46.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f576bfff700 1 -- 172.21.15.7:0/70808658 learned_addr learned my addr 172.21.15.7:0/70808658 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:37:46.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f5770ed1700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c14f9d0 0x7f576c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:46.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f57716d2700 1 --2- 172.21.15.7:0/70808658 >> v2:172.21.15.143:3300/0 conn(0x7f576c0abe20 0x7f576c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:46.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917995738 0 0) 0x7f576c0c2f30 con 0x7f576c14fda0 2023-03-03T19:37:46.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.037+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f575c0035b0 con 0x7f576c14fda0 2023-03-03T19:37:46.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4187495960 0 0) 0x7f575c0035b0 con 0x7f576c14fda0 2023-03-03T19:37:46.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c14f9d0 msgr2=0x7f576c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:37:46.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c14f9d0 0x7f576c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:46.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 >> v2:172.21.15.143:3300/0 conn(0x7f576c0abe20 msgr2=0x7f576c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:37:46.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 --2- 172.21.15.7:0/70808658 >> v2:172.21.15.143:3300/0 conn(0x7f576c0abe20 0x7f576c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c0c34b0 con 0x7f576c14fda0 2023-03-03T19:37:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f5758001670 con 0x7f576c14fda0 2023-03-03T19:37:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 >> v1:172.21.15.7:6789/0 conn(0x7f576c14fda0 legacy=0x7f576c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c003a70 0x7f575c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c004850 0x7f575c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c14f9d0 0x7f575c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c0c31d0 con 0x7f575c004850 2023-03-03T19:37:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c1524a0 con 0x7f575c003a70 2023-03-03T19:37:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c00bb10 con 0x7f576c14f9d0 2023-03-03T19:37:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576bfff700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c004850 0x7f575c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f57716d2700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c14f9d0 0x7f575c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f5770ed1700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c003a70 0x7f575c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576bfff700 1 -- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c14f9d0 msgr2=0x7f575c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:37:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576bfff700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c14f9d0 0x7f575c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:46.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576bfff700 1 -- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c003a70 msgr2=0x7f575c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:37:46.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576bfff700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c003a70 0x7f575c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:46.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576bfff700 1 -- 172.21.15.7:0/70808658 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0035b0 con 0x7f575c004850 2023-03-03T19:37:46.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576bfff700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c004850 0x7f575c004cc0 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7f575800c370 tx=0x7f575c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2ca91d0b767bb9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:37:46.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758007c60 con 0x7f575c004850 2023-03-03T19:37:46.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575800ce70 con 0x7f575c004850 2023-03-03T19:37:46.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.038+0000 7f576b7fe700 1 -- 172.21.15.7:0/70808658 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758005b80 con 0x7f575c004850 2023-03-03T19:37:46.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.039+0000 7f5773135700 1 -- 172.21.15.7:0/70808658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c004850 msgr2=0x7f575c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:46.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.039+0000 7f5773135700 1 --2- 172.21.15.7:0/70808658 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f575c004850 0x7f575c004cc0 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7f575800c370 tx=0x7f575c005880 comp rx=0 tx=0).stop 2023-03-03T19:37:46.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.039+0000 7f57716d2700 1 -- 172.21.15.7:0/70808658 reap_dead start 2023-03-03T19:37:46.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.039+0000 7f5773135700 1 -- 172.21.15.7:0/70808658 shutdown_connections 2023-03-03T19:37:46.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.039+0000 7f5773135700 1 -- 172.21.15.7:0/70808658 >> 172.21.15.7:0/70808658 conn(0x7f576c0c8900 msgr2=0x7f576c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:37:46.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.039+0000 7f5773135700 1 -- 172.21.15.7:0/70808658 shutdown_connections 2023-03-03T19:37:46.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.039+0000 7f5773135700 1 -- 172.21.15.7:0/70808658 wait complete. 2023-03-03T19:37:46.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.040+0000 7f5773135700 1 Processor -- start 2023-03-03T19:37:46.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.040+0000 7f5773135700 1 -- start start 2023-03-03T19:37:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f5773135700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c004850 0x7f576c0c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f5773135700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c155470 0x7f576c0c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f5773135700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f576c14d180 0x7f576c14c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f5773135700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c06b710 con 0x7f576c14d180 2023-03-03T19:37:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f5773135700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c06ade0 con 0x7f575c004850 2023-03-03T19:37:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f5773135700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576c1ee100 con 0x7f576c155470 2023-03-03T19:37:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f57716d2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f576c14d180 0x7f576c14c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f57716d2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f576c14d180 0x7f576c14c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42352/0 (socket says 172.21.15.7:42352) 2023-03-03T19:37:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f57716d2700 1 -- 172.21.15.7:0/662833688 learned_addr learned my addr 172.21.15.7:0/662833688 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:37:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f5770ed1700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c004850 0x7f576c0c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.041+0000 7f576bfff700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c155470 0x7f576c0c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f57716d2700 1 -- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c155470 msgr2=0x7f576c0c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f57716d2700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c155470 0x7f576c0c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f57716d2700 1 -- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c004850 msgr2=0x7f576c0c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f57716d2700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c004850 0x7f576c0c72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f57716d2700 1 -- 172.21.15.7:0/662833688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c0ac350 con 0x7f576c14d180 2023-03-03T19:37:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f57716d2700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f576c14d180 0x7f576c14c5f0 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7f5764011c80 tx=0x7f5764011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:37:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f57697fa700 1 -- 172.21.15.7:0/662833688 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576400e040 con 0x7f576c14d180 2023-03-03T19:37:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f57697fa700 1 -- 172.21.15.7:0/662833688 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576400f040 con 0x7f576c14d180 2023-03-03T19:37:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f5773135700 1 -- 172.21.15.7:0/662833688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c0ab630 con 0x7f576c14d180 2023-03-03T19:37:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f57697fa700 1 -- 172.21.15.7:0/662833688 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576400fe00 con 0x7f576c14d180 2023-03-03T19:37:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.042+0000 7f5773135700 1 -- 172.21.15.7:0/662833688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c071fa0 con 0x7f576c14d180 2023-03-03T19:37:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.043+0000 7f5773135700 1 -- 172.21.15.7:0/662833688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576c1edbd0 con 0x7f576c14d180 2023-03-03T19:37:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.043+0000 7f57697fa700 1 -- 172.21.15.7:0/662833688 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5764015430 con 0x7f576c14d180 2023-03-03T19:37:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.044+0000 7f57697fa700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5750068320 0x7f575006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:37:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.044+0000 7f57697fa700 1 -- 172.21.15.7:0/662833688 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5764086660 con 0x7f576c14d180 2023-03-03T19:37:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.044+0000 7f5770ed1700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5750068320 0x7f575006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:37:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.045+0000 7f5770ed1700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5750068320 0x7f575006a7d0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f5760009b00 tx=0x7f5760009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:37:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.048+0000 7f57697fa700 1 -- 172.21.15.7:0/662833688 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f57640529a0 con 0x7f576c14d180 2023-03-03T19:37:46.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.288+0000 7f5773135700 1 -- 172.21.15.7:0/662833688 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f576c074bb0 con 0x7f576c14d180 2023-03-03T19:37:46.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.288+0000 7f57697fa700 1 -- 172.21.15.7:0/662833688 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5764055fc0 con 0x7f576c14d180 2023-03-03T19:37:46.289 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:37:46.289 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:37:46.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 -- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5750068320 msgr2=0x7f575006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5750068320 0x7f575006a7d0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f5760009b00 tx=0x7f5760009210 comp rx=0 tx=0).stop 2023-03-03T19:37:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 -- 172.21.15.7:0/662833688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f576c14d180 msgr2=0x7f576c14c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:37:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f576c14d180 0x7f576c14c5f0 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7f5764011c80 tx=0x7f5764011f90 comp rx=0 tx=0).stop 2023-03-03T19:37:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 -- 172.21.15.7:0/662833688 shutdown_connections 2023-03-03T19:37:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5750068320 0x7f575006a7d0 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f576c14d180 0x7f576c14c5f0 unknown :-1 s=CLOSED pgs=3545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:46.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f575c004850 0x7f576c0c72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:46.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 --2- 172.21.15.7:0/662833688 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f576c155470 0x7f576c0c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:37:46.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 -- 172.21.15.7:0/662833688 >> 172.21.15.7:0/662833688 conn(0x7f576c0c8900 msgr2=0x7f576c151560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:37:46.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 -- 172.21.15.7:0/662833688 shutdown_connections 2023-03-03T19:37:46.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:37:46.290+0000 7f5773135700 1 -- 172.21.15.7:0/662833688 wait complete. 2023-03-03T19:37:46.293 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:37:46.302 INFO:tasks.mds_thrash.fs.[b]:waiting for 52 secs before thrashing 2023-03-03T19:37:50.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:37:50.240 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:38:20.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:38:20.314 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:38:38.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:38:38.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.442+0000 7fd718449700 1 Processor -- start 2023-03-03T19:38:38.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.442+0000 7fd718449700 1 -- start start 2023-03-03T19:38:38.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.443+0000 7fd718449700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd71014f9d0 0x7fd710156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:38:38.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.443+0000 7fd718449700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7100abe20 0x7fd7101529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:38:38.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.443+0000 7fd718449700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7100c2f30 con 0x7fd71014fda0 2023-03-03T19:38:38.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.443+0000 7fd718449700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7100c31d0 con 0x7fd71014f9d0 2023-03-03T19:38:38.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.443+0000 7fd718449700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7101524a0 con 0x7fd7100abe20 2023-03-03T19:38:38.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.443+0000 7fd7159e4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd71014fda0 0x7fd710071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36374/0 (socket says 172.21.15.7:36374) 2023-03-03T19:38:38.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.443+0000 7fd7159e4700 1 -- 172.21.15.7:0/4153882638 learned_addr learned my addr 172.21.15.7:0/4153882638 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:38:38.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615758988 0 0) 0x7fd7100c2f30 con 0x7fd71014fda0 2023-03-03T19:38:38.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7161e5700 1 --2- 172.21.15.7:0/4153882638 >> v2:172.21.15.143:3300/0 conn(0x7fd71014f9d0 0x7fd710156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:38:38.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7169e6700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7100abe20 0x7fd7101529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:38:38.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6fc0035b0 con 0x7fd71014fda0 2023-03-03T19:38:38.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1795515654 0 0) 0x7fd6fc0035b0 con 0x7fd71014fda0 2023-03-03T19:38:38.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7100abe20 msgr2=0x7fd7101529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:38:38.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7100abe20 0x7fd7101529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:38:38.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 >> v2:172.21.15.143:3300/0 conn(0x7fd71014f9d0 msgr2=0x7fd710156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:38:38.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 --2- 172.21.15.7:0/4153882638 >> v2:172.21.15.143:3300/0 conn(0x7fd71014f9d0 0x7fd710156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:38:38.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7100c34b0 con 0x7fd71014fda0 2023-03-03T19:38:38.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd70c001670 con 0x7fd71014fda0 2023-03-03T19:38:38.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 >> v1:172.21.15.7:6789/0 conn(0x7fd71014fda0 legacy=0x7fd710071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:38:38.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7100abe20 0x7fd6fc003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:38:38.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6fc003ff0 0x7fd6fc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:38:38.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6fc004d90 0x7fd6fc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:38:38.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7100c31d0 con 0x7fd6fc003ff0 2023-03-03T19:38:38.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7101524a0 con 0x7fd6fc004d90 2023-03-03T19:38:38.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc00bb70 con 0x7fd7100abe20 2023-03-03T19:38:38.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7159e4700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6fc003ff0 0x7fd6fc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:38:38.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7161e5700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7100abe20 0x7fd6fc003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:38:38.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7169e6700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6fc004d90 0x7fd6fc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:38:38.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7159e4700 1 -- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7100abe20 msgr2=0x7fd6fc003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:38:38.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7159e4700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd7100abe20 0x7fd6fc003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:38:38.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7159e4700 1 -- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6fc004d90 msgr2=0x7fd6fc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:38:38.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7159e4700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd6fc004d90 0x7fd6fc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:38:38.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.444+0000 7fd7159e4700 1 -- 172.21.15.7:0/4153882638 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc0035b0 con 0x7fd6fc003ff0 2023-03-03T19:38:38.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd7159e4700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6fc003ff0 0x7fd6fc004850 secure :-1 s=READY pgs=3547 cs=0 l=1 rev1=1 crypto rx=0x7fd70c001050 tx=0x7fd6fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6cbeb093061d23 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:38:38.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c007c60 con 0x7fd6fc003ff0 2023-03-03T19:38:38.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c00ce50 con 0x7fd6fc003ff0 2023-03-03T19:38:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd7151e3700 1 -- 172.21.15.7:0/4153882638 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c005ba0 con 0x7fd6fc003ff0 2023-03-03T19:38:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd718449700 1 -- 172.21.15.7:0/4153882638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6fc003ff0 msgr2=0x7fd6fc004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:38:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd718449700 1 --2- 172.21.15.7:0/4153882638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd6fc003ff0 0x7fd6fc004850 secure :-1 s=READY pgs=3547 cs=0 l=1 rev1=1 crypto rx=0x7fd70c001050 tx=0x7fd6fc005880 comp rx=0 tx=0).stop 2023-03-03T19:38:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd7169e6700 1 -- 172.21.15.7:0/4153882638 reap_dead start 2023-03-03T19:38:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd718449700 1 -- 172.21.15.7:0/4153882638 shutdown_connections 2023-03-03T19:38:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd718449700 1 -- 172.21.15.7:0/4153882638 >> 172.21.15.7:0/4153882638 conn(0x7fd7100c8900 msgr2=0x7fd7100c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:38:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd718449700 1 -- 172.21.15.7:0/4153882638 shutdown_connections 2023-03-03T19:38:38.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.445+0000 7fd718449700 1 -- 172.21.15.7:0/4153882638 wait complete. 2023-03-03T19:38:38.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.447+0000 7fd718449700 1 Processor -- start 2023-03-03T19:38:38.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.447+0000 7fd718449700 1 -- start start 2023-03-03T19:38:38.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.447+0000 7fd718449700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6fc003ff0 0x7fd71006f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:38:38.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.447+0000 7fd718449700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7101ed550 0x7fd7100c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:38:38.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.447+0000 7fd718449700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7100c1d20 0x7fd710143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:38:38.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.447+0000 7fd718449700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71006a2b0 con 0x7fd7100c1d20 2023-03-03T19:38:38.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.447+0000 7fd718449700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71006b710 con 0x7fd7101ed550 2023-03-03T19:38:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.447+0000 7fd718449700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71006b9f0 con 0x7fd6fc003ff0 2023-03-03T19:38:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.447+0000 7fd7169e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7100c1d20 0x7fd710143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:38:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd7169e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7100c1d20 0x7fd710143910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42408/0 (socket says 172.21.15.7:42408) 2023-03-03T19:38:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd7169e6700 1 -- 172.21.15.7:0/2150709166 learned_addr learned my addr 172.21.15.7:0/2150709166 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:38:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd7159e4700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7101ed550 0x7fd7100c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:38:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd7161e5700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6fc003ff0 0x7fd71006f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:38:38.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd7169e6700 1 -- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6fc003ff0 msgr2=0x7fd71006f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:38:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd7169e6700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6fc003ff0 0x7fd71006f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:38:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd7169e6700 1 -- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7101ed550 msgr2=0x7fd7100c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:38:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd7169e6700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7101ed550 0x7fd7100c17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:38:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd7169e6700 1 -- 172.21.15.7:0/2150709166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7100ac350 con 0x7fd7100c1d20 2023-03-03T19:38:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd7169e6700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7100c1d20 0x7fd710143910 secure :-1 s=READY pgs=3548 cs=0 l=1 rev1=1 crypto rx=0x7fd7100ab150 tx=0x7fd708011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:38:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd706ffd700 1 -- 172.21.15.7:0/2150709166 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70800e040 con 0x7fd7100c1d20 2023-03-03T19:38:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd706ffd700 1 -- 172.21.15.7:0/2150709166 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70800f040 con 0x7fd7100c1d20 2023-03-03T19:38:38.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd706ffd700 1 -- 172.21.15.7:0/2150709166 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70800fdb0 con 0x7fd7100c1d20 2023-03-03T19:38:38.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd718449700 1 -- 172.21.15.7:0/2150709166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7100ab630 con 0x7fd7100c1d20 2023-03-03T19:38:38.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.448+0000 7fd718449700 1 -- 172.21.15.7:0/2150709166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd710071fa0 con 0x7fd7100c1d20 2023-03-03T19:38:38.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.449+0000 7fd718449700 1 -- 172.21.15.7:0/2150709166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd710074bb0 con 0x7fd7100c1d20 2023-03-03T19:38:38.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.450+0000 7fd706ffd700 1 -- 172.21.15.7:0/2150709166 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd708022460 con 0x7fd7100c1d20 2023-03-03T19:38:38.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.450+0000 7fd706ffd700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6f40682f0 0x7fd6f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:38:38.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.450+0000 7fd706ffd700 1 -- 172.21.15.7:0/2150709166 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd708086780 con 0x7fd7100c1d20 2023-03-03T19:38:38.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.450+0000 7fd7161e5700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6f40682f0 0x7fd6f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:38:38.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.451+0000 7fd7161e5700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6f40682f0 0x7fd6f406a7a0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fd700009ba0 tx=0x7fd700008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:38:38.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.454+0000 7fd706ffd700 1 -- 172.21.15.7:0/2150709166 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd708052ac0 con 0x7fd7100c1d20 2023-03-03T19:38:38.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.691+0000 7fd718449700 1 -- 172.21.15.7:0/2150709166 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7100c9340 con 0x7fd7100c1d20 2023-03-03T19:38:38.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.691+0000 7fd706ffd700 1 -- 172.21.15.7:0/2150709166 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd7080560e0 con 0x7fd7100c1d20 2023-03-03T19:38:38.692 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:38:38.692 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:38:38.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.693+0000 7fd718449700 1 -- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6f40682f0 msgr2=0x7fd6f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:38:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.693+0000 7fd718449700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6f40682f0 0x7fd6f406a7a0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fd700009ba0 tx=0x7fd700008040 comp rx=0 tx=0).stop 2023-03-03T19:38:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.693+0000 7fd718449700 1 -- 172.21.15.7:0/2150709166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7100c1d20 msgr2=0x7fd710143910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:38:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.693+0000 7fd718449700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7100c1d20 0x7fd710143910 secure :-1 s=READY pgs=3548 cs=0 l=1 rev1=1 crypto rx=0x7fd7100ab150 tx=0x7fd708011f80 comp rx=0 tx=0).stop 2023-03-03T19:38:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.693+0000 7fd718449700 1 -- 172.21.15.7:0/2150709166 shutdown_connections 2023-03-03T19:38:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.693+0000 7fd718449700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd6f40682f0 0x7fd6f406a7a0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:38:38.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.693+0000 7fd718449700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd7100c1d20 0x7fd710143910 unknown :-1 s=CLOSED pgs=3548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:38:38.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.693+0000 7fd718449700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd6fc003ff0 0x7fd71006f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:38:38.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.693+0000 7fd718449700 1 --2- 172.21.15.7:0/2150709166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7101ed550 0x7fd7100c17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:38:38.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.693+0000 7fd718449700 1 -- 172.21.15.7:0/2150709166 >> 172.21.15.7:0/2150709166 conn(0x7fd7100c8900 msgr2=0x7fd7101518c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:38:38.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.694+0000 7fd718449700 1 -- 172.21.15.7:0/2150709166 shutdown_connections 2023-03-03T19:38:38.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:38:38.694+0000 7fd718449700 1 -- 172.21.15.7:0/2150709166 wait complete. 2023-03-03T19:38:38.696 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:38:38.706 INFO:tasks.mds_thrash.fs.[b]:waiting for 86 secs before thrashing 2023-03-03T19:38:50.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:38:50.449 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:39:02.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:39:02.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.499+0000 7fca82d05700 1 Processor -- start 2023-03-03T19:39:02.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.500+0000 7fca82d05700 1 -- start start 2023-03-03T19:39:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.500+0000 7fca82d05700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca7c072250 0x7fca7c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.500+0000 7fca82d05700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c1510d0 con 0x7fca7c0c1ce0 2023-03-03T19:39:02.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.500+0000 7fca82d05700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c0c3740 con 0x7fca7c071e80 2023-03-03T19:39:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.500+0000 7fca82d05700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7c070670 con 0x7fca7c072250 2023-03-03T19:39:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.500+0000 7fca812a2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fca7c0c1ce0 0x7fca7c15ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36414/0 (socket says 172.21.15.7:36414) 2023-03-03T19:39:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.500+0000 7fca812a2700 1 -- 172.21.15.7:0/2919743493 learned_addr learned my addr 172.21.15.7:0/2919743493 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:39:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7bfff700 1 --2- 172.21.15.7:0/2919743493 >> v2:172.21.15.143:3300/0 conn(0x7fca7c072250 0x7fca7c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199610632 0 0) 0x7fca7c1510d0 con 0x7fca7c0c1ce0 2023-03-03T19:39:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca6c0057c0 con 0x7fca7c0c1ce0 2023-03-03T19:39:02.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 9251212 0 0) 0x7fca6c0057c0 con 0x7fca7c0c1ce0 2023-03-03T19:39:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 >> v2:172.21.15.143:3300/0 conn(0x7fca7c072250 msgr2=0x7fca7c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:39:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 --2- 172.21.15.7:0/2919743493 >> v2:172.21.15.143:3300/0 conn(0x7fca7c072250 0x7fca7c14e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 >> v1:172.21.15.143:6789/0 conn(0x7fca7c071e80 legacy=0x7fca7c151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0c39e0 con 0x7fca7c0c1ce0 2023-03-03T19:39:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fca740019f0 con 0x7fca7c0c1ce0 2023-03-03T19:39:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 >> v1:172.21.15.7:6789/0 conn(0x7fca7c0c1ce0 legacy=0x7fca7c15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:02.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca6c005c80 0x7fca6c006050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:02.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c006a60 0x7fca6c006ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:02.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca6c007410 0x7fca6c00dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:02.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c070670 con 0x7fca6c005c80 2023-03-03T19:39:02.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c00e320 con 0x7fca6c007410 2023-03-03T19:39:02.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c00e4e0 con 0x7fca6c006a60 2023-03-03T19:39:02.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca80aa1700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca6c005c80 0x7fca6c006050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:02.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca812a2700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca6c007410 0x7fca6c00dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:02.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.501+0000 7fca7bfff700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c006a60 0x7fca6c006ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:02.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca80aa1700 1 -- 172.21.15.7:0/2919743493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c006a60 msgr2=0x7fca6c006ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:39:02.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca80aa1700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c006a60 0x7fca6c006ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca80aa1700 1 -- 172.21.15.7:0/2919743493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca6c007410 msgr2=0x7fca6c00dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:39:02.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca80aa1700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca6c007410 0x7fca6c00dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca80aa1700 1 -- 172.21.15.7:0/2919743493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c0057c0 con 0x7fca6c005c80 2023-03-03T19:39:02.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca812a2700 1 -- 172.21.15.7:0/2919743493 reap_dead start 2023-03-03T19:39:02.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca80aa1700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca6c005c80 0x7fca6c006050 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 crypto rx=0x7fca70007fd0 tx=0x7fca6c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2279e04a553b4ea server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:39:02.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70001d80 con 0x7fca6c005c80 2023-03-03T19:39:02.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca700052d0 con 0x7fca6c005c80 2023-03-03T19:39:02.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca7b7fe700 1 -- 172.21.15.7:0/2919743493 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70001420 con 0x7fca6c005c80 2023-03-03T19:39:02.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca82d05700 1 -- 172.21.15.7:0/2919743493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca6c005c80 msgr2=0x7fca6c006050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:02.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca82d05700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca6c005c80 0x7fca6c006050 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 crypto rx=0x7fca70007fd0 tx=0x7fca6c007a90 comp rx=0 tx=0).stop 2023-03-03T19:39:02.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca82d05700 1 -- 172.21.15.7:0/2919743493 shutdown_connections 2023-03-03T19:39:02.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca82d05700 1 --2- 172.21.15.7:0/2919743493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca6c005c80 0x7fca6c006050 unknown :-1 s=CLOSED pgs=3550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.502+0000 7fca82d05700 1 -- 172.21.15.7:0/2919743493 >> 172.21.15.7:0/2919743493 conn(0x7fca7c0a94d0 msgr2=0x7fca7c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:39:02.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.503+0000 7fca82d05700 1 -- 172.21.15.7:0/2919743493 shutdown_connections 2023-03-03T19:39:02.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.503+0000 7fca82d05700 1 -- 172.21.15.7:0/2919743493 wait complete. 2023-03-03T19:39:02.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.504+0000 7fca82d05700 1 Processor -- start 2023-03-03T19:39:02.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.504+0000 7fca82d05700 1 -- start start 2023-03-03T19:39:02.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca82d05700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c005c80 0x7fca7c1edab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:02.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca82d05700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca7c14ba40 0x7fca7c0ac4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:02.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca82d05700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca7c0c62e0 0x7fca7c0c6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:02.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca82d05700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0bd780 con 0x7fca7c0c62e0 2023-03-03T19:39:02.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca82d05700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c1edff0 con 0x7fca7c14ba40 2023-03-03T19:39:02.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca82d05700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0bd1c0 con 0x7fca6c005c80 2023-03-03T19:39:02.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca812a2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca7c0c62e0 0x7fca7c0c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:02.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca812a2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca7c0c62e0 0x7fca7c0c6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42446/0 (socket says 172.21.15.7:42446) 2023-03-03T19:39:02.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca812a2700 1 -- 172.21.15.7:0/1048617935 learned_addr learned my addr 172.21.15.7:0/1048617935 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:39:02.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca80aa1700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c005c80 0x7fca7c1edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:02.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca7bfff700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca7c14ba40 0x7fca7c0ac4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:02.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca812a2700 1 -- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c005c80 msgr2=0x7fca7c1edab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:02.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca812a2700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c005c80 0x7fca7c1edab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca812a2700 1 -- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca7c14ba40 msgr2=0x7fca7c0ac4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca812a2700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca7c14ba40 0x7fca7c0ac4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca812a2700 1 -- 172.21.15.7:0/1048617935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c06c9a0 con 0x7fca7c0c62e0 2023-03-03T19:39:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca7bfff700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca7c14ba40 0x7fca7c0ac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:39:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca80aa1700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c005c80 0x7fca7c1edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:39:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca812a2700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca7c0c62e0 0x7fca7c0c6780 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 crypto rx=0x7fca7c071650 tx=0x7fca7400a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:39:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca797fa700 1 -- 172.21.15.7:0/1048617935 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7400ac40 con 0x7fca7c0c62e0 2023-03-03T19:39:02.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca797fa700 1 -- 172.21.15.7:0/1048617935 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7400ada0 con 0x7fca7c0c62e0 2023-03-03T19:39:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca82d05700 1 -- 172.21.15.7:0/1048617935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7c151300 con 0x7fca7c0c62e0 2023-03-03T19:39:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.505+0000 7fca797fa700 1 -- 172.21.15.7:0/1048617935 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca74013630 con 0x7fca7c0c62e0 2023-03-03T19:39:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.506+0000 7fca82d05700 1 -- 172.21.15.7:0/1048617935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7c06cb30 con 0x7fca7c0c62e0 2023-03-03T19:39:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.506+0000 7fca82d05700 1 -- 172.21.15.7:0/1048617935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c1510a0 con 0x7fca7c0c62e0 2023-03-03T19:39:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.507+0000 7fca797fa700 1 -- 172.21.15.7:0/1048617935 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fca74013790 con 0x7fca7c0c62e0 2023-03-03T19:39:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.507+0000 7fca797fa700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 0x7fca6006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.507+0000 7fca797fa700 1 -- 172.21.15.7:0/1048617935 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fca74086a00 con 0x7fca7c0c62e0 2023-03-03T19:39:02.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.507+0000 7fca80aa1700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 0x7fca6006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:02.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.508+0000 7fca80aa1700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 0x7fca6006a7a0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fca70015040 tx=0x7fca7000a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:39:02.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.512+0000 7fca797fa700 1 -- 172.21.15.7:0/1048617935 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fca74052da0 con 0x7fca7c0c62e0 2023-03-03T19:39:02.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.752+0000 7fca82d05700 1 -- 172.21.15.7:0/1048617935 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca7c074bb0 con 0x7fca7c0c62e0 2023-03-03T19:39:02.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.753+0000 7fca797fa700 1 -- 172.21.15.7:0/1048617935 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fca740563c0 con 0x7fca7c0c62e0 2023-03-03T19:39:02.755 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:39:02.755 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:39:02.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 -- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 msgr2=0x7fca6006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:02.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 0x7fca6006a7a0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fca70015040 tx=0x7fca7000a000 comp rx=0 tx=0).stop 2023-03-03T19:39:02.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 -- 172.21.15.7:0/1048617935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca7c0c62e0 msgr2=0x7fca7c0c6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:02.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca7c0c62e0 0x7fca7c0c6780 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 crypto rx=0x7fca7c071650 tx=0x7fca7400a5b0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 -- 172.21.15.7:0/1048617935 shutdown_connections 2023-03-03T19:39:02.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca600682f0 0x7fca6006a7a0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca7c0c62e0 0x7fca7c0c6780 unknown :-1 s=CLOSED pgs=3551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca6c005c80 0x7fca7c1edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 --2- 172.21.15.7:0/1048617935 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca7c14ba40 0x7fca7c0ac4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:02.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 -- 172.21.15.7:0/1048617935 >> 172.21.15.7:0/1048617935 conn(0x7fca7c0a94d0 msgr2=0x7fca7c0c22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:39:02.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 -- 172.21.15.7:0/1048617935 shutdown_connections 2023-03-03T19:39:02.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:02.755+0000 7fca82d05700 1 -- 172.21.15.7:0/1048617935 wait complete. 2023-03-03T19:39:02.758 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:39:02.770 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-03-03T19:39:20.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:39:20.578 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:39:44.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:39:44.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.910+0000 7fef07bc3700 1 Processor -- start 2023-03-03T19:39:44.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.910+0000 7fef07bc3700 1 -- start start 2023-03-03T19:39:44.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.911+0000 7fef07bc3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef00151d40 0x7fef000736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:44.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.911+0000 7fef07bc3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef000c2930 0x7fef00157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:44.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.911+0000 7fef07bc3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0006d430 con 0x7fef00073c30 2023-03-03T19:39:44.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.911+0000 7fef07bc3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0006c2a0 con 0x7fef00151d40 2023-03-03T19:39:44.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.911+0000 7fef07bc3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef001586e0 con 0x7fef000c2930 2023-03-03T19:39:44.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.911+0000 7fef0595f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fef00151d40 0x7fef000736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:44.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.911+0000 7fef06160700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fef00073c30 0x7fef00157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36450/0 (socket says 172.21.15.7:36450) 2023-03-03T19:39:44.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.911+0000 7fef06160700 1 -- 172.21.15.7:0/1847708890 learned_addr learned my addr 172.21.15.7:0/1847708890 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:39:44.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.911+0000 7fef0515e700 1 --2- 172.21.15.7:0/1847708890 >> v2:172.21.15.143:3300/0 conn(0x7fef000c2930 0x7fef00157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:44.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef0595f700 1 -- 172.21.15.7:0/1847708890 >> v2:172.21.15.143:3300/0 conn(0x7fef000c2930 msgr2=0x7fef00157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:39:44.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef0595f700 1 --2- 172.21.15.7:0/1847708890 >> v2:172.21.15.143:3300/0 conn(0x7fef000c2930 0x7fef00157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:44.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef0595f700 1 -- 172.21.15.7:0/1847708890 >> v1:172.21.15.7:6789/0 conn(0x7fef00073c30 legacy=0x7fef00157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:44.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef0595f700 1 -- 172.21.15.7:0/1847708890 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0006cb00 con 0x7fef00151d40 2023-03-03T19:39:44.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef0595f700 1 --2- 172.21.15.7:0/1847708890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00151d40 0x7fef000736f0 secure :-1 s=READY pgs=3553 cs=0 l=1 rev1=1 crypto rx=0x7feef0009840 tx=0x7feef0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5cf1d0329ba4742 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:39:44.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef0495d700 1 -- 172.21.15.7:0/1847708890 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0011830 con 0x7fef00151d40 2023-03-03T19:39:44.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef0495d700 1 -- 172.21.15.7:0/1847708890 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef0011990 con 0x7fef00151d40 2023-03-03T19:39:44.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef0495d700 1 -- 172.21.15.7:0/1847708890 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0011d20 con 0x7fef00151d40 2023-03-03T19:39:44.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef07bc3700 1 -- 172.21.15.7:0/1847708890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00151d40 msgr2=0x7fef000736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:44.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef07bc3700 1 --2- 172.21.15.7:0/1847708890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00151d40 0x7fef000736f0 secure :-1 s=READY pgs=3553 cs=0 l=1 rev1=1 crypto rx=0x7feef0009840 tx=0x7feef0009b50 comp rx=0 tx=0).stop 2023-03-03T19:39:44.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef07bc3700 1 -- 172.21.15.7:0/1847708890 shutdown_connections 2023-03-03T19:39:44.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.912+0000 7fef07bc3700 1 --2- 172.21.15.7:0/1847708890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00151d40 0x7fef000736f0 unknown :-1 s=CLOSED pgs=3553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:44.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.913+0000 7fef07bc3700 1 --2- 172.21.15.7:0/1847708890 >> v2:172.21.15.143:3300/0 conn(0x7fef000c2930 0x7fef00157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:44.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.913+0000 7fef07bc3700 1 -- 172.21.15.7:0/1847708890 >> 172.21.15.7:0/1847708890 conn(0x7fef000a94d0 msgr2=0x7fef000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:39:44.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.913+0000 7fef07bc3700 1 -- 172.21.15.7:0/1847708890 shutdown_connections 2023-03-03T19:39:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.913+0000 7fef07bc3700 1 -- 172.21.15.7:0/1847708890 wait complete. 2023-03-03T19:39:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.914+0000 7fef07bc3700 1 Processor -- start 2023-03-03T19:39:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.914+0000 7fef07bc3700 1 -- start start 2023-03-03T19:39:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef07bc3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00073c30 0x7fef00156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef07bc3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef000c2930 0x7fef000be5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef07bc3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef00151d40 0x7fef000beb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef07bc3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef001578d0 con 0x7fef00073c30 2023-03-03T19:39:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef07bc3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef000c1040 con 0x7fef00151d40 2023-03-03T19:39:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef07bc3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef00072480 con 0x7fef000c2930 2023-03-03T19:39:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef0595f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00073c30 0x7fef00156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef0595f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00073c30 0x7fef00156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42472/0 (socket says 172.21.15.7:42472) 2023-03-03T19:39:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef0595f700 1 -- 172.21.15.7:0/1325953636 learned_addr learned my addr 172.21.15.7:0/1325953636 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:39:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef0595f700 1 -- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef000c2930 msgr2=0x7fef000be5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef06160700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef00151d40 0x7fef000beb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef0595f700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef000c2930 0x7fef000be5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef0595f700 1 -- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef00151d40 msgr2=0x7fef000beb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef0595f700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef00151d40 0x7fef000beb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef0595f700 1 -- 172.21.15.7:0/1325953636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef00094a0 con 0x7fef00073c30 2023-03-03T19:39:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef0595f700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00073c30 0x7fef00156ba0 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 crypto rx=0x7fef0006bb70 tx=0x7feef0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:39:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7feef67fc700 1 -- 172.21.15.7:0/1325953636 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef00038a0 con 0x7fef00073c30 2023-03-03T19:39:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7feef67fc700 1 -- 172.21.15.7:0/1325953636 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef001f070 con 0x7fef00073c30 2023-03-03T19:39:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef07bc3700 1 -- 172.21.15.7:0/1325953636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0006d0c0 con 0x7fef00073c30 2023-03-03T19:39:44.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7feef67fc700 1 -- 172.21.15.7:0/1325953636 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0003c20 con 0x7fef00073c30 2023-03-03T19:39:44.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.915+0000 7fef07bc3700 1 -- 172.21.15.7:0/1325953636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0006cde0 con 0x7fef00073c30 2023-03-03T19:39:44.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.916+0000 7fef07bc3700 1 -- 172.21.15.7:0/1325953636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0006bea0 con 0x7fef00073c30 2023-03-03T19:39:44.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.917+0000 7feef67fc700 1 -- 172.21.15.7:0/1325953636 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feef000f050 con 0x7fef00073c30 2023-03-03T19:39:44.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.917+0000 7feef67fc700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 0x7feee406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:39:44.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.917+0000 7feef67fc700 1 -- 172.21.15.7:0/1325953636 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7feef0021070 con 0x7fef00073c30 2023-03-03T19:39:44.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.918+0000 7fef0515e700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 0x7feee406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:39:44.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.918+0000 7fef0515e700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 0x7feee406a7a0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fef000ab420 tx=0x7feefc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:39:44.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:44.922+0000 7feef67fc700 1 -- 172.21.15.7:0/1325953636 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7feef0053e40 con 0x7fef00073c30 2023-03-03T19:39:45.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.160+0000 7fef07bc3700 1 -- 172.21.15.7:0/1325953636 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef00074bb0 con 0x7fef00073c30 2023-03-03T19:39:45.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.161+0000 7feef67fc700 1 -- 172.21.15.7:0/1325953636 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7feef0057460 con 0x7fef00073c30 2023-03-03T19:39:45.161 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:39:45.162 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:39:45.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 -- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 msgr2=0x7feee406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:45.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 0x7feee406a7a0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fef000ab420 tx=0x7feefc008040 comp rx=0 tx=0).stop 2023-03-03T19:39:45.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 -- 172.21.15.7:0/1325953636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00073c30 msgr2=0x7fef00156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:39:45.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00073c30 0x7fef00156ba0 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 crypto rx=0x7fef0006bb70 tx=0x7feef0004750 comp rx=0 tx=0).stop 2023-03-03T19:39:45.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 -- 172.21.15.7:0/1325953636 shutdown_connections 2023-03-03T19:39:45.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef00151d40 0x7fef000beb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:45.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7feee40682f0 0x7feee406a7a0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:45.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef00073c30 0x7fef00156ba0 unknown :-1 s=CLOSED pgs=3554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 --2- 172.21.15.7:0/1325953636 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef000c2930 0x7fef000be5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:39:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 -- 172.21.15.7:0/1325953636 >> 172.21.15.7:0/1325953636 conn(0x7fef000a94d0 msgr2=0x7fef000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:39:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 -- 172.21.15.7:0/1325953636 shutdown_connections 2023-03-03T19:39:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:39:45.163+0000 7fef07bc3700 1 -- 172.21.15.7:0/1325953636 wait complete. 2023-03-03T19:39:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:39:45.175 INFO:tasks.mds_thrash.fs.[a]:waiting for 96 secs before thrashing 2023-03-03T19:39:50.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:39:50.718 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:39:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T19:39:59.998+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T19:40:04.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:40:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.846+0000 7fd58e3b6700 1 Processor -- start 2023-03-03T19:40:04.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.846+0000 7fd58e3b6700 1 -- start start 2023-03-03T19:40:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.847+0000 7fd58e3b6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd588151b20 0x7fd588071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:40:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.847+0000 7fd58e3b6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588154f10 0x7fd588158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:40:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.847+0000 7fd58e3b6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd588151ef0 con 0x7fd588071d60 2023-03-03T19:40:04.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.847+0000 7fd58e3b6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd588157b20 con 0x7fd588151b20 2023-03-03T19:40:04.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.847+0000 7fd58e3b6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd588155410 con 0x7fd588154f10 2023-03-03T19:40:04.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.847+0000 7fd5877fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588154f10 0x7fd588158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:40:04.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.847+0000 7fd587fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd588151b20 0x7fd588071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:40:04.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd5877fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588154f10 0x7fd588158000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:50570/0 (socket says 172.21.15.7:50570) 2023-03-03T19:40:04.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd587fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd588151b20 0x7fd588071820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:38272/0 (socket says 172.21.15.7:38272) 2023-03-03T19:40:04.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.847+0000 7fd58c953700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd588071d60 0x7fd5880c17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:39998/0 (socket says 172.21.15.7:39998) 2023-03-03T19:40:04.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd5877fe700 1 -- 172.21.15.7:0/3399198276 learned_addr learned my addr 172.21.15.7:0/3399198276 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:40:04.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd5877fe700 1 -- 172.21.15.7:0/3399198276 >> v2:172.21.15.143:3300/0 conn(0x7fd588151b20 msgr2=0x7fd588071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:40:04.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd5877fe700 1 --2- 172.21.15.7:0/3399198276 >> v2:172.21.15.143:3300/0 conn(0x7fd588151b20 0x7fd588071820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:40:04.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd5877fe700 1 -- 172.21.15.7:0/3399198276 >> v1:172.21.15.143:6789/0 conn(0x7fd588071d60 legacy=0x7fd5880c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:40:04.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd5877fe700 1 -- 172.21.15.7:0/3399198276 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58806f640 con 0x7fd588154f10 2023-03-03T19:40:04.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd587fff700 1 --2- 172.21.15.7:0/3399198276 >> v2:172.21.15.143:3300/0 conn(0x7fd588151b20 0x7fd588071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:40:04.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd5877fe700 1 --2- 172.21.15.7:0/3399198276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588154f10 0x7fd588158000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0098f0 tx=0x7fd57c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f54e35396ddb9ce1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:40:04.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd586ffd700 1 -- 172.21.15.7:0/3399198276 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c011830 con 0x7fd588154f10 2023-03-03T19:40:04.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd586ffd700 1 -- 172.21.15.7:0/3399198276 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c011990 con 0x7fd588154f10 2023-03-03T19:40:04.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.848+0000 7fd586ffd700 1 -- 172.21.15.7:0/3399198276 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c011d20 con 0x7fd588154f10 2023-03-03T19:40:04.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.849+0000 7fd58e3b6700 1 -- 172.21.15.7:0/3399198276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588154f10 msgr2=0x7fd588158000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:40:04.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.849+0000 7fd58e3b6700 1 --2- 172.21.15.7:0/3399198276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588154f10 0x7fd588158000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0098f0 tx=0x7fd57c009c00 comp rx=0 tx=0).stop 2023-03-03T19:40:04.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.849+0000 7fd58e3b6700 1 -- 172.21.15.7:0/3399198276 shutdown_connections 2023-03-03T19:40:04.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.849+0000 7fd58e3b6700 1 --2- 172.21.15.7:0/3399198276 >> v2:172.21.15.143:3300/0 conn(0x7fd588151b20 0x7fd588071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:40:04.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.849+0000 7fd58e3b6700 1 --2- 172.21.15.7:0/3399198276 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588154f10 0x7fd588158000 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:40:04.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.849+0000 7fd58e3b6700 1 -- 172.21.15.7:0/3399198276 >> 172.21.15.7:0/3399198276 conn(0x7fd5880c8900 msgr2=0x7fd5880c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:40:04.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.849+0000 7fd58e3b6700 1 -- 172.21.15.7:0/3399198276 shutdown_connections 2023-03-03T19:40:04.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.849+0000 7fd58e3b6700 1 -- 172.21.15.7:0/3399198276 wait complete. 2023-03-03T19:40:04.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.850+0000 7fd58e3b6700 1 Processor -- start 2023-03-03T19:40:04.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.851+0000 7fd58e3b6700 1 -- start start 2023-03-03T19:40:04.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd58e3b6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd588071d60 0x7fd58814bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:40:04.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd58e3b6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588151b20 0x7fd5880c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:40:04.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd58e3b6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd588154f10 0x7fd588143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:40:04.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd58e3b6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58814d2c0 con 0x7fd588071d60 2023-03-03T19:40:04.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd58e3b6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58814c300 con 0x7fd588154f10 2023-03-03T19:40:04.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd58e3b6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd588150e90 con 0x7fd588151b20 2023-03-03T19:40:04.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd587fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd588071d60 0x7fd58814bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:40:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd587fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd588071d60 0x7fd58814bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42504/0 (socket says 172.21.15.7:42504) 2023-03-03T19:40:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd587fff700 1 -- 172.21.15.7:0/446035904 learned_addr learned my addr 172.21.15.7:0/446035904 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:40:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd5877fe700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588151b20 0x7fd5880c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:40:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd58c953700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd588154f10 0x7fd588143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:40:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd587fff700 1 -- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588151b20 msgr2=0x7fd5880c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:40:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd587fff700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588151b20 0x7fd5880c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:40:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd587fff700 1 -- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd588154f10 msgr2=0x7fd588143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:40:04.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd587fff700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd588154f10 0x7fd588143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:40:04.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd587fff700 1 -- 172.21.15.7:0/446035904 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5780094a0 con 0x7fd588071d60 2023-03-03T19:40:04.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.852+0000 7fd587fff700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd588071d60 0x7fd58814bdc0 secure :-1 s=READY pgs=3555 cs=0 l=1 rev1=1 crypto rx=0x7fd57800ebe0 tx=0x7fd57800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:40:04.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.854+0000 7fd584ff9700 1 -- 172.21.15.7:0/446035904 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57800c850 con 0x7fd588071d60 2023-03-03T19:40:04.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.854+0000 7fd584ff9700 1 -- 172.21.15.7:0/446035904 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57800c9b0 con 0x7fd588071d60 2023-03-03T19:40:04.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.854+0000 7fd58e3b6700 1 -- 172.21.15.7:0/446035904 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c009550 con 0x7fd588071d60 2023-03-03T19:40:04.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.854+0000 7fd584ff9700 1 -- 172.21.15.7:0/446035904 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57800cd40 con 0x7fd588071d60 2023-03-03T19:40:04.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.854+0000 7fd58e3b6700 1 -- 172.21.15.7:0/446035904 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58806fc00 con 0x7fd588071d60 2023-03-03T19:40:04.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.855+0000 7fd58e3b6700 1 -- 172.21.15.7:0/446035904 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58806f920 con 0x7fd588071d60 2023-03-03T19:40:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.856+0000 7fd584ff9700 1 -- 172.21.15.7:0/446035904 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd578019030 con 0x7fd588071d60 2023-03-03T19:40:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.856+0000 7fd584ff9700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5740682f0 0x7fd57406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:40:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.856+0000 7fd584ff9700 1 -- 172.21.15.7:0/446035904 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd5780861f0 con 0x7fd588071d60 2023-03-03T19:40:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.857+0000 7fd5877fe700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5740682f0 0x7fd57406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:40:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.857+0000 7fd5877fe700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5740682f0 0x7fd57406a7a0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fd588155340 tx=0x7fd57c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:40:04.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:04.861+0000 7fd584ff9700 1 -- 172.21.15.7:0/446035904 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd5780524a0 con 0x7fd588071d60 2023-03-03T19:40:05.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.096+0000 7fd58e3b6700 1 -- 172.21.15.7:0/446035904 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5880aae90 con 0x7fd588071d60 2023-03-03T19:40:05.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.097+0000 7fd584ff9700 1 -- 172.21.15.7:0/446035904 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd57800cea0 con 0x7fd588071d60 2023-03-03T19:40:05.099 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:40:05.099 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:40:05.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 -- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5740682f0 msgr2=0x7fd57406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:40:05.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5740682f0 0x7fd57406a7a0 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7fd588155340 tx=0x7fd57c008030 comp rx=0 tx=0).stop 2023-03-03T19:40:05.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 -- 172.21.15.7:0/446035904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd588071d60 msgr2=0x7fd58814bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:40:05.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd588071d60 0x7fd58814bdc0 secure :-1 s=READY pgs=3555 cs=0 l=1 rev1=1 crypto rx=0x7fd57800ebe0 tx=0x7fd57800eef0 comp rx=0 tx=0).stop 2023-03-03T19:40:05.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 -- 172.21.15.7:0/446035904 shutdown_connections 2023-03-03T19:40:05.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd588154f10 0x7fd588143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:40:05.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd5740682f0 0x7fd57406a7a0 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:40:05.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd588071d60 0x7fd58814bdc0 unknown :-1 s=CLOSED pgs=3555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:40:05.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 --2- 172.21.15.7:0/446035904 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd588151b20 0x7fd5880c1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:40:05.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 -- 172.21.15.7:0/446035904 >> 172.21.15.7:0/446035904 conn(0x7fd5880c8900 msgr2=0x7fd58814b370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:40:05.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 -- 172.21.15.7:0/446035904 shutdown_connections 2023-03-03T19:40:05.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:40:05.099+0000 7fd58e3b6700 1 -- 172.21.15.7:0/446035904 wait complete. 2023-03-03T19:40:05.103 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:40:05.113 INFO:tasks.mds_thrash.fs.[b]:waiting for 55 secs before thrashing 2023-03-03T19:40:20.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:40:20.793 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:40:50.869 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:40:50.873 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:41:00.114 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:41:00.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.249+0000 7fba23706700 1 Processor -- start 2023-03-03T19:41:00.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.249+0000 7fba23706700 1 -- start start 2023-03-03T19:41:00.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba23706700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fba1c0caa30 0x7fba1c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:00.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba23706700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba1c0c00e0 0x7fba1c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:00.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba23706700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1c0c7310 con 0x7fba1c0cae00 2023-03-03T19:41:00.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba23706700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1c0cb240 con 0x7fba1c0c00e0 2023-03-03T19:41:00.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba23706700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1c0bfd10 con 0x7fba1c0caa30 2023-03-03T19:41:00.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba20ca1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba1c0c00e0 0x7fba1c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:00.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba20ca1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba1c0c00e0 0x7fba1c154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42570/0 (socket says 172.21.15.7:42570) 2023-03-03T19:41:00.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba20ca1700 1 -- 172.21.15.7:0/342352326 learned_addr learned my addr 172.21.15.7:0/342352326 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:41:00.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba214a2700 1 --2- 172.21.15.7:0/342352326 >> v2:172.21.15.143:3300/0 conn(0x7fba1c0caa30 0x7fba1c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:00.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba20ca1700 1 -- 172.21.15.7:0/342352326 >> v2:172.21.15.143:3300/0 conn(0x7fba1c0caa30 msgr2=0x7fba1c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:41:00.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba20ca1700 1 --2- 172.21.15.7:0/342352326 >> v2:172.21.15.143:3300/0 conn(0x7fba1c0caa30 0x7fba1c0c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:00.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba20ca1700 1 -- 172.21.15.7:0/342352326 >> v1:172.21.15.143:6789/0 conn(0x7fba1c0cae00 legacy=0x7fba1c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:41:00.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.250+0000 7fba20ca1700 1 -- 172.21.15.7:0/342352326 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c0bfa30 con 0x7fba1c0c00e0 2023-03-03T19:41:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba20ca1700 1 --2- 172.21.15.7:0/342352326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0c00e0 0x7fba1c154ff0 secure :-1 s=READY pgs=3556 cs=0 l=1 rev1=1 crypto rx=0x7fba1c0c81b0 tx=0x7fba18009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7eba21f3e195c9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:41:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba13fff700 1 -- 172.21.15.7:0/342352326 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18005e30 con 0x7fba1c0c00e0 2023-03-03T19:41:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba13fff700 1 -- 172.21.15.7:0/342352326 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba18006030 con 0x7fba1c0c00e0 2023-03-03T19:41:00.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba13fff700 1 -- 172.21.15.7:0/342352326 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba180063e0 con 0x7fba1c0c00e0 2023-03-03T19:41:00.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba23706700 1 -- 172.21.15.7:0/342352326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0c00e0 msgr2=0x7fba1c154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba23706700 1 --2- 172.21.15.7:0/342352326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0c00e0 0x7fba1c154ff0 secure :-1 s=READY pgs=3556 cs=0 l=1 rev1=1 crypto rx=0x7fba1c0c81b0 tx=0x7fba18009bd0 comp rx=0 tx=0).stop 2023-03-03T19:41:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba23706700 1 -- 172.21.15.7:0/342352326 shutdown_connections 2023-03-03T19:41:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba23706700 1 --2- 172.21.15.7:0/342352326 >> v2:172.21.15.143:3300/0 conn(0x7fba1c0caa30 0x7fba1c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba23706700 1 --2- 172.21.15.7:0/342352326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0c00e0 0x7fba1c154ff0 unknown :-1 s=CLOSED pgs=3556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba23706700 1 -- 172.21.15.7:0/342352326 >> 172.21.15.7:0/342352326 conn(0x7fba1c066e40 msgr2=0x7fba1c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:41:00.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba23706700 1 -- 172.21.15.7:0/342352326 shutdown_connections 2023-03-03T19:41:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.251+0000 7fba23706700 1 -- 172.21.15.7:0/342352326 wait complete. 2023-03-03T19:41:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba23706700 1 Processor -- start 2023-03-03T19:41:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba23706700 1 -- start start 2023-03-03T19:41:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba23706700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1c0c00e0 0x7fba1c1ea160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba23706700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0caa30 0x7fba1c14c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba23706700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1c0cae00 0x7fba1c14b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba23706700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c0c19e0 con 0x7fba1c0caa30 2023-03-03T19:41:00.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba23706700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1ea6a0 con 0x7fba1c0c00e0 2023-03-03T19:41:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba23706700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c0c42b0 con 0x7fba1c0cae00 2023-03-03T19:41:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba20ca1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0caa30 0x7fba1c14c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba20ca1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0caa30 0x7fba1c14c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42574/0 (socket says 172.21.15.7:42574) 2023-03-03T19:41:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.253+0000 7fba20ca1700 1 -- 172.21.15.7:0/2271644973 learned_addr learned my addr 172.21.15.7:0/2271644973 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:41:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba214a2700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1c0c00e0 0x7fba1c1ea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba21ca3700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1c0cae00 0x7fba1c14b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:00.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba20ca1700 1 -- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1c0cae00 msgr2=0x7fba1c14b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:00.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba20ca1700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1c0cae00 0x7fba1c14b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:00.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba20ca1700 1 -- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1c0c00e0 msgr2=0x7fba1c1ea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:00.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba20ca1700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1c0c00e0 0x7fba1c1ea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:00.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba20ca1700 1 -- 172.21.15.7:0/2271644973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba18009500 con 0x7fba1c0caa30 2023-03-03T19:41:00.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba214a2700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1c0c00e0 0x7fba1c1ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:41:00.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba20ca1700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0caa30 0x7fba1c14c920 secure :-1 s=READY pgs=3557 cs=0 l=1 rev1=1 crypto rx=0x7fba18006000 tx=0x7fba18004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:41:00.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba11ffb700 1 -- 172.21.15.7:0/2271644973 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18004250 con 0x7fba1c0caa30 2023-03-03T19:41:00.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba11ffb700 1 -- 172.21.15.7:0/2271644973 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba180043b0 con 0x7fba1c0caa30 2023-03-03T19:41:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba23706700 1 -- 172.21.15.7:0/2271644973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1c1547f0 con 0x7fba1c0caa30 2023-03-03T19:41:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba11ffb700 1 -- 172.21.15.7:0/2271644973 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba180036a0 con 0x7fba1c0caa30 2023-03-03T19:41:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.254+0000 7fba23706700 1 -- 172.21.15.7:0/2271644973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1c154510 con 0x7fba1c0caa30 2023-03-03T19:41:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.255+0000 7fba23706700 1 -- 172.21.15.7:0/2271644973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba1c154ad0 con 0x7fba1c0caa30 2023-03-03T19:41:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.256+0000 7fba11ffb700 1 -- 172.21.15.7:0/2271644973 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fba1800f050 con 0x7fba1c0caa30 2023-03-03T19:41:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.256+0000 7fba11ffb700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 0x7fba0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:00.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.256+0000 7fba11ffb700 1 -- 172.21.15.7:0/2271644973 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fba18087d80 con 0x7fba1c0caa30 2023-03-03T19:41:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.256+0000 7fba214a2700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 0x7fba0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.257+0000 7fba214a2700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 0x7fba0006a7a0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fba0c006fd0 tx=0x7fba0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:41:00.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.261+0000 7fba11ffb700 1 -- 172.21.15.7:0/2271644973 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fba18054120 con 0x7fba1c0caa30 2023-03-03T19:41:00.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.502+0000 7fba23706700 1 -- 172.21.15.7:0/2271644973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba1c074bb0 con 0x7fba1c0caa30 2023-03-03T19:41:00.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.503+0000 7fba11ffb700 1 -- 172.21.15.7:0/2271644973 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fba1805b680 con 0x7fba1c0caa30 2023-03-03T19:41:00.504 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:41:00.504 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:41:00.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 -- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 msgr2=0x7fba0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:00.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 0x7fba0006a7a0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fba0c006fd0 tx=0x7fba0c008040 comp rx=0 tx=0).stop 2023-03-03T19:41:00.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 -- 172.21.15.7:0/2271644973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0caa30 msgr2=0x7fba1c14c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:00.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0caa30 0x7fba1c14c920 secure :-1 s=READY pgs=3557 cs=0 l=1 rev1=1 crypto rx=0x7fba18006000 tx=0x7fba18004750 comp rx=0 tx=0).stop 2023-03-03T19:41:00.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 -- 172.21.15.7:0/2271644973 shutdown_connections 2023-03-03T19:41:00.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fba1c0cae00 0x7fba1c14b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:00.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba1c0c00e0 0x7fba1c1ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:00.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fba000682f0 0x7fba0006a7a0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:00.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 --2- 172.21.15.7:0/2271644973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba1c0caa30 0x7fba1c14c920 unknown :-1 s=CLOSED pgs=3557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:00.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 -- 172.21.15.7:0/2271644973 >> 172.21.15.7:0/2271644973 conn(0x7fba1c066e40 msgr2=0x7fba1c0c62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:41:00.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 -- 172.21.15.7:0/2271644973 shutdown_connections 2023-03-03T19:41:00.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:00.505+0000 7fba23706700 1 -- 172.21.15.7:0/2271644973 wait complete. 2023-03-03T19:41:00.508 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:41:00.517 INFO:tasks.mds_thrash.fs.[b]:waiting for 50 secs before thrashing 2023-03-03T19:41:20.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:41:20.953 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:41:21.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:41:21.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.315+0000 7f6f04da9700 1 Processor -- start 2023-03-03T19:41:21.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.315+0000 7f6f04da9700 1 -- start start 2023-03-03T19:41:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.316+0000 7f6f04da9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f00151d40 0x7f6f000736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.316+0000 7f6f04da9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6f000c2930 0x7f6f00157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.316+0000 7f6f04da9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0006d430 con 0x7f6f00073c30 2023-03-03T19:41:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.316+0000 7f6f04da9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0006c2a0 con 0x7f6f000c2930 2023-03-03T19:41:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.316+0000 7f6f04da9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f001586e0 con 0x7f6f00151d40 2023-03-03T19:41:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.316+0000 7f6efe59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f00151d40 0x7f6f000736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.316+0000 7f6efdd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6f000c2930 0x7f6f00157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.316+0000 7f6efed9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f6f00073c30 0x7f6f00157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:40088/0 (socket says 172.21.15.7:40088) 2023-03-03T19:41:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.316+0000 7f6efed9d700 1 -- 172.21.15.7:0/1052622159 learned_addr learned my addr 172.21.15.7:0/1052622159 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:41:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.317+0000 7f6efdd9b700 1 -- 172.21.15.7:0/1052622159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f00151d40 msgr2=0x7f6f000736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:41:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.317+0000 7f6efdd9b700 1 --2- 172.21.15.7:0/1052622159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f00151d40 0x7f6f000736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.317+0000 7f6efdd9b700 1 -- 172.21.15.7:0/1052622159 >> v1:172.21.15.143:6789/0 conn(0x7f6f00073c30 legacy=0x7f6f00157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.317+0000 7f6efdd9b700 1 -- 172.21.15.7:0/1052622159 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0006cb00 con 0x7f6f000c2930 2023-03-03T19:41:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.317+0000 7f6efd59a700 1 -- 172.21.15.7:0/1052622159 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979724800 0 0) 0x7f6f0006d430 con 0x7f6f00073c30 2023-03-03T19:41:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.317+0000 7f6efdd9b700 1 --2- 172.21.15.7:0/1052622159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f000c2930 0x7f6f00157a90 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009840 tx=0x7f6ef0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34cc5da7d700c405 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:41:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.317+0000 7f6efd59a700 1 -- 172.21.15.7:0/1052622159 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0011830 con 0x7f6f000c2930 2023-03-03T19:41:21.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.317+0000 7f6efd59a700 1 -- 172.21.15.7:0/1052622159 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef0011990 con 0x7f6f000c2930 2023-03-03T19:41:21.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.317+0000 7f6efd59a700 1 -- 172.21.15.7:0/1052622159 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0011d20 con 0x7f6f000c2930 2023-03-03T19:41:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.318+0000 7f6f04da9700 1 -- 172.21.15.7:0/1052622159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f000c2930 msgr2=0x7f6f00157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.318+0000 7f6f04da9700 1 --2- 172.21.15.7:0/1052622159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f000c2930 0x7f6f00157a90 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009840 tx=0x7f6ef0009b50 comp rx=0 tx=0).stop 2023-03-03T19:41:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.318+0000 7f6f04da9700 1 -- 172.21.15.7:0/1052622159 shutdown_connections 2023-03-03T19:41:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.318+0000 7f6f04da9700 1 --2- 172.21.15.7:0/1052622159 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f00151d40 0x7f6f000736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.318+0000 7f6f04da9700 1 --2- 172.21.15.7:0/1052622159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f000c2930 0x7f6f00157a90 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:21.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.318+0000 7f6f04da9700 1 -- 172.21.15.7:0/1052622159 >> 172.21.15.7:0/1052622159 conn(0x7f6f000a94d0 msgr2=0x7f6f000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:41:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.318+0000 7f6f04da9700 1 -- 172.21.15.7:0/1052622159 shutdown_connections 2023-03-03T19:41:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.318+0000 7f6f04da9700 1 -- 172.21.15.7:0/1052622159 wait complete. 2023-03-03T19:41:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.319+0000 7f6f04da9700 1 Processor -- start 2023-03-03T19:41:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.319+0000 7f6f04da9700 1 -- start start 2023-03-03T19:41:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6f04da9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f00073c30 0x7f6f000c0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6f04da9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f000c2930 0x7f6f0014e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6f04da9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f00151d40 0x7f6f0014e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:21.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6f04da9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00150bf0 con 0x7f6f00151d40 2023-03-03T19:41:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6f04da9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f001577f0 con 0x7f6f000c2930 2023-03-03T19:41:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6f04da9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f000acd80 con 0x7f6f00073c30 2023-03-03T19:41:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f00151d40 0x7f6f0014e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f00151d40 0x7f6f0014e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42594/0 (socket says 172.21.15.7:42594) 2023-03-03T19:41:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efed9d700 1 -- 172.21.15.7:0/3486930758 learned_addr learned my addr 172.21.15.7:0/3486930758 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:41:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efdd9b700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f000c2930 0x7f6f0014e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efe59c700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f00073c30 0x7f6f000c0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efdd9b700 1 -- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f00073c30 msgr2=0x7f6f000c0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:21.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efdd9b700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f00073c30 0x7f6f000c0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:21.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efdd9b700 1 -- 172.21.15.7:0/3486930758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f00151d40 msgr2=0x7f6f0014e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:21.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efdd9b700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f00151d40 0x7f6f0014e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:21.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efdd9b700 1 -- 172.21.15.7:0/3486930758 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef00094a0 con 0x7f6f000c2930 2023-03-03T19:41:21.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efed9d700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f00151d40 0x7f6f0014e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:41:21.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.320+0000 7f6efdd9b700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f000c2930 0x7f6f0014e100 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0016000 tx=0x7f6ef0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:41:21.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.321+0000 7f6eeeffd700 1 -- 172.21.15.7:0/3486930758 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0004510 con 0x7f6f000c2930 2023-03-03T19:41:21.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.321+0000 7f6eeeffd700 1 -- 172.21.15.7:0/3486930758 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef001e070 con 0x7f6f000c2930 2023-03-03T19:41:21.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.321+0000 7f6f04da9700 1 -- 172.21.15.7:0/3486930758 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0006d0c0 con 0x7f6f000c2930 2023-03-03T19:41:21.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.321+0000 7f6eeeffd700 1 -- 172.21.15.7:0/3486930758 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00038e0 con 0x7f6f000c2930 2023-03-03T19:41:21.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.321+0000 7f6f04da9700 1 -- 172.21.15.7:0/3486930758 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0006cde0 con 0x7f6f000c2930 2023-03-03T19:41:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.322+0000 7f6f04da9700 1 -- 172.21.15.7:0/3486930758 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0006bea0 con 0x7f6f000c2930 2023-03-03T19:41:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.322+0000 7f6eeeffd700 1 -- 172.21.15.7:0/3486930758 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6ef000f050 con 0x7f6f000c2930 2023-03-03T19:41:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.322+0000 7f6eeeffd700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee00682f0 0x7f6ee006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.323+0000 7f6eeeffd700 1 -- 172.21.15.7:0/3486930758 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6ef0086f00 con 0x7f6f000c2930 2023-03-03T19:41:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.323+0000 7f6efe59c700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee00682f0 0x7f6ee006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:21.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.323+0000 7f6efe59c700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee00682f0 0x7f6ee006a7a0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009be0 tx=0x7f6ef4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:41:21.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.327+0000 7f6eeeffd700 1 -- 172.21.15.7:0/3486930758 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6ef00532a0 con 0x7f6f000c2930 2023-03-03T19:41:21.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.568+0000 7f6f04da9700 1 -- 172.21.15.7:0/3486930758 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f000acbe0 con 0x7f6f000c2930 2023-03-03T19:41:21.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.569+0000 7f6eeeffd700 1 -- 172.21.15.7:0/3486930758 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6f000acbe0 con 0x7f6f000c2930 2023-03-03T19:41:21.570 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:41:21.570 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:41:21.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 -- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee00682f0 msgr2=0x7f6ee006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:21.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee00682f0 0x7f6ee006a7a0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009be0 tx=0x7f6ef4009210 comp rx=0 tx=0).stop 2023-03-03T19:41:21.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 -- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f000c2930 msgr2=0x7f6f0014e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:21.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f000c2930 0x7f6f0014e100 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0016000 tx=0x7f6ef0004750 comp rx=0 tx=0).stop 2023-03-03T19:41:21.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 -- 172.21.15.7:0/3486930758 shutdown_connections 2023-03-03T19:41:21.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6ee00682f0 0x7f6ee006a7a0 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:21.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f00151d40 0x7f6f0014e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:21.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f00073c30 0x7f6f000c0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:21.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 --2- 172.21.15.7:0/3486930758 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f000c2930 0x7f6f0014e100 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:21.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 -- 172.21.15.7:0/3486930758 >> 172.21.15.7:0/3486930758 conn(0x7f6f000a94d0 msgr2=0x7f6f000c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:41:21.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 -- 172.21.15.7:0/3486930758 shutdown_connections 2023-03-03T19:41:21.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:21.571+0000 7f6f04da9700 1 -- 172.21.15.7:0/3486930758 wait complete. 2023-03-03T19:41:21.574 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:41:21.583 INFO:tasks.mds_thrash.fs.[a]:waiting for 46 secs before thrashing 2023-03-03T19:41:50.519 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:41:50.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.657+0000 7f201fedf700 1 Processor -- start 2023-03-03T19:41:50.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.657+0000 7f201fedf700 1 -- start start 2023-03-03T19:41:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.658+0000 7f201fedf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 0x7f20180c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.658+0000 7f201fedf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2018155950 0x7f2018155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.658+0000 7f201fedf700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20180713c0 con 0x7f20180aac00 2023-03-03T19:41:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.658+0000 7f201fedf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2018156310 con 0x7f2018155950 2023-03-03T19:41:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.658+0000 7f201fedf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2018152c70 con 0x7f2018071c30 2023-03-03T19:41:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.658+0000 7f201e47c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2018155950 0x7f2018155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.658+0000 7f201dc7b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f20180aac00 0x7f20180beb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36600/0 (socket says 172.21.15.7:36600) 2023-03-03T19:41:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.658+0000 7f201dc7b700 1 -- 172.21.15.7:0/237093903 learned_addr learned my addr 172.21.15.7:0/237093903 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:41:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201d47a700 1 --2- 172.21.15.7:0/237093903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 0x7f20180c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201e47c700 1 -- 172.21.15.7:0/237093903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 msgr2=0x7f20180c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:41:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201e47c700 1 --2- 172.21.15.7:0/237093903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 0x7f20180c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201e47c700 1 -- 172.21.15.7:0/237093903 >> v1:172.21.15.7:6789/0 conn(0x7f20180aac00 legacy=0x7f20180beb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201e47c700 1 -- 172.21.15.7:0/237093903 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20180c4930 con 0x7f2018155950 2023-03-03T19:41:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201cc79700 1 -- 172.21.15.7:0/237093903 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65753127 0 0) 0x7f20180713c0 con 0x7f20180aac00 2023-03-03T19:41:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201d47a700 1 --2- 172.21.15.7:0/237093903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 0x7f20180c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:41:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201e47c700 1 --2- 172.21.15.7:0/237093903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2018155950 0x7f2018155dd0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7f2010009d90 tx=0x7f201000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a19e0a2a96f6914 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:41:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201cc79700 1 -- 172.21.15.7:0/237093903 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201000e070 con 0x7f2018155950 2023-03-03T19:41:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201cc79700 1 -- 172.21.15.7:0/237093903 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201000bbd0 con 0x7f2018155950 2023-03-03T19:41:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.659+0000 7f201cc79700 1 -- 172.21.15.7:0/237093903 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2010012430 con 0x7f2018155950 2023-03-03T19:41:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.660+0000 7f201fedf700 1 -- 172.21.15.7:0/237093903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2018155950 msgr2=0x7f2018155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.660+0000 7f201fedf700 1 --2- 172.21.15.7:0/237093903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2018155950 0x7f2018155dd0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7f2010009d90 tx=0x7f201000b330 comp rx=0 tx=0).stop 2023-03-03T19:41:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.660+0000 7f201fedf700 1 -- 172.21.15.7:0/237093903 shutdown_connections 2023-03-03T19:41:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.660+0000 7f201fedf700 1 --2- 172.21.15.7:0/237093903 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2018155950 0x7f2018155dd0 unknown :-1 s=CLOSED pgs=3559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.660+0000 7f201fedf700 1 --2- 172.21.15.7:0/237093903 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 0x7f20180c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.660+0000 7f201fedf700 1 -- 172.21.15.7:0/237093903 >> 172.21.15.7:0/237093903 conn(0x7f20180736f0 msgr2=0x7f2018073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:41:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.660+0000 7f201fedf700 1 -- 172.21.15.7:0/237093903 shutdown_connections 2023-03-03T19:41:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.660+0000 7f201fedf700 1 -- 172.21.15.7:0/237093903 wait complete. 2023-03-03T19:41:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.661+0000 7f201fedf700 1 Processor -- start 2023-03-03T19:41:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.661+0000 7f201fedf700 1 -- start start 2023-03-03T19:41:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201fedf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 0x7f20180c7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201fedf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20180aac00 0x7f2018150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201fedf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2018155950 0x7f20180c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201fedf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20180c7c80 con 0x7f20180aac00 2023-03-03T19:41:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201fedf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20180c67e0 con 0x7f2018155950 2023-03-03T19:41:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201fedf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2018151160 con 0x7f2018071c30 2023-03-03T19:41:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201d47a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20180aac00 0x7f2018150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201d47a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20180aac00 0x7f2018150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42628/0 (socket says 172.21.15.7:42628) 2023-03-03T19:41:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201d47a700 1 -- 172.21.15.7:0/2552856291 learned_addr learned my addr 172.21.15.7:0/2552856291 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:41:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201dc7b700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 0x7f20180c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201e47c700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2018155950 0x7f20180c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201d47a700 1 -- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 msgr2=0x7f20180c7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201d47a700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 0x7f20180c7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201d47a700 1 -- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2018155950 msgr2=0x7f20180c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201d47a700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2018155950 0x7f20180c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201d47a700 1 -- 172.21.15.7:0/2552856291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2010009a70 con 0x7f20180aac00 2023-03-03T19:41:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201d47a700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20180aac00 0x7f2018150440 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7f201400e840 tx=0x7f201400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:41:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f200e7fc700 1 -- 172.21.15.7:0/2552856291 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014013610 con 0x7f20180aac00 2023-03-03T19:41:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f200e7fc700 1 -- 172.21.15.7:0/2552856291 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2014013770 con 0x7f20180aac00 2023-03-03T19:41:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.662+0000 7f201fedf700 1 -- 172.21.15.7:0/2552856291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20180c4b80 con 0x7f20180aac00 2023-03-03T19:41:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.663+0000 7f200e7fc700 1 -- 172.21.15.7:0/2552856291 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014013b20 con 0x7f20180aac00 2023-03-03T19:41:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.663+0000 7f201fedf700 1 -- 172.21.15.7:0/2552856291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2018152360 con 0x7f20180aac00 2023-03-03T19:41:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.664+0000 7f201fedf700 1 -- 172.21.15.7:0/2552856291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2018152620 con 0x7f20180aac00 2023-03-03T19:41:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.664+0000 7f200e7fc700 1 -- 172.21.15.7:0/2552856291 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2014023070 con 0x7f20180aac00 2023-03-03T19:41:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.665+0000 7f200e7fc700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ffc0682f0 0x7f1ffc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:41:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.665+0000 7f200e7fc700 1 -- 172.21.15.7:0/2552856291 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2014085de0 con 0x7f20180aac00 2023-03-03T19:41:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.665+0000 7f201dc7b700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ffc0682f0 0x7f1ffc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:41:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.665+0000 7f201dc7b700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ffc0682f0 0x7f1ffc06a7a0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f2008000f60 tx=0x7f2008009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:41:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.669+0000 7f200e7fc700 1 -- 172.21.15.7:0/2552856291 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2014052180 con 0x7f20180aac00 2023-03-03T19:41:50.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.910+0000 7f201fedf700 1 -- 172.21.15.7:0/2552856291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20180aafd0 con 0x7f20180aac00 2023-03-03T19:41:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.910+0000 7f200e7fc700 1 -- 172.21.15.7:0/2552856291 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f20140557a0 con 0x7f20180aac00 2023-03-03T19:41:50.911 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:41:50.911 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:41:50.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 -- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ffc0682f0 msgr2=0x7f1ffc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:50.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ffc0682f0 0x7f1ffc06a7a0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f2008000f60 tx=0x7f2008009040 comp rx=0 tx=0).stop 2023-03-03T19:41:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 -- 172.21.15.7:0/2552856291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20180aac00 msgr2=0x7f2018150440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:41:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20180aac00 0x7f2018150440 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7f201400e840 tx=0x7f201400ec00 comp rx=0 tx=0).stop 2023-03-03T19:41:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 -- 172.21.15.7:0/2552856291 shutdown_connections 2023-03-03T19:41:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2018155950 0x7f20180c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2018071c30 0x7f20180c7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ffc0682f0 0x7f1ffc06a7a0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 --2- 172.21.15.7:0/2552856291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f20180aac00 0x7f2018150440 unknown :-1 s=CLOSED pgs=3560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:41:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 -- 172.21.15.7:0/2552856291 >> 172.21.15.7:0/2552856291 conn(0x7f20180736f0 msgr2=0x7f20181dc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:41:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 -- 172.21.15.7:0/2552856291 shutdown_connections 2023-03-03T19:41:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:41:50.913+0000 7f201fedf700 1 -- 172.21.15.7:0/2552856291 wait complete. 2023-03-03T19:41:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:41:50.925 INFO:tasks.mds_thrash.fs.[b]:waiting for 72 secs before thrashing 2023-03-03T19:41:51.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:41:51.021 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:42:07.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:42:07.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.725+0000 7ff5ca6b9700 1 Processor -- start 2023-03-03T19:42:07.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.725+0000 7ff5ca6b9700 1 -- start start 2023-03-03T19:42:07.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.725+0000 7ff5ca6b9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5c40aad20 0x7ff5c4157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:07.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.725+0000 7ff5ca6b9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c41588f0 con 0x7ff5c414b020 2023-03-03T19:42:07.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.725+0000 7ff5ca6b9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c40728b0 con 0x7ff5c40aa030 2023-03-03T19:42:07.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.725+0000 7ff5ca6b9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c4155070 con 0x7ff5c40aad20 2023-03-03T19:42:07.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c3fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff5c414b020 0x7ff5c40717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36628/0 (socket says 172.21.15.7:36628) 2023-03-03T19:42:07.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c37fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5c40aad20 0x7ff5c4157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:07.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c3fff700 1 -- 172.21.15.7:0/19736760 learned_addr learned my addr 172.21.15.7:0/19736760 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:42:07.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c37fe700 1 -- 172.21.15.7:0/19736760 >> v1:172.21.15.143:6789/0 conn(0x7ff5c40aa030 legacy=0x7ff5c41581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:42:07.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c37fe700 1 -- 172.21.15.7:0/19736760 >> v1:172.21.15.7:6789/0 conn(0x7ff5c414b020 legacy=0x7ff5c40717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:07.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/19736760 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235747392 0 0) 0x7ff5c41588f0 con 0x7ff5c414b020 2023-03-03T19:42:07.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c37fe700 1 -- 172.21.15.7:0/19736760 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c406f5b0 con 0x7ff5c40aad20 2023-03-03T19:42:07.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c37fe700 1 --2- 172.21.15.7:0/19736760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c40aad20 0x7ff5c4157ca0 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009840 tx=0x7ff5b8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47437c65a22b3589 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:42:07.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/19736760 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b8004230 con 0x7ff5c40aad20 2023-03-03T19:42:07.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/19736760 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b8004390 con 0x7ff5c40aad20 2023-03-03T19:42:07.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.726+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/19736760 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b80036a0 con 0x7ff5c40aad20 2023-03-03T19:42:07.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.727+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/19736760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c40aad20 msgr2=0x7ff5c4157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:07.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.727+0000 7ff5ca6b9700 1 --2- 172.21.15.7:0/19736760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c40aad20 0x7ff5c4157ca0 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7ff5b8009840 tx=0x7ff5b8009b50 comp rx=0 tx=0).stop 2023-03-03T19:42:07.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.727+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/19736760 shutdown_connections 2023-03-03T19:42:07.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.727+0000 7ff5ca6b9700 1 --2- 172.21.15.7:0/19736760 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c40aad20 0x7ff5c4157ca0 unknown :-1 s=CLOSED pgs=3562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:07.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.727+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/19736760 >> 172.21.15.7:0/19736760 conn(0x7ff5c40c8900 msgr2=0x7ff5c40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:42:07.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.727+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/19736760 shutdown_connections 2023-03-03T19:42:07.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.727+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/19736760 wait complete. 2023-03-03T19:42:07.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.728+0000 7ff5ca6b9700 1 Processor -- start 2023-03-03T19:42:07.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.728+0000 7ff5ca6b9700 1 -- start start 2023-03-03T19:42:07.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5ca6b9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c40aa030 0x7ff5c41452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:07.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5ca6b9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c40aad20 0x7ff5c41dfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:07.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5ca6b9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c414b020 0x7ff5c41e02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:07.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5ca6b9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c41525f0 con 0x7ff5c414b020 2023-03-03T19:42:07.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5ca6b9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4145fd0 con 0x7ff5c40aa030 2023-03-03T19:42:07.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5ca6b9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c41435f0 con 0x7ff5c40aad20 2023-03-03T19:42:07.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5c8c56700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c414b020 0x7ff5c41e02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:07.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5c8c56700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c414b020 0x7ff5c41e02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42658/0 (socket says 172.21.15.7:42658) 2023-03-03T19:42:07.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5c8c56700 1 -- 172.21.15.7:0/259543665 learned_addr learned my addr 172.21.15.7:0/259543665 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:42:07.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5c3fff700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c40aa030 0x7ff5c41452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:07.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.729+0000 7ff5c37fe700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c40aad20 0x7ff5c41dfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:07.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c8c56700 1 -- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c40aad20 msgr2=0x7ff5c41dfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:07.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c8c56700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c40aad20 0x7ff5c41dfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:07.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c8c56700 1 -- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c40aa030 msgr2=0x7ff5c41452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:07.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c8c56700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c40aa030 0x7ff5c41452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:07.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c8c56700 1 -- 172.21.15.7:0/259543665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b80094a0 con 0x7ff5c414b020 2023-03-03T19:42:07.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c3fff700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c40aa030 0x7ff5c41452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:42:07.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c37fe700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c40aad20 0x7ff5c41dfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:42:07.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c8c56700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c414b020 0x7ff5c41e02f0 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00b970 tx=0x7ff5bc00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:42:07.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c0ff9700 1 -- 172.21.15.7:0/259543665 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc00c9b0 con 0x7ff5c414b020 2023-03-03T19:42:07.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c0ff9700 1 -- 172.21.15.7:0/259543665 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc00cb10 con 0x7ff5c414b020 2023-03-03T19:42:07.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/259543665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c406fb70 con 0x7ff5c414b020 2023-03-03T19:42:07.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5c0ff9700 1 -- 172.21.15.7:0/259543665 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc014440 con 0x7ff5c414b020 2023-03-03T19:42:07.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.730+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/259543665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c406f890 con 0x7ff5c414b020 2023-03-03T19:42:07.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.731+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/259543665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c4155350 con 0x7ff5c414b020 2023-03-03T19:42:07.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.732+0000 7ff5c0ff9700 1 -- 172.21.15.7:0/259543665 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff5bc01b030 con 0x7ff5c414b020 2023-03-03T19:42:07.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.732+0000 7ff5c0ff9700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5b00682f0 0x7ff5b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:07.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.732+0000 7ff5c0ff9700 1 -- 172.21.15.7:0/259543665 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff5bc085c40 con 0x7ff5c414b020 2023-03-03T19:42:07.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.732+0000 7ff5c3fff700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5b00682f0 0x7ff5b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:07.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.733+0000 7ff5c3fff700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5b00682f0 0x7ff5b006a7a0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4000f60 tx=0x7ff5b4007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:42:07.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.736+0000 7ff5c0ff9700 1 -- 172.21.15.7:0/259543665 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff5bc051fe0 con 0x7ff5c414b020 2023-03-03T19:42:07.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.977+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/259543665 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5c4074bb0 con 0x7ff5c414b020 2023-03-03T19:42:07.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.978+0000 7ff5c0ff9700 1 -- 172.21.15.7:0/259543665 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff5bc055600 con 0x7ff5c414b020 2023-03-03T19:42:07.979 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:42:07.979 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:42:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.979+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5b00682f0 msgr2=0x7ff5b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.979+0000 7ff5ca6b9700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5b00682f0 0x7ff5b006a7a0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4000f60 tx=0x7ff5b4007d30 comp rx=0 tx=0).stop 2023-03-03T19:42:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.979+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/259543665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c414b020 msgr2=0x7ff5c41e02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.979+0000 7ff5ca6b9700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c414b020 0x7ff5c41e02f0 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00b970 tx=0x7ff5bc00bd30 comp rx=0 tx=0).stop 2023-03-03T19:42:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.979+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/259543665 shutdown_connections 2023-03-03T19:42:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.979+0000 7ff5ca6b9700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff5b00682f0 0x7ff5b006a7a0 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.979+0000 7ff5ca6b9700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5c414b020 0x7ff5c41e02f0 unknown :-1 s=CLOSED pgs=3563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.979+0000 7ff5ca6b9700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5c40aa030 0x7ff5c41452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.979+0000 7ff5ca6b9700 1 --2- 172.21.15.7:0/259543665 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff5c40aad20 0x7ff5c41dfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.979+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/259543665 >> 172.21.15.7:0/259543665 conn(0x7ff5c40c8900 msgr2=0x7ff5c4148820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:42:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.980+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/259543665 shutdown_connections 2023-03-03T19:42:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:07.980+0000 7ff5ca6b9700 1 -- 172.21.15.7:0/259543665 wait complete. 2023-03-03T19:42:07.982 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:42:07.992 INFO:tasks.mds_thrash.fs.[a]:waiting for 10 secs before thrashing 2023-03-03T19:42:17.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:42:18.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.131+0000 7f3acdd43700 1 Processor -- start 2023-03-03T19:42:18.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.131+0000 7f3acdd43700 1 -- start start 2023-03-03T19:42:18.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3acdd43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac80aad20 0x7f3ac8157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:18.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3acdd43700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3ac80aa030 0x7f3ac81581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:18.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3acdd43700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac80728b0 con 0x7f3ac814b020 2023-03-03T19:42:18.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3acdd43700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac8155070 con 0x7f3ac80aa030 2023-03-03T19:42:18.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3acdd43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac806f5b0 con 0x7f3ac80aad20 2023-03-03T19:42:18.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3ac80aa030 0x7f3ac81581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:18.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3ac80aa030 0x7f3ac81581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42680/0 (socket says 172.21.15.7:42680) 2023-03-03T19:42:18.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac7fff700 1 -- 172.21.15.7:0/2631470553 learned_addr learned my addr 172.21.15.7:0/2631470553 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:42:18.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac6ffd700 1 --2- 172.21.15.7:0/2631470553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac80aad20 0x7f3ac8157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:18.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac7fff700 1 -- 172.21.15.7:0/2631470553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac80aad20 msgr2=0x7f3ac8157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:42:18.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac7fff700 1 --2- 172.21.15.7:0/2631470553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac80aad20 0x7f3ac8157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:18.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac7fff700 1 -- 172.21.15.7:0/2631470553 >> v1:172.21.15.143:6789/0 conn(0x7f3ac814b020 legacy=0x7f3ac80717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:18.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac7fff700 1 -- 172.21.15.7:0/2631470553 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac806f890 con 0x7f3ac80aa030 2023-03-03T19:42:18.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac7fff700 1 --2- 172.21.15.7:0/2631470553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aa030 0x7f3ac81581e0 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009e50 tx=0x7f3ac000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d253d98db75424 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:42:18.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac67fc700 1 -- 172.21.15.7:0/2631470553 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac000e070 con 0x7f3ac80aa030 2023-03-03T19:42:18.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac67fc700 1 -- 172.21.15.7:0/2631470553 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac000b910 con 0x7f3ac80aa030 2023-03-03T19:42:18.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.132+0000 7f3ac67fc700 1 -- 172.21.15.7:0/2631470553 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac000bcc0 con 0x7f3ac80aa030 2023-03-03T19:42:18.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.133+0000 7f3acdd43700 1 -- 172.21.15.7:0/2631470553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aa030 msgr2=0x7f3ac81581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:18.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.133+0000 7f3acdd43700 1 --2- 172.21.15.7:0/2631470553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aa030 0x7f3ac81581e0 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009e50 tx=0x7f3ac000b330 comp rx=0 tx=0).stop 2023-03-03T19:42:18.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.133+0000 7f3acdd43700 1 -- 172.21.15.7:0/2631470553 shutdown_connections 2023-03-03T19:42:18.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.133+0000 7f3acdd43700 1 --2- 172.21.15.7:0/2631470553 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aa030 0x7f3ac81581e0 unknown :-1 s=CLOSED pgs=3564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:18.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.133+0000 7f3acdd43700 1 --2- 172.21.15.7:0/2631470553 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac80aad20 0x7f3ac8157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:18.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.133+0000 7f3acdd43700 1 -- 172.21.15.7:0/2631470553 >> 172.21.15.7:0/2631470553 conn(0x7f3ac80c8900 msgr2=0x7f3ac80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:42:18.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.133+0000 7f3acdd43700 1 -- 172.21.15.7:0/2631470553 shutdown_connections 2023-03-03T19:42:18.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.133+0000 7f3acdd43700 1 -- 172.21.15.7:0/2631470553 wait complete. 2023-03-03T19:42:18.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.134+0000 7f3acdd43700 1 Processor -- start 2023-03-03T19:42:18.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.134+0000 7f3acdd43700 1 -- start start 2023-03-03T19:42:18.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3acdd43700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac80aa030 0x7f3ac81eb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:18.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3acdd43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aad20 0x7f3ac81ed640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:18.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3acdd43700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac814b020 0x7f3ac8145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:18.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3acdd43700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac81e8890 con 0x7f3ac80aad20 2023-03-03T19:42:18.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3acdd43700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac8147d60 con 0x7f3ac80aa030 2023-03-03T19:42:18.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3acdd43700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac81ee150 con 0x7f3ac814b020 2023-03-03T19:42:18.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3ac6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aad20 0x7f3ac81ed640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:18.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3ac6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aad20 0x7f3ac81ed640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42682/0 (socket says 172.21.15.7:42682) 2023-03-03T19:42:18.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3ac6ffd700 1 -- 172.21.15.7:0/3676980408 learned_addr learned my addr 172.21.15.7:0/3676980408 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:42:18.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3ac7fff700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac814b020 0x7f3ac8145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:18.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.135+0000 7f3ac77fe700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac80aa030 0x7f3ac81eb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:18.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3ac6ffd700 1 -- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac814b020 msgr2=0x7f3ac8145560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:18.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3ac6ffd700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac814b020 0x7f3ac8145560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:18.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3ac6ffd700 1 -- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac80aa030 msgr2=0x7f3ac81eb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:18.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3ac6ffd700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac80aa030 0x7f3ac81eb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:18.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3ac6ffd700 1 -- 172.21.15.7:0/3676980408 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac0009b30 con 0x7f3ac80aad20 2023-03-03T19:42:18.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3ac6ffd700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aad20 0x7f3ac81ed640 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00e840 tx=0x7f3abc00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:42:18.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3aaffff700 1 -- 172.21.15.7:0/3676980408 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc009590 con 0x7f3ac80aad20 2023-03-03T19:42:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3aaffff700 1 -- 172.21.15.7:0/3676980408 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3abc013680 con 0x7f3ac80aad20 2023-03-03T19:42:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3acdd43700 1 -- 172.21.15.7:0/3676980408 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac8155350 con 0x7f3ac80aad20 2023-03-03T19:42:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3aaffff700 1 -- 172.21.15.7:0/3676980408 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc013a10 con 0x7f3ac80aad20 2023-03-03T19:42:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.136+0000 7f3acdd43700 1 -- 172.21.15.7:0/3676980408 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac806fb70 con 0x7f3ac80aad20 2023-03-03T19:42:18.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.137+0000 7f3acdd43700 1 -- 172.21.15.7:0/3676980408 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac8071ef0 con 0x7f3ac80aad20 2023-03-03T19:42:18.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.137+0000 7f3aaffff700 1 -- 172.21.15.7:0/3676980408 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3abc019030 con 0x7f3ac80aad20 2023-03-03T19:42:18.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.138+0000 7f3aaffff700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ab40682f0 0x7f3ab406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:18.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.138+0000 7f3aaffff700 1 -- 172.21.15.7:0/3676980408 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3abc085e70 con 0x7f3ac80aad20 2023-03-03T19:42:18.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.138+0000 7f3ac77fe700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ab40682f0 0x7f3ab406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:18.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.139+0000 7f3ac77fe700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ab40682f0 0x7f3ab406a7a0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80c3620 tx=0x7f3ab8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:42:18.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.142+0000 7f3aaffff700 1 -- 172.21.15.7:0/3676980408 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3abc0521b0 con 0x7f3ac80aad20 2023-03-03T19:42:18.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.381+0000 7f3acdd43700 1 -- 172.21.15.7:0/3676980408 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ac80a9810 con 0x7f3ac80aad20 2023-03-03T19:42:18.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.381+0000 7f3aaffff700 1 -- 172.21.15.7:0/3676980408 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3abc0557d0 con 0x7f3ac80aad20 2023-03-03T19:42:18.383 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:42:18.383 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:42:18.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 -- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ab40682f0 msgr2=0x7f3ab406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:18.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ab40682f0 0x7f3ab406a7a0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80c3620 tx=0x7f3ab8009040 comp rx=0 tx=0).stop 2023-03-03T19:42:18.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 -- 172.21.15.7:0/3676980408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aad20 msgr2=0x7f3ac81ed640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:18.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aad20 0x7f3ac81ed640 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00e840 tx=0x7f3abc00eb50 comp rx=0 tx=0).stop 2023-03-03T19:42:18.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 -- 172.21.15.7:0/3676980408 shutdown_connections 2023-03-03T19:42:18.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ac814b020 0x7f3ac8145560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac80aa030 0x7f3ac81eb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3ab40682f0 0x7f3ab406a7a0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 --2- 172.21.15.7:0/3676980408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac80aad20 0x7f3ac81ed640 unknown :-1 s=CLOSED pgs=3565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 -- 172.21.15.7:0/3676980408 >> 172.21.15.7:0/3676980408 conn(0x7f3ac80c8900 msgr2=0x7f3ac80c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:42:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 -- 172.21.15.7:0/3676980408 shutdown_connections 2023-03-03T19:42:18.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:18.383+0000 7f3acdd43700 1 -- 172.21.15.7:0/3676980408 wait complete. 2023-03-03T19:42:18.387 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:42:18.398 INFO:tasks.mds_thrash.fs.[a]:waiting for 11 secs before thrashing 2023-03-03T19:42:21.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:42:21.088 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:42:29.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:42:29.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.538+0000 7f98cc87f700 1 Processor -- start 2023-03-03T19:42:29.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.538+0000 7f98cc87f700 1 -- start start 2023-03-03T19:42:29.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.539+0000 7f98cc87f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f98c80c20d0 0x7f98c81562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.539+0000 7f98cc87f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c27c0 0x7f98c80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.539+0000 7f98cc87f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c8155e60 con 0x7f98c814f860 2023-03-03T19:42:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.539+0000 7f98cc87f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c806d350 con 0x7f98c80c20d0 2023-03-03T19:42:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.539+0000 7f98cc87f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c81558e0 con 0x7f98c80c27c0 2023-03-03T19:42:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.539+0000 7f98c659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f98c814f860 0x7f98c8152d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36676/0 (socket says 172.21.15.7:36676) 2023-03-03T19:42:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.539+0000 7f98c659c700 1 -- 172.21.15.7:0/503189315 learned_addr learned my addr 172.21.15.7:0/503189315 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:42:29.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.539+0000 7f98c6d9d700 1 --2- 172.21.15.7:0/503189315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c27c0 0x7f98c80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:29.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.539+0000 7f98c5d9b700 1 --2- 172.21.15.7:0/503189315 >> v2:172.21.15.143:3300/0 conn(0x7f98c80c20d0 0x7f98c81562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:29.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c559a700 1 -- 172.21.15.7:0/503189315 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120365988 0 0) 0x7f98c8155e60 con 0x7f98c814f860 2023-03-03T19:42:29.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c559a700 1 -- 172.21.15.7:0/503189315 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b00035b0 con 0x7f98c814f860 2023-03-03T19:42:29.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c5d9b700 1 -- 172.21.15.7:0/503189315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c27c0 msgr2=0x7f98c80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:42:29.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c5d9b700 1 --2- 172.21.15.7:0/503189315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c27c0 0x7f98c80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:29.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c5d9b700 1 -- 172.21.15.7:0/503189315 >> v1:172.21.15.7:6789/0 conn(0x7f98c814f860 legacy=0x7f98c8152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:29.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c5d9b700 1 -- 172.21.15.7:0/503189315 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c806cd90 con 0x7f98c80c20d0 2023-03-03T19:42:29.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c6d9d700 1 --2- 172.21.15.7:0/503189315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c27c0 0x7f98c80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:42:29.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c5d9b700 1 --2- 172.21.15.7:0/503189315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c80c20d0 0x7f98c81562f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009840 tx=0x7f98b8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5135a5bbed52763e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:42:29.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c559a700 1 -- 172.21.15.7:0/503189315 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8011830 con 0x7f98c80c20d0 2023-03-03T19:42:29.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c559a700 1 -- 172.21.15.7:0/503189315 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b8011990 con 0x7f98c80c20d0 2023-03-03T19:42:29.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.540+0000 7f98c559a700 1 -- 172.21.15.7:0/503189315 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8011d20 con 0x7f98c80c20d0 2023-03-03T19:42:29.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.541+0000 7f98cc87f700 1 -- 172.21.15.7:0/503189315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c80c20d0 msgr2=0x7f98c81562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:29.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.541+0000 7f98cc87f700 1 --2- 172.21.15.7:0/503189315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c80c20d0 0x7f98c81562f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009840 tx=0x7f98b8009b50 comp rx=0 tx=0).stop 2023-03-03T19:42:29.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.541+0000 7f98cc87f700 1 -- 172.21.15.7:0/503189315 shutdown_connections 2023-03-03T19:42:29.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.541+0000 7f98cc87f700 1 --2- 172.21.15.7:0/503189315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c80c27c0 0x7f98c80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:29.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.541+0000 7f98cc87f700 1 --2- 172.21.15.7:0/503189315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c80c20d0 0x7f98c81562f0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:29.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.541+0000 7f98cc87f700 1 -- 172.21.15.7:0/503189315 >> 172.21.15.7:0/503189315 conn(0x7f98c80a94d0 msgr2=0x7f98c80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:42:29.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.541+0000 7f98cc87f700 1 -- 172.21.15.7:0/503189315 shutdown_connections 2023-03-03T19:42:29.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.541+0000 7f98cc87f700 1 -- 172.21.15.7:0/503189315 wait complete. 2023-03-03T19:42:29.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.542+0000 7f98cc87f700 1 Processor -- start 2023-03-03T19:42:29.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.542+0000 7f98cc87f700 1 -- start start 2023-03-03T19:42:29.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98cc87f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c80c20d0 0x7f98c814d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:29.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98cc87f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c27c0 0x7f98c814dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:29.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98cc87f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c814f860 0x7f98c81ed960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:29.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98cc87f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c80718b0 con 0x7f98c80c27c0 2023-03-03T19:42:29.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98cc87f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c8150950 con 0x7f98c80c20d0 2023-03-03T19:42:29.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98cc87f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c80715d0 con 0x7f98c814f860 2023-03-03T19:42:29.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98c5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c27c0 0x7f98c814dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:29.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98c5d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c27c0 0x7f98c814dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42702/0 (socket says 172.21.15.7:42702) 2023-03-03T19:42:29.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98c5d9b700 1 -- 172.21.15.7:0/4095677720 learned_addr learned my addr 172.21.15.7:0/4095677720 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:42:29.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98c659c700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c80c20d0 0x7f98c814d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:29.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98c5d9b700 1 -- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c814f860 msgr2=0x7f98c81ed960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:29.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98c5d9b700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c814f860 0x7f98c81ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:29.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98c5d9b700 1 -- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c80c20d0 msgr2=0x7f98c814d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:29.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98c5d9b700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c80c20d0 0x7f98c814d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:29.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.543+0000 7f98c5d9b700 1 -- 172.21.15.7:0/4095677720 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c0009a70 con 0x7f98c80c27c0 2023-03-03T19:42:29.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.544+0000 7f98c5d9b700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c27c0 0x7f98c814dca0 secure :-1 s=READY pgs=3567 cs=0 l=1 rev1=1 crypto rx=0x7f98b8000c00 tx=0x7f98b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:42:29.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.544+0000 7f98b6ffd700 1 -- 172.21.15.7:0/4095677720 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b80036a0 con 0x7f98c80c27c0 2023-03-03T19:42:29.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.544+0000 7f98b6ffd700 1 -- 172.21.15.7:0/4095677720 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b801f070 con 0x7f98c80c27c0 2023-03-03T19:42:29.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.544+0000 7f98cc87f700 1 -- 172.21.15.7:0/4095677720 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b80094a0 con 0x7f98c80c27c0 2023-03-03T19:42:29.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.544+0000 7f98b6ffd700 1 -- 172.21.15.7:0/4095677720 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8003a60 con 0x7f98c80c27c0 2023-03-03T19:42:29.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.544+0000 7f98cc87f700 1 -- 172.21.15.7:0/4095677720 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c8155af0 con 0x7f98c80c27c0 2023-03-03T19:42:29.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.545+0000 7f98cc87f700 1 -- 172.21.15.7:0/4095677720 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c806d070 con 0x7f98c80c27c0 2023-03-03T19:42:29.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.545+0000 7f98b6ffd700 1 -- 172.21.15.7:0/4095677720 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f98b800f050 con 0x7f98c80c27c0 2023-03-03T19:42:29.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.546+0000 7f98b6ffd700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98a80682f0 0x7f98a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:42:29.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.546+0000 7f98b6ffd700 1 -- 172.21.15.7:0/4095677720 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f98b8021070 con 0x7f98c80c27c0 2023-03-03T19:42:29.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.546+0000 7f98c659c700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98a80682f0 0x7f98a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:42:29.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.546+0000 7f98c659c700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98a80682f0 0x7f98a806a7a0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0011e0 tx=0x7f98bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:42:29.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.550+0000 7f98b6ffd700 1 -- 172.21.15.7:0/4095677720 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f98b8053d30 con 0x7f98c80c27c0 2023-03-03T19:42:29.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.789+0000 7f98cc87f700 1 -- 172.21.15.7:0/4095677720 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98c814fc30 con 0x7f98c80c27c0 2023-03-03T19:42:29.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.790+0000 7f98b6ffd700 1 -- 172.21.15.7:0/4095677720 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f98b8057350 con 0x7f98c80c27c0 2023-03-03T19:42:29.790 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:42:29.791 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:42:29.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 -- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98a80682f0 msgr2=0x7f98a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:29.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98a80682f0 0x7f98a806a7a0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0011e0 tx=0x7f98bc009040 comp rx=0 tx=0).stop 2023-03-03T19:42:29.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 -- 172.21.15.7:0/4095677720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c27c0 msgr2=0x7f98c814dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:42:29.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c27c0 0x7f98c814dca0 secure :-1 s=READY pgs=3567 cs=0 l=1 rev1=1 crypto rx=0x7f98b8000c00 tx=0x7f98b8004750 comp rx=0 tx=0).stop 2023-03-03T19:42:29.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 -- 172.21.15.7:0/4095677720 shutdown_connections 2023-03-03T19:42:29.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f98c814f860 0x7f98c81ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98c80c20d0 0x7f98c814d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f98a80682f0 0x7f98a806a7a0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 --2- 172.21.15.7:0/4095677720 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98c80c27c0 0x7f98c814dca0 unknown :-1 s=CLOSED pgs=3567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:42:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 -- 172.21.15.7:0/4095677720 >> 172.21.15.7:0/4095677720 conn(0x7f98c80a94d0 msgr2=0x7f98c80c51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:42:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 -- 172.21.15.7:0/4095677720 shutdown_connections 2023-03-03T19:42:29.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:42:29.792+0000 7f98cc87f700 1 -- 172.21.15.7:0/4095677720 wait complete. 2023-03-03T19:42:29.795 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:42:29.804 INFO:tasks.mds_thrash.fs.[a]:waiting for 34 secs before thrashing 2023-03-03T19:42:51.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:42:51.143 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:43:02.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:43:03.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.064+0000 7efda6930700 1 Processor -- start 2023-03-03T19:43:03.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.064+0000 7efda6930700 1 -- start start 2023-03-03T19:43:03.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.065+0000 7efda6930700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efda014fda0 0x7efda0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.065+0000 7efda6930700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda00c2f30 con 0x7efda00abe20 2023-03-03T19:43:03.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.065+0000 7efda6930700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda00c31d0 con 0x7efda014f9d0 2023-03-03T19:43:03.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.065+0000 7efda6930700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda01524a0 con 0x7efda014fda0 2023-03-03T19:43:03.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.065+0000 7efd9f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7efda014fda0 0x7efda0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:03.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.065+0000 7efda4ecd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7efda00abe20 0x7efda01529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36704/0 (socket says 172.21.15.7:36704) 2023-03-03T19:43:03.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.065+0000 7efda4ecd700 1 -- 172.21.15.7:0/4267124537 learned_addr learned my addr 172.21.15.7:0/4267124537 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:43:03.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.066+0000 7efd9f7fe700 1 -- 172.21.15.7:0/4267124537 >> v1:172.21.15.143:6789/0 conn(0x7efda014f9d0 legacy=0x7efda0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:03.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.066+0000 7efd9f7fe700 1 -- 172.21.15.7:0/4267124537 >> v1:172.21.15.7:6789/0 conn(0x7efda00abe20 legacy=0x7efda01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:03.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.066+0000 7efd9effd700 1 -- 172.21.15.7:0/4267124537 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803894697 0 0) 0x7efda00c2f30 con 0x7efda00abe20 2023-03-03T19:43:03.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.066+0000 7efd9f7fe700 1 -- 172.21.15.7:0/4267124537 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda00c34b0 con 0x7efda014fda0 2023-03-03T19:43:03.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.066+0000 7efd9f7fe700 1 --2- 172.21.15.7:0/4267124537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014fda0 0x7efda0156430 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 crypto rx=0x7efd90009840 tx=0x7efd90009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518c89aa7d1530a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:03.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.066+0000 7efd9effd700 1 -- 172.21.15.7:0/4267124537 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90011830 con 0x7efda014fda0 2023-03-03T19:43:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.066+0000 7efd9effd700 1 -- 172.21.15.7:0/4267124537 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90011990 con 0x7efda014fda0 2023-03-03T19:43:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.066+0000 7efd9effd700 1 -- 172.21.15.7:0/4267124537 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90011d20 con 0x7efda014fda0 2023-03-03T19:43:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.066+0000 7efda6930700 1 -- 172.21.15.7:0/4267124537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014fda0 msgr2=0x7efda0156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:03.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.066+0000 7efda6930700 1 --2- 172.21.15.7:0/4267124537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014fda0 0x7efda0156430 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 crypto rx=0x7efd90009840 tx=0x7efd90009b50 comp rx=0 tx=0).stop 2023-03-03T19:43:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.067+0000 7efda6930700 1 -- 172.21.15.7:0/4267124537 shutdown_connections 2023-03-03T19:43:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.067+0000 7efda6930700 1 --2- 172.21.15.7:0/4267124537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014fda0 0x7efda0156430 unknown :-1 s=CLOSED pgs=3569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.067+0000 7efda6930700 1 -- 172.21.15.7:0/4267124537 >> 172.21.15.7:0/4267124537 conn(0x7efda00c8900 msgr2=0x7efda00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:43:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.067+0000 7efda6930700 1 -- 172.21.15.7:0/4267124537 shutdown_connections 2023-03-03T19:43:03.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.067+0000 7efda6930700 1 -- 172.21.15.7:0/4267124537 wait complete. 2023-03-03T19:43:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.068+0000 7efda6930700 1 Processor -- start 2023-03-03T19:43:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.068+0000 7efda6930700 1 -- start start 2023-03-03T19:43:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efda6930700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efda00abe20 0x7efda01dee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efda6930700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014f9d0 0x7efda00c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efda6930700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efda014fda0 0x7efda01ddb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efda6930700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda014b6f0 con 0x7efda014f9d0 2023-03-03T19:43:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efda6930700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda01ecfd0 con 0x7efda00abe20 2023-03-03T19:43:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efda6930700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda014ce70 con 0x7efda014fda0 2023-03-03T19:43:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014f9d0 0x7efda00c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014f9d0 0x7efda00c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42728/0 (socket says 172.21.15.7:42728) 2023-03-03T19:43:03.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9f7fe700 1 -- 172.21.15.7:0/2774843403 learned_addr learned my addr 172.21.15.7:0/2774843403 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:43:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9ffff700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efda00abe20 0x7efda01dee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efda4ecd700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efda014fda0 0x7efda01ddb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9f7fe700 1 -- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efda014fda0 msgr2=0x7efda01ddb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9f7fe700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efda014fda0 0x7efda01ddb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9f7fe700 1 -- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efda00abe20 msgr2=0x7efda01dee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9f7fe700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efda00abe20 0x7efda01dee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9f7fe700 1 -- 172.21.15.7:0/2774843403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd900094a0 con 0x7efda014f9d0 2023-03-03T19:43:03.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9ffff700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efda00abe20 0x7efda01dee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:43:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9f7fe700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014f9d0 0x7efda00c1a50 secure :-1 s=READY pgs=3570 cs=0 l=1 rev1=1 crypto rx=0x7efd90013000 tx=0x7efd90004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9cff9700 1 -- 172.21.15.7:0/2774843403 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9001a8c0 con 0x7efda014f9d0 2023-03-03T19:43:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.069+0000 7efd9cff9700 1 -- 172.21.15.7:0/2774843403 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9001aa20 con 0x7efda014f9d0 2023-03-03T19:43:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.070+0000 7efda6930700 1 -- 172.21.15.7:0/2774843403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda00ac350 con 0x7efda014f9d0 2023-03-03T19:43:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.070+0000 7efd9cff9700 1 -- 172.21.15.7:0/2774843403 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9001ad90 con 0x7efda014f9d0 2023-03-03T19:43:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.070+0000 7efda6930700 1 -- 172.21.15.7:0/2774843403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda00ab630 con 0x7efda014f9d0 2023-03-03T19:43:03.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.070+0000 7efda6930700 1 -- 172.21.15.7:0/2774843403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda0071fa0 con 0x7efda014f9d0 2023-03-03T19:43:03.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.071+0000 7efd9cff9700 1 -- 172.21.15.7:0/2774843403 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efd9002b030 con 0x7efda014f9d0 2023-03-03T19:43:03.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.071+0000 7efd9cff9700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd800682f0 0x7efd8006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.071+0000 7efd9cff9700 1 -- 172.21.15.7:0/2774843403 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efd900874c0 con 0x7efda014f9d0 2023-03-03T19:43:03.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.072+0000 7efd9ffff700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd800682f0 0x7efd8006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:03.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.072+0000 7efd9ffff700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd800682f0 0x7efd8006a7a0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7efd88000f60 tx=0x7efd88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:03.083 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.076+0000 7efd9cff9700 1 -- 172.21.15.7:0/2774843403 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efd90053910 con 0x7efda014f9d0 2023-03-03T19:43:03.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.311+0000 7efda6930700 1 -- 172.21.15.7:0/2774843403 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efda0074c00 con 0x7efda014f9d0 2023-03-03T19:43:03.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.311+0000 7efd9cff9700 1 -- 172.21.15.7:0/2774843403 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efd90056f30 con 0x7efda014f9d0 2023-03-03T19:43:03.313 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:43:03.313 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:43:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 -- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd800682f0 msgr2=0x7efd8006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd800682f0 0x7efd8006a7a0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7efd88000f60 tx=0x7efd88009040 comp rx=0 tx=0).stop 2023-03-03T19:43:03.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 -- 172.21.15.7:0/2774843403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014f9d0 msgr2=0x7efda00c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014f9d0 0x7efda00c1a50 secure :-1 s=READY pgs=3570 cs=0 l=1 rev1=1 crypto rx=0x7efd90013000 tx=0x7efd90004750 comp rx=0 tx=0).stop 2023-03-03T19:43:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 -- 172.21.15.7:0/2774843403 shutdown_connections 2023-03-03T19:43:03.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efda014fda0 0x7efda01ddb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efda00abe20 0x7efda01dee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efd800682f0 0x7efd8006a7a0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 --2- 172.21.15.7:0/2774843403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efda014f9d0 0x7efda00c1a50 unknown :-1 s=CLOSED pgs=3570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 -- 172.21.15.7:0/2774843403 >> 172.21.15.7:0/2774843403 conn(0x7efda00c8900 msgr2=0x7efda0151aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:43:03.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 -- 172.21.15.7:0/2774843403 shutdown_connections 2023-03-03T19:43:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.314+0000 7efda6930700 1 -- 172.21.15.7:0/2774843403 wait complete. 2023-03-03T19:43:03.319 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:43:03.327 INFO:tasks.mds_thrash.fs.[b]:waiting for 23 secs before thrashing 2023-03-03T19:43:03.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:43:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.935+0000 7f6b97d38700 1 Processor -- start 2023-03-03T19:43:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.935+0000 7f6b97d38700 1 -- start start 2023-03-03T19:43:03.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b97d38700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b900c5ee0 0x7f6b900aa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b97d38700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6b90154e40 0x7f6b900c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b97d38700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b90155670 con 0x7f6b900a9af0 2023-03-03T19:43:03.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b97d38700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b900aae70 con 0x7f6b900c5ee0 2023-03-03T19:43:03.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b97d38700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b900c2b40 con 0x7f6b90154e40 2023-03-03T19:43:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b95ad4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b900c5ee0 0x7f6b900aa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b95ad4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6b900c5ee0 0x7f6b900aa5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42734/0 (socket says 172.21.15.7:42734) 2023-03-03T19:43:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b95ad4700 1 -- 172.21.15.7:0/1624140651 learned_addr learned my addr 172.21.15.7:0/1624140651 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:43:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b95ad4700 1 -- 172.21.15.7:0/1624140651 >> v2:172.21.15.143:3300/0 conn(0x7f6b90154e40 msgr2=0x7f6b900c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:43:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b95ad4700 1 --2- 172.21.15.7:0/1624140651 >> v2:172.21.15.143:3300/0 conn(0x7f6b90154e40 0x7f6b900c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b95ad4700 1 -- 172.21.15.7:0/1624140651 >> v1:172.21.15.143:6789/0 conn(0x7f6b900a9af0 legacy=0x7f6b90154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:43:03.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.936+0000 7f6b95ad4700 1 -- 172.21.15.7:0/1624140651 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b90155390 con 0x7f6b900c5ee0 2023-03-03T19:43:03.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b95ad4700 1 --2- 172.21.15.7:0/1624140651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 0x7f6b900aa5e0 secure :-1 s=READY pgs=3571 cs=0 l=1 rev1=1 crypto rx=0x7f6b900aab20 tx=0x7f6b8c00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2fbf4594b18e95 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:03.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b94ad2700 1 -- 172.21.15.7:0/1624140651 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c00dcf0 con 0x7f6b900c5ee0 2023-03-03T19:43:03.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b94ad2700 1 -- 172.21.15.7:0/1624140651 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c00de50 con 0x7f6b900c5ee0 2023-03-03T19:43:03.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b94ad2700 1 -- 172.21.15.7:0/1624140651 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c004250 con 0x7f6b900c5ee0 2023-03-03T19:43:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b97d38700 1 -- 172.21.15.7:0/1624140651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 msgr2=0x7f6b900aa5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b97d38700 1 --2- 172.21.15.7:0/1624140651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 0x7f6b900aa5e0 secure :-1 s=READY pgs=3571 cs=0 l=1 rev1=1 crypto rx=0x7f6b900aab20 tx=0x7f6b8c00be70 comp rx=0 tx=0).stop 2023-03-03T19:43:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b97d38700 1 -- 172.21.15.7:0/1624140651 shutdown_connections 2023-03-03T19:43:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b97d38700 1 --2- 172.21.15.7:0/1624140651 >> v2:172.21.15.143:3300/0 conn(0x7f6b90154e40 0x7f6b900c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b97d38700 1 --2- 172.21.15.7:0/1624140651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 0x7f6b900aa5e0 unknown :-1 s=CLOSED pgs=3571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b97d38700 1 -- 172.21.15.7:0/1624140651 >> 172.21.15.7:0/1624140651 conn(0x7f6b900a9430 msgr2=0x7f6b900c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:43:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b97d38700 1 -- 172.21.15.7:0/1624140651 shutdown_connections 2023-03-03T19:43:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.937+0000 7f6b97d38700 1 -- 172.21.15.7:0/1624140651 wait complete. 2023-03-03T19:43:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.939+0000 7f6b97d38700 1 Processor -- start 2023-03-03T19:43:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.939+0000 7f6b97d38700 1 -- start start 2023-03-03T19:43:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b97d38700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b900a9af0 0x7f6b90152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b97d38700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 0x7f6b90070b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b97d38700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b90154e40 0x7f6b9014de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b97d38700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9014c5f0 con 0x7f6b900c5ee0 2023-03-03T19:43:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b97d38700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90071830 con 0x7f6b900a9af0 2023-03-03T19:43:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b97d38700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9014d6d0 con 0x7f6b90154e40 2023-03-03T19:43:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b952d3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 0x7f6b90070b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b952d3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 0x7f6b90070b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42740/0 (socket says 172.21.15.7:42740) 2023-03-03T19:43:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b952d3700 1 -- 172.21.15.7:0/2814130227 learned_addr learned my addr 172.21.15.7:0/2814130227 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:43:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b95ad4700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b900a9af0 0x7f6b90152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b962d5700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b90154e40 0x7f6b9014de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b952d3700 1 -- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b90154e40 msgr2=0x7f6b9014de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b952d3700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b90154e40 0x7f6b9014de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b952d3700 1 -- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b900a9af0 msgr2=0x7f6b90152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b952d3700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b900a9af0 0x7f6b90152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b952d3700 1 -- 172.21.15.7:0/2814130227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c00b7f0 con 0x7f6b900c5ee0 2023-03-03T19:43:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b952d3700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 0x7f6b90070b60 secure :-1 s=READY pgs=3572 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000b470 tx=0x7f6b8000b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.940+0000 7f6b867fc700 1 -- 172.21.15.7:0/2814130227 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b80011830 con 0x7f6b900c5ee0 2023-03-03T19:43:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.941+0000 7f6b867fc700 1 -- 172.21.15.7:0/2814130227 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b800119b0 con 0x7f6b900c5ee0 2023-03-03T19:43:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.941+0000 7f6b97d38700 1 -- 172.21.15.7:0/2814130227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b900c1c00 con 0x7f6b900c5ee0 2023-03-03T19:43:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.941+0000 7f6b867fc700 1 -- 172.21.15.7:0/2814130227 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b80011d60 con 0x7f6b900c5ee0 2023-03-03T19:43:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.941+0000 7f6b97d38700 1 -- 172.21.15.7:0/2814130227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b900c1920 con 0x7f6b900c5ee0 2023-03-03T19:43:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.942+0000 7f6b97d38700 1 -- 172.21.15.7:0/2814130227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b900c62b0 con 0x7f6b900c5ee0 2023-03-03T19:43:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.942+0000 7f6b867fc700 1 -- 172.21.15.7:0/2814130227 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6b80021070 con 0x7f6b900c5ee0 2023-03-03T19:43:03.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.942+0000 7f6b867fc700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b740682f0 0x7f6b7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:03.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.943+0000 7f6b867fc700 1 -- 172.21.15.7:0/2814130227 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6b80086070 con 0x7f6b900c5ee0 2023-03-03T19:43:03.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.943+0000 7f6b95ad4700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b740682f0 0x7f6b7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:03.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.943+0000 7f6b95ad4700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b740682f0 0x7f6b7406a7a0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00b340 tx=0x7f6b8c01d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:03.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:03.947+0000 7f6b867fc700 1 -- 172.21.15.7:0/2814130227 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b80052320 con 0x7f6b900c5ee0 2023-03-03T19:43:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.185+0000 7f6b97d38700 1 -- 172.21.15.7:0/2814130227 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b90074bb0 con 0x7f6b900c5ee0 2023-03-03T19:43:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.185+0000 7f6b867fc700 1 -- 172.21.15.7:0/2814130227 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6b80015090 con 0x7f6b900c5ee0 2023-03-03T19:43:04.186 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:43:04.186 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:43:04.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.187+0000 7f6b97d38700 1 -- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b740682f0 msgr2=0x7f6b7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:04.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.187+0000 7f6b97d38700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b740682f0 0x7f6b7406a7a0 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c00b340 tx=0x7f6b8c01d040 comp rx=0 tx=0).stop 2023-03-03T19:43:04.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.187+0000 7f6b97d38700 1 -- 172.21.15.7:0/2814130227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 msgr2=0x7f6b90070b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.187+0000 7f6b97d38700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 0x7f6b90070b60 secure :-1 s=READY pgs=3572 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000b470 tx=0x7f6b8000b780 comp rx=0 tx=0).stop 2023-03-03T19:43:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.188+0000 7f6b97d38700 1 -- 172.21.15.7:0/2814130227 shutdown_connections 2023-03-03T19:43:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.188+0000 7f6b97d38700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b90154e40 0x7f6b9014de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.188+0000 7f6b97d38700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b900a9af0 0x7f6b90152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.188+0000 7f6b97d38700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b740682f0 0x7f6b7406a7a0 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.188+0000 7f6b97d38700 1 --2- 172.21.15.7:0/2814130227 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b900c5ee0 0x7f6b90070b60 unknown :-1 s=CLOSED pgs=3572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.188+0000 7f6b97d38700 1 -- 172.21.15.7:0/2814130227 >> 172.21.15.7:0/2814130227 conn(0x7f6b900a9430 msgr2=0x7f6b900c4980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:43:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.188+0000 7f6b97d38700 1 -- 172.21.15.7:0/2814130227 shutdown_connections 2023-03-03T19:43:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:04.188+0000 7f6b97d38700 1 -- 172.21.15.7:0/2814130227 wait complete. 2023-03-03T19:43:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:43:04.200 INFO:tasks.mds_thrash.fs.[a]:waiting for 56 secs before thrashing 2023-03-03T19:43:21.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:43:21.306 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:43:26.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:43:26.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.466+0000 7f23dc5df700 1 Processor -- start 2023-03-03T19:43:26.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.466+0000 7f23dc5df700 1 -- start start 2023-03-03T19:43:26.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.467+0000 7f23dc5df700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f23d4151de0 0x7f23d40c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:26.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.467+0000 7f23dc5df700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d40c1a40 0x7f23d4157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:26.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.467+0000 7f23dc5df700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23d40c5140 0x7f23d406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:26.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.467+0000 7f23dc5df700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d406e370 con 0x7f23d40c5140 2023-03-03T19:43:26.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.467+0000 7f23dc5df700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d406dad0 con 0x7f23d4151de0 2023-03-03T19:43:26.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.467+0000 7f23dc5df700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d406ddb0 con 0x7f23d40c1a40 2023-03-03T19:43:26.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.467+0000 7f23dab7c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23d40c5140 0x7f23d406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:26.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.467+0000 7f23dab7c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23d40c5140 0x7f23d406a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42772/0 (socket says 172.21.15.7:42772) 2023-03-03T19:43:26.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.467+0000 7f23dab7c700 1 -- 172.21.15.7:0/1843258024 learned_addr learned my addr 172.21.15.7:0/1843258024 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:43:26.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.468+0000 7f23dab7c700 1 -- 172.21.15.7:0/1843258024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d40c1a40 msgr2=0x7f23d4157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:43:26.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.468+0000 7f23dab7c700 1 --2- 172.21.15.7:0/1843258024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d40c1a40 0x7f23d4157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.468+0000 7f23dab7c700 1 -- 172.21.15.7:0/1843258024 >> v2:172.21.15.143:3300/0 conn(0x7f23d4151de0 msgr2=0x7f23d40c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:43:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.468+0000 7f23dab7c700 1 --2- 172.21.15.7:0/1843258024 >> v2:172.21.15.143:3300/0 conn(0x7f23d4151de0 0x7f23d40c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.468+0000 7f23dab7c700 1 -- 172.21.15.7:0/1843258024 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d406e090 con 0x7f23d40c5140 2023-03-03T19:43:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.468+0000 7f23dab7c700 1 --2- 172.21.15.7:0/1843258024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d40c5140 0x7f23d406a2b0 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 crypto rx=0x7f23d40c76a0 tx=0x7f23cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d15734762324f3f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.468+0000 7f23d9379700 1 -- 172.21.15.7:0/1843258024 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc00e070 con 0x7f23d40c5140 2023-03-03T19:43:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.468+0000 7f23d9379700 1 -- 172.21.15.7:0/1843258024 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23cc00b9c0 con 0x7f23d40c5140 2023-03-03T19:43:26.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.468+0000 7f23d9379700 1 -- 172.21.15.7:0/1843258024 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc00bd70 con 0x7f23d40c5140 2023-03-03T19:43:26.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.469+0000 7f23dc5df700 1 -- 172.21.15.7:0/1843258024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d40c5140 msgr2=0x7f23d406a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:26.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.469+0000 7f23dc5df700 1 --2- 172.21.15.7:0/1843258024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d40c5140 0x7f23d406a2b0 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 crypto rx=0x7f23d40c76a0 tx=0x7f23cc00b330 comp rx=0 tx=0).stop 2023-03-03T19:43:26.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.469+0000 7f23dc5df700 1 -- 172.21.15.7:0/1843258024 shutdown_connections 2023-03-03T19:43:26.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.469+0000 7f23dc5df700 1 --2- 172.21.15.7:0/1843258024 >> v2:172.21.15.143:3300/0 conn(0x7f23d4151de0 0x7f23d40c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.469+0000 7f23dc5df700 1 --2- 172.21.15.7:0/1843258024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d40c5140 0x7f23d406a2b0 unknown :-1 s=CLOSED pgs=3573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.469+0000 7f23dc5df700 1 --2- 172.21.15.7:0/1843258024 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d40c1a40 0x7f23d4157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.469+0000 7f23dc5df700 1 -- 172.21.15.7:0/1843258024 >> 172.21.15.7:0/1843258024 conn(0x7f23d40a9430 msgr2=0x7f23d40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:43:26.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.469+0000 7f23dc5df700 1 -- 172.21.15.7:0/1843258024 shutdown_connections 2023-03-03T19:43:26.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.469+0000 7f23dc5df700 1 -- 172.21.15.7:0/1843258024 wait complete. 2023-03-03T19:43:26.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.470+0000 7f23dc5df700 1 Processor -- start 2023-03-03T19:43:26.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.470+0000 7f23dc5df700 1 -- start start 2023-03-03T19:43:26.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dc5df700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d40c1a40 0x7f23d406cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:26.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dc5df700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d40c5140 0x7f23d4071490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:26.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dc5df700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d4151de0 0x7f23d406fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:26.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dc5df700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d41ecfd0 con 0x7f23d4151de0 2023-03-03T19:43:26.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dc5df700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d41edb50 con 0x7f23d40c5140 2023-03-03T19:43:26.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dc5df700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d41ed220 con 0x7f23d40c1a40 2023-03-03T19:43:26.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dab7c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d4151de0 0x7f23d406fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:26.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dab7c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d4151de0 0x7f23d406fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42778/0 (socket says 172.21.15.7:42778) 2023-03-03T19:43:26.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dab7c700 1 -- 172.21.15.7:0/1196249473 learned_addr learned my addr 172.21.15.7:0/1196249473 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:43:26.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23da37b700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d40c1a40 0x7f23d406cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:26.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23d9b7a700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d40c5140 0x7f23d4071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:26.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dab7c700 1 -- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d40c1a40 msgr2=0x7f23d406cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:26.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dab7c700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d40c1a40 0x7f23d406cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dab7c700 1 -- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d40c5140 msgr2=0x7f23d4071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:26.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dab7c700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d40c5140 0x7f23d4071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dab7c700 1 -- 172.21.15.7:0/1196249473 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc009b30 con 0x7f23d4151de0 2023-03-03T19:43:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.471+0000 7f23dab7c700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d4151de0 0x7f23d406fc30 secure :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 crypto rx=0x7f23cc012000 tx=0x7f23cc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.472+0000 7f23caffd700 1 -- 172.21.15.7:0/1196249473 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc004310 con 0x7f23d4151de0 2023-03-03T19:43:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.472+0000 7f23caffd700 1 -- 172.21.15.7:0/1196249473 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23cc004470 con 0x7f23d4151de0 2023-03-03T19:43:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.472+0000 7f23dc5df700 1 -- 172.21.15.7:0/1196249473 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d4066de0 con 0x7f23d4151de0 2023-03-03T19:43:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.472+0000 7f23caffd700 1 -- 172.21.15.7:0/1196249473 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23cc004820 con 0x7f23d4151de0 2023-03-03T19:43:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.472+0000 7f23dc5df700 1 -- 172.21.15.7:0/1196249473 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d40c4c20 con 0x7f23d4151de0 2023-03-03T19:43:26.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.473+0000 7f23dc5df700 1 -- 172.21.15.7:0/1196249473 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d41580e0 con 0x7f23d4151de0 2023-03-03T19:43:26.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.473+0000 7f23caffd700 1 -- 172.21.15.7:0/1196249473 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f23cc01c440 con 0x7f23d4151de0 2023-03-03T19:43:26.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.474+0000 7f23caffd700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23b8068320 0x7f23b806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:26.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.474+0000 7f23caffd700 1 -- 172.21.15.7:0/1196249473 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f23cc087910 con 0x7f23d4151de0 2023-03-03T19:43:26.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.474+0000 7f23da37b700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23b8068320 0x7f23b806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:26.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.475+0000 7f23da37b700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23b8068320 0x7f23b806a7d0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f23d0006fd0 tx=0x7f23d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.478+0000 7f23caffd700 1 -- 172.21.15.7:0/1196249473 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f23cc053c50 con 0x7f23d4151de0 2023-03-03T19:43:26.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.717+0000 7f23dc5df700 1 -- 172.21.15.7:0/1196249473 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23d4074bb0 con 0x7f23d4151de0 2023-03-03T19:43:26.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.718+0000 7f23caffd700 1 -- 172.21.15.7:0/1196249473 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f23cc057270 con 0x7f23d4151de0 2023-03-03T19:43:26.719 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:43:26.719 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:43:26.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 -- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23b8068320 msgr2=0x7f23b806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23b8068320 0x7f23b806a7d0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f23d0006fd0 tx=0x7f23d0008040 comp rx=0 tx=0).stop 2023-03-03T19:43:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 -- 172.21.15.7:0/1196249473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d4151de0 msgr2=0x7f23d406fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d4151de0 0x7f23d406fc30 secure :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 crypto rx=0x7f23cc012000 tx=0x7f23cc004ad0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 -- 172.21.15.7:0/1196249473 shutdown_connections 2023-03-03T19:43:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f23b8068320 0x7f23b806a7d0 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23d4151de0 0x7f23d406fc30 unknown :-1 s=CLOSED pgs=3574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f23d40c1a40 0x7f23d406cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 --2- 172.21.15.7:0/1196249473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f23d40c5140 0x7f23d4071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:26.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 -- 172.21.15.7:0/1196249473 >> 172.21.15.7:0/1196249473 conn(0x7f23d40a9430 msgr2=0x7f23d40bfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:43:26.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 -- 172.21.15.7:0/1196249473 shutdown_connections 2023-03-03T19:43:26.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:26.720+0000 7f23dc5df700 1 -- 172.21.15.7:0/1196249473 wait complete. 2023-03-03T19:43:26.722 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:43:26.732 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before thrashing 2023-03-03T19:43:30.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:43:30.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.872+0000 7f6f8d76b700 1 Processor -- start 2023-03-03T19:43:30.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.872+0000 7f6f8d76b700 1 -- start start 2023-03-03T19:43:30.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.872+0000 7f6f8d76b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f880c00e0 0x7f6f880c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:30.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.872+0000 7f6f8d76b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6f88154ff0 0x7f6f880cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:30.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.872+0000 7f6f8d76b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f880c7310 con 0x7f6f880caa30 2023-03-03T19:43:30.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.872+0000 7f6f8d76b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f880bfd10 con 0x7f6f88154ff0 2023-03-03T19:43:30.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.872+0000 7f6f8d76b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880bfa30 con 0x7f6f880c00e0 2023-03-03T19:43:30.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f86ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6f880caa30 0x7f6f880c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36762/0 (socket says 172.21.15.7:36762) 2023-03-03T19:43:30.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f86ffd700 1 -- 172.21.15.7:0/2142529974 learned_addr learned my addr 172.21.15.7:0/2142529974 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:43:30.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f877fe700 1 --2- 172.21.15.7:0/2142529974 >> v2:172.21.15.143:3300/0 conn(0x7f6f88154ff0 0x7f6f880cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:30.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f867fc700 1 --2- 172.21.15.7:0/2142529974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f880c00e0 0x7f6f880c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:30.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f85ffb700 1 -- 172.21.15.7:0/2142529974 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303059183 0 0) 0x7f6f880c7310 con 0x7f6f880caa30 2023-03-03T19:43:30.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f85ffb700 1 -- 172.21.15.7:0/2142529974 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f700035b0 con 0x7f6f880caa30 2023-03-03T19:43:30.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f867fc700 1 -- 172.21.15.7:0/2142529974 >> v2:172.21.15.143:3300/0 conn(0x7f6f88154ff0 msgr2=0x7f6f880cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:43:30.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f867fc700 1 --2- 172.21.15.7:0/2142529974 >> v2:172.21.15.143:3300/0 conn(0x7f6f88154ff0 0x7f6f880cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:30.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f867fc700 1 -- 172.21.15.7:0/2142529974 >> v1:172.21.15.7:6789/0 conn(0x7f6f880caa30 legacy=0x7f6f880c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:30.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f85ffb700 1 -- 172.21.15.7:0/2142529974 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2265718456 0 0) 0x7f6f700035b0 con 0x7f6f880caa30 2023-03-03T19:43:30.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f867fc700 1 -- 172.21.15.7:0/2142529974 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f88154510 con 0x7f6f880c00e0 2023-03-03T19:43:30.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f877fe700 1 --2- 172.21.15.7:0/2142529974 >> v2:172.21.15.143:3300/0 conn(0x7f6f88154ff0 0x7f6f880cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:43:30.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.873+0000 7f6f867fc700 1 --2- 172.21.15.7:0/2142529974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f880c00e0 0x7f6f880c5c60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009820 tx=0x7f6f7c009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97484fd7bbc50329 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:30.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f85ffb700 1 -- 172.21.15.7:0/2142529974 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78001670 con 0x7f6f880c00e0 2023-03-03T19:43:30.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f85ffb700 1 -- 172.21.15.7:0/2142529974 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c005c30 con 0x7f6f880c00e0 2023-03-03T19:43:30.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f85ffb700 1 -- 172.21.15.7:0/2142529974 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c004030 con 0x7f6f880c00e0 2023-03-03T19:43:30.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f8d76b700 1 -- 172.21.15.7:0/2142529974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f880c00e0 msgr2=0x7f6f880c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:30.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f8d76b700 1 --2- 172.21.15.7:0/2142529974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f880c00e0 0x7f6f880c5c60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009820 tx=0x7f6f7c009b30 comp rx=0 tx=0).stop 2023-03-03T19:43:30.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f8d76b700 1 -- 172.21.15.7:0/2142529974 shutdown_connections 2023-03-03T19:43:30.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f8d76b700 1 --2- 172.21.15.7:0/2142529974 >> v2:172.21.15.143:3300/0 conn(0x7f6f88154ff0 0x7f6f880cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:30.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f8d76b700 1 --2- 172.21.15.7:0/2142529974 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f880c00e0 0x7f6f880c5c60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:30.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f8d76b700 1 -- 172.21.15.7:0/2142529974 >> 172.21.15.7:0/2142529974 conn(0x7f6f88066e40 msgr2=0x7f6f88071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:43:30.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f8d76b700 1 -- 172.21.15.7:0/2142529974 shutdown_connections 2023-03-03T19:43:30.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.874+0000 7f6f8d76b700 1 -- 172.21.15.7:0/2142529974 wait complete. 2023-03-03T19:43:30.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f8d76b700 1 Processor -- start 2023-03-03T19:43:30.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f8d76b700 1 -- start start 2023-03-03T19:43:30.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f8d76b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f880c00e0 0x7f6f88157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:30.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f8d76b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f880caa30 0x7f6f88149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:30.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f8d76b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f88154ff0 0x7f6f880c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:30.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f8d76b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8814c200 con 0x7f6f880caa30 2023-03-03T19:43:30.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f8d76b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f881edae0 con 0x7f6f880c00e0 2023-03-03T19:43:30.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f8d76b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f881eddc0 con 0x7f6f88154ff0 2023-03-03T19:43:30.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f867fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f880caa30 0x7f6f88149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:30.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f867fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f880caa30 0x7f6f88149dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42788/0 (socket says 172.21.15.7:42788) 2023-03-03T19:43:30.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.876+0000 7f6f867fc700 1 -- 172.21.15.7:0/11011492 learned_addr learned my addr 172.21.15.7:0/11011492 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:43:30.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f86ffd700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f880c00e0 0x7f6f88157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:30.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f877fe700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f88154ff0 0x7f6f880c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:30.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f867fc700 1 -- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f88154ff0 msgr2=0x7f6f880c3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:30.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f867fc700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f88154ff0 0x7f6f880c3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:30.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f867fc700 1 -- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f880c00e0 msgr2=0x7f6f88157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:30.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f867fc700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f880c00e0 0x7f6f88157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:30.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f867fc700 1 -- 172.21.15.7:0/11011492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f80009a70 con 0x7f6f880caa30 2023-03-03T19:43:30.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f867fc700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f880caa30 0x7f6f88149dd0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c000c00 tx=0x7f6f7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:30.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f6f7fe700 1 -- 172.21.15.7:0/11011492 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c003b30 con 0x7f6f880caa30 2023-03-03T19:43:30.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f6f7fe700 1 -- 172.21.15.7:0/11011492 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c003cb0 con 0x7f6f880caa30 2023-03-03T19:43:30.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f8d76b700 1 -- 172.21.15.7:0/11011492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c009430 con 0x7f6f880caa30 2023-03-03T19:43:30.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f6f7fe700 1 -- 172.21.15.7:0/11011492 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c01d670 con 0x7f6f880caa30 2023-03-03T19:43:30.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.877+0000 7f6f8d76b700 1 -- 172.21.15.7:0/11011492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f88154ad0 con 0x7f6f880caa30 2023-03-03T19:43:30.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.878+0000 7f6f8d76b700 1 -- 172.21.15.7:0/11011492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f881547f0 con 0x7f6f880caa30 2023-03-03T19:43:30.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.879+0000 7f6f6f7fe700 1 -- 172.21.15.7:0/11011492 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6f7c02d070 con 0x7f6f880caa30 2023-03-03T19:43:30.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.879+0000 7f6f6f7fe700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f740682f0 0x7f6f7406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:43:30.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.879+0000 7f6f6f7fe700 1 -- 172.21.15.7:0/11011492 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6f7c087190 con 0x7f6f880caa30 2023-03-03T19:43:30.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.879+0000 7f6f86ffd700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f740682f0 0x7f6f7406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:43:30.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.880+0000 7f6f86ffd700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f740682f0 0x7f6f7406a7a0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f6f78000f60 tx=0x7f6f78001230 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:43:30.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:30.883+0000 7f6f6f7fe700 1 -- 172.21.15.7:0/11011492 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6f7c053530 con 0x7f6f880caa30 2023-03-03T19:43:31.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.125+0000 7f6f8d76b700 1 -- 172.21.15.7:0/11011492 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f880bf650 con 0x7f6f880caa30 2023-03-03T19:43:31.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.126+0000 7f6f6f7fe700 1 -- 172.21.15.7:0/11011492 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6f7c006d80 con 0x7f6f880caa30 2023-03-03T19:43:31.127 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:43:31.127 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:43:31.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 -- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f740682f0 msgr2=0x7f6f7406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f740682f0 0x7f6f7406a7a0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f6f78000f60 tx=0x7f6f78001230 comp rx=0 tx=0).stop 2023-03-03T19:43:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 -- 172.21.15.7:0/11011492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f880caa30 msgr2=0x7f6f88149dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:43:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f880caa30 0x7f6f88149dd0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c000c00 tx=0x7f6f7c004750 comp rx=0 tx=0).stop 2023-03-03T19:43:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 -- 172.21.15.7:0/11011492 shutdown_connections 2023-03-03T19:43:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6f88154ff0 0x7f6f880c3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f880c00e0 0x7f6f88157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6f740682f0 0x7f6f7406a7a0 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:31.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 --2- 172.21.15.7:0/11011492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f880caa30 0x7f6f88149dd0 unknown :-1 s=CLOSED pgs=3576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:43:31.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 -- 172.21.15.7:0/11011492 >> 172.21.15.7:0/11011492 conn(0x7f6f88066e40 msgr2=0x7f6f88151230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:43:31.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 -- 172.21.15.7:0/11011492 shutdown_connections 2023-03-03T19:43:31.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:43:31.128+0000 7f6f8d76b700 1 -- 172.21.15.7:0/11011492 wait complete. 2023-03-03T19:43:31.130 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:43:31.140 INFO:tasks.mds_thrash.fs.[b]:waiting for 91 secs before thrashing 2023-03-03T19:43:51.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:43:51.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:44:00.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:44:00.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.336+0000 7fd865a15700 1 Processor -- start 2023-03-03T19:44:00.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.336+0000 7fd865a15700 1 -- start start 2023-03-03T19:44:00.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.336+0000 7fd865a15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c5c60 0x7fd860154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:00.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.336+0000 7fd865a15700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8600c7310 con 0x7fd8600c00e0 2023-03-03T19:44:00.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.336+0000 7fd865a15700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8600bfd10 con 0x7fd8600caa30 2023-03-03T19:44:00.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.336+0000 7fd865a15700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600bfa30 con 0x7fd8600c5c60 2023-03-03T19:44:00.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85e7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd8600c00e0 0x7fd86015cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36782/0 (socket says 172.21.15.7:36782) 2023-03-03T19:44:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85e7fc700 1 -- 172.21.15.7:0/2663441402 learned_addr learned my addr 172.21.15.7:0/2663441402 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:44:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85f7fe700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c5c60 0x7fd860154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453332114 0 0) 0x7fd8600c7310 con 0x7fd8600c00e0 2023-03-03T19:44:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd84c0035b0 con 0x7fd8600c00e0 2023-03-03T19:44:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196935784 0 0) 0x7fd84c0035b0 con 0x7fd8600c00e0 2023-03-03T19:44:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c5c60 msgr2=0x7fd860154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:44:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c5c60 0x7fd860154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 >> v1:172.21.15.143:6789/0 conn(0x7fd8600caa30 legacy=0x7fd8600c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860154510 con 0x7fd8600c00e0 2023-03-03T19:44:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd8480017f0 con 0x7fd8600c00e0 2023-03-03T19:44:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 >> v1:172.21.15.7:6789/0 conn(0x7fd8600c00e0 legacy=0x7fd86015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd84c003aa0 0x7fd84c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c5c60 0x7fd84c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84c004d90 0x7fd84c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:00.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600bfa30 con 0x7fd84c003aa0 2023-03-03T19:44:00.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c00bb10 con 0x7fd84c004d90 2023-03-03T19:44:00.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c00bcd0 con 0x7fd8600c5c60 2023-03-03T19:44:00.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85effd700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd84c003aa0 0x7fd84c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:00.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85e7fc700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c5c60 0x7fd84c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:00.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.337+0000 7fd85f7fe700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84c004d90 0x7fd84c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:00.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd85effd700 1 -- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c5c60 msgr2=0x7fd84c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:44:00.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd85effd700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8600c5c60 0x7fd84c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:00.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd85effd700 1 -- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84c004d90 msgr2=0x7fd84c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:44:00.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd85effd700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd84c004d90 0x7fd84c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:00.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd85effd700 1 -- 172.21.15.7:0/2663441402 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c0035b0 con 0x7fd84c003aa0 2023-03-03T19:44:00.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd85effd700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd84c003aa0 0x7fd84c003e70 secure :-1 s=READY pgs=3578 cs=0 l=1 rev1=1 crypto rx=0x7fd8500018b0 tx=0x7fd84c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76657d75a88b67cd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:44:00.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850001370 con 0x7fd84c003aa0 2023-03-03T19:44:00.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85000c6b0 con 0x7fd84c003aa0 2023-03-03T19:44:00.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd85dffb700 1 -- 172.21.15.7:0/2663441402 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85000ca60 con 0x7fd84c003aa0 2023-03-03T19:44:00.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd865a15700 1 -- 172.21.15.7:0/2663441402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd84c003aa0 msgr2=0x7fd84c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:00.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd865a15700 1 --2- 172.21.15.7:0/2663441402 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd84c003aa0 0x7fd84c003e70 secure :-1 s=READY pgs=3578 cs=0 l=1 rev1=1 crypto rx=0x7fd8500018b0 tx=0x7fd84c005880 comp rx=0 tx=0).stop 2023-03-03T19:44:00.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd85f7fe700 1 -- 172.21.15.7:0/2663441402 reap_dead start 2023-03-03T19:44:00.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd865a15700 1 -- 172.21.15.7:0/2663441402 shutdown_connections 2023-03-03T19:44:00.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd865a15700 1 -- 172.21.15.7:0/2663441402 >> 172.21.15.7:0/2663441402 conn(0x7fd860066e40 msgr2=0x7fd860071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:44:00.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.338+0000 7fd865a15700 1 -- 172.21.15.7:0/2663441402 shutdown_connections 2023-03-03T19:44:00.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.339+0000 7fd865a15700 1 -- 172.21.15.7:0/2663441402 wait complete. 2023-03-03T19:44:00.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.340+0000 7fd865a15700 1 Processor -- start 2023-03-03T19:44:00.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.340+0000 7fd865a15700 1 -- start start 2023-03-03T19:44:00.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd865a15700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 0x7fd8601562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:00.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd865a15700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86014d630 0x7fd86014daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:00.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd865a15700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8601570f0 0x7fd860157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:00.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd865a15700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600c2260 con 0x7fd8601570f0 2023-03-03T19:44:00.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd865a15700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600c3c30 con 0x7fd86014d630 2023-03-03T19:44:00.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd865a15700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600c2f90 con 0x7fd84c003aa0 2023-03-03T19:44:00.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8601570f0 0x7fd860157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:00.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8601570f0 0x7fd860157590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42818/0 (socket says 172.21.15.7:42818) 2023-03-03T19:44:00.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85f7fe700 1 -- 172.21.15.7:0/3623235198 learned_addr learned my addr 172.21.15.7:0/3623235198 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:44:00.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85effd700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 0x7fd8601562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:00.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85e7fc700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86014d630 0x7fd86014daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:00.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85f7fe700 1 -- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 msgr2=0x7fd8601562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:00.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85f7fe700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 0x7fd8601562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:00.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85f7fe700 1 -- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86014d630 msgr2=0x7fd86014daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:00.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85f7fe700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86014d630 0x7fd86014daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:00.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85f7fe700 1 -- 172.21.15.7:0/3623235198 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860154ad0 con 0x7fd8601570f0 2023-03-03T19:44:00.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85effd700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 0x7fd8601562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:44:00.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd85f7fe700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8601570f0 0x7fd860157590 secure :-1 s=READY pgs=3579 cs=0 l=1 rev1=1 crypto rx=0x7fd858007d10 tx=0x7fd8580114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:44:00.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd857fff700 1 -- 172.21.15.7:0/3623235198 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85800e070 con 0x7fd8601570f0 2023-03-03T19:44:00.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd857fff700 1 -- 172.21.15.7:0/3623235198 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd858003c30 con 0x7fd8601570f0 2023-03-03T19:44:00.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd865a15700 1 -- 172.21.15.7:0/3623235198 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8601547f0 con 0x7fd8601570f0 2023-03-03T19:44:00.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd857fff700 1 -- 172.21.15.7:0/3623235198 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858003de0 con 0x7fd8601570f0 2023-03-03T19:44:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.341+0000 7fd865a15700 1 -- 172.21.15.7:0/3623235198 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8600bf650 con 0x7fd8601570f0 2023-03-03T19:44:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.342+0000 7fd865a15700 1 -- 172.21.15.7:0/3623235198 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860074bb0 con 0x7fd8601570f0 2023-03-03T19:44:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.343+0000 7fd857fff700 1 -- 172.21.15.7:0/3623235198 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd8580119e0 con 0x7fd8601570f0 2023-03-03T19:44:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.343+0000 7fd857fff700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8400682f0 0x7fd84006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.343+0000 7fd857fff700 1 -- 172.21.15.7:0/3623235198 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd858086770 con 0x7fd8601570f0 2023-03-03T19:44:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.344+0000 7fd85effd700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8400682f0 0x7fd84006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:00.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.344+0000 7fd85effd700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8400682f0 0x7fd84006a7a0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fd8500018b0 tx=0x7fd850005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:44:00.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.348+0000 7fd857fff700 1 -- 172.21.15.7:0/3623235198 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd858052b10 con 0x7fd8601570f0 2023-03-03T19:44:00.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.584+0000 7fd865a15700 1 -- 172.21.15.7:0/3623235198 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8600c3d70 con 0x7fd8601570f0 2023-03-03T19:44:00.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.585+0000 7fd857fff700 1 -- 172.21.15.7:0/3623235198 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd8600c3d70 con 0x7fd8601570f0 2023-03-03T19:44:00.586 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:44:00.586 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:44:00.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 -- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8400682f0 msgr2=0x7fd84006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:00.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8400682f0 0x7fd84006a7a0 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fd8500018b0 tx=0x7fd850005040 comp rx=0 tx=0).stop 2023-03-03T19:44:00.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 -- 172.21.15.7:0/3623235198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8601570f0 msgr2=0x7fd860157590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:00.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8601570f0 0x7fd860157590 secure :-1 s=READY pgs=3579 cs=0 l=1 rev1=1 crypto rx=0x7fd858007d10 tx=0x7fd8580114b0 comp rx=0 tx=0).stop 2023-03-03T19:44:00.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 -- 172.21.15.7:0/3623235198 shutdown_connections 2023-03-03T19:44:00.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8400682f0 0x7fd84006a7a0 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:00.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8601570f0 0x7fd860157590 unknown :-1 s=CLOSED pgs=3579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:00.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd84c003aa0 0x7fd8601562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:00.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 --2- 172.21.15.7:0/3623235198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd86014d630 0x7fd86014daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:00.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 -- 172.21.15.7:0/3623235198 >> 172.21.15.7:0/3623235198 conn(0x7fd860066e40 msgr2=0x7fd860071710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:44:00.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 -- 172.21.15.7:0/3623235198 shutdown_connections 2023-03-03T19:44:00.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:00.587+0000 7fd865a15700 1 -- 172.21.15.7:0/3623235198 wait complete. 2023-03-03T19:44:00.589 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:44:00.599 INFO:tasks.mds_thrash.fs.[a]:waiting for 59 secs before thrashing 2023-03-03T19:44:21.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:44:21.816 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:44:51.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:44:51.947 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:44:59.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:44:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.738+0000 7f55bf671700 1 Processor -- start 2023-03-03T19:44:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.738+0000 7f55bf671700 1 -- start start 2023-03-03T19:44:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.739+0000 7f55bf671700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55b8155390 0x7f55b80c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.739+0000 7f55bf671700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f55b8151770 0x7f55b8151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.739+0000 7f55bf671700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b8154910 con 0x7f55b806e020 2023-03-03T19:44:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.739+0000 7f55bf671700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b8070b90 con 0x7f55b8151770 2023-03-03T19:44:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.739+0000 7f55bf671700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b8151320 con 0x7f55b8155390 2023-03-03T19:44:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.739+0000 7f55bdc0e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f55b8151770 0x7f55b8151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.739+0000 7f55bdc0e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f55b8151770 0x7f55b8151bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42862/0 (socket says 172.21.15.7:42862) 2023-03-03T19:44:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.739+0000 7f55bdc0e700 1 -- 172.21.15.7:0/2764759765 learned_addr learned my addr 172.21.15.7:0/2764759765 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:44:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.740+0000 7f55bd40d700 1 --2- 172.21.15.7:0/2764759765 >> v2:172.21.15.143:3300/0 conn(0x7f55b8155390 0x7f55b80c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.740+0000 7f55bdc0e700 1 -- 172.21.15.7:0/2764759765 >> v2:172.21.15.143:3300/0 conn(0x7f55b8155390 msgr2=0x7f55b80c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:44:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.740+0000 7f55bdc0e700 1 --2- 172.21.15.7:0/2764759765 >> v2:172.21.15.143:3300/0 conn(0x7f55b8155390 0x7f55b80c3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.740+0000 7f55bdc0e700 1 -- 172.21.15.7:0/2764759765 >> v1:172.21.15.143:6789/0 conn(0x7f55b806e020 legacy=0x7f55b8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.740+0000 7f55bdc0e700 1 -- 172.21.15.7:0/2764759765 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b80702a0 con 0x7f55b8151770 2023-03-03T19:44:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.740+0000 7f55bdc0e700 1 --2- 172.21.15.7:0/2764759765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8151770 0x7f55b8151bd0 secure :-1 s=READY pgs=3580 cs=0 l=1 rev1=1 crypto rx=0x7f55b806d700 tx=0x7f55b000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502662d44faca7bd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:44:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.740+0000 7f55affff700 1 -- 172.21.15.7:0/2764759765 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b0010070 con 0x7f55b8151770 2023-03-03T19:44:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.740+0000 7f55affff700 1 -- 172.21.15.7:0/2764759765 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b000dd90 con 0x7f55b8151770 2023-03-03T19:44:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.740+0000 7f55affff700 1 -- 172.21.15.7:0/2764759765 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b000b310 con 0x7f55b8151770 2023-03-03T19:44:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.741+0000 7f55bf671700 1 -- 172.21.15.7:0/2764759765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8151770 msgr2=0x7f55b8151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.741+0000 7f55bf671700 1 --2- 172.21.15.7:0/2764759765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8151770 0x7f55b8151bd0 secure :-1 s=READY pgs=3580 cs=0 l=1 rev1=1 crypto rx=0x7f55b806d700 tx=0x7f55b000d430 comp rx=0 tx=0).stop 2023-03-03T19:44:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.741+0000 7f55bf671700 1 -- 172.21.15.7:0/2764759765 shutdown_connections 2023-03-03T19:44:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.741+0000 7f55bf671700 1 --2- 172.21.15.7:0/2764759765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8151770 0x7f55b8151bd0 unknown :-1 s=CLOSED pgs=3580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.741+0000 7f55bf671700 1 --2- 172.21.15.7:0/2764759765 >> v2:172.21.15.143:3300/0 conn(0x7f55b8155390 0x7f55b80c3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.741+0000 7f55bf671700 1 -- 172.21.15.7:0/2764759765 >> 172.21.15.7:0/2764759765 conn(0x7f55b806c110 msgr2=0x7f55b806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:44:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.741+0000 7f55bf671700 1 -- 172.21.15.7:0/2764759765 shutdown_connections 2023-03-03T19:44:59.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.741+0000 7f55bf671700 1 -- 172.21.15.7:0/2764759765 wait complete. 2023-03-03T19:44:59.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.742+0000 7f55bf671700 1 Processor -- start 2023-03-03T19:44:59.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.742+0000 7f55bf671700 1 -- start start 2023-03-03T19:44:59.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bf671700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55b806e020 0x7f55b80ac490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:59.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bf671700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b8151770 0x7f55b8153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:59.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bf671700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8155390 0x7f55b8148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:59.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bf671700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b814c140 con 0x7f55b8155390 2023-03-03T19:44:59.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bf671700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b814c480 con 0x7f55b8151770 2023-03-03T19:44:59.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bf671700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b80aa9e0 con 0x7f55b806e020 2023-03-03T19:44:59.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bdc0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8155390 0x7f55b8148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:59.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bdc0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8155390 0x7f55b8148d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42868/0 (socket says 172.21.15.7:42868) 2023-03-03T19:44:59.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bdc0e700 1 -- 172.21.15.7:0/3240695573 learned_addr learned my addr 172.21.15.7:0/3240695573 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:44:59.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bdc0e700 1 -- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55b806e020 msgr2=0x7f55b80ac490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:59.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bdc0e700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55b806e020 0x7f55b80ac490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:59.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bdc0e700 1 -- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b8151770 msgr2=0x7f55b8153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:59.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bdc0e700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b8151770 0x7f55b8153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bdc0e700 1 -- 172.21.15.7:0/3240695573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b000bd40 con 0x7f55b8155390 2023-03-03T19:44:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bdc0e700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8155390 0x7f55b8148d90 secure :-1 s=READY pgs=3581 cs=0 l=1 rev1=1 crypto rx=0x7f55b000d470 tx=0x7f55b0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:44:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55adffb700 1 -- 172.21.15.7:0/3240695573 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b0009a10 con 0x7f55b8155390 2023-03-03T19:44:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55adffb700 1 -- 172.21.15.7:0/3240695573 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b0009b70 con 0x7f55b8155390 2023-03-03T19:44:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55adffb700 1 -- 172.21.15.7:0/3240695573 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b000b4c0 con 0x7f55b8155390 2023-03-03T19:44:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.743+0000 7f55bf671700 1 -- 172.21.15.7:0/3240695573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b8070820 con 0x7f55b8155390 2023-03-03T19:44:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.744+0000 7f55bf671700 1 -- 172.21.15.7:0/3240695573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b8070540 con 0x7f55b8155390 2023-03-03T19:44:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.744+0000 7f55bf671700 1 -- 172.21.15.7:0/3240695573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b8155760 con 0x7f55b8155390 2023-03-03T19:44:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.745+0000 7f55adffb700 1 -- 172.21.15.7:0/3240695573 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f55b0023030 con 0x7f55b8155390 2023-03-03T19:44:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.745+0000 7f55adffb700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f559c0682f0 0x7f559c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:44:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.745+0000 7f55adffb700 1 -- 172.21.15.7:0/3240695573 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f55b0087a40 con 0x7f55b8155390 2023-03-03T19:44:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.746+0000 7f55bd40d700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f559c0682f0 0x7f559c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:44:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.746+0000 7f55bd40d700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f559c0682f0 0x7f559c06a7a0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f55a80098d0 tx=0x7f55a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:44:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.750+0000 7f55adffb700 1 -- 172.21.15.7:0/3240695573 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f55b0053de0 con 0x7f55b8155390 2023-03-03T19:44:59.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.985+0000 7f55bf671700 1 -- 172.21.15.7:0/3240695573 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55b8074bb0 con 0x7f55b8155390 2023-03-03T19:44:59.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.986+0000 7f55adffb700 1 -- 172.21.15.7:0/3240695573 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f55b0057400 con 0x7f55b8155390 2023-03-03T19:44:59.987 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:44:59.987 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:44:59.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 -- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f559c0682f0 msgr2=0x7f559c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:59.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f559c0682f0 0x7f559c06a7a0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f55a80098d0 tx=0x7f55a8008040 comp rx=0 tx=0).stop 2023-03-03T19:44:59.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 -- 172.21.15.7:0/3240695573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8155390 msgr2=0x7f55b8148d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:44:59.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8155390 0x7f55b8148d90 secure :-1 s=READY pgs=3581 cs=0 l=1 rev1=1 crypto rx=0x7f55b000d470 tx=0x7f55b0004ad0 comp rx=0 tx=0).stop 2023-03-03T19:44:59.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 -- 172.21.15.7:0/3240695573 shutdown_connections 2023-03-03T19:44:59.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f559c0682f0 0x7f559c06a7a0 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:59.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55b8155390 0x7f55b8148d90 unknown :-1 s=CLOSED pgs=3581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:59.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f55b806e020 0x7f55b80ac490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:59.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 --2- 172.21.15.7:0/3240695573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55b8151770 0x7f55b8153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:44:59.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 -- 172.21.15.7:0/3240695573 >> 172.21.15.7:0/3240695573 conn(0x7f55b806c110 msgr2=0x7f55b80c2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:44:59.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 -- 172.21.15.7:0/3240695573 shutdown_connections 2023-03-03T19:44:59.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:44:59.988+0000 7f55bf671700 1 -- 172.21.15.7:0/3240695573 wait complete. 2023-03-03T19:44:59.991 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:45:00.001 INFO:tasks.mds_thrash.fs.[a]:waiting for 19 secs before thrashing 2023-03-03T19:45:02.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:45:02.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.280+0000 7f4e6eeca700 1 Processor -- start 2023-03-03T19:45:02.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.280+0000 7f4e6eeca700 1 -- start start 2023-03-03T19:45:02.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.281+0000 7f4e6eeca700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e6814f9d0 0x7f4e68156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:02.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.281+0000 7f4e6eeca700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e680abe20 0x7f4e681529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:02.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.281+0000 7f4e6eeca700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e680c2f30 con 0x7f4e6814fda0 2023-03-03T19:45:02.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.281+0000 7f4e6eeca700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e680c31d0 con 0x7f4e680abe20 2023-03-03T19:45:02.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.281+0000 7f4e6eeca700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e681524a0 con 0x7f4e6814f9d0 2023-03-03T19:45:02.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e6d467700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e680abe20 0x7f4e681529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:02.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e6d467700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4e680abe20 0x7f4e681529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42876/0 (socket says 172.21.15.7:42876) 2023-03-03T19:45:02.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e6cc66700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4e6814f9d0 0x7f4e68156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:02.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e6d467700 1 -- 172.21.15.7:0/3242571037 learned_addr learned my addr 172.21.15.7:0/3242571037 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:45:02.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e6d467700 1 -- 172.21.15.7:0/3242571037 >> v2:172.21.15.143:3300/0 conn(0x7f4e6814f9d0 msgr2=0x7f4e68156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:45:02.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e6d467700 1 --2- 172.21.15.7:0/3242571037 >> v2:172.21.15.143:3300/0 conn(0x7f4e6814f9d0 0x7f4e68156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:02.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e6d467700 1 -- 172.21.15.7:0/3242571037 >> v1:172.21.15.143:6789/0 conn(0x7f4e6814fda0 legacy=0x7f4e68071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:02.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e6d467700 1 -- 172.21.15.7:0/3242571037 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e680c34b0 con 0x7f4e680abe20 2023-03-03T19:45:02.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e6d467700 1 --2- 172.21.15.7:0/3242571037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 0x7f4e681529c0 secure :-1 s=READY pgs=3582 cs=0 l=1 rev1=1 crypto rx=0x7f4e680710a0 tx=0x7f4e6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8b2d6c827b6fe2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:02.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e677fe700 1 -- 172.21.15.7:0/3242571037 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6000e070 con 0x7f4e680abe20 2023-03-03T19:45:02.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e677fe700 1 -- 172.21.15.7:0/3242571037 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6000b910 con 0x7f4e680abe20 2023-03-03T19:45:02.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.282+0000 7f4e677fe700 1 -- 172.21.15.7:0/3242571037 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6000bcc0 con 0x7f4e680abe20 2023-03-03T19:45:02.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.283+0000 7f4e6eeca700 1 -- 172.21.15.7:0/3242571037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 msgr2=0x7f4e681529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:02.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.283+0000 7f4e6eeca700 1 --2- 172.21.15.7:0/3242571037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 0x7f4e681529c0 secure :-1 s=READY pgs=3582 cs=0 l=1 rev1=1 crypto rx=0x7f4e680710a0 tx=0x7f4e6000b330 comp rx=0 tx=0).stop 2023-03-03T19:45:02.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.283+0000 7f4e6eeca700 1 -- 172.21.15.7:0/3242571037 shutdown_connections 2023-03-03T19:45:02.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.283+0000 7f4e6eeca700 1 --2- 172.21.15.7:0/3242571037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 0x7f4e681529c0 unknown :-1 s=CLOSED pgs=3582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:02.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.283+0000 7f4e6eeca700 1 --2- 172.21.15.7:0/3242571037 >> v2:172.21.15.143:3300/0 conn(0x7f4e6814f9d0 0x7f4e68156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:02.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.283+0000 7f4e6eeca700 1 -- 172.21.15.7:0/3242571037 >> 172.21.15.7:0/3242571037 conn(0x7f4e680c8900 msgr2=0x7f4e680c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:45:02.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.283+0000 7f4e6eeca700 1 -- 172.21.15.7:0/3242571037 shutdown_connections 2023-03-03T19:45:02.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.283+0000 7f4e6eeca700 1 -- 172.21.15.7:0/3242571037 wait complete. 2023-03-03T19:45:02.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.284+0000 7f4e6eeca700 1 Processor -- start 2023-03-03T19:45:02.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.284+0000 7f4e6eeca700 1 -- start start 2023-03-03T19:45:02.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6eeca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 0x7f4e680c24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:02.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6eeca700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6814f9d0 0x7f4e6814eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:02.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6eeca700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6814fda0 0x7f4e680bd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:02.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6eeca700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e680aa670 con 0x7f4e680abe20 2023-03-03T19:45:02.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6eeca700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e681eb790 con 0x7f4e6814fda0 2023-03-03T19:45:02.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6eeca700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e680aa3b0 con 0x7f4e6814f9d0 2023-03-03T19:45:02.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6cc66700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 0x7f4e680c24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:02.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6cc66700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 0x7f4e680c24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42878/0 (socket says 172.21.15.7:42878) 2023-03-03T19:45:02.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6cc66700 1 -- 172.21.15.7:0/500502949 learned_addr learned my addr 172.21.15.7:0/500502949 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:45:02.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6d467700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6814fda0 0x7f4e680bd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:02.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6cc66700 1 -- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6814f9d0 msgr2=0x7f4e6814eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:02.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e67fff700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6814f9d0 0x7f4e6814eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:02.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6cc66700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6814f9d0 0x7f4e6814eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:02.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6cc66700 1 -- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6814fda0 msgr2=0x7f4e680bd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:02.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6cc66700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6814fda0 0x7f4e680bd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:02.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6cc66700 1 -- 172.21.15.7:0/500502949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e60009b30 con 0x7f4e680abe20 2023-03-03T19:45:02.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.285+0000 7f4e6d467700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6814fda0 0x7f4e680bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:45:02.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.286+0000 7f4e6cc66700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 0x7f4e680c24b0 secure :-1 s=READY pgs=3583 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00e9b0 tx=0x7f4e5c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:02.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.286+0000 7f4e657fa700 1 -- 172.21.15.7:0/500502949 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c009710 con 0x7f4e680abe20 2023-03-03T19:45:02.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.286+0000 7f4e657fa700 1 -- 172.21.15.7:0/500502949 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c00c850 con 0x7f4e680abe20 2023-03-03T19:45:02.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.286+0000 7f4e657fa700 1 -- 172.21.15.7:0/500502949 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c00cbc0 con 0x7f4e680abe20 2023-03-03T19:45:02.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.286+0000 7f4e6eeca700 1 -- 172.21.15.7:0/500502949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e680ac350 con 0x7f4e680abe20 2023-03-03T19:45:02.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.286+0000 7f4e6eeca700 1 -- 172.21.15.7:0/500502949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e680ab630 con 0x7f4e680abe20 2023-03-03T19:45:02.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.287+0000 7f4e6eeca700 1 -- 172.21.15.7:0/500502949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e68071fa0 con 0x7f4e680abe20 2023-03-03T19:45:02.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.288+0000 7f4e657fa700 1 -- 172.21.15.7:0/500502949 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4e5c019030 con 0x7f4e680abe20 2023-03-03T19:45:02.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.288+0000 7f4e657fa700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e4c0682f0 0x7f4e4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:02.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.288+0000 7f4e657fa700 1 -- 172.21.15.7:0/500502949 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4e5c086820 con 0x7f4e680abe20 2023-03-03T19:45:02.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.288+0000 7f4e67fff700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e4c0682f0 0x7f4e4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:02.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.289+0000 7f4e67fff700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e4c0682f0 0x7f4e4c06a7a0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f4e680c4410 tx=0x7f4e54009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:02.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.292+0000 7f4e657fa700 1 -- 172.21.15.7:0/500502949 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4e5c056690 con 0x7f4e680abe20 2023-03-03T19:45:02.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.529+0000 7f4e6eeca700 1 -- 172.21.15.7:0/500502949 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e68074bb0 con 0x7f4e680abe20 2023-03-03T19:45:02.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.530+0000 7f4e657fa700 1 -- 172.21.15.7:0/500502949 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4e5c056280 con 0x7f4e680abe20 2023-03-03T19:45:02.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:45:02.531 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:45:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.531+0000 7f4e6eeca700 1 -- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e4c0682f0 msgr2=0x7f4e4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.531+0000 7f4e6eeca700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e4c0682f0 0x7f4e4c06a7a0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f4e680c4410 tx=0x7f4e54009040 comp rx=0 tx=0).stop 2023-03-03T19:45:02.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.531+0000 7f4e6eeca700 1 -- 172.21.15.7:0/500502949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 msgr2=0x7f4e680c24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.531+0000 7f4e6eeca700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 0x7f4e680c24b0 secure :-1 s=READY pgs=3583 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00e9b0 tx=0x7f4e5c00ed70 comp rx=0 tx=0).stop 2023-03-03T19:45:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.532+0000 7f4e6eeca700 1 -- 172.21.15.7:0/500502949 shutdown_connections 2023-03-03T19:45:02.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.532+0000 7f4e6eeca700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e6814fda0 0x7f4e680bd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:02.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.532+0000 7f4e6eeca700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4e4c0682f0 0x7f4e4c06a7a0 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:02.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.532+0000 7f4e6eeca700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e680abe20 0x7f4e680c24b0 unknown :-1 s=CLOSED pgs=3583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:02.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.532+0000 7f4e6eeca700 1 --2- 172.21.15.7:0/500502949 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4e6814f9d0 0x7f4e6814eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:02.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.532+0000 7f4e6eeca700 1 -- 172.21.15.7:0/500502949 >> 172.21.15.7:0/500502949 conn(0x7f4e680c8900 msgr2=0x7f4e680c6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:45:02.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.532+0000 7f4e6eeca700 1 -- 172.21.15.7:0/500502949 shutdown_connections 2023-03-03T19:45:02.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:02.532+0000 7f4e6eeca700 1 -- 172.21.15.7:0/500502949 wait complete. 2023-03-03T19:45:02.536 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:45:02.544 INFO:tasks.mds_thrash.fs.[b]:waiting for 53 secs before thrashing 2023-03-03T19:45:19.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:45:19.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.142+0000 7f32c9650700 1 Processor -- start 2023-03-03T19:45:19.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.142+0000 7f32c9650700 1 -- start start 2023-03-03T19:45:19.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c9650700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f32c40c5c60 0x7f32c4154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:19.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c9650700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c40c7310 con 0x7f32c40c00e0 2023-03-03T19:45:19.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c9650700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c40bfd10 con 0x7f32c40caa30 2023-03-03T19:45:19.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c9650700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c40c60a0 con 0x7f32c40c5c60 2023-03-03T19:45:19.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c27fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f32c40c00e0 0x7f32c415cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36890/0 (socket says 172.21.15.7:36890) 2023-03-03T19:45:19.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c27fc700 1 -- 172.21.15.7:0/4227679913 learned_addr learned my addr 172.21.15.7:0/4227679913 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:45:19.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c37fe700 1 --2- 172.21.15.7:0/4227679913 >> v2:172.21.15.7:3300/0 conn(0x7f32c40c5c60 0x7f32c4154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:19.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c1ffb700 1 -- 172.21.15.7:0/4227679913 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621678265 0 0) 0x7f32c40c7310 con 0x7f32c40c00e0 2023-03-03T19:45:19.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c1ffb700 1 -- 172.21.15.7:0/4227679913 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ac0035b0 con 0x7f32c40c00e0 2023-03-03T19:45:19.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c37fe700 1 -- 172.21.15.7:0/4227679913 >> v1:172.21.15.143:6789/0 conn(0x7f32c40caa30 legacy=0x7f32c40c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:19.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c37fe700 1 -- 172.21.15.7:0/4227679913 >> v1:172.21.15.7:6789/0 conn(0x7f32c40c00e0 legacy=0x7f32c415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:19.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.143+0000 7f32c37fe700 1 -- 172.21.15.7:0/4227679913 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c40bfa30 con 0x7f32c40c5c60 2023-03-03T19:45:19.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c37fe700 1 --2- 172.21.15.7:0/4227679913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40c5c60 0x7f32c4154ff0 secure :-1 s=READY pgs=3585 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009d90 tx=0x7f32bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee29da74c2e6899b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:19.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c1ffb700 1 -- 172.21.15.7:0/4227679913 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc00e070 con 0x7f32c40c5c60 2023-03-03T19:45:19.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c1ffb700 1 -- 172.21.15.7:0/4227679913 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32bc00bb20 con 0x7f32c40c5c60 2023-03-03T19:45:19.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c1ffb700 1 -- 172.21.15.7:0/4227679913 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc012430 con 0x7f32c40c5c60 2023-03-03T19:45:19.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c9650700 1 -- 172.21.15.7:0/4227679913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40c5c60 msgr2=0x7f32c4154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:19.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c9650700 1 --2- 172.21.15.7:0/4227679913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40c5c60 0x7f32c4154ff0 secure :-1 s=READY pgs=3585 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009d90 tx=0x7f32bc00b330 comp rx=0 tx=0).stop 2023-03-03T19:45:19.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c9650700 1 -- 172.21.15.7:0/4227679913 shutdown_connections 2023-03-03T19:45:19.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c9650700 1 --2- 172.21.15.7:0/4227679913 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40c5c60 0x7f32c4154ff0 unknown :-1 s=CLOSED pgs=3585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:19.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c9650700 1 -- 172.21.15.7:0/4227679913 >> 172.21.15.7:0/4227679913 conn(0x7f32c4066e40 msgr2=0x7f32c4071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:45:19.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c9650700 1 -- 172.21.15.7:0/4227679913 shutdown_connections 2023-03-03T19:45:19.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.144+0000 7f32c9650700 1 -- 172.21.15.7:0/4227679913 wait complete. 2023-03-03T19:45:19.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.146+0000 7f32c9650700 1 Processor -- start 2023-03-03T19:45:19.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.146+0000 7f32c9650700 1 -- start start 2023-03-03T19:45:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.146+0000 7f32c9650700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32c40c00e0 0x7f32c40c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.146+0000 7f32c9650700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c40c5c60 0x7f32c40c3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.146+0000 7f32c9650700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40caa30 0x7f32c4148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c9650700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c4156700 con 0x7f32c40caa30 2023-03-03T19:45:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c9650700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c4155330 con 0x7f32c40c5c60 2023-03-03T19:45:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c9650700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c4155ea0 con 0x7f32c40c00e0 2023-03-03T19:45:19.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40caa30 0x7f32c4148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:19.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40caa30 0x7f32c4148ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42918/0 (socket says 172.21.15.7:42918) 2023-03-03T19:45:19.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c37fe700 1 -- 172.21.15.7:0/1573184624 learned_addr learned my addr 172.21.15.7:0/1573184624 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:45:19.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c27fc700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c40c5c60 0x7f32c40c3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:19.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c2ffd700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32c40c00e0 0x7f32c40c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:19.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c37fe700 1 -- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32c40c00e0 msgr2=0x7f32c40c2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:19.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c37fe700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32c40c00e0 0x7f32c40c2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:19.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c37fe700 1 -- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c40c5c60 msgr2=0x7f32c40c3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:19.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c37fe700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c40c5c60 0x7f32c40c3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:19.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c37fe700 1 -- 172.21.15.7:0/1573184624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc009a70 con 0x7f32c40caa30 2023-03-03T19:45:19.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c27fc700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c40c5c60 0x7f32c40c3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:45:19.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c37fe700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40caa30 0x7f32c4148ac0 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 crypto rx=0x7f32bc014040 tx=0x7f32bc004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:19.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32ab7fe700 1 -- 172.21.15.7:0/1573184624 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc027480 con 0x7f32c40caa30 2023-03-03T19:45:19.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32ab7fe700 1 -- 172.21.15.7:0/1573184624 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32bc020070 con 0x7f32c40caa30 2023-03-03T19:45:19.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c9650700 1 -- 172.21.15.7:0/1573184624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c41547f0 con 0x7f32c40caa30 2023-03-03T19:45:19.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32ab7fe700 1 -- 172.21.15.7:0/1573184624 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc027820 con 0x7f32c40caa30 2023-03-03T19:45:19.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.147+0000 7f32c9650700 1 -- 172.21.15.7:0/1573184624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c4154510 con 0x7f32c40caa30 2023-03-03T19:45:19.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.148+0000 7f32c9650700 1 -- 172.21.15.7:0/1573184624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c4154ad0 con 0x7f32c40caa30 2023-03-03T19:45:19.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.149+0000 7f32ab7fe700 1 -- 172.21.15.7:0/1573184624 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f32bc02e070 con 0x7f32c40caa30 2023-03-03T19:45:19.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.149+0000 7f32ab7fe700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32b00682f0 0x7f32b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:19.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.149+0000 7f32ab7fe700 1 -- 172.21.15.7:0/1573184624 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f32bc022070 con 0x7f32c40caa30 2023-03-03T19:45:19.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.150+0000 7f32c2ffd700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32b00682f0 0x7f32b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:19.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.150+0000 7f32c2ffd700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32b00682f0 0x7f32b006a7a0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f32b8000f60 tx=0x7f32b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:19.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.154+0000 7f32ab7fe700 1 -- 172.21.15.7:0/1573184624 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f32bc054350 con 0x7f32c40caa30 2023-03-03T19:45:19.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.392+0000 7f32c9650700 1 -- 172.21.15.7:0/1573184624 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32c4074bb0 con 0x7f32c40caa30 2023-03-03T19:45:19.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.392+0000 7f32ab7fe700 1 -- 172.21.15.7:0/1573184624 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f32bc057970 con 0x7f32c40caa30 2023-03-03T19:45:19.393 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:45:19.394 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:45:19.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 -- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32b00682f0 msgr2=0x7f32b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:19.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32b00682f0 0x7f32b006a7a0 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f32b8000f60 tx=0x7f32b8009040 comp rx=0 tx=0).stop 2023-03-03T19:45:19.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 -- 172.21.15.7:0/1573184624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40caa30 msgr2=0x7f32c4148ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:19.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40caa30 0x7f32c4148ac0 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 crypto rx=0x7f32bc014040 tx=0x7f32bc004ad0 comp rx=0 tx=0).stop 2023-03-03T19:45:19.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 -- 172.21.15.7:0/1573184624 shutdown_connections 2023-03-03T19:45:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f32b00682f0 0x7f32b006a7a0 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f32c40caa30 0x7f32c4148ac0 unknown :-1 s=CLOSED pgs=3586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f32c40c00e0 0x7f32c40c2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 --2- 172.21.15.7:0/1573184624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32c40c5c60 0x7f32c40c3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 -- 172.21.15.7:0/1573184624 >> 172.21.15.7:0/1573184624 conn(0x7f32c4066e40 msgr2=0x7f32c4153900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:45:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 -- 172.21.15.7:0/1573184624 shutdown_connections 2023-03-03T19:45:19.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:19.394+0000 7f32c9650700 1 -- 172.21.15.7:0/1573184624 wait complete. 2023-03-03T19:45:19.397 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:45:19.407 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-03-03T19:45:22.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:45:22.093 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:45:45.408 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:45:45.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.549+0000 7f8b1fc19700 1 Processor -- start 2023-03-03T19:45:45.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.550+0000 7f8b1fc19700 1 -- start start 2023-03-03T19:45:45.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.550+0000 7f8b1fc19700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8b18151cd0 0x7f8b18154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:45.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.550+0000 7f8b1fc19700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b1806c6c0 0x7f8b180736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:45.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.550+0000 7f8b1fc19700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b18157ff0 con 0x7f8b180c2e30 2023-03-03T19:45:45.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.550+0000 7f8b1fc19700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1806d640 con 0x7f8b1806c6c0 2023-03-03T19:45:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.550+0000 7f8b1fc19700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b1806cda0 con 0x7f8b18151cd0 2023-03-03T19:45:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1e1b6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b1806c6c0 0x7f8b180736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1e1b6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8b1806c6c0 0x7f8b180736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42938/0 (socket says 172.21.15.7:42938) 2023-03-03T19:45:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1e1b6700 1 -- 172.21.15.7:0/226814109 learned_addr learned my addr 172.21.15.7:0/226814109 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:45:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1d9b5700 1 --2- 172.21.15.7:0/226814109 >> v2:172.21.15.143:3300/0 conn(0x7f8b18151cd0 0x7f8b18154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:45.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1e1b6700 1 -- 172.21.15.7:0/226814109 >> v2:172.21.15.143:3300/0 conn(0x7f8b18151cd0 msgr2=0x7f8b18154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:45:45.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1e1b6700 1 --2- 172.21.15.7:0/226814109 >> v2:172.21.15.143:3300/0 conn(0x7f8b18151cd0 0x7f8b18154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:45.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1e1b6700 1 -- 172.21.15.7:0/226814109 >> v1:172.21.15.143:6789/0 conn(0x7f8b180c2e30 legacy=0x7f8b18158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:45:45.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1e1b6700 1 -- 172.21.15.7:0/226814109 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1806d080 con 0x7f8b1806c6c0 2023-03-03T19:45:45.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1e1b6700 1 --2- 172.21.15.7:0/226814109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 0x7f8b180736f0 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 crypto rx=0x7f8b1806f340 tx=0x7f8b1000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89e19f124fc8cd0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:45.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1c9b3700 1 -- 172.21.15.7:0/226814109 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1000e070 con 0x7f8b1806c6c0 2023-03-03T19:45:45.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1c9b3700 1 -- 172.21.15.7:0/226814109 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b10003a20 con 0x7f8b1806c6c0 2023-03-03T19:45:45.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.551+0000 7f8b1c9b3700 1 -- 172.21.15.7:0/226814109 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b10003dd0 con 0x7f8b1806c6c0 2023-03-03T19:45:45.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.552+0000 7f8b1fc19700 1 -- 172.21.15.7:0/226814109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 msgr2=0x7f8b180736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:45.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.552+0000 7f8b1fc19700 1 --2- 172.21.15.7:0/226814109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 0x7f8b180736f0 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 crypto rx=0x7f8b1806f340 tx=0x7f8b1000b330 comp rx=0 tx=0).stop 2023-03-03T19:45:45.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.552+0000 7f8b1fc19700 1 -- 172.21.15.7:0/226814109 shutdown_connections 2023-03-03T19:45:45.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.552+0000 7f8b1fc19700 1 --2- 172.21.15.7:0/226814109 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 0x7f8b180736f0 unknown :-1 s=CLOSED pgs=3587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:45.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.552+0000 7f8b1fc19700 1 --2- 172.21.15.7:0/226814109 >> v2:172.21.15.143:3300/0 conn(0x7f8b18151cd0 0x7f8b18154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:45.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.552+0000 7f8b1fc19700 1 -- 172.21.15.7:0/226814109 >> 172.21.15.7:0/226814109 conn(0x7f8b180a94d0 msgr2=0x7f8b180a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:45:45.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.552+0000 7f8b1fc19700 1 -- 172.21.15.7:0/226814109 shutdown_connections 2023-03-03T19:45:45.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.552+0000 7f8b1fc19700 1 -- 172.21.15.7:0/226814109 wait complete. 2023-03-03T19:45:45.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.553+0000 7f8b1fc19700 1 Processor -- start 2023-03-03T19:45:45.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.553+0000 7f8b1fc19700 1 -- start start 2023-03-03T19:45:45.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1fc19700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 0x7f8b181de3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:45.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1fc19700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b180c2e30 0x7f8b1814b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:45.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1fc19700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b18151cd0 0x7f8b1814bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:45.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1fc19700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1814cf00 con 0x7f8b1806c6c0 2023-03-03T19:45:45.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1fc19700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1814ab10 con 0x7f8b180c2e30 2023-03-03T19:45:45.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1fc19700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1814a380 con 0x7f8b18151cd0 2023-03-03T19:45:45.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1d9b5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 0x7f8b181de3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:45.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1d9b5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 0x7f8b181de3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42940/0 (socket says 172.21.15.7:42940) 2023-03-03T19:45:45.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1d9b5700 1 -- 172.21.15.7:0/854942750 learned_addr learned my addr 172.21.15.7:0/854942750 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:45:45.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1d1b4700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b180c2e30 0x7f8b1814b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:45.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1e1b6700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b18151cd0 0x7f8b1814bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:45.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1d9b5700 1 -- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b18151cd0 msgr2=0x7f8b1814bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:45.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1d9b5700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b18151cd0 0x7f8b1814bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:45.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1d9b5700 1 -- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b180c2e30 msgr2=0x7f8b1814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:45.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1d9b5700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b180c2e30 0x7f8b1814b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:45.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1d9b5700 1 -- 172.21.15.7:0/854942750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b10009b30 con 0x7f8b1806c6c0 2023-03-03T19:45:45.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b1d9b5700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 0x7f8b181de3f0 secure :-1 s=READY pgs=3588 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400d800 tx=0x7f8b1400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:45.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.554+0000 7f8b0a7fc700 1 -- 172.21.15.7:0/854942750 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b140125f0 con 0x7f8b1806c6c0 2023-03-03T19:45:45.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.555+0000 7f8b0a7fc700 1 -- 172.21.15.7:0/854942750 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b14012750 con 0x7f8b1806c6c0 2023-03-03T19:45:45.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.555+0000 7f8b1fc19700 1 -- 172.21.15.7:0/854942750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b18157c80 con 0x7f8b1806c6c0 2023-03-03T19:45:45.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.555+0000 7f8b0a7fc700 1 -- 172.21.15.7:0/854942750 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14012b00 con 0x7f8b1806c6c0 2023-03-03T19:45:45.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.555+0000 7f8b1fc19700 1 -- 172.21.15.7:0/854942750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1806d360 con 0x7f8b1806c6c0 2023-03-03T19:45:45.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.556+0000 7f8b1fc19700 1 -- 172.21.15.7:0/854942750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b18154a50 con 0x7f8b1806c6c0 2023-03-03T19:45:45.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.556+0000 7f8b0a7fc700 1 -- 172.21.15.7:0/854942750 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8b14012c60 con 0x7f8b1806c6c0 2023-03-03T19:45:45.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.556+0000 7f8b0a7fc700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8afc0682f0 0x7f8afc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:45.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.557+0000 7f8b0a7fc700 1 -- 172.21.15.7:0/854942750 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8b14086600 con 0x7f8b1806c6c0 2023-03-03T19:45:45.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.557+0000 7f8b1d1b4700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8afc0682f0 0x7f8afc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:45.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.557+0000 7f8b1d1b4700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8afc0682f0 0x7f8afc06a7a0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f8b1814af80 tx=0x7f8b0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:45.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.561+0000 7f8b0a7fc700 1 -- 172.21.15.7:0/854942750 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8b140529a0 con 0x7f8b1806c6c0 2023-03-03T19:45:45.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.800+0000 7f8b1fc19700 1 -- 172.21.15.7:0/854942750 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b1814cd70 con 0x7f8b1806c6c0 2023-03-03T19:45:45.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.800+0000 7f8b0a7fc700 1 -- 172.21.15.7:0/854942750 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8b14055fc0 con 0x7f8b1806c6c0 2023-03-03T19:45:45.802 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:45:45.802 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:45:45.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.802+0000 7f8b1fc19700 1 -- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8afc0682f0 msgr2=0x7f8afc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:45.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.802+0000 7f8b1fc19700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8afc0682f0 0x7f8afc06a7a0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f8b1814af80 tx=0x7f8b0c008040 comp rx=0 tx=0).stop 2023-03-03T19:45:45.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.802+0000 7f8b1fc19700 1 -- 172.21.15.7:0/854942750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 msgr2=0x7f8b181de3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:45.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.802+0000 7f8b1fc19700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 0x7f8b181de3f0 secure :-1 s=READY pgs=3588 cs=0 l=1 rev1=1 crypto rx=0x7f8b1400d800 tx=0x7f8b1400dbc0 comp rx=0 tx=0).stop 2023-03-03T19:45:45.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.802+0000 7f8b1fc19700 1 -- 172.21.15.7:0/854942750 shutdown_connections 2023-03-03T19:45:45.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.802+0000 7f8b1fc19700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8b18151cd0 0x7f8b1814bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:45.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.802+0000 7f8b1fc19700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8afc0682f0 0x7f8afc06a7a0 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:45.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.802+0000 7f8b1fc19700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8b1806c6c0 0x7f8b181de3f0 unknown :-1 s=CLOSED pgs=3588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:45.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.802+0000 7f8b1fc19700 1 --2- 172.21.15.7:0/854942750 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b180c2e30 0x7f8b1814b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:45.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.802+0000 7f8b1fc19700 1 -- 172.21.15.7:0/854942750 >> 172.21.15.7:0/854942750 conn(0x7f8b180a94d0 msgr2=0x7f8b1814d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:45:45.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.803+0000 7f8b1fc19700 1 -- 172.21.15.7:0/854942750 shutdown_connections 2023-03-03T19:45:45.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:45.803+0000 7f8b1fc19700 1 -- 172.21.15.7:0/854942750 wait complete. 2023-03-03T19:45:45.807 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:45:45.815 INFO:tasks.mds_thrash.fs.[a]:waiting for 13 secs before thrashing 2023-03-03T19:45:52.162 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:45:52.166 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:45:55.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:45:55.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.679+0000 7f166ac4e700 1 Processor -- start 2023-03-03T19:45:55.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.679+0000 7f166ac4e700 1 -- start start 2023-03-03T19:45:55.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.680+0000 7f166ac4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a2b0 0x7f166406a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:55.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.680+0000 7f166ac4e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16641558f0 con 0x7f16640a6370 2023-03-03T19:45:55.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.680+0000 7f166ac4e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640a6180 con 0x7f16640c5cb0 2023-03-03T19:45:55.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.680+0000 7f166ac4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166406bab0 con 0x7f166406a2b0 2023-03-03T19:45:55.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.680+0000 7f1663fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f16640a6370 0x7f1664155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36938/0 (socket says 172.21.15.7:36938) 2023-03-03T19:45:55.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.680+0000 7f1663fff700 1 -- 172.21.15.7:0/1033399211 learned_addr learned my addr 172.21.15.7:0/1033399211 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:45:55.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.680+0000 7f16689ea700 1 --2- 172.21.15.7:0/1033399211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a2b0 0x7f166406a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:55.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16637fe700 1 -- 172.21.15.7:0/1033399211 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332817816 0 0) 0x7f16641558f0 con 0x7f16640a6370 2023-03-03T19:45:55.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16637fe700 1 -- 172.21.15.7:0/1033399211 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16540035b0 con 0x7f16640a6370 2023-03-03T19:45:55.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16637fe700 1 -- 172.21.15.7:0/1033399211 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52818125 0 0) 0x7f16640a6180 con 0x7f16640c5cb0 2023-03-03T19:45:55.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16637fe700 1 -- 172.21.15.7:0/1033399211 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16641558f0 con 0x7f16640c5cb0 2023-03-03T19:45:55.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16637fe700 1 -- 172.21.15.7:0/1033399211 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2127625958 0 0) 0x7f16540035b0 con 0x7f16640a6370 2023-03-03T19:45:55.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16689ea700 1 -- 172.21.15.7:0/1033399211 >> v1:172.21.15.143:6789/0 conn(0x7f16640c5cb0 legacy=0x7f166415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:55.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16689ea700 1 -- 172.21.15.7:0/1033399211 >> v1:172.21.15.7:6789/0 conn(0x7f16640a6370 legacy=0x7f1664155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:55.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16689ea700 1 -- 172.21.15.7:0/1033399211 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640a5ea0 con 0x7f166406a2b0 2023-03-03T19:45:55.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16689ea700 1 --2- 172.21.15.7:0/1033399211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a2b0 0x7f166406a680 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1658009870 tx=0x7f1658009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51b2e97e2b3452d2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:55.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16637fe700 1 -- 172.21.15.7:0/1033399211 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1650001670 con 0x7f166406a2b0 2023-03-03T19:45:55.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16637fe700 1 -- 172.21.15.7:0/1033399211 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1658004030 con 0x7f166406a2b0 2023-03-03T19:45:55.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.681+0000 7f16637fe700 1 -- 172.21.15.7:0/1033399211 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16580043c0 con 0x7f166406a2b0 2023-03-03T19:45:55.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.682+0000 7f166ac4e700 1 -- 172.21.15.7:0/1033399211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a2b0 msgr2=0x7f166406a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:55.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.682+0000 7f166ac4e700 1 --2- 172.21.15.7:0/1033399211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a2b0 0x7f166406a680 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1658009870 tx=0x7f1658009b80 comp rx=0 tx=0).stop 2023-03-03T19:45:55.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.682+0000 7f166ac4e700 1 -- 172.21.15.7:0/1033399211 shutdown_connections 2023-03-03T19:45:55.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.682+0000 7f166ac4e700 1 --2- 172.21.15.7:0/1033399211 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f166406a2b0 0x7f166406a680 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:55.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.682+0000 7f166ac4e700 1 -- 172.21.15.7:0/1033399211 >> 172.21.15.7:0/1033399211 conn(0x7f16640a9430 msgr2=0x7f16640c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:45:55.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.682+0000 7f166ac4e700 1 -- 172.21.15.7:0/1033399211 shutdown_connections 2023-03-03T19:45:55.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.682+0000 7f166ac4e700 1 -- 172.21.15.7:0/1033399211 wait complete. 2023-03-03T19:45:55.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.683+0000 7f166ac4e700 1 Processor -- start 2023-03-03T19:45:55.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.683+0000 7f166ac4e700 1 -- start start 2023-03-03T19:45:55.688 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f166ac4e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f166406a2b0 0x7f166414cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:55.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f166ac4e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16640a6370 0x7f16641ea190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:55.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f166ac4e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640c5cb0 0x7f16641ea6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:55.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f166ac4e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1664149b00 con 0x7f166406a2b0 2023-03-03T19:45:55.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f166ac4e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16641eac10 con 0x7f16640a6370 2023-03-03T19:45:55.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f166ac4e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1664145b50 con 0x7f16640c5cb0 2023-03-03T19:45:55.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f16689ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f166406a2b0 0x7f166414cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:55.689 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f16689ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f166406a2b0 0x7f166414cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42964/0 (socket says 172.21.15.7:42964) 2023-03-03T19:45:55.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f16689ea700 1 -- 172.21.15.7:0/1061954009 learned_addr learned my addr 172.21.15.7:0/1061954009 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:45:55.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f1663fff700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16640a6370 0x7f16641ea190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:55.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f16691eb700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640c5cb0 0x7f16641ea6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:55.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f16689ea700 1 -- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640c5cb0 msgr2=0x7f16641ea6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:55.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f16689ea700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640c5cb0 0x7f16641ea6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:55.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f16689ea700 1 -- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16640a6370 msgr2=0x7f16641ea190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:55.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f16689ea700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16640a6370 0x7f16641ea190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:55.690 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f16689ea700 1 -- 172.21.15.7:0/1061954009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16580094a0 con 0x7f166406a2b0 2023-03-03T19:45:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f1663fff700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16640a6370 0x7f16641ea190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:45:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.684+0000 7f16689ea700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f166406a2b0 0x7f166414cc10 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 crypto rx=0x7f16580097c0 tx=0x7f1658004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.685+0000 7f16617fa700 1 -- 172.21.15.7:0/1061954009 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658003d70 con 0x7f166406a2b0 2023-03-03T19:45:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.685+0000 7f16617fa700 1 -- 172.21.15.7:0/1061954009 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165801f070 con 0x7f166406a2b0 2023-03-03T19:45:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.685+0000 7f166ac4e700 1 -- 172.21.15.7:0/1061954009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1664153160 con 0x7f166406a2b0 2023-03-03T19:45:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.685+0000 7f16617fa700 1 -- 172.21.15.7:0/1061954009 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658011870 con 0x7f166406a2b0 2023-03-03T19:45:55.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.685+0000 7f166ac4e700 1 -- 172.21.15.7:0/1061954009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1664152e80 con 0x7f166406a2b0 2023-03-03T19:45:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.686+0000 7f166ac4e700 1 -- 172.21.15.7:0/1061954009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16640c1500 con 0x7f166406a2b0 2023-03-03T19:45:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.686+0000 7f16617fa700 1 -- 172.21.15.7:0/1061954009 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f165802a020 con 0x7f166406a2b0 2023-03-03T19:45:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.686+0000 7f16617fa700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1648068340 0x7f164806a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.686+0000 7f16617fa700 1 -- 172.21.15.7:0/1061954009 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1658087650 con 0x7f166406a2b0 2023-03-03T19:45:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.687+0000 7f1663fff700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1648068340 0x7f164806a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:55.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.687+0000 7f1663fff700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1648068340 0x7f164806a7f0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f16500017a0 tx=0x7f1650001550 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:55.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.691+0000 7f16617fa700 1 -- 172.21.15.7:0/1061954009 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1658053a90 con 0x7f166406a2b0 2023-03-03T19:45:55.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.929+0000 7f166ac4e700 1 -- 172.21.15.7:0/1061954009 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16640a6be0 con 0x7f166406a2b0 2023-03-03T19:45:55.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.929+0000 7f16617fa700 1 -- 172.21.15.7:0/1061954009 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f16580570b0 con 0x7f166406a2b0 2023-03-03T19:45:55.930 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:45:55.930 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:45:55.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 -- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1648068340 msgr2=0x7f164806a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:55.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1648068340 0x7f164806a7f0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f16500017a0 tx=0x7f1650001550 comp rx=0 tx=0).stop 2023-03-03T19:45:55.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 -- 172.21.15.7:0/1061954009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f166406a2b0 msgr2=0x7f166414cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:55.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f166406a2b0 0x7f166414cc10 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 crypto rx=0x7f16580097c0 tx=0x7f1658004750 comp rx=0 tx=0).stop 2023-03-03T19:45:55.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 -- 172.21.15.7:0/1061954009 shutdown_connections 2023-03-03T19:45:55.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f16640c5cb0 0x7f16641ea6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:55.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1648068340 0x7f164806a7f0 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:55.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f166406a2b0 0x7f166414cc10 unknown :-1 s=CLOSED pgs=3590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:55.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 --2- 172.21.15.7:0/1061954009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16640a6370 0x7f16641ea190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:55.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 -- 172.21.15.7:0/1061954009 >> 172.21.15.7:0/1061954009 conn(0x7f16640a9430 msgr2=0x7f16640c3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:45:55.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 -- 172.21.15.7:0/1061954009 shutdown_connections 2023-03-03T19:45:55.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:55.931+0000 7f166ac4e700 1 -- 172.21.15.7:0/1061954009 wait complete. 2023-03-03T19:45:55.934 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:45:55.944 INFO:tasks.mds_thrash.fs.[b]:waiting for 88 secs before thrashing 2023-03-03T19:45:58.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:45:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.954+0000 7fd3554a4700 1 Processor -- start 2023-03-03T19:45:58.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.954+0000 7fd3554a4700 1 -- start start 2023-03-03T19:45:58.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.955+0000 7fd3554a4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fd3500aad20 0x7fd350157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:58.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.955+0000 7fd3554a4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3500aa030 0x7fd350158330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:58.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.955+0000 7fd3554a4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3500a9c20 con 0x7fd35014b0d0 2023-03-03T19:45:58.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.955+0000 7fd3554a4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd350154f10 con 0x7fd3500aad20 2023-03-03T19:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.955+0000 7fd3554a4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500ac3c0 con 0x7fd3500aa030 2023-03-03T19:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.955+0000 7fd34effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd35014b0d0 0x7fd350071800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36952/0 (socket says 172.21.15.7:36952) 2023-03-03T19:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.955+0000 7fd34effd700 1 -- 172.21.15.7:0/915101174 learned_addr learned my addr 172.21.15.7:0/915101174 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223098177 0 0) 0x7fd3500a9c20 con 0x7fd35014b0d0 2023-03-03T19:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34f7fe700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3500aa030 0x7fd350158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34e7fc700 1 --2- 172.21.15.7:0/915101174 >> v2:172.21.15.143:3300/0 conn(0x7fd3500aad20 0x7fd350157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:58.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3380035b0 con 0x7fd35014b0d0 2023-03-03T19:45:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3287086331 0 0) 0x7fd3380035b0 con 0x7fd35014b0d0 2023-03-03T19:45:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3500aa030 msgr2=0x7fd350158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:45:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3500aa030 0x7fd350158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 >> v2:172.21.15.143:3300/0 conn(0x7fd3500aad20 msgr2=0x7fd350157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:45:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 --2- 172.21.15.7:0/915101174 >> v2:172.21.15.143:3300/0 conn(0x7fd3500aad20 0x7fd350157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35006fc70 con 0x7fd35014b0d0 2023-03-03T19:45:58.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fd344001670 con 0x7fd35014b0d0 2023-03-03T19:45:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 >> v1:172.21.15.7:6789/0 conn(0x7fd35014b0d0 legacy=0x7fd350071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3500aa030 0x7fd338003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 0x7fd338004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd338004d90 0x7fd338005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd350154f10 con 0x7fd338003ff0 2023-03-03T19:45:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500ac3c0 con 0x7fd338004d90 2023-03-03T19:45:58.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33800bac0 con 0x7fd3500aa030 2023-03-03T19:45:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34e7fc700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 0x7fd338004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34effd700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3500aa030 0x7fd338003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34f7fe700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd338004d90 0x7fd338005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34e7fc700 1 -- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3500aa030 msgr2=0x7fd338003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:45:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34e7fc700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd3500aa030 0x7fd338003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34e7fc700 1 -- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd338004d90 msgr2=0x7fd338005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:45:58.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34e7fc700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd338004d90 0x7fd338005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:58.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.956+0000 7fd34e7fc700 1 -- 172.21.15.7:0/915101174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3380035b0 con 0x7fd338003ff0 2023-03-03T19:45:58.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd34e7fc700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 0x7fd338004850 secure :-1 s=READY pgs=3592 cs=0 l=1 rev1=1 crypto rx=0x7fd3500ab1a0 tx=0x7fd338005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84639d40be45c97c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:58.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34000e6c0 con 0x7fd338003ff0 2023-03-03T19:45:58.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34000e820 con 0x7fd338003ff0 2023-03-03T19:45:58.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd34dffb700 1 -- 172.21.15.7:0/915101174 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34000eb90 con 0x7fd338003ff0 2023-03-03T19:45:58.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd3554a4700 1 -- 172.21.15.7:0/915101174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 msgr2=0x7fd338004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:58.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd3554a4700 1 --2- 172.21.15.7:0/915101174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 0x7fd338004850 secure :-1 s=READY pgs=3592 cs=0 l=1 rev1=1 crypto rx=0x7fd3500ab1a0 tx=0x7fd338005880 comp rx=0 tx=0).stop 2023-03-03T19:45:58.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd34f7fe700 1 -- 172.21.15.7:0/915101174 reap_dead start 2023-03-03T19:45:58.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd3554a4700 1 -- 172.21.15.7:0/915101174 shutdown_connections 2023-03-03T19:45:58.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd3554a4700 1 -- 172.21.15.7:0/915101174 >> 172.21.15.7:0/915101174 conn(0x7fd3500c8900 msgr2=0x7fd3500c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:45:58.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd3554a4700 1 -- 172.21.15.7:0/915101174 shutdown_connections 2023-03-03T19:45:58.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.957+0000 7fd3554a4700 1 -- 172.21.15.7:0/915101174 wait complete. 2023-03-03T19:45:58.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.959+0000 7fd3554a4700 1 Processor -- start 2023-03-03T19:45:58.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.959+0000 7fd3554a4700 1 -- start start 2023-03-03T19:45:58.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.959+0000 7fd3554a4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 0x7fd350146330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:58.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.959+0000 7fd3554a4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3501439a0 0x7fd350143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:58.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.959+0000 7fd3554a4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd350144a20 0x7fd350144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:58.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.959+0000 7fd3554a4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500c5690 con 0x7fd338003ff0 2023-03-03T19:45:58.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.959+0000 7fd3554a4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd350145400 con 0x7fd3501439a0 2023-03-03T19:45:58.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.959+0000 7fd3554a4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd350155a20 con 0x7fd350144a20 2023-03-03T19:45:58.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 0x7fd350146330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:58.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 0x7fd350146330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42984/0 (socket says 172.21.15.7:42984) 2023-03-03T19:45:58.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34effd700 1 -- 172.21.15.7:0/3528952477 learned_addr learned my addr 172.21.15.7:0/3528952477 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:45:58.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34f7fe700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd350144a20 0x7fd350144ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:58.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34e7fc700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3501439a0 0x7fd350143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:58.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34effd700 1 -- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd350144a20 msgr2=0x7fd350144ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:58.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34effd700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd350144a20 0x7fd350144ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:58.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34effd700 1 -- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3501439a0 msgr2=0x7fd350143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:58.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34effd700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3501439a0 0x7fd350143e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:58.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34effd700 1 -- 172.21.15.7:0/3528952477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35006f6b0 con 0x7fd338003ff0 2023-03-03T19:45:58.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34f7fe700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd350144a20 0x7fd350144ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:45:58.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd34effd700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 0x7fd350146330 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 crypto rx=0x7fd344007340 tx=0x7fd344007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:58.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd3377fe700 1 -- 172.21.15.7:0/3528952477 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34400f6c0 con 0x7fd338003ff0 2023-03-03T19:45:58.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd3377fe700 1 -- 172.21.15.7:0/3528952477 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34400f840 con 0x7fd338003ff0 2023-03-03T19:45:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd3554a4700 1 -- 172.21.15.7:0/3528952477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd350155160 con 0x7fd338003ff0 2023-03-03T19:45:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd3377fe700 1 -- 172.21.15.7:0/3528952477 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34400fbf0 con 0x7fd338003ff0 2023-03-03T19:45:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.960+0000 7fd3554a4700 1 -- 172.21.15.7:0/3528952477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35006f990 con 0x7fd338003ff0 2023-03-03T19:45:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.961+0000 7fd3554a4700 1 -- 172.21.15.7:0/3528952477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd350074bb0 con 0x7fd338003ff0 2023-03-03T19:45:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.962+0000 7fd3377fe700 1 -- 172.21.15.7:0/3528952477 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd34400fd50 con 0x7fd338003ff0 2023-03-03T19:45:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.962+0000 7fd3377fe700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd33c0682f0 0x7fd33c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:45:58.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.962+0000 7fd3377fe700 1 -- 172.21.15.7:0/3528952477 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd344086280 con 0x7fd338003ff0 2023-03-03T19:45:58.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.962+0000 7fd34e7fc700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd33c0682f0 0x7fd33c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:45:58.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.963+0000 7fd34e7fc700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd33c0682f0 0x7fd33c06a7a0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fd3501443f0 tx=0x7fd340008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:45:58.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:58.966+0000 7fd3377fe700 1 -- 172.21.15.7:0/3528952477 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd344052620 con 0x7fd338003ff0 2023-03-03T19:45:59.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.203+0000 7fd3554a4700 1 -- 172.21.15.7:0/3528952477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd35006f490 con 0x7fd338003ff0 2023-03-03T19:45:59.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.203+0000 7fd3377fe700 1 -- 172.21.15.7:0/3528952477 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd344055c40 con 0x7fd338003ff0 2023-03-03T19:45:59.204 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:45:59.204 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:45:59.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.205+0000 7fd3554a4700 1 -- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd33c0682f0 msgr2=0x7fd33c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:59.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.205+0000 7fd3554a4700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd33c0682f0 0x7fd33c06a7a0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7fd3501443f0 tx=0x7fd340008030 comp rx=0 tx=0).stop 2023-03-03T19:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.205+0000 7fd3554a4700 1 -- 172.21.15.7:0/3528952477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 msgr2=0x7fd350146330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.205+0000 7fd3554a4700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 0x7fd350146330 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 crypto rx=0x7fd344007340 tx=0x7fd344007650 comp rx=0 tx=0).stop 2023-03-03T19:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.206+0000 7fd3554a4700 1 -- 172.21.15.7:0/3528952477 shutdown_connections 2023-03-03T19:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.206+0000 7fd3554a4700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd350144a20 0x7fd350144ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.206+0000 7fd3554a4700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd33c0682f0 0x7fd33c06a7a0 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.206+0000 7fd3554a4700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd338003ff0 0x7fd350146330 unknown :-1 s=CLOSED pgs=3593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:59.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.206+0000 7fd3554a4700 1 --2- 172.21.15.7:0/3528952477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3501439a0 0x7fd350143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:45:59.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.206+0000 7fd3554a4700 1 -- 172.21.15.7:0/3528952477 >> 172.21.15.7:0/3528952477 conn(0x7fd3500c8900 msgr2=0x7fd3500c3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:45:59.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.206+0000 7fd3554a4700 1 -- 172.21.15.7:0/3528952477 shutdown_connections 2023-03-03T19:45:59.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:45:59.206+0000 7fd3554a4700 1 -- 172.21.15.7:0/3528952477 wait complete. 2023-03-03T19:45:59.208 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:45:59.218 INFO:tasks.mds_thrash.fs.[a]:waiting for 98 secs before thrashing 2023-03-03T19:46:22.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:46:22.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:46:52.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:46:52.382 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:47:22.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:47:22.495 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:47:23.945 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:47:24.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.083+0000 7f6617c9a700 1 Processor -- start 2023-03-03T19:47:24.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.083+0000 7f6617c9a700 1 -- start start 2023-03-03T19:47:24.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.084+0000 7f6617c9a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6610151ab0 0x7f66100736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:24.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.084+0000 7f6617c9a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6610073c30 0x7f661006bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:24.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.084+0000 7f6617c9a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661006d780 con 0x7f661006c3e0 2023-03-03T19:47:24.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.084+0000 7f6617c9a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661006cb70 con 0x7f6610073c30 2023-03-03T19:47:24.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.084+0000 7f6617c9a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661006ce50 con 0x7f6610151ab0 2023-03-03T19:47:24.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.084+0000 7f6615235700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6610073c30 0x7f661006bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:24.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.084+0000 7f6616237700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f661006c3e0 0x7f6610157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37018/0 (socket says 172.21.15.7:37018) 2023-03-03T19:47:24.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.084+0000 7f6616237700 1 -- 172.21.15.7:0/727947439 learned_addr learned my addr 172.21.15.7:0/727947439 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:47:24.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.084+0000 7f6615a36700 1 --2- 172.21.15.7:0/727947439 >> v2:172.21.15.143:3300/0 conn(0x7f6610151ab0 0x7f66100736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:24.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6615235700 1 -- 172.21.15.7:0/727947439 >> v2:172.21.15.143:3300/0 conn(0x7f6610151ab0 msgr2=0x7f66100736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:47:24.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6615235700 1 --2- 172.21.15.7:0/727947439 >> v2:172.21.15.143:3300/0 conn(0x7f6610151ab0 0x7f66100736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:24.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6615235700 1 -- 172.21.15.7:0/727947439 >> v1:172.21.15.7:6789/0 conn(0x7f661006c3e0 legacy=0x7f6610157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6615235700 1 -- 172.21.15.7:0/727947439 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661006d130 con 0x7f6610073c30 2023-03-03T19:47:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6615235700 1 --2- 172.21.15.7:0/727947439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610073c30 0x7f661006bea0 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 crypto rx=0x7f660c009840 tx=0x7f660c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b0a35dbcae517 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6614a34700 1 -- 172.21.15.7:0/727947439 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c011830 con 0x7f6610073c30 2023-03-03T19:47:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6614a34700 1 -- 172.21.15.7:0/727947439 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c011990 con 0x7f6610073c30 2023-03-03T19:47:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6614a34700 1 -- 172.21.15.7:0/727947439 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c011d20 con 0x7f6610073c30 2023-03-03T19:47:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6617c9a700 1 -- 172.21.15.7:0/727947439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610073c30 msgr2=0x7f661006bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6617c9a700 1 --2- 172.21.15.7:0/727947439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610073c30 0x7f661006bea0 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 crypto rx=0x7f660c009840 tx=0x7f660c009b50 comp rx=0 tx=0).stop 2023-03-03T19:47:24.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6617c9a700 1 -- 172.21.15.7:0/727947439 shutdown_connections 2023-03-03T19:47:24.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6617c9a700 1 --2- 172.21.15.7:0/727947439 >> v2:172.21.15.143:3300/0 conn(0x7f6610151ab0 0x7f66100736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:24.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.085+0000 7f6617c9a700 1 --2- 172.21.15.7:0/727947439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610073c30 0x7f661006bea0 unknown :-1 s=CLOSED pgs=3595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:24.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.086+0000 7f6617c9a700 1 -- 172.21.15.7:0/727947439 >> 172.21.15.7:0/727947439 conn(0x7f66100a94d0 msgr2=0x7f66100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:47:24.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.086+0000 7f6617c9a700 1 -- 172.21.15.7:0/727947439 shutdown_connections 2023-03-03T19:47:24.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.086+0000 7f6617c9a700 1 -- 172.21.15.7:0/727947439 wait complete. 2023-03-03T19:47:24.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.087+0000 7f6617c9a700 1 Processor -- start 2023-03-03T19:47:24.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.087+0000 7f6617c9a700 1 -- start start 2023-03-03T19:47:24.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.087+0000 7f6617c9a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661006c3e0 0x7f661014c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:24.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.087+0000 7f6617c9a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6610073c30 0x7f661014ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:24.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.087+0000 7f6617c9a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610151ab0 0x7f6610145ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:24.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.087+0000 7f6617c9a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66100bf750 con 0x7f6610151ab0 2023-03-03T19:47:24.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.087+0000 7f6617c9a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66100c0e90 con 0x7f661006c3e0 2023-03-03T19:47:24.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.087+0000 7f6617c9a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661014c8b0 con 0x7f6610073c30 2023-03-03T19:47:24.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6616237700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610151ab0 0x7f6610145ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:24.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6616237700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610151ab0 0x7f6610145ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43044/0 (socket says 172.21.15.7:43044) 2023-03-03T19:47:24.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6616237700 1 -- 172.21.15.7:0/3501667683 learned_addr learned my addr 172.21.15.7:0/3501667683 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:47:24.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6615a36700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661006c3e0 0x7f661014c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:24.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6615235700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6610073c30 0x7f661014ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:24.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6616237700 1 -- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6610073c30 msgr2=0x7f661014ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:24.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6616237700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6610073c30 0x7f661014ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:24.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6616237700 1 -- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661006c3e0 msgr2=0x7f661014c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:24.093 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6616237700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661006c3e0 0x7f661014c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:24.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6616237700 1 -- 172.21.15.7:0/3501667683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c0094a0 con 0x7f6610151ab0 2023-03-03T19:47:24.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6615a36700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661006c3e0 0x7f661014c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:47:24.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6615235700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6610073c30 0x7f661014ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:47:24.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6616237700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610151ab0 0x7f6610145ba0 secure :-1 s=READY pgs=3596 cs=0 l=1 rev1=1 crypto rx=0x7f660800cbf0 tx=0x7f66080021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:24.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f66067fc700 1 -- 172.21.15.7:0/3501667683 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660800ab40 con 0x7f6610151ab0 2023-03-03T19:47:24.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f66067fc700 1 -- 172.21.15.7:0/3501667683 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660800aca0 con 0x7f6610151ab0 2023-03-03T19:47:24.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6617c9a700 1 -- 172.21.15.7:0/3501667683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6610151e80 con 0x7f6610151ab0 2023-03-03T19:47:24.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f66067fc700 1 -- 172.21.15.7:0/3501667683 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6608012600 con 0x7f6610151ab0 2023-03-03T19:47:24.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.088+0000 7f6617c9a700 1 -- 172.21.15.7:0/3501667683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661006d410 con 0x7f6610151ab0 2023-03-03T19:47:24.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.089+0000 7f6617c9a700 1 -- 172.21.15.7:0/3501667683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6610152010 con 0x7f6610151ab0 2023-03-03T19:47:24.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.090+0000 7f66067fc700 1 -- 172.21.15.7:0/3501667683 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6608019030 con 0x7f6610151ab0 2023-03-03T19:47:24.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.090+0000 7f66067fc700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65f40682f0 0x7f65f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:24.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.090+0000 7f66067fc700 1 -- 172.21.15.7:0/3501667683 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6608085c00 con 0x7f6610151ab0 2023-03-03T19:47:24.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.090+0000 7f6615a36700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65f40682f0 0x7f65f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.091+0000 7f6615a36700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65f40682f0 0x7f65f406a7a0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f6600007a50 tx=0x7f6600008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:24.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.095+0000 7f66067fc700 1 -- 172.21.15.7:0/3501667683 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f660804b5e0 con 0x7f6610151ab0 2023-03-03T19:47:24.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.336+0000 7f6617c9a700 1 -- 172.21.15.7:0/3501667683 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66100bf890 con 0x7f6610151ab0 2023-03-03T19:47:24.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.337+0000 7f66067fc700 1 -- 172.21.15.7:0/3501667683 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f66100bf890 con 0x7f6610151ab0 2023-03-03T19:47:24.337 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:47:24.338 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:47:24.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.338+0000 7f6617c9a700 1 -- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65f40682f0 msgr2=0x7f65f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:24.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.338+0000 7f6617c9a700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65f40682f0 0x7f65f406a7a0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f6600007a50 tx=0x7f6600008040 comp rx=0 tx=0).stop 2023-03-03T19:47:24.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.338+0000 7f6617c9a700 1 -- 172.21.15.7:0/3501667683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610151ab0 msgr2=0x7f6610145ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.338+0000 7f6617c9a700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610151ab0 0x7f6610145ba0 secure :-1 s=READY pgs=3596 cs=0 l=1 rev1=1 crypto rx=0x7f660800cbf0 tx=0x7f66080021c0 comp rx=0 tx=0).stop 2023-03-03T19:47:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.339+0000 7f6617c9a700 1 -- 172.21.15.7:0/3501667683 shutdown_connections 2023-03-03T19:47:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.339+0000 7f6617c9a700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f65f40682f0 0x7f65f406a7a0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.339+0000 7f6617c9a700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610151ab0 0x7f6610145ba0 unknown :-1 s=CLOSED pgs=3596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.339+0000 7f6617c9a700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661006c3e0 0x7f661014c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.339+0000 7f6617c9a700 1 --2- 172.21.15.7:0/3501667683 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6610073c30 0x7f661014ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.339+0000 7f6617c9a700 1 -- 172.21.15.7:0/3501667683 >> 172.21.15.7:0/3501667683 conn(0x7f66100a94d0 msgr2=0x7f6610156bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:47:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.339+0000 7f6617c9a700 1 -- 172.21.15.7:0/3501667683 shutdown_connections 2023-03-03T19:47:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:24.339+0000 7f6617c9a700 1 -- 172.21.15.7:0/3501667683 wait complete. 2023-03-03T19:47:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:47:24.351 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before thrashing 2023-03-03T19:47:29.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:47:29.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.490+0000 7fcdc4e37700 1 Processor -- start 2023-03-03T19:47:29.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.490+0000 7fcdc4e37700 1 -- start start 2023-03-03T19:47:29.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.492+0000 7fcdc4e37700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcdc00c2680 0x7fcdc01593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:29.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.492+0000 7fcdc4e37700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdc00a8bd0 0x7fcdc00aba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:29.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.492+0000 7fcdc4e37700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc00a9de0 con 0x7fcdc00abf90 2023-03-03T19:47:29.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.492+0000 7fcdc4e37700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc00aa5f0 con 0x7fcdc00c2680 2023-03-03T19:47:29.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.492+0000 7fcdc4e37700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc006b9b0 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.492+0000 7fcdbe59c700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcdc00c2680 0x7fcdc01593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:29.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.492+0000 7fcdbed9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcdc00abf90 0x7fcdc006c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37032/0 (socket says 172.21.15.7:37032) 2023-03-03T19:47:29.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.492+0000 7fcdbed9d700 1 -- 172.21.15.7:0/1747824363 learned_addr learned my addr 172.21.15.7:0/1747824363 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:47:29.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.492+0000 7fcdbdd9b700 1 --2- 172.21.15.7:0/1747824363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdc00a8bd0 0x7fcdc00aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:29.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.493+0000 7fcdbd59a700 1 -- 172.21.15.7:0/1747824363 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054042246 0 0) 0x7fcdc00a9de0 con 0x7fcdc00abf90 2023-03-03T19:47:29.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.493+0000 7fcdbd59a700 1 -- 172.21.15.7:0/1747824363 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcda80035b0 con 0x7fcdc00abf90 2023-03-03T19:47:29.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.493+0000 7fcdbe59c700 1 -- 172.21.15.7:0/1747824363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdc00a8bd0 msgr2=0x7fcdc00aba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:47:29.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.493+0000 7fcdbe59c700 1 --2- 172.21.15.7:0/1747824363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdc00a8bd0 0x7fcdc00aba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.493+0000 7fcdbe59c700 1 -- 172.21.15.7:0/1747824363 >> v1:172.21.15.7:6789/0 conn(0x7fcdc00abf90 legacy=0x7fcdc006c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.493+0000 7fcdbe59c700 1 -- 172.21.15.7:0/1747824363 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc00c2a50 con 0x7fcdc00c2680 2023-03-03T19:47:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.493+0000 7fcdbe59c700 1 --2- 172.21.15.7:0/1747824363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdc00c2680 0x7fcdc01593f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009840 tx=0x7fcdb0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be9bfa206a17bf9a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.493+0000 7fcdbd59a700 1 -- 172.21.15.7:0/1747824363 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0011830 con 0x7fcdc00c2680 2023-03-03T19:47:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.493+0000 7fcdbd59a700 1 -- 172.21.15.7:0/1747824363 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb0011990 con 0x7fcdc00c2680 2023-03-03T19:47:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.493+0000 7fcdbd59a700 1 -- 172.21.15.7:0/1747824363 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0011d20 con 0x7fcdc00c2680 2023-03-03T19:47:29.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.494+0000 7fcdc4e37700 1 -- 172.21.15.7:0/1747824363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdc00c2680 msgr2=0x7fcdc01593f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.494+0000 7fcdc4e37700 1 --2- 172.21.15.7:0/1747824363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdc00c2680 0x7fcdc01593f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009840 tx=0x7fcdb0009b50 comp rx=0 tx=0).stop 2023-03-03T19:47:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.494+0000 7fcdc4e37700 1 -- 172.21.15.7:0/1747824363 shutdown_connections 2023-03-03T19:47:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.494+0000 7fcdc4e37700 1 --2- 172.21.15.7:0/1747824363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdc00c2680 0x7fcdc01593f0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.494+0000 7fcdc4e37700 1 --2- 172.21.15.7:0/1747824363 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdc00a8bd0 0x7fcdc00aba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.494+0000 7fcdc4e37700 1 -- 172.21.15.7:0/1747824363 >> 172.21.15.7:0/1747824363 conn(0x7fcdc00c62e0 msgr2=0x7fcdc00c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:47:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.494+0000 7fcdc4e37700 1 -- 172.21.15.7:0/1747824363 shutdown_connections 2023-03-03T19:47:29.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.494+0000 7fcdc4e37700 1 -- 172.21.15.7:0/1747824363 wait complete. 2023-03-03T19:47:29.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.496+0000 7fcdc4e37700 1 Processor -- start 2023-03-03T19:47:29.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.496+0000 7fcdc4e37700 1 -- start start 2023-03-03T19:47:29.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdc4e37700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdc00a8bd0 0x7fcdc0152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdc4e37700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdc00abf90 0x7fcdc015b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdc4e37700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdc00c2680 0x7fcdc015be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdc4e37700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc00a3860 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdc4e37700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc01eba60 con 0x7fcdc00c2680 2023-03-03T19:47:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdc4e37700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc006f210 con 0x7fcdc00abf90 2023-03-03T19:47:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbe59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdc00a8bd0 0x7fcdc0152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:29.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbe59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdc00a8bd0 0x7fcdc0152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43056/0 (socket says 172.21.15.7:43056) 2023-03-03T19:47:29.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbe59c700 1 -- 172.21.15.7:0/344866585 learned_addr learned my addr 172.21.15.7:0/344866585 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:47:29.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbdd9b700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdc00abf90 0x7fcdc015b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:29.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbed9d700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdc00c2680 0x7fcdc015be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:29.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbe59c700 1 -- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdc00abf90 msgr2=0x7fcdc015b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:29.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbe59c700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdc00abf90 0x7fcdc015b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:29.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbe59c700 1 -- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdc00c2680 msgr2=0x7fcdc015be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:29.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbe59c700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdc00c2680 0x7fcdc015be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:29.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbe59c700 1 -- 172.21.15.7:0/344866585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb00094a0 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbed9d700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdc00c2680 0x7fcdc015be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:47:29.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdbe59c700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdc00a8bd0 0x7fcdc0152600 secure :-1 s=READY pgs=3598 cs=0 l=1 rev1=1 crypto rx=0x7fcdb00090b0 tx=0x7fcdb0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:29.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdaeffd700 1 -- 172.21.15.7:0/344866585 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0004400 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.497+0000 7fcdaeffd700 1 -- 172.21.15.7:0/344866585 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdb001f070 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.498+0000 7fcdc4e37700 1 -- 172.21.15.7:0/344866585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc00aa030 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.498+0000 7fcdaeffd700 1 -- 172.21.15.7:0/344866585 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb00038c0 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.498+0000 7fcdc4e37700 1 -- 172.21.15.7:0/344866585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc00a9b10 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.498+0000 7fcdc4e37700 1 -- 172.21.15.7:0/344866585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc00aa310 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.499+0000 7fcdaeffd700 1 -- 172.21.15.7:0/344866585 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcdb002a020 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.499+0000 7fcdaeffd700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcda00682f0 0x7fcda006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.499+0000 7fcdaeffd700 1 -- 172.21.15.7:0/344866585 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcdb0087530 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.500+0000 7fcdbdd9b700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcda00682f0 0x7fcda006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.500+0000 7fcdbdd9b700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcda00682f0 0x7fcda006a7a0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00c5ea0 tx=0x7fcdb4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:29.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.504+0000 7fcdaeffd700 1 -- 172.21.15.7:0/344866585 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcdb0053850 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.743+0000 7fcdc4e37700 1 -- 172.21.15.7:0/344866585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdc0074bb0 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.743+0000 7fcdaeffd700 1 -- 172.21.15.7:0/344866585 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcdb0056e70 con 0x7fcdc00a8bd0 2023-03-03T19:47:29.745 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:47:29.746 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:47:29.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 -- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcda00682f0 msgr2=0x7fcda006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcda00682f0 0x7fcda006a7a0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7fcdc00c5ea0 tx=0x7fcdb4009210 comp rx=0 tx=0).stop 2023-03-03T19:47:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 -- 172.21.15.7:0/344866585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdc00a8bd0 msgr2=0x7fcdc0152600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdc00a8bd0 0x7fcdc0152600 secure :-1 s=READY pgs=3598 cs=0 l=1 rev1=1 crypto rx=0x7fcdb00090b0 tx=0x7fcdb0004750 comp rx=0 tx=0).stop 2023-03-03T19:47:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 -- 172.21.15.7:0/344866585 shutdown_connections 2023-03-03T19:47:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdc00c2680 0x7fcdc015be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcda00682f0 0x7fcda006a7a0 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdc00a8bd0 0x7fcdc0152600 unknown :-1 s=CLOSED pgs=3598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 --2- 172.21.15.7:0/344866585 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcdc00abf90 0x7fcdc015b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 -- 172.21.15.7:0/344866585 >> 172.21.15.7:0/344866585 conn(0x7fcdc00c62e0 msgr2=0x7fcdc01575b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:47:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 -- 172.21.15.7:0/344866585 shutdown_connections 2023-03-03T19:47:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:29.746+0000 7fcdc4e37700 1 -- 172.21.15.7:0/344866585 wait complete. 2023-03-03T19:47:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:47:29.759 INFO:tasks.mds_thrash.fs.[b]:waiting for 54 secs before thrashing 2023-03-03T19:47:37.219 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:47:37.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.357+0000 7fbc67327700 1 Processor -- start 2023-03-03T19:47:37.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.357+0000 7fbc67327700 1 -- start start 2023-03-03T19:47:37.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc67327700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbc6014f9d0 0x7fbc60156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:37.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc67327700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbc600abe20 0x7fbc601529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:37.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc67327700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600c2f30 con 0x7fbc6014fda0 2023-03-03T19:47:37.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc67327700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc600c31d0 con 0x7fbc6014f9d0 2023-03-03T19:47:37.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc67327700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc601524a0 con 0x7fbc600abe20 2023-03-03T19:47:37.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc650c3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbc6014f9d0 0x7fbc60156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:37.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc648c2700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbc6014fda0 0x7fbc60071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37046/0 (socket says 172.21.15.7:37046) 2023-03-03T19:47:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc648c2700 1 -- 172.21.15.7:0/2463928888 learned_addr learned my addr 172.21.15.7:0/2463928888 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:47:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc650c3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbc6014f9d0 0x7fbc60156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43064/0 (socket says 172.21.15.7:43064) 2023-03-03T19:47:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc658c4700 1 --2- 172.21.15.7:0/2463928888 >> v2:172.21.15.143:3300/0 conn(0x7fbc600abe20 0x7fbc601529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc650c3700 1 -- 172.21.15.7:0/2463928888 >> v2:172.21.15.143:3300/0 conn(0x7fbc600abe20 msgr2=0x7fbc601529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:47:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc650c3700 1 --2- 172.21.15.7:0/2463928888 >> v2:172.21.15.143:3300/0 conn(0x7fbc600abe20 0x7fbc601529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc650c3700 1 -- 172.21.15.7:0/2463928888 >> v1:172.21.15.7:6789/0 conn(0x7fbc6014fda0 legacy=0x7fbc60071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:37.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc650c3700 1 -- 172.21.15.7:0/2463928888 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600c34b0 con 0x7fbc6014f9d0 2023-03-03T19:47:37.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc57fff700 1 -- 172.21.15.7:0/2463928888 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492134157 0 0) 0x7fbc600c2f30 con 0x7fbc6014fda0 2023-03-03T19:47:37.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc658c4700 1 --2- 172.21.15.7:0/2463928888 >> v2:172.21.15.143:3300/0 conn(0x7fbc600abe20 0x7fbc601529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:47:37.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.358+0000 7fbc650c3700 1 --2- 172.21.15.7:0/2463928888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 0x7fbc60156430 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7fbc60071230 tx=0x7fbc50009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a29f21bcf045e9fc server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:37.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc57fff700 1 -- 172.21.15.7:0/2463928888 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50011830 con 0x7fbc6014f9d0 2023-03-03T19:47:37.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc57fff700 1 -- 172.21.15.7:0/2463928888 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc50011990 con 0x7fbc6014f9d0 2023-03-03T19:47:37.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc57fff700 1 -- 172.21.15.7:0/2463928888 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50011d20 con 0x7fbc6014f9d0 2023-03-03T19:47:37.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc67327700 1 -- 172.21.15.7:0/2463928888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 msgr2=0x7fbc60156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc67327700 1 --2- 172.21.15.7:0/2463928888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 0x7fbc60156430 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7fbc60071230 tx=0x7fbc50009b20 comp rx=0 tx=0).stop 2023-03-03T19:47:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc67327700 1 -- 172.21.15.7:0/2463928888 shutdown_connections 2023-03-03T19:47:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc67327700 1 --2- 172.21.15.7:0/2463928888 >> v2:172.21.15.143:3300/0 conn(0x7fbc600abe20 0x7fbc601529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc67327700 1 --2- 172.21.15.7:0/2463928888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 0x7fbc60156430 unknown :-1 s=CLOSED pgs=3600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc67327700 1 -- 172.21.15.7:0/2463928888 >> 172.21.15.7:0/2463928888 conn(0x7fbc600c8900 msgr2=0x7fbc600c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:47:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc67327700 1 -- 172.21.15.7:0/2463928888 shutdown_connections 2023-03-03T19:47:37.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.359+0000 7fbc67327700 1 -- 172.21.15.7:0/2463928888 wait complete. 2023-03-03T19:47:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.360+0000 7fbc67327700 1 Processor -- start 2023-03-03T19:47:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc67327700 1 -- start start 2023-03-03T19:47:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc67327700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc600abe20 0x7fbc6006e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc67327700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 0x7fbc601dbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc67327700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc6014fda0 0x7fbc601dcb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:37.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc67327700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600c1c80 con 0x7fbc6014f9d0 2023-03-03T19:47:37.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc67327700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6006e430 con 0x7fbc6014fda0 2023-03-03T19:47:37.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc67327700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600c1ed0 con 0x7fbc600abe20 2023-03-03T19:47:37.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc648c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 0x7fbc601dbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:37.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc648c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 0x7fbc601dbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43072/0 (socket says 172.21.15.7:43072) 2023-03-03T19:47:37.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc648c2700 1 -- 172.21.15.7:0/2085703547 learned_addr learned my addr 172.21.15.7:0/2085703547 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:47:37.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc658c4700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc6014fda0 0x7fbc601dcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.361+0000 7fbc650c3700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc600abe20 0x7fbc6006e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc648c2700 1 -- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc600abe20 msgr2=0x7fbc6006e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc648c2700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc600abe20 0x7fbc6006e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc648c2700 1 -- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc6014fda0 msgr2=0x7fbc601dcb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc648c2700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc6014fda0 0x7fbc601dcb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc648c2700 1 -- 172.21.15.7:0/2085703547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc500094a0 con 0x7fbc6014f9d0 2023-03-03T19:47:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc648c2700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 0x7fbc601dbba0 secure :-1 s=READY pgs=3601 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00c540 tx=0x7fbc5c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:37.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc55ffb700 1 -- 172.21.15.7:0/2085703547 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c004c80 con 0x7fbc6014f9d0 2023-03-03T19:47:37.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc55ffb700 1 -- 172.21.15.7:0/2085703547 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c004de0 con 0x7fbc6014f9d0 2023-03-03T19:47:37.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc55ffb700 1 -- 172.21.15.7:0/2085703547 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c007310 con 0x7fbc6014f9d0 2023-03-03T19:47:37.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc67327700 1 -- 172.21.15.7:0/2085703547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc600ac350 con 0x7fbc6014f9d0 2023-03-03T19:47:37.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.362+0000 7fbc67327700 1 -- 172.21.15.7:0/2085703547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc600ab630 con 0x7fbc6014f9d0 2023-03-03T19:47:37.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.363+0000 7fbc67327700 1 -- 172.21.15.7:0/2085703547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc60071fa0 con 0x7fbc6014f9d0 2023-03-03T19:47:37.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.363+0000 7fbc55ffb700 1 -- 172.21.15.7:0/2085703547 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbc5c01a020 con 0x7fbc6014f9d0 2023-03-03T19:47:37.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.364+0000 7fbc55ffb700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbc440682c0 0x7fbc4406a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.364+0000 7fbc55ffb700 1 -- 172.21.15.7:0/2085703547 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbc5c085d90 con 0x7fbc6014f9d0 2023-03-03T19:47:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.364+0000 7fbc650c3700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbc440682c0 0x7fbc4406a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.365+0000 7fbc650c3700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbc440682c0 0x7fbc4406a770 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7fbc500090b0 tx=0x7fbc50008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:37.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.368+0000 7fbc55ffb700 1 -- 172.21.15.7:0/2085703547 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbc5c0521e0 con 0x7fbc6014f9d0 2023-03-03T19:47:37.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.604+0000 7fbc67327700 1 -- 172.21.15.7:0/2085703547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc6006e5a0 con 0x7fbc6014f9d0 2023-03-03T19:47:37.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.604+0000 7fbc55ffb700 1 -- 172.21.15.7:0/2085703547 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbc5c055800 con 0x7fbc6014f9d0 2023-03-03T19:47:37.606 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:47:37.606 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:47:37.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.606+0000 7fbc67327700 1 -- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbc440682c0 msgr2=0x7fbc4406a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:37.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.606+0000 7fbc67327700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbc440682c0 0x7fbc4406a770 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7fbc500090b0 tx=0x7fbc50008030 comp rx=0 tx=0).stop 2023-03-03T19:47:37.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.606+0000 7fbc67327700 1 -- 172.21.15.7:0/2085703547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 msgr2=0x7fbc601dbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:37.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.606+0000 7fbc67327700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 0x7fbc601dbba0 secure :-1 s=READY pgs=3601 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00c540 tx=0x7fbc5c00c900 comp rx=0 tx=0).stop 2023-03-03T19:47:37.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.607+0000 7fbc67327700 1 -- 172.21.15.7:0/2085703547 shutdown_connections 2023-03-03T19:47:37.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.607+0000 7fbc67327700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbc6014fda0 0x7fbc601dcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:37.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.607+0000 7fbc67327700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbc600abe20 0x7fbc6006e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:37.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.607+0000 7fbc67327700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbc440682c0 0x7fbc4406a770 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:37.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.607+0000 7fbc67327700 1 --2- 172.21.15.7:0/2085703547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc6014f9d0 0x7fbc601dbba0 unknown :-1 s=CLOSED pgs=3601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:37.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.607+0000 7fbc67327700 1 -- 172.21.15.7:0/2085703547 >> 172.21.15.7:0/2085703547 conn(0x7fbc600c8900 msgr2=0x7fbc60155470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:47:37.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.607+0000 7fbc67327700 1 -- 172.21.15.7:0/2085703547 shutdown_connections 2023-03-03T19:47:37.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:37.607+0000 7fbc67327700 1 -- 172.21.15.7:0/2085703547 wait complete. 2023-03-03T19:47:37.610 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:47:37.620 INFO:tasks.mds_thrash.fs.[a]:waiting for 12 secs before thrashing 2023-03-03T19:47:49.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:47:49.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.760+0000 7f5048d06700 1 Processor -- start 2023-03-03T19:47:49.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.760+0000 7f5048d06700 1 -- start start 2023-03-03T19:47:49.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.761+0000 7f5048d06700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f504414fda0 0x7f5044156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:49.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.761+0000 7f5048d06700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50440abe20 0x7f50441529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:49.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.761+0000 7f5048d06700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50440c2f30 con 0x7f504414f9d0 2023-03-03T19:47:49.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.761+0000 7f5048d06700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50440c31d0 con 0x7f50440abe20 2023-03-03T19:47:49.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.761+0000 7f5048d06700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50441524a0 con 0x7f504414fda0 2023-03-03T19:47:49.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.761+0000 7f5042d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50440abe20 0x7f50441529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:49.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.761+0000 7f504259c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f504414f9d0 0x7f5044071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37070/0 (socket says 172.21.15.7:37070) 2023-03-03T19:47:49.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.761+0000 7f504259c700 1 -- 172.21.15.7:0/3011004866 learned_addr learned my addr 172.21.15.7:0/3011004866 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:47:49.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.761+0000 7f5041d9b700 1 --2- 172.21.15.7:0/3011004866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f504414fda0 0x7f5044156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:49.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f5042d9d700 1 -- 172.21.15.7:0/3011004866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f504414fda0 msgr2=0x7f5044156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:47:49.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f5042d9d700 1 --2- 172.21.15.7:0/3011004866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f504414fda0 0x7f5044156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:49.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f5042d9d700 1 -- 172.21.15.7:0/3011004866 >> v1:172.21.15.7:6789/0 conn(0x7f504414f9d0 legacy=0x7f5044071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:49.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f504159a700 1 -- 172.21.15.7:0/3011004866 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452407122 0 0) 0x7f50440c2f30 con 0x7f504414f9d0 2023-03-03T19:47:49.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f5042d9d700 1 -- 172.21.15.7:0/3011004866 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50440c34b0 con 0x7f50440abe20 2023-03-03T19:47:49.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f5042d9d700 1 --2- 172.21.15.7:0/3011004866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50440abe20 0x7f50441529c0 secure :-1 s=READY pgs=3603 cs=0 l=1 rev1=1 crypto rx=0x7f503c009d90 tx=0x7f503c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd4edc5a9396f8a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:49.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f504159a700 1 -- 172.21.15.7:0/3011004866 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c00e070 con 0x7f50440abe20 2023-03-03T19:47:49.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f504159a700 1 -- 172.21.15.7:0/3011004866 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c00bb20 con 0x7f50440abe20 2023-03-03T19:47:49.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f504159a700 1 -- 172.21.15.7:0/3011004866 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c012430 con 0x7f50440abe20 2023-03-03T19:47:49.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f5048d06700 1 -- 172.21.15.7:0/3011004866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50440abe20 msgr2=0x7f50441529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:49.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.762+0000 7f5048d06700 1 --2- 172.21.15.7:0/3011004866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50440abe20 0x7f50441529c0 secure :-1 s=READY pgs=3603 cs=0 l=1 rev1=1 crypto rx=0x7f503c009d90 tx=0x7f503c00b330 comp rx=0 tx=0).stop 2023-03-03T19:47:49.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.763+0000 7f5048d06700 1 -- 172.21.15.7:0/3011004866 shutdown_connections 2023-03-03T19:47:49.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.763+0000 7f5048d06700 1 --2- 172.21.15.7:0/3011004866 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50440abe20 0x7f50441529c0 unknown :-1 s=CLOSED pgs=3603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:49.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.763+0000 7f5048d06700 1 --2- 172.21.15.7:0/3011004866 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f504414fda0 0x7f5044156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:49.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.763+0000 7f5048d06700 1 -- 172.21.15.7:0/3011004866 >> 172.21.15.7:0/3011004866 conn(0x7f50440c8900 msgr2=0x7f50440c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:47:49.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.763+0000 7f5048d06700 1 -- 172.21.15.7:0/3011004866 shutdown_connections 2023-03-03T19:47:49.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.763+0000 7f5048d06700 1 -- 172.21.15.7:0/3011004866 wait complete. 2023-03-03T19:47:49.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.764+0000 7f5048d06700 1 Processor -- start 2023-03-03T19:47:49.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.764+0000 7f5048d06700 1 -- start start 2023-03-03T19:47:49.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5048d06700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50440abe20 0x7f504406e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:49.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5048d06700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f504414f9d0 0x7f50441e9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:49.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5048d06700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f504414fda0 0x7f50440c1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:49.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5048d06700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504406ee90 con 0x7f504414fda0 2023-03-03T19:47:49.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5048d06700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504406e530 con 0x7f504414f9d0 2023-03-03T19:47:49.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5048d06700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50441ecc30 con 0x7f50440abe20 2023-03-03T19:47:49.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5042d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f504414fda0 0x7f50440c1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:49.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5042d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f504414fda0 0x7f50440c1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43100/0 (socket says 172.21.15.7:43100) 2023-03-03T19:47:49.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5042d9d700 1 -- 172.21.15.7:0/1225489507 learned_addr learned my addr 172.21.15.7:0/1225489507 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:47:49.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f504259c700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50440abe20 0x7f504406e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:49.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5041d9b700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f504414f9d0 0x7f50441e9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:49.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5042d9d700 1 -- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50440abe20 msgr2=0x7f504406e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:49.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5042d9d700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50440abe20 0x7f504406e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:49.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5042d9d700 1 -- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f504414f9d0 msgr2=0x7f50441e9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:49.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5042d9d700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f504414f9d0 0x7f50441e9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:49.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5042d9d700 1 -- 172.21.15.7:0/1225489507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c009a70 con 0x7f504414fda0 2023-03-03T19:47:49.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f504259c700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50440abe20 0x7f504406e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:47:49.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5041d9b700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f504414f9d0 0x7f50441e9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:47:49.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.765+0000 7f5042d9d700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f504414fda0 0x7f50440c1a80 secure :-1 s=READY pgs=3604 cs=0 l=1 rev1=1 crypto rx=0x7f503c00b680 tx=0x7f503c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:49.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.766+0000 7f5032ffd700 1 -- 172.21.15.7:0/1225489507 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c012430 con 0x7f504414fda0 2023-03-03T19:47:49.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.766+0000 7f5032ffd700 1 -- 172.21.15.7:0/1225489507 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c020070 con 0x7f504414fda0 2023-03-03T19:47:49.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.766+0000 7f5048d06700 1 -- 172.21.15.7:0/1225489507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50440ac350 con 0x7f504414fda0 2023-03-03T19:47:49.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.766+0000 7f5032ffd700 1 -- 172.21.15.7:0/1225489507 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c004280 con 0x7f504414fda0 2023-03-03T19:47:49.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.766+0000 7f5048d06700 1 -- 172.21.15.7:0/1225489507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50440ab630 con 0x7f504414fda0 2023-03-03T19:47:49.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.767+0000 7f5048d06700 1 -- 172.21.15.7:0/1225489507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5044071fa0 con 0x7f504414fda0 2023-03-03T19:47:49.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.767+0000 7f5032ffd700 1 -- 172.21.15.7:0/1225489507 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f503c0043e0 con 0x7f504414fda0 2023-03-03T19:47:49.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.768+0000 7f5032ffd700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50240682f0 0x7f502406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:47:49.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.768+0000 7f5032ffd700 1 -- 172.21.15.7:0/1225489507 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f503c022070 con 0x7f504414fda0 2023-03-03T19:47:49.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.768+0000 7f504259c700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50240682f0 0x7f502406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:47:49.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.769+0000 7f504259c700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50240682f0 0x7f502406a7a0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f5038000f60 tx=0x7f5038009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:47:49.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:49.772+0000 7f5032ffd700 1 -- 172.21.15.7:0/1225489507 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f503c054c90 con 0x7f504414fda0 2023-03-03T19:47:50.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.010+0000 7f5048d06700 1 -- 172.21.15.7:0/1225489507 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5044074bb0 con 0x7f504414fda0 2023-03-03T19:47:50.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.010+0000 7f5032ffd700 1 -- 172.21.15.7:0/1225489507 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f503c0263e0 con 0x7f504414fda0 2023-03-03T19:47:50.011 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:47:50.011 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:47:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 -- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50240682f0 msgr2=0x7f502406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50240682f0 0x7f502406a7a0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f5038000f60 tx=0x7f5038009040 comp rx=0 tx=0).stop 2023-03-03T19:47:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 -- 172.21.15.7:0/1225489507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f504414fda0 msgr2=0x7f50440c1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:47:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f504414fda0 0x7f50440c1a80 secure :-1 s=READY pgs=3604 cs=0 l=1 rev1=1 crypto rx=0x7f503c00b680 tx=0x7f503c004ad0 comp rx=0 tx=0).stop 2023-03-03T19:47:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 -- 172.21.15.7:0/1225489507 shutdown_connections 2023-03-03T19:47:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f50240682f0 0x7f502406a7a0 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f504414fda0 0x7f50440c1a80 unknown :-1 s=CLOSED pgs=3604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f50440abe20 0x7f504406e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 --2- 172.21.15.7:0/1225489507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f504414f9d0 0x7f50441e9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:47:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 -- 172.21.15.7:0/1225489507 >> 172.21.15.7:0/1225489507 conn(0x7f50440c8900 msgr2=0x7f504414efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:47:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 -- 172.21.15.7:0/1225489507 shutdown_connections 2023-03-03T19:47:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:47:50.012+0000 7f5048d06700 1 -- 172.21.15.7:0/1225489507 wait complete. 2023-03-03T19:47:50.015 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:47:50.025 INFO:tasks.mds_thrash.fs.[a]:waiting for 44 secs before thrashing 2023-03-03T19:47:52.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:47:52.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:48:22.735 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:48:22.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:48:23.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:48:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.897+0000 7f47c34d4700 1 Processor -- start 2023-03-03T19:48:23.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.898+0000 7f47c34d4700 1 -- start start 2023-03-03T19:48:23.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.898+0000 7f47c34d4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f47bc151770 0x7f47bc151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:23.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.898+0000 7f47c34d4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47bc154910 con 0x7f47bc155390 2023-03-03T19:48:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.898+0000 7f47c34d4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47bc0c4540 con 0x7f47bc06e020 2023-03-03T19:48:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.898+0000 7f47c34d4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47bc070b90 con 0x7f47bc151770 2023-03-03T19:48:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47c1270700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f47bc155390 0x7f47bc0c3e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37112/0 (socket says 172.21.15.7:37112) 2023-03-03T19:48:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47c1270700 1 -- 172.21.15.7:0/330724711 learned_addr learned my addr 172.21.15.7:0/330724711 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:48:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092201233 0 0) 0x7f47bc154910 con 0x7f47bc155390 2023-03-03T19:48:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47a80035b0 con 0x7f47bc155390 2023-03-03T19:48:23.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47c1a71700 1 --2- 172.21.15.7:0/330724711 >> v2:172.21.15.143:3300/0 conn(0x7f47bc151770 0x7f47bc151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:23.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3977051844 0 0) 0x7f47a80035b0 con 0x7f47bc155390 2023-03-03T19:48:23.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 >> v2:172.21.15.143:3300/0 conn(0x7f47bc151770 msgr2=0x7f47bc151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:48:23.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 --2- 172.21.15.7:0/330724711 >> v2:172.21.15.143:3300/0 conn(0x7f47bc151770 0x7f47bc151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:23.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 >> v1:172.21.15.143:6789/0 conn(0x7f47bc06e020 legacy=0x7f47bc071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:23.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc0702a0 con 0x7f47bc155390 2023-03-03T19:48:23.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f47b8001670 con 0x7f47bc155390 2023-03-03T19:48:23.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 >> v1:172.21.15.7:6789/0 conn(0x7f47bc155390 legacy=0x7f47bc0c3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:23.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47a8003a70 0x7f47a8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:23.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a8004850 0x7f47a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:23.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47a8005250 0x7f47a800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:23.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc070b90 con 0x7f47a8004850 2023-03-03T19:48:23.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a800c190 con 0x7f47a8003a70 2023-03-03T19:48:23.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a800c350 con 0x7f47a8005250 2023-03-03T19:48:23.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47c0a6f700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a8004850 0x7f47a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:23.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.899+0000 7f47c1270700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47a8003a70 0x7f47a8003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47c1a71700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47a8005250 0x7f47a800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47c0a6f700 1 -- 172.21.15.7:0/330724711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47a8005250 msgr2=0x7f47a800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:48:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47c0a6f700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47a8005250 0x7f47a800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47c0a6f700 1 -- 172.21.15.7:0/330724711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47a8003a70 msgr2=0x7f47a8003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:48:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47c0a6f700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47a8003a70 0x7f47a8003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47c0a6f700 1 -- 172.21.15.7:0/330724711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a80035b0 con 0x7f47a8004850 2023-03-03T19:48:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47c1a71700 1 -- 172.21.15.7:0/330724711 reap_dead start 2023-03-03T19:48:23.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47c1270700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47a8003a70 0x7f47a8003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:48:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47c0a6f700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a8004850 0x7f47a8004cc0 secure :-1 s=READY pgs=3606 cs=0 l=1 rev1=1 crypto rx=0x7f47b0007fd0 tx=0x7f47a8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0fecadd6b7d4c9 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:48:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b00052d0 con 0x7f47a8004850 2023-03-03T19:48:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b0005430 con 0x7f47a8004850 2023-03-03T19:48:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.900+0000 7f47affff700 1 -- 172.21.15.7:0/330724711 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b00038c0 con 0x7f47a8004850 2023-03-03T19:48:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.901+0000 7f47c34d4700 1 -- 172.21.15.7:0/330724711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a8004850 msgr2=0x7f47a8004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.901+0000 7f47c34d4700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a8004850 0x7f47a8004cc0 secure :-1 s=READY pgs=3606 cs=0 l=1 rev1=1 crypto rx=0x7f47b0007fd0 tx=0x7f47a8005880 comp rx=0 tx=0).stop 2023-03-03T19:48:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.901+0000 7f47c34d4700 1 -- 172.21.15.7:0/330724711 shutdown_connections 2023-03-03T19:48:23.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.901+0000 7f47c34d4700 1 --2- 172.21.15.7:0/330724711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a8004850 0x7f47a8004cc0 unknown :-1 s=CLOSED pgs=3606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:23.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.901+0000 7f47c34d4700 1 -- 172.21.15.7:0/330724711 >> 172.21.15.7:0/330724711 conn(0x7f47bc06c110 msgr2=0x7f47bc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:48:23.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.901+0000 7f47c34d4700 1 -- 172.21.15.7:0/330724711 shutdown_connections 2023-03-03T19:48:23.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.901+0000 7f47c34d4700 1 -- 172.21.15.7:0/330724711 wait complete. 2023-03-03T19:48:23.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.902+0000 7f47c34d4700 1 Processor -- start 2023-03-03T19:48:23.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.902+0000 7f47c34d4700 1 -- start start 2023-03-03T19:48:23.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c34d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47a8004850 0x7f47bc0abdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:23.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c34d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47bc0c6c00 0x7f47bc13dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:23.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c34d4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc0c7070 0x7f47bc13e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:23.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c34d4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc1e86a0 con 0x7f47bc0c7070 2023-03-03T19:48:23.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c34d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc13e680 con 0x7f47bc0c6c00 2023-03-03T19:48:23.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c34d4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc1e7a20 con 0x7f47a8004850 2023-03-03T19:48:23.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c1a71700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc0c7070 0x7f47bc13e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:23.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c1a71700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc0c7070 0x7f47bc13e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43148/0 (socket says 172.21.15.7:43148) 2023-03-03T19:48:23.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c1a71700 1 -- 172.21.15.7:0/3175987485 learned_addr learned my addr 172.21.15.7:0/3175987485 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:48:23.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c0a6f700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47bc0c6c00 0x7f47bc13dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:23.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c1270700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47a8004850 0x7f47bc0abdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:23.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c1a71700 1 -- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47a8004850 msgr2=0x7f47bc0abdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:23.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c1a71700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47a8004850 0x7f47bc0abdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:23.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c1a71700 1 -- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47bc0c6c00 msgr2=0x7f47bc13dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:23.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c1a71700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47bc0c6c00 0x7f47bc13dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:23.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c1a71700 1 -- 172.21.15.7:0/3175987485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc070820 con 0x7f47bc0c7070 2023-03-03T19:48:23.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c0a6f700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47bc0c6c00 0x7f47bc13dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:48:23.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47c1a71700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc0c7070 0x7f47bc13e140 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 crypto rx=0x7f47bc14eee0 tx=0x7f47b40114b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:48:23.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47adffb700 1 -- 172.21.15.7:0/3175987485 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b400e070 con 0x7f47bc0c7070 2023-03-03T19:48:23.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.903+0000 7f47adffb700 1 -- 172.21.15.7:0/3175987485 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b4003c30 con 0x7f47bc0c7070 2023-03-03T19:48:23.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.904+0000 7f47c34d4700 1 -- 172.21.15.7:0/3175987485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc070540 con 0x7f47bc0c7070 2023-03-03T19:48:23.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.904+0000 7f47adffb700 1 -- 172.21.15.7:0/3175987485 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4003de0 con 0x7f47bc0c7070 2023-03-03T19:48:23.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.904+0000 7f47c34d4700 1 -- 172.21.15.7:0/3175987485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc155760 con 0x7f47bc0c7070 2023-03-03T19:48:23.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.905+0000 7f47c34d4700 1 -- 172.21.15.7:0/3175987485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc074bb0 con 0x7f47bc0c7070 2023-03-03T19:48:23.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.905+0000 7f47adffb700 1 -- 172.21.15.7:0/3175987485 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f47b4011990 con 0x7f47bc0c7070 2023-03-03T19:48:23.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.906+0000 7f47adffb700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47a00682f0 0x7f47a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:23.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.906+0000 7f47adffb700 1 -- 172.21.15.7:0/3175987485 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f47b4086790 con 0x7f47bc0c7070 2023-03-03T19:48:23.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.906+0000 7f47c1270700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47a00682f0 0x7f47a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:23.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.906+0000 7f47c1270700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47a00682f0 0x7f47a006a7a0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f47b8007fd0 tx=0x7f47b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:48:23.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:23.910+0000 7f47adffb700 1 -- 172.21.15.7:0/3175987485 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47b4052a40 con 0x7f47bc0c7070 2023-03-03T19:48:24.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.152+0000 7f47c34d4700 1 -- 172.21.15.7:0/3175987485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47bc0c1da0 con 0x7f47bc0c7070 2023-03-03T19:48:24.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.152+0000 7f47adffb700 1 -- 172.21.15.7:0/3175987485 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f47b4056060 con 0x7f47bc0c7070 2023-03-03T19:48:24.153 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:48:24.153 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:48:24.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 -- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47a00682f0 msgr2=0x7f47a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:24.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47a00682f0 0x7f47a006a7a0 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f47b8007fd0 tx=0x7f47b8009040 comp rx=0 tx=0).stop 2023-03-03T19:48:24.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 -- 172.21.15.7:0/3175987485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc0c7070 msgr2=0x7f47bc13e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:24.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc0c7070 0x7f47bc13e140 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 crypto rx=0x7f47bc14eee0 tx=0x7f47b40114b0 comp rx=0 tx=0).stop 2023-03-03T19:48:24.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 -- 172.21.15.7:0/3175987485 shutdown_connections 2023-03-03T19:48:24.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47a00682f0 0x7f47a006a7a0 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:24.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47bc0c7070 0x7f47bc13e140 unknown :-1 s=CLOSED pgs=3607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:24.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47a8004850 0x7f47bc0abdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:24.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 --2- 172.21.15.7:0/3175987485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47bc0c6c00 0x7f47bc13dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:24.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 -- 172.21.15.7:0/3175987485 >> 172.21.15.7:0/3175987485 conn(0x7f47bc06c110 msgr2=0x7f47bc150e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:48:24.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.154+0000 7f47c34d4700 1 -- 172.21.15.7:0/3175987485 shutdown_connections 2023-03-03T19:48:24.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:24.155+0000 7f47c34d4700 1 -- 172.21.15.7:0/3175987485 wait complete. 2023-03-03T19:48:24.157 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:48:24.167 INFO:tasks.mds_thrash.fs.[b]:waiting for 25 secs before thrashing 2023-03-03T19:48:34.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:48:34.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.164+0000 7f4dcd7b3700 1 Processor -- start 2023-03-03T19:48:34.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.164+0000 7f4dcd7b3700 1 -- start start 2023-03-03T19:48:34.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dcd7b3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4dc80caa30 0x7f4dc80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:34.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dcd7b3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4dc80c00e0 0x7f4dc8154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:34.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dcd7b3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc80c7310 con 0x7f4dc80cae00 2023-03-03T19:48:34.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dcd7b3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc80cb240 con 0x7f4dc80caa30 2023-03-03T19:48:34.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dcd7b3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dc80bfd10 con 0x7f4dc80c00e0 2023-03-03T19:48:34.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4dc80caa30 0x7f4dc80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:34.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc77fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4dc80cae00 0x7f4dc80c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37138/0 (socket says 172.21.15.7:37138) 2023-03-03T19:48:34.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc77fe700 1 -- 172.21.15.7:0/2502147933 learned_addr learned my addr 172.21.15.7:0/2502147933 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:48:34.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc6ffd700 1 --2- 172.21.15.7:0/2502147933 >> v2:172.21.15.7:3300/0 conn(0x7f4dc80caa30 0x7f4dc80c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43154/0 (socket says 172.21.15.7:43154) 2023-03-03T19:48:34.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc67fc700 1 --2- 172.21.15.7:0/2502147933 >> v2:172.21.15.143:3300/0 conn(0x7f4dc80c00e0 0x7f4dc8154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:34.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc6ffd700 1 -- 172.21.15.7:0/2502147933 >> v2:172.21.15.143:3300/0 conn(0x7f4dc80c00e0 msgr2=0x7f4dc8154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:48:34.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc6ffd700 1 --2- 172.21.15.7:0/2502147933 >> v2:172.21.15.143:3300/0 conn(0x7f4dc80c00e0 0x7f4dc8154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:34.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc6ffd700 1 -- 172.21.15.7:0/2502147933 >> v1:172.21.15.7:6789/0 conn(0x7f4dc80cae00 legacy=0x7f4dc80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:34.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc6ffd700 1 -- 172.21.15.7:0/2502147933 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc80bfa30 con 0x7f4dc80caa30 2023-03-03T19:48:34.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc5ffb700 1 -- 172.21.15.7:0/2502147933 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192101913 0 0) 0x7f4dc80c7310 con 0x7f4dc80cae00 2023-03-03T19:48:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.165+0000 7f4dc6ffd700 1 --2- 172.21.15.7:0/2502147933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80caa30 0x7f4dc80c5c60 secure :-1 s=READY pgs=3609 cs=0 l=1 rev1=1 crypto rx=0x7f4dc80ca910 tx=0x7f4dbc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a9a47772360c49 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:48:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dc5ffb700 1 -- 172.21.15.7:0/2502147933 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc011830 con 0x7f4dc80caa30 2023-03-03T19:48:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dc5ffb700 1 -- 172.21.15.7:0/2502147933 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc011990 con 0x7f4dc80caa30 2023-03-03T19:48:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dc5ffb700 1 -- 172.21.15.7:0/2502147933 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc011d20 con 0x7f4dc80caa30 2023-03-03T19:48:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/2502147933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80caa30 msgr2=0x7f4dc80c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dcd7b3700 1 --2- 172.21.15.7:0/2502147933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80caa30 0x7f4dc80c5c60 secure :-1 s=READY pgs=3609 cs=0 l=1 rev1=1 crypto rx=0x7f4dc80ca910 tx=0x7f4dbc009b20 comp rx=0 tx=0).stop 2023-03-03T19:48:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/2502147933 shutdown_connections 2023-03-03T19:48:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dcd7b3700 1 --2- 172.21.15.7:0/2502147933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80caa30 0x7f4dc80c5c60 unknown :-1 s=CLOSED pgs=3609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dcd7b3700 1 --2- 172.21.15.7:0/2502147933 >> v2:172.21.15.143:3300/0 conn(0x7f4dc80c00e0 0x7f4dc8154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:34.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/2502147933 >> 172.21.15.7:0/2502147933 conn(0x7f4dc8066e40 msgr2=0x7f4dc8071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:48:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/2502147933 shutdown_connections 2023-03-03T19:48:34.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.166+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/2502147933 wait complete. 2023-03-03T19:48:34.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.168+0000 7f4dcd7b3700 1 Processor -- start 2023-03-03T19:48:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.168+0000 7f4dcd7b3700 1 -- start start 2023-03-03T19:48:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.168+0000 7f4dcd7b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dc80c00e0 0x7f4dc81ddcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.168+0000 7f4dcd7b3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc80caa30 0x7f4dc813f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:34.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.168+0000 7f4dcd7b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80cae00 0x7f4dc80c2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.168+0000 7f4dcd7b3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc81de550 con 0x7f4dc80cae00 2023-03-03T19:48:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.168+0000 7f4dcd7b3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc81df520 con 0x7f4dc80caa30 2023-03-03T19:48:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.168+0000 7f4dcd7b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc81563d0 con 0x7f4dc80c00e0 2023-03-03T19:48:34.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80cae00 0x7f4dc80c2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80cae00 0x7f4dc80c2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43164/0 (socket says 172.21.15.7:43164) 2023-03-03T19:48:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc77fe700 1 -- 172.21.15.7:0/3254783988 learned_addr learned my addr 172.21.15.7:0/3254783988 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:48:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc6ffd700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dc80c00e0 0x7f4dc81ddcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:34.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc67fc700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc80caa30 0x7f4dc813f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc77fe700 1 -- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dc80c00e0 msgr2=0x7f4dc81ddcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc77fe700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dc80c00e0 0x7f4dc81ddcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc77fe700 1 -- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc80caa30 msgr2=0x7f4dc813f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:34.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc77fe700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc80caa30 0x7f4dc813f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc77fe700 1 -- 172.21.15.7:0/3254783988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc0094a0 con 0x7f4dc80cae00 2023-03-03T19:48:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dc77fe700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80cae00 0x7f4dc80c2b60 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000cb60 tx=0x7f4dc000ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:48:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4daf7fe700 1 -- 172.21.15.7:0/3254783988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc000a9e0 con 0x7f4dc80cae00 2023-03-03T19:48:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4daf7fe700 1 -- 172.21.15.7:0/3254783988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc000ab40 con 0x7f4dc80cae00 2023-03-03T19:48:34.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/3254783988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc81547f0 con 0x7f4dc80cae00 2023-03-03T19:48:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4daf7fe700 1 -- 172.21.15.7:0/3254783988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc00070d0 con 0x7f4dc80cae00 2023-03-03T19:48:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.169+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/3254783988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc8154510 con 0x7f4dc80cae00 2023-03-03T19:48:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.170+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/3254783988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc8154ad0 con 0x7f4dc80cae00 2023-03-03T19:48:34.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.171+0000 7f4daf7fe700 1 -- 172.21.15.7:0/3254783988 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4dc001a030 con 0x7f4dc80cae00 2023-03-03T19:48:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.171+0000 7f4daf7fe700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4db40682f0 0x7f4db406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.171+0000 7f4daf7fe700 1 -- 172.21.15.7:0/3254783988 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4dc0085d00 con 0x7f4dc80cae00 2023-03-03T19:48:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.171+0000 7f4dc6ffd700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4db40682f0 0x7f4db406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.172+0000 7f4dc6ffd700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4db40682f0 0x7f4db406a7a0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0090b0 tx=0x7f4dbc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:48:34.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.176+0000 7f4daf7fe700 1 -- 172.21.15.7:0/3254783988 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4dc0089030 con 0x7f4dc80cae00 2023-03-03T19:48:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.413+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/3254783988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dc81588a0 con 0x7f4dc80cae00 2023-03-03T19:48:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.414+0000 7f4daf7fe700 1 -- 172.21.15.7:0/3254783988 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4dc0018090 con 0x7f4dc80cae00 2023-03-03T19:48:34.416 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:48:34.416 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:48:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4db40682f0 msgr2=0x7f4db406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4db40682f0 0x7f4db406a7a0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0090b0 tx=0x7f4dbc008030 comp rx=0 tx=0).stop 2023-03-03T19:48:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/3254783988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80cae00 msgr2=0x7f4dc80c2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80cae00 0x7f4dc80c2b60 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000cb60 tx=0x7f4dc000ce70 comp rx=0 tx=0).stop 2023-03-03T19:48:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/3254783988 shutdown_connections 2023-03-03T19:48:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f4db40682f0 0x7f4db406a7a0 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4dc80cae00 0x7f4dc80c2b60 unknown :-1 s=CLOSED pgs=3610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4dc80c00e0 0x7f4dc81ddcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 --2- 172.21.15.7:0/3254783988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc80caa30 0x7f4dc813f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/3254783988 >> 172.21.15.7:0/3254783988 conn(0x7f4dc8066e40 msgr2=0x7f4dc8065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:48:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/3254783988 shutdown_connections 2023-03-03T19:48:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:34.416+0000 7f4dcd7b3700 1 -- 172.21.15.7:0/3254783988 wait complete. 2023-03-03T19:48:34.420 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:48:34.429 INFO:tasks.mds_thrash.fs.[a]:waiting for 102 secs before thrashing 2023-03-03T19:48:49.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:48:49.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.307+0000 7f52b8804700 1 Processor -- start 2023-03-03T19:48:49.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.307+0000 7f52b8804700 1 -- start start 2023-03-03T19:48:49.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.308+0000 7f52b8804700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f52b4071e80 0x7f52b4151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:49.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.308+0000 7f52b8804700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b4072250 0x7f52b414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:49.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.308+0000 7f52b8804700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b41510d0 con 0x7f52b40c1ce0 2023-03-03T19:48:49.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.308+0000 7f52b8804700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b40c3740 con 0x7f52b4071e80 2023-03-03T19:48:49.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.308+0000 7f52b8804700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b4070670 con 0x7f52b4072250 2023-03-03T19:48:49.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.309+0000 7f52b2d9d700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f52b40c1ce0 0x7f52b41546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:40676/0 (socket says 172.21.15.7:40676) 2023-03-03T19:48:49.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.309+0000 7f52b2d9d700 1 -- 172.21.15.7:0/1831809538 learned_addr learned my addr 172.21.15.7:0/1831809538 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:48:49.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.309+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389820613 0 0) 0x7f52b41510d0 con 0x7f52b40c1ce0 2023-03-03T19:48:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.309+0000 7f52b1d9b700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b4072250 0x7f52b414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.309+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a00035b0 con 0x7f52b40c1ce0 2023-03-03T19:48:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.309+0000 7f52b259c700 1 --2- 172.21.15.7:0/1831809538 >> v2:172.21.15.143:3300/0 conn(0x7f52b4071e80 0x7f52b4151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 <== mon.1 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1522988347 0 0) 0x7f52a00035b0 con 0x7f52b40c1ce0 2023-03-03T19:48:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b4072250 msgr2=0x7f52b414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:48:49.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b4072250 0x7f52b414e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 >> v2:172.21.15.143:3300/0 conn(0x7f52b4071e80 msgr2=0x7f52b4151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:48:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 --2- 172.21.15.7:0/1831809538 >> v2:172.21.15.143:3300/0 conn(0x7f52b4071e80 0x7f52b4151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b40c39e0 con 0x7f52b40c1ce0 2023-03-03T19:48:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 <== mon.1 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f52ac0019f0 con 0x7f52b40c1ce0 2023-03-03T19:48:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 >> v1:172.21.15.143:6789/0 conn(0x7f52b40c1ce0 legacy=0x7f52b41546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:49.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a0003a70 0x7f52a0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52a0004850 0x7f52a0004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b4072250 0x7f52a0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b40c3740 con 0x7f52a0004850 2023-03-03T19:48:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b4070670 con 0x7f52a0003a70 2023-03-03T19:48:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a000ba40 con 0x7f52b4072250 2023-03-03T19:48:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b1d9b700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52a0004850 0x7f52a0004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b259c700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a0003a70 0x7f52a0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:49.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b2d9d700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b4072250 0x7f52a0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:49.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b1d9b700 1 -- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b4072250 msgr2=0x7f52a0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:48:49.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b1d9b700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b4072250 0x7f52a0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b1d9b700 1 -- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a0003a70 msgr2=0x7f52a0003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:48:49.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b1d9b700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a0003a70 0x7f52a0003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b1d9b700 1 -- 172.21.15.7:0/1831809538 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a00035b0 con 0x7f52a0004850 2023-03-03T19:48:49.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.310+0000 7f52b1d9b700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52a0004850 0x7f52a0004cc0 secure :-1 s=READY pgs=3611 cs=0 l=1 rev1=1 crypto rx=0x7f529c009b30 tx=0x7f52a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d00a57e76e053f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:48:49.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.311+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c00e6c0 con 0x7f52a0004850 2023-03-03T19:48:49.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.311+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529c00e820 con 0x7f52a0004850 2023-03-03T19:48:49.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.311+0000 7f52b159a700 1 -- 172.21.15.7:0/1831809538 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c00eb90 con 0x7f52a0004850 2023-03-03T19:48:49.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.311+0000 7f52b8804700 1 -- 172.21.15.7:0/1831809538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52a0004850 msgr2=0x7f52a0004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:49.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.311+0000 7f52b8804700 1 --2- 172.21.15.7:0/1831809538 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52a0004850 0x7f52a0004cc0 secure :-1 s=READY pgs=3611 cs=0 l=1 rev1=1 crypto rx=0x7f529c009b30 tx=0x7f52a0005880 comp rx=0 tx=0).stop 2023-03-03T19:48:49.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.311+0000 7f52b2d9d700 1 -- 172.21.15.7:0/1831809538 reap_dead start 2023-03-03T19:48:49.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.311+0000 7f52b8804700 1 -- 172.21.15.7:0/1831809538 shutdown_connections 2023-03-03T19:48:49.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.311+0000 7f52b8804700 1 -- 172.21.15.7:0/1831809538 >> 172.21.15.7:0/1831809538 conn(0x7f52b40a94d0 msgr2=0x7f52b40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:48:49.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.311+0000 7f52b8804700 1 -- 172.21.15.7:0/1831809538 shutdown_connections 2023-03-03T19:48:49.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.311+0000 7f52b8804700 1 -- 172.21.15.7:0/1831809538 wait complete. 2023-03-03T19:48:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b8804700 1 Processor -- start 2023-03-03T19:48:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b8804700 1 -- start start 2023-03-03T19:48:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b8804700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a0004850 0x7f52b40c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b8804700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b414f950 0x7f52b4144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b8804700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b40c6780 0x7f52b4144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b8804700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b41dab70 con 0x7f52b414f950 2023-03-03T19:48:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b8804700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b4144da0 con 0x7f52a0004850 2023-03-03T19:48:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b8804700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b41506f0 con 0x7f52b40c6780 2023-03-03T19:48:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b414f950 0x7f52b4144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b1d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b414f950 0x7f52b4144320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43186/0 (socket says 172.21.15.7:43186) 2023-03-03T19:48:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.313+0000 7f52b1d9b700 1 -- 172.21.15.7:0/3493511441 learned_addr learned my addr 172.21.15.7:0/3493511441 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:48:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52b2d9d700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b40c6780 0x7f52b4144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52b259c700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a0004850 0x7f52b40c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:49.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52b1d9b700 1 -- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b40c6780 msgr2=0x7f52b4144860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52b1d9b700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b40c6780 0x7f52b4144860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52b1d9b700 1 -- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a0004850 msgr2=0x7f52b40c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52b1d9b700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a0004850 0x7f52b40c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52b1d9b700 1 -- 172.21.15.7:0/3493511441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b406c9a0 con 0x7f52b414f950 2023-03-03T19:48:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52b1d9b700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b414f950 0x7f52b4144320 secure :-1 s=READY pgs=3612 cs=0 l=1 rev1=1 crypto rx=0x7f529c00e4b0 tx=0x7f529c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:48:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52aaffd700 1 -- 172.21.15.7:0/3493511441 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c00e6c0 con 0x7f52b414f950 2023-03-03T19:48:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52aaffd700 1 -- 172.21.15.7:0/3493511441 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529c013b60 con 0x7f52b414f950 2023-03-03T19:48:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52b8804700 1 -- 172.21.15.7:0/3493511441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b4151300 con 0x7f52b414f950 2023-03-03T19:48:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52aaffd700 1 -- 172.21.15.7:0/3493511441 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c01c450 con 0x7f52b414f950 2023-03-03T19:48:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.314+0000 7f52b8804700 1 -- 172.21.15.7:0/3493511441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b406cb30 con 0x7f52b414f950 2023-03-03T19:48:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.315+0000 7f52b8804700 1 -- 172.21.15.7:0/3493511441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b41dae50 con 0x7f52b414f950 2023-03-03T19:48:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.316+0000 7f52aaffd700 1 -- 172.21.15.7:0/3493511441 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f529c019030 con 0x7f52b414f950 2023-03-03T19:48:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.316+0000 7f52aaffd700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5294068320 0x7f529406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:48:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.316+0000 7f52aaffd700 1 -- 172.21.15.7:0/3493511441 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f529c086d30 con 0x7f52b414f950 2023-03-03T19:48:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.316+0000 7f52b259c700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5294068320 0x7f529406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:48:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.317+0000 7f52b259c700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5294068320 0x7f529406a7d0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009c60 tx=0x7f52a4009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:48:49.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.321+0000 7f52aaffd700 1 -- 172.21.15.7:0/3493511441 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f529c053050 con 0x7f52b414f950 2023-03-03T19:48:49.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.560+0000 7f52b8804700 1 -- 172.21.15.7:0/3493511441 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52b4074bb0 con 0x7f52b414f950 2023-03-03T19:48:49.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.561+0000 7f52aaffd700 1 -- 172.21.15.7:0/3493511441 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f529c056670 con 0x7f52b414f950 2023-03-03T19:48:49.561 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:48:49.562 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:48:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.562+0000 7f52b8804700 1 -- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5294068320 msgr2=0x7f529406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.562+0000 7f52b8804700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5294068320 0x7f529406a7d0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009c60 tx=0x7f52a4009210 comp rx=0 tx=0).stop 2023-03-03T19:48:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.562+0000 7f52b8804700 1 -- 172.21.15.7:0/3493511441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b414f950 msgr2=0x7f52b4144320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:48:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.562+0000 7f52b8804700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b414f950 0x7f52b4144320 secure :-1 s=READY pgs=3612 cs=0 l=1 rev1=1 crypto rx=0x7f529c00e4b0 tx=0x7f529c0047c0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.563+0000 7f52b8804700 1 -- 172.21.15.7:0/3493511441 shutdown_connections 2023-03-03T19:48:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.563+0000 7f52b8804700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f52b40c6780 0x7f52b4144860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.563+0000 7f52b8804700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f52a0004850 0x7f52b40c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.563+0000 7f52b8804700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5294068320 0x7f529406a7d0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.563+0000 7f52b8804700 1 --2- 172.21.15.7:0/3493511441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52b414f950 0x7f52b4144320 unknown :-1 s=CLOSED pgs=3612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:48:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.563+0000 7f52b8804700 1 -- 172.21.15.7:0/3493511441 >> 172.21.15.7:0/3493511441 conn(0x7f52b40a94d0 msgr2=0x7f52b4151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:48:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.563+0000 7f52b8804700 1 -- 172.21.15.7:0/3493511441 shutdown_connections 2023-03-03T19:48:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:48:49.563+0000 7f52b8804700 1 -- 172.21.15.7:0/3493511441 wait complete. 2023-03-03T19:48:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:48:49.576 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T19:48:52.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:48:52.803 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:49:22.873 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:49:22.877 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:49:51.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:49:51.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.715+0000 7fddd5c91700 1 Processor -- start 2023-03-03T19:49:51.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.715+0000 7fddd5c91700 1 -- start start 2023-03-03T19:49:51.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.716+0000 7fddd5c91700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fddd0152e30 0x7fddd00c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:49:51.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.716+0000 7fddd5c91700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd00beab0 0x7fddd00c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:49:51.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.716+0000 7fddd5c91700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd01523b0 con 0x7fddd0155d80 2023-03-03T19:49:51.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.716+0000 7fddd5c91700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd0156930 con 0x7fddd0152e30 2023-03-03T19:49:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.716+0000 7fddd5c91700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd0072010 con 0x7fddd00beab0 2023-03-03T19:49:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.716+0000 7fddcf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fddd0152e30 0x7fddd00c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:49:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.716+0000 7fddcf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fddd0152e30 0x7fddd00c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43228/0 (socket says 172.21.15.7:43228) 2023-03-03T19:49:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.716+0000 7fddcf7fe700 1 -- 172.21.15.7:0/2858387418 learned_addr learned my addr 172.21.15.7:0/2858387418 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:49:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.717+0000 7fddceffd700 1 --2- 172.21.15.7:0/2858387418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd00beab0 0x7fddd00c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:49:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.717+0000 7fddcf7fe700 1 -- 172.21.15.7:0/2858387418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd00beab0 msgr2=0x7fddd00c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:49:51.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.717+0000 7fddcf7fe700 1 --2- 172.21.15.7:0/2858387418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd00beab0 0x7fddd00c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:49:51.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.717+0000 7fddcf7fe700 1 -- 172.21.15.7:0/2858387418 >> v1:172.21.15.143:6789/0 conn(0x7fddd0155d80 legacy=0x7fddd0156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:49:51.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.717+0000 7fddcf7fe700 1 -- 172.21.15.7:0/2858387418 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd00aaaf0 con 0x7fddd0152e30 2023-03-03T19:49:51.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.717+0000 7fddcf7fe700 1 --2- 172.21.15.7:0/2858387418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd0152e30 0x7fddd00c43d0 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7fddc00098f0 tx=0x7fddc0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402be4ae82b6ae5d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:49:51.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.717+0000 7fddce7fc700 1 -- 172.21.15.7:0/2858387418 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc000b8a0 con 0x7fddd0152e30 2023-03-03T19:49:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.717+0000 7fddce7fc700 1 -- 172.21.15.7:0/2858387418 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc000ba00 con 0x7fddd0152e30 2023-03-03T19:49:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.717+0000 7fddce7fc700 1 -- 172.21.15.7:0/2858387418 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc000bd90 con 0x7fddd0152e30 2023-03-03T19:49:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.718+0000 7fddd5c91700 1 -- 172.21.15.7:0/2858387418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd0152e30 msgr2=0x7fddd00c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:49:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.718+0000 7fddd5c91700 1 --2- 172.21.15.7:0/2858387418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd0152e30 0x7fddd00c43d0 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7fddc00098f0 tx=0x7fddc0009c00 comp rx=0 tx=0).stop 2023-03-03T19:49:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.718+0000 7fddd5c91700 1 -- 172.21.15.7:0/2858387418 shutdown_connections 2023-03-03T19:49:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.718+0000 7fddd5c91700 1 --2- 172.21.15.7:0/2858387418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd0152e30 0x7fddd00c43d0 unknown :-1 s=CLOSED pgs=3613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:49:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.718+0000 7fddd5c91700 1 --2- 172.21.15.7:0/2858387418 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd00beab0 0x7fddd00c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:49:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.718+0000 7fddd5c91700 1 -- 172.21.15.7:0/2858387418 >> 172.21.15.7:0/2858387418 conn(0x7fddd00736f0 msgr2=0x7fddd0073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:49:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.718+0000 7fddd5c91700 1 -- 172.21.15.7:0/2858387418 shutdown_connections 2023-03-03T19:49:51.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.718+0000 7fddd5c91700 1 -- 172.21.15.7:0/2858387418 wait complete. 2023-03-03T19:49:51.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.719+0000 7fddd5c91700 1 Processor -- start 2023-03-03T19:49:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.719+0000 7fddd5c91700 1 -- start start 2023-03-03T19:49:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddd5c91700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd00beab0 0x7fddd01ea3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:49:51.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddd5c91700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd0152e30 0x7fddd00c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:49:51.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddd5c91700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd0155d80 0x7fddd006ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:49:51.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddd5c91700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd01ea930 con 0x7fddd00beab0 2023-03-03T19:49:51.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddd5c91700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd006bbf0 con 0x7fddd0152e30 2023-03-03T19:49:51.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddd5c91700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd006bed0 con 0x7fddd0155d80 2023-03-03T19:49:51.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddcf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd00beab0 0x7fddd01ea3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:49:51.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddcf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd00beab0 0x7fddd01ea3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43232/0 (socket says 172.21.15.7:43232) 2023-03-03T19:49:51.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddcf7fe700 1 -- 172.21.15.7:0/1292869527 learned_addr learned my addr 172.21.15.7:0/1292869527 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:49:51.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddceffd700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd0152e30 0x7fddd00c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:49:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddcffff700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd0155d80 0x7fddd006ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:49:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddcf7fe700 1 -- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd0155d80 msgr2=0x7fddd006ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:49:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddcf7fe700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd0155d80 0x7fddd006ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:49:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddcf7fe700 1 -- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd0152e30 msgr2=0x7fddd00c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:49:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddcf7fe700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd0152e30 0x7fddd00c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:49:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddcf7fe700 1 -- 172.21.15.7:0/1292869527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc0009550 con 0x7fddd00beab0 2023-03-03T19:49:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddceffd700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd0152e30 0x7fddd00c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:49:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.720+0000 7fddcf7fe700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd00beab0 0x7fddd01ea3f0 secure :-1 s=READY pgs=3614 cs=0 l=1 rev1=1 crypto rx=0x7fddc0000c00 tx=0x7fddc0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:49:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.721+0000 7fddb7fff700 1 -- 172.21.15.7:0/1292869527 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc0004480 con 0x7fddd00beab0 2023-03-03T19:49:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.721+0000 7fddb7fff700 1 -- 172.21.15.7:0/1292869527 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc001f070 con 0x7fddd00beab0 2023-03-03T19:49:51.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.721+0000 7fddd5c91700 1 -- 172.21.15.7:0/1292869527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd0071c70 con 0x7fddd00beab0 2023-03-03T19:49:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.721+0000 7fddb7fff700 1 -- 172.21.15.7:0/1292869527 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc00038a0 con 0x7fddd00beab0 2023-03-03T19:49:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.721+0000 7fddd5c91700 1 -- 172.21.15.7:0/1292869527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd00aadd0 con 0x7fddd00beab0 2023-03-03T19:49:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.722+0000 7fddd5c91700 1 -- 172.21.15.7:0/1292869527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd0153200 con 0x7fddd00beab0 2023-03-03T19:49:51.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.723+0000 7fddb7fff700 1 -- 172.21.15.7:0/1292869527 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fddc002a020 con 0x7fddd00beab0 2023-03-03T19:49:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.723+0000 7fddb7fff700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddbc0682f0 0x7fddbc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:49:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.724+0000 7fddb7fff700 1 -- 172.21.15.7:0/1292869527 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fddc0087920 con 0x7fddd00beab0 2023-03-03T19:49:51.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.724+0000 7fddceffd700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddbc0682f0 0x7fddbc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:49:51.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.724+0000 7fddceffd700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddbc0682f0 0x7fddbc06a7a0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fddd00c2ae0 tx=0x7fddc4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:49:51.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.728+0000 7fddb7fff700 1 -- 172.21.15.7:0/1292869527 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fddc0053bd0 con 0x7fddd00beab0 2023-03-03T19:49:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.968+0000 7fddd5c91700 1 -- 172.21.15.7:0/1292869527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddd0074bb0 con 0x7fddd00beab0 2023-03-03T19:49:51.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.968+0000 7fddb7fff700 1 -- 172.21.15.7:0/1292869527 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fddc00571f0 con 0x7fddd00beab0 2023-03-03T19:49:51.971 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:49:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:49:51.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.970+0000 7fddd5c91700 1 -- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddbc0682f0 msgr2=0x7fddbc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:49:51.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.970+0000 7fddd5c91700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddbc0682f0 0x7fddbc06a7a0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fddd00c2ae0 tx=0x7fddc4008040 comp rx=0 tx=0).stop 2023-03-03T19:49:51.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.970+0000 7fddd5c91700 1 -- 172.21.15.7:0/1292869527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd00beab0 msgr2=0x7fddd01ea3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:49:51.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.970+0000 7fddd5c91700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd00beab0 0x7fddd01ea3f0 secure :-1 s=READY pgs=3614 cs=0 l=1 rev1=1 crypto rx=0x7fddc0000c00 tx=0x7fddc0004750 comp rx=0 tx=0).stop 2023-03-03T19:49:51.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.970+0000 7fddd5c91700 1 -- 172.21.15.7:0/1292869527 shutdown_connections 2023-03-03T19:49:51.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.971+0000 7fddd5c91700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fddd0155d80 0x7fddd006ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:49:51.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.971+0000 7fddd5c91700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fddbc0682f0 0x7fddbc06a7a0 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:49:51.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.971+0000 7fddd5c91700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddd00beab0 0x7fddd01ea3f0 unknown :-1 s=CLOSED pgs=3614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:49:51.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.971+0000 7fddd5c91700 1 --2- 172.21.15.7:0/1292869527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fddd0152e30 0x7fddd00c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:49:51.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.971+0000 7fddd5c91700 1 -- 172.21.15.7:0/1292869527 >> 172.21.15.7:0/1292869527 conn(0x7fddd00736f0 msgr2=0x7fddd0156220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:49:51.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.971+0000 7fddd5c91700 1 -- 172.21.15.7:0/1292869527 shutdown_connections 2023-03-03T19:49:51.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:49:51.971+0000 7fddd5c91700 1 -- 172.21.15.7:0/1292869527 wait complete. 2023-03-03T19:49:51.974 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:49:51.984 INFO:tasks.mds_thrash.fs.[b]:waiting for 19 secs before thrashing 2023-03-03T19:49:52.943 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:49:52.946 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:50:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T19:49:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T19:50:10.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:50:11.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.125+0000 7f5dbea69700 1 Processor -- start 2023-03-03T19:50:11.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.125+0000 7f5dbea69700 1 -- start start 2023-03-03T19:50:11.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.126+0000 7f5dbea69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814f9d0 0x7f5db8156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:11.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.126+0000 7f5dbea69700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5db814fda0 0x7f5db8071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:11.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.126+0000 7f5dbea69700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db80c2f30 con 0x7f5db80abe20 2023-03-03T19:50:11.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.126+0000 7f5dbea69700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db80c31d0 con 0x7f5db814fda0 2023-03-03T19:50:11.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.126+0000 7f5dbea69700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db81524a0 con 0x7f5db814f9d0 2023-03-03T19:50:11.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.126+0000 7f5db7fff700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5db814fda0 0x7f5db8071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:11.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.126+0000 7f5dbc805700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814f9d0 0x7f5db8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:11.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.126+0000 7f5dbd006700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f5db80abe20 0x7f5db81529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:40762/0 (socket says 172.21.15.7:40762) 2023-03-03T19:50:11.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.126+0000 7f5dbd006700 1 -- 172.21.15.7:0/4082548171 learned_addr learned my addr 172.21.15.7:0/4082548171 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:50:11.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.127+0000 7f5dbc805700 1 -- 172.21.15.7:0/4082548171 >> v2:172.21.15.143:3300/0 conn(0x7f5db814fda0 msgr2=0x7f5db8071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:50:11.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.127+0000 7f5dbc805700 1 --2- 172.21.15.7:0/4082548171 >> v2:172.21.15.143:3300/0 conn(0x7f5db814fda0 0x7f5db8071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:11.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.127+0000 7f5dbc805700 1 -- 172.21.15.7:0/4082548171 >> v1:172.21.15.143:6789/0 conn(0x7f5db80abe20 legacy=0x7f5db81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:11.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.127+0000 7f5dbc805700 1 -- 172.21.15.7:0/4082548171 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db80c34b0 con 0x7f5db814f9d0 2023-03-03T19:50:11.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.127+0000 7f5db7fff700 1 --2- 172.21.15.7:0/4082548171 >> v2:172.21.15.143:3300/0 conn(0x7f5db814fda0 0x7f5db8071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:50:11.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.127+0000 7f5db77fe700 1 -- 172.21.15.7:0/4082548171 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983945735 0 0) 0x7f5db80c2f30 con 0x7f5db80abe20 2023-03-03T19:50:11.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.127+0000 7f5dbc805700 1 --2- 172.21.15.7:0/4082548171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814f9d0 0x7f5db8156430 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009840 tx=0x7f5dac009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f032b20b5fc72258 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:11.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.127+0000 7f5db77fe700 1 -- 172.21.15.7:0/4082548171 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac011830 con 0x7f5db814f9d0 2023-03-03T19:50:11.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.127+0000 7f5db77fe700 1 -- 172.21.15.7:0/4082548171 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dac011990 con 0x7f5db814f9d0 2023-03-03T19:50:11.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.127+0000 7f5db77fe700 1 -- 172.21.15.7:0/4082548171 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac011d20 con 0x7f5db814f9d0 2023-03-03T19:50:11.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.128+0000 7f5dbea69700 1 -- 172.21.15.7:0/4082548171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814f9d0 msgr2=0x7f5db8156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:11.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.128+0000 7f5dbea69700 1 --2- 172.21.15.7:0/4082548171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814f9d0 0x7f5db8156430 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009840 tx=0x7f5dac009b50 comp rx=0 tx=0).stop 2023-03-03T19:50:11.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.128+0000 7f5dbea69700 1 -- 172.21.15.7:0/4082548171 shutdown_connections 2023-03-03T19:50:11.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.128+0000 7f5dbea69700 1 --2- 172.21.15.7:0/4082548171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814f9d0 0x7f5db8156430 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:11.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.128+0000 7f5dbea69700 1 --2- 172.21.15.7:0/4082548171 >> v2:172.21.15.143:3300/0 conn(0x7f5db814fda0 0x7f5db8071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:11.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.128+0000 7f5dbea69700 1 -- 172.21.15.7:0/4082548171 >> 172.21.15.7:0/4082548171 conn(0x7f5db80c8900 msgr2=0x7f5db80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:50:11.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.128+0000 7f5dbea69700 1 -- 172.21.15.7:0/4082548171 shutdown_connections 2023-03-03T19:50:11.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.128+0000 7f5dbea69700 1 -- 172.21.15.7:0/4082548171 wait complete. 2023-03-03T19:50:11.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.129+0000 7f5dbea69700 1 Processor -- start 2023-03-03T19:50:11.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.129+0000 7f5dbea69700 1 -- start start 2023-03-03T19:50:11.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbea69700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db80abe20 0x7f5db81de140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:11.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbea69700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5db814f9d0 0x7f5db81de680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:11.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbea69700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814fda0 0x7f5db806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:11.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbea69700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db81debc0 con 0x7f5db80abe20 2023-03-03T19:50:11.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbea69700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db80aa790 con 0x7f5db814f9d0 2023-03-03T19:50:11.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbea69700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db806e690 con 0x7f5db814fda0 2023-03-03T19:50:11.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbc805700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db80abe20 0x7f5db81de140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:11.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbc805700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db80abe20 0x7f5db81de140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43264/0 (socket says 172.21.15.7:43264) 2023-03-03T19:50:11.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbc805700 1 -- 172.21.15.7:0/124413921 learned_addr learned my addr 172.21.15.7:0/124413921 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:50:11.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5db7fff700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5db814f9d0 0x7f5db81de680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:11.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbd006700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814fda0 0x7f5db806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:11.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbc805700 1 -- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814fda0 msgr2=0x7f5db806ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:11.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbc805700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814fda0 0x7f5db806ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:11.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbc805700 1 -- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5db814f9d0 msgr2=0x7f5db81de680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:11.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbc805700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5db814f9d0 0x7f5db81de680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:11.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.130+0000 7f5dbc805700 1 -- 172.21.15.7:0/124413921 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40094a0 con 0x7f5db80abe20 2023-03-03T19:50:11.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.131+0000 7f5dbc805700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db80abe20 0x7f5db81de140 secure :-1 s=READY pgs=3615 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009fd0 tx=0x7f5dac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:11.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.131+0000 7f5db57fa700 1 -- 172.21.15.7:0/124413921 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac0038a0 con 0x7f5db80abe20 2023-03-03T19:50:11.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.131+0000 7f5db57fa700 1 -- 172.21.15.7:0/124413921 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dac01f070 con 0x7f5db80abe20 2023-03-03T19:50:11.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.131+0000 7f5db57fa700 1 -- 172.21.15.7:0/124413921 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac003c40 con 0x7f5db80abe20 2023-03-03T19:50:11.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.131+0000 7f5dbea69700 1 -- 172.21.15.7:0/124413921 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dac0094a0 con 0x7f5db80abe20 2023-03-03T19:50:11.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.131+0000 7f5dbea69700 1 -- 172.21.15.7:0/124413921 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db80ac350 con 0x7f5db80abe20 2023-03-03T19:50:11.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.132+0000 7f5dbea69700 1 -- 172.21.15.7:0/124413921 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db80ab630 con 0x7f5db80abe20 2023-03-03T19:50:11.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.132+0000 7f5db57fa700 1 -- 172.21.15.7:0/124413921 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5dac00f050 con 0x7f5db80abe20 2023-03-03T19:50:11.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.132+0000 7f5db57fa700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d9c0682f0 0x7f5d9c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:11.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.133+0000 7f5db57fa700 1 -- 172.21.15.7:0/124413921 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5dac021070 con 0x7f5db80abe20 2023-03-03T19:50:11.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.133+0000 7f5db7fff700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d9c0682f0 0x7f5d9c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:11.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.133+0000 7f5db7fff700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d9c0682f0 0x7f5d9c06a7a0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f5db813fae0 tx=0x7f5da4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:11.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.137+0000 7f5db57fa700 1 -- 172.21.15.7:0/124413921 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5dac053e00 con 0x7f5db80abe20 2023-03-03T19:50:11.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.377+0000 7f5dbea69700 1 -- 172.21.15.7:0/124413921 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5db80c3e10 con 0x7f5db80abe20 2023-03-03T19:50:11.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.377+0000 7f5db57fa700 1 -- 172.21.15.7:0/124413921 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5dac010380 con 0x7f5db80abe20 2023-03-03T19:50:11.378 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:50:11.378 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:50:11.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 -- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d9c0682f0 msgr2=0x7f5d9c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:11.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d9c0682f0 0x7f5d9c06a7a0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f5db813fae0 tx=0x7f5da4008040 comp rx=0 tx=0).stop 2023-03-03T19:50:11.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 -- 172.21.15.7:0/124413921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db80abe20 msgr2=0x7f5db81de140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db80abe20 0x7f5db81de140 secure :-1 s=READY pgs=3615 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009fd0 tx=0x7f5dac004750 comp rx=0 tx=0).stop 2023-03-03T19:50:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 -- 172.21.15.7:0/124413921 shutdown_connections 2023-03-03T19:50:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5db814fda0 0x7f5db806ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:11.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d9c0682f0 0x7f5d9c06a7a0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5db80abe20 0x7f5db81de140 unknown :-1 s=CLOSED pgs=3615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 --2- 172.21.15.7:0/124413921 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5db814f9d0 0x7f5db81de680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 -- 172.21.15.7:0/124413921 >> 172.21.15.7:0/124413921 conn(0x7f5db80c8900 msgr2=0x7f5db81528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:50:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.379+0000 7f5dbea69700 1 -- 172.21.15.7:0/124413921 shutdown_connections 2023-03-03T19:50:11.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:11.380+0000 7f5dbea69700 1 -- 172.21.15.7:0/124413921 wait complete. 2023-03-03T19:50:11.383 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:50:11.392 INFO:tasks.mds_thrash.fs.[b]:waiting for 91 secs before thrashing 2023-03-03T19:50:16.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:50:16.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.565+0000 7f3af0531700 1 Processor -- start 2023-03-03T19:50:16.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.565+0000 7f3af0531700 1 -- start start 2023-03-03T19:50:16.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.566+0000 7f3af0531700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae8155390 0x7f3ae80c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.566+0000 7f3af0531700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae8154910 con 0x7f3ae8151770 2023-03-03T19:50:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.566+0000 7f3af0531700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae8152300 con 0x7f3ae806e020 2023-03-03T19:50:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.566+0000 7f3af0531700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae8070b90 con 0x7f3ae8155390 2023-03-03T19:50:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.566+0000 7f3aeeace700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3ae8151770 0x7f3ae8151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37254/0 (socket says 172.21.15.7:37254) 2023-03-03T19:50:16.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.566+0000 7f3aeeace700 1 -- 172.21.15.7:0/3921531927 learned_addr learned my addr 172.21.15.7:0/3921531927 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:50:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aee2cd700 1 --2- 172.21.15.7:0/3921531927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae8155390 0x7f3ae80c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aed2cb700 1 -- 172.21.15.7:0/3921531927 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882173530 0 0) 0x7f3ae8154910 con 0x7f3ae8151770 2023-03-03T19:50:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aed2cb700 1 -- 172.21.15.7:0/3921531927 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad40035b0 con 0x7f3ae8151770 2023-03-03T19:50:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aed2cb700 1 -- 172.21.15.7:0/3921531927 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627160861 0 0) 0x7f3ae8152300 con 0x7f3ae806e020 2023-03-03T19:50:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aed2cb700 1 -- 172.21.15.7:0/3921531927 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae8154910 con 0x7f3ae806e020 2023-03-03T19:50:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aee2cd700 1 -- 172.21.15.7:0/3921531927 >> v1:172.21.15.143:6789/0 conn(0x7f3ae806e020 legacy=0x7f3ae8071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aee2cd700 1 -- 172.21.15.7:0/3921531927 >> v1:172.21.15.7:6789/0 conn(0x7f3ae8151770 legacy=0x7f3ae8151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:16.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aed2cb700 1 -- 172.21.15.7:0/3921531927 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285386239 0 0) 0x7f3ad40035b0 con 0x7f3ae8151770 2023-03-03T19:50:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aee2cd700 1 -- 172.21.15.7:0/3921531927 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae80702a0 con 0x7f3ae8155390 2023-03-03T19:50:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aee2cd700 1 --2- 172.21.15.7:0/3921531927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae8155390 0x7f3ae80c3e30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009870 tx=0x7f3ae4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d4143ae5a814232 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aed2cb700 1 -- 172.21.15.7:0/3921531927 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae00019f0 con 0x7f3ae8155390 2023-03-03T19:50:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aed2cb700 1 -- 172.21.15.7:0/3921531927 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4004030 con 0x7f3ae8155390 2023-03-03T19:50:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.567+0000 7f3aed2cb700 1 -- 172.21.15.7:0/3921531927 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae40043c0 con 0x7f3ae8155390 2023-03-03T19:50:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.568+0000 7f3af0531700 1 -- 172.21.15.7:0/3921531927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae8155390 msgr2=0x7f3ae80c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:16.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.568+0000 7f3af0531700 1 --2- 172.21.15.7:0/3921531927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae8155390 0x7f3ae80c3e30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009870 tx=0x7f3ae4009b80 comp rx=0 tx=0).stop 2023-03-03T19:50:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.568+0000 7f3af0531700 1 -- 172.21.15.7:0/3921531927 shutdown_connections 2023-03-03T19:50:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.568+0000 7f3af0531700 1 --2- 172.21.15.7:0/3921531927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae8155390 0x7f3ae80c3e30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.568+0000 7f3af0531700 1 -- 172.21.15.7:0/3921531927 >> 172.21.15.7:0/3921531927 conn(0x7f3ae806c110 msgr2=0x7f3ae806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:50:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.568+0000 7f3af0531700 1 -- 172.21.15.7:0/3921531927 shutdown_connections 2023-03-03T19:50:16.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.568+0000 7f3af0531700 1 -- 172.21.15.7:0/3921531927 wait complete. 2023-03-03T19:50:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.570+0000 7f3af0531700 1 Processor -- start 2023-03-03T19:50:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.570+0000 7f3af0531700 1 -- start start 2023-03-03T19:50:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3af0531700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae806e020 0x7f3ae80c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3af0531700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ae8151770 0x7f3ae8141de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:16.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3af0531700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ae8155390 0x7f3ae8142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3af0531700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae814c730 con 0x7f3ae8151770 2023-03-03T19:50:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3af0531700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae814c400 con 0x7f3ae8155390 2023-03-03T19:50:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3af0531700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80ac010 con 0x7f3ae806e020 2023-03-03T19:50:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aedacc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ae8151770 0x7f3ae8141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aedacc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ae8151770 0x7f3ae8141de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43278/0 (socket says 172.21.15.7:43278) 2023-03-03T19:50:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aedacc700 1 -- 172.21.15.7:0/4101065131 learned_addr learned my addr 172.21.15.7:0/4101065131 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:50:16.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aee2cd700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae806e020 0x7f3ae80c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aedacc700 1 -- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae806e020 msgr2=0x7f3ae80c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aedacc700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae806e020 0x7f3ae80c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aeeace700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ae8155390 0x7f3ae8142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aedacc700 1 -- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ae8155390 msgr2=0x7f3ae8142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aedacc700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ae8155390 0x7f3ae8142320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aedacc700 1 -- 172.21.15.7:0/4101065131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae40094a0 con 0x7f3ae8151770 2023-03-03T19:50:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3aedacc700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ae8151770 0x7f3ae8141de0 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 crypto rx=0x7f3adc00c500 tx=0x7f3adc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:16.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3adaffd700 1 -- 172.21.15.7:0/4101065131 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc0052d0 con 0x7f3ae8151770 2023-03-03T19:50:16.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3adaffd700 1 -- 172.21.15.7:0/4101065131 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3adc005450 con 0x7f3ae8151770 2023-03-03T19:50:16.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3af0531700 1 -- 172.21.15.7:0/4101065131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae8070820 con 0x7f3ae8151770 2023-03-03T19:50:16.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3adaffd700 1 -- 172.21.15.7:0/4101065131 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc0038c0 con 0x7f3ae8151770 2023-03-03T19:50:16.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.571+0000 7f3af0531700 1 -- 172.21.15.7:0/4101065131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae8070540 con 0x7f3ae8151770 2023-03-03T19:50:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.572+0000 7f3af0531700 1 -- 172.21.15.7:0/4101065131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae8155760 con 0x7f3ae8151770 2023-03-03T19:50:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.573+0000 7f3adaffd700 1 -- 172.21.15.7:0/4101065131 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3adc019030 con 0x7f3ae8151770 2023-03-03T19:50:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.573+0000 7f3adaffd700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3acc0682f0 0x7f3acc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.573+0000 7f3adaffd700 1 -- 172.21.15.7:0/4101065131 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3adc0854e0 con 0x7f3ae8151770 2023-03-03T19:50:16.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.573+0000 7f3aee2cd700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3acc0682f0 0x7f3acc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.574+0000 7f3aee2cd700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3acc0682f0 0x7f3acc06a7a0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40097c0 tx=0x7f3ae4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:16.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.578+0000 7f3adaffd700 1 -- 172.21.15.7:0/4101065131 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3adc051880 con 0x7f3ae8151770 2023-03-03T19:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.814+0000 7f3af0531700 1 -- 172.21.15.7:0/4101065131 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ae814c260 con 0x7f3ae8151770 2023-03-03T19:50:16.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.815+0000 7f3adaffd700 1 -- 172.21.15.7:0/4101065131 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3ae814c260 con 0x7f3ae8151770 2023-03-03T19:50:16.816 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:50:16.816 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.816+0000 7f3af0531700 1 -- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3acc0682f0 msgr2=0x7f3acc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:16.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.816+0000 7f3af0531700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3acc0682f0 0x7f3acc06a7a0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40097c0 tx=0x7f3ae4008030 comp rx=0 tx=0).stop 2023-03-03T19:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.816+0000 7f3af0531700 1 -- 172.21.15.7:0/4101065131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ae8151770 msgr2=0x7f3ae8141de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.816+0000 7f3af0531700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ae8151770 0x7f3ae8141de0 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 crypto rx=0x7f3adc00c500 tx=0x7f3adc00c8c0 comp rx=0 tx=0).stop 2023-03-03T19:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.816+0000 7f3af0531700 1 -- 172.21.15.7:0/4101065131 shutdown_connections 2023-03-03T19:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.816+0000 7f3af0531700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ae8155390 0x7f3ae8142320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.816+0000 7f3af0531700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3ae806e020 0x7f3ae80c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.816+0000 7f3af0531700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3acc0682f0 0x7f3acc06a7a0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:16.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.816+0000 7f3af0531700 1 --2- 172.21.15.7:0/4101065131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ae8151770 0x7f3ae8141de0 unknown :-1 s=CLOSED pgs=3617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.817+0000 7f3af0531700 1 -- 172.21.15.7:0/4101065131 >> 172.21.15.7:0/4101065131 conn(0x7f3ae806c110 msgr2=0x7f3ae814ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.817+0000 7f3af0531700 1 -- 172.21.15.7:0/4101065131 shutdown_connections 2023-03-03T19:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:16.817+0000 7f3af0531700 1 -- 172.21.15.7:0/4101065131 wait complete. 2023-03-03T19:50:16.819 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:50:16.830 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before thrashing 2023-03-03T19:50:22.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:50:22.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.973+0000 7ff67f0aa700 1 Processor -- start 2023-03-03T19:50:22.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.973+0000 7ff67f0aa700 1 -- start start 2023-03-03T19:50:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.974+0000 7ff67f0aa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff678072250 0x7ff67814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.974+0000 7ff67f0aa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6781510d0 con 0x7ff678071e80 2023-03-03T19:50:22.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.974+0000 7ff67f0aa700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6780c3740 con 0x7ff6780c1ce0 2023-03-03T19:50:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.974+0000 7ff67f0aa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff678070670 con 0x7ff678072250 2023-03-03T19:50:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.974+0000 7ff67ce46700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff678071e80 0x7ff678151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37268/0 (socket says 172.21.15.7:37268) 2023-03-03T19:50:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.974+0000 7ff67ce46700 1 -- 172.21.15.7:0/2926274566 learned_addr learned my addr 172.21.15.7:0/2926274566 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:50:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.974+0000 7ff677fff700 1 --2- 172.21.15.7:0/2926274566 >> v2:172.21.15.143:3300/0 conn(0x7ff678072250 0x7ff67814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.974+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402889540 0 0) 0x7ff6781510d0 con 0x7ff678071e80 2023-03-03T19:50:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6680035b0 con 0x7ff678071e80 2023-03-03T19:50:22.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1587361816 0 0) 0x7ff6680035b0 con 0x7ff678071e80 2023-03-03T19:50:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 >> v2:172.21.15.143:3300/0 conn(0x7ff678072250 msgr2=0x7ff67814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:50:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 --2- 172.21.15.7:0/2926274566 >> v2:172.21.15.143:3300/0 conn(0x7ff678072250 0x7ff67814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 >> v1:172.21.15.143:6789/0 conn(0x7ff6780c1ce0 legacy=0x7ff67815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6780c39e0 con 0x7ff678071e80 2023-03-03T19:50:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7ff66c001670 con 0x7ff678071e80 2023-03-03T19:50:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 >> v1:172.21.15.7:6789/0 conn(0x7ff678071e80 legacy=0x7ff678151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff668003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:22.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff668004850 0x7ff668004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff668005250 0x7ff66800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff678070670 con 0x7ff668003a70 2023-03-03T19:50:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66800c190 con 0x7ff668005250 2023-03-03T19:50:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66800c350 con 0x7ff668004850 2023-03-03T19:50:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff67ce46700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff668003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff677fff700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff668004850 0x7ff668004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:22.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff67d647700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff668005250 0x7ff66800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff67ce46700 1 -- 172.21.15.7:0/2926274566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff668004850 msgr2=0x7ff668004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:50:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff67ce46700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff668004850 0x7ff668004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff67ce46700 1 -- 172.21.15.7:0/2926274566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff668005250 msgr2=0x7ff66800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:50:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff67ce46700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff668005250 0x7ff66800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff67ce46700 1 -- 172.21.15.7:0/2926274566 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6680035b0 con 0x7ff668003a70 2023-03-03T19:50:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff67d647700 1 -- 172.21.15.7:0/2926274566 reap_dead start 2023-03-03T19:50:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.975+0000 7ff67ce46700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff668003e80 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00c210 tx=0x7ff668005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02e43da3a1b80a2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:22.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.976+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c007c10 con 0x7ff668003a70 2023-03-03T19:50:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.976+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66c00cdd0 con 0x7ff668003a70 2023-03-03T19:50:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.976+0000 7ff6777fe700 1 -- 172.21.15.7:0/2926274566 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c005ba0 con 0x7ff668003a70 2023-03-03T19:50:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.976+0000 7ff67f0aa700 1 -- 172.21.15.7:0/2926274566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 msgr2=0x7ff668003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.976+0000 7ff67f0aa700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff668003e80 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00c210 tx=0x7ff668005880 comp rx=0 tx=0).stop 2023-03-03T19:50:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.976+0000 7ff67f0aa700 1 -- 172.21.15.7:0/2926274566 shutdown_connections 2023-03-03T19:50:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.976+0000 7ff67f0aa700 1 --2- 172.21.15.7:0/2926274566 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff668003e80 unknown :-1 s=CLOSED pgs=3619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:22.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.976+0000 7ff67f0aa700 1 -- 172.21.15.7:0/2926274566 >> 172.21.15.7:0/2926274566 conn(0x7ff6780a94d0 msgr2=0x7ff6780a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:50:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.976+0000 7ff67f0aa700 1 -- 172.21.15.7:0/2926274566 shutdown_connections 2023-03-03T19:50:22.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.976+0000 7ff67f0aa700 1 -- 172.21.15.7:0/2926274566 wait complete. 2023-03-03T19:50:22.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.977+0000 7ff67f0aa700 1 Processor -- start 2023-03-03T19:50:22.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.977+0000 7ff67f0aa700 1 -- start start 2023-03-03T19:50:22.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff67f0aa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff67814ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:22.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff67f0aa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6781474e0 0x7ff678146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:22.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff67f0aa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6781490a0 0x7ff678149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:22.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff67f0aa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6781511a0 con 0x7ff668003a70 2023-03-03T19:50:22.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff67f0aa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67814f480 con 0x7ff6781474e0 2023-03-03T19:50:22.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff67f0aa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67814a7a0 con 0x7ff6781490a0 2023-03-03T19:50:22.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff67ce46700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff67814ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:22.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff67ce46700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff67814ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43300/0 (socket says 172.21.15.7:43300) 2023-03-03T19:50:22.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff67ce46700 1 -- 172.21.15.7:0/3104844505 learned_addr learned my addr 172.21.15.7:0/3104844505 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:50:22.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff67d647700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6781490a0 0x7ff678149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:22.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.978+0000 7ff677fff700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6781474e0 0x7ff678146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:22.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff67ce46700 1 -- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6781490a0 msgr2=0x7ff678149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:22.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff67ce46700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6781490a0 0x7ff678149d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:22.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff67ce46700 1 -- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6781474e0 msgr2=0x7ff678146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:22.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff67ce46700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6781474e0 0x7ff678146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:22.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff67ce46700 1 -- 172.21.15.7:0/3104844505 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67806c9a0 con 0x7ff668003a70 2023-03-03T19:50:22.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff677fff700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6781474e0 0x7ff678146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:50:22.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff67d647700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6781490a0 0x7ff678149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:50:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff67ce46700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff67814ef40 secure :-1 s=READY pgs=3620 cs=0 l=1 rev1=1 crypto rx=0x7ff66800c160 tx=0x7ff66c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff6757fa700 1 -- 172.21.15.7:0/3104844505 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c004c80 con 0x7ff668003a70 2023-03-03T19:50:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff6757fa700 1 -- 172.21.15.7:0/3104844505 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66c004e00 con 0x7ff668003a70 2023-03-03T19:50:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff67f0aa700 1 -- 172.21.15.7:0/3104844505 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff678151300 con 0x7ff668003a70 2023-03-03T19:50:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff6757fa700 1 -- 172.21.15.7:0/3104844505 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c007310 con 0x7ff668003a70 2023-03-03T19:50:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.979+0000 7ff67f0aa700 1 -- 172.21.15.7:0/3104844505 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67806cb30 con 0x7ff668003a70 2023-03-03T19:50:22.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.980+0000 7ff67f0aa700 1 -- 172.21.15.7:0/3104844505 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff678074bb0 con 0x7ff668003a70 2023-03-03T19:50:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.981+0000 7ff6757fa700 1 -- 172.21.15.7:0/3104844505 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff66c010070 con 0x7ff668003a70 2023-03-03T19:50:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.981+0000 7ff6757fa700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff65c0682f0 0x7ff65c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.981+0000 7ff6757fa700 1 -- 172.21.15.7:0/3104844505 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff66c086a20 con 0x7ff668003a70 2023-03-03T19:50:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.981+0000 7ff677fff700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff65c0682f0 0x7ff65c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:22.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.982+0000 7ff677fff700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff65c0682f0 0x7ff65c06a7a0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7ff67806aac0 tx=0x7ff664008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:22.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:22.986+0000 7ff6757fa700 1 -- 172.21.15.7:0/3104844505 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff66c052d40 con 0x7ff668003a70 2023-03-03T19:50:23.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:50:23.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:50:23.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.239+0000 7ff67f0aa700 1 -- 172.21.15.7:0/3104844505 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff678147f80 con 0x7ff668003a70 2023-03-03T19:50:23.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.240+0000 7ff6757fa700 1 -- 172.21.15.7:0/3104844505 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff66c056360 con 0x7ff668003a70 2023-03-03T19:50:23.241 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:50:23.242 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:50:23.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 -- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff65c0682f0 msgr2=0x7ff65c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:23.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff65c0682f0 0x7ff65c06a7a0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7ff67806aac0 tx=0x7ff664008040 comp rx=0 tx=0).stop 2023-03-03T19:50:23.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 -- 172.21.15.7:0/3104844505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 msgr2=0x7ff67814ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:23.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff67814ef40 secure :-1 s=READY pgs=3620 cs=0 l=1 rev1=1 crypto rx=0x7ff66800c160 tx=0x7ff66c001a00 comp rx=0 tx=0).stop 2023-03-03T19:50:23.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 -- 172.21.15.7:0/3104844505 shutdown_connections 2023-03-03T19:50:23.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff6781490a0 0x7ff678149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:23.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff65c0682f0 0x7ff65c06a7a0 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:23.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff668003a70 0x7ff67814ef40 unknown :-1 s=CLOSED pgs=3620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:23.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 --2- 172.21.15.7:0/3104844505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff6781474e0 0x7ff678146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:23.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 -- 172.21.15.7:0/3104844505 >> 172.21.15.7:0/3104844505 conn(0x7ff6780a94d0 msgr2=0x7ff678151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:50:23.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 -- 172.21.15.7:0/3104844505 shutdown_connections 2023-03-03T19:50:23.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:23.242+0000 7ff67f0aa700 1 -- 172.21.15.7:0/3104844505 wait complete. 2023-03-03T19:50:23.246 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:50:23.255 INFO:tasks.mds_thrash.fs.[a]:waiting for 20 secs before thrashing 2023-03-03T19:50:43.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:50:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.394+0000 7f5c454b0700 1 Processor -- start 2023-03-03T19:50:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.394+0000 7f5c454b0700 1 -- start start 2023-03-03T19:50:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.395+0000 7f5c454b0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5c400c00e0 0x7f5c400c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.395+0000 7f5c454b0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c400c7310 con 0x7f5c40154ff0 2023-03-03T19:50:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.395+0000 7f5c454b0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c400bfd10 con 0x7f5c400caa30 2023-03-03T19:50:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.395+0000 7f5c454b0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c40155430 con 0x7f5c400c00e0 2023-03-03T19:50:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.395+0000 7f5c3f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5c40154ff0 0x7f5c4015ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37310/0 (socket says 172.21.15.7:37310) 2023-03-03T19:50:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.395+0000 7f5c3f7fe700 1 -- 172.21.15.7:0/686557029 learned_addr learned my addr 172.21.15.7:0/686557029 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:50:43.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.395+0000 7f5c3e7fc700 1 --2- 172.21.15.7:0/686557029 >> v2:172.21.15.143:3300/0 conn(0x7f5c400c00e0 0x7f5c400c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.395+0000 7f5c3dffb700 1 -- 172.21.15.7:0/686557029 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056400246 0 0) 0x7f5c400c7310 con 0x7f5c40154ff0 2023-03-03T19:50:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.395+0000 7f5c3dffb700 1 -- 172.21.15.7:0/686557029 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c0035b0 con 0x7f5c40154ff0 2023-03-03T19:50:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.396+0000 7f5c3e7fc700 1 -- 172.21.15.7:0/686557029 >> v1:172.21.15.143:6789/0 conn(0x7f5c400caa30 legacy=0x7f5c400c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.396+0000 7f5c3e7fc700 1 -- 172.21.15.7:0/686557029 >> v1:172.21.15.7:6789/0 conn(0x7f5c40154ff0 legacy=0x7f5c4015ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.396+0000 7f5c3e7fc700 1 -- 172.21.15.7:0/686557029 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c400bfa30 con 0x7f5c400c00e0 2023-03-03T19:50:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.396+0000 7f5c3e7fc700 1 --2- 172.21.15.7:0/686557029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c400c00e0 0x7f5c400c5c60 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009820 tx=0x7f5c30009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b00f0ae496eed555 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.396+0000 7f5c3dffb700 1 -- 172.21.15.7:0/686557029 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30005e30 con 0x7f5c400c00e0 2023-03-03T19:50:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.396+0000 7f5c3dffb700 1 -- 172.21.15.7:0/686557029 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c30006030 con 0x7f5c400c00e0 2023-03-03T19:50:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.396+0000 7f5c3dffb700 1 -- 172.21.15.7:0/686557029 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c300063a0 con 0x7f5c400c00e0 2023-03-03T19:50:43.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.397+0000 7f5c454b0700 1 -- 172.21.15.7:0/686557029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c400c00e0 msgr2=0x7f5c400c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:43.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.397+0000 7f5c454b0700 1 --2- 172.21.15.7:0/686557029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c400c00e0 0x7f5c400c5c60 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009820 tx=0x7f5c30009b30 comp rx=0 tx=0).stop 2023-03-03T19:50:43.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.397+0000 7f5c454b0700 1 -- 172.21.15.7:0/686557029 shutdown_connections 2023-03-03T19:50:43.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.397+0000 7f5c454b0700 1 --2- 172.21.15.7:0/686557029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c400c00e0 0x7f5c400c5c60 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.397+0000 7f5c454b0700 1 -- 172.21.15.7:0/686557029 >> 172.21.15.7:0/686557029 conn(0x7f5c40066e40 msgr2=0x7f5c40071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:50:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.397+0000 7f5c454b0700 1 -- 172.21.15.7:0/686557029 shutdown_connections 2023-03-03T19:50:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.397+0000 7f5c454b0700 1 -- 172.21.15.7:0/686557029 wait complete. 2023-03-03T19:50:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.398+0000 7f5c454b0700 1 Processor -- start 2023-03-03T19:50:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.398+0000 7f5c454b0700 1 -- start start 2023-03-03T19:50:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c454b0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c400c00e0 0x7f5c40157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c454b0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c400caa30 0x7f5c400c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c454b0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c40154ff0 0x7f5c40156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c454b0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c401de870 con 0x7f5c400c00e0 2023-03-03T19:50:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c454b0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c400c4a10 con 0x7f5c400caa30 2023-03-03T19:50:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c454b0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4014d480 con 0x7f5c40154ff0 2023-03-03T19:50:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c400c00e0 0x7f5c40157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c400c00e0 0x7f5c40157170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43332/0 (socket says 172.21.15.7:43332) 2023-03-03T19:50:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3effd700 1 -- 172.21.15.7:0/1126001399 learned_addr learned my addr 172.21.15.7:0/1126001399 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:50:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3e7fc700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c400caa30 0x7f5c400c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3f7fe700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c40154ff0 0x7f5c40156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3effd700 1 -- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c40154ff0 msgr2=0x7f5c40156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3effd700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c40154ff0 0x7f5c40156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3effd700 1 -- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c400caa30 msgr2=0x7f5c400c37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3effd700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c400caa30 0x7f5c400c37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3effd700 1 -- 172.21.15.7:0/1126001399 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c30009430 con 0x7f5c400c00e0 2023-03-03T19:50:43.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3e7fc700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c400caa30 0x7f5c400c37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:50:43.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.399+0000 7f5c3effd700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c400c00e0 0x7f5c40157170 secure :-1 s=READY pgs=3622 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c6b0 tx=0x7f5c2800ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:43.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.400+0000 7f5c377fe700 1 -- 172.21.15.7:0/1126001399 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c280052d0 con 0x7f5c400c00e0 2023-03-03T19:50:43.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.400+0000 7f5c377fe700 1 -- 172.21.15.7:0/1126001399 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c28005430 con 0x7f5c400c00e0 2023-03-03T19:50:43.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.400+0000 7f5c454b0700 1 -- 172.21.15.7:0/1126001399 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c401547f0 con 0x7f5c400c00e0 2023-03-03T19:50:43.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.400+0000 7f5c377fe700 1 -- 172.21.15.7:0/1126001399 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c280038c0 con 0x7f5c400c00e0 2023-03-03T19:50:43.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.400+0000 7f5c454b0700 1 -- 172.21.15.7:0/1126001399 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c40154510 con 0x7f5c400c00e0 2023-03-03T19:50:43.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.401+0000 7f5c454b0700 1 -- 172.21.15.7:0/1126001399 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c40154ad0 con 0x7f5c400c00e0 2023-03-03T19:50:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.401+0000 7f5c377fe700 1 -- 172.21.15.7:0/1126001399 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5c28019030 con 0x7f5c400c00e0 2023-03-03T19:50:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.401+0000 7f5c377fe700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 0x7f5c2006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.402+0000 7f5c377fe700 1 -- 172.21.15.7:0/1126001399 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5c28085510 con 0x7f5c400c00e0 2023-03-03T19:50:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.402+0000 7f5c3e7fc700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 0x7f5c2006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.402+0000 7f5c3e7fc700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 0x7f5c2006a7a0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009b70 tx=0x7f5c30008070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:43.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.406+0000 7f5c377fe700 1 -- 172.21.15.7:0/1126001399 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5c28051850 con 0x7f5c400c00e0 2023-03-03T19:50:43.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.645+0000 7f5c454b0700 1 -- 172.21.15.7:0/1126001399 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c40074bb0 con 0x7f5c400c00e0 2023-03-03T19:50:43.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.645+0000 7f5c377fe700 1 -- 172.21.15.7:0/1126001399 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5c28054e70 con 0x7f5c400c00e0 2023-03-03T19:50:43.648 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:50:43.648 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:50:43.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 -- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 msgr2=0x7f5c2006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:43.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 0x7f5c2006a7a0 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009b70 tx=0x7f5c30008070 comp rx=0 tx=0).stop 2023-03-03T19:50:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 -- 172.21.15.7:0/1126001399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c400c00e0 msgr2=0x7f5c40157170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c400c00e0 0x7f5c40157170 secure :-1 s=READY pgs=3622 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c6b0 tx=0x7f5c2800ca70 comp rx=0 tx=0).stop 2023-03-03T19:50:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 -- 172.21.15.7:0/1126001399 shutdown_connections 2023-03-03T19:50:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5c40154ff0 0x7f5c40156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5c200682f0 0x7f5c2006a7a0 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5c400c00e0 0x7f5c40157170 unknown :-1 s=CLOSED pgs=3622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 --2- 172.21.15.7:0/1126001399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5c400caa30 0x7f5c400c37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:43.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 -- 172.21.15.7:0/1126001399 >> 172.21.15.7:0/1126001399 conn(0x7f5c40066e40 msgr2=0x7f5c4006b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:50:43.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 -- 172.21.15.7:0/1126001399 shutdown_connections 2023-03-03T19:50:43.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:43.648+0000 7f5c454b0700 1 -- 172.21.15.7:0/1126001399 wait complete. 2023-03-03T19:50:43.651 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:50:43.661 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2023-03-03T19:50:53.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:50:53.082 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:50:58.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:50:58.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.802+0000 7fbacdedc700 1 Processor -- start 2023-03-03T19:50:58.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.802+0000 7fbacdedc700 1 -- start start 2023-03-03T19:50:58.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.802+0000 7fbacdedc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbac8151cd0 0x7fbac8154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:58.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.802+0000 7fbacdedc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbac806c6c0 0x7fbac80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:58.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.802+0000 7fbacdedc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac8157ff0 con 0x7fbac80c2e30 2023-03-03T19:50:58.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.802+0000 7fbacdedc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac806d640 con 0x7fbac806c6c0 2023-03-03T19:50:58.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.802+0000 7fbacdedc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac806cda0 con 0x7fbac8151cd0 2023-03-03T19:50:58.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbac806c6c0 0x7fbac80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:58.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbac806c6c0 0x7fbac80736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43356/0 (socket says 172.21.15.7:43356) 2023-03-03T19:50:58.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac7fff700 1 -- 172.21.15.7:0/2950367116 learned_addr learned my addr 172.21.15.7:0/2950367116 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:50:58.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac77fe700 1 --2- 172.21.15.7:0/2950367116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbac8151cd0 0x7fbac8154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:58.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac7fff700 1 -- 172.21.15.7:0/2950367116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbac8151cd0 msgr2=0x7fbac8154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:50:58.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac7fff700 1 --2- 172.21.15.7:0/2950367116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbac8151cd0 0x7fbac8154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:58.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac7fff700 1 -- 172.21.15.7:0/2950367116 >> v1:172.21.15.143:6789/0 conn(0x7fbac80c2e30 legacy=0x7fbac8158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:50:58.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac7fff700 1 -- 172.21.15.7:0/2950367116 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac806d080 con 0x7fbac806c6c0 2023-03-03T19:50:58.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac7fff700 1 --2- 172.21.15.7:0/2950367116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac806c6c0 0x7fbac80736f0 secure :-1 s=READY pgs=3623 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009e50 tx=0x7fbac000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573f41a216e03eeb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:58.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac67fc700 1 -- 172.21.15.7:0/2950367116 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac000e070 con 0x7fbac806c6c0 2023-03-03T19:50:58.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac67fc700 1 -- 172.21.15.7:0/2950367116 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac0003a20 con 0x7fbac806c6c0 2023-03-03T19:50:58.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.803+0000 7fbac67fc700 1 -- 172.21.15.7:0/2950367116 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac0003dd0 con 0x7fbac806c6c0 2023-03-03T19:50:58.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.804+0000 7fbacdedc700 1 -- 172.21.15.7:0/2950367116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac806c6c0 msgr2=0x7fbac80736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:58.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.804+0000 7fbacdedc700 1 --2- 172.21.15.7:0/2950367116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac806c6c0 0x7fbac80736f0 secure :-1 s=READY pgs=3623 cs=0 l=1 rev1=1 crypto rx=0x7fbac0009e50 tx=0x7fbac000b330 comp rx=0 tx=0).stop 2023-03-03T19:50:58.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.804+0000 7fbacdedc700 1 -- 172.21.15.7:0/2950367116 shutdown_connections 2023-03-03T19:50:58.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.804+0000 7fbacdedc700 1 --2- 172.21.15.7:0/2950367116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac806c6c0 0x7fbac80736f0 unknown :-1 s=CLOSED pgs=3623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:58.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.804+0000 7fbacdedc700 1 --2- 172.21.15.7:0/2950367116 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbac8151cd0 0x7fbac8154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:58.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.804+0000 7fbacdedc700 1 -- 172.21.15.7:0/2950367116 >> 172.21.15.7:0/2950367116 conn(0x7fbac80a94d0 msgr2=0x7fbac80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:50:58.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.804+0000 7fbacdedc700 1 -- 172.21.15.7:0/2950367116 shutdown_connections 2023-03-03T19:50:58.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.804+0000 7fbacdedc700 1 -- 172.21.15.7:0/2950367116 wait complete. 2023-03-03T19:50:58.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.805+0000 7fbacdedc700 1 Processor -- start 2023-03-03T19:50:58.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.806+0000 7fbacdedc700 1 -- start start 2023-03-03T19:50:58.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbacdedc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbac806c6c0 0x7fbac81dee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:58.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbacdedc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbac80c2e30 0x7fbac814db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:58.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbacdedc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8151cd0 0x7fbac8152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:58.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbacdedc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac81514b0 con 0x7fbac8151cd0 2023-03-03T19:50:58.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbacdedc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac81e06e0 con 0x7fbac806c6c0 2023-03-03T19:50:58.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbacdedc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac81e09c0 con 0x7fbac80c2e30 2023-03-03T19:50:58.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8151cd0 0x7fbac8152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:58.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8151cd0 0x7fbac8152510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43362/0 (socket says 172.21.15.7:43362) 2023-03-03T19:50:58.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac7fff700 1 -- 172.21.15.7:0/1648409156 learned_addr learned my addr 172.21.15.7:0/1648409156 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:50:58.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac77fe700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbac806c6c0 0x7fbac81dee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:58.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac6ffd700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbac80c2e30 0x7fbac814db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:58.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac7fff700 1 -- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbac80c2e30 msgr2=0x7fbac814db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac7fff700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbac80c2e30 0x7fbac814db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac7fff700 1 -- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbac806c6c0 msgr2=0x7fbac81dee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac7fff700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbac806c6c0 0x7fbac81dee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac7fff700 1 -- 172.21.15.7:0/1648409156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac0009b30 con 0x7fbac8151cd0 2023-03-03T19:50:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbac7fff700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8151cd0 0x7fbac8152510 secure :-1 s=READY pgs=3624 cs=0 l=1 rev1=1 crypto rx=0x7fbac80744a0 tx=0x7fbac0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.807+0000 7fbaaffff700 1 -- 172.21.15.7:0/1648409156 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac0004440 con 0x7fbac8151cd0 2023-03-03T19:50:58.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.808+0000 7fbaaffff700 1 -- 172.21.15.7:0/1648409156 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac00045c0 con 0x7fbac8151cd0 2023-03-03T19:50:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.808+0000 7fbacdedc700 1 -- 172.21.15.7:0/1648409156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac8157c80 con 0x7fbac8151cd0 2023-03-03T19:50:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.808+0000 7fbaaffff700 1 -- 172.21.15.7:0/1648409156 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac0004970 con 0x7fbac8151cd0 2023-03-03T19:50:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.808+0000 7fbacdedc700 1 -- 172.21.15.7:0/1648409156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac806d360 con 0x7fbac8151cd0 2023-03-03T19:50:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.809+0000 7fbacdedc700 1 -- 172.21.15.7:0/1648409156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac8154a50 con 0x7fbac8151cd0 2023-03-03T19:50:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.809+0000 7fbaaffff700 1 -- 172.21.15.7:0/1648409156 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbac002c070 con 0x7fbac8151cd0 2023-03-03T19:50:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.809+0000 7fbaaffff700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbab40682f0 0x7fbab406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:50:58.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.809+0000 7fbaaffff700 1 -- 172.21.15.7:0/1648409156 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbac0087750 con 0x7fbac8151cd0 2023-03-03T19:50:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.810+0000 7fbac77fe700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbab40682f0 0x7fbab406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:50:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.810+0000 7fbac77fe700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbab40682f0 0x7fbab406a7a0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009930 tx=0x7fbab8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:50:58.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:58.814+0000 7fbaaffff700 1 -- 172.21.15.7:0/1648409156 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbac0053a00 con 0x7fbac8151cd0 2023-03-03T19:50:59.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.055+0000 7fbacdedc700 1 -- 172.21.15.7:0/1648409156 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac81eb2f0 con 0x7fbac8151cd0 2023-03-03T19:50:59.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.056+0000 7fbaaffff700 1 -- 172.21.15.7:0/1648409156 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbac0057020 con 0x7fbac8151cd0 2023-03-03T19:50:59.057 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:50:59.058 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:50:59.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 -- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbab40682f0 msgr2=0x7fbab406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:59.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbab40682f0 0x7fbab406a7a0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009930 tx=0x7fbab8008040 comp rx=0 tx=0).stop 2023-03-03T19:50:59.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 -- 172.21.15.7:0/1648409156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8151cd0 msgr2=0x7fbac8152510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:50:59.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8151cd0 0x7fbac8152510 secure :-1 s=READY pgs=3624 cs=0 l=1 rev1=1 crypto rx=0x7fbac80744a0 tx=0x7fbac0004ad0 comp rx=0 tx=0).stop 2023-03-03T19:50:59.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 -- 172.21.15.7:0/1648409156 shutdown_connections 2023-03-03T19:50:59.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbab40682f0 0x7fbab406a7a0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:59.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbac8151cd0 0x7fbac8152510 unknown :-1 s=CLOSED pgs=3624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:59.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbac806c6c0 0x7fbac81dee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:59.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 --2- 172.21.15.7:0/1648409156 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbac80c2e30 0x7fbac814db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:50:59.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 -- 172.21.15.7:0/1648409156 >> 172.21.15.7:0/1648409156 conn(0x7fbac80a94d0 msgr2=0x7fbac80719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:50:59.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 -- 172.21.15.7:0/1648409156 shutdown_connections 2023-03-03T19:50:59.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:50:59.058+0000 7fbacdedc700 1 -- 172.21.15.7:0/1648409156 wait complete. 2023-03-03T19:50:59.061 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:50:59.070 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-03-03T19:51:23.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:51:23.144 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:51:42.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:51:42.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.533+0000 7f80b49bf700 1 Processor -- start 2023-03-03T19:51:42.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.533+0000 7f80b49bf700 1 -- start start 2023-03-03T19:51:42.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.534+0000 7f80b49bf700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80b014de50 0x7f80b006a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.534+0000 7f80b49bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b006a7f0 0x7f80b00c2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.534+0000 7f80b49bf700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b00a65b0 con 0x7f80b00c65c0 2023-03-03T19:51:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.534+0000 7f80b49bf700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b0154e00 con 0x7f80b014de50 2023-03-03T19:51:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.534+0000 7f80b49bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b00c3370 con 0x7f80b006a7f0 2023-03-03T19:51:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.534+0000 7f80ae59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80b014de50 0x7f80b006a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.534+0000 7f80ae59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f80b014de50 0x7f80b006a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43394/0 (socket says 172.21.15.7:43394) 2023-03-03T19:51:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.535+0000 7f80ae59c700 1 -- 172.21.15.7:0/3434467166 learned_addr learned my addr 172.21.15.7:0/3434467166 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:51:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.535+0000 7f80aed9d700 1 --2- 172.21.15.7:0/3434467166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b006a7f0 0x7f80b00c2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.535+0000 7f80ae59c700 1 -- 172.21.15.7:0/3434467166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b006a7f0 msgr2=0x7f80b00c2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:51:42.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.535+0000 7f80ae59c700 1 --2- 172.21.15.7:0/3434467166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b006a7f0 0x7f80b00c2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.535+0000 7f80ae59c700 1 -- 172.21.15.7:0/3434467166 >> v1:172.21.15.143:6789/0 conn(0x7f80b00c65c0 legacy=0x7f80b00c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:42.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.535+0000 7f80ae59c700 1 -- 172.21.15.7:0/3434467166 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b01541f0 con 0x7f80b014de50 2023-03-03T19:51:42.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.535+0000 7f80ae59c700 1 --2- 172.21.15.7:0/3434467166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b014de50 0x7f80b006a2b0 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 crypto rx=0x7f809800bbc0 tx=0x7f809800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbeefae51d7ac4ce server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:51:42.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.535+0000 7f80ad59a700 1 -- 172.21.15.7:0/3434467166 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809800dd50 con 0x7f80b014de50 2023-03-03T19:51:42.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.535+0000 7f80ad59a700 1 -- 172.21.15.7:0/3434467166 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8098010470 con 0x7f80b014de50 2023-03-03T19:51:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.535+0000 7f80ad59a700 1 -- 172.21.15.7:0/3434467166 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098010800 con 0x7f80b014de50 2023-03-03T19:51:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.536+0000 7f80b49bf700 1 -- 172.21.15.7:0/3434467166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b014de50 msgr2=0x7f80b006a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.536+0000 7f80b49bf700 1 --2- 172.21.15.7:0/3434467166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b014de50 0x7f80b006a2b0 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 crypto rx=0x7f809800bbc0 tx=0x7f809800bed0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.536+0000 7f80b49bf700 1 -- 172.21.15.7:0/3434467166 shutdown_connections 2023-03-03T19:51:42.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.536+0000 7f80b49bf700 1 --2- 172.21.15.7:0/3434467166 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b006a7f0 0x7f80b00c2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.536+0000 7f80b49bf700 1 --2- 172.21.15.7:0/3434467166 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b014de50 0x7f80b006a2b0 unknown :-1 s=CLOSED pgs=3625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.536+0000 7f80b49bf700 1 -- 172.21.15.7:0/3434467166 >> 172.21.15.7:0/3434467166 conn(0x7f80b00a9430 msgr2=0x7f80b00c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:51:42.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.536+0000 7f80b49bf700 1 -- 172.21.15.7:0/3434467166 shutdown_connections 2023-03-03T19:51:42.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.536+0000 7f80b49bf700 1 -- 172.21.15.7:0/3434467166 wait complete. 2023-03-03T19:51:42.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.537+0000 7f80b49bf700 1 Processor -- start 2023-03-03T19:51:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.537+0000 7f80b49bf700 1 -- start start 2023-03-03T19:51:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80b49bf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b006a7f0 0x7f80b014acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80b49bf700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b00c65c0 0x7f80b006cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80b49bf700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80b014de50 0x7f80b006d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80b49bf700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b006db20 con 0x7f80b006a7f0 2023-03-03T19:51:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80b49bf700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b01e6900 con 0x7f80b014de50 2023-03-03T19:51:42.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80b49bf700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b006d790 con 0x7f80b00c65c0 2023-03-03T19:51:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80ae59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b006a7f0 0x7f80b014acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80ae59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b006a7f0 0x7f80b014acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43400/0 (socket says 172.21.15.7:43400) 2023-03-03T19:51:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80ae59c700 1 -- 172.21.15.7:0/2195763425 learned_addr learned my addr 172.21.15.7:0/2195763425 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:51:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80ae59c700 1 -- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b00c65c0 msgr2=0x7f80b006cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80ae59c700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b00c65c0 0x7f80b006cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80aed9d700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80b014de50 0x7f80b006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80ae59c700 1 -- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80b014de50 msgr2=0x7f80b006d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80ae59c700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80b014de50 0x7f80b006d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80ae59c700 1 -- 172.21.15.7:0/2195763425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809800b7f0 con 0x7f80b006a7f0 2023-03-03T19:51:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.538+0000 7f80ae59c700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b006a7f0 0x7f80b014acf0 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7f809800bf10 tx=0x7f8098004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:51:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.539+0000 7f80a6ffd700 1 -- 172.21.15.7:0/2195763425 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809801f070 con 0x7f80b006a7f0 2023-03-03T19:51:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.539+0000 7f80a6ffd700 1 -- 172.21.15.7:0/2195763425 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80980039b0 con 0x7f80b006a7f0 2023-03-03T19:51:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.539+0000 7f80b49bf700 1 -- 172.21.15.7:0/2195763425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b01547b0 con 0x7f80b006a7f0 2023-03-03T19:51:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.539+0000 7f80a6ffd700 1 -- 172.21.15.7:0/2195763425 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098003d40 con 0x7f80b006a7f0 2023-03-03T19:51:42.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.539+0000 7f80b49bf700 1 -- 172.21.15.7:0/2195763425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b01544d0 con 0x7f80b006a7f0 2023-03-03T19:51:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.540+0000 7f80b49bf700 1 -- 172.21.15.7:0/2195763425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b0154a90 con 0x7f80b006a7f0 2023-03-03T19:51:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.540+0000 7f80a6ffd700 1 -- 172.21.15.7:0/2195763425 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8098019400 con 0x7f80b006a7f0 2023-03-03T19:51:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.541+0000 7f80a6ffd700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80900682f0 0x7f809006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.541+0000 7f80a6ffd700 1 -- 172.21.15.7:0/2195763425 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8098089f40 con 0x7f80b006a7f0 2023-03-03T19:51:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.541+0000 7f80add9b700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80900682f0 0x7f809006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.541+0000 7f80add9b700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80900682f0 0x7f809006a7a0 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f80a0000f60 tx=0x7f80a0007a40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:51:42.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.545+0000 7f80a6ffd700 1 -- 172.21.15.7:0/2195763425 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8098056190 con 0x7f80b006a7f0 2023-03-03T19:51:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.788+0000 7f80b49bf700 1 -- 172.21.15.7:0/2195763425 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80b0152e20 con 0x7f80b006a7f0 2023-03-03T19:51:42.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.788+0000 7f80a6ffd700 1 -- 172.21.15.7:0/2195763425 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f80980597b0 con 0x7f80b006a7f0 2023-03-03T19:51:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:51:42.790 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:51:42.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.790+0000 7f80b49bf700 1 -- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80900682f0 msgr2=0x7f809006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:42.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.790+0000 7f80b49bf700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80900682f0 0x7f809006a7a0 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f80a0000f60 tx=0x7f80a0007a40 comp rx=0 tx=0).stop 2023-03-03T19:51:42.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.790+0000 7f80b49bf700 1 -- 172.21.15.7:0/2195763425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b006a7f0 msgr2=0x7f80b014acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:42.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.790+0000 7f80b49bf700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b006a7f0 0x7f80b014acf0 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7f809800bf10 tx=0x7f8098004750 comp rx=0 tx=0).stop 2023-03-03T19:51:42.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.790+0000 7f80b49bf700 1 -- 172.21.15.7:0/2195763425 shutdown_connections 2023-03-03T19:51:42.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.790+0000 7f80b49bf700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80b014de50 0x7f80b006d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.790+0000 7f80b49bf700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f80900682f0 0x7f809006a7a0 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.790+0000 7f80b49bf700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80b006a7f0 0x7f80b014acf0 unknown :-1 s=CLOSED pgs=3626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.790+0000 7f80b49bf700 1 --2- 172.21.15.7:0/2195763425 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80b00c65c0 0x7f80b006cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.790+0000 7f80b49bf700 1 -- 172.21.15.7:0/2195763425 >> 172.21.15.7:0/2195763425 conn(0x7f80b00a9430 msgr2=0x7f80b00a7350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:51:42.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.791+0000 7f80b49bf700 1 -- 172.21.15.7:0/2195763425 shutdown_connections 2023-03-03T19:51:42.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.791+0000 7f80b49bf700 1 -- 172.21.15.7:0/2195763425 wait complete. 2023-03-03T19:51:42.797 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:51:42.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:51:42.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.929+0000 7f812bfa2700 1 Processor -- start 2023-03-03T19:51:42.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.929+0000 7f812bfa2700 1 -- start start 2023-03-03T19:51:42.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812bfa2700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8124151d40 0x7f81240736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812bfa2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81240c2930 0x7f8124157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812bfa2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124073c30 0x7f8124157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812bfa2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812406c2a0 con 0x7f81240c2930 2023-03-03T19:51:42.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812bfa2700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f812406d430 con 0x7f8124151d40 2023-03-03T19:51:42.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812bfa2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812406cb00 con 0x7f8124073c30 2023-03-03T19:51:42.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812953d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81240c2930 0x7f8124157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812953d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81240c2930 0x7f8124157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43408/0 (socket says 172.21.15.7:43408) 2023-03-03T19:51:42.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812953d700 1 -- 172.21.15.7:0/2422601587 learned_addr learned my addr 172.21.15.7:0/2422601587 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:51:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812a53f700 1 --2- 172.21.15.7:0/2422601587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124073c30 0x7f8124157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f8129d3e700 1 --2- 172.21.15.7:0/2422601587 >> v2:172.21.15.143:3300/0 conn(0x7f8124151d40 0x7f81240736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812953d700 1 -- 172.21.15.7:0/2422601587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124073c30 msgr2=0x7f8124157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:51:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812953d700 1 --2- 172.21.15.7:0/2422601587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124073c30 0x7f8124157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812953d700 1 -- 172.21.15.7:0/2422601587 >> v2:172.21.15.143:3300/0 conn(0x7f8124151d40 msgr2=0x7f81240736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:51:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812953d700 1 --2- 172.21.15.7:0/2422601587 >> v2:172.21.15.143:3300/0 conn(0x7f8124151d40 0x7f81240736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.930+0000 7f812953d700 1 -- 172.21.15.7:0/2422601587 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812406cde0 con 0x7f81240c2930 2023-03-03T19:51:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f812953d700 1 --2- 172.21.15.7:0/2422601587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81240c2930 0x7f8124157a90 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7f812406e440 tx=0x7f8120009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e699c80359d1188a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:51:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f8128d3c700 1 -- 172.21.15.7:0/2422601587 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812000b870 con 0x7f81240c2930 2023-03-03T19:51:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f8128d3c700 1 -- 172.21.15.7:0/2422601587 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812000b9d0 con 0x7f81240c2930 2023-03-03T19:51:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f8128d3c700 1 -- 172.21.15.7:0/2422601587 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812000bd60 con 0x7f81240c2930 2023-03-03T19:51:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f812bfa2700 1 -- 172.21.15.7:0/2422601587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81240c2930 msgr2=0x7f8124157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f812bfa2700 1 --2- 172.21.15.7:0/2422601587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81240c2930 0x7f8124157a90 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7f812406e440 tx=0x7f8120009bd0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f812bfa2700 1 -- 172.21.15.7:0/2422601587 shutdown_connections 2023-03-03T19:51:42.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f812bfa2700 1 --2- 172.21.15.7:0/2422601587 >> v2:172.21.15.143:3300/0 conn(0x7f8124151d40 0x7f81240736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f812bfa2700 1 --2- 172.21.15.7:0/2422601587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124073c30 0x7f8124157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f812bfa2700 1 --2- 172.21.15.7:0/2422601587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81240c2930 0x7f8124157a90 unknown :-1 s=CLOSED pgs=3627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f812bfa2700 1 -- 172.21.15.7:0/2422601587 >> 172.21.15.7:0/2422601587 conn(0x7f81240a94d0 msgr2=0x7f81240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:51:42.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f812bfa2700 1 -- 172.21.15.7:0/2422601587 shutdown_connections 2023-03-03T19:51:42.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.931+0000 7f812bfa2700 1 -- 172.21.15.7:0/2422601587 wait complete. 2023-03-03T19:51:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f812bfa2700 1 Processor -- start 2023-03-03T19:51:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f812bfa2700 1 -- start start 2023-03-03T19:51:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f812bfa2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8124073c30 0x7f8124153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f812bfa2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81240c2930 0x7f81240bf830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f812bfa2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124151d40 0x7f81240c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f812bfa2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81240be5b0 con 0x7f8124073c30 2023-03-03T19:51:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f812bfa2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8124154bf0 con 0x7f81240c2930 2023-03-03T19:51:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f812bfa2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8124154ed0 con 0x7f8124151d40 2023-03-03T19:51:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f8129d3e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8124073c30 0x7f8124153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f8129d3e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8124073c30 0x7f8124153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43414/0 (socket says 172.21.15.7:43414) 2023-03-03T19:51:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f8129d3e700 1 -- 172.21.15.7:0/3019569394 learned_addr learned my addr 172.21.15.7:0/3019569394 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:51:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.933+0000 7f812953d700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81240c2930 0x7f81240bf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f812a53f700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124151d40 0x7f81240c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f8129d3e700 1 -- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124151d40 msgr2=0x7f81240c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f8129d3e700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124151d40 0x7f81240c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f8129d3e700 1 -- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81240c2930 msgr2=0x7f81240bf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f8129d3e700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81240c2930 0x7f81240bf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f8129d3e700 1 -- 172.21.15.7:0/3019569394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8120009550 con 0x7f8124073c30 2023-03-03T19:51:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f812a53f700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124151d40 0x7f81240c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:51:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f8129d3e700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8124073c30 0x7f8124153260 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 crypto rx=0x7f811400d800 tx=0x7f811400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:51:42.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f811a7fc700 1 -- 172.21.15.7:0/3019569394 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81140125f0 con 0x7f8124073c30 2023-03-03T19:51:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f811a7fc700 1 -- 172.21.15.7:0/3019569394 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8114012750 con 0x7f8124073c30 2023-03-03T19:51:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f812bfa2700 1 -- 172.21.15.7:0/3019569394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812406bea0 con 0x7f8124073c30 2023-03-03T19:51:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f811a7fc700 1 -- 172.21.15.7:0/3019569394 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8114012b00 con 0x7f8124073c30 2023-03-03T19:51:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.934+0000 7f812bfa2700 1 -- 172.21.15.7:0/3019569394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812406d0c0 con 0x7f8124073c30 2023-03-03T19:51:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.935+0000 7f812bfa2700 1 -- 172.21.15.7:0/3019569394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812406c030 con 0x7f8124073c30 2023-03-03T19:51:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.935+0000 7f811a7fc700 1 -- 172.21.15.7:0/3019569394 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8114012c60 con 0x7f8124073c30 2023-03-03T19:51:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.936+0000 7f811a7fc700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81080682f0 0x7f810806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:42.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.936+0000 7f811a7fc700 1 -- 172.21.15.7:0/3019569394 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f81140865d0 con 0x7f8124073c30 2023-03-03T19:51:42.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.936+0000 7f812953d700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81080682f0 0x7f810806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:42.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.937+0000 7f812953d700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81080682f0 0x7f810806a7a0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f8120000c00 tx=0x7f8120008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:51:42.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:42.940+0000 7f811a7fc700 1 -- 172.21.15.7:0/3019569394 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f81140528f0 con 0x7f8124073c30 2023-03-03T19:51:43.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.179+0000 7f812bfa2700 1 -- 172.21.15.7:0/3019569394 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81240bebe0 con 0x7f8124073c30 2023-03-03T19:51:43.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.180+0000 7f811a7fc700 1 -- 172.21.15.7:0/3019569394 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8114055f10 con 0x7f8124073c30 2023-03-03T19:51:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:51:43.181 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:51:43.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 -- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81080682f0 msgr2=0x7f810806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:43.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81080682f0 0x7f810806a7a0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f8120000c00 tx=0x7f8120008030 comp rx=0 tx=0).stop 2023-03-03T19:51:43.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 -- 172.21.15.7:0/3019569394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8124073c30 msgr2=0x7f8124153260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:43.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8124073c30 0x7f8124153260 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 crypto rx=0x7f811400d800 tx=0x7f811400db10 comp rx=0 tx=0).stop 2023-03-03T19:51:43.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 -- 172.21.15.7:0/3019569394 shutdown_connections 2023-03-03T19:51:43.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8124151d40 0x7f81240c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:43.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f81080682f0 0x7f810806a7a0 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:43.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8124073c30 0x7f8124153260 unknown :-1 s=CLOSED pgs=3628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:43.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 --2- 172.21.15.7:0/3019569394 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81240c2930 0x7f81240bf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:43.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 -- 172.21.15.7:0/3019569394 >> 172.21.15.7:0/3019569394 conn(0x7f81240a94d0 msgr2=0x7f81240a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:51:43.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 -- 172.21.15.7:0/3019569394 shutdown_connections 2023-03-03T19:51:43.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:43.182+0000 7f812bfa2700 1 -- 172.21.15.7:0/3019569394 wait complete. 2023-03-03T19:51:43.184 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:51:43.195 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before thrashing 2023-03-03T19:51:47.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:51:47.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.335+0000 7f68d7f70700 1 Processor -- start 2023-03-03T19:51:47.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.335+0000 7f68d7f70700 1 -- start start 2023-03-03T19:51:47.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.336+0000 7f68d7f70700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d014b020 0x7f68d00717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:47.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.336+0000 7f68d7f70700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d00728b0 con 0x7f68d0071d20 2023-03-03T19:51:47.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.336+0000 7f68d7f70700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d0155070 con 0x7f68d00aad20 2023-03-03T19:51:47.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.336+0000 7f68d7f70700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d006f5b0 con 0x7f68d014b020 2023-03-03T19:51:47.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.336+0000 7f68d650d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f68d0071d20 0x7f68d01583b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37412/0 (socket says 172.21.15.7:37412) 2023-03-03T19:51:47.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.336+0000 7f68d650d700 1 -- 172.21.15.7:0/3520995246 learned_addr learned my addr 172.21.15.7:0/3520995246 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:51:47.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.336+0000 7f68d5d0c700 1 --2- 172.21.15.7:0/3520995246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d014b020 0x7f68d00717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:47.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.336+0000 7f68d4d0a700 1 -- 172.21.15.7:0/3520995246 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048314295 0 0) 0x7f68d00728b0 con 0x7f68d0071d20 2023-03-03T19:51:47.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.336+0000 7f68d4d0a700 1 -- 172.21.15.7:0/3520995246 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68bc0035b0 con 0x7f68d0071d20 2023-03-03T19:51:47.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.337+0000 7f68d4d0a700 1 -- 172.21.15.7:0/3520995246 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778544083 0 0) 0x7f68d0155070 con 0x7f68d00aad20 2023-03-03T19:51:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.337+0000 7f68d4d0a700 1 -- 172.21.15.7:0/3520995246 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d00728b0 con 0x7f68d00aad20 2023-03-03T19:51:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.337+0000 7f68d5d0c700 1 -- 172.21.15.7:0/3520995246 >> v1:172.21.15.143:6789/0 conn(0x7f68d00aad20 legacy=0x7f68d0157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.337+0000 7f68d5d0c700 1 -- 172.21.15.7:0/3520995246 >> v1:172.21.15.7:6789/0 conn(0x7f68d0071d20 legacy=0x7f68d01583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.337+0000 7f68d5d0c700 1 -- 172.21.15.7:0/3520995246 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d006f890 con 0x7f68d014b020 2023-03-03T19:51:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.337+0000 7f68d5d0c700 1 --2- 172.21.15.7:0/3520995246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d014b020 0x7f68d00717e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009870 tx=0x7f68c0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a21af04aa4d3770 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:51:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.337+0000 7f68d4d0a700 1 -- 172.21.15.7:0/3520995246 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0011830 con 0x7f68d014b020 2023-03-03T19:51:47.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.337+0000 7f68d4d0a700 1 -- 172.21.15.7:0/3520995246 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c0011990 con 0x7f68d014b020 2023-03-03T19:51:47.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.337+0000 7f68d4d0a700 1 -- 172.21.15.7:0/3520995246 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0011d20 con 0x7f68d014b020 2023-03-03T19:51:47.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.338+0000 7f68d7f70700 1 -- 172.21.15.7:0/3520995246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d014b020 msgr2=0x7f68d00717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:47.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.338+0000 7f68d7f70700 1 --2- 172.21.15.7:0/3520995246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d014b020 0x7f68d00717e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009870 tx=0x7f68c0009b80 comp rx=0 tx=0).stop 2023-03-03T19:51:47.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.338+0000 7f68d7f70700 1 -- 172.21.15.7:0/3520995246 shutdown_connections 2023-03-03T19:51:47.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.338+0000 7f68d7f70700 1 --2- 172.21.15.7:0/3520995246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d014b020 0x7f68d00717e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:47.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.338+0000 7f68d7f70700 1 -- 172.21.15.7:0/3520995246 >> 172.21.15.7:0/3520995246 conn(0x7f68d00c8900 msgr2=0x7f68d00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:51:47.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.338+0000 7f68d7f70700 1 -- 172.21.15.7:0/3520995246 shutdown_connections 2023-03-03T19:51:47.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.338+0000 7f68d7f70700 1 -- 172.21.15.7:0/3520995246 wait complete. 2023-03-03T19:51:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.339+0000 7f68d7f70700 1 Processor -- start 2023-03-03T19:51:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.339+0000 7f68d7f70700 1 -- start start 2023-03-03T19:51:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d7f70700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d0071d20 0x7f68d01e93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d7f70700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d00aad20 0x7f68d01560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d7f70700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d014b020 0x7f68d014b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d7f70700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d01ec9b0 con 0x7f68d014b020 2023-03-03T19:51:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d7f70700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d01ecc90 con 0x7f68d00aad20 2023-03-03T19:51:47.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d7f70700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d01ecf70 con 0x7f68d0071d20 2023-03-03T19:51:47.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d650d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d014b020 0x7f68d014b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:47.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d650d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d014b020 0x7f68d014b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43438/0 (socket says 172.21.15.7:43438) 2023-03-03T19:51:47.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d650d700 1 -- 172.21.15.7:0/2220915374 learned_addr learned my addr 172.21.15.7:0/2220915374 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:51:47.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d5d0c700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d0071d20 0x7f68d01e93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:47.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d550b700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d00aad20 0x7f68d01560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:47.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d650d700 1 -- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d0071d20 msgr2=0x7f68d01e93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:47.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d650d700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d0071d20 0x7f68d01e93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d650d700 1 -- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d00aad20 msgr2=0x7f68d01560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d650d700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d00aad20 0x7f68d01560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d650d700 1 -- 172.21.15.7:0/2220915374 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c00094a0 con 0x7f68d014b020 2023-03-03T19:51:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d650d700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d014b020 0x7f68d014b630 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 crypto rx=0x7f68c800cba0 tx=0x7f68c800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:51:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68c67fc700 1 -- 172.21.15.7:0/2220915374 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c800aa40 con 0x7f68d014b020 2023-03-03T19:51:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68c67fc700 1 -- 172.21.15.7:0/2220915374 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c800aba0 con 0x7f68d014b020 2023-03-03T19:51:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.340+0000 7f68d7f70700 1 -- 172.21.15.7:0/2220915374 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d0155350 con 0x7f68d014b020 2023-03-03T19:51:47.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.341+0000 7f68c67fc700 1 -- 172.21.15.7:0/2220915374 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8012400 con 0x7f68d014b020 2023-03-03T19:51:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.341+0000 7f68d7f70700 1 -- 172.21.15.7:0/2220915374 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d006fb70 con 0x7f68d014b020 2023-03-03T19:51:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.341+0000 7f68d7f70700 1 -- 172.21.15.7:0/2220915374 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d0074bb0 con 0x7f68d014b020 2023-03-03T19:51:47.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.342+0000 7f68c67fc700 1 -- 172.21.15.7:0/2220915374 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f68c8019030 con 0x7f68d014b020 2023-03-03T19:51:47.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.342+0000 7f68c67fc700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68b40682f0 0x7f68b406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:51:47.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.342+0000 7f68c67fc700 1 -- 172.21.15.7:0/2220915374 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f68c8085d10 con 0x7f68d014b020 2023-03-03T19:51:47.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.343+0000 7f68d5d0c700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68b40682f0 0x7f68b406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:51:47.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.343+0000 7f68d5d0c700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68b40682f0 0x7f68b406a7a0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f68c00090b0 tx=0x7f68c001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:51:47.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.347+0000 7f68c67fc700 1 -- 172.21.15.7:0/2220915374 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68c8052030 con 0x7f68d014b020 2023-03-03T19:51:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.583+0000 7f68d7f70700 1 -- 172.21.15.7:0/2220915374 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68d014c6e0 con 0x7f68d014b020 2023-03-03T19:51:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.584+0000 7f68c67fc700 1 -- 172.21.15.7:0/2220915374 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f68c8055650 con 0x7f68d014b020 2023-03-03T19:51:47.585 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:51:47.586 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:51:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.585+0000 7f68d7f70700 1 -- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68b40682f0 msgr2=0x7f68b406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.585+0000 7f68d7f70700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68b40682f0 0x7f68b406a7a0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f68c00090b0 tx=0x7f68c001c040 comp rx=0 tx=0).stop 2023-03-03T19:51:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.585+0000 7f68d7f70700 1 -- 172.21.15.7:0/2220915374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d014b020 msgr2=0x7f68d014b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:51:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.585+0000 7f68d7f70700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d014b020 0x7f68d014b630 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 crypto rx=0x7f68c800cba0 tx=0x7f68c800ceb0 comp rx=0 tx=0).stop 2023-03-03T19:51:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.585+0000 7f68d7f70700 1 -- 172.21.15.7:0/2220915374 shutdown_connections 2023-03-03T19:51:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.585+0000 7f68d7f70700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68b40682f0 0x7f68b406a7a0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.585+0000 7f68d7f70700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f68d014b020 0x7f68d014b630 unknown :-1 s=CLOSED pgs=3630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.585+0000 7f68d7f70700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f68d0071d20 0x7f68d01e93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.585+0000 7f68d7f70700 1 --2- 172.21.15.7:0/2220915374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68d00aad20 0x7f68d01560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:51:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.585+0000 7f68d7f70700 1 -- 172.21.15.7:0/2220915374 >> 172.21.15.7:0/2220915374 conn(0x7f68d00c8900 msgr2=0x7f68d0158360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:51:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.586+0000 7f68d7f70700 1 -- 172.21.15.7:0/2220915374 shutdown_connections 2023-03-03T19:51:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:51:47.586+0000 7f68d7f70700 1 -- 172.21.15.7:0/2220915374 wait complete. 2023-03-03T19:51:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:51:47.599 INFO:tasks.mds_thrash.fs.[b]:waiting for 92 secs before thrashing 2023-03-03T19:51:53.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:51:53.291 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:52:05.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:52:05.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.212+0000 7f704437f700 1 Processor -- start 2023-03-03T19:52:05.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.212+0000 7f704437f700 1 -- start start 2023-03-03T19:52:05.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.212+0000 7f704437f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703c0c17f0 0x7f703c06d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:05.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.212+0000 7f704437f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f703c0c5190 0x7f703c06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:05.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704437f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c14ebe0 con 0x7f703c0c6010 2023-03-03T19:52:05.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704437f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f703c152100 con 0x7f703c0c5190 2023-03-03T19:52:05.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704437f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703c0c2e70 con 0x7f703c0c17f0 2023-03-03T19:52:05.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704291c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f703c0c5190 0x7f703c06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:52:05.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704211b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f703c0c6010 0x7f703c0c4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37434/0 (socket says 172.21.15.7:37434) 2023-03-03T19:52:05.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704211b700 1 -- 172.21.15.7:0/88259111 learned_addr learned my addr 172.21.15.7:0/88259111 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:52:05.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704291c700 1 -- 172.21.15.7:0/88259111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703c0c17f0 msgr2=0x7f703c06d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:52:05.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704191a700 1 --2- 172.21.15.7:0/88259111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703c0c17f0 0x7f703c06d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:52:05.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704291c700 1 --2- 172.21.15.7:0/88259111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703c0c17f0 0x7f703c06d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:05.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f7041119700 1 -- 172.21.15.7:0/88259111 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586213679 0 0) 0x7f703c14ebe0 con 0x7f703c0c6010 2023-03-03T19:52:05.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704291c700 1 -- 172.21.15.7:0/88259111 >> v1:172.21.15.7:6789/0 conn(0x7f703c0c6010 legacy=0x7f703c0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:05.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704291c700 1 -- 172.21.15.7:0/88259111 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c0aa680 con 0x7f703c0c5190 2023-03-03T19:52:05.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f704291c700 1 --2- 172.21.15.7:0/88259111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c5190 0x7f703c06dbe0 secure :-1 s=READY pgs=3632 cs=0 l=1 rev1=1 crypto rx=0x7f7034009d90 tx=0x7f703400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea72d8a48a15a38 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:52:05.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f7041119700 1 -- 172.21.15.7:0/88259111 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703400e070 con 0x7f703c0c5190 2023-03-03T19:52:05.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f7041119700 1 -- 172.21.15.7:0/88259111 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703400bbd0 con 0x7f703c0c5190 2023-03-03T19:52:05.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.213+0000 7f7041119700 1 -- 172.21.15.7:0/88259111 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034012430 con 0x7f703c0c5190 2023-03-03T19:52:05.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.214+0000 7f704437f700 1 -- 172.21.15.7:0/88259111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c5190 msgr2=0x7f703c06dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:05.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.214+0000 7f704437f700 1 --2- 172.21.15.7:0/88259111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c5190 0x7f703c06dbe0 secure :-1 s=READY pgs=3632 cs=0 l=1 rev1=1 crypto rx=0x7f7034009d90 tx=0x7f703400b330 comp rx=0 tx=0).stop 2023-03-03T19:52:05.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.214+0000 7f704437f700 1 -- 172.21.15.7:0/88259111 shutdown_connections 2023-03-03T19:52:05.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.214+0000 7f704437f700 1 --2- 172.21.15.7:0/88259111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c5190 0x7f703c06dbe0 unknown :-1 s=CLOSED pgs=3632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:05.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.214+0000 7f704437f700 1 --2- 172.21.15.7:0/88259111 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703c0c17f0 0x7f703c06d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:05.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.214+0000 7f704437f700 1 -- 172.21.15.7:0/88259111 >> 172.21.15.7:0/88259111 conn(0x7f703c0a9430 msgr2=0x7f703c0c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:52:05.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.214+0000 7f704437f700 1 -- 172.21.15.7:0/88259111 shutdown_connections 2023-03-03T19:52:05.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.214+0000 7f704437f700 1 -- 172.21.15.7:0/88259111 wait complete. 2023-03-03T19:52:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.215+0000 7f704437f700 1 Processor -- start 2023-03-03T19:52:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.215+0000 7f704437f700 1 -- start start 2023-03-03T19:52:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704437f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c17f0 0x7f703c1ded90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704437f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703c0c5190 0x7f703c149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704437f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f703c0c6010 0x7f703c14a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704437f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c153520 con 0x7f703c0c17f0 2023-03-03T19:52:05.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704437f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c06fbb0 con 0x7f703c0c6010 2023-03-03T19:52:05.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704437f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703c14a8d0 con 0x7f703c0c5190 2023-03-03T19:52:05.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704211b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c17f0 0x7f703c1ded90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:52:05.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704211b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c17f0 0x7f703c1ded90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43460/0 (socket says 172.21.15.7:43460) 2023-03-03T19:52:05.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704211b700 1 -- 172.21.15.7:0/1057832327 learned_addr learned my addr 172.21.15.7:0/1057832327 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:52:05.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704211b700 1 -- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703c0c5190 msgr2=0x7f703c149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:05.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704211b700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703c0c5190 0x7f703c149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:05.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704211b700 1 -- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f703c0c6010 msgr2=0x7f703c14a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:05.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704211b700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f703c0c6010 0x7f703c14a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:05.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704211b700 1 -- 172.21.15.7:0/1057832327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7034009a70 con 0x7f703c0c17f0 2023-03-03T19:52:05.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.216+0000 7f704211b700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c17f0 0x7f703c1ded90 secure :-1 s=READY pgs=3633 cs=0 l=1 rev1=1 crypto rx=0x7f703800c750 tx=0x7f703800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:52:05.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.217+0000 7f702effd700 1 -- 172.21.15.7:0/1057832327 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038004c80 con 0x7f703c0c17f0 2023-03-03T19:52:05.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.217+0000 7f702effd700 1 -- 172.21.15.7:0/1057832327 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7038004e00 con 0x7f703c0c17f0 2023-03-03T19:52:05.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.217+0000 7f704437f700 1 -- 172.21.15.7:0/1057832327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c0c2b00 con 0x7f703c0c17f0 2023-03-03T19:52:05.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.217+0000 7f702effd700 1 -- 172.21.15.7:0/1057832327 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038007310 con 0x7f703c0c17f0 2023-03-03T19:52:05.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.217+0000 7f704437f700 1 -- 172.21.15.7:0/1057832327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c0aa960 con 0x7f703c0c17f0 2023-03-03T19:52:05.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.218+0000 7f704437f700 1 -- 172.21.15.7:0/1057832327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703c0c32e0 con 0x7f703c0c17f0 2023-03-03T19:52:05.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.218+0000 7f702effd700 1 -- 172.21.15.7:0/1057832327 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f703801a020 con 0x7f703c0c17f0 2023-03-03T19:52:05.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.218+0000 7f702effd700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70200682f0 0x7f702006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:05.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.219+0000 7f702effd700 1 -- 172.21.15.7:0/1057832327 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7038085d90 con 0x7f703c0c17f0 2023-03-03T19:52:05.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.219+0000 7f704191a700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70200682f0 0x7f702006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:52:05.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.220+0000 7f704191a700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70200682f0 0x7f702006a7a0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f703c06fb40 tx=0x7f7030008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:52:05.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.223+0000 7f702effd700 1 -- 172.21.15.7:0/1057832327 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f70380520d0 con 0x7f703c0c17f0 2023-03-03T19:52:05.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.459+0000 7f704437f700 1 -- 172.21.15.7:0/1057832327 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f703c1e96b0 con 0x7f703c0c17f0 2023-03-03T19:52:05.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.460+0000 7f702effd700 1 -- 172.21.15.7:0/1057832327 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f703c1e96b0 con 0x7f703c0c17f0 2023-03-03T19:52:05.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:52:05.461 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:52:05.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 -- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70200682f0 msgr2=0x7f702006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:05.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70200682f0 0x7f702006a7a0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f703c06fb40 tx=0x7f7030008040 comp rx=0 tx=0).stop 2023-03-03T19:52:05.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 -- 172.21.15.7:0/1057832327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c17f0 msgr2=0x7f703c1ded90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:05.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c17f0 0x7f703c1ded90 secure :-1 s=READY pgs=3633 cs=0 l=1 rev1=1 crypto rx=0x7f703800c750 tx=0x7f703800cb10 comp rx=0 tx=0).stop 2023-03-03T19:52:05.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 -- 172.21.15.7:0/1057832327 shutdown_connections 2023-03-03T19:52:05.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f703c0c6010 0x7f703c14a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:05.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f70200682f0 0x7f702006a7a0 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:05.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f703c0c17f0 0x7f703c1ded90 unknown :-1 s=CLOSED pgs=3633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:05.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 --2- 172.21.15.7:0/1057832327 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f703c0c5190 0x7f703c149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:05.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 -- 172.21.15.7:0/1057832327 >> 172.21.15.7:0/1057832327 conn(0x7f703c0a9430 msgr2=0x7f703c06db60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:52:05.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 -- 172.21.15.7:0/1057832327 shutdown_connections 2023-03-03T19:52:05.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:05.462+0000 7f704437f700 1 -- 172.21.15.7:0/1057832327 wait complete. 2023-03-03T19:52:05.465 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:52:05.474 INFO:tasks.mds_thrash.fs.[a]:waiting for 54 secs before thrashing 2023-03-03T19:52:23.400 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:52:23.404 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:52:53.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:52:53.479 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:52:59.476 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:52:59.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.614+0000 7ff0e6744700 1 Processor -- start 2023-03-03T19:52:59.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.615+0000 7ff0e6744700 1 -- start start 2023-03-03T19:52:59.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.615+0000 7ff0e6744700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0e0151b20 0x7ff0e00736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:59.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.615+0000 7ff0e6744700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e006cb70 con 0x7ff0e0073c30 2023-03-03T19:52:59.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.615+0000 7ff0e6744700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e006ce50 con 0x7ff0e00c2950 2023-03-03T19:52:59.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.615+0000 7ff0e6744700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e006c2f0 con 0x7ff0e0151b20 2023-03-03T19:52:59.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.615+0000 7ff0dffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff0e0151b20 0x7ff0e00736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:52:59.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.615+0000 7ff0e4ce1700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff0e0073c30 0x7ff0e01581f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37500/0 (socket says 172.21.15.7:37500) 2023-03-03T19:52:59.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.616+0000 7ff0e4ce1700 1 -- 172.21.15.7:0/1728470437 learned_addr learned my addr 172.21.15.7:0/1728470437 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:52:59.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.616+0000 7ff0dffff700 1 -- 172.21.15.7:0/1728470437 >> v1:172.21.15.143:6789/0 conn(0x7ff0e00c2950 legacy=0x7ff0e0157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:52:59.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.616+0000 7ff0dffff700 1 -- 172.21.15.7:0/1728470437 >> v1:172.21.15.7:6789/0 conn(0x7ff0e0073c30 legacy=0x7ff0e01581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:59.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.616+0000 7ff0dffff700 1 -- 172.21.15.7:0/1728470437 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e006d130 con 0x7ff0e0151b20 2023-03-03T19:52:59.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.616+0000 7ff0deffd700 1 -- 172.21.15.7:0/1728470437 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499614961 0 0) 0x7ff0e006cb70 con 0x7ff0e0073c30 2023-03-03T19:52:59.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.616+0000 7ff0dffff700 1 --2- 172.21.15.7:0/1728470437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 0x7ff0e00736f0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009840 tx=0x7ff0d0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b1d7353a89a475 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:52:59.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.616+0000 7ff0deffd700 1 -- 172.21.15.7:0/1728470437 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0004230 con 0x7ff0e0151b20 2023-03-03T19:52:59.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.616+0000 7ff0deffd700 1 -- 172.21.15.7:0/1728470437 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d0004390 con 0x7ff0e0151b20 2023-03-03T19:52:59.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.616+0000 7ff0deffd700 1 -- 172.21.15.7:0/1728470437 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d00036a0 con 0x7ff0e0151b20 2023-03-03T19:52:59.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.617+0000 7ff0e6744700 1 -- 172.21.15.7:0/1728470437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 msgr2=0x7ff0e00736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:59.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.617+0000 7ff0e6744700 1 --2- 172.21.15.7:0/1728470437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 0x7ff0e00736f0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009840 tx=0x7ff0d0009b50 comp rx=0 tx=0).stop 2023-03-03T19:52:59.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.617+0000 7ff0e6744700 1 -- 172.21.15.7:0/1728470437 shutdown_connections 2023-03-03T19:52:59.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.617+0000 7ff0e6744700 1 --2- 172.21.15.7:0/1728470437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 0x7ff0e00736f0 unknown :-1 s=CLOSED pgs=3636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:59.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.617+0000 7ff0e6744700 1 -- 172.21.15.7:0/1728470437 >> 172.21.15.7:0/1728470437 conn(0x7ff0e00a94d0 msgr2=0x7ff0e00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:52:59.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.617+0000 7ff0e6744700 1 -- 172.21.15.7:0/1728470437 shutdown_connections 2023-03-03T19:52:59.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.617+0000 7ff0e6744700 1 -- 172.21.15.7:0/1728470437 wait complete. 2023-03-03T19:52:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.618+0000 7ff0e6744700 1 Processor -- start 2023-03-03T19:52:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.618+0000 7ff0e6744700 1 -- start start 2023-03-03T19:52:59.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e6744700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e0073c30 0x7ff0e01523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e6744700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e00c2950 0x7ff0e00c0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e6744700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 0x7ff0e01506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e6744700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00bf640 con 0x7ff0e0151b20 2023-03-03T19:52:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e6744700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00c0730 con 0x7ff0e0073c30 2023-03-03T19:52:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e6744700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e00bf2a0 con 0x7ff0e00c2950 2023-03-03T19:52:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e4ce1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 0x7ff0e01506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:52:59.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e4ce1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 0x7ff0e01506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43526/0 (socket says 172.21.15.7:43526) 2023-03-03T19:52:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e4ce1700 1 -- 172.21.15.7:0/257472662 learned_addr learned my addr 172.21.15.7:0/257472662 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:52:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0dffff700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e0073c30 0x7ff0e01523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:52:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0df7fe700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e00c2950 0x7ff0e00c0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:52:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e4ce1700 1 -- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e00c2950 msgr2=0x7ff0e00c0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e4ce1700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e00c2950 0x7ff0e00c0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e4ce1700 1 -- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e0073c30 msgr2=0x7ff0e01523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:59.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e4ce1700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e0073c30 0x7ff0e01523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:59.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0e4ce1700 1 -- 172.21.15.7:0/257472662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d00094a0 con 0x7ff0e0151b20 2023-03-03T19:52:59.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.619+0000 7ff0df7fe700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e00c2950 0x7ff0e00c0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:52:59.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.620+0000 7ff0e4ce1700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 0x7ff0e01506c0 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800cb60 tx=0x7ff0d800ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:52:59.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.620+0000 7ff0dcff9700 1 -- 172.21.15.7:0/257472662 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d800aa40 con 0x7ff0e0151b20 2023-03-03T19:52:59.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.620+0000 7ff0dcff9700 1 -- 172.21.15.7:0/257472662 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d800aba0 con 0x7ff0e0151b20 2023-03-03T19:52:59.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.620+0000 7ff0e6744700 1 -- 172.21.15.7:0/257472662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e0151ef0 con 0x7ff0e0151b20 2023-03-03T19:52:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.620+0000 7ff0dcff9700 1 -- 172.21.15.7:0/257472662 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80070d0 con 0x7ff0e0151b20 2023-03-03T19:52:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.620+0000 7ff0e6744700 1 -- 172.21.15.7:0/257472662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e006d410 con 0x7ff0e0151b20 2023-03-03T19:52:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.621+0000 7ff0e6744700 1 -- 172.21.15.7:0/257472662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e006bea0 con 0x7ff0e0151b20 2023-03-03T19:52:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.621+0000 7ff0dcff9700 1 -- 172.21.15.7:0/257472662 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff0d8010070 con 0x7ff0e0151b20 2023-03-03T19:52:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.622+0000 7ff0dcff9700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 0x7ff0cc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:52:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.622+0000 7ff0dcff9700 1 -- 172.21.15.7:0/257472662 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff0d8085ad0 con 0x7ff0e0151b20 2023-03-03T19:52:59.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.622+0000 7ff0dffff700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 0x7ff0cc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:52:59.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.622+0000 7ff0dffff700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 0x7ff0cc06a7a0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00090b0 tx=0x7ff0d001c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:52:59.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.626+0000 7ff0dcff9700 1 -- 172.21.15.7:0/257472662 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff0d8051e70 con 0x7ff0e0151b20 2023-03-03T19:52:59.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.865+0000 7ff0e6744700 1 -- 172.21.15.7:0/257472662 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0e01528f0 con 0x7ff0e0151b20 2023-03-03T19:52:59.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.865+0000 7ff0dcff9700 1 -- 172.21.15.7:0/257472662 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff0d8018090 con 0x7ff0e0151b20 2023-03-03T19:52:59.867 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:52:59.868 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:52:59.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 -- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 msgr2=0x7ff0cc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:59.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 0x7ff0cc06a7a0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00090b0 tx=0x7ff0d001c040 comp rx=0 tx=0).stop 2023-03-03T19:52:59.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 -- 172.21.15.7:0/257472662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 msgr2=0x7ff0e01506c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:52:59.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 0x7ff0e01506c0 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800cb60 tx=0x7ff0d800ce70 comp rx=0 tx=0).stop 2023-03-03T19:52:59.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 -- 172.21.15.7:0/257472662 shutdown_connections 2023-03-03T19:52:59.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff0cc0682f0 0x7ff0cc06a7a0 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:59.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0e0151b20 0x7ff0e01506c0 unknown :-1 s=CLOSED pgs=3637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:59.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0e0073c30 0x7ff0e01523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:59.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 --2- 172.21.15.7:0/257472662 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff0e00c2950 0x7ff0e00c0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:52:59.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 -- 172.21.15.7:0/257472662 >> 172.21.15.7:0/257472662 conn(0x7ff0e00a94d0 msgr2=0x7ff0e0158180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:52:59.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 -- 172.21.15.7:0/257472662 shutdown_connections 2023-03-03T19:52:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:52:59.867+0000 7ff0e6744700 1 -- 172.21.15.7:0/257472662 wait complete. 2023-03-03T19:52:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:52:59.881 INFO:tasks.mds_thrash.fs.[a]:waiting for 40 secs before thrashing 2023-03-03T19:53:19.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:53:19.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.739+0000 7fd24aaab700 1 Processor -- start 2023-03-03T19:53:19.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.739+0000 7fd24aaab700 1 -- start start 2023-03-03T19:53:19.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.740+0000 7fd24aaab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd244071e80 0x7fd244151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:19.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.740+0000 7fd24aaab700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd244072250 0x7fd24414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:19.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.740+0000 7fd24aaab700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2441510d0 con 0x7fd2440c1ce0 2023-03-03T19:53:19.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.740+0000 7fd24aaab700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2440c3740 con 0x7fd244072250 2023-03-03T19:53:19.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.740+0000 7fd24aaab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd244070670 con 0x7fd244071e80 2023-03-03T19:53:19.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.740+0000 7fd243fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd244072250 0x7fd24414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:19.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.740+0000 7fd243fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd244072250 0x7fd24414e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43546/0 (socket says 172.21.15.7:43546) 2023-03-03T19:53:19.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.740+0000 7fd243fff700 1 -- 172.21.15.7:0/3793902129 learned_addr learned my addr 172.21.15.7:0/3793902129 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:53:19.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.740+0000 7fd248847700 1 --2- 172.21.15.7:0/3793902129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd244071e80 0x7fd244151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:19.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.741+0000 7fd243fff700 1 -- 172.21.15.7:0/3793902129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd244071e80 msgr2=0x7fd244151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:53:19.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.741+0000 7fd243fff700 1 --2- 172.21.15.7:0/3793902129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd244071e80 0x7fd244151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:19.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.741+0000 7fd243fff700 1 -- 172.21.15.7:0/3793902129 >> v1:172.21.15.143:6789/0 conn(0x7fd2440c1ce0 legacy=0x7fd2441546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:19.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.741+0000 7fd243fff700 1 -- 172.21.15.7:0/3793902129 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2440c39e0 con 0x7fd244072250 2023-03-03T19:53:19.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.741+0000 7fd243fff700 1 --2- 172.21.15.7:0/3793902129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 0x7fd24414e1f0 secure :-1 s=READY pgs=3638 cs=0 l=1 rev1=1 crypto rx=0x7fd2300098f0 tx=0x7fd230009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e563418e1943aff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:53:19.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.741+0000 7fd2437fe700 1 -- 172.21.15.7:0/3793902129 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230011830 con 0x7fd244072250 2023-03-03T19:53:19.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.741+0000 7fd2437fe700 1 -- 172.21.15.7:0/3793902129 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd230011990 con 0x7fd244072250 2023-03-03T19:53:19.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.741+0000 7fd2437fe700 1 -- 172.21.15.7:0/3793902129 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230011d20 con 0x7fd244072250 2023-03-03T19:53:19.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.742+0000 7fd24aaab700 1 -- 172.21.15.7:0/3793902129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 msgr2=0x7fd24414e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:19.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.742+0000 7fd24aaab700 1 --2- 172.21.15.7:0/3793902129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 0x7fd24414e1f0 secure :-1 s=READY pgs=3638 cs=0 l=1 rev1=1 crypto rx=0x7fd2300098f0 tx=0x7fd230009c00 comp rx=0 tx=0).stop 2023-03-03T19:53:19.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.742+0000 7fd24aaab700 1 -- 172.21.15.7:0/3793902129 shutdown_connections 2023-03-03T19:53:19.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.742+0000 7fd24aaab700 1 --2- 172.21.15.7:0/3793902129 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd244071e80 0x7fd244151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:19.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.742+0000 7fd24aaab700 1 --2- 172.21.15.7:0/3793902129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 0x7fd24414e1f0 unknown :-1 s=CLOSED pgs=3638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:19.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.742+0000 7fd24aaab700 1 -- 172.21.15.7:0/3793902129 >> 172.21.15.7:0/3793902129 conn(0x7fd2440a94d0 msgr2=0x7fd2440a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:53:19.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.742+0000 7fd24aaab700 1 -- 172.21.15.7:0/3793902129 shutdown_connections 2023-03-03T19:53:19.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.742+0000 7fd24aaab700 1 -- 172.21.15.7:0/3793902129 wait complete. 2023-03-03T19:53:19.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.743+0000 7fd24aaab700 1 Processor -- start 2023-03-03T19:53:19.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.743+0000 7fd24aaab700 1 -- start start 2023-03-03T19:53:19.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.744+0000 7fd24aaab700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd244071e80 0x7fd2440c2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:19.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd24aaab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 0x7fd244142b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:19.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd24aaab700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2440c1ce0 0x7fd2441430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:19.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd24aaab700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2441435f0 con 0x7fd244072250 2023-03-03T19:53:19.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd24aaab700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd244146fa0 con 0x7fd244071e80 2023-03-03T19:53:19.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd24aaab700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd244143750 con 0x7fd2440c1ce0 2023-03-03T19:53:19.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd243fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 0x7fd244142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:19.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd243fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 0x7fd244142b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43552/0 (socket says 172.21.15.7:43552) 2023-03-03T19:53:19.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd243fff700 1 -- 172.21.15.7:0/3252632293 learned_addr learned my addr 172.21.15.7:0/3252632293 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:53:19.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd248847700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd244071e80 0x7fd2440c2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:19.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd249048700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2440c1ce0 0x7fd2441430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:19.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd243fff700 1 -- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2440c1ce0 msgr2=0x7fd2441430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:19.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd243fff700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2440c1ce0 0x7fd2441430b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:19.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd243fff700 1 -- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd244071e80 msgr2=0x7fd2440c2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:19.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd243fff700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd244071e80 0x7fd2440c2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:19.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd243fff700 1 -- 172.21.15.7:0/3252632293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd230009550 con 0x7fd244072250 2023-03-03T19:53:19.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd249048700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2440c1ce0 0x7fd2441430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:53:19.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd243fff700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 0x7fd244142b70 secure :-1 s=READY pgs=3639 cs=0 l=1 rev1=1 crypto rx=0x7fd230009c40 tx=0x7fd230004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:53:19.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd2417fa700 1 -- 172.21.15.7:0/3252632293 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230004480 con 0x7fd244072250 2023-03-03T19:53:19.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd2417fa700 1 -- 172.21.15.7:0/3252632293 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23001f070 con 0x7fd244072250 2023-03-03T19:53:19.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd24aaab700 1 -- 172.21.15.7:0/3252632293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24406c9a0 con 0x7fd244072250 2023-03-03T19:53:19.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd2417fa700 1 -- 172.21.15.7:0/3252632293 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2300038a0 con 0x7fd244072250 2023-03-03T19:53:19.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.745+0000 7fd24aaab700 1 -- 172.21.15.7:0/3252632293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd244151300 con 0x7fd244072250 2023-03-03T19:53:19.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.747+0000 7fd24aaab700 1 -- 172.21.15.7:0/3252632293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24406cb30 con 0x7fd244072250 2023-03-03T19:53:19.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.747+0000 7fd2417fa700 1 -- 172.21.15.7:0/3252632293 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd23002a020 con 0x7fd244072250 2023-03-03T19:53:19.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.747+0000 7fd2417fa700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd228068320 0x7fd22806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:19.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.748+0000 7fd2417fa700 1 -- 172.21.15.7:0/3252632293 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd2300876f0 con 0x7fd244072250 2023-03-03T19:53:19.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.748+0000 7fd248847700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd228068320 0x7fd22806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:19.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.748+0000 7fd248847700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd228068320 0x7fd22806a7d0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fd238009be0 tx=0x7fd238009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:53:19.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.752+0000 7fd2417fa700 1 -- 172.21.15.7:0/3252632293 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd230053a90 con 0x7fd244072250 2023-03-03T19:53:19.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.994+0000 7fd24aaab700 1 -- 172.21.15.7:0/3252632293 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd244074bb0 con 0x7fd244072250 2023-03-03T19:53:19.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.994+0000 7fd2417fa700 1 -- 172.21.15.7:0/3252632293 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd2300570b0 con 0x7fd244072250 2023-03-03T19:53:19.996 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:53:19.996 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:53:19.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.996+0000 7fd24aaab700 1 -- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd228068320 msgr2=0x7fd22806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:19.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.996+0000 7fd24aaab700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd228068320 0x7fd22806a7d0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fd238009be0 tx=0x7fd238009210 comp rx=0 tx=0).stop 2023-03-03T19:53:19.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.996+0000 7fd24aaab700 1 -- 172.21.15.7:0/3252632293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 msgr2=0x7fd244142b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:19.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.996+0000 7fd24aaab700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 0x7fd244142b70 secure :-1 s=READY pgs=3639 cs=0 l=1 rev1=1 crypto rx=0x7fd230009c40 tx=0x7fd230004750 comp rx=0 tx=0).stop 2023-03-03T19:53:19.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.996+0000 7fd24aaab700 1 -- 172.21.15.7:0/3252632293 shutdown_connections 2023-03-03T19:53:19.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.996+0000 7fd24aaab700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd2440c1ce0 0x7fd2441430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:19.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.996+0000 7fd24aaab700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd244071e80 0x7fd2440c2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:19.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.996+0000 7fd24aaab700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd228068320 0x7fd22806a7d0 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:19.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.996+0000 7fd24aaab700 1 --2- 172.21.15.7:0/3252632293 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd244072250 0x7fd244142b70 unknown :-1 s=CLOSED pgs=3639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:19.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.996+0000 7fd24aaab700 1 -- 172.21.15.7:0/3252632293 >> 172.21.15.7:0/3252632293 conn(0x7fd2440a94d0 msgr2=0x7fd24406dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:53:19.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.997+0000 7fd24aaab700 1 -- 172.21.15.7:0/3252632293 shutdown_connections 2023-03-03T19:53:19.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:19.997+0000 7fd24aaab700 1 -- 172.21.15.7:0/3252632293 wait complete. 2023-03-03T19:53:20.000 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:53:20.010 INFO:tasks.mds_thrash.fs.[b]:waiting for 99 secs before thrashing 2023-03-03T19:53:23.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:53:23.606 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:53:39.884 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:53:40.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.022+0000 7fdeeeb6b700 1 Processor -- start 2023-03-03T19:53:40.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.023+0000 7fdeeeb6b700 1 -- start start 2023-03-03T19:53:40.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.023+0000 7fdeeeb6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee8071e80 0x7fdee8151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:40.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.023+0000 7fdeeeb6b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee81510d0 con 0x7fdee8072250 2023-03-03T19:53:40.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.023+0000 7fdeeeb6b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee80c3740 con 0x7fdee814e900 2023-03-03T19:53:40.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.023+0000 7fdeeeb6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8070670 con 0x7fdee8071e80 2023-03-03T19:53:40.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.023+0000 7fdee7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdee8072250 0x7fdee814e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37544/0 (socket says 172.21.15.7:37544) 2023-03-03T19:53:40.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.023+0000 7fdee7fff700 1 -- 172.21.15.7:0/3255340388 learned_addr learned my addr 172.21.15.7:0/3255340388 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:53:40.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdeec907700 1 --2- 172.21.15.7:0/3255340388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee8071e80 0x7fdee8151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:40.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdee77fe700 1 -- 172.21.15.7:0/3255340388 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500853088 0 0) 0x7fdee81510d0 con 0x7fdee8072250 2023-03-03T19:53:40.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdee77fe700 1 -- 172.21.15.7:0/3255340388 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded80035b0 con 0x7fdee8072250 2023-03-03T19:53:40.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdee77fe700 1 -- 172.21.15.7:0/3255340388 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376635006 0 0) 0x7fdee80c3740 con 0x7fdee814e900 2023-03-03T19:53:40.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdee77fe700 1 -- 172.21.15.7:0/3255340388 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee81510d0 con 0x7fdee814e900 2023-03-03T19:53:40.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdeec907700 1 -- 172.21.15.7:0/3255340388 >> v1:172.21.15.143:6789/0 conn(0x7fdee814e900 legacy=0x7fdee815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:40.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdeec907700 1 -- 172.21.15.7:0/3255340388 >> v1:172.21.15.7:6789/0 conn(0x7fdee8072250 legacy=0x7fdee814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdeec907700 1 -- 172.21.15.7:0/3255340388 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee80c39e0 con 0x7fdee8071e80 2023-03-03T19:53:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdeec907700 1 --2- 172.21.15.7:0/3255340388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee8071e80 0x7fdee8151820 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009870 tx=0x7fdedc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f8b3f87dfc35752 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:53:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdee77fe700 1 -- 172.21.15.7:0/3255340388 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc011830 con 0x7fdee8071e80 2023-03-03T19:53:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdee77fe700 1 -- 172.21.15.7:0/3255340388 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdedc011990 con 0x7fdee8071e80 2023-03-03T19:53:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.024+0000 7fdee77fe700 1 -- 172.21.15.7:0/3255340388 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc011d20 con 0x7fdee8071e80 2023-03-03T19:53:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.025+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/3255340388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee8071e80 msgr2=0x7fdee8151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:40.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.025+0000 7fdeeeb6b700 1 --2- 172.21.15.7:0/3255340388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee8071e80 0x7fdee8151820 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009870 tx=0x7fdedc009b80 comp rx=0 tx=0).stop 2023-03-03T19:53:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.025+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/3255340388 shutdown_connections 2023-03-03T19:53:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.025+0000 7fdeeeb6b700 1 --2- 172.21.15.7:0/3255340388 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee8071e80 0x7fdee8151820 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.025+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/3255340388 >> 172.21.15.7:0/3255340388 conn(0x7fdee80a94d0 msgr2=0x7fdee80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:53:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.025+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/3255340388 shutdown_connections 2023-03-03T19:53:40.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.025+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/3255340388 wait complete. 2023-03-03T19:53:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.027+0000 7fdeeeb6b700 1 Processor -- start 2023-03-03T19:53:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.027+0000 7fdeeeb6b700 1 -- start start 2023-03-03T19:53:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeeeb6b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee8071e80 0x7fdee80c6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeeeb6b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdee8072250 0x7fdee8149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeeeb6b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee814e900 0x7fdee8149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:40.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeeeb6b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8150d10 con 0x7fdee8071e80 2023-03-03T19:53:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeeeb6b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8149e90 con 0x7fdee8072250 2023-03-03T19:53:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeeeb6b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee80c58e0 con 0x7fdee814e900 2023-03-03T19:53:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeec907700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee8071e80 0x7fdee80c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeec907700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee8071e80 0x7fdee80c6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43570/0 (socket says 172.21.15.7:43570) 2023-03-03T19:53:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeec907700 1 -- 172.21.15.7:0/1421592551 learned_addr learned my addr 172.21.15.7:0/1421592551 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:53:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdee7fff700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdee8072250 0x7fdee8149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:40.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeed108700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee814e900 0x7fdee8149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeec907700 1 -- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee814e900 msgr2=0x7fdee8149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeec907700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee814e900 0x7fdee8149950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeec907700 1 -- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdee8072250 msgr2=0x7fdee8149410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeec907700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdee8072250 0x7fdee8149410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeec907700 1 -- 172.21.15.7:0/1421592551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc0094a0 con 0x7fdee8071e80 2023-03-03T19:53:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeed108700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee814e900 0x7fdee8149950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:53:40.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.028+0000 7fdeec907700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee8071e80 0x7fdee80c6650 secure :-1 s=READY pgs=3641 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0090b0 tx=0x7fdedc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:53:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.029+0000 7fdee57fa700 1 -- 172.21.15.7:0/1421592551 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc01a760 con 0x7fdee8071e80 2023-03-03T19:53:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.029+0000 7fdee57fa700 1 -- 172.21.15.7:0/1421592551 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdedc01a8c0 con 0x7fdee8071e80 2023-03-03T19:53:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.029+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/1421592551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee806c9a0 con 0x7fdee8071e80 2023-03-03T19:53:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.029+0000 7fdee57fa700 1 -- 172.21.15.7:0/1421592551 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc01ac30 con 0x7fdee8071e80 2023-03-03T19:53:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.029+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/1421592551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee8151300 con 0x7fdee8071e80 2023-03-03T19:53:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.030+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/1421592551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee806cb30 con 0x7fdee8071e80 2023-03-03T19:53:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.030+0000 7fdee57fa700 1 -- 172.21.15.7:0/1421592551 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdedc02b030 con 0x7fdee8071e80 2023-03-03T19:53:40.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.031+0000 7fdee57fa700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdecc0682f0 0x7fdecc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:53:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.031+0000 7fdee57fa700 1 -- 172.21.15.7:0/1421592551 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fdedc0872b0 con 0x7fdee8071e80 2023-03-03T19:53:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.031+0000 7fdee7fff700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdecc0682f0 0x7fdecc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:53:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.031+0000 7fdee7fff700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdecc0682f0 0x7fdecc06a7a0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fdee8150980 tx=0x7fded4007e00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:53:40.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.035+0000 7fdee57fa700 1 -- 172.21.15.7:0/1421592551 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fdedc053700 con 0x7fdee8071e80 2023-03-03T19:53:40.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.275+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/1421592551 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdee80c2120 con 0x7fdee8071e80 2023-03-03T19:53:40.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.275+0000 7fdee57fa700 1 -- 172.21.15.7:0/1421592551 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fdedc015090 con 0x7fdee8071e80 2023-03-03T19:53:40.277 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:53:40.277 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:53:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdecc0682f0 msgr2=0x7fdecc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdecc0682f0 0x7fdecc06a7a0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fdee8150980 tx=0x7fded4007e00 comp rx=0 tx=0).stop 2023-03-03T19:53:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/1421592551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee8071e80 msgr2=0x7fdee80c6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:53:40.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee8071e80 0x7fdee80c6650 secure :-1 s=READY pgs=3641 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0090b0 tx=0x7fdedc004750 comp rx=0 tx=0).stop 2023-03-03T19:53:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/1421592551 shutdown_connections 2023-03-03T19:53:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fdee814e900 0x7fdee8149950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fdecc0682f0 0x7fdecc06a7a0 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdee8071e80 0x7fdee80c6650 unknown :-1 s=CLOSED pgs=3641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 --2- 172.21.15.7:0/1421592551 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdee8072250 0x7fdee8149410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:53:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/1421592551 >> 172.21.15.7:0/1421592551 conn(0x7fdee80a94d0 msgr2=0x7fdee814b190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:53:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/1421592551 shutdown_connections 2023-03-03T19:53:40.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:53:40.277+0000 7fdeeeb6b700 1 -- 172.21.15.7:0/1421592551 wait complete. 2023-03-03T19:53:40.280 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:53:40.291 INFO:tasks.mds_thrash.fs.[a]:waiting for 83 secs before thrashing 2023-03-03T19:53:53.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:53:53.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:54:23.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:54:23.762 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:54:53.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:54:53.829 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:54:59.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:54:59.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.151+0000 7fb8747e0700 1 Processor -- start 2023-03-03T19:54:59.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.152+0000 7fb8747e0700 1 -- start start 2023-03-03T19:54:59.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb8747e0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb86c156490 0x7fb86c15dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:54:59.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb8747e0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c1568d0 con 0x7fb86c0beab0 2023-03-03T19:54:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb8747e0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c1523b0 con 0x7fb86c152e30 2023-03-03T19:54:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb8747e0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c072010 con 0x7fb86c156490 2023-03-03T19:54:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb871d7b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb86c0beab0 0x7fb86c155d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37624/0 (socket says 172.21.15.7:37624) 2023-03-03T19:54:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb871d7b700 1 -- 172.21.15.7:0/819130620 learned_addr learned my addr 172.21.15.7:0/819130620 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:54:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb872d7d700 1 --2- 172.21.15.7:0/819130620 >> v2:172.21.15.143:3300/0 conn(0x7fb86c156490 0x7fb86c15dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:54:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55742580 0 0) 0x7fb86c1568d0 con 0x7fb86c0beab0 2023-03-03T19:54:59.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8580035b0 con 0x7fb86c0beab0 2023-03-03T19:54:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2648384165 0 0) 0x7fb8580035b0 con 0x7fb86c0beab0 2023-03-03T19:54:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 >> v2:172.21.15.143:3300/0 conn(0x7fb86c156490 msgr2=0x7fb86c15dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:54:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 --2- 172.21.15.7:0/819130620 >> v2:172.21.15.143:3300/0 conn(0x7fb86c156490 0x7fb86c15dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 >> v1:172.21.15.143:6789/0 conn(0x7fb86c152e30 legacy=0x7fb86c0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:54:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c0aaaf0 con 0x7fb86c0beab0 2023-03-03T19:54:59.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb868001670 con 0x7fb86c0beab0 2023-03-03T19:54:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 >> v1:172.21.15.7:6789/0 conn(0x7fb86c0beab0 legacy=0x7fb86c155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:54:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb858003a70 0x7fb858003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:54:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb858004850 0x7fb858004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:54:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb858005250 0x7fb85800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:54:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c072010 con 0x7fb858005250 2023-03-03T19:54:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85800c190 con 0x7fb858004850 2023-03-03T19:54:59.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.153+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85800c350 con 0x7fb858003a70 2023-03-03T19:54:59.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb872d7d700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb858005250 0x7fb85800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:54:59.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb87257c700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb858003a70 0x7fb858003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:54:59.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb871d7b700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb858004850 0x7fb858004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:54:59.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb872d7d700 1 -- 172.21.15.7:0/819130620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb858003a70 msgr2=0x7fb858003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:54:59.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb872d7d700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb858003a70 0x7fb858003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb872d7d700 1 -- 172.21.15.7:0/819130620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb858004850 msgr2=0x7fb858004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:54:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb872d7d700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb858004850 0x7fb858004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb872d7d700 1 -- 172.21.15.7:0/819130620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8580035b0 con 0x7fb858005250 2023-03-03T19:54:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb872d7d700 1 -- 172.21.15.7:0/819130620 reap_dead start 2023-03-03T19:54:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb872d7d700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb858005250 0x7fb85800ba10 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 crypto rx=0x7fb864009f90 tx=0x7fb858005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b9168cf60e23c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:54:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb864003a20 con 0x7fb858005250 2023-03-03T19:54:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86400f040 con 0x7fb858005250 2023-03-03T19:54:59.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.154+0000 7fb87157a700 1 -- 172.21.15.7:0/819130620 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb864003dc0 con 0x7fb858005250 2023-03-03T19:54:59.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.155+0000 7fb8747e0700 1 -- 172.21.15.7:0/819130620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb858005250 msgr2=0x7fb85800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:54:59.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.155+0000 7fb8747e0700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb858005250 0x7fb85800ba10 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 crypto rx=0x7fb864009f90 tx=0x7fb858005880 comp rx=0 tx=0).stop 2023-03-03T19:54:59.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.155+0000 7fb8747e0700 1 -- 172.21.15.7:0/819130620 shutdown_connections 2023-03-03T19:54:59.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.155+0000 7fb8747e0700 1 --2- 172.21.15.7:0/819130620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb858005250 0x7fb85800ba10 unknown :-1 s=CLOSED pgs=3643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.155+0000 7fb8747e0700 1 -- 172.21.15.7:0/819130620 >> 172.21.15.7:0/819130620 conn(0x7fb86c0736f0 msgr2=0x7fb86c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:54:59.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.155+0000 7fb8747e0700 1 -- 172.21.15.7:0/819130620 shutdown_connections 2023-03-03T19:54:59.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.155+0000 7fb8747e0700 1 -- 172.21.15.7:0/819130620 wait complete. 2023-03-03T19:54:59.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.156+0000 7fb8747e0700 1 Processor -- start 2023-03-03T19:54:59.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.156+0000 7fb8747e0700 1 -- start start 2023-03-03T19:54:59.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb8747e0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb858005250 0x7fb86c0c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:54:59.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb8747e0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb86c06bea0 0x7fb86c148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:54:59.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb8747e0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86c149290 0x7fb86c0ac600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:54:59.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb8747e0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c06bb20 con 0x7fb86c06bea0 2023-03-03T19:54:59.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb8747e0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c149700 con 0x7fb858005250 2023-03-03T19:54:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb8747e0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c06afa0 con 0x7fb86c149290 2023-03-03T19:54:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb871d7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb86c06bea0 0x7fb86c148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:54:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb871d7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb86c06bea0 0x7fb86c148d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43656/0 (socket says 172.21.15.7:43656) 2023-03-03T19:54:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb871d7b700 1 -- 172.21.15.7:0/3655472947 learned_addr learned my addr 172.21.15.7:0/3655472947 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:54:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb87257c700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb858005250 0x7fb86c0c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:54:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb872d7d700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86c149290 0x7fb86c0ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:54:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb871d7b700 1 -- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86c149290 msgr2=0x7fb86c0ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:54:59.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb871d7b700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86c149290 0x7fb86c0ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb871d7b700 1 -- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb858005250 msgr2=0x7fb86c0c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:54:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb871d7b700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb858005250 0x7fb86c0c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb871d7b700 1 -- 172.21.15.7:0/3655472947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c071c70 con 0x7fb86c06bea0 2023-03-03T19:54:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb872d7d700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86c149290 0x7fb86c0ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:54:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb87257c700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb858005250 0x7fb86c0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:54:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.157+0000 7fb871d7b700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb86c06bea0 0x7fb86c148d50 secure :-1 s=READY pgs=3644 cs=0 l=1 rev1=1 crypto rx=0x7fb86800e730 tx=0x7fb86800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:54:59.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.158+0000 7fb85effd700 1 -- 172.21.15.7:0/3655472947 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868012590 con 0x7fb86c06bea0 2023-03-03T19:54:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.158+0000 7fb85effd700 1 -- 172.21.15.7:0/3655472947 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8680126f0 con 0x7fb86c06bea0 2023-03-03T19:54:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.158+0000 7fb8747e0700 1 -- 172.21.15.7:0/3655472947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c0aadd0 con 0x7fb86c06bea0 2023-03-03T19:54:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.158+0000 7fb85effd700 1 -- 172.21.15.7:0/3655472947 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868012aa0 con 0x7fb86c06bea0 2023-03-03T19:54:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.158+0000 7fb8747e0700 1 -- 172.21.15.7:0/3655472947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c153200 con 0x7fb86c06bea0 2023-03-03T19:54:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.159+0000 7fb8747e0700 1 -- 172.21.15.7:0/3655472947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c0ac450 con 0x7fb86c06bea0 2023-03-03T19:54:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.159+0000 7fb85effd700 1 -- 172.21.15.7:0/3655472947 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb868012c00 con 0x7fb86c06bea0 2023-03-03T19:54:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.160+0000 7fb85effd700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8500682f0 0x7fb85006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:54:59.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.160+0000 7fb85effd700 1 -- 172.21.15.7:0/3655472947 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb868085fd0 con 0x7fb86c06bea0 2023-03-03T19:54:59.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.160+0000 7fb87257c700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8500682f0 0x7fb85006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:54:59.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.160+0000 7fb87257c700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8500682f0 0x7fb85006a7a0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fb8600016d0 tx=0x7fb860009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:54:59.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.164+0000 7fb85effd700 1 -- 172.21.15.7:0/3655472947 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb868052310 con 0x7fb86c06bea0 2023-03-03T19:54:59.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.405+0000 7fb8747e0700 1 -- 172.21.15.7:0/3655472947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb86c0ad310 con 0x7fb86c06bea0 2023-03-03T19:54:59.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.406+0000 7fb85effd700 1 -- 172.21.15.7:0/3655472947 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb86c0ad310 con 0x7fb86c06bea0 2023-03-03T19:54:59.407 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:54:59.407 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:54:59.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 -- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8500682f0 msgr2=0x7fb85006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:54:59.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8500682f0 0x7fb85006a7a0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fb8600016d0 tx=0x7fb860009040 comp rx=0 tx=0).stop 2023-03-03T19:54:59.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 -- 172.21.15.7:0/3655472947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb86c06bea0 msgr2=0x7fb86c148d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:54:59.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb86c06bea0 0x7fb86c148d50 secure :-1 s=READY pgs=3644 cs=0 l=1 rev1=1 crypto rx=0x7fb86800e730 tx=0x7fb86800eaf0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 -- 172.21.15.7:0/3655472947 shutdown_connections 2023-03-03T19:54:59.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb86c149290 0x7fb86c0ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb858005250 0x7fb86c0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb8500682f0 0x7fb85006a7a0 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 --2- 172.21.15.7:0/3655472947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb86c06bea0 0x7fb86c148d50 unknown :-1 s=CLOSED pgs=3644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:54:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 -- 172.21.15.7:0/3655472947 >> 172.21.15.7:0/3655472947 conn(0x7fb86c0736f0 msgr2=0x7fb86c0c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:54:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 -- 172.21.15.7:0/3655472947 shutdown_connections 2023-03-03T19:54:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:54:59.408+0000 7fb8747e0700 1 -- 172.21.15.7:0/3655472947 wait complete. 2023-03-03T19:54:59.410 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:54:59.420 INFO:tasks.mds_thrash.fs.[b]:waiting for 53 secs before thrashing 2023-03-03T19:55:03.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:55:03.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.428+0000 7f2fc75e9700 1 Processor -- start 2023-03-03T19:55:03.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.429+0000 7f2fc75e9700 1 -- start start 2023-03-03T19:55:03.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.429+0000 7f2fc75e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc0072250 0x7f2fc014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:03.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.429+0000 7f2fc75e9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc01510d0 con 0x7f2fc0071e80 2023-03-03T19:55:03.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.429+0000 7f2fc75e9700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc00c3740 con 0x7f2fc00c1ce0 2023-03-03T19:55:03.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.429+0000 7f2fc75e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc0070670 con 0x7f2fc0072250 2023-03-03T19:55:03.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.430+0000 7f2fc5385700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2fc0071e80 0x7f2fc0151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37646/0 (socket says 172.21.15.7:37646) 2023-03-03T19:55:03.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.430+0000 7f2fc5385700 1 -- 172.21.15.7:0/3561740190 learned_addr learned my addr 172.21.15.7:0/3561740190 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:55:03.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.430+0000 7f2fc4b84700 1 --2- 172.21.15.7:0/3561740190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc0072250 0x7f2fc014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:03.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.430+0000 7f2fb3fff700 1 -- 172.21.15.7:0/3561740190 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186265657 0 0) 0x7f2fc01510d0 con 0x7f2fc0071e80 2023-03-03T19:55:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.430+0000 7f2fb3fff700 1 -- 172.21.15.7:0/3561740190 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fac0035b0 con 0x7f2fc0071e80 2023-03-03T19:55:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.430+0000 7f2fc4b84700 1 -- 172.21.15.7:0/3561740190 >> v1:172.21.15.143:6789/0 conn(0x7f2fc00c1ce0 legacy=0x7f2fc015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:03.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.430+0000 7f2fb3fff700 1 -- 172.21.15.7:0/3561740190 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421340476 0 0) 0x7f2fc00c3740 con 0x7f2fc00c1ce0 2023-03-03T19:55:03.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.430+0000 7f2fc4b84700 1 -- 172.21.15.7:0/3561740190 >> v1:172.21.15.7:6789/0 conn(0x7f2fc0071e80 legacy=0x7f2fc0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:03.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.430+0000 7f2fc4b84700 1 -- 172.21.15.7:0/3561740190 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00c39e0 con 0x7f2fc0072250 2023-03-03T19:55:03.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.430+0000 7f2fc4b84700 1 --2- 172.21.15.7:0/3561740190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc0072250 0x7f2fc014e1f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4009870 tx=0x7f2fb4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b559768f338a7b31 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:55:03.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.431+0000 7f2fb3fff700 1 -- 172.21.15.7:0/3561740190 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4011830 con 0x7f2fc0072250 2023-03-03T19:55:03.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.431+0000 7f2fb3fff700 1 -- 172.21.15.7:0/3561740190 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb4011990 con 0x7f2fc0072250 2023-03-03T19:55:03.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.431+0000 7f2fb3fff700 1 -- 172.21.15.7:0/3561740190 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4011d20 con 0x7f2fc0072250 2023-03-03T19:55:03.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.431+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3561740190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc0072250 msgr2=0x7f2fc014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:03.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.431+0000 7f2fc75e9700 1 --2- 172.21.15.7:0/3561740190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc0072250 0x7f2fc014e1f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4009870 tx=0x7f2fb4009b80 comp rx=0 tx=0).stop 2023-03-03T19:55:03.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.431+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3561740190 shutdown_connections 2023-03-03T19:55:03.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.431+0000 7f2fc75e9700 1 --2- 172.21.15.7:0/3561740190 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc0072250 0x7f2fc014e1f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:03.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.431+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3561740190 >> 172.21.15.7:0/3561740190 conn(0x7f2fc00a94d0 msgr2=0x7f2fc00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:55:03.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.431+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3561740190 shutdown_connections 2023-03-03T19:55:03.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.431+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3561740190 wait complete. 2023-03-03T19:55:03.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.432+0000 7f2fc75e9700 1 Processor -- start 2023-03-03T19:55:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.432+0000 7f2fc75e9700 1 -- start start 2023-03-03T19:55:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc75e9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fc0071e80 0x7f2fc014d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc75e9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fc0072250 0x7f2fc01e5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc75e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc00c1ce0 0x7f2fc01e6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:03.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc75e9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00c7050 con 0x7f2fc0072250 2023-03-03T19:55:03.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc75e9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc0153a00 con 0x7f2fc0071e80 2023-03-03T19:55:03.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc75e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc00c6bc0 con 0x7f2fc00c1ce0 2023-03-03T19:55:03.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc4b84700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fc0072250 0x7f2fc01e5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:03.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc4b84700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fc0072250 0x7f2fc01e5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43674/0 (socket says 172.21.15.7:43674) 2023-03-03T19:55:03.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc4b84700 1 -- 172.21.15.7:0/3038015376 learned_addr learned my addr 172.21.15.7:0/3038015376 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:55:03.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc5385700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fc0071e80 0x7f2fc014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:03.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.433+0000 7f2fc5b86700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc00c1ce0 0x7f2fc01e6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:03.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fc4b84700 1 -- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc00c1ce0 msgr2=0x7f2fc01e6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:03.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fc4b84700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc00c1ce0 0x7f2fc01e6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:03.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fc4b84700 1 -- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fc0071e80 msgr2=0x7f2fc014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:03.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fc4b84700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fc0071e80 0x7f2fc014d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:03.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fc4b84700 1 -- 172.21.15.7:0/3038015376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb40094a0 con 0x7f2fc0072250 2023-03-03T19:55:03.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fc4b84700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fc0072250 0x7f2fc01e5bd0 secure :-1 s=READY pgs=3646 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40097c0 tx=0x7f2fb4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:55:03.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fb1ffb700 1 -- 172.21.15.7:0/3038015376 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb401a9f0 con 0x7f2fc0072250 2023-03-03T19:55:03.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fb1ffb700 1 -- 172.21.15.7:0/3038015376 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb401ab50 con 0x7f2fc0072250 2023-03-03T19:55:03.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3038015376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc0151f30 con 0x7f2fc0072250 2023-03-03T19:55:03.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fb1ffb700 1 -- 172.21.15.7:0/3038015376 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb40243f0 con 0x7f2fc0072250 2023-03-03T19:55:03.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.434+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3038015376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc0151300 con 0x7f2fc0072250 2023-03-03T19:55:03.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.435+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3038015376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc006c9a0 con 0x7f2fc0072250 2023-03-03T19:55:03.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.436+0000 7f2fb1ffb700 1 -- 172.21.15.7:0/3038015376 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2fb400f050 con 0x7f2fc0072250 2023-03-03T19:55:03.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.436+0000 7f2fb1ffb700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fa40682f0 0x7f2fa406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:03.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.436+0000 7f2fb1ffb700 1 -- 172.21.15.7:0/3038015376 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2fb40873f0 con 0x7f2fc0072250 2023-03-03T19:55:03.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.436+0000 7f2fc5385700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fa40682f0 0x7f2fa406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:03.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.437+0000 7f2fc5385700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fa40682f0 0x7f2fa406a7a0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc007fd0 tx=0x7f2fbc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:55:03.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.440+0000 7f2fb1ffb700 1 -- 172.21.15.7:0/3038015376 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2fb4053710 con 0x7f2fc0072250 2023-03-03T19:55:03.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.678+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3038015376 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fc0074bb0 con 0x7f2fc0072250 2023-03-03T19:55:03.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.679+0000 7f2fb1ffb700 1 -- 172.21.15.7:0/3038015376 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2fb4015090 con 0x7f2fc0072250 2023-03-03T19:55:03.682 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:55:03.682 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:55:03.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fa40682f0 msgr2=0x7f2fa406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:03.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fa40682f0 0x7f2fa406a7a0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc007fd0 tx=0x7f2fbc009040 comp rx=0 tx=0).stop 2023-03-03T19:55:03.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3038015376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fc0072250 msgr2=0x7f2fc01e5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:03.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fc0072250 0x7f2fc01e5bd0 secure :-1 s=READY pgs=3646 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40097c0 tx=0x7f2fb4004750 comp rx=0 tx=0).stop 2023-03-03T19:55:03.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3038015376 shutdown_connections 2023-03-03T19:55:03.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2fc00c1ce0 0x7f2fc01e6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:03.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2fc0071e80 0x7f2fc014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:03.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2fa40682f0 0x7f2fa406a7a0 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:03.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 --2- 172.21.15.7:0/3038015376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2fc0072250 0x7f2fc01e5bd0 unknown :-1 s=CLOSED pgs=3646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:03.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3038015376 >> 172.21.15.7:0/3038015376 conn(0x7f2fc00a94d0 msgr2=0x7f2fc0147ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:55:03.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3038015376 shutdown_connections 2023-03-03T19:55:03.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:03.681+0000 7f2fc75e9700 1 -- 172.21.15.7:0/3038015376 wait complete. 2023-03-03T19:55:03.685 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:55:03.694 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2023-03-03T19:55:23.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:55:23.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:55:30.696 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:55:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.846+0000 7fe4ce203700 1 Processor -- start 2023-03-03T19:55:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.846+0000 7fe4ce203700 1 -- start start 2023-03-03T19:55:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.846+0000 7fe4ce203700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c814b020 0x7fe4c80717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.846+0000 7fe4ce203700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe4c8071d20 0x7fe4c81583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:30.854 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.846+0000 7fe4ce203700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c81588f0 con 0x7fe4c80aad20 2023-03-03T19:55:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.846+0000 7fe4ce203700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c80728b0 con 0x7fe4c8071d20 2023-03-03T19:55:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.846+0000 7fe4ce203700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8155070 con 0x7fe4c814b020 2023-03-03T19:55:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c6ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe4c80aad20 0x7fe4c8157ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37680/0 (socket says 172.21.15.7:37680) 2023-03-03T19:55:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c6ffd700 1 -- 172.21.15.7:0/1226690183 learned_addr learned my addr 172.21.15.7:0/1226690183 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:55:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c77fe700 1 --2- 172.21.15.7:0/1226690183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c814b020 0x7fe4c80717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c7fff700 1 --2- 172.21.15.7:0/1226690183 >> v2:172.21.15.143:3300/0 conn(0x7fe4c8071d20 0x7fe4c81583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:30.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c67fc700 1 -- 172.21.15.7:0/1226690183 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375933210 0 0) 0x7fe4c81588f0 con 0x7fe4c80aad20 2023-03-03T19:55:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c67fc700 1 -- 172.21.15.7:0/1226690183 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b40035b0 con 0x7fe4c80aad20 2023-03-03T19:55:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c67fc700 1 -- 172.21.15.7:0/1226690183 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 9124187 0 0) 0x7fe4b40035b0 con 0x7fe4c80aad20 2023-03-03T19:55:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c77fe700 1 -- 172.21.15.7:0/1226690183 >> v2:172.21.15.143:3300/0 conn(0x7fe4c8071d20 msgr2=0x7fe4c81583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:55:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c77fe700 1 --2- 172.21.15.7:0/1226690183 >> v2:172.21.15.143:3300/0 conn(0x7fe4c8071d20 0x7fe4c81583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c77fe700 1 -- 172.21.15.7:0/1226690183 >> v1:172.21.15.7:6789/0 conn(0x7fe4c80aad20 legacy=0x7fe4c8157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c77fe700 1 -- 172.21.15.7:0/1226690183 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c806f5b0 con 0x7fe4c814b020 2023-03-03T19:55:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c7fff700 1 --2- 172.21.15.7:0/1226690183 >> v2:172.21.15.143:3300/0 conn(0x7fe4c8071d20 0x7fe4c81583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:55:30.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.847+0000 7fe4c77fe700 1 --2- 172.21.15.7:0/1226690183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c814b020 0x7fe4c80717e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009840 tx=0x7fe4b8009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7221338ffb3163a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:55:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4c67fc700 1 -- 172.21.15.7:0/1226690183 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0001670 con 0x7fe4c814b020 2023-03-03T19:55:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4c67fc700 1 -- 172.21.15.7:0/1226690183 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b8004030 con 0x7fe4c814b020 2023-03-03T19:55:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4c67fc700 1 -- 172.21.15.7:0/1226690183 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b80043a0 con 0x7fe4c814b020 2023-03-03T19:55:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4ce203700 1 -- 172.21.15.7:0/1226690183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c814b020 msgr2=0x7fe4c80717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4ce203700 1 --2- 172.21.15.7:0/1226690183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c814b020 0x7fe4c80717e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009840 tx=0x7fe4b8009b50 comp rx=0 tx=0).stop 2023-03-03T19:55:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4ce203700 1 -- 172.21.15.7:0/1226690183 shutdown_connections 2023-03-03T19:55:30.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4ce203700 1 --2- 172.21.15.7:0/1226690183 >> v2:172.21.15.143:3300/0 conn(0x7fe4c8071d20 0x7fe4c81583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4ce203700 1 --2- 172.21.15.7:0/1226690183 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c814b020 0x7fe4c80717e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4ce203700 1 -- 172.21.15.7:0/1226690183 >> 172.21.15.7:0/1226690183 conn(0x7fe4c80c8900 msgr2=0x7fe4c80c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:55:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4ce203700 1 -- 172.21.15.7:0/1226690183 shutdown_connections 2023-03-03T19:55:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.848+0000 7fe4ce203700 1 -- 172.21.15.7:0/1226690183 wait complete. 2023-03-03T19:55:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.849+0000 7fe4ce203700 1 Processor -- start 2023-03-03T19:55:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4ce203700 1 -- start start 2023-03-03T19:55:30.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4ce203700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c8071d20 0x7fe4c81ddc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4ce203700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c80aad20 0x7fe4c8144010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4ce203700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4c814b020 0x7fe4c81ed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4ce203700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8153fe0 con 0x7fe4c8071d20 2023-03-03T19:55:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4ce203700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c81edce0 con 0x7fe4c814b020 2023-03-03T19:55:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4ce203700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c81edfc0 con 0x7fe4c80aad20 2023-03-03T19:55:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4c77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c8071d20 0x7fe4c81ddc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4c77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c8071d20 0x7fe4c81ddc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43706/0 (socket says 172.21.15.7:43706) 2023-03-03T19:55:30.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4c77fe700 1 -- 172.21.15.7:0/1342433781 learned_addr learned my addr 172.21.15.7:0/1342433781 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:55:30.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.850+0000 7fe4c6ffd700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c80aad20 0x7fe4c8144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:30.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4c7fff700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4c814b020 0x7fe4c81ed4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:30.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4c77fe700 1 -- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c80aad20 msgr2=0x7fe4c8144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:30.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4c77fe700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c80aad20 0x7fe4c8144010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:30.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4c77fe700 1 -- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4c814b020 msgr2=0x7fe4c81ed4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:30.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4c77fe700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4c814b020 0x7fe4c81ed4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:30.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4c77fe700 1 -- 172.21.15.7:0/1342433781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c0009a70 con 0x7fe4c8071d20 2023-03-03T19:55:30.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4c77fe700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c8071d20 0x7fe4c81ddc10 secure :-1 s=READY pgs=3648 cs=0 l=1 rev1=1 crypto rx=0x7fe4b80090b0 tx=0x7fe4b8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:55:30.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4bffff700 1 -- 172.21.15.7:0/1342433781 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8004590 con 0x7fe4c8071d20 2023-03-03T19:55:30.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4ce203700 1 -- 172.21.15.7:0/1342433781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b80094a0 con 0x7fe4c8071d20 2023-03-03T19:55:30.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4ce203700 1 -- 172.21.15.7:0/1342433781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c806fb70 con 0x7fe4c8071d20 2023-03-03T19:55:30.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4bffff700 1 -- 172.21.15.7:0/1342433781 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b8011630 con 0x7fe4c8071d20 2023-03-03T19:55:30.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.851+0000 7fe4bffff700 1 -- 172.21.15.7:0/1342433781 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b80119a0 con 0x7fe4c8071d20 2023-03-03T19:55:30.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.852+0000 7fe4ce203700 1 -- 172.21.15.7:0/1342433781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ac0052e0 con 0x7fe4c8071d20 2023-03-03T19:55:30.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.853+0000 7fe4bffff700 1 -- 172.21.15.7:0/1342433781 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe4b8017020 con 0x7fe4c8071d20 2023-03-03T19:55:30.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.853+0000 7fe4bffff700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4a80682f0 0x7fe4a806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:30.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.853+0000 7fe4bffff700 1 -- 172.21.15.7:0/1342433781 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe4b80876b0 con 0x7fe4c8071d20 2023-03-03T19:55:30.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.853+0000 7fe4c6ffd700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4a80682f0 0x7fe4a806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:30.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.854+0000 7fe4c6ffd700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4a80682f0 0x7fe4a806a7a0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8157780 tx=0x7fe4b0000ed0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:55:30.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:30.858+0000 7fe4bffff700 1 -- 172.21.15.7:0/1342433781 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe4b80539d0 con 0x7fe4c8071d20 2023-03-03T19:55:31.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.096+0000 7fe4ce203700 1 -- 172.21.15.7:0/1342433781 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4ac005b10 con 0x7fe4c8071d20 2023-03-03T19:55:31.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.097+0000 7fe4bffff700 1 -- 172.21.15.7:0/1342433781 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe4b8056ff0 con 0x7fe4c8071d20 2023-03-03T19:55:31.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:55:31.098 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:55:31.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.098+0000 7fe4ce203700 1 -- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4a80682f0 msgr2=0x7fe4a806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:31.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.098+0000 7fe4ce203700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4a80682f0 0x7fe4a806a7a0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8157780 tx=0x7fe4b0000ed0 comp rx=0 tx=0).stop 2023-03-03T19:55:31.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.098+0000 7fe4ce203700 1 -- 172.21.15.7:0/1342433781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c8071d20 msgr2=0x7fe4c81ddc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.098+0000 7fe4ce203700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c8071d20 0x7fe4c81ddc10 secure :-1 s=READY pgs=3648 cs=0 l=1 rev1=1 crypto rx=0x7fe4b80090b0 tx=0x7fe4b8004750 comp rx=0 tx=0).stop 2023-03-03T19:55:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.099+0000 7fe4ce203700 1 -- 172.21.15.7:0/1342433781 shutdown_connections 2023-03-03T19:55:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.099+0000 7fe4ce203700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe4c814b020 0x7fe4c81ed4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.099+0000 7fe4ce203700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe4a80682f0 0x7fe4a806a7a0 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.099+0000 7fe4ce203700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe4c8071d20 0x7fe4c81ddc10 unknown :-1 s=CLOSED pgs=3648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.099+0000 7fe4ce203700 1 --2- 172.21.15.7:0/1342433781 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe4c80aad20 0x7fe4c8144010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:31.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.099+0000 7fe4ce203700 1 -- 172.21.15.7:0/1342433781 >> 172.21.15.7:0/1342433781 conn(0x7fe4c80c8900 msgr2=0x7fe4c8156c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:55:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.099+0000 7fe4ce203700 1 -- 172.21.15.7:0/1342433781 shutdown_connections 2023-03-03T19:55:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:31.099+0000 7fe4ce203700 1 -- 172.21.15.7:0/1342433781 wait complete. 2023-03-03T19:55:31.101 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:55:31.111 INFO:tasks.mds_thrash.fs.[a]:waiting for 44 secs before thrashing 2023-03-03T19:55:52.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:55:52.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.560+0000 7f2a348a8700 1 Processor -- start 2023-03-03T19:55:52.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.560+0000 7f2a348a8700 1 -- start start 2023-03-03T19:55:52.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.561+0000 7f2a348a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014f9d0 0x7f2a30156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:52.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.561+0000 7f2a348a8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a3014fda0 0x7f2a30071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:52.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.561+0000 7f2a348a8700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a300c2f30 con 0x7f2a300abe20 2023-03-03T19:55:52.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.561+0000 7f2a348a8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a300c31d0 con 0x7f2a3014fda0 2023-03-03T19:55:52.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.561+0000 7f2a348a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a301524a0 con 0x7f2a3014f9d0 2023-03-03T19:55:52.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2dd9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a3014fda0 0x7f2a30071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:52.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2dd9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2a3014fda0 0x7f2a30071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43736/0 (socket says 172.21.15.7:43736) 2023-03-03T19:55:52.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2dd9b700 1 -- 172.21.15.7:0/948785707 learned_addr learned my addr 172.21.15.7:0/948785707 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:55:52.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2e59c700 1 --2- 172.21.15.7:0/948785707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014f9d0 0x7f2a30156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:52.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2dd9b700 1 -- 172.21.15.7:0/948785707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014f9d0 msgr2=0x7f2a30156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:55:52.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2dd9b700 1 --2- 172.21.15.7:0/948785707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014f9d0 0x7f2a30156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:52.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2dd9b700 1 -- 172.21.15.7:0/948785707 >> v1:172.21.15.143:6789/0 conn(0x7f2a300abe20 legacy=0x7f2a301529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:55:52.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2dd9b700 1 -- 172.21.15.7:0/948785707 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a300c34b0 con 0x7f2a3014fda0 2023-03-03T19:55:52.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2e59c700 1 --2- 172.21.15.7:0/948785707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014f9d0 0x7f2a30156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:55:52.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2dd9b700 1 --2- 172.21.15.7:0/948785707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014fda0 0x7f2a30071890 secure :-1 s=READY pgs=3649 cs=0 l=1 rev1=1 crypto rx=0x7f2a240098f0 tx=0x7f2a24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff798f010d5b9e0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:55:52.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2d59a700 1 -- 172.21.15.7:0/948785707 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a24004230 con 0x7f2a3014fda0 2023-03-03T19:55:52.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2d59a700 1 -- 172.21.15.7:0/948785707 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a24004390 con 0x7f2a3014fda0 2023-03-03T19:55:52.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.562+0000 7f2a2d59a700 1 -- 172.21.15.7:0/948785707 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a240036a0 con 0x7f2a3014fda0 2023-03-03T19:55:52.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.563+0000 7f2a348a8700 1 -- 172.21.15.7:0/948785707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014fda0 msgr2=0x7f2a30071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:52.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.563+0000 7f2a348a8700 1 --2- 172.21.15.7:0/948785707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014fda0 0x7f2a30071890 secure :-1 s=READY pgs=3649 cs=0 l=1 rev1=1 crypto rx=0x7f2a240098f0 tx=0x7f2a24009c00 comp rx=0 tx=0).stop 2023-03-03T19:55:52.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.563+0000 7f2a348a8700 1 -- 172.21.15.7:0/948785707 shutdown_connections 2023-03-03T19:55:52.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.563+0000 7f2a348a8700 1 --2- 172.21.15.7:0/948785707 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014f9d0 0x7f2a30156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:52.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.563+0000 7f2a348a8700 1 --2- 172.21.15.7:0/948785707 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014fda0 0x7f2a30071890 unknown :-1 s=CLOSED pgs=3649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:52.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.563+0000 7f2a348a8700 1 -- 172.21.15.7:0/948785707 >> 172.21.15.7:0/948785707 conn(0x7f2a300c8900 msgr2=0x7f2a300c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:55:52.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.563+0000 7f2a348a8700 1 -- 172.21.15.7:0/948785707 shutdown_connections 2023-03-03T19:55:52.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.563+0000 7f2a348a8700 1 -- 172.21.15.7:0/948785707 wait complete. 2023-03-03T19:55:52.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.564+0000 7f2a348a8700 1 Processor -- start 2023-03-03T19:55:52.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.564+0000 7f2a348a8700 1 -- start start 2023-03-03T19:55:52.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a348a8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a300abe20 0x7f2a301ea1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:52.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a348a8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014f9d0 0x7f2a301dee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:52.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a348a8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014fda0 0x7f2a30145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:52.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a348a8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a301e9d60 con 0x7f2a3014f9d0 2023-03-03T19:55:52.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a348a8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a301df3c0 con 0x7f2a300abe20 2023-03-03T19:55:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a348a8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a301ed6b0 con 0x7f2a3014fda0 2023-03-03T19:55:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014f9d0 0x7f2a301dee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014f9d0 0x7f2a301dee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43744/0 (socket says 172.21.15.7:43744) 2023-03-03T19:55:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2dd9b700 1 -- 172.21.15.7:0/2214763222 learned_addr learned my addr 172.21.15.7:0/2214763222 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:55:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2e59c700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a300abe20 0x7f2a301ea1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2ed9d700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014fda0 0x7f2a30145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2dd9b700 1 -- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014fda0 msgr2=0x7f2a30145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2dd9b700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014fda0 0x7f2a30145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:52.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2dd9b700 1 -- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a300abe20 msgr2=0x7f2a301ea1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:52.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2dd9b700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a300abe20 0x7f2a301ea1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:52.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2dd9b700 1 -- 172.21.15.7:0/2214763222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a24009550 con 0x7f2a3014f9d0 2023-03-03T19:55:52.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2e59c700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a300abe20 0x7f2a301ea1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:55:52.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.565+0000 7f2a2ed9d700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014fda0 0x7f2a30145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:55:52.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.566+0000 7f2a2dd9b700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014f9d0 0x7f2a301dee80 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009130 tx=0x7f2a24004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:55:52.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.566+0000 7f2a1effd700 1 -- 172.21.15.7:0/2214763222 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2400b860 con 0x7f2a3014f9d0 2023-03-03T19:55:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.566+0000 7f2a1effd700 1 -- 172.21.15.7:0/2214763222 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2400b9c0 con 0x7f2a3014f9d0 2023-03-03T19:55:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.566+0000 7f2a348a8700 1 -- 172.21.15.7:0/2214763222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a300ac350 con 0x7f2a3014f9d0 2023-03-03T19:55:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.566+0000 7f2a1effd700 1 -- 172.21.15.7:0/2214763222 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2400bd70 con 0x7f2a3014f9d0 2023-03-03T19:55:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.566+0000 7f2a348a8700 1 -- 172.21.15.7:0/2214763222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a300ab630 con 0x7f2a3014f9d0 2023-03-03T19:55:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.567+0000 7f2a348a8700 1 -- 172.21.15.7:0/2214763222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a300c3e10 con 0x7f2a3014f9d0 2023-03-03T19:55:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.567+0000 7f2a1effd700 1 -- 172.21.15.7:0/2214763222 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2a24017020 con 0x7f2a3014f9d0 2023-03-03T19:55:52.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.568+0000 7f2a1effd700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a100682f0 0x7f2a1006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:55:52.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.568+0000 7f2a1effd700 1 -- 172.21.15.7:0/2214763222 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2a24087af0 con 0x7f2a3014f9d0 2023-03-03T19:55:52.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.568+0000 7f2a2e59c700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a100682f0 0x7f2a1006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:55:52.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.569+0000 7f2a2e59c700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a100682f0 0x7f2a1006a7a0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f2a200089b0 tx=0x7f2a20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:55:52.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.572+0000 7f2a1effd700 1 -- 172.21.15.7:0/2214763222 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f2a24053e90 con 0x7f2a3014f9d0 2023-03-03T19:55:52.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.810+0000 7f2a348a8700 1 -- 172.21.15.7:0/2214763222 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a30074bb0 con 0x7f2a3014f9d0 2023-03-03T19:55:52.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.811+0000 7f2a1effd700 1 -- 172.21.15.7:0/2214763222 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2a240574b0 con 0x7f2a3014f9d0 2023-03-03T19:55:52.813 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:55:52.813 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:55:52.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 -- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a100682f0 msgr2=0x7f2a1006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:52.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a100682f0 0x7f2a1006a7a0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f2a200089b0 tx=0x7f2a20008040 comp rx=0 tx=0).stop 2023-03-03T19:55:52.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 -- 172.21.15.7:0/2214763222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014f9d0 msgr2=0x7f2a301dee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:55:52.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014f9d0 0x7f2a301dee80 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009130 tx=0x7f2a24004750 comp rx=0 tx=0).stop 2023-03-03T19:55:52.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 -- 172.21.15.7:0/2214763222 shutdown_connections 2023-03-03T19:55:52.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2a3014fda0 0x7f2a30145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:52.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2a300abe20 0x7f2a301ea1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:52.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f2a100682f0 0x7f2a1006a7a0 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:52.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 --2- 172.21.15.7:0/2214763222 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a3014f9d0 0x7f2a301dee80 unknown :-1 s=CLOSED pgs=3650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:55:52.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 -- 172.21.15.7:0/2214763222 >> 172.21.15.7:0/2214763222 conn(0x7f2a300c8900 msgr2=0x7f2a300c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:55:52.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 -- 172.21.15.7:0/2214763222 shutdown_connections 2023-03-03T19:55:52.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:55:52.813+0000 7f2a348a8700 1 -- 172.21.15.7:0/2214763222 wait complete. 2023-03-03T19:55:52.816 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:55:52.826 INFO:tasks.mds_thrash.fs.[b]:waiting for 85 secs before thrashing 2023-03-03T19:55:53.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:55:53.972 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:56:15.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:56:15.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.250+0000 7f53c7a83700 1 Processor -- start 2023-03-03T19:56:15.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.251+0000 7f53c7a83700 1 -- start start 2023-03-03T19:56:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.251+0000 7f53c7a83700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c014fda0 0x7f53c0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:56:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.251+0000 7f53c7a83700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f53c00abe20 0x7f53c01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:56:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.251+0000 7f53c7a83700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c00c2f30 con 0x7f53c014f9d0 2023-03-03T19:56:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.251+0000 7f53c7a83700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c00c31d0 con 0x7f53c00abe20 2023-03-03T19:56:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.251+0000 7f53c7a83700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c01524a0 con 0x7f53c014fda0 2023-03-03T19:56:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.251+0000 7f53c581f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f53c014f9d0 0x7f53c0071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37738/0 (socket says 172.21.15.7:37738) 2023-03-03T19:56:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.251+0000 7f53c581f700 1 -- 172.21.15.7:0/719601174 learned_addr learned my addr 172.21.15.7:0/719601174 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:56:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c501e700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c014fda0 0x7f53c0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:56:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c6020700 1 --2- 172.21.15.7:0/719601174 >> v2:172.21.15.143:3300/0 conn(0x7f53c00abe20 0x7f53c01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:56:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733262369 0 0) 0x7f53c00c2f30 con 0x7f53c014f9d0 2023-03-03T19:56:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53ac0035b0 con 0x7f53c014f9d0 2023-03-03T19:56:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1607640293 0 0) 0x7f53ac0035b0 con 0x7f53c014f9d0 2023-03-03T19:56:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c014fda0 msgr2=0x7f53c0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:56:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c014fda0 0x7f53c0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:56:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 >> v2:172.21.15.143:3300/0 conn(0x7f53c00abe20 msgr2=0x7f53c01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:56:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 --2- 172.21.15.7:0/719601174 >> v2:172.21.15.143:3300/0 conn(0x7f53c00abe20 0x7f53c01529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:56:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c00c34b0 con 0x7f53c014f9d0 2023-03-03T19:56:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f53bc001670 con 0x7f53c014f9d0 2023-03-03T19:56:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c6020700 1 --2- 172.21.15.7:0/719601174 >> v2:172.21.15.143:3300/0 conn(0x7f53c00abe20 0x7f53c01529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:56:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 >> v1:172.21.15.7:6789/0 conn(0x7f53c014f9d0 legacy=0x7f53c0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:56:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c014fda0 0x7f53ac003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:56:15.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 0x7f53ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:56:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53ac004d90 0x7f53ac005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:56:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00c31d0 con 0x7f53ac003ff0 2023-03-03T19:56:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c01524a0 con 0x7f53ac004d90 2023-03-03T19:56:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac00bb70 con 0x7f53c014fda0 2023-03-03T19:56:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c501e700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 0x7f53ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:56:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.252+0000 7f53c581f700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c014fda0 0x7f53ac003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:56:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.253+0000 7f53c6020700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53ac004d90 0x7f53ac005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:56:15.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.253+0000 7f53c501e700 1 -- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c014fda0 msgr2=0x7f53ac003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:56:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.253+0000 7f53c501e700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c014fda0 0x7f53ac003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:56:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.253+0000 7f53c501e700 1 -- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53ac004d90 msgr2=0x7f53ac005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:56:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.253+0000 7f53c501e700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53ac004d90 0x7f53ac005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:56:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.253+0000 7f53c501e700 1 -- 172.21.15.7:0/719601174 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac0035b0 con 0x7f53ac003ff0 2023-03-03T19:56:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.253+0000 7f53c501e700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 0x7f53ac004850 secure :-1 s=READY pgs=3652 cs=0 l=1 rev1=1 crypto rx=0x7f53c01501d0 tx=0x7f53ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2acd9f2397e1a59 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:56:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.253+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b400e6c0 con 0x7f53ac003ff0 2023-03-03T19:56:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.253+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b400e820 con 0x7f53ac003ff0 2023-03-03T19:56:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.253+0000 7f53c481d700 1 -- 172.21.15.7:0/719601174 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b400ebb0 con 0x7f53ac003ff0 2023-03-03T19:56:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.254+0000 7f53c7a83700 1 -- 172.21.15.7:0/719601174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 msgr2=0x7f53ac004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:56:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.254+0000 7f53c7a83700 1 --2- 172.21.15.7:0/719601174 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 0x7f53ac004850 secure :-1 s=READY pgs=3652 cs=0 l=1 rev1=1 crypto rx=0x7f53c01501d0 tx=0x7f53ac005880 comp rx=0 tx=0).stop 2023-03-03T19:56:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.254+0000 7f53c6020700 1 -- 172.21.15.7:0/719601174 reap_dead start 2023-03-03T19:56:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.254+0000 7f53c7a83700 1 -- 172.21.15.7:0/719601174 shutdown_connections 2023-03-03T19:56:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.254+0000 7f53c7a83700 1 -- 172.21.15.7:0/719601174 >> 172.21.15.7:0/719601174 conn(0x7f53c00c8900 msgr2=0x7f53c00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:56:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.254+0000 7f53c7a83700 1 -- 172.21.15.7:0/719601174 shutdown_connections 2023-03-03T19:56:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.254+0000 7f53c7a83700 1 -- 172.21.15.7:0/719601174 wait complete. 2023-03-03T19:56:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.255+0000 7f53c7a83700 1 Processor -- start 2023-03-03T19:56:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.255+0000 7f53c7a83700 1 -- start start 2023-03-03T19:56:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c7a83700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 0x7f53c00c2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:56:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c7a83700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c0154af0 0x7f53c006d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:56:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c7a83700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53c00a94c0 0x7f53c00a9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:56:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c7a83700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c006df10 con 0x7f53ac003ff0 2023-03-03T19:56:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c7a83700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c01eb3e0 con 0x7f53c00a94c0 2023-03-03T19:56:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c7a83700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00be870 con 0x7f53c0154af0 2023-03-03T19:56:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c581f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 0x7f53c00c2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:56:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c581f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 0x7f53c00c2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43770/0 (socket says 172.21.15.7:43770) 2023-03-03T19:56:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c581f700 1 -- 172.21.15.7:0/1536800632 learned_addr learned my addr 172.21.15.7:0/1536800632 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:56:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c6020700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53c00a94c0 0x7f53c00a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:56:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c501e700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c0154af0 0x7f53c006d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:56:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c581f700 1 -- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c0154af0 msgr2=0x7f53c006d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:56:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c581f700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c0154af0 0x7f53c006d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:56:15.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c581f700 1 -- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53c00a94c0 msgr2=0x7f53c00a9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:56:15.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c581f700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53c00a94c0 0x7f53c00a9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:56:15.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c581f700 1 -- 172.21.15.7:0/1536800632 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c00ac350 con 0x7f53ac003ff0 2023-03-03T19:56:15.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c6020700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53c00a94c0 0x7f53c00a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:56:15.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c581f700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 0x7f53c00c2330 secure :-1 s=READY pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7f53bc007310 tx=0x7f53bc007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:56:15.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53b27fc700 1 -- 172.21.15.7:0/1536800632 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc00f660 con 0x7f53ac003ff0 2023-03-03T19:56:15.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53b27fc700 1 -- 172.21.15.7:0/1536800632 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53bc00f820 con 0x7f53ac003ff0 2023-03-03T19:56:15.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53b27fc700 1 -- 172.21.15.7:0/1536800632 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53bc00fbd0 con 0x7f53ac003ff0 2023-03-03T19:56:15.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c7a83700 1 -- 172.21.15.7:0/1536800632 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c00ab630 con 0x7f53ac003ff0 2023-03-03T19:56:15.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.256+0000 7f53c7a83700 1 -- 172.21.15.7:0/1536800632 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c0071fa0 con 0x7f53ac003ff0 2023-03-03T19:56:15.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.257+0000 7f53c7a83700 1 -- 172.21.15.7:0/1536800632 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c0074bb0 con 0x7f53ac003ff0 2023-03-03T19:56:15.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.258+0000 7f53b27fc700 1 -- 172.21.15.7:0/1536800632 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f53bc00fd30 con 0x7f53ac003ff0 2023-03-03T19:56:15.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.258+0000 7f53b27fc700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53a40682f0 0x7f53a406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:56:15.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.258+0000 7f53b27fc700 1 -- 172.21.15.7:0/1536800632 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f53bc086230 con 0x7f53ac003ff0 2023-03-03T19:56:15.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.259+0000 7f53c501e700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53a40682f0 0x7f53a406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:56:15.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.259+0000 7f53c501e700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53a40682f0 0x7f53a406a7a0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f53b40090b0 tx=0x7f53b4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:56:15.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.263+0000 7f53b27fc700 1 -- 172.21.15.7:0/1536800632 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f53bc0525d0 con 0x7f53ac003ff0 2023-03-03T19:56:15.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.504+0000 7f53c7a83700 1 -- 172.21.15.7:0/1536800632 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53c01504c0 con 0x7f53ac003ff0 2023-03-03T19:56:15.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.504+0000 7f53b27fc700 1 -- 172.21.15.7:0/1536800632 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f53bc055bf0 con 0x7f53ac003ff0 2023-03-03T19:56:15.506 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:56:15.507 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:56:15.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.506+0000 7f53c7a83700 1 -- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53a40682f0 msgr2=0x7f53a406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:56:15.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.506+0000 7f53c7a83700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53a40682f0 0x7f53a406a7a0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f53b40090b0 tx=0x7f53b4008030 comp rx=0 tx=0).stop 2023-03-03T19:56:15.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.506+0000 7f53c7a83700 1 -- 172.21.15.7:0/1536800632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 msgr2=0x7f53c00c2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:56:15.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.506+0000 7f53c7a83700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 0x7f53c00c2330 secure :-1 s=READY pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7f53bc007310 tx=0x7f53bc007620 comp rx=0 tx=0).stop 2023-03-03T19:56:15.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.507+0000 7f53c7a83700 1 -- 172.21.15.7:0/1536800632 shutdown_connections 2023-03-03T19:56:15.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.507+0000 7f53c7a83700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53c00a94c0 0x7f53c00a9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:56:15.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.507+0000 7f53c7a83700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f53a40682f0 0x7f53a406a7a0 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:56:15.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.507+0000 7f53c7a83700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53ac003ff0 0x7f53c00c2330 unknown :-1 s=CLOSED pgs=3653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:56:15.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.507+0000 7f53c7a83700 1 --2- 172.21.15.7:0/1536800632 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f53c0154af0 0x7f53c006d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:56:15.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.507+0000 7f53c7a83700 1 -- 172.21.15.7:0/1536800632 >> 172.21.15.7:0/1536800632 conn(0x7f53c00c8900 msgr2=0x7f53c0151c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:56:15.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.507+0000 7f53c7a83700 1 -- 172.21.15.7:0/1536800632 shutdown_connections 2023-03-03T19:56:15.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:56:15.507+0000 7f53c7a83700 1 -- 172.21.15.7:0/1536800632 wait complete. 2023-03-03T19:56:15.510 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:56:15.520 INFO:tasks.mds_thrash.fs.[a]:waiting for 48 secs before thrashing 2023-03-03T19:56:24.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:56:24.049 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:56:54.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:56:54.151 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:57:03.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:57:03.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.659+0000 7f83bfab4700 1 Processor -- start 2023-03-03T19:57:03.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.660+0000 7f83bfab4700 1 -- start start 2023-03-03T19:57:03.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.660+0000 7f83bfab4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f83b806c480 0x7f83b81581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.660+0000 7f83bfab4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b806d430 con 0x7f83b80c2930 2023-03-03T19:57:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.660+0000 7f83bfab4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b81586e0 con 0x7f83b8151d40 2023-03-03T19:57:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.660+0000 7f83bfab4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b806c2a0 con 0x7f83b806c480 2023-03-03T19:57:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bd04f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f83b80c2930 0x7f83b8157a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37792/0 (socket says 172.21.15.7:37792) 2023-03-03T19:57:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bd04f700 1 -- 172.21.15.7:0/3640339775 learned_addr learned my addr 172.21.15.7:0/3640339775 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:57:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83be051700 1 --2- 172.21.15.7:0/3640339775 >> v2:172.21.15.143:3300/0 conn(0x7f83b806c480 0x7f83b81581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258340350 0 0) 0x7f83b806d430 con 0x7f83b80c2930 2023-03-03T19:57:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a40035b0 con 0x7f83b80c2930 2023-03-03T19:57:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3519721261 0 0) 0x7f83a40035b0 con 0x7f83b80c2930 2023-03-03T19:57:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 >> v2:172.21.15.143:3300/0 conn(0x7f83b806c480 msgr2=0x7f83b81581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:57:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 --2- 172.21.15.7:0/3640339775 >> v2:172.21.15.143:3300/0 conn(0x7f83b806c480 0x7f83b81581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 >> v1:172.21.15.143:6789/0 conn(0x7f83b8151d40 legacy=0x7f83b80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b806cb00 con 0x7f83b80c2930 2023-03-03T19:57:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f83b4001670 con 0x7f83b80c2930 2023-03-03T19:57:03.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 >> v1:172.21.15.7:6789/0 conn(0x7f83b80c2930 legacy=0x7f83b8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:03.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83a4003aa0 0x7f83a4003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:03.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83a4004850 0x7f83a4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:03.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83a4005320 0x7f83a400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:03.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b806c2a0 con 0x7f83a4005320 2023-03-03T19:57:03.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a400c250 con 0x7f83a4004850 2023-03-03T19:57:03.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a400c410 con 0x7f83a4003aa0 2023-03-03T19:57:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83be051700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83a4005320 0x7f83a400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bd850700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83a4003aa0 0x7f83a4003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.661+0000 7f83bd04f700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83a4004850 0x7f83a4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.662+0000 7f83be051700 1 -- 172.21.15.7:0/3640339775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83a4003aa0 msgr2=0x7f83a4003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:57:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.662+0000 7f83be051700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83a4003aa0 0x7f83a4003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.662+0000 7f83be051700 1 -- 172.21.15.7:0/3640339775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83a4004850 msgr2=0x7f83a4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:57:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.662+0000 7f83be051700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83a4004850 0x7f83a4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.662+0000 7f83be051700 1 -- 172.21.15.7:0/3640339775 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a40035b0 con 0x7f83a4005320 2023-03-03T19:57:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.662+0000 7f83be051700 1 -- 172.21.15.7:0/3640339775 reap_dead start 2023-03-03T19:57:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.662+0000 7f83be051700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83a4005320 0x7f83a400ba10 secure :-1 s=READY pgs=3655 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009fc0 tx=0x7f83a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb2d8f190b8541c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:57:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.662+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0003a20 con 0x7f83a4005320 2023-03-03T19:57:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.662+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b000f040 con 0x7f83a4005320 2023-03-03T19:57:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.662+0000 7f83bc84e700 1 -- 172.21.15.7:0/3640339775 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b0003de0 con 0x7f83a4005320 2023-03-03T19:57:03.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.663+0000 7f83bfab4700 1 -- 172.21.15.7:0/3640339775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83a4005320 msgr2=0x7f83a400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:03.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.663+0000 7f83bfab4700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83a4005320 0x7f83a400ba10 secure :-1 s=READY pgs=3655 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009fc0 tx=0x7f83a4005880 comp rx=0 tx=0).stop 2023-03-03T19:57:03.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.663+0000 7f83bfab4700 1 -- 172.21.15.7:0/3640339775 shutdown_connections 2023-03-03T19:57:03.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.663+0000 7f83bfab4700 1 --2- 172.21.15.7:0/3640339775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83a4005320 0x7f83a400ba10 unknown :-1 s=CLOSED pgs=3655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.663+0000 7f83bfab4700 1 -- 172.21.15.7:0/3640339775 >> 172.21.15.7:0/3640339775 conn(0x7f83b80a94d0 msgr2=0x7f83b80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:57:03.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.663+0000 7f83bfab4700 1 -- 172.21.15.7:0/3640339775 shutdown_connections 2023-03-03T19:57:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.663+0000 7f83bfab4700 1 -- 172.21.15.7:0/3640339775 wait complete. 2023-03-03T19:57:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.664+0000 7f83bfab4700 1 Processor -- start 2023-03-03T19:57:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.664+0000 7f83bfab4700 1 -- start start 2023-03-03T19:57:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bfab4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83a4005320 0x7f83b8152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bfab4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83b8152970 0x7f83b8144540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bfab4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83b8144a80 0x7f83b80ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bfab4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b8153410 con 0x7f83b8152970 2023-03-03T19:57:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bfab4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b8153100 con 0x7f83a4005320 2023-03-03T19:57:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bfab4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b81eded0 con 0x7f83b8144a80 2023-03-03T19:57:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd04f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83b8152970 0x7f83b8144540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd04f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83b8152970 0x7f83b8144540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43824/0 (socket says 172.21.15.7:43824) 2023-03-03T19:57:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd04f700 1 -- 172.21.15.7:0/690316777 learned_addr learned my addr 172.21.15.7:0/690316777 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:57:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd850700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83a4005320 0x7f83b8152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83be051700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83b8144a80 0x7f83b80ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd04f700 1 -- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83b8144a80 msgr2=0x7f83b80ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd04f700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83b8144a80 0x7f83b80ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd04f700 1 -- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83a4005320 msgr2=0x7f83b8152430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd04f700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83a4005320 0x7f83b8152430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd04f700 1 -- 172.21.15.7:0/690316777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b806d0c0 con 0x7f83b8152970 2023-03-03T19:57:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83be051700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83b8144a80 0x7f83b80ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:57:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd850700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83a4005320 0x7f83b8152430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:57:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bd04f700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83b8152970 0x7f83b8144540 secure :-1 s=READY pgs=3656 cs=0 l=1 rev1=1 crypto rx=0x7f83b40072f0 tx=0x7f83b40076b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:57:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83ae7fc700 1 -- 172.21.15.7:0/690316777 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4001050 con 0x7f83b8152970 2023-03-03T19:57:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83ae7fc700 1 -- 172.21.15.7:0/690316777 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b40015e0 con 0x7f83b8152970 2023-03-03T19:57:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83ae7fc700 1 -- 172.21.15.7:0/690316777 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b400fab0 con 0x7f83b8152970 2023-03-03T19:57:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bfab4700 1 -- 172.21.15.7:0/690316777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b806cde0 con 0x7f83b8152970 2023-03-03T19:57:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.665+0000 7f83bfab4700 1 -- 172.21.15.7:0/690316777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b806bea0 con 0x7f83b8152970 2023-03-03T19:57:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.666+0000 7f83bfab4700 1 -- 172.21.15.7:0/690316777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b8074bb0 con 0x7f83b8152970 2023-03-03T19:57:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.667+0000 7f83ae7fc700 1 -- 172.21.15.7:0/690316777 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f83b400fc10 con 0x7f83b8152970 2023-03-03T19:57:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.667+0000 7f83ae7fc700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f839c0682f0 0x7f839c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.667+0000 7f83ae7fc700 1 -- 172.21.15.7:0/690316777 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f83b4086190 con 0x7f83b8152970 2023-03-03T19:57:03.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.667+0000 7f83bd850700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f839c0682f0 0x7f839c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:03.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.668+0000 7f83bd850700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f839c0682f0 0x7f839c06a7a0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f83a8000c00 tx=0x7f83a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:57:03.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.672+0000 7f83ae7fc700 1 -- 172.21.15.7:0/690316777 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f83b4052530 con 0x7f83b8152970 2023-03-03T19:57:03.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.908+0000 7f83bfab4700 1 -- 172.21.15.7:0/690316777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83b80a9fb0 con 0x7f83b8152970 2023-03-03T19:57:03.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.908+0000 7f83ae7fc700 1 -- 172.21.15.7:0/690316777 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f83b400fea0 con 0x7f83b8152970 2023-03-03T19:57:03.909 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:57:03.910 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:57:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 -- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f839c0682f0 msgr2=0x7f839c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f839c0682f0 0x7f839c06a7a0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f83a8000c00 tx=0x7f83a8009040 comp rx=0 tx=0).stop 2023-03-03T19:57:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 -- 172.21.15.7:0/690316777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83b8152970 msgr2=0x7f83b8144540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83b8152970 0x7f83b8144540 secure :-1 s=READY pgs=3656 cs=0 l=1 rev1=1 crypto rx=0x7f83b40072f0 tx=0x7f83b40076b0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 -- 172.21.15.7:0/690316777 shutdown_connections 2023-03-03T19:57:03.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83b8144a80 0x7f83b80ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83a4005320 0x7f83b8152430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f839c0682f0 0x7f839c06a7a0 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 --2- 172.21.15.7:0/690316777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83b8152970 0x7f83b8144540 unknown :-1 s=CLOSED pgs=3656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:03.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 -- 172.21.15.7:0/690316777 >> 172.21.15.7:0/690316777 conn(0x7f83b80a94d0 msgr2=0x7f83b806a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:57:03.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 -- 172.21.15.7:0/690316777 shutdown_connections 2023-03-03T19:57:03.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:03.910+0000 7f83bfab4700 1 -- 172.21.15.7:0/690316777 wait complete. 2023-03-03T19:57:03.913 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:57:03.923 INFO:tasks.mds_thrash.fs.[a]:waiting for 62 secs before thrashing 2023-03-03T19:57:17.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:57:17.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.963+0000 7fbdf4ad0700 1 Processor -- start 2023-03-03T19:57:17.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.963+0000 7fbdf4ad0700 1 -- start start 2023-03-03T19:57:17.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.964+0000 7fbdf4ad0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf014b340 0x7fbdf0152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:17.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.964+0000 7fbdf4ad0700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdf00c2790 0x7fbdf0152b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:17.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.964+0000 7fbdf4ad0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf006c920 con 0x7fbdf0151140 2023-03-03T19:57:17.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.964+0000 7fbdf4ad0700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf00abd90 con 0x7fbdf00c2790 2023-03-03T19:57:17.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.964+0000 7fbdf4ad0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00c1300 con 0x7fbdf014b340 2023-03-03T19:57:17.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdedd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdf00c2790 0x7fbdf0152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:17.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdee59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf014b340 0x7fbdf0152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:17.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdee59c700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf014b340 0x7fbdf0152630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:51914/0 (socket says 172.21.15.7:51914) 2023-03-03T19:57:17.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdedd9b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdf00c2790 0x7fbdf0152b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:39616/0 (socket says 172.21.15.7:39616) 2023-03-03T19:57:17.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdee59c700 1 -- 172.21.15.7:0/3712742293 learned_addr learned my addr 172.21.15.7:0/3712742293 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:57:17.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdeed9d700 1 --1- 172.21.15.7:0/3712742293 >> v1:172.21.15.143:6789/0 conn(0x7fbdf0151140 0x7fbdf0151590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:41346/0 (socket says 172.21.15.7:41346) 2023-03-03T19:57:17.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdee59c700 1 -- 172.21.15.7:0/3712742293 >> v2:172.21.15.143:3300/0 conn(0x7fbdf00c2790 msgr2=0x7fbdf0152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:57:17.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdee59c700 1 --2- 172.21.15.7:0/3712742293 >> v2:172.21.15.143:3300/0 conn(0x7fbdf00c2790 0x7fbdf0152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:17.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdee59c700 1 -- 172.21.15.7:0/3712742293 >> v1:172.21.15.143:6789/0 conn(0x7fbdf0151140 legacy=0x7fbdf0151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:17.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdee59c700 1 -- 172.21.15.7:0/3712742293 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf006bd50 con 0x7fbdf014b340 2023-03-03T19:57:17.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdedd9b700 1 --2- 172.21.15.7:0/3712742293 >> v2:172.21.15.143:3300/0 conn(0x7fbdf00c2790 0x7fbdf0152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T19:57:17.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.965+0000 7fbdee59c700 1 --2- 172.21.15.7:0/3712742293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf014b340 0x7fbdf0152630 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbde40098f0 tx=0x7fbde4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=157d428fcf454552 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:57:17.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbded59a700 1 -- 172.21.15.7:0/3712742293 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4011830 con 0x7fbdf014b340 2023-03-03T19:57:17.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbded59a700 1 -- 172.21.15.7:0/3712742293 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde4011990 con 0x7fbdf014b340 2023-03-03T19:57:17.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbded59a700 1 -- 172.21.15.7:0/3712742293 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4011d20 con 0x7fbdf014b340 2023-03-03T19:57:17.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/3712742293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf014b340 msgr2=0x7fbdf0152630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:17.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbdf4ad0700 1 --2- 172.21.15.7:0/3712742293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf014b340 0x7fbdf0152630 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbde40098f0 tx=0x7fbde4009c00 comp rx=0 tx=0).stop 2023-03-03T19:57:17.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/3712742293 shutdown_connections 2023-03-03T19:57:17.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbdf4ad0700 1 --2- 172.21.15.7:0/3712742293 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf014b340 0x7fbdf0152630 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbdf4ad0700 1 --2- 172.21.15.7:0/3712742293 >> v2:172.21.15.143:3300/0 conn(0x7fbdf00c2790 0x7fbdf0152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/3712742293 >> 172.21.15.7:0/3712742293 conn(0x7fbdf00c62e0 msgr2=0x7fbdf00c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:57:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/3712742293 shutdown_connections 2023-03-03T19:57:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.966+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/3712742293 wait complete. 2023-03-03T19:57:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.968+0000 7fbdf4ad0700 1 Processor -- start 2023-03-03T19:57:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.968+0000 7fbdf4ad0700 1 -- start start 2023-03-03T19:57:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdf4ad0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf00c2790 0x7fbdf0144170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:17.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdf4ad0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf014b340 0x7fbdf01e8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:17.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdf4ad0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf0151140 0x7fbdf01e9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:17.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdf4ad0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00aa760 con 0x7fbdf0151140 2023-03-03T19:57:17.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdf4ad0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf01529a0 con 0x7fbdf014b340 2023-03-03T19:57:17.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdf4ad0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00c51c0 con 0x7fbdf00c2790 2023-03-03T19:57:17.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdeed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf0151140 0x7fbdf01e9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:17.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdeed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf0151140 0x7fbdf01e9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43852/0 (socket says 172.21.15.7:43852) 2023-03-03T19:57:17.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdeed9d700 1 -- 172.21.15.7:0/1040560809 learned_addr learned my addr 172.21.15.7:0/1040560809 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:57:17.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdedd9b700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf014b340 0x7fbdf01e8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:17.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdee59c700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf00c2790 0x7fbdf0144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:17.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdeed9d700 1 -- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf00c2790 msgr2=0x7fbdf0144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:17.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdeed9d700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf00c2790 0x7fbdf0144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:17.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdeed9d700 1 -- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf014b340 msgr2=0x7fbdf01e8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:17.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdeed9d700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf014b340 0x7fbdf01e8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:17.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.969+0000 7fbdeed9d700 1 -- 172.21.15.7:0/1040560809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde000b740 con 0x7fbdf0151140 2023-03-03T19:57:17.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.970+0000 7fbdedd9b700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf014b340 0x7fbdf01e8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:57:17.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.970+0000 7fbdeed9d700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf0151140 0x7fbdf01e9350 secure :-1 s=READY pgs=3657 cs=0 l=1 rev1=1 crypto rx=0x7fbde800cbc0 tx=0x7fbde800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:57:17.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.970+0000 7fbddeffd700 1 -- 172.21.15.7:0/1040560809 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde800ab50 con 0x7fbdf0151140 2023-03-03T19:57:17.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.970+0000 7fbddeffd700 1 -- 172.21.15.7:0/1040560809 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde800acb0 con 0x7fbdf0151140 2023-03-03T19:57:17.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.970+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/1040560809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde4009550 con 0x7fbdf0151140 2023-03-03T19:57:17.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.970+0000 7fbddeffd700 1 -- 172.21.15.7:0/1040560809 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8007270 con 0x7fbdf0151140 2023-03-03T19:57:17.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.970+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/1040560809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf006c2d0 con 0x7fbdf0151140 2023-03-03T19:57:17.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.971+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/1040560809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf006bff0 con 0x7fbdf0151140 2023-03-03T19:57:17.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.971+0000 7fbddeffd700 1 -- 172.21.15.7:0/1040560809 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbde8019030 con 0x7fbdf0151140 2023-03-03T19:57:17.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.972+0000 7fbddeffd700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdd00682f0 0x7fbdd006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:57:17.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.972+0000 7fbddeffd700 1 -- 172.21.15.7:0/1040560809 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbde8085d20 con 0x7fbdf0151140 2023-03-03T19:57:17.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.972+0000 7fbdee59c700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdd00682f0 0x7fbdd006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:57:17.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.973+0000 7fbdee59c700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdd00682f0 0x7fbdd006a7a0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fbde4014040 tx=0x7fbde4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:57:17.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:17.977+0000 7fbddeffd700 1 -- 172.21.15.7:0/1040560809 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbde804b5e0 con 0x7fbdf0151140 2023-03-03T19:57:18.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.214+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/1040560809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdf006c5b0 con 0x7fbdf0151140 2023-03-03T19:57:18.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.215+0000 7fbddeffd700 1 -- 172.21.15.7:0/1040560809 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbde8017020 con 0x7fbdf0151140 2023-03-03T19:57:18.217 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:57:18.217 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:57:18.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdd00682f0 msgr2=0x7fbdd006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:18.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdd00682f0 0x7fbdd006a7a0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fbde4014040 tx=0x7fbde4008030 comp rx=0 tx=0).stop 2023-03-03T19:57:18.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/1040560809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf0151140 msgr2=0x7fbdf01e9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:57:18.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf0151140 0x7fbdf01e9350 secure :-1 s=READY pgs=3657 cs=0 l=1 rev1=1 crypto rx=0x7fbde800cbc0 tx=0x7fbde800cf80 comp rx=0 tx=0).stop 2023-03-03T19:57:18.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/1040560809 shutdown_connections 2023-03-03T19:57:18.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdd00682f0 0x7fbdd006a7a0 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:18.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdf0151140 0x7fbdf01e9350 unknown :-1 s=CLOSED pgs=3657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:18.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdf00c2790 0x7fbdf0144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:18.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 --2- 172.21.15.7:0/1040560809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf014b340 0x7fbdf01e8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:57:18.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/1040560809 >> 172.21.15.7:0/1040560809 conn(0x7fbdf00c62e0 msgr2=0x7fbdf0152bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:57:18.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/1040560809 shutdown_connections 2023-03-03T19:57:18.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:57:18.217+0000 7fbdf4ad0700 1 -- 172.21.15.7:0/1040560809 wait complete. 2023-03-03T19:57:18.221 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:57:18.231 INFO:tasks.mds_thrash.fs.[b]:waiting for 57 secs before thrashing 2023-03-03T19:57:24.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:57:24.224 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:57:54.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:57:54.288 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:58:05.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:58:06.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.062+0000 7f96948bd700 1 Processor -- start 2023-03-03T19:58:06.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.063+0000 7f96948bd700 1 -- start start 2023-03-03T19:58:06.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.063+0000 7f96948bd700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f9690151cd0 0x7f9690154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:06.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.063+0000 7f96948bd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9690157ff0 con 0x7f96900c2e30 2023-03-03T19:58:06.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.063+0000 7f96948bd700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969006d640 con 0x7f969006c6c0 2023-03-03T19:58:06.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.063+0000 7f96948bd700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969006cda0 con 0x7f9690151cd0 2023-03-03T19:58:06.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968dd9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f96900c2e30 0x7f9690158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37878/0 (socket says 172.21.15.7:37878) 2023-03-03T19:58:06.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968dd9b700 1 -- 172.21.15.7:0/1076661591 learned_addr learned my addr 172.21.15.7:0/1076661591 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:58:06.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968e59c700 1 --2- 172.21.15.7:0/1076661591 >> v2:172.21.15.143:3300/0 conn(0x7f9690151cd0 0x7f9690154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:06.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055829290 0 0) 0x7f9690157ff0 con 0x7f96900c2e30 2023-03-03T19:58:06.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f967c0035b0 con 0x7f96900c2e30 2023-03-03T19:58:06.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935937239 0 0) 0x7f969006d640 con 0x7f969006c6c0 2023-03-03T19:58:06.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9690157ff0 con 0x7f969006c6c0 2023-03-03T19:58:06.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552085752 0 0) 0x7f967c0035b0 con 0x7f96900c2e30 2023-03-03T19:58:06.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 >> v2:172.21.15.143:3300/0 conn(0x7f9690151cd0 msgr2=0x7f9690154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:06.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 --2- 172.21.15.7:0/1076661591 >> v2:172.21.15.143:3300/0 conn(0x7f9690151cd0 0x7f9690154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:06.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 >> v1:172.21.15.143:6789/0 conn(0x7f969006c6c0 legacy=0x7f96900736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:06.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969006d080 con 0x7f96900c2e30 2023-03-03T19:58:06.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f9678001670 con 0x7f96900c2e30 2023-03-03T19:58:06.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 >> v1:172.21.15.7:6789/0 conn(0x7f96900c2e30 legacy=0x7f9690158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:06.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c003c00 0x7f967c003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:06.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c004850 0x7f967c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:06.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c005310 0x7f967c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:06.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969006cda0 con 0x7f967c004850 2023-03-03T19:58:06.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c00c210 con 0x7f967c005310 2023-03-03T19:58:06.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c00c3d0 con 0x7f967c003c00 2023-03-03T19:58:06.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.064+0000 7f968dd9b700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c004850 0x7f967c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:06.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968e59c700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c003c00 0x7f967c003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:06.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968ed9d700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c005310 0x7f967c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:06.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968dd9b700 1 -- 172.21.15.7:0/1076661591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c003c00 msgr2=0x7f967c003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:06.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968dd9b700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c003c00 0x7f967c003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:06.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968dd9b700 1 -- 172.21.15.7:0/1076661591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c005310 msgr2=0x7f967c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:06.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968dd9b700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c005310 0x7f967c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:06.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968dd9b700 1 -- 172.21.15.7:0/1076661591 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c0035b0 con 0x7f967c004850 2023-03-03T19:58:06.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968ed9d700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f967c005310 0x7f967c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:58:06.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968e59c700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c003c00 0x7f967c003ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T19:58:06.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968ed9d700 1 -- 172.21.15.7:0/1076661591 reap_dead start 2023-03-03T19:58:06.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968dd9b700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c004850 0x7f967c004cc0 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 crypto rx=0x7f967800c210 tx=0x7f967c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e86cb588d071cc2 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:06.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967800cdb0 con 0x7f967c004850 2023-03-03T19:58:06.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9678001af0 con 0x7f967c004850 2023-03-03T19:58:06.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.065+0000 7f968d59a700 1 -- 172.21.15.7:0/1076661591 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678005ba0 con 0x7f967c004850 2023-03-03T19:58:06.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.066+0000 7f96948bd700 1 -- 172.21.15.7:0/1076661591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c004850 msgr2=0x7f967c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:06.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.066+0000 7f96948bd700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c004850 0x7f967c004cc0 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 crypto rx=0x7f967800c210 tx=0x7f967c005880 comp rx=0 tx=0).stop 2023-03-03T19:58:06.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.066+0000 7f96948bd700 1 -- 172.21.15.7:0/1076661591 shutdown_connections 2023-03-03T19:58:06.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.066+0000 7f96948bd700 1 --2- 172.21.15.7:0/1076661591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f967c004850 0x7f967c004cc0 unknown :-1 s=CLOSED pgs=3659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:06.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.066+0000 7f96948bd700 1 -- 172.21.15.7:0/1076661591 >> 172.21.15.7:0/1076661591 conn(0x7f96900a94d0 msgr2=0x7f96900a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:58:06.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.066+0000 7f96948bd700 1 -- 172.21.15.7:0/1076661591 shutdown_connections 2023-03-03T19:58:06.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.066+0000 7f96948bd700 1 -- 172.21.15.7:0/1076661591 wait complete. 2023-03-03T19:58:06.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.067+0000 7f96948bd700 1 Processor -- start 2023-03-03T19:58:06.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.067+0000 7f96948bd700 1 -- start start 2023-03-03T19:58:06.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f96948bd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c004850 0x7f96900c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:06.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f96948bd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96900c5460 0x7f9690155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:06.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f96948bd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9690155e40 0x7f9690147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f96948bd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96901df980 con 0x7f9690155e40 2023-03-03T19:58:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f96948bd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96901562b0 con 0x7f96900c5460 2023-03-03T19:58:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f96948bd700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96900c69f0 con 0x7f967c004850 2023-03-03T19:58:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9690155e40 0x7f9690147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9690155e40 0x7f9690147e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43914/0 (socket says 172.21.15.7:43914) 2023-03-03T19:58:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968ed9d700 1 -- 172.21.15.7:0/1828284789 learned_addr learned my addr 172.21.15.7:0/1828284789 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:58:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968dd9b700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96900c5460 0x7f9690155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968e59c700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c004850 0x7f96900c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968ed9d700 1 -- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c004850 msgr2=0x7f96900c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968ed9d700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c004850 0x7f96900c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968ed9d700 1 -- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96900c5460 msgr2=0x7f9690155900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:06.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968ed9d700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96900c5460 0x7f9690155900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968ed9d700 1 -- 172.21.15.7:0/1828284789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9690157c80 con 0x7f9690155e40 2023-03-03T19:58:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968dd9b700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96900c5460 0x7f9690155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:58:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968e59c700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c004850 0x7f96900c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:58:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.068+0000 7f968ed9d700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9690155e40 0x7f9690147e10 secure :-1 s=READY pgs=3660 cs=0 l=1 rev1=1 crypto rx=0x7f969006e230 tx=0x7f968800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.069+0000 7f9686ffd700 1 -- 172.21.15.7:0/1828284789 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9688003c10 con 0x7f9690155e40 2023-03-03T19:58:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.069+0000 7f9686ffd700 1 -- 172.21.15.7:0/1828284789 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9688003d90 con 0x7f9690155e40 2023-03-03T19:58:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.069+0000 7f96948bd700 1 -- 172.21.15.7:0/1828284789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969006d360 con 0x7f9690155e40 2023-03-03T19:58:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.069+0000 7f9686ffd700 1 -- 172.21.15.7:0/1828284789 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9688005310 con 0x7f9690155e40 2023-03-03T19:58:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.069+0000 7f96948bd700 1 -- 172.21.15.7:0/1828284789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9690158480 con 0x7f9690155e40 2023-03-03T19:58:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.070+0000 7f96948bd700 1 -- 172.21.15.7:0/1828284789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96901e0500 con 0x7f9690155e40 2023-03-03T19:58:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.070+0000 7f9686ffd700 1 -- 172.21.15.7:0/1828284789 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f9688005470 con 0x7f9690155e40 2023-03-03T19:58:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.070+0000 7f9686ffd700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96700682c0 0x7f967006a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.071+0000 7f9686ffd700 1 -- 172.21.15.7:0/1828284789 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f96880862d0 con 0x7f9690155e40 2023-03-03T19:58:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.071+0000 7f968e59c700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96700682c0 0x7f967006a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.071+0000 7f968e59c700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96700682c0 0x7f967006a770 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f969004f440 tx=0x7f9680008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.075+0000 7f9686ffd700 1 -- 172.21.15.7:0/1828284789 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9688052670 con 0x7f9690155e40 2023-03-03T19:58:06.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.314+0000 7f96948bd700 1 -- 172.21.15.7:0/1828284789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9690074bb0 con 0x7f9690155e40 2023-03-03T19:58:06.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.315+0000 7f9686ffd700 1 -- 172.21.15.7:0/1828284789 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9688055c90 con 0x7f9690155e40 2023-03-03T19:58:06.316 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:58:06.317 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:58:06.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 -- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96700682c0 msgr2=0x7f967006a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:06.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96700682c0 0x7f967006a770 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f969004f440 tx=0x7f9680008040 comp rx=0 tx=0).stop 2023-03-03T19:58:06.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 -- 172.21.15.7:0/1828284789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9690155e40 msgr2=0x7f9690147e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:06.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9690155e40 0x7f9690147e10 secure :-1 s=READY pgs=3660 cs=0 l=1 rev1=1 crypto rx=0x7f969006e230 tx=0x7f968800ee70 comp rx=0 tx=0).stop 2023-03-03T19:58:06.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 -- 172.21.15.7:0/1828284789 shutdown_connections 2023-03-03T19:58:06.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f96700682c0 0x7f967006a770 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:06.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9690155e40 0x7f9690147e10 unknown :-1 s=CLOSED pgs=3660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:06.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f967c004850 0x7f96900c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:06.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 --2- 172.21.15.7:0/1828284789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f96900c5460 0x7f9690155900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:06.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 -- 172.21.15.7:0/1828284789 >> 172.21.15.7:0/1828284789 conn(0x7f96900a94d0 msgr2=0x7f969014e350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:58:06.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 -- 172.21.15.7:0/1828284789 shutdown_connections 2023-03-03T19:58:06.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:06.317+0000 7f96948bd700 1 -- 172.21.15.7:0/1828284789 wait complete. 2023-03-03T19:58:06.322 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:58:06.329 INFO:tasks.mds_thrash.fs.[a]:waiting for 44 secs before thrashing 2023-03-03T19:58:15.232 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:58:15.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.369+0000 7fde86b66700 1 Processor -- start 2023-03-03T19:58:15.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.369+0000 7fde86b66700 1 -- start start 2023-03-03T19:58:15.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde86b66700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde800aad20 0x7fde80157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:15.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde86b66700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fde800aa030 0x7fde801581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:15.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde86b66700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800728b0 con 0x7fde8014b020 2023-03-03T19:58:15.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde86b66700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde80155070 con 0x7fde800aa030 2023-03-03T19:58:15.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde86b66700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8006f5b0 con 0x7fde800aad20 2023-03-03T19:58:15.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde84902700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fde8014b020 0x7fde800717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37904/0 (socket says 172.21.15.7:37904) 2023-03-03T19:58:15.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde84902700 1 -- 172.21.15.7:0/860119643 learned_addr learned my addr 172.21.15.7:0/860119643 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:58:15.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7ffff700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde800aad20 0x7fde80157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde85103700 1 --2- 172.21.15.7:0/860119643 >> v2:172.21.15.143:3300/0 conn(0x7fde800aa030 0x7fde801581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302028485 0 0) 0x7fde800728b0 con 0x7fde8014b020 2023-03-03T19:58:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde700035b0 con 0x7fde8014b020 2023-03-03T19:58:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2600677177 0 0) 0x7fde700035b0 con 0x7fde8014b020 2023-03-03T19:58:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde800aad20 msgr2=0x7fde80157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde800aad20 0x7fde80157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:15.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 >> v2:172.21.15.143:3300/0 conn(0x7fde800aa030 msgr2=0x7fde801581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 --2- 172.21.15.7:0/860119643 >> v2:172.21.15.143:3300/0 conn(0x7fde800aa030 0x7fde801581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8006f890 con 0x7fde8014b020 2023-03-03T19:58:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fde74001670 con 0x7fde8014b020 2023-03-03T19:58:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 >> v1:172.21.15.7:6789/0 conn(0x7fde8014b020 legacy=0x7fde800717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.370+0000 7fde7f7fe700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde800aad20 0x7fde70003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde7f7fe700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde70003ff0 0x7fde70004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde7f7fe700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde70004d90 0x7fde70005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:15.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde80155070 con 0x7fde70004d90 2023-03-03T19:58:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8006f5b0 con 0x7fde70003ff0 2023-03-03T19:58:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7000bac0 con 0x7fde800aad20 2023-03-03T19:58:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde85103700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde70004d90 0x7fde70005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde84902700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde800aad20 0x7fde70003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde7ffff700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde70003ff0 0x7fde70004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde85103700 1 -- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde800aad20 msgr2=0x7fde70003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:15.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde85103700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde800aad20 0x7fde70003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde85103700 1 -- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde70003ff0 msgr2=0x7fde70004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde85103700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde70003ff0 0x7fde70004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde85103700 1 -- 172.21.15.7:0/860119643 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde700035b0 con 0x7fde70004d90 2023-03-03T19:58:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde85103700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde70004d90 0x7fde70005200 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 crypto rx=0x7fde78009f90 tx=0x7fde70005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a9749639a57159 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7800ec30 con 0x7fde70004d90 2023-03-03T19:58:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7800f040 con 0x7fde70004d90 2023-03-03T19:58:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.371+0000 7fde7f7fe700 1 -- 172.21.15.7:0/860119643 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78014480 con 0x7fde70004d90 2023-03-03T19:58:15.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.372+0000 7fde86b66700 1 -- 172.21.15.7:0/860119643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde70004d90 msgr2=0x7fde70005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.372+0000 7fde86b66700 1 --2- 172.21.15.7:0/860119643 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde70004d90 0x7fde70005200 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 crypto rx=0x7fde78009f90 tx=0x7fde70005880 comp rx=0 tx=0).stop 2023-03-03T19:58:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.372+0000 7fde85103700 1 -- 172.21.15.7:0/860119643 reap_dead start 2023-03-03T19:58:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.372+0000 7fde86b66700 1 -- 172.21.15.7:0/860119643 shutdown_connections 2023-03-03T19:58:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.372+0000 7fde86b66700 1 -- 172.21.15.7:0/860119643 >> 172.21.15.7:0/860119643 conn(0x7fde800c8900 msgr2=0x7fde800c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:58:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.372+0000 7fde86b66700 1 -- 172.21.15.7:0/860119643 shutdown_connections 2023-03-03T19:58:15.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.372+0000 7fde86b66700 1 -- 172.21.15.7:0/860119643 wait complete. 2023-03-03T19:58:15.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.374+0000 7fde86b66700 1 Processor -- start 2023-03-03T19:58:15.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.374+0000 7fde86b66700 1 -- start start 2023-03-03T19:58:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde86b66700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde70004d90 0x7fde80156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde86b66700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde80157320 0x7fde800c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde86b66700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde8014f690 0x7fde80142bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde86b66700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde801ea4f0 con 0x7fde8014f690 2023-03-03T19:58:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde86b66700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde80155650 con 0x7fde80157320 2023-03-03T19:58:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde86b66700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde801565a0 con 0x7fde70004d90 2023-03-03T19:58:15.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde85103700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde8014f690 0x7fde80142bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde85103700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde8014f690 0x7fde80142bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43940/0 (socket says 172.21.15.7:43940) 2023-03-03T19:58:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde85103700 1 -- 172.21.15.7:0/3047791982 learned_addr learned my addr 172.21.15.7:0/3047791982 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:58:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde7ffff700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde80157320 0x7fde800c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde84902700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde70004d90 0x7fde80156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde85103700 1 -- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde70004d90 msgr2=0x7fde80156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde85103700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde70004d90 0x7fde80156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde85103700 1 -- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde80157320 msgr2=0x7fde800c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:15.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde85103700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde80157320 0x7fde800c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde85103700 1 -- 172.21.15.7:0/3047791982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde80155350 con 0x7fde8014f690 2023-03-03T19:58:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.375+0000 7fde85103700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde8014f690 0x7fde80142bc0 secure :-1 s=READY pgs=3663 cs=0 l=1 rev1=1 crypto rx=0x7fde78017040 tx=0x7fde78009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.376+0000 7fde7d7fa700 1 -- 172.21.15.7:0/3047791982 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78009410 con 0x7fde8014f690 2023-03-03T19:58:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.376+0000 7fde7d7fa700 1 -- 172.21.15.7:0/3047791982 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde78014a70 con 0x7fde8014f690 2023-03-03T19:58:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.376+0000 7fde86b66700 1 -- 172.21.15.7:0/3047791982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8006fb70 con 0x7fde8014f690 2023-03-03T19:58:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.376+0000 7fde7d7fa700 1 -- 172.21.15.7:0/3047791982 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78014e20 con 0x7fde8014f690 2023-03-03T19:58:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.376+0000 7fde86b66700 1 -- 172.21.15.7:0/3047791982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde80071ef0 con 0x7fde8014f690 2023-03-03T19:58:15.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.376+0000 7fde86b66700 1 -- 172.21.15.7:0/3047791982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde801ea8c0 con 0x7fde8014f690 2023-03-03T19:58:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.377+0000 7fde7d7fa700 1 -- 172.21.15.7:0/3047791982 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fde7801d420 con 0x7fde8014f690 2023-03-03T19:58:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.377+0000 7fde7d7fa700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde640682f0 0x7fde6406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.377+0000 7fde7d7fa700 1 -- 172.21.15.7:0/3047791982 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fde78087530 con 0x7fde8014f690 2023-03-03T19:58:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.378+0000 7fde84902700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde640682f0 0x7fde6406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:15.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.378+0000 7fde84902700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde640682f0 0x7fde6406a7a0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fde800c7ec0 tx=0x7fde74007980 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:15.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.382+0000 7fde7d7fa700 1 -- 172.21.15.7:0/3047791982 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fde780537f0 con 0x7fde8014f690 2023-03-03T19:58:15.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.617+0000 7fde86b66700 1 -- 172.21.15.7:0/3047791982 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde800c4ca0 con 0x7fde8014f690 2023-03-03T19:58:15.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.618+0000 7fde7d7fa700 1 -- 172.21.15.7:0/3047791982 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fde78056e10 con 0x7fde8014f690 2023-03-03T19:58:15.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:58:15.620 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:58:15.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.619+0000 7fde86b66700 1 -- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde640682f0 msgr2=0x7fde6406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:15.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.619+0000 7fde86b66700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde640682f0 0x7fde6406a7a0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fde800c7ec0 tx=0x7fde74007980 comp rx=0 tx=0).stop 2023-03-03T19:58:15.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.619+0000 7fde86b66700 1 -- 172.21.15.7:0/3047791982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde8014f690 msgr2=0x7fde80142bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:15.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.619+0000 7fde86b66700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde8014f690 0x7fde80142bc0 secure :-1 s=READY pgs=3663 cs=0 l=1 rev1=1 crypto rx=0x7fde78017040 tx=0x7fde78009750 comp rx=0 tx=0).stop 2023-03-03T19:58:15.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.620+0000 7fde86b66700 1 -- 172.21.15.7:0/3047791982 shutdown_connections 2023-03-03T19:58:15.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.620+0000 7fde86b66700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fde640682f0 0x7fde6406a7a0 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:15.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.620+0000 7fde86b66700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fde8014f690 0x7fde80142bc0 unknown :-1 s=CLOSED pgs=3663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:15.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.620+0000 7fde86b66700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fde70004d90 0x7fde80156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:15.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.620+0000 7fde86b66700 1 --2- 172.21.15.7:0/3047791982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fde80157320 0x7fde800c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:15.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.620+0000 7fde86b66700 1 -- 172.21.15.7:0/3047791982 >> 172.21.15.7:0/3047791982 conn(0x7fde800c8900 msgr2=0x7fde800c2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:58:15.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.620+0000 7fde86b66700 1 -- 172.21.15.7:0/3047791982 shutdown_connections 2023-03-03T19:58:15.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:15.620+0000 7fde86b66700 1 -- 172.21.15.7:0/3047791982 wait complete. 2023-03-03T19:58:15.623 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:58:15.634 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before thrashing 2023-03-03T19:58:16.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:58:16.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.772+0000 7f4583b50700 1 Processor -- start 2023-03-03T19:58:16.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.772+0000 7f4583b50700 1 -- start start 2023-03-03T19:58:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f4583b50700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c071990 0x7f457c071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f4583b50700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f457c151c00 0x7f457c0722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f4583b50700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c070360 con 0x7f457c14b460 2023-03-03T19:58:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f4583b50700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f457c1521a0 con 0x7f457c151c00 2023-03-03T19:58:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f4583b50700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457c06a5e0 con 0x7f457c071990 2023-03-03T19:58:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45810eb700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f457c14b460 0x7f457c06aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37926/0 (socket says 172.21.15.7:37926) 2023-03-03T19:58:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45810eb700 1 -- 172.21.15.7:0/1986873785 learned_addr learned my addr 172.21.15.7:0/1986873785 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T19:58:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45820ed700 1 --2- 172.21.15.7:0/1986873785 >> v2:172.21.15.143:3300/0 conn(0x7f457c151c00 0x7f457c0722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45818ec700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c071990 0x7f457c071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374573639 0 0) 0x7f457c070360 con 0x7f457c14b460 2023-03-03T19:58:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45680035b0 con 0x7f457c14b460 2023-03-03T19:58:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 331392738 0 0) 0x7f45680035b0 con 0x7f457c14b460 2023-03-03T19:58:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c071990 msgr2=0x7f457c071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45808ea700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c071990 0x7f457c071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 >> v2:172.21.15.143:3300/0 conn(0x7f457c151c00 msgr2=0x7f457c0722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.773+0000 7f45808ea700 1 --2- 172.21.15.7:0/1986873785 >> v2:172.21.15.143:3300/0 conn(0x7f457c151c00 0x7f457c0722a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0c38b0 con 0x7f457c14b460 2023-03-03T19:58:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f456c001670 con 0x7f457c14b460 2023-03-03T19:58:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 >> v1:172.21.15.7:6789/0 conn(0x7f457c14b460 legacy=0x7f457c06aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c071990 0x7f4568003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4568003ff0 0x7f4568004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4568004d90 0x7f4568005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c1521a0 con 0x7f4568004d90 2023-03-03T19:58:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c06a5e0 con 0x7f4568003ff0 2023-03-03T19:58:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456800dd80 con 0x7f457c071990 2023-03-03T19:58:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45820ed700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4568004d90 0x7f4568005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45818ec700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c071990 0x7f4568003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45810eb700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4568003ff0 0x7f4568004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45820ed700 1 -- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c071990 msgr2=0x7f4568003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45820ed700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c071990 0x7f4568003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45820ed700 1 -- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4568003ff0 msgr2=0x7f4568004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45820ed700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4568003ff0 0x7f4568004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45820ed700 1 -- 172.21.15.7:0/1986873785 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45680035b0 con 0x7f4568004d90 2023-03-03T19:58:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45820ed700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4568004d90 0x7f4568005200 secure :-1 s=READY pgs=3665 cs=0 l=1 rev1=1 crypto rx=0x7f4574009f90 tx=0x7f4568005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c917766ca2651f9f server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457400ec30 con 0x7f4568004d90 2023-03-03T19:58:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457400f040 con 0x7f4568004d90 2023-03-03T19:58:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.774+0000 7f45808ea700 1 -- 172.21.15.7:0/1986873785 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574014480 con 0x7f4568004d90 2023-03-03T19:58:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.775+0000 7f4583b50700 1 -- 172.21.15.7:0/1986873785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4568004d90 msgr2=0x7f4568005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.775+0000 7f4583b50700 1 --2- 172.21.15.7:0/1986873785 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4568004d90 0x7f4568005200 secure :-1 s=READY pgs=3665 cs=0 l=1 rev1=1 crypto rx=0x7f4574009f90 tx=0x7f4568005880 comp rx=0 tx=0).stop 2023-03-03T19:58:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.775+0000 7f45820ed700 1 -- 172.21.15.7:0/1986873785 reap_dead start 2023-03-03T19:58:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.775+0000 7f4583b50700 1 -- 172.21.15.7:0/1986873785 shutdown_connections 2023-03-03T19:58:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.775+0000 7f4583b50700 1 -- 172.21.15.7:0/1986873785 >> 172.21.15.7:0/1986873785 conn(0x7f457c0a94d0 msgr2=0x7f457c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:58:16.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.775+0000 7f4583b50700 1 -- 172.21.15.7:0/1986873785 shutdown_connections 2023-03-03T19:58:16.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.775+0000 7f4583b50700 1 -- 172.21.15.7:0/1986873785 wait complete. 2023-03-03T19:58:16.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.776+0000 7f4583b50700 1 Processor -- start 2023-03-03T19:58:16.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.776+0000 7f4583b50700 1 -- start start 2023-03-03T19:58:16.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f4583b50700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c1e9330 0x7f457c150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:16.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f4583b50700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c13eb30 0x7f457c145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:16.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f4583b50700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f457c1456b0 0x7f457c1e9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:16.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f4583b50700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c0bf250 con 0x7f457c1456b0 2023-03-03T19:58:16.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f4583b50700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c142c10 con 0x7f457c13eb30 2023-03-03T19:58:16.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f4583b50700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457c142ef0 con 0x7f457c1e9330 2023-03-03T19:58:16.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f45820ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f457c1456b0 0x7f457c1e9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:16.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f45820ed700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f457c1456b0 0x7f457c1e9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43962/0 (socket says 172.21.15.7:43962) 2023-03-03T19:58:16.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f45820ed700 1 -- 172.21.15.7:0/2023269434 learned_addr learned my addr 172.21.15.7:0/2023269434 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:58:16.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f45818ec700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c1e9330 0x7f457c150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:16.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f45810eb700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c13eb30 0x7f457c145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:16.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f45820ed700 1 -- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c1e9330 msgr2=0x7f457c150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:16.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f45820ed700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c1e9330 0x7f457c150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:16.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f45820ed700 1 -- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c13eb30 msgr2=0x7f457c145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:16.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f45820ed700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c13eb30 0x7f457c145170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:16.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.777+0000 7f45820ed700 1 -- 172.21.15.7:0/2023269434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0c3b50 con 0x7f457c1456b0 2023-03-03T19:58:16.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.778+0000 7f45810eb700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c13eb30 0x7f457c145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:58:16.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.778+0000 7f45820ed700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f457c1456b0 0x7f457c1e9700 secure :-1 s=READY pgs=3666 cs=0 l=1 rev1=1 crypto rx=0x7f456800dcd0 tx=0x7f4574009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:16.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.778+0000 7f45727fc700 1 -- 172.21.15.7:0/2023269434 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574009410 con 0x7f457c1456b0 2023-03-03T19:58:16.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.778+0000 7f45727fc700 1 -- 172.21.15.7:0/2023269434 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4574014b20 con 0x7f457c1456b0 2023-03-03T19:58:16.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.778+0000 7f4583b50700 1 -- 172.21.15.7:0/2023269434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c06a2b0 con 0x7f457c1456b0 2023-03-03T19:58:16.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.778+0000 7f45727fc700 1 -- 172.21.15.7:0/2023269434 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574014d10 con 0x7f457c1456b0 2023-03-03T19:58:16.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.778+0000 7f4583b50700 1 -- 172.21.15.7:0/2023269434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c071d60 con 0x7f457c1456b0 2023-03-03T19:58:16.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.779+0000 7f4583b50700 1 -- 172.21.15.7:0/2023269434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c1430b0 con 0x7f457c1456b0 2023-03-03T19:58:16.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.779+0000 7f45727fc700 1 -- 172.21.15.7:0/2023269434 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f457401d420 con 0x7f457c1456b0 2023-03-03T19:58:16.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.780+0000 7f45727fc700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 0x7f456006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:16.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.780+0000 7f45727fc700 1 -- 172.21.15.7:0/2023269434 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4574086f00 con 0x7f457c1456b0 2023-03-03T19:58:16.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.780+0000 7f45818ec700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 0x7f456006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:16.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.781+0000 7f45818ec700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 0x7f456006a7a0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f4578009ba0 tx=0x7f4578008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:16.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:16.785+0000 7f45727fc700 1 -- 172.21.15.7:0/2023269434 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f4574053240 con 0x7f457c1456b0 2023-03-03T19:58:17.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.025+0000 7f4583b50700 1 -- 172.21.15.7:0/2023269434 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f457c074bb0 con 0x7f457c1456b0 2023-03-03T19:58:17.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.025+0000 7f45727fc700 1 -- 172.21.15.7:0/2023269434 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f4574056860 con 0x7f457c1456b0 2023-03-03T19:58:17.028 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:58:17.028 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:58:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 -- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 msgr2=0x7f456006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 0x7f456006a7a0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f4578009ba0 tx=0x7f4578008040 comp rx=0 tx=0).stop 2023-03-03T19:58:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 -- 172.21.15.7:0/2023269434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f457c1456b0 msgr2=0x7f457c1e9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f457c1456b0 0x7f457c1e9700 secure :-1 s=READY pgs=3666 cs=0 l=1 rev1=1 crypto rx=0x7f456800dcd0 tx=0x7f4574009750 comp rx=0 tx=0).stop 2023-03-03T19:58:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 -- 172.21.15.7:0/2023269434 shutdown_connections 2023-03-03T19:58:17.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f45600682f0 0x7f456006a7a0 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f457c1456b0 0x7f457c1e9700 unknown :-1 s=CLOSED pgs=3666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f457c1e9330 0x7f457c150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 --2- 172.21.15.7:0/2023269434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f457c13eb30 0x7f457c145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:17.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 -- 172.21.15.7:0/2023269434 >> 172.21.15.7:0/2023269434 conn(0x7f457c0a94d0 msgr2=0x7f457c06a970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:58:17.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 -- 172.21.15.7:0/2023269434 shutdown_connections 2023-03-03T19:58:17.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:17.027+0000 7f4583b50700 1 -- 172.21.15.7:0/2023269434 wait complete. 2023-03-03T19:58:17.034 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:58:17.041 INFO:tasks.mds_thrash.fs.[b]:waiting for 15 secs before thrashing 2023-03-03T19:58:24.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:58:24.359 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:58:32.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:58:32.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.182+0000 7f77276b3700 1 Processor -- start 2023-03-03T19:58:32.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.182+0000 7f77276b3700 1 -- start start 2023-03-03T19:58:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f77276b3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7720071e80 0x7f7720151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f77276b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7720072250 0x7f772014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f77276b3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f77200c1ce0 0x7f772014e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f77276b3700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77201510d0 con 0x7f7720071e80 2023-03-03T19:58:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f77276b3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77200c3740 con 0x7f77200c1ce0 2023-03-03T19:58:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f77276b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7720070670 con 0x7f7720072250 2023-03-03T19:58:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f772544f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7720071e80 0x7f7720151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f772544f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7720071e80 0x7f7720151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43976/0 (socket says 172.21.15.7:43976) 2023-03-03T19:58:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f772544f700 1 -- 172.21.15.7:0/2306605915 learned_addr learned my addr 172.21.15.7:0/2306605915 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:58:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f7724c4e700 1 --2- 172.21.15.7:0/2306605915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7720072250 0x7f772014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f7725c50700 1 --2- 172.21.15.7:0/2306605915 >> v2:172.21.15.143:3300/0 conn(0x7f77200c1ce0 0x7f772014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f772544f700 1 -- 172.21.15.7:0/2306605915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7720072250 msgr2=0x7f772014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f772544f700 1 --2- 172.21.15.7:0/2306605915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7720072250 0x7f772014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f772544f700 1 -- 172.21.15.7:0/2306605915 >> v2:172.21.15.143:3300/0 conn(0x7f77200c1ce0 msgr2=0x7f772014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f772544f700 1 --2- 172.21.15.7:0/2306605915 >> v2:172.21.15.143:3300/0 conn(0x7f77200c1ce0 0x7f772014e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f772544f700 1 -- 172.21.15.7:0/2306605915 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77200c39e0 con 0x7f7720071e80 2023-03-03T19:58:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.183+0000 7f772544f700 1 --2- 172.21.15.7:0/2306605915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720071e80 0x7f7720151820 secure :-1 s=READY pgs=3667 cs=0 l=1 rev1=1 crypto rx=0x7f77200714c0 tx=0x7f771c00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c224ae4ee5a4ed server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f7717fff700 1 -- 172.21.15.7:0/2306605915 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c00dd30 con 0x7f7720071e80 2023-03-03T19:58:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f7717fff700 1 -- 172.21.15.7:0/2306605915 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771c00de90 con 0x7f7720071e80 2023-03-03T19:58:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f7717fff700 1 -- 172.21.15.7:0/2306605915 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c010690 con 0x7f7720071e80 2023-03-03T19:58:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f77276b3700 1 -- 172.21.15.7:0/2306605915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720071e80 msgr2=0x7f7720151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f77276b3700 1 --2- 172.21.15.7:0/2306605915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720071e80 0x7f7720151820 secure :-1 s=READY pgs=3667 cs=0 l=1 rev1=1 crypto rx=0x7f77200714c0 tx=0x7f771c00be70 comp rx=0 tx=0).stop 2023-03-03T19:58:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f77276b3700 1 -- 172.21.15.7:0/2306605915 shutdown_connections 2023-03-03T19:58:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f77276b3700 1 --2- 172.21.15.7:0/2306605915 >> v2:172.21.15.143:3300/0 conn(0x7f77200c1ce0 0x7f772014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f77276b3700 1 --2- 172.21.15.7:0/2306605915 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720071e80 0x7f7720151820 unknown :-1 s=CLOSED pgs=3667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f77276b3700 1 --2- 172.21.15.7:0/2306605915 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7720072250 0x7f772014e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f77276b3700 1 -- 172.21.15.7:0/2306605915 >> 172.21.15.7:0/2306605915 conn(0x7f77200a94d0 msgr2=0x7f77200a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:58:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f77276b3700 1 -- 172.21.15.7:0/2306605915 shutdown_connections 2023-03-03T19:58:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.184+0000 7f77276b3700 1 -- 172.21.15.7:0/2306605915 wait complete. 2023-03-03T19:58:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.185+0000 7f77276b3700 1 Processor -- start 2023-03-03T19:58:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f77276b3700 1 -- start start 2023-03-03T19:58:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f77276b3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7720071e80 0x7f7720147570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f77276b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720072250 0x7f77201e8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f77276b3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77200c1ce0 0x7f7720148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f77276b3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77200ac440 con 0x7f7720072250 2023-03-03T19:58:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f77276b3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77201e98c0 con 0x7f7720071e80 2023-03-03T19:58:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f77276b3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77200ac6e0 con 0x7f77200c1ce0 2023-03-03T19:58:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f7724c4e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720072250 0x7f77201e8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f7724c4e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720072250 0x7f77201e8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43984/0 (socket says 172.21.15.7:43984) 2023-03-03T19:58:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f7724c4e700 1 -- 172.21.15.7:0/100743812 learned_addr learned my addr 172.21.15.7:0/100743812 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:58:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.186+0000 7f7725c50700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77200c1ce0 0x7f7720148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f772544f700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7720071e80 0x7f7720147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f7724c4e700 1 -- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77200c1ce0 msgr2=0x7f7720148700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:32.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f7724c4e700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77200c1ce0 0x7f7720148700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f7724c4e700 1 -- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7720071e80 msgr2=0x7f7720147570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f7724c4e700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7720071e80 0x7f7720147570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f7724c4e700 1 -- 172.21.15.7:0/100743812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c00b7f0 con 0x7f7720072250 2023-03-03T19:58:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f7724c4e700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720072250 0x7f77201e8f80 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7f771000e810 tx=0x7f771000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f7715ffb700 1 -- 172.21.15.7:0/100743812 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710004230 con 0x7f7720072250 2023-03-03T19:58:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f7715ffb700 1 -- 172.21.15.7:0/100743812 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7710004390 con 0x7f7720072250 2023-03-03T19:58:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f77276b3700 1 -- 172.21.15.7:0/100743812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772006c9a0 con 0x7f7720072250 2023-03-03T19:58:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f7715ffb700 1 -- 172.21.15.7:0/100743812 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100036a0 con 0x7f7720072250 2023-03-03T19:58:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.187+0000 7f77276b3700 1 -- 172.21.15.7:0/100743812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7720151300 con 0x7f7720072250 2023-03-03T19:58:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.188+0000 7f77276b3700 1 -- 172.21.15.7:0/100743812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772006cb30 con 0x7f7720072250 2023-03-03T19:58:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.189+0000 7f7715ffb700 1 -- 172.21.15.7:0/100743812 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7710003800 con 0x7f7720072250 2023-03-03T19:58:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.189+0000 7f7715ffb700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77040682f0 0x7f770406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.190+0000 7f7715ffb700 1 -- 172.21.15.7:0/100743812 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7710085d40 con 0x7f7720072250 2023-03-03T19:58:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.190+0000 7f772544f700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77040682f0 0x7f770406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.190+0000 7f772544f700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77040682f0 0x7f770406a7a0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f771c00beb0 tx=0x7f771c012000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.194+0000 7f7715ffb700 1 -- 172.21.15.7:0/100743812 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f77100520e0 con 0x7f7720072250 2023-03-03T19:58:32.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.435+0000 7f77276b3700 1 -- 172.21.15.7:0/100743812 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7720074bb0 con 0x7f7720072250 2023-03-03T19:58:32.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.436+0000 7f7715ffb700 1 -- 172.21.15.7:0/100743812 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7710055700 con 0x7f7720072250 2023-03-03T19:58:32.437 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:58:32.437 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:58:32.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 -- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77040682f0 msgr2=0x7f770406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:32.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77040682f0 0x7f770406a7a0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f771c00beb0 tx=0x7f771c012000 comp rx=0 tx=0).stop 2023-03-03T19:58:32.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 -- 172.21.15.7:0/100743812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720072250 msgr2=0x7f77201e8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:32.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720072250 0x7f77201e8f80 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7f771000e810 tx=0x7f771000eb20 comp rx=0 tx=0).stop 2023-03-03T19:58:32.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 -- 172.21.15.7:0/100743812 shutdown_connections 2023-03-03T19:58:32.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f77200c1ce0 0x7f7720148700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7720071e80 0x7f7720147570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f77040682f0 0x7f770406a7a0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 --2- 172.21.15.7:0/100743812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7720072250 0x7f77201e8f80 unknown :-1 s=CLOSED pgs=3668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:32.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 -- 172.21.15.7:0/100743812 >> 172.21.15.7:0/100743812 conn(0x7f77200a94d0 msgr2=0x7f772014b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:58:32.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 -- 172.21.15.7:0/100743812 shutdown_connections 2023-03-03T19:58:32.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:32.438+0000 7f77276b3700 1 -- 172.21.15.7:0/100743812 wait complete. 2023-03-03T19:58:32.441 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:58:32.451 INFO:tasks.mds_thrash.fs.[b]:waiting for 93 secs before thrashing 2023-03-03T19:58:50.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:58:50.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.470+0000 7fc7d5e96700 1 Processor -- start 2023-03-03T19:58:50.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.470+0000 7fc7d5e96700 1 -- start start 2023-03-03T19:58:50.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7d5e96700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7d014f9d0 0x7fc7d0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:50.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7d5e96700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 0x7fc7d01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:50.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7d5e96700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d00c2f30 con 0x7fc7d014fda0 2023-03-03T19:58:50.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7d5e96700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d00c31d0 con 0x7fc7d014f9d0 2023-03-03T19:58:50.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7d5e96700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d01524a0 con 0x7fc7d00abe20 2023-03-03T19:58:50.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7cf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7d014f9d0 0x7fc7d0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:50.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7cf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc7d014f9d0 0x7fc7d0156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44006/0 (socket says 172.21.15.7:44006) 2023-03-03T19:58:50.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7cf7fe700 1 -- 172.21.15.7:0/2493944386 learned_addr learned my addr 172.21.15.7:0/2493944386 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:58:50.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7cffff700 1 --2- 172.21.15.7:0/2493944386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 0x7fc7d01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:50.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7cf7fe700 1 -- 172.21.15.7:0/2493944386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 msgr2=0x7fc7d01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:58:50.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/2493944386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 0x7fc7d01529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:50.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7cf7fe700 1 -- 172.21.15.7:0/2493944386 >> v1:172.21.15.143:6789/0 conn(0x7fc7d014fda0 legacy=0x7fc7d0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:50.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7cf7fe700 1 -- 172.21.15.7:0/2493944386 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d00c34b0 con 0x7fc7d014f9d0 2023-03-03T19:58:50.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.471+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/2493944386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 0x7fc7d0156430 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00098f0 tx=0x7fc7c0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d887dacc1f4a3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:50.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7ce7fc700 1 -- 172.21.15.7:0/2493944386 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000b8a0 con 0x7fc7d014f9d0 2023-03-03T19:58:50.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7ce7fc700 1 -- 172.21.15.7:0/2493944386 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c000ba00 con 0x7fc7d014f9d0 2023-03-03T19:58:50.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7ce7fc700 1 -- 172.21.15.7:0/2493944386 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000bd90 con 0x7fc7d014f9d0 2023-03-03T19:58:50.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7d5e96700 1 -- 172.21.15.7:0/2493944386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 msgr2=0x7fc7d0156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:50.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7d5e96700 1 --2- 172.21.15.7:0/2493944386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 0x7fc7d0156430 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00098f0 tx=0x7fc7c0009c00 comp rx=0 tx=0).stop 2023-03-03T19:58:50.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7d5e96700 1 -- 172.21.15.7:0/2493944386 shutdown_connections 2023-03-03T19:58:50.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7d5e96700 1 --2- 172.21.15.7:0/2493944386 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 0x7fc7d01529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:50.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7d5e96700 1 --2- 172.21.15.7:0/2493944386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 0x7fc7d0156430 unknown :-1 s=CLOSED pgs=3669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:50.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7d5e96700 1 -- 172.21.15.7:0/2493944386 >> 172.21.15.7:0/2493944386 conn(0x7fc7d00c8900 msgr2=0x7fc7d00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:58:50.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7d5e96700 1 -- 172.21.15.7:0/2493944386 shutdown_connections 2023-03-03T19:58:50.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.472+0000 7fc7d5e96700 1 -- 172.21.15.7:0/2493944386 wait complete. 2023-03-03T19:58:50.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.473+0000 7fc7d5e96700 1 Processor -- start 2023-03-03T19:58:50.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.474+0000 7fc7d5e96700 1 -- start start 2023-03-03T19:58:50.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.474+0000 7fc7d5e96700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 0x7fc7d014b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:50.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.474+0000 7fc7d5e96700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 0x7fc7d01dea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:50.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.474+0000 7fc7d5e96700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d014fda0 0x7fc7d01defd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:50.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.474+0000 7fc7d5e96700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d006e4d0 con 0x7fc7d014f9d0 2023-03-03T19:58:50.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.474+0000 7fc7d5e96700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d006ea50 con 0x7fc7d014fda0 2023-03-03T19:58:50.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.474+0000 7fc7d5e96700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d0152150 con 0x7fc7d00abe20 2023-03-03T19:58:50.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7ceffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 0x7fc7d01dea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:50.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7cf7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 0x7fc7d014b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:50.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7ceffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 0x7fc7d01dea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44014/0 (socket says 172.21.15.7:44014) 2023-03-03T19:58:50.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7cf7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 0x7fc7d014b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:52084/0 (socket says 172.21.15.7:52084) 2023-03-03T19:58:50.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7ceffd700 1 -- 172.21.15.7:0/66114714 learned_addr learned my addr 172.21.15.7:0/66114714 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:58:50.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7cffff700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d014fda0 0x7fc7d01defd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:50.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7ceffd700 1 -- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 msgr2=0x7fc7d014b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:50.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7ceffd700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 0x7fc7d014b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:50.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7ceffd700 1 -- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d014fda0 msgr2=0x7fc7d01defd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:50.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7ceffd700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d014fda0 0x7fc7d01defd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:50.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7ceffd700 1 -- 172.21.15.7:0/66114714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c0009550 con 0x7fc7d014f9d0 2023-03-03T19:58:50.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7ceffd700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 0x7fc7d01dea90 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7fc7b800c4b0 tx=0x7fc7b800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:50.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7c7fff700 1 -- 172.21.15.7:0/66114714 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b80052d0 con 0x7fc7d014f9d0 2023-03-03T19:58:50.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7c7fff700 1 -- 172.21.15.7:0/66114714 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b8005430 con 0x7fc7d014f9d0 2023-03-03T19:58:50.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7d5e96700 1 -- 172.21.15.7:0/66114714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d00ac350 con 0x7fc7d014f9d0 2023-03-03T19:58:50.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7c7fff700 1 -- 172.21.15.7:0/66114714 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b80038c0 con 0x7fc7d014f9d0 2023-03-03T19:58:50.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.475+0000 7fc7d5e96700 1 -- 172.21.15.7:0/66114714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d00ab630 con 0x7fc7d014f9d0 2023-03-03T19:58:50.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.476+0000 7fc7d5e96700 1 -- 172.21.15.7:0/66114714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d0071fa0 con 0x7fc7d014f9d0 2023-03-03T19:58:50.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.477+0000 7fc7c7fff700 1 -- 172.21.15.7:0/66114714 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc7b8019030 con 0x7fc7d014f9d0 2023-03-03T19:58:50.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.477+0000 7fc7c7fff700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:58:50.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.477+0000 7fc7c7fff700 1 -- 172.21.15.7:0/66114714 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc7b8085350 con 0x7fc7d014f9d0 2023-03-03T19:58:50.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.477+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:58:50.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.478+0000 7fc7cf7fe700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009840 tx=0x7fc7c0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:58:50.488 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.482+0000 7fc7c7fff700 1 -- 172.21.15.7:0/66114714 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc7b80516f0 con 0x7fc7d014f9d0 2023-03-03T19:58:50.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.719+0000 7fc7d5e96700 1 -- 172.21.15.7:0/66114714 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7d0074bb0 con 0x7fc7d014f9d0 2023-03-03T19:58:50.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.720+0000 7fc7c7fff700 1 -- 172.21.15.7:0/66114714 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc7b8054d10 con 0x7fc7d014f9d0 2023-03-03T19:58:50.721 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:58:50.721 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:58:50.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 -- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 msgr2=0x7fc7b006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009840 tx=0x7fc7c0008030 comp rx=0 tx=0).stop 2023-03-03T19:58:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 -- 172.21.15.7:0/66114714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 msgr2=0x7fc7d01dea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:58:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 0x7fc7d01dea90 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7fc7b800c4b0 tx=0x7fc7b800c7c0 comp rx=0 tx=0).stop 2023-03-03T19:58:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 -- 172.21.15.7:0/66114714 shutdown_connections 2023-03-03T19:58:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7d014fda0 0x7fc7d01defd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc7d00abe20 0x7fc7d014b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc7b00682f0 0x7fc7b006a7a0 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 --2- 172.21.15.7:0/66114714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc7d014f9d0 0x7fc7d01dea90 unknown :-1 s=CLOSED pgs=3670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:58:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 -- 172.21.15.7:0/66114714 >> 172.21.15.7:0/66114714 conn(0x7fc7d00c8900 msgr2=0x7fc7d00c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:58:50.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 -- 172.21.15.7:0/66114714 shutdown_connections 2023-03-03T19:58:50.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:58:50.722+0000 7fc7d5e96700 1 -- 172.21.15.7:0/66114714 wait complete. 2023-03-03T19:58:50.725 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:58:50.735 INFO:tasks.mds_thrash.fs.[a]:waiting for 61 secs before thrashing 2023-03-03T19:58:54.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:58:54.440 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:59:24.518 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:59:24.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:59:51.737 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T19:59:51.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.876+0000 7fe967170700 1 Processor -- start 2023-03-03T19:59:51.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.876+0000 7fe967170700 1 -- start start 2023-03-03T19:59:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe967170700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fe960071e80 0x7fe960151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:59:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe967170700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe960072250 0x7fe96014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:59:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe967170700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9601510d0 con 0x7fe9600c1ce0 2023-03-03T19:59:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe967170700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9600c3740 con 0x7fe960072250 2023-03-03T19:59:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe967170700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe960070670 con 0x7fe960071e80 2023-03-03T19:59:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe95ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe960072250 0x7fe96014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:59:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe95ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe960072250 0x7fe96014e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44070/0 (socket says 172.21.15.7:44070) 2023-03-03T19:59:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe95ffff700 1 -- 172.21.15.7:0/2449974113 learned_addr learned my addr 172.21.15.7:0/2449974113 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:59:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe964f0c700 1 --2- 172.21.15.7:0/2449974113 >> v2:172.21.15.143:3300/0 conn(0x7fe960071e80 0x7fe960151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:59:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe95ffff700 1 -- 172.21.15.7:0/2449974113 >> v2:172.21.15.143:3300/0 conn(0x7fe960071e80 msgr2=0x7fe960151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T19:59:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe95ffff700 1 --2- 172.21.15.7:0/2449974113 >> v2:172.21.15.143:3300/0 conn(0x7fe960071e80 0x7fe960151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:59:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe95ffff700 1 -- 172.21.15.7:0/2449974113 >> v1:172.21.15.143:6789/0 conn(0x7fe9600c1ce0 legacy=0x7fe9601546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:59:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe95ffff700 1 -- 172.21.15.7:0/2449974113 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9600c39e0 con 0x7fe960072250 2023-03-03T19:59:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.877+0000 7fe95ffff700 1 --2- 172.21.15.7:0/2449974113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960072250 0x7fe96014e1f0 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 crypto rx=0x7fe94c0098f0 tx=0x7fe94c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c763df91be4a369 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:59:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe95f7fe700 1 -- 172.21.15.7:0/2449974113 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c00b860 con 0x7fe960072250 2023-03-03T19:59:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe95f7fe700 1 -- 172.21.15.7:0/2449974113 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c00b9c0 con 0x7fe960072250 2023-03-03T19:59:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe95f7fe700 1 -- 172.21.15.7:0/2449974113 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c00bd50 con 0x7fe960072250 2023-03-03T19:59:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe967170700 1 -- 172.21.15.7:0/2449974113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960072250 msgr2=0x7fe96014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:59:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe967170700 1 --2- 172.21.15.7:0/2449974113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960072250 0x7fe96014e1f0 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 crypto rx=0x7fe94c0098f0 tx=0x7fe94c009c00 comp rx=0 tx=0).stop 2023-03-03T19:59:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe967170700 1 -- 172.21.15.7:0/2449974113 shutdown_connections 2023-03-03T19:59:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe967170700 1 --2- 172.21.15.7:0/2449974113 >> v2:172.21.15.143:3300/0 conn(0x7fe960071e80 0x7fe960151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:59:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe967170700 1 --2- 172.21.15.7:0/2449974113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960072250 0x7fe96014e1f0 unknown :-1 s=CLOSED pgs=3671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:59:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe967170700 1 -- 172.21.15.7:0/2449974113 >> 172.21.15.7:0/2449974113 conn(0x7fe9600a94d0 msgr2=0x7fe9600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:59:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe967170700 1 -- 172.21.15.7:0/2449974113 shutdown_connections 2023-03-03T19:59:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.878+0000 7fe967170700 1 -- 172.21.15.7:0/2449974113 wait complete. 2023-03-03T19:59:51.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.879+0000 7fe967170700 1 Processor -- start 2023-03-03T19:59:51.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.879+0000 7fe967170700 1 -- start start 2023-03-03T19:59:51.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.880+0000 7fe967170700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960071e80 0x7fe960147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:59:51.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.880+0000 7fe967170700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe960072250 0x7fe960147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:59:51.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.880+0000 7fe967170700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9600c1ce0 0x7fe9601eb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:59:51.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.880+0000 7fe967170700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96014a980 con 0x7fe960071e80 2023-03-03T19:59:51.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.880+0000 7fe967170700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9601de970 con 0x7fe960072250 2023-03-03T19:59:51.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.880+0000 7fe967170700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96014a070 con 0x7fe9600c1ce0 2023-03-03T19:59:51.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.880+0000 7fe964f0c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960071e80 0x7fe960147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:59:51.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.880+0000 7fe964f0c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960071e80 0x7fe960147350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44074/0 (socket says 172.21.15.7:44074) 2023-03-03T19:59:51.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.880+0000 7fe964f0c700 1 -- 172.21.15.7:0/1623583068 learned_addr learned my addr 172.21.15.7:0/1623583068 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T19:59:51.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe96570d700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9600c1ce0 0x7fe9601eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:59:51.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe95ffff700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe960072250 0x7fe960147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:59:51.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe964f0c700 1 -- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9600c1ce0 msgr2=0x7fe9601eb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:59:51.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe964f0c700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9600c1ce0 0x7fe9601eb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:59:51.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe964f0c700 1 -- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe960072250 msgr2=0x7fe960147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:59:51.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe964f0c700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe960072250 0x7fe960147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:59:51.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe964f0c700 1 -- 172.21.15.7:0/1623583068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c009550 con 0x7fe960071e80 2023-03-03T19:59:51.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe95ffff700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe960072250 0x7fe960147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T19:59:51.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe964f0c700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960071e80 0x7fe960147350 secure :-1 s=READY pgs=3672 cs=0 l=1 rev1=1 crypto rx=0x7fe95400e810 tx=0x7fe95400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:59:51.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe95d7fa700 1 -- 172.21.15.7:0/1623583068 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe954013610 con 0x7fe960071e80 2023-03-03T19:59:51.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe95d7fa700 1 -- 172.21.15.7:0/1623583068 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe954013770 con 0x7fe960071e80 2023-03-03T19:59:51.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe967170700 1 -- 172.21.15.7:0/1623583068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96006c9a0 con 0x7fe960071e80 2023-03-03T19:59:51.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe95d7fa700 1 -- 172.21.15.7:0/1623583068 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe954013b20 con 0x7fe960071e80 2023-03-03T19:59:51.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.881+0000 7fe967170700 1 -- 172.21.15.7:0/1623583068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe960151300 con 0x7fe960071e80 2023-03-03T19:59:51.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.882+0000 7fe967170700 1 -- 172.21.15.7:0/1623583068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96006cb30 con 0x7fe960071e80 2023-03-03T19:59:51.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.883+0000 7fe95d7fa700 1 -- 172.21.15.7:0/1623583068 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe954023070 con 0x7fe960071e80 2023-03-03T19:59:51.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.883+0000 7fe95d7fa700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe944068320 0x7fe94406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T19:59:51.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.883+0000 7fe95d7fa700 1 -- 172.21.15.7:0/1623583068 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe954085e40 con 0x7fe960071e80 2023-03-03T19:59:51.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.883+0000 7fe95ffff700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe944068320 0x7fe94406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T19:59:51.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.884+0000 7fe95ffff700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe944068320 0x7fe94406a7d0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fe9600c1780 tx=0x7fe94c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T19:59:51.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:51.887+0000 7fe95d7fa700 1 -- 172.21.15.7:0/1623583068 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe954052160 con 0x7fe960071e80 2023-03-03T19:59:52.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.126+0000 7fe967170700 1 -- 172.21.15.7:0/1623583068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe960074bb0 con 0x7fe960071e80 2023-03-03T19:59:52.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.127+0000 7fe95d7fa700 1 -- 172.21.15.7:0/1623583068 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe954085020 con 0x7fe960071e80 2023-03-03T19:59:52.128 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T19:59:52.128 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T19:59:52.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 -- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe944068320 msgr2=0x7fe94406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:59:52.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe944068320 0x7fe94406a7d0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fe9600c1780 tx=0x7fe94c008030 comp rx=0 tx=0).stop 2023-03-03T19:59:52.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 -- 172.21.15.7:0/1623583068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960071e80 msgr2=0x7fe960147350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T19:59:52.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960071e80 0x7fe960147350 secure :-1 s=READY pgs=3672 cs=0 l=1 rev1=1 crypto rx=0x7fe95400e810 tx=0x7fe95400ebd0 comp rx=0 tx=0).stop 2023-03-03T19:59:52.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 -- 172.21.15.7:0/1623583068 shutdown_connections 2023-03-03T19:59:52.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe9600c1ce0 0x7fe9601eb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:59:52.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe944068320 0x7fe94406a7d0 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:59:52.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe960071e80 0x7fe960147350 unknown :-1 s=CLOSED pgs=3672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:59:52.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 --2- 172.21.15.7:0/1623583068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe960072250 0x7fe960147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T19:59:52.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 -- 172.21.15.7:0/1623583068 >> 172.21.15.7:0/1623583068 conn(0x7fe9600a94d0 msgr2=0x7fe960151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T19:59:52.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 -- 172.21.15.7:0/1623583068 shutdown_connections 2023-03-03T19:59:52.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T19:59:52.129+0000 7fe967170700 1 -- 172.21.15.7:0/1623583068 wait complete. 2023-03-03T19:59:52.132 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T19:59:52.142 INFO:tasks.mds_thrash.fs.[a]:waiting for 80 secs before thrashing 2023-03-03T19:59:54.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T19:59:54.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:00:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T19:59:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T20:00:05.452 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:00:05.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.591+0000 7f69b45e6700 1 Processor -- start 2023-03-03T20:00:05.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.591+0000 7f69b45e6700 1 -- start start 2023-03-03T20:00:05.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.592+0000 7f69b45e6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f69ac0c2930 0x7f69ac157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:00:05.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.592+0000 7f69b45e6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69ac06c480 0x7f69ac157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:00:05.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.592+0000 7f69b45e6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ac06d430 con 0x7f69ac151d40 2023-03-03T20:00:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.592+0000 7f69b45e6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ac06c2a0 con 0x7f69ac0c2930 2023-03-03T20:00:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.592+0000 7f69b45e6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac06cb00 con 0x7f69ac06c480 2023-03-03T20:00:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.592+0000 7f69b2382700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f69ac151d40 0x7f69ac0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38066/0 (socket says 172.21.15.7:38066) 2023-03-03T20:00:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.592+0000 7f69b2382700 1 -- 172.21.15.7:0/3617602993 learned_addr learned my addr 172.21.15.7:0/3617602993 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:00:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b2b83700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69ac06c480 0x7f69ac157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:00:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243848272 0 0) 0x7f69ac06d430 con 0x7f69ac151d40 2023-03-03T20:00:05.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69980035b0 con 0x7f69ac151d40 2023-03-03T20:00:05.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1b81700 1 --2- 172.21.15.7:0/3617602993 >> v2:172.21.15.143:3300/0 conn(0x7f69ac0c2930 0x7f69ac157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:00:05.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3265872133 0 0) 0x7f69980035b0 con 0x7f69ac151d40 2023-03-03T20:00:05.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69ac06c480 msgr2=0x7f69ac157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:00:05.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69ac06c480 0x7f69ac157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:00:05.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 >> v2:172.21.15.143:3300/0 conn(0x7f69ac0c2930 msgr2=0x7f69ac157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:00:05.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 --2- 172.21.15.7:0/3617602993 >> v2:172.21.15.143:3300/0 conn(0x7f69ac0c2930 0x7f69ac157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:00:05.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac06cde0 con 0x7f69ac151d40 2023-03-03T20:00:05.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f69a8001670 con 0x7f69ac151d40 2023-03-03T20:00:05.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 >> v1:172.21.15.7:6789/0 conn(0x7f69ac151d40 legacy=0x7f69ac0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:00:05.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6998003a70 0x7f6998003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:00:05.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69ac06c480 0x7f6998004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:00:05.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6998004d90 0x7f6998005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:00:05.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac06c2a0 con 0x7f6998003a70 2023-03-03T20:00:05.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac06cb00 con 0x7f6998004d90 2023-03-03T20:00:05.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699800bb40 con 0x7f69ac06c480 2023-03-03T20:00:05.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b2382700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6998003a70 0x7f6998003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:00:05.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b1b81700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69ac06c480 0x7f6998004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:00:05.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.593+0000 7f69b2b83700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6998004d90 0x7f6998005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:00:05.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b2382700 1 -- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69ac06c480 msgr2=0x7f6998004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:00:05.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b2382700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f69ac06c480 0x7f6998004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:00:05.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b2382700 1 -- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6998004d90 msgr2=0x7f6998005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:00:05.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b2382700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6998004d90 0x7f6998005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:00:05.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b2382700 1 -- 172.21.15.7:0/3617602993 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69980035b0 con 0x7f6998003a70 2023-03-03T20:00:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b2382700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6998003a70 0x7f6998003e80 secure :-1 s=READY pgs=3674 cs=0 l=1 rev1=1 crypto rx=0x7f69a800c320 tx=0x7f6998005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd27964e0f5ebf79 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:00:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a800ce60 con 0x7f6998003a70 2023-03-03T20:00:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a8005960 con 0x7f6998003a70 2023-03-03T20:00:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b1380700 1 -- 172.21.15.7:0/3617602993 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8005cf0 con 0x7f6998003a70 2023-03-03T20:00:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b45e6700 1 -- 172.21.15.7:0/3617602993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6998003a70 msgr2=0x7f6998003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:00:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b45e6700 1 --2- 172.21.15.7:0/3617602993 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6998003a70 0x7f6998003e80 secure :-1 s=READY pgs=3674 cs=0 l=1 rev1=1 crypto rx=0x7f69a800c320 tx=0x7f6998005880 comp rx=0 tx=0).stop 2023-03-03T20:00:05.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b2b83700 1 -- 172.21.15.7:0/3617602993 reap_dead start 2023-03-03T20:00:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b45e6700 1 -- 172.21.15.7:0/3617602993 shutdown_connections 2023-03-03T20:00:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b45e6700 1 -- 172.21.15.7:0/3617602993 >> 172.21.15.7:0/3617602993 conn(0x7f69ac0a94d0 msgr2=0x7f69ac0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:00:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b45e6700 1 -- 172.21.15.7:0/3617602993 shutdown_connections 2023-03-03T20:00:05.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.594+0000 7f69b45e6700 1 -- 172.21.15.7:0/3617602993 wait complete. 2023-03-03T20:00:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.596+0000 7f69b45e6700 1 Processor -- start 2023-03-03T20:00:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.596+0000 7f69b45e6700 1 -- start start 2023-03-03T20:00:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.596+0000 7f69b45e6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6998003a70 0x7f69ac156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:00:05.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.596+0000 7f69b45e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69ac148cb0 0x7f69ac147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:00:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.596+0000 7f69b45e6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69ac148090 0x7f69ac1ed9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:00:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.596+0000 7f69b45e6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac1ea180 con 0x7f69ac148cb0 2023-03-03T20:00:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.596+0000 7f69b45e6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac148780 con 0x7f69ac148090 2023-03-03T20:00:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.596+0000 7f69b45e6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac1489d0 con 0x7f6998003a70 2023-03-03T20:00:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.596+0000 7f69b1b81700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69ac148cb0 0x7f69ac147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:00:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b1b81700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69ac148cb0 0x7f69ac147b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44100/0 (socket says 172.21.15.7:44100) 2023-03-03T20:00:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b1b81700 1 -- 172.21.15.7:0/3935619345 learned_addr learned my addr 172.21.15.7:0/3935619345 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:00:05.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b1b81700 1 -- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6998003a70 msgr2=0x7f69ac156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:00:05.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b1b81700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6998003a70 0x7f69ac156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:00:05.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b1b81700 1 -- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69ac148090 msgr2=0x7f69ac1ed9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:00:05.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b2b83700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69ac148090 0x7f69ac1ed9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:00:05.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b1b81700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69ac148090 0x7f69ac1ed9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:00:05.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b1b81700 1 -- 172.21.15.7:0/3935619345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac073e00 con 0x7f69ac148cb0 2023-03-03T20:00:05.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b2b83700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69ac148090 0x7f69ac1ed9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:00:05.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b1b81700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69ac148cb0 0x7f69ac147b50 secure :-1 s=READY pgs=3675 cs=0 l=1 rev1=1 crypto rx=0x7f69a0011aa0 tx=0x7f69a0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:00:05.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f699effd700 1 -- 172.21.15.7:0/3935619345 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a000f980 con 0x7f69ac148cb0 2023-03-03T20:00:05.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f699effd700 1 -- 172.21.15.7:0/3935619345 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a000fb00 con 0x7f69ac148cb0 2023-03-03T20:00:05.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b45e6700 1 -- 172.21.15.7:0/3935619345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac06d0c0 con 0x7f69ac148cb0 2023-03-03T20:00:05.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f699effd700 1 -- 172.21.15.7:0/3935619345 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a000feb0 con 0x7f69ac148cb0 2023-03-03T20:00:05.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.597+0000 7f69b45e6700 1 -- 172.21.15.7:0/3935619345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac06bea0 con 0x7f69ac148cb0 2023-03-03T20:00:05.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.598+0000 7f69b45e6700 1 -- 172.21.15.7:0/3935619345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac074bb0 con 0x7f69ac148cb0 2023-03-03T20:00:05.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.599+0000 7f699effd700 1 -- 172.21.15.7:0/3935619345 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f69a0014440 con 0x7f69ac148cb0 2023-03-03T20:00:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.599+0000 7f699effd700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69900682f0 0x7f699006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:00:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.599+0000 7f699effd700 1 -- 172.21.15.7:0/3935619345 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f69a00869b0 con 0x7f69ac148cb0 2023-03-03T20:00:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.599+0000 7f69b2382700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69900682f0 0x7f699006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:00:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.600+0000 7f69b2382700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69900682f0 0x7f699006a7a0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f69ac157400 tx=0x7f69a800f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:00:05.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.603+0000 7f699effd700 1 -- 172.21.15.7:0/3935619345 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f69a0052e60 con 0x7f69ac148cb0 2023-03-03T20:00:05.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.842+0000 7f69b45e6700 1 -- 172.21.15.7:0/3935619345 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69ac071950 con 0x7f69ac148cb0 2023-03-03T20:00:05.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.843+0000 7f699effd700 1 -- 172.21.15.7:0/3935619345 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f69a0057e60 con 0x7f69ac148cb0 2023-03-03T20:00:05.844 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:00:05.844 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:00:05.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 -- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69900682f0 msgr2=0x7f699006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:00:05.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69900682f0 0x7f699006a7a0 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f69ac157400 tx=0x7f69a800f000 comp rx=0 tx=0).stop 2023-03-03T20:00:05.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 -- 172.21.15.7:0/3935619345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69ac148cb0 msgr2=0x7f69ac147b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:00:05.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69ac148cb0 0x7f69ac147b50 secure :-1 s=READY pgs=3675 cs=0 l=1 rev1=1 crypto rx=0x7f69a0011aa0 tx=0x7f69a0011e60 comp rx=0 tx=0).stop 2023-03-03T20:00:05.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 -- 172.21.15.7:0/3935619345 shutdown_connections 2023-03-03T20:00:05.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69ac148090 0x7f69ac1ed9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:00:05.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6998003a70 0x7f69ac156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:00:05.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f69900682f0 0x7f699006a7a0 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:00:05.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 --2- 172.21.15.7:0/3935619345 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69ac148cb0 0x7f69ac147b50 unknown :-1 s=CLOSED pgs=3675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:00:05.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 -- 172.21.15.7:0/3935619345 >> 172.21.15.7:0/3935619345 conn(0x7f69ac0a94d0 msgr2=0x7f69ac1558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:00:05.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 -- 172.21.15.7:0/3935619345 shutdown_connections 2023-03-03T20:00:05.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:00:05.845+0000 7f69b45e6700 1 -- 172.21.15.7:0/3935619345 wait complete. 2023-03-03T20:00:05.848 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:00:05.858 INFO:tasks.mds_thrash.fs.[b]:waiting for 78 secs before thrashing 2023-03-03T20:00:24.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:00:24.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:00:54.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:00:54.756 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:01:12.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:01:12.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.284+0000 7f6bfd5d6700 1 Processor -- start 2023-03-03T20:01:12.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.284+0000 7f6bfd5d6700 1 -- start start 2023-03-03T20:01:12.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bfd5d6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 0x7f6bf806b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:12.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bfd5d6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bf806be60 0x7f6bf80ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:12.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bfd5d6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf814fd60 con 0x7f6bf814f990 2023-03-03T20:01:12.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bfd5d6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bf8153340 con 0x7f6bf806be60 2023-03-03T20:01:12.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bfd5d6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf80c29c0 con 0x7f6bf8152ed0 2023-03-03T20:01:12.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bf77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bf806be60 0x7f6bf80ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:12.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bf77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bf806be60 0x7f6bf80ab5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44146/0 (socket says 172.21.15.7:44146) 2023-03-03T20:01:12.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bf77fe700 1 -- 172.21.15.7:0/1367334252 learned_addr learned my addr 172.21.15.7:0/1367334252 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:01:12.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bf67fc700 1 --2- 172.21.15.7:0/1367334252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 0x7f6bf806b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:12.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bf77fe700 1 -- 172.21.15.7:0/1367334252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 msgr2=0x7f6bf806b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:01:12.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bf77fe700 1 --2- 172.21.15.7:0/1367334252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 0x7f6bf806b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:12.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bf77fe700 1 -- 172.21.15.7:0/1367334252 >> v1:172.21.15.143:6789/0 conn(0x7f6bf814f990 legacy=0x7f6bf81562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:12.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.285+0000 7f6bf77fe700 1 -- 172.21.15.7:0/1367334252 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf80c2400 con 0x7f6bf806be60 2023-03-03T20:01:12.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bf77fe700 1 --2- 172.21.15.7:0/1367334252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 0x7f6bf80ab5e0 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009e50 tx=0x7f6bf000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848c400d9553bfe8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:01:12.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bf5ffb700 1 -- 172.21.15.7:0/1367334252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf000e070 con 0x7f6bf806be60 2023-03-03T20:01:12.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bf5ffb700 1 -- 172.21.15.7:0/1367334252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf000b9c0 con 0x7f6bf806be60 2023-03-03T20:01:12.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bf5ffb700 1 -- 172.21.15.7:0/1367334252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf000bd70 con 0x7f6bf806be60 2023-03-03T20:01:12.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/1367334252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 msgr2=0x7f6bf80ab5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:12.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bfd5d6700 1 --2- 172.21.15.7:0/1367334252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 0x7f6bf80ab5e0 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009e50 tx=0x7f6bf000b330 comp rx=0 tx=0).stop 2023-03-03T20:01:12.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/1367334252 shutdown_connections 2023-03-03T20:01:12.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bfd5d6700 1 --2- 172.21.15.7:0/1367334252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 0x7f6bf80ab5e0 unknown :-1 s=CLOSED pgs=3676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:12.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bfd5d6700 1 --2- 172.21.15.7:0/1367334252 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 0x7f6bf806b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:12.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/1367334252 >> 172.21.15.7:0/1367334252 conn(0x7f6bf80c62e0 msgr2=0x7f6bf80c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:01:12.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/1367334252 shutdown_connections 2023-03-03T20:01:12.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.286+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/1367334252 wait complete. 2023-03-03T20:01:12.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.288+0000 7f6bfd5d6700 1 Processor -- start 2023-03-03T20:01:12.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.288+0000 7f6bfd5d6700 1 -- start start 2023-03-03T20:01:12.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.288+0000 7f6bfd5d6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 0x7f6bf81513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:12.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bfd5d6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf814f990 0x7f6bf8147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:12.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bfd5d6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 0x7f6bf8147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:12.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bfd5d6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf8146530 con 0x7f6bf806be60 2023-03-03T20:01:12.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bfd5d6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf8154420 con 0x7f6bf814f990 2023-03-03T20:01:12.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bfd5d6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf8153850 con 0x7f6bf8152ed0 2023-03-03T20:01:12.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 0x7f6bf81513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:12.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 0x7f6bf81513d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44148/0 (socket says 172.21.15.7:44148) 2023-03-03T20:01:12.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf6ffd700 1 -- 172.21.15.7:0/2738384033 learned_addr learned my addr 172.21.15.7:0/2738384033 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:01:12.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf77fe700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 0x7f6bf8147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:12.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf67fc700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf814f990 0x7f6bf8147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:12.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf6ffd700 1 -- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 msgr2=0x7f6bf8147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:12.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf6ffd700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 0x7f6bf8147fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:12.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf6ffd700 1 -- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf814f990 msgr2=0x7f6bf8147aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:12.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf6ffd700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf814f990 0x7f6bf8147aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:12.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf6ffd700 1 -- 172.21.15.7:0/2738384033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf0009b30 con 0x7f6bf806be60 2023-03-03T20:01:12.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf77fe700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 0x7f6bf8147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:01:12.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf67fc700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf814f990 0x7f6bf8147aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:01:12.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bf6ffd700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 0x7f6bf81513d0 secure :-1 s=READY pgs=3677 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00c450 tx=0x7f6bec00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:01:12.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bdf7fe700 1 -- 172.21.15.7:0/2738384033 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec0052d0 con 0x7f6bf806be60 2023-03-03T20:01:12.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bdf7fe700 1 -- 172.21.15.7:0/2738384033 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bec005430 con 0x7f6bf806be60 2023-03-03T20:01:12.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/2738384033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf81523f0 con 0x7f6bf806be60 2023-03-03T20:01:12.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bdf7fe700 1 -- 172.21.15.7:0/2738384033 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bec0038c0 con 0x7f6bf806be60 2023-03-03T20:01:12.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.289+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/2738384033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf80c26e0 con 0x7f6bf806be60 2023-03-03T20:01:12.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.290+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/2738384033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf81526d0 con 0x7f6bf806be60 2023-03-03T20:01:12.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.291+0000 7f6bdf7fe700 1 -- 172.21.15.7:0/2738384033 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6bec019030 con 0x7f6bf806be60 2023-03-03T20:01:12.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.291+0000 7f6bdf7fe700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6be40682f0 0x7f6be406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:12.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.292+0000 7f6bdf7fe700 1 -- 172.21.15.7:0/2738384033 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6bec085340 con 0x7f6bf806be60 2023-03-03T20:01:12.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.292+0000 7f6bf67fc700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6be40682f0 0x7f6be406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:12.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.292+0000 7f6bf67fc700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6be40682f0 0x7f6be406a7a0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009a00 tx=0x7f6be8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:01:12.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.296+0000 7f6bdf7fe700 1 -- 172.21.15.7:0/2738384033 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6bec0516e0 con 0x7f6bf806be60 2023-03-03T20:01:12.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.535+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/2738384033 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bf8074bb0 con 0x7f6bf806be60 2023-03-03T20:01:12.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.535+0000 7f6bdf7fe700 1 -- 172.21.15.7:0/2738384033 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6bec054d00 con 0x7f6bf806be60 2023-03-03T20:01:12.536 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:01:12.536 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:01:12.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6be40682f0 msgr2=0x7f6be406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:12.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6be40682f0 0x7f6be406a7a0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009a00 tx=0x7f6be8008040 comp rx=0 tx=0).stop 2023-03-03T20:01:12.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/2738384033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 msgr2=0x7f6bf81513d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:12.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 0x7f6bf81513d0 secure :-1 s=READY pgs=3677 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00c450 tx=0x7f6bec00c810 comp rx=0 tx=0).stop 2023-03-03T20:01:12.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/2738384033 shutdown_connections 2023-03-03T20:01:12.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6bf8152ed0 0x7f6bf8147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:12.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6be40682f0 0x7f6be406a7a0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:12.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bf806be60 0x7f6bf81513d0 unknown :-1 s=CLOSED pgs=3677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:12.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 --2- 172.21.15.7:0/2738384033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6bf814f990 0x7f6bf8147aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:12.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/2738384033 >> 172.21.15.7:0/2738384033 conn(0x7f6bf80c62e0 msgr2=0x7f6bf814cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:01:12.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/2738384033 shutdown_connections 2023-03-03T20:01:12.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:12.537+0000 7f6bfd5d6700 1 -- 172.21.15.7:0/2738384033 wait complete. 2023-03-03T20:01:12.540 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:01:12.550 INFO:tasks.mds_thrash.fs.[a]:waiting for 46 secs before thrashing 2023-03-03T20:01:23.859 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:01:23.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.997+0000 7f5580532700 1 Processor -- start 2023-03-03T20:01:23.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.997+0000 7f5580532700 1 -- start start 2023-03-03T20:01:23.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.998+0000 7f5580532700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5578155410 0x7f55780bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:23.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.998+0000 7f5580532700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f55780c5b00 0x7f55780bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:24.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.998+0000 7f5580532700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55780c08d0 con 0x7f5578151e80 2023-03-03T20:01:24.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.998+0000 7f5580532700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5578152250 con 0x7f5578155410 2023-03-03T20:01:24.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.998+0000 7f5580532700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55780c0fb0 con 0x7f55780c5b00 2023-03-03T20:01:24.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557dacd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5578155410 0x7f55780bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:24.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557e2ce700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5578151e80 0x7f55780c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38144/0 (socket says 172.21.15.7:38144) 2023-03-03T20:01:24.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557e2ce700 1 -- 172.21.15.7:0/4020077206 learned_addr learned my addr 172.21.15.7:0/4020077206 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:01:24.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557eacf700 1 --2- 172.21.15.7:0/4020077206 >> v2:172.21.15.143:3300/0 conn(0x7f55780c5b00 0x7f55780bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:24.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557dacd700 1 -- 172.21.15.7:0/4020077206 >> v2:172.21.15.143:3300/0 conn(0x7f55780c5b00 msgr2=0x7f55780bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:01:24.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557dacd700 1 --2- 172.21.15.7:0/4020077206 >> v2:172.21.15.143:3300/0 conn(0x7f55780c5b00 0x7f55780bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557dacd700 1 -- 172.21.15.7:0/4020077206 >> v1:172.21.15.7:6789/0 conn(0x7f5578151e80 legacy=0x7f55780c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:24.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557d2cc700 1 -- 172.21.15.7:0/4020077206 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266699348 0 0) 0x7f55780c08d0 con 0x7f5578151e80 2023-03-03T20:01:24.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557dacd700 1 -- 172.21.15.7:0/4020077206 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5578154930 con 0x7f5578155410 2023-03-03T20:01:24.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557dacd700 1 --2- 172.21.15.7:0/4020077206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 0x7f55780bf8e0 secure :-1 s=READY pgs=3679 cs=0 l=1 rev1=1 crypto rx=0x7f557806bc30 tx=0x7f5568009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1a6c37c306c167 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:01:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557d2cc700 1 -- 172.21.15.7:0/4020077206 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568011830 con 0x7f5578155410 2023-03-03T20:01:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557d2cc700 1 -- 172.21.15.7:0/4020077206 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5568011990 con 0x7f5578155410 2023-03-03T20:01:24.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:23.999+0000 7f557d2cc700 1 -- 172.21.15.7:0/4020077206 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568011d20 con 0x7f5578155410 2023-03-03T20:01:24.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.000+0000 7f5580532700 1 -- 172.21.15.7:0/4020077206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 msgr2=0x7f55780bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:24.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.000+0000 7f5580532700 1 --2- 172.21.15.7:0/4020077206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 0x7f55780bf8e0 secure :-1 s=READY pgs=3679 cs=0 l=1 rev1=1 crypto rx=0x7f557806bc30 tx=0x7f5568009b20 comp rx=0 tx=0).stop 2023-03-03T20:01:24.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.000+0000 7f5580532700 1 -- 172.21.15.7:0/4020077206 shutdown_connections 2023-03-03T20:01:24.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.000+0000 7f5580532700 1 --2- 172.21.15.7:0/4020077206 >> v2:172.21.15.143:3300/0 conn(0x7f55780c5b00 0x7f55780bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.000+0000 7f5580532700 1 --2- 172.21.15.7:0/4020077206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 0x7f55780bf8e0 unknown :-1 s=CLOSED pgs=3679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.000+0000 7f5580532700 1 -- 172.21.15.7:0/4020077206 >> 172.21.15.7:0/4020077206 conn(0x7f5578071630 msgr2=0x7f557806b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:01:24.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.000+0000 7f5580532700 1 -- 172.21.15.7:0/4020077206 shutdown_connections 2023-03-03T20:01:24.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.000+0000 7f5580532700 1 -- 172.21.15.7:0/4020077206 wait complete. 2023-03-03T20:01:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.001+0000 7f5580532700 1 Processor -- start 2023-03-03T20:01:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.001+0000 7f5580532700 1 -- start start 2023-03-03T20:01:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f5580532700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55780c5b00 0x7f557814dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f5580532700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5578151e80 0x7f55781585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:24.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f5580532700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 0x7f55780c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:24.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f5580532700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5578157fd0 con 0x7f5578155410 2023-03-03T20:01:24.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f5580532700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55780c9fe0 con 0x7f55780c5b00 2023-03-03T20:01:24.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f5580532700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55780c9a20 con 0x7f5578151e80 2023-03-03T20:01:24.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557eacf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 0x7f55780c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:24.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557eacf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 0x7f55780c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44174/0 (socket says 172.21.15.7:44174) 2023-03-03T20:01:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557eacf700 1 -- 172.21.15.7:0/427997171 learned_addr learned my addr 172.21.15.7:0/427997171 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:01:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557e2ce700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55780c5b00 0x7f557814dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557dacd700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5578151e80 0x7f55781585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557eacf700 1 -- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5578151e80 msgr2=0x7f55781585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557eacf700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5578151e80 0x7f55781585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557eacf700 1 -- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55780c5b00 msgr2=0x7f557814dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:24.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557eacf700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55780c5b00 0x7f557814dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557eacf700 1 -- 172.21.15.7:0/427997171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55680094a0 con 0x7f5578155410 2023-03-03T20:01:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557dacd700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5578151e80 0x7f55781585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:01:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.002+0000 7f557eacf700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 0x7f55780c8a60 secure :-1 s=READY pgs=3680 cs=0 l=1 rev1=1 crypto rx=0x7f5570007e40 tx=0x7f557000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:01:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.003+0000 7f556effd700 1 -- 172.21.15.7:0/427997171 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557000ede0 con 0x7f5578155410 2023-03-03T20:01:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.003+0000 7f556effd700 1 -- 172.21.15.7:0/427997171 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5570014420 con 0x7f5578155410 2023-03-03T20:01:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.003+0000 7f5580532700 1 -- 172.21.15.7:0/427997171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5578154ef0 con 0x7f5578155410 2023-03-03T20:01:24.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.003+0000 7f556effd700 1 -- 172.21.15.7:0/427997171 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55700147d0 con 0x7f5578155410 2023-03-03T20:01:24.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.003+0000 7f5580532700 1 -- 172.21.15.7:0/427997171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5578154c10 con 0x7f5578155410 2023-03-03T20:01:24.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.004+0000 7f5580532700 1 -- 172.21.15.7:0/427997171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55780c5220 con 0x7f5578155410 2023-03-03T20:01:24.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.004+0000 7f556effd700 1 -- 172.21.15.7:0/427997171 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5570012050 con 0x7f5578155410 2023-03-03T20:01:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.005+0000 7f556effd700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f555c0682f0 0x7f555c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.005+0000 7f556effd700 1 -- 172.21.15.7:0/427997171 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5570086780 con 0x7f5578155410 2023-03-03T20:01:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.005+0000 7f557e2ce700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f555c0682f0 0x7f555c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.005+0000 7f557e2ce700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f555c0682f0 0x7f555c06a7a0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f5574007fd0 tx=0x7f5574007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:01:24.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.009+0000 7f556effd700 1 -- 172.21.15.7:0/427997171 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5570052b20 con 0x7f5578155410 2023-03-03T20:01:24.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.247+0000 7f5580532700 1 -- 172.21.15.7:0/427997171 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55780c0af0 con 0x7f5578155410 2023-03-03T20:01:24.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.247+0000 7f556effd700 1 -- 172.21.15.7:0/427997171 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5570056140 con 0x7f5578155410 2023-03-03T20:01:24.249 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:01:24.249 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:01:24.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 -- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f555c0682f0 msgr2=0x7f555c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:24.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f555c0682f0 0x7f555c06a7a0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f5574007fd0 tx=0x7f5574007de0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 -- 172.21.15.7:0/427997171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 msgr2=0x7f55780c8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:24.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 0x7f55780c8a60 secure :-1 s=READY pgs=3680 cs=0 l=1 rev1=1 crypto rx=0x7f5570007e40 tx=0x7f557000e3c0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 -- 172.21.15.7:0/427997171 shutdown_connections 2023-03-03T20:01:24.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f555c0682f0 0x7f555c06a7a0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5578155410 0x7f55780c8a60 unknown :-1 s=CLOSED pgs=3680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55780c5b00 0x7f557814dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 --2- 172.21.15.7:0/427997171 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5578151e80 0x7f55781585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:24.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 -- 172.21.15.7:0/427997171 >> 172.21.15.7:0/427997171 conn(0x7f5578071630 msgr2=0x7f55781513f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:01:24.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 -- 172.21.15.7:0/427997171 shutdown_connections 2023-03-03T20:01:24.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:24.249+0000 7f5580532700 1 -- 172.21.15.7:0/427997171 wait complete. 2023-03-03T20:01:24.254 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:01:24.262 INFO:tasks.mds_thrash.fs.[b]:waiting for 63 secs before thrashing 2023-03-03T20:01:24.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:01:24.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:01:54.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:01:54.912 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:01:58.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:01:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.690+0000 7fc8a058d700 1 Processor -- start 2023-03-03T20:01:58.691 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.690+0000 7fc8a058d700 1 -- start start 2023-03-03T20:01:58.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.691+0000 7fc8a058d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc8980caa30 0x7fc8980c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.691+0000 7fc8a058d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fc898154ff0 0x7fc8980cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.691+0000 7fc8a058d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8980c7310 con 0x7fc8980c00e0 2023-03-03T20:01:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.691+0000 7fc8a058d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc898155430 con 0x7fc8980caa30 2023-03-03T20:01:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.691+0000 7fc8a058d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8980bfd10 con 0x7fc898154ff0 2023-03-03T20:01:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.691+0000 7fc89db28700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc8980c00e0 0x7fc8980c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38194/0 (socket says 172.21.15.7:38194) 2023-03-03T20:01:58.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.691+0000 7fc89e329700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc8980caa30 0x7fc8980c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:58.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.691+0000 7fc89db28700 1 -- 172.21.15.7:0/3906744822 learned_addr learned my addr 172.21.15.7:0/3906744822 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:01:58.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.692+0000 7fc89eb2a700 1 --2- 172.21.15.7:0/3906744822 >> v2:172.21.15.143:3300/0 conn(0x7fc898154ff0 0x7fc8980cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:58.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.692+0000 7fc89e329700 1 -- 172.21.15.7:0/3906744822 >> v2:172.21.15.143:3300/0 conn(0x7fc898154ff0 msgr2=0x7fc8980cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:01:58.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.692+0000 7fc89e329700 1 --2- 172.21.15.7:0/3906744822 >> v2:172.21.15.143:3300/0 conn(0x7fc898154ff0 0x7fc8980cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:58.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.692+0000 7fc89e329700 1 -- 172.21.15.7:0/3906744822 >> v1:172.21.15.7:6789/0 conn(0x7fc8980c00e0 legacy=0x7fc8980c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:58.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.692+0000 7fc89e329700 1 -- 172.21.15.7:0/3906744822 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8980bfa30 con 0x7fc8980caa30 2023-03-03T20:01:58.694 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.692+0000 7fc89d327700 1 -- 172.21.15.7:0/3906744822 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466997133 0 0) 0x7fc8980c7310 con 0x7fc8980c00e0 2023-03-03T20:01:58.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.692+0000 7fc89e329700 1 --2- 172.21.15.7:0/3906744822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980caa30 0x7fc8980c5c60 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 crypto rx=0x7fc898071530 tx=0x7fc888009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c05c0b0eb4e52d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:01:58.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.692+0000 7fc89d327700 1 -- 172.21.15.7:0/3906744822 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc888011830 con 0x7fc8980caa30 2023-03-03T20:01:58.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.692+0000 7fc89d327700 1 -- 172.21.15.7:0/3906744822 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc888011990 con 0x7fc8980caa30 2023-03-03T20:01:58.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.692+0000 7fc89d327700 1 -- 172.21.15.7:0/3906744822 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc888011d20 con 0x7fc8980caa30 2023-03-03T20:01:58.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.693+0000 7fc8a058d700 1 -- 172.21.15.7:0/3906744822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980caa30 msgr2=0x7fc8980c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.693+0000 7fc8a058d700 1 --2- 172.21.15.7:0/3906744822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980caa30 0x7fc8980c5c60 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 crypto rx=0x7fc898071530 tx=0x7fc888009b20 comp rx=0 tx=0).stop 2023-03-03T20:01:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.693+0000 7fc8a058d700 1 -- 172.21.15.7:0/3906744822 shutdown_connections 2023-03-03T20:01:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.693+0000 7fc8a058d700 1 --2- 172.21.15.7:0/3906744822 >> v2:172.21.15.143:3300/0 conn(0x7fc898154ff0 0x7fc8980cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.693+0000 7fc8a058d700 1 --2- 172.21.15.7:0/3906744822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980caa30 0x7fc8980c5c60 unknown :-1 s=CLOSED pgs=3682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.693+0000 7fc8a058d700 1 -- 172.21.15.7:0/3906744822 >> 172.21.15.7:0/3906744822 conn(0x7fc898066e40 msgr2=0x7fc898071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:01:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.693+0000 7fc8a058d700 1 -- 172.21.15.7:0/3906744822 shutdown_connections 2023-03-03T20:01:58.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.693+0000 7fc8a058d700 1 -- 172.21.15.7:0/3906744822 wait complete. 2023-03-03T20:01:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.694+0000 7fc8a058d700 1 Processor -- start 2023-03-03T20:01:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.694+0000 7fc8a058d700 1 -- start start 2023-03-03T20:01:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc8a058d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980c00e0 0x7fc898157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc8a058d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8980caa30 0x7fc898157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc8a058d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc898154ff0 0x7fc89813ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc8a058d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981dec00 con 0x7fc8980c00e0 2023-03-03T20:01:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc8a058d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8980c29d0 con 0x7fc898154ff0 2023-03-03T20:01:58.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc8a058d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981dda40 con 0x7fc8980caa30 2023-03-03T20:01:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89e329700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980c00e0 0x7fc898157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89e329700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980c00e0 0x7fc898157230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44218/0 (socket says 172.21.15.7:44218) 2023-03-03T20:01:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89e329700 1 -- 172.21.15.7:0/4066955790 learned_addr learned my addr 172.21.15.7:0/4066955790 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:01:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89db28700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8980caa30 0x7fc898157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89eb2a700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc898154ff0 0x7fc89813ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89e329700 1 -- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8980caa30 msgr2=0x7fc898157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:58.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89e329700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8980caa30 0x7fc898157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89e329700 1 -- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc898154ff0 msgr2=0x7fc89813ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89e329700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc898154ff0 0x7fc89813ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89e329700 1 -- 172.21.15.7:0/4066955790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8880094a0 con 0x7fc8980c00e0 2023-03-03T20:01:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89db28700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8980caa30 0x7fc898157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:01:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89eb2a700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc898154ff0 0x7fc89813ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:01:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc89e329700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980c00e0 0x7fc898157230 secure :-1 s=READY pgs=3683 cs=0 l=1 rev1=1 crypto rx=0x7fc8880090b0 tx=0x7fc888004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:01:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.695+0000 7fc88effd700 1 -- 172.21.15.7:0/4066955790 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8880038a0 con 0x7fc8980c00e0 2023-03-03T20:01:58.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.696+0000 7fc88effd700 1 -- 172.21.15.7:0/4066955790 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88801f070 con 0x7fc8980c00e0 2023-03-03T20:01:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.696+0000 7fc8a058d700 1 -- 172.21.15.7:0/4066955790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8981547f0 con 0x7fc8980c00e0 2023-03-03T20:01:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.696+0000 7fc88effd700 1 -- 172.21.15.7:0/4066955790 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc888003c20 con 0x7fc8980c00e0 2023-03-03T20:01:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.696+0000 7fc8a058d700 1 -- 172.21.15.7:0/4066955790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc898154510 con 0x7fc8980c00e0 2023-03-03T20:01:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.697+0000 7fc8a058d700 1 -- 172.21.15.7:0/4066955790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc898154ad0 con 0x7fc8980c00e0 2023-03-03T20:01:58.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.697+0000 7fc88effd700 1 -- 172.21.15.7:0/4066955790 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc88800f050 con 0x7fc8980c00e0 2023-03-03T20:01:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.697+0000 7fc88effd700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 0x7fc87c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:01:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.697+0000 7fc88effd700 1 -- 172.21.15.7:0/4066955790 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc888021070 con 0x7fc8980c00e0 2023-03-03T20:01:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.698+0000 7fc89db28700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 0x7fc87c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:01:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.698+0000 7fc89db28700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 0x7fc87c06a7a0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7fc894001360 tx=0x7fc894009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:01:58.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.702+0000 7fc88effd700 1 -- 172.21.15.7:0/4066955790 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc888053ec0 con 0x7fc8980c00e0 2023-03-03T20:01:58.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.938+0000 7fc8a058d700 1 -- 172.21.15.7:0/4066955790 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc898074bb0 con 0x7fc8980c00e0 2023-03-03T20:01:58.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.938+0000 7fc88effd700 1 -- 172.21.15.7:0/4066955790 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc888015090 con 0x7fc8980c00e0 2023-03-03T20:01:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:01:58.940 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:01:58.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 -- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 msgr2=0x7fc87c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 0x7fc87c06a7a0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7fc894001360 tx=0x7fc894009040 comp rx=0 tx=0).stop 2023-03-03T20:01:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 -- 172.21.15.7:0/4066955790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980c00e0 msgr2=0x7fc898157230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:01:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980c00e0 0x7fc898157230 secure :-1 s=READY pgs=3683 cs=0 l=1 rev1=1 crypto rx=0x7fc8880090b0 tx=0x7fc888004750 comp rx=0 tx=0).stop 2023-03-03T20:01:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 -- 172.21.15.7:0/4066955790 shutdown_connections 2023-03-03T20:01:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc898154ff0 0x7fc89813ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:58.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc87c0682f0 0x7fc87c06a7a0 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8980c00e0 0x7fc898157230 unknown :-1 s=CLOSED pgs=3683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 --2- 172.21.15.7:0/4066955790 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc8980caa30 0x7fc898157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:01:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 -- 172.21.15.7:0/4066955790 >> 172.21.15.7:0/4066955790 conn(0x7fc898066e40 msgr2=0x7fc898151210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:01:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 -- 172.21.15.7:0/4066955790 shutdown_connections 2023-03-03T20:01:58.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:01:58.941+0000 7fc8a058d700 1 -- 172.21.15.7:0/4066955790 wait complete. 2023-03-03T20:01:58.944 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:01:58.953 INFO:tasks.mds_thrash.fs.[a]:waiting for 63 secs before thrashing 2023-03-03T20:02:24.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:02:24.987 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:02:27.263 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:02:27.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.402+0000 7fa6155f3700 1 Processor -- start 2023-03-03T20:02:27.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.402+0000 7fa6155f3700 1 -- start start 2023-03-03T20:02:27.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.402+0000 7fa6155f3700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa6100b0450 0x7fa6100c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:02:27.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.402+0000 7fa6155f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 0x7fa6100ca610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:02:27.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.402+0000 7fa6155f3700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6100c5a70 con 0x7fa6101519e0 2023-03-03T20:02:27.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa6155f3700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa610154d60 con 0x7fa6100b0450 2023-03-03T20:02:27.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa6155f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa610151db0 con 0x7fa6100c5e80 2023-03-03T20:02:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60f7fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 0x7fa6100ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:02:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60e7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa6100b0450 0x7fa6100c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:02:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60e7fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa6100b0450 0x7fa6100c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:40020/0 (socket says 172.21.15.7:40020) 2023-03-03T20:02:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60e7fc700 1 -- 172.21.15.7:0/4036328185 learned_addr learned my addr 172.21.15.7:0/4036328185 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:02:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60effd700 1 --1- 172.21.15.7:0/4036328185 >> v1:172.21.15.143:6789/0 conn(0x7fa6101519e0 0x7fa6100bf650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:41744/0 (socket says 172.21.15.7:41744) 2023-03-03T20:02:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60f7fe700 1 -- 172.21.15.7:0/4036328185 >> v2:172.21.15.143:3300/0 conn(0x7fa6100b0450 msgr2=0x7fa6100c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:02:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60f7fe700 1 --2- 172.21.15.7:0/4036328185 >> v2:172.21.15.143:3300/0 conn(0x7fa6100b0450 0x7fa6100c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:02:27.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60f7fe700 1 -- 172.21.15.7:0/4036328185 >> v1:172.21.15.143:6789/0 conn(0x7fa6101519e0 legacy=0x7fa6100bf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:02:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60f7fe700 1 -- 172.21.15.7:0/4036328185 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6100c7f20 con 0x7fa6100c5e80 2023-03-03T20:02:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60dffb700 1 -- 172.21.15.7:0/4036328185 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604994201 0 0) 0x7fa6100c5a70 con 0x7fa6101519e0 2023-03-03T20:02:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.403+0000 7fa60e7fc700 1 --2- 172.21.15.7:0/4036328185 >> v2:172.21.15.143:3300/0 conn(0x7fa6100b0450 0x7fa6100c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:02:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa60f7fe700 1 --2- 172.21.15.7:0/4036328185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 0x7fa6100ca610 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa608009d90 tx=0x7fa60800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acc0c6cc9576db01 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:02:27.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa60dffb700 1 -- 172.21.15.7:0/4036328185 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60800e070 con 0x7fa6100c5e80 2023-03-03T20:02:27.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa60dffb700 1 -- 172.21.15.7:0/4036328185 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60800bb20 con 0x7fa6100c5e80 2023-03-03T20:02:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa60dffb700 1 -- 172.21.15.7:0/4036328185 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa608012430 con 0x7fa6100c5e80 2023-03-03T20:02:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa6155f3700 1 -- 172.21.15.7:0/4036328185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 msgr2=0x7fa6100ca610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:02:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa6155f3700 1 --2- 172.21.15.7:0/4036328185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 0x7fa6100ca610 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa608009d90 tx=0x7fa60800b330 comp rx=0 tx=0).stop 2023-03-03T20:02:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa6155f3700 1 -- 172.21.15.7:0/4036328185 shutdown_connections 2023-03-03T20:02:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa6155f3700 1 --2- 172.21.15.7:0/4036328185 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 0x7fa6100ca610 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:02:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa6155f3700 1 --2- 172.21.15.7:0/4036328185 >> v2:172.21.15.143:3300/0 conn(0x7fa6100b0450 0x7fa6100c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:02:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa6155f3700 1 -- 172.21.15.7:0/4036328185 >> 172.21.15.7:0/4036328185 conn(0x7fa610066a20 msgr2=0x7fa6100714c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:02:27.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa6155f3700 1 -- 172.21.15.7:0/4036328185 shutdown_connections 2023-03-03T20:02:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.404+0000 7fa6155f3700 1 -- 172.21.15.7:0/4036328185 wait complete. 2023-03-03T20:02:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.406+0000 7fa6155f3700 1 Processor -- start 2023-03-03T20:02:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.406+0000 7fa6155f3700 1 -- start start 2023-03-03T20:02:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa6155f3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6100b0450 0x7fa610152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:02:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa6155f3700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 0x7fa6101ee290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:02:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa6155f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6101519e0 0x7fa61014a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:02:27.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa6155f3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6101ee7d0 con 0x7fa6100b0450 2023-03-03T20:02:27.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa6155f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa610150a20 con 0x7fa6101519e0 2023-03-03T20:02:27.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa6155f3700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6100c3f00 con 0x7fa6100c5e80 2023-03-03T20:02:27.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6100b0450 0x7fa610152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:02:27.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6100b0450 0x7fa610152690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44250/0 (socket says 172.21.15.7:44250) 2023-03-03T20:02:27.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60effd700 1 -- 172.21.15.7:0/3103119711 learned_addr learned my addr 172.21.15.7:0/3103119711 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:02:27.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60e7fc700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 0x7fa6101ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:02:27.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60f7fe700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6101519e0 0x7fa61014a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:02:27.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60effd700 1 -- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 msgr2=0x7fa6101ee290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:02:27.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60effd700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 0x7fa6101ee290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:02:27.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60effd700 1 -- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6101519e0 msgr2=0x7fa61014a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:02:27.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60effd700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6101519e0 0x7fa61014a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:02:27.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60effd700 1 -- 172.21.15.7:0/3103119711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa604009550 con 0x7fa6100b0450 2023-03-03T20:02:27.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa60effd700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6100b0450 0x7fa610152690 secure :-1 s=READY pgs=3684 cs=0 l=1 rev1=1 crypto rx=0x7fa60000c550 tx=0x7fa60000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:02:27.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa5f77fe700 1 -- 172.21.15.7:0/3103119711 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa600004c80 con 0x7fa6100b0450 2023-03-03T20:02:27.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa5f77fe700 1 -- 172.21.15.7:0/3103119711 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa600004e00 con 0x7fa6100b0450 2023-03-03T20:02:27.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.407+0000 7fa6155f3700 1 -- 172.21.15.7:0/3103119711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa608009a70 con 0x7fa6100b0450 2023-03-03T20:02:27.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.408+0000 7fa5f77fe700 1 -- 172.21.15.7:0/3103119711 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa600007310 con 0x7fa6100b0450 2023-03-03T20:02:27.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.408+0000 7fa6155f3700 1 -- 172.21.15.7:0/3103119711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa610155230 con 0x7fa6100b0450 2023-03-03T20:02:27.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.408+0000 7fa6155f3700 1 -- 172.21.15.7:0/3103119711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa610154f50 con 0x7fa6100b0450 2023-03-03T20:02:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.409+0000 7fa5f77fe700 1 -- 172.21.15.7:0/3103119711 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa60001a020 con 0x7fa6100b0450 2023-03-03T20:02:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.409+0000 7fa5f77fe700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5fc0682f0 0x7fa5fc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:02:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.409+0000 7fa5f77fe700 1 -- 172.21.15.7:0/3103119711 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa600085da0 con 0x7fa6100b0450 2023-03-03T20:02:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.410+0000 7fa60e7fc700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5fc0682f0 0x7fa5fc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:02:27.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.410+0000 7fa60e7fc700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5fc0682f0 0x7fa5fc06a7a0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fa604009cf0 tx=0x7fa604008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:02:27.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.414+0000 7fa5f77fe700 1 -- 172.21.15.7:0/3103119711 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa6000521f0 con 0x7fa6100b0450 2023-03-03T20:02:27.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.650+0000 7fa6155f3700 1 -- 172.21.15.7:0/3103119711 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6100c2fa0 con 0x7fa6100b0450 2023-03-03T20:02:27.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.651+0000 7fa5f77fe700 1 -- 172.21.15.7:0/3103119711 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa600055810 con 0x7fa6100b0450 2023-03-03T20:02:27.652 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:02:27.652 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:02:27.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 -- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5fc0682f0 msgr2=0x7fa5fc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:02:27.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5fc0682f0 0x7fa5fc06a7a0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fa604009cf0 tx=0x7fa604008040 comp rx=0 tx=0).stop 2023-03-03T20:02:27.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 -- 172.21.15.7:0/3103119711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6100b0450 msgr2=0x7fa610152690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:02:27.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6100b0450 0x7fa610152690 secure :-1 s=READY pgs=3684 cs=0 l=1 rev1=1 crypto rx=0x7fa60000c550 tx=0x7fa60000c910 comp rx=0 tx=0).stop 2023-03-03T20:02:27.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 -- 172.21.15.7:0/3103119711 shutdown_connections 2023-03-03T20:02:27.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6101519e0 0x7fa61014a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:02:27.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5fc0682f0 0x7fa5fc06a7a0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:02:27.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa6100b0450 0x7fa610152690 unknown :-1 s=CLOSED pgs=3684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:02:27.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 --2- 172.21.15.7:0/3103119711 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa6100c5e80 0x7fa6101ee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:02:27.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 -- 172.21.15.7:0/3103119711 >> 172.21.15.7:0/3103119711 conn(0x7fa610066a20 msgr2=0x7fa61014d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:02:27.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 -- 172.21.15.7:0/3103119711 shutdown_connections 2023-03-03T20:02:27.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:02:27.653+0000 7fa6155f3700 1 -- 172.21.15.7:0/3103119711 wait complete. 2023-03-03T20:02:27.656 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:02:27.666 INFO:tasks.mds_thrash.fs.[b]:waiting for 114 secs before thrashing 2023-03-03T20:02:55.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:02:55.068 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:03:01.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:03:02.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.093+0000 7ff9bf0e9700 1 Processor -- start 2023-03-03T20:03:02.094 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.093+0000 7ff9bf0e9700 1 -- start start 2023-03-03T20:03:02.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.094+0000 7ff9bf0e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8151ab0 0x7ff9b80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:02.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.094+0000 7ff9bf0e9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9b8073c30 0x7ff9b806bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:02.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.094+0000 7ff9bf0e9700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff9b806c3e0 0x7ff9b8157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:02.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.094+0000 7ff9bf0e9700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b806d780 con 0x7ff9b8073c30 2023-03-03T20:03:02.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.094+0000 7ff9bf0e9700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b806cb70 con 0x7ff9b806c3e0 2023-03-03T20:03:02.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.094+0000 7ff9bf0e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b806ce50 con 0x7ff9b8151ab0 2023-03-03T20:03:02.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.094+0000 7ff9b7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9b8073c30 0x7ff9b806bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:02.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.094+0000 7ff9b7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff9b8073c30 0x7ff9b806bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44286/0 (socket says 172.21.15.7:44286) 2023-03-03T20:03:02.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.094+0000 7ff9b7fff700 1 -- 172.21.15.7:0/1212987927 learned_addr learned my addr 172.21.15.7:0/1212987927 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:03:02.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9bce85700 1 --2- 172.21.15.7:0/1212987927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8151ab0 0x7ff9b80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:02.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9bd686700 1 --2- 172.21.15.7:0/1212987927 >> v2:172.21.15.143:3300/0 conn(0x7ff9b806c3e0 0x7ff9b8157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:02.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9b7fff700 1 -- 172.21.15.7:0/1212987927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8151ab0 msgr2=0x7ff9b80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:03:02.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9b7fff700 1 --2- 172.21.15.7:0/1212987927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8151ab0 0x7ff9b80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9b7fff700 1 -- 172.21.15.7:0/1212987927 >> v2:172.21.15.143:3300/0 conn(0x7ff9b806c3e0 msgr2=0x7ff9b8157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:03:02.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9b7fff700 1 --2- 172.21.15.7:0/1212987927 >> v2:172.21.15.143:3300/0 conn(0x7ff9b806c3e0 0x7ff9b8157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9b7fff700 1 -- 172.21.15.7:0/1212987927 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b806d130 con 0x7ff9b8073c30 2023-03-03T20:03:02.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9b7fff700 1 --2- 172.21.15.7:0/1212987927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 0x7ff9b806bea0 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80c3db0 tx=0x7ff9a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6a44c84bec34f6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:03:02.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9b77fe700 1 -- 172.21.15.7:0/1212987927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a400b870 con 0x7ff9b8073c30 2023-03-03T20:03:02.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9b77fe700 1 -- 172.21.15.7:0/1212987927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a400b9d0 con 0x7ff9b8073c30 2023-03-03T20:03:02.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.095+0000 7ff9b77fe700 1 -- 172.21.15.7:0/1212987927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a400bd60 con 0x7ff9b8073c30 2023-03-03T20:03:02.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.096+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/1212987927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 msgr2=0x7ff9b806bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:02.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.096+0000 7ff9bf0e9700 1 --2- 172.21.15.7:0/1212987927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 0x7ff9b806bea0 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80c3db0 tx=0x7ff9a4009bd0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.096+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/1212987927 shutdown_connections 2023-03-03T20:03:02.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.096+0000 7ff9bf0e9700 1 --2- 172.21.15.7:0/1212987927 >> v2:172.21.15.143:3300/0 conn(0x7ff9b806c3e0 0x7ff9b8157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.096+0000 7ff9bf0e9700 1 --2- 172.21.15.7:0/1212987927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8151ab0 0x7ff9b80736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.096+0000 7ff9bf0e9700 1 --2- 172.21.15.7:0/1212987927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8073c30 0x7ff9b806bea0 unknown :-1 s=CLOSED pgs=3685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.096+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/1212987927 >> 172.21.15.7:0/1212987927 conn(0x7ff9b80a94d0 msgr2=0x7ff9b80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:03:02.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.096+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/1212987927 shutdown_connections 2023-03-03T20:03:02.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.096+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/1212987927 wait complete. 2023-03-03T20:03:02.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.097+0000 7ff9bf0e9700 1 Processor -- start 2023-03-03T20:03:02.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.097+0000 7ff9bf0e9700 1 -- start start 2023-03-03T20:03:02.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bf0e9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b806c3e0 0x7ff9b80aa440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:02.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bf0e9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8073c30 0x7ff9b814c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:02.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bf0e9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b8151ab0 0x7ff9b80aac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:02.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bf0e9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b814ab80 con 0x7ff9b806c3e0 2023-03-03T20:03:02.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bf0e9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b81470e0 con 0x7ff9b8151ab0 2023-03-03T20:03:02.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bf0e9700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b814bb90 con 0x7ff9b8073c30 2023-03-03T20:03:02.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bce85700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b806c3e0 0x7ff9b80aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:02.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bce85700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b806c3e0 0x7ff9b80aa440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44290/0 (socket says 172.21.15.7:44290) 2023-03-03T20:03:02.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bce85700 1 -- 172.21.15.7:0/4006965067 learned_addr learned my addr 172.21.15.7:0/4006965067 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:03:02.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9b7fff700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8073c30 0x7ff9b814c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:02.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bd686700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b8151ab0 0x7ff9b80aac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:02.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bce85700 1 -- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8073c30 msgr2=0x7ff9b814c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:02.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bce85700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8073c30 0x7ff9b814c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bce85700 1 -- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b8151ab0 msgr2=0x7ff9b80aac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:02.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bce85700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b8151ab0 0x7ff9b80aac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bce85700 1 -- 172.21.15.7:0/4006965067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a4009550 con 0x7ff9b806c3e0 2023-03-03T20:03:02.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.098+0000 7ff9bce85700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b806c3e0 0x7ff9b80aa440 secure :-1 s=READY pgs=3686 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00d830 tx=0x7ff9ac00dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:03:02.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.099+0000 7ff9b57fa700 1 -- 172.21.15.7:0/4006965067 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac0125f0 con 0x7ff9b806c3e0 2023-03-03T20:03:02.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.099+0000 7ff9b57fa700 1 -- 172.21.15.7:0/4006965067 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ac012750 con 0x7ff9b806c3e0 2023-03-03T20:03:02.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.099+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/4006965067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b8151e80 con 0x7ff9b806c3e0 2023-03-03T20:03:02.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.099+0000 7ff9b57fa700 1 -- 172.21.15.7:0/4006965067 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac012b00 con 0x7ff9b806c3e0 2023-03-03T20:03:02.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.099+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/4006965067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b806d410 con 0x7ff9b806c3e0 2023-03-03T20:03:02.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.100+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/4006965067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b8152010 con 0x7ff9b806c3e0 2023-03-03T20:03:02.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.100+0000 7ff9b57fa700 1 -- 172.21.15.7:0/4006965067 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff9ac012c60 con 0x7ff9b806c3e0 2023-03-03T20:03:02.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.101+0000 7ff9b57fa700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 0x7ff99c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:02.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.101+0000 7ff9b57fa700 1 -- 172.21.15.7:0/4006965067 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff9ac086640 con 0x7ff9b806c3e0 2023-03-03T20:03:02.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.101+0000 7ff9b7fff700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 0x7ff99c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:02.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.101+0000 7ff9b7fff700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 0x7ff99c06a7a0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4000c00 tx=0x7ff9a4008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:03:02.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.105+0000 7ff9b57fa700 1 -- 172.21.15.7:0/4006965067 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff9ac0529e0 con 0x7ff9b806c3e0 2023-03-03T20:03:02.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.344+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/4006965067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9b8074bb0 con 0x7ff9b806c3e0 2023-03-03T20:03:02.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.344+0000 7ff9b57fa700 1 -- 172.21.15.7:0/4006965067 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff9ac056000 con 0x7ff9b806c3e0 2023-03-03T20:03:02.346 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:03:02.346 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:03:02.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 msgr2=0x7ff99c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:02.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 0x7ff99c06a7a0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4000c00 tx=0x7ff9a4008030 comp rx=0 tx=0).stop 2023-03-03T20:03:02.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/4006965067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b806c3e0 msgr2=0x7ff9b80aa440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:02.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b806c3e0 0x7ff9b80aa440 secure :-1 s=READY pgs=3686 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00d830 tx=0x7ff9ac00dbf0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/4006965067 shutdown_connections 2023-03-03T20:03:02.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9b8151ab0 0x7ff9b80aac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff99c0682f0 0x7ff99c06a7a0 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b806c3e0 0x7ff9b80aa440 unknown :-1 s=CLOSED pgs=3686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 --2- 172.21.15.7:0/4006965067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff9b8073c30 0x7ff9b814c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:02.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/4006965067 >> 172.21.15.7:0/4006965067 conn(0x7ff9b80a94d0 msgr2=0x7ff9b81558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:03:02.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/4006965067 shutdown_connections 2023-03-03T20:03:02.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:02.346+0000 7ff9bf0e9700 1 -- 172.21.15.7:0/4006965067 wait complete. 2023-03-03T20:03:02.351 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:03:02.359 INFO:tasks.mds_thrash.fs.[a]:waiting for 45 secs before thrashing 2023-03-03T20:03:25.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:03:25.143 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:03:47.360 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:03:47.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.498+0000 7fa709f30700 1 Processor -- start 2023-03-03T20:03:47.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.498+0000 7fa709f30700 1 -- start start 2023-03-03T20:03:47.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa709f30700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa704155410 0x7fa7040bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:47.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa709f30700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa7040c5b00 0x7fa7040bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:47.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa709f30700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7040c08d0 con 0x7fa704151e80 2023-03-03T20:03:47.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa709f30700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa704152250 con 0x7fa704155410 2023-03-03T20:03:47.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa709f30700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7040c0fb0 con 0x7fa7040c5b00 2023-03-03T20:03:47.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa702ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa704155410 0x7fa7040bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:47.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa7037fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa704151e80 0x7fa7040c6aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38310/0 (socket says 172.21.15.7:38310) 2023-03-03T20:03:47.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa7037fe700 1 -- 172.21.15.7:0/3594630639 learned_addr learned my addr 172.21.15.7:0/3594630639 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:03:47.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa703fff700 1 --2- 172.21.15.7:0/3594630639 >> v2:172.21.15.143:3300/0 conn(0x7fa7040c5b00 0x7fa7040bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:47.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa702ffd700 1 -- 172.21.15.7:0/3594630639 >> v2:172.21.15.143:3300/0 conn(0x7fa7040c5b00 msgr2=0x7fa7040bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:03:47.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa702ffd700 1 --2- 172.21.15.7:0/3594630639 >> v2:172.21.15.143:3300/0 conn(0x7fa7040c5b00 0x7fa7040bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:47.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa702ffd700 1 -- 172.21.15.7:0/3594630639 >> v1:172.21.15.7:6789/0 conn(0x7fa704151e80 legacy=0x7fa7040c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:47.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa702ffd700 1 -- 172.21.15.7:0/3594630639 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa704154930 con 0x7fa704155410 2023-03-03T20:03:47.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa7027fc700 1 -- 172.21.15.7:0/3594630639 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593137349 0 0) 0x7fa7040c08d0 con 0x7fa704151e80 2023-03-03T20:03:47.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.499+0000 7fa703fff700 1 --2- 172.21.15.7:0/3594630639 >> v2:172.21.15.143:3300/0 conn(0x7fa7040c5b00 0x7fa7040bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:03:47.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa702ffd700 1 --2- 172.21.15.7:0/3594630639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa704155410 0x7fa7040bf8e0 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 crypto rx=0x7fa70406bc30 tx=0x7fa6ec009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b69bffb09d053eb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:03:47.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa7027fc700 1 -- 172.21.15.7:0/3594630639 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec011830 con 0x7fa704155410 2023-03-03T20:03:47.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa7027fc700 1 -- 172.21.15.7:0/3594630639 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec011990 con 0x7fa704155410 2023-03-03T20:03:47.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa7027fc700 1 -- 172.21.15.7:0/3594630639 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec011d20 con 0x7fa704155410 2023-03-03T20:03:47.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa709f30700 1 -- 172.21.15.7:0/3594630639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa704155410 msgr2=0x7fa7040bf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:47.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa709f30700 1 --2- 172.21.15.7:0/3594630639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa704155410 0x7fa7040bf8e0 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 crypto rx=0x7fa70406bc30 tx=0x7fa6ec009b20 comp rx=0 tx=0).stop 2023-03-03T20:03:47.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa709f30700 1 -- 172.21.15.7:0/3594630639 shutdown_connections 2023-03-03T20:03:47.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa709f30700 1 --2- 172.21.15.7:0/3594630639 >> v2:172.21.15.143:3300/0 conn(0x7fa7040c5b00 0x7fa7040bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:47.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa709f30700 1 --2- 172.21.15.7:0/3594630639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa704155410 0x7fa7040bf8e0 unknown :-1 s=CLOSED pgs=3688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa709f30700 1 -- 172.21.15.7:0/3594630639 >> 172.21.15.7:0/3594630639 conn(0x7fa704071630 msgr2=0x7fa70406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:03:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa709f30700 1 -- 172.21.15.7:0/3594630639 shutdown_connections 2023-03-03T20:03:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.500+0000 7fa709f30700 1 -- 172.21.15.7:0/3594630639 wait complete. 2023-03-03T20:03:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.502+0000 7fa709f30700 1 Processor -- start 2023-03-03T20:03:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.502+0000 7fa709f30700 1 -- start start 2023-03-03T20:03:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa709f30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7040c5b00 0x7fa70414a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:47.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa709f30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa704151e80 0x7fa70414fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:47.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa709f30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa704155410 0x7fa7041e12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:47.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa709f30700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7041edfe0 con 0x7fa7040c5b00 2023-03-03T20:03:47.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa709f30700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7041ee510 con 0x7fa704151e80 2023-03-03T20:03:47.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa709f30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7041ee230 con 0x7fa704155410 2023-03-03T20:03:47.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa7037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7040c5b00 0x7fa70414a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:47.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa7037fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7040c5b00 0x7fa70414a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44336/0 (socket says 172.21.15.7:44336) 2023-03-03T20:03:47.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa7037fe700 1 -- 172.21.15.7:0/1856490855 learned_addr learned my addr 172.21.15.7:0/1856490855 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:03:47.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa703fff700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa704155410 0x7fa7041e12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:47.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.503+0000 7fa702ffd700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa704151e80 0x7fa70414fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:47.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa7037fe700 1 -- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa704155410 msgr2=0x7fa7041e12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:47.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa7037fe700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa704155410 0x7fa7041e12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:47.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa7037fe700 1 -- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa704151e80 msgr2=0x7fa70414fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:47.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa7037fe700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa704151e80 0x7fa70414fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:47.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa7037fe700 1 -- 172.21.15.7:0/1856490855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec0094a0 con 0x7fa7040c5b00 2023-03-03T20:03:47.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa702ffd700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa704151e80 0x7fa70414fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:03:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa703fff700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa704155410 0x7fa7041e12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:03:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa7037fe700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7040c5b00 0x7fa70414a1c0 secure :-1 s=READY pgs=3689 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400c750 tx=0x7fa6f400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:03:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa6fbfff700 1 -- 172.21.15.7:0/1856490855 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4004c80 con 0x7fa7040c5b00 2023-03-03T20:03:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa6fbfff700 1 -- 172.21.15.7:0/1856490855 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f4004e00 con 0x7fa7040c5b00 2023-03-03T20:03:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa709f30700 1 -- 172.21.15.7:0/1856490855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa704154ef0 con 0x7fa7040c5b00 2023-03-03T20:03:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa6fbfff700 1 -- 172.21.15.7:0/1856490855 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4007310 con 0x7fa7040c5b00 2023-03-03T20:03:47.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.504+0000 7fa709f30700 1 -- 172.21.15.7:0/1856490855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa704154c10 con 0x7fa7040c5b00 2023-03-03T20:03:47.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.505+0000 7fa709f30700 1 -- 172.21.15.7:0/1856490855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7040c5220 con 0x7fa7040c5b00 2023-03-03T20:03:47.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.505+0000 7fa6fbfff700 1 -- 172.21.15.7:0/1856490855 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa6f401a020 con 0x7fa7040c5b00 2023-03-03T20:03:47.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.506+0000 7fa6fbfff700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 0x7fa6e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:03:47.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.506+0000 7fa6fbfff700 1 -- 172.21.15.7:0/1856490855 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa6f4085dd0 con 0x7fa7040c5b00 2023-03-03T20:03:47.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.506+0000 7fa702ffd700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 0x7fa6e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:03:47.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.507+0000 7fa702ffd700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 0x7fa6e406a7a0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7fa70406bc30 tx=0x7fa6ec008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:03:47.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.510+0000 7fa6fbfff700 1 -- 172.21.15.7:0/1856490855 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa6f4052220 con 0x7fa7040c5b00 2023-03-03T20:03:47.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.751+0000 7fa709f30700 1 -- 172.21.15.7:0/1856490855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7040c95e0 con 0x7fa7040c5b00 2023-03-03T20:03:47.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.751+0000 7fa6fbfff700 1 -- 172.21.15.7:0/1856490855 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa6f4055840 con 0x7fa7040c5b00 2023-03-03T20:03:47.753 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:03:47.753 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:03:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 -- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 msgr2=0x7fa6e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 0x7fa6e406a7a0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7fa70406bc30 tx=0x7fa6ec008030 comp rx=0 tx=0).stop 2023-03-03T20:03:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 -- 172.21.15.7:0/1856490855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7040c5b00 msgr2=0x7fa70414a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:03:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7040c5b00 0x7fa70414a1c0 secure :-1 s=READY pgs=3689 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400c750 tx=0x7fa6f400cb10 comp rx=0 tx=0).stop 2023-03-03T20:03:47.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 -- 172.21.15.7:0/1856490855 shutdown_connections 2023-03-03T20:03:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa704155410 0x7fa7041e12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa6e40682f0 0x7fa6e406a7a0 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7040c5b00 0x7fa70414a1c0 unknown :-1 s=CLOSED pgs=3689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 --2- 172.21.15.7:0/1856490855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa704151e80 0x7fa70414fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:03:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 -- 172.21.15.7:0/1856490855 >> 172.21.15.7:0/1856490855 conn(0x7fa704071630 msgr2=0x7fa7040c3190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:03:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 -- 172.21.15.7:0/1856490855 shutdown_connections 2023-03-03T20:03:47.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:03:47.753+0000 7fa709f30700 1 -- 172.21.15.7:0/1856490855 wait complete. 2023-03-03T20:03:47.757 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:03:47.767 INFO:tasks.mds_thrash.fs.[a]:waiting for 92 secs before thrashing 2023-03-03T20:03:55.291 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:03:55.294 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:04:21.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:04:21.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.806+0000 7fca5a6f8700 1 Processor -- start 2023-03-03T20:04:21.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.806+0000 7fca5a6f8700 1 -- start start 2023-03-03T20:04:21.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca5a6f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca54151d40 0x7fca540736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:04:21.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca5a6f8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fca540c2930 0x7fca54157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:04:21.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca5a6f8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca54073c30 0x7fca54157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:04:21.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca5a6f8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca5406c2a0 con 0x7fca54073c30 2023-03-03T20:04:21.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca5a6f8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca5406d430 con 0x7fca540c2930 2023-03-03T20:04:21.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca5a6f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5406cb00 con 0x7fca54151d40 2023-03-03T20:04:21.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca58c95700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca54073c30 0x7fca54157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:04:21.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca58c95700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fca54073c30 0x7fca54157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44360/0 (socket says 172.21.15.7:44360) 2023-03-03T20:04:21.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca58c95700 1 -- 172.21.15.7:0/62455587 learned_addr learned my addr 172.21.15.7:0/62455587 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:04:21.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca537fe700 1 --2- 172.21.15.7:0/62455587 >> v2:172.21.15.143:3300/0 conn(0x7fca540c2930 0x7fca54157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:04:21.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca53fff700 1 --2- 172.21.15.7:0/62455587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca54151d40 0x7fca540736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:04:21.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca58c95700 1 -- 172.21.15.7:0/62455587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca54151d40 msgr2=0x7fca540736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:04:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca58c95700 1 --2- 172.21.15.7:0/62455587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca54151d40 0x7fca540736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca58c95700 1 -- 172.21.15.7:0/62455587 >> v2:172.21.15.143:3300/0 conn(0x7fca540c2930 msgr2=0x7fca54157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:04:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca58c95700 1 --2- 172.21.15.7:0/62455587 >> v2:172.21.15.143:3300/0 conn(0x7fca540c2930 0x7fca54157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca58c95700 1 -- 172.21.15.7:0/62455587 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5406cde0 con 0x7fca54073c30 2023-03-03T20:04:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca537fe700 1 --2- 172.21.15.7:0/62455587 >> v2:172.21.15.143:3300/0 conn(0x7fca540c2930 0x7fca54157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:04:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.807+0000 7fca58c95700 1 --2- 172.21.15.7:0/62455587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca54073c30 0x7fca54157fd0 secure :-1 s=READY pgs=3690 cs=0 l=1 rev1=1 crypto rx=0x7fca5406b820 tx=0x7fca4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb07c9e70cad806 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:04:21.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca52ffd700 1 -- 172.21.15.7:0/62455587 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c00e070 con 0x7fca54073c30 2023-03-03T20:04:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca52ffd700 1 -- 172.21.15.7:0/62455587 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4c00b9c0 con 0x7fca54073c30 2023-03-03T20:04:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca52ffd700 1 -- 172.21.15.7:0/62455587 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c00bd70 con 0x7fca54073c30 2023-03-03T20:04:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca5a6f8700 1 -- 172.21.15.7:0/62455587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca54073c30 msgr2=0x7fca54157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:04:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca5a6f8700 1 --2- 172.21.15.7:0/62455587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca54073c30 0x7fca54157fd0 secure :-1 s=READY pgs=3690 cs=0 l=1 rev1=1 crypto rx=0x7fca5406b820 tx=0x7fca4c00b330 comp rx=0 tx=0).stop 2023-03-03T20:04:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca5a6f8700 1 -- 172.21.15.7:0/62455587 shutdown_connections 2023-03-03T20:04:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca5a6f8700 1 --2- 172.21.15.7:0/62455587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca54073c30 0x7fca54157fd0 unknown :-1 s=CLOSED pgs=3690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca5a6f8700 1 --2- 172.21.15.7:0/62455587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca54151d40 0x7fca540736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:21.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca5a6f8700 1 --2- 172.21.15.7:0/62455587 >> v2:172.21.15.143:3300/0 conn(0x7fca540c2930 0x7fca54157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:21.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca5a6f8700 1 -- 172.21.15.7:0/62455587 >> 172.21.15.7:0/62455587 conn(0x7fca540a94d0 msgr2=0x7fca540a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:04:21.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca5a6f8700 1 -- 172.21.15.7:0/62455587 shutdown_connections 2023-03-03T20:04:21.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.808+0000 7fca5a6f8700 1 -- 172.21.15.7:0/62455587 wait complete. 2023-03-03T20:04:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.810+0000 7fca5a6f8700 1 Processor -- start 2023-03-03T20:04:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.810+0000 7fca5a6f8700 1 -- start start 2023-03-03T20:04:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca5a6f8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca54073c30 0x7fca541defc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:04:21.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca5a6f8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca540c2930 0x7fca541546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:04:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca5a6f8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca54151d40 0x7fca541523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:04:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca5a6f8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca541deb30 con 0x7fca540c2930 2023-03-03T20:04:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca5a6f8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5414e5a0 con 0x7fca54073c30 2023-03-03T20:04:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca5a6f8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5414e880 con 0x7fca54151d40 2023-03-03T20:04:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca537fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca540c2930 0x7fca541546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:04:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca537fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca540c2930 0x7fca541546c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44364/0 (socket says 172.21.15.7:44364) 2023-03-03T20:04:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca537fe700 1 -- 172.21.15.7:0/916555236 learned_addr learned my addr 172.21.15.7:0/916555236 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:04:21.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca58c95700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca54151d40 0x7fca541523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:04:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca53fff700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca54073c30 0x7fca541defc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:04:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca537fe700 1 -- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca54151d40 msgr2=0x7fca541523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:04:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca537fe700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca54151d40 0x7fca541523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca537fe700 1 -- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca54073c30 msgr2=0x7fca541defc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:04:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca537fe700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca54073c30 0x7fca541defc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca537fe700 1 -- 172.21.15.7:0/916555236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c009b30 con 0x7fca540c2930 2023-03-03T20:04:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.811+0000 7fca53fff700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca54073c30 0x7fca541defc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:04:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.812+0000 7fca537fe700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca540c2930 0x7fca541546c0 secure :-1 s=READY pgs=3691 cs=0 l=1 rev1=1 crypto rx=0x7fca4400e810 tx=0x7fca4400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:04:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.812+0000 7fca50ff9700 1 -- 172.21.15.7:0/916555236 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44013610 con 0x7fca540c2930 2023-03-03T20:04:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.812+0000 7fca50ff9700 1 -- 172.21.15.7:0/916555236 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca44013770 con 0x7fca540c2930 2023-03-03T20:04:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.812+0000 7fca5a6f8700 1 -- 172.21.15.7:0/916555236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5406bea0 con 0x7fca540c2930 2023-03-03T20:04:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.812+0000 7fca50ff9700 1 -- 172.21.15.7:0/916555236 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44013b20 con 0x7fca540c2930 2023-03-03T20:04:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.812+0000 7fca5a6f8700 1 -- 172.21.15.7:0/916555236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5406d0c0 con 0x7fca540c2930 2023-03-03T20:04:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.813+0000 7fca5a6f8700 1 -- 172.21.15.7:0/916555236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5406c030 con 0x7fca540c2930 2023-03-03T20:04:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.814+0000 7fca50ff9700 1 -- 172.21.15.7:0/916555236 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fca44023070 con 0x7fca540c2930 2023-03-03T20:04:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.814+0000 7fca50ff9700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca400682f0 0x7fca4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:04:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.814+0000 7fca50ff9700 1 -- 172.21.15.7:0/916555236 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fca440862c0 con 0x7fca540c2930 2023-03-03T20:04:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.815+0000 7fca53fff700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca400682f0 0x7fca4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:04:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.815+0000 7fca53fff700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca400682f0 0x7fca4006a7a0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7fca480079d0 tx=0x7fca48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:04:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:21.818+0000 7fca50ff9700 1 -- 172.21.15.7:0/916555236 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fca440525f0 con 0x7fca540c2930 2023-03-03T20:04:22.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.055+0000 7fca5a6f8700 1 -- 172.21.15.7:0/916555236 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca54074bb0 con 0x7fca540c2930 2023-03-03T20:04:22.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.056+0000 7fca50ff9700 1 -- 172.21.15.7:0/916555236 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fca44055c10 con 0x7fca540c2930 2023-03-03T20:04:22.056 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:04:22.057 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:04:22.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.057+0000 7fca5a6f8700 1 -- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca400682f0 msgr2=0x7fca4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:04:22.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.057+0000 7fca5a6f8700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca400682f0 0x7fca4006a7a0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7fca480079d0 tx=0x7fca48008040 comp rx=0 tx=0).stop 2023-03-03T20:04:22.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.057+0000 7fca5a6f8700 1 -- 172.21.15.7:0/916555236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca540c2930 msgr2=0x7fca541546c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:04:22.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.057+0000 7fca5a6f8700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca540c2930 0x7fca541546c0 secure :-1 s=READY pgs=3691 cs=0 l=1 rev1=1 crypto rx=0x7fca4400e810 tx=0x7fca4400eb20 comp rx=0 tx=0).stop 2023-03-03T20:04:22.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.057+0000 7fca5a6f8700 1 -- 172.21.15.7:0/916555236 shutdown_connections 2023-03-03T20:04:22.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.057+0000 7fca5a6f8700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fca54151d40 0x7fca541523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:22.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.057+0000 7fca5a6f8700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca54073c30 0x7fca541defc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:22.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.057+0000 7fca5a6f8700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fca400682f0 0x7fca4006a7a0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:22.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.057+0000 7fca5a6f8700 1 --2- 172.21.15.7:0/916555236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fca540c2930 0x7fca541546c0 unknown :-1 s=CLOSED pgs=3691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:04:22.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.057+0000 7fca5a6f8700 1 -- 172.21.15.7:0/916555236 >> 172.21.15.7:0/916555236 conn(0x7fca540a94d0 msgr2=0x7fca540a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:04:22.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.058+0000 7fca5a6f8700 1 -- 172.21.15.7:0/916555236 shutdown_connections 2023-03-03T20:04:22.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:04:22.058+0000 7fca5a6f8700 1 -- 172.21.15.7:0/916555236 wait complete. 2023-03-03T20:04:22.060 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:04:22.070 INFO:tasks.mds_thrash.fs.[b]:waiting for 40 secs before thrashing 2023-03-03T20:04:25.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:04:25.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:04:55.493 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:04:55.496 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:05:02.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:05:02.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.209+0000 7f78a9c0d700 1 Processor -- start 2023-03-03T20:05:02.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.209+0000 7f78a9c0d700 1 -- start start 2023-03-03T20:05:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a9c0d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a40c6010 0x7f78a40c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a9c0d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f78a40c4fc0 0x7f78a406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a9c0d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a414ebe0 con 0x7f78a40c17f0 2023-03-03T20:05:02.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a9c0d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a4152100 con 0x7f78a40c4fc0 2023-03-03T20:05:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a9c0d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a40c2e70 con 0x7f78a40c6010 2023-03-03T20:05:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a2ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f78a40c17f0 0x7f78a406d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38384/0 (socket says 172.21.15.7:38384) 2023-03-03T20:05:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a2ffd700 1 -- 172.21.15.7:0/4065430392 learned_addr learned my addr 172.21.15.7:0/4065430392 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:05:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a3fff700 1 --2- 172.21.15.7:0/4065430392 >> v2:172.21.15.143:3300/0 conn(0x7f78a40c4fc0 0x7f78a406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a37fe700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a40c6010 0x7f78a40c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476254447 0 0) 0x7f78a414ebe0 con 0x7f78a40c17f0 2023-03-03T20:05:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.210+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f788c0035b0 con 0x7f78a40c17f0 2023-03-03T20:05:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1745663434 0 0) 0x7f788c0035b0 con 0x7f78a40c17f0 2023-03-03T20:05:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a40c6010 msgr2=0x7f78a40c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:05:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a40c6010 0x7f78a40c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 >> v2:172.21.15.143:3300/0 conn(0x7f78a40c4fc0 msgr2=0x7f78a406ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:05:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 --2- 172.21.15.7:0/4065430392 >> v2:172.21.15.143:3300/0 conn(0x7f78a40c4fc0 0x7f78a406ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a40aa680 con 0x7f78a40c17f0 2023-03-03T20:05:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f7898001670 con 0x7f78a40c17f0 2023-03-03T20:05:02.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a3fff700 1 --2- 172.21.15.7:0/4065430392 >> v2:172.21.15.143:3300/0 conn(0x7f78a40c4fc0 0x7f78a406ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:05:02.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 >> v1:172.21.15.7:6789/0 conn(0x7f78a40c17f0 legacy=0x7f78a406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:02.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c003a70 0x7f788c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:02.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c004850 0x7f788c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:02.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f788c005200 0x7f788c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:02.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a4152100 con 0x7f788c004850 2023-03-03T20:05:02.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a40c2e70 con 0x7f788c003a70 2023-03-03T20:05:02.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c00bfd0 con 0x7f788c005200 2023-03-03T20:05:02.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a37fe700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a40c6010 0x7f78a40c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:05:02.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a2ffd700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c004850 0x7f788c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:02.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a3fff700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f788c005200 0x7f788c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:02.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a37fe700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c003a70 0x7f788c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:02.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a2ffd700 1 -- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f788c005200 msgr2=0x7f788c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:05:02.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a2ffd700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f788c005200 0x7f788c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a2ffd700 1 -- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c003a70 msgr2=0x7f788c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:05:02.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a2ffd700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c003a70 0x7f788c003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a2ffd700 1 -- 172.21.15.7:0/4065430392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c0035b0 con 0x7f788c004850 2023-03-03T20:05:02.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a37fe700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c003a70 0x7f788c003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:05:02.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a3fff700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f788c005200 0x7f788c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:05:02.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a2ffd700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c004850 0x7f788c004cc0 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 crypto rx=0x7f789800c2f0 tx=0x7f788c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33fe9ab650ec112 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:05:02.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789800ce30 con 0x7f788c004850 2023-03-03T20:05:02.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7898005960 con 0x7f788c004850 2023-03-03T20:05:02.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.211+0000 7f78a27fc700 1 -- 172.21.15.7:0/4065430392 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898005d10 con 0x7f788c004850 2023-03-03T20:05:02.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.212+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4065430392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c004850 msgr2=0x7f788c004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:02.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.212+0000 7f78a9c0d700 1 --2- 172.21.15.7:0/4065430392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f788c004850 0x7f788c004cc0 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 crypto rx=0x7f789800c2f0 tx=0x7f788c005880 comp rx=0 tx=0).stop 2023-03-03T20:05:02.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.212+0000 7f78a3fff700 1 -- 172.21.15.7:0/4065430392 reap_dead start 2023-03-03T20:05:02.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.212+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4065430392 shutdown_connections 2023-03-03T20:05:02.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.212+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4065430392 >> 172.21.15.7:0/4065430392 conn(0x7f78a40a9430 msgr2=0x7f78a40c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:05:02.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.212+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4065430392 shutdown_connections 2023-03-03T20:05:02.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.212+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4065430392 wait complete. 2023-03-03T20:05:02.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.213+0000 7f78a9c0d700 1 Processor -- start 2023-03-03T20:05:02.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.213+0000 7f78a9c0d700 1 -- start start 2023-03-03T20:05:02.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a9c0d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c004850 0x7f78a406a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:02.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a9c0d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78a41edba0 0x7f78a41ea250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:02.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a9c0d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a406a7f0 0x7f78a41ea790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:02.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a9c0d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a41db650 con 0x7f78a41edba0 2023-03-03T20:05:02.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a9c0d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a41ee0a0 con 0x7f788c004850 2023-03-03T20:05:02.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a9c0d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a41526a0 con 0x7f78a406a7f0 2023-03-03T20:05:02.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78a41edba0 0x7f78a41ea250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:02.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78a41edba0 0x7f78a41ea250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44418/0 (socket says 172.21.15.7:44418) 2023-03-03T20:05:02.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a2ffd700 1 -- 172.21.15.7:0/4025587809 learned_addr learned my addr 172.21.15.7:0/4025587809 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:05:02.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a3fff700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a406a7f0 0x7f78a41ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:02.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.214+0000 7f78a37fe700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c004850 0x7f78a406a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:02.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f78a2ffd700 1 -- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a406a7f0 msgr2=0x7f78a41ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:02.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f78a2ffd700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a406a7f0 0x7f78a41ea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f78a2ffd700 1 -- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c004850 msgr2=0x7f78a406a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:02.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f78a2ffd700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c004850 0x7f78a406a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f78a2ffd700 1 -- 172.21.15.7:0/4025587809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a40c2b00 con 0x7f78a41edba0 2023-03-03T20:05:02.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f78a2ffd700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78a41edba0 0x7f78a41ea250 secure :-1 s=READY pgs=3694 cs=0 l=1 rev1=1 crypto rx=0x7f7898015040 tx=0x7f78a4066cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:05:02.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f788bfff700 1 -- 172.21.15.7:0/4025587809 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789800cea0 con 0x7f78a41edba0 2023-03-03T20:05:02.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f788bfff700 1 -- 172.21.15.7:0/4025587809 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7898004a80 con 0x7f78a41edba0 2023-03-03T20:05:02.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f788bfff700 1 -- 172.21.15.7:0/4025587809 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898004e30 con 0x7f78a41edba0 2023-03-03T20:05:02.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4025587809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a40aa960 con 0x7f78a41edba0 2023-03-03T20:05:02.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.215+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4025587809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a40c5420 con 0x7f78a41edba0 2023-03-03T20:05:02.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.216+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4025587809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a4074bb0 con 0x7f78a41edba0 2023-03-03T20:05:02.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.216+0000 7f788bfff700 1 -- 172.21.15.7:0/4025587809 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7898010070 con 0x7f78a41edba0 2023-03-03T20:05:02.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.217+0000 7f788bfff700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78900682f0 0x7f789006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:02.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.217+0000 7f788bfff700 1 -- 172.21.15.7:0/4025587809 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f78980873b0 con 0x7f78a41edba0 2023-03-03T20:05:02.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.217+0000 7f78a37fe700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78900682f0 0x7f789006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:02.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.217+0000 7f78a37fe700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78900682f0 0x7f789006a7a0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f7894009470 tx=0x7f7894008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:05:02.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.221+0000 7f788bfff700 1 -- 172.21.15.7:0/4025587809 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7898053660 con 0x7f78a41edba0 2023-03-03T20:05:02.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.461+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4025587809 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78a40c15d0 con 0x7f78a41edba0 2023-03-03T20:05:02.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.461+0000 7f788bfff700 1 -- 172.21.15.7:0/4025587809 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7898007da0 con 0x7f78a41edba0 2023-03-03T20:05:02.462 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:05:02.463 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:05:02.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.463+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78900682f0 msgr2=0x7f789006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:02.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.463+0000 7f78a9c0d700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78900682f0 0x7f789006a7a0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f7894009470 tx=0x7f7894008040 comp rx=0 tx=0).stop 2023-03-03T20:05:02.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.463+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4025587809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78a41edba0 msgr2=0x7f78a41ea250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.463+0000 7f78a9c0d700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78a41edba0 0x7f78a41ea250 secure :-1 s=READY pgs=3694 cs=0 l=1 rev1=1 crypto rx=0x7f7898015040 tx=0x7f78a4066cf0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.463+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4025587809 shutdown_connections 2023-03-03T20:05:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.463+0000 7f78a9c0d700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f78a406a7f0 0x7f78a41ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.463+0000 7f78a9c0d700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f788c004850 0x7f78a406a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.463+0000 7f78a9c0d700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f78900682f0 0x7f789006a7a0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.463+0000 7f78a9c0d700 1 --2- 172.21.15.7:0/4025587809 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f78a41edba0 0x7f78a41ea250 unknown :-1 s=CLOSED pgs=3694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:02.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.463+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4025587809 >> 172.21.15.7:0/4025587809 conn(0x7f78a40a9430 msgr2=0x7f78a40c1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:05:02.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.464+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4025587809 shutdown_connections 2023-03-03T20:05:02.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:02.464+0000 7f78a9c0d700 1 -- 172.21.15.7:0/4025587809 wait complete. 2023-03-03T20:05:02.466 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:05:02.476 INFO:tasks.mds_thrash.fs.[b]:waiting for 72 secs before thrashing 2023-03-03T20:05:19.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:05:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.900+0000 7f99d65dc700 1 Processor -- start 2023-03-03T20:05:19.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.900+0000 7f99d65dc700 1 -- start start 2023-03-03T20:05:19.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.901+0000 7f99d65dc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f99d0072250 0x7f99d014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:19.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.901+0000 7f99d65dc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 0x7f99d014e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:19.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.901+0000 7f99d65dc700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d01510d0 con 0x7f99d0071e80 2023-03-03T20:05:19.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.901+0000 7f99d65dc700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99d00c3740 con 0x7f99d0072250 2023-03-03T20:05:19.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.901+0000 7f99d65dc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d0070670 con 0x7f99d00c1ce0 2023-03-03T20:05:19.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.901+0000 7f99d4b79700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 0x7f99d014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:19.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.901+0000 7f99cf7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f99d0072250 0x7f99d014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:19.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.901+0000 7f99cffff700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f99d0071e80 0x7f99d0151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:41940/0 (socket says 172.21.15.7:41940) 2023-03-03T20:05:19.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.901+0000 7f99cffff700 1 -- 172.21.15.7:0/2434797339 learned_addr learned my addr 172.21.15.7:0/2434797339 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:05:19.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.902+0000 7f99d4b79700 1 -- 172.21.15.7:0/2434797339 >> v2:172.21.15.143:3300/0 conn(0x7f99d0072250 msgr2=0x7f99d014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:05:19.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.902+0000 7f99d4b79700 1 --2- 172.21.15.7:0/2434797339 >> v2:172.21.15.143:3300/0 conn(0x7f99d0072250 0x7f99d014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:19.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.902+0000 7f99d4b79700 1 -- 172.21.15.7:0/2434797339 >> v1:172.21.15.143:6789/0 conn(0x7f99d0071e80 legacy=0x7f99d0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:19.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.902+0000 7f99d4b79700 1 -- 172.21.15.7:0/2434797339 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d00c39e0 con 0x7f99d00c1ce0 2023-03-03T20:05:19.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.902+0000 7f99cf7fe700 1 --2- 172.21.15.7:0/2434797339 >> v2:172.21.15.143:3300/0 conn(0x7f99d0072250 0x7f99d014e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:05:19.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.902+0000 7f99d4b79700 1 --2- 172.21.15.7:0/2434797339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 0x7f99d014e730 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009d90 tx=0x7f99c800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9032d4fbb1b1260 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:05:19.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.902+0000 7f99ceffd700 1 -- 172.21.15.7:0/2434797339 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c800e070 con 0x7f99d00c1ce0 2023-03-03T20:05:19.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.902+0000 7f99ceffd700 1 -- 172.21.15.7:0/2434797339 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99c800bb20 con 0x7f99d00c1ce0 2023-03-03T20:05:19.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.902+0000 7f99ceffd700 1 -- 172.21.15.7:0/2434797339 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c8012430 con 0x7f99d00c1ce0 2023-03-03T20:05:19.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.903+0000 7f99d65dc700 1 -- 172.21.15.7:0/2434797339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 msgr2=0x7f99d014e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:19.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.903+0000 7f99d65dc700 1 --2- 172.21.15.7:0/2434797339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 0x7f99d014e730 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009d90 tx=0x7f99c800b330 comp rx=0 tx=0).stop 2023-03-03T20:05:19.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.903+0000 7f99d65dc700 1 -- 172.21.15.7:0/2434797339 shutdown_connections 2023-03-03T20:05:19.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.903+0000 7f99d65dc700 1 --2- 172.21.15.7:0/2434797339 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 0x7f99d014e730 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:19.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.903+0000 7f99d65dc700 1 --2- 172.21.15.7:0/2434797339 >> v2:172.21.15.143:3300/0 conn(0x7f99d0072250 0x7f99d014e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:19.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.903+0000 7f99d65dc700 1 -- 172.21.15.7:0/2434797339 >> 172.21.15.7:0/2434797339 conn(0x7f99d00a94d0 msgr2=0x7f99d00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:05:19.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.903+0000 7f99d65dc700 1 -- 172.21.15.7:0/2434797339 shutdown_connections 2023-03-03T20:05:19.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.903+0000 7f99d65dc700 1 -- 172.21.15.7:0/2434797339 wait complete. 2023-03-03T20:05:19.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.904+0000 7f99d65dc700 1 Processor -- start 2023-03-03T20:05:19.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.904+0000 7f99d65dc700 1 -- start start 2023-03-03T20:05:19.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99d65dc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99d0071e80 0x7f99d014aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:19.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99d65dc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d0072250 0x7f99d014ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:19.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99d65dc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 0x7f99d014d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:19.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99d65dc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d00c5160 con 0x7f99d0071e80 2023-03-03T20:05:19.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99d65dc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d0073f70 con 0x7f99d0072250 2023-03-03T20:05:19.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99d65dc700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99d00736f0 con 0x7f99d00c1ce0 2023-03-03T20:05:19.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99cffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99d0071e80 0x7f99d014aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:19.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99cffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99d0071e80 0x7f99d014aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44446/0 (socket says 172.21.15.7:44446) 2023-03-03T20:05:19.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99cffff700 1 -- 172.21.15.7:0/1128432412 learned_addr learned my addr 172.21.15.7:0/1128432412 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:05:19.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99cf7fe700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d0072250 0x7f99d014ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:19.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99d4b79700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 0x7f99d014d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:19.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99cffff700 1 -- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 msgr2=0x7f99d014d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:19.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99cffff700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 0x7f99d014d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:19.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99cffff700 1 -- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d0072250 msgr2=0x7f99d014ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:19.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99cffff700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d0072250 0x7f99d014ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:19.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99cffff700 1 -- 172.21.15.7:0/1128432412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c00094a0 con 0x7f99d0071e80 2023-03-03T20:05:19.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99d4b79700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 0x7f99d014d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:05:19.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.905+0000 7f99cffff700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99d0071e80 0x7f99d014aac0 secure :-1 s=READY pgs=3695 cs=0 l=1 rev1=1 crypto rx=0x7f99b800c520 tx=0x7f99b800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:05:19.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.906+0000 7f99ccff9700 1 -- 172.21.15.7:0/1128432412 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b8005d70 con 0x7f99d0071e80 2023-03-03T20:05:19.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.906+0000 7f99ccff9700 1 -- 172.21.15.7:0/1128432412 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b80070d0 con 0x7f99d0071e80 2023-03-03T20:05:19.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.906+0000 7f99d65dc700 1 -- 172.21.15.7:0/1128432412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c8009a70 con 0x7f99d0071e80 2023-03-03T20:05:19.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.906+0000 7f99ccff9700 1 -- 172.21.15.7:0/1128432412 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b8007440 con 0x7f99d0071e80 2023-03-03T20:05:19.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.906+0000 7f99d65dc700 1 -- 172.21.15.7:0/1128432412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d0151f30 con 0x7f99d0071e80 2023-03-03T20:05:19.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.907+0000 7f99d65dc700 1 -- 172.21.15.7:0/1128432412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d0151300 con 0x7f99d0071e80 2023-03-03T20:05:19.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.907+0000 7f99ccff9700 1 -- 172.21.15.7:0/1128432412 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f99b8019030 con 0x7f99d0071e80 2023-03-03T20:05:19.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.907+0000 7f99ccff9700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99b0068320 0x7f99b006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:19.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.907+0000 7f99ccff9700 1 -- 172.21.15.7:0/1128432412 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f99b80853f0 con 0x7f99d0071e80 2023-03-03T20:05:19.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.908+0000 7f99cf7fe700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99b0068320 0x7f99b006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:19.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.908+0000 7f99cf7fe700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99b0068320 0x7f99b006a7d0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f99d00be070 tx=0x7f99c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:05:19.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:19.912+0000 7f99ccff9700 1 -- 172.21.15.7:0/1128432412 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f99b804b5e0 con 0x7f99d0071e80 2023-03-03T20:05:20.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.151+0000 7f99d65dc700 1 -- 172.21.15.7:0/1128432412 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99d006c9a0 con 0x7f99d0071e80 2023-03-03T20:05:20.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.151+0000 7f99ccff9700 1 -- 172.21.15.7:0/1128432412 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f99b8051840 con 0x7f99d0071e80 2023-03-03T20:05:20.152 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:05:20.152 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:05:20.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 -- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99b0068320 msgr2=0x7f99b006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:20.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99b0068320 0x7f99b006a7d0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f99d00be070 tx=0x7f99c0008040 comp rx=0 tx=0).stop 2023-03-03T20:05:20.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 -- 172.21.15.7:0/1128432412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99d0071e80 msgr2=0x7f99d014aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:20.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99d0071e80 0x7f99d014aac0 secure :-1 s=READY pgs=3695 cs=0 l=1 rev1=1 crypto rx=0x7f99b800c520 tx=0x7f99b800c8e0 comp rx=0 tx=0).stop 2023-03-03T20:05:20.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 -- 172.21.15.7:0/1128432412 shutdown_connections 2023-03-03T20:05:20.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f99d00c1ce0 0x7f99d014d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:20.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f99b0068320 0x7f99b006a7d0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:20.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f99d0071e80 0x7f99d014aac0 unknown :-1 s=CLOSED pgs=3695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:20.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 --2- 172.21.15.7:0/1128432412 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99d0072250 0x7f99d014ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:20.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 -- 172.21.15.7:0/1128432412 >> 172.21.15.7:0/1128432412 conn(0x7f99d00a94d0 msgr2=0x7f99d00c20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:05:20.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 -- 172.21.15.7:0/1128432412 shutdown_connections 2023-03-03T20:05:20.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:20.153+0000 7f99d65dc700 1 -- 172.21.15.7:0/1128432412 wait complete. 2023-03-03T20:05:20.156 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:05:20.166 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before thrashing 2023-03-03T20:05:21.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:05:21.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.305+0000 7f6b7191e700 1 Processor -- start 2023-03-03T20:05:21.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.305+0000 7f6b7191e700 1 -- start start 2023-03-03T20:05:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b7191e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6b6c0c00e0 0x7f6b6c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b7191e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6c0c7310 con 0x7f6b6c154ff0 2023-03-03T20:05:21.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b7191e700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6c0bfd10 con 0x7f6b6c0caa30 2023-03-03T20:05:21.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b7191e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b6c155430 con 0x7f6b6c0c00e0 2023-03-03T20:05:21.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b6b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6b6c154ff0 0x7f6b6c15ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38440/0 (socket says 172.21.15.7:38440) 2023-03-03T20:05:21.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b6b7fe700 1 -- 172.21.15.7:0/4000251060 learned_addr learned my addr 172.21.15.7:0/4000251060 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:05:21.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b6a7fc700 1 --2- 172.21.15.7:0/4000251060 >> v2:172.21.15.143:3300/0 conn(0x7f6b6c0c00e0 0x7f6b6c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:21.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b69ffb700 1 -- 172.21.15.7:0/4000251060 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597465708 0 0) 0x7f6b6c0c7310 con 0x7f6b6c154ff0 2023-03-03T20:05:21.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b69ffb700 1 -- 172.21.15.7:0/4000251060 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b540035b0 con 0x7f6b6c154ff0 2023-03-03T20:05:21.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b6a7fc700 1 -- 172.21.15.7:0/4000251060 >> v1:172.21.15.143:6789/0 conn(0x7f6b6c0caa30 legacy=0x7f6b6c0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:21.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b6a7fc700 1 -- 172.21.15.7:0/4000251060 >> v1:172.21.15.7:6789/0 conn(0x7f6b6c154ff0 legacy=0x7f6b6c15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:21.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b6a7fc700 1 -- 172.21.15.7:0/4000251060 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c0bfa30 con 0x7f6b6c0c00e0 2023-03-03T20:05:21.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.306+0000 7f6b6a7fc700 1 --2- 172.21.15.7:0/4000251060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b6c0c00e0 0x7f6b6c0c5c60 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009820 tx=0x7f6b60009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55dd1b6cf6f4132e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:05:21.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.307+0000 7f6b69ffb700 1 -- 172.21.15.7:0/4000251060 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60005e30 con 0x7f6b6c0c00e0 2023-03-03T20:05:21.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.307+0000 7f6b69ffb700 1 -- 172.21.15.7:0/4000251060 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b60006030 con 0x7f6b6c0c00e0 2023-03-03T20:05:21.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.307+0000 7f6b69ffb700 1 -- 172.21.15.7:0/4000251060 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b600063a0 con 0x7f6b6c0c00e0 2023-03-03T20:05:21.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.307+0000 7f6b7191e700 1 -- 172.21.15.7:0/4000251060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b6c0c00e0 msgr2=0x7f6b6c0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:21.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.307+0000 7f6b7191e700 1 --2- 172.21.15.7:0/4000251060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b6c0c00e0 0x7f6b6c0c5c60 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009820 tx=0x7f6b60009b30 comp rx=0 tx=0).stop 2023-03-03T20:05:21.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.307+0000 7f6b7191e700 1 -- 172.21.15.7:0/4000251060 shutdown_connections 2023-03-03T20:05:21.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.307+0000 7f6b7191e700 1 --2- 172.21.15.7:0/4000251060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b6c0c00e0 0x7f6b6c0c5c60 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:21.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.307+0000 7f6b7191e700 1 -- 172.21.15.7:0/4000251060 >> 172.21.15.7:0/4000251060 conn(0x7f6b6c066e40 msgr2=0x7f6b6c071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:05:21.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.307+0000 7f6b7191e700 1 -- 172.21.15.7:0/4000251060 shutdown_connections 2023-03-03T20:05:21.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.307+0000 7f6b7191e700 1 -- 172.21.15.7:0/4000251060 wait complete. 2023-03-03T20:05:21.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.309+0000 7f6b7191e700 1 Processor -- start 2023-03-03T20:05:21.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.309+0000 7f6b7191e700 1 -- start start 2023-03-03T20:05:21.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.309+0000 7f6b7191e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b6c0c00e0 0x7f6b6c1ddcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:21.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.309+0000 7f6b7191e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b6c0caa30 0x7f6b6c0c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:21.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.309+0000 7f6b7191e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b6c154ff0 0x7f6b6c13eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:21.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.309+0000 7f6b7191e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c0c1c40 con 0x7f6b6c0caa30 2023-03-03T20:05:21.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.309+0000 7f6b7191e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c1ea890 con 0x7f6b6c154ff0 2023-03-03T20:05:21.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.309+0000 7f6b7191e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c1eab70 con 0x7f6b6c0c00e0 2023-03-03T20:05:21.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b6c0caa30 0x7f6b6c0c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:21.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6a7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b6c0caa30 0x7f6b6c0c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44464/0 (socket says 172.21.15.7:44464) 2023-03-03T20:05:21.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6a7fc700 1 -- 172.21.15.7:0/1273457533 learned_addr learned my addr 172.21.15.7:0/1273457533 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:05:21.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6b7fe700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b6c154ff0 0x7f6b6c13eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:21.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6affd700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b6c0c00e0 0x7f6b6c1ddcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:21.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6a7fc700 1 -- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b6c0c00e0 msgr2=0x7f6b6c1ddcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:21.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6a7fc700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b6c0c00e0 0x7f6b6c1ddcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:21.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6a7fc700 1 -- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b6c154ff0 msgr2=0x7f6b6c13eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:21.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6a7fc700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b6c154ff0 0x7f6b6c13eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:21.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6a7fc700 1 -- 172.21.15.7:0/1273457533 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b60009430 con 0x7f6b6c0caa30 2023-03-03T20:05:21.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b6a7fc700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b6c0caa30 0x7f6b6c0c4440 secure :-1 s=READY pgs=3697 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009fd0 tx=0x7f6b60004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:05:21.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b53fff700 1 -- 172.21.15.7:0/1273457533 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60006850 con 0x7f6b6c0caa30 2023-03-03T20:05:21.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b53fff700 1 -- 172.21.15.7:0/1273457533 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b60020070 con 0x7f6b6c0caa30 2023-03-03T20:05:21.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b7191e700 1 -- 172.21.15.7:0/1273457533 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6c1547f0 con 0x7f6b6c0caa30 2023-03-03T20:05:21.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b53fff700 1 -- 172.21.15.7:0/1273457533 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60004260 con 0x7f6b6c0caa30 2023-03-03T20:05:21.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.310+0000 7f6b7191e700 1 -- 172.21.15.7:0/1273457533 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6c154510 con 0x7f6b6c0caa30 2023-03-03T20:05:21.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.311+0000 7f6b7191e700 1 -- 172.21.15.7:0/1273457533 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6c154ad0 con 0x7f6b6c0caa30 2023-03-03T20:05:21.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.312+0000 7f6b53fff700 1 -- 172.21.15.7:0/1273457533 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6b6002b020 con 0x7f6b6c0caa30 2023-03-03T20:05:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.312+0000 7f6b53fff700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b580682f0 0x7f6b5806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:05:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.312+0000 7f6b53fff700 1 -- 172.21.15.7:0/1273457533 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6b600876f0 con 0x7f6b6c0caa30 2023-03-03T20:05:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.312+0000 7f6b6affd700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b580682f0 0x7f6b5806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:05:21.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.313+0000 7f6b6affd700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b580682f0 0x7f6b5806a7a0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c000f60 tx=0x7f6b5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:05:21.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.317+0000 7f6b53fff700 1 -- 172.21.15.7:0/1273457533 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6b60053a90 con 0x7f6b6c0caa30 2023-03-03T20:05:21.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.556+0000 7f6b7191e700 1 -- 172.21.15.7:0/1273457533 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b6c074bb0 con 0x7f6b6c0caa30 2023-03-03T20:05:21.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.556+0000 7f6b53fff700 1 -- 172.21.15.7:0/1273457533 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6b600570b0 con 0x7f6b6c0caa30 2023-03-03T20:05:21.557 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:05:21.557 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:05:21.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 -- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b580682f0 msgr2=0x7f6b5806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:21.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b580682f0 0x7f6b5806a7a0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c000f60 tx=0x7f6b5c009040 comp rx=0 tx=0).stop 2023-03-03T20:05:21.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 -- 172.21.15.7:0/1273457533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b6c0caa30 msgr2=0x7f6b6c0c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:05:21.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b6c0caa30 0x7f6b6c0c4440 secure :-1 s=READY pgs=3697 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009fd0 tx=0x7f6b60004750 comp rx=0 tx=0).stop 2023-03-03T20:05:21.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 -- 172.21.15.7:0/1273457533 shutdown_connections 2023-03-03T20:05:21.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b6c154ff0 0x7f6b6c13eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:21.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6b6c0c00e0 0x7f6b6c1ddcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:21.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6b580682f0 0x7f6b5806a7a0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:21.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 --2- 172.21.15.7:0/1273457533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b6c0caa30 0x7f6b6c0c4440 unknown :-1 s=CLOSED pgs=3697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:05:21.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 -- 172.21.15.7:0/1273457533 >> 172.21.15.7:0/1273457533 conn(0x7f6b6c066e40 msgr2=0x7f6b6c071680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:05:21.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.558+0000 7f6b7191e700 1 -- 172.21.15.7:0/1273457533 shutdown_connections 2023-03-03T20:05:21.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:05:21.559+0000 7f6b7191e700 1 -- 172.21.15.7:0/1273457533 wait complete. 2023-03-03T20:05:21.561 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:05:21.571 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-03-03T20:05:25.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:05:25.554 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:05:55.741 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:05:55.744 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:06:14.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:06:14.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.610+0000 7f1f0c01a700 1 Processor -- start 2023-03-03T20:06:14.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.610+0000 7f1f0c01a700 1 -- start start 2023-03-03T20:06:14.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f0c01a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f04071990 0x7f1f04071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:14.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f0c01a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f0414b460 0x7f1f040722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:14.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f0c01a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f04070360 con 0x7f1f04151c00 2023-03-03T20:06:14.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f0c01a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f041521a0 con 0x7f1f0414b460 2023-03-03T20:06:14.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f0c01a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0406a5e0 con 0x7f1f04071990 2023-03-03T20:06:14.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f0a5b7700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1f04151c00 0x7f1f0406aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38502/0 (socket says 172.21.15.7:38502) 2023-03-03T20:06:14.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f095b5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1f0414b460 0x7f1f040722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:14.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f0a5b7700 1 -- 172.21.15.7:0/2101967645 learned_addr learned my addr 172.21.15.7:0/2101967645 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:06:14.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f09db6700 1 --2- 172.21.15.7:0/2101967645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f04071990 0x7f1f04071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:14.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f08db4700 1 -- 172.21.15.7:0/2101967645 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234306397 0 0) 0x7f1f04070360 con 0x7f1f04151c00 2023-03-03T20:06:14.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.611+0000 7f1f08db4700 1 -- 172.21.15.7:0/2101967645 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef00035b0 con 0x7f1f04151c00 2023-03-03T20:06:14.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.612+0000 7f1f095b5700 1 -- 172.21.15.7:0/2101967645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f04071990 msgr2=0x7f1f04071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:06:14.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.612+0000 7f1f095b5700 1 --2- 172.21.15.7:0/2101967645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f04071990 0x7f1f04071d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:14.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.612+0000 7f1f095b5700 1 -- 172.21.15.7:0/2101967645 >> v1:172.21.15.7:6789/0 conn(0x7f1f04151c00 legacy=0x7f1f0406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:14.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.612+0000 7f1f095b5700 1 -- 172.21.15.7:0/2101967645 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f040c38b0 con 0x7f1f0414b460 2023-03-03T20:06:14.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.612+0000 7f1f095b5700 1 --2- 172.21.15.7:0/2101967645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0414b460 0x7f1f040722a0 secure :-1 s=READY pgs=3699 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009840 tx=0x7f1f00009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e572e4801fd13d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:06:14.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.612+0000 7f1f08db4700 1 -- 172.21.15.7:0/2101967645 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00011830 con 0x7f1f0414b460 2023-03-03T20:06:14.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.612+0000 7f1f08db4700 1 -- 172.21.15.7:0/2101967645 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f00011990 con 0x7f1f0414b460 2023-03-03T20:06:14.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.612+0000 7f1f08db4700 1 -- 172.21.15.7:0/2101967645 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00011d20 con 0x7f1f0414b460 2023-03-03T20:06:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.612+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2101967645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0414b460 msgr2=0x7f1f040722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.613+0000 7f1f0c01a700 1 --2- 172.21.15.7:0/2101967645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0414b460 0x7f1f040722a0 secure :-1 s=READY pgs=3699 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009840 tx=0x7f1f00009b50 comp rx=0 tx=0).stop 2023-03-03T20:06:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.613+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2101967645 shutdown_connections 2023-03-03T20:06:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.613+0000 7f1f0c01a700 1 --2- 172.21.15.7:0/2101967645 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f04071990 0x7f1f04071d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.613+0000 7f1f0c01a700 1 --2- 172.21.15.7:0/2101967645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f0414b460 0x7f1f040722a0 unknown :-1 s=CLOSED pgs=3699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.613+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2101967645 >> 172.21.15.7:0/2101967645 conn(0x7f1f040a94d0 msgr2=0x7f1f040a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:06:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.613+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2101967645 shutdown_connections 2023-03-03T20:06:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.613+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2101967645 wait complete. 2023-03-03T20:06:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.614+0000 7f1f0c01a700 1 Processor -- start 2023-03-03T20:06:14.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.614+0000 7f1f0c01a700 1 -- start start 2023-03-03T20:06:14.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0c01a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f04071990 0x7f1f04145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:14.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0c01a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f04151c00 0x7f1f041ec6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:14.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0c01a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f040c1500 0x7f1f041ecc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:14.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0c01a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040aa750 con 0x7f1f040c1500 2023-03-03T20:06:14.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0c01a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f04151720 con 0x7f1f04071990 2023-03-03T20:06:14.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0c01a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f041ed170 con 0x7f1f04151c00 2023-03-03T20:06:14.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0a5b7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f040c1500 0x7f1f041ecc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:14.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0a5b7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f040c1500 0x7f1f041ecc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44528/0 (socket says 172.21.15.7:44528) 2023-03-03T20:06:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0a5b7700 1 -- 172.21.15.7:0/2383982356 learned_addr learned my addr 172.21.15.7:0/2383982356 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:06:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f095b5700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f04151c00 0x7f1f041ec6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f09db6700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f04071990 0x7f1f04145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0a5b7700 1 -- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f04151c00 msgr2=0x7f1f041ec6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0a5b7700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f04151c00 0x7f1f041ec6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0a5b7700 1 -- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f04071990 msgr2=0x7f1f04145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0a5b7700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f04071990 0x7f1f04145f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:14.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0a5b7700 1 -- 172.21.15.7:0/2383982356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f000094a0 con 0x7f1f040c1500 2023-03-03T20:06:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0a5b7700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f040c1500 0x7f1f041ecc30 secure :-1 s=READY pgs=3700 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00caa0 tx=0x7f1efc00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:06:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1efa7fc700 1 -- 172.21.15.7:0/2383982356 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc00a9e0 con 0x7f1f040c1500 2023-03-03T20:06:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1efa7fc700 1 -- 172.21.15.7:0/2383982356 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1efc00ab40 con 0x7f1f040c1500 2023-03-03T20:06:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1efa7fc700 1 -- 172.21.15.7:0/2383982356 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc012400 con 0x7f1f040c1500 2023-03-03T20:06:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.615+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2383982356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f040c3b50 con 0x7f1f040c1500 2023-03-03T20:06:14.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.616+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2383982356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0406a2b0 con 0x7f1f040c1500 2023-03-03T20:06:14.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.617+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2383982356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f0406b180 con 0x7f1f040c1500 2023-03-03T20:06:14.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.617+0000 7f1efa7fc700 1 -- 172.21.15.7:0/2383982356 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1efc019030 con 0x7f1f040c1500 2023-03-03T20:06:14.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.617+0000 7f1efa7fc700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 0x7f1ee806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:14.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.617+0000 7f1efa7fc700 1 -- 172.21.15.7:0/2383982356 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1efc085d80 con 0x7f1f040c1500 2023-03-03T20:06:14.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.618+0000 7f1f09db6700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 0x7f1ee806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:14.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.618+0000 7f1f09db6700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 0x7f1ee806a7a0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009b70 tx=0x7f1ef4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:06:14.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.622+0000 7f1efa7fc700 1 -- 172.21.15.7:0/2383982356 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1efc052120 con 0x7f1f040c1500 2023-03-03T20:06:14.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.855+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2383982356 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f041ee280 con 0x7f1f040c1500 2023-03-03T20:06:14.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.856+0000 7f1efa7fc700 1 -- 172.21.15.7:0/2383982356 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1efc055740 con 0x7f1f040c1500 2023-03-03T20:06:14.857 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:06:14.857 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:06:14.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 msgr2=0x7f1ee806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:14.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 0x7f1ee806a7a0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009b70 tx=0x7f1ef4008040 comp rx=0 tx=0).stop 2023-03-03T20:06:14.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2383982356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f040c1500 msgr2=0x7f1f041ecc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:14.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f040c1500 0x7f1f041ecc30 secure :-1 s=READY pgs=3700 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00caa0 tx=0x7f1efc00cdb0 comp rx=0 tx=0).stop 2023-03-03T20:06:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2383982356 shutdown_connections 2023-03-03T20:06:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ee80682f0 0x7f1ee806a7a0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f040c1500 0x7f1f041ecc30 unknown :-1 s=CLOSED pgs=3700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1f04071990 0x7f1f04145f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 --2- 172.21.15.7:0/2383982356 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1f04151c00 0x7f1f041ec6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2383982356 >> 172.21.15.7:0/2383982356 conn(0x7f1f040a94d0 msgr2=0x7f1f040a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:06:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2383982356 shutdown_connections 2023-03-03T20:06:14.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:14.858+0000 7f1f0c01a700 1 -- 172.21.15.7:0/2383982356 wait complete. 2023-03-03T20:06:14.861 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:06:14.871 INFO:tasks.mds_thrash.fs.[b]:waiting for 35 secs before thrashing 2023-03-03T20:06:25.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:06:25.861 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:06:26.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:06:26.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.710+0000 7ff90813a700 1 Processor -- start 2023-03-03T20:06:26.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.711+0000 7ff90813a700 1 -- start start 2023-03-03T20:06:26.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.711+0000 7ff90813a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7ff9000ac220 0x7ff9000736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:26.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.711+0000 7ff90813a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9000ac900 con 0x7ff900152b60 2023-03-03T20:06:26.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.711+0000 7ff90813a700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9000acb50 con 0x7ff900073c30 2023-03-03T20:06:26.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.711+0000 7ff90813a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff900071e30 con 0x7ff9000ac220 2023-03-03T20:06:26.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.712+0000 7ff9066d7700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff900152b60 0x7ff90015ccf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38518/0 (socket says 172.21.15.7:38518) 2023-03-03T20:06:26.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.712+0000 7ff9066d7700 1 -- 172.21.15.7:0/4137078494 learned_addr learned my addr 172.21.15.7:0/4137078494 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:06:26.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.712+0000 7ff905ed6700 1 --2- 172.21.15.7:0/4137078494 >> v2:172.21.15.143:3300/0 conn(0x7ff9000ac220 0x7ff9000736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:26.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.712+0000 7ff904ed4700 1 -- 172.21.15.7:0/4137078494 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505802138 0 0) 0x7ff9000ac900 con 0x7ff900152b60 2023-03-03T20:06:26.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.712+0000 7ff904ed4700 1 -- 172.21.15.7:0/4137078494 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec0035b0 con 0x7ff900152b60 2023-03-03T20:06:26.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.712+0000 7ff905ed6700 1 -- 172.21.15.7:0/4137078494 >> v1:172.21.15.143:6789/0 conn(0x7ff900073c30 legacy=0x7ff900152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:26.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.712+0000 7ff904ed4700 1 -- 172.21.15.7:0/4137078494 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4066424258 0 0) 0x7ff8ec0035b0 con 0x7ff900152b60 2023-03-03T20:06:26.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.712+0000 7ff905ed6700 1 -- 172.21.15.7:0/4137078494 >> v1:172.21.15.7:6789/0 conn(0x7ff900152b60 legacy=0x7ff90015ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:26.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.712+0000 7ff905ed6700 1 -- 172.21.15.7:0/4137078494 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9001559c0 con 0x7ff9000ac220 2023-03-03T20:06:26.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff905ed6700 1 --2- 172.21.15.7:0/4137078494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9000ac220 0x7ff9000736f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009840 tx=0x7ff8fc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec3b735535379a05 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:06:26.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff904ed4700 1 -- 172.21.15.7:0/4137078494 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80019f0 con 0x7ff9000ac220 2023-03-03T20:06:26.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff904ed4700 1 -- 172.21.15.7:0/4137078494 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8fc004030 con 0x7ff9000ac220 2023-03-03T20:06:26.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff904ed4700 1 -- 172.21.15.7:0/4137078494 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc0043c0 con 0x7ff9000ac220 2023-03-03T20:06:26.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff90813a700 1 -- 172.21.15.7:0/4137078494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9000ac220 msgr2=0x7ff9000736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:26.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff90813a700 1 --2- 172.21.15.7:0/4137078494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9000ac220 0x7ff9000736f0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009840 tx=0x7ff8fc009b50 comp rx=0 tx=0).stop 2023-03-03T20:06:26.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff90813a700 1 -- 172.21.15.7:0/4137078494 shutdown_connections 2023-03-03T20:06:26.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff90813a700 1 --2- 172.21.15.7:0/4137078494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9000ac220 0x7ff9000736f0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:26.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff90813a700 1 -- 172.21.15.7:0/4137078494 >> 172.21.15.7:0/4137078494 conn(0x7ff9000a94d0 msgr2=0x7ff9000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:06:26.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff90813a700 1 -- 172.21.15.7:0/4137078494 shutdown_connections 2023-03-03T20:06:26.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.713+0000 7ff90813a700 1 -- 172.21.15.7:0/4137078494 wait complete. 2023-03-03T20:06:26.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff90813a700 1 Processor -- start 2023-03-03T20:06:26.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff90813a700 1 -- start start 2023-03-03T20:06:26.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff90813a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff900073c30 0x7ff9001db9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:26.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff90813a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9000ac220 0x7ff9001e80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:26.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff90813a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff900152b60 0x7ff9001ded20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:26.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff90813a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9001e8630 con 0x7ff900152b60 2023-03-03T20:06:26.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff90813a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9000c1000 con 0x7ff9000ac220 2023-03-03T20:06:26.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff90813a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9000c12e0 con 0x7ff900073c30 2023-03-03T20:06:26.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff9066d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff900152b60 0x7ff9001ded20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:26.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff9066d7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff900152b60 0x7ff9001ded20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44544/0 (socket says 172.21.15.7:44544) 2023-03-03T20:06:26.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff9066d7700 1 -- 172.21.15.7:0/725310407 learned_addr learned my addr 172.21.15.7:0/725310407 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:06:26.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff9056d5700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9000ac220 0x7ff9001e80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.715+0000 7ff905ed6700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff900073c30 0x7ff9001db9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff9066d7700 1 -- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff900073c30 msgr2=0x7ff9001db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff9066d7700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff900073c30 0x7ff9001db9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff9066d7700 1 -- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9000ac220 msgr2=0x7ff9001e80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff9066d7700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9000ac220 0x7ff9001e80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:26.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff9066d7700 1 -- 172.21.15.7:0/725310407 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0094a0 con 0x7ff900152b60 2023-03-03T20:06:26.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff9066d7700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff900152b60 0x7ff9001ded20 secure :-1 s=READY pgs=3702 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800cbe0 tx=0x7ff8f800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:06:26.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff8f67fc700 1 -- 172.21.15.7:0/725310407 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f800aa60 con 0x7ff900152b60 2023-03-03T20:06:26.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff8f67fc700 1 -- 172.21.15.7:0/725310407 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f800abe0 con 0x7ff900152b60 2023-03-03T20:06:26.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff90813a700 1 -- 172.21.15.7:0/725310407 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff900155f80 con 0x7ff900152b60 2023-03-03T20:06:26.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff8f67fc700 1 -- 172.21.15.7:0/725310407 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80070d0 con 0x7ff900152b60 2023-03-03T20:06:26.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.716+0000 7ff90813a700 1 -- 172.21.15.7:0/725310407 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff900155ca0 con 0x7ff900152b60 2023-03-03T20:06:26.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.717+0000 7ff90813a700 1 -- 172.21.15.7:0/725310407 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff900156260 con 0x7ff900152b60 2023-03-03T20:06:26.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.718+0000 7ff8f67fc700 1 -- 172.21.15.7:0/725310407 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ff8f801a030 con 0x7ff900152b60 2023-03-03T20:06:26.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.718+0000 7ff8f67fc700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8e40682f0 0x7ff8e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:26.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.718+0000 7ff8f67fc700 1 -- 172.21.15.7:0/725310407 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7ff8f8085d60 con 0x7ff900152b60 2023-03-03T20:06:26.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.718+0000 7ff905ed6700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8e40682f0 0x7ff8e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:26.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.719+0000 7ff905ed6700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8e40682f0 0x7ff8e406a7a0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc0090b0 tx=0x7ff8fc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:06:26.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.722+0000 7ff8f67fc700 1 -- 172.21.15.7:0/725310407 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7ff8f8052100 con 0x7ff900152b60 2023-03-03T20:06:26.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.958+0000 7ff90813a700 1 -- 172.21.15.7:0/725310407 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9000c1800 con 0x7ff900152b60 2023-03-03T20:06:26.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.959+0000 7ff8f67fc700 1 -- 172.21.15.7:0/725310407 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7ff8f8018090 con 0x7ff900152b60 2023-03-03T20:06:26.960 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:06:26.960 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:06:26.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 -- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8e40682f0 msgr2=0x7ff8e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:26.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8e40682f0 0x7ff8e406a7a0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc0090b0 tx=0x7ff8fc008030 comp rx=0 tx=0).stop 2023-03-03T20:06:26.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 -- 172.21.15.7:0/725310407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff900152b60 msgr2=0x7ff9001ded20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:26.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff900152b60 0x7ff9001ded20 secure :-1 s=READY pgs=3702 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800cbe0 tx=0x7ff8f800cef0 comp rx=0 tx=0).stop 2023-03-03T20:06:26.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 -- 172.21.15.7:0/725310407 shutdown_connections 2023-03-03T20:06:26.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7ff8e40682f0 0x7ff8e406a7a0 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:26.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff900152b60 0x7ff9001ded20 unknown :-1 s=CLOSED pgs=3702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:26.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7ff900073c30 0x7ff9001db9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:26.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 --2- 172.21.15.7:0/725310407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9000ac220 0x7ff9001e80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:26.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 -- 172.21.15.7:0/725310407 >> 172.21.15.7:0/725310407 conn(0x7ff9000a94d0 msgr2=0x7ff9000a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:06:26.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 -- 172.21.15.7:0/725310407 shutdown_connections 2023-03-03T20:06:26.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:26.961+0000 7ff90813a700 1 -- 172.21.15.7:0/725310407 wait complete. 2023-03-03T20:06:26.964 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:06:26.973 INFO:tasks.mds_thrash.fs.[a]:waiting for 53 secs before thrashing 2023-03-03T20:06:49.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:06:50.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.010+0000 7fa671450700 1 Processor -- start 2023-03-03T20:06:50.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.010+0000 7fa671450700 1 -- start start 2023-03-03T20:06:50.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa671450700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa66c151cc0 0x7fa66c0ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:50.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa671450700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa66c0cac10 0x7fa66c0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa671450700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c0c7ca0 con 0x7fa66c0b00b0 2023-03-03T20:06:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa671450700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66c152090 con 0x7fa66c151cc0 2023-03-03T20:06:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa671450700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66c0c6cf0 con 0x7fa66c0cac10 2023-03-03T20:06:50.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa66affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa66c151cc0 0x7fa66c0ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa66affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa66c151cc0 0x7fa66c0ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44562/0 (socket says 172.21.15.7:44562) 2023-03-03T20:06:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa66affd700 1 -- 172.21.15.7:0/604318988 learned_addr learned my addr 172.21.15.7:0/604318988 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:06:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa66b7fe700 1 --2- 172.21.15.7:0/604318988 >> v2:172.21.15.143:3300/0 conn(0x7fa66c0cac10 0x7fa66c0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa66affd700 1 -- 172.21.15.7:0/604318988 >> v2:172.21.15.143:3300/0 conn(0x7fa66c0cac10 msgr2=0x7fa66c0c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:06:50.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa66affd700 1 --2- 172.21.15.7:0/604318988 >> v2:172.21.15.143:3300/0 conn(0x7fa66c0cac10 0x7fa66c0c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:50.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa66affd700 1 -- 172.21.15.7:0/604318988 >> v1:172.21.15.143:6789/0 conn(0x7fa66c0b00b0 legacy=0x7fa66c0c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:06:50.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa66affd700 1 -- 172.21.15.7:0/604318988 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c1548f0 con 0x7fa66c151cc0 2023-03-03T20:06:50.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.011+0000 7fa66affd700 1 --2- 172.21.15.7:0/604318988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c151cc0 0x7fa66c0ca6d0 secure :-1 s=READY pgs=3703 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0c1700 tx=0x7fa654009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4feb46c7ff3c2663 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:06:50.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa669ffb700 1 -- 172.21.15.7:0/604318988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65400b870 con 0x7fa66c151cc0 2023-03-03T20:06:50.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa669ffb700 1 -- 172.21.15.7:0/604318988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65400b9d0 con 0x7fa66c151cc0 2023-03-03T20:06:50.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa669ffb700 1 -- 172.21.15.7:0/604318988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65400bd60 con 0x7fa66c151cc0 2023-03-03T20:06:50.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa671450700 1 -- 172.21.15.7:0/604318988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c151cc0 msgr2=0x7fa66c0ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:50.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa671450700 1 --2- 172.21.15.7:0/604318988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c151cc0 0x7fa66c0ca6d0 secure :-1 s=READY pgs=3703 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0c1700 tx=0x7fa654009bd0 comp rx=0 tx=0).stop 2023-03-03T20:06:50.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa671450700 1 -- 172.21.15.7:0/604318988 shutdown_connections 2023-03-03T20:06:50.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa671450700 1 --2- 172.21.15.7:0/604318988 >> v2:172.21.15.143:3300/0 conn(0x7fa66c0cac10 0x7fa66c0c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:50.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa671450700 1 --2- 172.21.15.7:0/604318988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c151cc0 0x7fa66c0ca6d0 unknown :-1 s=CLOSED pgs=3703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:50.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa671450700 1 -- 172.21.15.7:0/604318988 >> 172.21.15.7:0/604318988 conn(0x7fa66c06b980 msgr2=0x7fa66c071610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:06:50.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa671450700 1 -- 172.21.15.7:0/604318988 shutdown_connections 2023-03-03T20:06:50.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.012+0000 7fa671450700 1 -- 172.21.15.7:0/604318988 wait complete. 2023-03-03T20:06:50.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa671450700 1 Processor -- start 2023-03-03T20:06:50.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa671450700 1 -- start start 2023-03-03T20:06:50.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa671450700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0b00b0 0x7fa66c0c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:50.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa671450700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa66c0cac10 0x7fa66c1ed4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:50.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa671450700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa66c151cc0 0x7fa66c1ed9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:50.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa671450700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c14c8e0 con 0x7fa66c0b00b0 2023-03-03T20:06:50.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa671450700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c0c46d0 con 0x7fa66c151cc0 2023-03-03T20:06:50.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa671450700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c1436e0 con 0x7fa66c0cac10 2023-03-03T20:06:50.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa66affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0b00b0 0x7fa66c0c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:50.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa66affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0b00b0 0x7fa66c0c9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44568/0 (socket says 172.21.15.7:44568) 2023-03-03T20:06:50.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.014+0000 7fa66affd700 1 -- 172.21.15.7:0/2319847552 learned_addr learned my addr 172.21.15.7:0/2319847552 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:06:50.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa66b7fe700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa66c151cc0 0x7fa66c1ed9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:50.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa66affd700 1 -- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa66c0cac10 msgr2=0x7fa66c1ed4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:50.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa66a7fc700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa66c0cac10 0x7fa66c1ed4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:50.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa66affd700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa66c0cac10 0x7fa66c1ed4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa66affd700 1 -- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa66c151cc0 msgr2=0x7fa66c1ed9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa66affd700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa66c151cc0 0x7fa66c1ed9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa66affd700 1 -- 172.21.15.7:0/2319847552 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa654009550 con 0x7fa66c0b00b0 2023-03-03T20:06:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa66affd700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0b00b0 0x7fa66c0c9930 secure :-1 s=READY pgs=3704 cs=0 l=1 rev1=1 crypto rx=0x7fa654000c00 tx=0x7fa654004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:06:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa6637fe700 1 -- 172.21.15.7:0/2319847552 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa654004520 con 0x7fa66c0b00b0 2023-03-03T20:06:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa6637fe700 1 -- 172.21.15.7:0/2319847552 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65401e070 con 0x7fa66c0b00b0 2023-03-03T20:06:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa6637fe700 1 -- 172.21.15.7:0/2319847552 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540038a0 con 0x7fa66c0b00b0 2023-03-03T20:06:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa671450700 1 -- 172.21.15.7:0/2319847552 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66c154eb0 con 0x7fa66c0b00b0 2023-03-03T20:06:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.015+0000 7fa671450700 1 -- 172.21.15.7:0/2319847552 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66c154bd0 con 0x7fa66c0b00b0 2023-03-03T20:06:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.016+0000 7fa671450700 1 -- 172.21.15.7:0/2319847552 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66c155190 con 0x7fa66c0b00b0 2023-03-03T20:06:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.017+0000 7fa6637fe700 1 -- 172.21.15.7:0/2319847552 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa65400f050 con 0x7fa66c0b00b0 2023-03-03T20:06:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.017+0000 7fa6637fe700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa64c0682f0 0x7fa64c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:06:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.017+0000 7fa6637fe700 1 -- 172.21.15.7:0/2319847552 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa654020070 con 0x7fa66c0b00b0 2023-03-03T20:06:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.017+0000 7fa66a7fc700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa64c0682f0 0x7fa64c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:06:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.018+0000 7fa66a7fc700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa64c0682f0 0x7fa64c06a7a0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fa66c1e1cf0 tx=0x7fa65c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:06:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.022+0000 7fa6637fe700 1 -- 172.21.15.7:0/2319847552 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa654053a90 con 0x7fa66c0b00b0 2023-03-03T20:06:50.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.260+0000 7fa671450700 1 -- 172.21.15.7:0/2319847552 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa66c074bb0 con 0x7fa66c0b00b0 2023-03-03T20:06:50.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.261+0000 7fa6637fe700 1 -- 172.21.15.7:0/2319847552 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa6540570b0 con 0x7fa66c0b00b0 2023-03-03T20:06:50.262 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:06:50.262 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:06:50.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 -- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa64c0682f0 msgr2=0x7fa64c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:50.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa64c0682f0 0x7fa64c06a7a0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fa66c1e1cf0 tx=0x7fa65c009040 comp rx=0 tx=0).stop 2023-03-03T20:06:50.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 -- 172.21.15.7:0/2319847552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0b00b0 msgr2=0x7fa66c0c9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:06:50.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0b00b0 0x7fa66c0c9930 secure :-1 s=READY pgs=3704 cs=0 l=1 rev1=1 crypto rx=0x7fa654000c00 tx=0x7fa654004750 comp rx=0 tx=0).stop 2023-03-03T20:06:50.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 -- 172.21.15.7:0/2319847552 shutdown_connections 2023-03-03T20:06:50.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa66c151cc0 0x7fa66c1ed9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:50.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa64c0682f0 0x7fa64c06a7a0 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:50.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa66c0b00b0 0x7fa66c0c9930 unknown :-1 s=CLOSED pgs=3704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:50.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 --2- 172.21.15.7:0/2319847552 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa66c0cac10 0x7fa66c1ed4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:06:50.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 -- 172.21.15.7:0/2319847552 >> 172.21.15.7:0/2319847552 conn(0x7fa66c06b980 msgr2=0x7fa66c152a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:06:50.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 -- 172.21.15.7:0/2319847552 shutdown_connections 2023-03-03T20:06:50.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:06:50.263+0000 7fa671450700 1 -- 172.21.15.7:0/2319847552 wait complete. 2023-03-03T20:06:50.265 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:06:50.275 INFO:tasks.mds_thrash.fs.[b]:waiting for 72 secs before thrashing 2023-03-03T20:06:56.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:06:56.019 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:07:19.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:07:20.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.115+0000 7f7c1330f700 1 Processor -- start 2023-03-03T20:07:20.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.115+0000 7f7c1330f700 1 -- start start 2023-03-03T20:07:20.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.115+0000 7f7c1330f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 0x7f7c0c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:07:20.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.115+0000 7f7c1330f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c0c06c480 0x7f7c0c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:07:20.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c1330f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c06d430 con 0x7f7c0c151d40 2023-03-03T20:07:20.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c1330f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0c06c2a0 con 0x7f7c0c06c480 2023-03-03T20:07:20.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c1330f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c06cb00 con 0x7f7c0c0c2930 2023-03-03T20:07:20.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c118ac700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c0c06c480 0x7f7c0c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:07:20.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c118ac700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7c0c06c480 0x7f7c0c157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44594/0 (socket says 172.21.15.7:44594) 2023-03-03T20:07:20.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c110ab700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7c0c151d40 0x7f7c0c0736f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38570/0 (socket says 172.21.15.7:38570) 2023-03-03T20:07:20.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c118ac700 1 -- 172.21.15.7:0/1733577977 learned_addr learned my addr 172.21.15.7:0/1733577977 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:07:20.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c108aa700 1 --2- 172.21.15.7:0/1733577977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 0x7f7c0c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:07:20.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c118ac700 1 -- 172.21.15.7:0/1733577977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 msgr2=0x7f7c0c157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:07:20.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c118ac700 1 --2- 172.21.15.7:0/1733577977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 0x7f7c0c157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c118ac700 1 -- 172.21.15.7:0/1733577977 >> v1:172.21.15.7:6789/0 conn(0x7f7c0c151d40 legacy=0x7f7c0c0736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:07:20.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c118ac700 1 -- 172.21.15.7:0/1733577977 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c06cde0 con 0x7f7c0c06c480 2023-03-03T20:07:20.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c118ac700 1 --2- 172.21.15.7:0/1733577977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c06c480 0x7f7c0c157fd0 secure :-1 s=READY pgs=3706 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009e50 tx=0x7f7c0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f859aff92efd5b81 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:07:20.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c03fff700 1 -- 172.21.15.7:0/1733577977 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400e070 con 0x7f7c0c06c480 2023-03-03T20:07:20.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c03fff700 1 -- 172.21.15.7:0/1733577977 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0400b9c0 con 0x7f7c0c06c480 2023-03-03T20:07:20.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.116+0000 7f7c03fff700 1 -- 172.21.15.7:0/1733577977 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400bd70 con 0x7f7c0c06c480 2023-03-03T20:07:20.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.117+0000 7f7c1330f700 1 -- 172.21.15.7:0/1733577977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c06c480 msgr2=0x7f7c0c157fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:07:20.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.117+0000 7f7c1330f700 1 --2- 172.21.15.7:0/1733577977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c06c480 0x7f7c0c157fd0 secure :-1 s=READY pgs=3706 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009e50 tx=0x7f7c0400b330 comp rx=0 tx=0).stop 2023-03-03T20:07:20.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.117+0000 7f7c1330f700 1 -- 172.21.15.7:0/1733577977 shutdown_connections 2023-03-03T20:07:20.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.117+0000 7f7c1330f700 1 --2- 172.21.15.7:0/1733577977 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c06c480 0x7f7c0c157fd0 unknown :-1 s=CLOSED pgs=3706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.117+0000 7f7c1330f700 1 --2- 172.21.15.7:0/1733577977 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 0x7f7c0c157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.117+0000 7f7c1330f700 1 -- 172.21.15.7:0/1733577977 >> 172.21.15.7:0/1733577977 conn(0x7f7c0c0a94d0 msgr2=0x7f7c0c0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:07:20.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.117+0000 7f7c1330f700 1 -- 172.21.15.7:0/1733577977 shutdown_connections 2023-03-03T20:07:20.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.117+0000 7f7c1330f700 1 -- 172.21.15.7:0/1733577977 wait complete. 2023-03-03T20:07:20.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.118+0000 7f7c1330f700 1 Processor -- start 2023-03-03T20:07:20.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.118+0000 7f7c1330f700 1 -- start start 2023-03-03T20:07:20.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c1330f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c0c06c480 0x7f7c0c0c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:07:20.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c1330f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 0x7f7c0c14d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:07:20.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c1330f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c151d40 0x7f7c0c155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:07:20.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c1330f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0c6ea0 con 0x7f7c0c151d40 2023-03-03T20:07:20.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c1330f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c0aa2b0 con 0x7f7c0c06c480 2023-03-03T20:07:20.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c1330f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0c157380 con 0x7f7c0c0c2930 2023-03-03T20:07:20.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c118ac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c151d40 0x7f7c0c155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:07:20.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c118ac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c151d40 0x7f7c0c155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44600/0 (socket says 172.21.15.7:44600) 2023-03-03T20:07:20.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c118ac700 1 -- 172.21.15.7:0/607360767 learned_addr learned my addr 172.21.15.7:0/607360767 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:07:20.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c108aa700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 0x7f7c0c14d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:07:20.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c110ab700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c0c06c480 0x7f7c0c0c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:07:20.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c118ac700 1 -- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 msgr2=0x7f7c0c14d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:07:20.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c118ac700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 0x7f7c0c14d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c118ac700 1 -- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c0c06c480 msgr2=0x7f7c0c0c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:07:20.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c118ac700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c0c06c480 0x7f7c0c0c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c118ac700 1 -- 172.21.15.7:0/607360767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c04009b30 con 0x7f7c0c151d40 2023-03-03T20:07:20.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c108aa700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 0x7f7c0c14d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:07:20.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.119+0000 7f7c110ab700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c0c06c480 0x7f7c0c0c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:07:20.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.120+0000 7f7c118ac700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c151d40 0x7f7c0c155af0 secure :-1 s=READY pgs=3707 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c06bb70 tx=0x7f7c04004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:07:20.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.120+0000 7f7c01ffb700 1 -- 172.21.15.7:0/607360767 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c04004250 con 0x7f7c0c151d40 2023-03-03T20:07:20.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.120+0000 7f7c01ffb700 1 -- 172.21.15.7:0/607360767 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c040043b0 con 0x7f7c0c151d40 2023-03-03T20:07:20.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.120+0000 7f7c1330f700 1 -- 172.21.15.7:0/607360767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0c073e00 con 0x7f7c0c151d40 2023-03-03T20:07:20.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.120+0000 7f7c01ffb700 1 -- 172.21.15.7:0/607360767 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c04004760 con 0x7f7c0c151d40 2023-03-03T20:07:20.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.120+0000 7f7c1330f700 1 -- 172.21.15.7:0/607360767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c06d0c0 con 0x7f7c0c151d40 2023-03-03T20:07:20.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.121+0000 7f7c1330f700 1 -- 172.21.15.7:0/607360767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0c06bea0 con 0x7f7c0c151d40 2023-03-03T20:07:20.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.122+0000 7f7c01ffb700 1 -- 172.21.15.7:0/607360767 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7c0402b070 con 0x7f7c0c151d40 2023-03-03T20:07:20.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.122+0000 7f7c01ffb700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bf00682f0 0x7f7bf006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:07:20.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.122+0000 7f7c01ffb700 1 -- 172.21.15.7:0/607360767 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7c04087390 con 0x7f7c0c151d40 2023-03-03T20:07:20.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.122+0000 7f7c110ab700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bf00682f0 0x7f7bf006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:07:20.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.123+0000 7f7c110ab700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bf00682f0 0x7f7bf006a7a0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc000f60 tx=0x7f7bfc007df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:07:20.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.126+0000 7f7c01ffb700 1 -- 172.21.15.7:0/607360767 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7c04053730 con 0x7f7c0c151d40 2023-03-03T20:07:20.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.362+0000 7f7c1330f700 1 -- 172.21.15.7:0/607360767 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c0c0c6680 con 0x7f7c0c151d40 2023-03-03T20:07:20.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.362+0000 7f7c01ffb700 1 -- 172.21.15.7:0/607360767 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7c0c0c6680 con 0x7f7c0c151d40 2023-03-03T20:07:20.363 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:07:20.364 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:07:20.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 -- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bf00682f0 msgr2=0x7f7bf006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:07:20.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bf00682f0 0x7f7bf006a7a0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc000f60 tx=0x7f7bfc007df0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 -- 172.21.15.7:0/607360767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c151d40 msgr2=0x7f7c0c155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:07:20.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c151d40 0x7f7c0c155af0 secure :-1 s=READY pgs=3707 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c06bb70 tx=0x7f7c04004ad0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 -- 172.21.15.7:0/607360767 shutdown_connections 2023-03-03T20:07:20.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7bf00682f0 0x7f7bf006a7a0 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c0c151d40 0x7f7c0c155af0 unknown :-1 s=CLOSED pgs=3707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c0c06c480 0x7f7c0c0c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 --2- 172.21.15.7:0/607360767 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7c0c0c2930 0x7f7c0c14d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:07:20.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 -- 172.21.15.7:0/607360767 >> 172.21.15.7:0/607360767 conn(0x7f7c0c0a94d0 msgr2=0x7f7c0c158280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:07:20.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 -- 172.21.15.7:0/607360767 shutdown_connections 2023-03-03T20:07:20.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:07:20.364+0000 7f7c1330f700 1 -- 172.21.15.7:0/607360767 wait complete. 2023-03-03T20:07:20.367 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:07:20.377 INFO:tasks.mds_thrash.fs.[a]:waiting for 109 secs before thrashing 2023-03-03T20:07:26.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:07:26.163 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:07:56.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:07:56.294 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:08:02.276 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:08:02.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.415+0000 7fceb14ff700 1 Processor -- start 2023-03-03T20:08:02.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.415+0000 7fceb14ff700 1 -- start start 2023-03-03T20:08:02.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.415+0000 7fceb14ff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fceac0caa30 0x7fceac0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:08:02.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.415+0000 7fceb14ff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0c00e0 0x7fceac154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:08:02.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.415+0000 7fceb14ff700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceac0cb280 con 0x7fceac0cae00 2023-03-03T20:08:02.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.415+0000 7fceb14ff700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceac0c7310 con 0x7fceac0caa30 2023-03-03T20:08:02.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.415+0000 7fceb14ff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceac0bfd10 con 0x7fceac0c00e0 2023-03-03T20:08:02.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fceaaffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fceac0caa30 0x7fceac0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:08:02.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fceab7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fceac0cae00 0x7fceac0c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38620/0 (socket says 172.21.15.7:38620) 2023-03-03T20:08:02.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fceab7fe700 1 -- 172.21.15.7:0/397516940 learned_addr learned my addr 172.21.15.7:0/397516940 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:08:02.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fceaa7fc700 1 --2- 172.21.15.7:0/397516940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0c00e0 0x7fceac154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:08:02.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fceaaffd700 1 -- 172.21.15.7:0/397516940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0c00e0 msgr2=0x7fceac154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:08:02.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fceaaffd700 1 --2- 172.21.15.7:0/397516940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0c00e0 0x7fceac154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:08:02.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fceaaffd700 1 -- 172.21.15.7:0/397516940 >> v1:172.21.15.7:6789/0 conn(0x7fceac0cae00 legacy=0x7fceac0c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:08:02.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fceaaffd700 1 -- 172.21.15.7:0/397516940 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac0bfa30 con 0x7fceac0caa30 2023-03-03T20:08:02.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fcea9ffb700 1 -- 172.21.15.7:0/397516940 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622664188 0 0) 0x7fceac0cb280 con 0x7fceac0cae00 2023-03-03T20:08:02.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fceaaffd700 1 --2- 172.21.15.7:0/397516940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0caa30 0x7fceac0c5c60 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009840 tx=0x7fce9c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9687032a9beb9e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:08:02.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fcea9ffb700 1 -- 172.21.15.7:0/397516940 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c011830 con 0x7fceac0caa30 2023-03-03T20:08:02.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fcea9ffb700 1 -- 172.21.15.7:0/397516940 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c011990 con 0x7fceac0caa30 2023-03-03T20:08:02.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.416+0000 7fcea9ffb700 1 -- 172.21.15.7:0/397516940 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c011d20 con 0x7fceac0caa30 2023-03-03T20:08:02.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.417+0000 7fceb14ff700 1 -- 172.21.15.7:0/397516940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0caa30 msgr2=0x7fceac0c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:08:02.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.417+0000 7fceb14ff700 1 --2- 172.21.15.7:0/397516940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0caa30 0x7fceac0c5c60 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009840 tx=0x7fce9c009b50 comp rx=0 tx=0).stop 2023-03-03T20:08:02.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.417+0000 7fceb14ff700 1 -- 172.21.15.7:0/397516940 shutdown_connections 2023-03-03T20:08:02.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.417+0000 7fceb14ff700 1 --2- 172.21.15.7:0/397516940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0caa30 0x7fceac0c5c60 unknown :-1 s=CLOSED pgs=3709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:08:02.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.417+0000 7fceb14ff700 1 --2- 172.21.15.7:0/397516940 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0c00e0 0x7fceac154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:08:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.417+0000 7fceb14ff700 1 -- 172.21.15.7:0/397516940 >> 172.21.15.7:0/397516940 conn(0x7fceac066e40 msgr2=0x7fceac071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:08:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.417+0000 7fceb14ff700 1 -- 172.21.15.7:0/397516940 shutdown_connections 2023-03-03T20:08:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.417+0000 7fceb14ff700 1 -- 172.21.15.7:0/397516940 wait complete. 2023-03-03T20:08:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.418+0000 7fceb14ff700 1 Processor -- start 2023-03-03T20:08:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.418+0000 7fceb14ff700 1 -- start start 2023-03-03T20:08:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceb14ff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0c00e0 0x7fceac0c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:08:02.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceb14ff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceac0caa30 0x7fceac157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:08:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceb14ff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0cae00 0x7fceac1ed120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:08:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceb14ff700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac1dda40 con 0x7fceac0c00e0 2023-03-03T20:08:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceb14ff700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac158860 con 0x7fceac0caa30 2023-03-03T20:08:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceb14ff700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceac0c4760 con 0x7fceac0cae00 2023-03-03T20:08:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceaaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0c00e0 0x7fceac0c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:08:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceaaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0c00e0 0x7fceac0c26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44642/0 (socket says 172.21.15.7:44642) 2023-03-03T20:08:02.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceaaffd700 1 -- 172.21.15.7:0/2132562414 learned_addr learned my addr 172.21.15.7:0/2132562414 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:08:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceab7fe700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0cae00 0x7fceac1ed120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:08:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceaa7fc700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceac0caa30 0x7fceac157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:08:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceaaffd700 1 -- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0cae00 msgr2=0x7fceac1ed120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:08:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceaaffd700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0cae00 0x7fceac1ed120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:08:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceaaffd700 1 -- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceac0caa30 msgr2=0x7fceac157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:08:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceaaffd700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceac0caa30 0x7fceac157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:08:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceaaffd700 1 -- 172.21.15.7:0/2132562414 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c0094a0 con 0x7fceac0c00e0 2023-03-03T20:08:02.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceab7fe700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0cae00 0x7fceac1ed120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:08:02.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.419+0000 7fceaa7fc700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceac0caa30 0x7fceac157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:08:02.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.420+0000 7fceaaffd700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0c00e0 0x7fceac0c26d0 secure :-1 s=READY pgs=3710 cs=0 l=1 rev1=1 crypto rx=0x7fceac06b270 tx=0x7fce9c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:08:02.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.420+0000 7fce937fe700 1 -- 172.21.15.7:0/2132562414 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c0038a0 con 0x7fceac0c00e0 2023-03-03T20:08:02.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.420+0000 7fce937fe700 1 -- 172.21.15.7:0/2132562414 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c01f070 con 0x7fceac0c00e0 2023-03-03T20:08:02.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.420+0000 7fceb14ff700 1 -- 172.21.15.7:0/2132562414 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceac1547f0 con 0x7fceac0c00e0 2023-03-03T20:08:02.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.420+0000 7fce937fe700 1 -- 172.21.15.7:0/2132562414 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c003c20 con 0x7fceac0c00e0 2023-03-03T20:08:02.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.420+0000 7fceb14ff700 1 -- 172.21.15.7:0/2132562414 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceac154510 con 0x7fceac0c00e0 2023-03-03T20:08:02.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.421+0000 7fceb14ff700 1 -- 172.21.15.7:0/2132562414 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceac154ad0 con 0x7fceac0c00e0 2023-03-03T20:08:02.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.421+0000 7fce937fe700 1 -- 172.21.15.7:0/2132562414 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fce9c00f050 con 0x7fceac0c00e0 2023-03-03T20:08:02.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.422+0000 7fce937fe700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce980682f0 0x7fce9806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:08:02.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.422+0000 7fce937fe700 1 -- 172.21.15.7:0/2132562414 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fce9c021070 con 0x7fceac0c00e0 2023-03-03T20:08:02.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.422+0000 7fceaa7fc700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce980682f0 0x7fce9806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:08:02.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.422+0000 7fceaa7fc700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce980682f0 0x7fce9806a7a0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7fceac14de80 tx=0x7fcea0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:08:02.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.426+0000 7fce937fe700 1 -- 172.21.15.7:0/2132562414 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fce9c053e00 con 0x7fceac0c00e0 2023-03-03T20:08:02.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.664+0000 7fceb14ff700 1 -- 172.21.15.7:0/2132562414 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fceac0c2520 con 0x7fceac0c00e0 2023-03-03T20:08:02.667 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.665+0000 7fce937fe700 1 -- 172.21.15.7:0/2132562414 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fce9c057420 con 0x7fceac0c00e0 2023-03-03T20:08:02.667 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:08:02.667 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:08:02.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 -- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce980682f0 msgr2=0x7fce9806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:08:02.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce980682f0 0x7fce9806a7a0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7fceac14de80 tx=0x7fcea0008040 comp rx=0 tx=0).stop 2023-03-03T20:08:02.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 -- 172.21.15.7:0/2132562414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0c00e0 msgr2=0x7fceac0c26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:08:02.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0c00e0 0x7fceac0c26d0 secure :-1 s=READY pgs=3710 cs=0 l=1 rev1=1 crypto rx=0x7fceac06b270 tx=0x7fce9c004750 comp rx=0 tx=0).stop 2023-03-03T20:08:02.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 -- 172.21.15.7:0/2132562414 shutdown_connections 2023-03-03T20:08:02.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fceac0cae00 0x7fceac1ed120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:08:02.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fce980682f0 0x7fce9806a7a0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:08:02.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fceac0c00e0 0x7fceac0c26d0 unknown :-1 s=CLOSED pgs=3710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:08:02.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 --2- 172.21.15.7:0/2132562414 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fceac0caa30 0x7fceac157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:08:02.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 -- 172.21.15.7:0/2132562414 >> 172.21.15.7:0/2132562414 conn(0x7fceac066e40 msgr2=0x7fceac153c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:08:02.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 -- 172.21.15.7:0/2132562414 shutdown_connections 2023-03-03T20:08:02.670 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:08:02.667+0000 7fceb14ff700 1 -- 172.21.15.7:0/2132562414 wait complete. 2023-03-03T20:08:02.671 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:08:02.681 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2023-03-03T20:08:26.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:08:26.371 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:08:56.449 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:08:56.454 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:09:04.683 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:09:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.820+0000 7f7b14e8b700 1 Processor -- start 2023-03-03T20:09:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.820+0000 7f7b14e8b700 1 -- start start 2023-03-03T20:09:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.821+0000 7f7b14e8b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b100ac220 0x7f7b100736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.821+0000 7f7b14e8b700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7b10152b60 0x7f7b1015ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.821+0000 7f7b14e8b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b100ac900 con 0x7f7b10073c30 2023-03-03T20:09:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.821+0000 7f7b14e8b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b10071e30 con 0x7f7b100ac220 2023-03-03T20:09:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.821+0000 7f7b14e8b700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b100acb50 con 0x7f7b10152b60 2023-03-03T20:09:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.821+0000 7f7b0e59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b100ac220 0x7f7b100736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0e59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b100ac220 0x7f7b100736f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44682/0 (socket says 172.21.15.7:44682) 2023-03-03T20:09:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0e59c700 1 -- 172.21.15.7:0/195271030 learned_addr learned my addr 172.21.15.7:0/195271030 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:09:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0ed9d700 1 --2- 172.21.15.7:0/195271030 >> v2:172.21.15.143:3300/0 conn(0x7f7b10152b60 0x7f7b1015ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0e59c700 1 -- 172.21.15.7:0/195271030 >> v2:172.21.15.143:3300/0 conn(0x7f7b10152b60 msgr2=0x7f7b1015ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:09:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0e59c700 1 --2- 172.21.15.7:0/195271030 >> v2:172.21.15.143:3300/0 conn(0x7f7b10152b60 0x7f7b1015ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0e59c700 1 -- 172.21.15.7:0/195271030 >> v1:172.21.15.143:6789/0 conn(0x7f7b10073c30 legacy=0x7f7b10152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:09:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0e59c700 1 -- 172.21.15.7:0/195271030 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b101559c0 con 0x7f7b100ac220 2023-03-03T20:09:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0e59c700 1 --2- 172.21.15.7:0/195271030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 0x7f7b100736f0 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 crypto rx=0x7f7b10072070 tx=0x7f7b00009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f283212233dfc4b8 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:09:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0d59a700 1 -- 172.21.15.7:0/195271030 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0000b870 con 0x7f7b100ac220 2023-03-03T20:09:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0d59a700 1 -- 172.21.15.7:0/195271030 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0000b9d0 con 0x7f7b100ac220 2023-03-03T20:09:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.822+0000 7f7b0d59a700 1 -- 172.21.15.7:0/195271030 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0000bd60 con 0x7f7b100ac220 2023-03-03T20:09:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.823+0000 7f7b14e8b700 1 -- 172.21.15.7:0/195271030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 msgr2=0x7f7b100736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.823+0000 7f7b14e8b700 1 --2- 172.21.15.7:0/195271030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 0x7f7b100736f0 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 crypto rx=0x7f7b10072070 tx=0x7f7b00009bd0 comp rx=0 tx=0).stop 2023-03-03T20:09:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.823+0000 7f7b14e8b700 1 -- 172.21.15.7:0/195271030 shutdown_connections 2023-03-03T20:09:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.823+0000 7f7b14e8b700 1 --2- 172.21.15.7:0/195271030 >> v2:172.21.15.143:3300/0 conn(0x7f7b10152b60 0x7f7b1015ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.823+0000 7f7b14e8b700 1 --2- 172.21.15.7:0/195271030 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 0x7f7b100736f0 unknown :-1 s=CLOSED pgs=3711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.823+0000 7f7b14e8b700 1 -- 172.21.15.7:0/195271030 >> 172.21.15.7:0/195271030 conn(0x7f7b100a94d0 msgr2=0x7f7b100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:09:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.823+0000 7f7b14e8b700 1 -- 172.21.15.7:0/195271030 shutdown_connections 2023-03-03T20:09:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.823+0000 7f7b14e8b700 1 -- 172.21.15.7:0/195271030 wait complete. 2023-03-03T20:09:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.824+0000 7f7b14e8b700 1 Processor -- start 2023-03-03T20:09:04.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.824+0000 7f7b14e8b700 1 -- start start 2023-03-03T20:09:04.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b14e8b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b10073c30 0x7f7b100c10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:04.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b14e8b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 0x7f7b1014ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:04.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b14e8b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b10152b60 0x7f7b1014a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:04.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b14e8b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1014f660 con 0x7f7b100ac220 2023-03-03T20:09:04.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b14e8b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1014b930 con 0x7f7b10073c30 2023-03-03T20:09:04.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b14e8b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b100c1600 con 0x7f7b10152b60 2023-03-03T20:09:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b0dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 0x7f7b1014ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b0dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 0x7f7b1014ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44688/0 (socket says 172.21.15.7:44688) 2023-03-03T20:09:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b0dd9b700 1 -- 172.21.15.7:0/3780600709 learned_addr learned my addr 172.21.15.7:0/3780600709 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:09:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b0e59c700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b10073c30 0x7f7b100c10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b0ed9d700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b10152b60 0x7f7b1014a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b0dd9b700 1 -- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b10152b60 msgr2=0x7f7b1014a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.825+0000 7f7b0dd9b700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b10152b60 0x7f7b1014a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.826+0000 7f7b0dd9b700 1 -- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b10073c30 msgr2=0x7f7b100c10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.826+0000 7f7b0dd9b700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b10073c30 0x7f7b100c10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.826+0000 7f7b0dd9b700 1 -- 172.21.15.7:0/3780600709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00009550 con 0x7f7b100ac220 2023-03-03T20:09:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.826+0000 7f7b0dd9b700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 0x7f7b1014ea60 secure :-1 s=READY pgs=3712 cs=0 l=1 rev1=1 crypto rx=0x7f7af800c450 tx=0x7f7af800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:09:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.826+0000 7f7b06ffd700 1 -- 172.21.15.7:0/3780600709 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af80052d0 con 0x7f7b100ac220 2023-03-03T20:09:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.826+0000 7f7b06ffd700 1 -- 172.21.15.7:0/3780600709 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af8005430 con 0x7f7b100ac220 2023-03-03T20:09:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.826+0000 7f7b14e8b700 1 -- 172.21.15.7:0/3780600709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b10155f80 con 0x7f7b100ac220 2023-03-03T20:09:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.826+0000 7f7b06ffd700 1 -- 172.21.15.7:0/3780600709 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af80038c0 con 0x7f7b100ac220 2023-03-03T20:09:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.826+0000 7f7b14e8b700 1 -- 172.21.15.7:0/3780600709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b10155ca0 con 0x7f7b100ac220 2023-03-03T20:09:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.827+0000 7f7b14e8b700 1 -- 172.21.15.7:0/3780600709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b10156260 con 0x7f7b100ac220 2023-03-03T20:09:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.827+0000 7f7b06ffd700 1 -- 172.21.15.7:0/3780600709 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7af8019030 con 0x7f7b100ac220 2023-03-03T20:09:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.828+0000 7f7b06ffd700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af00682f0 0x7f7af006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.828+0000 7f7b06ffd700 1 -- 172.21.15.7:0/3780600709 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7af80852f0 con 0x7f7b100ac220 2023-03-03T20:09:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.828+0000 7f7b0e59c700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af00682f0 0x7f7af006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.828+0000 7f7b0e59c700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af00682f0 0x7f7af006a7a0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f7b00000c00 tx=0x7f7b00008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:09:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:04.832+0000 7f7b06ffd700 1 -- 172.21.15.7:0/3780600709 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7af8051690 con 0x7f7b100ac220 2023-03-03T20:09:05.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.069+0000 7f7b14e8b700 1 -- 172.21.15.7:0/3780600709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b10074bb0 con 0x7f7b100ac220 2023-03-03T20:09:05.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.070+0000 7f7b06ffd700 1 -- 172.21.15.7:0/3780600709 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7af800ce60 con 0x7f7b100ac220 2023-03-03T20:09:05.072 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:09:05.072 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:09:05.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 -- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af00682f0 msgr2=0x7f7af006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:05.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af00682f0 0x7f7af006a7a0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f7b00000c00 tx=0x7f7b00008030 comp rx=0 tx=0).stop 2023-03-03T20:09:05.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 -- 172.21.15.7:0/3780600709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 msgr2=0x7f7b1014ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:05.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 0x7f7b1014ea60 secure :-1 s=READY pgs=3712 cs=0 l=1 rev1=1 crypto rx=0x7f7af800c450 tx=0x7f7af800c760 comp rx=0 tx=0).stop 2023-03-03T20:09:05.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 -- 172.21.15.7:0/3780600709 shutdown_connections 2023-03-03T20:09:05.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7b10152b60 0x7f7b1014a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:05.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7b10073c30 0x7f7b100c10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7af00682f0 0x7f7af006a7a0 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 --2- 172.21.15.7:0/3780600709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b100ac220 0x7f7b1014ea60 unknown :-1 s=CLOSED pgs=3712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 -- 172.21.15.7:0/3780600709 >> 172.21.15.7:0/3780600709 conn(0x7f7b100a94d0 msgr2=0x7f7b10070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:09:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 -- 172.21.15.7:0/3780600709 shutdown_connections 2023-03-03T20:09:05.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:05.072+0000 7f7b14e8b700 1 -- 172.21.15.7:0/3780600709 wait complete. 2023-03-03T20:09:05.076 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:09:05.086 INFO:tasks.mds_thrash.fs.[b]:waiting for 42 secs before thrashing 2023-03-03T20:09:09.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:09:09.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.515+0000 7fef796ac700 1 Processor -- start 2023-03-03T20:09:09.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.515+0000 7fef796ac700 1 -- start start 2023-03-03T20:09:09.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.516+0000 7fef796ac700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef7406c910 0x7fef74073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:09.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.516+0000 7fef796ac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406bea0 0x7fef74158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:09.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.516+0000 7fef796ac700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7406c3e0 con 0x7fef740736f0 2023-03-03T20:09:09.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.516+0000 7fef796ac700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef74158960 con 0x7fef7406c910 2023-03-03T20:09:09.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.516+0000 7fef796ac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef740c32e0 con 0x7fef7406bea0 2023-03-03T20:09:09.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef737fe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406bea0 0x7fef74158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:09.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef727fc700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fef7406c910 0x7fef74073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:09.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef72ffd700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fef740736f0 0x7fef74157d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:42198/0 (socket says 172.21.15.7:42198) 2023-03-03T20:09:09.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef72ffd700 1 -- 172.21.15.7:0/3571367652 learned_addr learned my addr 172.21.15.7:0/3571367652 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:09:09.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef737fe700 1 -- 172.21.15.7:0/3571367652 >> v2:172.21.15.143:3300/0 conn(0x7fef7406c910 msgr2=0x7fef74073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:09:09.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef737fe700 1 --2- 172.21.15.7:0/3571367652 >> v2:172.21.15.143:3300/0 conn(0x7fef7406c910 0x7fef74073ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:09.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef737fe700 1 -- 172.21.15.7:0/3571367652 >> v1:172.21.15.143:6789/0 conn(0x7fef740736f0 legacy=0x7fef74157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:09.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef737fe700 1 -- 172.21.15.7:0/3571367652 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7406c630 con 0x7fef7406bea0 2023-03-03T20:09:09.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef727fc700 1 --2- 172.21.15.7:0/3571367652 >> v2:172.21.15.143:3300/0 conn(0x7fef7406c910 0x7fef74073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:09:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef737fe700 1 --2- 172.21.15.7:0/3571367652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406bea0 0x7fef74158420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fef6c009d90 tx=0x7fef6c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31c5acfa8aa8e643 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:09:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef71ffb700 1 -- 172.21.15.7:0/3571367652 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c00e070 con 0x7fef7406bea0 2023-03-03T20:09:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef71ffb700 1 -- 172.21.15.7:0/3571367652 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c00bb20 con 0x7fef7406bea0 2023-03-03T20:09:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.517+0000 7fef71ffb700 1 -- 172.21.15.7:0/3571367652 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c012430 con 0x7fef7406bea0 2023-03-03T20:09:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.518+0000 7fef796ac700 1 -- 172.21.15.7:0/3571367652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406bea0 msgr2=0x7fef74158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.518+0000 7fef796ac700 1 --2- 172.21.15.7:0/3571367652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406bea0 0x7fef74158420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fef6c009d90 tx=0x7fef6c00b330 comp rx=0 tx=0).stop 2023-03-03T20:09:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.518+0000 7fef796ac700 1 -- 172.21.15.7:0/3571367652 shutdown_connections 2023-03-03T20:09:09.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.518+0000 7fef796ac700 1 --2- 172.21.15.7:0/3571367652 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406bea0 0x7fef74158420 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.518+0000 7fef796ac700 1 --2- 172.21.15.7:0/3571367652 >> v2:172.21.15.143:3300/0 conn(0x7fef7406c910 0x7fef74073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.518+0000 7fef796ac700 1 -- 172.21.15.7:0/3571367652 >> 172.21.15.7:0/3571367652 conn(0x7fef740a94d0 msgr2=0x7fef740a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:09:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.518+0000 7fef796ac700 1 -- 172.21.15.7:0/3571367652 shutdown_connections 2023-03-03T20:09:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.518+0000 7fef796ac700 1 -- 172.21.15.7:0/3571367652 wait complete. 2023-03-03T20:09:09.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.519+0000 7fef796ac700 1 Processor -- start 2023-03-03T20:09:09.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.519+0000 7fef796ac700 1 -- start start 2023-03-03T20:09:09.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef796ac700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef7406bea0 0x7fef74150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:09.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef796ac700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406c910 0x7fef740c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:09.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef796ac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef740736f0 0x7fef741df130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:09.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef796ac700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef741505d0 con 0x7fef740736f0 2023-03-03T20:09:09.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef796ac700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef74149a30 con 0x7fef7406bea0 2023-03-03T20:09:09.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef796ac700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef741ed4d0 con 0x7fef7406c910 2023-03-03T20:09:09.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef737fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef740736f0 0x7fef741df130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:09.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef737fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef740736f0 0x7fef741df130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44708/0 (socket says 172.21.15.7:44708) 2023-03-03T20:09:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef737fe700 1 -- 172.21.15.7:0/690494864 learned_addr learned my addr 172.21.15.7:0/690494864 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:09:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef727fc700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406c910 0x7fef740c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:09.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.520+0000 7fef72ffd700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef7406bea0 0x7fef74150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef737fe700 1 -- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406c910 msgr2=0x7fef740c3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef737fe700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406c910 0x7fef740c3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef737fe700 1 -- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef7406bea0 msgr2=0x7fef74150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:09.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef737fe700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef7406bea0 0x7fef74150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef737fe700 1 -- 172.21.15.7:0/690494864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef5c0094a0 con 0x7fef740736f0 2023-03-03T20:09:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef727fc700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406c910 0x7fef740c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:09:09.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef737fe700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef740736f0 0x7fef741df130 secure :-1 s=READY pgs=3713 cs=0 l=1 rev1=1 crypto rx=0x7fef6c014040 tx=0x7fef6c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:09:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef6b7fe700 1 -- 172.21.15.7:0/690494864 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c012430 con 0x7fef740736f0 2023-03-03T20:09:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef6b7fe700 1 -- 172.21.15.7:0/690494864 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c020070 con 0x7fef740736f0 2023-03-03T20:09:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef796ac700 1 -- 172.21.15.7:0/690494864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6c009a70 con 0x7fef740736f0 2023-03-03T20:09:09.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef6b7fe700 1 -- 172.21.15.7:0/690494864 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c003ea0 con 0x7fef740736f0 2023-03-03T20:09:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.521+0000 7fef796ac700 1 -- 172.21.15.7:0/690494864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef740c3b60 con 0x7fef740736f0 2023-03-03T20:09:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.522+0000 7fef796ac700 1 -- 172.21.15.7:0/690494864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef740c3580 con 0x7fef740736f0 2023-03-03T20:09:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.523+0000 7fef6b7fe700 1 -- 172.21.15.7:0/690494864 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fef6c02e070 con 0x7fef740736f0 2023-03-03T20:09:09.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.523+0000 7fef6b7fe700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef540682f0 0x7fef5406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:09.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.523+0000 7fef6b7fe700 1 -- 172.21.15.7:0/690494864 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fef6c022070 con 0x7fef740736f0 2023-03-03T20:09:09.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.523+0000 7fef72ffd700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef540682f0 0x7fef5406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:09.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.524+0000 7fef72ffd700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef540682f0 0x7fef5406a7a0 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7fef64005910 tx=0x7fef64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:09:09.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.527+0000 7fef6b7fe700 1 -- 172.21.15.7:0/690494864 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fef6c054280 con 0x7fef740736f0 2023-03-03T20:09:09.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.764+0000 7fef796ac700 1 -- 172.21.15.7:0/690494864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef74074bb0 con 0x7fef740736f0 2023-03-03T20:09:09.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.765+0000 7fef6b7fe700 1 -- 172.21.15.7:0/690494864 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fef6c02c090 con 0x7fef740736f0 2023-03-03T20:09:09.767 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:09:09.767 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:09:09.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.766+0000 7fef796ac700 1 -- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef540682f0 msgr2=0x7fef5406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:09.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.766+0000 7fef796ac700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef540682f0 0x7fef5406a7a0 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7fef64005910 tx=0x7fef64009040 comp rx=0 tx=0).stop 2023-03-03T20:09:09.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.766+0000 7fef796ac700 1 -- 172.21.15.7:0/690494864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef740736f0 msgr2=0x7fef741df130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:09.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.766+0000 7fef796ac700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef740736f0 0x7fef741df130 secure :-1 s=READY pgs=3713 cs=0 l=1 rev1=1 crypto rx=0x7fef6c014040 tx=0x7fef6c004ad0 comp rx=0 tx=0).stop 2023-03-03T20:09:09.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.766+0000 7fef796ac700 1 -- 172.21.15.7:0/690494864 shutdown_connections 2023-03-03T20:09:09.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.766+0000 7fef796ac700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fef540682f0 0x7fef5406a7a0 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:09.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.766+0000 7fef796ac700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fef740736f0 0x7fef741df130 unknown :-1 s=CLOSED pgs=3713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:09.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.766+0000 7fef796ac700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef7406bea0 0x7fef74150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:09.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.766+0000 7fef796ac700 1 --2- 172.21.15.7:0/690494864 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fef7406c910 0x7fef740c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:09.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.767+0000 7fef796ac700 1 -- 172.21.15.7:0/690494864 >> 172.21.15.7:0/690494864 conn(0x7fef740a94d0 msgr2=0x7fef740c28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:09:09.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.767+0000 7fef796ac700 1 -- 172.21.15.7:0/690494864 shutdown_connections 2023-03-03T20:09:09.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:09.767+0000 7fef796ac700 1 -- 172.21.15.7:0/690494864 wait complete. 2023-03-03T20:09:09.770 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:09:09.781 INFO:tasks.mds_thrash.fs.[a]:waiting for 86 secs before thrashing 2023-03-03T20:09:26.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:09:26.600 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:09:47.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:09:47.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.226+0000 7f5b70760700 1 Processor -- start 2023-03-03T20:09:47.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.226+0000 7f5b70760700 1 -- start start 2023-03-03T20:09:47.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b70760700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5b6806d870 0x7f5b681550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:47.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b70760700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b680c7a60 con 0x7f5b680c3e30 2023-03-03T20:09:47.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b70760700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b680c3620 con 0x7f5b680c1da0 2023-03-03T20:09:47.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b70760700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b680c2af0 con 0x7f5b6806d870 2023-03-03T20:09:47.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b6e4fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5b6806d870 0x7f5b681550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b6ecfd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5b680c3e30 0x7f5b6815ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38724/0 (socket says 172.21.15.7:38724) 2023-03-03T20:09:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b6e4fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5b6806d870 0x7f5b681550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44740/0 (socket says 172.21.15.7:44740) 2023-03-03T20:09:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b6ecfd700 1 -- 172.21.15.7:0/618812788 learned_addr learned my addr 172.21.15.7:0/618812788 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:09:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b6e4fc700 1 -- 172.21.15.7:0/618812788 >> v1:172.21.15.143:6789/0 conn(0x7f5b680c1da0 legacy=0x7f5b680c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b6e4fc700 1 -- 172.21.15.7:0/618812788 >> v1:172.21.15.7:6789/0 conn(0x7f5b680c3e30 legacy=0x7f5b6815ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.227+0000 7f5b6e4fc700 1 -- 172.21.15.7:0/618812788 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b681545e0 con 0x7f5b6806d870 2023-03-03T20:09:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b6e4fc700 1 --2- 172.21.15.7:0/618812788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b6806d870 0x7f5b681550c0 secure :-1 s=READY pgs=3715 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009840 tx=0x7f5b58009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347927b62a5855bb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:09:47.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b6d4fa700 1 -- 172.21.15.7:0/618812788 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58011830 con 0x7f5b6806d870 2023-03-03T20:09:47.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b6d4fa700 1 -- 172.21.15.7:0/618812788 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b58011990 con 0x7f5b6806d870 2023-03-03T20:09:47.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b6d4fa700 1 -- 172.21.15.7:0/618812788 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58011d20 con 0x7f5b6806d870 2023-03-03T20:09:47.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b70760700 1 -- 172.21.15.7:0/618812788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b6806d870 msgr2=0x7f5b681550c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:47.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b70760700 1 --2- 172.21.15.7:0/618812788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b6806d870 0x7f5b681550c0 secure :-1 s=READY pgs=3715 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009840 tx=0x7f5b58009b50 comp rx=0 tx=0).stop 2023-03-03T20:09:47.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b70760700 1 -- 172.21.15.7:0/618812788 shutdown_connections 2023-03-03T20:09:47.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b70760700 1 --2- 172.21.15.7:0/618812788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b6806d870 0x7f5b681550c0 unknown :-1 s=CLOSED pgs=3715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:47.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b70760700 1 -- 172.21.15.7:0/618812788 >> 172.21.15.7:0/618812788 conn(0x7f5b6806c110 msgr2=0x7f5b6806c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:09:47.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b70760700 1 -- 172.21.15.7:0/618812788 shutdown_connections 2023-03-03T20:09:47.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.228+0000 7f5b70760700 1 -- 172.21.15.7:0/618812788 wait complete. 2023-03-03T20:09:47.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.230+0000 7f5b70760700 1 Processor -- start 2023-03-03T20:09:47.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.230+0000 7f5b70760700 1 -- start start 2023-03-03T20:09:47.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.230+0000 7f5b70760700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b6806d870 0x7f5b680c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:47.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.230+0000 7f5b70760700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b680c1da0 0x7f5b680ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:47.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.230+0000 7f5b70760700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b680c3e30 0x7f5b6813cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:47.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.230+0000 7f5b70760700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b68156710 con 0x7f5b680c3e30 2023-03-03T20:09:47.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.230+0000 7f5b70760700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b68157030 con 0x7f5b6806d870 2023-03-03T20:09:47.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.230+0000 7f5b70760700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b681569f0 con 0x7f5b680c1da0 2023-03-03T20:09:47.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6ecfd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b680c3e30 0x7f5b6813cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:47.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6ecfd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b680c3e30 0x7f5b6813cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44750/0 (socket says 172.21.15.7:44750) 2023-03-03T20:09:47.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6ecfd700 1 -- 172.21.15.7:0/595111068 learned_addr learned my addr 172.21.15.7:0/595111068 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:09:47.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6dcfb700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b680c1da0 0x7f5b680ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:47.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6e4fc700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b6806d870 0x7f5b680c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:47.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6ecfd700 1 -- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b680c1da0 msgr2=0x7f5b680ac9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:47.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6ecfd700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b680c1da0 0x7f5b680ac9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:47.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6ecfd700 1 -- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b6806d870 msgr2=0x7f5b680c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:47.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6ecfd700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b6806d870 0x7f5b680c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:47.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6ecfd700 1 -- 172.21.15.7:0/595111068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b580094a0 con 0x7f5b680c3e30 2023-03-03T20:09:47.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b6ecfd700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b680c3e30 0x7f5b6813cd20 secure :-1 s=READY pgs=3716 cs=0 l=1 rev1=1 crypto rx=0x7f5b6000cbf0 tx=0x7f5b600021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:09:47.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b5effd700 1 -- 172.21.15.7:0/595111068 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6000ab40 con 0x7f5b680c3e30 2023-03-03T20:09:47.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b5effd700 1 -- 172.21.15.7:0/595111068 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6000aca0 con 0x7f5b680c3e30 2023-03-03T20:09:47.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b5effd700 1 -- 172.21.15.7:0/595111068 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60012600 con 0x7f5b680c3e30 2023-03-03T20:09:47.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b70760700 1 -- 172.21.15.7:0/595111068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b68154ba0 con 0x7f5b680c3e30 2023-03-03T20:09:47.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.231+0000 7f5b70760700 1 -- 172.21.15.7:0/595111068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b681548c0 con 0x7f5b680c3e30 2023-03-03T20:09:47.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.232+0000 7f5b70760700 1 -- 172.21.15.7:0/595111068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6806dc40 con 0x7f5b680c3e30 2023-03-03T20:09:47.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.233+0000 7f5b5effd700 1 -- 172.21.15.7:0/595111068 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5b60019030 con 0x7f5b680c3e30 2023-03-03T20:09:47.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.233+0000 7f5b5effd700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b4c0682f0 0x7f5b4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:09:47.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.233+0000 7f5b5effd700 1 -- 172.21.15.7:0/595111068 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5b60085c00 con 0x7f5b680c3e30 2023-03-03T20:09:47.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.233+0000 7f5b6e4fc700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b4c0682f0 0x7f5b4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:09:47.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.234+0000 7f5b6e4fc700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b4c0682f0 0x7f5b4c06a7a0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f5b580090b0 tx=0x7f5b5801c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:09:47.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.238+0000 7f5b5effd700 1 -- 172.21.15.7:0/595111068 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5b6004b5e0 con 0x7f5b680c3e30 2023-03-03T20:09:47.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.477+0000 7f5b70760700 1 -- 172.21.15.7:0/595111068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b68156850 con 0x7f5b680c3e30 2023-03-03T20:09:47.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.477+0000 7f5b5effd700 1 -- 172.21.15.7:0/595111068 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5b68156850 con 0x7f5b680c3e30 2023-03-03T20:09:47.478 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:09:47.479 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:09:47.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 -- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b4c0682f0 msgr2=0x7f5b4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:47.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b4c0682f0 0x7f5b4c06a7a0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f5b580090b0 tx=0x7f5b5801c040 comp rx=0 tx=0).stop 2023-03-03T20:09:47.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 -- 172.21.15.7:0/595111068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b680c3e30 msgr2=0x7f5b6813cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:09:47.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b680c3e30 0x7f5b6813cd20 secure :-1 s=READY pgs=3716 cs=0 l=1 rev1=1 crypto rx=0x7f5b6000cbf0 tx=0x7f5b600021c0 comp rx=0 tx=0).stop 2023-03-03T20:09:47.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 -- 172.21.15.7:0/595111068 shutdown_connections 2023-03-03T20:09:47.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5b4c0682f0 0x7f5b4c06a7a0 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:47.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b680c3e30 0x7f5b6813cd20 unknown :-1 s=CLOSED pgs=3716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:47.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5b6806d870 0x7f5b680c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:47.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 --2- 172.21.15.7:0/595111068 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5b680c1da0 0x7f5b680ac9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:09:47.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 -- 172.21.15.7:0/595111068 >> 172.21.15.7:0/595111068 conn(0x7f5b6806c110 msgr2=0x7f5b680c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:09:47.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 -- 172.21.15.7:0/595111068 shutdown_connections 2023-03-03T20:09:47.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:09:47.479+0000 7f5b70760700 1 -- 172.21.15.7:0/595111068 wait complete. 2023-03-03T20:09:47.482 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:09:47.493 INFO:tasks.mds_thrash.fs.[b]:waiting for 111 secs before thrashing 2023-03-03T20:09:56.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:09:56.745 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:10:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T20:09:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T20:10:26.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:10:26.832 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:10:35.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:10:35.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.919+0000 7fbdd698f700 1 Processor -- start 2023-03-03T20:10:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.919+0000 7fbdd698f700 1 -- start start 2023-03-03T20:10:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdd698f700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fbdd0072250 0x7fbdd014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:10:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdd698f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd01510d0 con 0x7fbdd0071e80 2023-03-03T20:10:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdd698f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd00c3740 con 0x7fbdd00c1ce0 2023-03-03T20:10:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdd698f700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd0070670 con 0x7fbdd0072250 2023-03-03T20:10:35.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdcffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbdd0071e80 0x7fbdd0151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38780/0 (socket says 172.21.15.7:38780) 2023-03-03T20:10:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdcffff700 1 -- 172.21.15.7:0/933651517 learned_addr learned my addr 172.21.15.7:0/933651517 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:10:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260345589 0 0) 0x7fbdd01510d0 con 0x7fbdd0071e80 2023-03-03T20:10:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdcf7fe700 1 --2- 172.21.15.7:0/933651517 >> v2:172.21.15.143:3300/0 conn(0x7fbdd0072250 0x7fbdd014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:10:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdbc0035b0 con 0x7fbdd0071e80 2023-03-03T20:10:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3144126552 0 0) 0x7fbdbc0035b0 con 0x7fbdd0071e80 2023-03-03T20:10:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 >> v2:172.21.15.143:3300/0 conn(0x7fbdd0072250 msgr2=0x7fbdd014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:10:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdceffd700 1 --2- 172.21.15.7:0/933651517 >> v2:172.21.15.143:3300/0 conn(0x7fbdd0072250 0x7fbdd014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:10:35.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.920+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 >> v1:172.21.15.143:6789/0 conn(0x7fbdd00c1ce0 legacy=0x7fbdd015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:10:35.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd00c39e0 con 0x7fbdd0071e80 2023-03-03T20:10:35.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fbdc0001670 con 0x7fbdd0071e80 2023-03-03T20:10:35.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 >> v1:172.21.15.7:6789/0 conn(0x7fbdd0071e80 legacy=0x7fbdd0151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:10:35.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 --2- 172.21.15.7:0/933651517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc003a70 0x7fbdbc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:10:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 --2- 172.21.15.7:0/933651517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc004850 0x7fbdbc004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:10:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 --2- 172.21.15.7:0/933651517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc005300 0x7fbdbc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:10:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0070670 con 0x7fbdbc005300 2023-03-03T20:10:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc00c230 con 0x7fbdbc003a70 2023-03-03T20:10:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc00c3f0 con 0x7fbdbc004850 2023-03-03T20:10:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdd4f2c700 1 --2- 172.21.15.7:0/933651517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc005300 0x7fbdbc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:10:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdcf7fe700 1 --2- 172.21.15.7:0/933651517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc004850 0x7fbdbc004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:10:35.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdd4f2c700 1 -- 172.21.15.7:0/933651517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc004850 msgr2=0x7fbdbc004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:10:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdd4f2c700 1 --2- 172.21.15.7:0/933651517 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdbc004850 0x7fbdbc004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:10:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdd4f2c700 1 -- 172.21.15.7:0/933651517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc003a70 msgr2=0x7fbdbc003e80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-03T20:10:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdd4f2c700 1 --2- 172.21.15.7:0/933651517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc003a70 0x7fbdbc003e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:10:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdd4f2c700 1 -- 172.21.15.7:0/933651517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc0035b0 con 0x7fbdbc005300 2023-03-03T20:10:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdd4f2c700 1 -- 172.21.15.7:0/933651517 reap_dead start 2023-03-03T20:10:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdd4f2c700 1 --2- 172.21.15.7:0/933651517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc005300 0x7fbdbc00ba10 secure :-1 s=READY pgs=3718 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c160 tx=0x7fbdbc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2aadb4d939ed15c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:10:35.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8001db0 con 0x7fbdbc005300 2023-03-03T20:10:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc800cca0 con 0x7fbdbc005300 2023-03-03T20:10:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.921+0000 7fbdceffd700 1 -- 172.21.15.7:0/933651517 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8003c40 con 0x7fbdbc005300 2023-03-03T20:10:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.922+0000 7fbdd698f700 1 -- 172.21.15.7:0/933651517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc005300 msgr2=0x7fbdbc00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:10:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.922+0000 7fbdd698f700 1 --2- 172.21.15.7:0/933651517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc005300 0x7fbdbc00ba10 secure :-1 s=READY pgs=3718 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c160 tx=0x7fbdbc005880 comp rx=0 tx=0).stop 2023-03-03T20:10:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.922+0000 7fbdd698f700 1 -- 172.21.15.7:0/933651517 shutdown_connections 2023-03-03T20:10:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.922+0000 7fbdd698f700 1 --2- 172.21.15.7:0/933651517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdbc005300 0x7fbdbc00ba10 unknown :-1 s=CLOSED pgs=3718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:10:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.922+0000 7fbdd698f700 1 -- 172.21.15.7:0/933651517 >> 172.21.15.7:0/933651517 conn(0x7fbdd00a94d0 msgr2=0x7fbdd00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:10:35.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.922+0000 7fbdd698f700 1 -- 172.21.15.7:0/933651517 shutdown_connections 2023-03-03T20:10:35.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.922+0000 7fbdd698f700 1 -- 172.21.15.7:0/933651517 wait complete. 2023-03-03T20:10:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.923+0000 7fbdd698f700 1 Processor -- start 2023-03-03T20:10:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.923+0000 7fbdd698f700 1 -- start start 2023-03-03T20:10:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd698f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc005300 0x7fbdd01435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:10:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd698f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd0150df0 0x7fbdd0145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:10:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd698f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd0149f40 0x7fbdd014a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:10:35.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd698f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00aa1d0 con 0x7fbdd0149f40 2023-03-03T20:10:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd698f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd01eac80 con 0x7fbdbc005300 2023-03-03T20:10:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd698f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00aa490 con 0x7fbdd0150df0 2023-03-03T20:10:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd4f2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd0149f40 0x7fbdd014a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:10:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd4f2c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd0149f40 0x7fbdd014a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44814/0 (socket says 172.21.15.7:44814) 2023-03-03T20:10:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd4f2c700 1 -- 172.21.15.7:0/1069011690 learned_addr learned my addr 172.21.15.7:0/1069011690 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:10:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdcffff700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc005300 0x7fbdd01435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:10:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdcf7fe700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd0150df0 0x7fbdd0145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:10:35.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd4f2c700 1 -- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd0150df0 msgr2=0x7fbdd0145a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:10:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd4f2c700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd0150df0 0x7fbdd0145a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:10:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd4f2c700 1 -- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc005300 msgr2=0x7fbdd01435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:10:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd4f2c700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc005300 0x7fbdd01435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:10:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.924+0000 7fbdd4f2c700 1 -- 172.21.15.7:0/1069011690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd006c9a0 con 0x7fbdd0149f40 2023-03-03T20:10:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.925+0000 7fbdd4f2c700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd0149f40 0x7fbdd014a3e0 secure :-1 s=READY pgs=3719 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0072670 tx=0x7fbdc80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:10:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.925+0000 7fbdccff9700 1 -- 172.21.15.7:0/1069011690 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc80050d0 con 0x7fbdd0149f40 2023-03-03T20:10:35.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.925+0000 7fbdccff9700 1 -- 172.21.15.7:0/1069011690 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc8005250 con 0x7fbdd0149f40 2023-03-03T20:10:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.925+0000 7fbdd698f700 1 -- 172.21.15.7:0/1069011690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd0151300 con 0x7fbdd0149f40 2023-03-03T20:10:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.925+0000 7fbdccff9700 1 -- 172.21.15.7:0/1069011690 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8005600 con 0x7fbdd0149f40 2023-03-03T20:10:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.925+0000 7fbdd698f700 1 -- 172.21.15.7:0/1069011690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd006cb30 con 0x7fbdd0149f40 2023-03-03T20:10:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.926+0000 7fbdd698f700 1 -- 172.21.15.7:0/1069011690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd0074bb0 con 0x7fbdd0149f40 2023-03-03T20:10:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.926+0000 7fbdccff9700 1 -- 172.21.15.7:0/1069011690 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbdc8010050 con 0x7fbdd0149f40 2023-03-03T20:10:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.926+0000 7fbdccff9700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb0068320 0x7fbdb006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:10:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.927+0000 7fbdccff9700 1 -- 172.21.15.7:0/1069011690 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fbdc8087e30 con 0x7fbdd0149f40 2023-03-03T20:10:35.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.927+0000 7fbdcffff700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb0068320 0x7fbdb006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:10:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.927+0000 7fbdcffff700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb0068320 0x7fbdb006a7d0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0000c00 tx=0x7fbdc0001730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:10:35.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:35.931+0000 7fbdccff9700 1 -- 172.21.15.7:0/1069011690 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fbdc8054170 con 0x7fbdd0149f40 2023-03-03T20:10:36.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.168+0000 7fbdd698f700 1 -- 172.21.15.7:0/1069011690 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdd00c1560 con 0x7fbdd0149f40 2023-03-03T20:10:36.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.168+0000 7fbdccff9700 1 -- 172.21.15.7:0/1069011690 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fbdc8057790 con 0x7fbdd0149f40 2023-03-03T20:10:36.169 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:10:36.170 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:10:36.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 -- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb0068320 msgr2=0x7fbdb006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:10:36.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb0068320 0x7fbdb006a7d0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0000c00 tx=0x7fbdc0001730 comp rx=0 tx=0).stop 2023-03-03T20:10:36.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 -- 172.21.15.7:0/1069011690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd0149f40 msgr2=0x7fbdd014a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:10:36.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd0149f40 0x7fbdd014a3e0 secure :-1 s=READY pgs=3719 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0072670 tx=0x7fbdc80014a0 comp rx=0 tx=0).stop 2023-03-03T20:10:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 -- 172.21.15.7:0/1069011690 shutdown_connections 2023-03-03T20:10:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fbdb0068320 0x7fbdb006a7d0 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:10:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbdd0149f40 0x7fbdd014a3e0 unknown :-1 s=CLOSED pgs=3719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:10:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdbc005300 0x7fbdd01435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:10:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 --2- 172.21.15.7:0/1069011690 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fbdd0150df0 0x7fbdd0145a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:10:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 -- 172.21.15.7:0/1069011690 >> 172.21.15.7:0/1069011690 conn(0x7fbdd00a94d0 msgr2=0x7fbdd014dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:10:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 -- 172.21.15.7:0/1069011690 shutdown_connections 2023-03-03T20:10:36.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:10:36.170+0000 7fbdd698f700 1 -- 172.21.15.7:0/1069011690 wait complete. 2023-03-03T20:10:36.173 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:10:36.183 INFO:tasks.mds_thrash.fs.[a]:waiting for 39 secs before thrashing 2023-03-03T20:10:56.964 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:10:56.968 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:11:15.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:11:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.322+0000 7f3d4d26b700 1 Processor -- start 2023-03-03T20:11:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.322+0000 7f3d4d26b700 1 -- start start 2023-03-03T20:11:15.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.323+0000 7f3d4d26b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4814f9d0 0x7f3d48156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:15.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.323+0000 7f3d4d26b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d480abe20 0x7f3d481529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:15.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.323+0000 7f3d4d26b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d480c2f30 con 0x7f3d4814fda0 2023-03-03T20:11:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.323+0000 7f3d4d26b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d480c31d0 con 0x7f3d480abe20 2023-03-03T20:11:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.323+0000 7f3d4d26b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d481524a0 con 0x7f3d4814f9d0 2023-03-03T20:11:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.323+0000 7f3d477fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d480abe20 0x7f3d481529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.323+0000 7f3d477fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3d480abe20 0x7f3d481529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44834/0 (socket says 172.21.15.7:44834) 2023-03-03T20:11:15.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.323+0000 7f3d477fe700 1 -- 172.21.15.7:0/1643250315 learned_addr learned my addr 172.21.15.7:0/1643250315 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:11:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.323+0000 7f3d46ffd700 1 --2- 172.21.15.7:0/1643250315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4814f9d0 0x7f3d48156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d477fe700 1 -- 172.21.15.7:0/1643250315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4814f9d0 msgr2=0x7f3d48156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:11:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d477fe700 1 --2- 172.21.15.7:0/1643250315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4814f9d0 0x7f3d48156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d477fe700 1 -- 172.21.15.7:0/1643250315 >> v1:172.21.15.143:6789/0 conn(0x7f3d4814fda0 legacy=0x7f3d48071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:15.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d477fe700 1 -- 172.21.15.7:0/1643250315 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d480c34b0 con 0x7f3d480abe20 2023-03-03T20:11:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d477fe700 1 --2- 172.21.15.7:0/1643250315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 0x7f3d481529c0 secure :-1 s=READY pgs=3720 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009e50 tx=0x7f3d4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04b04a079108bfb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:11:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d45ffb700 1 -- 172.21.15.7:0/1643250315 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4000e070 con 0x7f3d480abe20 2023-03-03T20:11:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d45ffb700 1 -- 172.21.15.7:0/1643250315 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4000b9c0 con 0x7f3d480abe20 2023-03-03T20:11:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d45ffb700 1 -- 172.21.15.7:0/1643250315 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4000bd70 con 0x7f3d480abe20 2023-03-03T20:11:15.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d4d26b700 1 -- 172.21.15.7:0/1643250315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 msgr2=0x7f3d481529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d4d26b700 1 --2- 172.21.15.7:0/1643250315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 0x7f3d481529c0 secure :-1 s=READY pgs=3720 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009e50 tx=0x7f3d4000b330 comp rx=0 tx=0).stop 2023-03-03T20:11:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d4d26b700 1 -- 172.21.15.7:0/1643250315 shutdown_connections 2023-03-03T20:11:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d4d26b700 1 --2- 172.21.15.7:0/1643250315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 0x7f3d481529c0 unknown :-1 s=CLOSED pgs=3720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.324+0000 7f3d4d26b700 1 --2- 172.21.15.7:0/1643250315 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4814f9d0 0x7f3d48156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.325+0000 7f3d4d26b700 1 -- 172.21.15.7:0/1643250315 >> 172.21.15.7:0/1643250315 conn(0x7f3d480c8900 msgr2=0x7f3d480c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:11:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.325+0000 7f3d4d26b700 1 -- 172.21.15.7:0/1643250315 shutdown_connections 2023-03-03T20:11:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.325+0000 7f3d4d26b700 1 -- 172.21.15.7:0/1643250315 wait complete. 2023-03-03T20:11:15.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.326+0000 7f3d4d26b700 1 Processor -- start 2023-03-03T20:11:15.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.326+0000 7f3d4d26b700 1 -- start start 2023-03-03T20:11:15.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d4d26b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 0x7f3d4806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:15.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d4d26b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4814f9d0 0x7f3d48152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:15.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d4d26b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4814fda0 0x7f3d481de0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:15.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d4d26b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4806f3f0 con 0x7f3d480abe20 2023-03-03T20:11:15.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d4d26b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d481db7a0 con 0x7f3d4814fda0 2023-03-03T20:11:15.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d4d26b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d481dba80 con 0x7f3d4814f9d0 2023-03-03T20:11:15.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d46ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 0x7f3d4806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:15.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d46ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 0x7f3d4806ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44836/0 (socket says 172.21.15.7:44836) 2023-03-03T20:11:15.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d46ffd700 1 -- 172.21.15.7:0/909303829 learned_addr learned my addr 172.21.15.7:0/909303829 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:11:15.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d477fe700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4814fda0 0x7f3d481de0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:15.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d467fc700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4814f9d0 0x7f3d48152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:15.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d46ffd700 1 -- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4814f9d0 msgr2=0x7f3d48152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:15.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d46ffd700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4814f9d0 0x7f3d48152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:15.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d46ffd700 1 -- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4814fda0 msgr2=0x7f3d481de0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:15.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d46ffd700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4814fda0 0x7f3d481de0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:15.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d46ffd700 1 -- 172.21.15.7:0/909303829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d40009b30 con 0x7f3d480abe20 2023-03-03T20:11:15.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d477fe700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4814fda0 0x7f3d481de0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:11:15.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d46ffd700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 0x7f3d4806ed60 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800e810 tx=0x7f3d3800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:11:15.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d3f7fe700 1 -- 172.21.15.7:0/909303829 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d38013610 con 0x7f3d480abe20 2023-03-03T20:11:15.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d3f7fe700 1 -- 172.21.15.7:0/909303829 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d38013770 con 0x7f3d480abe20 2023-03-03T20:11:15.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d4d26b700 1 -- 172.21.15.7:0/909303829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d480ac350 con 0x7f3d480abe20 2023-03-03T20:11:15.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.327+0000 7f3d3f7fe700 1 -- 172.21.15.7:0/909303829 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d38013b20 con 0x7f3d480abe20 2023-03-03T20:11:15.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.328+0000 7f3d4d26b700 1 -- 172.21.15.7:0/909303829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d480ab630 con 0x7f3d480abe20 2023-03-03T20:11:15.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.328+0000 7f3d4d26b700 1 -- 172.21.15.7:0/909303829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d48071fa0 con 0x7f3d480abe20 2023-03-03T20:11:15.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.329+0000 7f3d3f7fe700 1 -- 172.21.15.7:0/909303829 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f3d38023070 con 0x7f3d480abe20 2023-03-03T20:11:15.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.329+0000 7f3d3f7fe700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 0x7f3d2806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:15.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.330+0000 7f3d3f7fe700 1 -- 172.21.15.7:0/909303829 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f3d38085f50 con 0x7f3d480abe20 2023-03-03T20:11:15.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.330+0000 7f3d467fc700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 0x7f3d2806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:15.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.330+0000 7f3d467fc700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 0x7f3d2806a7a0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f3d481e8490 tx=0x7f3d30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:11:15.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.334+0000 7f3d3f7fe700 1 -- 172.21.15.7:0/909303829 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f3d38052180 con 0x7f3d480abe20 2023-03-03T20:11:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.572+0000 7f3d4d26b700 1 -- 172.21.15.7:0/909303829 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d480ac4e0 con 0x7f3d480abe20 2023-03-03T20:11:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.573+0000 7f3d3f7fe700 1 -- 172.21.15.7:0/909303829 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f3d380557a0 con 0x7f3d480abe20 2023-03-03T20:11:15.575 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:11:15.576 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:11:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 -- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 msgr2=0x7f3d2806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 0x7f3d2806a7a0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f3d481e8490 tx=0x7f3d30009040 comp rx=0 tx=0).stop 2023-03-03T20:11:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 -- 172.21.15.7:0/909303829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 msgr2=0x7f3d4806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:15.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 0x7f3d4806ed60 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800e810 tx=0x7f3d3800ebd0 comp rx=0 tx=0).stop 2023-03-03T20:11:15.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 -- 172.21.15.7:0/909303829 shutdown_connections 2023-03-03T20:11:15.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d4814fda0 0x7f3d481de0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:15.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f3d280682f0 0x7f3d2806a7a0 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:15.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d480abe20 0x7f3d4806ed60 unknown :-1 s=CLOSED pgs=3721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:15.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 --2- 172.21.15.7:0/909303829 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3d4814f9d0 0x7f3d48152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:15.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 -- 172.21.15.7:0/909303829 >> 172.21.15.7:0/909303829 conn(0x7f3d480c8900 msgr2=0x7f3d48151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:11:15.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 -- 172.21.15.7:0/909303829 shutdown_connections 2023-03-03T20:11:15.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:15.575+0000 7f3d4d26b700 1 -- 172.21.15.7:0/909303829 wait complete. 2023-03-03T20:11:15.579 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:11:15.590 INFO:tasks.mds_thrash.fs.[a]:waiting for 115 secs before thrashing 2023-03-03T20:11:27.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:11:27.096 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:11:38.494 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:11:38.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.633+0000 7f414a419700 1 Processor -- start 2023-03-03T20:11:38.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.633+0000 7f414a419700 1 -- start start 2023-03-03T20:11:38.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f414a419700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4144151b20 0x7f4144071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:38.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f414a419700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f4144154f10 0x7f4144158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:38.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f414a419700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4144071d60 0x7f4144158540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:38.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f414a419700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4144151ef0 con 0x7f4144151b20 2023-03-03T20:11:38.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f414a419700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4144157b20 con 0x7f4144154f10 2023-03-03T20:11:38.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f414a419700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4144155410 con 0x7f4144071d60 2023-03-03T20:11:38.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f4143fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4144151b20 0x7f4144071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:38.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f4143fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4144151b20 0x7f4144071820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44856/0 (socket says 172.21.15.7:44856) 2023-03-03T20:11:38.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f4143fff700 1 -- 172.21.15.7:0/87638613 learned_addr learned my addr 172.21.15.7:0/87638613 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:11:38.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f41437fe700 1 --2- 172.21.15.7:0/87638613 >> v2:172.21.15.143:3300/0 conn(0x7f4144154f10 0x7f4144158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:38.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f41489b6700 1 --2- 172.21.15.7:0/87638613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4144071d60 0x7f4144158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:38.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f4143fff700 1 -- 172.21.15.7:0/87638613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4144071d60 msgr2=0x7f4144158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:11:38.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f4143fff700 1 --2- 172.21.15.7:0/87638613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4144071d60 0x7f4144158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f4143fff700 1 -- 172.21.15.7:0/87638613 >> v2:172.21.15.143:3300/0 conn(0x7f4144154f10 msgr2=0x7f4144158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:11:38.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f4143fff700 1 --2- 172.21.15.7:0/87638613 >> v2:172.21.15.143:3300/0 conn(0x7f4144154f10 0x7f4144158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f4143fff700 1 -- 172.21.15.7:0/87638613 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414406f640 con 0x7f4144151b20 2023-03-03T20:11:38.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f4143fff700 1 --2- 172.21.15.7:0/87638613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 0x7f4144071820 secure :-1 s=READY pgs=3722 cs=0 l=1 rev1=1 crypto rx=0x7f4144070850 tx=0x7f4134009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68576c58e5f3498 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:11:38.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.634+0000 7f4142ffd700 1 -- 172.21.15.7:0/87638613 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413400b870 con 0x7f4144151b20 2023-03-03T20:11:38.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f4142ffd700 1 -- 172.21.15.7:0/87638613 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413400b9d0 con 0x7f4144151b20 2023-03-03T20:11:38.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f4142ffd700 1 -- 172.21.15.7:0/87638613 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413400bd60 con 0x7f4144151b20 2023-03-03T20:11:38.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f414a419700 1 -- 172.21.15.7:0/87638613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 msgr2=0x7f4144071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:38.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f414a419700 1 --2- 172.21.15.7:0/87638613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 0x7f4144071820 secure :-1 s=READY pgs=3722 cs=0 l=1 rev1=1 crypto rx=0x7f4144070850 tx=0x7f4134009bd0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f414a419700 1 -- 172.21.15.7:0/87638613 shutdown_connections 2023-03-03T20:11:38.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f414a419700 1 --2- 172.21.15.7:0/87638613 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4144071d60 0x7f4144158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f414a419700 1 --2- 172.21.15.7:0/87638613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 0x7f4144071820 unknown :-1 s=CLOSED pgs=3722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f414a419700 1 --2- 172.21.15.7:0/87638613 >> v2:172.21.15.143:3300/0 conn(0x7f4144154f10 0x7f4144158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f414a419700 1 -- 172.21.15.7:0/87638613 >> 172.21.15.7:0/87638613 conn(0x7f41440c8900 msgr2=0x7f41440c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:11:38.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f414a419700 1 -- 172.21.15.7:0/87638613 shutdown_connections 2023-03-03T20:11:38.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.635+0000 7f414a419700 1 -- 172.21.15.7:0/87638613 wait complete. 2023-03-03T20:11:38.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.636+0000 7f414a419700 1 Processor -- start 2023-03-03T20:11:38.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.636+0000 7f414a419700 1 -- start start 2023-03-03T20:11:38.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f414a419700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4144071d60 0x7f4144143cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:38.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f414a419700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 0x7f41441560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:38.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f414a419700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4144154f10 0x7f4144144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:38.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f414a419700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4144147380 con 0x7f4144151b20 2023-03-03T20:11:38.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f414a419700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4144157610 con 0x7f4144071d60 2023-03-03T20:11:38.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f414a419700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4144145b20 con 0x7f4144154f10 2023-03-03T20:11:38.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f41437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 0x7f41441560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:38.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f4143fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4144071d60 0x7f4144143cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:38.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f41437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 0x7f41441560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44864/0 (socket says 172.21.15.7:44864) 2023-03-03T20:11:38.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f41437fe700 1 -- 172.21.15.7:0/1791251165 learned_addr learned my addr 172.21.15.7:0/1791251165 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:11:38.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.637+0000 7f41489b6700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4144154f10 0x7f4144144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:38.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f41437fe700 1 -- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4144154f10 msgr2=0x7f4144144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:38.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f41437fe700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4144154f10 0x7f4144144820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f41437fe700 1 -- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4144071d60 msgr2=0x7f4144143cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:38.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f41437fe700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4144071d60 0x7f4144143cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f41437fe700 1 -- 172.21.15.7:0/1791251165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4134009550 con 0x7f4144151b20 2023-03-03T20:11:38.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f41437fe700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 0x7f41441560b0 secure :-1 s=READY pgs=3723 cs=0 l=1 rev1=1 crypto rx=0x7f413800e840 tx=0x7f413800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:11:38.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f4140ff9700 1 -- 172.21.15.7:0/1791251165 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413800c070 con 0x7f4144151b20 2023-03-03T20:11:38.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f4140ff9700 1 -- 172.21.15.7:0/1791251165 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4138013610 con 0x7f4144151b20 2023-03-03T20:11:38.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f414a419700 1 -- 172.21.15.7:0/1791251165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414406fc00 con 0x7f4144151b20 2023-03-03T20:11:38.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f4140ff9700 1 -- 172.21.15.7:0/1791251165 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41380139a0 con 0x7f4144151b20 2023-03-03T20:11:38.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.638+0000 7f414a419700 1 -- 172.21.15.7:0/1791251165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f414406f920 con 0x7f4144151b20 2023-03-03T20:11:38.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.639+0000 7f414a419700 1 -- 172.21.15.7:0/1791251165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4144074bb0 con 0x7f4144151b20 2023-03-03T20:11:38.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.640+0000 7f4140ff9700 1 -- 172.21.15.7:0/1791251165 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4138023070 con 0x7f4144151b20 2023-03-03T20:11:38.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.640+0000 7f4140ff9700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41300682f0 0x7f413006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:11:38.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.640+0000 7f4140ff9700 1 -- 172.21.15.7:0/1791251165 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4138085e30 con 0x7f4144151b20 2023-03-03T20:11:38.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.640+0000 7f4143fff700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41300682f0 0x7f413006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:11:38.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.641+0000 7f4143fff700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41300682f0 0x7f413006a7a0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f4134009840 tx=0x7f4134008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:11:38.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.644+0000 7f4140ff9700 1 -- 172.21.15.7:0/1791251165 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f41380521d0 con 0x7f4144151b20 2023-03-03T20:11:38.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.882+0000 7f414a419700 1 -- 172.21.15.7:0/1791251165 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41440aae10 con 0x7f4144151b20 2023-03-03T20:11:38.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.883+0000 7f4140ff9700 1 -- 172.21.15.7:0/1791251165 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f41380557f0 con 0x7f4144151b20 2023-03-03T20:11:38.884 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:11:38.884 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:11:38.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 -- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41300682f0 msgr2=0x7f413006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41300682f0 0x7f413006a7a0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f4134009840 tx=0x7f4134008030 comp rx=0 tx=0).stop 2023-03-03T20:11:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 -- 172.21.15.7:0/1791251165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 msgr2=0x7f41441560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:11:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 0x7f41441560b0 secure :-1 s=READY pgs=3723 cs=0 l=1 rev1=1 crypto rx=0x7f413800e840 tx=0x7f413800eb50 comp rx=0 tx=0).stop 2023-03-03T20:11:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 -- 172.21.15.7:0/1791251165 shutdown_connections 2023-03-03T20:11:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f4144154f10 0x7f4144144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4144071d60 0x7f4144143cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f41300682f0 0x7f413006a7a0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 --2- 172.21.15.7:0/1791251165 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4144151b20 0x7f41441560b0 unknown :-1 s=CLOSED pgs=3723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:11:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 -- 172.21.15.7:0/1791251165 >> 172.21.15.7:0/1791251165 conn(0x7f41440c8900 msgr2=0x7f41440ac3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:11:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 -- 172.21.15.7:0/1791251165 shutdown_connections 2023-03-03T20:11:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:11:38.885+0000 7f414a419700 1 -- 172.21.15.7:0/1791251165 wait complete. 2023-03-03T20:11:38.888 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:11:38.899 INFO:tasks.mds_thrash.fs.[b]:waiting for 48 secs before thrashing 2023-03-03T20:11:57.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:11:57.209 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:12:26.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:12:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.038+0000 7f8e75093700 1 Processor -- start 2023-03-03T20:12:27.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.038+0000 7f8e75093700 1 -- start start 2023-03-03T20:12:27.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e75093700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e7014e560 0x7f8e70071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:12:27.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e75093700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c4850 0x7f8e70154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:12:27.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e75093700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e70072920 con 0x7f8e700c43d0 2023-03-03T20:12:27.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e75093700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e7006bef0 con 0x7f8e7014e560 2023-03-03T20:12:27.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e75093700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e700c4cb0 con 0x7f8e700c4850 2023-03-03T20:12:27.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e6ed9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8e7014e560 0x7f8e70071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:12:27.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e6e59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8e700c43d0 0x7f8e70154590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38868/0 (socket says 172.21.15.7:38868) 2023-03-03T20:12:27.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e6e59c700 1 -- 172.21.15.7:0/3432409576 learned_addr learned my addr 172.21.15.7:0/3432409576 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:12:27.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e6f59e700 1 --2- 172.21.15.7:0/3432409576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c4850 0x7f8e70154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:12:27.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e6ed9d700 1 -- 172.21.15.7:0/3432409576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c4850 msgr2=0x7f8e70154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:12:27.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e6ed9d700 1 --2- 172.21.15.7:0/3432409576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c4850 0x7f8e70154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:12:27.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e6ed9d700 1 -- 172.21.15.7:0/3432409576 >> v1:172.21.15.7:6789/0 conn(0x7f8e700c43d0 legacy=0x7f8e70154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:12:27.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e6dd9b700 1 -- 172.21.15.7:0/3432409576 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40399876 0 0) 0x7f8e60001110 con 0x7f8e700c43d0 2023-03-03T20:12:27.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e6ed9d700 1 -- 172.21.15.7:0/3432409576 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e701551e0 con 0x7f8e7014e560 2023-03-03T20:12:27.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.039+0000 7f8e6f59e700 1 --2- 172.21.15.7:0/3432409576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c4850 0x7f8e70154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:12:27.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e6ed9d700 1 --2- 172.21.15.7:0/3432409576 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e7014e560 0x7f8e70071c70 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 crypto rx=0x7f8e64009840 tx=0x7f8e64009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41078d1fa992898 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:12:27.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e6dd9b700 1 -- 172.21.15.7:0/3432409576 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e64011830 con 0x7f8e7014e560 2023-03-03T20:12:27.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e6dd9b700 1 -- 172.21.15.7:0/3432409576 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e64011990 con 0x7f8e7014e560 2023-03-03T20:12:27.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e6dd9b700 1 -- 172.21.15.7:0/3432409576 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e64011d20 con 0x7f8e7014e560 2023-03-03T20:12:27.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e75093700 1 -- 172.21.15.7:0/3432409576 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e7014e560 msgr2=0x7f8e70071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:12:27.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e75093700 1 --2- 172.21.15.7:0/3432409576 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e7014e560 0x7f8e70071c70 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 crypto rx=0x7f8e64009840 tx=0x7f8e64009b50 comp rx=0 tx=0).stop 2023-03-03T20:12:27.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e75093700 1 -- 172.21.15.7:0/3432409576 shutdown_connections 2023-03-03T20:12:27.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e75093700 1 --2- 172.21.15.7:0/3432409576 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c4850 0x7f8e70154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:12:27.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e75093700 1 --2- 172.21.15.7:0/3432409576 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e7014e560 0x7f8e70071c70 unknown :-1 s=CLOSED pgs=3725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:12:27.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e75093700 1 -- 172.21.15.7:0/3432409576 >> 172.21.15.7:0/3432409576 conn(0x7f8e700736f0 msgr2=0x7f8e70073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:12:27.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e75093700 1 -- 172.21.15.7:0/3432409576 shutdown_connections 2023-03-03T20:12:27.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.040+0000 7f8e75093700 1 -- 172.21.15.7:0/3432409576 wait complete. 2023-03-03T20:12:27.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.042+0000 7f8e75093700 1 Processor -- start 2023-03-03T20:12:27.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.042+0000 7f8e75093700 1 -- start start 2023-03-03T20:12:27.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e75093700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c43d0 0x7f8e7014f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:12:27.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e75093700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e700c4850 0x7f8e70151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:12:27.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e75093700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7014e560 0x7f8e701539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:12:27.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e75093700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e70150130 con 0x7f8e700c4850 2023-03-03T20:12:27.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e75093700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7014f470 con 0x7f8e7014e560 2023-03-03T20:12:27.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e75093700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e70151870 con 0x7f8e700c43d0 2023-03-03T20:12:27.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e700c4850 0x7f8e70151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:12:27.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6e59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e700c4850 0x7f8e70151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44894/0 (socket says 172.21.15.7:44894) 2023-03-03T20:12:27.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6e59c700 1 -- 172.21.15.7:0/3294142886 learned_addr learned my addr 172.21.15.7:0/3294142886 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:12:27.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6ed9d700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c43d0 0x7f8e7014f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:12:27.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6f59e700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7014e560 0x7f8e701539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:12:27.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6e59c700 1 -- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c43d0 msgr2=0x7f8e7014f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:12:27.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6e59c700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c43d0 0x7f8e7014f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:12:27.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6e59c700 1 -- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7014e560 msgr2=0x7f8e701539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:12:27.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6e59c700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7014e560 0x7f8e701539d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:12:27.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6e59c700 1 -- 172.21.15.7:0/3294142886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e640094a0 con 0x7f8e700c4850 2023-03-03T20:12:27.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e6e59c700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e700c4850 0x7f8e70151220 secure :-1 s=READY pgs=3726 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000c760 tx=0x7f8e6000cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:12:27.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e577fe700 1 -- 172.21.15.7:0/3294142886 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e60004c80 con 0x7f8e700c4850 2023-03-03T20:12:27.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e577fe700 1 -- 172.21.15.7:0/3294142886 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e60004e00 con 0x7f8e700c4850 2023-03-03T20:12:27.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.043+0000 7f8e75093700 1 -- 172.21.15.7:0/3294142886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e700aacc0 con 0x7f8e700c4850 2023-03-03T20:12:27.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.044+0000 7f8e577fe700 1 -- 172.21.15.7:0/3294142886 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e60007310 con 0x7f8e700c4850 2023-03-03T20:12:27.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.044+0000 7f8e75093700 1 -- 172.21.15.7:0/3294142886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7006b910 con 0x7f8e700c4850 2023-03-03T20:12:27.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.044+0000 7f8e75093700 1 -- 172.21.15.7:0/3294142886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e7006bb80 con 0x7f8e700c4850 2023-03-03T20:12:27.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.045+0000 7f8e577fe700 1 -- 172.21.15.7:0/3294142886 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8e6001a020 con 0x7f8e700c4850 2023-03-03T20:12:27.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.045+0000 7f8e577fe700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e5c0682f0 0x7f8e5c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:12:27.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.045+0000 7f8e577fe700 1 -- 172.21.15.7:0/3294142886 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8e60085c50 con 0x7f8e700c4850 2023-03-03T20:12:27.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.046+0000 7f8e6ed9d700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e5c0682f0 0x7f8e5c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:12:27.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.046+0000 7f8e6ed9d700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e5c0682f0 0x7f8e5c06a7a0 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f8e700adf90 tx=0x7f8e64008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:12:27.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.050+0000 7f8e577fe700 1 -- 172.21.15.7:0/3294142886 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8e60051ff0 con 0x7f8e700c4850 2023-03-03T20:12:27.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.294+0000 7f8e75093700 1 -- 172.21.15.7:0/3294142886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e70074bb0 con 0x7f8e700c4850 2023-03-03T20:12:27.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.295+0000 7f8e577fe700 1 -- 172.21.15.7:0/3294142886 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8e60055610 con 0x7f8e700c4850 2023-03-03T20:12:27.297 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:12:27.297 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:12:27.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 -- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e5c0682f0 msgr2=0x7f8e5c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:12:27.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e5c0682f0 0x7f8e5c06a7a0 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f8e700adf90 tx=0x7f8e64008030 comp rx=0 tx=0).stop 2023-03-03T20:12:27.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 -- 172.21.15.7:0/3294142886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e700c4850 msgr2=0x7f8e70151220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:12:27.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e700c4850 0x7f8e70151220 secure :-1 s=READY pgs=3726 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000c760 tx=0x7f8e6000cb20 comp rx=0 tx=0).stop 2023-03-03T20:12:27.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 -- 172.21.15.7:0/3294142886 shutdown_connections 2023-03-03T20:12:27.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e7014e560 0x7f8e701539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:12:27.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8e700c43d0 0x7f8e7014f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:12:27.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f8e5c0682f0 0x7f8e5c06a7a0 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:12:27.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 --2- 172.21.15.7:0/3294142886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8e700c4850 0x7f8e70151220 unknown :-1 s=CLOSED pgs=3726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:12:27.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 -- 172.21.15.7:0/3294142886 >> 172.21.15.7:0/3294142886 conn(0x7f8e700736f0 msgr2=0x7f8e7014bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:12:27.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 -- 172.21.15.7:0/3294142886 shutdown_connections 2023-03-03T20:12:27.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:12:27.297+0000 7f8e75093700 1 -- 172.21.15.7:0/3294142886 wait complete. 2023-03-03T20:12:27.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:12:27.303 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:12:27.305 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:12:27.311 INFO:tasks.mds_thrash.fs.[b]:waiting for 96 secs before thrashing 2023-03-03T20:12:57.443 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:12:57.447 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:13:10.591 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:13:10.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.729+0000 7f6908900700 1 Processor -- start 2023-03-03T20:13:10.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.729+0000 7f6908900700 1 -- start start 2023-03-03T20:13:10.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.730+0000 7f6908900700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f69040ab0c0 0x7f69040c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:13:10.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.730+0000 7f6908900700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f690406a750 0x7f6904158150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:13:10.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.730+0000 7f6908900700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6904158690 con 0x7f690406a2b0 2023-03-03T20:13:10.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.730+0000 7f6908900700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6904154530 con 0x7f690406a750 2023-03-03T20:13:10.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.730+0000 7f6908900700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6904154810 con 0x7f69040ab0c0 2023-03-03T20:13:10.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f6902d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f690406a750 0x7f6904158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:13:10.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f6901d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f690406a2b0 0x7f6904157a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38900/0 (socket says 172.21.15.7:38900) 2023-03-03T20:13:10.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f6901d9b700 1 -- 172.21.15.7:0/1729479317 learned_addr learned my addr 172.21.15.7:0/1729479317 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:13:10.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f690259c700 1 --2- 172.21.15.7:0/1729479317 >> v2:172.21.15.143:3300/0 conn(0x7f69040ab0c0 0x7f69040c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:13:10.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f6902d9d700 1 -- 172.21.15.7:0/1729479317 >> v2:172.21.15.143:3300/0 conn(0x7f69040ab0c0 msgr2=0x7f69040c1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:13:10.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f6902d9d700 1 --2- 172.21.15.7:0/1729479317 >> v2:172.21.15.143:3300/0 conn(0x7f69040ab0c0 0x7f69040c1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:13:10.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f6902d9d700 1 -- 172.21.15.7:0/1729479317 >> v1:172.21.15.7:6789/0 conn(0x7f690406a2b0 legacy=0x7f6904157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:13:10.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f6902d9d700 1 -- 172.21.15.7:0/1729479317 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6904154af0 con 0x7f690406a750 2023-03-03T20:13:10.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f690159a700 1 -- 172.21.15.7:0/1729479317 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320316939 0 0) 0x7f6904158690 con 0x7f690406a2b0 2023-03-03T20:13:10.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f690259c700 1 --2- 172.21.15.7:0/1729479317 >> v2:172.21.15.143:3300/0 conn(0x7f69040ab0c0 0x7f69040c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:13:10.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f6902d9d700 1 --2- 172.21.15.7:0/1729479317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a750 0x7f6904158150 secure :-1 s=READY pgs=3728 cs=0 l=1 rev1=1 crypto rx=0x7f69040c3560 tx=0x7f68fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eee694f30cfe9ed server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:13:10.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f690159a700 1 -- 172.21.15.7:0/1729479317 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00e070 con 0x7f690406a750 2023-03-03T20:13:10.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f690159a700 1 -- 172.21.15.7:0/1729479317 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc00bbd0 con 0x7f690406a750 2023-03-03T20:13:10.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.731+0000 7f690159a700 1 -- 172.21.15.7:0/1729479317 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc012430 con 0x7f690406a750 2023-03-03T20:13:10.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.732+0000 7f6908900700 1 -- 172.21.15.7:0/1729479317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a750 msgr2=0x7f6904158150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:13:10.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.732+0000 7f6908900700 1 --2- 172.21.15.7:0/1729479317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a750 0x7f6904158150 secure :-1 s=READY pgs=3728 cs=0 l=1 rev1=1 crypto rx=0x7f69040c3560 tx=0x7f68fc00b330 comp rx=0 tx=0).stop 2023-03-03T20:13:10.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.732+0000 7f6908900700 1 -- 172.21.15.7:0/1729479317 shutdown_connections 2023-03-03T20:13:10.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.732+0000 7f6908900700 1 --2- 172.21.15.7:0/1729479317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a750 0x7f6904158150 unknown :-1 s=CLOSED pgs=3728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:13:10.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.732+0000 7f6908900700 1 --2- 172.21.15.7:0/1729479317 >> v2:172.21.15.143:3300/0 conn(0x7f69040ab0c0 0x7f69040c1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:13:10.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.732+0000 7f6908900700 1 -- 172.21.15.7:0/1729479317 >> 172.21.15.7:0/1729479317 conn(0x7f69040a9430 msgr2=0x7f69040c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:13:10.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.732+0000 7f6908900700 1 -- 172.21.15.7:0/1729479317 shutdown_connections 2023-03-03T20:13:10.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.732+0000 7f6908900700 1 -- 172.21.15.7:0/1729479317 wait complete. 2023-03-03T20:13:10.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.733+0000 7f6908900700 1 Processor -- start 2023-03-03T20:13:10.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.733+0000 7f6908900700 1 -- start start 2023-03-03T20:13:10.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f6908900700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a2b0 0x7f69041e9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:13:10.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f6908900700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f690406a750 0x7f69041ed210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:13:10.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f6908900700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69040ab0c0 0x7f690414f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:13:10.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f6908900700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69040723b0 con 0x7f690406a2b0 2023-03-03T20:13:10.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f6908900700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041ea480 con 0x7f69040ab0c0 2023-03-03T20:13:10.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f6908900700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6904154180 con 0x7f690406a750 2023-03-03T20:13:10.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f690259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a2b0 0x7f69041e9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:13:10.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f690259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a2b0 0x7f69041e9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44926/0 (socket says 172.21.15.7:44926) 2023-03-03T20:13:10.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f690259c700 1 -- 172.21.15.7:0/611206154 learned_addr learned my addr 172.21.15.7:0/611206154 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:13:10.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f6902d9d700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69040ab0c0 0x7f690414f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:13:10.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.734+0000 7f6901d9b700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f690406a750 0x7f69041ed210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:13:10.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f690259c700 1 -- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f690406a750 msgr2=0x7f69041ed210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:13:10.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f690259c700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f690406a750 0x7f69041ed210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:13:10.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f690259c700 1 -- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69040ab0c0 msgr2=0x7f690414f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:13:10.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f690259c700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69040ab0c0 0x7f690414f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:13:10.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f690259c700 1 -- 172.21.15.7:0/611206154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc009a70 con 0x7f690406a2b0 2023-03-03T20:13:10.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f690259c700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a2b0 0x7f69041e9f40 secure :-1 s=READY pgs=3729 cs=0 l=1 rev1=1 crypto rx=0x7f68f400e810 tx=0x7f68f400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:13:10.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f68f2ffd700 1 -- 172.21.15.7:0/611206154 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4013610 con 0x7f690406a2b0 2023-03-03T20:13:10.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f68f2ffd700 1 -- 172.21.15.7:0/611206154 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f4013770 con 0x7f690406a2b0 2023-03-03T20:13:10.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f6908900700 1 -- 172.21.15.7:0/611206154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69041550b0 con 0x7f690406a2b0 2023-03-03T20:13:10.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f68f2ffd700 1 -- 172.21.15.7:0/611206154 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4013b20 con 0x7f690406a2b0 2023-03-03T20:13:10.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.735+0000 7f6908900700 1 -- 172.21.15.7:0/611206154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6904154dd0 con 0x7f690406a2b0 2023-03-03T20:13:10.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.736+0000 7f6908900700 1 -- 172.21.15.7:0/611206154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69040ab490 con 0x7f690406a2b0 2023-03-03T20:13:10.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.736+0000 7f68f2ffd700 1 -- 172.21.15.7:0/611206154 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f68f4023070 con 0x7f690406a2b0 2023-03-03T20:13:10.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.737+0000 7f68f2ffd700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e40682f0 0x7f68e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:13:10.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.737+0000 7f68f2ffd700 1 -- 172.21.15.7:0/611206154 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f68f4085ca0 con 0x7f690406a2b0 2023-03-03T20:13:10.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.737+0000 7f6901d9b700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e40682f0 0x7f68e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:13:10.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.737+0000 7f6901d9b700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e40682f0 0x7f68e406a7a0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f69040c6510 tx=0x7f68f8007e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:13:10.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.741+0000 7f68f2ffd700 1 -- 172.21.15.7:0/611206154 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f68f408a070 con 0x7f690406a2b0 2023-03-03T20:13:10.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.979+0000 7f6908900700 1 -- 172.21.15.7:0/611206154 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6904074bb0 con 0x7f690406a2b0 2023-03-03T20:13:10.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.980+0000 7f68f2ffd700 1 -- 172.21.15.7:0/611206154 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f68f4055a70 con 0x7f690406a2b0 2023-03-03T20:13:10.985 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:13:10.986 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:13:10.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 -- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e40682f0 msgr2=0x7f68e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:13:10.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e40682f0 0x7f68e406a7a0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f69040c6510 tx=0x7f68f8007e70 comp rx=0 tx=0).stop 2023-03-03T20:13:10.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 -- 172.21.15.7:0/611206154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a2b0 msgr2=0x7f69041e9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:13:10.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a2b0 0x7f69041e9f40 secure :-1 s=READY pgs=3729 cs=0 l=1 rev1=1 crypto rx=0x7f68f400e810 tx=0x7f68f400eb20 comp rx=0 tx=0).stop 2023-03-03T20:13:10.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 -- 172.21.15.7:0/611206154 shutdown_connections 2023-03-03T20:13:10.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69040ab0c0 0x7f690414f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:13:10.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f68e40682f0 0x7f68e406a7a0 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:13:10.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f690406a2b0 0x7f69041e9f40 unknown :-1 s=CLOSED pgs=3729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:13:10.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 --2- 172.21.15.7:0/611206154 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f690406a750 0x7f69041ed210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:13:10.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 -- 172.21.15.7:0/611206154 >> 172.21.15.7:0/611206154 conn(0x7f69040a9430 msgr2=0x7f69040c5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:13:10.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 -- 172.21.15.7:0/611206154 shutdown_connections 2023-03-03T20:13:10.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:13:10.982+0000 7f6908900700 1 -- 172.21.15.7:0/611206154 wait complete. 2023-03-03T20:13:10.989 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:13:10.994 INFO:tasks.mds_thrash.fs.[a]:waiting for 78 secs before thrashing 2023-03-03T20:13:27.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:13:27.584 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:13:57.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:13:57.661 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:14:03.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:14:03.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.452+0000 7f12665d8700 1 Processor -- start 2023-03-03T20:14:03.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.452+0000 7f12665d8700 1 -- start start 2023-03-03T20:14:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f12665d8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12600caa30 0x7f12600c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f12665d8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1260154ff0 0x7f12600cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f12665d8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12600c7310 con 0x7f12600c00e0 2023-03-03T20:14:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f12665d8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1260155430 con 0x7f12600caa30 2023-03-03T20:14:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f12665d8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12600bfd10 con 0x7f1260154ff0 2023-03-03T20:14:03.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f125ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f12600caa30 0x7f12600c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f125f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f12600c00e0 0x7f12600c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38954/0 (socket says 172.21.15.7:38954) 2023-03-03T20:14:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f125f7fe700 1 -- 172.21.15.7:0/2576638192 learned_addr learned my addr 172.21.15.7:0/2576638192 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:14:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f1264b75700 1 --2- 172.21.15.7:0/2576638192 >> v2:172.21.15.143:3300/0 conn(0x7f1260154ff0 0x7f12600cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f125effd700 1 -- 172.21.15.7:0/2576638192 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943919751 0 0) 0x7f12600c7310 con 0x7f12600c00e0 2023-03-03T20:14:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f125effd700 1 -- 172.21.15.7:0/2576638192 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12480035b0 con 0x7f12600c00e0 2023-03-03T20:14:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f125ffff700 1 -- 172.21.15.7:0/2576638192 >> v2:172.21.15.143:3300/0 conn(0x7f1260154ff0 msgr2=0x7f12600cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:14:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f125ffff700 1 --2- 172.21.15.7:0/2576638192 >> v2:172.21.15.143:3300/0 conn(0x7f1260154ff0 0x7f12600cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:03.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f125ffff700 1 -- 172.21.15.7:0/2576638192 >> v1:172.21.15.7:6789/0 conn(0x7f12600c00e0 legacy=0x7f12600c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.453+0000 7f125ffff700 1 -- 172.21.15.7:0/2576638192 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600bfa30 con 0x7f12600caa30 2023-03-03T20:14:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f125ffff700 1 --2- 172.21.15.7:0/2576638192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600caa30 0x7f12600c5c60 secure :-1 s=READY pgs=3731 cs=0 l=1 rev1=1 crypto rx=0x7f1260071530 tx=0x7f1250009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5a6fd81763f626 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:14:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f125effd700 1 -- 172.21.15.7:0/2576638192 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250011830 con 0x7f12600caa30 2023-03-03T20:14:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f125effd700 1 -- 172.21.15.7:0/2576638192 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1250011990 con 0x7f12600caa30 2023-03-03T20:14:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f125effd700 1 -- 172.21.15.7:0/2576638192 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250011d20 con 0x7f12600caa30 2023-03-03T20:14:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f12665d8700 1 -- 172.21.15.7:0/2576638192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600caa30 msgr2=0x7f12600c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:03.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f12665d8700 1 --2- 172.21.15.7:0/2576638192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600caa30 0x7f12600c5c60 secure :-1 s=READY pgs=3731 cs=0 l=1 rev1=1 crypto rx=0x7f1260071530 tx=0x7f1250009b20 comp rx=0 tx=0).stop 2023-03-03T20:14:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f12665d8700 1 -- 172.21.15.7:0/2576638192 shutdown_connections 2023-03-03T20:14:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f12665d8700 1 --2- 172.21.15.7:0/2576638192 >> v2:172.21.15.143:3300/0 conn(0x7f1260154ff0 0x7f12600cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f12665d8700 1 --2- 172.21.15.7:0/2576638192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600caa30 0x7f12600c5c60 unknown :-1 s=CLOSED pgs=3731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f12665d8700 1 -- 172.21.15.7:0/2576638192 >> 172.21.15.7:0/2576638192 conn(0x7f1260066e40 msgr2=0x7f1260071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:14:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f12665d8700 1 -- 172.21.15.7:0/2576638192 shutdown_connections 2023-03-03T20:14:03.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.454+0000 7f12665d8700 1 -- 172.21.15.7:0/2576638192 wait complete. 2023-03-03T20:14:03.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.456+0000 7f12665d8700 1 Processor -- start 2023-03-03T20:14:03.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.456+0000 7f12665d8700 1 -- start start 2023-03-03T20:14:03.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.456+0000 7f12665d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600c00e0 0x7f12600c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:03.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.456+0000 7f12665d8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12600caa30 0x7f126014d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:03.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.456+0000 7f12665d8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1260154ff0 0x7f126014d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:03.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.456+0000 7f12665d8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600c31c0 con 0x7f12600c00e0 2023-03-03T20:14:03.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f12665d8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12601ee040 con 0x7f1260154ff0 2023-03-03T20:14:03.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f12665d8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600c2b00 con 0x7f12600caa30 2023-03-03T20:14:03.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600c00e0 0x7f12600c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:03.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600c00e0 0x7f12600c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44978/0 (socket says 172.21.15.7:44978) 2023-03-03T20:14:03.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125ffff700 1 -- 172.21.15.7:0/4229081675 learned_addr learned my addr 172.21.15.7:0/4229081675 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:14:03.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f1264b75700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1260154ff0 0x7f126014d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:03.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125f7fe700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12600caa30 0x7f126014d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:03.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125ffff700 1 -- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12600caa30 msgr2=0x7f126014d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:03.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125ffff700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12600caa30 0x7f126014d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:03.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125ffff700 1 -- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1260154ff0 msgr2=0x7f126014d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:03.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125ffff700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1260154ff0 0x7f126014d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:03.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125ffff700 1 -- 172.21.15.7:0/4229081675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12500094a0 con 0x7f12600c00e0 2023-03-03T20:14:03.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f1264b75700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1260154ff0 0x7f126014d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:14:03.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125ffff700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600c00e0 0x7f12600c6be0 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 crypto rx=0x7f1250009b60 tx=0x7f1250004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:14:03.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125cff9700 1 -- 172.21.15.7:0/4229081675 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12500036a0 con 0x7f12600c00e0 2023-03-03T20:14:03.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125cff9700 1 -- 172.21.15.7:0/4229081675 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125001f070 con 0x7f12600c00e0 2023-03-03T20:14:03.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f12665d8700 1 -- 172.21.15.7:0/4229081675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12601547f0 con 0x7f12600c00e0 2023-03-03T20:14:03.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f125cff9700 1 -- 172.21.15.7:0/4229081675 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250003a40 con 0x7f12600c00e0 2023-03-03T20:14:03.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.457+0000 7f12665d8700 1 -- 172.21.15.7:0/4229081675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1260154510 con 0x7f12600c00e0 2023-03-03T20:14:03.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.458+0000 7f12665d8700 1 -- 172.21.15.7:0/4229081675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1260154ad0 con 0x7f12600c00e0 2023-03-03T20:14:03.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.459+0000 7f125cff9700 1 -- 172.21.15.7:0/4229081675 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f125000f050 con 0x7f12600c00e0 2023-03-03T20:14:03.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.459+0000 7f125cff9700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f124c0682f0 0x7f124c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:03.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.459+0000 7f125cff9700 1 -- 172.21.15.7:0/4229081675 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1250021070 con 0x7f12600c00e0 2023-03-03T20:14:03.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.460+0000 7f125f7fe700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f124c0682f0 0x7f124c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:03.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.460+0000 7f125f7fe700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f124c0682f0 0x7f124c06a7a0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f1254007fd0 tx=0x7f1254009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:14:03.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.464+0000 7f125cff9700 1 -- 172.21.15.7:0/4229081675 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1250053e20 con 0x7f12600c00e0 2023-03-03T20:14:03.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.706+0000 7f12665d8700 1 -- 172.21.15.7:0/4229081675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12600c3300 con 0x7f12600c00e0 2023-03-03T20:14:03.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.706+0000 7f125cff9700 1 -- 172.21.15.7:0/4229081675 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f12600c3300 con 0x7f12600c00e0 2023-03-03T20:14:03.707 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:14:03.708 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:14:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 -- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f124c0682f0 msgr2=0x7f124c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f124c0682f0 0x7f124c06a7a0 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f1254007fd0 tx=0x7f1254009040 comp rx=0 tx=0).stop 2023-03-03T20:14:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 -- 172.21.15.7:0/4229081675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600c00e0 msgr2=0x7f12600c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600c00e0 0x7f12600c6be0 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 crypto rx=0x7f1250009b60 tx=0x7f1250004750 comp rx=0 tx=0).stop 2023-03-03T20:14:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 -- 172.21.15.7:0/4229081675 shutdown_connections 2023-03-03T20:14:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1260154ff0 0x7f126014d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f124c0682f0 0x7f124c06a7a0 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12600c00e0 0x7f12600c6be0 unknown :-1 s=CLOSED pgs=3732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 --2- 172.21.15.7:0/4229081675 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f12600caa30 0x7f126014d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 -- 172.21.15.7:0/4229081675 >> 172.21.15.7:0/4229081675 conn(0x7f1260066e40 msgr2=0x7f12601520a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:14:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 -- 172.21.15.7:0/4229081675 shutdown_connections 2023-03-03T20:14:03.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:03.708+0000 7f12665d8700 1 -- 172.21.15.7:0/4229081675 wait complete. 2023-03-03T20:14:03.711 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:14:03.721 INFO:tasks.mds_thrash.fs.[b]:waiting for 29 secs before thrashing 2023-03-03T20:14:27.806 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:14:27.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:14:28.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:14:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.133+0000 7f561dd67700 1 Processor -- start 2023-03-03T20:14:29.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.133+0000 7f561dd67700 1 -- start start 2023-03-03T20:14:29.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f561dd67700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f561814b0d0 0x7f5618071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:29.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f561dd67700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56180aad20 0x7f5618157df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:29.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f561dd67700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56180a9c20 con 0x7f5618071d40 2023-03-03T20:14:29.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f561dd67700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5618154f10 con 0x7f561814b0d0 2023-03-03T20:14:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f561dd67700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56180ac3c0 con 0x7f56180aad20 2023-03-03T20:14:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f5617fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5618071d40 0x7f5618158330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38984/0 (socket says 172.21.15.7:38984) 2023-03-03T20:14:29.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f5617fff700 1 -- 172.21.15.7:0/1249177835 learned_addr learned my addr 172.21.15.7:0/1249177835 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:14:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f5616ffd700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56180aad20 0x7f5618157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f56177fe700 1 --2- 172.21.15.7:0/1249177835 >> v2:172.21.15.143:3300/0 conn(0x7f561814b0d0 0x7f5618071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964715841 0 0) 0x7f56180a9c20 con 0x7f5618071d40 2023-03-03T20:14:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.134+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56040035b0 con 0x7f5618071d40 2023-03-03T20:14:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4069218651 0 0) 0x7f56040035b0 con 0x7f5618071d40 2023-03-03T20:14:29.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56180aad20 msgr2=0x7f5618157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:14:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56180aad20 0x7f5618157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 >> v2:172.21.15.143:3300/0 conn(0x7f561814b0d0 msgr2=0x7f5618071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:14:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 --2- 172.21.15.7:0/1249177835 >> v2:172.21.15.143:3300/0 conn(0x7f561814b0d0 0x7f5618071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561806fc70 con 0x7f5618071d40 2023-03-03T20:14:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f56100019f0 con 0x7f5618071d40 2023-03-03T20:14:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 >> v1:172.21.15.7:6789/0 conn(0x7f5618071d40 legacy=0x7f5618158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56180aad20 0x7f5604003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5604003ff0 0x7f5604004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604004d90 0x7f5604005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5618154f10 con 0x7f5604004d90 2023-03-03T20:14:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56180ac3c0 con 0x7f5604003ff0 2023-03-03T20:14:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560400bac0 con 0x7f56180aad20 2023-03-03T20:14:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f5617fff700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604004d90 0x7f5604005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56177fe700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56180aad20 0x7f5604003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f5616ffd700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5604003ff0 0x7f5604004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f5617fff700 1 -- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56180aad20 msgr2=0x7f5604003a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:14:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f5617fff700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f56180aad20 0x7f5604003a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f5617fff700 1 -- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5604003ff0 msgr2=0x7f5604004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:14:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f5617fff700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5604003ff0 0x7f5604004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:29.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f5617fff700 1 -- 172.21.15.7:0/1249177835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56040035b0 con 0x7f5604004d90 2023-03-03T20:14:29.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f5617fff700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604004d90 0x7f5604005200 secure :-1 s=READY pgs=3734 cs=0 l=1 rev1=1 crypto rx=0x7f56100013d0 tx=0x7f5604005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d7470d8f97bf9d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:14:29.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.135+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561000a340 con 0x7f5604004d90 2023-03-03T20:14:29.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.136+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5610012600 con 0x7f5604004d90 2023-03-03T20:14:29.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.136+0000 7f56167fc700 1 -- 172.21.15.7:0/1249177835 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56100129b0 con 0x7f5604004d90 2023-03-03T20:14:29.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.136+0000 7f561dd67700 1 -- 172.21.15.7:0/1249177835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604004d90 msgr2=0x7f5604005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:29.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.136+0000 7f561dd67700 1 --2- 172.21.15.7:0/1249177835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5604004d90 0x7f5604005200 secure :-1 s=READY pgs=3734 cs=0 l=1 rev1=1 crypto rx=0x7f56100013d0 tx=0x7f5604005880 comp rx=0 tx=0).stop 2023-03-03T20:14:29.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.136+0000 7f5617fff700 1 -- 172.21.15.7:0/1249177835 reap_dead start 2023-03-03T20:14:29.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.136+0000 7f561dd67700 1 -- 172.21.15.7:0/1249177835 shutdown_connections 2023-03-03T20:14:29.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.136+0000 7f561dd67700 1 -- 172.21.15.7:0/1249177835 >> 172.21.15.7:0/1249177835 conn(0x7f56180c8900 msgr2=0x7f56180c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:14:29.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.136+0000 7f561dd67700 1 -- 172.21.15.7:0/1249177835 shutdown_connections 2023-03-03T20:14:29.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.136+0000 7f561dd67700 1 -- 172.21.15.7:0/1249177835 wait complete. 2023-03-03T20:14:29.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.137+0000 7f561dd67700 1 Processor -- start 2023-03-03T20:14:29.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f561dd67700 1 -- start start 2023-03-03T20:14:29.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f561dd67700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5618145af0 0x7f56181de000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:29.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f561dd67700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5618146340 0x7f56181e1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:29.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f561dd67700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56181ddaf0 0x7f56180c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:29.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f561dd67700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5618154560 con 0x7f5618145af0 2023-03-03T20:14:29.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f561dd67700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5618151c00 con 0x7f56181ddaf0 2023-03-03T20:14:29.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f561dd67700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5618152e40 con 0x7f5618146340 2023-03-03T20:14:29.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f56177fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5618145af0 0x7f56181de000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:29.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f56177fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5618145af0 0x7f56181de000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45012/0 (socket says 172.21.15.7:45012) 2023-03-03T20:14:29.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f56177fe700 1 -- 172.21.15.7:0/488022291 learned_addr learned my addr 172.21.15.7:0/488022291 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:14:29.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f5616ffd700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5618146340 0x7f56181e1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:29.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.138+0000 7f5617fff700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56181ddaf0 0x7f56180c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:29.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f56177fe700 1 -- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5618146340 msgr2=0x7f56181e1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f56177fe700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5618146340 0x7f56181e1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f56177fe700 1 -- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56181ddaf0 msgr2=0x7f56180c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f56177fe700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56181ddaf0 0x7f56180c7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f56177fe700 1 -- 172.21.15.7:0/488022291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561806f6b0 con 0x7f5618145af0 2023-03-03T20:14:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f5617fff700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56181ddaf0 0x7f56180c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:14:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f56177fe700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5618145af0 0x7f56181de000 secure :-1 s=READY pgs=3735 cs=0 l=1 rev1=1 crypto rx=0x7f5608011ad0 tx=0x7f5608011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:14:29.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f560ffff700 1 -- 172.21.15.7:0/488022291 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560800f940 con 0x7f5618145af0 2023-03-03T20:14:29.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f560ffff700 1 -- 172.21.15.7:0/488022291 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560800faa0 con 0x7f5618145af0 2023-03-03T20:14:29.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f561dd67700 1 -- 172.21.15.7:0/488022291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5618155160 con 0x7f5618145af0 2023-03-03T20:14:29.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f560ffff700 1 -- 172.21.15.7:0/488022291 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560800fe50 con 0x7f5618145af0 2023-03-03T20:14:29.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.139+0000 7f561dd67700 1 -- 172.21.15.7:0/488022291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561806f990 con 0x7f5618145af0 2023-03-03T20:14:29.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.140+0000 7f561dd67700 1 -- 172.21.15.7:0/488022291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56180c9410 con 0x7f5618145af0 2023-03-03T20:14:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.141+0000 7f560ffff700 1 -- 172.21.15.7:0/488022291 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5608014440 con 0x7f5618145af0 2023-03-03T20:14:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.141+0000 7f560ffff700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f80682f0 0x7f55f806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.141+0000 7f560ffff700 1 -- 172.21.15.7:0/488022291 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f56080869e0 con 0x7f5618145af0 2023-03-03T20:14:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.141+0000 7f5616ffd700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f80682f0 0x7f55f806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.142+0000 7f5616ffd700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f80682f0 0x7f55f806a7a0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f5618146710 tx=0x7f5600009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:14:29.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.145+0000 7f560ffff700 1 -- 172.21.15.7:0/488022291 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5608052d20 con 0x7f5618145af0 2023-03-03T20:14:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.388+0000 7f561dd67700 1 -- 172.21.15.7:0/488022291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5618074bb0 con 0x7f5618145af0 2023-03-03T20:14:29.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.389+0000 7f560ffff700 1 -- 172.21.15.7:0/488022291 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5608056340 con 0x7f5618145af0 2023-03-03T20:14:29.391 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:14:29.391 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:14:29.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 -- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f80682f0 msgr2=0x7f55f806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f80682f0 0x7f55f806a7a0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f5618146710 tx=0x7f5600009210 comp rx=0 tx=0).stop 2023-03-03T20:14:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 -- 172.21.15.7:0/488022291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5618145af0 msgr2=0x7f56181de000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5618145af0 0x7f56181de000 secure :-1 s=READY pgs=3735 cs=0 l=1 rev1=1 crypto rx=0x7f5608011ad0 tx=0x7f5608011e90 comp rx=0 tx=0).stop 2023-03-03T20:14:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 -- 172.21.15.7:0/488022291 shutdown_connections 2023-03-03T20:14:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f56181ddaf0 0x7f56180c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:29.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f55f80682f0 0x7f55f806a7a0 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:29.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5618145af0 0x7f56181de000 unknown :-1 s=CLOSED pgs=3735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:29.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 --2- 172.21.15.7:0/488022291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5618146340 0x7f56181e1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:29.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 -- 172.21.15.7:0/488022291 >> 172.21.15.7:0/488022291 conn(0x7f56180c8900 msgr2=0x7f56180ad0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:14:29.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 -- 172.21.15.7:0/488022291 shutdown_connections 2023-03-03T20:14:29.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:29.391+0000 7f561dd67700 1 -- 172.21.15.7:0/488022291 wait complete. 2023-03-03T20:14:29.395 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:14:29.405 INFO:tasks.mds_thrash.fs.[a]:waiting for 118 secs before thrashing 2023-03-03T20:14:32.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:14:32.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.860+0000 7f47d5bb8700 1 Processor -- start 2023-03-03T20:14:32.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.860+0000 7f47d5bb8700 1 -- start start 2023-03-03T20:14:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47d5bb8700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f47d014b0d0 0x7f47d0071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:32.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47d5bb8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d0071d40 0x7f47d0158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:32.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47d5bb8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d00a9c20 con 0x7f47d00aad20 2023-03-03T20:14:32.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47d5bb8700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d0154f10 con 0x7f47d014b0d0 2023-03-03T20:14:32.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47d5bb8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d00ac3c0 con 0x7f47d0071d40 2023-03-03T20:14:32.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47ceffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f47d00aad20 0x7f47d0157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39006/0 (socket says 172.21.15.7:39006) 2023-03-03T20:14:32.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47ceffd700 1 -- 172.21.15.7:0/1672409808 learned_addr learned my addr 172.21.15.7:0/1672409808 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:14:32.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47cffff700 1 --2- 172.21.15.7:0/1672409808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d0071d40 0x7f47d0158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:32.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47cf7fe700 1 --2- 172.21.15.7:0/1672409808 >> v2:172.21.15.143:3300/0 conn(0x7f47d014b0d0 0x7f47d0071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:32.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47ce7fc700 1 -- 172.21.15.7:0/1672409808 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927133644 0 0) 0x7f47d00a9c20 con 0x7f47d00aad20 2023-03-03T20:14:32.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47ce7fc700 1 -- 172.21.15.7:0/1672409808 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47b80035b0 con 0x7f47d00aad20 2023-03-03T20:14:32.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47cffff700 1 -- 172.21.15.7:0/1672409808 >> v2:172.21.15.143:3300/0 conn(0x7f47d014b0d0 msgr2=0x7f47d0071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:14:32.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47cffff700 1 --2- 172.21.15.7:0/1672409808 >> v2:172.21.15.143:3300/0 conn(0x7f47d014b0d0 0x7f47d0071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:32.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47cffff700 1 -- 172.21.15.7:0/1672409808 >> v1:172.21.15.7:6789/0 conn(0x7f47d00aad20 legacy=0x7f47d0157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:32.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47cffff700 1 -- 172.21.15.7:0/1672409808 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d006fc70 con 0x7f47d0071d40 2023-03-03T20:14:32.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.861+0000 7f47cf7fe700 1 --2- 172.21.15.7:0/1672409808 >> v2:172.21.15.143:3300/0 conn(0x7f47d014b0d0 0x7f47d0071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:14:32.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47cffff700 1 --2- 172.21.15.7:0/1672409808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d0071d40 0x7f47d0158500 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009d90 tx=0x7f47c800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0a2e44f69ee7673 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:14:32.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47ce7fc700 1 -- 172.21.15.7:0/1672409808 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c800e070 con 0x7f47d0071d40 2023-03-03T20:14:32.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47ce7fc700 1 -- 172.21.15.7:0/1672409808 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c800bb20 con 0x7f47d0071d40 2023-03-03T20:14:32.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47ce7fc700 1 -- 172.21.15.7:0/1672409808 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8012430 con 0x7f47d0071d40 2023-03-03T20:14:32.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47d5bb8700 1 -- 172.21.15.7:0/1672409808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d0071d40 msgr2=0x7f47d0158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:32.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47d5bb8700 1 --2- 172.21.15.7:0/1672409808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d0071d40 0x7f47d0158500 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f47c8009d90 tx=0x7f47c800b330 comp rx=0 tx=0).stop 2023-03-03T20:14:32.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47d5bb8700 1 -- 172.21.15.7:0/1672409808 shutdown_connections 2023-03-03T20:14:32.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47d5bb8700 1 --2- 172.21.15.7:0/1672409808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d0071d40 0x7f47d0158500 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:32.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47d5bb8700 1 --2- 172.21.15.7:0/1672409808 >> v2:172.21.15.143:3300/0 conn(0x7f47d014b0d0 0x7f47d0071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:32.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47d5bb8700 1 -- 172.21.15.7:0/1672409808 >> 172.21.15.7:0/1672409808 conn(0x7f47d00c8900 msgr2=0x7f47d00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:14:32.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47d5bb8700 1 -- 172.21.15.7:0/1672409808 shutdown_connections 2023-03-03T20:14:32.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.862+0000 7f47d5bb8700 1 -- 172.21.15.7:0/1672409808 wait complete. 2023-03-03T20:14:32.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.864+0000 7f47d5bb8700 1 Processor -- start 2023-03-03T20:14:32.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.864+0000 7f47d5bb8700 1 -- start start 2023-03-03T20:14:32.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.864+0000 7f47d5bb8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47d0071d40 0x7f47d0156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:32.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.864+0000 7f47d5bb8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47d00aad20 0x7f47d01ea720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:32.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47d5bb8700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d014b0d0 0x7f47d01eac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:32.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47d5bb8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d01eb1a0 con 0x7f47d0071d40 2023-03-03T20:14:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47d5bb8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d01e1980 con 0x7f47d00aad20 2023-03-03T20:14:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47d5bb8700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d01ee5d0 con 0x7f47d014b0d0 2023-03-03T20:14:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47d0071d40 0x7f47d0156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:32.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47d0071d40 0x7f47d0156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45030/0 (socket says 172.21.15.7:45030) 2023-03-03T20:14:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47cf7fe700 1 -- 172.21.15.7:0/654473589 learned_addr learned my addr 172.21.15.7:0/654473589 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:14:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47cffff700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d014b0d0 0x7f47d01eac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47ceffd700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47d00aad20 0x7f47d01ea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47cf7fe700 1 -- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d014b0d0 msgr2=0x7f47d01eac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47cf7fe700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d014b0d0 0x7f47d01eac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47cf7fe700 1 -- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47d00aad20 msgr2=0x7f47d01ea720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:32.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47cf7fe700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47d00aad20 0x7f47d01ea720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47cf7fe700 1 -- 172.21.15.7:0/654473589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c00094a0 con 0x7f47d0071d40 2023-03-03T20:14:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47cf7fe700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47d0071d40 0x7f47d0156b40 secure :-1 s=READY pgs=3737 cs=0 l=1 rev1=1 crypto rx=0x7f47c000ebe0 tx=0x7f47c000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:14:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47b7fff700 1 -- 172.21.15.7:0/654473589 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c0009760 con 0x7f47d0071d40 2023-03-03T20:14:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47b7fff700 1 -- 172.21.15.7:0/654473589 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c000c850 con 0x7f47d0071d40 2023-03-03T20:14:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47b7fff700 1 -- 172.21.15.7:0/654473589 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c000cbe0 con 0x7f47d0071d40 2023-03-03T20:14:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47d5bb8700 1 -- 172.21.15.7:0/654473589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c8009a70 con 0x7f47d0071d40 2023-03-03T20:14:32.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.865+0000 7f47d5bb8700 1 -- 172.21.15.7:0/654473589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d006f6b0 con 0x7f47d0071d40 2023-03-03T20:14:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.867+0000 7f47d5bb8700 1 -- 172.21.15.7:0/654473589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d0155160 con 0x7f47d0071d40 2023-03-03T20:14:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.867+0000 7f47b7fff700 1 -- 172.21.15.7:0/654473589 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f47c0019030 con 0x7f47d0071d40 2023-03-03T20:14:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.867+0000 7f47b7fff700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47bc0682f0 0x7f47bc06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:14:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.867+0000 7f47b7fff700 1 -- 172.21.15.7:0/654473589 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f47c0085f90 con 0x7f47d0071d40 2023-03-03T20:14:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.868+0000 7f47ceffd700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47bc0682f0 0x7f47bc06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:14:32.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.868+0000 7f47ceffd700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47bc0682f0 0x7f47bc06a7a0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f47d00c5ad0 tx=0x7f47c4007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:14:32.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:32.872+0000 7f47b7fff700 1 -- 172.21.15.7:0/654473589 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f47c0052330 con 0x7f47d0071d40 2023-03-03T20:14:33.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.111+0000 7f47d5bb8700 1 -- 172.21.15.7:0/654473589 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47d006f990 con 0x7f47d0071d40 2023-03-03T20:14:33.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.111+0000 7f47b7fff700 1 -- 172.21.15.7:0/654473589 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f47c0055950 con 0x7f47d0071d40 2023-03-03T20:14:33.114 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:14:33.114 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:14:33.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 -- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47bc0682f0 msgr2=0x7f47bc06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:33.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47bc0682f0 0x7f47bc06a7a0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f47d00c5ad0 tx=0x7f47c4007dc0 comp rx=0 tx=0).stop 2023-03-03T20:14:33.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 -- 172.21.15.7:0/654473589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47d0071d40 msgr2=0x7f47d0156b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:14:33.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47d0071d40 0x7f47d0156b40 secure :-1 s=READY pgs=3737 cs=0 l=1 rev1=1 crypto rx=0x7f47c000ebe0 tx=0x7f47c000eef0 comp rx=0 tx=0).stop 2023-03-03T20:14:33.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 -- 172.21.15.7:0/654473589 shutdown_connections 2023-03-03T20:14:33.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f47d014b0d0 0x7f47d01eac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:33.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f47bc0682f0 0x7f47bc06a7a0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:33.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47d0071d40 0x7f47d0156b40 unknown :-1 s=CLOSED pgs=3737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:33.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 --2- 172.21.15.7:0/654473589 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47d00aad20 0x7f47d01ea720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:14:33.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 -- 172.21.15.7:0/654473589 >> 172.21.15.7:0/654473589 conn(0x7f47d00c8900 msgr2=0x7f47d00c4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:14:33.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 -- 172.21.15.7:0/654473589 shutdown_connections 2023-03-03T20:14:33.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:14:33.114+0000 7f47d5bb8700 1 -- 172.21.15.7:0/654473589 wait complete. 2023-03-03T20:14:33.117 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:14:33.127 INFO:tasks.mds_thrash.fs.[b]:waiting for 79 secs before thrashing 2023-03-03T20:14:57.953 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:14:57.957 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:15:28.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:15:28.073 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:15:52.129 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:15:52.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.266+0000 7fcc0bbfe700 1 Processor -- start 2023-03-03T20:15:52.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.266+0000 7fcc0bbfe700 1 -- start start 2023-03-03T20:15:52.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc0bbfe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc0406c6c0 0x7fcc040736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:15:52.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc0bbfe700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc04157ff0 con 0x7fcc04151cd0 2023-03-03T20:15:52.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc0bbfe700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0406d640 con 0x7fcc040c2e30 2023-03-03T20:15:52.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc0bbfe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0406cda0 con 0x7fcc0406c6c0 2023-03-03T20:15:52.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc0999a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcc04151cd0 0x7fcc04154510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39048/0 (socket says 172.21.15.7:39048) 2023-03-03T20:15:52.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc0999a700 1 -- 172.21.15.7:0/2022345623 learned_addr learned my addr 172.21.15.7:0/2022345623 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:15:52.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711033480 0 0) 0x7fcc04157ff0 con 0x7fcc04151cd0 2023-03-03T20:15:52.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc0a19b700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc0406c6c0 0x7fcc040736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:15:52.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf00035b0 con 0x7fcc04151cd0 2023-03-03T20:15:52.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 123461776 0 0) 0x7fcbf00035b0 con 0x7fcc04151cd0 2023-03-03T20:15:52.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc0406c6c0 msgr2=0x7fcc040736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:15:52.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc08998700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc0406c6c0 0x7fcc040736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:15:52.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.267+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 >> v1:172.21.15.143:6789/0 conn(0x7fcc040c2e30 legacy=0x7fcc04158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:15:52.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0406d080 con 0x7fcc04151cd0 2023-03-03T20:15:52.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fcbf4001670 con 0x7fcc04151cd0 2023-03-03T20:15:52.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 >> v1:172.21.15.7:6789/0 conn(0x7fcc04151cd0 legacy=0x7fcc04154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:15:52.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbf0003aa0 0x7fcbf0003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:15:52.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc0406c6c0 0x7fcbf0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:15:52.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 0x7fcbf0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:15:52.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0406cda0 con 0x7fcbf0004d90 2023-03-03T20:15:52.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf000bc90 con 0x7fcbf0003aa0 2023-03-03T20:15:52.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf000be50 con 0x7fcc0406c6c0 2023-03-03T20:15:52.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc0a19b700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 0x7fcbf0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:15:52.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc0999a700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbf0003aa0 0x7fcbf0003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:15:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc09199700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc0406c6c0 0x7fcbf0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:15:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc0a19b700 1 -- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc0406c6c0 msgr2=0x7fcbf0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:15:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc0a19b700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc0406c6c0 0x7fcbf0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:15:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc0a19b700 1 -- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbf0003aa0 msgr2=0x7fcbf0003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:15:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc0a19b700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcbf0003aa0 0x7fcbf0003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:15:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc0a19b700 1 -- 172.21.15.7:0/2022345623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf00035b0 con 0x7fcbf0004d90 2023-03-03T20:15:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc0a19b700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 0x7fcbf0005200 secure :-1 s=READY pgs=3739 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009fc0 tx=0x7fcbf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948880bdc958d4d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:15:52.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc00ec30 con 0x7fcbf0004d90 2023-03-03T20:15:52.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc00f040 con 0x7fcbf0004d90 2023-03-03T20:15:52.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.268+0000 7fcc08998700 1 -- 172.21.15.7:0/2022345623 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc014480 con 0x7fcbf0004d90 2023-03-03T20:15:52.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.269+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/2022345623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 msgr2=0x7fcbf0005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:15:52.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.269+0000 7fcc0bbfe700 1 --2- 172.21.15.7:0/2022345623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 0x7fcbf0005200 secure :-1 s=READY pgs=3739 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009fc0 tx=0x7fcbf0005880 comp rx=0 tx=0).stop 2023-03-03T20:15:52.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.269+0000 7fcc0a19b700 1 -- 172.21.15.7:0/2022345623 reap_dead start 2023-03-03T20:15:52.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.269+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/2022345623 shutdown_connections 2023-03-03T20:15:52.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.269+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/2022345623 >> 172.21.15.7:0/2022345623 conn(0x7fcc040a94d0 msgr2=0x7fcc040a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:15:52.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.269+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/2022345623 shutdown_connections 2023-03-03T20:15:52.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.269+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/2022345623 wait complete. 2023-03-03T20:15:52.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.270+0000 7fcc0bbfe700 1 Processor -- start 2023-03-03T20:15:52.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.270+0000 7fcc0bbfe700 1 -- start start 2023-03-03T20:15:52.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0bbfe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 0x7fcc04157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:15:52.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0bbfe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc040c5460 0x7fcc04152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0bbfe700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc041528b0 0x7fcc041ed450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0bbfe700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc041543d0 con 0x7fcbf0004d90 2023-03-03T20:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0bbfe700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc04156910 con 0x7fcc040c5460 2023-03-03T20:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0bbfe700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc04155900 con 0x7fcc041528b0 2023-03-03T20:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0999a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 0x7fcc04157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0999a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 0x7fcc04157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45082/0 (socket says 172.21.15.7:45082) 2023-03-03T20:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0999a700 1 -- 172.21.15.7:0/3270613886 learned_addr learned my addr 172.21.15.7:0/3270613886 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0a19b700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc041528b0 0x7fcc041ed450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc09199700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc040c5460 0x7fcc04152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0999a700 1 -- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc041528b0 msgr2=0x7fcc041ed450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0999a700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc041528b0 0x7fcc041ed450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0999a700 1 -- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc040c5460 msgr2=0x7fcc04152370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0999a700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc040c5460 0x7fcc04152370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:15:52.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.271+0000 7fcc0999a700 1 -- 172.21.15.7:0/3270613886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc04157c80 con 0x7fcbf0004d90 2023-03-03T20:15:52.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.272+0000 7fcc0999a700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 0x7fcc04157510 secure :-1 s=READY pgs=3740 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40072f0 tx=0x7fcbf4007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:15:52.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.272+0000 7fcbfa7fc700 1 -- 172.21.15.7:0/3270613886 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4011580 con 0x7fcbf0004d90 2023-03-03T20:15:52.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.272+0000 7fcbfa7fc700 1 -- 172.21.15.7:0/3270613886 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf4011720 con 0x7fcbf0004d90 2023-03-03T20:15:52.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.272+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/3270613886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0406d360 con 0x7fcbf0004d90 2023-03-03T20:15:52.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.272+0000 7fcbfa7fc700 1 -- 172.21.15.7:0/3270613886 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4011ad0 con 0x7fcbf0004d90 2023-03-03T20:15:52.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.272+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/3270613886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc040736f0 con 0x7fcbf0004d90 2023-03-03T20:15:52.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.273+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/3270613886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc04074bb0 con 0x7fcbf0004d90 2023-03-03T20:15:52.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.273+0000 7fcbfa7fc700 1 -- 172.21.15.7:0/3270613886 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcbf4011c30 con 0x7fcbf0004d90 2023-03-03T20:15:52.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.274+0000 7fcbfa7fc700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbe80682c0 0x7fcbe806a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:15:52.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.274+0000 7fcbfa7fc700 1 -- 172.21.15.7:0/3270613886 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcbf4086200 con 0x7fcbf0004d90 2023-03-03T20:15:52.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.274+0000 7fcc09199700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbe80682c0 0x7fcbe806a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:15:52.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.275+0000 7fcc09199700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbe80682c0 0x7fcbe806a770 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fcc00007fd0 tx=0x7fcc00009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:15:52.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.278+0000 7fcbfa7fc700 1 -- 172.21.15.7:0/3270613886 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcbf40525a0 con 0x7fcbf0004d90 2023-03-03T20:15:52.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.516+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/3270613886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc040a9fb0 con 0x7fcbf0004d90 2023-03-03T20:15:52.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.517+0000 7fcbfa7fc700 1 -- 172.21.15.7:0/3270613886 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcbf4055bc0 con 0x7fcbf0004d90 2023-03-03T20:15:52.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:15:52.518 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:15:52.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbe80682c0 msgr2=0x7fcbe806a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:15:52.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbe80682c0 0x7fcbe806a770 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fcc00007fd0 tx=0x7fcc00009040 comp rx=0 tx=0).stop 2023-03-03T20:15:52.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/3270613886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 msgr2=0x7fcc04157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:15:52.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 0x7fcc04157510 secure :-1 s=READY pgs=3740 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40072f0 tx=0x7fcbf4007600 comp rx=0 tx=0).stop 2023-03-03T20:15:52.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/3270613886 shutdown_connections 2023-03-03T20:15:52.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcc041528b0 0x7fcc041ed450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:15:52.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcbe80682c0 0x7fcbe806a770 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:15:52.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcbf0004d90 0x7fcc04157510 unknown :-1 s=CLOSED pgs=3740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:15:52.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 --2- 172.21.15.7:0/3270613886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc040c5460 0x7fcc04152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:15:52.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/3270613886 >> 172.21.15.7:0/3270613886 conn(0x7fcc040a94d0 msgr2=0x7fcc040715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:15:52.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/3270613886 shutdown_connections 2023-03-03T20:15:52.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:15:52.519+0000 7fcc0bbfe700 1 -- 172.21.15.7:0/3270613886 wait complete. 2023-03-03T20:15:52.522 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:15:52.532 INFO:tasks.mds_thrash.fs.[b]:waiting for 114 secs before thrashing 2023-03-03T20:15:58.188 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:15:58.191 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:16:27.406 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:16:27.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.544+0000 7fa542321700 1 Processor -- start 2023-03-03T20:16:27.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.545+0000 7fa542321700 1 -- start start 2023-03-03T20:16:27.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.545+0000 7fa542321700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa53c155390 0x7fa53c0c3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:27.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.545+0000 7fa542321700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fa53c06e020 0x7fa53c071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:27.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.545+0000 7fa542321700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53c154910 con 0x7fa53c072520 2023-03-03T20:16:27.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.545+0000 7fa542321700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c070b90 con 0x7fa53c155390 2023-03-03T20:16:27.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.545+0000 7fa542321700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c151320 con 0x7fa53c06e020 2023-03-03T20:16:27.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.545+0000 7fa53bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa53c155390 0x7fa53c0c3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:27.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.545+0000 7fa5408be700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa53c072520 0x7fa53c151770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39086/0 (socket says 172.21.15.7:39086) 2023-03-03T20:16:27.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.545+0000 7fa5408be700 1 -- 172.21.15.7:0/2116685104 learned_addr learned my addr 172.21.15.7:0/2116685104 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:16:27.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53b7fe700 1 --2- 172.21.15.7:0/2116685104 >> v2:172.21.15.143:3300/0 conn(0x7fa53c06e020 0x7fa53c071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:27.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53bfff700 1 -- 172.21.15.7:0/2116685104 >> v2:172.21.15.143:3300/0 conn(0x7fa53c06e020 msgr2=0x7fa53c071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:16:27.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53bfff700 1 --2- 172.21.15.7:0/2116685104 >> v2:172.21.15.143:3300/0 conn(0x7fa53c06e020 0x7fa53c071fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:27.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53bfff700 1 -- 172.21.15.7:0/2116685104 >> v1:172.21.15.7:6789/0 conn(0x7fa53c072520 legacy=0x7fa53c151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:27.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53bfff700 1 -- 172.21.15.7:0/2116685104 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c0702a0 con 0x7fa53c155390 2023-03-03T20:16:27.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53affd700 1 -- 172.21.15.7:0/2116685104 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201936327 0 0) 0x7fa53c154910 con 0x7fa53c072520 2023-03-03T20:16:27.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53b7fe700 1 --2- 172.21.15.7:0/2116685104 >> v2:172.21.15.143:3300/0 conn(0x7fa53c06e020 0x7fa53c071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:16:27.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53bfff700 1 --2- 172.21.15.7:0/2116685104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c155390 0x7fa53c0c3e30 secure :-1 s=READY pgs=3742 cs=0 l=1 rev1=1 crypto rx=0x7fa53c0ac9e0 tx=0x7fa52c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754ab62322ce4fac server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:16:27.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53affd700 1 -- 172.21.15.7:0/2116685104 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c011830 con 0x7fa53c155390 2023-03-03T20:16:27.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53affd700 1 -- 172.21.15.7:0/2116685104 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52c011990 con 0x7fa53c155390 2023-03-03T20:16:27.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.546+0000 7fa53affd700 1 -- 172.21.15.7:0/2116685104 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c011d20 con 0x7fa53c155390 2023-03-03T20:16:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.547+0000 7fa542321700 1 -- 172.21.15.7:0/2116685104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c155390 msgr2=0x7fa53c0c3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.547+0000 7fa542321700 1 --2- 172.21.15.7:0/2116685104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c155390 0x7fa53c0c3e30 secure :-1 s=READY pgs=3742 cs=0 l=1 rev1=1 crypto rx=0x7fa53c0ac9e0 tx=0x7fa52c009b20 comp rx=0 tx=0).stop 2023-03-03T20:16:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.547+0000 7fa542321700 1 -- 172.21.15.7:0/2116685104 shutdown_connections 2023-03-03T20:16:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.547+0000 7fa542321700 1 --2- 172.21.15.7:0/2116685104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c155390 0x7fa53c0c3e30 unknown :-1 s=CLOSED pgs=3742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.547+0000 7fa542321700 1 --2- 172.21.15.7:0/2116685104 >> v2:172.21.15.143:3300/0 conn(0x7fa53c06e020 0x7fa53c071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:27.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.547+0000 7fa542321700 1 -- 172.21.15.7:0/2116685104 >> 172.21.15.7:0/2116685104 conn(0x7fa53c06c110 msgr2=0x7fa53c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:16:27.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.547+0000 7fa542321700 1 -- 172.21.15.7:0/2116685104 shutdown_connections 2023-03-03T20:16:27.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.547+0000 7fa542321700 1 -- 172.21.15.7:0/2116685104 wait complete. 2023-03-03T20:16:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.548+0000 7fa542321700 1 Processor -- start 2023-03-03T20:16:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.548+0000 7fa542321700 1 -- start start 2023-03-03T20:16:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa542321700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c06e020 0x7fa53c1e9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa542321700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c072520 0x7fa53c1ed9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:27.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa542321700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c155390 0x7fa53c14c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:27.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa542321700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c0c0d10 con 0x7fa53c072520 2023-03-03T20:16:27.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa542321700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c1e9f80 con 0x7fa53c06e020 2023-03-03T20:16:27.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa542321700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c14e790 con 0x7fa53c155390 2023-03-03T20:16:27.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa53b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c072520 0x7fa53c1ed9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:27.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa53b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c072520 0x7fa53c1ed9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45114/0 (socket says 172.21.15.7:45114) 2023-03-03T20:16:27.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa53b7fe700 1 -- 172.21.15.7:0/1415042291 learned_addr learned my addr 172.21.15.7:0/1415042291 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:16:27.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa5408be700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c155390 0x7fa53c14c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:27.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa53bfff700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c06e020 0x7fa53c1e9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:27.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa53b7fe700 1 -- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c155390 msgr2=0x7fa53c14c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:27.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa53b7fe700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c155390 0x7fa53c14c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:27.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa53b7fe700 1 -- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c06e020 msgr2=0x7fa53c1e9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:27.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa53b7fe700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c06e020 0x7fa53c1e9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:27.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.549+0000 7fa53b7fe700 1 -- 172.21.15.7:0/1415042291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c0094a0 con 0x7fa53c072520 2023-03-03T20:16:27.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.550+0000 7fa53b7fe700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c072520 0x7fa53c1ed9c0 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 crypto rx=0x7fa53000c780 tx=0x7fa53000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:16:27.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.550+0000 7fa538ff9700 1 -- 172.21.15.7:0/1415042291 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530011800 con 0x7fa53c072520 2023-03-03T20:16:27.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.550+0000 7fa538ff9700 1 -- 172.21.15.7:0/1415042291 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa530011960 con 0x7fa53c072520 2023-03-03T20:16:27.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.550+0000 7fa542321700 1 -- 172.21.15.7:0/1415042291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53c070820 con 0x7fa53c072520 2023-03-03T20:16:27.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.550+0000 7fa538ff9700 1 -- 172.21.15.7:0/1415042291 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530011cd0 con 0x7fa53c072520 2023-03-03T20:16:27.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.550+0000 7fa542321700 1 -- 172.21.15.7:0/1415042291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c070540 con 0x7fa53c072520 2023-03-03T20:16:27.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.551+0000 7fa542321700 1 -- 172.21.15.7:0/1415042291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c155760 con 0x7fa53c072520 2023-03-03T20:16:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.551+0000 7fa538ff9700 1 -- 172.21.15.7:0/1415042291 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa530020070 con 0x7fa53c072520 2023-03-03T20:16:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.552+0000 7fa538ff9700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 0x7fa52806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.552+0000 7fa538ff9700 1 -- 172.21.15.7:0/1415042291 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa530086010 con 0x7fa53c072520 2023-03-03T20:16:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.552+0000 7fa53bfff700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 0x7fa52806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:27.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.552+0000 7fa53bfff700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 0x7fa52806a7a0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0090b0 tx=0x7fa52c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:16:27.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.556+0000 7fa538ff9700 1 -- 172.21.15.7:0/1415042291 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa530052330 con 0x7fa53c072520 2023-03-03T20:16:27.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.794+0000 7fa542321700 1 -- 172.21.15.7:0/1415042291 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa53c1edf00 con 0x7fa53c072520 2023-03-03T20:16:27.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.794+0000 7fa538ff9700 1 -- 172.21.15.7:0/1415042291 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa53c1edf00 con 0x7fa53c072520 2023-03-03T20:16:27.795 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:16:27.795 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:16:27.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.796+0000 7fa542321700 1 -- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 msgr2=0x7fa52806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:27.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.796+0000 7fa542321700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 0x7fa52806a7a0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0090b0 tx=0x7fa52c01c040 comp rx=0 tx=0).stop 2023-03-03T20:16:27.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.796+0000 7fa542321700 1 -- 172.21.15.7:0/1415042291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c072520 msgr2=0x7fa53c1ed9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:27.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.796+0000 7fa542321700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c072520 0x7fa53c1ed9c0 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 crypto rx=0x7fa53000c780 tx=0x7fa53000cb40 comp rx=0 tx=0).stop 2023-03-03T20:16:27.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.796+0000 7fa542321700 1 -- 172.21.15.7:0/1415042291 shutdown_connections 2023-03-03T20:16:27.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.796+0000 7fa542321700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa53c155390 0x7fa53c14c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:27.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.796+0000 7fa542321700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa53c06e020 0x7fa53c1e9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:27.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.797+0000 7fa542321700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa5280682f0 0x7fa52806a7a0 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:27.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.797+0000 7fa542321700 1 --2- 172.21.15.7:0/1415042291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c072520 0x7fa53c1ed9c0 unknown :-1 s=CLOSED pgs=3743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:27.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.797+0000 7fa542321700 1 -- 172.21.15.7:0/1415042291 >> 172.21.15.7:0/1415042291 conn(0x7fa53c06c110 msgr2=0x7fa53c150980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:16:27.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.797+0000 7fa542321700 1 -- 172.21.15.7:0/1415042291 shutdown_connections 2023-03-03T20:16:27.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:27.797+0000 7fa542321700 1 -- 172.21.15.7:0/1415042291 wait complete. 2023-03-03T20:16:27.799 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:16:27.809 INFO:tasks.mds_thrash.fs.[a]:waiting for 21 secs before thrashing 2023-03-03T20:16:28.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:16:28.274 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:16:48.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:16:48.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.949+0000 7f19995da700 1 Processor -- start 2023-03-03T20:16:48.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.949+0000 7f19995da700 1 -- start start 2023-03-03T20:16:48.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.950+0000 7f19995da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f199406c2f0 0x7f19941584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:48.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.950+0000 7f19995da700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199406d780 con 0x7f199406bea0 2023-03-03T20:16:48.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.950+0000 7f19995da700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199406c770 con 0x7f1994151ab0 2023-03-03T20:16:48.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.950+0000 7f19995da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1994158a10 con 0x7f199406c2f0 2023-03-03T20:16:48.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.950+0000 7f19927fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f199406bea0 0x7f1994157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39118/0 (socket says 172.21.15.7:39118) 2023-03-03T20:16:48.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.950+0000 7f19927fc700 1 -- 172.21.15.7:0/2669097260 learned_addr learned my addr 172.21.15.7:0/2669097260 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:16:48.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.950+0000 7f19937fe700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f199406c2f0 0x7f19941584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:48.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.950+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631012039 0 0) 0x7f199406d780 con 0x7f199406bea0 2023-03-03T20:16:48.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f197c0035b0 con 0x7f199406bea0 2023-03-03T20:16:48.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2331999758 0 0) 0x7f197c0035b0 con 0x7f199406bea0 2023-03-03T20:16:48.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f199406c2f0 msgr2=0x7f19941584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:16:48.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f199406c2f0 0x7f19941584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:48.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 >> v1:172.21.15.143:6789/0 conn(0x7f1994151ab0 legacy=0x7f19940736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:48.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199406cb70 con 0x7f199406bea0 2023-03-03T20:16:48.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f1984001670 con 0x7f199406bea0 2023-03-03T20:16:48.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 >> v1:172.21.15.7:6789/0 conn(0x7f199406bea0 legacy=0x7f1994157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:48.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f199406c2f0 0x7f197c003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:48.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f197c004020 0x7f197c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:48.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f197c004d90 0x7f197c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:48.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994158a10 con 0x7f197c004020 2023-03-03T20:16:48.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c00bc00 con 0x7f197c004d90 2023-03-03T20:16:48.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197c00bdc0 con 0x7f199406c2f0 2023-03-03T20:16:48.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f19927fc700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f197c004020 0x7f197c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:48.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f1992ffd700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f199406c2f0 0x7f197c003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:48.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f19937fe700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f197c004d90 0x7f197c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:48.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f19927fc700 1 -- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f199406c2f0 msgr2=0x7f197c003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:16:48.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f19927fc700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f199406c2f0 0x7f197c003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:48.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f19927fc700 1 -- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f197c004d90 msgr2=0x7f197c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:16:48.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f19927fc700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f197c004d90 0x7f197c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:48.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.951+0000 7f19927fc700 1 -- 172.21.15.7:0/2669097260 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197c0035b0 con 0x7f197c004020 2023-03-03T20:16:48.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f19927fc700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f197c004020 0x7f197c004850 secure :-1 s=READY pgs=3745 cs=0 l=1 rev1=1 crypto rx=0x7f198400c270 tx=0x7f197c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1244ccc57cac95fb server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:16:48.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1984007c10 con 0x7f197c004020 2023-03-03T20:16:48.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198400cd90 con 0x7f197c004020 2023-03-03T20:16:48.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f1991ffb700 1 -- 172.21.15.7:0/2669097260 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1984005ba0 con 0x7f197c004020 2023-03-03T20:16:48.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f19995da700 1 -- 172.21.15.7:0/2669097260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f197c004020 msgr2=0x7f197c004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:48.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f19995da700 1 --2- 172.21.15.7:0/2669097260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f197c004020 0x7f197c004850 secure :-1 s=READY pgs=3745 cs=0 l=1 rev1=1 crypto rx=0x7f198400c270 tx=0x7f197c005880 comp rx=0 tx=0).stop 2023-03-03T20:16:48.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f19937fe700 1 -- 172.21.15.7:0/2669097260 reap_dead start 2023-03-03T20:16:48.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f19995da700 1 -- 172.21.15.7:0/2669097260 shutdown_connections 2023-03-03T20:16:48.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f19995da700 1 -- 172.21.15.7:0/2669097260 >> 172.21.15.7:0/2669097260 conn(0x7f19940a94d0 msgr2=0x7f19940a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:16:48.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f19995da700 1 -- 172.21.15.7:0/2669097260 shutdown_connections 2023-03-03T20:16:48.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.952+0000 7f19995da700 1 -- 172.21.15.7:0/2669097260 wait complete. 2023-03-03T20:16:48.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.954+0000 7f19995da700 1 Processor -- start 2023-03-03T20:16:48.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.954+0000 7f19995da700 1 -- start start 2023-03-03T20:16:48.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.954+0000 7f19995da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f197c004020 0x7f19940c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:48.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.954+0000 7f19995da700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f199414bf60 0x7f1994146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:48.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.954+0000 7f19995da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1994146880 0x7f19941eb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:48.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.954+0000 7f19995da700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199414bbc0 con 0x7f1994146880 2023-03-03T20:16:48.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.954+0000 7f19995da700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19941ece60 con 0x7f199414bf60 2023-03-03T20:16:48.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.954+0000 7f19995da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199414b8e0 con 0x7f197c004020 2023-03-03T20:16:48.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.954+0000 7f19937fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1994146880 0x7f19941eb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:48.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19937fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1994146880 0x7f19941eb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45154/0 (socket says 172.21.15.7:45154) 2023-03-03T20:16:48.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19937fe700 1 -- 172.21.15.7:0/3442591948 learned_addr learned my addr 172.21.15.7:0/3442591948 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:16:48.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f1992ffd700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f197c004020 0x7f19940c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:48.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19927fc700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f199414bf60 0x7f1994146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:48.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19937fe700 1 -- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f197c004020 msgr2=0x7f19940c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:48.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19937fe700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f197c004020 0x7f19940c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:48.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19937fe700 1 -- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f199414bf60 msgr2=0x7f1994146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:48.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19937fe700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f199414bf60 0x7f1994146340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:48.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19937fe700 1 -- 172.21.15.7:0/3442591948 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199406d130 con 0x7f1994146880 2023-03-03T20:16:48.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19937fe700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1994146880 0x7f19941eb380 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 crypto rx=0x7f199406b780 tx=0x7f198c011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:16:48.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f197b7fe700 1 -- 172.21.15.7:0/3442591948 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c00e040 con 0x7f1994146880 2023-03-03T20:16:48.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f197b7fe700 1 -- 172.21.15.7:0/3442591948 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c00f040 con 0x7f1994146880 2023-03-03T20:16:48.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19995da700 1 -- 172.21.15.7:0/3442591948 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199406ce50 con 0x7f1994146880 2023-03-03T20:16:48.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f197b7fe700 1 -- 172.21.15.7:0/3442591948 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c00fbe0 con 0x7f1994146880 2023-03-03T20:16:48.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.955+0000 7f19995da700 1 -- 172.21.15.7:0/3442591948 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199406d410 con 0x7f1994146880 2023-03-03T20:16:48.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.956+0000 7f19995da700 1 -- 172.21.15.7:0/3442591948 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199414c3e0 con 0x7f1994146880 2023-03-03T20:16:48.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.957+0000 7f197b7fe700 1 -- 172.21.15.7:0/3442591948 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f198c00fd40 con 0x7f1994146880 2023-03-03T20:16:48.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.957+0000 7f197b7fe700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 0x7f198006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:16:48.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.957+0000 7f197b7fe700 1 -- 172.21.15.7:0/3442591948 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f198c086a70 con 0x7f1994146880 2023-03-03T20:16:48.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.957+0000 7f1992ffd700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 0x7f198006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:16:48.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.958+0000 7f1992ffd700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 0x7f198006a7a0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f1988007fd0 tx=0x7f1988009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:16:48.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:48.962+0000 7f197b7fe700 1 -- 172.21.15.7:0/3442591948 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f199414c3e0 con 0x7f1994146880 2023-03-03T20:16:49.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.196+0000 7f19995da700 1 -- 172.21.15.7:0/3442591948 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f199414c3e0 con 0x7f1994146880 2023-03-03T20:16:49.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.197+0000 7f197b7fe700 1 -- 172.21.15.7:0/3442591948 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f199414c3e0 con 0x7f1994146880 2023-03-03T20:16:49.200 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:16:49.201 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:16:49.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 -- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 msgr2=0x7f198006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:49.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 0x7f198006a7a0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f1988007fd0 tx=0x7f1988009040 comp rx=0 tx=0).stop 2023-03-03T20:16:49.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 -- 172.21.15.7:0/3442591948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1994146880 msgr2=0x7f19941eb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:16:49.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1994146880 0x7f19941eb380 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 crypto rx=0x7f199406b780 tx=0x7f198c011e20 comp rx=0 tx=0).stop 2023-03-03T20:16:49.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 -- 172.21.15.7:0/3442591948 shutdown_connections 2023-03-03T20:16:49.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f19800682f0 0x7f198006a7a0 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:49.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1994146880 0x7f19941eb380 unknown :-1 s=CLOSED pgs=3746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:49.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f197c004020 0x7f19940c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:49.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 --2- 172.21.15.7:0/3442591948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f199414bf60 0x7f1994146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:16:49.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 -- 172.21.15.7:0/3442591948 >> 172.21.15.7:0/3442591948 conn(0x7f19940a94d0 msgr2=0x7f1994156840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:16:49.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 -- 172.21.15.7:0/3442591948 shutdown_connections 2023-03-03T20:16:49.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:16:49.199+0000 7f19995da700 1 -- 172.21.15.7:0/3442591948 wait complete. 2023-03-03T20:16:49.203 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:16:49.213 INFO:tasks.mds_thrash.fs.[a]:waiting for 46 secs before thrashing 2023-03-03T20:16:58.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:16:58.493 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:17:28.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:17:28.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:17:35.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:17:35.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.351+0000 7f354fb6f700 1 Processor -- start 2023-03-03T20:17:35.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.351+0000 7f354fb6f700 1 -- start start 2023-03-03T20:17:35.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.352+0000 7f354fb6f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3548071e80 0x7f3548151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:35.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.352+0000 7f354fb6f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35481510d0 con 0x7f354814e900 2023-03-03T20:17:35.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.352+0000 7f354fb6f700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35480c3740 con 0x7f3548072250 2023-03-03T20:17:35.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.352+0000 7f354fb6f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3548070670 con 0x7f3548071e80 2023-03-03T20:17:35.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.352+0000 7f354e10c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f354814e900 0x7f354815ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39176/0 (socket says 172.21.15.7:39176) 2023-03-03T20:17:35.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.352+0000 7f354e10c700 1 -- 172.21.15.7:0/3195057927 learned_addr learned my addr 172.21.15.7:0/3195057927 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:17:35.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354d90b700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3548071e80 0x7f3548151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:35.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962252766 0 0) 0x7f35481510d0 con 0x7f354814e900 2023-03-03T20:17:35.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35340035b0 con 0x7f354814e900 2023-03-03T20:17:35.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3383996831 0 0) 0x7f35340035b0 con 0x7f354814e900 2023-03-03T20:17:35.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3548071e80 msgr2=0x7f3548151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:17:35.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3548071e80 0x7f3548151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:35.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 >> v1:172.21.15.143:6789/0 conn(0x7f3548072250 legacy=0x7f354814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:35.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35480c39e0 con 0x7f354814e900 2023-03-03T20:17:35.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f35400019f0 con 0x7f354814e900 2023-03-03T20:17:35.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 >> v1:172.21.15.7:6789/0 conn(0x7f354814e900 legacy=0x7f354815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:35.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3548071e80 0x7f3534003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:35.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534004020 0x7f3534004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:35.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3534004d90 0x7f3534005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:35.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3548070670 con 0x7f3534004d90 2023-03-03T20:17:35.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353400bc00 con 0x7f3534004020 2023-03-03T20:17:35.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353400bdc0 con 0x7f3548071e80 2023-03-03T20:17:35.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354e10c700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3534004d90 0x7f3534005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:35.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354d90b700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3548071e80 0x7f3534003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:35.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.353+0000 7f354d10a700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534004020 0x7f3534004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:35.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354e10c700 1 -- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3548071e80 msgr2=0x7f3534003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:17:35.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354e10c700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f3548071e80 0x7f3534003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:35.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354e10c700 1 -- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534004020 msgr2=0x7f3534004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:17:35.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354e10c700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534004020 0x7f3534004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:35.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354e10c700 1 -- 172.21.15.7:0/3195057927 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35340035b0 con 0x7f3534004d90 2023-03-03T20:17:35.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354e10c700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3534004d90 0x7f3534005200 secure :-1 s=READY pgs=3748 cs=0 l=1 rev1=1 crypto rx=0x7f354000a410 tx=0x7f3534005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c378ad41c18df26d server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:17:35.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540010690 con 0x7f3534004d90 2023-03-03T20:17:35.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35400107f0 con 0x7f3534004d90 2023-03-03T20:17:35.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354c909700 1 -- 172.21.15.7:0/3195057927 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540010ba0 con 0x7f3534004d90 2023-03-03T20:17:35.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354fb6f700 1 -- 172.21.15.7:0/3195057927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3534004d90 msgr2=0x7f3534005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:35.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354fb6f700 1 --2- 172.21.15.7:0/3195057927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3534004d90 0x7f3534005200 secure :-1 s=READY pgs=3748 cs=0 l=1 rev1=1 crypto rx=0x7f354000a410 tx=0x7f3534005880 comp rx=0 tx=0).stop 2023-03-03T20:17:35.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354e10c700 1 -- 172.21.15.7:0/3195057927 reap_dead start 2023-03-03T20:17:35.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.354+0000 7f354fb6f700 1 -- 172.21.15.7:0/3195057927 shutdown_connections 2023-03-03T20:17:35.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.355+0000 7f354fb6f700 1 -- 172.21.15.7:0/3195057927 >> 172.21.15.7:0/3195057927 conn(0x7f35480a94d0 msgr2=0x7f35480a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:17:35.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.355+0000 7f354fb6f700 1 -- 172.21.15.7:0/3195057927 shutdown_connections 2023-03-03T20:17:35.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.355+0000 7f354fb6f700 1 -- 172.21.15.7:0/3195057927 wait complete. 2023-03-03T20:17:35.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.356+0000 7f354fb6f700 1 Processor -- start 2023-03-03T20:17:35.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.356+0000 7f354fb6f700 1 -- start start 2023-03-03T20:17:35.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.356+0000 7f354fb6f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35481da9a0 0x7f3548147bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:35.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.356+0000 7f354fb6f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35481dad70 0x7f35481e6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.356+0000 7f354fb6f700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35481e92e0 0x7f35481e6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.356+0000 7f354fb6f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35481e5e70 con 0x7f35481dad70 2023-03-03T20:17:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.356+0000 7f354fb6f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35481e9850 con 0x7f35481da9a0 2023-03-03T20:17:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.356+0000 7f354fb6f700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35481e9b30 con 0x7f35481e92e0 2023-03-03T20:17:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354d10a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35481dad70 0x7f35481e6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354d10a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35481dad70 0x7f35481e6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45206/0 (socket says 172.21.15.7:45206) 2023-03-03T20:17:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354d10a700 1 -- 172.21.15.7:0/159145204 learned_addr learned my addr 172.21.15.7:0/159145204 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:17:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354e10c700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35481e92e0 0x7f35481e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354d90b700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35481da9a0 0x7f3548147bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354d10a700 1 -- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35481e92e0 msgr2=0x7f35481e6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354d10a700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35481e92e0 0x7f35481e6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354d10a700 1 -- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35481da9a0 msgr2=0x7f3548147bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354d10a700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35481da9a0 0x7f3548147bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354d10a700 1 -- 172.21.15.7:0/159145204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354806c9a0 con 0x7f35481dad70 2023-03-03T20:17:35.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354d10a700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35481dad70 0x7f35481e6300 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 crypto rx=0x7f353c00f820 tx=0x7f353c00fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:17:35.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f353a7fc700 1 -- 172.21.15.7:0/159145204 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c0052e0 con 0x7f35481dad70 2023-03-03T20:17:35.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f353a7fc700 1 -- 172.21.15.7:0/159145204 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c005440 con 0x7f35481dad70 2023-03-03T20:17:35.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354fb6f700 1 -- 172.21.15.7:0/159145204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3548151300 con 0x7f35481dad70 2023-03-03T20:17:35.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f353a7fc700 1 -- 172.21.15.7:0/159145204 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c0038a0 con 0x7f35481dad70 2023-03-03T20:17:35.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.357+0000 7f354fb6f700 1 -- 172.21.15.7:0/159145204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354806cb30 con 0x7f35481dad70 2023-03-03T20:17:35.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.358+0000 7f354fb6f700 1 -- 172.21.15.7:0/159145204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3548074bb0 con 0x7f35481dad70 2023-03-03T20:17:35.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.359+0000 7f353a7fc700 1 -- 172.21.15.7:0/159145204 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f353c003a00 con 0x7f35481dad70 2023-03-03T20:17:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.359+0000 7f353a7fc700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f352c0682f0 0x7f352c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.359+0000 7f353a7fc700 1 -- 172.21.15.7:0/159145204 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f353c086040 con 0x7f35481dad70 2023-03-03T20:17:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.360+0000 7f354d90b700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f352c0682f0 0x7f352c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.360+0000 7f354d90b700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f352c0682f0 0x7f352c06a7a0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f35480bab00 tx=0x7f3544008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:17:35.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.364+0000 7f353a7fc700 1 -- 172.21.15.7:0/159145204 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f353c052490 con 0x7f35481dad70 2023-03-03T20:17:35.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.601+0000 7f354fb6f700 1 -- 172.21.15.7:0/159145204 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35480a9fb0 con 0x7f35481dad70 2023-03-03T20:17:35.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.602+0000 7f353a7fc700 1 -- 172.21.15.7:0/159145204 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f353c055ab0 con 0x7f35481dad70 2023-03-03T20:17:35.603 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:17:35.603 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:17:35.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 -- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f352c0682f0 msgr2=0x7f352c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:35.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f352c0682f0 0x7f352c06a7a0 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f35480bab00 tx=0x7f3544008040 comp rx=0 tx=0).stop 2023-03-03T20:17:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 -- 172.21.15.7:0/159145204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35481dad70 msgr2=0x7f35481e6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35481dad70 0x7f35481e6300 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 crypto rx=0x7f353c00f820 tx=0x7f353c00fbe0 comp rx=0 tx=0).stop 2023-03-03T20:17:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 -- 172.21.15.7:0/159145204 shutdown_connections 2023-03-03T20:17:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f35481e92e0 0x7f35481e6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:35.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f35481da9a0 0x7f3548147bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f352c0682f0 0x7f352c06a7a0 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 --2- 172.21.15.7:0/159145204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35481dad70 0x7f35481e6300 unknown :-1 s=CLOSED pgs=3749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 -- 172.21.15.7:0/159145204 >> 172.21.15.7:0/159145204 conn(0x7f35480a94d0 msgr2=0x7f3548151720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:17:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 -- 172.21.15.7:0/159145204 shutdown_connections 2023-03-03T20:17:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:35.604+0000 7f354fb6f700 1 -- 172.21.15.7:0/159145204 wait complete. 2023-03-03T20:17:35.606 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:17:35.616 INFO:tasks.mds_thrash.fs.[a]:waiting for 102 secs before thrashing 2023-03-03T20:17:46.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:17:46.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.670+0000 7fd8aad07700 1 Processor -- start 2023-03-03T20:17:46.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.670+0000 7fd8aad07700 1 -- start start 2023-03-03T20:17:46.673 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.671+0000 7fd8aad07700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd8a40aad20 0x7fd8a4157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:46.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.671+0000 7fd8aad07700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a41588f0 con 0x7fd8a40aa030 2023-03-03T20:17:46.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.671+0000 7fd8aad07700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a40728b0 con 0x7fd8a414b020 2023-03-03T20:17:46.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.671+0000 7fd8aad07700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a4155070 con 0x7fd8a40aad20 2023-03-03T20:17:46.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.671+0000 7fd8a92a4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd8a40aa030 0x7fd8a41581e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39214/0 (socket says 172.21.15.7:39214) 2023-03-03T20:17:46.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.671+0000 7fd8a3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd8a40aad20 0x7fd8a4157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:46.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.671+0000 7fd8a92a4700 1 -- 172.21.15.7:0/3416409880 learned_addr learned my addr 172.21.15.7:0/3416409880 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:17:46.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.672+0000 7fd8a37fe700 1 -- 172.21.15.7:0/3416409880 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463971606 0 0) 0x7fd8a41588f0 con 0x7fd8a40aa030 2023-03-03T20:17:46.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.672+0000 7fd8a37fe700 1 -- 172.21.15.7:0/3416409880 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8940035b0 con 0x7fd8a40aa030 2023-03-03T20:17:46.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.672+0000 7fd8a3fff700 1 -- 172.21.15.7:0/3416409880 >> v1:172.21.15.143:6789/0 conn(0x7fd8a414b020 legacy=0x7fd8a40717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:46.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.672+0000 7fd8a3fff700 1 -- 172.21.15.7:0/3416409880 >> v1:172.21.15.7:6789/0 conn(0x7fd8a40aa030 legacy=0x7fd8a41581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:46.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.672+0000 7fd8a3fff700 1 -- 172.21.15.7:0/3416409880 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a406f5b0 con 0x7fd8a40aad20 2023-03-03T20:17:46.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.672+0000 7fd8a3fff700 1 --2- 172.21.15.7:0/3416409880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aad20 0x7fd8a4157ca0 secure :-1 s=READY pgs=3751 cs=0 l=1 rev1=1 crypto rx=0x7fd890009840 tx=0x7fd890009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c50b0c959f5dba server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:17:46.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.672+0000 7fd8a37fe700 1 -- 172.21.15.7:0/3416409880 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd890011830 con 0x7fd8a40aad20 2023-03-03T20:17:46.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.672+0000 7fd8a37fe700 1 -- 172.21.15.7:0/3416409880 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd890011990 con 0x7fd8a40aad20 2023-03-03T20:17:46.676 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.672+0000 7fd8a37fe700 1 -- 172.21.15.7:0/3416409880 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd890011d20 con 0x7fd8a40aad20 2023-03-03T20:17:46.678 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.673+0000 7fd8aad07700 1 -- 172.21.15.7:0/3416409880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aad20 msgr2=0x7fd8a4157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.673+0000 7fd8aad07700 1 --2- 172.21.15.7:0/3416409880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aad20 0x7fd8a4157ca0 secure :-1 s=READY pgs=3751 cs=0 l=1 rev1=1 crypto rx=0x7fd890009840 tx=0x7fd890009b50 comp rx=0 tx=0).stop 2023-03-03T20:17:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.673+0000 7fd8aad07700 1 -- 172.21.15.7:0/3416409880 shutdown_connections 2023-03-03T20:17:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.673+0000 7fd8aad07700 1 --2- 172.21.15.7:0/3416409880 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aad20 0x7fd8a4157ca0 unknown :-1 s=CLOSED pgs=3751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.673+0000 7fd8aad07700 1 -- 172.21.15.7:0/3416409880 >> 172.21.15.7:0/3416409880 conn(0x7fd8a40c8900 msgr2=0x7fd8a40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:17:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.673+0000 7fd8aad07700 1 -- 172.21.15.7:0/3416409880 shutdown_connections 2023-03-03T20:17:46.679 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.673+0000 7fd8aad07700 1 -- 172.21.15.7:0/3416409880 wait complete. 2023-03-03T20:17:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.674+0000 7fd8aad07700 1 Processor -- start 2023-03-03T20:17:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.674+0000 7fd8aad07700 1 -- start start 2023-03-03T20:17:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8aad07700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aa030 0x7fd8a40c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8aad07700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8a40aad20 0x7fd8a41ea0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8aad07700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8a414b020 0x7fd8a41ea620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8aad07700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a414eaa0 con 0x7fd8a40aa030 2023-03-03T20:17:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8aad07700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a4154180 con 0x7fd8a414b020 2023-03-03T20:17:46.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8aad07700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a4151200 con 0x7fd8a40aad20 2023-03-03T20:17:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a8aa3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aa030 0x7fd8a40c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a8aa3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aa030 0x7fd8a40c75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45236/0 (socket says 172.21.15.7:45236) 2023-03-03T20:17:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a8aa3700 1 -- 172.21.15.7:0/3899908175 learned_addr learned my addr 172.21.15.7:0/3899908175 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:17:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a8aa3700 1 -- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8a40aad20 msgr2=0x7fd8a41ea0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a8aa3700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8a40aad20 0x7fd8a41ea0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a8aa3700 1 -- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8a414b020 msgr2=0x7fd8a41ea620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:46.681 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a92a4700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8a414b020 0x7fd8a41ea620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a8aa3700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8a414b020 0x7fd8a41ea620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a8aa3700 1 -- 172.21.15.7:0/3899908175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8900094a0 con 0x7fd8a40aa030 2023-03-03T20:17:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a8aa3700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aa030 0x7fd8a40c75d0 secure :-1 s=READY pgs=3752 cs=0 l=1 rev1=1 crypto rx=0x7fd89800c450 tx=0x7fd89800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:17:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a17fa700 1 -- 172.21.15.7:0/3899908175 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8980052d0 con 0x7fd8a40aa030 2023-03-03T20:17:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.675+0000 7fd8a17fa700 1 -- 172.21.15.7:0/3899908175 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd898005430 con 0x7fd8a40aa030 2023-03-03T20:17:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.676+0000 7fd8a17fa700 1 -- 172.21.15.7:0/3899908175 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8980038c0 con 0x7fd8a40aa030 2023-03-03T20:17:46.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.676+0000 7fd8aad07700 1 -- 172.21.15.7:0/3899908175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a406fb70 con 0x7fd8a40aa030 2023-03-03T20:17:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.676+0000 7fd8aad07700 1 -- 172.21.15.7:0/3899908175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a406f890 con 0x7fd8a40aa030 2023-03-03T20:17:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.676+0000 7fd8aad07700 1 -- 172.21.15.7:0/3899908175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a4155350 con 0x7fd8a40aa030 2023-03-03T20:17:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.677+0000 7fd8a17fa700 1 -- 172.21.15.7:0/3899908175 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd898019030 con 0x7fd8a40aa030 2023-03-03T20:17:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.677+0000 7fd8a17fa700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8880682f0 0x7fd88806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:46.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.677+0000 7fd8a17fa700 1 -- 172.21.15.7:0/3899908175 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd898085440 con 0x7fd8a40aa030 2023-03-03T20:17:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.677+0000 7fd8a3fff700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8880682f0 0x7fd88806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.678+0000 7fd8a3fff700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8880682f0 0x7fd88806a7a0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7fd890009840 tx=0x7fd890010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:17:46.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.682+0000 7fd8a17fa700 1 -- 172.21.15.7:0/3899908175 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd8980517e0 con 0x7fd8a40aa030 2023-03-03T20:17:46.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.920+0000 7fd8aad07700 1 -- 172.21.15.7:0/3899908175 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8a4074bb0 con 0x7fd8a40aa030 2023-03-03T20:17:46.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.920+0000 7fd8a17fa700 1 -- 172.21.15.7:0/3899908175 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd898054e00 con 0x7fd8a40aa030 2023-03-03T20:17:46.924 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:17:46.924 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:17:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 -- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8880682f0 msgr2=0x7fd88806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:46.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8880682f0 0x7fd88806a7a0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7fd890009840 tx=0x7fd890010000 comp rx=0 tx=0).stop 2023-03-03T20:17:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 -- 172.21.15.7:0/3899908175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aa030 msgr2=0x7fd8a40c75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aa030 0x7fd8a40c75d0 secure :-1 s=READY pgs=3752 cs=0 l=1 rev1=1 crypto rx=0x7fd89800c450 tx=0x7fd89800c810 comp rx=0 tx=0).stop 2023-03-03T20:17:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 -- 172.21.15.7:0/3899908175 shutdown_connections 2023-03-03T20:17:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8a414b020 0x7fd8a41ea620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd8880682f0 0x7fd88806a7a0 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8a40aa030 0x7fd8a40c75d0 unknown :-1 s=CLOSED pgs=3752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 --2- 172.21.15.7:0/3899908175 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd8a40aad20 0x7fd8a41ea0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:46.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 -- 172.21.15.7:0/3899908175 >> 172.21.15.7:0/3899908175 conn(0x7fd8a40c8900 msgr2=0x7fd8a40c17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:17:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 -- 172.21.15.7:0/3899908175 shutdown_connections 2023-03-03T20:17:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:46.922+0000 7fd8aad07700 1 -- 172.21.15.7:0/3899908175 wait complete. 2023-03-03T20:17:46.927 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:17:46.936 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before thrashing 2023-03-03T20:17:47.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:17:48.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.077+0000 7f73726e4700 1 Processor -- start 2023-03-03T20:17:48.079 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.077+0000 7f73726e4700 1 -- start start 2023-03-03T20:17:48.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f73726e4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f736c0c4780 0x7f736c154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:48.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f73726e4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f736c0c4b50 con 0x7f736c158130 2023-03-03T20:17:48.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f73726e4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f736c0a9d40 con 0x7f736c071290 2023-03-03T20:17:48.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f73726e4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f736c0a94c0 con 0x7f736c0c4780 2023-03-03T20:17:48.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f736bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f736c0c4780 0x7f736c154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:48.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f7370c81700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f736c158130 0x7f736c0c4f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39228/0 (socket says 172.21.15.7:39228) 2023-03-03T20:17:48.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f7370c81700 1 -- 172.21.15.7:0/3123538938 learned_addr learned my addr 172.21.15.7:0/3123538938 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:17:48.081 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f736bfff700 1 -- 172.21.15.7:0/3123538938 >> v1:172.21.15.143:6789/0 conn(0x7f736c071290 legacy=0x7f736c0c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:17:48.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f736bfff700 1 -- 172.21.15.7:0/3123538938 >> v1:172.21.15.7:6789/0 conn(0x7f736c158130 legacy=0x7f736c0c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:48.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f736bfff700 1 -- 172.21.15.7:0/3123538938 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c0a9780 con 0x7f736c0c4780 2023-03-03T20:17:48.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.078+0000 7f736bfff700 1 --2- 172.21.15.7:0/3123538938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 0x7f736c154fb0 secure :-1 s=READY pgs=3754 cs=0 l=1 rev1=1 crypto rx=0x7f735c009840 tx=0x7f735c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4662681ab45bcd server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:17:48.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.079+0000 7f736affd700 1 -- 172.21.15.7:0/3123538938 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c004230 con 0x7f736c0c4780 2023-03-03T20:17:48.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.079+0000 7f736affd700 1 -- 172.21.15.7:0/3123538938 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735c004390 con 0x7f736c0c4780 2023-03-03T20:17:48.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.079+0000 7f736affd700 1 -- 172.21.15.7:0/3123538938 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735c0036a0 con 0x7f736c0c4780 2023-03-03T20:17:48.084 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.079+0000 7f73726e4700 1 -- 172.21.15.7:0/3123538938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 msgr2=0x7f736c154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:48.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.079+0000 7f73726e4700 1 --2- 172.21.15.7:0/3123538938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 0x7f736c154fb0 secure :-1 s=READY pgs=3754 cs=0 l=1 rev1=1 crypto rx=0x7f735c009840 tx=0x7f735c009b50 comp rx=0 tx=0).stop 2023-03-03T20:17:48.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.079+0000 7f73726e4700 1 -- 172.21.15.7:0/3123538938 shutdown_connections 2023-03-03T20:17:48.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.079+0000 7f73726e4700 1 --2- 172.21.15.7:0/3123538938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 0x7f736c154fb0 unknown :-1 s=CLOSED pgs=3754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:48.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.079+0000 7f73726e4700 1 -- 172.21.15.7:0/3123538938 >> 172.21.15.7:0/3123538938 conn(0x7f736c0736f0 msgr2=0x7f736c073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:17:48.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.079+0000 7f73726e4700 1 -- 172.21.15.7:0/3123538938 shutdown_connections 2023-03-03T20:17:48.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.079+0000 7f73726e4700 1 -- 172.21.15.7:0/3123538938 wait complete. 2023-03-03T20:17:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.080+0000 7f73726e4700 1 Processor -- start 2023-03-03T20:17:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f73726e4700 1 -- start start 2023-03-03T20:17:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f73726e4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f736c071290 0x7f736c1eae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f73726e4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 0x7f736c14ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f73726e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f736c158130 0x7f736c14e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f73726e4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c1eb360 con 0x7f736c0c4780 2023-03-03T20:17:48.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f73726e4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c0c8be0 con 0x7f736c158130 2023-03-03T20:17:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f73726e4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736c14d700 con 0x7f736c071290 2023-03-03T20:17:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f736b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 0x7f736c14ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f736b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 0x7f736c14ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45252/0 (socket says 172.21.15.7:45252) 2023-03-03T20:17:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f736b7fe700 1 -- 172.21.15.7:0/793293244 learned_addr learned my addr 172.21.15.7:0/793293244 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:17:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.081+0000 7f736bfff700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f736c071290 0x7f736c1eae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f7370c81700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f736c158130 0x7f736c14e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f736b7fe700 1 -- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f736c071290 msgr2=0x7f736c1eae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:48.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f736b7fe700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f736c071290 0x7f736c1eae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f736b7fe700 1 -- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f736c158130 msgr2=0x7f736c14e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f736b7fe700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f736c158130 0x7f736c14e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f736b7fe700 1 -- 172.21.15.7:0/793293244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0094a0 con 0x7f736c0c4780 2023-03-03T20:17:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f736b7fe700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 0x7f736c14ba40 secure :-1 s=READY pgs=3755 cs=0 l=1 rev1=1 crypto rx=0x7f735400c4a0 tx=0x7f735400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:17:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f7368ff9700 1 -- 172.21.15.7:0/793293244 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73540054e0 con 0x7f736c0c4780 2023-03-03T20:17:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f7368ff9700 1 -- 172.21.15.7:0/793293244 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7354003680 con 0x7f736c0c4780 2023-03-03T20:17:48.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f73726e4700 1 -- 172.21.15.7:0/793293244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736c157c10 con 0x7f736c0c4780 2023-03-03T20:17:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f7368ff9700 1 -- 172.21.15.7:0/793293244 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354003a30 con 0x7f736c0c4780 2023-03-03T20:17:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.082+0000 7f73726e4700 1 -- 172.21.15.7:0/793293244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736c0a9a60 con 0x7f736c0c4780 2023-03-03T20:17:48.089 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.083+0000 7f73726e4700 1 -- 172.21.15.7:0/793293244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736c0c5620 con 0x7f736c0c4780 2023-03-03T20:17:48.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.084+0000 7f7368ff9700 1 -- 172.21.15.7:0/793293244 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7354019030 con 0x7f736c0c4780 2023-03-03T20:17:48.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.084+0000 7f7368ff9700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f734c0682f0 0x7f734c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:17:48.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.084+0000 7f7368ff9700 1 -- 172.21.15.7:0/793293244 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7354085370 con 0x7f736c0c4780 2023-03-03T20:17:48.090 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.084+0000 7f736bfff700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f734c0682f0 0x7f734c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:17:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.085+0000 7f736bfff700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f734c0682f0 0x7f734c06a7a0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f735c0090b0 tx=0x7f735c01c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:17:48.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.089+0000 7f7368ff9700 1 -- 172.21.15.7:0/793293244 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7354051710 con 0x7f736c0c4780 2023-03-03T20:17:48.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.324+0000 7f73726e4700 1 -- 172.21.15.7:0/793293244 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f736c074bb0 con 0x7f736c0c4780 2023-03-03T20:17:48.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.325+0000 7f7368ff9700 1 -- 172.21.15.7:0/793293244 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7354017090 con 0x7f736c0c4780 2023-03-03T20:17:48.327 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:17:48.327 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:17:48.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 -- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f734c0682f0 msgr2=0x7f734c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f734c0682f0 0x7f734c06a7a0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f735c0090b0 tx=0x7f735c01c040 comp rx=0 tx=0).stop 2023-03-03T20:17:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 -- 172.21.15.7:0/793293244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 msgr2=0x7f736c14ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:17:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 0x7f736c14ba40 secure :-1 s=READY pgs=3755 cs=0 l=1 rev1=1 crypto rx=0x7f735400c4a0 tx=0x7f735400c7b0 comp rx=0 tx=0).stop 2023-03-03T20:17:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 -- 172.21.15.7:0/793293244 shutdown_connections 2023-03-03T20:17:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f736c158130 0x7f736c14e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f736c071290 0x7f736c1eae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f734c0682f0 0x7f734c06a7a0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 --2- 172.21.15.7:0/793293244 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f736c0c4780 0x7f736c14ba40 unknown :-1 s=CLOSED pgs=3755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:17:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 -- 172.21.15.7:0/793293244 >> 172.21.15.7:0/793293244 conn(0x7f736c0736f0 msgr2=0x7f736c155d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:17:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 -- 172.21.15.7:0/793293244 shutdown_connections 2023-03-03T20:17:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:17:48.327+0000 7f73726e4700 1 -- 172.21.15.7:0/793293244 wait complete. 2023-03-03T20:17:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:17:48.340 INFO:tasks.mds_thrash.fs.[b]:waiting for 54 secs before thrashing 2023-03-03T20:17:58.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:17:58.633 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:18:28.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:18:28.717 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:18:42.342 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:18:42.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.481+0000 7efc24239700 1 Processor -- start 2023-03-03T20:18:42.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.481+0000 7efc24239700 1 -- start start 2023-03-03T20:18:42.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.482+0000 7efc24239700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efc1c14f9d0 0x7efc1c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:18:42.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.482+0000 7efc24239700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 0x7efc1c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:18:42.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.482+0000 7efc24239700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c0c2f30 con 0x7efc1c0abe20 2023-03-03T20:18:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.482+0000 7efc24239700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1c0c31d0 con 0x7efc1c14f9d0 2023-03-03T20:18:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.482+0000 7efc24239700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c1524a0 con 0x7efc1c14fda0 2023-03-03T20:18:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.482+0000 7efc217d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 0x7efc1c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:18:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.482+0000 7efc21fd5700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7efc1c14f9d0 0x7efc1c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:18:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.482+0000 7efc217d4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 0x7efc1c071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:53380/0 (socket says 172.21.15.7:53380) 2023-03-03T20:18:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.482+0000 7efc217d4700 1 -- 172.21.15.7:0/4137970697 learned_addr learned my addr 172.21.15.7:0/4137970697 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:18:42.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.483+0000 7efc217d4700 1 -- 172.21.15.7:0/4137970697 >> v2:172.21.15.143:3300/0 conn(0x7efc1c14f9d0 msgr2=0x7efc1c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:18:42.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.483+0000 7efc217d4700 1 --2- 172.21.15.7:0/4137970697 >> v2:172.21.15.143:3300/0 conn(0x7efc1c14f9d0 0x7efc1c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:18:42.491 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.483+0000 7efc217d4700 1 -- 172.21.15.7:0/4137970697 >> v1:172.21.15.143:6789/0 conn(0x7efc1c0abe20 legacy=0x7efc1c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:18:42.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.483+0000 7efc217d4700 1 -- 172.21.15.7:0/4137970697 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c0c34b0 con 0x7efc1c14fda0 2023-03-03T20:18:42.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.483+0000 7efc217d4700 1 --2- 172.21.15.7:0/4137970697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 0x7efc1c071890 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efc180098f0 tx=0x7efc18009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ce54f4791dbd6a4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:18:42.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.483+0000 7efc20fd3700 1 -- 172.21.15.7:0/4137970697 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc18011830 con 0x7efc1c14fda0 2023-03-03T20:18:42.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.483+0000 7efc20fd3700 1 -- 172.21.15.7:0/4137970697 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc18011990 con 0x7efc1c14fda0 2023-03-03T20:18:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.483+0000 7efc20fd3700 1 -- 172.21.15.7:0/4137970697 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc18011d20 con 0x7efc1c14fda0 2023-03-03T20:18:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.484+0000 7efc24239700 1 -- 172.21.15.7:0/4137970697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 msgr2=0x7efc1c071890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:18:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.484+0000 7efc24239700 1 --2- 172.21.15.7:0/4137970697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 0x7efc1c071890 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efc180098f0 tx=0x7efc18009c00 comp rx=0 tx=0).stop 2023-03-03T20:18:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.484+0000 7efc24239700 1 -- 172.21.15.7:0/4137970697 shutdown_connections 2023-03-03T20:18:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.484+0000 7efc24239700 1 --2- 172.21.15.7:0/4137970697 >> v2:172.21.15.143:3300/0 conn(0x7efc1c14f9d0 0x7efc1c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:18:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.484+0000 7efc24239700 1 --2- 172.21.15.7:0/4137970697 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 0x7efc1c071890 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:18:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.484+0000 7efc24239700 1 -- 172.21.15.7:0/4137970697 >> 172.21.15.7:0/4137970697 conn(0x7efc1c0c8900 msgr2=0x7efc1c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:18:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.484+0000 7efc24239700 1 -- 172.21.15.7:0/4137970697 shutdown_connections 2023-03-03T20:18:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.484+0000 7efc24239700 1 -- 172.21.15.7:0/4137970697 wait complete. 2023-03-03T20:18:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.485+0000 7efc24239700 1 Processor -- start 2023-03-03T20:18:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.485+0000 7efc24239700 1 -- start start 2023-03-03T20:18:42.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc24239700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc1c0abe20 0x7efc1c0c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:18:42.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc24239700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c14f9d0 0x7efc1c1dd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:18:42.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc24239700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 0x7efc1c1dda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:18:42.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc24239700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c06a980 con 0x7efc1c0abe20 2023-03-03T20:18:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc24239700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c1e9690 con 0x7efc1c14f9d0 2023-03-03T20:18:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc24239700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c1db7a0 con 0x7efc1c14fda0 2023-03-03T20:18:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc21fd5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc1c0abe20 0x7efc1c0c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:18:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc21fd5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc1c0abe20 0x7efc1c0c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45314/0 (socket says 172.21.15.7:45314) 2023-03-03T20:18:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc21fd5700 1 -- 172.21.15.7:0/2639430428 learned_addr learned my addr 172.21.15.7:0/2639430428 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:18:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc217d4700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c14f9d0 0x7efc1c1dd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:18:42.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc227d6700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 0x7efc1c1dda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:18:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc21fd5700 1 -- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 msgr2=0x7efc1c1dda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:18:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc21fd5700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 0x7efc1c1dda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:18:42.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc21fd5700 1 -- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c14f9d0 msgr2=0x7efc1c1dd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:18:42.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc21fd5700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c14f9d0 0x7efc1c1dd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:18:42.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc21fd5700 1 -- 172.21.15.7:0/2639430428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc18009550 con 0x7efc1c0abe20 2023-03-03T20:18:42.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc21fd5700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc1c0abe20 0x7efc1c0c1a50 secure :-1 s=READY pgs=3756 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00e9b0 tx=0x7efc0c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:18:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc127fc700 1 -- 172.21.15.7:0/2639430428 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c009710 con 0x7efc1c0abe20 2023-03-03T20:18:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.486+0000 7efc127fc700 1 -- 172.21.15.7:0/2639430428 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c00c850 con 0x7efc1c0abe20 2023-03-03T20:18:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.487+0000 7efc24239700 1 -- 172.21.15.7:0/2639430428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c0ac350 con 0x7efc1c0abe20 2023-03-03T20:18:42.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.487+0000 7efc127fc700 1 -- 172.21.15.7:0/2639430428 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c00cbc0 con 0x7efc1c0abe20 2023-03-03T20:18:42.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.487+0000 7efc24239700 1 -- 172.21.15.7:0/2639430428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c0ab630 con 0x7efc1c0abe20 2023-03-03T20:18:42.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.488+0000 7efc24239700 1 -- 172.21.15.7:0/2639430428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c0c3e10 con 0x7efc1c0abe20 2023-03-03T20:18:42.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.488+0000 7efc127fc700 1 -- 172.21.15.7:0/2639430428 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efc0c019030 con 0x7efc1c0abe20 2023-03-03T20:18:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.489+0000 7efc127fc700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc000682f0 0x7efc0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:18:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.489+0000 7efc127fc700 1 -- 172.21.15.7:0/2639430428 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7efc0c086160 con 0x7efc1c0abe20 2023-03-03T20:18:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.489+0000 7efc217d4700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc000682f0 0x7efc0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:18:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.489+0000 7efc217d4700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc000682f0 0x7efc0006a7a0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7efc1c1501d0 tx=0x7efc18008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:18:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.493+0000 7efc127fc700 1 -- 172.21.15.7:0/2639430428 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7efc0c052500 con 0x7efc1c0abe20 2023-03-03T20:18:42.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.729+0000 7efc24239700 1 -- 172.21.15.7:0/2639430428 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc1c0c2670 con 0x7efc1c0abe20 2023-03-03T20:18:42.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.730+0000 7efc127fc700 1 -- 172.21.15.7:0/2639430428 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7efc0c017090 con 0x7efc1c0abe20 2023-03-03T20:18:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:18:42.731 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:18:42.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 -- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc000682f0 msgr2=0x7efc0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:18:42.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc000682f0 0x7efc0006a7a0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7efc1c1501d0 tx=0x7efc18008030 comp rx=0 tx=0).stop 2023-03-03T20:18:42.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 -- 172.21.15.7:0/2639430428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc1c0abe20 msgr2=0x7efc1c0c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:18:42.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc1c0abe20 0x7efc1c0c1a50 secure :-1 s=READY pgs=3756 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00e9b0 tx=0x7efc0c00ed70 comp rx=0 tx=0).stop 2023-03-03T20:18:42.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 -- 172.21.15.7:0/2639430428 shutdown_connections 2023-03-03T20:18:42.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7efc1c14fda0 0x7efc1c1dda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:18:42.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7efc000682f0 0x7efc0006a7a0 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:18:42.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efc1c0abe20 0x7efc1c0c1a50 unknown :-1 s=CLOSED pgs=3756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:18:42.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 --2- 172.21.15.7:0/2639430428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c14f9d0 0x7efc1c1dd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:18:42.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 -- 172.21.15.7:0/2639430428 >> 172.21.15.7:0/2639430428 conn(0x7efc1c0c8900 msgr2=0x7efc1c0c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:18:42.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 -- 172.21.15.7:0/2639430428 shutdown_connections 2023-03-03T20:18:42.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:18:42.732+0000 7efc24239700 1 -- 172.21.15.7:0/2639430428 wait complete. 2023-03-03T20:18:42.735 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:18:42.745 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-03-03T20:18:58.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:18:58.784 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:19:17.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:19:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.755+0000 7f0f0ae12700 1 Processor -- start 2023-03-03T20:19:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.755+0000 7f0f0ae12700 1 -- start start 2023-03-03T20:19:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.756+0000 7f0f0ae12700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f0f04151b20 0x7f0f04071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:19:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.756+0000 7f0f0ae12700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f04154f10 0x7f0f04158000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:19:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.756+0000 7f0f0ae12700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f04151ef0 con 0x7f0f04071d60 2023-03-03T20:19:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.756+0000 7f0f0ae12700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f04157b20 con 0x7f0f04154f10 2023-03-03T20:19:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.756+0000 7f0f0ae12700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f04155410 con 0x7f0f04151b20 2023-03-03T20:19:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.756+0000 7f0f03fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f04154f10 0x7f0f04158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:19:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.756+0000 7f0f03fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0f04154f10 0x7f0f04158000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45350/0 (socket says 172.21.15.7:45350) 2023-03-03T20:19:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.756+0000 7f0f03fff700 1 -- 172.21.15.7:0/2114481921 learned_addr learned my addr 172.21.15.7:0/2114481921 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:19:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.757+0000 7f0f08bae700 1 --2- 172.21.15.7:0/2114481921 >> v2:172.21.15.143:3300/0 conn(0x7f0f04151b20 0x7f0f04071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:19:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.757+0000 7f0f03fff700 1 -- 172.21.15.7:0/2114481921 >> v2:172.21.15.143:3300/0 conn(0x7f0f04151b20 msgr2=0x7f0f04071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:19:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.757+0000 7f0f03fff700 1 --2- 172.21.15.7:0/2114481921 >> v2:172.21.15.143:3300/0 conn(0x7f0f04151b20 0x7f0f04071820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:19:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.757+0000 7f0f03fff700 1 -- 172.21.15.7:0/2114481921 >> v1:172.21.15.143:6789/0 conn(0x7f0f04071d60 legacy=0x7f0f040c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:19:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.757+0000 7f0f03fff700 1 -- 172.21.15.7:0/2114481921 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0406f640 con 0x7f0f04154f10 2023-03-03T20:19:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.757+0000 7f0f03fff700 1 --2- 172.21.15.7:0/2114481921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04154f10 0x7f0f04158000 secure :-1 s=READY pgs=3757 cs=0 l=1 rev1=1 crypto rx=0x7f0f040708e0 tx=0x7f0ef0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a5f7dabbd1ad1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:19:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.757+0000 7f0f037fe700 1 -- 172.21.15.7:0/2114481921 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0004230 con 0x7f0f04154f10 2023-03-03T20:19:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.757+0000 7f0f037fe700 1 -- 172.21.15.7:0/2114481921 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef0004390 con 0x7f0f04154f10 2023-03-03T20:19:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.757+0000 7f0f037fe700 1 -- 172.21.15.7:0/2114481921 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef00036a0 con 0x7f0f04154f10 2023-03-03T20:19:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.758+0000 7f0f0ae12700 1 -- 172.21.15.7:0/2114481921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04154f10 msgr2=0x7f0f04158000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:19:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.758+0000 7f0f0ae12700 1 --2- 172.21.15.7:0/2114481921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04154f10 0x7f0f04158000 secure :-1 s=READY pgs=3757 cs=0 l=1 rev1=1 crypto rx=0x7f0f040708e0 tx=0x7f0ef0009bd0 comp rx=0 tx=0).stop 2023-03-03T20:19:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.758+0000 7f0f0ae12700 1 -- 172.21.15.7:0/2114481921 shutdown_connections 2023-03-03T20:19:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.758+0000 7f0f0ae12700 1 --2- 172.21.15.7:0/2114481921 >> v2:172.21.15.143:3300/0 conn(0x7f0f04151b20 0x7f0f04071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:19:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.758+0000 7f0f0ae12700 1 --2- 172.21.15.7:0/2114481921 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04154f10 0x7f0f04158000 unknown :-1 s=CLOSED pgs=3757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:19:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.758+0000 7f0f0ae12700 1 -- 172.21.15.7:0/2114481921 >> 172.21.15.7:0/2114481921 conn(0x7f0f040c8900 msgr2=0x7f0f040c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:19:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.758+0000 7f0f0ae12700 1 -- 172.21.15.7:0/2114481921 shutdown_connections 2023-03-03T20:19:17.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.758+0000 7f0f0ae12700 1 -- 172.21.15.7:0/2114481921 wait complete. 2023-03-03T20:19:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.759+0000 7f0f0ae12700 1 Processor -- start 2023-03-03T20:19:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.759+0000 7f0f0ae12700 1 -- start start 2023-03-03T20:19:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f0ae12700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04071d60 0x7f0f04149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:19:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f0ae12700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f04151b20 0x7f0f041ed3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:19:17.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f0ae12700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f04154f10 0x7f0f041ed8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:19:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f0ae12700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f040c6b50 con 0x7f0f04071d60 2023-03-03T20:19:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f0ae12700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0414fbe0 con 0x7f0f04151b20 2023-03-03T20:19:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f0ae12700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f040c63b0 con 0x7f0f04154f10 2023-03-03T20:19:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f08bae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04071d60 0x7f0f04149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:19:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f08bae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04071d60 0x7f0f04149570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45354/0 (socket says 172.21.15.7:45354) 2023-03-03T20:19:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f08bae700 1 -- 172.21.15.7:0/4039672835 learned_addr learned my addr 172.21.15.7:0/4039672835 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:19:17.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f03fff700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f04151b20 0x7f0f041ed3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:19:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f093af700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f04154f10 0x7f0f041ed8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:19:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f08bae700 1 -- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f04154f10 msgr2=0x7f0f041ed8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:19:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f08bae700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f04154f10 0x7f0f041ed8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:19:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f08bae700 1 -- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f04151b20 msgr2=0x7f0f041ed3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:19:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f08bae700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f04151b20 0x7f0f041ed3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:19:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f08bae700 1 -- 172.21.15.7:0/4039672835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef0009550 con 0x7f0f04071d60 2023-03-03T20:19:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f093af700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f04154f10 0x7f0f041ed8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:19:17.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.760+0000 7f0f08bae700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04071d60 0x7f0f04149570 secure :-1 s=READY pgs=3758 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800d830 tx=0x7f0ef800dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:19:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.761+0000 7f0f017fa700 1 -- 172.21.15.7:0/4039672835 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80125f0 con 0x7f0f04071d60 2023-03-03T20:19:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.761+0000 7f0f017fa700 1 -- 172.21.15.7:0/4039672835 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef8012750 con 0x7f0f04071d60 2023-03-03T20:19:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.761+0000 7f0f0ae12700 1 -- 172.21.15.7:0/4039672835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0406fc00 con 0x7f0f04071d60 2023-03-03T20:19:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.761+0000 7f0f017fa700 1 -- 172.21.15.7:0/4039672835 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8012b00 con 0x7f0f04071d60 2023-03-03T20:19:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.761+0000 7f0f0ae12700 1 -- 172.21.15.7:0/4039672835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0406f920 con 0x7f0f04071d60 2023-03-03T20:19:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.762+0000 7f0f0ae12700 1 -- 172.21.15.7:0/4039672835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f040aae90 con 0x7f0f04071d60 2023-03-03T20:19:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.762+0000 7f0f017fa700 1 -- 172.21.15.7:0/4039672835 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0ef8012c60 con 0x7f0f04071d60 2023-03-03T20:19:17.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.762+0000 7f0f017fa700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 0x7f0ee806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:19:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.763+0000 7f0f017fa700 1 -- 172.21.15.7:0/4039672835 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0ef8086660 con 0x7f0f04071d60 2023-03-03T20:19:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.763+0000 7f0f03fff700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 0x7f0ee806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:19:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.763+0000 7f0f03fff700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 0x7f0ee806a7a0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f0f04155340 tx=0x7f0ef0008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:19:17.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:17.767+0000 7f0f017fa700 1 -- 172.21.15.7:0/4039672835 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0ef8052a00 con 0x7f0f04071d60 2023-03-03T20:19:18.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.004+0000 7f0f0ae12700 1 -- 172.21.15.7:0/4039672835 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f040c69c0 con 0x7f0f04071d60 2023-03-03T20:19:18.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.004+0000 7f0f017fa700 1 -- 172.21.15.7:0/4039672835 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f0ef8056020 con 0x7f0f04071d60 2023-03-03T20:19:18.006 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:19:18.006 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:19:18.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.006+0000 7f0f0ae12700 1 -- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 msgr2=0x7f0ee806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:19:18.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.006+0000 7f0f0ae12700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 0x7f0ee806a7a0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f0f04155340 tx=0x7f0ef0008030 comp rx=0 tx=0).stop 2023-03-03T20:19:18.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.007+0000 7f0f0ae12700 1 -- 172.21.15.7:0/4039672835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04071d60 msgr2=0x7f0f04149570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:19:18.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.007+0000 7f0f0ae12700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04071d60 0x7f0f04149570 secure :-1 s=READY pgs=3758 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800d830 tx=0x7f0ef800dbf0 comp rx=0 tx=0).stop 2023-03-03T20:19:18.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.007+0000 7f0f0ae12700 1 -- 172.21.15.7:0/4039672835 shutdown_connections 2023-03-03T20:19:18.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.007+0000 7f0f0ae12700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f0f04154f10 0x7f0f041ed8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:19:18.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.007+0000 7f0f0ae12700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f0ee80682f0 0x7f0ee806a7a0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:19:18.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.007+0000 7f0f0ae12700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0f04071d60 0x7f0f04149570 unknown :-1 s=CLOSED pgs=3758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:19:18.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.007+0000 7f0f0ae12700 1 --2- 172.21.15.7:0/4039672835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f04151b20 0x7f0f041ed3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:19:18.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.007+0000 7f0f0ae12700 1 -- 172.21.15.7:0/4039672835 >> 172.21.15.7:0/4039672835 conn(0x7f0f040c8900 msgr2=0x7f0f04149e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:19:18.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.007+0000 7f0f0ae12700 1 -- 172.21.15.7:0/4039672835 shutdown_connections 2023-03-03T20:19:18.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:19:18.007+0000 7f0f0ae12700 1 -- 172.21.15.7:0/4039672835 wait complete. 2023-03-03T20:19:18.009 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:19:18.019 INFO:tasks.mds_thrash.fs.[a]:waiting for 71 secs before thrashing 2023-03-03T20:19:28.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:19:28.854 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:19:58.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:19:58.917 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:19:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T20:19:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T20:20:04.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:20:04.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.884+0000 7fb5f12d1700 1 Processor -- start 2023-03-03T20:20:04.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.884+0000 7fb5f12d1700 1 -- start start 2023-03-03T20:20:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.885+0000 7fb5f12d1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec0c2790 0x7fb5ec152d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.885+0000 7fb5f12d1700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5ec1546f0 0x7fb5ec154b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.885+0000 7fb5f12d1700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ec06c920 con 0x7fb5ec14b340 2023-03-03T20:20:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.885+0000 7fb5f12d1700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ec0abd90 con 0x7fb5ec1546f0 2023-03-03T20:20:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.885+0000 7fb5f12d1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec0c1300 con 0x7fb5ec0c2790 2023-03-03T20:20:04.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.885+0000 7fb5eb7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5ec1546f0 0x7fb5ec154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.885+0000 7fb5ea7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec0c2790 0x7fb5ec152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5eb7fe700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb5ec1546f0 0x7fb5ec154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:41170/0 (socket says 172.21.15.7:41170) 2023-03-03T20:20:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5ea7fc700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec0c2790 0x7fb5ec152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:53466/0 (socket says 172.21.15.7:53466) 2023-03-03T20:20:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/2053589432 learned_addr learned my addr 172.21.15.7:0/2053589432 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:20:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5eaffd700 1 --1- 172.21.15.7:0/2053589432 >> v1:172.21.15.143:6789/0 conn(0x7fb5ec14b340 0x7fb5ec152630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:42892/0 (socket says 172.21.15.7:42892) 2023-03-03T20:20:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/2053589432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec0c2790 msgr2=0x7fb5ec152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:20:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5eb7fe700 1 --2- 172.21.15.7:0/2053589432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec0c2790 0x7fb5ec152d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:04.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/2053589432 >> v1:172.21.15.143:6789/0 conn(0x7fb5ec14b340 legacy=0x7fb5ec152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5eb7fe700 1 -- 172.21.15.7:0/2053589432 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec153280 con 0x7fb5ec1546f0 2023-03-03T20:20:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5ea7fc700 1 --2- 172.21.15.7:0/2053589432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec0c2790 0x7fb5ec152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:20:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5eb7fe700 1 --2- 172.21.15.7:0/2053589432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ec1546f0 0x7fb5ec154b40 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009e50 tx=0x7fb5e400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1d473739e814414 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:20:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5e9ffb700 1 -- 172.21.15.7:0/2053589432 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400e070 con 0x7fb5ec1546f0 2023-03-03T20:20:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5e9ffb700 1 -- 172.21.15.7:0/2053589432 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e400b910 con 0x7fb5ec1546f0 2023-03-03T20:20:04.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.886+0000 7fb5e9ffb700 1 -- 172.21.15.7:0/2053589432 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400bcc0 con 0x7fb5ec1546f0 2023-03-03T20:20:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.887+0000 7fb5f12d1700 1 -- 172.21.15.7:0/2053589432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ec1546f0 msgr2=0x7fb5ec154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.887+0000 7fb5f12d1700 1 --2- 172.21.15.7:0/2053589432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ec1546f0 0x7fb5ec154b40 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009e50 tx=0x7fb5e400b330 comp rx=0 tx=0).stop 2023-03-03T20:20:04.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.887+0000 7fb5f12d1700 1 -- 172.21.15.7:0/2053589432 shutdown_connections 2023-03-03T20:20:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.887+0000 7fb5f12d1700 1 --2- 172.21.15.7:0/2053589432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ec1546f0 0x7fb5ec154b40 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.887+0000 7fb5f12d1700 1 --2- 172.21.15.7:0/2053589432 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec0c2790 0x7fb5ec152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.887+0000 7fb5f12d1700 1 -- 172.21.15.7:0/2053589432 >> 172.21.15.7:0/2053589432 conn(0x7fb5ec0c62e0 msgr2=0x7fb5ec0c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:20:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.887+0000 7fb5f12d1700 1 -- 172.21.15.7:0/2053589432 shutdown_connections 2023-03-03T20:20:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.887+0000 7fb5f12d1700 1 -- 172.21.15.7:0/2053589432 wait complete. 2023-03-03T20:20:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.889+0000 7fb5f12d1700 1 Processor -- start 2023-03-03T20:20:04.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.889+0000 7fb5f12d1700 1 -- start start 2023-03-03T20:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.889+0000 7fb5f12d1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5ec0c2790 0x7fb5ec0a9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.889+0000 7fb5f12d1700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec14b340 0x7fb5ec0aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5f12d1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ec1546f0 0x7fb5ec1e8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5f12d1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec0a9730 con 0x7fb5ec0c2790 2023-03-03T20:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5f12d1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec147550 con 0x7fb5ec1546f0 2023-03-03T20:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5f12d1700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec0c56b0 con 0x7fb5ec14b340 2023-03-03T20:20:04.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5ec0c2790 0x7fb5ec0a9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5ec0c2790 0x7fb5ec0a9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45398/0 (socket says 172.21.15.7:45398) 2023-03-03T20:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eaffd700 1 -- 172.21.15.7:0/4008540348 learned_addr learned my addr 172.21.15.7:0/4008540348 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eb7fe700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ec1546f0 0x7fb5ec1e8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5ea7fc700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec14b340 0x7fb5ec0aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eaffd700 1 -- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec14b340 msgr2=0x7fb5ec0aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eaffd700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec14b340 0x7fb5ec0aa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eaffd700 1 -- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ec1546f0 msgr2=0x7fb5ec1e8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:04.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eaffd700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ec1546f0 0x7fb5ec1e8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eaffd700 1 -- 172.21.15.7:0/4008540348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc00b740 con 0x7fb5ec0c2790 2023-03-03T20:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5ea7fc700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec14b340 0x7fb5ec0aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eb7fe700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ec1546f0 0x7fb5ec1e8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5eaffd700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5ec0c2790 0x7fb5ec0a9b50 secure :-1 s=READY pgs=3759 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000c500 tx=0x7fb5e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5d37fe700 1 -- 172.21.15.7:0/4008540348 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e0005cb0 con 0x7fb5ec0c2790 2023-03-03T20:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5f12d1700 1 -- 172.21.15.7:0/4008540348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e4009b30 con 0x7fb5ec0c2790 2023-03-03T20:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5d37fe700 1 -- 172.21.15.7:0/4008540348 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e0005e10 con 0x7fb5ec0c2790 2023-03-03T20:20:04.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5d37fe700 1 -- 172.21.15.7:0/4008540348 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e00072d0 con 0x7fb5ec0c2790 2023-03-03T20:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.890+0000 7fb5f12d1700 1 -- 172.21.15.7:0/4008540348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ec06bff0 con 0x7fb5ec0c2790 2023-03-03T20:20:04.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.891+0000 7fb5f12d1700 1 -- 172.21.15.7:0/4008540348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ec06bd50 con 0x7fb5ec0c2790 2023-03-03T20:20:04.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.892+0000 7fb5d37fe700 1 -- 172.21.15.7:0/4008540348 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb5e0019030 con 0x7fb5ec0c2790 2023-03-03T20:20:04.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.892+0000 7fb5d37fe700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5d80682f0 0x7fb5d806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:04.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.892+0000 7fb5d37fe700 1 -- 172.21.15.7:0/4008540348 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb5e00853a0 con 0x7fb5ec0c2790 2023-03-03T20:20:04.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.893+0000 7fb5ea7fc700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5d80682f0 0x7fb5d806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:04.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.893+0000 7fb5ea7fc700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5d80682f0 0x7fb5d806a7a0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0c4ab0 tx=0x7fb5dc00b420 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:20:04.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:04.897+0000 7fb5d37fe700 1 -- 172.21.15.7:0/4008540348 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb5e004b5e0 con 0x7fb5ec0c2790 2023-03-03T20:20:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.133+0000 7fb5f12d1700 1 -- 172.21.15.7:0/4008540348 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5ec06c2d0 con 0x7fb5ec0c2790 2023-03-03T20:20:05.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.133+0000 7fb5d37fe700 1 -- 172.21.15.7:0/4008540348 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb5e0017090 con 0x7fb5ec0c2790 2023-03-03T20:20:05.135 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:20:05.136 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:20:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 -- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5d80682f0 msgr2=0x7fb5d806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5d80682f0 0x7fb5d806a7a0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0c4ab0 tx=0x7fb5dc00b420 comp rx=0 tx=0).stop 2023-03-03T20:20:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 -- 172.21.15.7:0/4008540348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5ec0c2790 msgr2=0x7fb5ec0a9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5ec0c2790 0x7fb5ec0a9b50 secure :-1 s=READY pgs=3759 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000c500 tx=0x7fb5e000c8c0 comp rx=0 tx=0).stop 2023-03-03T20:20:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 -- 172.21.15.7:0/4008540348 shutdown_connections 2023-03-03T20:20:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5ec1546f0 0x7fb5ec1e8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb5d80682f0 0x7fb5d806a7a0 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb5ec0c2790 0x7fb5ec0a9b50 unknown :-1 s=CLOSED pgs=3759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 --2- 172.21.15.7:0/4008540348 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb5ec14b340 0x7fb5ec0aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 -- 172.21.15.7:0/4008540348 >> 172.21.15.7:0/4008540348 conn(0x7fb5ec0c62e0 msgr2=0x7fb5ec144210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:20:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 -- 172.21.15.7:0/4008540348 shutdown_connections 2023-03-03T20:20:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:05.136+0000 7fb5f12d1700 1 -- 172.21.15.7:0/4008540348 wait complete. 2023-03-03T20:20:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:20:05.149 INFO:tasks.mds_thrash.fs.[b]:waiting for 22 secs before thrashing 2023-03-03T20:20:27.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:20:27.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.289+0000 7fc1b8209700 1 Processor -- start 2023-03-03T20:20:27.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.289+0000 7fc1b8209700 1 -- start start 2023-03-03T20:20:27.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b8209700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1b01530d0 0x7fc1b0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:27.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b8209700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00c2f30 con 0x7fc1b014fda0 2023-03-03T20:20:27.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b8209700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00c31d0 con 0x7fc1b014f9d0 2023-03-03T20:20:27.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b8209700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b01524a0 con 0x7fc1b01530d0 2023-03-03T20:20:27.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b67a6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1b01530d0 0x7fc1b0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:27.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b67a6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc1b01530d0 0x7fc1b0156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45428/0 (socket says 172.21.15.7:45428) 2023-03-03T20:20:27.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b57a4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc1b014fda0 0x7fc1b01529c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39406/0 (socket says 172.21.15.7:39406) 2023-03-03T20:20:27.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b67a6700 1 -- 172.21.15.7:0/2375355503 learned_addr learned my addr 172.21.15.7:0/2375355503 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:20:27.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b67a6700 1 -- 172.21.15.7:0/2375355503 >> v1:172.21.15.143:6789/0 conn(0x7fc1b014f9d0 legacy=0x7fc1b0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:27.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b67a6700 1 -- 172.21.15.7:0/2375355503 >> v1:172.21.15.7:6789/0 conn(0x7fc1b014fda0 legacy=0x7fc1b01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:27.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.290+0000 7fc1b67a6700 1 -- 172.21.15.7:0/2375355503 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00c34b0 con 0x7fc1b01530d0 2023-03-03T20:20:27.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b67a6700 1 --2- 172.21.15.7:0/2375355503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 0x7fc1b0156430 secure :-1 s=READY pgs=3761 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009e50 tx=0x7fc1a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69eda9873caf847 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:20:27.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b4fa3700 1 -- 172.21.15.7:0/2375355503 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800e070 con 0x7fc1b01530d0 2023-03-03T20:20:27.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b4fa3700 1 -- 172.21.15.7:0/2375355503 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a800b9c0 con 0x7fc1b01530d0 2023-03-03T20:20:27.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b4fa3700 1 -- 172.21.15.7:0/2375355503 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800bd70 con 0x7fc1b01530d0 2023-03-03T20:20:27.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b8209700 1 -- 172.21.15.7:0/2375355503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 msgr2=0x7fc1b0156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:27.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b8209700 1 --2- 172.21.15.7:0/2375355503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 0x7fc1b0156430 secure :-1 s=READY pgs=3761 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009e50 tx=0x7fc1a800b330 comp rx=0 tx=0).stop 2023-03-03T20:20:27.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b8209700 1 -- 172.21.15.7:0/2375355503 shutdown_connections 2023-03-03T20:20:27.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b8209700 1 --2- 172.21.15.7:0/2375355503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 0x7fc1b0156430 unknown :-1 s=CLOSED pgs=3761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:27.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b8209700 1 -- 172.21.15.7:0/2375355503 >> 172.21.15.7:0/2375355503 conn(0x7fc1b00c8900 msgr2=0x7fc1b00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:20:27.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b8209700 1 -- 172.21.15.7:0/2375355503 shutdown_connections 2023-03-03T20:20:27.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.291+0000 7fc1b8209700 1 -- 172.21.15.7:0/2375355503 wait complete. 2023-03-03T20:20:27.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.292+0000 7fc1b8209700 1 Processor -- start 2023-03-03T20:20:27.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b8209700 1 -- start start 2023-03-03T20:20:27.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b8209700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1b014f9d0 0x7fc1b01e0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:27.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b8209700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1b014fda0 0x7fc1b01dd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:27.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b8209700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 0x7fc1b01ddc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:27.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b8209700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b0155a30 con 0x7fc1b01530d0 2023-03-03T20:20:27.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b8209700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b01e9440 con 0x7fc1b014fda0 2023-03-03T20:20:27.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b8209700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b01ea410 con 0x7fc1b014f9d0 2023-03-03T20:20:27.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b67a6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 0x7fc1b01ddc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:27.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b67a6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 0x7fc1b01ddc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45434/0 (socket says 172.21.15.7:45434) 2023-03-03T20:20:27.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b67a6700 1 -- 172.21.15.7:0/1419163653 learned_addr learned my addr 172.21.15.7:0/1419163653 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:20:27.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b5fa5700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1b014f9d0 0x7fc1b01e0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:27.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.293+0000 7fc1b57a4700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1b014fda0 0x7fc1b01dd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:27.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1b67a6700 1 -- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1b014f9d0 msgr2=0x7fc1b01e0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:27.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1b67a6700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1b014f9d0 0x7fc1b01e0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:27.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1b67a6700 1 -- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1b014fda0 msgr2=0x7fc1b01dd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1b67a6700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1b014fda0 0x7fc1b01dd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1b67a6700 1 -- 172.21.15.7:0/1419163653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a8009b30 con 0x7fc1b01530d0 2023-03-03T20:20:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1b57a4700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1b014fda0 0x7fc1b01dd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:20:27.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1b67a6700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 0x7fc1b01ddc10 secure :-1 s=READY pgs=3762 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8012000 tx=0x7fc1a8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:20:27.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1a67fc700 1 -- 172.21.15.7:0/1419163653 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a80046a0 con 0x7fc1b01530d0 2023-03-03T20:20:27.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1a67fc700 1 -- 172.21.15.7:0/1419163653 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a8004800 con 0x7fc1b01530d0 2023-03-03T20:20:27.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1b8209700 1 -- 172.21.15.7:0/1419163653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b00ac350 con 0x7fc1b01530d0 2023-03-03T20:20:27.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1a67fc700 1 -- 172.21.15.7:0/1419163653 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8004970 con 0x7fc1b01530d0 2023-03-03T20:20:27.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.294+0000 7fc1b8209700 1 -- 172.21.15.7:0/1419163653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b00ab630 con 0x7fc1b01530d0 2023-03-03T20:20:27.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.295+0000 7fc1b8209700 1 -- 172.21.15.7:0/1419163653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b0071fa0 con 0x7fc1b01530d0 2023-03-03T20:20:27.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.295+0000 7fc1a67fc700 1 -- 172.21.15.7:0/1419163653 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fc1a8022030 con 0x7fc1b01530d0 2023-03-03T20:20:27.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.296+0000 7fc1a67fc700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1940682f0 0x7fc19406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:27.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.296+0000 7fc1a67fc700 1 -- 172.21.15.7:0/1419163653 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fc1a80879b0 con 0x7fc1b01530d0 2023-03-03T20:20:27.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.296+0000 7fc1b5fa5700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1940682f0 0x7fc19406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:27.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.296+0000 7fc1b5fa5700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1940682f0 0x7fc19406a7a0 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0156400 tx=0x7fc1ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:20:27.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.300+0000 7fc1a67fc700 1 -- 172.21.15.7:0/1419163653 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fc1a8053d50 con 0x7fc1b01530d0 2023-03-03T20:20:27.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.540+0000 7fc1b8209700 1 -- 172.21.15.7:0/1419163653 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1b0074bb0 con 0x7fc1b01530d0 2023-03-03T20:20:27.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.540+0000 7fc1a67fc700 1 -- 172.21.15.7:0/1419163653 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fc1a8057370 con 0x7fc1b01530d0 2023-03-03T20:20:27.542 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:20:27.542 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:20:27.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 -- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1940682f0 msgr2=0x7fc19406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:27.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1940682f0 0x7fc19406a7a0 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0156400 tx=0x7fc1ac009040 comp rx=0 tx=0).stop 2023-03-03T20:20:27.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 -- 172.21.15.7:0/1419163653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 msgr2=0x7fc1b01ddc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:27.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 0x7fc1b01ddc10 secure :-1 s=READY pgs=3762 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8012000 tx=0x7fc1a8004ad0 comp rx=0 tx=0).stop 2023-03-03T20:20:27.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 -- 172.21.15.7:0/1419163653 shutdown_connections 2023-03-03T20:20:27.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fc1940682f0 0x7fc19406a7a0 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:27.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1b01530d0 0x7fc1b01ddc10 unknown :-1 s=CLOSED pgs=3762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:27.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fc1b014f9d0 0x7fc1b01e0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:27.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 --2- 172.21.15.7:0/1419163653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1b014fda0 0x7fc1b01dd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:27.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 -- 172.21.15.7:0/1419163653 >> 172.21.15.7:0/1419163653 conn(0x7fc1b00c8900 msgr2=0x7fc1b0150200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:20:27.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 -- 172.21.15.7:0/1419163653 shutdown_connections 2023-03-03T20:20:27.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:27.542+0000 7fc1b8209700 1 -- 172.21.15.7:0/1419163653 wait complete. 2023-03-03T20:20:27.546 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:20:27.555 INFO:tasks.mds_thrash.fs.[b]:waiting for 96 secs before thrashing 2023-03-03T20:20:28.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:20:28.997 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:20:29.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:20:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.171+0000 7f2218bfa700 1 Processor -- start 2023-03-03T20:20:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.171+0000 7f2218bfa700 1 -- start start 2023-03-03T20:20:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.172+0000 7f2218bfa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2214071e80 0x7f2214151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.172+0000 7f2218bfa700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f221414e900 0x7f221415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.172+0000 7f2218bfa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22141510d0 con 0x7f2214072250 2023-03-03T20:20:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.172+0000 7f2218bfa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22140c3740 con 0x7f2214071e80 2023-03-03T20:20:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.172+0000 7f2218bfa700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2214070670 con 0x7f221414e900 2023-03-03T20:20:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.172+0000 7f221259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2214071e80 0x7f2214151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.172+0000 7f2211d9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2214072250 0x7f221414e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39418/0 (socket says 172.21.15.7:39418) 2023-03-03T20:20:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.172+0000 7f2211d9b700 1 -- 172.21.15.7:0/2837386429 learned_addr learned my addr 172.21.15.7:0/2837386429 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:20:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.172+0000 7f2212d9d700 1 --2- 172.21.15.7:0/2837386429 >> v2:172.21.15.143:3300/0 conn(0x7f221414e900 0x7f221415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f221259c700 1 -- 172.21.15.7:0/2837386429 >> v2:172.21.15.143:3300/0 conn(0x7f221414e900 msgr2=0x7f221415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:20:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f221259c700 1 --2- 172.21.15.7:0/2837386429 >> v2:172.21.15.143:3300/0 conn(0x7f221414e900 0x7f221415ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f221259c700 1 -- 172.21.15.7:0/2837386429 >> v1:172.21.15.7:6789/0 conn(0x7f2214072250 legacy=0x7f221414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f221259c700 1 -- 172.21.15.7:0/2837386429 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22140c39e0 con 0x7f2214071e80 2023-03-03T20:20:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f221159a700 1 -- 172.21.15.7:0/2837386429 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127717147 0 0) 0x7f22141510d0 con 0x7f2214072250 2023-03-03T20:20:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f2212d9d700 1 --2- 172.21.15.7:0/2837386429 >> v2:172.21.15.143:3300/0 conn(0x7f221414e900 0x7f221415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:20:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f221259c700 1 --2- 172.21.15.7:0/2837386429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214071e80 0x7f2214151820 secure :-1 s=READY pgs=3764 cs=0 l=1 rev1=1 crypto rx=0x7f2214070de0 tx=0x7f21fc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189f1485857c99ff server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:20:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f221159a700 1 -- 172.21.15.7:0/2837386429 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc011830 con 0x7f2214071e80 2023-03-03T20:20:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f221159a700 1 -- 172.21.15.7:0/2837386429 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc011990 con 0x7f2214071e80 2023-03-03T20:20:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f221159a700 1 -- 172.21.15.7:0/2837386429 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc011d20 con 0x7f2214071e80 2023-03-03T20:20:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f2218bfa700 1 -- 172.21.15.7:0/2837386429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214071e80 msgr2=0x7f2214151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.173+0000 7f2218bfa700 1 --2- 172.21.15.7:0/2837386429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214071e80 0x7f2214151820 secure :-1 s=READY pgs=3764 cs=0 l=1 rev1=1 crypto rx=0x7f2214070de0 tx=0x7f21fc009b20 comp rx=0 tx=0).stop 2023-03-03T20:20:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.174+0000 7f2218bfa700 1 -- 172.21.15.7:0/2837386429 shutdown_connections 2023-03-03T20:20:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.174+0000 7f2218bfa700 1 --2- 172.21.15.7:0/2837386429 >> v2:172.21.15.143:3300/0 conn(0x7f221414e900 0x7f221415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.174+0000 7f2218bfa700 1 --2- 172.21.15.7:0/2837386429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214071e80 0x7f2214151820 unknown :-1 s=CLOSED pgs=3764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.174+0000 7f2218bfa700 1 -- 172.21.15.7:0/2837386429 >> 172.21.15.7:0/2837386429 conn(0x7f22140a94d0 msgr2=0x7f22140a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:20:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.174+0000 7f2218bfa700 1 -- 172.21.15.7:0/2837386429 shutdown_connections 2023-03-03T20:20:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.174+0000 7f2218bfa700 1 -- 172.21.15.7:0/2837386429 wait complete. 2023-03-03T20:20:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.175+0000 7f2218bfa700 1 Processor -- start 2023-03-03T20:20:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.175+0000 7f2218bfa700 1 -- start start 2023-03-03T20:20:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.176+0000 7f2218bfa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2214071e80 0x7f22141ed6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.176+0000 7f2218bfa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214072250 0x7f2214150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.176+0000 7f2218bfa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f221414e900 0x7f2214154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.176+0000 7f2218bfa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22141eb0e0 con 0x7f2214072250 2023-03-03T20:20:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.176+0000 7f2218bfa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22141edc20 con 0x7f221414e900 2023-03-03T20:20:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.176+0000 7f2218bfa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22140c7110 con 0x7f2214071e80 2023-03-03T20:20:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.176+0000 7f2211d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214072250 0x7f2214150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.176+0000 7f2211d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214072250 0x7f2214150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45444/0 (socket says 172.21.15.7:45444) 2023-03-03T20:20:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.176+0000 7f2211d9b700 1 -- 172.21.15.7:0/1785669587 learned_addr learned my addr 172.21.15.7:0/1785669587 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:20:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f221259c700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2214071e80 0x7f22141ed6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:29.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f2212d9d700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f221414e900 0x7f2214154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f2211d9b700 1 -- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2214071e80 msgr2=0x7f22141ed6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f2211d9b700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2214071e80 0x7f22141ed6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f2211d9b700 1 -- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f221414e900 msgr2=0x7f2214154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f2211d9b700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f221414e900 0x7f2214154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f2211d9b700 1 -- 172.21.15.7:0/1785669587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc0094a0 con 0x7f2214072250 2023-03-03T20:20:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f221259c700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2214071e80 0x7f22141ed6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:20:29.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f2211d9b700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214072250 0x7f2214150200 secure :-1 s=READY pgs=3765 cs=0 l=1 rev1=1 crypto rx=0x7f220400c570 tx=0x7f220400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:20:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f220affd700 1 -- 172.21.15.7:0/1785669587 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2204004c80 con 0x7f2214072250 2023-03-03T20:20:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f220affd700 1 -- 172.21.15.7:0/1785669587 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2204004de0 con 0x7f2214072250 2023-03-03T20:20:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f2218bfa700 1 -- 172.21.15.7:0/1785669587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221406c9a0 con 0x7f2214072250 2023-03-03T20:20:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f220affd700 1 -- 172.21.15.7:0/1785669587 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2204007310 con 0x7f2214072250 2023-03-03T20:20:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.177+0000 7f2218bfa700 1 -- 172.21.15.7:0/1785669587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2214151300 con 0x7f2214072250 2023-03-03T20:20:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.178+0000 7f2218bfa700 1 -- 172.21.15.7:0/1785669587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f221406cb30 con 0x7f2214072250 2023-03-03T20:20:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.178+0000 7f220affd700 1 -- 172.21.15.7:0/1785669587 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f220401a020 con 0x7f2214072250 2023-03-03T20:20:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.179+0000 7f220affd700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21f40682f0 0x7f21f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:20:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.179+0000 7f220affd700 1 -- 172.21.15.7:0/1785669587 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2204085e40 con 0x7f2214072250 2023-03-03T20:20:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.179+0000 7f221259c700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21f40682f0 0x7f21f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:20:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.180+0000 7f221259c700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21f40682f0 0x7f21f406a7a0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0090b0 tx=0x7f21fc008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:20:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.183+0000 7f220affd700 1 -- 172.21.15.7:0/1785669587 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f22040521a0 con 0x7f2214072250 2023-03-03T20:20:29.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.422+0000 7f2218bfa700 1 -- 172.21.15.7:0/1785669587 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22141eb220 con 0x7f2214072250 2023-03-03T20:20:29.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.422+0000 7f220affd700 1 -- 172.21.15.7:0/1785669587 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f22141eb220 con 0x7f2214072250 2023-03-03T20:20:29.423 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:20:29.423 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:20:29.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 -- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21f40682f0 msgr2=0x7f21f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:29.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21f40682f0 0x7f21f406a7a0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0090b0 tx=0x7f21fc008030 comp rx=0 tx=0).stop 2023-03-03T20:20:29.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 -- 172.21.15.7:0/1785669587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214072250 msgr2=0x7f2214150200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:20:29.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214072250 0x7f2214150200 secure :-1 s=READY pgs=3765 cs=0 l=1 rev1=1 crypto rx=0x7f220400c570 tx=0x7f220400c930 comp rx=0 tx=0).stop 2023-03-03T20:20:29.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 -- 172.21.15.7:0/1785669587 shutdown_connections 2023-03-03T20:20:29.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f221414e900 0x7f2214154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:29.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2214071e80 0x7f22141ed6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:29.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f21f40682f0 0x7f21f406a7a0 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:29.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 --2- 172.21.15.7:0/1785669587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2214072250 0x7f2214150200 unknown :-1 s=CLOSED pgs=3765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:20:29.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 -- 172.21.15.7:0/1785669587 >> 172.21.15.7:0/1785669587 conn(0x7f22140a94d0 msgr2=0x7f221406da80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:20:29.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 -- 172.21.15.7:0/1785669587 shutdown_connections 2023-03-03T20:20:29.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:20:29.424+0000 7f2218bfa700 1 -- 172.21.15.7:0/1785669587 wait complete. 2023-03-03T20:20:29.426 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:20:29.438 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2023-03-03T20:20:59.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:20:59.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:21:29.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:21:29.292 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:21:48.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:21:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.577+0000 7f13bf31d700 1 Processor -- start 2023-03-03T20:21:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.577+0000 7f13bf31d700 1 -- start start 2023-03-03T20:21:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13bf31d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f13b806c6c0 0x7f13b80736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:21:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13bf31d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b8157ff0 con 0x7f13b80c2e30 2023-03-03T20:21:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13bf31d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b806d640 con 0x7f13b8151cd0 2023-03-03T20:21:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13bf31d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b806cda0 con 0x7f13b806c6c0 2023-03-03T20:21:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13bc8b8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f13b80c2e30 0x7f13b8158480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39464/0 (socket says 172.21.15.7:39464) 2023-03-03T20:21:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13bc8b8700 1 -- 172.21.15.7:0/2138682808 learned_addr learned my addr 172.21.15.7:0/2138682808 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:21:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13bd8ba700 1 --2- 172.21.15.7:0/2138682808 >> v2:172.21.15.143:3300/0 conn(0x7f13b806c6c0 0x7f13b80736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:21:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526344992 0 0) 0x7f13b8157ff0 con 0x7f13b80c2e30 2023-03-03T20:21:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a40035b0 con 0x7f13b80c2e30 2023-03-03T20:21:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3572717052 0 0) 0x7f13a40035b0 con 0x7f13b80c2e30 2023-03-03T20:21:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 >> v2:172.21.15.143:3300/0 conn(0x7f13b806c6c0 msgr2=0x7f13b80736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:21:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13affff700 1 --2- 172.21.15.7:0/2138682808 >> v2:172.21.15.143:3300/0 conn(0x7f13b806c6c0 0x7f13b80736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:21:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 >> v1:172.21.15.143:6789/0 conn(0x7f13b8151cd0 legacy=0x7f13b8154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:21:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.578+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b806d080 con 0x7f13b80c2e30 2023-03-03T20:21:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f13a8001670 con 0x7f13b80c2e30 2023-03-03T20:21:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 >> v1:172.21.15.7:6789/0 conn(0x7f13b80c2e30 legacy=0x7f13b8158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:21:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13a4003a70 0x7f13a4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:21:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13a4004850 0x7f13a4004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:21:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a4005320 0x7f13a400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:21:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b806cda0 con 0x7f13a4003a70 2023-03-03T20:21:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a400c230 con 0x7f13a4005320 2023-03-03T20:21:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a400c3f0 con 0x7f13a4004850 2023-03-03T20:21:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13bd0b9700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13a4003a70 0x7f13a4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:21:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13bc8b8700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13a4004850 0x7f13a4004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:21:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13bd8ba700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a4005320 0x7f13a400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:21:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13bd0b9700 1 -- 172.21.15.7:0/2138682808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13a4004850 msgr2=0x7f13a4004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:21:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13bd0b9700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13a4004850 0x7f13a4004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:21:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13bd0b9700 1 -- 172.21.15.7:0/2138682808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a4005320 msgr2=0x7f13a400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:21:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13bd0b9700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a4005320 0x7f13a400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:21:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13bd0b9700 1 -- 172.21.15.7:0/2138682808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a40035b0 con 0x7f13a4003a70 2023-03-03T20:21:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13bd8ba700 1 -- 172.21.15.7:0/2138682808 reap_dead start 2023-03-03T20:21:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13bd0b9700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13a4003a70 0x7f13a4003e80 secure :-1 s=READY pgs=3767 cs=0 l=1 rev1=1 crypto rx=0x7f13b4001800 tx=0x7f13a4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e4a235e71e427 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:21:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b40012f0 con 0x7f13a4003a70 2023-03-03T20:21:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b4005310 con 0x7f13a4003a70 2023-03-03T20:21:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.579+0000 7f13affff700 1 -- 172.21.15.7:0/2138682808 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b4006860 con 0x7f13a4003a70 2023-03-03T20:21:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.580+0000 7f13bf31d700 1 -- 172.21.15.7:0/2138682808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13a4003a70 msgr2=0x7f13a4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:21:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.580+0000 7f13bf31d700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13a4003a70 0x7f13a4003e80 secure :-1 s=READY pgs=3767 cs=0 l=1 rev1=1 crypto rx=0x7f13b4001800 tx=0x7f13a4005880 comp rx=0 tx=0).stop 2023-03-03T20:21:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.580+0000 7f13bf31d700 1 -- 172.21.15.7:0/2138682808 shutdown_connections 2023-03-03T20:21:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.580+0000 7f13bf31d700 1 --2- 172.21.15.7:0/2138682808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13a4003a70 0x7f13a4003e80 unknown :-1 s=CLOSED pgs=3767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:21:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.580+0000 7f13bf31d700 1 -- 172.21.15.7:0/2138682808 >> 172.21.15.7:0/2138682808 conn(0x7f13b80a94d0 msgr2=0x7f13b80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:21:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.580+0000 7f13bf31d700 1 -- 172.21.15.7:0/2138682808 shutdown_connections 2023-03-03T20:21:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.580+0000 7f13bf31d700 1 -- 172.21.15.7:0/2138682808 wait complete. 2023-03-03T20:21:48.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.581+0000 7f13bf31d700 1 Processor -- start 2023-03-03T20:21:48.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.581+0000 7f13bf31d700 1 -- start start 2023-03-03T20:21:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bf31d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a4003a70 0x7f13b80acf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:21:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bf31d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13b80ac9d0 0x7f13b81ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:21:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bf31d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13b80c64b0 0x7f13b8155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:21:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bf31d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80aa810 con 0x7f13b80c64b0 2023-03-03T20:21:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bf31d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b80c7090 con 0x7f13a4003a70 2023-03-03T20:21:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bf31d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b81540b0 con 0x7f13b80ac9d0 2023-03-03T20:21:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bd8ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13b80c64b0 0x7f13b8155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:21:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bd8ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13b80c64b0 0x7f13b8155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45498/0 (socket says 172.21.15.7:45498) 2023-03-03T20:21:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bd8ba700 1 -- 172.21.15.7:0/4032412839 learned_addr learned my addr 172.21.15.7:0/4032412839 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:21:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bc8b8700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13b80ac9d0 0x7f13b81ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:21:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bd0b9700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a4003a70 0x7f13b80acf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:21:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bd8ba700 1 -- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13b80ac9d0 msgr2=0x7f13b81ee260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:21:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bd8ba700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13b80ac9d0 0x7f13b81ee260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:21:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bd8ba700 1 -- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a4003a70 msgr2=0x7f13b80acf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:21:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bd8ba700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a4003a70 0x7f13b80acf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:21:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bd8ba700 1 -- 172.21.15.7:0/4032412839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b8157c80 con 0x7f13b80c64b0 2023-03-03T20:21:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.582+0000 7f13bc8b8700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13b80ac9d0 0x7f13b81ee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:21:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.583+0000 7f13bd8ba700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13b80c64b0 0x7f13b8155900 secure :-1 s=READY pgs=3768 cs=0 l=1 rev1=1 crypto rx=0x7f13a4005830 tx=0x7f13b0011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:21:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.583+0000 7f13adffb700 1 -- 172.21.15.7:0/4032412839 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b000e040 con 0x7f13b80c64b0 2023-03-03T20:21:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.583+0000 7f13adffb700 1 -- 172.21.15.7:0/4032412839 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b000f040 con 0x7f13b80c64b0 2023-03-03T20:21:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.583+0000 7f13bf31d700 1 -- 172.21.15.7:0/4032412839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b806d360 con 0x7f13b80c64b0 2023-03-03T20:21:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.583+0000 7f13adffb700 1 -- 172.21.15.7:0/4032412839 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b000fde0 con 0x7f13b80c64b0 2023-03-03T20:21:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.583+0000 7f13bf31d700 1 -- 172.21.15.7:0/4032412839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b8158480 con 0x7f13b80c64b0 2023-03-03T20:21:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.584+0000 7f13bf31d700 1 -- 172.21.15.7:0/4032412839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b81511e0 con 0x7f13b80c64b0 2023-03-03T20:21:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.584+0000 7f13adffb700 1 -- 172.21.15.7:0/4032412839 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f13b0022460 con 0x7f13b80c64b0 2023-03-03T20:21:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.585+0000 7f13adffb700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f139c0682f0 0x7f139c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:21:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.585+0000 7f13adffb700 1 -- 172.21.15.7:0/4032412839 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f13b0086860 con 0x7f13b80c64b0 2023-03-03T20:21:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.585+0000 7f13bd0b9700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f139c0682f0 0x7f139c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:21:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.586+0000 7f13bd0b9700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f139c0682f0 0x7f139c06a7a0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f13b4001800 tx=0x7f13b4009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:21:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.589+0000 7f13adffb700 1 -- 172.21.15.7:0/4032412839 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f13b0052ba0 con 0x7f13b80c64b0 2023-03-03T20:21:48.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.829+0000 7f13bf31d700 1 -- 172.21.15.7:0/4032412839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13b81514c0 con 0x7f13b80c64b0 2023-03-03T20:21:48.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.830+0000 7f13adffb700 1 -- 172.21.15.7:0/4032412839 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f13b00561c0 con 0x7f13b80c64b0 2023-03-03T20:21:48.831 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:21:48.831 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:21:48.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 -- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f139c0682f0 msgr2=0x7f139c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:21:48.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f139c0682f0 0x7f139c06a7a0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f13b4001800 tx=0x7f13b4009030 comp rx=0 tx=0).stop 2023-03-03T20:21:48.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 -- 172.21.15.7:0/4032412839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13b80c64b0 msgr2=0x7f13b8155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:21:48.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13b80c64b0 0x7f13b8155900 secure :-1 s=READY pgs=3768 cs=0 l=1 rev1=1 crypto rx=0x7f13a4005830 tx=0x7f13b0011f80 comp rx=0 tx=0).stop 2023-03-03T20:21:48.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 -- 172.21.15.7:0/4032412839 shutdown_connections 2023-03-03T20:21:48.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f139c0682f0 0x7f139c06a7a0 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:21:48.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13b80c64b0 0x7f13b8155900 unknown :-1 s=CLOSED pgs=3768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:21:48.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a4003a70 0x7f13b80acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:21:48.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 --2- 172.21.15.7:0/4032412839 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f13b80ac9d0 0x7f13b81ee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:21:48.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 -- 172.21.15.7:0/4032412839 >> 172.21.15.7:0/4032412839 conn(0x7f13b80a94d0 msgr2=0x7f13b814d620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:21:48.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 -- 172.21.15.7:0/4032412839 shutdown_connections 2023-03-03T20:21:48.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:21:48.832+0000 7f13bf31d700 1 -- 172.21.15.7:0/4032412839 wait complete. 2023-03-03T20:21:48.834 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:21:48.844 INFO:tasks.mds_thrash.fs.[a]:waiting for 110 secs before thrashing 2023-03-03T20:21:59.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:21:59.355 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:22:03.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:22:03.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.694+0000 7faf5fdf2700 1 Processor -- start 2023-03-03T20:22:03.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.695+0000 7faf5fdf2700 1 -- start start 2023-03-03T20:22:03.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.695+0000 7faf5fdf2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf58072250 0x7faf5814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:03.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.695+0000 7faf5fdf2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faf580c1ce0 0x7faf5814e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:03.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.695+0000 7faf5fdf2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf581510d0 con 0x7faf58071e80 2023-03-03T20:22:03.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.695+0000 7faf5fdf2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf580c3740 con 0x7faf580c1ce0 2023-03-03T20:22:03.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.695+0000 7faf5fdf2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf58070670 con 0x7faf58072250 2023-03-03T20:22:03.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5e38f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faf580c1ce0 0x7faf5814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:03.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5db8e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faf58071e80 0x7faf58151820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39486/0 (socket says 172.21.15.7:39486) 2023-03-03T20:22:03.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5db8e700 1 -- 172.21.15.7:0/4025849651 learned_addr learned my addr 172.21.15.7:0/4025849651 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:22:03.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5d38d700 1 --2- 172.21.15.7:0/4025849651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf58072250 0x7faf5814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:03.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5e38f700 1 -- 172.21.15.7:0/4025849651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf58072250 msgr2=0x7faf5814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:22:03.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5e38f700 1 --2- 172.21.15.7:0/4025849651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf58072250 0x7faf5814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5e38f700 1 -- 172.21.15.7:0/4025849651 >> v1:172.21.15.7:6789/0 conn(0x7faf58071e80 legacy=0x7faf58151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:03.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5cb8c700 1 -- 172.21.15.7:0/4025849651 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555324408 0 0) 0x7faf581510d0 con 0x7faf58071e80 2023-03-03T20:22:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5e38f700 1 -- 172.21.15.7:0/4025849651 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf580c39e0 con 0x7faf580c1ce0 2023-03-03T20:22:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5e38f700 1 --2- 172.21.15.7:0/4025849651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 0x7faf5814e730 secure :-1 s=READY pgs=3770 cs=0 l=1 rev1=1 crypto rx=0x7faf50009d90 tx=0x7faf5000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67cc87f1aaec0ae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:22:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5cb8c700 1 -- 172.21.15.7:0/4025849651 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5000e070 con 0x7faf580c1ce0 2023-03-03T20:22:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5cb8c700 1 -- 172.21.15.7:0/4025849651 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5000bbd0 con 0x7faf580c1ce0 2023-03-03T20:22:03.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.696+0000 7faf5cb8c700 1 -- 172.21.15.7:0/4025849651 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50012430 con 0x7faf580c1ce0 2023-03-03T20:22:03.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.697+0000 7faf5fdf2700 1 -- 172.21.15.7:0/4025849651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 msgr2=0x7faf5814e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:03.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.697+0000 7faf5fdf2700 1 --2- 172.21.15.7:0/4025849651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 0x7faf5814e730 secure :-1 s=READY pgs=3770 cs=0 l=1 rev1=1 crypto rx=0x7faf50009d90 tx=0x7faf5000b330 comp rx=0 tx=0).stop 2023-03-03T20:22:03.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.697+0000 7faf5fdf2700 1 -- 172.21.15.7:0/4025849651 shutdown_connections 2023-03-03T20:22:03.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.697+0000 7faf5fdf2700 1 --2- 172.21.15.7:0/4025849651 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 0x7faf5814e730 unknown :-1 s=CLOSED pgs=3770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.697+0000 7faf5fdf2700 1 --2- 172.21.15.7:0/4025849651 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf58072250 0x7faf5814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.697+0000 7faf5fdf2700 1 -- 172.21.15.7:0/4025849651 >> 172.21.15.7:0/4025849651 conn(0x7faf580a94d0 msgr2=0x7faf580a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:22:03.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.697+0000 7faf5fdf2700 1 -- 172.21.15.7:0/4025849651 shutdown_connections 2023-03-03T20:22:03.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.697+0000 7faf5fdf2700 1 -- 172.21.15.7:0/4025849651 wait complete. 2023-03-03T20:22:03.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.699+0000 7faf5fdf2700 1 Processor -- start 2023-03-03T20:22:03.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.699+0000 7faf5fdf2700 1 -- start start 2023-03-03T20:22:03.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.699+0000 7faf5fdf2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf58071e80 0x7faf5814a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:03.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.699+0000 7faf5fdf2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf58072250 0x7faf58148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:03.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5fdf2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 0x7faf58145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:03.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5fdf2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5813f460 con 0x7faf580c1ce0 2023-03-03T20:22:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5fdf2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5814a750 con 0x7faf58072250 2023-03-03T20:22:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5fdf2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf58149000 con 0x7faf58071e80 2023-03-03T20:22:03.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5e38f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 0x7faf58145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5e38f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 0x7faf58145600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45516/0 (socket says 172.21.15.7:45516) 2023-03-03T20:22:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5e38f700 1 -- 172.21.15.7:0/1436551995 learned_addr learned my addr 172.21.15.7:0/1436551995 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:22:03.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5db8e700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf58071e80 0x7faf5814a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:03.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5d38d700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf58072250 0x7faf58148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:03.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5e38f700 1 -- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf58071e80 msgr2=0x7faf5814a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:03.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5e38f700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf58071e80 0x7faf5814a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5e38f700 1 -- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf58072250 msgr2=0x7faf58148700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:03.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5e38f700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf58072250 0x7faf58148700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5e38f700 1 -- 172.21.15.7:0/1436551995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf50009a70 con 0x7faf580c1ce0 2023-03-03T20:22:03.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5e38f700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 0x7faf58145600 secure :-1 s=READY pgs=3771 cs=0 l=1 rev1=1 crypto rx=0x7faf5806b550 tx=0x7faf50004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:22:03.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf4a7fc700 1 -- 172.21.15.7:0/1436551995 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50012430 con 0x7faf580c1ce0 2023-03-03T20:22:03.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf4a7fc700 1 -- 172.21.15.7:0/1436551995 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf50020070 con 0x7faf580c1ce0 2023-03-03T20:22:03.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf5fdf2700 1 -- 172.21.15.7:0/1436551995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf58151f30 con 0x7faf580c1ce0 2023-03-03T20:22:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.700+0000 7faf4a7fc700 1 -- 172.21.15.7:0/1436551995 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf500098f0 con 0x7faf580c1ce0 2023-03-03T20:22:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.701+0000 7faf5fdf2700 1 -- 172.21.15.7:0/1436551995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf58151300 con 0x7faf580c1ce0 2023-03-03T20:22:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.701+0000 7faf5fdf2700 1 -- 172.21.15.7:0/1436551995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5806c9a0 con 0x7faf580c1ce0 2023-03-03T20:22:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.702+0000 7faf4a7fc700 1 -- 172.21.15.7:0/1436551995 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7faf50004040 con 0x7faf580c1ce0 2023-03-03T20:22:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.702+0000 7faf4a7fc700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf3c0682f0 0x7faf3c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.702+0000 7faf4a7fc700 1 -- 172.21.15.7:0/1436551995 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7faf50022070 con 0x7faf580c1ce0 2023-03-03T20:22:03.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.703+0000 7faf5db8e700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf3c0682f0 0x7faf3c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:03.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.703+0000 7faf5db8e700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf3c0682f0 0x7faf3c06a7a0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7faf54007fd0 tx=0x7faf54007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:22:03.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.707+0000 7faf4a7fc700 1 -- 172.21.15.7:0/1436551995 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7faf50054740 con 0x7faf580c1ce0 2023-03-03T20:22:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.942+0000 7faf5fdf2700 1 -- 172.21.15.7:0/1436551995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf58074bb0 con 0x7faf580c1ce0 2023-03-03T20:22:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.943+0000 7faf4a7fc700 1 -- 172.21.15.7:0/1436551995 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7faf50057d60 con 0x7faf580c1ce0 2023-03-03T20:22:03.945 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:22:03.945 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:22:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.944+0000 7faf5fdf2700 1 -- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf3c0682f0 msgr2=0x7faf3c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.944+0000 7faf5fdf2700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf3c0682f0 0x7faf3c06a7a0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7faf54007fd0 tx=0x7faf54007de0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.944+0000 7faf5fdf2700 1 -- 172.21.15.7:0/1436551995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 msgr2=0x7faf58145600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.944+0000 7faf5fdf2700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 0x7faf58145600 secure :-1 s=READY pgs=3771 cs=0 l=1 rev1=1 crypto rx=0x7faf5806b550 tx=0x7faf50004ad0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.945+0000 7faf5fdf2700 1 -- 172.21.15.7:0/1436551995 shutdown_connections 2023-03-03T20:22:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.945+0000 7faf5fdf2700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7faf3c0682f0 0x7faf3c06a7a0 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.945+0000 7faf5fdf2700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faf580c1ce0 0x7faf58145600 unknown :-1 s=CLOSED pgs=3771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.945+0000 7faf5fdf2700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7faf58071e80 0x7faf5814a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.945+0000 7faf5fdf2700 1 --2- 172.21.15.7:0/1436551995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faf58072250 0x7faf58148700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.945+0000 7faf5fdf2700 1 -- 172.21.15.7:0/1436551995 >> 172.21.15.7:0/1436551995 conn(0x7faf580a94d0 msgr2=0x7faf5814dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:22:03.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.945+0000 7faf5fdf2700 1 -- 172.21.15.7:0/1436551995 shutdown_connections 2023-03-03T20:22:03.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:03.945+0000 7faf5fdf2700 1 -- 172.21.15.7:0/1436551995 wait complete. 2023-03-03T20:22:03.948 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:22:03.958 INFO:tasks.mds_thrash.fs.[b]:waiting for 32 secs before thrashing 2023-03-03T20:22:29.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:22:29.441 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:22:35.960 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:22:36.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.098+0000 7f6317274700 1 Processor -- start 2023-03-03T20:22:36.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.099+0000 7f6317274700 1 -- start start 2023-03-03T20:22:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.099+0000 7f6317274700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6310071e80 0x7f6310151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:36.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.099+0000 7f6317274700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63101510d0 con 0x7f631014e900 2023-03-03T20:22:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.099+0000 7f6317274700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63100c3740 con 0x7f6310072250 2023-03-03T20:22:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.099+0000 7f6317274700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6310070670 con 0x7f6310071e80 2023-03-03T20:22:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6315811700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f631014e900 0x7f631015ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39528/0 (socket says 172.21.15.7:39528) 2023-03-03T20:22:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6315811700 1 -- 172.21.15.7:0/1055048476 learned_addr learned my addr 172.21.15.7:0/1055048476 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:22:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6315010700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6310071e80 0x7f6310151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837821546 0 0) 0x7f63101510d0 con 0x7f631014e900 2023-03-03T20:22:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62fc0035b0 con 0x7f631014e900 2023-03-03T20:22:36.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 114981242 0 0) 0x7f62fc0035b0 con 0x7f631014e900 2023-03-03T20:22:36.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6310071e80 msgr2=0x7f6310151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:22:36.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6310071e80 0x7f6310151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:36.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 >> v1:172.21.15.143:6789/0 conn(0x7f6310072250 legacy=0x7f631014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:36.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63100c39e0 con 0x7f631014e900 2023-03-03T20:22:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f63080019f0 con 0x7f631014e900 2023-03-03T20:22:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 >> v1:172.21.15.7:6789/0 conn(0x7f631014e900 legacy=0x7f631015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62fc003aa0 0x7f62fc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6310071e80 0x7f62fc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62fc004d90 0x7f62fc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6310070670 con 0x7f62fc003aa0 2023-03-03T20:22:36.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc00bb10 con 0x7f62fc004d90 2023-03-03T20:22:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc00dee0 con 0x7f6310071e80 2023-03-03T20:22:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f6315010700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62fc003aa0 0x7f62fc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.100+0000 7f631480f700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6310071e80 0x7f62fc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6315811700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62fc004d90 0x7f62fc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6315010700 1 -- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6310071e80 msgr2=0x7f62fc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:22:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6315010700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6310071e80 0x7f62fc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6315010700 1 -- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62fc004d90 msgr2=0x7f62fc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:22:36.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6315010700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62fc004d90 0x7f62fc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6315010700 1 -- 172.21.15.7:0/1055048476 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc0035b0 con 0x7f62fc003aa0 2023-03-03T20:22:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6315811700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62fc004d90 0x7f62fc005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:22:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6315010700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62fc003aa0 0x7f62fc003e70 secure :-1 s=READY pgs=3773 cs=0 l=1 rev1=1 crypto rx=0x7f630c009b60 tx=0x7f62fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd03fcf16d9a2581 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:22:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c00e6c0 con 0x7f62fc003aa0 2023-03-03T20:22:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c00e820 con 0x7f62fc003aa0 2023-03-03T20:22:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.101+0000 7f6303fff700 1 -- 172.21.15.7:0/1055048476 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c00eb90 con 0x7f62fc003aa0 2023-03-03T20:22:36.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.102+0000 7f6317274700 1 -- 172.21.15.7:0/1055048476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62fc003aa0 msgr2=0x7f62fc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.102+0000 7f6317274700 1 --2- 172.21.15.7:0/1055048476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62fc003aa0 0x7f62fc003e70 secure :-1 s=READY pgs=3773 cs=0 l=1 rev1=1 crypto rx=0x7f630c009b60 tx=0x7f62fc005880 comp rx=0 tx=0).stop 2023-03-03T20:22:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.102+0000 7f6315811700 1 -- 172.21.15.7:0/1055048476 reap_dead start 2023-03-03T20:22:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.102+0000 7f6317274700 1 -- 172.21.15.7:0/1055048476 shutdown_connections 2023-03-03T20:22:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.102+0000 7f6317274700 1 -- 172.21.15.7:0/1055048476 >> 172.21.15.7:0/1055048476 conn(0x7f63100a94d0 msgr2=0x7f63100a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:22:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.102+0000 7f6317274700 1 -- 172.21.15.7:0/1055048476 shutdown_connections 2023-03-03T20:22:36.108 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.102+0000 7f6317274700 1 -- 172.21.15.7:0/1055048476 wait complete. 2023-03-03T20:22:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.103+0000 7f6317274700 1 Processor -- start 2023-03-03T20:22:36.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.103+0000 7f6317274700 1 -- start start 2023-03-03T20:22:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f6317274700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62fc003aa0 0x7f63100c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f6317274700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631014a050 0x7f63101473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f6317274700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6310147900 0x7f63100abe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f6317274700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63100aba00 con 0x7f631014a050 2023-03-03T20:22:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f6317274700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631014a440 con 0x7f6310147900 2023-03-03T20:22:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f6317274700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6310147d80 con 0x7f62fc003aa0 2023-03-03T20:22:36.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f631480f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631014a050 0x7f63101473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f631480f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631014a050 0x7f63101473c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45558/0 (socket says 172.21.15.7:45558) 2023-03-03T20:22:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f631480f700 1 -- 172.21.15.7:0/2851214562 learned_addr learned my addr 172.21.15.7:0/2851214562 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:22:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f631480f700 1 -- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62fc003aa0 msgr2=0x7f63100c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f631480f700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62fc003aa0 0x7f63100c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f631480f700 1 -- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6310147900 msgr2=0x7f63100abe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f6315811700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6310147900 0x7f63100abe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:36.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f631480f700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6310147900 0x7f63100abe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f631480f700 1 -- 172.21.15.7:0/2851214562 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631006c9a0 con 0x7f631014a050 2023-03-03T20:22:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f631480f700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631014a050 0x7f63101473c0 secure :-1 s=READY pgs=3774 cs=0 l=1 rev1=1 crypto rx=0x7f630400f7b0 tx=0x7f630400fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:22:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.104+0000 7f6301ffb700 1 -- 172.21.15.7:0/2851214562 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63040052e0 con 0x7f631014a050 2023-03-03T20:22:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.105+0000 7f6301ffb700 1 -- 172.21.15.7:0/2851214562 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6304005460 con 0x7f631014a050 2023-03-03T20:22:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.105+0000 7f6301ffb700 1 -- 172.21.15.7:0/2851214562 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63040038c0 con 0x7f631014a050 2023-03-03T20:22:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.105+0000 7f6317274700 1 -- 172.21.15.7:0/2851214562 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6310151300 con 0x7f631014a050 2023-03-03T20:22:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.105+0000 7f6317274700 1 -- 172.21.15.7:0/2851214562 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631006cb30 con 0x7f631014a050 2023-03-03T20:22:36.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.106+0000 7f6317274700 1 -- 172.21.15.7:0/2851214562 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63100ab760 con 0x7f631014a050 2023-03-03T20:22:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.106+0000 7f6301ffb700 1 -- 172.21.15.7:0/2851214562 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6304003a20 con 0x7f631014a050 2023-03-03T20:22:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.106+0000 7f6301ffb700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62f40682f0 0x7f62f406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:22:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.107+0000 7f6301ffb700 1 -- 172.21.15.7:0/2851214562 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f63040862a0 con 0x7f631014a050 2023-03-03T20:22:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.107+0000 7f6315010700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62f40682f0 0x7f62f406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:22:36.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.107+0000 7f6315010700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62f40682f0 0x7f62f406a7a0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f630c0090b0 tx=0x7f630c013000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:22:36.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.111+0000 7f6301ffb700 1 -- 172.21.15.7:0/2851214562 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6304052550 con 0x7f631014a050 2023-03-03T20:22:36.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.351+0000 7f6317274700 1 -- 172.21.15.7:0/2851214562 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63100abb40 con 0x7f631014a050 2023-03-03T20:22:36.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.352+0000 7f6301ffb700 1 -- 172.21.15.7:0/2851214562 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f63100abb40 con 0x7f631014a050 2023-03-03T20:22:36.353 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:22:36.353 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:22:36.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 -- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62f40682f0 msgr2=0x7f62f406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:36.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62f40682f0 0x7f62f406a7a0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f630c0090b0 tx=0x7f630c013000 comp rx=0 tx=0).stop 2023-03-03T20:22:36.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 -- 172.21.15.7:0/2851214562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631014a050 msgr2=0x7f63101473c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:22:36.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631014a050 0x7f63101473c0 secure :-1 s=READY pgs=3774 cs=0 l=1 rev1=1 crypto rx=0x7f630400f7b0 tx=0x7f630400fb70 comp rx=0 tx=0).stop 2023-03-03T20:22:36.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 -- 172.21.15.7:0/2851214562 shutdown_connections 2023-03-03T20:22:36.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6310147900 0x7f63100abe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:36.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f62fc003aa0 0x7f63100c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:36.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f62f40682f0 0x7f62f406a7a0 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:36.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 --2- 172.21.15.7:0/2851214562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f631014a050 0x7f63101473c0 unknown :-1 s=CLOSED pgs=3774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:22:36.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 -- 172.21.15.7:0/2851214562 >> 172.21.15.7:0/2851214562 conn(0x7f63100a94d0 msgr2=0x7f631006d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:22:36.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 -- 172.21.15.7:0/2851214562 shutdown_connections 2023-03-03T20:22:36.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:22:36.354+0000 7f6317274700 1 -- 172.21.15.7:0/2851214562 wait complete. 2023-03-03T20:22:36.357 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:22:36.367 INFO:tasks.mds_thrash.fs.[b]:waiting for 28 secs before thrashing 2023-03-03T20:22:59.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:22:59.506 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:23:04.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:23:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.510+0000 7f024383e700 1 Processor -- start 2023-03-03T20:23:04.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.510+0000 7f024383e700 1 -- start start 2023-03-03T20:23:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f024383e700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f023c13a0f0 0x7f023c0c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f024383e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f023c1586c0 0x7f023c06d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:04.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f024383e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f023c0c1da0 0x7f023c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f024383e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f023c154840 con 0x7f023c1586c0 2023-03-03T20:23:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f024383e700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f023c154560 con 0x7f023c13a0f0 2023-03-03T20:23:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f024383e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023c157be0 con 0x7f023c0c1da0 2023-03-03T20:23:04.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f0240dd9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f023c1586c0 0x7f023c06d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f0240dd9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f023c1586c0 0x7f023c06d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45598/0 (socket says 172.21.15.7:45598) 2023-03-03T20:23:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f0240dd9700 1 -- 172.21.15.7:0/3318939083 learned_addr learned my addr 172.21.15.7:0/3318939083 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:23:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f02415da700 1 --2- 172.21.15.7:0/3318939083 >> v2:172.21.15.143:3300/0 conn(0x7f023c13a0f0 0x7f023c0c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:04.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f0241ddb700 1 --2- 172.21.15.7:0/3318939083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f023c0c1da0 0x7f023c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:04.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f0240dd9700 1 -- 172.21.15.7:0/3318939083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f023c0c1da0 msgr2=0x7f023c06d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:23:04.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f0240dd9700 1 --2- 172.21.15.7:0/3318939083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f023c0c1da0 0x7f023c06d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f0240dd9700 1 -- 172.21.15.7:0/3318939083 >> v2:172.21.15.143:3300/0 conn(0x7f023c13a0f0 msgr2=0x7f023c0c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:23:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f0240dd9700 1 --2- 172.21.15.7:0/3318939083 >> v2:172.21.15.143:3300/0 conn(0x7f023c13a0f0 0x7f023c0c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f0240dd9700 1 -- 172.21.15.7:0/3318939083 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c157ec0 con 0x7f023c1586c0 2023-03-03T20:23:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.511+0000 7f0240dd9700 1 --2- 172.21.15.7:0/3318939083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 0x7f023c06d240 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 crypto rx=0x7f023c06cf70 tx=0x7f0238009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aceb19c04a38ed6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:23:04.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f0233fff700 1 -- 172.21.15.7:0/3318939083 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023800b870 con 0x7f023c1586c0 2023-03-03T20:23:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f0233fff700 1 -- 172.21.15.7:0/3318939083 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023800b9d0 con 0x7f023c1586c0 2023-03-03T20:23:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f0233fff700 1 -- 172.21.15.7:0/3318939083 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023800bd60 con 0x7f023c1586c0 2023-03-03T20:23:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f024383e700 1 -- 172.21.15.7:0/3318939083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 msgr2=0x7f023c06d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f024383e700 1 --2- 172.21.15.7:0/3318939083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 0x7f023c06d240 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 crypto rx=0x7f023c06cf70 tx=0x7f0238009bd0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f024383e700 1 -- 172.21.15.7:0/3318939083 shutdown_connections 2023-03-03T20:23:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f024383e700 1 --2- 172.21.15.7:0/3318939083 >> v2:172.21.15.143:3300/0 conn(0x7f023c13a0f0 0x7f023c0c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f024383e700 1 --2- 172.21.15.7:0/3318939083 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f023c0c1da0 0x7f023c06d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f024383e700 1 --2- 172.21.15.7:0/3318939083 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 0x7f023c06d240 unknown :-1 s=CLOSED pgs=3775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f024383e700 1 -- 172.21.15.7:0/3318939083 >> 172.21.15.7:0/3318939083 conn(0x7f023c06c110 msgr2=0x7f023c06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:23:04.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f024383e700 1 -- 172.21.15.7:0/3318939083 shutdown_connections 2023-03-03T20:23:04.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.512+0000 7f024383e700 1 -- 172.21.15.7:0/3318939083 wait complete. 2023-03-03T20:23:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f024383e700 1 Processor -- start 2023-03-03T20:23:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f024383e700 1 -- start start 2023-03-03T20:23:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f024383e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c0c1da0 0x7f023c071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:04.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f024383e700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f023c13a0f0 0x7f023c155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f024383e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 0x7f023c155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f024383e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c1ed360 con 0x7f023c1586c0 2023-03-03T20:23:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f024383e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c1ee1c0 con 0x7f023c0c1da0 2023-03-03T20:23:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f024383e700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023c13dad0 con 0x7f023c13a0f0 2023-03-03T20:23:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f0241ddb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 0x7f023c155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f0241ddb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 0x7f023c155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45608/0 (socket says 172.21.15.7:45608) 2023-03-03T20:23:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f0241ddb700 1 -- 172.21.15.7:0/3001494187 learned_addr learned my addr 172.21.15.7:0/3001494187 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:23:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f0240dd9700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f023c13a0f0 0x7f023c155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.514+0000 7f02415da700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c0c1da0 0x7f023c071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f0241ddb700 1 -- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f023c13a0f0 msgr2=0x7f023c155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f0241ddb700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f023c13a0f0 0x7f023c155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f0241ddb700 1 -- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c0c1da0 msgr2=0x7f023c071150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f0241ddb700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c0c1da0 0x7f023c071150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f0241ddb700 1 -- 172.21.15.7:0/3001494187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0238009550 con 0x7f023c1586c0 2023-03-03T20:23:04.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f0241ddb700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 0x7f023c155ca0 secure :-1 s=READY pgs=3776 cs=0 l=1 rev1=1 crypto rx=0x7f0234007e70 tx=0x7f023400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:23:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f0231ffb700 1 -- 172.21.15.7:0/3001494187 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023400ed30 con 0x7f023c1586c0 2023-03-03T20:23:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f0231ffb700 1 -- 172.21.15.7:0/3001494187 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023400eeb0 con 0x7f023c1586c0 2023-03-03T20:23:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f024383e700 1 -- 172.21.15.7:0/3001494187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023c06e1a0 con 0x7f023c1586c0 2023-03-03T20:23:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f0231ffb700 1 -- 172.21.15.7:0/3001494187 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0234014640 con 0x7f023c1586c0 2023-03-03T20:23:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.515+0000 7f024383e700 1 -- 172.21.15.7:0/3001494187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023c1581a0 con 0x7f023c1586c0 2023-03-03T20:23:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.516+0000 7f024383e700 1 -- 172.21.15.7:0/3001494187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023c06e330 con 0x7f023c1586c0 2023-03-03T20:23:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.517+0000 7f0231ffb700 1 -- 172.21.15.7:0/3001494187 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f0234012050 con 0x7f023c1586c0 2023-03-03T20:23:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.517+0000 7f0231ffb700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02200682f0 0x7f022006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.517+0000 7f0231ffb700 1 -- 172.21.15.7:0/3001494187 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f0234086710 con 0x7f023c1586c0 2023-03-03T20:23:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.517+0000 7f02415da700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02200682f0 0x7f022006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.518+0000 7f02415da700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02200682f0 0x7f022006a7a0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f022c006b20 tx=0x7f022c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:23:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.522+0000 7f0231ffb700 1 -- 172.21.15.7:0/3001494187 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f0234052ab0 con 0x7f023c1586c0 2023-03-03T20:23:04.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.759+0000 7f024383e700 1 -- 172.21.15.7:0/3001494187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f023c074bb0 con 0x7f023c1586c0 2023-03-03T20:23:04.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.760+0000 7f0231ffb700 1 -- 172.21.15.7:0/3001494187 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f02340560d0 con 0x7f023c1586c0 2023-03-03T20:23:04.761 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:23:04.761 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:23:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 -- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02200682f0 msgr2=0x7f022006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02200682f0 0x7f022006a7a0 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f022c006b20 tx=0x7f022c008040 comp rx=0 tx=0).stop 2023-03-03T20:23:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 -- 172.21.15.7:0/3001494187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 msgr2=0x7f023c155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 0x7f023c155ca0 secure :-1 s=READY pgs=3776 cs=0 l=1 rev1=1 crypto rx=0x7f0234007e70 tx=0x7f023400e3c0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 -- 172.21.15.7:0/3001494187 shutdown_connections 2023-03-03T20:23:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f02200682f0 0x7f022006a7a0 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f023c1586c0 0x7f023c155ca0 unknown :-1 s=CLOSED pgs=3776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f023c0c1da0 0x7f023c071150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 --2- 172.21.15.7:0/3001494187 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f023c13a0f0 0x7f023c155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 -- 172.21.15.7:0/3001494187 >> 172.21.15.7:0/3001494187 conn(0x7f023c06c110 msgr2=0x7f023c137570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:23:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 -- 172.21.15.7:0/3001494187 shutdown_connections 2023-03-03T20:23:04.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:04.762+0000 7f024383e700 1 -- 172.21.15.7:0/3001494187 wait complete. 2023-03-03T20:23:04.765 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:23:04.775 INFO:tasks.mds_thrash.fs.[b]:waiting for 40 secs before thrashing 2023-03-03T20:23:29.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:23:29.596 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:23:38.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.984+0000 7f1af8734700 1 Processor -- start 2023-03-03T20:23:38.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.984+0000 7f1af8734700 1 -- start start 2023-03-03T20:23:38.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.985+0000 7f1af8734700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af014f9d0 0x7f1af0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:38.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.985+0000 7f1af8734700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f1af014fda0 0x7f1af0071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:38.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.985+0000 7f1af8734700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1af00abe20 0x7f1af01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.985+0000 7f1af8734700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af00c2f30 con 0x7f1af00abe20 2023-03-03T20:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.985+0000 7f1af8734700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af00c31d0 con 0x7f1af014fda0 2023-03-03T20:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.985+0000 7f1af8734700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af01524a0 con 0x7f1af014f9d0 2023-03-03T20:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.985+0000 7f1af6cd1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1af00abe20 0x7f1af01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.985+0000 7f1af6cd1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1af00abe20 0x7f1af01529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45638/0 (socket says 172.21.15.7:45638) 2023-03-03T20:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.985+0000 7f1af6cd1700 1 -- 172.21.15.7:0/4233610964 learned_addr learned my addr 172.21.15.7:0/4233610964 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:23:38.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.985+0000 7f1af5ccf700 1 --2- 172.21.15.7:0/4233610964 >> v2:172.21.15.143:3300/0 conn(0x7f1af014fda0 0x7f1af0071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:38.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af64d0700 1 --2- 172.21.15.7:0/4233610964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af014f9d0 0x7f1af0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:38.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af6cd1700 1 -- 172.21.15.7:0/4233610964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af014f9d0 msgr2=0x7f1af0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:23:38.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af6cd1700 1 --2- 172.21.15.7:0/4233610964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af014f9d0 0x7f1af0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:38.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af6cd1700 1 -- 172.21.15.7:0/4233610964 >> v2:172.21.15.143:3300/0 conn(0x7f1af014fda0 msgr2=0x7f1af0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:23:38.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af6cd1700 1 --2- 172.21.15.7:0/4233610964 >> v2:172.21.15.143:3300/0 conn(0x7f1af014fda0 0x7f1af0071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:38.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af6cd1700 1 -- 172.21.15.7:0/4233610964 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af00c34b0 con 0x7f1af00abe20 2023-03-03T20:23:38.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af64d0700 1 --2- 172.21.15.7:0/4233610964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af014f9d0 0x7f1af0156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:23:38.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af6cd1700 1 --2- 172.21.15.7:0/4233610964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af00abe20 0x7f1af01529c0 secure :-1 s=READY pgs=3777 cs=0 l=1 rev1=1 crypto rx=0x7f1af00ab220 tx=0x7f1ae800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350f7a2d41ff5361 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:23:38.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af54ce700 1 -- 172.21.15.7:0/4233610964 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae800e070 con 0x7f1af00abe20 2023-03-03T20:23:38.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af54ce700 1 -- 172.21.15.7:0/4233610964 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae800b9c0 con 0x7f1af00abe20 2023-03-03T20:23:38.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af54ce700 1 -- 172.21.15.7:0/4233610964 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae800bd70 con 0x7f1af00abe20 2023-03-03T20:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af8734700 1 -- 172.21.15.7:0/4233610964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af00abe20 msgr2=0x7f1af01529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.986+0000 7f1af8734700 1 --2- 172.21.15.7:0/4233610964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af00abe20 0x7f1af01529c0 secure :-1 s=READY pgs=3777 cs=0 l=1 rev1=1 crypto rx=0x7f1af00ab220 tx=0x7f1ae800b330 comp rx=0 tx=0).stop 2023-03-03T20:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.987+0000 7f1af8734700 1 -- 172.21.15.7:0/4233610964 shutdown_connections 2023-03-03T20:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.987+0000 7f1af8734700 1 --2- 172.21.15.7:0/4233610964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af00abe20 0x7f1af01529c0 unknown :-1 s=CLOSED pgs=3777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.987+0000 7f1af8734700 1 --2- 172.21.15.7:0/4233610964 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af014f9d0 0x7f1af0156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.987+0000 7f1af8734700 1 --2- 172.21.15.7:0/4233610964 >> v2:172.21.15.143:3300/0 conn(0x7f1af014fda0 0x7f1af0071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:38.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.987+0000 7f1af8734700 1 -- 172.21.15.7:0/4233610964 >> 172.21.15.7:0/4233610964 conn(0x7f1af00c8900 msgr2=0x7f1af00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.987+0000 7f1af8734700 1 -- 172.21.15.7:0/4233610964 shutdown_connections 2023-03-03T20:23:38.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.987+0000 7f1af8734700 1 -- 172.21.15.7:0/4233610964 wait complete. 2023-03-03T20:23:38.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.988+0000 7f1af8734700 1 Processor -- start 2023-03-03T20:23:38.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.988+0000 7f1af8734700 1 -- start start 2023-03-03T20:23:38.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af8734700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af00abe20 0x7f1af01e7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:38.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af8734700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1af014f9d0 0x7f1af014b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:38.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af8734700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af014fda0 0x7f1af0149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:38.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af8734700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af0144f50 con 0x7f1af014fda0 2023-03-03T20:23:38.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af8734700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af014dc50 con 0x7f1af014f9d0 2023-03-03T20:23:38.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af8734700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af01e7770 con 0x7f1af00abe20 2023-03-03T20:23:38.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af6cd1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af014fda0 0x7f1af0149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:38.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af6cd1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af014fda0 0x7f1af0149180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45644/0 (socket says 172.21.15.7:45644) 2023-03-03T20:23:38.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af6cd1700 1 -- 172.21.15.7:0/2241704067 learned_addr learned my addr 172.21.15.7:0/2241704067 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:23:38.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af5ccf700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1af014f9d0 0x7f1af014b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:38.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af64d0700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af00abe20 0x7f1af01e7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:38.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af6cd1700 1 -- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af00abe20 msgr2=0x7f1af01e7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:38.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af6cd1700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af00abe20 0x7f1af01e7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:38.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af6cd1700 1 -- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1af014f9d0 msgr2=0x7f1af014b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:38.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af6cd1700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1af014f9d0 0x7f1af014b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:38.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af6cd1700 1 -- 172.21.15.7:0/2241704067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae8009b30 con 0x7f1af014fda0 2023-03-03T20:23:38.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af5ccf700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1af014f9d0 0x7f1af014b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:23:38.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af64d0700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af00abe20 0x7f1af01e7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:23:38.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af6cd1700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af014fda0 0x7f1af0149180 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 crypto rx=0x7f1af0072520 tx=0x7f1ae8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:23:38.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1ae6ffd700 1 -- 172.21.15.7:0/2241704067 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8004250 con 0x7f1af014fda0 2023-03-03T20:23:38.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1ae6ffd700 1 -- 172.21.15.7:0/2241704067 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae80043d0 con 0x7f1af014fda0 2023-03-03T20:23:38.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1ae6ffd700 1 -- 172.21.15.7:0/2241704067 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8004780 con 0x7f1af014fda0 2023-03-03T20:23:38.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af8734700 1 -- 172.21.15.7:0/2241704067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af00ac350 con 0x7f1af014fda0 2023-03-03T20:23:38.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.989+0000 7f1af8734700 1 -- 172.21.15.7:0/2241704067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af00ab630 con 0x7f1af014fda0 2023-03-03T20:23:38.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.990+0000 7f1af8734700 1 -- 172.21.15.7:0/2241704067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af00c3e10 con 0x7f1af014fda0 2023-03-03T20:23:38.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.991+0000 7f1ae6ffd700 1 -- 172.21.15.7:0/2241704067 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f1ae802b070 con 0x7f1af014fda0 2023-03-03T20:23:38.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.991+0000 7f1ae6ffd700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ad40682f0 0x7f1ad406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:39.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.991+0000 7f1ae6ffd700 1 -- 172.21.15.7:0/2241704067 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f1ae80872e0 con 0x7f1af014fda0 2023-03-03T20:23:39.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.991+0000 7f1af64d0700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ad40682f0 0x7f1ad406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:39.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.992+0000 7f1af64d0700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ad40682f0 0x7f1ad406a7a0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00089b0 tx=0x7f1ae0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:23:39.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:38.996+0000 7f1ae6ffd700 1 -- 172.21.15.7:0/2241704067 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f1ae8053590 con 0x7f1af014fda0 2023-03-03T20:23:39.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.230+0000 7f1af8734700 1 -- 172.21.15.7:0/2241704067 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1af00ad0c0 con 0x7f1af014fda0 2023-03-03T20:23:39.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.231+0000 7f1ae6ffd700 1 -- 172.21.15.7:0/2241704067 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f1ae8029090 con 0x7f1af014fda0 2023-03-03T20:23:39.232 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:23:39.232 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:23:39.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 -- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ad40682f0 msgr2=0x7f1ad406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:39.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ad40682f0 0x7f1ad406a7a0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00089b0 tx=0x7f1ae0008040 comp rx=0 tx=0).stop 2023-03-03T20:23:39.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 -- 172.21.15.7:0/2241704067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af014fda0 msgr2=0x7f1af0149180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:39.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af014fda0 0x7f1af0149180 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 crypto rx=0x7f1af0072520 tx=0x7f1ae8004ad0 comp rx=0 tx=0).stop 2023-03-03T20:23:39.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 -- 172.21.15.7:0/2241704067 shutdown_connections 2023-03-03T20:23:39.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f1ad40682f0 0x7f1ad406a7a0 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:39.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1af014fda0 0x7f1af0149180 unknown :-1 s=CLOSED pgs=3778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:39.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f1af00abe20 0x7f1af01e7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:39.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 --2- 172.21.15.7:0/2241704067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1af014f9d0 0x7f1af014b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:39.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 -- 172.21.15.7:0/2241704067 >> 172.21.15.7:0/2241704067 conn(0x7f1af00c8900 msgr2=0x7f1af00c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:23:39.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 -- 172.21.15.7:0/2241704067 shutdown_connections 2023-03-03T20:23:39.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:39.233+0000 7f1af8734700 1 -- 172.21.15.7:0/2241704067 wait complete. 2023-03-03T20:23:39.235 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:23:39.246 INFO:tasks.mds_thrash.fs.[a]:waiting for 107 secs before thrashing 2023-03-03T20:23:44.777 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:23:44.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.915+0000 7f09e4333700 1 Processor -- start 2023-03-03T20:23:44.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.915+0000 7f09e4333700 1 -- start start 2023-03-03T20:23:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.916+0000 7f09e4333700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc151ab0 0x7f09dc0736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.916+0000 7f09e4333700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f09dc073c30 0x7f09dc06bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.916+0000 7f09e4333700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09dc06d780 con 0x7f09dc06c3e0 2023-03-03T20:23:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.916+0000 7f09e4333700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09dc06cb70 con 0x7f09dc073c30 2023-03-03T20:23:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.916+0000 7f09e4333700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc06ce50 con 0x7f09dc151ab0 2023-03-03T20:23:44.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.916+0000 7f09e28d0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f09dc06c3e0 0x7f09dc157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39636/0 (socket says 172.21.15.7:39636) 2023-03-03T20:23:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.916+0000 7f09e28d0700 1 -- 172.21.15.7:0/912482044 learned_addr learned my addr 172.21.15.7:0/912482044 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:23:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.916+0000 7f09e18ce700 1 --2- 172.21.15.7:0/912482044 >> v2:172.21.15.143:3300/0 conn(0x7f09dc073c30 0x7f09dc06bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.916+0000 7f09e20cf700 1 --2- 172.21.15.7:0/912482044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc151ab0 0x7f09dc0736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e10cd700 1 -- 172.21.15.7:0/912482044 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964862703 0 0) 0x7f09dc06d780 con 0x7f09dc06c3e0 2023-03-03T20:23:44.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e10cd700 1 -- 172.21.15.7:0/912482044 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c80035b0 con 0x7f09dc06c3e0 2023-03-03T20:23:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e20cf700 1 -- 172.21.15.7:0/912482044 >> v2:172.21.15.143:3300/0 conn(0x7f09dc073c30 msgr2=0x7f09dc06bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:23:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e20cf700 1 --2- 172.21.15.7:0/912482044 >> v2:172.21.15.143:3300/0 conn(0x7f09dc073c30 0x7f09dc06bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e20cf700 1 -- 172.21.15.7:0/912482044 >> v1:172.21.15.7:6789/0 conn(0x7f09dc06c3e0 legacy=0x7f09dc157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e20cf700 1 -- 172.21.15.7:0/912482044 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc06d130 con 0x7f09dc151ab0 2023-03-03T20:23:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e18ce700 1 --2- 172.21.15.7:0/912482044 >> v2:172.21.15.143:3300/0 conn(0x7f09dc073c30 0x7f09dc06bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:23:44.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e20cf700 1 --2- 172.21.15.7:0/912482044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc151ab0 0x7f09dc0736f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009840 tx=0x7f09cc009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdecbf608c61cba4 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:23:44.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e10cd700 1 -- 172.21.15.7:0/912482044 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc011830 con 0x7f09dc151ab0 2023-03-03T20:23:44.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e10cd700 1 -- 172.21.15.7:0/912482044 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc011990 con 0x7f09dc151ab0 2023-03-03T20:23:44.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.917+0000 7f09e10cd700 1 -- 172.21.15.7:0/912482044 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc011d20 con 0x7f09dc151ab0 2023-03-03T20:23:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.918+0000 7f09e4333700 1 -- 172.21.15.7:0/912482044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc151ab0 msgr2=0x7f09dc0736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.918+0000 7f09e4333700 1 --2- 172.21.15.7:0/912482044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc151ab0 0x7f09dc0736f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009840 tx=0x7f09cc009b50 comp rx=0 tx=0).stop 2023-03-03T20:23:44.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.918+0000 7f09e4333700 1 -- 172.21.15.7:0/912482044 shutdown_connections 2023-03-03T20:23:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.918+0000 7f09e4333700 1 --2- 172.21.15.7:0/912482044 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc151ab0 0x7f09dc0736f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.918+0000 7f09e4333700 1 --2- 172.21.15.7:0/912482044 >> v2:172.21.15.143:3300/0 conn(0x7f09dc073c30 0x7f09dc06bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.918+0000 7f09e4333700 1 -- 172.21.15.7:0/912482044 >> 172.21.15.7:0/912482044 conn(0x7f09dc0a94d0 msgr2=0x7f09dc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:23:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.918+0000 7f09e4333700 1 -- 172.21.15.7:0/912482044 shutdown_connections 2023-03-03T20:23:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.918+0000 7f09e4333700 1 -- 172.21.15.7:0/912482044 wait complete. 2023-03-03T20:23:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.919+0000 7f09e4333700 1 Processor -- start 2023-03-03T20:23:44.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.919+0000 7f09e4333700 1 -- start start 2023-03-03T20:23:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e4333700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc06c3e0 0x7f09dc1ed620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e4333700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc073c30 0x7f09dc1e0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e4333700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc151ab0 0x7f09dc0aa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e4333700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc14c750 con 0x7f09dc06c3e0 2023-03-03T20:23:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e4333700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc0c6b10 con 0x7f09dc151ab0 2023-03-03T20:23:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e4333700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09dc0abfa0 con 0x7f09dc073c30 2023-03-03T20:23:44.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e20cf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc06c3e0 0x7f09dc1ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e20cf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc06c3e0 0x7f09dc1ed620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45658/0 (socket says 172.21.15.7:45658) 2023-03-03T20:23:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e20cf700 1 -- 172.21.15.7:0/3097567073 learned_addr learned my addr 172.21.15.7:0/3097567073 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:23:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e18ce700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc073c30 0x7f09dc1e0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e28d0700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc151ab0 0x7f09dc0aa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e20cf700 1 -- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc073c30 msgr2=0x7f09dc1e0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e20cf700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc073c30 0x7f09dc1e0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e20cf700 1 -- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc151ab0 msgr2=0x7f09dc0aa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:44.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e20cf700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc151ab0 0x7f09dc0aa790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.920+0000 7f09e20cf700 1 -- 172.21.15.7:0/3097567073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d80094a0 con 0x7f09dc06c3e0 2023-03-03T20:23:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.921+0000 7f09e20cf700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc06c3e0 0x7f09dc1ed620 secure :-1 s=READY pgs=3780 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0090b0 tx=0x7f09cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:23:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.921+0000 7f09d2ffd700 1 -- 172.21.15.7:0/3097567073 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc004590 con 0x7f09dc06c3e0 2023-03-03T20:23:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.921+0000 7f09d2ffd700 1 -- 172.21.15.7:0/3097567073 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc01e070 con 0x7f09dc06c3e0 2023-03-03T20:23:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.921+0000 7f09e4333700 1 -- 172.21.15.7:0/3097567073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09cc0094a0 con 0x7f09dc06c3e0 2023-03-03T20:23:44.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.921+0000 7f09d2ffd700 1 -- 172.21.15.7:0/3097567073 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc0038e0 con 0x7f09dc06c3e0 2023-03-03T20:23:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.921+0000 7f09e4333700 1 -- 172.21.15.7:0/3097567073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09dc151e80 con 0x7f09dc06c3e0 2023-03-03T20:23:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.922+0000 7f09e4333700 1 -- 172.21.15.7:0/3097567073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09dc06d410 con 0x7f09dc06c3e0 2023-03-03T20:23:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.922+0000 7f09d2ffd700 1 -- 172.21.15.7:0/3097567073 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f09cc00f050 con 0x7f09dc06c3e0 2023-03-03T20:23:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.923+0000 7f09d2ffd700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09c00682f0 0x7f09c006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:23:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.923+0000 7f09d2ffd700 1 -- 172.21.15.7:0/3097567073 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f09cc086f20 con 0x7f09dc06c3e0 2023-03-03T20:23:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.923+0000 7f09e18ce700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09c00682f0 0x7f09c006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:23:44.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.924+0000 7f09e18ce700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09c00682f0 0x7f09c006a7a0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0c1680 tx=0x7f09d8009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:23:44.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:44.927+0000 7f09d2ffd700 1 -- 172.21.15.7:0/3097567073 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f09cc0b85e0 con 0x7f09dc06c3e0 2023-03-03T20:23:45.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.163+0000 7f09e4333700 1 -- 172.21.15.7:0/3097567073 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09dc152010 con 0x7f09dc06c3e0 2023-03-03T20:23:45.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.164+0000 7f09d2ffd700 1 -- 172.21.15.7:0/3097567073 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f09cc015090 con 0x7f09dc06c3e0 2023-03-03T20:23:45.165 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:23:45.165 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:23:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 -- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09c00682f0 msgr2=0x7f09c006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09c00682f0 0x7f09c006a7a0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0c1680 tx=0x7f09d8009210 comp rx=0 tx=0).stop 2023-03-03T20:23:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 -- 172.21.15.7:0/3097567073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc06c3e0 msgr2=0x7f09dc1ed620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:23:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc06c3e0 0x7f09dc1ed620 secure :-1 s=READY pgs=3780 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0090b0 tx=0x7f09cc004750 comp rx=0 tx=0).stop 2023-03-03T20:23:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 -- 172.21.15.7:0/3097567073 shutdown_connections 2023-03-03T20:23:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f09dc151ab0 0x7f09dc0aa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:45.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f09c00682f0 0x7f09c006a7a0 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:45.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc06c3e0 0x7f09dc1ed620 unknown :-1 s=CLOSED pgs=3780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:45.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 --2- 172.21.15.7:0/3097567073 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f09dc073c30 0x7f09dc1e0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:23:45.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 -- 172.21.15.7:0/3097567073 >> 172.21.15.7:0/3097567073 conn(0x7f09dc0a94d0 msgr2=0x7f09dc0a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:23:45.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 -- 172.21.15.7:0/3097567073 shutdown_connections 2023-03-03T20:23:45.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:23:45.166+0000 7f09e4333700 1 -- 172.21.15.7:0/3097567073 wait complete. 2023-03-03T20:23:45.169 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:23:45.179 INFO:tasks.mds_thrash.fs.[b]:waiting for 84 secs before thrashing 2023-03-03T20:23:59.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:23:59.654 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:24:29.719 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:24:29.724 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:24:59.799 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:24:59.803 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:25:09.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:25:09.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.318+0000 7f90a01e4700 1 Processor -- start 2023-03-03T20:25:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.319+0000 7f90a01e4700 1 -- start start 2023-03-03T20:25:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.320+0000 7f90a01e4700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f90980c6010 0x7f90980c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:25:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.320+0000 7f90a01e4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909814ebe0 con 0x7f90980c17f0 2023-03-03T20:25:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.320+0000 7f90a01e4700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90980c2e70 con 0x7f90980c4fc0 2023-03-03T20:25:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.320+0000 7f90a01e4700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9098152100 con 0x7f90980c6010 2023-03-03T20:25:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.320+0000 7f909d77f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f90980c17f0 0x7f909806d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39674/0 (socket says 172.21.15.7:39674) 2023-03-03T20:25:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.320+0000 7f909d77f700 1 -- 172.21.15.7:0/154418698 learned_addr learned my addr 172.21.15.7:0/154418698 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:25:09.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.320+0000 7f909df80700 1 --2- 172.21.15.7:0/154418698 >> v2:172.21.15.143:3300/0 conn(0x7f90980c6010 0x7f90980c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:25:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.320+0000 7f909cf7e700 1 -- 172.21.15.7:0/154418698 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511716517 0 0) 0x7f909814ebe0 con 0x7f90980c17f0 2023-03-03T20:25:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.320+0000 7f909cf7e700 1 -- 172.21.15.7:0/154418698 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90840035b0 con 0x7f90980c17f0 2023-03-03T20:25:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.321+0000 7f909cf7e700 1 -- 172.21.15.7:0/154418698 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3764683826 0 0) 0x7f90840035b0 con 0x7f90980c17f0 2023-03-03T20:25:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.321+0000 7f909df80700 1 -- 172.21.15.7:0/154418698 >> v1:172.21.15.143:6789/0 conn(0x7f90980c4fc0 legacy=0x7f909806ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:09.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.321+0000 7f909df80700 1 -- 172.21.15.7:0/154418698 >> v1:172.21.15.7:6789/0 conn(0x7f90980c17f0 legacy=0x7f909806d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.321+0000 7f909df80700 1 -- 172.21.15.7:0/154418698 --> v2:172.21.15.143:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90980aa680 con 0x7f90980c6010 2023-03-03T20:25:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.321+0000 7f909df80700 1 --2- 172.21.15.7:0/154418698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90980c6010 0x7f90980c4a80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f9094009840 tx=0x7f9094009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a020c2110d0580 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:25:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.321+0000 7f909cf7e700 1 -- 172.21.15.7:0/154418698 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9088001670 con 0x7f90980c6010 2023-03-03T20:25:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.321+0000 7f909cf7e700 1 -- 172.21.15.7:0/154418698 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9094004030 con 0x7f90980c6010 2023-03-03T20:25:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.321+0000 7f909cf7e700 1 -- 172.21.15.7:0/154418698 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90940043c0 con 0x7f90980c6010 2023-03-03T20:25:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.321+0000 7f90a01e4700 1 -- 172.21.15.7:0/154418698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90980c6010 msgr2=0x7f90980c4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.322+0000 7f90a01e4700 1 --2- 172.21.15.7:0/154418698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90980c6010 0x7f90980c4a80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f9094009840 tx=0x7f9094009b50 comp rx=0 tx=0).stop 2023-03-03T20:25:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.322+0000 7f90a01e4700 1 -- 172.21.15.7:0/154418698 shutdown_connections 2023-03-03T20:25:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.322+0000 7f90a01e4700 1 --2- 172.21.15.7:0/154418698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90980c6010 0x7f90980c4a80 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.322+0000 7f90a01e4700 1 -- 172.21.15.7:0/154418698 >> 172.21.15.7:0/154418698 conn(0x7f90980a9430 msgr2=0x7f90980c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:25:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.322+0000 7f90a01e4700 1 -- 172.21.15.7:0/154418698 shutdown_connections 2023-03-03T20:25:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.322+0000 7f90a01e4700 1 -- 172.21.15.7:0/154418698 wait complete. 2023-03-03T20:25:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.323+0000 7f90a01e4700 1 Processor -- start 2023-03-03T20:25:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.323+0000 7f90a01e4700 1 -- start start 2023-03-03T20:25:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.324+0000 7f90a01e4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90980c17f0 0x7f90981eb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:25:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.324+0000 7f90a01e4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90980c4fc0 0x7f90980c0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:25:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.324+0000 7f90a01e4700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90980c6010 0x7f90981dbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:25:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.324+0000 7f90a01e4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90981e8530 con 0x7f90980c4fc0 2023-03-03T20:25:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.324+0000 7f90a01e4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90981ebd50 con 0x7f90980c17f0 2023-03-03T20:25:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.324+0000 7f90a01e4700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90981e7920 con 0x7f90980c6010 2023-03-03T20:25:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.324+0000 7f909d77f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90980c4fc0 0x7f90980c0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:25:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.324+0000 7f909d77f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90980c4fc0 0x7f90980c0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45700/0 (socket says 172.21.15.7:45700) 2023-03-03T20:25:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.324+0000 7f909df80700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90980c17f0 0x7f90981eb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:25:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.324+0000 7f909d77f700 1 -- 172.21.15.7:0/498985759 learned_addr learned my addr 172.21.15.7:0/498985759 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:25:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f909e781700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90980c6010 0x7f90981dbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:25:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f909d77f700 1 -- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90980c6010 msgr2=0x7f90981dbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f909d77f700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90980c6010 0x7f90981dbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f909d77f700 1 -- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90980c17f0 msgr2=0x7f90981eb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f909d77f700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90980c17f0 0x7f90981eb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f909d77f700 1 -- 172.21.15.7:0/498985759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90940094a0 con 0x7f90980c4fc0 2023-03-03T20:25:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f909d77f700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90980c4fc0 0x7f90980c0140 secure :-1 s=READY pgs=3782 cs=0 l=1 rev1=1 crypto rx=0x7f90880011e0 tx=0x7f908800ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:25:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f908e7fc700 1 -- 172.21.15.7:0/498985759 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9088004c80 con 0x7f90980c4fc0 2023-03-03T20:25:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f908e7fc700 1 -- 172.21.15.7:0/498985759 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9088004de0 con 0x7f90980c4fc0 2023-03-03T20:25:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f90a01e4700 1 -- 172.21.15.7:0/498985759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90980c2b00 con 0x7f90980c4fc0 2023-03-03T20:25:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f908e7fc700 1 -- 172.21.15.7:0/498985759 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90880072d0 con 0x7f90980c4fc0 2023-03-03T20:25:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.325+0000 7f90a01e4700 1 -- 172.21.15.7:0/498985759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90980aa960 con 0x7f90980c4fc0 2023-03-03T20:25:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.326+0000 7f90a01e4700 1 -- 172.21.15.7:0/498985759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90980c3300 con 0x7f90980c4fc0 2023-03-03T20:25:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.326+0000 7f908e7fc700 1 -- 172.21.15.7:0/498985759 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f908801a020 con 0x7f90980c4fc0 2023-03-03T20:25:09.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.327+0000 7f908e7fc700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f907c0682f0 0x7f907c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:25:09.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.327+0000 7f908e7fc700 1 -- 172.21.15.7:0/498985759 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f9088085e70 con 0x7f90980c4fc0 2023-03-03T20:25:09.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.327+0000 7f909df80700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f907c0682f0 0x7f907c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:25:09.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.328+0000 7f909df80700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f907c0682f0 0x7f907c06a7a0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f90940090b0 tx=0x7f9094008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:25:09.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.331+0000 7f908e7fc700 1 -- 172.21.15.7:0/498985759 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f9088052210 con 0x7f90980c4fc0 2023-03-03T20:25:09.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.569+0000 7f90a01e4700 1 -- 172.21.15.7:0/498985759 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90980c4980 con 0x7f90980c4fc0 2023-03-03T20:25:09.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.569+0000 7f908e7fc700 1 -- 172.21.15.7:0/498985759 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f9088055830 con 0x7f90980c4fc0 2023-03-03T20:25:09.570 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:25:09.571 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:25:09.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.571+0000 7f90a01e4700 1 -- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f907c0682f0 msgr2=0x7f907c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:09.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.571+0000 7f90a01e4700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f907c0682f0 0x7f907c06a7a0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f90940090b0 tx=0x7f9094008030 comp rx=0 tx=0).stop 2023-03-03T20:25:09.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.571+0000 7f90a01e4700 1 -- 172.21.15.7:0/498985759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90980c4fc0 msgr2=0x7f90980c0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:09.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.571+0000 7f90a01e4700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90980c4fc0 0x7f90980c0140 secure :-1 s=READY pgs=3782 cs=0 l=1 rev1=1 crypto rx=0x7f90880011e0 tx=0x7f908800ca50 comp rx=0 tx=0).stop 2023-03-03T20:25:09.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.571+0000 7f90a01e4700 1 -- 172.21.15.7:0/498985759 shutdown_connections 2023-03-03T20:25:09.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.571+0000 7f90a01e4700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f90980c6010 0x7f90981dbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:09.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.571+0000 7f90a01e4700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90980c17f0 0x7f90981eb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:09.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.571+0000 7f90a01e4700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f907c0682f0 0x7f907c06a7a0 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:09.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.571+0000 7f90a01e4700 1 --2- 172.21.15.7:0/498985759 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90980c4fc0 0x7f90980c0140 unknown :-1 s=CLOSED pgs=3782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:09.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.571+0000 7f90a01e4700 1 -- 172.21.15.7:0/498985759 >> 172.21.15.7:0/498985759 conn(0x7f90980a9430 msgr2=0x7f909806df50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:25:09.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.572+0000 7f90a01e4700 1 -- 172.21.15.7:0/498985759 shutdown_connections 2023-03-03T20:25:09.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:09.572+0000 7f90a01e4700 1 -- 172.21.15.7:0/498985759 wait complete. 2023-03-03T20:25:09.574 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:25:09.585 INFO:tasks.mds_thrash.fs.[b]:waiting for 64 secs before thrashing 2023-03-03T20:25:26.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:25:26.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.384+0000 7fd8a038b700 1 Processor -- start 2023-03-03T20:25:26.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.384+0000 7fd8a038b700 1 -- start start 2023-03-03T20:25:26.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.385+0000 7fd8a038b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 0x7fd8980736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:25:26.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.385+0000 7fd8a038b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89806cb70 con 0x7fd898073c30 2023-03-03T20:25:26.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.385+0000 7fd8a038b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89806ce50 con 0x7fd8980c2950 2023-03-03T20:25:26.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.385+0000 7fd8a038b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd89806c2f0 con 0x7fd898151b20 2023-03-03T20:25:26.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.385+0000 7fd89e928700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd898073c30 0x7fd8981581f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39698/0 (socket says 172.21.15.7:39698) 2023-03-03T20:25:26.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.385+0000 7fd89e928700 1 -- 172.21.15.7:0/3690922316 learned_addr learned my addr 172.21.15.7:0/3690922316 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:25:26.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.385+0000 7fd89e127700 1 --2- 172.21.15.7:0/3690922316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 0x7fd8980736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:25:26.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.385+0000 7fd89d125700 1 -- 172.21.15.7:0/3690922316 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217621692 0 0) 0x7fd89806cb70 con 0x7fd898073c30 2023-03-03T20:25:26.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.386+0000 7fd89d125700 1 -- 172.21.15.7:0/3690922316 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8840035b0 con 0x7fd898073c30 2023-03-03T20:25:26.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.386+0000 7fd89d125700 1 -- 172.21.15.7:0/3690922316 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355339552 0 0) 0x7fd89806ce50 con 0x7fd8980c2950 2023-03-03T20:25:26.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.386+0000 7fd89d125700 1 -- 172.21.15.7:0/3690922316 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd89806cb70 con 0x7fd8980c2950 2023-03-03T20:25:26.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.386+0000 7fd89e127700 1 -- 172.21.15.7:0/3690922316 >> v1:172.21.15.143:6789/0 conn(0x7fd8980c2950 legacy=0x7fd898157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:26.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.386+0000 7fd89e127700 1 -- 172.21.15.7:0/3690922316 >> v1:172.21.15.7:6789/0 conn(0x7fd898073c30 legacy=0x7fd8981581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:26.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.386+0000 7fd89e127700 1 -- 172.21.15.7:0/3690922316 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89806d130 con 0x7fd898151b20 2023-03-03T20:25:26.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.386+0000 7fd89e127700 1 --2- 172.21.15.7:0/3690922316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 0x7fd8980736f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd888009870 tx=0x7fd888009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bd7326ccb02c921 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:25:26.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.386+0000 7fd89d125700 1 -- 172.21.15.7:0/3690922316 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd888011830 con 0x7fd898151b20 2023-03-03T20:25:26.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.386+0000 7fd89d125700 1 -- 172.21.15.7:0/3690922316 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd888011990 con 0x7fd898151b20 2023-03-03T20:25:26.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.386+0000 7fd89d125700 1 -- 172.21.15.7:0/3690922316 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd888011d00 con 0x7fd898151b20 2023-03-03T20:25:26.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.387+0000 7fd8a038b700 1 -- 172.21.15.7:0/3690922316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 msgr2=0x7fd8980736f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:26.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.387+0000 7fd8a038b700 1 --2- 172.21.15.7:0/3690922316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 0x7fd8980736f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd888009870 tx=0x7fd888009b80 comp rx=0 tx=0).stop 2023-03-03T20:25:26.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.387+0000 7fd8a038b700 1 -- 172.21.15.7:0/3690922316 shutdown_connections 2023-03-03T20:25:26.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.387+0000 7fd8a038b700 1 --2- 172.21.15.7:0/3690922316 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 0x7fd8980736f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:26.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.387+0000 7fd8a038b700 1 -- 172.21.15.7:0/3690922316 >> 172.21.15.7:0/3690922316 conn(0x7fd8980a94d0 msgr2=0x7fd8980a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:25:26.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.387+0000 7fd8a038b700 1 -- 172.21.15.7:0/3690922316 shutdown_connections 2023-03-03T20:25:26.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.387+0000 7fd8a038b700 1 -- 172.21.15.7:0/3690922316 wait complete. 2023-03-03T20:25:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.388+0000 7fd8a038b700 1 Processor -- start 2023-03-03T20:25:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.388+0000 7fd8a038b700 1 -- start start 2023-03-03T20:25:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd8a038b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd898073c30 0x7fd8980c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:25:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd8a038b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8980c2950 0x7fd898152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:25:26.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd8a038b700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 0x7fd8980c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:25:26.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd8a038b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8980be430 con 0x7fd8980c2950 2023-03-03T20:25:26.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd8a038b700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8980be710 con 0x7fd898073c30 2023-03-03T20:25:26.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd8a038b700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8981ea840 con 0x7fd898151b20 2023-03-03T20:25:26.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89d926700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8980c2950 0x7fd898152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:25:26.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89d926700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8980c2950 0x7fd898152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45724/0 (socket says 172.21.15.7:45724) 2023-03-03T20:25:26.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89d926700 1 -- 172.21.15.7:0/1478184062 learned_addr learned my addr 172.21.15.7:0/1478184062 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:25:26.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89e127700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd898073c30 0x7fd8980c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:25:26.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89e928700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 0x7fd8980c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:25:26.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89d926700 1 -- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 msgr2=0x7fd8980c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:26.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89d926700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 0x7fd8980c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:26.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89d926700 1 -- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd898073c30 msgr2=0x7fd8980c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:26.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89d926700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd898073c30 0x7fd8980c0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:26.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89d926700 1 -- 172.21.15.7:0/1478184062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8880094a0 con 0x7fd8980c2950 2023-03-03T20:25:26.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.389+0000 7fd89d926700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8980c2950 0x7fd898152570 secure :-1 s=READY pgs=3784 cs=0 l=1 rev1=1 crypto rx=0x7fd89400c410 tx=0x7fd89400c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:25:26.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.390+0000 7fd88effd700 1 -- 172.21.15.7:0/1478184062 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8940052d0 con 0x7fd8980c2950 2023-03-03T20:25:26.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.390+0000 7fd88effd700 1 -- 172.21.15.7:0/1478184062 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd894005450 con 0x7fd8980c2950 2023-03-03T20:25:26.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.390+0000 7fd8a038b700 1 -- 172.21.15.7:0/1478184062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd898151ef0 con 0x7fd8980c2950 2023-03-03T20:25:26.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.390+0000 7fd88effd700 1 -- 172.21.15.7:0/1478184062 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8940038c0 con 0x7fd8980c2950 2023-03-03T20:25:26.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.390+0000 7fd8a038b700 1 -- 172.21.15.7:0/1478184062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89806d410 con 0x7fd8980c2950 2023-03-03T20:25:26.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.391+0000 7fd8a038b700 1 -- 172.21.15.7:0/1478184062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89806bea0 con 0x7fd8980c2950 2023-03-03T20:25:26.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.391+0000 7fd88effd700 1 -- 172.21.15.7:0/1478184062 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd894019030 con 0x7fd8980c2950 2023-03-03T20:25:26.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.391+0000 7fd88effd700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd87c0682f0 0x7fd87c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:25:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.392+0000 7fd88effd700 1 -- 172.21.15.7:0/1478184062 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd894085320 con 0x7fd8980c2950 2023-03-03T20:25:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.392+0000 7fd89e127700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd87c0682f0 0x7fd87c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:25:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.392+0000 7fd89e127700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd87c0682f0 0x7fd87c06a7a0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fd8880090b0 tx=0x7fd888008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:25:26.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.396+0000 7fd88effd700 1 -- 172.21.15.7:0/1478184062 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd8940516c0 con 0x7fd8980c2950 2023-03-03T20:25:26.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.633+0000 7fd8a038b700 1 -- 172.21.15.7:0/1478184062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8981ead30 con 0x7fd8980c2950 2023-03-03T20:25:26.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.634+0000 7fd88effd700 1 -- 172.21.15.7:0/1478184062 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd894054ce0 con 0x7fd8980c2950 2023-03-03T20:25:26.636 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:25:26.636 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:25:26.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 -- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd87c0682f0 msgr2=0x7fd87c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:26.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd87c0682f0 0x7fd87c06a7a0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fd8880090b0 tx=0x7fd888008030 comp rx=0 tx=0).stop 2023-03-03T20:25:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 -- 172.21.15.7:0/1478184062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8980c2950 msgr2=0x7fd898152570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:25:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8980c2950 0x7fd898152570 secure :-1 s=READY pgs=3784 cs=0 l=1 rev1=1 crypto rx=0x7fd89400c410 tx=0x7fd89400c720 comp rx=0 tx=0).stop 2023-03-03T20:25:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 -- 172.21.15.7:0/1478184062 shutdown_connections 2023-03-03T20:25:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd898151b20 0x7fd8980c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd898073c30 0x7fd8980c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd87c0682f0 0x7fd87c06a7a0 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 --2- 172.21.15.7:0/1478184062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8980c2950 0x7fd898152570 unknown :-1 s=CLOSED pgs=3784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:25:26.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 -- 172.21.15.7:0/1478184062 >> 172.21.15.7:0/1478184062 conn(0x7fd8980a94d0 msgr2=0x7fd8981580f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:25:26.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 -- 172.21.15.7:0/1478184062 shutdown_connections 2023-03-03T20:25:26.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:25:26.636+0000 7fd8a038b700 1 -- 172.21.15.7:0/1478184062 wait complete. 2023-03-03T20:25:26.642 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:25:26.650 INFO:tasks.mds_thrash.fs.[a]:waiting for 86 secs before thrashing 2023-03-03T20:25:29.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:25:29.881 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:25:59.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:25:59.945 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:26:13.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:26:13.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.737+0000 7fd9c20fa700 1 Processor -- start 2023-03-03T20:26:13.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.737+0000 7fd9c20fa700 1 -- start start 2023-03-03T20:26:13.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9c20fa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9bc151e80 0x7fd9bc0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:13.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9c20fa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd9bc0c6fe0 0x7fd9bc0bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:13.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9c20fa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9bc0c08d0 con 0x7fd9bc155410 2023-03-03T20:26:13.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9c20fa700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9bc152250 con 0x7fd9bc0c6fe0 2023-03-03T20:26:13.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9c20fa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0c0fb0 con 0x7fd9bc151e80 2023-03-03T20:26:13.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9bbfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd9bc0c6fe0 0x7fd9bc0bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:13.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9baffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd9bc155410 0x7fd9bc0bf8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39746/0 (socket says 172.21.15.7:39746) 2023-03-03T20:26:13.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9baffd700 1 -- 172.21.15.7:0/1786900133 learned_addr learned my addr 172.21.15.7:0/1786900133 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:26:13.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9bb7fe700 1 --2- 172.21.15.7:0/1786900133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9bc151e80 0x7fd9bc0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:13.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9bbfff700 1 -- 172.21.15.7:0/1786900133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9bc151e80 msgr2=0x7fd9bc0c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:26:13.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9bbfff700 1 --2- 172.21.15.7:0/1786900133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9bc151e80 0x7fd9bc0c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:13.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9bbfff700 1 -- 172.21.15.7:0/1786900133 >> v1:172.21.15.7:6789/0 conn(0x7fd9bc155410 legacy=0x7fd9bc0bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:13.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9ba7fc700 1 -- 172.21.15.7:0/1786900133 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962807838 0 0) 0x7fd9bc0c08d0 con 0x7fd9bc155410 2023-03-03T20:26:13.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9bbfff700 1 -- 172.21.15.7:0/1786900133 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc154930 con 0x7fd9bc0c6fe0 2023-03-03T20:26:13.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.738+0000 7fd9bbfff700 1 --2- 172.21.15.7:0/1786900133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc0c6fe0 0x7fd9bc0bfff0 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009dc0 tx=0x7fd9b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07f78acbf9c2b3e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:26:13.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9ba7fc700 1 -- 172.21.15.7:0/1786900133 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b400e070 con 0x7fd9bc0c6fe0 2023-03-03T20:26:13.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9ba7fc700 1 -- 172.21.15.7:0/1786900133 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b400bb90 con 0x7fd9bc0c6fe0 2023-03-03T20:26:13.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9ba7fc700 1 -- 172.21.15.7:0/1786900133 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4012430 con 0x7fd9bc0c6fe0 2023-03-03T20:26:13.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9c20fa700 1 -- 172.21.15.7:0/1786900133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc0c6fe0 msgr2=0x7fd9bc0bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:13.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9c20fa700 1 --2- 172.21.15.7:0/1786900133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc0c6fe0 0x7fd9bc0bfff0 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009dc0 tx=0x7fd9b400b330 comp rx=0 tx=0).stop 2023-03-03T20:26:13.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9c20fa700 1 -- 172.21.15.7:0/1786900133 shutdown_connections 2023-03-03T20:26:13.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9c20fa700 1 --2- 172.21.15.7:0/1786900133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc0c6fe0 0x7fd9bc0bfff0 unknown :-1 s=CLOSED pgs=3787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:13.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9c20fa700 1 --2- 172.21.15.7:0/1786900133 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9bc151e80 0x7fd9bc0c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:13.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9c20fa700 1 -- 172.21.15.7:0/1786900133 >> 172.21.15.7:0/1786900133 conn(0x7fd9bc071630 msgr2=0x7fd9bc06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:26:13.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9c20fa700 1 -- 172.21.15.7:0/1786900133 shutdown_connections 2023-03-03T20:26:13.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.739+0000 7fd9c20fa700 1 -- 172.21.15.7:0/1786900133 wait complete. 2023-03-03T20:26:13.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.740+0000 7fd9c20fa700 1 Processor -- start 2023-03-03T20:26:13.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.741+0000 7fd9c20fa700 1 -- start start 2023-03-03T20:26:13.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.741+0000 7fd9c20fa700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9bc0c6fe0 0x7fd9bc0c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:13.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.741+0000 7fd9c20fa700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9bc151e80 0x7fd9bc14ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:13.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.741+0000 7fd9c20fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc155410 0x7fd9bc0c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:13.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.741+0000 7fd9c20fa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0c8ee0 con 0x7fd9bc155410 2023-03-03T20:26:13.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.741+0000 7fd9c20fa700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0c9740 con 0x7fd9bc0c6fe0 2023-03-03T20:26:13.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.741+0000 7fd9c20fa700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0ca7d0 con 0x7fd9bc151e80 2023-03-03T20:26:13.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.741+0000 7fd9bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc155410 0x7fd9bc0c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:13.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.741+0000 7fd9bbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc155410 0x7fd9bc0c23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45776/0 (socket says 172.21.15.7:45776) 2023-03-03T20:26:13.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.741+0000 7fd9bbfff700 1 -- 172.21.15.7:0/2512039416 learned_addr learned my addr 172.21.15.7:0/2512039416 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:26:13.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9baffd700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9bc151e80 0x7fd9bc14ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:13.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9bb7fe700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9bc0c6fe0 0x7fd9bc0c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:13.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9bbfff700 1 -- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9bc151e80 msgr2=0x7fd9bc14ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:13.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9bbfff700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9bc151e80 0x7fd9bc14ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:13.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9bbfff700 1 -- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9bc0c6fe0 msgr2=0x7fd9bc0c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:13.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9bbfff700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9bc0c6fe0 0x7fd9bc0c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:13.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9bbfff700 1 -- 172.21.15.7:0/2512039416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b4009a70 con 0x7fd9bc155410 2023-03-03T20:26:13.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9bbfff700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc155410 0x7fd9bc0c23d0 secure :-1 s=READY pgs=3788 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0cb4b0 tx=0x7fd9b4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:26:13.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9b3fff700 1 -- 172.21.15.7:0/2512039416 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4012430 con 0x7fd9bc155410 2023-03-03T20:26:13.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9b3fff700 1 -- 172.21.15.7:0/2512039416 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b4020070 con 0x7fd9bc155410 2023-03-03T20:26:13.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9c20fa700 1 -- 172.21.15.7:0/2512039416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9bc154ef0 con 0x7fd9bc155410 2023-03-03T20:26:13.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9b3fff700 1 -- 172.21.15.7:0/2512039416 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4004280 con 0x7fd9bc155410 2023-03-03T20:26:13.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.742+0000 7fd9c20fa700 1 -- 172.21.15.7:0/2512039416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9bc154c10 con 0x7fd9bc155410 2023-03-03T20:26:13.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.743+0000 7fd9c20fa700 1 -- 172.21.15.7:0/2512039416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc0c5220 con 0x7fd9bc155410 2023-03-03T20:26:13.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.744+0000 7fd9b3fff700 1 -- 172.21.15.7:0/2512039416 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd9b40043e0 con 0x7fd9bc155410 2023-03-03T20:26:13.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.744+0000 7fd9b3fff700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd99c0682f0 0x7fd99c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:13.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.744+0000 7fd9b3fff700 1 -- 172.21.15.7:0/2512039416 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd9b4022070 con 0x7fd9bc155410 2023-03-03T20:26:13.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.744+0000 7fd9bb7fe700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd99c0682f0 0x7fd99c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:13.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.745+0000 7fd9bb7fe700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd99c0682f0 0x7fd99c06a7a0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00b9b0 tx=0x7fd9ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:26:13.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.748+0000 7fd9b3fff700 1 -- 172.21.15.7:0/2512039416 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd9b4054970 con 0x7fd9bc155410 2023-03-03T20:26:13.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.987+0000 7fd9c20fa700 1 -- 172.21.15.7:0/2512039416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9bc1529e0 con 0x7fd9bc155410 2023-03-03T20:26:13.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.988+0000 7fd9b3fff700 1 -- 172.21.15.7:0/2512039416 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd9b4057f90 con 0x7fd9bc155410 2023-03-03T20:26:13.989 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:26:13.990 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:26:13.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.989+0000 7fd9c20fa700 1 -- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd99c0682f0 msgr2=0x7fd99c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:13.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.989+0000 7fd9c20fa700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd99c0682f0 0x7fd99c06a7a0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00b9b0 tx=0x7fd9ac008040 comp rx=0 tx=0).stop 2023-03-03T20:26:13.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.989+0000 7fd9c20fa700 1 -- 172.21.15.7:0/2512039416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc155410 msgr2=0x7fd9bc0c23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:13.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.989+0000 7fd9c20fa700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc155410 0x7fd9bc0c23d0 secure :-1 s=READY pgs=3788 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0cb4b0 tx=0x7fd9b4004ad0 comp rx=0 tx=0).stop 2023-03-03T20:26:13.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.989+0000 7fd9c20fa700 1 -- 172.21.15.7:0/2512039416 shutdown_connections 2023-03-03T20:26:13.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.989+0000 7fd9c20fa700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd99c0682f0 0x7fd99c06a7a0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:13.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.989+0000 7fd9c20fa700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd9bc155410 0x7fd9bc0c23d0 unknown :-1 s=CLOSED pgs=3788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:13.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.990+0000 7fd9c20fa700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9bc0c6fe0 0x7fd9bc0c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:13.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.990+0000 7fd9c20fa700 1 --2- 172.21.15.7:0/2512039416 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd9bc151e80 0x7fd9bc14ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:13.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.990+0000 7fd9c20fa700 1 -- 172.21.15.7:0/2512039416 >> 172.21.15.7:0/2512039416 conn(0x7fd9bc071630 msgr2=0x7fd9bc1513d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:26:13.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.990+0000 7fd9c20fa700 1 -- 172.21.15.7:0/2512039416 shutdown_connections 2023-03-03T20:26:13.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:13.990+0000 7fd9c20fa700 1 -- 172.21.15.7:0/2512039416 wait complete. 2023-03-03T20:26:13.993 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:26:14.003 INFO:tasks.mds_thrash.fs.[b]:waiting for 104 secs before thrashing 2023-03-03T20:26:30.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:26:30.017 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:26:52.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:26:52.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.792+0000 7f7955f75700 1 Processor -- start 2023-03-03T20:26:52.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.792+0000 7f7955f75700 1 -- start start 2023-03-03T20:26:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f7955f75700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7950072250 0x7f795014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f7955f75700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79501510d0 con 0x7f79500c1ce0 2023-03-03T20:26:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f7955f75700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79500c3740 con 0x7f7950071e80 2023-03-03T20:26:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f7955f75700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7950070670 con 0x7f7950072250 2023-03-03T20:26:52.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f794effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7950072250 0x7f795014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:52.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f794ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f79500c1ce0 0x7f795015ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39794/0 (socket says 172.21.15.7:39794) 2023-03-03T20:26:52.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f794effd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7950072250 0x7f795014e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45812/0 (socket says 172.21.15.7:45812) 2023-03-03T20:26:52.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f794ffff700 1 -- 172.21.15.7:0/3053510570 learned_addr learned my addr 172.21.15.7:0/3053510570 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:26:52.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f794effd700 1 -- 172.21.15.7:0/3053510570 >> v1:172.21.15.143:6789/0 conn(0x7f7950071e80 legacy=0x7f7950151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:52.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f794effd700 1 -- 172.21.15.7:0/3053510570 >> v1:172.21.15.7:6789/0 conn(0x7f79500c1ce0 legacy=0x7f795015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:52.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f794e7fc700 1 -- 172.21.15.7:0/3053510570 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699527907 0 0) 0x7f79501510d0 con 0x7f79500c1ce0 2023-03-03T20:26:52.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.793+0000 7f794effd700 1 -- 172.21.15.7:0/3053510570 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79500c39e0 con 0x7f7950072250 2023-03-03T20:26:52.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f794effd700 1 --2- 172.21.15.7:0/3053510570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 0x7f795014e1f0 secure :-1 s=READY pgs=3791 cs=0 l=1 rev1=1 crypto rx=0x7f7938009870 tx=0x7f7938009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798a36fc5c18df67 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:26:52.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f794e7fc700 1 -- 172.21.15.7:0/3053510570 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938011830 con 0x7f7950072250 2023-03-03T20:26:52.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f794e7fc700 1 -- 172.21.15.7:0/3053510570 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7938011990 con 0x7f7950072250 2023-03-03T20:26:52.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f794e7fc700 1 -- 172.21.15.7:0/3053510570 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938011d20 con 0x7f7950072250 2023-03-03T20:26:52.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f7955f75700 1 -- 172.21.15.7:0/3053510570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 msgr2=0x7f795014e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:52.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f7955f75700 1 --2- 172.21.15.7:0/3053510570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 0x7f795014e1f0 secure :-1 s=READY pgs=3791 cs=0 l=1 rev1=1 crypto rx=0x7f7938009870 tx=0x7f7938009b80 comp rx=0 tx=0).stop 2023-03-03T20:26:52.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f7955f75700 1 -- 172.21.15.7:0/3053510570 shutdown_connections 2023-03-03T20:26:52.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f7955f75700 1 --2- 172.21.15.7:0/3053510570 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 0x7f795014e1f0 unknown :-1 s=CLOSED pgs=3791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f7955f75700 1 -- 172.21.15.7:0/3053510570 >> 172.21.15.7:0/3053510570 conn(0x7f79500a94d0 msgr2=0x7f79500a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:26:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f7955f75700 1 -- 172.21.15.7:0/3053510570 shutdown_connections 2023-03-03T20:26:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.794+0000 7f7955f75700 1 -- 172.21.15.7:0/3053510570 wait complete. 2023-03-03T20:26:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.795+0000 7f7955f75700 1 Processor -- start 2023-03-03T20:26:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.796+0000 7f7955f75700 1 -- start start 2023-03-03T20:26:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.796+0000 7f7955f75700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7950071e80 0x7f79501da940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.796+0000 7f7955f75700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 0x7f795014f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:52.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.796+0000 7f7955f75700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79500c1ce0 0x7f7950142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:52.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.796+0000 7f7955f75700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7950151010 con 0x7f7950072250 2023-03-03T20:26:52.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.796+0000 7f7955f75700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79501db020 con 0x7f7950071e80 2023-03-03T20:26:52.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.796+0000 7f7955f75700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7950150670 con 0x7f79500c1ce0 2023-03-03T20:26:52.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.796+0000 7f794effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 0x7f795014f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:52.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 0x7f795014f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45818/0 (socket says 172.21.15.7:45818) 2023-03-03T20:26:52.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794effd700 1 -- 172.21.15.7:0/1630221446 learned_addr learned my addr 172.21.15.7:0/1630221446 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:26:52.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794effd700 1 -- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79500c1ce0 msgr2=0x7f7950142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:52.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794f7fe700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7950071e80 0x7f79501da940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:52.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794ffff700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79500c1ce0 0x7f7950142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:52.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794effd700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79500c1ce0 0x7f7950142590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:52.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794effd700 1 -- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7950071e80 msgr2=0x7f79501da940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:52.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794effd700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7950071e80 0x7f79501da940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:52.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794effd700 1 -- 172.21.15.7:0/1630221446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79380094a0 con 0x7f7950072250 2023-03-03T20:26:52.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794f7fe700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7950071e80 0x7f79501da940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:26:52.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f794effd700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 0x7f795014f630 secure :-1 s=READY pgs=3792 cs=0 l=1 rev1=1 crypto rx=0x7f7938000c00 tx=0x7f7938004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:26:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f7947fff700 1 -- 172.21.15.7:0/1630221446 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79380045b0 con 0x7f7950072250 2023-03-03T20:26:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f7947fff700 1 -- 172.21.15.7:0/1630221446 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793801f070 con 0x7f7950072250 2023-03-03T20:26:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f7955f75700 1 -- 172.21.15.7:0/1630221446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7950151f30 con 0x7f7950072250 2023-03-03T20:26:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f7947fff700 1 -- 172.21.15.7:0/1630221446 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938003ae0 con 0x7f7950072250 2023-03-03T20:26:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.797+0000 7f7955f75700 1 -- 172.21.15.7:0/1630221446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7950151300 con 0x7f7950072250 2023-03-03T20:26:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.798+0000 7f7955f75700 1 -- 172.21.15.7:0/1630221446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795006c9a0 con 0x7f7950072250 2023-03-03T20:26:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.799+0000 7f7947fff700 1 -- 172.21.15.7:0/1630221446 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f793800f050 con 0x7f7950072250 2023-03-03T20:26:52.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.799+0000 7f7947fff700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79300682f0 0x7f793006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:26:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.799+0000 7f7947fff700 1 -- 172.21.15.7:0/1630221446 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f7938021070 con 0x7f7950072250 2023-03-03T20:26:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.799+0000 7f794f7fe700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79300682f0 0x7f793006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:26:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.800+0000 7f794f7fe700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79300682f0 0x7f793006a7a0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f7940000f60 tx=0x7f7940009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:26:52.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:52.803+0000 7f7947fff700 1 -- 172.21.15.7:0/1630221446 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f79380540a0 con 0x7f7950072250 2023-03-03T20:26:53.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.041+0000 7f7955f75700 1 -- 172.21.15.7:0/1630221446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7950151150 con 0x7f7950072250 2023-03-03T20:26:53.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.042+0000 7f7947fff700 1 -- 172.21.15.7:0/1630221446 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f7938015020 con 0x7f7950072250 2023-03-03T20:26:53.043 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:26:53.043 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:26:53.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 -- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79300682f0 msgr2=0x7f793006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:53.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79300682f0 0x7f793006a7a0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f7940000f60 tx=0x7f7940009040 comp rx=0 tx=0).stop 2023-03-03T20:26:53.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 -- 172.21.15.7:0/1630221446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 msgr2=0x7f795014f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:26:53.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 0x7f795014f630 secure :-1 s=READY pgs=3792 cs=0 l=1 rev1=1 crypto rx=0x7f7938000c00 tx=0x7f7938004750 comp rx=0 tx=0).stop 2023-03-03T20:26:53.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 -- 172.21.15.7:0/1630221446 shutdown_connections 2023-03-03T20:26:53.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f79500c1ce0 0x7f7950142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:53.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7950071e80 0x7f79501da940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:53.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f79300682f0 0x7f793006a7a0 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:53.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 --2- 172.21.15.7:0/1630221446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7950072250 0x7f795014f630 unknown :-1 s=CLOSED pgs=3792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:26:53.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 -- 172.21.15.7:0/1630221446 >> 172.21.15.7:0/1630221446 conn(0x7f79500a94d0 msgr2=0x7f7950065fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:26:53.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 -- 172.21.15.7:0/1630221446 shutdown_connections 2023-03-03T20:26:53.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:26:53.044+0000 7f7955f75700 1 -- 172.21.15.7:0/1630221446 wait complete. 2023-03-03T20:26:53.046 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:26:53.056 INFO:tasks.mds_thrash.fs.[a]:waiting for 48 secs before thrashing 2023-03-03T20:27:00.077 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:27:00.080 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:27:30.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:27:30.157 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:27:41.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:27:41.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.196+0000 7fb9b8596700 1 Processor -- start 2023-03-03T20:27:41.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.196+0000 7fb9b8596700 1 -- start start 2023-03-03T20:27:41.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b8596700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9b014fda0 0x7fb9b0156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:41.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b8596700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb9b00abe20 0x7fb9b01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:41.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b8596700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b00c2f30 con 0x7fb9b014f9d0 2023-03-03T20:27:41.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b8596700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b00c31d0 con 0x7fb9b00abe20 2023-03-03T20:27:41.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b8596700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b01524a0 con 0x7fb9b014fda0 2023-03-03T20:27:41.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b6332700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb9b014f9d0 0x7fb9b0071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39830/0 (socket says 172.21.15.7:39830) 2023-03-03T20:27:41.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b6332700 1 -- 172.21.15.7:0/1817577053 learned_addr learned my addr 172.21.15.7:0/1817577053 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:27:41.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b6b33700 1 --2- 172.21.15.7:0/1817577053 >> v2:172.21.15.7:3300/0 conn(0x7fb9b00abe20 0x7fb9b01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:41.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b5b31700 1 --2- 172.21.15.7:0/1817577053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9b014fda0 0x7fb9b0156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:41.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b5330700 1 -- 172.21.15.7:0/1817577053 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270432609 0 0) 0x7fb9b00c2f30 con 0x7fb9b014f9d0 2023-03-03T20:27:41.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b5330700 1 -- 172.21.15.7:0/1817577053 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb99c0035b0 con 0x7fb9b014f9d0 2023-03-03T20:27:41.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b6b33700 1 -- 172.21.15.7:0/1817577053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9b014fda0 msgr2=0x7fb9b0156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:27:41.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b6b33700 1 --2- 172.21.15.7:0/1817577053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9b014fda0 0x7fb9b0156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:41.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b6b33700 1 -- 172.21.15.7:0/1817577053 >> v1:172.21.15.7:6789/0 conn(0x7fb9b014f9d0 legacy=0x7fb9b0071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:41.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.197+0000 7fb9b6b33700 1 -- 172.21.15.7:0/1817577053 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b00c34b0 con 0x7fb9b00abe20 2023-03-03T20:27:41.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b6b33700 1 --2- 172.21.15.7:0/1817577053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 0x7fb9b01529c0 secure :-1 s=READY pgs=3794 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009d90 tx=0x7fb9a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa20a42b617472c6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:27:41.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b5330700 1 -- 172.21.15.7:0/1817577053 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a800e070 con 0x7fb9b00abe20 2023-03-03T20:27:41.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b5330700 1 -- 172.21.15.7:0/1817577053 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a800bb20 con 0x7fb9b00abe20 2023-03-03T20:27:41.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b5330700 1 -- 172.21.15.7:0/1817577053 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8012430 con 0x7fb9b00abe20 2023-03-03T20:27:41.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b8596700 1 -- 172.21.15.7:0/1817577053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 msgr2=0x7fb9b01529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:41.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b8596700 1 --2- 172.21.15.7:0/1817577053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 0x7fb9b01529c0 secure :-1 s=READY pgs=3794 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009d90 tx=0x7fb9a800b330 comp rx=0 tx=0).stop 2023-03-03T20:27:41.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b8596700 1 -- 172.21.15.7:0/1817577053 shutdown_connections 2023-03-03T20:27:41.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b8596700 1 --2- 172.21.15.7:0/1817577053 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 0x7fb9b01529c0 unknown :-1 s=CLOSED pgs=3794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:41.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b8596700 1 --2- 172.21.15.7:0/1817577053 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9b014fda0 0x7fb9b0156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:41.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b8596700 1 -- 172.21.15.7:0/1817577053 >> 172.21.15.7:0/1817577053 conn(0x7fb9b00c8900 msgr2=0x7fb9b00c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:27:41.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b8596700 1 -- 172.21.15.7:0/1817577053 shutdown_connections 2023-03-03T20:27:41.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.198+0000 7fb9b8596700 1 -- 172.21.15.7:0/1817577053 wait complete. 2023-03-03T20:27:41.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.199+0000 7fb9b8596700 1 Processor -- start 2023-03-03T20:27:41.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b8596700 1 -- start start 2023-03-03T20:27:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b8596700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 0x7fb9b01216f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b8596700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9b014f9d0 0x7fb9b00c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b8596700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9b014fda0 0x7fb9b006e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b8596700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b0121e80 con 0x7fb9b00abe20 2023-03-03T20:27:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b8596700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b01221d0 con 0x7fb9b014fda0 2023-03-03T20:27:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b8596700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b006af60 con 0x7fb9b014f9d0 2023-03-03T20:27:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b6332700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 0x7fb9b01216f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:41.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b6332700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 0x7fb9b01216f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45856/0 (socket says 172.21.15.7:45856) 2023-03-03T20:27:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b6332700 1 -- 172.21.15.7:0/408689261 learned_addr learned my addr 172.21.15.7:0/408689261 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:27:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b5b31700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9b014f9d0 0x7fb9b00c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.200+0000 7fb9b6b33700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9b014fda0 0x7fb9b006e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9b6332700 1 -- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9b014f9d0 msgr2=0x7fb9b00c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9b6332700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9b014f9d0 0x7fb9b00c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9b6332700 1 -- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9b014fda0 msgr2=0x7fb9b006e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:41.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9b6332700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9b014fda0 0x7fb9b006e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:41.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9b6332700 1 -- 172.21.15.7:0/408689261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a8009a70 con 0x7fb9b00abe20 2023-03-03T20:27:41.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9b6b33700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9b014fda0 0x7fb9b006e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:27:41.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9b6332700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 0x7fb9b01216f0 secure :-1 s=READY pgs=3795 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00c730 tx=0x7fb9ac00ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:27:41.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9a6ffd700 1 -- 172.21.15.7:0/408689261 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac005d70 con 0x7fb9b00abe20 2023-03-03T20:27:41.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9a6ffd700 1 -- 172.21.15.7:0/408689261 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ac0070d0 con 0x7fb9b00abe20 2023-03-03T20:27:41.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9b8596700 1 -- 172.21.15.7:0/408689261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b00ac350 con 0x7fb9b00abe20 2023-03-03T20:27:41.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9a6ffd700 1 -- 172.21.15.7:0/408689261 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac007460 con 0x7fb9b00abe20 2023-03-03T20:27:41.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.201+0000 7fb9b8596700 1 -- 172.21.15.7:0/408689261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b00ab630 con 0x7fb9b00abe20 2023-03-03T20:27:41.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.202+0000 7fb9b8596700 1 -- 172.21.15.7:0/408689261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b0071fa0 con 0x7fb9b00abe20 2023-03-03T20:27:41.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.203+0000 7fb9a6ffd700 1 -- 172.21.15.7:0/408689261 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb9ac019030 con 0x7fb9b00abe20 2023-03-03T20:27:41.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.203+0000 7fb9a6ffd700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9940682f0 0x7fb99406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:41.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.203+0000 7fb9a6ffd700 1 -- 172.21.15.7:0/408689261 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb9ac085530 con 0x7fb9b00abe20 2023-03-03T20:27:41.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.203+0000 7fb9b5b31700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9940682f0 0x7fb99406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:41.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.204+0000 7fb9b5b31700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9940682f0 0x7fb99406a7a0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fb9b01463c0 tx=0x7fb9a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:27:41.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.208+0000 7fb9a6ffd700 1 -- 172.21.15.7:0/408689261 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb9ac0518d0 con 0x7fb9b00abe20 2023-03-03T20:27:41.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.447+0000 7fb9b8596700 1 -- 172.21.15.7:0/408689261 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9b0074bb0 con 0x7fb9b00abe20 2023-03-03T20:27:41.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.448+0000 7fb9a6ffd700 1 -- 172.21.15.7:0/408689261 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb9ac054ef0 con 0x7fb9b00abe20 2023-03-03T20:27:41.449 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:27:41.450 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:27:41.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 -- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9940682f0 msgr2=0x7fb99406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:41.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9940682f0 0x7fb99406a7a0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fb9b01463c0 tx=0x7fb9a0008040 comp rx=0 tx=0).stop 2023-03-03T20:27:41.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 -- 172.21.15.7:0/408689261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 msgr2=0x7fb9b01216f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:41.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 0x7fb9b01216f0 secure :-1 s=READY pgs=3795 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac00c730 tx=0x7fb9ac00ca40 comp rx=0 tx=0).stop 2023-03-03T20:27:41.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 -- 172.21.15.7:0/408689261 shutdown_connections 2023-03-03T20:27:41.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb9b014fda0 0x7fb9b006e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:41.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb9940682f0 0x7fb99406a7a0 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:41.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb9b00abe20 0x7fb9b01216f0 unknown :-1 s=CLOSED pgs=3795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:41.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 --2- 172.21.15.7:0/408689261 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb9b014f9d0 0x7fb9b00c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:41.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 -- 172.21.15.7:0/408689261 >> 172.21.15.7:0/408689261 conn(0x7fb9b00c8900 msgr2=0x7fb9b014efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:27:41.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 -- 172.21.15.7:0/408689261 shutdown_connections 2023-03-03T20:27:41.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:41.450+0000 7fb9b8596700 1 -- 172.21.15.7:0/408689261 wait complete. 2023-03-03T20:27:41.455 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:27:41.463 INFO:tasks.mds_thrash.fs.[a]:waiting for 82 secs before thrashing 2023-03-03T20:27:58.004 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:27:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.142+0000 7f76670da700 1 Processor -- start 2023-03-03T20:27:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.143+0000 7f76670da700 1 -- start start 2023-03-03T20:27:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.143+0000 7f76670da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7660071e80 0x7f7660151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.143+0000 7f76670da700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f7660072250 0x7f766014e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.143+0000 7f76670da700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76601510d0 con 0x7f76600c1ce0 2023-03-03T20:27:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.143+0000 7f76670da700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76600c3740 con 0x7f7660072250 2023-03-03T20:27:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.143+0000 7f76670da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7660070670 con 0x7f7660071e80 2023-03-03T20:27:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f7665677700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f76600c1ce0 0x7f76601546c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39860/0 (socket says 172.21.15.7:39860) 2023-03-03T20:27:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f7665677700 1 -- 172.21.15.7:0/3334515618 learned_addr learned my addr 172.21.15.7:0/3334515618 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:27:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765ffff700 1 --2- 172.21.15.7:0/3334515618 >> v2:172.21.15.143:3300/0 conn(0x7f7660072250 0x7f766014e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112446037 0 0) 0x7f76601510d0 con 0x7f76600c1ce0 2023-03-03T20:27:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76500035b0 con 0x7f76600c1ce0 2023-03-03T20:27:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f7664e76700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7660071e80 0x7f7660151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2599372871 0 0) 0x7f76500035b0 con 0x7f76600c1ce0 2023-03-03T20:27:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7660071e80 msgr2=0x7f7660151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:27:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7660071e80 0x7f7660151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 >> v2:172.21.15.143:3300/0 conn(0x7f7660072250 msgr2=0x7f766014e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:27:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 --2- 172.21.15.7:0/3334515618 >> v2:172.21.15.143:3300/0 conn(0x7f7660072250 0x7f766014e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76600c39e0 con 0x7f76600c1ce0 2023-03-03T20:27:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f76580019f0 con 0x7f76600c1ce0 2023-03-03T20:27:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 >> v1:172.21.15.7:6789/0 conn(0x7f76600c1ce0 legacy=0x7f76601546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7650003a70 0x7f7650003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 0x7f7650004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7650005200 0x7f765000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76600c3740 con 0x7f7650004850 2023-03-03T20:27:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7660070670 con 0x7f7650003a70 2023-03-03T20:27:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765000c050 con 0x7f7650005200 2023-03-03T20:27:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f7664e76700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7660071e80 0x7f7660151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:27:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f765ffff700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 0x7f7650004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.144+0000 7f7665677700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7650005200 0x7f765000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f7664e76700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7650003a70 0x7f7650003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f765ffff700 1 -- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7650005200 msgr2=0x7f765000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:27:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f765ffff700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7650005200 0x7f765000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f765ffff700 1 -- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7650003a70 msgr2=0x7f7650003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:27:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f765ffff700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7650003a70 0x7f7650003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f765ffff700 1 -- 172.21.15.7:0/3334515618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76500035b0 con 0x7f7650004850 2023-03-03T20:27:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f7665677700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7650005200 0x7f765000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:27:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f7664e76700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7650003a70 0x7f7650003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:27:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f765ffff700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 0x7f7650004cc0 secure :-1 s=READY pgs=3798 cs=0 l=1 rev1=1 crypto rx=0x7f764c009b30 tx=0x7f7650005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8001c5ef68ced2ee server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:27:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c00e6c0 con 0x7f7650004850 2023-03-03T20:27:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764c00e820 con 0x7f7650004850 2023-03-03T20:27:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f765f7fe700 1 -- 172.21.15.7:0/3334515618 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c00eb90 con 0x7f7650004850 2023-03-03T20:27:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f76670da700 1 -- 172.21.15.7:0/3334515618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 msgr2=0x7f7650004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f76670da700 1 --2- 172.21.15.7:0/3334515618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 0x7f7650004cc0 secure :-1 s=READY pgs=3798 cs=0 l=1 rev1=1 crypto rx=0x7f764c009b30 tx=0x7f7650005880 comp rx=0 tx=0).stop 2023-03-03T20:27:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.145+0000 7f7665677700 1 -- 172.21.15.7:0/3334515618 reap_dead start 2023-03-03T20:27:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.146+0000 7f76670da700 1 -- 172.21.15.7:0/3334515618 shutdown_connections 2023-03-03T20:27:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.146+0000 7f76670da700 1 -- 172.21.15.7:0/3334515618 >> 172.21.15.7:0/3334515618 conn(0x7f76600a94d0 msgr2=0x7f76600a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:27:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.146+0000 7f76670da700 1 -- 172.21.15.7:0/3334515618 shutdown_connections 2023-03-03T20:27:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.146+0000 7f76670da700 1 -- 172.21.15.7:0/3334515618 wait complete. 2023-03-03T20:27:58.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.147+0000 7f76670da700 1 Processor -- start 2023-03-03T20:27:58.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.147+0000 7f76670da700 1 -- start start 2023-03-03T20:27:58.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.148+0000 7f76670da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 0x7f766014ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:58.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.148+0000 7f76670da700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76601e7bf0 0x7f7660143630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:58.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.148+0000 7f76670da700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7660150b30 0x7f7660143b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:58.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.148+0000 7f76670da700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76600ad1b0 con 0x7f7650004850 2023-03-03T20:27:58.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.148+0000 7f76670da700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76600bac30 con 0x7f76601e7bf0 2023-03-03T20:27:58.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.148+0000 7f76670da700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76600abc40 con 0x7f7660150b30 2023-03-03T20:27:58.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.148+0000 7f7664e76700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 0x7f766014ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:58.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f7664e76700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 0x7f766014ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45888/0 (socket says 172.21.15.7:45888) 2023-03-03T20:27:58.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f7664e76700 1 -- 172.21.15.7:0/1366753883 learned_addr learned my addr 172.21.15.7:0/1366753883 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:27:58.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f7665677700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7660150b30 0x7f7660143b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f765ffff700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76601e7bf0 0x7f7660143630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f7664e76700 1 -- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7660150b30 msgr2=0x7f7660143b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f7664e76700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7660150b30 0x7f7660143b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f7664e76700 1 -- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76601e7bf0 msgr2=0x7f7660143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f7664e76700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76601e7bf0 0x7f7660143630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f7664e76700 1 -- 172.21.15.7:0/1366753883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766006c9a0 con 0x7f7650004850 2023-03-03T20:27:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f7664e76700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 0x7f766014ac80 secure :-1 s=READY pgs=3799 cs=0 l=1 rev1=1 crypto rx=0x7f7654010810 tx=0x7f7654010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:27:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f765d7fa700 1 -- 172.21.15.7:0/1366753883 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654015880 con 0x7f7650004850 2023-03-03T20:27:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f765d7fa700 1 -- 172.21.15.7:0/1366753883 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765400c050 con 0x7f7650004850 2023-03-03T20:27:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f76670da700 1 -- 172.21.15.7:0/1366753883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7660151300 con 0x7f7650004850 2023-03-03T20:27:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f765d7fa700 1 -- 172.21.15.7:0/1366753883 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654015be0 con 0x7f7650004850 2023-03-03T20:27:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.149+0000 7f76670da700 1 -- 172.21.15.7:0/1366753883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766006cb30 con 0x7f7650004850 2023-03-03T20:27:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.150+0000 7f76670da700 1 -- 172.21.15.7:0/1366753883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76600a98a0 con 0x7f7650004850 2023-03-03T20:27:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.151+0000 7f765d7fa700 1 -- 172.21.15.7:0/1366753883 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f7654026070 con 0x7f7650004850 2023-03-03T20:27:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.151+0000 7f765d7fa700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7644068320 0x7f764406a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:27:58.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.151+0000 7f765d7fa700 1 -- 172.21.15.7:0/1366753883 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f76540868b0 con 0x7f7650004850 2023-03-03T20:27:58.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.151+0000 7f765ffff700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7644068320 0x7f764406a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:27:58.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.152+0000 7f765ffff700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7644068320 0x7f764406a7d0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f7660151220 tx=0x7f764c008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:27:58.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.155+0000 7f765d7fa700 1 -- 172.21.15.7:0/1366753883 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f7654052bd0 con 0x7f7650004850 2023-03-03T20:27:58.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.395+0000 7f76670da700 1 -- 172.21.15.7:0/1366753883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76601eb3e0 con 0x7f7650004850 2023-03-03T20:27:58.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.395+0000 7f765d7fa700 1 -- 172.21.15.7:0/1366753883 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f76540561f0 con 0x7f7650004850 2023-03-03T20:27:58.396 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:27:58.396 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:27:58.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 -- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7644068320 msgr2=0x7f764406a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:58.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7644068320 0x7f764406a7d0 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f7660151220 tx=0x7f764c008030 comp rx=0 tx=0).stop 2023-03-03T20:27:58.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 -- 172.21.15.7:0/1366753883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 msgr2=0x7f766014ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:27:58.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 0x7f766014ac80 secure :-1 s=READY pgs=3799 cs=0 l=1 rev1=1 crypto rx=0x7f7654010810 tx=0x7f7654010bd0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 -- 172.21.15.7:0/1366753883 shutdown_connections 2023-03-03T20:27:58.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f7660150b30 0x7f7660143b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f7644068320 0x7f764406a7d0 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7650004850 0x7f766014ac80 unknown :-1 s=CLOSED pgs=3799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 --2- 172.21.15.7:0/1366753883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76601e7bf0 0x7f7660143630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:27:58.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 -- 172.21.15.7:0/1366753883 >> 172.21.15.7:0/1366753883 conn(0x7f76600a94d0 msgr2=0x7f76600c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:27:58.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 -- 172.21.15.7:0/1366753883 shutdown_connections 2023-03-03T20:27:58.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:27:58.397+0000 7f76670da700 1 -- 172.21.15.7:0/1366753883 wait complete. 2023-03-03T20:27:58.401 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:27:58.411 INFO:tasks.mds_thrash.fs.[b]:waiting for 55 secs before thrashing 2023-03-03T20:28:00.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:28:00.219 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:28:30.293 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:28:30.296 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:28:53.412 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:28:53.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.551+0000 7fb749994700 1 Processor -- start 2023-03-03T20:28:53.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.551+0000 7fb749994700 1 -- start start 2023-03-03T20:28:53.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.552+0000 7fb749994700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fb744156490 0x7fb74415dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:28:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.552+0000 7fb749994700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7441568d0 con 0x7fb744152e30 2023-03-03T20:28:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.552+0000 7fb749994700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7441523b0 con 0x7fb7440beab0 2023-03-03T20:28:53.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.552+0000 7fb749994700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb744072010 con 0x7fb744156490 2023-03-03T20:28:53.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.552+0000 7fb742ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb744152e30 0x7fb7440c43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39906/0 (socket says 172.21.15.7:39906) 2023-03-03T20:28:53.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.552+0000 7fb742ffd700 1 -- 172.21.15.7:0/2029744994 learned_addr learned my addr 172.21.15.7:0/2029744994 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:28:53.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb7437fe700 1 --2- 172.21.15.7:0/2029744994 >> v2:172.21.15.143:3300/0 conn(0x7fb744156490 0x7fb74415dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:28:53.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362564541 0 0) 0x7fb7441568d0 con 0x7fb744152e30 2023-03-03T20:28:53.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb72c0035b0 con 0x7fb744152e30 2023-03-03T20:28:53.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1940160613 0 0) 0x7fb72c0035b0 con 0x7fb744152e30 2023-03-03T20:28:53.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 >> v2:172.21.15.143:3300/0 conn(0x7fb744156490 msgr2=0x7fb74415dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:28:53.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 --2- 172.21.15.7:0/2029744994 >> v2:172.21.15.143:3300/0 conn(0x7fb744156490 0x7fb74415dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:28:53.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 >> v1:172.21.15.143:6789/0 conn(0x7fb7440beab0 legacy=0x7fb744155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:28:53.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7440aaaf0 con 0x7fb744152e30 2023-03-03T20:28:53.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fb734001670 con 0x7fb744152e30 2023-03-03T20:28:53.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 >> v1:172.21.15.7:6789/0 conn(0x7fb744152e30 legacy=0x7fb7440c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:28:53.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb72c003a70 0x7fb72c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:28:53.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb72c004850 0x7fb72c004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:28:53.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb72c005250 0x7fb72c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:28:53.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744072010 con 0x7fb72c003a70 2023-03-03T20:28:53.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c00c190 con 0x7fb72c005250 2023-03-03T20:28:53.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c00c350 con 0x7fb72c004850 2023-03-03T20:28:53.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb742ffd700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb72c003a70 0x7fb72c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:28:53.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb7437fe700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb72c005250 0x7fb72c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:28:53.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.553+0000 7fb7427fc700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb72c004850 0x7fb72c004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:28:53.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb742ffd700 1 -- 172.21.15.7:0/2029744994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb72c004850 msgr2=0x7fb72c004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:28:53.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb742ffd700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb72c004850 0x7fb72c004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:28:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb742ffd700 1 -- 172.21.15.7:0/2029744994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb72c005250 msgr2=0x7fb72c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:28:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb742ffd700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb72c005250 0x7fb72c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:28:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb7437fe700 1 -- 172.21.15.7:0/2029744994 reap_dead start 2023-03-03T20:28:53.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb742ffd700 1 -- 172.21.15.7:0/2029744994 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0035b0 con 0x7fb72c003a70 2023-03-03T20:28:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb742ffd700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb72c003a70 0x7fb72c003e80 secure :-1 s=READY pgs=3802 cs=0 l=1 rev1=1 crypto rx=0x7fb73400c210 tx=0x7fb72c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c39ccc19ca1b1b server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:28:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734007c10 con 0x7fb72c003a70 2023-03-03T20:28:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73400cd30 con 0x7fb72c003a70 2023-03-03T20:28:53.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb741ffb700 1 -- 172.21.15.7:0/2029744994 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73400cea0 con 0x7fb72c003a70 2023-03-03T20:28:53.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb749994700 1 -- 172.21.15.7:0/2029744994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb72c003a70 msgr2=0x7fb72c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:28:53.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.554+0000 7fb749994700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb72c003a70 0x7fb72c003e80 secure :-1 s=READY pgs=3802 cs=0 l=1 rev1=1 crypto rx=0x7fb73400c210 tx=0x7fb72c005880 comp rx=0 tx=0).stop 2023-03-03T20:28:53.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.555+0000 7fb749994700 1 -- 172.21.15.7:0/2029744994 shutdown_connections 2023-03-03T20:28:53.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.555+0000 7fb749994700 1 --2- 172.21.15.7:0/2029744994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb72c003a70 0x7fb72c003e80 unknown :-1 s=CLOSED pgs=3802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:28:53.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.555+0000 7fb749994700 1 -- 172.21.15.7:0/2029744994 >> 172.21.15.7:0/2029744994 conn(0x7fb7440736f0 msgr2=0x7fb744073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:28:53.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.555+0000 7fb749994700 1 -- 172.21.15.7:0/2029744994 shutdown_connections 2023-03-03T20:28:53.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.555+0000 7fb749994700 1 -- 172.21.15.7:0/2029744994 wait complete. 2023-03-03T20:28:53.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.556+0000 7fb749994700 1 Processor -- start 2023-03-03T20:28:53.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.556+0000 7fb749994700 1 -- start start 2023-03-03T20:28:53.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb749994700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb72c003a70 0x7fb7440addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:28:53.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb749994700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7441e74d0 0x7fb7440be050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:28:53.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb749994700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7441eafc0 0x7fb7441eb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:28:53.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb749994700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7441eb980 con 0x7fb7441eafc0 2023-03-03T20:28:53.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb749994700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7441eab70 con 0x7fb7441e74d0 2023-03-03T20:28:53.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb749994700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7440ae300 con 0x7fb72c003a70 2023-03-03T20:28:53.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb7437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7441eafc0 0x7fb7441eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:28:53.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb7437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7441eafc0 0x7fb7441eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45942/0 (socket says 172.21.15.7:45942) 2023-03-03T20:28:53.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb7437fe700 1 -- 172.21.15.7:0/3932141246 learned_addr learned my addr 172.21.15.7:0/3932141246 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:28:53.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb742ffd700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb72c003a70 0x7fb7440addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:28:53.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb7427fc700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7441e74d0 0x7fb7440be050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:28:53.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb7437fe700 1 -- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb72c003a70 msgr2=0x7fb7440addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:28:53.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb7437fe700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb72c003a70 0x7fb7440addc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:28:53.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb7437fe700 1 -- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7441e74d0 msgr2=0x7fb7440be050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:28:53.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb7437fe700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7441e74d0 0x7fb7440be050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:28:53.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb7437fe700 1 -- 172.21.15.7:0/3932141246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744071c70 con 0x7fb7441eafc0 2023-03-03T20:28:53.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb742ffd700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb72c003a70 0x7fb7440addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:28:53.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb7437fe700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7441eafc0 0x7fb7441eb440 secure :-1 s=READY pgs=3803 cs=0 l=1 rev1=1 crypto rx=0x7fb74414ff90 tx=0x7fb73c011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:28:53.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb72bfff700 1 -- 172.21.15.7:0/3932141246 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00e040 con 0x7fb7441eafc0 2023-03-03T20:28:53.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb72bfff700 1 -- 172.21.15.7:0/3932141246 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c00f040 con 0x7fb7441eafc0 2023-03-03T20:28:53.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb749994700 1 -- 172.21.15.7:0/3932141246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7440aadd0 con 0x7fb7441eafc0 2023-03-03T20:28:53.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.557+0000 7fb72bfff700 1 -- 172.21.15.7:0/3932141246 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00fd50 con 0x7fb7441eafc0 2023-03-03T20:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.558+0000 7fb749994700 1 -- 172.21.15.7:0/3932141246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb744153200 con 0x7fb7441eafc0 2023-03-03T20:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.558+0000 7fb749994700 1 -- 172.21.15.7:0/3932141246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb744074bb0 con 0x7fb7441eafc0 2023-03-03T20:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.559+0000 7fb72bfff700 1 -- 172.21.15.7:0/3932141246 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fb73c022460 con 0x7fb7441eafc0 2023-03-03T20:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.560+0000 7fb72bfff700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7300682f0 0x7fb73006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.560+0000 7fb72bfff700 1 -- 172.21.15.7:0/3932141246 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fb73c0874e0 con 0x7fb7441eafc0 2023-03-03T20:28:53.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.560+0000 7fb742ffd700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7300682f0 0x7fb73006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:28:53.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.561+0000 7fb742ffd700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7300682f0 0x7fb73006a7a0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fb734015040 tx=0x7fb734007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:28:53.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.564+0000 7fb72bfff700 1 -- 172.21.15.7:0/3932141246 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fb73c0538c0 con 0x7fb7441eafc0 2023-03-03T20:28:53.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.803+0000 7fb749994700 1 -- 172.21.15.7:0/3932141246 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7441eace0 con 0x7fb7441eafc0 2023-03-03T20:28:53.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.804+0000 7fb72bfff700 1 -- 172.21.15.7:0/3932141246 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fb7441eace0 con 0x7fb7441eafc0 2023-03-03T20:28:53.805 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:28:53.805 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:28:53.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 -- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7300682f0 msgr2=0x7fb73006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:28:53.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7300682f0 0x7fb73006a7a0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fb734015040 tx=0x7fb734007000 comp rx=0 tx=0).stop 2023-03-03T20:28:53.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 -- 172.21.15.7:0/3932141246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7441eafc0 msgr2=0x7fb7441eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:28:53.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7441eafc0 0x7fb7441eb440 secure :-1 s=READY pgs=3803 cs=0 l=1 rev1=1 crypto rx=0x7fb74414ff90 tx=0x7fb73c011f80 comp rx=0 tx=0).stop 2023-03-03T20:28:53.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 -- 172.21.15.7:0/3932141246 shutdown_connections 2023-03-03T20:28:53.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fb7300682f0 0x7fb73006a7a0 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:28:53.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7441eafc0 0x7fb7441eb440 unknown :-1 s=CLOSED pgs=3803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:28:53.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fb72c003a70 0x7fb7440addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:28:53.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 --2- 172.21.15.7:0/3932141246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7441e74d0 0x7fb7440be050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:28:53.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 -- 172.21.15.7:0/3932141246 >> 172.21.15.7:0/3932141246 conn(0x7fb7440736f0 msgr2=0x7fb7440bd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:28:53.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 -- 172.21.15.7:0/3932141246 shutdown_connections 2023-03-03T20:28:53.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:28:53.806+0000 7fb749994700 1 -- 172.21.15.7:0/3932141246 wait complete. 2023-03-03T20:28:53.809 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:28:53.819 INFO:tasks.mds_thrash.fs.[b]:waiting for 20 secs before thrashing 2023-03-03T20:29:00.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:29:00.358 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:29:03.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:29:03.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.602+0000 7f4092d76700 1 Processor -- start 2023-03-03T20:29:03.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.602+0000 7f4092d76700 1 -- start start 2023-03-03T20:29:03.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.602+0000 7f4092d76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 0x7f408c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:03.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.602+0000 7f4092d76700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f408c14fda0 0x7f408c071890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:03.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.602+0000 7f4092d76700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f408c0abe20 0x7f408c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:03.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.602+0000 7f4092d76700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f408c0c2f30 con 0x7f408c0abe20 2023-03-03T20:29:03.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.602+0000 7f4092d76700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f408c0c31d0 con 0x7f408c14fda0 2023-03-03T20:29:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.602+0000 7f4092d76700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408c1524a0 con 0x7f408c14f9d0 2023-03-03T20:29:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f4091313700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f408c0abe20 0x7f408c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f4091313700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f408c0abe20 0x7f408c1529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45950/0 (socket says 172.21.15.7:45950) 2023-03-03T20:29:03.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f4091313700 1 -- 172.21.15.7:0/291279558 learned_addr learned my addr 172.21.15.7:0/291279558 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:29:03.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f408bfff700 1 --2- 172.21.15.7:0/291279558 >> v2:172.21.15.143:3300/0 conn(0x7f408c14fda0 0x7f408c071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:03.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f4090b12700 1 --2- 172.21.15.7:0/291279558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 0x7f408c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:03.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f4091313700 1 -- 172.21.15.7:0/291279558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 msgr2=0x7f408c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:29:03.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f4091313700 1 --2- 172.21.15.7:0/291279558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 0x7f408c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f4091313700 1 -- 172.21.15.7:0/291279558 >> v2:172.21.15.143:3300/0 conn(0x7f408c14fda0 msgr2=0x7f408c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:29:03.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f4091313700 1 --2- 172.21.15.7:0/291279558 >> v2:172.21.15.143:3300/0 conn(0x7f408c14fda0 0x7f408c071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f4091313700 1 -- 172.21.15.7:0/291279558 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c0c34b0 con 0x7f408c0abe20 2023-03-03T20:29:03.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f4091313700 1 --2- 172.21.15.7:0/291279558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c0abe20 0x7f408c1529c0 secure :-1 s=READY pgs=3804 cs=0 l=1 rev1=1 crypto rx=0x7f408c0ab220 tx=0x7f408400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce5b7c0a771afbe server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:03.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f408b7fe700 1 -- 172.21.15.7:0/291279558 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408400e070 con 0x7f408c0abe20 2023-03-03T20:29:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f408b7fe700 1 -- 172.21.15.7:0/291279558 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408400b9c0 con 0x7f408c0abe20 2023-03-03T20:29:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.603+0000 7f408b7fe700 1 -- 172.21.15.7:0/291279558 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408400bd70 con 0x7f408c0abe20 2023-03-03T20:29:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.604+0000 7f4092d76700 1 -- 172.21.15.7:0/291279558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c0abe20 msgr2=0x7f408c1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.604+0000 7f4092d76700 1 --2- 172.21.15.7:0/291279558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c0abe20 0x7f408c1529c0 secure :-1 s=READY pgs=3804 cs=0 l=1 rev1=1 crypto rx=0x7f408c0ab220 tx=0x7f408400b330 comp rx=0 tx=0).stop 2023-03-03T20:29:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.604+0000 7f4092d76700 1 -- 172.21.15.7:0/291279558 shutdown_connections 2023-03-03T20:29:03.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.604+0000 7f4092d76700 1 --2- 172.21.15.7:0/291279558 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c0abe20 0x7f408c1529c0 unknown :-1 s=CLOSED pgs=3804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.604+0000 7f4092d76700 1 --2- 172.21.15.7:0/291279558 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 0x7f408c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.604+0000 7f4092d76700 1 --2- 172.21.15.7:0/291279558 >> v2:172.21.15.143:3300/0 conn(0x7f408c14fda0 0x7f408c071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.604+0000 7f4092d76700 1 -- 172.21.15.7:0/291279558 >> 172.21.15.7:0/291279558 conn(0x7f408c0c8900 msgr2=0x7f408c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:29:03.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.604+0000 7f4092d76700 1 -- 172.21.15.7:0/291279558 shutdown_connections 2023-03-03T20:29:03.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.604+0000 7f4092d76700 1 -- 172.21.15.7:0/291279558 wait complete. 2023-03-03T20:29:03.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.606+0000 7f4092d76700 1 Processor -- start 2023-03-03T20:29:03.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.606+0000 7f4092d76700 1 -- start start 2023-03-03T20:29:03.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4092d76700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f408c0abe20 0x7f408c14c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:03.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4092d76700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 0x7f408c1558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:03.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4092d76700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c14fda0 0x7f408c1e73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4092d76700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c06f4f0 con 0x7f408c14fda0 2023-03-03T20:29:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4092d76700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c0bf660 con 0x7f408c0abe20 2023-03-03T20:29:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4092d76700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408c1ea8b0 con 0x7f408c14f9d0 2023-03-03T20:29:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4091313700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c14fda0 0x7f408c1e73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4090b12700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f408c0abe20 0x7f408c14c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:03.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f408bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 0x7f408c1558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4090b12700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f408c0abe20 0x7f408c14c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.7:41726/0 (socket says 172.21.15.7:41726) 2023-03-03T20:29:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f408bfff700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 0x7f408c1558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3301/0 says I am v2:172.21.15.7:54026/0 (socket says 172.21.15.7:54026) 2023-03-03T20:29:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4090b12700 1 -- 172.21.15.7:0/3255180509 learned_addr learned my addr 172.21.15.7:0/3255180509 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:29:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.607+0000 7f4091313700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c14fda0 0x7f408c1e73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45956/0 (socket says 172.21.15.7:45956) 2023-03-03T20:29:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f4091313700 1 -- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 msgr2=0x7f408c1558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:03.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f4091313700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 0x7f408c1558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f4091313700 1 -- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f408c0abe20 msgr2=0x7f408c14c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:03.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f4091313700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f408c0abe20 0x7f408c14c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f4091313700 1 -- 172.21.15.7:0/3255180509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4084009b30 con 0x7f408c14fda0 2023-03-03T20:29:03.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f408bfff700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 0x7f408c1558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:29:03.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f4090b12700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f408c0abe20 0x7f408c14c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:29:03.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f4091313700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c14fda0 0x7f408c1e73a0 secure :-1 s=READY pgs=3805 cs=0 l=1 rev1=1 crypto rx=0x7f4084015000 tx=0x7f4084004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:03.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f40897fa700 1 -- 172.21.15.7:0/3255180509 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40840099b0 con 0x7f408c14fda0 2023-03-03T20:29:03.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f40897fa700 1 -- 172.21.15.7:0/3255180509 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4084004250 con 0x7f408c14fda0 2023-03-03T20:29:03.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f4092d76700 1 -- 172.21.15.7:0/3255180509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c0ac350 con 0x7f408c14fda0 2023-03-03T20:29:03.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f40897fa700 1 -- 172.21.15.7:0/3255180509 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4084004600 con 0x7f408c14fda0 2023-03-03T20:29:03.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.608+0000 7f4092d76700 1 -- 172.21.15.7:0/3255180509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c0ab630 con 0x7f408c14fda0 2023-03-03T20:29:03.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.609+0000 7f4092d76700 1 -- 172.21.15.7:0/3255180509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c0c3e10 con 0x7f408c14fda0 2023-03-03T20:29:03.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.610+0000 7f40897fa700 1 -- 172.21.15.7:0/3255180509 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f4084004760 con 0x7f408c14fda0 2023-03-03T20:29:03.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.610+0000 7f40897fa700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40700682f0 0x7f407006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:03.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.610+0000 7f40897fa700 1 -- 172.21.15.7:0/3255180509 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f4084087240 con 0x7f408c14fda0 2023-03-03T20:29:03.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.610+0000 7f4090b12700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40700682f0 0x7f407006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:03.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.611+0000 7f4090b12700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40700682f0 0x7f407006a7a0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f4080009e80 tx=0x7f4080008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:03.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.615+0000 7f40897fa700 1 -- 172.21.15.7:0/3255180509 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f40840535e0 con 0x7f408c14fda0 2023-03-03T20:29:03.855 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.854+0000 7f4092d76700 1 -- 172.21.15.7:0/3255180509 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f408c074bb0 con 0x7f408c14fda0 2023-03-03T20:29:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.855+0000 7f40897fa700 1 -- 172.21.15.7:0/3255180509 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f408401f090 con 0x7f408c14fda0 2023-03-03T20:29:03.856 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:29:03.856 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:29:03.857 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 -- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40700682f0 msgr2=0x7f407006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:03.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40700682f0 0x7f407006a7a0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f4080009e80 tx=0x7f4080008040 comp rx=0 tx=0).stop 2023-03-03T20:29:03.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 -- 172.21.15.7:0/3255180509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c14fda0 msgr2=0x7f408c1e73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:03.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c14fda0 0x7f408c1e73a0 secure :-1 s=READY pgs=3805 cs=0 l=1 rev1=1 crypto rx=0x7f4084015000 tx=0x7f4084004ad0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 -- 172.21.15.7:0/3255180509 shutdown_connections 2023-03-03T20:29:03.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f40700682f0 0x7f407006a7a0 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.858 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f408c14fda0 0x7f408c1e73a0 unknown :-1 s=CLOSED pgs=3805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f408c0abe20 0x7f408c14c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 --2- 172.21.15.7:0/3255180509 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f408c14f9d0 0x7f408c1558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 -- 172.21.15.7:0/3255180509 >> 172.21.15.7:0/3255180509 conn(0x7f408c0c8900 msgr2=0x7f408c0c7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:29:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 -- 172.21.15.7:0/3255180509 shutdown_connections 2023-03-03T20:29:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:03.857+0000 7f4092d76700 1 -- 172.21.15.7:0/3255180509 wait complete. 2023-03-03T20:29:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:29:03.870 INFO:tasks.mds_thrash.fs.[a]:waiting for 14 secs before thrashing 2023-03-03T20:29:13.820 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:29:13.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.958+0000 7fa24b1c0700 1 Processor -- start 2023-03-03T20:29:13.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.958+0000 7fa24b1c0700 1 -- start start 2023-03-03T20:29:13.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.959+0000 7fa24b1c0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa24413a0f0 0x7fa2440c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:13.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.959+0000 7fa24b1c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 0x7fa24406d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.959+0000 7fa24b1c0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa24406de90 con 0x7fa2441586c0 2023-03-03T20:29:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.959+0000 7fa24b1c0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa244154840 con 0x7fa24413a0f0 2023-03-03T20:29:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.959+0000 7fa24b1c0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa244154560 con 0x7fa2440c1da0 2023-03-03T20:29:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.959+0000 7fa248f5c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa24413a0f0 0x7fa2440c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:13.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.959+0000 7fa248f5c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa24413a0f0 0x7fa2440c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45962/0 (socket says 172.21.15.7:45962) 2023-03-03T20:29:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.959+0000 7fa248f5c700 1 -- 172.21.15.7:0/615929551 learned_addr learned my addr 172.21.15.7:0/615929551 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:29:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.959+0000 7fa24975d700 1 --2- 172.21.15.7:0/615929551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 0x7fa24406d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa248f5c700 1 -- 172.21.15.7:0/615929551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 msgr2=0x7fa24406d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:29:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa248f5c700 1 --2- 172.21.15.7:0/615929551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 0x7fa24406d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa248f5c700 1 -- 172.21.15.7:0/615929551 >> v1:172.21.15.143:6789/0 conn(0x7fa2441586c0 legacy=0x7fa24406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:29:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa248f5c700 1 -- 172.21.15.7:0/615929551 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa244157be0 con 0x7fa24413a0f0 2023-03-03T20:29:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa24975d700 1 --2- 172.21.15.7:0/615929551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 0x7fa24406d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:29:13.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa248f5c700 1 --2- 172.21.15.7:0/615929551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 0x7fa2440c07d0 secure :-1 s=READY pgs=3806 cs=0 l=1 rev1=1 crypto rx=0x7fa2380098f0 tx=0x7fa238009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d6edc309ccd4ae server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa2437fe700 1 -- 172.21.15.7:0/615929551 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa238004230 con 0x7fa24413a0f0 2023-03-03T20:29:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa2437fe700 1 -- 172.21.15.7:0/615929551 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa238004390 con 0x7fa24413a0f0 2023-03-03T20:29:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa2437fe700 1 -- 172.21.15.7:0/615929551 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2380036a0 con 0x7fa24413a0f0 2023-03-03T20:29:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa24b1c0700 1 -- 172.21.15.7:0/615929551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 msgr2=0x7fa2440c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa24b1c0700 1 --2- 172.21.15.7:0/615929551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 0x7fa2440c07d0 secure :-1 s=READY pgs=3806 cs=0 l=1 rev1=1 crypto rx=0x7fa2380098f0 tx=0x7fa238009c00 comp rx=0 tx=0).stop 2023-03-03T20:29:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa24b1c0700 1 -- 172.21.15.7:0/615929551 shutdown_connections 2023-03-03T20:29:13.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa24b1c0700 1 --2- 172.21.15.7:0/615929551 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 0x7fa24406d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa24b1c0700 1 --2- 172.21.15.7:0/615929551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 0x7fa2440c07d0 unknown :-1 s=CLOSED pgs=3806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.960+0000 7fa24b1c0700 1 -- 172.21.15.7:0/615929551 >> 172.21.15.7:0/615929551 conn(0x7fa24406c110 msgr2=0x7fa24406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:29:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.961+0000 7fa24b1c0700 1 -- 172.21.15.7:0/615929551 shutdown_connections 2023-03-03T20:29:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.961+0000 7fa24b1c0700 1 -- 172.21.15.7:0/615929551 wait complete. 2023-03-03T20:29:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.962+0000 7fa24b1c0700 1 Processor -- start 2023-03-03T20:29:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.962+0000 7fa24b1c0700 1 -- start start 2023-03-03T20:29:13.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.962+0000 7fa24b1c0700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 0x7fa2440c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:13.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.962+0000 7fa24b1c0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 0x7fa244071520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:13.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa24b1c0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2441586c0 0x7fa244134400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:13.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa24b1c0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2440c5df0 con 0x7fa24413a0f0 2023-03-03T20:29:13.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa24b1c0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2440723c0 con 0x7fa2441586c0 2023-03-03T20:29:13.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa24b1c0700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa244071090 con 0x7fa2440c1da0 2023-03-03T20:29:13.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa243fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 0x7fa244071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:13.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa243fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 0x7fa244071520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45968/0 (socket says 172.21.15.7:45968) 2023-03-03T20:29:13.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa243fff700 1 -- 172.21.15.7:0/1167285988 learned_addr learned my addr 172.21.15.7:0/1167285988 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:29:13.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa248f5c700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 0x7fa2440c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:13.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa24975d700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2441586c0 0x7fa244134400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:13.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa243fff700 1 -- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 msgr2=0x7fa2440c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:13.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa243fff700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 0x7fa2440c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:13.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa243fff700 1 -- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2441586c0 msgr2=0x7fa244134400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:13.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa243fff700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2441586c0 0x7fa244134400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:13.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa243fff700 1 -- 172.21.15.7:0/1167285988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa238009550 con 0x7fa24413a0f0 2023-03-03T20:29:13.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa243fff700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 0x7fa244071520 secure :-1 s=READY pgs=3807 cs=0 l=1 rev1=1 crypto rx=0x7fa23000c500 tx=0x7fa23000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa2417fa700 1 -- 172.21.15.7:0/1167285988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa230011790 con 0x7fa24413a0f0 2023-03-03T20:29:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa2417fa700 1 -- 172.21.15.7:0/1167285988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2300118f0 con 0x7fa24413a0f0 2023-03-03T20:29:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa24b1c0700 1 -- 172.21.15.7:0/1167285988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2441581a0 con 0x7fa24413a0f0 2023-03-03T20:29:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa2417fa700 1 -- 172.21.15.7:0/1167285988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa230011ca0 con 0x7fa24413a0f0 2023-03-03T20:29:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.963+0000 7fa24b1c0700 1 -- 172.21.15.7:0/1167285988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa244157ec0 con 0x7fa24413a0f0 2023-03-03T20:29:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.964+0000 7fa24b1c0700 1 -- 172.21.15.7:0/1167285988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24406e1a0 con 0x7fa24413a0f0 2023-03-03T20:29:13.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.965+0000 7fa2417fa700 1 -- 172.21.15.7:0/1167285988 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fa230010050 con 0x7fa24413a0f0 2023-03-03T20:29:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.965+0000 7fa2417fa700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa228068320 0x7fa22806a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:13.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.965+0000 7fa2417fa700 1 -- 172.21.15.7:0/1167285988 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fa230085ca0 con 0x7fa24413a0f0 2023-03-03T20:29:13.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.965+0000 7fa248f5c700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa228068320 0x7fa22806a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:13.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.966+0000 7fa248f5c700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa228068320 0x7fa22806a7d0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7fa238009130 tx=0x7fa238008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:13.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:13.970+0000 7fa2417fa700 1 -- 172.21.15.7:0/1167285988 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fa23008a070 con 0x7fa24413a0f0 2023-03-03T20:29:14.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.206+0000 7fa24b1c0700 1 -- 172.21.15.7:0/1167285988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa244074bb0 con 0x7fa24413a0f0 2023-03-03T20:29:14.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.207+0000 7fa2417fa700 1 -- 172.21.15.7:0/1167285988 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fa230052040 con 0x7fa24413a0f0 2023-03-03T20:29:14.208 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:29:14.209 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:29:14.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 -- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa228068320 msgr2=0x7fa22806a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:14.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa228068320 0x7fa22806a7d0 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7fa238009130 tx=0x7fa238008030 comp rx=0 tx=0).stop 2023-03-03T20:29:14.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 -- 172.21.15.7:0/1167285988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 msgr2=0x7fa244071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:14.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 0x7fa244071520 secure :-1 s=READY pgs=3807 cs=0 l=1 rev1=1 crypto rx=0x7fa23000c500 tx=0x7fa23000c810 comp rx=0 tx=0).stop 2023-03-03T20:29:14.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 -- 172.21.15.7:0/1167285988 shutdown_connections 2023-03-03T20:29:14.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa2441586c0 0x7fa244134400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:14.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fa2440c1da0 0x7fa2440c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:14.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fa228068320 0x7fa22806a7d0 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:14.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 --2- 172.21.15.7:0/1167285988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa24413a0f0 0x7fa244071520 unknown :-1 s=CLOSED pgs=3807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:14.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 -- 172.21.15.7:0/1167285988 >> 172.21.15.7:0/1167285988 conn(0x7fa24406c110 msgr2=0x7fa2440736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:29:14.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 -- 172.21.15.7:0/1167285988 shutdown_connections 2023-03-03T20:29:14.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:14.209+0000 7fa24b1c0700 1 -- 172.21.15.7:0/1167285988 wait complete. 2023-03-03T20:29:14.211 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:29:14.222 INFO:tasks.mds_thrash.fs.[b]:waiting for 35 secs before thrashing 2023-03-03T20:29:17.871 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:29:18.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.010+0000 7f256eeda700 1 Processor -- start 2023-03-03T20:29:18.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.010+0000 7f256eeda700 1 -- start start 2023-03-03T20:29:18.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.011+0000 7f256eeda700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 0x7f2568071d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:18.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.011+0000 7f256eeda700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2568070360 con 0x7f2568151c00 2023-03-03T20:29:18.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.011+0000 7f256eeda700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f256806a5e0 con 0x7f256814b460 2023-03-03T20:29:18.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.011+0000 7f256eeda700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25681521a0 con 0x7f2568071990 2023-03-03T20:29:18.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.011+0000 7f256d477700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2568151c00 0x7f256806b180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39956/0 (socket says 172.21.15.7:39956) 2023-03-03T20:29:18.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.011+0000 7f256d477700 1 -- 172.21.15.7:0/1930392218 learned_addr learned my addr 172.21.15.7:0/1930392218 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:29:18.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.011+0000 7f256cc76700 1 --2- 172.21.15.7:0/1930392218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 0x7f2568071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:18.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.011+0000 7f25677fe700 1 -- 172.21.15.7:0/1930392218 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007349839 0 0) 0x7f2568070360 con 0x7f2568151c00 2023-03-03T20:29:18.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.011+0000 7f25677fe700 1 -- 172.21.15.7:0/1930392218 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25580035b0 con 0x7f2568151c00 2023-03-03T20:29:18.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.012+0000 7f25677fe700 1 -- 172.21.15.7:0/1930392218 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558419924 0 0) 0x7f256806a5e0 con 0x7f256814b460 2023-03-03T20:29:18.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.012+0000 7f256cc76700 1 -- 172.21.15.7:0/1930392218 >> v1:172.21.15.143:6789/0 conn(0x7f256814b460 legacy=0x7f256806aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:18.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.012+0000 7f256cc76700 1 -- 172.21.15.7:0/1930392218 >> v1:172.21.15.7:6789/0 conn(0x7f2568151c00 legacy=0x7f256806b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:18.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.012+0000 7f256cc76700 1 -- 172.21.15.7:0/1930392218 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25680c38b0 con 0x7f2568071990 2023-03-03T20:29:18.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.012+0000 7f256cc76700 1 --2- 172.21.15.7:0/1930392218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 0x7f2568071d60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f255c009870 tx=0x7f255c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba9eb5eba8fb26d6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:18.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.012+0000 7f25677fe700 1 -- 172.21.15.7:0/1930392218 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25600019f0 con 0x7f2568071990 2023-03-03T20:29:18.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.012+0000 7f25677fe700 1 -- 172.21.15.7:0/1930392218 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255c004030 con 0x7f2568071990 2023-03-03T20:29:18.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.012+0000 7f25677fe700 1 -- 172.21.15.7:0/1930392218 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c0043c0 con 0x7f2568071990 2023-03-03T20:29:18.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.013+0000 7f256eeda700 1 -- 172.21.15.7:0/1930392218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 msgr2=0x7f2568071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:18.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.013+0000 7f256eeda700 1 --2- 172.21.15.7:0/1930392218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 0x7f2568071d60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f255c009870 tx=0x7f255c009b80 comp rx=0 tx=0).stop 2023-03-03T20:29:18.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.013+0000 7f256eeda700 1 -- 172.21.15.7:0/1930392218 shutdown_connections 2023-03-03T20:29:18.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.013+0000 7f256eeda700 1 --2- 172.21.15.7:0/1930392218 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 0x7f2568071d60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:18.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.013+0000 7f256eeda700 1 -- 172.21.15.7:0/1930392218 >> 172.21.15.7:0/1930392218 conn(0x7f25680a94d0 msgr2=0x7f25680a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:29:18.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.013+0000 7f256eeda700 1 -- 172.21.15.7:0/1930392218 shutdown_connections 2023-03-03T20:29:18.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.013+0000 7f256eeda700 1 -- 172.21.15.7:0/1930392218 wait complete. 2023-03-03T20:29:18.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.014+0000 7f256eeda700 1 Processor -- start 2023-03-03T20:29:18.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.014+0000 7f256eeda700 1 -- start start 2023-03-03T20:29:18.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f256eeda700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 0x7f25681de260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:18.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f256eeda700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2568151c00 0x7f2568147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:18.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f256eeda700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25681483c0 0x7f25681defb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:18.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f256eeda700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25681de7a0 con 0x7f2568151c00 2023-03-03T20:29:18.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f256eeda700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2568152570 con 0x7f25681483c0 2023-03-03T20:29:18.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f256eeda700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2568152b30 con 0x7f2568071990 2023-03-03T20:29:18.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f2567fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2568151c00 0x7f2568147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:18.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f2567fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2568151c00 0x7f2568147c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45982/0 (socket says 172.21.15.7:45982) 2023-03-03T20:29:18.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f2567fff700 1 -- 172.21.15.7:0/3818521313 learned_addr learned my addr 172.21.15.7:0/3818521313 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:29:18.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f256cc76700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 0x7f25681de260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:18.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f256d477700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25681483c0 0x7f25681defb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:18.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f2567fff700 1 -- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 msgr2=0x7f25681de260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:18.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f2567fff700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 0x7f25681de260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:18.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f2567fff700 1 -- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25681483c0 msgr2=0x7f25681defb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:18.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f2567fff700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25681483c0 0x7f25681defb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:18.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f2567fff700 1 -- 172.21.15.7:0/3818521313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c0094a0 con 0x7f2568151c00 2023-03-03T20:29:18.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.015+0000 7f2567fff700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2568151c00 0x7f2568147c10 secure :-1 s=READY pgs=3809 cs=0 l=1 rev1=1 crypto rx=0x7f25540011e0 tx=0x7f255400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:18.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.016+0000 7f25657fa700 1 -- 172.21.15.7:0/3818521313 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25540054e0 con 0x7f2568151c00 2023-03-03T20:29:18.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.016+0000 7f25657fa700 1 -- 172.21.15.7:0/3818521313 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2554003680 con 0x7f2568151c00 2023-03-03T20:29:18.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.016+0000 7f256eeda700 1 -- 172.21.15.7:0/3818521313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25680c3b50 con 0x7f2568151c00 2023-03-03T20:29:18.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.016+0000 7f25657fa700 1 -- 172.21.15.7:0/3818521313 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554003a30 con 0x7f2568151c00 2023-03-03T20:29:18.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.016+0000 7f256eeda700 1 -- 172.21.15.7:0/3818521313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256806a2b0 con 0x7f2568151c00 2023-03-03T20:29:18.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.016+0000 7f256eeda700 1 -- 172.21.15.7:0/3818521313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25680722a0 con 0x7f2568151c00 2023-03-03T20:29:18.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.017+0000 7f25657fa700 1 -- 172.21.15.7:0/3818521313 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f2554019030 con 0x7f2568151c00 2023-03-03T20:29:18.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.017+0000 7f25657fa700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f254c0682f0 0x7f254c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:18.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.017+0000 7f25657fa700 1 -- 172.21.15.7:0/3818521313 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f2554085400 con 0x7f2568151c00 2023-03-03T20:29:18.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.019+0000 7f256cc76700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f254c0682f0 0x7f254c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:18.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.020+0000 7f256cc76700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f254c0682f0 0x7f254c06a7a0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f255c000c00 tx=0x7f255c008000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:18.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.022+0000 7f25657fa700 1 -- 172.21.15.7:0/3818521313 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f25540517a0 con 0x7f2568151c00 2023-03-03T20:29:18.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.256+0000 7f256eeda700 1 -- 172.21.15.7:0/3818521313 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2568152ca0 con 0x7f2568151c00 2023-03-03T20:29:18.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.257+0000 7f25657fa700 1 -- 172.21.15.7:0/3818521313 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f2554017090 con 0x7f2568151c00 2023-03-03T20:29:18.258 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:29:18.258 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:29:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 -- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f254c0682f0 msgr2=0x7f254c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f254c0682f0 0x7f254c06a7a0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f255c000c00 tx=0x7f255c008000 comp rx=0 tx=0).stop 2023-03-03T20:29:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 -- 172.21.15.7:0/3818521313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2568151c00 msgr2=0x7f2568147c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2568151c00 0x7f2568147c10 secure :-1 s=READY pgs=3809 cs=0 l=1 rev1=1 crypto rx=0x7f25540011e0 tx=0x7f255400ca30 comp rx=0 tx=0).stop 2023-03-03T20:29:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 -- 172.21.15.7:0/3818521313 shutdown_connections 2023-03-03T20:29:18.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25681483c0 0x7f25681defb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:18.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f2568071990 0x7f25681de260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:18.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f254c0682f0 0x7f254c06a7a0 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:18.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 --2- 172.21.15.7:0/3818521313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2568151c00 0x7f2568147c10 unknown :-1 s=CLOSED pgs=3809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:18.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 -- 172.21.15.7:0/3818521313 >> 172.21.15.7:0/3818521313 conn(0x7f25680a94d0 msgr2=0x7f256806aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:29:18.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 -- 172.21.15.7:0/3818521313 shutdown_connections 2023-03-03T20:29:18.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:18.259+0000 7f256eeda700 1 -- 172.21.15.7:0/3818521313 wait complete. 2023-03-03T20:29:18.262 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:29:18.272 INFO:tasks.mds_thrash.fs.[a]:waiting for 76 secs before thrashing 2023-03-03T20:29:30.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:29:30.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:29:49.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:29:49.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.362+0000 7f01c45c6700 1 Processor -- start 2023-03-03T20:29:49.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.362+0000 7f01c45c6700 1 -- start start 2023-03-03T20:29:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.362+0000 7f01c45c6700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f01bc152e30 0x7f01bc0c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.362+0000 7f01c45c6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01bc0beab0 0x7f01bc0c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.362+0000 7f01c45c6700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01bc156910 con 0x7f01bc155d80 2023-03-03T20:29:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.362+0000 7f01c45c6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01bc1523b0 con 0x7f01bc0beab0 2023-03-03T20:29:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.362+0000 7f01c45c6700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01bc072010 con 0x7f01bc152e30 2023-03-03T20:29:49.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.362+0000 7f01c1b61700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01bc0beab0 0x7f01bc0c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c1b61700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f01bc0beab0 0x7f01bc0c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46010/0 (socket says 172.21.15.7:46010) 2023-03-03T20:29:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c1b61700 1 -- 172.21.15.7:0/636553184 learned_addr learned my addr 172.21.15.7:0/636553184 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:29:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c2362700 1 --2- 172.21.15.7:0/636553184 >> v2:172.21.15.143:3300/0 conn(0x7f01bc152e30 0x7f01bc0c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c1b61700 1 -- 172.21.15.7:0/636553184 >> v2:172.21.15.143:3300/0 conn(0x7f01bc152e30 msgr2=0x7f01bc0c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:29:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c1b61700 1 --2- 172.21.15.7:0/636553184 >> v2:172.21.15.143:3300/0 conn(0x7f01bc152e30 0x7f01bc0c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c1b61700 1 -- 172.21.15.7:0/636553184 >> v1:172.21.15.143:6789/0 conn(0x7f01bc155d80 legacy=0x7f01bc156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:29:49.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c1b61700 1 -- 172.21.15.7:0/636553184 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc0aaaf0 con 0x7f01bc0beab0 2023-03-03T20:29:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c1b61700 1 --2- 172.21.15.7:0/636553184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc0beab0 0x7f01bc0c4910 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0c60b0 tx=0x7f01b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be4f8241b2c8172 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c1360700 1 -- 172.21.15.7:0/636553184 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0004230 con 0x7f01bc0beab0 2023-03-03T20:29:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c1360700 1 -- 172.21.15.7:0/636553184 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b0004390 con 0x7f01bc0beab0 2023-03-03T20:29:49.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.363+0000 7f01c1360700 1 -- 172.21.15.7:0/636553184 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b00036a0 con 0x7f01bc0beab0 2023-03-03T20:29:49.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.364+0000 7f01c45c6700 1 -- 172.21.15.7:0/636553184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc0beab0 msgr2=0x7f01bc0c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:49.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.364+0000 7f01c45c6700 1 --2- 172.21.15.7:0/636553184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc0beab0 0x7f01bc0c4910 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0c60b0 tx=0x7f01b0009bd0 comp rx=0 tx=0).stop 2023-03-03T20:29:49.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.364+0000 7f01c45c6700 1 -- 172.21.15.7:0/636553184 shutdown_connections 2023-03-03T20:29:49.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.364+0000 7f01c45c6700 1 --2- 172.21.15.7:0/636553184 >> v2:172.21.15.143:3300/0 conn(0x7f01bc152e30 0x7f01bc0c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:49.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.364+0000 7f01c45c6700 1 --2- 172.21.15.7:0/636553184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc0beab0 0x7f01bc0c4910 unknown :-1 s=CLOSED pgs=3810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:49.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.364+0000 7f01c45c6700 1 -- 172.21.15.7:0/636553184 >> 172.21.15.7:0/636553184 conn(0x7f01bc0736f0 msgr2=0x7f01bc073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:29:49.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.364+0000 7f01c45c6700 1 -- 172.21.15.7:0/636553184 shutdown_connections 2023-03-03T20:29:49.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.364+0000 7f01c45c6700 1 -- 172.21.15.7:0/636553184 wait complete. 2023-03-03T20:29:49.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.365+0000 7f01c45c6700 1 Processor -- start 2023-03-03T20:29:49.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.365+0000 7f01c45c6700 1 -- start start 2023-03-03T20:29:49.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c45c6700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc0beab0 0x7f01bc0c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:49.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c45c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01bc152e30 0x7f01bc0ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:49.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c45c6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc155d80 0x7f01bc1e75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:49.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c45c6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0725a0 con 0x7f01bc155d80 2023-03-03T20:29:49.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c45c6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc1e7b10 con 0x7f01bc152e30 2023-03-03T20:29:49.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c45c6700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0c2620 con 0x7f01bc0beab0 2023-03-03T20:29:49.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c2b63700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc155d80 0x7f01bc1e75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:49.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c2b63700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc155d80 0x7f01bc1e75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46020/0 (socket says 172.21.15.7:46020) 2023-03-03T20:29:49.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c2b63700 1 -- 172.21.15.7:0/2532707055 learned_addr learned my addr 172.21.15.7:0/2532707055 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:29:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c1b61700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01bc152e30 0x7f01bc0ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c2362700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc0beab0 0x7f01bc0c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:49.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c2b63700 1 -- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc0beab0 msgr2=0x7f01bc0c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:49.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c2b63700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc0beab0 0x7f01bc0c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:49.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c2b63700 1 -- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01bc152e30 msgr2=0x7f01bc0ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:49.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c2b63700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01bc152e30 0x7f01bc0ae070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:49.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c2b63700 1 -- 172.21.15.7:0/2532707055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0009550 con 0x7f01bc155d80 2023-03-03T20:29:49.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.366+0000 7f01c2b63700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc155d80 0x7f01bc1e75d0 secure :-1 s=READY pgs=3811 cs=0 l=1 rev1=1 crypto rx=0x7f01b400cc40 tx=0x7f01b400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:49.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.367+0000 7f01aeffd700 1 -- 172.21.15.7:0/2532707055 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b400aa80 con 0x7f01bc155d80 2023-03-03T20:29:49.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.367+0000 7f01aeffd700 1 -- 172.21.15.7:0/2532707055 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b400abe0 con 0x7f01bc155d80 2023-03-03T20:29:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.367+0000 7f01c45c6700 1 -- 172.21.15.7:0/2532707055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01bc071c70 con 0x7f01bc155d80 2023-03-03T20:29:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.367+0000 7f01aeffd700 1 -- 172.21.15.7:0/2532707055 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4012420 con 0x7f01bc155d80 2023-03-03T20:29:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.367+0000 7f01c45c6700 1 -- 172.21.15.7:0/2532707055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01bc0aadd0 con 0x7f01bc155d80 2023-03-03T20:29:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.368+0000 7f01c45c6700 1 -- 172.21.15.7:0/2532707055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01bc153200 con 0x7f01bc155d80 2023-03-03T20:29:49.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.368+0000 7f01aeffd700 1 -- 172.21.15.7:0/2532707055 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f01b4022070 con 0x7f01bc155d80 2023-03-03T20:29:49.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.369+0000 7f01aeffd700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 0x7f01a006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:29:49.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.369+0000 7f01aeffd700 1 -- 172.21.15.7:0/2532707055 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f01b40866e0 con 0x7f01bc155d80 2023-03-03T20:29:49.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.369+0000 7f01c2362700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 0x7f01a006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:29:49.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.369+0000 7f01c2362700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 0x7f01a006a7a0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0aa220 tx=0x7f01b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:29:49.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.373+0000 7f01aeffd700 1 -- 172.21.15.7:0/2532707055 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f01b4052a40 con 0x7f01bc155d80 2023-03-03T20:29:49.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.608+0000 7f01c45c6700 1 -- 172.21.15.7:0/2532707055 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01bc0ac1b0 con 0x7f01bc155d80 2023-03-03T20:29:49.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.608+0000 7f01aeffd700 1 -- 172.21.15.7:0/2532707055 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f01bc0ac1b0 con 0x7f01bc155d80 2023-03-03T20:29:49.610 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:29:49.610 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:29:49.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 -- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 msgr2=0x7f01a006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:49.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 0x7f01a006a7a0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f01bc0aa220 tx=0x7f01b8008040 comp rx=0 tx=0).stop 2023-03-03T20:29:49.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 -- 172.21.15.7:0/2532707055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc155d80 msgr2=0x7f01bc1e75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:29:49.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc155d80 0x7f01bc1e75d0 secure :-1 s=READY pgs=3811 cs=0 l=1 rev1=1 crypto rx=0x7f01b400cc40 tx=0x7f01b400cf50 comp rx=0 tx=0).stop 2023-03-03T20:29:49.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 -- 172.21.15.7:0/2532707055 shutdown_connections 2023-03-03T20:29:49.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f01a00682f0 0x7f01a006a7a0 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:49.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f01bc155d80 0x7f01bc1e75d0 unknown :-1 s=CLOSED pgs=3811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:49.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f01bc0beab0 0x7f01bc0c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:49.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 --2- 172.21.15.7:0/2532707055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01bc152e30 0x7f01bc0ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:29:49.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 -- 172.21.15.7:0/2532707055 >> 172.21.15.7:0/2532707055 conn(0x7f01bc0736f0 msgr2=0x7f01bc0c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:29:49.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 -- 172.21.15.7:0/2532707055 shutdown_connections 2023-03-03T20:29:49.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:29:49.610+0000 7f01c45c6700 1 -- 172.21.15.7:0/2532707055 wait complete. 2023-03-03T20:29:49.617 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:29:49.624 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2023-03-03T20:30:00.000 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-03-03T20:29:59.999+0000 7f6f774ea700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 filesystems are degraded; 2 filesystems are offline; 2 mds daemons damaged 2023-03-03T20:30:00.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:30:00.590 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:30:16.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:30:16.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.764+0000 7f80b003d700 1 Processor -- start 2023-03-03T20:30:16.765 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.764+0000 7f80b003d700 1 -- start start 2023-03-03T20:30:16.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.765+0000 7f80b003d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c4850 0x7f80a8154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:16.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.765+0000 7f80b003d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a8072920 con 0x7f80a814e560 2023-03-03T20:30:16.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.765+0000 7f80b003d700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a806bef0 con 0x7f80a80c43d0 2023-03-03T20:30:16.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.765+0000 7f80b003d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8072380 con 0x7f80a80c4850 2023-03-03T20:30:16.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.765+0000 7f80addd9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f80a814e560 0x7f80a8071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40020/0 (socket says 172.21.15.7:40020) 2023-03-03T20:30:16.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.765+0000 7f80addd9700 1 -- 172.21.15.7:0/738250528 learned_addr learned my addr 172.21.15.7:0/738250528 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:30:16.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80ae5da700 1 --2- 172.21.15.7:0/738250528 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c4850 0x7f80a8154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:16.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80acdd7700 1 -- 172.21.15.7:0/738250528 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327483526 0 0) 0x7f8098001110 con 0x7f80a814e560 2023-03-03T20:30:16.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80acdd7700 1 -- 172.21.15.7:0/738250528 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80940035b0 con 0x7f80a814e560 2023-03-03T20:30:16.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80acdd7700 1 -- 172.21.15.7:0/738250528 <== mon.1 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312300576 0 0) 0x7f80a806bef0 con 0x7f80a80c43d0 2023-03-03T20:30:16.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80acdd7700 1 -- 172.21.15.7:0/738250528 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8098001110 con 0x7f80a80c43d0 2023-03-03T20:30:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80acdd7700 1 -- 172.21.15.7:0/738250528 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3681866549 0 0) 0x7f80940035b0 con 0x7f80a814e560 2023-03-03T20:30:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80ae5da700 1 -- 172.21.15.7:0/738250528 >> v1:172.21.15.143:6789/0 conn(0x7f80a80c43d0 legacy=0x7f80a8154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80ae5da700 1 -- 172.21.15.7:0/738250528 >> v1:172.21.15.7:6789/0 conn(0x7f80a814e560 legacy=0x7f80a8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80ae5da700 1 -- 172.21.15.7:0/738250528 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a80c4cb0 con 0x7f80a80c4850 2023-03-03T20:30:16.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80ae5da700 1 --2- 172.21.15.7:0/738250528 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c4850 0x7f80a8154ca0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009dc0 tx=0x7f80a000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe408f928825906e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:30:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.766+0000 7f80acdd7700 1 -- 172.21.15.7:0/738250528 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a000e070 con 0x7f80a80c4850 2023-03-03T20:30:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.767+0000 7f80acdd7700 1 -- 172.21.15.7:0/738250528 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a0003a20 con 0x7f80a80c4850 2023-03-03T20:30:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.767+0000 7f80acdd7700 1 -- 172.21.15.7:0/738250528 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098001800 con 0x7f80a80c4850 2023-03-03T20:30:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.767+0000 7f80b003d700 1 -- 172.21.15.7:0/738250528 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c4850 msgr2=0x7f80a8154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:16.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.767+0000 7f80b003d700 1 --2- 172.21.15.7:0/738250528 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c4850 0x7f80a8154ca0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009dc0 tx=0x7f80a000b330 comp rx=0 tx=0).stop 2023-03-03T20:30:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.767+0000 7f80b003d700 1 -- 172.21.15.7:0/738250528 shutdown_connections 2023-03-03T20:30:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.767+0000 7f80b003d700 1 --2- 172.21.15.7:0/738250528 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c4850 0x7f80a8154ca0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.767+0000 7f80b003d700 1 -- 172.21.15.7:0/738250528 >> 172.21.15.7:0/738250528 conn(0x7f80a80736f0 msgr2=0x7f80a8073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:30:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.767+0000 7f80b003d700 1 -- 172.21.15.7:0/738250528 shutdown_connections 2023-03-03T20:30:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.767+0000 7f80b003d700 1 -- 172.21.15.7:0/738250528 wait complete. 2023-03-03T20:30:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.768+0000 7f80b003d700 1 Processor -- start 2023-03-03T20:30:16.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.768+0000 7f80b003d700 1 -- start start 2023-03-03T20:30:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80b003d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c43d0 0x7f80a80bef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80b003d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a80c4850 0x7f80a80c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80b003d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a814e560 0x7f80a81e9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80b003d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a814e930 con 0x7f80a80c4850 2023-03-03T20:30:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80b003d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a80beaf0 con 0x7f80a814e560 2023-03-03T20:30:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80b003d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a81ed9e0 con 0x7f80a80c43d0 2023-03-03T20:30:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80ad5d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a80c4850 0x7f80a80c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:16.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80addd9700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c43d0 0x7f80a80bef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80ad5d8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a80c4850 0x7f80a80c7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46048/0 (socket says 172.21.15.7:46048) 2023-03-03T20:30:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80ad5d8700 1 -- 172.21.15.7:0/3251376529 learned_addr learned my addr 172.21.15.7:0/3251376529 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:30:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.769+0000 7f80ae5da700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a814e560 0x7f80a81e9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f80ad5d8700 1 -- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c43d0 msgr2=0x7f80a80bef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f80ad5d8700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c43d0 0x7f80a80bef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:16.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f80ad5d8700 1 -- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a814e560 msgr2=0x7f80a81e9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f80ad5d8700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a814e560 0x7f80a81e9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f80ad5d8700 1 -- 172.21.15.7:0/3251376529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a0009a70 con 0x7f80a80c4850 2023-03-03T20:30:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f80ad5d8700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a80c4850 0x7f80a80c7ba0 secure :-1 s=READY pgs=3813 cs=0 l=1 rev1=1 crypto rx=0x7f80a400c550 tx=0x7f80a400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:30:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f809e7fc700 1 -- 172.21.15.7:0/3251376529 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a40052d0 con 0x7f80a80c4850 2023-03-03T20:30:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f809e7fc700 1 -- 172.21.15.7:0/3251376529 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a4005430 con 0x7f80a80c4850 2023-03-03T20:30:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f80b003d700 1 -- 172.21.15.7:0/3251376529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a806b910 con 0x7f80a80c4850 2023-03-03T20:30:16.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f809e7fc700 1 -- 172.21.15.7:0/3251376529 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a40038c0 con 0x7f80a80c4850 2023-03-03T20:30:16.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.770+0000 7f80b003d700 1 -- 172.21.15.7:0/3251376529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a81551e0 con 0x7f80a80c4850 2023-03-03T20:30:16.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.771+0000 7f80b003d700 1 -- 172.21.15.7:0/3251376529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a80aacc0 con 0x7f80a80c4850 2023-03-03T20:30:16.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.771+0000 7f809e7fc700 1 -- 172.21.15.7:0/3251376529 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f80a4019030 con 0x7f80a80c4850 2023-03-03T20:30:16.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.772+0000 7f809e7fc700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f808c0682f0 0x7f808c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:16.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.772+0000 7f809e7fc700 1 -- 172.21.15.7:0/3251376529 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f80a40853c0 con 0x7f80a80c4850 2023-03-03T20:30:16.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.772+0000 7f80addd9700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f808c0682f0 0x7f808c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:16.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.773+0000 7f80addd9700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f808c0682f0 0x7f808c06a7a0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f8098001370 tx=0x7f8098007ba0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:30:16.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:16.776+0000 7f809e7fc700 1 -- 172.21.15.7:0/3251376529 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f80a4051760 con 0x7f80a80c4850 2023-03-03T20:30:17.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.017+0000 7f80b003d700 1 -- 172.21.15.7:0/3251376529 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80a80bedd0 con 0x7f80a80c4850 2023-03-03T20:30:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.017+0000 7f809e7fc700 1 -- 172.21.15.7:0/3251376529 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f80a4054d80 con 0x7f80a80c4850 2023-03-03T20:30:17.018 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:30:17.019 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:30:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 -- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f808c0682f0 msgr2=0x7f808c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f808c0682f0 0x7f808c06a7a0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f8098001370 tx=0x7f8098007ba0 comp rx=0 tx=0).stop 2023-03-03T20:30:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 -- 172.21.15.7:0/3251376529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a80c4850 msgr2=0x7f80a80c7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a80c4850 0x7f80a80c7ba0 secure :-1 s=READY pgs=3813 cs=0 l=1 rev1=1 crypto rx=0x7f80a400c550 tx=0x7f80a400c860 comp rx=0 tx=0).stop 2023-03-03T20:30:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 -- 172.21.15.7:0/3251376529 shutdown_connections 2023-03-03T20:30:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80a814e560 0x7f80a81e9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f80a80c43d0 0x7f80a80bef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f808c0682f0 0x7f808c06a7a0 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 --2- 172.21.15.7:0/3251376529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f80a80c4850 0x7f80a80c7ba0 unknown :-1 s=CLOSED pgs=3813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 -- 172.21.15.7:0/3251376529 >> 172.21.15.7:0/3251376529 conn(0x7f80a80736f0 msgr2=0x7f80a80a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:30:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 -- 172.21.15.7:0/3251376529 shutdown_connections 2023-03-03T20:30:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:17.019+0000 7f80b003d700 1 -- 172.21.15.7:0/3251376529 wait complete. 2023-03-03T20:30:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:30:17.032 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-03-03T20:30:30.725 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:30:30.729 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:30:34.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:30:34.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.408+0000 7fcd6f56d700 1 Processor -- start 2023-03-03T20:30:34.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.408+0000 7fcd6f56d700 1 -- start start 2023-03-03T20:30:34.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.409+0000 7fcd6f56d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd680beab0 0x7fcd68155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:34.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.409+0000 7fcd6f56d700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7fcd681562c0 0x7fcd6815cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:34.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.409+0000 7fcd6f56d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd681523b0 con 0x7fcd68152e30 2023-03-03T20:30:34.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.409+0000 7fcd6f56d700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd68072010 con 0x7fcd681562c0 2023-03-03T20:30:34.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.409+0000 7fcd6f56d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680aaaf0 con 0x7fcd680beab0 2023-03-03T20:30:34.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.409+0000 7fcd6d309700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcd68152e30 0x7fcd680c43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40048/0 (socket says 172.21.15.7:40048) 2023-03-03T20:30:34.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.409+0000 7fcd6d309700 1 -- 172.21.15.7:0/1725665493 learned_addr learned my addr 172.21.15.7:0/1725665493 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:30:34.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.409+0000 7fcd6db0a700 1 --2- 172.21.15.7:0/1725665493 >> v2:172.21.15.143:3300/0 conn(0x7fcd681562c0 0x7fcd6815cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:34.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.409+0000 7fcd6cb08700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd680beab0 0x7fcd68155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118116295 0 0) 0x7fcd681523b0 con 0x7fcd68152e30 2023-03-03T20:30:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd540035b0 con 0x7fcd68152e30 2023-03-03T20:30:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4118403291 0 0) 0x7fcd540035b0 con 0x7fcd68152e30 2023-03-03T20:30:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd680beab0 msgr2=0x7fcd68155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:30:34.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd680beab0 0x7fcd68155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 >> v2:172.21.15.143:3300/0 conn(0x7fcd681562c0 msgr2=0x7fcd6815cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:30:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 --2- 172.21.15.7:0/1725665493 >> v2:172.21.15.143:3300/0 conn(0x7fcd681562c0 0x7fcd6815cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd680aadd0 con 0x7fcd68152e30 2023-03-03T20:30:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7fcd64001670 con 0x7fcd68152e30 2023-03-03T20:30:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 >> v1:172.21.15.7:6789/0 conn(0x7fcd68152e30 legacy=0x7fcd680c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd54003a70 0x7fcd54003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:34.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd54004850 0x7fcd54004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd54005200 0x7fcd5400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd68072010 con 0x7fcd54005200 2023-03-03T20:30:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680aaaf0 con 0x7fcd54004850 2023-03-03T20:30:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5400bfd0 con 0x7fcd54003a70 2023-03-03T20:30:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd6cb08700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd680beab0 0x7fcd68155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:30:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd6db0a700 1 --2- 172.21.15.7:0/1725665493 >> v2:172.21.15.143:3300/0 conn(0x7fcd681562c0 0x7fcd6815cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:30:34.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd6db0a700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd54005200 0x7fcd5400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd6d309700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd54003a70 0x7fcd54003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd6cb08700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd54004850 0x7fcd54004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd6db0a700 1 -- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd54003a70 msgr2=0x7fcd54003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:30:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd6db0a700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd54003a70 0x7fcd54003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd6db0a700 1 -- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd54004850 msgr2=0x7fcd54004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:30:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd6db0a700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd54004850 0x7fcd54004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.410+0000 7fcd6db0a700 1 -- 172.21.15.7:0/1725665493 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd540035b0 con 0x7fcd54005200 2023-03-03T20:30:34.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd6cb08700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd54004850 0x7fcd54004cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:30:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd6db0a700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd54005200 0x7fcd5400ba10 secure :-1 s=READY pgs=3815 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000e350 tx=0x7fcd54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eafcf5f25efdf2c6 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:30:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6000ee10 con 0x7fcd54005200 2023-03-03T20:30:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6000f040 con 0x7fcd54005200 2023-03-03T20:30:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd5bfff700 1 -- 172.21.15.7:0/1725665493 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60003c40 con 0x7fcd54005200 2023-03-03T20:30:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1725665493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd54005200 msgr2=0x7fcd5400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd6f56d700 1 --2- 172.21.15.7:0/1725665493 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd54005200 0x7fcd5400ba10 secure :-1 s=READY pgs=3815 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000e350 tx=0x7fcd54005880 comp rx=0 tx=0).stop 2023-03-03T20:30:34.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd6db0a700 1 -- 172.21.15.7:0/1725665493 reap_dead start 2023-03-03T20:30:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1725665493 shutdown_connections 2023-03-03T20:30:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1725665493 >> 172.21.15.7:0/1725665493 conn(0x7fcd680736f0 msgr2=0x7fcd68073ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:30:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.411+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1725665493 shutdown_connections 2023-03-03T20:30:34.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.412+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1725665493 wait complete. 2023-03-03T20:30:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.413+0000 7fcd6f56d700 1 Processor -- start 2023-03-03T20:30:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.413+0000 7fcd6f56d700 1 -- start start 2023-03-03T20:30:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.413+0000 7fcd6f56d700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd54005200 0x7fcd68151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.413+0000 7fcd6f56d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680c2330 0x7fcd68154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.413+0000 7fcd6f56d700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd68154fa0 0x7fcd6814ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:34.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.413+0000 7fcd6f56d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680ad260 con 0x7fcd680c2330 2023-03-03T20:30:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.413+0000 7fcd6f56d700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd68152480 con 0x7fcd68154fa0 2023-03-03T20:30:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.413+0000 7fcd6f56d700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd681e7e60 con 0x7fcd54005200 2023-03-03T20:30:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6cb08700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680c2330 0x7fcd68154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6cb08700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680c2330 0x7fcd68154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46082/0 (socket says 172.21.15.7:46082) 2023-03-03T20:30:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6cb08700 1 -- 172.21.15.7:0/1446863630 learned_addr learned my addr 172.21.15.7:0/1446863630 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:30:34.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6d309700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd54005200 0x7fcd68151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6db0a700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd68154fa0 0x7fcd6814ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6cb08700 1 -- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd54005200 msgr2=0x7fcd68151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6cb08700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd54005200 0x7fcd68151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6cb08700 1 -- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd68154fa0 msgr2=0x7fcd6814ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6cb08700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd68154fa0 0x7fcd6814ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6cb08700 1 -- 172.21.15.7:0/1446863630 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68153200 con 0x7fcd680c2330 2023-03-03T20:30:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6cb08700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680c2330 0x7fcd68154a60 secure :-1 s=READY pgs=3816 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c0118e0 tx=0x7fcd5c011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:30:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd59ffb700 1 -- 172.21.15.7:0/1446863630 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c00fa80 con 0x7fcd680c2330 2023-03-03T20:30:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd59ffb700 1 -- 172.21.15.7:0/1446863630 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5c00fbe0 con 0x7fcd680c2330 2023-03-03T20:30:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1446863630 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd68071c70 con 0x7fcd680c2330 2023-03-03T20:30:34.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd59ffb700 1 -- 172.21.15.7:0/1446863630 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c011e80 con 0x7fcd680c2330 2023-03-03T20:30:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.414+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1446863630 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd68153390 con 0x7fcd680c2330 2023-03-03T20:30:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.415+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1446863630 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd68074bb0 con 0x7fcd680c2330 2023-03-03T20:30:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.416+0000 7fcd59ffb700 1 -- 172.21.15.7:0/1446863630 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fcd5c012040 con 0x7fcd680c2330 2023-03-03T20:30:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.416+0000 7fcd59ffb700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd4c0682f0 0x7fcd4c06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:30:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.416+0000 7fcd59ffb700 1 -- 172.21.15.7:0/1446863630 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fcd5c086920 con 0x7fcd680c2330 2023-03-03T20:30:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.417+0000 7fcd6d309700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd4c0682f0 0x7fcd4c06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:30:34.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.417+0000 7fcd6d309700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd4c0682f0 0x7fcd4c06a7a0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fcd64001050 tx=0x7fcd64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:30:34.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.421+0000 7fcd59ffb700 1 -- 172.21.15.7:0/1446863630 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fcd5c052bd0 con 0x7fcd680c2330 2023-03-03T20:30:34.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.665+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1446863630 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd681e7fa0 con 0x7fcd680c2330 2023-03-03T20:30:34.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.665+0000 7fcd59ffb700 1 -- 172.21.15.7:0/1446863630 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fcd5c0561f0 con 0x7fcd680c2330 2023-03-03T20:30:34.666 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:30:34.666 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:30:34.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd4c0682f0 msgr2=0x7fcd4c06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:34.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd4c0682f0 0x7fcd4c06a7a0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fcd64001050 tx=0x7fcd64009040 comp rx=0 tx=0).stop 2023-03-03T20:30:34.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1446863630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680c2330 msgr2=0x7fcd68154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:30:34.668 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680c2330 0x7fcd68154a60 secure :-1 s=READY pgs=3816 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c0118e0 tx=0x7fcd5c011ca0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1446863630 shutdown_connections 2023-03-03T20:30:34.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd68154fa0 0x7fcd6814ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fcd54005200 0x7fcd68151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fcd4c0682f0 0x7fcd4c06a7a0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 --2- 172.21.15.7:0/1446863630 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd680c2330 0x7fcd68154a60 unknown :-1 s=CLOSED pgs=3816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:30:34.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1446863630 >> 172.21.15.7:0/1446863630 conn(0x7fcd680736f0 msgr2=0x7fcd680bd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:30:34.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1446863630 shutdown_connections 2023-03-03T20:30:34.669 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:30:34.667+0000 7fcd6f56d700 1 -- 172.21.15.7:0/1446863630 wait complete. 2023-03-03T20:30:34.670 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:30:34.680 INFO:tasks.mds_thrash.fs.[a]:waiting for 35 secs before thrashing 2023-03-03T20:31:00.838 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:31:00.841 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:31:09.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:31:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.819+0000 7fd8104c2700 1 Processor -- start 2023-03-03T20:31:09.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.819+0000 7fd8104c2700 1 -- start start 2023-03-03T20:31:09.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.820+0000 7fd8104c2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd808072250 0x7fd80814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.820+0000 7fd8104c2700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8081510d0 con 0x7fd8080c1ce0 2023-03-03T20:31:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.820+0000 7fd8104c2700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8080c3740 con 0x7fd808071e80 2023-03-03T20:31:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.820+0000 7fd8104c2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd808070670 con 0x7fd808072250 2023-03-03T20:31:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.820+0000 7fd80da5d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd808072250 0x7fd80814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.820+0000 7fd80ea5f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd8080c1ce0 0x7fd80815ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40084/0 (socket says 172.21.15.7:40084) 2023-03-03T20:31:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.820+0000 7fd80ea5f700 1 -- 172.21.15.7:0/3330800214 learned_addr learned my addr 172.21.15.7:0/3330800214 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:31:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.821+0000 7fd80da5d700 1 -- 172.21.15.7:0/3330800214 >> v1:172.21.15.143:6789/0 conn(0x7fd808071e80 legacy=0x7fd808151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:31:09.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.821+0000 7fd80da5d700 1 -- 172.21.15.7:0/3330800214 >> v1:172.21.15.7:6789/0 conn(0x7fd8080c1ce0 legacy=0x7fd80815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:09.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.821+0000 7fd80da5d700 1 -- 172.21.15.7:0/3330800214 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8080c39e0 con 0x7fd808072250 2023-03-03T20:31:09.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.821+0000 7fd80da5d700 1 --2- 172.21.15.7:0/3330800214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808072250 0x7fd80814e1f0 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 crypto rx=0x7fd804009870 tx=0x7fd804009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dcbeece18282a6a server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:31:09.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.821+0000 7fd80d25c700 1 -- 172.21.15.7:0/3330800214 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804004230 con 0x7fd808072250 2023-03-03T20:31:09.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.821+0000 7fd80d25c700 1 -- 172.21.15.7:0/3330800214 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd804004390 con 0x7fd808072250 2023-03-03T20:31:09.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.821+0000 7fd80d25c700 1 -- 172.21.15.7:0/3330800214 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8040036a0 con 0x7fd808072250 2023-03-03T20:31:09.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.821+0000 7fd8104c2700 1 -- 172.21.15.7:0/3330800214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808072250 msgr2=0x7fd80814e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:09.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.821+0000 7fd8104c2700 1 --2- 172.21.15.7:0/3330800214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808072250 0x7fd80814e1f0 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 crypto rx=0x7fd804009870 tx=0x7fd804009b80 comp rx=0 tx=0).stop 2023-03-03T20:31:09.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.822+0000 7fd8104c2700 1 -- 172.21.15.7:0/3330800214 shutdown_connections 2023-03-03T20:31:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.822+0000 7fd8104c2700 1 --2- 172.21.15.7:0/3330800214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808072250 0x7fd80814e1f0 unknown :-1 s=CLOSED pgs=3818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.822+0000 7fd8104c2700 1 -- 172.21.15.7:0/3330800214 >> 172.21.15.7:0/3330800214 conn(0x7fd8080a94d0 msgr2=0x7fd8080a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:31:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.822+0000 7fd8104c2700 1 -- 172.21.15.7:0/3330800214 shutdown_connections 2023-03-03T20:31:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.822+0000 7fd8104c2700 1 -- 172.21.15.7:0/3330800214 wait complete. 2023-03-03T20:31:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.823+0000 7fd8104c2700 1 Processor -- start 2023-03-03T20:31:09.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.823+0000 7fd8104c2700 1 -- start start 2023-03-03T20:31:09.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd8104c2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808071e80 0x7fd80814c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:09.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd8104c2700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd808072250 0x7fd80814c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:09.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd8104c2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8080c1ce0 0x7fd808143630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:09.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd8104c2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80814d920 con 0x7fd808071e80 2023-03-03T20:31:09.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd8104c2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8080c5d10 con 0x7fd8080c1ce0 2023-03-03T20:31:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd8104c2700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8080c5ff0 con 0x7fd808072250 2023-03-03T20:31:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd80e25e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808071e80 0x7fd80814c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd80e25e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808071e80 0x7fd80814c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46106/0 (socket says 172.21.15.7:46106) 2023-03-03T20:31:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd80e25e700 1 -- 172.21.15.7:0/1288360611 learned_addr learned my addr 172.21.15.7:0/1288360611 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:31:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd80ea5f700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8080c1ce0 0x7fd808143630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.824+0000 7fd80da5d700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd808072250 0x7fd80814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:09.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd80e25e700 1 -- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd808072250 msgr2=0x7fd80814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd80e25e700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd808072250 0x7fd80814c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd80e25e700 1 -- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8080c1ce0 msgr2=0x7fd808143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd80e25e700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8080c1ce0 0x7fd808143630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd80e25e700 1 -- 172.21.15.7:0/1288360611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8040094a0 con 0x7fd808071e80 2023-03-03T20:31:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd80ea5f700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8080c1ce0 0x7fd808143630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-03T20:31:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd80e25e700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808071e80 0x7fd80814c310 secure :-1 s=READY pgs=3819 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800c4d0 tx=0x7fd7f800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:31:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd7feffd700 1 -- 172.21.15.7:0/1288360611 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8011790 con 0x7fd808071e80 2023-03-03T20:31:09.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd7feffd700 1 -- 172.21.15.7:0/1288360611 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f80118f0 con 0x7fd808071e80 2023-03-03T20:31:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd8104c2700 1 -- 172.21.15.7:0/1288360611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd808151f30 con 0x7fd808071e80 2023-03-03T20:31:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd7feffd700 1 -- 172.21.15.7:0/1288360611 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8011ca0 con 0x7fd808071e80 2023-03-03T20:31:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.825+0000 7fd8104c2700 1 -- 172.21.15.7:0/1288360611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd808151300 con 0x7fd808071e80 2023-03-03T20:31:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.826+0000 7fd8104c2700 1 -- 172.21.15.7:0/1288360611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80806c9a0 con 0x7fd808071e80 2023-03-03T20:31:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.826+0000 7fd7feffd700 1 -- 172.21.15.7:0/1288360611 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fd7f8010050 con 0x7fd808071e80 2023-03-03T20:31:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.827+0000 7fd7feffd700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7ec0682f0 0x7fd7ec06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:09.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.827+0000 7fd7feffd700 1 -- 172.21.15.7:0/1288360611 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fd7f8085e30 con 0x7fd808071e80 2023-03-03T20:31:09.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.827+0000 7fd80da5d700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7ec0682f0 0x7fd7ec06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:09.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.828+0000 7fd80da5d700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7ec0682f0 0x7fd7ec06a7a0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fd8080c1780 tx=0x7fd804008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:31:09.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:09.831+0000 7fd7feffd700 1 -- 172.21.15.7:0/1288360611 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fd7f80521d0 con 0x7fd808071e80 2023-03-03T20:31:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.070+0000 7fd8104c2700 1 -- 172.21.15.7:0/1288360611 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8081db9e0 con 0x7fd808071e80 2023-03-03T20:31:10.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.071+0000 7fd7feffd700 1 -- 172.21.15.7:0/1288360611 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fd7f80557f0 con 0x7fd808071e80 2023-03-03T20:31:10.074 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:31:10.074 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:31:10.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 -- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7ec0682f0 msgr2=0x7fd7ec06a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:10.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7ec0682f0 0x7fd7ec06a7a0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fd8080c1780 tx=0x7fd804008030 comp rx=0 tx=0).stop 2023-03-03T20:31:10.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 -- 172.21.15.7:0/1288360611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808071e80 msgr2=0x7fd80814c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:10.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808071e80 0x7fd80814c310 secure :-1 s=READY pgs=3819 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800c4d0 tx=0x7fd7f800c890 comp rx=0 tx=0).stop 2023-03-03T20:31:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 -- 172.21.15.7:0/1288360611 shutdown_connections 2023-03-03T20:31:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd8080c1ce0 0x7fd808143630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fd7ec0682f0 0x7fd7ec06a7a0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd808071e80 0x7fd80814c310 unknown :-1 s=CLOSED pgs=3819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 --2- 172.21.15.7:0/1288360611 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fd808072250 0x7fd80814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 -- 172.21.15.7:0/1288360611 >> 172.21.15.7:0/1288360611 conn(0x7fd8080a94d0 msgr2=0x7fd808147d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:31:10.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 -- 172.21.15.7:0/1288360611 shutdown_connections 2023-03-03T20:31:10.077 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:10.073+0000 7fd8104c2700 1 -- 172.21.15.7:0/1288360611 wait complete. 2023-03-03T20:31:10.077 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:31:10.085 INFO:tasks.mds_thrash.fs.[a]:waiting for 41 secs before thrashing 2023-03-03T20:31:30.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:31:30.995 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:31:51.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:31:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.225+0000 7f876aebb700 1 Processor -- start 2023-03-03T20:31:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.226+0000 7f876aebb700 1 -- start start 2023-03-03T20:31:51.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.226+0000 7f876aebb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f8764151ab0 0x7f87640736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:51.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.226+0000 7f876aebb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8764073c30 0x7f876406bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:51.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.226+0000 7f876aebb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876406d780 con 0x7f876406c3e0 2023-03-03T20:31:51.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.226+0000 7f876aebb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876406cb70 con 0x7f8764073c30 2023-03-03T20:31:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.226+0000 7f876aebb700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f876406ce50 con 0x7f8764151ab0 2023-03-03T20:31:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f8763fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8764073c30 0x7f876406bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f8769458700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f876406c3e0 0x7f8764157dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40116/0 (socket says 172.21.15.7:40116) 2023-03-03T20:31:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f8769458700 1 -- 172.21.15.7:0/3782070492 learned_addr learned my addr 172.21.15.7:0/3782070492 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:31:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f8768c57700 1 --2- 172.21.15.7:0/3782070492 >> v2:172.21.15.143:3300/0 conn(0x7f8764151ab0 0x7f87640736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f8763fff700 1 -- 172.21.15.7:0/3782070492 >> v2:172.21.15.143:3300/0 conn(0x7f8764151ab0 msgr2=0x7f87640736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:31:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f8763fff700 1 --2- 172.21.15.7:0/3782070492 >> v2:172.21.15.143:3300/0 conn(0x7f8764151ab0 0x7f87640736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f8763fff700 1 -- 172.21.15.7:0/3782070492 >> v1:172.21.15.7:6789/0 conn(0x7f876406c3e0 legacy=0x7f8764157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f8763fff700 1 -- 172.21.15.7:0/3782070492 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876406d130 con 0x7f8764073c30 2023-03-03T20:31:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f8768c57700 1 --2- 172.21.15.7:0/3782070492 >> v2:172.21.15.143:3300/0 conn(0x7f8764151ab0 0x7f87640736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:31:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f87637fe700 1 -- 172.21.15.7:0/3782070492 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313032314 0 0) 0x7f876406d780 con 0x7f876406c3e0 2023-03-03T20:31:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f8763fff700 1 --2- 172.21.15.7:0/3782070492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8764073c30 0x7f876406bea0 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 crypto rx=0x7f8750009840 tx=0x7f8750009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99f83db88989e1 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:31:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f87637fe700 1 -- 172.21.15.7:0/3782070492 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8750011830 con 0x7f8764073c30 2023-03-03T20:31:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f87637fe700 1 -- 172.21.15.7:0/3782070492 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8750011990 con 0x7f8764073c30 2023-03-03T20:31:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.227+0000 7f87637fe700 1 -- 172.21.15.7:0/3782070492 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8750011d20 con 0x7f8764073c30 2023-03-03T20:31:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.228+0000 7f876aebb700 1 -- 172.21.15.7:0/3782070492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8764073c30 msgr2=0x7f876406bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.228+0000 7f876aebb700 1 --2- 172.21.15.7:0/3782070492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8764073c30 0x7f876406bea0 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 crypto rx=0x7f8750009840 tx=0x7f8750009b50 comp rx=0 tx=0).stop 2023-03-03T20:31:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.228+0000 7f876aebb700 1 -- 172.21.15.7:0/3782070492 shutdown_connections 2023-03-03T20:31:51.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.228+0000 7f876aebb700 1 --2- 172.21.15.7:0/3782070492 >> v2:172.21.15.143:3300/0 conn(0x7f8764151ab0 0x7f87640736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:51.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.228+0000 7f876aebb700 1 --2- 172.21.15.7:0/3782070492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8764073c30 0x7f876406bea0 unknown :-1 s=CLOSED pgs=3821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:51.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.228+0000 7f876aebb700 1 -- 172.21.15.7:0/3782070492 >> 172.21.15.7:0/3782070492 conn(0x7f87640a94d0 msgr2=0x7f87640a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:31:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.228+0000 7f876aebb700 1 -- 172.21.15.7:0/3782070492 shutdown_connections 2023-03-03T20:31:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.228+0000 7f876aebb700 1 -- 172.21.15.7:0/3782070492 wait complete. 2023-03-03T20:31:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.229+0000 7f876aebb700 1 Processor -- start 2023-03-03T20:31:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.229+0000 7f876aebb700 1 -- start start 2023-03-03T20:31:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f876aebb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876406c3e0 0x7f87640c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f876aebb700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8764073c30 0x7f87640c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f876aebb700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8764151ab0 0x7f876414bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f876aebb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87640c0af0 con 0x7f876406c3e0 2023-03-03T20:31:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f876aebb700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8764154550 con 0x7f8764073c30 2023-03-03T20:31:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f876aebb700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87641ed360 con 0x7f8764151ab0 2023-03-03T20:31:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8768c57700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876406c3e0 0x7f87640c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8768c57700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876406c3e0 0x7f87640c1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46140/0 (socket says 172.21.15.7:46140) 2023-03-03T20:31:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8768c57700 1 -- 172.21.15.7:0/3542417195 learned_addr learned my addr 172.21.15.7:0/3542417195 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:31:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8763fff700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8764073c30 0x7f87640c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8769458700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8764151ab0 0x7f876414bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8768c57700 1 -- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8764151ab0 msgr2=0x7f876414bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8768c57700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8764151ab0 0x7f876414bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8768c57700 1 -- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8764073c30 msgr2=0x7f87640c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8768c57700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8764073c30 0x7f87640c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8768c57700 1 -- 172.21.15.7:0/3542417195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87500094a0 con 0x7f876406c3e0 2023-03-03T20:31:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.230+0000 7f8768c57700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876406c3e0 0x7f87640c1120 secure :-1 s=READY pgs=3822 cs=0 l=1 rev1=1 crypto rx=0x7f875800e760 tx=0x7f875800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:31:51.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.231+0000 7f87617fa700 1 -- 172.21.15.7:0/3542417195 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758013610 con 0x7f876406c3e0 2023-03-03T20:31:51.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.231+0000 7f87617fa700 1 -- 172.21.15.7:0/3542417195 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8758013770 con 0x7f876406c3e0 2023-03-03T20:31:51.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.231+0000 7f876aebb700 1 -- 172.21.15.7:0/3542417195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8764151e80 con 0x7f876406c3e0 2023-03-03T20:31:51.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.231+0000 7f87617fa700 1 -- 172.21.15.7:0/3542417195 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758013b20 con 0x7f876406c3e0 2023-03-03T20:31:51.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.231+0000 7f876aebb700 1 -- 172.21.15.7:0/3542417195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876406d410 con 0x7f876406c3e0 2023-03-03T20:31:51.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.232+0000 7f876aebb700 1 -- 172.21.15.7:0/3542417195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8764152010 con 0x7f876406c3e0 2023-03-03T20:31:51.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.232+0000 7f87617fa700 1 -- 172.21.15.7:0/3542417195 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8758023070 con 0x7f876406c3e0 2023-03-03T20:31:51.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.233+0000 7f87617fa700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87480682f0 0x7f874806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:31:51.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.233+0000 7f87617fa700 1 -- 172.21.15.7:0/3542417195 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f8758085db0 con 0x7f876406c3e0 2023-03-03T20:31:51.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.233+0000 7f8763fff700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87480682f0 0x7f874806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:31:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.233+0000 7f8763fff700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87480682f0 0x7f874806a7a0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f87500090b0 tx=0x7f8750008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:31:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.237+0000 7f87617fa700 1 -- 172.21.15.7:0/3542417195 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f8758052150 con 0x7f876406c3e0 2023-03-03T20:31:51.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.474+0000 7f876aebb700 1 -- 172.21.15.7:0/3542417195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8764074bb0 con 0x7f876406c3e0 2023-03-03T20:31:51.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.474+0000 7f87617fa700 1 -- 172.21.15.7:0/3542417195 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f8758055770 con 0x7f876406c3e0 2023-03-03T20:31:51.477 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:31:51.478 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:31:51.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.476+0000 7f876aebb700 1 -- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87480682f0 msgr2=0x7f874806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.476+0000 7f876aebb700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87480682f0 0x7f874806a7a0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f87500090b0 tx=0x7f8750008030 comp rx=0 tx=0).stop 2023-03-03T20:31:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.476+0000 7f876aebb700 1 -- 172.21.15.7:0/3542417195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876406c3e0 msgr2=0x7f87640c1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:31:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.476+0000 7f876aebb700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876406c3e0 0x7f87640c1120 secure :-1 s=READY pgs=3822 cs=0 l=1 rev1=1 crypto rx=0x7f875800e760 tx=0x7f875800eb20 comp rx=0 tx=0).stop 2023-03-03T20:31:51.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.476+0000 7f876aebb700 1 -- 172.21.15.7:0/3542417195 shutdown_connections 2023-03-03T20:31:51.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.476+0000 7f876aebb700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f8764151ab0 0x7f876414bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:51.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.476+0000 7f876aebb700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f87480682f0 0x7f874806a7a0 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:51.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.476+0000 7f876aebb700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f876406c3e0 0x7f87640c1120 unknown :-1 s=CLOSED pgs=3822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:51.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.476+0000 7f876aebb700 1 --2- 172.21.15.7:0/3542417195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8764073c30 0x7f87640c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:31:51.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.476+0000 7f876aebb700 1 -- 172.21.15.7:0/3542417195 >> 172.21.15.7:0/3542417195 conn(0x7f87640a94d0 msgr2=0x7f8764150cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:31:51.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.477+0000 7f876aebb700 1 -- 172.21.15.7:0/3542417195 shutdown_connections 2023-03-03T20:31:51.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:31:51.477+0000 7f876aebb700 1 -- 172.21.15.7:0/3542417195 wait complete. 2023-03-03T20:31:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:31:51.490 INFO:tasks.mds_thrash.fs.[a]:waiting for 76 secs before thrashing 2023-03-03T20:32:01.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:32:01.074 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:32:06.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:32:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.171+0000 7f5d23a30700 1 Processor -- start 2023-03-03T20:32:06.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.171+0000 7f5d23a30700 1 -- start start 2023-03-03T20:32:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.172+0000 7f5d23a30700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5d1c14fda0 0x7f5d1c156430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.172+0000 7f5d23a30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c0abe20 0x7f5d1c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.172+0000 7f5d23a30700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1c0c2f30 con 0x7f5d1c14f9d0 2023-03-03T20:32:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.172+0000 7f5d23a30700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1c0c31d0 con 0x7f5d1c14fda0 2023-03-03T20:32:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.172+0000 7f5d23a30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c1524a0 con 0x7f5d1c0abe20 2023-03-03T20:32:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.172+0000 7f5d21fcd700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c0abe20 0x7f5d1c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.172+0000 7f5d20fcb700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f5d1c14fda0 0x7f5d1c156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.172+0000 7f5d217cc700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f5d1c14f9d0 0x7f5d1c071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.7:43658/0 (socket says 172.21.15.7:43658) 2023-03-03T20:32:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.172+0000 7f5d217cc700 1 -- 172.21.15.7:0/2773853504 learned_addr learned my addr 172.21.15.7:0/2773853504 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:32:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.173+0000 7f5d21fcd700 1 -- 172.21.15.7:0/2773853504 >> v2:172.21.15.143:3300/0 conn(0x7f5d1c14fda0 msgr2=0x7f5d1c156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:32:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.173+0000 7f5d21fcd700 1 --2- 172.21.15.7:0/2773853504 >> v2:172.21.15.143:3300/0 conn(0x7f5d1c14fda0 0x7f5d1c156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:06.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.173+0000 7f5d21fcd700 1 -- 172.21.15.7:0/2773853504 >> v1:172.21.15.143:6789/0 conn(0x7f5d1c14f9d0 legacy=0x7f5d1c071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:06.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.173+0000 7f5d21fcd700 1 -- 172.21.15.7:0/2773853504 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1c0c34b0 con 0x7f5d1c0abe20 2023-03-03T20:32:06.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.173+0000 7f5d20fcb700 1 --2- 172.21.15.7:0/2773853504 >> v2:172.21.15.143:3300/0 conn(0x7f5d1c14fda0 0x7f5d1c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-03T20:32:06.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.173+0000 7f5d21fcd700 1 --2- 172.21.15.7:0/2773853504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c0abe20 0x7f5d1c1529c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009d90 tx=0x7f5d1400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8948978fd48f26e server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:32:06.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.173+0000 7f5d13fff700 1 -- 172.21.15.7:0/2773853504 <== mon.2 v2:172.21.15.143:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1400e070 con 0x7f5d1c0abe20 2023-03-03T20:32:06.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.173+0000 7f5d13fff700 1 -- 172.21.15.7:0/2773853504 <== mon.2 v2:172.21.15.143:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d14003a20 con 0x7f5d1c0abe20 2023-03-03T20:32:06.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.173+0000 7f5d13fff700 1 -- 172.21.15.7:0/2773853504 <== mon.2 v2:172.21.15.143:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d14003dd0 con 0x7f5d1c0abe20 2023-03-03T20:32:06.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.174+0000 7f5d23a30700 1 -- 172.21.15.7:0/2773853504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c0abe20 msgr2=0x7f5d1c1529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:06.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.174+0000 7f5d23a30700 1 --2- 172.21.15.7:0/2773853504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c0abe20 0x7f5d1c1529c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009d90 tx=0x7f5d1400b330 comp rx=0 tx=0).stop 2023-03-03T20:32:06.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.174+0000 7f5d23a30700 1 -- 172.21.15.7:0/2773853504 shutdown_connections 2023-03-03T20:32:06.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.174+0000 7f5d23a30700 1 --2- 172.21.15.7:0/2773853504 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c0abe20 0x7f5d1c1529c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:06.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.174+0000 7f5d23a30700 1 --2- 172.21.15.7:0/2773853504 >> v2:172.21.15.143:3300/0 conn(0x7f5d1c14fda0 0x7f5d1c156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:06.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.174+0000 7f5d23a30700 1 -- 172.21.15.7:0/2773853504 >> 172.21.15.7:0/2773853504 conn(0x7f5d1c0c8900 msgr2=0x7f5d1c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:32:06.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.174+0000 7f5d23a30700 1 -- 172.21.15.7:0/2773853504 shutdown_connections 2023-03-03T20:32:06.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.174+0000 7f5d23a30700 1 -- 172.21.15.7:0/2773853504 wait complete. 2023-03-03T20:32:06.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.175+0000 7f5d23a30700 1 Processor -- start 2023-03-03T20:32:06.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.175+0000 7f5d23a30700 1 -- start start 2023-03-03T20:32:06.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d23a30700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d1c0abe20 0x7f5d1c06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d23a30700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1c14f9d0 0x7f5d1c1db700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d23a30700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c14fda0 0x7f5d1c14e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d23a30700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c06d990 con 0x7f5d1c0abe20 2023-03-03T20:32:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d23a30700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c06ed00 con 0x7f5d1c14f9d0 2023-03-03T20:32:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d23a30700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c06d620 con 0x7f5d1c14fda0 2023-03-03T20:32:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d217cc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d1c0abe20 0x7f5d1c06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:06.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d217cc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d1c0abe20 0x7f5d1c06efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46164/0 (socket says 172.21.15.7:46164) 2023-03-03T20:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d217cc700 1 -- 172.21.15.7:0/2267074155 learned_addr learned my addr 172.21.15.7:0/2267074155 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d20fcb700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1c14f9d0 0x7f5d1c1db700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d21fcd700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c14fda0 0x7f5d1c14e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d217cc700 1 -- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c14fda0 msgr2=0x7f5d1c14e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d217cc700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c14fda0 0x7f5d1c14e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d217cc700 1 -- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1c14f9d0 msgr2=0x7f5d1c1db700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:06.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d217cc700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1c14f9d0 0x7f5d1c1db700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.176+0000 7f5d217cc700 1 -- 172.21.15.7:0/2267074155 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d180094a0 con 0x7f5d1c0abe20 2023-03-03T20:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.177+0000 7f5d217cc700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d1c0abe20 0x7f5d1c06efe0 secure :-1 s=READY pgs=3823 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00c780 tx=0x7f5d0c00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.177+0000 7f5d11ffb700 1 -- 172.21.15.7:0/2267074155 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c004c80 con 0x7f5d1c0abe20 2023-03-03T20:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.177+0000 7f5d11ffb700 1 -- 172.21.15.7:0/2267074155 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c004de0 con 0x7f5d1c0abe20 2023-03-03T20:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.177+0000 7f5d23a30700 1 -- 172.21.15.7:0/2267074155 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d14009a70 con 0x7f5d1c0abe20 2023-03-03T20:32:06.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.177+0000 7f5d11ffb700 1 -- 172.21.15.7:0/2267074155 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c007310 con 0x7f5d1c0abe20 2023-03-03T20:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.177+0000 7f5d23a30700 1 -- 172.21.15.7:0/2267074155 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1c0ac350 con 0x7f5d1c0abe20 2023-03-03T20:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.178+0000 7f5d23a30700 1 -- 172.21.15.7:0/2267074155 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1c0ab630 con 0x7f5d1c0abe20 2023-03-03T20:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.178+0000 7f5d11ffb700 1 -- 172.21.15.7:0/2267074155 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f5d0c01a020 con 0x7f5d1c0abe20 2023-03-03T20:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.179+0000 7f5d11ffb700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d000682f0 0x7f5d0006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.179+0000 7f5d11ffb700 1 -- 172.21.15.7:0/2267074155 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f5d0c085d50 con 0x7f5d1c0abe20 2023-03-03T20:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.179+0000 7f5d20fcb700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d000682f0 0x7f5d0006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:06.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.179+0000 7f5d20fcb700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d000682f0 0x7f5d0006a7a0 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c14f210 tx=0x7f5d18009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:32:06.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.183+0000 7f5d11ffb700 1 -- 172.21.15.7:0/2267074155 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f5d0c0520f0 con 0x7f5d1c0abe20 2023-03-03T20:32:06.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.418+0000 7f5d23a30700 1 -- 172.21.15.7:0/2267074155 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d1c071fa0 con 0x7f5d1c0abe20 2023-03-03T20:32:06.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.419+0000 7f5d11ffb700 1 -- 172.21.15.7:0/2267074155 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f5d0c055710 con 0x7f5d1c0abe20 2023-03-03T20:32:06.420 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:32:06.420 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:32:06.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 -- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d000682f0 msgr2=0x7f5d0006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:06.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d000682f0 0x7f5d0006a7a0 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c14f210 tx=0x7f5d18009210 comp rx=0 tx=0).stop 2023-03-03T20:32:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 -- 172.21.15.7:0/2267074155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d1c0abe20 msgr2=0x7f5d1c06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d1c0abe20 0x7f5d1c06efe0 secure :-1 s=READY pgs=3823 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c00c780 tx=0x7f5d0c00ca90 comp rx=0 tx=0).stop 2023-03-03T20:32:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 -- 172.21.15.7:0/2267074155 shutdown_connections 2023-03-03T20:32:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5d1c14fda0 0x7f5d1c14e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f5d000682f0 0x7f5d0006a7a0 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d1c0abe20 0x7f5d1c06efe0 unknown :-1 s=CLOSED pgs=3823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:06.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 --2- 172.21.15.7:0/2267074155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5d1c14f9d0 0x7f5d1c1db700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:06.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 -- 172.21.15.7:0/2267074155 >> 172.21.15.7:0/2267074155 conn(0x7f5d1c0c8900 msgr2=0x7f5d1c0c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:32:06.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 -- 172.21.15.7:0/2267074155 shutdown_connections 2023-03-03T20:32:06.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:06.421+0000 7f5d23a30700 1 -- 172.21.15.7:0/2267074155 wait complete. 2023-03-03T20:32:06.423 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:32:06.438 INFO:tasks.mds_thrash.fs.[b]:waiting for 41 secs before thrashing 2023-03-03T20:32:31.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:32:31.178 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:32:47.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:32:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.578+0000 7f520b284700 1 Processor -- start 2023-03-03T20:32:47.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.578+0000 7f520b284700 1 -- start start 2023-03-03T20:32:47.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f520b284700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5204151d40 0x7f52040736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:47.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f520b284700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f52040c2930 0x7f5204157a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:47.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f520b284700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520406d430 con 0x7f5204073c30 2023-03-03T20:32:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f520b284700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520406c2a0 con 0x7f52040c2930 2023-03-03T20:32:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f520b284700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52041586e0 con 0x7f5204151d40 2023-03-03T20:32:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f5209821700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5204073c30 0x7f5204157fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40180/0 (socket says 172.21.15.7:40180) 2023-03-03T20:32:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f5209821700 1 -- 172.21.15.7:0/1462968597 learned_addr learned my addr 172.21.15.7:0/1462968597 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:32:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941085859 0 0) 0x7f520406d430 con 0x7f5204073c30 2023-03-03T20:32:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f5209020700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5204151d40 0x7f52040736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:47.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f00035b0 con 0x7f5204073c30 2023-03-03T20:32:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f520881f700 1 --2- 172.21.15.7:0/1462968597 >> v2:172.21.15.143:3300/0 conn(0x7f52040c2930 0x7f5204157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.579+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701795337 0 0) 0x7f51f00035b0 con 0x7f5204073c30 2023-03-03T20:32:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5204151d40 msgr2=0x7f52040736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:32:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5204151d40 0x7f52040736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 >> v2:172.21.15.143:3300/0 conn(0x7f52040c2930 msgr2=0x7f5204157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:32:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 --2- 172.21.15.7:0/1462968597 >> v2:172.21.15.143:3300/0 conn(0x7f52040c2930 0x7f5204157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520406cb00 con 0x7f5204073c30 2023-03-03T20:32:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 978054248 0 0) 0x7f51fc0019f0 con 0x7f5204073c30 2023-03-03T20:32:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 >> v1:172.21.15.7:6789/0 conn(0x7f5204073c30 legacy=0x7f5204157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f0003a70 0x7f51f0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5204151d40 0x7f51f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51f0004d90 0x7f51f0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520406c2a0 con 0x7f51f0003a70 2023-03-03T20:32:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52041586e0 con 0x7f51f0004d90 2023-03-03T20:32:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f000baf0 con 0x7f5204151d40 2023-03-03T20:32:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f5209020700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f0003a70 0x7f51f0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f520881f700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5204151d40 0x7f51f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f5209821700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51f0004d90 0x7f51f0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f5209020700 1 -- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5204151d40 msgr2=0x7f51f0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:32:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f5209020700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f5204151d40 0x7f51f0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f5209020700 1 -- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51f0004d90 msgr2=0x7f51f0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:32:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f5209020700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f51f0004d90 0x7f51f0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f5209020700 1 -- 172.21.15.7:0/1462968597 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f00035b0 con 0x7f51f0003a70 2023-03-03T20:32:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f5209020700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f0003a70 0x7f51f0003e80 secure :-1 s=READY pgs=3825 cs=0 l=1 rev1=1 crypto rx=0x7f5200009b60 tx=0x7f51f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd205427767e325 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:32:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520000e6c0 con 0x7f51f0003a70 2023-03-03T20:32:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520000e820 con 0x7f51f0003a70 2023-03-03T20:32:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.580+0000 7f51fbfff700 1 -- 172.21.15.7:0/1462968597 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520000eb90 con 0x7f51f0003a70 2023-03-03T20:32:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.581+0000 7f520b284700 1 -- 172.21.15.7:0/1462968597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f0003a70 msgr2=0x7f51f0003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.581+0000 7f520b284700 1 --2- 172.21.15.7:0/1462968597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f0003a70 0x7f51f0003e80 secure :-1 s=READY pgs=3825 cs=0 l=1 rev1=1 crypto rx=0x7f5200009b60 tx=0x7f51f0005880 comp rx=0 tx=0).stop 2023-03-03T20:32:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.581+0000 7f5209821700 1 -- 172.21.15.7:0/1462968597 reap_dead start 2023-03-03T20:32:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.581+0000 7f520b284700 1 -- 172.21.15.7:0/1462968597 shutdown_connections 2023-03-03T20:32:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.581+0000 7f520b284700 1 -- 172.21.15.7:0/1462968597 >> 172.21.15.7:0/1462968597 conn(0x7f52040a94d0 msgr2=0x7f52040a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:32:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.581+0000 7f520b284700 1 -- 172.21.15.7:0/1462968597 shutdown_connections 2023-03-03T20:32:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.581+0000 7f520b284700 1 -- 172.21.15.7:0/1462968597 wait complete. 2023-03-03T20:32:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.582+0000 7f520b284700 1 Processor -- start 2023-03-03T20:32:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.582+0000 7f520b284700 1 -- start start 2023-03-03T20:32:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f520b284700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f0003a70 0x7f52041ed1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f520b284700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52041ed700 0x7f52041eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f520b284700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520414e6a0 0x7f52041eb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f520b284700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040bea60 con 0x7f52041ed700 2023-03-03T20:32:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f520b284700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52040aa470 con 0x7f520414e6a0 2023-03-03T20:32:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f520b284700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52040be430 con 0x7f51f0003a70 2023-03-03T20:32:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f520881f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52041ed700 0x7f52041eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f520881f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52041ed700 0x7f52041eb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46210/0 (socket says 172.21.15.7:46210) 2023-03-03T20:32:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f520881f700 1 -- 172.21.15.7:0/2975780533 learned_addr learned my addr 172.21.15.7:0/2975780533 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:32:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f5209020700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f0003a70 0x7f52041ed1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.583+0000 7f5209821700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520414e6a0 0x7f52041eb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f520881f700 1 -- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f0003a70 msgr2=0x7f52041ed1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f520881f700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f0003a70 0x7f52041ed1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f520881f700 1 -- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520414e6a0 msgr2=0x7f52041eb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f520881f700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520414e6a0 0x7f52041eb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f520881f700 1 -- 172.21.15.7:0/2975780533 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520406d0c0 con 0x7f52041ed700 2023-03-03T20:32:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f5209821700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520414e6a0 0x7f52041eb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-03T20:32:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f520881f700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52041ed700 0x7f52041eb330 secure :-1 s=READY pgs=3826 cs=0 l=1 rev1=1 crypto rx=0x7f51f4011ad0 tx=0x7f51f4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:32:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f51f9ffb700 1 -- 172.21.15.7:0/2975780533 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f400fa60 con 0x7f52041ed700 2023-03-03T20:32:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f51f9ffb700 1 -- 172.21.15.7:0/2975780533 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f400fbc0 con 0x7f52041ed700 2023-03-03T20:32:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f520b284700 1 -- 172.21.15.7:0/2975780533 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520406cde0 con 0x7f52041ed700 2023-03-03T20:32:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f51f9ffb700 1 -- 172.21.15.7:0/2975780533 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f4014440 con 0x7f52041ed700 2023-03-03T20:32:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.584+0000 7f520b284700 1 -- 172.21.15.7:0/2975780533 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520406bea0 con 0x7f52041ed700 2023-03-03T20:32:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.585+0000 7f520b284700 1 -- 172.21.15.7:0/2975780533 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5204071b00 con 0x7f52041ed700 2023-03-03T20:32:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.586+0000 7f51f9ffb700 1 -- 172.21.15.7:0/2975780533 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f51f40145a0 con 0x7f52041ed700 2023-03-03T20:32:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.586+0000 7f51f9ffb700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51e80682f0 0x7f51e806a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:32:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.586+0000 7f51f9ffb700 1 -- 172.21.15.7:0/2975780533 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f51f40869a0 con 0x7f52041ed700 2023-03-03T20:32:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.586+0000 7f5209020700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51e80682f0 0x7f51e806a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:32:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.587+0000 7f5209020700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51e80682f0 0x7f51e806a7a0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f52000090b0 tx=0x7f5200008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:32:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.591+0000 7f51f9ffb700 1 -- 172.21.15.7:0/2975780533 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f51f4052d40 con 0x7f52041ed700 2023-03-03T20:32:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.828+0000 7f520b284700 1 -- 172.21.15.7:0/2975780533 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5204074bb0 con 0x7f52041ed700 2023-03-03T20:32:47.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.829+0000 7f51f9ffb700 1 -- 172.21.15.7:0/2975780533 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f51f4056360 con 0x7f52041ed700 2023-03-03T20:32:47.830 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:32:47.830 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:32:47.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 -- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51e80682f0 msgr2=0x7f51e806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51e80682f0 0x7f51e806a7a0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f52000090b0 tx=0x7f5200008030 comp rx=0 tx=0).stop 2023-03-03T20:32:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 -- 172.21.15.7:0/2975780533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52041ed700 msgr2=0x7f52041eb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:32:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52041ed700 0x7f52041eb330 secure :-1 s=READY pgs=3826 cs=0 l=1 rev1=1 crypto rx=0x7f51f4011ad0 tx=0x7f51f4011e90 comp rx=0 tx=0).stop 2023-03-03T20:32:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 -- 172.21.15.7:0/2975780533 shutdown_connections 2023-03-03T20:32:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f520414e6a0 0x7f52041eb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f51f0003a70 0x7f52041ed1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:47.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f51e80682f0 0x7f51e806a7a0 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:47.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 --2- 172.21.15.7:0/2975780533 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52041ed700 0x7f52041eb330 unknown :-1 s=CLOSED pgs=3826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:32:47.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 -- 172.21.15.7:0/2975780533 >> 172.21.15.7:0/2975780533 conn(0x7f52040a94d0 msgr2=0x7f52040c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:32:47.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 -- 172.21.15.7:0/2975780533 shutdown_connections 2023-03-03T20:32:47.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:32:47.832+0000 7f520b284700 1 -- 172.21.15.7:0/2975780533 wait complete. 2023-03-03T20:32:47.834 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:32:47.844 INFO:tasks.mds_thrash.fs.[b]:waiting for 25 secs before thrashing 2023-03-03T20:33:01.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:33:01.254 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:33:07.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:33:07.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.629+0000 7f83fe049700 1 Processor -- start 2023-03-03T20:33:07.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.629+0000 7f83fe049700 1 -- start start 2023-03-03T20:33:07.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83fe049700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f83f80caa30 0x7f83f80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:07.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83fe049700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f83f8154ff0 0x7f83f80cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:07.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83fe049700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f80c7310 con 0x7f83f80c00e0 2023-03-03T20:33:07.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83fe049700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f8155430 con 0x7f83f80caa30 2023-03-03T20:33:07.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83fe049700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f80bfd10 con 0x7f83f8154ff0 2023-03-03T20:33:07.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83f77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f83f80caa30 0x7f83f80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:07.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83f77fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f83f80caa30 0x7f83f80c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46230/0 (socket says 172.21.15.7:46230) 2023-03-03T20:33:07.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83f77fe700 1 -- 172.21.15.7:0/3646387711 learned_addr learned my addr 172.21.15.7:0/3646387711 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:33:07.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83f7fff700 1 --2- 172.21.15.7:0/3646387711 >> v2:172.21.15.143:3300/0 conn(0x7f83f8154ff0 0x7f83f80cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:07.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83f77fe700 1 -- 172.21.15.7:0/3646387711 >> v2:172.21.15.143:3300/0 conn(0x7f83f8154ff0 msgr2=0x7f83f80cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:33:07.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83f77fe700 1 --2- 172.21.15.7:0/3646387711 >> v2:172.21.15.143:3300/0 conn(0x7f83f8154ff0 0x7f83f80cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83f77fe700 1 -- 172.21.15.7:0/3646387711 >> v1:172.21.15.143:6789/0 conn(0x7f83f80c00e0 legacy=0x7f83f80c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:07.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.630+0000 7f83f77fe700 1 -- 172.21.15.7:0/3646387711 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f80bfa30 con 0x7f83f80caa30 2023-03-03T20:33:07.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83f77fe700 1 --2- 172.21.15.7:0/3646387711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f80caa30 0x7f83f80c5c60 secure :-1 s=READY pgs=3827 cs=0 l=1 rev1=1 crypto rx=0x7f83f8071530 tx=0x7f83e8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e7604a392af13 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:33:07.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83f67fc700 1 -- 172.21.15.7:0/3646387711 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e8011830 con 0x7f83f80caa30 2023-03-03T20:33:07.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83f67fc700 1 -- 172.21.15.7:0/3646387711 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e8011990 con 0x7f83f80caa30 2023-03-03T20:33:07.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83f67fc700 1 -- 172.21.15.7:0/3646387711 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e8011d20 con 0x7f83f80caa30 2023-03-03T20:33:07.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83fe049700 1 -- 172.21.15.7:0/3646387711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f80caa30 msgr2=0x7f83f80c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:07.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83fe049700 1 --2- 172.21.15.7:0/3646387711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f80caa30 0x7f83f80c5c60 secure :-1 s=READY pgs=3827 cs=0 l=1 rev1=1 crypto rx=0x7f83f8071530 tx=0x7f83e8009bd0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83fe049700 1 -- 172.21.15.7:0/3646387711 shutdown_connections 2023-03-03T20:33:07.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83fe049700 1 --2- 172.21.15.7:0/3646387711 >> v2:172.21.15.143:3300/0 conn(0x7f83f8154ff0 0x7f83f80cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83fe049700 1 --2- 172.21.15.7:0/3646387711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f80caa30 0x7f83f80c5c60 unknown :-1 s=CLOSED pgs=3827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83fe049700 1 -- 172.21.15.7:0/3646387711 >> 172.21.15.7:0/3646387711 conn(0x7f83f8066e40 msgr2=0x7f83f8071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:33:07.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83fe049700 1 -- 172.21.15.7:0/3646387711 shutdown_connections 2023-03-03T20:33:07.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.631+0000 7f83fe049700 1 -- 172.21.15.7:0/3646387711 wait complete. 2023-03-03T20:33:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.633+0000 7f83fe049700 1 Processor -- start 2023-03-03T20:33:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.633+0000 7f83fe049700 1 -- start start 2023-03-03T20:33:07.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.633+0000 7f83fe049700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83f80c00e0 0x7f83f814a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.633+0000 7f83fe049700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83f80caa30 0x7f83f814a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.633+0000 7f83fe049700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f8154ff0 0x7f83f81ed9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.633+0000 7f83fe049700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f814ba30 con 0x7f83f8154ff0 2023-03-03T20:33:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.633+0000 7f83fe049700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f8157550 con 0x7f83f80caa30 2023-03-03T20:33:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.633+0000 7f83fe049700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f814ab60 con 0x7f83f80c00e0 2023-03-03T20:33:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83f7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f8154ff0 0x7f83f81ed9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83f7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f8154ff0 0x7f83f81ed9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46240/0 (socket says 172.21.15.7:46240) 2023-03-03T20:33:07.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83f7fff700 1 -- 172.21.15.7:0/993670059 learned_addr learned my addr 172.21.15.7:0/993670059 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:33:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83f7fff700 1 -- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83f80c00e0 msgr2=0x7f83f814a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83f7fff700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83f80c00e0 0x7f83f814a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83f7fff700 1 -- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83f80caa30 msgr2=0x7f83f814a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83f7fff700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83f80caa30 0x7f83f814a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83f7fff700 1 -- 172.21.15.7:0/993670059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e8009550 con 0x7f83f8154ff0 2023-03-03T20:33:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83f7fff700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f8154ff0 0x7f83f81ed9c0 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 crypto rx=0x7f83f000e800 tx=0x7f83f000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:33:07.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83dffff700 1 -- 172.21.15.7:0/993670059 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f00095b0 con 0x7f83f8154ff0 2023-03-03T20:33:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83dffff700 1 -- 172.21.15.7:0/993670059 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f0013680 con 0x7f83f8154ff0 2023-03-03T20:33:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83dffff700 1 -- 172.21.15.7:0/993670059 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f0013a10 con 0x7f83f8154ff0 2023-03-03T20:33:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83fe049700 1 -- 172.21.15.7:0/993670059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f81547f0 con 0x7f83f8154ff0 2023-03-03T20:33:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.634+0000 7f83fe049700 1 -- 172.21.15.7:0/993670059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f8154510 con 0x7f83f8154ff0 2023-03-03T20:33:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.635+0000 7f83fe049700 1 -- 172.21.15.7:0/993670059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f8154ad0 con 0x7f83f8154ff0 2023-03-03T20:33:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.636+0000 7f83dffff700 1 -- 172.21.15.7:0/993670059 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f83f0019030 con 0x7f83f8154ff0 2023-03-03T20:33:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.636+0000 7f83dffff700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 0x7f83e406a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:07.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.636+0000 7f83dffff700 1 -- 172.21.15.7:0/993670059 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f83f0085fa0 con 0x7f83f8154ff0 2023-03-03T20:33:07.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.637+0000 7f83f77fe700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 0x7f83e406a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:07.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.637+0000 7f83f77fe700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 0x7f83e406a7a0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009fd0 tx=0x7f83e8010000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:33:07.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.641+0000 7f83dffff700 1 -- 172.21.15.7:0/993670059 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f83f00522e0 con 0x7f83f8154ff0 2023-03-03T20:33:07.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.876+0000 7f83fe049700 1 -- 172.21.15.7:0/993670059 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83f8074bb0 con 0x7f83f8154ff0 2023-03-03T20:33:07.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.877+0000 7f83dffff700 1 -- 172.21.15.7:0/993670059 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f83f0055900 con 0x7f83f8154ff0 2023-03-03T20:33:07.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:33:07.878 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:33:07.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 -- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 msgr2=0x7f83e406a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 0x7f83e406a7a0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009fd0 tx=0x7f83e8010000 comp rx=0 tx=0).stop 2023-03-03T20:33:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 -- 172.21.15.7:0/993670059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f8154ff0 msgr2=0x7f83f81ed9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f8154ff0 0x7f83f81ed9c0 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 crypto rx=0x7f83f000e800 tx=0x7f83f000ebc0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 -- 172.21.15.7:0/993670059 shutdown_connections 2023-03-03T20:33:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f83e40682f0 0x7f83e406a7a0 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83f8154ff0 0x7f83f81ed9c0 unknown :-1 s=CLOSED pgs=3828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f83f80c00e0 0x7f83f814a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 --2- 172.21.15.7:0/993670059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83f80caa30 0x7f83f814a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:07.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 -- 172.21.15.7:0/993670059 >> 172.21.15.7:0/993670059 conn(0x7f83f8066e40 msgr2=0x7f83f80736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:33:07.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 -- 172.21.15.7:0/993670059 shutdown_connections 2023-03-03T20:33:07.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:07.879+0000 7f83fe049700 1 -- 172.21.15.7:0/993670059 wait complete. 2023-03-03T20:33:07.881 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:33:07.892 INFO:tasks.mds_thrash.fs.[a]:waiting for 98 secs before thrashing 2023-03-03T20:33:12.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:33:12.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.985+0000 7f6e65b9a700 1 Processor -- start 2023-03-03T20:33:12.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.985+0000 7f6e65b9a700 1 -- start start 2023-03-03T20:33:12.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e65b9a700 1 --2- >> v2:172.21.15.143:3300/0 conn(0x7f6e600c00e0 0x7f6e600c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:12.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e65b9a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e60154ff0 0x7f6e600cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:12.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e65b9a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e600c7310 con 0x7f6e600caa30 2023-03-03T20:33:12.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e65b9a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e60155430 con 0x7f6e60154ff0 2023-03-03T20:33:12.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e65b9a700 1 -- --> v2:172.21.15.143:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e600bfd10 con 0x7f6e600c00e0 2023-03-03T20:33:12.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e5ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6e60154ff0 0x7f6e600cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:12.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e5f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6e600caa30 0x7f6e600c6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40228/0 (socket says 172.21.15.7:40228) 2023-03-03T20:33:12.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e5f7fe700 1 -- 172.21.15.7:0/3925410971 learned_addr learned my addr 172.21.15.7:0/3925410971 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-03-03T20:33:12.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e5effd700 1 --2- 172.21.15.7:0/3925410971 >> v2:172.21.15.143:3300/0 conn(0x7f6e600c00e0 0x7f6e600c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:12.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e5ffff700 1 -- 172.21.15.7:0/3925410971 >> v2:172.21.15.143:3300/0 conn(0x7f6e600c00e0 msgr2=0x7f6e600c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:33:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e5ffff700 1 --2- 172.21.15.7:0/3925410971 >> v2:172.21.15.143:3300/0 conn(0x7f6e600c00e0 0x7f6e600c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e5ffff700 1 -- 172.21.15.7:0/3925410971 >> v1:172.21.15.7:6789/0 conn(0x7f6e600caa30 legacy=0x7f6e600c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e5e7fc700 1 -- 172.21.15.7:0/3925410971 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594718363 0 0) 0x7f6e600c7310 con 0x7f6e600caa30 2023-03-03T20:33:12.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.986+0000 7f6e5ffff700 1 -- 172.21.15.7:0/3925410971 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e600bfa30 con 0x7f6e60154ff0 2023-03-03T20:33:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e5ffff700 1 --2- 172.21.15.7:0/3925410971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 0x7f6e600cae00 secure :-1 s=READY pgs=3830 cs=0 l=1 rev1=1 crypto rx=0x7f6e600ca910 tx=0x7f6e5800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ddb657e665c78f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:33:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e5e7fc700 1 -- 172.21.15.7:0/3925410971 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5800e070 con 0x7f6e60154ff0 2023-03-03T20:33:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e5e7fc700 1 -- 172.21.15.7:0/3925410971 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5800bbd0 con 0x7f6e60154ff0 2023-03-03T20:33:12.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e5e7fc700 1 -- 172.21.15.7:0/3925410971 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58012430 con 0x7f6e60154ff0 2023-03-03T20:33:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e65b9a700 1 -- 172.21.15.7:0/3925410971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 msgr2=0x7f6e600cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e65b9a700 1 --2- 172.21.15.7:0/3925410971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 0x7f6e600cae00 secure :-1 s=READY pgs=3830 cs=0 l=1 rev1=1 crypto rx=0x7f6e600ca910 tx=0x7f6e5800b330 comp rx=0 tx=0).stop 2023-03-03T20:33:12.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e65b9a700 1 -- 172.21.15.7:0/3925410971 shutdown_connections 2023-03-03T20:33:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e65b9a700 1 --2- 172.21.15.7:0/3925410971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 0x7f6e600cae00 unknown :-1 s=CLOSED pgs=3830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e65b9a700 1 --2- 172.21.15.7:0/3925410971 >> v2:172.21.15.143:3300/0 conn(0x7f6e600c00e0 0x7f6e600c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e65b9a700 1 -- 172.21.15.7:0/3925410971 >> 172.21.15.7:0/3925410971 conn(0x7f6e60066e40 msgr2=0x7f6e60071580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:33:12.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e65b9a700 1 -- 172.21.15.7:0/3925410971 shutdown_connections 2023-03-03T20:33:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.987+0000 7f6e65b9a700 1 -- 172.21.15.7:0/3925410971 wait complete. 2023-03-03T20:33:12.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.989+0000 7f6e65b9a700 1 Processor -- start 2023-03-03T20:33:13.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.989+0000 7f6e65b9a700 1 -- start start 2023-03-03T20:33:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e65b9a700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e600c00e0 0x7f6e600c2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e65b9a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e600caa30 0x7f6e601e9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:13.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e65b9a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 0x7f6e6014c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e65b9a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e60147010 con 0x7f6e60154ff0 2023-03-03T20:33:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e65b9a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e601ea410 con 0x7f6e600caa30 2023-03-03T20:33:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e65b9a700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6014d560 con 0x7f6e600c00e0 2023-03-03T20:33:13.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e5ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 0x7f6e6014c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e5ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 0x7f6e6014c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46258/0 (socket says 172.21.15.7:46258) 2023-03-03T20:33:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e5ffff700 1 -- 172.21.15.7:0/431936233 learned_addr learned my addr 172.21.15.7:0/431936233 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:33:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e5effd700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e600caa30 0x7f6e601e9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:13.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.990+0000 7f6e5f7fe700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e600c00e0 0x7f6e600c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e5ffff700 1 -- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e600c00e0 msgr2=0x7f6e600c2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e5ffff700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e600c00e0 0x7f6e600c2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e5ffff700 1 -- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e600caa30 msgr2=0x7f6e601e9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:13.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e5ffff700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e600caa30 0x7f6e601e9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e5ffff700 1 -- 172.21.15.7:0/431936233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e58009a70 con 0x7f6e60154ff0 2023-03-03T20:33:13.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e5ffff700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 0x7f6e6014c980 secure :-1 s=READY pgs=3831 cs=0 l=1 rev1=1 crypto rx=0x7f6e6006b270 tx=0x7f6e58004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:33:13.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e57fff700 1 -- 172.21.15.7:0/431936233 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58012430 con 0x7f6e60154ff0 2023-03-03T20:33:13.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e57fff700 1 -- 172.21.15.7:0/431936233 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e58020070 con 0x7f6e60154ff0 2023-03-03T20:33:13.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e65b9a700 1 -- 172.21.15.7:0/431936233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e601547f0 con 0x7f6e60154ff0 2023-03-03T20:33:13.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e57fff700 1 -- 172.21.15.7:0/431936233 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58004280 con 0x7f6e60154ff0 2023-03-03T20:33:13.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.991+0000 7f6e65b9a700 1 -- 172.21.15.7:0/431936233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e60154510 con 0x7f6e60154ff0 2023-03-03T20:33:13.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.992+0000 7f6e65b9a700 1 -- 172.21.15.7:0/431936233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e60154ad0 con 0x7f6e60154ff0 2023-03-03T20:33:13.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.992+0000 7f6e57fff700 1 -- 172.21.15.7:0/431936233 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f6e580043e0 con 0x7f6e60154ff0 2023-03-03T20:33:13.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.993+0000 7f6e57fff700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6e400682f0 0x7f6e4006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:13.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.993+0000 7f6e57fff700 1 -- 172.21.15.7:0/431936233 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7f6e58022070 con 0x7f6e60154ff0 2023-03-03T20:33:13.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.993+0000 7f6e5f7fe700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6e400682f0 0x7f6e4006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:13.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.994+0000 7f6e5f7fe700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6e400682f0 0x7f6e4006a7a0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f6e50007fd0 tx=0x7f6e50007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:33:13.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:12.997+0000 7f6e57fff700 1 -- 172.21.15.7:0/431936233 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7f6e58054550 con 0x7f6e60154ff0 2023-03-03T20:33:13.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.234+0000 7f6e65b9a700 1 -- 172.21.15.7:0/431936233 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e600c1f40 con 0x7f6e60154ff0 2023-03-03T20:33:13.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.235+0000 7f6e57fff700 1 -- 172.21.15.7:0/431936233 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7f6e600c1f40 con 0x7f6e60154ff0 2023-03-03T20:33:13.236 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:33:13.236 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:33:13.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 -- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6e400682f0 msgr2=0x7f6e4006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:13.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6e400682f0 0x7f6e4006a7a0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f6e50007fd0 tx=0x7f6e50007de0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 -- 172.21.15.7:0/431936233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 msgr2=0x7f6e6014c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:13.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 0x7f6e6014c980 secure :-1 s=READY pgs=3831 cs=0 l=1 rev1=1 crypto rx=0x7f6e6006b270 tx=0x7f6e58004ad0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 -- 172.21.15.7:0/431936233 shutdown_connections 2023-03-03T20:33:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7f6e400682f0 0x7f6e4006a7a0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e60154ff0 0x7f6e6014c980 unknown :-1 s=CLOSED pgs=3831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7f6e600c00e0 0x7f6e600c2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 --2- 172.21.15.7:0/431936233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e600caa30 0x7f6e601e9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 -- 172.21.15.7:0/431936233 >> 172.21.15.7:0/431936233 conn(0x7f6e60066e40 msgr2=0x7f6e60151230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:33:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 -- 172.21.15.7:0/431936233 shutdown_connections 2023-03-03T20:33:13.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.237+0000 7f6e65b9a700 1 -- 172.21.15.7:0/431936233 wait complete. 2023-03-03T20:33:13.240 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:33:13.250 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-03T20:33:13.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.374+0000 7fe0390ee700 1 Processor -- start 2023-03-03T20:33:13.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.374+0000 7fe0390ee700 1 -- start start 2023-03-03T20:33:13.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe0390ee700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe03413a0f0 0x7fe0340c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:13.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe0390ee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0340c1da0 0x7fe03406d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:13.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe0390ee700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe03406de90 con 0x7fe0341586c0 2023-03-03T20:33:13.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe0390ee700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe034154840 con 0x7fe03413a0f0 2023-03-03T20:33:13.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe0390ee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe034154560 con 0x7fe0340c1da0 2023-03-03T20:33:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe032d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe03413a0f0 0x7fe0340c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe032d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe03413a0f0 0x7fe0340c07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46262/0 (socket says 172.21.15.7:46262) 2023-03-03T20:33:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe032d9d700 1 -- 172.21.15.7:0/3480209029 learned_addr learned my addr 172.21.15.7:0/3480209029 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:33:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe03359e700 1 --2- 172.21.15.7:0/3480209029 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0340c1da0 0x7fe03406d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe032d9d700 1 -- 172.21.15.7:0/3480209029 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0340c1da0 msgr2=0x7fe03406d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-03T20:33:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe032d9d700 1 --2- 172.21.15.7:0/3480209029 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0340c1da0 0x7fe03406d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe032d9d700 1 -- 172.21.15.7:0/3480209029 >> v1:172.21.15.143:6789/0 conn(0x7fe0341586c0 legacy=0x7fe03406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:13.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe032d9d700 1 -- 172.21.15.7:0/3480209029 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe034157be0 con 0x7fe03413a0f0 2023-03-03T20:33:13.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.375+0000 7fe032d9d700 1 --2- 172.21.15.7:0/3480209029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 0x7fe0340c07d0 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 crypto rx=0x7fe0280098f0 tx=0x7fe028009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19625ad72aeae47c server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:33:13.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe031d9b700 1 -- 172.21.15.7:0/3480209029 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02800b8a0 con 0x7fe03413a0f0 2023-03-03T20:33:13.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe031d9b700 1 -- 172.21.15.7:0/3480209029 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02800ba00 con 0x7fe03413a0f0 2023-03-03T20:33:13.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe031d9b700 1 -- 172.21.15.7:0/3480209029 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02800bd90 con 0x7fe03413a0f0 2023-03-03T20:33:13.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe0390ee700 1 -- 172.21.15.7:0/3480209029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 msgr2=0x7fe0340c07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:13.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe0390ee700 1 --2- 172.21.15.7:0/3480209029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 0x7fe0340c07d0 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 crypto rx=0x7fe0280098f0 tx=0x7fe028009c00 comp rx=0 tx=0).stop 2023-03-03T20:33:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe0390ee700 1 -- 172.21.15.7:0/3480209029 shutdown_connections 2023-03-03T20:33:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe0390ee700 1 --2- 172.21.15.7:0/3480209029 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0340c1da0 0x7fe03406d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe0390ee700 1 --2- 172.21.15.7:0/3480209029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 0x7fe0340c07d0 unknown :-1 s=CLOSED pgs=3832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe0390ee700 1 -- 172.21.15.7:0/3480209029 >> 172.21.15.7:0/3480209029 conn(0x7fe03406c110 msgr2=0x7fe03406c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:33:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe0390ee700 1 -- 172.21.15.7:0/3480209029 shutdown_connections 2023-03-03T20:33:13.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.376+0000 7fe0390ee700 1 -- 172.21.15.7:0/3480209029 wait complete. 2023-03-03T20:33:13.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.378+0000 7fe0390ee700 1 Processor -- start 2023-03-03T20:33:13.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.378+0000 7fe0390ee700 1 -- start start 2023-03-03T20:33:13.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.378+0000 7fe0390ee700 1 --2- >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0340c1da0 0x7fe0340aa700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:13.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.378+0000 7fe0390ee700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 0x7fe0340c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:13.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.378+0000 7fe0390ee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0341586c0 0x7fe0340c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:13.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.378+0000 7fe0390ee700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe034071970 con 0x7fe03413a0f0 2023-03-03T20:33:13.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.378+0000 7fe0390ee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe034070ae0 con 0x7fe0341586c0 2023-03-03T20:33:13.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.378+0000 7fe0390ee700 1 -- --> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe034070d60 con 0x7fe0340c1da0 2023-03-03T20:33:13.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe03259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 0x7fe0340c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:13.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe03259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 0x7fe0340c6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46270/0 (socket says 172.21.15.7:46270) 2023-03-03T20:33:13.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe03259c700 1 -- 172.21.15.7:0/3428124506 learned_addr learned my addr 172.21.15.7:0/3428124506 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-03-03T20:33:13.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe03359e700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0341586c0 0x7fe0340c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:13.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe032d9d700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0340c1da0 0x7fe0340aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:13.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe03259c700 1 -- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0340c1da0 msgr2=0x7fe0340aa700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:13.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe03259c700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0340c1da0 0x7fe0340aa700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe03259c700 1 -- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0341586c0 msgr2=0x7fe0340c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:13.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe03259c700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0341586c0 0x7fe0340c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe03259c700 1 -- 172.21.15.7:0/3428124506 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe028009550 con 0x7fe03413a0f0 2023-03-03T20:33:13.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe03259c700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 0x7fe0340c6b00 secure :-1 s=READY pgs=3833 cs=0 l=1 rev1=1 crypto rx=0x7fe02400c4e0 tx=0x7fe02400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:33:13.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe01b7fe700 1 -- 172.21.15.7:0/3428124506 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0240052d0 con 0x7fe03413a0f0 2023-03-03T20:33:13.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe01b7fe700 1 -- 172.21.15.7:0/3428124506 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe024005430 con 0x7fe03413a0f0 2023-03-03T20:33:13.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe0390ee700 1 -- 172.21.15.7:0/3428124506 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0341581a0 con 0x7fe03413a0f0 2023-03-03T20:33:13.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe01b7fe700 1 -- 172.21.15.7:0/3428124506 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0240038c0 con 0x7fe03413a0f0 2023-03-03T20:33:13.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.379+0000 7fe0390ee700 1 -- 172.21.15.7:0/3428124506 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe034157ec0 con 0x7fe03413a0f0 2023-03-03T20:33:13.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.380+0000 7fe0390ee700 1 -- 172.21.15.7:0/3428124506 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03406e1a0 con 0x7fe03413a0f0 2023-03-03T20:33:13.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.381+0000 7fe01b7fe700 1 -- 172.21.15.7:0/3428124506 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fe024019030 con 0x7fe03413a0f0 2023-03-03T20:33:13.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.381+0000 7fe01b7fe700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0200682f0 0x7fe02006a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-03T20:33:13.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.381+0000 7fe01b7fe700 1 -- 172.21.15.7:0/3428124506 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7276+0+0 (secure 0 0 0) 0x7fe0240851f0 con 0x7fe03413a0f0 2023-03-03T20:33:13.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.381+0000 7fe032d9d700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0200682f0 0x7fe02006a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-03T20:33:13.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.382+0000 7fe032d9d700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0200682f0 0x7fe02006a7a0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7fe028009c40 tx=0x7fe028008030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-03T20:33:13.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.386+0000 7fe01b7fe700 1 -- 172.21.15.7:0/3428124506 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173508 (secure 0 0 0) 0x7fe024051590 con 0x7fe03413a0f0 2023-03-03T20:33:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.626+0000 7fe0390ee700 1 -- 172.21.15.7:0/3428124506 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe03406de90 con 0x7fe03413a0f0 2023-03-03T20:33:13.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.627+0000 7fe01b7fe700 1 -- 172.21.15.7:0/3428124506 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5741 (secure 0 0 0) 0x7fe03406de90 con 0x7fe03413a0f0 2023-03-03T20:33:13.627 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-03-03T20:33:13.628 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/2988899186","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":2988899186},{"type":"v1","addr":"172.21.15.143:6835","nonce":2988899186}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4769,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/1716699873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":1716699873},{"type":"v1","addr":"172.21.15.143:6837","nonce":1716699873}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":4897,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6834/4107062379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4107062379},{"type":"v1","addr":"172.21.15.7:6834","nonce":4107062379}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5178,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/121229783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6833","nonce":121229783},{"type":"v1","addr":"172.21.15.7:6835","nonce":121229783}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:34.516962+0000","modified":"2023-03-03T08:45:11.325826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-03T08:43:43.971492+0000","modified":"2023-03-03T08:45:32.315314+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-03-03T20:33:13.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 -- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0200682f0 msgr2=0x7fe02006a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:13.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0200682f0 0x7fe02006a7a0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7fe028009c40 tx=0x7fe028008030 comp rx=0 tx=0).stop 2023-03-03T20:33:13.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 -- 172.21.15.7:0/3428124506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 msgr2=0x7fe0340c6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-03T20:33:13.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 0x7fe0340c6b00 secure :-1 s=READY pgs=3833 cs=0 l=1 rev1=1 crypto rx=0x7fe02400c4e0 tx=0x7fe02400c7f0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 -- 172.21.15.7:0/3428124506 shutdown_connections 2023-03-03T20:33:13.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0341586c0 0x7fe0340c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:3301/0,v1:172.21.15.143:6790/0] conn(0x7fe0340c1da0 0x7fe0340aa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.143:6832/96367,v1:172.21.15.143:6833/96367] conn(0x7fe0200682f0 0x7fe02006a7a0 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 --2- 172.21.15.7:0/3428124506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe03413a0f0 0x7fe0340c6b00 unknown :-1 s=CLOSED pgs=3833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-03T20:33:13.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 -- 172.21.15.7:0/3428124506 >> 172.21.15.7:0/3428124506 conn(0x7fe03406c110 msgr2=0x7fe034139150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-03T20:33:13.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 -- 172.21.15.7:0/3428124506 shutdown_connections 2023-03-03T20:33:13.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-03-03T20:33:13.629+0000 7fe0390ee700 1 -- 172.21.15.7:0/3428124506 wait complete. 2023-03-03T20:33:13.633 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2023-03-03T20:33:13.642 INFO:tasks.mds_thrash.fs.[b]:waiting for 44 secs before thrashing 2023-03-03T20:33:31.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:33:31.346 DEBUG:teuthology.orchestra.run.smithi143:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-03T20:33:37.907 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-03-03T20:33:37.908 DEBUG:teuthology.task.console_log:Killing console logger for smithi007 2023-03-03T20:33:37.910 DEBUG:teuthology.task.console_log:Killing console logger for smithi143 2023-03-03T20:33:37.910 DEBUG:teuthology.exit:Finished running handlers